Merge LA.UM.7.8.9.C1.08.00.00.516.007 branch via qcom-msm-4.9 into android-msm-bluecross-4.9
Including 'qcom/release/LA.UM.7.8.9.08.00.00.478.137 in this merge.
git checkout partner/android-msm-bluecross-4.9 -- drivers/media/platform/msm/camera/ \
drivers/media/platform/msm/camera_v2/ \
include/uapi/media/cam_req_mgr.h
Conflicts:
arch/arm64/boot/dts/qcom/sdm845-670-usb-common.dtsi
block/bio.c
block/blk-core.c
block/blk-merge.c
drivers/char/adsprpc.c
drivers/gpio/gpio-msm-smp2p-test.c
drivers/gpu/drm/msm/sde/sde_crtc.c
drivers/gpu/drm/msm/sde/sde_encoder.c
drivers/media/platform/msm/camera/cam_core/cam_context.c
drivers/media/platform/msm/camera/cam_core/cam_node.c
drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/cam_ife_hw_mgr.c
drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_core.c
drivers/media/platform/msm/camera/cam_sensor_module/cam_ois/cam_ois_core.c
drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_util.c
drivers/media/platform/msm/camera_v2/sensor/csiphy/msm_csiphy.c
drivers/regulator/qpnp-labibb-regulator.c
drivers/usb/dwc3/dwc3-msm.c
fs/crypto/Makefile
fs/crypto/fscrypt_ice.h
fs/crypto/fscrypt_private.h
fs/crypto/keyinfo.c
fs/direct-io.c
fs/ext4/super.c
fs/f2fs/data.c
fs/f2fs/f2fs.h
fs/f2fs/file.c
fs/f2fs/segment.c
fs/f2fs/super.c
fs/f2fs/sysfs.c
include/linux/blk_types.h
include/linux/bvec.h
include/linux/fscrypt.h
include/linux/fscrypt_supp.h
kernel/events/core.c
net/packet/af_packet.c
security/pfe/pfk.c
security/pfe/pfk_ext4.c
security/pfe/pfk_fscrypt.c
Bug: 121228685
Signed-off-by: Wilson Sung <wilsonsung@google.com>
Change-Id: I20b3f0f5170a1f27e9424cd1394764d3ea267e33
diff --git a/.gitignore b/.gitignore
index 4105cfb..30c8898 100644
--- a/.gitignore
+++ b/.gitignore
@@ -118,3 +118,6 @@
# fetched Android config fragments
kernel/configs/android-*.cfg
+
+# Qualcomm out-of-tree audio drivers
+/techpack/audio
diff --git a/Documentation/ABI/obsolete/sysfs-block-zram b/Documentation/ABI/obsolete/sysfs-block-zram
deleted file mode 100644
index 720ea92..0000000
--- a/Documentation/ABI/obsolete/sysfs-block-zram
+++ /dev/null
@@ -1,119 +0,0 @@
-What: /sys/block/zram<id>/num_reads
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The num_reads file is read-only and specifies the number of
- reads (failed or successful) done on this device.
- Now accessible via zram<id>/stat node.
-
-What: /sys/block/zram<id>/num_writes
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The num_writes file is read-only and specifies the number of
- writes (failed or successful) done on this device.
- Now accessible via zram<id>/stat node.
-
-What: /sys/block/zram<id>/invalid_io
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The invalid_io file is read-only and specifies the number of
- non-page-size-aligned I/O requests issued to this device.
- Now accessible via zram<id>/io_stat node.
-
-What: /sys/block/zram<id>/failed_reads
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The failed_reads file is read-only and specifies the number of
- failed reads happened on this device.
- Now accessible via zram<id>/io_stat node.
-
-What: /sys/block/zram<id>/failed_writes
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The failed_writes file is read-only and specifies the number of
- failed writes happened on this device.
- Now accessible via zram<id>/io_stat node.
-
-What: /sys/block/zram<id>/notify_free
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The notify_free file is read-only. Depending on device usage
- scenario it may account a) the number of pages freed because
- of swap slot free notifications or b) the number of pages freed
- because of REQ_DISCARD requests sent by bio. The former ones
- are sent to a swap block device when a swap slot is freed, which
- implies that this disk is being used as a swap disk. The latter
- ones are sent by filesystem mounted with discard option,
- whenever some data blocks are getting discarded.
- Now accessible via zram<id>/io_stat node.
-
-What: /sys/block/zram<id>/zero_pages
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The zero_pages file is read-only and specifies number of zero
- filled pages written to this disk. No memory is allocated for
- such pages.
- Now accessible via zram<id>/mm_stat node.
-
-What: /sys/block/zram<id>/orig_data_size
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The orig_data_size file is read-only and specifies uncompressed
- size of data stored in this disk. This excludes zero-filled
- pages (zero_pages) since no memory is allocated for them.
- Unit: bytes
- Now accessible via zram<id>/mm_stat node.
-
-What: /sys/block/zram<id>/compr_data_size
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The compr_data_size file is read-only and specifies compressed
- size of data stored in this disk. So, compression ratio can be
- calculated using orig_data_size and this statistic.
- Unit: bytes
- Now accessible via zram<id>/mm_stat node.
-
-What: /sys/block/zram<id>/mem_used_total
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The mem_used_total file is read-only and specifies the amount
- of memory, including allocator fragmentation and metadata
- overhead, allocated for this disk. So, allocator space
- efficiency can be calculated using compr_data_size and this
- statistic.
- Unit: bytes
- Now accessible via zram<id>/mm_stat node.
-
-What: /sys/block/zram<id>/mem_used_max
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The mem_used_max file is read/write and specifies the amount
- of maximum memory zram have consumed to store compressed data.
- For resetting the value, you should write "0". Otherwise,
- you could see -EINVAL.
- Unit: bytes
- Downgraded to write-only node: so it's possible to set new
- value only; its current value is stored in zram<id>/mm_stat
- node.
-
-What: /sys/block/zram<id>/mem_limit
-Date: August 2015
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The mem_limit file is read/write and specifies the maximum
- amount of memory ZRAM can use to store the compressed data.
- The limit could be changed in run time and "0" means disable
- the limit. No limit is the initial state. Unit: bytes
- Downgraded to write-only node: so it's possible to set new
- value only; its current value is stored in zram<id>/mm_stat
- node.
diff --git a/Documentation/ABI/testing/procfs-smaps_rollup b/Documentation/ABI/testing/procfs-smaps_rollup
new file mode 100644
index 0000000..0a54ed0
--- /dev/null
+++ b/Documentation/ABI/testing/procfs-smaps_rollup
@@ -0,0 +1,31 @@
+What: /proc/pid/smaps_rollup
+Date: August 2017
+Contact: Daniel Colascione <dancol@google.com>
+Description:
+ This file provides pre-summed memory information for a
+ process. The format is identical to /proc/pid/smaps,
+ except instead of an entry for each VMA in a process,
+ smaps_rollup has a single entry (tagged "[rollup]")
+ for which each field is the sum of the corresponding
+ fields from all the maps in /proc/pid/smaps.
+ For more details, see the procfs man page.
+
+ Typical output looks like this:
+
+ 00100000-ff709000 ---p 00000000 00:00 0 [rollup]
+ Rss: 884 kB
+ Pss: 385 kB
+ Shared_Clean: 696 kB
+ Shared_Dirty: 0 kB
+ Private_Clean: 120 kB
+ Private_Dirty: 68 kB
+ Referenced: 884 kB
+ Anonymous: 68 kB
+ LazyFree: 0 kB
+ AnonHugePages: 0 kB
+ ShmemPmdMapped: 0 kB
+ Shared_Hugetlb: 0 kB
+ Private_Hugetlb: 0 kB
+ Swap: 0 kB
+ SwapPss: 0 kB
+ Locked: 385 kB
diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI/testing/sysfs-block-zram
index 4518d30..3dd19cc 100644
--- a/Documentation/ABI/testing/sysfs-block-zram
+++ b/Documentation/ABI/testing/sysfs-block-zram
@@ -22,41 +22,6 @@
device. The reset operation frees all the memory associated
with this device.
-What: /sys/block/zram<id>/num_reads
-Date: August 2010
-Contact: Nitin Gupta <ngupta@vflare.org>
-Description:
- The num_reads file is read-only and specifies the number of
- reads (failed or successful) done on this device.
-
-What: /sys/block/zram<id>/num_writes
-Date: August 2010
-Contact: Nitin Gupta <ngupta@vflare.org>
-Description:
- The num_writes file is read-only and specifies the number of
- writes (failed or successful) done on this device.
-
-What: /sys/block/zram<id>/invalid_io
-Date: August 2010
-Contact: Nitin Gupta <ngupta@vflare.org>
-Description:
- The invalid_io file is read-only and specifies the number of
- non-page-size-aligned I/O requests issued to this device.
-
-What: /sys/block/zram<id>/failed_reads
-Date: February 2014
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The failed_reads file is read-only and specifies the number of
- failed reads happened on this device.
-
-What: /sys/block/zram<id>/failed_writes
-Date: February 2014
-Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
-Description:
- The failed_writes file is read-only and specifies the number of
- failed writes happened on this device.
-
What: /sys/block/zram<id>/max_comp_streams
Date: February 2014
Contact: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>
@@ -73,74 +38,24 @@
available and selected compression algorithms, change
compression algorithm selection.
-What: /sys/block/zram<id>/notify_free
-Date: August 2010
-Contact: Nitin Gupta <ngupta@vflare.org>
-Description:
- The notify_free file is read-only. Depending on device usage
- scenario it may account a) the number of pages freed because
- of swap slot free notifications or b) the number of pages freed
- because of REQ_DISCARD requests sent by bio. The former ones
- are sent to a swap block device when a swap slot is freed, which
- implies that this disk is being used as a swap disk. The latter
- ones are sent by filesystem mounted with discard option,
- whenever some data blocks are getting discarded.
-
-What: /sys/block/zram<id>/zero_pages
-Date: August 2010
-Contact: Nitin Gupta <ngupta@vflare.org>
-Description:
- The zero_pages file is read-only and specifies number of zero
- filled pages written to this disk. No memory is allocated for
- such pages.
-
-What: /sys/block/zram<id>/orig_data_size
-Date: August 2010
-Contact: Nitin Gupta <ngupta@vflare.org>
-Description:
- The orig_data_size file is read-only and specifies uncompressed
- size of data stored in this disk. This excludes zero-filled
- pages (zero_pages) since no memory is allocated for them.
- Unit: bytes
-
-What: /sys/block/zram<id>/compr_data_size
-Date: August 2010
-Contact: Nitin Gupta <ngupta@vflare.org>
-Description:
- The compr_data_size file is read-only and specifies compressed
- size of data stored in this disk. So, compression ratio can be
- calculated using orig_data_size and this statistic.
- Unit: bytes
-
-What: /sys/block/zram<id>/mem_used_total
-Date: August 2010
-Contact: Nitin Gupta <ngupta@vflare.org>
-Description:
- The mem_used_total file is read-only and specifies the amount
- of memory, including allocator fragmentation and metadata
- overhead, allocated for this disk. So, allocator space
- efficiency can be calculated using compr_data_size and this
- statistic.
- Unit: bytes
-
What: /sys/block/zram<id>/mem_used_max
Date: August 2014
Contact: Minchan Kim <minchan@kernel.org>
Description:
- The mem_used_max file is read/write and specifies the amount
- of maximum memory zram have consumed to store compressed data.
- For resetting the value, you should write "0". Otherwise,
- you could see -EINVAL.
+ The mem_used_max file is write-only and is used to reset
+ the counter of maximum memory zram have consumed to store
+ compressed data. For resetting the value, you should write
+ "0". Otherwise, you could see -EINVAL.
Unit: bytes
What: /sys/block/zram<id>/mem_limit
Date: August 2014
Contact: Minchan Kim <minchan@kernel.org>
Description:
- The mem_limit file is read/write and specifies the maximum
- amount of memory ZRAM can use to store the compressed data. The
- limit could be changed in run time and "0" means disable the
- limit. No limit is the initial state. Unit: bytes
+ The mem_limit file is write-only and specifies the maximum
+ amount of memory ZRAM can use to store the compressed data.
+ The limit could be changed in run time and "0" means disable
+ the limit. No limit is the initial state. Unit: bytes
What: /sys/block/zram<id>/compact
Date: August 2015
@@ -175,3 +90,50 @@
device's debugging info useful for kernel developers. Its
format is not documented intentionally and may change
anytime without any notice.
+
+What: /sys/block/zram<id>/backing_dev
+Date: June 2017
+Contact: Minchan Kim <minchan@kernel.org>
+Description:
+ The backing_dev file is read-write and set up backing
+ device for zram to write incompressible pages.
+ For using, user should enable CONFIG_ZRAM_WRITEBACK.
+
+What: /sys/block/zram<id>/idle
+Date: November 2018
+Contact: Minchan Kim <minchan@kernel.org>
+Description:
+ idle file is write-only and mark zram slot as idle.
+ If system has mounted debugfs, user can see which slots
+ are idle via /sys/kernel/debug/zram/zram<id>/block_state
+
+What: /sys/block/zram<id>/writeback
+Date: November 2018
+Contact: Minchan Kim <minchan@kernel.org>
+Description:
+ The writeback file is write-only and trigger idle and/or
+ huge page writeback to backing device.
+
+What: /sys/block/zram<id>/bd_stat
+Date: November 2018
+Contact: Minchan Kim <minchan@kernel.org>
+Description:
+ The bd_stat file is read-only and represents backing device's
+ statistics (bd_count, bd_reads, bd_writes) in a format
+ similar to block layer statistics file format.
+
+What: /sys/block/zram<id>/writeback_limit_enable
+Date: November 2018
+Contact: Minchan Kim <minchan@kernel.org>
+Description:
+ The writeback_limit_enable file is read-write and specifies
+ eable of writeback_limit feature. "1" means enable the feature.
+ No limit "0" is the initial state.
+
+What: /sys/block/zram<id>/writeback_limit
+Date: November 2018
+Contact: Minchan Kim <minchan@kernel.org>
+Description:
+ The writeback_limit file is read-write and specifies the maximum
+ amount of writeback ZRAM can do. The limit could be changed
+ in run time.
diff --git a/Documentation/ABI/testing/sysfs-class-led-driver-cs40l20 b/Documentation/ABI/testing/sysfs-class-led-driver-cs40l20
new file mode 100644
index 0000000..9b3ebe1
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-class-led-driver-cs40l20
@@ -0,0 +1,38 @@
+What: /sys/class/leds/vibrator/device/cp_trigger_index
+Date: Jan 2018
+KernelVersion: 4.9
+Contact: David Lin <dtwlin@google.com>
+Description:
+ Sets the Control Port trigger index in the waveform library to
+ be played on the subsequent activate.
+
+What: /sys/class/leds/vibrator/device/f0_measured
+Date: Jan 2018
+KernelVersion: 4.9
+Contact: David Lin <dtwlin@google.com>
+Description:
+ Reading it to obtain the fundamental frequency measured in
+ Q10.14 format.
+
+What: /sys/class/leds/vibrator/device/f0_stored
+Date: Jan 2018
+KernelVersion: 4.9
+Contact: David Lin <dtwlin@google.com>
+Description:
+ Sets the fundamental frequency to be used either PWLE and
+ waveform playback
+
+
+What: /sys/class/leds/vibrator/device/redc_measured
+Date: Jan 2018
+KernelVersion: 4.9
+Contact: David Lin <dtwlin@google.com>
+Description:
+ Reading it to obtain the resistence DC measured in Q7.17
+ format. It must be scaled by 5.687 as shown in the following
+ example:
+
+ $ cat /sys/class/leds/vibrator/device/redc_measured
+
+ ReDC (Ohms) = redc_measured / 2^17 * 5.687
+
diff --git a/Documentation/ABI/testing/sysfs-class-typec b/Documentation/ABI/testing/sysfs-class-typec
new file mode 100644
index 0000000..5be552e
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-class-typec
@@ -0,0 +1,291 @@
+USB Type-C port devices (eg. /sys/class/typec/port0/)
+
+What: /sys/class/typec/<port>/data_role
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ The supported USB data roles. This attribute can be used for
+ requesting data role swapping on the port. Swapping is supported
+ as synchronous operation, so write(2) to the attribute will not
+ return until the operation has finished. The attribute is
+ notified about role changes so that poll(2) on the attribute
+ wakes up. Change on the role will also generate uevent
+ KOBJ_CHANGE on the port. The current role is show in brackets,
+ for example "[host] device" when DRP port is in host mode.
+
+ Valid values: host, device
+
+What: /sys/class/typec/<port>/power_role
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ The supported power roles. This attribute can be used to request
+ power role swap on the port when the port supports USB Power
+ Delivery. Swapping is supported as synchronous operation, so
+ write(2) to the attribute will not return until the operation
+ has finished. The attribute is notified about role changes so
+ that poll(2) on the attribute wakes up. Change on the role will
+ also generate uevent KOBJ_CHANGE. The current role is show in
+ brackets, for example "[source] sink" when in source mode.
+
+ Valid values: source, sink
+
+What: /sys/class/typec/<port>/port_type
+Date: May 2017
+Contact: Badhri Jagan Sridharan <Badhri@google.com>
+Description:
+ Indicates the type of the port. This attribute can be used for
+ requesting a change in the port type. Port type change is
+ supported as a synchronous operation, so write(2) to the
+ attribute will not return until the operation has finished.
+
+ Valid values:
+ - source (The port will behave as source only DFP port)
+ - sink (The port will behave as sink only UFP port)
+ - dual (The port will behave as dual-role-data and
+ dual-role-power port)
+
+What: /sys/class/typec/<port>/vconn_source
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Shows is the port VCONN Source. This attribute can be used to
+ request VCONN swap to change the VCONN Source during connection
+ when both the port and the partner support USB Power Delivery.
+ Swapping is supported as synchronous operation, so write(2) to
+ the attribute will not return until the operation has finished.
+ The attribute is notified about VCONN source changes so that
+ poll(2) on the attribute wakes up. Change on VCONN source also
+ generates uevent KOBJ_CHANGE.
+
+ Valid values:
+ - "no" when the port is not the VCONN Source
+ - "yes" when the port is the VCONN Source
+
+What: /sys/class/typec/<port>/power_operation_mode
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Shows the current power operational mode the port is in. The
+ power operation mode means current level for VBUS. In case USB
+ Power Delivery communication is used for negotiating the levels,
+ power operation mode should show "usb_power_delivery".
+
+ Valid values:
+ - default
+ - 1.5A
+ - 3.0A
+ - usb_power_delivery
+
+What: /sys/class/typec/<port>/preferred_role
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ The user space can notify the driver about the preferred role.
+ It should be handled as enabling of Try.SRC or Try.SNK, as
+ defined in USB Type-C specification, in the port drivers. By
+ default the preferred role should come from the platform.
+
+ Valid values: source, sink, none (to remove preference)
+
+What: /sys/class/typec/<port>/supported_accessory_modes
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Space separated list of accessory modes, defined in the USB
+ Type-C specification, the port supports.
+
+What: /sys/class/typec/<port>/usb_power_delivery_revision
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Revision number of the supported USB Power Delivery
+ specification, or 0 when USB Power Delivery is not supported.
+
+What: /sys/class/typec/<port>/usb_typec_revision
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Revision number of the supported USB Type-C specification.
+
+
+USB Type-C partner devices (eg. /sys/class/typec/port0-partner/)
+
+What: /sys/class/typec/<port>-partner/accessory_mode
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Shows the Accessory Mode name when the partner is an Accessory.
+ The Accessory Modes are defined in USB Type-C Specification.
+
+What: /sys/class/typec/<port>-partner/supports_usb_power_delivery
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Shows if the partner supports USB Power Delivery communication:
+ Valid values: yes, no
+
+What: /sys/class/typec/<port>-partner>/identity/
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ This directory appears only if the port device driver is capable
+ of showing the result of Discover Identity USB power delivery
+ command. That will not always be possible even when USB power
+ delivery is supported, for example when USB power delivery
+ communication for the port is mostly handled in firmware. If the
+ directory exists, it will have an attribute file for every VDO
+ in Discover Identity command result.
+
+What: /sys/class/typec/<port>-partner/identity/id_header
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ ID Header VDO part of Discover Identity command result. The
+ value will show 0 until Discover Identity command result becomes
+ available. The value can be polled.
+
+What: /sys/class/typec/<port>-partner/identity/cert_stat
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Cert Stat VDO part of Discover Identity command result. The
+ value will show 0 until Discover Identity command result becomes
+ available. The value can be polled.
+
+What: /sys/class/typec/<port>-partner/identity/product
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Product VDO part of Discover Identity command result. The value
+ will show 0 until Discover Identity command result becomes
+ available. The value can be polled.
+
+
+USB Type-C cable devices (eg. /sys/class/typec/port0-cable/)
+
+Note: Electronically Marked Cables will have a device also for one cable plug
+(eg. /sys/class/typec/port0-plug0). If the cable is active and has also SOP
+Double Prime controller (USB Power Deliver specification ch. 2.4) it will have
+second device also for the other plug. Both plugs may have alternate modes as
+described in USB Type-C and USB Power Delivery specifications.
+
+What: /sys/class/typec/<port>-cable/type
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Shows if the cable is active.
+ Valid values: active, passive
+
+What: /sys/class/typec/<port>-cable/plug_type
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Shows type of the plug on the cable:
+ - type-a - Standard A
+ - type-b - Standard B
+ - type-c
+ - captive
+
+What: /sys/class/typec/<port>-cable/identity/
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ This directory appears only if the port device driver is capable
+ of showing the result of Discover Identity USB power delivery
+ command. That will not always be possible even when USB power
+ delivery is supported. If the directory exists, it will have an
+ attribute for every VDO returned by Discover Identity command.
+
+What: /sys/class/typec/<port>-cable/identity/id_header
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ ID Header VDO part of Discover Identity command result. The
+ value will show 0 until Discover Identity command result becomes
+ available. The value can be polled.
+
+What: /sys/class/typec/<port>-cable/identity/cert_stat
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Cert Stat VDO part of Discover Identity command result. The
+ value will show 0 until Discover Identity command result becomes
+ available. The value can be polled.
+
+What: /sys/class/typec/<port>-cable/identity/product
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Product VDO part of Discover Identity command result. The value
+ will show 0 until Discover Identity command result becomes
+ available. The value can be polled.
+
+
+Alternate Mode devices.
+
+The alternate modes will have Standard or Vendor ID (SVID) assigned by USB-IF.
+The ports, partners and cable plugs can have alternate modes. A supported SVID
+will consist of a set of modes. Every SVID a port/partner/plug supports will
+have a device created for it, and every supported mode for a supported SVID will
+have its own directory under that device. Below <dev> refers to the device for
+the alternate mode.
+
+What: /sys/class/typec/<port|partner|cable>/<dev>/svid
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ The SVID (Standard or Vendor ID) assigned by USB-IF for this
+ alternate mode.
+
+What: /sys/class/typec/<port|partner|cable>/<dev>/mode<index>/
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Every supported mode will have its own directory. The name of
+ a mode will be "mode<index>" (for example mode1), where <index>
+ is the actual index to the mode VDO returned by Discover Modes
+ USB power delivery command.
+
+What: /sys/class/typec/<port|partner|cable>/<dev>/mode<index>/description
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Shows description of the mode. The description is optional for
+ the drivers, just like with the Billboard Devices.
+
+What: /sys/class/typec/<port|partner|cable>/<dev>/mode<index>/vdo
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Shows the VDO in hexadecimal returned by Discover Modes command
+ for this mode.
+
+What: /sys/class/typec/<port|partner|cable>/<dev>/mode<index>/active
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Shows if the mode is active or not. The attribute can be used
+ for entering/exiting the mode with partners and cable plugs, and
+ with the port alternate modes it can be used for disabling
+ support for specific alternate modes. Entering/exiting modes is
+ supported as synchronous operation so write(2) to the attribute
+ does not return until the enter/exit mode operation has
+ finished. The attribute is notified when the mode is
+ entered/exited so poll(2) on the attribute wakes up.
+ Entering/exiting a mode will also generate uevent KOBJ_CHANGE.
+
+ Valid values: yes, no
+
+What: /sys/class/typec/<port>/<dev>/mode<index>/supported_roles
+Date: April 2017
+Contact: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+Description:
+ Space separated list of the supported roles.
+
+ This attribute is available for the devices describing the
+ alternate modes a port supports, and it will not be exposed with
+ the devices presenting the alternate modes the partners or cable
+ plugs support.
+
+ Valid values: source, sink
diff --git a/Documentation/ABI/testing/sysfs-devices-platform-vd6281 b/Documentation/ABI/testing/sysfs-devices-platform-vd6281
new file mode 100644
index 0000000..e08b26b
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-devices-platform-vd6281
@@ -0,0 +1,23 @@
+What: /sys/devices/platform/<I2C controller address>/rainbow_enable
+Date: October 2017
+Contact: Xu Han <xuhanyz@google.com>
+Description:
+ Write to this file to enable/disable rainbow sensor vd6281
+ 0 - Enable
+ 1 - Disable
+
+What: /sys/devices/platform/<I2C controller address>/rainbow_write_byte
+Date: October 2017
+Contact: Xu Han <xuhanyz@google.com>
+Description:
+ Write to this file to write a byte to rainbow sensor vd6281. The
+ 1st byte is the address and 2nd byte is the data. A negative error
+ is returned in case of failure.
+
+What: /sys/devices/platform/<I2C controller address>/rainbow_read_byte
+Date: October 2017
+Contact: Xu Han <xuhanyz@google.com>
+Description:
+ Write to this file to read a byte to rainbow sensor vd6281. The
+ 1st byte is the address to read. Read data is returned in case of
+ success. A negative error is returned in case of failure.
diff --git a/Documentation/ABI/testing/sysfs-devices-ramoops b/Documentation/ABI/testing/sysfs-devices-ramoops
new file mode 100644
index 0000000..c20ce8b
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-devices-ramoops
@@ -0,0 +1,35 @@
+What: /sys/devices/virtual/ramoops/pstore/aes_key
+Date: July 24, 2017
+KernelVersion: 4.4
+Contact: Patrick Tjin <pattjin@google.com>
+Description: This file is used to communicate the AES key used to decrypt the
+ alternate buffer.
+
+What: /sys/devices/virtual/ramoops/pstore/aes_key_iv
+Date: July 24, 2017
+KernelVersion: 4.4
+Contact: Patrick Tjin <pattjin@google.com>
+Description: This file is used to communicate the AES IV used to decrypt the
+ alternate buffer.
+
+What: /sys/devices/virtual/ramoops/pstore/aes_key_tag
+Date: July 24, 2017
+KernelVersion: 4.4
+Contact: Patrick Tjin <pattjin@google.com>
+Description: This file is used to communicate the AES tag used to decrypt the
+ alternate buffer.
+
+What: /sys/devices/virtual/ramoops/pstore/use_alt
+Date: July 24, 2017
+KernelVersion: 4.4
+Contact: Patrick Tjin <pattjin@google.com>
+Description: Writing a 1 to this file will cause the ramoops to pivot to
+ using the alternate buffer. (This includes decryption, the
+ AES keys should have been given to the driver prior to writing
+ this node.)
+
+What: /sys/devices/virtual/ramoops/pstore/decrypt_state
+Date: July 9, 2018
+KernelVersion: 4.4
+Contact: jonechou <jonechou@google.com>
+Description: This file is used to query the decrypt state of console-ramoops-0
diff --git a/Documentation/ABI/testing/sysfs-devices-soc-ufs b/Documentation/ABI/testing/sysfs-devices-soc-ufs
new file mode 100644
index 0000000..a7e0034
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-devices-soc-ufs
@@ -0,0 +1,35 @@
+What: /sys/devices/soc/X.ufshc/erasesize
+Date: April 2018
+contact: Jaegeuk Kim <jaegeuk@google.com>
+Description:
+ This file shows the size of erase operation reported by UFS.
+
+What: /sys/devices/soc/X.ufshc/health
+Date: September 2017
+contact: Jaegeuk Kim <jaegeuk@google.com>
+Description:
+ This directory contains health information reported by UFS.
+ - length must be 25h
+ - type must be 09h
+ - eol represent
+ 00h: Not defined
+ 01h: Normal
+ 02h: Warning
+ 03h: Critical
+ - lifetimeA/B
+ 00h: Not defined
+ 01h: 0% ~ 10% device life time used
+ 02h: 10% ~ 20% device life time used
+ 03h: 20% ~ 30% device life time used
+ 04h: 30% ~ 40% device life time used
+ 05h: 40% ~ 50% device life time used
+ 06h: 50% ~ 60% device life time used
+ 07h: 60% ~ 70% device life time used
+ 08h: 70% ~ 80% device life time used
+ 09h: 80% ~ 90% device life time used
+ 0Ah: 90% ~ 100% device life time used
+ 0Bh: Exceeded its maximum estimated device life time
+ - erasecount
+ average erase count
+ - totalwrite
+ total host data writes in 100MB
diff --git a/Documentation/ABI/testing/sysfs-devices-system-cpu b/Documentation/ABI/testing/sysfs-devices-system-cpu
index dfd56ec..069e8d5 100644
--- a/Documentation/ABI/testing/sysfs-devices-system-cpu
+++ b/Documentation/ABI/testing/sysfs-devices-system-cpu
@@ -355,6 +355,8 @@
/sys/devices/system/cpu/vulnerabilities/meltdown
/sys/devices/system/cpu/vulnerabilities/spectre_v1
/sys/devices/system/cpu/vulnerabilities/spectre_v2
+ /sys/devices/system/cpu/vulnerabilities/spec_store_bypass
+ /sys/devices/system/cpu/vulnerabilities/l1tf
Date: January 2018
Contact: Linux kernel mailing list <linux-kernel@vger.kernel.org>
Description: Information about CPU vulnerabilities
@@ -366,3 +368,26 @@
"Not affected" CPU is not affected by the vulnerability
"Vulnerable" CPU is affected and no mitigation in effect
"Mitigation: $M" CPU is affected and mitigation $M is in effect
+
+ Details about the l1tf file can be found in
+ Documentation/admin-guide/l1tf.rst
+
+What: /sys/devices/system/cpu/smt
+ /sys/devices/system/cpu/smt/active
+ /sys/devices/system/cpu/smt/control
+Date: June 2018
+Contact: Linux kernel mailing list <linux-kernel@vger.kernel.org>
+Description: Control Symetric Multi Threading (SMT)
+
+ active: Tells whether SMT is active (enabled and siblings online)
+
+ control: Read/write interface to control SMT. Possible
+ values:
+
+ "on" SMT is enabled
+ "off" SMT is disabled
+ "forceoff" SMT is force disabled. Cannot be changed.
+ "notsupported" SMT is not supported by the CPU
+
+ If control status is "forceoff" or "notsupported" writes
+ are rejected.
diff --git a/Documentation/ABI/testing/sysfs-devices-ufs b/Documentation/ABI/testing/sysfs-devices-ufs
new file mode 100644
index 0000000..a9f02c7
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-devices-ufs
@@ -0,0 +1,282 @@
+What: /sys/devices/platform/soc/<....>ufshc/manual_gc
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Control UFS to give a time for internal Garbage Collections,
+ given manual_gc_hold seconds.
+
+What: /sys/devices/platform/soc/<....>ufshc/manual_gc_hold
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Give the seconds to UFS for internal GCs.
+
+What: /sys/devices/platform/soc/<....>ufshc/slowio_read_us
+Date: May, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Contains watermark value in micro seconds unit for slow
+ UFS read I/O logging and counting. Can be updated by writing
+ value to.
+
+What: /sys/devices/platform/soc/<....>ufshc/slowio_read_cnt
+Date: May, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Contains the number of UFS read I/O requests which took more
+ or equal to the defined slow I/O water mark time.
+ Can be reset by writing any value.
+
+What: /sys/devices/platform/soc/<....>ufshc/slowio_write_us
+Date: May, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Contains watermark value in micro seconds unit for slow
+ UFS write I/O logging and counting. Can be updated by writing
+ value to.
+
+What: /sys/devices/platform/soc/<....>ufshc/slowio_write_cnt
+Date: May, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Contains the number of UFS write I/O requests which took more
+ or equal to the defined slow I/O water mark time.
+ Can be reset by writing any value.
+
+What: /sys/devices/platform/soc/<....>ufshc/slowio_unmap_us
+Date: May, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Contains watermark value in micro seconds unit for slow
+ UFS unmap I/O logging and counting. Can be updated by writing
+ value to.
+
+What: /sys/devices/platform/soc/<....>ufshc/slowio_unmap_cnt
+Date: May, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Contains the number of UFS unmap I/O requests which took more
+ or equal to the defined slow I/O water mark time.
+ Can be reset by writing any value.
+
+What: /sys/devices/platform/soc/<....>ufshc/slowio_sync_us
+Date: May, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Contains watermark value in micro seconds unit for slow
+ UFS sync I/O logging and counting. Can be updated by writing
+ value to.
+
+What: /sys/devices/platform/soc/<....>ufshc/slowio_sync_cnt
+Date: May, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Contains the number of UFS sync I/O requests which took more
+ or equal to the defined slow I/O water mark time.
+ Can be reset by writing any value.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/enabled
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Enable or disable impaired storage emulation.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/read_model
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Degraded UFS performance model is applied for read requests.
+ It could be set with 'ftlfrag' or 'none'; the former enables
+ model based delay emulation and the later disables the feature.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/read_delay_percent
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ This knob can be used to control target delay for UFS read requests.
+ It receives a precentage value, and the default value is 100, which means
+ no change. When 200 is written, the target latency with delay will be
+ twice of an input latency, which is either the actual latency or a latency
+ chosen by the degraded performance model when fragmented ftl emulation
+ feature is enabled.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/read_delay_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ This knob is used to control target delay for UFS read requsts.
+ For each read request, this value will be added to the target delay.
+ For instance, when 200 is set, all read requests will be delayed
+ by 200 microseconds. The default value is 0, which means no delay.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/write_model
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Degraded UFS performance model is applied for write requests.
+ It could be set with 'ftlfrag' or 'none'; the former enables
+ model based delay emulation and the later disables the feature.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/write_delay_percent
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ This knob can be used to control target delay for UFS write requests.
+ It receives a precentage value, and the default value is 100, which means
+ no change. When 200 is written, the target latency with delay will be
+ twice of an input latency, which is either the actual latency or a latency
+ chosen by the degraded performance model when fragmented ftl emulation
+ feature is enabled.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/write_delay_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ This knob is used to control target delay for UFS write requsts.
+ For each write request, this value will be added to the target delay.
+ For instance, when 200 is set, all write requests will be delayed
+ by 200 microseconds. The default value is 0, which means no delay.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/advanced_control/read_max_delayed_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Maximum read request latency can be set by this entry. For instance,
+ when 1000 is set to this entry, the emulated read latency will not
+ be bigger than 1ms.
+ Default value is 0, which means no limitation.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/advanced_control/read_skip_delay_cnt
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ This knob can be used to control delay emulation frequency.
+ When the default 0 is set, all read requests are delayed.
+ When 1 is written, every other read request is delayed.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/advanced_control/write_max_delayed_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Maximum write request latency can be set by this entry. For instance,
+ when 1000 is set to this entry, the emulated write latency will not
+ be bigger than 1ms.
+ Default value is 0, which means no limitation.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/advanced_control/write_skip_delay_cnt
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ This knob can be used to control delay emulation frequency.
+ When the default 0 is set, all write requests are delayed.
+ When 1 is written, every other write request is delayed.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/stat/total_read_cnt
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Total number of read requests can be checked by reading this entry.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/stat/total_read_original_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Aggregated original read latencies without added delay can be checked
+ by reading this entry.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/stat/total_read_delayed_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Aggregated read latencies with delay emulation can be checked
+ by reading this entry.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/stat/total_write_cnt
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Total number of write requests can be checked by reading this entry.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/stat/total_write_original_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Aggregated original write latencies without added delay can be checked
+ by reading this entry.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/stat/total_write_delayed_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Aggregated write latencies with delay emulation can be checked
+ by reading this entry.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/delay_stat/max_read_delay_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Maximum read delay in microseconds, targeted by delay emulation,
+ can be checked by reading this entry.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/delay_stat/max_read_delay_error_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Added delay might be different from the intended delay value.
+ By reading this entry, observed maximum read delay error value can be
+ checked in microsecond unit.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/delay_stat/total_read_delay_error_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Added delay might be different from the intended delay value.
+ For all delayed read requests, the additional delay value than intended delay
+ is aggrated to this entry.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/delay_stat/max_write_delay_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Maximum write delay in microseconds, targeted by delay emulation,
+ can be checked by reading this entry.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/delay_stat/max_write_delay_error_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Added delay might be different from the intended delay value.
+ By reading this entry, observed maximum write delay error value can be
+ checked in microsecond unit.
+ Can be reset together with all other statistics entries when
+ any value is written to.
+
+What: /sys/devices/platform/soc/<....>ufshc/impaired/delay_stat/total_write_delay_error_us
+Date: December, 2018
+Contact: "Hyojun Kim" <hyojun@google.com>
+Description:
+ Added delay might be different from the intended delay value.
+ For all delayed write requests, the additional delay value than intended delay
+ is aggrated to this entry.
+ Can be reset together with all other statistics entries when
+ any value is written to.
diff --git a/Documentation/ABI/testing/sysfs-devices-ufshpb b/Documentation/ABI/testing/sysfs-devices-ufshpb
new file mode 100644
index 0000000..0e76cee
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-devices-ufshpb
@@ -0,0 +1,77 @@
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/HPBVersion
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Show HPB version info.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/add_evict_count
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Show the count of HPB entry evictions.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/hit_count
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Show the count of HPB entry hits.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/map_loading
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Request HPB map loading.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/read_16_disable
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Disable READ16 for HPB communication.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/active_block_status
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Show all the active block.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/count_reset
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Reset all the tracking counts.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/is_active_group
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Show active group info.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/map_req_count
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Show the count to request HPB map entry.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/active_list
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Show active block list.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/get_info_from_lba
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Show HPB entry info given LBA.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/map_cmd_disable
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Disable HPB map command.
+
+What: /sys/devices/platform/soc/<....>ufshc/ufshpb_lu#num/miss_count
+Date: April, 2018
+Contact: "Jaegeuk Kim" <jaegeuk@google.com>
+Description:
+ Show the count of HPB entry misses.
diff --git a/Documentation/ABI/testing/sysfs-driver-mnh b/Documentation/ABI/testing/sysfs-driver-mnh
new file mode 100644
index 0000000..f839a94
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-driver-mnh
@@ -0,0 +1,62 @@
+What: /sys/devices/virtual/misc/mnh_sm/mnh_clk/cpu_freq
+Date: November 2017
+Contact: Trevor Bunker <trevorbunker@google.com>
+Description:
+ Reads to this file return the current MNH CPU frequency in MHz.
+ Writes to this file will change the MNH CPU frequency. The only
+ accepted values for inputs are "200", "400", "600", "800", and
+ "950".
+
+What: /sys/devices/virtual/misc/mnh_sm/mnh_clk/ipu_freq
+Date: November 2017
+Contact: Trevor Bunker <trevorbunker@google.com>
+Description:
+ Reads to this file return the current MNH IPU frequency in MHz.
+ Writes to this file will change the MNH IPU frequency. The only
+ accepted values for inputs are "100", "200", "300", "400", and
+ "425".
+
+What: /sys/devices/virtual/misc/mnh_sm/mnh_clk/lpddr_freq
+Date: November 2017
+Contact: Trevor Bunker <trevorbunker@google.com>
+Description:
+ Reads to this file return the current MNH LPDDR Frequency Set
+ Point (FSP). Writes to this file will change the MNH LPDDR
+ FSP. The only accepted values for inputs are "0" "1", "2", and
+ "3".
+
+What: /sys/devices/virtual/misc/mnh_sm/mnh_clk/ipu_clk_src
+Date: November 2017
+Contact: Trevor Bunker <trevorbunker@google.com>
+Description:
+ Reads to this file return the current IPU clock source setting.
+ The valid values are: "0", for CPU_IPU PLL; and "1" for IPU
+ PLL.
+
+What: /sys/devices/virtual/misc/mnh_sm/mnh_clk/sys200_mode
+Date: November 2017
+Contact: Trevor Bunker <trevorbunker@google.com>
+Description:
+ Reads to this file return a boolean indicating if the MNH
+ device is operating in sys200 mode. By writing a "1" to this
+ file, you can force the MNH device to enter sys200 mode.
+ However, the only way to leave sys200 mode is to set the CPU,
+ IPU, or LPDDR frequencies with the sysfs files listed above.
+
+What: /sys/devices/virtual/misc/mnh_sm/mnh_clk/ipu_clock_gating
+Date: November 2017
+Contact: Trevor Bunker <trevorbunker@google.com>
+Description:
+ Reads to this file return a boolean indicating if the IPU is
+ clock gated. By writing a "0" or "1" to this file, you can
+ either disable or enable, respectively, IPU clock gating.
+
+What: /sys/devices/virtual/misc/mnh_sm/mnh_clk/bypass_clock_gating
+Date: November 2017
+Contact: Trevor Bunker <trevorbunker@google.com>
+Description:
+ A small subset of peripherals can be clock gated during MNH
+ bypass mode for power savings. Reads to this file return a
+ boolean indicating if this subset is clock gated. By writing a
+ "0" or "1" to this file, you can either disable or enable,
+ respectively, clock gating to this subset.
diff --git a/Documentation/ABI/testing/sysfs-fs-f2fs b/Documentation/ABI/testing/sysfs-fs-f2fs
index 3bbb9fe..e09db01 100644
--- a/Documentation/ABI/testing/sysfs-fs-f2fs
+++ b/Documentation/ABI/testing/sysfs-fs-f2fs
@@ -92,6 +92,15 @@
Description:
Controls the number of trials to find a victim segment.
+What: /sys/fs/f2fs/<disk>/migration_granularity
+Date: October 2018
+Contact: "Chao Yu" <yuchao0@huawei.com>
+Description:
+ Controls migration granularity of garbage collection on large
+ section, it can let GC move partial segment{s} of one section
+ in one GC cycle, so that dispersing heavy overhead GC to
+ multiple lightweight one.
+
What: /sys/fs/f2fs/<disk>/dir_level
Date: March 2014
Contact: "Jaegeuk Kim" <jaegeuk.kim@samsung.com>
@@ -121,7 +130,22 @@
Date: January 2016
Contact: "Jaegeuk Kim" <jaegeuk@kernel.org>
Description:
- Controls the idle timing.
+ Controls the idle timing for all paths other than
+ discard and gc path.
+
+What: /sys/fs/f2fs/<disk>/discard_idle_interval
+Date: September 2018
+Contact: "Chao Yu" <yuchao0@huawei.com>
+Contact: "Sahitya Tummala" <stummala@codeaurora.org>
+Description:
+ Controls the idle timing for discard path.
+
+What: /sys/fs/f2fs/<disk>/gc_idle_interval
+Date: September 2018
+Contact: "Chao Yu" <yuchao0@huawei.com>
+Contact: "Sahitya Tummala" <stummala@codeaurora.org>
+Description:
+ Controls the idle timing for gc path.
What: /sys/fs/f2fs/<disk>/iostat_enable
Date: August 2017
diff --git a/Documentation/ABI/testing/sysfs-misc-access b/Documentation/ABI/testing/sysfs-misc-access
new file mode 100644
index 0000000..25ca44d
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-misc-access
@@ -0,0 +1,13 @@
+What: /sys/class/misc/access-<region>/label
+Date: May 16, 2017
+KernelVersion: 3.18
+Contact: Patrick Tjin <pattjin@google.com>
+Description: This file exposes the region which is controlled by the driver.
+
+What: /sys/class/misc/access-<region>/size
+Date: June 6, 2017
+KernelVersion: 3.18
+Contact: Patrick Tjin <pattjin@google.com>
+Description: This file exposes the size of the region which is controlled
+ by the driver.
+
diff --git a/Documentation/ABI/testing/sysfs-power b/Documentation/ABI/testing/sysfs-power
index 50b368d..f523e5a 100644
--- a/Documentation/ABI/testing/sysfs-power
+++ b/Documentation/ABI/testing/sysfs-power
@@ -7,30 +7,35 @@
subsystem.
What: /sys/power/state
-Date: May 2014
+Date: November 2016
Contact: Rafael J. Wysocki <rjw@rjwysocki.net>
Description:
The /sys/power/state file controls system sleep states.
Reading from this file returns the available sleep state
- labels, which may be "mem", "standby", "freeze" and "disk"
- (hibernation). The meanings of the first three labels depend on
- the relative_sleep_states command line argument as follows:
- 1) relative_sleep_states = 1
- "mem", "standby", "freeze" represent non-hibernation sleep
- states from the deepest ("mem", always present) to the
- shallowest ("freeze"). "standby" and "freeze" may or may
- not be present depending on the capabilities of the
- platform. "freeze" can only be present if "standby" is
- present.
- 2) relative_sleep_states = 0 (default)
- "mem" - "suspend-to-RAM", present if supported.
- "standby" - "power-on suspend", present if supported.
- "freeze" - "suspend-to-idle", always present.
+ labels, which may be "mem" (suspend), "standby" (power-on
+ suspend), "freeze" (suspend-to-idle) and "disk" (hibernation).
- Writing to this file one of these strings causes the system to
- transition into the corresponding state, if available. See
- Documentation/power/states.txt for a description of what
- "suspend-to-RAM", "power-on suspend" and "suspend-to-idle" mean.
+ Writing one of the above strings to this file causes the system
+ to transition into the corresponding state, if available.
+
+ See Documentation/power/states.txt for more information.
+
+What: /sys/power/mem_sleep
+Date: November 2016
+Contact: Rafael J. Wysocki <rjw@rjwysocki.net>
+Description:
+ The /sys/power/mem_sleep file controls the operating mode of
+ system suspend. Reading from it returns the available modes
+ as "s2idle" (always present), "shallow" and "deep" (present if
+ supported). The mode that will be used on subsequent attempts
+ to suspend the system (by writing "mem" to the /sys/power/state
+ file described above) is enclosed in square brackets.
+
+ Writing one of the above strings to this file causes the mode
+ represented by it to be used on subsequent attempts to suspend
+ the system.
+
+ See Documentation/power/states.txt for more information.
What: /sys/power/disk
Date: September 2006
diff --git a/Documentation/Changes b/Documentation/Changes
index 22797a1..76d6dc0 100644
--- a/Documentation/Changes
+++ b/Documentation/Changes
@@ -33,7 +33,7 @@
GNU make 3.80 make --version
binutils 2.12 ld -v
util-linux 2.10o fdformat --version
-module-init-tools 0.9.10 depmod -V
+kmod 13 depmod -V
e2fsprogs 1.41.4 e2fsck -V
jfsutils 1.1.3 fsck.jfs -V
reiserfsprogs 3.6.3 reiserfsck -V
@@ -143,12 +143,6 @@
reproduce the Oops with that option, then you can still decode that Oops
with ksymoops.
-Module-Init-Tools
------------------
-
-A new module loader is now in the kernel that requires ``module-init-tools``
-to use. It is backward compatible with the 2.4.x series kernels.
-
Mkinitrd
--------
@@ -363,16 +357,17 @@
- <ftp://ftp.kernel.org/pub/linux/utils/util-linux/>
+Kmod
+----
+
+- <https://www.kernel.org/pub/linux/utils/kernel/kmod/>
+- <https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git>
+
Ksymoops
--------
- <ftp://ftp.kernel.org/pub/linux/utils/kernel/ksymoops/v2.4/>
-Module-Init-Tools
------------------
-
-- <ftp://ftp.kernel.org/pub/linux/kernel/people/rusty/modules/>
-
Mkinitrd
--------
diff --git a/Documentation/accounting/psi.txt b/Documentation/accounting/psi.txt
new file mode 100644
index 0000000..b8ca28b
--- /dev/null
+++ b/Documentation/accounting/psi.txt
@@ -0,0 +1,73 @@
+================================
+PSI - Pressure Stall Information
+================================
+
+:Date: April, 2018
+:Author: Johannes Weiner <hannes@cmpxchg.org>
+
+When CPU, memory or IO devices are contended, workloads experience
+latency spikes, throughput losses, and run the risk of OOM kills.
+
+Without an accurate measure of such contention, users are forced to
+either play it safe and under-utilize their hardware resources, or
+roll the dice and frequently suffer the disruptions resulting from
+excessive overcommit.
+
+The psi feature identifies and quantifies the disruptions caused by
+such resource crunches and the time impact it has on complex workloads
+or even entire systems.
+
+Having an accurate measure of productivity losses caused by resource
+scarcity aids users in sizing workloads to hardware--or provisioning
+hardware according to workload demand.
+
+As psi aggregates this information in realtime, systems can be managed
+dynamically using techniques such as load shedding, migrating jobs to
+other systems or data centers, or strategically pausing or killing low
+priority or restartable batch jobs.
+
+This allows maximizing hardware utilization without sacrificing
+workload health or risking major disruptions such as OOM kills.
+
+Pressure interface
+==================
+
+Pressure information for each resource is exported through the
+respective file in /proc/pressure/ -- cpu, memory, and io.
+
+The format for CPU is as such:
+
+some avg10=0.00 avg60=0.00 avg300=0.00 total=0
+
+and for memory and IO:
+
+some avg10=0.00 avg60=0.00 avg300=0.00 total=0
+full avg10=0.00 avg60=0.00 avg300=0.00 total=0
+
+The "some" line indicates the share of time in which at least some
+tasks are stalled on a given resource.
+
+The "full" line indicates the share of time in which all non-idle
+tasks are stalled on a given resource simultaneously. In this state
+actual CPU cycles are going to waste, and a workload that spends
+extended time in this state is considered to be thrashing. This has
+severe impact on performance, and it's useful to distinguish this
+situation from a state where some tasks are stalled but the CPU is
+still doing productive work. As such, time spent in this subset of the
+stall state is tracked separately and exported in the "full" averages.
+
+The ratios are tracked as recent trends over ten, sixty, and three
+hundred second windows, which gives insight into short term events as
+well as medium and long term trends. The total absolute stall time is
+tracked and exported as well, to allow detection of latency spikes
+which wouldn't necessarily make a dent in the time averages, or to
+average trends over custom time frames.
+
+Cgroup2 interface
+=================
+
+In a system with a CONFIG_CGROUP=y kernel and the cgroup2 filesystem
+mounted, pressure stall information is also tracked for tasks grouped
+into cgroups. Each subdirectory in the cgroupfs mountpoint contains
+cpu.pressure, memory.pressure, and io.pressure files; the format is
+the same as the /proc/pressure/ files.
diff --git a/Documentation/blockdev/zram.txt b/Documentation/blockdev/zram.txt
index 0535ae1..cb230a7 100644
--- a/Documentation/blockdev/zram.txt
+++ b/Documentation/blockdev/zram.txt
@@ -156,47 +156,24 @@
A brief description of exported device attributes. For more details please
read Documentation/ABI/testing/sysfs-block-zram.
-Name access description
----- ------ -----------
-disksize RW show and set the device's disk size
-initstate RO shows the initialization state of the device
-reset WO trigger device reset
-num_reads RO the number of reads
-failed_reads RO the number of failed reads
-num_write RO the number of writes
-failed_writes RO the number of failed writes
-invalid_io RO the number of non-page-size-aligned I/O requests
-max_comp_streams RW the number of possible concurrent compress operations
-comp_algorithm RW show and change the compression algorithm
-notify_free RO the number of notifications to free pages (either
- slot free notifications or REQ_DISCARD requests)
-zero_pages RO the number of zero filled pages written to this disk
-orig_data_size RO uncompressed size of data stored in this disk
-compr_data_size RO compressed size of data stored in this disk
-mem_used_total RO the amount of memory allocated for this disk
-mem_used_max RW the maximum amount of memory zram have consumed to
- store the data (to reset this counter to the actual
- current value, write 1 to this attribute)
-mem_limit RW the maximum amount of memory ZRAM can use to store
- the compressed data
-pages_compacted RO the number of pages freed during compaction
- (available only via zram<id>/mm_stat node)
-compact WO trigger memory compaction
-debug_stat RO this file is used for zram debugging purposes
+Name access description
+---- ------ -----------
+disksize RW show and set the device's disk size
+initstate RO shows the initialization state of the device
+reset WO trigger device reset
+mem_used_max WO reset the `mem_used_max' counter (see later)
+mem_limit WO specifies the maximum amount of memory ZRAM can use
+ to store the compressed data
+writeback_limit WO specifies the maximum amount of write IO zram can
+ write out to backing device as 4KB unit
+writeback_limit_enable RW show and set writeback_limit feature
+max_comp_streams RW the number of possible concurrent compress operations
+comp_algorithm RW show and change the compression algorithm
+compact WO trigger memory compaction
+debug_stat RO this file is used for zram debugging purposes
+backing_dev RW set up backend storage for zram to write out
+idle WO mark allocated slot as idle
-WARNING
-=======
-per-stat sysfs attributes are considered to be deprecated.
-The basic strategy is:
--- the existing RW nodes will be downgraded to WO nodes (in linux 4.11)
--- deprecated RO sysfs nodes will eventually be removed (in linux 4.11)
-
-The list of deprecated attributes can be found here:
-Documentation/ABI/obsolete/sysfs-block-zram
-
-Basically, every attribute that has its own read accessible sysfs node
-(e.g. num_reads) *AND* is accessible via one of the stat files (zram<id>/stat
-or zram<id>/io_stat or zram<id>/mm_stat) is considered to be deprecated.
User space is advised to use the following files to read the device statistics.
@@ -211,22 +188,52 @@
layer and, thus, not available in zram<id>/stat file. It consists of a
single line of text and contains the following stats separated by
whitespace:
- failed_reads
- failed_writes
- invalid_io
- notify_free
+ failed_reads the number of failed reads
+ failed_writes the number of failed writes
+ invalid_io the number of non-page-size-aligned I/O requests
+ notify_free Depending on device usage scenario it may account
+ a) the number of pages freed because of swap slot free
+ notifications or b) the number of pages freed because of
+ REQ_DISCARD requests sent by bio. The former ones are
+ sent to a swap block device when a swap slot is freed,
+ which implies that this disk is being used as a swap disk.
+ The latter ones are sent by filesystem mounted with
+ discard option, whenever some data blocks are getting
+ discarded.
File /sys/block/zram<id>/mm_stat
The stat file represents device's mm statistics. It consists of a single
line of text and contains the following stats separated by whitespace:
- orig_data_size
- compr_data_size
- mem_used_total
- mem_limit
- mem_used_max
- zero_pages
- num_migrated
+ orig_data_size uncompressed size of data stored in this disk.
+ This excludes same-element-filled pages (same_pages) since
+ no memory is allocated for them.
+ Unit: bytes
+ compr_data_size compressed size of data stored in this disk
+ mem_used_total the amount of memory allocated for this disk. This
+ includes allocator fragmentation and metadata overhead,
+ allocated for this disk. So, allocator space efficiency
+ can be calculated using compr_data_size and this statistic.
+ Unit: bytes
+ mem_limit the maximum amount of memory ZRAM can use to store
+ the compressed data
+ mem_used_max the maximum amount of memory zram have consumed to
+ store the data
+ same_pages the number of same element filled pages written to this disk.
+ No memory is allocated for such pages.
+ pages_compacted the number of pages freed during compaction
+ huge_pages the number of incompressible pages
+
+File /sys/block/zram<id>/bd_stat
+
+The stat file represents device's backing device statistics. It consists of
+a single line of text and contains the following stats separated by whitespace:
+ bd_count size of data written in backing device.
+ Unit: 4K bytes
+ bd_reads the number of reads from backing device
+ Unit: 4K bytes
+ bd_writes the number of writes to backing device
+ Unit: 4K bytes
9) Deactivate:
swapoff /dev/zram0
@@ -241,5 +248,108 @@
resets the disksize to zero. You must set the disksize again
before reusing the device.
+* Optional Feature
+
+= writeback
+
+With CONFIG_ZRAM_WRITEBACK, zram can write idle/incompressible page
+to backing storage rather than keeping it in memory.
+To use the feature, admin should set up backing device via
+
+ "echo /dev/sda5 > /sys/block/zramX/backing_dev"
+
+before disksize setting. It supports only partition at this moment.
+If admin want to use incompressible page writeback, they could do via
+
+ "echo huge > /sys/block/zramX/write"
+
+To use idle page writeback, first, user need to declare zram pages
+as idle.
+
+ "echo all > /sys/block/zramX/idle"
+
+From now on, any pages on zram are idle pages. The idle mark
+will be removed until someone request access of the block.
+IOW, unless there is access request, those pages are still idle pages.
+
+Admin can request writeback of those idle pages at right timing via
+
+ "echo idle > /sys/block/zramX/writeback"
+
+With the command, zram writeback idle pages from memory to the storage.
+
+If there are lots of write IO with flash device, potentially, it has
+flash wearout problem so that admin needs to design write limitation
+to guarantee storage health for entire product life.
+
+To overcome the concern, zram supports "writeback_limit" feature.
+The "writeback_limit_enable"'s default value is 0 so that it doesn't limit
+any writeback. IOW, if admin want to apply writeback budget, he should
+enable writeback_limit_enable via
+
+ $ echo 1 > /sys/block/zramX/writeback_limit_enable
+
+Once writeback_limit_enable is set, zram doesn't allow any writeback
+until admin set the budget via /sys/block/zramX/writeback_limit.
+
+(If admin doesn't enable writeback_limit_enable, writeback_limit's value
+assigned via /sys/block/zramX/writeback_limit is meaningless.)
+
+If admin want to limit writeback as per-day 400M, he could do it
+like below.
+
+ $ MB_SHIFT=20
+ $ 4K_SHIFT=12
+ $ echo $((400<<MB_SHIFT>>4K_SHIFT)) > \
+ /sys/block/zram0/writeback_limit.
+ $ echo 1 > /sys/block/zram0/writeback_limit_enable
+
+If admin want to allow further write again once the bugdet is exausted,
+he could do it like below
+
+ $ echo $((400<<MB_SHIFT>>4K_SHIFT)) > \
+ /sys/block/zram0/writeback_limit
+
+If admin want to see remaining writeback budget since he set,
+
+ $ cat /sys/block/zramX/writeback_limit
+
+If admin want to disable writeback limit, he could do
+
+ $ echo 0 > /sys/block/zramX/writeback_limit_enable
+
+The writeback_limit count will reset whenever you reset zram(e.g.,
+system reboot, echo 1 > /sys/block/zramX/reset) so keeping how many of
+writeback happened until you reset the zram to allocate extra writeback
+budget in next setting is user's job.
+
+If admin want to measure writeback count in a certain period, he could
+know it via /sys/block/zram0/bd_stat's 3rd column.
+
+= memory tracking
+
+With CONFIG_ZRAM_MEMORY_TRACKING, user can know information of the
+zram block. It could be useful to catch cold or incompressible
+pages of the process with*pagemap.
+If you enable the feature, you could see block state via
+/sys/kernel/debug/zram/zram0/block_state". The output is as follows,
+
+ 300 75.033841 .wh.
+ 301 63.806904 s...
+ 302 63.806919 ..hi
+
+First column is zram's block index.
+Second column is access time since the system was booted
+Third column is state of the block.
+(s: same page
+w: written page to backing store
+h: huge page
+i: idle page)
+
+First line of above example says 300th block is accessed at 75.033841sec
+and the block's state is huge so it is written back to the backing
+storage. It's a debugging feature so anyone shouldn't rely on it to work
+properly.
+
Nitin Gupta
ngupta@vflare.org
diff --git a/Documentation/device-mapper/thin-provisioning.txt b/Documentation/device-mapper/thin-provisioning.txt
index 1699a55..ef63996 100644
--- a/Documentation/device-mapper/thin-provisioning.txt
+++ b/Documentation/device-mapper/thin-provisioning.txt
@@ -112,9 +112,11 @@
free space on the data device drops below this level then a dm event
will be triggered which a userspace daemon should catch allowing it to
extend the pool device. Only one such event will be sent.
-Resuming a device with a new table itself triggers an event so the
-userspace daemon can use this to detect a situation where a new table
-already exceeds the threshold.
+
+No special event is triggered if a just resumed device's free space is below
+the low water mark. However, resuming a device always triggers an
+event; a userspace daemon should verify that free space exceeds the low
+water mark when handling this event.
A low water mark for the metadata device is maintained in the kernel and
will trigger a dm event if free space on the metadata device drops below
diff --git a/Documentation/devicetree/bindings/arm/google-b1c1.txt b/Documentation/devicetree/bindings/arm/google-b1c1.txt
new file mode 100644
index 0000000..b954b42
--- /dev/null
+++ b/Documentation/devicetree/bindings/arm/google-b1c1.txt
@@ -0,0 +1,20 @@
+Google Platforms Device Tree Bindings
+---------------------------------------
+
+Google boards
+
+Required root node properties:
+- compatible: it should be "google,b1c1-sdm845";
+- qcom,board-id: qcom board-id as described in [1]. The following rules are used
+ for Google boards:
+ - Platform Type ID: must use 0x05 (which is OEM code reserved by QCOM)
+ - Major ID: global Google-wide major for the product
+ - Minor ID: product-specific Minor as HW version (EVT, DVT etc)
+ - Subtype: product-specific Subtype as HW modification
+
+Example:
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00010205 0>;
+
+
+[1] Documentation/devicetree/bindings/arm/msm/board-id.txt
diff --git a/Documentation/devicetree/bindings/cs40l2x.txt b/Documentation/devicetree/bindings/cs40l2x.txt
new file mode 100644
index 0000000..87de6080
--- /dev/null
+++ b/Documentation/devicetree/bindings/cs40l2x.txt
@@ -0,0 +1,119 @@
+CS40L20 Boosted Haptics Driver
+
+Required properties:
+
+ - compatible : One of "cirrus,cs40l20", "cirrus,cs40l25", "cirrus,cs40l25a"
+ or "cirrus,cs40l25b".
+
+ - reg : The I2C slave address of the device.
+
+ - VA-supply, VP-supply : Regulators for the device's VA and VP supplies,
+ respectively. See the following:
+ Documentation/devicetree/bindings/regulator/regulator.txt
+
+ - cirrus,boost-ind-nanohenry : Boost inductor value, expressed in nH. Valid
+ values include 1000, 1200, 1500 and 2200.
+
+ - cirrus,boost-cap-microfarad : Total equivalent boost capacitance on the VBST
+ and VAMP pins, derated at 11 volts DC. The value must be rounded to the
+ nearest integer and expressed in uF.
+
+ - cirrus,boost-ipk-milliamp : Boost inductor peak current, expressed in mA.
+ Valid values range from 1600 to 4500 (inclusive) in steps of 50.
+
+Optional properties:
+
+ - reset-gpios : GPIO used for resetting the device.
+
+ - cirrus,refclk-gpio2 : Boolean for configuring the device to expect its
+ 32.768-kHz reference clock on the REFCLK/GPIO2 pin. If this property is
+ omitted, the device expects its 32.768-kHz reference clock on the
+ ASP_BCLK/REFCLK pin. This property is ignored for CS40L25A and CS40L25B
+ devices as reference clock selection is configured automatically.
+
+ - cirrus,f0-default : Default LRA resonant frequency (f0), expressed as
+ follows: cirrus,f0-default = f0 (Hz) * 2^14. This value represents the
+ frequency used during playback of PWLE segments specified with frequency
+ equal to f0; it also serves as the unit-specific f0 input to the click
+ compensation algorithm. It can be overwritten at a later time by writing
+ to the f0_stored sysfs control.
+
+ If this value is omitted or specified as zero, the measurement recorded in
+ the f0_measured sysfs control is used. If LRA diagnostics has not been
+ administered and f0_measured is uninitialized, 2621440 (160 Hz) is used.
+
+ - cirrus,f0-min : Minimum LRA resonant frequency (f0) that may be written to
+ the f0_stored sysfs control, expressed using the same numerical format as
+ cirrus,f0-default. If this value is omitted or specified as zero, no lower-
+ bound validation is performed.
+
+ - cirrus,f0-max : Maximum LRA resonant frequency (f0) that may be written to
+ the f0_stored sysfs control, expressed using the same numerical format as
+ cirrus,f0-default. If this value is omitted or specified as zero, no upper-
+ bound validation is performed.
+
+ - cirrus,redc-default : Default LRA series resistance (ReDC), expressed as
+ follows: cirrus,redc-default = ReDC (ohms) / 5.857 * 2^17. This value
+ represents the unit-specific ReDC input to the click compensation algorithm.
+ It can be overwritten at a later time by writing to the redc_stored sysfs
+ control.
+
+ If this value is omitted or specified as zero, the measurement recorded in
+ the redc_measured sysfs control is used. If LRA diagnostics has not been
+ administered and redc_measured is uninitialized, 340787 (15.2 ohms) is used.
+
+ - cirrus,redc-min : Minimum LRA series resistance (ReDC) that may be written
+ to the redc_stored sysfs control, expressed using the same numerical format
+ as cirrus,redc-default. If this value is omitted or specified as zero, no
+ lower-bound validation is performed.
+
+ - cirrus,redc-max : Maximum LRA series resistance (ReDC) that may be written
+ to the redc_stored sysfs control, expressed using the same numerical format
+ as cirrus,redc-default. If this value is omitted or specified as zero, no
+ upper-bound validation is performed.
+
+ - cirrus,gpio1-rise-index : Specifies the wavetable index mapped to GPIO1
+ rising edges. If this value is omitted, specified as zero or exceeds the
+ maximum available index in the wavetable, GPIO1 rising edges are mapped to
+ index 1.
+
+ - cirrus,gpio1-fall-index : Specifies the wavetable index mapped to GPIO1
+ falling edges. If this value is omitted, specified as zero or exceeds the
+ maximum available index in the wavetable, GPIO1 falling edges are mapped to
+ index 2.
+
+ - cirrus,gpio1-fall-timeout : Specifies the number of 48-kHz periods for
+ which the device remains in the active state in search of a GPIO1 falling
+ edge, following a GPIO1 rising edge (the latter of which renders the device
+ active). If a GPIO1 falling edge does not arrive within this timeout, the
+ device automatically returns to the standby state and the subsequent GPIO1
+ falling edge is ignored. If this value is omitted or exceeds the maximum
+ timeout (8388607) then a default of 240000 is assumed (corresponding to
+ 240000 / 48000 = 5 seconds). If this value is specified as zero, the timeout
+ is effectively disabled.
+
+ - cirrus,gpio1-mode : Specifies the operating mode of the GPIO1 pin, equal to
+ one of the following.
+
+ 0 = enabled by default
+ 1 = disabled by default
+ 2 = disabled by default, but automatically enabled and disabled upon suspend
+ and resume, respectively
+
+ If this value is omitted or given an invalid value, mode 0 (enabled by
+ default) is assumed. Regardless of mode, the GPIO1 pin can be enabled or
+ disabled at will from user space using the gpio1_enable sysfs control.
+
+Example:
+
+ cs40l20: cs40l20@43 {
+ compatible = "cirrus,cs40l20";
+ reg = <0x43>;
+ reset-gpios = <&gpio0 54 0>;
+ VA-supply = <&dummy_vreg>;
+ VP-supply = <&dummy_vreg>;
+ cirrus,boost-ind-nanohenry = <1000>;
+ cirrus,boost-cap-microfarad = <4>;
+ cirrus,boost-ipk-milliamp = <4500>;
+ cirrus,refclk-gpio2;
+ };
diff --git a/Documentation/devicetree/bindings/display/msm/sde.txt b/Documentation/devicetree/bindings/display/msm/sde.txt
index 0589165..1c13e47 100644
--- a/Documentation/devicetree/bindings/display/msm/sde.txt
+++ b/Documentation/devicetree/bindings/display/msm/sde.txt
@@ -76,6 +76,9 @@
mixer number corresponding to the array index.
Optional properties:
+- qcom,sde-vsync-source GPIO VSYNC/TE Source for the SDE encoder. For
+ example, on SDM845 use 0 for GPIO10, 1 for
+ GPIO11, and 2 for GPIO12.
- clock-rate: List of clock rates in Hz.
- clock-max-rate: List of maximum clock rate in Hz that this device supports.
- connectors: Specifies the connector components.
diff --git a/Documentation/devicetree/bindings/dma/mv-xor-v2.txt b/Documentation/devicetree/bindings/dma/mv-xor-v2.txt
index 217a90e..9c38bbe 100644
--- a/Documentation/devicetree/bindings/dma/mv-xor-v2.txt
+++ b/Documentation/devicetree/bindings/dma/mv-xor-v2.txt
@@ -11,7 +11,11 @@
interrupts.
Optional properties:
-- clocks: Optional reference to the clock used by the XOR engine.
+- clocks: Optional reference to the clocks used by the XOR engine.
+- clock-names: mandatory if there is a second clock, in this case the
+ name must be "core" for the first clock and "reg" for the second
+ one
+
Example:
diff --git a/Documentation/devicetree/bindings/drm/msm/google-bl.txt b/Documentation/devicetree/bindings/drm/msm/google-bl.txt
new file mode 100644
index 0000000..7708936
--- /dev/null
+++ b/Documentation/devicetree/bindings/drm/msm/google-bl.txt
@@ -0,0 +1,146 @@
+Google binned low power mode panel
+
+This is an extension of QTI's mdss-dsi-panel to handle additional command
+sequences for low power modes.
+
+Required properties:
+- compatible: "google,dsi_binned_lp"
+
+
+==============================================================
+
+Google low power display modes
+
+Required properties:
+- compatible: "google,lp-modes"
+- label: Name of the entry
+- dsi-lp-brightness-threshold:
+ Max brightness supported by the mode
+- dsi-lp-command: Command sequence to enter this mode
+- dsi-lp-command-state: Command state used in this mode
+
+Example:
+
+/ {
+ ...
+ google,lp-modes {
+ lp-mode@0 {
+ label = "off";
+ google,dsi-lp-brightness-threshold = <0>;
+ google,dsi-lp-command = [
+ 05 01 00 00 00 00 01 22
+ ];
+ google,dsi-lp-command-state = "dsi_lp_mode";
+ };
+
+ lp-mode@1 {
+ label = "high";
+ google,dsi-lp-brightness-threshold = <255>;
+ google,dsi-lp-command = [
+ 39 01 00 00 00 00 03 51 01 FF
+ 05 01 00 00 00 00 01 12
+ ];
+ google,dsi-lp-command-state = "dsi_lp_mode";
+ };
+};
+
+==============================================================
+
+Google high brightness display mode
+
+This is an extension of QTI's mdss-dsi-panel to handle special display modes
+that allow for exceptionally high brightness.
+
+High brightness mode (HBM) is specified through 1 or more user brightness
+ranges. Conceptually, Each HBM range contains,
+1) User brightness threshold
+2) Panel brightness range
+3) DSI command (optional)
+
+HBM ranges must be sorted by brightness threshold and not overlap.
+An HBM range covers user brightness levels starting from the threshold, up to
+but not including the next HBM mode's threshold.
+The last HBM range covers user brightness levels starting from its threshold,
+up to and including qcom,mdss-brightness-max-level.
+
+When user brightness is updated, the appropriate HBM range is selected and the
+corresponding DSI command, if present, is sent to the panel. Then the user's
+brightness is linearly mapped to the panel's brightness range and the panel is
+updated if needed.
+
+Note that there is currently no support for backlight LUTs during HBM mode.
+
+HBM support also includes optional brightness dimming support. Brightness
+dimming refers to some panels' ability to gradually change brightness over
+time, smoothing the transition between current brightness and the latest
+brightness level requested by the host. The driver uses this feature, if
+supported by the panel, to smooth brightness changes as HBM is enabled, changes
+ranges, or disabled.
+To implement this support,
+1) HBM range entry sequences (dsi-hbm-range-entry-command) or exit sequences
+(dsi-hbm-exit-command) will include commands necessary to enable brightness
+dimming
+2) The corresponding HBM node will specify the number of frames dimming is
+expected to last on the panel side
+3) The same node will specify a command to send to the panel at the end of the
+brightness dimming duration, to disable brightness dimming.
+
+HBM uses the following properties. Each property is required, unless specified
+as optional:
+- google,hbm-ranges: Parent list of HBM ranges.
+- hbm-range@n: An HBM range, starting at n=0.
+- dsi-hbm-exit-command: Optional command sequence to exit HBM.
+- dsi-hbm-commands-state: Optional command state used for HBM commands.
+- dsi-hbm-exit-num-dimming-frames: Optional number of frames indicating
+ brightness dimming duration. Mandatory if dsi-hbm-exit-dimming-stop-command
+ is specified.
+- dsi-hbm-exit-dimming-stop-command: Optional command to send at the end of the
+ brightness dimming duration, to disable dimming. Mandatory if
+ dsi-hbm-exit-num-dimming-frames is specified.
+
+Each hbm-range@n contains the following properties. Each property is required,
+unless specified as optional:
+- dsi-hbm-range-brightness-threshold: Starting user brightness for this HBM
+ range.
+- dsi-hbm-range-entry-command: Optional command sequence to enter this range.
+- dsi-hbm-range-commands-state: Optional command state used in this range.
+- dsi-hbm-range-bl-min-level: Start of panel brightness range
+ (inclusive).
+- dsi-hbm-range-bl-max-level: End of panel brightness range (inclusive).
+- dsi-hbm-range-num-dimming-frames: Optional. Analogous to
+ dsi-hbm-exit-num-dimming-frames, but for a given HBM range.
+- dsi-hbm-range-dimming-stop-command: Optional. Analogous to
+ dsi-hbm-exit-dimming-stop-command, but for a given HBM range.
+
+Example:
+
+/ {
+ ...
+ google,hbm-ranges {
+ google,dsi-hbm-exit-command = [
+ 15 01 00 00 00 00 02 53 20
+ ];
+ google,dsi-hbm-commands-state = "dsi_lp_mode";
+
+ hbm-range@0 {
+ google,dsi-hbm-range-brightness-threshold = <1>;
+
+ google,dsi-hbm-range-bl-min-level = <0x5>;
+ google,dsi-hbm-range-bl-max-level = <0x3ff>;
+ google,dsi-hbm-range-entry-command = [
+ 15 01 00 00 00 00 02 53 20
+ ];
+ google,dsi-hbm-range-commands-state = "dsi_lp_mode";
+ };
+
+ hbm-range@1 {
+ google,dsi-hbm-range-brightness-threshold = <170>;
+
+ google,dsi-hbm-range-bl-min-level = <0x3ff>;
+ google,dsi-hbm-range-bl-max-level = <0x3ff>;
+ google,dsi-hbm-range-entry-command = [
+ 15 01 00 00 00 00 02 53 E0
+ ];
+ google,dsi-hbm-range-commands-state = "dsi_lp_mode";
+ };
+ };
diff --git a/Documentation/devicetree/bindings/drm/msm/mdss-dsi-panel.txt b/Documentation/devicetree/bindings/drm/msm/mdss-dsi-panel.txt
index 24290c8..104aec9 100644
--- a/Documentation/devicetree/bindings/drm/msm/mdss-dsi-panel.txt
+++ b/Documentation/devicetree/bindings/drm/msm/mdss-dsi-panel.txt
@@ -187,6 +187,16 @@
255 = default value.
- qcom,mdss-brightness-max-level: Specifies the max brightness level supported.
255 = default value.
+- qcom,mdss-dsi-bl-lut: Backlight LUT; maps userspace brightness levels 0 through
+ qcom,mdss-brightness-max-level to panel brightness levels. Specifying
+ this LUT overrides qcom,mdss-dsi-bl-min-level and
+ qcom,mdss-dsi-bl-max-level.
+ If unspecified, this LUT is unused and unallocated. Instead, userspace
+ brightness gets mapped linearly, during runtime, to the range defined by
+ qcom,mdss-dsi-bl-min-level and qcom,mdss-dsi-bl-max-level.
+- qcom,mdss-dsi-bl-vr-min-safe-level: Specify a minimum safe panel brightness
+ level during VR mode. This limit comes into effect only when
+ attempting to change brightness level while VR mode is active.
- qcom,mdss-dsi-interleave-mode: Specifies interleave mode.
0 = default value.
- qcom,mdss-dsi-panel-type: Specifies the panel operating mode.
@@ -422,12 +432,22 @@
setting the panel power mode.
"dsi_lp_mode" = DSI low power mode
"dsi_hs_mode" = DSI high speed mode
+- qcom,mdss-dsi-vr-command: An optional byte stream containing commands to switch the panel
+ into a mode designed for virtual reality applications.
+- qcom,mdss-dsi-vr-command-state: The control state for sending qcom,mdss-dsi-vr-command.
+ "dsi_lp_mode" = DSI low power mode (default)
+ "dsi_hs_mode" = DSI high speed mode
+- qcom,mdss-dsi-novr-command: If qcom,mdss-dsi-vr-command is defined, a byte stream containing
+ commands to switch the panel out of virtual-reality mode to normal operation.
+- qcom,mdss-dsi-novr-command-state: The control state for sending qcom,mdss-dsi-novr-command
+ See qcom,mdss-dsi-vr-command-state.
- qcom,mdss-dsi-panel-status-check-mode:Specifies the panel status check method for ESD recovery.
"bta_check" = Uses BTA to check the panel status
"reg_read" = Reads panel status register to check the panel status
"reg_read_nt35596" = Reads panel status register to check the panel
status for NT35596 panel.
"te_signal_check" = Uses TE signal behaviour to check the panel status
+ "irq_check" = Uses IRQ signal behaviour to check the panel status.
- qcom,mdss-dsi-panel-status-read-length: Integer array that specify the expected read-back length of values
for each of panel registers. Each length is corresponding to number of
returned parameters of register introduced in specification.
@@ -519,8 +539,22 @@
to identify the default topology for the
display. The first set is indexed by the
value 0.
+- qcom,mdss-dsi-hs-pps boolean: Send PPS command in high speed (default false)
+- google,mdss-dsi-panel-vendor string: panel vendor name
+- google,mdss-dsi-panel-sn-location Panel serial number is read by MIPI command. And it is defined by a
+ set of 3 values in the order:
+ - address of DDIC's register
+ - number of start byte. example: 0 means start from byte 0.
+ - length of the SN
- qcom,mdss-dsi-dma-schedule-line: An integer value indicates the line number after vertical active
region, at which command DMA needs to be triggered.
+- google,dsi-esd-irq Specifies the pinctrl setting if qcom,mdss-dsi-panel-status-check-mode
+ is "irq_check".
+- google,esd-irq-mode Specifies the interrupt line behaviour for ESD recovery.
+ "trigger_rising" = Triggered when signal is changed from low to high.
+ "trigger_falling" = Triggered when signal is changed from high to low.
+ "trigger_high" = Triggered when signal is high.
+ "trigger_low" = Triggered when signal is low.
Required properties for sub-nodes: None
Optional properties:
diff --git a/Documentation/devicetree/bindings/input/ti,drv2624.txt b/Documentation/devicetree/bindings/input/ti,drv2624.txt
new file mode 100644
index 0000000..49bc01b
--- /dev/null
+++ b/Documentation/devicetree/bindings/input/ti,drv2624.txt
@@ -0,0 +1,33 @@
+* Texas Instruments - drv2624 Haptics driver
+
+Required properties:
+ - compatible - "ti,drv2624" - DRV2624
+ - reg - I2C slave address
+ - ti,reset-gpio - GPIO to reset the device
+
+Optional properties:
+ - ti,irq-gpio - GPIO to receive the interrupt
+ - ti,voltage-comp - Voltage compensation for any resistive losses in the
+ driver IC.
+ - ti,ol-lra-frequency - LRA frequency for open-loop mode
+ - ti,bemf-factor - Sets the feedback gain for closed-loop operation
+ - ti,bemf-gain - Sets the analog gain of the back-EMF amplifier
+ - ti,blanking-time - Time waiting for BEMF to settle. Before ADC
+ sampling
+ - ti,idiss-time - Sime waiting for inductor current to discharge
+ - ti,zc-det-time - Zero crossing detection time
+ - ti,lra-wave-shape - Selects which shape to use for driving the LRA
+ when in open loop mode. 0 - Square Wave, 1 - Sine Wave.
+ - ti,waveform-interval - Sets the internal memory playback interval to
+ either 5 ms or 1 ms. 0 - 5 ms, 1 - 1 ms.
+
+Example:
+
+drv2624: drv2624@5a {
+ compatible = "ti,drv2624";
+ reg = <0x5a>;
+ ti,reset-gpio = <&tlmm 95 0>;
+};
+
+For more product information please see the link below:
+http://www.ti.com/product/drv2624
diff --git a/Documentation/devicetree/bindings/input/touchscreen/ftm4_ts.txt b/Documentation/devicetree/bindings/input/touchscreen/ftm4_ts.txt
new file mode 100644
index 0000000..6221b40
--- /dev/null
+++ b/Documentation/devicetree/bindings/input/touchscreen/ftm4_ts.txt
@@ -0,0 +1,40 @@
+FTS Capacitive touch screen controller (FingerTipS)
+----------------
+
+Required properties:
+- compatible: "stm,ftm4_fts"
+- stm,irq_gpio: Interrupt GPIO
+- stm,irq_type: Type of IRQ (edge/level and high/low)
+- stm,num_lines: Number of RX(SenseChannelLength), TX(ForceChannelLength)
+- stm,max_coords: Maximum X/Y values reported to the client
+- stm,regulator_dvdd: 1.8V digital supply
+- stm,regulator_avdd: 3.3V analog supply
+- stm,firmware_name: Firmware file to load
+
+Optional properties:
+- stm,switch_gpio: Used to control external I2C switch
+
+Node should be a child node to a relevant I2C bus.
+
+Example:
+&qupv3_se2_i2c {
+ ftm4_fts@49 {
+ status = "ok";
+ compatible = "stm,ftm4_fts";
+ reg = <0x49>;
+
+ pinctrl-names = "on_state", "off_state";
+ pinctrl-0 = <&ts_active>;
+ pinctrl-1 = <&ts_suspend>;
+
+ stm,switch_gpio= <&tlmm 128 0>;
+ stm,irq_gpio = <&tlmm 125 0>;
+ stm,irq_type = <0x2004>; /* Trigger high */
+ stm,num_lines = <32 16>;
+ stm,max_coords = <1440 2880>;
+ stm,regulator_dvdd = "pm8998_l14";
+ stm,regulator_avdd = "pm8998_l19";
+
+ stm,firmware_name = "ftm4_fw.ftb";
+ };
+};
diff --git a/Documentation/devicetree/bindings/misc/access_ramoops.txt b/Documentation/devicetree/bindings/misc/access_ramoops.txt
new file mode 100644
index 0000000..6ae5d42
--- /dev/null
+++ b/Documentation/devicetree/bindings/misc/access_ramoops.txt
@@ -0,0 +1,14 @@
+access_ramoops driver
+=====================
+
+The access_ramoops driver allows userspace access to the persistent RAM storage
+used for oops and panics as well as the metadata required to decrypt that data.
+
+Required properties:
+
+- compatible: must be "access_ramoops"
+
+- memory-region: phandle to a region of memory that is preserved between reboots
+
+- label: name of the region, will be used to create the device node /dev/<label>
+
diff --git a/Documentation/devicetree/bindings/misc/citadel-spi.txt b/Documentation/devicetree/bindings/misc/citadel-spi.txt
new file mode 100644
index 0000000..661f093
--- /dev/null
+++ b/Documentation/devicetree/bindings/misc/citadel-spi.txt
@@ -0,0 +1,14 @@
+* Google Inc. Citadel SoC
+
+Required properties:
+- compatible: Should contain "google,citadel".
+- reg: SPI address for chip
+- spi-max-frequency: max SPI frequency supported by the device
+
+Example:
+ citadel@0 {
+ compatible = "google,citadel";
+ reg = <0>;
+ spi-max-frequency = <50000000>;
+ };
+};
diff --git a/Documentation/devicetree/bindings/misc/keydebug.txt b/Documentation/devicetree/bindings/misc/keydebug.txt
new file mode 100644
index 0000000..ed7a1c9
--- /dev/null
+++ b/Documentation/devicetree/bindings/misc/keydebug.txt
@@ -0,0 +1,27 @@
+keydebug driver
+=========================
+
+When a system is locked up and unresponsive, a user may long press key to force
+reset it. keydebug driver add debug function to collect more information in this
+path before system restart.
+
+Collects:
+-Top CPU usage tasks
+-Call stack of uninterruptible tasks
+-Call stack of each online core
+
+Required properties:
+- compatible: must be "keydebug"
+
+- keys_down: The input key-code used to active debug function count down.
+- key_down_delay: The time the input key need to be hold to trigger debug
+ event, value is in milliseconds.
+- dbg_fn_delay: The delay time to active debug function, value is in
+ milliseconds.
+Example:
+ keydebug {
+ compatible = "keydebug";
+ key_down_delay = <7000>;
+ keys_down = <116>;
+ dbg_fn_delay = <2500>;
+ }
diff --git a/Documentation/devicetree/bindings/net/dsa/b53.txt b/Documentation/devicetree/bindings/net/dsa/b53.txt
index d6c6e41..6192f02 100644
--- a/Documentation/devicetree/bindings/net/dsa/b53.txt
+++ b/Documentation/devicetree/bindings/net/dsa/b53.txt
@@ -10,6 +10,7 @@
"brcm,bcm53128"
"brcm,bcm5365"
"brcm,bcm5395"
+ "brcm,bcm5389"
"brcm,bcm5397"
"brcm,bcm5398"
diff --git a/Documentation/devicetree/bindings/net/dsa/qca8k.txt b/Documentation/devicetree/bindings/net/dsa/qca8k.txt
index 9c67ee4..bbcb255 100644
--- a/Documentation/devicetree/bindings/net/dsa/qca8k.txt
+++ b/Documentation/devicetree/bindings/net/dsa/qca8k.txt
@@ -2,7 +2,10 @@
Required properties:
-- compatible: should be "qca,qca8337"
+- compatible: should be one of:
+ "qca,qca8334"
+ "qca,qca8337"
+
- #size-cells: must be 0
- #address-cells: must be 1
@@ -14,6 +17,20 @@
referencing the internal PHY connected to it. The CPU port of this switch is
always port 0.
+A CPU port node has the following optional node:
+
+- fixed-link : Fixed-link subnode describing a link to a non-MDIO
+ managed entity. See
+ Documentation/devicetree/bindings/net/fixed-link.txt
+ for details.
+
+For QCA8K the 'fixed-link' sub-node supports only the following properties:
+
+- 'speed' (integer, mandatory), to indicate the link speed. Accepted
+ values are 10, 100 and 1000
+- 'full-duplex' (boolean, optional), to indicate that full duplex is
+ used. When absent, half duplex is assumed.
+
Example:
@@ -53,6 +70,10 @@
label = "cpu";
ethernet = <&gmac1>;
phy-mode = "rgmii";
+ fixed-link {
+ speed = 1000;
+ full-duplex;
+ };
};
port@1 {
diff --git a/Documentation/devicetree/bindings/net/macb.txt b/Documentation/devicetree/bindings/net/macb.txt
index 1506e94..d1f435c 100644
--- a/Documentation/devicetree/bindings/net/macb.txt
+++ b/Documentation/devicetree/bindings/net/macb.txt
@@ -10,6 +10,7 @@
Use "cdns,pc302-gem" for Picochip picoXcell pc302 and later devices based on
the Cadence GEM, or the generic form: "cdns,gem".
Use "atmel,sama5d2-gem" for the GEM IP (10/100) available on Atmel sama5d2 SoCs.
+ Use "atmel,sama5d3-macb" for the 10/100Mbit IP available on Atmel sama5d3 SoCs.
Use "atmel,sama5d3-gem" for the Gigabit IP available on Atmel sama5d3 SoCs.
Use "atmel,sama5d4-gem" for the GEM IP (10/100) available on Atmel sama5d4 SoCs.
Use "cdns,zynq-gem" Xilinx Zynq-7xxx SoC.
diff --git a/Documentation/devicetree/bindings/net/meson-dwmac.txt b/Documentation/devicetree/bindings/net/meson-dwmac.txt
index 89e62dd..da37da0 100644
--- a/Documentation/devicetree/bindings/net/meson-dwmac.txt
+++ b/Documentation/devicetree/bindings/net/meson-dwmac.txt
@@ -10,6 +10,7 @@
- "amlogic,meson6-dwmac"
- "amlogic,meson8b-dwmac"
- "amlogic,meson-gxbb-dwmac"
+ - "amlogic,meson-axg-dwmac"
Additionally "snps,dwmac" and any applicable more
detailed version number described in net/stmmac.txt
should be used.
diff --git a/Documentation/devicetree/bindings/nfc/ese/pn81a.txt b/Documentation/devicetree/bindings/nfc/ese/pn81a.txt
new file mode 100644
index 0000000..3624018
--- /dev/null
+++ b/Documentation/devicetree/bindings/nfc/ese/pn81a.txt
@@ -0,0 +1,23 @@
+* NXP PN81A Embedded Secure Element
+
+PN81A is part of an NXP NFC controller and depends on
+access to eSE power via the controller.
+
+Required properties:
+- compatible: Should be "nxp,pn81a"
+- reg: SPI chip select.
+- spi-max-frequency: SPI maximum speed.
+- nxp,nfcc: Name of the i2c device where an nq-nci device is mapped.
+- nxp,clear-n: Optional GPIO for clearing a inbound application GPIO.
+
+Examples:
+
+&spi_bus {
+ pn81a@0 {
+ compatible = "nxp,pn81a";
+ reg = <0>
+ spi-max-frequency = <8000000>;
+ nxp,nfcc = "8-0028";
+ nxp,clear-n = <&tlmm 29 0x00>;
+ };
+};
diff --git a/Documentation/devicetree/bindings/pinctrl/meson,pinctrl.txt b/Documentation/devicetree/bindings/pinctrl/meson,pinctrl.txt
index fe7fe0b..1b98817 100644
--- a/Documentation/devicetree/bindings/pinctrl/meson,pinctrl.txt
+++ b/Documentation/devicetree/bindings/pinctrl/meson,pinctrl.txt
@@ -3,8 +3,10 @@
Required properties for the root node:
- compatible: one of "amlogic,meson8-cbus-pinctrl"
"amlogic,meson8b-cbus-pinctrl"
+ "amlogic,meson8m2-cbus-pinctrl"
"amlogic,meson8-aobus-pinctrl"
"amlogic,meson8b-aobus-pinctrl"
+ "amlogic,meson8m2-aobus-pinctrl"
"amlogic,meson-gxbb-periphs-pinctrl"
"amlogic,meson-gxbb-aobus-pinctrl"
- reg: address and size of registers controlling irq functionality
diff --git a/Documentation/devicetree/bindings/power/supply/google_charger.txt b/Documentation/devicetree/bindings/power/supply/google_charger.txt
new file mode 100644
index 0000000..0008946
--- /dev/null
+++ b/Documentation/devicetree/bindings/power/supply/google_charger.txt
@@ -0,0 +1,46 @@
+google_charger
+~~~~~~~~~~~~~~
+
+Required properties:
+ - compatible : "google,charger"
+ - google,chg-power-supply: name of the charge control power supply
+ - google,bat-power-supply: name of the Fuel Gauge power supply
+ - google,chg-update-interval: delay in ms to wait before re-running the work
+ item
+ - google,chg-battery-capacity: the nominal capacity used to compute the
+ effective 1C charge rate
+ - google,chg-temp-limits: the temperature buckets limits defining the indexes
+ of the constant charge limits (row index in google,chg-cc-limits)
+ - google,chg-cv-limits: steps voltage limits (column index in
+ google,chg-cc-limits)
+ - google,chg-cc-limits: the array of constant current charge rates given in %
+ of google,chg-battery-capacity
+
+Optional properties:
+ - google,wlc-power-supply: name of wireless charing input power supply
+ - google,cv-hw-resolution: the constant voltage setting resolution in micro
+ volt defaults to 25000
+ - google,cc-hw-resolution: the constant current setting resolution in micro amp
+ defaults to 25000
+ - google,cv-range-accuracy: the voltage margin in micro volt used at taper to
+ determine whether battery voltage should be raised in an attempt to match
+ tier voltage for ir drop compensation. Defaults to cv-hw-resolution / 2.
+ - google,cv-debounce-cnt: the number of cv-update-interval to wait after
+ entering taper or after raising charging voltage for ir drop compensation
+ before a new attempt. Defaults to 3 (eg 6 seconds for default
+ cv-update-interval).
+ - google,cv-update-interval: delay in ms to wait before re-running the work
+ item while in taper. Defaults to 2000 (2 seconds)
+ - google,cv-tier-ov-delay: the number of cv-update-interval to wait (penalty)
+ during ir drop compensation before a new attempt to raise voltage when
+ battery voltage is found above tier limit. Defaults to 10 cycles (20
+ seconds for default cv-update-interval).
+ - google,chg-tolerance: the deci-percent tolerance we need to give the PMIC to
+ ensure actual current is below chg-cc-limits (divide by 10 to get
+ percentage points). Max allowed tolerance is 25%.
+- google,fv-uv-margin-dpct: the deci-percent margin on vtier that charger float
+ voltage must not exceed when calculating target voltage during IRDROP
+ compensation. Default 0 (disabled), set to 1030 for 3%.
+- google,cv-tier-switch-cnt: the number of consecutive poll cyles with current
+ under next tier that are needed before switching charge tier.
+ Default 3 poll cycles.
\ No newline at end of file
diff --git a/Documentation/devicetree/bindings/power/supply/max1720x_battery.txt b/Documentation/devicetree/bindings/power/supply/max1720x_battery.txt
new file mode 100644
index 0000000..b3c3f7fa
--- /dev/null
+++ b/Documentation/devicetree/bindings/power/supply/max1720x_battery.txt
@@ -0,0 +1,27 @@
+MAX17201/MAX17205 Fuel Gauge
+----------------------------
+
+Required properties :
+ - compatible : "maxim,max1720x"
+
+Optional properties :
+ - maxim,psy-type-unknown: if set, forces poser supply type to be
+ POWER_SUPPLY_TYPE_UNKNOWN rather than POWER_SUPPLY_TYPE_BATTERY
+
+ - io-channels: see Documentation/devicetree/bindings/iio/iio-bindings.txt
+ - io-channel-names: see Documentation/devicetree/bindings/iio/iio-bindings.txt
+ - maxim,batt-id-range-pct: tolerance in percent on the resitance ID value. if
+ not provided, defaults to 20%.
+
+ - maxim,config: node in which children would have a "maxim,batt-id-kohm"
+ property and a "maxim,regval" to allow applying configuration changes given a
+ battery.
+ - maxim,batt-id-kohm: the value of the battery resitance ID the config applies
+ to.
+ - maxim,n_regval: An array of u16 that are pairs of register address and
+ corresponding values. These values will be written to the shadow registers
+ of Maxim fuel gauge their address must be within 0x80 - 0xDF to be considered
+ - maxim,r_regval: An array of u16 that are pairs of register address and
+ corresponding values. These values will be written to the ModelGauge m5
+ Register their address must be within 0x00 - 0x4D, 0xB0 - 0xDF to be
+ considered.
diff --git a/Documentation/devicetree/bindings/power/supply/overheat_mitigation.txt b/Documentation/devicetree/bindings/power/supply/overheat_mitigation.txt
new file mode 100644
index 0000000..acdbbef
--- /dev/null
+++ b/Documentation/devicetree/bindings/power/supply/overheat_mitigation.txt
@@ -0,0 +1,11 @@
+google,overheat_mitigation
+~~~~~~~~~~~~~~
+
+Required properties:
+ - compatible : "google,overheat_mitigation"
+ - google,begin-mitigation-temp: USB port temperature threshold in decidegrees
+ Celsius at which to begin overheat mitigation
+ - google,end-mitigation-temp: USB port temperature threshold in decidegrees
+ Celsius at which to end overheat mitigation
+ - google,port-overheat-work-interval: period in milliseconds at which the
+ driver should monitor the overheat status
diff --git a/Documentation/devicetree/bindings/power/supply/qcom/qpnp-smb2.txt b/Documentation/devicetree/bindings/power/supply/qcom/qpnp-smb2.txt
index afa8009..d088865 100644
--- a/Documentation/devicetree/bindings/power/supply/qcom/qpnp-smb2.txt
+++ b/Documentation/devicetree/bindings/power/supply/qcom/qpnp-smb2.txt
@@ -92,6 +92,15 @@
maximum charge current in mA for each thermal
level.
+- google,wlc-thermal-mitigation
+ Usage: optional
+ Value type: Array of <u32>
+ Definition: Array of DC input current limit values for
+ different system thermal mitigation levels.
+ This should be a flat array that denotes the
+ maximum DC input current in micro-amps for each
+ thermal level.
+
- io-channels
Usage: optional
Value type: List of <phandle u32>
diff --git a/Documentation/devicetree/bindings/regulator/qpnp-labibb-regulator.txt b/Documentation/devicetree/bindings/regulator/qpnp-labibb-regulator.txt
index a5e607d..e5a2c28 100644
--- a/Documentation/devicetree/bindings/regulator/qpnp-labibb-regulator.txt
+++ b/Documentation/devicetree/bindings/regulator/qpnp-labibb-regulator.txt
@@ -52,6 +52,8 @@
and IBB regulators. This will be mostly
needed when LAB and IBB are operating
in standalone mode to vote for MBG.
+- qcom,aod-mode: A boolean property which indicates if
+ AOD mode is supported for AMOLED panel.
Following properties are available only for PM660A:
@@ -352,6 +354,11 @@
qcom,pmic-revid = <&pmi8994_revid>;
qcom,skip-2nd-swire-cmd;
+ /* For AMOLED */
+ qcom,qpnp-labibb-mode = "amoled";
+ qcom,swire-control;
+ qcom,aod-mode;
+
lab_regulator: qcom,lab@de00 {
reg = <0xde00 0x100>;
reg-names = "lab";
diff --git a/Documentation/devicetree/bindings/reserved-memory/ramoops.txt b/Documentation/devicetree/bindings/reserved-memory/ramoops.txt
index e81f821..92ec6e7 100644
--- a/Documentation/devicetree/bindings/reserved-memory/ramoops.txt
+++ b/Documentation/devicetree/bindings/reserved-memory/ramoops.txt
@@ -22,7 +22,10 @@
- compatible: must be "ramoops"
-- reg: region of memory that is preserved between reboots
+- memory-region: region of memory that is preserved between reboots
+- alt-memory-region: region of memory that is preserved between reboots
+ which contains an AES encrypted copy of the persistent
+ RAM storage.
Optional properties:
diff --git a/Documentation/devicetree/bindings/soc/qcom/modemsmem.txt b/Documentation/devicetree/bindings/soc/qcom/modemsmem.txt
new file mode 100644
index 0000000..9e9fb91
--- /dev/null
+++ b/Documentation/devicetree/bindings/soc/qcom/modemsmem.txt
@@ -0,0 +1,14 @@
+modemsmem (Modem Shared Memory) Driver
+
+The modemsmem driver is used to notify with modem subsystem about
+hardware and software information via SMD channel.
+
+Required properties:
+
+- compatible: name of the component used for driver matching, should be:
+ "modemsmem"
+
+Example:
+ modemsmem {
+ compatible = "modemsmem";
+ };
diff --git a/Documentation/devicetree/bindings/sound/cs35l36.txt b/Documentation/devicetree/bindings/sound/cs35l36.txt
new file mode 100644
index 0000000..a31ab0e
--- /dev/null
+++ b/Documentation/devicetree/bindings/sound/cs35l36.txt
@@ -0,0 +1,150 @@
+CS35L36 Speaker Amplifier
+
+Required properties:
+
+ - compatible : "cirrus,cs35l36"
+
+ - reg : the I2C address of the device for I2C
+
+ - VA-supply, VP-supply : power supplies for the device,
+ as covered in
+ Documentation/devicetree/bindings/regulator/regulator.txt.
+
+Optional properties:
+ - cirrus,sclk-force-output : Audio serial port SCLK force
+ output control. Forces the SCLK to continue to drive even
+ if no ASP_TXn channels are enabled.
+
+ - cirrus,lrclk-force-output : Audio serial port LRCLK force
+ output control. Forces the LRCLK to continue to drive even
+ if no ASP_TXn channels are enabled.
+
+ - cirrus,right-channel-amp : Boolean to determine which channel
+ the amplifier is to receive the audio data on. If present the
+ amplifier receives data on the right channel of I2S data.
+ If not present the amplifier receives data on the left
+ channel of I2S data
+
+ - cirrus,boost-ctl-millivolt : Boost Voltage Value. Configures the boost
+ converter's output voltage in mV. The range is from 2550mV to 10000mV with
+ increments of 50mV.
+ (Default) VP
+
+ - cirrus,boost-ctl-select : Boost converter control source selection.
+ Selects the source of the BST_CTL target VBST voltage for the boost
+ converter to generate.
+ 0x00 - Control Port Value
+ 0x01 - Class H Tracking
+ 0x10 - MultiDevice Sync Value
+
+ - cirrus,boost-peak-milliamp : Boost-converter peak current limit in mA.
+ Configures the peak current by monitoring the current through the boost FET.
+ Range starts at 1600mA and goes to a maximum of 4500mA with increments of
+ 50mA.
+ (Default) 4.50 Amps
+
+ - cirrus,amp-gain-zc : Boolean to determine if to use Amplifier gain-change
+ zero-cross
+
+ - cirrus,ldm-mode-select : Amplifier level-dependent muting enable mode selection.
+ Configures whether the amplifier level-dependent muting is enabled for the various
+ blocks when the selected audio source has been below the AMP_LDM_THLD for a period
+ of time configured by AMP_LDM_DELAY. Any and all bits can be set.
+
+ 00000 - All level-dependent muting functionality disabled (Default)
+ xxxx1 - Amplifier level-dependent muting detection enabled (OUT+/-; switching)
+ xxx1x - Boost converter DCM detection enabled
+ xx1xx - VMON low power mode detection enabled
+ x1xxx - IMON low power mode detection enabled
+ 1xxxx - VP brownout prevention detection enabled
+
+ - cirrus,pdm-ldm-exit : Boolean : Amplifier PDM mode digital volume ramp enable.
+ When AMP_PDM_MODE = 1 controls whether the digital volume soft ramp is enabled
+ or disabled when exiting a level-dependent muted condition. The AMP_RAMP_PDM_DN
+ control is used to control desired ramp rate.
+
+ - cirrus,pdm-ldm-enter : Boolean: Amplifier PDM mode digital volume ramp enable.
+ When AMP_PDM_MODE = 1 controls whether the digital volume soft ramp is enabled
+ or disabled when entering a level-dependent muted condition. The AMP_RAMP_PDM_UP
+ control is used to control desired ramp rate.
+
+ - cirrus,pdm-channel-select : PDM Channel select.
+ 0x00 (Default) : Selects Channel A data when the PDM audio input is selected.
+ 0x01 : Selects Channel B data when the PDM audio input is selected
+
+ - cirrus,weak-fet-disable : Boolean : The strength of the output drivers is
+ reduced when operating in a Weak-FET Drive Mode and must not be used to drive
+ a large load.
+
+ - cirrus,classh-wk-fet-delay : Weak-FET entry delay. Controls the delay
+ (in ms) before the Class H algorithm switches to the weak-FET voltage
+ (after the audio falls and remains below the value specified in WKFET_AMP_THLD).
+
+ 0 = 0ms
+ 1 = 5ms
+ 2 = 10ms
+ 3 = 50ms
+ 4 = 100ms (Default)
+ 5 = 200ms
+ 6 = 500ms
+ 7 = 1000ms
+
+ - cirrus,classh-weak-fet-thld-millivolt : Weak-FET amplifier drive threshold.
+ Configures the signal threshold at which the PWM output stage enters
+ weak-FET operation. The range is 50mV to 700mV in 50mV increments.
+
+ - cirrus,temp-warn-threshold : Amplifier overtemperature warning threshold.
+ Configures the threshold at which the overtemperature warning condition occurs.
+ When the threshold is met, the overtemperature warning attenuation is applied
+ and the TEMP_WARN_EINT interrupt status bit is set.
+ If TEMP_WARN_MASK = 0, INTb is asserted.
+
+ 0 = 105C
+ 1 = 115C
+ 2 = 125C (Default)
+ 3 = 135C
+
+Example:
+
+cs35l36: cs35l36@40 {
+ #sound-dai-cells = <1>;
+ compatible = "cirrus,cs35l36";
+ reg = <0x40>;
+ reset-gpios = <&gpio0 54 0>;
+ interrupt-parent = <&gpio0>;
+ interrupts = <55 8>;
+ VA-supply = <&dummy_vreg>;
+ VP-supply = <&dummy_vreg>;
+ cirrus,left-channel-amp;
+ cirrus,boost-ind-nanohenry = <1000>; /* 1uH */
+ cirrus,boost-ctl-millivolt = <10000>;
+ cirrus,boost-peak-milliamp = <4500>;
+ cirrus,amp-gain-zc;
+ cirrus,ldm-mode-select = <0x01>;
+ cirrus,pdm-ldm-exit;
+ cirrus,pdm-ldm-enter;
+ cirrus,weak-fet-delay = <0x04>;
+ cirrus,weak-fet-thld = <0x01>;
+ cirrus,temp-warn-threshold = <0x01>;
+
+ cirrus,irq-config {
+ cirrus,irq-drive-select = <0x01>;
+ cirrus,irq-polarity = <0x01>;
+ cirrus,irq-gpio-select = <0x01>;
+ cirrus,irq-output-enable = <0x01>;
+ cirrus,irq-src-select = <0x01>;
+ };
+
+ /* Slot Number, Output Select */
+ cirrus,asp-tx-config {
+ cirrus,tx1-config = /bits/ 8 <0x00 0x18>;
+ cirrus,tx2-config = /bits/ 8 <0x01 0x19>;
+ /*
+ * cirrus,tx3-config = /bits/ 8 <0x02 0x28>;
+ * cirrus,tx4-config = /bits/ 8 <0x03 0x29>;
+ * cirrus,tx5-config = /bits/ 8 <0x04 0x20>;
+ * cirrus,tx6-config = /bits/ 8 <0x05 0x00>;
+ */
+ };
+ };
+};
diff --git a/Documentation/devicetree/bindings/sound/max98927.txt b/Documentation/devicetree/bindings/sound/max98927.txt
new file mode 100644
index 0000000..d610879
--- /dev/null
+++ b/Documentation/devicetree/bindings/sound/max98927.txt
@@ -0,0 +1,39 @@
+max98927 audio CODEC
+
+This device supports I2C.
+
+Required properties:
+
+ - compatible : "maxim,max98927"
+
+ - maxim,vmon-slot-no : slot number used to send voltage information
+ or in inteleave mode this will be used as
+ interleave slot.
+ This property can be set to values from 0 to 15 for slot 0 to 15.
+ Default value is 0.
+
+ - maxim,imon-slot-no : slot number used to send current information
+ This property can be set to values from 0 to 15 for slot 0 to 15.
+ Default value is 0.
+
+ - maxim,interleave-mode : When using two MAX98927 in a system it is
+ possible to create ADC data that that will
+ overflow the frame size. Digital Audio Interleave
+ mode provides a means to output VMON and IMON data
+ from two devices on a single DOUT line when running
+ smaller frames sizes such as 32 BCLKS per LRCLK or
+ 48 BCLKS per LRCLK.
+ This property can be set to values from 0 to 1 for off and on.
+ Default value is 0.
+
+ - reg : the I2C address of the device for I2C
+
+Example:
+
+codec: max98927@3a {
+ compatible = "maxim,max98927";
+ maxim,vmon-slot-no = <0>;
+ maxim,imon-slot-no = <1>;
+ maxim,interleave-mode = <0>;
+ reg = <0x3a>;
+};
diff --git a/Documentation/devicetree/bindings/spi/qcom,spi-geni-qcom.txt b/Documentation/devicetree/bindings/spi/qcom,spi-geni-qcom.txt
index 866d004..f584c482 100644
--- a/Documentation/devicetree/bindings/spi/qcom,spi-geni-qcom.txt
+++ b/Documentation/devicetree/bindings/spi/qcom,spi-geni-qcom.txt
@@ -28,6 +28,7 @@
Optional properties:
- qcom,rt: Specifies if the framework worker thread for this
controller device should have "real-time" priority.
+-qcom,disable-autosuspend: Specifies to disable runtime PM auto suspend.
SPI slave nodes must be children of the SPI master node and can contain
properties described in Documentation/devicetree/bindings/spi/spi-bus.txt
diff --git a/Documentation/devicetree/bindings/usb/qpnp-pdphy.txt b/Documentation/devicetree/bindings/usb/qpnp-pdphy.txt
index ab2bbe4..e138cb1 100644
--- a/Documentation/devicetree/bindings/usb/qpnp-pdphy.txt
+++ b/Documentation/devicetree/bindings/usb/qpnp-pdphy.txt
@@ -34,6 +34,10 @@
Triggers when a message was received but had to
be discarded due to the RX buffer still in use
by SW.
+- goog,port-type: Should be set to one of following bindings:
+ <TYPEC_PORT_DFP>
+ <TYPEC_PORT_UFP>
+ <TYPEC_PORT_DRP>
Optional properties:
- vbus-supply: Regulator that enables VBUS source output
@@ -46,6 +50,58 @@
- qcom,default-sink-caps: List of 32-bit values representing the nominal sink
capabilities in voltage (millivolts) and current
(milliamps) pairs.
+- goog,src-pdo: Required when goog,port-type is set to <TYPEC_PORT_DFP>
+ or <TYPEC_PORT_DRP>.
+ A set of 32-bit unsigned integer cells represents the
+ Source Capabilities of the board. The number of the
+ cells should be multiple of 4, at least 4 cells, and at
+ most 28 cells. Every group of 4-tuple-cell represents
+ one Capability.
+
+ Format of each 4-tuple:
+ < pdo_type power_attr1 power_attr2 power_attr3 >
+ where pdo_type should be set to one of the following:
+ 1. PDO_TYPE_FIXED
+ 2. PDO_TYPE_BATT
+ 3. PDO_TYPE_VAR
+ And power_attrX varies from pdo_type
+
+ When pdo_type is PDO_TYPE_FIXED:
+ power_attr1 is Voltage (mv)
+ power_attr2 is Maximum Current (ma)
+ power_attr3 should be set to 0
+
+ When pdo_type is PDO_TYPE_BATT:
+ power_attr1 is Minimum Voltage (mv)
+ power_attr2 is Maximum Voltage (mv)
+ power_attr3 is Maximum Allowable Power (mw)
+
+ When pdo_type is PDO_TYPE_VAR:
+ power_attr1 is Minimum Voltage (mv)
+ power_attr2 is Maximum Voltage (mv)
+ power_attr3 is Maximum Current (ma)
+
+- goog,snk-pdo: Required when goog,port-type is set to <TYPEC_PORT_UFP>
+ or <TYPEC_PORT_DRP>.
+ It represents the Sink Capabilities of the board with
+ the same format as that of goog,src-pdo
+- goog,max-snk-mv: Maximum acceptable sinking voltage when selecting PDO in
+ power negotiation
+- goog,max-snk-ma: Maximum acceptable sinking current when selecting PDO in
+ power negotiation
+- goog,max-snk-mw: Maximum acceptable power limit when specifying available
+ current in power negotiation
+- goog,op-snk-mw: Required when goog,port-type is set to <TYPEC_PORT_UFP>
+ or <TYPEC_PORT_DRP>.
+ Minimum operating power
+ Capability Mismatch flag will be set if this value is
+ bigger than the calculated power in selected PDO.
+- goog,default-role: Specifying Type-C try role
+ This value should be set to one of the following:
+ <TYPEC_SINK>
+ <TYPEC_SOURCE>
+- goog,try-role-hw: Set this property if try.{src,snk} is implemented in
+ hardware.
Example:
qcom,qpnp-pdphy@1700 {
@@ -71,4 +127,15 @@
qcom,default-sink-caps = <5000 3000>, /* 5V @ 3A */
<9000 3000>, /* 9V @ 3A */
<12000 2250>; /* 12V @ 2.25A */
+
+ goog,src-pdo = <PDO_TYPE_FIXED 5000 900 0>; /* 5V @ 0.9A */
+ goog,snk-pdo = <PDO_TYPE_FIXED 5000 3000 0>, /* 5V @ 3A */
+ <PDO_TYPE_FIXED 9000 3000 0>; /* 9V @ 3A */
+ goog,max-snk-mv = <9000>;
+ goog,max-snk-ma = <3000>;
+ goog,max-snk-mw = <27000>;
+ goog,op-snk-mw = <7600>;
+ goog,port-type = <TYPEC_PORT_DRP>;
+ goog,default-role = <TYPEC_SINK>;
+ goog,try-role-hw;
};
diff --git a/Documentation/devicetree/bindings/vendor-prefixes.txt b/Documentation/devicetree/bindings/vendor-prefixes.txt
index e1be5fd..cf1b3d7 100644
--- a/Documentation/devicetree/bindings/vendor-prefixes.txt
+++ b/Documentation/devicetree/bindings/vendor-prefixes.txt
@@ -265,6 +265,7 @@
spansion Spansion Inc.
sprd Spreadtrum Communications Inc.
st STMicroelectronics
+stm STMicroelectronics
starry Starry Electronic Technology (ShenZhen) Co., LTD
startek Startek
ste ST-Ericsson
diff --git a/Documentation/dma-buf-sharing.txt b/Documentation/dma-buf-sharing.txt
index ca44c58..249007e 100644
--- a/Documentation/dma-buf-sharing.txt
+++ b/Documentation/dma-buf-sharing.txt
@@ -439,6 +439,25 @@
cases. Userspace can use this to detect support for discovering the dma-buf
size using llseek.
+Dma-buf memory usage tracking
+----------------------------
+- To provide methods for userspace to track the dma-buf usage across different
+ processes, dma-buf provides several interfaces to read the allocation detail of
+ dma-bufs and the process that own them. First of all, a mini-filesystem in
+ dma-buf will assign a unique inode to each dma-buf. By Calling stat(2) on
+ each entry, the caller can get a unique ID (st_ino), the buffer's size
+ (st_size) and the number of pages assigned to each dma-buffer. The inode
+ information is also exposed in /sys/kernel/debug/dma_buf/bufinfo so in the case
+ where a buffer is mmap()ed into a process’s address space but all remaining
+ fds have been closed, we can still get the dma-buf information and try to
+ accociate it with the process by searching the proc/pid/maps and looking for
+ the corresponding inode number exposed in dma-buf debug fs. Additionally,
+ dma-buf support assigning names to dma-bufs from userspace. This information
+ can be helpful for tracking and accounting shared buffers based on their usage
+ and original purpose. Last but not least, the show_fdinfo() handler in
+ dma_buf_file_operations can print the file_count and name of each buffer in
+ /proc/pid/fdinfo/fd.
+
Miscellaneous notes
-------------------
diff --git a/Documentation/filesystems/f2fs.txt b/Documentation/filesystems/f2fs.txt
index 193a034..b640a37 100644
--- a/Documentation/filesystems/f2fs.txt
+++ b/Documentation/filesystems/f2fs.txt
@@ -154,6 +154,27 @@
enabled by default.
data_flush Enable data flushing before checkpoint in order to
persist data of regular and symlink.
+fault_injection=%d Enable fault injection in all supported types with
+ specified injection rate.
+fault_type=%d Support configuring fault injection type, should be
+ enabled with fault_injection option, fault type value
+ is shown below, it supports single or combined type.
+ Type_Name Type_Value
+ FAULT_KMALLOC 0x000000001
+ FAULT_KVMALLOC 0x000000002
+ FAULT_PAGE_ALLOC 0x000000004
+ FAULT_PAGE_GET 0x000000008
+ FAULT_ALLOC_BIO 0x000000010
+ FAULT_ALLOC_NID 0x000000020
+ FAULT_ORPHAN 0x000000040
+ FAULT_BLOCK 0x000000080
+ FAULT_DIR_DEPTH 0x000000100
+ FAULT_EVICT_INODE 0x000000200
+ FAULT_TRUNCATE 0x000000400
+ FAULT_READ_IO 0x000000800
+ FAULT_CHECKPOINT 0x000001000
+ FAULT_DISCARD 0x000002000
+ FAULT_WRITE_IO 0x000004000
mode=%s Control block allocation mode which supports "adaptive"
and "lfs". In "lfs" mode, there should be no random
writes towards main area.
@@ -190,6 +211,11 @@
non-atomic files likewise "nobarrier" mount option.
test_dummy_encryption Enable dummy encryption, which provides a fake fscrypt
context. The fake fscrypt context is used by xfstests.
+checkpoint=%s Set to "disable" to turn off checkpointing. Set to "enable"
+ to reenable checkpointing. Is enabled by default. While
+ disabled, any unmounting or unexpected shutdowns will cause
+ the filesystem contents to appear as they did when the
+ filesystem was mounted with that option.
================================================================================
DEBUGFS ENTRIES
diff --git a/Documentation/filesystems/fscrypt.rst b/Documentation/filesystems/fscrypt.rst
new file mode 100644
index 0000000..3a7b6052
--- /dev/null
+++ b/Documentation/filesystems/fscrypt.rst
@@ -0,0 +1,641 @@
+=====================================
+Filesystem-level encryption (fscrypt)
+=====================================
+
+Introduction
+============
+
+fscrypt is a library which filesystems can hook into to support
+transparent encryption of files and directories.
+
+Note: "fscrypt" in this document refers to the kernel-level portion,
+implemented in ``fs/crypto/``, as opposed to the userspace tool
+`fscrypt <https://github.com/google/fscrypt>`_. This document only
+covers the kernel-level portion. For command-line examples of how to
+use encryption, see the documentation for the userspace tool `fscrypt
+<https://github.com/google/fscrypt>`_. Also, it is recommended to use
+the fscrypt userspace tool, or other existing userspace tools such as
+`fscryptctl <https://github.com/google/fscryptctl>`_ or `Android's key
+management system
+<https://source.android.com/security/encryption/file-based>`_, over
+using the kernel's API directly. Using existing tools reduces the
+chance of introducing your own security bugs. (Nevertheless, for
+completeness this documentation covers the kernel's API anyway.)
+
+Unlike dm-crypt, fscrypt operates at the filesystem level rather than
+at the block device level. This allows it to encrypt different files
+with different keys and to have unencrypted files on the same
+filesystem. This is useful for multi-user systems where each user's
+data-at-rest needs to be cryptographically isolated from the others.
+However, except for filenames, fscrypt does not encrypt filesystem
+metadata.
+
+Unlike eCryptfs, which is a stacked filesystem, fscrypt is integrated
+directly into supported filesystems --- currently ext4, F2FS, and
+UBIFS. This allows encrypted files to be read and written without
+caching both the decrypted and encrypted pages in the pagecache,
+thereby nearly halving the memory used and bringing it in line with
+unencrypted files. Similarly, half as many dentries and inodes are
+needed. eCryptfs also limits encrypted filenames to 143 bytes,
+causing application compatibility issues; fscrypt allows the full 255
+bytes (NAME_MAX). Finally, unlike eCryptfs, the fscrypt API can be
+used by unprivileged users, with no need to mount anything.
+
+fscrypt does not support encrypting files in-place. Instead, it
+supports marking an empty directory as encrypted. Then, after
+userspace provides the key, all regular files, directories, and
+symbolic links created in that directory tree are transparently
+encrypted.
+
+Threat model
+============
+
+Offline attacks
+---------------
+
+Provided that userspace chooses a strong encryption key, fscrypt
+protects the confidentiality of file contents and filenames in the
+event of a single point-in-time permanent offline compromise of the
+block device content. fscrypt does not protect the confidentiality of
+non-filename metadata, e.g. file sizes, file permissions, file
+timestamps, and extended attributes. Also, the existence and location
+of holes (unallocated blocks which logically contain all zeroes) in
+files is not protected.
+
+fscrypt is not guaranteed to protect confidentiality or authenticity
+if an attacker is able to manipulate the filesystem offline prior to
+an authorized user later accessing the filesystem.
+
+Online attacks
+--------------
+
+fscrypt (and storage encryption in general) can only provide limited
+protection, if any at all, against online attacks. In detail:
+
+fscrypt is only resistant to side-channel attacks, such as timing or
+electromagnetic attacks, to the extent that the underlying Linux
+Cryptographic API algorithms are. If a vulnerable algorithm is used,
+such as a table-based implementation of AES, it may be possible for an
+attacker to mount a side channel attack against the online system.
+Side channel attacks may also be mounted against applications
+consuming decrypted data.
+
+After an encryption key has been provided, fscrypt is not designed to
+hide the plaintext file contents or filenames from other users on the
+same system, regardless of the visibility of the keyring key.
+Instead, existing access control mechanisms such as file mode bits,
+POSIX ACLs, LSMs, or mount namespaces should be used for this purpose.
+Also note that as long as the encryption keys are *anywhere* in
+memory, an online attacker can necessarily compromise them by mounting
+a physical attack or by exploiting any kernel security vulnerability
+which provides an arbitrary memory read primitive.
+
+While it is ostensibly possible to "evict" keys from the system,
+recently accessed encrypted files will remain accessible at least
+until the filesystem is unmounted or the VFS caches are dropped, e.g.
+using ``echo 2 > /proc/sys/vm/drop_caches``. Even after that, if the
+RAM is compromised before being powered off, it will likely still be
+possible to recover portions of the plaintext file contents, if not
+some of the encryption keys as well. (Since Linux v4.12, all
+in-kernel keys related to fscrypt are sanitized before being freed.
+However, userspace would need to do its part as well.)
+
+Currently, fscrypt does not prevent a user from maliciously providing
+an incorrect key for another user's existing encrypted files. A
+protection against this is planned.
+
+Key hierarchy
+=============
+
+Master Keys
+-----------
+
+Each encrypted directory tree is protected by a *master key*. Master
+keys can be up to 64 bytes long, and must be at least as long as the
+greater of the key length needed by the contents and filenames
+encryption modes being used. For example, if AES-256-XTS is used for
+contents encryption, the master key must be 64 bytes (512 bits). Note
+that the XTS mode is defined to require a key twice as long as that
+required by the underlying block cipher.
+
+To "unlock" an encrypted directory tree, userspace must provide the
+appropriate master key. There can be any number of master keys, each
+of which protects any number of directory trees on any number of
+filesystems.
+
+Userspace should generate master keys either using a cryptographically
+secure random number generator, or by using a KDF (Key Derivation
+Function). Note that whenever a KDF is used to "stretch" a
+lower-entropy secret such as a passphrase, it is critical that a KDF
+designed for this purpose be used, such as scrypt, PBKDF2, or Argon2.
+
+Per-file keys
+-------------
+
+Since each master key can protect many files, it is necessary to
+"tweak" the encryption of each file so that the same plaintext in two
+files doesn't map to the same ciphertext, or vice versa. In most
+cases, fscrypt does this by deriving per-file keys. When a new
+encrypted inode (regular file, directory, or symlink) is created,
+fscrypt randomly generates a 16-byte nonce and stores it in the
+inode's encryption xattr. Then, it uses a KDF (Key Derivation
+Function) to derive the file's key from the master key and nonce.
+
+The Adiantum encryption mode (see `Encryption modes and usage`_) is
+special, since it accepts longer IVs and is suitable for both contents
+and filenames encryption. For it, a "direct key" option is offered
+where the file's nonce is included in the IVs and the master key is
+used for encryption directly. This improves performance; however,
+users must not use the same master key for any other encryption mode.
+
+Below, the KDF and design considerations are described in more detail.
+
+The current KDF works by encrypting the master key with AES-128-ECB,
+using the file's nonce as the AES key. The output is used as the
+derived key. If the output is longer than needed, then it is
+truncated to the needed length.
+
+Note: this KDF meets the primary security requirement, which is to
+produce unique derived keys that preserve the entropy of the master
+key, assuming that the master key is already a good pseudorandom key.
+However, it is nonstandard and has some problems such as being
+reversible, so it is generally considered to be a mistake! It may be
+replaced with HKDF or another more standard KDF in the future.
+
+Key derivation was chosen over key wrapping because wrapped keys would
+require larger xattrs which would be less likely to fit in-line in the
+filesystem's inode table, and there didn't appear to be any
+significant advantages to key wrapping. In particular, currently
+there is no requirement to support unlocking a file with multiple
+alternative master keys or to support rotating master keys. Instead,
+the master keys may be wrapped in userspace, e.g. as is done by the
+`fscrypt <https://github.com/google/fscrypt>`_ tool.
+
+Including the inode number in the IVs was considered. However, it was
+rejected as it would have prevented ext4 filesystems from being
+resized, and by itself still wouldn't have been sufficient to prevent
+the same key from being directly reused for both XTS and CTS-CBC.
+
+Encryption modes and usage
+==========================
+
+fscrypt allows one encryption mode to be specified for file contents
+and one encryption mode to be specified for filenames. Different
+directory trees are permitted to use different encryption modes.
+Currently, the following pairs of encryption modes are supported:
+
+- AES-256-XTS for contents and AES-256-CTS-CBC for filenames
+- AES-128-CBC for contents and AES-128-CTS-CBC for filenames
+- Adiantum for both contents and filenames
+
+If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.
+
+AES-128-CBC was added only for low-powered embedded devices with
+crypto accelerators such as CAAM or CESA that do not support XTS.
+
+Adiantum is a (primarily) stream cipher-based mode that is fast even
+on CPUs without dedicated crypto instructions. It's also a true
+wide-block mode, unlike XTS. It can also eliminate the need to derive
+per-file keys. However, it depends on the security of two primitives,
+XChaCha12 and AES-256, rather than just one. See the paper
+"Adiantum: length-preserving encryption for entry-level processors"
+(https://eprint.iacr.org/2018/720.pdf) for more details. To use
+Adiantum, CONFIG_CRYPTO_ADIANTUM must be enabled. Also, fast
+implementations of ChaCha and NHPoly1305 should be enabled, e.g.
+CONFIG_CRYPTO_CHACHA20_NEON and CONFIG_CRYPTO_NHPOLY1305_NEON for ARM.
+
+New encryption modes can be added relatively easily, without changes
+to individual filesystems. However, authenticated encryption (AE)
+modes are not currently supported because of the difficulty of dealing
+with ciphertext expansion.
+
+Contents encryption
+-------------------
+
+For file contents, each filesystem block is encrypted independently.
+Currently, only the case where the filesystem block size is equal to
+the system's page size (usually 4096 bytes) is supported.
+
+Each block's IV is set to the logical block number within the file as
+a little endian number, except that:
+
+- With CBC mode encryption, ESSIV is also used. Specifically, each IV
+ is encrypted with AES-256 where the AES-256 key is the SHA-256 hash
+ of the file's data encryption key.
+
+- In the "direct key" configuration (FS_POLICY_FLAG_DIRECT_KEY set in
+ the fscrypt_policy), the file's nonce is also appended to the IV.
+ Currently this is only allowed with the Adiantum encryption mode.
+
+Filenames encryption
+--------------------
+
+For filenames, each full filename is encrypted at once. Because of
+the requirements to retain support for efficient directory lookups and
+filenames of up to 255 bytes, the same IV is used for every filename
+in a directory.
+
+However, each encrypted directory still uses a unique key; or
+alternatively (for the "direct key" configuration) has the file's
+nonce included in the IVs. Thus, IV reuse is limited to within a
+single directory.
+
+With CTS-CBC, the IV reuse means that when the plaintext filenames
+share a common prefix at least as long as the cipher block size (16
+bytes for AES), the corresponding encrypted filenames will also share
+a common prefix. This is undesirable. Adiantum does not have this
+weakness, as it is a wide-block encryption mode.
+
+All supported filenames encryption modes accept any plaintext length
+>= 16 bytes; cipher block alignment is not required. However,
+filenames shorter than 16 bytes are NUL-padded to 16 bytes before
+being encrypted. In addition, to reduce leakage of filename lengths
+via their ciphertexts, all filenames are NUL-padded to the next 4, 8,
+16, or 32-byte boundary (configurable). 32 is recommended since this
+provides the best confidentiality, at the cost of making directory
+entries consume slightly more space. Note that since NUL (``\0``) is
+not otherwise a valid character in filenames, the padding will never
+produce duplicate plaintexts.
+
+Symbolic link targets are considered a type of filename and are
+encrypted in the same way as filenames in directory entries, except
+that IV reuse is not a problem as each symlink has its own inode.
+
+User API
+========
+
+Setting an encryption policy
+----------------------------
+
+The FS_IOC_SET_ENCRYPTION_POLICY ioctl sets an encryption policy on an
+empty directory or verifies that a directory or regular file already
+has the specified encryption policy. It takes in a pointer to a
+:c:type:`struct fscrypt_policy`, defined as follows::
+
+ #define FS_KEY_DESCRIPTOR_SIZE 8
+
+ struct fscrypt_policy {
+ __u8 version;
+ __u8 contents_encryption_mode;
+ __u8 filenames_encryption_mode;
+ __u8 flags;
+ __u8 master_key_descriptor[FS_KEY_DESCRIPTOR_SIZE];
+ };
+
+This structure must be initialized as follows:
+
+- ``version`` must be 0.
+
+- ``contents_encryption_mode`` and ``filenames_encryption_mode`` must
+ be set to constants from ``<linux/fs.h>`` which identify the
+ encryption modes to use. If unsure, use
+ FS_ENCRYPTION_MODE_AES_256_XTS (1) for ``contents_encryption_mode``
+ and FS_ENCRYPTION_MODE_AES_256_CTS (4) for
+ ``filenames_encryption_mode``.
+
+- ``flags`` must contain a value from ``<linux/fs.h>`` which
+ identifies the amount of NUL-padding to use when encrypting
+ filenames. If unsure, use FS_POLICY_FLAGS_PAD_32 (0x3).
+ In addition, if the chosen encryption modes are both
+ FS_ENCRYPTION_MODE_ADIANTUM, this can contain
+ FS_POLICY_FLAG_DIRECT_KEY to specify that the master key should be
+ used directly, without key derivation.
+
+- ``master_key_descriptor`` specifies how to find the master key in
+ the keyring; see `Adding keys`_. It is up to userspace to choose a
+ unique ``master_key_descriptor`` for each master key. The e4crypt
+ and fscrypt tools use the first 8 bytes of
+ ``SHA-512(SHA-512(master_key))``, but this particular scheme is not
+ required. Also, the master key need not be in the keyring yet when
+ FS_IOC_SET_ENCRYPTION_POLICY is executed. However, it must be added
+ before any files can be created in the encrypted directory.
+
+If the file is not yet encrypted, then FS_IOC_SET_ENCRYPTION_POLICY
+verifies that the file is an empty directory. If so, the specified
+encryption policy is assigned to the directory, turning it into an
+encrypted directory. After that, and after providing the
+corresponding master key as described in `Adding keys`_, all regular
+files, directories (recursively), and symlinks created in the
+directory will be encrypted, inheriting the same encryption policy.
+The filenames in the directory's entries will be encrypted as well.
+
+Alternatively, if the file is already encrypted, then
+FS_IOC_SET_ENCRYPTION_POLICY validates that the specified encryption
+policy exactly matches the actual one. If they match, then the ioctl
+returns 0. Otherwise, it fails with EEXIST. This works on both
+regular files and directories, including nonempty directories.
+
+Note that the ext4 filesystem does not allow the root directory to be
+encrypted, even if it is empty. Users who want to encrypt an entire
+filesystem with one key should consider using dm-crypt instead.
+
+FS_IOC_SET_ENCRYPTION_POLICY can fail with the following errors:
+
+- ``EACCES``: the file is not owned by the process's uid, nor does the
+ process have the CAP_FOWNER capability in a namespace with the file
+ owner's uid mapped
+- ``EEXIST``: the file is already encrypted with an encryption policy
+ different from the one specified
+- ``EINVAL``: an invalid encryption policy was specified (invalid
+ version, mode(s), or flags)
+- ``ENOTDIR``: the file is unencrypted and is a regular file, not a
+ directory
+- ``ENOTEMPTY``: the file is unencrypted and is a nonempty directory
+- ``ENOTTY``: this type of filesystem does not implement encryption
+- ``EOPNOTSUPP``: the kernel was not configured with encryption
+ support for this filesystem, or the filesystem superblock has not
+ had encryption enabled on it. (For example, to use encryption on an
+ ext4 filesystem, CONFIG_EXT4_ENCRYPTION must be enabled in the
+ kernel config, and the superblock must have had the "encrypt"
+ feature flag enabled using ``tune2fs -O encrypt`` or ``mkfs.ext4 -O
+ encrypt``.)
+- ``EPERM``: this directory may not be encrypted, e.g. because it is
+ the root directory of an ext4 filesystem
+- ``EROFS``: the filesystem is readonly
+
+Getting an encryption policy
+----------------------------
+
+The FS_IOC_GET_ENCRYPTION_POLICY ioctl retrieves the :c:type:`struct
+fscrypt_policy`, if any, for a directory or regular file. See above
+for the struct definition. No additional permissions are required
+beyond the ability to open the file.
+
+FS_IOC_GET_ENCRYPTION_POLICY can fail with the following errors:
+
+- ``EINVAL``: the file is encrypted, but it uses an unrecognized
+ encryption context format
+- ``ENODATA``: the file is not encrypted
+- ``ENOTTY``: this type of filesystem does not implement encryption
+- ``EOPNOTSUPP``: the kernel was not configured with encryption
+ support for this filesystem
+
+Note: if you only need to know whether a file is encrypted or not, on
+most filesystems it is also possible to use the FS_IOC_GETFLAGS ioctl
+and check for FS_ENCRYPT_FL, or to use the statx() system call and
+check for STATX_ATTR_ENCRYPTED in stx_attributes.
+
+Getting the per-filesystem salt
+-------------------------------
+
+Some filesystems, such as ext4 and F2FS, also support the deprecated
+ioctl FS_IOC_GET_ENCRYPTION_PWSALT. This ioctl retrieves a randomly
+generated 16-byte value stored in the filesystem superblock. This
+value is intended to used as a salt when deriving an encryption key
+from a passphrase or other low-entropy user credential.
+
+FS_IOC_GET_ENCRYPTION_PWSALT is deprecated. Instead, prefer to
+generate and manage any needed salt(s) in userspace.
+
+Adding keys
+-----------
+
+To provide a master key, userspace must add it to an appropriate
+keyring using the add_key() system call (see:
+``Documentation/security/keys/core.rst``). The key type must be
+"logon"; keys of this type are kept in kernel memory and cannot be
+read back by userspace. The key description must be "fscrypt:"
+followed by the 16-character lower case hex representation of the
+``master_key_descriptor`` that was set in the encryption policy. The
+key payload must conform to the following structure::
+
+ #define FS_MAX_KEY_SIZE 64
+
+ struct fscrypt_key {
+ u32 mode;
+ u8 raw[FS_MAX_KEY_SIZE];
+ u32 size;
+ };
+
+``mode`` is ignored; just set it to 0. The actual key is provided in
+``raw`` with ``size`` indicating its size in bytes. That is, the
+bytes ``raw[0..size-1]`` (inclusive) are the actual key.
+
+The key description prefix "fscrypt:" may alternatively be replaced
+with a filesystem-specific prefix such as "ext4:". However, the
+filesystem-specific prefixes are deprecated and should not be used in
+new programs.
+
+There are several different types of keyrings in which encryption keys
+may be placed, such as a session keyring, a user session keyring, or a
+user keyring. Each key must be placed in a keyring that is "attached"
+to all processes that might need to access files encrypted with it, in
+the sense that request_key() will find the key. Generally, if only
+processes belonging to a specific user need to access a given
+encrypted directory and no session keyring has been installed, then
+that directory's key should be placed in that user's user session
+keyring or user keyring. Otherwise, a session keyring should be
+installed if needed, and the key should be linked into that session
+keyring, or in a keyring linked into that session keyring.
+
+Note: introducing the complex visibility semantics of keyrings here
+was arguably a mistake --- especially given that by design, after any
+process successfully opens an encrypted file (thereby setting up the
+per-file key), possessing the keyring key is not actually required for
+any process to read/write the file until its in-memory inode is
+evicted. In the future there probably should be a way to provide keys
+directly to the filesystem instead, which would make the intended
+semantics clearer.
+
+Access semantics
+================
+
+With the key
+------------
+
+With the encryption key, encrypted regular files, directories, and
+symlinks behave very similarly to their unencrypted counterparts ---
+after all, the encryption is intended to be transparent. However,
+astute users may notice some differences in behavior:
+
+- Unencrypted files, or files encrypted with a different encryption
+ policy (i.e. different key, modes, or flags), cannot be renamed or
+ linked into an encrypted directory; see `Encryption policy
+ enforcement`_. Attempts to do so will fail with EPERM. However,
+ encrypted files can be renamed within an encrypted directory, or
+ into an unencrypted directory.
+
+- Direct I/O is not supported on encrypted files. Attempts to use
+ direct I/O on such files will fall back to buffered I/O.
+
+- The fallocate operations FALLOC_FL_COLLAPSE_RANGE,
+ FALLOC_FL_INSERT_RANGE, and FALLOC_FL_ZERO_RANGE are not supported
+ on encrypted files and will fail with EOPNOTSUPP.
+
+- Online defragmentation of encrypted files is not supported. The
+ EXT4_IOC_MOVE_EXT and F2FS_IOC_MOVE_RANGE ioctls will fail with
+ EOPNOTSUPP.
+
+- The ext4 filesystem does not support data journaling with encrypted
+ regular files. It will fall back to ordered data mode instead.
+
+- DAX (Direct Access) is not supported on encrypted files.
+
+- The st_size of an encrypted symlink will not necessarily give the
+ length of the symlink target as required by POSIX. It will actually
+ give the length of the ciphertext, which will be slightly longer
+ than the plaintext due to NUL-padding and an extra 2-byte overhead.
+
+- The maximum length of an encrypted symlink is 2 bytes shorter than
+ the maximum length of an unencrypted symlink. For example, on an
+ EXT4 filesystem with a 4K block size, unencrypted symlinks can be up
+ to 4095 bytes long, while encrypted symlinks can only be up to 4093
+ bytes long (both lengths excluding the terminating null).
+
+Note that mmap *is* supported. This is possible because the pagecache
+for an encrypted file contains the plaintext, not the ciphertext.
+
+Without the key
+---------------
+
+Some filesystem operations may be performed on encrypted regular
+files, directories, and symlinks even before their encryption key has
+been provided:
+
+- File metadata may be read, e.g. using stat().
+
+- Directories may be listed, in which case the filenames will be
+ listed in an encoded form derived from their ciphertext. The
+ current encoding algorithm is described in `Filename hashing and
+ encoding`_. The algorithm is subject to change, but it is
+ guaranteed that the presented filenames will be no longer than
+ NAME_MAX bytes, will not contain the ``/`` or ``\0`` characters, and
+ will uniquely identify directory entries.
+
+ The ``.`` and ``..`` directory entries are special. They are always
+ present and are not encrypted or encoded.
+
+- Files may be deleted. That is, nondirectory files may be deleted
+ with unlink() as usual, and empty directories may be deleted with
+ rmdir() as usual. Therefore, ``rm`` and ``rm -r`` will work as
+ expected.
+
+- Symlink targets may be read and followed, but they will be presented
+ in encrypted form, similar to filenames in directories. Hence, they
+ are unlikely to point to anywhere useful.
+
+Without the key, regular files cannot be opened or truncated.
+Attempts to do so will fail with ENOKEY. This implies that any
+regular file operations that require a file descriptor, such as
+read(), write(), mmap(), fallocate(), and ioctl(), are also forbidden.
+
+Also without the key, files of any type (including directories) cannot
+be created or linked into an encrypted directory, nor can a name in an
+encrypted directory be the source or target of a rename, nor can an
+O_TMPFILE temporary file be created in an encrypted directory. All
+such operations will fail with ENOKEY.
+
+It is not currently possible to backup and restore encrypted files
+without the encryption key. This would require special APIs which
+have not yet been implemented.
+
+Encryption policy enforcement
+=============================
+
+After an encryption policy has been set on a directory, all regular
+files, directories, and symbolic links created in that directory
+(recursively) will inherit that encryption policy. Special files ---
+that is, named pipes, device nodes, and UNIX domain sockets --- will
+not be encrypted.
+
+Except for those special files, it is forbidden to have unencrypted
+files, or files encrypted with a different encryption policy, in an
+encrypted directory tree. Attempts to link or rename such a file into
+an encrypted directory will fail with EPERM. This is also enforced
+during ->lookup() to provide limited protection against offline
+attacks that try to disable or downgrade encryption in known locations
+where applications may later write sensitive data. It is recommended
+that systems implementing a form of "verified boot" take advantage of
+this by validating all top-level encryption policies prior to access.
+
+Implementation details
+======================
+
+Encryption context
+------------------
+
+An encryption policy is represented on-disk by a :c:type:`struct
+fscrypt_context`. It is up to individual filesystems to decide where
+to store it, but normally it would be stored in a hidden extended
+attribute. It should *not* be exposed by the xattr-related system
+calls such as getxattr() and setxattr() because of the special
+semantics of the encryption xattr. (In particular, there would be
+much confusion if an encryption policy were to be added to or removed
+from anything other than an empty directory.) The struct is defined
+as follows::
+
+ #define FS_KEY_DESCRIPTOR_SIZE 8
+ #define FS_KEY_DERIVATION_NONCE_SIZE 16
+
+ struct fscrypt_context {
+ u8 format;
+ u8 contents_encryption_mode;
+ u8 filenames_encryption_mode;
+ u8 flags;
+ u8 master_key_descriptor[FS_KEY_DESCRIPTOR_SIZE];
+ u8 nonce[FS_KEY_DERIVATION_NONCE_SIZE];
+ };
+
+Note that :c:type:`struct fscrypt_context` contains the same
+information as :c:type:`struct fscrypt_policy` (see `Setting an
+encryption policy`_), except that :c:type:`struct fscrypt_context`
+also contains a nonce. The nonce is randomly generated by the kernel
+and is used to derive the inode's encryption key as described in
+`Per-file keys`_.
+
+Data path changes
+-----------------
+
+For the read path (->readpage()) of regular files, filesystems can
+read the ciphertext into the page cache and decrypt it in-place. The
+page lock must be held until decryption has finished, to prevent the
+page from becoming visible to userspace prematurely.
+
+For the write path (->writepage()) of regular files, filesystems
+cannot encrypt data in-place in the page cache, since the cached
+plaintext must be preserved. Instead, filesystems must encrypt into a
+temporary buffer or "bounce page", then write out the temporary
+buffer. Some filesystems, such as UBIFS, already use temporary
+buffers regardless of encryption. Other filesystems, such as ext4 and
+F2FS, have to allocate bounce pages specially for encryption.
+
+Filename hashing and encoding
+-----------------------------
+
+Modern filesystems accelerate directory lookups by using indexed
+directories. An indexed directory is organized as a tree keyed by
+filename hashes. When a ->lookup() is requested, the filesystem
+normally hashes the filename being looked up so that it can quickly
+find the corresponding directory entry, if any.
+
+With encryption, lookups must be supported and efficient both with and
+without the encryption key. Clearly, it would not work to hash the
+plaintext filenames, since the plaintext filenames are unavailable
+without the key. (Hashing the plaintext filenames would also make it
+impossible for the filesystem's fsck tool to optimize encrypted
+directories.) Instead, filesystems hash the ciphertext filenames,
+i.e. the bytes actually stored on-disk in the directory entries. When
+asked to do a ->lookup() with the key, the filesystem just encrypts
+the user-supplied name to get the ciphertext.
+
+Lookups without the key are more complicated. The raw ciphertext may
+contain the ``\0`` and ``/`` characters, which are illegal in
+filenames. Therefore, readdir() must base64-encode the ciphertext for
+presentation. For most filenames, this works fine; on ->lookup(), the
+filesystem just base64-decodes the user-supplied name to get back to
+the raw ciphertext.
+
+However, for very long filenames, base64 encoding would cause the
+filename length to exceed NAME_MAX. To prevent this, readdir()
+actually presents long filenames in an abbreviated form which encodes
+a strong "hash" of the ciphertext filename, along with the optional
+filesystem-specific hash(es) needed for directory lookups. This
+allows the filesystem to still, with a high degree of confidence, map
+the filename given in ->lookup() back to a particular directory entry
+that was previously listed by readdir(). See :c:type:`struct
+fscrypt_digested_name` in the source for more details.
+
+Note that the precise way that filenames are presented to userspace
+without the key is subject to change in the future. It is only meant
+as a way to temporarily present valid filenames so that commands like
+``rm -r`` work as expected on encrypted directories.
diff --git a/Documentation/filesystems/overlayfs.txt b/Documentation/filesystems/overlayfs.txt
index bcbf971..ddd7a48 100644
--- a/Documentation/filesystems/overlayfs.txt
+++ b/Documentation/filesystems/overlayfs.txt
@@ -82,6 +82,23 @@
such as metadata and extended attributes are reported for the upper
directory only. These attributes of the lower directory are hidden.
+credentials
+-----------
+
+By default, all access to the upper, lower and work directories is the
+recorded mounter's MAC and DAC credentials. The incoming accesses are
+checked against the caller's credentials.
+
+If the principles of least privilege are applied, the mounter's
+credentials might not overlap the credentials of the caller's when
+accessing the overlayfs filesystem. For example, a file that a lower
+DAC privileged caller can execute, is MAC denied to the generally
+higher DAC privileged mounter, to prevent an attack vector. One
+option is to turn off override_creds in the mount options; all
+subsequent operations after mount on the filesystem will be only the
+caller's credentials. This option default is set in the CONFIG
+OVERLAY_FS_OVERRIDE_CREDS or in the module option override_creds.
+
whiteouts and opaque directories
--------------------------------
diff --git a/Documentation/hwmon/ina2xx b/Documentation/hwmon/ina2xx
index cfd31d9..f8bf140 100644
--- a/Documentation/hwmon/ina2xx
+++ b/Documentation/hwmon/ina2xx
@@ -32,7 +32,7 @@
Datasheet: Publicly available at the Texas Instruments website
http://www.ti.com/
-Author: Lothar Felten <l-felten@ti.com>
+Author: Lothar Felten <lothar.felten@gmail.com>
Description
-----------
diff --git a/Documentation/index.rst b/Documentation/index.rst
index c53d089..213399a 100644
--- a/Documentation/index.rst
+++ b/Documentation/index.rst
@@ -12,6 +12,7 @@
:maxdepth: 2
kernel-documentation
+ l1tf
development-process/index
dev-tools/tools
driver-api/index
diff --git a/Documentation/input/event-codes.txt b/Documentation/input/event-codes.txt
index 36ea940..575415f 100644
--- a/Documentation/input/event-codes.txt
+++ b/Documentation/input/event-codes.txt
@@ -301,7 +301,10 @@
INPUT_PROP_ACCELEROMETER
-------------------------
Directional axes on this device (absolute and/or relative x, y, z) represent
-accelerometer data. All other axes retain their meaning. A device must not mix
+accelerometer data. Some devices also report gyroscope data, which devices
+can report through the rotational axes (absolute and/or relative rx, ry, rz).
+
+All other axes retain their meaning. A device must not mix
regular directional axes and accelerometer axes on the same event node.
Guidelines:
diff --git a/Documentation/kernel-parameters.txt b/Documentation/kernel-parameters.txt
index 0b8f21f..86addee 100644
--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -314,7 +314,6 @@
This facility can be used to prevent such uncontrolled
GPE floodings.
Format: <int>
- Support masking of GPEs numbered from 0x00 to 0x7f.
acpi_no_auto_serialize [HW,ACPI]
Disable auto-serialization of AML methods
@@ -1100,12 +1099,6 @@
nopku [X86] Disable Memory Protection Keys CPU feature found
in some Intel CPUs.
- eagerfpu= [X86]
- on enable eager fpu restore
- off disable eager fpu restore
- auto selects the default scheme, which automatically
- enables eagerfpu restore for xsaveopt.
-
module.async_probe [KNL]
Enable asynchronous probe on this module.
@@ -2032,10 +2025,84 @@
(virtualized real and unpaged mode) on capable
Intel chips. Default is 1 (enabled)
+ kvm-intel.vmentry_l1d_flush=[KVM,Intel] Mitigation for L1 Terminal Fault
+ CVE-2018-3620.
+
+ Valid arguments: never, cond, always
+
+ always: L1D cache flush on every VMENTER.
+ cond: Flush L1D on VMENTER only when the code between
+ VMEXIT and VMENTER can leak host memory.
+ never: Disables the mitigation
+
+ Default is cond (do L1 cache flush in specific instances)
+
kvm-intel.vpid= [KVM,Intel] Disable Virtual Processor Identification
feature (tagged TLBs) on capable Intel chips.
Default is 1 (enabled)
+ l1tf= [X86] Control mitigation of the L1TF vulnerability on
+ affected CPUs
+
+ The kernel PTE inversion protection is unconditionally
+ enabled and cannot be disabled.
+
+ full
+ Provides all available mitigations for the
+ L1TF vulnerability. Disables SMT and
+ enables all mitigations in the
+ hypervisors, i.e. unconditional L1D flush.
+
+ SMT control and L1D flush control via the
+ sysfs interface is still possible after
+ boot. Hypervisors will issue a warning
+ when the first VM is started in a
+ potentially insecure configuration,
+ i.e. SMT enabled or L1D flush disabled.
+
+ full,force
+ Same as 'full', but disables SMT and L1D
+ flush runtime control. Implies the
+ 'nosmt=force' command line option.
+ (i.e. sysfs control of SMT is disabled.)
+
+ flush
+ Leaves SMT enabled and enables the default
+ hypervisor mitigation, i.e. conditional
+ L1D flush.
+
+ SMT control and L1D flush control via the
+ sysfs interface is still possible after
+ boot. Hypervisors will issue a warning
+ when the first VM is started in a
+ potentially insecure configuration,
+ i.e. SMT enabled or L1D flush disabled.
+
+ flush,nosmt
+
+ Disables SMT and enables the default
+ hypervisor mitigation.
+
+ SMT control and L1D flush control via the
+ sysfs interface is still possible after
+ boot. Hypervisors will issue a warning
+ when the first VM is started in a
+ potentially insecure configuration,
+ i.e. SMT enabled or L1D flush disabled.
+
+ flush,nowarn
+ Same as 'flush', but hypervisors will not
+ warn when a VM is started in a potentially
+ insecure configuration.
+
+ off
+ Disables hypervisor mitigations and doesn't
+ emit any warnings.
+
+ Default is 'flush'.
+
+ For details see: Documentation/admin-guide/l1tf.rst
+
l2cr= [PPC]
l3cr= [PPC]
@@ -2361,6 +2428,12 @@
memory contents and reserves bad memory
regions that are detected.
+ mem_sleep_default= [SUSPEND] Default system suspend mode:
+ s2idle - Suspend-To-Idle
+ shallow - Power-On Suspend or equivalent (if supported)
+ deep - Suspend-To-RAM or equivalent (if supported)
+ See Documentation/power/states.txt.
+
meye.*= [HW] Set MotionEye Camera parameters
See Documentation/video4linux/meye.txt.
@@ -2662,6 +2735,9 @@
noalign [KNL,ARM]
+ noaltinstr [S390] Disables alternative instructions patching
+ (CPU alternatives feature).
+
noapic [SMP,APIC] Tells the kernel to not make use of any
IOAPICs that may be present in the system.
@@ -2713,11 +2789,18 @@
nosmt [KNL,S390] Disable symmetric multithreading (SMT).
Equivalent to smt=1.
+ [KNL,x86] Disable symmetric multithreading (SMT).
+ nosmt=force: Force disable SMT, cannot be undone
+ via the sysfs control file.
+
nospectre_v2 [X86] Disable all mitigations for the Spectre variant 2
(indirect branch prediction) vulnerability. System may
allow data leaks with this option, which is equivalent
to spectre_v2=off.
+ nospec_store_bypass_disable
+ [HW] Disable all mitigations for the Speculative Store Bypass vulnerability
+
noxsave [BUGS=X86] Disables x86 extended register state save
and restore using xsave. The kernel will fallback to
enabling legacy floating-point and sse state.
@@ -3724,13 +3807,6 @@
[KNL, SMP] Set scheduler's default relax_domain_level.
See Documentation/cgroup-v1/cpusets.txt.
- relative_sleep_states=
- [SUSPEND] Use sleep state labeling where the deepest
- state available other than hibernation is always "mem".
- Format: { "0" | "1" }
- 0 -- Traditional sleep state labels.
- 1 -- Relative sleep state labels.
-
reserve= [KNL,BUGS] Force the kernel to ignore some iomem area
reservetop= [X86-32]
@@ -3992,11 +4068,70 @@
Not specifying this option is equivalent to
spectre_v2=auto.
+ spec_store_bypass_disable=
+ [HW] Control Speculative Store Bypass (SSB) Disable mitigation
+ (Speculative Store Bypass vulnerability)
+
+ Certain CPUs are vulnerable to an exploit against a
+ a common industry wide performance optimization known
+ as "Speculative Store Bypass" in which recent stores
+ to the same memory location may not be observed by
+ later loads during speculative execution. The idea
+ is that such stores are unlikely and that they can
+ be detected prior to instruction retirement at the
+ end of a particular speculation execution window.
+
+ In vulnerable processors, the speculatively forwarded
+ store can be used in a cache side channel attack, for
+ example to read memory to which the attacker does not
+ directly have access (e.g. inside sandboxed code).
+
+ This parameter controls whether the Speculative Store
+ Bypass optimization is used.
+
+ on - Unconditionally disable Speculative Store Bypass
+ off - Unconditionally enable Speculative Store Bypass
+ auto - Kernel detects whether the CPU model contains an
+ implementation of Speculative Store Bypass and
+ picks the most appropriate mitigation. If the
+ CPU is not vulnerable, "off" is selected. If the
+ CPU is vulnerable the default mitigation is
+ architecture and Kconfig dependent. See below.
+ prctl - Control Speculative Store Bypass per thread
+ via prctl. Speculative Store Bypass is enabled
+ for a process by default. The state of the control
+ is inherited on fork.
+ seccomp - Same as "prctl" above, but all seccomp threads
+ will disable SSB unless they explicitly opt out.
+
+ Not specifying this option is equivalent to
+ spec_store_bypass_disable=auto.
+
+ Default mitigations:
+ X86: If CONFIG_SECCOMP=y "seccomp", otherwise "prctl"
+
spia_io_base= [HW,MTD]
spia_fio_base=
spia_pedr=
spia_peddr=
+ ssbd= [ARM64,HW]
+ Speculative Store Bypass Disable control
+
+ On CPUs that are vulnerable to the Speculative
+ Store Bypass vulnerability and offer a
+ firmware based mitigation, this parameter
+ indicates how the mitigation should be used:
+
+ force-on: Unconditionally enable mitigation for
+ for both kernel and userspace
+ force-off: Unconditionally disable mitigation for
+ for both kernel and userspace
+ kernel: Always enable mitigation in the
+ kernel, and offer a prctl interface
+ to allow userspace to register its
+ interest in being mitigated too.
+
stack_guard_gap= [MM]
override the default stack gap protection. The value
is in page units and it defines how many pages prior
diff --git a/Documentation/l1tf.rst b/Documentation/l1tf.rst
new file mode 100644
index 0000000..bae52b84
--- /dev/null
+++ b/Documentation/l1tf.rst
@@ -0,0 +1,610 @@
+L1TF - L1 Terminal Fault
+========================
+
+L1 Terminal Fault is a hardware vulnerability which allows unprivileged
+speculative access to data which is available in the Level 1 Data Cache
+when the page table entry controlling the virtual address, which is used
+for the access, has the Present bit cleared or other reserved bits set.
+
+Affected processors
+-------------------
+
+This vulnerability affects a wide range of Intel processors. The
+vulnerability is not present on:
+
+ - Processors from AMD, Centaur and other non Intel vendors
+
+ - Older processor models, where the CPU family is < 6
+
+ - A range of Intel ATOM processors (Cedarview, Cloverview, Lincroft,
+ Penwell, Pineview, Silvermont, Airmont, Merrifield)
+
+ - The Intel XEON PHI family
+
+ - Intel processors which have the ARCH_CAP_RDCL_NO bit set in the
+ IA32_ARCH_CAPABILITIES MSR. If the bit is set the CPU is not affected
+ by the Meltdown vulnerability either. These CPUs should become
+ available by end of 2018.
+
+Whether a processor is affected or not can be read out from the L1TF
+vulnerability file in sysfs. See :ref:`l1tf_sys_info`.
+
+Related CVEs
+------------
+
+The following CVE entries are related to the L1TF vulnerability:
+
+ ============= ================= ==============================
+ CVE-2018-3615 L1 Terminal Fault SGX related aspects
+ CVE-2018-3620 L1 Terminal Fault OS, SMM related aspects
+ CVE-2018-3646 L1 Terminal Fault Virtualization related aspects
+ ============= ================= ==============================
+
+Problem
+-------
+
+If an instruction accesses a virtual address for which the relevant page
+table entry (PTE) has the Present bit cleared or other reserved bits set,
+then speculative execution ignores the invalid PTE and loads the referenced
+data if it is present in the Level 1 Data Cache, as if the page referenced
+by the address bits in the PTE was still present and accessible.
+
+While this is a purely speculative mechanism and the instruction will raise
+a page fault when it is retired eventually, the pure act of loading the
+data and making it available to other speculative instructions opens up the
+opportunity for side channel attacks to unprivileged malicious code,
+similar to the Meltdown attack.
+
+While Meltdown breaks the user space to kernel space protection, L1TF
+allows to attack any physical memory address in the system and the attack
+works across all protection domains. It allows an attack of SGX and also
+works from inside virtual machines because the speculation bypasses the
+extended page table (EPT) protection mechanism.
+
+
+Attack scenarios
+----------------
+
+1. Malicious user space
+^^^^^^^^^^^^^^^^^^^^^^^
+
+ Operating Systems store arbitrary information in the address bits of a
+ PTE which is marked non present. This allows a malicious user space
+ application to attack the physical memory to which these PTEs resolve.
+ In some cases user-space can maliciously influence the information
+ encoded in the address bits of the PTE, thus making attacks more
+ deterministic and more practical.
+
+ The Linux kernel contains a mitigation for this attack vector, PTE
+ inversion, which is permanently enabled and has no performance
+ impact. The kernel ensures that the address bits of PTEs, which are not
+ marked present, never point to cacheable physical memory space.
+
+ A system with an up to date kernel is protected against attacks from
+ malicious user space applications.
+
+2. Malicious guest in a virtual machine
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+ The fact that L1TF breaks all domain protections allows malicious guest
+ OSes, which can control the PTEs directly, and malicious guest user
+ space applications, which run on an unprotected guest kernel lacking the
+ PTE inversion mitigation for L1TF, to attack physical host memory.
+
+ A special aspect of L1TF in the context of virtualization is symmetric
+ multi threading (SMT). The Intel implementation of SMT is called
+ HyperThreading. The fact that Hyperthreads on the affected processors
+ share the L1 Data Cache (L1D) is important for this. As the flaw allows
+ only to attack data which is present in L1D, a malicious guest running
+ on one Hyperthread can attack the data which is brought into the L1D by
+ the context which runs on the sibling Hyperthread of the same physical
+ core. This context can be host OS, host user space or a different guest.
+
+ If the processor does not support Extended Page Tables, the attack is
+ only possible, when the hypervisor does not sanitize the content of the
+ effective (shadow) page tables.
+
+ While solutions exist to mitigate these attack vectors fully, these
+ mitigations are not enabled by default in the Linux kernel because they
+ can affect performance significantly. The kernel provides several
+ mechanisms which can be utilized to address the problem depending on the
+ deployment scenario. The mitigations, their protection scope and impact
+ are described in the next sections.
+
+ The default mitigations and the rationale for choosing them are explained
+ at the end of this document. See :ref:`default_mitigations`.
+
+.. _l1tf_sys_info:
+
+L1TF system information
+-----------------------
+
+The Linux kernel provides a sysfs interface to enumerate the current L1TF
+status of the system: whether the system is vulnerable, and which
+mitigations are active. The relevant sysfs file is:
+
+/sys/devices/system/cpu/vulnerabilities/l1tf
+
+The possible values in this file are:
+
+ =========================== ===============================
+ 'Not affected' The processor is not vulnerable
+ 'Mitigation: PTE Inversion' The host protection is active
+ =========================== ===============================
+
+If KVM/VMX is enabled and the processor is vulnerable then the following
+information is appended to the 'Mitigation: PTE Inversion' part:
+
+ - SMT status:
+
+ ===================== ================
+ 'VMX: SMT vulnerable' SMT is enabled
+ 'VMX: SMT disabled' SMT is disabled
+ ===================== ================
+
+ - L1D Flush mode:
+
+ ================================ ====================================
+ 'L1D vulnerable' L1D flushing is disabled
+
+ 'L1D conditional cache flushes' L1D flush is conditionally enabled
+
+ 'L1D cache flushes' L1D flush is unconditionally enabled
+ ================================ ====================================
+
+The resulting grade of protection is discussed in the following sections.
+
+
+Host mitigation mechanism
+-------------------------
+
+The kernel is unconditionally protected against L1TF attacks from malicious
+user space running on the host.
+
+
+Guest mitigation mechanisms
+---------------------------
+
+.. _l1d_flush:
+
+1. L1D flush on VMENTER
+^^^^^^^^^^^^^^^^^^^^^^^
+
+ To make sure that a guest cannot attack data which is present in the L1D
+ the hypervisor flushes the L1D before entering the guest.
+
+ Flushing the L1D evicts not only the data which should not be accessed
+ by a potentially malicious guest, it also flushes the guest
+ data. Flushing the L1D has a performance impact as the processor has to
+ bring the flushed guest data back into the L1D. Depending on the
+ frequency of VMEXIT/VMENTER and the type of computations in the guest
+ performance degradation in the range of 1% to 50% has been observed. For
+ scenarios where guest VMEXIT/VMENTER are rare the performance impact is
+ minimal. Virtio and mechanisms like posted interrupts are designed to
+ confine the VMEXITs to a bare minimum, but specific configurations and
+ application scenarios might still suffer from a high VMEXIT rate.
+
+ The kernel provides two L1D flush modes:
+ - conditional ('cond')
+ - unconditional ('always')
+
+ The conditional mode avoids L1D flushing after VMEXITs which execute
+ only audited code paths before the corresponding VMENTER. These code
+ paths have been verified that they cannot expose secrets or other
+ interesting data to an attacker, but they can leak information about the
+ address space layout of the hypervisor.
+
+ Unconditional mode flushes L1D on all VMENTER invocations and provides
+ maximum protection. It has a higher overhead than the conditional
+ mode. The overhead cannot be quantified correctly as it depends on the
+ workload scenario and the resulting number of VMEXITs.
+
+ The general recommendation is to enable L1D flush on VMENTER. The kernel
+ defaults to conditional mode on affected processors.
+
+ **Note**, that L1D flush does not prevent the SMT problem because the
+ sibling thread will also bring back its data into the L1D which makes it
+ attackable again.
+
+ L1D flush can be controlled by the administrator via the kernel command
+ line and sysfs control files. See :ref:`mitigation_control_command_line`
+ and :ref:`mitigation_control_kvm`.
+
+.. _guest_confinement:
+
+2. Guest VCPU confinement to dedicated physical cores
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+ To address the SMT problem, it is possible to make a guest or a group of
+ guests affine to one or more physical cores. The proper mechanism for
+ that is to utilize exclusive cpusets to ensure that no other guest or
+ host tasks can run on these cores.
+
+ If only a single guest or related guests run on sibling SMT threads on
+ the same physical core then they can only attack their own memory and
+ restricted parts of the host memory.
+
+ Host memory is attackable, when one of the sibling SMT threads runs in
+ host OS (hypervisor) context and the other in guest context. The amount
+ of valuable information from the host OS context depends on the context
+ which the host OS executes, i.e. interrupts, soft interrupts and kernel
+ threads. The amount of valuable data from these contexts cannot be
+ declared as non-interesting for an attacker without deep inspection of
+ the code.
+
+ **Note**, that assigning guests to a fixed set of physical cores affects
+ the ability of the scheduler to do load balancing and might have
+ negative effects on CPU utilization depending on the hosting
+ scenario. Disabling SMT might be a viable alternative for particular
+ scenarios.
+
+ For further information about confining guests to a single or to a group
+ of cores consult the cpusets documentation:
+
+ https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt
+
+.. _interrupt_isolation:
+
+3. Interrupt affinity
+^^^^^^^^^^^^^^^^^^^^^
+
+ Interrupts can be made affine to logical CPUs. This is not universally
+ true because there are types of interrupts which are truly per CPU
+ interrupts, e.g. the local timer interrupt. Aside of that multi queue
+ devices affine their interrupts to single CPUs or groups of CPUs per
+ queue without allowing the administrator to control the affinities.
+
+ Moving the interrupts, which can be affinity controlled, away from CPUs
+ which run untrusted guests, reduces the attack vector space.
+
+ Whether the interrupts with are affine to CPUs, which run untrusted
+ guests, provide interesting data for an attacker depends on the system
+ configuration and the scenarios which run on the system. While for some
+ of the interrupts it can be assumed that they won't expose interesting
+ information beyond exposing hints about the host OS memory layout, there
+ is no way to make general assumptions.
+
+ Interrupt affinity can be controlled by the administrator via the
+ /proc/irq/$NR/smp_affinity[_list] files. Limited documentation is
+ available at:
+
+ https://www.kernel.org/doc/Documentation/IRQ-affinity.txt
+
+.. _smt_control:
+
+4. SMT control
+^^^^^^^^^^^^^^
+
+ To prevent the SMT issues of L1TF it might be necessary to disable SMT
+ completely. Disabling SMT can have a significant performance impact, but
+ the impact depends on the hosting scenario and the type of workloads.
+ The impact of disabling SMT needs also to be weighted against the impact
+ of other mitigation solutions like confining guests to dedicated cores.
+
+ The kernel provides a sysfs interface to retrieve the status of SMT and
+ to control it. It also provides a kernel command line interface to
+ control SMT.
+
+ The kernel command line interface consists of the following options:
+
+ =========== ==========================================================
+ nosmt Affects the bring up of the secondary CPUs during boot. The
+ kernel tries to bring all present CPUs online during the
+ boot process. "nosmt" makes sure that from each physical
+ core only one - the so called primary (hyper) thread is
+ activated. Due to a design flaw of Intel processors related
+ to Machine Check Exceptions the non primary siblings have
+ to be brought up at least partially and are then shut down
+ again. "nosmt" can be undone via the sysfs interface.
+
+ nosmt=force Has the same effect as "nosmt" but it does not allow to
+ undo the SMT disable via the sysfs interface.
+ =========== ==========================================================
+
+ The sysfs interface provides two files:
+
+ - /sys/devices/system/cpu/smt/control
+ - /sys/devices/system/cpu/smt/active
+
+ /sys/devices/system/cpu/smt/control:
+
+ This file allows to read out the SMT control state and provides the
+ ability to disable or (re)enable SMT. The possible states are:
+
+ ============== ===================================================
+ on SMT is supported by the CPU and enabled. All
+ logical CPUs can be onlined and offlined without
+ restrictions.
+
+ off SMT is supported by the CPU and disabled. Only
+ the so called primary SMT threads can be onlined
+ and offlined without restrictions. An attempt to
+ online a non-primary sibling is rejected
+
+ forceoff Same as 'off' but the state cannot be controlled.
+ Attempts to write to the control file are rejected.
+
+ notsupported The processor does not support SMT. It's therefore
+ not affected by the SMT implications of L1TF.
+ Attempts to write to the control file are rejected.
+ ============== ===================================================
+
+ The possible states which can be written into this file to control SMT
+ state are:
+
+ - on
+ - off
+ - forceoff
+
+ /sys/devices/system/cpu/smt/active:
+
+ This file reports whether SMT is enabled and active, i.e. if on any
+ physical core two or more sibling threads are online.
+
+ SMT control is also possible at boot time via the l1tf kernel command
+ line parameter in combination with L1D flush control. See
+ :ref:`mitigation_control_command_line`.
+
+5. Disabling EPT
+^^^^^^^^^^^^^^^^
+
+ Disabling EPT for virtual machines provides full mitigation for L1TF even
+ with SMT enabled, because the effective page tables for guests are
+ managed and sanitized by the hypervisor. Though disabling EPT has a
+ significant performance impact especially when the Meltdown mitigation
+ KPTI is enabled.
+
+ EPT can be disabled in the hypervisor via the 'kvm-intel.ept' parameter.
+
+There is ongoing research and development for new mitigation mechanisms to
+address the performance impact of disabling SMT or EPT.
+
+.. _mitigation_control_command_line:
+
+Mitigation control on the kernel command line
+---------------------------------------------
+
+The kernel command line allows to control the L1TF mitigations at boot
+time with the option "l1tf=". The valid arguments for this option are:
+
+ ============ =============================================================
+ full Provides all available mitigations for the L1TF
+ vulnerability. Disables SMT and enables all mitigations in
+ the hypervisors, i.e. unconditional L1D flushing
+
+ SMT control and L1D flush control via the sysfs interface
+ is still possible after boot. Hypervisors will issue a
+ warning when the first VM is started in a potentially
+ insecure configuration, i.e. SMT enabled or L1D flush
+ disabled.
+
+ full,force Same as 'full', but disables SMT and L1D flush runtime
+ control. Implies the 'nosmt=force' command line option.
+ (i.e. sysfs control of SMT is disabled.)
+
+ flush Leaves SMT enabled and enables the default hypervisor
+ mitigation, i.e. conditional L1D flushing
+
+ SMT control and L1D flush control via the sysfs interface
+ is still possible after boot. Hypervisors will issue a
+ warning when the first VM is started in a potentially
+ insecure configuration, i.e. SMT enabled or L1D flush
+ disabled.
+
+ flush,nosmt Disables SMT and enables the default hypervisor mitigation,
+ i.e. conditional L1D flushing.
+
+ SMT control and L1D flush control via the sysfs interface
+ is still possible after boot. Hypervisors will issue a
+ warning when the first VM is started in a potentially
+ insecure configuration, i.e. SMT enabled or L1D flush
+ disabled.
+
+ flush,nowarn Same as 'flush', but hypervisors will not warn when a VM is
+ started in a potentially insecure configuration.
+
+ off Disables hypervisor mitigations and doesn't emit any
+ warnings.
+ ============ =============================================================
+
+The default is 'flush'. For details about L1D flushing see :ref:`l1d_flush`.
+
+
+.. _mitigation_control_kvm:
+
+Mitigation control for KVM - module parameter
+-------------------------------------------------------------
+
+The KVM hypervisor mitigation mechanism, flushing the L1D cache when
+entering a guest, can be controlled with a module parameter.
+
+The option/parameter is "kvm-intel.vmentry_l1d_flush=". It takes the
+following arguments:
+
+ ============ ==============================================================
+ always L1D cache flush on every VMENTER.
+
+ cond Flush L1D on VMENTER only when the code between VMEXIT and
+ VMENTER can leak host memory which is considered
+ interesting for an attacker. This still can leak host memory
+ which allows e.g. to determine the hosts address space layout.
+
+ never Disables the mitigation
+ ============ ==============================================================
+
+The parameter can be provided on the kernel command line, as a module
+parameter when loading the modules and at runtime modified via the sysfs
+file:
+
+/sys/module/kvm_intel/parameters/vmentry_l1d_flush
+
+The default is 'cond'. If 'l1tf=full,force' is given on the kernel command
+line, then 'always' is enforced and the kvm-intel.vmentry_l1d_flush
+module parameter is ignored and writes to the sysfs file are rejected.
+
+
+Mitigation selection guide
+--------------------------
+
+1. No virtualization in use
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+ The system is protected by the kernel unconditionally and no further
+ action is required.
+
+2. Virtualization with trusted guests
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+ If the guest comes from a trusted source and the guest OS kernel is
+ guaranteed to have the L1TF mitigations in place the system is fully
+ protected against L1TF and no further action is required.
+
+ To avoid the overhead of the default L1D flushing on VMENTER the
+ administrator can disable the flushing via the kernel command line and
+ sysfs control files. See :ref:`mitigation_control_command_line` and
+ :ref:`mitigation_control_kvm`.
+
+
+3. Virtualization with untrusted guests
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+3.1. SMT not supported or disabled
+""""""""""""""""""""""""""""""""""
+
+ If SMT is not supported by the processor or disabled in the BIOS or by
+ the kernel, it's only required to enforce L1D flushing on VMENTER.
+
+ Conditional L1D flushing is the default behaviour and can be tuned. See
+ :ref:`mitigation_control_command_line` and :ref:`mitigation_control_kvm`.
+
+3.2. EPT not supported or disabled
+""""""""""""""""""""""""""""""""""
+
+ If EPT is not supported by the processor or disabled in the hypervisor,
+ the system is fully protected. SMT can stay enabled and L1D flushing on
+ VMENTER is not required.
+
+ EPT can be disabled in the hypervisor via the 'kvm-intel.ept' parameter.
+
+3.3. SMT and EPT supported and active
+"""""""""""""""""""""""""""""""""""""
+
+ If SMT and EPT are supported and active then various degrees of
+ mitigations can be employed:
+
+ - L1D flushing on VMENTER:
+
+ L1D flushing on VMENTER is the minimal protection requirement, but it
+ is only potent in combination with other mitigation methods.
+
+ Conditional L1D flushing is the default behaviour and can be tuned. See
+ :ref:`mitigation_control_command_line` and :ref:`mitigation_control_kvm`.
+
+ - Guest confinement:
+
+ Confinement of guests to a single or a group of physical cores which
+ are not running any other processes, can reduce the attack surface
+ significantly, but interrupts, soft interrupts and kernel threads can
+ still expose valuable data to a potential attacker. See
+ :ref:`guest_confinement`.
+
+ - Interrupt isolation:
+
+ Isolating the guest CPUs from interrupts can reduce the attack surface
+ further, but still allows a malicious guest to explore a limited amount
+ of host physical memory. This can at least be used to gain knowledge
+ about the host address space layout. The interrupts which have a fixed
+ affinity to the CPUs which run the untrusted guests can depending on
+ the scenario still trigger soft interrupts and schedule kernel threads
+ which might expose valuable information. See
+ :ref:`interrupt_isolation`.
+
+The above three mitigation methods combined can provide protection to a
+certain degree, but the risk of the remaining attack surface has to be
+carefully analyzed. For full protection the following methods are
+available:
+
+ - Disabling SMT:
+
+ Disabling SMT and enforcing the L1D flushing provides the maximum
+ amount of protection. This mitigation is not depending on any of the
+ above mitigation methods.
+
+ SMT control and L1D flushing can be tuned by the command line
+ parameters 'nosmt', 'l1tf', 'kvm-intel.vmentry_l1d_flush' and at run
+ time with the matching sysfs control files. See :ref:`smt_control`,
+ :ref:`mitigation_control_command_line` and
+ :ref:`mitigation_control_kvm`.
+
+ - Disabling EPT:
+
+ Disabling EPT provides the maximum amount of protection as well. It is
+ not depending on any of the above mitigation methods. SMT can stay
+ enabled and L1D flushing is not required, but the performance impact is
+ significant.
+
+ EPT can be disabled in the hypervisor via the 'kvm-intel.ept'
+ parameter.
+
+3.4. Nested virtual machines
+""""""""""""""""""""""""""""
+
+When nested virtualization is in use, three operating systems are involved:
+the bare metal hypervisor, the nested hypervisor and the nested virtual
+machine. VMENTER operations from the nested hypervisor into the nested
+guest will always be processed by the bare metal hypervisor. If KVM is the
+bare metal hypervisor it wiil:
+
+ - Flush the L1D cache on every switch from the nested hypervisor to the
+ nested virtual machine, so that the nested hypervisor's secrets are not
+ exposed to the nested virtual machine;
+
+ - Flush the L1D cache on every switch from the nested virtual machine to
+ the nested hypervisor; this is a complex operation, and flushing the L1D
+ cache avoids that the bare metal hypervisor's secrets are exposed to the
+ nested virtual machine;
+
+ - Instruct the nested hypervisor to not perform any L1D cache flush. This
+ is an optimization to avoid double L1D flushing.
+
+
+.. _default_mitigations:
+
+Default mitigations
+-------------------
+
+ The kernel default mitigations for vulnerable processors are:
+
+ - PTE inversion to protect against malicious user space. This is done
+ unconditionally and cannot be controlled.
+
+ - L1D conditional flushing on VMENTER when EPT is enabled for
+ a guest.
+
+ The kernel does not by default enforce the disabling of SMT, which leaves
+ SMT systems vulnerable when running untrusted guests with EPT enabled.
+
+ The rationale for this choice is:
+
+ - Force disabling SMT can break existing setups, especially with
+ unattended updates.
+
+ - If regular users run untrusted guests on their machine, then L1TF is
+ just an add on to other malware which might be embedded in an untrusted
+ guest, e.g. spam-bots or attacks on the local network.
+
+ There is no technical way to prevent a user from running untrusted code
+ on their machines blindly.
+
+ - It's technically extremely unlikely and from today's knowledge even
+ impossible that L1TF can be exploited via the most popular attack
+ mechanisms like JavaScript because these mechanisms have no way to
+ control PTEs. If this would be possible and not other mitigation would
+ be possible, then the default might be different.
+
+ - The administrators of cloud and hosting setups have to carefully
+ analyze the risk for their scenarios and make the appropriate
+ mitigation choices, which might even vary across their deployed
+ machines and also result in other changes of their overall setup.
+ There is no way for the kernel to provide a sensible default for this
+ kind of scenarios.
diff --git a/Documentation/leds/leds-class.txt b/Documentation/leds/leds-class.txt
index f1f7ec9..ebdca55 100644
--- a/Documentation/leds/leds-class.txt
+++ b/Documentation/leds/leds-class.txt
@@ -51,6 +51,10 @@
above leaves scope for further attributes should they be needed. If sections
of the name don't apply, just leave that section blank.
+Optionally, the driver may choose to register with the LED_BRIGHTNESS_FAST flag.
+This flag indicates that the driver implements the brightness_set() callback
+function using a fastpath so the LED core can use hrtimer if the driver requires
+high precision for the trigger timing.
Brightness setting API
======================
diff --git a/Documentation/misc-devices/access_ramoops.txt b/Documentation/misc-devices/access_ramoops.txt
new file mode 100644
index 0000000..9bb3581
--- /dev/null
+++ b/Documentation/misc-devices/access_ramoops.txt
@@ -0,0 +1,11 @@
+Kernel driver access_ramoops
+============================
+
+Author: Patrick Tjin <pattjin@google.com>
+
+
+Description
+-----------
+
+The access_ramoops driver allows userspace access to the persistent RAM storage
+used for oops and panics as well as the metadata required to decrypt that data.
diff --git a/Documentation/networking/ip-sysctl.txt b/Documentation/networking/ip-sysctl.txt
index e52a472..827622e 100644
--- a/Documentation/networking/ip-sysctl.txt
+++ b/Documentation/networking/ip-sysctl.txt
@@ -122,14 +122,11 @@
IP Fragmentation:
-ipfrag_high_thresh - INTEGER
- Maximum memory used to reassemble IP fragments. When
- ipfrag_high_thresh bytes of memory is allocated for this purpose,
- the fragment handler will toss packets until ipfrag_low_thresh
- is reached. This also serves as a maximum limit to namespaces
- different from the initial one.
+ipfrag_high_thresh - LONG INTEGER
+ Maximum memory used to reassemble IP fragments.
-ipfrag_low_thresh - INTEGER
+ipfrag_low_thresh - LONG INTEGER
+ (Obsolete since linux-4.17)
Maximum memory used to reassemble IP fragments before the kernel
begins to remove incomplete fragment queues to free up resources.
The kernel still accepts new fragments for defragmentation.
diff --git a/Documentation/networking/netdev-FAQ.txt b/Documentation/networking/netdev-FAQ.txt
index a20b2fa..56af008 100644
--- a/Documentation/networking/netdev-FAQ.txt
+++ b/Documentation/networking/netdev-FAQ.txt
@@ -168,6 +168,15 @@
dash marker line as described in Documentation/SubmittingPatches to
temporarily embed that information into the patch that you send.
+Q: Are all networking bug fixes backported to all stable releases?
+
+A: Due to capacity, Dave could only take care of the backports for the last
+ 2 stable releases. For earlier stable releases, each stable branch maintainer
+ is supposed to take care of them. If you find any patch is missing from an
+ earlier stable branch, please notify stable@vger.kernel.org with either a
+ commit ID or a formal patch backported, and CC Dave and other relevant
+ networking developers.
+
Q: Someone said that the comment style and coding convention is different
for the networking content. Is this true?
diff --git a/Documentation/power/states.txt b/Documentation/power/states.txt
index 50f3ef9..008ecb5 100644
--- a/Documentation/power/states.txt
+++ b/Documentation/power/states.txt
@@ -8,25 +8,41 @@
The states are represented by strings that can be read or written to the
/sys/power/state file. Those strings may be "mem", "standby", "freeze" and
-"disk", where the last one always represents hibernation (Suspend-To-Disk) and
-the meaning of the remaining ones depends on the relative_sleep_states command
-line argument.
+"disk", where the last three always represent Power-On Suspend (if supported),
+Suspend-To-Idle and hibernation (Suspend-To-Disk), respectively.
-For relative_sleep_states=1, the strings "mem", "standby" and "freeze" label the
-available non-hibernation sleep states from the deepest to the shallowest,
-respectively. In that case, "mem" is always present in /sys/power/state,
-because there is at least one non-hibernation sleep state in every system. If
-the given system supports two non-hibernation sleep states, "standby" is present
-in /sys/power/state in addition to "mem". If the system supports three
-non-hibernation sleep states, "freeze" will be present in /sys/power/state in
-addition to "mem" and "standby".
+The meaning of the "mem" string is controlled by the /sys/power/mem_sleep file.
+It contains strings representing the available modes of system suspend that may
+be triggered by writing "mem" to /sys/power/state. These modes are "s2idle"
+(Suspend-To-Idle), "shallow" (Power-On Suspend) and "deep" (Suspend-To-RAM).
+The "s2idle" mode is always available, while the other ones are only available
+if supported by the platform (if not supported, the strings representing them
+are not present in /sys/power/mem_sleep). The string representing the suspend
+mode to be used subsequently is enclosed in square brackets. Writing one of
+the other strings present in /sys/power/mem_sleep to it causes the suspend mode
+to be used subsequently to change to the one represented by that string.
-For relative_sleep_states=0, which is the default, the following descriptions
-apply.
+Consequently, there are two ways to cause the system to go into the
+Suspend-To-Idle sleep state. The first one is to write "freeze" directly to
+/sys/power/state. The second one is to write "s2idle" to /sys/power/mem_sleep
+and then to wrtie "mem" to /sys/power/state. Similarly, there are two ways
+to cause the system to go into the Power-On Suspend sleep state (the strings to
+write to the control files in that case are "standby" or "shallow" and "mem",
+respectively) if that state is supported by the platform. In turn, there is
+only one way to cause the system to go into the Suspend-To-RAM state (write
+"deep" into /sys/power/mem_sleep and "mem" into /sys/power/state).
-state: Suspend-To-Idle
+The default suspend mode (ie. the one to be used without writing anything into
+/sys/power/mem_sleep) is either "deep" (if Suspend-To-RAM is supported) or
+"s2idle", but it can be overridden by the value of the "mem_sleep_default"
+parameter in the kernel command line.
+
+The properties of all of the sleep states are described below.
+
+
+State: Suspend-To-Idle
ACPI state: S0
-Label: "freeze"
+Label: "s2idle" ("freeze")
This state is a generic, pure software, light-weight, system sleep state.
It allows more energy to be saved relative to runtime idle by freezing user
@@ -35,13 +51,13 @@
spend more time in their idle states.
This state can be used for platforms without Power-On Suspend/Suspend-to-RAM
-support, or it can be used in addition to Suspend-to-RAM (memory sleep)
-to provide reduced resume latency. It is always supported.
+support, or it can be used in addition to Suspend-to-RAM to provide reduced
+resume latency. It is always supported.
State: Standby / Power-On Suspend
ACPI State: S1
-Label: "standby"
+Label: "shallow" ("standby")
This state, if supported, offers moderate, though real, power savings, while
providing a relatively low-latency transition back to a working system. No
@@ -58,7 +74,7 @@
State: Suspend-to-RAM
ACPI State: S3
-Label: "mem"
+Label: "deep"
This state, if supported, offers significant power savings as everything in the
system is put into a low-power state, except for memory, which should be placed
diff --git a/Documentation/printk-formats.txt b/Documentation/printk-formats.txt
index 5962949..d2fbeeb 100644
--- a/Documentation/printk-formats.txt
+++ b/Documentation/printk-formats.txt
@@ -279,11 +279,10 @@
%pC pll1
%pCn pll1
- %pCr 1560000000
For printing struct clk structures. '%pC' and '%pCn' print the name
(Common Clock Framework) or address (legacy clock framework) of the
- structure; '%pCr' prints the current clock rate.
+ structure.
Passed by reference.
diff --git a/Documentation/scsi/ufs-impaired-storage.txt b/Documentation/scsi/ufs-impaired-storage.txt
new file mode 100644
index 0000000..1493f21
--- /dev/null
+++ b/Documentation/scsi/ufs-impaired-storage.txt
@@ -0,0 +1,48 @@
+
+UFS-Impaired-Storage
+====================
+
+Contents
+--------
+
+1. Overview
+2. Quick reference guide
+3. Detailed information
+
+1. Overview
+-----------
+
+Flash storage performance can be degraded over time due to the storage and
+file system aging as well as complicated internal structure.
+Impaired Storage emulate degraded storage latencies by adding delay for each
+I/O request.
+
+
+2. Quick reference guide
+------------------------
+
+* Case1: delay all read requests by 1.5x
+ echo 150 > /sys/.../impaired/read_delay_percent
+ echo 1 > /sys/.../impaired/enabled
+
+* Case 2: delay all write requests by 300us
+ echo 300 > /sys/.../impaired/write_delay_us
+ echo 1 > /sys/.../impaired/enabled
+
+* Case 3: delay both read and write requests by 2x, then add 100us
+ echo 200 > /sys/.../impaired/read_delay_percent
+ echo 200 > /sys/.../impaired/write_delay_percent
+ echo 100 > /sys/.../impaired/read_delay_us
+ echo 100 > /sys/.../impaired/write_delay_us
+ echo 1 > /sys/.../impaired/enabled
+
+* Case 4: emulate impaired read latencies.
+ echo fragftl > /sys/.../impaired/read_model
+ echo 1 > /sys/.../impaired/enabled
+
+
+3. Detailed information
+-----------------------
+
+http://go/impairedstorage
+
diff --git a/Documentation/spec_ctrl.txt b/Documentation/spec_ctrl.txt
new file mode 100644
index 0000000..32f3d55
--- /dev/null
+++ b/Documentation/spec_ctrl.txt
@@ -0,0 +1,94 @@
+===================
+Speculation Control
+===================
+
+Quite some CPUs have speculation-related misfeatures which are in
+fact vulnerabilities causing data leaks in various forms even across
+privilege domains.
+
+The kernel provides mitigation for such vulnerabilities in various
+forms. Some of these mitigations are compile-time configurable and some
+can be supplied on the kernel command line.
+
+There is also a class of mitigations which are very expensive, but they can
+be restricted to a certain set of processes or tasks in controlled
+environments. The mechanism to control these mitigations is via
+:manpage:`prctl(2)`.
+
+There are two prctl options which are related to this:
+
+ * PR_GET_SPECULATION_CTRL
+
+ * PR_SET_SPECULATION_CTRL
+
+PR_GET_SPECULATION_CTRL
+-----------------------
+
+PR_GET_SPECULATION_CTRL returns the state of the speculation misfeature
+which is selected with arg2 of prctl(2). The return value uses bits 0-3 with
+the following meaning:
+
+==== ===================== ===================================================
+Bit Define Description
+==== ===================== ===================================================
+0 PR_SPEC_PRCTL Mitigation can be controlled per task by
+ PR_SET_SPECULATION_CTRL.
+1 PR_SPEC_ENABLE The speculation feature is enabled, mitigation is
+ disabled.
+2 PR_SPEC_DISABLE The speculation feature is disabled, mitigation is
+ enabled.
+3 PR_SPEC_FORCE_DISABLE Same as PR_SPEC_DISABLE, but cannot be undone. A
+ subsequent prctl(..., PR_SPEC_ENABLE) will fail.
+==== ===================== ===================================================
+
+If all bits are 0 the CPU is not affected by the speculation misfeature.
+
+If PR_SPEC_PRCTL is set, then the per-task control of the mitigation is
+available. If not set, prctl(PR_SET_SPECULATION_CTRL) for the speculation
+misfeature will fail.
+
+PR_SET_SPECULATION_CTRL
+-----------------------
+
+PR_SET_SPECULATION_CTRL allows to control the speculation misfeature, which
+is selected by arg2 of :manpage:`prctl(2)` per task. arg3 is used to hand
+in the control value, i.e. either PR_SPEC_ENABLE or PR_SPEC_DISABLE or
+PR_SPEC_FORCE_DISABLE.
+
+Common error codes
+------------------
+======= =================================================================
+Value Meaning
+======= =================================================================
+EINVAL The prctl is not implemented by the architecture or unused
+ prctl(2) arguments are not 0.
+
+ENODEV arg2 is selecting a not supported speculation misfeature.
+======= =================================================================
+
+PR_SET_SPECULATION_CTRL error codes
+-----------------------------------
+======= =================================================================
+Value Meaning
+======= =================================================================
+0 Success
+
+ERANGE arg3 is incorrect, i.e. it's neither PR_SPEC_ENABLE nor
+ PR_SPEC_DISABLE nor PR_SPEC_FORCE_DISABLE.
+
+ENXIO Control of the selected speculation misfeature is not possible.
+ See PR_GET_SPECULATION_CTRL.
+
+EPERM Speculation was disabled with PR_SPEC_FORCE_DISABLE and caller
+ tried to enable it again.
+======= =================================================================
+
+Speculation misfeature controls
+-------------------------------
+- PR_SPEC_STORE_BYPASS: Speculative Store Bypass
+
+ Invocations:
+ * prctl(PR_GET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, 0, 0, 0);
+ * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_ENABLE, 0, 0);
+ * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_DISABLE, 0, 0);
+ * prctl(PR_SET_SPECULATION_CTRL, PR_SPEC_STORE_BYPASS, PR_SPEC_FORCE_DISABLE, 0, 0);
diff --git a/Documentation/sysctl/fs.txt b/Documentation/sysctl/fs.txt
index 35e17f7..af5859b 100644
--- a/Documentation/sysctl/fs.txt
+++ b/Documentation/sysctl/fs.txt
@@ -34,7 +34,9 @@
- overflowgid
- pipe-user-pages-hard
- pipe-user-pages-soft
+- protected_fifos
- protected_hardlinks
+- protected_regular
- protected_symlinks
- suid_dumpable
- super-max
@@ -182,6 +184,24 @@
==============================================================
+protected_fifos:
+
+The intent of this protection is to avoid unintentional writes to
+an attacker-controlled FIFO, where a program expected to create a regular
+file.
+
+When set to "0", writing to FIFOs is unrestricted.
+
+When set to "1" don't allow O_CREAT open on FIFOs that we don't own
+in world writable sticky directories, unless they are owned by the
+owner of the directory.
+
+When set to "2" it also applies to group writable sticky directories.
+
+This protection is based on the restrictions in Openwall.
+
+==============================================================
+
protected_hardlinks:
A long-standing class of security issues is the hardlink-based
@@ -202,6 +222,22 @@
==============================================================
+protected_regular:
+
+This protection is similar to protected_fifos, but it
+avoids writes to an attacker-controlled regular file, where a program
+expected to create one.
+
+When set to "0", writing to regular files is unrestricted.
+
+When set to "1" don't allow O_CREAT open on regular files that we
+don't own in world writable sticky directories, unless they are
+owned by the owner of the directory.
+
+When set to "2" it also applies to group writable sticky directories.
+
+==============================================================
+
protected_symlinks:
A long-standing class of security issues is the symlink-based
diff --git a/Documentation/usb/typec.rst b/Documentation/usb/typec.rst
new file mode 100644
index 0000000..b67a467
--- /dev/null
+++ b/Documentation/usb/typec.rst
@@ -0,0 +1,184 @@
+
+USB Type-C connector class
+==========================
+
+Introduction
+------------
+
+The typec class is meant for describing the USB Type-C ports in a system to the
+user space in unified fashion. The class is designed to provide nothing else
+except the user space interface implementation in hope that it can be utilized
+on as many platforms as possible.
+
+The platforms are expected to register every USB Type-C port they have with the
+class. In a normal case the registration will be done by a USB Type-C or PD PHY
+driver, but it may be a driver for firmware interface such as UCSI, driver for
+USB PD controller or even driver for Thunderbolt3 controller. This document
+considers the component registering the USB Type-C ports with the class as "port
+driver".
+
+On top of showing the capabilities, the class also offer user space control over
+the roles and alternate modes of ports, partners and cable plugs when the port
+driver is capable of supporting those features.
+
+The class provides an API for the port drivers described in this document. The
+attributes are described in Documentation/ABI/testing/sysfs-class-typec.
+
+User space interface
+--------------------
+Every port will be presented as its own device under /sys/class/typec/. The
+first port will be named "port0", the second "port1" and so on.
+
+When connected, the partner will be presented also as its own device under
+/sys/class/typec/. The parent of the partner device will always be the port it
+is attached to. The partner attached to port "port0" will be named
+"port0-partner". Full path to the device would be
+/sys/class/typec/port0/port0-partner/.
+
+The cable and the two plugs on it may also be optionally presented as their own
+devices under /sys/class/typec/. The cable attached to the port "port0" port
+will be named port0-cable and the plug on the SOP Prime end (see USB Power
+Delivery Specification ch. 2.4) will be named "port0-plug0" and on the SOP
+Double Prime end "port0-plug1". The parent of a cable will always be the port,
+and the parent of the cable plugs will always be the cable.
+
+If the port, partner or cable plug supports Alternate Modes, every supported
+Alternate Mode SVID will have their own device describing them. Note that the
+Alternate Mode devices will not be attached to the typec class. The parent of an
+alternate mode will be the device that supports it, so for example an alternate
+mode of port0-partner will be presented under /sys/class/typec/port0-partner/.
+Every mode that is supported will have its own group under the Alternate Mode
+device named "mode<index>", for example /sys/class/typec/port0/<alternate
+mode>/mode1/. The requests for entering/exiting a mode can be done with "active"
+attribute file in that group.
+
+Driver API
+----------
+
+Registering the ports
+~~~~~~~~~~~~~~~~~~~~~
+
+The port drivers will describe every Type-C port they control with struct
+typec_capability data structure, and register them with the following API:
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_register_port typec_unregister_port
+
+When registering the ports, the prefer_role member in struct typec_capability
+deserves special notice. If the port that is being registered does not have
+initial role preference, which means the port does not execute Try.SNK or
+Try.SRC by default, the member must have value TYPEC_NO_PREFERRED_ROLE.
+Otherwise if the port executes Try.SNK by default, the member must have value
+TYPEC_DEVICE, and with Try.SRC the value must be TYPEC_HOST.
+
+Registering Partners
+~~~~~~~~~~~~~~~~~~~~
+
+After successful connection of a partner, the port driver needs to register the
+partner with the class. Details about the partner need to be described in struct
+typec_partner_desc. The class copies the details of the partner during
+registration. The class offers the following API for registering/unregistering
+partners.
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_register_partner typec_unregister_partner
+
+The class will provide a handle to struct typec_partner if the registration was
+successful, or NULL.
+
+If the partner is USB Power Delivery capable, and the port driver is able to
+show the result of Discover Identity command, the partner descriptor structure
+should include handle to struct usb_pd_identity instance. The class will then
+create a sysfs directory for the identity under the partner device. The result
+of Discover Identity command can then be reported with the following API:
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_partner_set_identity
+
+Registering Cables
+~~~~~~~~~~~~~~~~~~
+
+After successful connection of a cable that supports USB Power Delivery
+Structured VDM "Discover Identity", the port driver needs to register the cable
+and one or two plugs, depending if there is CC Double Prime controller present
+in the cable or not. So a cable capable of SOP Prime communication, but not SOP
+Double Prime communication, should only have one plug registered. For more
+information about SOP communication, please read chapter about it from the
+latest USB Power Delivery specification.
+
+The plugs are represented as their own devices. The cable is registered first,
+followed by registration of the cable plugs. The cable will be the parent device
+for the plugs. Details about the cable need to be described in struct
+typec_cable_desc and about a plug in struct typec_plug_desc. The class copies
+the details during registration. The class offers the following API for
+registering/unregistering cables and their plugs:
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_register_cable typec_unregister_cable typec_register_plug
+ typec_unregister_plug
+
+The class will provide a handle to struct typec_cable and struct typec_plug if
+the registration is successful, or NULL if it isn't.
+
+If the cable is USB Power Delivery capable, and the port driver is able to show
+the result of Discover Identity command, the cable descriptor structure should
+include handle to struct usb_pd_identity instance. The class will then create a
+sysfs directory for the identity under the cable device. The result of Discover
+Identity command can then be reported with the following API:
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_cable_set_identity
+
+Notifications
+~~~~~~~~~~~~~
+
+When the partner has executed a role change, or when the default roles change
+during connection of a partner or cable, the port driver must use the following
+APIs to report it to the class:
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_set_data_role typec_set_pwr_role typec_set_vconn_role
+ typec_set_pwr_opmode
+
+Alternate Modes
+~~~~~~~~~~~~~~~
+
+USB Type-C ports, partners and cable plugs may support Alternate Modes. Each
+Alternate Mode will have identifier called SVID, which is either a Standard ID
+given by USB-IF or vendor ID, and each supported SVID can have 1 - 6 modes. The
+class provides struct typec_mode_desc for describing individual mode of a SVID,
+and struct typec_altmode_desc which is a container for all the supported modes.
+
+Ports that support Alternate Modes need to register each SVID they support with
+the following API:
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_port_register_altmode
+
+If a partner or cable plug provides a list of SVIDs as response to USB Power
+Delivery Structured VDM Discover SVIDs message, each SVID needs to be
+registered.
+
+API for the partners:
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_partner_register_altmode
+
+API for the Cable Plugs:
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_plug_register_altmode
+
+So ports, partners and cable plugs will register the alternate modes with their
+own functions, but the registration will always return a handle to struct
+typec_altmode on success, or NULL. The unregistration will happen with the same
+function:
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_unregister_altmode
+
+If a partner or cable plug enters or exits a mode, the port driver needs to
+notify the class with the following API:
+
+.. kernel-doc:: drivers/usb/typec/typec.c
+ :functions: typec_altmode_update_active
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 1f5eab4..3ff58a8 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -122,14 +122,15 @@
privileged user (CAP_SYS_ADMIN).
-4.3 KVM_GET_MSR_INDEX_LIST
+4.3 KVM_GET_MSR_INDEX_LIST, KVM_GET_MSR_FEATURE_INDEX_LIST
-Capability: basic
+Capability: basic, KVM_CAP_GET_MSR_FEATURES for KVM_GET_MSR_FEATURE_INDEX_LIST
Architectures: x86
-Type: system
+Type: system ioctl
Parameters: struct kvm_msr_list (in/out)
Returns: 0 on success; -1 on error
Errors:
+ EFAULT: the msr index list cannot be read from or written to
E2BIG: the msr index list is to be to fit in the array specified by
the user.
@@ -138,16 +139,23 @@
__u32 indices[0];
};
-This ioctl returns the guest msrs that are supported. The list varies
-by kvm version and host processor, but does not change otherwise. The
-user fills in the size of the indices array in nmsrs, and in return
-kvm adjusts nmsrs to reflect the actual number of msrs and fills in
-the indices array with their numbers.
+The user fills in the size of the indices array in nmsrs, and in return
+kvm adjusts nmsrs to reflect the actual number of msrs and fills in the
+indices array with their numbers.
+
+KVM_GET_MSR_INDEX_LIST returns the guest msrs that are supported. The list
+varies by kvm version and host processor, but does not change otherwise.
Note: if kvm indicates supports MCE (KVM_CAP_MCE), then the MCE bank MSRs are
not returned in the MSR list, as different vcpus can have a different number
of banks, as set via the KVM_X86_SETUP_MCE ioctl.
+KVM_GET_MSR_FEATURE_INDEX_LIST returns the list of MSRs that can be passed
+to the KVM_GET_MSRS system ioctl. This lets userspace probe host capabilities
+and processor features that are exposed via MSRs (e.g., VMX capabilities).
+This list also varies by kvm version and host processor, but does not change
+otherwise.
+
4.4 KVM_CHECK_EXTENSION
@@ -474,14 +482,22 @@
4.18 KVM_GET_MSRS
-Capability: basic
+Capability: basic (vcpu), KVM_CAP_GET_MSR_FEATURES (system)
Architectures: x86
-Type: vcpu ioctl
+Type: system ioctl, vcpu ioctl
Parameters: struct kvm_msrs (in/out)
-Returns: 0 on success, -1 on error
+Returns: number of msrs successfully returned;
+ -1 on error
+When used as a system ioctl:
+Reads the values of MSR-based features that are available for the VM. This
+is similar to KVM_GET_SUPPORTED_CPUID, but it returns MSR indices and values.
+The list of msr-based features can be obtained using KVM_GET_MSR_FEATURE_INDEX_LIST
+in a system ioctl.
+
+When used as a vcpu ioctl:
Reads model-specific registers from the vcpu. Supported msr indices can
-be obtained using KVM_GET_MSR_INDEX_LIST.
+be obtained using KVM_GET_MSR_INDEX_LIST in a system ioctl.
struct kvm_msrs {
__u32 nmsrs; /* number of msrs in entries */
@@ -2118,6 +2134,9 @@
ARM 64-bit FP registers have the following id bit patterns:
0x4030 0000 0012 0 <regno:12>
+ARM firmware pseudo-registers have the following bit pattern:
+ 0x4030 0000 0014 <regno:16>
+
arm64 registers are mapped using the lower 32 bits. The upper 16 of
that is the register group type, or coprocessor number:
@@ -2134,6 +2153,9 @@
arm64 system registers have the following id bit patterns:
0x6030 0000 0013 <op0:2> <op1:3> <crn:4> <crm:4> <op2:3>
+arm64 firmware pseudo-registers have the following bit pattern:
+ 0x6030 0000 0014 <regno:16>
+
MIPS registers are mapped using the lower 32 bits. The upper 16 of that is
the register group type:
@@ -2656,7 +2678,8 @@
and execute guest code when KVM_RUN is called.
- KVM_ARM_VCPU_EL1_32BIT: Starts the CPU in a 32bit mode.
Depends on KVM_CAP_ARM_EL1_32BIT (arm64 only).
- - KVM_ARM_VCPU_PSCI_0_2: Emulate PSCI v0.2 for the CPU.
+ - KVM_ARM_VCPU_PSCI_0_2: Emulate PSCI v0.2 (or a future revision
+ backward compatible with v0.2) for the CPU.
Depends on KVM_CAP_ARM_PSCI_0_2.
- KVM_ARM_VCPU_PMU_V3: Emulate PMUv3 for the CPU.
Depends on KVM_CAP_ARM_PMU_V3.
diff --git a/Documentation/virtual/kvm/arm/psci.txt b/Documentation/virtual/kvm/arm/psci.txt
new file mode 100644
index 0000000..aafdab8
--- /dev/null
+++ b/Documentation/virtual/kvm/arm/psci.txt
@@ -0,0 +1,30 @@
+KVM implements the PSCI (Power State Coordination Interface)
+specification in order to provide services such as CPU on/off, reset
+and power-off to the guest.
+
+The PSCI specification is regularly updated to provide new features,
+and KVM implements these updates if they make sense from a virtualization
+point of view.
+
+This means that a guest booted on two different versions of KVM can
+observe two different "firmware" revisions. This could cause issues if
+a given guest is tied to a particular PSCI revision (unlikely), or if
+a migration causes a different PSCI version to be exposed out of the
+blue to an unsuspecting guest.
+
+In order to remedy this situation, KVM exposes a set of "firmware
+pseudo-registers" that can be manipulated using the GET/SET_ONE_REG
+interface. These registers can be saved/restored by userspace, and set
+to a convenient value if required.
+
+The following register is defined:
+
+* KVM_REG_ARM_PSCI_VERSION:
+
+ - Only valid if the vcpu has the KVM_ARM_VCPU_PSCI_0_2 feature set
+ (and thus has already been initialized)
+ - Returns the current PSCI version on GET_ONE_REG (defaulting to the
+ highest PSCI version implemented by KVM and compatible with v0.2)
+ - Allows any PSCI version implemented by KVM and compatible with
+ v0.2 to be set with SET_ONE_REG
+ - Affects the whole VM (even if the register view is per-vcpu)
diff --git a/MAINTAINERS b/MAINTAINERS
index a419303..7369cbe 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -11482,6 +11482,7 @@
STABLE BRANCH
M: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
+M: Sasha Levin <sashal@kernel.org>
L: stable@vger.kernel.org
S: Supported
F: Documentation/stable_kernel_rules.txt
@@ -12640,6 +12641,15 @@
F: include/linux/usb.h
F: include/linux/usb/
+USB TYPEC SUBSYSTEM
+M: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+L: linux-usb@vger.kernel.org
+S: Maintained
+F: Documentation/ABI/testing/sysfs-class-typec
+F: Documentation/usb/typec.rst
+F: drivers/usb/typec/
+F: include/linux/usb/typec.h
+
USB UHCI DRIVER
M: Alan Stern <stern@rowland.harvard.edu>
L: linux-usb@vger.kernel.org
diff --git a/Makefile b/Makefile
index 188a66d..6ace9fc 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,6 @@
VERSION = 4
PATCHLEVEL = 9
-SUBLEVEL = 96
+SUBLEVEL = 150
EXTRAVERSION =
NAME = Roaring Lionus
@@ -348,14 +348,16 @@
# Make variables (CC, etc...)
AS = $(CROSS_COMPILE)as
LD = $(CROSS_COMPILE)ld
-REAL_CC = $(CROSS_COMPILE)gcc
LDGOLD = $(CROSS_COMPILE)ld.gold
+REAL_CC = $(CROSS_COMPILE)gcc
+CC = $(CROSS_COMPILE)gcc
CPP = $(CC) -E
AR = $(CROSS_COMPILE)ar
NM = $(CROSS_COMPILE)nm
STRIP = $(CROSS_COMPILE)strip
OBJCOPY = $(CROSS_COMPILE)objcopy
OBJDUMP = $(CROSS_COMPILE)objdump
+DTC = scripts/dtc/dtc
AWK = awk
GENKSYMS = scripts/genksyms/genksyms
INSTALLKERNEL := installkernel
@@ -399,7 +401,7 @@
KBUILD_AFLAGS := -D__ASSEMBLY__
KBUILD_CFLAGS := -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs \
- -fno-strict-aliasing -fno-common \
+ -fno-strict-aliasing -fno-common -fshort-wchar \
-Werror-implicit-function-declaration \
-Wno-format-security \
-std=gnu89
@@ -417,12 +419,13 @@
export VERSION PATCHLEVEL SUBLEVEL KERNELRELEASE KERNELVERSION
export ARCH SRCARCH CONFIG_SHELL HOSTCC HOSTCFLAGS CROSS_COMPILE AS LD CC
-export CPP AR NM STRIP OBJCOPY OBJDUMP
+export CPP AR NM STRIP OBJCOPY OBJDUMP DTC
export MAKE AWK GENKSYMS INSTALLKERNEL PERL PYTHON UTS_MACHINE
export HOSTCXX HOSTCXXFLAGS LDFLAGS_MODULE CHECK CHECKFLAGS
-export KBUILD_CPPFLAGS NOSTDINC_FLAGS LINUXINCLUDE OBJCOPYFLAGS LDFLAGS
-export KBUILD_CFLAGS CFLAGS_KERNEL CFLAGS_MODULE CFLAGS_KASAN CFLAGS_UBSAN
+export KBUILD_CPPFLAGS NOSTDINC_FLAGS LINUXINCLUDE OBJCOPYFLAGS LDFLAGS DTC_FLAGS
+export KBUILD_CFLAGS CFLAGS_KERNEL CFLAGS_MODULE
+export CFLAGS_KASAN CFLAGS_KASAN_NOSANITIZE CFLAGS_UBSAN
export KBUILD_AFLAGS AFLAGS_KERNEL AFLAGS_MODULE
export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_LDFLAGS_MODULE
export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL
@@ -513,6 +516,43 @@
endif
endif
+ifeq ($(cc-name),clang)
+ifneq ($(CROSS_COMPILE),)
+CLANG_TRIPLE ?= $(CROSS_COMPILE)
+CLANG_TARGET := --target=$(notdir $(CLANG_TRIPLE:%-=%))
+GCC_TOOLCHAIN_DIR := $(dir $(shell which $(LD)))
+CLANG_PREFIX := --prefix=$(GCC_TOOLCHAIN_DIR)
+GCC_TOOLCHAIN := $(realpath $(GCC_TOOLCHAIN_DIR)/..)
+endif
+ifneq ($(GCC_TOOLCHAIN),)
+CLANG_GCC_TC := --gcc-toolchain=$(GCC_TOOLCHAIN)
+endif
+KBUILD_CFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC) $(CLANG_PREFIX)
+KBUILD_AFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC) $(CLANG_PREFIX)
+KBUILD_CPPFLAGS += $(call cc-option,-Qunused-arguments,)
+KBUILD_CFLAGS += $(call cc-disable-warning, unused-variable)
+KBUILD_CFLAGS += $(call cc-disable-warning, format-invalid-specifier)
+KBUILD_CFLAGS += $(call cc-disable-warning, gnu)
+KBUILD_CFLAGS += $(call cc-disable-warning, address-of-packed-member)
+KBUILD_CFLAGS += $(call cc-disable-warning, duplicate-decl-specifier)
+# Quiet clang warning: comparison of unsigned expression < 0 is always false
+KBUILD_CFLAGS += $(call cc-disable-warning, tautological-compare)
+# CLANG uses a _MergedGlobals as optimization, but this breaks modpost, as the
+# source of a reference will be _MergedGlobals and not on of the whitelisted names.
+# See modpost pattern 2
+KBUILD_CFLAGS += $(call cc-option, -mno-global-merge,)
+KBUILD_CFLAGS += $(call cc-option, -fcatch-undefined-behavior)
+KBUILD_CFLAGS += $(call cc-option, -no-integrated-as)
+KBUILD_AFLAGS += $(call cc-option, -no-integrated-as)
+else
+
+# These warnings generated too much noise in a regular build.
+# Use make W=1 to enable them (see scripts/Makefile.build)
+KBUILD_CFLAGS += $(call cc-disable-warning, unused-but-set-variable)
+KBUILD_CFLAGS += $(call cc-disable-warning, unused-const-variable)
+endif
+
+
ifeq ($(mixed-targets),1)
# ===========================================================================
# We're called with mixed targets (*config and build targets).
@@ -622,12 +662,6 @@
# Defaults to vmlinux, but the arch makefile usually adds further targets
all: vmlinux
-KBUILD_CFLAGS += $(call cc-option,-fno-PIE)
-KBUILD_AFLAGS += $(call cc-option,-fno-PIE)
-CFLAGS_GCOV := -fprofile-arcs -ftest-coverage -fno-tree-loop-im $(call cc-disable-warning,maybe-uninitialized,)
-CFLAGS_KCOV := $(call cc-option,-fsanitize-coverage=trace-pc,)
-export CFLAGS_GCOV CFLAGS_KCOV
-
# Make toolchain changes before including arch/$(SRCARCH)/Makefile to ensure
# ar/cc/ld-* macros return correct values.
ifdef CONFIG_LTO_CLANG
@@ -635,6 +669,8 @@
LDFINAL_vmlinux := $(LD)
LD := $(LDGOLD)
LDFLAGS += -plugin LLVMgold.so
+LDFLAGS += -plugin-opt=-function-sections
+LDFLAGS += -plugin-opt=-data-sections
# use llvm-ar for building symbol tables from IR files, and llvm-dis instead
# of objdump for processing symbol versions and exports
LLVM_AR := llvm-ar
@@ -642,6 +678,14 @@
export LLVM_AR LLVM_DIS
endif
+KBUILD_CFLAGS += $(call cc-option,-fno-PIE)
+KBUILD_AFLAGS += $(call cc-option,-fno-PIE)
+CFLAGS_GCOV := -fprofile-arcs -ftest-coverage \
+ $(call cc-option,-fno-tree-loop-im) \
+ $(call cc-disable-warning,maybe-uninitialized,)
+CFLAGS_KCOV := $(call cc-option,-fsanitize-coverage=trace-pc,)
+export CFLAGS_GCOV CFLAGS_KCOV
+
# The arch Makefile can set ARCH_{CPP,A,C}FLAGS to override the default
# values of the respective KBUILD_* variables
ARCH_CPPFLAGS :=
@@ -654,6 +698,7 @@
KBUILD_CFLAGS += $(call cc-disable-warning, format-truncation)
KBUILD_CFLAGS += $(call cc-disable-warning, format-overflow)
KBUILD_CFLAGS += $(call cc-disable-warning, int-in-bool-context)
+KBUILD_CFLAGS += $(call cc-disable-warning, attribute-alias)
ifdef CONFIG_LD_DEAD_CODE_DATA_ELIMINATION
KBUILD_CFLAGS += $(call cc-option,-ffunction-sections,)
@@ -707,6 +752,13 @@
export DISABLE_CFI
endif
+ifdef CONFIG_SHADOW_CALL_STACK
+scs-flags := -fsanitize=shadow-call-stack
+KBUILD_CFLAGS += $(scs-flags)
+DISABLE_SCS := -fno-sanitize=shadow-call-stack
+export DISABLE_SCS
+endif
+
ifdef CONFIG_CC_OPTIMIZE_FOR_SIZE
KBUILD_CFLAGS += $(call cc-option,-Oz,-Os)
KBUILD_CFLAGS += $(call cc-disable-warning,maybe-uninitialized,)
@@ -721,6 +773,10 @@
KBUILD_CFLAGS += $(call cc-ifversion, -lt, 0409, \
$(call cc-disable-warning,maybe-uninitialized,))
+ifdef CONFIG_CC_WERROR
+KBUILD_CFLAGS += -Werror
+endif
+
# Tell gcc to never replace conditional load with a non-conditional one
KBUILD_CFLAGS += $(call cc-option,--param=allow-store-data-races=0)
@@ -768,40 +824,6 @@
endif
KBUILD_CFLAGS += $(stackp-flag)
-ifeq ($(cc-name),clang)
-ifneq ($(CROSS_COMPILE),)
-CLANG_TRIPLE ?= $(CROSS_COMPILE)
-CLANG_TARGET := --target=$(notdir $(CLANG_TRIPLE:%-=%))
-GCC_TOOLCHAIN := $(realpath $(dir $(shell which $(LD)))/..)
-endif
-ifneq ($(GCC_TOOLCHAIN),)
-CLANG_GCC_TC := --gcc-toolchain=$(GCC_TOOLCHAIN)
-endif
-KBUILD_CFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC)
-KBUILD_AFLAGS += $(CLANG_TARGET) $(CLANG_GCC_TC)
-KBUILD_CPPFLAGS += $(call cc-option,-Qunused-arguments,)
-KBUILD_CFLAGS += $(call cc-disable-warning, unused-variable)
-KBUILD_CFLAGS += $(call cc-disable-warning, format-invalid-specifier)
-KBUILD_CFLAGS += $(call cc-disable-warning, gnu)
-KBUILD_CFLAGS += $(call cc-disable-warning, address-of-packed-member)
-KBUILD_CFLAGS += $(call cc-disable-warning, duplicate-decl-specifier)
-# Quiet clang warning: comparison of unsigned expression < 0 is always false
-KBUILD_CFLAGS += $(call cc-disable-warning, tautological-compare)
-# CLANG uses a _MergedGlobals as optimization, but this breaks modpost, as the
-# source of a reference will be _MergedGlobals and not on of the whitelisted names.
-# See modpost pattern 2
-KBUILD_CFLAGS += $(call cc-option, -mno-global-merge,)
-KBUILD_CFLAGS += $(call cc-option, -fcatch-undefined-behavior)
-KBUILD_CFLAGS += $(call cc-option, -no-integrated-as)
-KBUILD_AFLAGS += $(call cc-option, -no-integrated-as)
-else
-
-# These warnings generated too much noise in a regular build.
-# Use make W=1 to enable them (see scripts/Makefile.build)
-KBUILD_CFLAGS += $(call cc-disable-warning, unused-but-set-variable)
-KBUILD_CFLAGS += $(call cc-disable-warning, unused-const-variable)
-endif
-
ifdef CONFIG_FRAME_POINTER
KBUILD_CFLAGS += -fno-omit-frame-pointer -fno-optimize-sibling-calls
else
@@ -867,6 +889,9 @@
# disable pointer signed / unsigned warnings in gcc 4.0
KBUILD_CFLAGS += $(call cc-disable-warning, pointer-sign)
+# disable stringop warnings in gcc 8+
+KBUILD_CFLAGS += $(call cc-disable-warning, stringop-truncation)
+
# disable invalid "can't wrap" optimizations for signed / pointers
KBUILD_CFLAGS += $(call cc-option,-fno-strict-overflow)
@@ -1193,6 +1218,11 @@
@echo Cannot use CONFIG_CFI: $(cfi-flags) not supported by compiler >&2 && exit 1
endif
endif
+ifdef scs-flags
+ ifeq ($(call cc-option, $(scs-flags)),)
+ @echo Cannot use CONFIG_SHADOW_CALL_STACK: $(scs-flags) not supported by compiler >&2 && exit 1
+ endif
+endif
@:
# Generate some files
@@ -1203,11 +1233,15 @@
uts_len := 64
define filechk_utsrelease.h
- if [ `echo -n "$(KERNELRELEASE)" | wc -c ` -gt $(uts_len) ]; then \
- echo '"$(KERNELRELEASE)" exceeds $(uts_len) characters' >&2; \
- exit 1; \
- fi; \
- (echo \#define UTS_RELEASE \"$(KERNELRELEASE)\";)
+ if [ `echo -n "$(KERNELRELEASE)" | wc -c ` -gt $(uts_len) ]; then \
+ echo '"$(KERNELRELEASE)" exceeds $(uts_len) characters' >&2; \
+ exit 1; \
+ fi; \
+ if [ -n "$(BUILD_NUMBER)" ]; then \
+ (echo \#define UTS_RELEASE \"$(KERNELRELEASE)-ab$(BUILD_NUMBER)\";) \
+ else \
+ (echo \#define UTS_RELEASE \"$(KERNELRELEASE)\";) \
+ fi
endef
define filechk_version.h
diff --git a/PREUPLOAD.cfg b/PREUPLOAD.cfg
new file mode 100644
index 0000000..9124589
--- /dev/null
+++ b/PREUPLOAD.cfg
@@ -0,0 +1,2 @@
+[Hook Scripts]
+checkpatch_hook = ${REPO_ROOT}/build/static_analysis/checkpatch.sh --git_sha1 ${PREUPLOAD_COMMIT}
diff --git a/arch/Kconfig b/arch/Kconfig
index 4fa799b..364d5da 100644
--- a/arch/Kconfig
+++ b/arch/Kconfig
@@ -5,6 +5,9 @@
config KEXEC_CORE
bool
+config HOTPLUG_SMT
+ bool
+
config OPROFILE
tristate "OProfile system profiling"
depends on PROFILING
@@ -163,7 +166,7 @@
config KRETPROBES
def_bool y
- depends on KPROBES && HAVE_KRETPROBES
+ depends on KPROBES && HAVE_KRETPROBES && ROP_PROTECTION_NONE
config USER_RETURN_NOTIFIER
bool
@@ -571,6 +574,37 @@
If you select this option, the kernel builds a fast look-up table of
CFI check functions in loaded modules to reduce overhead.
+config ARCH_SUPPORTS_SHADOW_CALL_STACK
+ bool
+ help
+ An architecture should select this if it supports clang's Shadow
+ Call Stack, has asm/scs.h, and implements runtime support for shadow
+ stack switching.
+
+choice
+ prompt "Return-oriented programming (ROP) protection"
+ default ROP_PROTECTION_NONE
+ help
+ This option controls kernel protections against return-oriented
+ programming (ROP) attacks, which involve overwriting function return
+ addresses.
+
+config ROP_PROTECTION_NONE
+ bool "None"
+
+config SHADOW_CALL_STACK
+ bool "clang Shadow Call Stack (EXPERIMENTAL)"
+ depends on ARCH_SUPPORTS_SHADOW_CALL_STACK
+ help
+ This option enables clang's Shadow Call Stack, which uses a shadow
+ stack to protect function return addresses from being overwritten by
+ an attacker. More information can be found from clang's
+ documentation:
+
+ https://clang.llvm.org/docs/ShadowCallStack.html
+
+endchoice
+
config HAVE_ARCH_WITHIN_STACK_FRAMES
bool
help
diff --git a/arch/alpha/include/asm/futex.h b/arch/alpha/include/asm/futex.h
index f939794..5647469 100644
--- a/arch/alpha/include/asm/futex.h
+++ b/arch/alpha/include/asm/futex.h
@@ -29,18 +29,10 @@
: "r" (uaddr), "r"(oparg) \
: "memory")
-static inline int futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
+static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval,
+ u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
pagefault_disable();
@@ -66,17 +58,9 @@ static inline int futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/alpha/include/asm/termios.h b/arch/alpha/include/asm/termios.h
index 7fde0f8..51ed90b 100644
--- a/arch/alpha/include/asm/termios.h
+++ b/arch/alpha/include/asm/termios.h
@@ -72,9 +72,15 @@
})
#define user_termios_to_kernel_termios(k, u) \
- copy_from_user(k, u, sizeof(struct termios))
+ copy_from_user(k, u, sizeof(struct termios2))
#define kernel_termios_to_user_termios(u, k) \
+ copy_to_user(u, k, sizeof(struct termios2))
+
+#define user_termios_to_kernel_termios_1(k, u) \
+ copy_from_user(k, u, sizeof(struct termios))
+
+#define kernel_termios_to_user_termios_1(u, k) \
copy_to_user(u, k, sizeof(struct termios))
#endif /* _ALPHA_TERMIOS_H */
diff --git a/arch/alpha/include/asm/xchg.h b/arch/alpha/include/asm/xchg.h
index 0ca9724..7081e52 100644
--- a/arch/alpha/include/asm/xchg.h
+++ b/arch/alpha/include/asm/xchg.h
@@ -11,6 +11,10 @@
* Atomic exchange.
* Since it can be used to implement critical sections
* it must clobber "memory" (also for interrupts in UP).
+ *
+ * The leading and the trailing memory barriers guarantee that these
+ * operations are fully ordered.
+ *
*/
static inline unsigned long
@@ -18,6 +22,7 @@ ____xchg(_u8, volatile char *m, unsigned long val)
{
unsigned long ret, tmp, addr64;
+ smp_mb();
__asm__ __volatile__(
" andnot %4,7,%3\n"
" insbl %1,%4,%1\n"
@@ -42,6 +47,7 @@ ____xchg(_u16, volatile short *m, unsigned long val)
{
unsigned long ret, tmp, addr64;
+ smp_mb();
__asm__ __volatile__(
" andnot %4,7,%3\n"
" inswl %1,%4,%1\n"
@@ -66,6 +72,7 @@ ____xchg(_u32, volatile int *m, unsigned long val)
{
unsigned long dummy;
+ smp_mb();
__asm__ __volatile__(
"1: ldl_l %0,%4\n"
" bis $31,%3,%1\n"
@@ -86,6 +93,7 @@ ____xchg(_u64, volatile long *m, unsigned long val)
{
unsigned long dummy;
+ smp_mb();
__asm__ __volatile__(
"1: ldq_l %0,%4\n"
" bis $31,%3,%1\n"
@@ -127,10 +135,12 @@ ____xchg(, volatile void *ptr, unsigned long x, int size)
* store NEW in MEM. Return the initial value in MEM. Success is
* indicated by comparing RETURN with OLD.
*
- * The memory barrier should be placed in SMP only when we actually
- * make the change. If we don't change anything (so if the returned
- * prev is equal to old) then we aren't acquiring anything new and
- * we don't need any memory barrier as far I can tell.
+ * The leading and the trailing memory barriers guarantee that these
+ * operations are fully ordered.
+ *
+ * The trailing memory barrier is placed in SMP unconditionally, in
+ * order to guarantee that dependency ordering is preserved when a
+ * dependency is headed by an unsuccessful operation.
*/
static inline unsigned long
@@ -138,6 +148,7 @@ ____cmpxchg(_u8, volatile char *m, unsigned char old, unsigned char new)
{
unsigned long prev, tmp, cmp, addr64;
+ smp_mb();
__asm__ __volatile__(
" andnot %5,7,%4\n"
" insbl %1,%5,%1\n"
@@ -149,8 +160,8 @@ ____cmpxchg(_u8, volatile char *m, unsigned char old, unsigned char new)
" or %1,%2,%2\n"
" stq_c %2,0(%4)\n"
" beq %2,3f\n"
- __ASM__MB
"2:\n"
+ __ASM__MB
".subsection 2\n"
"3: br 1b\n"
".previous"
@@ -165,6 +176,7 @@ ____cmpxchg(_u16, volatile short *m, unsigned short old, unsigned short new)
{
unsigned long prev, tmp, cmp, addr64;
+ smp_mb();
__asm__ __volatile__(
" andnot %5,7,%4\n"
" inswl %1,%5,%1\n"
@@ -176,8 +188,8 @@ ____cmpxchg(_u16, volatile short *m, unsigned short old, unsigned short new)
" or %1,%2,%2\n"
" stq_c %2,0(%4)\n"
" beq %2,3f\n"
- __ASM__MB
"2:\n"
+ __ASM__MB
".subsection 2\n"
"3: br 1b\n"
".previous"
@@ -192,6 +204,7 @@ ____cmpxchg(_u32, volatile int *m, int old, int new)
{
unsigned long prev, cmp;
+ smp_mb();
__asm__ __volatile__(
"1: ldl_l %0,%5\n"
" cmpeq %0,%3,%1\n"
@@ -199,8 +212,8 @@ ____cmpxchg(_u32, volatile int *m, int old, int new)
" mov %4,%1\n"
" stl_c %1,%2\n"
" beq %1,3f\n"
- __ASM__MB
"2:\n"
+ __ASM__MB
".subsection 2\n"
"3: br 1b\n"
".previous"
@@ -215,6 +228,7 @@ ____cmpxchg(_u64, volatile long *m, unsigned long old, unsigned long new)
{
unsigned long prev, cmp;
+ smp_mb();
__asm__ __volatile__(
"1: ldq_l %0,%5\n"
" cmpeq %0,%3,%1\n"
@@ -222,8 +236,8 @@ ____cmpxchg(_u64, volatile long *m, unsigned long old, unsigned long new)
" mov %4,%1\n"
" stq_c %1,%2\n"
" beq %1,3f\n"
- __ASM__MB
"2:\n"
+ __ASM__MB
".subsection 2\n"
"3: br 1b\n"
".previous"
diff --git a/arch/alpha/include/uapi/asm/ioctls.h b/arch/alpha/include/uapi/asm/ioctls.h
index f30c94a..7ee8ab5 100644
--- a/arch/alpha/include/uapi/asm/ioctls.h
+++ b/arch/alpha/include/uapi/asm/ioctls.h
@@ -31,6 +31,11 @@
#define TCXONC _IO('t', 30)
#define TCFLSH _IO('t', 31)
+#define TCGETS2 _IOR('T', 42, struct termios2)
+#define TCSETS2 _IOW('T', 43, struct termios2)
+#define TCSETSW2 _IOW('T', 44, struct termios2)
+#define TCSETSF2 _IOW('T', 45, struct termios2)
+
#define TIOCSWINSZ _IOW('t', 103, struct winsize)
#define TIOCGWINSZ _IOR('t', 104, struct winsize)
#define TIOCSTART _IO('t', 110) /* start output, like ^Q */
diff --git a/arch/alpha/include/uapi/asm/termbits.h b/arch/alpha/include/uapi/asm/termbits.h
index 879dd35..483c7ec 100644
--- a/arch/alpha/include/uapi/asm/termbits.h
+++ b/arch/alpha/include/uapi/asm/termbits.h
@@ -25,6 +25,19 @@ struct termios {
speed_t c_ospeed; /* output speed */
};
+/* Alpha has identical termios and termios2 */
+
+struct termios2 {
+ tcflag_t c_iflag; /* input mode flags */
+ tcflag_t c_oflag; /* output mode flags */
+ tcflag_t c_cflag; /* control mode flags */
+ tcflag_t c_lflag; /* local mode flags */
+ cc_t c_cc[NCCS]; /* control characters */
+ cc_t c_line; /* line discipline (== c_cc[19]) */
+ speed_t c_ispeed; /* input speed */
+ speed_t c_ospeed; /* output speed */
+};
+
/* Alpha has matching termios and ktermios */
struct ktermios {
@@ -147,6 +160,7 @@ struct ktermios {
#define B3000000 00034
#define B3500000 00035
#define B4000000 00036
+#define BOTHER 00037
#define CSIZE 00001400
#define CS5 00000000
@@ -164,6 +178,9 @@ struct ktermios {
#define CMSPAR 010000000000 /* mark or space (stick) parity */
#define CRTSCTS 020000000000 /* flow control */
+#define CIBAUD 07600000
+#define IBSHIFT 16
+
/* c_lflag bits */
#define ISIG 0x00000080
#define ICANON 0x00000100
diff --git a/arch/alpha/kernel/osf_sys.c b/arch/alpha/kernel/osf_sys.c
index 4f95577..6e0d549 100644
--- a/arch/alpha/kernel/osf_sys.c
+++ b/arch/alpha/kernel/osf_sys.c
@@ -526,24 +526,19 @@ SYSCALL_DEFINE4(osf_mount, unsigned long, typenr, const char __user *, path,
SYSCALL_DEFINE1(osf_utsname, char __user *, name)
{
int error;
+ char tmp[5 * 32];
down_read(&uts_sem);
- error = -EFAULT;
- if (copy_to_user(name + 0, utsname()->sysname, 32))
- goto out;
- if (copy_to_user(name + 32, utsname()->nodename, 32))
- goto out;
- if (copy_to_user(name + 64, utsname()->release, 32))
- goto out;
- if (copy_to_user(name + 96, utsname()->version, 32))
- goto out;
- if (copy_to_user(name + 128, utsname()->machine, 32))
- goto out;
+ memcpy(tmp + 0 * 32, utsname()->sysname, 32);
+ memcpy(tmp + 1 * 32, utsname()->nodename, 32);
+ memcpy(tmp + 2 * 32, utsname()->release, 32);
+ memcpy(tmp + 3 * 32, utsname()->version, 32);
+ memcpy(tmp + 4 * 32, utsname()->machine, 32);
+ up_read(&uts_sem);
- error = 0;
- out:
- up_read(&uts_sem);
- return error;
+ if (copy_to_user(name, tmp, sizeof(tmp)))
+ return -EFAULT;
+ return 0;
}
SYSCALL_DEFINE0(getpagesize)
@@ -561,24 +556,22 @@ SYSCALL_DEFINE0(getdtablesize)
*/
SYSCALL_DEFINE2(osf_getdomainname, char __user *, name, int, namelen)
{
- unsigned len;
- int i;
+ int len, err = 0;
+ char *kname;
+ char tmp[32];
- if (!access_ok(VERIFY_WRITE, name, namelen))
- return -EFAULT;
-
- len = namelen;
- if (len > 32)
- len = 32;
+ if (namelen < 0 || namelen > 32)
+ namelen = 32;
down_read(&uts_sem);
- for (i = 0; i < len; ++i) {
- __put_user(utsname()->domainname[i], name + i);
- if (utsname()->domainname[i] == '\0')
- break;
- }
+ kname = utsname()->domainname;
+ len = strnlen(kname, namelen);
+ len = min(len + 1, namelen);
+ memcpy(tmp, kname, len);
up_read(&uts_sem);
+ if (copy_to_user(name, tmp, len))
+ return -EFAULT;
return 0;
}
@@ -741,13 +734,14 @@ SYSCALL_DEFINE3(osf_sysinfo, int, command, char __user *, buf, long, count)
};
unsigned long offset;
const char *res;
- long len, err = -EINVAL;
+ long len;
+ char tmp[__NEW_UTS_LEN + 1];
offset = command-1;
if (offset >= ARRAY_SIZE(sysinfo_table)) {
/* Digital UNIX has a few unpublished interfaces here */
printk("sysinfo(%d)", command);
- goto out;
+ return -EINVAL;
}
down_read(&uts_sem);
@@ -755,13 +749,11 @@ SYSCALL_DEFINE3(osf_sysinfo, int, command, char __user *, buf, long, count)
len = strlen(res)+1;
if ((unsigned long)len > (unsigned long)count)
len = count;
- if (copy_to_user(buf, res, len))
- err = -EFAULT;
- else
- err = 0;
+ memcpy(tmp, res, len);
up_read(&uts_sem);
- out:
- return err;
+ if (copy_to_user(buf, tmp, len))
+ return -EFAULT;
+ return 0;
}
SYSCALL_DEFINE5(osf_getsysinfo, unsigned long, op, void __user *, buffer,
diff --git a/arch/arc/Kconfig b/arch/arc/Kconfig
index 249e101..c7a081c 100644
--- a/arch/arc/Kconfig
+++ b/arch/arc/Kconfig
@@ -105,7 +105,7 @@
choice
prompt "ARC Instruction Set"
- default ISA_ARCOMPACT
+ default ISA_ARCV2
config ISA_ARCOMPACT
bool "ARCompact ISA"
@@ -495,7 +495,6 @@
config ARC_EMUL_UNALIGNED
bool "Emulate unaligned memory access (userspace only)"
- default N
select SYSCTL_ARCH_UNALIGN_NO_WARN
select SYSCTL_ARCH_UNALIGN_ALLOW
depends on ISA_ARCOMPACT
diff --git a/arch/arc/Makefile b/arch/arc/Makefile
index 19cce22..fd79faa 100644
--- a/arch/arc/Makefile
+++ b/arch/arc/Makefile
@@ -8,34 +8,12 @@
UTS_MACHINE := arc
-ifeq ($(CROSS_COMPILE),)
-ifndef CONFIG_CPU_BIG_ENDIAN
-CROSS_COMPILE := arc-linux-
-else
-CROSS_COMPILE := arceb-linux-
-endif
-endif
+KBUILD_DEFCONFIG := nsim_hs_defconfig
-KBUILD_DEFCONFIG := nsim_700_defconfig
-
-cflags-y += -fno-common -pipe -fno-builtin -D__linux__
+cflags-y += -fno-common -pipe -fno-builtin -mmedium-calls -D__linux__
cflags-$(CONFIG_ISA_ARCOMPACT) += -mA7
cflags-$(CONFIG_ISA_ARCV2) += -mcpu=archs
-is_700 = $(shell $(CC) -dM -E - < /dev/null | grep -q "ARC700" && echo 1 || echo 0)
-
-ifdef CONFIG_ISA_ARCOMPACT
-ifeq ($(is_700), 0)
- $(error Toolchain not configured for ARCompact builds)
-endif
-endif
-
-ifdef CONFIG_ISA_ARCV2
-ifeq ($(is_700), 1)
- $(error Toolchain not configured for ARCv2 builds)
-endif
-endif
-
ifdef CONFIG_ARC_CURR_IN_REG
# For a global register defintion, make sure it gets passed to every file
# We had a customer reported bug where some code built in kernel was NOT using
@@ -89,7 +67,7 @@
# --build-id w/o "-marclinux". Default arc-elf32-ld is OK
ldflags-$(upto_gcc44) += -marclinux
-LIBGCC := $(shell $(CC) $(cflags-y) --print-libgcc-file-name)
+LIBGCC = $(shell $(CC) $(cflags-y) --print-libgcc-file-name)
# Modules with short calls might break for calls into builtin-kernel
KBUILD_CFLAGS_MODULE += -mlong-calls -mno-millicode
@@ -141,16 +119,3 @@
archclean:
$(Q)$(MAKE) $(clean)=$(boot)
-
-# Hacks to enable final link due to absence of link-time branch relexation
-# and gcc choosing optimal(shorter) branches at -O3
-#
-# vineetg Feb 2010: -mlong-calls switched off for overall kernel build
-# However lib/decompress_inflate.o (.init.text) calls
-# zlib_inflate_workspacesize (.text) causing relocation errors.
-# Thus forcing all exten calls in this file to be long calls
-export CFLAGS_decompress_inflate.o = -mmedium-calls
-export CFLAGS_initramfs.o = -mmedium-calls
-ifdef CONFIG_SMP
-export CFLAGS_core.o = -mmedium-calls
-endif
diff --git a/arch/arc/configs/axs101_defconfig b/arch/arc/configs/axs101_defconfig
index 0a0eaf0..cdb00af 100644
--- a/arch/arc/configs/axs101_defconfig
+++ b/arch/arc/configs/axs101_defconfig
@@ -1,5 +1,4 @@
CONFIG_DEFAULT_HOSTNAME="ARCLinux"
-# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
# CONFIG_CROSS_MEMORY_ATTACH is not set
@@ -11,12 +10,12 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../arc_initramfs/"
CONFIG_EMBEDDED=y
CONFIG_PERF_EVENTS=y
# CONFIG_VM_EVENT_COUNTERS is not set
# CONFIG_SLUB_DEBUG is not set
# CONFIG_COMPAT_BRK is not set
+CONFIG_ISA_ARCOMPACT=y
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
CONFIG_MODULE_UNLOAD=y
@@ -98,6 +97,7 @@
CONFIG_NTFS_FS=y
CONFIG_TMPFS=y
CONFIG_NFS_FS=y
+CONFIG_NFS_V3_ACL=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
diff --git a/arch/arc/configs/axs103_defconfig b/arch/arc/configs/axs103_defconfig
index 2233f57..02c766d 100644
--- a/arch/arc/configs/axs103_defconfig
+++ b/arch/arc/configs/axs103_defconfig
@@ -1,5 +1,4 @@
CONFIG_DEFAULT_HOSTNAME="ARCLinux"
-# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
# CONFIG_CROSS_MEMORY_ATTACH is not set
@@ -11,7 +10,6 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../../arc_initramfs_hs/"
CONFIG_EMBEDDED=y
CONFIG_PERF_EVENTS=y
# CONFIG_VM_EVENT_COUNTERS is not set
@@ -99,6 +97,7 @@
CONFIG_NTFS_FS=y
CONFIG_TMPFS=y
CONFIG_NFS_FS=y
+CONFIG_NFS_V3_ACL=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
diff --git a/arch/arc/configs/axs103_smp_defconfig b/arch/arc/configs/axs103_smp_defconfig
index 1108747..8c16093 100644
--- a/arch/arc/configs/axs103_smp_defconfig
+++ b/arch/arc/configs/axs103_smp_defconfig
@@ -1,5 +1,4 @@
CONFIG_DEFAULT_HOSTNAME="ARCLinux"
-# CONFIG_SWAP is not set
CONFIG_SYSVIPC=y
CONFIG_POSIX_MQUEUE=y
# CONFIG_CROSS_MEMORY_ATTACH is not set
@@ -11,7 +10,6 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../../arc_initramfs_hs/"
CONFIG_EMBEDDED=y
CONFIG_PERF_EVENTS=y
# CONFIG_VM_EVENT_COUNTERS is not set
@@ -100,6 +98,7 @@
CONFIG_NTFS_FS=y
CONFIG_TMPFS=y
CONFIG_NFS_FS=y
+CONFIG_NFS_V3_ACL=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
diff --git a/arch/arc/configs/nps_defconfig b/arch/arc/configs/nps_defconfig
index ede625c..397742c 100644
--- a/arch/arc/configs/nps_defconfig
+++ b/arch/arc/configs/nps_defconfig
@@ -15,6 +15,7 @@
CONFIG_EMBEDDED=y
CONFIG_PERF_EVENTS=y
# CONFIG_COMPAT_BRK is not set
+CONFIG_ISA_ARCOMPACT=y
CONFIG_KPROBES=y
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
@@ -75,6 +76,7 @@
CONFIG_TMPFS=y
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_NFS_FS=y
+CONFIG_NFS_V3_ACL=y
CONFIG_ROOT_NFS=y
CONFIG_DEBUG_INFO=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
diff --git a/arch/arc/configs/nsim_700_defconfig b/arch/arc/configs/nsim_700_defconfig
index b0066a7..cbc6d06 100644
--- a/arch/arc/configs/nsim_700_defconfig
+++ b/arch/arc/configs/nsim_700_defconfig
@@ -11,12 +11,12 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../arc_initramfs/"
CONFIG_KALLSYMS_ALL=y
CONFIG_EMBEDDED=y
CONFIG_PERF_EVENTS=y
# CONFIG_SLUB_DEBUG is not set
# CONFIG_COMPAT_BRK is not set
+CONFIG_ISA_ARCOMPACT=y
CONFIG_KPROBES=y
CONFIG_MODULES=y
# CONFIG_LBDAF is not set
diff --git a/arch/arc/configs/nsim_hs_defconfig b/arch/arc/configs/nsim_hs_defconfig
index ebe9ebb..1dbb661 100644
--- a/arch/arc/configs/nsim_hs_defconfig
+++ b/arch/arc/configs/nsim_hs_defconfig
@@ -11,7 +11,6 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../../arc_initramfs_hs/"
CONFIG_KALLSYMS_ALL=y
CONFIG_EMBEDDED=y
CONFIG_PERF_EVENTS=y
diff --git a/arch/arc/configs/nsim_hs_smp_defconfig b/arch/arc/configs/nsim_hs_smp_defconfig
index 4bde432..cb36a69 100644
--- a/arch/arc/configs/nsim_hs_smp_defconfig
+++ b/arch/arc/configs/nsim_hs_smp_defconfig
@@ -9,7 +9,6 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../arc_initramfs_hs/"
CONFIG_KALLSYMS_ALL=y
CONFIG_EMBEDDED=y
CONFIG_PERF_EVENTS=y
diff --git a/arch/arc/configs/nsimosci_defconfig b/arch/arc/configs/nsimosci_defconfig
index f6fb3d26..d34b838 100644
--- a/arch/arc/configs/nsimosci_defconfig
+++ b/arch/arc/configs/nsimosci_defconfig
@@ -11,12 +11,12 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../arc_initramfs/"
CONFIG_KALLSYMS_ALL=y
CONFIG_EMBEDDED=y
CONFIG_PERF_EVENTS=y
# CONFIG_SLUB_DEBUG is not set
# CONFIG_COMPAT_BRK is not set
+CONFIG_ISA_ARCOMPACT=y
CONFIG_KPROBES=y
CONFIG_MODULES=y
# CONFIG_LBDAF is not set
@@ -71,5 +71,6 @@
CONFIG_TMPFS=y
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_NFS_FS=y
+CONFIG_NFS_V3_ACL=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
diff --git a/arch/arc/configs/nsimosci_hs_defconfig b/arch/arc/configs/nsimosci_hs_defconfig
index b9f0fe0..e8c7dd7 100644
--- a/arch/arc/configs/nsimosci_hs_defconfig
+++ b/arch/arc/configs/nsimosci_hs_defconfig
@@ -11,7 +11,6 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../arc_initramfs_hs/"
CONFIG_KALLSYMS_ALL=y
CONFIG_EMBEDDED=y
CONFIG_PERF_EVENTS=y
@@ -70,5 +69,6 @@
CONFIG_TMPFS=y
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_NFS_FS=y
+CONFIG_NFS_V3_ACL=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
diff --git a/arch/arc/configs/nsimosci_hs_smp_defconfig b/arch/arc/configs/nsimosci_hs_smp_defconfig
index 6da71ba..100d7bf 100644
--- a/arch/arc/configs/nsimosci_hs_smp_defconfig
+++ b/arch/arc/configs/nsimosci_hs_smp_defconfig
@@ -9,7 +9,6 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../arc_initramfs_hs/"
CONFIG_PERF_EVENTS=y
# CONFIG_COMPAT_BRK is not set
CONFIG_KPROBES=y
@@ -81,6 +80,7 @@
CONFIG_TMPFS=y
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_NFS_FS=y
+CONFIG_NFS_V3_ACL=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
# CONFIG_ENABLE_MUST_CHECK is not set
CONFIG_FTRACE=y
diff --git a/arch/arc/configs/tb10x_defconfig b/arch/arc/configs/tb10x_defconfig
index 4c51183..493966c 100644
--- a/arch/arc/configs/tb10x_defconfig
+++ b/arch/arc/configs/tb10x_defconfig
@@ -19,6 +19,7 @@
# CONFIG_AIO is not set
CONFIG_EMBEDDED=y
# CONFIG_COMPAT_BRK is not set
+CONFIG_ISA_ARCOMPACT=y
CONFIG_SLAB=y
CONFIG_MODULES=y
CONFIG_MODULE_FORCE_LOAD=y
diff --git a/arch/arc/configs/vdk_hs38_defconfig b/arch/arc/configs/vdk_hs38_defconfig
index c0d6a01..b1d38af 100644
--- a/arch/arc/configs/vdk_hs38_defconfig
+++ b/arch/arc/configs/vdk_hs38_defconfig
@@ -88,6 +88,7 @@
CONFIG_TMPFS=y
CONFIG_JFFS2_FS=y
CONFIG_NFS_FS=y
+CONFIG_NFS_V3_ACL=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
diff --git a/arch/arc/configs/vdk_hs38_smp_defconfig b/arch/arc/configs/vdk_hs38_smp_defconfig
index 969b206..2d103f7 100644
--- a/arch/arc/configs/vdk_hs38_smp_defconfig
+++ b/arch/arc/configs/vdk_hs38_smp_defconfig
@@ -87,6 +87,7 @@
CONFIG_TMPFS=y
CONFIG_JFFS2_FS=y
CONFIG_NFS_FS=y
+CONFIG_NFS_V3_ACL=y
CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
# CONFIG_ENABLE_WARN_DEPRECATED is not set
diff --git a/arch/arc/configs/zebu_hs_defconfig b/arch/arc/configs/zebu_hs_defconfig
index 9f6166b..3346829b 100644
--- a/arch/arc/configs/zebu_hs_defconfig
+++ b/arch/arc/configs/zebu_hs_defconfig
@@ -11,7 +11,6 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../../arc_initramfs_hs/"
CONFIG_EXPERT=y
CONFIG_PERF_EVENTS=y
# CONFIG_COMPAT_BRK is not set
diff --git a/arch/arc/configs/zebu_hs_smp_defconfig b/arch/arc/configs/zebu_hs_smp_defconfig
index 44e9693..4471f9c 100644
--- a/arch/arc/configs/zebu_hs_smp_defconfig
+++ b/arch/arc/configs/zebu_hs_smp_defconfig
@@ -11,7 +11,6 @@
# CONFIG_UTS_NS is not set
# CONFIG_PID_NS is not set
CONFIG_BLK_DEV_INITRD=y
-CONFIG_INITRAMFS_SOURCE="../../arc_initramfs_hs/"
CONFIG_EMBEDDED=y
CONFIG_PERF_EVENTS=y
# CONFIG_VM_EVENT_COUNTERS is not set
diff --git a/arch/arc/include/asm/atomic.h b/arch/arc/include/asm/atomic.h
index 54b54da..49112f7 100644
--- a/arch/arc/include/asm/atomic.h
+++ b/arch/arc/include/asm/atomic.h
@@ -84,7 +84,7 @@ static inline int atomic_fetch_##op(int i, atomic_t *v) \
"1: llock %[orig], [%[ctr]] \n" \
" " #asm_op " %[val], %[orig], %[i] \n" \
" scond %[val], [%[ctr]] \n" \
- " \n" \
+ " bnz 1b \n" \
: [val] "=&r" (val), \
[orig] "=&r" (orig) \
: [ctr] "r" (&v->counter), \
diff --git a/arch/arc/include/asm/delay.h b/arch/arc/include/asm/delay.h
index d5da211..03d6bb0 100644
--- a/arch/arc/include/asm/delay.h
+++ b/arch/arc/include/asm/delay.h
@@ -17,8 +17,11 @@
#ifndef __ASM_ARC_UDELAY_H
#define __ASM_ARC_UDELAY_H
+#include <asm-generic/types.h>
#include <asm/param.h> /* HZ */
+extern unsigned long loops_per_jiffy;
+
static inline void __delay(unsigned long loops)
{
__asm__ __volatile__(
diff --git a/arch/arc/include/asm/futex.h b/arch/arc/include/asm/futex.h
index 11e1b1f..eb887dd 100644
--- a/arch/arc/include/asm/futex.h
+++ b/arch/arc/include/asm/futex.h
@@ -73,20 +73,11 @@
#endif
-static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval,
+ u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(int)))
- return -EFAULT;
-
#ifndef CONFIG_ARC_HAS_LLSC
preempt_disable(); /* to guarantee atomic r-m-w of futex op */
#endif
@@ -118,30 +109,9 @@ static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
preempt_enable();
#endif
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ:
- ret = (oldval == cmparg);
- break;
- case FUTEX_OP_CMP_NE:
- ret = (oldval != cmparg);
- break;
- case FUTEX_OP_CMP_LT:
- ret = (oldval < cmparg);
- break;
- case FUTEX_OP_CMP_GE:
- ret = (oldval >= cmparg);
- break;
- case FUTEX_OP_CMP_LE:
- ret = (oldval <= cmparg);
- break;
- case FUTEX_OP_CMP_GT:
- ret = (oldval > cmparg);
- break;
- default:
- ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/arc/include/asm/io.h b/arch/arc/include/asm/io.h
index c22b181..2f39d9b 100644
--- a/arch/arc/include/asm/io.h
+++ b/arch/arc/include/asm/io.h
@@ -12,6 +12,7 @@
#include <linux/types.h>
#include <asm/byteorder.h>
#include <asm/page.h>
+#include <asm/unaligned.h>
#ifdef CONFIG_ISA_ARCV2
#include <asm/barrier.h>
@@ -94,6 +95,42 @@ static inline u32 __raw_readl(const volatile void __iomem *addr)
return w;
}
+/*
+ * {read,write}s{b,w,l}() repeatedly access the same IO address in
+ * native endianness in 8-, 16-, 32-bit chunks {into,from} memory,
+ * @count times
+ */
+#define __raw_readsx(t,f) \
+static inline void __raw_reads##f(const volatile void __iomem *addr, \
+ void *ptr, unsigned int count) \
+{ \
+ bool is_aligned = ((unsigned long)ptr % ((t) / 8)) == 0; \
+ u##t *buf = ptr; \
+ \
+ if (!count) \
+ return; \
+ \
+ /* Some ARC CPU's don't support unaligned accesses */ \
+ if (is_aligned) { \
+ do { \
+ u##t x = __raw_read##f(addr); \
+ *buf++ = x; \
+ } while (--count); \
+ } else { \
+ do { \
+ u##t x = __raw_read##f(addr); \
+ put_unaligned(x, buf++); \
+ } while (--count); \
+ } \
+}
+
+#define __raw_readsb __raw_readsb
+__raw_readsx(8, b)
+#define __raw_readsw __raw_readsw
+__raw_readsx(16, w)
+#define __raw_readsl __raw_readsl
+__raw_readsx(32, l)
+
#define __raw_writeb __raw_writeb
static inline void __raw_writeb(u8 b, volatile void __iomem *addr)
{
@@ -126,6 +163,35 @@ static inline void __raw_writel(u32 w, volatile void __iomem *addr)
}
+#define __raw_writesx(t,f) \
+static inline void __raw_writes##f(volatile void __iomem *addr, \
+ const void *ptr, unsigned int count) \
+{ \
+ bool is_aligned = ((unsigned long)ptr % ((t) / 8)) == 0; \
+ const u##t *buf = ptr; \
+ \
+ if (!count) \
+ return; \
+ \
+ /* Some ARC CPU's don't support unaligned accesses */ \
+ if (is_aligned) { \
+ do { \
+ __raw_write##f(*buf++, addr); \
+ } while (--count); \
+ } else { \
+ do { \
+ __raw_write##f(get_unaligned(buf++), addr); \
+ } while (--count); \
+ } \
+}
+
+#define __raw_writesb __raw_writesb
+__raw_writesx(8, b)
+#define __raw_writesw __raw_writesw
+__raw_writesx(16, w)
+#define __raw_writesl __raw_writesl
+__raw_writesx(32, l)
+
/*
* MMIO can also get buffered/optimized in micro-arch, so barriers needed
* Based on ARM model for the typical use case
@@ -141,10 +207,16 @@ static inline void __raw_writel(u32 w, volatile void __iomem *addr)
#define readb(c) ({ u8 __v = readb_relaxed(c); __iormb(); __v; })
#define readw(c) ({ u16 __v = readw_relaxed(c); __iormb(); __v; })
#define readl(c) ({ u32 __v = readl_relaxed(c); __iormb(); __v; })
+#define readsb(p,d,l) ({ __raw_readsb(p,d,l); __iormb(); })
+#define readsw(p,d,l) ({ __raw_readsw(p,d,l); __iormb(); })
+#define readsl(p,d,l) ({ __raw_readsl(p,d,l); __iormb(); })
#define writeb(v,c) ({ __iowmb(); writeb_relaxed(v,c); })
#define writew(v,c) ({ __iowmb(); writew_relaxed(v,c); })
#define writel(v,c) ({ __iowmb(); writel_relaxed(v,c); })
+#define writesb(p,d,l) ({ __iowmb(); __raw_writesb(p,d,l); })
+#define writesw(p,d,l) ({ __iowmb(); __raw_writesw(p,d,l); })
+#define writesl(p,d,l) ({ __iowmb(); __raw_writesl(p,d,l); })
/*
* Relaxed API for drivers which can handle barrier ordering themselves
diff --git a/arch/arc/include/asm/mach_desc.h b/arch/arc/include/asm/mach_desc.h
index c28e6c3..871f3cb 100644
--- a/arch/arc/include/asm/mach_desc.h
+++ b/arch/arc/include/asm/mach_desc.h
@@ -34,9 +34,7 @@ struct machine_desc {
const char *name;
const char **dt_compat;
void (*init_early)(void);
-#ifdef CONFIG_SMP
void (*init_per_cpu)(unsigned int);
-#endif
void (*init_machine)(void);
void (*init_late)(void);
diff --git a/arch/arc/include/asm/page.h b/arch/arc/include/asm/page.h
index 296c342..ffb5f33 100644
--- a/arch/arc/include/asm/page.h
+++ b/arch/arc/include/asm/page.h
@@ -105,7 +105,7 @@ typedef pte_t * pgtable_t;
#define virt_addr_valid(kaddr) pfn_valid(virt_to_pfn(kaddr))
/* Default Permissions for stack/heaps pages (Non Executable) */
-#define VM_DATA_DEFAULT_FLAGS (VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE)
+#define VM_DATA_DEFAULT_FLAGS (VM_READ | VM_WRITE | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC)
#define WANT_PAGE_VIRTUAL 1
diff --git a/arch/arc/include/asm/pgtable.h b/arch/arc/include/asm/pgtable.h
index e94ca72..c10f5cb 100644
--- a/arch/arc/include/asm/pgtable.h
+++ b/arch/arc/include/asm/pgtable.h
@@ -378,7 +378,7 @@ void update_mmu_cache(struct vm_area_struct *vma, unsigned long address,
/* Decode a PTE containing swap "identifier "into constituents */
#define __swp_type(pte_lookalike) (((pte_lookalike).val) & 0x1f)
-#define __swp_offset(pte_lookalike) ((pte_lookalike).val << 13)
+#define __swp_offset(pte_lookalike) ((pte_lookalike).val >> 13)
/* NOPs, to keep generic kernel happy */
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
diff --git a/arch/arc/kernel/irq.c b/arch/arc/kernel/irq.c
index 538b36a..62b1850 100644
--- a/arch/arc/kernel/irq.c
+++ b/arch/arc/kernel/irq.c
@@ -31,10 +31,10 @@ void __init init_IRQ(void)
/* a SMP H/w block could do IPI IRQ request here */
if (plat_smp_ops.init_per_cpu)
plat_smp_ops.init_per_cpu(smp_processor_id());
+#endif
if (machine_desc->init_per_cpu)
machine_desc->init_per_cpu(smp_processor_id());
-#endif
}
/*
diff --git a/arch/arc/kernel/process.c b/arch/arc/kernel/process.c
index a41a79a..3ce1213 100644
--- a/arch/arc/kernel/process.c
+++ b/arch/arc/kernel/process.c
@@ -44,7 +44,8 @@ SYSCALL_DEFINE0(arc_gettls)
SYSCALL_DEFINE3(arc_usr_cmpxchg, int *, uaddr, int, expected, int, new)
{
struct pt_regs *regs = current_pt_regs();
- int uval = -EFAULT;
+ u32 uval;
+ int ret;
/*
* This is only for old cores lacking LLOCK/SCOND, which by defintion
@@ -57,23 +58,47 @@ SYSCALL_DEFINE3(arc_usr_cmpxchg, int *, uaddr, int, expected, int, new)
/* Z indicates to userspace if operation succeded */
regs->status32 &= ~STATUS_Z_MASK;
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(int)))
- return -EFAULT;
+ ret = access_ok(VERIFY_WRITE, uaddr, sizeof(*uaddr));
+ if (!ret)
+ goto fail;
+again:
preempt_disable();
- if (__get_user(uval, uaddr))
- goto done;
+ ret = __get_user(uval, uaddr);
+ if (ret)
+ goto fault;
- if (uval == expected) {
- if (!__put_user(new, uaddr))
- regs->status32 |= STATUS_Z_MASK;
- }
+ if (uval != expected)
+ goto out;
-done:
+ ret = __put_user(new, uaddr);
+ if (ret)
+ goto fault;
+
+ regs->status32 |= STATUS_Z_MASK;
+
+out:
+ preempt_enable();
+ return uval;
+
+fault:
preempt_enable();
- return uval;
+ if (unlikely(ret != -EFAULT))
+ goto fail;
+
+ down_read(¤t->mm->mmap_sem);
+ ret = fixup_user_fault(current, current->mm, (unsigned long) uaddr,
+ FAULT_FLAG_WRITE, NULL);
+ up_read(¤t->mm->mmap_sem);
+
+ if (likely(!ret))
+ goto again;
+
+fail:
+ force_sig(SIGSEGV, current);
+ return ret;
}
void arch_cpu_idle(void)
@@ -188,6 +213,26 @@ int copy_thread(unsigned long clone_flags,
task_thread_info(current)->thr_ptr;
}
+
+ /*
+ * setup usermode thread pointer #1:
+ * when child is picked by scheduler, __switch_to() uses @c_callee to
+ * populate usermode callee regs: this works (despite being in a kernel
+ * function) since special return path for child @ret_from_fork()
+ * ensures those regs are not clobbered all the way to RTIE to usermode
+ */
+ c_callee->r25 = task_thread_info(p)->thr_ptr;
+
+#ifdef CONFIG_ARC_CURR_IN_REG
+ /*
+ * setup usermode thread pointer #2:
+ * however for this special use of r25 in kernel, __switch_to() sets
+ * r25 for kernel needs and only in the final return path is usermode
+ * r25 setup, from pt_regs->user_r25. So set that up as well
+ */
+ c_regs->user_r25 = c_callee->r25;
+#endif
+
return 0;
}
diff --git a/arch/arc/mm/cache.c b/arch/arc/mm/cache.c
index bbdfeb3..fefe357 100644
--- a/arch/arc/mm/cache.c
+++ b/arch/arc/mm/cache.c
@@ -840,7 +840,7 @@ void flush_cache_mm(struct mm_struct *mm)
void flush_cache_page(struct vm_area_struct *vma, unsigned long u_vaddr,
unsigned long pfn)
{
- unsigned int paddr = pfn << PAGE_SHIFT;
+ phys_addr_t paddr = pfn << PAGE_SHIFT;
u_vaddr &= PAGE_MASK;
@@ -860,8 +860,9 @@ void flush_anon_page(struct vm_area_struct *vma, struct page *page,
unsigned long u_vaddr)
{
/* TBD: do we really need to clear the kernel mapping */
- __flush_dcache_page(page_address(page), u_vaddr);
- __flush_dcache_page(page_address(page), page_address(page));
+ __flush_dcache_page((phys_addr_t)page_address(page), u_vaddr);
+ __flush_dcache_page((phys_addr_t)page_address(page),
+ (phys_addr_t)page_address(page));
}
diff --git a/arch/arc/plat-eznps/include/plat/ctop.h b/arch/arc/plat-eznps/include/plat/ctop.h
index 9d6718c..3c401ce 100644
--- a/arch/arc/plat-eznps/include/plat/ctop.h
+++ b/arch/arc/plat-eznps/include/plat/ctop.h
@@ -21,6 +21,7 @@
#error "Incorrect ctop.h include"
#endif
+#include <linux/types.h>
#include <soc/nps/common.h>
/* core auxiliary registers */
diff --git a/arch/arm/Makefile b/arch/arm/Makefile
index f66f377..82945ce 100644
--- a/arch/arm/Makefile
+++ b/arch/arm/Makefile
@@ -108,7 +108,7 @@
tune-y := $(tune-y)
ifeq ($(CONFIG_AEABI),y)
-CFLAGS_ABI :=-mabi=aapcs-linux -mno-thumb-interwork -mfpu=vfp
+CFLAGS_ABI :=-mabi=aapcs-linux -mfpu=vfp
else
CFLAGS_ABI :=$(call cc-option,-mapcs-32,-mabi=apcs-gnu) $(call cc-option,-mno-thumb-interwork,)
endif
diff --git a/arch/arm/boot/compressed/Makefile b/arch/arm/boot/compressed/Makefile
index d50430c..552c7d7 100644
--- a/arch/arm/boot/compressed/Makefile
+++ b/arch/arm/boot/compressed/Makefile
@@ -112,7 +112,7 @@
CFLAGS_fdt_rw.o := $(nossp_flags)
CFLAGS_fdt_wip.o := $(nossp_flags)
-ccflags-y := -fpic -mno-single-pic-base -fno-builtin -I$(obj)
+ccflags-y := -fpic $(call cc-option,-mno-single-pic-base,) -fno-builtin -I$(obj)
asflags-y := -DZIMAGE
# Supply kernel BSS size to the decompressor via a linker symbol.
diff --git a/arch/arm/boot/compressed/efi-header.S b/arch/arm/boot/compressed/efi-header.S
index 9d5dc4f..3f7d1b7 100644
--- a/arch/arm/boot/compressed/efi-header.S
+++ b/arch/arm/boot/compressed/efi-header.S
@@ -17,14 +17,12 @@
@ there.
.inst 'M' | ('Z' << 8) | (0x1310 << 16) @ tstne r0, #0x4d000
#else
- mov r0, r0
+ W(mov) r0, r0
#endif
.endm
.macro __EFI_HEADER
#ifdef CONFIG_EFI_STUB
- b __efi_start
-
.set start_offset, __efi_start - start
.org start + 0x3c
@
diff --git a/arch/arm/boot/compressed/head.S b/arch/arm/boot/compressed/head.S
index 51fc9fb..b114faa 100644
--- a/arch/arm/boot/compressed/head.S
+++ b/arch/arm/boot/compressed/head.S
@@ -130,19 +130,22 @@
.rept 7
__nop
.endr
- ARM( mov r0, r0 )
- ARM( b 1f )
- THUMB( badr r12, 1f )
- THUMB( bx r12 )
+#ifndef CONFIG_THUMB2_KERNEL
+ mov r0, r0
+#else
+ AR_CLASS( sub pc, pc, #3 ) @ A/R: switch to Thumb2 mode
+ M_CLASS( nop.w ) @ M: already in Thumb2 mode
+ .thumb
+#endif
+ W(b) 1f
.word _magic_sig @ Magic numbers to help the loader
.word _magic_start @ absolute load/run zImage address
.word _magic_end @ zImage end address
.word 0x04030201 @ endianness flag
- THUMB( .thumb )
-1: __EFI_HEADER
-
+ __EFI_HEADER
+1:
ARM_BE8( setend be ) @ go BE8 if compiled for BE8
AR_CLASS( mrs r9, cpsr )
#ifdef CONFIG_ARM_VIRT_EXT
diff --git a/arch/arm/boot/dts/am3517.dtsi b/arch/arm/boot/dts/am3517.dtsi
index 0db19d3..d022b6b 100644
--- a/arch/arm/boot/dts/am3517.dtsi
+++ b/arch/arm/boot/dts/am3517.dtsi
@@ -74,6 +74,11 @@
};
};
+/* Table Table 5-79 of the TRM shows 480ab000 is reserved */
+&usb_otg_hs {
+ status = "disabled";
+};
+
&iva {
status = "disabled";
};
diff --git a/arch/arm/boot/dts/am437x-sk-evm.dts b/arch/arm/boot/dts/am437x-sk-evm.dts
index 319d942..6482ada 100644
--- a/arch/arm/boot/dts/am437x-sk-evm.dts
+++ b/arch/arm/boot/dts/am437x-sk-evm.dts
@@ -533,6 +533,8 @@
touchscreen-size-x = <480>;
touchscreen-size-y = <272>;
+
+ wakeup-source;
};
tlv320aic3106: tlv320aic3106@1b {
diff --git a/arch/arm/boot/dts/bcm-cygnus.dtsi b/arch/arm/boot/dts/bcm-cygnus.dtsi
index fabc9f3..5ad6153 100644
--- a/arch/arm/boot/dts/bcm-cygnus.dtsi
+++ b/arch/arm/boot/dts/bcm-cygnus.dtsi
@@ -128,7 +128,7 @@
reg = <0x18008000 0x100>;
#address-cells = <1>;
#size-cells = <0>;
- interrupts = <GIC_SPI 85 IRQ_TYPE_NONE>;
+ interrupts = <GIC_SPI 85 IRQ_TYPE_LEVEL_HIGH>;
clock-frequency = <100000>;
status = "disabled";
};
@@ -157,7 +157,7 @@
reg = <0x1800b000 0x100>;
#address-cells = <1>;
#size-cells = <0>;
- interrupts = <GIC_SPI 86 IRQ_TYPE_NONE>;
+ interrupts = <GIC_SPI 86 IRQ_TYPE_LEVEL_HIGH>;
clock-frequency = <100000>;
status = "disabled";
};
@@ -168,7 +168,7 @@
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 0>;
- interrupt-map = <0 0 0 0 &gic GIC_SPI 100 IRQ_TYPE_NONE>;
+ interrupt-map = <0 0 0 0 &gic GIC_SPI 100 IRQ_TYPE_LEVEL_HIGH>;
linux,pci-domain = <0>;
@@ -190,10 +190,10 @@
compatible = "brcm,iproc-msi";
msi-controller;
interrupt-parent = <&gic>;
- interrupts = <GIC_SPI 96 IRQ_TYPE_NONE>,
- <GIC_SPI 97 IRQ_TYPE_NONE>,
- <GIC_SPI 98 IRQ_TYPE_NONE>,
- <GIC_SPI 99 IRQ_TYPE_NONE>;
+ interrupts = <GIC_SPI 96 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 97 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 98 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 99 IRQ_TYPE_LEVEL_HIGH>;
};
};
@@ -203,7 +203,7 @@
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 0>;
- interrupt-map = <0 0 0 0 &gic GIC_SPI 106 IRQ_TYPE_NONE>;
+ interrupt-map = <0 0 0 0 &gic GIC_SPI 106 IRQ_TYPE_LEVEL_HIGH>;
linux,pci-domain = <1>;
@@ -225,10 +225,10 @@
compatible = "brcm,iproc-msi";
msi-controller;
interrupt-parent = <&gic>;
- interrupts = <GIC_SPI 102 IRQ_TYPE_NONE>,
- <GIC_SPI 103 IRQ_TYPE_NONE>,
- <GIC_SPI 104 IRQ_TYPE_NONE>,
- <GIC_SPI 105 IRQ_TYPE_NONE>;
+ interrupts = <GIC_SPI 102 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 103 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 104 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 105 IRQ_TYPE_LEVEL_HIGH>;
};
};
diff --git a/arch/arm/boot/dts/bcm-nsp.dtsi b/arch/arm/boot/dts/bcm-nsp.dtsi
index 65e0db1..6e3d3b5 100644
--- a/arch/arm/boot/dts/bcm-nsp.dtsi
+++ b/arch/arm/boot/dts/bcm-nsp.dtsi
@@ -288,7 +288,7 @@
reg = <0x38000 0x50>;
#address-cells = <1>;
#size-cells = <0>;
- interrupts = <GIC_SPI 89 IRQ_TYPE_NONE>;
+ interrupts = <GIC_SPI 89 IRQ_TYPE_LEVEL_HIGH>;
clock-frequency = <100000>;
};
@@ -375,7 +375,7 @@
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 0>;
- interrupt-map = <0 0 0 0 &gic GIC_SPI 131 IRQ_TYPE_NONE>;
+ interrupt-map = <0 0 0 0 &gic GIC_SPI 131 IRQ_TYPE_LEVEL_HIGH>;
linux,pci-domain = <0>;
@@ -397,10 +397,10 @@
compatible = "brcm,iproc-msi";
msi-controller;
interrupt-parent = <&gic>;
- interrupts = <GIC_SPI 127 IRQ_TYPE_NONE>,
- <GIC_SPI 128 IRQ_TYPE_NONE>,
- <GIC_SPI 129 IRQ_TYPE_NONE>,
- <GIC_SPI 130 IRQ_TYPE_NONE>;
+ interrupts = <GIC_SPI 127 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 128 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 129 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 130 IRQ_TYPE_LEVEL_HIGH>;
brcm,pcie-msi-inten;
};
};
@@ -411,7 +411,7 @@
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 0>;
- interrupt-map = <0 0 0 0 &gic GIC_SPI 137 IRQ_TYPE_NONE>;
+ interrupt-map = <0 0 0 0 &gic GIC_SPI 137 IRQ_TYPE_LEVEL_HIGH>;
linux,pci-domain = <1>;
@@ -433,10 +433,10 @@
compatible = "brcm,iproc-msi";
msi-controller;
interrupt-parent = <&gic>;
- interrupts = <GIC_SPI 133 IRQ_TYPE_NONE>,
- <GIC_SPI 134 IRQ_TYPE_NONE>,
- <GIC_SPI 135 IRQ_TYPE_NONE>,
- <GIC_SPI 136 IRQ_TYPE_NONE>;
+ interrupts = <GIC_SPI 133 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 135 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 136 IRQ_TYPE_LEVEL_HIGH>;
brcm,pcie-msi-inten;
};
};
@@ -447,7 +447,7 @@
#interrupt-cells = <1>;
interrupt-map-mask = <0 0 0 0>;
- interrupt-map = <0 0 0 0 &gic GIC_SPI 143 IRQ_TYPE_NONE>;
+ interrupt-map = <0 0 0 0 &gic GIC_SPI 143 IRQ_TYPE_LEVEL_HIGH>;
linux,pci-domain = <2>;
@@ -469,10 +469,10 @@
compatible = "brcm,iproc-msi";
msi-controller;
interrupt-parent = <&gic>;
- interrupts = <GIC_SPI 139 IRQ_TYPE_NONE>,
- <GIC_SPI 140 IRQ_TYPE_NONE>,
- <GIC_SPI 141 IRQ_TYPE_NONE>,
- <GIC_SPI 142 IRQ_TYPE_NONE>;
+ interrupts = <GIC_SPI 139 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 140 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 141 IRQ_TYPE_LEVEL_HIGH>,
+ <GIC_SPI 142 IRQ_TYPE_LEVEL_HIGH>;
brcm,pcie-msi-inten;
};
};
diff --git a/arch/arm/boot/dts/bcm283x.dtsi b/arch/arm/boot/dts/bcm283x.dtsi
index 74dd21b..3156300 100644
--- a/arch/arm/boot/dts/bcm283x.dtsi
+++ b/arch/arm/boot/dts/bcm283x.dtsi
@@ -3,6 +3,11 @@
#include <dt-bindings/clock/bcm2835-aux.h>
#include <dt-bindings/gpio/gpio.h>
+/* firmware-provided startup stubs live here, where the secondary CPUs are
+ * spinning.
+ */
+/memreserve/ 0x00000000 0x00001000;
+
/* This include file covers the common peripherals and configuration between
* bcm2835 and bcm2836 implementations, leaving the CPU configuration to
* bcm2835.dtsi and bcm2836.dtsi.
@@ -146,8 +151,8 @@
i2s: i2s@7e203000 {
compatible = "brcm,bcm2835-i2s";
- reg = <0x7e203000 0x20>,
- <0x7e101098 0x02>;
+ reg = <0x7e203000 0x24>;
+ clocks = <&clocks BCM2835_CLOCK_PCM>;
dmas = <&dma 2>,
<&dma 3>;
diff --git a/arch/arm/boot/dts/bcm63138.dtsi b/arch/arm/boot/dts/bcm63138.dtsi
index d0560e8..547369c 100644
--- a/arch/arm/boot/dts/bcm63138.dtsi
+++ b/arch/arm/boot/dts/bcm63138.dtsi
@@ -105,21 +105,23 @@
global_timer: timer@1e200 {
compatible = "arm,cortex-a9-global-timer";
reg = <0x1e200 0x20>;
- interrupts = <GIC_PPI 11 IRQ_TYPE_LEVEL_HIGH>;
+ interrupts = <GIC_PPI 11 IRQ_TYPE_EDGE_RISING>;
clocks = <&axi_clk>;
};
local_timer: local-timer@1e600 {
compatible = "arm,cortex-a9-twd-timer";
reg = <0x1e600 0x20>;
- interrupts = <GIC_PPI 13 IRQ_TYPE_LEVEL_HIGH>;
+ interrupts = <GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(2) |
+ IRQ_TYPE_EDGE_RISING)>;
clocks = <&axi_clk>;
};
twd_watchdog: watchdog@1e620 {
compatible = "arm,cortex-a9-twd-wdt";
reg = <0x1e620 0x20>;
- interrupts = <GIC_PPI 14 IRQ_TYPE_LEVEL_HIGH>;
+ interrupts = <GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(2) |
+ IRQ_TYPE_LEVEL_HIGH)>;
};
armpll: armpll {
@@ -157,7 +159,7 @@
serial0: serial@600 {
compatible = "brcm,bcm6345-uart";
reg = <0x600 0x1b>;
- interrupts = <GIC_SPI 32 0>;
+ interrupts = <GIC_SPI 32 IRQ_TYPE_LEVEL_HIGH>;
clocks = <&periph_clk>;
clock-names = "periph";
status = "disabled";
@@ -166,7 +168,7 @@
serial1: serial@620 {
compatible = "brcm,bcm6345-uart";
reg = <0x620 0x1b>;
- interrupts = <GIC_SPI 33 0>;
+ interrupts = <GIC_SPI 33 IRQ_TYPE_LEVEL_HIGH>;
clocks = <&periph_clk>;
clock-names = "periph";
status = "disabled";
@@ -179,7 +181,7 @@
reg = <0x2000 0x600>, <0xf0 0x10>;
reg-names = "nand", "nand-int-base";
status = "disabled";
- interrupts = <GIC_SPI 38 0>;
+ interrupts = <GIC_SPI 38 IRQ_TYPE_LEVEL_HIGH>;
interrupt-names = "nand";
};
diff --git a/arch/arm/boot/dts/bcm958625hr.dts b/arch/arm/boot/dts/bcm958625hr.dts
index a1658d0..cf0de77 100644
--- a/arch/arm/boot/dts/bcm958625hr.dts
+++ b/arch/arm/boot/dts/bcm958625hr.dts
@@ -49,7 +49,7 @@
memory {
device_type = "memory";
- reg = <0x60000000 0x80000000>;
+ reg = <0x60000000 0x20000000>;
};
gpio-restart {
diff --git a/arch/arm/boot/dts/da850.dtsi b/arch/arm/boot/dts/da850.dtsi
index f79e1b9..51ab92f 100644
--- a/arch/arm/boot/dts/da850.dtsi
+++ b/arch/arm/boot/dts/da850.dtsi
@@ -377,11 +377,7 @@
gpio-controller;
#gpio-cells = <2>;
reg = <0x226000 0x1000>;
- interrupts = <42 IRQ_TYPE_EDGE_BOTH
- 43 IRQ_TYPE_EDGE_BOTH 44 IRQ_TYPE_EDGE_BOTH
- 45 IRQ_TYPE_EDGE_BOTH 46 IRQ_TYPE_EDGE_BOTH
- 47 IRQ_TYPE_EDGE_BOTH 48 IRQ_TYPE_EDGE_BOTH
- 49 IRQ_TYPE_EDGE_BOTH 50 IRQ_TYPE_EDGE_BOTH>;
+ interrupts = <42 43 44 45 46 47 48 49 50>;
ti,ngpio = <144>;
ti,davinci-gpio-unbanked = <0>;
status = "disabled";
diff --git a/arch/arm/boot/dts/dra7.dtsi b/arch/arm/boot/dts/dra7.dtsi
index ce54a70..a1a9280 100644
--- a/arch/arm/boot/dts/dra7.dtsi
+++ b/arch/arm/boot/dts/dra7.dtsi
@@ -1770,7 +1770,7 @@
};
};
- dcan1: can@481cc000 {
+ dcan1: can@4ae3c000 {
compatible = "ti,dra7-d_can";
ti,hwmods = "dcan1";
reg = <0x4ae3c000 0x2000>;
@@ -1780,7 +1780,7 @@
status = "disabled";
};
- dcan2: can@481d0000 {
+ dcan2: can@48480000 {
compatible = "ti,dra7-d_can";
ti,hwmods = "dcan2";
reg = <0x48480000 0x2000>;
diff --git a/arch/arm/boot/dts/emev2.dtsi b/arch/arm/boot/dts/emev2.dtsi
index cd11940..fd6f9ce 100644
--- a/arch/arm/boot/dts/emev2.dtsi
+++ b/arch/arm/boot/dts/emev2.dtsi
@@ -30,13 +30,13 @@
#address-cells = <1>;
#size-cells = <0>;
- cpu@0 {
+ cpu0: cpu@0 {
device_type = "cpu";
compatible = "arm,cortex-a9";
reg = <0>;
clock-frequency = <533000000>;
};
- cpu@1 {
+ cpu1: cpu@1 {
device_type = "cpu";
compatible = "arm,cortex-a9";
reg = <1>;
@@ -56,6 +56,7 @@
compatible = "arm,cortex-a9-pmu";
interrupts = <GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 121 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-affinity = <&cpu0>, <&cpu1>;
};
clocks@e0110000 {
diff --git a/arch/arm/boot/dts/exynos3250.dtsi b/arch/arm/boot/dts/exynos3250.dtsi
index e9d2556..2a531be 100644
--- a/arch/arm/boot/dts/exynos3250.dtsi
+++ b/arch/arm/boot/dts/exynos3250.dtsi
@@ -80,6 +80,22 @@
compatible = "arm,cortex-a7";
reg = <1>;
clock-frequency = <1000000000>;
+ clocks = <&cmu CLK_ARM_CLK>;
+ clock-names = "cpu";
+ #cooling-cells = <2>;
+
+ operating-points = <
+ 1000000 1150000
+ 900000 1112500
+ 800000 1075000
+ 700000 1037500
+ 600000 1000000
+ 500000 962500
+ 400000 925000
+ 300000 887500
+ 200000 850000
+ 100000 850000
+ >;
};
};
diff --git a/arch/arm/boot/dts/exynos4210.dtsi b/arch/arm/boot/dts/exynos4210.dtsi
index 2d9b029..b0c550e 100644
--- a/arch/arm/boot/dts/exynos4210.dtsi
+++ b/arch/arm/boot/dts/exynos4210.dtsi
@@ -52,8 +52,6 @@
400000 975000
200000 950000
>;
- cooling-min-level = <4>;
- cooling-max-level = <2>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -61,6 +59,19 @@
device_type = "cpu";
compatible = "arm,cortex-a9";
reg = <0x901>;
+ clocks = <&clock CLK_ARM_CLK>;
+ clock-names = "cpu";
+ clock-latency = <160000>;
+
+ operating-points = <
+ 1200000 1250000
+ 1000000 1150000
+ 800000 1075000
+ 500000 975000
+ 400000 975000
+ 200000 950000
+ >;
+ #cooling-cells = <2>; /* min followed by max */
};
};
diff --git a/arch/arm/boot/dts/exynos4412.dtsi b/arch/arm/boot/dts/exynos4412.dtsi
index 3ebdf01..63b1c5a 100644
--- a/arch/arm/boot/dts/exynos4412.dtsi
+++ b/arch/arm/boot/dts/exynos4412.dtsi
@@ -33,8 +33,6 @@
clocks = <&clock CLK_ARM_CLK>;
clock-names = "cpu";
operating-points-v2 = <&cpu0_opp_table>;
- cooling-min-level = <13>;
- cooling-max-level = <7>;
#cooling-cells = <2>; /* min followed by max */
};
diff --git a/arch/arm/boot/dts/exynos5250.dtsi b/arch/arm/boot/dts/exynos5250.dtsi
index 64de33d..ecc73f26 100644
--- a/arch/arm/boot/dts/exynos5250.dtsi
+++ b/arch/arm/boot/dts/exynos5250.dtsi
@@ -57,38 +57,106 @@
device_type = "cpu";
compatible = "arm,cortex-a15";
reg = <0>;
- clock-frequency = <1700000000>;
clocks = <&clock CLK_ARM_CLK>;
clock-names = "cpu";
- clock-latency = <140000>;
-
- operating-points = <
- 1700000 1300000
- 1600000 1250000
- 1500000 1225000
- 1400000 1200000
- 1300000 1150000
- 1200000 1125000
- 1100000 1100000
- 1000000 1075000
- 900000 1050000
- 800000 1025000
- 700000 1012500
- 600000 1000000
- 500000 975000
- 400000 950000
- 300000 937500
- 200000 925000
- >;
- cooling-min-level = <15>;
- cooling-max-level = <9>;
+ operating-points-v2 = <&cpu0_opp_table>;
#cooling-cells = <2>; /* min followed by max */
};
cpu@1 {
device_type = "cpu";
compatible = "arm,cortex-a15";
reg = <1>;
- clock-frequency = <1700000000>;
+ clocks = <&clock CLK_ARM_CLK>;
+ clock-names = "cpu";
+ operating-points-v2 = <&cpu0_opp_table>;
+ #cooling-cells = <2>; /* min followed by max */
+ };
+ };
+
+ cpu0_opp_table: opp_table0 {
+ compatible = "operating-points-v2";
+ opp-shared;
+
+ opp-200000000 {
+ opp-hz = /bits/ 64 <200000000>;
+ opp-microvolt = <925000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-300000000 {
+ opp-hz = /bits/ 64 <300000000>;
+ opp-microvolt = <937500>;
+ clock-latency-ns = <140000>;
+ };
+ opp-400000000 {
+ opp-hz = /bits/ 64 <400000000>;
+ opp-microvolt = <950000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-500000000 {
+ opp-hz = /bits/ 64 <500000000>;
+ opp-microvolt = <975000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-600000000 {
+ opp-hz = /bits/ 64 <600000000>;
+ opp-microvolt = <1000000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-700000000 {
+ opp-hz = /bits/ 64 <700000000>;
+ opp-microvolt = <1012500>;
+ clock-latency-ns = <140000>;
+ };
+ opp-800000000 {
+ opp-hz = /bits/ 64 <800000000>;
+ opp-microvolt = <1025000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-900000000 {
+ opp-hz = /bits/ 64 <900000000>;
+ opp-microvolt = <1050000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-1000000000 {
+ opp-hz = /bits/ 64 <1000000000>;
+ opp-microvolt = <1075000>;
+ clock-latency-ns = <140000>;
+ opp-suspend;
+ };
+ opp-1100000000 {
+ opp-hz = /bits/ 64 <1100000000>;
+ opp-microvolt = <1100000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-1200000000 {
+ opp-hz = /bits/ 64 <1200000000>;
+ opp-microvolt = <1125000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-1300000000 {
+ opp-hz = /bits/ 64 <1300000000>;
+ opp-microvolt = <1150000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-1400000000 {
+ opp-hz = /bits/ 64 <1400000000>;
+ opp-microvolt = <1200000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-1500000000 {
+ opp-hz = /bits/ 64 <1500000000>;
+ opp-microvolt = <1225000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-1600000000 {
+ opp-hz = /bits/ 64 <1600000000>;
+ opp-microvolt = <1250000>;
+ clock-latency-ns = <140000>;
+ };
+ opp-1700000000 {
+ opp-hz = /bits/ 64 <1700000000>;
+ opp-microvolt = <1300000>;
+ clock-latency-ns = <140000>;
};
};
diff --git a/arch/arm/boot/dts/exynos5420-cpus.dtsi b/arch/arm/boot/dts/exynos5420-cpus.dtsi
index 5c052d7..7e6b555 100644
--- a/arch/arm/boot/dts/exynos5420-cpus.dtsi
+++ b/arch/arm/boot/dts/exynos5420-cpus.dtsi
@@ -33,8 +33,6 @@
clock-frequency = <1800000000>;
cci-control-port = <&cci_control1>;
operating-points-v2 = <&cluster_a15_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <11>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -45,8 +43,6 @@
clock-frequency = <1800000000>;
cci-control-port = <&cci_control1>;
operating-points-v2 = <&cluster_a15_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <11>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -57,8 +53,6 @@
clock-frequency = <1800000000>;
cci-control-port = <&cci_control1>;
operating-points-v2 = <&cluster_a15_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <11>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -69,8 +63,6 @@
clock-frequency = <1800000000>;
cci-control-port = <&cci_control1>;
operating-points-v2 = <&cluster_a15_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <11>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -82,8 +74,6 @@
clock-frequency = <1000000000>;
cci-control-port = <&cci_control0>;
operating-points-v2 = <&cluster_a7_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <7>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -94,8 +84,6 @@
clock-frequency = <1000000000>;
cci-control-port = <&cci_control0>;
operating-points-v2 = <&cluster_a7_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <7>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -106,8 +94,6 @@
clock-frequency = <1000000000>;
cci-control-port = <&cci_control0>;
operating-points-v2 = <&cluster_a7_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <7>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -118,8 +104,6 @@
clock-frequency = <1000000000>;
cci-control-port = <&cci_control0>;
operating-points-v2 = <&cluster_a7_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <7>;
#cooling-cells = <2>; /* min followed by max */
};
};
diff --git a/arch/arm/boot/dts/exynos5422-cpus.dtsi b/arch/arm/boot/dts/exynos5422-cpus.dtsi
index bf3c6f1..c8afdf8 100644
--- a/arch/arm/boot/dts/exynos5422-cpus.dtsi
+++ b/arch/arm/boot/dts/exynos5422-cpus.dtsi
@@ -32,8 +32,6 @@
clock-frequency = <1000000000>;
cci-control-port = <&cci_control0>;
operating-points-v2 = <&cluster_a7_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <11>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -44,8 +42,6 @@
clock-frequency = <1000000000>;
cci-control-port = <&cci_control0>;
operating-points-v2 = <&cluster_a7_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <11>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -56,8 +52,6 @@
clock-frequency = <1000000000>;
cci-control-port = <&cci_control0>;
operating-points-v2 = <&cluster_a7_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <11>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -68,8 +62,6 @@
clock-frequency = <1000000000>;
cci-control-port = <&cci_control0>;
operating-points-v2 = <&cluster_a7_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <11>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -81,8 +73,6 @@
clock-frequency = <1800000000>;
cci-control-port = <&cci_control1>;
operating-points-v2 = <&cluster_a15_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <15>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -93,8 +83,6 @@
clock-frequency = <1800000000>;
cci-control-port = <&cci_control1>;
operating-points-v2 = <&cluster_a15_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <15>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -105,8 +93,6 @@
clock-frequency = <1800000000>;
cci-control-port = <&cci_control1>;
operating-points-v2 = <&cluster_a15_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <15>;
#cooling-cells = <2>; /* min followed by max */
};
@@ -117,8 +103,6 @@
clock-frequency = <1800000000>;
cci-control-port = <&cci_control1>;
operating-points-v2 = <&cluster_a15_opp_table>;
- cooling-min-level = <0>;
- cooling-max-level = <15>;
#cooling-cells = <2>; /* min followed by max */
};
};
diff --git a/arch/arm/boot/dts/imx53-qsb-common.dtsi b/arch/arm/boot/dts/imx53-qsb-common.dtsi
index c05e7cf..c8a6a68 100644
--- a/arch/arm/boot/dts/imx53-qsb-common.dtsi
+++ b/arch/arm/boot/dts/imx53-qsb-common.dtsi
@@ -130,6 +130,17 @@
};
};
+&cpu0 {
+ /* CPU rated to 1GHz, not 1.2GHz as per the default settings */
+ operating-points = <
+ /* kHz uV */
+ 166666 850000
+ 400000 900000
+ 800000 1050000
+ 1000000 1200000
+ >;
+};
+
&esdhc1 {
pinctrl-names = "default";
pinctrl-0 = <&pinctrl_esdhc1>;
diff --git a/arch/arm/boot/dts/imx6q.dtsi b/arch/arm/boot/dts/imx6q.dtsi
index e9a5d0b..908b269 100644
--- a/arch/arm/boot/dts/imx6q.dtsi
+++ b/arch/arm/boot/dts/imx6q.dtsi
@@ -96,7 +96,7 @@
clocks = <&clks IMX6Q_CLK_ECSPI5>,
<&clks IMX6Q_CLK_ECSPI5>;
clock-names = "ipg", "per";
- dmas = <&sdma 11 7 1>, <&sdma 12 7 2>;
+ dmas = <&sdma 11 8 1>, <&sdma 12 8 2>;
dma-names = "rx", "tx";
status = "disabled";
};
diff --git a/arch/arm/boot/dts/imx6qdl-wandboard.dtsi b/arch/arm/boot/dts/imx6qdl-wandboard.dtsi
index 47c9554..2b9c2be 100644
--- a/arch/arm/boot/dts/imx6qdl-wandboard.dtsi
+++ b/arch/arm/boot/dts/imx6qdl-wandboard.dtsi
@@ -88,7 +88,6 @@
clocks = <&clks IMX6QDL_CLK_CKO>;
VDDA-supply = <®_2p5v>;
VDDIO-supply = <®_3p3v>;
- lrclk-strength = <3>;
};
};
diff --git a/arch/arm/boot/dts/imx6sx.dtsi b/arch/arm/boot/dts/imx6sx.dtsi
index 1a473e8..a885052 100644
--- a/arch/arm/boot/dts/imx6sx.dtsi
+++ b/arch/arm/boot/dts/imx6sx.dtsi
@@ -1280,7 +1280,7 @@
/* non-prefetchable memory */
0x82000000 0 0x08000000 0x08000000 0 0x00f00000>;
num-lanes = <1>;
- interrupts = <GIC_SPI 123 IRQ_TYPE_LEVEL_HIGH>;
+ interrupts = <GIC_SPI 120 IRQ_TYPE_LEVEL_HIGH>;
clocks = <&clks IMX6SX_CLK_PCIE_REF_125M>,
<&clks IMX6SX_CLK_PCIE_AXI>,
<&clks IMX6SX_CLK_LVDS1_OUT>,
diff --git a/arch/arm/boot/dts/imx7d-cl-som-imx7.dts b/arch/arm/boot/dts/imx7d-cl-som-imx7.dts
index 58b09bf..2051306 100644
--- a/arch/arm/boot/dts/imx7d-cl-som-imx7.dts
+++ b/arch/arm/boot/dts/imx7d-cl-som-imx7.dts
@@ -213,37 +213,37 @@
&iomuxc {
pinctrl_enet1: enet1grp {
fsl,pins = <
- MX7D_PAD_SD2_CD_B__ENET1_MDIO 0x3
- MX7D_PAD_SD2_WP__ENET1_MDC 0x3
- MX7D_PAD_ENET1_RGMII_TXC__ENET1_RGMII_TXC 0x1
- MX7D_PAD_ENET1_RGMII_TD0__ENET1_RGMII_TD0 0x1
- MX7D_PAD_ENET1_RGMII_TD1__ENET1_RGMII_TD1 0x1
- MX7D_PAD_ENET1_RGMII_TD2__ENET1_RGMII_TD2 0x1
- MX7D_PAD_ENET1_RGMII_TD3__ENET1_RGMII_TD3 0x1
- MX7D_PAD_ENET1_RGMII_TX_CTL__ENET1_RGMII_TX_CTL 0x1
- MX7D_PAD_ENET1_RGMII_RXC__ENET1_RGMII_RXC 0x1
- MX7D_PAD_ENET1_RGMII_RD0__ENET1_RGMII_RD0 0x1
- MX7D_PAD_ENET1_RGMII_RD1__ENET1_RGMII_RD1 0x1
- MX7D_PAD_ENET1_RGMII_RD2__ENET1_RGMII_RD2 0x1
- MX7D_PAD_ENET1_RGMII_RD3__ENET1_RGMII_RD3 0x1
- MX7D_PAD_ENET1_RGMII_RX_CTL__ENET1_RGMII_RX_CTL 0x1
+ MX7D_PAD_SD2_CD_B__ENET1_MDIO 0x30
+ MX7D_PAD_SD2_WP__ENET1_MDC 0x30
+ MX7D_PAD_ENET1_RGMII_TXC__ENET1_RGMII_TXC 0x11
+ MX7D_PAD_ENET1_RGMII_TD0__ENET1_RGMII_TD0 0x11
+ MX7D_PAD_ENET1_RGMII_TD1__ENET1_RGMII_TD1 0x11
+ MX7D_PAD_ENET1_RGMII_TD2__ENET1_RGMII_TD2 0x11
+ MX7D_PAD_ENET1_RGMII_TD3__ENET1_RGMII_TD3 0x11
+ MX7D_PAD_ENET1_RGMII_TX_CTL__ENET1_RGMII_TX_CTL 0x11
+ MX7D_PAD_ENET1_RGMII_RXC__ENET1_RGMII_RXC 0x11
+ MX7D_PAD_ENET1_RGMII_RD0__ENET1_RGMII_RD0 0x11
+ MX7D_PAD_ENET1_RGMII_RD1__ENET1_RGMII_RD1 0x11
+ MX7D_PAD_ENET1_RGMII_RD2__ENET1_RGMII_RD2 0x11
+ MX7D_PAD_ENET1_RGMII_RD3__ENET1_RGMII_RD3 0x11
+ MX7D_PAD_ENET1_RGMII_RX_CTL__ENET1_RGMII_RX_CTL 0x11
>;
};
pinctrl_enet2: enet2grp {
fsl,pins = <
- MX7D_PAD_EPDC_GDSP__ENET2_RGMII_TXC 0x1
- MX7D_PAD_EPDC_SDCE2__ENET2_RGMII_TD0 0x1
- MX7D_PAD_EPDC_SDCE3__ENET2_RGMII_TD1 0x1
- MX7D_PAD_EPDC_GDCLK__ENET2_RGMII_TD2 0x1
- MX7D_PAD_EPDC_GDOE__ENET2_RGMII_TD3 0x1
- MX7D_PAD_EPDC_GDRL__ENET2_RGMII_TX_CTL 0x1
- MX7D_PAD_EPDC_SDCE1__ENET2_RGMII_RXC 0x1
- MX7D_PAD_EPDC_SDCLK__ENET2_RGMII_RD0 0x1
- MX7D_PAD_EPDC_SDLE__ENET2_RGMII_RD1 0x1
- MX7D_PAD_EPDC_SDOE__ENET2_RGMII_RD2 0x1
- MX7D_PAD_EPDC_SDSHR__ENET2_RGMII_RD3 0x1
- MX7D_PAD_EPDC_SDCE0__ENET2_RGMII_RX_CTL 0x1
+ MX7D_PAD_EPDC_GDSP__ENET2_RGMII_TXC 0x11
+ MX7D_PAD_EPDC_SDCE2__ENET2_RGMII_TD0 0x11
+ MX7D_PAD_EPDC_SDCE3__ENET2_RGMII_TD1 0x11
+ MX7D_PAD_EPDC_GDCLK__ENET2_RGMII_TD2 0x11
+ MX7D_PAD_EPDC_GDOE__ENET2_RGMII_TD3 0x11
+ MX7D_PAD_EPDC_GDRL__ENET2_RGMII_TX_CTL 0x11
+ MX7D_PAD_EPDC_SDCE1__ENET2_RGMII_RXC 0x11
+ MX7D_PAD_EPDC_SDCLK__ENET2_RGMII_RD0 0x11
+ MX7D_PAD_EPDC_SDLE__ENET2_RGMII_RD1 0x11
+ MX7D_PAD_EPDC_SDOE__ENET2_RGMII_RD2 0x11
+ MX7D_PAD_EPDC_SDSHR__ENET2_RGMII_RD3 0x11
+ MX7D_PAD_EPDC_SDCE0__ENET2_RGMII_RX_CTL 0x11
>;
};
diff --git a/arch/arm/boot/dts/imx7d-nitrogen7.dts b/arch/arm/boot/dts/imx7d-nitrogen7.dts
index ce08f18..080a4ea 100644
--- a/arch/arm/boot/dts/imx7d-nitrogen7.dts
+++ b/arch/arm/boot/dts/imx7d-nitrogen7.dts
@@ -117,13 +117,17 @@
compatible = "regulator-fixed";
regulator-min-microvolt = <3300000>;
regulator-max-microvolt = <3300000>;
- clocks = <&clks IMX7D_CLKO2_ROOT_DIV>;
- clock-names = "slow";
regulator-name = "reg_wlan";
startup-delay-us = <70000>;
gpio = <&gpio4 21 GPIO_ACTIVE_HIGH>;
enable-active-high;
};
+
+ usdhc2_pwrseq: usdhc2_pwrseq {
+ compatible = "mmc-pwrseq-simple";
+ clocks = <&clks IMX7D_CLKO2_ROOT_DIV>;
+ clock-names = "ext_clock";
+ };
};
&adc1 {
@@ -430,6 +434,7 @@
bus-width = <4>;
non-removable;
vmmc-supply = <®_wlan>;
+ mmc-pwrseq = <&usdhc2_pwrseq>;
cap-power-off-card;
keep-power-in-suspend;
status = "okay";
diff --git a/arch/arm/boot/dts/logicpd-som-lv.dtsi b/arch/arm/boot/dts/logicpd-som-lv.dtsi
index e262fa9..876ed5f 100644
--- a/arch/arm/boot/dts/logicpd-som-lv.dtsi
+++ b/arch/arm/boot/dts/logicpd-som-lv.dtsi
@@ -122,7 +122,7 @@
};
&mmc3 {
- interrupts-extended = <&intc 94 &omap3_pmx_core2 0x46>;
+ interrupts-extended = <&intc 94 &omap3_pmx_core 0x136>;
pinctrl-0 = <&mmc3_pins &wl127x_gpio>;
pinctrl-names = "default";
vmmc-supply = <&wl12xx_vmmc>;
diff --git a/arch/arm/boot/dts/qcom-msm8974-lge-nexus5-hammerhead.dts b/arch/arm/boot/dts/qcom-msm8974-lge-nexus5-hammerhead.dts
index c0fb4a6..386b93c 100644
--- a/arch/arm/boot/dts/qcom-msm8974-lge-nexus5-hammerhead.dts
+++ b/arch/arm/boot/dts/qcom-msm8974-lge-nexus5-hammerhead.dts
@@ -188,6 +188,8 @@
regulator-max-microvolt = <2950000>;
regulator-boot-on;
+ regulator-system-load = <200000>;
+ regulator-allow-set-load;
};
l21 {
diff --git a/arch/arm/boot/dts/r8a7791-porter.dts b/arch/arm/boot/dts/r8a7791-porter.dts
index 6761d11..db0239c 100644
--- a/arch/arm/boot/dts/r8a7791-porter.dts
+++ b/arch/arm/boot/dts/r8a7791-porter.dts
@@ -428,7 +428,7 @@
"dclkin.0", "dclkin.1";
ports {
- port@1 {
+ port@0 {
endpoint {
remote-endpoint = <&adv7511_in>;
};
diff --git a/arch/arm/boot/dts/sama5d3_emac.dtsi b/arch/arm/boot/dts/sama5d3_emac.dtsi
index 7cb235e..6e9e1c2 100644
--- a/arch/arm/boot/dts/sama5d3_emac.dtsi
+++ b/arch/arm/boot/dts/sama5d3_emac.dtsi
@@ -41,7 +41,7 @@
};
macb1: ethernet@f802c000 {
- compatible = "cdns,at91sam9260-macb", "cdns,macb";
+ compatible = "atmel,sama5d3-macb", "cdns,at91sam9260-macb", "cdns,macb";
reg = <0xf802c000 0x100>;
interrupts = <35 IRQ_TYPE_LEVEL_HIGH 3>;
pinctrl-names = "default";
diff --git a/arch/arm/boot/dts/sh73a0.dtsi b/arch/arm/boot/dts/sh73a0.dtsi
index 032fe2f..6b0cc22 100644
--- a/arch/arm/boot/dts/sh73a0.dtsi
+++ b/arch/arm/boot/dts/sh73a0.dtsi
@@ -22,7 +22,7 @@
#address-cells = <1>;
#size-cells = <0>;
- cpu@0 {
+ cpu0: cpu@0 {
device_type = "cpu";
compatible = "arm,cortex-a9";
reg = <0>;
@@ -30,7 +30,7 @@
power-domains = <&pd_a2sl>;
next-level-cache = <&L2>;
};
- cpu@1 {
+ cpu1: cpu@1 {
device_type = "cpu";
compatible = "arm,cortex-a9";
reg = <1>;
@@ -89,6 +89,7 @@
compatible = "arm,cortex-a9-pmu";
interrupts = <GIC_SPI 55 IRQ_TYPE_LEVEL_HIGH>,
<GIC_SPI 56 IRQ_TYPE_LEVEL_HIGH>;
+ interrupt-affinity = <&cpu0>, <&cpu1>;
};
cmt1: timer@e6138000 {
diff --git a/arch/arm/boot/dts/socfpga.dtsi b/arch/arm/boot/dts/socfpga.dtsi
index 9f48141..f0702d8 100644
--- a/arch/arm/boot/dts/socfpga.dtsi
+++ b/arch/arm/boot/dts/socfpga.dtsi
@@ -759,7 +759,7 @@
timer@fffec600 {
compatible = "arm,cortex-a9-twd-timer";
reg = <0xfffec600 0x100>;
- interrupts = <1 13 0xf04>;
+ interrupts = <1 13 0xf01>;
clocks = <&mpu_periph_clk>;
};
diff --git a/arch/arm/boot/dts/tegra30-cardhu.dtsi b/arch/arm/boot/dts/tegra30-cardhu.dtsi
index f11012b..cfcf5dc 100644
--- a/arch/arm/boot/dts/tegra30-cardhu.dtsi
+++ b/arch/arm/boot/dts/tegra30-cardhu.dtsi
@@ -205,6 +205,7 @@
#address-cells = <1>;
#size-cells = <0>;
reg = <0x70>;
+ reset-gpio = <&gpio TEGRA_GPIO(BB, 0) GPIO_ACTIVE_LOW>;
};
};
diff --git a/arch/arm/configs/imx_v4_v5_defconfig b/arch/arm/configs/imx_v4_v5_defconfig
index 5f013c9f..8290c9a 100644
--- a/arch/arm/configs/imx_v4_v5_defconfig
+++ b/arch/arm/configs/imx_v4_v5_defconfig
@@ -145,9 +145,11 @@
CONFIG_USB_CHIPIDEA=y
CONFIG_USB_CHIPIDEA_UDC=y
CONFIG_USB_CHIPIDEA_HOST=y
+CONFIG_USB_CHIPIDEA_ULPI=y
CONFIG_NOP_USB_XCEIV=y
CONFIG_USB_GADGET=y
CONFIG_USB_ETH=m
+CONFIG_USB_ULPI_BUS=y
CONFIG_MMC=y
CONFIG_MMC_SDHCI=y
CONFIG_MMC_SDHCI_PLTFM=y
diff --git a/arch/arm/configs/imx_v6_v7_defconfig b/arch/arm/configs/imx_v6_v7_defconfig
index 8ec4dbb..47c3fb8 100644
--- a/arch/arm/configs/imx_v6_v7_defconfig
+++ b/arch/arm/configs/imx_v6_v7_defconfig
@@ -361,6 +361,7 @@
CONFIG_UDF_FS=m
CONFIG_MSDOS_FS=m
CONFIG_VFAT_FS=y
+CONFIG_TMPFS_POSIX_ACL=y
CONFIG_JFFS2_FS=y
CONFIG_UBIFS_FS=y
CONFIG_NFS_FS=y
diff --git a/arch/arm/firmware/trusted_foundations.c b/arch/arm/firmware/trusted_foundations.c
index 3fb1b5a..689e656 100644
--- a/arch/arm/firmware/trusted_foundations.c
+++ b/arch/arm/firmware/trusted_foundations.c
@@ -31,21 +31,25 @@
static unsigned long cpu_boot_addr;
-static void __naked tf_generic_smc(u32 type, u32 arg1, u32 arg2)
+static void tf_generic_smc(u32 type, u32 arg1, u32 arg2)
{
+ register u32 r0 asm("r0") = type;
+ register u32 r1 asm("r1") = arg1;
+ register u32 r2 asm("r2") = arg2;
+
asm volatile(
".arch_extension sec\n\t"
- "stmfd sp!, {r4 - r11, lr}\n\t"
+ "stmfd sp!, {r4 - r11}\n\t"
__asmeq("%0", "r0")
__asmeq("%1", "r1")
__asmeq("%2", "r2")
"mov r3, #0\n\t"
"mov r4, #0\n\t"
"smc #0\n\t"
- "ldmfd sp!, {r4 - r11, pc}"
+ "ldmfd sp!, {r4 - r11}\n\t"
:
- : "r" (type), "r" (arg1), "r" (arg2)
- : "memory");
+ : "r" (r0), "r" (r1), "r" (r2)
+ : "memory", "r3", "r12", "lr");
}
static int tf_set_cpu_boot_addr(int cpu, unsigned long boot_addr)
diff --git a/arch/arm/include/asm/assembler.h b/arch/arm/include/asm/assembler.h
index 12f99fd..e616f61 100644
--- a/arch/arm/include/asm/assembler.h
+++ b/arch/arm/include/asm/assembler.h
@@ -445,11 +445,23 @@ THUMB( orr \reg , \reg , #PSR_T_BIT )
.size \name , . - \name
.endm
+ .macro csdb
+#ifdef CONFIG_THUMB2_KERNEL
+ .inst.w 0xf3af8014
+#else
+ .inst 0xe320f014
+#endif
+ .endm
+
.macro check_uaccess, addr:req, size:req, limit:req, tmp:req, bad:req
#ifndef CONFIG_CPU_USE_DOMAINS
adds \tmp, \addr, #\size - 1
sbcccs \tmp, \tmp, \limit
bcs \bad
+#ifdef CONFIG_CPU_SPECTRE
+ movcs \addr, #0
+ csdb
+#endif
#endif
.endm
@@ -534,4 +546,14 @@ THUMB( orr \reg , \reg , #PSR_T_BIT )
#endif
.endm
+#ifdef CONFIG_KPROBES
+#define _ASM_NOKPROBE(entry) \
+ .pushsection "_kprobe_blacklist", "aw" ; \
+ .balign 4 ; \
+ .long entry; \
+ .popsection
+#else
+#define _ASM_NOKPROBE(entry)
+#endif
+
#endif /* __ASM_ASSEMBLER_H__ */
diff --git a/arch/arm/include/asm/barrier.h b/arch/arm/include/asm/barrier.h
index f5d6981..513e03d 100644
--- a/arch/arm/include/asm/barrier.h
+++ b/arch/arm/include/asm/barrier.h
@@ -16,6 +16,12 @@
#define isb(option) __asm__ __volatile__ ("isb " #option : : : "memory")
#define dsb(option) __asm__ __volatile__ ("dsb " #option : : : "memory")
#define dmb(option) __asm__ __volatile__ ("dmb " #option : : : "memory")
+#ifdef CONFIG_THUMB2_KERNEL
+#define CSDB ".inst.w 0xf3af8014"
+#else
+#define CSDB ".inst 0xe320f014"
+#endif
+#define csdb() __asm__ __volatile__(CSDB : : : "memory")
#elif defined(CONFIG_CPU_XSC3) || __LINUX_ARM_ARCH__ == 6
#define isb(x) __asm__ __volatile__ ("mcr p15, 0, %0, c7, c5, 4" \
: : "r" (0) : "memory")
@@ -36,6 +42,13 @@
#define dmb(x) __asm__ __volatile__ ("" : : : "memory")
#endif
+#ifndef CSDB
+#define CSDB
+#endif
+#ifndef csdb
+#define csdb()
+#endif
+
#ifdef CONFIG_ARM_HEAVY_MB
extern void (*soc_mb)(void);
extern void arm_heavy_mb(void);
@@ -62,6 +75,25 @@ extern void arm_heavy_mb(void);
#define __smp_rmb() __smp_mb()
#define __smp_wmb() dmb(ishst)
+#ifdef CONFIG_CPU_SPECTRE
+static inline unsigned long array_index_mask_nospec(unsigned long idx,
+ unsigned long sz)
+{
+ unsigned long mask;
+
+ asm volatile(
+ "cmp %1, %2\n"
+ " sbc %0, %1, %1\n"
+ CSDB
+ : "=r" (mask)
+ : "r" (idx), "Ir" (sz)
+ : "cc");
+
+ return mask;
+}
+#define array_index_mask_nospec array_index_mask_nospec
+#endif
+
#include <asm-generic/barrier.h>
#endif /* !__ASSEMBLY__ */
diff --git a/arch/arm/include/asm/bugs.h b/arch/arm/include/asm/bugs.h
index a97f1ea..73a99c7 100644
--- a/arch/arm/include/asm/bugs.h
+++ b/arch/arm/include/asm/bugs.h
@@ -10,12 +10,14 @@
#ifndef __ASM_BUGS_H
#define __ASM_BUGS_H
-#ifdef CONFIG_MMU
extern void check_writebuffer_bugs(void);
-#define check_bugs() check_writebuffer_bugs()
+#ifdef CONFIG_MMU
+extern void check_bugs(void);
+extern void check_other_bugs(void);
#else
#define check_bugs() do { } while (0)
+#define check_other_bugs() do { } while (0)
#endif
#endif
diff --git a/arch/arm/include/asm/cp15.h b/arch/arm/include/asm/cp15.h
index dbdbce1..b74b174 100644
--- a/arch/arm/include/asm/cp15.h
+++ b/arch/arm/include/asm/cp15.h
@@ -64,6 +64,9 @@
#define __write_sysreg(v, r, w, c, t) asm volatile(w " " c : : "r" ((t)(v)))
#define write_sysreg(v, ...) __write_sysreg(v, __VA_ARGS__)
+#define BPIALL __ACCESS_CP15(c7, 0, c5, 6)
+#define ICIALLU __ACCESS_CP15(c7, 0, c5, 0)
+
extern unsigned long cr_alignment; /* defined in entry-armv.S */
static inline unsigned long get_cr(void)
diff --git a/arch/arm/include/asm/cputype.h b/arch/arm/include/asm/cputype.h
index b62eaeb..c55db1e 100644
--- a/arch/arm/include/asm/cputype.h
+++ b/arch/arm/include/asm/cputype.h
@@ -76,8 +76,16 @@
#define ARM_CPU_PART_CORTEX_A12 0x4100c0d0
#define ARM_CPU_PART_CORTEX_A17 0x4100c0e0
#define ARM_CPU_PART_CORTEX_A15 0x4100c0f0
+#define ARM_CPU_PART_CORTEX_A53 0x4100d030
+#define ARM_CPU_PART_CORTEX_A57 0x4100d070
+#define ARM_CPU_PART_CORTEX_A72 0x4100d080
+#define ARM_CPU_PART_CORTEX_A73 0x4100d090
+#define ARM_CPU_PART_CORTEX_A75 0x4100d0a0
#define ARM_CPU_PART_MASK 0xff00fff0
+/* Broadcom cores */
+#define ARM_CPU_PART_BRAHMA_B15 0x420000f0
+
/* DEC implemented cores */
#define ARM_CPU_PART_SA1100 0x4400a110
diff --git a/arch/arm/include/asm/futex.h b/arch/arm/include/asm/futex.h
index 6795368..cc41438 100644
--- a/arch/arm/include/asm/futex.h
+++ b/arch/arm/include/asm/futex.h
@@ -128,20 +128,10 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
#endif /* !SMP */
static inline int
-futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
+arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret, tmp;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
-
#ifndef CONFIG_SMP
preempt_disable();
#endif
@@ -172,17 +162,9 @@ futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
preempt_enable();
#endif
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/arm/include/asm/kgdb.h b/arch/arm/include/asm/kgdb.h
index 0a9d5dd..6949c7d 100644
--- a/arch/arm/include/asm/kgdb.h
+++ b/arch/arm/include/asm/kgdb.h
@@ -76,7 +76,7 @@ extern int kgdb_fault_expected;
#define KGDB_MAX_NO_CPUS 1
#define BUFMAX 400
-#define NUMREGBYTES (DBG_MAX_REG_NUM << 2)
+#define NUMREGBYTES (GDB_MAX_REGS << 2)
#define NUMCRITREGBYTES (32 << 2)
#define _R0 0
diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h
index 8ef0538..24f3ec7 100644
--- a/arch/arm/include/asm/kvm_asm.h
+++ b/arch/arm/include/asm/kvm_asm.h
@@ -61,8 +61,6 @@ struct kvm_vcpu;
extern char __kvm_hyp_init[];
extern char __kvm_hyp_init_end[];
-extern char __kvm_hyp_vector[];
-
extern void __kvm_flush_vm_context(void);
extern void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa);
extern void __kvm_tlb_flush_vmid(struct kvm *kvm);
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 9fe1043..2fda7e9 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -21,6 +21,7 @@
#include <linux/types.h>
#include <linux/kvm_types.h>
+#include <asm/cputype.h>
#include <asm/kvm.h>
#include <asm/kvm_asm.h>
#include <asm/kvm_mmio.h>
@@ -78,6 +79,9 @@ struct kvm_arch {
/* Interrupt controller */
struct vgic_dist vgic;
int max_vcpus;
+
+ /* Mandated version of PSCI */
+ u32 psci_version;
};
#define KVM_NR_MEM_OBJS 40
@@ -320,8 +324,29 @@ static inline int kvm_arm_vcpu_arch_has_attr(struct kvm_vcpu *vcpu,
static inline bool kvm_arm_harden_branch_predictor(void)
{
+ switch(read_cpuid_part()) {
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+ case ARM_CPU_PART_BRAHMA_B15:
+ case ARM_CPU_PART_CORTEX_A12:
+ case ARM_CPU_PART_CORTEX_A15:
+ case ARM_CPU_PART_CORTEX_A17:
+ return true;
+#endif
+ default:
+ return false;
+ }
+}
+
+#define KVM_SSBD_UNKNOWN -1
+#define KVM_SSBD_FORCE_DISABLE 0
+#define KVM_SSBD_KERNEL 1
+#define KVM_SSBD_FORCE_ENABLE 2
+#define KVM_SSBD_MITIGATED 3
+
+static inline int kvm_arm_have_ssbd(void)
+{
/* No way to detect it yet, pretend it is not there. */
- return false;
+ return KVM_SSBD_UNKNOWN;
}
#endif /* __ARM_KVM_HOST_H__ */
diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index d10e362..d263957 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -28,6 +28,13 @@
*/
#define kern_hyp_va(kva) (kva)
+/* Contrary to arm64, there is no need to generate a PC-relative address */
+#define hyp_symbol_addr(s) \
+ ({ \
+ typeof(s) *addr = &(s); \
+ addr; \
+ })
+
/*
* KVM_MMU_CACHE_MIN_PAGES is the number of stage2 page table translation levels.
*/
@@ -223,9 +230,46 @@ static inline unsigned int kvm_get_vmid_bits(void)
return 8;
}
+/*
+ * We are not in the kvm->srcu critical section most of the time, so we take
+ * the SRCU read lock here. Since we copy the data from the user page, we
+ * can immediately drop the lock again.
+ */
+static inline int kvm_read_guest_lock(struct kvm *kvm,
+ gpa_t gpa, void *data, unsigned long len)
+{
+ int srcu_idx = srcu_read_lock(&kvm->srcu);
+ int ret = kvm_read_guest(kvm, gpa, data, len);
+
+ srcu_read_unlock(&kvm->srcu, srcu_idx);
+
+ return ret;
+}
+
static inline void *kvm_get_hyp_vector(void)
{
- return kvm_ksym_ref(__kvm_hyp_vector);
+ switch(read_cpuid_part()) {
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+ case ARM_CPU_PART_CORTEX_A12:
+ case ARM_CPU_PART_CORTEX_A17:
+ {
+ extern char __kvm_hyp_vector_bp_inv[];
+ return kvm_ksym_ref(__kvm_hyp_vector_bp_inv);
+ }
+
+ case ARM_CPU_PART_BRAHMA_B15:
+ case ARM_CPU_PART_CORTEX_A15:
+ {
+ extern char __kvm_hyp_vector_ic_inv[];
+ return kvm_ksym_ref(__kvm_hyp_vector_ic_inv);
+ }
+#endif
+ default:
+ {
+ extern char __kvm_hyp_vector[];
+ return kvm_ksym_ref(__kvm_hyp_vector);
+ }
+ }
}
static inline int kvm_map_vectors(void)
@@ -233,6 +277,11 @@ static inline int kvm_map_vectors(void)
return 0;
}
+static inline int hyp_map_aux_data(void)
+{
+ return 0;
+}
+
#endif /* !__ASSEMBLY__ */
#endif /* __ARM_KVM_MMU_H__ */
diff --git a/arch/arm/include/asm/proc-fns.h b/arch/arm/include/asm/proc-fns.h
index 8877ad5..f379f5f 100644
--- a/arch/arm/include/asm/proc-fns.h
+++ b/arch/arm/include/asm/proc-fns.h
@@ -37,6 +37,10 @@ extern struct processor {
*/
void (*_proc_init)(void);
/*
+ * Check for processor bugs
+ */
+ void (*check_bugs)(void);
+ /*
* Disable any processor specifics
*/
void (*_proc_fin)(void);
diff --git a/arch/arm/include/asm/system_misc.h b/arch/arm/include/asm/system_misc.h
index 906623e..be866cf 100644
--- a/arch/arm/include/asm/system_misc.h
+++ b/arch/arm/include/asm/system_misc.h
@@ -7,6 +7,7 @@
#include <linux/linkage.h>
#include <linux/irqflags.h>
#include <linux/reboot.h>
+#include <linux/percpu.h>
extern void cpu_init(void);
@@ -14,6 +15,20 @@ void soft_restart(unsigned long);
extern void (*arm_pm_restart)(enum reboot_mode reboot_mode, const char *cmd);
extern void (*arm_pm_idle)(void);
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+typedef void (*harden_branch_predictor_fn_t)(void);
+DECLARE_PER_CPU(harden_branch_predictor_fn_t, harden_branch_predictor_fn);
+static inline void harden_branch_predictor(void)
+{
+ harden_branch_predictor_fn_t fn = per_cpu(harden_branch_predictor_fn,
+ smp_processor_id());
+ if (fn)
+ fn();
+}
+#else
+#define harden_branch_predictor() do { } while (0)
+#endif
+
#define UDBG_UNDEFINED (1 << 0)
#define UDBG_SYSCALL (1 << 1)
#define UDBG_BADABORT (1 << 2)
diff --git a/arch/arm/include/asm/thread_info.h b/arch/arm/include/asm/thread_info.h
index f23454d..a0e42ca 100644
--- a/arch/arm/include/asm/thread_info.h
+++ b/arch/arm/include/asm/thread_info.h
@@ -126,8 +126,8 @@ struct user_vfp_exc;
extern int vfp_preserve_user_clear_hwstate(struct user_vfp __user *,
struct user_vfp_exc __user *);
-extern int vfp_restore_user_hwstate(struct user_vfp __user *,
- struct user_vfp_exc __user *);
+extern int vfp_restore_user_hwstate(struct user_vfp *,
+ struct user_vfp_exc *);
#endif
/*
diff --git a/arch/arm/include/asm/uaccess.h b/arch/arm/include/asm/uaccess.h
index b7e0125..7b174601 100644
--- a/arch/arm/include/asm/uaccess.h
+++ b/arch/arm/include/asm/uaccess.h
@@ -115,6 +115,13 @@ static inline void set_fs(mm_segment_t fs)
flag; })
/*
+ * This is a type: either unsigned long, if the argument fits into
+ * that type, or otherwise unsigned long long.
+ */
+#define __inttype(x) \
+ __typeof__(__builtin_choose_expr(sizeof(x) > sizeof(0UL), 0ULL, 0UL))
+
+/*
* Single-value transfer routines. They automatically use the right
* size if we just have the right pointer type. Note that the functions
* which read from user space (*get_*) need to take care not to leak
@@ -183,7 +190,7 @@ extern int __get_user_64t_4(void *);
({ \
unsigned long __limit = current_thread_info()->addr_limit - 1; \
register const typeof(*(p)) __user *__p asm("r0") = (p);\
- register typeof(x) __r2 asm("r2"); \
+ register __inttype(x) __r2 asm("r2"); \
register unsigned long __l asm("r1") = __limit; \
register int __e asm("r0"); \
unsigned int __ua_flags = uaccess_save_and_enable(); \
@@ -273,6 +280,16 @@ static inline void set_fs(mm_segment_t fs)
#define user_addr_max() \
(segment_eq(get_fs(), KERNEL_DS) ? ~0UL : get_fs())
+#ifdef CONFIG_CPU_SPECTRE
+/*
+ * When mitigating Spectre variant 1, it is not worth fixing the non-
+ * verifying accessors, because we need to add verification of the
+ * address space there. Force these to use the standard get_user()
+ * version instead.
+ */
+#define __get_user(x, ptr) get_user(x, ptr)
+#else
+
/*
* The "__xxx" versions of the user access functions do not verify the
* address space - it must have been done previously with a separate
@@ -289,12 +306,6 @@ static inline void set_fs(mm_segment_t fs)
__gu_err; \
})
-#define __get_user_error(x, ptr, err) \
-({ \
- __get_user_err((x), (ptr), err); \
- (void) 0; \
-})
-
#define __get_user_err(x, ptr, err) \
do { \
unsigned long __gu_addr = (unsigned long)(ptr); \
@@ -354,6 +365,7 @@ do { \
#define __get_user_asm_word(x, addr, err) \
__get_user_asm(x, addr, err, ldr)
+#endif
#define __put_user_switch(x, ptr, __err, __fn) \
diff --git a/arch/arm/include/asm/vdso.h b/arch/arm/include/asm/vdso.h
index d0295f1..ff65b6d 100644
--- a/arch/arm/include/asm/vdso.h
+++ b/arch/arm/include/asm/vdso.h
@@ -11,8 +11,6 @@ struct mm_struct;
void arm_install_vdso(struct mm_struct *mm, unsigned long addr);
-extern char vdso_start, vdso_end;
-
extern unsigned int vdso_total_pages;
#else /* CONFIG_VDSO */
diff --git a/arch/arm/include/asm/vdso_datapage.h b/arch/arm/include/asm/vdso_datapage.h
index 9be2594..0120852 100644
--- a/arch/arm/include/asm/vdso_datapage.h
+++ b/arch/arm/include/asm/vdso_datapage.h
@@ -24,21 +24,38 @@
#include <asm/page.h>
+#ifndef _VDSO_WTM_CLOCK_SEC_T
+#define _VDSO_WTM_CLOCK_SEC_T
+typedef u32 vdso_wtm_clock_nsec_t;
+#endif
+
+#ifndef _VDSO_XTIME_CLOCK_SEC_T
+#define _VDSO_XTIME_CLOCK_SEC_T
+typedef u32 vdso_xtime_clock_sec_t;
+#endif
+
+#ifndef _VDSO_RAW_TIME_SEC_T
+#define _VDSO_RAW_TIME_SEC_T
+typedef u32 vdso_raw_time_sec_t;
+#endif
+
/* Try to be cache-friendly on systems that don't implement the
* generic timer: fit the unconditionally updated fields in the first
* 32 bytes.
*/
struct vdso_data {
- u32 seq_count; /* sequence count - odd during updates */
- u16 tk_is_cntvct; /* fall back to syscall if false */
+ u32 tb_seq_count; /* sequence count - odd during updates */
+ u16 use_syscall; /* fall back to syscall if true */
u16 cs_shift; /* clocksource shift */
u32 xtime_coarse_sec; /* coarse time */
u32 xtime_coarse_nsec;
- u32 wtm_clock_sec; /* wall to monotonic offset */
- u32 wtm_clock_nsec;
- u32 xtime_clock_sec; /* CLOCK_REALTIME - seconds */
- u32 cs_mult; /* clocksource multiplier */
+ /* wall to monotonic offset */
+ u32 wtm_clock_sec;
+ vdso_wtm_clock_nsec_t wtm_clock_nsec;
+ /* CLOCK_REALTIME - seconds */
+ vdso_xtime_clock_sec_t xtime_clock_sec;
+ u32 cs_mono_mult; /* clocksource multiplier */
u64 cs_cycle_last; /* last cycle value */
u64 cs_mask; /* clocksource mask */
@@ -46,6 +63,14 @@ struct vdso_data {
u64 xtime_clock_snsec; /* CLOCK_REALTIME sub-ns base */
u32 tz_minuteswest; /* timezone info for gettimeofday(2) */
u32 tz_dsttime;
+
+ u32 btm_sec; /* monotonic to boot time */
+ u32 btm_nsec;
+ /* Raw clocksource multipler */
+ u32 cs_raw_mult;
+ /* Raw time */
+ vdso_raw_time_sec_t raw_time_sec;
+ u32 raw_time_nsec;
};
union vdso_data_store {
diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h
index b38c10c..0b8cf31 100644
--- a/arch/arm/include/uapi/asm/kvm.h
+++ b/arch/arm/include/uapi/asm/kvm.h
@@ -173,6 +173,12 @@ struct kvm_arch_memory_slot {
#define KVM_REG_ARM_VFP_FPINST 0x1009
#define KVM_REG_ARM_VFP_FPINST2 0x100A
+/* KVM-as-firmware specific pseudo-registers */
+#define KVM_REG_ARM_FW (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
+#define KVM_REG_ARM_FW_REG(r) (KVM_REG_ARM | KVM_REG_SIZE_U64 | \
+ KVM_REG_ARM_FW | ((r) & 0xffff))
+#define KVM_REG_ARM_PSCI_VERSION KVM_REG_ARM_FW_REG(0)
+
/* Device Control API: ARM VGIC */
#define KVM_DEV_ARM_VGIC_GRP_ADDR 0
#define KVM_DEV_ARM_VGIC_GRP_DIST_REGS 1
diff --git a/arch/arm/kernel/Makefile b/arch/arm/kernel/Makefile
index ad325a8..adb9add 100644
--- a/arch/arm/kernel/Makefile
+++ b/arch/arm/kernel/Makefile
@@ -30,6 +30,7 @@
obj-y += entry-armv.o
endif
+obj-$(CONFIG_MMU) += bugs.o
obj-$(CONFIG_CPU_IDLE) += cpuidle.o
obj-$(CONFIG_ISA_DMA_API) += dma.o
obj-$(CONFIG_FIQ) += fiq.o fiqasm.o
diff --git a/arch/arm/kernel/bugs.c b/arch/arm/kernel/bugs.c
new file mode 100644
index 0000000..7be5113
--- /dev/null
+++ b/arch/arm/kernel/bugs.c
@@ -0,0 +1,18 @@
+// SPDX-Identifier: GPL-2.0
+#include <linux/init.h>
+#include <asm/bugs.h>
+#include <asm/proc-fns.h>
+
+void check_other_bugs(void)
+{
+#ifdef MULTI_CPU
+ if (processor.check_bugs)
+ processor.check_bugs();
+#endif
+}
+
+void __init check_bugs(void)
+{
+ check_writebuffer_bugs();
+ check_other_bugs();
+}
diff --git a/arch/arm/kernel/entry-common.S b/arch/arm/kernel/entry-common.S
index 10c3283..56be67ec 100644
--- a/arch/arm/kernel/entry-common.S
+++ b/arch/arm/kernel/entry-common.S
@@ -223,9 +223,7 @@
tst r10, #_TIF_SYSCALL_WORK @ are we tracing syscalls?
bne __sys_trace
- cmp scno, #NR_syscalls @ check upper syscall limit
- badr lr, ret_fast_syscall @ return address
- ldrcc pc, [tbl, scno, lsl #2] @ call sys_* routine
+ invoke_syscall tbl, scno, r10, ret_fast_syscall
add r1, sp, #S_OFF
2: cmp scno, #(__ARM_NR_BASE - __NR_SYSCALL_BASE)
@@ -258,14 +256,8 @@
mov r1, scno
add r0, sp, #S_OFF
bl syscall_trace_enter
-
- badr lr, __sys_trace_return @ return address
- mov scno, r0 @ syscall number (possibly new)
- add r1, sp, #S_R0 + S_OFF @ pointer to regs
- cmp scno, #NR_syscalls @ check upper syscall limit
- ldmccia r1, {r0 - r6} @ have to reload r0 - r6
- stmccia sp, {r4, r5} @ and update the stack args
- ldrcc pc, [tbl, scno, lsl #2] @ call sys_* routine
+ mov scno, r0
+ invoke_syscall tbl, scno, r10, __sys_trace_return, reload=1
cmp scno, #-1 @ skip the syscall?
bne 2b
add sp, sp, #S_OFF @ restore stack
@@ -317,6 +309,10 @@
bic scno, r0, #__NR_OABI_SYSCALL_BASE
cmp scno, #__NR_syscall - __NR_SYSCALL_BASE
cmpne scno, #NR_syscalls @ check range
+#ifdef CONFIG_CPU_SPECTRE
+ movhs scno, #0
+ csdb
+#endif
stmloia sp, {r5, r6} @ shuffle args
movlo r0, r1
movlo r1, r2
diff --git a/arch/arm/kernel/entry-header.S b/arch/arm/kernel/entry-header.S
index e056c9a..fa7c6e5c 100644
--- a/arch/arm/kernel/entry-header.S
+++ b/arch/arm/kernel/entry-header.S
@@ -377,6 +377,31 @@
#endif
.endm
+ .macro invoke_syscall, table, nr, tmp, ret, reload=0
+#ifdef CONFIG_CPU_SPECTRE
+ mov \tmp, \nr
+ cmp \tmp, #NR_syscalls @ check upper syscall limit
+ movcs \tmp, #0
+ csdb
+ badr lr, \ret @ return address
+ .if \reload
+ add r1, sp, #S_R0 + S_OFF @ pointer to regs
+ ldmccia r1, {r0 - r6} @ reload r0-r6
+ stmccia sp, {r4, r5} @ update stack arguments
+ .endif
+ ldrcc pc, [\table, \tmp, lsl #2] @ call sys_* routine
+#else
+ cmp \nr, #NR_syscalls @ check upper syscall limit
+ badr lr, \ret @ return address
+ .if \reload
+ add r1, sp, #S_R0 + S_OFF @ pointer to regs
+ ldmccia r1, {r0 - r6} @ reload r0-r6
+ stmccia sp, {r4, r5} @ update stack arguments
+ .endif
+ ldrcc pc, [\table, \nr, lsl #2] @ call sys_* routine
+#endif
+ .endm
+
/*
* These are the registers used in the syscall handler, and allow us to
* have in theory up to 7 arguments to a function - r0 to r6.
diff --git a/arch/arm/kernel/signal.c b/arch/arm/kernel/signal.c
index 7b8f214..6bee5c9 100644
--- a/arch/arm/kernel/signal.c
+++ b/arch/arm/kernel/signal.c
@@ -107,21 +107,20 @@ static int preserve_vfp_context(struct vfp_sigframe __user *frame)
return vfp_preserve_user_clear_hwstate(&frame->ufp, &frame->ufp_exc);
}
-static int restore_vfp_context(struct vfp_sigframe __user *frame)
+static int restore_vfp_context(struct vfp_sigframe __user *auxp)
{
- unsigned long magic;
- unsigned long size;
- int err = 0;
+ struct vfp_sigframe frame;
+ int err;
- __get_user_error(magic, &frame->magic, err);
- __get_user_error(size, &frame->size, err);
+ err = __copy_from_user(&frame, (char __user *) auxp, sizeof(frame));
if (err)
- return -EFAULT;
- if (magic != VFP_MAGIC || size != VFP_STORAGE_SIZE)
+ return err;
+
+ if (frame.magic != VFP_MAGIC || frame.size != VFP_STORAGE_SIZE)
return -EINVAL;
- return vfp_restore_user_hwstate(&frame->ufp, &frame->ufp_exc);
+ return vfp_restore_user_hwstate(&frame.ufp, &frame.ufp_exc);
}
#endif
@@ -141,6 +140,7 @@ struct rt_sigframe {
static int restore_sigframe(struct pt_regs *regs, struct sigframe __user *sf)
{
+ struct sigcontext context;
struct aux_sigframe __user *aux;
sigset_t set;
int err;
@@ -149,23 +149,26 @@ static int restore_sigframe(struct pt_regs *regs, struct sigframe __user *sf)
if (err == 0)
set_current_blocked(&set);
- __get_user_error(regs->ARM_r0, &sf->uc.uc_mcontext.arm_r0, err);
- __get_user_error(regs->ARM_r1, &sf->uc.uc_mcontext.arm_r1, err);
- __get_user_error(regs->ARM_r2, &sf->uc.uc_mcontext.arm_r2, err);
- __get_user_error(regs->ARM_r3, &sf->uc.uc_mcontext.arm_r3, err);
- __get_user_error(regs->ARM_r4, &sf->uc.uc_mcontext.arm_r4, err);
- __get_user_error(regs->ARM_r5, &sf->uc.uc_mcontext.arm_r5, err);
- __get_user_error(regs->ARM_r6, &sf->uc.uc_mcontext.arm_r6, err);
- __get_user_error(regs->ARM_r7, &sf->uc.uc_mcontext.arm_r7, err);
- __get_user_error(regs->ARM_r8, &sf->uc.uc_mcontext.arm_r8, err);
- __get_user_error(regs->ARM_r9, &sf->uc.uc_mcontext.arm_r9, err);
- __get_user_error(regs->ARM_r10, &sf->uc.uc_mcontext.arm_r10, err);
- __get_user_error(regs->ARM_fp, &sf->uc.uc_mcontext.arm_fp, err);
- __get_user_error(regs->ARM_ip, &sf->uc.uc_mcontext.arm_ip, err);
- __get_user_error(regs->ARM_sp, &sf->uc.uc_mcontext.arm_sp, err);
- __get_user_error(regs->ARM_lr, &sf->uc.uc_mcontext.arm_lr, err);
- __get_user_error(regs->ARM_pc, &sf->uc.uc_mcontext.arm_pc, err);
- __get_user_error(regs->ARM_cpsr, &sf->uc.uc_mcontext.arm_cpsr, err);
+ err |= __copy_from_user(&context, &sf->uc.uc_mcontext, sizeof(context));
+ if (err == 0) {
+ regs->ARM_r0 = context.arm_r0;
+ regs->ARM_r1 = context.arm_r1;
+ regs->ARM_r2 = context.arm_r2;
+ regs->ARM_r3 = context.arm_r3;
+ regs->ARM_r4 = context.arm_r4;
+ regs->ARM_r5 = context.arm_r5;
+ regs->ARM_r6 = context.arm_r6;
+ regs->ARM_r7 = context.arm_r7;
+ regs->ARM_r8 = context.arm_r8;
+ regs->ARM_r9 = context.arm_r9;
+ regs->ARM_r10 = context.arm_r10;
+ regs->ARM_fp = context.arm_fp;
+ regs->ARM_ip = context.arm_ip;
+ regs->ARM_sp = context.arm_sp;
+ regs->ARM_lr = context.arm_lr;
+ regs->ARM_pc = context.arm_pc;
+ regs->ARM_cpsr = context.arm_cpsr;
+ }
err |= !valid_user_regs(regs);
diff --git a/arch/arm/kernel/smp.c b/arch/arm/kernel/smp.c
index a03a99a..40bc867 100644
--- a/arch/arm/kernel/smp.c
+++ b/arch/arm/kernel/smp.c
@@ -29,6 +29,7 @@
#include <linux/irq_work.h>
#include <linux/atomic.h>
+#include <asm/bugs.h>
#include <asm/smp.h>
#include <asm/cacheflush.h>
#include <asm/cpu.h>
@@ -400,6 +401,9 @@ asmlinkage void secondary_start_kernel(void)
* before we continue - which happens after __cpu_up returns.
*/
set_cpu_online(cpu, true);
+
+ check_other_bugs();
+
complete(&cpu_running);
local_irq_enable();
diff --git a/arch/arm/kernel/suspend.c b/arch/arm/kernel/suspend.c
index 9a2f882..134f0d4 100644
--- a/arch/arm/kernel/suspend.c
+++ b/arch/arm/kernel/suspend.c
@@ -1,6 +1,7 @@
#include <linux/init.h>
#include <linux/slab.h>
+#include <asm/bugs.h>
#include <asm/cacheflush.h>
#include <asm/idmap.h>
#include <asm/pgalloc.h>
@@ -34,6 +35,7 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long))
cpu_switch_mm(mm->pgd, mm);
local_flush_bp_all();
local_flush_tlb_all();
+ check_other_bugs();
}
return ret;
diff --git a/arch/arm/kernel/sys_oabi-compat.c b/arch/arm/kernel/sys_oabi-compat.c
index 5f221ac..640748e 100644
--- a/arch/arm/kernel/sys_oabi-compat.c
+++ b/arch/arm/kernel/sys_oabi-compat.c
@@ -328,9 +328,11 @@ asmlinkage long sys_oabi_semtimedop(int semid,
return -ENOMEM;
err = 0;
for (i = 0; i < nsops; i++) {
- __get_user_error(sops[i].sem_num, &tsops->sem_num, err);
- __get_user_error(sops[i].sem_op, &tsops->sem_op, err);
- __get_user_error(sops[i].sem_flg, &tsops->sem_flg, err);
+ struct oabi_sembuf osb;
+ err |= __copy_from_user(&osb, tsops, sizeof(osb));
+ sops[i].sem_num = osb.sem_num;
+ sops[i].sem_op = osb.sem_op;
+ sops[i].sem_flg = osb.sem_flg;
tsops++;
}
if (timeout) {
diff --git a/arch/arm/kernel/traps.c b/arch/arm/kernel/traps.c
index 1b30489..aa316a7 100644
--- a/arch/arm/kernel/traps.c
+++ b/arch/arm/kernel/traps.c
@@ -19,6 +19,7 @@
#include <linux/uaccess.h>
#include <linux/hardirq.h>
#include <linux/kdebug.h>
+#include <linux/kprobes.h>
#include <linux/module.h>
#include <linux/kexec.h>
#include <linux/bug.h>
@@ -415,7 +416,8 @@ void unregister_undef_hook(struct undef_hook *hook)
raw_spin_unlock_irqrestore(&undef_lock, flags);
}
-static int call_undef_hook(struct pt_regs *regs, unsigned int instr)
+static nokprobe_inline
+int call_undef_hook(struct pt_regs *regs, unsigned int instr)
{
struct undef_hook *hook;
unsigned long flags;
@@ -488,6 +490,7 @@ asmlinkage void __exception do_undefinstr(struct pt_regs *regs)
arm_notify_die("Oops - undefined instruction", regs, &info, 0, 6);
}
+NOKPROBE_SYMBOL(do_undefinstr)
/*
* Handle FIQ similarly to NMI on x86 systems.
diff --git a/arch/arm/kernel/vdso.c b/arch/arm/kernel/vdso.c
index 53cf86c..2385f02 100644
--- a/arch/arm/kernel/vdso.c
+++ b/arch/arm/kernel/vdso.c
@@ -39,6 +39,8 @@
static struct page **vdso_text_pagelist;
+extern char vdso_start[], vdso_end[];
+
/* Total number of pages needed for the data and text portions of the VDSO. */
unsigned int vdso_total_pages __ro_after_init;
@@ -171,6 +173,8 @@ static void __init patch_vdso(void *ehdr)
if (!cntvct_ok) {
vdso_nullpatch_one(&einfo, "__vdso_gettimeofday");
vdso_nullpatch_one(&einfo, "__vdso_clock_gettime");
+ vdso_nullpatch_one(&einfo, "__vdso_clock_getres");
+ /* do not zero out __vdso_time, no cntvct_ok dependency */
}
}
@@ -179,13 +183,13 @@ static int __init vdso_init(void)
unsigned int text_pages;
int i;
- if (memcmp(&vdso_start, "\177ELF", 4)) {
+ if (memcmp(vdso_start, "\177ELF", 4)) {
pr_err("VDSO is not a valid ELF object!\n");
return -ENOEXEC;
}
- text_pages = (&vdso_end - &vdso_start) >> PAGE_SHIFT;
- pr_debug("vdso: %i text pages at base %p\n", text_pages, &vdso_start);
+ text_pages = (vdso_end - vdso_start) >> PAGE_SHIFT;
+ pr_debug("vdso: %i text pages at base %p\n", text_pages, vdso_start);
/* Allocate the VDSO text pagelist */
vdso_text_pagelist = kcalloc(text_pages, sizeof(struct page *),
@@ -200,7 +204,7 @@ static int __init vdso_init(void)
for (i = 0; i < text_pages; i++) {
struct page *page;
- page = virt_to_page(&vdso_start + i * PAGE_SIZE);
+ page = virt_to_page(vdso_start + i * PAGE_SIZE);
vdso_text_pagelist[i] = page;
}
@@ -211,7 +215,7 @@ static int __init vdso_init(void)
cntvct_ok = cntvct_functional();
- patch_vdso(&vdso_start);
+ patch_vdso(vdso_start);
return 0;
}
@@ -256,14 +260,14 @@ void arm_install_vdso(struct mm_struct *mm, unsigned long addr)
static void vdso_write_begin(struct vdso_data *vdata)
{
- ++vdso_data->seq_count;
+ ++vdso_data->tb_seq_count;
smp_wmb(); /* Pairs with smp_rmb in vdso_read_retry */
}
static void vdso_write_end(struct vdso_data *vdata)
{
smp_wmb(); /* Pairs with smp_rmb in vdso_read_begin */
- ++vdso_data->seq_count;
+ ++vdso_data->tb_seq_count;
}
static bool tk_is_cntvct(const struct timekeeper *tk)
@@ -287,10 +291,10 @@ static bool tk_is_cntvct(const struct timekeeper *tk)
* counter again, making it even, indicating to userspace that the
* update is finished.
*
- * Userspace is expected to sample seq_count before reading any other
- * fields from the data page. If seq_count is odd, userspace is
+ * Userspace is expected to sample tb_seq_count before reading any other
+ * fields from the data page. If tb_seq_count is odd, userspace is
* expected to wait until it becomes even. After copying data from
- * the page, userspace must sample seq_count again; if it has changed
+ * the page, userspace must sample tb_seq_count again; if it has changed
* from its previous value, userspace must retry the whole sequence.
*
* Calls to update_vsyscall are serialized by the timekeeping core.
@@ -308,20 +312,28 @@ void update_vsyscall(struct timekeeper *tk)
vdso_write_begin(vdso_data);
- vdso_data->tk_is_cntvct = tk_is_cntvct(tk);
+ vdso_data->use_syscall = !tk_is_cntvct(tk);
vdso_data->xtime_coarse_sec = tk->xtime_sec;
vdso_data->xtime_coarse_nsec = (u32)(tk->tkr_mono.xtime_nsec >>
tk->tkr_mono.shift);
vdso_data->wtm_clock_sec = wtm->tv_sec;
vdso_data->wtm_clock_nsec = wtm->tv_nsec;
- if (vdso_data->tk_is_cntvct) {
+ if (!vdso_data->use_syscall) {
+ struct timespec btm = ktime_to_timespec(tk->offs_boot);
+
vdso_data->cs_cycle_last = tk->tkr_mono.cycle_last;
+ vdso_data->raw_time_sec = tk->raw_sec;
+ vdso_data->raw_time_nsec = tk->tkr_raw.xtime_nsec;
vdso_data->xtime_clock_sec = tk->xtime_sec;
vdso_data->xtime_clock_snsec = tk->tkr_mono.xtime_nsec;
- vdso_data->cs_mult = tk->tkr_mono.mult;
+ vdso_data->cs_mono_mult = tk->tkr_mono.mult;
+ vdso_data->cs_raw_mult = tk->tkr_raw.mult;
+ /* tkr_mono.shift == tkr_raw.shift */
vdso_data->cs_shift = tk->tkr_mono.shift;
vdso_data->cs_mask = tk->tkr_mono.mask;
+ vdso_data->btm_sec = btm.tv_sec;
+ vdso_data->btm_nsec = btm.tv_nsec;
}
vdso_write_end(vdso_data);
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index ef6595c..a670c70 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -51,8 +51,8 @@
__asm__(".arch_extension virt");
#endif
+DEFINE_PER_CPU(kvm_cpu_context_t, kvm_host_cpu_state);
static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
-static kvm_cpu_context_t __percpu *kvm_host_cpu_state;
static unsigned long hyp_default_vectors;
/* Per-CPU variable containing the currently running vcpu. */
@@ -338,7 +338,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
}
vcpu->cpu = cpu;
- vcpu->arch.host_cpu_context = this_cpu_ptr(kvm_host_cpu_state);
+ vcpu->arch.host_cpu_context = this_cpu_ptr(&kvm_host_cpu_state);
kvm_arm_set_running_vcpu(vcpu);
}
@@ -1092,8 +1092,6 @@ static void cpu_init_hyp_mode(void *dummy)
__cpu_init_hyp_mode(pgd_ptr, hyp_stack_ptr, vector_ptr);
__cpu_init_stage2();
-
- kvm_arm_init_debug();
}
static void cpu_hyp_reinit(void)
@@ -1108,6 +1106,8 @@ static void cpu_hyp_reinit(void)
if (__hyp_get_vectors() == hyp_default_vectors)
cpu_init_hyp_mode(NULL);
}
+
+ kvm_arm_init_debug();
}
static void cpu_hyp_reset(void)
@@ -1199,19 +1199,8 @@ static inline void hyp_cpu_pm_exit(void)
}
#endif
-static void teardown_common_resources(void)
-{
- free_percpu(kvm_host_cpu_state);
-}
-
static int init_common_resources(void)
{
- kvm_host_cpu_state = alloc_percpu(kvm_cpu_context_t);
- if (!kvm_host_cpu_state) {
- kvm_err("Cannot allocate host CPU state\n");
- return -ENOMEM;
- }
-
/* set size of VMID supported by CPU */
kvm_vmid_bits = kvm_get_vmid_bits();
kvm_info("%d-bit VMID\n", kvm_vmid_bits);
@@ -1369,7 +1358,7 @@ static int init_hyp_mode(void)
for_each_possible_cpu(cpu) {
kvm_cpu_context_t *cpu_ctxt;
- cpu_ctxt = per_cpu_ptr(kvm_host_cpu_state, cpu);
+ cpu_ctxt = per_cpu_ptr(&kvm_host_cpu_state, cpu);
err = create_hyp_mappings(cpu_ctxt, cpu_ctxt + 1, PAGE_HYP);
if (err) {
@@ -1378,6 +1367,12 @@ static int init_hyp_mode(void)
}
}
+ err = hyp_map_aux_data();
+ if (err) {
+ kvm_err("Cannot map host auxilary data: %d\n", err);
+ goto out_err;
+ }
+
kvm_info("Hyp mode initialized successfully\n");
return 0;
@@ -1447,7 +1442,6 @@ int kvm_arch_init(void *opaque)
out_hyp:
teardown_hyp_mode();
out_err:
- teardown_common_resources();
return err;
}
diff --git a/arch/arm/kvm/guest.c b/arch/arm/kvm/guest.c
index 9aca920..630117d 100644
--- a/arch/arm/kvm/guest.c
+++ b/arch/arm/kvm/guest.c
@@ -22,6 +22,7 @@
#include <linux/module.h>
#include <linux/vmalloc.h>
#include <linux/fs.h>
+#include <kvm/arm_psci.h>
#include <asm/cputype.h>
#include <asm/uaccess.h>
#include <asm/kvm.h>
@@ -176,6 +177,7 @@ static unsigned long num_core_regs(void)
unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu)
{
return num_core_regs() + kvm_arm_num_coproc_regs(vcpu)
+ + kvm_arm_get_fw_num_regs(vcpu)
+ NUM_TIMER_REGS;
}
@@ -196,6 +198,11 @@ int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
uindices++;
}
+ ret = kvm_arm_copy_fw_reg_indices(vcpu, uindices);
+ if (ret)
+ return ret;
+ uindices += kvm_arm_get_fw_num_regs(vcpu);
+
ret = copy_timer_indices(vcpu, uindices);
if (ret)
return ret;
@@ -214,6 +221,9 @@ int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_CORE)
return get_core_reg(vcpu, reg);
+ if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_FW)
+ return kvm_arm_get_fw_reg(vcpu, reg);
+
if (is_timer_reg(reg->id))
return get_timer_reg(vcpu, reg);
@@ -230,6 +240,9 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_CORE)
return set_core_reg(vcpu, reg);
+ if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_FW)
+ return kvm_arm_set_fw_reg(vcpu, reg);
+
if (is_timer_reg(reg->id))
return set_timer_reg(vcpu, reg);
diff --git a/arch/arm/kvm/hyp/hyp-entry.S b/arch/arm/kvm/hyp/hyp-entry.S
index 96beb53..64d4a39 100644
--- a/arch/arm/kvm/hyp/hyp-entry.S
+++ b/arch/arm/kvm/hyp/hyp-entry.S
@@ -16,6 +16,7 @@
* Foundation, 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
+#include <linux/arm-smccc.h>
#include <linux/linkage.h>
#include <asm/kvm_arm.h>
#include <asm/kvm_asm.h>
@@ -71,6 +72,90 @@
W(b) hyp_irq
W(b) hyp_fiq
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+ .align 5
+__kvm_hyp_vector_ic_inv:
+ .global __kvm_hyp_vector_ic_inv
+
+ /*
+ * We encode the exception entry in the bottom 3 bits of
+ * SP, and we have to guarantee to be 8 bytes aligned.
+ */
+ W(add) sp, sp, #1 /* Reset 7 */
+ W(add) sp, sp, #1 /* Undef 6 */
+ W(add) sp, sp, #1 /* Syscall 5 */
+ W(add) sp, sp, #1 /* Prefetch abort 4 */
+ W(add) sp, sp, #1 /* Data abort 3 */
+ W(add) sp, sp, #1 /* HVC 2 */
+ W(add) sp, sp, #1 /* IRQ 1 */
+ W(nop) /* FIQ 0 */
+
+ mcr p15, 0, r0, c7, c5, 0 /* ICIALLU */
+ isb
+
+ b decode_vectors
+
+ .align 5
+__kvm_hyp_vector_bp_inv:
+ .global __kvm_hyp_vector_bp_inv
+
+ /*
+ * We encode the exception entry in the bottom 3 bits of
+ * SP, and we have to guarantee to be 8 bytes aligned.
+ */
+ W(add) sp, sp, #1 /* Reset 7 */
+ W(add) sp, sp, #1 /* Undef 6 */
+ W(add) sp, sp, #1 /* Syscall 5 */
+ W(add) sp, sp, #1 /* Prefetch abort 4 */
+ W(add) sp, sp, #1 /* Data abort 3 */
+ W(add) sp, sp, #1 /* HVC 2 */
+ W(add) sp, sp, #1 /* IRQ 1 */
+ W(nop) /* FIQ 0 */
+
+ mcr p15, 0, r0, c7, c5, 6 /* BPIALL */
+ isb
+
+decode_vectors:
+
+#ifdef CONFIG_THUMB2_KERNEL
+ /*
+ * Yet another silly hack: Use VPIDR as a temp register.
+ * Thumb2 is really a pain, as SP cannot be used with most
+ * of the bitwise instructions. The vect_br macro ensures
+ * things gets cleaned-up.
+ */
+ mcr p15, 4, r0, c0, c0, 0 /* VPIDR */
+ mov r0, sp
+ and r0, r0, #7
+ sub sp, sp, r0
+ push {r1, r2}
+ mov r1, r0
+ mrc p15, 4, r0, c0, c0, 0 /* VPIDR */
+ mrc p15, 0, r2, c0, c0, 0 /* MIDR */
+ mcr p15, 4, r2, c0, c0, 0 /* VPIDR */
+#endif
+
+.macro vect_br val, targ
+ARM( eor sp, sp, #\val )
+ARM( tst sp, #7 )
+ARM( eorne sp, sp, #\val )
+
+THUMB( cmp r1, #\val )
+THUMB( popeq {r1, r2} )
+
+ beq \targ
+.endm
+
+ vect_br 0, hyp_fiq
+ vect_br 1, hyp_irq
+ vect_br 2, hyp_hvc
+ vect_br 3, hyp_dabt
+ vect_br 4, hyp_pabt
+ vect_br 5, hyp_svc
+ vect_br 6, hyp_undef
+ vect_br 7, hyp_reset
+#endif
+
.macro invalid_vector label, cause
.align
\label: mov r0, #\cause
@@ -118,7 +203,7 @@
lsr r2, r2, #16
and r2, r2, #0xff
cmp r2, #0
- bne guest_trap @ Guest called HVC
+ bne guest_hvc_trap @ Guest called HVC
/*
* Getting here means host called HVC, we shift parameters and branch
@@ -131,7 +216,14 @@
mrceq p15, 4, r0, c12, c0, 0 @ get HVBAR
beq 1f
- push {lr}
+ /*
+ * Pushing r2 here is just a way of keeping the stack aligned to
+ * 8 bytes on any path that can trigger a HYP exception. Here,
+ * we may well be about to jump into the guest, and the guest
+ * exit would otherwise be badly decoded by our fancy
+ * "decode-exception-without-a-branch" code...
+ */
+ push {r2, lr}
mov lr, r0
mov r0, r1
@@ -141,9 +233,23 @@
THUMB( orr lr, #1)
blx lr @ Call the HYP function
- pop {lr}
+ pop {r2, lr}
1: eret
+guest_hvc_trap:
+ movw r2, #:lower16:ARM_SMCCC_ARCH_WORKAROUND_1
+ movt r2, #:upper16:ARM_SMCCC_ARCH_WORKAROUND_1
+ ldr r0, [sp] @ Guest's r0
+ teq r0, r2
+ bne guest_trap
+ add sp, sp, #12
+ @ Returns:
+ @ r0 = 0
+ @ r1 = HSR value (perfectly predictable)
+ @ r2 = ARM_SMCCC_ARCH_WORKAROUND_1
+ mov r0, #0
+ eret
+
guest_trap:
load_vcpu r0 @ Load VCPU pointer to r0
diff --git a/arch/arm/kvm/mmu.c b/arch/arm/kvm/mmu.c
index 7f868d9..b3d268a 100644
--- a/arch/arm/kvm/mmu.c
+++ b/arch/arm/kvm/mmu.c
@@ -894,19 +894,35 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache
pmd = stage2_get_pmd(kvm, cache, addr);
VM_BUG_ON(!pmd);
- /*
- * Mapping in huge pages should only happen through a fault. If a
- * page is merged into a transparent huge page, the individual
- * subpages of that huge page should be unmapped through MMU
- * notifiers before we get here.
- *
- * Merging of CompoundPages is not supported; they should become
- * splitting first, unmapped, merged, and mapped back in on-demand.
- */
- VM_BUG_ON(pmd_present(*pmd) && pmd_pfn(*pmd) != pmd_pfn(*new_pmd));
-
old_pmd = *pmd;
if (pmd_present(old_pmd)) {
+ /*
+ * Multiple vcpus faulting on the same PMD entry, can
+ * lead to them sequentially updating the PMD with the
+ * same value. Following the break-before-make
+ * (pmd_clear() followed by tlb_flush()) process can
+ * hinder forward progress due to refaults generated
+ * on missing translations.
+ *
+ * Skip updating the page table if the entry is
+ * unchanged.
+ */
+ if (pmd_val(old_pmd) == pmd_val(*new_pmd))
+ return 0;
+
+ /*
+ * Mapping in huge pages should only happen through a
+ * fault. If a page is merged into a transparent huge
+ * page, the individual subpages of that huge page
+ * should be unmapped through MMU notifiers before we
+ * get here.
+ *
+ * Merging of CompoundPages is not supported; they
+ * should become splitting first, unmapped, merged,
+ * and mapped back in on-demand.
+ */
+ VM_BUG_ON(pmd_pfn(old_pmd) != pmd_pfn(*new_pmd));
+
pmd_clear(pmd);
kvm_tlb_flush_vmid_ipa(kvm, addr);
} else {
@@ -962,6 +978,10 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache,
/* Create 2nd stage page table mapping - Level 3 */
old_pte = *pte;
if (pte_present(old_pte)) {
+ /* Skip page table update if there is no change */
+ if (pte_val(old_pte) == pte_val(*new_pte))
+ return 0;
+
kvm_set_pte(pte, __pte(0));
kvm_tlb_flush_vmid_ipa(kvm, addr);
} else {
diff --git a/arch/arm/kvm/psci.c b/arch/arm/kvm/psci.c
index 3d96225..83365be 100644
--- a/arch/arm/kvm/psci.c
+++ b/arch/arm/kvm/psci.c
@@ -18,6 +18,7 @@
#include <linux/arm-smccc.h>
#include <linux/preempt.h>
#include <linux/kvm_host.h>
+#include <linux/uaccess.h>
#include <linux/wait.h>
#include <asm/cputype.h>
@@ -402,7 +403,7 @@ static int kvm_psci_call(struct kvm_vcpu *vcpu)
int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
{
u32 func_id = smccc_get_function(vcpu);
- u32 val = PSCI_RET_NOT_SUPPORTED;
+ u32 val = SMCCC_RET_NOT_SUPPORTED;
u32 feature;
switch (func_id) {
@@ -414,7 +415,21 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
switch(feature) {
case ARM_SMCCC_ARCH_WORKAROUND_1:
if (kvm_arm_harden_branch_predictor())
- val = 0;
+ val = SMCCC_RET_SUCCESS;
+ break;
+ case ARM_SMCCC_ARCH_WORKAROUND_2:
+ switch (kvm_arm_have_ssbd()) {
+ case KVM_SSBD_FORCE_DISABLE:
+ case KVM_SSBD_UNKNOWN:
+ break;
+ case KVM_SSBD_KERNEL:
+ val = SMCCC_RET_SUCCESS;
+ break;
+ case KVM_SSBD_FORCE_ENABLE:
+ case KVM_SSBD_MITIGATED:
+ val = SMCCC_RET_NOT_REQUIRED;
+ break;
+ }
break;
}
break;
@@ -425,3 +440,62 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
smccc_set_retval(vcpu, val, 0, 0, 0);
return 1;
}
+
+int kvm_arm_get_fw_num_regs(struct kvm_vcpu *vcpu)
+{
+ return 1; /* PSCI version */
+}
+
+int kvm_arm_copy_fw_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
+{
+ if (put_user(KVM_REG_ARM_PSCI_VERSION, uindices))
+ return -EFAULT;
+
+ return 0;
+}
+
+int kvm_arm_get_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
+{
+ if (reg->id == KVM_REG_ARM_PSCI_VERSION) {
+ void __user *uaddr = (void __user *)(long)reg->addr;
+ u64 val;
+
+ val = kvm_psci_version(vcpu, vcpu->kvm);
+ if (copy_to_user(uaddr, &val, KVM_REG_SIZE(reg->id)))
+ return -EFAULT;
+
+ return 0;
+ }
+
+ return -EINVAL;
+}
+
+int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
+{
+ if (reg->id == KVM_REG_ARM_PSCI_VERSION) {
+ void __user *uaddr = (void __user *)(long)reg->addr;
+ bool wants_02;
+ u64 val;
+
+ if (copy_from_user(&val, uaddr, KVM_REG_SIZE(reg->id)))
+ return -EFAULT;
+
+ wants_02 = test_bit(KVM_ARM_VCPU_PSCI_0_2, vcpu->arch.features);
+
+ switch (val) {
+ case KVM_ARM_PSCI_0_1:
+ if (wants_02)
+ return -EINVAL;
+ vcpu->kvm->arch.psci_version = val;
+ return 0;
+ case KVM_ARM_PSCI_0_2:
+ case KVM_ARM_PSCI_1_0:
+ if (!wants_02)
+ return -EINVAL;
+ vcpu->kvm->arch.psci_version = val;
+ return 0;
+ }
+ }
+
+ return -EINVAL;
+}
diff --git a/arch/arm/lib/copy_from_user.S b/arch/arm/lib/copy_from_user.S
index 7a4b060..a826df3 100644
--- a/arch/arm/lib/copy_from_user.S
+++ b/arch/arm/lib/copy_from_user.S
@@ -90,6 +90,15 @@
.text
ENTRY(arm_copy_from_user)
+#ifdef CONFIG_CPU_SPECTRE
+ get_thread_info r3
+ ldr r3, [r3, #TI_ADDR_LIMIT]
+ adds ip, r1, r2 @ ip=addr+size
+ sub r3, r3, #1 @ addr_limit - 1
+ cmpcc ip, r3 @ if (addr+size > addr_limit - 1)
+ movcs r1, #0 @ addr = NULL
+ csdb
+#endif
#include "copy_template.S"
diff --git a/arch/arm/lib/getuser.S b/arch/arm/lib/getuser.S
index df73914..746e780 100644
--- a/arch/arm/lib/getuser.S
+++ b/arch/arm/lib/getuser.S
@@ -38,6 +38,7 @@
mov r0, #0
ret lr
ENDPROC(__get_user_1)
+_ASM_NOKPROBE(__get_user_1)
ENTRY(__get_user_2)
check_uaccess r0, 2, r1, r2, __get_user_bad
@@ -58,6 +59,7 @@
mov r0, #0
ret lr
ENDPROC(__get_user_2)
+_ASM_NOKPROBE(__get_user_2)
ENTRY(__get_user_4)
check_uaccess r0, 4, r1, r2, __get_user_bad
@@ -65,6 +67,7 @@
mov r0, #0
ret lr
ENDPROC(__get_user_4)
+_ASM_NOKPROBE(__get_user_4)
ENTRY(__get_user_8)
check_uaccess r0, 8, r1, r2, __get_user_bad8
@@ -78,6 +81,7 @@
mov r0, #0
ret lr
ENDPROC(__get_user_8)
+_ASM_NOKPROBE(__get_user_8)
#ifdef __ARMEB__
ENTRY(__get_user_32t_8)
@@ -91,6 +95,7 @@
mov r0, #0
ret lr
ENDPROC(__get_user_32t_8)
+_ASM_NOKPROBE(__get_user_32t_8)
ENTRY(__get_user_64t_1)
check_uaccess r0, 1, r1, r2, __get_user_bad8
@@ -98,6 +103,7 @@
mov r0, #0
ret lr
ENDPROC(__get_user_64t_1)
+_ASM_NOKPROBE(__get_user_64t_1)
ENTRY(__get_user_64t_2)
check_uaccess r0, 2, r1, r2, __get_user_bad8
@@ -114,6 +120,7 @@
mov r0, #0
ret lr
ENDPROC(__get_user_64t_2)
+_ASM_NOKPROBE(__get_user_64t_2)
ENTRY(__get_user_64t_4)
check_uaccess r0, 4, r1, r2, __get_user_bad8
@@ -121,6 +128,7 @@
mov r0, #0
ret lr
ENDPROC(__get_user_64t_4)
+_ASM_NOKPROBE(__get_user_64t_4)
#endif
__get_user_bad8:
@@ -131,6 +139,8 @@
ret lr
ENDPROC(__get_user_bad)
ENDPROC(__get_user_bad8)
+_ASM_NOKPROBE(__get_user_bad)
+_ASM_NOKPROBE(__get_user_bad8)
.pushsection __ex_table, "a"
.long 1b, __get_user_bad
diff --git a/arch/arm/mach-exynos/suspend.c b/arch/arm/mach-exynos/suspend.c
index 06332f6..3e1430a 100644
--- a/arch/arm/mach-exynos/suspend.c
+++ b/arch/arm/mach-exynos/suspend.c
@@ -252,6 +252,7 @@ static int __init exynos_pmu_irq_init(struct device_node *node,
NULL);
if (!domain) {
iounmap(pmu_base_addr);
+ pmu_base_addr = NULL;
return -ENOMEM;
}
diff --git a/arch/arm/mach-hisi/hotplug.c b/arch/arm/mach-hisi/hotplug.c
index a129aae..909bb24 100644
--- a/arch/arm/mach-hisi/hotplug.c
+++ b/arch/arm/mach-hisi/hotplug.c
@@ -148,13 +148,20 @@ static int hi3xxx_hotplug_init(void)
struct device_node *node;
node = of_find_compatible_node(NULL, NULL, "hisilicon,sysctrl");
- if (node) {
- ctrl_base = of_iomap(node, 0);
- id = HI3620_CTRL;
- return 0;
+ if (!node) {
+ id = ERROR_CTRL;
+ return -ENOENT;
}
- id = ERROR_CTRL;
- return -ENOENT;
+
+ ctrl_base = of_iomap(node, 0);
+ of_node_put(node);
+ if (!ctrl_base) {
+ id = ERROR_CTRL;
+ return -ENOMEM;
+ }
+
+ id = HI3620_CTRL;
+ return 0;
}
void hi3xxx_set_cpu(int cpu, bool enable)
@@ -173,11 +180,15 @@ static bool hix5hd2_hotplug_init(void)
struct device_node *np;
np = of_find_compatible_node(NULL, NULL, "hisilicon,cpuctrl");
- if (np) {
- ctrl_base = of_iomap(np, 0);
- return true;
- }
- return false;
+ if (!np)
+ return false;
+
+ ctrl_base = of_iomap(np, 0);
+ of_node_put(np);
+ if (!ctrl_base)
+ return false;
+
+ return true;
}
void hix5hd2_set_cpu(int cpu, bool enable)
@@ -219,10 +230,10 @@ void hip01_set_cpu(int cpu, bool enable)
if (!ctrl_base) {
np = of_find_compatible_node(NULL, NULL, "hisilicon,hip01-sysctrl");
- if (np)
- ctrl_base = of_iomap(np, 0);
- else
- BUG();
+ BUG_ON(!np);
+ ctrl_base = of_iomap(np, 0);
+ of_node_put(np);
+ BUG_ON(!ctrl_base);
}
if (enable) {
diff --git a/arch/arm/mach-imx/cpuidle-imx6sx.c b/arch/arm/mach-imx/cpuidle-imx6sx.c
index c5a5c3a..edb888a 100644
--- a/arch/arm/mach-imx/cpuidle-imx6sx.c
+++ b/arch/arm/mach-imx/cpuidle-imx6sx.c
@@ -108,7 +108,7 @@ int __init imx6sx_cpuidle_init(void)
* except for power up sw2iso which need to be
* larger than LDO ramp up time.
*/
- imx_gpc_set_arm_power_up_timing(2, 1);
+ imx_gpc_set_arm_power_up_timing(0xf, 1);
imx_gpc_set_arm_power_down_timing(1, 1);
return cpuidle_register(&imx6sx_cpuidle_driver, NULL);
diff --git a/arch/arm/mach-mmp/cputype.h b/arch/arm/mach-mmp/cputype.h
index 8a3b56d..8f94add 100644
--- a/arch/arm/mach-mmp/cputype.h
+++ b/arch/arm/mach-mmp/cputype.h
@@ -43,10 +43,12 @@ static inline int cpu_is_pxa910(void)
#define cpu_is_pxa910() (0)
#endif
-#ifdef CONFIG_CPU_MMP2
+#if defined(CONFIG_CPU_MMP2) || defined(CONFIG_MACH_MMP2_DT)
static inline int cpu_is_mmp2(void)
{
- return (((read_cpuid_id() >> 8) & 0xff) == 0x58);
+ return (((read_cpuid_id() >> 8) & 0xff) == 0x58) &&
+ (((mmp_chip_id & 0xfff) == 0x410) ||
+ ((mmp_chip_id & 0xfff) == 0x610));
}
#else
#define cpu_is_mmp2() (0)
diff --git a/arch/arm/mach-mvebu/pmsu.c b/arch/arm/mach-mvebu/pmsu.c
index f39bd51..faaf7c3 100644
--- a/arch/arm/mach-mvebu/pmsu.c
+++ b/arch/arm/mach-mvebu/pmsu.c
@@ -116,8 +116,8 @@ void mvebu_pmsu_set_cpu_boot_addr(int hw_cpu, void *boot_addr)
PMSU_BOOT_ADDR_REDIRECT_OFFSET(hw_cpu));
}
-extern unsigned char mvebu_boot_wa_start;
-extern unsigned char mvebu_boot_wa_end;
+extern unsigned char mvebu_boot_wa_start[];
+extern unsigned char mvebu_boot_wa_end[];
/*
* This function sets up the boot address workaround needed for SMP
@@ -130,7 +130,7 @@ int mvebu_setup_boot_addr_wa(unsigned int crypto_eng_target,
phys_addr_t resume_addr_reg)
{
void __iomem *sram_virt_base;
- u32 code_len = &mvebu_boot_wa_end - &mvebu_boot_wa_start;
+ u32 code_len = mvebu_boot_wa_end - mvebu_boot_wa_start;
mvebu_mbus_del_window(BOOTROM_BASE, BOOTROM_SIZE);
mvebu_mbus_add_window_by_id(crypto_eng_target, crypto_eng_attribute,
diff --git a/arch/arm/mach-omap1/board-ams-delta.c b/arch/arm/mach-omap1/board-ams-delta.c
index 6613a6f..c4b634c 100644
--- a/arch/arm/mach-omap1/board-ams-delta.c
+++ b/arch/arm/mach-omap1/board-ams-delta.c
@@ -511,6 +511,9 @@ static void modem_pm(struct uart_port *port, unsigned int state, unsigned old)
{
struct modem_private_data *priv = port->private_data;
+ if (!priv)
+ return;
+
if (IS_ERR(priv->regulator))
return;
diff --git a/arch/arm/mach-omap1/clock.c b/arch/arm/mach-omap1/clock.c
index 4f5fd4a..034b894 100644
--- a/arch/arm/mach-omap1/clock.c
+++ b/arch/arm/mach-omap1/clock.c
@@ -1031,17 +1031,17 @@ static int clk_debugfs_register_one(struct clk *c)
return -ENOMEM;
c->dent = d;
- d = debugfs_create_u8("usecount", S_IRUGO, c->dent, (u8 *)&c->usecount);
+ d = debugfs_create_u8("usecount", S_IRUGO, c->dent, &c->usecount);
if (!d) {
err = -ENOMEM;
goto err_out;
}
- d = debugfs_create_u32("rate", S_IRUGO, c->dent, (u32 *)&c->rate);
+ d = debugfs_create_ulong("rate", S_IRUGO, c->dent, &c->rate);
if (!d) {
err = -ENOMEM;
goto err_out;
}
- d = debugfs_create_x32("flags", S_IRUGO, c->dent, (u32 *)&c->flags);
+ d = debugfs_create_x8("flags", S_IRUGO, c->dent, &c->flags);
if (!d) {
err = -ENOMEM;
goto err_out;
diff --git a/arch/arm/mach-omap2/omap-smp.c b/arch/arm/mach-omap2/omap-smp.c
index b4de3da..f7f36da 100644
--- a/arch/arm/mach-omap2/omap-smp.c
+++ b/arch/arm/mach-omap2/omap-smp.c
@@ -104,6 +104,45 @@ void omap5_erratum_workaround_801819(void)
static inline void omap5_erratum_workaround_801819(void) { }
#endif
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+/*
+ * Configure ACR and enable ACTLR[0] (Enable invalidates of BTB with
+ * ICIALLU) to activate the workaround for secondary Core.
+ * NOTE: it is assumed that the primary core's configuration is done
+ * by the boot loader (kernel will detect a misconfiguration and complain
+ * if this is not done).
+ *
+ * In General Purpose(GP) devices, ACR bit settings can only be done
+ * by ROM code in "secure world" using the smc call and there is no
+ * option to update the "firmware" on such devices. This also works for
+ * High security(HS) devices, as a backup option in case the
+ * "update" is not done in the "security firmware".
+ */
+static void omap5_secondary_harden_predictor(void)
+{
+ u32 acr, acr_mask;
+
+ asm volatile ("mrc p15, 0, %0, c1, c0, 1" : "=r" (acr));
+
+ /*
+ * ACTLR[0] (Enable invalidates of BTB with ICIALLU)
+ */
+ acr_mask = BIT(0);
+
+ /* Do we already have it done.. if yes, skip expensive smc */
+ if ((acr & acr_mask) == acr_mask)
+ return;
+
+ acr |= acr_mask;
+ omap_smc1(OMAP5_DRA7_MON_SET_ACR_INDEX, acr);
+
+ pr_debug("%s: ARM ACR setup for CVE_2017_5715 applied on CPU%d\n",
+ __func__, smp_processor_id());
+}
+#else
+static inline void omap5_secondary_harden_predictor(void) { }
+#endif
+
static void omap4_secondary_init(unsigned int cpu)
{
/*
@@ -126,6 +165,8 @@ static void omap4_secondary_init(unsigned int cpu)
set_cntfreq();
/* Configure ACR to disable streaming WA for 801819 */
omap5_erratum_workaround_801819();
+ /* Enable ACR to allow for ICUALLU workaround */
+ omap5_secondary_harden_predictor();
}
/*
diff --git a/arch/arm/mach-omap2/omap_hwmod_reset.c b/arch/arm/mach-omap2/omap_hwmod_reset.c
index b68f9c0..d5ddba0 100644
--- a/arch/arm/mach-omap2/omap_hwmod_reset.c
+++ b/arch/arm/mach-omap2/omap_hwmod_reset.c
@@ -92,11 +92,13 @@ static void omap_rtc_wait_not_busy(struct omap_hwmod *oh)
*/
void omap_hwmod_rtc_unlock(struct omap_hwmod *oh)
{
- local_irq_disable();
+ unsigned long flags;
+
+ local_irq_save(flags);
omap_rtc_wait_not_busy(oh);
omap_hwmod_write(OMAP_RTC_KICK0_VALUE, oh, OMAP_RTC_KICK0_REG);
omap_hwmod_write(OMAP_RTC_KICK1_VALUE, oh, OMAP_RTC_KICK1_REG);
- local_irq_enable();
+ local_irq_restore(flags);
}
/**
@@ -110,9 +112,11 @@ void omap_hwmod_rtc_unlock(struct omap_hwmod *oh)
*/
void omap_hwmod_rtc_lock(struct omap_hwmod *oh)
{
- local_irq_disable();
+ unsigned long flags;
+
+ local_irq_save(flags);
omap_rtc_wait_not_busy(oh);
omap_hwmod_write(0x0, oh, OMAP_RTC_KICK0_REG);
omap_hwmod_write(0x0, oh, OMAP_RTC_KICK1_REG);
- local_irq_enable();
+ local_irq_restore(flags);
}
diff --git a/arch/arm/mach-omap2/pm.c b/arch/arm/mach-omap2/pm.c
index 678d2a3..3202015 100644
--- a/arch/arm/mach-omap2/pm.c
+++ b/arch/arm/mach-omap2/pm.c
@@ -225,7 +225,7 @@ static void omap_pm_end(void)
cpu_idle_poll_ctrl(false);
}
-static void omap_pm_finish(void)
+static void omap_pm_wake(void)
{
if (cpu_is_omap34xx())
omap_prcm_irq_complete();
@@ -235,7 +235,7 @@ static const struct platform_suspend_ops omap_pm_ops = {
.begin = omap_pm_begin,
.end = omap_pm_end,
.enter = omap_pm_enter,
- .finish = omap_pm_finish,
+ .wake = omap_pm_wake,
.valid = suspend_valid_only_mem,
};
diff --git a/arch/arm/mach-omap2/prm44xx.c b/arch/arm/mach-omap2/prm44xx.c
index 3076800..8c50528 100644
--- a/arch/arm/mach-omap2/prm44xx.c
+++ b/arch/arm/mach-omap2/prm44xx.c
@@ -344,7 +344,7 @@ static void omap44xx_prm_reconfigure_io_chain(void)
* to occur, WAKEUPENABLE bits must be set in the pad mux registers, and
* omap44xx_prm_reconfigure_io_chain() must be called. No return value.
*/
-static void __init omap44xx_prm_enable_io_wakeup(void)
+static void omap44xx_prm_enable_io_wakeup(void)
{
s32 inst = omap4_prmst_get_prm_dev_inst();
diff --git a/arch/arm/mach-omap2/timer.c b/arch/arm/mach-omap2/timer.c
index b2f2448..a4cab28 100644
--- a/arch/arm/mach-omap2/timer.c
+++ b/arch/arm/mach-omap2/timer.c
@@ -136,12 +136,6 @@ static struct clock_event_device clockevent_gpt = {
.tick_resume = omap2_gp_timer_shutdown,
};
-static struct property device_disabled = {
- .name = "status",
- .length = sizeof("disabled"),
- .value = "disabled",
-};
-
static const struct of_device_id omap_timer_match[] __initconst = {
{ .compatible = "ti,omap2420-timer", },
{ .compatible = "ti,omap3430-timer", },
@@ -183,8 +177,17 @@ static struct device_node * __init omap_get_timer_dt(const struct of_device_id *
of_get_property(np, "ti,timer-secure", NULL)))
continue;
- if (!of_device_is_compatible(np, "ti,omap-counter32k"))
- of_add_property(np, &device_disabled);
+ if (!of_device_is_compatible(np, "ti,omap-counter32k")) {
+ struct property *prop;
+
+ prop = kzalloc(sizeof(*prop), GFP_KERNEL);
+ if (!prop)
+ return NULL;
+ prop->name = "status";
+ prop->value = "disabled";
+ prop->length = strlen(prop->value);
+ of_add_property(np, prop);
+ }
return np;
}
diff --git a/arch/arm/mach-orion5x/Kconfig b/arch/arm/mach-orion5x/Kconfig
index 89bb0fc..72905a4 100644
--- a/arch/arm/mach-orion5x/Kconfig
+++ b/arch/arm/mach-orion5x/Kconfig
@@ -57,7 +57,6 @@
config MACH_DNS323
bool "D-Link DNS-323"
- select GENERIC_NET_UTILS
select I2C_BOARDINFO if I2C
help
Say 'Y' here if you want your kernel to support the
@@ -65,7 +64,6 @@
config MACH_TS209
bool "QNAP TS-109/TS-209"
- select GENERIC_NET_UTILS
help
Say 'Y' here if you want your kernel to support the
QNAP TS-109/TS-209 platform.
@@ -107,7 +105,6 @@
config MACH_TS409
bool "QNAP TS-409"
- select GENERIC_NET_UTILS
help
Say 'Y' here if you want your kernel to support the
QNAP TS-409 platform.
diff --git a/arch/arm/mach-orion5x/dns323-setup.c b/arch/arm/mach-orion5x/dns323-setup.c
index cd483bf..d13344b 100644
--- a/arch/arm/mach-orion5x/dns323-setup.c
+++ b/arch/arm/mach-orion5x/dns323-setup.c
@@ -173,10 +173,42 @@ static struct mv643xx_eth_platform_data dns323_eth_data = {
.phy_addr = MV643XX_ETH_PHY_ADDR(8),
};
+/* dns323_parse_hex_*() taken from tsx09-common.c; should a common copy of these
+ * functions be kept somewhere?
+ */
+static int __init dns323_parse_hex_nibble(char n)
+{
+ if (n >= '0' && n <= '9')
+ return n - '0';
+
+ if (n >= 'A' && n <= 'F')
+ return n - 'A' + 10;
+
+ if (n >= 'a' && n <= 'f')
+ return n - 'a' + 10;
+
+ return -1;
+}
+
+static int __init dns323_parse_hex_byte(const char *b)
+{
+ int hi;
+ int lo;
+
+ hi = dns323_parse_hex_nibble(b[0]);
+ lo = dns323_parse_hex_nibble(b[1]);
+
+ if (hi < 0 || lo < 0)
+ return -1;
+
+ return (hi << 4) | lo;
+}
+
static int __init dns323_read_mac_addr(void)
{
u_int8_t addr[6];
- void __iomem *mac_page;
+ int i;
+ char *mac_page;
/* MAC address is stored as a regular ol' string in /dev/mtdblock4
* (0x007d0000-0x00800000) starting at offset 196480 (0x2ff80).
@@ -185,8 +217,23 @@ static int __init dns323_read_mac_addr(void)
if (!mac_page)
return -ENOMEM;
- if (!mac_pton((__force const char *) mac_page, addr))
- goto error_fail;
+ /* Sanity check the string we're looking at */
+ for (i = 0; i < 5; i++) {
+ if (*(mac_page + (i * 3) + 2) != ':') {
+ goto error_fail;
+ }
+ }
+
+ for (i = 0; i < 6; i++) {
+ int byte;
+
+ byte = dns323_parse_hex_byte(mac_page + (i * 3));
+ if (byte < 0) {
+ goto error_fail;
+ }
+
+ addr[i] = byte;
+ }
iounmap(mac_page);
printk("DNS-323: Found ethernet MAC address: %pM\n", addr);
diff --git a/arch/arm/mach-orion5x/tsx09-common.c b/arch/arm/mach-orion5x/tsx09-common.c
index 8977498..905d4f2 100644
--- a/arch/arm/mach-orion5x/tsx09-common.c
+++ b/arch/arm/mach-orion5x/tsx09-common.c
@@ -53,12 +53,53 @@ struct mv643xx_eth_platform_data qnap_tsx09_eth_data = {
.phy_addr = MV643XX_ETH_PHY_ADDR(8),
};
+static int __init qnap_tsx09_parse_hex_nibble(char n)
+{
+ if (n >= '0' && n <= '9')
+ return n - '0';
+
+ if (n >= 'A' && n <= 'F')
+ return n - 'A' + 10;
+
+ if (n >= 'a' && n <= 'f')
+ return n - 'a' + 10;
+
+ return -1;
+}
+
+static int __init qnap_tsx09_parse_hex_byte(const char *b)
+{
+ int hi;
+ int lo;
+
+ hi = qnap_tsx09_parse_hex_nibble(b[0]);
+ lo = qnap_tsx09_parse_hex_nibble(b[1]);
+
+ if (hi < 0 || lo < 0)
+ return -1;
+
+ return (hi << 4) | lo;
+}
+
static int __init qnap_tsx09_check_mac_addr(const char *addr_str)
{
u_int8_t addr[6];
+ int i;
- if (!mac_pton(addr_str, addr))
- return -1;
+ for (i = 0; i < 6; i++) {
+ int byte;
+
+ /*
+ * Enforce "xx:xx:xx:xx:xx:xx\n" format.
+ */
+ if (addr_str[(i * 3) + 2] != ((i < 5) ? ':' : '\n'))
+ return -1;
+
+ byte = qnap_tsx09_parse_hex_byte(addr_str + (i * 3));
+ if (byte < 0)
+ return -1;
+ addr[i] = byte;
+ }
printk(KERN_INFO "tsx09: found ethernet mac address %pM\n", addr);
@@ -77,12 +118,12 @@ void __init qnap_tsx09_find_mac_addr(u32 mem_base, u32 size)
unsigned long addr;
for (addr = mem_base; addr < (mem_base + size); addr += 1024) {
- void __iomem *nor_page;
+ char *nor_page;
int ret = 0;
nor_page = ioremap(addr, 1024);
if (nor_page != NULL) {
- ret = qnap_tsx09_check_mac_addr((__force const char *)nor_page);
+ ret = qnap_tsx09_check_mac_addr(nor_page);
iounmap(nor_page);
}
diff --git a/arch/arm/mach-pxa/irq.c b/arch/arm/mach-pxa/irq.c
index 9c10248..4e8c211 100644
--- a/arch/arm/mach-pxa/irq.c
+++ b/arch/arm/mach-pxa/irq.c
@@ -185,7 +185,7 @@ static int pxa_irq_suspend(void)
{
int i;
- for (i = 0; i < pxa_internal_irq_nr / 32; i++) {
+ for (i = 0; i < DIV_ROUND_UP(pxa_internal_irq_nr, 32); i++) {
void __iomem *base = irq_base(i);
saved_icmr[i] = __raw_readl(base + ICMR);
@@ -204,7 +204,7 @@ static void pxa_irq_resume(void)
{
int i;
- for (i = 0; i < pxa_internal_irq_nr / 32; i++) {
+ for (i = 0; i < DIV_ROUND_UP(pxa_internal_irq_nr, 32); i++) {
void __iomem *base = irq_base(i);
__raw_writel(saved_icmr[i], base + ICMR);
diff --git a/arch/arm/mach-rockchip/Kconfig b/arch/arm/mach-rockchip/Kconfig
index 9ad84cd..5ed8fa5 100644
--- a/arch/arm/mach-rockchip/Kconfig
+++ b/arch/arm/mach-rockchip/Kconfig
@@ -16,6 +16,7 @@
select ROCKCHIP_TIMER
select ARM_GLOBAL_TIMER
select CLKSRC_ARM_GLOBAL_TIMER_SCHED_CLOCK
+ select PM
help
Support for Rockchip's Cortex-A9 Single-to-Quad-Core-SoCs
containing the RK2928, RK30xx and RK31xx series.
diff --git a/arch/arm/mm/Kconfig b/arch/arm/mm/Kconfig
index c1799dd..7f3760f 100644
--- a/arch/arm/mm/Kconfig
+++ b/arch/arm/mm/Kconfig
@@ -396,6 +396,7 @@
select CPU_CP15_MPU if !MMU
select CPU_HAS_ASID if MMU
select CPU_PABRT_V7
+ select CPU_SPECTRE if MMU
select CPU_TLB_V7 if MMU
# ARMv7M
@@ -800,6 +801,28 @@
help
Say Y here to disable branch prediction. If unsure, say N.
+config CPU_SPECTRE
+ bool
+
+config HARDEN_BRANCH_PREDICTOR
+ bool "Harden the branch predictor against aliasing attacks" if EXPERT
+ depends on CPU_SPECTRE
+ default y
+ help
+ Speculation attacks against some high-performance processors rely
+ on being able to manipulate the branch predictor for a victim
+ context by executing aliasing branches in the attacker context.
+ Such attacks can be partially mitigated against by clearing
+ internal branch predictor state and limiting the prediction
+ logic in some situations.
+
+ This config option will take CPU-specific actions to harden
+ the branch predictor against aliasing attacks and may rely on
+ specific instruction sequences or control bits being set by
+ the system firmware.
+
+ If unsure, say Y.
+
config TLS_REG_EMUL
bool
select NEED_KUSER_HELPERS
diff --git a/arch/arm/mm/Makefile b/arch/arm/mm/Makefile
index e869824..92d47c8 100644
--- a/arch/arm/mm/Makefile
+++ b/arch/arm/mm/Makefile
@@ -94,7 +94,7 @@
obj-$(CONFIG_CPU_FEROCEON) += proc-feroceon.o
obj-$(CONFIG_CPU_V6) += proc-v6.o
obj-$(CONFIG_CPU_V6K) += proc-v6.o
-obj-$(CONFIG_CPU_V7) += proc-v7.o
+obj-$(CONFIG_CPU_V7) += proc-v7.o proc-v7-bugs.o
obj-$(CONFIG_CPU_V7M) += proc-v7m.o
AFLAGS_proc-v6.o :=-Wa,-march=armv6
diff --git a/arch/arm/mm/cache-v7.S b/arch/arm/mm/cache-v7.S
index 2ddf364..08975bd 100644
--- a/arch/arm/mm/cache-v7.S
+++ b/arch/arm/mm/cache-v7.S
@@ -359,14 +359,16 @@
ALT_UP(W(nop))
#endif
mcrne p15, 0, r0, c7, c14, 1 @ clean & invalidate D / U line
+ addne r0, r0, r2
tst r1, r3
bic r1, r1, r3
mcrne p15, 0, r1, c7, c14, 1 @ clean & invalidate D / U line
-1:
- mcr p15, 0, r0, c7, c6, 1 @ invalidate D / U line
- add r0, r0, r2
cmp r0, r1
+1:
+ mcrlo p15, 0, r0, c7, c6, 1 @ invalidate D / U line
+ addlo r0, r0, r2
+ cmplo r0, r1
blo 1b
dsb st
ret lr
diff --git a/arch/arm/mm/cache-v7m.S b/arch/arm/mm/cache-v7m.S
index 816a7e4..d299277 100644
--- a/arch/arm/mm/cache-v7m.S
+++ b/arch/arm/mm/cache-v7m.S
@@ -73,9 +73,11 @@
/*
* dcimvac: Invalidate data cache line by MVA to PoC
*/
-.macro dcimvac, rt, tmp
- v7m_cacheop \rt, \tmp, V7M_SCB_DCIMVAC
+.irp c,,eq,ne,cs,cc,mi,pl,vs,vc,hi,ls,ge,lt,gt,le,hs,lo
+.macro dcimvac\c, rt, tmp
+ v7m_cacheop \rt, \tmp, V7M_SCB_DCIMVAC, \c
.endm
+.endr
/*
* dccmvau: Clean data cache line by MVA to PoU
@@ -369,14 +371,16 @@
tst r0, r3
bic r0, r0, r3
dccimvacne r0, r3
+ addne r0, r0, r2
subne r3, r2, #1 @ restore r3, corrupted by v7m's dccimvac
tst r1, r3
bic r1, r1, r3
dccimvacne r1, r3
-1:
- dcimvac r0, r3
- add r0, r0, r2
cmp r0, r1
+1:
+ dcimvaclo r0, r3
+ addlo r0, r0, r2
+ cmplo r0, r1
blo 1b
dsb st
ret lr
diff --git a/arch/arm/mm/fault.c b/arch/arm/mm/fault.c
index a9ef54d..d6c9dee 100644
--- a/arch/arm/mm/fault.c
+++ b/arch/arm/mm/fault.c
@@ -163,6 +163,9 @@ __do_user_fault(struct task_struct *tsk, unsigned long addr,
{
struct siginfo si;
+ if (addr > TASK_SIZE)
+ harden_branch_predictor();
+
#ifdef CONFIG_DEBUG_USER
if (((user_debug & UDBG_SEGV) && (sig == SIGSEGV)) ||
((user_debug & UDBG_BUS) && (sig == SIGBUS))) {
diff --git a/arch/arm/mm/init.c b/arch/arm/mm/init.c
index cae69148a..10a8099 100644
--- a/arch/arm/mm/init.c
+++ b/arch/arm/mm/init.c
@@ -822,19 +822,28 @@ int __mark_rodata_ro(void *unused)
return 0;
}
+static int kernel_set_to_readonly __read_mostly;
+
void mark_rodata_ro(void)
{
+ kernel_set_to_readonly = 1;
stop_machine(__mark_rodata_ro, NULL, NULL);
}
void set_kernel_text_rw(void)
{
+ if (!kernel_set_to_readonly)
+ return;
+
set_section_perms(ro_perms, ARRAY_SIZE(ro_perms), false,
current->active_mm);
}
void set_kernel_text_ro(void)
{
+ if (!kernel_set_to_readonly)
+ return;
+
set_section_perms(ro_perms, ARRAY_SIZE(ro_perms), true,
current->active_mm);
}
diff --git a/arch/arm/mm/ioremap.c b/arch/arm/mm/ioremap.c
index 203728d..3da63a6 100644
--- a/arch/arm/mm/ioremap.c
+++ b/arch/arm/mm/ioremap.c
@@ -474,7 +474,7 @@ void pci_ioremap_set_mem_type(int mem_type)
int pci_ioremap_io(unsigned int offset, phys_addr_t phys_addr)
{
- BUG_ON(offset + SZ_64K > IO_SPACE_LIMIT);
+ BUG_ON(offset + SZ_64K - 1 > IO_SPACE_LIMIT);
return ioremap_page_range(PCI_IO_VIRT_BASE + offset,
PCI_IO_VIRT_BASE + offset + SZ_64K,
diff --git a/arch/arm/mm/proc-macros.S b/arch/arm/mm/proc-macros.S
index 60bd916..53bf9ed 100644
--- a/arch/arm/mm/proc-macros.S
+++ b/arch/arm/mm/proc-macros.S
@@ -274,13 +274,14 @@
mcr p15, 0, ip, c7, c10, 4 @ data write barrier
.endm
-.macro define_processor_functions name:req, dabort:req, pabort:req, nommu=0, suspend=0
+.macro define_processor_functions name:req, dabort:req, pabort:req, nommu=0, suspend=0, bugs=0
.type \name\()_processor_functions, #object
.align 2
ENTRY(\name\()_processor_functions)
.word \dabort
.word \pabort
.word cpu_\name\()_proc_init
+ .word \bugs
.word cpu_\name\()_proc_fin
.word cpu_\name\()_reset
.word cpu_\name\()_do_idle
diff --git a/arch/arm/mm/proc-v7-2level.S b/arch/arm/mm/proc-v7-2level.S
index c6141a5..f8d45ad 100644
--- a/arch/arm/mm/proc-v7-2level.S
+++ b/arch/arm/mm/proc-v7-2level.S
@@ -41,11 +41,6 @@
* even on Cortex-A8 revisions not affected by 430973.
* If IBE is not set, the flush BTAC/BTB won't do anything.
*/
-ENTRY(cpu_ca8_switch_mm)
-#ifdef CONFIG_MMU
- mov r2, #0
- mcr p15, 0, r2, c7, c5, 6 @ flush BTAC/BTB
-#endif
ENTRY(cpu_v7_switch_mm)
#ifdef CONFIG_MMU
mmid r1, r1 @ get mm->context.id
@@ -66,7 +61,6 @@
#endif
bx lr
ENDPROC(cpu_v7_switch_mm)
-ENDPROC(cpu_ca8_switch_mm)
/*
* cpu_v7_set_pte_ext(ptep, pte)
diff --git a/arch/arm/mm/proc-v7-bugs.c b/arch/arm/mm/proc-v7-bugs.c
new file mode 100644
index 0000000..5544b82
--- /dev/null
+++ b/arch/arm/mm/proc-v7-bugs.c
@@ -0,0 +1,174 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/arm-smccc.h>
+#include <linux/kernel.h>
+#include <linux/psci.h>
+#include <linux/smp.h>
+
+#include <asm/cp15.h>
+#include <asm/cputype.h>
+#include <asm/proc-fns.h>
+#include <asm/system_misc.h>
+
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+DEFINE_PER_CPU(harden_branch_predictor_fn_t, harden_branch_predictor_fn);
+
+extern void cpu_v7_iciallu_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm);
+extern void cpu_v7_bpiall_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm);
+extern void cpu_v7_smc_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm);
+extern void cpu_v7_hvc_switch_mm(phys_addr_t pgd_phys, struct mm_struct *mm);
+
+static void harden_branch_predictor_bpiall(void)
+{
+ write_sysreg(0, BPIALL);
+}
+
+static void harden_branch_predictor_iciallu(void)
+{
+ write_sysreg(0, ICIALLU);
+}
+
+static void __maybe_unused call_smc_arch_workaround_1(void)
+{
+ arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_1, NULL);
+}
+
+static void __maybe_unused call_hvc_arch_workaround_1(void)
+{
+ arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_WORKAROUND_1, NULL);
+}
+
+static void cpu_v7_spectre_init(void)
+{
+ const char *spectre_v2_method = NULL;
+ int cpu = smp_processor_id();
+
+ if (per_cpu(harden_branch_predictor_fn, cpu))
+ return;
+
+ switch (read_cpuid_part()) {
+ case ARM_CPU_PART_CORTEX_A8:
+ case ARM_CPU_PART_CORTEX_A9:
+ case ARM_CPU_PART_CORTEX_A12:
+ case ARM_CPU_PART_CORTEX_A17:
+ case ARM_CPU_PART_CORTEX_A73:
+ case ARM_CPU_PART_CORTEX_A75:
+ if (processor.switch_mm != cpu_v7_bpiall_switch_mm)
+ goto bl_error;
+ per_cpu(harden_branch_predictor_fn, cpu) =
+ harden_branch_predictor_bpiall;
+ spectre_v2_method = "BPIALL";
+ break;
+
+ case ARM_CPU_PART_CORTEX_A15:
+ case ARM_CPU_PART_BRAHMA_B15:
+ if (processor.switch_mm != cpu_v7_iciallu_switch_mm)
+ goto bl_error;
+ per_cpu(harden_branch_predictor_fn, cpu) =
+ harden_branch_predictor_iciallu;
+ spectre_v2_method = "ICIALLU";
+ break;
+
+#ifdef CONFIG_ARM_PSCI
+ default:
+ /* Other ARM CPUs require no workaround */
+ if (read_cpuid_implementor() == ARM_CPU_IMP_ARM)
+ break;
+ /* fallthrough */
+ /* Cortex A57/A72 require firmware workaround */
+ case ARM_CPU_PART_CORTEX_A57:
+ case ARM_CPU_PART_CORTEX_A72: {
+ struct arm_smccc_res res;
+
+ if (psci_ops.smccc_version == SMCCC_VERSION_1_0)
+ break;
+
+ switch (psci_ops.conduit) {
+ case PSCI_CONDUIT_HVC:
+ arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID,
+ ARM_SMCCC_ARCH_WORKAROUND_1, &res);
+ if ((int)res.a0 != 0)
+ break;
+ if (processor.switch_mm != cpu_v7_hvc_switch_mm && cpu)
+ goto bl_error;
+ per_cpu(harden_branch_predictor_fn, cpu) =
+ call_hvc_arch_workaround_1;
+ processor.switch_mm = cpu_v7_hvc_switch_mm;
+ spectre_v2_method = "hypervisor";
+ break;
+
+ case PSCI_CONDUIT_SMC:
+ arm_smccc_1_1_smc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID,
+ ARM_SMCCC_ARCH_WORKAROUND_1, &res);
+ if ((int)res.a0 != 0)
+ break;
+ if (processor.switch_mm != cpu_v7_smc_switch_mm && cpu)
+ goto bl_error;
+ per_cpu(harden_branch_predictor_fn, cpu) =
+ call_smc_arch_workaround_1;
+ processor.switch_mm = cpu_v7_smc_switch_mm;
+ spectre_v2_method = "firmware";
+ break;
+
+ default:
+ break;
+ }
+ }
+#endif
+ }
+
+ if (spectre_v2_method)
+ pr_info("CPU%u: Spectre v2: using %s workaround\n",
+ smp_processor_id(), spectre_v2_method);
+ return;
+
+bl_error:
+ pr_err("CPU%u: Spectre v2: incorrect context switching function, system vulnerable\n",
+ cpu);
+}
+#else
+static void cpu_v7_spectre_init(void)
+{
+}
+#endif
+
+static __maybe_unused bool cpu_v7_check_auxcr_set(bool *warned,
+ u32 mask, const char *msg)
+{
+ u32 aux_cr;
+
+ asm("mrc p15, 0, %0, c1, c0, 1" : "=r" (aux_cr));
+
+ if ((aux_cr & mask) != mask) {
+ if (!*warned)
+ pr_err("CPU%u: %s", smp_processor_id(), msg);
+ *warned = true;
+ return false;
+ }
+ return true;
+}
+
+static DEFINE_PER_CPU(bool, spectre_warned);
+
+static bool check_spectre_auxcr(bool *warned, u32 bit)
+{
+ return IS_ENABLED(CONFIG_HARDEN_BRANCH_PREDICTOR) &&
+ cpu_v7_check_auxcr_set(warned, bit,
+ "Spectre v2: firmware did not set auxiliary control register IBE bit, system vulnerable\n");
+}
+
+void cpu_v7_ca8_ibe(void)
+{
+ if (check_spectre_auxcr(this_cpu_ptr(&spectre_warned), BIT(6)))
+ cpu_v7_spectre_init();
+}
+
+void cpu_v7_ca15_ibe(void)
+{
+ if (check_spectre_auxcr(this_cpu_ptr(&spectre_warned), BIT(0)))
+ cpu_v7_spectre_init();
+}
+
+void cpu_v7_bugs_init(void)
+{
+ cpu_v7_spectre_init();
+}
diff --git a/arch/arm/mm/proc-v7.S b/arch/arm/mm/proc-v7.S
index d00d52c..850c22b 100644
--- a/arch/arm/mm/proc-v7.S
+++ b/arch/arm/mm/proc-v7.S
@@ -9,6 +9,7 @@
*
* This is the "shell" of the ARMv7 processor support.
*/
+#include <linux/arm-smccc.h>
#include <linux/init.h>
#include <linux/linkage.h>
#include <asm/assembler.h>
@@ -88,6 +89,37 @@
ret lr
ENDPROC(cpu_v7_dcache_clean_area)
+#ifdef CONFIG_ARM_PSCI
+ .arch_extension sec
+ENTRY(cpu_v7_smc_switch_mm)
+ stmfd sp!, {r0 - r3}
+ movw r0, #:lower16:ARM_SMCCC_ARCH_WORKAROUND_1
+ movt r0, #:upper16:ARM_SMCCC_ARCH_WORKAROUND_1
+ smc #0
+ ldmfd sp!, {r0 - r3}
+ b cpu_v7_switch_mm
+ENDPROC(cpu_v7_smc_switch_mm)
+ .arch_extension virt
+ENTRY(cpu_v7_hvc_switch_mm)
+ stmfd sp!, {r0 - r3}
+ movw r0, #:lower16:ARM_SMCCC_ARCH_WORKAROUND_1
+ movt r0, #:upper16:ARM_SMCCC_ARCH_WORKAROUND_1
+ hvc #0
+ ldmfd sp!, {r0 - r3}
+ b cpu_v7_switch_mm
+ENDPROC(cpu_v7_hvc_switch_mm)
+#endif
+ENTRY(cpu_v7_iciallu_switch_mm)
+ mov r3, #0
+ mcr p15, 0, r3, c7, c5, 0 @ ICIALLU
+ b cpu_v7_switch_mm
+ENDPROC(cpu_v7_iciallu_switch_mm)
+ENTRY(cpu_v7_bpiall_switch_mm)
+ mov r3, #0
+ mcr p15, 0, r3, c7, c5, 6 @ flush BTAC/BTB
+ b cpu_v7_switch_mm
+ENDPROC(cpu_v7_bpiall_switch_mm)
+
string cpu_v7_name, "ARMv7 Processor"
.align
@@ -153,31 +185,6 @@
ENDPROC(cpu_v7_do_resume)
#endif
-/*
- * Cortex-A8
- */
- globl_equ cpu_ca8_proc_init, cpu_v7_proc_init
- globl_equ cpu_ca8_proc_fin, cpu_v7_proc_fin
- globl_equ cpu_ca8_reset, cpu_v7_reset
- globl_equ cpu_ca8_do_idle, cpu_v7_do_idle
- globl_equ cpu_ca8_dcache_clean_area, cpu_v7_dcache_clean_area
- globl_equ cpu_ca8_set_pte_ext, cpu_v7_set_pte_ext
- globl_equ cpu_ca8_suspend_size, cpu_v7_suspend_size
-#ifdef CONFIG_ARM_CPU_SUSPEND
- globl_equ cpu_ca8_do_suspend, cpu_v7_do_suspend
- globl_equ cpu_ca8_do_resume, cpu_v7_do_resume
-#endif
-
-/*
- * Cortex-A9 processor functions
- */
- globl_equ cpu_ca9mp_proc_init, cpu_v7_proc_init
- globl_equ cpu_ca9mp_proc_fin, cpu_v7_proc_fin
- globl_equ cpu_ca9mp_reset, cpu_v7_reset
- globl_equ cpu_ca9mp_do_idle, cpu_v7_do_idle
- globl_equ cpu_ca9mp_dcache_clean_area, cpu_v7_dcache_clean_area
- globl_equ cpu_ca9mp_switch_mm, cpu_v7_switch_mm
- globl_equ cpu_ca9mp_set_pte_ext, cpu_v7_set_pte_ext
.globl cpu_ca9mp_suspend_size
.equ cpu_ca9mp_suspend_size, cpu_v7_suspend_size + 4 * 2
#ifdef CONFIG_ARM_CPU_SUSPEND
@@ -541,12 +548,79 @@
__INITDATA
+ .weak cpu_v7_bugs_init
+
@ define struct processor (see <asm/proc-fns.h> and proc-macros.S)
- define_processor_functions v7, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
-#ifndef CONFIG_ARM_LPAE
- define_processor_functions ca8, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
- define_processor_functions ca9mp, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
+ define_processor_functions v7, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_bugs_init
+
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+ @ generic v7 bpiall on context switch
+ globl_equ cpu_v7_bpiall_proc_init, cpu_v7_proc_init
+ globl_equ cpu_v7_bpiall_proc_fin, cpu_v7_proc_fin
+ globl_equ cpu_v7_bpiall_reset, cpu_v7_reset
+ globl_equ cpu_v7_bpiall_do_idle, cpu_v7_do_idle
+ globl_equ cpu_v7_bpiall_dcache_clean_area, cpu_v7_dcache_clean_area
+ globl_equ cpu_v7_bpiall_set_pte_ext, cpu_v7_set_pte_ext
+ globl_equ cpu_v7_bpiall_suspend_size, cpu_v7_suspend_size
+#ifdef CONFIG_ARM_CPU_SUSPEND
+ globl_equ cpu_v7_bpiall_do_suspend, cpu_v7_do_suspend
+ globl_equ cpu_v7_bpiall_do_resume, cpu_v7_do_resume
#endif
+ define_processor_functions v7_bpiall, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_bugs_init
+
+#define HARDENED_BPIALL_PROCESSOR_FUNCTIONS v7_bpiall_processor_functions
+#else
+#define HARDENED_BPIALL_PROCESSOR_FUNCTIONS v7_processor_functions
+#endif
+
+#ifndef CONFIG_ARM_LPAE
+ @ Cortex-A8 - always needs bpiall switch_mm implementation
+ globl_equ cpu_ca8_proc_init, cpu_v7_proc_init
+ globl_equ cpu_ca8_proc_fin, cpu_v7_proc_fin
+ globl_equ cpu_ca8_reset, cpu_v7_reset
+ globl_equ cpu_ca8_do_idle, cpu_v7_do_idle
+ globl_equ cpu_ca8_dcache_clean_area, cpu_v7_dcache_clean_area
+ globl_equ cpu_ca8_set_pte_ext, cpu_v7_set_pte_ext
+ globl_equ cpu_ca8_switch_mm, cpu_v7_bpiall_switch_mm
+ globl_equ cpu_ca8_suspend_size, cpu_v7_suspend_size
+#ifdef CONFIG_ARM_CPU_SUSPEND
+ globl_equ cpu_ca8_do_suspend, cpu_v7_do_suspend
+ globl_equ cpu_ca8_do_resume, cpu_v7_do_resume
+#endif
+ define_processor_functions ca8, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_ca8_ibe
+
+ @ Cortex-A9 - needs more registers preserved across suspend/resume
+ @ and bpiall switch_mm for hardening
+ globl_equ cpu_ca9mp_proc_init, cpu_v7_proc_init
+ globl_equ cpu_ca9mp_proc_fin, cpu_v7_proc_fin
+ globl_equ cpu_ca9mp_reset, cpu_v7_reset
+ globl_equ cpu_ca9mp_do_idle, cpu_v7_do_idle
+ globl_equ cpu_ca9mp_dcache_clean_area, cpu_v7_dcache_clean_area
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+ globl_equ cpu_ca9mp_switch_mm, cpu_v7_bpiall_switch_mm
+#else
+ globl_equ cpu_ca9mp_switch_mm, cpu_v7_switch_mm
+#endif
+ globl_equ cpu_ca9mp_set_pte_ext, cpu_v7_set_pte_ext
+ define_processor_functions ca9mp, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_bugs_init
+#endif
+
+ @ Cortex-A15 - needs iciallu switch_mm for hardening
+ globl_equ cpu_ca15_proc_init, cpu_v7_proc_init
+ globl_equ cpu_ca15_proc_fin, cpu_v7_proc_fin
+ globl_equ cpu_ca15_reset, cpu_v7_reset
+ globl_equ cpu_ca15_do_idle, cpu_v7_do_idle
+ globl_equ cpu_ca15_dcache_clean_area, cpu_v7_dcache_clean_area
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+ globl_equ cpu_ca15_switch_mm, cpu_v7_iciallu_switch_mm
+#else
+ globl_equ cpu_ca15_switch_mm, cpu_v7_switch_mm
+#endif
+ globl_equ cpu_ca15_set_pte_ext, cpu_v7_set_pte_ext
+ globl_equ cpu_ca15_suspend_size, cpu_v7_suspend_size
+ globl_equ cpu_ca15_do_suspend, cpu_v7_do_suspend
+ globl_equ cpu_ca15_do_resume, cpu_v7_do_resume
+ define_processor_functions ca15, dabort=v7_early_abort, pabort=v7_pabort, suspend=1, bugs=cpu_v7_ca15_ibe
#ifdef CONFIG_CPU_PJ4B
define_processor_functions pj4b, dabort=v7_early_abort, pabort=v7_pabort, suspend=1
#endif
@@ -653,7 +727,7 @@
__v7_ca12mp_proc_info:
.long 0x410fc0d0
.long 0xff0ffff0
- __v7_proc __v7_ca12mp_proc_info, __v7_ca12mp_setup
+ __v7_proc __v7_ca12mp_proc_info, __v7_ca12mp_setup, proc_fns = HARDENED_BPIALL_PROCESSOR_FUNCTIONS
.size __v7_ca12mp_proc_info, . - __v7_ca12mp_proc_info
/*
@@ -663,7 +737,7 @@
__v7_ca15mp_proc_info:
.long 0x410fc0f0
.long 0xff0ffff0
- __v7_proc __v7_ca15mp_proc_info, __v7_ca15mp_setup
+ __v7_proc __v7_ca15mp_proc_info, __v7_ca15mp_setup, proc_fns = ca15_processor_functions
.size __v7_ca15mp_proc_info, . - __v7_ca15mp_proc_info
/*
@@ -673,7 +747,7 @@
__v7_b15mp_proc_info:
.long 0x420f00f0
.long 0xff0ffff0
- __v7_proc __v7_b15mp_proc_info, __v7_b15mp_setup
+ __v7_proc __v7_b15mp_proc_info, __v7_b15mp_setup, proc_fns = ca15_processor_functions
.size __v7_b15mp_proc_info, . - __v7_b15mp_proc_info
/*
@@ -683,9 +757,25 @@
__v7_ca17mp_proc_info:
.long 0x410fc0e0
.long 0xff0ffff0
- __v7_proc __v7_ca17mp_proc_info, __v7_ca17mp_setup
+ __v7_proc __v7_ca17mp_proc_info, __v7_ca17mp_setup, proc_fns = HARDENED_BPIALL_PROCESSOR_FUNCTIONS
.size __v7_ca17mp_proc_info, . - __v7_ca17mp_proc_info
+ /* ARM Ltd. Cortex A73 processor */
+ .type __v7_ca73_proc_info, #object
+__v7_ca73_proc_info:
+ .long 0x410fd090
+ .long 0xff0ffff0
+ __v7_proc __v7_ca73_proc_info, __v7_setup, proc_fns = HARDENED_BPIALL_PROCESSOR_FUNCTIONS
+ .size __v7_ca73_proc_info, . - __v7_ca73_proc_info
+
+ /* ARM Ltd. Cortex A75 processor */
+ .type __v7_ca75_proc_info, #object
+__v7_ca75_proc_info:
+ .long 0x410fd0a0
+ .long 0xff0ffff0
+ __v7_proc __v7_ca75_proc_info, __v7_setup, proc_fns = HARDENED_BPIALL_PROCESSOR_FUNCTIONS
+ .size __v7_ca75_proc_info, . - __v7_ca75_proc_info
+
/*
* Qualcomm Inc. Krait processors.
*/
diff --git a/arch/arm/plat-omap/dmtimer.c b/arch/arm/plat-omap/dmtimer.c
index 7a327bd..ebef8aa 100644
--- a/arch/arm/plat-omap/dmtimer.c
+++ b/arch/arm/plat-omap/dmtimer.c
@@ -890,11 +890,8 @@ static int omap_dm_timer_probe(struct platform_device *pdev)
timer->irq = irq->start;
timer->pdev = pdev;
- /* Skip pm_runtime_enable for OMAP1 */
- if (!(timer->capability & OMAP_TIMER_NEEDS_RESET)) {
- pm_runtime_enable(dev);
- pm_runtime_irq_safe(dev);
- }
+ pm_runtime_enable(dev);
+ pm_runtime_irq_safe(dev);
if (!timer->reserved) {
ret = pm_runtime_get_sync(dev);
diff --git a/arch/arm/probes/kprobes/opt-arm.c b/arch/arm/probes/kprobes/opt-arm.c
index bcdecc2..b2aa9b3 100644
--- a/arch/arm/probes/kprobes/opt-arm.c
+++ b/arch/arm/probes/kprobes/opt-arm.c
@@ -165,13 +165,14 @@ optimized_callback(struct optimized_kprobe *op, struct pt_regs *regs)
{
unsigned long flags;
struct kprobe *p = &op->kp;
- struct kprobe_ctlblk *kcb = get_kprobe_ctlblk();
+ struct kprobe_ctlblk *kcb;
/* Save skipped registers */
regs->ARM_pc = (unsigned long)op->kp.addr;
regs->ARM_ORIG_r0 = ~0UL;
local_irq_save(flags);
+ kcb = get_kprobe_ctlblk();
if (kprobe_running()) {
kprobes_inc_nmissed_count(&op->kp);
@@ -191,6 +192,7 @@ optimized_callback(struct optimized_kprobe *op, struct pt_regs *regs)
local_irq_restore(flags);
}
+NOKPROBE_SYMBOL(optimized_callback)
int arch_prepare_optimized_kprobe(struct optimized_kprobe *op, struct kprobe *orig)
{
diff --git a/arch/arm/vdso/Makefile b/arch/arm/vdso/Makefile
index 59a8fa7..1ce912a 100644
--- a/arch/arm/vdso/Makefile
+++ b/arch/arm/vdso/Makefile
@@ -26,8 +26,11 @@
CFLAGS_REMOVE_vgettimeofday.o = -pg -Os
CFLAGS_vgettimeofday.o = -O2
-# Disable gcov profiling for VDSO code
+# Disable gcov, kasan, ubsan and kcov profiling for VDSO code
GCOV_PROFILE := n
+KASAN_SANITIZE := n
+UBSAN_SANITIZE := n
+KCOV_INSTRUMENT := n
# Force dependency
$(obj)/vdso.o : $(obj)/vdso.so
diff --git a/arch/arm/vdso/compiler.h b/arch/arm/vdso/compiler.h
new file mode 100644
index 0000000..6fd88be
--- /dev/null
+++ b/arch/arm/vdso/compiler.h
@@ -0,0 +1,75 @@
+/*
+ * Userspace implementations of fallback calls
+ *
+ * Copyright (C) 2017 Cavium, Inc.
+ * Copyright (C) 2012 ARM Limited
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Author: Will Deacon <will.deacon@arm.com>
+ * Rewriten into C by: Andrew Pinski <apinski@cavium.com>
+ */
+
+#ifndef __VDSO_COMPILER_H
+#define __VDSO_COMPILER_H
+
+#include <asm/arch_timer.h> /* for arch_counter_get_cntvct() */
+#include <asm/processor.h> /* for cpu_relax() */
+#include <asm/unistd.h>
+#include <linux/compiler.h>
+#include <linux/hrtimer.h> /* for LOW_RES_NSEC and MONOTONIC_RES_NSEC */
+#include <linux/time.h> /* for NSEC_PER_SEC */
+
+#ifndef CONFIG_AEABI
+#error This code depends on AEABI system call conventions
+#endif
+
+#ifdef CONFIG_ARM_ARCH_TIMER
+#define ARCH_PROVIDES_TIMER
+#endif
+
+#define DEFINE_FALLBACK(name, type_arg1, name_arg1, type_arg2, name_arg2) \
+static notrace long name##_fallback(type_arg1 _##name_arg1, \
+ type_arg2 _##name_arg2) \
+{ \
+ register type_arg1 name_arg1 asm("r0") = _##name_arg1; \
+ register type_arg2 name_arg2 asm("r1") = _##name_arg2; \
+ register long ret asm ("r0"); \
+ register long nr asm("r7") = __NR_##name; \
+ \
+ asm volatile( \
+ " swi #0\n" \
+ : "=r" (ret) \
+ : "r" (name_arg1), "r" (name_arg2), "r" (nr) \
+ : "memory"); \
+ \
+ return ret; \
+}
+
+#define arch_vdso_read_counter() arch_counter_get_cntvct()
+
+/* Avoid unresolved references emitted by GCC */
+
+void __aeabi_unwind_cpp_pr0(void)
+{
+}
+
+void __aeabi_unwind_cpp_pr1(void)
+{
+}
+
+void __aeabi_unwind_cpp_pr2(void)
+{
+}
+
+#endif /* __VDSO_COMPILER_H */
diff --git a/arch/arm/vdso/datapage.h b/arch/arm/vdso/datapage.h
new file mode 100644
index 0000000..e3088bd
--- /dev/null
+++ b/arch/arm/vdso/datapage.h
@@ -0,0 +1,25 @@
+/*
+ * Userspace implementations of __get_datapage
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __VDSO_DATAPAGE_H
+#define __VDSO_DATAPAGE_H
+
+#include <linux/types.h>
+#include <asm/vdso_datapage.h>
+
+extern const struct vdso_data *__get_datapage(void);
+
+#endif /* __VDSO_DATAPAGE_H */
diff --git a/arch/arm/vdso/vdso.lds.S b/arch/arm/vdso/vdso.lds.S
index 89ca89f..1eb5770 100644
--- a/arch/arm/vdso/vdso.lds.S
+++ b/arch/arm/vdso/vdso.lds.S
@@ -82,6 +82,8 @@
global:
__vdso_clock_gettime;
__vdso_gettimeofday;
+ __vdso_clock_getres;
+ __vdso_time;
local: *;
};
}
diff --git a/arch/arm/vdso/vgettimeofday.c b/arch/arm/vdso/vgettimeofday.c
index 79214d5..4b241fe6 100644
--- a/arch/arm/vdso/vgettimeofday.c
+++ b/arch/arm/vdso/vgettimeofday.c
@@ -1,282 +1,3 @@
-/*
- * Copyright 2015 Mentor Graphics Corporation.
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License
- * as published by the Free Software Foundation; version 2 of the
- * License.
- *
- * This program is distributed in the hope that it will be useful, but
- * WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
-
-#include <linux/compiler.h>
-#include <linux/hrtimer.h>
-#include <linux/time.h>
-#include <asm/arch_timer.h>
-#include <asm/barrier.h>
-#include <asm/bug.h>
-#include <asm/page.h>
-#include <asm/unistd.h>
-#include <asm/vdso_datapage.h>
-
-#ifndef CONFIG_AEABI
-#error This code depends on AEABI system call conventions
-#endif
-
-extern struct vdso_data *__get_datapage(void);
-
-static notrace u32 __vdso_read_begin(const struct vdso_data *vdata)
-{
- u32 seq;
-repeat:
- seq = ACCESS_ONCE(vdata->seq_count);
- if (seq & 1) {
- cpu_relax();
- goto repeat;
- }
- return seq;
-}
-
-static notrace u32 vdso_read_begin(const struct vdso_data *vdata)
-{
- u32 seq;
-
- seq = __vdso_read_begin(vdata);
-
- smp_rmb(); /* Pairs with smp_wmb in vdso_write_end */
- return seq;
-}
-
-static notrace int vdso_read_retry(const struct vdso_data *vdata, u32 start)
-{
- smp_rmb(); /* Pairs with smp_wmb in vdso_write_begin */
- return vdata->seq_count != start;
-}
-
-static notrace long clock_gettime_fallback(clockid_t _clkid,
- struct timespec *_ts)
-{
- register struct timespec *ts asm("r1") = _ts;
- register clockid_t clkid asm("r0") = _clkid;
- register long ret asm ("r0");
- register long nr asm("r7") = __NR_clock_gettime;
-
- asm volatile(
- " swi #0\n"
- : "=r" (ret)
- : "r" (clkid), "r" (ts), "r" (nr)
- : "memory");
-
- return ret;
-}
-
-static notrace int do_realtime_coarse(struct timespec *ts,
- struct vdso_data *vdata)
-{
- u32 seq;
-
- do {
- seq = vdso_read_begin(vdata);
-
- ts->tv_sec = vdata->xtime_coarse_sec;
- ts->tv_nsec = vdata->xtime_coarse_nsec;
-
- } while (vdso_read_retry(vdata, seq));
-
- return 0;
-}
-
-static notrace int do_monotonic_coarse(struct timespec *ts,
- struct vdso_data *vdata)
-{
- struct timespec tomono;
- u32 seq;
-
- do {
- seq = vdso_read_begin(vdata);
-
- ts->tv_sec = vdata->xtime_coarse_sec;
- ts->tv_nsec = vdata->xtime_coarse_nsec;
-
- tomono.tv_sec = vdata->wtm_clock_sec;
- tomono.tv_nsec = vdata->wtm_clock_nsec;
-
- } while (vdso_read_retry(vdata, seq));
-
- ts->tv_sec += tomono.tv_sec;
- timespec_add_ns(ts, tomono.tv_nsec);
-
- return 0;
-}
-
-#ifdef CONFIG_ARM_ARCH_TIMER
-
-static notrace u64 get_ns(struct vdso_data *vdata)
-{
- u64 cycle_delta;
- u64 cycle_now;
- u64 nsec;
-
- cycle_now = arch_counter_get_cntvct();
-
- cycle_delta = (cycle_now - vdata->cs_cycle_last) & vdata->cs_mask;
-
- nsec = (cycle_delta * vdata->cs_mult) + vdata->xtime_clock_snsec;
- nsec >>= vdata->cs_shift;
-
- return nsec;
-}
-
-static notrace int do_realtime(struct timespec *ts, struct vdso_data *vdata)
-{
- u64 nsecs;
- u32 seq;
-
- do {
- seq = vdso_read_begin(vdata);
-
- if (!vdata->tk_is_cntvct)
- return -1;
-
- ts->tv_sec = vdata->xtime_clock_sec;
- nsecs = get_ns(vdata);
-
- } while (vdso_read_retry(vdata, seq));
-
- ts->tv_nsec = 0;
- timespec_add_ns(ts, nsecs);
-
- return 0;
-}
-
-static notrace int do_monotonic(struct timespec *ts, struct vdso_data *vdata)
-{
- struct timespec tomono;
- u64 nsecs;
- u32 seq;
-
- do {
- seq = vdso_read_begin(vdata);
-
- if (!vdata->tk_is_cntvct)
- return -1;
-
- ts->tv_sec = vdata->xtime_clock_sec;
- nsecs = get_ns(vdata);
-
- tomono.tv_sec = vdata->wtm_clock_sec;
- tomono.tv_nsec = vdata->wtm_clock_nsec;
-
- } while (vdso_read_retry(vdata, seq));
-
- ts->tv_sec += tomono.tv_sec;
- ts->tv_nsec = 0;
- timespec_add_ns(ts, nsecs + tomono.tv_nsec);
-
- return 0;
-}
-
-#else /* CONFIG_ARM_ARCH_TIMER */
-
-static notrace int do_realtime(struct timespec *ts, struct vdso_data *vdata)
-{
- return -1;
-}
-
-static notrace int do_monotonic(struct timespec *ts, struct vdso_data *vdata)
-{
- return -1;
-}
-
-#endif /* CONFIG_ARM_ARCH_TIMER */
-
-notrace int __vdso_clock_gettime(clockid_t clkid, struct timespec *ts)
-{
- struct vdso_data *vdata;
- int ret = -1;
-
- vdata = __get_datapage();
-
- switch (clkid) {
- case CLOCK_REALTIME_COARSE:
- ret = do_realtime_coarse(ts, vdata);
- break;
- case CLOCK_MONOTONIC_COARSE:
- ret = do_monotonic_coarse(ts, vdata);
- break;
- case CLOCK_REALTIME:
- ret = do_realtime(ts, vdata);
- break;
- case CLOCK_MONOTONIC:
- ret = do_monotonic(ts, vdata);
- break;
- default:
- break;
- }
-
- if (ret)
- ret = clock_gettime_fallback(clkid, ts);
-
- return ret;
-}
-
-static notrace long gettimeofday_fallback(struct timeval *_tv,
- struct timezone *_tz)
-{
- register struct timezone *tz asm("r1") = _tz;
- register struct timeval *tv asm("r0") = _tv;
- register long ret asm ("r0");
- register long nr asm("r7") = __NR_gettimeofday;
-
- asm volatile(
- " swi #0\n"
- : "=r" (ret)
- : "r" (tv), "r" (tz), "r" (nr)
- : "memory");
-
- return ret;
-}
-
-notrace int __vdso_gettimeofday(struct timeval *tv, struct timezone *tz)
-{
- struct timespec ts;
- struct vdso_data *vdata;
- int ret;
-
- vdata = __get_datapage();
-
- ret = do_realtime(&ts, vdata);
- if (ret)
- return gettimeofday_fallback(tv, tz);
-
- if (tv) {
- tv->tv_sec = ts.tv_sec;
- tv->tv_usec = ts.tv_nsec / 1000;
- }
- if (tz) {
- tz->tz_minuteswest = vdata->tz_minuteswest;
- tz->tz_dsttime = vdata->tz_dsttime;
- }
-
- return ret;
-}
-
-/* Avoid unresolved references emitted by GCC */
-
-void __aeabi_unwind_cpp_pr0(void)
-{
-}
-
-void __aeabi_unwind_cpp_pr1(void)
-{
-}
-
-void __aeabi_unwind_cpp_pr2(void)
-{
-}
+#include "compiler.h"
+#include "datapage.h"
+#include "../../../lib/vdso/vgettimeofday.c"
diff --git a/arch/arm/vfp/vfpmodule.c b/arch/arm/vfp/vfpmodule.c
index 5629d75..8e5e979 100644
--- a/arch/arm/vfp/vfpmodule.c
+++ b/arch/arm/vfp/vfpmodule.c
@@ -597,13 +597,11 @@ int vfp_preserve_user_clear_hwstate(struct user_vfp __user *ufp,
}
/* Sanitise and restore the current VFP state from the provided structures. */
-int vfp_restore_user_hwstate(struct user_vfp __user *ufp,
- struct user_vfp_exc __user *ufp_exc)
+int vfp_restore_user_hwstate(struct user_vfp *ufp, struct user_vfp_exc *ufp_exc)
{
struct thread_info *thread = current_thread_info();
struct vfp_hard_struct *hwstate = &thread->vfpstate.hard;
unsigned long fpexc;
- int err = 0;
/* Disable VFP to avoid corrupting the new thread state. */
vfp_flush_hwstate(thread);
@@ -612,17 +610,16 @@ int vfp_restore_user_hwstate(struct user_vfp __user *ufp,
* Copy the floating point registers. There can be unused
* registers see asm/hwcap.h for details.
*/
- err |= __copy_from_user(&hwstate->fpregs, &ufp->fpregs,
- sizeof(hwstate->fpregs));
+ memcpy(&hwstate->fpregs, &ufp->fpregs, sizeof(hwstate->fpregs));
/*
* Copy the status and control register.
*/
- __get_user_error(hwstate->fpscr, &ufp->fpscr, err);
+ hwstate->fpscr = ufp->fpscr;
/*
* Sanitise and restore the exception registers.
*/
- __get_user_error(fpexc, &ufp_exc->fpexc, err);
+ fpexc = ufp_exc->fpexc;
/* Ensure the VFP is enabled. */
fpexc |= FPEXC_EN;
@@ -631,10 +628,10 @@ int vfp_restore_user_hwstate(struct user_vfp __user *ufp,
fpexc &= ~(FPEXC_EX | FPEXC_FP2V);
hwstate->fpexc = fpexc;
- __get_user_error(hwstate->fpinst, &ufp_exc->fpinst, err);
- __get_user_error(hwstate->fpinst2, &ufp_exc->fpinst2, err);
+ hwstate->fpinst = ufp_exc->fpinst;
+ hwstate->fpinst2 = ufp_exc->fpinst2;
- return err ? -EFAULT : 0;
+ return 0;
}
/*
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 9c01a31..2527058 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -17,6 +17,7 @@
select ARCH_HAS_TICK_BROADCAST if GENERIC_CLOCKEVENTS_BROADCAST
select ARCH_USE_CMPXCHG_LOCKREF
select ARCH_SUPPORTS_LTO_CLANG
+ select ARCH_SUPPORTS_SHADOW_CALL_STACK
select ARCH_SUPPORTS_ATOMIC_RMW
select ARCH_SUPPORTS_NUMA_BALANCING
select ARCH_WANT_COMPAT_IPC_PARSE_VERSION
@@ -24,6 +25,8 @@
select ARCH_HAS_UBSAN_SANITIZE_ALL
select ARM_AMBA
select ARM_ARCH_TIMER
+ select HAVE_KERNEL_GZIP
+ select HAVE_KERNEL_LZ4
select ARM_GIC
select AUDIT_ARCH_COMPAT_GENERIC
select ARM_GIC_V2M if PCI
@@ -837,6 +840,15 @@
If unsure, say N.
+config ARM64_SSBD
+ bool "Speculative Store Bypass Disable" if EXPERT
+ default y
+ help
+ This enables mitigation of the bypassing of previous stores
+ by speculative loads.
+
+ If unsure, say Y.
+
menuconfig ARMV8_DEPRECATED
bool "Emulate deprecated/obsolete ARMv8 instructions"
depends on COMPAT
@@ -1136,26 +1148,6 @@
DTBs to be built by default (instead of a standalone Image.gz.)
The image will built in arch/arm64/boot/Image.gz-dtb
-choice
- prompt "Appended DTB Kernel Image name"
- depends on BUILD_ARM64_APPENDED_DTB_IMAGE
- help
- Enabling this option will cause a specific kernel image Image or
- Image.gz to be used for final image creation.
- The image will built in arch/arm64/boot/IMAGE-NAME-dtb
-
- config IMG_GZ_DTB
- bool "Image.gz-dtb"
- config IMG_DTB
- bool "Image-dtb"
-endchoice
-
-config BUILD_ARM64_APPENDED_KERNEL_IMAGE_NAME
- string
- depends on BUILD_ARM64_APPENDED_DTB_IMAGE
- default "Image.gz-dtb" if IMG_GZ_DTB
- default "Image-dtb" if IMG_DTB
-
config BUILD_ARM64_APPENDED_DTB_IMAGE_NAMES
string "Default dtb names"
depends on BUILD_ARM64_APPENDED_DTB_IMAGE
@@ -1163,34 +1155,21 @@
Space separated list of names of dtbs to append when
building a concatenated Image.gz-dtb.
-choice
- prompt "Kernel compression method"
- default BUILD_ARM64_KERNEL_COMPRESSION_GZIP
- help
- Allows choice between gzip compressed or uncompressed
- kernel image
+config BUILD_ARM64_DTC
+ string "dtc to use"
-config BUILD_ARM64_KERNEL_COMPRESSION_GZIP
- bool "Build compressed kernel image"
- help
- Build a kernel image using gzip
- compression with concatenated dtb.
- gzip is based on the DEFLATE
- algorithm.
+config BUILD_ARM64_DTC_FLAGS
+ string "extra dtc flags"
-config BUILD_ARM64_UNCOMPRESSED_KERNEL
- bool "Build uncompressed kernel image"
- help
- Build a kernel image without
- compression and with
- concatenated dtb.
-endchoice
+config BUILD_ARM64_APPLY_DTBO
+ bool "apply device tree overlay"
config BUILD_ARM64_DT_OVERLAY
bool "enable DT overlay compilation support"
depends on OF
help
This option enables support for DT overlay compilation.
+
endmenu
menu "Userspace binary formats"
@@ -1216,10 +1195,65 @@
If you want to execute 32-bit userspace applications, say Y.
+config KUSER_HELPERS
+ bool "Enable the kuser helpers page in 32-bit processes"
+ depends on COMPAT
+ default y
+ help
+ Warning: disabling this option may break 32-bit applications.
+
+ Provide kuser helpers in a special purpose fixed-address page. The
+ kernel provides helper code to userspace in read-only form at a fixed
+ location to allow userspace to be independent of the CPU type fitted
+ to the system. This permits 32-bit binaries to be run on ARMv6 through
+ to ARMv8 without modification.
+
+ See Documentation/arm/kernel_user_helpers.txt for details.
+
+ However, the fixed-address nature of these helpers can be used by ROP
+ (return-orientated programming) authors when creating exploits.
+
+ If all of the 32-bit binaries and libraries that run on your platform
+ are built specifically for your platform, and make no use of these
+ helpers, then you can turn this option off to hinder such exploits.
+ However, in that case, if a binary or library relying on those helpers
+ is run, it will receive a SIGSEGV signal, which will terminate the
+ program. Typically, binaries compiled for ARMv7 or later do not use
+ the kuser helpers.
+
+ Say N here only if you are absolutely certain that you do not need
+ these helpers; otherwise, the safe option is to say Y (the default
+ for now)
+
config SYSVIPC_COMPAT
def_bool y
depends on COMPAT && SYSVIPC
+config COMPAT_VDSO
+ bool "32-bit vDSO"
+ depends on COMPAT
+ select ARM_ARCH_TIMER_VCT_ACCESS
+ default n
+ help
+ Warning: a 32-bit toolchain is necessary to build the vDSO. You
+ must explicitly define which toolchain should be used by setting
+ CROSS_COMPILE_ARM32 to the prefix of the 32-bit toolchain (same format
+ as CROSS_COMPILE). If CROSS_COMPILE_ARM32 is empty, a warning will be
+ printed and the kernel will be built as if COMPAT_VDSO had not been
+ set. If CROSS_COMPILE_ARM32 is set to an invalid prefix, compilation
+ will be aborted.
+
+ Provide a vDSO to 32-bit processes. It includes the symbols provided
+ by the vDSO from the 32-bit kernel, so that a 32-bit libc can use
+ the compat vDSO without modification. It also provides sigreturn
+ trampolines, replacing the sigreturn page.
+
+config CROSS_COMPILE_ARM32
+ string "32-bit toolchain prefix"
+ help
+ Same as setting CROSS_COMPILE_ARM32 in the environment, but saved for
+ future builds. The environment variable overrides this config option.
+
endmenu
menu "Power management options"
diff --git a/arch/arm64/Kconfig.platforms b/arch/arm64/Kconfig.platforms
index a92f511..d056458 100644
--- a/arch/arm64/Kconfig.platforms
+++ b/arch/arm64/Kconfig.platforms
@@ -216,6 +216,7 @@
select GPIOLIB
select PINCTRL
select PINCTRL_ROCKCHIP
+ select PM
select ROCKCHIP_TIMER
help
This enables support for the ARMv8 based Rockchip chipsets,
@@ -321,4 +322,9 @@
help
This enables support for Xilinx ZynqMP Family
+config BOARD_B1C1
+ bool "Google B1C1 Family"
+ depends on ARCH_QCOM
+ select ARCH_SDM845
+
endmenu
diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile
index 2b265a7..2dc83b9 100644
--- a/arch/arm64/Makefile
+++ b/arch/arm64/Makefile
@@ -10,7 +10,7 @@
#
# Copyright (C) 1995-2001 by Russell King
-LDFLAGS_vmlinux :=-p --no-undefined -X
+LDFLAGS_vmlinux :=--no-undefined -X
CPPFLAGS_vmlinux.lds = -DTEXT_OFFSET=$(TEXT_OFFSET)
GZFLAGS :=-9
@@ -50,18 +50,51 @@
endif
endif
+ifeq ($(CONFIG_COMPAT_VDSO), y)
+ CROSS_COMPILE_ARM32 ?= $(CONFIG_CROSS_COMPILE_ARM32:"%"=%)
+
+ # Check that the user has provided a valid prefix for the 32-bit toolchain.
+ # To prevent selecting the system $(cc-name) by default, the prefix is not
+ # allowed to be empty, unlike CROSS_COMPILE. In the unlikely event that the
+ # system $(cc-name) is actually the 32-bit ARM compiler to be used, the
+ # variable can be set to the dirname (e.g. CROSS_COMPILE_ARM32=/usr/bin/).
+ # Note: this Makefile is read both before and after regenerating the config
+ # (if needed). Any warning appearing before the config has been regenerated
+ # should be ignored. If the error is triggered and you set
+ # CONFIG_CROSS_COMPILE_ARM32, set CROSS_COMPILE_ARM32 to an appropriate value
+ # when invoking make and fix CONFIG_CROSS_COMPILE_ARM32.
+ ifeq ($(CROSS_COMPILE_ARM32),)
+ $(error CROSS_COMPILE_ARM32 not defined or empty, the compat vDSO will not be built)
+ else ifeq ($(cc-name),clang)
+ export CLANG_TRIPLE_ARM32 ?= $(CROSS_COMPILE_ARM32)
+ export CLANG_TARGET_ARM32 := --target=$(notdir $(CLANG_TRIPLE_ARM32:%-=%))
+ export CONFIG_VDSO32 := y
+ vdso32 := -DCONFIG_VDSO32=1
+ else ifeq ($(shell which $(CROSS_COMPILE_ARM32)$(cc-name) 2> /dev/null),)
+ $(error $(CROSS_COMPILE_ARM32)$(cc-name) not found, check CROSS_COMPILE_ARM32)
+ else
+ export CROSS_COMPILE_ARM32
+ export CONFIG_VDSO32 := y
+ vdso32 := -DCONFIG_VDSO32=1
+ endif
+endif
+
ifeq ($(cc-name),clang)
# This is a workaround for https://bugs.llvm.org/show_bug.cgi?id=30792.
# TODO: revert when this is fixed in LLVM.
-KBUILD_CFLAGS += -mno-implicit-float
+KBUILD_CFLAGS += -mno-implicit-float
else
-KBUILD_CFLAGS += -mgeneral-regs-only
+KBUILD_CFLAGS += -mgeneral-regs-only
endif
-KBUILD_CFLAGS += $(lseinstr)
+KBUILD_CFLAGS += $(lseinstr) $(vdso32)
KBUILD_CFLAGS += -fno-asynchronous-unwind-tables
KBUILD_CFLAGS += $(call cc-option, -mpc-relative-literal-loads)
KBUILD_CFLAGS += -fno-pic
-KBUILD_AFLAGS += $(lseinstr)
+KBUILD_AFLAGS += $(lseinstr) $(vdso32)
+
+ifeq ($(CONFIG_SHADOW_CALL_STACK), y)
+KBUILD_CFLAGS += -ffixed-x18
+endif
ifeq ($(CONFIG_CPU_BIG_ENDIAN), y)
KBUILD_CPPFLAGS += -mbig-endian
@@ -81,7 +114,7 @@
KBUILD_CFLAGS_MODULE += -mcmodel=large
ifeq ($(CONFIG_LTO_CLANG), y)
# Code model is not stored in LLVM IR, so we need to pass it also to LLVMgold
-LDFLAGS += -plugin-opt=-code-model=large
+KBUILD_LDFLAGS_MODULE += -plugin-opt=-code-model=large
endif
endif
@@ -122,24 +155,39 @@
libs-y := arch/arm64/lib/ $(libs-y)
core-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a
-ifeq ($(CONFIG_BUILD_ARM64_KERNEL_COMPRESSION_GZIP),y)
-KBUILD_IMAGE := Image.gz
-else
-KBUILD_IMAGE := Image
-endif
+suffix_$(CONFIG_KERNEL_GZIP) = gz
+suffix_$(CONFIG_KERNEL_LZ4) = lz4
# Default target when executing plain make
ifeq ($(CONFIG_BUILD_ARM64_APPENDED_DTB_IMAGE),y)
-KBUILD_IMAGE := $(addsuffix -dtb,$(KBUILD_IMAGE))
+KBUILD_IMAGE := Image.$(suffix_y)-dtb
+else
+KBUILD_IMAGE := Image.$(suffix_y)
endif
KBUILD_DTBS := dtbs
+KBUILD_DTBO_IMG := dtbo.img
+
ifeq ($(CONFIG_BUILD_ARM64_DT_OVERLAY),y)
export DTC_FLAGS := -@
endif
-all: $(KBUILD_IMAGE) $(KBUILD_DTBS)
+ifeq ($(CONFIG_BUILD_ARM64_APPLY_DTBO),y)
+ KBUILD_APPLY_DTBO := apply_dtbo
+endif
+
+BUILD_DTC_PATH = $(subst $\",,$(CONFIG_BUILD_ARM64_DTC))
+ifneq ($(BUILD_DTC_PATH),)
+ DTC = $(BUILD_DTC_PATH)
+endif
+
+BUILD_DTC_FLAGS = $(subst $\",,$(CONFIG_BUILD_ARM64_DTC_FLAGS))
+ifneq ($(BUILD_DTC_FLAGS),)
+ DTC_FLAGS += $(BUILD_DTC_FLAGS)
+endif
+
+all: $(KBUILD_IMAGE) $(KBUILD_DTBS) $(KBUILD_DTBO_IMG) $(KBUILD_APPLY_DTBO)
boot := arch/arm64/boot
@@ -163,9 +211,17 @@
dtbs_install:
$(Q)$(MAKE) $(dtbinst)=$(boot)/dts
-Image-dtb Image.gz-dtb: vmlinux scripts dtbs
+Image-dtb Image.gz-dtb Image.lz4-dtb: vmlinux scripts dtbs
$(Q)$(MAKE) $(build)=$(boot) $(boot)/$@
+dtbo.img: $(boot)/dts/dtboimg.cfg dtbs
+ $(Q)$(MAKE) $(build)=$(boot) $(boot)/$@
+
+
+PHONY += apply_dtbo
+dtboapply := -f $(if $(KBUILD_SRC),$(srctree)/)scripts/Makefile.dtboapply obj
+apply_dtbo: Image.lz4-dtb dtbo.img
+ $(Q)$(MAKE) $(dtboapply)=$(boot) KERNEL_DTB=Image.lz4-dtb DTBO=dtbo.img
PHONY += vdso_install
vdso_install:
$(Q)$(MAKE) $(build)=arch/arm64/kernel/vdso $@
@@ -175,6 +231,9 @@
$(Q)$(MAKE) $(clean)=$(boot)
$(Q)$(MAKE) $(clean)=$(boot)/dts
+CLEAN_FILES += $(boot)/Image $(boot)/Image-dtb $(boot)/Image.* $(boot)/dtbo.img
+MRPROPER_FILES += $(boot)/Image $(boot)/Image-dtb $(boot)/Image.* $(boot)/dtbo.img
+
# We need to generate vdso-offsets.h before compiling certain files in kernel/.
# In order to do that, we should use the archprepare target, but we can't since
# asm-offsets.h is included in some files used to generate vdso-offsets.h, and
@@ -184,9 +243,12 @@
prepare: vdso_prepare
vdso_prepare: prepare0
$(Q)$(MAKE) $(build)=arch/arm64/kernel/vdso include/generated/vdso-offsets.h
+ $(if $(CONFIG_VDSO32),$(Q)$(MAKE) $(build)=arch/arm64/kernel/vdso32 \
+ include/generated/vdso32-offsets.h)
define archhelp
echo '* Image.gz - Compressed kernel image (arch/$(ARCH)/boot/Image.gz)'
+ echo '* Image.lz4 - Compressed kernel image (arch/$(ARCH)/boot/Image.lz4)'
echo ' Image - Uncompressed kernel image (arch/$(ARCH)/boot/Image)'
echo '* dtbs - Build device tree blobs for enabled boards'
echo ' dtbs_install - Install dtbs to $(INSTALL_DTBS_PATH)'
diff --git a/arch/arm64/boot/.gitignore b/arch/arm64/boot/.gitignore
index 34e3520..08375ea 100644
--- a/arch/arm64/boot/.gitignore
+++ b/arch/arm64/boot/.gitignore
@@ -2,3 +2,4 @@
Image-dtb
Image.gz
Image.gz-dtb
+dtbo.img
diff --git a/arch/arm64/boot/Makefile b/arch/arm64/boot/Makefile
index ccdfa1f..dfafe91 100644
--- a/arch/arm64/boot/Makefile
+++ b/arch/arm64/boot/Makefile
@@ -18,7 +18,7 @@
OBJCOPYFLAGS_Image :=-O binary -R .note -R .note.gnu.build-id -R .comment -S
-targets := Image Image.gz
+targets := Image Image.gz Image.lz4
DTB_NAMES := $(subst $\",,$(CONFIG_BUILD_ARM64_APPENDED_DTB_IMAGE_NAMES))
ifneq ($(DTB_NAMES),)
@@ -27,6 +27,7 @@
else
DTB_OBJS := $(shell find $(obj)/dts/ -name \*.dtb)
endif
+DTBO_OBJS := $(shell find -L $(obj)/dts/ -name \*.dtbo)
# Add RTIC DTB to the DTB list if RTIC MPGen is enabled
ifdef RTIC_MPGEN
@@ -45,11 +46,7 @@
$(obj)/Image.bz2: $(obj)/Image FORCE
$(call if_changed,bzip2)
-$(obj)/Image-dtb-hdr: $(obj)/Image FORCE
- echo -n 'UNCOMPRESSED_IMG' > $@ && \
- $(call size_append, $(filter-out FORCE,$^)) >> $@
-
-$(obj)/Image-dtb: $(obj)/Image-dtb-hdr $(obj)/Image $(DTB_OBJS) FORCE
+$(obj)/Image-dtb: $(obj)/Image $(DTB_OBJS) FORCE
$(call if_changed,cat)
$(obj)/Image.gz: $(obj)/Image FORCE
@@ -67,6 +64,12 @@
$(obj)/Image.gz-dtb: $(obj)/Image.gz $(DTB_OBJS) FORCE
$(call if_changed,cat)
+$(obj)/Image.lz4-dtb: $(obj)/Image.lz4 $(DTB_OBJS) FORCE
+ $(call if_changed,cat)
+
+$(obj)/dtbo.img: $(obj)/dts/dtboimg.cfg $(DTBO_OBJS) FORCE
+ $(call if_changed,mkdtimg,4096) # align dtbo.img to 4kB
+
install:
$(CONFIG_SHELL) $(srctree)/$(src)/install.sh $(KERNELRELEASE) \
$(obj)/Image System.map "$(INSTALL_PATH)"
diff --git a/arch/arm64/boot/dts/.gitignore b/arch/arm64/boot/dts/.gitignore
index b60ed20..77ed402c 100644
--- a/arch/arm64/boot/dts/.gitignore
+++ b/arch/arm64/boot/dts/.gitignore
@@ -1 +1,2 @@
*.dtb
+*.dtbo
diff --git a/arch/arm64/boot/dts/Makefile b/arch/arm64/boot/dts/Makefile
index 7ad2cf0..288ec82 100644
--- a/arch/arm64/boot/dts/Makefile
+++ b/arch/arm64/boot/dts/Makefile
@@ -9,6 +9,7 @@
dts-dirs += exynos
dts-dirs += freescale
dts-dirs += hisilicon
+dts-dirs += google
dts-dirs += marvell
dts-dirs += mediatek
dts-dirs += nvidia
diff --git a/arch/arm64/boot/dts/altera/socfpga_stratix10.dtsi b/arch/arm64/boot/dts/altera/socfpga_stratix10.dtsi
index c2b9bcb..e79f3de 100644
--- a/arch/arm64/boot/dts/altera/socfpga_stratix10.dtsi
+++ b/arch/arm64/boot/dts/altera/socfpga_stratix10.dtsi
@@ -249,7 +249,7 @@
sysmgr: sysmgr@ffd12000 {
compatible = "altr,sys-mgr", "syscon";
- reg = <0xffd12000 0x1000>;
+ reg = <0xffd12000 0x228>;
};
/* Local timer */
diff --git a/arch/arm64/boot/dts/broadcom/ns2.dtsi b/arch/arm64/boot/dts/broadcom/ns2.dtsi
index a16b1b3..8a94ec8 100644
--- a/arch/arm64/boot/dts/broadcom/ns2.dtsi
+++ b/arch/arm64/boot/dts/broadcom/ns2.dtsi
@@ -393,7 +393,7 @@
reg = <0x66080000 0x100>;
#address-cells = <1>;
#size-cells = <0>;
- interrupts = <GIC_SPI 394 IRQ_TYPE_NONE>;
+ interrupts = <GIC_SPI 394 IRQ_TYPE_LEVEL_HIGH>;
clock-frequency = <100000>;
status = "disabled";
};
@@ -421,7 +421,7 @@
reg = <0x660b0000 0x100>;
#address-cells = <1>;
#size-cells = <0>;
- interrupts = <GIC_SPI 395 IRQ_TYPE_NONE>;
+ interrupts = <GIC_SPI 395 IRQ_TYPE_LEVEL_HIGH>;
clock-frequency = <100000>;
status = "disabled";
};
diff --git a/arch/arm64/boot/dts/dtboimg.cfg b/arch/arm64/boot/dts/dtboimg.cfg
new file mode 100644
index 0000000..e2e1d08
--- /dev/null
+++ b/arch/arm64/boot/dts/dtboimg.cfg
@@ -0,0 +1,55 @@
+arch/arm64/boot/dts/google/sdm845-v2-b1c1-devboard.dtbo
+ id=0x102
+ rev=0x1
+
+arch/arm64/boot/dts/google/sdm845-b1-proto.dtbo
+ id=0x203
+ rev=0x1
+
+arch/arm64/boot/dts/google/sdm845-b1-proto-v1.1.dtbo
+ id=0x204
+ rev=0x1
+
+arch/arm64/boot/dts/google/sdm845-b1c1-dev2.dtbo
+ id=0x202
+ rev=0x2
+
+arch/arm64/boot/dts/google/sdm845-b1-evt1.dtbo
+ id=0x20A
+ rev=0x2
+
+arch/arm64/boot/dts/google/sdm845-b1-evt2.dtbo
+ id=0x20F
+ rev=0x2
+
+arch/arm64/boot/dts/google/sdm845-b1-evt2.1.dtbo
+ id=0x210
+ rev=0x2
+
+arch/arm64/boot/dts/google/sdm845-b1-dvt.dtbo
+ id=0x214
+ rev=0x2
+
+arch/arm64/boot/dts/google/sdm845-b1-dvt1.1.dtbo
+ id=0x215
+ rev=0x2
+
+arch/arm64/boot/dts/google/sdm845-c1-proto.dtbo
+ id=0x103
+ rev=0x2
+
+arch/arm64/boot/dts/google/sdm845-c1-evt.dtbo
+ id=0x10A
+ rev=0x2
+
+arch/arm64/boot/dts/google/sdm845-c1-evt1.1.dtbo
+ id=0x10B
+ rev=0x2
+
+arch/arm64/boot/dts/google/sdm845-c1-dvt.dtbo
+ id=0x114
+ rev=0x2
+
+arch/arm64/boot/dts/google/sdm845-c1-dvt1.1.dtbo
+ id=0x115
+ rev=0x2
diff --git a/arch/arm64/boot/dts/google/Makefile b/arch/arm64/boot/dts/google/Makefile
new file mode 100644
index 0000000..024be78
--- /dev/null
+++ b/arch/arm64/boot/dts/google/Makefile
@@ -0,0 +1,43 @@
+ifeq ($(CONFIG_BUILD_ARM64_DT_OVERLAY),y)
+
+# V2 SDM845
+dtbo-$(CONFIG_BOARD_B1C1) += \
+ sdm845-v2-b1c1-devboard.dtbo \
+ sdm845-b1-proto.dtbo \
+ sdm845-b1-proto-v1.1.dtbo
+
+sdm845-v2-b1c1-devboard.dtbo-base := ../qcom/sdm845-v2.dtb
+sdm845-b1-proto.dtbo-base := ../qcom/sdm845-v2.dtb
+sdm845-b1-proto-v1.1.dtbo-base := ../qcom/sdm845-v2.dtb
+
+# V2.1 SDM845
+dtbo-$(CONFIG_BOARD_B1C1) += \
+ sdm845-b1c1-dev2.dtbo \
+ sdm845-b1-evt1.dtbo \
+ sdm845-b1-evt2.dtbo \
+ sdm845-b1-evt2.1.dtbo \
+ sdm845-b1-dvt.dtbo \
+ sdm845-b1-dvt1.1.dtbo \
+ sdm845-c1-proto.dtbo \
+ sdm845-c1-evt.dtbo \
+ sdm845-c1-evt1.1.dtbo \
+ sdm845-c1-dvt.dtbo \
+ sdm845-c1-dvt1.1.dtbo
+
+sdm845-b1c1-dev2.dtbo-base := ../qcom/sdm845-v2.1.dtb
+sdm845-b1-evt1.dtbo-base := ../qcom/sdm845-v2.1.dtb
+sdm845-b1-evt2.dtbo-base := ../qcom/sdm845-v2.1.dtb
+sdm845-b1-evt2.1.dtbo-base := ../qcom/sdm845-v2.1.dtb
+sdm845-b1-dvt.dtbo-base := ../qcom/sdm845-v2.1.dtb
+sdm845-b1-dvt1.1.dtbo-base := ../qcom/sdm845-v2.1.dtb
+sdm845-c1-proto.dtbo-base := ../qcom/sdm845-v2.1.dtb
+sdm845-c1-evt.dtbo-base := ../qcom/sdm845-v2.1.dtb
+sdm845-c1-evt1.1.dtbo-base := ../qcom/sdm845-v2.1.dtb
+sdm845-c1-dvt.dtbo-base := ../qcom/sdm845-v2.1.dtb
+sdm845-c1-dvt1.1.dtbo-base := ../qcom/sdm845-v2.1.dtb
+
+endif
+
+always := $(dtb-y)
+subdir-y := $(dts-dirs)
+clean-files := *.dtb
diff --git a/arch/arm64/boot/dts/google/batterydata-maxim-b1-100k.dtsi b/arch/arm64/boot/dts/google/batterydata-maxim-b1-100k.dtsi
new file mode 100644
index 0000000..b76358e
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-maxim-b1-100k.dtsi
@@ -0,0 +1,39 @@
+maxim,n_regval = /bits/ 16 <
+ /* Sunwoda_B1_LWN_EVT1_Vempty3.3_ALERT_LearnCFG_2362.INI */
+ /* Generated on 2018/03/05 16:20:09 */
+ 0x80 0x0021 /* nXTable0 */
+ 0x81 0x002A /* nXTable1 */
+ 0x82 0x0314 /* nXTable2 */
+ 0x83 0x0C21 /* nXTable3 */
+ 0x84 0x0F2A /* nXTable4 */
+ 0x85 0x1670 /* nXTable5 */
+ 0x86 0x1A40 /* nXTable6 */
+ 0x87 0x0EDA /* nXTable7 */
+ 0x88 0x07EB /* nXTable8 */
+ 0x89 0x06F0 /* nXTable9 */
+ 0x8A 0x0887 /* nXTable10 */
+ 0x8B 0x07BB /* nXTable11 */
+ 0x90 0x7D0B /* nOCVTable0 */
+ 0x91 0x7D37 /* nOCVTable1 */
+ 0x92 0xAE25 /* nOCVTable2 */
+ 0x93 0xB83B /* nOCVTable3 */
+ 0x94 0xB9CC /* nOCVTable4 */
+ 0x95 0xBB09 /* nOCVTable5 */
+ 0x96 0xBDFC /* nOCVTable6 */
+ 0x97 0xBED3 /* nOCVTable7 */
+ 0x98 0xC105 /* nOCVTable8 */
+ 0x99 0xC62D /* nOCVTable9 */
+ 0x9A 0xC9C2 /* nOCVTable10 */
+ 0x9B 0xCCB4 /* nOCVTable11 */
+ 0x9E 0xA561 /* nVEmpty: VE=3300 VR=3880 */
+ 0xB1 0x0204 /* nRippleCfg */
+ 0xB5 0x9C01 /* nPackCfg */
+ 0xB6 0x0000 /* nRelaxCfg */
+ 0xB9 0xA00E /* nNVCfg1 */
+ 0xBB 0x0002 /* nSBSCfg */
+ 0xC3 0x7F80 /* nIAlrtTh: Disabled */
+ 0xC7 0x0000 /* nTTFCfg */
+ 0xC9 0x002F /* nTCurve */
+ 0xCA 0xF1F1 /* nTGain */
+ 0xCB 0x198A /* nTOff */
+>;
diff --git a/arch/arm64/boot/dts/google/batterydata-maxim-b1-10k.dtsi b/arch/arm64/boot/dts/google/batterydata-maxim-b1-10k.dtsi
new file mode 100644
index 0000000..9b7f605
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-maxim-b1-10k.dtsi
@@ -0,0 +1,39 @@
+maxim,n_regval = /bits/ 16 <
+ /* Desay_B1_LSN_EVT1_Vempty3.3_ALERT_LearnCFG_2360.INI */
+ /* Generated on 2018/03/02 09:14:40 */
+ 0x80 0x0020 /* nXTable0 */
+ 0x81 0x004E /* nXTable1 */
+ 0x82 0x0AEC /* nXTable2 */
+ 0x83 0x0D01 /* nXTable3 */
+ 0x84 0x1EBB /* nXTable4 */
+ 0x85 0x1E4E /* nXTable5 */
+ 0x86 0x10F0 /* nXTable6 */
+ 0x87 0x10D9 /* nXTable7 */
+ 0x88 0x08D7 /* nXTable8 */
+ 0x89 0x0800 /* nXTable9 */
+ 0x8A 0x07A7 /* nXTable10 */
+ 0x8B 0x0744 /* nXTable11 */
+ 0x90 0x88DB /* nOCVTable0 */
+ 0x91 0x8908 /* nOCVTable1 */
+ 0x92 0xB54A /* nOCVTable2 */
+ 0x93 0xBA9B /* nOCVTable3 */
+ 0x94 0xBC0D /* nOCVTable4 */
+ 0x95 0xBCA4 /* nOCVTable5 */
+ 0x96 0xBEBC /* nOCVTable6 */
+ 0x97 0xBFE4 /* nOCVTable7 */
+ 0x98 0xC114 /* nOCVTable8 */
+ 0x99 0xC78D /* nOCVTable9 */
+ 0x9A 0xCA83 /* nOCVTable10 */
+ 0x9B 0xCD92 /* nOCVTable11 */
+ 0x9E 0xA561 /* nVEmpty: VE=3300 VR=3880 */
+ 0xB1 0x0204 /* nRippleCfg */
+ 0xB5 0x9C01 /* nPackCfg */
+ 0xB6 0x0000 /* nRelaxCfg */
+ 0xB9 0xA00E /* nNVCfg1 */
+ 0xBB 0x0002 /* nSBSCfg */
+ 0xC3 0x7F80 /* nIAlrtTh: Disabled */
+ 0xC7 0x0000 /* nTTFCfg */
+ 0xC9 0x002F /* nTCurve */
+ 0xCA 0xF1F1 /* nTGain */
+ 0xCB 0x198A /* nTOff */
+>;
diff --git a/arch/arm64/boot/dts/google/batterydata-maxim-b1-22k.dtsi b/arch/arm64/boot/dts/google/batterydata-maxim-b1-22k.dtsi
new file mode 100644
index 0000000..cc410d6b
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-maxim-b1-22k.dtsi
@@ -0,0 +1,39 @@
+maxim,n_regval = /bits/ 16 <
+ /* Sunwoda_B1_LSN_EVT1_Vempty3.3_ALERT_LearnCFG_2365.INI */
+ /* Generated on 2018/03/02 09:20:27 */
+ 0x80 0x0021 /* nXTable0 */
+ 0x81 0x0027 /* nXTable1 */
+ 0x82 0x0397 /* nXTable2 */
+ 0x83 0x0A81 /* nXTable3 */
+ 0x84 0x18E8 /* nXTable4 */
+ 0x85 0x1BCF /* nXTable5 */
+ 0x86 0x0FF0 /* nXTable6 */
+ 0x87 0x0A38 /* nXTable7 */
+ 0x88 0x0989 /* nXTable8 */
+ 0x89 0x0800 /* nXTable9 */
+ 0x8A 0x0797 /* nXTable10 */
+ 0x8B 0x0744 /* nXTable11 */
+ 0x90 0x7D0B /* nOCVTable0 */
+ 0x91 0x7D47 /* nOCVTable1 */
+ 0x92 0xB1BD /* nOCVTable2 */
+ 0x93 0xB91B /* nOCVTable3 */
+ 0x94 0xBB3D /* nOCVTable4 */
+ 0x95 0xBCED /* nOCVTable5 */
+ 0x96 0xBEEC /* nOCVTable6 */
+ 0x97 0xC124 /* nOCVTable7 */
+ 0x98 0xC27F /* nOCVTable8 */
+ 0x99 0xC80D /* nOCVTable9 */
+ 0x9A 0xCB04 /* nOCVTable10 */
+ 0x9B 0xCDF1 /* nOCVTable11 */
+ 0x9E 0xA561 /* nVEmpty: VE=3300 VR=3880 */
+ 0xB1 0x0204 /* nRippleCfg */
+ 0xB5 0x9C01 /* nPackCfg */
+ 0xB6 0x0000 /* nRelaxCfg */
+ 0xB9 0xA00E /* nNVCfg1 */
+ 0xBB 0x0002 /* nSBSCfg */
+ 0xC3 0x7F80 /* nIAlrtTh: Disabled */
+ 0xC7 0x0000 /* nTTFCfg */
+ 0xC9 0x002F /* nTCurve */
+ 0xCA 0xF1F1 /* nTGain */
+ 0xCB 0x198A /* nTOff */
+>;
diff --git a/arch/arm64/boot/dts/google/batterydata-maxim-b1-47k.dtsi b/arch/arm64/boot/dts/google/batterydata-maxim-b1-47k.dtsi
new file mode 100644
index 0000000..0a119d3
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-maxim-b1-47k.dtsi
@@ -0,0 +1,39 @@
+maxim,n_regval = /bits/ 16 <
+ /* Desay_B1_ATL_EVT1_Vempty3.3_ALERT__LearnCGF_2361.INI */
+ /* Generated on 2018/03/02 09:16:25 */
+ 0x80 0x0020 /* nXTable0 */
+ 0x81 0x004F /* nXTable1 */
+ 0x82 0x01E1 /* nXTable2 */
+ 0x83 0x1081 /* nXTable3 */
+ 0x84 0x13F7 /* nXTable4 */
+ 0x85 0x1DEE /* nXTable5 */
+ 0x86 0x0EB0 /* nXTable6 */
+ 0x87 0x0C98 /* nXTable7 */
+ 0x88 0x08C3 /* nXTable8 */
+ 0x89 0x0800 /* nXTable9 */
+ 0x8A 0x0737 /* nXTable10 */
+ 0x8B 0x0766 /* nXTable11 */
+ 0x90 0x7EBB /* nOCVTable0 */
+ 0x91 0x7F27 /* nOCVTable1 */
+ 0x92 0xAF4C /* nOCVTable2 */
+ 0x93 0xB9FB /* nOCVTable3 */
+ 0x94 0xBB4D /* nOCVTable4 */
+ 0x95 0xBC7B /* nOCVTable5 */
+ 0x96 0xBF3C /* nOCVTable6 */
+ 0x97 0xC0D5 /* nOCVTable7 */
+ 0x98 0xC2D9 /* nOCVTable8 */
+ 0x99 0xC8BD /* nOCVTable9 */
+ 0x9A 0xCD55 /* nOCVTable10 */
+ 0x9B 0xD255 /* nOCVTable11 */
+ 0x9E 0xA561 /* nVEmpty: VE=3300 VR=3880 */
+ 0xB1 0x0204 /* nRippleCfg */
+ 0xB5 0x9C01 /* nPackCfg */
+ 0xB6 0x0000 /* nRelaxCfg */
+ 0xB9 0xA00E /* nNVCfg1 */
+ 0xBB 0x0002 /* nSBSCfg */
+ 0xC3 0x7F80 /* nIAlrtTh: Disabled */
+ 0xC7 0x0000 /* nTTFCfg */
+ 0xC9 0x002F /* nTCurve */
+ 0xCA 0xF1F1 /* nTGain */
+ 0xCB 0x198A /* nTOff */
+>;
diff --git a/arch/arm64/boot/dts/google/batterydata-maxim-b1-68k.dtsi b/arch/arm64/boot/dts/google/batterydata-maxim-b1-68k.dtsi
new file mode 100644
index 0000000..5ffa513
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-maxim-b1-68k.dtsi
@@ -0,0 +1,39 @@
+maxim,n_regval = /bits/ 16 <
+ /* Sunwoda_B1_ATL_EVT1_Vempty3.3_ALERT_LearnCFG_2367.INI */
+ /* Generated on 2018/03/06 09:39:12 */
+ 0x80 0x0021 /* nXTable0 */
+ 0x81 0x0081 /* nXTable1 */
+ 0x82 0x03E1 /* nXTable2 */
+ 0x83 0x0D81 /* nXTable3 */
+ 0x84 0x1604 /* nXTable4 */
+ 0x85 0x24B6 /* nXTable5 */
+ 0x86 0x12F0 /* nXTable6 */
+ 0x87 0x0869 /* nXTable7 */
+ 0x88 0x0A01 /* nXTable8 */
+ 0x89 0x07E0 /* nXTable9 */
+ 0x8A 0x0797 /* nXTable10 */
+ 0x8B 0x0700 /* nXTable11 */
+ 0x90 0x946B /* nOCVTable0 */
+ 0x91 0x94C7 /* nOCVTable1 */
+ 0x92 0xB549 /* nOCVTable2 */
+ 0x93 0xB8FB /* nOCVTable3 */
+ 0x94 0xBB5D /* nOCVTable4 */
+ 0x95 0xBC87 /* nOCVTable5 */
+ 0x96 0xBF0C /* nOCVTable6 */
+ 0x97 0xC116 /* nOCVTable7 */
+ 0x98 0xC322 /* nOCVTable8 */
+ 0x99 0xC86D /* nOCVTable9 */
+ 0x9A 0xCAD1 /* nOCVTable10 */
+ 0x9B 0xCE17 /* nOCVTable11 */
+ 0x9E 0xA561 /* nVEmpty: VE=3300 VR=3880 */
+ 0xB1 0x0204 /* nRippleCfg */
+ 0xB5 0x9C01 /* nPackCfg */
+ 0xB6 0x0000 /* nRelaxCfg */
+ 0xB9 0xA00E /* nNVCfg1 */
+ 0xBB 0x0002 /* nSBSCfg */
+ 0xC3 0x7F80 /* nIAlrtTh: Disabled */
+ 0xC7 0x0000 /* nTTFCfg */
+ 0xC9 0x002F /* nTCurve */
+ 0xCA 0xF1F1 /* nTGain */
+ 0xCB 0x198A /* nTOff */
+>;
diff --git a/arch/arm64/boot/dts/google/batterydata-maxim-c1-10k.dtsi b/arch/arm64/boot/dts/google/batterydata-maxim-c1-10k.dtsi
new file mode 100644
index 0000000..016ea01
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-maxim-c1-10k.dtsi
@@ -0,0 +1,39 @@
+maxim,n_regval = /bits/ 16 <
+ /* Desay_C1_Proto_ATL_ALERT_LearnCFG_2363.INI */
+ /* Generated on 2018/03/08 22:14:08 */
+ 0x80 0x0021 /* nXTable0 */
+ 0x81 0x0022 /* nXTable1 */
+ 0x82 0x02A3 /* nXTable2 */
+ 0x83 0x0EF1 /* nXTable3 */
+ 0x84 0x18C9 /* nXTable4 */
+ 0x85 0x1610 /* nXTable5 */
+ 0x86 0x0E80 /* nXTable6 */
+ 0x87 0x0F48 /* nXTable7 */
+ 0x88 0x085B /* nXTable8 */
+ 0x89 0x0800 /* nXTable9 */
+ 0x8A 0x08C6 /* nXTable10 */
+ 0x8B 0x06CC /* nXTable11 */
+ 0x90 0x7D0B /* nOCVTable0 */
+ 0x91 0x7D48 /* nOCVTable1 */
+ 0x92 0xAE41 /* nOCVTable2 */
+ 0x93 0xBA2B /* nOCVTable3 */
+ 0x94 0xBB8E /* nOCVTable4 */
+ 0x95 0xBD34 /* nOCVTable5 */
+ 0x96 0xBF5C /* nOCVTable6 */
+ 0x97 0xC0E5 /* nOCVTable7 */
+ 0x98 0xC251 /* nOCVTable8 */
+ 0x99 0xC7CD /* nOCVTable9 */
+ 0x9A 0xCA93 /* nOCVTable10 */
+ 0x9B 0xCD68 /* nOCVTable11 */
+ 0x9E 0xA561 /* nVEmpty: VE=3300 VR=3880 */
+ 0xB1 0x0204 /* nRippleCfg */
+ 0xB5 0x9C01 /* nPackCfg */
+ 0xB6 0x0000 /* nRelaxCfg */
+ 0xB9 0xA00E /* nNVCfg1 */
+ 0xBB 0x0002 /* nSBSCfg */
+ 0xC3 0x7F80 /* nIAlrtTh: Disabled */
+ 0xC7 0x0000 /* nTTFCfg */
+ 0xC9 0x002F /* nTCurve */
+ 0xCA 0xF1F1 /* nTGain */
+ 0xCB 0x198A /* nTOff */
+>;
diff --git a/arch/arm64/boot/dts/google/batterydata-maxim-c1-180k.dtsi b/arch/arm64/boot/dts/google/batterydata-maxim-c1-180k.dtsi
new file mode 100644
index 0000000..016ea01
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-maxim-c1-180k.dtsi
@@ -0,0 +1,39 @@
+maxim,n_regval = /bits/ 16 <
+ /* Desay_C1_Proto_ATL_ALERT_LearnCFG_2363.INI */
+ /* Generated on 2018/03/08 22:14:08 */
+ 0x80 0x0021 /* nXTable0 */
+ 0x81 0x0022 /* nXTable1 */
+ 0x82 0x02A3 /* nXTable2 */
+ 0x83 0x0EF1 /* nXTable3 */
+ 0x84 0x18C9 /* nXTable4 */
+ 0x85 0x1610 /* nXTable5 */
+ 0x86 0x0E80 /* nXTable6 */
+ 0x87 0x0F48 /* nXTable7 */
+ 0x88 0x085B /* nXTable8 */
+ 0x89 0x0800 /* nXTable9 */
+ 0x8A 0x08C6 /* nXTable10 */
+ 0x8B 0x06CC /* nXTable11 */
+ 0x90 0x7D0B /* nOCVTable0 */
+ 0x91 0x7D48 /* nOCVTable1 */
+ 0x92 0xAE41 /* nOCVTable2 */
+ 0x93 0xBA2B /* nOCVTable3 */
+ 0x94 0xBB8E /* nOCVTable4 */
+ 0x95 0xBD34 /* nOCVTable5 */
+ 0x96 0xBF5C /* nOCVTable6 */
+ 0x97 0xC0E5 /* nOCVTable7 */
+ 0x98 0xC251 /* nOCVTable8 */
+ 0x99 0xC7CD /* nOCVTable9 */
+ 0x9A 0xCA93 /* nOCVTable10 */
+ 0x9B 0xCD68 /* nOCVTable11 */
+ 0x9E 0xA561 /* nVEmpty: VE=3300 VR=3880 */
+ 0xB1 0x0204 /* nRippleCfg */
+ 0xB5 0x9C01 /* nPackCfg */
+ 0xB6 0x0000 /* nRelaxCfg */
+ 0xB9 0xA00E /* nNVCfg1 */
+ 0xBB 0x0002 /* nSBSCfg */
+ 0xC3 0x7F80 /* nIAlrtTh: Disabled */
+ 0xC7 0x0000 /* nTTFCfg */
+ 0xC9 0x002F /* nTCurve */
+ 0xCA 0xF1F1 /* nTGain */
+ 0xCB 0x198A /* nTOff */
+>;
diff --git a/arch/arm64/boot/dts/google/batterydata-maxim-c1-47k.dtsi b/arch/arm64/boot/dts/google/batterydata-maxim-c1-47k.dtsi
new file mode 100644
index 0000000..7d69ab9
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-maxim-c1-47k.dtsi
@@ -0,0 +1,39 @@
+maxim,n_regval = /bits/ 16 <
+ /* Sunwoda_C1_Proto_ATL_ALERT_LearnCFG_2380.INI */
+ /* Generated on 2018/03/13 18:14:44 */
+ 0x80 0x0021 /* nXTable0 */
+ 0x81 0x0022 /* nXTable1 */
+ 0x82 0x04F3 /* nXTable2 */
+ 0x83 0x0DF1 /* nXTable3 */
+ 0x84 0x1B6D /* nXTable4 */
+ 0x85 0x1976 /* nXTable5 */
+ 0x86 0x0FB0 /* nXTable6 */
+ 0x87 0x11A9 /* nXTable7 */
+ 0x88 0x0951 /* nXTable8 */
+ 0x89 0x0780 /* nXTable9 */
+ 0x8A 0x0756 /* nXTable10 */
+ 0x8B 0x0611 /* nXTable11 */
+ 0x90 0x7D0B /* nOCVTable0 */
+ 0x91 0x7D68 /* nOCVTable1 */
+ 0x92 0xB2F3 /* nOCVTable2 */
+ 0x93 0xBA1B /* nOCVTable3 */
+ 0x94 0xBBCD /* nOCVTable4 */
+ 0x95 0xBCCC /* nOCVTable5 */
+ 0x96 0xBEAC /* nOCVTable6 */
+ 0x97 0xC015 /* nOCVTable7 */
+ 0x98 0xC144 /* nOCVTable8 */
+ 0x99 0xC82D /* nOCVTable9 */
+ 0x9A 0xCDA6 /* nOCVTable10 */
+ 0x9B 0xD48E /* nOCVTable11 */
+ 0x9E 0xA561 /* nVEmpty: VE=3300 VR=3880 */
+ 0xB1 0x0204 /* nRippleCfg */
+ 0xB5 0x9C01 /* nPackCfg */
+ 0xB6 0x0000 /* nRelaxCfg */
+ 0xB9 0xA00E /* nNVCfg1 */
+ 0xBB 0x0002 /* nSBSCfg */
+ 0xC3 0x7F80 /* nIAlrtTh: Disabled */
+ 0xC7 0x0000 /* nTTFCfg */
+ 0xC9 0x002F /* nTCurve */
+ 0xCA 0xF1F1 /* nTGain */
+ 0xCB 0x198A /* nTOff */
+>;
diff --git a/arch/arm64/boot/dts/google/batterydata-maxim-c1-68k.dtsi b/arch/arm64/boot/dts/google/batterydata-maxim-c1-68k.dtsi
new file mode 100644
index 0000000..27658ad
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-maxim-c1-68k.dtsi
@@ -0,0 +1,39 @@
+maxim,n_regval = /bits/ 16 <
+ /* Sunwoda_C1_Proto_COSLIGHT_ALERT_LearnCFG_2381.INI */
+ /* Generated on 2018/03/13 18:17:33 */
+ 0x80 0x0020 /* nXTable0 */
+ 0x81 0x0025 /* nXTable1 */
+ 0x82 0x0290 /* nXTable2 */
+ 0x83 0x1601 /* nXTable3 */
+ 0x84 0x0B27 /* nXTable4 */
+ 0x85 0x2D4F /* nXTable5 */
+ 0x86 0x1BA0 /* nXTable6 */
+ 0x87 0x1027 /* nXTable7 */
+ 0x88 0x0FFF /* nXTable8 */
+ 0x89 0x0A20 /* nXTable9 */
+ 0x8A 0x07B7 /* nXTable10 */
+ 0x8B 0x0788 /* nXTable11 */
+ 0x90 0x7D0B /* nOCVTable0 */
+ 0x91 0x7D56 /* nOCVTable1 */
+ 0x92 0xAD07 /* nOCVTable2 */
+ 0x93 0xB85B /* nOCVTable3 */
+ 0x94 0xB9DC /* nOCVTable4 */
+ 0x95 0xBBE6 /* nOCVTable5 */
+ 0x96 0xBD8C /* nOCVTable6 */
+ 0x97 0xBEC1 /* nOCVTable7 */
+ 0x98 0xC03A /* nOCVTable8 */
+ 0x99 0xC5ED /* nOCVTable9 */
+ 0x9A 0xC8D2 /* nOCVTable10 */
+ 0x9B 0xCD14 /* nOCVTable11 */
+ 0x9E 0xA561 /* nVEmpty: VE=3300 VR=3880 */
+ 0xB1 0x0204 /* nRippleCfg */
+ 0xB5 0x9C01 /* nPackCfg */
+ 0xB6 0x0000 /* nRelaxCfg */
+ 0xB9 0xA00E /* nNVCfg1 */
+ 0xBB 0x0002 /* nSBSCfg */
+ 0xC3 0x7F80 /* nIAlrtTh: Disabled */
+ 0xC7 0x0000 /* nTTFCfg */
+ 0xC9 0x002F /* nTCurve */
+ 0xCA 0xF1F1 /* nTGain */
+ 0xCB 0x198A /* nTOff */
+>;
diff --git a/arch/arm64/boot/dts/google/batterydata-qcom-b1-100k.dtsi b/arch/arm64/boot/dts/google/batterydata-qcom-b1-100k.dtsi
new file mode 100644
index 0000000..59e7b96
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-qcom-b1-100k.dtsi
@@ -0,0 +1,70 @@
+
+qcom,b1-100k {
+ qcom, = <24>;
+ /* #3353347_Google_B1SwdLwn_2970mAh_averaged_MasterSlave_Feb27th2018*/
+ qcom,max-voltage-uv = <4400000>;
+ qcom,nom-batt-capacity-mah = <2970>;
+ qcom,batt-id-kohm = <100>;
+ qcom,battery-beta = <4360>;
+ qcom,battery-type = "b1-100k";
+ qcom,checksum = <0xB33E>;
+ qcom,gui-version = "PMI8998GUI - 2.0.0.58";
+ qcom,fg-profile-data = [
+ 01 1F 55 06
+ A4 03 0A F4
+ D4 1C FF FB
+ B4 0D 97 0A
+ 1A 18 5E 23
+ 05 45 9F 53
+ 72 00 00 00
+ 11 00 00 00
+ 00 00 03 00
+ 1F CD 6B B4
+ 2C 00 08 00
+ 23 E2 5F E4
+ 1F 05 E1 01
+ 5E 15 39 F5
+ DE F4 8B 2A
+ 38 06 09 20
+ 27 00 14 00
+ C3 1F AF 05
+ 4B 0A 15 FD
+ BD 1C 1B 03
+ E7 15 39 12
+ 8E 18 EA 23
+ C8 4D 7D 5B
+ 6B 00 00 00
+ 0D 00 00 00
+ 00 00 82 CC
+ D8 C2 F9 B2
+ 1D 00 00 00
+ 52 00 5F E4
+ 4C 06 A9 F3
+ 23 ED 07 03
+ 17 F5 50 22
+ B6 33 CC FF
+ 07 10 00 00
+ AB 0B 66 46
+ 1D 00 40 00
+ CF 01 0A FA
+ FF 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ ];
+};
diff --git a/arch/arm64/boot/dts/google/batterydata-qcom-b1-10k.dtsi b/arch/arm64/boot/dts/google/batterydata-qcom-b1-10k.dtsi
new file mode 100644
index 0000000..e5547ff
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-qcom-b1-10k.dtsi
@@ -0,0 +1,71 @@
+
+qcom,b1-10k {
+ qcom, = <24>;
+ /* #3353338_Google_B1DsyLsn_2970mAh_averaged_MasterSlave_Feb27th2018*/
+ qcom,max-voltage-uv = <4400000>;
+ qcom,nom-batt-capacity-mah = <2970>;
+ qcom,batt-id-kohm = <10>;
+ qcom,battery-beta = <4360>;
+ qcom,fg-cc-cv-threshold-mv = <4390>;
+ qcom,battery-type = "b1-10k";
+ qcom,checksum = <0xBD5A>;
+ qcom,gui-version = "PMI8998GUI - 2.0.0.58";
+ qcom,fg-profile-data = [
+ 17 20 1C 05
+ C3 0A 31 FC
+ F8 1C 99 01
+ EA 04 99 03
+ F6 17 BE 23
+ 76 44 49 5A
+ 85 00 00 00
+ 10 00 00 00
+ 00 00 22 A5
+ 4D CD 96 B3
+ 20 00 08 00
+ 6B DB D2 07
+ 27 06 20 01
+ E2 04 CA 0B
+ 67 EC 7D 32
+ 1F 06 09 20
+ 27 00 14 00
+ AB 1F AC 05
+ 66 0A 5E 06
+ C3 1C 18 03
+ D5 15 63 12
+ 7B 18 27 2A
+ 5B 4D 3F 62
+ 67 00 00 00
+ 0C 00 00 00
+ 00 00 9A D5
+ 3E CA BD BB
+ 1A 00 00 00
+ 07 EA D2 07
+ 63 FD EB F2
+ 31 F4 05 03
+ 52 EA 76 1A
+ B1 33 CC FF
+ 07 10 00 00
+ 99 0B 66 46
+ 1A 00 40 00
+ B0 01 0A FA
+ FF 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ ];
+};
diff --git a/arch/arm64/boot/dts/google/batterydata-qcom-b1-22k.dtsi b/arch/arm64/boot/dts/google/batterydata-qcom-b1-22k.dtsi
new file mode 100644
index 0000000..afbf81a
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-qcom-b1-22k.dtsi
@@ -0,0 +1,71 @@
+
+qcom,b1-22k {
+ qcom, = <24>;
+ /* #3353346_Google_B1SwdLsn_2970mAh_averaged_MasterSlave_Feb27th2018*/
+ qcom,max-voltage-uv = <4400000>;
+ qcom,nom-batt-capacity-mah = <2970>;
+ qcom,batt-id-kohm = <22>;
+ qcom,battery-beta = <4360>;
+ qcom,fg-cc-cv-threshold-mv = <4390>;
+ qcom,battery-type = "b1-22k";
+ qcom,checksum = <0x74FF>;
+ qcom,gui-version = "PMI8998GUI - 2.0.0.58";
+ qcom,fg-profile-data = [
+ 5E 1F EE 05
+ 25 0A 5F FD
+ F0 1C 83 FB
+ 42 04 27 0A
+ 71 18 49 23
+ 00 45 C1 53
+ 7A 00 00 00
+ 11 00 00 00
+ 00 00 15 BD
+ A2 CD 2E C2
+ 22 00 08 00
+ FB DB BC E4
+ 5D FC 33 FA
+ 85 06 C3 0B
+ 43 DB 2F 32
+ 22 06 09 20
+ 27 00 14 00
+ 90 1F D9 05
+ 38 0A 27 FD
+ C7 1C 09 03
+ E3 15 51 12
+ 7F 18 27 2A
+ 5B 4D 40 62
+ 67 00 00 00
+ 0C 00 00 00
+ 00 00 0C CC
+ CE CA 6B B4
+ 1C 00 00 00
+ 47 EA BC E4
+ C6 06 97 F2
+ 97 F5 24 03
+ 19 FA 56 1A
+ B3 33 CC FF
+ 07 10 00 00
+ A1 0B 66 46
+ 1C 00 40 00
+ AD 01 0A FA
+ FF 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ ];
+};
diff --git a/arch/arm64/boot/dts/google/batterydata-qcom-b1-47k.dtsi b/arch/arm64/boot/dts/google/batterydata-qcom-b1-47k.dtsi
new file mode 100644
index 0000000..b4a0851
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-qcom-b1-47k.dtsi
@@ -0,0 +1,71 @@
+
+qcom,b1-47k {
+ qcom, = <24>;
+ /* #3353334_Google_B1DsyAtl_2970mAh_averaged_MasterSlave_Feb27th2018*/
+ qcom,max-voltage-uv = <4400000>;
+ qcom,nom-batt-capacity-mah = <2970>;
+ qcom,batt-id-kohm = <47>;
+ qcom,battery-beta = <4360>;
+ qcom,fg-cc-cv-threshold-mv = <4390>;
+ qcom,battery-type = "b1-47k";
+ qcom,checksum = <0x393A>;
+ qcom,gui-version = "PMI8998GUI - 2.0.0.58";
+ qcom,fg-profile-data = [
+ FD 1F 43 05
+ 9C 0A 46 06
+ F9 1C 99 01
+ E6 04 98 03
+ B5 17 D1 23
+ 7E 44 33 5A
+ 85 00 00 00
+ 11 00 00 00
+ 00 00 CD C4
+ 36 C5 68 C3
+ 1E 00 08 00
+ 74 DB 1A E4
+ DB FD 8A EB
+ 97 FB 82 12
+ 8F DA C4 32
+ 19 06 09 20
+ 27 00 14 00
+ DF 1F 75 05
+ 86 0A 91 FC
+ D1 1C D0 02
+ 43 0C 03 12
+ 91 18 D3 23
+ E8 4D 2E 5B
+ 6D 00 00 00
+ 0E 00 00 00
+ 00 00 F8 07
+ 89 BB AD CA
+ 19 00 00 00
+ 0A EB 1A E4
+ 1D 07 01 EB
+ 64 EA 23 0A
+ 87 FA 79 1A
+ AE 33 CC FF
+ 07 10 00 00
+ B6 0B 66 46
+ 19 00 40 00
+ B1 01 0A FA
+ FF 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ ];
+};
diff --git a/arch/arm64/boot/dts/google/batterydata-qcom-b1-68k.dtsi b/arch/arm64/boot/dts/google/batterydata-qcom-b1-68k.dtsi
new file mode 100644
index 0000000..26a79e7
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-qcom-b1-68k.dtsi
@@ -0,0 +1,71 @@
+
+qcom,b1-68k {
+ qcom, = <24>;
+ /* #3353425_Google_B1SWDATL_2970mAh_averaged_MasterSlave_Mar23rd2018*/
+ qcom,max-voltage-uv = <4400000>;
+ qcom,nom-batt-capacity-mah = <2970>;
+ qcom,batt-id-kohm = <68>;
+ qcom,battery-beta = <4360>;
+ qcom,fg-cc-cv-threshold-mv = <4390>;
+ qcom,battery-type = "b1-68k";
+ qcom,checksum = <0xDF92>;
+ qcom,gui-version = "PMI8998GUI - 2.0.0.58";
+ qcom,fg-profile-data = [
+ 65 1F EA 05
+ 25 0A B2 06
+ F6 1C B6 01
+ 63 04 18 0A
+ 2B 18 99 23
+ 9B 44 31 5A
+ 7B 00 00 00
+ 10 00 00 00
+ 00 00 FC 07
+ 36 C5 DE C3
+ 1F 00 08 00
+ D8 DB BF 07
+ F4 06 B3 EB
+ B4 01 6C 12
+ D9 FC 45 2B
+ 19 06 09 20
+ 27 00 14 00
+ EB 1F 6B 05
+ 8B 0A 8D FC
+ D6 1C C0 02
+ 65 0C DD 0B
+ 8E 18 D8 23
+ E4 4D 34 5B
+ 6F 00 00 00
+ 0E 00 00 00
+ 00 00 40 CD
+ D8 BB 96 B3
+ 1B 00 00 00
+ 1C EB BF 07
+ DF F4 5C EA
+ 2E EA DA 03
+ AB F3 C5 1A
+ AB 33 CC FF
+ 07 10 00 00
+ A6 0B 66 46
+ 1B 00 40 00
+ A4 01 0A FA
+ FF 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ ];
+};
diff --git a/arch/arm64/boot/dts/google/batterydata-qcom-c1-10k.dtsi b/arch/arm64/boot/dts/google/batterydata-qcom-c1-10k.dtsi
new file mode 100644
index 0000000..979449f
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-qcom-c1-10k.dtsi
@@ -0,0 +1,71 @@
+
+qcom,c1-10k {
+ qcom, = <24>;
+ /* #3353404_Google_C1DsyAtl_3480mAh_averaged_MasterSlave_Feb27th2018*/
+ qcom,max-voltage-uv = <4400000>;
+ qcom,nom-batt-capacity-mah = <3480>;
+ qcom,batt-id-kohm = <10>;
+ qcom,battery-beta = <4360>;
+ qcom,fg-cc-cv-threshold-mv = <4390>;
+ qcom,battery-type = "c1-10k";
+ qcom,checksum = <0xEC06>;
+ qcom,gui-version = "PMI8998GUI - 2.0.0.58";
+ qcom,fg-profile-data = [
+ AC 1F 8A 05
+ 78 0A 5B 06
+ F1 1C C0 01
+ 45 04 26 0A
+ 0C 18 77 23
+ EB 44 BF 53
+ 7C 00 00 00
+ 11 00 00 00
+ 00 00 61 CD
+ E1 C5 86 CA
+ 1A 00 08 00
+ 0C DB 9D E4
+ BD FD 8F 00
+ 97 02 FE 12
+ BC 06 42 2B
+ 1B 06 09 20
+ 27 00 14 00
+ 04 20 3E 05
+ B8 0A 12 06
+ CE 1C E0 02
+ 23 0C 15 12
+ 90 18 D1 23
+ E9 4D 2C 5B
+ 6D 00 00 00
+ 0E 00 00 00
+ 00 00 54 CC
+ 89 C2 24 CA
+ 17 00 00 00
+ A2 EA 9D E4
+ 8F F4 BC EA
+ 42 F2 7B 0A
+ 8F FD 4B 1A
+ 9E 33 CC FF
+ 07 10 00 00
+ 9D 0D 66 46
+ 17 00 40 00
+ B9 01 0A FA
+ FF 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ ];
+};
diff --git a/arch/arm64/boot/dts/google/batterydata-qcom-c1-180k.dtsi b/arch/arm64/boot/dts/google/batterydata-qcom-c1-180k.dtsi
new file mode 100644
index 0000000..2487be1
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-qcom-c1-180k.dtsi
@@ -0,0 +1,71 @@
+
+qcom,c1-180k {
+ qcom, = <24>;
+ /* #3353407_Google_C1DsyCos_3480mAh_averaged_MasterSlave_Feb27th2018*/
+ qcom,max-voltage-uv = <4400000>;
+ qcom,nom-batt-capacity-mah = <3480>;
+ qcom,batt-id-kohm = <180>;
+ qcom,battery-beta = <4360>;
+ qcom,fg-cc-cv-threshold-mv = <4390>;
+ qcom,battery-type = "c1-180k";
+ qcom,checksum = <0xC7C8>;
+ qcom,gui-version = "PMI8998GUI - 2.0.0.58";
+ qcom,fg-profile-data = [
+ 55 1F 05 FC
+ 03 0A DD 06
+ E6 1C 7F FB
+ 5E 04 0C 0A
+ FA 17 64 23
+ 12 45 76 53
+ 7D 00 00 00
+ 12 00 00 00
+ 00 00 B1 C2
+ EB CC 08 B5
+ 25 00 08 00
+ 11 DA 9B E5
+ 3E 05 CA 01
+ C9 15 C6 01
+ 1A E5 90 2A
+ 34 06 09 20
+ 27 00 14 00
+ 9E 1F DF 05
+ 21 0A B7 06
+ BD 1C 15 03
+ EB 15 37 12
+ 8C 18 CA 23
+ EF 4D 23 5B
+ 6A 00 00 00
+ 0E 00 00 00
+ 00 00 F6 07
+ CB C2 65 CB
+ 1A 00 00 00
+ C3 E3 9B E5
+ 6D FC 05 01
+ ED ED E2 03
+ 6F 02 C0 22
+ C6 33 CC FF
+ 07 10 00 00
+ C5 0D 66 46
+ 1A 00 40 00
+ BC 01 0A FA
+ FF 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ ];
+};
diff --git a/arch/arm64/boot/dts/google/batterydata-qcom-c1-47k.dtsi b/arch/arm64/boot/dts/google/batterydata-qcom-c1-47k.dtsi
new file mode 100644
index 0000000..1f91228
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-qcom-c1-47k.dtsi
@@ -0,0 +1,71 @@
+
+qcom,c1-47k {
+ qcom, = <24>;
+ /* #3353414_Google_C1SWDATL_3480mAh_averaged_MasterSlave_Mar23rd2018*/
+ qcom,max-voltage-uv = <4400000>;
+ qcom,nom-batt-capacity-mah = <3480>;
+ qcom,batt-id-kohm = <47>;
+ qcom,battery-beta = <4360>;
+ qcom,fg-cc-cv-threshold-mv = <4390>;
+ qcom,battery-type = "c1-47k";
+ qcom,checksum = <0xDFAB>;
+ qcom,gui-version = "PMI8998GUI - 2.0.0.58";
+ qcom,fg-profile-data = [
+ EB 20 3C 04
+ 5B 0B 92 05
+ 04 1D 74 01
+ 65 05 1D 03
+ F1 17 92 23
+ D0 44 E2 53
+ 8F 00 00 00
+ 11 00 00 00
+ 00 00 29 BC
+ CA CD 9D BA
+ 1F 00 08 00
+ 27 DB 4E E4
+ F8 06 75 00
+ 15 02 9A 12
+ B2 07 FF 2A
+ 1C 06 09 20
+ 27 00 14 00
+ 07 20 33 05
+ C7 0A FE 05
+ C8 1C F2 02
+ 00 0C 2A 12
+ 94 18 C9 23
+ EE 4D 24 5B
+ 6C 00 00 00
+ 0E 00 00 00
+ 00 00 5B CC
+ 19 C3 27 C3
+ 1A 00 00 00
+ E9 EA 4E E4
+ 29 07 FB EA
+ EE E2 84 03
+ DB FC CA 1A
+ 9C 33 CC FF
+ 07 10 00 00
+ 95 0D 66 46
+ 1A 00 40 00
+ 94 01 0A FA
+ FF 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ ];
+};
diff --git a/arch/arm64/boot/dts/google/batterydata-qcom-c1-68k.dtsi b/arch/arm64/boot/dts/google/batterydata-qcom-c1-68k.dtsi
new file mode 100644
index 0000000..2bab225
--- /dev/null
+++ b/arch/arm64/boot/dts/google/batterydata-qcom-c1-68k.dtsi
@@ -0,0 +1,71 @@
+
+qcom,c1-68k {
+ qcom, = <24>;
+ /* #3353422_Google_C1SWDCOS_3480mAh_averaged_MasterSlave_Mar23rd2018*/
+ qcom,max-voltage-uv = <4400000>;
+ qcom,nom-batt-capacity-mah = <3480>;
+ qcom,batt-id-kohm = <68>;
+ qcom,battery-beta = <4360>;
+ qcom,fg-cc-cv-threshold-mv = <4390>;
+ qcom,battery-type = "c1-68k";
+ qcom,checksum = <0x5417>;
+ qcom,gui-version = "PMI8998GUI - 2.0.0.58";
+ qcom,fg-profile-data = [
+ 95 1E B5 FD
+ CF 02 6D 07
+ C0 1C 4A 02
+ 1E 0D 49 0B
+ DD 17 98 23
+ BC 44 0B 5A
+ 66 00 00 00
+ 11 00 00 00
+ 00 00 D8 C2
+ A2 CD 0D A2
+ 2A 00 08 00
+ 5A DB 25 E5
+ 2D 05 D1 01
+ 7E 15 0A 00
+ A1 D4 41 2B
+ 35 06 09 20
+ 27 00 14 00
+ E8 1F 99 05
+ 4B 0A 2D FD
+ CE 1C D7 02
+ 62 0C BE 0B
+ 93 18 AF 23
+ 1C 44 D9 5A
+ 71 00 00 00
+ 0E 00 00 00
+ 00 00 6E CC
+ 55 C2 89 C2
+ 1E 00 00 00
+ 06 EA 25 E5
+ 3E 06 E1 F3
+ 71 F5 6C 02
+ 86 02 3C 23
+ C9 33 CC FF
+ 07 10 00 00
+ C1 0D 66 46
+ 1E 00 40 00
+ A7 01 0A FA
+ FF 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ 00 00 00 00
+ ];
+};
diff --git a/arch/arm64/boot/dts/google/dsi-panel-s6e3ha8-dsc-wqhd-cmd.dtsi b/arch/arm64/boot/dts/google/dsi-panel-s6e3ha8-dsc-wqhd-cmd.dtsi
new file mode 100644
index 0000000..3f2fa1c
--- /dev/null
+++ b/arch/arm64/boot/dts/google/dsi-panel-s6e3ha8-dsc-wqhd-cmd.dtsi
@@ -0,0 +1,259 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&mdss_mdp {
+ dsi_s6e3ha8_dsc_wqhd_cmd: qcom,mdss_dsi_s6e3ha8 {
+ compatible = "google,dsi_binned_lp";
+
+ qcom,mdss-dsi-panel-name = "S6E3HA8 6.3 command mode panel";
+ google,mdss-dsi-panel-vendor = "SAMSUNG";
+ google,mdss-dsi-panel-sn-location = <0xA1 6 7>;
+ qcom,mdss-dsi-panel-type = "dsi_cmd_mode";
+ qcom,mdss-dsi-virtual-channel-id = <0>;
+ qcom,mdss-dsi-bpp = <24>;
+ qcom,mdss-dsi-color-order = "rgb_swap_rgb";
+ qcom,mdss-dsi-underflow-color = <0xff>;
+ qcom,mdss-dsi-traffic-mode = "non_burst_sync_event";
+ qcom,mdss-dsi-lane-map = "lane_map_0123";
+ qcom,mdss-dsi-panel-broadcast-mode;
+ qcom,mdss-dsi-lane-0-state;
+ qcom,mdss-dsi-lane-1-state;
+ qcom,mdss-dsi-lane-2-state;
+ qcom,mdss-dsi-lane-3-state;
+ qcom,mdss-dsi-reset-sequence = <1 10>, <0 1>, <1 0>;
+
+ qcom,mdss-dsi-bl-pmic-control-type = "bl_ctrl_dcs";
+ qcom,mdss-dsi-bl-vr-min-safe-level = <0xa>;
+ qcom,mdss-dsi-bl-min-level = <0x5>;
+ qcom,mdss-dsi-bl-max-level = <0x3ff>;
+ qcom,mdss-dsi-panel-hdr-enabled;
+ qcom,mdss-dsi-panel-hdr-color-primaries = <14500 15500 32000
+ 17000 15500 30000 8000 3000>;
+ qcom,mdss-dsi-panel-peak-brightness = <5530000>;
+ qcom,mdss-dsi-panel-blackness-level = <40>;
+
+ qcom,mdss-dsi-dma-trigger = "trigger_sw";
+ qcom,mdss-dsi-mdp-trigger = "none";
+ qcom,mdss-dsi-wr-mem-start = <0x2c>;
+ qcom,mdss-dsi-wr-mem-continue = <0x3c>;
+ qcom,mdss-dsi-te-pin-select = <1>;
+ qcom,mdss-dsi-te-dcs-command = <1>;
+
+ qcom,mdss-dsi-te-check-enable;
+ qcom,mdss-dsi-te-using-te-pin;
+ /* qcom,esd-check-enabled; */
+ qcom,mdss-dsi-panel-status-check-mode = "te_signal_check";
+
+ qcom,mdss-dsi-init-delay-us = <5000>;
+
+ qcom,mdss-pan-physical-width-dimension = <70>;
+ qcom,mdss-pan-physical-height-dimension = <144>;
+
+ qcom,mdss-dsi-display-timings {
+ timing@0 {
+ qcom,mdss-dsi-panel-framerate = <60>;
+ qcom,mdss-dsi-panel-width = <1440>;
+ qcom,mdss-dsi-panel-height = <2960>;
+ qcom,mdss-dsi-h-front-porch = <116>;
+ qcom,mdss-dsi-h-back-porch = <116>;
+ qcom,mdss-dsi-h-pulse-width = <44>;
+ qcom,mdss-dsi-h-sync-skew = <0>;
+ qcom,mdss-dsi-v-back-porch = <80>;
+ qcom,mdss-dsi-v-front-porch = <124>;
+ qcom,mdss-dsi-v-pulse-width = <120>;
+ qcom,mdss-dsi-h-left-border = <0>;
+ qcom,mdss-dsi-h-right-border = <0>;
+ qcom,mdss-dsi-v-top-border = <0>;
+ qcom,mdss-dsi-v-bottom-border = <0>;
+ qcom,mdss-dsi-on-command = [
+ 07 01 00 00 00 00 01 01 /* Compression Enable */
+ 05 01 00 00 78 00 01 11 /* sleep out, Wait 120 ms */
+ 15 01 00 00 00 00 02 53 20 /* Write Control Display */
+ 05 01 00 00 00 00 02 35 00 /* TE on */
+ 39 01 00 00 00 00 03 F0 5A 5A /* Unlock */
+ 39 01 00 00 00 00 0A B9 01 B0 81 09 00 00 00 11 03 /* Sync display with touch */
+ 39 01 00 00 00 00 03 F0 A5 A5 /* Lock */
+ 05 01 00 00 00 00 01 29 /* display on */
+ ];
+ qcom,mdss-dsi-on-command-state = "dsi_lp_mode";
+ qcom,mdss-dsi-off-command = [
+ 05 01 00 00 0A 00 02 28 00 /* display off */
+ 05 01 00 00 78 00 02 10 00 /* sleep in */
+ ];
+
+ qcom,mdss-dsi-vr-command = [
+ 39 01 00 00 00 00 03 F0 5A 5A
+ 15 01 00 00 00 00 02 B8 11
+ 39 01 00 00 00 00 05 B1 02 54 0C 00
+ 15 01 00 00 00 00 02 B0 07
+ 15 01 00 00 00 00 02 B1 07
+ 15 01 00 00 00 00 02 B0 10
+ 39 01 00 00 00 00 03 CB 03 0E
+ 15 01 00 00 00 00 02 B0 25
+ 15 01 00 00 00 00 02 CB 15
+ 15 01 00 00 00 00 02 B0 43
+ 15 01 00 00 00 00 02 CB 8F
+ 15 01 00 00 00 00 02 F7 03
+ 39 01 00 00 00 00 03 F0 A5 A5
+ 39 01 00 00 00 00 03 51 00 BB
+ 15 01 00 00 00 00 02 53 20
+ ];
+
+ qcom,mdss-dsi-novr-command = [
+ 39 01 00 00 00 00 03 F0 5A 5A
+ 15 01 00 00 00 00 02 B8 15
+ 39 01 00 00 00 00 05 B1 00 0C 0C A0
+ 15 01 00 00 00 00 02 B0 08
+ 15 01 00 00 00 00 02 B1 0C
+ 15 01 00 00 00 00 02 B0 10
+ 39 01 00 00 00 00 03 CB 0B 06
+ 15 01 00 00 00 00 02 B0 25
+ 15 01 00 00 00 00 02 CB 09
+ 15 01 00 00 00 00 02 B0 43
+ 15 01 00 00 00 00 02 CB 0F
+ 15 01 00 00 14 00 02 F7 03
+ 39 01 00 00 00 00 03 F0 A5 A5
+ ];
+ qcom,mdss-dsi-off-command-state = "dsi_hs_mode";
+
+ qcom,mdss-dsi-nolp-command = [
+ 15 01 00 00 00 00 02 53 20
+ ];
+
+ qcom,compression-mode = "dsc";
+ qcom,mdss-dsc-slice-height = <40>;
+ qcom,mdss-dsc-slice-width = <720>;
+ qcom,mdss-dsc-slice-per-pkt = <1>;
+ qcom,mdss-dsc-bit-per-component = <8>;
+ qcom,mdss-dsc-bit-per-pixel = <8>;
+ qcom,mdss-dsc-block-prediction-enable;
+
+ /* 2xLM, 2xDSC Enc, 1xDSI Intf */
+ qcom,display-topology = <2 2 1>;
+ qcom,default-topology-index = <0>;
+ };
+ };
+
+ google,lp-modes {
+ lp-mode@0 {
+ label = "off";
+
+ google,dsi-lp-brightness-threshold = <0>;
+ google,dsi-lp-command = [
+ 39 01 00 00 00 00 03 51 00 00 /* turn screen dark */
+ 15 01 00 00 00 00 02 53 20 /* disable HLPM */
+ ];
+ google,dsi-lp-command-state = "dsi_lp_mode";
+ };
+
+ lp-mode@1 {
+ label = "low";
+
+ google,dsi-lp-brightness-threshold = <10>;
+ google,dsi-lp-command = [
+ 39 01 00 00 00 00 03 F0 5A 5A /* Unlock */
+ 15 01 00 00 00 00 02 B0 05
+ 15 01 00 00 00 00 02 BB 84
+ 15 01 00 00 00 00 02 53 03
+ 15 01 00 00 00 00 02 F7 03
+ 39 01 00 00 00 00 03 F0 A5 A5 /* Lock */
+ ];
+ google,dsi-lp-command-state = "dsi_lp_mode";
+ };
+
+ lp-mode@2 {
+ label = "high";
+ google,dsi-lp-brightness-threshold = <255>;
+ google,dsi-lp-command = [
+ 39 01 00 00 00 00 03 F0 5A 5A /* Unlock */
+ 15 01 00 00 00 00 02 B0 05
+ 15 01 00 00 00 00 02 BB 04
+ 15 01 00 00 00 00 02 53 02
+ 15 01 00 00 00 00 02 F7 03
+ 39 01 00 00 00 00 03 F0 A5 A5 /* Lock */
+ ];
+ google,dsi-lp-command-state = "dsi_lp_mode";
+ };
+ };
+
+ google,hbm-ranges {
+ google,dsi-hbm-exit-command = [
+ /* Disable high brightness and enable
+ * brightness dimming
+ */
+ 15 01 00 00 00 00 02 53 28
+ ];
+ google,dsi-hbm-exit-num-dimming-frames = <32>;
+ google,dsi-hbm-exit-dimming-stop-command = [
+ /* Disable high brightness and disable
+ * brightness dimming
+ */
+ 15 01 00 00 00 00 02 53 20
+ ];
+ google,dsi-hbm-commands-state = "dsi_lp_mode";
+
+ hbm-range@0 {
+ google,dsi-hbm-range-brightness-threshold = <1>;
+
+ google,dsi-hbm-range-bl-min-level = <0x5>;
+ google,dsi-hbm-range-bl-max-level = <0x3ff>;
+ google,dsi-hbm-range-entry-command = [
+ /* Disable high brightness and enable
+ * brightness dimming
+ */
+ 15 01 00 00 00 00 02 53 28
+ ];
+ google,dsi-hbm-range-num-dimming-frames = <32>;
+ google,dsi-hbm-range-dimming-stop-command = [
+ /* Disable high brightness and disable
+ * brightness dimming
+ */
+ 15 01 00 00 00 00 02 53 20
+ ];
+ google,dsi-hbm-range-commands-state =
+ "dsi_lp_mode";
+ };
+
+ hbm-range@1 {
+ /* Normal brightness mode covers ~2 to 400
+ * nits, using panel brightness values up to
+ * 0x3ff. HBM extends the range to 600 nits,
+ * and remaps user brightness 1-255 to cover
+ * this extended range. In this extended range,
+ * 400 nits / 0x3ff corresponds to user
+ * brightness 170. Panel brightness is not
+ * adjustable in the 400-600 nit segment.
+ */
+ google,dsi-hbm-range-brightness-threshold =
+ <170>;
+
+ google,dsi-hbm-range-bl-min-level = <0x3ff>;
+ google,dsi-hbm-range-bl-max-level = <0x3ff>;
+
+ google,dsi-hbm-range-entry-command = [
+ /* Enable high brightness */
+ 15 01 00 00 00 00 02 53 E8
+ ];
+ google,dsi-hbm-range-num-dimming-frames = <32>;
+ google,dsi-hbm-range-dimming-stop-command = [
+ /* Enable high brightness and disable
+ * brightness dimming
+ */
+ 15 01 00 00 00 00 02 53 E0
+ ];
+
+ google,dsi-hbm-range-commands-state =
+ "dsi_lp_mode";
+ };
+ };
+ };
+};
diff --git a/arch/arm64/boot/dts/google/dsi-panel-sw43402-dsc-qhd-cmd.dtsi b/arch/arm64/boot/dts/google/dsi-panel-sw43402-dsc-qhd-cmd.dtsi
new file mode 100644
index 0000000..51f474b
--- /dev/null
+++ b/arch/arm64/boot/dts/google/dsi-panel-sw43402-dsc-qhd-cmd.dtsi
@@ -0,0 +1,141 @@
+/* Copyright (c) 2017, LGE Inc. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&mdss_mdp {
+ dsi_sw43402_dsc_qhd_cmd: qcom,mdss_dsi_sw43402_dsc_qhd_cmd {
+ qcom,mdss-dsi-panel-name = "SW43402 cmd mode dsc dsi panel";
+ qcom,mdss-dsi-panel-type = "dsi_cmd_mode";
+ qcom,mdss-dsi-virtual-channel-id = <0>;
+ qcom,mdss-dsi-stream = <0>;
+ qcom,mdss-dsi-bpp = <24>;
+ qcom,mdss-dsi-color-order = "rgb_swap_rgb";
+ qcom,mdss-dsi-underflow-color = <0xff>;
+ qcom,mdss-dsi-border-color = <0>;
+ qcom,mdss-dsi-traffic-mode = "burst_mode";
+ qcom,mdss-dsi-lane-map = "lane_map_0123";
+ qcom,mdss-dsi-bllp-eof-power-mode;
+ qcom,mdss-dsi-bllp-power-mode;
+ /* qcom,partial-update-enabled = "single-roi"; */
+ /* qcom,panel-roi-alignment = <1440 16 1440 16 1440 16>; */
+ qcom,mdss-dsi-lane-0-state;
+ qcom,mdss-dsi-lane-1-state;
+ qcom,mdss-dsi-lane-2-state;
+ qcom,mdss-dsi-lane-3-state;
+ qcom,mdss-dsi-reset-sequence = <1 10>, <0 10>, <1 10>;
+ qcom,mdss-dsi-dma-trigger = "trigger_sw";
+ qcom,mdss-dsi-mdp-trigger = "none";
+ qcom,mdss-dsi-te-pin-select = <1>;
+ qcom,mdss-dsi-wr-mem-start = <0x2c>;
+ qcom,mdss-dsi-wr-mem-continue = <0x3c>;
+ qcom,mdss-dsi-te-dcs-command = <1>;
+ qcom,mdss-dsi-te-check-enable;
+ qcom,mdss-dsi-te-using-te-pin;
+ // todo: salidoa qcom,mdss-dsi-lp11-init;
+ qcom,mdss-dsi-tx-eot-append;
+ qcom,mdss-dsi-force-clock-lane-hs = <1>;
+
+ /* TODO: salidoa: Check if these lp-mode settings belong in
+ * the timing section
+ */
+ qcom,mdss-dsi-lp-mode-on = [15 01 00 00 00 00 02 53 08];
+ qcom,mdss-dsi-lp-mode-off = [15 01 00 00 00 00 02 53 00];
+ qcom,mdss-dsi-lp-change-state = "dsi_hs_mode";
+ qcom,mode-control-dsi-state = "dsi_lp_mode";
+
+ qcom,alpm-off-command = [
+ 05 01 00 00 00 00 01 38 /* IDLE off */
+ ];
+ qcom,alpm-low-command = [
+ 05 01 00 00 00 00 01 39 /* IDLE on */
+ 15 01 00 00 00 00 02 51 40 /* brightness control */
+ ];
+ qcom,alpm-high-command = [
+ 05 01 00 00 00 00 01 39 /* IDLE on */
+ 15 01 00 00 00 00 02 51 80 /* brightness control */
+ ];
+ qcom,mdss-pan-physical-width-dimension = <68>;
+ qcom,mdss-pan-physical-height-dimension = <136>;
+
+ qcom,mdss-dsi-bl-pmic-control-type = "bl_ctrl_dcs";
+ qcom,mdss-dsi-bl-min-level = <23>;
+ qcom,mdss-dsi-bl-max-level = <223>;
+
+ qcom,mdss-dsi-display-timings {
+ timing@0 {
+ qcom,mdss-dsi-panel-framerate = <60>;
+ qcom,mdss-dsi-panel-width = <1440>;
+ qcom,mdss-dsi-panel-height = <2880>;
+ qcom,mdss-dsi-h-front-porch = <20>;
+ qcom,mdss-dsi-h-back-porch = <20>;
+ qcom,mdss-dsi-h-pulse-width = <32>;
+ qcom,mdss-dsi-h-sync-skew = <0>;
+ qcom,mdss-dsi-v-back-porch = <20>;
+ qcom,mdss-dsi-v-front-porch = <20>;
+ qcom,mdss-dsi-v-pulse-width = <4>;
+ qcom,mdss-dsi-h-left-border = <0>;
+ qcom,mdss-dsi-h-right-border = <0>;
+ qcom,mdss-dsi-v-top-border = <0>;
+ qcom,mdss-dsi-v-bottom-border = <0>;
+ qcom,mdss-dsi-on-command = [
+ 39 01 00 00 00 00 03 B0
+ 20 43
+ 39 01 00 00 00 00 03 DF
+ 08 00
+ 39 01 00 00 00 00 03 B0
+ A5 00
+ 05 01 00 00 00 00 01 35
+ 05 01 00 00 3C 00 01 11
+ 15 01 00 00 00 00 02 53
+ 00
+ 15 01 00 00 00 00 02 55
+ 0C
+ 39 01 00 00 00 00 03 FB
+ 03 77
+ 39 01 00 00 00 00 06 ED
+ 13 00 07 00 00
+ 39 01 00 00 00 00 13 C9
+ 00 00 00 24 33 AE
+ FF FF FF 00 1F 3F
+ 5E 7D 91 AF CF FF
+ 39 01 00 00 14 00 0F E2
+ 20 0D 08 A8 0A AA
+ 04 A4 83 83 83 5C
+ 5C 5C
+ 05 01 00 00 00 00 01 29
+ ];
+ qcom,mdss-dsi-off-command = [
+ 39 01 00 00 00 00 05 E8
+ 08 90 18 05
+ 05 01 00 00 00 00 01 28
+ 05 01 00 00 64 00 01 10
+ ];
+ qcom,mdss-dsi-on-command-state = "dsi_lp_mode";
+ qcom,mdss-dsi-off-command-state = "dsi_lp_mode";
+ qcom,mdss-dsi-h-sync-pulse = <0>;
+ qcom,compression-mode = "dsc";
+ qcom,mdss-dsc-slice-height = <16>;
+ qcom,mdss-dsc-slice-width = <720>;
+ qcom,mdss-dsc-slice-per-pkt = <1>;
+ qcom,mdss-dsc-bit-per-component = <8>;
+ qcom,mdss-dsc-bit-per-pixel = <8>;
+ qcom,mdss-dsc-block-prediction-enable;
+
+
+ /* 2xLM, 2xDSC Enc, 1xDSI Intf */
+ qcom,display-topology = <2 2 1>;
+ qcom,default-topology-index = <0>;
+ };
+ };
+
+ };
+};
+
diff --git a/arch/arm64/boot/dts/google/dsi-panel-sw43408-dsc-fhd-cmd.dtsi b/arch/arm64/boot/dts/google/dsi-panel-sw43408-dsc-fhd-cmd.dtsi
new file mode 100644
index 0000000..e36ce04
--- /dev/null
+++ b/arch/arm64/boot/dts/google/dsi-panel-sw43408-dsc-fhd-cmd.dtsi
@@ -0,0 +1,279 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&mdss_mdp {
+ dsi_sw43408_dsc_fhd_cmd: qcom,mdss_dsi_sw43408_dsc_fhd_cmd {
+ compatible = "google,dsi_binned_lp";
+
+ qcom,mdss-dsi-panel-name = "sw43408 cmd mode dsc dsi panel";
+ google,mdss-dsi-panel-vendor = "LG";
+ google,mdss-dsi-panel-sn-location = <0xA9 0 9>;
+ qcom,mdss-dsi-panel-type = "dsi_cmd_mode";
+ qcom,mdss-dsi-virtual-channel-id = <0>;
+ qcom,mdss-dsi-stream = <0>;
+ qcom,mdss-dsi-bpp = <24>;
+ qcom,mdss-dsi-color-order = "rgb_swap_rgb";
+ qcom,mdss-dsi-underflow-color = <0xff>;
+ qcom,mdss-dsi-border-color = <0>;
+ qcom,mdss-dsi-traffic-mode = "burst_mode";
+ qcom,mdss-dsi-lane-map = "lane_map_0123";
+ /* qcom,partial-update-enabled = "single-roi"; */
+ /* qcom,panel-roi-alignment = <1440 16 1440 16 1440 16>; */
+ qcom,mdss-dsi-lane-0-state;
+ qcom,mdss-dsi-lane-1-state;
+ qcom,mdss-dsi-lane-2-state;
+ qcom,mdss-dsi-lane-3-state;
+ qcom,mdss-dsi-reset-sequence = <1 9>, <0 1>, <1 10>;
+ qcom,mdss-dsi-dma-trigger = "trigger_sw";
+ qcom,mdss-dsi-mdp-trigger = "none";
+ qcom,mdss-dsi-te-pin-select = <1>;
+ qcom,mdss-dsi-wr-mem-start = <0x2c>;
+ qcom,mdss-dsi-wr-mem-continue = <0x3c>;
+ qcom,mdss-dsi-te-dcs-command = <1>;
+ qcom,mdss-dsi-te-check-enable;
+ qcom,mdss-dsi-te-using-te-pin;
+ qcom,mdss-dsi-tx-eot-append;
+ qcom,mdss-dsi-force-clock-lane-hs = <1>;
+
+ qcom,mdss-pan-physical-width-dimension = <62>;
+ qcom,mdss-pan-physical-height-dimension = <124>;
+
+ qcom,mdss-dsi-hs-pps;
+ qcom,mdss-dsi-init-delay-us = <5000>;
+ qcom,mdss-dsi-lp11-init;
+
+ qcom,mdss-dsi-bl-pmic-control-type = "bl_ctrl_dcs";
+ qcom,mdss-dsi-bl-min-level = <0x8>; /* 2.74 nits */
+ qcom,mdss-dsi-bl-max-level = <0x384>; /* 399.81 nits */
+ qcom,mdss-dsi-panel-hdr-enabled;
+ qcom,mdss-dsi-panel-hdr-color-primaries = <14500 15500 32000
+ 17000 15500 30000 8000 3000>;
+ qcom,mdss-dsi-panel-peak-brightness = <6200000>;
+ qcom,mdss-dsi-panel-blackness-level = <42>;
+ google,mdss-dsi-lp-brightness-compensation = <0>;
+
+ qcom,mdss-dsi-display-timings {
+ timing@0 {
+ qcom,mdss-dsi-panel-framerate = <60>;
+ qcom,mdss-dsi-panel-width = <1080>;
+ qcom,mdss-dsi-panel-height = <2160>;
+ qcom,mdss-dsi-h-front-porch = <20>;
+ qcom,mdss-dsi-h-back-porch = <20>;
+ qcom,mdss-dsi-h-pulse-width = <32>;
+ qcom,mdss-dsi-h-sync-skew = <0>;
+ qcom,mdss-dsi-v-back-porch = <20>;
+ qcom,mdss-dsi-v-front-porch = <20>;
+ qcom,mdss-dsi-v-pulse-width = <4>;
+ qcom,mdss-dsi-h-left-border = <0>;
+ qcom,mdss-dsi-h-right-border = <0>;
+ qcom,mdss-dsi-v-top-border = <0>;
+ qcom,mdss-dsi-v-bottom-border = <0>;
+ qcom,mdss-dsi-init-delay-us = <10000>;
+ qcom,mdss-dsi-on-command = [
+ 15 01 00 00 00 00 02 26 02
+ 15 01 00 00 00 00 02 35 00
+ 39 01 00 00 00 00 03 53 0C 30
+ 39 01 00 00 00 00 07 55 00 70 DF 00 70 DF
+ 39 01 00 00 00 00 04 F7 01 49 0C
+ 05 01 00 00 87 00 01 11
+ 07 01 00 00 00 00 01 11
+ 15 01 00 00 00 00 02 B0 AC
+ 39 01 00 00 00 00 13 CD
+ 00 00 00 19 19 19 19 19
+ 19 19 19 19 19 19 19 19
+ 16 16
+ 39 01 00 00 00 00 06 CB 80 5C 07 03 28
+ 39 01 00 00 00 00 04 C0 02 02 0F
+ 39 01 00 00 00 00 08 E5
+ 00 3A 00 3A 00 0E 10
+ 39 01 00 00 00 00 2A B5
+ 75 60 2D 5D 80 00 0A 0B
+ 00 05 0B 00 80 0D 0E 40
+ 00 0C 00 16 00 B8 00 80
+ 0D 0E 40 00 0C 00 16 00
+ B8 00 81 00 03 03 03 01
+ 01
+ 39 01 00 00 00 00 07 55 04 61 DB 04 70 DB
+ 15 01 00 00 00 00 02 B0 CA
+ 05 01 00 00 32 00 01 29
+ ];
+ qcom,mdss-dsi-off-command = [
+ 05 01 00 00 00 00 01 28
+ 05 01 00 00 64 00 01 10
+ ];
+
+ qcom,mdss-dsi-lp1-command = [
+ 05 01 00 00 00 00 01 22
+ 39 01 00 00 00 00 05 30 00 00 08 6F
+ 39 01 00 00 00 00 05 31 00 00 04 37
+ 15 01 00 00 00 00 02 B0 AC
+ 39 01 00 00 00 00 07 55 04 61 DB 04 78 DB
+ 39 01 00 00 00 00 0D B4 2A 40 25 22 41 41 41 0A 10 50 11 03
+ 39 01 00 00 3C 00 06 CB 80 5C 07 09 28
+ ];
+
+ qcom,mdss-dsi-nolp-command = [
+ 05 01 00 00 00 00 01 22
+ 15 01 00 00 00 00 02 B0 AC
+ 39 01 00 00 00 00 0D B4 2A 02 25 22 41 41 41 0A 10 50 11 00
+ 39 01 00 00 00 00 06 CB 80 5C 07 03 28
+ 39 01 00 00 00 00 03 51 00 00
+ 05 01 00 00 3C 00 01 13
+ ];
+
+ qcom,mdss-dsi-vr-command = [
+ 05 01 00 00 64 00 01 28
+ /* display off; wait 100ms */
+ 15 01 00 00 00 00 02 B0 AC
+ /* MCS Access */
+ 39 01 00 00 00 00 08 E5
+ 00 3E 00 3E 00 0E 30
+ 39 01 00 00 00 00 2A B5
+ 75 60 2D 5D 80 00 0A 0B
+ 00 05 0B 00 80 0D 0E 40
+ 00 0C 00 16 00 B8 00 80
+ 0D 0E 40 00 0C 00 16 00
+ B8 00 81 00 03 1F 1F 02
+ 02
+ /* VR mode: enable EM inversion */
+ 15 01 00 00 00 00 02 B0 CA
+ /* Don't MCS Access */
+ 39 01 00 00 00 00 07 55 04 70 DB 04 70 DB
+ /* Write ACL - turn off edge rounding */
+ 39 01 00 00 00 00 03 51 01 A0
+ /* vr mode band1 (50 nit ) */
+ 39 01 00 00 00 00 03 53 2C 30
+ /* vr mode enable */
+ 05 01 00 00 00 00 01 29
+ /* display on */
+ ];
+
+ qcom,mdss-dsi-novr-command = [
+ 05 01 00 00 64 00 01 28
+ /* display off; wait 100ms */
+ 15 01 00 00 00 00 02 B0 AC
+ /* MCS Access */
+ 39 01 00 00 00 00 08 E5
+ 00 3A 00 3A 00 0E 10
+ 39 01 00 00 00 00 2A B5
+ 75 60 2D 5D 80 00 0A 0B
+ 00 05 0B 00 80 0D 0E 40
+ 00 0C 00 16 00 B8 00 80
+ 0D 0E 40 00 0C 00 16 00
+ B8 00 81 00 03 03 03 01
+ 01
+ /* Normal mode: disable EM inversion */
+ 15 01 00 00 00 00 02 B0 CA
+ /* Don't MCS Access */
+ 39 01 00 00 00 00 03 53 0C 30
+ /* vr mode disable */
+ 39 01 00 00 00 00 07 55 04 61 DB 04 70 DB
+ /* Write ACL - turn on edge rounding */
+ 05 01 00 00 00 00 01 29
+ /* display on */
+ ];
+
+ qcom,mdss-dsi-on-command-state = "dsi_lp_mode";
+ qcom,mdss-dsi-off-command-state = "dsi_lp_mode";
+ qcom,mdss-dsi-h-sync-pulse = <0>;
+ qcom,compression-mode = "dsc";
+ qcom,mdss-dsc-slice-height = <16>;
+ qcom,mdss-dsc-slice-width = <540>;
+ qcom,mdss-dsc-slice-per-pkt = <1>;
+ qcom,mdss-dsc-bit-per-component = <8>;
+ qcom,mdss-dsc-bit-per-pixel = <8>;
+ qcom,mdss-dsc-block-prediction-enable;
+
+ /* 2xLM, 2xDSC Enc, 1xDSI Intf */
+ qcom,display-topology = <2 2 1>;
+ qcom,default-topology-index = <0>;
+ };
+ };
+
+ google,lp-modes {
+ lp-mode@0 {
+ label = "off";
+
+ google,dsi-lp-brightness-threshold = <0>;
+ google,dsi-lp-command = [
+ 05 01 00 00 00 00 01 22
+ ];
+ google,dsi-lp-command-state = "dsi_lp_mode";
+ };
+
+ lp-mode@1 {
+ label = "dim";
+
+ google,dsi-lp-brightness-threshold = <5>;
+ google,dsi-lp-command = [
+ 39 01 00 00 00 00 03 51 00 6B
+ 05 01 00 00 00 00 01 12
+ ];
+ google,dsi-lp-command-state = "dsi_lp_mode";
+ };
+ lp-mode@2 {
+ label = "low";
+
+ google,dsi-lp-brightness-threshold = <10>;
+ google,dsi-lp-command = [
+ 39 01 00 00 00 00 03 51 00 E0
+ 05 01 00 00 00 00 01 12
+ ];
+ google,dsi-lp-command-state = "dsi_lp_mode";
+ };
+
+ lp-mode@3 {
+ label = "high";
+
+ google,dsi-lp-brightness-threshold = <255>;
+ google,dsi-lp-command = [
+ 39 01 00 00 00 00 03 51 03 87
+ 05 01 00 00 00 00 01 12
+ ];
+ google,dsi-lp-command-state = "dsi_lp_mode";
+ };
+ };
+
+ google,hbm-ranges {
+ google,dsi-hbm-exit-command = [
+ /* Enable brightness dimming */
+ 39 01 00 00 00 00 03 53 4C 30
+ ];
+ google,dsi-hbm-exit-num-dimming-frames = <32>;
+ google,dsi-hbm-exit-dimming-stop-command = [
+ /* Disable brightness dimming */
+ 39 01 00 00 00 00 03 53 0C 30
+ ];
+ google,dsi-hbm-commands-state = "dsi_lp_mode";
+
+ hbm-range@0 {
+ google,dsi-hbm-range-brightness-threshold = <1>;
+
+ google,dsi-hbm-range-bl-min-level = <0x8>;
+ google,dsi-hbm-range-bl-max-level = <0x3ff>;
+
+ google,dsi-hbm-range-entry-command = [
+ /* Enable brightness dimming */
+ 39 01 00 00 00 00 03 53 4C 30
+ ];
+ google,dsi-hbm-range-num-dimming-frames = <32>;
+ google,dsi-hbm-range-dimming-stop-command = [
+ /* Disable brightness dimming */
+ 39 01 00 00 00 00 03 53 0C 30
+ ];
+ google,dsi-hbm-range-commands-state =
+ "dsi_lp_mode";
+ };
+ };
+ };
+};
+
diff --git a/arch/arm64/boot/dts/google/sdm845-audio-overlay.dtsi b/arch/arm64/boot/dts/google/sdm845-audio-overlay.dtsi
new file mode 100644
index 0000000..8cebaff2
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-audio-overlay.dtsi
@@ -0,0 +1,133 @@
+/*
+ * Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include "sdm845-wcd.dtsi"
+#include <dt-bindings/clock/qcom,audio-ext-clk.h>
+
+&soc {
+ wcd_usbc_analog_en1_gpio: msm_cdc_pinctrl@49 {
+ compatible = "qcom,msm-cdc-pinctrl";
+ pinctrl-names = "aud_active", "aud_sleep";
+ pinctrl-0 = <&wcd_usbc_analog_en1_active>;
+ pinctrl-1 = <&wcd_usbc_analog_en1_idle>;
+ };
+
+ wcd9xxx_intc: wcd9xxx-irq {
+ status = "ok";
+ compatible = "qcom,wcd9xxx-irq";
+ interrupt-controller;
+ #interrupt-cells = <1>;
+ interrupt-parent = <&tlmm>;
+ qcom,gpio-connect = <&tlmm 54 0>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&wcd_intr_default>;
+ };
+
+ clock_audio_lnbb: audio_ext_clk_lnbb {
+ status = "ok";
+ compatible = "qcom,audio-ref-clk";
+ clock-names = "osr_clk";
+ clocks = <&clock_rpmh RPMH_LN_BB_CLK2>;
+ qcom,node_has_rpm_clock;
+ #clock-cells = <1>;
+ };
+
+ wcd_rst_gpio: msm_cdc_pinctrl@64 {
+ compatible = "qcom,msm-cdc-pinctrl";
+ qcom,cdc-rst-n-gpio = <&tlmm 64 0>;
+ pinctrl-names = "aud_active", "aud_sleep";
+ pinctrl-0 = <&cdc_reset_active>;
+ pinctrl-1 = <&cdc_reset_sleep>;
+ };
+
+ qocm,wcd-dsp-glink {
+ compatible = "qcom,wcd-dsp-glink";
+ };
+
+ qcom,wcd-dsp-mgr {
+ compatible = "qcom,wcd-dsp-mgr";
+ qcom,wdsp-components = <&wcd934x_cdc 0>,
+ <&wcd_spi_0 1>,
+ <&glink_spi_xprt_wdsp 2>;
+ qcom,img-filename = "cpe_9340";
+ };
+};
+
+&slim_aud {
+ wcd934x_cdc: tavil_codec {
+ compatible = "qcom,tavil-slim-pgd";
+ elemental-addr = [00 01 50 02 17 02];
+
+ interrupt-parent = <&wcd9xxx_intc>;
+ interrupts = <0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
+ 17 18 19 20 21 22 23 24 25 26 27 28 29
+ 30 31>;
+
+ qcom,wcd-rst-gpio-node = <&wcd_rst_gpio>;
+
+ clock-names = "wcd_clk";
+ clocks = <&clock_audio_lnbb AUDIO_PMIC_LNBB_CLK>;
+
+ cdc-vdd-buck-supply = <&pm8998_s4>;
+ qcom,cdc-vdd-buck-voltage = <1800000 1800000>;
+ qcom,cdc-vdd-buck-current = <650000>;
+
+ cdc-buck-sido-supply = <&pm8998_s4>;
+ qcom,cdc-buck-sido-voltage = <1800000 1800000>;
+ qcom,cdc-buck-sido-current = <250000>;
+
+ cdc-vdd-tx-h-supply = <&pm8998_s4>;
+ qcom,cdc-vdd-tx-h-voltage = <1800000 1800000>;
+ qcom,cdc-vdd-tx-h-current = <25000>;
+
+ cdc-vdd-rx-h-supply = <&pm8998_s4>;
+ qcom,cdc-vdd-rx-h-voltage = <1800000 1800000>;
+ qcom,cdc-vdd-rx-h-current = <25000>;
+
+ cdc-vddpx-1-supply = <&pm8998_s4>;
+ qcom,cdc-vddpx-1-voltage = <1800000 1800000>;
+ qcom,cdc-vddpx-1-current = <10000>;
+
+ qcom,cdc-static-supplies = "cdc-vdd-buck",
+ "cdc-buck-sido",
+ "cdc-vdd-tx-h",
+ "cdc-vdd-rx-h",
+ "cdc-vddpx-1";
+
+ qcom,cdc-micbias1-mv = <1800>;
+ qcom,cdc-micbias2-mv = <1800>;
+ qcom,cdc-micbias3-mv = <1800>;
+ qcom,cdc-micbias4-mv = <1800>;
+
+ qcom,cdc-mclk-clk-rate = <9600000>;
+ qcom,cdc-slim-ifd = "tavil-slim-ifd";
+ qcom,cdc-slim-ifd-elemental-addr = [00 00 50 02 17 02];
+ qcom,cdc-dmic-sample-rate = <4800000>;
+ qcom,cdc-mad-dmic-rate = <600000>;
+
+ qcom,wdsp-cmpnt-dev-name = "tavil_codec";
+
+ wcd_spi_0: wcd_spi {
+ compatible = "qcom,wcd-spi-v2";
+ qcom,master-bus-num = <0>;
+ qcom,chip-select = <0>;
+ qcom,max-frequency = <24000000>;
+ qcom,mem-base-addr = <0x100000>;
+
+ wcd_spi_ac {
+ compatible = "qcom,wcd-spi-ac";
+ };
+ };
+
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-audio-cal.dtsi b/arch/arm64/boot/dts/google/sdm845-b1-audio-cal.dtsi
new file mode 100644
index 0000000..bc4c4d3
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-audio-cal.dtsi
@@ -0,0 +1,21 @@
+&soc {
+ msm_cirrus_playback {
+ /* the resistance is 7.7296 ohm */
+ cirrus,top-speaker-impedance = <10811>;
+
+ /* the percentage of tolerance is 10 */
+ cirrus,top-speaker-tolerance = <10>;
+
+ /* the resistance is 6.9567 ohm */
+ cirrus,top-speaker-mean = <9730>;
+
+ /* the resistance is 7.0497 ohm */
+ cirrus,bottom-speaker-impedance = <9860>;
+
+ /* the percentage of tolerance is 10 */
+ cirrus,bottom-speaker-tolerance = <10>;
+
+ /* the resistance is 6.3447 ohm */
+ cirrus,bottom-speaker-mean = <8874>;
+ };
+};
\ No newline at end of file
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-battery.dtsi b/arch/arm64/boot/dts/google/sdm845-b1-battery.dtsi
new file mode 100644
index 0000000..2f78c6a
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-battery.dtsi
@@ -0,0 +1,91 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&vendor {
+ b1_batterydata: qcom,battery-data {
+ qcom,batt-id-range-pct = <10>;
+ /* Desay using Lishen cell - 10k
+ * Sunwoda using Lishen cell - 22k
+ * Desay using ATL cell - 47k
+ * Sunwoda using ATL cell - 68k
+ * Sunwoda using Liwinon cell - 100k
+ */
+ #include "batterydata-qcom-b1-10k.dtsi"
+ #include "batterydata-qcom-b1-22k.dtsi"
+ #include "batterydata-qcom-b1-47k.dtsi"
+ #include "batterydata-qcom-b1-68k.dtsi"
+ #include "batterydata-qcom-b1-100k.dtsi"
+ };
+};
+
+&pmi8998_fg {
+ qcom,battery-data = <&b1_batterydata>;
+ qcom,fg-chg-term-current = <145>;
+ qcom,fg-sys-term-current = <(-181)>;
+};
+
+&google_charger {
+ google,chg-battery-capacity = <2915>;
+ google,wlc-power-supply = "wireless";
+};
+
+&vendor {
+ b1_maxim_config: maxim,config {
+ maxim,b1-10k {
+ maxim,batt-id-kohm = <10>;
+#include "batterydata-maxim-b1-10k.dtsi"
+ };
+ maxim,b1-22k {
+ maxim,batt-id-kohm = <22>;
+#include "batterydata-maxim-b1-22k.dtsi"
+ };
+ maxim,b1-47k {
+ maxim,batt-id-kohm = <47>;
+#include "batterydata-maxim-b1-47k.dtsi"
+ };
+ maxim,b1-68k {
+ maxim,batt-id-kohm = <68>;
+#include "batterydata-maxim-b1-68k.dtsi"
+ };
+ maxim,b1-180k {
+ maxim,batt-id-kohm = <100>;
+#include "batterydata-maxim-b1-100k.dtsi"
+ };
+ };
+};
+
+&max1720x_fg {
+ maxim,config = <&b1_maxim_config>;
+ maxim,n_regval = /bits/ 16 <
+ 0x9C 0x0300 /* nIChgTerm = 240.0 mA */
+ 0xB7 0x33C1 /* nConvgCfg: RepLow=6% VoltLowOff=140mV MinSlopeX=8/16 RepL_per_stage=1% */
+ 0x9F 0x2606 /* nLearnCfg: Filt Empty */
+ 0xB0 0x7217 /* nConfig: SS TS VS Ten Aen dSOCen TAlrtEn */
+ 0xC0 0xFF00 /* nVAlrtTh: disabled */
+ 0xC1 0x7F80 /* nTAlrtTh: disabled */
+ 0xC2 0xFF00 /* nSAlrtTh: disabled */
+ 0x9D 0x0EB4 /* nFilterCfg: TEMP=90.0s MIX=12.8h VOLT=90.0s CURR=5.625s */
+ 0xC6 0x5A05 /* nFullSOCThr = 90% */
+ 0xD5 0x2D7D /* nRFastVShdn: rFast=87mohm VShdn=2.5V */
+ 0xB8 0xCDF0 /* nNVCfg0: enOCV enX enCfg enFCfg enLCfg enICT enCG enVE enDC */
+ 0xB9 0xB00E /* nNVCfg1: enTGO enFTh enRFVSH enAT enCRV enCTE */
+ 0xBA 0xFF0A /* nNVCfg2: enT enSOC enMMT enMMV enMMC enVT enFC enIAvg CYCLESpSAVE=5.0 */
+ 0xB3 0x0B80 /* nDesignCap: 2944 mAh */
+ >;
+};
+
+&soc {
+ google,overheat_mitigation {
+ google,begin-mitigation-temp = <550>;
+ google,end-mitigation-temp = <450>;
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-common-v2.1.dtsi b/arch/arm64/boot/dts/google/sdm845-b1-common-v2.1.dtsi
new file mode 100644
index 0000000..29a1e22
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-common-v2.1.dtsi
@@ -0,0 +1,5 @@
+#include "sdm845-b1-common.dtsi"
+
+/ {
+ qcom,msm-id = <321 0x20001>;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-common-v2.dtsi b/arch/arm64/boot/dts/google/sdm845-b1-common-v2.dtsi
new file mode 100644
index 0000000..ac01ac5
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-common-v2.dtsi
@@ -0,0 +1,5 @@
+#include "sdm845-b1-common.dtsi"
+
+/ {
+ qcom,msm-id = <321 0x20000>;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-common.dtsi b/arch/arm64/boot/dts/google/sdm845-b1-common.dtsi
new file mode 100644
index 0000000..c5bca53
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-common.dtsi
@@ -0,0 +1,30 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include "sdm845-b1c1-common.dtsi"
+
+#include "sdm845-b1-touch.dtsi"
+#include "sdm845-b1c1-audio.dtsi"
+#include "sdm845-b1-battery.dtsi"
+
+&dsi_sw43408_cmd_display {
+ qcom,dsi-display-active;
+};
+
+&p9221 {
+ fod = [a8 28 86 34 82 38 9a 12 9c 0b 84 65];
+ fod_epp = [a4 2a 86 36 84 3d 9a 14 9d 0b a2 fb];
+};
+
+&actuator_rear {
+ param-index = <5>;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-dvt.dts b/arch/arm64/boot/dts/google/sdm845-b1-dvt.dts
new file mode 100644
index 0000000..83df05a
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-dvt.dts
@@ -0,0 +1,32 @@
+/* Copyright (c) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-b1-common-v2.1.dtsi"
+#include "sdm845-patherm.dtsi"
+#include "sdm845-usbctherm.dtsi"
+#include "sdm845-b1-usb-v3.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 B1 DVT";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00021405 0>;
+};
+
+&nq {
+ ese {
+ loader_scripts_path =
+ "/vendor/etc/ese/loaderservice_updater_dvt_";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-dvt1.1.dts b/arch/arm64/boot/dts/google/sdm845-b1-dvt1.1.dts
new file mode 100644
index 0000000..618fd2c
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-dvt1.1.dts
@@ -0,0 +1,26 @@
+/* Copyright (c) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-b1-common-v2.1.dtsi"
+#include "sdm845-patherm.dtsi"
+#include "sdm845-usbctherm.dtsi"
+#include "sdm845-b1-usb-v3.dtsi"
+#include "sdm845-b1-audio-cal.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 B1 DVT1.1";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00021505 0>;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-evt1-panel.dtsi b/arch/arm64/boot/dts/google/sdm845-b1-evt1-panel.dtsi
new file mode 100644
index 0000000..4cdeacb
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-evt1-panel.dtsi
@@ -0,0 +1,92 @@
+/* Copyright (c) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/* We need override the display on sequence for EVT1 devices. */
+&dsi_sw43408_dsc_fhd_cmd {
+ qcom,mdss-dsi-display-timings {
+ timing@0 {
+ qcom,mdss-dsi-on-command = [
+ 15 01 00 00 00 00 02 35 00
+
+ 39 01 00 00 00 00 03 53 0C 30
+
+ 05 01 00 00 C8 00 01 11
+ /* TODO: delay specified as 100ms. 0xC8 translates to 200 ms */
+
+ /* PPS */
+ 07 01 00 00 00 00 01 11
+
+ 0A 01 00 00 00 00 80 11
+ 00 00 89 30 80 08 70 04
+ 38 00 10 02 1C 02 1C 02
+ 00 02 0E 00 20 01 84 00
+ 07 00 0C 06 67 06 5C 18
+ 00 10 F0 03 0C 20 00 06
+ 0B 0B 33 0E 1C 2A 38 46
+ 54 62 69 70 77 79 7B 7D
+ 7E 01 02 01 00 09 40 09
+ BE 19 FC 19 FA 19 F8 1A
+ 38 1A 78 1A B6 2A F6 2B
+ 34 2B 74 3B 74 6B F4 00
+ 00 00 00 00 00 00 00 00
+ 00 00 00 00 00 00 00 00
+ 00 00 00 00 00 00 00 00
+ 00 00 00 00 00 00 00 00
+ 00 00 00 00 00 00 00
+
+ 15 01 00 00 00 00 02 B0 AC
+
+ 15 01 00 00 00 00 02 26 02
+
+ /* Edge Round coordination loaded from Flash to SRAM */
+ 15 01 00 00 00 00 02 B0 AC /* MCS Access */
+
+ 39 01 00 00 00 00 07 F7
+ 01 49 0C FF 7F 04 /* Flash memory power on */
+
+ 39 01 00 00 00 00 06 CB 80 5C 07 03 28
+
+ 39 01 00 00 00 00 06 CC
+ 02 00 00 00 41 /* Flash memory type check */
+
+ 39 01 00 00 00 00 13 CD
+ 00 00 00 20 20 20 20 20
+ 24 24 28 28 28 20 20 20
+ 20 16 /* VSSEL band setting */
+
+ 15 01 00 00 00 00 02 EC 10 /* Data read enable from Flash */
+ 15 01 00 00 00 00 02 F3 AC /* Don't load OTP */
+
+
+ /* Edge Round Blur(Anti-Aliasing) Func. ON */
+ 15 01 00 00 00 00 02 B0 55 /* MLPIS Access */
+
+ 39 01 00 00 00 00 0C D7
+ 03 02 00 02 00 01 00 01
+ 69 01 00 /* Edge Round Control1 */
+
+ 39 01 00 00 00 00 0E D8
+ 00 02 00 02 00 02 00 01
+ 00 01 00 02 1C /* Edge Round Control2 */
+
+ 39 01 00 00 00 00 07 55
+ 34 70 CB 04 70 DB /* Edge Round function on */
+
+ 15 01 00 00 00 00 02 B0 CA /* Lock MCS Access */
+
+ 05 01 00 00 00 00 01 29
+ ];
+ };
+ };
+};
+
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-evt1.dts b/arch/arm64/boot/dts/google/sdm845-b1-evt1.dts
new file mode 100644
index 0000000..d80e7cf
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-evt1.dts
@@ -0,0 +1,49 @@
+/* Copyright (c) 2017 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-b1-common-v2.1.dtsi"
+#include "sdm845-b1c1-proto-evt-audio.dtsi"
+#include "sdm845-b1c1-audio-cal.dtsi"
+#include "sdm845-b1c1-thermal-v2.dtsi"
+#include "sdm845-b1-evt1-panel.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 B1 EVT v1.0";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00020A05 0>;
+};
+
+&p9221 {
+ status = "disabled";
+};
+
+&vd6281 {
+ hw-version = <20>; /* cut2.0 */
+};
+
+&nq {
+ ese {
+ loader_scripts_path =
+ "/vendor/etc/ese/loaderservice_updater_eng_";
+ };
+};
+
+&pmi8998_pdphy {
+ /delete-property/ goog,wlc-supported;
+};
+
+&google_charger {
+ /delete-property/ google,wlc-power-supply;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-evt2.1.dts b/arch/arm64/boot/dts/google/sdm845-b1-evt2.1.dts
new file mode 100644
index 0000000..deabd95
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-evt2.1.dts
@@ -0,0 +1,53 @@
+/* Copyright (c) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-b1-common-v2.1.dtsi"
+#include "sdm845-patherm.dtsi"
+#include "sdm845-usbctherm.dtsi"
+#include "sdm845-b1c1-usb-v2.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 B1 EVT v2.1";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00021005 0>;
+};
+
+/* We need override the display on sequence for EVT2 devices. */
+&dsi_sw43408_dsc_fhd_cmd {
+ qcom,mdss-dsi-display-timings {
+ timing@0 {
+ qcom,mdss-dsi-on-command = [
+ 15 01 00 00 00 00 02 26 02
+ 15 01 00 00 00 00 02 35 00
+ 39 01 00 00 00 00 03 53 0C 30
+ 39 01 00 00 00 00 07 55 34 61 CB 04 70 DB
+ 05 01 00 00 05 00 01 11
+ 07 01 00 00 00 00 01 11
+ 15 01 00 00 00 00 02 B0 AC
+ 39 01 00 00 00 00 13 CD 00 00 00 19 19 19
+ 19 19 19 19 19 19 19 19 19 19 16 16
+ 39 01 00 00 82 00 06 CB 80 5C 07 03 28
+ 05 01 00 00 32 00 01 29
+ ];
+ };
+ };
+};
+
+&nq {
+ ese {
+ loader_scripts_path =
+ "/vendor/etc/ese/loaderservice_updater_eng_";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-evt2.dts b/arch/arm64/boot/dts/google/sdm845-b1-evt2.dts
new file mode 100644
index 0000000..95285b6
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-evt2.dts
@@ -0,0 +1,37 @@
+/* Copyright (c) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-b1-common-v2.1.dtsi"
+#include "sdm845-b1c1-audio-cal.dtsi"
+#include "sdm845-patherm.dtsi"
+#include "sdm845-b1c1-thermal-v2.dtsi"
+#include "sdm845-b1c1-usb-v2.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 B1 EVT v2.0";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00020F05 0>;
+};
+
+&p9221 {
+ idt,gpio_slct = <0>;
+};
+
+&nq {
+ ese {
+ loader_scripts_path =
+ "/vendor/etc/ese/loaderservice_updater_eng_";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-proto-v1.1.dts b/arch/arm64/boot/dts/google/sdm845-b1-proto-v1.1.dts
new file mode 100644
index 0000000..9fd2090
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-proto-v1.1.dts
@@ -0,0 +1,421 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+
+/dts-v1/;
+
+#include "sdm845-b1-common-v2.dtsi"
+#include "sdm845-b1c1-proto-evt-audio.dtsi"
+#include "sdm845-b1c1-audio-cal.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 B1 proto v1.1 Board";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00020405 0>;
+};
+
+/* Workaround: Disable front normal camera, which has some MIPI issue. */
+&sensor_front {
+ status = "disabled";
+};
+
+/* B1 P1 has easel spi connected to se11 */
+&aliases {
+ spi10 = "/soc/spi@a8c000";
+};
+&qupv3_se0_spi {
+ status = "disabled";
+};
+&qupv3_se11_spi {
+ status = "okay";
+ mnh-spi@0 {
+ compatible = "intel,mnh-spi";
+ reg = <0>;
+ spi-max-frequency = <25000000>;
+ spi-cpol;
+ spi-cpha;
+ };
+ ese@0 {
+ status = "disabled";
+ };
+};
+
+/* Easel is required for camera */
+&sensor_front {
+ uses-easel;
+};
+&sensor_front_aux {
+ uses-easel;
+};
+&sensor_rear {
+ uses-easel;
+};
+
+/* We need override the display on sequence for P1.1 devices. */
+&dsi_sw43408_dsc_fhd_cmd {
+ qcom,mdss-dsi-bl-min-level = <0xf0>;
+ qcom,mdss-dsi-bl-max-level = <0x332>;
+ qcom,mdss-dsi-display-timings {
+ timing@0 {
+ qcom,mdss-dsi-on-command = [
+ 07 01 00 00 00 00 01 11
+ 15 01 00 00 00 00 02 B0 AC
+ 39 01 00 00 00 00 05 2A 00 00 04 37
+ 39 01 00 00 00 00 05 2B 00 00 08 6F
+ 39 01 00 00 00 00 0E C8
+ 0B 0B 0B 0B 0B 00 02 01
+ 0B 07 0A 09 08
+ 39 01 00 00 00 00 0E C9
+ 0B 0B 0B 0B 0B 00 02 01
+ 0B 07 0A 09 08
+ // Duty: 6VST
+ 39 01 00 00 00 00 08 E5
+ 00 3E 00 3E 00 3E 10
+ 39 01 00 00 00 00 3D E6
+ 20 00 31 20 00 31 20 00
+ 31 20 00 31 20 00 31 20
+ 00 31 10 BF 72 10 BF 72
+ 01 E2 4E 01 E2 4E 20 00
+ 31 20 00 31 20 00 31 20
+ 00 31 20 00 31 20 00 31
+ 10 BF 72 10 BF 72 01 E2
+ 4E 01 E2 4E
+ 39 01 00 00 00 00 1F E7
+ 20 00 31 20 00 31 02 1A
+ 16 20 00 31 20 88 A8 20
+ 00 31 20 00 31 02 1A 16
+ 20 00 31 20 88 A8
+ 39 01 00 00 00 00 0D B4
+ 2A 02 25 22 41 41 41 0A
+ 10 50 11 00
+ // 1VST?
+ 39 01 00 00 00 00 13 B5
+ 75 60 2D 5D 80 00 0A 0B
+ 00 05 0B 00 80 0D 0E 40
+ 00 0C
+ //OSC 48.9MHz 60 Hz
+ 39 01 00 00 00 00 0C B6
+ 60 30 00 50 20 00 50 20
+ 40 14 14
+ 39 01 00 00 00 00 06 B7 00 05 00 04 37
+ 39 01 00 00 00 00 10 B8
+ 2C 01 2C 01 FF 24 01 2C
+ 01 FF 33 01 1F 01 1F
+ 39 01 00 00 00 00 13 BF
+ 30 FF 11 33 00 00 00 00
+ 00 00 00 00 00 00 33 01
+ 00 20
+ 39 01 00 00 00 00 04 C0 02 09 0F
+ 39 01 00 00 00 00 19 C2
+ 70 8A 0A 0A 44 8C 4C 44
+ 8C 4C 0A ED AD 54 24 00
+ 00 64 24 CB 64 97 17 64
+ 39 01 00 00 00 00 12 C3
+ 10 15 1F B9 AD 4D B0 50
+ B7 57 22 02 C8 20 83 0C
+ 07
+ 39 01 00 00 00 00 12 C4
+ 14 34 13 22 32 22 22 22
+ 24 24 30 22 22 33 22 43
+ 12
+ // Command-video switch setting
+ 39 01 00 00 00 00 06 C6 0F 03 03 00 00
+
+ 39 01 00 00 00 00 05 CA 08 99 18 05
+ 39 01 00 00 00 00 0F CB
+ 80 5C 07 03 28 54 4F 5A
+ 33 19 60 00 00 00
+ 39 01 00 00 00 00 13 CD
+ 00 00 00 20 20 20 20 20
+ 24 24 28 28 28 20 20 20
+ 20 16
+ 15 01 00 00 00 00 02 26 02
+ 39 01 00 00 00 00 19 D0
+ E5 20 20 20 20 42 00 00
+ 00 E5 42 00 00 00 08 00
+ C4 0D 0D 93 93 20 00 00
+ 39 01 00 00 00 00 2D D1
+ BF A3 47 0B D3 6A D3 73
+ 73 63 94 03 FF FF 7F AF
+ BF 7F FF 7F 05 FF 7F FF
+ 7F 58 01 AF 4F FF FF FF
+ F9 F3 ED FF E7 E1 DB D5
+ 3F CF C9 C3
+ 39 01 00 00 00 00 37 D2
+ 30 F4 3F 1F 4E 55 5F 8C
+ A1 30 DC 3F 3F 46 58 5C
+ 8E 9F BF 6A 3F 20 46 5A
+ 5C 87 9A 80 3B 7F D6 CE
+ BF B6 CC CD 80 07 3F D7
+ CC C2 BA CB CF 80 FF 3F
+ C2 C2 BF B4 CA CA
+ 39 01 00 00 00 00 37 D3
+ 30 F0 3F 32 4C 5D 61 92
+ A4 30 D8 3F 4C 4E 58 60
+ 93 A5 BF 60 3F 2C 50 5A
+ 5E 8D 9F 00 eF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 D4
+ 30 EC 3F 45 4C 5C 65 98
+ A8 30 D6 3F 54 4C 5A 64
+ 97 A8 BF 5B 3F 34 4C 5C
+ 62 92 A4 80 3B 7F D6 CE
+ BF B6 CE CD 80 07 3F D7
+ CC C2 BA CB CF 80 FF 3F
+ C2 C2 BF B4 CA CA
+ 39 01 00 00 00 00 37 D5
+ 30 E4 3F 42 60 50 68 99
+ A8 30 CC 3F 64 64 59 65
+ 99 AA BF 4C 3F 44 60 50
+ 65 94 A4 00 eF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 D6
+ 30 D4 3F 52 50 5E 6B 98
+ B2 30 C0 3F 72 58 5B 68
+ 9A B0 BF 39 3F 40 58 5C
+ 6A 97 AC 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 D7
+ 30 BC 3F 66 4A 60 6A 9E
+ B4 30 A9 3F 94 56 5C 69
+ 9E B2 BF 1B 3F 54 4C 66
+ 6A 97 AE 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 D8
+ 30 BE 3F 6B 58 5F 6D A1
+ B7 30 AB 3F C4 78 62 6B
+ A1 B6 BF 1E 3F 64 64 62
+ 6E 9D B0 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 D9
+ 30 71 3F 7C 7A 5C 6C A4
+ B8 30 63 3F CF C8 73 6C
+ A4 B9 3F BA 3F 7C 77 65
+ 70 A1 B6 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 DD
+ 30 83 3F 7C 7A AA 96 B2
+ C0 30 78 3F CF C8 73 BA
+ BE C8 3F D1 3F 7C 77 65
+ 9B B9 C1 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 DE
+ B0 0A 3F 7C 7A AA 96 B2
+ C0 30 F5 3F CF C8 73 BA
+ BE C8 BF 59 3F 7C 77 65
+ 9B B9 C1 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 DF
+ 30 A0 3F 2A 40 8D 92 BA
+ CA 30 8B 3E F3 7C 8E 91
+ BB CD 3F DC 3F B0 9A 92
+ 8C B7 C4 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 F3 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 E0
+ 30 A6 3F 7A 2D B0 BA D4
+ DE 30 93 3E 83 D5 C4 BC
+ D5 DF 3F DF 3F 10 E0 D0
+ C2 D6 E0 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 F3 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 E1
+ 30 D8 3F AB AE AF AB CA
+ DF 30 DF 3F BF B5 E6 C0
+ CE C5 BF 30 3F B7 AB BE
+ 8F B3 C7 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 F3 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 E2
+ B0 27 7F 41 6F 76 6F 95
+ 9D B0 00 7F 51 6E 76 70
+ 97 9E BF 9A 7F 62 6B 70
+ 66 8D 92 00 eF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 E3
+ 30 B3 3F 4A 6F 80 84 B3
+ C2 30 9B 3F 53 92 85 84
+ B5 C2 3F FF 3F 60 94 84
+ 80 AC BC 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 F3 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 1F E4
+ 00 EF 3F 7A 9D 6A 8F A8
+ BC 80 4D 3E 83 7E 74 83
+ AD A8 00 F3 3F 10 7C 80
+ 70 B2 BA 20 01 CC
+ // TE Enable: 0x39 0xCE 0x27 0x10 0x08 0x80
+ 39 01 00 00 00 00 05 CE 27 10 08 80
+ 15 01 00 00 00 00 02 35 00
+ 39 01 00 00 00 00 03 53 0C 30
+ 39 01 00 00 00 00 07 55
+ 04 70 DB 04 70 DB
+ 15 01 00 00 00 00 02 F3 AC
+ 15 01 00 00 00 00 02 B0 AC
+ // TODO salidoa: No delay between 11/29? or delay? 200 msec>?
+ 05 01 00 00 C8 00 01 11
+ // TODO salidoa: write protect twice? 15 01 00 00 00 00 02 B0 AC
+ // TODO salidoa: check delay after 29?
+ 05 01 00 00 C8 00 01 29
+ ];
+ };
+ };
+};
+
+/* B1 proto has curved screen edges, so it requires its own firmware tuning */
+&qupv3_se2_i2c {
+ st_fts@49 {
+ st,firmware_name = "ftm5_p1_fw.ftb";
+ };
+};
+
+#include "sdm845-b1c1-haptics-drv2624-overlay.dtsi"
+
+/* B1 P1.1 has the bottom speaker amp in inverted polarity */
+&cs35l36_codec1 {
+ cirrus,amp-pcm-inv;
+ cirrus,imon-pol-inv;
+ cirrus,vmon-pol-inv;
+};
+
+/* Disable wchg on I2C10 */
+&p9221 {
+ status = "disabled";
+};
+
+&qupv3_se7_spi {
+ status = "disable";
+};
+
+&qupv3_se8_spi {
+ status = "ok";
+};
+
+&aliases {
+ spi0 = "/soc/spi@a80000";
+};
+
+&cam_sensor_front2_active {
+ mux {
+ pins = "gpio9", "gpio93";
+ };
+ config {
+ pins = "gpio9", "gpio93";
+ };
+};
+
+&cam_sensor_front2_suspend {
+ mux {
+ pins = "gpio9", "gpio93";
+ };
+ config {
+ pins = "gpio9", "gpio93";
+ };
+};
+
+&eeprom_front {
+ pinctrl-0 = <&cam_sensor_mclk1_active
+ &cam_sensor_front_active>;
+ pinctrl-1 = <&cam_sensor_mclk1_suspend
+ &cam_sensor_front_suspend>;
+ gpios = <&tlmm 14 0>,
+ <&tlmm 21 0>,
+ <&tlmm 8 0>;
+ clocks = <&clock_camcc CAM_CC_MCLK1_CLK>;
+};
+
+&eeprom_front_aux {
+ gpios = <&tlmm 15 0>,
+ <&tlmm 9 0>,
+ <&tlmm 93 0>;
+};
+
+&sensor_rear {
+ csiphy-sd-index = <1>;
+};
+
+&sensor_front_aux {
+ gpios = <&tlmm 15 0>,
+ <&tlmm 9 0>,
+ <&tlmm 93 0>;
+};
+
+&sensor_front {
+ csiphy-sd-index = <0>;
+ pinctrl-0 = <&cam_sensor_mclk1_active
+ &cam_sensor_front_active>;
+ pinctrl-1 = <&cam_sensor_mclk1_suspend
+ &cam_sensor_front_suspend>;
+ gpios = <&tlmm 14 0>,
+ <&tlmm 21 0>,
+ <&tlmm 8 0>;
+ clocks = <&clock_camcc CAM_CC_MCLK1_CLK>;
+};
+
+&ctdl_ap_irq {
+ mux {
+ pins = "gpio96";
+ };
+
+ config {
+ pins = "gpio96";
+ };
+};
+
+&ap_ctdl_irq {
+ mux {
+ pins = "gpio94";
+ };
+
+ config {
+ pins = "gpio94";
+ };
+};
+
+&qupv3_se4_spi {
+ citadel@0 {
+ interrupts = <96 0x0>;
+ citadel,ctdl_ap_irq = <&tlmm 96 0x0>;
+ };
+};
+
+&thermal_zones {
+ fps-therm-adc {
+ status = "disabled";
+ };
+};
+
+&google_charger {
+ /delete-property/ google,wlc-power-supply;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-proto.dts b/arch/arm64/boot/dts/google/sdm845-b1-proto.dts
new file mode 100644
index 0000000..9fa53ec
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-proto.dts
@@ -0,0 +1,483 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+
+/dts-v1/;
+
+#include "sdm845-b1-common-v2.dtsi"
+#include "sdm845-b1c1-proto-evt-audio.dtsi"
+#include "sdm845-b1c1-audio-cal.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 B1 proto Board";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00020305 0>;
+};
+
+/* B1 P1 has easel spi connected to se11 */
+&aliases {
+ spi10 = "/soc/spi@a8c000";
+};
+&qupv3_se0_spi {
+ status = "disabled";
+};
+&qupv3_se11_spi {
+ status = "okay";
+ mnh-spi@0 {
+ compatible = "intel,mnh-spi";
+ reg = <0>;
+ spi-max-frequency = <25000000>;
+ spi-cpol;
+ spi-cpha;
+ };
+ ese@0 {
+ status = "disabled";
+ };
+};
+
+/* Easel is required for camera */
+&sensor_front {
+ uses-easel;
+};
+&sensor_front_aux {
+ uses-easel;
+};
+&sensor_rear {
+ uses-easel;
+};
+
+/*
+ * B1 P1 requires a SW workaround for easel because of a schematic error. This
+ * will be resolved in P1.1 schematic and is not needed there.
+ */
+&s2mpb04_gpio {
+ p1_workaround;
+};
+
+/*
+ * B1 P1 requires the speaker amp to be strictly in i2s (instead of tdm mode)
+ * This will be resolved in P1.1 schematic and is not needed there.
+ */
+&snd_934x {
+ qcom,model = "sdm845-tavil-snd-card";
+};
+
+/* path to speaker boosters */
+&dai_mi2s3 {
+ /* AP in SD0 (gpio60), AP out SD1 (gpio61); */
+ /* SD0; in; ("rx-lines" is playback path): mask=1 */
+ /* SD1; out ("tx-lines" is capture path): mask=2 */
+ pinctrl-names = "default", "sleep";
+ pinctrl-0 = <&quat_mi2s_mclk_active &quat_mi2s_active
+ &quat_mi2s_sd0_active &quat_mi2s_sd1_active>;
+ pinctrl-1 = <&quat_mi2s_mclk_sleep &quat_mi2s_sleep
+ &quat_mi2s_sd0_sleep &quat_mi2s_sd1_sleep>;
+};
+
+&dai_quat_tdm_rx_0 {
+ pinctrl-2 = <>;
+ pinctrl-3 = <>;
+};
+
+&dai_quat_tdm_tx_0 {
+ pinctrl-2 = <>;
+ pinctrl-3 = <>;
+};
+
+&mdss_mdp {
+ qcom,sde-vsync-source = <0>;
+};
+
+&sde_te_active {
+ mux {
+ pins = "gpio10";
+ };
+ config {
+ pins = "gpio10";
+ };
+};
+
+&sde_te_suspend {
+ mux {
+ pins = "gpio10";
+ };
+ config {
+ pins = "gpio10";
+ };
+};
+
+&nfc_enable_default {
+ mux {
+ /* 12: NFC ENABLE , 23: NFC_SE_PWR_REQ, 122:ESE WAKEUP INT */
+ pins = "gpio12", "gpio23", "gpio122";
+ };
+ config {
+ pins = "gpio12", "gpio23", "gpio122";
+ };
+};
+
+&nq {
+ status = "disabled";
+};
+
+#include "sdm845-b1c1-haptics-drv2624-overlay.dtsi"
+
+&qupv3_se7_spi {
+ status = "disable";
+};
+
+&qupv3_se8_spi {
+ status = "ok";
+};
+
+&aliases {
+ spi0 = "/soc/spi@a80000";
+};
+
+/* Disable wchg on I2C10 */
+&p9221 {
+ status = "disabled";
+};
+
+&cam_sensor_front2_active {
+ mux {
+ pins = "gpio9", "gpio93";
+ };
+ config {
+ pins = "gpio9", "gpio93";
+ };
+};
+
+&cam_sensor_front2_suspend {
+ mux {
+ pins = "gpio9", "gpio93";
+ };
+ config {
+ pins = "gpio9", "gpio93";
+ };
+};
+
+&eeprom_front {
+ pinctrl-0 = <&cam_sensor_mclk1_active
+ &cam_sensor_front_active>;
+ pinctrl-1 = <&cam_sensor_mclk1_suspend
+ &cam_sensor_front_suspend>;
+ gpios = <&tlmm 14 0>,
+ <&tlmm 21 0>,
+ <&tlmm 8 0>;
+ clocks = <&clock_camcc CAM_CC_MCLK1_CLK>;
+};
+
+&eeprom_front_aux {
+ gpios = <&tlmm 15 0>,
+ <&tlmm 9 0>,
+ <&tlmm 93 0>;
+};
+
+&sensor_rear {
+ csiphy-sd-index = <1>;
+};
+
+&sensor_front_aux {
+ gpios = <&tlmm 15 0>,
+ <&tlmm 9 0>,
+ <&tlmm 93 0>;
+};
+
+&sensor_front {
+ csiphy-sd-index = <0>;
+ pinctrl-0 = <&cam_sensor_mclk1_active
+ &cam_sensor_front_active>;
+ pinctrl-1 = <&cam_sensor_mclk1_suspend
+ &cam_sensor_front_suspend>;
+ gpios = <&tlmm 14 0>,
+ <&tlmm 21 0>,
+ <&tlmm 8 0>;
+ clocks = <&clock_camcc CAM_CC_MCLK1_CLK>;
+};
+
+&ctdl_ap_irq {
+ mux {
+ pins = "gpio96";
+ };
+
+ config {
+ pins = "gpio96";
+ };
+};
+
+&ap_ctdl_irq {
+ mux {
+ pins = "gpio94";
+ };
+
+ config {
+ pins = "gpio94";
+ };
+};
+
+&qupv3_se4_spi {
+ citadel@0 {
+ interrupts = <96 0x0>;
+ citadel,ctdl_ap_irq = <&tlmm 96 0x0>;
+ };
+};
+
+/* We need override the display on sequence for P1 devices. */
+&dsi_sw43408_dsc_fhd_cmd {
+ qcom,mdss-dsi-bl-min-level = <0xf0>;
+ qcom,mdss-dsi-bl-max-level = <0x332>;
+ qcom,mdss-dsi-display-timings {
+ timing@0 {
+ qcom,mdss-dsi-on-command = [
+ 07 01 00 00 00 00 01 11
+ 15 01 00 00 00 00 02 B0 AC
+ 39 01 00 00 00 00 05 2A 00 00 04 37
+ 39 01 00 00 00 00 05 2B 00 00 08 6F
+ 39 01 00 00 00 00 0E C8
+ 0B 0B 0B 0B 0B 00 02 01
+ 0B 07 0A 09 08
+ 39 01 00 00 00 00 0E C9
+ 0B 0B 0B 0B 0B 00 02 01
+ 0B 07 0A 09 08
+ // Duty: 6VST
+ 39 01 00 00 00 00 08 E5
+ 00 3E 00 3E 00 3E 10
+ 39 01 00 00 00 00 3D E6
+ 20 00 31 20 00 31 20 00
+ 31 20 00 31 20 00 31 20
+ 00 31 10 BF 72 10 BF 72
+ 01 E2 4E 01 E2 4E 20 00
+ 31 20 00 31 20 00 31 20
+ 00 31 20 00 31 20 00 31
+ 10 BF 72 10 BF 72 01 E2
+ 4E 01 E2 4E
+ 39 01 00 00 00 00 1F E7
+ 20 00 31 20 00 31 02 1A
+ 16 20 00 31 20 88 A8 20
+ 00 31 20 00 31 02 1A 16
+ 20 00 31 20 88 A8
+ 39 01 00 00 00 00 0D B4
+ 2A 02 25 22 41 41 41 0A
+ 10 50 11 00
+ // 1VST?
+ 39 01 00 00 00 00 13 B5
+ 75 60 2D 5D 80 00 0A 0B
+ 00 05 0B 00 80 0D 0E 40
+ 00 0C
+ //OSC 48.9MHz 60 Hz
+ 39 01 00 00 00 00 0C B6
+ 60 30 00 50 20 00 50 20
+ 40 14 14
+ 39 01 00 00 00 00 06 B7 00 05 00 04 37
+ 39 01 00 00 00 00 10 B8
+ 2C 01 2C 01 FF 24 01 2C
+ 01 FF 33 01 1F 01 1F
+ 39 01 00 00 00 00 13 BF
+ 30 FF 11 33 00 00 00 00
+ 00 00 00 00 00 00 33 01
+ 00 20
+ 39 01 00 00 00 00 04 C0 02 09 0F
+ 39 01 00 00 00 00 19 C2
+ 70 8A 0A 0A 44 8C 4C 44
+ 8C 4C 0A ED AD 54 24 00
+ 00 64 24 CB 64 97 17 64
+ 39 01 00 00 00 00 12 C3
+ 10 15 1F B9 AD 4D B0 50
+ B7 57 22 02 C8 20 83 0C
+ 07
+ 39 01 00 00 00 00 12 C4
+ 14 34 13 22 32 22 22 22
+ 24 24 30 22 22 33 22 43
+ 12
+ // Command-video switch setting
+ 39 01 00 00 00 00 06 C6 0F 03 03 00 00
+
+ 39 01 00 00 00 00 05 CA 08 99 18 05
+ 39 01 00 00 00 00 0F CB
+ 80 5C 07 03 28 54 4F 5A
+ 33 19 60 00 00 00
+ 39 01 00 00 00 00 13 CD
+ 00 00 00 20 20 20 20 20
+ 24 24 28 28 28 20 20 20
+ 20 16
+ 15 01 00 00 00 00 02 26 02
+ 39 01 00 00 00 00 19 D0
+ E5 20 20 20 20 42 00 00
+ 00 E5 42 00 00 00 08 00
+ C4 0D 0D 93 93 20 00 00
+ 39 01 00 00 00 00 2D D1
+ BF A3 47 0B D3 6A D3 73
+ 73 63 94 03 FF FF 7F AF
+ BF 7F FF 7F 05 FF 7F FF
+ 7F 58 01 AF 4F FF FF FF
+ F9 F3 ED FF E7 E1 DB D5
+ 3F CF C9 C3
+ 39 01 00 00 00 00 37 D2
+ 30 F4 3F 1F 4E 55 5F 8C
+ A1 30 DC 3F 3F 46 58 5C
+ 8E 9F BF 6A 3F 20 46 5A
+ 5C 87 9A 80 3B 7F D6 CE
+ BF B6 CC CD 80 07 3F D7
+ CC C2 BA CB CF 80 FF 3F
+ C2 C2 BF B4 CA CA
+ 39 01 00 00 00 00 37 D3
+ 30 F0 3F 32 4C 5D 61 92
+ A4 30 D8 3F 4C 4E 58 60
+ 93 A5 BF 60 3F 2C 50 5A
+ 5E 8D 9F 00 eF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 D4
+ 30 EC 3F 45 4C 5C 65 98
+ A8 30 D6 3F 54 4C 5A 64
+ 97 A8 BF 5B 3F 34 4C 5C
+ 62 92 A4 80 3B 7F D6 CE
+ BF B6 CE CD 80 07 3F D7
+ CC C2 BA CB CF 80 FF 3F
+ C2 C2 BF B4 CA CA
+ 39 01 00 00 00 00 37 D5
+ 30 E4 3F 42 60 50 68 99
+ A8 30 CC 3F 64 64 59 65
+ 99 AA BF 4C 3F 44 60 50
+ 65 94 A4 00 eF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 D6
+ 30 D4 3F 52 50 5E 6B 98
+ B2 30 C0 3F 72 58 5B 68
+ 9A B0 BF 39 3F 40 58 5C
+ 6A 97 AC 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 D7
+ 30 BC 3F 66 4A 60 6A 9E
+ B4 30 A9 3F 94 56 5C 69
+ 9E B2 BF 1B 3F 54 4C 66
+ 6A 97 AE 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 D8
+ 30 BE 3F 6B 58 5F 6D A1
+ B7 30 AB 3F C4 78 62 6B
+ A1 B6 BF 1E 3F 64 64 62
+ 6E 9D B0 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 D9
+ 30 71 3F 7C 7A 5C 6C A4
+ B8 30 63 3F CF C8 73 6C
+ A4 B9 3F BA 3F 7C 77 65
+ 70 A1 B6 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 DD
+ 30 83 3F 7C 7A AA 96 B2
+ C0 30 78 3F CF C8 73 BA
+ BE C8 3F D1 3F 7C 77 65
+ 9B B9 C1 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 DE
+ B0 0A 3F 7C 7A AA 96 B2
+ C0 30 F5 3F CF C8 73 BA
+ BE C8 BF 59 3F 7C 77 65
+ 9B B9 C1 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 DF
+ 30 A0 3F 2A 40 8D 92 BA
+ CA 30 8B 3E F3 7C 8E 91
+ BB CD 3F DC 3F B0 9A 92
+ 8C B7 C4 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 F3 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 E0
+ 30 A6 3F 7A 2D B0 BA D4
+ DE 30 93 3E 83 D5 C4 BC
+ D5 DF 3F DF 3F 10 E0 D0
+ C2 D6 E0 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 F3 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 E1
+ 30 D8 3F AB AE AF AB CA
+ DF 30 DF 3F BF B5 E6 C0
+ CE C5 BF 30 3F B7 AB BE
+ 8F B3 C7 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 F3 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 E2
+ B0 27 7F 41 6F 76 6F 95
+ 9D B0 00 7F 51 6E 76 70
+ 97 9E BF 9A 7F 62 6B 70
+ 66 8D 92 00 eF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 FF 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 37 E3
+ 30 B3 3F 4A 6F 80 84 B3
+ C2 30 9B 3F 53 92 85 84
+ B5 C2 3F FF 3F 60 94 84
+ 80 AC BC 00 EF 3F 7A 9D
+ 6A 8F A8 BC 80 4D 3E 83
+ 7E 74 83 AD A8 00 F3 3F
+ 10 7C 80 70 B2 BA
+ 39 01 00 00 00 00 1F E4
+ 00 EF 3F 7A 9D 6A 8F A8
+ BC 80 4D 3E 83 7E 74 83
+ AD A8 00 F3 3F 10 7C 80
+ 70 B2 BA 20 01 CC
+ // TE Enable: 0x39 0xCE 0x27 0x10 0x08 0x80
+ 39 01 00 00 00 00 05 CE 27 10 08 80
+ 15 01 00 00 00 00 02 35 00
+ 39 01 00 00 00 00 03 53 0C 30
+ 39 01 00 00 00 00 07 55
+ 04 70 DB 04 70 DB
+ 15 01 00 00 00 00 02 F3 AC
+ 15 01 00 00 00 00 02 B0 AC
+ // TODO salidoa: No delay between 11/29? or delay? 200 msec>?
+ 05 01 00 00 C8 00 01 11
+ // TODO salidoa: write protect twice? 15 01 00 00 00 00 02 B0 AC
+ // TODO salidoa: check delay after 29?
+ 05 01 00 00 C8 00 01 29
+ ];
+ };
+ };
+};
+
+/* B1 proto has curved screen edges, so it requires its own firmware tuning */
+&qupv3_se2_i2c {
+ st_fts@49 {
+ st,firmware_name = "ftm5_p1_fw.ftb";
+ };
+};
+
+&thermal_zones {
+ fps-therm-adc {
+ status = "disabled";
+ };
+};
+
+&google_charger {
+ /delete-property/ google,wlc-power-supply;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-touch.dtsi b/arch/arm64/boot/dts/google/sdm845-b1-touch.dtsi
new file mode 100644
index 0000000..04ef436
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-touch.dtsi
@@ -0,0 +1,64 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+/* Setting 3.3V per FTM5 porting guide
+ * There is a concern that PMI8998 VBOB is only 3.312 and is the
+ * reference for L19, but it is matching the Taimen.
+ */
+&pm8998_l19 {
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ qcom,init-voltage = <3300000>;
+ status = "okay";
+};
+
+&pm8998_l14 {
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <1800000>;
+ qcom,init-voltage = <1800000>;
+ qcom,init-mode = <RPMH_REGULATOR_MODE_HPM>;
+ status = "okay";
+};
+
+&rpmh_regulator_ldoa14 {
+ qcom,supported-modes = <RPMH_REGULATOR_MODE_HPM>;
+};
+
+/* AP Touch I2C SDA is gpio27, SCL is gpio28. (QUP2)
+ * Reset gpio99, int gpio125.
+ */
+
+&qupv3_se2_i2c {
+ status = "ok";
+ qcom,clk-freq-out = <1000000>;
+
+ st_fts@49 {
+ status = "ok";
+ compatible = "st,fts";
+ reg = <0x49>;
+ interrupt-parent = <&tlmm>;
+ interrupts = <125 IRQ_TYPE_LEVEL_LOW>;
+ vdd-supply = <&pm8998_l14>; /* 1.8V */
+ avdd-supply = <&pm8998_l19>; /* 3.3V */
+ pinctrl-names = "pmx_ts_active", "pmx_ts_suspend";
+ pinctrl-0 = <&ts_int_active &ts_reset_active>;
+ pinctrl-1 = <&ts_int_suspend>;
+
+ st,irq-gpio = <&tlmm 125 0>;
+ st,reset-gpio = <&tlmm 99 0>;
+ st,switch_gpio = <&tlmm 136 0>;
+ st,max-coords = <1079 2159>;
+ st,regulator_dvdd = "vdd";
+ st,regulator_avdd = "avdd";
+ /* todo st,firmware_name = "ftm4_fw.ftb"; */
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1-usb-v3.dtsi b/arch/arm64/boot/dts/google/sdm845-b1-usb-v3.dtsi
new file mode 100644
index 0000000..643ca181
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1-usb-v3.dtsi
@@ -0,0 +1,24 @@
+/* Overrides phy tuning parameters for B1 DVT, B1 DVT1.1 or later*/
+
+&qusb_phy0 {
+ qcom,qusb-phy-init-seq =
+ /* <value reg_offset> */
+ <0x23 0x210 /* PWR_CTRL1 */
+ 0x03 0x04 /* PLL_ANALOG_CONTROLS_TWO */
+ 0x7c 0x18c /* PLL_CLOCK_INVERTERS */
+ 0x80 0x2c /* PLL_CMODE */
+ 0x0a 0x184 /* PLL_LOCK_DELAY */
+ 0x19 0xb4 /* PLL_DIGITAL_TIMERS_TWO */
+ 0x40 0x194 /* PLL_BIAS_CONTROL_1 */
+ 0x20 0x198 /* PLL_BIAS_CONTROL_2 */
+ 0x21 0x214 /* PWR_CTRL2 */
+ 0x00 0x220 /* IMP_CTRL1 */
+ 0x58 0x224 /* IMP_CTRL2 */
+ 0x37 0x240 /* TUNE1 */
+ 0x29 0x244 /* TUNE2 */
+ 0xca 0x248 /* TUNE3 */
+ 0x04 0x24c /* TUNE4 */
+ 0x03 0x250 /* TUNE5 */
+ 0x00 0x23c /* CHG_CTRL2 */
+ 0x22 0x210>; /* PWR_CTRL1 */
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-audio-cal.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-audio-cal.dtsi
new file mode 100644
index 0000000..f0b330d
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-audio-cal.dtsi
@@ -0,0 +1,5 @@
+&soc {
+ msm_cirrus_playback {
+ cirrus,swap_calibration;
+ };
+};
\ No newline at end of file
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-audio.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-audio.dtsi
new file mode 100644
index 0000000..c1bffb7
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-audio.dtsi
@@ -0,0 +1,247 @@
+/* AUDIO: RESET: GPIO64, INTR1: GPIO54, INTR2: GPIO53, EXT_CLK: PM845.68 LN_BB_CLK3, SLIMBUS_CLK: GPIO70, SLIMBUS_DATA[0..1]: GPIO71,GPIO72 */
+
+#include <dt-bindings/clock/qcom,audio-ext-clk.h>
+#include "sdm845-audio-overlay.dtsi"
+
+&aliases {
+ spi0 = "/soc/spi@89c000";
+};
+
+&pm8998_gpios {
+ gpio@cc00 { /* GPIO 13 */
+ status = "disabled";
+ };
+};
+
+&tlmm {
+ cs35l36_pins {
+ cs35l36_reset_top: cs35l36_reset_top {
+ mux {
+ pins = "gpio75";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio75";
+ drive-strength = <2>;
+ bias-disable;
+ };
+ };
+
+ cs35l36_irq_top: cs35l36_irq_top {
+ mux {
+ pins = "gpio40";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio40";
+ drive-strength = <2>;
+ bias-pull-up;
+ };
+ };
+ cs35l36_reset_bottom: cs35l36_reset_bottom {
+ mux {
+ pins = "gpio112";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio112";
+ drive-strength = <2>;
+ bias-disable;
+ };
+ };
+
+ cs35l36_irq_bottom: cs35l36_irq_bottom {
+ mux {
+ pins = "gpio115";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio115";
+ drive-strength = <2>;
+ bias-pull-up;
+ };
+ };
+ };
+};
+
+&clock_audio_lnbb {
+ clocks = <&clock_rpmh RPMH_LN_BB_CLK3>;
+};
+
+&wcd_usbc_analog_en1_gpio {
+ status = "disabled";
+};
+
+&snd_934x {
+ qcom,model = "sdm845-tavil-b1-snd-card";
+ qcom,audio-routing =
+ "AIF4 VI", "MCLK",
+ "RX_BIAS", "MCLK",
+ "MADINPUT", "MCLK",
+ "AMIC1", "Flicker Mic",
+ "DMIC0", "MIC BIAS1",
+ "MIC BIAS1", "Digital Mic0",
+ "DMIC3", "MIC BIAS3",
+ "MIC BIAS3", "Digital Mic1",
+ "DMIC4", "MIC BIAS4",
+ "MIC BIAS4", "Digital Mic2";
+
+ qcom,msm-mbhc-hphl-swh = <0>;
+ qcom,msm-mbhc-gnd-swh = <0>;
+ qcom,wsa-max-devs = <0>;
+ qcom,msm-mbhc-hs-mic-max-threshold-mv = <1700>;
+ qcom,msm-mbhc-hs-mic-min-threshold-mv = <50>;
+ qcom,tavil-mclk-clk-freq = <9600000>;
+
+ asoc-codec = <&stub_codec>, <&ext_disp_audio_codec>;
+ asoc-codec-names = "msm-stub-codec.1",
+ "msm-ext-disp-audio-codec-rx";
+
+ pinctrl-names = "quat-mi2s-active", "quat-mi2s-sleep",
+ "quat-tdm-active", "quat-tdm-sleep";
+ pinctrl-0 = <&quat_mi2s_active &quat_mi2s_sd0_active
+ &quat_mi2s_sd1_active>;
+ pinctrl-1 = <&quat_mi2s_sleep &quat_mi2s_sd0_sleep
+ &quat_mi2s_sd1_sleep>;
+ pinctrl-2 = <&quat_tdm_active &quat_tdm_dout_active
+ &quat_tdm_din_active>;
+ pinctrl-3 = <&quat_tdm_sleep &quat_tdm_dout_sleep
+ &quat_tdm_din_sleep>;
+};
+
+&msm_dai_tdm_quat_rx {
+ qcom,msm-cpudai-tdm-clk-rate = <12288000>;
+ qcom,msm-cpudai-tdm-sync-mode = <0>;
+ qcom,msm-cpudai-tdm-invert-sync = <0>;
+};
+&dai_quat_tdm_rx_0 {
+ qcom,msm-cpudai-tdm-header-start-offset = <0>;
+ qcom,msm-cpudai-tdm-header-width = <2>;
+ qcom,msm-cpudai-tdm-header-num-frame-repeat = <8>;
+};
+
+&msm_dai_tdm_quat_tx {
+ qcom,msm-cpudai-tdm-clk-rate = <12288000>;
+ qcom,msm-cpudai-tdm-sync-mode = <0>;
+ qcom,msm-cpudai-tdm-invert-sync = <0>;
+};
+
+&soc {
+ msm_cirrus_playback {
+ compatible = "cirrus,msm-cirrus-playback";
+ usecase-names = "Playback";
+
+ /* the resistance is 7.9999 ohm */
+ cirrus,top-speaker-impedance = <11189>;
+
+ /* the percentage of tolerance is 10 */
+ cirrus,top-speaker-tolerance = <10>;
+
+ /* the resistance is 7.6796 ohm */
+ cirrus,top-speaker-mean = <10741>;
+
+ /* the resistance is 6.7994 ohm */
+ cirrus,bottom-speaker-impedance = <9510>;
+
+ /* the percentage of tolerance is 10 */
+ cirrus,bottom-speaker-tolerance = <10>;
+
+ /* the resistance is 6.6593 ohm */
+ cirrus,bottom-speaker-mean = <9314>;
+ };
+};
+
+&routing {
+ qcom,msm-pcm-config = "b1";
+};
+
+/* path to speaker boosters */
+&dai_quat_tdm_rx_0 {
+ pinctrl-names = "default", "sleep";
+ pinctrl-2 = <&quat_tdm_active &quat_tdm_dout_active
+ &quat_tdm_din_active>;
+ pinctrl-3 = <&quat_tdm_sleep &quat_tdm_dout_sleep
+ &quat_tdm_din_sleep>;
+};
+
+&dai_quat_tdm_tx_0 {
+ pinctrl-names = "default", "sleep";
+ pinctrl-2 = <&quat_tdm_active &quat_tdm_dout_active
+ &quat_tdm_din_active>;
+ pinctrl-3 = <&quat_tdm_sleep &quat_tdm_dout_sleep
+ &quat_tdm_din_sleep>;
+};
+
+/* speaker boosters */
+&qupv3_se12_i2c {
+ status = "ok";
+ cs35l36_codec1: cs35l36@40 {
+ /* bottom speaker - portrait right */
+ compatible = "cirrus,cs35l36";
+ reg = <0x40>;
+ reset-gpios = <&tlmm 112 0>;
+ irq-gpio = <&tlmm 115 0>;
+ VA-supply = <&pm8998_s4>;
+ interrupt-parent = <&tlmm>;
+ interrupts = <115 IRQ_TYPE_LEVEL_LOW>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&cs35l36_reset_bottom &cs35l36_irq_bottom>;
+ cirrus,boost-ind-nanohenry = <1000>;
+ cirrus,boost-ctl-select = <0x01>;
+ cirrus,boost-ctl-millivolt = <10000>;
+ cirrus,boost-peak-milliamp = <4500>;
+ cirrus,amp-gain-zc;
+ cirrus,ldm-mode-select = <0x01>;
+ cirrus,pdm-ldm-exit;
+ cirrus,pdm-ldm-enter;
+ cirrus,weak-fet-delay = <0x04>;
+ cirrus,weak-fet-thld = <0x01>;
+ cirrus,temp-warn-threshold = <0x01>;
+ cirrus,multi-amp-mode;
+
+ cirrus,irq-config {
+ cirrus,irq-drive-select = <0x00>;
+ cirrus,irq-polarity = <0x00>;
+ cirrus,irq-gpio-select = <0x00>;
+ cirrus,irq-output-enable = <0x01>;
+ cirrus,irq-src-select = <0x01>;
+ };
+ };
+
+ cs35l36_codec2: cs35l36@41 {
+ /* top speaker - portrait left */
+ compatible = "cirrus,cs35l36";
+ reg = <0x41>;
+ reset-gpios = <&tlmm 75 0>;
+ irq-gpio = <&tlmm 40 0>;
+ VA-supply = <&pm8998_s4>;
+ interrupt-parent = <&tlmm>;
+ interrupts = <40 IRQ_TYPE_LEVEL_LOW>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&cs35l36_reset_top &cs35l36_irq_top>;
+ cirrus,boost-ind-nanohenry = <1000>;
+ cirrus,boost-ctl-select = <0x01>;
+ cirrus,boost-ctl-millivolt = <10000>;
+ cirrus,boost-peak-milliamp = <4500>;
+ cirrus,amp-gain-zc;
+ cirrus,ldm-mode-select = <0x01>;
+ cirrus,pdm-ldm-exit;
+ cirrus,pdm-ldm-enter;
+ cirrus,weak-fet-delay = <0x04>;
+ cirrus,weak-fet-thld = <0x01>;
+ cirrus,temp-warn-threshold = <0x01>;
+ cirrus,multi-amp-mode;
+
+ cirrus,irq-config {
+ cirrus,irq-drive-select = <0x00>;
+ cirrus,irq-polarity = <0x00>;
+ cirrus,irq-gpio-select = <0x00>;
+ cirrus,irq-output-enable = <0x01>;
+ cirrus,irq-src-select = <0x01>;
+ };
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-battery.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-battery.dtsi
new file mode 100644
index 0000000..65a435d
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-battery.dtsi
@@ -0,0 +1,120 @@
+
+&qupv3_se10_i2c {
+#include "../qcom/smb1355.dtsi"
+};
+
+&smb1355_charger_0 {
+ status = "ok";
+};
+
+&smb1355_charger_1 {
+ status = "ok";
+};
+
+&pmi8998_pdphy {
+ vbus-supply = <&smb2_vbus>;
+ ext-vbus-supply = <&ext_5v_boost>;
+};
+
+&pmi8998_charger {
+ google,boost_to_5_1V;
+ qcom,dc-icl-ua = <1100000>;
+ google,batt_psy_is_bms;
+ google,usb-port-tz-name = "usbc-therm-adc";
+ google,wlc-thermal-mitigation = <1100000 500000 250000 110000>;
+ google,low-batt-threshold = <2>;
+ qcom,hvdcp-disable;
+};
+
+&pmi8998_fg {
+ qcom,battery-thermal-coefficients = [d8 50 ff];
+ qcom,cl-max-decrement = <5>;
+ qcom,hold-soc-while-full;
+ qcom,linearize-soc;
+ qcom,fg-cutoff-voltage = <3300>;
+ qcom,fg-empty-voltage = <2600>;
+ qcom,fg-jeita-hyst-temp = <1>;
+ qcom,fg-jeita-thresholds = <0 0 55 55>;
+ qcom,fg-force-load-profile;
+ google,fg-soc-irq-disable;
+};
+
+&tlmm {
+ fg {
+ fg_int_default: fg_int_default {
+ mux {
+ /* GPIO 62 Maxim FG status change Interrupt */
+ pins = "gpio62";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio62";
+ drive-strength = <2>; /* 2 MA */
+ bias-pull-up;
+ };
+ };
+
+ };
+};
+
+/* smart battery: maxim fuel gauge */
+&qupv3_se12_i2c {
+ status = "ok";
+ max1720x_fg: max1720x@36 {
+ compatible = "maxim,max1720x";
+ reg-names = "m5db", "nvram";
+ reg = <0x36 0x0b>;
+ interrupt-parent = <&tlmm>;
+ interrupt-names = "fg_irq";
+ interrupts = <62 IRQ_TYPE_LEVEL_LOW>;
+ fg,irq-gpio = <&tlmm 62 0x00>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&fg_int_default>;
+
+ io-channels = <&pmi8998_rradc 0>;
+ io-channel-names = "rradc_batt_id";
+ maxim,batt-id-range-pct = <15>;
+
+ maxim,nconvgcfg-temp-hysteresis = <10>;
+ maxim,nconvgcfg-temp-limits = /bits/ 16
+ <(-70) (-45) (-20) (-3) 14 30 55 80
+ 96 113 130 155 180 230>;
+ maxim,nconvgcfg-values = /bits/ 16
+ <0xe7c0 0xd7c0 0xc7c0 0xb7c0 0xa740 0x9740 0x86c0 0x7640
+ 0x65c0 0x54c0 0x4440 0x3340 0x2240 0x1140>;
+ };
+};
+
+&soc {
+ google_charger: google,charger {
+ compatible = "google,charger";
+ google,chg-power-supply = "battery";
+ google,bat-power-supply = "maxfg";
+ google,fv-uv-margin-dpct = <1020>;
+ google,chg-update-interval = <30000>;
+ google,chg-temp-limits = <0 100 200 420 460 480 550>;
+ google,chg-cv-limits = <4200000 4300000 4400000>;
+ google,chg-cc-limits = <
+ 30 10 0
+ 50 30 30
+ 100 70 50
+ 80 50 50
+ 50 50 0
+ 30 0 0
+ >;
+ google,chg-cc-tolerance = <25>;
+ };
+
+ google_overheat_mitigation: google,overheat_mitigation {
+ compatible = "google,overheat_mitigation";
+ google,port-overheat-work-interval = <1000>;
+ google,accessory-monitoring-period = <150>;
+ /* Must maintain < 1/15 duty cycle */
+ google,polling-freq = <3>;
+ };
+};
+
+&bcl_sensor {
+ google,bat-power-supply = "maxfg";
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-citadel.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-citadel.dtsi
new file mode 100644
index 0000000..a22a0ec
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-citadel.dtsi
@@ -0,0 +1,69 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&tlmm {
+ citadel {
+ ctdl_ap_irq: ctdl_ap_irq {
+ mux {
+ pins = "gpio129";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio129";
+ drive-strength = <2>; /* 2 MA */
+ input-enable;
+ bias-disable;
+ };
+ };
+
+ ap_ctdl_irq: ap_ctdl_irq {
+ mux {
+ pins = "gpio135";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio135";
+ drive-strength = <2>; /* 2 MA */
+ bias-pull-down;
+ output-high;
+ };
+ };
+ };
+};
+
+&qupv3_se4_spi_pins {
+ citadel_pin_defaults: citadel_pin_defaults {
+ mux {
+ pins = "gpio89", "gpio90", "gpio91", "gpio92";
+ function = "qup4";
+ };
+ };
+};
+
+&qupv3_se4_spi {
+ status = "okay";
+ pinctrl-0 = <&citadel_pin_defaults>;
+ pinctrl-1 = <&citadel_pin_defaults>;
+ citadel@0 {
+ compatible = "google,citadel";
+ interrupt-parent = <&tlmm>;
+ interrupts = <129 0x0>;
+ citadel,ctdl_ap_irq = <&tlmm 129 0x0>;
+ citadel,ctdl_rst = <&pmi8998_gpios 12 GPIO_ACTIVE_HIGH>;
+ pinctrl-names = "ap_ctdl_irq";
+ pinctrl-0 = <&ap_ctdl_irq>;
+ reg = <0>;
+ spi-max-frequency = <1200000>;
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-common.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-common.dtsi
new file mode 100644
index 0000000..abe26e2
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-common.dtsi
@@ -0,0 +1,232 @@
+/* Copyright (c) 2016-2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+/* Common B1C1 includes go here (i.e. not display) */
+/* TODO: Split this file into bluetooth/keys/display/memory/power/thermal */
+
+/plugin/;
+
+#include <dt-bindings/clock/qcom,gcc-sdm845.h>
+#include <dt-bindings/clock/qcom,camcc-sdm845.h>
+#include <dt-bindings/clock/qcom,dispcc-sdm845.h>
+#include <dt-bindings/clock/qcom,rpmh.h>
+#include <dt-bindings/interrupt-controller/arm-gic.h>
+#include <dt-bindings/regulator/qcom,rpmh-regulator.h>
+#include <dt-bindings/clock/mdss-10nm-pll-clk.h>
+
+#include "../qcom/sdm845-pmic-overlay.dtsi"
+#include "../qcom/sdm845-pinctrl-overlay.dtsi"
+#include <dt-bindings/gpio/gpio.h>
+#include <dt-bindings/clock/qcom,camcc-sdm845.h>
+#include "sdm845-camera-sensor.dtsi"
+#include "sdm845-b1c1-display.dtsi"
+#include "sdm845-b1c1-pinctrl.dtsi"
+#include "sdm845-b1c1-haptics.dtsi"
+#include "sdm845-b1c1-nfc.dtsi"
+#include "sdm845-b1c1-citadel.dtsi"
+#include "sdm845-b1c1-easel.dtsi"
+#include "sdm845-b1c1-rainbow.dtsi"
+#include "sdm845-b1c1-fingerprint.dtsi"
+#include "sdm845-b1c1-regulator.dtsi"
+#include "sdm845-b1c1-wcharger.dtsi"
+#include "sdm845-b1c1-memory.dtsi"
+#include "sdm845-b1c1-thermal.dtsi"
+#include "sdm845-b1c1-usb.dtsi"
+#include "sdm845-b1c1-battery.dtsi"
+
+&vendor {
+ bluetooth: bt_wcn3990 {
+ compatible = "qca,wcn3990";
+ qca,bt-vdd-io-supply = <&pm8998_s3>;
+ qca,bt-vdd-xtal-supply = <&pm8998_s5>;
+ qca,bt-vdd-core-supply = <&pm8998_l7>;
+ qca,bt-vdd-pa-supply = <&pm8998_l17>;
+ qca,bt-vdd-ldo-supply = <&pm8998_l25>;
+
+ qca,bt-vdd-io-voltage-level = <1352000 1352000>;
+ qca,bt-vdd-xtal-voltage-level = <2040000 2040000>;
+ qca,bt-vdd-core-voltage-level = <1800000 1800000>;
+ qca,bt-vdd-pa-voltage-level = <1304000 1304000>;
+ qca,bt-vdd-ldo-voltage-level = <3312000 3312000>;
+
+ qca,bt-vdd-io-current-level = <1>; /* LPM/PFM */
+ qca,bt-vdd-xtal-current-level = <1>; /* LPM/PFM */
+ qca,bt-vdd-core-current-level = <1>; /* LPM/PFM */
+ qca,bt-vdd-pa-current-level = <1>; /* LPM/PFM */
+ qca,bt-vdd-ldo-current-level = <1>; /* LPM/PFM */
+ };
+};
+
+&soc {
+ gpio_keys {
+ compatible = "gpio-keys";
+ label = "gpio-keys";
+
+ pinctrl-names = "default";
+ pinctrl-0 = <&key_vol_up_default>;
+
+ vol_up {
+ label = "volume_up";
+ gpios = <&pm8998_gpios 6 GPIO_ACTIVE_LOW>;
+ linux,input-type = <1>;
+ linux,code = <115>;
+ gpio-key,wakeup;
+ debounce-interval = <15>;
+ linux,can-disable;
+ };
+ };
+
+ qcom,msm-imem@146bf000 {
+ restart_info@a94 {
+ compatible = "msm-imem-restart_info";
+ reg = <0xa94 100>;
+ };
+ };
+
+ modemsmem {
+ compatible = "modemsmem";
+ };
+
+ bootloader_log {
+ compatible = "qcom,bldr_log";
+ reg = <0xA1800000 0x5C00>, <0xA1806000 0x5C00>;
+ reg-names = "bl_log", "bl_old_log";
+ memory-region = <&debug_info>;
+ };
+
+ bootloader_kinfo {
+ compatible = "bootloader_kinfo";
+ reg = <0xA1805C00 0x3C0>;
+ reg-names = "kinfo";
+ memory-region = <&debug_info>;
+ };
+
+ ramoops {
+ compatible = "ramoops";
+ memory-region = <&ramoops_mem>;
+ alt-memory-region = <&alt_ramoops_mem>;
+ record-size = <0x20000>;
+ console-size = <0x100000>;
+ pmsg-size = <0x80000>;
+ };
+
+ access_ramoops@0 {
+ compatible = "access_ramoops";
+ label = "metadata";
+ memory-region = <&ramoops_meta_mem>;
+ };
+
+ access_ramoops@1 {
+ compatible = "access_ramoops";
+ label = "ramoops";
+ memory-region = <&alt_ramoops_mem>;
+ };
+
+ keydebug {
+ compatible = "keydebug";
+ key_down_delay = <7000>;
+ keys_down = <116>;
+ dbg_fn_delay = <2500>;
+ };
+};
+
+&ufsphy_mem {
+ compatible = "qcom,ufs-phy-qmp-v3";
+
+ vdda-phy-supply = <&pm8998_l1>; /* 0.88v */
+ vdda-pll-supply = <&pm8998_l26>; /* 1.2v */
+ vdda-phy-max-microamp = <62900>;
+ vdda-pll-max-microamp = <18300>;
+
+ status = "ok";
+};
+
+&ufshc_mem {
+ vdd-hba-supply = <&ufs_phy_gdsc>;
+ vdd-hba-fixed-regulator;
+ vcc-supply = <&pm8998_l20>;
+ vcc-voltage-level = <2950000 2960000>;
+ vccq2-supply = <&pm8998_s4>;
+ vcc-max-microamp = <600000>;
+ vccq2-max-microamp = <600000>;
+
+ qcom,vddp-ref-clk-supply = <&pm8998_l2>;
+ qcom,vddp-ref-clk-max-microamp = <100>;
+
+ status = "ok";
+};
+
+&pcie0 {
+ status = "disabled";
+};
+
+&pmi8998_switch1 {
+ pinctrl-names = "led_enable", "led_disable";
+ pinctrl-0 = <&flash_led3_front_en>;
+ pinctrl-1 = <&flash_led3_front_dis>;
+};
+
+&qupv3_se9_2uart {
+ status = "ok";
+};
+
+&qupv3_se7_spi {
+ status = "ok";
+ qcom,disable-autosuspend;
+};
+
+&qupv3_se3_i2c {
+ status = "ok";
+};
+
+&qupv3_se10_i2c {
+ status = "ok";
+};
+
+&qupv3_se6_4uart {
+ status = "ok";
+};
+
+&qcom_ion {
+ qcom,ion-heap@30 {
+ reg = <30>;
+ memory-region = <&easel_mem>;
+ qcom,ion-heap-type = "CARVEOUT";
+ };
+};
+
+&thermal_zones {
+ vbat_too_low {
+ trips {
+ low-vbat {
+ temperature = <2800>;
+ };
+ };
+ };
+};
+
+&spmi_bus {
+ qcom,pm8998@0 {
+ qcom,power-on@800 {
+ interrupts = <0x0 0x8 0x0 IRQ_TYPE_NONE>,
+ <0x0 0x8 0x1 IRQ_TYPE_NONE>,
+ <0x0 0x8 0x3 IRQ_TYPE_NONE>,
+ <0x0 0x8 0x4 IRQ_TYPE_NONE>,
+ <0x0 0x8 0x5 IRQ_TYPE_NONE>;
+ interrupt-names = "kpdpwr", "resin", "kpdpwr-bark",
+ "resin-bark", "kpdpwr-resin-bark";
+ qcom,pon_1 {
+ qcom,use-bark;
+ };
+ };
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-dev2.dts b/arch/arm64/boot/dts/google/sdm845-b1c1-dev2.dts
new file mode 100644
index 0000000..7a0e0d9
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-dev2.dts
@@ -0,0 +1,37 @@
+/* Copyright (c) 2017 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-b1-common-v2.1.dtsi"
+#include "sdm845-b1c1-proto-evt-audio.dtsi"
+#include "sdm845-b1c1-audio-cal.dtsi"
+#include "sdm845-b1c1-thermal-v2.dtsi"
+#include "sdm845-b1-evt1-panel.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 B1C1 Dev v2.0 board";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00020205 0>;
+};
+
+&vd6281 {
+ hw-version = <20>; /* cut2.0 */
+};
+
+&nq {
+ ese {
+ loader_scripts_path =
+ "/vendor/etc/ese/loaderservice_updater_eng_";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-devboard-audio.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-devboard-audio.dtsi
new file mode 100644
index 0000000..beafead
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-devboard-audio.dtsi
@@ -0,0 +1,85 @@
+/* AUDIO: RESET: GPIO64, INTR1: GPIO54, INTR2: GPIO53, EXT_CLK: PM845.68
+ * LN_BB_CLK3, SLIMBUS_CLK: GPIO70, SLIMBUS_DATA[0..1]: GPIO71,GPIO72 */
+
+#include <dt-bindings/clock/qcom,audio-ext-clk.h>
+#include "sdm845-audio-overlay.dtsi"
+
+&pm8998_gpios {
+ gpio@cc00 { /* GPIO 13 */
+ status = "disabled";
+ };
+};
+
+&clock_audio_lnbb {
+ clocks = <&clock_rpmh RPMH_LN_BB_CLK3>;
+};
+
+&wcd_usbc_analog_en1_gpio {
+ status = "disabled";
+};
+
+&snd_934x {
+ compatible = "qcom,sdm845-asoc-snd-tavil-max98927";
+ qcom,audio-routing =
+ "AIF4 VI", "MCLK",
+ "RX_BIAS", "MCLK",
+ "MADINPUT", "MCLK",
+ "DMIC0", "MIC BIAS1",
+ "MIC BIAS1", "Digital Mic0",
+ "DMIC1", "MIC BIAS1",
+ "MIC BIAS1", "Digital Mic1",
+ "DMIC2", "MIC BIAS3",
+ "MIC BIAS3", "Digital Mic2",
+ "DMIC5", "MIC BIAS4",
+ "MIC BIAS4", "Digital Mic3";
+
+ qcom,msm-mbhc-hphl-swh = <0>;
+ qcom,msm-mbhc-gnd-swh = <0>;
+ qcom,wsa-max-devs = <0>;
+ qcom,msm-mbhc-hs-mic-max-threshold-mv = <1700>;
+ qcom,msm-mbhc-hs-mic-min-threshold-mv = <50>;
+
+ qcom,tavil-mclk-clk-freq = <9600000>;
+ asoc-codec = <&stub_codec>, <&ext_disp_audio_codec>;
+ asoc-codec-names = "msm-stub-codec.1",
+ "msm-ext-disp-audio-codec-rx";
+
+ pinctrl-names = "aud_active", "aud_sleep";
+ pinctrl-0 = <&wcd_usbc_analog_en2_active>;
+ pinctrl-1 = <&wcd_usbc_analog_en2_idle>;
+};
+
+/* path to speaker boosters */
+&dai_mi2s3 {
+ /* AP in SD0 (gpio60), AP out SD1 (gpio61); */
+ /* SD0; in; ("rx-lines" is playback path): mask=1 */
+ /* SD1; out ("tx-lines" is capture path): mask=2 */
+ pinctrl-names = "default", "sleep";
+ pinctrl-0 =
+ <&quat_mi2s_mclk_active &quat_mi2s_active &quat_mi2s_sd0_active
+ &quat_mi2s_sd1_active>;
+ pinctrl-1 =
+ <&quat_mi2s_mclk_sleep &quat_mi2s_sleep &quat_mi2s_sd0_sleep
+ &quat_mi2s_sd1_sleep>;
+};
+
+/* speaker boosters */
+&qupv3_se12_i2c {
+ status = "ok";
+ max98927_codec1: max98927@39 {
+ compatible = "maxim,max98927";
+ reset-gpios = <&tlmm 75 0x00>;
+ maxim,vmon-slot-no = <0>;
+ maxim,imon-slot-no = <1>;
+ maxim,interleave-mode = <0>;
+ reg = <0x39>;
+ };
+ max98927_codec2: max98927@3a {
+ compatible = "maxim,max98927";
+ reset-gpios = <&tlmm 75 0x00>;
+ maxim,vmon-slot-no = <2>;
+ maxim,imon-slot-no = <3>;
+ maxim,interleave-mode = <0>;
+ reg = <0x3a>;
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-devboard-common.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-devboard-common.dtsi
new file mode 100644
index 0000000..c00ce18
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-devboard-common.dtsi
@@ -0,0 +1,245 @@
+/* Copyright (c) 2016, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include "sdm845-b1c1-common.dtsi"
+
+#include "sdm845-taimen-touch-stm-ftm4.dtsi"
+#include "sdm845-b1c1-devboard-audio.dtsi"
+#include "sdm845-b1c1-devboard-wcharger.dtsi"
+
+&sensor_front_aux {
+ csiphy-sd-index = <2>;
+};
+
+&mdss_mdp {
+ qcom,sde-vsync-source = <0>;
+};
+
+&dsi_sw43402_cmd_display {
+ qcom,dsi-display-active;
+};
+
+&sde_te_active {
+ mux {
+ pins = "gpio10";
+ };
+ config {
+ pins = "gpio10";
+ };
+};
+
+&sde_te_suspend {
+ mux {
+ pins = "gpio10";
+ };
+ config {
+ pins = "gpio10";
+ };
+};
+
+&nq {
+ status = "disabled";
+};
+
+&nfc_enable_default {
+ mux {
+ /* 12: NFC ENABLE, 24:ESE Enable, 122:ESE WAKEUP INT */
+ pins = "gpio12", "gpio24", "gpio122";
+ };
+ config {
+ pins = "gpio12", "gpio24", "gpio122";
+ };
+};
+
+/* B1 P1 has easel spi connected to se11 */
+&aliases {
+ spi10 = "/soc/spi@a8c000";
+};
+&qupv3_se0_spi {
+ status = "disabled";
+};
+&qupv3_se11_spi {
+ status = "okay";
+ mnh-spi@0 {
+ compatible = "intel,mnh-spi";
+ reg = <0>;
+ spi-max-frequency = <25000000>;
+ spi-cpol;
+ spi-cpha;
+ };
+ ese@0 {
+ status = "disabled";
+ };
+};
+
+/* devboard doesn't have easel PMIC, so we need a pull-down on resetb */
+&easel_pmic {
+ pinctrl-0 = <&s2mpb04_pon_default &s2mpb04_intb_default
+ &s2mpb04_resetb_devboard>;
+};
+
+/* override thermistor configs for devboard only */
+&pm8998_vadc {
+ chan@4d {
+ label = "msm_therm";
+ };
+
+ chan@4e {
+ label = "emmc_therm";
+ };
+
+ chan@4f {
+ label = "pa_therm";
+ };
+
+ chan@51 {
+ label = "quiet_therm";
+ };
+};
+
+&pm8998_adc_tm {
+ chan@4d {
+ label = "msm_therm";
+ };
+
+ chan@4e {
+ label = "emmc_therm";
+ };
+
+ chan@4f {
+ label = "pa_therm";
+ };
+
+ chan@51 {
+ label = "quiet_therm";
+ };
+};
+
+&thermal_zones {
+ sdm-therm-adc {
+ status = "disabled";
+ };
+ fps-therm-adc {
+ status = "disabled";
+ };
+ usbc-therm-adc {
+ status = "disabled";
+ };
+ pmi_therm-adc {
+ status = "disabled";
+ };
+ msm-therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x4d>;
+ thermal-governor = "user_space";
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+
+ emmc-therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x4e>;
+ thermal-governor = "user_space";
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+
+ pa_therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x4f>;
+ thermal-governor = "user_space";
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+
+ quiet-therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x51>;
+ thermal-governor = "user_space";
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+};
+
+&soc {
+ /delete-node/google,port-overheat-mitigation;
+};
+
+&qupv3_se7_spi {
+ status = "disabled";
+};
+
+&qupv3_se8_spi {
+ status = "ok";
+ /* Sleep the SPI lines when not in use. This allows SLPI to share the
+ * bus with the AP. */
+ pinctrl-1 = <&qupv3_se8_spi_sleep>;
+};
+
+&aliases {
+ spi0 = "/soc/spi@a80000";
+};
+
+&ctdl_ap_irq {
+ mux {
+ pins = "gpio96";
+ };
+
+ config {
+ pins = "gpio96";
+ };
+};
+
+&ap_ctdl_irq {
+ mux {
+ pins = "gpio94";
+ };
+
+ config {
+ pins = "gpio94";
+ };
+};
+
+&qupv3_se4_spi {
+ citadel@0 {
+ interrupts = <96 0x0>;
+ citadel,ctdl_ap_irq = <&tlmm 96 0x0>;
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-devboard-wcharger.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-devboard-wcharger.dtsi
new file mode 100644
index 0000000..2e3d6ce5
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-devboard-wcharger.dtsi
@@ -0,0 +1,40 @@
+/*
+ * P9221 eval board on B1C1 dev board
+ * I2C pins 85/86 QUP_L0(5)
+ * INT GPIO 132
+ *
+ * Connect:
+ * P9221 VOUT to DC_IN
+ * P9221 J1-2 (INT) to J2-17
+ * P9221 J1-3 (SCL) to J20-3
+ * P9221 J1-4 (SDA) to J20-13
+ * P9221 J1-5 (GND) to J20-1
+ */
+
+&p9221_int {
+ mux {
+ pins = "gpio132";
+ };
+ config {
+ pins = "gpio132";
+ };
+};
+
+&qupv3_se5_i2c {
+ status = "okay";
+ p9221@61 {
+ compatible = "idt,p9221";
+ reg = <0x61>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&p9221_int>;
+ interrupt-parent = <&tlmm>;
+ interrupts = <132 0>;
+ idt,irq_gpio = <&tlmm 132 0x00>;
+ status = "ok";
+ };
+};
+
+/* Disable wchg on I2C10 */
+&p9221 {
+ status = "disabled";
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-display.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-display.dtsi
new file mode 100644
index 0000000..0de6ebb
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-display.dtsi
@@ -0,0 +1,328 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include "../qcom/dsi-panel-sim-video.dtsi"
+#include "dsi-panel-sw43402-dsc-qhd-cmd.dtsi"
+#include "dsi-panel-sw43408-dsc-fhd-cmd.dtsi"
+#include "dsi-panel-s6e3ha8-dsc-wqhd-cmd.dtsi"
+
+&sde_te_active {
+ mux {
+ pins = "gpio12";
+ };
+ config {
+ pins = "gpio12";
+ };
+};
+
+&sde_te_suspend {
+ mux {
+ pins = "gpio12";
+ };
+ config {
+ pins = "gpio12";
+ };
+};
+
+&tlmm {
+ se8_spiflash: se8_spiflash {
+ mux {
+ pins = "gpio65", "gpio66", "gpio67", "gpio68";
+ function = "gpio";
+ };
+ config {
+ pins = "gpio65", "gpio66", "gpio67", "gpio68";
+ input-enable;
+ bias-disable;
+ };
+ };
+};
+
+&soc {
+ dsi_sim_vid_display: qcom,dsi-display@4 {
+ compatible = "qcom,dsi-display";
+ label = "dsi_sim_vid_display";
+ qcom,display-type = "primary";
+
+ qcom,dsi-ctrl = <&mdss_dsi0>;
+ qcom,dsi-phy = <&mdss_dsi_phy0>;
+ clocks = <&mdss_dsi0_pll BYTECLK_MUX_0_CLK>,
+ <&mdss_dsi0_pll PCLK_MUX_0_CLK>;
+ clock-names = "src_byte_clk", "src_pixel_clk";
+
+ pinctrl-names = "panel_active", "panel_suspend";
+ pinctrl-0 = <&sde_dsi_active &sde_te_active>;
+ pinctrl-1 = <&sde_dsi_suspend &sde_te_suspend>;
+
+ qcom,dsi-panel = <&dsi_sim_vid>;
+ };
+
+ dsi_sw43402_cmd_display: qcom,dsi-display@50 {
+ compatible = "qcom,dsi-display";
+ label = "dsi_sw43402_cmd_display";
+ qcom,display-type = "primary";
+
+ qcom,dsi-ctrl = <&mdss_dsi0>;
+ qcom,dsi-phy = <&mdss_dsi_phy0>;
+ clocks = <&mdss_dsi0_pll BYTECLK_MUX_0_CLK>,
+ <&mdss_dsi0_pll PCLK_MUX_0_CLK>;
+ clock-names = "src_byte_clk", "src_pixel_clk";
+
+ pinctrl-names = "panel_active", "panel_suspend";
+ pinctrl-0 = <&sde_dsi_active &sde_te_active &sde_pmgpio_active>;
+ pinctrl-1 = <&sde_dsi_suspend &sde_te_suspend &sde_pmgpio_suspend>;
+
+ qcom,dsi-panel = <&dsi_sw43402_dsc_qhd_cmd>;
+
+ vddio-supply = <&pm8998_l14>;
+ vdd-supply = <&pm8998_l28>;
+ };
+
+ dsi_sw43408_cmd_display: qcom,dsi-display@51 {
+ compatible = "qcom,dsi-display";
+ label = "dsi_sw43408_cmd_display";
+ qcom,display-type = "primary";
+
+ qcom,dsi-ctrl = <&mdss_dsi0>;
+ qcom,dsi-phy = <&mdss_dsi_phy0>;
+ clocks = <&mdss_dsi0_pll BYTECLK_MUX_0_CLK>,
+ <&mdss_dsi0_pll PCLK_MUX_0_CLK>;
+ clock-names = "src_byte_clk", "src_pixel_clk";
+
+ pinctrl-names = "panel_active", "panel_suspend";
+ pinctrl-0 = <&sde_dsi_active &sde_te_active>;
+ pinctrl-1 = <&sde_dsi_suspend &sde_te_suspend>;
+
+ qcom,dsi-panel = <&dsi_sw43408_dsc_fhd_cmd>;
+
+ vddi-supply = <&pm8998_l14>;
+ vpnl-supply = <&pm8998_l28>;
+ lab-supply = <&lab_regulator>;
+ };
+
+ dsi_s6e3ha8_cmd_display: qcom,dsi-display@52 {
+ compatible = "qcom,dsi-display";
+ label = "dsi_s6e3ha8_cmd_display";
+ qcom,display-type = "primary";
+
+ qcom,dsi-ctrl = <&mdss_dsi0>;
+ qcom,dsi-phy = <&mdss_dsi_phy0>;
+ clocks = <&mdss_dsi0_pll BYTECLK_MUX_0_CLK>,
+ <&mdss_dsi0_pll PCLK_MUX_0_CLK>;
+ clock-names = "src_byte_clk", "src_pixel_clk";
+
+ pinctrl-names = "panel_active", "panel_suspend";
+ pinctrl-0 = <&sde_dsi_active &sde_te_active &se8_spiflash>;
+ pinctrl-1 = <&sde_dsi_suspend &sde_te_suspend>;
+
+ qcom,dsi-panel = <&dsi_s6e3ha8_dsc_wqhd_cmd>;
+
+ vddi-supply = <&pm8998_l14>;
+ vci-supply = <&pm8998_l28>;
+ lab-supply = <&lab_regulator>;
+ };
+
+ sde_wb: qcom,wb-display@0 {
+ compatible = "qcom,wb-display";
+ cell-index = <0>;
+ label = "wb_display";
+ };
+
+ ext_disp: qcom,msm-ext-disp {
+ compatible = "qcom,msm-ext-disp";
+
+ ext_disp_audio_codec: qcom,msm-ext-disp-audio-codec-rx {
+ compatible = "qcom,msm-ext-disp-audio-codec-rx";
+ };
+ };
+};
+
+/*
+ * Reset is GPIO6
+ * VSYNC/TE is GPIO12 (vsync-source 2)
+ * VDDI is 1p8, L14A source (first one up)
+ * VPNL (POLED_VCI??) is 3.0 L28A source. (second one up)
+ * POLED_GPO0 is also PM845 GPIO2 - > (do not drive)
+ * POLED_GPO1 is also PM845 GPIO5 - > (do not drive)
+ * Single DSI (dsi0)
+ */
+
+
+&mdss_dsi1 {
+ status = "disabled";
+};
+
+&sde_dp {
+ status = "disabled";
+};
+
+&mdss_mdp {
+ connectors = <&sde_rscc &sde_wb>;
+ qcom,sde-vsync-source = <2>;
+};
+
+&dsi_sw43402_dsc_qhd_cmd {
+ qcom,panel-supply-entries = <&dsi_panel_pwr_supply_vdd_no_labibb>;
+
+ qcom,mdss-dsi-t-clk-post = <0x0B>;
+ qcom,mdss-dsi-t-clk-pre = <0x23>;
+
+ qcom,platform-reset-gpio = <&tlmm 6 0>;
+ qcom,mdss-dsi-display-timings {
+ timing@0 {
+ qcom,mdss-dsi-panel-phy-timings =
+ [00 15 05 05 20 1F 05 05 03 03 04 00];
+ };
+ };
+};
+
+&dsi_sw43408_dsc_fhd_cmd {
+ qcom,panel-supply-entries = <&dsi_panel_pwr_supply_sw43408>;
+ qcom,mdss-dsi-t-clk-post = <0x09>;
+ qcom,mdss-dsi-t-clk-pre = <0x1A>;
+
+ qcom,platform-reset-gpio = <&tlmm 6 0>;
+ qcom,mdss-dsi-display-timings {
+ timing@0 {
+ qcom,mdss-dsi-panel-phy-timings =
+ [00 0E 03 03 1E 1D 04 04 02 03 04 00];
+ };
+ };
+};
+
+&dsi_s6e3ha8_dsc_wqhd_cmd {
+ qcom,panel-supply-entries = <&dsi_panel_pwr_supply_s6e3ha8>;
+
+ qcom,platform-reset-gpio = <&tlmm 6 0>;
+
+ qcom,mdss-dsi-t-clk-post = <0x0D>;
+ qcom,mdss-dsi-t-clk-pre = <0x2F>;
+ qcom,mdss-dsi-display-timings {
+ timing@0 {
+ qcom,mdss-dsi-panel-phy-timings =
+ [00 1E 08 07 24 22 08 08 05 03 04 00];
+ };
+ };
+};
+
+&dsi_sim_vid {
+ qcom,panel-supply-entries = <&dsi_panel_pwr_supply_vdd_no_labibb>;
+ qcom,mdss-dsi-bl-pmic-control-type = "bl_ctrl_wled";
+ qcom,platform-reset-gpio = <&tlmm 6 0>;
+ qcom,mdss-dsi-t-clk-post = <0x0d>;
+ qcom,mdss-dsi-t-clk-pre = <0x2d>;
+ qcom,mdss-dsi-display-timings {
+ timing@0{
+ qcom,mdss-dsi-panel-phy-timings = [00 1c 07 07 23 21 07
+ 07 05 03 04 00];
+ qcom,display-topology = <1 0 1>,
+ <2 0 1>;
+ qcom,default-topology-index = <0>;
+ };
+ };
+};
+
+&pmi8998_wled {
+ status = "okay";
+ qcom,disp-type-amoled;
+ qcom,led-strings-list = [02 03];
+ qcom,avdd-target-voltage-mv = <7600>;
+ qcom,en-cabc;
+};
+
+&labibb {
+ status = "okay";
+ qcom,qpnp-labibb-mode = "amoled";
+ qcom,swire-control;
+ qcom,aod-mode;
+};
+
+&pm8998_l28 {
+ qcom,init-voltage = <3000000>;
+};
+
+&soc {
+ dsi_panel_pwr_supply_vdd_no_labibb: dsi_panel_pwr_supply_vdd_no_labibb {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ qcom,panel-supply-entry@0 {
+ reg = <0>;
+ qcom,supply-name = "vddio";
+ qcom,supply-min-voltage = <1800000>;
+ qcom,supply-max-voltage = <1800000>;
+ qcom,supply-enable-load = <62000>;
+ qcom,supply-disable-load = <80>;
+ qcom,supply-post-on-sleep = <20>;
+ };
+
+ qcom,panel-supply-entry@1 {
+ reg = <1>;
+ qcom,supply-name = "vdd";
+ qcom,supply-min-voltage = <3000000>;
+ qcom,supply-max-voltage = <3000000>;
+ qcom,supply-enable-load = <857000>;
+ qcom,supply-disable-load = <0>;
+ qcom,supply-post-on-sleep = <0>;
+ };
+ };
+
+ dsi_panel_pwr_supply_sw43408: dsi_panel_pwr_supply_sw43408 {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ qcom,panel-supply-entry@0 {
+ reg = <0>;
+ qcom,supply-name = "vddi";
+ qcom,supply-min-voltage = <1800000>;
+ qcom,supply-max-voltage = <1800000>;
+ qcom,supply-enable-load = <62000>;
+ qcom,supply-disable-load = <80>;
+ qcom,supply-post-on-sleep = <0>;
+ };
+
+ qcom,panel-supply-entry@1 {
+ reg = <1>;
+ qcom,supply-name = "vpnl";
+ qcom,supply-min-voltage = <3000000>;
+ qcom,supply-max-voltage = <3000000>;
+ qcom,supply-enable-load = <857000>;
+ qcom,supply-disable-load = <0>;
+ qcom,supply-post-on-sleep = <1>;
+ };
+ };
+
+ dsi_panel_pwr_supply_s6e3ha8: dsi_panel_pwr_supply_s6e3ha8 {
+ #address-cells = <1>;
+ #size-cells = <0>;
+
+ qcom,panel-supply-entry@0 {
+ reg = <0>;
+ qcom,supply-name = "vddi";
+ qcom,supply-min-voltage = <1800000>;
+ qcom,supply-max-voltage = <1800000>;
+ qcom,supply-enable-load = <100000>;
+ qcom,supply-disable-load = <100>;
+ qcom,supply-post-on-sleep = <0>;
+ };
+
+ qcom,panel-supply-entry@1 {
+ reg = <1>;
+ qcom,supply-name = "vci";
+ qcom,supply-min-voltage = <3000000>;
+ qcom,supply-max-voltage = <3000000>;
+ qcom,supply-enable-load = <100000>;
+ qcom,supply-disable-load = <100>;
+ qcom,supply-post-on-sleep = <0>;
+ };
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-easel.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-easel.dtsi
new file mode 100644
index 0000000..68e0eef
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-easel.dtsi
@@ -0,0 +1,236 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <dt-bindings/interrupt-controller/irq.h>
+
+&aliases {
+ serial1 = "/soc/qcom,qup_uart@88c000";
+ spi10 = "/soc/spi@880000";
+};
+
+&pcie1 {
+ /* QCOM's GICM implementation is not compatible with Easel */
+ qcom,msi-gicm-addr = <>;
+ qcom,msi-gicm-base = <>;
+
+ /* reduce endpoint reset latency */
+ qcom,ep-latency = <1>;
+
+ /* WAKE# optional sideband signal not connected */
+ wake-gpio = <>;
+ /* NOTE: signals routed to wrong pin on Binder and B1C1 */
+ perst-gpio = <&tlmm 35 0>;
+
+ /* Remove pcie1_wake_default from pin muxing */
+ pinctrl-0 = <&pcie1_clkreq_default &pcie1_perst_default>;
+};
+
+&tlmm {
+ pcie1 {
+ pcie1_perst_default: pcie1_perst_default {
+ mux {
+ pins = "gpio35";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio35";
+ drive-strength = <2>;
+ bias-pull-down;
+ };
+ };
+ };
+
+ s2mpb04_pins {
+ s2mpb04_pon_default: s2mpb04_pon_default {
+ mux {
+ pins = "gpio29";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio29";
+ drive-strength = <2>;
+ bias-disable;
+ };
+ };
+
+ s2mpb04_intb_default: s2mpb04_intb_default {
+ mux {
+ pins = "gpio30";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio30";
+ drive-strength = <2>;
+ bias-pull-up;
+ };
+ };
+
+ s2mpb04_resetb_default: s2mpb04_resetb_default {
+ mux {
+ pins = "gpio114";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio114";
+ drive-strength = <2>;
+ bias-disable;
+ };
+ };
+ s2mpb04_resetb_devboard: s2mpb04_resetb_devboard {
+ mux {
+ pins = "gpio114";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio114";
+ drive-strength = <2>;
+ bias-pull-down;
+ };
+ };
+ };
+
+ easel_pins {
+ easel_ready_default: easel_ready_default {
+ mux {
+ pins = "gpio52";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio52";
+ drive-strength = <2>;
+ bias-pull-down;
+ };
+ };
+
+ easel_boot_mode_default: easel_boot_mode_default {
+ mux {
+ pins = "gpio11";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio11";
+ drive-strength = <2>;
+ bias-disable;
+ };
+ };
+ };
+};
+
+&qupv3_se10_i2c {
+ status = "okay";
+ easel_pmic: s2mpb04@66 {
+ compatible = "samsung,s2mpb04";
+ reg = <0x66>;
+
+ interrupt-parent = <&tlmm>;
+ interrupts = <30 IRQ_TYPE_EDGE_FALLING>;
+ interrupt-names = "s2mpb04_intb";
+
+ pinctrl-names = "default";
+ pinctrl-0 = <&s2mpb04_pon_default &s2mpb04_intb_default
+ &s2mpb04_resetb_default>;
+
+ samsung,pon-gpio = <&tlmm 29 0x00>;
+ samsung,intb-gpio = <&tlmm 30 0x00>;
+ samsung,resetb-gpio = <&tlmm 114 0x00>;
+
+ s2mpb04_regulator: regulator {
+ compatible = "samsung,s2mpb04-regulator";
+ };
+
+ s2mpb04_gpio: gpio {
+ compatible = "samsung,s2mpb04-gpio";
+ gpio-controller;
+ #gpio-cells = <2>;
+ };
+
+ s2mpb04_thermal: sensor {
+ compatible = "samsung,s2mpb04-thermal";
+ #thermal-sensor-cells = <0>;
+ };
+ };
+};
+
+&qupv3_se0_spi {
+ status = "okay";
+ mnh-spi@0 {
+ compatible = "intel,mnh-spi";
+ reg = <0>;
+ spi-max-frequency = <25000000>;
+ spi-cpol;
+ spi-cpha;
+ };
+};
+
+/*
+ * Default client of SE3 is the I2C for NFC, enabling console requires
+ * modifying devcfg to enable 2-wire UART
+ */
+&qupv3_se3_2uart {
+ status = "disabled";
+};
+
+&soc {
+ mnh-sm {
+ compatible = "intel,mnh-sm";
+
+ clocks = <&clock_rpmh RPMH_LN_BB_CLK2>;
+ clock-names = "ref_clk";
+
+ pinctrl-names = "default";
+ pinctrl-0 = <&easel_ready_default &easel_boot_mode_default>;
+
+ soc-pwr-good-gpio = <&s2mpb04_gpio 0 0x00>;
+ ddr-pad-iso-n-gpio = <&s2mpb04_gpio 1 0x00>;
+ ready-gpio = <&tlmm 52 0x00>;
+ boot-mode-gpio = <&tlmm 11 0x00>;
+ };
+
+ mnh_thermal: mnh_thermal {
+ compatible = "intel,mnh_thermal";
+ #thermal-sensor-cells = <1>;
+ #cooling-cells = <2>;
+ };
+
+ thermal-zones {
+ mnh_ipu1 {
+ polling-delay-passive = <1000>;/* milliseconds */
+ polling-delay = <0>;/* milliseconds */
+ thermal-sensors = <&mnh_thermal 0>;
+ };
+
+ mnh_ipu2 {
+ polling-delay-passive = <1000>;/* milliseconds */
+ polling-delay = <0>;/* milliseconds */
+ thermal-sensors = <&mnh_thermal 1>;
+ };
+
+ mnh_cpu {
+ polling-delay-passive = <1000>;/* milliseconds */
+ polling-delay = <0>;/* milliseconds */
+ thermal-sensors = <&mnh_thermal 2>;
+ };
+
+ mnh_lpddr {
+ polling-delay-passive = <1000>;/* milliseconds */
+ polling-delay = <0>;/* milliseconds */
+ thermal-sensors = <&mnh_thermal 3>;
+ };
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-fingerprint.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-fingerprint.dtsi
new file mode 100644
index 0000000..be1fd0a
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-fingerprint.dtsi
@@ -0,0 +1,74 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&soc {
+ fp_fpc1020 {
+ status = "ok";
+ compatible = "fpc,fpc1020";
+ interrupt-parent = <&tlmm>;
+ interrupts = <121 0x0>;
+ fpc,gpio_rst = <&tlmm 7 0x0>;
+ fpc,gpio_irq = <&tlmm 121 0x0>;
+ vdd_io-supply = <&pm8998_s4>;
+
+ pinctrl-names = "fpc1020_reset_reset",
+ "fpc1020_reset_active",
+ "fpc1020_irq_active";
+
+ pinctrl-0 = <&fp_reset_l>;
+ pinctrl-1 = <&fp_reset_l_output_high>;
+ pinctrl-2 = <&fp_to_ap_irq_default>;
+ };
+};
+
+&tlmm {
+ fp_reset_l: fp_reset_l {
+ mux {
+ pins = "gpio7";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio7";
+ drive-strength = <2>;
+ bias-disable;
+ output-low;
+ };
+ };
+
+ fp_reset_l_output_high: fp_reset_l_output_high {
+ mux {
+ pins = "gpio7";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio7";
+ drive-strength = <2>;
+ bias-disable;
+ output-high;
+ };
+ };
+
+ fp_to_ap_irq_default: fp_to_ap_irq_default {
+ mux {
+ pins = "gpio121";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio121";
+ drive-strength = <2>;
+ bias-pull-down;
+ };
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-haptics-drv2624-overlay.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-haptics-drv2624-overlay.dtsi
new file mode 100644
index 0000000..2cbaf72
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-haptics-drv2624-overlay.dtsi
@@ -0,0 +1,101 @@
+/* B1C1 Haptics Overlay that removes CS40L20 and add DRV2624 */
+
+/* Add PM GPIO15 to the list of allowed GPIOs. */
+&pm8998_gpios {
+ interrupts = <0x0 0xc0 0 IRQ_TYPE_NONE>,
+ <0x0 0xc1 0 IRQ_TYPE_NONE>,
+ <0x0 0xc3 0 IRQ_TYPE_NONE>,
+ <0x0 0xc4 0 IRQ_TYPE_NONE>,
+ <0x0 0xc5 0 IRQ_TYPE_NONE>,
+ <0x0 0xc6 0 IRQ_TYPE_NONE>,
+ <0x0 0xc7 0 IRQ_TYPE_NONE>,
+ <0x0 0xc8 0 IRQ_TYPE_NONE>,
+ <0x0 0xc9 0 IRQ_TYPE_NONE>,
+ <0x0 0xca 0 IRQ_TYPE_NONE>,
+ <0x0 0xcb 0 IRQ_TYPE_NONE>,
+ <0x0 0xcc 0 IRQ_TYPE_NONE>,
+ <0x0 0xcd 0 IRQ_TYPE_NONE>,
+ <0x0 0xce 0 IRQ_TYPE_NONE>,
+ <0x0 0xcf 0 IRQ_TYPE_NONE>,
+ <0x0 0xd0 0 IRQ_TYPE_NONE>,
+ <0x0 0xd1 0 IRQ_TYPE_NONE>,
+ <0x0 0xd2 0 IRQ_TYPE_NONE>,
+ <0x0 0xd4 0 IRQ_TYPE_NONE>,
+ <0x0 0xd6 0 IRQ_TYPE_NONE>;
+ interrupt-names = "pm8998_gpio1", "pm8998_gpio2",
+ "pm8998_gpio4", "pm8998_gpio5",
+ "pm8998_gpio6", "pm8998_gpio7",
+ "pm8998_gpio8", "pm8998_gpio9",
+ "pm8998_gpio10", "pm8998_gpio11",
+ "pm8998_gpio12", "pm8998_gpio13",
+ "pm8998_gpio14", "pm8998_gpio15",
+ "pm8998_gpio16",
+ "pm8998_gpio17", "pm8998_gpio18",
+ "pm8998_gpio19", "pm8998_gpio21",
+ "pm8998_gpio23";
+ qcom,gpios-disallowed = <3 20 22 24 25 26>;
+};
+
+&tlmm {
+ drv2624_rst: drv2624_rst {
+ mux {
+ pins = "gpio87";
+ function = "gpio";
+ };
+ config {
+ drive-strength = <2>;
+ output-low;
+ };
+ };
+ drv2624_intz: drv2624_intz {
+ mux {
+ pins = "gpio88";
+ function = "gpio";
+ };
+ config {
+ drive-strength = <2>;
+ bias-pull-up;
+ };
+ };
+};
+
+&pm8998_gpios {
+ drv2624_pm_gpio {
+ drv2624_pwr: drv2624_pwr {
+ pins = "gpio15";
+ function = "normal";
+ output-high;
+ power-source = <0>;
+ };
+ };
+};
+
+&qupv3_se5_i2c {
+ status = "okay";
+ qcom,clk-freq-out = <400000>;
+ drv2624@5a {
+ compatible = "ti,drv2624";
+ pinctrl-names = "default";
+ pinctrl-0 = <&drv2624_intz &drv2624_rst &drv2624_pwr>;
+ reg = <0x5a>;
+ ti,reset-gpio = <&tlmm 87 0>;
+ ti,irq-gpio = <&tlmm 88 0>;
+ ti,smart-loop = <0>; /* 0-closeloop, 1-openloop */
+ ti,actuator = <1>; /* 0-ERM, 1-LRA */
+ ti,rated-voltage = <77>; /* 1.8 Vrms */
+ ti,odclamp-voltage = <165>; /* 3.5 Vpeak */
+ ti,lra-frequency = <155>; /* f0 = 155Hz */
+ ti,ol-lra-frequency = <155>; /* off-f0 in OL */
+ ti,bemf-factor = <188>;
+ ti,bemf-gain = <1>;
+ ti,voltage-comp = <9>;
+ ti,blanking-time = <2>; /* 50 us */
+ ti,idiss-time = <2>; /* 50 us */
+ ti,zc-det-time = <0>; /* 100 us*/
+ ti,lra-wave-shape = <0>; /* Square wave */
+ ti,waveform-interval = <1>; /* 1 ms */
+ };
+ cs40l20@43 {
+ status = "disabled";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-haptics.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-haptics.dtsi
new file mode 100644
index 0000000..a49f9dc
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-haptics.dtsi
@@ -0,0 +1,51 @@
+/* I2C pins 85/86 QUP_L0(5)
+ * NRST 87
+ * TRIG_INTZ 88
+ *
+ * HAP_PMIC_EN: PM845 GPIO 15
+ */
+
+&tlmm {
+ cs40l20_rst:cs40l20_rst {
+ mux {
+ pins = "gpio87";
+ function = "gpio";
+ };
+ config {
+ pins = "gpio87";
+ drive-strength = <2>;
+ output-low;
+ };
+ };
+};
+
+&qupv3_se5_spi {
+ status = "disabled";
+};
+
+&qupv3_se5_i2c {
+ status = "okay";
+ qcom,clk-freq-out = <1000000>;
+ cs40l20@43 {
+ compatible = "cirrus,cs40l20";
+ pinctrl-names = "default";
+ pinctrl-0 = <&cs40l20_rst>;
+ reset-gpios = <&tlmm 87 0>;
+ reg = <0x43>;
+ cirrus,boost-ind-nanohenry = <1000>;
+ cirrus,boost-cap-microfarad = <4>;
+ cirrus,boost-ipk-milliamp = <4500>;
+ cirrus,f0-default = <2514000>; /* 153 Hz */
+ cirrus,f0-min = <2424832>; /* 148 Hz */
+ cirrus,f0-max = <2703360>; /* 165 Hz */
+ cirrus,redc-default = <327728>; /* 15 Ohm */
+ cirrus,redc-min = <302112>; /* 13.5 Ohm */
+ cirrus,redc-max = <369248>; /* 16.5 Ohm */
+ cirrus,gpio1-rise-index = <5>; /* 100% HEAVY_CLICK */
+ cirrus,gpio1-fall-index = <4>; /* 75% HEAVY_CLICK */
+ cirrus,gpio1-fall-timeout = <47278>; /* 985 ms */
+ cirrus,gpio1-mode = <2>; /* enabled only in suspend mode */
+ status = "ok";
+ };
+};
+
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-memory.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-memory.dtsi
new file mode 100644
index 0000000..75e9237
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-memory.dtsi
@@ -0,0 +1,82 @@
+/* Copyright (c) 2016-2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+/* B1C1 memory includes go here */
+
+/*
+ the following carveout items are still defined in sdm845.dtsi:
+ region address size
+ ========== ========== =========
+ hyp_region: 0x85700000 0x600000
+ xbl_region: 0x85e00000 0x100000
+ removed_region: 0x85fc0000 0x2f40000
+ pil_camera_mem: 0x8bf00000 0x500000
+ pil_adsp_mem: 0x8c500000 0x1a00000
+ wlan_fw_region: 0x8df00000 0x100000
+ */
+
+&pil_modem_mem {
+ reg = <0 0x8e000000 0 0x9800000>;
+};
+
+&pil_video_mem {
+ reg = <0 0x97800000 0 0x500000>;
+};
+
+&pil_cdsp_mem {
+ reg = <0 0x97D00000 0 0x800000>;
+};
+
+&pil_mba_mem {
+ reg = <0 0x98500000 0 0x200000>;
+};
+
+&pil_slpi_mem {
+ reg = <0 0x98700000 0 0x1400000>;
+};
+
+&pil_spss_mem {
+ reg = <0 0x99B00000 0 0x100000>;
+};
+
+&reserved_memory {
+ easel_mem: easel_mem@9f800000 {
+ compatible = "removed-dma-pool";
+ reg = <0 0x9f800000 0 0x2000000>;
+ label = "easel_mem";
+ };
+
+ debug_info: debug_info@0 {
+ compatible = "removed-dma-pool";
+ no-map;
+ reg = <0 0xa1800000 0 0x00010000>;
+ };
+
+ ramoops_mem: ramoops_region@a1810000 {
+ compatible = "removed-dma-pool";
+ no-map;
+ reg = <0 0xa1810000 0 0x200000>;
+ };
+
+ alt_ramoops_mem: alt_ramoops_region@a1a10000 {
+ compatible = "removed-dma-pool";
+ no-map;
+ reg = <0 0xa1a10000 0 0x200000>;
+ };
+
+ ramoops_meta_mem: ramoops_meta_region@a1c10000 {
+ compatible = "removed-dma-pool";
+ no-map;
+ reg = <0 0xa1c10000 0 0x1000>;
+ };
+
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-nfc.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-nfc.dtsi
new file mode 100644
index 0000000..e7df35f
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-nfc.dtsi
@@ -0,0 +1,69 @@
+&tlmm {
+ nfc {
+ /* define the real stuff */
+ nfc_int_default: nfc_int_default {
+ /* default state */
+ mux {
+ /* GPIO 63 NFC Read Interrupt */
+ pins = "gpio63";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio63";
+ drive-strength = <2>; /* 2 MA */
+ bias-pull-down;
+ };
+ };
+
+ nfc_enable_default: nfc_enable_default {
+ /* default state */
+ mux {
+ /* 25: NFC ENABLE, 23: NFC_SE_PWR_REQ */
+ /* 79: NFC_FW_DL_REQ_FROM_HOST */
+ pins = "gpio25", "gpio23", "gpio79";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio25", "gpio23", "gpio79";
+ drive-strength = <2>; /* 2 MA */
+ bias-pull-down;
+ };
+ };
+ };
+};
+
+&qupv3_se3_i2c {
+ status = "okay";
+ nq: nq@28 {
+ status = "okay";
+ compatible = "qcom,nq-nci";
+ reg = <0x28>;
+ interrupt-parent = <&tlmm>;
+ interrupt-names = "nfc_irq";
+ interrupts = <63 0>;
+ qcom,nq-ven = <&tlmm 25 0x00>;
+ qcom,nq-irq = <&tlmm 63 0x00>;
+ qcom,nq-firm = <&tlmm 79 0x00>;
+ qcom,nq-esepwr = <&tlmm 23 0x00>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&nfc_int_default &nfc_enable_default>;
+ qcom,nq-postinit-cmd = /bits/ 8
+ <0x5 0x20 0x02 0x05 0x01 0xA0 0xF2 0x01 0x01>;
+ };
+};
+
+&qupv3_se11_spi {
+ status = "ok";
+
+ ese@0 {
+ compatible = "nxp,pn81a";
+ reg = <0>;
+ spi-max-frequency = <10000000>;
+ /* i2c NFC device handles eSE power:
+ * i2c_1, reg 0x28
+ */
+ nxp,nfcc = "1-0028";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-pinctrl.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-pinctrl.dtsi
new file mode 100644
index 0000000..7de5386
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-pinctrl.dtsi
@@ -0,0 +1,74 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&tlmm {
+ pmx_sde {
+ sde_dsi_active {
+ mux {
+ pins = "gpio6";
+ };
+ config {
+ pins = "gpio6";
+ };
+ };
+ sde_dsi_suspend {
+ mux {
+ pins = "gpio6";
+ };
+ config {
+ pins = "gpio6";
+ };
+ };
+ };
+ ts_mux {
+ ts_int_active: ts_int_active {
+ mux {
+ pins = "gpio125";
+ function = "gpio";
+ };
+ config {
+ pins = "gpio125";
+ drive-strength = <2>;
+ bias-disable;
+ };
+ };
+ ts_int_suspend: ts_int_suspend {
+ mux {
+ pins = "gpio125";
+ function = "gpio";
+ };
+ config {
+ pins = "gpio125";
+ drive-strength = <2>;
+ bias-disable;
+ };
+ };
+ };
+};
+
+&pm8998_gpios {
+ /* Don't drive these, for display */
+ sde_pmgpio_active: sde_pmgpio_active {
+ pins = "gpio2", "gpio5";
+ function = "normal";
+ input-enable;
+ bias-disable;
+ power-source = <0>;
+ };
+ sde_pmgpio_suspend: sde_pmgpio_suspend {
+ pins = "gpio2", "gpio5";
+ function = "normal";
+ input-enable;
+ bias-disable;
+ power-source = <0>;
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-proto-evt-audio.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-proto-evt-audio.dtsi
new file mode 100644
index 0000000..cb6fab0
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-proto-evt-audio.dtsi
@@ -0,0 +1,17 @@
+&cs35l36_codec1 {
+ reset-gpios = <&tlmm 75 0>;
+ irq-gpio = <&tlmm 40 0>;
+ interrupts = <40 IRQ_TYPE_LEVEL_LOW>;
+ pinctrl-0 = <&cs35l36_reset_top &cs35l36_irq_top>;
+ cirrus,irq-config {
+ cirrus,irq-shared;
+ };
+};
+
+
+&cs35l36_codec2 {
+ pinctrl-names = "unused";
+ cirrus,irq-config {
+ cirrus,irq-shared;
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-rainbow.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-rainbow.dtsi
new file mode 100644
index 0000000..354ed635
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-rainbow.dtsi
@@ -0,0 +1,27 @@
+/*
+ * Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&cam_cci {
+ /* VD6281 */
+ status = "ok";
+ vd6281: qcom,rainbow@20 {
+ compatible = "qcom,rainbow";
+ cci-master = <0>;
+ reg = <0x20>;
+ vdd-supply = <&pm8998_lvs1>;
+ regulator-names = "vdd";
+ hw-version = <21>; /* cut2.1 */
+ status = "ok";
+ };
+};
+
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-regulator.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-regulator.dtsi
new file mode 100644
index 0000000..c9f57a4
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-regulator.dtsi
@@ -0,0 +1,26 @@
+/*
+ * Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&soc {
+ /* Force LDO 19 (touch 3.3v) to stay in HPM. */
+ rpmh-regulator-ldoa19 {
+ qcom,supported-modes = <RPMH_REGULATOR_MODE_HPM>;
+ };
+
+ rpmh-regulator-ldoa22 {
+ qcom,supported-modes = <>;
+ pm8998_l22: regulator-l22 {
+ qcom,init-mode = <>;
+ };
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-thermal-v2.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-thermal-v2.dtsi
new file mode 100644
index 0000000..99e6cff
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-thermal-v2.dtsi
@@ -0,0 +1,64 @@
+
+/* Supported hardware: B1 EVT1 => B1 EVT2, C1 P1 => C1 EVT1 */
+
+&thermal_zones {
+ fps-therm-adc {
+ status = "disabled";
+ };
+
+ usbc-therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x4f>;
+ thermal-governor = "user_space";
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+
+ backup-charge {
+ thermal-sensors = <&pm8998_adc_tm 0x4e>;
+
+ trips {
+ batt-charging_lv1 {
+ temperature = <42000>;
+ };
+ batt-charging_lv2 {
+ temperature = <44000>;
+ };
+ batt-charging_lv3 {
+ temperature = <46000>;
+ };
+ batt-charging_lv4 {
+ temperature = <50000>;
+ };
+ };
+ };
+ quiet-therm-adc {
+ trips {
+ shutdown-config {
+ temperature = <62000>;
+ hysteresis = <1000>;
+ type = "critical";
+ };
+ };
+ };
+};
+
+&pm8998_vadc {
+ chan@4f {
+ label = "usbc_therm";
+ };
+};
+
+&pm8998_adc_tm {
+ chan@4f {
+ label = "usbc_therm";
+ qcom,btm-channel-number = <0x80>;
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-thermal.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-thermal.dtsi
new file mode 100644
index 0000000..ba428fb
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-thermal.dtsi
@@ -0,0 +1,438 @@
+
+&bcl_sensor {
+ label = "bcl_sensor";
+};
+
+&pm8998_vadc {
+ chan@83 {
+ label = "vph_pwr";
+ reg = <0x83>;
+ qcom,decimation = <2>;
+ qcom,pre-div-channel-scaling = <1>;
+ qcom,calibration-type = "absolute";
+ qcom,scale-function = <0>;
+ qcom,hw-settle-time = <0>;
+ qcom,fast-avg-setup = <0>;
+ };
+
+ chan@85 {
+ label = "vcoin";
+ reg = <0x85>;
+ qcom,decimation = <2>;
+ qcom,pre-div-channel-scaling = <1>;
+ qcom,calibration-type = "absolute";
+ qcom,scale-function = <0>;
+ qcom,hw-settle-time = <0>;
+ qcom,fast-avg-setup = <0>;
+ };
+
+ chan@4c {
+ label = "xo_therm";
+ reg = <0x4c>;
+ qcom,decimation = <2>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <4>;
+ qcom,hw-settle-time = <2>;
+ qcom,fast-avg-setup = <0>;
+ };
+
+ chan@4d {
+ label = "sdm_therm";
+ reg = <0x4d>;
+ qcom,decimation = <2>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,fast-avg-setup = <0>;
+ };
+
+ chan@4e {
+ label = "quiet_therm";
+ reg = <0x4e>;
+ qcom,decimation = <2>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,fast-avg-setup = <0>;
+ };
+
+ chan@4f {
+ label = "fps_therm";
+ reg = <0x4f>;
+ qcom,decimation = <2>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,fast-avg-setup = <0>;
+ };
+
+ chan@51 {
+ label = "pmi_therm";
+ reg = <0x51>;
+ qcom,decimation = <2>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,fast-avg-setup = <0>;
+ };
+};
+
+&pm8998_adc_tm {
+ chan@83 {
+ label = "vph_pwr";
+ reg = <0x83>;
+ qcom,pre-div-channel-scaling = <1>;
+ qcom,calibration-type = "absolute";
+ qcom,scale-function = <0>;
+ qcom,hw-settle-time = <0>;
+ qcom,btm-channel-number = <0x60>;
+ };
+
+ chan@4c {
+ label = "xo_therm";
+ reg = <0x4c>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <4>;
+ qcom,hw-settle-time = <2>;
+ qcom,btm-channel-number = <0x68>;
+ qcom,thermal-node;
+ };
+
+ chan@4d {
+ label = "sdm_therm";
+ reg = <0x4d>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,btm-channel-number = <0x70>;
+ qcom,thermal-node;
+ };
+
+ chan@4e {
+ label = "quiet_therm";
+ reg = <0x4e>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,btm-channel-number = <0x78>;
+ qcom,thermal-node;
+ };
+
+ chan@4f {
+ label = "fps_therm";
+ reg = <0x4f>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,btm-channel-number = <0x98>;
+ qcom,thermal-node;
+ };
+
+ chan@51 {
+ label = "pmi_therm";
+ reg = <0x51>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,btm-channel-number = <0x88>;
+ qcom,thermal-node;
+ };
+};
+
+&low_vbat {
+ temperature = <3200>;
+ hysteresis = <100>;
+ type = "passive";
+};
+
+&low_soc {
+ temperature = <10>;
+ hysteresis = <0>;
+ type = "passive";
+};
+
+&ibat_high {
+ temperature = <5000>;
+ hysteresis = <200>;
+ type = "passive";
+};
+
+&thermal_zones {
+ vbat {
+ polling-delay-passive = <100>;
+ cooling-maps {
+ vbat_cpu4 {
+ trip = <&low_vbat>;
+ cooling-device =
+ <&CPU4 18 18>;
+ };
+ vbat_cpu5 {
+ trip = <&low_vbat>;
+ cooling-device =
+ <&CPU5 18 18>;
+ };
+ vbat_map6 {
+ trip = <&low_vbat>;
+ cooling-device =
+ <&CPU6 18 18>;
+ };
+ vbat_map7 {
+ trip = <&low_vbat>;
+ cooling-device =
+ <&CPU7 18 18>;
+ };
+ mnh_thermal {
+ trip = <&low_vbat>;
+ cooling-device =
+ <&mnh_thermal 3 3>;
+ };
+ };
+ };
+ soc {
+ polling-delay-passive = <0>;
+ cooling-maps {
+ soc_cpu4 {
+ trip = <&low_soc>;
+ cooling-device =
+ <&CPU4 18 18>;
+ };
+ soc_cpu5 {
+ trip = <&low_soc>;
+ cooling-device =
+ <&CPU5 18 18>;
+ };
+ soc_map6 {
+ trip = <&low_soc>;
+ cooling-device =
+ <&CPU6 18 18>;
+ };
+ soc_map7 {
+ trip = <&low_soc>;
+ cooling-device =
+ <&CPU7 18 18>;
+ };
+ };
+ };
+ ibat-high {
+ polling-delay-passive = <100>;
+ cooling-maps {
+ ibat_cpu4 {
+ trip = <&ibat_high>;
+ cooling-device =
+ <&CPU4 18 18>;
+ };
+ ibat_cpu5 {
+ trip = <&ibat_high>;
+ cooling-device =
+ <&CPU5 18 18>;
+ };
+ ibat_cpu6 {
+ trip = <&ibat_high>;
+ cooling-device =
+ <&CPU6 18 18>;
+ };
+ ibat_cpu7 {
+ trip = <&ibat_high>;
+ cooling-device =
+ <&CPU7 18 18>;
+ };
+ };
+ };
+
+ xo-therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x4c>;
+ thermal-governor = "user_space";
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+
+ sdm-therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x4d>;
+ thermal-governor = "user_space";
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+
+ quiet-therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x4e>;
+ thermal-governor = "user_space";
+ disable-thermal-zone;
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+
+ backup-charge {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x4f>;
+ thermal-governor = "step_wise";
+ disable-thermal-zone;
+
+ trips {
+ batt_charging_lv1: batt-charging_lv1 {
+ temperature = <39000>;
+ hysteresis = <1000>;
+ type = "passive";
+ };
+ batt_charging_lv2: batt-charging_lv2 {
+ temperature = <41000>;
+ hysteresis = <1000>;
+ type = "passive";
+ };
+ batt_charging_lv3: batt-charging_lv3 {
+ temperature = <43000>;
+ hysteresis = <1000>;
+ type = "passive";
+ };
+ batt_charging_lv4: batt-charging_lv4 {
+ temperature = <48000>;
+ hysteresis = <1000>;
+ type = "passive";
+ };
+ };
+ cooling-maps {
+ charging_cdev_lv1 {
+ trip = <&batt_charging_lv1>;
+ cooling-device = <&pmi8998_charger 1 1>;
+ };
+ charging_cdev_lv2 {
+ trip = <&batt_charging_lv2>;
+ cooling-device = <&pmi8998_charger 2 2>;
+ };
+ charging_cdev_lv3 {
+ trip = <&batt_charging_lv3>;
+ cooling-device = <&pmi8998_charger 3 3>;
+ };
+ charging_cdev_lv4 {
+ trip = <&batt_charging_lv4>;
+ cooling-device = <&pmi8998_charger 4 4>;
+ };
+ };
+ };
+
+ fps-therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x4f>;
+ thermal-governor = "user_space";
+ disable-thermal-zone;
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ shutdown-config {
+ temperature = <57000>;
+ hysteresis = <1000>;
+ type = "critical";
+ };
+ };
+ };
+
+ pmi_therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x51>;
+ thermal-governor = "user_space";
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+
+ aoss0-usr {
+ status = "disabled";
+ };
+
+ aoss1-usr {
+ status = "disabled";
+ };
+
+ camera-usr {
+ status = "disabled";
+ };
+
+ compute-hvx-usr {
+ status = "disabled";
+ };
+
+ ddr-usr {
+ status = "disabled";
+ };
+
+ gold-virt-max-step {
+ status = "disabled";
+ };
+
+ gpu-virt-max-step {
+ status = "disabled";
+ };
+
+ kryo-l3-0-usr {
+ status = "disabled";
+ };
+
+ kryo-l3-1-usr {
+ status = "disabled";
+ };
+
+ mdm-core-usr {
+ status = "disabled";
+ };
+
+ mdm-dsp-usr {
+ status = "disabled";
+ };
+
+ mmss-usr {
+ status = "disabled";
+ };
+
+ silv-virt-max-step {
+ status = "disabled";
+ };
+
+ wlan-usr {
+ status = "disabled";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-usb-v2.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-usb-v2.dtsi
new file mode 100644
index 0000000..6a8ba8b
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-usb-v2.dtsi
@@ -0,0 +1,24 @@
+/* Overrides phy tuning parameters for B1 EVT2, B1 EVT2.1 */
+
+&qusb_phy0 {
+ qcom,qusb-phy-init-seq =
+ /* <value reg_offset> */
+ <0x23 0x210 /* PWR_CTRL1 */
+ 0x03 0x04 /* PLL_ANALOG_CONTROLS_TWO */
+ 0x7c 0x18c /* PLL_CLOCK_INVERTERS */
+ 0x80 0x2c /* PLL_CMODE */
+ 0x0a 0x184 /* PLL_LOCK_DELAY */
+ 0x19 0xb4 /* PLL_DIGITAL_TIMERS_TWO */
+ 0x40 0x194 /* PLL_BIAS_CONTROL_1 */
+ 0x19 0x198 /* PLL_BIAS_CONTROL_2 */
+ 0x21 0x214 /* PWR_CTRL2 */
+ 0x00 0x220 /* IMP_CTRL1 */
+ 0x58 0x224 /* IMP_CTRL2 */
+ 0x7F 0x240 /* TUNE1 */
+ 0x29 0x244 /* TUNE2 */
+ 0xca 0x248 /* TUNE3 */
+ 0x04 0x24c /* TUNE4 */
+ 0x03 0x250 /* TUNE5 */
+ 0x00 0x23c /* CHG_CTRL2 */
+ 0x22 0x210>; /* PWR_CTRL1 */
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-usb.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-usb.dtsi
new file mode 100644
index 0000000..188ce0d
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-usb.dtsi
@@ -0,0 +1,109 @@
+
+#include <dt-bindings/usb/typec.h>
+
+&vendor {
+ extcon_usb1: extcon_usb1 {
+ compatible = "linux,extcon-usb-gpio";
+ vbus-gpio = <&pmi8998_gpios 8 GPIO_ACTIVE_HIGH>;
+
+ pinctrl-names = "default";
+ pinctrl-0 = <&usb2_vbus_det_default>;
+ };
+};
+
+&ext_5v_boost {
+ status = "ok";
+};
+
+&usb0 {
+ google,switch-vbus = <250>;
+};
+
+&tlmm {
+ /**
+ * HS enumeration WAR: Add an external pull down resistor (1 K
+ * ~ 1.2 K to USB_DP pin via an external switch controlled by
+ * GPIO104 to enable pull down resistance needed for HS
+ * enumeration.
+ */
+ atest_usb13_active: atest_usb13_active {
+ mux {
+ pins = "gpio104";
+ function = "gpio";
+ };
+ config {
+ pins = "gpio104";
+ drive-strength = <12>;
+ bias-pull-up;
+ };
+ };
+ atest_usb13_suspend: atest_usb13_suspend {
+ mux {
+ pins = "gpio104";
+ function = "gpio";
+ };
+ config {
+ pins = "gpio104";
+ bias-pull-down;
+ drive-strength = <2>;
+ };
+ };
+};
+
+&qusb_phy0 {
+ /* Primary USB port related QUSB2 PHY */
+ pinctrl-names = "atest_usb13_suspend", "atest_usb13_active";
+ pinctrl-0 = <&atest_usb13_suspend>;
+ pinctrl-1 = <&atest_usb13_active>;
+
+ qcom,qusb-phy-reg-offset =
+ <0x240 /* QUSB2PHY_PORT_TUNE1 */
+ 0x1a0 /* QUSB2PHY_PLL_COMMON_STATUS_ONE */
+ 0x210 /* QUSB2PHY_PWR_CTRL1 */
+ 0x230 /* QUSB2PHY_INTR_CTRL */
+ 0x0a8 /* QUSB2PHY_PLL_CORE_INPUT_OVERRIDE */
+ 0x254 /* QUSB2PHY_TEST1 */
+ 0x194 /* PLL_BIAS_CONTROL_1 */
+ 0x198 /* PLL_BIAS_CONTROL_2 */
+ 0x228 /* QUSB2PHY_SQ_CTRL1 */
+ 0x22c /* QUSB2PHY_SQ_CTRL2 */
+ 0x27c>; /* QUSB2PHY_DEBUG_CTRL1 */
+
+ qcom,qusb-phy-init-seq =
+ /* <value reg_offset> */
+ <0x23 0x210 /* PWR_CTRL1 */
+ 0x03 0x04 /* PLL_ANALOG_CONTROLS_TWO */
+ 0x7c 0x18c /* PLL_CLOCK_INVERTERS */
+ 0x80 0x2c /* PLL_CMODE */
+ 0x0a 0x184 /* PLL_LOCK_DELAY */
+ 0x19 0xb4 /* PLL_DIGITAL_TIMERS_TWO */
+ 0x40 0x194 /* PLL_BIAS_CONTROL_1 */
+ 0x20 0x198 /* PLL_BIAS_CONTROL_2 */
+ 0x21 0x214 /* PWR_CTRL2 */
+ 0x00 0x220 /* IMP_CTRL1 */
+ 0x58 0x224 /* IMP_CTRL2 */
+ 0x75 0x240 /* TUNE1 */
+ 0x29 0x244 /* TUNE2 */
+ 0xca 0x248 /* TUNE3 */
+ 0x04 0x24c /* TUNE4 */
+ 0x03 0x250 /* TUNE5 */
+ 0x00 0x23c /* CHG_CTRL2 */
+ 0x22 0x210>; /* PWR_CTRL1 */
+
+ google,skip_efuse_reg;
+};
+
+&pmi8998_pdphy {
+ goog,src-pdo = <PDO_TYPE_FIXED 5000 900 0>; /* 5V @ 0.9A */
+ goog,snk-pdo = <PDO_TYPE_FIXED 5000 3000 0>, /* 5V @ 3A */
+ <PDO_TYPE_FIXED 9000 3000 0>; /* 9V @ 3A */
+ goog,max-snk-mv = <9000>;
+ goog,max-snk-ma = <3000>;
+ goog,max-snk-mw = <27000>;
+ goog,op-snk-mw = <7600>;
+ goog,port-type = <TYPEC_PORT_DRP>;
+ goog,default-role = <TYPEC_SINK>;
+ goog,try-role-hw;
+
+ goog,wlc-supported;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-b1c1-wcharger.dtsi b/arch/arm64/boot/dts/google/sdm845-b1c1-wcharger.dtsi
new file mode 100644
index 0000000..7c9206a
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-b1c1-wcharger.dtsi
@@ -0,0 +1,54 @@
+/*
+ * Wireless charger:
+ * Signal GPIO Type
+ * ----------------------- --------------- ---------
+ * INT 124 interrupt
+ * QI_EN_L 122 output
+ * WLC_1V8_PG 36 interrupt
+ * WLC_BPP_EPP_SLCT 37 output
+ * I2C(10) 55(SDA)/56(SCL) i2c bus
+ */
+
+&tlmm {
+ p9221_det_int: p9221_det_int {
+ mux {
+ pins = "gpio36";
+ function = "gpio";
+ };
+ config {
+ pins = "gpio36";
+ drive-strength = <2>;
+ bias-pull-down;
+ };
+ };
+
+ p9221_int: p9221_int {
+ mux {
+ pins = "gpio124";
+ function = "gpio";
+ };
+ config {
+ pins = "gpio124";
+ drive-strength = <2>;
+ bias-pull-up;
+ };
+ };
+};
+
+&qupv3_se10_i2c {
+ p9221: p9221@61 {
+ compatible = "idt,p9221";
+ reg = <0x61>;
+ pinctrl-names = "default";
+ pinctrl-0 = <&p9221_int
+ &p9221_det_int>;
+ interrupt-parent = <&tlmm>;
+ interrupts = <124 0>,
+ <36 0>;
+ idt,gpio_qien = <&tlmm 122 0x00>;
+ idt,gpio_slct = <&tlmm 37 0x00>;
+ idt,irq_gpio = <&tlmm 124 0x00>;
+ idt,irq_det_gpio = <&tlmm 36 0x00>;
+ status = "ok";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-audio-cal.dtsi b/arch/arm64/boot/dts/google/sdm845-c1-audio-cal.dtsi
new file mode 100644
index 0000000..9b73280
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-audio-cal.dtsi
@@ -0,0 +1,21 @@
+&soc {
+ msm_cirrus_playback {
+ /* the resistance is 7.8197 ohm */
+ cirrus,top-speaker-impedance = <10937>;
+
+ /* the percentage of tolerance is 10 */
+ cirrus,top-speaker-tolerance = <10>;
+
+ /* the resistance is 7.0375 ohm */
+ cirrus,top-speaker-mean = <9843>;
+
+ /* the resistance is 6.3697 ohm */
+ cirrus,bottom-speaker-impedance = <8909>;
+
+ /* the percentage of tolerance is 10 */
+ cirrus,bottom-speaker-tolerance = <10>;
+
+ /* the resistance is 5.5797 ohm */
+ cirrus,bottom-speaker-mean = <7804>;
+ };
+};
\ No newline at end of file
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-battery.dtsi b/arch/arm64/boot/dts/google/sdm845-c1-battery.dtsi
new file mode 100644
index 0000000..d42fd46
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-battery.dtsi
@@ -0,0 +1,89 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&vendor {
+ c1_batterydata: qcom,battery-data {
+ qcom,batt-id-range-pct = <10>;
+ /*
+ * C1 regular
+ * --------------
+ * Desay using ATL cell - 10k
+ * Sunwoda using ATL cell - 47k
+ * Sunwoda using Coslight cell - 68k
+ * Desay using Coslight cell - 180k
+ */
+ #include "batterydata-qcom-c1-10k.dtsi"
+ #include "batterydata-qcom-c1-47k.dtsi"
+ #include "batterydata-qcom-c1-68k.dtsi"
+ #include "batterydata-qcom-c1-180k.dtsi"
+ };
+};
+
+&pmi8998_fg {
+ qcom,battery-data = <&c1_batterydata>;
+ qcom,fg-chg-term-current = <171>;
+ qcom,fg-sys-term-current = <(-214)>;
+};
+
+&google_charger {
+ google,chg-battery-capacity = <3430>;
+ google,wlc-power-supply = "wireless";
+};
+
+&vendor {
+ c1_maxim_config: maxim,config {
+ maxim,c1-10k {
+ maxim,batt-id-kohm = <10>;
+#include "batterydata-maxim-c1-10k.dtsi"
+ };
+ maxim,c1-47k {
+ maxim,batt-id-kohm = <47>;
+#include "batterydata-maxim-c1-47k.dtsi"
+ };
+ maxim,c1-68k {
+ maxim,batt-id-kohm = <68>;
+#include "batterydata-maxim-c1-68k.dtsi"
+ };
+ maxim,c1-180k {
+ maxim,batt-id-kohm = <180>;
+#include "batterydata-maxim-c1-180k.dtsi"
+ };
+ };
+};
+
+&max1720x_fg {
+ maxim,config = <&c1_maxim_config>;
+ maxim,n_regval = /bits/ 16 <
+ 0x9C 0x0380 /* nIChgTerm = 280.0 mA */
+ 0xB7 0x33C1 /* nConvgCfg: RepLow=6% VoltLowOff=140mV MinSlopeX=8/16 RepL_per_stage=1% */
+ 0x9F 0x2606 /* nLearnCfg: Filt Empty */
+ 0xB0 0x7217 /* nConfig: SS TS VS Ten Aen dSOCen TAlrtEn */
+ 0xC0 0xFF00 /* nVAlrtTh: disabled */
+ 0xC1 0x7F80 /* nTAlrtTh: disabled */
+ 0xC2 0xFF00 /* nSAlrtTh: disabled */
+ 0x9D 0x0EB4 /* nFilterCfg: TEMP=90.0s MIX=12.8h VOLT=90.0s CURR=5.625s */
+ 0xC6 0x5A05 /* nFullSOCThr = 90% */
+ 0xD5 0x307D /* nRFastVShdn: rFast=93mohm VShdn=2.5V */
+ 0xB8 0xCDF0 /* nNVCfg0: enOCV enX enCfg enFCfg enLCfg enICT enCG enVE enDC */
+ 0xB9 0xB00E /* nNVCfg1: enTGO enFTh enRFVSH enAT enCRV enCTE */
+ 0xBA 0xFF0A /* nNVCfg2: enT enSOC enMMT enMMV enMMC enVT enFC enIAvg CYCLESpSAVE=5.0 */
+ 0xB3 0x0D80 /* nDesignCap: 3456 mAh */
+ >;
+};
+
+&soc {
+ google,overheat_mitigation {
+ google,begin-mitigation-temp = <500>;
+ google,end-mitigation-temp = <400>;
+ };
+};
+
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-common.dtsi b/arch/arm64/boot/dts/google/sdm845-c1-common.dtsi
new file mode 100644
index 0000000..4e217b06
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-common.dtsi
@@ -0,0 +1,61 @@
+/* Copyright (c) 2017 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include "sdm845-b1c1-common.dtsi"
+
+#include "sdm845-c1-touch.dtsi"
+#include "sdm845-b1c1-audio.dtsi"
+#include "sdm845-c1-battery.dtsi"
+
+/ {
+ qcom,msm-id = <321 0x20001>;
+};
+
+&dsi_s6e3ha8_cmd_display {
+ qcom,dsi-display-active;
+};
+
+/* This overylay is required in order for acdbloader to load different files */
+&soc {
+ sound-tavil {
+ qcom,model = "sdm845-tavil-c1-snd-card";
+ };
+
+ msm_cirrus_playback {
+ /* the resistance is 7.8698 ohm */
+ cirrus,top-speaker-mean = <11007>;
+
+ /* the resistance is 6.1996 ohm */
+ cirrus,bottom-speaker-impedance = <8671>;
+
+ /* the resistance is 6.276 ohm */
+ cirrus,bottom-speaker-mean = <8783>;
+ };
+};
+
+&routing {
+ qcom,msm-pcm-config = "c1";
+};
+
+&mdss_dsi0 {
+ qcom,null-insertion-enabled;
+};
+
+&p9221 {
+ fod = [a4 2a 86 36 84 3d 9a 14 9d 0b 85 6f];
+ fod_epp = [b0 24 90 42 88 52 88 53 91 36 9c f0];
+};
+
+&actuator_rear {
+ param-index = <8>;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-dvt.dts b/arch/arm64/boot/dts/google/sdm845-c1-dvt.dts
new file mode 100644
index 0000000..3293fb4
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-dvt.dts
@@ -0,0 +1,32 @@
+/* Copyright (c) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-c1-common.dtsi"
+#include "sdm845-patherm.dtsi"
+#include "sdm845-usbctherm.dtsi"
+#include "sdm845-c1-usb-v3.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 C1 DVT";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00011405 0>;
+};
+
+&nq {
+ ese {
+ loader_scripts_path =
+ "/vendor/etc/ese/loaderservice_updater_dvt_";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-dvt1.1.dts b/arch/arm64/boot/dts/google/sdm845-c1-dvt1.1.dts
new file mode 100644
index 0000000..1fa23b8
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-dvt1.1.dts
@@ -0,0 +1,26 @@
+/* Copyright (c) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-c1-common.dtsi"
+#include "sdm845-patherm.dtsi"
+#include "sdm845-usbctherm.dtsi"
+#include "sdm845-c1-usb-v3.dtsi"
+#include "sdm845-c1-audio-cal.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 C1 DVT1.1";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00011505 0>;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-evt.dts b/arch/arm64/boot/dts/google/sdm845-c1-evt.dts
new file mode 100644
index 0000000..9ad0be0
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-evt.dts
@@ -0,0 +1,41 @@
+/* Copyright (c) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-c1-common.dtsi"
+#include "sdm845-c1-proto-panel.dtsi"
+#include "sdm845-b1c1-audio-cal.dtsi"
+#include "sdm845-patherm.dtsi"
+#include "sdm845-b1c1-thermal-v2.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 C1 EVT v1.0";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00010A05 0>;
+};
+
+&p9221 {
+ pinctrl-0 = <&p9221_int>;
+ interrupts = <124 0>;
+ idt,irq_det_gpio = <0>;
+ idt,gpio_qien = <0>;
+ idt,gpio_slct = <0>;
+};
+
+&nq {
+ ese {
+ loader_scripts_path =
+ "/vendor/etc/ese/loaderservice_updater_eng_";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-evt1.1.dts b/arch/arm64/boot/dts/google/sdm845-c1-evt1.1.dts
new file mode 100644
index 0000000..4136f34
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-evt1.1.dts
@@ -0,0 +1,32 @@
+/* Copyright (c) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-c1-common.dtsi"
+#include "sdm845-c1-proto-panel.dtsi"
+#include "sdm845-patherm.dtsi"
+#include "sdm845-usbctherm.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 C1 EVT v1.1";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00010B05 0>;
+};
+
+&nq {
+ ese {
+ loader_scripts_path =
+ "/vendor/etc/ese/loaderservice_updater_eng_";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-proto-panel.dtsi b/arch/arm64/boot/dts/google/sdm845-c1-proto-panel.dtsi
new file mode 100644
index 0000000..f40a76e
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-proto-panel.dtsi
@@ -0,0 +1,50 @@
+/* Copyright (c) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+&dsi_s6e3ha8_dsc_wqhd_cmd {
+ qcom,mdss-brightness-max-level = <255>;
+ qcom,mdss-dsi-bl-lut = <
+ 0x0 0x5d 0x71 0x83 0x95 0xa6 0xb6 0xc1
+ 0xcb 0xd5 0xde 0xe8 0xf1 0xfb 0x105 0x10e
+ 0x117 0x11e 0x125 0x12c 0x132 0x139 0x140 0x147
+ 0x14d 0x154 0x15b 0x162 0x169 0x16f 0x175 0x17b
+ 0x180 0x185 0x18a 0x190 0x195 0x19a 0x19f 0x1a5
+ 0x1aa 0x1b0 0x1b6 0x1bb 0x1c1 0x1c6 0x1cc 0x1d1
+ 0x1d5 0x1da 0x1de 0x1e3 0x1e7 0x1ec 0x1f1 0x1f5
+ 0x1fa 0x1fe 0x203 0x208 0x20c 0x211 0x216 0x21b
+ 0x220 0x225 0x22a 0x22f 0x233 0x237 0x23b 0x23f
+ 0x242 0x246 0x249 0x24c 0x24f 0x252 0x255 0x258
+ 0x25c 0x25f 0x263 0x266 0x269 0x26d 0x270 0x273
+ 0x275 0x278 0x27b 0x27d 0x280 0x283 0x286 0x289
+ 0x28d 0x290 0x293 0x296 0x299 0x29c 0x29f 0x2a1
+ 0x2a4 0x2a7 0x2aa 0x2ac 0x2af 0x2b1 0x2b3 0x2b6
+ 0x2b8 0x2bb 0x2bd 0x2c0 0x2c3 0x2c6 0x2c9 0x2cb
+ 0x2ce 0x2d1 0x2d4 0x2d6 0x2d8 0x2da 0x2dc 0x2de
+ 0x2e0 0x2e2 0x2e5 0x2e7 0x2ea 0x2ee 0x2f1 0x2f4
+ 0x2f8 0x2fb 0x2fd 0x300 0x302 0x305 0x307 0x30a
+ 0x30c 0x30f 0x311 0x313 0x316 0x318 0x31a 0x31c
+ 0x31f 0x321 0x323 0x326 0x328 0x32a 0x32d 0x32f
+ 0x332 0x334 0x336 0x339 0x33c 0x33e 0x341 0x344
+ 0x347 0x349 0x34c 0x34e 0x350 0x352 0x354 0x356
+ 0x358 0x35b 0x35d 0x35f 0x361 0x363 0x365 0x367
+ 0x36a 0x36c 0x36e 0x370 0x372 0x375 0x377 0x37a
+ 0x37c 0x37f 0x382 0x384 0x387 0x389 0x38b 0x38d
+ 0x38f 0x391 0x393 0x395 0x397 0x399 0x39b 0x39d
+ 0x39f 0x3a1 0x3a3 0x3a5 0x3a7 0x3a8 0x3aa 0x3ac
+ 0x3ae 0x3b0 0x3b3 0x3b5 0x3b7 0x3ba 0x3bc 0x3be
+ 0x3c0 0x3c3 0x3c5 0x3c7 0x3c9 0x3cb 0x3cd 0x3cf
+ 0x3d0 0x3d2 0x3d4 0x3d6 0x3d8 0x3da 0x3dc 0x3de
+ 0x3df 0x3e1 0x3e3 0x3e5 0x3e6 0x3e8 0x3ea 0x3ec
+ 0x3ee 0x3f1 0x3f3 0x3f5 0x3f8 0x3fa 0x3fd 0x3ff
+ >;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-proto.dts b/arch/arm64/boot/dts/google/sdm845-c1-proto.dts
new file mode 100644
index 0000000..4b7c94b
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-proto.dts
@@ -0,0 +1,49 @@
+/* Copyright (c) 2017 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/dts-v1/;
+
+#include "sdm845-c1-common.dtsi"
+#include "sdm845-c1-proto-panel.dtsi"
+#include "sdm845-b1c1-proto-evt-audio.dtsi"
+#include "sdm845-b1c1-audio-cal.dtsi"
+#include "sdm845-b1c1-thermal-v2.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 C1 Proto v1.0";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,board-id = <0x00010305 0>;
+};
+
+&p9221 {
+ status = "disabled";
+};
+
+&vd6281 {
+ hw-version = <20>; /* cut2.0 */
+};
+
+&nq {
+ ese {
+ loader_scripts_path =
+ "/vendor/etc/ese/loaderservice_updater_eng_";
+ };
+};
+
+&pmi8998_pdphy {
+ /delete-property/ goog,wlc-supported;
+};
+
+&google_charger {
+ /delete-property/ google,wlc-power-supply;
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-touch.dtsi b/arch/arm64/boot/dts/google/sdm845-c1-touch.dtsi
new file mode 100644
index 0000000..c1b70c2
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-touch.dtsi
@@ -0,0 +1,74 @@
+/* Copyright (c) 2017 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/* VDDI = 1.8v, AVDD = 3.0v ~ 3.3v */
+&pm8998_l19 {
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ qcom,init-voltage = <3300000>;
+ status = "okay";
+};
+
+&pm8998_l14 {
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <1800000>;
+ qcom,init-voltage = <1800000>;
+ qcom,init-mode = <RPMH_REGULATOR_MODE_HPM>;
+ status = "okay";
+};
+
+&rpmh_regulator_ldoa14 {
+ qcom,supported-modes = <RPMH_REGULATOR_MODE_HPM>;
+};
+
+&tlmm {
+ sec_ts_reset_active: sec_ts_reset_active {
+ mux {
+ pins = "gpio99";
+ function = "gpio";
+ };
+ config {
+ pins = "gpio99";
+ drive-strength = <2>;
+ bias-disable;
+ output-high;
+ };
+ };
+};
+
+/* AP Touch I2C SDA is gpio27, SCL is gpio28. (QUP2)
+ * Reset gpio99, int gpio125.
+ */
+
+&qupv3_se2_i2c {
+ status = "ok";
+ qcom,clk-freq-out = <400000>;
+
+ touchscreen@48 {
+ status = "ok";
+ compatible = "sec,sec_ts";
+ reg = <0x48>;
+ pinctrl-names = "on_state", "off_state";
+ pinctrl-0 = <&ts_int_active &sec_ts_reset_active>;
+ pinctrl-1 = <&ts_int_suspend>;
+
+ sec,irq_gpio = <&tlmm 125 0>; /* required */
+ sec,switch_gpio = <&tlmm 136 0>;
+ sec,reset_gpio = <&tlmm 99 0>;
+ //sec,irq_type = ?; /* optional - IRQF flags - default is IRQF_TRIGGER_LOW | IRQF_ONESHOT */
+ sec,max_coords = <1440 2960>;
+ sec,firmware_name = "s6sy761.fw";
+ sec,regulator_dvdd = "pm8998_l14"; /* required */
+ sec,regulator_avdd = "pm8998_l19";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-c1-usb-v3.dtsi b/arch/arm64/boot/dts/google/sdm845-c1-usb-v3.dtsi
new file mode 100644
index 0000000..eba8cbf
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-c1-usb-v3.dtsi
@@ -0,0 +1,24 @@
+/* Overrides phy tuning parameters for C1 DVT, C1 DVT1.1 and later*/
+
+&qusb_phy0 {
+ qcom,qusb-phy-init-seq =
+ /* <value reg_offset> */
+ <0x23 0x210 /* PWR_CTRL1 */
+ 0x03 0x04 /* PLL_ANALOG_CONTROLS_TWO */
+ 0x7c 0x18c /* PLL_CLOCK_INVERTERS */
+ 0x80 0x2c /* PLL_CMODE */
+ 0x0a 0x184 /* PLL_LOCK_DELAY */
+ 0x19 0xb4 /* PLL_DIGITAL_TIMERS_TWO */
+ 0x40 0x194 /* PLL_BIAS_CONTROL_1 */
+ 0x28 0x198 /* PLL_BIAS_CONTROL_2 */
+ 0x21 0x214 /* PWR_CTRL2 */
+ 0x00 0x220 /* IMP_CTRL1 */
+ 0x58 0x224 /* IMP_CTRL2 */
+ 0x35 0x240 /* TUNE1 */
+ 0x29 0x244 /* TUNE2 */
+ 0xca 0x248 /* TUNE3 */
+ 0x04 0x24c /* TUNE4 */
+ 0x03 0x250 /* TUNE5 */
+ 0x00 0x23c /* CHG_CTRL2 */
+ 0x22 0x210>; /* PWR_CTRL1 */
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-camera-sensor.dtsi b/arch/arm64/boot/dts/google/sdm845-camera-sensor.dtsi
new file mode 100644
index 0000000..0ac91ba
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-camera-sensor.dtsi
@@ -0,0 +1,444 @@
+/*
+ * Copyright (c) 2016-2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&tlmm {
+ cam_sensor_rear_active: cam_sensor_rear_active {
+ /* RESET, AVDD LDO */
+ mux {
+ pins = "gpio80","gpio16";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio80","gpio16";
+ bias-disable; /* No PULL */
+ drive-strength = <2>; /* 2 MA */
+ };
+ };
+
+ cam_sensor_rear_suspend: cam_sensor_rear_suspend {
+ /* RESET, AVDD LDO */
+ mux {
+ pins = "gpio80","gpio16";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio80","gpio16";
+ bias-pull-down; /* PULL DOWN */
+ drive-strength = <2>; /* 2 MA */
+ };
+ };
+
+ cam_sensor_front_active: cam_sensor_front_active {
+ /* RESET AVDD_LDO*/
+ mux {
+ pins = "gpio21", "gpio8";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio21", "gpio8";
+ bias-disable; /* No PULL */
+ drive-strength = <2>; /* 2 MA */
+ };
+ };
+
+ cam_sensor_front_suspend: cam_sensor_front_suspend {
+ /* RESET AVDD_LDO*/
+ mux {
+ pins = "gpio21", "gpio8";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio21", "gpio8";
+ bias-pull-down; /* PULL DOWN */
+ drive-strength = <2>; /* 2 MA */
+ };
+ };
+
+ cam_sensor_front2_active: cam_sensor_front2_active {
+ /* RESET AVDD_LDO*/
+ mux {
+ pins = "gpio9", "gpio14";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio9", "gpio14";
+ bias-disable; /* No PULL */
+ drive-strength = <2>; /* 2 MA */
+ };
+ };
+
+ cam_sensor_front2_suspend: cam_sensor_front2_suspend {
+ /* RESET AVDD_LDO*/
+ mux {
+ pins = "gpio9", "gpio14";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio9", "gpio14";
+ bias-pull-down; /* PULL DOWN */
+ drive-strength = <2>; /* 2 MA */
+ };
+ };
+};
+
+&cci0_suspend {
+ mux {
+ /* CLK, DATA */
+ pins = "gpio17","gpio18";
+ function = "cci_i2c";
+ };
+ config {
+ pins = "gpio17","gpio18";
+ /* bus has external pull-up, don't pull down */
+ bias-disable;
+ };
+};
+
+&soc {
+ led_flash_rear: qcom,camera-flash@0 {
+ cell-index = <0>;
+ reg = <0x00 0x00>;
+ compatible = "qcom,camera-flash";
+ flash-source = <&pmi8998_flash0 &pmi8998_flash1>;
+ torch-source = <&pmi8998_torch0 &pmi8998_torch1>;
+ switch-source = <&pmi8998_switch0>;
+ status = "ok";
+ };
+
+ led_flash_front: qcom,camera-flash@1 {
+ cell-index = <1>;
+ reg = <0x01 0x00>;
+ compatible = "qcom,camera-flash";
+ flash-source = <&pmi8998_flash2>;
+ torch-source = <&pmi8998_torch2>;
+ switch-source = <&pmi8998_switch1>;
+ status = "ok";
+ };
+
+ actuator_regulator: gpio-regulator@0 {
+ compatible = "regulator-fixed";
+ reg = <0x00 0x00>;
+ regulator-name = "actuator_regulator";
+ regulator-min-microvolt = <2864000>;
+ regulator-max-microvolt = <2864000>;
+ regulator-enable-ramp-delay = <100>;
+ enable-active-high;
+ vin-supply = <&pm8998_l22>;
+ };
+};
+
+&cam_cci {
+ actuator_rear: qcom,actuator@0 {
+ cell-index = <0>;
+ reg = <0x0>;
+ compatible = "qcom,actuator";
+ cci-master = <0>;
+ cam_vaf-supply = <&actuator_regulator>;
+ regulator-names = "cam_vaf";
+ rgltr-cntrl-support;
+ rgltr-min-voltage = <2864000>;
+ rgltr-max-voltage = <2864000>;
+ rgltr-load-current = <0>;
+ };
+
+ actuator_front: qcom,actuator@1 {
+ cell-index = <1>;
+ reg = <0x1>;
+ compatible = "qcom,actuator";
+ cci-master = <1>;
+ cam_vaf-supply = <&actuator_regulator>;
+ regulator-names = "cam_vaf";
+ rgltr-cntrl-support;
+ rgltr-min-voltage = <2864000>;
+ rgltr-max-voltage = <2864000>;
+ rgltr-load-current = <0>;
+ };
+
+ ois_rear: qcom,ois@0 {
+ cell-index = <0>;
+ reg = <0x0>;
+ compatible = "qcom,ois";
+ cci-master = <0>;
+ cam_vaf-supply = <&actuator_regulator>;
+ regulator-names = "cam_vaf";
+ rgltr-cntrl-support;
+ rgltr-min-voltage = <2864000>;
+ rgltr-max-voltage = <2864000>;
+ rgltr-load-current = <0>;
+ };
+
+ eeprom_front: qcom,eeprom@0 {
+ cell-index = <0>;
+ compatible = "qcom,eeprom";
+ reg = <0x0>;
+ cam_vio-supply = <&pm8998_lvs1>;
+ cam_vana-supply = <&pmi8998_bob>;
+ cam_vdig-supply = <&pm8998_s3>;
+ cam_clk-supply = <&titan_top_gdsc>;
+ regulator-names = "cam_vio", "cam_vana", "cam_vdig",
+ "cam_clk";
+ rgltr-cntrl-support;
+ rgltr-min-voltage = <0 3312000 1352000 0>;
+ rgltr-max-voltage = <0 3600000 1352000 0>;
+ rgltr-load-current = <0 80000 105000 0>;
+ gpio-no-mux = <0>;
+ pinctrl-names = "cam_default", "cam_suspend";
+ pinctrl-0 = <&cam_sensor_mclk2_active
+ &cam_sensor_front_active>;
+ pinctrl-1 = <&cam_sensor_mclk2_suspend
+ &cam_sensor_front_suspend>;
+ gpios = <&tlmm 15 0>,
+ <&tlmm 21 0>,
+ <&tlmm 8 0>;
+ gpio-reset = <1>;
+ gpio-vana = <2>;
+ gpio-req-tbl-num = <0 1 2>;
+ gpio-req-tbl-flags = <1 0 0>;
+ gpio-req-tbl-label = "CAMIF_MCLK1",
+ "CAM_RESET1",
+ "CAM_VANA1";
+ sensor-position = <1>;
+ sensor-mode = <0>;
+ cci-master = <1>;
+ status = "ok";
+ clocks = <&clock_camcc CAM_CC_MCLK2_CLK>;
+ clock-names = "cam_clk";
+ clock-cntl-level = "turbo";
+ clock-rates = <24000000>;
+ };
+
+ eeprom_rear: qcom,eeprom@1 {
+ cell-index = <1>;
+ compatible = "qcom,eeprom";
+ reg = <0x1>;
+ cam_vdig-supply = <&pm8998_s3>;
+ cam_vio-supply = <&pm8998_lvs1>;
+ cam_vana-supply = <&pmi8998_bob>;
+ cam_clk-supply = <&titan_top_gdsc>;
+ regulator-names = "cam_vdig", "cam_vio", "cam_vana",
+ "cam_clk";
+ rgltr-cntrl-support;
+ rgltr-min-voltage = <1352000 0 3312000 0>;
+ rgltr-max-voltage = <1352000 0 3600000 0>;
+ rgltr-load-current = <105000 0 80000 0>;
+ gpio-no-mux = <0>;
+ pinctrl-names = "cam_default", "cam_suspend";
+ pinctrl-0 = <&cam_sensor_mclk0_active
+ &cam_sensor_rear_active>;
+ pinctrl-1 = <&cam_sensor_mclk0_suspend
+ &cam_sensor_rear_suspend>;
+ gpios = <&tlmm 13 0>,
+ <&tlmm 80 0>,
+ <&tlmm 16 0>;
+ gpio-reset = <1>;
+ gpio-vana = <2>;
+ gpio-req-tbl-num = <0 1 2>;
+ gpio-req-tbl-flags = <1 0 0>;
+ gpio-req-tbl-label = "CAMIF_MCLK0",
+ "CAM_RESET0",
+ "CAM_VANA0";
+ sensor-position = <0>;
+ sensor-mode = <0>;
+ cci-master = <0>;
+ status = "ok";
+ clocks = <&clock_camcc CAM_CC_MCLK0_CLK>;
+ clock-names = "cam_clk";
+ clock-cntl-level = "turbo";
+ clock-rates = <24000000>;
+ };
+
+ eeprom_front_aux: qcom,eeprom@2 {
+ cell-index = <2>;
+ compatible = "qcom,eeprom";
+ reg = <0x2>;
+ cam_vio-supply = <&pm8998_lvs1>;
+ cam_vana-supply = <&pmi8998_bob>;
+ cam_vdig-supply = <&pm8998_s3>;
+ cam_clk-supply = <&titan_top_gdsc>;
+ regulator-names = "cam_vio", "cam_vana", "cam_vdig",
+ "cam_clk";
+ rgltr-cntrl-support;
+ rgltr-min-voltage = <0 3312000 1352000 0>;
+ rgltr-max-voltage = <0 3600000 1352000 0>;
+ rgltr-load-current = <0 80000 105000 0>;
+ gpio-no-mux = <0>;
+ pinctrl-names = "cam_default", "cam_suspend";
+ pinctrl-0 = <&cam_sensor_mclk2_active
+ &cam_sensor_front2_active>;
+ pinctrl-1 = <&cam_sensor_mclk2_suspend
+ &cam_sensor_front2_suspend>;
+ gpios = <&tlmm 15 0>,
+ <&tlmm 9 0>,
+ <&tlmm 14 0>;
+ gpio-reset = <1>;
+ gpio-vana = <2>;
+ gpio-req-tbl-num = <0 1 2>;
+ gpio-req-tbl-flags = <1 0 0>;
+ gpio-req-tbl-label = "CAMIF_MCLK2",
+ "CAM_RESET2",
+ "CAM_VANA2";
+ sensor-position = <1>;
+ sensor-mode = <0>;
+ cci-master = <1>;
+ status = "ok";
+ clocks = <&clock_camcc CAM_CC_MCLK2_CLK>;
+ clock-names = "cam_clk";
+ clock-cntl-level = "turbo";
+ clock-rates = <24000000>;
+ };
+
+ sensor_rear: qcom,cam-sensor@0 {
+ cell-index = <0>;
+ compatible = "qcom,cam-sensor";
+ reg = <0x0>;
+ csiphy-sd-index = <0>;
+ sensor-position-roll = <90>;
+ sensor-position-pitch = <0>;
+ sensor-position-yaw = <180>;
+ led-flash-src = <&led_flash_rear>;
+ actuator-src = <&actuator_rear>;
+ eeprom-src = <&eeprom_rear>;
+ ois-src = <&ois_rear>;
+ cam_vdig-supply = <&pm8998_s3>;
+ cam_vio-supply = <&pm8998_lvs1>;
+ cam_vana-supply = <&pmi8998_bob>;
+ cam_clk-supply = <&titan_top_gdsc>;
+ regulator-names = "cam_vdig", "cam_vio", "cam_vana",
+ "cam_clk";
+ rgltr-cntrl-support;
+ rgltr-min-voltage = <1352000 0 3312000 0>;
+ rgltr-max-voltage = <1352000 0 3600000 0>;
+ rgltr-load-current = <105000 0 80000 0>;
+ gpio-no-mux = <0>;
+ pinctrl-names = "cam_default", "cam_suspend";
+ pinctrl-0 = <&cam_sensor_mclk0_active
+ &cam_sensor_rear_active>;
+ pinctrl-1 = <&cam_sensor_mclk0_suspend
+ &cam_sensor_rear_suspend>;
+ gpios = <&tlmm 13 0>,
+ <&tlmm 80 0>,
+ <&tlmm 16 0>;
+ gpio-reset = <1>;
+ gpio-vana = <2>;
+ gpio-req-tbl-num = <0 1 2>;
+ gpio-req-tbl-flags = <1 0 0>;
+ gpio-req-tbl-label = "CAMIF_MCLK0",
+ "CAM_RESET0",
+ "CAM_VANA0";
+ sensor-mode = <0>;
+ cci-master = <0>;
+ status = "ok";
+ clocks = <&clock_camcc CAM_CC_MCLK0_CLK>;
+ clock-names = "cam_clk";
+ clock-cntl-level = "turbo";
+ clock-rates = <24000000>;
+ };
+
+ sensor_front: qcom,cam-sensor@1 {
+ cell-index = <1>;
+ compatible = "qcom,cam-sensor";
+ reg = <0x1>;
+ csiphy-sd-index = <2>;
+ sensor-position-roll = <90>;
+ sensor-position-pitch = <0>;
+ sensor-position-yaw = <0>;
+ actuator-src = <&actuator_front>;
+ led-flash-src = <&led_flash_front>;
+ eeprom-src = <&eeprom_front>;
+ cam_vio-supply = <&pm8998_lvs1>;
+ cam_vana-supply = <&pmi8998_bob>;
+ cam_vdig-supply = <&pm8998_s3>;
+ cam_clk-supply = <&titan_top_gdsc>;
+ cam_vaf-supply = <&actuator_regulator>;
+ regulator-names = "cam_vio", "cam_vana", "cam_vdig",
+ "cam_clk", "cam_vaf";
+ rgltr-cntrl-support;
+ rgltr-min-voltage = <0 3312000 1352000 0 2864000>;
+ rgltr-max-voltage = <0 3600000 1352000 0 2864000>;
+ rgltr-load-current = <0 80000 105000 0 0>;
+ gpio-no-mux = <0>;
+ pinctrl-names = "cam_default", "cam_suspend";
+ pinctrl-0 = <&cam_sensor_mclk2_active
+ &cam_sensor_front_active>;
+ pinctrl-1 = <&cam_sensor_mclk2_suspend
+ &cam_sensor_front_suspend>;
+ gpios = <&tlmm 15 0>,
+ <&tlmm 21 0>,
+ <&tlmm 8 0>;
+ gpio-reset = <1>;
+ gpio-vana = <2>;
+ gpio-req-tbl-num = <0 1 2>;
+ gpio-req-tbl-flags = <1 0 0>;
+ gpio-req-tbl-label = "CAMIF_MCLK1",
+ "CAM_RESET1",
+ "CAM_VANA1";
+ sensor-mode = <0>;
+ cci-master = <1>;
+ status = "ok";
+ clocks = <&clock_camcc CAM_CC_MCLK2_CLK>;
+ clock-names = "cam_clk";
+ clock-cntl-level = "turbo";
+ clock-rates = <24000000>;
+ };
+
+ sensor_front_aux: qcom,cam-sensor@2 {
+ cell-index = <2>;
+ compatible = "qcom,cam-sensor";
+ reg = <0x02>;
+ csiphy-sd-index = <1>;
+ sensor-position-roll = <90>;
+ sensor-position-pitch = <0>;
+ sensor-position-yaw = <0>;
+ eeprom-src = <&eeprom_front_aux>;
+ led-flash-src = <&led_flash_front>;
+ cam_vio-supply = <&pm8998_lvs1>;
+ cam_vana-supply = <&pmi8998_bob>;
+ cam_vdig-supply = <&pm8998_s3>;
+ cam_clk-supply = <&titan_top_gdsc>;
+ regulator-names = "cam_vio", "cam_vana", "cam_vdig",
+ "cam_clk";
+ rgltr-cntrl-support;
+ rgltr-min-voltage = <0 3312000 1352000 0>;
+ rgltr-max-voltage = <0 3600000 1352000 0>;
+ rgltr-load-current = <0 80000 105000 0>;
+ gpio-no-mux = <0>;
+ pinctrl-names = "cam_default", "cam_suspend";
+ pinctrl-0 = <&cam_sensor_mclk2_active
+ &cam_sensor_front2_active>;
+ pinctrl-1 = <&cam_sensor_mclk2_suspend
+ &cam_sensor_front2_suspend>;
+ gpios = <&tlmm 15 0>,
+ <&tlmm 9 0>,
+ <&tlmm 14 0>;
+ gpio-reset = <1>;
+ gpio-vana = <2>;
+ gpio-req-tbl-num = <0 1 2>;
+ gpio-req-tbl-flags = <1 0 0>;
+ gpio-req-tbl-label = "CAMIF_MCLK2",
+ "CAM_RESET2",
+ "CAM_VANA2";
+ sensor-mode = <0>;
+ cci-master = <1>;
+ status = "ok";
+ clocks = <&clock_camcc CAM_CC_MCLK2_CLK>;
+ clock-names = "cam_clk";
+ clock-cntl-level = "turbo";
+ clock-rates = <24000000>;
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-patherm.dtsi b/arch/arm64/boot/dts/google/sdm845-patherm.dtsi
new file mode 100644
index 0000000..a4582c6
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-patherm.dtsi
@@ -0,0 +1,43 @@
+
+&pm8998_vadc {
+ chan@50 {
+ label = "pa_therm";
+ reg = <0x50>;
+ qcom,decimation = <2>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,fast-avg-setup = <0>;
+ };
+};
+
+&pm8998_adc_tm {
+ chan@50 {
+ label = "pa_therm";
+ reg = <0x50>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,btm-channel-number = <0x90>;
+ qcom,thermal-node;
+ };
+};
+
+&thermal_zones {
+ pa-therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x50>;
+ thermal-governor = "user_space";
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-taimen-touch-stm-ftm4.dtsi b/arch/arm64/boot/dts/google/sdm845-taimen-touch-stm-ftm4.dtsi
new file mode 100644
index 0000000..8591e34
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-taimen-touch-stm-ftm4.dtsi
@@ -0,0 +1,65 @@
+/* Copyright (c) 2014, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+/* Setting 3.3V to match Taimen (coded and measured.)
+ * There is a concern that PMI8998 VBOB is only 3.312 and is the
+ * reference for L19, but it is matching the Taimen, so let's leave
+ * it for now (Re-investigate for Proto display).
+ */
+&pm8998_l19 {
+ regulator-min-microvolt = <3300000>;
+ regulator-max-microvolt = <3300000>;
+ qcom,init-voltage = <3300000>;
+ status = "okay";
+};
+
+&pm8998_l14 {
+ regulator-min-microvolt = <1800000>;
+ regulator-max-microvolt = <1800000>;
+ qcom,init-voltage = <1800000>;
+ qcom,init-mode = <RPMH_REGULATOR_MODE_HPM>;
+ status = "okay";
+};
+
+&rpmh_regulator_ldoa14 {
+ qcom,supported-modes = <RPMH_REGULATOR_MODE_HPM>;
+};
+
+/* TOUCH_HOST_SEL (gpio128): Keep it low for AP to use touch.
+ * AP Touch I2C SDA is gpio27, SCL is gpio28. (QUP2)
+ * Reset gpio99, int gpio125, AP/SLPI switch is 128. (Reset isn't used?).
+ */
+
+&qupv3_se2_i2c {
+ status = "ok";
+ qcom,clk-freq-out = <400000>;
+
+ ftm4_fts@49 {
+ status = "ok";
+ compatible = "stm,ftm4_fts";
+ reg = <0x49>;
+
+ pinctrl-names = "on_state", "off_state";
+ pinctrl-0 = <&ts_int_active &ts_reset_active>;
+ pinctrl-1 = <&ts_int_suspend>;
+
+ stm,switch_gpio= <&tlmm 128 0>;
+ stm,irq_gpio = <&tlmm 125 0>;
+ stm,irq_type = <0x2008>;
+ stm,num_lines = <32 16>;
+ stm,max_coords = <1440 2880>;
+ stm,regulator_dvdd = "pm8998_l14";
+ stm,regulator_avdd = "pm8998_l19";
+
+ stm,firmware_name = "ftm4_fw.ftb";
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-usbctherm.dtsi b/arch/arm64/boot/dts/google/sdm845-usbctherm.dtsi
new file mode 100644
index 0000000..d24be28
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-usbctherm.dtsi
@@ -0,0 +1,56 @@
+&pm8998_gpios {
+ usbc_therm_gpio {
+ usbc_therm_gpiocfg: usbc_therm_gpiocfg {
+ pins = "gpio11";
+ function="analog";
+ bias-high-impedance;
+ };
+ };
+};
+
+&pm8998_vadc {
+ pinctrl-names = "default";
+ pinctrl-0 = <&usbc_therm_gpiocfg>;
+
+ /* This node can't be disabled w/ status = "disable" */
+ chan@55 {
+ label = "usbc_therm";
+ reg = <0x55>;
+ qcom,decimation = <2>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,fast-avg-setup = <0>;
+ };
+};
+
+&pm8998_adc_tm {
+ chan@55 {
+ label = "usbc_therm";
+ reg = <0x55>;
+ qcom,pre-div-channel-scaling = <0>;
+ qcom,calibration-type = "ratiometric";
+ qcom,scale-function = <2>;
+ qcom,hw-settle-time = <2>;
+ qcom,btm-channel-number = <0x80>;
+ qcom,thermal-node;
+ };
+};
+
+&thermal_zones {
+ usbc-therm-adc {
+ polling-delay-passive = <0>;
+ polling-delay = <0>;
+ thermal-sensors = <&pm8998_adc_tm 0x55>;
+ thermal-governor = "user_space";
+
+ trips {
+ active-config0 {
+ temperature = <125000>;
+ hysteresis = <10000>;
+ type = "passive";
+ };
+ };
+ };
+};
diff --git a/arch/arm64/boot/dts/google/sdm845-v2-b1c1-devboard.dts b/arch/arm64/boot/dts/google/sdm845-v2-b1c1-devboard.dts
new file mode 100644
index 0000000..fcd9fa3
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-v2-b1c1-devboard.dts
@@ -0,0 +1,26 @@
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+
+/dts-v1/;
+
+#include "sdm845-b1c1-devboard-common.dtsi"
+#include "sdm845-b1c1-audio-cal.dtsi"
+
+/ {
+ model = "Google Inc. MSM sdm845 V2 Dev Board";
+ compatible = "google,b1c1-sdm845", "qcom,sdm845";
+ qcom,msm-id = <321 0x20000>;
+ qcom,board-id = <0x00010205 0>;
+};
+
+#include "sdm845-b1c1-haptics-drv2624-overlay.dtsi"
diff --git a/arch/arm64/boot/dts/google/sdm845-wcd.dtsi b/arch/arm64/boot/dts/google/sdm845-wcd.dtsi
new file mode 100644
index 0000000..aab91d6
--- /dev/null
+++ b/arch/arm64/boot/dts/google/sdm845-wcd.dtsi
@@ -0,0 +1,167 @@
+/* Copyright (c) 2016-2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+&slim_aud {
+ tavil_codec {
+ wcd: wcd_pinctrl@5 {
+ compatible = "qcom,wcd-pinctrl";
+ qcom,num-gpios = <5>;
+ gpio-controller;
+ #gpio-cells = <2>;
+
+ us_euro_sw_wcd_active: us_euro_sw_wcd_active {
+ mux {
+ pins = "gpio1";
+ };
+
+ config {
+ pins = "gpio1";
+ output-high;
+ };
+ };
+
+ us_euro_sw_wcd_sleep: us_euro_sw_wcd_sleep {
+ mux {
+ pins = "gpio1";
+ };
+
+ config {
+ pins = "gpio1";
+ output-low;
+ };
+ };
+
+ spkr_1_wcd_en_active: spkr_1_wcd_en_active {
+ mux {
+ pins = "gpio2";
+ };
+
+ config {
+ pins = "gpio2";
+ output-high;
+ };
+ };
+
+ spkr_1_wcd_en_sleep: spkr_1_wcd_en_sleep {
+ mux {
+ pins = "gpio2";
+ };
+
+ config {
+ pins = "gpio2";
+ input-enable;
+ };
+ };
+
+ spkr_2_wcd_en_active: spkr_2_sd_n_active {
+ mux {
+ pins = "gpio3";
+ };
+
+ config {
+ pins = "gpio3";
+ output-high;
+ };
+ };
+
+ spkr_2_wcd_en_sleep: spkr_2_sd_n_sleep {
+ mux {
+ pins = "gpio3";
+ };
+
+ config {
+ pins = "gpio3";
+ input-enable;
+ };
+ };
+
+ hph_en0_wcd_active: hph_en0_wcd_active {
+ mux {
+ pins = "gpio4";
+ };
+
+ config {
+ pins = "gpio4";
+ output-high;
+ };
+ };
+
+ hph_en0_wcd_sleep: hph_en0_wcd_sleep {
+ mux {
+ pins = "gpio4";
+ };
+
+ config {
+ pins = "gpio4";
+ output-low;
+ };
+ };
+
+ hph_en1_wcd_active: hph_en1_wcd_active {
+ mux {
+ pins = "gpio5";
+ };
+
+ config {
+ pins = "gpio5";
+ output-high;
+ };
+ };
+
+ hph_en1_wcd_sleep: hph_en1_wcd_sleep {
+ mux {
+ pins = "gpio5";
+ };
+
+ config {
+ pins = "gpio5";
+ output-low;
+ };
+ };
+ };
+
+ wsa_spkr_wcd_sd1: msm_cdc_pinctrll {
+ compatible = "qcom,msm-cdc-pinctrl";
+ pinctrl-names = "aud_active", "aud_sleep";
+ pinctrl-0 = <&spkr_1_wcd_en_active>;
+ pinctrl-1 = <&spkr_1_wcd_en_sleep>;
+ };
+
+ wsa_spkr_wcd_sd2: msm_cdc_pinctrlr {
+ compatible = "qcom,msm-cdc-pinctrl";
+ pinctrl-names = "aud_active", "aud_sleep";
+ pinctrl-0 = <&spkr_2_wcd_en_active>;
+ pinctrl-1 = <&spkr_2_wcd_en_sleep>;
+ };
+
+ tavil_us_euro_sw: msm_cdc_pinctrl_us_euro_sw {
+ compatible = "qcom,msm-cdc-pinctrl";
+ pinctrl-names = "aud_active", "aud_sleep";
+ pinctrl-0 = <&us_euro_sw_wcd_active>;
+ pinctrl-1 = <&us_euro_sw_wcd_sleep>;
+ };
+
+ tavil_hph_en0: msm_cdc_pinctrl_hph_en0 {
+ compatible = "qcom,msm-cdc-pinctrl";
+ pinctrl-names = "aud_active", "aud_sleep";
+ pinctrl-0 = <&hph_en0_wcd_active>;
+ pinctrl-1 = <&hph_en0_wcd_sleep>;
+ };
+
+ tavil_hph_en1: msm_cdc_pinctrl_hph_en1 {
+ compatible = "qcom,msm-cdc-pinctrl";
+ pinctrl-names = "aud_active", "aud_sleep";
+ pinctrl-0 = <&hph_en1_wcd_active>;
+ pinctrl-1 = <&hph_en1_wcd_sleep>;
+ };
+ };
+};
diff --git a/arch/arm64/boot/dts/qcom/Makefile b/arch/arm64/boot/dts/qcom/Makefile
index 4db459a..045e3bf 100644
--- a/arch/arm64/boot/dts/qcom/Makefile
+++ b/arch/arm64/boot/dts/qcom/Makefile
@@ -1,3 +1,6 @@
+# Do not include QC default DTBs for B1C1 builds.
+ifneq ($(CONFIG_BOARD_B1C1),y)
+
dtb-$(CONFIG_ARCH_QCOM) += apq8016-sbc.dtb msm8916-mtp.dtb
dtb-$(CONFIG_ARCH_QCOM) += msm8996-mtp.dtb
dtb-$(CONFIG_ARCH_QCOM) += apq8096-db820c.dtb
@@ -495,6 +498,8 @@
endif
+endif #CONFIG_BOARD_B1C1
+
always := $(dtb-y)
subdir-y := $(dts-dirs)
clean-files := *.dtb
diff --git a/arch/arm64/boot/dts/qcom/apq8016-sbc.dtsi b/arch/arm64/boot/dts/qcom/apq8016-sbc.dtsi
index bb062b5..601be61 100644
--- a/arch/arm64/boot/dts/qcom/apq8016-sbc.dtsi
+++ b/arch/arm64/boot/dts/qcom/apq8016-sbc.dtsi
@@ -170,7 +170,7 @@
led@6 {
label = "apq8016-sbc:blue:bt";
gpios = <&pm8916_mpps 3 GPIO_ACTIVE_HIGH>;
- linux,default-trigger = "bt";
+ linux,default-trigger = "bluetooth-power";
default-state = "off";
};
};
diff --git a/arch/arm64/boot/dts/qcom/msm-audio-lpass.dtsi b/arch/arm64/boot/dts/qcom/msm-audio-lpass.dtsi
index 13e5187..00acdd3 100644
--- a/arch/arm64/boot/dts/qcom/msm-audio-lpass.dtsi
+++ b/arch/arm64/boot/dts/qcom/msm-audio-lpass.dtsi
@@ -509,7 +509,7 @@
};
};
- qcom,msm-dai-tdm-quat-rx {
+ msm_dai_tdm_quat_rx: qcom,msm-dai-tdm-quat-rx {
compatible = "qcom,msm-dai-tdm";
qcom,msm-cpudai-tdm-group-id = <37168>;
qcom,msm-cpudai-tdm-group-num-ports = <1>;
@@ -528,7 +528,7 @@
};
};
- qcom,msm-dai-tdm-quat-tx {
+ msm_dai_tdm_quat_tx: qcom,msm-dai-tdm-quat-tx {
compatible = "qcom,msm-dai-tdm";
qcom,msm-cpudai-tdm-group-id = <37169>;
qcom,msm-cpudai-tdm-group-num-ports = <1>;
diff --git a/arch/arm64/boot/dts/qcom/msm8996.dtsi b/arch/arm64/boot/dts/qcom/msm8996.dtsi
index 338f82a..2c93de7 100644
--- a/arch/arm64/boot/dts/qcom/msm8996.dtsi
+++ b/arch/arm64/boot/dts/qcom/msm8996.dtsi
@@ -326,8 +326,8 @@
blsp2_spi5: spi@075ba000{
compatible = "qcom,spi-qup-v2.2.1";
reg = <0x075ba000 0x600>;
- interrupts = <GIC_SPI 107 IRQ_TYPE_LEVEL_HIGH>;
- clocks = <&gcc GCC_BLSP2_QUP5_SPI_APPS_CLK>,
+ interrupts = <GIC_SPI 106 IRQ_TYPE_LEVEL_HIGH>;
+ clocks = <&gcc GCC_BLSP2_QUP6_SPI_APPS_CLK>,
<&gcc GCC_BLSP2_AHB_CLK>;
clock-names = "core", "iface";
pinctrl-names = "default", "sleep";
diff --git a/arch/arm64/boot/dts/qcom/pmi8998.dtsi b/arch/arm64/boot/dts/qcom/pmi8998.dtsi
index 2f4b00e..50ed5d7 100644
--- a/arch/arm64/boot/dts/qcom/pmi8998.dtsi
+++ b/arch/arm64/boot/dts/qcom/pmi8998.dtsi
@@ -869,6 +869,7 @@
polling-delay-passive = <0>;
polling-delay = <0>;
thermal-sensors = <&pmi8998_tz>;
+ disable-thermal-zone;
trips {
pmi8998_trip0: pmi8998-trip0 {
diff --git a/arch/arm64/boot/dts/qcom/sdm670-camera.dtsi b/arch/arm64/boot/dts/qcom/sdm670-camera.dtsi
index 348ba6f..0cf0617 100644
--- a/arch/arm64/boot/dts/qcom/sdm670-camera.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm670-camera.dtsi
@@ -911,6 +911,9 @@
cam_ipe0: qcom,ipe0 {
cell-index = <0>;
compatible = "qcom,cam-ipe";
+ reg = <0xac87000 0x3000>;
+ reg-names = "ipe0_top";
+ reg-cam-base = <0x87000>;
regulator-names = "ipe0-vdd";
ipe0-vdd-supply = <&ipe_0_gdsc>;
clock-names = "ipe_0_ahb_clk",
@@ -938,6 +941,9 @@
cam_ipe1: qcom,ipe1 {
cell-index = <1>;
compatible = "qcom,cam-ipe";
+ reg = <0xac91000 0x3000>;
+ reg-names = "ipe1_top";
+ reg-cam-base = <0x91000>;
regulator-names = "ipe1-vdd";
ipe1-vdd-supply = <&ipe_1_gdsc>;
clock-names = "ipe_1_ahb_clk",
@@ -965,6 +971,9 @@
cam_bps: qcom,bps {
cell-index = <0>;
compatible = "qcom,cam-bps";
+ reg = <0xac6f000 0x3000>;
+ reg-names = "bps_top";
+ reg-cam-base = <0x6f000>;
regulator-names = "bps-vdd";
bps-vdd-supply = <&bps_gdsc>;
clock-names = "bps_ahb_clk",
diff --git a/arch/arm64/boot/dts/qcom/sdm845-670-usb-common.dtsi b/arch/arm64/boot/dts/qcom/sdm845-670-usb-common.dtsi
index c8b0e16..25f05c1 100644
--- a/arch/arm64/boot/dts/qcom/sdm845-670-usb-common.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm845-670-usb-common.dtsi
@@ -151,7 +151,7 @@
0x0a 0x184 /* PLL_LOCK_DELAY */
0x19 0xb4 /* PLL_DIGITAL_TIMERS_TWO */
0x40 0x194 /* PLL_BIAS_CONTROL_1 */
- 0x20 0x198 /* PLL_BIAS_CONTROL_2 */
+ 0x19 0x198 /* PLL_BIAS_CONTROL_2 */
0x21 0x214 /* PWR_CTRL2 */
0x00 0x220 /* IMP_CTRL1 */
0x58 0x224 /* IMP_CTRL2 */
@@ -431,6 +431,7 @@
0x22c /* QUSB2PHY_SQ_CTRL2 */
0x27c>; /* QUSB2PHY_DEBUG_CTRL1 */
+ qcom,qusb-bias-ctrl2-default = <0x20>;
qcom,qusb-phy-init-seq =
/* <value reg_offset> */
<0x23 0x210 /* PWR_CTRL1 */
diff --git a/arch/arm64/boot/dts/qcom/sdm845-camera.dtsi b/arch/arm64/boot/dts/qcom/sdm845-camera.dtsi
index 2e2de74..dd769eb 100644
--- a/arch/arm64/boot/dts/qcom/sdm845-camera.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm845-camera.dtsi
@@ -881,6 +881,9 @@
cam_ipe0: qcom,ipe0 {
cell-index = <0>;
compatible = "qcom,cam-ipe";
+ reg = <0xac87000 0x3000>;
+ reg-names = "ipe0_top";
+ reg-cam-base = <0x87000>;
regulator-names = "ipe0-vdd";
ipe0-vdd-supply = <&ipe_0_gdsc>;
clock-names = "ipe_0_ahb_clk",
@@ -908,6 +911,9 @@
cam_ipe1: qcom,ipe1 {
cell-index = <1>;
compatible = "qcom,cam-ipe";
+ reg = <0xac91000 0x3000>;
+ reg-names = "ipe1_top";
+ reg-cam-base = <0x91000>;
regulator-names = "ipe1-vdd";
ipe1-vdd-supply = <&ipe_1_gdsc>;
clock-names = "ipe_1_ahb_clk",
@@ -935,6 +941,9 @@
cam_bps: qcom,bps {
cell-index = <0>;
compatible = "qcom,cam-bps";
+ reg = <0xac6f000 0x3000>;
+ reg-names = "bps_top";
+ reg-cam-base = <0x6f000>;
regulator-names = "bps-vdd";
bps-vdd-supply = <&bps_gdsc>;
clock-names = "bps_ahb_clk",
diff --git a/arch/arm64/boot/dts/qcom/sdm845-ion.dtsi b/arch/arm64/boot/dts/qcom/sdm845-ion.dtsi
index 7d83184..d6f2920 100644
--- a/arch/arm64/boot/dts/qcom/sdm845-ion.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm845-ion.dtsi
@@ -11,7 +11,7 @@
*/
&soc {
- qcom,ion {
+ qcom_ion: qcom,ion {
compatible = "qcom,msm-ion";
#address-cells = <1>;
#size-cells = <0>;
diff --git a/arch/arm64/boot/dts/qcom/sdm845-pcie.dtsi b/arch/arm64/boot/dts/qcom/sdm845-pcie.dtsi
index c00c2646..d2c2137 100644
--- a/arch/arm64/boot/dts/qcom/sdm845-pcie.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm845-pcie.dtsi
@@ -475,8 +475,7 @@
RPMH_REGULATOR_LEVEL_NOM 0>;
qcom,l1-supported;
- qcom,l1ss-supported;
- qcom,aux-clk-sync;
+ /* BUG 67735049: L1ss doesn't work */
qcom,ep-latency = <10>;
diff --git a/arch/arm64/boot/dts/qcom/sdm845-pinctrl.dtsi b/arch/arm64/boot/dts/qcom/sdm845-pinctrl.dtsi
index 53f8084..9323cb3 100644
--- a/arch/arm64/boot/dts/qcom/sdm845-pinctrl.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm845-pinctrl.dtsi
@@ -1983,6 +1983,34 @@
};
};
+ qupv3_se3_2uart_pins: qupv3_se3_2uart_pins {
+ qupv3_se3_2uart_active: qupv3_se3_2uart_active {
+ mux {
+ pins = "gpio43", "gpio44";
+ function = "qup3";
+ };
+
+ config {
+ pins = "gpio43", "gpio44";
+ drive-strength = <2>;
+ bias-disable;
+ };
+ };
+
+ qupv3_se3_2uart_sleep: qupv3_se3_2uart_sleep {
+ mux {
+ pins = "gpio43", "gpio44";
+ function = "gpio";
+ };
+
+ config {
+ pins = "gpio43", "gpio44";
+ drive-strength = <2>;
+ bias-disable;
+ };
+ };
+ };
+
nfc {
nfc_int_active: nfc_int_active {
/* active state */
diff --git a/arch/arm64/boot/dts/qcom/sdm845-qupv3.dtsi b/arch/arm64/boot/dts/qcom/sdm845-qupv3.dtsi
index 097c3ac..2ab4ea20 100644
--- a/arch/arm64/boot/dts/qcom/sdm845-qupv3.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm845-qupv3.dtsi
@@ -423,6 +423,22 @@
/* 2-wire UART */
+ qupv3_se3_2uart: qcom,qup_uart@88c000 {
+ compatible = "qcom,msm-geni-console", "qcom,msm-geni-uart";
+ reg = <0x88c000 0x4000>;
+ reg-names = "se_phys";
+ clock-names = "se-clk", "m-ahb", "s-ahb";
+ clocks = <&clock_gcc GCC_QUPV3_WRAP0_S3_CLK>,
+ <&clock_gcc GCC_QUPV3_WRAP_0_M_AHB_CLK>,
+ <&clock_gcc GCC_QUPV3_WRAP_0_S_AHB_CLK>;
+ pinctrl-names = "default", "sleep";
+ pinctrl-0 = <&qupv3_se3_2uart_active>;
+ pinctrl-1 = <&qupv3_se3_2uart_sleep>;
+ interrupts = <GIC_SPI 604 0>;
+ qcom,wrapper-core = <&qupv3_0>;
+ status = "disabled";
+ };
+
/* Debug UART Instance for CDP/MTP platform */
qupv3_se9_2uart: qcom,qup_uart@0xa84000 {
compatible = "qcom,msm-geni-console";
diff --git a/arch/arm64/boot/dts/qcom/sdm845-regulator.dtsi b/arch/arm64/boot/dts/qcom/sdm845-regulator.dtsi
index 34beda4..b6bdf786 100644
--- a/arch/arm64/boot/dts/qcom/sdm845-regulator.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm845-regulator.dtsi
@@ -429,7 +429,7 @@
};
};
- rpmh-regulator-ldoa14 {
+ rpmh_regulator_ldoa14: rpmh-regulator-ldoa14 {
compatible = "qcom,rpmh-vrm-regulator";
mboxes = <&apps_rsc 0>;
qcom,resource-name = "ldoa14";
diff --git a/arch/arm64/boot/dts/qcom/sdm845-sde.dtsi b/arch/arm64/boot/dts/qcom/sdm845-sde.dtsi
index bfcebf6..f4905e6 100644
--- a/arch/arm64/boot/dts/qcom/sdm845-sde.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm845-sde.dtsi
@@ -258,33 +258,13 @@
/* data and reg bus scale settings */
qcom,sde-data-bus {
- qcom,msm-bus,name = "mdss_sde_mnoc";
+ qcom,msm-bus,name = "mdss_sde";
qcom,msm-bus,num-cases = <3>;
qcom,msm-bus,num-paths = <2>;
qcom,msm-bus,vectors-KBps =
- <22 773 0 0>, <23 773 0 0>,
- <22 773 0 6400000>, <23 773 0 6400000>,
- <22 773 0 6400000>, <23 773 0 6400000>;
- };
-
- qcom,sde-llcc-bus {
- qcom,msm-bus,name = "mdss_sde_llcc";
- qcom,msm-bus,num-cases = <3>;
- qcom,msm-bus,num-paths = <1>;
- qcom,msm-bus,vectors-KBps =
- <132 770 0 0>,
- <132 770 0 6400000>,
- <132 770 0 6400000>;
- };
-
- qcom,sde-ebi-bus {
- qcom,msm-bus,name = "mdss_sde_ebi";
- qcom,msm-bus,num-cases = <3>;
- qcom,msm-bus,num-paths = <1>;
- qcom,msm-bus,vectors-KBps =
- <129 512 0 0>,
- <129 512 0 6400000>,
- <129 512 0 6400000>;
+ <22 512 0 0>, <23 512 0 0>,
+ <22 512 0 6400000>, <23 512 0 6400000>,
+ <22 512 0 6400000>, <23 512 0 6400000>;
};
qcom,sde-reg-bus {
@@ -456,7 +436,6 @@
clock-names = "byte_clk", "byte_clk_rcg", "byte_intf_clk",
"pixel_clk", "pixel_clk_rcg",
"esc_clk";
- qcom,null-insertion-enabled;
qcom,ctrl-supply-entries {
#address-cells = <1>;
#size-cells = <0>;
diff --git a/arch/arm64/boot/dts/qcom/sdm845-v2.dtsi b/arch/arm64/boot/dts/qcom/sdm845-v2.dtsi
index c6ed0fd..418b48e 100644
--- a/arch/arm64/boot/dts/qcom/sdm845-v2.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm845-v2.dtsi
@@ -255,8 +255,7 @@
&devfreq_l3lat_0 {
qcom,core-dev-table =
- < 300000 300000000 >,
- < 480000 403200000 >,
+ < 300000 403200000 >,
< 652800 480000000 >,
< 748800 576000000 >,
< 902400 652800000 >,
@@ -299,9 +298,13 @@
&devfreq_cpufreq {
mincpubw-cpufreq {
+ cpu-to-dev-map-0 =
+ < 1708800 MHZ_TO_MBPS(200, 4) >,
+ < 1766400 MHZ_TO_MBPS(451, 4) >;
cpu-to-dev-map-4 =
< 1881600 MHZ_TO_MBPS(200, 4) >,
- < 2400000 MHZ_TO_MBPS(1017, 4) >;
+ < 2400000 MHZ_TO_MBPS(1017, 4) >,
+ < 2745600 MHZ_TO_MBPS(1804, 4) >;
};
};
@@ -410,23 +413,23 @@
&energy_costs {
CPU_COST_0: core-cost0 {
busy-cost-data = <
- 300000 12
- 403200 17
- 480000 21
- 576000 27
- 652800 31
- 748800 37
- 825600 42
- 902400 47
- 979200 52
- 1056000 57
- 1132800 62
- 1228800 70
- 1324800 78
- 1420800 89
- 1516800 103
- 1612800 122
- 1689600 141
+ 300000 21
+ 403200 26
+ 480000 31
+ 576000 36
+ 652800 41
+ 748800 46
+ 825600 53
+ 902400 59
+ 979200 64
+ 1056000 70
+ 1132800 79
+ 1228800 89
+ 1324800 99
+ 1420800 108
+ 1516800 118
+ 1612800 134
+ 1689600 143
1766400 160
>;
idle-cost-data = <
@@ -435,45 +438,43 @@
};
CPU_COST_1: core-cost1 {
busy-cost-data = <
- 300000 189
- 403200 523
- 480000 763
- 576000 1052
- 652800 1273
- 748800 1536
- 825600 1736
- 902400 1926
- 979200 2108
- 1056000 2284
- 1132800 2456
- 1209600 2628
- 1286400 2804
- 1363200 2992
- 1459200 3255
- 1536000 3499
- 1612800 3786
- 1689600 4128
- 1766400 4535
- 1843200 5019
- 1920000 5583
- 1996800 6226
- 2092800 7120
- 2169600 7876
- 2246400 8628
- 2323200 9344
- 2400000 10030
- 2476800 10806
- 2553600 12045
- 2649600 15686
- 2745600 25586
- 2764800 30000
- 2784000 35000
- 2803200 40000
- 2841600 50000
- 2956800 60000
+ 300000 90
+ 403200 113
+ 480000 135
+ 576000 157
+ 652800 175
+ 748800 198
+ 825600 210
+ 902400 233
+ 979200 263
+ 1056000 285
+ 1132800 322
+ 1209600 341
+ 1286400 379
+ 1363200 410
+ 1459200 462
+ 1536000 503
+ 1612800 542
+ 1689600 600
+ 1766400 649
+ 1843200 717
+ 1920000 813
+ 1996800 866
+ 2092800 1025
+ 2169600 1054
+ 2246400 1130
+ 2323200 1254
+ 2400000 1356
+ 2476800 1503
+ 2553600 1745
+ 2649600 2106
+ 2745600 2200
+ 2764800 2300
+ 2784000 2350
+ 2803200 2400
>;
idle-cost-data = <
- 100 80 60 40
+ 25 20 15 10
>;
};
CLUSTER_COST_0: cluster-cost0 {
diff --git a/arch/arm64/boot/dts/qcom/sdm845.dtsi b/arch/arm64/boot/dts/qcom/sdm845.dtsi
index c6144757..e4d87018 100644
--- a/arch/arm64/boot/dts/qcom/sdm845.dtsi
+++ b/arch/arm64/boot/dts/qcom/sdm845.dtsi
@@ -35,14 +35,11 @@
qcom,msm-id = <321 0x10000>;
interrupt-parent = <&pdc>;
- aliases {
+ aliases: aliases {
ufshc1 = &ufshc_mem; /* Embedded UFS slot */
pci-domain0 = &pcie0;
pci-domain1 = &pcie1;
sdhc2 = &sdhc_2; /* SDC2 SD card slot */
- };
-
- aliases {
serial0 = &qupv3_se9_2uart;
spi0 = &qupv3_se8_spi;
i2c0 = &qupv3_se10_i2c;
@@ -492,28 +489,7 @@
compatible = "simple-bus";
};
- firmware: firmware {
- android {
- compatible = "android,firmware";
- vbmeta {
- compatible = "android,vbmeta";
- parts = "vbmeta,boot,system,vendor,dtbo";
- };
-
- fstab {
- compatible = "android,fstab";
- vendor {
- compatible = "android,vendor";
- dev = "/dev/block/platform/soc/1d84000.ufshc/by-name/vendor";
- type = "ext4";
- mnt_flags = "ro,barrier=1,discard";
- fsmgr_flags = "wait,slotselect,avb";
- };
- };
- };
- };
-
- reserved-memory {
+ reserved_memory: reserved-memory {
#address-cells = <2>;
#size-cells = <2>;
ranges;
@@ -2493,6 +2469,7 @@
clock-frequency = <25000000>;
qcom,ipc-gpio = <&tlmm 121 0>;
qcom,finger-detect-gpio = <&pm8998_gpios 5 0>;
+ status = "disabled";
};
qcom_seecom: qseecom@86d00000 {
@@ -2677,7 +2654,6 @@
qcom,ipa-wdi2;
qcom,use-64-bit-dma-mask;
qcom,arm-smmu;
- qcom,smmu-fast-map;
qcom,bandwidth-vote-for-ipa;
qcom,msm-bus,name = "ipa";
qcom,msm-bus,num-cases = <5>;
@@ -2800,6 +2776,7 @@
ipa_smmu_ap: ipa_smmu_ap {
compatible = "qcom,ipa-smmu-ap-cb";
+ qcom,smmu-s1-bypass;
iommus = <&apps_smmu 0x720 0x0>;
qcom,iova-mapping = <0x20000000 0x40000000>;
qcom,additional-mapping =
@@ -2809,6 +2786,7 @@
ipa_smmu_wlan: ipa_smmu_wlan {
compatible = "qcom,ipa-smmu-wlan-cb";
+ qcom,smmu-s1-bypass;
iommus = <&apps_smmu 0x721 0x0>;
qcom,additional-mapping =
/* ipa-uc ram */
@@ -2817,6 +2795,7 @@
ipa_smmu_uc: ipa_smmu_uc {
compatible = "qcom,ipa-smmu-uc-cb";
+ qcom,smmu-s1-bypass;
iommus = <&apps_smmu 0x722 0x0>;
qcom,iova-mapping = <0x40000000 0x20000000>;
};
@@ -2976,7 +2955,6 @@
<DCC_READ 0x011360b0 1 0>,
<DCC_READ 0x0113e030 2 0>,
<DCC_READ 0x01141000 1 0>,
- <DCC_READ 0x01142028 1 0>,
<DCC_READ 0x01148058 4 0>,
<DCC_READ 0x01160410 3 0>,
<DCC_READ 0x011604a0 1 0>,
@@ -2987,7 +2965,6 @@
<DCC_READ 0x011b6044 4 0>,
<DCC_READ 0x011be030 2 0>,
<DCC_READ 0x011c1000 1 0>,
- <DCC_READ 0x011c2028 1 0>,
<DCC_READ 0x011c8058 4 0>,
<DCC_READ 0x011e0410 3 0>,
<DCC_READ 0x011e04a0 1 0>,
@@ -2999,7 +2976,6 @@
<DCC_READ 0x012360B0 1 0>,
<DCC_READ 0x0123E030 2 0>,
<DCC_READ 0x01241000 1 0>,
- <DCC_READ 0x01242028 1 0>,
<DCC_READ 0x01248058 4 0>,
<DCC_READ 0x01260410 3 0>,
<DCC_READ 0x012604a0 1 0>,
@@ -3012,7 +2988,6 @@
<DCC_READ 0x012b60b0 1 0>,
<DCC_READ 0x012be030 2 0>,
<DCC_READ 0x012c1000 1 0>,
- <DCC_READ 0x012c2028 1 0>,
<DCC_READ 0x012c8058 4 0>,
<DCC_READ 0x012e0410 3 0>,
<DCC_READ 0x012e04a0 1 0>,
@@ -3021,7 +2996,6 @@
<DCC_READ 0x012e6418 1 0>,
<DCC_READ 0x01380900 8 0>,
<DCC_READ 0x01380d00 5 0>,
- <DCC_READ 0x01350110 4 0>,
<DCC_READ 0x01430280 1 0>,
<DCC_READ 0x01430288 1 0>,
<DCC_READ 0x0143028c 7 0>,
@@ -3030,7 +3004,6 @@
<DCC_READ 0x011360b0 1 0>,
<DCC_READ 0x0113e030 2 0>,
<DCC_READ 0x01141000 1 0>,
- <DCC_READ 0x01142028 1 0>,
<DCC_READ 0x01148058 4 0>,
<DCC_READ 0x01160410 3 0>,
<DCC_READ 0x011604a0 1 0>,
@@ -3041,7 +3014,6 @@
<DCC_READ 0x011b6044 4 0>,
<DCC_READ 0x011be030 2 0>,
<DCC_READ 0x011c1000 1 0>,
- <DCC_READ 0x011c2028 1 0>,
<DCC_READ 0x011c8058 4 0>,
<DCC_READ 0x011e0410 3 0>,
<DCC_READ 0x011e04a0 1 0>,
@@ -3053,7 +3025,6 @@
<DCC_READ 0x012360b0 1 0>,
<DCC_READ 0x0123e030 2 0>,
<DCC_READ 0x01241000 1 0>,
- <DCC_READ 0x01242028 1 0>,
<DCC_READ 0x01248058 4 0>,
<DCC_READ 0x01260410 3 0>,
<DCC_READ 0x012604a0 1 0>,
@@ -3066,7 +3037,6 @@
<DCC_READ 0x012b60b0 1 0>,
<DCC_READ 0x012be030 2 0>,
<DCC_READ 0x012C1000 1 0>,
- <DCC_READ 0x012C2028 1 0>,
<DCC_READ 0x012C8058 4 0>,
<DCC_READ 0x012e0410 3 0>,
<DCC_READ 0x012e04a0 1 0>,
@@ -3075,7 +3045,6 @@
<DCC_READ 0x012e6418 1 0>,
<DCC_READ 0x01380900 8 0>,
<DCC_READ 0x01380d00 5 0>,
- <DCC_READ 0x01350110 4 0>,
<DCC_READ 0x01430280 1 0>,
<DCC_READ 0x01430288 1 0>,
<DCC_READ 0x0143028c 7 0>,
@@ -3117,6 +3086,7 @@
vdd-3.3-ch0-supply = <&pm8998_l25>;
qcom,vdd-0.8-cx-mx-config = <800000 800000>;
qcom,vdd-3.3-ch0-config = <3104000 3312000>;
+ qcom,smmu-s1-bypass;
};
qmi-tmd-devices {
diff --git a/arch/arm64/configs/b1c1_defconfig b/arch/arm64/configs/b1c1_defconfig
new file mode 100644
index 0000000..b0de454
--- /dev/null
+++ b/arch/arm64/configs/b1c1_defconfig
@@ -0,0 +1,663 @@
+CONFIG_KERNEL_LZ4=y
+# CONFIG_FHANDLE is not set
+CONFIG_AUDIT=y
+# CONFIG_AUDITSYSCALL is not set
+CONFIG_NO_HZ=y
+CONFIG_HIGH_RES_TIMERS=y
+CONFIG_IRQ_TIME_ACCOUNTING=y
+CONFIG_SCHED_WALT=y
+CONFIG_TASKSTATS=y
+CONFIG_TASK_XACCT=y
+CONFIG_TASK_IO_ACCOUNTING=y
+CONFIG_PSI=y
+CONFIG_RCU_EXPERT=y
+CONFIG_RCU_FAST_NO_HZ=y
+CONFIG_RCU_NOCB_CPU=y
+CONFIG_RCU_NOCB_CPU_ALL=y
+CONFIG_IKCONFIG=y
+CONFIG_IKCONFIG_PROC=y
+CONFIG_LOG_CPU_MAX_BUF_SHIFT=17
+CONFIG_CGROUP_FREEZER=y
+CONFIG_CPUSETS=y
+CONFIG_CGROUP_CPUACCT=y
+CONFIG_CGROUP_SCHEDTUNE=y
+CONFIG_MEMCG=y
+CONFIG_MEMCG_SWAP=y
+CONFIG_RT_GROUP_SCHED=y
+CONFIG_CGROUP_BPF=y
+CONFIG_SCHED_CORE_CTL=y
+CONFIG_NAMESPACES=y
+# CONFIG_UTS_NS is not set
+# CONFIG_PID_NS is not set
+CONFIG_SCHED_AUTOGROUP=y
+CONFIG_SCHED_TUNE=y
+CONFIG_DEFAULT_USE_ENERGY_AWARE=y
+CONFIG_BLK_DEV_INITRD=y
+# CONFIG_RD_BZIP2 is not set
+# CONFIG_RD_LZMA is not set
+# CONFIG_RD_XZ is not set
+# CONFIG_RD_LZO is not set
+# CONFIG_RD_LZ4 is not set
+# CONFIG_SYSFS_SYSCALL is not set
+CONFIG_KALLSYMS_ALL=y
+CONFIG_BPF_SYSCALL=y
+# CONFIG_MEMBARRIER is not set
+CONFIG_EMBEDDED=y
+# CONFIG_COMPAT_BRK is not set
+CONFIG_PROFILING=y
+CONFIG_KPROBES=y
+CONFIG_JUMP_LABEL=y
+CONFIG_CC_STACKPROTECTOR_STRONG=y
+CONFIG_LTO_CLANG=y
+CONFIG_CFI_CLANG=y
+CONFIG_SHADOW_CALL_STACK=y
+CONFIG_ARCH_MMAP_RND_COMPAT_BITS=16
+CONFIG_MODULES=y
+CONFIG_MODULE_UNLOAD=y
+CONFIG_MODULE_FORCE_UNLOAD=y
+CONFIG_MODVERSIONS=y
+CONFIG_MODULE_SIG=y
+CONFIG_MODULE_SIG_FORCE=y
+CONFIG_MODULE_SIG_SHA512=y
+# CONFIG_BLK_DEV_BSG is not set
+CONFIG_PARTITION_ADVANCED=y
+CONFIG_ARCH_QCOM=y
+CONFIG_BOARD_B1C1=y
+CONFIG_PCI=y
+CONFIG_PCI_MSM=y
+CONFIG_SCHED_MC=y
+CONFIG_NR_CPUS=8
+CONFIG_PREEMPT=y
+CONFIG_HZ_300=y
+CONFIG_PROCESS_RECLAIM=y
+CONFIG_CMA=y
+CONFIG_ZSMALLOC=y
+CONFIG_MM_EVENT_STAT=y
+CONFIG_BALANCE_ANON_FILE_RECLAIM=y
+CONFIG_IDLE_PAGE_TRACKING=y
+CONFIG_SECCOMP=y
+CONFIG_HARDEN_BRANCH_PREDICTOR=y
+CONFIG_PSCI_BP_HARDENING=y
+CONFIG_ARMV8_DEPRECATED=y
+CONFIG_SWP_EMULATION=y
+CONFIG_CP15_BARRIER_EMULATION=y
+CONFIG_SETEND_EMULATION=y
+# CONFIG_ARM64_VHE is not set
+CONFIG_RANDOMIZE_BASE=y
+# CONFIG_EFI is not set
+CONFIG_BUILD_ARM64_APPENDED_DTB_IMAGE=y
+CONFIG_BUILD_ARM64_DTC="dtc"
+CONFIG_BUILD_ARM64_DTC_FLAGS="-q"
+CONFIG_BUILD_ARM64_DT_OVERLAY=y
+# CONFIG_CORE_DUMP_DEFAULT_ELF_HEADERS is not set
+CONFIG_COMPAT=y
+CONFIG_COMPAT_VDSO=y
+CONFIG_PM_AUTOSLEEP=y
+CONFIG_PM_WAKELOCKS=y
+CONFIG_PM_WAKELOCKS_LIMIT=0
+# CONFIG_PM_WAKELOCKS_GC is not set
+CONFIG_CPU_IDLE=y
+CONFIG_CPU_FREQ=y
+CONFIG_CPU_FREQ_STAT=y
+CONFIG_CPU_FREQ_GOV_POWERSAVE=y
+CONFIG_CPU_FREQ_GOV_USERSPACE=y
+CONFIG_CPU_FREQ_GOV_SCHEDUTIL=y
+CONFIG_NET=y
+CONFIG_PACKET=y
+CONFIG_UNIX=y
+CONFIG_XFRM_USER=y
+CONFIG_XFRM_STATISTICS=y
+CONFIG_NET_KEY=y
+CONFIG_INET=y
+CONFIG_IP_MULTICAST=y
+CONFIG_IP_ADVANCED_ROUTER=y
+CONFIG_IP_MULTIPLE_TABLES=y
+CONFIG_IP_ROUTE_VERBOSE=y
+CONFIG_IP_PNP=y
+CONFIG_IP_PNP_DHCP=y
+CONFIG_NET_IPVTI=y
+CONFIG_INET_AH=y
+CONFIG_INET_ESP=y
+CONFIG_INET_IPCOMP=y
+CONFIG_INET_UDP_DIAG=y
+CONFIG_INET_DIAG_DESTROY=y
+CONFIG_IPV6_ROUTER_PREF=y
+CONFIG_IPV6_ROUTE_INFO=y
+CONFIG_IPV6_OPTIMISTIC_DAD=y
+CONFIG_INET6_AH=y
+CONFIG_INET6_ESP=y
+CONFIG_INET6_IPCOMP=y
+CONFIG_IPV6_MIP6=y
+CONFIG_IPV6_VTI=y
+CONFIG_IPV6_MULTIPLE_TABLES=y
+CONFIG_IPV6_SUBTREES=y
+CONFIG_NETFILTER=y
+# CONFIG_BRIDGE_NETFILTER is not set
+CONFIG_NF_CONNTRACK=y
+CONFIG_NF_CONNTRACK_SECMARK=y
+CONFIG_NF_CONNTRACK_EVENTS=y
+CONFIG_NF_CT_PROTO_DCCP=y
+CONFIG_NF_CT_PROTO_SCTP=y
+CONFIG_NF_CT_PROTO_UDPLITE=y
+CONFIG_NF_CONNTRACK_AMANDA=y
+CONFIG_NF_CONNTRACK_FTP=y
+CONFIG_NF_CONNTRACK_H323=y
+CONFIG_NF_CONNTRACK_IRC=y
+CONFIG_NF_CONNTRACK_NETBIOS_NS=y
+CONFIG_NF_CONNTRACK_PPTP=y
+CONFIG_NF_CONNTRACK_SANE=y
+CONFIG_NF_CONNTRACK_TFTP=y
+CONFIG_NF_CT_NETLINK=y
+CONFIG_NETFILTER_XT_TARGET_CLASSIFY=y
+CONFIG_NETFILTER_XT_TARGET_CONNMARK=y
+CONFIG_NETFILTER_XT_TARGET_CONNSECMARK=y
+CONFIG_NETFILTER_XT_TARGET_IDLETIMER=y
+CONFIG_NETFILTER_XT_TARGET_HARDIDLETIMER=y
+CONFIG_NETFILTER_XT_TARGET_LOG=y
+CONFIG_NETFILTER_XT_TARGET_MARK=y
+CONFIG_NETFILTER_XT_TARGET_NFLOG=y
+CONFIG_NETFILTER_XT_TARGET_NFQUEUE=y
+CONFIG_NETFILTER_XT_TARGET_NOTRACK=y
+CONFIG_NETFILTER_XT_TARGET_TEE=y
+CONFIG_NETFILTER_XT_TARGET_TPROXY=y
+CONFIG_NETFILTER_XT_TARGET_TRACE=y
+CONFIG_NETFILTER_XT_TARGET_SECMARK=y
+CONFIG_NETFILTER_XT_TARGET_TCPMSS=y
+CONFIG_NETFILTER_XT_MATCH_BPF=y
+CONFIG_NETFILTER_XT_MATCH_COMMENT=y
+CONFIG_NETFILTER_XT_MATCH_CONNLIMIT=y
+CONFIG_NETFILTER_XT_MATCH_CONNMARK=y
+CONFIG_NETFILTER_XT_MATCH_CONNTRACK=y
+CONFIG_NETFILTER_XT_MATCH_DSCP=y
+CONFIG_NETFILTER_XT_MATCH_ESP=y
+CONFIG_NETFILTER_XT_MATCH_HASHLIMIT=y
+CONFIG_NETFILTER_XT_MATCH_HELPER=y
+CONFIG_NETFILTER_XT_MATCH_IPRANGE=y
+# CONFIG_NETFILTER_XT_MATCH_L2TP is not set
+CONFIG_NETFILTER_XT_MATCH_LENGTH=y
+CONFIG_NETFILTER_XT_MATCH_LIMIT=y
+CONFIG_NETFILTER_XT_MATCH_MAC=y
+CONFIG_NETFILTER_XT_MATCH_MARK=y
+CONFIG_NETFILTER_XT_MATCH_MULTIPORT=y
+CONFIG_NETFILTER_XT_MATCH_POLICY=y
+CONFIG_NETFILTER_XT_MATCH_PKTTYPE=y
+CONFIG_NETFILTER_XT_MATCH_QTAGUID=y
+CONFIG_NETFILTER_XT_MATCH_QUOTA=y
+CONFIG_NETFILTER_XT_MATCH_QUOTA2=y
+CONFIG_NETFILTER_XT_MATCH_SOCKET=y
+CONFIG_NETFILTER_XT_MATCH_STATE=y
+CONFIG_NETFILTER_XT_MATCH_STATISTIC=y
+CONFIG_NETFILTER_XT_MATCH_STRING=y
+CONFIG_NETFILTER_XT_MATCH_TIME=y
+CONFIG_NETFILTER_XT_MATCH_U32=y
+CONFIG_NF_CONNTRACK_IPV4=y
+CONFIG_IP_NF_IPTABLES=y
+CONFIG_IP_NF_MATCH_AH=y
+CONFIG_IP_NF_MATCH_ECN=y
+CONFIG_IP_NF_MATCH_RPFILTER=y
+CONFIG_IP_NF_MATCH_TTL=y
+CONFIG_IP_NF_FILTER=y
+CONFIG_IP_NF_TARGET_REJECT=y
+CONFIG_IP_NF_NAT=y
+CONFIG_IP_NF_TARGET_MASQUERADE=y
+CONFIG_IP_NF_TARGET_NETMAP=y
+CONFIG_IP_NF_TARGET_REDIRECT=y
+CONFIG_IP_NF_MANGLE=y
+CONFIG_IP_NF_RAW=y
+CONFIG_IP_NF_SECURITY=y
+CONFIG_IP_NF_ARPTABLES=y
+CONFIG_IP_NF_ARPFILTER=y
+CONFIG_IP_NF_ARP_MANGLE=y
+CONFIG_NF_CONNTRACK_IPV6=y
+CONFIG_IP6_NF_IPTABLES=y
+CONFIG_IP6_NF_IPTABLES_128=y
+CONFIG_IP6_NF_MATCH_RPFILTER=y
+CONFIG_IP6_NF_FILTER=y
+CONFIG_IP6_NF_TARGET_REJECT=y
+CONFIG_IP6_NF_MANGLE=y
+CONFIG_IP6_NF_RAW=y
+CONFIG_BRIDGE_NF_EBTABLES=y
+CONFIG_BRIDGE_EBT_BROUTE=y
+CONFIG_L2TP=y
+CONFIG_L2TP_V3=y
+CONFIG_L2TP_IP=y
+CONFIG_L2TP_ETH=y
+CONFIG_BRIDGE=y
+CONFIG_NET_SCHED=y
+CONFIG_NET_SCH_HTB=y
+CONFIG_NET_SCH_PRIO=y
+CONFIG_NET_SCH_MULTIQ=y
+CONFIG_NET_SCH_INGRESS=y
+CONFIG_NET_CLS_FW=y
+CONFIG_NET_CLS_U32=y
+CONFIG_NET_CLS_BPF=y
+CONFIG_NET_EMATCH=y
+CONFIG_NET_EMATCH_CMP=y
+CONFIG_NET_EMATCH_NBYTE=y
+CONFIG_NET_EMATCH_U32=y
+CONFIG_NET_EMATCH_META=y
+CONFIG_NET_EMATCH_TEXT=y
+CONFIG_NET_CLS_ACT=y
+CONFIG_NET_ACT_GACT=y
+CONFIG_NET_ACT_MIRRED=y
+CONFIG_NET_ACT_SKBEDIT=y
+CONFIG_RMNET_DATA=y
+CONFIG_RMNET_DATA_FC=y
+CONFIG_SOCKEV_NLMCAST=y
+CONFIG_BT=y
+CONFIG_MSM_BT_POWER=y
+CONFIG_CFG80211=y
+CONFIG_CFG80211_CERTIFICATION_ONUS=y
+CONFIG_CFG80211_REG_CELLULAR_HINTS=y
+CONFIG_CFG80211_INTERNAL_REGDB=y
+# CONFIG_CFG80211_CRDA_SUPPORT is not set
+CONFIG_RFKILL=y
+CONFIG_NFC_NQ=y
+CONFIG_NFC_NQ_PN81A=y
+CONFIG_IPC_ROUTER=y
+CONFIG_IPC_ROUTER_SECURITY=y
+CONFIG_FW_LOADER_USER_HELPER_FALLBACK=y
+CONFIG_DMA_CMA=y
+CONFIG_ZRAM=y
+CONFIG_ZRAM_WRITEBACK=y
+CONFIG_BLK_DEV_LOOP=y
+CONFIG_BLK_DEV_RAM=y
+CONFIG_BLK_DEV_RAM_SIZE=8192
+CONFIG_QSEECOM=y
+CONFIG_UID_SYS_STATS=y
+CONFIG_MEMORY_STATE_TIME=y
+CONFIG_CITADEL=y
+CONFIG_GOOGLE_EASEL=y
+CONFIG_GOOGLE_EASEL_AP=y
+CONFIG_CS40L2X_HAPTIC=y
+CONFIG_ACCESS_RAMOOPS=y
+CONFIG_MNH_PCIE_HOST=y
+CONFIG_MNH_SM_HOST=y
+CONFIG_MNH_THERMAL_HOST=y
+CONFIG_MNH_PCIE_MULTIPLE_MSI=y
+CONFIG_MNH_SIG=y
+CONFIG_SCSI=y
+CONFIG_BLK_DEV_SD=y
+CONFIG_CHR_DEV_SG=y
+CONFIG_SCSI_CONSTANTS=y
+CONFIG_SCSI_LOGGING=y
+CONFIG_SCSI_SCAN_ASYNC=y
+CONFIG_SCSI_UFSHCD=y
+CONFIG_SCSI_UFSHCD_PLATFORM=y
+CONFIG_SCSI_UFS_QCOM=y
+CONFIG_SCSI_UFS_QCOM_ICE=y
+CONFIG_SCSI_UFSHCD_CMD_LOGGING=y
+CONFIG_MD=y
+CONFIG_BLK_DEV_DM=y
+CONFIG_DM_CRYPT=y
+CONFIG_DM_DEFAULT_KEY=y
+CONFIG_DM_REQ_CRYPT=y
+CONFIG_DM_UEVENT=y
+CONFIG_DM_VERITY=y
+CONFIG_DM_VERITY_FEC=y
+CONFIG_NETDEVICES=y
+CONFIG_BONDING=y
+CONFIG_DUMMY=y
+CONFIG_TUN=y
+# CONFIG_ETHERNET is not set
+CONFIG_PPP=y
+CONFIG_PPP_BSDCOMP=y
+CONFIG_PPP_DEFLATE=y
+CONFIG_PPP_FILTER=y
+CONFIG_PPP_MPPE=y
+CONFIG_PPP_MULTILINK=y
+CONFIG_PPPOE=y
+CONFIG_PPPOL2TP=y
+CONFIG_PPPOLAC=y
+CONFIG_PPPOPNS=y
+CONFIG_PPP_ASYNC=y
+CONFIG_PPP_SYNC_TTY=y
+CONFIG_USB_RTL8150=y
+CONFIG_USB_RTL8152=y
+CONFIG_USB_USBNET=y
+# CONFIG_USB_NET_NET1080 is not set
+# CONFIG_USB_NET_CDC_SUBSET is not set
+# CONFIG_USB_NET_ZAURUS is not set
+CONFIG_WCNSS_MEM_PRE_ALLOC=y
+CONFIG_CLD_LL_CORE=y
+CONFIG_WLAN_FEATURE_SARV1_TO_SARV2=y
+CONFIG_CNSS_GENL=y
+CONFIG_INPUT_EVDEV=y
+CONFIG_INPUT_KEYDEBUG=y
+CONFIG_KEYBOARD_GPIO=y
+# CONFIG_INPUT_MOUSE is not set
+CONFIG_INPUT_JOYSTICK=y
+CONFIG_JOYSTICK_XPAD=y
+CONFIG_JOYSTICK_XPAD_FF=y
+CONFIG_JOYSTICK_XPAD_LEDS=y
+CONFIG_INPUT_TOUCHSCREEN=y
+CONFIG_TOUCHSCREEN_FTS=m
+CONFIG_TOUCHSCREEN_FTM4=y
+CONFIG_TOUCHSCREEN_SEC_TS=m
+CONFIG_TOUCHSCREEN_TBN=y
+CONFIG_INPUT_MISC=y
+CONFIG_INPUT_QPNP_POWER_ON=y
+CONFIG_INPUT_KEYCHORD=y
+CONFIG_INPUT_UINPUT=y
+CONFIG_INPUT_DRV2624_HAPTICS=y
+CONFIG_INPUT_VD6281=y
+CONFIG_FPR_FPC=y
+# CONFIG_SERIO_SERPORT is not set
+# CONFIG_VT is not set
+# CONFIG_LEGACY_PTYS is not set
+# CONFIG_DEVMEM is not set
+# CONFIG_DEVKMEM is not set
+CONFIG_SERIAL_MSM_GENI=y
+CONFIG_SERIAL_MSM_GENI_CONSOLE=y
+CONFIG_DIAG_CHAR=y
+CONFIG_HW_RANDOM=y
+CONFIG_HW_RANDOM_MSM_LEGACY=y
+# CONFIG_DEVPORT is not set
+CONFIG_MSM_ADSPRPC=y
+CONFIG_I2C_CHARDEV=y
+CONFIG_I2C_QCOM_GENI=y
+CONFIG_SPI=y
+CONFIG_SPI_QUP=y
+CONFIG_SPI_QCOM_GENI=y
+CONFIG_SPI_SPIDEV=y
+CONFIG_SLIMBUS_MSM_NGD=y
+CONFIG_SPMI=y
+CONFIG_PINCTRL_SDM845=y
+CONFIG_PINCTRL_QCOM_SPMI_PMIC=y
+CONFIG_GPIOLIB=y
+CONFIG_GPIO_SYSFS=y
+CONFIG_POWER_RESET_QCOM=y
+CONFIG_QCOM_DLOAD_MODE=y
+CONFIG_POWER_RESET_XGENE=y
+CONFIG_POWER_RESET_SYSCON=y
+CONFIG_BATTERY_MAX1720X=y
+CONFIG_GOOGLE_CHARGER=y
+CONFIG_USB_OVERHEAT_MITIGATION=y
+CONFIG_QPNP_FG_GEN3=y
+# CONFIG_QPNP_FG_GEN3_LEGACY_CYCLE_COUNT is not set
+CONFIG_SMB1355_SLAVE_CHARGER=y
+CONFIG_QPNP_SMB2=y
+CONFIG_CHARGER_P9221=y
+CONFIG_SENSORS_QPNP_ADC_VOLTAGE=y
+CONFIG_THERMAL=y
+CONFIG_THERMAL_WRITABLE_TRIPS=y
+CONFIG_THERMAL_DEFAULT_GOV_USER_SPACE=y
+CONFIG_THERMAL_GOV_STEP_WISE=y
+CONFIG_THERMAL_GOV_LOW_LIMITS=y
+CONFIG_CPU_THERMAL=y
+CONFIG_DEVFREQ_THERMAL=y
+CONFIG_THERMAL_EMULATION=y
+CONFIG_QCOM_SPMI_TEMP_ALARM=y
+CONFIG_THERMAL_QPNP=y
+CONFIG_THERMAL_QPNP_ADC_TM=y
+CONFIG_THERMAL_TSENS=y
+CONFIG_MSM_BCL_PERIPHERAL_CTL=y
+CONFIG_QTI_THERMAL_LIMITS_DCVS=y
+CONFIG_QTI_VIRTUAL_SENSOR=y
+CONFIG_QTI_AOP_REG_COOLING_DEVICE=y
+CONFIG_QTI_QMI_COOLING_DEVICE=y
+CONFIG_REGULATOR_COOLING_DEVICE=y
+CONFIG_MFD_I2C_PMIC=y
+CONFIG_MFD_SPMI_PMIC=y
+CONFIG_REGULATOR_FIXED_VOLTAGE=y
+CONFIG_REGULATOR_PROXY_CONSUMER=y
+CONFIG_REGULATOR_QPNP_LABIBB=y
+CONFIG_REGULATOR_QPNP=y
+CONFIG_REGULATOR_REFGEN=y
+CONFIG_REGULATOR_RPMH=y
+CONFIG_REGULATOR_STUB=y
+CONFIG_MEDIA_SUPPORT=y
+CONFIG_MEDIA_CAMERA_SUPPORT=y
+CONFIG_MEDIA_DIGITAL_TV_SUPPORT=y
+CONFIG_MEDIA_CONTROLLER=y
+CONFIG_VIDEO_V4L2_SUBDEV_API=y
+CONFIG_VIDEO_FIXED_MINOR_RANGES=y
+CONFIG_V4L_PLATFORM_DRIVERS=y
+CONFIG_SPECTRA_CAMERA=y
+CONFIG_MSM_VIDC_V4L2=y
+CONFIG_MSM_VIDC_GOVERNORS=y
+CONFIG_MSM_SDE_ROTATOR=y
+CONFIG_CAMERA_FW_UPDATE=y
+CONFIG_QCOM_KGSL=y
+CONFIG_DRM=y
+CONFIG_DRM_SDE_RSC=y
+CONFIG_FB_VIRTUAL=y
+CONFIG_BACKLIGHT_LCD_SUPPORT=y
+# CONFIG_LCD_CLASS_DEVICE is not set
+CONFIG_BACKLIGHT_CLASS_DEVICE=y
+CONFIG_SOUND=y
+CONFIG_SND=y
+CONFIG_SND_DYNAMIC_MINORS=y
+CONFIG_SND_USB_AUDIO=y
+CONFIG_SND_USB_AUDIO_QMI=y
+CONFIG_SND_SOC=y
+CONFIG_SND_SOC_MAX98927=y
+CONFIG_UHID=y
+CONFIG_HID_APPLE=y
+CONFIG_HID_DRAGONRISE=y
+CONFIG_HID_ELECOM=y
+CONFIG_HID_MAGICMOUSE=y
+CONFIG_HID_MICROSOFT=y
+CONFIG_HID_MULTITOUCH=y
+CONFIG_HID_PANTHERLORD=y
+CONFIG_PANTHERLORD_FF=y
+CONFIG_HID_PLANTRONICS=y
+CONFIG_HID_SONY=y
+CONFIG_SONY_FF=y
+CONFIG_HID_GREENASIA=y
+CONFIG_GREENASIA_FF=y
+CONFIG_HID_WIIMOTE=y
+CONFIG_USB_HIDDEV=y
+CONFIG_USB_XHCI_HCD=y
+CONFIG_USB_STORAGE=y
+CONFIG_USB_DWC3=y
+CONFIG_USB_DWC3_MSM=y
+CONFIG_USB_ISP1760=y
+CONFIG_USB_ISP1760_HOST_ROLE=y
+CONFIG_USB_PD_ENGINE=y
+CONFIG_QPNP_USB_PDPHY=y
+CONFIG_USB_EHSET_TEST_FIXTURE=y
+CONFIG_USB_LINK_LAYER_TEST=y
+CONFIG_NOP_USB_XCEIV=y
+CONFIG_DUAL_ROLE_USB_INTF=y
+CONFIG_USB_MSM_SSPHY_QMP=y
+CONFIG_MSM_QUSB_PHY=y
+CONFIG_USB_GADGET=y
+CONFIG_USB_GADGET_VBUS_DRAW=500
+CONFIG_USB_CONFIGFS=y
+CONFIG_USB_CONFIGFS_MASS_STORAGE=y
+CONFIG_USB_CONFIGFS_F_FS=y
+CONFIG_USB_CONFIGFS_F_MTP=y
+CONFIG_USB_CONFIGFS_F_PTP=y
+CONFIG_USB_CONFIGFS_F_ACC=y
+CONFIG_USB_CONFIGFS_F_AUDIO_SRC=y
+CONFIG_USB_CONFIGFS_UEVENT=y
+CONFIG_USB_CONFIGFS_F_MIDI=y
+CONFIG_USB_CONFIGFS_F_HID=y
+CONFIG_USB_CONFIGFS_F_DIAG=y
+CONFIG_USB_CONFIGFS_F_CDEV=y
+CONFIG_USB_CONFIGFS_F_CCID=y
+CONFIG_USB_CONFIGFS_F_GSI=y
+CONFIG_USB_CONFIGFS_F_QDSS=y
+CONFIG_NEW_LEDS=y
+CONFIG_LEDS_CLASS=y
+CONFIG_LEDS_QPNP=y
+CONFIG_LEDS_QPNP_FLASH_V2=y
+CONFIG_LEDS_QPNP_WLED=y
+CONFIG_LEDS_TRIGGERS=y
+CONFIG_LEDS_TRIGGER_TRANSIENT=y
+CONFIG_EDAC=y
+CONFIG_EDAC_MM_EDAC=y
+CONFIG_EDAC_KRYO3XX_ARM64=y
+CONFIG_EDAC_KRYO3XX_ARM64_PANIC_ON_UE=y
+CONFIG_RTC_CLASS=y
+CONFIG_RTC_DRV_QPNP=y
+CONFIG_DMADEVICES=y
+CONFIG_QCOM_GPI_DMA=y
+CONFIG_UIO=y
+CONFIG_UIO_MSM_SHAREDMEM=y
+CONFIG_STAGING=y
+CONFIG_ASHMEM=y
+CONFIG_ION=y
+CONFIG_ION_MSM=y
+CONFIG_S2MPB04=y
+CONFIG_GOOGLE_WLAN_MAC=y
+CONFIG_GSI=y
+CONFIG_IPA3=y
+CONFIG_RMNET_IPA3=y
+CONFIG_RNDIS_IPA=y
+CONFIG_IPA_UT=y
+CONFIG_SPS=y
+CONFIG_SPS_SUPPORT_NDP_BAM=y
+CONFIG_QPNP_COINCELL=y
+CONFIG_QPNP_REVID=y
+CONFIG_USB_BAM=y
+CONFIG_QCOM_GENI_SE=y
+CONFIG_MSM_GCC_SDM845=y
+CONFIG_MSM_VIDEOCC_SDM845=y
+CONFIG_MSM_CAMCC_SDM845=y
+CONFIG_MSM_DISPCC_SDM845=y
+CONFIG_CLOCK_QPNP_DIV=y
+CONFIG_MSM_CLK_RPMH=y
+CONFIG_CLOCK_CPU_OSM=y
+CONFIG_MSM_GPUCC_SDM845=y
+CONFIG_MSM_CLK_AOP_QMP=y
+CONFIG_QCOM_MDSS_PLL=y
+CONFIG_REMOTE_SPINLOCK_MSM=y
+CONFIG_MSM_QMP=y
+CONFIG_IOMMU_IO_PGTABLE_FAST=y
+CONFIG_ARM_SMMU=y
+CONFIG_QCOM_LAZY_MAPPING=y
+CONFIG_QCOM_RUN_QUEUE_STATS=y
+CONFIG_QCOM_LLCC=y
+CONFIG_QCOM_SDM845_LLCC=y
+CONFIG_MSM_SERVICE_LOCATOR=y
+CONFIG_MSM_SERVICE_NOTIFIER=y
+CONFIG_MSM_BOOT_STATS=y
+CONFIG_MSM_CORE_HANG_DETECT=y
+CONFIG_MSM_GLADIATOR_HANG_DETECT=y
+CONFIG_QCOM_EUD=y
+CONFIG_QCOM_WATCHDOG_V2=y
+CONFIG_QCOM_MEMORY_DUMP_V2=y
+CONFIG_QCOM_BUS_SCALING=y
+CONFIG_QCOM_BUS_CONFIG_RPMH=y
+CONFIG_QCOM_SECURE_BUFFER=y
+CONFIG_QCOM_EARLY_RANDOM=y
+CONFIG_MSM_SMEM=y
+CONFIG_MSM_GLINK=y
+CONFIG_MSM_GLINK_LOOPBACK_SERVER=y
+CONFIG_MSM_GLINK_SMEM_NATIVE_XPRT=y
+CONFIG_MSM_GLINK_SPI_XPRT=y
+CONFIG_MSM_SPCOM=y
+CONFIG_MSM_SPSS_UTILS=y
+CONFIG_TRACER_PKT=y
+CONFIG_QTI_RPMH_API=y
+CONFIG_MSM_SMP2P=y
+CONFIG_MSM_IPC_ROUTER_GLINK_XPRT=y
+CONFIG_MSM_QMI_INTERFACE=y
+CONFIG_MSM_GLINK_PKT=y
+CONFIG_MSM_SUBSYSTEM_RESTART=y
+CONFIG_MSM_PIL=y
+CONFIG_MSM_SYSMON_GLINK_COMM=y
+CONFIG_MSM_PIL_SSR_GENERIC=y
+CONFIG_MSM_PIL_MSS_QDSP6V5=y
+CONFIG_ICNSS=y
+CONFIG_QCOM_COMMAND_DB=y
+CONFIG_MSM_CDSP_LOADER=y
+CONFIG_QCOM_SMCINVOKE=y
+CONFIG_MSM_EVENT_TIMER=y
+CONFIG_MSM_PM=y
+CONFIG_QTI_RPM_STATS_LOG=y
+CONFIG_QCOM_FORCE_WDOG_BITE_ON_PANIC=y
+CONFIG_QMP_DEBUGFS_CLIENT=y
+CONFIG_QSEE_IPC_IRQ_BRIDGE=y
+CONFIG_MODEMSMEM=y
+CONFIG_QCOM_BIMC_BWMON=y
+CONFIG_ARM_MEMLAT_MON=y
+CONFIG_QCOMCCI_HWMON=y
+CONFIG_QCOM_M4M_HWMON=y
+CONFIG_DEVFREQ_GOV_QCOM_BW_HWMON=y
+CONFIG_DEVFREQ_GOV_QCOM_CACHE_HWMON=y
+CONFIG_DEVFREQ_GOV_MEMLAT=y
+CONFIG_DEVFREQ_SIMPLE_DEV=y
+CONFIG_QCOM_DEVFREQ_DEVBW=y
+CONFIG_EXTCON=y
+CONFIG_EXTCON_USB_GPIO=y
+CONFIG_IIO=y
+CONFIG_QCOM_RRADC=y
+CONFIG_PWM=y
+CONFIG_PWM_QPNP=y
+CONFIG_ARM_GIC_V3_ACL=y
+CONFIG_ANDROID=y
+CONFIG_ANDROID_BINDER_IPC=y
+CONFIG_SENSORS_SSC=y
+CONFIG_MSM_TZ_LOG=y
+CONFIG_EXT4_FS=y
+CONFIG_EXT4_FS_SECURITY=y
+CONFIG_F2FS_FS=y
+CONFIG_F2FS_FS_SECURITY=y
+CONFIG_F2FS_FS_ENCRYPTION=y
+CONFIG_F2FS_FS_VERITY=y
+CONFIG_QUOTA=y
+CONFIG_QUOTA_NETLINK_INTERFACE=y
+CONFIG_QFMT_V2=y
+CONFIG_FUSE_FS=y
+CONFIG_OVERLAY_FS=y
+CONFIG_VFAT_FS=y
+CONFIG_TMPFS_POSIX_ACL=y
+CONFIG_ECRYPT_FS=y
+CONFIG_ECRYPT_FS_MESSAGING=y
+CONFIG_SDCARD_FS=y
+CONFIG_PSTORE=y
+CONFIG_PSTORE_CONSOLE=y
+CONFIG_PSTORE_PMSG=y
+CONFIG_PSTORE_RAM=y
+CONFIG_BLDR_DEBUG_LOG=y
+# CONFIG_NETWORK_FILESYSTEMS is not set
+CONFIG_NLS_CODEPAGE_437=y
+CONFIG_NLS_ISO8859_1=y
+CONFIG_PRINTK_TIME=y
+CONFIG_DYNAMIC_DEBUG=y
+CONFIG_DEBUG_INFO=y
+CONFIG_MAGIC_SYSRQ=y
+CONFIG_DETECT_HUNG_TASK=y
+CONFIG_PANIC_TIMEOUT=1
+CONFIG_SCHEDSTATS=y
+# CONFIG_DEBUG_PREEMPT is not set
+CONFIG_IPC_LOGGING=y
+CONFIG_QCOM_RTB=y
+CONFIG_QCOM_RTB_SEPARATE_CPUS=y
+CONFIG_CPU_FREQ_SWITCH_PROFILER=y
+CONFIG_CC_WERROR=y
+CONFIG_PID_IN_CONTEXTIDR=y
+CONFIG_DEBUG_ALIGN_RODATA=y
+CONFIG_PFK=y
+CONFIG_SECURITY_PERF_EVENTS_RESTRICT=y
+CONFIG_SECURITY=y
+CONFIG_HARDENED_USERCOPY=y
+CONFIG_FORTIFY_SOURCE=y
+CONFIG_SECURITY_SELINUX=y
+CONFIG_SECURITY_SMACK=y
+CONFIG_CRYPTO_GCM=y
+CONFIG_CRYPTO_XCBC=y
+CONFIG_CRYPTO_MD4=y
+CONFIG_CRYPTO_TWOFISH=y
+CONFIG_CRYPTO_LZ4=y
+CONFIG_CRYPTO_ANSI_CPRNG=y
+CONFIG_CRYPTO_DRBG_HASH=y
+CONFIG_CRYPTO_DRBG_CTR=y
+CONFIG_CRYPTO_DEV_QCOM_MSM_QCE=y
+CONFIG_CRYPTO_DEV_QCRYPTO=y
+CONFIG_CRYPTO_DEV_QCEDEV=y
+CONFIG_CRYPTO_DEV_QCOM_ICE=y
+CONFIG_SYSTEM_TRUSTED_KEYS="certs/esl_key.pem"
+CONFIG_ARM64_CRYPTO=y
+CONFIG_CRYPTO_SHA1_ARM64_CE=y
+CONFIG_CRYPTO_SHA2_ARM64_CE=y
+CONFIG_CRYPTO_GHASH_ARM64_CE=y
+CONFIG_CRYPTO_AES_ARM64_CE_CCM=y
+CONFIG_CRYPTO_AES_ARM64_CE_BLK=y
+CONFIG_CRYPTO_AES_ARM64_NEON_BLK=y
+CONFIG_CRYPTO_CRC32_ARM64=y
+CONFIG_XZ_DEC=y
+CONFIG_QMI_ENCDEC=y
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index dab2cb0..b4c4d82 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -260,6 +260,8 @@
CONFIG_GPIO_PCA953X=y
CONFIG_GPIO_PCA953X_IRQ=y
CONFIG_GPIO_MAX77620=y
+CONFIG_POWER_AVS=y
+CONFIG_ROCKCHIP_IODOMAIN=y
CONFIG_POWER_RESET_MSM=y
CONFIG_BATTERY_BQ27XXX=y
CONFIG_POWER_RESET_XGENE=y
diff --git a/arch/arm64/include/asm/alternative.h b/arch/arm64/include/asm/alternative.h
index 6e1cb8c..7e842dc 100644
--- a/arch/arm64/include/asm/alternative.h
+++ b/arch/arm64/include/asm/alternative.h
@@ -4,6 +4,8 @@
#include <asm/cpucaps.h>
#include <asm/insn.h>
+#define ARM64_CB_PATCH ARM64_NCAPS
+
#ifndef __ASSEMBLY__
#include <linux/init.h>
@@ -11,6 +13,8 @@
#include <linux/stddef.h>
#include <linux/stringify.h>
+extern int alternatives_applied;
+
struct alt_instr {
s32 orig_offset; /* offset to original instruction */
s32 alt_offset; /* offset to replacement instruction */
@@ -19,12 +23,19 @@ struct alt_instr {
u8 alt_len; /* size of new instruction(s), <= orig_len */
};
+typedef void (*alternative_cb_t)(struct alt_instr *alt,
+ __le32 *origptr, __le32 *updptr, int nr_inst);
+
void __init apply_alternatives_all(void);
void apply_alternatives(void *start, size_t length);
-#define ALTINSTR_ENTRY(feature) \
+#define ALTINSTR_ENTRY(feature,cb) \
" .word 661b - .\n" /* label */ \
+ " .if " __stringify(cb) " == 0\n" \
" .word 663f - .\n" /* new instruction */ \
+ " .else\n" \
+ " .word " __stringify(cb) "- .\n" /* callback */ \
+ " .endif\n" \
" .hword " __stringify(feature) "\n" /* feature bit */ \
" .byte 662b-661b\n" /* source len */ \
" .byte 664f-663f\n" /* replacement len */
@@ -42,15 +53,18 @@ void apply_alternatives(void *start, size_t length);
* but most assemblers die if insn1 or insn2 have a .inst. This should
* be fixed in a binutils release posterior to 2.25.51.0.2 (anything
* containing commit 4e4d08cf7399b606 or c1baaddf8861).
+ *
+ * Alternatives with callbacks do not generate replacement instructions.
*/
-#define __ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg_enabled) \
+#define __ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg_enabled, cb) \
".if "__stringify(cfg_enabled)" == 1\n" \
"661:\n\t" \
oldinstr "\n" \
"662:\n" \
".pushsection .altinstructions,\"a\"\n" \
- ALTINSTR_ENTRY(feature) \
+ ALTINSTR_ENTRY(feature,cb) \
".popsection\n" \
+ " .if " __stringify(cb) " == 0\n" \
".pushsection .altinstr_replacement, \"a\"\n" \
"663:\n\t" \
newinstr "\n" \
@@ -58,11 +72,17 @@ void apply_alternatives(void *start, size_t length);
".popsection\n\t" \
".org . - (664b-663b) + (662b-661b)\n\t" \
".org . - (662b-661b) + (664b-663b)\n" \
+ ".else\n\t" \
+ "663:\n\t" \
+ "664:\n\t" \
+ ".endif\n" \
".endif\n"
#define _ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg, ...) \
- __ALTERNATIVE_CFG(oldinstr, newinstr, feature, IS_ENABLED(cfg))
+ __ALTERNATIVE_CFG(oldinstr, newinstr, feature, IS_ENABLED(cfg), 0)
+#define ALTERNATIVE_CB(oldinstr, cb) \
+ __ALTERNATIVE_CFG(oldinstr, "NOT_AN_INSTRUCTION", ARM64_CB_PATCH, 1, cb)
#else
#include <asm/assembler.h>
@@ -129,6 +149,14 @@ void apply_alternatives(void *start, size_t length);
661:
.endm
+.macro alternative_cb cb
+ .set .Lasm_alt_mode, 0
+ .pushsection .altinstructions, "a"
+ altinstruction_entry 661f, \cb, ARM64_CB_PATCH, 662f-661f, 0
+ .popsection
+661:
+.endm
+
/*
* Provide the other half of the alternative code sequence.
*/
@@ -155,6 +183,13 @@ void apply_alternatives(void *start, size_t length);
.endm
/*
+ * Callback-based alternative epilogue
+ */
+.macro alternative_cb_end
+662:
+.endm
+
+/*
* Provides a trivial alternative or default sequence consisting solely
* of NOPs. The number of NOPs is chosen automatically to match the
* previous case.
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index c7749d8..d9e329d 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -266,7 +266,11 @@ lr .req x30 // link register
*/
.macro adr_this_cpu, dst, sym, tmp
adr_l \dst, \sym
+alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
mrs \tmp, tpidr_el1
+alternative_else
+ mrs \tmp, tpidr_el2
+alternative_endif
add \dst, \dst, \tmp
.endm
@@ -277,7 +281,11 @@ lr .req x30 // link register
*/
.macro ldr_this_cpu dst, sym, tmp
adr_l \dst, \sym
+alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
mrs \tmp, tpidr_el1
+alternative_else
+ mrs \tmp, tpidr_el2
+alternative_endif
ldr \dst, [\dst, \tmp]
.endm
@@ -471,6 +479,10 @@ alternative_endif
mrs \rd, sp_el0
.endm
+ .macro pte_to_phys, phys, pte
+ and \phys, \pte, #(((1 << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT)
+ .endm
+
/*
* Check the MIDR_EL1 of the current CPU for a given model and a range of
* variant/revision. See asm/cputype.h for the macros used below.
@@ -510,8 +522,4 @@ alternative_endif
.Ldone\@:
.endm
- .macro pte_to_phys, phys, pte
- and \phys, \pte, #(((1 << (48 - PAGE_SHIFT)) - 1) << PAGE_SHIFT)
- .endm
-
#endif /* __ASM_ASSEMBLER_H */
diff --git a/arch/arm64/include/asm/atomic_lse.h b/arch/arm64/include/asm/atomic_lse.h
index 7457ce0..d32a016 100644
--- a/arch/arm64/include/asm/atomic_lse.h
+++ b/arch/arm64/include/asm/atomic_lse.h
@@ -117,7 +117,7 @@ static inline void atomic_and(int i, atomic_t *v)
/* LSE atomics */
" mvn %w[i], %w[i]\n"
" stclr %w[i], %[v]")
- : [i] "+r" (w0), [v] "+Q" (v->counter)
+ : [i] "+&r" (w0), [v] "+Q" (v->counter)
: "r" (x1)
: __LL_SC_CLOBBERS);
}
@@ -135,7 +135,7 @@ static inline int atomic_fetch_and##name(int i, atomic_t *v) \
/* LSE atomics */ \
" mvn %w[i], %w[i]\n" \
" ldclr" #mb " %w[i], %w[i], %[v]") \
- : [i] "+r" (w0), [v] "+Q" (v->counter) \
+ : [i] "+&r" (w0), [v] "+Q" (v->counter) \
: "r" (x1) \
: __LL_SC_CLOBBERS, ##cl); \
\
@@ -161,7 +161,7 @@ static inline void atomic_sub(int i, atomic_t *v)
/* LSE atomics */
" neg %w[i], %w[i]\n"
" stadd %w[i], %[v]")
- : [i] "+r" (w0), [v] "+Q" (v->counter)
+ : [i] "+&r" (w0), [v] "+Q" (v->counter)
: "r" (x1)
: __LL_SC_CLOBBERS);
}
@@ -180,7 +180,7 @@ static inline int atomic_sub_return##name(int i, atomic_t *v) \
" neg %w[i], %w[i]\n" \
" ldadd" #mb " %w[i], w30, %[v]\n" \
" add %w[i], %w[i], w30") \
- : [i] "+r" (w0), [v] "+Q" (v->counter) \
+ : [i] "+&r" (w0), [v] "+Q" (v->counter) \
: "r" (x1) \
: __LL_SC_CLOBBERS , ##cl); \
\
@@ -207,7 +207,7 @@ static inline int atomic_fetch_sub##name(int i, atomic_t *v) \
/* LSE atomics */ \
" neg %w[i], %w[i]\n" \
" ldadd" #mb " %w[i], %w[i], %[v]") \
- : [i] "+r" (w0), [v] "+Q" (v->counter) \
+ : [i] "+&r" (w0), [v] "+Q" (v->counter) \
: "r" (x1) \
: __LL_SC_CLOBBERS, ##cl); \
\
@@ -314,7 +314,7 @@ static inline void atomic64_and(long i, atomic64_t *v)
/* LSE atomics */
" mvn %[i], %[i]\n"
" stclr %[i], %[v]")
- : [i] "+r" (x0), [v] "+Q" (v->counter)
+ : [i] "+&r" (x0), [v] "+Q" (v->counter)
: "r" (x1)
: __LL_SC_CLOBBERS);
}
@@ -332,7 +332,7 @@ static inline long atomic64_fetch_and##name(long i, atomic64_t *v) \
/* LSE atomics */ \
" mvn %[i], %[i]\n" \
" ldclr" #mb " %[i], %[i], %[v]") \
- : [i] "+r" (x0), [v] "+Q" (v->counter) \
+ : [i] "+&r" (x0), [v] "+Q" (v->counter) \
: "r" (x1) \
: __LL_SC_CLOBBERS, ##cl); \
\
@@ -358,7 +358,7 @@ static inline void atomic64_sub(long i, atomic64_t *v)
/* LSE atomics */
" neg %[i], %[i]\n"
" stadd %[i], %[v]")
- : [i] "+r" (x0), [v] "+Q" (v->counter)
+ : [i] "+&r" (x0), [v] "+Q" (v->counter)
: "r" (x1)
: __LL_SC_CLOBBERS);
}
@@ -377,7 +377,7 @@ static inline long atomic64_sub_return##name(long i, atomic64_t *v) \
" neg %[i], %[i]\n" \
" ldadd" #mb " %[i], x30, %[v]\n" \
" add %[i], %[i], x30") \
- : [i] "+r" (x0), [v] "+Q" (v->counter) \
+ : [i] "+&r" (x0), [v] "+Q" (v->counter) \
: "r" (x1) \
: __LL_SC_CLOBBERS, ##cl); \
\
@@ -404,7 +404,7 @@ static inline long atomic64_fetch_sub##name(long i, atomic64_t *v) \
/* LSE atomics */ \
" neg %[i], %[i]\n" \
" ldadd" #mb " %[i], %[i], %[v]") \
- : [i] "+r" (x0), [v] "+Q" (v->counter) \
+ : [i] "+&r" (x0), [v] "+Q" (v->counter) \
: "r" (x1) \
: __LL_SC_CLOBBERS, ##cl); \
\
@@ -516,7 +516,7 @@ static inline long __cmpxchg_double##name(unsigned long old1, \
" eor %[old1], %[old1], %[oldval1]\n" \
" eor %[old2], %[old2], %[oldval2]\n" \
" orr %[old1], %[old1], %[old2]") \
- : [old1] "+r" (x0), [old2] "+r" (x1), \
+ : [old1] "+&r" (x0), [old2] "+&r" (x1), \
[v] "+Q" (*(unsigned long *)ptr) \
: [new1] "r" (x2), [new2] "r" (x3), [ptr] "r" (x4), \
[oldval1] "r" (oldval1), [oldval2] "r" (oldval2) \
diff --git a/arch/arm64/include/asm/cachetype.h b/arch/arm64/include/asm/cachetype.h
index f558869..877d478 100644
--- a/arch/arm64/include/asm/cachetype.h
+++ b/arch/arm64/include/asm/cachetype.h
@@ -22,6 +22,11 @@
#define CTR_L1IP_MASK 3
#define CTR_CWG_SHIFT 24
#define CTR_CWG_MASK 15
+#define CTR_DMINLINE_SHIFT 16
+#define CTR_IMINLINE_SHIFT 0
+
+#define CTR_CACHE_MINLINE_MASK \
+ ((0xf << CTR_DMINLINE_SHIFT) | (0xf << CTR_IMINLINE_SHIFT))
#define ICACHE_POLICY_RESERVED 0
#define ICACHE_POLICY_AIVIVT 1
diff --git a/arch/arm64/include/asm/cmpxchg.h b/arch/arm64/include/asm/cmpxchg.h
index ae852ad..0f2e1ab 100644
--- a/arch/arm64/include/asm/cmpxchg.h
+++ b/arch/arm64/include/asm/cmpxchg.h
@@ -229,7 +229,9 @@ static inline void __cmpwait_case_##name(volatile void *ptr, \
unsigned long tmp; \
\
asm volatile( \
- " ldxr" #sz "\t%" #w "[tmp], %[v]\n" \
+ " sevl\n" \
+ " wfe\n" \
+ " ldxr" #sz "\t%" #w "[tmp], %[v]\n" \
" eor %" #w "[tmp], %" #w "[tmp], %" #w "[val]\n" \
" cbnz %" #w "[tmp], 1f\n" \
" wfe\n" \
diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index d64bf94..898fbaa 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -38,7 +38,9 @@
#define ARM64_UNMAP_KERNEL_AT_EL0 16
#define ARM64_HARDEN_BRANCH_PREDICTOR 17
+#define ARM64_SSBD 18
+#define ARM64_MISMATCHED_CACHE_TYPE 19
-#define ARM64_NCAPS 18
+#define ARM64_NCAPS 20
#endif /* __ASM_CPUCAPS_H */
diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h
index b3423f5..ddaeb43 100644
--- a/arch/arm64/include/asm/cpufeature.h
+++ b/arch/arm64/include/asm/cpufeature.h
@@ -9,8 +9,6 @@
#ifndef __ASM_CPUFEATURE_H
#define __ASM_CPUFEATURE_H
-#include <linux/jump_label.h>
-
#include <asm/cpucaps.h>
#include <asm/hwcap.h>
#include <asm/sysreg.h>
@@ -27,6 +25,8 @@
#ifndef __ASSEMBLY__
+#include <linux/bug.h>
+#include <linux/jump_label.h>
#include <linux/kernel.h>
extern const char *machine_name;
@@ -98,6 +98,7 @@ struct arm64_cpu_capabilities {
extern DECLARE_BITMAP(cpu_hwcaps, ARM64_NCAPS);
extern struct static_key_false cpu_hwcap_keys[ARM64_NCAPS];
+extern struct static_key_false arm64_const_caps_ready;
bool this_cpu_has_cap(unsigned int cap);
@@ -106,14 +107,27 @@ static inline bool cpu_have_feature(unsigned int num)
return elf_hwcap & (1UL << num);
}
+/* System capability check for constant caps */
+static inline bool __cpus_have_const_cap(int num)
+{
+ if (num >= ARM64_NCAPS)
+ return false;
+ return static_branch_unlikely(&cpu_hwcap_keys[num]);
+}
+
static inline bool cpus_have_cap(unsigned int num)
{
if (num >= ARM64_NCAPS)
return false;
- if (__builtin_constant_p(num))
- return static_branch_unlikely(&cpu_hwcap_keys[num]);
+ return test_bit(num, cpu_hwcaps);
+}
+
+static inline bool cpus_have_const_cap(int num)
+{
+ if (static_branch_likely(&arm64_const_caps_ready))
+ return __cpus_have_const_cap(num);
else
- return test_bit(num, cpu_hwcaps);
+ return cpus_have_cap(num);
}
static inline void cpus_set_cap(unsigned int num)
@@ -123,7 +137,6 @@ static inline void cpus_set_cap(unsigned int num)
num, ARM64_NCAPS);
} else {
__set_bit(num, cpu_hwcaps);
- static_branch_enable(&cpu_hwcap_keys[num]);
}
}
@@ -202,7 +215,7 @@ static inline bool cpu_supports_mixed_endian_el0(void)
static inline bool system_supports_32bit_el0(void)
{
- return cpus_have_cap(ARM64_HAS_32BIT_EL0);
+ return cpus_have_const_cap(ARM64_HAS_32BIT_EL0);
}
static inline bool system_supports_mixed_endian_el0(void)
@@ -216,6 +229,28 @@ static inline bool system_uses_ttbr0_pan(void)
!cpus_have_cap(ARM64_HAS_PAN);
}
+#define ARM64_SSBD_UNKNOWN -1
+#define ARM64_SSBD_FORCE_DISABLE 0
+#define ARM64_SSBD_KERNEL 1
+#define ARM64_SSBD_FORCE_ENABLE 2
+#define ARM64_SSBD_MITIGATED 3
+
+static inline int arm64_get_ssbd_state(void)
+{
+#ifdef CONFIG_ARM64_SSBD
+ extern int ssbd_state;
+ return ssbd_state;
+#else
+ return ARM64_SSBD_UNKNOWN;
+#endif
+}
+
+#ifdef CONFIG_ARM64_SSBD
+void arm64_set_ssbd_mitigation(bool state);
+#else
+static inline void arm64_set_ssbd_mitigation(bool state) {}
+#endif
+
#endif /* __ASSEMBLY__ */
#endif
diff --git a/arch/arm64/include/asm/cputype.h b/arch/arm64/include/asm/cputype.h
index 9a6b81a..d20708857 100644
--- a/arch/arm64/include/asm/cputype.h
+++ b/arch/arm64/include/asm/cputype.h
@@ -57,7 +57,7 @@
((partnum) << MIDR_PARTNUM_SHIFT))
#define MIDR_CPU_VAR_REV(var, rev) \
- (((var) << MIDR_VARIANT_SHIFT) | (rev))
+ (((var) << MIDR_VARIANT_SHIFT) | (rev))
#define MIDR_CPU_MODEL_MASK (MIDR_IMPLEMENTOR_MASK | MIDR_PARTNUM_MASK | \
MIDR_ARCHITECTURE_MASK)
@@ -77,6 +77,7 @@
#define ARM_CPU_PART_AEM_V8 0xD0F
#define ARM_CPU_PART_FOUNDATION 0xD00
+#define ARM_CPU_PART_CORTEX_A55 0xD05
#define ARM_CPU_PART_CORTEX_A57 0xD07
#define ARM_CPU_PART_CORTEX_A72 0xD08
#define ARM_CPU_PART_CORTEX_A53 0xD03
@@ -97,6 +98,7 @@
#define BRCM_CPU_PART_VULCAN 0x516
#define MIDR_CORTEX_A53 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A53)
+#define MIDR_CORTEX_A55 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A55)
#define MIDR_CORTEX_A57 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A57)
#define MIDR_CORTEX_A72 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A72)
#define MIDR_CORTEX_A73 MIDR_CPU_MODEL(ARM_CPU_IMP_ARM, ARM_CPU_PART_CORTEX_A73)
diff --git a/arch/arm64/include/asm/elf.h b/arch/arm64/include/asm/elf.h
index 40a8a94..d46c194 100644
--- a/arch/arm64/include/asm/elf.h
+++ b/arch/arm64/include/asm/elf.h
@@ -140,11 +140,12 @@ typedef struct user_fpsimd_state elf_fpregset_t;
#define SET_PERSONALITY(ex) clear_thread_flag(TIF_32BIT);
/* update AT_VECTOR_SIZE_ARCH if the number of NEW_AUX_ENT entries changes */
-#define ARCH_DLINFO \
+#define _SET_AUX_ENT_VDSO \
do { \
NEW_AUX_ENT(AT_SYSINFO_EHDR, \
- (elf_addr_t)current->mm->context.vdso); \
+ (Elf64_Off)current->mm->context.vdso); \
} while (0)
+#define ARCH_DLINFO _SET_AUX_ENT_VDSO
#define ARCH_HAS_SETUP_ADDITIONAL_PAGES
struct linux_binprm;
@@ -184,7 +185,11 @@ typedef compat_elf_greg_t compat_elf_gregset_t[COMPAT_ELF_NGREG];
#define compat_start_thread compat_start_thread
#define COMPAT_SET_PERSONALITY(ex) set_thread_flag(TIF_32BIT);
+#ifdef CONFIG_VDSO32
+#define COMPAT_ARCH_DLINFO _SET_AUX_ENT_VDSO
+#else
#define COMPAT_ARCH_DLINFO
+#endif
extern int aarch32_setup_vectors_page(struct linux_binprm *bprm,
int uses_interp);
#define compat_arch_setup_additional_pages \
diff --git a/arch/arm64/include/asm/futex.h b/arch/arm64/include/asm/futex.h
index c5bc52e..a891bb6 100644
--- a/arch/arm64/include/asm/futex.h
+++ b/arch/arm64/include/asm/futex.h
@@ -48,20 +48,9 @@ do { \
} while (0)
static inline int
-futex_atomic_op_inuser(unsigned int encoded_op, u32 __user *_uaddr)
+arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (int)(encoded_op << 8) >> 20;
- int cmparg = (int)(encoded_op << 20) >> 20;
int oldval = 0, ret, tmp;
- u32 __user *uaddr = __uaccess_mask_ptr(_uaddr);
-
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1U << (oparg & 0x1f);
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
pagefault_disable();
@@ -92,17 +81,9 @@ futex_atomic_op_inuser(unsigned int encoded_op, u32 __user *_uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/arm64/include/asm/jump_label.h b/arch/arm64/include/asm/jump_label.h
index 1b5e0e8..7e2b3e3 100644
--- a/arch/arm64/include/asm/jump_label.h
+++ b/arch/arm64/include/asm/jump_label.h
@@ -28,7 +28,7 @@
static __always_inline bool arch_static_branch(struct static_key *key, bool branch)
{
- asm goto("1: nop\n\t"
+ asm_volatile_goto("1: nop\n\t"
".pushsection __jump_table, \"aw\"\n\t"
".align 3\n\t"
".quad 1b, %l[l_yes], %c0\n\t"
@@ -42,7 +42,7 @@ static __always_inline bool arch_static_branch(struct static_key *key, bool bran
static __always_inline bool arch_static_branch_jump(struct static_key *key, bool branch)
{
- asm goto("1: b %l[l_yes]\n\t"
+ asm_volatile_goto("1: b %l[l_yes]\n\t"
".pushsection __jump_table, \"aw\"\n\t"
".align 3\n\t"
".quad 1b, %l[l_yes], %c0\n\t"
diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h
index 0dbc1c6..68dedca 100644
--- a/arch/arm64/include/asm/kvm_arm.h
+++ b/arch/arm64/include/asm/kvm_arm.h
@@ -99,7 +99,7 @@
TCR_EL2_ORGN0_MASK | TCR_EL2_IRGN0_MASK | TCR_EL2_T0SZ_MASK)
/* VTCR_EL2 Registers bits */
-#define VTCR_EL2_RES1 (1 << 31)
+#define VTCR_EL2_RES1 (1U << 31)
#define VTCR_EL2_HD (1 << 22)
#define VTCR_EL2_HA (1 << 21)
#define VTCR_EL2_PS_MASK TCR_EL2_PS_MASK
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index ec3553eb..8f5cf83 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -33,6 +33,10 @@
#define KVM_ARM64_DEBUG_DIRTY_SHIFT 0
#define KVM_ARM64_DEBUG_DIRTY (1 << KVM_ARM64_DEBUG_DIRTY_SHIFT)
+#define VCPU_WORKAROUND_2_FLAG_SHIFT 0
+#define VCPU_WORKAROUND_2_FLAG (_AC(1, UL) << VCPU_WORKAROUND_2_FLAG_SHIFT)
+
+/* Translate a kernel address of @sym into its equivalent linear mapping */
#define kvm_ksym_ref(sym) \
({ \
void *val = &sym; \
@@ -65,6 +69,43 @@ extern u32 __kvm_get_mdcr_el2(void);
extern u32 __init_stage2_translation(void);
+/* Home-grown __this_cpu_{ptr,read} variants that always work at HYP */
+#define __hyp_this_cpu_ptr(sym) \
+ ({ \
+ void *__ptr = hyp_symbol_addr(sym); \
+ __ptr += read_sysreg(tpidr_el2); \
+ (typeof(&sym))__ptr; \
+ })
+
+#define __hyp_this_cpu_read(sym) \
+ ({ \
+ *__hyp_this_cpu_ptr(sym); \
+ })
+
+#else /* __ASSEMBLY__ */
+
+.macro hyp_adr_this_cpu reg, sym, tmp
+ adr_l \reg, \sym
+ mrs \tmp, tpidr_el2
+ add \reg, \reg, \tmp
+.endm
+
+.macro hyp_ldr_this_cpu reg, sym, tmp
+ adr_l \reg, \sym
+ mrs \tmp, tpidr_el2
+ ldr \reg, [\reg, \tmp]
+.endm
+
+.macro get_host_ctxt reg, tmp
+ hyp_adr_this_cpu \reg, kvm_host_cpu_state, \tmp
+.endm
+
+.macro get_vcpu_ptr vcpu, ctxt
+ get_host_ctxt \ctxt, \vcpu
+ ldr \vcpu, [\ctxt, #HOST_CONTEXT_VCPU]
+ kern_hyp_va \vcpu
+.endm
+
#endif
#endif /* __ARM_KVM_ASM_H__ */
diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index fe39e68..ba0d52c 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -42,6 +42,11 @@ void kvm_inject_vabt(struct kvm_vcpu *vcpu);
void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr);
void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr);
+static inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu)
+{
+ return !(vcpu->arch.hcr_el2 & HCR_RW);
+}
+
static inline void vcpu_reset_hcr(struct kvm_vcpu *vcpu)
{
vcpu->arch.hcr_el2 = HCR_GUEST_FLAGS;
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 37d56e8..4cdfbd0 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -24,6 +24,7 @@
#include <linux/types.h>
#include <linux/kvm_types.h>
+#include <asm/cpufeature.h>
#include <asm/kvm.h>
#include <asm/kvm_asm.h>
#include <asm/kvm_mmio.h>
@@ -73,6 +74,9 @@ struct kvm_arch {
/* Timer */
struct arch_timer_kvm timer;
+
+ /* Mandated version of PSCI */
+ u32 psci_version;
};
#define KVM_NR_MEM_OBJS 40
@@ -193,6 +197,8 @@ struct kvm_cpu_context {
u64 sys_regs[NR_SYS_REGS];
u32 copro[NR_COPRO_REGS];
};
+
+ struct kvm_vcpu *__hyp_running_vcpu;
};
typedef struct kvm_cpu_context kvm_cpu_context_t;
@@ -207,6 +213,9 @@ struct kvm_vcpu_arch {
/* Exception Information */
struct kvm_vcpu_fault_info fault;
+ /* State of various workarounds, see kvm_asm.h for bit assignment */
+ u64 workaround_flags;
+
/* Guest debug state */
u64 debug_flags;
@@ -350,15 +359,33 @@ int kvm_perf_teardown(void);
struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
+void __kvm_set_tpidr_el2(u64 tpidr_el2);
+DECLARE_PER_CPU(kvm_cpu_context_t, kvm_host_cpu_state);
+
static inline void __cpu_init_hyp_mode(phys_addr_t pgd_ptr,
unsigned long hyp_stack_ptr,
unsigned long vector_ptr)
{
+ u64 tpidr_el2;
+
/*
- * Call initialization code, and switch to the full blown
- * HYP code.
+ * Call initialization code, and switch to the full blown HYP code.
+ * If the cpucaps haven't been finalized yet, something has gone very
+ * wrong, and hyp will crash and burn when it uses any
+ * cpus_have_const_cap() wrapper.
*/
+ BUG_ON(!static_branch_likely(&arm64_const_caps_ready));
__kvm_call_hyp((void *)pgd_ptr, hyp_stack_ptr, vector_ptr);
+
+ /*
+ * Calculate the raw per-cpu offset without a translation from the
+ * kernel's mapping to the linear mapping, and store it in tpidr_el2
+ * so that we can use adr_l to access per-cpu variables in EL2.
+ */
+ tpidr_el2 = (u64)this_cpu_ptr(&kvm_host_cpu_state)
+ - (u64)kvm_ksym_ref(kvm_host_cpu_state);
+
+ kvm_call_hyp(__kvm_set_tpidr_el2, tpidr_el2);
}
void __kvm_hyp_teardown(void);
@@ -395,7 +422,30 @@ static inline void __cpu_init_stage2(void)
static inline bool kvm_arm_harden_branch_predictor(void)
{
- return cpus_have_cap(ARM64_HARDEN_BRANCH_PREDICTOR);
+ return cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR);
+}
+
+#define KVM_SSBD_UNKNOWN -1
+#define KVM_SSBD_FORCE_DISABLE 0
+#define KVM_SSBD_KERNEL 1
+#define KVM_SSBD_FORCE_ENABLE 2
+#define KVM_SSBD_MITIGATED 3
+
+static inline int kvm_arm_have_ssbd(void)
+{
+ switch (arm64_get_ssbd_state()) {
+ case ARM64_SSBD_FORCE_DISABLE:
+ return KVM_SSBD_FORCE_DISABLE;
+ case ARM64_SSBD_KERNEL:
+ return KVM_SSBD_KERNEL;
+ case ARM64_SSBD_FORCE_ENABLE:
+ return KVM_SSBD_FORCE_ENABLE;
+ case ARM64_SSBD_MITIGATED:
+ return KVM_SSBD_MITIGATED;
+ case ARM64_SSBD_UNKNOWN:
+ default:
+ return KVM_SSBD_UNKNOWN;
+ }
}
#endif /* __ARM64_KVM_HOST_H__ */
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 36d2aba..4287acb 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -131,6 +131,26 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
#define kern_hyp_va(v) ((typeof(v))(__kern_hyp_va((unsigned long)(v))))
/*
+ * Obtain the PC-relative address of a kernel symbol
+ * s: symbol
+ *
+ * The goal of this macro is to return a symbol's address based on a
+ * PC-relative computation, as opposed to a loading the VA from a
+ * constant pool or something similar. This works well for HYP, as an
+ * absolute VA is guaranteed to be wrong. Only use this if trying to
+ * obtain the address of a symbol (i.e. not something you obtained by
+ * following a pointer).
+ */
+#define hyp_symbol_addr(s) \
+ ({ \
+ typeof(s) *addr; \
+ asm("adrp %0, %1\n" \
+ "add %0, %0, :lo12:%1\n" \
+ : "=r" (addr) : "S" (&s)); \
+ addr; \
+ })
+
+/*
* We currently only support a 40bit IPA.
*/
#define KVM_PHYS_SHIFT (40)
@@ -313,6 +333,22 @@ static inline unsigned int kvm_get_vmid_bits(void)
return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
}
+/*
+ * We are not in the kvm->srcu critical section most of the time, so we take
+ * the SRCU read lock here. Since we copy the data from the user page, we
+ * can immediately drop the lock again.
+ */
+static inline int kvm_read_guest_lock(struct kvm *kvm,
+ gpa_t gpa, void *data, unsigned long len)
+{
+ int srcu_idx = srcu_read_lock(&kvm->srcu);
+ int ret = kvm_read_guest(kvm, gpa, data, len);
+
+ srcu_read_unlock(&kvm->srcu, srcu_idx);
+
+ return ret;
+}
+
#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
#include <asm/mmu.h>
@@ -325,7 +361,7 @@ static inline void *kvm_get_hyp_vector(void)
vect = __bp_harden_hyp_vecs_start +
data->hyp_vectors_slot * SZ_2K;
- if (!cpus_have_cap(ARM64_HAS_VIRT_HOST_EXTN))
+ if (!cpus_have_const_cap(ARM64_HAS_VIRT_HOST_EXTN))
vect = lm_alias(vect);
}
@@ -351,5 +387,29 @@ static inline int kvm_map_vectors(void)
}
#endif
+#ifdef CONFIG_ARM64_SSBD
+DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+
+static inline int hyp_map_aux_data(void)
+{
+ int cpu, err;
+
+ for_each_possible_cpu(cpu) {
+ u64 *ptr;
+
+ ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu);
+ err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
+ if (err)
+ return err;
+ }
+ return 0;
+}
+#else
+static inline int hyp_map_aux_data(void)
+{
+ return 0;
+}
+#endif
+
#endif /* __ASSEMBLY__ */
#endif /* __ARM64_KVM_MMU_H__ */
diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h
index 24c780d..1464b50 100644
--- a/arch/arm64/include/asm/mmu.h
+++ b/arch/arm64/include/asm/mmu.h
@@ -38,7 +38,7 @@ typedef struct {
static inline bool arm64_kernel_unmapped_at_el0(void)
{
return IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0) &&
- cpus_have_cap(ARM64_UNMAP_KERNEL_AT_EL0);
+ cpus_have_const_cap(ARM64_UNMAP_KERNEL_AT_EL0);
}
typedef void (*bp_hardening_cb_t)(void);
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index d7a3c62..20b03a02 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -16,11 +16,15 @@
#ifndef __ASM_PERCPU_H
#define __ASM_PERCPU_H
+#include <asm/alternative.h>
#include <asm/stack_pointer.h>
static inline void set_my_cpu_offset(unsigned long off)
{
- asm volatile("msr tpidr_el1, %0" :: "r" (off) : "memory");
+ asm volatile(ALTERNATIVE("msr tpidr_el1, %0",
+ "msr tpidr_el2, %0",
+ ARM64_HAS_VIRT_HOST_EXTN)
+ :: "r" (off) : "memory");
}
static inline unsigned long __my_cpu_offset(void)
@@ -31,7 +35,10 @@ static inline unsigned long __my_cpu_offset(void)
* We want to allow caching the value, so avoid using volatile and
* instead use a fake stack read to hazard against barrier().
*/
- asm("mrs %0, tpidr_el1" : "=r" (off) :
+ asm(ALTERNATIVE("mrs %0, tpidr_el1",
+ "mrs %0, tpidr_el2",
+ ARM64_HAS_VIRT_HOST_EXTN)
+ : "=r" (off) :
"Q" (*(const unsigned long *)current_stack_pointer));
return off;
@@ -86,6 +93,7 @@ static inline unsigned long __percpu_##op(void *ptr, \
: [val] "Ir" (val)); \
break; \
default: \
+ ret = 0; \
BUILD_BUG(); \
} \
\
@@ -115,6 +123,7 @@ static inline unsigned long __percpu_read(void *ptr, int size)
ret = ACCESS_ONCE(*(u64 *)ptr);
break;
default:
+ ret = 0;
BUILD_BUG();
}
@@ -184,6 +193,7 @@ static inline unsigned long __percpu_xchg(void *ptr, unsigned long val,
: [val] "r" (val));
break;
default:
+ ret = 0;
BUILD_BUG();
}
diff --git a/arch/arm64/include/asm/processor.h b/arch/arm64/include/asm/processor.h
index d8039a1..10cf9ae 100644
--- a/arch/arm64/include/asm/processor.h
+++ b/arch/arm64/include/asm/processor.h
@@ -62,9 +62,9 @@
#define STACK_TOP_MAX TASK_SIZE_64
#ifdef CONFIG_COMPAT
-#define AARCH32_VECTORS_BASE 0xffff0000
+#define AARCH32_KUSER_HELPERS_BASE 0xffff0000
#define STACK_TOP (test_thread_flag(TIF_32BIT) ? \
- AARCH32_VECTORS_BASE : STACK_TOP_MAX)
+ AARCH32_KUSER_HELPERS_BASE : STACK_TOP_MAX)
#else
#define STACK_TOP STACK_TOP_MAX
#endif /* CONFIG_COMPAT */
@@ -193,20 +193,20 @@ extern struct task_struct *cpu_switch_to(struct task_struct *prev,
#define ARCH_HAS_PREFETCH
static inline void prefetch(const void *ptr)
{
- asm volatile("prfm pldl1keep, %a0\n" : : "p" (ptr));
+ asm volatile("prfm pldl1keep, [%x0]\n" : : "r" (ptr));
}
#define ARCH_HAS_PREFETCHW
static inline void prefetchw(const void *ptr)
{
- asm volatile("prfm pstl1keep, %a0\n" : : "p" (ptr));
+ asm volatile("prfm pstl1keep, [%x0]\n" : : "r" (ptr));
}
#define ARCH_HAS_SPINLOCK_PREFETCH
static inline void spin_lock_prefetch(const void *ptr)
{
asm volatile(ARM64_LSE_ATOMIC_INSN(
- "prfm pstl1strm, %a0",
+ "prfm pstl1strm, [%x0]",
"nop") : : "p" (ptr));
}
diff --git a/arch/arm64/include/asm/scs.h b/arch/arm64/include/asm/scs.h
new file mode 100644
index 0000000..026bb8e
--- /dev/null
+++ b/arch/arm64/include/asm/scs.h
@@ -0,0 +1,53 @@
+#ifndef _ASM_SCS_H
+#define _ASM_SCS_H
+
+#ifndef __ASSEMBLY__
+
+#include <linux/scs.h>
+
+#ifdef CONFIG_SHADOW_CALL_STACK
+
+static inline void scs_save(struct task_struct *tsk)
+{
+ void *s;
+
+ asm volatile("mov %0, x18" : "=r" (s));
+ task_set_scs(tsk, s);
+}
+
+static inline void scs_load(struct task_struct *tsk)
+{
+ asm volatile("mov x18, %0" : : "r" (task_scs(tsk)));
+ task_set_scs(tsk, NULL);
+}
+
+static inline void scs_thread_switch(struct task_struct *prev,
+ struct task_struct *next)
+{
+ scs_save(prev);
+ scs_load(next);
+
+ if (unlikely(scs_corrupted(prev)))
+ panic("corrupted shadow stack detected inside scheduler\n");
+}
+
+#else /* CONFIG_SHADOW_CALL_STACK */
+
+static inline void scs_save(struct task_struct *tsk)
+{
+}
+
+static inline void scs_load(struct task_struct *tsk)
+{
+}
+
+static inline void scs_thread_switch(struct task_struct *prev,
+ struct task_struct *next)
+{
+}
+
+#endif /* CONFIG_SHADOW_CALL_STACK */
+
+#endif /* __ASSEMBLY __ */
+
+#endif /* _ASM_SCS_H */
diff --git a/arch/arm64/include/asm/signal32.h b/arch/arm64/include/asm/signal32.h
index 81abea0..bcd0e13 100644
--- a/arch/arm64/include/asm/signal32.h
+++ b/arch/arm64/include/asm/signal32.h
@@ -20,7 +20,51 @@
#ifdef CONFIG_COMPAT
#include <linux/compat.h>
-#define AARCH32_KERN_SIGRET_CODE_OFFSET 0x500
+struct compat_sigcontext {
+ /* We always set these two fields to 0 */
+ compat_ulong_t trap_no;
+ compat_ulong_t error_code;
+
+ compat_ulong_t oldmask;
+ compat_ulong_t arm_r0;
+ compat_ulong_t arm_r1;
+ compat_ulong_t arm_r2;
+ compat_ulong_t arm_r3;
+ compat_ulong_t arm_r4;
+ compat_ulong_t arm_r5;
+ compat_ulong_t arm_r6;
+ compat_ulong_t arm_r7;
+ compat_ulong_t arm_r8;
+ compat_ulong_t arm_r9;
+ compat_ulong_t arm_r10;
+ compat_ulong_t arm_fp;
+ compat_ulong_t arm_ip;
+ compat_ulong_t arm_sp;
+ compat_ulong_t arm_lr;
+ compat_ulong_t arm_pc;
+ compat_ulong_t arm_cpsr;
+ compat_ulong_t fault_address;
+};
+
+struct compat_ucontext {
+ compat_ulong_t uc_flags;
+ compat_uptr_t uc_link;
+ compat_stack_t uc_stack;
+ struct compat_sigcontext uc_mcontext;
+ compat_sigset_t uc_sigmask;
+ int __unused[32 - (sizeof(compat_sigset_t) / sizeof(int))];
+ compat_ulong_t uc_regspace[128] __aligned(8);
+};
+
+struct compat_sigframe {
+ struct compat_ucontext uc;
+ compat_ulong_t retcode[2];
+};
+
+struct compat_rt_sigframe {
+ struct compat_siginfo info;
+ struct compat_sigframe sig;
+};
int compat_setup_frame(int usig, struct ksignal *ksig, sigset_t *set,
struct pt_regs *regs);
diff --git a/arch/arm64/include/asm/spinlock.h b/arch/arm64/include/asm/spinlock.h
index 55082cc..b15a5abe 100644
--- a/arch/arm64/include/asm/spinlock.h
+++ b/arch/arm64/include/asm/spinlock.h
@@ -140,8 +140,8 @@ static inline int arch_spin_trylock(arch_spinlock_t *lock)
" cbnz %w1, 1f\n"
" add %w1, %w0, %3\n"
" casa %w0, %w1, %2\n"
- " and %w1, %w1, #0xffff\n"
- " eor %w1, %w1, %w0, lsr #16\n"
+ " sub %w1, %w1, %3\n"
+ " eor %w1, %w1, %w0\n"
"1:")
: "=&r" (lockval), "=&r" (tmp), "+Q" (*lock)
: "I" (1 << TICKET_SHIFT)
diff --git a/arch/arm64/include/asm/stacktrace.h b/arch/arm64/include/asm/stacktrace.h
index 801a16db..7d2a15a 100644
--- a/arch/arm64/include/asm/stacktrace.h
+++ b/arch/arm64/include/asm/stacktrace.h
@@ -23,7 +23,7 @@ struct stackframe {
unsigned long sp;
unsigned long pc;
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
- unsigned int graph;
+ int graph;
#endif
};
diff --git a/arch/arm64/include/asm/thread_info.h b/arch/arm64/include/asm/thread_info.h
index ba3a69a..40e3f26 100644
--- a/arch/arm64/include/asm/thread_info.h
+++ b/arch/arm64/include/asm/thread_info.h
@@ -45,6 +45,9 @@ struct thread_info {
u64 ttbr0; /* saved TTBR0_EL1 */
#endif
int preempt_count; /* 0 => preemptable, <0 => bug */
+#ifdef CONFIG_SHADOW_CALL_STACK
+ void *shadow_call_stack;
+#endif
};
#define INIT_THREAD_INFO(tsk) \
@@ -89,6 +92,7 @@ struct thread_info {
#define TIF_RESTORE_SIGMASK 20
#define TIF_SINGLESTEP 21
#define TIF_32BIT 22 /* 32bit process */
+#define TIF_SSBD 23 /* Wants SSB mitigation */
#define TIF_MM_RELEASED 24
#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)
diff --git a/arch/arm64/include/asm/uaccess.h b/arch/arm64/include/asm/uaccess.h
index 6c35d21..de21caa 100644
--- a/arch/arm64/include/asm/uaccess.h
+++ b/arch/arm64/include/asm/uaccess.h
@@ -312,7 +312,7 @@ do { \
(err), ARM64_HAS_UAO); \
break; \
case 8: \
- __get_user_asm("ldr", "ldtr", "%", __gu_val, (ptr), \
+ __get_user_asm("ldr", "ldtr", "%x", __gu_val, (ptr), \
(err), ARM64_HAS_UAO); \
break; \
default: \
@@ -384,7 +384,7 @@ do { \
(err), ARM64_HAS_UAO); \
break; \
case 8: \
- __put_user_asm("str", "sttr", "%", __pu_val, (ptr), \
+ __put_user_asm("str", "sttr", "%x", __pu_val, (ptr), \
(err), ARM64_HAS_UAO); \
break; \
default: \
diff --git a/arch/arm64/include/asm/vdso.h b/arch/arm64/include/asm/vdso.h
index 839ce00..f2a9523 100644
--- a/arch/arm64/include/asm/vdso.h
+++ b/arch/arm64/include/asm/vdso.h
@@ -28,6 +28,9 @@
#ifndef __ASSEMBLY__
#include <generated/vdso-offsets.h>
+#ifdef CONFIG_VDSO32
+#include <generated/vdso32-offsets.h>
+#endif
#define VDSO_SYMBOL(base, name) \
({ \
diff --git a/arch/arm64/include/asm/vdso_datapage.h b/arch/arm64/include/asm/vdso_datapage.h
index 2b9a637..d776978 100644
--- a/arch/arm64/include/asm/vdso_datapage.h
+++ b/arch/arm64/include/asm/vdso_datapage.h
@@ -20,16 +20,37 @@
#ifndef __ASSEMBLY__
+#ifndef _VDSO_WTM_CLOCK_SEC_T
+#define _VDSO_WTM_CLOCK_SEC_T
+typedef __u64 vdso_wtm_clock_nsec_t;
+#endif
+
+#ifndef _VDSO_XTIME_CLOCK_SEC_T
+#define _VDSO_XTIME_CLOCK_SEC_T
+typedef __u64 vdso_xtime_clock_sec_t;
+#endif
+
+#ifndef _VDSO_RAW_TIME_SEC_T
+#define _VDSO_RAW_TIME_SEC_T
+typedef __u64 vdso_raw_time_sec_t;
+#endif
+
+#define USE_SYSCALL 0x1
+#define USE_SYSCALL_32 0x2
+#define USE_SYSCALL_64 0x4
+
struct vdso_data {
__u64 cs_cycle_last; /* Timebase at clocksource init */
- __u64 raw_time_sec; /* Raw time */
+ vdso_raw_time_sec_t raw_time_sec; /* Raw time */
__u64 raw_time_nsec;
- __u64 xtime_clock_sec; /* Kernel time */
- __u64 xtime_clock_nsec;
+ vdso_xtime_clock_sec_t xtime_clock_sec; /* Kernel time */
+ __u64 xtime_clock_snsec;
__u64 xtime_coarse_sec; /* Coarse time */
__u64 xtime_coarse_nsec;
__u64 wtm_clock_sec; /* Wall to monotonic time */
- __u64 wtm_clock_nsec;
+ vdso_wtm_clock_nsec_t wtm_clock_nsec;
+ __u32 btm_sec; /* monotonic to boot time */
+ __u32 btm_nsec;
__u32 tb_seq_count; /* Timebase sequence counter */
/* cs_* members must be adjacent and in this order (ldp accesses) */
__u32 cs_mono_mult; /* NTP-adjusted clocksource multiplier */
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 3051f86..702de7a 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -195,6 +195,12 @@ struct kvm_arch_memory_slot {
#define KVM_REG_ARM_TIMER_CNT ARM64_SYS_REG(3, 3, 14, 3, 2)
#define KVM_REG_ARM_TIMER_CVAL ARM64_SYS_REG(3, 3, 14, 0, 2)
+/* KVM-as-firmware specific pseudo-registers */
+#define KVM_REG_ARM_FW (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
+#define KVM_REG_ARM_FW_REG(r) (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
+ KVM_REG_ARM_FW | ((r) & 0xffff))
+#define KVM_REG_ARM_PSCI_VERSION KVM_REG_ARM_FW_REG(0)
+
/* Device Control API: ARM VGIC */
#define KVM_DEV_ARM_VGIC_GRP_ADDR 0
#define KVM_DEV_ARM_VGIC_GRP_DIST_REGS 1
diff --git a/arch/arm64/include/uapi/asm/setup.h b/arch/arm64/include/uapi/asm/setup.h
index 9cf2e46..7474c81 100644
--- a/arch/arm64/include/uapi/asm/setup.h
+++ b/arch/arm64/include/uapi/asm/setup.h
@@ -21,6 +21,6 @@
#include <linux/types.h>
-#define COMMAND_LINE_SIZE 2048
+#define COMMAND_LINE_SIZE 4096
#endif
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 446eabd..bc9a65a 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -27,8 +27,12 @@
$(obj)/%.stub.o: $(obj)/%.o FORCE
$(call if_changed,objcopy)
-arm64-obj-$(CONFIG_COMPAT) += sys32.o kuser32.o signal32.o \
+arm64-obj-$(CONFIG_COMPAT) += sys32.o signal32.o \
sys_compat.o entry32.o
+ifneq ($(CONFIG_VDSO32),y)
+arm64-obj-$(CONFIG_COMPAT) += sigreturn32.o
+endif
+arm64-obj-$(CONFIG_KUSER_HELPERS) += kuser32.o
arm64-obj-$(CONFIG_FUNCTION_TRACER) += ftrace.o entry-ftrace.o
arm64-obj-$(CONFIG_MODULES) += arm64ksyms.o module.o
arm64-obj-$(CONFIG_ARM64_MODULE_PLTS) += module-plts.o
@@ -51,12 +55,14 @@
arm64-obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o
arm64-obj-$(CONFIG_KEXEC) += machine_kexec.o relocate_kernel.o \
cpu-reset.o
+arm64-obj-$(CONFIG_ARM64_SSBD) += ssbd.o
ifeq ($(CONFIG_KVM),y)
arm64-obj-$(CONFIG_HARDEN_BRANCH_PREDICTOR) += bpi.o
endif
obj-y += $(arm64-obj-y) vdso/ probes/
+obj-$(CONFIG_VDSO32) += vdso32/
obj-m += $(arm64-obj-m)
head-y := head.o
extra-y += $(head-y) vmlinux.lds
diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c
index 06d650f..0917480 100644
--- a/arch/arm64/kernel/alternative.c
+++ b/arch/arm64/kernel/alternative.c
@@ -28,10 +28,12 @@
#include <asm/sections.h>
#include <linux/stop_machine.h>
-#define __ALT_PTR(a,f) (u32 *)((void *)&(a)->f + (a)->f)
+#define __ALT_PTR(a,f) ((void *)&(a)->f + (a)->f)
#define ALT_ORIG_PTR(a) __ALT_PTR(a, orig_offset)
#define ALT_REPL_PTR(a) __ALT_PTR(a, alt_offset)
+int alternatives_applied;
+
struct alt_region {
struct alt_instr *begin;
struct alt_instr *end;
@@ -105,31 +107,52 @@ static u32 get_alt_insn(struct alt_instr *alt, u32 *insnptr, u32 *altinsnptr)
return insn;
}
+static void patch_alternative(struct alt_instr *alt,
+ __le32 *origptr, __le32 *updptr, int nr_inst)
+{
+ __le32 *replptr;
+ int i;
+
+ replptr = ALT_REPL_PTR(alt);
+ for (i = 0; i < nr_inst; i++) {
+ u32 insn;
+
+ insn = get_alt_insn(alt, origptr + i, replptr + i);
+ updptr[i] = cpu_to_le32(insn);
+ }
+}
+
static void __apply_alternatives(void *alt_region)
{
struct alt_instr *alt;
struct alt_region *region = alt_region;
- u32 *origptr, *replptr;
+ __le32 *origptr;
+ alternative_cb_t alt_cb;
for (alt = region->begin; alt < region->end; alt++) {
- u32 insn;
- int i, nr_inst;
+ int nr_inst;
- if (!cpus_have_cap(alt->cpufeature))
+ /* Use ARM64_CB_PATCH as an unconditional patch */
+ if (alt->cpufeature < ARM64_CB_PATCH &&
+ !cpus_have_cap(alt->cpufeature))
continue;
- BUG_ON(alt->alt_len != alt->orig_len);
+ if (alt->cpufeature == ARM64_CB_PATCH)
+ BUG_ON(alt->alt_len != 0);
+ else
+ BUG_ON(alt->alt_len != alt->orig_len);
pr_info_once("patching kernel code\n");
origptr = ALT_ORIG_PTR(alt);
- replptr = ALT_REPL_PTR(alt);
- nr_inst = alt->alt_len / sizeof(insn);
+ nr_inst = alt->orig_len / AARCH64_INSN_SIZE;
- for (i = 0; i < nr_inst; i++) {
- insn = get_alt_insn(alt, origptr + i, replptr + i);
- *(origptr + i) = cpu_to_le32(insn);
- }
+ if (alt->cpufeature < ARM64_CB_PATCH)
+ alt_cb = patch_alternative;
+ else
+ alt_cb = ALT_REPL_PTR(alt);
+
+ alt_cb(alt, origptr, origptr, nr_inst);
flush_icache_range((uintptr_t)origptr,
(uintptr_t)(origptr + nr_inst));
@@ -142,7 +165,6 @@ static void __apply_alternatives(void *alt_region)
*/
static int __apply_alternatives_multi_stop(void *unused)
{
- static int patched = 0;
struct alt_region region = {
.begin = (struct alt_instr *)__alt_instructions,
.end = (struct alt_instr *)__alt_instructions_end,
@@ -150,14 +172,14 @@ static int __apply_alternatives_multi_stop(void *unused)
/* We always have a CPU 0 at this point (__init) */
if (smp_processor_id()) {
- while (!READ_ONCE(patched))
+ while (!READ_ONCE(alternatives_applied))
cpu_relax();
isb();
} else {
- BUG_ON(patched);
+ BUG_ON(alternatives_applied);
__apply_alternatives(®ion);
/* Barriers provided by the cache flushing */
- WRITE_ONCE(patched, 1);
+ WRITE_ONCE(alternatives_applied, 1);
}
return 0;
diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c
index 5d2d356..41df4662 100644
--- a/arch/arm64/kernel/asm-offsets.c
+++ b/arch/arm64/kernel/asm-offsets.c
@@ -27,6 +27,7 @@
#include <asm/fixmap.h>
#include <asm/thread_info.h>
#include <asm/memory.h>
+#include <asm/signal32.h>
#include <asm/smp_plat.h>
#include <asm/suspend.h>
#include <asm/vdso_datapage.h>
@@ -43,6 +44,9 @@ int main(void)
#ifdef CONFIG_ARM64_SW_TTBR0_PAN
DEFINE(TSK_TI_TTBR0, offsetof(struct task_struct, thread_info.ttbr0));
#endif
+#ifdef CONFIG_SHADOW_CALL_STACK
+ DEFINE(TSK_TI_SCS, offsetof(struct task_struct, thread_info.shadow_call_stack));
+#endif
DEFINE(TSK_STACK, offsetof(struct task_struct, stack));
BLANK();
DEFINE(THREAD_CPU_CONTEXT, offsetof(struct task_struct, thread.cpu_context));
@@ -78,6 +82,18 @@ int main(void)
DEFINE(S_ORIG_ADDR_LIMIT, offsetof(struct pt_regs, orig_addr_limit));
DEFINE(S_FRAME_SIZE, sizeof(struct pt_regs));
BLANK();
+#ifdef CONFIG_COMPAT
+ DEFINE(COMPAT_SIGFRAME_REGS_OFFSET,
+ offsetof(struct compat_sigframe, uc) +
+ offsetof(struct compat_ucontext, uc_mcontext) +
+ offsetof(struct compat_sigcontext, arm_r0));
+ DEFINE(COMPAT_RT_SIGFRAME_REGS_OFFSET,
+ offsetof(struct compat_rt_sigframe, sig) +
+ offsetof(struct compat_sigframe, uc) +
+ offsetof(struct compat_ucontext, uc_mcontext) +
+ offsetof(struct compat_sigcontext, arm_r0));
+ BLANK();
+#endif
DEFINE(MM_CONTEXT_ID, offsetof(struct mm_struct, context.id.counter));
BLANK();
DEFINE(VMA_VM_MM, offsetof(struct vm_area_struct, vm_mm));
@@ -91,50 +107,18 @@ int main(void)
DEFINE(DMA_TO_DEVICE, DMA_TO_DEVICE);
DEFINE(DMA_FROM_DEVICE, DMA_FROM_DEVICE);
BLANK();
- DEFINE(CLOCK_REALTIME, CLOCK_REALTIME);
- DEFINE(CLOCK_MONOTONIC, CLOCK_MONOTONIC);
- DEFINE(CLOCK_MONOTONIC_RAW, CLOCK_MONOTONIC_RAW);
- DEFINE(CLOCK_REALTIME_RES, MONOTONIC_RES_NSEC);
- DEFINE(CLOCK_REALTIME_COARSE, CLOCK_REALTIME_COARSE);
- DEFINE(CLOCK_MONOTONIC_COARSE,CLOCK_MONOTONIC_COARSE);
- DEFINE(CLOCK_COARSE_RES, LOW_RES_NSEC);
- DEFINE(NSEC_PER_SEC, NSEC_PER_SEC);
- BLANK();
- DEFINE(VDSO_CS_CYCLE_LAST, offsetof(struct vdso_data, cs_cycle_last));
- DEFINE(VDSO_RAW_TIME_SEC, offsetof(struct vdso_data, raw_time_sec));
- DEFINE(VDSO_RAW_TIME_NSEC, offsetof(struct vdso_data, raw_time_nsec));
- DEFINE(VDSO_XTIME_CLK_SEC, offsetof(struct vdso_data, xtime_clock_sec));
- DEFINE(VDSO_XTIME_CLK_NSEC, offsetof(struct vdso_data, xtime_clock_nsec));
- DEFINE(VDSO_XTIME_CRS_SEC, offsetof(struct vdso_data, xtime_coarse_sec));
- DEFINE(VDSO_XTIME_CRS_NSEC, offsetof(struct vdso_data, xtime_coarse_nsec));
- DEFINE(VDSO_WTM_CLK_SEC, offsetof(struct vdso_data, wtm_clock_sec));
- DEFINE(VDSO_WTM_CLK_NSEC, offsetof(struct vdso_data, wtm_clock_nsec));
- DEFINE(VDSO_TB_SEQ_COUNT, offsetof(struct vdso_data, tb_seq_count));
- DEFINE(VDSO_CS_MONO_MULT, offsetof(struct vdso_data, cs_mono_mult));
- DEFINE(VDSO_CS_RAW_MULT, offsetof(struct vdso_data, cs_raw_mult));
- DEFINE(VDSO_CS_SHIFT, offsetof(struct vdso_data, cs_shift));
- DEFINE(VDSO_TZ_MINWEST, offsetof(struct vdso_data, tz_minuteswest));
- DEFINE(VDSO_TZ_DSTTIME, offsetof(struct vdso_data, tz_dsttime));
- DEFINE(VDSO_USE_SYSCALL, offsetof(struct vdso_data, use_syscall));
- BLANK();
- DEFINE(TVAL_TV_SEC, offsetof(struct timeval, tv_sec));
- DEFINE(TVAL_TV_USEC, offsetof(struct timeval, tv_usec));
- DEFINE(TSPEC_TV_SEC, offsetof(struct timespec, tv_sec));
- DEFINE(TSPEC_TV_NSEC, offsetof(struct timespec, tv_nsec));
- BLANK();
- DEFINE(TZ_MINWEST, offsetof(struct timezone, tz_minuteswest));
- DEFINE(TZ_DSTTIME, offsetof(struct timezone, tz_dsttime));
- BLANK();
DEFINE(CPU_BOOT_STACK, offsetof(struct secondary_data, stack));
DEFINE(CPU_BOOT_TASK, offsetof(struct secondary_data, task));
BLANK();
#ifdef CONFIG_KVM_ARM_HOST
DEFINE(VCPU_CONTEXT, offsetof(struct kvm_vcpu, arch.ctxt));
+ DEFINE(VCPU_WORKAROUND_FLAGS, offsetof(struct kvm_vcpu, arch.workaround_flags));
DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs));
DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs));
DEFINE(CPU_FP_REGS, offsetof(struct kvm_regs, fp_regs));
DEFINE(VCPU_FPEXC32_EL2, offsetof(struct kvm_vcpu, arch.ctxt.sys_regs[FPEXC32_EL2]));
DEFINE(VCPU_HOST_CONTEXT, offsetof(struct kvm_vcpu, arch.host_cpu_context));
+ DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu));
#endif
#ifdef CONFIG_CPU_PM
DEFINE(CPU_SUSPEND_SZ, sizeof(struct cpu_suspend_ctx));
diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S
index f736a6f..6f9847f 100644
--- a/arch/arm64/kernel/cpu-reset.S
+++ b/arch/arm64/kernel/cpu-reset.S
@@ -44,11 +44,11 @@
mov x0, #HVC_SOFT_RESTART
hvc #0 // no return
-1: mov x18, x1 // entry
+1: mov x8, x1 // entry
mov x0, x2 // arg0
mov x1, x3 // arg1
mov x2, x4 // arg2
- br x18
+ br x8
ENDPROC(__cpu_soft_restart)
.popsection
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 49e548f..202f6304 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -17,6 +17,7 @@
*/
#include <linux/types.h>
+#include <asm/cachetype.h>
#include <asm/cpu.h>
#include <asm/cputype.h>
#include <asm/cpufeature.h>
@@ -31,12 +32,18 @@ is_affected_midr_range(const struct arm64_cpu_capabilities *entry, int scope)
}
static bool
-has_mismatched_cache_line_size(const struct arm64_cpu_capabilities *entry,
- int scope)
+has_mismatched_cache_type(const struct arm64_cpu_capabilities *entry,
+ int scope)
{
+ u64 mask = CTR_CACHE_MINLINE_MASK;
+
+ /* Skip matching the min line sizes for cache type check */
+ if (entry->capability == ARM64_MISMATCHED_CACHE_TYPE)
+ mask ^= arm64_ftr_reg_ctrel0.strict_mask;
+
WARN_ON(scope != SCOPE_LOCAL_CPU || preemptible());
- return (read_cpuid_cachetype() & arm64_ftr_reg_ctrel0.strict_mask) !=
- (arm64_ftr_reg_ctrel0.sys_val & arm64_ftr_reg_ctrel0.strict_mask);
+ return (read_cpuid_cachetype() & mask) !=
+ (arm64_ftr_reg_ctrel0.sys_val & mask);
}
static int cpu_enable_trap_ctr_access(void *__unused)
@@ -137,13 +144,18 @@ static void install_bp_hardening_cb(const struct arm64_cpu_capabilities *entry,
#include <linux/psci.h>
#ifdef CONFIG_PSCI_BP_HARDENING
+static void psci_ops_get_version(void)
+{
+ psci_ops.get_version();
+}
+
static int enable_psci_bp_hardening(void *data)
{
const struct arm64_cpu_capabilities *entry = data;
if (psci_ops.get_version)
install_bp_hardening_cb(entry,
- (bp_hardening_cb_t)psci_ops.get_version,
+ psci_ops_get_version,
__psci_hyp_bp_inval_start,
__psci_hyp_bp_inval_end);
return 0;
@@ -177,7 +189,7 @@ static int enable_smccc_arch_workaround_1(void *data)
case PSCI_CONDUIT_HVC:
arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID,
ARM_SMCCC_ARCH_WORKAROUND_1, &res);
- if (res.a0)
+ if ((int)res.a0 < 0)
return 0;
cb = call_hvc_arch_workaround_1;
smccc_start = __smccc_workaround_1_hvc_start;
@@ -187,7 +199,7 @@ static int enable_smccc_arch_workaround_1(void *data)
case PSCI_CONDUIT_SMC:
arm_smccc_1_1_smc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID,
ARM_SMCCC_ARCH_WORKAROUND_1, &res);
- if (res.a0)
+ if ((int)res.a0 < 0)
return 0;
cb = call_smc_arch_workaround_1;
smccc_start = __smccc_workaround_1_smc_start;
@@ -204,6 +216,178 @@ static int enable_smccc_arch_workaround_1(void *data)
}
#endif /* CONFIG_HARDEN_BRANCH_PREDICTOR */
+#ifdef CONFIG_ARM64_SSBD
+DEFINE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+
+int ssbd_state __read_mostly = ARM64_SSBD_KERNEL;
+
+static const struct ssbd_options {
+ const char *str;
+ int state;
+} ssbd_options[] = {
+ { "force-on", ARM64_SSBD_FORCE_ENABLE, },
+ { "force-off", ARM64_SSBD_FORCE_DISABLE, },
+ { "kernel", ARM64_SSBD_KERNEL, },
+};
+
+static int __init ssbd_cfg(char *buf)
+{
+ int i;
+
+ if (!buf || !buf[0])
+ return -EINVAL;
+
+ for (i = 0; i < ARRAY_SIZE(ssbd_options); i++) {
+ int len = strlen(ssbd_options[i].str);
+
+ if (strncmp(buf, ssbd_options[i].str, len))
+ continue;
+
+ ssbd_state = ssbd_options[i].state;
+ return 0;
+ }
+
+ return -EINVAL;
+}
+early_param("ssbd", ssbd_cfg);
+
+void __init arm64_update_smccc_conduit(struct alt_instr *alt,
+ __le32 *origptr, __le32 *updptr,
+ int nr_inst)
+{
+ u32 insn;
+
+ BUG_ON(nr_inst != 1);
+
+ switch (psci_ops.conduit) {
+ case PSCI_CONDUIT_HVC:
+ insn = aarch64_insn_get_hvc_value();
+ break;
+ case PSCI_CONDUIT_SMC:
+ insn = aarch64_insn_get_smc_value();
+ break;
+ default:
+ return;
+ }
+
+ *updptr = cpu_to_le32(insn);
+}
+
+void __init arm64_enable_wa2_handling(struct alt_instr *alt,
+ __le32 *origptr, __le32 *updptr,
+ int nr_inst)
+{
+ BUG_ON(nr_inst != 1);
+ /*
+ * Only allow mitigation on EL1 entry/exit and guest
+ * ARCH_WORKAROUND_2 handling if the SSBD state allows it to
+ * be flipped.
+ */
+ if (arm64_get_ssbd_state() == ARM64_SSBD_KERNEL)
+ *updptr = cpu_to_le32(aarch64_insn_gen_nop());
+}
+
+void arm64_set_ssbd_mitigation(bool state)
+{
+ switch (psci_ops.conduit) {
+ case PSCI_CONDUIT_HVC:
+ arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_WORKAROUND_2, state, NULL);
+ break;
+
+ case PSCI_CONDUIT_SMC:
+ arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, state, NULL);
+ break;
+
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
+}
+
+static bool has_ssbd_mitigation(const struct arm64_cpu_capabilities *entry,
+ int scope)
+{
+ struct arm_smccc_res res;
+ bool required = true;
+ s32 val;
+
+ WARN_ON(scope != SCOPE_LOCAL_CPU || preemptible());
+
+ if (psci_ops.smccc_version == SMCCC_VERSION_1_0) {
+ ssbd_state = ARM64_SSBD_UNKNOWN;
+ return false;
+ }
+
+ switch (psci_ops.conduit) {
+ case PSCI_CONDUIT_HVC:
+ arm_smccc_1_1_hvc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID,
+ ARM_SMCCC_ARCH_WORKAROUND_2, &res);
+ break;
+
+ case PSCI_CONDUIT_SMC:
+ arm_smccc_1_1_smc(ARM_SMCCC_ARCH_FEATURES_FUNC_ID,
+ ARM_SMCCC_ARCH_WORKAROUND_2, &res);
+ break;
+
+ default:
+ ssbd_state = ARM64_SSBD_UNKNOWN;
+ return false;
+ }
+
+ val = (s32)res.a0;
+
+ switch (val) {
+ case SMCCC_RET_NOT_SUPPORTED:
+ ssbd_state = ARM64_SSBD_UNKNOWN;
+ return false;
+
+ case SMCCC_RET_NOT_REQUIRED:
+ pr_info_once("%s mitigation not required\n", entry->desc);
+ ssbd_state = ARM64_SSBD_MITIGATED;
+ return false;
+
+ case SMCCC_RET_SUCCESS:
+ required = true;
+ break;
+
+ case 1: /* Mitigation not required on this CPU */
+ required = false;
+ break;
+
+ default:
+ WARN_ON(1);
+ return false;
+ }
+
+ switch (ssbd_state) {
+ case ARM64_SSBD_FORCE_DISABLE:
+ pr_info_once("%s disabled from command-line\n", entry->desc);
+ arm64_set_ssbd_mitigation(false);
+ required = false;
+ break;
+
+ case ARM64_SSBD_KERNEL:
+ if (required) {
+ __this_cpu_write(arm64_ssbd_callback_required, 1);
+ arm64_set_ssbd_mitigation(true);
+ }
+ break;
+
+ case ARM64_SSBD_FORCE_ENABLE:
+ pr_info_once("%s forced from command-line\n", entry->desc);
+ arm64_set_ssbd_mitigation(true);
+ required = true;
+ break;
+
+ default:
+ WARN_ON(1);
+ break;
+ }
+
+ return required;
+}
+#endif /* CONFIG_ARM64_SSBD */
+
#define MIDR_RANGE(model, min, max) \
.def_scope = SCOPE_LOCAL_CPU, \
.matches = is_affected_midr_range, \
@@ -294,7 +478,14 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
{
.desc = "Mismatched cache line size",
.capability = ARM64_MISMATCHED_CACHE_LINE_SIZE,
- .matches = has_mismatched_cache_line_size,
+ .matches = has_mismatched_cache_type,
+ .def_scope = SCOPE_LOCAL_CPU,
+ .enable = cpu_enable_trap_ctr_access,
+ },
+ {
+ .desc = "Mismatched cache type",
+ .capability = ARM64_MISMATCHED_CACHE_TYPE,
+ .matches = has_mismatched_cache_type,
.def_scope = SCOPE_LOCAL_CPU,
.enable = cpu_enable_trap_ctr_access,
},
@@ -348,6 +539,14 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
.enable = enable_smccc_arch_workaround_1,
},
#endif
+#ifdef CONFIG_ARM64_SSBD
+ {
+ .desc = "Speculative Store Bypass Disable",
+ .def_scope = SCOPE_LOCAL_CPU,
+ .capability = ARM64_SSBD,
+ .matches = has_ssbd_mitigation,
+ },
+#endif
{
}
};
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 675bf45..789a7c7 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -29,6 +29,7 @@
#include <asm/cpu_ops.h>
#include <asm/mmu_context.h>
#include <asm/processor.h>
+#include <asm/scs.h>
#include <asm/sysreg.h>
#include <asm/virt.h>
@@ -153,7 +154,7 @@ static const struct arm64_ftr_bits ftr_ctr[] = {
ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, 28, 3, 0),
ARM64_FTR_BITS(FTR_STRICT, FTR_HIGHER_SAFE, 24, 4, 0), /* CWG */
ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, 20, 4, 0), /* ERG */
- ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, 16, 4, 1), /* DminLine */
+ ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, CTR_DMINLINE_SHIFT, 4, 1),
/*
* Linux can handle differing I-cache policies. Userspace JITs will
* make use of *minLine.
@@ -161,7 +162,7 @@ static const struct arm64_ftr_bits ftr_ctr[] = {
*/
ARM64_FTR_BITS(FTR_NONSTRICT, FTR_EXACT, 14, 2, ICACHE_POLICY_AIVIVT), /* L1Ip */
ARM64_FTR_BITS(FTR_STRICT, FTR_EXACT, 4, 10, 0), /* RAZ */
- ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, 0, 4, 0), /* IminLine */
+ ARM64_FTR_BITS(FTR_STRICT, FTR_LOWER_SAFE, CTR_IMINLINE_SHIFT, 4, 0),
ARM64_FTR_END,
};
@@ -762,7 +763,7 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry,
* ThunderX leads to apparent I-cache corruption of kernel text, which
* ends as well as you might imagine. Don't even try.
*/
- if (cpus_have_cap(ARM64_WORKAROUND_CAVIUM_27456)) {
+ if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_27456)) {
str = "ARM64_WORKAROUND_CAVIUM_27456";
__kpti_forced = -1;
}
@@ -804,10 +805,14 @@ static int __nocfi kpti_install_ng_mappings(void *__unused)
remap_fn = (void *)__pa_symbol(idmap_kpti_install_ng_mappings);
+ scs_save(current);
+
cpu_install_idmap();
remap_fn(cpu, num_online_cpus(), __pa_symbol(swapper_pg_dir));
cpu_uninstall_idmap();
+ scs_load(current);
+
if (!cpu)
kpti_applied = true;
@@ -825,9 +830,25 @@ static int __init parse_kpti(char *str)
__kpti_forced = enabled ? 1 : -1;
return 0;
}
-__setup("kpti=", parse_kpti);
+early_param("kpti", parse_kpti);
#endif /* CONFIG_UNMAP_KERNEL_AT_EL0 */
+static int cpu_copy_el2regs(void *__unused)
+{
+ /*
+ * Copy register values that aren't redirected by hardware.
+ *
+ * Before code patching, we only set tpidr_el1, all CPUs need to copy
+ * this value to tpidr_el2 before we patch the code. Once we've done
+ * that, freshly-onlined CPUs will set tpidr_el2, so we don't need to
+ * do anything here.
+ */
+ if (!alternatives_applied)
+ write_sysreg(read_sysreg(tpidr_el1), tpidr_el2);
+
+ return 0;
+}
+
static const struct arm64_cpu_capabilities arm64_features[] = {
{
.desc = "GIC system register CPU interface",
@@ -894,6 +915,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
.capability = ARM64_HAS_VIRT_HOST_EXTN,
.def_scope = SCOPE_SYSTEM,
.matches = runs_at_el2,
+ .enable = cpu_copy_el2regs,
},
{
.desc = "32-bit EL0 Support",
@@ -1051,8 +1073,16 @@ void update_cpu_capabilities(const struct arm64_cpu_capabilities *caps,
*/
void __init enable_cpu_capabilities(const struct arm64_cpu_capabilities *caps)
{
- for (; caps->matches; caps++)
- if (caps->enable && cpus_have_cap(caps->capability))
+ for (; caps->matches; caps++) {
+ unsigned int num = caps->capability;
+
+ if (!cpus_have_cap(num))
+ continue;
+
+ /* Ensure cpus_have_const_cap(num) works */
+ static_branch_enable(&cpu_hwcap_keys[num]);
+
+ if (caps->enable) {
/*
* Use stop_machine() as it schedules the work allowing
* us to modify PSTATE, instead of on_each_cpu() which
@@ -1060,6 +1090,8 @@ void __init enable_cpu_capabilities(const struct arm64_cpu_capabilities *caps)
* we return.
*/
stop_machine(caps->enable, (void *)caps, cpu_online_mask);
+ }
+ }
}
/*
@@ -1163,6 +1195,14 @@ static void __init setup_feature_capabilities(void)
enable_cpu_capabilities(arm64_features);
}
+DEFINE_STATIC_KEY_FALSE(arm64_const_caps_ready);
+EXPORT_SYMBOL(arm64_const_caps_ready);
+
+static void __init mark_const_caps_ready(void)
+{
+ static_branch_enable(&arm64_const_caps_ready);
+}
+
extern const struct arm64_cpu_capabilities arm64_errata[];
bool this_cpu_has_cap(unsigned int cap)
@@ -1179,6 +1219,7 @@ void __init setup_cpu_features(void)
/* Set the CPU feature capabilies */
setup_feature_capabilities();
enable_errata_workarounds();
+ mark_const_caps_ready();
setup_elf_hwcaps(arm64_elf_hwcaps);
if (system_supports_32bit_el0())
@@ -1203,5 +1244,5 @@ void __init setup_cpu_features(void)
static bool __maybe_unused
cpufeature_pan_not_uao(const struct arm64_cpu_capabilities *entry, int __unused)
{
- return (cpus_have_cap(ARM64_HAS_PAN) && !cpus_have_cap(ARM64_HAS_UAO));
+ return (cpus_have_const_cap(ARM64_HAS_PAN) && !cpus_have_const_cap(ARM64_HAS_UAO));
}
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 7613ed1..ef2ff05 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -18,6 +18,7 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
+#include <linux/arm-smccc.h>
#include <linux/init.h>
#include <linux/linkage.h>
@@ -97,6 +98,25 @@
add \dst, \dst, #(\sym - .entry.tramp.text)
.endm
+ // This macro corrupts x0-x3. It is the caller's duty
+ // to save/restore them if required.
+ .macro apply_ssbd, state, targ, tmp1, tmp2
+#ifdef CONFIG_ARM64_SSBD
+alternative_cb arm64_enable_wa2_handling
+ b \targ
+alternative_cb_end
+ ldr_this_cpu \tmp2, arm64_ssbd_callback_required, \tmp1
+ cbz \tmp2, \targ
+ ldr \tmp2, [tsk, #TSK_TI_FLAGS]
+ tbnz \tmp2, #TIF_SSBD, \targ
+ mov w0, #ARM_SMCCC_ARCH_WORKAROUND_2
+ mov w1, #\state
+alternative_cb arm64_update_smccc_conduit
+ nop // Patched to SMC/HVC #0
+alternative_cb_end
+#endif
+ .endm
+
.macro kernel_entry, el, regsize = 64
.if \regsize == 32
mov w0, w0 // zero upper 32 bits of x0
@@ -123,7 +143,20 @@
ldr x19, [tsk, #TSK_TI_FLAGS] // since we can unmask debug
disable_step_tsk x19, x20 // exceptions when scheduling.
+ apply_ssbd 1, 1f, x22, x23
+
+#ifdef CONFIG_ARM64_SSBD
+ ldp x0, x1, [sp, #16 * 0]
+ ldp x2, x3, [sp, #16 * 1]
+#endif
+1:
+
mov x29, xzr // fp pointed to user-space
+
+#ifdef CONFIG_SHADOW_CALL_STACK
+ ldr x18, [tsk, #TSK_TI_SCS] // Restore shadow call stack
+ str xzr, [tsk, #TSK_TI_SCS]
+#endif
.else
add x21, sp, #S_FRAME_SIZE
get_thread_info tsk
@@ -203,6 +236,12 @@
ct_user_enter
.endif
+#ifdef CONFIG_SHADOW_CALL_STACK
+ .if \el == 0
+ str x18, [tsk, #TSK_TI_SCS] // Save shadow call stack
+ .endif
+#endif
+
#ifdef CONFIG_ARM64_SW_TTBR0_PAN
/*
* Restore access to TTBR0_EL1. If returning to EL0, no need for SPSR
@@ -251,6 +290,8 @@
alternative_else_nop_endif
#endif
3:
+ apply_ssbd 0, 5f, x0, x1
+5:
.endif
msr elr_el1, x21 // set up the return data
@@ -291,6 +332,9 @@
.macro irq_stack_entry
mov x19, sp // preserve the original sp
+#ifdef CONFIG_SHADOW_CALL_STACK
+ mov x20, x18 // preserve the original shadow stack
+#endif
/*
* Compare sp with the base of the task stack.
@@ -309,6 +353,11 @@
/* switch to the irq stack */
mov sp, x26
+#ifdef CONFIG_SHADOW_CALL_STACK
+ /* also switch to the irq shadow stack */
+ adr_this_cpu x18, irq_shadow_call_stack, x26
+#endif
+
/*
* Add a dummy stack frame, this non-standard format is fixed up
* by unwind_frame()
@@ -325,6 +374,10 @@
*/
.macro irq_stack_exit
mov sp, x19
+#ifdef CONFIG_SHADOW_CALL_STACK
+ /* x20 is also preserved */
+ mov x18, x20
+#endif
.endm
/*
diff --git a/arch/arm64/kernel/head.S b/arch/arm64/kernel/head.S
index d479185..f5c64d9 100644
--- a/arch/arm64/kernel/head.S
+++ b/arch/arm64/kernel/head.S
@@ -37,6 +37,7 @@
#include <asm/pgtable-hwdef.h>
#include <asm/pgtable.h>
#include <asm/page.h>
+#include <asm/scs.h>
#include <asm/smp.h>
#include <asm/sysreg.h>
#include <asm/thread_info.h>
@@ -438,6 +439,10 @@
stp xzr, x30, [sp, #-16]!
mov x29, sp
+#ifdef CONFIG_SHADOW_CALL_STACK
+ adr_l x18, init_shadow_call_stack // Set shadow call stack
+#endif
+
str_l x21, __fdt_pointer, x5 // Save FDT pointer
ldr_l x4, kimage_vaddr // Save the offset between
@@ -451,7 +456,6 @@
sub x2, x2, x0
bl __pi_memset
dsb ishst // Make zero page visible to PTW
-
#ifdef CONFIG_KASAN
bl kasan_early_init
#endif
@@ -705,6 +709,10 @@
mov sp, x1
ldr x2, [x0, #CPU_BOOT_TASK]
msr sp_el0, x2
+#ifdef CONFIG_SHADOW_CALL_STACK
+ ldr x18, [x2, #TSK_TI_SCS] // Set shadow call stack
+ str xzr, [x2, #TSK_TI_SCS]
+#endif
mov x29, #0
b secondary_start_kernel
ENDPROC(__secondary_switched)
diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c
index 8bed26a..db603cd 100644
--- a/arch/arm64/kernel/hibernate.c
+++ b/arch/arm64/kernel/hibernate.c
@@ -305,6 +305,17 @@ int swsusp_arch_suspend(void)
sleep_cpu = -EINVAL;
__cpu_suspend_exit();
+
+ /*
+ * Just in case the boot kernel did turn the SSBD
+ * mitigation off behind our back, let's set the state
+ * to what we expect it to be.
+ */
+ switch (arm64_get_ssbd_state()) {
+ case ARM64_SSBD_FORCE_ENABLE:
+ case ARM64_SSBD_KERNEL:
+ arm64_set_ssbd_mitigation(true);
+ }
}
local_dbg_restore(flags);
diff --git a/arch/arm64/kernel/irq.c b/arch/arm64/kernel/irq.c
index 2386b26..86d18c8 100644
--- a/arch/arm64/kernel/irq.c
+++ b/arch/arm64/kernel/irq.c
@@ -27,12 +27,18 @@
#include <linux/init.h>
#include <linux/irqchip.h>
#include <linux/seq_file.h>
+#include <asm/scs.h>
unsigned long irq_err_count;
/* irq stack only needs to be 16 byte aligned - not IRQ_STACK_SIZE aligned. */
DEFINE_PER_CPU(unsigned long [IRQ_STACK_SIZE/sizeof(long)], irq_stack) __aligned(16);
+#ifdef CONFIG_SHADOW_CALL_STACK
+DEFINE_PER_CPU(unsigned long [SCS_SIZE/sizeof(long)], irq_shadow_call_stack)
+ __aligned(SCS_SIZE);
+#endif
+
int arch_show_interrupts(struct seq_file *p, int prec)
{
show_ipi_list(p, prec);
diff --git a/arch/arm64/kernel/kuser32.S b/arch/arm64/kernel/kuser32.S
index 997e6b2..d15b5c2 100644
--- a/arch/arm64/kernel/kuser32.S
+++ b/arch/arm64/kernel/kuser32.S
@@ -20,16 +20,13 @@
*
* AArch32 user helpers.
*
- * Each segment is 32-byte aligned and will be moved to the top of the high
- * vector page. New segments (if ever needed) must be added in front of
- * existing ones. This mechanism should be used only for things that are
- * really small and justified, and not be abused freely.
+ * These helpers are provided for compatibility with AArch32 binaries that
+ * still need them. They are installed at a fixed address by
+ * aarch32_setup_additional_pages().
*
* See Documentation/arm/kernel_user_helpers.txt for formal definitions.
*/
-#include <asm/unistd.h>
-
.align 5
.globl __kuser_helper_start
__kuser_helper_start:
@@ -77,42 +74,3 @@
.word ((__kuser_helper_end - __kuser_helper_start) >> 5)
.globl __kuser_helper_end
__kuser_helper_end:
-
-/*
- * AArch32 sigreturn code
- *
- * For ARM syscalls, the syscall number has to be loaded into r7.
- * We do not support an OABI userspace.
- *
- * For Thumb syscalls, we also pass the syscall number via r7. We therefore
- * need two 16-bit instructions.
- */
- .globl __aarch32_sigret_code_start
-__aarch32_sigret_code_start:
-
- /*
- * ARM Code
- */
- .byte __NR_compat_sigreturn, 0x70, 0xa0, 0xe3 // mov r7, #__NR_compat_sigreturn
- .byte __NR_compat_sigreturn, 0x00, 0x00, 0xef // svc #__NR_compat_sigreturn
-
- /*
- * Thumb code
- */
- .byte __NR_compat_sigreturn, 0x27 // svc #__NR_compat_sigreturn
- .byte __NR_compat_sigreturn, 0xdf // mov r7, #__NR_compat_sigreturn
-
- /*
- * ARM code
- */
- .byte __NR_compat_rt_sigreturn, 0x70, 0xa0, 0xe3 // mov r7, #__NR_compat_rt_sigreturn
- .byte __NR_compat_rt_sigreturn, 0x00, 0x00, 0xef // svc #__NR_compat_rt_sigreturn
-
- /*
- * Thumb code
- */
- .byte __NR_compat_rt_sigreturn, 0x27 // svc #__NR_compat_rt_sigreturn
- .byte __NR_compat_rt_sigreturn, 0xdf // mov r7, #__NR_compat_rt_sigreturn
-
- .globl __aarch32_sigret_code_end
-__aarch32_sigret_code_end:
diff --git a/arch/arm64/kernel/probes/kprobes.c b/arch/arm64/kernel/probes/kprobes.c
index f5077ea..bf387b0 100644
--- a/arch/arm64/kernel/probes/kprobes.c
+++ b/arch/arm64/kernel/probes/kprobes.c
@@ -274,7 +274,7 @@ static int __kprobes reenter_kprobe(struct kprobe *p,
break;
case KPROBE_HIT_SS:
case KPROBE_REENTER:
- pr_warn("Unrecoverable kprobe detected at %p.\n", p->addr);
+ pr_warn("Unrecoverable kprobe detected.\n");
dump_kprobe(p);
BUG();
break;
@@ -639,6 +639,7 @@ void __kprobes __used *trampoline_probe_handler(struct pt_regs *regs)
return (void *)orig_ret_address;
}
+#ifdef CONFIG_KRETPROBES
void __kprobes arch_prepare_kretprobe(struct kretprobe_instance *ri,
struct pt_regs *regs)
{
@@ -652,6 +653,7 @@ int __kprobes arch_trampoline_kprobe(struct kprobe *p)
{
return 0;
}
+#endif
int __init arch_init_kprobes(void)
{
diff --git a/arch/arm64/kernel/process.c b/arch/arm64/kernel/process.c
index 08ca9dc..c7bd9ed 100644
--- a/arch/arm64/kernel/process.c
+++ b/arch/arm64/kernel/process.c
@@ -55,6 +55,7 @@
#include <asm/mmu_context.h>
#include <asm/processor.h>
#include <asm/stacktrace.h>
+#include <asm/scs.h>
#ifdef CONFIG_CC_STACKPROTECTOR
#include <linux/stackprotector.h>
@@ -360,7 +361,7 @@ int copy_thread(unsigned long clone_flags, unsigned long stack_start,
memset(childregs, 0, sizeof(struct pt_regs));
childregs->pstate = PSR_MODE_EL1h;
if (IS_ENABLED(CONFIG_ARM64_UAO) &&
- cpus_have_cap(ARM64_HAS_UAO))
+ cpus_have_const_cap(ARM64_HAS_UAO))
childregs->pstate |= PSR_UAO_BIT;
p->thread.cpu_context.x19 = stack_start;
p->thread.cpu_context.x20 = stk_sz;
@@ -428,6 +429,8 @@ struct task_struct *__switch_to(struct task_struct *prev,
entry_task_switch(next);
uao_thread_switch(next);
+ scs_thread_switch(prev, next);
+
/*
* Complete any pending TLB or cache maintenance on this CPU in case
* the thread migrates to a different CPU.
diff --git a/arch/arm64/kernel/signal32.c b/arch/arm64/kernel/signal32.c
index b7063de..b6e8ff7 100644
--- a/arch/arm64/kernel/signal32.c
+++ b/arch/arm64/kernel/signal32.c
@@ -28,42 +28,7 @@
#include <asm/signal32.h>
#include <asm/uaccess.h>
#include <asm/unistd.h>
-
-struct compat_sigcontext {
- /* We always set these two fields to 0 */
- compat_ulong_t trap_no;
- compat_ulong_t error_code;
-
- compat_ulong_t oldmask;
- compat_ulong_t arm_r0;
- compat_ulong_t arm_r1;
- compat_ulong_t arm_r2;
- compat_ulong_t arm_r3;
- compat_ulong_t arm_r4;
- compat_ulong_t arm_r5;
- compat_ulong_t arm_r6;
- compat_ulong_t arm_r7;
- compat_ulong_t arm_r8;
- compat_ulong_t arm_r9;
- compat_ulong_t arm_r10;
- compat_ulong_t arm_fp;
- compat_ulong_t arm_ip;
- compat_ulong_t arm_sp;
- compat_ulong_t arm_lr;
- compat_ulong_t arm_pc;
- compat_ulong_t arm_cpsr;
- compat_ulong_t fault_address;
-};
-
-struct compat_ucontext {
- compat_ulong_t uc_flags;
- compat_uptr_t uc_link;
- compat_stack_t uc_stack;
- struct compat_sigcontext uc_mcontext;
- compat_sigset_t uc_sigmask;
- int __unused[32 - (sizeof (compat_sigset_t) / sizeof (int))];
- compat_ulong_t uc_regspace[128] __attribute__((__aligned__(8)));
-};
+#include <asm/vdso.h>
struct compat_vfp_sigframe {
compat_ulong_t magic;
@@ -91,16 +56,6 @@ struct compat_aux_sigframe {
unsigned long end_magic;
} __attribute__((__aligned__(8)));
-struct compat_sigframe {
- struct compat_ucontext uc;
- compat_ulong_t retcode[2];
-};
-
-struct compat_rt_sigframe {
- struct compat_siginfo info;
- struct compat_sigframe sig;
-};
-
#define _BLOCKABLE (~(sigmask(SIGKILL) | sigmask(SIGSTOP)))
static inline int put_sigset_t(compat_sigset_t __user *uset, sigset_t *set)
@@ -484,14 +439,27 @@ static void compat_setup_return(struct pt_regs *regs, struct k_sigaction *ka,
retcode = ptr_to_compat(ka->sa.sa_restorer);
} else {
/* Set up sigreturn pointer */
+#ifdef CONFIG_VDSO32
+ void *vdso_base = current->mm->context.vdso;
+ void *trampoline =
+ (ka->sa.sa_flags & SA_SIGINFO
+ ? (thumb
+ ? VDSO_SYMBOL(vdso_base, compat_rt_sigreturn_thumb)
+ : VDSO_SYMBOL(vdso_base, compat_rt_sigreturn_arm))
+ : (thumb
+ ? VDSO_SYMBOL(vdso_base, compat_sigreturn_thumb)
+ : VDSO_SYMBOL(vdso_base, compat_sigreturn_arm)));
+
+ retcode = ptr_to_compat(trampoline) + thumb;
+#else
+ void *sigreturn_base = current->mm->context.vdso;
unsigned int idx = thumb << 1;
if (ka->sa.sa_flags & SA_SIGINFO)
idx += 3;
- retcode = AARCH32_VECTORS_BASE +
- AARCH32_KERN_SIGRET_CODE_OFFSET +
- (idx << 2) + thumb;
+ retcode = ptr_to_compat(sigreturn_base) + (idx << 2) + thumb;
+#endif
}
regs->regs[0] = usig;
diff --git a/arch/arm64/kernel/sigreturn32.S b/arch/arm64/kernel/sigreturn32.S
new file mode 100644
index 0000000..6ecda4d
--- /dev/null
+++ b/arch/arm64/kernel/sigreturn32.S
@@ -0,0 +1,67 @@
+/*
+ * sigreturn trampolines for AArch32.
+ *
+ * Copyright (C) 2005-2011 Nicolas Pitre <nico@fluxnic.net>
+ * Copyright (C) 2012 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ *
+ * AArch32 sigreturn code
+ *
+ * For ARM syscalls, the syscall number has to be loaded into r7.
+ * We do not support an OABI userspace.
+ *
+ * For Thumb syscalls, we also pass the syscall number via r7. We therefore
+ * need two 16-bit instructions.
+ */
+
+#include <asm/unistd.h>
+
+ .globl __aarch32_sigret_code_start
+__aarch32_sigret_code_start:
+
+ /*
+ * ARM Code
+ */
+ // mov r7, #__NR_compat_sigreturn
+ .byte __NR_compat_sigreturn, 0x70, 0xa0, 0xe3
+ // svc #__NR_compat_sigreturn
+ .byte __NR_compat_sigreturn, 0x00, 0x00, 0xef
+
+ /*
+ * Thumb code
+ */
+ // svc #__NR_compat_sigreturn
+ .byte __NR_compat_sigreturn, 0x27
+ // mov r7, #__NR_compat_sigreturn
+ .byte __NR_compat_sigreturn, 0xdf
+
+ /*
+ * ARM code
+ */
+ // mov r7, #__NR_compat_rt_sigreturn
+ .byte __NR_compat_rt_sigreturn, 0x70, 0xa0, 0xe3
+ // svc #__NR_compat_rt_sigreturn
+ .byte __NR_compat_rt_sigreturn, 0x00, 0x00, 0xef
+
+ /*
+ * Thumb code
+ */
+ // svc #__NR_compat_rt_sigreturn
+ .byte __NR_compat_rt_sigreturn, 0x27
+ // mov r7, #__NR_compat_rt_sigreturn
+ .byte __NR_compat_rt_sigreturn, 0xdf
+
+ .globl __aarch32_sigret_code_end
+__aarch32_sigret_code_end:
diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index 623dd48..d5046e4 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -49,6 +49,7 @@
#include <asm/pgtable.h>
#include <asm/pgalloc.h>
#include <asm/processor.h>
+#include <asm/scs.h>
#include <asm/smp_plat.h>
#include <asm/sections.h>
#include <asm/tlbflush.h>
@@ -212,7 +213,7 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
* This is the secondary CPU boot entry. We're using this CPUs
* idle thread stack, but a set of temporary page tables.
*/
-asmlinkage void secondary_start_kernel(void)
+asmlinkage notrace void secondary_start_kernel(void)
{
struct mm_struct *mm = &init_mm;
unsigned int cpu;
@@ -377,6 +378,9 @@ void cpu_die(void)
{
unsigned int cpu = smp_processor_id();
+ /* Save the shadow stack pointer before exiting the idle task */
+ scs_save(current);
+
idle_task_exit();
local_irq_disable();
diff --git a/arch/arm64/kernel/ssbd.c b/arch/arm64/kernel/ssbd.c
new file mode 100644
index 0000000..0560738
--- /dev/null
+++ b/arch/arm64/kernel/ssbd.c
@@ -0,0 +1,108 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2018 ARM Ltd, All Rights Reserved.
+ */
+
+#include <linux/errno.h>
+#include <linux/prctl.h>
+#include <linux/sched.h>
+#include <linux/thread_info.h>
+
+#include <asm/cpufeature.h>
+
+/*
+ * prctl interface for SSBD
+ */
+static int ssbd_prctl_set(struct task_struct *task, unsigned long ctrl)
+{
+ int state = arm64_get_ssbd_state();
+
+ /* Unsupported */
+ if (state == ARM64_SSBD_UNKNOWN)
+ return -EINVAL;
+
+ /* Treat the unaffected/mitigated state separately */
+ if (state == ARM64_SSBD_MITIGATED) {
+ switch (ctrl) {
+ case PR_SPEC_ENABLE:
+ return -EPERM;
+ case PR_SPEC_DISABLE:
+ case PR_SPEC_FORCE_DISABLE:
+ return 0;
+ }
+ }
+
+ /*
+ * Things are a bit backward here: the arm64 internal API
+ * *enables the mitigation* when the userspace API *disables
+ * speculation*. So much fun.
+ */
+ switch (ctrl) {
+ case PR_SPEC_ENABLE:
+ /* If speculation is force disabled, enable is not allowed */
+ if (state == ARM64_SSBD_FORCE_ENABLE ||
+ task_spec_ssb_force_disable(task))
+ return -EPERM;
+ task_clear_spec_ssb_disable(task);
+ clear_tsk_thread_flag(task, TIF_SSBD);
+ break;
+ case PR_SPEC_DISABLE:
+ if (state == ARM64_SSBD_FORCE_DISABLE)
+ return -EPERM;
+ task_set_spec_ssb_disable(task);
+ set_tsk_thread_flag(task, TIF_SSBD);
+ break;
+ case PR_SPEC_FORCE_DISABLE:
+ if (state == ARM64_SSBD_FORCE_DISABLE)
+ return -EPERM;
+ task_set_spec_ssb_disable(task);
+ task_set_spec_ssb_force_disable(task);
+ set_tsk_thread_flag(task, TIF_SSBD);
+ break;
+ default:
+ return -ERANGE;
+ }
+
+ return 0;
+}
+
+int arch_prctl_spec_ctrl_set(struct task_struct *task, unsigned long which,
+ unsigned long ctrl)
+{
+ switch (which) {
+ case PR_SPEC_STORE_BYPASS:
+ return ssbd_prctl_set(task, ctrl);
+ default:
+ return -ENODEV;
+ }
+}
+
+static int ssbd_prctl_get(struct task_struct *task)
+{
+ switch (arm64_get_ssbd_state()) {
+ case ARM64_SSBD_UNKNOWN:
+ return -EINVAL;
+ case ARM64_SSBD_FORCE_ENABLE:
+ return PR_SPEC_DISABLE;
+ case ARM64_SSBD_KERNEL:
+ if (task_spec_ssb_force_disable(task))
+ return PR_SPEC_PRCTL | PR_SPEC_FORCE_DISABLE;
+ if (task_spec_ssb_disable(task))
+ return PR_SPEC_PRCTL | PR_SPEC_DISABLE;
+ return PR_SPEC_PRCTL | PR_SPEC_ENABLE;
+ case ARM64_SSBD_FORCE_DISABLE:
+ return PR_SPEC_ENABLE;
+ default:
+ return PR_SPEC_NOT_AFFECTED;
+ }
+}
+
+int arch_prctl_spec_ctrl_get(struct task_struct *task, unsigned long which)
+{
+ switch (which) {
+ case PR_SPEC_STORE_BYPASS:
+ return ssbd_prctl_get(task);
+ default:
+ return -ENODEV;
+ }
+}
diff --git a/arch/arm64/kernel/stacktrace.c b/arch/arm64/kernel/stacktrace.c
index f7ce3d2..4fa6d84 100644
--- a/arch/arm64/kernel/stacktrace.c
+++ b/arch/arm64/kernel/stacktrace.c
@@ -73,6 +73,11 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame)
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
if (tsk->ret_stack &&
(frame->pc == (unsigned long)return_to_handler)) {
+ if (WARN_ON_ONCE(frame->graph == -1))
+ return -EINVAL;
+ if (frame->graph < -1)
+ frame->graph += FTRACE_NOTRACE_DEPTH;
+
/*
* This is a case where function graph tracer has
* modified a return address (LR) in a stack frame
diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c
index 1e3be90..e12f2d0 100644
--- a/arch/arm64/kernel/suspend.c
+++ b/arch/arm64/kernel/suspend.c
@@ -61,6 +61,14 @@ void notrace __cpu_suspend_exit(void)
*/
if (hw_breakpoint_restore)
hw_breakpoint_restore(cpu);
+
+ /*
+ * On resume, firmware implementing dynamic mitigation will
+ * have turned the mitigation on. If the user has forcefully
+ * disabled it, make sure their wishes are obeyed.
+ */
+ if (arm64_get_ssbd_state() == ARM64_SSBD_FORCE_DISABLE)
+ arm64_set_ssbd_mitigation(false);
}
/*
diff --git a/arch/arm64/kernel/time.c b/arch/arm64/kernel/time.c
index 5977969..5d9076e 100644
--- a/arch/arm64/kernel/time.c
+++ b/arch/arm64/kernel/time.c
@@ -53,7 +53,7 @@ unsigned long profile_pc(struct pt_regs *regs)
frame.sp = regs->sp;
frame.pc = regs->pc;
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
- frame.graph = -1; /* no task info */
+ frame.graph = current->curr_ret_stack;
#endif
do {
int ret = unwind_frame(NULL, &frame);
diff --git a/arch/arm64/kernel/vdso.c b/arch/arm64/kernel/vdso.c
index e8f759f..11b624e 100644
--- a/arch/arm64/kernel/vdso.c
+++ b/arch/arm64/kernel/vdso.c
@@ -1,5 +1,7 @@
/*
- * VDSO implementation for AArch64 and vector page setup for AArch32.
+ * Additional userspace pages setup for AArch64 and AArch32.
+ * - AArch64: vDSO pages setup, vDSO data page update.
+ * - AArch32: sigreturn and kuser helpers pages setup.
*
* Copyright (C) 2012 ARM Limited
*
@@ -26,6 +28,7 @@
#include <linux/gfp.h>
#include <linux/kernel.h>
#include <linux/mm.h>
+#include <linux/moduleparam.h>
#include <linux/sched.h>
#include <linux/signal.h>
#include <linux/slab.h>
@@ -37,8 +40,24 @@
#include <asm/vdso.h>
#include <asm/vdso_datapage.h>
-extern char vdso_start[], vdso_end[];
-static unsigned long vdso_pages __ro_after_init;
+#ifdef USE_SYSCALL
+#if defined(__LP64__)
+static int enable_64 = 1;
+module_param(enable_64, int, 0600);
+MODULE_PARM_DESC(enable_64, "enable vDSO for aarch64 0=off");
+#endif
+#if (!defined(__LP64) || (defined(CONFIG_COMPAT) && defined(CONFIG_VDSO32)))
+static int enable_32 = 1;
+module_param(enable_32, int, 0600);
+MODULE_PARM_DESC(enable_32, "enable vDSO for aarch64 0=off");
+#endif
+#endif
+
+struct vdso_mappings {
+ unsigned long num_code_pages;
+ struct vm_special_mapping data_mapping;
+ struct vm_special_mapping code_mapping;
+};
/*
* The vDSO data page.
@@ -53,151 +72,265 @@ struct vdso_data *vdso_data = &vdso_data_store.data;
/*
* Create and map the vectors page for AArch32 tasks.
*/
-static struct page *vectors_page[1] __ro_after_init;
+#if !defined(CONFIG_VDSO32) || defined(CONFIG_KUSER_HELPERS)
+static struct page *vectors_page[] __ro_after_init;
+static const struct vm_special_mapping compat_vdso_spec[] = {
+ {
+ /* Must be named [sigpage] for compatibility with arm. */
+ .name = "[sigpage]",
+ .pages = &vectors_page[0],
+ },
+#ifdef CONFIG_KUSER_HELPERS
+ {
+ .name = "[kuserhelpers]",
+ .pages = &vectors_page[1],
+ },
+#endif
+};
+static struct page *vectors_page[ARRAY_SIZE(compat_vdso_spec)] __ro_after_init;
+#endif
static int __init alloc_vectors_page(void)
{
+#ifdef CONFIG_KUSER_HELPERS
extern char __kuser_helper_start[], __kuser_helper_end[];
+ size_t kuser_sz = __kuser_helper_end - __kuser_helper_start;
+ unsigned long kuser_vpage;
+#endif
+
+#ifndef CONFIG_VDSO32
extern char __aarch32_sigret_code_start[], __aarch32_sigret_code_end[];
+ size_t sigret_sz =
+ __aarch32_sigret_code_end - __aarch32_sigret_code_start;
+ unsigned long sigret_vpage;
- int kuser_sz = __kuser_helper_end - __kuser_helper_start;
- int sigret_sz = __aarch32_sigret_code_end - __aarch32_sigret_code_start;
- unsigned long vpage;
-
- vpage = get_zeroed_page(GFP_ATOMIC);
-
- if (!vpage)
+ sigret_vpage = get_zeroed_page(GFP_ATOMIC);
+ if (!sigret_vpage)
return -ENOMEM;
+#endif
- /* kuser helpers */
- memcpy((void *)vpage + 0x1000 - kuser_sz, __kuser_helper_start,
- kuser_sz);
+#ifdef CONFIG_KUSER_HELPERS
+ kuser_vpage = get_zeroed_page(GFP_ATOMIC);
+ if (!kuser_vpage) {
+#ifndef CONFIG_VDSO32
+ free_page(sigret_vpage);
+#endif
+ return -ENOMEM;
+ }
+#endif
+#ifndef CONFIG_VDSO32
/* sigreturn code */
- memcpy((void *)vpage + AARCH32_KERN_SIGRET_CODE_OFFSET,
- __aarch32_sigret_code_start, sigret_sz);
+ memcpy((void *)sigret_vpage, __aarch32_sigret_code_start, sigret_sz);
+ flush_icache_range(sigret_vpage, sigret_vpage + PAGE_SIZE);
+ vectors_page[0] = virt_to_page(sigret_vpage);
+#endif
- flush_icache_range(vpage, vpage + PAGE_SIZE);
- vectors_page[0] = virt_to_page(vpage);
+#ifdef CONFIG_KUSER_HELPERS
+ /* kuser helpers */
+ memcpy((void *)kuser_vpage + 0x1000 - kuser_sz, __kuser_helper_start,
+ kuser_sz);
+ flush_icache_range(kuser_vpage, kuser_vpage + PAGE_SIZE);
+ vectors_page[1] = virt_to_page(kuser_vpage);
+#endif
return 0;
}
arch_initcall(alloc_vectors_page);
+#ifndef CONFIG_VDSO32
int aarch32_setup_vectors_page(struct linux_binprm *bprm, int uses_interp)
{
struct mm_struct *mm = current->mm;
- unsigned long addr = AARCH32_VECTORS_BASE;
- static const struct vm_special_mapping spec = {
- .name = "[vectors]",
- .pages = vectors_page,
-
- };
+ unsigned long addr;
void *ret;
if (down_write_killable(&mm->mmap_sem))
return -EINTR;
+ addr = get_unmapped_area(NULL, 0, PAGE_SIZE, 0, 0);
+ if (IS_ERR_VALUE(addr)) {
+ ret = ERR_PTR(addr);
+ goto out;
+ }
+
+ ret = _install_special_mapping(mm, addr, PAGE_SIZE,
+ VM_READ|VM_EXEC|
+ VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
+ &compat_vdso_spec[0]);
+ if (IS_ERR(ret))
+ goto out;
+
current->mm->context.vdso = (void *)addr;
- /* Map vectors page at the high address. */
- ret = _install_special_mapping(mm, addr, PAGE_SIZE,
+#ifdef CONFIG_KUSER_HELPERS
+ /* Map the kuser helpers at the ABI-defined high address. */
+ ret = _install_special_mapping(mm, AARCH32_KUSER_HELPERS_BASE,
+ PAGE_SIZE,
VM_READ|VM_EXEC|VM_MAYREAD|VM_MAYEXEC,
- &spec);
-
+ &compat_vdso_spec[1]);
+#endif
+out:
up_write(&mm->mmap_sem);
return PTR_ERR_OR_ZERO(ret);
}
+#endif /* !CONFIG_VDSO32 */
#endif /* CONFIG_COMPAT */
-static struct vm_special_mapping vdso_spec[2] __ro_after_init = {
- {
- .name = "[vvar]",
- },
- {
- .name = "[vdso]",
- },
-};
-
-static int __init vdso_init(void)
+static int __init vdso_mappings_init(const char *name,
+ const char *code_start,
+ const char *code_end,
+ struct vdso_mappings *mappings)
{
- int i;
+ unsigned long i, vdso_pages;
struct page **vdso_pagelist;
unsigned long pfn;
- if (memcmp(vdso_start, "\177ELF", 4)) {
- pr_err("vDSO is not a valid ELF object!\n");
+ if (memcmp(code_start, "\177ELF", 4)) {
+ pr_err("%sis not a valid ELF object!\n", name);
return -EINVAL;
}
- vdso_pages = (vdso_end - vdso_start) >> PAGE_SHIFT;
- pr_info("vdso: %ld pages (%ld code @ %p, %ld data @ %p)\n",
- vdso_pages + 1, vdso_pages, vdso_start, 1L, vdso_data);
+ vdso_pages = (code_end - code_start) >> PAGE_SHIFT;
+ pr_info("%s: %ld pages (%ld code @ %p, %ld data @ %p)\n",
+ name, vdso_pages + 1, vdso_pages, code_start, 1L, vdso_data);
/* Allocate the vDSO pagelist, plus a page for the data. */
- vdso_pagelist = kcalloc(vdso_pages + 1, sizeof(struct page *),
- GFP_KERNEL);
+ /*
+ * Allocate space for storing pointers to the vDSO code pages + the
+ * data page. The pointers must have the same lifetime as the mappings,
+ * which are static, so there is no need to keep track of the pointer
+ * array to free it.
+ */
+ vdso_pagelist = kmalloc_array(vdso_pages + 1, sizeof(struct page *),
+ GFP_KERNEL);
if (vdso_pagelist == NULL)
return -ENOMEM;
/* Grab the vDSO data page. */
vdso_pagelist[0] = phys_to_page(__pa_symbol(vdso_data));
-
/* Grab the vDSO code pages. */
- pfn = sym_to_pfn(vdso_start);
+ pfn = sym_to_pfn(code_start);
for (i = 0; i < vdso_pages; i++)
vdso_pagelist[i + 1] = pfn_to_page(pfn + i);
- vdso_spec[0].pages = &vdso_pagelist[0];
- vdso_spec[1].pages = &vdso_pagelist[1];
+ /* Populate the special mapping structures */
+ mappings->data_mapping = (struct vm_special_mapping) {
+ .name = "[vvar]",
+ .pages = &vdso_pagelist[0],
+ };
+
+ mappings->code_mapping = (struct vm_special_mapping) {
+ .name = "[vdso]",
+ .pages = &vdso_pagelist[1],
+ };
+
+ mappings->num_code_pages = vdso_pages;
return 0;
}
+
+#ifdef CONFIG_COMPAT
+#ifdef CONFIG_VDSO32
+
+static struct vdso_mappings vdso32_mappings __ro_after_init;
+
+static int __init vdso32_init(void)
+{
+ extern char vdso32_start[], vdso32_end[];
+
+ return vdso_mappings_init("vdso32", vdso32_start, vdso32_end,
+ &vdso32_mappings);
+}
+arch_initcall(vdso32_init);
+
+#endif /* CONFIG_VDSO32 */
+#endif /* CONFIG_COMPAT */
+
+static struct vdso_mappings vdso_mappings __ro_after_init;
+
+static int __init vdso_init(void)
+{
+ extern char vdso_start[], vdso_end[];
+
+ return vdso_mappings_init("vdso", vdso_start, vdso_end,
+ &vdso_mappings);
+}
arch_initcall(vdso_init);
-int arch_setup_additional_pages(struct linux_binprm *bprm,
- int uses_interp)
+static int vdso_setup(struct mm_struct *mm,
+ const struct vdso_mappings *mappings)
{
- struct mm_struct *mm = current->mm;
unsigned long vdso_base, vdso_text_len, vdso_mapping_len;
void *ret;
- vdso_text_len = vdso_pages << PAGE_SHIFT;
+ vdso_text_len = mappings->num_code_pages << PAGE_SHIFT;
/* Be sure to map the data page */
vdso_mapping_len = vdso_text_len + PAGE_SIZE;
- if (down_write_killable(&mm->mmap_sem))
- return -EINTR;
vdso_base = get_unmapped_area(NULL, 0, vdso_mapping_len, 0, 0);
- if (IS_ERR_VALUE(vdso_base)) {
- ret = ERR_PTR(vdso_base);
- goto up_fail;
- }
+ if (IS_ERR_VALUE(vdso_base))
+ return PTR_ERR_OR_ZERO(ERR_PTR(vdso_base));
+
ret = _install_special_mapping(mm, vdso_base, PAGE_SIZE,
VM_READ|VM_MAYREAD,
- &vdso_spec[0]);
+ &mappings->data_mapping);
if (IS_ERR(ret))
- goto up_fail;
+ return PTR_ERR_OR_ZERO(ret);
vdso_base += PAGE_SIZE;
- mm->context.vdso = (void *)vdso_base;
ret = _install_special_mapping(mm, vdso_base, vdso_text_len,
VM_READ|VM_EXEC|
VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
- &vdso_spec[1]);
- if (IS_ERR(ret))
- goto up_fail;
+ &mappings->code_mapping);
+ if (!IS_ERR(ret))
+ mm->context.vdso = (void *)vdso_base;
+ return PTR_ERR_OR_ZERO(ret);
+}
+
+#ifdef CONFIG_COMPAT
+#ifdef CONFIG_VDSO32
+int aarch32_setup_vectors_page(struct linux_binprm *bprm, int uses_interp)
+{
+ struct mm_struct *mm = current->mm;
+ void *ret;
+
+ if (down_write_killable(&mm->mmap_sem))
+ return -EINTR;
+
+ ret = ERR_PTR(vdso_setup(mm, &vdso32_mappings));
+#ifdef CONFIG_KUSER_HELPERS
+ if (!IS_ERR(ret))
+ /* Map the kuser helpers at the ABI-defined high address. */
+ ret = _install_special_mapping(mm, AARCH32_KUSER_HELPERS_BASE,
+ PAGE_SIZE,
+ VM_READ|VM_EXEC|
+ VM_MAYREAD|VM_MAYEXEC,
+ &compat_vdso_spec[1]);
+#endif
up_write(&mm->mmap_sem);
- return 0;
-up_fail:
- mm->context.vdso = NULL;
+ return PTR_ERR_OR_ZERO(ret);
+}
+#endif /* CONFIG_VDSO32 */
+#endif /* CONFIG_COMPAT */
+
+int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
+{
+ struct mm_struct *mm = current->mm;
+ int ret;
+
+ if (down_write_killable(&mm->mmap_sem))
+ return -EINTR;
+
+ ret = vdso_setup(mm, &vdso_mappings);
+
up_write(&mm->mmap_sem);
- return PTR_ERR(ret);
+ return ret;
}
/*
@@ -207,6 +340,23 @@ void update_vsyscall(struct timekeeper *tk)
{
u32 use_syscall = !tk->tkr_mono.clock->archdata.vdso_direct;
+#ifdef USE_SYSCALL
+ if (use_syscall) {
+ use_syscall = USE_SYSCALL | USE_SYSCALL_32 | USE_SYSCALL_64;
+ } else {
+#if (defined(__LP64__))
+ if (!enable_64)
+#endif
+ use_syscall = USE_SYSCALL_64;
+#if (!defined(__LP64) || (defined(CONFIG_COMPAT) && defined(CONFIG_VDSO32)))
+ if (!enable_32)
+#endif
+ use_syscall |= USE_SYSCALL_32;
+ if (use_syscall == (USE_SYSCALL_32 | USE_SYSCALL_64))
+ use_syscall |= USE_SYSCALL;
+ }
+#endif
+
++vdso_data->tb_seq_count;
smp_wmb();
@@ -217,17 +367,25 @@ void update_vsyscall(struct timekeeper *tk)
vdso_data->wtm_clock_sec = tk->wall_to_monotonic.tv_sec;
vdso_data->wtm_clock_nsec = tk->wall_to_monotonic.tv_nsec;
+#ifdef USE_SYSCALL
+ if (!(use_syscall & USE_SYSCALL)) {
+#else
if (!use_syscall) {
+#endif
+ struct timespec btm = ktime_to_timespec(tk->offs_boot);
+
/* tkr_mono.cycle_last == tkr_raw.cycle_last */
vdso_data->cs_cycle_last = tk->tkr_mono.cycle_last;
vdso_data->raw_time_sec = tk->raw_sec;
vdso_data->raw_time_nsec = tk->tkr_raw.xtime_nsec;
vdso_data->xtime_clock_sec = tk->xtime_sec;
- vdso_data->xtime_clock_nsec = tk->tkr_mono.xtime_nsec;
+ vdso_data->xtime_clock_snsec = tk->tkr_mono.xtime_nsec;
vdso_data->cs_mono_mult = tk->tkr_mono.mult;
vdso_data->cs_raw_mult = tk->tkr_raw.mult;
/* tkr_mono.shift == tkr_raw.shift */
vdso_data->cs_shift = tk->tkr_mono.shift;
+ vdso_data->btm_sec = btm.tv_sec;
+ vdso_data->btm_nsec = btm.tv_nsec;
}
smp_wmb();
diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
index 88fef38..8c70015 100644
--- a/arch/arm64/kernel/vdso/Makefile
+++ b/arch/arm64/kernel/vdso/Makefile
@@ -5,19 +5,33 @@
# Heavily based on the vDSO Makefiles for other archs.
#
-obj-vdso := gettimeofday.o note.o sigreturn.o
+obj-vdso-s := note.o sigreturn.o
+obj-vdso-c := vgettimeofday.o
# Build rules
-targets := $(obj-vdso) vdso.so vdso.so.dbg
-obj-vdso := $(addprefix $(obj)/, $(obj-vdso))
+targets := $(obj-vdso-s) $(obj-vdso-c) vdso.so vdso.so.dbg
+obj-vdso-s := $(addprefix $(obj)/, $(obj-vdso-s))
+obj-vdso-c := $(addprefix $(obj)/, $(obj-vdso-c))
+obj-vdso := $(obj-vdso-c) $(obj-vdso-s)
-ccflags-y := -shared -fno-common -fno-builtin
+ccflags-y := -shared -fno-common -fno-builtin -fno-stack-protector -ffixed-x18
+ccflags-y += -DDISABLE_BRANCH_PROFILING
ccflags-y += -nostdlib -Wl,-soname=linux-vdso.so.1 \
$(call cc-ldoption, -Wl$(comma)--hash-style=sysv)
-ccflags-y += $(DISABLE_LTO)
+ccflags-y += $(DISABLE_LTO) $(DISABLE_SCS)
+
+# Force -O2 to avoid libgcc dependencies
+CFLAGS_REMOVE_vgettimeofday.o = -pg -Os
+CFLAGS_vgettimeofday.o = -O2 -fPIC
+ifneq ($(cc-name),clang)
+CFLAGS_vgettimeofday.o += -mcmodel=tiny
+endif
# Disable gcov profiling for VDSO code
GCOV_PROFILE := n
+KASAN_SANITIZE := n
+UBSAN_SANITIZE := n
+KCOV_INSTRUMENT := n
# Workaround for bare-metal (ELF) toolchains that neglect to pass -shared
# down to collect2, resulting in silent corruption of the vDSO image.
@@ -50,12 +64,17 @@
$(call if_changed,vdsosym)
# Assembly rules for the .S files
-$(obj-vdso): %.o: %.S FORCE
+$(obj-vdso-s): %.o: %.S FORCE
$(call if_changed_dep,vdsoas)
+$(obj-vdso-c): %.o: %.c FORCE
+ $(call if_changed_dep,vdsocc)
+
# Actual build commands
quiet_cmd_vdsold = VDSOL $@
cmd_vdsold = $(CC) $(c_flags) -Wl,-n -Wl,-T $^ -o $@
+quiet_cmd_vdsocc = VDSOC $@
+ cmd_vdsocc = ${CC} $(c_flags) -c -o $@ $<
quiet_cmd_vdsoas = VDSOA $@
cmd_vdsoas = $(CC) $(a_flags) -c -o $@ $<
diff --git a/arch/arm64/kernel/vdso/compiler.h b/arch/arm64/kernel/vdso/compiler.h
new file mode 100644
index 0000000..fb27545
--- /dev/null
+++ b/arch/arm64/kernel/vdso/compiler.h
@@ -0,0 +1,70 @@
+/*
+ * Userspace implementations of fallback calls
+ *
+ * Copyright (C) 2017 Cavium, Inc.
+ * Copyright (C) 2012 ARM Limited
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Author: Will Deacon <will.deacon@arm.com>
+ * Rewriten into C by: Andrew Pinski <apinski@cavium.com>
+ */
+
+#ifndef __VDSO_COMPILER_H
+#define __VDSO_COMPILER_H
+
+#include <asm/processor.h> /* for cpu_relax() */
+#include <asm/sysreg.h> /* for read_sysreg() */
+#include <asm/unistd.h>
+#include <linux/compiler.h>
+#include <linux/hrtimer.h> /* for LOW_RES_NSEC and MONOTONIC_RES_NSEC */
+
+#ifdef CONFIG_ARM_ARCH_TIMER
+#define ARCH_PROVIDES_TIMER
+#endif
+
+#define DEFINE_FALLBACK(name, type_arg1, name_arg1, type_arg2, name_arg2) \
+static notrace long name##_fallback(type_arg1 _##name_arg1, \
+ type_arg2 _##name_arg2) \
+{ \
+ register type_arg1 name_arg1 asm("x0") = _##name_arg1; \
+ register type_arg2 name_arg2 asm("x1") = _##name_arg2; \
+ register long ret asm ("x0"); \
+ register long nr asm("x8") = __NR_##name; \
+ \
+ asm volatile( \
+ " svc #0\n" \
+ : "=r" (ret) \
+ : "r" (name_arg1), "r" (name_arg2), "r" (nr) \
+ : "memory"); \
+ \
+ return ret; \
+}
+
+/*
+ * AArch64 implementation of arch_counter_get_cntvct() suitable for vdso
+ */
+static __always_inline notrace u64 arch_vdso_read_counter(void)
+{
+ /* Read the virtual counter. */
+ isb();
+ return read_sysreg(cntvct_el0);
+}
+
+/* Rename exported vdso functions */
+#define __vdso_clock_gettime __kernel_clock_gettime
+#define __vdso_gettimeofday __kernel_gettimeofday
+#define __vdso_clock_getres __kernel_clock_getres
+#define __vdso_time __kernel_time
+
+#endif /* __VDSO_COMPILER_H */
diff --git a/arch/arm64/kernel/vdso/datapage.h b/arch/arm64/kernel/vdso/datapage.h
new file mode 100644
index 0000000..be86a6074
--- /dev/null
+++ b/arch/arm64/kernel/vdso/datapage.h
@@ -0,0 +1,59 @@
+/*
+ * Userspace implementations of __get_datapage
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __VDSO_DATAPAGE_H
+#define __VDSO_DATAPAGE_H
+
+#include <linux/bitops.h>
+#include <linux/types.h>
+#include <asm/vdso_datapage.h>
+
+/*
+ * We use the hidden visibility to prevent the compiler from generating a GOT
+ * relocation. Not only is going through a GOT useless (the entry couldn't and
+ * mustn't be overridden by another library), it does not even work: the linker
+ * cannot generate an absolute address to the data page.
+ *
+ * With the hidden visibility, the compiler simply generates a PC-relative
+ * relocation (R_ARM_REL32), and this is what we need.
+ */
+extern const struct vdso_data _vdso_data __attribute__((visibility("hidden")));
+
+static inline const struct vdso_data *__get_datapage(void)
+{
+ const struct vdso_data *ret;
+ /*
+ * This simply puts &_vdso_data into ret. The reason why we don't use
+ * `ret = &_vdso_data` is that the compiler tends to optimise this in a
+ * very suboptimal way: instead of keeping &_vdso_data in a register,
+ * it goes through a relocation almost every time _vdso_data must be
+ * accessed (even in subfunctions). This is both time and space
+ * consuming: each relocation uses a word in the code section, and it
+ * has to be loaded at runtime.
+ *
+ * This trick hides the assignment from the compiler. Since it cannot
+ * track where the pointer comes from, it will only use one relocation
+ * where __get_datapage() is called, and then keep the result in a
+ * register.
+ */
+ asm("" : "=r"(ret) : "0"(&_vdso_data));
+ return ret;
+}
+
+/* We can only guarantee 56 bits of precision. */
+#define ARCH_CLOCK_FIXED_MASK GENMASK_ULL(55, 0)
+
+#endif /* __VDSO_DATAPAGE_H */
diff --git a/arch/arm64/kernel/vdso/gettimeofday.S b/arch/arm64/kernel/vdso/gettimeofday.S
deleted file mode 100644
index c39872a..0000000
--- a/arch/arm64/kernel/vdso/gettimeofday.S
+++ /dev/null
@@ -1,328 +0,0 @@
-/*
- * Userspace implementations of gettimeofday() and friends.
- *
- * Copyright (C) 2012 ARM Limited
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- *
- * Author: Will Deacon <will.deacon@arm.com>
- */
-
-#include <linux/linkage.h>
-#include <asm/asm-offsets.h>
-#include <asm/unistd.h>
-
-#define NSEC_PER_SEC_LO16 0xca00
-#define NSEC_PER_SEC_HI16 0x3b9a
-
-vdso_data .req x6
-seqcnt .req w7
-w_tmp .req w8
-x_tmp .req x8
-
-/*
- * Conventions for macro arguments:
- * - An argument is write-only if its name starts with "res".
- * - All other arguments are read-only, unless otherwise specified.
- */
-
- .macro seqcnt_acquire
-9999: ldr seqcnt, [vdso_data, #VDSO_TB_SEQ_COUNT]
- tbnz seqcnt, #0, 9999b
- dmb ishld
- .endm
-
- .macro seqcnt_check fail
- dmb ishld
- ldr w_tmp, [vdso_data, #VDSO_TB_SEQ_COUNT]
- cmp w_tmp, seqcnt
- b.ne \fail
- .endm
-
- .macro syscall_check fail
- ldr w_tmp, [vdso_data, #VDSO_USE_SYSCALL]
- cbnz w_tmp, \fail
- .endm
-
- .macro get_nsec_per_sec res
- mov \res, #NSEC_PER_SEC_LO16
- movk \res, #NSEC_PER_SEC_HI16, lsl #16
- .endm
-
- /*
- * Returns the clock delta, in nanoseconds left-shifted by the clock
- * shift.
- */
- .macro get_clock_shifted_nsec res, cycle_last, mult
- /* Read the virtual counter. */
- isb
- mrs x_tmp, cntvct_el0
- /* Calculate cycle delta and convert to ns. */
- sub \res, x_tmp, \cycle_last
- /* We can only guarantee 56 bits of precision. */
- movn x_tmp, #0xff00, lsl #48
- and \res, x_tmp, \res
- mul \res, \res, \mult
- .endm
-
- /*
- * Returns in res_{sec,nsec} the REALTIME timespec, based on the
- * "wall time" (xtime) and the clock_mono delta.
- */
- .macro get_ts_realtime res_sec, res_nsec, \
- clock_nsec, xtime_sec, xtime_nsec, nsec_to_sec
- add \res_nsec, \clock_nsec, \xtime_nsec
- udiv x_tmp, \res_nsec, \nsec_to_sec
- add \res_sec, \xtime_sec, x_tmp
- msub \res_nsec, x_tmp, \nsec_to_sec, \res_nsec
- .endm
-
- /*
- * Returns in res_{sec,nsec} the timespec based on the clock_raw delta,
- * used for CLOCK_MONOTONIC_RAW.
- */
- .macro get_ts_clock_raw res_sec, res_nsec, clock_nsec, nsec_to_sec
- udiv \res_sec, \clock_nsec, \nsec_to_sec
- msub \res_nsec, \res_sec, \nsec_to_sec, \clock_nsec
- .endm
-
- /* sec and nsec are modified in place. */
- .macro add_ts sec, nsec, ts_sec, ts_nsec, nsec_to_sec
- /* Add timespec. */
- add \sec, \sec, \ts_sec
- add \nsec, \nsec, \ts_nsec
-
- /* Normalise the new timespec. */
- cmp \nsec, \nsec_to_sec
- b.lt 9999f
- sub \nsec, \nsec, \nsec_to_sec
- add \sec, \sec, #1
-9999:
- cmp \nsec, #0
- b.ge 9998f
- add \nsec, \nsec, \nsec_to_sec
- sub \sec, \sec, #1
-9998:
- .endm
-
- .macro clock_gettime_return, shift=0
- .if \shift == 1
- lsr x11, x11, x12
- .endif
- stp x10, x11, [x1, #TSPEC_TV_SEC]
- mov x0, xzr
- ret
- .endm
-
- .macro jump_slot jumptable, index, label
- .if (. - \jumptable) != 4 * (\index)
- .error "Jump slot index mismatch"
- .endif
- b \label
- .endm
-
- .text
-
-/* int __kernel_gettimeofday(struct timeval *tv, struct timezone *tz); */
-ENTRY(__kernel_gettimeofday)
- .cfi_startproc
- adr vdso_data, _vdso_data
- /* If tv is NULL, skip to the timezone code. */
- cbz x0, 2f
-
- /* Compute the time of day. */
-1: seqcnt_acquire
- syscall_check fail=4f
- ldr x10, [vdso_data, #VDSO_CS_CYCLE_LAST]
- /* w11 = cs_mono_mult, w12 = cs_shift */
- ldp w11, w12, [vdso_data, #VDSO_CS_MONO_MULT]
- ldp x13, x14, [vdso_data, #VDSO_XTIME_CLK_SEC]
- seqcnt_check fail=1b
-
- get_nsec_per_sec res=x9
- lsl x9, x9, x12
-
- get_clock_shifted_nsec res=x15, cycle_last=x10, mult=x11
- get_ts_realtime res_sec=x10, res_nsec=x11, \
- clock_nsec=x15, xtime_sec=x13, xtime_nsec=x14, nsec_to_sec=x9
-
- /* Convert ns to us. */
- mov x13, #1000
- lsl x13, x13, x12
- udiv x11, x11, x13
- stp x10, x11, [x0, #TVAL_TV_SEC]
-2:
- /* If tz is NULL, return 0. */
- cbz x1, 3f
- ldp w4, w5, [vdso_data, #VDSO_TZ_MINWEST]
- stp w4, w5, [x1, #TZ_MINWEST]
-3:
- mov x0, xzr
- ret
-4:
- /* Syscall fallback. */
- mov x8, #__NR_gettimeofday
- svc #0
- ret
- .cfi_endproc
-ENDPROC(__kernel_gettimeofday)
-
-#define JUMPSLOT_MAX CLOCK_MONOTONIC_COARSE
-
-/* int __kernel_clock_gettime(clockid_t clock_id, struct timespec *tp); */
-ENTRY(__kernel_clock_gettime)
- .cfi_startproc
- cmp w0, #JUMPSLOT_MAX
- b.hi syscall
- adr vdso_data, _vdso_data
- adr x_tmp, jumptable
- add x_tmp, x_tmp, w0, uxtw #2
- br x_tmp
-
- ALIGN
-jumptable:
- jump_slot jumptable, CLOCK_REALTIME, realtime
- jump_slot jumptable, CLOCK_MONOTONIC, monotonic
- b syscall
- b syscall
- jump_slot jumptable, CLOCK_MONOTONIC_RAW, monotonic_raw
- jump_slot jumptable, CLOCK_REALTIME_COARSE, realtime_coarse
- jump_slot jumptable, CLOCK_MONOTONIC_COARSE, monotonic_coarse
-
- .if (. - jumptable) != 4 * (JUMPSLOT_MAX + 1)
- .error "Wrong jumptable size"
- .endif
-
- ALIGN
-realtime:
- seqcnt_acquire
- syscall_check fail=syscall
- ldr x10, [vdso_data, #VDSO_CS_CYCLE_LAST]
- /* w11 = cs_mono_mult, w12 = cs_shift */
- ldp w11, w12, [vdso_data, #VDSO_CS_MONO_MULT]
- ldp x13, x14, [vdso_data, #VDSO_XTIME_CLK_SEC]
- seqcnt_check fail=realtime
-
- /* All computations are done with left-shifted nsecs. */
- get_nsec_per_sec res=x9
- lsl x9, x9, x12
-
- get_clock_shifted_nsec res=x15, cycle_last=x10, mult=x11
- get_ts_realtime res_sec=x10, res_nsec=x11, \
- clock_nsec=x15, xtime_sec=x13, xtime_nsec=x14, nsec_to_sec=x9
- clock_gettime_return, shift=1
-
- ALIGN
-monotonic:
- seqcnt_acquire
- syscall_check fail=syscall
- ldr x10, [vdso_data, #VDSO_CS_CYCLE_LAST]
- /* w11 = cs_mono_mult, w12 = cs_shift */
- ldp w11, w12, [vdso_data, #VDSO_CS_MONO_MULT]
- ldp x13, x14, [vdso_data, #VDSO_XTIME_CLK_SEC]
- ldp x3, x4, [vdso_data, #VDSO_WTM_CLK_SEC]
- seqcnt_check fail=monotonic
-
- /* All computations are done with left-shifted nsecs. */
- lsl x4, x4, x12
- get_nsec_per_sec res=x9
- lsl x9, x9, x12
-
- get_clock_shifted_nsec res=x15, cycle_last=x10, mult=x11
- get_ts_realtime res_sec=x10, res_nsec=x11, \
- clock_nsec=x15, xtime_sec=x13, xtime_nsec=x14, nsec_to_sec=x9
-
- add_ts sec=x10, nsec=x11, ts_sec=x3, ts_nsec=x4, nsec_to_sec=x9
- clock_gettime_return, shift=1
-
- ALIGN
-monotonic_raw:
- seqcnt_acquire
- syscall_check fail=syscall
- ldr x10, [vdso_data, #VDSO_CS_CYCLE_LAST]
- /* w11 = cs_raw_mult, w12 = cs_shift */
- ldp w12, w11, [vdso_data, #VDSO_CS_SHIFT]
- ldp x13, x14, [vdso_data, #VDSO_RAW_TIME_SEC]
- seqcnt_check fail=monotonic_raw
-
- /* All computations are done with left-shifted nsecs. */
- get_nsec_per_sec res=x9
- lsl x9, x9, x12
-
- get_clock_shifted_nsec res=x15, cycle_last=x10, mult=x11
- get_ts_clock_raw res_sec=x10, res_nsec=x11, \
- clock_nsec=x15, nsec_to_sec=x9
-
- add_ts sec=x10, nsec=x11, ts_sec=x13, ts_nsec=x14, nsec_to_sec=x9
- clock_gettime_return, shift=1
-
- ALIGN
-realtime_coarse:
- seqcnt_acquire
- ldp x10, x11, [vdso_data, #VDSO_XTIME_CRS_SEC]
- seqcnt_check fail=realtime_coarse
- clock_gettime_return
-
- ALIGN
-monotonic_coarse:
- seqcnt_acquire
- ldp x10, x11, [vdso_data, #VDSO_XTIME_CRS_SEC]
- ldp x13, x14, [vdso_data, #VDSO_WTM_CLK_SEC]
- seqcnt_check fail=monotonic_coarse
-
- /* Computations are done in (non-shifted) nsecs. */
- get_nsec_per_sec res=x9
- add_ts sec=x10, nsec=x11, ts_sec=x13, ts_nsec=x14, nsec_to_sec=x9
- clock_gettime_return
-
- ALIGN
-syscall: /* Syscall fallback. */
- mov x8, #__NR_clock_gettime
- svc #0
- ret
- .cfi_endproc
-ENDPROC(__kernel_clock_gettime)
-
-/* int __kernel_clock_getres(clockid_t clock_id, struct timespec *res); */
-ENTRY(__kernel_clock_getres)
- .cfi_startproc
- cmp w0, #CLOCK_REALTIME
- ccmp w0, #CLOCK_MONOTONIC, #0x4, ne
- ccmp w0, #CLOCK_MONOTONIC_RAW, #0x4, ne
- b.ne 1f
-
- ldr x2, 5f
- b 2f
-1:
- cmp w0, #CLOCK_REALTIME_COARSE
- ccmp w0, #CLOCK_MONOTONIC_COARSE, #0x4, ne
- b.ne 4f
- ldr x2, 6f
-2:
- cbz x1, 3f
- stp xzr, x2, [x1]
-
-3: /* res == NULL. */
- mov w0, wzr
- ret
-
-4: /* Syscall fallback. */
- mov x8, #__NR_clock_getres
- svc #0
- ret
-5:
- .quad CLOCK_REALTIME_RES
-6:
- .quad CLOCK_COARSE_RES
- .cfi_endproc
-ENDPROC(__kernel_clock_getres)
diff --git a/arch/arm64/kernel/vdso/vdso.lds.S b/arch/arm64/kernel/vdso/vdso.lds.S
index beca249..9de0ffc 100644
--- a/arch/arm64/kernel/vdso/vdso.lds.S
+++ b/arch/arm64/kernel/vdso/vdso.lds.S
@@ -88,6 +88,7 @@
__kernel_gettimeofday;
__kernel_clock_gettime;
__kernel_clock_getres;
+ __kernel_time;
local: *;
};
}
diff --git a/arch/arm64/kernel/vdso/vgettimeofday.c b/arch/arm64/kernel/vdso/vgettimeofday.c
new file mode 100644
index 0000000..b73d401
--- /dev/null
+++ b/arch/arm64/kernel/vdso/vgettimeofday.c
@@ -0,0 +1,3 @@
+#include "compiler.h"
+#include "datapage.h"
+#include "../../../../lib/vdso/vgettimeofday.c"
diff --git a/arch/arm64/kernel/vdso32/.gitignore b/arch/arm64/kernel/vdso32/.gitignore
new file mode 100644
index 0000000..4fea950
--- /dev/null
+++ b/arch/arm64/kernel/vdso32/.gitignore
@@ -0,0 +1,2 @@
+vdso.lds
+vdso.so.raw
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
new file mode 100644
index 0000000..837e877
--- /dev/null
+++ b/arch/arm64/kernel/vdso32/Makefile
@@ -0,0 +1,172 @@
+#
+# Building a vDSO image for AArch32.
+#
+# Author: Kevin Brodsky <kevin.brodsky@arm.com>
+# A mix between the arm64 and arm vDSO Makefiles.
+
+ifeq ($(cc-name),clang)
+ CC_ARM32 := $(CC) $(CLANG_TARGET_ARM32) -no-integrated-as
+else
+ CC_ARM32 := $(CROSS_COMPILE_ARM32)$(cc-name)
+endif
+
+# Same as cc-*option, but using CC_ARM32 instead of CC
+cc32-option = $(call try-run,\
+ $(CC_ARM32) $(1) -c -x c /dev/null -o "$$TMP",$(1),$(2))
+cc32-disable-warning = $(call try-run,\
+ $(CC_ARM32) -W$(strip $(1)) -c -x c /dev/null -o "$$TMP",-Wno-$(strip $(1)))
+cc32-ldoption = $(call try-run,\
+ $(CC_ARM32) $(1) -nostdlib -x c /dev/null -o "$$TMP",$(1),$(2))
+
+# We cannot use the global flags to compile the vDSO files, the main reason
+# being that the 32-bit compiler may be older than the main (64-bit) compiler
+# and therefore may not understand flags set using $(cc-option ...). Besides,
+# arch-specific options should be taken from the arm Makefile instead of the
+# arm64 one.
+# As a result we set our own flags here.
+
+# From top-level Makefile
+# NOSTDINC_FLAGS
+VDSO_CPPFLAGS := -nostdinc -isystem $(shell $(CC_ARM32) -print-file-name=include)
+VDSO_CPPFLAGS += $(LINUXINCLUDE)
+VDSO_CPPFLAGS += $(KBUILD_CPPFLAGS)
+
+# Common C and assembly flags
+# From top-level Makefile
+VDSO_CAFLAGS := $(VDSO_CPPFLAGS)
+VDSO_CAFLAGS += $(call cc32-option,-fno-PIE)
+ifdef CONFIG_DEBUG_INFO
+VDSO_CAFLAGS += -g
+endif
+ifeq ($(shell $(CONFIG_SHELL) $(srctree)/scripts/gcc-goto.sh $(CC_ARM32)), y)
+VDSO_CAFLAGS += -DCC_HAVE_ASM_GOTO
+endif
+
+# From arm Makefile
+VDSO_CAFLAGS += $(call cc32-option,-fno-dwarf2-cfi-asm)
+VDSO_CAFLAGS += -mabi=aapcs-linux -mfloat-abi=soft
+ifeq ($(CONFIG_CPU_BIG_ENDIAN), y)
+VDSO_CAFLAGS += -mbig-endian
+else
+VDSO_CAFLAGS += -mlittle-endian
+endif
+
+# From arm vDSO Makefile
+VDSO_CAFLAGS += -fPIC -fno-builtin -fno-stack-protector
+VDSO_CAFLAGS += -DDISABLE_BRANCH_PROFILING
+
+# Try to compile for ARMv8. If the compiler is too old and doesn't support it,
+# fall back to v7. There is no easy way to check for what architecture the code
+# is being compiled, so define a macro specifying that (see arch/arm/Makefile).
+VDSO_CAFLAGS += $(call cc32-option,-march=armv8-a -D__LINUX_ARM_ARCH__=8,\
+ -march=armv7-a -D__LINUX_ARM_ARCH__=7)
+
+VDSO_CFLAGS := $(VDSO_CAFLAGS)
+# KBUILD_CFLAGS from top-level Makefile
+VDSO_CFLAGS += -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs \
+ -fno-strict-aliasing -fno-common \
+ -Werror-implicit-function-declaration \
+ -Wno-format-security \
+ -std=gnu89
+VDSO_CFLAGS += -O2
+# Some useful compiler-dependent flags from top-level Makefile
+VDSO_CFLAGS += $(call cc32-option,-Wdeclaration-after-statement,)
+VDSO_CFLAGS += $(call cc32-option,-Wno-pointer-sign)
+VDSO_CFLAGS += $(call cc32-option,-fno-strict-overflow)
+VDSO_CFLAGS += $(call cc32-option,-Werror=strict-prototypes)
+VDSO_CFLAGS += $(call cc32-option,-Werror=date-time)
+VDSO_CFLAGS += $(call cc32-option,-Werror=incompatible-pointer-types)
+
+# The 32-bit compiler does not provide 128-bit integers, which are used in
+# some headers that are indirectly included from the vDSO code.
+# This hack makes the compiler happy and should trigger a warning/error if
+# variables of such type are referenced.
+VDSO_CFLAGS += -D__uint128_t='void*'
+# Silence some warnings coming from headers that operate on long's
+# (on GCC 4.8 or older, there is unfortunately no way to silence this warning)
+VDSO_CFLAGS += $(call cc32-disable-warning,shift-count-overflow)
+VDSO_CFLAGS += -Wno-int-to-pointer-cast
+
+VDSO_AFLAGS := $(VDSO_CAFLAGS)
+VDSO_AFLAGS += -D__ASSEMBLY__
+
+VDSO_LDFLAGS := $(VDSO_CPPFLAGS)
+# From arm vDSO Makefile
+VDSO_LDFLAGS += -Wl,-Bsymbolic -Wl,--no-undefined -Wl,-soname=linux-vdso.so.1
+VDSO_LDFLAGS += -Wl,-z,max-page-size=4096 -Wl,-z,common-page-size=4096
+VDSO_LDFLAGS += -nostdlib -shared -mfloat-abi=soft
+VDSO_LDFLAGS += $(call cc32-ldoption,-Wl$(comma)--hash-style=sysv)
+VDSO_LDFLAGS += $(call cc32-ldoption,-Wl$(comma)--build-id)
+VDSO_LDFLAGS += $(call cc32-ldoption,-fuse-ld=bfd)
+
+
+# Borrow vdsomunge.c from the arm vDSO
+# We have to use a relative path because scripts/Makefile.host prefixes
+# $(hostprogs-y) with $(obj)
+munge := ../../../arm/vdso/vdsomunge
+hostprogs-y := $(munge)
+
+c-obj-vdso := vgettimeofday.o
+asm-obj-vdso := sigreturn.o
+
+# Build rules
+targets := $(c-obj-vdso) $(asm-obj-vdso) vdso.so vdso.so.dbg vdso.so.raw
+c-obj-vdso := $(addprefix $(obj)/, $(c-obj-vdso))
+asm-obj-vdso := $(addprefix $(obj)/, $(asm-obj-vdso))
+obj-vdso := $(c-obj-vdso) $(asm-obj-vdso)
+
+obj-y += vdso.o
+extra-y += vdso.lds
+CPPFLAGS_vdso.lds += -P -C -U$(ARCH)
+
+# Force dependency (vdso.s includes vdso.so through incbin)
+$(obj)/vdso.o: $(obj)/vdso.so
+
+include/generated/vdso32-offsets.h: $(obj)/vdso.so.dbg FORCE
+ $(call if_changed,vdsosym)
+
+# Strip rule for vdso.so
+$(obj)/vdso.so: OBJCOPYFLAGS := -S
+$(obj)/vdso.so: $(obj)/vdso.so.dbg FORCE
+ $(call if_changed,objcopy)
+
+$(obj)/vdso.so.dbg: $(obj)/vdso.so.raw $(obj)/$(munge) FORCE
+ $(call if_changed,vdsomunge)
+
+# Link rule for the .so file, .lds has to be first
+$(obj)/vdso.so.raw: $(src)/vdso.lds $(obj-vdso) FORCE
+ $(call if_changed,vdsold)
+
+# Compilation rules for the vDSO sources
+$(filter-out vgettimeofday.o, $(c-obj-vdso)): %.o: %.c FORCE
+ $(call if_changed_dep,vdsocc)
+$(asm-obj-vdso): %.o: %.S FORCE
+ $(call if_changed_dep,vdsoas)
+
+# Actual build commands
+quiet_cmd_vdsold = VDSOL32 $@
+ cmd_vdsold = $(CC_ARM32) -Wp,-MD,$(depfile) $(VDSO_LDFLAGS) \
+ -Wl,-T $(filter %.lds,$^) $(filter %.o,$^) -o $@
+quiet_cmd_vdsocc = VDSOC32 $@
+ cmd_vdsocc = $(CC_ARM32) -Wp,-MD,$(depfile) $(VDSO_CFLAGS) -c -o $@ $<
+quiet_cmd_vdsoas = VDSOA32 $@
+ cmd_vdsoas = $(CC_ARM32) -Wp,-MD,$(depfile) $(VDSO_AFLAGS) -c -o $@ $<
+
+quiet_cmd_vdsomunge = MUNGE $@
+ cmd_vdsomunge = $(obj)/$(munge) $< $@
+
+# Generate vDSO offsets using helper script (borrowed from the 64-bit vDSO)
+gen-vdsosym := $(srctree)/$(src)/../vdso/gen_vdso_offsets.sh
+quiet_cmd_vdsosym = VDSOSYM $@
+# The AArch64 nm should be able to read an AArch32 binary
+ cmd_vdsosym = $(NM) $< | $(gen-vdsosym) | LC_ALL=C sort > $@
+
+# Install commands for the unstripped file
+quiet_cmd_vdso_install = INSTALL $@
+ cmd_vdso_install = cp $(obj)/$@.dbg $(MODLIB)/vdso/vdso32.so
+
+vdso.so: $(obj)/vdso.so.dbg
+ @mkdir -p $(MODLIB)/vdso
+ $(call cmd,vdso_install)
+
+vdso_install: vdso.so
diff --git a/arch/arm64/kernel/vdso32/compiler.h b/arch/arm64/kernel/vdso32/compiler.h
new file mode 100644
index 0000000..19a43fc
--- /dev/null
+++ b/arch/arm64/kernel/vdso32/compiler.h
@@ -0,0 +1,122 @@
+/*
+ * Userspace implementations of fallback calls
+ *
+ * Copyright (C) 2017 Cavium, Inc.
+ * Copyright (C) 2012 ARM Limited
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Author: Will Deacon <will.deacon@arm.com>
+ * Rewriten into C by: Andrew Pinski <apinski@cavium.com>
+ */
+
+#ifndef __VDSO_COMPILER_H
+#define __VDSO_COMPILER_H
+
+#include <generated/autoconf.h>
+#undef CONFIG_64BIT
+#include <asm/barrier.h> /* for isb() & dmb() */
+#include <asm/param.h> /* for HZ */
+#include <asm/unistd32.h>
+#include <linux/compiler.h>
+
+#ifdef CONFIG_ARM_ARCH_TIMER
+#define ARCH_PROVIDES_TIMER
+#endif
+
+/* can not include linux/time.h because of too much architectural cruft */
+#ifndef NSEC_PER_SEC
+#define NSEC_PER_SEC 1000000000L
+#endif
+
+/* can not include linux/jiffies.h because of too much architectural cruft */
+#ifndef TICK_NSEC
+#define TICK_NSEC ((NSEC_PER_SEC+HZ/2)/HZ)
+#endif
+
+/* can not include linux/hrtimer.h because of too much architectural cruft */
+#ifndef LOW_RES_NSEC
+#define LOW_RES_NSEC TICK_NSEC
+#ifdef ARCH_PROVIDES_TIMER
+#ifdef CONFIG_HIGH_RES_TIMERS
+# define HIGH_RES_NSEC 1
+# define MONOTONIC_RES_NSEC HIGH_RES_NSEC
+#else
+# define MONOTONIC_RES_NSEC LOW_RES_NSEC
+#endif
+#endif
+#endif
+
+#define DEFINE_FALLBACK(name, type_arg1, name_arg1, type_arg2, name_arg2) \
+static notrace long name##_fallback(type_arg1 _##name_arg1, \
+ type_arg2 _##name_arg2) \
+{ \
+ register type_arg1 name_arg1 asm("r0") = _##name_arg1; \
+ register type_arg2 name_arg2 asm("r1") = _##name_arg2; \
+ register long ret asm ("r0"); \
+ register long nr asm("r7") = __NR_##name; \
+ \
+ asm volatile( \
+ " swi #0\n" \
+ : "=r" (ret) \
+ : "r" (name_arg1), "r" (name_arg2), "r" (nr) \
+ : "memory"); \
+ \
+ return ret; \
+}
+
+/*
+ * AArch32 implementation of arch_counter_get_cntvct() suitable for vdso
+ */
+static __always_inline notrace u64 arch_vdso_read_counter(void)
+{
+ u64 res;
+
+ /* Read the virtual counter. */
+ isb();
+ asm volatile("mrrc p15, 1, %Q0, %R0, c14" : "=r" (res));
+
+ return res;
+}
+
+/*
+ * Can not include asm/processor.h to pick this up because of all the
+ * architectural components also included, so we open code a copy.
+ */
+static inline void cpu_relax(void)
+{
+ asm volatile("yield" ::: "memory");
+}
+
+#undef smp_rmb
+#if __LINUX_ARM_ARCH__ >= 8
+#define smp_rmb() dmb(ishld) /* ok on ARMv8 */
+#else
+#define smp_rmb() dmb(ish) /* ishld does not exist on ARMv7 */
+#endif
+
+/* Avoid unresolved references emitted by GCC */
+
+void __aeabi_unwind_cpp_pr0(void)
+{
+}
+
+void __aeabi_unwind_cpp_pr1(void)
+{
+}
+
+void __aeabi_unwind_cpp_pr2(void)
+{
+}
+
+#endif /* __VDSO_COMPILER_H */
diff --git a/arch/arm64/kernel/vdso32/datapage.h b/arch/arm64/kernel/vdso32/datapage.h
new file mode 100644
index 0000000..fe3e216
--- /dev/null
+++ b/arch/arm64/kernel/vdso32/datapage.h
@@ -0,0 +1 @@
+#include "../vdso/datapage.h"
diff --git a/arch/arm64/kernel/vdso32/sigreturn.S b/arch/arm64/kernel/vdso32/sigreturn.S
new file mode 100644
index 0000000..14e5f9c
--- /dev/null
+++ b/arch/arm64/kernel/vdso32/sigreturn.S
@@ -0,0 +1,76 @@
+/*
+ * Sigreturn trampolines for returning from a signal when the SA_RESTORER
+ * flag is not set.
+ *
+ * Copyright (C) 2016 ARM Limited
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Based on glibc's arm sa_restorer. While this is not strictly necessary, we
+ * provide both A32 and T32 versions, in accordance with the arm sigreturn
+ * code.
+ */
+
+#include <linux/linkage.h>
+#include <asm/asm-offsets.h>
+#include <asm/unistd32.h>
+
+.macro sigreturn_trampoline name, syscall, regs_offset
+ /*
+ * We provide directives for enabling stack unwinding through the
+ * trampoline. On arm, CFI directives are only used for debugging (and
+ * the vDSO is stripped of debug information), so only the arm-specific
+ * unwinding directives are useful here.
+ */
+ .fnstart
+ .save {r0-r15}
+ .pad #\regs_offset
+ /*
+ * It is necessary to start the unwind tables at least one instruction
+ * before the trampoline, as the unwinder will assume that the signal
+ * handler has been called from the trampoline, that is just before
+ * where the signal handler returns (mov r7, ...).
+ */
+ nop
+ENTRY(\name)
+ mov r7, #\syscall
+ svc #0
+ .fnend
+ /*
+ * We would like to use ENDPROC, but the macro uses @ which is a
+ * comment symbol for arm assemblers, so directly use .type with %
+ * instead.
+ */
+ .type \name, %function
+END(\name)
+.endm
+
+ .text
+
+ .arm
+ sigreturn_trampoline __kernel_sigreturn_arm, \
+ __NR_sigreturn, \
+ COMPAT_SIGFRAME_REGS_OFFSET
+
+ sigreturn_trampoline __kernel_rt_sigreturn_arm, \
+ __NR_rt_sigreturn, \
+ COMPAT_RT_SIGFRAME_REGS_OFFSET
+
+ .thumb
+ sigreturn_trampoline __kernel_sigreturn_thumb, \
+ __NR_sigreturn, \
+ COMPAT_SIGFRAME_REGS_OFFSET
+
+ sigreturn_trampoline __kernel_rt_sigreturn_thumb, \
+ __NR_rt_sigreturn, \
+ COMPAT_RT_SIGFRAME_REGS_OFFSET
diff --git a/arch/arm64/kernel/vdso32/vdso.S b/arch/arm64/kernel/vdso32/vdso.S
new file mode 100644
index 0000000..fe19ff7
--- /dev/null
+++ b/arch/arm64/kernel/vdso32/vdso.S
@@ -0,0 +1,32 @@
+/*
+ * Copyright (C) 2012 ARM Limited
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Author: Will Deacon <will.deacon@arm.com>
+ */
+
+#include <linux/init.h>
+#include <linux/linkage.h>
+#include <linux/const.h>
+#include <asm/page.h>
+
+ .globl vdso32_start, vdso32_end
+ .section .rodata
+ .balign PAGE_SIZE
+vdso32_start:
+ .incbin "arch/arm64/kernel/vdso32/vdso.so"
+ .balign PAGE_SIZE
+vdso32_end:
+
+ .previous
diff --git a/arch/arm64/kernel/vdso32/vdso.lds.S b/arch/arm64/kernel/vdso32/vdso.lds.S
new file mode 100644
index 0000000..f95cb1c
--- /dev/null
+++ b/arch/arm64/kernel/vdso32/vdso.lds.S
@@ -0,0 +1,95 @@
+/*
+ * Adapted from arm64 version.
+ *
+ * GNU linker script for the VDSO library.
+ *
+ * Copyright (C) 2012 ARM Limited
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Author: Will Deacon <will.deacon@arm.com>
+ * Heavily based on the vDSO linker scripts for other archs.
+ */
+
+#include <linux/const.h>
+#include <asm/page.h>
+#include <asm/vdso.h>
+
+OUTPUT_FORMAT("elf32-littlearm", "elf32-bigarm", "elf32-littlearm")
+OUTPUT_ARCH(arm)
+
+SECTIONS
+{
+ PROVIDE_HIDDEN(_vdso_data = . - PAGE_SIZE);
+ . = VDSO_LBASE + SIZEOF_HEADERS;
+
+ .hash : { *(.hash) } :text
+ .gnu.hash : { *(.gnu.hash) }
+ .dynsym : { *(.dynsym) }
+ .dynstr : { *(.dynstr) }
+ .gnu.version : { *(.gnu.version) }
+ .gnu.version_d : { *(.gnu.version_d) }
+ .gnu.version_r : { *(.gnu.version_r) }
+
+ .note : { *(.note.*) } :text :note
+
+ .dynamic : { *(.dynamic) } :text :dynamic
+
+ .rodata : { *(.rodata*) } :text
+
+ .text : { *(.text*) } :text =0xe7f001f2
+
+ .got : { *(.got) }
+ .rel.plt : { *(.rel.plt) }
+
+ /DISCARD/ : {
+ *(.note.GNU-stack)
+ *(.data .data.* .gnu.linkonce.d.* .sdata*)
+ *(.bss .sbss .dynbss .dynsbss)
+ }
+}
+
+/*
+ * We must supply the ELF program headers explicitly to get just one
+ * PT_LOAD segment, and set the flags explicitly to make segments read-only.
+ */
+PHDRS
+{
+ text PT_LOAD FLAGS(5) FILEHDR PHDRS; /* PF_R|PF_X */
+ dynamic PT_DYNAMIC FLAGS(4); /* PF_R */
+ note PT_NOTE FLAGS(4); /* PF_R */
+}
+
+VERSION
+{
+ LINUX_2.6 {
+ global:
+ __vdso_clock_gettime;
+ __vdso_gettimeofday;
+ __vdso_clock_getres;
+ __vdso_time;
+ __kernel_sigreturn_arm;
+ __kernel_sigreturn_thumb;
+ __kernel_rt_sigreturn_arm;
+ __kernel_rt_sigreturn_thumb;
+ local: *;
+ };
+}
+
+/*
+ * Make the sigreturn code visible to the kernel.
+ */
+VDSO_compat_sigreturn_arm = __kernel_sigreturn_arm;
+VDSO_compat_sigreturn_thumb = __kernel_sigreturn_thumb;
+VDSO_compat_rt_sigreturn_arm = __kernel_rt_sigreturn_arm;
+VDSO_compat_rt_sigreturn_thumb = __kernel_rt_sigreturn_thumb;
diff --git a/arch/arm64/kernel/vdso32/vgettimeofday.c b/arch/arm64/kernel/vdso32/vgettimeofday.c
new file mode 100644
index 0000000..b73d401
--- /dev/null
+++ b/arch/arm64/kernel/vdso32/vgettimeofday.c
@@ -0,0 +1,3 @@
+#include "compiler.h"
+#include "datapage.h"
+#include "../../../../lib/vdso/vgettimeofday.c"
diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c
index 3f9e157..e41a7b4 100644
--- a/arch/arm64/kvm/guest.c
+++ b/arch/arm64/kvm/guest.c
@@ -25,6 +25,7 @@
#include <linux/module.h>
#include <linux/vmalloc.h>
#include <linux/fs.h>
+#include <kvm/arm_psci.h>
#include <asm/cputype.h>
#include <asm/uaccess.h>
#include <asm/kvm.h>
@@ -56,6 +57,45 @@ static u64 core_reg_offset_from_id(u64 id)
return id & ~(KVM_REG_ARCH_MASK | KVM_REG_SIZE_MASK | KVM_REG_ARM_CORE);
}
+static int validate_core_offset(const struct kvm_one_reg *reg)
+{
+ u64 off = core_reg_offset_from_id(reg->id);
+ int size;
+
+ switch (off) {
+ case KVM_REG_ARM_CORE_REG(regs.regs[0]) ...
+ KVM_REG_ARM_CORE_REG(regs.regs[30]):
+ case KVM_REG_ARM_CORE_REG(regs.sp):
+ case KVM_REG_ARM_CORE_REG(regs.pc):
+ case KVM_REG_ARM_CORE_REG(regs.pstate):
+ case KVM_REG_ARM_CORE_REG(sp_el1):
+ case KVM_REG_ARM_CORE_REG(elr_el1):
+ case KVM_REG_ARM_CORE_REG(spsr[0]) ...
+ KVM_REG_ARM_CORE_REG(spsr[KVM_NR_SPSR - 1]):
+ size = sizeof(__u64);
+ break;
+
+ case KVM_REG_ARM_CORE_REG(fp_regs.vregs[0]) ...
+ KVM_REG_ARM_CORE_REG(fp_regs.vregs[31]):
+ size = sizeof(__uint128_t);
+ break;
+
+ case KVM_REG_ARM_CORE_REG(fp_regs.fpsr):
+ case KVM_REG_ARM_CORE_REG(fp_regs.fpcr):
+ size = sizeof(__u32);
+ break;
+
+ default:
+ return -EINVAL;
+ }
+
+ if (KVM_REG_SIZE(reg->id) == size &&
+ IS_ALIGNED(off, size / sizeof(__u32)))
+ return 0;
+
+ return -EINVAL;
+}
+
static int get_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
{
/*
@@ -75,6 +115,9 @@ static int get_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
(off + (KVM_REG_SIZE(reg->id) / sizeof(__u32))) >= nr_regs)
return -ENOENT;
+ if (validate_core_offset(reg))
+ return -EINVAL;
+
if (copy_to_user(uaddr, ((u32 *)regs) + off, KVM_REG_SIZE(reg->id)))
return -EFAULT;
@@ -97,6 +140,9 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
(off + (KVM_REG_SIZE(reg->id) / sizeof(__u32))) >= nr_regs)
return -ENOENT;
+ if (validate_core_offset(reg))
+ return -EINVAL;
+
if (KVM_REG_SIZE(reg->id) > sizeof(tmp))
return -EINVAL;
@@ -106,17 +152,25 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
}
if (off == KVM_REG_ARM_CORE_REG(regs.pstate)) {
- u32 mode = (*(u32 *)valp) & COMPAT_PSR_MODE_MASK;
+ u64 mode = (*(u64 *)valp) & COMPAT_PSR_MODE_MASK;
switch (mode) {
case COMPAT_PSR_MODE_USR:
+ if (!system_supports_32bit_el0())
+ return -EINVAL;
+ break;
case COMPAT_PSR_MODE_FIQ:
case COMPAT_PSR_MODE_IRQ:
case COMPAT_PSR_MODE_SVC:
case COMPAT_PSR_MODE_ABT:
case COMPAT_PSR_MODE_UND:
+ if (!vcpu_el1_is_32bit(vcpu))
+ return -EINVAL;
+ break;
case PSR_MODE_EL0t:
case PSR_MODE_EL1t:
case PSR_MODE_EL1h:
+ if (vcpu_el1_is_32bit(vcpu))
+ return -EINVAL;
break;
default:
err = -EINVAL;
@@ -205,7 +259,7 @@ static int get_timer_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
unsigned long kvm_arm_num_regs(struct kvm_vcpu *vcpu)
{
return num_core_regs() + kvm_arm_num_sys_reg_descs(vcpu)
- + NUM_TIMER_REGS;
+ + kvm_arm_get_fw_num_regs(vcpu) + NUM_TIMER_REGS;
}
/**
@@ -225,6 +279,11 @@ int kvm_arm_copy_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices)
uindices++;
}
+ ret = kvm_arm_copy_fw_reg_indices(vcpu, uindices);
+ if (ret)
+ return ret;
+ uindices += kvm_arm_get_fw_num_regs(vcpu);
+
ret = copy_timer_indices(vcpu, uindices);
if (ret)
return ret;
@@ -243,6 +302,9 @@ int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_CORE)
return get_core_reg(vcpu, reg);
+ if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_FW)
+ return kvm_arm_get_fw_reg(vcpu, reg);
+
if (is_timer_reg(reg->id))
return get_timer_reg(vcpu, reg);
@@ -259,6 +321,9 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg)
if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_CORE)
return set_core_reg(vcpu, reg);
+ if ((reg->id & KVM_REG_ARM_COPROC_MASK) == KVM_REG_ARM_FW)
+ return kvm_arm_set_fw_reg(vcpu, reg);
+
if (is_timer_reg(reg->id))
return set_timer_reg(vcpu, reg);
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index 4bbff90..db5efaf 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -118,6 +118,10 @@
kern_hyp_va x2
msr vbar_el2, x2
+ /* copy tpidr_el1 into tpidr_el2 for use by HYP */
+ mrs x1, tpidr_el1
+ msr tpidr_el2, x1
+
/* Hello, World! */
eret
ENDPROC(__kvm_hyp_init)
diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S
index 12ee62d..be38122 100644
--- a/arch/arm64/kvm/hyp/entry.S
+++ b/arch/arm64/kvm/hyp/entry.S
@@ -32,6 +32,7 @@
.pushsection .hyp.text, "ax"
.macro save_callee_saved_regs ctxt
+ str x18, [\ctxt, #CPU_XREG_OFFSET(18)]
stp x19, x20, [\ctxt, #CPU_XREG_OFFSET(19)]
stp x21, x22, [\ctxt, #CPU_XREG_OFFSET(21)]
stp x23, x24, [\ctxt, #CPU_XREG_OFFSET(23)]
@@ -47,6 +48,7 @@
ldp x25, x26, [\ctxt, #CPU_XREG_OFFSET(25)]
ldp x27, x28, [\ctxt, #CPU_XREG_OFFSET(27)]
ldp x29, lr, [\ctxt, #CPU_XREG_OFFSET(29)]
+ ldr x18, [\ctxt, #CPU_XREG_OFFSET(18)]
.endm
/*
@@ -62,9 +64,6 @@
// Store the host regs
save_callee_saved_regs x1
- // Store the host_ctxt for use at exit time
- str x1, [sp, #-16]!
-
add x18, x0, #VCPU_CONTEXT
// Restore guest regs x0-x17
@@ -78,12 +77,9 @@
ldp x14, x15, [x18, #CPU_XREG_OFFSET(14)]
ldp x16, x17, [x18, #CPU_XREG_OFFSET(16)]
- // Restore guest regs x19-x29, lr
+ // Restore guest regs x18-x29, lr
restore_callee_saved_regs x18
- // Restore guest reg x18
- ldr x18, [x18, #CPU_XREG_OFFSET(18)]
-
// Do not touch any register after this!
eret
ENDPROC(__guest_enter)
@@ -104,7 +100,7 @@
// Retrieve the guest regs x0-x1 from the stack
ldp x2, x3, [sp], #16 // x0, x1
- // Store the guest regs x0-x1 and x4-x18
+ // Store the guest regs x0-x1 and x4-x17
stp x2, x3, [x1, #CPU_XREG_OFFSET(0)]
stp x4, x5, [x1, #CPU_XREG_OFFSET(4)]
stp x6, x7, [x1, #CPU_XREG_OFFSET(6)]
@@ -113,13 +109,11 @@
stp x12, x13, [x1, #CPU_XREG_OFFSET(12)]
stp x14, x15, [x1, #CPU_XREG_OFFSET(14)]
stp x16, x17, [x1, #CPU_XREG_OFFSET(16)]
- str x18, [x1, #CPU_XREG_OFFSET(18)]
- // Store the guest regs x19-x29, lr
+ // Store the guest regs x18-x29, lr
save_callee_saved_regs x1
- // Restore the host_ctxt from the stack
- ldr x2, [sp], #16
+ get_host_ctxt x2, x3
// Now restore the host regs
restore_callee_saved_regs x2
@@ -159,6 +153,10 @@
ENDPROC(__guest_exit)
ENTRY(__fpsimd_guest_restore)
+ // x0: esr
+ // x1: vcpu
+ // x2-x29,lr: vcpu regs
+ // vcpu x0-x1 on the stack
stp x2, x3, [sp, #-16]!
stp x4, lr, [sp, #-16]!
@@ -173,7 +171,7 @@
alternative_endif
isb
- mrs x3, tpidr_el2
+ mov x3, x1
ldr x0, [x3, #VCPU_HOST_CONTEXT]
kern_hyp_va x0
diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index 4e9d50c..bf4988f 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -72,13 +72,8 @@
el1_sync: // Guest trapped into EL2
stp x0, x1, [sp, #-16]!
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- mrs x1, esr_el2
-alternative_else
- mrs x1, esr_el1
-alternative_endif
- lsr x0, x1, #ESR_ELx_EC_SHIFT
-
+ mrs x0, esr_el2
+ lsr x0, x0, #ESR_ELx_EC_SHIFT
cmp x0, #ESR_ELx_EC_HVC64
ccmp x0, #ESR_ELx_EC_HVC32, #4, ne
b.ne el1_trap
@@ -112,33 +107,73 @@
*/
ldr x1, [sp] // Guest's x0
eor w1, w1, #ARM_SMCCC_ARCH_WORKAROUND_1
+ cbz w1, wa_epilogue
+
+ /* ARM_SMCCC_ARCH_WORKAROUND_2 handling */
+ eor w1, w1, #(ARM_SMCCC_ARCH_WORKAROUND_1 ^ \
+ ARM_SMCCC_ARCH_WORKAROUND_2)
cbnz w1, el1_trap
- mov x0, x1
+
+#ifdef CONFIG_ARM64_SSBD
+alternative_cb arm64_enable_wa2_handling
+ b wa2_end
+alternative_cb_end
+ get_vcpu_ptr x2, x0
+ ldr x0, [x2, #VCPU_WORKAROUND_FLAGS]
+
+ // Sanitize the argument and update the guest flags
+ ldr x1, [sp, #8] // Guest's x1
+ clz w1, w1 // Murphy's device:
+ lsr w1, w1, #5 // w1 = !!w1 without using
+ eor w1, w1, #1 // the flags...
+ bfi x0, x1, #VCPU_WORKAROUND_2_FLAG_SHIFT, #1
+ str x0, [x2, #VCPU_WORKAROUND_FLAGS]
+
+ /* Check that we actually need to perform the call */
+ hyp_ldr_this_cpu x0, arm64_ssbd_callback_required, x2
+ cbz x0, wa2_end
+
+ mov w0, #ARM_SMCCC_ARCH_WORKAROUND_2
+ smc #0
+
+ /* Don't leak data from the SMC call */
+ mov x3, xzr
+wa2_end:
+ mov x2, xzr
+ mov x1, xzr
+#endif
+
+wa_epilogue:
+ mov x0, xzr
add sp, sp, #16
eret
el1_trap:
+ get_vcpu_ptr x1, x0
+
+ mrs x0, esr_el2
+ lsr x0, x0, #ESR_ELx_EC_SHIFT
/*
* x0: ESR_EC
+ * x1: vcpu pointer
*/
/* Guest accessed VFP/SIMD registers, save host, restore Guest */
cmp x0, #ESR_ELx_EC_FP_ASIMD
b.eq __fpsimd_guest_restore
- mrs x1, tpidr_el2
mov x0, #ARM_EXCEPTION_TRAP
b __guest_exit
el1_irq:
stp x0, x1, [sp, #-16]!
- mrs x1, tpidr_el2
+ get_vcpu_ptr x1, x0
mov x0, #ARM_EXCEPTION_IRQ
b __guest_exit
el1_error:
stp x0, x1, [sp, #-16]!
- mrs x1, tpidr_el2
+ get_vcpu_ptr x1, x0
mov x0, #ARM_EXCEPTION_EL1_SERROR
b __guest_exit
@@ -173,6 +208,11 @@
eret
ENDPROC(__hyp_do_panic)
+ENTRY(__hyp_panic)
+ get_host_ctxt x0, x1
+ b hyp_panic
+ENDPROC(__hyp_panic)
+
.macro invalid_vector label, target = __hyp_panic
.align 2
\label:
diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c
index 849ee8a..c6a76ea 100644
--- a/arch/arm64/kvm/hyp/switch.c
+++ b/arch/arm64/kvm/hyp/switch.c
@@ -15,6 +15,7 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
+#include <linux/arm-smccc.h>
#include <linux/types.h>
#include <linux/jump_label.h>
#include <uapi/linux/psci.h>
@@ -267,6 +268,39 @@ static void __hyp_text __skip_instr(struct kvm_vcpu *vcpu)
write_sysreg_el2(*vcpu_pc(vcpu), elr);
}
+static inline bool __hyp_text __needs_ssbd_off(struct kvm_vcpu *vcpu)
+{
+ if (!cpus_have_cap(ARM64_SSBD))
+ return false;
+
+ return !(vcpu->arch.workaround_flags & VCPU_WORKAROUND_2_FLAG);
+}
+
+static void __hyp_text __set_guest_arch_workaround_state(struct kvm_vcpu *vcpu)
+{
+#ifdef CONFIG_ARM64_SSBD
+ /*
+ * The host runs with the workaround always present. If the
+ * guest wants it disabled, so be it...
+ */
+ if (__needs_ssbd_off(vcpu) &&
+ __hyp_this_cpu_read(arm64_ssbd_callback_required))
+ arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 0, NULL);
+#endif
+}
+
+static void __hyp_text __set_host_arch_workaround_state(struct kvm_vcpu *vcpu)
+{
+#ifdef CONFIG_ARM64_SSBD
+ /*
+ * If the guest has disabled the workaround, bring it back on.
+ */
+ if (__needs_ssbd_off(vcpu) &&
+ __hyp_this_cpu_read(arm64_ssbd_callback_required))
+ arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 1, NULL);
+#endif
+}
+
int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
{
struct kvm_cpu_context *host_ctxt;
@@ -275,9 +309,9 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
u64 exit_code;
vcpu = kern_hyp_va(vcpu);
- write_sysreg(vcpu, tpidr_el2);
host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
__sysreg_save_host_state(host_ctxt);
@@ -297,6 +331,8 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
__sysreg_restore_guest_state(guest_ctxt);
__debug_restore_state(vcpu, kern_hyp_va(vcpu->arch.debug_ptr), guest_ctxt);
+ __set_guest_arch_workaround_state(vcpu);
+
/* Jump in the fire! */
again:
exit_code = __guest_enter(vcpu, host_ctxt);
@@ -353,6 +389,8 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
}
}
+ __set_host_arch_workaround_state(vcpu);
+
fp_enabled = __fpsimd_enabled();
__sysreg_save_guest_state(guest_ctxt);
@@ -378,7 +416,8 @@ int __hyp_text __kvm_vcpu_run(struct kvm_vcpu *vcpu)
static const char __hyp_panic_string[] = "HYP panic:\nPS:%08llx PC:%016llx ESR:%08llx\nFAR:%016llx HPFAR:%016llx PAR:%016llx\nVCPU:%p\n";
-static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par)
+static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par,
+ struct kvm_vcpu *vcpu)
{
unsigned long str_va;
@@ -392,35 +431,32 @@ static void __hyp_text __hyp_call_panic_nvhe(u64 spsr, u64 elr, u64 par)
__hyp_do_panic(str_va,
spsr, elr,
read_sysreg(esr_el2), read_sysreg_el2(far),
- read_sysreg(hpfar_el2), par,
- (void *)read_sysreg(tpidr_el2));
+ read_sysreg(hpfar_el2), par, vcpu);
}
-static void __hyp_text __hyp_call_panic_vhe(u64 spsr, u64 elr, u64 par)
+static void __hyp_text __hyp_call_panic_vhe(u64 spsr, u64 elr, u64 par,
+ struct kvm_vcpu *vcpu)
{
panic(__hyp_panic_string,
spsr, elr,
read_sysreg_el2(esr), read_sysreg_el2(far),
- read_sysreg(hpfar_el2), par,
- (void *)read_sysreg(tpidr_el2));
+ read_sysreg(hpfar_el2), par, vcpu);
}
static hyp_alternate_select(__hyp_call_panic,
__hyp_call_panic_nvhe, __hyp_call_panic_vhe,
ARM64_HAS_VIRT_HOST_EXTN);
-void __hyp_text __noreturn __hyp_panic(void)
+void __hyp_text __noreturn hyp_panic(struct kvm_cpu_context *host_ctxt)
{
+ struct kvm_vcpu *vcpu = NULL;
+
u64 spsr = read_sysreg_el2(spsr);
u64 elr = read_sysreg_el2(elr);
u64 par = read_sysreg(par_el1);
if (read_sysreg(vttbr_el2)) {
- struct kvm_vcpu *vcpu;
- struct kvm_cpu_context *host_ctxt;
-
- vcpu = (struct kvm_vcpu *)read_sysreg(tpidr_el2);
- host_ctxt = kern_hyp_va(vcpu->arch.host_cpu_context);
+ vcpu = host_ctxt->__hyp_running_vcpu;
__timer_save_state(vcpu);
__deactivate_traps(vcpu);
__deactivate_vm(vcpu);
@@ -428,7 +464,7 @@ void __hyp_text __noreturn __hyp_panic(void)
}
/* Call panic for real */
- __hyp_call_panic()(spsr, elr, par);
+ __hyp_call_panic()(spsr, elr, par, vcpu);
unreachable();
}
diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c
index 9341376..e19d89c 100644
--- a/arch/arm64/kvm/hyp/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/sysreg-sr.c
@@ -27,8 +27,8 @@ static void __hyp_text __sysreg_do_nothing(struct kvm_cpu_context *ctxt) { }
/*
* Non-VHE: Both host and guest must save everything.
*
- * VHE: Host must save tpidr*_el[01], actlr_el1, mdscr_el1, sp0, pc,
- * pstate, and guest must save everything.
+ * VHE: Host must save tpidr*_el0, actlr_el1, mdscr_el1, sp_el0,
+ * and guest must save everything.
*/
static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
@@ -36,11 +36,8 @@ static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt)
ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1);
ctxt->sys_regs[TPIDR_EL0] = read_sysreg(tpidr_el0);
ctxt->sys_regs[TPIDRRO_EL0] = read_sysreg(tpidrro_el0);
- ctxt->sys_regs[TPIDR_EL1] = read_sysreg(tpidr_el1);
ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1);
ctxt->gp_regs.regs.sp = read_sysreg(sp_el0);
- ctxt->gp_regs.regs.pc = read_sysreg_el2(elr);
- ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
}
static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
@@ -62,10 +59,13 @@ static void __hyp_text __sysreg_save_state(struct kvm_cpu_context *ctxt)
ctxt->sys_regs[AMAIR_EL1] = read_sysreg_el1(amair);
ctxt->sys_regs[CNTKCTL_EL1] = read_sysreg_el1(cntkctl);
ctxt->sys_regs[PAR_EL1] = read_sysreg(par_el1);
+ ctxt->sys_regs[TPIDR_EL1] = read_sysreg(tpidr_el1);
ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1);
ctxt->gp_regs.elr_el1 = read_sysreg_el1(elr);
ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg_el1(spsr);
+ ctxt->gp_regs.regs.pc = read_sysreg_el2(elr);
+ ctxt->gp_regs.regs.pstate = read_sysreg_el2(spsr);
}
static hyp_alternate_select(__sysreg_call_save_host_state,
@@ -89,11 +89,8 @@ static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctx
write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1);
write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0);
write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0);
- write_sysreg(ctxt->sys_regs[TPIDR_EL1], tpidr_el1);
write_sysreg(ctxt->sys_regs[MDSCR_EL1], mdscr_el1);
write_sysreg(ctxt->gp_regs.regs.sp, sp_el0);
- write_sysreg_el2(ctxt->gp_regs.regs.pc, elr);
- write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
}
static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
@@ -115,10 +112,13 @@ static void __hyp_text __sysreg_restore_state(struct kvm_cpu_context *ctxt)
write_sysreg_el1(ctxt->sys_regs[AMAIR_EL1], amair);
write_sysreg_el1(ctxt->sys_regs[CNTKCTL_EL1], cntkctl);
write_sysreg(ctxt->sys_regs[PAR_EL1], par_el1);
+ write_sysreg(ctxt->sys_regs[TPIDR_EL1], tpidr_el1);
write_sysreg(ctxt->gp_regs.sp_el1, sp_el1);
write_sysreg_el1(ctxt->gp_regs.elr_el1, elr);
write_sysreg_el1(ctxt->gp_regs.spsr[KVM_SPSR_EL1],spsr);
+ write_sysreg_el2(ctxt->gp_regs.regs.pc, elr);
+ write_sysreg_el2(ctxt->gp_regs.regs.pstate, spsr);
}
static hyp_alternate_select(__sysreg_call_restore_host_state,
@@ -183,3 +183,8 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu)
if (vcpu->arch.debug_flags & KVM_ARM64_DEBUG_DIRTY)
write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2);
}
+
+void __hyp_text __kvm_set_tpidr_el2(u64 tpidr_el2)
+{
+ asm("msr tpidr_el2, %0": : "r" (tpidr_el2));
+}
diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
index 5bc4608..29a27a0 100644
--- a/arch/arm64/kvm/reset.c
+++ b/arch/arm64/kvm/reset.c
@@ -135,6 +135,10 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu)
/* Reset PMU */
kvm_pmu_vcpu_reset(vcpu);
+ /* Default workaround setup is enabled (if supported) */
+ if (kvm_arm_have_ssbd() == KVM_SSBD_KERNEL)
+ vcpu->arch.workaround_flags |= VCPU_WORKAROUND_2_FLAG;
+
/* Reset timer */
return kvm_timer_vcpu_reset(vcpu, cpu_vtimer_irq);
}
diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
index c86b790..7d3ccc4 100644
--- a/arch/arm64/lib/Makefile
+++ b/arch/arm64/lib/Makefile
@@ -11,9 +11,8 @@
# when supported by the CPU. Result and argument registers are handled
# correctly, based on the function prototype.
lib-$(CONFIG_ARM64_LSE_ATOMICS) += atomic_ll_sc.o
-CFLAGS_atomic_ll_sc.o := -fcall-used-x0 -ffixed-x1 -ffixed-x2 \
+CFLAGS_atomic_ll_sc.o := -ffixed-x1 -ffixed-x2 \
-ffixed-x3 -ffixed-x4 -ffixed-x5 -ffixed-x6 \
-ffixed-x7 -fcall-saved-x8 -fcall-saved-x9 \
-fcall-saved-x10 -fcall-saved-x11 -fcall-saved-x12 \
- -fcall-saved-x13 -fcall-saved-x14 -fcall-saved-x15 \
- -fcall-saved-x18
+ -fcall-saved-x13 -fcall-saved-x14 -fcall-saved-x15
diff --git a/arch/arm64/lib/copy_page.S b/arch/arm64/lib/copy_page.S
index c3cd65e..ba84a99 100644
--- a/arch/arm64/lib/copy_page.S
+++ b/arch/arm64/lib/copy_page.S
@@ -30,9 +30,10 @@
*/
ENTRY(copy_page)
alternative_if ARM64_HAS_NO_HW_PREFETCH
- # Prefetch two cache lines ahead.
- prfm pldl1strm, [x1, #128]
- prfm pldl1strm, [x1, #256]
+ // Prefetch three cache lines ahead.
+ prfm pldl1strm, [x1, #128]
+ prfm pldl1strm, [x1, #256]
+ prfm pldl1strm, [x1, #384]
alternative_else_nop_endif
ldp x2, x3, [x1]
@@ -44,45 +45,45 @@
ldp x14, x15, [x1, #96]
ldp x16, x17, [x1, #112]
- mov x18, #(PAGE_SIZE - 128)
+ add x0, x0, #256
add x1, x1, #128
1:
- subs x18, x18, #128
+ tst x0, #(PAGE_SIZE - 1)
alternative_if ARM64_HAS_NO_HW_PREFETCH
- prfm pldl1strm, [x1, #384]
+ prfm pldl1strm, [x1, #384]
alternative_else_nop_endif
- stnp x2, x3, [x0]
+ stnp x2, x3, [x0, #-256]
ldp x2, x3, [x1]
- stnp x4, x5, [x0, #16]
+ stnp x4, x5, [x0, #-240]
ldp x4, x5, [x1, #16]
- stnp x6, x7, [x0, #32]
+ stnp x6, x7, [x0, #-224]
ldp x6, x7, [x1, #32]
- stnp x8, x9, [x0, #48]
+ stnp x8, x9, [x0, #-208]
ldp x8, x9, [x1, #48]
- stnp x10, x11, [x0, #64]
+ stnp x10, x11, [x0, #-192]
ldp x10, x11, [x1, #64]
- stnp x12, x13, [x0, #80]
+ stnp x12, x13, [x0, #-176]
ldp x12, x13, [x1, #80]
- stnp x14, x15, [x0, #96]
+ stnp x14, x15, [x0, #-160]
ldp x14, x15, [x1, #96]
- stnp x16, x17, [x0, #112]
+ stnp x16, x17, [x0, #-144]
ldp x16, x17, [x1, #112]
add x0, x0, #128
add x1, x1, #128
- b.gt 1b
+ b.ne 1b
- stnp x2, x3, [x0]
- stnp x4, x5, [x0, #16]
- stnp x6, x7, [x0, #32]
- stnp x8, x9, [x0, #48]
- stnp x10, x11, [x0, #64]
- stnp x12, x13, [x0, #80]
- stnp x14, x15, [x0, #96]
- stnp x16, x17, [x0, #112]
+ stnp x2, x3, [x0, #-256]
+ stnp x4, x5, [x0, #-240]
+ stnp x6, x7, [x0, #-224]
+ stnp x8, x9, [x0, #-208]
+ stnp x10, x11, [x0, #-192]
+ stnp x12, x13, [x0, #-176]
+ stnp x14, x15, [x0, #-160]
+ stnp x16, x17, [x0, #-144]
ret
ENDPROC(copy_page)
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 6befc9c..eb445e1 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -316,8 +316,12 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr,
struct task_struct *tsk;
struct mm_struct *mm;
int fault, sig, code;
+ bool was_major = false;
unsigned long vm_flags = VM_READ | VM_WRITE;
unsigned int mm_flags = FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_KILLABLE;
+ ktime_t event_ts;
+
+ mm_event_start(&event_ts);
if (notify_page_fault(regs, esr))
return 0;
@@ -399,6 +403,7 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr,
perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS, 1, regs, addr);
if (mm_flags & FAULT_FLAG_ALLOW_RETRY) {
if (fault & VM_FAULT_MAJOR) {
+ was_major = true;
tsk->maj_flt++;
perf_sw_event(PERF_COUNT_SW_PAGE_FAULTS_MAJ, 1, regs,
addr);
@@ -418,6 +423,14 @@ static int __kprobes do_page_fault(unsigned long addr, unsigned int esr,
}
}
+ if (was_major) {
+ if (fault & VM_FAULT_SWAP)
+ mm_event_end(MM_SWP_FAULT, event_ts);
+ else
+ mm_event_end(MM_MAJ_FAULT, event_ts);
+ } else {
+ mm_event_end(MM_MIN_FAULT, event_ts);
+ }
up_read(&mm->mmap_sem);
/*
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 2b35b67..c410c9e 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -146,11 +146,13 @@ static void __init zone_sizes_init(unsigned long min, unsigned long max)
#endif /* CONFIG_NUMA */
#ifdef CONFIG_HAVE_ARCH_PFN_VALID
-#define PFN_MASK ((1UL << (64 - PAGE_SHIFT)) - 1)
-
int pfn_valid(unsigned long pfn)
{
- return (pfn & PFN_MASK) == pfn && memblock_is_map_memory(pfn << PAGE_SHIFT);
+ phys_addr_t addr = pfn << PAGE_SHIFT;
+
+ if ((addr >> PAGE_SHIFT) != pfn)
+ return 0;
+ return memblock_is_map_memory(addr);
}
EXPORT_SYMBOL(pfn_valid);
#endif
@@ -471,11 +473,13 @@ void __init mem_init(void)
BUILD_BUG_ON(TASK_SIZE_32 > TASK_SIZE_64);
#endif
+#ifdef CONFIG_SPARSEMEM_VMEMMAP
/*
* Make sure we chose the upper bound of sizeof(struct page)
- * correctly.
+ * correctly when sizing the VMEMMAP array.
*/
BUILD_BUG_ON(sizeof(struct page) > (1 << STRUCT_PAGE_MAX_SHIFT));
+#endif
if (PAGE_SIZE >= 16384 && get_num_physpages() <= 128) {
extern int sysctl_overcommit_memory;
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 6e989eb..9e2ec8a 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -862,12 +862,12 @@ int pmd_clear_huge(pmd_t *pmd)
return 1;
}
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
{
return pud_none(*pud);
}
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
{
return pmd_none(*pmd);
}
diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S
index 3a9af60..a543da8 100644
--- a/arch/arm64/mm/proc.S
+++ b/arch/arm64/mm/proc.S
@@ -119,7 +119,7 @@
mrs x11, tpidr_el1
mrs x12, sp_el0
stp x2, x3, [x0]
- stp x4, xzr, [x0, #16]
+ stp x4, x18, [x0, #16]
stp x5, x6, [x0, #32]
stp x7, x8, [x0, #48]
stp x9, x10, [x0, #64]
@@ -135,7 +135,7 @@
.pushsection ".idmap.text", "awx"
ENTRY(cpu_do_resume)
ldp x2, x3, [x0]
- ldp x4, x5, [x0, #16]
+ ldp x4, x18, [x0, #16]
ldp x6, x8, [x0, #32]
ldp x9, x10, [x0, #48]
ldp x11, x12, [x0, #64]
@@ -241,8 +241,9 @@
.macro __idmap_kpti_put_pgtable_ent_ng, type
orr \type, \type, #PTE_NG // Same bit for blocks and pages
- str \type, [cur_\()\type\()p] // Update the entry and ensure it
- dc civac, cur_\()\type\()p // is visible to all CPUs.
+ str \type, [cur_\()\type\()p] // Update the entry and ensure
+ dmb sy // that it is visible to all
+ dc civac, cur_\()\type\()p // CPUs.
.endm
/*
@@ -481,6 +482,9 @@
cmp x9, #2
b.lt 1f
#ifdef CONFIG_ARM64_ERRATUM_1024718
+ /* Disable hardware DBM on Cortex-A55 r0p0, r0p1 & r1p0 */
+ cpu_midr_match MIDR_CORTEX_A55, MIDR_CPU_VAR_REV(0, 0), MIDR_CPU_VAR_REV(1, 0), x1, x2, x3, x4
+ cbnz x1, 1f
/* Disable hardware DBM on Kryo3S */
cpu_midr_match MIDR_KRYO3S, MIDR_CPU_VAR_REV(7, 12), \
MIDR_CPU_VAR_REV(7, 13), x1, x2, x3, x4
diff --git a/arch/frv/include/asm/futex.h b/arch/frv/include/asm/futex.h
index 4bea27f..2702bd8 100644
--- a/arch/frv/include/asm/futex.h
+++ b/arch/frv/include/asm/futex.h
@@ -7,7 +7,8 @@
#include <asm/errno.h>
#include <asm/uaccess.h>
-extern int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr);
+extern int arch_futex_atomic_op_inuser(int op, int oparg, int *oval,
+ u32 __user *uaddr);
static inline int
futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
diff --git a/arch/frv/kernel/futex.c b/arch/frv/kernel/futex.c
index d155ca9..37f7b2b 100644
--- a/arch/frv/kernel/futex.c
+++ b/arch/frv/kernel/futex.c
@@ -186,20 +186,10 @@ static inline int atomic_futex_op_xchg_xor(int oparg, u32 __user *uaddr, int *_o
/*
* do the futex operations
*/
-int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+int arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
-
pagefault_disable();
switch (op) {
@@ -225,18 +215,9 @@ int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS; break;
- }
- }
+ if (!ret)
+ *oval = oldval;
return ret;
-} /* end futex_atomic_op_inuser() */
+} /* end arch_futex_atomic_op_inuser() */
diff --git a/arch/hexagon/include/asm/bitops.h b/arch/hexagon/include/asm/bitops.h
index 5e4a59b..2691a18 100644
--- a/arch/hexagon/include/asm/bitops.h
+++ b/arch/hexagon/include/asm/bitops.h
@@ -211,7 +211,7 @@ static inline long ffz(int x)
* This is defined the same way as ffs.
* Note fls(0) = 0, fls(1) = 1, fls(0x80000000) = 32.
*/
-static inline long fls(int x)
+static inline int fls(int x)
{
int r;
@@ -232,7 +232,7 @@ static inline long fls(int x)
* the libc and compiler builtin ffs routines, therefore
* differs in spirit from the above ffz (man ffs).
*/
-static inline long ffs(int x)
+static inline int ffs(int x)
{
int r;
diff --git a/arch/hexagon/include/asm/futex.h b/arch/hexagon/include/asm/futex.h
index 7e597f8..c607b77 100644
--- a/arch/hexagon/include/asm/futex.h
+++ b/arch/hexagon/include/asm/futex.h
@@ -31,18 +31,9 @@
static inline int
-futex_atomic_op_inuser(int encoded_op, int __user *uaddr)
+arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(int)))
- return -EFAULT;
pagefault_disable();
@@ -72,30 +63,9 @@ futex_atomic_op_inuser(int encoded_op, int __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ:
- ret = (oldval == cmparg);
- break;
- case FUTEX_OP_CMP_NE:
- ret = (oldval != cmparg);
- break;
- case FUTEX_OP_CMP_LT:
- ret = (oldval < cmparg);
- break;
- case FUTEX_OP_CMP_GE:
- ret = (oldval >= cmparg);
- break;
- case FUTEX_OP_CMP_LE:
- ret = (oldval <= cmparg);
- break;
- case FUTEX_OP_CMP_GT:
- ret = (oldval > cmparg);
- break;
- default:
- ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/hexagon/kernel/dma.c b/arch/hexagon/kernel/dma.c
index b901778..0e2be48 100644
--- a/arch/hexagon/kernel/dma.c
+++ b/arch/hexagon/kernel/dma.c
@@ -68,7 +68,7 @@ static void *hexagon_dma_alloc_coherent(struct device *dev, size_t size,
panic("Can't create %s() memory pool!", __func__);
else
gen_pool_add(coherent_pool,
- pfn_to_virt(max_low_pfn),
+ (unsigned long)pfn_to_virt(max_low_pfn),
hexagon_coherent_pool_size, -1);
}
diff --git a/arch/ia64/include/asm/futex.h b/arch/ia64/include/asm/futex.h
index 76acbcd..6d67dc1 100644
--- a/arch/ia64/include/asm/futex.h
+++ b/arch/ia64/include/asm/futex.h
@@ -45,18 +45,9 @@ do { \
} while (0)
static inline int
-futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
+arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (! access_ok (VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
pagefault_disable();
@@ -84,17 +75,9 @@ futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/ia64/kernel/err_inject.c b/arch/ia64/kernel/err_inject.c
index 5ed0ea9..f851c9d 100644
--- a/arch/ia64/kernel/err_inject.c
+++ b/arch/ia64/kernel/err_inject.c
@@ -142,7 +142,7 @@ store_virtual_to_phys(struct device *dev, struct device_attribute *attr,
u64 virt_addr=simple_strtoull(buf, NULL, 16);
int ret;
- ret = get_user_pages(virt_addr, 1, FOLL_WRITE, NULL, NULL);
+ ret = get_user_pages_fast(virt_addr, 1, FOLL_WRITE, NULL);
if (ret<=0) {
#ifdef ERR_INJ_DEBUG
printk("Virtual address %lx is not existing.\n",virt_addr);
diff --git a/arch/m68k/coldfire/device.c b/arch/m68k/coldfire/device.c
index a0fc0c1..3e8be0f 100644
--- a/arch/m68k/coldfire/device.c
+++ b/arch/m68k/coldfire/device.c
@@ -135,7 +135,11 @@ static struct platform_device mcf_fec0 = {
.id = 0,
.num_resources = ARRAY_SIZE(mcf_fec0_resources),
.resource = mcf_fec0_resources,
- .dev.platform_data = FEC_PDATA,
+ .dev = {
+ .dma_mask = &mcf_fec0.dev.coherent_dma_mask,
+ .coherent_dma_mask = DMA_BIT_MASK(32),
+ .platform_data = FEC_PDATA,
+ }
};
#ifdef MCFFEC_BASE1
@@ -167,7 +171,11 @@ static struct platform_device mcf_fec1 = {
.id = 1,
.num_resources = ARRAY_SIZE(mcf_fec1_resources),
.resource = mcf_fec1_resources,
- .dev.platform_data = FEC_PDATA,
+ .dev = {
+ .dma_mask = &mcf_fec1.dev.coherent_dma_mask,
+ .coherent_dma_mask = DMA_BIT_MASK(32),
+ .platform_data = FEC_PDATA,
+ }
};
#endif /* MCFFEC_BASE1 */
#endif /* CONFIG_FEC */
diff --git a/arch/m68k/include/asm/mcf_pgalloc.h b/arch/m68k/include/asm/mcf_pgalloc.h
index fb95aed..dac7564 100644
--- a/arch/m68k/include/asm/mcf_pgalloc.h
+++ b/arch/m68k/include/asm/mcf_pgalloc.h
@@ -43,6 +43,7 @@ extern inline pmd_t *pmd_alloc_kernel(pgd_t *pgd, unsigned long address)
static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t page,
unsigned long address)
{
+ pgtable_page_dtor(page);
__free_page(page);
}
@@ -73,8 +74,9 @@ static inline struct page *pte_alloc_one(struct mm_struct *mm,
return page;
}
-extern inline void pte_free(struct mm_struct *mm, struct page *page)
+static inline void pte_free(struct mm_struct *mm, struct page *page)
{
+ pgtable_page_dtor(page);
__free_page(page);
}
diff --git a/arch/m68k/kernel/time.c b/arch/m68k/kernel/time.c
index 4e5aa2f..87160b4 100644
--- a/arch/m68k/kernel/time.c
+++ b/arch/m68k/kernel/time.c
@@ -14,6 +14,7 @@
#include <linux/export.h>
#include <linux/module.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/kernel.h>
#include <linux/param.h>
#include <linux/string.h>
diff --git a/arch/m68k/mm/kmap.c b/arch/m68k/mm/kmap.c
index 6e4955b..fcd52ce 100644
--- a/arch/m68k/mm/kmap.c
+++ b/arch/m68k/mm/kmap.c
@@ -88,7 +88,8 @@ static inline void free_io_area(void *addr)
for (p = &iolist ; (tmp = *p) ; p = &tmp->next) {
if (tmp->addr == addr) {
*p = tmp->next;
- __iounmap(tmp->addr, tmp->size);
+ /* remove gap added in get_io_area() */
+ __iounmap(tmp->addr, tmp->size - IO_SIZE);
kfree(tmp);
return;
}
diff --git a/arch/microblaze/boot/Makefile b/arch/microblaze/boot/Makefile
index 91d2068..0f3fe6a 100644
--- a/arch/microblaze/boot/Makefile
+++ b/arch/microblaze/boot/Makefile
@@ -21,17 +21,19 @@
quiet_cmd_cp = CP $< $@$2
cmd_cp = cat $< >$@$2 || (rm -f $@ && echo false)
-quiet_cmd_strip = STRIP $@
+quiet_cmd_strip = STRIP $< $@$2
cmd_strip = $(STRIP) -K microblaze_start -K _end -K __log_buf \
- -K _fdt_start vmlinux -o $@
+ -K _fdt_start $< -o $@$2
UIMAGE_LOADADDR = $(CONFIG_KERNEL_BASE_ADDR)
+UIMAGE_IN = $@
+UIMAGE_OUT = $@.ub
$(obj)/simpleImage.%: vmlinux FORCE
$(call if_changed,cp,.unstrip)
$(call if_changed,objcopy)
$(call if_changed,uimage)
- $(call if_changed,strip)
- @echo 'Kernel: $@ is ready' ' (#'`cat .version`')'
+ $(call if_changed,strip,.strip)
+ @echo 'Kernel: $(UIMAGE_OUT) is ready' ' (#'`cat .version`')'
clean-files += simpleImage.*.unstrip linux.bin.ub dts/*.dtb
diff --git a/arch/microblaze/include/asm/futex.h b/arch/microblaze/include/asm/futex.h
index 01848f0..a9dad9e 100644
--- a/arch/microblaze/include/asm/futex.h
+++ b/arch/microblaze/include/asm/futex.h
@@ -29,18 +29,9 @@
})
static inline int
-futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
pagefault_disable();
@@ -66,30 +57,9 @@ futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ:
- ret = (oldval == cmparg);
- break;
- case FUTEX_OP_CMP_NE:
- ret = (oldval != cmparg);
- break;
- case FUTEX_OP_CMP_LT:
- ret = (oldval < cmparg);
- break;
- case FUTEX_OP_CMP_GE:
- ret = (oldval >= cmparg);
- break;
- case FUTEX_OP_CMP_LE:
- ret = (oldval <= cmparg);
- break;
- case FUTEX_OP_CMP_GT:
- ret = (oldval > cmparg);
- break;
- default:
- ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/microblaze/kernel/heartbeat.c b/arch/microblaze/kernel/heartbeat.c
index 4643e3a..2022130 100644
--- a/arch/microblaze/kernel/heartbeat.c
+++ b/arch/microblaze/kernel/heartbeat.c
@@ -9,6 +9,7 @@
*/
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/io.h>
#include <asm/setup.h>
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index 2d2fd79..34fbbf8 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -95,6 +95,7 @@
select PCI_DRIVERS_GENERIC
select PINCTRL
select SMP_UP if SMP
+ select SWAP_IO_SPACE
select SYS_HAS_CPU_MIPS32_R1
select SYS_HAS_CPU_MIPS32_R2
select SYS_HAS_CPU_MIPS32_R6
diff --git a/arch/mips/ath79/common.c b/arch/mips/ath79/common.c
index d071a3a..fc97a11 100644
--- a/arch/mips/ath79/common.c
+++ b/arch/mips/ath79/common.c
@@ -58,7 +58,7 @@ EXPORT_SYMBOL_GPL(ath79_ddr_ctrl_init);
void ath79_ddr_wb_flush(u32 reg)
{
- void __iomem *flush_reg = ath79_ddr_wb_flush_base + reg;
+ void __iomem *flush_reg = ath79_ddr_wb_flush_base + (reg * 4);
/* Flush the DDR write buffer. */
__raw_writel(0x1, flush_reg);
diff --git a/arch/mips/ath79/setup.c b/arch/mips/ath79/setup.c
index f206daf..26a058d 100644
--- a/arch/mips/ath79/setup.c
+++ b/arch/mips/ath79/setup.c
@@ -40,6 +40,7 @@ static char ath79_sys_type[ATH79_SYS_TYPE_LEN];
static void ath79_restart(char *command)
{
+ local_irq_disable();
ath79_device_reset_set(AR71XX_RESET_FULL_CHIP);
for (;;)
if (cpu_wait)
diff --git a/arch/mips/boot/compressed/calc_vmlinuz_load_addr.c b/arch/mips/boot/compressed/calc_vmlinuz_load_addr.c
index 37fe58c..542c3ed 100644
--- a/arch/mips/boot/compressed/calc_vmlinuz_load_addr.c
+++ b/arch/mips/boot/compressed/calc_vmlinuz_load_addr.c
@@ -13,6 +13,7 @@
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
+#include "../../../../include/linux/sizes.h"
int main(int argc, char *argv[])
{
@@ -45,11 +46,11 @@ int main(int argc, char *argv[])
vmlinuz_load_addr = vmlinux_load_addr + vmlinux_size;
/*
- * Align with 16 bytes: "greater than that used for any standard data
- * types by a MIPS compiler." -- See MIPS Run Linux (Second Edition).
+ * Align with 64KB: KEXEC needs load sections to be aligned to PAGE_SIZE,
+ * which may be as large as 64KB depending on the kernel configuration.
*/
- vmlinuz_load_addr += (16 - vmlinux_size % 16);
+ vmlinuz_load_addr += (SZ_64K - vmlinux_size % SZ_64K);
printf("0x%llx\n", vmlinuz_load_addr);
diff --git a/arch/mips/cavium-octeon/executive/cvmx-helper.c b/arch/mips/cavium-octeon/executive/cvmx-helper.c
index 6456af6..59defc5 100644
--- a/arch/mips/cavium-octeon/executive/cvmx-helper.c
+++ b/arch/mips/cavium-octeon/executive/cvmx-helper.c
@@ -67,7 +67,7 @@ void (*cvmx_override_pko_queue_priority) (int pko_port,
void (*cvmx_override_ipd_port_setup) (int ipd_port);
/* Port count per interface */
-static int interface_port_count[5];
+static int interface_port_count[9];
/* Port last configured link info index by IPD/PKO port */
static cvmx_helper_link_info_t
@@ -290,7 +290,8 @@ static cvmx_helper_interface_mode_t __cvmx_get_mode_cn7xxx(int interface)
case 3:
return CVMX_HELPER_INTERFACE_MODE_LOOP;
case 4:
- return CVMX_HELPER_INTERFACE_MODE_RGMII;
+ /* TODO: Implement support for AGL (RGMII). */
+ return CVMX_HELPER_INTERFACE_MODE_DISABLED;
default:
return CVMX_HELPER_INTERFACE_MODE_DISABLED;
}
diff --git a/arch/mips/cavium-octeon/octeon-irq.c b/arch/mips/cavium-octeon/octeon-irq.c
index 6ed1ded..6420c83 100644
--- a/arch/mips/cavium-octeon/octeon-irq.c
+++ b/arch/mips/cavium-octeon/octeon-irq.c
@@ -2271,7 +2271,7 @@ static int __init octeon_irq_init_cib(struct device_node *ciu_node,
parent_irq = irq_of_parse_and_map(ciu_node, 0);
if (!parent_irq) {
- pr_err("ERROR: Couldn't acquire parent_irq for %s\n.",
+ pr_err("ERROR: Couldn't acquire parent_irq for %s\n",
ciu_node->name);
return -EINVAL;
}
@@ -2283,7 +2283,7 @@ static int __init octeon_irq_init_cib(struct device_node *ciu_node,
addr = of_get_address(ciu_node, 0, NULL, NULL);
if (!addr) {
- pr_err("ERROR: Couldn't acquire reg(0) %s\n.", ciu_node->name);
+ pr_err("ERROR: Couldn't acquire reg(0) %s\n", ciu_node->name);
return -EINVAL;
}
host_data->raw_reg = (u64)phys_to_virt(
@@ -2291,7 +2291,7 @@ static int __init octeon_irq_init_cib(struct device_node *ciu_node,
addr = of_get_address(ciu_node, 1, NULL, NULL);
if (!addr) {
- pr_err("ERROR: Couldn't acquire reg(1) %s\n.", ciu_node->name);
+ pr_err("ERROR: Couldn't acquire reg(1) %s\n", ciu_node->name);
return -EINVAL;
}
host_data->en_reg = (u64)phys_to_virt(
@@ -2299,7 +2299,7 @@ static int __init octeon_irq_init_cib(struct device_node *ciu_node,
r = of_property_read_u32(ciu_node, "cavium,max-bits", &val);
if (r) {
- pr_err("ERROR: Couldn't read cavium,max-bits from %s\n.",
+ pr_err("ERROR: Couldn't read cavium,max-bits from %s\n",
ciu_node->name);
return r;
}
@@ -2309,7 +2309,7 @@ static int __init octeon_irq_init_cib(struct device_node *ciu_node,
&octeon_irq_domain_cib_ops,
host_data);
if (!cib_domain) {
- pr_err("ERROR: Couldn't irq_domain_add_linear()\n.");
+ pr_err("ERROR: Couldn't irq_domain_add_linear()\n");
return -ENOMEM;
}
diff --git a/arch/mips/cavium-octeon/octeon-platform.c b/arch/mips/cavium-octeon/octeon-platform.c
index 37a932d..1ba6bcf 100644
--- a/arch/mips/cavium-octeon/octeon-platform.c
+++ b/arch/mips/cavium-octeon/octeon-platform.c
@@ -366,6 +366,7 @@ static int __init octeon_ehci_device_init(void)
return 0;
pd = of_find_device_by_node(ehci_node);
+ of_node_put(ehci_node);
if (!pd)
return 0;
@@ -428,6 +429,7 @@ static int __init octeon_ohci_device_init(void)
return 0;
pd = of_find_device_by_node(ohci_node);
+ of_node_put(ohci_node);
if (!pd)
return 0;
diff --git a/arch/mips/generic/init.c b/arch/mips/generic/init.c
index d493ccb..cf5b564 100644
--- a/arch/mips/generic/init.c
+++ b/arch/mips/generic/init.c
@@ -159,6 +159,7 @@ void __init arch_init_irq(void)
"mti,cpu-interrupt-controller");
if (!cpu_has_veic && !intc_node)
mips_cpu_irq_init();
+ of_node_put(intc_node);
irqchip_init();
}
diff --git a/arch/mips/include/asm/futex.h b/arch/mips/include/asm/futex.h
index 1de190b..a9e61ea 100644
--- a/arch/mips/include/asm/futex.h
+++ b/arch/mips/include/asm/futex.h
@@ -83,18 +83,9 @@
}
static inline int
-futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (! access_ok (VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
pagefault_disable();
@@ -125,17 +116,9 @@ futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/mips/include/asm/io.h b/arch/mips/include/asm/io.h
index ecabc00..06049b6 100644
--- a/arch/mips/include/asm/io.h
+++ b/arch/mips/include/asm/io.h
@@ -141,14 +141,14 @@ static inline void * phys_to_virt(unsigned long address)
/*
* ISA I/O bus memory addresses are 1:1 with the physical address.
*/
-static inline unsigned long isa_virt_to_bus(volatile void * address)
+static inline unsigned long isa_virt_to_bus(volatile void *address)
{
- return (unsigned long)address - PAGE_OFFSET;
+ return virt_to_phys(address);
}
-static inline void * isa_bus_to_virt(unsigned long address)
+static inline void *isa_bus_to_virt(unsigned long address)
{
- return (void *)(address + PAGE_OFFSET);
+ return phys_to_virt(address);
}
#define isa_page_to_bus page_to_phys
@@ -412,6 +412,8 @@ static inline type pfx##in##bwlq##p(unsigned long port) \
__val = *__addr; \
slow; \
\
+ /* prevent prefetching of coherent DMA data prematurely */ \
+ rmb(); \
return pfx##ioswab##bwlq(__addr, __val); \
}
diff --git a/arch/mips/include/asm/mach-ath79/ar71xx_regs.h b/arch/mips/include/asm/mach-ath79/ar71xx_regs.h
index aa3800c..d99ca86 100644
--- a/arch/mips/include/asm/mach-ath79/ar71xx_regs.h
+++ b/arch/mips/include/asm/mach-ath79/ar71xx_regs.h
@@ -167,7 +167,7 @@
#define AR71XX_AHB_DIV_MASK 0x7
#define AR724X_PLL_REG_CPU_CONFIG 0x00
-#define AR724X_PLL_REG_PCIE_CONFIG 0x18
+#define AR724X_PLL_REG_PCIE_CONFIG 0x10
#define AR724X_PLL_FB_SHIFT 0
#define AR724X_PLL_FB_MASK 0x3ff
diff --git a/arch/mips/include/asm/mach-ath79/ath79.h b/arch/mips/include/asm/mach-ath79/ath79.h
index 441faa9..6e6c0fe 100644
--- a/arch/mips/include/asm/mach-ath79/ath79.h
+++ b/arch/mips/include/asm/mach-ath79/ath79.h
@@ -134,6 +134,7 @@ static inline u32 ath79_pll_rr(unsigned reg)
static inline void ath79_reset_wr(unsigned reg, u32 val)
{
__raw_writel(val, ath79_reset_base + reg);
+ (void) __raw_readl(ath79_reset_base + reg); /* flush */
}
static inline u32 ath79_reset_rr(unsigned reg)
diff --git a/arch/mips/include/asm/mach-loongson64/irq.h b/arch/mips/include/asm/mach-loongson64/irq.h
index d18c45c..19ff9ce 100644
--- a/arch/mips/include/asm/mach-loongson64/irq.h
+++ b/arch/mips/include/asm/mach-loongson64/irq.h
@@ -9,7 +9,7 @@
#define MIPS_CPU_IRQ_BASE 56
#define LOONGSON_UART_IRQ (MIPS_CPU_IRQ_BASE + 2) /* UART */
-#define LOONGSON_HT1_IRQ (MIPS_CPU_IRQ_BASE + 3) /* HT1 */
+#define LOONGSON_BRIDGE_IRQ (MIPS_CPU_IRQ_BASE + 3) /* CASCADE */
#define LOONGSON_TIMER_IRQ (MIPS_CPU_IRQ_BASE + 7) /* CPU Timer */
#define LOONGSON_HT1_CFG_BASE loongson_sysconf.ht_control_base
diff --git a/arch/mips/include/asm/machine.h b/arch/mips/include/asm/machine.h
index 6b444cd..db930cd 100644
--- a/arch/mips/include/asm/machine.h
+++ b/arch/mips/include/asm/machine.h
@@ -52,7 +52,7 @@ mips_machine_is_compatible(const struct mips_machine *mach, const void *fdt)
if (!mach->matches)
return NULL;
- for (match = mach->matches; match->compatible; match++) {
+ for (match = mach->matches; match->compatible[0]; match++) {
if (fdt_node_check_compatible(fdt, 0, match->compatible) == 0)
return match;
}
diff --git a/arch/mips/include/asm/pgtable-64.h b/arch/mips/include/asm/pgtable-64.h
index 514cbc0..ef6f007 100644
--- a/arch/mips/include/asm/pgtable-64.h
+++ b/arch/mips/include/asm/pgtable-64.h
@@ -193,6 +193,11 @@ static inline int pmd_bad(pmd_t pmd)
static inline int pmd_present(pmd_t pmd)
{
+#ifdef CONFIG_MIPS_HUGE_TLB_SUPPORT
+ if (unlikely(pmd_val(pmd) & _PAGE_HUGE))
+ return pmd_val(pmd) & _PAGE_PRESENT;
+#endif
+
return pmd_val(pmd) != (unsigned long) invalid_pte_table;
}
diff --git a/arch/mips/include/asm/processor.h b/arch/mips/include/asm/processor.h
index 0d36c87..ad6f019 100644
--- a/arch/mips/include/asm/processor.h
+++ b/arch/mips/include/asm/processor.h
@@ -141,7 +141,7 @@ struct mips_fpu_struct {
#define NUM_DSP_REGS 6
-typedef __u32 dspreg_t;
+typedef unsigned long dspreg_t;
struct mips_dsp_state {
dspreg_t dspr[NUM_DSP_REGS];
diff --git a/arch/mips/include/asm/syscall.h b/arch/mips/include/asm/syscall.h
index d878825..deee3ac 100644
--- a/arch/mips/include/asm/syscall.h
+++ b/arch/mips/include/asm/syscall.h
@@ -51,7 +51,7 @@ static inline unsigned long mips_get_syscall_arg(unsigned long *arg,
#ifdef CONFIG_64BIT
case 4: case 5: case 6: case 7:
#ifdef CONFIG_MIPS32_O32
- if (test_thread_flag(TIF_32BIT_REGS))
+ if (test_tsk_thread_flag(task, TIF_32BIT_REGS))
return get_user(*arg, (int *)usp + n);
else
#endif
diff --git a/arch/mips/include/uapi/asm/inst.h b/arch/mips/include/uapi/asm/inst.h
index 77429d1..711d9b8 100644
--- a/arch/mips/include/uapi/asm/inst.h
+++ b/arch/mips/include/uapi/asm/inst.h
@@ -964,7 +964,7 @@ struct mm16_r3_format { /* Load from global pointer format */
struct mm16_r5_format { /* Load/store from stack pointer format */
__BITFIELD_FIELD(unsigned int opcode : 6,
__BITFIELD_FIELD(unsigned int rt : 5,
- __BITFIELD_FIELD(signed int simmediate : 5,
+ __BITFIELD_FIELD(unsigned int imm : 5,
__BITFIELD_FIELD(unsigned int : 16, /* Ignored */
;))))
};
diff --git a/arch/mips/jz4740/Platform b/arch/mips/jz4740/Platform
index 28448d35..a2a5a85 100644
--- a/arch/mips/jz4740/Platform
+++ b/arch/mips/jz4740/Platform
@@ -1,4 +1,4 @@
platform-$(CONFIG_MACH_INGENIC) += jz4740/
cflags-$(CONFIG_MACH_INGENIC) += -I$(srctree)/arch/mips/include/asm/mach-jz4740
load-$(CONFIG_MACH_INGENIC) += 0xffffffff80010000
-zload-$(CONFIG_MACH_INGENIC) += 0xffffffff80600000
+zload-$(CONFIG_MACH_INGENIC) += 0xffffffff81000000
diff --git a/arch/mips/kernel/crash.c b/arch/mips/kernel/crash.c
index 1723b17..e757f36 100644
--- a/arch/mips/kernel/crash.c
+++ b/arch/mips/kernel/crash.c
@@ -34,6 +34,9 @@ static void crash_shutdown_secondary(void *passed_regs)
if (!cpu_online(cpu))
return;
+ /* We won't be sent IPIs any more. */
+ set_cpu_online(cpu, false);
+
local_irq_disable();
if (!cpumask_test_cpu(cpu, &cpus_in_crash))
crash_save_cpu(regs, cpu);
diff --git a/arch/mips/kernel/machine_kexec.c b/arch/mips/kernel/machine_kexec.c
index 5972520..32b567e 100644
--- a/arch/mips/kernel/machine_kexec.c
+++ b/arch/mips/kernel/machine_kexec.c
@@ -96,6 +96,9 @@ machine_kexec(struct kimage *image)
*ptr = (unsigned long) phys_to_virt(*ptr);
}
+ /* Mark offline BEFORE disabling local irq. */
+ set_cpu_online(smp_processor_id(), false);
+
/*
* we do not want to be bothered.
*/
diff --git a/arch/mips/kernel/mcount.S b/arch/mips/kernel/mcount.S
index 2f7c734..0df911e 100644
--- a/arch/mips/kernel/mcount.S
+++ b/arch/mips/kernel/mcount.S
@@ -116,10 +116,20 @@
NESTED(_mcount, PT_SIZE, ra)
PTR_LA t1, ftrace_stub
PTR_L t2, ftrace_trace_function /* Prepare t2 for (1) */
- bne t1, t2, static_trace
+ beq t1, t2, fgraph_trace
nop
+ MCOUNT_SAVE_REGS
+
+ move a0, ra /* arg1: self return address */
+ jalr t2 /* (1) call *ftrace_trace_function */
+ move a1, AT /* arg2: parent's return address */
+
+ MCOUNT_RESTORE_REGS
+
+fgraph_trace:
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
+ PTR_LA t1, ftrace_stub
PTR_L t3, ftrace_graph_return
bne t1, t3, ftrace_graph_caller
nop
@@ -128,24 +138,11 @@
bne t1, t3, ftrace_graph_caller
nop
#endif
- b ftrace_stub
-#ifdef CONFIG_32BIT
- addiu sp, sp, 8
-#else
- nop
-#endif
-static_trace:
- MCOUNT_SAVE_REGS
-
- move a0, ra /* arg1: self return address */
- jalr t2 /* (1) call *ftrace_trace_function */
- move a1, AT /* arg2: parent's return address */
-
- MCOUNT_RESTORE_REGS
#ifdef CONFIG_32BIT
addiu sp, sp, 8
#endif
+
.globl ftrace_stub
ftrace_stub:
RETURN_BACK
diff --git a/arch/mips/kernel/process.c b/arch/mips/kernel/process.c
index 6e716a5..1cc133e 100644
--- a/arch/mips/kernel/process.c
+++ b/arch/mips/kernel/process.c
@@ -26,6 +26,7 @@
#include <linux/kallsyms.h>
#include <linux/random.h>
#include <linux/prctl.h>
+#include <linux/nmi.h>
#include <asm/asm.h>
#include <asm/bootinfo.h>
@@ -117,7 +118,6 @@ int copy_thread(unsigned long clone_flags, unsigned long usp,
struct thread_info *ti = task_thread_info(p);
struct pt_regs *childregs, *regs = current_pt_regs();
unsigned long childksp;
- p->set_child_tid = p->clear_child_tid = NULL;
childksp = (unsigned long)task_stack_page(p) + THREAD_SIZE - 32;
@@ -212,7 +212,7 @@ static inline int is_ra_save_ins(union mips_instruction *ip, int *poff)
if (ip->mm16_r5_format.rt != 31)
return 0;
- *poff = ip->mm16_r5_format.simmediate;
+ *poff = ip->mm16_r5_format.imm;
*poff = (*poff << 2) / sizeof(ulong);
return 1;
@@ -346,6 +346,7 @@ static int get_frame_info(struct mips_frame_info *info)
bool is_mmips = IS_ENABLED(CONFIG_CPU_MICROMIPS);
union mips_instruction insn, *ip, *ip_end;
const unsigned int max_insns = 128;
+ unsigned int last_insn_size = 0;
unsigned int i;
info->pc_offset = -1;
@@ -357,15 +358,19 @@ static int get_frame_info(struct mips_frame_info *info)
ip_end = (void *)ip + info->func_size;
- for (i = 0; i < max_insns && ip < ip_end; i++, ip++) {
+ for (i = 0; i < max_insns && ip < ip_end; i++) {
+ ip = (void *)ip + last_insn_size;
if (is_mmips && mm_insn_16bit(ip->halfword[0])) {
insn.halfword[0] = 0;
insn.halfword[1] = ip->halfword[0];
+ last_insn_size = 2;
} else if (is_mmips) {
insn.halfword[0] = ip->halfword[1];
insn.halfword[1] = ip->halfword[0];
+ last_insn_size = 4;
} else {
insn.word = ip->word;
+ last_insn_size = 4;
}
if (is_jump_ins(&insn))
@@ -387,8 +392,6 @@ static int get_frame_info(struct mips_frame_info *info)
tmp = (ip->halfword[0] >> 1);
info->frame_size = -(signed short)(tmp & 0xf);
}
- ip = (void *) &ip->halfword[1];
- ip--;
} else
#endif
info->frame_size = - ip->i_format.simmediate;
@@ -633,28 +636,42 @@ unsigned long arch_align_stack(unsigned long sp)
return sp & ALMASK;
}
-static void arch_dump_stack(void *info)
+static DEFINE_PER_CPU(struct call_single_data, backtrace_csd);
+static struct cpumask backtrace_csd_busy;
+
+static void handle_backtrace(void *info)
{
- struct pt_regs *regs;
+ nmi_cpu_backtrace(get_irq_regs());
+ cpumask_clear_cpu(smp_processor_id(), &backtrace_csd_busy);
+}
- regs = get_irq_regs();
+static void raise_backtrace(cpumask_t *mask)
+{
+ struct call_single_data *csd;
+ int cpu;
- if (regs)
- show_regs(regs);
+ for_each_cpu(cpu, mask) {
+ /*
+ * If we previously sent an IPI to the target CPU & it hasn't
+ * cleared its bit in the busy cpumask then it didn't handle
+ * our previous IPI & it's not safe for us to reuse the
+ * call_single_data_t.
+ */
+ if (cpumask_test_and_set_cpu(cpu, &backtrace_csd_busy)) {
+ pr_warn("Unable to send backtrace IPI to CPU%u - perhaps it hung?\n",
+ cpu);
+ continue;
+ }
- dump_stack();
+ csd = &per_cpu(backtrace_csd, cpu);
+ csd->func = handle_backtrace;
+ smp_call_function_single_async(cpu, csd);
+ }
}
void arch_trigger_cpumask_backtrace(const cpumask_t *mask, bool exclude_self)
{
- long this_cpu = get_cpu();
-
- if (cpumask_test_cpu(this_cpu, mask) && !exclude_self)
- dump_stack();
-
- smp_call_function_many(mask, arch_dump_stack, NULL, 1);
-
- put_cpu();
+ nmi_trigger_cpumask_backtrace(mask, exclude_self, raise_backtrace);
}
int mips_get_process_fp_mode(struct task_struct *task)
@@ -699,6 +716,10 @@ int mips_set_process_fp_mode(struct task_struct *task, unsigned int value)
if (value & ~known_bits)
return -EOPNOTSUPP;
+ /* Setting FRE without FR is not supported. */
+ if ((value & (PR_FP_MODE_FR | PR_FP_MODE_FRE)) == PR_FP_MODE_FRE)
+ return -EOPNOTSUPP;
+
/* Avoid inadvertently triggering emulation */
if ((value & PR_FP_MODE_FR) && raw_cpu_has_fpu &&
!(raw_current_cpu_data.fpu_id & MIPS_FPIR_F64))
diff --git a/arch/mips/kernel/ptrace.c b/arch/mips/kernel/ptrace.c
index 0c8ae2c..b702ba3 100644
--- a/arch/mips/kernel/ptrace.c
+++ b/arch/mips/kernel/ptrace.c
@@ -483,7 +483,7 @@ static int fpr_get_msa(struct task_struct *target,
/*
* Copy the floating-point context to the supplied NT_PRFPREG buffer.
* Choose the appropriate helper for general registers, and then copy
- * the FCSR register separately.
+ * the FCSR and FIR registers separately.
*/
static int fpr_get(struct task_struct *target,
const struct user_regset *regset,
@@ -491,6 +491,7 @@ static int fpr_get(struct task_struct *target,
void *kbuf, void __user *ubuf)
{
const int fcr31_pos = NUM_FPU_REGS * sizeof(elf_fpreg_t);
+ const int fir_pos = fcr31_pos + sizeof(u32);
int err;
if (sizeof(target->thread.fpu.fpr[0]) == sizeof(elf_fpreg_t))
@@ -503,6 +504,12 @@ static int fpr_get(struct task_struct *target,
err = user_regset_copyout(&pos, &count, &kbuf, &ubuf,
&target->thread.fpu.fcr31,
fcr31_pos, fcr31_pos + sizeof(u32));
+ if (err)
+ return err;
+
+ err = user_regset_copyout(&pos, &count, &kbuf, &ubuf,
+ &boot_cpu_data.fpu_id,
+ fir_pos, fir_pos + sizeof(u32));
return err;
}
@@ -551,7 +558,8 @@ static int fpr_set_msa(struct task_struct *target,
/*
* Copy the supplied NT_PRFPREG buffer to the floating-point context.
* Choose the appropriate helper for general registers, and then copy
- * the FCSR register separately.
+ * the FCSR register separately. Ignore the incoming FIR register
+ * contents though, as the register is read-only.
*
* We optimize for the case where `count % sizeof(elf_fpreg_t) == 0',
* which is supposed to have been guaranteed by the kernel before
@@ -565,6 +573,7 @@ static int fpr_set(struct task_struct *target,
const void *kbuf, const void __user *ubuf)
{
const int fcr31_pos = NUM_FPU_REGS * sizeof(elf_fpreg_t);
+ const int fir_pos = fcr31_pos + sizeof(u32);
u32 fcr31;
int err;
@@ -592,6 +601,11 @@ static int fpr_set(struct task_struct *target,
ptrace_setfcr31(target, fcr31);
}
+ if (count > 0)
+ err = user_regset_copyin_ignore(&pos, &count, &kbuf, &ubuf,
+ fir_pos,
+ fir_pos + sizeof(u32));
+
return err;
}
@@ -813,7 +827,7 @@ long arch_ptrace(struct task_struct *child, long request,
fregs = get_fpu_regs(child);
#ifdef CONFIG_32BIT
- if (test_thread_flag(TIF_32BIT_FPREGS)) {
+ if (test_tsk_thread_flag(child, TIF_32BIT_FPREGS)) {
/*
* The odd registers are actually the high
* order bits of the values stored in the even
@@ -824,7 +838,7 @@ long arch_ptrace(struct task_struct *child, long request,
break;
}
#endif
- tmp = get_fpr32(&fregs[addr - FPR_BASE], 0);
+ tmp = get_fpr64(&fregs[addr - FPR_BASE], 0);
break;
case PC:
tmp = regs->cp0_epc;
@@ -862,7 +876,7 @@ long arch_ptrace(struct task_struct *child, long request,
goto out;
}
dregs = __get_dsp_regs(child);
- tmp = (unsigned long) (dregs[addr - DSP_BASE]);
+ tmp = dregs[addr - DSP_BASE];
break;
}
case DSP_CONTROL:
@@ -902,7 +916,7 @@ long arch_ptrace(struct task_struct *child, long request,
init_fp_ctx(child);
#ifdef CONFIG_32BIT
- if (test_thread_flag(TIF_32BIT_FPREGS)) {
+ if (test_tsk_thread_flag(child, TIF_32BIT_FPREGS)) {
/*
* The odd registers are actually the high
* order bits of the values stored in the even
diff --git a/arch/mips/kernel/ptrace32.c b/arch/mips/kernel/ptrace32.c
index 5fcbdcd..4840af1 100644
--- a/arch/mips/kernel/ptrace32.c
+++ b/arch/mips/kernel/ptrace32.c
@@ -97,7 +97,7 @@ long compat_arch_ptrace(struct task_struct *child, compat_long_t request,
break;
}
fregs = get_fpu_regs(child);
- if (test_thread_flag(TIF_32BIT_FPREGS)) {
+ if (test_tsk_thread_flag(child, TIF_32BIT_FPREGS)) {
/*
* The odd registers are actually the high
* order bits of the values stored in the even
@@ -107,7 +107,7 @@ long compat_arch_ptrace(struct task_struct *child, compat_long_t request,
addr & 1);
break;
}
- tmp = get_fpr32(&fregs[addr - FPR_BASE], 0);
+ tmp = get_fpr64(&fregs[addr - FPR_BASE], 0);
break;
case PC:
tmp = regs->cp0_epc;
@@ -140,7 +140,7 @@ long compat_arch_ptrace(struct task_struct *child, compat_long_t request,
goto out;
}
dregs = __get_dsp_regs(child);
- tmp = (unsigned long) (dregs[addr - DSP_BASE]);
+ tmp = dregs[addr - DSP_BASE];
break;
}
case DSP_CONTROL:
@@ -204,7 +204,7 @@ long compat_arch_ptrace(struct task_struct *child, compat_long_t request,
sizeof(child->thread.fpu));
child->thread.fpu.fcr31 = 0;
}
- if (test_thread_flag(TIF_32BIT_FPREGS)) {
+ if (test_tsk_thread_flag(child, TIF_32BIT_FPREGS)) {
/*
* The odd registers are actually the high
* order bits of the values stored in the even
diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index bb1d9ff..8e07496 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -351,6 +351,7 @@ static void __show_regs(const struct pt_regs *regs)
void show_regs(struct pt_regs *regs)
{
__show_regs((struct pt_regs *)regs);
+ dump_stack();
}
void show_registers(struct pt_regs *regs)
diff --git a/arch/mips/kernel/vdso.c b/arch/mips/kernel/vdso.c
index f9dbfb1..c6297a0 100644
--- a/arch/mips/kernel/vdso.c
+++ b/arch/mips/kernel/vdso.c
@@ -14,12 +14,14 @@
#include <linux/init.h>
#include <linux/ioport.h>
#include <linux/irqchip/mips-gic.h>
+#include <linux/kernel.h>
#include <linux/mm.h>
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/timekeeper_internal.h>
#include <asm/abi.h>
+#include <asm/page.h>
#include <asm/vdso.h>
/* Kernel-provided data used by the VDSO. */
@@ -109,8 +111,8 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
/* Map delay slot emulation page */
base = mmap_region(NULL, STACK_TOP, PAGE_SIZE,
- VM_READ|VM_WRITE|VM_EXEC|
- VM_MAYREAD|VM_MAYWRITE|VM_MAYEXEC,
+ VM_READ | VM_EXEC |
+ VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC,
0);
if (IS_ERR_VALUE(base)) {
ret = base;
@@ -129,12 +131,30 @@ int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
vvar_size = gic_size + PAGE_SIZE;
size = vvar_size + image->size;
+ /*
+ * Find a region that's large enough for us to perform the
+ * colour-matching alignment below.
+ */
+ if (cpu_has_dc_aliases)
+ size += shm_align_mask + 1;
+
base = get_unmapped_area(NULL, 0, size, 0, 0);
if (IS_ERR_VALUE(base)) {
ret = base;
goto out;
}
+ /*
+ * If we suffer from dcache aliasing, ensure that the VDSO data page
+ * mapping is coloured the same as the kernel's mapping of that memory.
+ * This ensures that when the kernel updates the VDSO data userland
+ * will observe it without requiring cache invalidations.
+ */
+ if (cpu_has_dc_aliases) {
+ base = __ALIGN_MASK(base, shm_align_mask);
+ base += ((unsigned long)&vdso_data - gic_size) & shm_align_mask;
+ }
+
data_addr = base + gic_size;
vdso_addr = data_addr + PAGE_SIZE;
diff --git a/arch/mips/kvm/mips.c b/arch/mips/kvm/mips.c
index 29ec9ab..a2c46f5 100644
--- a/arch/mips/kvm/mips.c
+++ b/arch/mips/kvm/mips.c
@@ -42,7 +42,7 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {
{ "cache", VCPU_STAT(cache_exits), KVM_STAT_VCPU },
{ "signal", VCPU_STAT(signal_exits), KVM_STAT_VCPU },
{ "interrupt", VCPU_STAT(int_exits), KVM_STAT_VCPU },
- { "cop_unsuable", VCPU_STAT(cop_unusable_exits), KVM_STAT_VCPU },
+ { "cop_unusable", VCPU_STAT(cop_unusable_exits), KVM_STAT_VCPU },
{ "tlbmod", VCPU_STAT(tlbmod_exits), KVM_STAT_VCPU },
{ "tlbmiss_ld", VCPU_STAT(tlbmiss_ld_exits), KVM_STAT_VCPU },
{ "tlbmiss_st", VCPU_STAT(tlbmiss_st_exits), KVM_STAT_VCPU },
diff --git a/arch/mips/lib/multi3.c b/arch/mips/lib/multi3.c
index 111ad47..4c2483f 100644
--- a/arch/mips/lib/multi3.c
+++ b/arch/mips/lib/multi3.c
@@ -4,12 +4,12 @@
#include "libgcc.h"
/*
- * GCC 7 suboptimally generates __multi3 calls for mips64r6, so for that
- * specific case only we'll implement it here.
+ * GCC 7 & older can suboptimally generate __multi3 calls for mips64r6, so for
+ * that specific case only we implement that intrinsic here.
*
* See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82981
*/
-#if defined(CONFIG_64BIT) && defined(CONFIG_CPU_MIPSR6) && (__GNUC__ == 7)
+#if defined(CONFIG_64BIT) && defined(CONFIG_CPU_MIPSR6) && (__GNUC__ < 8)
/* multiply 64-bit values, low 64-bits returned */
static inline long long notrace dmulu(long long a, long long b)
diff --git a/arch/mips/loongson64/common/cs5536/cs5536_ohci.c b/arch/mips/loongson64/common/cs5536/cs5536_ohci.c
index f7c905e..92dc6ba 100644
--- a/arch/mips/loongson64/common/cs5536/cs5536_ohci.c
+++ b/arch/mips/loongson64/common/cs5536/cs5536_ohci.c
@@ -138,7 +138,7 @@ u32 pci_ohci_read_reg(int reg)
break;
case PCI_OHCI_INT_REG:
_rdmsr(DIVIL_MSR_REG(PIC_YSEL_LOW), &hi, &lo);
- if ((lo & 0x00000f00) == CS5536_USB_INTR)
+ if (((lo >> PIC_YSEL_LOW_USB_SHIFT) & 0xf) == CS5536_USB_INTR)
conf_data = 1;
break;
default:
diff --git a/arch/mips/loongson64/loongson-3/irq.c b/arch/mips/loongson64/loongson-3/irq.c
index 8e76490..027f53e 100644
--- a/arch/mips/loongson64/loongson-3/irq.c
+++ b/arch/mips/loongson64/loongson-3/irq.c
@@ -44,51 +44,8 @@ void mach_irq_dispatch(unsigned int pending)
}
}
-static struct irqaction cascade_irqaction = {
- .handler = no_action,
- .flags = IRQF_NO_SUSPEND,
- .name = "cascade",
-};
-
-static inline void mask_loongson_irq(struct irq_data *d)
-{
- clear_c0_status(0x100 << (d->irq - MIPS_CPU_IRQ_BASE));
- irq_disable_hazard();
-
- /* Workaround: UART IRQ may deliver to any core */
- if (d->irq == LOONGSON_UART_IRQ) {
- int cpu = smp_processor_id();
- int node_id = cpu_logical_map(cpu) / loongson_sysconf.cores_per_node;
- int core_id = cpu_logical_map(cpu) % loongson_sysconf.cores_per_node;
- u64 intenclr_addr = smp_group[node_id] |
- (u64)(&LOONGSON_INT_ROUTER_INTENCLR);
- u64 introuter_lpc_addr = smp_group[node_id] |
- (u64)(&LOONGSON_INT_ROUTER_LPC);
-
- *(volatile u32 *)intenclr_addr = 1 << 10;
- *(volatile u8 *)introuter_lpc_addr = 0x10 + (1<<core_id);
- }
-}
-
-static inline void unmask_loongson_irq(struct irq_data *d)
-{
- /* Workaround: UART IRQ may deliver to any core */
- if (d->irq == LOONGSON_UART_IRQ) {
- int cpu = smp_processor_id();
- int node_id = cpu_logical_map(cpu) / loongson_sysconf.cores_per_node;
- int core_id = cpu_logical_map(cpu) % loongson_sysconf.cores_per_node;
- u64 intenset_addr = smp_group[node_id] |
- (u64)(&LOONGSON_INT_ROUTER_INTENSET);
- u64 introuter_lpc_addr = smp_group[node_id] |
- (u64)(&LOONGSON_INT_ROUTER_LPC);
-
- *(volatile u32 *)intenset_addr = 1 << 10;
- *(volatile u8 *)introuter_lpc_addr = 0x10 + (1<<core_id);
- }
-
- set_c0_status(0x100 << (d->irq - MIPS_CPU_IRQ_BASE));
- irq_enable_hazard();
-}
+static inline void mask_loongson_irq(struct irq_data *d) { }
+static inline void unmask_loongson_irq(struct irq_data *d) { }
/* For MIPS IRQs which shared by all cores */
static struct irq_chip loongson_irq_chip = {
@@ -126,12 +83,11 @@ void __init mach_init_irq(void)
mips_cpu_irq_init();
init_i8259_irqs();
irq_set_chip_and_handler(LOONGSON_UART_IRQ,
- &loongson_irq_chip, handle_level_irq);
+ &loongson_irq_chip, handle_percpu_irq);
+ irq_set_chip_and_handler(LOONGSON_BRIDGE_IRQ,
+ &loongson_irq_chip, handle_percpu_irq);
- /* setup HT1 irq */
- setup_irq(LOONGSON_HT1_IRQ, &cascade_irqaction);
-
- set_c0_status(STATUSF_IP2 | STATUSF_IP6);
+ set_c0_status(STATUSF_IP2 | STATUSF_IP3 | STATUSF_IP6);
}
#ifdef CONFIG_HOTPLUG_CPU
diff --git a/arch/mips/math-emu/dsemul.c b/arch/mips/math-emu/dsemul.c
index 4a094f7..7b43298 100644
--- a/arch/mips/math-emu/dsemul.c
+++ b/arch/mips/math-emu/dsemul.c
@@ -211,8 +211,9 @@ int mips_dsemul(struct pt_regs *regs, mips_instruction ir,
{
int isa16 = get_isa16_mode(regs->cp0_epc);
mips_instruction break_math;
- struct emuframe __user *fr;
- int err, fr_idx;
+ unsigned long fr_uaddr;
+ struct emuframe fr;
+ int fr_idx, ret;
/* NOP is easy */
if (ir == 0)
@@ -247,27 +248,31 @@ int mips_dsemul(struct pt_regs *regs, mips_instruction ir,
fr_idx = alloc_emuframe();
if (fr_idx == BD_EMUFRAME_NONE)
return SIGBUS;
- fr = &dsemul_page()[fr_idx];
/* Retrieve the appropriately encoded break instruction */
break_math = BREAK_MATH(isa16);
/* Write the instructions to the frame */
if (isa16) {
- err = __put_user(ir >> 16,
- (u16 __user *)(&fr->emul));
- err |= __put_user(ir & 0xffff,
- (u16 __user *)((long)(&fr->emul) + 2));
- err |= __put_user(break_math >> 16,
- (u16 __user *)(&fr->badinst));
- err |= __put_user(break_math & 0xffff,
- (u16 __user *)((long)(&fr->badinst) + 2));
+ union mips_instruction _emul = {
+ .halfword = { ir >> 16, ir }
+ };
+ union mips_instruction _badinst = {
+ .halfword = { break_math >> 16, break_math }
+ };
+
+ fr.emul = _emul.word;
+ fr.badinst = _badinst.word;
} else {
- err = __put_user(ir, &fr->emul);
- err |= __put_user(break_math, &fr->badinst);
+ fr.emul = ir;
+ fr.badinst = break_math;
}
- if (unlikely(err)) {
+ /* Write the frame to user memory */
+ fr_uaddr = (unsigned long)&dsemul_page()[fr_idx];
+ ret = access_process_vm(current, fr_uaddr, &fr, sizeof(fr),
+ FOLL_FORCE | FOLL_WRITE);
+ if (unlikely(ret != sizeof(fr))) {
MIPS_FPU_EMU_INC_STATS(errors);
free_emuframe(fr_idx, current->mm);
return SIGBUS;
@@ -279,10 +284,7 @@ int mips_dsemul(struct pt_regs *regs, mips_instruction ir,
atomic_set(¤t->thread.bd_emu_frame, fr_idx);
/* Change user register context to execute the frame */
- regs->cp0_epc = (unsigned long)&fr->emul | isa16;
-
- /* Ensure the icache observes our newly written frame */
- flush_cache_sigtramp((unsigned long)&fr->emul);
+ regs->cp0_epc = fr_uaddr | isa16;
return 0;
}
diff --git a/arch/mips/mm/c-r4k.c b/arch/mips/mm/c-r4k.c
index 9d0107f..0ff379f 100644
--- a/arch/mips/mm/c-r4k.c
+++ b/arch/mips/mm/c-r4k.c
@@ -835,7 +835,8 @@ static void r4k_flush_icache_user_range(unsigned long start, unsigned long end)
static void r4k_dma_cache_wback_inv(unsigned long addr, unsigned long size)
{
/* Catch bad driver code */
- BUG_ON(size == 0);
+ if (WARN_ON(size == 0))
+ return;
preempt_disable();
if (cpu_has_inclusive_pcaches) {
@@ -851,9 +852,12 @@ static void r4k_dma_cache_wback_inv(unsigned long addr, unsigned long size)
/*
* Either no secondary cache or the available caches don't have the
* subset property so we have to flush the primary caches
- * explicitly
+ * explicitly.
+ * If we would need IPI to perform an INDEX-type operation, then
+ * we have to use the HIT-type alternative as IPI cannot be used
+ * here due to interrupts possibly being disabled.
*/
- if (size >= dcache_size) {
+ if (!r4k_op_needs_ipi(R4K_INDEX) && size >= dcache_size) {
r4k_blast_dcache();
} else {
R4600_HIT_CACHEOP_WAR_IMPL;
@@ -868,7 +872,8 @@ static void r4k_dma_cache_wback_inv(unsigned long addr, unsigned long size)
static void r4k_dma_cache_inv(unsigned long addr, unsigned long size)
{
/* Catch bad driver code */
- BUG_ON(size == 0);
+ if (WARN_ON(size == 0))
+ return;
preempt_disable();
if (cpu_has_inclusive_pcaches) {
@@ -890,7 +895,7 @@ static void r4k_dma_cache_inv(unsigned long addr, unsigned long size)
return;
}
- if (size >= dcache_size) {
+ if (!r4k_op_needs_ipi(R4K_INDEX) && size >= dcache_size) {
r4k_blast_dcache();
} else {
R4600_HIT_CACHEOP_WAR_IMPL;
diff --git a/arch/mips/mm/ioremap.c b/arch/mips/mm/ioremap.c
index 1f18962..0dbcd90b 100644
--- a/arch/mips/mm/ioremap.c
+++ b/arch/mips/mm/ioremap.c
@@ -9,6 +9,7 @@
#include <linux/export.h>
#include <asm/addrspace.h>
#include <asm/byteorder.h>
+#include <linux/ioport.h>
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/vmalloc.h>
@@ -97,6 +98,20 @@ static int remap_area_pages(unsigned long address, phys_addr_t phys_addr,
return error;
}
+static int __ioremap_check_ram(unsigned long start_pfn, unsigned long nr_pages,
+ void *arg)
+{
+ unsigned long i;
+
+ for (i = 0; i < nr_pages; i++) {
+ if (pfn_valid(start_pfn + i) &&
+ !PageReserved(pfn_to_page(start_pfn + i)))
+ return 1;
+ }
+
+ return 0;
+}
+
/*
* Generic mapping function (not visible outside):
*/
@@ -115,8 +130,8 @@ static int remap_area_pages(unsigned long address, phys_addr_t phys_addr,
void __iomem * __ioremap(phys_addr_t phys_addr, phys_addr_t size, unsigned long flags)
{
+ unsigned long offset, pfn, last_pfn;
struct vm_struct * area;
- unsigned long offset;
phys_addr_t last_addr;
void * addr;
@@ -136,18 +151,16 @@ void __iomem * __ioremap(phys_addr_t phys_addr, phys_addr_t size, unsigned long
return (void __iomem *) CKSEG1ADDR(phys_addr);
/*
- * Don't allow anybody to remap normal RAM that we're using..
+ * Don't allow anybody to remap RAM that may be allocated by the page
+ * allocator, since that could lead to races & data clobbering.
*/
- if (phys_addr < virt_to_phys(high_memory)) {
- char *t_addr, *t_end;
- struct page *page;
-
- t_addr = __va(phys_addr);
- t_end = t_addr + (size - 1);
-
- for(page = virt_to_page(t_addr); page <= virt_to_page(t_end); page++)
- if(!PageReserved(page))
- return NULL;
+ pfn = PFN_DOWN(phys_addr);
+ last_pfn = PFN_DOWN(last_addr);
+ if (walk_system_ram_range(pfn, last_pfn - pfn + 1, NULL,
+ __ioremap_check_ram) == 1) {
+ WARN_ONCE(1, "ioremap on RAM at %pa - %pa\n",
+ &phys_addr, &last_addr);
+ return NULL;
}
/*
diff --git a/arch/mips/pci/pci-legacy.c b/arch/mips/pci/pci-legacy.c
index 014649b..2d6886f 100644
--- a/arch/mips/pci/pci-legacy.c
+++ b/arch/mips/pci/pci-legacy.c
@@ -116,8 +116,12 @@ static void pcibios_scanbus(struct pci_controller *hose)
if (pci_has_flag(PCI_PROBE_ONLY)) {
pci_bus_claim_resources(bus);
} else {
+ struct pci_bus *child;
+
pci_bus_size_bridges(bus);
pci_bus_assign_resources(bus);
+ list_for_each_entry(child, &bus->children, node)
+ pcie_bus_configure_settings(child);
}
pci_bus_add_devices(bus);
}
diff --git a/arch/mips/pci/pci.c b/arch/mips/pci/pci.c
index f6325fa..64ae8c0 100644
--- a/arch/mips/pci/pci.c
+++ b/arch/mips/pci/pci.c
@@ -55,7 +55,7 @@ void pci_resource_to_user(const struct pci_dev *dev, int bar,
phys_addr_t size = resource_size(rsrc);
*start = fixup_bigphys_addr(rsrc->start, size);
- *end = rsrc->start + size;
+ *end = rsrc->start + size - 1;
}
int pci_mmap_page_range(struct pci_dev *dev, struct vm_area_struct *vma,
diff --git a/arch/mips/ralink/mt7620.c b/arch/mips/ralink/mt7620.c
index 0696142..9194b04 100644
--- a/arch/mips/ralink/mt7620.c
+++ b/arch/mips/ralink/mt7620.c
@@ -81,7 +81,7 @@ static struct rt2880_pmx_func pcie_rst_grp[] = {
};
static struct rt2880_pmx_func nd_sd_grp[] = {
FUNC("nand", MT7620_GPIO_MODE_NAND, 45, 15),
- FUNC("sd", MT7620_GPIO_MODE_SD, 45, 15)
+ FUNC("sd", MT7620_GPIO_MODE_SD, 47, 13)
};
static struct rt2880_pmx_group mt7620a_pinmux_data[] = {
diff --git a/arch/mips/txx9/rbtx4939/setup.c b/arch/mips/txx9/rbtx4939/setup.c
index 8b93730..fd26fad 100644
--- a/arch/mips/txx9/rbtx4939/setup.c
+++ b/arch/mips/txx9/rbtx4939/setup.c
@@ -186,7 +186,7 @@ static void __init rbtx4939_update_ioc_pen(void)
#define RBTX4939_MAX_7SEGLEDS 8
-#if IS_ENABLED(CONFIG_LEDS_CLASS)
+#if IS_BUILTIN(CONFIG_LEDS_CLASS)
static u8 led_val[RBTX4939_MAX_7SEGLEDS];
struct rbtx4939_led_data {
struct led_classdev cdev;
@@ -261,7 +261,7 @@ static inline void rbtx4939_led_setup(void)
static void __rbtx4939_7segled_putc(unsigned int pos, unsigned char val)
{
-#if IS_ENABLED(CONFIG_LEDS_CLASS)
+#if IS_BUILTIN(CONFIG_LEDS_CLASS)
unsigned long flags;
local_irq_save(flags);
/* bit7: reserved for LED class */
diff --git a/arch/openrisc/kernel/process.c b/arch/openrisc/kernel/process.c
index 7095dfe..9623721 100644
--- a/arch/openrisc/kernel/process.c
+++ b/arch/openrisc/kernel/process.c
@@ -152,8 +152,6 @@ copy_thread(unsigned long clone_flags, unsigned long usp,
top_of_kernel_stack = sp;
- p->set_child_tid = p->clear_child_tid = NULL;
-
/* Locate userspace context on stack... */
sp -= STACK_FRAME_OVERHEAD; /* redzone */
sp -= sizeof(struct pt_regs);
diff --git a/arch/parisc/Kconfig b/arch/parisc/Kconfig
index a14b865..3c37af1 100644
--- a/arch/parisc/Kconfig
+++ b/arch/parisc/Kconfig
@@ -184,7 +184,7 @@
config MLONGCALLS
bool "Enable the -mlong-calls compiler option for big kernels"
- def_bool y if (!MODULES)
+ default y
depends on PA8X00
help
If you configure the kernel to include many drivers built-in instead
diff --git a/arch/parisc/include/asm/barrier.h b/arch/parisc/include/asm/barrier.h
new file mode 100644
index 0000000..dbaaca84
--- /dev/null
+++ b/arch/parisc/include/asm/barrier.h
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_BARRIER_H
+#define __ASM_BARRIER_H
+
+#ifndef __ASSEMBLY__
+
+/* The synchronize caches instruction executes as a nop on systems in
+ which all memory references are performed in order. */
+#define synchronize_caches() __asm__ __volatile__ ("sync" : : : "memory")
+
+#if defined(CONFIG_SMP)
+#define mb() do { synchronize_caches(); } while (0)
+#define rmb() mb()
+#define wmb() mb()
+#define dma_rmb() mb()
+#define dma_wmb() mb()
+#else
+#define mb() barrier()
+#define rmb() barrier()
+#define wmb() barrier()
+#define dma_rmb() barrier()
+#define dma_wmb() barrier()
+#endif
+
+#define __smp_mb() mb()
+#define __smp_rmb() mb()
+#define __smp_wmb() mb()
+
+#include <asm-generic/barrier.h>
+
+#endif /* !__ASSEMBLY__ */
+#endif /* __ASM_BARRIER_H */
diff --git a/arch/parisc/include/asm/futex.h b/arch/parisc/include/asm/futex.h
index ac8bd58..06a1a88 100644
--- a/arch/parisc/include/asm/futex.h
+++ b/arch/parisc/include/asm/futex.h
@@ -32,22 +32,12 @@ _futex_spin_unlock_irqrestore(u32 __user *uaddr, unsigned long int *flags)
}
static inline int
-futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
+arch_futex_atomic_op_inuser(int op, int oparg, int *oval, u32 __user *uaddr)
{
unsigned long int flags;
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval, ret;
u32 tmp;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(*uaddr)))
- return -EFAULT;
-
_futex_spin_lock_irqsave(uaddr, &flags);
pagefault_disable();
@@ -85,17 +75,9 @@ futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
pagefault_enable();
_futex_spin_unlock_irqrestore(uaddr, &flags);
- if (ret == 0) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/parisc/include/asm/spinlock.h b/arch/parisc/include/asm/spinlock.h
index e32936c..7031483 100644
--- a/arch/parisc/include/asm/spinlock.h
+++ b/arch/parisc/include/asm/spinlock.h
@@ -26,7 +26,6 @@ static inline void arch_spin_lock_flags(arch_spinlock_t *x,
{
volatile unsigned int *a;
- mb();
a = __ldcw_align(x);
while (__ldcw(a) == 0)
while (*a == 0)
@@ -36,16 +35,15 @@ static inline void arch_spin_lock_flags(arch_spinlock_t *x,
local_irq_disable();
} else
cpu_relax();
- mb();
}
static inline void arch_spin_unlock(arch_spinlock_t *x)
{
volatile unsigned int *a;
- mb();
+
a = __ldcw_align(x);
- *a = 1;
mb();
+ *a = 1;
}
static inline int arch_spin_trylock(arch_spinlock_t *x)
@@ -53,10 +51,8 @@ static inline int arch_spin_trylock(arch_spinlock_t *x)
volatile unsigned int *a;
int ret;
- mb();
a = __ldcw_align(x);
ret = __ldcw(a) != 0;
- mb();
return ret;
}
diff --git a/arch/parisc/kernel/entry.S b/arch/parisc/kernel/entry.S
index e3d3e8e..63b140b 100644
--- a/arch/parisc/kernel/entry.S
+++ b/arch/parisc/kernel/entry.S
@@ -185,7 +185,7 @@
bv,n 0(%r3)
nop
.word 0 /* checksum (will be patched) */
- .word PA(os_hpmc) /* address of handler */
+ .word 0 /* address of handler */
.word 0 /* length of handler */
.endm
@@ -482,6 +482,8 @@
.macro tlb_unlock0 spc,tmp
#ifdef CONFIG_SMP
or,COND(=) %r0,\spc,%r0
+ sync
+ or,COND(=) %r0,\spc,%r0
stw \spc,0(\tmp)
#endif
.endm
diff --git a/arch/parisc/kernel/hpmc.S b/arch/parisc/kernel/hpmc.S
index 0fbd0a0..e88f4e7 100644
--- a/arch/parisc/kernel/hpmc.S
+++ b/arch/parisc/kernel/hpmc.S
@@ -83,7 +83,8 @@
.text
.import intr_save, code
-ENTRY_CFI(os_hpmc)
+ .align 16
+ENTRY(os_hpmc)
.os_hpmc:
/*
@@ -299,11 +300,14 @@
b .
nop
-ENDPROC_CFI(os_hpmc)
+ .align 16 /* make function length multiple of 16 bytes */
.os_hpmc_end:
__INITRODATA
- .export os_hpmc_size
+.globl os_hpmc_size
+ .align 4
+ .type os_hpmc_size, @object
+ .size os_hpmc_size, 4
os_hpmc_size:
.word .os_hpmc_end-.os_hpmc
diff --git a/arch/parisc/kernel/pacache.S b/arch/parisc/kernel/pacache.S
index 67b0f75..3e163df 100644
--- a/arch/parisc/kernel/pacache.S
+++ b/arch/parisc/kernel/pacache.S
@@ -354,6 +354,7 @@
.macro tlb_unlock la,flags,tmp
#ifdef CONFIG_SMP
ldi 1,\tmp
+ sync
stw \tmp,0(\la)
mtsm \flags
#endif
diff --git a/arch/parisc/kernel/syscall.S b/arch/parisc/kernel/syscall.S
index e775f80..5f7e57f 100644
--- a/arch/parisc/kernel/syscall.S
+++ b/arch/parisc/kernel/syscall.S
@@ -629,11 +629,12 @@
stw %r1, 4(%sr2,%r20)
#endif
/* The load and store could fail */
-1: ldw,ma 0(%r26), %r28
+1: ldw 0(%r26), %r28
sub,<> %r28, %r25, %r0
-2: stw,ma %r24, 0(%r26)
+2: stw %r24, 0(%r26)
/* Free lock */
- stw,ma %r20, 0(%sr2,%r20)
+ sync
+ stw %r20, 0(%sr2,%r20)
#if ENABLE_LWS_DEBUG
/* Clear thread register indicator */
stw %r0, 4(%sr2,%r20)
@@ -647,6 +648,7 @@
3:
/* Error occurred on load or store */
/* Free lock */
+ sync
stw %r20, 0(%sr2,%r20)
#if ENABLE_LWS_DEBUG
stw %r0, 4(%sr2,%r20)
@@ -796,30 +798,30 @@
ldo 1(%r0),%r28
/* 8bit CAS */
-13: ldb,ma 0(%r26), %r29
+13: ldb 0(%r26), %r29
sub,= %r29, %r25, %r0
b,n cas2_end
-14: stb,ma %r24, 0(%r26)
+14: stb %r24, 0(%r26)
b cas2_end
copy %r0, %r28
nop
nop
/* 16bit CAS */
-15: ldh,ma 0(%r26), %r29
+15: ldh 0(%r26), %r29
sub,= %r29, %r25, %r0
b,n cas2_end
-16: sth,ma %r24, 0(%r26)
+16: sth %r24, 0(%r26)
b cas2_end
copy %r0, %r28
nop
nop
/* 32bit CAS */
-17: ldw,ma 0(%r26), %r29
+17: ldw 0(%r26), %r29
sub,= %r29, %r25, %r0
b,n cas2_end
-18: stw,ma %r24, 0(%r26)
+18: stw %r24, 0(%r26)
b cas2_end
copy %r0, %r28
nop
@@ -827,10 +829,10 @@
/* 64bit CAS */
#ifdef CONFIG_64BIT
-19: ldd,ma 0(%r26), %r29
+19: ldd 0(%r26), %r29
sub,*= %r29, %r25, %r0
b,n cas2_end
-20: std,ma %r24, 0(%r26)
+20: std %r24, 0(%r26)
copy %r0, %r28
#else
/* Compare first word */
@@ -848,7 +850,8 @@
cas2_end:
/* Free lock */
- stw,ma %r20, 0(%sr2,%r20)
+ sync
+ stw %r20, 0(%sr2,%r20)
/* Enable interrupts */
ssm PSW_SM_I, %r0
/* Return to userspace, set no error */
@@ -858,6 +861,7 @@
22:
/* Error occurred on load or store */
/* Free lock */
+ sync
stw %r20, 0(%sr2,%r20)
ssm PSW_SM_I, %r0
ldo 1(%r0),%r28
diff --git a/arch/parisc/kernel/traps.c b/arch/parisc/kernel/traps.c
index 378df92..11c9169 100644
--- a/arch/parisc/kernel/traps.c
+++ b/arch/parisc/kernel/traps.c
@@ -826,7 +826,8 @@ void __init initialize_ivt(const void *iva)
for (i = 0; i < 8; i++)
*ivap++ = 0;
- /* Compute Checksum for HPMC handler */
+ /* Setup IVA and compute checksum for HPMC handler */
+ ivap[6] = (u32)__pa(os_hpmc);
length = os_hpmc_size;
ivap[7] = length;
diff --git a/arch/parisc/mm/init.c b/arch/parisc/mm/init.c
index e02ada3..b9db8e5 100644
--- a/arch/parisc/mm/init.c
+++ b/arch/parisc/mm/init.c
@@ -491,12 +491,8 @@ static void __init map_pages(unsigned long start_vaddr,
pte = pte_mkhuge(pte);
}
- if (address >= end_paddr) {
- if (force)
- break;
- else
- pte_val(pte) = 0;
- }
+ if (address >= end_paddr)
+ break;
set_pte(pg_table, pte);
diff --git a/arch/powerpc/boot/Makefile b/arch/powerpc/boot/Makefile
index 9d47f2e..bb69f39 100644
--- a/arch/powerpc/boot/Makefile
+++ b/arch/powerpc/boot/Makefile
@@ -92,7 +92,8 @@
libfdt := fdt.c fdt_ro.c fdt_wip.c fdt_sw.c fdt_rw.c fdt_strerror.c
libfdtheader := fdt.h libfdt.h libfdt_internal.h
-$(addprefix $(obj)/,$(libfdt) libfdt-wrapper.o simpleboot.o epapr.o opal.o): \
+$(addprefix $(obj)/,$(libfdt) libfdt-wrapper.o simpleboot.o epapr.o opal.o \
+ treeboot-akebono.o treeboot-currituck.o treeboot-iss4xx.o): \
$(addprefix $(obj)/,$(libfdtheader))
src-wlib-y := string.S crt0.S crtsavres.S stdio.c decompress.c main.c \
diff --git a/arch/powerpc/boot/crt0.S b/arch/powerpc/boot/crt0.S
index 12866cc..a3550e8f 100644
--- a/arch/powerpc/boot/crt0.S
+++ b/arch/powerpc/boot/crt0.S
@@ -15,7 +15,7 @@
RELA = 7
RELACOUNT = 0x6ffffff9
- .text
+ .data
/* A procedure descriptor used when booting this as a COFF file.
* When making COFF, this comes first in the link and we're
* linked at 0x500000.
@@ -23,6 +23,8 @@
.globl _zimage_start_opd
_zimage_start_opd:
.long 0x500000, 0, 0, 0
+ .text
+ b _zimage_start
#ifdef __powerpc64__
.balign 8
@@ -47,8 +49,10 @@
p_pstack: .long _platform_stack_top
#endif
- .weak _zimage_start
.globl _zimage_start
+ /* Clang appears to require the .weak directive to be after the symbol
+ * is defined. See https://bugs.llvm.org/show_bug.cgi?id=38921 */
+ .weak _zimage_start
_zimage_start:
.globl _zimage_start_lib
_zimage_start_lib:
diff --git a/arch/powerpc/include/asm/exception-64s.h b/arch/powerpc/include/asm/exception-64s.h
index 903e76a..e220010 100644
--- a/arch/powerpc/include/asm/exception-64s.h
+++ b/arch/powerpc/include/asm/exception-64s.h
@@ -51,6 +51,27 @@
#define EX_PPR 88 /* SMT thread status register (priority) */
#define EX_CTR 96
+#define STF_ENTRY_BARRIER_SLOT \
+ STF_ENTRY_BARRIER_FIXUP_SECTION; \
+ nop; \
+ nop; \
+ nop
+
+#define STF_EXIT_BARRIER_SLOT \
+ STF_EXIT_BARRIER_FIXUP_SECTION; \
+ nop; \
+ nop; \
+ nop; \
+ nop; \
+ nop; \
+ nop
+
+/*
+ * r10 must be free to use, r13 must be paca
+ */
+#define INTERRUPT_TO_KERNEL \
+ STF_ENTRY_BARRIER_SLOT
+
/*
* Macros for annotating the expected destination of (h)rfid
*
@@ -67,16 +88,19 @@
rfid
#define RFI_TO_USER \
+ STF_EXIT_BARRIER_SLOT; \
RFI_FLUSH_SLOT; \
rfid; \
b rfi_flush_fallback
#define RFI_TO_USER_OR_KERNEL \
+ STF_EXIT_BARRIER_SLOT; \
RFI_FLUSH_SLOT; \
rfid; \
b rfi_flush_fallback
#define RFI_TO_GUEST \
+ STF_EXIT_BARRIER_SLOT; \
RFI_FLUSH_SLOT; \
rfid; \
b rfi_flush_fallback
@@ -85,21 +109,25 @@
hrfid
#define HRFI_TO_USER \
+ STF_EXIT_BARRIER_SLOT; \
RFI_FLUSH_SLOT; \
hrfid; \
b hrfi_flush_fallback
#define HRFI_TO_USER_OR_KERNEL \
+ STF_EXIT_BARRIER_SLOT; \
RFI_FLUSH_SLOT; \
hrfid; \
b hrfi_flush_fallback
#define HRFI_TO_GUEST \
+ STF_EXIT_BARRIER_SLOT; \
RFI_FLUSH_SLOT; \
hrfid; \
b hrfi_flush_fallback
#define HRFI_TO_UNKNOWN \
+ STF_EXIT_BARRIER_SLOT; \
RFI_FLUSH_SLOT; \
hrfid; \
b hrfi_flush_fallback
@@ -225,6 +253,7 @@ END_FTR_SECTION_NESTED(ftr,ftr,943)
#define __EXCEPTION_PROLOG_1(area, extra, vec) \
OPT_SAVE_REG_TO_PACA(area+EX_PPR, r9, CPU_FTR_HAS_PPR); \
OPT_SAVE_REG_TO_PACA(area+EX_CFAR, r10, CPU_FTR_CFAR); \
+ INTERRUPT_TO_KERNEL; \
SAVE_CTR(r10, area); \
mfcr r9; \
extra(vec); \
diff --git a/arch/powerpc/include/asm/fadump.h b/arch/powerpc/include/asm/fadump.h
index 0031806..f93238a 100644
--- a/arch/powerpc/include/asm/fadump.h
+++ b/arch/powerpc/include/asm/fadump.h
@@ -190,9 +190,6 @@ struct fadump_crash_info_header {
struct cpumask online_mask;
};
-/* Crash memory ranges */
-#define INIT_CRASHMEM_RANGES (INIT_MEMBLOCK_REGIONS + 2)
-
struct fad_crash_memory_ranges {
unsigned long long base;
unsigned long long size;
diff --git a/arch/powerpc/include/asm/feature-fixups.h b/arch/powerpc/include/asm/feature-fixups.h
index 7b33234..0bf8202 100644
--- a/arch/powerpc/include/asm/feature-fixups.h
+++ b/arch/powerpc/include/asm/feature-fixups.h
@@ -189,6 +189,22 @@ void apply_feature_fixups(void);
void setup_feature_keys(void);
#endif
+#define STF_ENTRY_BARRIER_FIXUP_SECTION \
+953: \
+ .pushsection __stf_entry_barrier_fixup,"a"; \
+ .align 2; \
+954: \
+ FTR_ENTRY_OFFSET 953b-954b; \
+ .popsection;
+
+#define STF_EXIT_BARRIER_FIXUP_SECTION \
+955: \
+ .pushsection __stf_exit_barrier_fixup,"a"; \
+ .align 2; \
+956: \
+ FTR_ENTRY_OFFSET 955b-956b; \
+ .popsection;
+
#define RFI_FLUSH_FIXUP_SECTION \
951: \
.pushsection __rfi_flush_fixup,"a"; \
@@ -200,6 +216,9 @@ void setup_feature_keys(void);
#ifndef __ASSEMBLY__
+extern long stf_barrier_fallback;
+extern long __start___stf_entry_barrier_fixup, __stop___stf_entry_barrier_fixup;
+extern long __start___stf_exit_barrier_fixup, __stop___stf_exit_barrier_fixup;
extern long __start___rfi_flush_fixup, __stop___rfi_flush_fixup;
#endif
diff --git a/arch/powerpc/include/asm/futex.h b/arch/powerpc/include/asm/futex.h
index 2a9cf84..f4c7467f 100644
--- a/arch/powerpc/include/asm/futex.h
+++ b/arch/powerpc/include/asm/futex.h
@@ -31,18 +31,10 @@
: "b" (uaddr), "i" (-EFAULT), "r" (oparg) \
: "cr0", "memory")
-static inline int futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
+static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval,
+ u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (! access_ok (VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
pagefault_disable();
@@ -68,17 +60,9 @@ static inline int futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/powerpc/include/asm/hvcall.h b/arch/powerpc/include/asm/hvcall.h
index dc0996b..9d97810 100644
--- a/arch/powerpc/include/asm/hvcall.h
+++ b/arch/powerpc/include/asm/hvcall.h
@@ -313,6 +313,9 @@
#define H_CPU_CHAR_L1D_FLUSH_ORI30 (1ull << 61) // IBM bit 2
#define H_CPU_CHAR_L1D_FLUSH_TRIG2 (1ull << 60) // IBM bit 3
#define H_CPU_CHAR_L1D_THREAD_PRIV (1ull << 59) // IBM bit 4
+#define H_CPU_CHAR_BRANCH_HINTS_HONORED (1ull << 58) // IBM bit 5
+#define H_CPU_CHAR_THREAD_RECONFIG_CTRL (1ull << 57) // IBM bit 6
+#define H_CPU_CHAR_COUNT_CACHE_DISABLED (1ull << 56) // IBM bit 7
#define H_CPU_BEHAV_FAVOUR_SECURITY (1ull << 63) // IBM bit 0
#define H_CPU_BEHAV_L1D_FLUSH_PR (1ull << 62) // IBM bit 1
diff --git a/arch/powerpc/include/asm/io.h b/arch/powerpc/include/asm/io.h
index f6fda84..3b10f53 100644
--- a/arch/powerpc/include/asm/io.h
+++ b/arch/powerpc/include/asm/io.h
@@ -333,19 +333,13 @@ extern void _memcpy_toio(volatile void __iomem *dest, const void *src,
* their hooks, a bitfield is reserved for use by the platform near the
* top of MMIO addresses (not PIO, those have to cope the hard way).
*
- * This bit field is 12 bits and is at the top of the IO virtual
- * addresses PCI_IO_INDIRECT_TOKEN_MASK.
+ * The highest address in the kernel virtual space are:
*
- * The kernel virtual space is thus:
+ * d0003fffffffffff # with Hash MMU
+ * c00fffffffffffff # with Radix MMU
*
- * 0xD000000000000000 : vmalloc
- * 0xD000080000000000 : PCI PHB IO space
- * 0xD000080080000000 : ioremap
- * 0xD0000fffffffffff : end of ioremap region
- *
- * Since the top 4 bits are reserved as the region ID, we use thus
- * the next 12 bits and keep 4 bits available for the future if the
- * virtual address space is ever to be extended.
+ * The top 4 bits are reserved as the region ID on hash, leaving us 8 bits
+ * that can be used for the field.
*
* The direct IO mapping operations will then mask off those bits
* before doing the actual access, though that only happen when
@@ -357,8 +351,8 @@ extern void _memcpy_toio(volatile void __iomem *dest, const void *src,
*/
#ifdef CONFIG_PPC_INDIRECT_MMIO
-#define PCI_IO_IND_TOKEN_MASK 0x0fff000000000000ul
-#define PCI_IO_IND_TOKEN_SHIFT 48
+#define PCI_IO_IND_TOKEN_SHIFT 52
+#define PCI_IO_IND_TOKEN_MASK (0xfful << PCI_IO_IND_TOKEN_SHIFT)
#define PCI_FIX_ADDR(addr) \
((PCI_IO_ADDR)(((unsigned long)(addr)) & ~PCI_IO_IND_TOKEN_MASK))
#define PCI_GET_ADDR_TOKEN(addr) \
diff --git a/arch/powerpc/include/asm/irq_work.h b/arch/powerpc/include/asm/irq_work.h
index 744fd54..1bcc849 100644
--- a/arch/powerpc/include/asm/irq_work.h
+++ b/arch/powerpc/include/asm/irq_work.h
@@ -5,5 +5,6 @@ static inline bool arch_irq_work_has_interrupt(void)
{
return true;
}
+extern void arch_irq_work_raise(void);
#endif /* _ASM_POWERPC_IRQ_WORK_H */
diff --git a/arch/powerpc/include/asm/mpic.h b/arch/powerpc/include/asm/mpic.h
index 9869761..705f4dc 100644
--- a/arch/powerpc/include/asm/mpic.h
+++ b/arch/powerpc/include/asm/mpic.h
@@ -392,7 +392,14 @@ extern struct bus_type mpic_subsys;
#define MPIC_REGSET_TSI108 MPIC_REGSET(1) /* Tsi108/109 PIC */
/* Get the version of primary MPIC */
+#ifdef CONFIG_MPIC
extern u32 fsl_mpic_primary_get_version(void);
+#else
+static inline u32 fsl_mpic_primary_get_version(void)
+{
+ return 0;
+}
+#endif
/* Allocate the controller structure and setup the linux irq descs
* for the range if interrupts passed in. No HW initialization is
diff --git a/arch/powerpc/include/asm/security_features.h b/arch/powerpc/include/asm/security_features.h
new file mode 100644
index 0000000..44989b2
--- /dev/null
+++ b/arch/powerpc/include/asm/security_features.h
@@ -0,0 +1,85 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+/*
+ * Security related feature bit definitions.
+ *
+ * Copyright 2018, Michael Ellerman, IBM Corporation.
+ */
+
+#ifndef _ASM_POWERPC_SECURITY_FEATURES_H
+#define _ASM_POWERPC_SECURITY_FEATURES_H
+
+
+extern unsigned long powerpc_security_features;
+extern bool rfi_flush;
+
+/* These are bit flags */
+enum stf_barrier_type {
+ STF_BARRIER_NONE = 0x1,
+ STF_BARRIER_FALLBACK = 0x2,
+ STF_BARRIER_EIEIO = 0x4,
+ STF_BARRIER_SYNC_ORI = 0x8,
+};
+
+void setup_stf_barrier(void);
+void do_stf_barrier_fixups(enum stf_barrier_type types);
+
+static inline void security_ftr_set(unsigned long feature)
+{
+ powerpc_security_features |= feature;
+}
+
+static inline void security_ftr_clear(unsigned long feature)
+{
+ powerpc_security_features &= ~feature;
+}
+
+static inline bool security_ftr_enabled(unsigned long feature)
+{
+ return !!(powerpc_security_features & feature);
+}
+
+
+// Features indicating support for Spectre/Meltdown mitigations
+
+// The L1-D cache can be flushed with ori r30,r30,0
+#define SEC_FTR_L1D_FLUSH_ORI30 0x0000000000000001ull
+
+// The L1-D cache can be flushed with mtspr 882,r0 (aka SPRN_TRIG2)
+#define SEC_FTR_L1D_FLUSH_TRIG2 0x0000000000000002ull
+
+// ori r31,r31,0 acts as a speculation barrier
+#define SEC_FTR_SPEC_BAR_ORI31 0x0000000000000004ull
+
+// Speculation past bctr is disabled
+#define SEC_FTR_BCCTRL_SERIALISED 0x0000000000000008ull
+
+// Entries in L1-D are private to a SMT thread
+#define SEC_FTR_L1D_THREAD_PRIV 0x0000000000000010ull
+
+// Indirect branch prediction cache disabled
+#define SEC_FTR_COUNT_CACHE_DISABLED 0x0000000000000020ull
+
+
+// Features indicating need for Spectre/Meltdown mitigations
+
+// The L1-D cache should be flushed on MSR[HV] 1->0 transition (hypervisor to guest)
+#define SEC_FTR_L1D_FLUSH_HV 0x0000000000000040ull
+
+// The L1-D cache should be flushed on MSR[PR] 0->1 transition (kernel to userspace)
+#define SEC_FTR_L1D_FLUSH_PR 0x0000000000000080ull
+
+// A speculation barrier should be used for bounds checks (Spectre variant 1)
+#define SEC_FTR_BNDS_CHK_SPEC_BAR 0x0000000000000100ull
+
+// Firmware configuration indicates user favours security over performance
+#define SEC_FTR_FAVOUR_SECURITY 0x0000000000000200ull
+
+
+// Features enabled by default
+#define SEC_FTR_DEFAULT \
+ (SEC_FTR_L1D_FLUSH_HV | \
+ SEC_FTR_L1D_FLUSH_PR | \
+ SEC_FTR_BNDS_CHK_SPEC_BAR | \
+ SEC_FTR_FAVOUR_SECURITY)
+
+#endif /* _ASM_POWERPC_SECURITY_FEATURES_H */
diff --git a/arch/powerpc/include/asm/setup.h b/arch/powerpc/include/asm/setup.h
index 6825a67..3f160cd 100644
--- a/arch/powerpc/include/asm/setup.h
+++ b/arch/powerpc/include/asm/setup.h
@@ -48,7 +48,7 @@ enum l1d_flush_type {
L1D_FLUSH_MTTRIG = 0x8,
};
-void __init setup_rfi_flush(enum l1d_flush_type, bool enable);
+void setup_rfi_flush(enum l1d_flush_type, bool enable);
void do_rfi_flush_fixups(enum l1d_flush_type types);
#endif /* !__ASSEMBLY__ */
diff --git a/arch/powerpc/kernel/Makefile b/arch/powerpc/kernel/Makefile
index adb52d1..1388578 100644
--- a/arch/powerpc/kernel/Makefile
+++ b/arch/powerpc/kernel/Makefile
@@ -44,7 +44,7 @@
obj-$(CONFIG_VDSO32) += vdso32/
obj-$(CONFIG_HAVE_HW_BREAKPOINT) += hw_breakpoint.o
obj-$(CONFIG_PPC_BOOK3S_64) += cpu_setup_ppc970.o cpu_setup_pa6t.o
-obj-$(CONFIG_PPC_BOOK3S_64) += cpu_setup_power.o
+obj-$(CONFIG_PPC_BOOK3S_64) += cpu_setup_power.o security.o
obj-$(CONFIG_PPC_BOOK3S_64) += mce.o mce_power.o
obj-$(CONFIG_PPC_BOOK3E_64) += exceptions-64e.o idle_book3e.o
obj-$(CONFIG_PPC64) += vdso64/
diff --git a/arch/powerpc/kernel/cpu_setup_power.S b/arch/powerpc/kernel/cpu_setup_power.S
index 9e05c88..ff45d00 100644
--- a/arch/powerpc/kernel/cpu_setup_power.S
+++ b/arch/powerpc/kernel/cpu_setup_power.S
@@ -28,6 +28,7 @@
beqlr
li r0,0
mtspr SPRN_LPID,r0
+ mtspr SPRN_PCR,r0
mfspr r3,SPRN_LPCR
bl __init_LPCR
bl __init_tlb_power7
@@ -41,6 +42,7 @@
beqlr
li r0,0
mtspr SPRN_LPID,r0
+ mtspr SPRN_PCR,r0
mfspr r3,SPRN_LPCR
bl __init_LPCR
bl __init_tlb_power7
@@ -57,6 +59,7 @@
beqlr
li r0,0
mtspr SPRN_LPID,r0
+ mtspr SPRN_PCR,r0
mfspr r3,SPRN_LPCR
ori r3, r3, LPCR_PECEDH
bl __init_LPCR
@@ -78,6 +81,7 @@
beqlr
li r0,0
mtspr SPRN_LPID,r0
+ mtspr SPRN_PCR,r0
mfspr r3,SPRN_LPCR
ori r3, r3, LPCR_PECEDH
bl __init_LPCR
@@ -98,6 +102,7 @@
li r0,0
mtspr SPRN_LPID,r0
mtspr SPRN_PID,r0
+ mtspr SPRN_PCR,r0
mfspr r3,SPRN_LPCR
LOAD_REG_IMMEDIATE(r4, LPCR_PECEDH | LPCR_PECE_HVEE | LPCR_HVICE)
or r3, r3, r4
@@ -121,6 +126,7 @@
li r0,0
mtspr SPRN_LPID,r0
mtspr SPRN_PID,r0
+ mtspr SPRN_PCR,r0
mfspr r3,SPRN_LPCR
LOAD_REG_IMMEDIATE(r4, LPCR_PECEDH | LPCR_PECE_HVEE | LPCR_HVICE)
or r3, r3, r4
diff --git a/arch/powerpc/kernel/eeh.c b/arch/powerpc/kernel/eeh.c
index e5bfbf6..8336b90 100644
--- a/arch/powerpc/kernel/eeh.c
+++ b/arch/powerpc/kernel/eeh.c
@@ -169,6 +169,11 @@ static size_t eeh_dump_dev_log(struct eeh_dev *edev, char *buf, size_t len)
int n = 0, l = 0;
char buffer[128];
+ if (!pdn) {
+ pr_warn("EEH: Note: No error log for absent device.\n");
+ return 0;
+ }
+
n += scnprintf(buf+n, len-n, "%04x:%02x:%02x.%01x\n",
edev->phb->global_number, pdn->busno,
PCI_SLOT(pdn->devfn), PCI_FUNC(pdn->devfn));
diff --git a/arch/powerpc/kernel/eeh_driver.c b/arch/powerpc/kernel/eeh_driver.c
index 6ef8f0b..620e08d 100644
--- a/arch/powerpc/kernel/eeh_driver.c
+++ b/arch/powerpc/kernel/eeh_driver.c
@@ -207,18 +207,18 @@ static void *eeh_report_error(void *data, void *userdata)
if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe))
return NULL;
+
+ device_lock(&dev->dev);
dev->error_state = pci_channel_io_frozen;
driver = eeh_pcid_get(dev);
- if (!driver) return NULL;
+ if (!driver) goto out_no_dev;
eeh_disable_irq(dev);
if (!driver->err_handler ||
- !driver->err_handler->error_detected) {
- eeh_pcid_put(dev);
- return NULL;
- }
+ !driver->err_handler->error_detected)
+ goto out;
rc = driver->err_handler->error_detected(dev, pci_channel_io_frozen);
@@ -227,7 +227,10 @@ static void *eeh_report_error(void *data, void *userdata)
if (*res == PCI_ERS_RESULT_NONE) *res = rc;
edev->in_error = true;
+out:
eeh_pcid_put(dev);
+out_no_dev:
+ device_unlock(&dev->dev);
return NULL;
}
@@ -250,15 +253,14 @@ static void *eeh_report_mmio_enabled(void *data, void *userdata)
if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe))
return NULL;
+ device_lock(&dev->dev);
driver = eeh_pcid_get(dev);
- if (!driver) return NULL;
+ if (!driver) goto out_no_dev;
if (!driver->err_handler ||
!driver->err_handler->mmio_enabled ||
- (edev->mode & EEH_DEV_NO_HANDLER)) {
- eeh_pcid_put(dev);
- return NULL;
- }
+ (edev->mode & EEH_DEV_NO_HANDLER))
+ goto out;
rc = driver->err_handler->mmio_enabled(dev);
@@ -266,7 +268,10 @@ static void *eeh_report_mmio_enabled(void *data, void *userdata)
if (rc == PCI_ERS_RESULT_NEED_RESET) *res = rc;
if (*res == PCI_ERS_RESULT_NONE) *res = rc;
+out:
eeh_pcid_put(dev);
+out_no_dev:
+ device_unlock(&dev->dev);
return NULL;
}
@@ -289,20 +294,20 @@ static void *eeh_report_reset(void *data, void *userdata)
if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe))
return NULL;
+
+ device_lock(&dev->dev);
dev->error_state = pci_channel_io_normal;
driver = eeh_pcid_get(dev);
- if (!driver) return NULL;
+ if (!driver) goto out_no_dev;
eeh_enable_irq(dev);
if (!driver->err_handler ||
!driver->err_handler->slot_reset ||
(edev->mode & EEH_DEV_NO_HANDLER) ||
- (!edev->in_error)) {
- eeh_pcid_put(dev);
- return NULL;
- }
+ (!edev->in_error))
+ goto out;
rc = driver->err_handler->slot_reset(dev);
if ((*res == PCI_ERS_RESULT_NONE) ||
@@ -310,7 +315,10 @@ static void *eeh_report_reset(void *data, void *userdata)
if (*res == PCI_ERS_RESULT_DISCONNECT &&
rc == PCI_ERS_RESULT_NEED_RESET) *res = rc;
+out:
eeh_pcid_put(dev);
+out_no_dev:
+ device_unlock(&dev->dev);
return NULL;
}
@@ -361,10 +369,12 @@ static void *eeh_report_resume(void *data, void *userdata)
if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe))
return NULL;
+
+ device_lock(&dev->dev);
dev->error_state = pci_channel_io_normal;
driver = eeh_pcid_get(dev);
- if (!driver) return NULL;
+ if (!driver) goto out_no_dev;
was_in_error = edev->in_error;
edev->in_error = false;
@@ -374,13 +384,15 @@ static void *eeh_report_resume(void *data, void *userdata)
!driver->err_handler->resume ||
(edev->mode & EEH_DEV_NO_HANDLER) || !was_in_error) {
edev->mode &= ~EEH_DEV_NO_HANDLER;
- eeh_pcid_put(dev);
- return NULL;
+ goto out;
}
driver->err_handler->resume(dev);
+out:
eeh_pcid_put(dev);
+out_no_dev:
+ device_unlock(&dev->dev);
return NULL;
}
@@ -400,22 +412,25 @@ static void *eeh_report_failure(void *data, void *userdata)
if (!dev || eeh_dev_removed(edev) || eeh_pe_passed(edev->pe))
return NULL;
+
+ device_lock(&dev->dev);
dev->error_state = pci_channel_io_perm_failure;
driver = eeh_pcid_get(dev);
- if (!driver) return NULL;
+ if (!driver) goto out_no_dev;
eeh_disable_irq(dev);
if (!driver->err_handler ||
- !driver->err_handler->error_detected) {
- eeh_pcid_put(dev);
- return NULL;
- }
+ !driver->err_handler->error_detected)
+ goto out;
driver->err_handler->error_detected(dev, pci_channel_io_perm_failure);
+out:
eeh_pcid_put(dev);
+out_no_dev:
+ device_unlock(&dev->dev);
return NULL;
}
@@ -435,9 +450,11 @@ static void *eeh_add_virt_device(void *data, void *userdata)
driver = eeh_pcid_get(dev);
if (driver) {
- eeh_pcid_put(dev);
- if (driver->err_handler)
+ if (driver->err_handler) {
+ eeh_pcid_put(dev);
return NULL;
+ }
+ eeh_pcid_put(dev);
}
#ifdef CONFIG_PPC_POWERNV
@@ -474,17 +491,19 @@ static void *eeh_rmv_device(void *data, void *userdata)
if (eeh_dev_removed(edev))
return NULL;
- driver = eeh_pcid_get(dev);
- if (driver) {
- eeh_pcid_put(dev);
- if (removed &&
- eeh_pe_passed(edev->pe))
+ if (removed) {
+ if (eeh_pe_passed(edev->pe))
return NULL;
- if (removed &&
- driver->err_handler &&
- driver->err_handler->error_detected &&
- driver->err_handler->slot_reset)
- return NULL;
+ driver = eeh_pcid_get(dev);
+ if (driver) {
+ if (driver->err_handler &&
+ driver->err_handler->error_detected &&
+ driver->err_handler->slot_reset) {
+ eeh_pcid_put(dev);
+ return NULL;
+ }
+ eeh_pcid_put(dev);
+ }
}
/* Remove it from PCI subsystem */
diff --git a/arch/powerpc/kernel/entry_64.S b/arch/powerpc/kernel/entry_64.S
index 2dc52e6..e24ae0f 100644
--- a/arch/powerpc/kernel/entry_64.S
+++ b/arch/powerpc/kernel/entry_64.S
@@ -586,6 +586,7 @@
* actually hit this code path.
*/
+ isync
slbie r6
slbie r6 /* Workaround POWER5 < DD2.1 issue */
slbmte r7,r0
diff --git a/arch/powerpc/kernel/exceptions-64s.S b/arch/powerpc/kernel/exceptions-64s.S
index 94b5dfb..d50cc9b 100644
--- a/arch/powerpc/kernel/exceptions-64s.S
+++ b/arch/powerpc/kernel/exceptions-64s.S
@@ -846,7 +846,7 @@
#endif
-EXC_REAL_MASKABLE(decrementer, 0x900, 0x980)
+EXC_REAL_OOL_MASKABLE(decrementer, 0x900, 0x980)
EXC_VIRT_MASKABLE(decrementer, 0x4900, 0x4980, 0x900)
TRAMP_KVM(PACA_EXGEN, 0x900)
EXC_COMMON_ASYNC(decrementer_common, 0x900, timer_interrupt)
@@ -884,6 +884,7 @@
END_FTR_SECTION_IFSET(CPU_FTR_REAL_LE) \
mr r9,r13 ; \
GET_PACA(r13) ; \
+ INTERRUPT_TO_KERNEL ; \
mfspr r11,SPRN_SRR0 ; \
0:
@@ -1353,6 +1354,19 @@
##_H##RFI_TO_KERNEL; \
b .
+TRAMP_REAL_BEGIN(stf_barrier_fallback)
+ std r9,PACA_EXRFI+EX_R9(r13)
+ std r10,PACA_EXRFI+EX_R10(r13)
+ sync
+ ld r9,PACA_EXRFI+EX_R9(r13)
+ ld r10,PACA_EXRFI+EX_R10(r13)
+ ori 31,31,0
+ .rept 14
+ b 1f
+1:
+ .endr
+ blr
+
/*
* Real mode exceptions actually use this too, but alternate
* instruction code patches (which end up in the common .text area)
diff --git a/arch/powerpc/kernel/fadump.c b/arch/powerpc/kernel/fadump.c
index 8f0c7c5..0292504 100644
--- a/arch/powerpc/kernel/fadump.c
+++ b/arch/powerpc/kernel/fadump.c
@@ -35,6 +35,7 @@
#include <linux/crash_dump.h>
#include <linux/kobject.h>
#include <linux/sysfs.h>
+#include <linux/slab.h>
#include <asm/page.h>
#include <asm/prom.h>
@@ -48,8 +49,10 @@ static struct fadump_mem_struct fdm;
static const struct fadump_mem_struct *fdm_active;
static DEFINE_MUTEX(fadump_mutex);
-struct fad_crash_memory_ranges crash_memory_ranges[INIT_CRASHMEM_RANGES];
+struct fad_crash_memory_ranges *crash_memory_ranges;
+int crash_memory_ranges_size;
int crash_mem_ranges;
+int max_crash_mem_ranges;
/* Scan the Firmware Assisted dump configuration details. */
int __init early_init_dt_scan_fw_dump(unsigned long node,
@@ -362,9 +365,9 @@ static int __init early_fadump_reserve_mem(char *p)
}
early_param("fadump_reserve_mem", early_fadump_reserve_mem);
-static void register_fw_dump(struct fadump_mem_struct *fdm)
+static int register_fw_dump(struct fadump_mem_struct *fdm)
{
- int rc;
+ int rc, err;
unsigned int wait_time;
pr_debug("Registering for firmware-assisted kernel dump...\n");
@@ -381,7 +384,11 @@ static void register_fw_dump(struct fadump_mem_struct *fdm)
} while (wait_time);
+ err = -EIO;
switch (rc) {
+ default:
+ pr_err("Failed to register. Unknown Error(%d).\n", rc);
+ break;
case -1:
printk(KERN_ERR "Failed to register firmware-assisted kernel"
" dump. Hardware Error(%d).\n", rc);
@@ -389,18 +396,22 @@ static void register_fw_dump(struct fadump_mem_struct *fdm)
case -3:
printk(KERN_ERR "Failed to register firmware-assisted kernel"
" dump. Parameter Error(%d).\n", rc);
+ err = -EINVAL;
break;
case -9:
printk(KERN_ERR "firmware-assisted kernel dump is already "
" registered.");
fw_dump.dump_registered = 1;
+ err = -EEXIST;
break;
case 0:
printk(KERN_INFO "firmware-assisted kernel dump registration"
" is successful\n");
fw_dump.dump_registered = 1;
+ err = 0;
break;
}
+ return err;
}
void crash_fadump(struct pt_regs *regs, const char *str)
@@ -731,38 +742,88 @@ static int __init process_fadump(const struct fadump_mem_struct *fdm_active)
return 0;
}
-static inline void fadump_add_crash_memory(unsigned long long base,
- unsigned long long end)
+static void free_crash_memory_ranges(void)
+{
+ kfree(crash_memory_ranges);
+ crash_memory_ranges = NULL;
+ crash_memory_ranges_size = 0;
+ max_crash_mem_ranges = 0;
+}
+
+/*
+ * Allocate or reallocate crash memory ranges array in incremental units
+ * of PAGE_SIZE.
+ */
+static int allocate_crash_memory_ranges(void)
+{
+ struct fad_crash_memory_ranges *new_array;
+ u64 new_size;
+
+ new_size = crash_memory_ranges_size + PAGE_SIZE;
+ pr_debug("Allocating %llu bytes of memory for crash memory ranges\n",
+ new_size);
+
+ new_array = krealloc(crash_memory_ranges, new_size, GFP_KERNEL);
+ if (new_array == NULL) {
+ pr_err("Insufficient memory for setting up crash memory ranges\n");
+ free_crash_memory_ranges();
+ return -ENOMEM;
+ }
+
+ crash_memory_ranges = new_array;
+ crash_memory_ranges_size = new_size;
+ max_crash_mem_ranges = (new_size /
+ sizeof(struct fad_crash_memory_ranges));
+ return 0;
+}
+
+static inline int fadump_add_crash_memory(unsigned long long base,
+ unsigned long long end)
{
if (base == end)
- return;
+ return 0;
+
+ if (crash_mem_ranges == max_crash_mem_ranges) {
+ int ret;
+
+ ret = allocate_crash_memory_ranges();
+ if (ret)
+ return ret;
+ }
pr_debug("crash_memory_range[%d] [%#016llx-%#016llx], %#llx bytes\n",
crash_mem_ranges, base, end - 1, (end - base));
crash_memory_ranges[crash_mem_ranges].base = base;
crash_memory_ranges[crash_mem_ranges].size = end - base;
crash_mem_ranges++;
+ return 0;
}
-static void fadump_exclude_reserved_area(unsigned long long start,
+static int fadump_exclude_reserved_area(unsigned long long start,
unsigned long long end)
{
unsigned long long ra_start, ra_end;
+ int ret = 0;
ra_start = fw_dump.reserve_dump_area_start;
ra_end = ra_start + fw_dump.reserve_dump_area_size;
if ((ra_start < end) && (ra_end > start)) {
if ((start < ra_start) && (end > ra_end)) {
- fadump_add_crash_memory(start, ra_start);
- fadump_add_crash_memory(ra_end, end);
+ ret = fadump_add_crash_memory(start, ra_start);
+ if (ret)
+ return ret;
+
+ ret = fadump_add_crash_memory(ra_end, end);
} else if (start < ra_start) {
- fadump_add_crash_memory(start, ra_start);
+ ret = fadump_add_crash_memory(start, ra_start);
} else if (ra_end < end) {
- fadump_add_crash_memory(ra_end, end);
+ ret = fadump_add_crash_memory(ra_end, end);
}
} else
- fadump_add_crash_memory(start, end);
+ ret = fadump_add_crash_memory(start, end);
+
+ return ret;
}
static int fadump_init_elfcore_header(char *bufp)
@@ -802,10 +863,11 @@ static int fadump_init_elfcore_header(char *bufp)
* Traverse through memblock structure and setup crash memory ranges. These
* ranges will be used create PT_LOAD program headers in elfcore header.
*/
-static void fadump_setup_crash_memory_ranges(void)
+static int fadump_setup_crash_memory_ranges(void)
{
struct memblock_region *reg;
unsigned long long start, end;
+ int ret;
pr_debug("Setup crash memory ranges.\n");
crash_mem_ranges = 0;
@@ -816,7 +878,9 @@ static void fadump_setup_crash_memory_ranges(void)
* specified during fadump registration. We need to create a separate
* program header for this chunk with the correct offset.
*/
- fadump_add_crash_memory(RMA_START, fw_dump.boot_memory_size);
+ ret = fadump_add_crash_memory(RMA_START, fw_dump.boot_memory_size);
+ if (ret)
+ return ret;
for_each_memblock(memory, reg) {
start = (unsigned long long)reg->base;
@@ -825,8 +889,12 @@ static void fadump_setup_crash_memory_ranges(void)
start = fw_dump.boot_memory_size;
/* add this range excluding the reserved dump area. */
- fadump_exclude_reserved_area(start, end);
+ ret = fadump_exclude_reserved_area(start, end);
+ if (ret)
+ return ret;
}
+
+ return 0;
}
/*
@@ -946,19 +1014,22 @@ static unsigned long init_fadump_header(unsigned long addr)
return addr;
}
-static void register_fadump(void)
+static int register_fadump(void)
{
unsigned long addr;
void *vaddr;
+ int ret;
/*
* If no memory is reserved then we can not register for firmware-
* assisted dump.
*/
if (!fw_dump.reserve_dump_area_size)
- return;
+ return -ENODEV;
- fadump_setup_crash_memory_ranges();
+ ret = fadump_setup_crash_memory_ranges();
+ if (ret)
+ return ret;
addr = be64_to_cpu(fdm.rmr_region.destination_address) + be64_to_cpu(fdm.rmr_region.source_len);
/* Initialize fadump crash info header. */
@@ -969,7 +1040,7 @@ static void register_fadump(void)
fadump_create_elfcore_headers(vaddr);
/* register the future kernel dump with firmware. */
- register_fw_dump(&fdm);
+ return register_fw_dump(&fdm);
}
static int fadump_unregister_dump(struct fadump_mem_struct *fdm)
@@ -1033,6 +1104,10 @@ void fadump_cleanup(void)
init_fadump_mem_struct(&fdm,
be64_to_cpu(fdm_active->cpu_state_data.destination_address));
fadump_invalidate_dump(&fdm);
+ } else if (fw_dump.dump_registered) {
+ /* Un-register Firmware-assisted dump if it was registered. */
+ fadump_unregister_dump(&fdm);
+ free_crash_memory_ranges();
}
}
@@ -1151,7 +1226,6 @@ static ssize_t fadump_register_store(struct kobject *kobj,
switch (buf[0]) {
case '0':
if (fw_dump.dump_registered == 0) {
- ret = -EINVAL;
goto unlock_out;
}
/* Un-register Firmware-assisted dump */
@@ -1159,11 +1233,11 @@ static ssize_t fadump_register_store(struct kobject *kobj,
break;
case '1':
if (fw_dump.dump_registered == 1) {
- ret = -EINVAL;
+ ret = -EEXIST;
goto unlock_out;
}
/* Register Firmware-assisted dump */
- register_fadump();
+ ret = register_fadump();
break;
default:
ret = -EINVAL;
diff --git a/arch/powerpc/kernel/head_8xx.S b/arch/powerpc/kernel/head_8xx.S
index fb133a1..2274be5 100644
--- a/arch/powerpc/kernel/head_8xx.S
+++ b/arch/powerpc/kernel/head_8xx.S
@@ -769,7 +769,7 @@
tovirt(r6,r6)
lis r5, abatron_pteptrs@h
ori r5, r5, abatron_pteptrs@l
- stw r5, 0xf0(r0) /* Must match your Abatron config file */
+ stw r5, 0xf0(0) /* Must match your Abatron config file */
tophys(r5,r5)
stw r6, 0(r5)
diff --git a/arch/powerpc/kernel/hw_breakpoint.c b/arch/powerpc/kernel/hw_breakpoint.c
index 469d86d..532c585 100644
--- a/arch/powerpc/kernel/hw_breakpoint.c
+++ b/arch/powerpc/kernel/hw_breakpoint.c
@@ -175,8 +175,8 @@ int arch_validate_hwbkpt_settings(struct perf_event *bp)
if (cpu_has_feature(CPU_FTR_DAWR)) {
length_max = 512 ; /* 64 doublewords */
/* DAWR region can't cross 512 boundary */
- if ((bp->attr.bp_addr >> 10) !=
- ((bp->attr.bp_addr + bp->attr.bp_len - 1) >> 10))
+ if ((bp->attr.bp_addr >> 9) !=
+ ((bp->attr.bp_addr + bp->attr.bp_len - 1) >> 9))
return -EINVAL;
}
if (info->len >
diff --git a/arch/powerpc/kernel/machine_kexec.c b/arch/powerpc/kernel/machine_kexec.c
index 2694d07..9dafd7a 100644
--- a/arch/powerpc/kernel/machine_kexec.c
+++ b/arch/powerpc/kernel/machine_kexec.c
@@ -186,7 +186,12 @@ void __init reserve_crashkernel(void)
(unsigned long)(crashk_res.start >> 20),
(unsigned long)(memblock_phys_mem_size() >> 20));
- memblock_reserve(crashk_res.start, crash_size);
+ if (!memblock_is_region_memory(crashk_res.start, crash_size) ||
+ memblock_reserve(crashk_res.start, crash_size)) {
+ pr_err("Failed to reserve memory for crashkernel!\n");
+ crashk_res.start = crashk_res.end = 0;
+ return;
+ }
}
int overlaps_crashkernel(unsigned long start, unsigned long size)
diff --git a/arch/powerpc/kernel/msi.c b/arch/powerpc/kernel/msi.c
index dab616a..f219765 100644
--- a/arch/powerpc/kernel/msi.c
+++ b/arch/powerpc/kernel/msi.c
@@ -34,5 +34,10 @@ void arch_teardown_msi_irqs(struct pci_dev *dev)
{
struct pci_controller *phb = pci_bus_to_host(dev->bus);
- phb->controller_ops.teardown_msi_irqs(dev);
+ /*
+ * We can be called even when arch_setup_msi_irqs() returns -ENOSYS,
+ * so check the pointer again.
+ */
+ if (phb->controller_ops.teardown_msi_irqs)
+ phb->controller_ops.teardown_msi_irqs(dev);
}
diff --git a/arch/powerpc/kernel/pci_32.c b/arch/powerpc/kernel/pci_32.c
index 678f87a..97b02b8 100644
--- a/arch/powerpc/kernel/pci_32.c
+++ b/arch/powerpc/kernel/pci_32.c
@@ -11,6 +11,7 @@
#include <linux/sched.h>
#include <linux/errno.h>
#include <linux/bootmem.h>
+#include <linux/syscalls.h>
#include <linux/irq.h>
#include <linux/list.h>
#include <linux/of.h>
diff --git a/arch/powerpc/kernel/ptrace.c b/arch/powerpc/kernel/ptrace.c
index d973708..adfa63e 100644
--- a/arch/powerpc/kernel/ptrace.c
+++ b/arch/powerpc/kernel/ptrace.c
@@ -2380,6 +2380,7 @@ static int ptrace_set_debugreg(struct task_struct *task, unsigned long addr,
/* Create a new breakpoint request if one doesn't exist already */
hw_breakpoint_init(&attr);
attr.bp_addr = hw_brk.address;
+ attr.bp_len = 8;
arch_bp_generic_fields(hw_brk.type,
&attr.bp_type);
diff --git a/arch/powerpc/kernel/security.c b/arch/powerpc/kernel/security.c
new file mode 100644
index 0000000..2277df8
--- /dev/null
+++ b/arch/powerpc/kernel/security.c
@@ -0,0 +1,237 @@
+// SPDX-License-Identifier: GPL-2.0+
+//
+// Security related flags and so on.
+//
+// Copyright 2018, Michael Ellerman, IBM Corporation.
+
+#include <linux/kernel.h>
+#include <linux/debugfs.h>
+#include <linux/device.h>
+#include <linux/seq_buf.h>
+
+#include <asm/security_features.h>
+
+
+unsigned long powerpc_security_features __read_mostly = SEC_FTR_DEFAULT;
+
+ssize_t cpu_show_meltdown(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ bool thread_priv;
+
+ thread_priv = security_ftr_enabled(SEC_FTR_L1D_THREAD_PRIV);
+
+ if (rfi_flush || thread_priv) {
+ struct seq_buf s;
+ seq_buf_init(&s, buf, PAGE_SIZE - 1);
+
+ seq_buf_printf(&s, "Mitigation: ");
+
+ if (rfi_flush)
+ seq_buf_printf(&s, "RFI Flush");
+
+ if (rfi_flush && thread_priv)
+ seq_buf_printf(&s, ", ");
+
+ if (thread_priv)
+ seq_buf_printf(&s, "L1D private per thread");
+
+ seq_buf_printf(&s, "\n");
+
+ return s.len;
+ }
+
+ if (!security_ftr_enabled(SEC_FTR_L1D_FLUSH_HV) &&
+ !security_ftr_enabled(SEC_FTR_L1D_FLUSH_PR))
+ return sprintf(buf, "Not affected\n");
+
+ return sprintf(buf, "Vulnerable\n");
+}
+
+ssize_t cpu_show_spectre_v1(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ if (!security_ftr_enabled(SEC_FTR_BNDS_CHK_SPEC_BAR))
+ return sprintf(buf, "Not affected\n");
+
+ return sprintf(buf, "Vulnerable\n");
+}
+
+ssize_t cpu_show_spectre_v2(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ bool bcs, ccd, ori;
+ struct seq_buf s;
+
+ seq_buf_init(&s, buf, PAGE_SIZE - 1);
+
+ bcs = security_ftr_enabled(SEC_FTR_BCCTRL_SERIALISED);
+ ccd = security_ftr_enabled(SEC_FTR_COUNT_CACHE_DISABLED);
+ ori = security_ftr_enabled(SEC_FTR_SPEC_BAR_ORI31);
+
+ if (bcs || ccd) {
+ seq_buf_printf(&s, "Mitigation: ");
+
+ if (bcs)
+ seq_buf_printf(&s, "Indirect branch serialisation (kernel only)");
+
+ if (bcs && ccd)
+ seq_buf_printf(&s, ", ");
+
+ if (ccd)
+ seq_buf_printf(&s, "Indirect branch cache disabled");
+ } else
+ seq_buf_printf(&s, "Vulnerable");
+
+ if (ori)
+ seq_buf_printf(&s, ", ori31 speculation barrier enabled");
+
+ seq_buf_printf(&s, "\n");
+
+ return s.len;
+}
+
+/*
+ * Store-forwarding barrier support.
+ */
+
+static enum stf_barrier_type stf_enabled_flush_types;
+static bool no_stf_barrier;
+bool stf_barrier;
+
+static int __init handle_no_stf_barrier(char *p)
+{
+ pr_info("stf-barrier: disabled on command line.");
+ no_stf_barrier = true;
+ return 0;
+}
+
+early_param("no_stf_barrier", handle_no_stf_barrier);
+
+/* This is the generic flag used by other architectures */
+static int __init handle_ssbd(char *p)
+{
+ if (!p || strncmp(p, "auto", 5) == 0 || strncmp(p, "on", 2) == 0 ) {
+ /* Until firmware tells us, we have the barrier with auto */
+ return 0;
+ } else if (strncmp(p, "off", 3) == 0) {
+ handle_no_stf_barrier(NULL);
+ return 0;
+ } else
+ return 1;
+
+ return 0;
+}
+early_param("spec_store_bypass_disable", handle_ssbd);
+
+/* This is the generic flag used by other architectures */
+static int __init handle_no_ssbd(char *p)
+{
+ handle_no_stf_barrier(NULL);
+ return 0;
+}
+early_param("nospec_store_bypass_disable", handle_no_ssbd);
+
+static void stf_barrier_enable(bool enable)
+{
+ if (enable)
+ do_stf_barrier_fixups(stf_enabled_flush_types);
+ else
+ do_stf_barrier_fixups(STF_BARRIER_NONE);
+
+ stf_barrier = enable;
+}
+
+void setup_stf_barrier(void)
+{
+ enum stf_barrier_type type;
+ bool enable, hv;
+
+ hv = cpu_has_feature(CPU_FTR_HVMODE);
+
+ /* Default to fallback in case fw-features are not available */
+ if (cpu_has_feature(CPU_FTR_ARCH_300))
+ type = STF_BARRIER_EIEIO;
+ else if (cpu_has_feature(CPU_FTR_ARCH_207S))
+ type = STF_BARRIER_SYNC_ORI;
+ else if (cpu_has_feature(CPU_FTR_ARCH_206))
+ type = STF_BARRIER_FALLBACK;
+ else
+ type = STF_BARRIER_NONE;
+
+ enable = security_ftr_enabled(SEC_FTR_FAVOUR_SECURITY) &&
+ (security_ftr_enabled(SEC_FTR_L1D_FLUSH_PR) ||
+ (security_ftr_enabled(SEC_FTR_L1D_FLUSH_HV) && hv));
+
+ if (type == STF_BARRIER_FALLBACK) {
+ pr_info("stf-barrier: fallback barrier available\n");
+ } else if (type == STF_BARRIER_SYNC_ORI) {
+ pr_info("stf-barrier: hwsync barrier available\n");
+ } else if (type == STF_BARRIER_EIEIO) {
+ pr_info("stf-barrier: eieio barrier available\n");
+ }
+
+ stf_enabled_flush_types = type;
+
+ if (!no_stf_barrier)
+ stf_barrier_enable(enable);
+}
+
+ssize_t cpu_show_spec_store_bypass(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ if (stf_barrier && stf_enabled_flush_types != STF_BARRIER_NONE) {
+ const char *type;
+ switch (stf_enabled_flush_types) {
+ case STF_BARRIER_EIEIO:
+ type = "eieio";
+ break;
+ case STF_BARRIER_SYNC_ORI:
+ type = "hwsync";
+ break;
+ case STF_BARRIER_FALLBACK:
+ type = "fallback";
+ break;
+ default:
+ type = "unknown";
+ }
+ return sprintf(buf, "Mitigation: Kernel entry/exit barrier (%s)\n", type);
+ }
+
+ if (!security_ftr_enabled(SEC_FTR_L1D_FLUSH_HV) &&
+ !security_ftr_enabled(SEC_FTR_L1D_FLUSH_PR))
+ return sprintf(buf, "Not affected\n");
+
+ return sprintf(buf, "Vulnerable\n");
+}
+
+#ifdef CONFIG_DEBUG_FS
+static int stf_barrier_set(void *data, u64 val)
+{
+ bool enable;
+
+ if (val == 1)
+ enable = true;
+ else if (val == 0)
+ enable = false;
+ else
+ return -EINVAL;
+
+ /* Only do anything if we're changing state */
+ if (enable != stf_barrier)
+ stf_barrier_enable(enable);
+
+ return 0;
+}
+
+static int stf_barrier_get(void *data, u64 *val)
+{
+ *val = stf_barrier ? 1 : 0;
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fops_stf_barrier, stf_barrier_get, stf_barrier_set, "%llu\n");
+
+static __init int stf_barrier_debugfs_init(void)
+{
+ debugfs_create_file("stf_barrier", 0600, powerpc_debugfs_root, NULL, &fops_stf_barrier);
+ return 0;
+}
+device_initcall(stf_barrier_debugfs_init);
+#endif /* CONFIG_DEBUG_FS */
diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c
index f516ac5..bf0f712 100644
--- a/arch/powerpc/kernel/setup-common.c
+++ b/arch/powerpc/kernel/setup-common.c
@@ -228,14 +228,6 @@ static int show_cpuinfo(struct seq_file *m, void *v)
unsigned short maj;
unsigned short min;
- /* We only show online cpus: disable preempt (overzealous, I
- * knew) to prevent cpu going down. */
- preempt_disable();
- if (!cpu_online(cpu_id)) {
- preempt_enable();
- return 0;
- }
-
#ifdef CONFIG_SMP
pvr = per_cpu(cpu_pvr, cpu_id);
#else
@@ -340,9 +332,6 @@ static int show_cpuinfo(struct seq_file *m, void *v)
#ifdef CONFIG_SMP
seq_printf(m, "\n");
#endif
-
- preempt_enable();
-
/* If this is the last cpu, print the summary */
if (cpumask_next(cpu_id, cpu_online_mask) >= nr_cpu_ids)
show_cpuinfo_summary(m);
diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
index 5243501..fdba106 100644
--- a/arch/powerpc/kernel/setup_64.c
+++ b/arch/powerpc/kernel/setup_64.c
@@ -679,6 +679,7 @@ static int __init disable_hardlockup_detector(void)
return 0;
}
early_initcall(disable_hardlockup_detector);
+#endif /* CONFIG_HARDLOCKUP_DETECTOR */
#ifdef CONFIG_PPC_BOOK3S_64
static enum l1d_flush_type enabled_flush_types;
@@ -716,9 +717,6 @@ static void do_nothing(void *unused)
void rfi_flush_enable(bool enable)
{
- if (rfi_flush == enable)
- return;
-
if (enable) {
do_rfi_flush_fixups(enabled_flush_types);
on_each_cpu(do_nothing, NULL, 1);
@@ -728,11 +726,15 @@ void rfi_flush_enable(bool enable)
rfi_flush = enable;
}
-static void init_fallback_flush(void)
+static void __ref init_fallback_flush(void)
{
u64 l1d_size, limit;
int cpu;
+ /* Only allocate the fallback flush area once (at boot time). */
+ if (l1d_flush_fallback_area)
+ return;
+
l1d_size = ppc64_caches.dsize;
limit = min(safe_stack_limit(), ppc64_rma_size);
@@ -750,18 +752,18 @@ static void init_fallback_flush(void)
}
}
-void __init setup_rfi_flush(enum l1d_flush_type types, bool enable)
+void setup_rfi_flush(enum l1d_flush_type types, bool enable)
{
if (types & L1D_FLUSH_FALLBACK) {
- pr_info("rfi-flush: Using fallback displacement flush\n");
+ pr_info("rfi-flush: fallback displacement flush available\n");
init_fallback_flush();
}
if (types & L1D_FLUSH_ORI)
- pr_info("rfi-flush: Using ori type flush\n");
+ pr_info("rfi-flush: ori type flush available\n");
if (types & L1D_FLUSH_MTTRIG)
- pr_info("rfi-flush: Using mttrig type flush\n");
+ pr_info("rfi-flush: mttrig type flush available\n");
enabled_flush_types = types;
@@ -772,13 +774,19 @@ void __init setup_rfi_flush(enum l1d_flush_type types, bool enable)
#ifdef CONFIG_DEBUG_FS
static int rfi_flush_set(void *data, u64 val)
{
+ bool enable;
+
if (val == 1)
- rfi_flush_enable(true);
+ enable = true;
else if (val == 0)
- rfi_flush_enable(false);
+ enable = false;
else
return -EINVAL;
+ /* Only do anything if we're changing state */
+ if (enable != rfi_flush)
+ rfi_flush_enable(enable);
+
return 0;
}
@@ -797,13 +805,4 @@ static __init int rfi_flush_debugfs_init(void)
}
device_initcall(rfi_flush_debugfs_init);
#endif
-
-ssize_t cpu_show_meltdown(struct device *dev, struct device_attribute *attr, char *buf)
-{
- if (rfi_flush)
- return sprintf(buf, "Mitigation: RFI Flush\n");
-
- return sprintf(buf, "Vulnerable\n");
-}
#endif /* CONFIG_PPC_BOOK3S_64 */
-#endif
diff --git a/arch/powerpc/kernel/signal_32.c b/arch/powerpc/kernel/signal_32.c
index 27aa913..2bfa5a7 100644
--- a/arch/powerpc/kernel/signal_32.c
+++ b/arch/powerpc/kernel/signal_32.c
@@ -866,7 +866,23 @@ static long restore_tm_user_regs(struct pt_regs *regs,
/* If TM bits are set to the reserved value, it's an invalid context */
if (MSR_TM_RESV(msr_hi))
return 1;
- /* Pull in the MSR TM bits from the user context */
+
+ /*
+ * Disabling preemption, since it is unsafe to be preempted
+ * with MSR[TS] set without recheckpointing.
+ */
+ preempt_disable();
+
+ /*
+ * CAUTION:
+ * After regs->MSR[TS] being updated, make sure that get_user(),
+ * put_user() or similar functions are *not* called. These
+ * functions can generate page faults which will cause the process
+ * to be de-scheduled with MSR[TS] set but without calling
+ * tm_recheckpoint(). This can cause a bug.
+ *
+ * Pull in the MSR TM bits from the user context
+ */
regs->msr = (regs->msr & ~MSR_TS_MASK) | (msr_hi & MSR_TS_MASK);
/* Now, recheckpoint. This loads up all of the checkpointed (older)
* registers, including FP and V[S]Rs. After recheckpointing, the
@@ -891,6 +907,8 @@ static long restore_tm_user_regs(struct pt_regs *regs,
}
#endif
+ preempt_enable();
+
return 0;
}
#endif
diff --git a/arch/powerpc/kernel/signal_64.c b/arch/powerpc/kernel/signal_64.c
index 04e9225..d929afa 100644
--- a/arch/powerpc/kernel/signal_64.c
+++ b/arch/powerpc/kernel/signal_64.c
@@ -452,20 +452,6 @@ static long restore_tm_sigcontexts(struct task_struct *tsk,
if (MSR_TM_RESV(msr))
return -EINVAL;
- /* pull in MSR TS bits from user context */
- regs->msr = (regs->msr & ~MSR_TS_MASK) | (msr & MSR_TS_MASK);
-
- /*
- * Ensure that TM is enabled in regs->msr before we leave the signal
- * handler. It could be the case that (a) user disabled the TM bit
- * through the manipulation of the MSR bits in uc_mcontext or (b) the
- * TM bit was disabled because a sufficient number of context switches
- * happened whilst in the signal handler and load_tm overflowed,
- * disabling the TM bit. In either case we can end up with an illegal
- * TM state leading to a TM Bad Thing when we return to userspace.
- */
- regs->msr |= MSR_TM;
-
/* pull in MSR LE from user context */
regs->msr = (regs->msr & ~MSR_LE) | (msr & MSR_LE);
@@ -557,6 +543,34 @@ static long restore_tm_sigcontexts(struct task_struct *tsk,
tm_enable();
/* Make sure the transaction is marked as failed */
tsk->thread.tm_texasr |= TEXASR_FS;
+
+ /*
+ * Disabling preemption, since it is unsafe to be preempted
+ * with MSR[TS] set without recheckpointing.
+ */
+ preempt_disable();
+
+ /* pull in MSR TS bits from user context */
+ regs->msr = (regs->msr & ~MSR_TS_MASK) | (msr & MSR_TS_MASK);
+
+ /*
+ * Ensure that TM is enabled in regs->msr before we leave the signal
+ * handler. It could be the case that (a) user disabled the TM bit
+ * through the manipulation of the MSR bits in uc_mcontext or (b) the
+ * TM bit was disabled because a sufficient number of context switches
+ * happened whilst in the signal handler and load_tm overflowed,
+ * disabling the TM bit. In either case we can end up with an illegal
+ * TM state leading to a TM Bad Thing when we return to userspace.
+ *
+ * CAUTION:
+ * After regs->MSR[TS] being updated, make sure that get_user(),
+ * put_user() or similar functions are *not* called. These
+ * functions can generate page faults which will cause the process
+ * to be de-scheduled with MSR[TS] set but without calling
+ * tm_recheckpoint(). This can cause a bug.
+ */
+ regs->msr |= MSR_TM;
+
/* This loads the checkpointed FP/VEC state, if used */
tm_recheckpoint(&tsk->thread, msr);
@@ -570,6 +584,8 @@ static long restore_tm_sigcontexts(struct task_struct *tsk,
regs->msr |= MSR_VEC;
}
+ preempt_enable();
+
return err;
}
#endif
diff --git a/arch/powerpc/kernel/tm.S b/arch/powerpc/kernel/tm.S
index 3a2d041..f59b738 100644
--- a/arch/powerpc/kernel/tm.S
+++ b/arch/powerpc/kernel/tm.S
@@ -166,13 +166,27 @@
std r1, PACATMSCRATCH(r13)
ld r1, PACAR1(r13)
- /* Store the PPR in r11 and reset to decent value */
std r11, GPR11(r1) /* Temporary stash */
+ /*
+ * Move the saved user r1 to the kernel stack in case PACATMSCRATCH is
+ * clobbered by an exception once we turn on MSR_RI below.
+ */
+ ld r11, PACATMSCRATCH(r13)
+ std r11, GPR1(r1)
+
+ /*
+ * Store r13 away so we can free up the scratch SPR for the SLB fault
+ * handler (needed once we start accessing the thread_struct).
+ */
+ GET_SCRATCH0(r11)
+ std r11, GPR13(r1)
+
/* Reset MSR RI so we can take SLB faults again */
li r11, MSR_RI
mtmsrd r11, 1
+ /* Store the PPR in r11 and reset to decent value */
mfspr r11, SPRN_PPR
HMT_MEDIUM
@@ -197,11 +211,11 @@
SAVE_GPR(8, r7) /* user r8 */
SAVE_GPR(9, r7) /* user r9 */
SAVE_GPR(10, r7) /* user r10 */
- ld r3, PACATMSCRATCH(r13) /* user r1 */
+ ld r3, GPR1(r1) /* user r1 */
ld r4, GPR7(r1) /* user r7 */
ld r5, GPR11(r1) /* user r11 */
ld r6, GPR12(r1) /* user r12 */
- GET_SCRATCH0(8) /* user r13 */
+ ld r8, GPR13(r1) /* user r13 */
std r3, GPR1(r7)
std r4, GPR7(r7)
std r5, GPR11(r7)
diff --git a/arch/powerpc/kernel/vmlinux.lds.S b/arch/powerpc/kernel/vmlinux.lds.S
index b61fb79..c16fddb 100644
--- a/arch/powerpc/kernel/vmlinux.lds.S
+++ b/arch/powerpc/kernel/vmlinux.lds.S
@@ -134,6 +134,20 @@
#ifdef CONFIG_PPC64
. = ALIGN(8);
+ __stf_entry_barrier_fixup : AT(ADDR(__stf_entry_barrier_fixup) - LOAD_OFFSET) {
+ __start___stf_entry_barrier_fixup = .;
+ *(__stf_entry_barrier_fixup)
+ __stop___stf_entry_barrier_fixup = .;
+ }
+
+ . = ALIGN(8);
+ __stf_exit_barrier_fixup : AT(ADDR(__stf_exit_barrier_fixup) - LOAD_OFFSET) {
+ __start___stf_exit_barrier_fixup = .;
+ *(__stf_exit_barrier_fixup)
+ __stop___stf_exit_barrier_fixup = .;
+ }
+
+ . = ALIGN(8);
__rfi_flush_fixup : AT(ADDR(__rfi_flush_fixup) - LOAD_OFFSET) {
__start___rfi_flush_fixup = .;
*(__rfi_flush_fixup)
diff --git a/arch/powerpc/kvm/book3s_64_mmu_hv.c b/arch/powerpc/kvm/book3s_64_mmu_hv.c
index 05f09ae..915e89f 100644
--- a/arch/powerpc/kvm/book3s_64_mmu_hv.c
+++ b/arch/powerpc/kvm/book3s_64_mmu_hv.c
@@ -314,7 +314,7 @@ static int kvmppc_mmu_book3s_64_hv_xlate(struct kvm_vcpu *vcpu, gva_t eaddr,
unsigned long pp, key;
unsigned long v, gr;
__be64 *hptep;
- int index;
+ long int index;
int virtmode = vcpu->arch.shregs.msr & (data ? MSR_DR : MSR_IR);
/* Get SLB entry */
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 218cba2..0a2b247 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -3107,15 +3107,17 @@ static int kvmppc_hv_setup_htab_rma(struct kvm_vcpu *vcpu)
goto up_out;
psize = vma_kernel_pagesize(vma);
- porder = __ilog2(psize);
up_read(¤t->mm->mmap_sem);
/* We can handle 4k, 64k or 16M pages in the VRMA */
- err = -EINVAL;
- if (!(psize == 0x1000 || psize == 0x10000 ||
- psize == 0x1000000))
- goto out_srcu;
+ if (psize >= 0x1000000)
+ psize = 0x1000000;
+ else if (psize >= 0x10000)
+ psize = 0x10000;
+ else
+ psize = 0x1000;
+ porder = __ilog2(psize);
/* Update VRMASD field in the LPCR */
senc = slb_pgsize_encoding(psize);
diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
index 55fbc0c..79a180c 100644
--- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
+++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
@@ -299,7 +299,6 @@
stw r12, STACK_SLOT_TRAP(r1)
bl kvmhv_commence_exit
nop
- lwz r12, STACK_SLOT_TRAP(r1)
b kvmhv_switch_to_host
/*
@@ -1023,6 +1022,7 @@
secondary_too_late:
li r12, 0
+ stw r12, STACK_SLOT_TRAP(r1)
cmpdi r4, 0
beq 11f
stw r12, VCPU_TRAP(r4)
@@ -1266,12 +1266,12 @@
bl kvmhv_accumulate_time
#endif
+ stw r12, STACK_SLOT_TRAP(r1)
mr r3, r12
/* Increment exit count, poke other threads to exit */
bl kvmhv_commence_exit
nop
ld r9, HSTATE_KVM_VCPU(r13)
- lwz r12, VCPU_TRAP(r9)
/* Stop others sending VCPU interrupts to this physical CPU */
li r0, -1
@@ -1549,6 +1549,7 @@
* POWER7/POWER8 guest -> host partition switch code.
* We don't have to lock against tlbies but we do
* have to coordinate the hardware threads.
+ * Here STACK_SLOT_TRAP(r1) contains the trap number.
*/
kvmhv_switch_to_host:
/* Secondary threads wait for primary to do partition switch */
@@ -1599,11 +1600,11 @@
END_FTR_SECTION_IFSET(CPU_FTR_ARCH_207S)
/* If HMI, call kvmppc_realmode_hmi_handler() */
+ lwz r12, STACK_SLOT_TRAP(r1)
cmpwi r12, BOOK3S_INTERRUPT_HMI
bne 27f
bl kvmppc_realmode_hmi_handler
nop
- li r12, BOOK3S_INTERRUPT_HMI
/*
* At this point kvmppc_realmode_hmi_handler would have resync-ed
* the TB. Hence it is not required to subtract guest timebase
@@ -1678,6 +1679,7 @@
li r0, KVM_GUEST_MODE_NONE
stb r0, HSTATE_IN_GUEST(r13)
+ lwz r12, STACK_SLOT_TRAP(r1) /* return trap # in r12 */
ld r0, SFS+PPC_LR_STKOFF(r1)
addi r1, r1, SFS
mtlr r0
diff --git a/arch/powerpc/kvm/trace.h b/arch/powerpc/kvm/trace.h
index 2e0e67e..e8cedf3 100644
--- a/arch/powerpc/kvm/trace.h
+++ b/arch/powerpc/kvm/trace.h
@@ -5,8 +5,6 @@
#undef TRACE_SYSTEM
#define TRACE_SYSTEM kvm
-#define TRACE_INCLUDE_PATH .
-#define TRACE_INCLUDE_FILE trace
/*
* Tracepoint for guest mode entry.
@@ -119,4 +117,10 @@ TRACE_EVENT(kvm_check_requests,
#endif /* _TRACE_KVM_H */
/* This part must be outside protection */
+#undef TRACE_INCLUDE_PATH
+#undef TRACE_INCLUDE_FILE
+
+#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_FILE trace
+
#include <trace/define_trace.h>
diff --git a/arch/powerpc/kvm/trace_booke.h b/arch/powerpc/kvm/trace_booke.h
index 7ec534d..7eadbf4 100644
--- a/arch/powerpc/kvm/trace_booke.h
+++ b/arch/powerpc/kvm/trace_booke.h
@@ -5,8 +5,6 @@
#undef TRACE_SYSTEM
#define TRACE_SYSTEM kvm_booke
-#define TRACE_INCLUDE_PATH .
-#define TRACE_INCLUDE_FILE trace_booke
#define kvm_trace_symbol_exit \
{0, "CRITICAL"}, \
@@ -217,4 +215,11 @@ TRACE_EVENT(kvm_booke_queue_irqprio,
#endif
/* This part must be outside protection */
+
+#undef TRACE_INCLUDE_PATH
+#undef TRACE_INCLUDE_FILE
+
+#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_FILE trace_booke
+
#include <trace/define_trace.h>
diff --git a/arch/powerpc/kvm/trace_hv.h b/arch/powerpc/kvm/trace_hv.h
index fb21990..d9a21a7 100644
--- a/arch/powerpc/kvm/trace_hv.h
+++ b/arch/powerpc/kvm/trace_hv.h
@@ -8,8 +8,6 @@
#undef TRACE_SYSTEM
#define TRACE_SYSTEM kvm_hv
-#define TRACE_INCLUDE_PATH .
-#define TRACE_INCLUDE_FILE trace_hv
#define kvm_trace_symbol_hcall \
{H_REMOVE, "H_REMOVE"}, \
@@ -496,4 +494,11 @@ TRACE_EVENT(kvmppc_run_vcpu_exit,
#endif /* _TRACE_KVM_HV_H */
/* This part must be outside protection */
+
+#undef TRACE_INCLUDE_PATH
+#undef TRACE_INCLUDE_FILE
+
+#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_FILE trace_hv
+
#include <trace/define_trace.h>
diff --git a/arch/powerpc/kvm/trace_pr.h b/arch/powerpc/kvm/trace_pr.h
index d44f324..e8e2b9a 100644
--- a/arch/powerpc/kvm/trace_pr.h
+++ b/arch/powerpc/kvm/trace_pr.h
@@ -7,8 +7,6 @@
#undef TRACE_SYSTEM
#define TRACE_SYSTEM kvm_pr
-#define TRACE_INCLUDE_PATH .
-#define TRACE_INCLUDE_FILE trace_pr
TRACE_EVENT(kvm_book3s_reenter,
TP_PROTO(int r, struct kvm_vcpu *vcpu),
@@ -271,4 +269,11 @@ TRACE_EVENT(kvm_unmap_hva,
#endif /* _TRACE_KVM_H */
/* This part must be outside protection */
+
+#undef TRACE_INCLUDE_PATH
+#undef TRACE_INCLUDE_FILE
+
+#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_FILE trace_pr
+
#include <trace/define_trace.h>
diff --git a/arch/powerpc/lib/feature-fixups.c b/arch/powerpc/lib/feature-fixups.c
index 46c8338..cf1398e 100644
--- a/arch/powerpc/lib/feature-fixups.c
+++ b/arch/powerpc/lib/feature-fixups.c
@@ -22,6 +22,7 @@
#include <asm/page.h>
#include <asm/sections.h>
#include <asm/setup.h>
+#include <asm/security_features.h>
#include <asm/firmware.h>
#include <asm/setup.h>
@@ -117,6 +118,120 @@ void do_feature_fixups(unsigned long value, void *fixup_start, void *fixup_end)
}
#ifdef CONFIG_PPC_BOOK3S_64
+void do_stf_entry_barrier_fixups(enum stf_barrier_type types)
+{
+ unsigned int instrs[3], *dest;
+ long *start, *end;
+ int i;
+
+ start = PTRRELOC(&__start___stf_entry_barrier_fixup),
+ end = PTRRELOC(&__stop___stf_entry_barrier_fixup);
+
+ instrs[0] = 0x60000000; /* nop */
+ instrs[1] = 0x60000000; /* nop */
+ instrs[2] = 0x60000000; /* nop */
+
+ i = 0;
+ if (types & STF_BARRIER_FALLBACK) {
+ instrs[i++] = 0x7d4802a6; /* mflr r10 */
+ instrs[i++] = 0x60000000; /* branch patched below */
+ instrs[i++] = 0x7d4803a6; /* mtlr r10 */
+ } else if (types & STF_BARRIER_EIEIO) {
+ instrs[i++] = 0x7e0006ac; /* eieio + bit 6 hint */
+ } else if (types & STF_BARRIER_SYNC_ORI) {
+ instrs[i++] = 0x7c0004ac; /* hwsync */
+ instrs[i++] = 0xe94d0000; /* ld r10,0(r13) */
+ instrs[i++] = 0x63ff0000; /* ori 31,31,0 speculation barrier */
+ }
+
+ for (i = 0; start < end; start++, i++) {
+ dest = (void *)start + *start;
+
+ pr_devel("patching dest %lx\n", (unsigned long)dest);
+
+ patch_instruction(dest, instrs[0]);
+
+ if (types & STF_BARRIER_FALLBACK)
+ patch_branch(dest + 1, (unsigned long)&stf_barrier_fallback,
+ BRANCH_SET_LINK);
+ else
+ patch_instruction(dest + 1, instrs[1]);
+
+ patch_instruction(dest + 2, instrs[2]);
+ }
+
+ printk(KERN_DEBUG "stf-barrier: patched %d entry locations (%s barrier)\n", i,
+ (types == STF_BARRIER_NONE) ? "no" :
+ (types == STF_BARRIER_FALLBACK) ? "fallback" :
+ (types == STF_BARRIER_EIEIO) ? "eieio" :
+ (types == (STF_BARRIER_SYNC_ORI)) ? "hwsync"
+ : "unknown");
+}
+
+void do_stf_exit_barrier_fixups(enum stf_barrier_type types)
+{
+ unsigned int instrs[6], *dest;
+ long *start, *end;
+ int i;
+
+ start = PTRRELOC(&__start___stf_exit_barrier_fixup),
+ end = PTRRELOC(&__stop___stf_exit_barrier_fixup);
+
+ instrs[0] = 0x60000000; /* nop */
+ instrs[1] = 0x60000000; /* nop */
+ instrs[2] = 0x60000000; /* nop */
+ instrs[3] = 0x60000000; /* nop */
+ instrs[4] = 0x60000000; /* nop */
+ instrs[5] = 0x60000000; /* nop */
+
+ i = 0;
+ if (types & STF_BARRIER_FALLBACK || types & STF_BARRIER_SYNC_ORI) {
+ if (cpu_has_feature(CPU_FTR_HVMODE)) {
+ instrs[i++] = 0x7db14ba6; /* mtspr 0x131, r13 (HSPRG1) */
+ instrs[i++] = 0x7db04aa6; /* mfspr r13, 0x130 (HSPRG0) */
+ } else {
+ instrs[i++] = 0x7db243a6; /* mtsprg 2,r13 */
+ instrs[i++] = 0x7db142a6; /* mfsprg r13,1 */
+ }
+ instrs[i++] = 0x7c0004ac; /* hwsync */
+ instrs[i++] = 0xe9ad0000; /* ld r13,0(r13) */
+ instrs[i++] = 0x63ff0000; /* ori 31,31,0 speculation barrier */
+ if (cpu_has_feature(CPU_FTR_HVMODE)) {
+ instrs[i++] = 0x7db14aa6; /* mfspr r13, 0x131 (HSPRG1) */
+ } else {
+ instrs[i++] = 0x7db242a6; /* mfsprg r13,2 */
+ }
+ } else if (types & STF_BARRIER_EIEIO) {
+ instrs[i++] = 0x7e0006ac; /* eieio + bit 6 hint */
+ }
+
+ for (i = 0; start < end; start++, i++) {
+ dest = (void *)start + *start;
+
+ pr_devel("patching dest %lx\n", (unsigned long)dest);
+
+ patch_instruction(dest, instrs[0]);
+ patch_instruction(dest + 1, instrs[1]);
+ patch_instruction(dest + 2, instrs[2]);
+ patch_instruction(dest + 3, instrs[3]);
+ patch_instruction(dest + 4, instrs[4]);
+ patch_instruction(dest + 5, instrs[5]);
+ }
+ printk(KERN_DEBUG "stf-barrier: patched %d exit locations (%s barrier)\n", i,
+ (types == STF_BARRIER_NONE) ? "no" :
+ (types == STF_BARRIER_FALLBACK) ? "fallback" :
+ (types == STF_BARRIER_EIEIO) ? "eieio" :
+ (types == (STF_BARRIER_SYNC_ORI)) ? "hwsync"
+ : "unknown");
+}
+
+
+void do_stf_barrier_fixups(enum stf_barrier_type types)
+{
+ do_stf_entry_barrier_fixups(types);
+ do_stf_exit_barrier_fixups(types);
+}
+
void do_rfi_flush_fixups(enum l1d_flush_type types)
{
unsigned int instrs[3], *dest;
@@ -153,7 +268,14 @@ void do_rfi_flush_fixups(enum l1d_flush_type types)
patch_instruction(dest + 2, instrs[2]);
}
- printk(KERN_DEBUG "rfi-flush: patched %d locations\n", i);
+ printk(KERN_DEBUG "rfi-flush: patched %d locations (%s flush)\n", i,
+ (types == L1D_FLUSH_NONE) ? "no" :
+ (types == L1D_FLUSH_FALLBACK) ? "fallback displacement" :
+ (types & L1D_FLUSH_ORI) ? (types & L1D_FLUSH_MTTRIG)
+ ? "ori+mttrig type"
+ : "ori type" :
+ (types & L1D_FLUSH_MTTRIG) ? "mttrig type"
+ : "unknown");
}
#endif /* CONFIG_PPC_BOOK3S_64 */
diff --git a/arch/powerpc/mm/numa.c b/arch/powerpc/mm/numa.c
index a51c188..0ef83c2 100644
--- a/arch/powerpc/mm/numa.c
+++ b/arch/powerpc/mm/numa.c
@@ -551,7 +551,7 @@ static int numa_setup_cpu(unsigned long lcpu)
nid = of_node_to_nid_single(cpu);
out_present:
- if (nid < 0 || !node_online(nid))
+ if (nid < 0 || !node_possible(nid))
nid = first_online_node;
map_cpu_to_node(lcpu, nid);
@@ -904,6 +904,32 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
NODE_DATA(nid)->node_spanned_pages = spanned_pages;
}
+static void __init find_possible_nodes(void)
+{
+ struct device_node *rtas;
+ u32 numnodes, i;
+
+ if (min_common_depth <= 0)
+ return;
+
+ rtas = of_find_node_by_path("/rtas");
+ if (!rtas)
+ return;
+
+ if (of_property_read_u32_index(rtas,
+ "ibm,max-associativity-domains",
+ min_common_depth, &numnodes))
+ goto out;
+
+ for (i = 0; i < numnodes; i++) {
+ if (!node_possible(i))
+ node_set(i, node_possible_map);
+ }
+
+out:
+ of_node_put(rtas);
+}
+
void __init initmem_init(void)
{
int nid, cpu;
@@ -917,12 +943,15 @@ void __init initmem_init(void)
memblock_dump_all();
/*
- * Reduce the possible NUMA nodes to the online NUMA nodes,
- * since we do not support node hotplug. This ensures that we
- * lower the maximum NUMA node ID to what is actually present.
+ * Modify the set of possible NUMA nodes to reflect information
+ * available about the set of online nodes, and the set of nodes
+ * that we expect to make use of for this platform's affinity
+ * calculations.
*/
nodes_and(node_possible_map, node_possible_map, node_online_map);
+ find_possible_nodes();
+
for_each_online_node(nid) {
unsigned long start_pfn, end_pfn;
@@ -1260,7 +1289,7 @@ static long vphn_get_associativity(unsigned long cpu,
switch (rc) {
case H_FUNCTION:
- printk(KERN_INFO
+ printk_once(KERN_INFO
"VPHN is not supported. Disabling polling...\n");
stop_topology_update();
break;
@@ -1274,6 +1303,40 @@ static long vphn_get_associativity(unsigned long cpu,
return rc;
}
+static inline int find_and_online_cpu_nid(int cpu)
+{
+ __be32 associativity[VPHN_ASSOC_BUFSIZE] = {0};
+ int new_nid;
+
+ /* Use associativity from first thread for all siblings */
+ vphn_get_associativity(cpu, associativity);
+ new_nid = associativity_to_nid(associativity);
+ if (new_nid < 0 || !node_possible(new_nid))
+ new_nid = first_online_node;
+
+ if (NODE_DATA(new_nid) == NULL) {
+#ifdef CONFIG_MEMORY_HOTPLUG
+ /*
+ * Need to ensure that NODE_DATA is initialized for a node from
+ * available memory (see memblock_alloc_try_nid). If unable to
+ * init the node, then default to nearest node that has memory
+ * installed.
+ */
+ if (try_online_node(new_nid))
+ new_nid = first_online_node;
+#else
+ /*
+ * Default to using the nearest node that has memory installed.
+ * Otherwise, it would be necessary to patch the kernel MM code
+ * to deal with more memoryless-node error conditions.
+ */
+ new_nid = first_online_node;
+#endif
+ }
+
+ return new_nid;
+}
+
/*
* Update the CPU maps and sysfs entries for a single CPU when its NUMA
* characteristics change. This function doesn't perform any locking and is
@@ -1339,7 +1402,6 @@ int arch_update_cpu_topology(void)
{
unsigned int cpu, sibling, changed = 0;
struct topology_update_data *updates, *ud;
- __be32 associativity[VPHN_ASSOC_BUFSIZE] = {0};
cpumask_t updated_cpus;
struct device *dev;
int weight, new_nid, i = 0;
@@ -1374,11 +1436,7 @@ int arch_update_cpu_topology(void)
continue;
}
- /* Use associativity from first thread for all siblings */
- vphn_get_associativity(cpu, associativity);
- new_nid = associativity_to_nid(associativity);
- if (new_nid < 0 || !node_online(new_nid))
- new_nid = first_online_node;
+ new_nid = find_and_online_cpu_nid(cpu);
if (new_nid == numa_cpu_lookup_table[cpu]) {
cpumask_andnot(&cpu_associativity_changes_mask,
diff --git a/arch/powerpc/mm/slb.c b/arch/powerpc/mm/slb.c
index 48fc28b..64c9a91 100644
--- a/arch/powerpc/mm/slb.c
+++ b/arch/powerpc/mm/slb.c
@@ -68,14 +68,14 @@ static inline void slb_shadow_update(unsigned long ea, int ssize,
* updating it. No write barriers are needed here, provided
* we only update the current CPU's SLB shadow buffer.
*/
- p->save_area[index].esid = 0;
- p->save_area[index].vsid = cpu_to_be64(mk_vsid_data(ea, ssize, flags));
- p->save_area[index].esid = cpu_to_be64(mk_esid_data(ea, ssize, index));
+ WRITE_ONCE(p->save_area[index].esid, 0);
+ WRITE_ONCE(p->save_area[index].vsid, cpu_to_be64(mk_vsid_data(ea, ssize, flags)));
+ WRITE_ONCE(p->save_area[index].esid, cpu_to_be64(mk_esid_data(ea, ssize, index)));
}
static inline void slb_shadow_clear(enum slb_index index)
{
- get_slb_shadow()->save_area[index].esid = 0;
+ WRITE_ONCE(get_slb_shadow()->save_area[index].esid, 0);
}
static inline void create_shadowed_slbe(unsigned long ea, int ssize,
diff --git a/arch/powerpc/mm/tlb_nohash.c b/arch/powerpc/mm/tlb_nohash.c
index 0b50019..79cf21b 100644
--- a/arch/powerpc/mm/tlb_nohash.c
+++ b/arch/powerpc/mm/tlb_nohash.c
@@ -481,6 +481,9 @@ static void setup_page_sizes(void)
for (psize = 0; psize < MMU_PAGE_COUNT; ++psize) {
struct mmu_psize_def *def = &mmu_psize_defs[psize];
+ if (!def->shift)
+ continue;
+
if (tlb1ps & (1U << (def->shift - 10))) {
def->flags |= MMU_PAGE_SIZE_DIRECT;
diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c
index 7e706f3..9c58194 100644
--- a/arch/powerpc/net/bpf_jit_comp.c
+++ b/arch/powerpc/net/bpf_jit_comp.c
@@ -329,6 +329,9 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,
BUILD_BUG_ON(FIELD_SIZEOF(struct sk_buff, len) != 4);
PPC_LWZ_OFFS(r_A, r_skb, offsetof(struct sk_buff, len));
break;
+ case BPF_LDX | BPF_W | BPF_ABS: /* A = *((u32 *)(seccomp_data + K)); */
+ PPC_LWZ_OFFS(r_A, r_skb, K);
+ break;
case BPF_LDX | BPF_W | BPF_LEN: /* X = skb->len; */
PPC_LWZ_OFFS(r_X, r_skb, offsetof(struct sk_buff, len));
break;
diff --git a/arch/powerpc/net/bpf_jit_comp64.c b/arch/powerpc/net/bpf_jit_comp64.c
index be9d968..bdbbc32 100644
--- a/arch/powerpc/net/bpf_jit_comp64.c
+++ b/arch/powerpc/net/bpf_jit_comp64.c
@@ -207,25 +207,37 @@ static void bpf_jit_build_epilogue(u32 *image, struct codegen_context *ctx)
static void bpf_jit_emit_func_call(u32 *image, struct codegen_context *ctx, u64 func)
{
+ unsigned int i, ctx_idx = ctx->idx;
+
+ /* Load function address into r12 */
+ PPC_LI64(12, func);
+
+ /* For bpf-to-bpf function calls, the callee's address is unknown
+ * until the last extra pass. As seen above, we use PPC_LI64() to
+ * load the callee's address, but this may optimize the number of
+ * instructions required based on the nature of the address.
+ *
+ * Since we don't want the number of instructions emitted to change,
+ * we pad the optimized PPC_LI64() call with NOPs to guarantee that
+ * we always have a five-instruction sequence, which is the maximum
+ * that PPC_LI64() can emit.
+ */
+ for (i = ctx->idx - ctx_idx; i < 5; i++)
+ PPC_NOP();
+
#ifdef PPC64_ELF_ABI_v1
- /* func points to the function descriptor */
- PPC_LI64(b2p[TMP_REG_2], func);
- /* Load actual entry point from function descriptor */
- PPC_BPF_LL(b2p[TMP_REG_1], b2p[TMP_REG_2], 0);
- /* ... and move it to LR */
- PPC_MTLR(b2p[TMP_REG_1]);
/*
* Load TOC from function descriptor at offset 8.
* We can clobber r2 since we get called through a
* function pointer (so caller will save/restore r2)
* and since we don't use a TOC ourself.
*/
- PPC_BPF_LL(2, b2p[TMP_REG_2], 8);
-#else
- /* We can clobber r12 */
- PPC_FUNC_ADDR(12, func);
- PPC_MTLR(12);
+ PPC_BPF_LL(2, 12, 8);
+ /* Load actual entry point from function descriptor */
+ PPC_BPF_LL(12, 12, 0);
#endif
+
+ PPC_MTLR(12);
PPC_BLRL();
}
@@ -314,6 +326,7 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,
u64 imm64;
u8 *func;
u32 true_cond;
+ u32 tmp_idx;
/*
* addrs[] maps a BPF bytecode address into a real offset from
@@ -673,11 +686,7 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,
case BPF_STX | BPF_XADD | BPF_W:
/* Get EA into TMP_REG_1 */
PPC_ADDI(b2p[TMP_REG_1], dst_reg, off);
- /* error if EA is not word-aligned */
- PPC_ANDI(b2p[TMP_REG_2], b2p[TMP_REG_1], 0x03);
- PPC_BCC_SHORT(COND_EQ, (ctx->idx * 4) + 12);
- PPC_LI(b2p[BPF_REG_0], 0);
- PPC_JMP(exit_addr);
+ tmp_idx = ctx->idx * 4;
/* load value from memory into TMP_REG_2 */
PPC_BPF_LWARX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1], 0);
/* add value from src_reg into this */
@@ -685,32 +694,16 @@ static int bpf_jit_build_body(struct bpf_prog *fp, u32 *image,
/* store result back */
PPC_BPF_STWCX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1]);
/* we're done if this succeeded */
- PPC_BCC_SHORT(COND_EQ, (ctx->idx * 4) + (7*4));
- /* otherwise, let's try once more */
- PPC_BPF_LWARX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1], 0);
- PPC_ADD(b2p[TMP_REG_2], b2p[TMP_REG_2], src_reg);
- PPC_BPF_STWCX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1]);
- /* exit if the store was not successful */
- PPC_LI(b2p[BPF_REG_0], 0);
- PPC_BCC(COND_NE, exit_addr);
+ PPC_BCC_SHORT(COND_NE, tmp_idx);
break;
/* *(u64 *)(dst + off) += src */
case BPF_STX | BPF_XADD | BPF_DW:
PPC_ADDI(b2p[TMP_REG_1], dst_reg, off);
- /* error if EA is not doubleword-aligned */
- PPC_ANDI(b2p[TMP_REG_2], b2p[TMP_REG_1], 0x07);
- PPC_BCC_SHORT(COND_EQ, (ctx->idx * 4) + (3*4));
- PPC_LI(b2p[BPF_REG_0], 0);
- PPC_JMP(exit_addr);
+ tmp_idx = ctx->idx * 4;
PPC_BPF_LDARX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1], 0);
PPC_ADD(b2p[TMP_REG_2], b2p[TMP_REG_2], src_reg);
PPC_BPF_STDCX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1]);
- PPC_BCC_SHORT(COND_EQ, (ctx->idx * 4) + (7*4));
- PPC_BPF_LDARX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1], 0);
- PPC_ADD(b2p[TMP_REG_2], b2p[TMP_REG_2], src_reg);
- PPC_BPF_STDCX(b2p[TMP_REG_2], 0, b2p[TMP_REG_1]);
- PPC_LI(b2p[BPF_REG_0], 0);
- PPC_BCC(COND_NE, exit_addr);
+ PPC_BCC_SHORT(COND_NE, tmp_idx);
break;
/*
diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c
index bf94962..771edff 100644
--- a/arch/powerpc/perf/core-book3s.c
+++ b/arch/powerpc/perf/core-book3s.c
@@ -448,6 +448,16 @@ static void power_pmu_bhrb_read(struct cpu_hw_events *cpuhw)
/* invalid entry */
continue;
+ /*
+ * BHRB rolling buffer could very much contain the kernel
+ * addresses at this point. Check the privileges before
+ * exporting it to userspace (avoid exposure of regions
+ * where we could have speculative execution)
+ */
+ if (perf_paranoid_kernel() && !capable(CAP_SYS_ADMIN) &&
+ is_kernel_addr(addr))
+ continue;
+
/* Branches are read most recent first (ie. mfbhrb 0 is
* the most recent branch).
* There are two types of valid entries:
@@ -1188,6 +1198,7 @@ static void power_pmu_disable(struct pmu *pmu)
*/
write_mmcr0(cpuhw, val);
mb();
+ isync();
/*
* Disable instruction sampling if it was enabled
@@ -1196,12 +1207,26 @@ static void power_pmu_disable(struct pmu *pmu)
mtspr(SPRN_MMCRA,
cpuhw->mmcr[2] & ~MMCRA_SAMPLE_ENABLE);
mb();
+ isync();
}
cpuhw->disabled = 1;
cpuhw->n_added = 0;
ebb_switch_out(mmcr0);
+
+#ifdef CONFIG_PPC64
+ /*
+ * These are readable by userspace, may contain kernel
+ * addresses and are not switched by context switch, so clear
+ * them now to avoid leaking anything to userspace in general
+ * including to another process.
+ */
+ if (ppmu->flags & PPMU_ARCH_207S) {
+ mtspr(SPRN_SDAR, 0);
+ mtspr(SPRN_SIAR, 0);
+ }
+#endif
}
local_irq_restore(flags);
diff --git a/arch/powerpc/platforms/cell/cpufreq_spudemand.c b/arch/powerpc/platforms/cell/cpufreq_spudemand.c
index 88301e5..5d8e8b6 100644
--- a/arch/powerpc/platforms/cell/cpufreq_spudemand.c
+++ b/arch/powerpc/platforms/cell/cpufreq_spudemand.c
@@ -22,6 +22,7 @@
#include <linux/cpufreq.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/module.h>
#include <linux/timer.h>
#include <linux/workqueue.h>
@@ -48,7 +49,7 @@ static int calc_freq(struct spu_gov_info_struct *info)
cpu = info->policy->cpu;
busy_spus = atomic_read(&cbe_spu_info[cpu_to_node(cpu)].busy_spus);
- CALC_LOAD(info->busy_spus, EXP, busy_spus * FIXED_1);
+ info->busy_spus = calc_load(info->busy_spus, EXP, busy_spus * FIXED_1);
pr_debug("cpu %d: busy_spus=%d, info->busy_spus=%ld\n",
cpu, busy_spus, info->busy_spus);
diff --git a/arch/powerpc/platforms/cell/spufs/sched.c b/arch/powerpc/platforms/cell/spufs/sched.c
index 460f5f3..1149c34 100644
--- a/arch/powerpc/platforms/cell/spufs/sched.c
+++ b/arch/powerpc/platforms/cell/spufs/sched.c
@@ -24,6 +24,7 @@
#include <linux/errno.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/sched/rt.h>
#include <linux/kernel.h>
#include <linux/mm.h>
@@ -986,9 +987,9 @@ static void spu_calc_load(void)
unsigned long active_tasks; /* fixed-point */
active_tasks = count_active_contexts() * FIXED_1;
- CALC_LOAD(spu_avenrun[0], EXP_1, active_tasks);
- CALC_LOAD(spu_avenrun[1], EXP_5, active_tasks);
- CALC_LOAD(spu_avenrun[2], EXP_15, active_tasks);
+ spu_avenrun[0] = calc_load(spu_avenrun[0], EXP_1, active_tasks);
+ spu_avenrun[1] = calc_load(spu_avenrun[1], EXP_5, active_tasks);
+ spu_avenrun[2] = calc_load(spu_avenrun[2], EXP_15, active_tasks);
}
static void spusched_wake(unsigned long data)
@@ -1070,9 +1071,6 @@ void spuctx_switch_state(struct spu_context *ctx,
}
}
-#define LOAD_INT(x) ((x) >> FSHIFT)
-#define LOAD_FRAC(x) LOAD_INT(((x) & (FIXED_1-1)) * 100)
-
static int show_spu_loadavg(struct seq_file *s, void *private)
{
int a, b, c;
diff --git a/arch/powerpc/platforms/chrp/time.c b/arch/powerpc/platforms/chrp/time.c
index f803f4b..8608e35 100644
--- a/arch/powerpc/platforms/chrp/time.c
+++ b/arch/powerpc/platforms/chrp/time.c
@@ -27,6 +27,8 @@
#include <asm/sections.h>
#include <asm/time.h>
+#include <platforms/chrp/chrp.h>
+
extern spinlock_t rtc_lock;
#define NVRAM_AS0 0x74
@@ -62,7 +64,7 @@ long __init chrp_time_init(void)
return 0;
}
-int chrp_cmos_clock_read(int addr)
+static int chrp_cmos_clock_read(int addr)
{
if (nvram_as1 != 0)
outb(addr>>8, nvram_as1);
@@ -70,7 +72,7 @@ int chrp_cmos_clock_read(int addr)
return (inb(nvram_data));
}
-void chrp_cmos_clock_write(unsigned long val, int addr)
+static void chrp_cmos_clock_write(unsigned long val, int addr)
{
if (nvram_as1 != 0)
outb(addr>>8, nvram_as1);
diff --git a/arch/powerpc/platforms/embedded6xx/hlwd-pic.c b/arch/powerpc/platforms/embedded6xx/hlwd-pic.c
index 89c54de..bf4a125 100644
--- a/arch/powerpc/platforms/embedded6xx/hlwd-pic.c
+++ b/arch/powerpc/platforms/embedded6xx/hlwd-pic.c
@@ -35,6 +35,8 @@
*/
#define HW_BROADWAY_ICR 0x00
#define HW_BROADWAY_IMR 0x04
+#define HW_STARLET_ICR 0x08
+#define HW_STARLET_IMR 0x0c
/*
@@ -74,6 +76,9 @@ static void hlwd_pic_unmask(struct irq_data *d)
void __iomem *io_base = irq_data_get_irq_chip_data(d);
setbits32(io_base + HW_BROADWAY_IMR, 1 << irq);
+
+ /* Make sure the ARM (aka. Starlet) doesn't handle this interrupt. */
+ clrbits32(io_base + HW_STARLET_IMR, 1 << irq);
}
diff --git a/arch/powerpc/platforms/powermac/bootx_init.c b/arch/powerpc/platforms/powermac/bootx_init.c
index c3c9bbb..ba0964c 100644
--- a/arch/powerpc/platforms/powermac/bootx_init.c
+++ b/arch/powerpc/platforms/powermac/bootx_init.c
@@ -468,7 +468,7 @@ void __init bootx_init(unsigned long r3, unsigned long r4)
boot_infos_t *bi = (boot_infos_t *) r4;
unsigned long hdr;
unsigned long space;
- unsigned long ptr, x;
+ unsigned long ptr;
char *model;
unsigned long offset = reloc_offset();
@@ -562,6 +562,8 @@ void __init bootx_init(unsigned long r3, unsigned long r4)
* MMU switched OFF, so this should not be useful anymore.
*/
if (bi->version < 4) {
+ unsigned long x __maybe_unused;
+
bootx_printf("Touching pages...\n");
/*
diff --git a/arch/powerpc/platforms/powermac/setup.c b/arch/powerpc/platforms/powermac/setup.c
index 6b4e9d18..4929dd4 100644
--- a/arch/powerpc/platforms/powermac/setup.c
+++ b/arch/powerpc/platforms/powermac/setup.c
@@ -352,6 +352,7 @@ static int pmac_late_init(void)
}
machine_late_initcall(powermac, pmac_late_init);
+void note_bootable_part(dev_t dev, int part, int goodness);
/*
* This is __ref because we check for "initializing" before
* touching any of the __init sensitive things and "initializing"
diff --git a/arch/powerpc/platforms/powernv/opal-nvram.c b/arch/powerpc/platforms/powernv/opal-nvram.c
index 1bceb95..5584247 100644
--- a/arch/powerpc/platforms/powernv/opal-nvram.c
+++ b/arch/powerpc/platforms/powernv/opal-nvram.c
@@ -44,6 +44,10 @@ static ssize_t opal_nvram_read(char *buf, size_t count, loff_t *index)
return count;
}
+/*
+ * This can be called in the panic path with interrupts off, so use
+ * mdelay in that case.
+ */
static ssize_t opal_nvram_write(char *buf, size_t count, loff_t *index)
{
s64 rc = OPAL_BUSY;
@@ -58,10 +62,16 @@ static ssize_t opal_nvram_write(char *buf, size_t count, loff_t *index)
while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT) {
rc = opal_write_nvram(__pa(buf), count, off);
if (rc == OPAL_BUSY_EVENT) {
- msleep(OPAL_BUSY_DELAY_MS);
+ if (in_interrupt() || irqs_disabled())
+ mdelay(OPAL_BUSY_DELAY_MS);
+ else
+ msleep(OPAL_BUSY_DELAY_MS);
opal_poll_events(NULL);
} else if (rc == OPAL_BUSY) {
- msleep(OPAL_BUSY_DELAY_MS);
+ if (in_interrupt() || irqs_disabled())
+ mdelay(OPAL_BUSY_DELAY_MS);
+ else
+ msleep(OPAL_BUSY_DELAY_MS);
}
}
diff --git a/arch/powerpc/platforms/powernv/opal-rtc.c b/arch/powerpc/platforms/powernv/opal-rtc.c
index f886886..aa2a513 100644
--- a/arch/powerpc/platforms/powernv/opal-rtc.c
+++ b/arch/powerpc/platforms/powernv/opal-rtc.c
@@ -48,10 +48,12 @@ unsigned long __init opal_get_boot_time(void)
while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT) {
rc = opal_rtc_read(&__y_m_d, &__h_m_s_ms);
- if (rc == OPAL_BUSY_EVENT)
+ if (rc == OPAL_BUSY_EVENT) {
+ mdelay(OPAL_BUSY_DELAY_MS);
opal_poll_events(NULL);
- else if (rc == OPAL_BUSY)
- mdelay(10);
+ } else if (rc == OPAL_BUSY) {
+ mdelay(OPAL_BUSY_DELAY_MS);
+ }
}
if (rc != OPAL_SUCCESS)
return 0;
diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c
index 6c9a65b..7fb61eb 100644
--- a/arch/powerpc/platforms/powernv/opal.c
+++ b/arch/powerpc/platforms/powernv/opal.c
@@ -369,7 +369,7 @@ int opal_put_chars(uint32_t vtermno, const char *data, int total_len)
/* Closed or other error drop */
if (rc != OPAL_SUCCESS && rc != OPAL_BUSY &&
rc != OPAL_BUSY_EVENT) {
- written = total_len;
+ written += total_len;
break;
}
if (rc == OPAL_SUCCESS) {
diff --git a/arch/powerpc/platforms/powernv/pci-ioda.c b/arch/powerpc/platforms/powernv/pci-ioda.c
index f602307..8015e40 100644
--- a/arch/powerpc/platforms/powernv/pci-ioda.c
+++ b/arch/powerpc/platforms/powernv/pci-ioda.c
@@ -2623,7 +2623,7 @@ static long pnv_pci_ioda2_table_alloc_pages(int nid, __u64 bus_offset,
level_shift = entries_shift + 3;
level_shift = max_t(unsigned, level_shift, PAGE_SHIFT);
- if ((level_shift - 3) * levels + page_shift >= 60)
+ if ((level_shift - 3) * levels + page_shift >= 55)
return -EINVAL;
/* Allocate TCE table */
@@ -3124,12 +3124,49 @@ static void pnv_pci_ioda_create_dbgfs(void)
#endif /* CONFIG_DEBUG_FS */
}
+static void pnv_pci_enable_bridge(struct pci_bus *bus)
+{
+ struct pci_dev *dev = bus->self;
+ struct pci_bus *child;
+
+ /* Empty bus ? bail */
+ if (list_empty(&bus->devices))
+ return;
+
+ /*
+ * If there's a bridge associated with that bus enable it. This works
+ * around races in the generic code if the enabling is done during
+ * parallel probing. This can be removed once those races have been
+ * fixed.
+ */
+ if (dev) {
+ int rc = pci_enable_device(dev);
+ if (rc)
+ pci_err(dev, "Error enabling bridge (%d)\n", rc);
+ pci_set_master(dev);
+ }
+
+ /* Perform the same to child busses */
+ list_for_each_entry(child, &bus->children, node)
+ pnv_pci_enable_bridge(child);
+}
+
+static void pnv_pci_enable_bridges(void)
+{
+ struct pci_controller *hose;
+
+ list_for_each_entry(hose, &hose_list, list_node)
+ pnv_pci_enable_bridge(hose->bus);
+}
+
static void pnv_pci_ioda_fixup(void)
{
pnv_pci_ioda_setup_PEs();
pnv_pci_ioda_setup_iommu_api();
pnv_pci_ioda_create_dbgfs();
+ pnv_pci_enable_bridges();
+
#ifdef CONFIG_EEH
eeh_init();
eeh_addr_cache_build();
@@ -3424,7 +3461,6 @@ static void pnv_pci_ioda2_release_pe_dma(struct pnv_ioda_pe *pe)
WARN_ON(pe->table_group.group);
}
- pnv_pci_ioda2_table_free_pages(tbl);
iommu_free_table(tbl, "pnv");
}
diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
index 6f8b4c1..17203ab 100644
--- a/arch/powerpc/platforms/powernv/setup.c
+++ b/arch/powerpc/platforms/powernv/setup.c
@@ -37,53 +37,92 @@
#include <asm/smp.h>
#include <asm/tm.h>
#include <asm/setup.h>
+#include <asm/security_features.h>
#include "powernv.h"
+
+static bool fw_feature_is(const char *state, const char *name,
+ struct device_node *fw_features)
+{
+ struct device_node *np;
+ bool rc = false;
+
+ np = of_get_child_by_name(fw_features, name);
+ if (np) {
+ rc = of_property_read_bool(np, state);
+ of_node_put(np);
+ }
+
+ return rc;
+}
+
+static void init_fw_feat_flags(struct device_node *np)
+{
+ if (fw_feature_is("enabled", "inst-spec-barrier-ori31,31,0", np))
+ security_ftr_set(SEC_FTR_SPEC_BAR_ORI31);
+
+ if (fw_feature_is("enabled", "fw-bcctrl-serialized", np))
+ security_ftr_set(SEC_FTR_BCCTRL_SERIALISED);
+
+ if (fw_feature_is("enabled", "inst-l1d-flush-ori30,30,0", np))
+ security_ftr_set(SEC_FTR_L1D_FLUSH_ORI30);
+
+ if (fw_feature_is("enabled", "inst-l1d-flush-trig2", np))
+ security_ftr_set(SEC_FTR_L1D_FLUSH_TRIG2);
+
+ if (fw_feature_is("enabled", "fw-l1d-thread-split", np))
+ security_ftr_set(SEC_FTR_L1D_THREAD_PRIV);
+
+ if (fw_feature_is("enabled", "fw-count-cache-disabled", np))
+ security_ftr_set(SEC_FTR_COUNT_CACHE_DISABLED);
+
+ /*
+ * The features below are enabled by default, so we instead look to see
+ * if firmware has *disabled* them, and clear them if so.
+ */
+ if (fw_feature_is("disabled", "speculation-policy-favor-security", np))
+ security_ftr_clear(SEC_FTR_FAVOUR_SECURITY);
+
+ if (fw_feature_is("disabled", "needs-l1d-flush-msr-pr-0-to-1", np))
+ security_ftr_clear(SEC_FTR_L1D_FLUSH_PR);
+
+ if (fw_feature_is("disabled", "needs-l1d-flush-msr-hv-1-to-0", np))
+ security_ftr_clear(SEC_FTR_L1D_FLUSH_HV);
+
+ if (fw_feature_is("disabled", "needs-spec-barrier-for-bound-checks", np))
+ security_ftr_clear(SEC_FTR_BNDS_CHK_SPEC_BAR);
+}
+
static void pnv_setup_rfi_flush(void)
{
struct device_node *np, *fw_features;
enum l1d_flush_type type;
- int enable;
+ bool enable;
/* Default to fallback in case fw-features are not available */
type = L1D_FLUSH_FALLBACK;
- enable = 1;
np = of_find_node_by_name(NULL, "ibm,opal");
fw_features = of_get_child_by_name(np, "fw-features");
of_node_put(np);
if (fw_features) {
- np = of_get_child_by_name(fw_features, "inst-l1d-flush-trig2");
- if (np && of_property_read_bool(np, "enabled"))
+ init_fw_feat_flags(fw_features);
+ of_node_put(fw_features);
+
+ if (security_ftr_enabled(SEC_FTR_L1D_FLUSH_TRIG2))
type = L1D_FLUSH_MTTRIG;
- of_node_put(np);
-
- np = of_get_child_by_name(fw_features, "inst-l1d-flush-ori30,30,0");
- if (np && of_property_read_bool(np, "enabled"))
+ if (security_ftr_enabled(SEC_FTR_L1D_FLUSH_ORI30))
type = L1D_FLUSH_ORI;
-
- of_node_put(np);
-
- /* Enable unless firmware says NOT to */
- enable = 2;
- np = of_get_child_by_name(fw_features, "needs-l1d-flush-msr-hv-1-to-0");
- if (np && of_property_read_bool(np, "disabled"))
- enable--;
-
- of_node_put(np);
-
- np = of_get_child_by_name(fw_features, "needs-l1d-flush-msr-pr-0-to-1");
- if (np && of_property_read_bool(np, "disabled"))
- enable--;
-
- of_node_put(np);
- of_node_put(fw_features);
}
- setup_rfi_flush(type, enable > 0);
+ enable = security_ftr_enabled(SEC_FTR_FAVOUR_SECURITY) && \
+ (security_ftr_enabled(SEC_FTR_L1D_FLUSH_PR) || \
+ security_ftr_enabled(SEC_FTR_L1D_FLUSH_HV));
+
+ setup_rfi_flush(type, enable);
}
static void __init pnv_setup_arch(void)
@@ -91,6 +130,7 @@ static void __init pnv_setup_arch(void)
set_arch_panic_timeout(10, ARCH_PANIC_TIMEOUT);
pnv_setup_rfi_flush();
+ setup_stf_barrier();
/* Initialize SMP */
pnv_smp_init();
diff --git a/arch/powerpc/platforms/pseries/mobility.c b/arch/powerpc/platforms/pseries/mobility.c
index 6a5e746..3784a7a 100644
--- a/arch/powerpc/platforms/pseries/mobility.c
+++ b/arch/powerpc/platforms/pseries/mobility.c
@@ -314,6 +314,9 @@ void post_mobility_fixup(void)
printk(KERN_ERR "Post-mobility device tree update "
"failed: %d\n", rc);
+ /* Possibly switch to a new RFI flush type */
+ pseries_setup_rfi_flush();
+
return;
}
diff --git a/arch/powerpc/platforms/pseries/pseries.h b/arch/powerpc/platforms/pseries/pseries.h
index b1be7b7..62ff57c 100644
--- a/arch/powerpc/platforms/pseries/pseries.h
+++ b/arch/powerpc/platforms/pseries/pseries.h
@@ -79,4 +79,6 @@ extern struct pci_controller_ops pseries_pci_controller_ops;
unsigned long pseries_memory_block_size(void);
+void pseries_setup_rfi_flush(void);
+
#endif /* _PSERIES_PSERIES_H */
diff --git a/arch/powerpc/platforms/pseries/ras.c b/arch/powerpc/platforms/pseries/ras.c
index 904a677..8799d8a 100644
--- a/arch/powerpc/platforms/pseries/ras.c
+++ b/arch/powerpc/platforms/pseries/ras.c
@@ -346,7 +346,7 @@ static struct rtas_error_log *fwnmi_get_errinfo(struct pt_regs *regs)
}
savep = __va(regs->gpr[3]);
- regs->gpr[3] = savep[0]; /* restore original r3 */
+ regs->gpr[3] = be64_to_cpu(savep[0]); /* restore original r3 */
/* If it isn't an extended log we can use the per cpu 64bit buffer */
h = (struct rtas_error_log *)&savep[1];
@@ -357,7 +357,7 @@ static struct rtas_error_log *fwnmi_get_errinfo(struct pt_regs *regs)
int len, error_log_length;
error_log_length = 8 + rtas_error_extended_log_length(h);
- len = max_t(int, error_log_length, RTAS_ERROR_LOG_MAX);
+ len = min_t(int, error_log_length, RTAS_ERROR_LOG_MAX);
memset(global_mce_data_buf, 0, RTAS_ERROR_LOG_MAX);
memcpy(global_mce_data_buf, h, len);
errhdr = (struct rtas_error_log *)global_mce_data_buf;
diff --git a/arch/powerpc/platforms/pseries/setup.c b/arch/powerpc/platforms/pseries/setup.c
index 1845fc6..91ade77 100644
--- a/arch/powerpc/platforms/pseries/setup.c
+++ b/arch/powerpc/platforms/pseries/setup.c
@@ -66,6 +66,7 @@
#include <asm/reg.h>
#include <asm/plpar_wrappers.h>
#include <asm/kexec.h>
+#include <asm/security_features.h>
#include "pseries.h"
@@ -450,35 +451,78 @@ static void __init find_and_init_phbs(void)
of_pci_check_probe_only();
}
-static void pseries_setup_rfi_flush(void)
+static void init_cpu_char_feature_flags(struct h_cpu_char_result *result)
+{
+ /*
+ * The features below are disabled by default, so we instead look to see
+ * if firmware has *enabled* them, and set them if so.
+ */
+ if (result->character & H_CPU_CHAR_SPEC_BAR_ORI31)
+ security_ftr_set(SEC_FTR_SPEC_BAR_ORI31);
+
+ if (result->character & H_CPU_CHAR_BCCTRL_SERIALISED)
+ security_ftr_set(SEC_FTR_BCCTRL_SERIALISED);
+
+ if (result->character & H_CPU_CHAR_L1D_FLUSH_ORI30)
+ security_ftr_set(SEC_FTR_L1D_FLUSH_ORI30);
+
+ if (result->character & H_CPU_CHAR_L1D_FLUSH_TRIG2)
+ security_ftr_set(SEC_FTR_L1D_FLUSH_TRIG2);
+
+ if (result->character & H_CPU_CHAR_L1D_THREAD_PRIV)
+ security_ftr_set(SEC_FTR_L1D_THREAD_PRIV);
+
+ if (result->character & H_CPU_CHAR_COUNT_CACHE_DISABLED)
+ security_ftr_set(SEC_FTR_COUNT_CACHE_DISABLED);
+
+ /*
+ * The features below are enabled by default, so we instead look to see
+ * if firmware has *disabled* them, and clear them if so.
+ */
+ if (!(result->behaviour & H_CPU_BEHAV_FAVOUR_SECURITY))
+ security_ftr_clear(SEC_FTR_FAVOUR_SECURITY);
+
+ if (!(result->behaviour & H_CPU_BEHAV_L1D_FLUSH_PR))
+ security_ftr_clear(SEC_FTR_L1D_FLUSH_PR);
+
+ if (!(result->behaviour & H_CPU_BEHAV_BNDS_CHK_SPEC_BAR))
+ security_ftr_clear(SEC_FTR_BNDS_CHK_SPEC_BAR);
+}
+
+void pseries_setup_rfi_flush(void)
{
struct h_cpu_char_result result;
enum l1d_flush_type types;
bool enable;
long rc;
- /* Enable by default */
- enable = true;
+ /*
+ * Set features to the defaults assumed by init_cpu_char_feature_flags()
+ * so it can set/clear again any features that might have changed after
+ * migration, and in case the hypercall fails and it is not even called.
+ */
+ powerpc_security_features = SEC_FTR_DEFAULT;
rc = plpar_get_cpu_characteristics(&result);
- if (rc == H_SUCCESS) {
- types = L1D_FLUSH_NONE;
+ if (rc == H_SUCCESS)
+ init_cpu_char_feature_flags(&result);
- if (result.character & H_CPU_CHAR_L1D_FLUSH_TRIG2)
- types |= L1D_FLUSH_MTTRIG;
- if (result.character & H_CPU_CHAR_L1D_FLUSH_ORI30)
- types |= L1D_FLUSH_ORI;
+ /*
+ * We're the guest so this doesn't apply to us, clear it to simplify
+ * handling of it elsewhere.
+ */
+ security_ftr_clear(SEC_FTR_L1D_FLUSH_HV);
- /* Use fallback if nothing set in hcall */
- if (types == L1D_FLUSH_NONE)
- types = L1D_FLUSH_FALLBACK;
+ types = L1D_FLUSH_FALLBACK;
- if (!(result.behaviour & H_CPU_BEHAV_L1D_FLUSH_PR))
- enable = false;
- } else {
- /* Default to fallback if case hcall is not available */
- types = L1D_FLUSH_FALLBACK;
- }
+ if (security_ftr_enabled(SEC_FTR_L1D_FLUSH_TRIG2))
+ types |= L1D_FLUSH_MTTRIG;
+
+ if (security_ftr_enabled(SEC_FTR_L1D_FLUSH_ORI30))
+ types |= L1D_FLUSH_ORI;
+
+ enable = security_ftr_enabled(SEC_FTR_FAVOUR_SECURITY) && \
+ security_ftr_enabled(SEC_FTR_L1D_FLUSH_PR);
setup_rfi_flush(types, enable);
}
@@ -501,6 +545,7 @@ static void __init pSeries_setup_arch(void)
fwnmi_init();
pseries_setup_rfi_flush();
+ setup_stf_barrier();
/* By default, only probe PCI (can be overridden by rtas_pci) */
pci_add_flags(PCI_PROBE_ONLY);
diff --git a/arch/powerpc/sysdev/mpic.c b/arch/powerpc/sysdev/mpic.c
index b9aac95..f37567e 100644
--- a/arch/powerpc/sysdev/mpic.c
+++ b/arch/powerpc/sysdev/mpic.c
@@ -626,7 +626,7 @@ static inline u32 mpic_physmask(u32 cpumask)
int i;
u32 mask = 0;
- for (i = 0; i < min(32, NR_CPUS); ++i, cpumask >>= 1)
+ for (i = 0; i < min(32, NR_CPUS) && cpu_possible(i); ++i, cpumask >>= 1)
mask |= (cpumask & 1) << get_hard_smp_processor_id(i);
return mask;
}
diff --git a/arch/powerpc/sysdev/mpic_msgr.c b/arch/powerpc/sysdev/mpic_msgr.c
index db2286b..47fb336 100644
--- a/arch/powerpc/sysdev/mpic_msgr.c
+++ b/arch/powerpc/sysdev/mpic_msgr.c
@@ -196,7 +196,7 @@ static int mpic_msgr_probe(struct platform_device *dev)
/* IO map the message register block. */
of_address_to_resource(np, 0, &rsrc);
- msgr_block_addr = ioremap(rsrc.start, rsrc.end - rsrc.start);
+ msgr_block_addr = ioremap(rsrc.start, resource_size(&rsrc));
if (!msgr_block_addr) {
dev_err(&dev->dev, "Failed to iomap MPIC message registers");
return -EFAULT;
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 9aa0d04..1c4a595 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -118,6 +118,7 @@
select GENERIC_CLOCKEVENTS
select GENERIC_CPU_AUTOPROBE
select GENERIC_CPU_DEVICES if !SMP
+ select GENERIC_CPU_VULNERABILITIES
select GENERIC_FIND_FIRST_BIT
select GENERIC_SMP_IDLE_THREAD
select GENERIC_TIME_VSYSCALL
@@ -704,6 +705,51 @@
If unsure, say Y.
+config KERNEL_NOBP
+ def_bool n
+ prompt "Enable modified branch prediction for the kernel by default"
+ help
+ If this option is selected the kernel will switch to a modified
+ branch prediction mode if the firmware interface is available.
+ The modified branch prediction mode improves the behaviour in
+ regard to speculative execution.
+
+ With the option enabled the kernel parameter "nobp=0" or "nospec"
+ can be used to run the kernel in the normal branch prediction mode.
+
+ With the option disabled the modified branch prediction mode is
+ enabled with the "nobp=1" kernel parameter.
+
+ If unsure, say N.
+
+config EXPOLINE
+ def_bool n
+ prompt "Avoid speculative indirect branches in the kernel"
+ help
+ Compile the kernel with the expoline compiler options to guard
+ against kernel-to-user data leaks by avoiding speculative indirect
+ branches.
+ Requires a compiler with -mindirect-branch=thunk support for full
+ protection. The kernel may run slower.
+
+ If unsure, say N.
+
+choice
+ prompt "Expoline default"
+ depends on EXPOLINE
+ default EXPOLINE_FULL
+
+config EXPOLINE_OFF
+ bool "spectre_v2=off"
+
+config EXPOLINE_AUTO
+ bool "spectre_v2=auto"
+
+config EXPOLINE_FULL
+ bool "spectre_v2=on"
+
+endchoice
+
endmenu
menu "Power Management"
@@ -753,6 +799,7 @@
config SHARED_KERNEL
bool "VM shared kernel support"
depends on !JUMP_LABEL
+ depends on !ALTERNATIVES
help
Select this option, if you want to share the text segment of the
Linux kernel between different VM guests. This reduces memory
diff --git a/arch/s390/Makefile b/arch/s390/Makefile
index 54e0052..bef67c0 100644
--- a/arch/s390/Makefile
+++ b/arch/s390/Makefile
@@ -79,6 +79,16 @@
cflags-$(CONFIG_WARN_DYNAMIC_STACK) += -mwarn-dynamicstack
endif
+ifdef CONFIG_EXPOLINE
+ ifeq ($(call cc-option-yn,$(CC_FLAGS_MARCH) -mindirect-branch=thunk),y)
+ CC_FLAGS_EXPOLINE := -mindirect-branch=thunk
+ CC_FLAGS_EXPOLINE += -mfunction-return=thunk
+ CC_FLAGS_EXPOLINE += -mindirect-branch-table
+ export CC_FLAGS_EXPOLINE
+ cflags-y += $(CC_FLAGS_EXPOLINE) -DCC_USING_EXPOLINE
+ endif
+endif
+
ifdef CONFIG_FUNCTION_TRACER
# make use of hotpatch feature if the compiler supports it
cc_hotpatch := -mhotpatch=0,3
diff --git a/arch/s390/appldata/appldata_os.c b/arch/s390/appldata/appldata_os.c
index 69b23b2..731dc7e 100644
--- a/arch/s390/appldata/appldata_os.c
+++ b/arch/s390/appldata/appldata_os.c
@@ -17,15 +17,12 @@
#include <linux/kernel_stat.h>
#include <linux/netdevice.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <asm/appldata.h>
#include <asm/smp.h>
#include "appldata.h"
-
-#define LOAD_INT(x) ((x) >> FSHIFT)
-#define LOAD_FRAC(x) LOAD_INT(((x) & (FIXED_1-1)) * 100)
-
/*
* OS data
*
@@ -113,21 +110,21 @@ static void appldata_get_os_data(void *data)
j = 0;
for_each_online_cpu(i) {
os_data->os_cpu[j].per_cpu_user =
- cputime_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_USER]);
+ nsecs_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_USER]);
os_data->os_cpu[j].per_cpu_nice =
- cputime_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_NICE]);
+ nsecs_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_NICE]);
os_data->os_cpu[j].per_cpu_system =
- cputime_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_SYSTEM]);
+ nsecs_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_SYSTEM]);
os_data->os_cpu[j].per_cpu_idle =
- cputime_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_IDLE]);
+ nsecs_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_IDLE]);
os_data->os_cpu[j].per_cpu_irq =
- cputime_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_IRQ]);
+ nsecs_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_IRQ]);
os_data->os_cpu[j].per_cpu_softirq =
- cputime_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_SOFTIRQ]);
+ nsecs_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_SOFTIRQ]);
os_data->os_cpu[j].per_cpu_iowait =
- cputime_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_IOWAIT]);
+ nsecs_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_IOWAIT]);
os_data->os_cpu[j].per_cpu_steal =
- cputime_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_STEAL]);
+ nsecs_to_jiffies(kcpustat_cpu(i).cpustat[CPUTIME_STEAL]);
os_data->os_cpu[j].cpu_id = i;
j++;
}
diff --git a/arch/s390/crypto/crc32be-vx.S b/arch/s390/crypto/crc32be-vx.S
index 8013989..096affb 100644
--- a/arch/s390/crypto/crc32be-vx.S
+++ b/arch/s390/crypto/crc32be-vx.S
@@ -12,6 +12,7 @@
*/
#include <linux/linkage.h>
+#include <asm/nospec-insn.h>
#include <asm/vx-insn.h>
/* Vector register range containing CRC-32 constants */
@@ -66,6 +67,8 @@
.previous
+ GEN_BR_THUNK %r14
+
.text
/*
* The CRC-32 function(s) use these calling conventions:
@@ -202,6 +205,6 @@
.Ldone:
VLGVF %r2,%v2,3
- br %r14
+ BR_EX %r14
.previous
diff --git a/arch/s390/crypto/crc32le-vx.S b/arch/s390/crypto/crc32le-vx.S
index 17f2504..8dc98c1 100644
--- a/arch/s390/crypto/crc32le-vx.S
+++ b/arch/s390/crypto/crc32le-vx.S
@@ -13,6 +13,7 @@
*/
#include <linux/linkage.h>
+#include <asm/nospec-insn.h>
#include <asm/vx-insn.h>
/* Vector register range containing CRC-32 constants */
@@ -75,6 +76,7 @@
.previous
+ GEN_BR_THUNK %r14
.text
@@ -263,6 +265,6 @@
.Ldone:
VLGVF %r2,%v2,2
- br %r14
+ BR_EX %r14
.previous
diff --git a/arch/s390/include/asm/alternative-asm.h b/arch/s390/include/asm/alternative-asm.h
new file mode 100644
index 0000000..955d620
--- /dev/null
+++ b/arch/s390/include/asm/alternative-asm.h
@@ -0,0 +1,108 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_S390_ALTERNATIVE_ASM_H
+#define _ASM_S390_ALTERNATIVE_ASM_H
+
+#ifdef __ASSEMBLY__
+
+/*
+ * Check the length of an instruction sequence. The length may not be larger
+ * than 254 bytes and it has to be divisible by 2.
+ */
+.macro alt_len_check start,end
+ .if ( \end - \start ) > 254
+ .error "cpu alternatives does not support instructions blocks > 254 bytes\n"
+ .endif
+ .if ( \end - \start ) % 2
+ .error "cpu alternatives instructions length is odd\n"
+ .endif
+.endm
+
+/*
+ * Issue one struct alt_instr descriptor entry (need to put it into
+ * the section .altinstructions, see below). This entry contains
+ * enough information for the alternatives patching code to patch an
+ * instruction. See apply_alternatives().
+ */
+.macro alt_entry orig_start, orig_end, alt_start, alt_end, feature
+ .long \orig_start - .
+ .long \alt_start - .
+ .word \feature
+ .byte \orig_end - \orig_start
+ .byte \alt_end - \alt_start
+.endm
+
+/*
+ * Fill up @bytes with nops. The macro emits 6-byte nop instructions
+ * for the bulk of the area, possibly followed by a 4-byte and/or
+ * a 2-byte nop if the size of the area is not divisible by 6.
+ */
+.macro alt_pad_fill bytes
+ .fill ( \bytes ) / 6, 6, 0xc0040000
+ .fill ( \bytes ) % 6 / 4, 4, 0x47000000
+ .fill ( \bytes ) % 6 % 4 / 2, 2, 0x0700
+.endm
+
+/*
+ * Fill up @bytes with nops. If the number of bytes is larger
+ * than 6, emit a jg instruction to branch over all nops, then
+ * fill an area of size (@bytes - 6) with nop instructions.
+ */
+.macro alt_pad bytes
+ .if ( \bytes > 0 )
+ .if ( \bytes > 6 )
+ jg . + \bytes
+ alt_pad_fill \bytes - 6
+ .else
+ alt_pad_fill \bytes
+ .endif
+ .endif
+.endm
+
+/*
+ * Define an alternative between two instructions. If @feature is
+ * present, early code in apply_alternatives() replaces @oldinstr with
+ * @newinstr. ".skip" directive takes care of proper instruction padding
+ * in case @newinstr is longer than @oldinstr.
+ */
+.macro ALTERNATIVE oldinstr, newinstr, feature
+ .pushsection .altinstr_replacement,"ax"
+770: \newinstr
+771: .popsection
+772: \oldinstr
+773: alt_len_check 770b, 771b
+ alt_len_check 772b, 773b
+ alt_pad ( ( 771b - 770b ) - ( 773b - 772b ) )
+774: .pushsection .altinstructions,"a"
+ alt_entry 772b, 774b, 770b, 771b, \feature
+ .popsection
+.endm
+
+/*
+ * Define an alternative between two instructions. If @feature is
+ * present, early code in apply_alternatives() replaces @oldinstr with
+ * @newinstr. ".skip" directive takes care of proper instruction padding
+ * in case @newinstr is longer than @oldinstr.
+ */
+.macro ALTERNATIVE_2 oldinstr, newinstr1, feature1, newinstr2, feature2
+ .pushsection .altinstr_replacement,"ax"
+770: \newinstr1
+771: \newinstr2
+772: .popsection
+773: \oldinstr
+774: alt_len_check 770b, 771b
+ alt_len_check 771b, 772b
+ alt_len_check 773b, 774b
+ .if ( 771b - 770b > 772b - 771b )
+ alt_pad ( ( 771b - 770b ) - ( 774b - 773b ) )
+ .else
+ alt_pad ( ( 772b - 771b ) - ( 774b - 773b ) )
+ .endif
+775: .pushsection .altinstructions,"a"
+ alt_entry 773b, 775b, 770b, 771b,\feature1
+ alt_entry 773b, 775b, 771b, 772b,\feature2
+ .popsection
+.endm
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_S390_ALTERNATIVE_ASM_H */
diff --git a/arch/s390/include/asm/alternative.h b/arch/s390/include/asm/alternative.h
new file mode 100644
index 0000000..a720020
--- /dev/null
+++ b/arch/s390/include/asm/alternative.h
@@ -0,0 +1,149 @@
+#ifndef _ASM_S390_ALTERNATIVE_H
+#define _ASM_S390_ALTERNATIVE_H
+
+#ifndef __ASSEMBLY__
+
+#include <linux/types.h>
+#include <linux/stddef.h>
+#include <linux/stringify.h>
+
+struct alt_instr {
+ s32 instr_offset; /* original instruction */
+ s32 repl_offset; /* offset to replacement instruction */
+ u16 facility; /* facility bit set for replacement */
+ u8 instrlen; /* length of original instruction */
+ u8 replacementlen; /* length of new instruction */
+} __packed;
+
+void apply_alternative_instructions(void);
+void apply_alternatives(struct alt_instr *start, struct alt_instr *end);
+
+/*
+ * |661: |662: |6620 |663:
+ * +-----------+---------------------+
+ * | oldinstr | oldinstr_padding |
+ * | +----------+----------+
+ * | | | |
+ * | | >6 bytes |6/4/2 nops|
+ * | |6 bytes jg----------->
+ * +-----------+---------------------+
+ * ^^ static padding ^^
+ *
+ * .altinstr_replacement section
+ * +---------------------+-----------+
+ * |6641: |6651:
+ * | alternative instr 1 |
+ * +-----------+---------+- - - - - -+
+ * |6642: |6652: |
+ * | alternative instr 2 | padding
+ * +---------------------+- - - - - -+
+ * ^ runtime ^
+ *
+ * .altinstructions section
+ * +---------------------------------+
+ * | alt_instr entries for each |
+ * | alternative instr |
+ * +---------------------------------+
+ */
+
+#define b_altinstr(num) "664"#num
+#define e_altinstr(num) "665"#num
+
+#define e_oldinstr_pad_end "663"
+#define oldinstr_len "662b-661b"
+#define oldinstr_total_len e_oldinstr_pad_end"b-661b"
+#define altinstr_len(num) e_altinstr(num)"b-"b_altinstr(num)"b"
+#define oldinstr_pad_len(num) \
+ "-(((" altinstr_len(num) ")-(" oldinstr_len ")) > 0) * " \
+ "((" altinstr_len(num) ")-(" oldinstr_len "))"
+
+#define INSTR_LEN_SANITY_CHECK(len) \
+ ".if " len " > 254\n" \
+ "\t.error \"cpu alternatives does not support instructions " \
+ "blocks > 254 bytes\"\n" \
+ ".endif\n" \
+ ".if (" len ") %% 2\n" \
+ "\t.error \"cpu alternatives instructions length is odd\"\n" \
+ ".endif\n"
+
+#define OLDINSTR_PADDING(oldinstr, num) \
+ ".if " oldinstr_pad_len(num) " > 6\n" \
+ "\tjg " e_oldinstr_pad_end "f\n" \
+ "6620:\n" \
+ "\t.fill (" oldinstr_pad_len(num) " - (6620b-662b)) / 2, 2, 0x0700\n" \
+ ".else\n" \
+ "\t.fill " oldinstr_pad_len(num) " / 6, 6, 0xc0040000\n" \
+ "\t.fill " oldinstr_pad_len(num) " %% 6 / 4, 4, 0x47000000\n" \
+ "\t.fill " oldinstr_pad_len(num) " %% 6 %% 4 / 2, 2, 0x0700\n" \
+ ".endif\n"
+
+#define OLDINSTR(oldinstr, num) \
+ "661:\n\t" oldinstr "\n662:\n" \
+ OLDINSTR_PADDING(oldinstr, num) \
+ e_oldinstr_pad_end ":\n" \
+ INSTR_LEN_SANITY_CHECK(oldinstr_len)
+
+#define OLDINSTR_2(oldinstr, num1, num2) \
+ "661:\n\t" oldinstr "\n662:\n" \
+ ".if " altinstr_len(num1) " < " altinstr_len(num2) "\n" \
+ OLDINSTR_PADDING(oldinstr, num2) \
+ ".else\n" \
+ OLDINSTR_PADDING(oldinstr, num1) \
+ ".endif\n" \
+ e_oldinstr_pad_end ":\n" \
+ INSTR_LEN_SANITY_CHECK(oldinstr_len)
+
+#define ALTINSTR_ENTRY(facility, num) \
+ "\t.long 661b - .\n" /* old instruction */ \
+ "\t.long " b_altinstr(num)"b - .\n" /* alt instruction */ \
+ "\t.word " __stringify(facility) "\n" /* facility bit */ \
+ "\t.byte " oldinstr_total_len "\n" /* source len */ \
+ "\t.byte " altinstr_len(num) "\n" /* alt instruction len */
+
+#define ALTINSTR_REPLACEMENT(altinstr, num) /* replacement */ \
+ b_altinstr(num)":\n\t" altinstr "\n" e_altinstr(num) ":\n" \
+ INSTR_LEN_SANITY_CHECK(altinstr_len(num))
+
+/* alternative assembly primitive: */
+#define ALTERNATIVE(oldinstr, altinstr, facility) \
+ ".pushsection .altinstr_replacement, \"ax\"\n" \
+ ALTINSTR_REPLACEMENT(altinstr, 1) \
+ ".popsection\n" \
+ OLDINSTR(oldinstr, 1) \
+ ".pushsection .altinstructions,\"a\"\n" \
+ ALTINSTR_ENTRY(facility, 1) \
+ ".popsection\n"
+
+#define ALTERNATIVE_2(oldinstr, altinstr1, facility1, altinstr2, facility2)\
+ ".pushsection .altinstr_replacement, \"ax\"\n" \
+ ALTINSTR_REPLACEMENT(altinstr1, 1) \
+ ALTINSTR_REPLACEMENT(altinstr2, 2) \
+ ".popsection\n" \
+ OLDINSTR_2(oldinstr, 1, 2) \
+ ".pushsection .altinstructions,\"a\"\n" \
+ ALTINSTR_ENTRY(facility1, 1) \
+ ALTINSTR_ENTRY(facility2, 2) \
+ ".popsection\n"
+
+/*
+ * Alternative instructions for different CPU types or capabilities.
+ *
+ * This allows to use optimized instructions even on generic binary
+ * kernels.
+ *
+ * oldinstr is padded with jump and nops at compile time if altinstr is
+ * longer. altinstr is padded with jump and nops at run-time during patching.
+ *
+ * For non barrier like inlines please define new variants
+ * without volatile and memory clobber.
+ */
+#define alternative(oldinstr, altinstr, facility) \
+ asm volatile(ALTERNATIVE(oldinstr, altinstr, facility) : : : "memory")
+
+#define alternative_2(oldinstr, altinstr1, facility1, altinstr2, facility2) \
+ asm volatile(ALTERNATIVE_2(oldinstr, altinstr1, facility1, \
+ altinstr2, facility2) ::: "memory")
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_S390_ALTERNATIVE_H */
diff --git a/arch/s390/include/asm/barrier.h b/arch/s390/include/asm/barrier.h
index 5c8db3c..03b2e5b 100644
--- a/arch/s390/include/asm/barrier.h
+++ b/arch/s390/include/asm/barrier.h
@@ -48,6 +48,30 @@ do { \
#define __smp_mb__before_atomic() barrier()
#define __smp_mb__after_atomic() barrier()
+/**
+ * array_index_mask_nospec - generate a mask for array_idx() that is
+ * ~0UL when the bounds check succeeds and 0 otherwise
+ * @index: array element index
+ * @size: number of elements in array
+ */
+#define array_index_mask_nospec array_index_mask_nospec
+static inline unsigned long array_index_mask_nospec(unsigned long index,
+ unsigned long size)
+{
+ unsigned long mask;
+
+ if (__builtin_constant_p(size) && size > 0) {
+ asm(" clgr %2,%1\n"
+ " slbgr %0,%0\n"
+ :"=d" (mask) : "d" (size-1), "d" (index) :"cc");
+ return mask;
+ }
+ asm(" clgr %1,%2\n"
+ " slbgr %0,%0\n"
+ :"=d" (mask) : "d" (size), "d" (index) :"cc");
+ return ~mask;
+}
+
#include <asm-generic/barrier.h>
#endif /* __ASM_BARRIER_H */
diff --git a/arch/s390/include/asm/cpu_mf.h b/arch/s390/include/asm/cpu_mf.h
index 0351647..ee64e62 100644
--- a/arch/s390/include/asm/cpu_mf.h
+++ b/arch/s390/include/asm/cpu_mf.h
@@ -113,7 +113,7 @@ struct hws_basic_entry {
struct hws_diag_entry {
unsigned int def:16; /* 0-15 Data Entry Format */
- unsigned int R:14; /* 16-19 and 20-30 reserved */
+ unsigned int R:15; /* 16-19 and 20-30 reserved */
unsigned int I:1; /* 31 entry valid or invalid */
u8 data[]; /* Machine-dependent sample data */
} __packed;
@@ -129,7 +129,9 @@ struct hws_trailer_entry {
unsigned int f:1; /* 0 - Block Full Indicator */
unsigned int a:1; /* 1 - Alert request control */
unsigned int t:1; /* 2 - Timestamp format */
- unsigned long long:61; /* 3 - 63: Reserved */
+ unsigned int :29; /* 3 - 31: Reserved */
+ unsigned int bsdes:16; /* 32-47: size of basic SDE */
+ unsigned int dsdes:16; /* 48-63: size of diagnostic SDE */
};
unsigned long long flags; /* 0 - 63: All indicators */
};
diff --git a/arch/s390/include/asm/facility.h b/arch/s390/include/asm/facility.h
index 09b406d..5811e78 100644
--- a/arch/s390/include/asm/facility.h
+++ b/arch/s390/include/asm/facility.h
@@ -15,7 +15,25 @@
#include <linux/preempt.h>
#include <asm/lowcore.h>
-#define MAX_FACILITY_BIT (256*8) /* stfle_fac_list has 256 bytes */
+#define MAX_FACILITY_BIT (sizeof(((struct lowcore *)0)->stfle_fac_list) * 8)
+
+static inline void __set_facility(unsigned long nr, void *facilities)
+{
+ unsigned char *ptr = (unsigned char *) facilities;
+
+ if (nr >= MAX_FACILITY_BIT)
+ return;
+ ptr[nr >> 3] |= 0x80 >> (nr & 7);
+}
+
+static inline void __clear_facility(unsigned long nr, void *facilities)
+{
+ unsigned char *ptr = (unsigned char *) facilities;
+
+ if (nr >= MAX_FACILITY_BIT)
+ return;
+ ptr[nr >> 3] &= ~(0x80 >> (nr & 7));
+}
static inline int __test_facility(unsigned long nr, void *facilities)
{
diff --git a/arch/s390/include/asm/futex.h b/arch/s390/include/asm/futex.h
index a4811aa..8f8eec9e 100644
--- a/arch/s390/include/asm/futex.h
+++ b/arch/s390/include/asm/futex.h
@@ -21,17 +21,12 @@
: "0" (-EFAULT), "d" (oparg), "a" (uaddr), \
"m" (*uaddr) : "cc");
-static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval,
+ u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, newval, ret;
load_kernel_asce();
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
pagefault_disable();
switch (op) {
@@ -60,17 +55,9 @@ static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
}
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/s390/include/asm/kvm_host.h b/arch/s390/include/asm/kvm_host.h
index a41faf3..5792590 100644
--- a/arch/s390/include/asm/kvm_host.h
+++ b/arch/s390/include/asm/kvm_host.h
@@ -181,7 +181,8 @@ struct kvm_s390_sie_block {
__u16 ipa; /* 0x0056 */
__u32 ipb; /* 0x0058 */
__u32 scaoh; /* 0x005c */
- __u8 reserved60; /* 0x0060 */
+#define FPF_BPBC 0x20
+ __u8 fpf; /* 0x0060 */
__u8 ecb; /* 0x0061 */
__u8 ecb2; /* 0x0062 */
#define ECB3_AES 0x04
diff --git a/arch/s390/include/asm/lowcore.h b/arch/s390/include/asm/lowcore.h
index 7b93b78..ad4e0ce 100644
--- a/arch/s390/include/asm/lowcore.h
+++ b/arch/s390/include/asm/lowcore.h
@@ -135,7 +135,9 @@ struct lowcore {
/* Per cpu primary space access list */
__u32 paste[16]; /* 0x0400 */
- __u8 pad_0x04c0[0x0e00-0x0440]; /* 0x0440 */
+ /* br %r1 trampoline */
+ __u16 br_r1_trampoline; /* 0x0440 */
+ __u8 pad_0x0442[0x0e00-0x0442]; /* 0x0442 */
/*
* 0xe00 contains the address of the IPL Parameter Information
@@ -150,7 +152,8 @@ struct lowcore {
__u8 pad_0x0e20[0x0f00-0x0e20]; /* 0x0e20 */
/* Extended facility list */
- __u64 stfle_fac_list[32]; /* 0x0f00 */
+ __u64 stfle_fac_list[16]; /* 0x0f00 */
+ __u64 alt_stfle_fac_list[16]; /* 0x0f80 */
__u8 pad_0x1000[0x11b0-0x1000]; /* 0x1000 */
/* Pointer to vector register save area */
diff --git a/arch/s390/include/asm/nospec-branch.h b/arch/s390/include/asm/nospec-branch.h
new file mode 100644
index 0000000..b4bd8c4
--- /dev/null
+++ b/arch/s390/include/asm/nospec-branch.h
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_S390_EXPOLINE_H
+#define _ASM_S390_EXPOLINE_H
+
+#ifndef __ASSEMBLY__
+
+#include <linux/types.h>
+
+extern int nospec_disable;
+
+void nospec_init_branches(void);
+void nospec_auto_detect(void);
+void nospec_revert(s32 *start, s32 *end);
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_S390_EXPOLINE_H */
diff --git a/arch/s390/include/asm/nospec-insn.h b/arch/s390/include/asm/nospec-insn.h
new file mode 100644
index 0000000..9a56e73
--- /dev/null
+++ b/arch/s390/include/asm/nospec-insn.h
@@ -0,0 +1,195 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_S390_NOSPEC_ASM_H
+#define _ASM_S390_NOSPEC_ASM_H
+
+#include <asm/alternative-asm.h>
+#include <asm/asm-offsets.h>
+
+#ifdef __ASSEMBLY__
+
+#ifdef CONFIG_EXPOLINE
+
+_LC_BR_R1 = __LC_BR_R1
+
+/*
+ * The expoline macros are used to create thunks in the same format
+ * as gcc generates them. The 'comdat' section flag makes sure that
+ * the various thunks are merged into a single copy.
+ */
+ .macro __THUNK_PROLOG_NAME name
+ .pushsection .text.\name,"axG",@progbits,\name,comdat
+ .globl \name
+ .hidden \name
+ .type \name,@function
+\name:
+ .cfi_startproc
+ .endm
+
+ .macro __THUNK_EPILOG
+ .cfi_endproc
+ .popsection
+ .endm
+
+ .macro __THUNK_PROLOG_BR r1,r2
+ __THUNK_PROLOG_NAME __s390x_indirect_jump_r\r2\()use_r\r1
+ .endm
+
+ .macro __THUNK_PROLOG_BC d0,r1,r2
+ __THUNK_PROLOG_NAME __s390x_indirect_branch_\d0\()_\r2\()use_\r1
+ .endm
+
+ .macro __THUNK_BR r1,r2
+ jg __s390x_indirect_jump_r\r2\()use_r\r1
+ .endm
+
+ .macro __THUNK_BC d0,r1,r2
+ jg __s390x_indirect_branch_\d0\()_\r2\()use_\r1
+ .endm
+
+ .macro __THUNK_BRASL r1,r2,r3
+ brasl \r1,__s390x_indirect_jump_r\r3\()use_r\r2
+ .endm
+
+ .macro __DECODE_RR expand,reg,ruse
+ .set __decode_fail,1
+ .irp r1,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
+ .ifc \reg,%r\r1
+ .irp r2,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
+ .ifc \ruse,%r\r2
+ \expand \r1,\r2
+ .set __decode_fail,0
+ .endif
+ .endr
+ .endif
+ .endr
+ .if __decode_fail == 1
+ .error "__DECODE_RR failed"
+ .endif
+ .endm
+
+ .macro __DECODE_RRR expand,rsave,rtarget,ruse
+ .set __decode_fail,1
+ .irp r1,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
+ .ifc \rsave,%r\r1
+ .irp r2,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
+ .ifc \rtarget,%r\r2
+ .irp r3,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
+ .ifc \ruse,%r\r3
+ \expand \r1,\r2,\r3
+ .set __decode_fail,0
+ .endif
+ .endr
+ .endif
+ .endr
+ .endif
+ .endr
+ .if __decode_fail == 1
+ .error "__DECODE_RRR failed"
+ .endif
+ .endm
+
+ .macro __DECODE_DRR expand,disp,reg,ruse
+ .set __decode_fail,1
+ .irp r1,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
+ .ifc \reg,%r\r1
+ .irp r2,0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
+ .ifc \ruse,%r\r2
+ \expand \disp,\r1,\r2
+ .set __decode_fail,0
+ .endif
+ .endr
+ .endif
+ .endr
+ .if __decode_fail == 1
+ .error "__DECODE_DRR failed"
+ .endif
+ .endm
+
+ .macro __THUNK_EX_BR reg,ruse
+ # Be very careful when adding instructions to this macro!
+ # The ALTERNATIVE replacement code has a .+10 which targets
+ # the "br \reg" after the code has been patched.
+#ifdef CONFIG_HAVE_MARCH_Z10_FEATURES
+ exrl 0,555f
+ j .
+#else
+ .ifc \reg,%r1
+ ALTERNATIVE "ex %r0,_LC_BR_R1", ".insn ril,0xc60000000000,0,.+10", 35
+ j .
+ .else
+ larl \ruse,555f
+ ex 0,0(\ruse)
+ j .
+ .endif
+#endif
+555: br \reg
+ .endm
+
+ .macro __THUNK_EX_BC disp,reg,ruse
+#ifdef CONFIG_HAVE_MARCH_Z10_FEATURES
+ exrl 0,556f
+ j .
+#else
+ larl \ruse,556f
+ ex 0,0(\ruse)
+ j .
+#endif
+556: b \disp(\reg)
+ .endm
+
+ .macro GEN_BR_THUNK reg,ruse=%r1
+ __DECODE_RR __THUNK_PROLOG_BR,\reg,\ruse
+ __THUNK_EX_BR \reg,\ruse
+ __THUNK_EPILOG
+ .endm
+
+ .macro GEN_B_THUNK disp,reg,ruse=%r1
+ __DECODE_DRR __THUNK_PROLOG_BC,\disp,\reg,\ruse
+ __THUNK_EX_BC \disp,\reg,\ruse
+ __THUNK_EPILOG
+ .endm
+
+ .macro BR_EX reg,ruse=%r1
+557: __DECODE_RR __THUNK_BR,\reg,\ruse
+ .pushsection .s390_indirect_branches,"a",@progbits
+ .long 557b-.
+ .popsection
+ .endm
+
+ .macro B_EX disp,reg,ruse=%r1
+558: __DECODE_DRR __THUNK_BC,\disp,\reg,\ruse
+ .pushsection .s390_indirect_branches,"a",@progbits
+ .long 558b-.
+ .popsection
+ .endm
+
+ .macro BASR_EX rsave,rtarget,ruse=%r1
+559: __DECODE_RRR __THUNK_BRASL,\rsave,\rtarget,\ruse
+ .pushsection .s390_indirect_branches,"a",@progbits
+ .long 559b-.
+ .popsection
+ .endm
+
+#else
+ .macro GEN_BR_THUNK reg,ruse=%r1
+ .endm
+
+ .macro GEN_B_THUNK disp,reg,ruse=%r1
+ .endm
+
+ .macro BR_EX reg,ruse=%r1
+ br \reg
+ .endm
+
+ .macro B_EX disp,reg,ruse=%r1
+ b \disp(\reg)
+ .endm
+
+ .macro BASR_EX rsave,rtarget,ruse=%r1
+ basr \rsave,\rtarget
+ .endm
+#endif
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_S390_NOSPEC_ASM_H */
diff --git a/arch/s390/include/asm/processor.h b/arch/s390/include/asm/processor.h
index 6bcbbec..d584212 100644
--- a/arch/s390/include/asm/processor.h
+++ b/arch/s390/include/asm/processor.h
@@ -84,6 +84,7 @@ void cpu_detect_mhz_feature(void);
extern const struct seq_operations cpuinfo_op;
extern int sysctl_ieee_emulation_warnings;
extern void execve_tail(void);
+extern void __bpon(void);
/*
* User space process size: 2GB for 31 bit, 4TB or 8PT for 64 bit.
@@ -359,6 +360,9 @@ extern void memcpy_absolute(void *, void *, size_t);
memcpy_absolute(&(dest), &__tmp, sizeof(__tmp)); \
}
+extern int s390_isolate_bp(void);
+extern int s390_isolate_bp_guest(void);
+
#endif /* __ASSEMBLY__ */
#endif /* __ASM_S390_PROCESSOR_H */
diff --git a/arch/s390/include/asm/qdio.h b/arch/s390/include/asm/qdio.h
index 998b61c..4b39ba7 100644
--- a/arch/s390/include/asm/qdio.h
+++ b/arch/s390/include/asm/qdio.h
@@ -261,7 +261,6 @@ struct qdio_outbuf_state {
void *user;
};
-#define QDIO_OUTBUF_STATE_FLAG_NONE 0x00
#define QDIO_OUTBUF_STATE_FLAG_PENDING 0x01
#define CHSC_AC1_INITIATE_INPUTQ 0x80
diff --git a/arch/s390/include/asm/thread_info.h b/arch/s390/include/asm/thread_info.h
index f15c039..84f2ae4 100644
--- a/arch/s390/include/asm/thread_info.h
+++ b/arch/s390/include/asm/thread_info.h
@@ -79,6 +79,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
#define TIF_SECCOMP 5 /* secure computing */
#define TIF_SYSCALL_TRACEPOINT 6 /* syscall tracepoint instrumentation */
#define TIF_UPROBE 7 /* breakpointed or single-stepping */
+#define TIF_ISOLATE_BP 8 /* Run process with isolated BP */
+#define TIF_ISOLATE_BP_GUEST 9 /* Run KVM guests with isolated BP */
#define TIF_31BIT 16 /* 32bit process */
#define TIF_MEMDIE 17 /* is terminating due to OOM killer */
#define TIF_RESTORE_SIGMASK 18 /* restore signal mask in do_signal() */
@@ -94,6 +96,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src);
#define _TIF_SECCOMP _BITUL(TIF_SECCOMP)
#define _TIF_SYSCALL_TRACEPOINT _BITUL(TIF_SYSCALL_TRACEPOINT)
#define _TIF_UPROBE _BITUL(TIF_UPROBE)
+#define _TIF_ISOLATE_BP _BITUL(TIF_ISOLATE_BP)
+#define _TIF_ISOLATE_BP_GUEST _BITUL(TIF_ISOLATE_BP_GUEST)
#define _TIF_31BIT _BITUL(TIF_31BIT)
#define _TIF_SINGLE_STEP _BITUL(TIF_SINGLE_STEP)
diff --git a/arch/s390/include/uapi/asm/kvm.h b/arch/s390/include/uapi/asm/kvm.h
index a2ffec4..81c02e1 100644
--- a/arch/s390/include/uapi/asm/kvm.h
+++ b/arch/s390/include/uapi/asm/kvm.h
@@ -197,6 +197,7 @@ struct kvm_guest_debug_arch {
#define KVM_SYNC_VRS (1UL << 6)
#define KVM_SYNC_RICCB (1UL << 7)
#define KVM_SYNC_FPRS (1UL << 8)
+#define KVM_SYNC_BPBC (1UL << 10)
/* definition of registers in kvm_run */
struct kvm_sync_regs {
__u64 prefix; /* prefix register */
@@ -217,7 +218,9 @@ struct kvm_sync_regs {
};
__u8 reserved[512]; /* for future vector expansion */
__u32 fpc; /* valid on KVM_SYNC_VRS or KVM_SYNC_FPRS */
- __u8 padding[52]; /* riccb needs to be 64byte aligned */
+ __u8 bpbc : 1; /* bp mode */
+ __u8 reserved2 : 7;
+ __u8 padding1[51]; /* riccb needs to be 64byte aligned */
__u8 riccb[64]; /* runtime instrumentation controls block */
};
diff --git a/arch/s390/kernel/Makefile b/arch/s390/kernel/Makefile
index 1f0fe98..5b13997 100644
--- a/arch/s390/kernel/Makefile
+++ b/arch/s390/kernel/Makefile
@@ -42,6 +42,7 @@
CFLAGS_REMOVE_sclp.o += $(CC_FLAGS_MARCH)
CFLAGS_sclp.o += -march=z900
CFLAGS_REMOVE_als.o += $(CC_FLAGS_MARCH)
+CFLAGS_REMOVE_als.o += $(CC_FLAGS_EXPOLINE)
CFLAGS_als.o += -march=z900
AFLAGS_REMOVE_head.o += $(CC_FLAGS_MARCH)
AFLAGS_head.o += -march=z900
@@ -57,10 +58,14 @@
obj-y += debug.o irq.o ipl.o dis.o diag.o sclp.o vdso.o als.o
obj-y += sysinfo.o jump_label.o lgr.o os_info.o machine_kexec.o pgm_check.o
obj-y += runtime_instr.o cache.o fpu.o dumpstack.o
-obj-y += entry.o reipl.o relocate_kernel.o
+obj-y += entry.o reipl.o relocate_kernel.o alternative.o
+obj-y += nospec-branch.o
extra-y += head.o head64.o vmlinux.lds
+obj-$(CONFIG_SYSFS) += nospec-sysfs.o
+CFLAGS_REMOVE_nospec-branch.o += $(CC_FLAGS_EXPOLINE)
+
obj-$(CONFIG_MODULES) += module.o
obj-$(CONFIG_SMP) += smp.o
obj-$(CONFIG_SCHED_TOPOLOGY) += topology.o
diff --git a/arch/s390/kernel/alternative.c b/arch/s390/kernel/alternative.c
new file mode 100644
index 0000000..b57b293
--- /dev/null
+++ b/arch/s390/kernel/alternative.c
@@ -0,0 +1,112 @@
+#include <linux/module.h>
+#include <asm/alternative.h>
+#include <asm/facility.h>
+#include <asm/nospec-branch.h>
+
+#define MAX_PATCH_LEN (255 - 1)
+
+static int __initdata_or_module alt_instr_disabled;
+
+static int __init disable_alternative_instructions(char *str)
+{
+ alt_instr_disabled = 1;
+ return 0;
+}
+
+early_param("noaltinstr", disable_alternative_instructions);
+
+struct brcl_insn {
+ u16 opc;
+ s32 disp;
+} __packed;
+
+static u16 __initdata_or_module nop16 = 0x0700;
+static u32 __initdata_or_module nop32 = 0x47000000;
+static struct brcl_insn __initdata_or_module nop48 = {
+ 0xc004, 0
+};
+
+static const void *nops[] __initdata_or_module = {
+ &nop16,
+ &nop32,
+ &nop48
+};
+
+static void __init_or_module add_jump_padding(void *insns, unsigned int len)
+{
+ struct brcl_insn brcl = {
+ 0xc0f4,
+ len / 2
+ };
+
+ memcpy(insns, &brcl, sizeof(brcl));
+ insns += sizeof(brcl);
+ len -= sizeof(brcl);
+
+ while (len > 0) {
+ memcpy(insns, &nop16, 2);
+ insns += 2;
+ len -= 2;
+ }
+}
+
+static void __init_or_module add_padding(void *insns, unsigned int len)
+{
+ if (len > 6)
+ add_jump_padding(insns, len);
+ else if (len >= 2)
+ memcpy(insns, nops[len / 2 - 1], len);
+}
+
+static void __init_or_module __apply_alternatives(struct alt_instr *start,
+ struct alt_instr *end)
+{
+ struct alt_instr *a;
+ u8 *instr, *replacement;
+ u8 insnbuf[MAX_PATCH_LEN];
+
+ /*
+ * The scan order should be from start to end. A later scanned
+ * alternative code can overwrite previously scanned alternative code.
+ */
+ for (a = start; a < end; a++) {
+ int insnbuf_sz = 0;
+
+ instr = (u8 *)&a->instr_offset + a->instr_offset;
+ replacement = (u8 *)&a->repl_offset + a->repl_offset;
+
+ if (!__test_facility(a->facility,
+ S390_lowcore.alt_stfle_fac_list))
+ continue;
+
+ if (unlikely(a->instrlen % 2 || a->replacementlen % 2)) {
+ WARN_ONCE(1, "cpu alternatives instructions length is "
+ "odd, skipping patching\n");
+ continue;
+ }
+
+ memcpy(insnbuf, replacement, a->replacementlen);
+ insnbuf_sz = a->replacementlen;
+
+ if (a->instrlen > a->replacementlen) {
+ add_padding(insnbuf + a->replacementlen,
+ a->instrlen - a->replacementlen);
+ insnbuf_sz += a->instrlen - a->replacementlen;
+ }
+
+ s390_kernel_write(instr, insnbuf, insnbuf_sz);
+ }
+}
+
+void __init_or_module apply_alternatives(struct alt_instr *start,
+ struct alt_instr *end)
+{
+ if (!alt_instr_disabled)
+ __apply_alternatives(start, end);
+}
+
+extern struct alt_instr __alt_instructions[], __alt_instructions_end[];
+void __init apply_alternative_instructions(void)
+{
+ apply_alternatives(__alt_instructions, __alt_instructions_end);
+}
diff --git a/arch/s390/kernel/asm-offsets.c b/arch/s390/kernel/asm-offsets.c
index f3df9e0..85c8ead 100644
--- a/arch/s390/kernel/asm-offsets.c
+++ b/arch/s390/kernel/asm-offsets.c
@@ -175,6 +175,7 @@ int main(void)
OFFSET(__LC_MACHINE_FLAGS, lowcore, machine_flags);
OFFSET(__LC_GMAP, lowcore, gmap);
OFFSET(__LC_PASTE, lowcore, paste);
+ OFFSET(__LC_BR_R1, lowcore, br_r1_trampoline);
/* software defined ABI-relevant lowcore locations 0xe00 - 0xe20 */
OFFSET(__LC_DUMP_REIPL, lowcore, ipib);
/* hardware defined lowcore locations 0x1000 - 0x18ff */
diff --git a/arch/s390/kernel/base.S b/arch/s390/kernel/base.S
index 326f717..61fca54 100644
--- a/arch/s390/kernel/base.S
+++ b/arch/s390/kernel/base.S
@@ -8,18 +8,22 @@
#include <linux/linkage.h>
#include <asm/asm-offsets.h>
+#include <asm/nospec-insn.h>
#include <asm/ptrace.h>
#include <asm/sigp.h>
+ GEN_BR_THUNK %r9
+ GEN_BR_THUNK %r14
+
ENTRY(s390_base_mcck_handler)
basr %r13,0
0: lg %r15,__LC_PANIC_STACK # load panic stack
aghi %r15,-STACK_FRAME_OVERHEAD
larl %r1,s390_base_mcck_handler_fn
- lg %r1,0(%r1)
- ltgr %r1,%r1
+ lg %r9,0(%r1)
+ ltgr %r9,%r9
jz 1f
- basr %r14,%r1
+ BASR_EX %r14,%r9
1: la %r1,4095
lmg %r0,%r15,__LC_GPREGS_SAVE_AREA-4095(%r1)
lpswe __LC_MCK_OLD_PSW
@@ -36,10 +40,10 @@
basr %r13,0
0: aghi %r15,-STACK_FRAME_OVERHEAD
larl %r1,s390_base_ext_handler_fn
- lg %r1,0(%r1)
- ltgr %r1,%r1
+ lg %r9,0(%r1)
+ ltgr %r9,%r9
jz 1f
- basr %r14,%r1
+ BASR_EX %r14,%r9
1: lmg %r0,%r15,__LC_SAVE_AREA_ASYNC
ni __LC_EXT_OLD_PSW+1,0xfd # clear wait state bit
lpswe __LC_EXT_OLD_PSW
@@ -56,10 +60,10 @@
basr %r13,0
0: aghi %r15,-STACK_FRAME_OVERHEAD
larl %r1,s390_base_pgm_handler_fn
- lg %r1,0(%r1)
- ltgr %r1,%r1
+ lg %r9,0(%r1)
+ ltgr %r9,%r9
jz 1f
- basr %r14,%r1
+ BASR_EX %r14,%r9
lmg %r0,%r15,__LC_SAVE_AREA_SYNC
lpswe __LC_PGM_OLD_PSW
1: lpswe disabled_wait_psw-0b(%r13)
@@ -116,7 +120,7 @@
larl %r4,.Lcontinue_psw # Restore PSW flags
lpswe 0(%r4)
.Lcontinue:
- br %r14
+ BR_EX %r14
.align 16
.Lrestart_psw:
.long 0x00080000,0x80000000 + .Lrestart_part2
diff --git a/arch/s390/kernel/crash_dump.c b/arch/s390/kernel/crash_dump.c
index 5982544..1671352 100644
--- a/arch/s390/kernel/crash_dump.c
+++ b/arch/s390/kernel/crash_dump.c
@@ -401,11 +401,13 @@ static void *get_vmcoreinfo_old(unsigned long *size)
if (copy_oldmem_kernel(nt_name, addr + sizeof(note),
sizeof(nt_name) - 1))
return NULL;
- if (strcmp(nt_name, "VMCOREINFO") != 0)
+ if (strcmp(nt_name, VMCOREINFO_NOTE_NAME) != 0)
return NULL;
vmcoreinfo = kzalloc_panic(note.n_descsz);
- if (copy_oldmem_kernel(vmcoreinfo, addr + 24, note.n_descsz))
+ if (copy_oldmem_kernel(vmcoreinfo, addr + 24, note.n_descsz)) {
+ kfree(vmcoreinfo);
return NULL;
+ }
*size = note.n_descsz;
return vmcoreinfo;
}
@@ -415,15 +417,20 @@ static void *get_vmcoreinfo_old(unsigned long *size)
*/
static void *nt_vmcoreinfo(void *ptr)
{
+ const char *name = VMCOREINFO_NOTE_NAME;
unsigned long size;
void *vmcoreinfo;
vmcoreinfo = os_info_old_entry(OS_INFO_VMCOREINFO, &size);
- if (!vmcoreinfo)
- vmcoreinfo = get_vmcoreinfo_old(&size);
+ if (vmcoreinfo)
+ return nt_init_name(ptr, 0, vmcoreinfo, size, name);
+
+ vmcoreinfo = get_vmcoreinfo_old(&size);
if (!vmcoreinfo)
return ptr;
- return nt_init_name(ptr, 0, vmcoreinfo, size, "VMCOREINFO");
+ ptr = nt_init_name(ptr, 0, vmcoreinfo, size, name);
+ kfree(vmcoreinfo);
+ return ptr;
}
/*
diff --git a/arch/s390/kernel/early.c b/arch/s390/kernel/early.c
index 6257898..0c7a7d5 100644
--- a/arch/s390/kernel/early.c
+++ b/arch/s390/kernel/early.c
@@ -299,6 +299,11 @@ static noinline __init void setup_facility_list(void)
{
stfle(S390_lowcore.stfle_fac_list,
ARRAY_SIZE(S390_lowcore.stfle_fac_list));
+ memcpy(S390_lowcore.alt_stfle_fac_list,
+ S390_lowcore.stfle_fac_list,
+ sizeof(S390_lowcore.alt_stfle_fac_list));
+ if (!IS_ENABLED(CONFIG_KERNEL_NOBP))
+ __clear_facility(82, S390_lowcore.alt_stfle_fac_list);
}
static __init void detect_diag9c(void)
diff --git a/arch/s390/kernel/entry.S b/arch/s390/kernel/entry.S
index 3bc2825..771cfd2 100644
--- a/arch/s390/kernel/entry.S
+++ b/arch/s390/kernel/entry.S
@@ -24,6 +24,7 @@
#include <asm/setup.h>
#include <asm/nmi.h>
#include <asm/export.h>
+#include <asm/nospec-insn.h>
__PT_R0 = __PT_GPRS
__PT_R1 = __PT_GPRS + 8
@@ -105,6 +106,7 @@
j 3f
1: LAST_BREAK %r14
UPDATE_VTIME %r14,%r15,\timer
+ BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP
2: lg %r15,__LC_ASYNC_STACK # load async stack
3: la %r11,STACK_FRAME_OVERHEAD(%r15)
.endm
@@ -163,6 +165,72 @@
tm off+\addr, \mask
.endm
+ .macro BPOFF
+ .pushsection .altinstr_replacement, "ax"
+660: .long 0xb2e8c000
+ .popsection
+661: .long 0x47000000
+ .pushsection .altinstructions, "a"
+ .long 661b - .
+ .long 660b - .
+ .word 82
+ .byte 4
+ .byte 4
+ .popsection
+ .endm
+
+ .macro BPON
+ .pushsection .altinstr_replacement, "ax"
+662: .long 0xb2e8d000
+ .popsection
+663: .long 0x47000000
+ .pushsection .altinstructions, "a"
+ .long 663b - .
+ .long 662b - .
+ .word 82
+ .byte 4
+ .byte 4
+ .popsection
+ .endm
+
+ .macro BPENTER tif_ptr,tif_mask
+ .pushsection .altinstr_replacement, "ax"
+662: .word 0xc004, 0x0000, 0x0000 # 6 byte nop
+ .word 0xc004, 0x0000, 0x0000 # 6 byte nop
+ .popsection
+664: TSTMSK \tif_ptr,\tif_mask
+ jz . + 8
+ .long 0xb2e8d000
+ .pushsection .altinstructions, "a"
+ .long 664b - .
+ .long 662b - .
+ .word 82
+ .byte 12
+ .byte 12
+ .popsection
+ .endm
+
+ .macro BPEXIT tif_ptr,tif_mask
+ TSTMSK \tif_ptr,\tif_mask
+ .pushsection .altinstr_replacement, "ax"
+662: jnz . + 8
+ .long 0xb2e8d000
+ .popsection
+664: jz . + 8
+ .long 0xb2e8c000
+ .pushsection .altinstructions, "a"
+ .long 664b - .
+ .long 662b - .
+ .word 82
+ .byte 8
+ .byte 8
+ .popsection
+ .endm
+
+ GEN_BR_THUNK %r9
+ GEN_BR_THUNK %r14
+ GEN_BR_THUNK %r14,%r11
+
.section .kprobes.text, "ax"
.Ldummy:
/*
@@ -175,6 +243,11 @@
*/
nop 0
+ENTRY(__bpon)
+ .globl __bpon
+ BPON
+ BR_EX %r14
+
/*
* Scheduler resume function, called by switch_to
* gpr2 = (task_struct *) prev
@@ -201,9 +274,9 @@
mvc __LC_CURRENT_PID(4,%r0),__TASK_pid(%r3) # store pid of next
lmg %r6,%r15,__SF_GPRS(%r15) # load gprs of next task
TSTMSK __LC_MACHINE_FLAGS,MACHINE_FLAG_LPP
- bzr %r14
+ jz 0f
.insn s,0xb2800000,__LC_LPP # set program parameter
- br %r14
+0: BR_EX %r14
.L__critical_start:
@@ -215,9 +288,11 @@
*/
ENTRY(sie64a)
stmg %r6,%r14,__SF_GPRS(%r15) # save kernel registers
+ lg %r12,__LC_CURRENT
stg %r2,__SF_EMPTY(%r15) # save control block pointer
stg %r3,__SF_EMPTY+8(%r15) # save guest register save area
xc __SF_EMPTY+16(8,%r15),__SF_EMPTY+16(%r15) # reason code = 0
+ mvc __SF_EMPTY+24(8,%r15),__TI_flags(%r12) # copy thread flags
TSTMSK __LC_CPU_FLAGS,_CIF_FPU # load guest fp/vx registers ?
jno .Lsie_load_guest_gprs
brasl %r14,load_fpu_regs # load guest fp/vx regs
@@ -234,7 +309,11 @@
jnz .Lsie_skip
TSTMSK __LC_CPU_FLAGS,_CIF_FPU
jo .Lsie_skip # exit if fp/vx regs changed
+ BPEXIT __SF_EMPTY+24(%r15),(_TIF_ISOLATE_BP|_TIF_ISOLATE_BP_GUEST)
sie 0(%r14)
+.Lsie_exit:
+ BPOFF
+ BPENTER __SF_EMPTY+24(%r15),(_TIF_ISOLATE_BP|_TIF_ISOLATE_BP_GUEST)
.Lsie_skip:
ni __SIE_PROG0C+3(%r14),0xfe # no longer in SIE
lctlg %c1,%c1,__LC_USER_ASCE # load primary asce
@@ -255,9 +334,15 @@
sie_exit:
lg %r14,__SF_EMPTY+8(%r15) # load guest register save area
stmg %r0,%r13,0(%r14) # save guest gprs 0-13
+ xgr %r0,%r0 # clear guest registers to
+ xgr %r1,%r1 # prevent speculative use
+ xgr %r2,%r2
+ xgr %r3,%r3
+ xgr %r4,%r4
+ xgr %r5,%r5
lmg %r6,%r14,__SF_GPRS(%r15) # restore kernel registers
lg %r2,__SF_EMPTY+16(%r15) # return exit reason code
- br %r14
+ BR_EX %r14
.Lsie_fault:
lghi %r14,-EFAULT
stg %r14,__SF_EMPTY+16(%r15) # set exit reason code
@@ -280,6 +365,7 @@
stpt __LC_SYNC_ENTER_TIMER
.Lsysc_stmg:
stmg %r8,%r15,__LC_SAVE_AREA_SYNC
+ BPOFF
lg %r10,__LC_LAST_BREAK
lg %r12,__LC_THREAD_INFO
lghi %r14,_PIF_SYSCALL
@@ -289,12 +375,15 @@
LAST_BREAK %r13
.Lsysc_vtime:
UPDATE_VTIME %r10,%r13,__LC_SYNC_ENTER_TIMER
+ BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP
stmg %r0,%r7,__PT_R0(%r11)
mvc __PT_R8(64,%r11),__LC_SAVE_AREA_SYNC
mvc __PT_PSW(16,%r11),__LC_SVC_OLD_PSW
mvc __PT_INT_CODE(4,%r11),__LC_SVC_ILC
stg %r14,__PT_FLAGS(%r11)
.Lsysc_do_svc:
+ # clear user controlled register to prevent speculative use
+ xgr %r0,%r0
lg %r10,__TI_sysc_table(%r12) # address of system call table
llgh %r8,__PT_INT_CODE+2(%r11)
slag %r8,%r8,2 # shift and test for svc 0
@@ -312,7 +401,7 @@
lgf %r9,0(%r8,%r10) # get system call add.
TSTMSK __TI_flags(%r12),_TIF_TRACE
jnz .Lsysc_tracesys
- basr %r14,%r9 # call sys_xxxx
+ BASR_EX %r14,%r9 # call sys_xxxx
stg %r2,__PT_R2(%r11) # store return value
.Lsysc_return:
@@ -324,6 +413,7 @@
jnz .Lsysc_work # check for work
TSTMSK __LC_CPU_FLAGS,_CIF_WORK
jnz .Lsysc_work
+ BPEXIT __TI_flags(%r12),_TIF_ISOLATE_BP
.Lsysc_restore:
lg %r14,__LC_VDSO_PER_CPU
lmg %r0,%r10,__PT_R0(%r11)
@@ -451,7 +541,7 @@
lmg %r3,%r7,__PT_R3(%r11)
stg %r7,STACK_FRAME_OVERHEAD(%r15)
lg %r2,__PT_ORIG_GPR2(%r11)
- basr %r14,%r9 # call sys_xxx
+ BASR_EX %r14,%r9 # call sys_xxx
stg %r2,__PT_R2(%r11) # store return value
.Lsysc_tracenogo:
TSTMSK __TI_flags(%r12),_TIF_TRACE
@@ -475,7 +565,7 @@
lmg %r9,%r10,__PT_R9(%r11) # load gprs
ENTRY(kernel_thread_starter)
la %r2,0(%r10)
- basr %r14,%r9
+ BASR_EX %r14,%r9
j .Lsysc_tracenogo
/*
@@ -484,6 +574,7 @@
ENTRY(pgm_check_handler)
stpt __LC_SYNC_ENTER_TIMER
+ BPOFF
stmg %r8,%r15,__LC_SAVE_AREA_SYNC
lg %r10,__LC_LAST_BREAK
lg %r12,__LC_THREAD_INFO
@@ -508,6 +599,7 @@
j 3f
2: LAST_BREAK %r14
UPDATE_VTIME %r14,%r15,__LC_SYNC_ENTER_TIMER
+ BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP
lg %r15,__LC_KERNEL_STACK
lg %r14,__TI_task(%r12)
aghi %r14,__TASK_thread # pointer to thread_struct
@@ -517,6 +609,15 @@
mvc __THREAD_trap_tdb(256,%r14),0(%r13)
3: la %r11,STACK_FRAME_OVERHEAD(%r15)
stmg %r0,%r7,__PT_R0(%r11)
+ # clear user controlled registers to prevent speculative use
+ xgr %r0,%r0
+ xgr %r1,%r1
+ xgr %r2,%r2
+ xgr %r3,%r3
+ xgr %r4,%r4
+ xgr %r5,%r5
+ xgr %r6,%r6
+ xgr %r7,%r7
mvc __PT_R8(64,%r11),__LC_SAVE_AREA_SYNC
stmg %r8,%r9,__PT_PSW(%r11)
mvc __PT_INT_CODE(4,%r11),__LC_PGM_ILC
@@ -538,9 +639,9 @@
nill %r10,0x007f
sll %r10,2
je .Lpgm_return
- lgf %r1,0(%r10,%r1) # load address of handler routine
+ lgf %r9,0(%r10,%r1) # load address of handler routine
lgr %r2,%r11 # pass pointer to pt_regs
- basr %r14,%r1 # branch to interrupt-handler
+ BASR_EX %r14,%r9 # branch to interrupt-handler
.Lpgm_return:
LOCKDEP_SYS_EXIT
tm __PT_PSW+1(%r11),0x01 # returning to user ?
@@ -573,6 +674,7 @@
ENTRY(io_int_handler)
STCK __LC_INT_CLOCK
stpt __LC_ASYNC_ENTER_TIMER
+ BPOFF
stmg %r8,%r15,__LC_SAVE_AREA_ASYNC
lg %r10,__LC_LAST_BREAK
lg %r12,__LC_THREAD_INFO
@@ -580,6 +682,16 @@
lmg %r8,%r9,__LC_IO_OLD_PSW
SWITCH_ASYNC __LC_SAVE_AREA_ASYNC,__LC_ASYNC_ENTER_TIMER
stmg %r0,%r7,__PT_R0(%r11)
+ # clear user controlled registers to prevent speculative use
+ xgr %r0,%r0
+ xgr %r1,%r1
+ xgr %r2,%r2
+ xgr %r3,%r3
+ xgr %r4,%r4
+ xgr %r5,%r5
+ xgr %r6,%r6
+ xgr %r7,%r7
+ xgr %r10,%r10
mvc __PT_R8(64,%r11),__LC_SAVE_AREA_ASYNC
stmg %r8,%r9,__PT_PSW(%r11)
mvc __PT_INT_CODE(12,%r11),__LC_SUBCHANNEL_ID
@@ -614,9 +726,13 @@
lg %r14,__LC_VDSO_PER_CPU
lmg %r0,%r10,__PT_R0(%r11)
mvc __LC_RETURN_PSW(16),__PT_PSW(%r11)
+ tm __PT_PSW+1(%r11),0x01 # returning to user ?
+ jno .Lio_exit_kernel
+ BPEXIT __TI_flags(%r12),_TIF_ISOLATE_BP
.Lio_exit_timer:
stpt __LC_EXIT_TIMER
mvc __VDSO_ECTG_BASE(16,%r14),__LC_EXIT_TIMER
+.Lio_exit_kernel:
lmg %r11,%r15,__PT_R11(%r11)
lpswe __LC_RETURN_PSW
.Lio_done:
@@ -748,6 +864,7 @@
ENTRY(ext_int_handler)
STCK __LC_INT_CLOCK
stpt __LC_ASYNC_ENTER_TIMER
+ BPOFF
stmg %r8,%r15,__LC_SAVE_AREA_ASYNC
lg %r10,__LC_LAST_BREAK
lg %r12,__LC_THREAD_INFO
@@ -755,6 +872,16 @@
lmg %r8,%r9,__LC_EXT_OLD_PSW
SWITCH_ASYNC __LC_SAVE_AREA_ASYNC,__LC_ASYNC_ENTER_TIMER
stmg %r0,%r7,__PT_R0(%r11)
+ # clear user controlled registers to prevent speculative use
+ xgr %r0,%r0
+ xgr %r1,%r1
+ xgr %r2,%r2
+ xgr %r3,%r3
+ xgr %r4,%r4
+ xgr %r5,%r5
+ xgr %r6,%r6
+ xgr %r7,%r7
+ xgr %r10,%r10
mvc __PT_R8(64,%r11),__LC_SAVE_AREA_ASYNC
stmg %r8,%r9,__PT_PSW(%r11)
lghi %r1,__LC_EXT_PARAMS2
@@ -787,11 +914,12 @@
.Lpsw_idle_stcctm:
#endif
oi __LC_CPU_FLAGS+7,_CIF_ENABLED_WAIT
+ BPON
STCK __CLOCK_IDLE_ENTER(%r2)
stpt __TIMER_IDLE_ENTER(%r2)
.Lpsw_idle_lpsw:
lpswe __SF_EMPTY(%r15)
- br %r14
+ BR_EX %r14
.Lpsw_idle_end:
/*
@@ -805,7 +933,7 @@
lg %r2,__LC_CURRENT
aghi %r2,__TASK_thread
TSTMSK __LC_CPU_FLAGS,_CIF_FPU
- bor %r14
+ jo .Lsave_fpu_regs_exit
stfpc __THREAD_FPU_fpc(%r2)
.Lsave_fpu_regs_fpc_end:
lg %r3,__THREAD_FPU_regs(%r2)
@@ -835,7 +963,8 @@
std 15,120(%r3)
.Lsave_fpu_regs_done:
oi __LC_CPU_FLAGS+7,_CIF_FPU
- br %r14
+.Lsave_fpu_regs_exit:
+ BR_EX %r14
.Lsave_fpu_regs_end:
#if IS_ENABLED(CONFIG_KVM)
EXPORT_SYMBOL(save_fpu_regs)
@@ -855,7 +984,7 @@
lg %r4,__LC_CURRENT
aghi %r4,__TASK_thread
TSTMSK __LC_CPU_FLAGS,_CIF_FPU
- bnor %r14
+ jno .Lload_fpu_regs_exit
lfpc __THREAD_FPU_fpc(%r4)
TSTMSK __LC_MACHINE_FLAGS,MACHINE_FLAG_VX
lg %r4,__THREAD_FPU_regs(%r4) # %r4 <- reg save area
@@ -884,7 +1013,8 @@
ld 15,120(%r4)
.Lload_fpu_regs_done:
ni __LC_CPU_FLAGS+7,255-_CIF_FPU
- br %r14
+.Lload_fpu_regs_exit:
+ BR_EX %r14
.Lload_fpu_regs_end:
.L__critical_end:
@@ -894,6 +1024,7 @@
*/
ENTRY(mcck_int_handler)
STCK __LC_MCCK_CLOCK
+ BPOFF
la %r1,4095 # revalidate r1
spt __LC_CPU_TIMER_SAVE_AREA-4095(%r1) # revalidate cpu timer
lmg %r0,%r15,__LC_GPREGS_SAVE_AREA-4095(%r1)# revalidate gprs
@@ -925,6 +1056,16 @@
.Lmcck_skip:
lghi %r14,__LC_GPREGS_SAVE_AREA+64
stmg %r0,%r7,__PT_R0(%r11)
+ # clear user controlled registers to prevent speculative use
+ xgr %r0,%r0
+ xgr %r1,%r1
+ xgr %r2,%r2
+ xgr %r3,%r3
+ xgr %r4,%r4
+ xgr %r5,%r5
+ xgr %r6,%r6
+ xgr %r7,%r7
+ xgr %r10,%r10
mvc __PT_R8(64,%r11),0(%r14)
stmg %r8,%r9,__PT_PSW(%r11)
xc __PT_FLAGS(8,%r11),__PT_FLAGS(%r11)
@@ -950,6 +1091,7 @@
mvc __LC_RETURN_MCCK_PSW(16),__PT_PSW(%r11) # move return PSW
tm __LC_RETURN_MCCK_PSW+1,0x01 # returning to user ?
jno 0f
+ BPEXIT __TI_flags(%r12),_TIF_ISOLATE_BP
stpt __LC_EXIT_TIMER
mvc __VDSO_ECTG_BASE(16,%r14),__LC_EXIT_TIMER
0: lmg %r11,%r15,__PT_R11(%r11)
@@ -1045,7 +1187,7 @@
jl 0f
clg %r9,BASED(.Lcleanup_table+104) # .Lload_fpu_regs_end
jl .Lcleanup_load_fpu_regs
-0: br %r14
+0: BR_EX %r14,%r11
.align 8
.Lcleanup_table:
@@ -1070,11 +1212,12 @@
.quad .Lsie_done
.Lcleanup_sie:
+ BPENTER __SF_EMPTY+24(%r15),(_TIF_ISOLATE_BP|_TIF_ISOLATE_BP_GUEST)
lg %r9,__SF_EMPTY(%r15) # get control block pointer
ni __SIE_PROG0C+3(%r9),0xfe # no longer in SIE
lctlg %c1,%c1,__LC_USER_ASCE # load primary asce
larl %r9,sie_exit # skip forward to sie_exit
- br %r14
+ BR_EX %r14,%r11
#endif
.Lcleanup_system_call:
@@ -1116,7 +1259,8 @@
srag %r9,%r9,23
jz 0f
mvc __TI_last_break(8,%r12),16(%r11)
-0: # set up saved register r11
+0: BPENTER __TI_flags(%r12),_TIF_ISOLATE_BP
+ # set up saved register r11
lg %r15,__LC_KERNEL_STACK
la %r9,STACK_FRAME_OVERHEAD(%r15)
stg %r9,24(%r11) # r11 pt_regs pointer
@@ -1131,7 +1275,7 @@
stg %r15,56(%r11) # r15 stack pointer
# set new psw address and exit
larl %r9,.Lsysc_do_svc
- br %r14
+ BR_EX %r14,%r11
.Lcleanup_system_call_insn:
.quad system_call
.quad .Lsysc_stmg
@@ -1141,7 +1285,7 @@
.Lcleanup_sysc_tif:
larl %r9,.Lsysc_tif
- br %r14
+ BR_EX %r14,%r11
.Lcleanup_sysc_restore:
# check if stpt has been executed
@@ -1158,14 +1302,14 @@
mvc 0(64,%r11),__PT_R8(%r9)
lmg %r0,%r7,__PT_R0(%r9)
1: lmg %r8,%r9,__LC_RETURN_PSW
- br %r14
+ BR_EX %r14,%r11
.Lcleanup_sysc_restore_insn:
.quad .Lsysc_exit_timer
.quad .Lsysc_done - 4
.Lcleanup_io_tif:
larl %r9,.Lio_tif
- br %r14
+ BR_EX %r14,%r11
.Lcleanup_io_restore:
# check if stpt has been executed
@@ -1179,7 +1323,7 @@
mvc 0(64,%r11),__PT_R8(%r9)
lmg %r0,%r7,__PT_R0(%r9)
1: lmg %r8,%r9,__LC_RETURN_PSW
- br %r14
+ BR_EX %r14,%r11
.Lcleanup_io_restore_insn:
.quad .Lio_exit_timer
.quad .Lio_done - 4
@@ -1232,17 +1376,17 @@
# prepare return psw
nihh %r8,0xfcfd # clear irq & wait state bits
lg %r9,48(%r11) # return from psw_idle
- br %r14
+ BR_EX %r14,%r11
.Lcleanup_idle_insn:
.quad .Lpsw_idle_lpsw
.Lcleanup_save_fpu_regs:
larl %r9,save_fpu_regs
- br %r14
+ BR_EX %r14,%r11
.Lcleanup_load_fpu_regs:
larl %r9,load_fpu_regs
- br %r14
+ BR_EX %r14,%r11
/*
* Integer constants
@@ -1258,7 +1402,6 @@
.Lsie_critical_length:
.quad .Lsie_done - .Lsie_gmap
#endif
-
.section .rodata, "a"
#define SYSCALL(esame,emu) .long esame
.globl sys_call_table
diff --git a/arch/s390/kernel/ipl.c b/arch/s390/kernel/ipl.c
index 39127b6..df49f2a1 100644
--- a/arch/s390/kernel/ipl.c
+++ b/arch/s390/kernel/ipl.c
@@ -563,6 +563,7 @@ static struct kset *ipl_kset;
static void __ipl_run(void *unused)
{
+ __bpon();
diag308(DIAG308_LOAD_CLEAR, NULL);
if (MACHINE_IS_VM)
__cpcmd("IPL", NULL, 0, NULL);
diff --git a/arch/s390/kernel/irq.c b/arch/s390/kernel/irq.c
index 285d656..7ff9767 100644
--- a/arch/s390/kernel/irq.c
+++ b/arch/s390/kernel/irq.c
@@ -173,10 +173,9 @@ void do_softirq_own_stack(void)
new -= STACK_FRAME_OVERHEAD;
((struct stack_frame *) new)->back_chain = old;
asm volatile(" la 15,0(%0)\n"
- " basr 14,%2\n"
+ " brasl 14,__do_softirq\n"
" la 15,0(%1)\n"
- : : "a" (new), "a" (old),
- "a" (__do_softirq)
+ : : "a" (new), "a" (old)
: "0", "1", "2", "3", "4", "5", "14",
"cc", "memory" );
} else {
diff --git a/arch/s390/kernel/mcount.S b/arch/s390/kernel/mcount.S
index 9a17e44..be75e8e 100644
--- a/arch/s390/kernel/mcount.S
+++ b/arch/s390/kernel/mcount.S
@@ -8,13 +8,17 @@
#include <linux/linkage.h>
#include <asm/asm-offsets.h>
#include <asm/ftrace.h>
+#include <asm/nospec-insn.h>
#include <asm/ptrace.h>
#include <asm/export.h>
+ GEN_BR_THUNK %r1
+ GEN_BR_THUNK %r14
+
.section .kprobes.text, "ax"
ENTRY(ftrace_stub)
- br %r14
+ BR_EX %r14
#define STACK_FRAME_SIZE (STACK_FRAME_OVERHEAD + __PT_SIZE)
#define STACK_PTREGS (STACK_FRAME_OVERHEAD)
@@ -22,7 +26,7 @@
#define STACK_PTREGS_PSW (STACK_PTREGS + __PT_PSW)
ENTRY(_mcount)
- br %r14
+ BR_EX %r14
EXPORT_SYMBOL(_mcount)
@@ -52,7 +56,7 @@
#endif
lgr %r3,%r14
la %r5,STACK_PTREGS(%r15)
- basr %r14,%r1
+ BASR_EX %r14,%r1
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
# The j instruction gets runtime patched to a nop instruction.
# See ftrace_enable_ftrace_graph_caller.
@@ -67,7 +71,7 @@
#endif
lg %r1,(STACK_PTREGS_PSW+8)(%r15)
lmg %r2,%r15,(STACK_PTREGS_GPRS+2*8)(%r15)
- br %r1
+ BR_EX %r1
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
@@ -80,6 +84,6 @@
aghi %r15,STACK_FRAME_OVERHEAD
lgr %r14,%r2
lmg %r2,%r5,32(%r15)
- br %r14
+ BR_EX %r14
#endif
diff --git a/arch/s390/kernel/module.c b/arch/s390/kernel/module.c
index fbc0789..64ccfdf 100644
--- a/arch/s390/kernel/module.c
+++ b/arch/s390/kernel/module.c
@@ -31,6 +31,9 @@
#include <linux/kernel.h>
#include <linux/moduleloader.h>
#include <linux/bug.h>
+#include <asm/alternative.h>
+#include <asm/nospec-branch.h>
+#include <asm/facility.h>
#if 0
#define DEBUGP printk
@@ -167,7 +170,11 @@ int module_frob_arch_sections(Elf_Ehdr *hdr, Elf_Shdr *sechdrs,
me->arch.got_offset = me->core_layout.size;
me->core_layout.size += me->arch.got_size;
me->arch.plt_offset = me->core_layout.size;
- me->core_layout.size += me->arch.plt_size;
+ if (me->arch.plt_size) {
+ if (IS_ENABLED(CONFIG_EXPOLINE) && !nospec_disable)
+ me->arch.plt_size += PLT_ENTRY_SIZE;
+ me->core_layout.size += me->arch.plt_size;
+ }
return 0;
}
@@ -321,9 +328,20 @@ static int apply_rela(Elf_Rela *rela, Elf_Addr base, Elf_Sym *symtab,
unsigned int *ip;
ip = me->core_layout.base + me->arch.plt_offset +
info->plt_offset;
- ip[0] = 0x0d10e310; /* basr 1,0; lg 1,10(1); br 1 */
- ip[1] = 0x100a0004;
- ip[2] = 0x07f10000;
+ ip[0] = 0x0d10e310; /* basr 1,0 */
+ ip[1] = 0x100a0004; /* lg 1,10(1) */
+ if (IS_ENABLED(CONFIG_EXPOLINE) && !nospec_disable) {
+ unsigned int *ij;
+ ij = me->core_layout.base +
+ me->arch.plt_offset +
+ me->arch.plt_size - PLT_ENTRY_SIZE;
+ ip[2] = 0xa7f40000 + /* j __jump_r1 */
+ (unsigned int)(u16)
+ (((unsigned long) ij - 8 -
+ (unsigned long) ip) / 2);
+ } else {
+ ip[2] = 0x07f10000; /* br %r1 */
+ }
ip[3] = (unsigned int) (val >> 32);
ip[4] = (unsigned int) val;
info->plt_initialized = 1;
@@ -428,6 +446,45 @@ int module_finalize(const Elf_Ehdr *hdr,
const Elf_Shdr *sechdrs,
struct module *me)
{
+ const Elf_Shdr *s;
+ char *secstrings, *secname;
+ void *aseg;
+
+ if (IS_ENABLED(CONFIG_EXPOLINE) &&
+ !nospec_disable && me->arch.plt_size) {
+ unsigned int *ij;
+
+ ij = me->core_layout.base + me->arch.plt_offset +
+ me->arch.plt_size - PLT_ENTRY_SIZE;
+ if (test_facility(35)) {
+ ij[0] = 0xc6000000; /* exrl %r0,.+10 */
+ ij[1] = 0x0005a7f4; /* j . */
+ ij[2] = 0x000007f1; /* br %r1 */
+ } else {
+ ij[0] = 0x44000000 | (unsigned int)
+ offsetof(struct lowcore, br_r1_trampoline);
+ ij[1] = 0xa7f40000; /* j . */
+ }
+ }
+
+ secstrings = (void *)hdr + sechdrs[hdr->e_shstrndx].sh_offset;
+ for (s = sechdrs; s < sechdrs + hdr->e_shnum; s++) {
+ aseg = (void *) s->sh_addr;
+ secname = secstrings + s->sh_name;
+
+ if (!strcmp(".altinstructions", secname))
+ /* patch .altinstructions */
+ apply_alternatives(aseg, aseg + s->sh_size);
+
+ if (IS_ENABLED(CONFIG_EXPOLINE) &&
+ (!strncmp(".s390_indirect", secname, 14)))
+ nospec_revert(aseg, aseg + s->sh_size);
+
+ if (IS_ENABLED(CONFIG_EXPOLINE) &&
+ (!strncmp(".s390_return", secname, 12)))
+ nospec_revert(aseg, aseg + s->sh_size);
+ }
+
jump_label_apply_nops(me);
return 0;
}
diff --git a/arch/s390/kernel/nospec-branch.c b/arch/s390/kernel/nospec-branch.c
new file mode 100644
index 0000000..d5eed65
--- /dev/null
+++ b/arch/s390/kernel/nospec-branch.c
@@ -0,0 +1,166 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/module.h>
+#include <linux/device.h>
+#include <asm/facility.h>
+#include <asm/nospec-branch.h>
+
+static int __init nobp_setup_early(char *str)
+{
+ bool enabled;
+ int rc;
+
+ rc = kstrtobool(str, &enabled);
+ if (rc)
+ return rc;
+ if (enabled && test_facility(82)) {
+ /*
+ * The user explicitely requested nobp=1, enable it and
+ * disable the expoline support.
+ */
+ __set_facility(82, S390_lowcore.alt_stfle_fac_list);
+ if (IS_ENABLED(CONFIG_EXPOLINE))
+ nospec_disable = 1;
+ } else {
+ __clear_facility(82, S390_lowcore.alt_stfle_fac_list);
+ }
+ return 0;
+}
+early_param("nobp", nobp_setup_early);
+
+static int __init nospec_setup_early(char *str)
+{
+ __clear_facility(82, S390_lowcore.alt_stfle_fac_list);
+ return 0;
+}
+early_param("nospec", nospec_setup_early);
+
+static int __init nospec_report(void)
+{
+ if (IS_ENABLED(CC_USING_EXPOLINE) && !nospec_disable)
+ pr_info("Spectre V2 mitigation: execute trampolines.\n");
+ if (__test_facility(82, S390_lowcore.alt_stfle_fac_list))
+ pr_info("Spectre V2 mitigation: limited branch prediction.\n");
+ return 0;
+}
+arch_initcall(nospec_report);
+
+#ifdef CONFIG_EXPOLINE
+
+int nospec_disable = IS_ENABLED(CONFIG_EXPOLINE_OFF);
+
+static int __init nospectre_v2_setup_early(char *str)
+{
+ nospec_disable = 1;
+ return 0;
+}
+early_param("nospectre_v2", nospectre_v2_setup_early);
+
+void __init nospec_auto_detect(void)
+{
+ if (IS_ENABLED(CC_USING_EXPOLINE)) {
+ /*
+ * The kernel has been compiled with expolines.
+ * Keep expolines enabled and disable nobp.
+ */
+ nospec_disable = 0;
+ __clear_facility(82, S390_lowcore.alt_stfle_fac_list);
+ }
+ /*
+ * If the kernel has not been compiled with expolines the
+ * nobp setting decides what is done, this depends on the
+ * CONFIG_KERNEL_NP option and the nobp/nospec parameters.
+ */
+}
+
+static int __init spectre_v2_setup_early(char *str)
+{
+ if (str && !strncmp(str, "on", 2)) {
+ nospec_disable = 0;
+ __clear_facility(82, S390_lowcore.alt_stfle_fac_list);
+ }
+ if (str && !strncmp(str, "off", 3))
+ nospec_disable = 1;
+ if (str && !strncmp(str, "auto", 4))
+ nospec_auto_detect();
+ return 0;
+}
+early_param("spectre_v2", spectre_v2_setup_early);
+
+static void __init_or_module __nospec_revert(s32 *start, s32 *end)
+{
+ enum { BRCL_EXPOLINE, BRASL_EXPOLINE } type;
+ u8 *instr, *thunk, *br;
+ u8 insnbuf[6];
+ s32 *epo;
+
+ /* Second part of the instruction replace is always a nop */
+ for (epo = start; epo < end; epo++) {
+ instr = (u8 *) epo + *epo;
+ if (instr[0] == 0xc0 && (instr[1] & 0x0f) == 0x04)
+ type = BRCL_EXPOLINE; /* brcl instruction */
+ else if (instr[0] == 0xc0 && (instr[1] & 0x0f) == 0x05)
+ type = BRASL_EXPOLINE; /* brasl instruction */
+ else
+ continue;
+ thunk = instr + (*(int *)(instr + 2)) * 2;
+ if (thunk[0] == 0xc6 && thunk[1] == 0x00)
+ /* exrl %r0,<target-br> */
+ br = thunk + (*(int *)(thunk + 2)) * 2;
+ else if (thunk[0] == 0xc0 && (thunk[1] & 0x0f) == 0x00 &&
+ thunk[6] == 0x44 && thunk[7] == 0x00 &&
+ (thunk[8] & 0x0f) == 0x00 && thunk[9] == 0x00 &&
+ (thunk[1] & 0xf0) == (thunk[8] & 0xf0))
+ /* larl %rx,<target br> + ex %r0,0(%rx) */
+ br = thunk + (*(int *)(thunk + 2)) * 2;
+ else
+ continue;
+ /* Check for unconditional branch 0x07f? or 0x47f???? */
+ if ((br[0] & 0xbf) != 0x07 || (br[1] & 0xf0) != 0xf0)
+ continue;
+
+ memcpy(insnbuf + 2, (char[]) { 0x47, 0x00, 0x07, 0x00 }, 4);
+ switch (type) {
+ case BRCL_EXPOLINE:
+ insnbuf[0] = br[0];
+ insnbuf[1] = (instr[1] & 0xf0) | (br[1] & 0x0f);
+ if (br[0] == 0x47) {
+ /* brcl to b, replace with bc + nopr */
+ insnbuf[2] = br[2];
+ insnbuf[3] = br[3];
+ } else {
+ /* brcl to br, replace with bcr + nop */
+ }
+ break;
+ case BRASL_EXPOLINE:
+ insnbuf[1] = (instr[1] & 0xf0) | (br[1] & 0x0f);
+ if (br[0] == 0x47) {
+ /* brasl to b, replace with bas + nopr */
+ insnbuf[0] = 0x4d;
+ insnbuf[2] = br[2];
+ insnbuf[3] = br[3];
+ } else {
+ /* brasl to br, replace with basr + nop */
+ insnbuf[0] = 0x0d;
+ }
+ break;
+ }
+
+ s390_kernel_write(instr, insnbuf, 6);
+ }
+}
+
+void __init_or_module nospec_revert(s32 *start, s32 *end)
+{
+ if (nospec_disable)
+ __nospec_revert(start, end);
+}
+
+extern s32 __nospec_call_start[], __nospec_call_end[];
+extern s32 __nospec_return_start[], __nospec_return_end[];
+void __init nospec_init_branches(void)
+{
+ nospec_revert(__nospec_call_start, __nospec_call_end);
+ nospec_revert(__nospec_return_start, __nospec_return_end);
+}
+
+#endif /* CONFIG_EXPOLINE */
diff --git a/arch/s390/kernel/nospec-sysfs.c b/arch/s390/kernel/nospec-sysfs.c
new file mode 100644
index 0000000..8affad5
--- /dev/null
+++ b/arch/s390/kernel/nospec-sysfs.c
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/device.h>
+#include <linux/cpu.h>
+#include <asm/facility.h>
+#include <asm/nospec-branch.h>
+
+ssize_t cpu_show_spectre_v1(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ return sprintf(buf, "Mitigation: __user pointer sanitization\n");
+}
+
+ssize_t cpu_show_spectre_v2(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ if (IS_ENABLED(CC_USING_EXPOLINE) && !nospec_disable)
+ return sprintf(buf, "Mitigation: execute trampolines\n");
+ if (__test_facility(82, S390_lowcore.alt_stfle_fac_list))
+ return sprintf(buf, "Mitigation: limited branch prediction\n");
+ return sprintf(buf, "Vulnerable\n");
+}
diff --git a/arch/s390/kernel/perf_cpum_cf.c b/arch/s390/kernel/perf_cpum_cf.c
index 037c2a2..1238e7e 100644
--- a/arch/s390/kernel/perf_cpum_cf.c
+++ b/arch/s390/kernel/perf_cpum_cf.c
@@ -344,6 +344,8 @@ static int __hw_perf_event_init(struct perf_event *event)
break;
case PERF_TYPE_HARDWARE:
+ if (is_sampling_event(event)) /* No sampling support */
+ return -ENOENT;
ev = attr->config;
/* Count user space (problem-state) only */
if (!attr->exclude_user && attr->exclude_kernel) {
diff --git a/arch/s390/kernel/perf_cpum_sf.c b/arch/s390/kernel/perf_cpum_sf.c
index fcc634c..96e4fca 100644
--- a/arch/s390/kernel/perf_cpum_sf.c
+++ b/arch/s390/kernel/perf_cpum_sf.c
@@ -739,6 +739,10 @@ static int __hw_perf_event_init(struct perf_event *event)
*/
rate = 0;
if (attr->freq) {
+ if (!attr->sample_freq) {
+ err = -EINVAL;
+ goto out;
+ }
rate = freq_to_sample_rate(&si, attr->sample_freq);
rate = hw_limit_rate(&si, rate);
attr->freq = 0;
diff --git a/arch/s390/kernel/processor.c b/arch/s390/kernel/processor.c
index 81d0808..d856263 100644
--- a/arch/s390/kernel/processor.c
+++ b/arch/s390/kernel/processor.c
@@ -179,3 +179,21 @@ const struct seq_operations cpuinfo_op = {
.stop = c_stop,
.show = show_cpuinfo,
};
+
+int s390_isolate_bp(void)
+{
+ if (!test_facility(82))
+ return -EOPNOTSUPP;
+ set_thread_flag(TIF_ISOLATE_BP);
+ return 0;
+}
+EXPORT_SYMBOL(s390_isolate_bp);
+
+int s390_isolate_bp_guest(void)
+{
+ if (!test_facility(82))
+ return -EOPNOTSUPP;
+ set_thread_flag(TIF_ISOLATE_BP_GUEST);
+ return 0;
+}
+EXPORT_SYMBOL(s390_isolate_bp_guest);
diff --git a/arch/s390/kernel/reipl.S b/arch/s390/kernel/reipl.S
index 89ea8c2..70d635d 100644
--- a/arch/s390/kernel/reipl.S
+++ b/arch/s390/kernel/reipl.S
@@ -6,8 +6,11 @@
#include <linux/linkage.h>
#include <asm/asm-offsets.h>
+#include <asm/nospec-insn.h>
#include <asm/sigp.h>
+ GEN_BR_THUNK %r9
+
#
# Issue "store status" for the current CPU to its prefix page
# and call passed function afterwards
@@ -66,9 +69,9 @@
st %r4,0(%r1)
st %r5,4(%r1)
stg %r2,8(%r1)
- lgr %r1,%r2
+ lgr %r9,%r2
lgr %r2,%r3
- br %r1
+ BR_EX %r9
.section .bss
.align 8
diff --git a/arch/s390/kernel/setup.c b/arch/s390/kernel/setup.c
index e974e53..feb9d97 100644
--- a/arch/s390/kernel/setup.c
+++ b/arch/s390/kernel/setup.c
@@ -63,6 +63,8 @@
#include <asm/sclp.h>
#include <asm/sysinfo.h>
#include <asm/numa.h>
+#include <asm/alternative.h>
+#include <asm/nospec-branch.h>
#include "entry.h"
/*
@@ -335,7 +337,9 @@ static void __init setup_lowcore(void)
lc->machine_flags = S390_lowcore.machine_flags;
lc->stfl_fac_list = S390_lowcore.stfl_fac_list;
memcpy(lc->stfle_fac_list, S390_lowcore.stfle_fac_list,
- MAX_FACILITY_BIT/8);
+ sizeof(lc->stfle_fac_list));
+ memcpy(lc->alt_stfle_fac_list, S390_lowcore.alt_stfle_fac_list,
+ sizeof(lc->alt_stfle_fac_list));
if (MACHINE_HAS_VX)
lc->vector_save_area_addr =
(unsigned long) &lc->vector_save_area;
@@ -372,6 +376,7 @@ static void __init setup_lowcore(void)
#ifdef CONFIG_SMP
lc->spinlock_lockval = arch_spin_lockval(0);
#endif
+ lc->br_r1_trampoline = 0x07f1; /* br %r1 */
set_prefix((u32)(unsigned long) lc);
lowcore_ptr[0] = lc;
@@ -871,6 +876,9 @@ void __init setup_arch(char **cmdline_p)
init_mm.end_data = (unsigned long) &_edata;
init_mm.brk = (unsigned long) &_end;
+ if (IS_ENABLED(CONFIG_EXPOLINE_AUTO))
+ nospec_auto_detect();
+
parse_early_param();
#ifdef CONFIG_CRASH_DUMP
/* Deactivate elfcorehdr= kernel parameter */
@@ -931,6 +939,10 @@ void __init setup_arch(char **cmdline_p)
conmode_default();
set_preferred_console();
+ apply_alternative_instructions();
+ if (IS_ENABLED(CONFIG_EXPOLINE))
+ nospec_init_branches();
+
/* Setup zfcpdump support */
setup_zfcpdump();
diff --git a/arch/s390/kernel/smp.c b/arch/s390/kernel/smp.c
index 35531fe..0a31110 100644
--- a/arch/s390/kernel/smp.c
+++ b/arch/s390/kernel/smp.c
@@ -205,6 +205,7 @@ static int pcpu_alloc_lowcore(struct pcpu *pcpu, int cpu)
lc->panic_stack = panic_stack + PANIC_FRAME_OFFSET;
lc->cpu_nr = cpu;
lc->spinlock_lockval = arch_spin_lockval(cpu);
+ lc->br_r1_trampoline = 0x07f1; /* br %r1 */
if (MACHINE_HAS_VX)
lc->vector_save_area_addr =
(unsigned long) &lc->vector_save_area;
@@ -253,7 +254,9 @@ static void pcpu_prepare_secondary(struct pcpu *pcpu, int cpu)
__ctl_store(lc->cregs_save_area, 0, 15);
save_access_regs((unsigned int *) lc->access_regs_save_area);
memcpy(lc->stfle_fac_list, S390_lowcore.stfle_fac_list,
- MAX_FACILITY_BIT/8);
+ sizeof(lc->stfle_fac_list));
+ memcpy(lc->alt_stfle_fac_list, S390_lowcore.alt_stfle_fac_list,
+ sizeof(lc->alt_stfle_fac_list));
}
static void pcpu_attach_task(struct pcpu *pcpu, struct task_struct *tsk)
@@ -302,6 +305,7 @@ static void pcpu_delegate(struct pcpu *pcpu, void (*func)(void *),
mem_assign_absolute(lc->restart_fn, (unsigned long) func);
mem_assign_absolute(lc->restart_data, (unsigned long) data);
mem_assign_absolute(lc->restart_source, source_cpu);
+ __bpon();
asm volatile(
"0: sigp 0,%0,%2 # sigp restart to target cpu\n"
" brc 2,0b # busy, try again\n"
@@ -875,6 +879,7 @@ void __cpu_die(unsigned int cpu)
void __noreturn cpu_die(void)
{
idle_task_exit();
+ __bpon();
pcpu_sigp_retry(pcpu_devices + smp_processor_id(), SIGP_STOP, 0);
for (;;) ;
}
diff --git a/arch/s390/kernel/swsusp.S b/arch/s390/kernel/swsusp.S
index 2d6b6e8..4e76aaf 100644
--- a/arch/s390/kernel/swsusp.S
+++ b/arch/s390/kernel/swsusp.S
@@ -12,6 +12,7 @@
#include <asm/ptrace.h>
#include <asm/thread_info.h>
#include <asm/asm-offsets.h>
+#include <asm/nospec-insn.h>
#include <asm/sigp.h>
/*
@@ -23,6 +24,8 @@
* (see below) in the resume process.
* This function runs with disabled interrupts.
*/
+ GEN_BR_THUNK %r14
+
.section .text
ENTRY(swsusp_arch_suspend)
stmg %r6,%r15,__SF_GPRS(%r15)
@@ -102,7 +105,7 @@
spx 0x318(%r1)
lmg %r6,%r15,STACK_FRAME_OVERHEAD + __SF_GPRS(%r15)
lghi %r2,0
- br %r14
+ BR_EX %r14
/*
* Restore saved memory image to correct place and restore register context.
@@ -200,7 +203,7 @@
lghi %r1,0
sam31
sigp %r1,%r0,SIGP_SET_ARCHITECTURE
- basr %r14,%r3
+ brasl %r14,_sclp_print_early
larl %r3,.Ldisabled_wait_31
lpsw 0(%r3)
4:
@@ -266,7 +269,7 @@
/* Return 0 */
lmg %r6,%r15,STACK_FRAME_OVERHEAD + __SF_GPRS(%r15)
lghi %r2,0
- br %r14
+ BR_EX %r14
.section .data..nosave,"aw",@progbits
.align 8
diff --git a/arch/s390/kernel/uprobes.c b/arch/s390/kernel/uprobes.c
index 66956c0..3d04dfd 100644
--- a/arch/s390/kernel/uprobes.c
+++ b/arch/s390/kernel/uprobes.c
@@ -147,6 +147,15 @@ unsigned long arch_uretprobe_hijack_return_addr(unsigned long trampoline,
return orig;
}
+bool arch_uretprobe_is_alive(struct return_instance *ret, enum rp_check ctx,
+ struct pt_regs *regs)
+{
+ if (ctx == RP_CHECK_CHAIN_CALL)
+ return user_stack_pointer(regs) <= ret->stack;
+ else
+ return user_stack_pointer(regs) < ret->stack;
+}
+
/* Instruction Emulation */
static void adjust_psw_addr(psw_t *psw, unsigned long len)
diff --git a/arch/s390/kernel/vdso32/Makefile b/arch/s390/kernel/vdso32/Makefile
index 6cc9478..ca7c3c3 100644
--- a/arch/s390/kernel/vdso32/Makefile
+++ b/arch/s390/kernel/vdso32/Makefile
@@ -32,7 +32,7 @@
$(obj)/vdso32_wrapper.o : $(obj)/vdso32.so
# link rule for the .so file, .lds has to be first
-$(obj)/vdso32.so.dbg: $(src)/vdso32.lds $(obj-vdso32)
+$(obj)/vdso32.so.dbg: $(src)/vdso32.lds $(obj-vdso32) FORCE
$(call if_changed,vdso32ld)
# strip rule for the .so file
@@ -41,12 +41,12 @@
$(call if_changed,objcopy)
# assembly rules for the .S files
-$(obj-vdso32): %.o: %.S
+$(obj-vdso32): %.o: %.S FORCE
$(call if_changed_dep,vdso32as)
# actual build commands
quiet_cmd_vdso32ld = VDSO32L $@
- cmd_vdso32ld = $(CC) $(c_flags) -Wl,-T $^ -o $@
+ cmd_vdso32ld = $(CC) $(c_flags) -Wl,-T $(filter %.lds %.o,$^) -o $@
quiet_cmd_vdso32as = VDSO32A $@
cmd_vdso32as = $(CC) $(a_flags) -c -o $@ $<
diff --git a/arch/s390/kernel/vdso64/Makefile b/arch/s390/kernel/vdso64/Makefile
index 2d54c18..84af2b6 100644
--- a/arch/s390/kernel/vdso64/Makefile
+++ b/arch/s390/kernel/vdso64/Makefile
@@ -32,7 +32,7 @@
$(obj)/vdso64_wrapper.o : $(obj)/vdso64.so
# link rule for the .so file, .lds has to be first
-$(obj)/vdso64.so.dbg: $(src)/vdso64.lds $(obj-vdso64)
+$(obj)/vdso64.so.dbg: $(src)/vdso64.lds $(obj-vdso64) FORCE
$(call if_changed,vdso64ld)
# strip rule for the .so file
@@ -41,12 +41,12 @@
$(call if_changed,objcopy)
# assembly rules for the .S files
-$(obj-vdso64): %.o: %.S
+$(obj-vdso64): %.o: %.S FORCE
$(call if_changed_dep,vdso64as)
# actual build commands
quiet_cmd_vdso64ld = VDSO64L $@
- cmd_vdso64ld = $(CC) $(c_flags) -Wl,-T $^ -o $@
+ cmd_vdso64ld = $(CC) $(c_flags) -Wl,-T $(filter %.lds %.o,$^) -o $@
quiet_cmd_vdso64as = VDSO64A $@
cmd_vdso64as = $(CC) $(a_flags) -c -o $@ $<
diff --git a/arch/s390/kernel/vmlinux.lds.S b/arch/s390/kernel/vmlinux.lds.S
index 115bda2..dd96b46 100644
--- a/arch/s390/kernel/vmlinux.lds.S
+++ b/arch/s390/kernel/vmlinux.lds.S
@@ -99,6 +99,43 @@
EXIT_DATA
}
+ /*
+ * struct alt_inst entries. From the header (alternative.h):
+ * "Alternative instructions for different CPU types or capabilities"
+ * Think locking instructions on spinlocks.
+ * Note, that it is a part of __init region.
+ */
+ . = ALIGN(8);
+ .altinstructions : {
+ __alt_instructions = .;
+ *(.altinstructions)
+ __alt_instructions_end = .;
+ }
+
+ /*
+ * And here are the replacement instructions. The linker sticks
+ * them as binary blobs. The .altinstructions has enough data to
+ * get the address and the length of them to patch the kernel safely.
+ * Note, that it is a part of __init region.
+ */
+ .altinstr_replacement : {
+ *(.altinstr_replacement)
+ }
+
+ /*
+ * Table with the patch locations to undo expolines
+ */
+ .nospec_call_table : {
+ __nospec_call_start = . ;
+ *(.s390_indirect*)
+ __nospec_call_end = . ;
+ }
+ .nospec_return_table : {
+ __nospec_return_start = . ;
+ *(.s390_return*)
+ __nospec_return_end = . ;
+ }
+
/* early.c uses stsi, which requires page aligned data. */
. = ALIGN(PAGE_SIZE);
INIT_DATA_SECTION(0x100)
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index a70ff09..2032ab8 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -401,6 +401,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_S390_RI:
r = test_facility(64);
break;
+ case KVM_CAP_S390_BPB:
+ r = test_facility(82);
+ break;
default:
r = 0;
}
@@ -1713,6 +1716,8 @@ int kvm_arch_vcpu_init(struct kvm_vcpu *vcpu)
kvm_s390_set_prefix(vcpu, 0);
if (test_kvm_facility(vcpu->kvm, 64))
vcpu->run->kvm_valid_regs |= KVM_SYNC_RICCB;
+ if (test_kvm_facility(vcpu->kvm, 82))
+ vcpu->run->kvm_valid_regs |= KVM_SYNC_BPBC;
/* fprs can be synchronized via vrs, even if the guest has no vx. With
* MACHINE_HAS_VX, (load|store)_fpu_regs() will work with vrs format.
*/
@@ -1829,7 +1834,6 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
if (test_fp_ctl(current->thread.fpu.fpc))
/* User space provided an invalid FPC, let's clear it */
current->thread.fpu.fpc = 0;
-
save_access_regs(vcpu->arch.host_acrs);
restore_access_regs(vcpu->run->s.regs.acrs);
gmap_enable(vcpu->arch.enabled_gmap);
@@ -1877,6 +1881,7 @@ static void kvm_s390_vcpu_initial_reset(struct kvm_vcpu *vcpu)
current->thread.fpu.fpc = 0;
vcpu->arch.sie_block->gbea = 1;
vcpu->arch.sie_block->pp = 0;
+ vcpu->arch.sie_block->fpf &= ~FPF_BPBC;
vcpu->arch.pfault_token = KVM_S390_PFAULT_TOKEN_INVALID;
kvm_clear_async_pf_completion_queue(vcpu);
if (!kvm_s390_user_cpu_state_ctrl(vcpu->kvm))
@@ -2744,6 +2749,11 @@ static void sync_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
if (riccb->valid)
vcpu->arch.sie_block->ecb3 |= 0x01;
}
+ if ((kvm_run->kvm_dirty_regs & KVM_SYNC_BPBC) &&
+ test_kvm_facility(vcpu->kvm, 82)) {
+ vcpu->arch.sie_block->fpf &= ~FPF_BPBC;
+ vcpu->arch.sie_block->fpf |= kvm_run->s.regs.bpbc ? FPF_BPBC : 0;
+ }
kvm_run->kvm_dirty_regs = 0;
}
@@ -2762,6 +2772,7 @@ static void store_regs(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
kvm_run->s.regs.pft = vcpu->arch.pfault_token;
kvm_run->s.regs.pfs = vcpu->arch.pfault_select;
kvm_run->s.regs.pfc = vcpu->arch.pfault_compare;
+ kvm_run->s.regs.bpbc = (vcpu->arch.sie_block->fpf & FPF_BPBC) == FPF_BPBC;
}
int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *kvm_run)
diff --git a/arch/s390/kvm/sthyi.c b/arch/s390/kvm/sthyi.c
index 2f04ad1..029fd5e 100644
--- a/arch/s390/kvm/sthyi.c
+++ b/arch/s390/kvm/sthyi.c
@@ -174,17 +174,19 @@ static void fill_hdr(struct sthyi_sctns *sctns)
static void fill_stsi_mac(struct sthyi_sctns *sctns,
struct sysinfo_1_1_1 *sysinfo)
{
+ sclp_ocf_cpc_name_copy(sctns->mac.infmname);
+ if (*(u64 *)sctns->mac.infmname != 0)
+ sctns->mac.infmval1 |= MAC_NAME_VLD;
+
if (stsi(sysinfo, 1, 1, 1))
return;
- sclp_ocf_cpc_name_copy(sctns->mac.infmname);
-
memcpy(sctns->mac.infmtype, sysinfo->type, sizeof(sctns->mac.infmtype));
memcpy(sctns->mac.infmmanu, sysinfo->manufacturer, sizeof(sctns->mac.infmmanu));
memcpy(sctns->mac.infmpman, sysinfo->plant, sizeof(sctns->mac.infmpman));
memcpy(sctns->mac.infmseq, sysinfo->sequence, sizeof(sctns->mac.infmseq));
- sctns->mac.infmval1 |= MAC_ID_VLD | MAC_NAME_VLD;
+ sctns->mac.infmval1 |= MAC_ID_VLD;
}
static void fill_stsi_par(struct sthyi_sctns *sctns,
diff --git a/arch/s390/kvm/vsie.c b/arch/s390/kvm/vsie.c
index d8673e2..da246d9 100644
--- a/arch/s390/kvm/vsie.c
+++ b/arch/s390/kvm/vsie.c
@@ -156,7 +156,8 @@ static int shadow_crycb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
return set_validity_icpt(scb_s, 0x0039U);
/* copy only the wrapping keys */
- if (read_guest_real(vcpu, crycb_addr + 72, &vsie_page->crycb, 56))
+ if (read_guest_real(vcpu, crycb_addr + 72,
+ vsie_page->crycb.dea_wrapping_key_mask, 56))
return set_validity_icpt(scb_s, 0x0035U);
scb_s->ecb3 |= ecb3_flags;
@@ -217,6 +218,12 @@ static void unshadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
memcpy(scb_o->gcr, scb_s->gcr, 128);
scb_o->pp = scb_s->pp;
+ /* branch prediction */
+ if (test_kvm_facility(vcpu->kvm, 82)) {
+ scb_o->fpf &= ~FPF_BPBC;
+ scb_o->fpf |= scb_s->fpf & FPF_BPBC;
+ }
+
/* interrupt intercept */
switch (scb_s->icptcode) {
case ICPT_PROGI:
@@ -259,6 +266,7 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
scb_s->ecb3 = 0;
scb_s->ecd = 0;
scb_s->fac = 0;
+ scb_s->fpf = 0;
rc = prepare_cpuflags(vcpu, vsie_page);
if (rc)
@@ -316,6 +324,9 @@ static int shadow_scb(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
prefix_unmapped(vsie_page);
scb_s->ecb |= scb_o->ecb & 0x10U;
}
+ /* branch prediction */
+ if (test_kvm_facility(vcpu->kvm, 82))
+ scb_s->fpf |= scb_o->fpf & FPF_BPBC;
/* SIMD */
if (test_kvm_facility(vcpu->kvm, 129)) {
scb_s->eca |= scb_o->eca & 0x00020000U;
@@ -539,7 +550,7 @@ static int pin_blocks(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
gpa = scb_o->itdba & ~0xffUL;
if (gpa && (scb_s->ecb & 0x10U)) {
- if (!(gpa & ~0x1fffU)) {
+ if (!(gpa & ~0x1fffUL)) {
rc = set_validity_icpt(scb_s, 0x0080U);
goto unpin;
}
@@ -754,6 +765,7 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
{
struct kvm_s390_sie_block *scb_s = &vsie_page->scb_s;
struct kvm_s390_sie_block *scb_o = vsie_page->scb_o;
+ int guest_bp_isolation;
int rc;
handle_last_fault(vcpu, vsie_page);
@@ -764,6 +776,20 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
s390_handle_mcck();
srcu_read_unlock(&vcpu->kvm->srcu, vcpu->srcu_idx);
+
+ /* save current guest state of bp isolation override */
+ guest_bp_isolation = test_thread_flag(TIF_ISOLATE_BP_GUEST);
+
+ /*
+ * The guest is running with BPBC, so we have to force it on for our
+ * nested guest. This is done by enabling BPBC globally, so the BPBC
+ * control in the SCB (which the nested guest can modify) is simply
+ * ignored.
+ */
+ if (test_kvm_facility(vcpu->kvm, 82) &&
+ vcpu->arch.sie_block->fpf & FPF_BPBC)
+ set_thread_flag(TIF_ISOLATE_BP_GUEST);
+
local_irq_disable();
guest_enter_irqoff();
local_irq_enable();
@@ -773,6 +799,11 @@ static int do_vsie_run(struct kvm_vcpu *vcpu, struct vsie_page *vsie_page)
local_irq_disable();
guest_exit_irqoff();
local_irq_enable();
+
+ /* restore guest state for bp isolation override */
+ if (!guest_bp_isolation)
+ clear_thread_flag(TIF_ISOLATE_BP_GUEST);
+
vcpu->srcu_idx = srcu_read_lock(&vcpu->kvm->srcu);
if (rc > 0)
diff --git a/arch/s390/lib/mem.S b/arch/s390/lib/mem.S
index be9fa65..5ff0520 100644
--- a/arch/s390/lib/mem.S
+++ b/arch/s390/lib/mem.S
@@ -6,6 +6,9 @@
#include <linux/linkage.h>
#include <asm/export.h>
+#include <asm/nospec-insn.h>
+
+ GEN_BR_THUNK %r14
/*
* memset implementation
@@ -24,7 +27,7 @@
*/
ENTRY(memset)
ltgr %r4,%r4
- bzr %r14
+ jz .Lmemset_exit
ltgr %r3,%r3
jnz .Lmemset_fill
aghi %r4,-1
@@ -39,12 +42,13 @@
.Lmemset_clear_rest:
larl %r3,.Lmemset_xc
ex %r4,0(%r3)
- br %r14
+.Lmemset_exit:
+ BR_EX %r14
.Lmemset_fill:
stc %r3,0(%r2)
cghi %r4,1
lgr %r1,%r2
- ber %r14
+ je .Lmemset_fill_exit
aghi %r4,-2
srlg %r3,%r4,8
ltgr %r3,%r3
@@ -56,7 +60,8 @@
.Lmemset_fill_rest:
larl %r3,.Lmemset_mvc
ex %r4,0(%r3)
- br %r14
+.Lmemset_fill_exit:
+ BR_EX %r14
.Lmemset_xc:
xc 0(1,%r1),0(%r1)
.Lmemset_mvc:
@@ -70,7 +75,7 @@
*/
ENTRY(memcpy)
ltgr %r4,%r4
- bzr %r14
+ jz .Lmemcpy_exit
aghi %r4,-1
srlg %r5,%r4,8
ltgr %r5,%r5
@@ -79,7 +84,8 @@
.Lmemcpy_rest:
larl %r5,.Lmemcpy_mvc
ex %r4,0(%r5)
- br %r14
+.Lmemcpy_exit:
+ BR_EX %r14
.Lmemcpy_loop:
mvc 0(256,%r1),0(%r3)
la %r1,256(%r1)
diff --git a/arch/s390/mm/extmem.c b/arch/s390/mm/extmem.c
index 02042b6..e6665a6 100644
--- a/arch/s390/mm/extmem.c
+++ b/arch/s390/mm/extmem.c
@@ -79,7 +79,7 @@ struct qin64 {
struct dcss_segment {
struct list_head list;
char dcss_name[8];
- char res_name[15];
+ char res_name[16];
unsigned long start_addr;
unsigned long end;
atomic_t ref_count;
@@ -432,7 +432,7 @@ __segment_load (char *name, int do_nonshared, unsigned long *addr, unsigned long
memcpy(&seg->res_name, seg->dcss_name, 8);
EBCASC(seg->res_name, 8);
seg->res_name[8] = '\0';
- strncat(seg->res_name, " (DCSS)", 7);
+ strlcat(seg->res_name, " (DCSS)", sizeof(seg->res_name));
seg->res->name = seg->res_name;
rc = seg->vm_segtype;
if (rc == SEG_TYPE_SC ||
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index 661d9fe..ba2f218 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -462,6 +462,8 @@ static inline int do_exception(struct pt_regs *regs, int access)
/* No reason to continue if interrupted by SIGKILL. */
if ((fault & VM_FAULT_RETRY) && fatal_signal_pending(current)) {
fault = VM_FAULT_SIGNAL;
+ if (flags & FAULT_FLAG_RETRY_NOWAIT)
+ goto out_up;
goto out;
}
if (unlikely(fault & VM_FAULT_ERROR))
diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c
index cb2cd04..b6c85b7 100644
--- a/arch/s390/mm/gmap.c
+++ b/arch/s390/mm/gmap.c
@@ -686,6 +686,8 @@ void gmap_discard(struct gmap *gmap, unsigned long from, unsigned long to)
vmaddr |= gaddr & ~PMD_MASK;
/* Find vma in the parent mm */
vma = find_vma(gmap->mm, vmaddr);
+ if (!vma)
+ continue;
size = min(to - gaddr, PMD_SIZE - (gaddr & ~PMD_MASK));
zap_page_range(vma, vmaddr, size, NULL);
}
diff --git a/arch/s390/mm/pgalloc.c b/arch/s390/mm/pgalloc.c
index 995f785..781a044 100644
--- a/arch/s390/mm/pgalloc.c
+++ b/arch/s390/mm/pgalloc.c
@@ -26,7 +26,7 @@ static struct ctl_table page_table_sysctl[] = {
.data = &page_table_allocate_pgste,
.maxlen = sizeof(int),
.mode = S_IRUGO | S_IWUSR,
- .proc_handler = proc_dointvec,
+ .proc_handler = proc_dointvec_minmax,
.extra1 = &page_table_allocate_pgste_min,
.extra2 = &page_table_allocate_pgste_max,
},
diff --git a/arch/s390/net/bpf_jit.S b/arch/s390/net/bpf_jit.S
index a1c917d..fa716f2 100644
--- a/arch/s390/net/bpf_jit.S
+++ b/arch/s390/net/bpf_jit.S
@@ -8,6 +8,7 @@
*/
#include <linux/linkage.h>
+#include <asm/nospec-insn.h>
#include "bpf_jit.h"
/*
@@ -53,7 +54,7 @@
clg %r3,STK_OFF_HLEN(%r15); /* Offset + SIZE > hlen? */ \
jh sk_load_##NAME##_slow; \
LOAD %r14,-SIZE(%r3,%r12); /* Get data from skb */ \
- b OFF_OK(%r6); /* Return */ \
+ B_EX OFF_OK,%r6; /* Return */ \
\
sk_load_##NAME##_slow:; \
lgr %r2,%r7; /* Arg1 = skb pointer */ \
@@ -63,11 +64,14 @@
brasl %r14,skb_copy_bits; /* Get data from skb */ \
LOAD %r14,STK_OFF_TMP(%r15); /* Load from temp bufffer */ \
ltgr %r2,%r2; /* Set cc to (%r2 != 0) */ \
- br %r6; /* Return */
+ BR_EX %r6; /* Return */
sk_load_common(word, 4, llgf) /* r14 = *(u32 *) (skb->data+offset) */
sk_load_common(half, 2, llgh) /* r14 = *(u16 *) (skb->data+offset) */
+ GEN_BR_THUNK %r6
+ GEN_B_THUNK OFF_OK,%r6
+
/*
* Load 1 byte from SKB (optimized version)
*/
@@ -79,7 +83,7 @@
clg %r3,STK_OFF_HLEN(%r15) # Offset >= hlen?
jnl sk_load_byte_slow
llgc %r14,0(%r3,%r12) # Get byte from skb
- b OFF_OK(%r6) # Return OK
+ B_EX OFF_OK,%r6 # Return OK
sk_load_byte_slow:
lgr %r2,%r7 # Arg1 = skb pointer
@@ -89,7 +93,7 @@
brasl %r14,skb_copy_bits # Get data from skb
llgc %r14,STK_OFF_TMP(%r15) # Load result from temp buffer
ltgr %r2,%r2 # Set cc to (%r2 != 0)
- br %r6 # Return cc
+ BR_EX %r6 # Return cc
#define sk_negative_common(NAME, SIZE, LOAD) \
sk_load_##NAME##_slow_neg:; \
@@ -103,7 +107,7 @@
jz bpf_error; \
LOAD %r14,0(%r2); /* Get data from pointer */ \
xr %r3,%r3; /* Set cc to zero */ \
- br %r6; /* Return cc */
+ BR_EX %r6; /* Return cc */
sk_negative_common(word, 4, llgf)
sk_negative_common(half, 2, llgh)
@@ -112,4 +116,4 @@
bpf_error:
# force a return 0 from jit handler
ltgr %r15,%r15 # Set condition code
- br %r6
+ BR_EX %r6
diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c
index e8dee62..8bd25ae 100644
--- a/arch/s390/net/bpf_jit_comp.c
+++ b/arch/s390/net/bpf_jit_comp.c
@@ -24,6 +24,8 @@
#include <linux/bpf.h>
#include <asm/cacheflush.h>
#include <asm/dis.h>
+#include <asm/facility.h>
+#include <asm/nospec-branch.h>
#include "bpf_jit.h"
int bpf_jit_enable __read_mostly;
@@ -41,6 +43,8 @@ struct bpf_jit {
int base_ip; /* Base address for literal pool */
int ret0_ip; /* Address of return 0 */
int exit_ip; /* Address of exit */
+ int r1_thunk_ip; /* Address of expoline thunk for 'br %r1' */
+ int r14_thunk_ip; /* Address of expoline thunk for 'br %r14' */
int tail_call_start; /* Tail call start offset */
int labels[1]; /* Labels for local jumps */
};
@@ -251,6 +255,19 @@ static inline void reg_set_seen(struct bpf_jit *jit, u32 b1)
REG_SET_SEEN(b2); \
})
+#define EMIT6_PCREL_RILB(op, b, target) \
+({ \
+ int rel = (target - jit->prg) / 2; \
+ _EMIT6(op | reg_high(b) << 16 | rel >> 16, rel & 0xffff); \
+ REG_SET_SEEN(b); \
+})
+
+#define EMIT6_PCREL_RIL(op, target) \
+({ \
+ int rel = (target - jit->prg) / 2; \
+ _EMIT6(op | rel >> 16, rel & 0xffff); \
+})
+
#define _EMIT6_IMM(op, imm) \
({ \
unsigned int __imm = (imm); \
@@ -470,8 +487,43 @@ static void bpf_jit_epilogue(struct bpf_jit *jit)
EMIT4(0xb9040000, REG_2, BPF_REG_0);
/* Restore registers */
save_restore_regs(jit, REGS_RESTORE);
+ if (IS_ENABLED(CC_USING_EXPOLINE) && !nospec_disable) {
+ jit->r14_thunk_ip = jit->prg;
+ /* Generate __s390_indirect_jump_r14 thunk */
+ if (test_facility(35)) {
+ /* exrl %r0,.+10 */
+ EMIT6_PCREL_RIL(0xc6000000, jit->prg + 10);
+ } else {
+ /* larl %r1,.+14 */
+ EMIT6_PCREL_RILB(0xc0000000, REG_1, jit->prg + 14);
+ /* ex 0,0(%r1) */
+ EMIT4_DISP(0x44000000, REG_0, REG_1, 0);
+ }
+ /* j . */
+ EMIT4_PCREL(0xa7f40000, 0);
+ }
/* br %r14 */
_EMIT2(0x07fe);
+
+ if (IS_ENABLED(CC_USING_EXPOLINE) && !nospec_disable &&
+ (jit->seen & SEEN_FUNC)) {
+ jit->r1_thunk_ip = jit->prg;
+ /* Generate __s390_indirect_jump_r1 thunk */
+ if (test_facility(35)) {
+ /* exrl %r0,.+10 */
+ EMIT6_PCREL_RIL(0xc6000000, jit->prg + 10);
+ /* j . */
+ EMIT4_PCREL(0xa7f40000, 0);
+ /* br %r1 */
+ _EMIT2(0x07f1);
+ } else {
+ /* ex 0,S390_lowcore.br_r1_tampoline */
+ EMIT4_DISP(0x44000000, REG_0, REG_0,
+ offsetof(struct lowcore, br_r1_trampoline));
+ /* j . */
+ EMIT4_PCREL(0xa7f40000, 0);
+ }
+ }
}
/*
@@ -977,8 +1029,13 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, int i
/* lg %w1,<d(imm)>(%l) */
EMIT6_DISP_LH(0xe3000000, 0x0004, REG_W1, REG_0, REG_L,
EMIT_CONST_U64(func));
- /* basr %r14,%w1 */
- EMIT2(0x0d00, REG_14, REG_W1);
+ if (IS_ENABLED(CC_USING_EXPOLINE) && !nospec_disable) {
+ /* brasl %r14,__s390_indirect_jump_r1 */
+ EMIT6_PCREL_RILB(0xc0050000, REG_14, jit->r1_thunk_ip);
+ } else {
+ /* basr %r14,%w1 */
+ EMIT2(0x0d00, REG_14, REG_W1);
+ }
/* lgr %b0,%r2: load return value into %b0 */
EMIT4(0xb9040000, BPF_REG_0, REG_2);
if (bpf_helper_changes_skb_data((void *)func)) {
@@ -1327,6 +1384,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp)
goto free_addrs;
}
if (bpf_jit_prog(&jit, fp)) {
+ bpf_jit_binary_free(header);
fp = orig_fp;
goto free_addrs;
}
diff --git a/arch/s390/numa/numa.c b/arch/s390/numa/numa.c
index f576f10..e73a116 100644
--- a/arch/s390/numa/numa.c
+++ b/arch/s390/numa/numa.c
@@ -53,6 +53,7 @@ int __node_distance(int a, int b)
{
return mode->distance ? mode->distance(a, b) : 0;
}
+EXPORT_SYMBOL(__node_distance);
int numa_debug_enabled;
@@ -133,6 +134,8 @@ void __init numa_setup(void)
{
pr_info("NUMA mode: %s\n", mode->name);
nodes_clear(node_possible_map);
+ /* Initially attach all possible CPUs to node 0. */
+ cpumask_copy(&node_to_cpumask_map[0], cpu_possible_mask);
if (mode->setup)
mode->setup();
numa_setup_memory();
@@ -140,20 +143,6 @@ void __init numa_setup(void)
}
/*
- * numa_init_early() - Initialization initcall
- *
- * This runs when only one CPU is online and before the first
- * topology update is called for by the scheduler.
- */
-static int __init numa_init_early(void)
-{
- /* Attach all possible CPUs to node 0 for now. */
- cpumask_copy(&node_to_cpumask_map[0], cpu_possible_mask);
- return 0;
-}
-early_initcall(numa_init_early);
-
-/*
* numa_init_late() - Initialization initcall
*
* Register NUMA nodes.
diff --git a/arch/s390/pci/pci.c b/arch/s390/pci/pci.c
index 03a1d59..8757411 100644
--- a/arch/s390/pci/pci.c
+++ b/arch/s390/pci/pci.c
@@ -407,6 +407,8 @@ int arch_setup_msi_irqs(struct pci_dev *pdev, int nvec, int type)
hwirq = 0;
for_each_pci_msi_entry(msi, pdev) {
rc = -EIO;
+ if (hwirq >= msi_vecs)
+ break;
irq = irq_alloc_desc(0); /* Alloc irq on node 0 */
if (irq < 0)
goto out_msi;
diff --git a/arch/sh/drivers/heartbeat.c b/arch/sh/drivers/heartbeat.c
index 49bace4..c6d9604 100644
--- a/arch/sh/drivers/heartbeat.c
+++ b/arch/sh/drivers/heartbeat.c
@@ -21,6 +21,7 @@
#include <linux/init.h>
#include <linux/platform_device.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/timer.h>
#include <linux/io.h>
#include <linux/slab.h>
diff --git a/arch/sh/include/asm/futex.h b/arch/sh/include/asm/futex.h
index d007874..8f8cf94 100644
--- a/arch/sh/include/asm/futex.h
+++ b/arch/sh/include/asm/futex.h
@@ -27,21 +27,12 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
return atomic_futex_op_cmpxchg_inatomic(uval, uaddr, oldval, newval);
}
-static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+static inline int arch_futex_atomic_op_inuser(int op, u32 oparg, int *oval,
+ u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- u32 oparg = (encoded_op << 8) >> 20;
- u32 cmparg = (encoded_op << 20) >> 20;
u32 oldval, newval, prev;
int ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
-
pagefault_disable();
do {
@@ -80,17 +71,8 @@ static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = ((int)oldval < (int)cmparg); break;
- case FUTEX_OP_CMP_GE: ret = ((int)oldval >= (int)cmparg); break;
- case FUTEX_OP_CMP_LE: ret = ((int)oldval <= (int)cmparg); break;
- case FUTEX_OP_CMP_GT: ret = ((int)oldval > (int)cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
return ret;
}
diff --git a/arch/sh/kernel/entry-common.S b/arch/sh/kernel/entry-common.S
index c001f78..28cc612 100644
--- a/arch/sh/kernel/entry-common.S
+++ b/arch/sh/kernel/entry-common.S
@@ -255,7 +255,7 @@
mov.l @r8, r8
jsr @r8
nop
- bra __restore_all
+ bra ret_from_exception
nop
CFI_ENDPROC
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 8b4152f..cef42d4 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -290,9 +290,13 @@
depends on SPARC64 && SMP
config NODES_SHIFT
- int
- default "4"
+ int "Maximum NUMA Nodes (as a power of 2)"
+ range 4 5 if SPARC64
+ default "5"
depends on NEED_MULTIPLE_NODES
+ help
+ Specify the maximum number of NUMA Nodes available on the target
+ system. Increases memory reserved to accommodate various tables.
# Some NUMA nodes have memory ranges that span
# other nodes. Even though a pfn is valid and
diff --git a/arch/sparc/include/asm/atomic_64.h b/arch/sparc/include/asm/atomic_64.h
index 24827a3..89d299c 100644
--- a/arch/sparc/include/asm/atomic_64.h
+++ b/arch/sparc/include/asm/atomic_64.h
@@ -82,7 +82,11 @@ ATOMIC_OPS(xor)
#define atomic64_add_negative(i, v) (atomic64_add_return(i, v) < 0)
#define atomic_cmpxchg(v, o, n) (cmpxchg(&((v)->counter), (o), (n)))
-#define atomic_xchg(v, new) (xchg(&((v)->counter), new))
+
+static inline int atomic_xchg(atomic_t *v, int new)
+{
+ return xchg(&v->counter, new);
+}
static inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
diff --git a/arch/sparc/include/asm/cpudata_64.h b/arch/sparc/include/asm/cpudata_64.h
index 5b0ed48..aa2bf90 100644
--- a/arch/sparc/include/asm/cpudata_64.h
+++ b/arch/sparc/include/asm/cpudata_64.h
@@ -27,7 +27,7 @@ typedef struct {
unsigned short sock_id; /* physical package */
unsigned short core_id;
unsigned short max_cache_id; /* groupings of highest shared cache */
- unsigned short proc_id; /* strand (aka HW thread) id */
+ signed short proc_id; /* strand (aka HW thread) id */
} cpuinfo_sparc;
DECLARE_PER_CPU(cpuinfo_sparc, __cpu_data);
diff --git a/arch/sparc/include/asm/futex_64.h b/arch/sparc/include/asm/futex_64.h
index 4e899b0..1cfd89d 100644
--- a/arch/sparc/include/asm/futex_64.h
+++ b/arch/sparc/include/asm/futex_64.h
@@ -29,22 +29,14 @@
: "r" (uaddr), "r" (oparg), "i" (-EFAULT) \
: "memory")
-static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval,
+ u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret, tem;
- if (unlikely(!access_ok(VERIFY_WRITE, uaddr, sizeof(u32))))
- return -EFAULT;
if (unlikely((((unsigned long) uaddr) & 0x3UL)))
return -EINVAL;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
pagefault_disable();
switch (op) {
@@ -69,17 +61,9 @@ static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/sparc/include/asm/pgtable_64.h b/arch/sparc/include/asm/pgtable_64.h
index b6802b9..81ad06a 100644
--- a/arch/sparc/include/asm/pgtable_64.h
+++ b/arch/sparc/include/asm/pgtable_64.h
@@ -952,7 +952,7 @@ void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr,
pmd_t *pmd);
#define __HAVE_ARCH_PMDP_INVALIDATE
-extern void pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
+extern pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
pmd_t *pmdp);
#define __HAVE_ARCH_PGTABLE_DEPOSIT
diff --git a/arch/sparc/kernel/ds.c b/arch/sparc/kernel/ds.c
index f87a55d..9b3f2e2 100644
--- a/arch/sparc/kernel/ds.c
+++ b/arch/sparc/kernel/ds.c
@@ -908,7 +908,7 @@ static int register_services(struct ds_info *dp)
pbuf.req.handle = cp->handle;
pbuf.req.major = 1;
pbuf.req.minor = 0;
- strcpy(pbuf.req.svc_id, cp->service_id);
+ strcpy(pbuf.id_buf, cp->service_id);
err = __ds_send(lp, &pbuf, msg_len);
if (err > 0)
diff --git a/arch/sparc/kernel/led.c b/arch/sparc/kernel/led.c
index 3ae36f3..44a3ed9 100644
--- a/arch/sparc/kernel/led.c
+++ b/arch/sparc/kernel/led.c
@@ -8,6 +8,7 @@
#include <linux/jiffies.h>
#include <linux/timer.h>
#include <linux/uaccess.h>
+#include <linux/sched/loadavg.h>
#include <asm/auxio.h>
diff --git a/arch/sparc/kernel/pcic.c b/arch/sparc/kernel/pcic.c
index 24384e1..a7aeb03 100644
--- a/arch/sparc/kernel/pcic.c
+++ b/arch/sparc/kernel/pcic.c
@@ -602,7 +602,7 @@ void pcibios_fixup_bus(struct pci_bus *bus)
{
struct pci_dev *dev;
int i, has_io, has_mem;
- unsigned int cmd;
+ unsigned int cmd = 0;
struct linux_pcic *pcic;
/* struct linux_pbm_info* pbm = &pcic->pbm; */
int node;
diff --git a/arch/sparc/kernel/perf_event.c b/arch/sparc/kernel/perf_event.c
index 710f327..71e7f77 100644
--- a/arch/sparc/kernel/perf_event.c
+++ b/arch/sparc/kernel/perf_event.c
@@ -926,6 +926,8 @@ static void read_in_all_counters(struct cpu_hw_events *cpuc)
sparc_perf_event_update(cp, &cp->hw,
cpuc->current_idx[i]);
cpuc->current_idx[i] = PIC_NO_INDEX;
+ if (cp->hw.state & PERF_HES_STOPPED)
+ cp->hw.state |= PERF_HES_ARCH;
}
}
}
@@ -958,10 +960,12 @@ static void calculate_single_pcr(struct cpu_hw_events *cpuc)
enc = perf_event_get_enc(cpuc->events[i]);
cpuc->pcr[0] &= ~mask_for_index(idx);
- if (hwc->state & PERF_HES_STOPPED)
+ if (hwc->state & PERF_HES_ARCH) {
cpuc->pcr[0] |= nop_for_index(idx);
- else
+ } else {
cpuc->pcr[0] |= event_encoding(enc, idx);
+ hwc->state = 0;
+ }
}
out:
cpuc->pcr[0] |= cpuc->event[0]->hw.config_base;
@@ -987,6 +991,9 @@ static void calculate_multiple_pcrs(struct cpu_hw_events *cpuc)
cpuc->current_idx[i] = idx;
+ if (cp->hw.state & PERF_HES_ARCH)
+ continue;
+
sparc_pmu_start(cp, PERF_EF_RELOAD);
}
out:
@@ -1078,6 +1085,8 @@ static void sparc_pmu_start(struct perf_event *event, int flags)
event->hw.state = 0;
sparc_pmu_enable_event(cpuc, &event->hw, idx);
+
+ perf_event_update_userpage(event);
}
static void sparc_pmu_stop(struct perf_event *event, int flags)
@@ -1370,9 +1379,9 @@ static int sparc_pmu_add(struct perf_event *event, int ef_flags)
cpuc->events[n0] = event->hw.event_base;
cpuc->current_idx[n0] = PIC_NO_INDEX;
- event->hw.state = PERF_HES_UPTODATE;
+ event->hw.state = PERF_HES_UPTODATE | PERF_HES_STOPPED;
if (!(ef_flags & PERF_EF_START))
- event->hw.state |= PERF_HES_STOPPED;
+ event->hw.state |= PERF_HES_ARCH;
/*
* If group events scheduling transaction was started,
diff --git a/arch/sparc/kernel/sys_sparc_32.c b/arch/sparc/kernel/sys_sparc_32.c
index 646988d..740f43b 100644
--- a/arch/sparc/kernel/sys_sparc_32.c
+++ b/arch/sparc/kernel/sys_sparc_32.c
@@ -201,23 +201,27 @@ SYSCALL_DEFINE5(rt_sigaction, int, sig,
asmlinkage long sys_getdomainname(char __user *name, int len)
{
- int nlen, err;
-
+ int nlen, err;
+ char tmp[__NEW_UTS_LEN + 1];
+
if (len < 0)
return -EINVAL;
- down_read(&uts_sem);
-
+ down_read(&uts_sem);
+
nlen = strlen(utsname()->domainname) + 1;
err = -EINVAL;
if (nlen > len)
- goto out;
+ goto out_unlock;
+ memcpy(tmp, utsname()->domainname, nlen);
- err = -EFAULT;
- if (!copy_to_user(name, utsname()->domainname, nlen))
- err = 0;
+ up_read(&uts_sem);
-out:
+ if (copy_to_user(name, tmp, nlen))
+ return -EFAULT;
+ return 0;
+
+out_unlock:
up_read(&uts_sem);
return err;
}
diff --git a/arch/sparc/kernel/sys_sparc_64.c b/arch/sparc/kernel/sys_sparc_64.c
index 02e05e2..ebecbc9 100644
--- a/arch/sparc/kernel/sys_sparc_64.c
+++ b/arch/sparc/kernel/sys_sparc_64.c
@@ -524,23 +524,27 @@ extern void check_pending(int signum);
SYSCALL_DEFINE2(getdomainname, char __user *, name, int, len)
{
- int nlen, err;
+ int nlen, err;
+ char tmp[__NEW_UTS_LEN + 1];
if (len < 0)
return -EINVAL;
- down_read(&uts_sem);
-
+ down_read(&uts_sem);
+
nlen = strlen(utsname()->domainname) + 1;
err = -EINVAL;
if (nlen > len)
- goto out;
+ goto out_unlock;
+ memcpy(tmp, utsname()->domainname, nlen);
- err = -EFAULT;
- if (!copy_to_user(name, utsname()->domainname, nlen))
- err = 0;
+ up_read(&uts_sem);
-out:
+ if (copy_to_user(name, tmp, nlen))
+ return -EFAULT;
+ return 0;
+
+out_unlock:
up_read(&uts_sem);
return err;
}
diff --git a/arch/sparc/lib/multi3.S b/arch/sparc/lib/multi3.S
index d6b6c97..703127a 100644
--- a/arch/sparc/lib/multi3.S
+++ b/arch/sparc/lib/multi3.S
@@ -5,26 +5,26 @@
.align 4
ENTRY(__multi3) /* %o0 = u, %o1 = v */
mov %o1, %g1
- srl %o3, 0, %g4
- mulx %g4, %g1, %o1
+ srl %o3, 0, %o4
+ mulx %o4, %g1, %o1
srlx %g1, 0x20, %g3
- mulx %g3, %g4, %g5
- sllx %g5, 0x20, %o5
- srl %g1, 0, %g4
+ mulx %g3, %o4, %g7
+ sllx %g7, 0x20, %o5
+ srl %g1, 0, %o4
sub %o1, %o5, %o5
srlx %o5, 0x20, %o5
- addcc %g5, %o5, %g5
+ addcc %g7, %o5, %g7
srlx %o3, 0x20, %o5
- mulx %g4, %o5, %g4
+ mulx %o4, %o5, %o4
mulx %g3, %o5, %o5
sethi %hi(0x80000000), %g3
- addcc %g5, %g4, %g5
- srlx %g5, 0x20, %g5
+ addcc %g7, %o4, %g7
+ srlx %g7, 0x20, %g7
add %g3, %g3, %g3
movcc %xcc, %g0, %g3
- addcc %o5, %g5, %o5
- sllx %g4, 0x20, %g4
- add %o1, %g4, %o1
+ addcc %o5, %g7, %o5
+ sllx %o4, 0x20, %o4
+ add %o1, %o4, %o1
add %o5, %g3, %g2
mulx %g1, %o2, %g1
add %g1, %g2, %g1
diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c
index c56a195..349cb83 100644
--- a/arch/sparc/mm/tlb.c
+++ b/arch/sparc/mm/tlb.c
@@ -163,13 +163,10 @@ static void tlb_batch_pmd_scan(struct mm_struct *mm, unsigned long vaddr,
pte_unmap(pte);
}
-void set_pmd_at(struct mm_struct *mm, unsigned long addr,
- pmd_t *pmdp, pmd_t pmd)
+
+static void __set_pmd_acct(struct mm_struct *mm, unsigned long addr,
+ pmd_t orig, pmd_t pmd)
{
- pmd_t orig = *pmdp;
-
- *pmdp = pmd;
-
if (mm == &init_mm)
return;
@@ -219,17 +216,38 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr,
}
}
+void set_pmd_at(struct mm_struct *mm, unsigned long addr,
+ pmd_t *pmdp, pmd_t pmd)
+{
+ pmd_t orig = *pmdp;
+
+ *pmdp = pmd;
+ __set_pmd_acct(mm, addr, orig, pmd);
+}
+
+static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmdp, pmd_t pmd)
+{
+ pmd_t old;
+
+ do {
+ old = *pmdp;
+ } while (cmpxchg64(&pmdp->pmd, old.pmd, pmd.pmd) != old.pmd);
+ __set_pmd_acct(vma->vm_mm, address, old, pmd);
+
+ return old;
+}
+
/*
* This routine is only called when splitting a THP
*/
-void pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
+pmd_t pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
pmd_t *pmdp)
{
- pmd_t entry = *pmdp;
+ pmd_t old, entry;
- pmd_val(entry) &= ~_PAGE_VALID;
-
- set_pmd_at(vma->vm_mm, address, pmdp, entry);
+ entry = __pmd(pmd_val(*pmdp) & ~_PAGE_VALID);
+ old = pmdp_establish(vma, address, pmdp, entry);
flush_tlb_range(vma, address, address + HPAGE_PMD_SIZE);
/*
@@ -240,6 +258,8 @@ void pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
if ((pmd_val(entry) & _PAGE_PMD_HUGE) &&
!is_huge_zero_page(pmd_page(entry)))
(vma->vm_mm)->context.thp_pte_count--;
+
+ return old;
}
void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
diff --git a/arch/tile/include/asm/futex.h b/arch/tile/include/asm/futex.h
index e64a1b7..83c1e63 100644
--- a/arch/tile/include/asm/futex.h
+++ b/arch/tile/include/asm/futex.h
@@ -106,12 +106,9 @@
lock = __atomic_hashed_lock((int __force *)uaddr)
#endif
-static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+static inline int arch_futex_atomic_op_inuser(int op, u32 oparg, int *oval,
+ u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int uninitialized_var(val), ret;
__futex_prolog();
@@ -119,12 +116,6 @@ static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
/* The 32-bit futex code makes this assumption, so validate it here. */
BUILD_BUG_ON(sizeof(atomic_t) != sizeof(int));
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
-
pagefault_disable();
switch (op) {
case FUTEX_OP_SET:
@@ -148,30 +139,9 @@ static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
}
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ:
- ret = (val == cmparg);
- break;
- case FUTEX_OP_CMP_NE:
- ret = (val != cmparg);
- break;
- case FUTEX_OP_CMP_LT:
- ret = (val < cmparg);
- break;
- case FUTEX_OP_CMP_GE:
- ret = (val >= cmparg);
- break;
- case FUTEX_OP_CMP_LE:
- ret = (val <= cmparg);
- break;
- case FUTEX_OP_CMP_GT:
- ret = (val > cmparg);
- break;
- default:
- ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = val;
+
return ret;
}
diff --git a/arch/um/os-Linux/skas/process.c b/arch/um/os-Linux/skas/process.c
index 23025d6..0a99d45 100644
--- a/arch/um/os-Linux/skas/process.c
+++ b/arch/um/os-Linux/skas/process.c
@@ -578,6 +578,11 @@ int start_idle_thread(void *stack, jmp_buf *switch_buf)
fatal_sigsegv();
}
longjmp(*switch_buf, 1);
+
+ /* unreachable */
+ printk(UM_KERN_ERR "impossible long jump!");
+ fatal_sigsegv();
+ return 0;
}
void initial_thread_cb_skas(void (*proc)(void *), void *arg)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 957731d..eb431d8 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -148,6 +148,7 @@
select HAVE_UID16 if X86_32 || IA32_EMULATION
select HAVE_UNSTABLE_SCHED_CLOCK
select HAVE_USER_RETURN_NOTIFIER
+ select HOTPLUG_SMT if SMP
select IRQ_FORCED_THREADING
select MODULES_USE_ELF_RELA if X86_64
select MODULES_USE_ELF_REL if X86_32
diff --git a/arch/x86/boot/compressed/Makefile b/arch/x86/boot/compressed/Makefile
index 4669b3a..cda8e14 100644
--- a/arch/x86/boot/compressed/Makefile
+++ b/arch/x86/boot/compressed/Makefile
@@ -101,9 +101,13 @@
done
endef
+# We need to run two commands under "if_changed", so merge them into a
+# single invocation.
+quiet_cmd_check-and-link-vmlinux = LD $@
+ cmd_check-and-link-vmlinux = $(cmd_check_data_rel); $(cmd_ld)
+
$(obj)/vmlinux: $(vmlinux-objs-y) FORCE
- $(call if_changed,check_data_rel)
- $(call if_changed,ld)
+ $(call if_changed,check-and-link-vmlinux)
OBJCOPYFLAGS_vmlinux.bin := -R .comment -S
$(obj)/vmlinux.bin: vmlinux FORCE
diff --git a/arch/x86/boot/compressed/eboot.c b/arch/x86/boot/compressed/eboot.c
index cc69e37..c0ad1bb 100644
--- a/arch/x86/boot/compressed/eboot.c
+++ b/arch/x86/boot/compressed/eboot.c
@@ -330,7 +330,8 @@ __setup_efi_pci32(efi_pci_io_protocol_32 *pci, struct pci_setup_rom **__rom)
if (status != EFI_SUCCESS)
goto free_struct;
- memcpy(rom->romdata, pci->romimage, pci->romsize);
+ memcpy(rom->romdata, (void *)(unsigned long)pci->romimage,
+ pci->romsize);
return status;
free_struct:
@@ -436,7 +437,8 @@ __setup_efi_pci64(efi_pci_io_protocol_64 *pci, struct pci_setup_rom **__rom)
if (status != EFI_SUCCESS)
goto free_struct;
- memcpy(rom->romdata, pci->romimage, pci->romsize);
+ memcpy(rom->romdata, (void *)(unsigned long)pci->romimage,
+ pci->romsize);
return status;
free_struct:
diff --git a/arch/x86/boot/tools/build.c b/arch/x86/boot/tools/build.c
index 0702d25..039c4a6 100644
--- a/arch/x86/boot/tools/build.c
+++ b/arch/x86/boot/tools/build.c
@@ -390,6 +390,13 @@ int main(int argc, char ** argv)
die("Unable to mmap '%s': %m", argv[2]);
/* Number of 16-byte paragraphs, including space for a 4-byte CRC */
sys_size = (sz + 15 + 4) / 16;
+#ifdef CONFIG_EFI_STUB
+ /*
+ * COFF requires minimum 32-byte alignment of sections, and
+ * adding a signature is problematic without that alignment.
+ */
+ sys_size = (sys_size + 1) & ~1;
+#endif
/* Patch the setup code with the appropriate size parameters */
buf[0x1f1] = setup_sectors-1;
diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index 34b3fa2..9e32d40 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -2,6 +2,8 @@
# Arch-specific CryptoAPI modules.
#
+OBJECT_FILES_NON_STANDARD := y
+
avx_supported := $(call as-instr,vpxor %xmm0$(comma)%xmm0$(comma)%xmm0,yes,no)
avx2_supported := $(call as-instr,vpgatherdd %ymm0$(comma)(%eax$(comma)%ymm1\
$(comma)4)$(comma)%ymm2,yes,no)
diff --git a/arch/x86/crypto/chacha20_glue.c b/arch/x86/crypto/chacha20_glue.c
index f910d1d..0a5fedf 100644
--- a/arch/x86/crypto/chacha20_glue.c
+++ b/arch/x86/crypto/chacha20_glue.c
@@ -77,6 +77,7 @@ static int chacha20_simd(struct blkcipher_desc *desc, struct scatterlist *dst,
blkcipher_walk_init(&walk, dst, src, nbytes);
err = blkcipher_walk_virt_block(desc, &walk, CHACHA20_BLOCK_SIZE);
+ desc->flags &= ~CRYPTO_TFM_REQ_MAY_SLEEP;
crypto_chacha20_init(state, crypto_blkcipher_ctx(desc->tfm), walk.iv);
diff --git a/arch/x86/crypto/crc32c-intel_glue.c b/arch/x86/crypto/crc32c-intel_glue.c
index 60a391b..5773e11 100644
--- a/arch/x86/crypto/crc32c-intel_glue.c
+++ b/arch/x86/crypto/crc32c-intel_glue.c
@@ -48,26 +48,13 @@
#ifdef CONFIG_X86_64
/*
* use carryless multiply version of crc32c when buffer
- * size is >= 512 (when eager fpu is enabled) or
- * >= 1024 (when eager fpu is disabled) to account
+ * size is >= 512 to account
* for fpu state save/restore overhead.
*/
-#define CRC32C_PCL_BREAKEVEN_EAGERFPU 512
-#define CRC32C_PCL_BREAKEVEN_NOEAGERFPU 1024
+#define CRC32C_PCL_BREAKEVEN 512
asmlinkage unsigned int crc_pcl(const u8 *buffer, int len,
unsigned int crc_init);
-static int crc32c_pcl_breakeven = CRC32C_PCL_BREAKEVEN_EAGERFPU;
-#if defined(X86_FEATURE_EAGER_FPU)
-#define set_pcl_breakeven_point() \
-do { \
- if (!use_eager_fpu()) \
- crc32c_pcl_breakeven = CRC32C_PCL_BREAKEVEN_NOEAGERFPU; \
-} while (0)
-#else
-#define set_pcl_breakeven_point() \
- (crc32c_pcl_breakeven = CRC32C_PCL_BREAKEVEN_NOEAGERFPU)
-#endif
#endif /* CONFIG_X86_64 */
static u32 crc32c_intel_le_hw_byte(u32 crc, unsigned char const *data, size_t length)
@@ -190,7 +177,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,
* use faster PCL version if datasize is large enough to
* overcome kernel fpu state save/restore overhead
*/
- if (len >= crc32c_pcl_breakeven && irq_fpu_usable()) {
+ if (len >= CRC32C_PCL_BREAKEVEN && irq_fpu_usable()) {
kernel_fpu_begin();
*crcp = crc_pcl(data, len, *crcp);
kernel_fpu_end();
@@ -202,7 +189,7 @@ static int crc32c_pcl_intel_update(struct shash_desc *desc, const u8 *data,
static int __crc32c_pcl_intel_finup(u32 *crcp, const u8 *data, unsigned int len,
u8 *out)
{
- if (len >= crc32c_pcl_breakeven && irq_fpu_usable()) {
+ if (len >= CRC32C_PCL_BREAKEVEN && irq_fpu_usable()) {
kernel_fpu_begin();
*(__le32 *)out = ~cpu_to_le32(crc_pcl(data, len, *crcp));
kernel_fpu_end();
@@ -262,7 +249,6 @@ static int __init crc32c_intel_mod_init(void)
alg.update = crc32c_pcl_intel_update;
alg.finup = crc32c_pcl_intel_finup;
alg.digest = crc32c_pcl_intel_digest;
- set_pcl_breakeven_point();
}
#endif
return crypto_register_shash(&alg);
diff --git a/arch/x86/crypto/sha1-mb/Makefile b/arch/x86/crypto/sha1-mb/Makefile
index 2f87563..2e14acc 100644
--- a/arch/x86/crypto/sha1-mb/Makefile
+++ b/arch/x86/crypto/sha1-mb/Makefile
@@ -2,6 +2,8 @@
# Arch-specific CryptoAPI modules.
#
+OBJECT_FILES_NON_STANDARD := y
+
avx2_supported := $(call as-instr,vpgatherdd %ymm0$(comma)(%eax$(comma)%ymm1\
$(comma)4)$(comma)%ymm2,yes,no)
ifeq ($(avx2_supported),yes)
diff --git a/arch/x86/crypto/sha256-mb/Makefile b/arch/x86/crypto/sha256-mb/Makefile
index 41089e7..45b4fca 100644
--- a/arch/x86/crypto/sha256-mb/Makefile
+++ b/arch/x86/crypto/sha256-mb/Makefile
@@ -2,6 +2,8 @@
# Arch-specific CryptoAPI modules.
#
+OBJECT_FILES_NON_STANDARD := y
+
avx2_supported := $(call as-instr,vpgatherdd %ymm0$(comma)(%eax$(comma)%ymm1\
$(comma)4)$(comma)%ymm2,yes,no)
ifeq ($(avx2_supported),yes)
diff --git a/arch/x86/crypto/sha256-mb/sha256_mb_mgr_flush_avx2.S b/arch/x86/crypto/sha256-mb/sha256_mb_mgr_flush_avx2.S
index ec9bee6..b7f5042 100644
--- a/arch/x86/crypto/sha256-mb/sha256_mb_mgr_flush_avx2.S
+++ b/arch/x86/crypto/sha256-mb/sha256_mb_mgr_flush_avx2.S
@@ -265,7 +265,7 @@
vpinsrd $1, _args_digest+1*32(state, idx, 4), %xmm0, %xmm0
vpinsrd $2, _args_digest+2*32(state, idx, 4), %xmm0, %xmm0
vpinsrd $3, _args_digest+3*32(state, idx, 4), %xmm0, %xmm0
- vmovd _args_digest(state , idx, 4) , %xmm0
+ vmovd _args_digest+4*32(state, idx, 4), %xmm1
vpinsrd $1, _args_digest+5*32(state, idx, 4), %xmm1, %xmm1
vpinsrd $2, _args_digest+6*32(state, idx, 4), %xmm1, %xmm1
vpinsrd $3, _args_digest+7*32(state, idx, 4), %xmm1, %xmm1
diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S
index 3a31fd4..d764992 100644
--- a/arch/x86/entry/entry_64.S
+++ b/arch/x86/entry/entry_64.S
@@ -91,7 +91,7 @@
.endm
.macro TRACE_IRQS_IRETQ_DEBUG
- bt $9, EFLAGS(%rsp) /* interrupts off? */
+ btl $9, EFLAGS(%rsp) /* interrupts off? */
jnc 1f
TRACE_IRQS_ON_DEBUG
1:
@@ -485,7 +485,7 @@
#ifdef CONFIG_PREEMPT
/* Interrupts are off */
/* Check if we need preemption */
- bt $9, EFLAGS(%rsp) /* were interrupts off? */
+ btl $9, EFLAGS(%rsp) /* were interrupts off? */
jnc 1f
0: cmpl $0, PER_CPU_VAR(__preempt_count)
jnz 1f
@@ -769,7 +769,7 @@
call \do_sym
- jmp error_exit /* %ebx: no swapgs flag */
+ jmp error_exit
.endif
END(\sym)
.endm
@@ -1038,7 +1038,6 @@
/*
* Save all registers in pt_regs, and switch gs if needed.
- * Return: EBX=0: came from user mode; EBX=1: otherwise
*/
ENTRY(error_entry)
cld
@@ -1051,7 +1050,6 @@
* the kernel CR3 here.
*/
SWITCH_KERNEL_CR3
- xorl %ebx, %ebx
testb $3, CS+8(%rsp)
jz .Lerror_kernelspace
@@ -1082,7 +1080,6 @@
* for these here too.
*/
.Lerror_kernelspace:
- incl %ebx
leaq native_irq_return_iret(%rip), %rcx
cmpq %rcx, RIP+8(%rsp)
je .Lerror_bad_iret
@@ -1114,28 +1111,19 @@
/*
* Pretend that the exception came from user mode: set up pt_regs
- * as if we faulted immediately after IRET and clear EBX so that
- * error_exit knows that we will be returning to user mode.
+ * as if we faulted immediately after IRET.
*/
mov %rsp, %rdi
call fixup_bad_iret
mov %rax, %rsp
- decl %ebx
jmp .Lerror_entry_from_usermode_after_swapgs
END(error_entry)
-
-/*
- * On entry, EBX is a "return to kernel mode" flag:
- * 1: already in kernel mode, don't need SWAPGS
- * 0: user gsbase is loaded, we need SWAPGS and standard preparation for return to usermode
- */
ENTRY(error_exit)
- movl %ebx, %eax
DISABLE_INTERRUPTS(CLBR_NONE)
TRACE_IRQS_OFF
- testl %eax, %eax
- jnz retint_kernel
+ testb $3, CS(%rsp)
+ jz retint_kernel
jmp retint_user
END(error_exit)
diff --git a/arch/x86/entry/vdso/vclock_gettime.c b/arch/x86/entry/vdso/vclock_gettime.c
index 02223cb..1e96709 100644
--- a/arch/x86/entry/vdso/vclock_gettime.c
+++ b/arch/x86/entry/vdso/vclock_gettime.c
@@ -37,8 +37,9 @@ extern u8 pvclock_page
notrace static long vdso_fallback_gettime(long clock, struct timespec *ts)
{
long ret;
- asm("syscall" : "=a" (ret) :
- "0" (__NR_clock_gettime), "D" (clock), "S" (ts) : "memory");
+ asm ("syscall" : "=a" (ret), "=m" (*ts) :
+ "0" (__NR_clock_gettime), "D" (clock), "S" (ts) :
+ "memory", "rcx", "r11");
return ret;
}
@@ -46,8 +47,9 @@ notrace static long vdso_fallback_gtod(struct timeval *tv, struct timezone *tz)
{
long ret;
- asm("syscall" : "=a" (ret) :
- "0" (__NR_gettimeofday), "D" (tv), "S" (tz) : "memory");
+ asm ("syscall" : "=a" (ret), "=m" (*tv), "=m" (*tz) :
+ "0" (__NR_gettimeofday), "D" (tv), "S" (tz) :
+ "memory", "rcx", "r11");
return ret;
}
@@ -58,13 +60,13 @@ notrace static long vdso_fallback_gettime(long clock, struct timespec *ts)
{
long ret;
- asm(
+ asm (
"mov %%ebx, %%edx \n"
- "mov %2, %%ebx \n"
+ "mov %[clock], %%ebx \n"
"call __kernel_vsyscall \n"
"mov %%edx, %%ebx \n"
- : "=a" (ret)
- : "0" (__NR_clock_gettime), "g" (clock), "c" (ts)
+ : "=a" (ret), "=m" (*ts)
+ : "0" (__NR_clock_gettime), [clock] "g" (clock), "c" (ts)
: "memory", "edx");
return ret;
}
@@ -73,13 +75,13 @@ notrace static long vdso_fallback_gtod(struct timeval *tv, struct timezone *tz)
{
long ret;
- asm(
+ asm (
"mov %%ebx, %%edx \n"
- "mov %2, %%ebx \n"
+ "mov %[tv], %%ebx \n"
"call __kernel_vsyscall \n"
"mov %%edx, %%ebx \n"
- : "=a" (ret)
- : "0" (__NR_gettimeofday), "g" (tv), "c" (tz)
+ : "=a" (ret), "=m" (*tv), "=m" (*tz)
+ : "0" (__NR_gettimeofday), [tv] "g" (tv), "c" (tz)
: "memory", "edx");
return ret;
}
diff --git a/arch/x86/events/amd/ibs.c b/arch/x86/events/amd/ibs.c
index b26ee32..fd4484ae 100644
--- a/arch/x86/events/amd/ibs.c
+++ b/arch/x86/events/amd/ibs.c
@@ -578,7 +578,7 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)
{
struct cpu_perf_ibs *pcpu = this_cpu_ptr(perf_ibs->pcpu);
struct perf_event *event = pcpu->event;
- struct hw_perf_event *hwc = &event->hw;
+ struct hw_perf_event *hwc;
struct perf_sample_data data;
struct perf_raw_record raw;
struct pt_regs regs;
@@ -601,6 +601,10 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)
return 0;
}
+ if (WARN_ON_ONCE(!event))
+ goto fail;
+
+ hwc = &event->hw;
msr = hwc->config_base;
buf = ibs_data.regs;
rdmsrl(msr, *buf);
diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c
index f73796d..cadf999 100644
--- a/arch/x86/events/core.c
+++ b/arch/x86/events/core.c
@@ -26,6 +26,7 @@
#include <linux/cpu.h>
#include <linux/bitops.h>
#include <linux/device.h>
+#include <linux/nospec.h>
#include <asm/apic.h>
#include <asm/stacktrace.h>
@@ -303,17 +304,20 @@ set_ext_hw_attr(struct hw_perf_event *hwc, struct perf_event *event)
config = attr->config;
- cache_type = (config >> 0) & 0xff;
+ cache_type = (config >> 0) & 0xff;
if (cache_type >= PERF_COUNT_HW_CACHE_MAX)
return -EINVAL;
+ cache_type = array_index_nospec(cache_type, PERF_COUNT_HW_CACHE_MAX);
cache_op = (config >> 8) & 0xff;
if (cache_op >= PERF_COUNT_HW_CACHE_OP_MAX)
return -EINVAL;
+ cache_op = array_index_nospec(cache_op, PERF_COUNT_HW_CACHE_OP_MAX);
cache_result = (config >> 16) & 0xff;
if (cache_result >= PERF_COUNT_HW_CACHE_RESULT_MAX)
return -EINVAL;
+ cache_result = array_index_nospec(cache_result, PERF_COUNT_HW_CACHE_RESULT_MAX);
val = hw_cache_event_ids[cache_type][cache_op][cache_result];
@@ -420,6 +424,8 @@ int x86_setup_perfctr(struct perf_event *event)
if (attr->config >= x86_pmu.max_events)
return -EINVAL;
+ attr->config = array_index_nospec((unsigned long)attr->config, x86_pmu.max_events);
+
/*
* The generic map:
*/
@@ -431,26 +437,6 @@ int x86_setup_perfctr(struct perf_event *event)
if (config == -1LL)
return -EINVAL;
- /*
- * Branch tracing:
- */
- if (attr->config == PERF_COUNT_HW_BRANCH_INSTRUCTIONS &&
- !attr->freq && hwc->sample_period == 1) {
- /* BTS is not supported by this architecture. */
- if (!x86_pmu.bts_active)
- return -EOPNOTSUPP;
-
- /* BTS is currently only allowed for user-mode. */
- if (!attr->exclude_kernel)
- return -EOPNOTSUPP;
-
- /* disallow bts if conflicting events are present */
- if (x86_add_exclusive(x86_lbr_exclusive_lbr))
- return -EBUSY;
-
- event->destroy = hw_perf_lbr_event_destroy;
- }
-
hwc->config |= config;
return 0;
@@ -1149,16 +1135,13 @@ int x86_perf_event_set_period(struct perf_event *event)
per_cpu(pmc_prev_left[idx], smp_processor_id()) = left;
- if (!(hwc->flags & PERF_X86_EVENT_AUTO_RELOAD) ||
- local64_read(&hwc->prev_count) != (u64)-left) {
- /*
- * The hw event starts counting from this event offset,
- * mark it to be able to extra future deltas:
- */
- local64_set(&hwc->prev_count, (u64)-left);
+ /*
+ * The hw event starts counting from this event offset,
+ * mark it to be able to extra future deltas:
+ */
+ local64_set(&hwc->prev_count, (u64)-left);
- wrmsrl(hwc->event_base, (u64)(-left) & x86_pmu.cntval_mask);
- }
+ wrmsrl(hwc->event_base, (u64)(-left) & x86_pmu.cntval_mask);
/*
* Due to erratum on certan cpu we need
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 6f353a8..4f85607 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -2066,16 +2066,23 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
int bit, loops;
u64 status;
int handled;
+ int pmu_enabled;
cpuc = this_cpu_ptr(&cpu_hw_events);
/*
+ * Save the PMU state.
+ * It needs to be restored when leaving the handler.
+ */
+ pmu_enabled = cpuc->enabled;
+ /*
* No known reason to not always do late ACK,
* but just in case do it opt-in.
*/
if (!x86_pmu.late_ack)
apic_write(APIC_LVTPC, APIC_DM_NMI);
intel_bts_disable_local();
+ cpuc->enabled = 0;
__intel_pmu_disable_all();
handled = intel_pmu_drain_bts_buffer();
handled += intel_bts_interrupt();
@@ -2173,7 +2180,8 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
done:
/* Only restore PMU state when it's active. See x86_pmu_disable(). */
- if (cpuc->enabled)
+ cpuc->enabled = pmu_enabled;
+ if (pmu_enabled)
__intel_pmu_enable_all(0, true);
intel_bts_enable_local();
@@ -2190,16 +2198,7 @@ static int intel_pmu_handle_irq(struct pt_regs *regs)
static struct event_constraint *
intel_bts_constraints(struct perf_event *event)
{
- struct hw_perf_event *hwc = &event->hw;
- unsigned int hw_event, bts_event;
-
- if (event->attr.freq)
- return NULL;
-
- hw_event = hwc->config & INTEL_ARCH_EVENT_MASK;
- bts_event = x86_pmu.event_map(PERF_COUNT_HW_BRANCH_INSTRUCTIONS);
-
- if (unlikely(hw_event == bts_event && hwc->sample_period == 1))
+ if (unlikely(intel_pmu_has_bts(event)))
return &bts_constraint;
return NULL;
@@ -2814,6 +2813,39 @@ static unsigned long intel_pmu_free_running_flags(struct perf_event *event)
return flags;
}
+static int intel_pmu_bts_config(struct perf_event *event)
+{
+ struct perf_event_attr *attr = &event->attr;
+
+ if (unlikely(intel_pmu_has_bts(event))) {
+ /* BTS is not supported by this architecture. */
+ if (!x86_pmu.bts_active)
+ return -EOPNOTSUPP;
+
+ /* BTS is currently only allowed for user-mode. */
+ if (!attr->exclude_kernel)
+ return -EOPNOTSUPP;
+
+ /* disallow bts if conflicting events are present */
+ if (x86_add_exclusive(x86_lbr_exclusive_lbr))
+ return -EBUSY;
+
+ event->destroy = hw_perf_lbr_event_destroy;
+ }
+
+ return 0;
+}
+
+static int core_pmu_hw_config(struct perf_event *event)
+{
+ int ret = x86_pmu_hw_config(event);
+
+ if (ret)
+ return ret;
+
+ return intel_pmu_bts_config(event);
+}
+
static int intel_pmu_hw_config(struct perf_event *event)
{
int ret = x86_pmu_hw_config(event);
@@ -2821,6 +2853,10 @@ static int intel_pmu_hw_config(struct perf_event *event)
if (ret)
return ret;
+ ret = intel_pmu_bts_config(event);
+ if (ret)
+ return ret;
+
if (event->attr.precise_ip) {
if (!event->attr.freq) {
event->hw.flags |= PERF_X86_EVENT_AUTO_RELOAD;
@@ -2840,7 +2876,7 @@ static int intel_pmu_hw_config(struct perf_event *event)
/*
* BTS is set up earlier in this path, so don't account twice
*/
- if (!intel_pmu_has_bts(event)) {
+ if (!unlikely(intel_pmu_has_bts(event))) {
/* disallow lbr if conflicting events are present */
if (x86_add_exclusive(x86_lbr_exclusive_lbr))
return -EBUSY;
@@ -3019,7 +3055,7 @@ hsw_get_event_constraints(struct cpu_hw_events *cpuc, int idx,
* Therefore the effective (average) period matches the requested period,
* despite coarser hardware granularity.
*/
-static unsigned bdw_limit_period(struct perf_event *event, unsigned left)
+static u64 bdw_limit_period(struct perf_event *event, u64 left)
{
if ((event->hw.config & INTEL_ARCH_EVENT_MASK) ==
X86_CONFIG(.event=0xc0, .umask=0x01)) {
@@ -3257,7 +3293,7 @@ static __initconst const struct x86_pmu core_pmu = {
.enable_all = core_pmu_enable_all,
.enable = core_pmu_enable_event,
.disable = x86_pmu_disable_event,
- .hw_config = x86_pmu_hw_config,
+ .hw_config = core_pmu_hw_config,
.schedule_events = x86_schedule_events,
.eventsel = MSR_ARCH_PERFMON_EVENTSEL0,
.perfctr = MSR_ARCH_PERFMON_PERFCTR0,
diff --git a/arch/x86/events/intel/cstate.c b/arch/x86/events/intel/cstate.c
index 1076c9a..47d526c 100644
--- a/arch/x86/events/intel/cstate.c
+++ b/arch/x86/events/intel/cstate.c
@@ -90,6 +90,7 @@
#include <linux/module.h>
#include <linux/slab.h>
#include <linux/perf_event.h>
+#include <linux/nospec.h>
#include <asm/cpu_device_id.h>
#include <asm/intel-family.h>
#include "../perf_event.h"
@@ -300,6 +301,7 @@ static int cstate_pmu_event_init(struct perf_event *event)
} else if (event->pmu == &cstate_pkg_pmu) {
if (cfg >= PERF_CSTATE_PKG_EVENT_MAX)
return -EINVAL;
+ cfg = array_index_nospec((unsigned long)cfg, PERF_CSTATE_PKG_EVENT_MAX);
if (!pkg_msr[cfg].attr)
return -EINVAL;
event->hw.event_base = pkg_msr[cfg].msr;
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 8e7a3f1..f26e26e 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -1110,6 +1110,7 @@ static void setup_pebs_sample_data(struct perf_event *event,
if (pebs == NULL)
return;
+ regs->flags &= ~PERF_EFLAGS_EXACT;
sample_type = event->attr.sample_type;
dsrc = sample_type & PERF_SAMPLE_DATA_SRC;
@@ -1154,7 +1155,6 @@ static void setup_pebs_sample_data(struct perf_event *event,
*/
*regs = *iregs;
regs->flags = pebs->flags;
- set_linear_ip(regs, pebs->ip);
if (sample_type & PERF_SAMPLE_REGS_INTR) {
regs->ax = pebs->ax;
@@ -1190,13 +1190,22 @@ static void setup_pebs_sample_data(struct perf_event *event,
#endif
}
- if (event->attr.precise_ip > 1 && x86_pmu.intel_cap.pebs_format >= 2) {
- regs->ip = pebs->real_ip;
- regs->flags |= PERF_EFLAGS_EXACT;
- } else if (event->attr.precise_ip > 1 && intel_pmu_pebs_fixup_ip(regs))
- regs->flags |= PERF_EFLAGS_EXACT;
- else
- regs->flags &= ~PERF_EFLAGS_EXACT;
+ if (event->attr.precise_ip > 1) {
+ /* Haswell and later have the eventing IP, so use it: */
+ if (x86_pmu.intel_cap.pebs_format >= 2) {
+ set_linear_ip(regs, pebs->real_ip);
+ regs->flags |= PERF_EFLAGS_EXACT;
+ } else {
+ /* Otherwise use PEBS off-by-1 IP: */
+ set_linear_ip(regs, pebs->ip);
+
+ /* ... and try to fix it up using the LBR entries: */
+ if (intel_pmu_pebs_fixup_ip(regs))
+ regs->flags |= PERF_EFLAGS_EXACT;
+ }
+ } else
+ set_linear_ip(regs, pebs->ip);
+
if ((sample_type & PERF_SAMPLE_ADDR) &&
x86_pmu.intel_cap.pebs_format >= 1)
@@ -1263,17 +1272,84 @@ get_next_pebs_record_by_bit(void *base, void *top, int bit)
return NULL;
}
+/*
+ * Special variant of intel_pmu_save_and_restart() for auto-reload.
+ */
+static int
+intel_pmu_save_and_restart_reload(struct perf_event *event, int count)
+{
+ struct hw_perf_event *hwc = &event->hw;
+ int shift = 64 - x86_pmu.cntval_bits;
+ u64 period = hwc->sample_period;
+ u64 prev_raw_count, new_raw_count;
+ s64 new, old;
+
+ WARN_ON(!period);
+
+ /*
+ * drain_pebs() only happens when the PMU is disabled.
+ */
+ WARN_ON(this_cpu_read(cpu_hw_events.enabled));
+
+ prev_raw_count = local64_read(&hwc->prev_count);
+ rdpmcl(hwc->event_base_rdpmc, new_raw_count);
+ local64_set(&hwc->prev_count, new_raw_count);
+
+ /*
+ * Since the counter increments a negative counter value and
+ * overflows on the sign switch, giving the interval:
+ *
+ * [-period, 0]
+ *
+ * the difference between two consequtive reads is:
+ *
+ * A) value2 - value1;
+ * when no overflows have happened in between,
+ *
+ * B) (0 - value1) + (value2 - (-period));
+ * when one overflow happened in between,
+ *
+ * C) (0 - value1) + (n - 1) * (period) + (value2 - (-period));
+ * when @n overflows happened in between.
+ *
+ * Here A) is the obvious difference, B) is the extension to the
+ * discrete interval, where the first term is to the top of the
+ * interval and the second term is from the bottom of the next
+ * interval and C) the extension to multiple intervals, where the
+ * middle term is the whole intervals covered.
+ *
+ * An equivalent of C, by reduction, is:
+ *
+ * value2 - value1 + n * period
+ */
+ new = ((s64)(new_raw_count << shift) >> shift);
+ old = ((s64)(prev_raw_count << shift) >> shift);
+ local64_add(new - old + count * period, &event->count);
+
+ perf_event_update_userpage(event);
+
+ return 0;
+}
+
static void __intel_pmu_pebs_event(struct perf_event *event,
struct pt_regs *iregs,
void *base, void *top,
int bit, int count)
{
+ struct hw_perf_event *hwc = &event->hw;
struct perf_sample_data data;
struct pt_regs regs;
void *at = get_next_pebs_record_by_bit(base, top, bit);
- if (!intel_pmu_save_and_restart(event) &&
- !(event->hw.flags & PERF_X86_EVENT_AUTO_RELOAD))
+ if (hwc->flags & PERF_X86_EVENT_AUTO_RELOAD) {
+ /*
+ * Now, auto-reload is only enabled in fixed period mode.
+ * The reload value is always hwc->sample_period.
+ * May need to change it, if auto-reload is enabled in
+ * freq mode later.
+ */
+ intel_pmu_save_and_restart_reload(event, count);
+ } else if (!intel_pmu_save_and_restart(event))
return;
while (count > 1) {
@@ -1325,8 +1401,11 @@ static void intel_pmu_drain_pebs_core(struct pt_regs *iregs)
return;
n = top - at;
- if (n <= 0)
+ if (n <= 0) {
+ if (event->hw.flags & PERF_X86_EVENT_AUTO_RELOAD)
+ intel_pmu_save_and_restart_reload(event, 0);
return;
+ }
__intel_pmu_pebs_event(event, iregs, at, top, 0, n);
}
@@ -1349,8 +1428,22 @@ static void intel_pmu_drain_pebs_nhm(struct pt_regs *iregs)
ds->pebs_index = ds->pebs_buffer_base;
- if (unlikely(base >= top))
+ if (unlikely(base >= top)) {
+ /*
+ * The drain_pebs() could be called twice in a short period
+ * for auto-reload event in pmu::read(). There are no
+ * overflows have happened in between.
+ * It needs to call intel_pmu_save_and_restart_reload() to
+ * update the event->count for this case.
+ */
+ for_each_set_bit(bit, (unsigned long *)&cpuc->pebs_enabled,
+ x86_pmu.max_pebs_events) {
+ event = cpuc->events[bit];
+ if (event->hw.flags & PERF_X86_EVENT_AUTO_RELOAD)
+ intel_pmu_save_and_restart_reload(event, 0);
+ }
return;
+ }
for (at = base; at < top; at += x86_pmu.pebs_record_size) {
struct pebs_record_nhm *p = at;
diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c
index 5d103a8..10c1a5c 100644
--- a/arch/x86/events/intel/lbr.c
+++ b/arch/x86/events/intel/lbr.c
@@ -342,7 +342,7 @@ static void __intel_pmu_lbr_restore(struct x86_perf_task_context *task_ctx)
mask = x86_pmu.lbr_nr - 1;
tos = task_ctx->tos;
- for (i = 0; i < tos; i++) {
+ for (i = 0; i < task_ctx->valid_lbrs; i++) {
lbr_idx = (tos - i) & mask;
wrlbr_from(lbr_idx, task_ctx->lbr_from[i]);
wrlbr_to (lbr_idx, task_ctx->lbr_to[i]);
@@ -350,6 +350,15 @@ static void __intel_pmu_lbr_restore(struct x86_perf_task_context *task_ctx)
if (x86_pmu.intel_cap.lbr_format == LBR_FORMAT_INFO)
wrmsrl(MSR_LBR_INFO_0 + lbr_idx, task_ctx->lbr_info[i]);
}
+
+ for (; i < x86_pmu.lbr_nr; i++) {
+ lbr_idx = (tos - i) & mask;
+ wrlbr_from(lbr_idx, 0);
+ wrlbr_to(lbr_idx, 0);
+ if (x86_pmu.intel_cap.lbr_format == LBR_FORMAT_INFO)
+ wrmsrl(MSR_LBR_INFO_0 + lbr_idx, 0);
+ }
+
wrmsrl(x86_pmu.lbr_tos, tos);
task_ctx->lbr_stack_state = LBR_NONE;
}
@@ -357,7 +366,7 @@ static void __intel_pmu_lbr_restore(struct x86_perf_task_context *task_ctx)
static void __intel_pmu_lbr_save(struct x86_perf_task_context *task_ctx)
{
unsigned lbr_idx, mask;
- u64 tos;
+ u64 tos, from;
int i;
if (task_ctx->lbr_callstack_users == 0) {
@@ -367,13 +376,17 @@ static void __intel_pmu_lbr_save(struct x86_perf_task_context *task_ctx)
mask = x86_pmu.lbr_nr - 1;
tos = intel_pmu_lbr_tos();
- for (i = 0; i < tos; i++) {
+ for (i = 0; i < x86_pmu.lbr_nr; i++) {
lbr_idx = (tos - i) & mask;
- task_ctx->lbr_from[i] = rdlbr_from(lbr_idx);
+ from = rdlbr_from(lbr_idx);
+ if (!from)
+ break;
+ task_ctx->lbr_from[i] = from;
task_ctx->lbr_to[i] = rdlbr_to(lbr_idx);
if (x86_pmu.intel_cap.lbr_format == LBR_FORMAT_INFO)
rdmsrl(MSR_LBR_INFO_0 + lbr_idx, task_ctx->lbr_info[i]);
}
+ task_ctx->valid_lbrs = i;
task_ctx->tos = tos;
task_ctx->lbr_stack_state = LBR_VALID;
}
@@ -522,7 +535,7 @@ static void intel_pmu_lbr_read_32(struct cpu_hw_events *cpuc)
*/
static void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
{
- bool need_info = false;
+ bool need_info = false, call_stack = false;
unsigned long mask = x86_pmu.lbr_nr - 1;
int lbr_format = x86_pmu.intel_cap.lbr_format;
u64 tos = intel_pmu_lbr_tos();
@@ -533,7 +546,7 @@ static void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
if (cpuc->lbr_sel) {
need_info = !(cpuc->lbr_sel->config & LBR_NO_INFO);
if (cpuc->lbr_sel->config & LBR_CALL_STACK)
- num = tos;
+ call_stack = true;
}
for (i = 0; i < num; i++) {
@@ -546,6 +559,13 @@ static void intel_pmu_lbr_read_64(struct cpu_hw_events *cpuc)
from = rdlbr_from(lbr_idx);
to = rdlbr_to(lbr_idx);
+ /*
+ * Read LBR call stack entries
+ * until invalid entry (0s) is detected.
+ */
+ if (call_stack && !from)
+ break;
+
if (lbr_format == LBR_FORMAT_INFO && need_info) {
u64 info;
@@ -1175,4 +1195,8 @@ void intel_pmu_lbr_init_knl(void)
x86_pmu.lbr_sel_mask = LBR_SEL_MASK;
x86_pmu.lbr_sel_map = snb_lbr_sel_map;
+
+ /* Knights Landing does have MISPREDICT bit */
+ if (x86_pmu.intel_cap.lbr_format == LBR_FORMAT_LIP)
+ x86_pmu.intel_cap.lbr_format = LBR_FORMAT_EIP_FLAGS;
}
diff --git a/arch/x86/events/intel/uncore.c b/arch/x86/events/intel/uncore.c
index aec6cc9..4f36526 100644
--- a/arch/x86/events/intel/uncore.c
+++ b/arch/x86/events/intel/uncore.c
@@ -212,7 +212,7 @@ void uncore_perf_event_update(struct intel_uncore_box *box, struct perf_event *e
u64 prev_count, new_count, delta;
int shift;
- if (event->hw.idx >= UNCORE_PMC_IDX_FIXED)
+ if (event->hw.idx == UNCORE_PMC_IDX_FIXED)
shift = 64 - uncore_fixed_ctr_bits(box);
else
shift = 64 - uncore_perf_ctr_bits(box);
diff --git a/arch/x86/events/intel/uncore_nhmex.c b/arch/x86/events/intel/uncore_nhmex.c
index cda5693..83e2188 100644
--- a/arch/x86/events/intel/uncore_nhmex.c
+++ b/arch/x86/events/intel/uncore_nhmex.c
@@ -245,7 +245,7 @@ static void nhmex_uncore_msr_enable_event(struct intel_uncore_box *box, struct p
{
struct hw_perf_event *hwc = &event->hw;
- if (hwc->idx >= UNCORE_PMC_IDX_FIXED)
+ if (hwc->idx == UNCORE_PMC_IDX_FIXED)
wrmsrl(hwc->config_base, NHMEX_PMON_CTL_EN_BIT0);
else if (box->pmu->type->event_mask & NHMEX_PMON_CTL_EN_BIT0)
wrmsrl(hwc->config_base, hwc->config | NHMEX_PMON_CTL_EN_BIT22);
diff --git a/arch/x86/events/intel/uncore_snb.c b/arch/x86/events/intel/uncore_snb.c
index a3dcc12..8c70006 100644
--- a/arch/x86/events/intel/uncore_snb.c
+++ b/arch/x86/events/intel/uncore_snb.c
@@ -14,6 +14,25 @@
#define PCI_DEVICE_ID_INTEL_SKL_HQ_IMC 0x1910
#define PCI_DEVICE_ID_INTEL_SKL_SD_IMC 0x190f
#define PCI_DEVICE_ID_INTEL_SKL_SQ_IMC 0x191f
+#define PCI_DEVICE_ID_INTEL_KBL_Y_IMC 0x590c
+#define PCI_DEVICE_ID_INTEL_KBL_U_IMC 0x5904
+#define PCI_DEVICE_ID_INTEL_KBL_UQ_IMC 0x5914
+#define PCI_DEVICE_ID_INTEL_KBL_SD_IMC 0x590f
+#define PCI_DEVICE_ID_INTEL_KBL_SQ_IMC 0x591f
+#define PCI_DEVICE_ID_INTEL_CFL_2U_IMC 0x3ecc
+#define PCI_DEVICE_ID_INTEL_CFL_4U_IMC 0x3ed0
+#define PCI_DEVICE_ID_INTEL_CFL_4H_IMC 0x3e10
+#define PCI_DEVICE_ID_INTEL_CFL_6H_IMC 0x3ec4
+#define PCI_DEVICE_ID_INTEL_CFL_2S_D_IMC 0x3e0f
+#define PCI_DEVICE_ID_INTEL_CFL_4S_D_IMC 0x3e1f
+#define PCI_DEVICE_ID_INTEL_CFL_6S_D_IMC 0x3ec2
+#define PCI_DEVICE_ID_INTEL_CFL_8S_D_IMC 0x3e30
+#define PCI_DEVICE_ID_INTEL_CFL_4S_W_IMC 0x3e18
+#define PCI_DEVICE_ID_INTEL_CFL_6S_W_IMC 0x3ec6
+#define PCI_DEVICE_ID_INTEL_CFL_8S_W_IMC 0x3e31
+#define PCI_DEVICE_ID_INTEL_CFL_4S_S_IMC 0x3e33
+#define PCI_DEVICE_ID_INTEL_CFL_6S_S_IMC 0x3eca
+#define PCI_DEVICE_ID_INTEL_CFL_8S_S_IMC 0x3e32
/* SNB event control */
#define SNB_UNC_CTL_EV_SEL_MASK 0x000000ff
@@ -631,7 +650,82 @@ static const struct pci_device_id skl_uncore_pci_ids[] = {
PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_SKL_SQ_IMC),
.driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
},
-
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_KBL_Y_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_KBL_U_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_KBL_UQ_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_KBL_SD_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_KBL_SQ_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_2U_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_4U_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_4H_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_6H_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_2S_D_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_4S_D_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_6S_D_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_8S_D_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_4S_W_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_6S_W_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_8S_W_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_4S_S_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_6S_S_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
+ { /* IMC */
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_CFL_8S_S_IMC),
+ .driver_data = UNCORE_PCI_DEV_DATA(SNB_PCI_UNCORE_IMC, 0),
+ },
{ /* end: all zeroes */ },
};
@@ -680,6 +774,25 @@ static const struct imc_uncore_pci_dev desktop_imc_pci_ids[] = {
IMC_DEV(SKL_HQ_IMC, &skl_uncore_pci_driver), /* 6th Gen Core H Quad Core */
IMC_DEV(SKL_SD_IMC, &skl_uncore_pci_driver), /* 6th Gen Core S Dual Core */
IMC_DEV(SKL_SQ_IMC, &skl_uncore_pci_driver), /* 6th Gen Core S Quad Core */
+ IMC_DEV(KBL_Y_IMC, &skl_uncore_pci_driver), /* 7th Gen Core Y */
+ IMC_DEV(KBL_U_IMC, &skl_uncore_pci_driver), /* 7th Gen Core U */
+ IMC_DEV(KBL_UQ_IMC, &skl_uncore_pci_driver), /* 7th Gen Core U Quad Core */
+ IMC_DEV(KBL_SD_IMC, &skl_uncore_pci_driver), /* 7th Gen Core S Dual Core */
+ IMC_DEV(KBL_SQ_IMC, &skl_uncore_pci_driver), /* 7th Gen Core S Quad Core */
+ IMC_DEV(CFL_2U_IMC, &skl_uncore_pci_driver), /* 8th Gen Core U 2 Cores */
+ IMC_DEV(CFL_4U_IMC, &skl_uncore_pci_driver), /* 8th Gen Core U 4 Cores */
+ IMC_DEV(CFL_4H_IMC, &skl_uncore_pci_driver), /* 8th Gen Core H 4 Cores */
+ IMC_DEV(CFL_6H_IMC, &skl_uncore_pci_driver), /* 8th Gen Core H 6 Cores */
+ IMC_DEV(CFL_2S_D_IMC, &skl_uncore_pci_driver), /* 8th Gen Core S 2 Cores Desktop */
+ IMC_DEV(CFL_4S_D_IMC, &skl_uncore_pci_driver), /* 8th Gen Core S 4 Cores Desktop */
+ IMC_DEV(CFL_6S_D_IMC, &skl_uncore_pci_driver), /* 8th Gen Core S 6 Cores Desktop */
+ IMC_DEV(CFL_8S_D_IMC, &skl_uncore_pci_driver), /* 8th Gen Core S 8 Cores Desktop */
+ IMC_DEV(CFL_4S_W_IMC, &skl_uncore_pci_driver), /* 8th Gen Core S 4 Cores Work Station */
+ IMC_DEV(CFL_6S_W_IMC, &skl_uncore_pci_driver), /* 8th Gen Core S 6 Cores Work Station */
+ IMC_DEV(CFL_8S_W_IMC, &skl_uncore_pci_driver), /* 8th Gen Core S 8 Cores Work Station */
+ IMC_DEV(CFL_4S_S_IMC, &skl_uncore_pci_driver), /* 8th Gen Core S 4 Cores Server */
+ IMC_DEV(CFL_6S_S_IMC, &skl_uncore_pci_driver), /* 8th Gen Core S 6 Cores Server */
+ IMC_DEV(CFL_8S_S_IMC, &skl_uncore_pci_driver), /* 8th Gen Core S 8 Cores Server */
{ /* end marker */ }
};
diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c
index 6bc3694..8c2a9fa 100644
--- a/arch/x86/events/intel/uncore_snbep.c
+++ b/arch/x86/events/intel/uncore_snbep.c
@@ -3767,16 +3767,16 @@ static const struct pci_device_id skx_uncore_pci_ids[] = {
.driver_data = UNCORE_PCI_DEV_FULL_DATA(21, 5, SKX_PCI_UNCORE_M2PCIE, 3),
},
{ /* M3UPI0 Link 0 */
- PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x204C),
- .driver_data = UNCORE_PCI_DEV_FULL_DATA(18, 0, SKX_PCI_UNCORE_M3UPI, 0),
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x204D),
+ .driver_data = UNCORE_PCI_DEV_FULL_DATA(18, 1, SKX_PCI_UNCORE_M3UPI, 0),
},
{ /* M3UPI0 Link 1 */
- PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x204D),
- .driver_data = UNCORE_PCI_DEV_FULL_DATA(18, 1, SKX_PCI_UNCORE_M3UPI, 1),
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x204E),
+ .driver_data = UNCORE_PCI_DEV_FULL_DATA(18, 2, SKX_PCI_UNCORE_M3UPI, 1),
},
{ /* M3UPI1 Link 2 */
- PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x204C),
- .driver_data = UNCORE_PCI_DEV_FULL_DATA(18, 4, SKX_PCI_UNCORE_M3UPI, 2),
+ PCI_DEVICE(PCI_VENDOR_ID_INTEL, 0x204D),
+ .driver_data = UNCORE_PCI_DEV_FULL_DATA(18, 5, SKX_PCI_UNCORE_M3UPI, 2),
},
{ /* end: all zeroes */ }
};
diff --git a/arch/x86/events/msr.c b/arch/x86/events/msr.c
index 4bb3ec6..be0b196 100644
--- a/arch/x86/events/msr.c
+++ b/arch/x86/events/msr.c
@@ -1,4 +1,5 @@
#include <linux/perf_event.h>
+#include <linux/nospec.h>
#include <asm/intel-family.h>
enum perf_msr_id {
@@ -136,9 +137,6 @@ static int msr_event_init(struct perf_event *event)
if (event->attr.type != event->pmu->type)
return -ENOENT;
- if (cfg >= PERF_MSR_EVENT_MAX)
- return -EINVAL;
-
/* unsupported modes and filters */
if (event->attr.exclude_user ||
event->attr.exclude_kernel ||
@@ -149,6 +147,11 @@ static int msr_event_init(struct perf_event *event)
event->attr.sample_period) /* no sampling */
return -EINVAL;
+ if (cfg >= PERF_MSR_EVENT_MAX)
+ return -EINVAL;
+
+ cfg = array_index_nospec((unsigned long)cfg, PERF_MSR_EVENT_MAX);
+
if (!msr[cfg].attr)
return -EINVAL;
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index bcbb1d2..7ace39c 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -548,7 +548,7 @@ struct x86_pmu {
struct x86_pmu_quirk *quirks;
int perfctr_second_write;
bool late_ack;
- unsigned (*limit_period)(struct perf_event *event, unsigned l);
+ u64 (*limit_period)(struct perf_event *event, u64 l);
/*
* sysfs attrs
@@ -633,6 +633,7 @@ struct x86_perf_task_context {
u64 lbr_to[MAX_LBR_ENTRIES];
u64 lbr_info[MAX_LBR_ENTRIES];
int tos;
+ int valid_lbrs;
int lbr_callstack_users;
int lbr_stack_state;
};
@@ -834,11 +835,16 @@ static inline int amd_pmu_init(void)
static inline bool intel_pmu_has_bts(struct perf_event *event)
{
- if (event->attr.config == PERF_COUNT_HW_BRANCH_INSTRUCTIONS &&
- !event->attr.freq && event->hw.sample_period == 1)
- return true;
+ struct hw_perf_event *hwc = &event->hw;
+ unsigned int hw_event, bts_event;
- return false;
+ if (event->attr.freq)
+ return false;
+
+ hw_event = hwc->config & INTEL_ARCH_EVENT_MASK;
+ bts_event = x86_pmu.event_map(PERF_COUNT_HW_BRANCH_INSTRUCTIONS);
+
+ return hw_event == bts_event && hwc->sample_period == 1;
}
int intel_pmu_save_and_restart(struct perf_event *event);
diff --git a/arch/x86/include/asm/apic.h b/arch/x86/include/asm/apic.h
index f5aaf6c..2188b5a 100644
--- a/arch/x86/include/asm/apic.h
+++ b/arch/x86/include/asm/apic.h
@@ -12,6 +12,7 @@
#include <asm/mpspec.h>
#include <asm/msr.h>
#include <asm/idle.h>
+#include <asm/hardirq.h>
#define ARCH_APICTIMER_STOPS_ON_C3 1
@@ -633,6 +634,13 @@ extern int default_check_phys_apicid_present(int phys_apicid);
#endif
#endif /* CONFIG_X86_LOCAL_APIC */
+
+#ifdef CONFIG_SMP
+bool apic_id_is_primary_thread(unsigned int id);
+#else
+static inline bool apic_id_is_primary_thread(unsigned int id) { return false; }
+#endif
+
extern void irq_enter(void);
extern void irq_exit(void);
@@ -640,6 +648,7 @@ static inline void entering_irq(void)
{
irq_enter();
exit_idle();
+ kvm_set_cpu_l1tf_flush_l1d();
}
static inline void entering_ack_irq(void)
@@ -652,6 +661,7 @@ static inline void ipi_entering_ack_irq(void)
{
irq_enter();
ack_APIC_irq();
+ kvm_set_cpu_l1tf_flush_l1d();
}
static inline void exiting_irq(void)
diff --git a/arch/x86/include/asm/apm.h b/arch/x86/include/asm/apm.h
index 46e40ae..93eebc63 100644
--- a/arch/x86/include/asm/apm.h
+++ b/arch/x86/include/asm/apm.h
@@ -6,8 +6,6 @@
#ifndef _ASM_X86_MACH_DEFAULT_APM_H
#define _ASM_X86_MACH_DEFAULT_APM_H
-#include <asm/nospec-branch.h>
-
#ifdef APM_ZERO_SEGS
# define APM_DO_ZERO_SEGS \
"pushl %%ds\n\t" \
@@ -33,7 +31,6 @@ static inline void apm_bios_call_asm(u32 func, u32 ebx_in, u32 ecx_in,
* N.B. We do NOT need a cld after the BIOS call
* because we always save and restore the flags.
*/
- firmware_restrict_branch_speculation_start();
__asm__ __volatile__(APM_DO_ZERO_SEGS
"pushl %%edi\n\t"
"pushl %%ebp\n\t"
@@ -46,7 +43,6 @@ static inline void apm_bios_call_asm(u32 func, u32 ebx_in, u32 ecx_in,
"=S" (*esi)
: "a" (func), "b" (ebx_in), "c" (ecx_in)
: "memory", "cc");
- firmware_restrict_branch_speculation_end();
}
static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in,
@@ -59,7 +55,6 @@ static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in,
* N.B. We do NOT need a cld after the BIOS call
* because we always save and restore the flags.
*/
- firmware_restrict_branch_speculation_start();
__asm__ __volatile__(APM_DO_ZERO_SEGS
"pushl %%edi\n\t"
"pushl %%ebp\n\t"
@@ -72,7 +67,6 @@ static inline bool apm_bios_call_simple_asm(u32 func, u32 ebx_in,
"=S" (si)
: "a" (func), "b" (ebx_in), "c" (ecx_in)
: "memory", "cc");
- firmware_restrict_branch_speculation_end();
return error;
}
diff --git a/arch/x86/include/asm/asm.h b/arch/x86/include/asm/asm.h
index 8d8c24f..742712b 100644
--- a/arch/x86/include/asm/asm.h
+++ b/arch/x86/include/asm/asm.h
@@ -45,6 +45,65 @@
#define _ASM_SI __ASM_REG(si)
#define _ASM_DI __ASM_REG(di)
+#ifndef __x86_64__
+/* 32 bit */
+
+#define _ASM_ARG1 _ASM_AX
+#define _ASM_ARG2 _ASM_DX
+#define _ASM_ARG3 _ASM_CX
+
+#define _ASM_ARG1L eax
+#define _ASM_ARG2L edx
+#define _ASM_ARG3L ecx
+
+#define _ASM_ARG1W ax
+#define _ASM_ARG2W dx
+#define _ASM_ARG3W cx
+
+#define _ASM_ARG1B al
+#define _ASM_ARG2B dl
+#define _ASM_ARG3B cl
+
+#else
+/* 64 bit */
+
+#define _ASM_ARG1 _ASM_DI
+#define _ASM_ARG2 _ASM_SI
+#define _ASM_ARG3 _ASM_DX
+#define _ASM_ARG4 _ASM_CX
+#define _ASM_ARG5 r8
+#define _ASM_ARG6 r9
+
+#define _ASM_ARG1Q rdi
+#define _ASM_ARG2Q rsi
+#define _ASM_ARG3Q rdx
+#define _ASM_ARG4Q rcx
+#define _ASM_ARG5Q r8
+#define _ASM_ARG6Q r9
+
+#define _ASM_ARG1L edi
+#define _ASM_ARG2L esi
+#define _ASM_ARG3L edx
+#define _ASM_ARG4L ecx
+#define _ASM_ARG5L r8d
+#define _ASM_ARG6L r9d
+
+#define _ASM_ARG1W di
+#define _ASM_ARG2W si
+#define _ASM_ARG3W dx
+#define _ASM_ARG4W cx
+#define _ASM_ARG5W r8w
+#define _ASM_ARG6W r9w
+
+#define _ASM_ARG1B dil
+#define _ASM_ARG2B sil
+#define _ASM_ARG3B dl
+#define _ASM_ARG4B cl
+#define _ASM_ARG5B r8b
+#define _ASM_ARG6B r9b
+
+#endif
+
/*
* Macros to generate condition code outputs from inline assembly,
* The output operand must be type "bool".
diff --git a/arch/x86/include/asm/barrier.h b/arch/x86/include/asm/barrier.h
index 78d1c6a..eb53c2c 100644
--- a/arch/x86/include/asm/barrier.h
+++ b/arch/x86/include/asm/barrier.h
@@ -37,7 +37,7 @@ static inline unsigned long array_index_mask_nospec(unsigned long index,
{
unsigned long mask;
- asm ("cmp %1,%2; sbb %0,%0;"
+ asm volatile ("cmp %1,%2; sbb %0,%0;"
:"=r" (mask)
:"g"(size),"r" (index)
:"cc");
diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h
index a248531..c56c243 100644
--- a/arch/x86/include/asm/cpufeatures.h
+++ b/arch/x86/include/asm/cpufeatures.h
@@ -104,7 +104,6 @@
#define X86_FEATURE_EXTD_APICID ( 3*32+26) /* has extended APICID (8 bits) */
#define X86_FEATURE_AMD_DCM ( 3*32+27) /* multi-node processor */
#define X86_FEATURE_APERFMPERF ( 3*32+28) /* APERFMPERF */
-#define X86_FEATURE_EAGER_FPU ( 3*32+29) /* "eagerfpu" Non lazy FPU restore */
#define X86_FEATURE_NONSTOP_TSC_S3 ( 3*32+30) /* TSC doesn't stop in S3 state */
/* Intel-defined CPU features, CPUID level 0x00000001 (ecx), word 4 */
@@ -197,6 +196,9 @@
#define X86_FEATURE_RETPOLINE ( 7*32+12) /* "" Generic Retpoline mitigation for Spectre variant 2 */
#define X86_FEATURE_RETPOLINE_AMD ( 7*32+13) /* "" AMD Retpoline mitigation for Spectre variant 2 */
+#define X86_FEATURE_MSR_SPEC_CTRL ( 7*32+16) /* "" MSR SPEC_CTRL is implemented */
+#define X86_FEATURE_SSBD ( 7*32+17) /* Speculative Store Bypass Disable */
+
#define X86_FEATURE_RSB_CTXSW ( 7*32+19) /* "" Fill RSB on context switches */
/* Because the ALTERNATIVE scheme is for members of the X86_FEATURE club... */
@@ -204,6 +206,14 @@
#define X86_FEATURE_USE_IBPB ( 7*32+21) /* "" Indirect Branch Prediction Barrier enabled */
#define X86_FEATURE_USE_IBRS_FW ( 7*32+22) /* "" Use IBRS during runtime firmware calls */
+#define X86_FEATURE_SPEC_STORE_BYPASS_DISABLE ( 7*32+23) /* "" Disable Speculative Store Bypass. */
+#define X86_FEATURE_LS_CFG_SSBD ( 7*32+24) /* "" AMD SSBD implementation */
+#define X86_FEATURE_IBRS ( 7*32+25) /* Indirect Branch Restricted Speculation */
+#define X86_FEATURE_IBPB ( 7*32+26) /* Indirect Branch Prediction Barrier */
+#define X86_FEATURE_STIBP ( 7*32+27) /* Single Thread Indirect Branch Predictors */
+#define X86_FEATURE_ZEN ( 7*32+28) /* "" CPU is AMD family 0x17 (Zen) */
+#define X86_FEATURE_L1TF_PTEINV ( 7*32+29) /* "" L1TF workaround PTE inversion */
+#define X86_FEATURE_IBRS_ENHANCED ( 7*32+30) /* Enhanced IBRS */
/* Virtualization flags: Linux defined, word 8 */
#define X86_FEATURE_TPR_SHADOW ( 8*32+ 0) /* Intel TPR Shadow */
@@ -261,9 +271,10 @@
/* AMD-defined CPU features, CPUID level 0x80000008 (ebx), word 13 */
#define X86_FEATURE_CLZERO (13*32+0) /* CLZERO instruction */
#define X86_FEATURE_IRPERF (13*32+1) /* Instructions Retired Count */
-#define X86_FEATURE_IBPB (13*32+12) /* Indirect Branch Prediction Barrier */
-#define X86_FEATURE_IBRS (13*32+14) /* Indirect Branch Restricted Speculation */
-#define X86_FEATURE_STIBP (13*32+15) /* Single Thread Indirect Branch Predictors */
+#define X86_FEATURE_AMD_IBPB (13*32+12) /* Indirect Branch Prediction Barrier */
+#define X86_FEATURE_AMD_IBRS (13*32+14) /* Indirect Branch Restricted Speculation */
+#define X86_FEATURE_AMD_STIBP (13*32+15) /* Single Thread Indirect Branch Predictors */
+#define X86_FEATURE_VIRT_SSBD (13*32+25) /* Virtualized Speculative Store Bypass Disable */
/* Thermal and Power Management Leaf, CPUID level 0x00000006 (eax), word 14 */
#define X86_FEATURE_DTHERM (14*32+ 0) /* Digital Thermal Sensor */
@@ -299,13 +310,16 @@
#define X86_FEATURE_SUCCOR (17*32+1) /* Uncorrectable error containment and recovery */
#define X86_FEATURE_SMCA (17*32+3) /* Scalable MCA */
+
/* Intel-defined CPU features, CPUID level 0x00000007:0 (EDX), word 18 */
#define X86_FEATURE_AVX512_4VNNIW (18*32+ 2) /* AVX-512 Neural Network Instructions */
#define X86_FEATURE_AVX512_4FMAPS (18*32+ 3) /* AVX-512 Multiply Accumulation Single precision */
#define X86_FEATURE_PCONFIG (18*32+18) /* Intel PCONFIG */
#define X86_FEATURE_SPEC_CTRL (18*32+26) /* "" Speculation Control (IBRS + IBPB) */
#define X86_FEATURE_INTEL_STIBP (18*32+27) /* "" Single Thread Indirect Branch Predictors */
+#define X86_FEATURE_FLUSH_L1D (18*32+28) /* Flush L1D cache */
#define X86_FEATURE_ARCH_CAPABILITIES (18*32+29) /* IA32_ARCH_CAPABILITIES MSR (Intel) */
+#define X86_FEATURE_SPEC_CTRL_SSBD (18*32+31) /* "" Speculative Store Bypass Disable */
/*
* BUG word(s)
@@ -335,5 +349,7 @@
#define X86_BUG_CPU_MELTDOWN X86_BUG(14) /* CPU is affected by meltdown attack and needs kernel page table isolation */
#define X86_BUG_SPECTRE_V1 X86_BUG(15) /* CPU is affected by Spectre variant 1 attack with conditional branches */
#define X86_BUG_SPECTRE_V2 X86_BUG(16) /* CPU is affected by Spectre variant 2 attack with indirect branches */
+#define X86_BUG_SPEC_STORE_BYPASS X86_BUG(17) /* CPU is affected by speculative store bypass attack */
+#define X86_BUG_L1TF X86_BUG(18) /* CPU is affected by L1 Terminal Fault */
#endif /* _ASM_X86_CPUFEATURES_H */
diff --git a/arch/x86/include/asm/dmi.h b/arch/x86/include/asm/dmi.h
index 3c69fed..d8b9560 100644
--- a/arch/x86/include/asm/dmi.h
+++ b/arch/x86/include/asm/dmi.h
@@ -3,8 +3,8 @@
#include <linux/compiler.h>
#include <linux/init.h>
+#include <linux/io.h>
-#include <asm/io.h>
#include <asm/setup.h>
static __always_inline __init void *dmi_alloc(unsigned len)
diff --git a/arch/x86/include/asm/fpu/internal.h b/arch/x86/include/asm/fpu/internal.h
index 2737366..499d6ed 100644
--- a/arch/x86/include/asm/fpu/internal.h
+++ b/arch/x86/include/asm/fpu/internal.h
@@ -60,11 +60,6 @@ extern u64 fpu__get_supported_xfeatures_mask(void);
/*
* FPU related CPU feature flag helper routines:
*/
-static __always_inline __pure bool use_eager_fpu(void)
-{
- return static_cpu_has(X86_FEATURE_EAGER_FPU);
-}
-
static __always_inline __pure bool use_xsaveopt(void)
{
return static_cpu_has(X86_FEATURE_XSAVEOPT);
@@ -501,24 +496,6 @@ static inline int fpu_want_lazy_restore(struct fpu *fpu, unsigned int cpu)
}
-/*
- * Wrap lazy FPU TS handling in a 'hw fpregs activation/deactivation'
- * idiom, which is then paired with the sw-flag (fpregs_active) later on:
- */
-
-static inline void __fpregs_activate_hw(void)
-{
- if (!use_eager_fpu())
- clts();
-}
-
-static inline void __fpregs_deactivate_hw(void)
-{
- if (!use_eager_fpu())
- stts();
-}
-
-/* Must be paired with an 'stts' (fpregs_deactivate_hw()) after! */
static inline void __fpregs_deactivate(struct fpu *fpu)
{
WARN_ON_FPU(!fpu->fpregs_active);
@@ -528,7 +505,6 @@ static inline void __fpregs_deactivate(struct fpu *fpu)
trace_x86_fpu_regs_deactivated(fpu);
}
-/* Must be paired with a 'clts' (fpregs_activate_hw()) before! */
static inline void __fpregs_activate(struct fpu *fpu)
{
WARN_ON_FPU(fpu->fpregs_active);
@@ -554,22 +530,17 @@ static inline int fpregs_active(void)
}
/*
- * Encapsulate the CR0.TS handling together with the
- * software flag.
- *
* These generally need preemption protection to work,
* do try to avoid using these on their own.
*/
static inline void fpregs_activate(struct fpu *fpu)
{
- __fpregs_activate_hw();
__fpregs_activate(fpu);
}
static inline void fpregs_deactivate(struct fpu *fpu)
{
__fpregs_deactivate(fpu);
- __fpregs_deactivate_hw();
}
/*
@@ -596,8 +567,7 @@ switch_fpu_prepare(struct fpu *old_fpu, struct fpu *new_fpu, int cpu)
* or if the past 5 consecutive context-switches used math.
*/
fpu.preload = static_cpu_has(X86_FEATURE_FPU) &&
- new_fpu->fpstate_active &&
- (use_eager_fpu() || new_fpu->counter > 5);
+ new_fpu->fpstate_active;
if (old_fpu->fpregs_active) {
if (!copy_fpregs_to_fpstate(old_fpu))
@@ -611,18 +581,13 @@ switch_fpu_prepare(struct fpu *old_fpu, struct fpu *new_fpu, int cpu)
/* Don't change CR0.TS if we just switch! */
if (fpu.preload) {
- new_fpu->counter++;
__fpregs_activate(new_fpu);
trace_x86_fpu_regs_activated(new_fpu);
prefetch(&new_fpu->state);
- } else {
- __fpregs_deactivate_hw();
}
} else {
- old_fpu->counter = 0;
old_fpu->last_cpu = -1;
if (fpu.preload) {
- new_fpu->counter++;
if (fpu_want_lazy_restore(new_fpu, cpu))
fpu.preload = 0;
else
diff --git a/arch/x86/include/asm/fpu/types.h b/arch/x86/include/asm/fpu/types.h
index 48df486..3c80f5b 100644
--- a/arch/x86/include/asm/fpu/types.h
+++ b/arch/x86/include/asm/fpu/types.h
@@ -322,17 +322,6 @@ struct fpu {
unsigned char fpregs_active;
/*
- * @counter:
- *
- * This counter contains the number of consecutive context switches
- * during which the FPU stays used. If this is over a threshold, the
- * lazy FPU restore logic becomes eager, to save the trap overhead.
- * This is an unsigned char so that after 256 iterations the counter
- * wraps and the context switch behavior turns lazy again; this is to
- * deal with bursty apps that only use the FPU for a short time:
- */
- unsigned char counter;
- /*
* @state:
*
* In-memory copy of all FPU registers that we save/restore
@@ -340,29 +329,6 @@ struct fpu {
* the registers in the FPU are more recent than this state
* copy. If the task context-switches away then they get
* saved here and represent the FPU state.
- *
- * After context switches there may be a (short) time period
- * during which the in-FPU hardware registers are unchanged
- * and still perfectly match this state, if the tasks
- * scheduled afterwards are not using the FPU.
- *
- * This is the 'lazy restore' window of optimization, which
- * we track though 'fpu_fpregs_owner_ctx' and 'fpu->last_cpu'.
- *
- * We detect whether a subsequent task uses the FPU via setting
- * CR0::TS to 1, which causes any FPU use to raise a #NM fault.
- *
- * During this window, if the task gets scheduled again, we
- * might be able to skip having to do a restore from this
- * memory buffer to the hardware registers - at the cost of
- * incurring the overhead of #NM fault traps.
- *
- * Note that on modern CPUs that support the XSAVEOPT (or other
- * optimized XSAVE instructions), we don't use #NM traps anymore,
- * as the hardware can track whether FPU registers need saving
- * or not. On such CPUs we activate the non-lazy ('eagerfpu')
- * logic, which unconditionally saves/restores all FPU state
- * across context switches. (if FPU state exists.)
*/
union fpregs_state state;
/*
diff --git a/arch/x86/include/asm/futex.h b/arch/x86/include/asm/futex.h
index b4c1f54..f4dc9b6 100644
--- a/arch/x86/include/asm/futex.h
+++ b/arch/x86/include/asm/futex.h
@@ -41,20 +41,11 @@
"+m" (*uaddr), "=&r" (tem) \
: "r" (oparg), "i" (-EFAULT), "1" (0))
-static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval,
+ u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret, tem;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
-
pagefault_disable();
switch (op) {
@@ -80,30 +71,9 @@ static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ:
- ret = (oldval == cmparg);
- break;
- case FUTEX_OP_CMP_NE:
- ret = (oldval != cmparg);
- break;
- case FUTEX_OP_CMP_LT:
- ret = (oldval < cmparg);
- break;
- case FUTEX_OP_CMP_GE:
- ret = (oldval >= cmparg);
- break;
- case FUTEX_OP_CMP_LE:
- ret = (oldval <= cmparg);
- break;
- case FUTEX_OP_CMP_GT:
- ret = (oldval > cmparg);
- break;
- default:
- ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/arch/x86/include/asm/hardirq.h b/arch/x86/include/asm/hardirq.h
index 9b76cd3..9871659 100644
--- a/arch/x86/include/asm/hardirq.h
+++ b/arch/x86/include/asm/hardirq.h
@@ -2,10 +2,12 @@
#define _ASM_X86_HARDIRQ_H
#include <linux/threads.h>
-#include <linux/irq.h>
typedef struct {
- unsigned int __softirq_pending;
+ u16 __softirq_pending;
+#if IS_ENABLED(CONFIG_KVM_INTEL)
+ u8 kvm_cpu_l1tf_flush_l1d;
+#endif
unsigned int __nmi_count; /* arch dependent */
#ifdef CONFIG_X86_LOCAL_APIC
unsigned int apic_timer_irqs; /* arch dependent */
@@ -60,4 +62,24 @@ extern u64 arch_irq_stat_cpu(unsigned int cpu);
extern u64 arch_irq_stat(void);
#define arch_irq_stat arch_irq_stat
+
+#if IS_ENABLED(CONFIG_KVM_INTEL)
+static inline void kvm_set_cpu_l1tf_flush_l1d(void)
+{
+ __this_cpu_write(irq_stat.kvm_cpu_l1tf_flush_l1d, 1);
+}
+
+static inline void kvm_clear_cpu_l1tf_flush_l1d(void)
+{
+ __this_cpu_write(irq_stat.kvm_cpu_l1tf_flush_l1d, 0);
+}
+
+static inline bool kvm_get_cpu_l1tf_flush_l1d(void)
+{
+ return __this_cpu_read(irq_stat.kvm_cpu_l1tf_flush_l1d);
+}
+#else /* !IS_ENABLED(CONFIG_KVM_INTEL) */
+static inline void kvm_set_cpu_l1tf_flush_l1d(void) { }
+#endif /* IS_ENABLED(CONFIG_KVM_INTEL) */
+
#endif /* _ASM_X86_HARDIRQ_H */
diff --git a/arch/x86/include/asm/i8259.h b/arch/x86/include/asm/i8259.h
index 39bcefc..be6492c 100644
--- a/arch/x86/include/asm/i8259.h
+++ b/arch/x86/include/asm/i8259.h
@@ -2,6 +2,7 @@
#define _ASM_X86_I8259_H
#include <linux/delay.h>
+#include <asm/io.h>
extern unsigned int cached_irq_mask;
@@ -68,6 +69,11 @@ struct legacy_pic {
extern struct legacy_pic *legacy_pic;
extern struct legacy_pic null_legacy_pic;
+static inline bool has_legacy_pic(void)
+{
+ return legacy_pic != &null_legacy_pic;
+}
+
static inline int nr_legacy_irqs(void)
{
return legacy_pic->nr_legacy_irqs;
diff --git a/arch/x86/include/asm/irqflags.h b/arch/x86/include/asm/irqflags.h
index ac7692d..508a062 100644
--- a/arch/x86/include/asm/irqflags.h
+++ b/arch/x86/include/asm/irqflags.h
@@ -12,7 +12,9 @@
* Interrupt control:
*/
-static inline unsigned long native_save_fl(void)
+/* Declaration required for gcc < 4.9 to prevent -Werror=missing-prototypes */
+extern inline unsigned long native_save_fl(void);
+extern inline unsigned long native_save_fl(void)
{
unsigned long flags;
@@ -30,7 +32,8 @@ static inline unsigned long native_save_fl(void)
return flags;
}
-static inline void native_restore_fl(unsigned long flags)
+extern inline void native_restore_fl(unsigned long flags);
+extern inline void native_restore_fl(unsigned long flags)
{
asm volatile("push %0 ; popf"
: /* no output */
diff --git a/arch/x86/include/asm/kvm_emulate.h b/arch/x86/include/asm/kvm_emulate.h
index fc3c7e4..ae357d0 100644
--- a/arch/x86/include/asm/kvm_emulate.h
+++ b/arch/x86/include/asm/kvm_emulate.h
@@ -105,11 +105,12 @@ struct x86_emulate_ops {
* @addr: [IN ] Linear address from which to read.
* @val: [OUT] Value read from memory, zero-extended to 'u_long'.
* @bytes: [IN ] Number of bytes to read from memory.
+ * @system:[IN ] Whether the access is forced to be at CPL0.
*/
int (*read_std)(struct x86_emulate_ctxt *ctxt,
unsigned long addr, void *val,
unsigned int bytes,
- struct x86_exception *fault);
+ struct x86_exception *fault, bool system);
/*
* read_phys: Read bytes of standard (non-emulated/special) memory.
@@ -127,10 +128,11 @@ struct x86_emulate_ops {
* @addr: [IN ] Linear address to which to write.
* @val: [OUT] Value write to memory, zero-extended to 'u_long'.
* @bytes: [IN ] Number of bytes to write to memory.
+ * @system:[IN ] Whether the access is forced to be at CPL0.
*/
int (*write_std)(struct x86_emulate_ctxt *ctxt,
unsigned long addr, void *val, unsigned int bytes,
- struct x86_exception *fault);
+ struct x86_exception *fault, bool system);
/*
* fetch: Read bytes of standard (non-emulated/special) memory.
* Used for instruction fetch.
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 20cfeeb..9a8167b 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -17,6 +17,7 @@
#include <linux/tracepoint.h>
#include <linux/cpumask.h>
#include <linux/irq_work.h>
+#include <linux/irq.h>
#include <linux/kvm.h>
#include <linux/kvm_para.h>
@@ -485,6 +486,7 @@ struct kvm_vcpu_arch {
u64 smbase;
bool tpr_access_reporting;
u64 ia32_xss;
+ u64 microcode_version;
/*
* Paging state of the vcpu
@@ -659,6 +661,9 @@ struct kvm_vcpu_arch {
int pending_ioapic_eoi;
int pending_external_vector;
+
+ /* Flush the L1 Data cache for L1TF mitigation on VMENTER */
+ bool l1tf_flush_l1d;
};
struct kvm_lpage_info {
@@ -819,6 +824,7 @@ struct kvm_vcpu_stat {
u64 signal_exits;
u64 irq_window_exits;
u64 nmi_window_exits;
+ u64 l1d_flush;
u64 halt_exits;
u64 halt_successful_poll;
u64 halt_attempted_poll;
@@ -864,7 +870,7 @@ struct kvm_x86_ops {
int (*hardware_setup)(void); /* __init */
void (*hardware_unsetup)(void); /* __exit */
bool (*cpu_has_accelerated_tpr)(void);
- bool (*cpu_has_high_real_mode_segbase)(void);
+ bool (*has_emulated_msr)(int index);
void (*cpuid_update)(struct kvm_vcpu *vcpu);
int (*vm_init)(struct kvm *kvm);
@@ -1020,6 +1026,8 @@ struct kvm_x86_ops {
void (*cancel_hv_timer)(struct kvm_vcpu *vcpu);
void (*setup_mce)(struct kvm_vcpu *vcpu);
+
+ int (*get_msr_feature)(struct kvm_msr_entry *entry);
};
struct kvm_arch_async_pf {
@@ -1316,7 +1324,7 @@ asmlinkage void kvm_spurious_fault(void);
"cmpb $0, kvm_rebooting \n\t" \
"jne 668b \n\t" \
__ASM_SIZE(push) " $666b \n\t" \
- "call kvm_spurious_fault \n\t" \
+ "jmp kvm_spurious_fault \n\t" \
".popsection \n\t" \
_ASM_EXTABLE(666b, 667b)
@@ -1338,6 +1346,7 @@ void kvm_vcpu_reload_apic_access_page(struct kvm_vcpu *vcpu);
void kvm_arch_mmu_notifier_invalidate_page(struct kvm *kvm,
unsigned long address);
+u64 kvm_get_arch_capabilities(void);
void kvm_define_shared_msr(unsigned index, u32 msr);
int kvm_set_shared_msr(unsigned index, u64 val, u64 mask);
diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h
index 5a295bb..7336508 100644
--- a/arch/x86/include/asm/mmu_context.h
+++ b/arch/x86/include/asm/mmu_context.h
@@ -113,7 +113,7 @@ static inline int init_new_context(struct task_struct *tsk,
#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
if (cpu_feature_enabled(X86_FEATURE_OSPKE)) {
- /* pkey 0 is the default and always allocated */
+ /* pkey 0 is the default and allocated implicitly */
mm->context.pkey_allocation_map = 0x1;
/* -1 means unallocated or invalid */
mm->context.execute_only_pkey = -1;
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index c768bc1..bbbb9b1 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -40,6 +40,8 @@
#define MSR_IA32_SPEC_CTRL 0x00000048 /* Speculation Control */
#define SPEC_CTRL_IBRS (1 << 0) /* Indirect Branch Restricted Speculation */
#define SPEC_CTRL_STIBP (1 << 1) /* Single Thread Indirect Branch Predictors */
+#define SPEC_CTRL_SSBD_SHIFT 2 /* Speculative Store Bypass Disable bit */
+#define SPEC_CTRL_SSBD (1 << SPEC_CTRL_SSBD_SHIFT) /* Speculative Store Bypass Disable */
#define MSR_IA32_PRED_CMD 0x00000049 /* Prediction Command */
#define PRED_CMD_IBPB (1 << 0) /* Indirect Branch Prediction Barrier */
@@ -61,6 +63,18 @@
#define MSR_IA32_ARCH_CAPABILITIES 0x0000010a
#define ARCH_CAP_RDCL_NO (1 << 0) /* Not susceptible to Meltdown */
#define ARCH_CAP_IBRS_ALL (1 << 1) /* Enhanced IBRS support */
+#define ARCH_CAP_SKIP_VMENTRY_L1DFLUSH (1 << 3) /* Skip L1D flush on vmentry */
+#define ARCH_CAP_SSB_NO (1 << 4) /*
+ * Not susceptible to Speculative Store Bypass
+ * attack, so no Speculative Store Bypass
+ * control required.
+ */
+
+#define MSR_IA32_FLUSH_CMD 0x0000010b
+#define L1D_FLUSH (1 << 0) /*
+ * Writeback and invalidate the
+ * L1 data cache.
+ */
#define MSR_IA32_BBL_CR_CTL 0x00000119
#define MSR_IA32_BBL_CR_CTL3 0x0000011e
@@ -135,6 +149,7 @@
/* DEBUGCTLMSR bits (others vary by model): */
#define DEBUGCTLMSR_LBR (1UL << 0) /* last branch recording */
+#define DEBUGCTLMSR_BTF_SHIFT 1
#define DEBUGCTLMSR_BTF (1UL << 1) /* single-step on branches */
#define DEBUGCTLMSR_TR (1UL << 6)
#define DEBUGCTLMSR_BTS (1UL << 7)
@@ -315,6 +330,8 @@
#define MSR_AMD64_IBSOPDATA4 0xc001103d
#define MSR_AMD64_IBS_REG_COUNT_MAX 8 /* includes MSR_AMD64_IBSBRTARGET */
+#define MSR_AMD64_VIRT_SPEC_CTRL 0xc001011f
+
/* Fam 17h MSRs */
#define MSR_F17H_IRPERF 0xc00000e9
diff --git a/arch/x86/include/asm/nospec-branch.h b/arch/x86/include/asm/nospec-branch.h
index f928ad9..1b41321 100644
--- a/arch/x86/include/asm/nospec-branch.h
+++ b/arch/x86/include/asm/nospec-branch.h
@@ -215,6 +215,15 @@ enum spectre_v2_mitigation {
SPECTRE_V2_RETPOLINE_GENERIC,
SPECTRE_V2_RETPOLINE_AMD,
SPECTRE_V2_IBRS,
+ SPECTRE_V2_IBRS_ENHANCED,
+};
+
+/* The Speculative Store Bypass disable variants */
+enum ssb_mitigation {
+ SPEC_STORE_BYPASS_NONE,
+ SPEC_STORE_BYPASS_DISABLE,
+ SPEC_STORE_BYPASS_PRCTL,
+ SPEC_STORE_BYPASS_SECCOMP,
};
extern char __indirect_thunk_start[];
@@ -241,22 +250,27 @@ static inline void vmexit_fill_RSB(void)
#endif
}
-#define alternative_msr_write(_msr, _val, _feature) \
- asm volatile(ALTERNATIVE("", \
- "movl %[msr], %%ecx\n\t" \
- "movl %[val], %%eax\n\t" \
- "movl $0, %%edx\n\t" \
- "wrmsr", \
- _feature) \
- : : [msr] "i" (_msr), [val] "i" (_val) \
- : "eax", "ecx", "edx", "memory")
+static __always_inline
+void alternative_msr_write(unsigned int msr, u64 val, unsigned int feature)
+{
+ asm volatile(ALTERNATIVE("", "wrmsr", %c[feature])
+ : : "c" (msr),
+ "a" ((u32)val),
+ "d" ((u32)(val >> 32)),
+ [feature] "i" (feature)
+ : "memory");
+}
static inline void indirect_branch_prediction_barrier(void)
{
- alternative_msr_write(MSR_IA32_PRED_CMD, PRED_CMD_IBPB,
- X86_FEATURE_USE_IBPB);
+ u64 val = PRED_CMD_IBPB;
+
+ alternative_msr_write(MSR_IA32_PRED_CMD, val, X86_FEATURE_USE_IBPB);
}
+/* The Intel SPEC CTRL MSR base value cache */
+extern u64 x86_spec_ctrl_base;
+
/*
* With retpoline, we must use IBRS to restrict branch prediction
* before calling into firmware.
@@ -265,14 +279,18 @@ static inline void indirect_branch_prediction_barrier(void)
*/
#define firmware_restrict_branch_speculation_start() \
do { \
+ u64 val = x86_spec_ctrl_base | SPEC_CTRL_IBRS; \
+ \
preempt_disable(); \
- alternative_msr_write(MSR_IA32_SPEC_CTRL, SPEC_CTRL_IBRS, \
+ alternative_msr_write(MSR_IA32_SPEC_CTRL, val, \
X86_FEATURE_USE_IBRS_FW); \
} while (0)
#define firmware_restrict_branch_speculation_end() \
do { \
- alternative_msr_write(MSR_IA32_SPEC_CTRL, 0, \
+ u64 val = x86_spec_ctrl_base; \
+ \
+ alternative_msr_write(MSR_IA32_SPEC_CTRL, val, \
X86_FEATURE_USE_IBRS_FW); \
preempt_enable(); \
} while (0)
diff --git a/arch/x86/include/asm/orc_types.h b/arch/x86/include/asm/orc_types.h
new file mode 100644
index 0000000..7dc777a
--- /dev/null
+++ b/arch/x86/include/asm/orc_types.h
@@ -0,0 +1,107 @@
+/*
+ * Copyright (C) 2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef _ORC_TYPES_H
+#define _ORC_TYPES_H
+
+#include <linux/types.h>
+#include <linux/compiler.h>
+
+/*
+ * The ORC_REG_* registers are base registers which are used to find other
+ * registers on the stack.
+ *
+ * ORC_REG_PREV_SP, also known as DWARF Call Frame Address (CFA), is the
+ * address of the previous frame: the caller's SP before it called the current
+ * function.
+ *
+ * ORC_REG_UNDEFINED means the corresponding register's value didn't change in
+ * the current frame.
+ *
+ * The most commonly used base registers are SP and BP -- which the previous SP
+ * is usually based on -- and PREV_SP and UNDEFINED -- which the previous BP is
+ * usually based on.
+ *
+ * The rest of the base registers are needed for special cases like entry code
+ * and GCC realigned stacks.
+ */
+#define ORC_REG_UNDEFINED 0
+#define ORC_REG_PREV_SP 1
+#define ORC_REG_DX 2
+#define ORC_REG_DI 3
+#define ORC_REG_BP 4
+#define ORC_REG_SP 5
+#define ORC_REG_R10 6
+#define ORC_REG_R13 7
+#define ORC_REG_BP_INDIRECT 8
+#define ORC_REG_SP_INDIRECT 9
+#define ORC_REG_MAX 15
+
+/*
+ * ORC_TYPE_CALL: Indicates that sp_reg+sp_offset resolves to PREV_SP (the
+ * caller's SP right before it made the call). Used for all callable
+ * functions, i.e. all C code and all callable asm functions.
+ *
+ * ORC_TYPE_REGS: Used in entry code to indicate that sp_reg+sp_offset points
+ * to a fully populated pt_regs from a syscall, interrupt, or exception.
+ *
+ * ORC_TYPE_REGS_IRET: Used in entry code to indicate that sp_reg+sp_offset
+ * points to the iret return frame.
+ *
+ * The UNWIND_HINT macros are used only for the unwind_hint struct. They
+ * aren't used in struct orc_entry due to size and complexity constraints.
+ * Objtool converts them to real types when it converts the hints to orc
+ * entries.
+ */
+#define ORC_TYPE_CALL 0
+#define ORC_TYPE_REGS 1
+#define ORC_TYPE_REGS_IRET 2
+#define UNWIND_HINT_TYPE_SAVE 3
+#define UNWIND_HINT_TYPE_RESTORE 4
+
+#ifndef __ASSEMBLY__
+/*
+ * This struct is more or less a vastly simplified version of the DWARF Call
+ * Frame Information standard. It contains only the necessary parts of DWARF
+ * CFI, simplified for ease of access by the in-kernel unwinder. It tells the
+ * unwinder how to find the previous SP and BP (and sometimes entry regs) on
+ * the stack for a given code address. Each instance of the struct corresponds
+ * to one or more code locations.
+ */
+struct orc_entry {
+ s16 sp_offset;
+ s16 bp_offset;
+ unsigned sp_reg:4;
+ unsigned bp_reg:4;
+ unsigned type:2;
+};
+
+/*
+ * This struct is used by asm and inline asm code to manually annotate the
+ * location of registers on the stack for the ORC unwinder.
+ *
+ * Type can be either ORC_TYPE_* or UNWIND_HINT_TYPE_*.
+ */
+struct unwind_hint {
+ u32 ip;
+ s16 sp_offset;
+ u8 sp_reg;
+ u8 type;
+};
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ORC_TYPES_H */
diff --git a/arch/x86/include/asm/page_32_types.h b/arch/x86/include/asm/page_32_types.h
index 3bae496..2622984 100644
--- a/arch/x86/include/asm/page_32_types.h
+++ b/arch/x86/include/asm/page_32_types.h
@@ -28,8 +28,13 @@
#define N_EXCEPTION_STACKS 1
#ifdef CONFIG_X86_PAE
-/* 44=32+12, the limit we can fit into an unsigned long pfn */
-#define __PHYSICAL_MASK_SHIFT 44
+/*
+ * This is beyond the 44 bit limit imposed by the 32bit long pfns,
+ * but we need the full mask to make sure inverted PROT_NONE
+ * entries have all the host bits set in a guest.
+ * The real limit is still 44 bits.
+ */
+#define __PHYSICAL_MASK_SHIFT 52
#define __VIRTUAL_MASK_SHIFT 32
#else /* !CONFIG_X86_PAE */
diff --git a/arch/x86/include/asm/percpu.h b/arch/x86/include/asm/percpu.h
index 84f58de..2cb5d0f 100644
--- a/arch/x86/include/asm/percpu.h
+++ b/arch/x86/include/asm/percpu.h
@@ -184,22 +184,22 @@ do { \
typeof(var) pfo_ret__; \
switch (sizeof(var)) { \
case 1: \
- asm(op "b "__percpu_arg(1)",%0" \
+ asm volatile(op "b "__percpu_arg(1)",%0"\
: "=q" (pfo_ret__) \
: "m" (var)); \
break; \
case 2: \
- asm(op "w "__percpu_arg(1)",%0" \
+ asm volatile(op "w "__percpu_arg(1)",%0"\
: "=r" (pfo_ret__) \
: "m" (var)); \
break; \
case 4: \
- asm(op "l "__percpu_arg(1)",%0" \
+ asm volatile(op "l "__percpu_arg(1)",%0"\
: "=r" (pfo_ret__) \
: "m" (var)); \
break; \
case 8: \
- asm(op "q "__percpu_arg(1)",%0" \
+ asm volatile(op "q "__percpu_arg(1)",%0"\
: "=r" (pfo_ret__) \
: "m" (var)); \
break; \
diff --git a/arch/x86/include/asm/pgtable-2level.h b/arch/x86/include/asm/pgtable-2level.h
index fd74a11..89c5033 100644
--- a/arch/x86/include/asm/pgtable-2level.h
+++ b/arch/x86/include/asm/pgtable-2level.h
@@ -77,4 +77,21 @@ static inline unsigned long pte_bitop(unsigned long value, unsigned int rightshi
#define __pte_to_swp_entry(pte) ((swp_entry_t) { (pte).pte_low })
#define __swp_entry_to_pte(x) ((pte_t) { .pte = (x).val })
+/* No inverted PFNs on 2 level page tables */
+
+static inline u64 protnone_mask(u64 val)
+{
+ return 0;
+}
+
+static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask)
+{
+ return val;
+}
+
+static inline bool __pte_needs_invert(u64 val)
+{
+ return false;
+}
+
#endif /* _ASM_X86_PGTABLE_2LEVEL_H */
diff --git a/arch/x86/include/asm/pgtable-3level.h b/arch/x86/include/asm/pgtable-3level.h
index cdaa58c..095dbc2 100644
--- a/arch/x86/include/asm/pgtable-3level.h
+++ b/arch/x86/include/asm/pgtable-3level.h
@@ -1,6 +1,8 @@
#ifndef _ASM_X86_PGTABLE_3LEVEL_H
#define _ASM_X86_PGTABLE_3LEVEL_H
+#include <asm/atomic64_32.h>
+
/*
* Intel Physical Address Extension (PAE) Mode - three-level page
* tables on PPro+ CPUs.
@@ -142,10 +144,7 @@ static inline pte_t native_ptep_get_and_clear(pte_t *ptep)
{
pte_t res;
- /* xchg acts as a barrier before the setting of the high bits */
- res.pte_low = xchg(&ptep->pte_low, 0);
- res.pte_high = ptep->pte_high;
- ptep->pte_high = 0;
+ res.pte = (pteval_t)atomic64_xchg((atomic64_t *)ptep, 0);
return res;
}
@@ -177,11 +176,44 @@ static inline pmd_t native_pmdp_get_and_clear(pmd_t *pmdp)
#endif
/* Encode and de-code a swap entry */
+#define SWP_TYPE_BITS 5
+
+#define SWP_OFFSET_FIRST_BIT (_PAGE_BIT_PROTNONE + 1)
+
+/* We always extract/encode the offset by shifting it all the way up, and then down again */
+#define SWP_OFFSET_SHIFT (SWP_OFFSET_FIRST_BIT + SWP_TYPE_BITS)
+
#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > 5)
#define __swp_type(x) (((x).val) & 0x1f)
#define __swp_offset(x) ((x).val >> 5)
#define __swp_entry(type, offset) ((swp_entry_t){(type) | (offset) << 5})
-#define __pte_to_swp_entry(pte) ((swp_entry_t){ (pte).pte_high })
-#define __swp_entry_to_pte(x) ((pte_t){ { .pte_high = (x).val } })
+
+/*
+ * Normally, __swp_entry() converts from arch-independent swp_entry_t to
+ * arch-dependent swp_entry_t, and __swp_entry_to_pte() just stores the result
+ * to pte. But here we have 32bit swp_entry_t and 64bit pte, and need to use the
+ * whole 64 bits. Thus, we shift the "real" arch-dependent conversion to
+ * __swp_entry_to_pte() through the following helper macro based on 64bit
+ * __swp_entry().
+ */
+#define __swp_pteval_entry(type, offset) ((pteval_t) { \
+ (~(pteval_t)(offset) << SWP_OFFSET_SHIFT >> SWP_TYPE_BITS) \
+ | ((pteval_t)(type) << (64 - SWP_TYPE_BITS)) })
+
+#define __swp_entry_to_pte(x) ((pte_t){ .pte = \
+ __swp_pteval_entry(__swp_type(x), __swp_offset(x)) })
+/*
+ * Analogically, __pte_to_swp_entry() doesn't just extract the arch-dependent
+ * swp_entry_t, but also has to convert it from 64bit to the 32bit
+ * intermediate representation, using the following macros based on 64bit
+ * __swp_type() and __swp_offset().
+ */
+#define __pteval_swp_type(x) ((unsigned long)((x).pte >> (64 - SWP_TYPE_BITS)))
+#define __pteval_swp_offset(x) ((unsigned long)(~((x).pte) << SWP_TYPE_BITS >> SWP_OFFSET_SHIFT))
+
+#define __pte_to_swp_entry(pte) (__swp_entry(__pteval_swp_type(pte), \
+ __pteval_swp_offset(pte)))
+
+#include <asm/pgtable-invert.h>
#endif /* _ASM_X86_PGTABLE_3LEVEL_H */
diff --git a/arch/x86/include/asm/pgtable-invert.h b/arch/x86/include/asm/pgtable-invert.h
new file mode 100644
index 0000000..a0c1525
--- /dev/null
+++ b/arch/x86/include/asm/pgtable-invert.h
@@ -0,0 +1,41 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_PGTABLE_INVERT_H
+#define _ASM_PGTABLE_INVERT_H 1
+
+#ifndef __ASSEMBLY__
+
+/*
+ * A clear pte value is special, and doesn't get inverted.
+ *
+ * Note that even users that only pass a pgprot_t (rather
+ * than a full pte) won't trigger the special zero case,
+ * because even PAGE_NONE has _PAGE_PROTNONE | _PAGE_ACCESSED
+ * set. So the all zero case really is limited to just the
+ * cleared page table entry case.
+ */
+static inline bool __pte_needs_invert(u64 val)
+{
+ return val && !(val & _PAGE_PRESENT);
+}
+
+/* Get a mask to xor with the page table entry to get the correct pfn. */
+static inline u64 protnone_mask(u64 val)
+{
+ return __pte_needs_invert(val) ? ~0ull : 0;
+}
+
+static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask)
+{
+ /*
+ * When a PTE transitions from NONE to !NONE or vice-versa
+ * invert the PFN part to stop speculation.
+ * pte_pfn undoes this when needed.
+ */
+ if (__pte_needs_invert(oldval) != __pte_needs_invert(val))
+ val = (val & ~mask) | (~val & mask);
+ return val;
+}
+
+#endif /* __ASSEMBLY__ */
+
+#endif
diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h
index 5af0401..5736306 100644
--- a/arch/x86/include/asm/pgtable.h
+++ b/arch/x86/include/asm/pgtable.h
@@ -165,19 +165,34 @@ static inline int pte_special(pte_t pte)
return pte_flags(pte) & _PAGE_SPECIAL;
}
+/* Entries that were set to PROT_NONE are inverted */
+
+static inline u64 protnone_mask(u64 val);
+
static inline unsigned long pte_pfn(pte_t pte)
{
- return (pte_val(pte) & PTE_PFN_MASK) >> PAGE_SHIFT;
+ phys_addr_t pfn = pte_val(pte);
+ pfn ^= protnone_mask(pfn);
+ return (pfn & PTE_PFN_MASK) >> PAGE_SHIFT;
}
static inline unsigned long pmd_pfn(pmd_t pmd)
{
- return (pmd_val(pmd) & pmd_pfn_mask(pmd)) >> PAGE_SHIFT;
+ phys_addr_t pfn = pmd_val(pmd);
+ pfn ^= protnone_mask(pfn);
+ return (pfn & pmd_pfn_mask(pmd)) >> PAGE_SHIFT;
}
static inline unsigned long pud_pfn(pud_t pud)
{
- return (pud_val(pud) & pud_pfn_mask(pud)) >> PAGE_SHIFT;
+ phys_addr_t pfn = pud_val(pud);
+ pfn ^= protnone_mask(pfn);
+ return (pfn & pud_pfn_mask(pud)) >> PAGE_SHIFT;
+}
+
+static inline unsigned long pgd_pfn(pgd_t pgd)
+{
+ return (pgd_val(pgd) & PTE_PFN_MASK) >> PAGE_SHIFT;
}
#define pte_page(pte) pfn_to_page(pte_pfn(pte))
@@ -340,11 +355,6 @@ static inline pmd_t pmd_mkwrite(pmd_t pmd)
return pmd_set_flags(pmd, _PAGE_RW);
}
-static inline pmd_t pmd_mknotpresent(pmd_t pmd)
-{
- return pmd_clear_flags(pmd, _PAGE_PRESENT | _PAGE_PROTNONE);
-}
-
#ifdef CONFIG_HAVE_ARCH_SOFT_DIRTY
static inline int pte_soft_dirty(pte_t pte)
{
@@ -394,19 +404,58 @@ static inline pgprotval_t massage_pgprot(pgprot_t pgprot)
static inline pte_t pfn_pte(unsigned long page_nr, pgprot_t pgprot)
{
- return __pte(((phys_addr_t)page_nr << PAGE_SHIFT) |
- massage_pgprot(pgprot));
+ phys_addr_t pfn = (phys_addr_t)page_nr << PAGE_SHIFT;
+ pfn ^= protnone_mask(pgprot_val(pgprot));
+ pfn &= PTE_PFN_MASK;
+ return __pte(pfn | massage_pgprot(pgprot));
}
static inline pmd_t pfn_pmd(unsigned long page_nr, pgprot_t pgprot)
{
- return __pmd(((phys_addr_t)page_nr << PAGE_SHIFT) |
- massage_pgprot(pgprot));
+ phys_addr_t pfn = (phys_addr_t)page_nr << PAGE_SHIFT;
+ pfn ^= protnone_mask(pgprot_val(pgprot));
+ pfn &= PHYSICAL_PMD_PAGE_MASK;
+ return __pmd(pfn | massage_pgprot(pgprot));
}
+static inline pud_t pfn_pud(unsigned long page_nr, pgprot_t pgprot)
+{
+ phys_addr_t pfn = (phys_addr_t)page_nr << PAGE_SHIFT;
+ pfn ^= protnone_mask(pgprot_val(pgprot));
+ pfn &= PHYSICAL_PUD_PAGE_MASK;
+ return __pud(pfn | massage_pgprot(pgprot));
+}
+
+static inline pmd_t pmd_mknotpresent(pmd_t pmd)
+{
+ return pfn_pmd(pmd_pfn(pmd),
+ __pgprot(pmd_flags(pmd) & ~(_PAGE_PRESENT|_PAGE_PROTNONE)));
+}
+
+static inline pud_t pud_set_flags(pud_t pud, pudval_t set)
+{
+ pudval_t v = native_pud_val(pud);
+
+ return __pud(v | set);
+}
+
+static inline pud_t pud_clear_flags(pud_t pud, pudval_t clear)
+{
+ pudval_t v = native_pud_val(pud);
+
+ return __pud(v & ~clear);
+}
+
+static inline pud_t pud_mkhuge(pud_t pud)
+{
+ return pud_set_flags(pud, _PAGE_PSE);
+}
+
+static inline u64 flip_protnone_guard(u64 oldval, u64 val, u64 mask);
+
static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
{
- pteval_t val = pte_val(pte);
+ pteval_t val = pte_val(pte), oldval = val;
/*
* Chop off the NX bit (if present), and add the NX portion of
@@ -414,17 +463,17 @@ static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
*/
val &= _PAGE_CHG_MASK;
val |= massage_pgprot(newprot) & ~_PAGE_CHG_MASK;
-
+ val = flip_protnone_guard(oldval, val, PTE_PFN_MASK);
return __pte(val);
}
static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot)
{
- pmdval_t val = pmd_val(pmd);
+ pmdval_t val = pmd_val(pmd), oldval = val;
val &= _HPAGE_CHG_MASK;
val |= massage_pgprot(newprot) & ~_HPAGE_CHG_MASK;
-
+ val = flip_protnone_guard(oldval, val, PHYSICAL_PMD_PAGE_MASK);
return __pmd(val);
}
@@ -577,8 +626,7 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
* Currently stuck as a macro due to indirect forward reference to
* linux/mmzone.h's __section_mem_map_addr() definition:
*/
-#define pmd_page(pmd) \
- pfn_to_page((pmd_val(pmd) & pmd_pfn_mask(pmd)) >> PAGE_SHIFT)
+#define pmd_page(pmd) pfn_to_page(pmd_pfn(pmd))
/*
* the pmd page can be thought of an array like this: pmd_t[PTRS_PER_PMD]
@@ -646,8 +694,7 @@ static inline unsigned long pud_page_vaddr(pud_t pud)
* Currently stuck as a macro due to indirect forward reference to
* linux/mmzone.h's __section_mem_map_addr() definition:
*/
-#define pud_page(pud) \
- pfn_to_page((pud_val(pud) & pud_pfn_mask(pud)) >> PAGE_SHIFT)
+#define pud_page(pud) pfn_to_page(pud_pfn(pud))
/* Find an entry in the second-level page table.. */
static inline pmd_t *pmd_offset(pud_t *pud, unsigned long address)
@@ -687,7 +734,7 @@ static inline unsigned long pgd_page_vaddr(pgd_t pgd)
* Currently stuck as a macro due to indirect forward reference to
* linux/mmzone.h's __section_mem_map_addr() definition:
*/
-#define pgd_page(pgd) pfn_to_page(pgd_val(pgd) >> PAGE_SHIFT)
+#define pgd_page(pgd) pfn_to_page(pgd_pfn(pgd))
/* to find an entry in a page-table-directory. */
static inline unsigned long pud_index(unsigned long address)
@@ -1010,6 +1057,15 @@ static inline u16 pte_flags_pkey(unsigned long pte_flags)
#endif
}
+
+#define __HAVE_ARCH_PFN_MODIFY_ALLOWED 1
+extern bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot);
+
+static inline bool arch_has_pfn_modify_check(void)
+{
+ return boot_cpu_has_bug(X86_BUG_L1TF);
+}
+
#include <asm-generic/pgtable.h>
#endif /* __ASSEMBLY__ */
diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h
index ce97c8c6..221a32e 100644
--- a/arch/x86/include/asm/pgtable_64.h
+++ b/arch/x86/include/asm/pgtable_64.h
@@ -166,29 +166,49 @@ static inline int pgd_large(pgd_t pgd) { return 0; }
/*
* Encode and de-code a swap entry
*
- * | ... | 11| 10| 9|8|7|6|5| 4| 3|2|1|0| <- bit number
- * | ... |SW3|SW2|SW1|G|L|D|A|CD|WT|U|W|P| <- bit names
- * | OFFSET (14->63) | TYPE (9-13) |0|X|X|X| X| X|X|X|0| <- swp entry
+ * | ... | 11| 10| 9|8|7|6|5| 4| 3|2| 1|0| <- bit number
+ * | ... |SW3|SW2|SW1|G|L|D|A|CD|WT|U| W|P| <- bit names
+ * | TYPE (59-63) | ~OFFSET (9-58) |0|0|X|X| X| X|X|SD|0| <- swp entry
*
* G (8) is aliased and used as a PROT_NONE indicator for
* !present ptes. We need to start storing swap entries above
* there. We also need to avoid using A and D because of an
* erratum where they can be incorrectly set by hardware on
* non-present PTEs.
+ *
+ * SD (1) in swp entry is used to store soft dirty bit, which helps us
+ * remember soft dirty over page migration
+ *
+ * Bit 7 in swp entry should be 0 because pmd_present checks not only P,
+ * but also L and G.
+ *
+ * The offset is inverted by a binary not operation to make the high
+ * physical bits set.
*/
-#define SWP_TYPE_FIRST_BIT (_PAGE_BIT_PROTNONE + 1)
-#define SWP_TYPE_BITS 5
-/* Place the offset above the type: */
-#define SWP_OFFSET_FIRST_BIT (SWP_TYPE_FIRST_BIT + SWP_TYPE_BITS)
+#define SWP_TYPE_BITS 5
+
+#define SWP_OFFSET_FIRST_BIT (_PAGE_BIT_PROTNONE + 1)
+
+/* We always extract/encode the offset by shifting it all the way up, and then down again */
+#define SWP_OFFSET_SHIFT (SWP_OFFSET_FIRST_BIT+SWP_TYPE_BITS)
#define MAX_SWAPFILES_CHECK() BUILD_BUG_ON(MAX_SWAPFILES_SHIFT > SWP_TYPE_BITS)
-#define __swp_type(x) (((x).val >> (SWP_TYPE_FIRST_BIT)) \
- & ((1U << SWP_TYPE_BITS) - 1))
-#define __swp_offset(x) ((x).val >> SWP_OFFSET_FIRST_BIT)
-#define __swp_entry(type, offset) ((swp_entry_t) { \
- ((type) << (SWP_TYPE_FIRST_BIT)) \
- | ((offset) << SWP_OFFSET_FIRST_BIT) })
+/* Extract the high bits for type */
+#define __swp_type(x) ((x).val >> (64 - SWP_TYPE_BITS))
+
+/* Shift up (to get rid of type), then down to get value */
+#define __swp_offset(x) (~(x).val << SWP_TYPE_BITS >> SWP_OFFSET_SHIFT)
+
+/*
+ * Shift the offset up "too far" by TYPE bits, then down again
+ * The offset is inverted by a binary not operation to make the high
+ * physical bits set.
+ */
+#define __swp_entry(type, offset) ((swp_entry_t) { \
+ (~(unsigned long)(offset) << SWP_OFFSET_SHIFT >> SWP_TYPE_BITS) \
+ | ((unsigned long)(type) << (64-SWP_TYPE_BITS)) })
+
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val((pte)) })
#define __swp_entry_to_pte(x) ((pte_t) { .pte = (x).val })
@@ -215,6 +235,8 @@ extern void cleanup_highmap(void);
extern void init_extra_mapping_uc(unsigned long phys, unsigned long size);
extern void init_extra_mapping_wb(unsigned long phys, unsigned long size);
+#include <asm/pgtable-invert.h>
+
#endif /* !__ASSEMBLY__ */
#endif /* _ASM_X86_PGTABLE_64_H */
diff --git a/arch/x86/include/asm/pgtable_types.h b/arch/x86/include/asm/pgtable_types.h
index f1c8ac4..93d089c 100644
--- a/arch/x86/include/asm/pgtable_types.h
+++ b/arch/x86/include/asm/pgtable_types.h
@@ -97,15 +97,15 @@
/*
* Tracking soft dirty bit when a page goes to a swap is tricky.
* We need a bit which can be stored in pte _and_ not conflict
- * with swap entry format. On x86 bits 6 and 7 are *not* involved
- * into swap entry computation, but bit 6 is used for nonlinear
- * file mapping, so we borrow bit 7 for soft dirty tracking.
+ * with swap entry format. On x86 bits 1-4 are *not* involved
+ * into swap entry computation, but bit 7 is used for thp migration,
+ * so we borrow bit 1 for soft dirty tracking.
*
* Please note that this bit must be treated as swap dirty page
- * mark if and only if the PTE has present bit clear!
+ * mark if and only if the PTE/PMD has present bit clear!
*/
#ifdef CONFIG_MEM_SOFT_DIRTY
-#define _PAGE_SWP_SOFT_DIRTY _PAGE_PSE
+#define _PAGE_SWP_SOFT_DIRTY _PAGE_RW
#else
#define _PAGE_SWP_SOFT_DIRTY (_AT(pteval_t, 0))
#endif
@@ -134,7 +134,7 @@
*/
#define _PAGE_CHG_MASK (PTE_PFN_MASK | _PAGE_PCD | _PAGE_PWT | \
_PAGE_SPECIAL | _PAGE_ACCESSED | _PAGE_DIRTY | \
- _PAGE_SOFT_DIRTY)
+ _PAGE_SOFT_DIRTY | _PAGE_DEVMAP)
#define _HPAGE_CHG_MASK (_PAGE_CHG_MASK | _PAGE_PSE)
/* The ASID is the lower 12 bits of CR3 */
diff --git a/arch/x86/include/asm/pkeys.h b/arch/x86/include/asm/pkeys.h
index b3b09b9..c50d6dc 100644
--- a/arch/x86/include/asm/pkeys.h
+++ b/arch/x86/include/asm/pkeys.h
@@ -1,6 +1,8 @@
#ifndef _ASM_X86_PKEYS_H
#define _ASM_X86_PKEYS_H
+#define ARCH_DEFAULT_PKEY 0
+
#define arch_max_pkey() (boot_cpu_has(X86_FEATURE_OSPKE) ? 16 : 1)
extern int arch_set_user_pkey_access(struct task_struct *tsk, int pkey,
@@ -14,7 +16,7 @@ extern int __execute_only_pkey(struct mm_struct *mm);
static inline int execute_only_pkey(struct mm_struct *mm)
{
if (!boot_cpu_has(X86_FEATURE_OSPKE))
- return 0;
+ return ARCH_DEFAULT_PKEY;
return __execute_only_pkey(mm);
}
@@ -48,13 +50,21 @@ bool mm_pkey_is_allocated(struct mm_struct *mm, int pkey)
{
/*
* "Allocated" pkeys are those that have been returned
- * from pkey_alloc(). pkey 0 is special, and never
- * returned from pkey_alloc().
+ * from pkey_alloc() or pkey 0 which is allocated
+ * implicitly when the mm is created.
*/
- if (pkey <= 0)
+ if (pkey < 0)
return false;
if (pkey >= arch_max_pkey())
return false;
+ /*
+ * The exec-only pkey is set in the allocation map, but
+ * is not available to any of the user interfaces like
+ * mprotect_pkey().
+ */
+ if (pkey == mm->context.execute_only_pkey)
+ return false;
+
return mm_pkey_allocation_map(mm) & (1U << pkey);
}
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index ec15ca2..ee8c629 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -136,6 +136,8 @@ struct cpuinfo_x86 {
/* Index into per_cpu list: */
u16 cpu_index;
u32 microcode;
+ /* Address space bits used by the cache internally */
+ u8 x86_cache_bits;
};
#define X86_VENDOR_INTEL 0
@@ -173,6 +175,11 @@ extern const struct seq_operations cpuinfo_op;
extern void cpu_detect(struct cpuinfo_x86 *c);
+static inline unsigned long long l1tf_pfn_limit(void)
+{
+ return BIT_ULL(boot_cpu_data.x86_cache_bits - 1 - PAGE_SHIFT);
+}
+
extern void early_cpu_init(void);
extern void identify_boot_cpu(void);
extern void identify_secondary_cpu(struct cpuinfo_x86 *);
@@ -855,4 +862,16 @@ bool xen_set_default_idle(void);
void stop_this_cpu(void *dummy);
void df_debug(struct pt_regs *regs, long error_code);
+
+enum l1tf_mitigations {
+ L1TF_MITIGATION_OFF,
+ L1TF_MITIGATION_FLUSH_NOWARN,
+ L1TF_MITIGATION_FLUSH,
+ L1TF_MITIGATION_FLUSH_NOSMT,
+ L1TF_MITIGATION_FULL,
+ L1TF_MITIGATION_FULL_FORCE
+};
+
+extern enum l1tf_mitigations l1tf_mitigation;
+
#endif /* _ASM_X86_PROCESSOR_H */
diff --git a/arch/x86/include/asm/qspinlock.h b/arch/x86/include/asm/qspinlock.h
index eaba080..9e78e96 100644
--- a/arch/x86/include/asm/qspinlock.h
+++ b/arch/x86/include/asm/qspinlock.h
@@ -4,6 +4,29 @@
#include <asm/cpufeature.h>
#include <asm-generic/qspinlock_types.h>
#include <asm/paravirt.h>
+#include <asm/rmwcc.h>
+
+#define _Q_PENDING_LOOPS (1 << 9)
+
+#define queued_fetch_set_pending_acquire queued_fetch_set_pending_acquire
+
+static __always_inline bool __queued_RMW_btsl(struct qspinlock *lock)
+{
+ GEN_BINARY_RMWcc(LOCK_PREFIX "btsl", lock->val.counter,
+ "I", _Q_PENDING_OFFSET, "%0", c);
+}
+
+static __always_inline u32 queued_fetch_set_pending_acquire(struct qspinlock *lock)
+{
+ u32 val = 0;
+
+ if (__queued_RMW_btsl(lock))
+ val |= _Q_PENDING_VAL;
+
+ val |= atomic_read(&lock->val) & ~_Q_PENDING_MASK;
+
+ return val;
+}
#define queued_spin_unlock queued_spin_unlock
/**
@@ -14,7 +37,7 @@
*/
static inline void native_queued_spin_unlock(struct qspinlock *lock)
{
- smp_store_release((u8 *)lock, 0);
+ smp_store_release(&lock->locked, 0);
}
#ifdef CONFIG_PARAVIRT_SPINLOCKS
diff --git a/arch/x86/include/asm/qspinlock_paravirt.h b/arch/x86/include/asm/qspinlock_paravirt.h
index 9d55f9b..fc75415 100644
--- a/arch/x86/include/asm/qspinlock_paravirt.h
+++ b/arch/x86/include/asm/qspinlock_paravirt.h
@@ -21,8 +21,7 @@ PV_CALLEE_SAVE_REGS_THUNK(__pv_queued_spin_unlock_slowpath);
*
* void __pv_queued_spin_unlock(struct qspinlock *lock)
* {
- * struct __qspinlock *l = (void *)lock;
- * u8 lockval = cmpxchg(&l->locked, _Q_LOCKED_VAL, 0);
+ * u8 lockval = cmpxchg(&lock->locked, _Q_LOCKED_VAL, 0);
*
* if (likely(lockval == _Q_LOCKED_VAL))
* return;
diff --git a/arch/x86/include/asm/smp.h b/arch/x86/include/asm/smp.h
index 026ea82..d25fb6b 100644
--- a/arch/x86/include/asm/smp.h
+++ b/arch/x86/include/asm/smp.h
@@ -156,7 +156,6 @@ static inline int wbinvd_on_all_cpus(void)
wbinvd();
return 0;
}
-#define smp_num_siblings 1
#endif /* CONFIG_SMP */
extern unsigned disabled_cpus;
diff --git a/arch/x86/include/asm/spec-ctrl.h b/arch/x86/include/asm/spec-ctrl.h
new file mode 100644
index 0000000..ae7c2c5
--- /dev/null
+++ b/arch/x86/include/asm/spec-ctrl.h
@@ -0,0 +1,80 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _ASM_X86_SPECCTRL_H_
+#define _ASM_X86_SPECCTRL_H_
+
+#include <linux/thread_info.h>
+#include <asm/nospec-branch.h>
+
+/*
+ * On VMENTER we must preserve whatever view of the SPEC_CTRL MSR
+ * the guest has, while on VMEXIT we restore the host view. This
+ * would be easier if SPEC_CTRL were architecturally maskable or
+ * shadowable for guests but this is not (currently) the case.
+ * Takes the guest view of SPEC_CTRL MSR as a parameter and also
+ * the guest's version of VIRT_SPEC_CTRL, if emulated.
+ */
+extern void x86_virt_spec_ctrl(u64 guest_spec_ctrl, u64 guest_virt_spec_ctrl, bool guest);
+
+/**
+ * x86_spec_ctrl_set_guest - Set speculation control registers for the guest
+ * @guest_spec_ctrl: The guest content of MSR_SPEC_CTRL
+ * @guest_virt_spec_ctrl: The guest controlled bits of MSR_VIRT_SPEC_CTRL
+ * (may get translated to MSR_AMD64_LS_CFG bits)
+ *
+ * Avoids writing to the MSR if the content/bits are the same
+ */
+static inline
+void x86_spec_ctrl_set_guest(u64 guest_spec_ctrl, u64 guest_virt_spec_ctrl)
+{
+ x86_virt_spec_ctrl(guest_spec_ctrl, guest_virt_spec_ctrl, true);
+}
+
+/**
+ * x86_spec_ctrl_restore_host - Restore host speculation control registers
+ * @guest_spec_ctrl: The guest content of MSR_SPEC_CTRL
+ * @guest_virt_spec_ctrl: The guest controlled bits of MSR_VIRT_SPEC_CTRL
+ * (may get translated to MSR_AMD64_LS_CFG bits)
+ *
+ * Avoids writing to the MSR if the content/bits are the same
+ */
+static inline
+void x86_spec_ctrl_restore_host(u64 guest_spec_ctrl, u64 guest_virt_spec_ctrl)
+{
+ x86_virt_spec_ctrl(guest_spec_ctrl, guest_virt_spec_ctrl, false);
+}
+
+/* AMD specific Speculative Store Bypass MSR data */
+extern u64 x86_amd_ls_cfg_base;
+extern u64 x86_amd_ls_cfg_ssbd_mask;
+
+static inline u64 ssbd_tif_to_spec_ctrl(u64 tifn)
+{
+ BUILD_BUG_ON(TIF_SSBD < SPEC_CTRL_SSBD_SHIFT);
+ return (tifn & _TIF_SSBD) >> (TIF_SSBD - SPEC_CTRL_SSBD_SHIFT);
+}
+
+static inline unsigned long ssbd_spec_ctrl_to_tif(u64 spec_ctrl)
+{
+ BUILD_BUG_ON(TIF_SSBD < SPEC_CTRL_SSBD_SHIFT);
+ return (spec_ctrl & SPEC_CTRL_SSBD) << (TIF_SSBD - SPEC_CTRL_SSBD_SHIFT);
+}
+
+static inline u64 ssbd_tif_to_amd_ls_cfg(u64 tifn)
+{
+ return (tifn & _TIF_SSBD) ? x86_amd_ls_cfg_ssbd_mask : 0ULL;
+}
+
+#ifdef CONFIG_SMP
+extern void speculative_store_bypass_ht_init(void);
+#else
+static inline void speculative_store_bypass_ht_init(void) { }
+#endif
+
+extern void speculative_store_bypass_update(unsigned long tif);
+
+static inline void speculative_store_bypass_update_current(void)
+{
+ speculative_store_bypass_update(current_thread_info()->flags);
+}
+
+#endif
diff --git a/arch/x86/include/asm/suspend_64.h b/arch/x86/include/asm/suspend_64.h
index 6136a18..2bd96b4 100644
--- a/arch/x86/include/asm/suspend_64.h
+++ b/arch/x86/include/asm/suspend_64.h
@@ -42,8 +42,7 @@ struct saved_context {
set_debugreg((thread)->debugreg##register, register)
/* routines for saving/restoring kernel state */
-extern int acpi_save_state_mem(void);
-extern char core_restore_code;
-extern char restore_registers;
+extern char core_restore_code[];
+extern char restore_registers[];
#endif /* _ASM_X86_SUSPEND_64_H */
diff --git a/arch/x86/include/asm/thread_info.h b/arch/x86/include/asm/thread_info.h
index 89978b9..2d8788a 100644
--- a/arch/x86/include/asm/thread_info.h
+++ b/arch/x86/include/asm/thread_info.h
@@ -83,6 +83,7 @@ struct thread_info {
#define TIF_SIGPENDING 2 /* signal pending */
#define TIF_NEED_RESCHED 3 /* rescheduling necessary */
#define TIF_SINGLESTEP 4 /* reenable singlestep on user return*/
+#define TIF_SSBD 5 /* Reduced data speculation */
#define TIF_SYSCALL_EMU 6 /* syscall emulation active */
#define TIF_SYSCALL_AUDIT 7 /* syscall auditing active */
#define TIF_SECCOMP 8 /* secure computing */
@@ -104,8 +105,9 @@ struct thread_info {
#define _TIF_SYSCALL_TRACE (1 << TIF_SYSCALL_TRACE)
#define _TIF_NOTIFY_RESUME (1 << TIF_NOTIFY_RESUME)
#define _TIF_SIGPENDING (1 << TIF_SIGPENDING)
-#define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP)
#define _TIF_NEED_RESCHED (1 << TIF_NEED_RESCHED)
+#define _TIF_SINGLESTEP (1 << TIF_SINGLESTEP)
+#define _TIF_SSBD (1 << TIF_SSBD)
#define _TIF_SYSCALL_EMU (1 << TIF_SYSCALL_EMU)
#define _TIF_SYSCALL_AUDIT (1 << TIF_SYSCALL_AUDIT)
#define _TIF_SECCOMP (1 << TIF_SECCOMP)
@@ -139,7 +141,7 @@ struct thread_info {
/* flags to check in __switch_to() */
#define _TIF_WORK_CTXSW \
- (_TIF_IO_BITMAP|_TIF_NOTSC|_TIF_BLOCKSTEP)
+ (_TIF_IO_BITMAP|_TIF_NOTSC|_TIF_BLOCKSTEP|_TIF_SSBD)
#define _TIF_WORK_CTXSW_PREV (_TIF_WORK_CTXSW|_TIF_USER_RETURN_NOTIFY)
#define _TIF_WORK_CTXSW_NEXT (_TIF_WORK_CTXSW)
diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h
index 99185a0..686a58d 100644
--- a/arch/x86/include/asm/tlbflush.h
+++ b/arch/x86/include/asm/tlbflush.h
@@ -111,6 +111,16 @@ static inline void cr4_clear_bits(unsigned long mask)
}
}
+static inline void cr4_toggle_bits(unsigned long mask)
+{
+ unsigned long cr4;
+
+ cr4 = this_cpu_read(cpu_tlbstate.cr4);
+ cr4 ^= mask;
+ this_cpu_write(cpu_tlbstate.cr4, cr4);
+ __write_cr4(cr4);
+}
+
/* Read the CR4 shadow. */
static inline unsigned long cr4_read_shadow(void)
{
diff --git a/arch/x86/include/asm/topology.h b/arch/x86/include/asm/topology.h
index cf75871..1fbb174 100644
--- a/arch/x86/include/asm/topology.h
+++ b/arch/x86/include/asm/topology.h
@@ -129,13 +129,17 @@ static inline int topology_max_smt_threads(void)
}
int topology_update_package_map(unsigned int apicid, unsigned int cpu);
-extern int topology_phys_to_logical_pkg(unsigned int pkg);
+int topology_phys_to_logical_pkg(unsigned int pkg);
+bool topology_is_primary_thread(unsigned int cpu);
+bool topology_smt_supported(void);
#else
#define topology_max_packages() (1)
static inline int
topology_update_package_map(unsigned int apicid, unsigned int cpu) { return 0; }
static inline int topology_phys_to_logical_pkg(unsigned int pkg) { return 0; }
static inline int topology_max_smt_threads(void) { return 1; }
+static inline bool topology_is_primary_thread(unsigned int cpu) { return true; }
+static inline bool topology_smt_supported(void) { return false; }
#endif
static inline void arch_fix_phys_package_id(int num, u32 slot)
diff --git a/arch/x86/include/asm/trace/fpu.h b/arch/x86/include/asm/trace/fpu.h
index 9217ab1..342e597 100644
--- a/arch/x86/include/asm/trace/fpu.h
+++ b/arch/x86/include/asm/trace/fpu.h
@@ -14,7 +14,6 @@ DECLARE_EVENT_CLASS(x86_fpu,
__field(struct fpu *, fpu)
__field(bool, fpregs_active)
__field(bool, fpstate_active)
- __field(int, counter)
__field(u64, xfeatures)
__field(u64, xcomp_bv)
),
@@ -23,17 +22,15 @@ DECLARE_EVENT_CLASS(x86_fpu,
__entry->fpu = fpu;
__entry->fpregs_active = fpu->fpregs_active;
__entry->fpstate_active = fpu->fpstate_active;
- __entry->counter = fpu->counter;
if (boot_cpu_has(X86_FEATURE_OSXSAVE)) {
__entry->xfeatures = fpu->state.xsave.header.xfeatures;
__entry->xcomp_bv = fpu->state.xsave.header.xcomp_bv;
}
),
- TP_printk("x86/fpu: %p fpregs_active: %d fpstate_active: %d counter: %d xfeatures: %llx xcomp_bv: %llx",
+ TP_printk("x86/fpu: %p fpregs_active: %d fpstate_active: %d xfeatures: %llx xcomp_bv: %llx",
__entry->fpu,
__entry->fpregs_active,
__entry->fpstate_active,
- __entry->counter,
__entry->xfeatures,
__entry->xcomp_bv
)
diff --git a/arch/x86/include/asm/unwind_hints.h b/arch/x86/include/asm/unwind_hints.h
new file mode 100644
index 0000000..5e02b11
--- /dev/null
+++ b/arch/x86/include/asm/unwind_hints.h
@@ -0,0 +1,103 @@
+#ifndef _ASM_X86_UNWIND_HINTS_H
+#define _ASM_X86_UNWIND_HINTS_H
+
+#include "orc_types.h"
+
+#ifdef __ASSEMBLY__
+
+/*
+ * In asm, there are two kinds of code: normal C-type callable functions and
+ * the rest. The normal callable functions can be called by other code, and
+ * don't do anything unusual with the stack. Such normal callable functions
+ * are annotated with the ENTRY/ENDPROC macros. Most asm code falls in this
+ * category. In this case, no special debugging annotations are needed because
+ * objtool can automatically generate the ORC data for the ORC unwinder to read
+ * at runtime.
+ *
+ * Anything which doesn't fall into the above category, such as syscall and
+ * interrupt handlers, tends to not be called directly by other functions, and
+ * often does unusual non-C-function-type things with the stack pointer. Such
+ * code needs to be annotated such that objtool can understand it. The
+ * following CFI hint macros are for this type of code.
+ *
+ * These macros provide hints to objtool about the state of the stack at each
+ * instruction. Objtool starts from the hints and follows the code flow,
+ * making automatic CFI adjustments when it sees pushes and pops, filling out
+ * the debuginfo as necessary. It will also warn if it sees any
+ * inconsistencies.
+ */
+.macro UNWIND_HINT sp_reg=ORC_REG_SP sp_offset=0 type=ORC_TYPE_CALL
+#ifdef CONFIG_STACK_VALIDATION
+.Lunwind_hint_ip_\@:
+ .pushsection .discard.unwind_hints
+ /* struct unwind_hint */
+ .long .Lunwind_hint_ip_\@ - .
+ .short \sp_offset
+ .byte \sp_reg
+ .byte \type
+ .popsection
+#endif
+.endm
+
+.macro UNWIND_HINT_EMPTY
+ UNWIND_HINT sp_reg=ORC_REG_UNDEFINED
+.endm
+
+.macro UNWIND_HINT_REGS base=%rsp offset=0 indirect=0 extra=1 iret=0
+ .if \base == %rsp && \indirect
+ .set sp_reg, ORC_REG_SP_INDIRECT
+ .elseif \base == %rsp
+ .set sp_reg, ORC_REG_SP
+ .elseif \base == %rbp
+ .set sp_reg, ORC_REG_BP
+ .elseif \base == %rdi
+ .set sp_reg, ORC_REG_DI
+ .elseif \base == %rdx
+ .set sp_reg, ORC_REG_DX
+ .elseif \base == %r10
+ .set sp_reg, ORC_REG_R10
+ .else
+ .error "UNWIND_HINT_REGS: bad base register"
+ .endif
+
+ .set sp_offset, \offset
+
+ .if \iret
+ .set type, ORC_TYPE_REGS_IRET
+ .elseif \extra == 0
+ .set type, ORC_TYPE_REGS_IRET
+ .set sp_offset, \offset + (16*8)
+ .else
+ .set type, ORC_TYPE_REGS
+ .endif
+
+ UNWIND_HINT sp_reg=sp_reg sp_offset=sp_offset type=type
+.endm
+
+.macro UNWIND_HINT_IRET_REGS base=%rsp offset=0
+ UNWIND_HINT_REGS base=\base offset=\offset iret=1
+.endm
+
+.macro UNWIND_HINT_FUNC sp_offset=8
+ UNWIND_HINT sp_offset=\sp_offset
+.endm
+
+#else /* !__ASSEMBLY__ */
+
+#define UNWIND_HINT(sp_reg, sp_offset, type) \
+ "987: \n\t" \
+ ".pushsection .discard.unwind_hints\n\t" \
+ /* struct unwind_hint */ \
+ ".long 987b - .\n\t" \
+ ".short " __stringify(sp_offset) "\n\t" \
+ ".byte " __stringify(sp_reg) "\n\t" \
+ ".byte " __stringify(type) "\n\t" \
+ ".popsection\n\t"
+
+#define UNWIND_HINT_SAVE UNWIND_HINT(0, 0, UNWIND_HINT_TYPE_SAVE)
+
+#define UNWIND_HINT_RESTORE UNWIND_HINT(0, 0, UNWIND_HINT_TYPE_RESTORE)
+
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ASM_X86_UNWIND_HINTS_H */
diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h
index 9cbfbef..72cacb0 100644
--- a/arch/x86/include/asm/vmx.h
+++ b/arch/x86/include/asm/vmx.h
@@ -499,4 +499,15 @@ enum vm_instruction_error_number {
VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID = 28,
};
+enum vmx_l1d_flush_state {
+ VMENTER_L1D_FLUSH_AUTO,
+ VMENTER_L1D_FLUSH_NEVER,
+ VMENTER_L1D_FLUSH_COND,
+ VMENTER_L1D_FLUSH_ALWAYS,
+ VMENTER_L1D_FLUSH_EPT_DISABLED,
+ VMENTER_L1D_FLUSH_NOT_REQUIRED,
+};
+
+extern enum vmx_l1d_flush_state l1tf_vmx_mitigation;
+
#endif
diff --git a/arch/x86/include/uapi/asm/kvm.h b/arch/x86/include/uapi/asm/kvm.h
index 739c0c5..1bb90fa 100644
--- a/arch/x86/include/uapi/asm/kvm.h
+++ b/arch/x86/include/uapi/asm/kvm.h
@@ -356,5 +356,6 @@ struct kvm_sync_regs {
#define KVM_X86_QUIRK_LINT0_REENABLED (1 << 0)
#define KVM_X86_QUIRK_CD_NW_CLEARED (1 << 1)
+#define KVM_X86_QUIRK_LAPIC_MMIO_HOLE (1 << 2)
#endif /* _ASM_X86_KVM_H */
diff --git a/arch/x86/include/uapi/asm/msgbuf.h b/arch/x86/include/uapi/asm/msgbuf.h
index 809134c..90ab9a79 100644
--- a/arch/x86/include/uapi/asm/msgbuf.h
+++ b/arch/x86/include/uapi/asm/msgbuf.h
@@ -1 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef __ASM_X64_MSGBUF_H
+#define __ASM_X64_MSGBUF_H
+
+#if !defined(__x86_64__) || !defined(__ILP32__)
#include <asm-generic/msgbuf.h>
+#else
+/*
+ * The msqid64_ds structure for x86 architecture with x32 ABI.
+ *
+ * On x86-32 and x86-64 we can just use the generic definition, but
+ * x32 uses the same binary layout as x86_64, which is differnet
+ * from other 32-bit architectures.
+ */
+
+struct msqid64_ds {
+ struct ipc64_perm msg_perm;
+ __kernel_time_t msg_stime; /* last msgsnd time */
+ __kernel_time_t msg_rtime; /* last msgrcv time */
+ __kernel_time_t msg_ctime; /* last change time */
+ __kernel_ulong_t msg_cbytes; /* current number of bytes on queue */
+ __kernel_ulong_t msg_qnum; /* number of messages in queue */
+ __kernel_ulong_t msg_qbytes; /* max number of bytes on queue */
+ __kernel_pid_t msg_lspid; /* pid of last msgsnd */
+ __kernel_pid_t msg_lrpid; /* last receive pid */
+ __kernel_ulong_t __unused4;
+ __kernel_ulong_t __unused5;
+};
+
+#endif
+
+#endif /* __ASM_GENERIC_MSGBUF_H */
diff --git a/arch/x86/include/uapi/asm/shmbuf.h b/arch/x86/include/uapi/asm/shmbuf.h
index 83c05fc..644421f 100644
--- a/arch/x86/include/uapi/asm/shmbuf.h
+++ b/arch/x86/include/uapi/asm/shmbuf.h
@@ -1 +1,43 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+#ifndef __ASM_X86_SHMBUF_H
+#define __ASM_X86_SHMBUF_H
+
+#if !defined(__x86_64__) || !defined(__ILP32__)
#include <asm-generic/shmbuf.h>
+#else
+/*
+ * The shmid64_ds structure for x86 architecture with x32 ABI.
+ *
+ * On x86-32 and x86-64 we can just use the generic definition, but
+ * x32 uses the same binary layout as x86_64, which is differnet
+ * from other 32-bit architectures.
+ */
+
+struct shmid64_ds {
+ struct ipc64_perm shm_perm; /* operation perms */
+ size_t shm_segsz; /* size of segment (bytes) */
+ __kernel_time_t shm_atime; /* last attach time */
+ __kernel_time_t shm_dtime; /* last detach time */
+ __kernel_time_t shm_ctime; /* last change time */
+ __kernel_pid_t shm_cpid; /* pid of creator */
+ __kernel_pid_t shm_lpid; /* pid of last operator */
+ __kernel_ulong_t shm_nattch; /* no. of current attaches */
+ __kernel_ulong_t __unused4;
+ __kernel_ulong_t __unused5;
+};
+
+struct shminfo64 {
+ __kernel_ulong_t shmmax;
+ __kernel_ulong_t shmmin;
+ __kernel_ulong_t shmmni;
+ __kernel_ulong_t shmseg;
+ __kernel_ulong_t shmall;
+ __kernel_ulong_t __unused1;
+ __kernel_ulong_t __unused2;
+ __kernel_ulong_t __unused3;
+ __kernel_ulong_t __unused4;
+};
+
+#endif
+
+#endif /* __ASM_X86_SHMBUF_H */
diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
index 79076d7..a9ba968 100644
--- a/arch/x86/kernel/Makefile
+++ b/arch/x86/kernel/Makefile
@@ -29,6 +29,7 @@
OBJECT_FILES_NON_STANDARD_relocate_kernel_$(BITS).o := y
OBJECT_FILES_NON_STANDARD_mcount_$(BITS).o := y
OBJECT_FILES_NON_STANDARD_test_nx.o := y
+OBJECT_FILES_NON_STANDARD_paravirt_patch_$(BITS).o := y
# If instrumentation of this dir is enabled, boot hangs during first second.
# Probably could be more selective here, but note that files related to irqs,
@@ -55,6 +56,7 @@
obj-y += tsc.o tsc_msr.o io_delay.o rtc.o
obj-y += pci-iommu_table.o
obj-y += resource.o
+obj-y += irqflags.o
obj-y += process.o
obj-y += fpu/
diff --git a/arch/x86/kernel/acpi/Makefile b/arch/x86/kernel/acpi/Makefile
index 26b78d8..85a9e17 100644
--- a/arch/x86/kernel/acpi/Makefile
+++ b/arch/x86/kernel/acpi/Makefile
@@ -1,3 +1,5 @@
+OBJECT_FILES_NON_STANDARD_wakeup_$(BITS).o := y
+
obj-$(CONFIG_ACPI) += boot.o
obj-$(CONFIG_ACPI_SLEEP) += sleep.o wakeup_$(BITS).o
obj-$(CONFIG_ACPI_APEI) += apei.o
diff --git a/arch/x86/kernel/apic/apic.c b/arch/x86/kernel/apic/apic.c
index c6583ef..4f2af1e 100644
--- a/arch/x86/kernel/apic/apic.c
+++ b/arch/x86/kernel/apic/apic.c
@@ -34,6 +34,7 @@
#include <linux/dmi.h>
#include <linux/smp.h>
#include <linux/mm.h>
+#include <linux/irq.h>
#include <asm/trace/irq_vectors.h>
#include <asm/irq_remapping.h>
@@ -55,6 +56,7 @@
#include <asm/mce.h>
#include <asm/tsc.h>
#include <asm/hypervisor.h>
+#include <asm/irq_regs.h>
unsigned int num_processors;
@@ -1403,7 +1405,7 @@ void setup_local_APIC(void)
* TODO: set up through-local-APIC from through-I/O-APIC? --macro
*/
value = apic_read(APIC_LVT0) & APIC_LVT_MASKED;
- if (!cpu && (pic_mode || !value)) {
+ if (!cpu && (pic_mode || !value || skip_ioapic_setup)) {
value = APIC_DM_EXTINT;
apic_printk(APIC_VERBOSE, "enabled ExtINT on CPU#%d\n", cpu);
} else {
@@ -2041,6 +2043,23 @@ static int cpuid_to_apicid[] = {
[0 ... NR_CPUS - 1] = -1,
};
+#ifdef CONFIG_SMP
+/**
+ * apic_id_is_primary_thread - Check whether APIC ID belongs to a primary thread
+ * @id: APIC ID to check
+ */
+bool apic_id_is_primary_thread(unsigned int apicid)
+{
+ u32 mask;
+
+ if (smp_num_siblings == 1)
+ return true;
+ /* Isolate the SMT bit(s) in the APICID and check for 0 */
+ mask = (1U << (fls(smp_num_siblings) - 1)) - 1;
+ return !(apicid & mask);
+}
+#endif
+
/*
* Should use this API to allocate logical CPU IDs to keep nr_logical_cpuids
* and cpuid_to_apicid[] synchronized.
diff --git a/arch/x86/kernel/apic/htirq.c b/arch/x86/kernel/apic/htirq.c
index ae50d34..89d6e96 100644
--- a/arch/x86/kernel/apic/htirq.c
+++ b/arch/x86/kernel/apic/htirq.c
@@ -16,6 +16,8 @@
#include <linux/device.h>
#include <linux/pci.h>
#include <linux/htirq.h>
+#include <linux/irq.h>
+
#include <asm/irqdomain.h>
#include <asm/hw_irq.h>
#include <asm/apic.h>
diff --git a/arch/x86/kernel/apic/io_apic.c b/arch/x86/kernel/apic/io_apic.c
index cf89928..d34629d 100644
--- a/arch/x86/kernel/apic/io_apic.c
+++ b/arch/x86/kernel/apic/io_apic.c
@@ -32,6 +32,7 @@
#include <linux/mm.h>
#include <linux/interrupt.h>
+#include <linux/irq.h>
#include <linux/init.h>
#include <linux/delay.h>
#include <linux/sched.h>
diff --git a/arch/x86/kernel/apic/msi.c b/arch/x86/kernel/apic/msi.c
index 015bbf3..cfd17a3 100644
--- a/arch/x86/kernel/apic/msi.c
+++ b/arch/x86/kernel/apic/msi.c
@@ -12,6 +12,7 @@
*/
#include <linux/mm.h>
#include <linux/interrupt.h>
+#include <linux/irq.h>
#include <linux/pci.h>
#include <linux/dmar.h>
#include <linux/hpet.h>
diff --git a/arch/x86/kernel/apic/vector.c b/arch/x86/kernel/apic/vector.c
index 4922ab6..c6bd3f9 100644
--- a/arch/x86/kernel/apic/vector.c
+++ b/arch/x86/kernel/apic/vector.c
@@ -11,6 +11,7 @@
* published by the Free Software Foundation.
*/
#include <linux/interrupt.h>
+#include <linux/irq.h>
#include <linux/init.h>
#include <linux/compiler.h>
#include <linux/slab.h>
diff --git a/arch/x86/kernel/apm_32.c b/arch/x86/kernel/apm_32.c
index 51287cd..313a85a 100644
--- a/arch/x86/kernel/apm_32.c
+++ b/arch/x86/kernel/apm_32.c
@@ -239,6 +239,7 @@
#include <asm/olpc.h>
#include <asm/paravirt.h>
#include <asm/reboot.h>
+#include <asm/nospec-branch.h>
#if defined(CONFIG_APM_DISPLAY_BLANK) && defined(CONFIG_VT)
extern int (*console_blank_hook)(int);
@@ -613,11 +614,13 @@ static long __apm_bios_call(void *_call)
gdt[0x40 / 8] = bad_bios_desc;
apm_irq_save(flags);
+ firmware_restrict_branch_speculation_start();
APM_DO_SAVE_SEGS;
apm_bios_call_asm(call->func, call->ebx, call->ecx,
&call->eax, &call->ebx, &call->ecx, &call->edx,
&call->esi);
APM_DO_RESTORE_SEGS;
+ firmware_restrict_branch_speculation_end();
apm_irq_restore(flags);
gdt[0x40 / 8] = save_desc_40;
put_cpu();
@@ -689,10 +692,12 @@ static long __apm_bios_call_simple(void *_call)
gdt[0x40 / 8] = bad_bios_desc;
apm_irq_save(flags);
+ firmware_restrict_branch_speculation_start();
APM_DO_SAVE_SEGS;
error = apm_bios_call_simple_asm(call->func, call->ebx, call->ecx,
&call->eax);
APM_DO_RESTORE_SEGS;
+ firmware_restrict_branch_speculation_end();
apm_irq_restore(flags);
gdt[0x40 / 8] = save_desc_40;
put_cpu();
diff --git a/arch/x86/kernel/check.c b/arch/x86/kernel/check.c
index 145863d..a8b2158 100644
--- a/arch/x86/kernel/check.c
+++ b/arch/x86/kernel/check.c
@@ -30,6 +30,11 @@ static __init int set_corruption_check(char *arg)
ssize_t ret;
unsigned long val;
+ if (!arg) {
+ pr_err("memory_corruption_check config string not provided\n");
+ return -EINVAL;
+ }
+
ret = kstrtoul(arg, 10, &val);
if (ret)
return ret;
@@ -44,6 +49,11 @@ static __init int set_corruption_check_period(char *arg)
ssize_t ret;
unsigned long val;
+ if (!arg) {
+ pr_err("memory_corruption_check_period config string not provided\n");
+ return -EINVAL;
+ }
+
ret = kstrtoul(arg, 10, &val);
if (ret)
return ret;
@@ -58,6 +68,11 @@ static __init int set_corruption_check_size(char *arg)
char *end;
unsigned size;
+ if (!arg) {
+ pr_err("memory_corruption_check_size config string not provided\n");
+ return -EINVAL;
+ }
+
size = memparse(arg, &end);
if (*end == '\0')
diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
index c375bc6..4c2648b 100644
--- a/arch/x86/kernel/cpu/amd.c
+++ b/arch/x86/kernel/cpu/amd.c
@@ -9,6 +9,7 @@
#include <asm/processor.h>
#include <asm/apic.h>
#include <asm/cpu.h>
+#include <asm/spec-ctrl.h>
#include <asm/smp.h>
#include <asm/pci-direct.h>
#include <asm/delay.h>
@@ -295,13 +296,34 @@ static int nearby_node(int apicid)
}
#endif
+static void amd_get_topology_early(struct cpuinfo_x86 *c)
+{
+ if (cpu_has(c, X86_FEATURE_TOPOEXT))
+ smp_num_siblings = ((cpuid_ebx(0x8000001e) >> 8) & 0xff) + 1;
+}
+
+/*
+ * Fix up cpu_core_id for pre-F17h systems to be in the
+ * [0 .. cores_per_node - 1] range. Not really needed but
+ * kept so as not to break existing setups.
+ */
+static void legacy_fixup_core_id(struct cpuinfo_x86 *c)
+{
+ u32 cus_per_node;
+
+ if (c->x86 >= 0x17)
+ return;
+
+ cus_per_node = c->x86_max_cores / nodes_per_socket;
+ c->cpu_core_id %= cus_per_node;
+}
+
/*
* Fixup core topology information for
* (1) AMD multi-node processors
* Assumption: Number of cores in each internal node is the same.
* (2) AMD processors supporting compute units
*/
-#ifdef CONFIG_SMP
static void amd_get_topology(struct cpuinfo_x86 *c)
{
u8 node_id;
@@ -314,7 +336,6 @@ static void amd_get_topology(struct cpuinfo_x86 *c)
cpuid(0x8000001e, &eax, &ebx, &ecx, &edx);
node_id = ecx & 0xff;
- smp_num_siblings = ((ebx >> 8) & 0xff) + 1;
if (c->x86 == 0x15)
c->cu_id = ebx & 0xff;
@@ -352,18 +373,11 @@ static void amd_get_topology(struct cpuinfo_x86 *c)
} else
return;
- /* fixup multi-node processor information */
if (nodes_per_socket > 1) {
- u32 cus_per_node;
-
set_cpu_cap(c, X86_FEATURE_AMD_DCM);
- cus_per_node = c->x86_max_cores / nodes_per_socket;
-
- /* core id has to be in the [0 .. cores_per_node - 1] range */
- c->cpu_core_id %= cus_per_node;
+ legacy_fixup_core_id(c);
}
}
-#endif
/*
* On a AMD dual core setup the lower bits of the APIC id distinguish the cores.
@@ -371,7 +385,6 @@ static void amd_get_topology(struct cpuinfo_x86 *c)
*/
static void amd_detect_cmp(struct cpuinfo_x86 *c)
{
-#ifdef CONFIG_SMP
unsigned bits;
int cpu = smp_processor_id();
@@ -383,16 +396,11 @@ static void amd_detect_cmp(struct cpuinfo_x86 *c)
/* use socket ID also for last level cache */
per_cpu(cpu_llc_id, cpu) = c->phys_proc_id;
amd_get_topology(c);
-#endif
}
u16 amd_get_nb_id(int cpu)
{
- u16 id = 0;
-#ifdef CONFIG_SMP
- id = per_cpu(cpu_llc_id, cpu);
-#endif
- return id;
+ return per_cpu(cpu_llc_id, cpu);
}
EXPORT_SYMBOL_GPL(amd_get_nb_id);
@@ -542,10 +550,32 @@ static void bsp_init_amd(struct cpuinfo_x86 *c)
rdmsrl(MSR_FAM10H_NODE_ID, value);
nodes_per_socket = ((value >> 3) & 7) + 1;
}
+
+ if (c->x86 >= 0x15 && c->x86 <= 0x17) {
+ unsigned int bit;
+
+ switch (c->x86) {
+ case 0x15: bit = 54; break;
+ case 0x16: bit = 33; break;
+ case 0x17: bit = 10; break;
+ default: return;
+ }
+ /*
+ * Try to cache the base value so further operations can
+ * avoid RMW. If that faults, do not enable SSBD.
+ */
+ if (!rdmsrl_safe(MSR_AMD64_LS_CFG, &x86_amd_ls_cfg_base)) {
+ setup_force_cpu_cap(X86_FEATURE_LS_CFG_SSBD);
+ setup_force_cpu_cap(X86_FEATURE_SSBD);
+ x86_amd_ls_cfg_ssbd_mask = 1ULL << bit;
+ }
+ }
}
static void early_init_amd(struct cpuinfo_x86 *c)
{
+ u64 value;
+
early_init_amd_mc(c);
/*
@@ -612,6 +642,23 @@ static void early_init_amd(struct cpuinfo_x86 *c)
*/
if (cpu_has_amd_erratum(c, amd_erratum_400))
set_cpu_bug(c, X86_BUG_AMD_E400);
+
+
+ /* Re-enable TopologyExtensions if switched off by BIOS */
+ if (c->x86 == 0x15 &&
+ (c->x86_model >= 0x10 && c->x86_model <= 0x6f) &&
+ !cpu_has(c, X86_FEATURE_TOPOEXT)) {
+
+ if (msr_set_bit(0xc0011005, 54) > 0) {
+ rdmsrl(0xc0011005, value);
+ if (value & BIT_64(54)) {
+ set_cpu_cap(c, X86_FEATURE_TOPOEXT);
+ pr_info_once(FW_INFO "CPU: Re-enabling disabled Topology Extensions Support.\n");
+ }
+ }
+ }
+
+ amd_get_topology_early(c);
}
static void init_amd_k8(struct cpuinfo_x86 *c)
@@ -703,19 +750,6 @@ static void init_amd_bd(struct cpuinfo_x86 *c)
{
u64 value;
- /* re-enable TopologyExtensions if switched off by BIOS */
- if ((c->x86_model >= 0x10) && (c->x86_model <= 0x6f) &&
- !cpu_has(c, X86_FEATURE_TOPOEXT)) {
-
- if (msr_set_bit(0xc0011005, 54) > 0) {
- rdmsrl(0xc0011005, value);
- if (value & BIT_64(54)) {
- set_cpu_cap(c, X86_FEATURE_TOPOEXT);
- pr_info_once(FW_INFO "CPU: Re-enabling disabled Topology Extensions Support.\n");
- }
- }
- }
-
/*
* The way access filter has a performance penalty on some workloads.
* Disable it on the affected CPUs.
@@ -728,6 +762,17 @@ static void init_amd_bd(struct cpuinfo_x86 *c)
}
}
+static void init_amd_zn(struct cpuinfo_x86 *c)
+{
+ set_cpu_cap(c, X86_FEATURE_ZEN);
+ /*
+ * Fix erratum 1076: CPB feature bit not being set in CPUID. It affects
+ * all up to and including B1.
+ */
+ if (c->x86_model <= 1 && c->x86_stepping <= 1)
+ set_cpu_cap(c, X86_FEATURE_CPB);
+}
+
static void init_amd(struct cpuinfo_x86 *c)
{
u32 dummy;
@@ -758,6 +803,7 @@ static void init_amd(struct cpuinfo_x86 *c)
case 0x10: init_amd_gh(c); break;
case 0x12: init_amd_ln(c); break;
case 0x15: init_amd_bd(c); break;
+ case 0x17: init_amd_zn(c); break;
}
/* Enable workaround for FXSAVE leak */
@@ -766,15 +812,8 @@ static void init_amd(struct cpuinfo_x86 *c)
cpu_detect_cache_sizes(c);
- /* Multi core CPU? */
- if (c->extended_cpuid_level >= 0x80000008) {
- amd_detect_cmp(c);
- srat_detect_node(c);
- }
-
-#ifdef CONFIG_X86_32
- detect_ht(c);
-#endif
+ amd_detect_cmp(c);
+ srat_detect_node(c);
init_amd_cacheinfo(c);
@@ -824,8 +863,9 @@ static void init_amd(struct cpuinfo_x86 *c)
if (cpu_has(c, X86_FEATURE_3DNOW) || cpu_has(c, X86_FEATURE_LM))
set_cpu_cap(c, X86_FEATURE_3DNOWPREFETCH);
- /* AMD CPUs don't reset SS attributes on SYSRET */
- set_cpu_bug(c, X86_BUG_SYSRET_SS_ATTRS);
+ /* AMD CPUs don't reset SS attributes on SYSRET, Xen does. */
+ if (!cpu_has(c, X86_FEATURE_XENPV))
+ set_cpu_bug(c, X86_BUG_SYSRET_SS_ATTRS);
}
#ifdef CONFIG_X86_32
diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c
index b8b0b6e..6221166 100644
--- a/arch/x86/kernel/cpu/bugs.c
+++ b/arch/x86/kernel/cpu/bugs.c
@@ -11,34 +11,86 @@
#include <linux/utsname.h>
#include <linux/cpu.h>
#include <linux/module.h>
+#include <linux/nospec.h>
+#include <linux/prctl.h>
-#include <asm/nospec-branch.h>
+#include <asm/spec-ctrl.h>
#include <asm/cmdline.h>
#include <asm/bugs.h>
#include <asm/processor.h>
#include <asm/processor-flags.h>
#include <asm/fpu/internal.h>
#include <asm/msr.h>
+#include <asm/vmx.h>
#include <asm/paravirt.h>
#include <asm/alternative.h>
#include <asm/pgtable.h>
#include <asm/cacheflush.h>
#include <asm/intel-family.h>
+#include <asm/e820.h>
static void __init spectre_v2_select_mitigation(void);
+static void __init ssb_select_mitigation(void);
+static void __init l1tf_select_mitigation(void);
+
+/*
+ * Our boot-time value of the SPEC_CTRL MSR. We read it once so that any
+ * writes to SPEC_CTRL contain whatever reserved bits have been set.
+ */
+u64 __ro_after_init x86_spec_ctrl_base;
+EXPORT_SYMBOL_GPL(x86_spec_ctrl_base);
+
+/*
+ * The vendor and possibly platform specific bits which can be modified in
+ * x86_spec_ctrl_base.
+ */
+static u64 __ro_after_init x86_spec_ctrl_mask = SPEC_CTRL_IBRS;
+
+/*
+ * AMD specific MSR info for Speculative Store Bypass control.
+ * x86_amd_ls_cfg_ssbd_mask is initialized in identify_boot_cpu().
+ */
+u64 __ro_after_init x86_amd_ls_cfg_base;
+u64 __ro_after_init x86_amd_ls_cfg_ssbd_mask;
void __init check_bugs(void)
{
identify_boot_cpu();
+ /*
+ * identify_boot_cpu() initialized SMT support information, let the
+ * core code know.
+ */
+ cpu_smt_check_topology_early();
+
if (!IS_ENABLED(CONFIG_SMP)) {
pr_info("CPU: ");
print_cpu_info(&boot_cpu_data);
}
+ /*
+ * Read the SPEC_CTRL MSR to account for reserved bits which may
+ * have unknown values. AMD64_LS_CFG MSR is cached in the early AMD
+ * init code as it is not enumerated and depends on the family.
+ */
+ if (boot_cpu_has(X86_FEATURE_MSR_SPEC_CTRL))
+ rdmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base);
+
+ /* Allow STIBP in MSR_SPEC_CTRL if supported */
+ if (boot_cpu_has(X86_FEATURE_STIBP))
+ x86_spec_ctrl_mask |= SPEC_CTRL_STIBP;
+
/* Select the proper spectre mitigation before patching alternatives */
spectre_v2_select_mitigation();
+ /*
+ * Select proper mitigation for any exposure to the Speculative Store
+ * Bypass vulnerability.
+ */
+ ssb_select_mitigation();
+
+ l1tf_select_mitigation();
+
#ifdef CONFIG_X86_32
/*
* Check whether we are able to run this kernel safely on SMP.
@@ -87,12 +139,82 @@ static const char *spectre_v2_strings[] = {
[SPECTRE_V2_RETPOLINE_MINIMAL_AMD] = "Vulnerable: Minimal AMD ASM retpoline",
[SPECTRE_V2_RETPOLINE_GENERIC] = "Mitigation: Full generic retpoline",
[SPECTRE_V2_RETPOLINE_AMD] = "Mitigation: Full AMD retpoline",
+ [SPECTRE_V2_IBRS_ENHANCED] = "Mitigation: Enhanced IBRS",
};
#undef pr_fmt
#define pr_fmt(fmt) "Spectre V2 : " fmt
-static enum spectre_v2_mitigation spectre_v2_enabled = SPECTRE_V2_NONE;
+static enum spectre_v2_mitigation spectre_v2_enabled __ro_after_init =
+ SPECTRE_V2_NONE;
+
+void
+x86_virt_spec_ctrl(u64 guest_spec_ctrl, u64 guest_virt_spec_ctrl, bool setguest)
+{
+ u64 msrval, guestval, hostval = x86_spec_ctrl_base;
+ struct thread_info *ti = current_thread_info();
+
+ /* Is MSR_SPEC_CTRL implemented ? */
+ if (static_cpu_has(X86_FEATURE_MSR_SPEC_CTRL)) {
+ /*
+ * Restrict guest_spec_ctrl to supported values. Clear the
+ * modifiable bits in the host base value and or the
+ * modifiable bits from the guest value.
+ */
+ guestval = hostval & ~x86_spec_ctrl_mask;
+ guestval |= guest_spec_ctrl & x86_spec_ctrl_mask;
+
+ /* SSBD controlled in MSR_SPEC_CTRL */
+ if (static_cpu_has(X86_FEATURE_SPEC_CTRL_SSBD))
+ hostval |= ssbd_tif_to_spec_ctrl(ti->flags);
+
+ if (hostval != guestval) {
+ msrval = setguest ? guestval : hostval;
+ wrmsrl(MSR_IA32_SPEC_CTRL, msrval);
+ }
+ }
+
+ /*
+ * If SSBD is not handled in MSR_SPEC_CTRL on AMD, update
+ * MSR_AMD64_L2_CFG or MSR_VIRT_SPEC_CTRL if supported.
+ */
+ if (!static_cpu_has(X86_FEATURE_LS_CFG_SSBD) &&
+ !static_cpu_has(X86_FEATURE_VIRT_SSBD))
+ return;
+
+ /*
+ * If the host has SSBD mitigation enabled, force it in the host's
+ * virtual MSR value. If its not permanently enabled, evaluate
+ * current's TIF_SSBD thread flag.
+ */
+ if (static_cpu_has(X86_FEATURE_SPEC_STORE_BYPASS_DISABLE))
+ hostval = SPEC_CTRL_SSBD;
+ else
+ hostval = ssbd_tif_to_spec_ctrl(ti->flags);
+
+ /* Sanitize the guest value */
+ guestval = guest_virt_spec_ctrl & SPEC_CTRL_SSBD;
+
+ if (hostval != guestval) {
+ unsigned long tif;
+
+ tif = setguest ? ssbd_spec_ctrl_to_tif(guestval) :
+ ssbd_spec_ctrl_to_tif(hostval);
+
+ speculative_store_bypass_update(tif);
+ }
+}
+EXPORT_SYMBOL_GPL(x86_virt_spec_ctrl);
+
+static void x86_amd_ssb_disable(void)
+{
+ u64 msrval = x86_amd_ls_cfg_base | x86_amd_ls_cfg_ssbd_mask;
+
+ if (boot_cpu_has(X86_FEATURE_VIRT_SSBD))
+ wrmsrl(MSR_AMD64_VIRT_SPEC_CTRL, SPEC_CTRL_SSBD);
+ else if (boot_cpu_has(X86_FEATURE_LS_CFG_SSBD))
+ wrmsrl(MSR_AMD64_LS_CFG, msrval);
+}
#ifdef RETPOLINE
static bool spectre_v2_bad_module;
@@ -200,23 +322,6 @@ static enum spectre_v2_mitigation_cmd __init spectre_v2_parse_cmdline(void)
return cmd;
}
-/* Check for Skylake-like CPUs (for RSB handling) */
-static bool __init is_skylake_era(void)
-{
- if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL &&
- boot_cpu_data.x86 == 6) {
- switch (boot_cpu_data.x86_model) {
- case INTEL_FAM6_SKYLAKE_MOBILE:
- case INTEL_FAM6_SKYLAKE_DESKTOP:
- case INTEL_FAM6_SKYLAKE_X:
- case INTEL_FAM6_KABYLAKE_MOBILE:
- case INTEL_FAM6_KABYLAKE_DESKTOP:
- return true;
- }
- }
- return false;
-}
-
static void __init spectre_v2_select_mitigation(void)
{
enum spectre_v2_mitigation_cmd cmd = spectre_v2_parse_cmdline();
@@ -236,6 +341,13 @@ static void __init spectre_v2_select_mitigation(void)
case SPECTRE_V2_CMD_FORCE:
case SPECTRE_V2_CMD_AUTO:
+ if (boot_cpu_has(X86_FEATURE_IBRS_ENHANCED)) {
+ mode = SPECTRE_V2_IBRS_ENHANCED;
+ /* Force it so VMEXIT will restore correctly */
+ x86_spec_ctrl_base |= SPEC_CTRL_IBRS;
+ wrmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base);
+ goto specv2_set_mode;
+ }
if (IS_ENABLED(CONFIG_RETPOLINE))
goto retpoline_auto;
break;
@@ -273,26 +385,20 @@ static void __init spectre_v2_select_mitigation(void)
setup_force_cpu_cap(X86_FEATURE_RETPOLINE);
}
+specv2_set_mode:
spectre_v2_enabled = mode;
pr_info("%s\n", spectre_v2_strings[mode]);
/*
- * If neither SMEP nor PTI are available, there is a risk of
- * hitting userspace addresses in the RSB after a context switch
- * from a shallow call stack to a deeper one. To prevent this fill
- * the entire RSB, even when using IBRS.
+ * If spectre v2 protection has been enabled, unconditionally fill
+ * RSB during a context switch; this protects against two independent
+ * issues:
*
- * Skylake era CPUs have a separate issue with *underflow* of the
- * RSB, when they will predict 'ret' targets from the generic BTB.
- * The proper mitigation for this is IBRS. If IBRS is not supported
- * or deactivated in favour of retpolines the RSB fill on context
- * switch is required.
+ * - RSB underflow (and switch to BTB) on Skylake+
+ * - SpectreRSB variant of spectre v2 on X86_BUG_SPECTRE_V2 CPUs
*/
- if ((!boot_cpu_has(X86_FEATURE_KAISER) &&
- !boot_cpu_has(X86_FEATURE_SMEP)) || is_skylake_era()) {
- setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW);
- pr_info("Spectre v2 mitigation: Filling RSB on context switch\n");
- }
+ setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW);
+ pr_info("Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch\n");
/* Initialize Indirect Branch Prediction Barrier if supported */
if (boot_cpu_has(X86_FEATURE_IBPB)) {
@@ -302,41 +408,466 @@ static void __init spectre_v2_select_mitigation(void)
/*
* Retpoline means the kernel is safe because it has no indirect
- * branches. But firmware isn't, so use IBRS to protect that.
+ * branches. Enhanced IBRS protects firmware too, so, enable restricted
+ * speculation around firmware calls only when Enhanced IBRS isn't
+ * supported.
+ *
+ * Use "mode" to check Enhanced IBRS instead of boot_cpu_has(), because
+ * the user might select retpoline on the kernel command line and if
+ * the CPU supports Enhanced IBRS, kernel might un-intentionally not
+ * enable IBRS around firmware calls.
*/
- if (boot_cpu_has(X86_FEATURE_IBRS)) {
+ if (boot_cpu_has(X86_FEATURE_IBRS) && mode != SPECTRE_V2_IBRS_ENHANCED) {
setup_force_cpu_cap(X86_FEATURE_USE_IBRS_FW);
pr_info("Enabling Restricted Speculation for firmware calls\n");
}
}
#undef pr_fmt
+#define pr_fmt(fmt) "Speculative Store Bypass: " fmt
+
+static enum ssb_mitigation ssb_mode __ro_after_init = SPEC_STORE_BYPASS_NONE;
+
+/* The kernel command line selection */
+enum ssb_mitigation_cmd {
+ SPEC_STORE_BYPASS_CMD_NONE,
+ SPEC_STORE_BYPASS_CMD_AUTO,
+ SPEC_STORE_BYPASS_CMD_ON,
+ SPEC_STORE_BYPASS_CMD_PRCTL,
+ SPEC_STORE_BYPASS_CMD_SECCOMP,
+};
+
+static const char *ssb_strings[] = {
+ [SPEC_STORE_BYPASS_NONE] = "Vulnerable",
+ [SPEC_STORE_BYPASS_DISABLE] = "Mitigation: Speculative Store Bypass disabled",
+ [SPEC_STORE_BYPASS_PRCTL] = "Mitigation: Speculative Store Bypass disabled via prctl",
+ [SPEC_STORE_BYPASS_SECCOMP] = "Mitigation: Speculative Store Bypass disabled via prctl and seccomp",
+};
+
+static const struct {
+ const char *option;
+ enum ssb_mitigation_cmd cmd;
+} ssb_mitigation_options[] = {
+ { "auto", SPEC_STORE_BYPASS_CMD_AUTO }, /* Platform decides */
+ { "on", SPEC_STORE_BYPASS_CMD_ON }, /* Disable Speculative Store Bypass */
+ { "off", SPEC_STORE_BYPASS_CMD_NONE }, /* Don't touch Speculative Store Bypass */
+ { "prctl", SPEC_STORE_BYPASS_CMD_PRCTL }, /* Disable Speculative Store Bypass via prctl */
+ { "seccomp", SPEC_STORE_BYPASS_CMD_SECCOMP }, /* Disable Speculative Store Bypass via prctl and seccomp */
+};
+
+static enum ssb_mitigation_cmd __init ssb_parse_cmdline(void)
+{
+ enum ssb_mitigation_cmd cmd = SPEC_STORE_BYPASS_CMD_AUTO;
+ char arg[20];
+ int ret, i;
+
+ if (cmdline_find_option_bool(boot_command_line, "nospec_store_bypass_disable")) {
+ return SPEC_STORE_BYPASS_CMD_NONE;
+ } else {
+ ret = cmdline_find_option(boot_command_line, "spec_store_bypass_disable",
+ arg, sizeof(arg));
+ if (ret < 0)
+ return SPEC_STORE_BYPASS_CMD_AUTO;
+
+ for (i = 0; i < ARRAY_SIZE(ssb_mitigation_options); i++) {
+ if (!match_option(arg, ret, ssb_mitigation_options[i].option))
+ continue;
+
+ cmd = ssb_mitigation_options[i].cmd;
+ break;
+ }
+
+ if (i >= ARRAY_SIZE(ssb_mitigation_options)) {
+ pr_err("unknown option (%s). Switching to AUTO select\n", arg);
+ return SPEC_STORE_BYPASS_CMD_AUTO;
+ }
+ }
+
+ return cmd;
+}
+
+static enum ssb_mitigation __init __ssb_select_mitigation(void)
+{
+ enum ssb_mitigation mode = SPEC_STORE_BYPASS_NONE;
+ enum ssb_mitigation_cmd cmd;
+
+ if (!boot_cpu_has(X86_FEATURE_SSBD))
+ return mode;
+
+ cmd = ssb_parse_cmdline();
+ if (!boot_cpu_has_bug(X86_BUG_SPEC_STORE_BYPASS) &&
+ (cmd == SPEC_STORE_BYPASS_CMD_NONE ||
+ cmd == SPEC_STORE_BYPASS_CMD_AUTO))
+ return mode;
+
+ switch (cmd) {
+ case SPEC_STORE_BYPASS_CMD_AUTO:
+ case SPEC_STORE_BYPASS_CMD_SECCOMP:
+ /*
+ * Choose prctl+seccomp as the default mode if seccomp is
+ * enabled.
+ */
+ if (IS_ENABLED(CONFIG_SECCOMP))
+ mode = SPEC_STORE_BYPASS_SECCOMP;
+ else
+ mode = SPEC_STORE_BYPASS_PRCTL;
+ break;
+ case SPEC_STORE_BYPASS_CMD_ON:
+ mode = SPEC_STORE_BYPASS_DISABLE;
+ break;
+ case SPEC_STORE_BYPASS_CMD_PRCTL:
+ mode = SPEC_STORE_BYPASS_PRCTL;
+ break;
+ case SPEC_STORE_BYPASS_CMD_NONE:
+ break;
+ }
+
+ /*
+ * We have three CPU feature flags that are in play here:
+ * - X86_BUG_SPEC_STORE_BYPASS - CPU is susceptible.
+ * - X86_FEATURE_SSBD - CPU is able to turn off speculative store bypass
+ * - X86_FEATURE_SPEC_STORE_BYPASS_DISABLE - engage the mitigation
+ */
+ if (mode == SPEC_STORE_BYPASS_DISABLE) {
+ setup_force_cpu_cap(X86_FEATURE_SPEC_STORE_BYPASS_DISABLE);
+ /*
+ * Intel uses the SPEC CTRL MSR Bit(2) for this, while AMD uses
+ * a completely different MSR and bit dependent on family.
+ */
+ switch (boot_cpu_data.x86_vendor) {
+ case X86_VENDOR_INTEL:
+ x86_spec_ctrl_base |= SPEC_CTRL_SSBD;
+ x86_spec_ctrl_mask |= SPEC_CTRL_SSBD;
+ wrmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base);
+ break;
+ case X86_VENDOR_AMD:
+ x86_amd_ssb_disable();
+ break;
+ }
+ }
+
+ return mode;
+}
+
+static void ssb_select_mitigation(void)
+{
+ ssb_mode = __ssb_select_mitigation();
+
+ if (boot_cpu_has_bug(X86_BUG_SPEC_STORE_BYPASS))
+ pr_info("%s\n", ssb_strings[ssb_mode]);
+}
+
+#undef pr_fmt
+#define pr_fmt(fmt) "Speculation prctl: " fmt
+
+static int ssb_prctl_set(struct task_struct *task, unsigned long ctrl)
+{
+ bool update;
+
+ if (ssb_mode != SPEC_STORE_BYPASS_PRCTL &&
+ ssb_mode != SPEC_STORE_BYPASS_SECCOMP)
+ return -ENXIO;
+
+ switch (ctrl) {
+ case PR_SPEC_ENABLE:
+ /* If speculation is force disabled, enable is not allowed */
+ if (task_spec_ssb_force_disable(task))
+ return -EPERM;
+ task_clear_spec_ssb_disable(task);
+ update = test_and_clear_tsk_thread_flag(task, TIF_SSBD);
+ break;
+ case PR_SPEC_DISABLE:
+ task_set_spec_ssb_disable(task);
+ update = !test_and_set_tsk_thread_flag(task, TIF_SSBD);
+ break;
+ case PR_SPEC_FORCE_DISABLE:
+ task_set_spec_ssb_disable(task);
+ task_set_spec_ssb_force_disable(task);
+ update = !test_and_set_tsk_thread_flag(task, TIF_SSBD);
+ break;
+ default:
+ return -ERANGE;
+ }
+
+ /*
+ * If being set on non-current task, delay setting the CPU
+ * mitigation until it is next scheduled.
+ */
+ if (task == current && update)
+ speculative_store_bypass_update_current();
+
+ return 0;
+}
+
+int arch_prctl_spec_ctrl_set(struct task_struct *task, unsigned long which,
+ unsigned long ctrl)
+{
+ switch (which) {
+ case PR_SPEC_STORE_BYPASS:
+ return ssb_prctl_set(task, ctrl);
+ default:
+ return -ENODEV;
+ }
+}
+
+#ifdef CONFIG_SECCOMP
+void arch_seccomp_spec_mitigate(struct task_struct *task)
+{
+ if (ssb_mode == SPEC_STORE_BYPASS_SECCOMP)
+ ssb_prctl_set(task, PR_SPEC_FORCE_DISABLE);
+}
+#endif
+
+static int ssb_prctl_get(struct task_struct *task)
+{
+ switch (ssb_mode) {
+ case SPEC_STORE_BYPASS_DISABLE:
+ return PR_SPEC_DISABLE;
+ case SPEC_STORE_BYPASS_SECCOMP:
+ case SPEC_STORE_BYPASS_PRCTL:
+ if (task_spec_ssb_force_disable(task))
+ return PR_SPEC_PRCTL | PR_SPEC_FORCE_DISABLE;
+ if (task_spec_ssb_disable(task))
+ return PR_SPEC_PRCTL | PR_SPEC_DISABLE;
+ return PR_SPEC_PRCTL | PR_SPEC_ENABLE;
+ default:
+ if (boot_cpu_has_bug(X86_BUG_SPEC_STORE_BYPASS))
+ return PR_SPEC_ENABLE;
+ return PR_SPEC_NOT_AFFECTED;
+ }
+}
+
+int arch_prctl_spec_ctrl_get(struct task_struct *task, unsigned long which)
+{
+ switch (which) {
+ case PR_SPEC_STORE_BYPASS:
+ return ssb_prctl_get(task);
+ default:
+ return -ENODEV;
+ }
+}
+
+void x86_spec_ctrl_setup_ap(void)
+{
+ if (boot_cpu_has(X86_FEATURE_MSR_SPEC_CTRL))
+ wrmsrl(MSR_IA32_SPEC_CTRL, x86_spec_ctrl_base);
+
+ if (ssb_mode == SPEC_STORE_BYPASS_DISABLE)
+ x86_amd_ssb_disable();
+}
+
+#undef pr_fmt
+#define pr_fmt(fmt) "L1TF: " fmt
+
+/* Default mitigation for L1TF-affected CPUs */
+enum l1tf_mitigations l1tf_mitigation __ro_after_init = L1TF_MITIGATION_FLUSH;
+#if IS_ENABLED(CONFIG_KVM_INTEL)
+EXPORT_SYMBOL_GPL(l1tf_mitigation);
+#endif
+enum vmx_l1d_flush_state l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;
+EXPORT_SYMBOL_GPL(l1tf_vmx_mitigation);
+
+/*
+ * These CPUs all support 44bits physical address space internally in the
+ * cache but CPUID can report a smaller number of physical address bits.
+ *
+ * The L1TF mitigation uses the top most address bit for the inversion of
+ * non present PTEs. When the installed memory reaches into the top most
+ * address bit due to memory holes, which has been observed on machines
+ * which report 36bits physical address bits and have 32G RAM installed,
+ * then the mitigation range check in l1tf_select_mitigation() triggers.
+ * This is a false positive because the mitigation is still possible due to
+ * the fact that the cache uses 44bit internally. Use the cache bits
+ * instead of the reported physical bits and adjust them on the affected
+ * machines to 44bit if the reported bits are less than 44.
+ */
+static void override_cache_bits(struct cpuinfo_x86 *c)
+{
+ if (c->x86 != 6)
+ return;
+
+ switch (c->x86_model) {
+ case INTEL_FAM6_NEHALEM:
+ case INTEL_FAM6_WESTMERE:
+ case INTEL_FAM6_SANDYBRIDGE:
+ case INTEL_FAM6_IVYBRIDGE:
+ case INTEL_FAM6_HASWELL_CORE:
+ case INTEL_FAM6_HASWELL_ULT:
+ case INTEL_FAM6_HASWELL_GT3E:
+ case INTEL_FAM6_BROADWELL_CORE:
+ case INTEL_FAM6_BROADWELL_GT3E:
+ case INTEL_FAM6_SKYLAKE_MOBILE:
+ case INTEL_FAM6_SKYLAKE_DESKTOP:
+ case INTEL_FAM6_KABYLAKE_MOBILE:
+ case INTEL_FAM6_KABYLAKE_DESKTOP:
+ if (c->x86_cache_bits < 44)
+ c->x86_cache_bits = 44;
+ break;
+ }
+}
+
+static void __init l1tf_select_mitigation(void)
+{
+ u64 half_pa;
+
+ if (!boot_cpu_has_bug(X86_BUG_L1TF))
+ return;
+
+ override_cache_bits(&boot_cpu_data);
+
+ switch (l1tf_mitigation) {
+ case L1TF_MITIGATION_OFF:
+ case L1TF_MITIGATION_FLUSH_NOWARN:
+ case L1TF_MITIGATION_FLUSH:
+ break;
+ case L1TF_MITIGATION_FLUSH_NOSMT:
+ case L1TF_MITIGATION_FULL:
+ cpu_smt_disable(false);
+ break;
+ case L1TF_MITIGATION_FULL_FORCE:
+ cpu_smt_disable(true);
+ break;
+ }
+
+#if CONFIG_PGTABLE_LEVELS == 2
+ pr_warn("Kernel not compiled for PAE. No mitigation for L1TF\n");
+ return;
+#endif
+
+ half_pa = (u64)l1tf_pfn_limit() << PAGE_SHIFT;
+ if (e820_any_mapped(half_pa, ULLONG_MAX - half_pa, E820_RAM)) {
+ pr_warn("System has more than MAX_PA/2 memory. L1TF mitigation not effective.\n");
+ pr_info("You may make it effective by booting the kernel with mem=%llu parameter.\n",
+ half_pa);
+ pr_info("However, doing so will make a part of your RAM unusable.\n");
+ pr_info("Reading https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html might help you decide.\n");
+ return;
+ }
+
+ setup_force_cpu_cap(X86_FEATURE_L1TF_PTEINV);
+}
+
+static int __init l1tf_cmdline(char *str)
+{
+ if (!boot_cpu_has_bug(X86_BUG_L1TF))
+ return 0;
+
+ if (!str)
+ return -EINVAL;
+
+ if (!strcmp(str, "off"))
+ l1tf_mitigation = L1TF_MITIGATION_OFF;
+ else if (!strcmp(str, "flush,nowarn"))
+ l1tf_mitigation = L1TF_MITIGATION_FLUSH_NOWARN;
+ else if (!strcmp(str, "flush"))
+ l1tf_mitigation = L1TF_MITIGATION_FLUSH;
+ else if (!strcmp(str, "flush,nosmt"))
+ l1tf_mitigation = L1TF_MITIGATION_FLUSH_NOSMT;
+ else if (!strcmp(str, "full"))
+ l1tf_mitigation = L1TF_MITIGATION_FULL;
+ else if (!strcmp(str, "full,force"))
+ l1tf_mitigation = L1TF_MITIGATION_FULL_FORCE;
+
+ return 0;
+}
+early_param("l1tf", l1tf_cmdline);
+
+#undef pr_fmt
#ifdef CONFIG_SYSFS
+
+#define L1TF_DEFAULT_MSG "Mitigation: PTE Inversion"
+
+#if IS_ENABLED(CONFIG_KVM_INTEL)
+static const char *l1tf_vmx_states[] = {
+ [VMENTER_L1D_FLUSH_AUTO] = "auto",
+ [VMENTER_L1D_FLUSH_NEVER] = "vulnerable",
+ [VMENTER_L1D_FLUSH_COND] = "conditional cache flushes",
+ [VMENTER_L1D_FLUSH_ALWAYS] = "cache flushes",
+ [VMENTER_L1D_FLUSH_EPT_DISABLED] = "EPT disabled",
+ [VMENTER_L1D_FLUSH_NOT_REQUIRED] = "flush not necessary"
+};
+
+static ssize_t l1tf_show_state(char *buf)
+{
+ if (l1tf_vmx_mitigation == VMENTER_L1D_FLUSH_AUTO)
+ return sprintf(buf, "%s\n", L1TF_DEFAULT_MSG);
+
+ if (l1tf_vmx_mitigation == VMENTER_L1D_FLUSH_EPT_DISABLED ||
+ (l1tf_vmx_mitigation == VMENTER_L1D_FLUSH_NEVER &&
+ cpu_smt_control == CPU_SMT_ENABLED))
+ return sprintf(buf, "%s; VMX: %s\n", L1TF_DEFAULT_MSG,
+ l1tf_vmx_states[l1tf_vmx_mitigation]);
+
+ return sprintf(buf, "%s; VMX: %s, SMT %s\n", L1TF_DEFAULT_MSG,
+ l1tf_vmx_states[l1tf_vmx_mitigation],
+ cpu_smt_control == CPU_SMT_ENABLED ? "vulnerable" : "disabled");
+}
+#else
+static ssize_t l1tf_show_state(char *buf)
+{
+ return sprintf(buf, "%s\n", L1TF_DEFAULT_MSG);
+}
+#endif
+
+static ssize_t cpu_show_common(struct device *dev, struct device_attribute *attr,
+ char *buf, unsigned int bug)
+{
+ if (!boot_cpu_has_bug(bug))
+ return sprintf(buf, "Not affected\n");
+
+ switch (bug) {
+ case X86_BUG_CPU_MELTDOWN:
+ if (boot_cpu_has(X86_FEATURE_KAISER))
+ return sprintf(buf, "Mitigation: PTI\n");
+
+ break;
+
+ case X86_BUG_SPECTRE_V1:
+ return sprintf(buf, "Mitigation: __user pointer sanitization\n");
+
+ case X86_BUG_SPECTRE_V2:
+ return sprintf(buf, "%s%s%s%s\n", spectre_v2_strings[spectre_v2_enabled],
+ boot_cpu_has(X86_FEATURE_USE_IBPB) ? ", IBPB" : "",
+ boot_cpu_has(X86_FEATURE_USE_IBRS_FW) ? ", IBRS_FW" : "",
+ spectre_v2_module_string());
+
+ case X86_BUG_SPEC_STORE_BYPASS:
+ return sprintf(buf, "%s\n", ssb_strings[ssb_mode]);
+
+ case X86_BUG_L1TF:
+ if (boot_cpu_has(X86_FEATURE_L1TF_PTEINV))
+ return l1tf_show_state(buf);
+ break;
+ default:
+ break;
+ }
+
+ return sprintf(buf, "Vulnerable\n");
+}
+
ssize_t cpu_show_meltdown(struct device *dev, struct device_attribute *attr, char *buf)
{
- if (!boot_cpu_has_bug(X86_BUG_CPU_MELTDOWN))
- return sprintf(buf, "Not affected\n");
- if (boot_cpu_has(X86_FEATURE_KAISER))
- return sprintf(buf, "Mitigation: PTI\n");
- return sprintf(buf, "Vulnerable\n");
+ return cpu_show_common(dev, attr, buf, X86_BUG_CPU_MELTDOWN);
}
ssize_t cpu_show_spectre_v1(struct device *dev, struct device_attribute *attr, char *buf)
{
- if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V1))
- return sprintf(buf, "Not affected\n");
- return sprintf(buf, "Mitigation: __user pointer sanitization\n");
+ return cpu_show_common(dev, attr, buf, X86_BUG_SPECTRE_V1);
}
ssize_t cpu_show_spectre_v2(struct device *dev, struct device_attribute *attr, char *buf)
{
- if (!boot_cpu_has_bug(X86_BUG_SPECTRE_V2))
- return sprintf(buf, "Not affected\n");
+ return cpu_show_common(dev, attr, buf, X86_BUG_SPECTRE_V2);
+}
- return sprintf(buf, "%s%s%s%s\n", spectre_v2_strings[spectre_v2_enabled],
- boot_cpu_has(X86_FEATURE_USE_IBPB) ? ", IBPB" : "",
- boot_cpu_has(X86_FEATURE_USE_IBRS_FW) ? ", IBRS_FW" : "",
- spectre_v2_module_string());
+ssize_t cpu_show_spec_store_bypass(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ return cpu_show_common(dev, attr, buf, X86_BUG_SPEC_STORE_BYPASS);
+}
+
+ssize_t cpu_show_l1tf(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ return cpu_show_common(dev, attr, buf, X86_BUG_L1TF);
}
#endif
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index 301bbd1..3c01610 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -61,6 +61,13 @@ cpumask_var_t cpu_callin_mask;
/* representing cpus for which sibling maps can be computed */
cpumask_var_t cpu_sibling_setup_mask;
+/* Number of siblings per CPU package */
+int smp_num_siblings = 1;
+EXPORT_SYMBOL(smp_num_siblings);
+
+/* Last level cache ID of each logical CPU */
+DEFINE_PER_CPU_READ_MOSTLY(u16, cpu_llc_id) = BAD_APICID;
+
/* correctly size the local cpu masks */
void __init setup_cpu_local_masks(void)
{
@@ -606,33 +613,36 @@ static void cpu_detect_tlb(struct cpuinfo_x86 *c)
tlb_lld_4m[ENTRIES], tlb_lld_1g[ENTRIES]);
}
-void detect_ht(struct cpuinfo_x86 *c)
+int detect_ht_early(struct cpuinfo_x86 *c)
{
#ifdef CONFIG_SMP
u32 eax, ebx, ecx, edx;
- int index_msb, core_bits;
- static bool printed;
if (!cpu_has(c, X86_FEATURE_HT))
- return;
+ return -1;
if (cpu_has(c, X86_FEATURE_CMP_LEGACY))
- goto out;
+ return -1;
if (cpu_has(c, X86_FEATURE_XTOPOLOGY))
- return;
+ return -1;
cpuid(1, &eax, &ebx, &ecx, &edx);
smp_num_siblings = (ebx & 0xff0000) >> 16;
-
- if (smp_num_siblings == 1) {
+ if (smp_num_siblings == 1)
pr_info_once("CPU0: Hyper-Threading is disabled\n");
- goto out;
- }
+#endif
+ return 0;
+}
- if (smp_num_siblings <= 1)
- goto out;
+void detect_ht(struct cpuinfo_x86 *c)
+{
+#ifdef CONFIG_SMP
+ int index_msb, core_bits;
+
+ if (detect_ht_early(c) < 0)
+ return;
index_msb = get_count_order(smp_num_siblings);
c->phys_proc_id = apic->phys_pkg_id(c->initial_apicid, index_msb);
@@ -645,15 +655,6 @@ void detect_ht(struct cpuinfo_x86 *c)
c->cpu_core_id = apic->phys_pkg_id(c->initial_apicid, index_msb) &
((1 << core_bits) - 1);
-
-out:
- if (!printed && (c->x86_max_cores * smp_num_siblings) > 1) {
- pr_info("CPU: Physical Processor ID: %d\n",
- c->phys_proc_id);
- pr_info("CPU: Processor Core ID: %d\n",
- c->cpu_core_id);
- printed = 1;
- }
#endif
}
@@ -725,17 +726,32 @@ static void init_speculation_control(struct cpuinfo_x86 *c)
* and they also have a different bit for STIBP support. Also,
* a hypervisor might have set the individual AMD bits even on
* Intel CPUs, for finer-grained selection of what's available.
- *
- * We use the AMD bits in 0x8000_0008 EBX as the generic hardware
- * features, which are visible in /proc/cpuinfo and used by the
- * kernel. So set those accordingly from the Intel bits.
*/
if (cpu_has(c, X86_FEATURE_SPEC_CTRL)) {
set_cpu_cap(c, X86_FEATURE_IBRS);
set_cpu_cap(c, X86_FEATURE_IBPB);
+ set_cpu_cap(c, X86_FEATURE_MSR_SPEC_CTRL);
}
+
if (cpu_has(c, X86_FEATURE_INTEL_STIBP))
set_cpu_cap(c, X86_FEATURE_STIBP);
+
+ if (cpu_has(c, X86_FEATURE_SPEC_CTRL_SSBD) ||
+ cpu_has(c, X86_FEATURE_VIRT_SSBD))
+ set_cpu_cap(c, X86_FEATURE_SSBD);
+
+ if (cpu_has(c, X86_FEATURE_AMD_IBRS)) {
+ set_cpu_cap(c, X86_FEATURE_IBRS);
+ set_cpu_cap(c, X86_FEATURE_MSR_SPEC_CTRL);
+ }
+
+ if (cpu_has(c, X86_FEATURE_AMD_IBPB))
+ set_cpu_cap(c, X86_FEATURE_IBPB);
+
+ if (cpu_has(c, X86_FEATURE_AMD_STIBP)) {
+ set_cpu_cap(c, X86_FEATURE_STIBP);
+ set_cpu_cap(c, X86_FEATURE_MSR_SPEC_CTRL);
+ }
}
void get_cpu_cap(struct cpuinfo_x86 *c)
@@ -833,6 +849,13 @@ void get_cpu_cap(struct cpuinfo_x86 *c)
init_scattered_cpuid_features(c);
init_speculation_control(c);
+
+ /*
+ * Clear/Set all flags overridden by options, after probe.
+ * This needs to happen each time we re-probe, which may happen
+ * several times during CPU initialization.
+ */
+ apply_forced_caps(c);
}
static void identify_cpu_without_cpuid(struct cpuinfo_x86 *c)
@@ -859,6 +882,7 @@ static void identify_cpu_without_cpuid(struct cpuinfo_x86 *c)
}
}
#endif
+ c->x86_cache_bits = c->x86_phys_bits;
}
static const __initconst struct x86_cpu_id cpu_no_speculation[] = {
@@ -879,21 +903,78 @@ static const __initconst struct x86_cpu_id cpu_no_meltdown[] = {
{}
};
-static bool __init cpu_vulnerable_to_meltdown(struct cpuinfo_x86 *c)
+static const __initconst struct x86_cpu_id cpu_no_spec_store_bypass[] = {
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_PINEVIEW },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_LINCROFT },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_PENWELL },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_CLOVERVIEW },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_CEDARVIEW },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_SILVERMONT1 },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_AIRMONT },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_SILVERMONT2 },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_MERRIFIELD },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_CORE_YONAH },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_XEON_PHI_KNL },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_XEON_PHI_KNM },
+ { X86_VENDOR_CENTAUR, 5, },
+ { X86_VENDOR_INTEL, 5, },
+ { X86_VENDOR_NSC, 5, },
+ { X86_VENDOR_AMD, 0x12, },
+ { X86_VENDOR_AMD, 0x11, },
+ { X86_VENDOR_AMD, 0x10, },
+ { X86_VENDOR_AMD, 0xf, },
+ { X86_VENDOR_ANY, 4, },
+ {}
+};
+
+static const __initconst struct x86_cpu_id cpu_no_l1tf[] = {
+ /* in addition to cpu_no_speculation */
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_SILVERMONT1 },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_SILVERMONT2 },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_AIRMONT },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_MERRIFIELD },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_MOOREFIELD },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_GOLDMONT },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_DENVERTON },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_ATOM_GEMINI_LAKE },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_XEON_PHI_KNL },
+ { X86_VENDOR_INTEL, 6, INTEL_FAM6_XEON_PHI_KNM },
+ {}
+};
+
+static void __init cpu_set_bug_bits(struct cpuinfo_x86 *c)
{
u64 ia32_cap = 0;
- if (x86_match_cpu(cpu_no_meltdown))
- return false;
-
if (cpu_has(c, X86_FEATURE_ARCH_CAPABILITIES))
rdmsrl(MSR_IA32_ARCH_CAPABILITIES, ia32_cap);
+ if (!x86_match_cpu(cpu_no_spec_store_bypass) &&
+ !(ia32_cap & ARCH_CAP_SSB_NO))
+ setup_force_cpu_bug(X86_BUG_SPEC_STORE_BYPASS);
+
+ if (x86_match_cpu(cpu_no_speculation))
+ return;
+
+ setup_force_cpu_bug(X86_BUG_SPECTRE_V1);
+ setup_force_cpu_bug(X86_BUG_SPECTRE_V2);
+
+ if (ia32_cap & ARCH_CAP_IBRS_ALL)
+ setup_force_cpu_cap(X86_FEATURE_IBRS_ENHANCED);
+
+ if (x86_match_cpu(cpu_no_meltdown))
+ return;
+
/* Rogue Data Cache Load? No! */
if (ia32_cap & ARCH_CAP_RDCL_NO)
- return false;
+ return;
- return true;
+ setup_force_cpu_bug(X86_BUG_CPU_MELTDOWN);
+
+ if (x86_match_cpu(cpu_no_l1tf))
+ return;
+
+ setup_force_cpu_bug(X86_BUG_L1TF);
}
/*
@@ -942,12 +1023,7 @@ static void __init early_identify_cpu(struct cpuinfo_x86 *c)
setup_force_cpu_cap(X86_FEATURE_ALWAYS);
- if (!x86_match_cpu(cpu_no_speculation)) {
- if (cpu_vulnerable_to_meltdown(c))
- setup_force_cpu_bug(X86_BUG_CPU_MELTDOWN);
- setup_force_cpu_bug(X86_BUG_SPECTRE_V1);
- setup_force_cpu_bug(X86_BUG_SPECTRE_V2);
- }
+ cpu_set_bug_bits(c);
fpu__init_system(c);
@@ -1315,6 +1391,7 @@ void identify_secondary_cpu(struct cpuinfo_x86 *c)
#endif
mtrr_ap_init();
validate_apic_and_package_id(c);
+ x86_spec_ctrl_setup_ap();
}
struct msr_range {
diff --git a/arch/x86/kernel/cpu/cpu.h b/arch/x86/kernel/cpu/cpu.h
index 2584265..2275900 100644
--- a/arch/x86/kernel/cpu/cpu.h
+++ b/arch/x86/kernel/cpu/cpu.h
@@ -46,4 +46,9 @@ extern const struct cpu_dev *const __x86_cpu_dev_start[],
extern void get_cpu_cap(struct cpuinfo_x86 *c);
extern void cpu_detect_cache_sizes(struct cpuinfo_x86 *c);
+extern int detect_extended_topology_early(struct cpuinfo_x86 *c);
+extern int detect_ht_early(struct cpuinfo_x86 *c);
+
+extern void x86_spec_ctrl_setup_ap(void);
+
#endif /* ARCH_X86_CPU_H */
diff --git a/arch/x86/kernel/cpu/cyrix.c b/arch/x86/kernel/cpu/cyrix.c
index 455d8ada..d39cfb2 100644
--- a/arch/x86/kernel/cpu/cyrix.c
+++ b/arch/x86/kernel/cpu/cyrix.c
@@ -253,6 +253,7 @@ static void init_cyrix(struct cpuinfo_x86 *c)
break;
case 4: /* MediaGX/GXm or Geode GXM/GXLV/GX1 */
+ case 11: /* GX1 with inverted Device ID */
#ifdef CONFIG_PCI
{
u32 vendor, device;
diff --git a/arch/x86/kernel/cpu/intel.c b/arch/x86/kernel/cpu/intel.c
index 8fb1d65..cee0fec 100644
--- a/arch/x86/kernel/cpu/intel.c
+++ b/arch/x86/kernel/cpu/intel.c
@@ -109,6 +109,9 @@ static bool bad_spectre_microcode(struct cpuinfo_x86 *c)
if (cpu_has(c, X86_FEATURE_HYPERVISOR))
return false;
+ if (c->x86 != 6)
+ return false;
+
for (i = 0; i < ARRAY_SIZE(spectre_bad_microcodes); i++) {
if (c->x86_model == spectre_bad_microcodes[i].model &&
c->x86_stepping == spectre_bad_microcodes[i].stepping)
@@ -153,7 +156,10 @@ static void early_init_intel(struct cpuinfo_x86 *c)
setup_clear_cpu_cap(X86_FEATURE_IBPB);
setup_clear_cpu_cap(X86_FEATURE_STIBP);
setup_clear_cpu_cap(X86_FEATURE_SPEC_CTRL);
+ setup_clear_cpu_cap(X86_FEATURE_MSR_SPEC_CTRL);
setup_clear_cpu_cap(X86_FEATURE_INTEL_STIBP);
+ setup_clear_cpu_cap(X86_FEATURE_SSBD);
+ setup_clear_cpu_cap(X86_FEATURE_SPEC_CTRL_SSBD);
}
/*
@@ -280,6 +286,13 @@ static void early_init_intel(struct cpuinfo_x86 *c)
}
check_mpx_erratum(c);
+
+ /*
+ * Get the number of SMT siblings early from the extended topology
+ * leaf, if available. Otherwise try the legacy SMT detection.
+ */
+ if (detect_extended_topology_early(c) < 0)
+ detect_ht_early(c);
}
#ifdef CONFIG_X86_32
diff --git a/arch/x86/kernel/cpu/mcheck/mce-severity.c b/arch/x86/kernel/cpu/mcheck/mce-severity.c
index f46071c..3e0199e 100644
--- a/arch/x86/kernel/cpu/mcheck/mce-severity.c
+++ b/arch/x86/kernel/cpu/mcheck/mce-severity.c
@@ -143,6 +143,11 @@ static struct severity {
SER, MASK(MCI_STATUS_OVER|MCI_UC_SAR|MCI_ADDR|MCACOD, MCI_UC_SAR|MCI_ADDR|MCACOD_INSTR),
USER
),
+ MCESEV(
+ PANIC, "Data load in unrecoverable area of kernel",
+ SER, MASK(MCI_STATUS_OVER|MCI_UC_SAR|MCI_ADDR|MCACOD, MCI_UC_SAR|MCI_ADDR|MCACOD_DATA),
+ KERNEL
+ ),
#endif
MCESEV(
PANIC, "Action required: unknown MCACOD",
diff --git a/arch/x86/kernel/cpu/mcheck/mce.c b/arch/x86/kernel/cpu/mcheck/mce.c
index 7bbd50f..7e6163c 100644
--- a/arch/x86/kernel/cpu/mcheck/mce.c
+++ b/arch/x86/kernel/cpu/mcheck/mce.c
@@ -738,23 +738,25 @@ EXPORT_SYMBOL_GPL(machine_check_poll);
static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp,
struct pt_regs *regs)
{
- int i, ret = 0;
char *tmp;
+ int i;
for (i = 0; i < mca_cfg.banks; i++) {
m->status = mce_rdmsrl(msr_ops.status(i));
- if (m->status & MCI_STATUS_VAL) {
- __set_bit(i, validp);
- if (quirk_no_way_out)
- quirk_no_way_out(i, m, regs);
- }
+ if (!(m->status & MCI_STATUS_VAL))
+ continue;
+
+ __set_bit(i, validp);
+ if (quirk_no_way_out)
+ quirk_no_way_out(i, m, regs);
if (mce_severity(m, mca_cfg.tolerant, &tmp, true) >= MCE_PANIC_SEVERITY) {
+ mce_read_aux(m, i);
*msg = tmp;
- ret = 1;
+ return 1;
}
}
- return ret;
+ return 0;
}
/*
@@ -1140,13 +1142,18 @@ void do_machine_check(struct pt_regs *regs, long error_code)
lmce = m.mcgstatus & MCG_STATUS_LMCES;
/*
+ * Local machine check may already know that we have to panic.
+ * Broadcast machine check begins rendezvous in mce_start()
* Go through all banks in exclusion of the other CPUs. This way we
* don't report duplicated events on shared banks because the first one
- * to see it will clear it. If this is a Local MCE, then no need to
- * perform rendezvous.
+ * to see it will clear it.
*/
- if (!lmce)
+ if (lmce) {
+ if (no_way_out)
+ mce_panic("Fatal local machine check", &m, msg);
+ } else {
order = mce_start(&no_way_out);
+ }
for (i = 0; i < cfg->banks; i++) {
__clear_bit(i, toclear);
@@ -1222,12 +1229,17 @@ void do_machine_check(struct pt_regs *regs, long error_code)
no_way_out = worst >= MCE_PANIC_SEVERITY;
} else {
/*
- * Local MCE skipped calling mce_reign()
- * If we found a fatal error, we need to panic here.
+ * If there was a fatal machine check we should have
+ * already called mce_panic earlier in this function.
+ * Since we re-read the banks, we might have found
+ * something new. Check again to see if we found a
+ * fatal error. We call "mce_severity()" again to
+ * make sure we have the right "msg".
*/
- if (worst >= MCE_PANIC_SEVERITY && mca_cfg.tolerant < 3)
- mce_panic("Machine check from unknown source",
- NULL, NULL);
+ if (worst >= MCE_PANIC_SEVERITY && mca_cfg.tolerant < 3) {
+ mce_severity(&m, cfg->tolerant, &msg, true);
+ mce_panic("Local fatal machine check!", &m, msg);
+ }
}
/*
@@ -2385,9 +2397,6 @@ static ssize_t store_int_with_restart(struct device *s,
if (check_interval == old_check_interval)
return ret;
- if (check_interval < 1)
- check_interval = 1;
-
mutex_lock(&mce_sysfs_mutex);
mce_restart();
mutex_unlock(&mce_sysfs_mutex);
diff --git a/arch/x86/kernel/cpu/microcode/core.c b/arch/x86/kernel/cpu/microcode/core.c
index 0afaf00..b53a657 100644
--- a/arch/x86/kernel/cpu/microcode/core.c
+++ b/arch/x86/kernel/cpu/microcode/core.c
@@ -384,6 +384,24 @@ static void __exit microcode_dev_exit(void)
/* fake device for request_firmware */
static struct platform_device *microcode_pdev;
+static int check_online_cpus(void)
+{
+ unsigned int cpu;
+
+ /*
+ * Make sure all CPUs are online. It's fine for SMT to be disabled if
+ * all the primary threads are still online.
+ */
+ for_each_present_cpu(cpu) {
+ if (topology_is_primary_thread(cpu) && !cpu_online(cpu)) {
+ pr_err("Not all CPUs online, aborting microcode update.\n");
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
static int reload_for_cpu(int cpu)
{
struct ucode_cpu_info *uci = ucode_cpu_info + cpu;
@@ -418,7 +436,13 @@ static ssize_t reload_store(struct device *dev,
return size;
get_online_cpus();
+
+ ret = check_online_cpus();
+ if (ret)
+ goto put;
+
mutex_lock(µcode_mutex);
+
for_each_online_cpu(cpu) {
tmp_ret = reload_for_cpu(cpu);
if (tmp_ret != 0)
@@ -431,6 +455,8 @@ static ssize_t reload_store(struct device *dev,
if (!ret)
perf_check_microcode();
mutex_unlock(µcode_mutex);
+
+put:
put_online_cpus();
if (!ret)
diff --git a/arch/x86/kernel/cpu/microcode/intel.c b/arch/x86/kernel/cpu/microcode/intel.c
index 4bcd30c..79291d6 100644
--- a/arch/x86/kernel/cpu/microcode/intel.c
+++ b/arch/x86/kernel/cpu/microcode/intel.c
@@ -474,7 +474,6 @@ static void show_saved_mc(void)
*/
static void save_mc_for_early(u8 *mc)
{
-#ifdef CONFIG_HOTPLUG_CPU
/* Synchronization during CPU hotplug. */
static DEFINE_MUTEX(x86_cpu_microcode_mutex);
@@ -521,7 +520,6 @@ static void save_mc_for_early(u8 *mc)
out:
mutex_unlock(&x86_cpu_microcode_mutex);
-#endif
}
static bool __init load_builtin_intel_microcode(struct cpio_data *cp)
diff --git a/arch/x86/kernel/cpu/mtrr/if.c b/arch/x86/kernel/cpu/mtrr/if.c
index 6d9b455..d5b2a08 100644
--- a/arch/x86/kernel/cpu/mtrr/if.c
+++ b/arch/x86/kernel/cpu/mtrr/if.c
@@ -172,6 +172,8 @@ mtrr_ioctl(struct file *file, unsigned int cmd, unsigned long __arg)
struct mtrr_gentry gentry;
void __user *arg = (void __user *) __arg;
+ memset(&gentry, 0, sizeof(gentry));
+
switch (cmd) {
case MTRRIOC_ADD_ENTRY:
case MTRRIOC_SET_ENTRY:
diff --git a/arch/x86/kernel/cpu/topology.c b/arch/x86/kernel/cpu/topology.c
index cd53135..6b5a850 100644
--- a/arch/x86/kernel/cpu/topology.c
+++ b/arch/x86/kernel/cpu/topology.c
@@ -26,16 +26,13 @@
* exists, use it for populating initial_apicid and cpu topology
* detection.
*/
-void detect_extended_topology(struct cpuinfo_x86 *c)
+int detect_extended_topology_early(struct cpuinfo_x86 *c)
{
#ifdef CONFIG_SMP
- unsigned int eax, ebx, ecx, edx, sub_index;
- unsigned int ht_mask_width, core_plus_mask_width;
- unsigned int core_select_mask, core_level_siblings;
- static bool printed;
+ unsigned int eax, ebx, ecx, edx;
if (c->cpuid_level < 0xb)
- return;
+ return -1;
cpuid_count(0xb, SMT_LEVEL, &eax, &ebx, &ecx, &edx);
@@ -43,7 +40,7 @@ void detect_extended_topology(struct cpuinfo_x86 *c)
* check if the cpuid leaf 0xb is actually implemented.
*/
if (ebx == 0 || (LEAFB_SUBTYPE(ecx) != SMT_TYPE))
- return;
+ return -1;
set_cpu_cap(c, X86_FEATURE_XTOPOLOGY);
@@ -51,10 +48,30 @@ void detect_extended_topology(struct cpuinfo_x86 *c)
* initial apic id, which also represents 32-bit extended x2apic id.
*/
c->initial_apicid = edx;
+ smp_num_siblings = LEVEL_MAX_SIBLINGS(ebx);
+#endif
+ return 0;
+}
+
+/*
+ * Check for extended topology enumeration cpuid leaf 0xb and if it
+ * exists, use it for populating initial_apicid and cpu topology
+ * detection.
+ */
+void detect_extended_topology(struct cpuinfo_x86 *c)
+{
+#ifdef CONFIG_SMP
+ unsigned int eax, ebx, ecx, edx, sub_index;
+ unsigned int ht_mask_width, core_plus_mask_width;
+ unsigned int core_select_mask, core_level_siblings;
+
+ if (detect_extended_topology_early(c) < 0)
+ return;
/*
* Populate HT related information from sub-leaf level 0.
*/
+ cpuid_count(0xb, SMT_LEVEL, &eax, &ebx, &ecx, &edx);
core_level_siblings = smp_num_siblings = LEVEL_MAX_SIBLINGS(ebx);
core_plus_mask_width = ht_mask_width = BITS_SHIFT_NEXT_LEVEL(eax);
@@ -85,15 +102,5 @@ void detect_extended_topology(struct cpuinfo_x86 *c)
c->apicid = apic->phys_pkg_id(c->initial_apicid, 0);
c->x86_max_cores = (core_level_siblings / smp_num_siblings);
-
- if (!printed) {
- pr_info("CPU: Physical Processor ID: %d\n",
- c->phys_proc_id);
- if (c->x86_max_cores > 1)
- pr_info("CPU: Processor Core ID: %d\n",
- c->cpu_core_id);
- printed = 1;
- }
- return;
#endif
}
diff --git a/arch/x86/kernel/devicetree.c b/arch/x86/kernel/devicetree.c
index 3fe45f8..7a07b15 100644
--- a/arch/x86/kernel/devicetree.c
+++ b/arch/x86/kernel/devicetree.c
@@ -11,6 +11,7 @@
#include <linux/of_address.h>
#include <linux/of_platform.h>
#include <linux/of_irq.h>
+#include <linux/libfdt.h>
#include <linux/slab.h>
#include <linux/pci.h>
#include <linux/of_pci.h>
@@ -199,19 +200,22 @@ static struct of_ioapic_type of_ioapic_type[] =
static int dt_irqdomain_alloc(struct irq_domain *domain, unsigned int virq,
unsigned int nr_irqs, void *arg)
{
- struct of_phandle_args *irq_data = (void *)arg;
+ struct irq_fwspec *fwspec = (struct irq_fwspec *)arg;
struct of_ioapic_type *it;
struct irq_alloc_info tmp;
+ int type_index;
- if (WARN_ON(irq_data->args_count < 2))
- return -EINVAL;
- if (irq_data->args[1] >= ARRAY_SIZE(of_ioapic_type))
+ if (WARN_ON(fwspec->param_count < 2))
return -EINVAL;
- it = &of_ioapic_type[irq_data->args[1]];
+ type_index = fwspec->param[1];
+ if (type_index >= ARRAY_SIZE(of_ioapic_type))
+ return -EINVAL;
+
+ it = &of_ioapic_type[type_index];
ioapic_set_alloc_attr(&tmp, NUMA_NO_NODE, it->trigger, it->polarity);
tmp.ioapic_id = mpc_ioapic_id(mp_irqdomain_ioapic_idx(domain));
- tmp.ioapic_pin = irq_data->args[0];
+ tmp.ioapic_pin = fwspec->param[0];
return mp_irqdomain_alloc(domain, virq, nr_irqs, &tmp);
}
@@ -276,14 +280,15 @@ static void __init x86_flattree_get_config(void)
map_len = max(PAGE_SIZE - (initial_dtb & ~PAGE_MASK), (u64)128);
- initial_boot_params = dt = early_memremap(initial_dtb, map_len);
- size = of_get_flat_dt_size();
+ dt = early_memremap(initial_dtb, map_len);
+ size = fdt_totalsize(dt);
if (map_len < size) {
early_memunmap(dt, map_len);
- initial_boot_params = dt = early_memremap(initial_dtb, size);
+ dt = early_memremap(initial_dtb, size);
map_len = size;
}
+ early_init_dt_verify(dt);
unflatten_and_copy_device_tree();
early_memunmap(dt, map_len);
}
diff --git a/arch/x86/kernel/dumpstack.c b/arch/x86/kernel/dumpstack.c
index 85f854b..3576ece 100644
--- a/arch/x86/kernel/dumpstack.c
+++ b/arch/x86/kernel/dumpstack.c
@@ -15,6 +15,7 @@
#include <linux/bug.h>
#include <linux/nmi.h>
#include <linux/sysfs.h>
+#include <linux/kasan.h>
#include <asm/stacktrace.h>
#include <asm/unwind.h>
@@ -229,7 +230,10 @@ void oops_end(unsigned long flags, struct pt_regs *regs, int signr)
* We're not going to return, but we might be on an IST stack or
* have very little stack space left. Rewind the stack and kill
* the task.
+ * Before we rewind the stack, we have to tell KASAN that we're going to
+ * reuse the task stack and that existing poisons are invalid.
*/
+ kasan_unpoison_task_stack(current);
rewind_stack_do_exit(signr);
}
NOKPROBE_SYMBOL(oops_end);
diff --git a/arch/x86/kernel/fpu/core.c b/arch/x86/kernel/fpu/core.c
index 96d80df..fc96511 100644
--- a/arch/x86/kernel/fpu/core.c
+++ b/arch/x86/kernel/fpu/core.c
@@ -10,6 +10,7 @@
#include <asm/fpu/signal.h>
#include <asm/fpu/types.h>
#include <asm/traps.h>
+#include <asm/irq_regs.h>
#include <linux/hardirq.h>
#include <linux/pkeys.h>
@@ -58,27 +59,9 @@ static bool kernel_fpu_disabled(void)
return this_cpu_read(in_kernel_fpu);
}
-/*
- * Were we in an interrupt that interrupted kernel mode?
- *
- * On others, we can do a kernel_fpu_begin/end() pair *ONLY* if that
- * pair does nothing at all: the thread must not have fpu (so
- * that we don't try to save the FPU state), and TS must
- * be set (so that the clts/stts pair does nothing that is
- * visible in the interrupted kernel thread).
- *
- * Except for the eagerfpu case when we return true; in the likely case
- * the thread has FPU but we are not going to set/clear TS.
- */
static bool interrupted_kernel_fpu_idle(void)
{
- if (kernel_fpu_disabled())
- return false;
-
- if (use_eager_fpu())
- return true;
-
- return !current->thread.fpu.fpregs_active && (read_cr0() & X86_CR0_TS);
+ return !kernel_fpu_disabled();
}
/*
@@ -126,7 +109,6 @@ void __kernel_fpu_begin(void)
copy_fpregs_to_fpstate(fpu);
} else {
this_cpu_write(fpu_fpregs_owner_ctx, NULL);
- __fpregs_activate_hw();
}
}
EXPORT_SYMBOL(__kernel_fpu_begin);
@@ -137,8 +119,6 @@ void __kernel_fpu_end(void)
if (fpu->fpregs_active)
copy_kernel_to_fpregs(&fpu->state);
- else
- __fpregs_deactivate_hw();
kernel_fpu_enable();
}
@@ -200,10 +180,7 @@ void fpu__save(struct fpu *fpu)
trace_x86_fpu_before_save(fpu);
if (fpu->fpregs_active) {
if (!copy_fpregs_to_fpstate(fpu)) {
- if (use_eager_fpu())
- copy_kernel_to_fpregs(&fpu->state);
- else
- fpregs_deactivate(fpu);
+ copy_kernel_to_fpregs(&fpu->state);
}
}
trace_x86_fpu_after_save(fpu);
@@ -248,7 +225,6 @@ EXPORT_SYMBOL_GPL(fpstate_init);
int fpu__copy(struct fpu *dst_fpu, struct fpu *src_fpu)
{
- dst_fpu->counter = 0;
dst_fpu->fpregs_active = 0;
dst_fpu->last_cpu = -1;
@@ -261,8 +237,7 @@ int fpu__copy(struct fpu *dst_fpu, struct fpu *src_fpu)
* Don't let 'init optimized' areas of the XSAVE area
* leak into the child task:
*/
- if (use_eager_fpu())
- memset(&dst_fpu->state.xsave, 0, fpu_kernel_xstate_size);
+ memset(&dst_fpu->state.xsave, 0, fpu_kernel_xstate_size);
/*
* Save current FPU registers directly into the child
@@ -284,10 +259,7 @@ int fpu__copy(struct fpu *dst_fpu, struct fpu *src_fpu)
memcpy(&src_fpu->state, &dst_fpu->state,
fpu_kernel_xstate_size);
- if (use_eager_fpu())
- copy_kernel_to_fpregs(&src_fpu->state);
- else
- fpregs_deactivate(src_fpu);
+ copy_kernel_to_fpregs(&src_fpu->state);
}
preempt_enable();
@@ -460,7 +432,6 @@ void fpu__restore(struct fpu *fpu)
trace_x86_fpu_before_restore(fpu);
fpregs_activate(fpu);
copy_kernel_to_fpregs(&fpu->state);
- fpu->counter++;
trace_x86_fpu_after_restore(fpu);
kernel_fpu_enable();
}
@@ -478,7 +449,6 @@ EXPORT_SYMBOL_GPL(fpu__restore);
void fpu__drop(struct fpu *fpu)
{
preempt_disable();
- fpu->counter = 0;
if (fpu->fpregs_active) {
/* Ignore delayed exceptions from user space */
diff --git a/arch/x86/kernel/fpu/init.c b/arch/x86/kernel/fpu/init.c
index 6f0ab305..9f36578 100644
--- a/arch/x86/kernel/fpu/init.c
+++ b/arch/x86/kernel/fpu/init.c
@@ -15,10 +15,7 @@
*/
static void fpu__init_cpu_ctx_switch(void)
{
- if (!boot_cpu_has(X86_FEATURE_EAGER_FPU))
- stts();
- else
- clts();
+ clts();
}
/*
@@ -234,82 +231,16 @@ static void __init fpu__init_system_xstate_size_legacy(void)
}
/*
- * FPU context switching strategies:
- *
- * Against popular belief, we don't do lazy FPU saves, due to the
- * task migration complications it brings on SMP - we only do
- * lazy FPU restores.
- *
- * 'lazy' is the traditional strategy, which is based on setting
- * CR0::TS to 1 during context-switch (instead of doing a full
- * restore of the FPU state), which causes the first FPU instruction
- * after the context switch (whenever it is executed) to fault - at
- * which point we lazily restore the FPU state into FPU registers.
- *
- * Tasks are of course under no obligation to execute FPU instructions,
- * so it can easily happen that another context-switch occurs without
- * a single FPU instruction being executed. If we eventually switch
- * back to the original task (that still owns the FPU) then we have
- * not only saved the restores along the way, but we also have the
- * FPU ready to be used for the original task.
- *
- * 'lazy' is deprecated because it's almost never a performance win
- * and it's much more complicated than 'eager'.
- *
- * 'eager' switching is by default on all CPUs, there we switch the FPU
- * state during every context switch, regardless of whether the task
- * has used FPU instructions in that time slice or not. This is done
- * because modern FPU context saving instructions are able to optimize
- * state saving and restoration in hardware: they can detect both
- * unused and untouched FPU state and optimize accordingly.
- *
- * [ Note that even in 'lazy' mode we might optimize context switches
- * to use 'eager' restores, if we detect that a task is using the FPU
- * frequently. See the fpu->counter logic in fpu/internal.h for that. ]
- */
-static enum { ENABLE, DISABLE } eagerfpu = ENABLE;
-
-/*
* Find supported xfeatures based on cpu features and command-line input.
* This must be called after fpu__init_parse_early_param() is called and
* xfeatures_mask is enumerated.
*/
u64 __init fpu__get_supported_xfeatures_mask(void)
{
- /* Support all xfeatures known to us */
- if (eagerfpu != DISABLE)
- return XCNTXT_MASK;
-
- /* Warning of xfeatures being disabled for no eagerfpu mode */
- if (xfeatures_mask & XFEATURE_MASK_EAGER) {
- pr_err("x86/fpu: eagerfpu switching disabled, disabling the following xstate features: 0x%llx.\n",
- xfeatures_mask & XFEATURE_MASK_EAGER);
- }
-
- /* Return a mask that masks out all features requiring eagerfpu mode */
- return ~XFEATURE_MASK_EAGER;
+ return XCNTXT_MASK;
}
-/*
- * Disable features dependent on eagerfpu.
- */
-static void __init fpu__clear_eager_fpu_features(void)
-{
- setup_clear_cpu_cap(X86_FEATURE_MPX);
-}
-
-/*
- * Pick the FPU context switching strategy:
- *
- * When eagerfpu is AUTO or ENABLE, we ensure it is ENABLE if either of
- * the following is true:
- *
- * (1) the cpu has xsaveopt, as it has the optimization and doing eager
- * FPU switching has a relatively low cost compared to a plain xsave;
- * (2) the cpu has xsave features (e.g. MPX) that depend on eager FPU
- * switching. Should the kernel boot with noxsaveopt, we support MPX
- * with eager FPU switching at a higher cost.
- */
+/* Legacy code to initialize eager fpu mode. */
static void __init fpu__init_system_ctx_switch(void)
{
static bool on_boot_cpu __initdata = 1;
@@ -318,17 +249,6 @@ static void __init fpu__init_system_ctx_switch(void)
on_boot_cpu = 0;
WARN_ON_FPU(current->thread.fpu.fpstate_active);
-
- if (boot_cpu_has(X86_FEATURE_XSAVEOPT) && eagerfpu != DISABLE)
- eagerfpu = ENABLE;
-
- if (xfeatures_mask & XFEATURE_MASK_EAGER)
- eagerfpu = ENABLE;
-
- if (eagerfpu == ENABLE)
- setup_force_cpu_cap(X86_FEATURE_EAGER_FPU);
-
- printk(KERN_INFO "x86/fpu: Using '%s' FPU context switches.\n", eagerfpu == ENABLE ? "eager" : "lazy");
}
/*
@@ -337,11 +257,6 @@ static void __init fpu__init_system_ctx_switch(void)
*/
static void __init fpu__init_parse_early_param(void)
{
- if (cmdline_find_option_bool(boot_command_line, "eagerfpu=off")) {
- eagerfpu = DISABLE;
- fpu__clear_eager_fpu_features();
- }
-
if (cmdline_find_option_bool(boot_command_line, "no387"))
setup_clear_cpu_cap(X86_FEATURE_FPU);
diff --git a/arch/x86/kernel/fpu/signal.c b/arch/x86/kernel/fpu/signal.c
index 3ec0d2d..769831d 100644
--- a/arch/x86/kernel/fpu/signal.c
+++ b/arch/x86/kernel/fpu/signal.c
@@ -309,7 +309,6 @@ static int __fpu__restore_sig(void __user *buf, void __user *buf_fx, int size)
* thread's fpu state, reconstruct fxstate from the fsave
* header. Sanitize the copied state etc.
*/
- struct fpu *fpu = &tsk->thread.fpu;
struct user_i387_ia32_struct env;
int err = 0;
@@ -343,12 +342,10 @@ static int __fpu__restore_sig(void __user *buf, void __user *buf_fx, int size)
sanitize_restored_xstate(tsk, &env, xfeatures, fx_only);
}
+ local_bh_disable();
fpu->fpstate_active = 1;
- if (use_eager_fpu()) {
- preempt_disable();
- fpu__restore(fpu);
- preempt_enable();
- }
+ fpu__restore(fpu);
+ local_bh_enable();
return err;
} else {
diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c
index abfbb61b..e9d7f46 100644
--- a/arch/x86/kernel/fpu/xstate.c
+++ b/arch/x86/kernel/fpu/xstate.c
@@ -890,15 +890,6 @@ int arch_set_user_pkey_access(struct task_struct *tsk, int pkey,
*/
if (!boot_cpu_has(X86_FEATURE_OSPKE))
return -EINVAL;
- /*
- * For most XSAVE components, this would be an arduous task:
- * brining fpstate up to date with fpregs, updating fpstate,
- * then re-populating fpregs. But, for components that are
- * never lazily managed, we can just access the fpregs
- * directly. PKRU is never managed lazily, so we can just
- * manipulate it directly. Make sure it stays that way.
- */
- WARN_ON_ONCE(!use_eager_fpu());
/* Set the bits we need in PKRU: */
if (init_val & PKEY_DISABLE_ACCESS)
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 6bf09f5..5e06ffe 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -26,6 +26,7 @@
#include <asm/cacheflush.h>
#include <asm/kprobes.h>
+#include <asm/sections.h>
#include <asm/ftrace.h>
#include <asm/nops.h>
diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c
index 9512529..756634f 100644
--- a/arch/x86/kernel/hpet.c
+++ b/arch/x86/kernel/hpet.c
@@ -1,6 +1,7 @@
#include <linux/clocksource.h>
#include <linux/clockchips.h>
#include <linux/interrupt.h>
+#include <linux/irq.h>
#include <linux/export.h>
#include <linux/delay.h>
#include <linux/errno.h>
diff --git a/arch/x86/kernel/i8259.c b/arch/x86/kernel/i8259.c
index 4e3b8a5..26d5451 100644
--- a/arch/x86/kernel/i8259.c
+++ b/arch/x86/kernel/i8259.c
@@ -4,6 +4,7 @@
#include <linux/sched.h>
#include <linux/ioport.h>
#include <linux/interrupt.h>
+#include <linux/irq.h>
#include <linux/timex.h>
#include <linux/random.h>
#include <linux/init.h>
diff --git a/arch/x86/kernel/irq.c b/arch/x86/kernel/irq.c
index 8a7ad9f..c6f0ef1 100644
--- a/arch/x86/kernel/irq.c
+++ b/arch/x86/kernel/irq.c
@@ -10,6 +10,7 @@
#include <linux/ftrace.h>
#include <linux/delay.h>
#include <linux/export.h>
+#include <linux/irq.h>
#include <asm/apic.h>
#include <asm/io_apic.h>
diff --git a/arch/x86/kernel/irq_32.c b/arch/x86/kernel/irq_32.c
index 2763573..5aaa39a 100644
--- a/arch/x86/kernel/irq_32.c
+++ b/arch/x86/kernel/irq_32.c
@@ -10,6 +10,7 @@
#include <linux/seq_file.h>
#include <linux/interrupt.h>
+#include <linux/irq.h>
#include <linux/kernel_stat.h>
#include <linux/notifier.h>
#include <linux/cpu.h>
diff --git a/arch/x86/kernel/irq_64.c b/arch/x86/kernel/irq_64.c
index 9ebd0b0..bcd1b82 100644
--- a/arch/x86/kernel/irq_64.c
+++ b/arch/x86/kernel/irq_64.c
@@ -10,6 +10,7 @@
#include <linux/kernel_stat.h>
#include <linux/interrupt.h>
+#include <linux/irq.h>
#include <linux/seq_file.h>
#include <linux/delay.h>
#include <linux/ftrace.h>
diff --git a/arch/x86/kernel/irqflags.S b/arch/x86/kernel/irqflags.S
new file mode 100644
index 0000000..ddeeaac
--- /dev/null
+++ b/arch/x86/kernel/irqflags.S
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#include <asm/asm.h>
+#include <asm/export.h>
+#include <linux/linkage.h>
+
+/*
+ * unsigned long native_save_fl(void)
+ */
+ENTRY(native_save_fl)
+ pushf
+ pop %_ASM_AX
+ ret
+ENDPROC(native_save_fl)
+EXPORT_SYMBOL(native_save_fl)
+
+/*
+ * void native_restore_fl(unsigned long flags)
+ * %eax/%rdi: flags
+ */
+ENTRY(native_restore_fl)
+ push %_ASM_ARG1
+ popf
+ ret
+ENDPROC(native_restore_fl)
+EXPORT_SYMBOL(native_restore_fl)
diff --git a/arch/x86/kernel/irqinit.c b/arch/x86/kernel/irqinit.c
index f480b38..eeb77e5 100644
--- a/arch/x86/kernel/irqinit.c
+++ b/arch/x86/kernel/irqinit.c
@@ -4,6 +4,7 @@
#include <linux/sched.h>
#include <linux/ioport.h>
#include <linux/interrupt.h>
+#include <linux/irq.h>
#include <linux/timex.h>
#include <linux/random.h>
#include <linux/kprobes.h>
diff --git a/arch/x86/kernel/kexec-bzimage64.c b/arch/x86/kernel/kexec-bzimage64.c
index 3407b14..490f9be 100644
--- a/arch/x86/kernel/kexec-bzimage64.c
+++ b/arch/x86/kernel/kexec-bzimage64.c
@@ -529,7 +529,7 @@ static int bzImage64_cleanup(void *loader_data)
static int bzImage64_verify_sig(const char *kernel, unsigned long kernel_len)
{
return verify_pefile_signature(kernel, kernel_len,
- NULL,
+ VERIFY_USE_SECONDARY_KEYRING,
VERIFYING_KEXEC_PE_SIGNATURE);
}
#endif
diff --git a/arch/x86/kernel/kprobes/core.c b/arch/x86/kernel/kprobes/core.c
index 91c48cd..64a70b2 100644
--- a/arch/x86/kernel/kprobes/core.c
+++ b/arch/x86/kernel/kprobes/core.c
@@ -61,6 +61,7 @@
#include <asm/alternative.h>
#include <asm/insn.h>
#include <asm/debugreg.h>
+#include <asm/sections.h>
#include "common.h"
@@ -396,7 +397,6 @@ int __copy_instruction(u8 *dest, u8 *src)
newdisp = (u8 *) src + (s64) insn.displacement.value - (u8 *) dest;
if ((s64) (s32) newdisp != newdisp) {
pr_err("Kprobes error: new displacement does not fit into s32 (%llx)\n", newdisp);
- pr_err("\tSrc: %p, Dest: %p, old disp: %x\n", src, dest, insn.displacement.value);
return 0;
}
disp = (u8 *) dest + insn_offset_displacement(&insn);
@@ -414,25 +414,38 @@ void free_insn_page(void *page)
module_memfree(page);
}
+/* Prepare reljump right after instruction to boost */
+static void prepare_boost(struct kprobe *p, int length)
+{
+ if (can_boost(p->ainsn.insn, p->addr) &&
+ MAX_INSN_SIZE - length >= RELATIVEJUMP_SIZE) {
+ /*
+ * These instructions can be executed directly if it
+ * jumps back to correct address.
+ */
+ synthesize_reljump(p->ainsn.insn + length, p->addr + length);
+ p->ainsn.boostable = 1;
+ } else {
+ p->ainsn.boostable = -1;
+ }
+}
+
static int arch_copy_kprobe(struct kprobe *p)
{
- int ret;
+ int len;
set_memory_rw((unsigned long)p->ainsn.insn & PAGE_MASK, 1);
/* Copy an instruction with recovering if other optprobe modifies it.*/
- ret = __copy_instruction(p->ainsn.insn, p->addr);
- if (!ret)
+ len = __copy_instruction(p->ainsn.insn, p->addr);
+ if (!len)
return -EINVAL;
/*
* __copy_instruction can modify the displacement of the instruction,
* but it doesn't affect boostable check.
*/
- if (can_boost(p->ainsn.insn, p->addr))
- p->ainsn.boostable = 0;
- else
- p->ainsn.boostable = -1;
+ prepare_boost(p, len);
set_memory_ro((unsigned long)p->ainsn.insn & PAGE_MASK, 1);
@@ -599,8 +612,7 @@ static int reenter_kprobe(struct kprobe *p, struct pt_regs *regs,
* Raise a BUG or we'll continue in an endless reentering loop
* and eventually a stack overflow.
*/
- printk(KERN_WARNING "Unrecoverable kprobe detected at %p.\n",
- p->addr);
+ pr_err("Unrecoverable kprobe detected.\n");
dump_kprobe(p);
BUG();
default:
@@ -897,21 +909,6 @@ static void resume_execution(struct kprobe *p, struct pt_regs *regs,
break;
}
- if (p->ainsn.boostable == 0) {
- if ((regs->ip > copy_ip) &&
- (regs->ip - copy_ip) + 5 < MAX_INSN_SIZE) {
- /*
- * These instructions can be executed directly if it
- * jumps back to correct address.
- */
- synthesize_reljump((void *)regs->ip,
- (void *)orig_ip + (regs->ip - copy_ip));
- p->ainsn.boostable = 1;
- } else {
- p->ainsn.boostable = -1;
- }
- }
-
regs->ip += orig_ip - copy_ip;
no_change:
diff --git a/arch/x86/kernel/kprobes/opt.c b/arch/x86/kernel/kprobes/opt.c
index fa671b9..1009d63 100644
--- a/arch/x86/kernel/kprobes/opt.c
+++ b/arch/x86/kernel/kprobes/opt.c
@@ -28,6 +28,7 @@
#include <linux/kdebug.h>
#include <linux/kallsyms.h>
#include <linux/ftrace.h>
+#include <linux/frame.h>
#include <asm/text-patching.h>
#include <asm/cacheflush.h>
@@ -38,6 +39,7 @@
#include <asm/insn.h>
#include <asm/debugreg.h>
#include <asm/nospec-branch.h>
+#include <asm/sections.h>
#include "common.h"
@@ -91,6 +93,7 @@ static void synthesize_set_arg1(kprobe_opcode_t *addr, unsigned long val)
}
asm (
+ "optprobe_template_func:\n"
".global optprobe_template_entry\n"
"optprobe_template_entry:\n"
#ifdef CONFIG_X86_64
@@ -128,7 +131,12 @@ asm (
" popf\n"
#endif
".global optprobe_template_end\n"
- "optprobe_template_end:\n");
+ "optprobe_template_end:\n"
+ ".type optprobe_template_func, @function\n"
+ ".size optprobe_template_func, .-optprobe_template_func\n");
+
+void optprobe_template_func(void);
+STACK_FRAME_NON_STANDARD(optprobe_template_func);
#define TMPL_MOVE_IDX \
((long)&optprobe_template_val - (long)&optprobe_template_entry)
diff --git a/arch/x86/kernel/machine_kexec_32.c b/arch/x86/kernel/machine_kexec_32.c
index 469b23d..fd7e993 100644
--- a/arch/x86/kernel/machine_kexec_32.c
+++ b/arch/x86/kernel/machine_kexec_32.c
@@ -71,12 +71,17 @@ static void load_segments(void)
static void machine_kexec_free_page_tables(struct kimage *image)
{
free_page((unsigned long)image->arch.pgd);
+ image->arch.pgd = NULL;
#ifdef CONFIG_X86_PAE
free_page((unsigned long)image->arch.pmd0);
+ image->arch.pmd0 = NULL;
free_page((unsigned long)image->arch.pmd1);
+ image->arch.pmd1 = NULL;
#endif
free_page((unsigned long)image->arch.pte0);
+ image->arch.pte0 = NULL;
free_page((unsigned long)image->arch.pte1);
+ image->arch.pte1 = NULL;
}
static int machine_kexec_alloc_page_tables(struct kimage *image)
@@ -93,7 +98,6 @@ static int machine_kexec_alloc_page_tables(struct kimage *image)
!image->arch.pmd0 || !image->arch.pmd1 ||
#endif
!image->arch.pte0 || !image->arch.pte1) {
- machine_kexec_free_page_tables(image);
return -ENOMEM;
}
return 0;
diff --git a/arch/x86/kernel/machine_kexec_64.c b/arch/x86/kernel/machine_kexec_64.c
index a5784a1..eae59ca 100644
--- a/arch/x86/kernel/machine_kexec_64.c
+++ b/arch/x86/kernel/machine_kexec_64.c
@@ -37,8 +37,11 @@ static struct kexec_file_ops *kexec_file_loaders[] = {
static void free_transition_pgtable(struct kimage *image)
{
free_page((unsigned long)image->arch.pud);
+ image->arch.pud = NULL;
free_page((unsigned long)image->arch.pmd);
+ image->arch.pmd = NULL;
free_page((unsigned long)image->arch.pte);
+ image->arch.pte = NULL;
}
static int init_transition_pgtable(struct kimage *image, pgd_t *pgd)
@@ -79,7 +82,6 @@ static int init_transition_pgtable(struct kimage *image, pgd_t *pgd)
set_pte(pte, pfn_pte(paddr >> PAGE_SHIFT, PAGE_KERNEL_EXEC));
return 0;
err:
- free_transition_pgtable(image);
return result;
}
diff --git a/arch/x86/kernel/paravirt.c b/arch/x86/kernel/paravirt.c
index bbf3d59..bf9552b 100644
--- a/arch/x86/kernel/paravirt.c
+++ b/arch/x86/kernel/paravirt.c
@@ -88,10 +88,12 @@ unsigned paravirt_patch_call(void *insnbuf,
struct branch *b = insnbuf;
unsigned long delta = (unsigned long)target - (addr+5);
- if (tgt_clobbers & ~site_clobbers)
- return len; /* target would clobber too much for this site */
- if (len < 5)
+ if (len < 5) {
+#ifdef CONFIG_RETPOLINE
+ WARN_ONCE(1, "Failing to patch indirect CALL in %ps\n", (void *)addr);
+#endif
return len; /* call too long for patch site */
+ }
b->opcode = 0xe8; /* call */
b->delta = delta;
@@ -106,8 +108,12 @@ unsigned paravirt_patch_jmp(void *insnbuf, const void *target,
struct branch *b = insnbuf;
unsigned long delta = (unsigned long)target - (addr+5);
- if (len < 5)
+ if (len < 5) {
+#ifdef CONFIG_RETPOLINE
+ WARN_ONCE(1, "Failing to patch indirect JMP in %ps\n", (void *)addr);
+#endif
return len; /* call too long for patch site */
+ }
b->opcode = 0xe9; /* jmp */
b->delta = delta;
diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c
index 54b2711..e9195a1 100644
--- a/arch/x86/kernel/process.c
+++ b/arch/x86/kernel/process.c
@@ -33,6 +33,7 @@
#include <asm/mce.h>
#include <asm/vm86.h>
#include <asm/switch_to.h>
+#include <asm/spec-ctrl.h>
/*
* per-CPU TSS segments. Threads are completely 'soft' on Linux,
@@ -121,11 +122,6 @@ void flush_thread(void)
fpu__clear(&tsk->thread.fpu);
}
-static void hard_disable_TSC(void)
-{
- cr4_set_bits(X86_CR4_TSD);
-}
-
void disable_TSC(void)
{
preempt_disable();
@@ -134,15 +130,10 @@ void disable_TSC(void)
* Must flip the CPU state synchronously with
* TIF_NOTSC in the current running context.
*/
- hard_disable_TSC();
+ cr4_set_bits(X86_CR4_TSD);
preempt_enable();
}
-static void hard_enable_TSC(void)
-{
- cr4_clear_bits(X86_CR4_TSD);
-}
-
static void enable_TSC(void)
{
preempt_disable();
@@ -151,7 +142,7 @@ static void enable_TSC(void)
* Must flip the CPU state synchronously with
* TIF_NOTSC in the current running context.
*/
- hard_enable_TSC();
+ cr4_clear_bits(X86_CR4_TSD);
preempt_enable();
}
@@ -179,48 +170,199 @@ int set_tsc_mode(unsigned int val)
return 0;
}
-void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p,
- struct tss_struct *tss)
+static inline void switch_to_bitmap(struct tss_struct *tss,
+ struct thread_struct *prev,
+ struct thread_struct *next,
+ unsigned long tifp, unsigned long tifn)
{
- struct thread_struct *prev, *next;
-
- prev = &prev_p->thread;
- next = &next_p->thread;
-
- if (test_tsk_thread_flag(prev_p, TIF_BLOCKSTEP) ^
- test_tsk_thread_flag(next_p, TIF_BLOCKSTEP)) {
- unsigned long debugctl = get_debugctlmsr();
-
- debugctl &= ~DEBUGCTLMSR_BTF;
- if (test_tsk_thread_flag(next_p, TIF_BLOCKSTEP))
- debugctl |= DEBUGCTLMSR_BTF;
-
- update_debugctlmsr(debugctl);
- }
-
- if (test_tsk_thread_flag(prev_p, TIF_NOTSC) ^
- test_tsk_thread_flag(next_p, TIF_NOTSC)) {
- /* prev and next are different */
- if (test_tsk_thread_flag(next_p, TIF_NOTSC))
- hard_disable_TSC();
- else
- hard_enable_TSC();
- }
-
- if (test_tsk_thread_flag(next_p, TIF_IO_BITMAP)) {
+ if (tifn & _TIF_IO_BITMAP) {
/*
* Copy the relevant range of the IO bitmap.
* Normally this is 128 bytes or less:
*/
memcpy(tss->io_bitmap, next->io_bitmap_ptr,
max(prev->io_bitmap_max, next->io_bitmap_max));
- } else if (test_tsk_thread_flag(prev_p, TIF_IO_BITMAP)) {
+ } else if (tifp & _TIF_IO_BITMAP) {
/*
* Clear any possible leftover bits:
*/
memset(tss->io_bitmap, 0xff, prev->io_bitmap_max);
}
+}
+
+#ifdef CONFIG_SMP
+
+struct ssb_state {
+ struct ssb_state *shared_state;
+ raw_spinlock_t lock;
+ unsigned int disable_state;
+ unsigned long local_state;
+};
+
+#define LSTATE_SSB 0
+
+static DEFINE_PER_CPU(struct ssb_state, ssb_state);
+
+void speculative_store_bypass_ht_init(void)
+{
+ struct ssb_state *st = this_cpu_ptr(&ssb_state);
+ unsigned int this_cpu = smp_processor_id();
+ unsigned int cpu;
+
+ st->local_state = 0;
+
+ /*
+ * Shared state setup happens once on the first bringup
+ * of the CPU. It's not destroyed on CPU hotunplug.
+ */
+ if (st->shared_state)
+ return;
+
+ raw_spin_lock_init(&st->lock);
+
+ /*
+ * Go over HT siblings and check whether one of them has set up the
+ * shared state pointer already.
+ */
+ for_each_cpu(cpu, topology_sibling_cpumask(this_cpu)) {
+ if (cpu == this_cpu)
+ continue;
+
+ if (!per_cpu(ssb_state, cpu).shared_state)
+ continue;
+
+ /* Link it to the state of the sibling: */
+ st->shared_state = per_cpu(ssb_state, cpu).shared_state;
+ return;
+ }
+
+ /*
+ * First HT sibling to come up on the core. Link shared state of
+ * the first HT sibling to itself. The siblings on the same core
+ * which come up later will see the shared state pointer and link
+ * themself to the state of this CPU.
+ */
+ st->shared_state = st;
+}
+
+/*
+ * Logic is: First HT sibling enables SSBD for both siblings in the core
+ * and last sibling to disable it, disables it for the whole core. This how
+ * MSR_SPEC_CTRL works in "hardware":
+ *
+ * CORE_SPEC_CTRL = THREAD0_SPEC_CTRL | THREAD1_SPEC_CTRL
+ */
+static __always_inline void amd_set_core_ssb_state(unsigned long tifn)
+{
+ struct ssb_state *st = this_cpu_ptr(&ssb_state);
+ u64 msr = x86_amd_ls_cfg_base;
+
+ if (!static_cpu_has(X86_FEATURE_ZEN)) {
+ msr |= ssbd_tif_to_amd_ls_cfg(tifn);
+ wrmsrl(MSR_AMD64_LS_CFG, msr);
+ return;
+ }
+
+ if (tifn & _TIF_SSBD) {
+ /*
+ * Since this can race with prctl(), block reentry on the
+ * same CPU.
+ */
+ if (__test_and_set_bit(LSTATE_SSB, &st->local_state))
+ return;
+
+ msr |= x86_amd_ls_cfg_ssbd_mask;
+
+ raw_spin_lock(&st->shared_state->lock);
+ /* First sibling enables SSBD: */
+ if (!st->shared_state->disable_state)
+ wrmsrl(MSR_AMD64_LS_CFG, msr);
+ st->shared_state->disable_state++;
+ raw_spin_unlock(&st->shared_state->lock);
+ } else {
+ if (!__test_and_clear_bit(LSTATE_SSB, &st->local_state))
+ return;
+
+ raw_spin_lock(&st->shared_state->lock);
+ st->shared_state->disable_state--;
+ if (!st->shared_state->disable_state)
+ wrmsrl(MSR_AMD64_LS_CFG, msr);
+ raw_spin_unlock(&st->shared_state->lock);
+ }
+}
+#else
+static __always_inline void amd_set_core_ssb_state(unsigned long tifn)
+{
+ u64 msr = x86_amd_ls_cfg_base | ssbd_tif_to_amd_ls_cfg(tifn);
+
+ wrmsrl(MSR_AMD64_LS_CFG, msr);
+}
+#endif
+
+static __always_inline void amd_set_ssb_virt_state(unsigned long tifn)
+{
+ /*
+ * SSBD has the same definition in SPEC_CTRL and VIRT_SPEC_CTRL,
+ * so ssbd_tif_to_spec_ctrl() just works.
+ */
+ wrmsrl(MSR_AMD64_VIRT_SPEC_CTRL, ssbd_tif_to_spec_ctrl(tifn));
+}
+
+static __always_inline void intel_set_ssb_state(unsigned long tifn)
+{
+ u64 msr = x86_spec_ctrl_base | ssbd_tif_to_spec_ctrl(tifn);
+
+ wrmsrl(MSR_IA32_SPEC_CTRL, msr);
+}
+
+static __always_inline void __speculative_store_bypass_update(unsigned long tifn)
+{
+ if (static_cpu_has(X86_FEATURE_VIRT_SSBD))
+ amd_set_ssb_virt_state(tifn);
+ else if (static_cpu_has(X86_FEATURE_LS_CFG_SSBD))
+ amd_set_core_ssb_state(tifn);
+ else
+ intel_set_ssb_state(tifn);
+}
+
+void speculative_store_bypass_update(unsigned long tif)
+{
+ preempt_disable();
+ __speculative_store_bypass_update(tif);
+ preempt_enable();
+}
+
+void __switch_to_xtra(struct task_struct *prev_p, struct task_struct *next_p,
+ struct tss_struct *tss)
+{
+ struct thread_struct *prev, *next;
+ unsigned long tifp, tifn;
+
+ prev = &prev_p->thread;
+ next = &next_p->thread;
+
+ tifn = READ_ONCE(task_thread_info(next_p)->flags);
+ tifp = READ_ONCE(task_thread_info(prev_p)->flags);
+ switch_to_bitmap(tss, prev, next, tifp, tifn);
+
propagate_user_return_notify(prev_p, next_p);
+
+ if ((tifp & _TIF_BLOCKSTEP || tifn & _TIF_BLOCKSTEP) &&
+ arch_has_block_step()) {
+ unsigned long debugctl, msk;
+
+ rdmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
+ debugctl &= ~DEBUGCTLMSR_BTF;
+ msk = tifn & _TIF_BLOCKSTEP;
+ debugctl |= (msk >> TIF_BLOCKSTEP) << DEBUGCTLMSR_BTF_SHIFT;
+ wrmsrl(MSR_IA32_DEBUGCTLMSR, debugctl);
+ }
+
+ if ((tifp ^ tifn) & _TIF_NOTSC)
+ cr4_toggle_bits(X86_CR4_TSD);
+
+ if ((tifp ^ tifn) & _TIF_SSBD)
+ __speculative_store_bypass_update(tifn);
}
/*
diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c
index dffe81d..a266181 100644
--- a/arch/x86/kernel/process_64.c
+++ b/arch/x86/kernel/process_64.c
@@ -360,6 +360,7 @@ start_thread(struct pt_regs *regs, unsigned long new_ip, unsigned long new_sp)
start_thread_common(regs, new_ip, new_sp,
__USER_CS, __USER_DS, 0);
}
+EXPORT_SYMBOL_GPL(start_thread);
#ifdef CONFIG_COMPAT
void compat_start_thread(struct pt_regs *regs, u32 new_ip, u32 new_sp)
diff --git a/arch/x86/kernel/quirks.c b/arch/x86/kernel/quirks.c
index 0bee04d..b57100a 100644
--- a/arch/x86/kernel/quirks.c
+++ b/arch/x86/kernel/quirks.c
@@ -643,12 +643,19 @@ static void quirk_intel_brickland_xeon_ras_cap(struct pci_dev *pdev)
/* Skylake */
static void quirk_intel_purley_xeon_ras_cap(struct pci_dev *pdev)
{
- u32 capid0;
+ u32 capid0, capid5;
pci_read_config_dword(pdev, 0x84, &capid0);
+ pci_read_config_dword(pdev, 0x98, &capid5);
- if ((capid0 & 0xc0) == 0xc0)
+ /*
+ * CAPID0{7:6} indicate whether this is an advanced RAS SKU
+ * CAPID5{8:5} indicate that various NVDIMM usage modes are
+ * enabled, so memory machine check recovery is also enabled.
+ */
+ if ((capid0 & 0xc0) == 0xc0 || (capid5 & 0x1e0))
static_branch_inc(&mcsafe_key);
+
}
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x0ec3, quirk_intel_brickland_xeon_ras_cap);
DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x2fc0, quirk_intel_brickland_xeon_ras_cap);
diff --git a/arch/x86/kernel/reboot.c b/arch/x86/kernel/reboot.c
index 03f21db..4a12362 100644
--- a/arch/x86/kernel/reboot.c
+++ b/arch/x86/kernel/reboot.c
@@ -9,6 +9,7 @@
#include <linux/sched.h>
#include <linux/tboot.h>
#include <linux/delay.h>
+#include <linux/frame.h>
#include <acpi/reboot.h>
#include <asm/io.h>
#include <asm/apic.h>
@@ -127,6 +128,7 @@ void __noreturn machine_real_restart(unsigned int type)
#ifdef CONFIG_APM_MODULE
EXPORT_SYMBOL(machine_real_restart);
#endif
+STACK_FRAME_NON_STANDARD(machine_real_restart);
/*
* Some Apple MacBook and MacBookPro's needs reboot=p to be able to reboot
diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c
index 6b55012..49960ec 100644
--- a/arch/x86/kernel/setup.c
+++ b/arch/x86/kernel/setup.c
@@ -854,6 +854,12 @@ void __init setup_arch(char **cmdline_p)
memblock_reserve(__pa_symbol(_text),
(unsigned long)__bss_stop - (unsigned long)_text);
+ /*
+ * Make sure page 0 is always reserved because on systems with
+ * L1TF its contents can be leaked to user processes.
+ */
+ memblock_reserve(0, PAGE_SIZE);
+
early_reserve_initrd();
/*
diff --git a/arch/x86/kernel/smp.c b/arch/x86/kernel/smp.c
index ea217ca..2863ad3 100644
--- a/arch/x86/kernel/smp.c
+++ b/arch/x86/kernel/smp.c
@@ -271,6 +271,7 @@ __visible void __irq_entry smp_reschedule_interrupt(struct pt_regs *regs)
/*
* KVM uses this interrupt to force a cpu out of guest mode
*/
+ kvm_set_cpu_l1tf_flush_l1d();
}
__visible void __irq_entry smp_trace_reschedule_interrupt(struct pt_regs *regs)
diff --git a/arch/x86/kernel/smpboot.c b/arch/x86/kernel/smpboot.c
index e803d72..ef38bc1 100644
--- a/arch/x86/kernel/smpboot.c
+++ b/arch/x86/kernel/smpboot.c
@@ -75,13 +75,8 @@
#include <asm/i8259.h>
#include <asm/realmode.h>
#include <asm/misc.h>
-
-/* Number of siblings per CPU package */
-int smp_num_siblings = 1;
-EXPORT_SYMBOL(smp_num_siblings);
-
-/* Last level cache ID of each logical CPU */
-DEFINE_PER_CPU_READ_MOSTLY(u16, cpu_llc_id) = BAD_APICID;
+#include <asm/spec-ctrl.h>
+#include <asm/hw_irq.h>
/* representing HT siblings of each logical CPU */
DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_sibling_map);
@@ -229,6 +224,8 @@ static void notrace start_secondary(void *unused)
*/
check_tsc_sync_target();
+ speculative_store_bypass_ht_init();
+
/*
* Lock vector_lock and initialize the vectors on this cpu
* before setting the cpu online. We must set it online with
@@ -293,6 +290,23 @@ int topology_update_package_map(unsigned int pkg, unsigned int cpu)
}
/**
+ * topology_is_primary_thread - Check whether CPU is the primary SMT thread
+ * @cpu: CPU to check
+ */
+bool topology_is_primary_thread(unsigned int cpu)
+{
+ return apic_id_is_primary_thread(per_cpu(x86_cpu_to_apicid, cpu));
+}
+
+/**
+ * topology_smt_supported - Check whether SMT is supported by the CPUs
+ */
+bool topology_smt_supported(void)
+{
+ return smp_num_siblings > 1;
+}
+
+/**
* topology_phys_to_logical_pkg - Map a physical package id to a logical
*
* Returns logical package id or -1 if not found
@@ -1325,6 +1339,8 @@ void __init native_smp_prepare_cpus(unsigned int max_cpus)
set_mtrr_aps_delayed_init();
smp_quirk_init_udelay();
+
+ speculative_store_bypass_ht_init();
}
void arch_enable_nonboot_cpus_begin(void)
@@ -1492,6 +1508,7 @@ static void remove_siblinginfo(int cpu)
cpumask_clear(topology_core_cpumask(cpu));
c->phys_proc_id = 0;
c->cpu_core_id = 0;
+ c->booted_cores = 0;
cpumask_clear_cpu(cpu, cpu_sibling_setup_mask);
recompute_smt_state();
}
@@ -1591,6 +1608,8 @@ static inline void mwait_play_dead(void)
void *mwait_ptr;
int i;
+ if (boot_cpu_data.x86_vendor == X86_VENDOR_AMD)
+ return;
if (!this_cpu_has(X86_FEATURE_MWAIT))
return;
if (!this_cpu_has(X86_FEATURE_CLFLUSH))
diff --git a/arch/x86/kernel/time.c b/arch/x86/kernel/time.c
index d39c091..89d1190 100644
--- a/arch/x86/kernel/time.c
+++ b/arch/x86/kernel/time.c
@@ -11,6 +11,7 @@
#include <linux/clockchips.h>
#include <linux/interrupt.h>
+#include <linux/irq.h>
#include <linux/i8253.h>
#include <linux/time.h>
#include <linux/export.h>
@@ -23,7 +24,7 @@
#include <asm/time.h>
#ifdef CONFIG_X86_64
-__visible volatile unsigned long jiffies __cacheline_aligned = INITIAL_JIFFIES;
+__visible volatile unsigned long jiffies __cacheline_aligned_in_smp = INITIAL_JIFFIES;
#endif
unsigned long profile_pc(struct pt_regs *regs)
diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c
index f214293..5bbfa2f 100644
--- a/arch/x86/kernel/traps.c
+++ b/arch/x86/kernel/traps.c
@@ -799,16 +799,18 @@ static void math_error(struct pt_regs *regs, int error_code, int trapnr)
char *str = (trapnr == X86_TRAP_MF) ? "fpu exception" :
"simd exception";
- if (notify_die(DIE_TRAP, str, regs, error_code, trapnr, SIGFPE) == NOTIFY_STOP)
- return;
cond_local_irq_enable(regs);
if (!user_mode(regs)) {
- if (!fixup_exception(regs, trapnr)) {
- task->thread.error_code = error_code;
- task->thread.trap_nr = trapnr;
+ if (fixup_exception(regs, trapnr))
+ return;
+
+ task->thread.error_code = error_code;
+ task->thread.trap_nr = trapnr;
+
+ if (notify_die(DIE_TRAP, str, regs, error_code,
+ trapnr, SIGFPE) != NOTIFY_STOP)
die(str, regs, error_code);
- }
return;
}
diff --git a/arch/x86/kernel/tsc.c b/arch/x86/kernel/tsc.c
index bbfb03ec..769c370 100644
--- a/arch/x86/kernel/tsc.c
+++ b/arch/x86/kernel/tsc.c
@@ -24,6 +24,7 @@
#include <asm/geode.h>
#include <asm/apic.h>
#include <asm/intel-family.h>
+#include <asm/i8259.h>
unsigned int __read_mostly cpu_khz; /* TSC clocks / usec, not used here */
EXPORT_SYMBOL(cpu_khz);
@@ -409,7 +410,7 @@ static unsigned long calc_hpet_ref(u64 deltatsc, u64 hpet1, u64 hpet2)
hpet2 -= hpet1;
tmp = ((u64)hpet2 * hpet_readl(HPET_PERIOD));
do_div(tmp, 1000000);
- do_div(deltatsc, tmp);
+ deltatsc = div64_u64(deltatsc, tmp);
return (unsigned long) deltatsc;
}
@@ -456,6 +457,20 @@ static unsigned long pit_calibrate_tsc(u32 latch, unsigned long ms, int loopmin)
unsigned long tscmin, tscmax;
int pitcnt;
+ if (!has_legacy_pic()) {
+ /*
+ * Relies on tsc_early_delay_calibrate() to have given us semi
+ * usable udelay(), wait for the same 50ms we would have with
+ * the PIT loop below.
+ */
+ udelay(10 * USEC_PER_MSEC);
+ udelay(10 * USEC_PER_MSEC);
+ udelay(10 * USEC_PER_MSEC);
+ udelay(10 * USEC_PER_MSEC);
+ udelay(10 * USEC_PER_MSEC);
+ return ULONG_MAX;
+ }
+
/* Set the Gate high, disable speaker */
outb((inb(0x61) & ~0x02) | 0x01, 0x61);
@@ -580,6 +595,9 @@ static unsigned long quick_pit_calibrate(void)
u64 tsc, delta;
unsigned long d1, d2;
+ if (!has_legacy_pic())
+ return 0;
+
/* Set the Gate high, disable speaker */
outb((inb(0x61) & ~0x02) | 0x01, 0x61);
diff --git a/arch/x86/kernel/tsc_msr.c b/arch/x86/kernel/tsc_msr.c
index 0fe720d..3f818ce 100644
--- a/arch/x86/kernel/tsc_msr.c
+++ b/arch/x86/kernel/tsc_msr.c
@@ -12,6 +12,7 @@
#include <asm/setup.h>
#include <asm/apic.h>
#include <asm/param.h>
+#include <asm/tsc.h>
#define MAX_NUM_FREQS 9
diff --git a/arch/x86/kernel/uprobes.c b/arch/x86/kernel/uprobes.c
index 495c776..e78a6b1 100644
--- a/arch/x86/kernel/uprobes.c
+++ b/arch/x86/kernel/uprobes.c
@@ -290,7 +290,7 @@ static int uprobe_init_insn(struct arch_uprobe *auprobe, struct insn *insn, bool
insn_init(insn, auprobe->insn, sizeof(auprobe->insn), x86_64);
/* has the side-effect of processing the entire instruction */
insn_get_length(insn);
- if (WARN_ON_ONCE(!insn_complete(insn)))
+ if (!insn_complete(insn))
return -ENOEXEC;
if (is_prefix_bad(insn))
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index c7194e9..e783a5d 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -352,7 +352,6 @@
DISCARDS
/DISCARD/ : {
*(.eh_frame)
- *(__func_stack_frame_non_standard)
}
}
diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index 93f924d..c17d389 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -16,7 +16,6 @@
#include <linux/export.h>
#include <linux/vmalloc.h>
#include <linux/uaccess.h>
-#include <asm/fpu/internal.h> /* For use_eager_fpu. Ugh! */
#include <asm/user.h>
#include <asm/fpu/xstate.h>
#include "cpuid.h"
@@ -114,8 +113,7 @@ int kvm_update_cpuid(struct kvm_vcpu *vcpu)
if (best && (best->eax & (F(XSAVES) | F(XSAVEC))))
best->ebx = xstate_required_size(vcpu->arch.xcr0, true);
- if (use_eager_fpu())
- kvm_x86_ops->fpu_activate(vcpu);
+ kvm_x86_ops->fpu_activate(vcpu);
/*
* The existing code assumes virtual address is 48-bit in the canonical
@@ -357,7 +355,7 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
/* cpuid 0x80000008.ebx */
const u32 kvm_cpuid_8000_0008_ebx_x86_features =
- F(IBPB) | F(IBRS);
+ F(AMD_IBPB) | F(AMD_IBRS) | F(VIRT_SSBD);
/* cpuid 0xC0000001.edx */
const u32 kvm_cpuid_C000_0001_edx_x86_features =
@@ -382,7 +380,7 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
/* cpuid 7.0.edx*/
const u32 kvm_cpuid_7_0_edx_x86_features =
- F(SPEC_CTRL) | F(ARCH_CAPABILITIES);
+ F(SPEC_CTRL) | F(SPEC_CTRL_SSBD) | F(ARCH_CAPABILITIES);
/* all calls to cpuid_count() should be made on the same cpu */
get_cpu();
@@ -468,6 +466,11 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
entry->ecx &= ~F(PKU);
entry->edx &= kvm_cpuid_7_0_edx_x86_features;
cpuid_mask(&entry->edx, CPUID_7_EDX);
+ /*
+ * We emulate ARCH_CAPABILITIES in software even
+ * if the host doesn't support it.
+ */
+ entry->edx |= F(ARCH_CAPABILITIES);
} else {
entry->ebx = 0;
entry->ecx = 0;
@@ -618,13 +621,20 @@ static inline int __do_cpuid_ent(struct kvm_cpuid_entry2 *entry, u32 function,
g_phys_as = phys_as;
entry->eax = g_phys_as | (virt_as << 8);
entry->edx = 0;
- /* IBRS and IBPB aren't necessarily present in hardware cpuid */
- if (boot_cpu_has(X86_FEATURE_IBPB))
- entry->ebx |= F(IBPB);
- if (boot_cpu_has(X86_FEATURE_IBRS))
- entry->ebx |= F(IBRS);
+ /*
+ * IBRS, IBPB and VIRT_SSBD aren't necessarily present in
+ * hardware cpuid
+ */
+ if (boot_cpu_has(X86_FEATURE_AMD_IBPB))
+ entry->ebx |= F(AMD_IBPB);
+ if (boot_cpu_has(X86_FEATURE_AMD_IBRS))
+ entry->ebx |= F(AMD_IBRS);
+ if (boot_cpu_has(X86_FEATURE_VIRT_SSBD))
+ entry->ebx |= F(VIRT_SSBD);
entry->ebx &= kvm_cpuid_8000_0008_ebx_x86_features;
cpuid_mask(&entry->ebx, CPUID_8000_0008_EBX);
+ if (boot_cpu_has(X86_FEATURE_LS_CFG_SSBD))
+ entry->ebx |= F(VIRT_SSBD);
break;
}
case 0x80000019:
diff --git a/arch/x86/kvm/cpuid.h b/arch/x86/kvm/cpuid.h
index d1beb71..8a841b9 100644
--- a/arch/x86/kvm/cpuid.h
+++ b/arch/x86/kvm/cpuid.h
@@ -165,21 +165,21 @@ static inline bool guest_cpuid_has_ibpb(struct kvm_vcpu *vcpu)
struct kvm_cpuid_entry2 *best;
best = kvm_find_cpuid_entry(vcpu, 0x80000008, 0);
- if (best && (best->ebx & bit(X86_FEATURE_IBPB)))
+ if (best && (best->ebx & bit(X86_FEATURE_AMD_IBPB)))
return true;
best = kvm_find_cpuid_entry(vcpu, 7, 0);
return best && (best->edx & bit(X86_FEATURE_SPEC_CTRL));
}
-static inline bool guest_cpuid_has_ibrs(struct kvm_vcpu *vcpu)
+static inline bool guest_cpuid_has_spec_ctrl(struct kvm_vcpu *vcpu)
{
struct kvm_cpuid_entry2 *best;
best = kvm_find_cpuid_entry(vcpu, 0x80000008, 0);
- if (best && (best->ebx & bit(X86_FEATURE_IBRS)))
+ if (best && (best->ebx & bit(X86_FEATURE_AMD_IBRS)))
return true;
best = kvm_find_cpuid_entry(vcpu, 7, 0);
- return best && (best->edx & bit(X86_FEATURE_SPEC_CTRL));
+ return best && (best->edx & (bit(X86_FEATURE_SPEC_CTRL) | bit(X86_FEATURE_SPEC_CTRL_SSBD)));
}
static inline bool guest_cpuid_has_arch_capabilities(struct kvm_vcpu *vcpu)
@@ -190,6 +190,15 @@ static inline bool guest_cpuid_has_arch_capabilities(struct kvm_vcpu *vcpu)
return best && (best->edx & bit(X86_FEATURE_ARCH_CAPABILITIES));
}
+static inline bool guest_cpuid_has_virt_ssbd(struct kvm_vcpu *vcpu)
+{
+ struct kvm_cpuid_entry2 *best;
+
+ best = kvm_find_cpuid_entry(vcpu, 0x80000008, 0);
+ return best && (best->ebx & bit(X86_FEATURE_VIRT_SSBD));
+}
+
+
/*
* NRIPS is provided through cpuidfn 0x8000000a.edx bit 3
diff --git a/arch/x86/kvm/emulate.c b/arch/x86/kvm/emulate.c
index c8d5738..510cfc0 100644
--- a/arch/x86/kvm/emulate.c
+++ b/arch/x86/kvm/emulate.c
@@ -802,6 +802,19 @@ static inline int jmp_rel(struct x86_emulate_ctxt *ctxt, int rel)
return assign_eip_near(ctxt, ctxt->_eip + rel);
}
+static int linear_read_system(struct x86_emulate_ctxt *ctxt, ulong linear,
+ void *data, unsigned size)
+{
+ return ctxt->ops->read_std(ctxt, linear, data, size, &ctxt->exception, true);
+}
+
+static int linear_write_system(struct x86_emulate_ctxt *ctxt,
+ ulong linear, void *data,
+ unsigned int size)
+{
+ return ctxt->ops->write_std(ctxt, linear, data, size, &ctxt->exception, true);
+}
+
static int segmented_read_std(struct x86_emulate_ctxt *ctxt,
struct segmented_address addr,
void *data,
@@ -813,7 +826,7 @@ static int segmented_read_std(struct x86_emulate_ctxt *ctxt,
rc = linearize(ctxt, addr, size, false, &linear);
if (rc != X86EMUL_CONTINUE)
return rc;
- return ctxt->ops->read_std(ctxt, linear, data, size, &ctxt->exception);
+ return ctxt->ops->read_std(ctxt, linear, data, size, &ctxt->exception, false);
}
static int segmented_write_std(struct x86_emulate_ctxt *ctxt,
@@ -827,7 +840,7 @@ static int segmented_write_std(struct x86_emulate_ctxt *ctxt,
rc = linearize(ctxt, addr, size, true, &linear);
if (rc != X86EMUL_CONTINUE)
return rc;
- return ctxt->ops->write_std(ctxt, linear, data, size, &ctxt->exception);
+ return ctxt->ops->write_std(ctxt, linear, data, size, &ctxt->exception, false);
}
/*
@@ -1500,8 +1513,7 @@ static int read_interrupt_descriptor(struct x86_emulate_ctxt *ctxt,
return emulate_gp(ctxt, index << 3 | 0x2);
addr = dt.address + index * 8;
- return ctxt->ops->read_std(ctxt, addr, desc, sizeof *desc,
- &ctxt->exception);
+ return linear_read_system(ctxt, addr, desc, sizeof *desc);
}
static void get_descriptor_table_ptr(struct x86_emulate_ctxt *ctxt,
@@ -1564,8 +1576,7 @@ static int read_segment_descriptor(struct x86_emulate_ctxt *ctxt,
if (rc != X86EMUL_CONTINUE)
return rc;
- return ctxt->ops->read_std(ctxt, *desc_addr_p, desc, sizeof(*desc),
- &ctxt->exception);
+ return linear_read_system(ctxt, *desc_addr_p, desc, sizeof(*desc));
}
/* allowed just for 8 bytes segments */
@@ -1579,8 +1590,7 @@ static int write_segment_descriptor(struct x86_emulate_ctxt *ctxt,
if (rc != X86EMUL_CONTINUE)
return rc;
- return ctxt->ops->write_std(ctxt, addr, desc, sizeof *desc,
- &ctxt->exception);
+ return linear_write_system(ctxt, addr, desc, sizeof *desc);
}
static int __load_segment_descriptor(struct x86_emulate_ctxt *ctxt,
@@ -1741,8 +1751,7 @@ static int __load_segment_descriptor(struct x86_emulate_ctxt *ctxt,
return ret;
}
} else if (ctxt->mode == X86EMUL_MODE_PROT64) {
- ret = ctxt->ops->read_std(ctxt, desc_addr+8, &base3,
- sizeof(base3), &ctxt->exception);
+ ret = linear_read_system(ctxt, desc_addr+8, &base3, sizeof(base3));
if (ret != X86EMUL_CONTINUE)
return ret;
if (is_noncanonical_address(get_desc_base(&seg_desc) |
@@ -2055,11 +2064,11 @@ static int __emulate_int_real(struct x86_emulate_ctxt *ctxt, int irq)
eip_addr = dt.address + (irq << 2);
cs_addr = dt.address + (irq << 2) + 2;
- rc = ops->read_std(ctxt, cs_addr, &cs, 2, &ctxt->exception);
+ rc = linear_read_system(ctxt, cs_addr, &cs, 2);
if (rc != X86EMUL_CONTINUE)
return rc;
- rc = ops->read_std(ctxt, eip_addr, &eip, 2, &ctxt->exception);
+ rc = linear_read_system(ctxt, eip_addr, &eip, 2);
if (rc != X86EMUL_CONTINUE)
return rc;
@@ -2903,12 +2912,12 @@ static bool emulator_io_port_access_allowed(struct x86_emulate_ctxt *ctxt,
#ifdef CONFIG_X86_64
base |= ((u64)base3) << 32;
#endif
- r = ops->read_std(ctxt, base + 102, &io_bitmap_ptr, 2, NULL);
+ r = ops->read_std(ctxt, base + 102, &io_bitmap_ptr, 2, NULL, true);
if (r != X86EMUL_CONTINUE)
return false;
if (io_bitmap_ptr + port/8 > desc_limit_scaled(&tr_seg))
return false;
- r = ops->read_std(ctxt, base + io_bitmap_ptr + port/8, &perm, 2, NULL);
+ r = ops->read_std(ctxt, base + io_bitmap_ptr + port/8, &perm, 2, NULL, true);
if (r != X86EMUL_CONTINUE)
return false;
if ((perm >> bit_idx) & mask)
@@ -3037,35 +3046,30 @@ static int task_switch_16(struct x86_emulate_ctxt *ctxt,
u16 tss_selector, u16 old_tss_sel,
ulong old_tss_base, struct desc_struct *new_desc)
{
- const struct x86_emulate_ops *ops = ctxt->ops;
struct tss_segment_16 tss_seg;
int ret;
u32 new_tss_base = get_desc_base(new_desc);
- ret = ops->read_std(ctxt, old_tss_base, &tss_seg, sizeof tss_seg,
- &ctxt->exception);
+ ret = linear_read_system(ctxt, old_tss_base, &tss_seg, sizeof tss_seg);
if (ret != X86EMUL_CONTINUE)
return ret;
save_state_to_tss16(ctxt, &tss_seg);
- ret = ops->write_std(ctxt, old_tss_base, &tss_seg, sizeof tss_seg,
- &ctxt->exception);
+ ret = linear_write_system(ctxt, old_tss_base, &tss_seg, sizeof tss_seg);
if (ret != X86EMUL_CONTINUE)
return ret;
- ret = ops->read_std(ctxt, new_tss_base, &tss_seg, sizeof tss_seg,
- &ctxt->exception);
+ ret = linear_read_system(ctxt, new_tss_base, &tss_seg, sizeof tss_seg);
if (ret != X86EMUL_CONTINUE)
return ret;
if (old_tss_sel != 0xffff) {
tss_seg.prev_task_link = old_tss_sel;
- ret = ops->write_std(ctxt, new_tss_base,
- &tss_seg.prev_task_link,
- sizeof tss_seg.prev_task_link,
- &ctxt->exception);
+ ret = linear_write_system(ctxt, new_tss_base,
+ &tss_seg.prev_task_link,
+ sizeof tss_seg.prev_task_link);
if (ret != X86EMUL_CONTINUE)
return ret;
}
@@ -3181,38 +3185,34 @@ static int task_switch_32(struct x86_emulate_ctxt *ctxt,
u16 tss_selector, u16 old_tss_sel,
ulong old_tss_base, struct desc_struct *new_desc)
{
- const struct x86_emulate_ops *ops = ctxt->ops;
struct tss_segment_32 tss_seg;
int ret;
u32 new_tss_base = get_desc_base(new_desc);
u32 eip_offset = offsetof(struct tss_segment_32, eip);
u32 ldt_sel_offset = offsetof(struct tss_segment_32, ldt_selector);
- ret = ops->read_std(ctxt, old_tss_base, &tss_seg, sizeof tss_seg,
- &ctxt->exception);
+ ret = linear_read_system(ctxt, old_tss_base, &tss_seg, sizeof tss_seg);
if (ret != X86EMUL_CONTINUE)
return ret;
save_state_to_tss32(ctxt, &tss_seg);
/* Only GP registers and segment selectors are saved */
- ret = ops->write_std(ctxt, old_tss_base + eip_offset, &tss_seg.eip,
- ldt_sel_offset - eip_offset, &ctxt->exception);
+ ret = linear_write_system(ctxt, old_tss_base + eip_offset, &tss_seg.eip,
+ ldt_sel_offset - eip_offset);
if (ret != X86EMUL_CONTINUE)
return ret;
- ret = ops->read_std(ctxt, new_tss_base, &tss_seg, sizeof tss_seg,
- &ctxt->exception);
+ ret = linear_read_system(ctxt, new_tss_base, &tss_seg, sizeof tss_seg);
if (ret != X86EMUL_CONTINUE)
return ret;
if (old_tss_sel != 0xffff) {
tss_seg.prev_task_link = old_tss_sel;
- ret = ops->write_std(ctxt, new_tss_base,
- &tss_seg.prev_task_link,
- sizeof tss_seg.prev_task_link,
- &ctxt->exception);
+ ret = linear_write_system(ctxt, new_tss_base,
+ &tss_seg.prev_task_link,
+ sizeof tss_seg.prev_task_link);
if (ret != X86EMUL_CONTINUE)
return ret;
}
diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
index 5c3d416f..c863056 100644
--- a/arch/x86/kvm/lapic.c
+++ b/arch/x86/kvm/lapic.c
@@ -57,7 +57,7 @@
#define APIC_BUS_CYCLE_NS 1
/* #define apic_debug(fmt,arg...) printk(KERN_WARNING fmt,##arg) */
-#define apic_debug(fmt, arg...)
+#define apic_debug(fmt, arg...) do {} while (0)
/* 14 is the version for Xeon and Pentium 8.4.8*/
#define APIC_VERSION (0x14UL | ((KVM_APIC_LVT_NUM - 1) << 16))
@@ -299,8 +299,16 @@ void kvm_apic_set_version(struct kvm_vcpu *vcpu)
if (!lapic_in_kernel(vcpu))
return;
+ /*
+ * KVM emulates 82093AA datasheet (with in-kernel IOAPIC implementation)
+ * which doesn't have EOI register; Some buggy OSes (e.g. Windows with
+ * Hyper-V role) disable EOI broadcast in lapic not checking for IOAPIC
+ * version first and level-triggered interrupts never get EOIed in
+ * IOAPIC.
+ */
feat = kvm_find_cpuid_entry(apic->vcpu, 0x1, 0);
- if (feat && (feat->ecx & (1 << (X86_FEATURE_X2APIC & 31))))
+ if (feat && (feat->ecx & (1 << (X86_FEATURE_X2APIC & 31))) &&
+ !ioapic_in_kernel(vcpu->kvm))
v |= APIC_LVR_DIRECTED_EOI;
kvm_lapic_set_reg(apic, APIC_LVR, v);
}
@@ -1212,9 +1220,8 @@ EXPORT_SYMBOL_GPL(kvm_lapic_reg_read);
static int apic_mmio_in_range(struct kvm_lapic *apic, gpa_t addr)
{
- return kvm_apic_hw_enabled(apic) &&
- addr >= apic->base_address &&
- addr < apic->base_address + LAPIC_MMIO_LENGTH;
+ return addr >= apic->base_address &&
+ addr < apic->base_address + LAPIC_MMIO_LENGTH;
}
static int apic_mmio_read(struct kvm_vcpu *vcpu, struct kvm_io_device *this,
@@ -1226,6 +1233,15 @@ static int apic_mmio_read(struct kvm_vcpu *vcpu, struct kvm_io_device *this,
if (!apic_mmio_in_range(apic, address))
return -EOPNOTSUPP;
+ if (!kvm_apic_hw_enabled(apic) || apic_x2apic_mode(apic)) {
+ if (!kvm_check_has_quirk(vcpu->kvm,
+ KVM_X86_QUIRK_LAPIC_MMIO_HOLE))
+ return -EOPNOTSUPP;
+
+ memset(data, 0xff, len);
+ return 0;
+ }
+
kvm_lapic_reg_read(apic, offset, len, data);
return 0;
@@ -1638,6 +1654,14 @@ static int apic_mmio_write(struct kvm_vcpu *vcpu, struct kvm_io_device *this,
if (!apic_mmio_in_range(apic, address))
return -EOPNOTSUPP;
+ if (!kvm_apic_hw_enabled(apic) || apic_x2apic_mode(apic)) {
+ if (!kvm_check_has_quirk(vcpu->kvm,
+ KVM_X86_QUIRK_LAPIC_MMIO_HOLE))
+ return -EOPNOTSUPP;
+
+ return 0;
+ }
+
/*
* APIC register must be aligned on 128-bits boundary.
* 32/64/128 bits registers must be accessed thru 32 bits.
diff --git a/arch/x86/kvm/mmu.c b/arch/x86/kvm/mmu.c
index a16c066..676edfc 100644
--- a/arch/x86/kvm/mmu.c
+++ b/arch/x86/kvm/mmu.c
@@ -698,7 +698,7 @@ static int mmu_topup_memory_cache_page(struct kvm_mmu_memory_cache *cache,
if (cache->nobjs >= min)
return 0;
while (cache->nobjs < ARRAY_SIZE(cache->objects)) {
- page = (void *)__get_free_page(GFP_KERNEL);
+ page = (void *)__get_free_page(GFP_KERNEL_ACCOUNT);
if (!page)
return -ENOMEM;
cache->objects[cache->nobjs++] = page;
@@ -4297,9 +4297,9 @@ static bool need_remote_flush(u64 old, u64 new)
}
static u64 mmu_pte_write_fetch_gpte(struct kvm_vcpu *vcpu, gpa_t *gpa,
- const u8 *new, int *bytes)
+ int *bytes)
{
- u64 gentry;
+ u64 gentry = 0;
int r;
/*
@@ -4311,22 +4311,12 @@ static u64 mmu_pte_write_fetch_gpte(struct kvm_vcpu *vcpu, gpa_t *gpa,
/* Handle a 32-bit guest writing two halves of a 64-bit gpte */
*gpa &= ~(gpa_t)7;
*bytes = 8;
- r = kvm_vcpu_read_guest(vcpu, *gpa, &gentry, 8);
- if (r)
- gentry = 0;
- new = (const u8 *)&gentry;
}
- switch (*bytes) {
- case 4:
- gentry = *(const u32 *)new;
- break;
- case 8:
- gentry = *(const u64 *)new;
- break;
- default:
- gentry = 0;
- break;
+ if (*bytes == 4 || *bytes == 8) {
+ r = kvm_vcpu_read_guest_atomic(vcpu, *gpa, &gentry, *bytes);
+ if (r)
+ gentry = 0;
}
return gentry;
@@ -4437,8 +4427,6 @@ static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,
pgprintk("%s: gpa %llx bytes %d\n", __func__, gpa, bytes);
- gentry = mmu_pte_write_fetch_gpte(vcpu, &gpa, new, &bytes);
-
/*
* No need to care whether allocation memory is successful
* or not since pte prefetch is skiped if it does not have
@@ -4447,6 +4435,9 @@ static void kvm_mmu_pte_write(struct kvm_vcpu *vcpu, gpa_t gpa,
mmu_topup_memory_caches(vcpu);
spin_lock(&vcpu->kvm->mmu_lock);
+
+ gentry = mmu_pte_write_fetch_gpte(vcpu, &gpa, &bytes);
+
++vcpu->kvm->stat.mmu_pte_write;
kvm_mmu_audit(vcpu, AUDIT_PRE_PTE_WRITE);
diff --git a/arch/x86/kvm/svm.c b/arch/x86/kvm/svm.c
index aaa93b4..fa1b0e3 100644
--- a/arch/x86/kvm/svm.c
+++ b/arch/x86/kvm/svm.c
@@ -36,6 +36,7 @@
#include <linux/slab.h>
#include <linux/amd-iommu.h>
#include <linux/hashtable.h>
+#include <linux/frame.h>
#include <asm/apic.h>
#include <asm/perf_event.h>
@@ -45,7 +46,7 @@
#include <asm/kvm_para.h>
#include <asm/irq_remapping.h>
#include <asm/microcode.h>
-#include <asm/nospec-branch.h>
+#include <asm/spec-ctrl.h>
#include <asm/virtext.h>
#include "trace.h"
@@ -174,6 +175,8 @@ struct vcpu_svm {
uint64_t sysenter_eip;
uint64_t tsc_aux;
+ u64 msr_decfg;
+
u64 next_rip;
u64 host_user_msrs[NR_HOST_SAVE_USER_MSRS];
@@ -185,6 +188,12 @@ struct vcpu_svm {
} host;
u64 spec_ctrl;
+ /*
+ * Contains guest-controlled bits of VIRT_SPEC_CTRL, which will be
+ * translated into the appropriate L2_CFG bits on the host to
+ * perform speculative control.
+ */
+ u64 virt_spec_ctrl;
u32 *msrpm;
@@ -1324,20 +1333,23 @@ static u64 *avic_get_physical_id_entry(struct kvm_vcpu *vcpu, int index)
static int avic_init_access_page(struct kvm_vcpu *vcpu)
{
struct kvm *kvm = vcpu->kvm;
- int ret;
+ int ret = 0;
+ mutex_lock(&kvm->slots_lock);
if (kvm->arch.apic_access_page_done)
- return 0;
+ goto out;
- ret = x86_set_memory_region(kvm,
- APIC_ACCESS_PAGE_PRIVATE_MEMSLOT,
- APIC_DEFAULT_PHYS_BASE,
- PAGE_SIZE);
+ ret = __x86_set_memory_region(kvm,
+ APIC_ACCESS_PAGE_PRIVATE_MEMSLOT,
+ APIC_DEFAULT_PHYS_BASE,
+ PAGE_SIZE);
if (ret)
- return ret;
+ goto out;
kvm->arch.apic_access_page_done = true;
- return 0;
+out:
+ mutex_unlock(&kvm->slots_lock);
+ return ret;
}
static int avic_init_backing_page(struct kvm_vcpu *vcpu)
@@ -1560,7 +1572,9 @@ static void svm_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
u32 dummy;
u32 eax = 1;
+ vcpu->arch.microcode_version = 0x01000065;
svm->spec_ctrl = 0;
+ svm->virt_spec_ctrl = 0;
if (!init_event) {
svm->vcpu.arch.apic_base = APIC_DEFAULT_PHYS_BASE |
@@ -1661,21 +1675,31 @@ static struct kvm_vcpu *svm_create_vcpu(struct kvm *kvm, unsigned int id)
return ERR_PTR(err);
}
+static void svm_clear_current_vmcb(struct vmcb *vmcb)
+{
+ int i;
+
+ for_each_online_cpu(i)
+ cmpxchg(&per_cpu(svm_data, i)->current_vmcb, vmcb, NULL);
+}
+
static void svm_free_vcpu(struct kvm_vcpu *vcpu)
{
struct vcpu_svm *svm = to_svm(vcpu);
+ /*
+ * The vmcb page can be recycled, causing a false negative in
+ * svm_vcpu_load(). So, ensure that no logical CPU has this
+ * vmcb page recorded as its current vmcb.
+ */
+ svm_clear_current_vmcb(svm->vmcb);
+
__free_page(pfn_to_page(svm->vmcb_pa >> PAGE_SHIFT));
__free_pages(virt_to_page(svm->msrpm), MSRPM_ALLOC_ORDER);
__free_page(virt_to_page(svm->nested.hsave));
__free_pages(virt_to_page(svm->nested.msrpm), MSRPM_ALLOC_ORDER);
kvm_vcpu_uninit(vcpu);
kmem_cache_free(kvm_vcpu_cache, svm);
- /*
- * The vmcb page can be recycled, causing a false negative in
- * svm_vcpu_load(). So do a full IBPB now.
- */
- indirect_branch_prediction_barrier();
}
static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)
@@ -2116,6 +2140,8 @@ static int pf_interception(struct vcpu_svm *svm)
u32 error_code;
int r = 1;
+ svm->vcpu.arch.l1tf_flush_l1d = true;
+
switch (svm->apf_reason) {
default:
error_code = svm->vmcb->control.exit_info_1;
@@ -3475,6 +3501,22 @@ static int cr8_write_interception(struct vcpu_svm *svm)
return 0;
}
+static int svm_get_msr_feature(struct kvm_msr_entry *msr)
+{
+ msr->data = 0;
+
+ switch (msr->index) {
+ case MSR_F10H_DECFG:
+ if (boot_cpu_has(X86_FEATURE_LFENCE_RDTSC))
+ msr->data |= MSR_F10H_DECFG_LFENCE_SERIALIZE;
+ break;
+ default:
+ return 1;
+ }
+
+ return 0;
+}
+
static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
{
struct vcpu_svm *svm = to_svm(vcpu);
@@ -3545,13 +3587,17 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
break;
case MSR_IA32_SPEC_CTRL:
if (!msr_info->host_initiated &&
- !guest_cpuid_has_ibrs(vcpu))
+ !guest_cpuid_has_spec_ctrl(vcpu))
return 1;
msr_info->data = svm->spec_ctrl;
break;
- case MSR_IA32_UCODE_REV:
- msr_info->data = 0x01000065;
+ case MSR_AMD64_VIRT_SPEC_CTRL:
+ if (!msr_info->host_initiated &&
+ !guest_cpuid_has_virt_ssbd(vcpu))
+ return 1;
+
+ msr_info->data = svm->virt_spec_ctrl;
break;
case MSR_F15H_IC_CFG: {
@@ -3570,6 +3616,9 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
msr_info->data = 0x1E;
}
break;
+ case MSR_F10H_DECFG:
+ msr_info->data = svm->msr_decfg;
+ break;
default:
return kvm_get_msr_common(vcpu, msr_info);
}
@@ -3643,7 +3692,7 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
break;
case MSR_IA32_SPEC_CTRL:
if (!msr->host_initiated &&
- !guest_cpuid_has_ibrs(vcpu))
+ !guest_cpuid_has_spec_ctrl(vcpu))
return 1;
/* The STIBP bit doesn't fault even if it's not advertised */
@@ -3684,6 +3733,16 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
break;
set_msr_interception(svm->msrpm, MSR_IA32_PRED_CMD, 0, 1);
break;
+ case MSR_AMD64_VIRT_SPEC_CTRL:
+ if (!msr->host_initiated &&
+ !guest_cpuid_has_virt_ssbd(vcpu))
+ return 1;
+
+ if (data & ~SPEC_CTRL_SSBD)
+ return 1;
+
+ svm->virt_spec_ctrl = data;
+ break;
case MSR_STAR:
svm->vmcb->save.star = data;
break;
@@ -3748,6 +3807,24 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr)
case MSR_VM_IGNNE:
vcpu_unimpl(vcpu, "unimplemented wrmsr: 0x%x data 0x%llx\n", ecx, data);
break;
+ case MSR_F10H_DECFG: {
+ struct kvm_msr_entry msr_entry;
+
+ msr_entry.index = msr->index;
+ if (svm_get_msr_feature(&msr_entry))
+ return 1;
+
+ /* Check the supported bits */
+ if (data & ~msr_entry.data)
+ return 1;
+
+ /* Don't allow the guest to change a bit, #GP */
+ if (!msr->host_initiated && (data ^ msr_entry.data))
+ return 1;
+
+ svm->msr_decfg = data;
+ break;
+ }
case MSR_IA32_APICBASE:
if (kvm_vcpu_apicv_active(vcpu))
avic_update_vapic_bar(to_svm(vcpu), data);
@@ -4909,16 +4986,15 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
clgi();
- local_irq_enable();
-
/*
* If this vCPU has touched SPEC_CTRL, restore the guest's value if
* it's non-zero. Since vmentry is serialising on affected CPUs, there
* is no need to worry about the conditional branch over the wrmsr
* being speculatively taken.
*/
- if (svm->spec_ctrl)
- native_wrmsrl(MSR_IA32_SPEC_CTRL, svm->spec_ctrl);
+ x86_spec_ctrl_set_guest(svm->spec_ctrl, svm->virt_spec_ctrl);
+
+ local_irq_enable();
asm volatile (
"push %%" _ASM_BP "; \n\t"
@@ -5012,6 +5088,18 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
#endif
);
+ /* Eliminate branch target predictions from guest mode */
+ vmexit_fill_RSB();
+
+#ifdef CONFIG_X86_64
+ wrmsrl(MSR_GS_BASE, svm->host.gs_base);
+#else
+ loadsegment(fs, svm->host.fs);
+#ifndef CONFIG_X86_32_LAZY_GS
+ loadsegment(gs, svm->host.gs);
+#endif
+#endif
+
/*
* We do not use IBRS in the kernel. If this vCPU has used the
* SPEC_CTRL MSR it may have left it on; save the value and
@@ -5030,25 +5118,12 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
if (unlikely(!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL)))
svm->spec_ctrl = native_read_msr(MSR_IA32_SPEC_CTRL);
- if (svm->spec_ctrl)
- native_wrmsrl(MSR_IA32_SPEC_CTRL, 0);
-
- /* Eliminate branch target predictions from guest mode */
- vmexit_fill_RSB();
-
-#ifdef CONFIG_X86_64
- wrmsrl(MSR_GS_BASE, svm->host.gs_base);
-#else
- loadsegment(fs, svm->host.fs);
-#ifndef CONFIG_X86_32_LAZY_GS
- loadsegment(gs, svm->host.gs);
-#endif
-#endif
-
reload_tss(vcpu);
local_irq_disable();
+ x86_spec_ctrl_restore_host(svm->spec_ctrl, svm->virt_spec_ctrl);
+
vcpu->arch.cr2 = svm->vmcb->save.cr2;
vcpu->arch.regs[VCPU_REGS_RAX] = svm->vmcb->save.rax;
vcpu->arch.regs[VCPU_REGS_RSP] = svm->vmcb->save.rsp;
@@ -5089,6 +5164,7 @@ static void svm_vcpu_run(struct kvm_vcpu *vcpu)
mark_all_clean(svm->vmcb);
}
+STACK_FRAME_NON_STANDARD(svm_vcpu_run);
static void svm_set_cr3(struct kvm_vcpu *vcpu, unsigned long root)
{
@@ -5145,7 +5221,7 @@ static bool svm_cpu_has_accelerated_tpr(void)
return false;
}
-static bool svm_has_high_real_mode_segbase(void)
+static bool svm_has_emulated_msr(int index)
{
return true;
}
@@ -5462,7 +5538,7 @@ static struct kvm_x86_ops svm_x86_ops __ro_after_init = {
.hardware_enable = svm_hardware_enable,
.hardware_disable = svm_hardware_disable,
.cpu_has_accelerated_tpr = svm_cpu_has_accelerated_tpr,
- .cpu_has_high_real_mode_segbase = svm_has_high_real_mode_segbase,
+ .has_emulated_msr = svm_has_emulated_msr,
.vcpu_create = svm_create_vcpu,
.vcpu_free = svm_free_vcpu,
@@ -5478,6 +5554,7 @@ static struct kvm_x86_ops svm_x86_ops __ro_after_init = {
.vcpu_unblocking = svm_vcpu_unblocking,
.update_bp_intercept = update_bp_intercept,
+ .get_msr_feature = svm_get_msr_feature,
.get_msr = svm_get_msr,
.set_msr = svm_set_msr,
.get_segment_base = svm_get_segment_base,
diff --git a/arch/x86/kvm/vmx.c b/arch/x86/kvm/vmx.c
index ff2030f..726cc44 100644
--- a/arch/x86/kvm/vmx.c
+++ b/arch/x86/kvm/vmx.c
@@ -33,6 +33,7 @@
#include <linux/slab.h>
#include <linux/tboot.h>
#include <linux/hrtimer.h>
+#include <linux/frame.h>
#include <linux/nospec.h>
#include "kvm_cache_regs.h"
#include "x86.h"
@@ -50,7 +51,7 @@
#include <asm/apic.h>
#include <asm/irq_remapping.h>
#include <asm/microcode.h>
-#include <asm/nospec-branch.h>
+#include <asm/spec-ctrl.h>
#include "trace.h"
#include "pmu.h"
@@ -188,6 +189,156 @@ module_param(ple_window_max, int, S_IRUGO);
extern const ulong vmx_return;
+static DEFINE_STATIC_KEY_FALSE(vmx_l1d_should_flush);
+static DEFINE_STATIC_KEY_FALSE(vmx_l1d_flush_cond);
+static DEFINE_MUTEX(vmx_l1d_flush_mutex);
+
+/* Storage for pre module init parameter parsing */
+static enum vmx_l1d_flush_state __read_mostly vmentry_l1d_flush_param = VMENTER_L1D_FLUSH_AUTO;
+
+static const struct {
+ const char *option;
+ bool for_parse;
+} vmentry_l1d_param[] = {
+ [VMENTER_L1D_FLUSH_AUTO] = {"auto", true},
+ [VMENTER_L1D_FLUSH_NEVER] = {"never", true},
+ [VMENTER_L1D_FLUSH_COND] = {"cond", true},
+ [VMENTER_L1D_FLUSH_ALWAYS] = {"always", true},
+ [VMENTER_L1D_FLUSH_EPT_DISABLED] = {"EPT disabled", false},
+ [VMENTER_L1D_FLUSH_NOT_REQUIRED] = {"not required", false},
+};
+
+#define L1D_CACHE_ORDER 4
+static void *vmx_l1d_flush_pages;
+
+static int vmx_setup_l1d_flush(enum vmx_l1d_flush_state l1tf)
+{
+ struct page *page;
+ unsigned int i;
+
+ if (!enable_ept) {
+ l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_EPT_DISABLED;
+ return 0;
+ }
+
+ if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES)) {
+ u64 msr;
+
+ rdmsrl(MSR_IA32_ARCH_CAPABILITIES, msr);
+ if (msr & ARCH_CAP_SKIP_VMENTRY_L1DFLUSH) {
+ l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_NOT_REQUIRED;
+ return 0;
+ }
+ }
+
+ /* If set to auto use the default l1tf mitigation method */
+ if (l1tf == VMENTER_L1D_FLUSH_AUTO) {
+ switch (l1tf_mitigation) {
+ case L1TF_MITIGATION_OFF:
+ l1tf = VMENTER_L1D_FLUSH_NEVER;
+ break;
+ case L1TF_MITIGATION_FLUSH_NOWARN:
+ case L1TF_MITIGATION_FLUSH:
+ case L1TF_MITIGATION_FLUSH_NOSMT:
+ l1tf = VMENTER_L1D_FLUSH_COND;
+ break;
+ case L1TF_MITIGATION_FULL:
+ case L1TF_MITIGATION_FULL_FORCE:
+ l1tf = VMENTER_L1D_FLUSH_ALWAYS;
+ break;
+ }
+ } else if (l1tf_mitigation == L1TF_MITIGATION_FULL_FORCE) {
+ l1tf = VMENTER_L1D_FLUSH_ALWAYS;
+ }
+
+ if (l1tf != VMENTER_L1D_FLUSH_NEVER && !vmx_l1d_flush_pages &&
+ !boot_cpu_has(X86_FEATURE_FLUSH_L1D)) {
+ page = alloc_pages(GFP_KERNEL, L1D_CACHE_ORDER);
+ if (!page)
+ return -ENOMEM;
+ vmx_l1d_flush_pages = page_address(page);
+
+ /*
+ * Initialize each page with a different pattern in
+ * order to protect against KSM in the nested
+ * virtualization case.
+ */
+ for (i = 0; i < 1u << L1D_CACHE_ORDER; ++i) {
+ memset(vmx_l1d_flush_pages + i * PAGE_SIZE, i + 1,
+ PAGE_SIZE);
+ }
+ }
+
+ l1tf_vmx_mitigation = l1tf;
+
+ if (l1tf != VMENTER_L1D_FLUSH_NEVER)
+ static_branch_enable(&vmx_l1d_should_flush);
+ else
+ static_branch_disable(&vmx_l1d_should_flush);
+
+ if (l1tf == VMENTER_L1D_FLUSH_COND)
+ static_branch_enable(&vmx_l1d_flush_cond);
+ else
+ static_branch_disable(&vmx_l1d_flush_cond);
+ return 0;
+}
+
+static int vmentry_l1d_flush_parse(const char *s)
+{
+ unsigned int i;
+
+ if (s) {
+ for (i = 0; i < ARRAY_SIZE(vmentry_l1d_param); i++) {
+ if (vmentry_l1d_param[i].for_parse &&
+ sysfs_streq(s, vmentry_l1d_param[i].option))
+ return i;
+ }
+ }
+ return -EINVAL;
+}
+
+static int vmentry_l1d_flush_set(const char *s, const struct kernel_param *kp)
+{
+ int l1tf, ret;
+
+ l1tf = vmentry_l1d_flush_parse(s);
+ if (l1tf < 0)
+ return l1tf;
+
+ if (!boot_cpu_has(X86_BUG_L1TF))
+ return 0;
+
+ /*
+ * Has vmx_init() run already? If not then this is the pre init
+ * parameter parsing. In that case just store the value and let
+ * vmx_init() do the proper setup after enable_ept has been
+ * established.
+ */
+ if (l1tf_vmx_mitigation == VMENTER_L1D_FLUSH_AUTO) {
+ vmentry_l1d_flush_param = l1tf;
+ return 0;
+ }
+
+ mutex_lock(&vmx_l1d_flush_mutex);
+ ret = vmx_setup_l1d_flush(l1tf);
+ mutex_unlock(&vmx_l1d_flush_mutex);
+ return ret;
+}
+
+static int vmentry_l1d_flush_get(char *s, const struct kernel_param *kp)
+{
+ if (WARN_ON_ONCE(l1tf_vmx_mitigation >= ARRAY_SIZE(vmentry_l1d_param)))
+ return sprintf(s, "???\n");
+
+ return sprintf(s, "%s\n", vmentry_l1d_param[l1tf_vmx_mitigation].option);
+}
+
+static const struct kernel_param_ops vmentry_l1d_flush_ops = {
+ .set = vmentry_l1d_flush_set,
+ .get = vmentry_l1d_flush_get,
+};
+module_param_cb(vmentry_l1d_flush, &vmentry_l1d_flush_ops, NULL, 0644);
+
#define NR_AUTOLOAD_MSRS 8
struct vmcs {
@@ -540,6 +691,11 @@ static inline int pi_test_sn(struct pi_desc *pi_desc)
(unsigned long *)&pi_desc->control);
}
+struct vmx_msrs {
+ unsigned int nr;
+ struct vmx_msr_entry val[NR_AUTOLOAD_MSRS];
+};
+
struct vcpu_vmx {
struct kvm_vcpu vcpu;
unsigned long host_rsp;
@@ -572,9 +728,8 @@ struct vcpu_vmx {
struct loaded_vmcs *loaded_vmcs;
bool __launched; /* temporary, used in vmx_vcpu_run */
struct msr_autoload {
- unsigned nr;
- struct vmx_msr_entry guest[NR_AUTOLOAD_MSRS];
- struct vmx_msr_entry host[NR_AUTOLOAD_MSRS];
+ struct vmx_msrs guest;
+ struct vmx_msrs host;
} msr_autoload;
struct {
int loaded;
@@ -922,7 +1077,7 @@ static void copy_vmcs12_to_shadow(struct vcpu_vmx *vmx);
static void copy_shadow_to_vmcs12(struct vcpu_vmx *vmx);
static int alloc_identity_pagetable(struct kvm *kvm);
static void vmx_update_msr_bitmap(struct kvm_vcpu *vcpu);
-static void __always_inline vmx_disable_intercept_for_msr(unsigned long *msr_bitmap,
+static __always_inline void vmx_disable_intercept_for_msr(unsigned long *msr_bitmap,
u32 msr, int type);
static DEFINE_PER_CPU(struct vmcs *, vmxarea);
@@ -1919,9 +2074,20 @@ static void clear_atomic_switch_msr_special(struct vcpu_vmx *vmx,
vm_exit_controls_clearbit(vmx, exit);
}
+static int find_msr(struct vmx_msrs *m, unsigned int msr)
+{
+ unsigned int i;
+
+ for (i = 0; i < m->nr; ++i) {
+ if (m->val[i].index == msr)
+ return i;
+ }
+ return -ENOENT;
+}
+
static void clear_atomic_switch_msr(struct vcpu_vmx *vmx, unsigned msr)
{
- unsigned i;
+ int i;
struct msr_autoload *m = &vmx->msr_autoload;
switch (msr) {
@@ -1942,18 +2108,21 @@ static void clear_atomic_switch_msr(struct vcpu_vmx *vmx, unsigned msr)
}
break;
}
+ i = find_msr(&m->guest, msr);
+ if (i < 0)
+ goto skip_guest;
+ --m->guest.nr;
+ m->guest.val[i] = m->guest.val[m->guest.nr];
+ vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, m->guest.nr);
- for (i = 0; i < m->nr; ++i)
- if (m->guest[i].index == msr)
- break;
-
- if (i == m->nr)
+skip_guest:
+ i = find_msr(&m->host, msr);
+ if (i < 0)
return;
- --m->nr;
- m->guest[i] = m->guest[m->nr];
- m->host[i] = m->host[m->nr];
- vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, m->nr);
- vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, m->nr);
+
+ --m->host.nr;
+ m->host.val[i] = m->host.val[m->host.nr];
+ vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, m->host.nr);
}
static void add_atomic_switch_msr_special(struct vcpu_vmx *vmx,
@@ -1968,9 +2137,9 @@ static void add_atomic_switch_msr_special(struct vcpu_vmx *vmx,
}
static void add_atomic_switch_msr(struct vcpu_vmx *vmx, unsigned msr,
- u64 guest_val, u64 host_val)
+ u64 guest_val, u64 host_val, bool entry_only)
{
- unsigned i;
+ int i, j = 0;
struct msr_autoload *m = &vmx->msr_autoload;
switch (msr) {
@@ -2005,24 +2174,31 @@ static void add_atomic_switch_msr(struct vcpu_vmx *vmx, unsigned msr,
wrmsrl(MSR_IA32_PEBS_ENABLE, 0);
}
- for (i = 0; i < m->nr; ++i)
- if (m->guest[i].index == msr)
- break;
+ i = find_msr(&m->guest, msr);
+ if (!entry_only)
+ j = find_msr(&m->host, msr);
- if (i == NR_AUTOLOAD_MSRS) {
+ if (i == NR_AUTOLOAD_MSRS || j == NR_AUTOLOAD_MSRS) {
printk_once(KERN_WARNING "Not enough msr switch entries. "
"Can't add msr %x\n", msr);
return;
- } else if (i == m->nr) {
- ++m->nr;
- vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, m->nr);
- vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, m->nr);
}
+ if (i < 0) {
+ i = m->guest.nr++;
+ vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, m->guest.nr);
+ }
+ m->guest.val[i].index = msr;
+ m->guest.val[i].value = guest_val;
- m->guest[i].index = msr;
- m->guest[i].value = guest_val;
- m->host[i].index = msr;
- m->host[i].value = host_val;
+ if (entry_only)
+ return;
+
+ if (j < 0) {
+ j = m->host.nr++;
+ vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, m->host.nr);
+ }
+ m->host.val[j].index = msr;
+ m->host.val[j].value = host_val;
}
static void reload_tss(void)
@@ -2079,7 +2255,7 @@ static bool update_transition_efer(struct vcpu_vmx *vmx, int efer_offset)
guest_efer &= ~EFER_LME;
if (guest_efer != host_efer)
add_atomic_switch_msr(vmx, MSR_EFER,
- guest_efer, host_efer);
+ guest_efer, host_efer, false);
return false;
} else {
guest_efer &= ~ignore_bits;
@@ -2558,6 +2734,8 @@ static void vmx_queue_exception(struct kvm_vcpu *vcpu, unsigned nr,
return;
}
+ WARN_ON_ONCE(vmx->emulation_required);
+
if (kvm_exception_is_soft(nr)) {
vmcs_write32(VM_ENTRY_INSTRUCTION_LEN,
vmx->vcpu.arch.event_exit_inst_len);
@@ -2991,6 +3169,11 @@ static inline bool vmx_feature_control_msr_valid(struct kvm_vcpu *vcpu,
return !(val & ~valid_bits);
}
+static int vmx_get_msr_feature(struct kvm_msr_entry *msr)
+{
+ return 1;
+}
+
/*
* Reads an msr value (of 'msr_index') into 'pdata'.
* Returns 0 on success, non-0 otherwise.
@@ -3020,7 +3203,7 @@ static int vmx_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
break;
case MSR_IA32_SPEC_CTRL:
if (!msr_info->host_initiated &&
- !guest_cpuid_has_ibrs(vcpu))
+ !guest_cpuid_has_spec_ctrl(vcpu))
return 1;
msr_info->data = to_vmx(vcpu)->spec_ctrl;
@@ -3137,11 +3320,11 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
break;
case MSR_IA32_SPEC_CTRL:
if (!msr_info->host_initiated &&
- !guest_cpuid_has_ibrs(vcpu))
+ !guest_cpuid_has_spec_ctrl(vcpu))
return 1;
/* The STIBP bit doesn't fault even if it's not advertised */
- if (data & ~(SPEC_CTRL_IBRS | SPEC_CTRL_STIBP))
+ if (data & ~(SPEC_CTRL_IBRS | SPEC_CTRL_STIBP | SPEC_CTRL_SSBD))
return 1;
vmx->spec_ctrl = data;
@@ -3241,7 +3424,7 @@ static int vmx_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
vcpu->arch.ia32_xss = data;
if (vcpu->arch.ia32_xss != host_xss)
add_atomic_switch_msr(vmx, MSR_IA32_XSS,
- vcpu->arch.ia32_xss, host_xss);
+ vcpu->arch.ia32_xss, host_xss, false);
else
clear_atomic_switch_msr(vmx, MSR_IA32_XSS);
break;
@@ -4689,7 +4872,7 @@ static void free_vpid(int vpid)
spin_unlock(&vmx_vpid_lock);
}
-static void __always_inline vmx_disable_intercept_for_msr(unsigned long *msr_bitmap,
+static __always_inline void vmx_disable_intercept_for_msr(unsigned long *msr_bitmap,
u32 msr, int type)
{
int f = sizeof(unsigned long);
@@ -4724,7 +4907,7 @@ static void __always_inline vmx_disable_intercept_for_msr(unsigned long *msr_bit
}
}
-static void __always_inline vmx_enable_intercept_for_msr(unsigned long *msr_bitmap,
+static __always_inline void vmx_enable_intercept_for_msr(unsigned long *msr_bitmap,
u32 msr, int type)
{
int f = sizeof(unsigned long);
@@ -4759,7 +4942,7 @@ static void __always_inline vmx_enable_intercept_for_msr(unsigned long *msr_bitm
}
}
-static void __always_inline vmx_set_intercept_for_msr(unsigned long *msr_bitmap,
+static __always_inline void vmx_set_intercept_for_msr(unsigned long *msr_bitmap,
u32 msr, int type, bool value)
{
if (value)
@@ -5262,9 +5445,9 @@ static int vmx_vcpu_setup(struct vcpu_vmx *vmx)
vmcs_write32(VM_EXIT_MSR_STORE_COUNT, 0);
vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, 0);
- vmcs_write64(VM_EXIT_MSR_LOAD_ADDR, __pa(vmx->msr_autoload.host));
+ vmcs_write64(VM_EXIT_MSR_LOAD_ADDR, __pa(vmx->msr_autoload.host.val));
vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, 0);
- vmcs_write64(VM_ENTRY_MSR_LOAD_ADDR, __pa(vmx->msr_autoload.guest));
+ vmcs_write64(VM_ENTRY_MSR_LOAD_ADDR, __pa(vmx->msr_autoload.guest.val));
if (vmcs_config.vmentry_ctrl & VM_ENTRY_LOAD_IA32_PAT)
vmcs_write64(GUEST_IA32_PAT, vmx->vcpu.arch.pat);
@@ -5284,8 +5467,7 @@ static int vmx_vcpu_setup(struct vcpu_vmx *vmx)
++vmx->nmsrs;
}
- if (boot_cpu_has(X86_FEATURE_ARCH_CAPABILITIES))
- rdmsrl(MSR_IA32_ARCH_CAPABILITIES, vmx->arch_capabilities);
+ vmx->arch_capabilities = kvm_get_arch_capabilities();
vm_exit_controls_init(vmx, vmcs_config.vmexit_ctrl);
@@ -5314,6 +5496,7 @@ static void vmx_vcpu_reset(struct kvm_vcpu *vcpu, bool init_event)
u64 cr0;
vmx->rmode.vm86_active = 0;
+ vcpu->arch.microcode_version = 0x100000000ULL;
vmx->spec_ctrl = 0;
vmx->soft_vnmi_blocked = 0;
@@ -5719,6 +5902,7 @@ static int handle_exception(struct kvm_vcpu *vcpu)
BUG_ON(enable_ept);
cr2 = vmcs_readl(EXIT_QUALIFICATION);
trace_kvm_page_fault(cr2, error_code);
+ vcpu->arch.l1tf_flush_l1d = true;
if (kvm_event_needs_reinjection(vcpu))
kvm_mmu_unprotect_page_virt(vcpu, cr2);
@@ -6364,9 +6548,24 @@ static int handle_ept_misconfig(struct kvm_vcpu *vcpu)
gpa = vmcs_read64(GUEST_PHYSICAL_ADDRESS);
if (!kvm_io_bus_write(vcpu, KVM_FAST_MMIO_BUS, gpa, 0, NULL)) {
- skip_emulated_instruction(vcpu);
trace_kvm_fast_mmio(gpa);
- return 1;
+ /*
+ * Doing kvm_skip_emulated_instruction() depends on undefined
+ * behavior: Intel's manual doesn't mandate
+ * VM_EXIT_INSTRUCTION_LEN to be set in VMCS when EPT MISCONFIG
+ * occurs and while on real hardware it was observed to be set,
+ * other hypervisors (namely Hyper-V) don't set it, we end up
+ * advancing IP with some random value. Disable fast mmio when
+ * running nested and keep it for real hardware in hope that
+ * VM_EXIT_INSTRUCTION_LEN will always be set correctly.
+ */
+ if (!static_cpu_has(X86_FEATURE_HYPERVISOR)) {
+ skip_emulated_instruction(vcpu);
+ return 1;
+ }
+ else
+ return x86_emulate_instruction(vcpu, gpa, EMULTYPE_SKIP,
+ NULL, 0) == EMULATE_DONE;
}
ret = handle_mmio_page_fault(vcpu, gpa, true);
@@ -6430,12 +6629,12 @@ static int handle_invalid_guest_state(struct kvm_vcpu *vcpu)
goto out;
}
- if (err != EMULATE_DONE) {
- vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
- vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_EMULATION;
- vcpu->run->internal.ndata = 0;
- return 0;
- }
+ if (err != EMULATE_DONE)
+ goto emulation_error;
+
+ if (vmx->emulation_required && !vmx->rmode.vm86_active &&
+ vcpu->arch.exception.pending)
+ goto emulation_error;
if (vcpu->arch.halt_request) {
vcpu->arch.halt_request = 0;
@@ -6451,6 +6650,12 @@ static int handle_invalid_guest_state(struct kvm_vcpu *vcpu)
out:
return ret;
+
+emulation_error:
+ vcpu->run->exit_reason = KVM_EXIT_INTERNAL_ERROR;
+ vcpu->run->internal.suberror = KVM_INTERNAL_ERROR_EMULATION;
+ vcpu->run->internal.ndata = 0;
+ return 0;
}
static int __grow_ple_window(int val)
@@ -6919,8 +7124,7 @@ static int nested_vmx_check_vmptr(struct kvm_vcpu *vcpu, int exit_reason,
vmcs_read32(VMX_INSTRUCTION_INFO), false, &gva))
return 1;
- if (kvm_read_guest_virt(&vcpu->arch.emulate_ctxt, gva, &vmptr,
- sizeof(vmptr), &e)) {
+ if (kvm_read_guest_virt(vcpu, gva, &vmptr, sizeof(vmptr), &e)) {
kvm_inject_page_fault(vcpu, &e);
return 1;
}
@@ -7077,6 +7281,8 @@ static int handle_vmon(struct kvm_vcpu *vcpu)
HRTIMER_MODE_REL_PINNED);
vmx->nested.preemption_timer.function = vmx_preemption_timer_fn;
+ vmx->nested.vpid02 = allocate_vpid();
+
vmx->nested.vmxon = true;
skip_emulated_instruction(vcpu);
@@ -7460,8 +7666,8 @@ static int handle_vmread(struct kvm_vcpu *vcpu)
vmx_instruction_info, true, &gva))
return 1;
/* _system ok, as nested_vmx_check_permission verified cpl=0 */
- kvm_write_guest_virt_system(&vcpu->arch.emulate_ctxt, gva,
- &field_value, (is_long_mode(vcpu) ? 8 : 4), NULL);
+ kvm_write_guest_virt_system(vcpu, gva, &field_value,
+ (is_long_mode(vcpu) ? 8 : 4), NULL);
}
nested_vmx_succeed(vcpu);
@@ -7496,8 +7702,8 @@ static int handle_vmwrite(struct kvm_vcpu *vcpu)
if (get_vmx_mem_address(vcpu, exit_qualification,
vmx_instruction_info, false, &gva))
return 1;
- if (kvm_read_guest_virt(&vcpu->arch.emulate_ctxt, gva,
- &field_value, (is_64_bit_mode(vcpu) ? 8 : 4), &e)) {
+ if (kvm_read_guest_virt(vcpu, gva, &field_value,
+ (is_64_bit_mode(vcpu) ? 8 : 4), &e)) {
kvm_inject_page_fault(vcpu, &e);
return 1;
}
@@ -7594,9 +7800,9 @@ static int handle_vmptrst(struct kvm_vcpu *vcpu)
vmx_instruction_info, true, &vmcs_gva))
return 1;
/* ok to use *_system, as nested_vmx_check_permission verified cpl=0 */
- if (kvm_write_guest_virt_system(&vcpu->arch.emulate_ctxt, vmcs_gva,
- (void *)&to_vmx(vcpu)->nested.current_vmptr,
- sizeof(u64), &e)) {
+ if (kvm_write_guest_virt_system(vcpu, vmcs_gva,
+ (void *)&to_vmx(vcpu)->nested.current_vmptr,
+ sizeof(u64), &e)) {
kvm_inject_page_fault(vcpu, &e);
return 1;
}
@@ -7650,8 +7856,7 @@ static int handle_invept(struct kvm_vcpu *vcpu)
if (get_vmx_mem_address(vcpu, vmcs_readl(EXIT_QUALIFICATION),
vmx_instruction_info, false, &gva))
return 1;
- if (kvm_read_guest_virt(&vcpu->arch.emulate_ctxt, gva, &operand,
- sizeof(operand), &e)) {
+ if (kvm_read_guest_virt(vcpu, gva, &operand, sizeof(operand), &e)) {
kvm_inject_page_fault(vcpu, &e);
return 1;
}
@@ -7714,8 +7919,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu)
if (get_vmx_mem_address(vcpu, vmcs_readl(EXIT_QUALIFICATION),
vmx_instruction_info, false, &gva))
return 1;
- if (kvm_read_guest_virt(&vcpu->arch.emulate_ctxt, gva, &vpid,
- sizeof(u32), &e)) {
+ if (kvm_read_guest_virt(vcpu, gva, &vpid, sizeof(u32), &e)) {
kvm_inject_page_fault(vcpu, &e);
return 1;
}
@@ -8477,6 +8681,76 @@ static int vmx_handle_exit(struct kvm_vcpu *vcpu)
}
}
+/*
+ * Software based L1D cache flush which is used when microcode providing
+ * the cache control MSR is not loaded.
+ *
+ * The L1D cache is 32 KiB on Nehalem and later microarchitectures, but to
+ * flush it is required to read in 64 KiB because the replacement algorithm
+ * is not exactly LRU. This could be sized at runtime via topology
+ * information but as all relevant affected CPUs have 32KiB L1D cache size
+ * there is no point in doing so.
+ */
+static void vmx_l1d_flush(struct kvm_vcpu *vcpu)
+{
+ int size = PAGE_SIZE << L1D_CACHE_ORDER;
+
+ /*
+ * This code is only executed when the the flush mode is 'cond' or
+ * 'always'
+ */
+ if (static_branch_likely(&vmx_l1d_flush_cond)) {
+ bool flush_l1d;
+
+ /*
+ * Clear the per-vcpu flush bit, it gets set again
+ * either from vcpu_run() or from one of the unsafe
+ * VMEXIT handlers.
+ */
+ flush_l1d = vcpu->arch.l1tf_flush_l1d;
+ vcpu->arch.l1tf_flush_l1d = false;
+
+ /*
+ * Clear the per-cpu flush bit, it gets set again from
+ * the interrupt handlers.
+ */
+ flush_l1d |= kvm_get_cpu_l1tf_flush_l1d();
+ kvm_clear_cpu_l1tf_flush_l1d();
+
+ if (!flush_l1d)
+ return;
+ }
+
+ vcpu->stat.l1d_flush++;
+
+ if (static_cpu_has(X86_FEATURE_FLUSH_L1D)) {
+ wrmsrl(MSR_IA32_FLUSH_CMD, L1D_FLUSH);
+ return;
+ }
+
+ asm volatile(
+ /* First ensure the pages are in the TLB */
+ "xorl %%eax, %%eax\n"
+ ".Lpopulate_tlb:\n\t"
+ "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t"
+ "addl $4096, %%eax\n\t"
+ "cmpl %%eax, %[size]\n\t"
+ "jne .Lpopulate_tlb\n\t"
+ "xorl %%eax, %%eax\n\t"
+ "cpuid\n\t"
+ /* Now fill the cache */
+ "xorl %%eax, %%eax\n"
+ ".Lfill_cache:\n"
+ "movzbl (%[flush_pages], %%" _ASM_AX "), %%ecx\n\t"
+ "addl $64, %%eax\n\t"
+ "cmpl %%eax, %[size]\n\t"
+ "jne .Lfill_cache\n\t"
+ "lfence\n"
+ :: [flush_pages] "r" (vmx_l1d_flush_pages),
+ [size] "r" (size)
+ : "eax", "ebx", "ecx", "edx");
+}
+
static void update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr)
{
struct vmcs12 *vmcs12 = get_vmcs12(vcpu);
@@ -8689,10 +8963,23 @@ static void vmx_handle_external_intr(struct kvm_vcpu *vcpu)
);
}
}
+STACK_FRAME_NON_STANDARD(vmx_handle_external_intr);
-static bool vmx_has_high_real_mode_segbase(void)
+static bool vmx_has_emulated_msr(int index)
{
- return enable_unrestricted_guest || emulate_invalid_guest_state;
+ switch (index) {
+ case MSR_IA32_SMBASE:
+ /*
+ * We cannot do SMM unless we can run the guest in big
+ * real mode.
+ */
+ return enable_unrestricted_guest || emulate_invalid_guest_state;
+ case MSR_AMD64_VIRT_SPEC_CTRL:
+ /* This is AMD only. */
+ return false;
+ default:
+ return true;
+ }
}
static bool vmx_mpx_supported(void)
@@ -8835,7 +9122,7 @@ static void atomic_switch_perf_msrs(struct vcpu_vmx *vmx)
clear_atomic_switch_msr(vmx, msrs[i].msr);
else
add_atomic_switch_msr(vmx, msrs[i].msr, msrs[i].guest,
- msrs[i].host);
+ msrs[i].host, false);
}
void vmx_arm_hv_timer(struct kvm_vcpu *vcpu)
@@ -8915,10 +9202,13 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
* is no need to worry about the conditional branch over the wrmsr
* being speculatively taken.
*/
- if (vmx->spec_ctrl)
- native_wrmsrl(MSR_IA32_SPEC_CTRL, vmx->spec_ctrl);
+ x86_spec_ctrl_set_guest(vmx->spec_ctrl, 0);
vmx->__launched = vmx->loaded_vmcs->launched;
+
+ if (static_branch_unlikely(&vmx_l1d_should_flush))
+ vmx_l1d_flush(vcpu);
+
asm(
/* Store host registers */
"push %%" _ASM_DX "; push %%" _ASM_BP ";"
@@ -9054,8 +9344,7 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
if (unlikely(!msr_write_intercepted(vcpu, MSR_IA32_SPEC_CTRL)))
vmx->spec_ctrl = native_read_msr(MSR_IA32_SPEC_CTRL);
- if (vmx->spec_ctrl)
- native_wrmsrl(MSR_IA32_SPEC_CTRL, 0);
+ x86_spec_ctrl_restore_host(vmx->spec_ctrl, 0);
/* Eliminate branch target predictions from guest mode */
vmexit_fill_RSB();
@@ -9118,6 +9407,7 @@ static void __noclone vmx_vcpu_run(struct kvm_vcpu *vcpu)
vmx_recover_nmi_blocking(vmx);
vmx_complete_interrupts(vmx);
}
+STACK_FRAME_NON_STANDARD(vmx_vcpu_run);
static void vmx_load_vmcs01(struct kvm_vcpu *vcpu)
{
@@ -9244,10 +9534,8 @@ static struct kvm_vcpu *vmx_create_vcpu(struct kvm *kvm, unsigned int id)
goto free_vmcs;
}
- if (nested) {
+ if (nested)
nested_vmx_setup_ctls_msrs(vmx);
- vmx->nested.vpid02 = allocate_vpid();
- }
vmx->nested.posted_intr_nv = -1;
vmx->nested.current_vmptr = -1ull;
@@ -9265,7 +9553,6 @@ static struct kvm_vcpu *vmx_create_vcpu(struct kvm *kvm, unsigned int id)
return &vmx->vcpu;
free_vmcs:
- free_vpid(vmx->nested.vpid02);
free_loaded_vmcs(vmx->loaded_vmcs);
free_msrs:
kfree(vmx->guest_msrs);
@@ -9279,6 +9566,37 @@ static struct kvm_vcpu *vmx_create_vcpu(struct kvm *kvm, unsigned int id)
return ERR_PTR(err);
}
+#define L1TF_MSG_SMT "L1TF CPU bug present and SMT on, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html for details.\n"
+#define L1TF_MSG_L1D "L1TF CPU bug present and virtualization mitigation disabled, data leak possible. See CVE-2018-3646 and https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html for details.\n"
+
+static int vmx_vm_init(struct kvm *kvm)
+{
+ if (boot_cpu_has(X86_BUG_L1TF) && enable_ept) {
+ switch (l1tf_mitigation) {
+ case L1TF_MITIGATION_OFF:
+ case L1TF_MITIGATION_FLUSH_NOWARN:
+ /* 'I explicitly don't care' is set */
+ break;
+ case L1TF_MITIGATION_FLUSH:
+ case L1TF_MITIGATION_FLUSH_NOSMT:
+ case L1TF_MITIGATION_FULL:
+ /*
+ * Warn upon starting the first VM in a potentially
+ * insecure environment.
+ */
+ if (cpu_smt_control == CPU_SMT_ENABLED)
+ pr_warn_once(L1TF_MSG_SMT);
+ if (l1tf_vmx_mitigation == VMENTER_L1D_FLUSH_NEVER)
+ pr_warn_once(L1TF_MSG_L1D);
+ break;
+ case L1TF_MITIGATION_FULL_FORCE:
+ /* Flush is enforced */
+ break;
+ }
+ }
+ return 0;
+}
+
static void __init vmx_check_processor_compat(void *rtn)
{
struct vmcs_config vmcs_conf;
@@ -10074,6 +10392,15 @@ static void prepare_vmcs02(struct kvm_vcpu *vcpu, struct vmcs12 *vmcs12)
vmx_set_constant_host_state(vmx);
/*
+ * Set the MSR load/store lists to match L0's settings.
+ */
+ vmcs_write32(VM_EXIT_MSR_STORE_COUNT, 0);
+ vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
+ vmcs_write64(VM_EXIT_MSR_LOAD_ADDR, __pa(vmx->msr_autoload.host.val));
+ vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, vmx->msr_autoload.guest.nr);
+ vmcs_write64(VM_ENTRY_MSR_LOAD_ADDR, __pa(vmx->msr_autoload.guest.val));
+
+ /*
* HOST_RSP is normally set correctly in vmx_vcpu_run() just before
* entry, but only if the current (host) sp changed from the value
* we wrote last (vmx->host_rsp). This cache is no longer relevant
@@ -10423,6 +10750,9 @@ static int nested_vmx_run(struct kvm_vcpu *vcpu, bool launch)
vmcs12->launch_state = 1;
+ /* Hide L1D cache contents from the nested guest. */
+ vmx->vcpu.arch.l1tf_flush_l1d = true;
+
if (vmcs12->guest_activity_state == GUEST_ACTIVITY_HLT)
return kvm_vcpu_halt(vcpu);
@@ -10917,6 +11247,8 @@ static void nested_vmx_vmexit(struct kvm_vcpu *vcpu, u32 exit_reason,
load_vmcs12_host_state(vcpu, vmcs12);
/* Update any VMCS fields that might have changed while L2 ran */
+ vmcs_write32(VM_EXIT_MSR_LOAD_COUNT, vmx->msr_autoload.host.nr);
+ vmcs_write32(VM_ENTRY_MSR_LOAD_COUNT, vmx->msr_autoload.guest.nr);
vmcs_write64(TSC_OFFSET, vcpu->arch.tsc_offset);
if (vmx->hv_deadline_tsc == -1)
vmcs_clear_bits(PIN_BASED_VM_EXEC_CONTROL,
@@ -11346,7 +11678,9 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = {
.hardware_enable = hardware_enable,
.hardware_disable = hardware_disable,
.cpu_has_accelerated_tpr = report_flexpriority,
- .cpu_has_high_real_mode_segbase = vmx_has_high_real_mode_segbase,
+ .has_emulated_msr = vmx_has_emulated_msr,
+
+ .vm_init = vmx_vm_init,
.vcpu_create = vmx_create_vcpu,
.vcpu_free = vmx_free_vcpu,
@@ -11357,6 +11691,7 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = {
.vcpu_put = vmx_vcpu_put,
.update_bp_intercept = update_exception_bitmap,
+ .get_msr_feature = vmx_get_msr_feature,
.get_msr = vmx_get_msr,
.set_msr = vmx_set_msr,
.get_segment_base = vmx_get_segment_base,
@@ -11467,13 +11802,54 @@ static struct kvm_x86_ops vmx_x86_ops __ro_after_init = {
.setup_mce = vmx_setup_mce,
};
+static void vmx_cleanup_l1d_flush(void)
+{
+ if (vmx_l1d_flush_pages) {
+ free_pages((unsigned long)vmx_l1d_flush_pages, L1D_CACHE_ORDER);
+ vmx_l1d_flush_pages = NULL;
+ }
+ /* Restore state so sysfs ignores VMX */
+ l1tf_vmx_mitigation = VMENTER_L1D_FLUSH_AUTO;
+}
+
+
+static void vmx_exit(void)
+{
+#ifdef CONFIG_KEXEC_CORE
+ RCU_INIT_POINTER(crash_vmclear_loaded_vmcss, NULL);
+ synchronize_rcu();
+#endif
+
+ kvm_exit();
+
+ vmx_cleanup_l1d_flush();
+}
+module_exit(vmx_exit)
+
static int __init vmx_init(void)
{
- int r = kvm_init(&vmx_x86_ops, sizeof(struct vcpu_vmx),
- __alignof__(struct vcpu_vmx), THIS_MODULE);
+ int r;
+
+ r = kvm_init(&vmx_x86_ops, sizeof(struct vcpu_vmx),
+ __alignof__(struct vcpu_vmx), THIS_MODULE);
if (r)
return r;
+ /*
+ * Must be called after kvm_init() so enable_ept is properly set
+ * up. Hand the parameter mitigation value in which was stored in
+ * the pre module init parser. If no parameter was given, it will
+ * contain 'auto' which will be turned into the default 'cond'
+ * mitigation mode.
+ */
+ if (boot_cpu_has(X86_BUG_L1TF)) {
+ r = vmx_setup_l1d_flush(vmentry_l1d_flush_param);
+ if (r) {
+ vmx_exit();
+ return r;
+ }
+ }
+
#ifdef CONFIG_KEXEC_CORE
rcu_assign_pointer(crash_vmclear_loaded_vmcss,
crash_vmclear_local_loaded_vmcss);
@@ -11481,16 +11857,4 @@ static int __init vmx_init(void)
return 0;
}
-
-static void __exit vmx_exit(void)
-{
-#ifdef CONFIG_KEXEC_CORE
- RCU_INIT_POINTER(crash_vmclear_loaded_vmcss, NULL);
- synchronize_rcu();
-#endif
-
- kvm_exit();
-}
-
module_init(vmx_init)
-module_exit(vmx_exit)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 3aaaf30..46e0ad7 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -180,6 +180,7 @@ struct kvm_stats_debugfs_item debugfs_entries[] = {
{ "insn_emulation_fail", VCPU_STAT(insn_emulation_fail) },
{ "irq_injections", VCPU_STAT(irq_injections) },
{ "nmi_injections", VCPU_STAT(nmi_injections) },
+ { "l1d_flush", VCPU_STAT(l1d_flush) },
{ "mmu_shadow_zapped", VM_STAT(mmu_shadow_zapped) },
{ "mmu_pte_write", VM_STAT(mmu_pte_write) },
{ "mmu_pte_updated", VM_STAT(mmu_pte_updated) },
@@ -1002,10 +1003,76 @@ static u32 emulated_msrs[] = {
MSR_IA32_MCG_CTL,
MSR_IA32_MCG_EXT_CTL,
MSR_IA32_SMBASE,
+ MSR_AMD64_VIRT_SPEC_CTRL,
};
static unsigned num_emulated_msrs;
+/*
+ * List of msr numbers which are used to expose MSR-based features that
+ * can be used by a hypervisor to validate requested CPU features.
+ */
+static u32 msr_based_features[] = {
+ MSR_F10H_DECFG,
+ MSR_IA32_UCODE_REV,
+ MSR_IA32_ARCH_CAPABILITIES,
+};
+
+static unsigned int num_msr_based_features;
+
+u64 kvm_get_arch_capabilities(void)
+{
+ u64 data;
+
+ rdmsrl_safe(MSR_IA32_ARCH_CAPABILITIES, &data);
+
+ /*
+ * If we're doing cache flushes (either "always" or "cond")
+ * we will do one whenever the guest does a vmlaunch/vmresume.
+ * If an outer hypervisor is doing the cache flush for us
+ * (VMENTER_L1D_FLUSH_NESTED_VM), we can safely pass that
+ * capability to the guest too, and if EPT is disabled we're not
+ * vulnerable. Overall, only VMENTER_L1D_FLUSH_NEVER will
+ * require a nested hypervisor to do a flush of its own.
+ */
+ if (l1tf_vmx_mitigation != VMENTER_L1D_FLUSH_NEVER)
+ data |= ARCH_CAP_SKIP_VMENTRY_L1DFLUSH;
+
+ return data;
+}
+EXPORT_SYMBOL_GPL(kvm_get_arch_capabilities);
+
+static int kvm_get_msr_feature(struct kvm_msr_entry *msr)
+{
+ switch (msr->index) {
+ case MSR_IA32_ARCH_CAPABILITIES:
+ msr->data = kvm_get_arch_capabilities();
+ break;
+ case MSR_IA32_UCODE_REV:
+ rdmsrl_safe(msr->index, &msr->data);
+ break;
+ default:
+ if (kvm_x86_ops->get_msr_feature(msr))
+ return 1;
+ }
+ return 0;
+}
+
+static int do_get_msr_feature(struct kvm_vcpu *vcpu, unsigned index, u64 *data)
+{
+ struct kvm_msr_entry msr;
+ int r;
+
+ msr.index = index;
+ r = kvm_get_msr_feature(&msr);
+ if (r)
+ return r;
+
+ *data = msr.data;
+
+ return 0;
+}
+
bool kvm_valid_efer(struct kvm_vcpu *vcpu, u64 efer)
{
if (efer & efer_reserved_bits)
@@ -2120,13 +2187,16 @@ int kvm_set_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
switch (msr) {
case MSR_AMD64_NB_CFG:
- case MSR_IA32_UCODE_REV:
case MSR_IA32_UCODE_WRITE:
case MSR_VM_HSAVE_PA:
case MSR_AMD64_PATCH_LOADER:
case MSR_AMD64_BU_CFG2:
break;
+ case MSR_IA32_UCODE_REV:
+ if (msr_info->host_initiated)
+ vcpu->arch.microcode_version = data;
+ break;
case MSR_EFER:
return set_efer(vcpu, data);
case MSR_K7_HWCR:
@@ -2401,7 +2471,7 @@ int kvm_get_msr_common(struct kvm_vcpu *vcpu, struct msr_data *msr_info)
msr_info->data = 0;
break;
case MSR_IA32_UCODE_REV:
- msr_info->data = 0x100000000ULL;
+ msr_info->data = vcpu->arch.microcode_version;
break;
case MSR_MTRRcap:
case 0x200 ... 0x2ff:
@@ -2544,13 +2614,11 @@ static int __msr_io(struct kvm_vcpu *vcpu, struct kvm_msrs *msrs,
int (*do_msr)(struct kvm_vcpu *vcpu,
unsigned index, u64 *data))
{
- int i, idx;
+ int i;
- idx = srcu_read_lock(&vcpu->kvm->srcu);
for (i = 0; i < msrs->nmsrs; ++i)
if (do_msr(vcpu, entries[i].index, &entries[i].data))
break;
- srcu_read_unlock(&vcpu->kvm->srcu, idx);
return i;
}
@@ -2650,6 +2718,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_ASSIGN_DEV_IRQ:
case KVM_CAP_PCI_2_3:
#endif
+ case KVM_CAP_GET_MSR_FEATURES:
r = 1;
break;
case KVM_CAP_ADJUST_CLOCK:
@@ -2664,7 +2733,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
* fringe case that is not enabled except via specific settings
* of the module parameters.
*/
- r = kvm_x86_ops->cpu_has_high_real_mode_segbase();
+ r = kvm_x86_ops->has_emulated_msr(MSR_IA32_SMBASE);
break;
case KVM_CAP_COALESCED_MMIO:
r = KVM_COALESCED_MMIO_PAGE_OFFSET;
@@ -2769,6 +2838,31 @@ long kvm_arch_dev_ioctl(struct file *filp,
goto out;
r = 0;
break;
+ case KVM_GET_MSR_FEATURE_INDEX_LIST: {
+ struct kvm_msr_list __user *user_msr_list = argp;
+ struct kvm_msr_list msr_list;
+ unsigned int n;
+
+ r = -EFAULT;
+ if (copy_from_user(&msr_list, user_msr_list, sizeof(msr_list)))
+ goto out;
+ n = msr_list.nmsrs;
+ msr_list.nmsrs = num_msr_based_features;
+ if (copy_to_user(user_msr_list, &msr_list, sizeof(msr_list)))
+ goto out;
+ r = -E2BIG;
+ if (n < msr_list.nmsrs)
+ goto out;
+ r = -EFAULT;
+ if (copy_to_user(user_msr_list->indices, &msr_based_features,
+ num_msr_based_features * sizeof(u32)))
+ goto out;
+ r = 0;
+ break;
+ }
+ case KVM_GET_MSRS:
+ r = msr_io(NULL, argp, do_get_msr_feature, 1);
+ break;
}
default:
r = -EINVAL;
@@ -3450,12 +3544,18 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
r = 0;
break;
}
- case KVM_GET_MSRS:
+ case KVM_GET_MSRS: {
+ int idx = srcu_read_lock(&vcpu->kvm->srcu);
r = msr_io(vcpu, argp, do_get_msr, 1);
+ srcu_read_unlock(&vcpu->kvm->srcu, idx);
break;
- case KVM_SET_MSRS:
+ }
+ case KVM_SET_MSRS: {
+ int idx = srcu_read_lock(&vcpu->kvm->srcu);
r = msr_io(vcpu, argp, do_set_msr, 0);
+ srcu_read_unlock(&vcpu->kvm->srcu, idx);
break;
+ }
case KVM_TPR_ACCESS_REPORTING: {
struct kvm_tpr_access_ctl tac;
@@ -4130,13 +4230,14 @@ long kvm_arch_vm_ioctl(struct file *filp,
mutex_unlock(&kvm->lock);
break;
case KVM_XEN_HVM_CONFIG: {
+ struct kvm_xen_hvm_config xhc;
r = -EFAULT;
- if (copy_from_user(&kvm->arch.xen_hvm_config, argp,
- sizeof(struct kvm_xen_hvm_config)))
+ if (copy_from_user(&xhc, argp, sizeof(xhc)))
goto out;
r = -EINVAL;
- if (kvm->arch.xen_hvm_config.flags)
+ if (xhc.flags)
goto out;
+ memcpy(&kvm->arch.xen_hvm_config, &xhc, sizeof(xhc));
r = 0;
break;
}
@@ -4226,20 +4327,27 @@ static void kvm_init_msr_list(void)
num_msrs_to_save = j;
for (i = j = 0; i < ARRAY_SIZE(emulated_msrs); i++) {
- switch (emulated_msrs[i]) {
- case MSR_IA32_SMBASE:
- if (!kvm_x86_ops->cpu_has_high_real_mode_segbase())
- continue;
- break;
- default:
- break;
- }
+ if (!kvm_x86_ops->has_emulated_msr(emulated_msrs[i]))
+ continue;
if (j < i)
emulated_msrs[j] = emulated_msrs[i];
j++;
}
num_emulated_msrs = j;
+
+ for (i = j = 0; i < ARRAY_SIZE(msr_based_features); i++) {
+ struct kvm_msr_entry msr;
+
+ msr.index = msr_based_features[i];
+ if (kvm_get_msr_feature(&msr))
+ continue;
+
+ if (j < i)
+ msr_based_features[j] = msr_based_features[i];
+ j++;
+ }
+ num_msr_based_features = j;
}
static int vcpu_mmio_write(struct kvm_vcpu *vcpu, gpa_t addr, int len,
@@ -4399,11 +4507,10 @@ static int kvm_fetch_guest_virt(struct x86_emulate_ctxt *ctxt,
return X86EMUL_CONTINUE;
}
-int kvm_read_guest_virt(struct x86_emulate_ctxt *ctxt,
+int kvm_read_guest_virt(struct kvm_vcpu *vcpu,
gva_t addr, void *val, unsigned int bytes,
struct x86_exception *exception)
{
- struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
u32 access = (kvm_x86_ops->get_cpl(vcpu) == 3) ? PFERR_USER_MASK : 0;
return kvm_read_guest_virt_helper(addr, val, bytes, vcpu, access,
@@ -4411,12 +4518,17 @@ int kvm_read_guest_virt(struct x86_emulate_ctxt *ctxt,
}
EXPORT_SYMBOL_GPL(kvm_read_guest_virt);
-static int kvm_read_guest_virt_system(struct x86_emulate_ctxt *ctxt,
- gva_t addr, void *val, unsigned int bytes,
- struct x86_exception *exception)
+static int emulator_read_std(struct x86_emulate_ctxt *ctxt,
+ gva_t addr, void *val, unsigned int bytes,
+ struct x86_exception *exception, bool system)
{
struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
- return kvm_read_guest_virt_helper(addr, val, bytes, vcpu, 0, exception);
+ u32 access = 0;
+
+ if (!system && kvm_x86_ops->get_cpl(vcpu) == 3)
+ access |= PFERR_USER_MASK;
+
+ return kvm_read_guest_virt_helper(addr, val, bytes, vcpu, access, exception);
}
static int kvm_read_guest_phys_system(struct x86_emulate_ctxt *ctxt,
@@ -4428,18 +4540,16 @@ static int kvm_read_guest_phys_system(struct x86_emulate_ctxt *ctxt,
return r < 0 ? X86EMUL_IO_NEEDED : X86EMUL_CONTINUE;
}
-int kvm_write_guest_virt_system(struct x86_emulate_ctxt *ctxt,
- gva_t addr, void *val,
- unsigned int bytes,
- struct x86_exception *exception)
+static int kvm_write_guest_virt_helper(gva_t addr, void *val, unsigned int bytes,
+ struct kvm_vcpu *vcpu, u32 access,
+ struct x86_exception *exception)
{
- struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
void *data = val;
int r = X86EMUL_CONTINUE;
while (bytes) {
gpa_t gpa = vcpu->arch.walk_mmu->gva_to_gpa(vcpu, addr,
- PFERR_WRITE_MASK,
+ access,
exception);
unsigned offset = addr & (PAGE_SIZE-1);
unsigned towrite = min(bytes, (unsigned)PAGE_SIZE - offset);
@@ -4460,6 +4570,30 @@ int kvm_write_guest_virt_system(struct x86_emulate_ctxt *ctxt,
out:
return r;
}
+
+static int emulator_write_std(struct x86_emulate_ctxt *ctxt, gva_t addr, void *val,
+ unsigned int bytes, struct x86_exception *exception,
+ bool system)
+{
+ struct kvm_vcpu *vcpu = emul_to_vcpu(ctxt);
+ u32 access = PFERR_WRITE_MASK;
+
+ if (!system && kvm_x86_ops->get_cpl(vcpu) == 3)
+ access |= PFERR_USER_MASK;
+
+ return kvm_write_guest_virt_helper(addr, val, bytes, vcpu,
+ access, exception);
+}
+
+int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu, gva_t addr, void *val,
+ unsigned int bytes, struct x86_exception *exception)
+{
+ /* kvm_write_guest_virt_system can pull in tons of pages. */
+ vcpu->arch.l1tf_flush_l1d = true;
+
+ return kvm_write_guest_virt_helper(addr, val, bytes, vcpu,
+ PFERR_WRITE_MASK, exception);
+}
EXPORT_SYMBOL_GPL(kvm_write_guest_virt_system);
static int vcpu_mmio_gva_to_gpa(struct kvm_vcpu *vcpu, unsigned long gva,
@@ -5184,8 +5318,8 @@ static void emulator_set_hflags(struct x86_emulate_ctxt *ctxt, unsigned emul_fla
static const struct x86_emulate_ops emulate_ops = {
.read_gpr = emulator_read_gpr,
.write_gpr = emulator_write_gpr,
- .read_std = kvm_read_guest_virt_system,
- .write_std = kvm_write_guest_virt_system,
+ .read_std = emulator_read_std,
+ .write_std = emulator_write_std,
.read_phys = kvm_read_guest_phys_system,
.fetch = kvm_fetch_guest_virt,
.read_emulated = emulator_read_emulated,
@@ -5555,6 +5689,8 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu,
bool writeback = true;
bool write_fault_to_spt = vcpu->arch.write_fault_to_shadow_pgtable;
+ vcpu->arch.l1tf_flush_l1d = true;
+
/*
* Clear write_fault_to_shadow_pgtable here to ensure it is
* never reused.
@@ -5571,7 +5707,8 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu,
* handle watchpoints yet, those would be handled in
* the emulate_ops.
*/
- if (kvm_vcpu_check_breakpoint(vcpu, &r))
+ if (!(emulation_type & EMULTYPE_SKIP) &&
+ kvm_vcpu_check_breakpoint(vcpu, &r))
return r;
ctxt->interruptibility = 0;
@@ -6525,7 +6662,8 @@ static void vcpu_scan_ioapic(struct kvm_vcpu *vcpu)
else {
if (vcpu->arch.apicv_active)
kvm_x86_ops->sync_pir_to_irr(vcpu);
- kvm_ioapic_scan_entry(vcpu, vcpu->arch.ioapic_handled_vectors);
+ if (ioapic_in_kernel(vcpu->kvm))
+ kvm_ioapic_scan_entry(vcpu, vcpu->arch.ioapic_handled_vectors);
}
bitmap_or((ulong *)eoi_exit_bitmap, vcpu->arch.ioapic_handled_vectors,
vcpu_to_synic(vcpu)->vec_bitmap, 256);
@@ -6910,6 +7048,7 @@ static int vcpu_run(struct kvm_vcpu *vcpu)
struct kvm *kvm = vcpu->kvm;
vcpu->srcu_idx = srcu_read_lock(&kvm->srcu);
+ vcpu->arch.l1tf_flush_l1d = true;
for (;;) {
if (kvm_vcpu_running(vcpu)) {
@@ -7263,6 +7402,7 @@ int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu,
{
struct msr_data apic_base_msr;
int mmu_reset_needed = 0;
+ int cpuid_update_needed = 0;
int pending_vec, max_bits, idx;
struct desc_ptr dt;
@@ -7294,8 +7434,10 @@ int kvm_arch_vcpu_ioctl_set_sregs(struct kvm_vcpu *vcpu,
vcpu->arch.cr0 = sregs->cr0;
mmu_reset_needed |= kvm_read_cr4(vcpu) != sregs->cr4;
+ cpuid_update_needed |= ((kvm_read_cr4(vcpu) ^ sregs->cr4) &
+ (X86_CR4_OSXSAVE | X86_CR4_PKE));
kvm_x86_ops->set_cr4(vcpu, sregs->cr4);
- if (sregs->cr4 & (X86_CR4_OSXSAVE | X86_CR4_PKE))
+ if (cpuid_update_needed)
kvm_update_cpuid(vcpu);
idx = srcu_read_lock(&vcpu->kvm->srcu);
@@ -7491,16 +7633,6 @@ void kvm_put_guest_fpu(struct kvm_vcpu *vcpu)
copy_fpregs_to_fpstate(&vcpu->arch.guest_fpu);
__kernel_fpu_end();
++vcpu->stat.fpu_reload;
- /*
- * If using eager FPU mode, or if the guest is a frequent user
- * of the FPU, just leave the FPU active for next time.
- * Every 255 times fpu_counter rolls over to 0; a guest that uses
- * the FPU in bursts will revert to loading it on demand.
- */
- if (!use_eager_fpu()) {
- if (++vcpu->fpu_counter < 5)
- kvm_make_request(KVM_REQ_DEACTIVATE_FPU, vcpu);
- }
trace_kvm_fpu(0);
}
@@ -7877,6 +8009,7 @@ void kvm_arch_vcpu_uninit(struct kvm_vcpu *vcpu)
void kvm_arch_sched_in(struct kvm_vcpu *vcpu, int cpu)
{
+ vcpu->arch.l1tf_flush_l1d = true;
kvm_x86_ops->sched_in(vcpu, cpu);
}
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index e8ff3e4..2133a18 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -161,11 +161,11 @@ int kvm_inject_realmode_interrupt(struct kvm_vcpu *vcpu, int irq, int inc_eip);
void kvm_write_tsc(struct kvm_vcpu *vcpu, struct msr_data *msr);
u64 get_kvmclock_ns(struct kvm *kvm);
-int kvm_read_guest_virt(struct x86_emulate_ctxt *ctxt,
+int kvm_read_guest_virt(struct kvm_vcpu *vcpu,
gva_t addr, void *val, unsigned int bytes,
struct x86_exception *exception);
-int kvm_write_guest_virt_system(struct x86_emulate_ctxt *ctxt,
+int kvm_write_guest_virt_system(struct kvm_vcpu *vcpu,
gva_t addr, void *val, unsigned int bytes,
struct x86_exception *exception);
diff --git a/arch/x86/lib/msr-reg.S b/arch/x86/lib/msr-reg.S
index c815564..10ffa7e 100644
--- a/arch/x86/lib/msr-reg.S
+++ b/arch/x86/lib/msr-reg.S
@@ -13,14 +13,14 @@
.macro op_safe_regs op
ENTRY(\op\()_safe_regs)
pushq %rbx
- pushq %rbp
+ pushq %r12
movq %rdi, %r10 /* Save pointer */
xorl %r11d, %r11d /* Return value */
movl (%rdi), %eax
movl 4(%rdi), %ecx
movl 8(%rdi), %edx
movl 12(%rdi), %ebx
- movl 20(%rdi), %ebp
+ movl 20(%rdi), %r12d
movl 24(%rdi), %esi
movl 28(%rdi), %edi
1: \op
@@ -29,10 +29,10 @@
movl %ecx, 4(%r10)
movl %edx, 8(%r10)
movl %ebx, 12(%r10)
- movl %ebp, 20(%r10)
+ movl %r12d, 20(%r10)
movl %esi, 24(%r10)
movl %edi, 28(%r10)
- popq %rbp
+ popq %r12
popq %rbx
ret
3:
diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c
index 54efa85..0da3945 100644
--- a/arch/x86/mm/fault.c
+++ b/arch/x86/mm/fault.c
@@ -23,6 +23,7 @@
#include <asm/vsyscall.h> /* emulate_vsyscall */
#include <asm/vm86.h> /* struct vm86 */
#include <asm/mmu_context.h> /* vma_pkey() */
+#include <asm/sections.h>
#define CREATE_TRACE_POINTS
#include <asm/trace/exceptions.h>
@@ -329,8 +330,6 @@ static noinline int vmalloc_fault(unsigned long address)
if (!(address >= VMALLOC_START && address < VMALLOC_END))
return -1;
- WARN_ON_ONCE(in_nmi());
-
/*
* Synchronize this task's top level page-table
* with the 'reference' page table.
diff --git a/arch/x86/mm/init.c b/arch/x86/mm/init.c
index f92bdb9..90801a8 100644
--- a/arch/x86/mm/init.c
+++ b/arch/x86/mm/init.c
@@ -4,6 +4,8 @@
#include <linux/swap.h>
#include <linux/memblock.h>
#include <linux/bootmem.h> /* for max_low_pfn */
+#include <linux/swapfile.h>
+#include <linux/swapops.h>
#include <asm/cacheflush.h>
#include <asm/e820.h>
@@ -653,7 +655,9 @@ void __init init_mem_mapping(void)
*/
int devmem_is_allowed(unsigned long pagenr)
{
- if (page_is_ram(pagenr)) {
+ if (region_intersects(PFN_PHYS(pagenr), PAGE_SIZE,
+ IORESOURCE_SYSTEM_RAM, IORES_DESC_NONE)
+ != REGION_DISJOINT) {
/*
* For disallowed memory regions in the low 1MB range,
* request that the page be shown as all zeros.
@@ -778,3 +782,26 @@ void update_cache_mode_entry(unsigned entry, enum page_cache_mode cache)
__cachemode2pte_tbl[cache] = __cm_idx2pte(entry);
__pte2cachemode_tbl[entry] = cache;
}
+
+#ifdef CONFIG_SWAP
+unsigned long max_swapfile_size(void)
+{
+ unsigned long pages;
+
+ pages = generic_max_swapfile_size();
+
+ if (boot_cpu_has_bug(X86_BUG_L1TF)) {
+ /* Limit the swap file size to MAX_PA/2 for L1TF workaround */
+ unsigned long long l1tf_limit = l1tf_pfn_limit();
+ /*
+ * We encode swap offsets also with 3 bits below those for pfn
+ * which makes the usable limit higher.
+ */
+#if CONFIG_PGTABLE_LEVELS > 2
+ l1tf_limit <<= PAGE_SHIFT - SWP_OFFSET_FIRST_BIT;
+#endif
+ pages = min_t(unsigned long long, l1tf_limit, pages);
+ }
+ return pages;
+}
+#endif
diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c
index 7df8e3a..d35d0e4 100644
--- a/arch/x86/mm/init_64.c
+++ b/arch/x86/mm/init_64.c
@@ -1014,8 +1014,7 @@ void __init mem_init(void)
after_bootmem = 1;
/* Register memory areas for /proc/kcore */
- kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR,
- PAGE_SIZE, KCORE_OTHER);
+ kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR, PAGE_SIZE, KCORE_USER);
mem_init_print_info(NULL);
}
diff --git a/arch/x86/mm/kaiser.c b/arch/x86/mm/kaiser.c
index ec678aa..3f729e2 100644
--- a/arch/x86/mm/kaiser.c
+++ b/arch/x86/mm/kaiser.c
@@ -20,6 +20,7 @@
#include <asm/desc.h>
#include <asm/cmdline.h>
#include <asm/vsyscall.h>
+#include <asm/sections.h>
int kaiser_enabled __read_mostly = 1;
EXPORT_SYMBOL(kaiser_enabled); /* for inlined TLB flush functions */
diff --git a/arch/x86/mm/kmmio.c b/arch/x86/mm/kmmio.c
index cadb82b..c695272 100644
--- a/arch/x86/mm/kmmio.c
+++ b/arch/x86/mm/kmmio.c
@@ -125,24 +125,29 @@ static struct kmmio_fault_page *get_kmmio_fault_page(unsigned long addr)
static void clear_pmd_presence(pmd_t *pmd, bool clear, pmdval_t *old)
{
+ pmd_t new_pmd;
pmdval_t v = pmd_val(*pmd);
if (clear) {
- *old = v & _PAGE_PRESENT;
- v &= ~_PAGE_PRESENT;
- } else /* presume this has been called with clear==true previously */
- v |= *old;
- set_pmd(pmd, __pmd(v));
+ *old = v;
+ new_pmd = pmd_mknotpresent(*pmd);
+ } else {
+ /* Presume this has been called with clear==true previously */
+ new_pmd = __pmd(*old);
+ }
+ set_pmd(pmd, new_pmd);
}
static void clear_pte_presence(pte_t *pte, bool clear, pteval_t *old)
{
pteval_t v = pte_val(*pte);
if (clear) {
- *old = v & _PAGE_PRESENT;
- v &= ~_PAGE_PRESENT;
- } else /* presume this has been called with clear==true previously */
- v |= *old;
- set_pte_atomic(pte, __pte(v));
+ *old = v;
+ /* Nothing should care about address */
+ pte_clear(&init_mm, 0, pte);
+ } else {
+ /* Presume this has been called with clear==true previously */
+ set_pte_atomic(pte, __pte(*old));
+ }
}
static int clear_page_presence(struct kmmio_fault_page *f, bool clear)
diff --git a/arch/x86/mm/mmap.c b/arch/x86/mm/mmap.c
index d2dc043..74609a9 100644
--- a/arch/x86/mm/mmap.c
+++ b/arch/x86/mm/mmap.c
@@ -121,3 +121,24 @@ const char *arch_vma_name(struct vm_area_struct *vma)
return "[mpx]";
return NULL;
}
+
+/*
+ * Only allow root to set high MMIO mappings to PROT_NONE.
+ * This prevents an unpriv. user to set them to PROT_NONE and invert
+ * them, then pointing to valid memory for L1TF speculation.
+ *
+ * Note: for locked down kernels may want to disable the root override.
+ */
+bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot)
+{
+ if (!boot_cpu_has_bug(X86_BUG_L1TF))
+ return true;
+ if (!__pte_needs_invert(pgprot_val(prot)))
+ return true;
+ /* If it's real memory always allow */
+ if (pfn_valid(pfn))
+ return true;
+ if (pfn >= l1tf_pfn_limit() && !capable(CAP_SYS_ADMIN))
+ return false;
+ return true;
+}
diff --git a/arch/x86/mm/numa_emulation.c b/arch/x86/mm/numa_emulation.c
index a8f90ce..dc6d990 100644
--- a/arch/x86/mm/numa_emulation.c
+++ b/arch/x86/mm/numa_emulation.c
@@ -60,7 +60,7 @@ static int __init emu_setup_memblk(struct numa_meminfo *ei,
eb->nid = nid;
if (emu_nid_to_phys[nid] == NUMA_NO_NODE)
- emu_nid_to_phys[nid] = nid;
+ emu_nid_to_phys[nid] = pb->nid;
pb->start += size;
if (pb->start >= pb->end) {
diff --git a/arch/x86/mm/pageattr.c b/arch/x86/mm/pageattr.c
index 73dcb0e1..1271bc9 100644
--- a/arch/x86/mm/pageattr.c
+++ b/arch/x86/mm/pageattr.c
@@ -279,9 +279,11 @@ static inline pgprot_t static_protections(pgprot_t prot, unsigned long address,
/*
* The .rodata section needs to be read-only. Using the pfn
- * catches all aliases.
+ * catches all aliases. This also includes __ro_after_init,
+ * so do not enforce until kernel_set_to_readonly is true.
*/
- if (within(pfn, __pa_symbol(__start_rodata) >> PAGE_SHIFT,
+ if (kernel_set_to_readonly &&
+ within(pfn, __pa_symbol(__start_rodata) >> PAGE_SHIFT,
__pa_symbol(__end_rodata) >> PAGE_SHIFT))
pgprot_val(forbidden) |= _PAGE_RW;
@@ -999,8 +1001,8 @@ static long populate_pmd(struct cpa_data *cpa,
pmd = pmd_offset(pud, start);
- set_pmd(pmd, __pmd(cpa->pfn << PAGE_SHIFT | _PAGE_PSE |
- massage_pgprot(pmd_pgprot)));
+ set_pmd(pmd, pmd_mkhuge(pfn_pmd(cpa->pfn,
+ canon_pgprot(pmd_pgprot))));
start += PMD_SIZE;
cpa->pfn += PMD_SIZE >> PAGE_SHIFT;
@@ -1072,8 +1074,8 @@ static long populate_pud(struct cpa_data *cpa, unsigned long start, pgd_t *pgd,
* Map everything starting from the Gb boundary, possibly with 1G pages
*/
while (boot_cpu_has(X86_FEATURE_GBPAGES) && end - start >= PUD_SIZE) {
- set_pud(pud, __pud(cpa->pfn << PAGE_SHIFT | _PAGE_PSE |
- massage_pgprot(pud_pgprot)));
+ set_pud(pud, pud_mkhuge(pfn_pud(cpa->pfn,
+ canon_pgprot(pud_pgprot))));
start += PUD_SIZE;
cpa->pfn += PUD_SIZE >> PAGE_SHIFT;
diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index b97ef29..e30baa8 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -1,5 +1,6 @@
#include <linux/mm.h>
#include <linux/gfp.h>
+#include <linux/hugetlb.h>
#include <asm/pgalloc.h>
#include <asm/pgtable.h>
#include <asm/tlb.h>
@@ -577,6 +578,10 @@ int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot)
(mtrr != MTRR_TYPE_WRBACK))
return 0;
+ /* Bail out if we are we on a populated non-leaf entry: */
+ if (pud_present(*pud) && !pud_huge(*pud))
+ return 0;
+
prot = pgprot_4k_2_large(prot);
set_pte((pte_t *)pud, pfn_pte(
@@ -605,6 +610,10 @@ int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot)
return 0;
}
+ /* Bail out if we are we on a populated non-leaf entry: */
+ if (pmd_present(*pmd) && !pmd_huge(*pmd))
+ return 0;
+
prot = pgprot_4k_2_large(prot);
set_pte((pte_t *)pmd, pfn_pte(
@@ -644,28 +653,50 @@ int pmd_clear_huge(pmd_t *pmd)
return 0;
}
+#ifdef CONFIG_X86_64
/**
* pud_free_pmd_page - Clear pud entry and free pmd page.
* @pud: Pointer to a PUD.
+ * @addr: Virtual address associated with pud.
*
- * Context: The pud range has been unmaped and TLB purged.
+ * Context: The pud range has been unmapped and TLB purged.
* Return: 1 if clearing the entry succeeded. 0 otherwise.
+ *
+ * NOTE: Callers must allow a single page allocation.
*/
-int pud_free_pmd_page(pud_t *pud)
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
{
- pmd_t *pmd;
+ pmd_t *pmd, *pmd_sv;
+ pte_t *pte;
int i;
if (pud_none(*pud))
return 1;
pmd = (pmd_t *)pud_page_vaddr(*pud);
+ pmd_sv = (pmd_t *)__get_free_page(GFP_KERNEL);
+ if (!pmd_sv)
+ return 0;
- for (i = 0; i < PTRS_PER_PMD; i++)
- if (!pmd_free_pte_page(&pmd[i]))
- return 0;
+ for (i = 0; i < PTRS_PER_PMD; i++) {
+ pmd_sv[i] = pmd[i];
+ if (!pmd_none(pmd[i]))
+ pmd_clear(&pmd[i]);
+ }
pud_clear(pud);
+
+ /* INVLPG to clear all paging-structure caches */
+ flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1);
+
+ for (i = 0; i < PTRS_PER_PMD; i++) {
+ if (!pmd_none(pmd_sv[i])) {
+ pte = (pte_t *)pmd_page_vaddr(pmd_sv[i]);
+ free_page((unsigned long)pte);
+ }
+ }
+
+ free_page((unsigned long)pmd_sv);
free_page((unsigned long)pmd);
return 1;
@@ -674,11 +705,12 @@ int pud_free_pmd_page(pud_t *pud)
/**
* pmd_free_pte_page - Clear pmd entry and free pte page.
* @pmd: Pointer to a PMD.
+ * @addr: Virtual address associated with pmd.
*
- * Context: The pmd range has been unmaped and TLB purged.
+ * Context: The pmd range has been unmapped and TLB purged.
* Return: 1 if clearing the entry succeeded. 0 otherwise.
*/
-int pmd_free_pte_page(pmd_t *pmd)
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
{
pte_t *pte;
@@ -687,8 +719,30 @@ int pmd_free_pte_page(pmd_t *pmd)
pte = (pte_t *)pmd_page_vaddr(*pmd);
pmd_clear(pmd);
+
+ /* INVLPG to clear all paging-structure caches */
+ flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1);
+
free_page((unsigned long)pte);
return 1;
}
+
+#else /* !CONFIG_X86_64 */
+
+int pud_free_pmd_page(pud_t *pud, unsigned long addr)
+{
+ return pud_none(*pud);
+}
+
+/*
+ * Disable free page handling on x86-PAE. This assures that ioremap()
+ * does not update sync'd pmd entries. See vmalloc_sync_one().
+ */
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
+{
+ return pmd_none(*pmd);
+}
+
+#endif /* CONFIG_X86_64 */
#endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */
diff --git a/arch/x86/mm/pkeys.c b/arch/x86/mm/pkeys.c
index f88ce0e..e2d2b3c 100644
--- a/arch/x86/mm/pkeys.c
+++ b/arch/x86/mm/pkeys.c
@@ -95,26 +95,27 @@ int __arch_override_mprotect_pkey(struct vm_area_struct *vma, int prot, int pkey
*/
if (pkey != -1)
return pkey;
- /*
- * Look for a protection-key-drive execute-only mapping
- * which is now being given permissions that are not
- * execute-only. Move it back to the default pkey.
- */
- if (vma_is_pkey_exec_only(vma) &&
- (prot & (PROT_READ|PROT_WRITE))) {
- return 0;
- }
+
/*
* The mapping is execute-only. Go try to get the
* execute-only protection key. If we fail to do that,
* fall through as if we do not have execute-only
- * support.
+ * support in this mm.
*/
if (prot == PROT_EXEC) {
pkey = execute_only_pkey(vma->vm_mm);
if (pkey > 0)
return pkey;
+ } else if (vma_is_pkey_exec_only(vma)) {
+ /*
+ * Protections are *not* PROT_EXEC, but the mapping
+ * is using the exec-only pkey. This mapping was
+ * PROT_EXEC and will no longer be. Move back to
+ * the default pkey.
+ */
+ return ARCH_DEFAULT_PKEY;
}
+
/*
* This is a vanilla, non-pkey mprotect (or we failed to
* setup execute-only), inherit the pkey from the VMA we
@@ -141,8 +142,7 @@ u32 init_pkru_value = PKRU_AD_KEY( 1) | PKRU_AD_KEY( 2) | PKRU_AD_KEY( 3) |
* Called from the FPU code when creating a fresh set of FPU
* registers. This is called from a very specific context where
* we know the FPU regstiers are safe for use and we can use PKRU
- * directly. The fact that PKRU is only available when we are
- * using eagerfpu mode makes this possible.
+ * directly.
*/
void copy_init_pkru_to_fpregs(void)
{
diff --git a/arch/x86/net/Makefile b/arch/x86/net/Makefile
index 90568c3..fefb4b6 100644
--- a/arch/x86/net/Makefile
+++ b/arch/x86/net/Makefile
@@ -1,4 +1,6 @@
#
# Arch-specific network modules
#
+OBJECT_FILES_NON_STANDARD_bpf_jit.o += y
+
obj-$(CONFIG_BPF_JIT) += bpf_jit.o bpf_jit_comp.o
diff --git a/arch/x86/platform/efi/Makefile b/arch/x86/platform/efi/Makefile
index 066619b..7a25502 100644
--- a/arch/x86/platform/efi/Makefile
+++ b/arch/x86/platform/efi/Makefile
@@ -1,4 +1,5 @@
OBJECT_FILES_NON_STANDARD_efi_thunk_$(BITS).o := y
+OBJECT_FILES_NON_STANDARD_efi_stub_$(BITS).o := y
obj-$(CONFIG_EFI) += quirks.o efi.o efi_$(BITS).o efi_stub_$(BITS).o
obj-$(CONFIG_ACPI_BGRT) += efi-bgrt.o
diff --git a/arch/x86/platform/efi/early_printk.c b/arch/x86/platform/efi/early_printk.c
index 5fdacb3..c3e6be1 100644
--- a/arch/x86/platform/efi/early_printk.c
+++ b/arch/x86/platform/efi/early_printk.c
@@ -179,7 +179,7 @@ early_efi_write(struct console *con, const char *str, unsigned int num)
num--;
}
- if (efi_x >= si->lfb_width) {
+ if (efi_x + font->width > si->lfb_width) {
efi_x = 0;
efi_y += font->height;
}
diff --git a/arch/x86/platform/efi/efi_64.c b/arch/x86/platform/efi/efi_64.c
index dcb2d9d..351a55d 100644
--- a/arch/x86/platform/efi/efi_64.c
+++ b/arch/x86/platform/efi/efi_64.c
@@ -45,6 +45,7 @@
#include <asm/realmode.h>
#include <asm/time.h>
#include <asm/pgalloc.h>
+#include <asm/sections.h>
/*
* We allocate runtime services regions bottom-up, starting from -4G, i.e.
diff --git a/arch/x86/platform/efi/quirks.c b/arch/x86/platform/efi/quirks.c
index 393a0c0..dee9939 100644
--- a/arch/x86/platform/efi/quirks.c
+++ b/arch/x86/platform/efi/quirks.c
@@ -13,6 +13,7 @@
#include <linux/dmi.h>
#include <asm/efi.h>
#include <asm/uv/uv.h>
+#include <asm/sections.h>
#define EFI_MIN_RESERVE 5120
diff --git a/arch/x86/platform/intel-mid/device_libs/platform_mrfld_wdt.c b/arch/x86/platform/intel-mid/device_libs/platform_mrfld_wdt.c
index 10bad1e..85e112e 100644
--- a/arch/x86/platform/intel-mid/device_libs/platform_mrfld_wdt.c
+++ b/arch/x86/platform/intel-mid/device_libs/platform_mrfld_wdt.c
@@ -18,6 +18,7 @@
#include <asm/intel-mid.h>
#include <asm/intel_scu_ipc.h>
#include <asm/io_apic.h>
+#include <asm/hw_irq.h>
#define TANGIER_EXT_TIMER0_MSI 12
diff --git a/arch/x86/platform/olpc/olpc-xo1-rtc.c b/arch/x86/platform/olpc/olpc-xo1-rtc.c
index a2b4efd..8e7ddd7 100644
--- a/arch/x86/platform/olpc/olpc-xo1-rtc.c
+++ b/arch/x86/platform/olpc/olpc-xo1-rtc.c
@@ -16,6 +16,7 @@
#include <asm/msr.h>
#include <asm/olpc.h>
+#include <asm/x86_init.h>
static void rtc_wake_on(struct device *dev)
{
@@ -75,6 +76,8 @@ static int __init xo1_rtc_init(void)
if (r)
return r;
+ x86_platform.legacy.rtc = 0;
+
device_init_wakeup(&xo1_rtc_device.dev, 1);
return 0;
}
diff --git a/arch/x86/platform/uv/tlb_uv.c b/arch/x86/platform/uv/tlb_uv.c
index 0f017518..16d4967 100644
--- a/arch/x86/platform/uv/tlb_uv.c
+++ b/arch/x86/platform/uv/tlb_uv.c
@@ -1283,6 +1283,7 @@ void uv_bau_message_interrupt(struct pt_regs *regs)
struct msg_desc msgdesc;
ack_APIC_irq();
+ kvm_set_cpu_l1tf_flush_l1d();
time_start = get_cycles();
bcp = &per_cpu(bau_control, smp_processor_id());
diff --git a/arch/x86/power/Makefile b/arch/x86/power/Makefile
index a6a198c..0504187 100644
--- a/arch/x86/power/Makefile
+++ b/arch/x86/power/Makefile
@@ -1,3 +1,5 @@
+OBJECT_FILES_NON_STANDARD_hibernate_asm_$(BITS).o := y
+
# __restore_processor_state() restores %gs after S3 resume and so should not
# itself be stack-protected
nostackp := $(call cc-option, -fno-stack-protector)
diff --git a/arch/x86/power/hibernate_32.c b/arch/x86/power/hibernate_32.c
index 9f14bd3..74b516c 100644
--- a/arch/x86/power/hibernate_32.c
+++ b/arch/x86/power/hibernate_32.c
@@ -142,7 +142,7 @@ static inline void resume_init_first_level_page_table(pgd_t *pg_dir)
#endif
}
-int swsusp_arch_resume(void)
+asmlinkage int swsusp_arch_resume(void)
{
int error;
diff --git a/arch/x86/power/hibernate_64.c b/arch/x86/power/hibernate_64.c
index 9634557..fef485b 100644
--- a/arch/x86/power/hibernate_64.c
+++ b/arch/x86/power/hibernate_64.c
@@ -126,7 +126,7 @@ static int relocate_restore_code(void)
if (!relocated_restore_code)
return -ENOMEM;
- memcpy((void *)relocated_restore_code, &core_restore_code, PAGE_SIZE);
+ memcpy((void *)relocated_restore_code, core_restore_code, PAGE_SIZE);
/* Make the page containing the relocated code executable */
pgd = (pgd_t *)__va(read_cr3()) + pgd_index(relocated_restore_code);
@@ -149,7 +149,7 @@ static int relocate_restore_code(void)
return 0;
}
-int swsusp_arch_resume(void)
+asmlinkage int swsusp_arch_resume(void)
{
int error;
@@ -197,8 +197,8 @@ int arch_hibernation_header_save(void *addr, unsigned int max_size)
if (max_size < sizeof(struct restore_data_record))
return -EOVERFLOW;
- rdr->jump_address = (unsigned long)&restore_registers;
- rdr->jump_address_phys = __pa_symbol(&restore_registers);
+ rdr->jump_address = (unsigned long)restore_registers;
+ rdr->jump_address_phys = __pa_symbol(restore_registers);
rdr->cr3 = restore_cr3;
rdr->magic = RESTORE_MAGIC;
return 0;
diff --git a/arch/x86/xen/Makefile b/arch/x86/xen/Makefile
index e47e527..4a54059 100644
--- a/arch/x86/xen/Makefile
+++ b/arch/x86/xen/Makefile
@@ -1,3 +1,6 @@
+OBJECT_FILES_NON_STANDARD_xen-asm_$(BITS).o := y
+OBJECT_FILES_NON_STANDARD_xen-pvh.o := y
+
ifdef CONFIG_FUNCTION_TRACER
# Do not profile debug and lowlevel utilities
CFLAGS_REMOVE_spinlock.o = -pg
diff --git a/arch/x86/xen/enlighten.c b/arch/x86/xen/enlighten.c
index 2bea87c..db7cf87 100644
--- a/arch/x86/xen/enlighten.c
+++ b/arch/x86/xen/enlighten.c
@@ -35,6 +35,7 @@
#include <linux/frame.h>
#include <linux/kexec.h>
+#include <linux/slab.h>
#include <xen/xen.h>
#include <xen/events.h>
@@ -75,6 +76,7 @@
#include <asm/mwait.h>
#include <asm/pci_x86.h>
#include <asm/cpu.h>
+#include <asm/unwind_hints.h>
#ifdef CONFIG_ACPI
#include <linux/acpi.h>
@@ -471,6 +473,12 @@ static void __init xen_init_cpuid_mask(void)
cpuid_leaf1_ecx_set_mask = (1 << (X86_FEATURE_MWAIT % 32));
}
+static void __init xen_init_capabilities(void)
+{
+ if (xen_pv_domain())
+ setup_force_cpu_cap(X86_FEATURE_XENPV);
+}
+
static void xen_set_debugreg(int reg, unsigned long val)
{
HYPERVISOR_set_debugreg(reg, val);
@@ -1452,10 +1460,12 @@ static void __ref xen_setup_gdt(int cpu)
* GDT. The new GDT has __KERNEL_CS with CS.L = 1
* and we are jumping to reload it.
*/
- asm volatile ("pushq %0\n"
+ asm volatile (UNWIND_HINT_SAVE
+ "pushq %0\n"
"leaq 1f(%%rip),%0\n"
"pushq %0\n"
"lretq\n"
+ UNWIND_HINT_RESTORE
"1:\n"
: "=&r" (dummy) : "0" (__KERNEL_CS));
@@ -1631,6 +1641,7 @@ asmlinkage __visible void __init xen_start_kernel(void)
xen_init_irq_ops();
xen_init_cpuid_mask();
+ xen_init_capabilities();
#ifdef CONFIG_X86_LOCAL_APIC
/*
@@ -1975,14 +1986,6 @@ bool xen_hvm_need_lapic(void)
}
EXPORT_SYMBOL_GPL(xen_hvm_need_lapic);
-static void xen_set_cpu_features(struct cpuinfo_x86 *c)
-{
- if (xen_pv_domain()) {
- clear_cpu_bug(c, X86_BUG_SYSRET_SS_ATTRS);
- set_cpu_cap(c, X86_FEATURE_XENPV);
- }
-}
-
static void xen_pin_vcpu(int cpu)
{
static bool disable_pinning;
@@ -2029,7 +2032,6 @@ const struct hypervisor_x86 x86_hyper_xen = {
.init_platform = xen_hvm_guest_init,
#endif
.x2apic_available = xen_x2apic_para_available,
- .set_cpu_features = xen_set_cpu_features,
.pin_vcpu = xen_pin_vcpu,
};
EXPORT_SYMBOL(x86_hyper_xen);
diff --git a/arch/x86/xen/mmu.c b/arch/x86/xen/mmu.c
index 418f1b8..c92f75f 100644
--- a/arch/x86/xen/mmu.c
+++ b/arch/x86/xen/mmu.c
@@ -1317,8 +1317,6 @@ void xen_flush_tlb_all(void)
struct mmuext_op *op;
struct multicall_space mcs;
- trace_xen_mmu_flush_tlb_all(0);
-
preempt_disable();
mcs = xen_mc_entry(sizeof(*op));
@@ -1336,8 +1334,6 @@ static void xen_flush_tlb(void)
struct mmuext_op *op;
struct multicall_space mcs;
- trace_xen_mmu_flush_tlb(0);
-
preempt_disable();
mcs = xen_mc_entry(sizeof(*op));
diff --git a/arch/x86/xen/pmu.c b/arch/x86/xen/pmu.c
index b9fc525..0b29a43 100644
--- a/arch/x86/xen/pmu.c
+++ b/arch/x86/xen/pmu.c
@@ -477,7 +477,7 @@ static void xen_convert_regs(const struct xen_pmu_regs *xen_regs,
irqreturn_t xen_pmu_irq_handler(int irq, void *dev_id)
{
int err, ret = IRQ_NONE;
- struct pt_regs regs;
+ struct pt_regs regs = {0};
const struct xen_pmu_data *xenpmu_data = get_xenpmu_data();
uint8_t xenpmu_flags = get_xenpmu_flags();
diff --git a/arch/x86/xen/setup.c b/arch/x86/xen/setup.c
index 9f21b0c..36bfafb 100644
--- a/arch/x86/xen/setup.c
+++ b/arch/x86/xen/setup.c
@@ -18,6 +18,7 @@
#include <asm/setup.h>
#include <asm/acpi.h>
#include <asm/numa.h>
+#include <asm/sections.h>
#include <asm/xen/hypervisor.h>
#include <asm/xen/hypercall.h>
diff --git a/arch/x86/xen/smp.c b/arch/x86/xen/smp.c
index a11540e..8eca26e 100644
--- a/arch/x86/xen/smp.c
+++ b/arch/x86/xen/smp.c
@@ -28,6 +28,7 @@
#include <xen/interface/vcpu.h>
#include <xen/interface/xenpmu.h>
+#include <asm/spec-ctrl.h>
#include <asm/xen/interface.h>
#include <asm/xen/hypercall.h>
@@ -87,6 +88,8 @@ static void cpu_bringup(void)
cpu_data(cpu).x86_max_cores = 1;
set_cpu_sibling_map(cpu);
+ speculative_store_bypass_ht_init();
+
xen_setup_cpu_clockevents();
notify_cpu_starting(cpu);
@@ -375,6 +378,8 @@ static void __init xen_smp_prepare_cpus(unsigned int max_cpus)
}
set_cpu_sibling_map(0);
+ speculative_store_bypass_ht_init();
+
xen_pmu_init(0);
if (xen_smp_intr_init(0))
diff --git a/arch/x86/xen/spinlock.c b/arch/x86/xen/spinlock.c
index 3d6e006..8d2c6f0 100644
--- a/arch/x86/xen/spinlock.c
+++ b/arch/x86/xen/spinlock.c
@@ -8,6 +8,7 @@
#include <linux/log2.h>
#include <linux/gfp.h>
#include <linux/slab.h>
+#include <linux/atomic.h>
#include <asm/paravirt.h>
@@ -19,6 +20,7 @@
static DEFINE_PER_CPU(int, lock_kicker_irq) = -1;
static DEFINE_PER_CPU(char *, irq_name);
+static DEFINE_PER_CPU(atomic_t, xen_qlock_wait_nest);
static bool xen_pvspin = true;
#include <asm/qspinlock.h>
@@ -40,33 +42,24 @@ static void xen_qlock_kick(int cpu)
static void xen_qlock_wait(u8 *byte, u8 val)
{
int irq = __this_cpu_read(lock_kicker_irq);
+ atomic_t *nest_cnt = this_cpu_ptr(&xen_qlock_wait_nest);
/* If kicker interrupts not initialized yet, just spin */
- if (irq == -1)
+ if (irq == -1 || in_nmi())
return;
- /* clear pending */
- xen_clear_irq_pending(irq);
- barrier();
+ /* Detect reentry. */
+ atomic_inc(nest_cnt);
- /*
- * We check the byte value after clearing pending IRQ to make sure
- * that we won't miss a wakeup event because of the clearing.
- *
- * The sync_clear_bit() call in xen_clear_irq_pending() is atomic.
- * So it is effectively a memory barrier for x86.
- */
- if (READ_ONCE(*byte) != val)
- return;
+ /* If irq pending already and no nested call clear it. */
+ if (atomic_read(nest_cnt) == 1 && xen_test_irq_pending(irq)) {
+ xen_clear_irq_pending(irq);
+ } else if (READ_ONCE(*byte) == val) {
+ /* Block until irq becomes pending (or a spurious wakeup) */
+ xen_poll_irq(irq);
+ }
- /*
- * If an interrupt happens here, it will leave the wakeup irq
- * pending, which will cause xen_poll_irq() to return
- * immediately.
- */
-
- /* Block until irq becomes pending (or perhaps a spurious wakeup) */
- xen_poll_irq(irq);
+ atomic_dec(nest_cnt);
}
static irqreturn_t dummy_handler(int irq, void *dev_id)
diff --git a/arch/xtensa/boot/Makefile b/arch/xtensa/boot/Makefile
index ca20a89..6c6877d 100644
--- a/arch/xtensa/boot/Makefile
+++ b/arch/xtensa/boot/Makefile
@@ -31,7 +31,7 @@
$(addprefix $(obj)/,$(host-progs))
$(Q)$(MAKE) $(build)=$(obj)/$@ $(MAKECMDGOALS)
-OBJCOPYFLAGS = --strip-all -R .comment -R .note.gnu.build-id -O binary
+OBJCOPYFLAGS = --strip-all -R .comment -R .notes -O binary
vmlinux.bin: vmlinux FORCE
$(call if_changed,objcopy)
diff --git a/arch/xtensa/include/asm/cacheasm.h b/arch/xtensa/include/asm/cacheasm.h
index 2041abb..34545ec 100644
--- a/arch/xtensa/include/asm/cacheasm.h
+++ b/arch/xtensa/include/asm/cacheasm.h
@@ -31,16 +31,32 @@
*
*/
- .macro __loop_cache_all ar at insn size line_width
+
+ .macro __loop_cache_unroll ar at insn size line_width max_immed
+
+ .if (1 << (\line_width)) > (\max_immed)
+ .set _reps, 1
+ .elseif (2 << (\line_width)) > (\max_immed)
+ .set _reps, 2
+ .else
+ .set _reps, 4
+ .endif
+
+ __loopi \ar, \at, \size, (_reps << (\line_width))
+ .set _index, 0
+ .rep _reps
+ \insn \ar, _index << (\line_width)
+ .set _index, _index + 1
+ .endr
+ __endla \ar, \at, _reps << (\line_width)
+
+ .endm
+
+
+ .macro __loop_cache_all ar at insn size line_width max_immed
movi \ar, 0
-
- __loopi \ar, \at, \size, (4 << (\line_width))
- \insn \ar, 0 << (\line_width)
- \insn \ar, 1 << (\line_width)
- \insn \ar, 2 << (\line_width)
- \insn \ar, 3 << (\line_width)
- __endla \ar, \at, 4 << (\line_width)
+ __loop_cache_unroll \ar, \at, \insn, \size, \line_width, \max_immed
.endm
@@ -57,14 +73,9 @@
.endm
- .macro __loop_cache_page ar at insn line_width
+ .macro __loop_cache_page ar at insn line_width max_immed
- __loopi \ar, \at, PAGE_SIZE, 4 << (\line_width)
- \insn \ar, 0 << (\line_width)
- \insn \ar, 1 << (\line_width)
- \insn \ar, 2 << (\line_width)
- \insn \ar, 3 << (\line_width)
- __endla \ar, \at, 4 << (\line_width)
+ __loop_cache_unroll \ar, \at, \insn, PAGE_SIZE, \line_width, \max_immed
.endm
@@ -72,7 +83,8 @@
.macro ___unlock_dcache_all ar at
#if XCHAL_DCACHE_LINE_LOCKABLE && XCHAL_DCACHE_SIZE
- __loop_cache_all \ar \at diu XCHAL_DCACHE_SIZE XCHAL_DCACHE_LINEWIDTH
+ __loop_cache_all \ar \at diu XCHAL_DCACHE_SIZE \
+ XCHAL_DCACHE_LINEWIDTH 240
#endif
.endm
@@ -81,7 +93,8 @@
.macro ___unlock_icache_all ar at
#if XCHAL_ICACHE_LINE_LOCKABLE && XCHAL_ICACHE_SIZE
- __loop_cache_all \ar \at iiu XCHAL_ICACHE_SIZE XCHAL_ICACHE_LINEWIDTH
+ __loop_cache_all \ar \at iiu XCHAL_ICACHE_SIZE \
+ XCHAL_ICACHE_LINEWIDTH 240
#endif
.endm
@@ -90,7 +103,8 @@
.macro ___flush_invalidate_dcache_all ar at
#if XCHAL_DCACHE_SIZE
- __loop_cache_all \ar \at diwbi XCHAL_DCACHE_SIZE XCHAL_DCACHE_LINEWIDTH
+ __loop_cache_all \ar \at diwbi XCHAL_DCACHE_SIZE \
+ XCHAL_DCACHE_LINEWIDTH 240
#endif
.endm
@@ -99,7 +113,8 @@
.macro ___flush_dcache_all ar at
#if XCHAL_DCACHE_SIZE
- __loop_cache_all \ar \at diwb XCHAL_DCACHE_SIZE XCHAL_DCACHE_LINEWIDTH
+ __loop_cache_all \ar \at diwb XCHAL_DCACHE_SIZE \
+ XCHAL_DCACHE_LINEWIDTH 240
#endif
.endm
@@ -108,8 +123,8 @@
.macro ___invalidate_dcache_all ar at
#if XCHAL_DCACHE_SIZE
- __loop_cache_all \ar \at dii __stringify(DCACHE_WAY_SIZE) \
- XCHAL_DCACHE_LINEWIDTH
+ __loop_cache_all \ar \at dii XCHAL_DCACHE_SIZE \
+ XCHAL_DCACHE_LINEWIDTH 1020
#endif
.endm
@@ -118,8 +133,8 @@
.macro ___invalidate_icache_all ar at
#if XCHAL_ICACHE_SIZE
- __loop_cache_all \ar \at iii __stringify(ICACHE_WAY_SIZE) \
- XCHAL_ICACHE_LINEWIDTH
+ __loop_cache_all \ar \at iii XCHAL_ICACHE_SIZE \
+ XCHAL_ICACHE_LINEWIDTH 1020
#endif
.endm
@@ -166,7 +181,7 @@
.macro ___flush_invalidate_dcache_page ar as
#if XCHAL_DCACHE_SIZE
- __loop_cache_page \ar \as dhwbi XCHAL_DCACHE_LINEWIDTH
+ __loop_cache_page \ar \as dhwbi XCHAL_DCACHE_LINEWIDTH 1020
#endif
.endm
@@ -175,7 +190,7 @@
.macro ___flush_dcache_page ar as
#if XCHAL_DCACHE_SIZE
- __loop_cache_page \ar \as dhwb XCHAL_DCACHE_LINEWIDTH
+ __loop_cache_page \ar \as dhwb XCHAL_DCACHE_LINEWIDTH 1020
#endif
.endm
@@ -184,7 +199,7 @@
.macro ___invalidate_dcache_page ar as
#if XCHAL_DCACHE_SIZE
- __loop_cache_page \ar \as dhi XCHAL_DCACHE_LINEWIDTH
+ __loop_cache_page \ar \as dhi XCHAL_DCACHE_LINEWIDTH 1020
#endif
.endm
@@ -193,7 +208,7 @@
.macro ___invalidate_icache_page ar as
#if XCHAL_ICACHE_SIZE
- __loop_cache_page \ar \as ihi XCHAL_ICACHE_LINEWIDTH
+ __loop_cache_page \ar \as ihi XCHAL_ICACHE_LINEWIDTH 1020
#endif
.endm
diff --git a/arch/xtensa/include/asm/futex.h b/arch/xtensa/include/asm/futex.h
index 72bfc1c..5bfbc1c 100644
--- a/arch/xtensa/include/asm/futex.h
+++ b/arch/xtensa/include/asm/futex.h
@@ -44,18 +44,10 @@
: "r" (uaddr), "I" (-EFAULT), "r" (oparg) \
: "memory")
-static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+static inline int arch_futex_atomic_op_inuser(int op, int oparg, int *oval,
+ u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
#if !XCHAL_HAVE_S32C1I
return -ENOSYS;
@@ -89,19 +81,10 @@ static inline int futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (ret)
- return ret;
+ if (!ret)
+ *oval = oldval;
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: return (oldval == cmparg);
- case FUTEX_OP_CMP_NE: return (oldval != cmparg);
- case FUTEX_OP_CMP_LT: return (oldval < cmparg);
- case FUTEX_OP_CMP_GE: return (oldval >= cmparg);
- case FUTEX_OP_CMP_LE: return (oldval <= cmparg);
- case FUTEX_OP_CMP_GT: return (oldval > cmparg);
- }
-
- return -ENOSYS;
+ return ret;
}
static inline int
diff --git a/arch/xtensa/include/asm/processor.h b/arch/xtensa/include/asm/processor.h
index b42d68b..521c1e7 100644
--- a/arch/xtensa/include/asm/processor.h
+++ b/arch/xtensa/include/asm/processor.h
@@ -24,7 +24,11 @@
# error Linux requires the Xtensa Windowed Registers Option.
#endif
-#define ARCH_SLAB_MINALIGN XCHAL_DATA_WIDTH
+/* Xtensa ABI requires stack alignment to be at least 16 */
+
+#define STACK_ALIGN (XCHAL_DATA_WIDTH > 16 ? XCHAL_DATA_WIDTH : 16)
+
+#define ARCH_SLAB_MINALIGN STACK_ALIGN
/*
* User space process size: 1 GB.
diff --git a/arch/xtensa/kernel/asm-offsets.c b/arch/xtensa/kernel/asm-offsets.c
index 8e10e35..f1af06b 100644
--- a/arch/xtensa/kernel/asm-offsets.c
+++ b/arch/xtensa/kernel/asm-offsets.c
@@ -91,14 +91,14 @@ int main(void)
DEFINE(THREAD_SP, offsetof (struct task_struct, thread.sp));
DEFINE(THREAD_CPENABLE, offsetof (struct thread_info, cpenable));
#if XTENSA_HAVE_COPROCESSORS
- DEFINE(THREAD_XTREGS_CP0, offsetof (struct thread_info, xtregs_cp));
- DEFINE(THREAD_XTREGS_CP1, offsetof (struct thread_info, xtregs_cp));
- DEFINE(THREAD_XTREGS_CP2, offsetof (struct thread_info, xtregs_cp));
- DEFINE(THREAD_XTREGS_CP3, offsetof (struct thread_info, xtregs_cp));
- DEFINE(THREAD_XTREGS_CP4, offsetof (struct thread_info, xtregs_cp));
- DEFINE(THREAD_XTREGS_CP5, offsetof (struct thread_info, xtregs_cp));
- DEFINE(THREAD_XTREGS_CP6, offsetof (struct thread_info, xtregs_cp));
- DEFINE(THREAD_XTREGS_CP7, offsetof (struct thread_info, xtregs_cp));
+ DEFINE(THREAD_XTREGS_CP0, offsetof(struct thread_info, xtregs_cp.cp0));
+ DEFINE(THREAD_XTREGS_CP1, offsetof(struct thread_info, xtregs_cp.cp1));
+ DEFINE(THREAD_XTREGS_CP2, offsetof(struct thread_info, xtregs_cp.cp2));
+ DEFINE(THREAD_XTREGS_CP3, offsetof(struct thread_info, xtregs_cp.cp3));
+ DEFINE(THREAD_XTREGS_CP4, offsetof(struct thread_info, xtregs_cp.cp4));
+ DEFINE(THREAD_XTREGS_CP5, offsetof(struct thread_info, xtregs_cp.cp5));
+ DEFINE(THREAD_XTREGS_CP6, offsetof(struct thread_info, xtregs_cp.cp6));
+ DEFINE(THREAD_XTREGS_CP7, offsetof(struct thread_info, xtregs_cp.cp7));
#endif
DEFINE(THREAD_XTREGS_USER, offsetof (struct thread_info, xtregs_user));
DEFINE(XTREGS_USER_SIZE, sizeof(xtregs_user_t));
diff --git a/arch/xtensa/kernel/head.S b/arch/xtensa/kernel/head.S
index 23ce62e..27c8e07 100644
--- a/arch/xtensa/kernel/head.S
+++ b/arch/xtensa/kernel/head.S
@@ -88,9 +88,12 @@
initialize_mmu
#if defined(CONFIG_MMU) && XCHAL_HAVE_PTP_MMU && XCHAL_HAVE_SPANNING_WAY
rsr a2, excsave1
- movi a3, 0x08000000
+ movi a3, XCHAL_KSEG_PADDR
+ bltu a2, a3, 1f
+ sub a2, a2, a3
+ movi a3, XCHAL_KSEG_SIZE
bgeu a2, a3, 1f
- movi a3, 0xd0000000
+ movi a3, XCHAL_KSEG_CACHED_VADDR
add a2, a2, a3
wsr a2, excsave1
1:
diff --git a/arch/xtensa/kernel/process.c b/arch/xtensa/kernel/process.c
index e0ded48..570307c 100644
--- a/arch/xtensa/kernel/process.c
+++ b/arch/xtensa/kernel/process.c
@@ -85,18 +85,21 @@ void coprocessor_release_all(struct thread_info *ti)
void coprocessor_flush_all(struct thread_info *ti)
{
- unsigned long cpenable;
+ unsigned long cpenable, old_cpenable;
int i;
preempt_disable();
+ RSR_CPENABLE(old_cpenable);
cpenable = ti->cpenable;
+ WSR_CPENABLE(cpenable);
for (i = 0; i < XCHAL_CP_MAX; i++) {
if ((cpenable & 1) != 0 && coprocessor_owner[i] == ti)
coprocessor_flush(ti, i);
cpenable >>= 1;
}
+ WSR_CPENABLE(old_cpenable);
preempt_enable();
}
diff --git a/arch/xtensa/kernel/traps.c b/arch/xtensa/kernel/traps.c
index ce37d5b..44bd9a37 100644
--- a/arch/xtensa/kernel/traps.c
+++ b/arch/xtensa/kernel/traps.c
@@ -334,7 +334,7 @@ do_unaligned_user (struct pt_regs *regs)
info.si_errno = 0;
info.si_code = BUS_ADRALN;
info.si_addr = (void *) regs->excvaddr;
- force_sig_info(SIGSEGV, &info, current);
+ force_sig_info(SIGBUS, &info, current);
}
#endif
diff --git a/arch/xtensa/kernel/vmlinux.lds.S b/arch/xtensa/kernel/vmlinux.lds.S
index 31411fc..e8358ea0 100644
--- a/arch/xtensa/kernel/vmlinux.lds.S
+++ b/arch/xtensa/kernel/vmlinux.lds.S
@@ -109,6 +109,7 @@
.fixup : { *(.fixup) }
EXCEPTION_TABLE(16)
+ NOTES
/* Data section */
_sdata = .;
diff --git a/block/bio.c b/block/bio.c
index a4c8c99..94f0be8 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -155,7 +155,7 @@ static void bio_put_slab(struct bio_set *bs)
unsigned int bvec_nr_vecs(unsigned short idx)
{
- return bvec_slabs[idx].nr_vecs;
+ return bvec_slabs[--idx].nr_vecs;
}
void bvec_free(mempool_t *pool, struct bio_vec *bv, unsigned int idx)
@@ -568,8 +568,11 @@ EXPORT_SYMBOL(bio_phys_segments);
static inline void bio_clone_crypt_key(struct bio *dst, const struct bio *src)
{
#ifdef CONFIG_PFK
- dst->bi_crypt_key = src->bi_crypt_key;
dst->bi_iter.bi_dun = src->bi_iter.bi_dun;
+#ifdef CONFIG_DM_DEFAULT_KEY
+ dst->bi_crypt_key = src->bi_crypt_key;
+ dst->bi_crypt_skip = src->bi_crypt_skip;
+#endif
dst->bi_dio_inode = src->bi_dio_inode;
#endif
}
@@ -598,10 +601,9 @@ void __bio_clone_fast(struct bio *bio, struct bio *bio_src)
bio->bi_opf = bio_src->bi_opf;
bio->bi_iter = bio_src->bi_iter;
bio->bi_io_vec = bio_src->bi_io_vec;
-#ifdef CONFIG_PFK
- bio->bi_dio_inode = bio_src->bi_dio_inode;
-#endif
+
bio_clone_crypt_key(bio, bio_src);
+
bio_clone_blkcg_association(bio, bio_src);
}
EXPORT_SYMBOL(__bio_clone_fast);
@@ -709,6 +711,7 @@ struct bio *bio_clone_bioset(struct bio *bio_src, gfp_t gfp_mask,
}
bio_clone_crypt_key(bio, bio_src);
+
bio_clone_blkcg_association(bio, bio_src);
return bio;
diff --git a/block/blk-core.c b/block/blk-core.c
index 68ec55a..abdea59 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -636,7 +636,6 @@ EXPORT_SYMBOL(blk_alloc_queue);
int blk_queue_enter(struct request_queue *q, bool nowait)
{
while (true) {
- int ret;
if (percpu_ref_tryget_live(&q->q_usage_counter))
return 0;
@@ -644,13 +643,11 @@ int blk_queue_enter(struct request_queue *q, bool nowait)
if (nowait)
return -EBUSY;
- ret = wait_event_interruptible(q->mq_freeze_wq,
- !atomic_read(&q->mq_freeze_depth) ||
- blk_queue_dying(q));
+ wait_event(q->mq_freeze_wq,
+ !atomic_read(&q->mq_freeze_depth) ||
+ blk_queue_dying(q));
if (blk_queue_dying(q))
return -ENODEV;
- if (ret)
- return ret;
}
}
@@ -1254,7 +1251,11 @@ static struct request *get_request(struct request_queue *q, int op,
trace_block_sleeprq(q, bio, op);
spin_unlock_irq(q->queue_lock);
- io_schedule();
+ /*
+ * FIXME: this should be io_schedule(). The timeout is there as a
+ * workaround for some io timeout problems.
+ */
+ io_schedule_timeout(5*HZ);
/*
* After sleeping, we become a "batching" process and will be able
@@ -1289,6 +1290,9 @@ static struct request *blk_old_get_request(struct request_queue *q, int rw,
/* q->queue_lock is unlocked at this point */
rq->__data_len = 0;
rq->__sector = (sector_t) -1;
+#ifdef CONFIG_PFK
+ rq->__dun = 0;
+#endif
rq->bio = rq->biotail = NULL;
return rq;
}
@@ -1530,7 +1534,7 @@ bool bio_attempt_front_merge(struct request_queue *q, struct request *req,
req->bio = bio;
#ifdef CONFIG_PFK
- WARN_ON(req->__dun || bio->bi_iter.bi_dun);
+ req->__dun = bio->bi_iter.bi_dun;
#endif
req->__sector = bio->bi_iter.bi_sector;
req->__data_len += bio->bi_iter.bi_size;
@@ -3591,7 +3595,7 @@ int __init blk_dev_init(void)
* them when printing them out.
*/
ssize_t
-blk_latency_hist_show(char* name, struct io_latency_state *s, char *buf,
+blk_latency_hist_show(char *name, struct io_latency_state *s, char *buf,
int buf_size)
{
int i;
@@ -3600,31 +3604,29 @@ blk_latency_hist_show(char* name, struct io_latency_state *s, char *buf,
int pct;
u_int64_t average;
- num_elem = s->latency_elems;
- if (num_elem > 0) {
- average = div64_u64(s->latency_sum, s->latency_elems);
- bytes_written += scnprintf(buf + bytes_written,
- buf_size - bytes_written,
- "IO svc_time %s Latency Histogram (n = %llu,"
- " average = %llu):\n", name, num_elem, average);
- for (i = 0;
- i < ARRAY_SIZE(latency_x_axis_us);
- i++) {
- elem = s->latency_y_axis[i];
- pct = div64_u64(elem * 100, num_elem);
- bytes_written += scnprintf(buf + bytes_written,
- PAGE_SIZE - bytes_written,
- "\t< %6lluus%15llu%15d%%\n",
- latency_x_axis_us[i],
- elem, pct);
- }
- /* Last element in y-axis table is overflow */
- elem = s->latency_y_axis[i];
- pct = div64_u64(elem * 100, num_elem);
- bytes_written += scnprintf(buf + bytes_written,
- PAGE_SIZE - bytes_written,
- "\t>=%6lluus%15llu%15d%%\n",
- latency_x_axis_us[i - 1], elem, pct);
+ num_elem = s->latency_elems;
+ if (num_elem > 0) {
+ average = div64_u64(s->latency_sum, s->latency_elems);
+ bytes_written += scnprintf(buf + bytes_written,
+ buf_size - bytes_written, "IO svc_time %s Latency"
+ " Histogram (n = %llu, average = %llu):\n", name,
+ num_elem, average);
+ for (i = 0; i < ARRAY_SIZE(latency_x_axis_us); i++) {
+ elem = s->latency_y_axis[i];
+ pct = div64_u64(elem * 100, num_elem);
+ bytes_written += scnprintf(buf + bytes_written,
+ PAGE_SIZE - bytes_written,
+ "\t< %6lluus%15llu%15d%%\n",
+ latency_x_axis_us[i],
+ elem, pct);
+ }
+ /* Last element in y-axis table is overflow */
+ elem = s->latency_y_axis[i];
+ pct = div64_u64(elem * 100, num_elem);
+ bytes_written += scnprintf(buf + bytes_written,
+ PAGE_SIZE - bytes_written,
+ "\t>=%6lluus%15llu%15d%%\n",
+ latency_x_axis_us[i - 1], elem, pct);
}
return bytes_written;
diff --git a/block/blk-lib.c b/block/blk-lib.c
index 46fe924..af1d26f 100644
--- a/block/blk-lib.c
+++ b/block/blk-lib.c
@@ -63,10 +63,18 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
unsigned int req_sects;
sector_t end_sect, tmp;
- /* Make sure bi_size doesn't overflow */
- req_sects = min_t(sector_t, nr_sects, UINT_MAX >> 9);
+ /*
+ * Issue in chunks of the user defined max discard setting,
+ * ensuring that bi_size doesn't overflow
+ */
+ req_sects = min_t(sector_t, nr_sects,
+ q->limits.max_discard_sectors);
+ if (!req_sects)
+ goto fail;
+ if (req_sects > UINT_MAX >> 9)
+ req_sects = UINT_MAX >> 9;
- /**
+ /*
* If splitting a request, and the next starting sector would be
* misaligned, stop the discard at the previous aligned sector.
*/
@@ -100,6 +108,14 @@ int __blkdev_issue_discard(struct block_device *bdev, sector_t sector,
*biop = bio;
return 0;
+
+fail:
+ if (bio) {
+ submit_bio_wait(bio);
+ bio_put(bio);
+ }
+ *biop = NULL;
+ return -EOPNOTSUPP;
}
EXPORT_SYMBOL(__blkdev_issue_discard);
diff --git a/block/blk-merge.c b/block/blk-merge.c
index 8e730d1..d9226f9 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -8,7 +8,7 @@
#include <linux/scatterlist.h>
#include <linux/pfk.h>
#include <trace/events/block.h>
-#include <linux/pfk.h>
+
#include "blk.h"
static struct bio *blk_bio_discard_split(struct request_queue *q,
@@ -584,6 +584,8 @@ int ll_back_merge_fn(struct request_queue *q, struct request *req,
if (blk_integrity_rq(req) &&
integrity_req_gap_back_merge(req, bio))
return 0;
+ if (blk_try_merge(req, bio) != ELEVATOR_BACK_MERGE)
+ return 0;
if (blk_rq_sectors(req) + bio_sectors(bio) >
blk_rq_get_max_sectors(req, blk_rq_pos(req))) {
req->cmd_flags |= REQ_NOMERGE;
@@ -608,6 +610,8 @@ int ll_front_merge_fn(struct request_queue *q, struct request *req,
if (blk_integrity_rq(req) &&
integrity_req_gap_front_merge(req, bio))
return 0;
+ if (blk_try_merge(req, bio) != ELEVATOR_FRONT_MERGE)
+ return 0;
if (blk_rq_sectors(req) + bio_sectors(bio) >
blk_rq_get_max_sectors(req, bio->bi_iter.bi_sector)) {
req->cmd_flags |= REQ_NOMERGE;
@@ -869,20 +873,20 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio)
!blk_write_same_mergeable(rq->bio, bio))
return false;
- if (crypto_not_mergeable(rq->bio, bio))
- return false;
return true;
}
int blk_try_merge(struct request *rq, struct bio *bio)
{
-#ifdef CONFIG_PFK
- if (blk_rq_dun(rq) || bio_dun(bio))
- return ELEVATOR_NO_MERGE;
-#endif
- if (blk_rq_pos(rq) + blk_rq_sectors(rq) == bio->bi_iter.bi_sector)
+ if (blk_rq_pos(rq) + blk_rq_sectors(rq) == bio->bi_iter.bi_sector) {
+ if (crypto_not_mergeable(rq->bio, bio))
+ return ELEVATOR_NO_MERGE;
return ELEVATOR_BACK_MERGE;
- else if (blk_rq_pos(rq) - bio_sectors(bio) == bio->bi_iter.bi_sector)
+ } else if (blk_rq_pos(rq) - bio_sectors(bio) ==
+ bio->bi_iter.bi_sector) {
+ if (crypto_not_mergeable(bio, rq->bio))
+ return ELEVATOR_NO_MERGE;
return ELEVATOR_FRONT_MERGE;
+ }
return ELEVATOR_NO_MERGE;
}
diff --git a/block/blk-settings.c b/block/blk-settings.c
index f679ae1..008b607 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -249,6 +249,7 @@ void blk_queue_max_hw_sectors(struct request_queue *q, unsigned int max_hw_secto
max_sectors = min_not_zero(max_hw_sectors, limits->max_dev_sectors);
max_sectors = min_t(unsigned int, max_sectors, BLK_DEF_MAX_SECTORS);
limits->max_sectors = max_sectors;
+ q->backing_dev_info->io_pages = max_sectors >> (PAGE_SHIFT - 9);
}
EXPORT_SYMBOL(blk_queue_max_hw_sectors);
diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c
index 07c75d0..8ae9e75 100644
--- a/block/blk-sysfs.c
+++ b/block/blk-sysfs.c
@@ -212,6 +212,7 @@ queue_max_sectors_store(struct request_queue *q, const char *page, size_t count)
spin_lock_irq(q->queue_lock);
q->limits.max_sectors = max_sectors_kb << 1;
+ q->backing_dev_info->io_pages = max_sectors_kb >> (PAGE_SHIFT - 10);
spin_unlock_irq(q->queue_lock);
return ret;
diff --git a/block/partitions/aix.c b/block/partitions/aix.c
index f3ed7b2..8e7d358 100644
--- a/block/partitions/aix.c
+++ b/block/partitions/aix.c
@@ -177,7 +177,7 @@ int aix_partition(struct parsed_partitions *state)
u32 vgda_sector = 0;
u32 vgda_len = 0;
int numlvs = 0;
- struct pvd *pvd;
+ struct pvd *pvd = NULL;
struct lv_info {
unsigned short pps_per_lv;
unsigned short pps_found;
@@ -231,10 +231,11 @@ int aix_partition(struct parsed_partitions *state)
if (lvip[i].pps_per_lv)
foundlvs += 1;
}
+ /* pvd loops depend on n[].name and lvip[].pps_per_lv */
+ pvd = alloc_pvd(state, vgda_sector + 17);
}
put_dev_sector(sect);
}
- pvd = alloc_pvd(state, vgda_sector + 17);
if (pvd) {
int numpps = be16_to_cpu(pvd->pp_count);
int psn_part1 = be32_to_cpu(pvd->psn_part1);
@@ -281,10 +282,14 @@ int aix_partition(struct parsed_partitions *state)
next_lp_ix += 1;
}
for (i = 0; i < state->limit; i += 1)
- if (lvip[i].pps_found && !lvip[i].lv_is_contiguous)
+ if (lvip[i].pps_found && !lvip[i].lv_is_contiguous) {
+ char tmp[sizeof(n[i].name) + 1]; // null char
+
+ snprintf(tmp, sizeof(tmp), "%s", n[i].name);
pr_warn("partition %s (%u pp's found) is "
"not contiguous\n",
- n[i].name, lvip[i].pps_found);
+ tmp, lvip[i].pps_found);
+ }
kfree(pvd);
}
kfree(n);
diff --git a/build.config b/build.config
new file mode 100644
index 0000000..dca53a2
--- /dev/null
+++ b/build.config
@@ -0,0 +1,6 @@
+KERNEL_DIR=private/msm-google
+UNSTRIPPED_MODULES="
+wlan.ko
+"
+. ${ROOT_DIR}/${KERNEL_DIR}/build.config.common.clang
+POST_DEFCONFIG_CMDS="check_defconfig"
diff --git a/build.config.common b/build.config.common
new file mode 100644
index 0000000..8ed3062
--- /dev/null
+++ b/build.config.common
@@ -0,0 +1,28 @@
+ARCH=arm64
+BRANCH=android-msm-bluecross-4.9
+CROSS_COMPILE=aarch64-linux-android-
+CROSS_COMPILE_ARM32=arm-linux-androideabi-
+DEFCONFIG=b1c1_defconfig
+EXTRA_CMDS=''
+CLANG_PREBUILT_BIN=prebuilts-master/clang/host/linux-x86/clang-r346389b/bin/
+LD_LIBRARY_PATH=${ROOT_DIR}/prebuilts-master/clang/host/linux-x86/clang-r346389b/lib64:$LD_LIBRARY_PATH
+export LD_LIBRARY_PATH
+LINUX_GCC_CROSS_COMPILE_PREBUILTS_BIN=prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin
+LINUX_GCC_CROSS_COMPILE_ARM32_PREBUILTS_BIN=prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.9/bin
+LZ4_PREBUILTS_BIN=prebuilts-master/misc/linux-x86/lz4
+DTC_PREBUILTS_BIN=prebuilts-master/misc/linux-x86/dtc
+LIBUFDT_PREBUILTS_BIN=prebuilts-master/misc/linux-x86/libufdt
+FILES="
+arch/arm64/boot/dtbo.img
+arch/arm64/boot/Image.lz4-dtb
+arch/arm64/boot/dts/qcom/sdm845-v2.1.dtb
+arch/arm64/boot/dts/qcom/sdm845-v2.dtb
+vmlinux
+System.map
+.config
+"
+EXT_MODULES="
+private/msm-google-modules/wlan/qcacld-3.0
+"
+IN_KERNEL_MODULES=1
+STOP_SHIP_TRACEPRINTK=1
diff --git a/build.config.common.clang b/build.config.common.clang
new file mode 100644
index 0000000..8ac509a
--- /dev/null
+++ b/build.config.common.clang
@@ -0,0 +1,3 @@
+. ${ROOT_DIR}/${KERNEL_DIR}/build.config.common
+CC=clang
+CLANG_TRIPLE=aarch64-linux-gnu-
diff --git a/build.config.debug_api b/build.config.debug_api
new file mode 100644
index 0000000..2437067
--- /dev/null
+++ b/build.config.debug_api
@@ -0,0 +1,23 @@
+KERNEL_DIR=private/msm-google
+. ${ROOT_DIR}/${KERNEL_DIR}/build.config.common
+KCFLAGS='-fno-tree-vrp'
+POST_DEFCONFIG_CMDS="update_debug_config"
+function update_debug_config() {
+ ${KERNEL_DIR}/scripts/config --file ${OUT_DIR}/.config \
+ -e CONFIG_DMA_API_DEBUG \
+ -e CONFIG_DEBUG_SHIRQ \
+ -e CONFIG_DEBUG_KOBJECT \
+ -e CONFIG_DEBUG_KOBJECT_RELEASE \
+ -e CONFIG_DEBUG_LIST \
+ -e CONFIG_DEBUG_PI_LIST \
+ -e CONFIG_DEBUG_NOTIFIERS \
+ -e CONFIG_DEBUG_CREDENTIALS \
+ -d LTO \
+ -d LTO_CLANG \
+ -d CFI \
+ -d CFI_PERMISSIVE \
+ -d CFI_CLANG \
+ -d SHADOW_CALL_STACK
+ (cd ${OUT_DIR} && \
+ make O=${OUT_DIR} $archsubarch CROSS_COMPILE=${CROSS_COMPILE} olddefconfig)
+}
diff --git a/build.config.debug_hang b/build.config.debug_hang
new file mode 100644
index 0000000..169ddea
--- /dev/null
+++ b/build.config.debug_hang
@@ -0,0 +1,24 @@
+KERNEL_DIR=private/msm-google
+. ${ROOT_DIR}/${KERNEL_DIR}/build.config.common
+KCFLAGS='-fno-tree-vrp'
+POST_DEFCONFIG_CMDS="update_debug_config"
+function update_debug_config() {
+ ${KERNEL_DIR}/scripts/config --file ${OUT_DIR}/.config \
+ -e CONFIG_LOCKUP_DETECTOR \
+ -e CONFIG_HARDLOCKUP_DETECTOR \
+ -e CONFIG_HARDLOCKUP_DETECTOR_OTHER_CPU \
+ -e CONFIG_BOOTPARAM_HARDLOCKUP_PANIC \
+ -e CONFIG_BOOTPARAM_SOFTLOCKUP_PANIC \
+ -e CONFIG_BOOTPARAM_HUNG_TASK_PANIC \
+ -e CONFIG_WQ_WATCHDOG \
+ -e CONFIG_PANIC_ON_RT_THROTTLING \
+ -e CONFIG_RCU_EQS_DEBUG \
+ -d LTO \
+ -d LTO_CLANG \
+ -d CFI \
+ -d CFI_PERMISSIVE \
+ -d CFI_CLANG \
+ -d SHADOW_CALL_STACK
+ (cd ${OUT_DIR} && \
+ make O=${OUT_DIR} $archsubarch CROSS_COMPILE=${CROSS_COMPILE} olddefconfig)
+}
diff --git a/build.config.debug_locking b/build.config.debug_locking
new file mode 100644
index 0000000..8d59c34
--- /dev/null
+++ b/build.config.debug_locking
@@ -0,0 +1,21 @@
+KERNEL_DIR=private/msm-google
+. ${ROOT_DIR}/${KERNEL_DIR}/build.config.common
+KCFLAGS='-fno-tree-vrp'
+POST_DEFCONFIG_CMDS="check_defconfig && update_debug_config"
+function update_debug_config() {
+ ${KERNEL_DIR}/scripts/config --file ${OUT_DIR}/.config \
+ -e CONFIG_DEBUG_RT_MUTEXES \
+ -e CONFIG_DEBUG_SPINLOCK \
+ -e CONFIG_DEBUG_SPINLOCK_PANIC_ON_BUG \
+ -e CONFIG_DEBUG_MUTEXES \
+ -e CONFIG_DEBUG_PREEMPT \
+ -e CONFIG_DEBUG_ATOMIC_SLEEP \
+ -d LTO \
+ -d LTO_CLANG \
+ -d CFI \
+ -d CFI_PERMISSIVE \
+ -d CFI_CLANG \
+ -d SHADOW_CALL_STACK
+ (cd ${OUT_DIR} && \
+ make O=${OUT_DIR} $archsubarch CROSS_COMPILE=${CROSS_COMPILE} olddefconfig)
+}
diff --git a/build.config.debug_memory b/build.config.debug_memory
new file mode 100644
index 0000000..6ee6a57
--- /dev/null
+++ b/build.config.debug_memory
@@ -0,0 +1,42 @@
+KERNEL_DIR=private/msm-google
+. ${ROOT_DIR}/${KERNEL_DIR}/build.config.common.clang
+POST_DEFCONFIG_CMDS="check_defconfig && update_debug_config"
+function update_debug_config() {
+ ${KERNEL_DIR}/scripts/config --file ${OUT_DIR}/.config \
+ -e CONFIG_DEBUG_OBJECTS \
+ -e CONFIG_DEBUG_OBJECTS_FREE \
+ -e CONFIG_DEBUG_OBJECTS_TIMERS \
+ -e CONFIG_DEBUG_OBJECTS_WORK \
+ -e CONFIG_DEBUG_OBJECTS_RCU_HEAD \
+ -e CONFIG_DEBUG_OBJECTS_PERCPU_COUNTER \
+ -e CONFIG_DEBUG_VM \
+ -e CONFIG_DEBUG_VM_VMACACHE \
+ -e CONFIG_DEBUG_VM_RB \
+ -e CONFIG_DEBUG_MEMORY_INIT \
+ -e CONFIG_DEBUG_STACK_USAGE \
+ -e CONFIG_DEBUG_PAGEALLOC \
+ -e CONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULT \
+ -e CONFIG_SCHED_STACK_END_CHECK \
+ -e CONFIG_SLUB_DEBUG \
+ -e CONFIG_SLUB_DEBUG_PANIC_ON \
+ -e CONFIG_PAGE_OWNER \
+ -e CONFIG_PAGE_OWNER_ENABLE_DEFAULT \
+ -e CONFIG_PAGE_POISONING \
+ -e CONFIG_PAGE_POISONING_ENABLE_DEFAULT \
+ -e CONFIG_DEBUG_SPINLOCK \
+ -e CONFIG_DEBUG_SPINLOCK_PANIC_ON_BUG \
+ -e CONFIG_DEBUG_MUTEXES \
+ -e CONFIG_DEBUG_ATOMIC_SLEEP \
+ -e CONFIG_DEBUG_LIST \
+ -e CONFIG_QCOM_RTB \
+ -e CONFIG_QCOM_RTB_SEPARATE_CPUS \
+ -e CONFIG_PANIC_ON_DATA_CORRUPTION \
+ -e CONFIG_REGMAP_ALLOW_WRITE_DEBUGFS \
+ -d LTO \
+ -d LTO_CLANG \
+ -d CFI \
+ -d CFI_PERMISSIVE \
+ -d CFI_CLANG
+ (cd ${OUT_DIR} && \
+ make O=${OUT_DIR} $archsubarch CROSS_COMPILE=${CROSS_COMPILE} olddefconfig)
+}
diff --git a/build.config.gcc b/build.config.gcc
new file mode 100644
index 0000000..3a78249
--- /dev/null
+++ b/build.config.gcc
@@ -0,0 +1,15 @@
+KERNEL_DIR=private/msm-google
+. ${ROOT_DIR}/${KERNEL_DIR}/build.config.common
+KCFLAGS='-fno-tree-vrp'
+POST_DEFCONFIG_CMDS="check_defconfig && update_gcc_config"
+function update_gcc_config() {
+ # Disable clang-specific options
+ ${KERNEL_DIR}/scripts/config --file ${OUT_DIR}/.config \
+ -d LTO_CLANG \
+ -d CFI \
+ -d CFI_PERMISSIVE \
+ -d CFI_CLANG \
+ -d SHADOW_CALL_STACK
+ (cd ${OUT_DIR} && \
+ make O=${OUT_DIR} $archsubarch CROSS_COMPILE=${CROSS_COMPILE} olddefconfig)
+}
diff --git a/build.config.kasan b/build.config.kasan
new file mode 100644
index 0000000..3b9abae
--- /dev/null
+++ b/build.config.kasan
@@ -0,0 +1,24 @@
+KERNEL_DIR=private/msm-google
+. ${ROOT_DIR}/${KERNEL_DIR}/build.config.common.clang
+POST_DEFCONFIG_CMDS="check_defconfig && update_kasan_config"
+function update_kasan_config() {
+ ${KERNEL_DIR}/scripts/config --file ${OUT_DIR}/.config \
+ -e CONFIG_KASAN \
+ -e CONFIG_KASAN_INLINE \
+ -e CONFIG_KASAN_PANIC_ON_WARN \
+ -e CONFIG_TEST_KASAN \
+ -e CONFIG_KCOV \
+ -e CONFIG_SLUB \
+ -d CONFIG_KASAN_OUTLINE \
+ -d CONFIG_RANDOMIZE_BASE \
+ -d CONFIG_CC_WERROR \
+ --set-val CONFIG_FRAME_WARN 0 \
+ -d LTO \
+ -d LTO_CLANG \
+ -d CFI \
+ -d CFI_PERMISSIVE \
+ -d CFI_CLANG \
+ -d SHADOW_CALL_STACK
+ (cd ${OUT_DIR} && \
+ make O=${OUT_DIR} $archsubarch CROSS_COMPILE=${CROSS_COMPILE} olddefconfig)
+}
diff --git a/build.config.no-cfi b/build.config.no-cfi
new file mode 100644
index 0000000..e71fc83
--- /dev/null
+++ b/build.config.no-cfi
@@ -0,0 +1,13 @@
+KERNEL_DIR=private/msm-google
+. ${ROOT_DIR}/${KERNEL_DIR}/build.config.common.clang
+POST_DEFCONFIG_CMDS="check_defconfig && update_nocfi_config"
+function update_nocfi_config() {
+ # Disable clang-specific options
+ ${KERNEL_DIR}/scripts/config --file ${OUT_DIR}/.config \
+ -d LTO_CLANG \
+ -d CFI \
+ -d CFI_PERMISSIVE \
+ -d CFI_CLANG
+ (cd ${OUT_DIR} && \
+ make O=${OUT_DIR} $archsubarch CROSS_COMPILE=${CROSS_COMPILE} olddefconfig)
+}
diff --git a/certs/esl_key.pem b/certs/esl_key.pem
new file mode 100644
index 0000000..ec1318b
--- /dev/null
+++ b/certs/esl_key.pem
@@ -0,0 +1,29 @@
+-----BEGIN CERTIFICATE-----
+MIIFADCCAuigAwIBAgIJAKL6ZJ3LUxTBMA0GCSqGSIb3DQEBBQUAMBAxDjAMBgNV
+BAMMBUVhc2VsMB4XDTE3MDQwNjIyMTExMloXDTE3MDUwNjIyMTExMlowEDEOMAwG
+A1UEAwwFRWFzZWwwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDUJuve
+l3PuMegBFYywMhkMmCuP1iTdp9isBpQ7OuNupmOhJCMDs0wBW7IG9lm7oxFQVLGX
+8GGkVCRQBqxWwPmWNIqbE7a7ZXkIaCcF2qrC8X+vYCe/wEw/WHrLAZxPLbt19yAK
++JNcnezWip06EupYIYkH3pjMQlhR+d+QpGs5zlOoGN2Il/nEsGk+GHO2nKCxBE79
+fCS/6cCy/GqHNo3ZykYJg6FzT9MyZzxDwm+Cg1ZH6Ok/AdbAeEy3v1ZQdgjNNqgz
+zb+IXitbAdZjT3D15+m3cy7s8Y3a+bGChjSjWCIkcHc419r9E6JfzJAFM1PITWVn
+6Z5fgLzO8RuIxkuzNalXJ6khvk72ibfvUffmUWuGs/jZh/YINUQ35ULOGBkrSuvX
+ZipbJryKZ61tm8w/sw0zKIHu/y8T3WNMtVF9raqKEjl5P4Wj0Jh39LZNAt2dyJVV
+g67HB0+Y+zdIn9n9Dy5WymFd8QR6tMF3ADb3gk/ztRbriudnidh9jZcLV8SWmePT
+UYwfCf+tWNE5RaASMrBIZy51vXD/HjiF23krmEM2xXk4nAlLbZ4rX52oQqs1vW1e
+Xf8NAi+g9+YpR7THI4iQoziLMl4m4bcSlOHDNRlO4b8LJje58/IuC7FOdFWc9VlK
+0Z+qlliF0eMMEpV/xJC5W7NEbW9yVnSe1wQQUwIDAQABo10wWzAMBgNVHRMBAf8E
+AjAAMAsGA1UdDwQEAwIHgDAdBgNVHQ4EFgQULZy4+2alImbLOwCz49szX635COQw
+HwYDVR0jBBgwFoAULZy4+2alImbLOwCz49szX635COQwDQYJKoZIhvcNAQEFBQAD
+ggIBAMDw/M/y18ywIupgD7CDOb/VZxzj54qeId5UkujHYNkT6vDliCg7CW+2oZWD
+aI6+BeEXieTrs/SHg7mxo3PhHzdvohGLe/kZnwdN6A7NSHKJjwcjv5jBCx395STo
++fXKV+wErn+blqSSSb/6gKchrxlKHr6PmD5QBBV9enHSIwsCnAG4KE1922X1JCSm
+ZGJUcPOocqr7clWYdWulQH75myIL+6bMQ7KSdn6evjFgLKYetqmroVcbUXEir3MP
+FNQM2AMFNcLIC3eBu5LNmRisjoK26ReA24ue7cYkxjEBBEQY/UOZwKm0aAYhBxgb
+mTsgRva0IvwactvvKMdcFokHKxHURcTqPJYTzxkKr5WQycJ+CQc1CbiwJygeJXHN
+w1xLUyHauSi10Uod17564UX7oYBHcwHLHICrwZWrPIqEG4uWcCHul0dabaZTQ3/2
+DIuW18dm+oToGbAKVXP757SN9GVJgmuhLMH5m/tIzO60qEIuFH75BgIIN8SzpY5x
+/aYCkxijTCPShbo0Y6d11PFj0TpeNncl6Xymedb1p03SiMejlE1gbM0k6Tp9daT8
+o/v+zm+yWWyxvr5hB+/udTmJHM7o+COu5jqDzWD++fNm487LC8BMa2F78XaLtWrT
+V0m8ghFKRjzF1w4LUvYNUFlxvNFUMUPc/06TJp1XnWNJvVPL
+-----END CERTIFICATE-----
diff --git a/certs/system_keyring.c b/certs/system_keyring.c
index 50979d6..2476650 100644
--- a/certs/system_keyring.c
+++ b/certs/system_keyring.c
@@ -14,6 +14,7 @@
#include <linux/sched.h>
#include <linux/cred.h>
#include <linux/err.h>
+#include <linux/verification.h>
#include <keys/asymmetric-type.h>
#include <keys/system_keyring.h>
#include <crypto/pkcs7.h>
@@ -207,7 +208,7 @@ int verify_pkcs7_signature(const void *data, size_t len,
if (!trusted_keys) {
trusted_keys = builtin_trusted_keys;
- } else if (trusted_keys == (void *)1UL) {
+ } else if (trusted_keys == VERIFY_USE_SECONDARY_KEYRING) {
#ifdef CONFIG_SECONDARY_TRUSTED_KEYRING
trusted_keys = secondary_trusted_keys;
#else
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 0f32afc..1ffcd85 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -102,6 +102,15 @@
select CRYPTO_ALGAPI
select CRYPTO_KPP2
+config CRYPTO_ACOMP2
+ tristate
+ select CRYPTO_ALGAPI2
+
+config CRYPTO_ACOMP
+ tristate
+ select CRYPTO_ALGAPI
+ select CRYPTO_ACOMP2
+
config CRYPTO_RSA
tristate "RSA algorithm"
select CRYPTO_AKCIPHER
@@ -138,6 +147,7 @@
select CRYPTO_BLKCIPHER2
select CRYPTO_AKCIPHER2
select CRYPTO_KPP2
+ select CRYPTO_ACOMP2
config CRYPTO_USER
tristate "Userspace cryptographic algorithm configuration"
@@ -1609,6 +1619,7 @@
config CRYPTO_LZ4
tristate "LZ4 compression algorithm"
select CRYPTO_ALGAPI
+ select CRYPTO_ACOMP2
select LZ4_COMPRESS
select LZ4_DECOMPRESS
help
@@ -1617,6 +1628,7 @@
config CRYPTO_LZ4HC
tristate "LZ4HC compression algorithm"
select CRYPTO_ALGAPI
+ select CRYPTO_ACOMP2
select LZ4HC_COMPRESS
select LZ4_DECOMPRESS
help
diff --git a/crypto/Makefile b/crypto/Makefile
index 5b08597..233ef36 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -51,6 +51,9 @@
rsa_generic-y += rsa-pkcs1pad.o
obj-$(CONFIG_CRYPTO_RSA) += rsa_generic.o
+obj-$(CONFIG_CRYPTO_ACOMP2) += acompress.o
+obj-$(CONFIG_CRYPTO_ACOMP2) += scompress.o
+
cryptomgr-y := algboss.o testmgr.o
obj-$(CONFIG_CRYPTO_MANAGER2) += cryptomgr.o
diff --git a/crypto/ablkcipher.c b/crypto/ablkcipher.c
index d676fc5..3bc0e76 100644
--- a/crypto/ablkcipher.c
+++ b/crypto/ablkcipher.c
@@ -70,11 +70,9 @@ static inline u8 *ablkcipher_get_spot(u8 *start, unsigned int len)
return max(start, end_page);
}
-static inline unsigned int ablkcipher_done_slow(struct ablkcipher_walk *walk,
- unsigned int bsize)
+static inline void ablkcipher_done_slow(struct ablkcipher_walk *walk,
+ unsigned int n)
{
- unsigned int n = bsize;
-
for (;;) {
unsigned int len_this_page = scatterwalk_pagelen(&walk->out);
@@ -86,17 +84,13 @@ static inline unsigned int ablkcipher_done_slow(struct ablkcipher_walk *walk,
n -= len_this_page;
scatterwalk_start(&walk->out, sg_next(walk->out.sg));
}
-
- return bsize;
}
-static inline unsigned int ablkcipher_done_fast(struct ablkcipher_walk *walk,
- unsigned int n)
+static inline void ablkcipher_done_fast(struct ablkcipher_walk *walk,
+ unsigned int n)
{
scatterwalk_advance(&walk->in, n);
scatterwalk_advance(&walk->out, n);
-
- return n;
}
static int ablkcipher_walk_next(struct ablkcipher_request *req,
@@ -106,39 +100,40 @@ int ablkcipher_walk_done(struct ablkcipher_request *req,
struct ablkcipher_walk *walk, int err)
{
struct crypto_tfm *tfm = req->base.tfm;
- unsigned int nbytes = 0;
+ unsigned int n; /* bytes processed */
+ bool more;
- if (likely(err >= 0)) {
- unsigned int n = walk->nbytes - err;
+ if (unlikely(err < 0))
+ goto finish;
- if (likely(!(walk->flags & ABLKCIPHER_WALK_SLOW)))
- n = ablkcipher_done_fast(walk, n);
- else if (WARN_ON(err)) {
+ n = walk->nbytes - err;
+ walk->total -= n;
+ more = (walk->total != 0);
+
+ if (likely(!(walk->flags & ABLKCIPHER_WALK_SLOW))) {
+ ablkcipher_done_fast(walk, n);
+ } else {
+ if (WARN_ON(err)) {
+ /* unexpected case; didn't process all bytes */
err = -EINVAL;
- goto err;
- } else
- n = ablkcipher_done_slow(walk, n);
-
- nbytes = walk->total - n;
- err = 0;
+ goto finish;
+ }
+ ablkcipher_done_slow(walk, n);
}
- scatterwalk_done(&walk->in, 0, nbytes);
- scatterwalk_done(&walk->out, 1, nbytes);
+ scatterwalk_done(&walk->in, 0, more);
+ scatterwalk_done(&walk->out, 1, more);
-err:
- walk->total = nbytes;
- walk->nbytes = nbytes;
-
- if (nbytes) {
+ if (more) {
crypto_yield(req->base.flags);
return ablkcipher_walk_next(req, walk);
}
-
+ err = 0;
+finish:
+ walk->nbytes = 0;
if (walk->iv != req->info)
memcpy(req->info, walk->iv, tfm->crt_ablkcipher.ivsize);
kfree(walk->iv_buffer);
-
return err;
}
EXPORT_SYMBOL_GPL(ablkcipher_walk_done);
@@ -372,6 +367,7 @@ static int crypto_ablkcipher_report(struct sk_buff *skb, struct crypto_alg *alg)
strncpy(rblkcipher.type, "ablkcipher", sizeof(rblkcipher.type));
strncpy(rblkcipher.geniv, alg->cra_ablkcipher.geniv ?: "<default>",
sizeof(rblkcipher.geniv));
+ rblkcipher.geniv[sizeof(rblkcipher.geniv) - 1] = '\0';
rblkcipher.blocksize = alg->cra_blocksize;
rblkcipher.min_keysize = alg->cra_ablkcipher.min_keysize;
@@ -446,6 +442,7 @@ static int crypto_givcipher_report(struct sk_buff *skb, struct crypto_alg *alg)
strncpy(rblkcipher.type, "givcipher", sizeof(rblkcipher.type));
strncpy(rblkcipher.geniv, alg->cra_ablkcipher.geniv ?: "<built-in>",
sizeof(rblkcipher.geniv));
+ rblkcipher.geniv[sizeof(rblkcipher.geniv) - 1] = '\0';
rblkcipher.blocksize = alg->cra_blocksize;
rblkcipher.min_keysize = alg->cra_ablkcipher.min_keysize;
diff --git a/crypto/acompress.c b/crypto/acompress.c
new file mode 100644
index 0000000..887783d
--- /dev/null
+++ b/crypto/acompress.c
@@ -0,0 +1,169 @@
+/*
+ * Asynchronous Compression operations
+ *
+ * Copyright (c) 2016, Intel Corporation
+ * Authors: Weigang Li <weigang.li@intel.com>
+ * Giovanni Cabiddu <giovanni.cabiddu@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ */
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/seq_file.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <linux/crypto.h>
+#include <crypto/algapi.h>
+#include <linux/cryptouser.h>
+#include <net/netlink.h>
+#include <crypto/internal/acompress.h>
+#include <crypto/internal/scompress.h>
+#include "internal.h"
+
+static const struct crypto_type crypto_acomp_type;
+
+#ifdef CONFIG_NET
+static int crypto_acomp_report(struct sk_buff *skb, struct crypto_alg *alg)
+{
+ struct crypto_report_acomp racomp;
+
+ strncpy(racomp.type, "acomp", sizeof(racomp.type));
+
+ if (nla_put(skb, CRYPTOCFGA_REPORT_ACOMP,
+ sizeof(struct crypto_report_acomp), &racomp))
+ goto nla_put_failure;
+ return 0;
+
+nla_put_failure:
+ return -EMSGSIZE;
+}
+#else
+static int crypto_acomp_report(struct sk_buff *skb, struct crypto_alg *alg)
+{
+ return -ENOSYS;
+}
+#endif
+
+static void crypto_acomp_show(struct seq_file *m, struct crypto_alg *alg)
+ __attribute__ ((unused));
+
+static void crypto_acomp_show(struct seq_file *m, struct crypto_alg *alg)
+{
+ seq_puts(m, "type : acomp\n");
+}
+
+static void crypto_acomp_exit_tfm(struct crypto_tfm *tfm)
+{
+ struct crypto_acomp *acomp = __crypto_acomp_tfm(tfm);
+ struct acomp_alg *alg = crypto_acomp_alg(acomp);
+
+ alg->exit(acomp);
+}
+
+static int crypto_acomp_init_tfm(struct crypto_tfm *tfm)
+{
+ struct crypto_acomp *acomp = __crypto_acomp_tfm(tfm);
+ struct acomp_alg *alg = crypto_acomp_alg(acomp);
+
+ if (tfm->__crt_alg->cra_type != &crypto_acomp_type)
+ return crypto_init_scomp_ops_async(tfm);
+
+ acomp->compress = alg->compress;
+ acomp->decompress = alg->decompress;
+ acomp->dst_free = alg->dst_free;
+ acomp->reqsize = alg->reqsize;
+
+ if (alg->exit)
+ acomp->base.exit = crypto_acomp_exit_tfm;
+
+ if (alg->init)
+ return alg->init(acomp);
+
+ return 0;
+}
+
+static unsigned int crypto_acomp_extsize(struct crypto_alg *alg)
+{
+ int extsize = crypto_alg_extsize(alg);
+
+ if (alg->cra_type != &crypto_acomp_type)
+ extsize += sizeof(struct crypto_scomp *);
+
+ return extsize;
+}
+
+static const struct crypto_type crypto_acomp_type = {
+ .extsize = crypto_acomp_extsize,
+ .init_tfm = crypto_acomp_init_tfm,
+#ifdef CONFIG_PROC_FS
+ .show = crypto_acomp_show,
+#endif
+ .report = crypto_acomp_report,
+ .maskclear = ~CRYPTO_ALG_TYPE_MASK,
+ .maskset = CRYPTO_ALG_TYPE_ACOMPRESS_MASK,
+ .type = CRYPTO_ALG_TYPE_ACOMPRESS,
+ .tfmsize = offsetof(struct crypto_acomp, base),
+};
+
+struct crypto_acomp *crypto_alloc_acomp(const char *alg_name, u32 type,
+ u32 mask)
+{
+ return crypto_alloc_tfm(alg_name, &crypto_acomp_type, type, mask);
+}
+EXPORT_SYMBOL_GPL(crypto_alloc_acomp);
+
+struct acomp_req *acomp_request_alloc(struct crypto_acomp *acomp)
+{
+ struct crypto_tfm *tfm = crypto_acomp_tfm(acomp);
+ struct acomp_req *req;
+
+ req = __acomp_request_alloc(acomp);
+ if (req && (tfm->__crt_alg->cra_type != &crypto_acomp_type))
+ return crypto_acomp_scomp_alloc_ctx(req);
+
+ return req;
+}
+EXPORT_SYMBOL_GPL(acomp_request_alloc);
+
+void acomp_request_free(struct acomp_req *req)
+{
+ struct crypto_acomp *acomp = crypto_acomp_reqtfm(req);
+ struct crypto_tfm *tfm = crypto_acomp_tfm(acomp);
+
+ if (tfm->__crt_alg->cra_type != &crypto_acomp_type)
+ crypto_acomp_scomp_free_ctx(req);
+
+ if (req->flags & CRYPTO_ACOMP_ALLOC_OUTPUT) {
+ acomp->dst_free(req->dst);
+ req->dst = NULL;
+ }
+
+ __acomp_request_free(req);
+}
+EXPORT_SYMBOL_GPL(acomp_request_free);
+
+int crypto_register_acomp(struct acomp_alg *alg)
+{
+ struct crypto_alg *base = &alg->base;
+
+ base->cra_type = &crypto_acomp_type;
+ base->cra_flags &= ~CRYPTO_ALG_TYPE_MASK;
+ base->cra_flags |= CRYPTO_ALG_TYPE_ACOMPRESS;
+
+ return crypto_register_alg(base);
+}
+EXPORT_SYMBOL_GPL(crypto_register_acomp);
+
+int crypto_unregister_acomp(struct acomp_alg *alg)
+{
+ return crypto_unregister_alg(&alg->base);
+}
+EXPORT_SYMBOL_GPL(crypto_unregister_acomp);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Asynchronous compression type");
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index ca50eeb1..b5953f1 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -157,16 +157,16 @@ static int alg_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
void *private;
int err;
- /* If caller uses non-allowed flag, return error. */
- if ((sa->salg_feat & ~allowed) || (sa->salg_mask & ~allowed))
- return -EINVAL;
-
if (sock->state == SS_CONNECTED)
return -EINVAL;
if (addr_len != sizeof(*sa))
return -EINVAL;
+ /* If caller uses non-allowed flag, return error. */
+ if ((sa->salg_feat & ~allowed) || (sa->salg_mask & ~allowed))
+ return -EINVAL;
+
sa->salg_type[sizeof(sa->salg_type) - 1] = 0;
sa->salg_name[sizeof(sa->salg_name) - 1] = 0;
diff --git a/crypto/api.c b/crypto/api.c
index e5c1abf..f12d6b9 100644
--- a/crypto/api.c
+++ b/crypto/api.c
@@ -216,7 +216,7 @@ struct crypto_alg *crypto_larval_lookup(const char *name, u32 type, u32 mask)
type &= mask;
alg = crypto_alg_lookup(name, type, mask);
- if (!alg) {
+ if (!alg && !(mask & CRYPTO_NOLOAD)) {
request_module("crypto-%s", name);
if (!((type ^ CRYPTO_ALG_NEED_FALLBACK) & mask &
diff --git a/crypto/asymmetric_keys/pkcs7_key_type.c b/crypto/asymmetric_keys/pkcs7_key_type.c
index 1063b64..b2aa925 100644
--- a/crypto/asymmetric_keys/pkcs7_key_type.c
+++ b/crypto/asymmetric_keys/pkcs7_key_type.c
@@ -62,7 +62,7 @@ static int pkcs7_preparse(struct key_preparsed_payload *prep)
return verify_pkcs7_signature(NULL, 0,
prep->data, prep->datalen,
- (void *)1UL, usage,
+ VERIFY_USE_SECONDARY_KEYRING, usage,
pkcs7_view_content, prep);
}
diff --git a/crypto/asymmetric_keys/pkcs7_trust.c b/crypto/asymmetric_keys/pkcs7_trust.c
index f6a009d..52e5ea3 100644
--- a/crypto/asymmetric_keys/pkcs7_trust.c
+++ b/crypto/asymmetric_keys/pkcs7_trust.c
@@ -106,6 +106,7 @@ static int pkcs7_validate_trust_one(struct pkcs7_message *pkcs7,
pr_devel("sinfo %u: Direct signer is key %x\n",
sinfo->index, key_serial(key));
x509 = NULL;
+ sig = sinfo->sig;
goto matched;
}
if (PTR_ERR(key) != -ENOKEY)
diff --git a/crypto/asymmetric_keys/x509_cert_parser.c b/crypto/asymmetric_keys/x509_cert_parser.c
index ce2df8c..7e6a43f 100644
--- a/crypto/asymmetric_keys/x509_cert_parser.c
+++ b/crypto/asymmetric_keys/x509_cert_parser.c
@@ -249,6 +249,15 @@ int x509_note_signature(void *context, size_t hdrlen,
return -EINVAL;
}
+ if (strcmp(ctx->cert->sig->pkey_algo, "rsa") == 0) {
+ /* Discard the BIT STRING metadata */
+ if (vlen < 1 || *(const u8 *)value != 0)
+ return -EBADMSG;
+
+ value++;
+ vlen--;
+ }
+
ctx->cert->raw_sig = value;
ctx->cert->raw_sig_size = vlen;
return 0;
diff --git a/crypto/authenc.c b/crypto/authenc.c
index a7e1ac7..c3180eb 100644
--- a/crypto/authenc.c
+++ b/crypto/authenc.c
@@ -108,6 +108,7 @@ static int crypto_authenc_setkey(struct crypto_aead *authenc, const u8 *key,
CRYPTO_TFM_RES_MASK);
out:
+ memzero_explicit(&keys, sizeof(keys));
return err;
badkey:
diff --git a/crypto/authencesn.c b/crypto/authencesn.c
index 18c94e1..49e7e85 100644
--- a/crypto/authencesn.c
+++ b/crypto/authencesn.c
@@ -90,6 +90,7 @@ static int crypto_authenc_esn_setkey(struct crypto_aead *authenc_esn, const u8 *
CRYPTO_TFM_RES_MASK);
out:
+ memzero_explicit(&keys, sizeof(keys));
return err;
badkey:
diff --git a/crypto/blkcipher.c b/crypto/blkcipher.c
index a832426..59a0936 100644
--- a/crypto/blkcipher.c
+++ b/crypto/blkcipher.c
@@ -70,19 +70,18 @@ static inline u8 *blkcipher_get_spot(u8 *start, unsigned int len)
return max(start, end_page);
}
-static inline unsigned int blkcipher_done_slow(struct blkcipher_walk *walk,
- unsigned int bsize)
+static inline void blkcipher_done_slow(struct blkcipher_walk *walk,
+ unsigned int bsize)
{
u8 *addr;
addr = (u8 *)ALIGN((unsigned long)walk->buffer, walk->alignmask + 1);
addr = blkcipher_get_spot(addr, bsize);
scatterwalk_copychunks(addr, &walk->out, bsize, 1);
- return bsize;
}
-static inline unsigned int blkcipher_done_fast(struct blkcipher_walk *walk,
- unsigned int n)
+static inline void blkcipher_done_fast(struct blkcipher_walk *walk,
+ unsigned int n)
{
if (walk->flags & BLKCIPHER_WALK_COPY) {
blkcipher_map_dst(walk);
@@ -96,49 +95,48 @@ static inline unsigned int blkcipher_done_fast(struct blkcipher_walk *walk,
scatterwalk_advance(&walk->in, n);
scatterwalk_advance(&walk->out, n);
-
- return n;
}
int blkcipher_walk_done(struct blkcipher_desc *desc,
struct blkcipher_walk *walk, int err)
{
- unsigned int nbytes = 0;
+ unsigned int n; /* bytes processed */
+ bool more;
- if (likely(err >= 0)) {
- unsigned int n = walk->nbytes - err;
+ if (unlikely(err < 0))
+ goto finish;
- if (likely(!(walk->flags & BLKCIPHER_WALK_SLOW)))
- n = blkcipher_done_fast(walk, n);
- else if (WARN_ON(err)) {
+ n = walk->nbytes - err;
+ walk->total -= n;
+ more = (walk->total != 0);
+
+ if (likely(!(walk->flags & BLKCIPHER_WALK_SLOW))) {
+ blkcipher_done_fast(walk, n);
+ } else {
+ if (WARN_ON(err)) {
+ /* unexpected case; didn't process all bytes */
err = -EINVAL;
- goto err;
- } else
- n = blkcipher_done_slow(walk, n);
-
- nbytes = walk->total - n;
- err = 0;
+ goto finish;
+ }
+ blkcipher_done_slow(walk, n);
}
- scatterwalk_done(&walk->in, 0, nbytes);
- scatterwalk_done(&walk->out, 1, nbytes);
+ scatterwalk_done(&walk->in, 0, more);
+ scatterwalk_done(&walk->out, 1, more);
-err:
- walk->total = nbytes;
- walk->nbytes = nbytes;
-
- if (nbytes) {
+ if (more) {
crypto_yield(desc->flags);
return blkcipher_walk_next(desc, walk);
}
-
+ err = 0;
+finish:
+ walk->nbytes = 0;
if (walk->iv != desc->info)
memcpy(desc->info, walk->iv, walk->ivsize);
if (walk->buffer != walk->page)
kfree(walk->buffer);
if (walk->page)
free_page((unsigned long)walk->page);
-
return err;
}
EXPORT_SYMBOL_GPL(blkcipher_walk_done);
@@ -512,6 +510,7 @@ static int crypto_blkcipher_report(struct sk_buff *skb, struct crypto_alg *alg)
strncpy(rblkcipher.type, "blkcipher", sizeof(rblkcipher.type));
strncpy(rblkcipher.geniv, alg->cra_blkcipher.geniv ?: "<default>",
sizeof(rblkcipher.geniv));
+ rblkcipher.geniv[sizeof(rblkcipher.geniv) - 1] = '\0';
rblkcipher.blocksize = alg->cra_blocksize;
rblkcipher.min_keysize = alg->cra_blkcipher.min_keysize;
diff --git a/crypto/crypto_user.c b/crypto/crypto_user.c
index 1c57054..a90404a 100644
--- a/crypto/crypto_user.c
+++ b/crypto/crypto_user.c
@@ -112,6 +112,21 @@ static int crypto_report_comp(struct sk_buff *skb, struct crypto_alg *alg)
return -EMSGSIZE;
}
+static int crypto_report_acomp(struct sk_buff *skb, struct crypto_alg *alg)
+{
+ struct crypto_report_acomp racomp;
+
+ strncpy(racomp.type, "acomp", sizeof(racomp.type));
+
+ if (nla_put(skb, CRYPTOCFGA_REPORT_ACOMP,
+ sizeof(struct crypto_report_acomp), &racomp))
+ goto nla_put_failure;
+ return 0;
+
+nla_put_failure:
+ return -EMSGSIZE;
+}
+
static int crypto_report_akcipher(struct sk_buff *skb, struct crypto_alg *alg)
{
struct crypto_report_akcipher rakcipher;
@@ -186,7 +201,11 @@ static int crypto_report_one(struct crypto_alg *alg,
goto nla_put_failure;
break;
+ case CRYPTO_ALG_TYPE_ACOMPRESS:
+ if (crypto_report_acomp(skb, alg))
+ goto nla_put_failure;
+ break;
case CRYPTO_ALG_TYPE_AKCIPHER:
if (crypto_report_akcipher(skb, alg))
goto nla_put_failure;
diff --git a/crypto/drbg.c b/crypto/drbg.c
index 942ddff..4bb5f93 100644
--- a/crypto/drbg.c
+++ b/crypto/drbg.c
@@ -1134,8 +1134,10 @@ static inline void drbg_dealloc_state(struct drbg_state *drbg)
if (!drbg)
return;
kzfree(drbg->Vbuf);
+ drbg->Vbuf = NULL;
drbg->V = NULL;
kzfree(drbg->Cbuf);
+ drbg->Cbuf = NULL;
drbg->C = NULL;
kzfree(drbg->scratchpadbuf);
drbg->scratchpadbuf = NULL;
diff --git a/crypto/lrw.c b/crypto/lrw.c
index 6f9908a..d38a382 100644
--- a/crypto/lrw.c
+++ b/crypto/lrw.c
@@ -132,7 +132,12 @@ static inline int get_index128(be128 *block)
return x + ffz(val);
}
- return x;
+ /*
+ * If we get here, then x == 128 and we are incrementing the counter
+ * from all ones to all zeros. This means we must return index 127, i.e.
+ * the one corresponding to key2*{ 1,...,1 }.
+ */
+ return 127;
}
static int crypt(struct blkcipher_desc *d,
diff --git a/crypto/lz4.c b/crypto/lz4.c
index aefbcea..99c1b2c 100644
--- a/crypto/lz4.c
+++ b/crypto/lz4.c
@@ -23,36 +23,53 @@
#include <linux/crypto.h>
#include <linux/vmalloc.h>
#include <linux/lz4.h>
+#include <crypto/internal/scompress.h>
struct lz4_ctx {
void *lz4_comp_mem;
};
+static void *lz4_alloc_ctx(struct crypto_scomp *tfm)
+{
+ void *ctx;
+
+ ctx = vmalloc(LZ4_MEM_COMPRESS);
+ if (!ctx)
+ return ERR_PTR(-ENOMEM);
+
+ return ctx;
+}
+
static int lz4_init(struct crypto_tfm *tfm)
{
struct lz4_ctx *ctx = crypto_tfm_ctx(tfm);
- ctx->lz4_comp_mem = vmalloc(LZ4_MEM_COMPRESS);
- if (!ctx->lz4_comp_mem)
+ ctx->lz4_comp_mem = lz4_alloc_ctx(NULL);
+ if (IS_ERR(ctx->lz4_comp_mem))
return -ENOMEM;
return 0;
}
+static void lz4_free_ctx(struct crypto_scomp *tfm, void *ctx)
+{
+ vfree(ctx);
+}
+
static void lz4_exit(struct crypto_tfm *tfm)
{
struct lz4_ctx *ctx = crypto_tfm_ctx(tfm);
- vfree(ctx->lz4_comp_mem);
+
+ lz4_free_ctx(NULL, ctx->lz4_comp_mem);
}
-static int lz4_compress_crypto(struct crypto_tfm *tfm, const u8 *src,
- unsigned int slen, u8 *dst, unsigned int *dlen)
+static int __lz4_compress_crypto(const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx)
{
- struct lz4_ctx *ctx = crypto_tfm_ctx(tfm);
size_t tmp_len = *dlen;
int err;
- err = lz4_compress(src, slen, dst, &tmp_len, ctx->lz4_comp_mem);
+ err = lz4_compress(src, slen, dst, &tmp_len, ctx);
if (err < 0)
return -EINVAL;
@@ -61,8 +78,23 @@ static int lz4_compress_crypto(struct crypto_tfm *tfm, const u8 *src,
return 0;
}
-static int lz4_decompress_crypto(struct crypto_tfm *tfm, const u8 *src,
- unsigned int slen, u8 *dst, unsigned int *dlen)
+static int lz4_scompress(struct crypto_scomp *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen,
+ void *ctx)
+{
+ return __lz4_compress_crypto(src, slen, dst, dlen, ctx);
+}
+
+static int lz4_compress_crypto(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen)
+{
+ struct lz4_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ return __lz4_compress_crypto(src, slen, dst, dlen, ctx->lz4_comp_mem);
+}
+
+static int __lz4_decompress_crypto(const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx)
{
int err;
size_t tmp_len = *dlen;
@@ -76,6 +108,20 @@ static int lz4_decompress_crypto(struct crypto_tfm *tfm, const u8 *src,
return err;
}
+static int lz4_sdecompress(struct crypto_scomp *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen,
+ void *ctx)
+{
+ return __lz4_decompress_crypto(src, slen, dst, dlen, NULL);
+}
+
+static int lz4_decompress_crypto(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst,
+ unsigned int *dlen)
+{
+ return __lz4_decompress_crypto(src, slen, dst, dlen, NULL);
+}
+
static struct crypto_alg alg_lz4 = {
.cra_name = "lz4",
.cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
@@ -89,14 +135,39 @@ static struct crypto_alg alg_lz4 = {
.coa_decompress = lz4_decompress_crypto } }
};
+static struct scomp_alg scomp = {
+ .alloc_ctx = lz4_alloc_ctx,
+ .free_ctx = lz4_free_ctx,
+ .compress = lz4_scompress,
+ .decompress = lz4_sdecompress,
+ .base = {
+ .cra_name = "lz4",
+ .cra_driver_name = "lz4-scomp",
+ .cra_module = THIS_MODULE,
+ }
+};
+
static int __init lz4_mod_init(void)
{
- return crypto_register_alg(&alg_lz4);
+ int ret;
+
+ ret = crypto_register_alg(&alg_lz4);
+ if (ret)
+ return ret;
+
+ ret = crypto_register_scomp(&scomp);
+ if (ret) {
+ crypto_unregister_alg(&alg_lz4);
+ return ret;
+ }
+
+ return ret;
}
static void __exit lz4_mod_fini(void)
{
crypto_unregister_alg(&alg_lz4);
+ crypto_unregister_scomp(&scomp);
}
module_init(lz4_mod_init);
diff --git a/crypto/lz4hc.c b/crypto/lz4hc.c
index a1d3b5b..75ffc4a 100644
--- a/crypto/lz4hc.c
+++ b/crypto/lz4hc.c
@@ -22,37 +22,53 @@
#include <linux/crypto.h>
#include <linux/vmalloc.h>
#include <linux/lz4.h>
+#include <crypto/internal/scompress.h>
struct lz4hc_ctx {
void *lz4hc_comp_mem;
};
+static void *lz4hc_alloc_ctx(struct crypto_scomp *tfm)
+{
+ void *ctx;
+
+ ctx = vmalloc(LZ4HC_MEM_COMPRESS);
+ if (!ctx)
+ return ERR_PTR(-ENOMEM);
+
+ return ctx;
+}
+
static int lz4hc_init(struct crypto_tfm *tfm)
{
struct lz4hc_ctx *ctx = crypto_tfm_ctx(tfm);
- ctx->lz4hc_comp_mem = vmalloc(LZ4HC_MEM_COMPRESS);
- if (!ctx->lz4hc_comp_mem)
+ ctx->lz4hc_comp_mem = lz4hc_alloc_ctx(NULL);
+ if (IS_ERR(ctx->lz4hc_comp_mem))
return -ENOMEM;
return 0;
}
+static void lz4hc_free_ctx(struct crypto_scomp *tfm, void *ctx)
+{
+ vfree(ctx);
+}
+
static void lz4hc_exit(struct crypto_tfm *tfm)
{
struct lz4hc_ctx *ctx = crypto_tfm_ctx(tfm);
- vfree(ctx->lz4hc_comp_mem);
+ lz4hc_free_ctx(NULL, ctx->lz4hc_comp_mem);
}
-static int lz4hc_compress_crypto(struct crypto_tfm *tfm, const u8 *src,
- unsigned int slen, u8 *dst, unsigned int *dlen)
+static int __lz4hc_compress_crypto(const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx)
{
- struct lz4hc_ctx *ctx = crypto_tfm_ctx(tfm);
size_t tmp_len = *dlen;
int err;
- err = lz4hc_compress(src, slen, dst, &tmp_len, ctx->lz4hc_comp_mem);
+ err = lz4hc_compress(src, slen, dst, &tmp_len, ctx);
if (err < 0)
return -EINVAL;
@@ -61,8 +77,25 @@ static int lz4hc_compress_crypto(struct crypto_tfm *tfm, const u8 *src,
return 0;
}
-static int lz4hc_decompress_crypto(struct crypto_tfm *tfm, const u8 *src,
- unsigned int slen, u8 *dst, unsigned int *dlen)
+static int lz4hc_scompress(struct crypto_scomp *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen,
+ void *ctx)
+{
+ return __lz4hc_compress_crypto(src, slen, dst, dlen, ctx);
+}
+
+static int lz4hc_compress_crypto(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst,
+ unsigned int *dlen)
+{
+ struct lz4hc_ctx *ctx = crypto_tfm_ctx(tfm);
+
+ return __lz4hc_compress_crypto(src, slen, dst, dlen,
+ ctx->lz4hc_comp_mem);
+}
+
+static int __lz4hc_decompress_crypto(const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx)
{
int err;
size_t tmp_len = *dlen;
@@ -76,6 +109,20 @@ static int lz4hc_decompress_crypto(struct crypto_tfm *tfm, const u8 *src,
return err;
}
+static int lz4hc_sdecompress(struct crypto_scomp *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen,
+ void *ctx)
+{
+ return __lz4hc_decompress_crypto(src, slen, dst, dlen, NULL);
+}
+
+static int lz4hc_decompress_crypto(struct crypto_tfm *tfm, const u8 *src,
+ unsigned int slen, u8 *dst,
+ unsigned int *dlen)
+{
+ return __lz4hc_decompress_crypto(src, slen, dst, dlen, NULL);
+}
+
static struct crypto_alg alg_lz4hc = {
.cra_name = "lz4hc",
.cra_flags = CRYPTO_ALG_TYPE_COMPRESS,
@@ -89,14 +136,39 @@ static struct crypto_alg alg_lz4hc = {
.coa_decompress = lz4hc_decompress_crypto } }
};
+static struct scomp_alg scomp = {
+ .alloc_ctx = lz4hc_alloc_ctx,
+ .free_ctx = lz4hc_free_ctx,
+ .compress = lz4hc_scompress,
+ .decompress = lz4hc_sdecompress,
+ .base = {
+ .cra_name = "lz4hc",
+ .cra_driver_name = "lz4hc-scomp",
+ .cra_module = THIS_MODULE,
+ }
+};
+
static int __init lz4hc_mod_init(void)
{
- return crypto_register_alg(&alg_lz4hc);
+ int ret;
+
+ ret = crypto_register_alg(&alg_lz4hc);
+ if (ret)
+ return ret;
+
+ ret = crypto_register_scomp(&scomp);
+ if (ret) {
+ crypto_unregister_alg(&alg_lz4hc);
+ return ret;
+ }
+
+ return ret;
}
static void __exit lz4hc_mod_fini(void)
{
crypto_unregister_alg(&alg_lz4hc);
+ crypto_unregister_scomp(&scomp);
}
module_init(lz4hc_mod_init);
diff --git a/crypto/scompress.c b/crypto/scompress.c
new file mode 100644
index 0000000..35e396d
--- /dev/null
+++ b/crypto/scompress.c
@@ -0,0 +1,356 @@
+/*
+ * Synchronous Compression operations
+ *
+ * Copyright 2015 LG Electronics Inc.
+ * Copyright (c) 2016, Intel Corporation
+ * Author: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ */
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/seq_file.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <linux/crypto.h>
+#include <linux/vmalloc.h>
+#include <crypto/algapi.h>
+#include <linux/cryptouser.h>
+#include <net/netlink.h>
+#include <linux/scatterlist.h>
+#include <crypto/scatterwalk.h>
+#include <crypto/internal/acompress.h>
+#include <crypto/internal/scompress.h>
+#include "internal.h"
+
+static const struct crypto_type crypto_scomp_type;
+static void * __percpu *scomp_src_scratches;
+static void * __percpu *scomp_dst_scratches;
+static int scomp_scratch_users;
+static DEFINE_MUTEX(scomp_lock);
+
+#ifdef CONFIG_NET
+static int crypto_scomp_report(struct sk_buff *skb, struct crypto_alg *alg)
+{
+ struct crypto_report_comp rscomp;
+
+ strncpy(rscomp.type, "scomp", sizeof(rscomp.type));
+
+ if (nla_put(skb, CRYPTOCFGA_REPORT_COMPRESS,
+ sizeof(struct crypto_report_comp), &rscomp))
+ goto nla_put_failure;
+ return 0;
+
+nla_put_failure:
+ return -EMSGSIZE;
+}
+#else
+static int crypto_scomp_report(struct sk_buff *skb, struct crypto_alg *alg)
+{
+ return -ENOSYS;
+}
+#endif
+
+static void crypto_scomp_show(struct seq_file *m, struct crypto_alg *alg)
+ __attribute__ ((unused));
+
+static void crypto_scomp_show(struct seq_file *m, struct crypto_alg *alg)
+{
+ seq_puts(m, "type : scomp\n");
+}
+
+static int crypto_scomp_init_tfm(struct crypto_tfm *tfm)
+{
+ return 0;
+}
+
+static void crypto_scomp_free_scratches(void * __percpu *scratches)
+{
+ int i;
+
+ if (!scratches)
+ return;
+
+ for_each_possible_cpu(i)
+ vfree(*per_cpu_ptr(scratches, i));
+
+ free_percpu(scratches);
+}
+
+static void * __percpu *crypto_scomp_alloc_scratches(void)
+{
+ void * __percpu *scratches;
+ int i;
+
+ scratches = alloc_percpu(void *);
+ if (!scratches)
+ return NULL;
+
+ for_each_possible_cpu(i) {
+ void *scratch;
+
+ scratch = vmalloc_node(SCOMP_SCRATCH_SIZE, cpu_to_node(i));
+ if (!scratch)
+ goto error;
+ *per_cpu_ptr(scratches, i) = scratch;
+ }
+
+ return scratches;
+
+error:
+ crypto_scomp_free_scratches(scratches);
+ return NULL;
+}
+
+static void crypto_scomp_free_all_scratches(void)
+{
+ if (!--scomp_scratch_users) {
+ crypto_scomp_free_scratches(scomp_src_scratches);
+ crypto_scomp_free_scratches(scomp_dst_scratches);
+ scomp_src_scratches = NULL;
+ scomp_dst_scratches = NULL;
+ }
+}
+
+static int crypto_scomp_alloc_all_scratches(void)
+{
+ if (!scomp_scratch_users++) {
+ scomp_src_scratches = crypto_scomp_alloc_scratches();
+ if (!scomp_src_scratches)
+ return -ENOMEM;
+ scomp_dst_scratches = crypto_scomp_alloc_scratches();
+ if (!scomp_dst_scratches)
+ return -ENOMEM;
+ }
+ return 0;
+}
+
+static void crypto_scomp_sg_free(struct scatterlist *sgl)
+{
+ int i, n;
+ struct page *page;
+
+ if (!sgl)
+ return;
+
+ n = sg_nents(sgl);
+ for_each_sg(sgl, sgl, n, i) {
+ page = sg_page(sgl);
+ if (page)
+ __free_page(page);
+ }
+
+ kfree(sgl);
+}
+
+static struct scatterlist *crypto_scomp_sg_alloc(size_t size, gfp_t gfp)
+{
+ struct scatterlist *sgl;
+ struct page *page;
+ int i, n;
+
+ n = ((size - 1) >> PAGE_SHIFT) + 1;
+
+ sgl = kmalloc_array(n, sizeof(struct scatterlist), gfp);
+ if (!sgl)
+ return NULL;
+
+ sg_init_table(sgl, n);
+
+ for (i = 0; i < n; i++) {
+ page = alloc_page(gfp);
+ if (!page)
+ goto err;
+ sg_set_page(sgl + i, page, PAGE_SIZE, 0);
+ }
+
+ return sgl;
+
+err:
+ sg_mark_end(sgl + i);
+ crypto_scomp_sg_free(sgl);
+ return NULL;
+}
+
+static int scomp_acomp_comp_decomp(struct acomp_req *req, int dir)
+{
+ struct crypto_acomp *tfm = crypto_acomp_reqtfm(req);
+ void **tfm_ctx = acomp_tfm_ctx(tfm);
+ struct crypto_scomp *scomp = *tfm_ctx;
+ void **ctx = acomp_request_ctx(req);
+ const int cpu = get_cpu();
+ u8 *scratch_src = *per_cpu_ptr(scomp_src_scratches, cpu);
+ u8 *scratch_dst = *per_cpu_ptr(scomp_dst_scratches, cpu);
+ int ret;
+
+ if (!req->src || !req->slen || req->slen > SCOMP_SCRATCH_SIZE) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (req->dst && !req->dlen) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (!req->dlen || req->dlen > SCOMP_SCRATCH_SIZE)
+ req->dlen = SCOMP_SCRATCH_SIZE;
+
+ scatterwalk_map_and_copy(scratch_src, req->src, 0, req->slen, 0);
+ if (dir)
+ ret = crypto_scomp_compress(scomp, scratch_src, req->slen,
+ scratch_dst, &req->dlen, *ctx);
+ else
+ ret = crypto_scomp_decompress(scomp, scratch_src, req->slen,
+ scratch_dst, &req->dlen, *ctx);
+ if (!ret) {
+ if (!req->dst) {
+ req->dst = crypto_scomp_sg_alloc(req->dlen,
+ req->base.flags & CRYPTO_TFM_REQ_MAY_SLEEP ?
+ GFP_KERNEL : GFP_ATOMIC);
+ if (!req->dst)
+ goto out;
+ }
+ scatterwalk_map_and_copy(scratch_dst, req->dst, 0, req->dlen,
+ 1);
+ }
+out:
+ put_cpu();
+ return ret;
+}
+
+static int scomp_acomp_compress(struct acomp_req *req)
+{
+ return scomp_acomp_comp_decomp(req, 1);
+}
+
+static int scomp_acomp_decompress(struct acomp_req *req)
+{
+ return scomp_acomp_comp_decomp(req, 0);
+}
+
+static void crypto_exit_scomp_ops_async(struct crypto_tfm *tfm)
+{
+ struct crypto_scomp **ctx = crypto_tfm_ctx(tfm);
+
+ crypto_free_scomp(*ctx);
+}
+
+int crypto_init_scomp_ops_async(struct crypto_tfm *tfm)
+{
+ struct crypto_alg *calg = tfm->__crt_alg;
+ struct crypto_acomp *crt = __crypto_acomp_tfm(tfm);
+ struct crypto_scomp **ctx = crypto_tfm_ctx(tfm);
+ struct crypto_scomp *scomp;
+
+ if (!crypto_mod_get(calg))
+ return -EAGAIN;
+
+ scomp = crypto_create_tfm(calg, &crypto_scomp_type);
+ if (IS_ERR(scomp)) {
+ crypto_mod_put(calg);
+ return PTR_ERR(scomp);
+ }
+
+ *ctx = scomp;
+ tfm->exit = crypto_exit_scomp_ops_async;
+
+ crt->compress = scomp_acomp_compress;
+ crt->decompress = scomp_acomp_decompress;
+ crt->dst_free = crypto_scomp_sg_free;
+ crt->reqsize = sizeof(void *);
+
+ return 0;
+}
+
+struct acomp_req *crypto_acomp_scomp_alloc_ctx(struct acomp_req *req)
+{
+ struct crypto_acomp *acomp = crypto_acomp_reqtfm(req);
+ struct crypto_tfm *tfm = crypto_acomp_tfm(acomp);
+ struct crypto_scomp **tfm_ctx = crypto_tfm_ctx(tfm);
+ struct crypto_scomp *scomp = *tfm_ctx;
+ void *ctx;
+
+ ctx = crypto_scomp_alloc_ctx(scomp);
+ if (IS_ERR(ctx)) {
+ kfree(req);
+ return NULL;
+ }
+
+ *req->__ctx = ctx;
+
+ return req;
+}
+
+void crypto_acomp_scomp_free_ctx(struct acomp_req *req)
+{
+ struct crypto_acomp *acomp = crypto_acomp_reqtfm(req);
+ struct crypto_tfm *tfm = crypto_acomp_tfm(acomp);
+ struct crypto_scomp **tfm_ctx = crypto_tfm_ctx(tfm);
+ struct crypto_scomp *scomp = *tfm_ctx;
+ void *ctx = *req->__ctx;
+
+ if (ctx)
+ crypto_scomp_free_ctx(scomp, ctx);
+}
+
+static const struct crypto_type crypto_scomp_type = {
+ .extsize = crypto_alg_extsize,
+ .init_tfm = crypto_scomp_init_tfm,
+#ifdef CONFIG_PROC_FS
+ .show = crypto_scomp_show,
+#endif
+ .report = crypto_scomp_report,
+ .maskclear = ~CRYPTO_ALG_TYPE_MASK,
+ .maskset = CRYPTO_ALG_TYPE_MASK,
+ .type = CRYPTO_ALG_TYPE_SCOMPRESS,
+ .tfmsize = offsetof(struct crypto_scomp, base),
+};
+
+int crypto_register_scomp(struct scomp_alg *alg)
+{
+ struct crypto_alg *base = &alg->base;
+ int ret = -ENOMEM;
+
+ mutex_lock(&scomp_lock);
+ if (crypto_scomp_alloc_all_scratches())
+ goto error;
+
+ base->cra_type = &crypto_scomp_type;
+ base->cra_flags &= ~CRYPTO_ALG_TYPE_MASK;
+ base->cra_flags |= CRYPTO_ALG_TYPE_SCOMPRESS;
+
+ ret = crypto_register_alg(base);
+ if (ret)
+ goto error;
+
+ mutex_unlock(&scomp_lock);
+ return ret;
+
+error:
+ crypto_scomp_free_all_scratches();
+ mutex_unlock(&scomp_lock);
+ return ret;
+}
+EXPORT_SYMBOL_GPL(crypto_register_scomp);
+
+int crypto_unregister_scomp(struct scomp_alg *alg)
+{
+ int ret;
+
+ mutex_lock(&scomp_lock);
+ ret = crypto_unregister_alg(&alg->base);
+ crypto_scomp_free_all_scratches();
+ mutex_unlock(&scomp_lock);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(crypto_unregister_scomp);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Synchronous compression type");
diff --git a/crypto/shash.c b/crypto/shash.c
index d5bd2f0..4f047c7 100644
--- a/crypto/shash.c
+++ b/crypto/shash.c
@@ -41,7 +41,7 @@ static int shash_setkey_unaligned(struct crypto_shash *tfm, const u8 *key,
int err;
absize = keylen + (alignmask & ~(crypto_tfm_ctx_alignment() - 1));
- buffer = kmalloc(absize, GFP_KERNEL);
+ buffer = kmalloc(absize, GFP_ATOMIC);
if (!buffer)
return -ENOMEM;
diff --git a/crypto/tcrypt.c b/crypto/tcrypt.c
index 2a07341..babbda2 100644
--- a/crypto/tcrypt.c
+++ b/crypto/tcrypt.c
@@ -729,6 +729,9 @@ static void test_ahash_speed_common(const char *algo, unsigned int secs,
break;
}
+ if (speed[i].klen)
+ crypto_ahash_setkey(tfm, tvmem[0], speed[i].klen);
+
pr_info("test%3u "
"(%5u byte blocks,%5u bytes per update,%4u updates): ",
i, speed[i].blen, speed[i].plen, speed[i].blen / speed[i].plen);
diff --git a/crypto/vmac.c b/crypto/vmac.c
index df76a81..bb2fc78 100644
--- a/crypto/vmac.c
+++ b/crypto/vmac.c
@@ -1,6 +1,10 @@
/*
- * Modified to interface to the Linux kernel
+ * VMAC: Message Authentication Code using Universal Hashing
+ *
+ * Reference: https://tools.ietf.org/html/draft-krovetz-vmac-01
+ *
* Copyright (c) 2009, Intel Corporation.
+ * Copyright (c) 2018, Google Inc.
*
* This program is free software; you can redistribute it and/or modify it
* under the terms and conditions of the GNU General Public License,
@@ -16,14 +20,15 @@
* Place - Suite 330, Boston, MA 02111-1307 USA.
*/
-/* --------------------------------------------------------------------------
- * VMAC and VHASH Implementation by Ted Krovetz (tdk@acm.org) and Wei Dai.
- * This implementation is herby placed in the public domain.
- * The authors offers no warranty. Use at your own risk.
- * Please send bug reports to the authors.
- * Last modified: 17 APR 08, 1700 PDT
- * ----------------------------------------------------------------------- */
+/*
+ * Derived from:
+ * VMAC and VHASH Implementation by Ted Krovetz (tdk@acm.org) and Wei Dai.
+ * This implementation is herby placed in the public domain.
+ * The authors offers no warranty. Use at your own risk.
+ * Last modified: 17 APR 08, 1700 PDT
+ */
+#include <asm/unaligned.h>
#include <linux/init.h>
#include <linux/types.h>
#include <linux/crypto.h>
@@ -31,10 +36,36 @@
#include <linux/scatterlist.h>
#include <asm/byteorder.h>
#include <crypto/scatterwalk.h>
-#include <crypto/vmac.h>
#include <crypto/internal/hash.h>
/*
+ * User definable settings.
+ */
+#define VMAC_TAG_LEN 64
+#define VMAC_KEY_SIZE 128/* Must be 128, 192 or 256 */
+#define VMAC_KEY_LEN (VMAC_KEY_SIZE/8)
+#define VMAC_NHBYTES 128/* Must 2^i for any 3 < i < 13 Standard = 128*/
+
+/* per-transform (per-key) context */
+struct vmac_tfm_ctx {
+ struct crypto_cipher *cipher;
+ u64 nhkey[(VMAC_NHBYTES/8)+2*(VMAC_TAG_LEN/64-1)];
+ u64 polykey[2*VMAC_TAG_LEN/64];
+ u64 l3key[2*VMAC_TAG_LEN/64];
+};
+
+/* per-request context */
+struct vmac_desc_ctx {
+ union {
+ u8 partial[VMAC_NHBYTES]; /* partial block */
+ __le64 partial_words[VMAC_NHBYTES / 8];
+ };
+ unsigned int partial_size; /* size of the partial block */
+ bool first_block_processed;
+ u64 polytmp[2*VMAC_TAG_LEN/64]; /* running total of L2-hash */
+};
+
+/*
* Constants and masks
*/
#define UINT64_C(x) x##ULL
@@ -318,13 +349,6 @@ static void poly_step_func(u64 *ahi, u64 *alo,
} while (0)
#endif
-static void vhash_abort(struct vmac_ctx *ctx)
-{
- ctx->polytmp[0] = ctx->polykey[0] ;
- ctx->polytmp[1] = ctx->polykey[1] ;
- ctx->first_block_processed = 0;
-}
-
static u64 l3hash(u64 p1, u64 p2, u64 k1, u64 k2, u64 len)
{
u64 rh, rl, t, z = 0;
@@ -364,280 +388,209 @@ static u64 l3hash(u64 p1, u64 p2, u64 k1, u64 k2, u64 len)
return rl;
}
-static void vhash_update(const unsigned char *m,
- unsigned int mbytes, /* Pos multiple of VMAC_NHBYTES */
- struct vmac_ctx *ctx)
+/* L1 and L2-hash one or more VMAC_NHBYTES-byte blocks */
+static void vhash_blocks(const struct vmac_tfm_ctx *tctx,
+ struct vmac_desc_ctx *dctx,
+ const __le64 *mptr, unsigned int blocks)
{
- u64 rh, rl, *mptr;
- const u64 *kptr = (u64 *)ctx->nhkey;
- int i;
- u64 ch, cl;
- u64 pkh = ctx->polykey[0];
- u64 pkl = ctx->polykey[1];
+ const u64 *kptr = tctx->nhkey;
+ const u64 pkh = tctx->polykey[0];
+ const u64 pkl = tctx->polykey[1];
+ u64 ch = dctx->polytmp[0];
+ u64 cl = dctx->polytmp[1];
+ u64 rh, rl;
- if (!mbytes)
- return;
-
- BUG_ON(mbytes % VMAC_NHBYTES);
-
- mptr = (u64 *)m;
- i = mbytes / VMAC_NHBYTES; /* Must be non-zero */
-
- ch = ctx->polytmp[0];
- cl = ctx->polytmp[1];
-
- if (!ctx->first_block_processed) {
- ctx->first_block_processed = 1;
+ if (!dctx->first_block_processed) {
+ dctx->first_block_processed = true;
nh_vmac_nhbytes(mptr, kptr, VMAC_NHBYTES/8, rh, rl);
rh &= m62;
ADD128(ch, cl, rh, rl);
mptr += (VMAC_NHBYTES/sizeof(u64));
- i--;
+ blocks--;
}
- while (i--) {
+ while (blocks--) {
nh_vmac_nhbytes(mptr, kptr, VMAC_NHBYTES/8, rh, rl);
rh &= m62;
poly_step(ch, cl, pkh, pkl, rh, rl);
mptr += (VMAC_NHBYTES/sizeof(u64));
}
- ctx->polytmp[0] = ch;
- ctx->polytmp[1] = cl;
+ dctx->polytmp[0] = ch;
+ dctx->polytmp[1] = cl;
}
-static u64 vhash(unsigned char m[], unsigned int mbytes,
- u64 *tagl, struct vmac_ctx *ctx)
+static int vmac_setkey(struct crypto_shash *tfm,
+ const u8 *key, unsigned int keylen)
{
- u64 rh, rl, *mptr;
- const u64 *kptr = (u64 *)ctx->nhkey;
- int i, remaining;
- u64 ch, cl;
- u64 pkh = ctx->polykey[0];
- u64 pkl = ctx->polykey[1];
+ struct vmac_tfm_ctx *tctx = crypto_shash_ctx(tfm);
+ __be64 out[2];
+ u8 in[16] = { 0 };
+ unsigned int i;
+ int err;
- mptr = (u64 *)m;
- i = mbytes / VMAC_NHBYTES;
- remaining = mbytes % VMAC_NHBYTES;
-
- if (ctx->first_block_processed) {
- ch = ctx->polytmp[0];
- cl = ctx->polytmp[1];
- } else if (i) {
- nh_vmac_nhbytes(mptr, kptr, VMAC_NHBYTES/8, ch, cl);
- ch &= m62;
- ADD128(ch, cl, pkh, pkl);
- mptr += (VMAC_NHBYTES/sizeof(u64));
- i--;
- } else if (remaining) {
- nh_16(mptr, kptr, 2*((remaining+15)/16), ch, cl);
- ch &= m62;
- ADD128(ch, cl, pkh, pkl);
- mptr += (VMAC_NHBYTES/sizeof(u64));
- goto do_l3;
- } else {/* Empty String */
- ch = pkh; cl = pkl;
- goto do_l3;
+ if (keylen != VMAC_KEY_LEN) {
+ crypto_shash_set_flags(tfm, CRYPTO_TFM_RES_BAD_KEY_LEN);
+ return -EINVAL;
}
- while (i--) {
- nh_vmac_nhbytes(mptr, kptr, VMAC_NHBYTES/8, rh, rl);
- rh &= m62;
- poly_step(ch, cl, pkh, pkl, rh, rl);
- mptr += (VMAC_NHBYTES/sizeof(u64));
- }
- if (remaining) {
- nh_16(mptr, kptr, 2*((remaining+15)/16), rh, rl);
- rh &= m62;
- poly_step(ch, cl, pkh, pkl, rh, rl);
- }
-
-do_l3:
- vhash_abort(ctx);
- remaining *= 8;
- return l3hash(ch, cl, ctx->l3key[0], ctx->l3key[1], remaining);
-}
-
-static u64 vmac(unsigned char m[], unsigned int mbytes,
- const unsigned char n[16], u64 *tagl,
- struct vmac_ctx_t *ctx)
-{
- u64 *in_n, *out_p;
- u64 p, h;
- int i;
-
- in_n = ctx->__vmac_ctx.cached_nonce;
- out_p = ctx->__vmac_ctx.cached_aes;
-
- i = n[15] & 1;
- if ((*(u64 *)(n+8) != in_n[1]) || (*(u64 *)(n) != in_n[0])) {
- in_n[0] = *(u64 *)(n);
- in_n[1] = *(u64 *)(n+8);
- ((unsigned char *)in_n)[15] &= 0xFE;
- crypto_cipher_encrypt_one(ctx->child,
- (unsigned char *)out_p, (unsigned char *)in_n);
-
- ((unsigned char *)in_n)[15] |= (unsigned char)(1-i);
- }
- p = be64_to_cpup(out_p + i);
- h = vhash(m, mbytes, (u64 *)0, &ctx->__vmac_ctx);
- return le64_to_cpu(p + h);
-}
-
-static int vmac_set_key(unsigned char user_key[], struct vmac_ctx_t *ctx)
-{
- u64 in[2] = {0}, out[2];
- unsigned i;
- int err = 0;
-
- err = crypto_cipher_setkey(ctx->child, user_key, VMAC_KEY_LEN);
+ err = crypto_cipher_setkey(tctx->cipher, key, keylen);
if (err)
return err;
/* Fill nh key */
- ((unsigned char *)in)[0] = 0x80;
- for (i = 0; i < sizeof(ctx->__vmac_ctx.nhkey)/8; i += 2) {
- crypto_cipher_encrypt_one(ctx->child,
- (unsigned char *)out, (unsigned char *)in);
- ctx->__vmac_ctx.nhkey[i] = be64_to_cpup(out);
- ctx->__vmac_ctx.nhkey[i+1] = be64_to_cpup(out+1);
- ((unsigned char *)in)[15] += 1;
+ in[0] = 0x80;
+ for (i = 0; i < ARRAY_SIZE(tctx->nhkey); i += 2) {
+ crypto_cipher_encrypt_one(tctx->cipher, (u8 *)out, in);
+ tctx->nhkey[i] = be64_to_cpu(out[0]);
+ tctx->nhkey[i+1] = be64_to_cpu(out[1]);
+ in[15]++;
}
/* Fill poly key */
- ((unsigned char *)in)[0] = 0xC0;
- in[1] = 0;
- for (i = 0; i < sizeof(ctx->__vmac_ctx.polykey)/8; i += 2) {
- crypto_cipher_encrypt_one(ctx->child,
- (unsigned char *)out, (unsigned char *)in);
- ctx->__vmac_ctx.polytmp[i] =
- ctx->__vmac_ctx.polykey[i] =
- be64_to_cpup(out) & mpoly;
- ctx->__vmac_ctx.polytmp[i+1] =
- ctx->__vmac_ctx.polykey[i+1] =
- be64_to_cpup(out+1) & mpoly;
- ((unsigned char *)in)[15] += 1;
+ in[0] = 0xC0;
+ in[15] = 0;
+ for (i = 0; i < ARRAY_SIZE(tctx->polykey); i += 2) {
+ crypto_cipher_encrypt_one(tctx->cipher, (u8 *)out, in);
+ tctx->polykey[i] = be64_to_cpu(out[0]) & mpoly;
+ tctx->polykey[i+1] = be64_to_cpu(out[1]) & mpoly;
+ in[15]++;
}
/* Fill ip key */
- ((unsigned char *)in)[0] = 0xE0;
- in[1] = 0;
- for (i = 0; i < sizeof(ctx->__vmac_ctx.l3key)/8; i += 2) {
+ in[0] = 0xE0;
+ in[15] = 0;
+ for (i = 0; i < ARRAY_SIZE(tctx->l3key); i += 2) {
do {
- crypto_cipher_encrypt_one(ctx->child,
- (unsigned char *)out, (unsigned char *)in);
- ctx->__vmac_ctx.l3key[i] = be64_to_cpup(out);
- ctx->__vmac_ctx.l3key[i+1] = be64_to_cpup(out+1);
- ((unsigned char *)in)[15] += 1;
- } while (ctx->__vmac_ctx.l3key[i] >= p64
- || ctx->__vmac_ctx.l3key[i+1] >= p64);
+ crypto_cipher_encrypt_one(tctx->cipher, (u8 *)out, in);
+ tctx->l3key[i] = be64_to_cpu(out[0]);
+ tctx->l3key[i+1] = be64_to_cpu(out[1]);
+ in[15]++;
+ } while (tctx->l3key[i] >= p64 || tctx->l3key[i+1] >= p64);
}
- /* Invalidate nonce/aes cache and reset other elements */
- ctx->__vmac_ctx.cached_nonce[0] = (u64)-1; /* Ensure illegal nonce */
- ctx->__vmac_ctx.cached_nonce[1] = (u64)0; /* Ensure illegal nonce */
- ctx->__vmac_ctx.first_block_processed = 0;
-
- return err;
-}
-
-static int vmac_setkey(struct crypto_shash *parent,
- const u8 *key, unsigned int keylen)
-{
- struct vmac_ctx_t *ctx = crypto_shash_ctx(parent);
-
- if (keylen != VMAC_KEY_LEN) {
- crypto_shash_set_flags(parent, CRYPTO_TFM_RES_BAD_KEY_LEN);
- return -EINVAL;
- }
-
- return vmac_set_key((u8 *)key, ctx);
-}
-
-static int vmac_init(struct shash_desc *pdesc)
-{
- return 0;
-}
-
-static int vmac_update(struct shash_desc *pdesc, const u8 *p,
- unsigned int len)
-{
- struct crypto_shash *parent = pdesc->tfm;
- struct vmac_ctx_t *ctx = crypto_shash_ctx(parent);
- int expand;
- int min;
-
- expand = VMAC_NHBYTES - ctx->partial_size > 0 ?
- VMAC_NHBYTES - ctx->partial_size : 0;
-
- min = len < expand ? len : expand;
-
- memcpy(ctx->partial + ctx->partial_size, p, min);
- ctx->partial_size += min;
-
- if (len < expand)
- return 0;
-
- vhash_update(ctx->partial, VMAC_NHBYTES, &ctx->__vmac_ctx);
- ctx->partial_size = 0;
-
- len -= expand;
- p += expand;
-
- if (len % VMAC_NHBYTES) {
- memcpy(ctx->partial, p + len - (len % VMAC_NHBYTES),
- len % VMAC_NHBYTES);
- ctx->partial_size = len % VMAC_NHBYTES;
- }
-
- vhash_update(p, len - len % VMAC_NHBYTES, &ctx->__vmac_ctx);
-
return 0;
}
-static int vmac_final(struct shash_desc *pdesc, u8 *out)
+static int vmac_init(struct shash_desc *desc)
{
- struct crypto_shash *parent = pdesc->tfm;
- struct vmac_ctx_t *ctx = crypto_shash_ctx(parent);
- vmac_t mac;
- u8 nonce[16] = {};
+ const struct vmac_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
+ struct vmac_desc_ctx *dctx = shash_desc_ctx(desc);
- /* vmac() ends up accessing outside the array bounds that
- * we specify. In appears to access up to the next 2-word
- * boundary. We'll just be uber cautious and zero the
- * unwritten bytes in the buffer.
- */
- if (ctx->partial_size) {
- memset(ctx->partial + ctx->partial_size, 0,
- VMAC_NHBYTES - ctx->partial_size);
+ dctx->partial_size = 0;
+ dctx->first_block_processed = false;
+ memcpy(dctx->polytmp, tctx->polykey, sizeof(dctx->polytmp));
+ return 0;
+}
+
+static int vmac_update(struct shash_desc *desc, const u8 *p, unsigned int len)
+{
+ const struct vmac_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
+ struct vmac_desc_ctx *dctx = shash_desc_ctx(desc);
+ unsigned int n;
+
+ if (dctx->partial_size) {
+ n = min(len, VMAC_NHBYTES - dctx->partial_size);
+ memcpy(&dctx->partial[dctx->partial_size], p, n);
+ dctx->partial_size += n;
+ p += n;
+ len -= n;
+ if (dctx->partial_size == VMAC_NHBYTES) {
+ vhash_blocks(tctx, dctx, dctx->partial_words, 1);
+ dctx->partial_size = 0;
+ }
}
- mac = vmac(ctx->partial, ctx->partial_size, nonce, NULL, ctx);
- memcpy(out, &mac, sizeof(vmac_t));
- memzero_explicit(&mac, sizeof(vmac_t));
- memset(&ctx->__vmac_ctx, 0, sizeof(struct vmac_ctx));
- ctx->partial_size = 0;
+
+ if (len >= VMAC_NHBYTES) {
+ n = round_down(len, VMAC_NHBYTES);
+ /* TODO: 'p' may be misaligned here */
+ vhash_blocks(tctx, dctx, (const __le64 *)p, n / VMAC_NHBYTES);
+ p += n;
+ len -= n;
+ }
+
+ if (len) {
+ memcpy(dctx->partial, p, len);
+ dctx->partial_size = len;
+ }
+
+ return 0;
+}
+
+static u64 vhash_final(const struct vmac_tfm_ctx *tctx,
+ struct vmac_desc_ctx *dctx)
+{
+ unsigned int partial = dctx->partial_size;
+ u64 ch = dctx->polytmp[0];
+ u64 cl = dctx->polytmp[1];
+
+ /* L1 and L2-hash the final block if needed */
+ if (partial) {
+ /* Zero-pad to next 128-bit boundary */
+ unsigned int n = round_up(partial, 16);
+ u64 rh, rl;
+
+ memset(&dctx->partial[partial], 0, n - partial);
+ nh_16(dctx->partial_words, tctx->nhkey, n / 8, rh, rl);
+ rh &= m62;
+ if (dctx->first_block_processed)
+ poly_step(ch, cl, tctx->polykey[0], tctx->polykey[1],
+ rh, rl);
+ else
+ ADD128(ch, cl, rh, rl);
+ }
+
+ /* L3-hash the 128-bit output of L2-hash */
+ return l3hash(ch, cl, tctx->l3key[0], tctx->l3key[1], partial * 8);
+}
+
+static int vmac_final(struct shash_desc *desc, u8 *out)
+{
+ const struct vmac_tfm_ctx *tctx = crypto_shash_ctx(desc->tfm);
+ struct vmac_desc_ctx *dctx = shash_desc_ctx(desc);
+ static const u8 nonce[16] = {}; /* TODO: this is insecure */
+ union {
+ u8 bytes[16];
+ __be64 pads[2];
+ } block;
+ int index;
+ u64 hash, pad;
+
+ /* Finish calculating the VHASH of the message */
+ hash = vhash_final(tctx, dctx);
+
+ /* Generate pseudorandom pad by encrypting the nonce */
+ memcpy(&block, nonce, 16);
+ index = block.bytes[15] & 1;
+ block.bytes[15] &= ~1;
+ crypto_cipher_encrypt_one(tctx->cipher, block.bytes, block.bytes);
+ pad = be64_to_cpu(block.pads[index]);
+
+ /* The VMAC is the sum of VHASH and the pseudorandom pad */
+ put_unaligned_le64(hash + pad, out);
return 0;
}
static int vmac_init_tfm(struct crypto_tfm *tfm)
{
- struct crypto_cipher *cipher;
- struct crypto_instance *inst = (void *)tfm->__crt_alg;
+ struct crypto_instance *inst = crypto_tfm_alg_instance(tfm);
struct crypto_spawn *spawn = crypto_instance_ctx(inst);
- struct vmac_ctx_t *ctx = crypto_tfm_ctx(tfm);
+ struct vmac_tfm_ctx *tctx = crypto_tfm_ctx(tfm);
+ struct crypto_cipher *cipher;
cipher = crypto_spawn_cipher(spawn);
if (IS_ERR(cipher))
return PTR_ERR(cipher);
- ctx->child = cipher;
+ tctx->cipher = cipher;
return 0;
}
static void vmac_exit_tfm(struct crypto_tfm *tfm)
{
- struct vmac_ctx_t *ctx = crypto_tfm_ctx(tfm);
- crypto_free_cipher(ctx->child);
+ struct vmac_tfm_ctx *tctx = crypto_tfm_ctx(tfm);
+
+ crypto_free_cipher(tctx->cipher);
}
static int vmac_create(struct crypto_template *tmpl, struct rtattr **tb)
@@ -655,6 +608,10 @@ static int vmac_create(struct crypto_template *tmpl, struct rtattr **tb)
if (IS_ERR(alg))
return PTR_ERR(alg);
+ err = -EINVAL;
+ if (alg->cra_blocksize != 16)
+ goto out_put_alg;
+
inst = shash_alloc_instance("vmac", alg);
err = PTR_ERR(inst);
if (IS_ERR(inst))
@@ -670,11 +627,12 @@ static int vmac_create(struct crypto_template *tmpl, struct rtattr **tb)
inst->alg.base.cra_blocksize = alg->cra_blocksize;
inst->alg.base.cra_alignmask = alg->cra_alignmask;
- inst->alg.digestsize = sizeof(vmac_t);
- inst->alg.base.cra_ctxsize = sizeof(struct vmac_ctx_t);
+ inst->alg.base.cra_ctxsize = sizeof(struct vmac_tfm_ctx);
inst->alg.base.cra_init = vmac_init_tfm;
inst->alg.base.cra_exit = vmac_exit_tfm;
+ inst->alg.descsize = sizeof(struct vmac_desc_ctx);
+ inst->alg.digestsize = VMAC_TAG_LEN / 8;
inst->alg.init = vmac_init;
inst->alg.update = vmac_update;
inst->alg.final = vmac_final;
diff --git a/drivers/acpi/acpi_lpss.c b/drivers/acpi/acpi_lpss.c
index 373657f..957d3fa 100644
--- a/drivers/acpi/acpi_lpss.c
+++ b/drivers/acpi/acpi_lpss.c
@@ -187,10 +187,12 @@ static const struct lpss_device_desc lpt_sdio_dev_desc = {
static const struct lpss_device_desc byt_pwm_dev_desc = {
.flags = LPSS_SAVE_CTX,
+ .prv_offset = 0x800,
};
static const struct lpss_device_desc bsw_pwm_dev_desc = {
.flags = LPSS_SAVE_CTX | LPSS_NO_D3_DELAY,
+ .prv_offset = 0x800,
};
static const struct lpss_device_desc byt_uart_dev_desc = {
@@ -276,9 +278,11 @@ static const struct acpi_device_id acpi_lpss_device_ids[] = {
{ "INT33FC", },
/* Braswell LPSS devices */
+ { "80862286", LPSS_ADDR(lpss_dma_desc) },
{ "80862288", LPSS_ADDR(bsw_pwm_dev_desc) },
{ "8086228A", LPSS_ADDR(bsw_uart_dev_desc) },
{ "8086228E", LPSS_ADDR(bsw_spi_dev_desc) },
+ { "808622C0", LPSS_ADDR(lpss_dma_desc) },
{ "808622C1", LPSS_ADDR(bsw_i2c_dev_desc) },
/* Broadwell LPSS devices */
diff --git a/drivers/acpi/acpi_pad.c b/drivers/acpi/acpi_pad.c
index eb76a4c..8ce203f 100644
--- a/drivers/acpi/acpi_pad.c
+++ b/drivers/acpi/acpi_pad.c
@@ -109,6 +109,7 @@ static void round_robin_cpu(unsigned int tsk_index)
cpumask_andnot(tmp, cpu_online_mask, pad_busy_cpus);
if (cpumask_empty(tmp)) {
mutex_unlock(&round_robin_lock);
+ free_cpumask_var(tmp);
return;
}
for_each_cpu(cpu, tmp) {
@@ -126,6 +127,8 @@ static void round_robin_cpu(unsigned int tsk_index)
mutex_unlock(&round_robin_lock);
set_cpus_allowed_ptr(current, cpumask_of(preferred_cpu));
+
+ free_cpumask_var(tmp);
}
static void exit_round_robin(unsigned int tsk_index)
diff --git a/drivers/acpi/acpi_platform.c b/drivers/acpi/acpi_platform.c
index 03250e1f..d92eaca 100644
--- a/drivers/acpi/acpi_platform.c
+++ b/drivers/acpi/acpi_platform.c
@@ -30,6 +30,7 @@ static const struct acpi_device_id forbidden_id_list[] = {
{"PNP0200", 0}, /* AT DMA Controller */
{"ACPI0009", 0}, /* IOxAPIC */
{"ACPI000A", 0}, /* IOAPIC */
+ {"SMB0001", 0}, /* ACPI SMBUS virtual device */
{"", 0},
};
diff --git a/drivers/acpi/acpi_video.c b/drivers/acpi/acpi_video.c
index 94e04c9..667dc5c 100644
--- a/drivers/acpi/acpi_video.c
+++ b/drivers/acpi/acpi_video.c
@@ -2069,6 +2069,25 @@ static int __init intel_opregion_present(void)
return opregion;
}
+static bool dmi_is_desktop(void)
+{
+ const char *chassis_type;
+
+ chassis_type = dmi_get_system_info(DMI_CHASSIS_TYPE);
+ if (!chassis_type)
+ return false;
+
+ if (!strcmp(chassis_type, "3") || /* 3: Desktop */
+ !strcmp(chassis_type, "4") || /* 4: Low Profile Desktop */
+ !strcmp(chassis_type, "5") || /* 5: Pizza Box */
+ !strcmp(chassis_type, "6") || /* 6: Mini Tower */
+ !strcmp(chassis_type, "7") || /* 7: Tower */
+ !strcmp(chassis_type, "11")) /* 11: Main Server Chassis */
+ return true;
+
+ return false;
+}
+
int acpi_video_register(void)
{
int ret = 0;
@@ -2089,8 +2108,12 @@ int acpi_video_register(void)
* win8 ready (where we also prefer the native backlight driver, so
* normally the acpi_video code should not register there anyways).
*/
- if (only_lcd == -1)
- only_lcd = acpi_osi_is_win8();
+ if (only_lcd == -1) {
+ if (dmi_is_desktop() && acpi_osi_is_win8())
+ only_lcd = true;
+ else
+ only_lcd = false;
+ }
dmi_check_system(video_dmi_table);
diff --git a/drivers/acpi/acpi_watchdog.c b/drivers/acpi/acpi_watchdog.c
index ce8fc68..396e358 100644
--- a/drivers/acpi/acpi_watchdog.c
+++ b/drivers/acpi/acpi_watchdog.c
@@ -17,18 +17,77 @@
#include "internal.h"
+#ifdef CONFIG_RTC_MC146818_LIB
+#include <linux/mc146818rtc.h>
+
+/*
+ * There are several systems where the WDAT table is accessing RTC SRAM to
+ * store persistent information. This does not work well with the Linux RTC
+ * driver so on those systems we skip WDAT driver and prefer iTCO_wdt
+ * instead.
+ *
+ * See also https://bugzilla.kernel.org/show_bug.cgi?id=199033.
+ */
+static bool acpi_watchdog_uses_rtc(const struct acpi_table_wdat *wdat)
+{
+ const struct acpi_wdat_entry *entries;
+ int i;
+
+ entries = (struct acpi_wdat_entry *)(wdat + 1);
+ for (i = 0; i < wdat->entries; i++) {
+ const struct acpi_generic_address *gas;
+
+ gas = &entries[i].register_region;
+ if (gas->space_id == ACPI_ADR_SPACE_SYSTEM_IO) {
+ switch (gas->address) {
+ case RTC_PORT(0):
+ case RTC_PORT(1):
+ case RTC_PORT(2):
+ case RTC_PORT(3):
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+#else
+static bool acpi_watchdog_uses_rtc(const struct acpi_table_wdat *wdat)
+{
+ return false;
+}
+#endif
+
+static const struct acpi_table_wdat *acpi_watchdog_get_wdat(void)
+{
+ const struct acpi_table_wdat *wdat = NULL;
+ acpi_status status;
+
+ if (acpi_disabled)
+ return NULL;
+
+ status = acpi_get_table(ACPI_SIG_WDAT, 0,
+ (struct acpi_table_header **)&wdat);
+ if (ACPI_FAILURE(status)) {
+ /* It is fine if there is no WDAT */
+ return NULL;
+ }
+
+ if (acpi_watchdog_uses_rtc(wdat)) {
+ pr_info("Skipping WDAT on this system because it uses RTC SRAM\n");
+ return NULL;
+ }
+
+ return wdat;
+}
+
/**
* Returns true if this system should prefer ACPI based watchdog instead of
* the native one (which are typically the same hardware).
*/
bool acpi_has_watchdog(void)
{
- struct acpi_table_header hdr;
-
- if (acpi_disabled)
- return false;
-
- return ACPI_SUCCESS(acpi_get_table_header(ACPI_SIG_WDAT, 0, &hdr));
+ return !!acpi_watchdog_get_wdat();
}
EXPORT_SYMBOL_GPL(acpi_has_watchdog);
@@ -41,12 +100,10 @@ void __init acpi_watchdog_init(void)
struct platform_device *pdev;
struct resource *resources;
size_t nresources = 0;
- acpi_status status;
int i;
- status = acpi_get_table(ACPI_SIG_WDAT, 0,
- (struct acpi_table_header **)&wdat);
- if (ACPI_FAILURE(status)) {
+ wdat = acpi_watchdog_get_wdat();
+ if (!wdat) {
/* It is fine if there is no WDAT */
return;
}
diff --git a/drivers/acpi/acpica/evevent.c b/drivers/acpi/acpica/evevent.c
index 80fc0b9..f362841 100644
--- a/drivers/acpi/acpica/evevent.c
+++ b/drivers/acpi/acpica/evevent.c
@@ -204,6 +204,7 @@ u32 acpi_ev_fixed_event_detect(void)
u32 fixed_status;
u32 fixed_enable;
u32 i;
+ acpi_status status;
ACPI_FUNCTION_NAME(ev_fixed_event_detect);
@@ -211,8 +212,12 @@ u32 acpi_ev_fixed_event_detect(void)
* Read the fixed feature status and enable registers, as all the cases
* depend on their values. Ignore errors here.
*/
- (void)acpi_hw_register_read(ACPI_REGISTER_PM1_STATUS, &fixed_status);
- (void)acpi_hw_register_read(ACPI_REGISTER_PM1_ENABLE, &fixed_enable);
+ status = acpi_hw_register_read(ACPI_REGISTER_PM1_STATUS, &fixed_status);
+ status |=
+ acpi_hw_register_read(ACPI_REGISTER_PM1_ENABLE, &fixed_enable);
+ if (ACPI_FAILURE(status)) {
+ return (int_status);
+ }
ACPI_DEBUG_PRINT((ACPI_DB_INTERRUPTS,
"Fixed Event Block: Enable %08X Status %08X\n",
diff --git a/drivers/acpi/acpica/nseval.c b/drivers/acpi/acpica/nseval.c
index 5d59cfc..c5d6701 100644
--- a/drivers/acpi/acpica/nseval.c
+++ b/drivers/acpi/acpica/nseval.c
@@ -308,6 +308,14 @@ acpi_status acpi_ns_evaluate(struct acpi_evaluate_info *info)
/* Map AE_CTRL_RETURN_VALUE to AE_OK, we are done with it */
status = AE_OK;
+ } else if (ACPI_FAILURE(status)) {
+
+ /* If return_object exists, delete it */
+
+ if (info->return_object) {
+ acpi_ut_remove_reference(info->return_object);
+ info->return_object = NULL;
+ }
}
ACPI_DEBUG_PRINT((ACPI_DB_NAMES,
diff --git a/drivers/acpi/nfit/core.c b/drivers/acpi/nfit/core.c
index 3874eec..ef32e57 100644
--- a/drivers/acpi/nfit/core.c
+++ b/drivers/acpi/nfit/core.c
@@ -201,6 +201,8 @@ int acpi_nfit_ctl(struct nvdimm_bus_descriptor *nd_desc, struct nvdimm *nvdimm,
const u8 *uuid;
int rc, i;
+ if (cmd_rc)
+ *cmd_rc = -EINVAL;
func = cmd;
if (cmd == ND_CMD_CALL) {
call_pkg = buf;
@@ -288,6 +290,8 @@ int acpi_nfit_ctl(struct nvdimm_bus_descriptor *nd_desc, struct nvdimm *nvdimm,
* If we return an error (like elsewhere) then caller wouldn't
* be able to rely upon data returned to make calculation.
*/
+ if (cmd_rc)
+ *cmd_rc = 0;
return 0;
}
diff --git a/drivers/acpi/pci_root.c b/drivers/acpi/pci_root.c
index bf601d4..b66815f 100644
--- a/drivers/acpi/pci_root.c
+++ b/drivers/acpi/pci_root.c
@@ -472,9 +472,11 @@ static void negotiate_os_control(struct acpi_pci_root *root, int *no_aspm)
}
control = OSC_PCI_EXPRESS_CAPABILITY_CONTROL
- | OSC_PCI_EXPRESS_NATIVE_HP_CONTROL
| OSC_PCI_EXPRESS_PME_CONTROL;
+ if (IS_ENABLED(CONFIG_HOTPLUG_PCI_PCIE))
+ control |= OSC_PCI_EXPRESS_NATIVE_HP_CONTROL;
+
if (pci_aer_available()) {
if (aer_acpi_firmware_first())
dev_info(&device->dev,
diff --git a/drivers/acpi/processor_perflib.c b/drivers/acpi/processor_perflib.c
index bb01dea..9825780 100644
--- a/drivers/acpi/processor_perflib.c
+++ b/drivers/acpi/processor_perflib.c
@@ -161,7 +161,7 @@ int acpi_processor_ppc_has_changed(struct acpi_processor *pr, int event_flag)
{
int ret;
- if (ignore_ppc) {
+ if (ignore_ppc || !pr->performance) {
/*
* Only when it is notification event, the _OST object
* will be evaluated. Otherwise it is skipped.
diff --git a/drivers/acpi/scan.c b/drivers/acpi/scan.c
index cf725d5..0792ec5 100644
--- a/drivers/acpi/scan.c
+++ b/drivers/acpi/scan.c
@@ -1422,6 +1422,8 @@ void acpi_init_device_object(struct acpi_device *device, acpi_handle handle,
device_initialize(&device->dev);
dev_set_uevent_suppress(&device->dev, true);
acpi_init_coherency(device);
+ /* Assume there are unmet deps until acpi_device_dep_initialize() runs */
+ device->dep_unmet = 1;
}
void acpi_device_add_finalize(struct acpi_device *device)
@@ -1445,6 +1447,15 @@ static int acpi_add_single_object(struct acpi_device **child,
}
acpi_init_device_object(device, handle, type, sta);
+ /*
+ * For ACPI_BUS_TYPE_DEVICE getting the status is delayed till here so
+ * that we can call acpi_bus_get_status() and use its quirk handling.
+ * Note this must be done before the get power-/wakeup_dev-flags calls.
+ */
+ if (type == ACPI_BUS_TYPE_DEVICE)
+ if (acpi_bus_get_status(device) < 0)
+ acpi_set_device_status(device, 0);
+
acpi_bus_get_power_flags(device);
acpi_bus_get_wakeup_device_flags(device);
@@ -1517,9 +1528,11 @@ static int acpi_bus_type_and_status(acpi_handle handle, int *type,
return -ENODEV;
*type = ACPI_BUS_TYPE_DEVICE;
- status = acpi_bus_get_status_handle(handle, sta);
- if (ACPI_FAILURE(status))
- *sta = 0;
+ /*
+ * acpi_add_single_object updates this once we've an acpi_device
+ * so that acpi_bus_get_status' quirk handling can be used.
+ */
+ *sta = ACPI_STA_DEFAULT;
break;
case ACPI_TYPE_PROCESSOR:
*type = ACPI_BUS_TYPE_PROCESSOR;
@@ -1621,6 +1634,8 @@ static void acpi_device_dep_initialize(struct acpi_device *adev)
acpi_status status;
int i;
+ adev->dep_unmet = 0;
+
if (!acpi_has_method(adev->handle, "_DEP"))
return;
diff --git a/drivers/acpi/sleep.c b/drivers/acpi/sleep.c
index 097d630..f0633169 100644
--- a/drivers/acpi/sleep.c
+++ b/drivers/acpi/sleep.c
@@ -330,6 +330,14 @@ static struct dmi_system_id acpisleep_dmi_table[] __initdata = {
DMI_MATCH(DMI_PRODUCT_NAME, "K54HR"),
},
},
+ {
+ .callback = init_nvs_save_s3,
+ .ident = "Asus 1025C",
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."),
+ DMI_MATCH(DMI_PRODUCT_NAME, "1025C"),
+ },
+ },
/*
* https://bugzilla.kernel.org/show_bug.cgi?id=189431
* Lenovo G50-45 is a platform later than 2012, but needs nvs memory
diff --git a/drivers/acpi/sysfs.c b/drivers/acpi/sysfs.c
index 34328e1..cf2a17b 100644
--- a/drivers/acpi/sysfs.c
+++ b/drivers/acpi/sysfs.c
@@ -726,14 +726,8 @@ static ssize_t counter_set(struct kobject *kobj,
* interface:
* echo unmask > /sys/firmware/acpi/interrupts/gpe00
*/
-
-/*
- * Currently, the GPE flooding prevention only supports to mask the GPEs
- * numbered from 00 to 7f.
- */
-#define ACPI_MASKABLE_GPE_MAX 0x80
-
-static u64 __initdata acpi_masked_gpes;
+#define ACPI_MASKABLE_GPE_MAX 0xFF
+static DECLARE_BITMAP(acpi_masked_gpes_map, ACPI_MASKABLE_GPE_MAX) __initdata;
static int __init acpi_gpe_set_masked_gpes(char *val)
{
@@ -741,7 +735,7 @@ static int __init acpi_gpe_set_masked_gpes(char *val)
if (kstrtou8(val, 0, &gpe) || gpe > ACPI_MASKABLE_GPE_MAX)
return -EINVAL;
- acpi_masked_gpes |= ((u64)1<<gpe);
+ set_bit(gpe, acpi_masked_gpes_map);
return 1;
}
@@ -753,15 +747,11 @@ void __init acpi_gpe_apply_masked_gpes(void)
acpi_status status;
u8 gpe;
- for (gpe = 0;
- gpe < min_t(u8, ACPI_MASKABLE_GPE_MAX, acpi_current_gpe_count);
- gpe++) {
- if (acpi_masked_gpes & ((u64)1<<gpe)) {
- status = acpi_get_gpe_device(gpe, &handle);
- if (ACPI_SUCCESS(status)) {
- pr_info("Masking GPE 0x%x.\n", gpe);
- (void)acpi_mask_gpe(handle, gpe, TRUE);
- }
+ for_each_set_bit(gpe, acpi_masked_gpes_map, ACPI_MASKABLE_GPE_MAX) {
+ status = acpi_get_gpe_device(gpe, &handle);
+ if (ACPI_SUCCESS(status)) {
+ pr_info("Masking GPE 0x%x.\n", gpe);
+ (void)acpi_mask_gpe(handle, gpe, TRUE);
}
}
}
diff --git a/drivers/amba/bus.c b/drivers/amba/bus.c
index 7394aac..93888cc 100644
--- a/drivers/amba/bus.c
+++ b/drivers/amba/bus.c
@@ -69,11 +69,12 @@ static ssize_t driver_override_show(struct device *_dev,
struct device_attribute *attr, char *buf)
{
struct amba_device *dev = to_amba_device(_dev);
+ ssize_t len;
- if (!dev->driver_override)
- return 0;
-
- return sprintf(buf, "%s\n", dev->driver_override);
+ device_lock(_dev);
+ len = sprintf(buf, "%s\n", dev->driver_override);
+ device_unlock(_dev);
+ return len;
}
static ssize_t driver_override_store(struct device *_dev,
@@ -81,7 +82,7 @@ static ssize_t driver_override_store(struct device *_dev,
const char *buf, size_t count)
{
struct amba_device *dev = to_amba_device(_dev);
- char *driver_override, *old = dev->driver_override, *cp;
+ char *driver_override, *old, *cp;
/* We need to keep extra room for a newline */
if (count >= (PAGE_SIZE - 1))
@@ -95,12 +96,15 @@ static ssize_t driver_override_store(struct device *_dev,
if (cp)
*cp = '\0';
+ device_lock(_dev);
+ old = dev->driver_override;
if (strlen(driver_override)) {
dev->driver_override = driver_override;
} else {
kfree(driver_override);
dev->driver_override = NULL;
}
+ device_unlock(_dev);
kfree(old);
diff --git a/drivers/android/binder.c b/drivers/android/binder.c
index 0c1ef82..4768de0 100644
--- a/drivers/android/binder.c
+++ b/drivers/android/binder.c
@@ -360,6 +360,8 @@ struct binder_error {
* (invariant after initialized)
* @inherit_rt: inherit RT scheduling policy from caller
* (invariant after initialized)
+ * @txn_security_ctx: require sender's security context
+ * (invariant after initialized)
* @async_todo: list of async work items
* (protected by @proc->inner_lock)
*
@@ -398,6 +400,7 @@ struct binder_node {
u8 sched_policy:2;
u8 inherit_rt:1;
u8 accept_fds:1;
+ u8 txn_security_ctx:1;
u8 min_priority;
};
bool has_async_transaction;
@@ -655,6 +658,7 @@ struct binder_transaction {
struct binder_priority saved_priority;
bool set_priority_called;
kuid_t sender_euid;
+ binder_uintptr_t security_ctx;
/**
* @lock: protects @from, @to_proc, and @to_thread
*
@@ -1364,6 +1368,7 @@ static struct binder_node *binder_init_node_ilocked(
node->min_priority = to_kernel_prio(node->sched_policy, priority);
node->accept_fds = !!(flags & FLAT_BINDER_FLAG_ACCEPTS_FDS);
node->inherit_rt = !!(flags & FLAT_BINDER_FLAG_INHERIT_RT);
+ node->txn_security_ctx = !!(flags & FLAT_BINDER_FLAG_TXN_SECURITY_CTX);
spin_lock_init(&node->lock);
INIT_LIST_HEAD(&node->work.entry);
INIT_LIST_HEAD(&node->async_todo);
@@ -1636,6 +1641,52 @@ static struct binder_ref *binder_get_ref_olocked(struct binder_proc *proc,
return NULL;
}
+
+static void dump_ref_desc_tree(struct binder_ref *new_ref, struct rb_node *n_in)
+{
+ struct binder_proc *proc = new_ref->proc;
+ uint32_t desc = new_ref->data.desc;
+ struct rb_node *n, *p;
+ struct binder_ref *ref;
+ int i = 0;
+
+ pr_info("BUG.%d:%d: dump of refs_by_desc rb tree, new desc=%u, n%sNULL\n",
+ proc->pid, current->pid, desc, n_in ? "!=" : "==");
+ if (n_in) {
+ ref = rb_entry(n_in, struct binder_ref, rb_node_desc);
+ pr_info("ref containing n: id %d desc %u n %d s %d w %d\n",
+ ref->data.debug_id, ref->data.desc,
+ ref->node->debug_id,
+ ref->data.strong, ref->data.weak);
+ }
+
+ for (n = rb_first(&proc->refs_by_desc); n != NULL; n = rb_next(n)) {
+ struct binder_ref *left=NULL, *right=NULL, *parent=NULL;
+
+ ref = rb_entry(n, struct binder_ref, rb_node_desc);
+ if (n->rb_left)
+ left = rb_entry(n->rb_left, struct binder_ref,
+ rb_node_desc);
+ if (n->rb_right)
+ right = rb_entry(n->rb_right, struct binder_ref,
+ rb_node_desc);
+ p = rb_parent(n);
+ if (p)
+ parent = rb_entry(p, struct binder_ref, rb_node_desc);
+
+ pr_info("%d: id %d desc %u%s%s%s n %d s %d w %d l %d r %d p %d\n",
+ i++, ref->data.debug_id, ref->data.desc,
+ (n == proc->refs_by_desc.rb_node) ? " (ROOT)" : "",
+ (ref->data.desc == desc) ? " (MATCH)" : "",
+ (n_in == n) ? " (BREAK)" : "",
+ ref->node->debug_id,
+ ref->data.strong, ref->data.weak,
+ left ? left->data.debug_id : -1,
+ right ? right->data.debug_id : -1,
+ parent ? parent->data.debug_id : -1);
+ }
+}
+
/**
* binder_get_ref_for_node_olocked() - get the ref associated with given node
* @proc: binder_proc that owns the ref
@@ -1703,8 +1754,10 @@ static struct binder_ref *binder_get_ref_for_node_olocked(
p = &(*p)->rb_left;
else if (new_ref->data.desc > ref->data.desc)
p = &(*p)->rb_right;
- else
+ else {
+ dump_ref_desc_tree(new_ref, n);
BUG();
+ }
}
rb_link_node(&new_ref->rb_node_desc, parent, p);
rb_insert_color(&new_ref->rb_node_desc, &proc->refs_by_desc);
@@ -2902,6 +2955,8 @@ static void binder_transaction(struct binder_proc *proc,
binder_size_t last_fixup_min_off = 0;
struct binder_context *context = proc->context;
int t_debug_id = atomic_inc_return(&binder_last_id);
+ char *secctx = NULL;
+ u32 secctx_sz = 0;
e = binder_transaction_log_add(&binder_transaction_log);
e->debug_id = t_debug_id;
@@ -3117,6 +3172,20 @@ static void binder_transaction(struct binder_proc *proc,
t->priority = target_proc->default_priority;
}
+ if (target_node && target_node->txn_security_ctx) {
+ u32 secid;
+
+ security_task_getsecid(proc->tsk, &secid);
+ ret = security_secid_to_secctx(secid, &secctx, &secctx_sz);
+ if (ret) {
+ return_error = BR_FAILED_REPLY;
+ return_error_param = ret;
+ return_error_line = __LINE__;
+ goto err_get_secctx_failed;
+ }
+ extra_buffers_size += ALIGN(secctx_sz, sizeof(u64));
+ }
+
trace_binder_transaction(reply, t, target_node);
t->buffer = binder_alloc_new_buf(&target_proc->alloc, tr->data_size,
@@ -3133,7 +3202,19 @@ static void binder_transaction(struct binder_proc *proc,
t->buffer = NULL;
goto err_binder_alloc_buf_failed;
}
- t->buffer->allow_user_free = 0;
+ if (secctx) {
+ size_t buf_offset = ALIGN(tr->data_size, sizeof(void *)) +
+ ALIGN(tr->offsets_size, sizeof(void *)) +
+ ALIGN(extra_buffers_size, sizeof(void *)) -
+ ALIGN(secctx_sz, sizeof(u64));
+ char *kptr = t->buffer->data + buf_offset;
+
+ t->security_ctx = (binder_uintptr_t)kptr +
+ binder_alloc_get_user_buffer_offset(&target_proc->alloc);
+ memcpy(kptr, secctx, secctx_sz);
+ security_release_secctx(secctx, secctx_sz);
+ secctx = NULL;
+ }
t->buffer->debug_id = t->debug_id;
t->buffer->transaction = t;
t->buffer->target_node = target_node;
@@ -3404,6 +3485,9 @@ static void binder_transaction(struct binder_proc *proc,
t->buffer->transaction = NULL;
binder_alloc_free_buf(&target_proc->alloc, t->buffer);
err_binder_alloc_buf_failed:
+ if (secctx)
+ security_release_secctx(secctx, secctx_sz);
+err_get_secctx_failed:
kfree(tcomplete);
binder_stats_deleted(BINDER_STAT_TRANSACTION_COMPLETE);
err_alloc_tcomplete_failed:
@@ -3629,14 +3713,18 @@ static int binder_thread_write(struct binder_proc *proc,
buffer = binder_alloc_prepare_to_free(&proc->alloc,
data_ptr);
- if (buffer == NULL) {
- binder_user_error("%d:%d BC_FREE_BUFFER u%016llx no match\n",
- proc->pid, thread->pid, (u64)data_ptr);
- break;
- }
- if (!buffer->allow_user_free) {
- binder_user_error("%d:%d BC_FREE_BUFFER u%016llx matched unreturned buffer\n",
- proc->pid, thread->pid, (u64)data_ptr);
+ if (IS_ERR_OR_NULL(buffer)) {
+ if (PTR_ERR(buffer) == -EPERM) {
+ binder_user_error(
+ "%d:%d BC_FREE_BUFFER u%016llx matched unreturned or currently freeing buffer\n",
+ proc->pid, thread->pid,
+ (u64)data_ptr);
+ } else {
+ binder_user_error(
+ "%d:%d BC_FREE_BUFFER u%016llx no match\n",
+ proc->pid, thread->pid,
+ (u64)data_ptr);
+ }
break;
}
binder_debug(BINDER_DEBUG_FREE_BUFFER,
@@ -4046,11 +4134,13 @@ static int binder_thread_read(struct binder_proc *proc,
while (1) {
uint32_t cmd;
- struct binder_transaction_data tr;
+ struct binder_transaction_data_secctx tr;
+ struct binder_transaction_data *trd = &tr.transaction_data;
struct binder_work *w = NULL;
struct list_head *list = NULL;
struct binder_transaction *t = NULL;
struct binder_thread *t_from;
+ size_t trsize = sizeof(*trd);
binder_inner_proc_lock(proc);
if (!binder_worklist_empty_ilocked(&thread->todo))
@@ -4245,41 +4335,47 @@ static int binder_thread_read(struct binder_proc *proc,
struct binder_node *target_node = t->buffer->target_node;
struct binder_priority node_prio;
- tr.target.ptr = target_node->ptr;
- tr.cookie = target_node->cookie;
+ trd->target.ptr = target_node->ptr;
+ trd->cookie = target_node->cookie;
node_prio.sched_policy = target_node->sched_policy;
node_prio.prio = target_node->min_priority;
binder_transaction_priority(current, t, node_prio,
target_node->inherit_rt);
cmd = BR_TRANSACTION;
} else {
- tr.target.ptr = 0;
- tr.cookie = 0;
+ trd->target.ptr = 0;
+ trd->cookie = 0;
cmd = BR_REPLY;
}
- tr.code = t->code;
- tr.flags = t->flags;
- tr.sender_euid = from_kuid(current_user_ns(), t->sender_euid);
+ trd->code = t->code;
+ trd->flags = t->flags;
+ trd->sender_euid = from_kuid(current_user_ns(), t->sender_euid);
t_from = binder_get_txn_from(t);
if (t_from) {
struct task_struct *sender = t_from->proc->tsk;
- tr.sender_pid = task_tgid_nr_ns(sender,
- task_active_pid_ns(current));
+ trd->sender_pid =
+ task_tgid_nr_ns(sender,
+ task_active_pid_ns(current));
} else {
- tr.sender_pid = 0;
+ trd->sender_pid = 0;
}
- tr.data_size = t->buffer->data_size;
- tr.offsets_size = t->buffer->offsets_size;
- tr.data.ptr.buffer = (binder_uintptr_t)
+ trd->data_size = t->buffer->data_size;
+ trd->offsets_size = t->buffer->offsets_size;
+ trd->data.ptr.buffer = (binder_uintptr_t)
((uintptr_t)t->buffer->data +
binder_alloc_get_user_buffer_offset(&proc->alloc));
- tr.data.ptr.offsets = tr.data.ptr.buffer +
+ trd->data.ptr.offsets = trd->data.ptr.buffer +
ALIGN(t->buffer->data_size,
sizeof(void *));
+ if (t->security_ctx) {
+ cmd = BR_TRANSACTION_SEC_CTX;
+ tr.secctx = t->security_ctx;
+ trsize = sizeof(tr);
+ }
if (put_user(cmd, (uint32_t __user *)ptr)) {
if (t_from)
binder_thread_dec_tmpref(t_from);
@@ -4290,7 +4386,7 @@ static int binder_thread_read(struct binder_proc *proc,
return -EFAULT;
}
ptr += sizeof(uint32_t);
- if (copy_to_user(ptr, &tr, sizeof(tr))) {
+ if (copy_to_user(ptr, &tr, trsize)) {
if (t_from)
binder_thread_dec_tmpref(t_from);
@@ -4299,7 +4395,7 @@ static int binder_thread_read(struct binder_proc *proc,
return -EFAULT;
}
- ptr += sizeof(tr);
+ ptr += trsize;
trace_binder_transaction_received(t);
binder_stat_br(proc, thread, cmd);
@@ -4307,16 +4403,18 @@ static int binder_thread_read(struct binder_proc *proc,
"%d:%d %s %d %d:%d, cmd %d size %zd-%zd ptr %016llx-%016llx\n",
proc->pid, thread->pid,
(cmd == BR_TRANSACTION) ? "BR_TRANSACTION" :
- "BR_REPLY",
+ (cmd == BR_TRANSACTION_SEC_CTX) ?
+ "BR_TRANSACTION_SEC_CTX" : "BR_REPLY",
t->debug_id, t_from ? t_from->proc->pid : 0,
t_from ? t_from->pid : 0, cmd,
t->buffer->data_size, t->buffer->offsets_size,
- (u64)tr.data.ptr.buffer, (u64)tr.data.ptr.offsets);
+ (u64)trd->data.ptr.buffer,
+ (u64)trd->data.ptr.offsets);
if (t_from)
binder_thread_dec_tmpref(t_from);
t->buffer->allow_user_free = 1;
- if (cmd == BR_TRANSACTION && !(t->flags & TF_ONE_WAY)) {
+ if (cmd != BR_REPLY && !(t->flags & TF_ONE_WAY)) {
binder_inner_proc_lock(thread->proc);
t->to_parent = thread->transaction_stack;
t->to_thread = thread;
@@ -4661,7 +4759,8 @@ static int binder_ioctl_write_read(struct file *filp,
return ret;
}
-static int binder_ioctl_set_ctx_mgr(struct file *filp)
+static int binder_ioctl_set_ctx_mgr(struct file *filp,
+ struct flat_binder_object *fbo)
{
int ret = 0;
struct binder_proc *proc = filp->private_data;
@@ -4690,7 +4789,7 @@ static int binder_ioctl_set_ctx_mgr(struct file *filp)
} else {
context->binder_context_mgr_uid = curr_euid;
}
- new_node = binder_new_node(proc, NULL);
+ new_node = binder_new_node(proc, fbo);
if (!new_node) {
ret = -ENOMEM;
goto out;
@@ -4708,6 +4807,42 @@ static int binder_ioctl_set_ctx_mgr(struct file *filp)
return ret;
}
+static int binder_ioctl_get_node_info_for_ref(struct binder_proc *proc,
+ struct binder_node_info_for_ref *info)
+{
+ struct binder_node *node;
+ struct binder_context *context = proc->context;
+ __u32 handle = info->handle;
+
+ if (info->strong_count || info->weak_count || info->reserved1 ||
+ info->reserved2 || info->reserved3) {
+ binder_user_error("%d BINDER_GET_NODE_INFO_FOR_REF: only handle may be non-zero.",
+ proc->pid);
+ return -EINVAL;
+ }
+
+ /* This ioctl may only be used by the context manager */
+ mutex_lock(&context->context_mgr_node_lock);
+ if (!context->binder_context_mgr_node ||
+ context->binder_context_mgr_node->proc != proc) {
+ mutex_unlock(&context->context_mgr_node_lock);
+ return -EPERM;
+ }
+ mutex_unlock(&context->context_mgr_node_lock);
+
+ node = binder_get_node_from_ref(proc, handle, true, NULL);
+ if (!node)
+ return -EINVAL;
+
+ info->strong_count = node->local_strong_refs +
+ node->internal_strong_refs;
+ info->weak_count = node->local_weak_refs;
+
+ binder_put_node(node);
+
+ return 0;
+}
+
static int binder_ioctl_get_node_debug_info(struct binder_proc *proc,
struct binder_node_debug_info *info) {
struct rb_node *n;
@@ -4776,8 +4911,20 @@ static long binder_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
binder_inner_proc_unlock(proc);
break;
}
+ case BINDER_SET_CONTEXT_MGR_EXT: {
+ struct flat_binder_object fbo;
+
+ if (copy_from_user(&fbo, ubuf, sizeof(fbo))) {
+ ret = -EINVAL;
+ goto err;
+ }
+ ret = binder_ioctl_set_ctx_mgr(filp, &fbo);
+ if (ret)
+ goto err;
+ break;
+ }
case BINDER_SET_CONTEXT_MGR:
- ret = binder_ioctl_set_ctx_mgr(filp);
+ ret = binder_ioctl_set_ctx_mgr(filp, NULL);
if (ret)
goto err;
break;
@@ -4801,6 +4948,25 @@ static long binder_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
}
break;
}
+ case BINDER_GET_NODE_INFO_FOR_REF: {
+ struct binder_node_info_for_ref info;
+
+ if (copy_from_user(&info, ubuf, sizeof(info))) {
+ ret = -EFAULT;
+ goto err;
+ }
+
+ ret = binder_ioctl_get_node_info_for_ref(proc, &info);
+ if (ret < 0)
+ goto err;
+
+ if (copy_to_user(ubuf, &info, sizeof(info))) {
+ ret = -EFAULT;
+ goto err;
+ }
+
+ break;
+ }
case BINDER_GET_NODE_DEBUG_INFO: {
struct binder_node_debug_info info;
diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c
index 1c76daa..d34d1eb 100644
--- a/drivers/android/binder_alloc.c
+++ b/drivers/android/binder_alloc.c
@@ -149,14 +149,12 @@ static struct binder_buffer *binder_alloc_prepare_to_free_locked(
else {
/*
* Guard against user threads attempting to
- * free the buffer twice
+ * free the buffer when in use by kernel or
+ * after it's already been freed.
*/
- if (buffer->free_in_progress) {
- pr_err("%d:%d FREE_BUFFER u%016llx user freed buffer twice\n",
- alloc->pid, current->pid, (u64)user_ptr);
- return NULL;
- }
- buffer->free_in_progress = 1;
+ if (!buffer->allow_user_free)
+ return ERR_PTR(-EPERM);
+ buffer->allow_user_free = 0;
return buffer;
}
}
@@ -461,7 +459,7 @@ struct binder_buffer *binder_alloc_new_buf_locked(struct binder_alloc *alloc,
rb_erase(best_fit, &alloc->free_buffers);
buffer->free = 0;
- buffer->free_in_progress = 0;
+ buffer->allow_user_free = 0;
binder_insert_allocated_buffer_locked(alloc, buffer);
binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC,
"%d: binder_alloc_buf size %zd got %pK\n",
@@ -924,14 +922,13 @@ enum lru_status binder_alloc_free_page(struct list_head *item,
index = page - alloc->pages;
page_addr = (uintptr_t)alloc->buffer + index * PAGE_SIZE;
+
+ mm = alloc->vma_vm_mm;
+ if (!mmget_not_zero(mm))
+ goto err_mmget;
+ if (!down_write_trylock(&mm->mmap_sem))
+ goto err_down_write_mmap_sem_failed;
vma = alloc->vma;
- if (vma) {
- if (!mmget_not_zero(alloc->vma_vm_mm))
- goto err_mmget;
- mm = alloc->vma_vm_mm;
- if (!down_write_trylock(&mm->mmap_sem))
- goto err_down_write_mmap_sem_failed;
- }
list_lru_isolate(lru, item);
spin_unlock(lock);
@@ -945,10 +942,9 @@ enum lru_status binder_alloc_free_page(struct list_head *item,
PAGE_SIZE, NULL);
trace_binder_unmap_user_end(alloc, index);
-
- up_write(&mm->mmap_sem);
- mmput(mm);
}
+ up_write(&mm->mmap_sem);
+ mmput(mm);
trace_binder_unmap_kernel_start(alloc, index);
diff --git a/drivers/android/binder_alloc.h b/drivers/android/binder_alloc.h
index 0b14530..c4b49e2 100644
--- a/drivers/android/binder_alloc.h
+++ b/drivers/android/binder_alloc.h
@@ -50,8 +50,7 @@ struct binder_buffer {
unsigned free:1;
unsigned allow_user_free:1;
unsigned async_transaction:1;
- unsigned free_in_progress:1;
- unsigned debug_id:28;
+ unsigned debug_id:29;
struct binder_transaction *transaction;
diff --git a/drivers/ata/ahci.c b/drivers/ata/ahci.c
index 4d4b5f6..5408a29 100644
--- a/drivers/ata/ahci.c
+++ b/drivers/ata/ahci.c
@@ -624,8 +624,11 @@ static void ahci_pci_save_initial_config(struct pci_dev *pdev,
static int ahci_pci_reset_controller(struct ata_host *host)
{
struct pci_dev *pdev = to_pci_dev(host->dev);
+ int rc;
- ahci_reset_controller(host);
+ rc = ahci_reset_controller(host);
+ if (rc)
+ return rc;
if (pdev->vendor == PCI_VENDOR_ID_INTEL) {
struct ahci_host_priv *hpriv = host->private_data;
@@ -1260,6 +1263,59 @@ static bool ahci_broken_suspend(struct pci_dev *pdev)
return strcmp(buf, dmi->driver_data) < 0;
}
+static bool ahci_broken_lpm(struct pci_dev *pdev)
+{
+ static const struct dmi_system_id sysids[] = {
+ /* Various Lenovo 50 series have LPM issues with older BIOSen */
+ {
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+ DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad X250"),
+ },
+ .driver_data = "20180406", /* 1.31 */
+ },
+ {
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+ DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad L450"),
+ },
+ .driver_data = "20180420", /* 1.28 */
+ },
+ {
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+ DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad T450s"),
+ },
+ .driver_data = "20180315", /* 1.33 */
+ },
+ {
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+ DMI_MATCH(DMI_PRODUCT_VERSION, "ThinkPad W541"),
+ },
+ /*
+ * Note date based on release notes, 2.35 has been
+ * reported to be good, but I've been unable to get
+ * a hold of the reporter to get the DMI BIOS date.
+ * TODO: fix this.
+ */
+ .driver_data = "20180310", /* 2.35 */
+ },
+ { } /* terminate list */
+ };
+ const struct dmi_system_id *dmi = dmi_first_match(sysids);
+ int year, month, date;
+ char buf[9];
+
+ if (!dmi)
+ return false;
+
+ dmi_get_date(DMI_BIOS_DATE, &year, &month, &date);
+ snprintf(buf, sizeof(buf), "%04d%02d%02d", year, month, date);
+
+ return strcmp(buf, dmi->driver_data) < 0;
+}
+
static bool ahci_broken_online(struct pci_dev *pdev)
{
#define ENCODE_BUSDEVFN(bus, slot, func) \
@@ -1626,6 +1682,12 @@ static int ahci_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
"quirky BIOS, skipping spindown on poweroff\n");
}
+ if (ahci_broken_lpm(pdev)) {
+ pi.flags |= ATA_FLAG_NO_LPM;
+ dev_warn(&pdev->dev,
+ "BIOS update required for Link Power Management support\n");
+ }
+
if (ahci_broken_suspend(pdev)) {
hpriv->flags |= AHCI_HFLAG_NO_SUSPEND;
dev_warn(&pdev->dev,
diff --git a/drivers/ata/libahci.c b/drivers/ata/libahci.c
index 0d028ea..f233ce6 100644
--- a/drivers/ata/libahci.c
+++ b/drivers/ata/libahci.c
@@ -35,6 +35,7 @@
#include <linux/kernel.h>
#include <linux/gfp.h>
#include <linux/module.h>
+#include <linux/nospec.h>
#include <linux/blkdev.h>
#include <linux/delay.h>
#include <linux/interrupt.h>
@@ -1124,10 +1125,12 @@ static ssize_t ahci_led_store(struct ata_port *ap, const char *buf,
/* get the slot number from the message */
pmp = (state & EM_MSG_LED_PMP_SLOT) >> 8;
- if (pmp < EM_MAX_SLOTS)
+ if (pmp < EM_MAX_SLOTS) {
+ pmp = array_index_nospec(pmp, EM_MAX_SLOTS);
emp = &pp->em_priv[pmp];
- else
+ } else {
return -EINVAL;
+ }
/* mask off the activity bits if we are in sw_activity
* mode, user should turn off sw_activity before setting
@@ -2129,6 +2132,8 @@ static void ahci_set_aggressive_devslp(struct ata_port *ap, bool sleep)
deto = 20;
}
+ /* Make dito, mdat, deto bits to 0s */
+ devslp &= ~GENMASK_ULL(24, 2);
devslp |= ((dito << PORT_DEVSLP_DITO_OFFSET) |
(mdat << PORT_DEVSLP_MDAT_OFFSET) |
(deto << PORT_DEVSLP_DETO_OFFSET) |
diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c
index e08c09f..35be49f 100644
--- a/drivers/ata/libata-core.c
+++ b/drivers/ata/libata-core.c
@@ -2385,6 +2385,9 @@ int ata_dev_configure(struct ata_device *dev)
(id[ATA_ID_SATA_CAPABILITY] & 0xe) == 0x2)
dev->horkage |= ATA_HORKAGE_NOLPM;
+ if (ap->flags & ATA_FLAG_NO_LPM)
+ dev->horkage |= ATA_HORKAGE_NOLPM;
+
if (dev->horkage & ATA_HORKAGE_NOLPM) {
ata_dev_warn(dev, "LPM support broken, forcing max_power\n");
dev->link->ap->target_lpm_policy = ATA_LPM_MAX_POWER;
@@ -4366,6 +4369,10 @@ static const struct ata_blacklist_entry ata_device_blacklist [] = {
/* https://bugzilla.kernel.org/show_bug.cgi?id=15573 */
{ "C300-CTFDDAC128MAG", "0001", ATA_HORKAGE_NONCQ, },
+ /* Some Sandisk SSDs lock up hard with NCQ enabled. Reported on
+ SD7SN6S256G and SD8SN8U256G */
+ { "SanDisk SD[78]SN*G", NULL, ATA_HORKAGE_NONCQ, },
+
/* devices which puke on READ_NATIVE_MAX */
{ "HDS724040KLSA80", "KFAOA20N", ATA_HORKAGE_BROKEN_HPA, },
{ "WDC WD3200JD-00KLB0", "WD-WCAMR1130137", ATA_HORKAGE_BROKEN_HPA },
@@ -4423,6 +4430,8 @@ static const struct ata_blacklist_entry ata_device_blacklist [] = {
ATA_HORKAGE_NOLPM, },
/* devices that don't properly handle queued TRIM commands */
+ { "Micron_M500IT_*", "MU01", ATA_HORKAGE_NO_NCQ_TRIM |
+ ATA_HORKAGE_ZERO_AFTER_TRIM, },
{ "Micron_M500_*", NULL, ATA_HORKAGE_NO_NCQ_TRIM |
ATA_HORKAGE_ZERO_AFTER_TRIM, },
{ "Crucial_CT*M500*", NULL, ATA_HORKAGE_NO_NCQ_TRIM |
@@ -4467,6 +4476,7 @@ static const struct ata_blacklist_entry ata_device_blacklist [] = {
{ "SSD*INTEL*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, },
{ "Samsung*SSD*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, },
{ "SAMSUNG*SSD*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, },
+ { "SAMSUNG*MZ7KM*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, },
{ "ST[1248][0248]0[FH]*", NULL, ATA_HORKAGE_ZERO_AFTER_TRIM, },
/*
@@ -6772,7 +6782,7 @@ static int __init ata_parse_force_one(char **cur,
}
force_ent->port = simple_strtoul(id, &endp, 10);
- if (p == endp || *endp != '\0') {
+ if (id == endp || *endp != '\0') {
*reason = "invalid port/link";
return -EINVAL;
}
diff --git a/drivers/ata/libata-eh.c b/drivers/ata/libata-eh.c
index 6475a13..90c3877 100644
--- a/drivers/ata/libata-eh.c
+++ b/drivers/ata/libata-eh.c
@@ -2282,12 +2282,16 @@ static void ata_eh_link_autopsy(struct ata_link *link)
if (qc->err_mask & ~AC_ERR_OTHER)
qc->err_mask &= ~AC_ERR_OTHER;
- /* SENSE_VALID trumps dev/unknown error and revalidation */
+ /*
+ * SENSE_VALID trumps dev/unknown error and revalidation. Upper
+ * layers will determine whether the command is worth retrying
+ * based on the sense data and device class/type. Otherwise,
+ * determine directly if the command is worth retrying using its
+ * error mask and flags.
+ */
if (qc->flags & ATA_QCFLAG_SENSE_VALID)
qc->err_mask &= ~(AC_ERR_DEV | AC_ERR_OTHER);
-
- /* determine whether the command is worth retrying */
- if (ata_eh_worth_retry(qc))
+ else if (ata_eh_worth_retry(qc))
qc->flags |= ATA_QCFLAG_RETRY;
/* accumulate error info */
diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
index 9babbc8..a3d60cc 100644
--- a/drivers/ata/libata-scsi.c
+++ b/drivers/ata/libata-scsi.c
@@ -3772,10 +3772,20 @@ static unsigned int ata_scsi_zbc_out_xlat(struct ata_queued_cmd *qc)
*/
goto invalid_param_len;
}
- if (block > dev->n_sectors)
- goto out_of_range;
all = cdb[14] & 0x1;
+ if (all) {
+ /*
+ * Ignore the block address (zone ID) as defined by ZBC.
+ */
+ block = 0;
+ } else if (block >= dev->n_sectors) {
+ /*
+ * Block must be a valid zone ID (a zone start LBA).
+ */
+ fp = 2;
+ goto invalid_fld;
+ }
if (ata_ncq_enabled(qc->dev) &&
ata_fpdma_zac_mgmt_out_supported(qc->dev)) {
@@ -3804,10 +3814,6 @@ static unsigned int ata_scsi_zbc_out_xlat(struct ata_queued_cmd *qc)
invalid_fld:
ata_scsi_set_invalid_field(qc->dev, scmd, fp, 0xff);
return 1;
- out_of_range:
- /* "Logical Block Address out of range" */
- ata_scsi_set_sense(qc->dev, scmd, ILLEGAL_REQUEST, 0x21, 0x00);
- return 1;
invalid_param_len:
/* "Parameter list length error" */
ata_scsi_set_sense(qc->dev, scmd, ILLEGAL_REQUEST, 0x1a, 0x0);
@@ -4156,7 +4162,7 @@ static inline void ata_scsi_dump_cdb(struct ata_port *ap,
#ifdef ATA_DEBUG
struct scsi_device *scsidev = cmd->device;
- DPRINTK("CDB (%u:%d,%d,%d) %9ph\n",
+ DPRINTK("CDB (%u:%d,%d,%lld) %9ph\n",
ap->print_id,
scsidev->channel, scsidev->id, scsidev->lun,
cmd->cmnd);
diff --git a/drivers/ata/libata-zpodd.c b/drivers/ata/libata-zpodd.c
index f3a65a3..0ad96c6 100644
--- a/drivers/ata/libata-zpodd.c
+++ b/drivers/ata/libata-zpodd.c
@@ -34,7 +34,7 @@ struct zpodd {
static int eject_tray(struct ata_device *dev)
{
struct ata_taskfile tf;
- const char cdb[] = { GPCMD_START_STOP_UNIT,
+ static const char cdb[ATAPI_CDB_LEN] = { GPCMD_START_STOP_UNIT,
0, 0, 0,
0x02, /* LoEj */
0, 0, 0, 0, 0, 0, 0,
@@ -55,7 +55,7 @@ static enum odd_mech_type zpodd_get_mech_type(struct ata_device *dev)
unsigned int ret;
struct rm_feature_desc *desc = (void *)(buf + 8);
struct ata_taskfile tf;
- char cdb[] = { GPCMD_GET_CONFIGURATION,
+ static const char cdb[] = { GPCMD_GET_CONFIGURATION,
2, /* only 1 feature descriptor requested */
0, 3, /* 3, removable medium feature */
0, 0, 0,/* reserved */
diff --git a/drivers/ata/sata_rcar.c b/drivers/ata/sata_rcar.c
index f72d601..e83a3d3 100644
--- a/drivers/ata/sata_rcar.c
+++ b/drivers/ata/sata_rcar.c
@@ -890,7 +890,10 @@ static int sata_rcar_probe(struct platform_device *pdev)
dev_err(&pdev->dev, "failed to get access to sata clock\n");
return PTR_ERR(priv->clk);
}
- clk_prepare_enable(priv->clk);
+
+ ret = clk_prepare_enable(priv->clk);
+ if (ret)
+ return ret;
host = ata_host_alloc(&pdev->dev, 1);
if (!host) {
@@ -970,8 +973,11 @@ static int sata_rcar_resume(struct device *dev)
struct ata_host *host = dev_get_drvdata(dev);
struct sata_rcar_priv *priv = host->private_data;
void __iomem *base = priv->base;
+ int ret;
- clk_prepare_enable(priv->clk);
+ ret = clk_prepare_enable(priv->clk);
+ if (ret)
+ return ret;
/* ack and mask */
iowrite32(0, base + SATAINTSTAT_REG);
@@ -988,8 +994,11 @@ static int sata_rcar_restore(struct device *dev)
{
struct ata_host *host = dev_get_drvdata(dev);
struct sata_rcar_priv *priv = host->private_data;
+ int ret;
- clk_prepare_enable(priv->clk);
+ ret = clk_prepare_enable(priv->clk);
+ if (ret)
+ return ret;
sata_rcar_setup_port(host);
diff --git a/drivers/atm/zatm.c b/drivers/atm/zatm.c
index d3dc954..a0b88f1 100644
--- a/drivers/atm/zatm.c
+++ b/drivers/atm/zatm.c
@@ -23,6 +23,7 @@
#include <linux/bitops.h>
#include <linux/wait.h>
#include <linux/slab.h>
+#include <linux/nospec.h>
#include <asm/byteorder.h>
#include <asm/string.h>
#include <asm/io.h>
@@ -1150,8 +1151,8 @@ static void eprom_get_byte(struct zatm_dev *zatm_dev, unsigned char *byte,
}
-static unsigned char eprom_try_esi(struct atm_dev *dev, unsigned short cmd,
- int offset, int swap)
+static int eprom_try_esi(struct atm_dev *dev, unsigned short cmd, int offset,
+ int swap)
{
unsigned char buf[ZEPROM_SIZE];
struct zatm_dev *zatm_dev;
@@ -1458,6 +1459,8 @@ static int zatm_ioctl(struct atm_dev *dev,unsigned int cmd,void __user *arg)
return -EFAULT;
if (pool < 0 || pool > ZATM_LAST_POOL)
return -EINVAL;
+ pool = array_index_nospec(pool,
+ ZATM_LAST_POOL + 1);
spin_lock_irqsave(&zatm_dev->lock, flags);
info = zatm_dev->pool_info[pool];
if (cmd == ZATM_GETPOOLZ) {
@@ -1480,6 +1483,8 @@ static int zatm_ioctl(struct atm_dev *dev,unsigned int cmd,void __user *arg)
return -EFAULT;
if (pool < 0 || pool > ZATM_LAST_POOL)
return -EINVAL;
+ pool = array_index_nospec(pool,
+ ZATM_LAST_POOL + 1);
if (copy_from_user(&info,
&((struct zatm_pool_req __user *) arg)->info,
sizeof(info))) return -EFAULT;
diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index 0335e23..fe1b856 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -85,6 +85,11 @@
require userspace firmware loading support, but a module built
out-of-tree does.
+config FW_CACHE
+ bool "Enable firmware caching during suspend"
+ depends on PM_SLEEP
+ default n
+
config FIRMWARE_IN_KERNEL
bool "Include in-kernel firmware blobs in kernel binary"
depends on FW_LOADER
diff --git a/drivers/base/bus.c b/drivers/base/bus.c
index e32a74e..09d8fce 100644
--- a/drivers/base/bus.c
+++ b/drivers/base/bus.c
@@ -186,10 +186,10 @@ static ssize_t unbind_store(struct device_driver *drv, const char *buf,
dev = bus_find_device_by_name(bus, NULL, buf);
if (dev && dev->driver == drv) {
- if (dev->parent) /* Needed for USB */
+ if (dev->parent && bus->need_parent_lock)
device_lock(dev->parent);
device_release_driver(dev);
- if (dev->parent)
+ if (dev->parent && bus->need_parent_lock)
device_unlock(dev->parent);
err = count;
}
@@ -213,12 +213,12 @@ static ssize_t bind_store(struct device_driver *drv, const char *buf,
dev = bus_find_device_by_name(bus, NULL, buf);
if (dev && dev->driver == NULL && driver_match_device(drv, dev)) {
- if (dev->parent) /* Needed for USB */
+ if (dev->parent && dev->bus->need_parent_lock)
device_lock(dev->parent);
device_lock(dev);
err = driver_probe_device(drv, dev);
device_unlock(dev);
- if (dev->parent)
+ if (dev->parent && dev->bus->need_parent_lock)
device_unlock(dev->parent);
if (err > 0) {
@@ -775,10 +775,10 @@ static int __must_check bus_rescan_devices_helper(struct device *dev,
int ret = 0;
if (!dev->driver) {
- if (dev->parent) /* Needed for USB */
+ if (dev->parent && dev->bus->need_parent_lock)
device_lock(dev->parent);
ret = device_attach(dev);
- if (dev->parent)
+ if (dev->parent && dev->bus->need_parent_lock)
device_unlock(dev->parent);
}
return ret < 0 ? ret : 0;
@@ -810,10 +810,10 @@ EXPORT_SYMBOL_GPL(bus_rescan_devices);
int device_reprobe(struct device *dev)
{
if (dev->driver) {
- if (dev->parent) /* Needed for USB */
+ if (dev->parent && dev->bus->need_parent_lock)
device_lock(dev->parent);
device_release_driver(dev);
- if (dev->parent)
+ if (dev->parent && dev->bus->need_parent_lock)
device_unlock(dev->parent);
}
return bus_rescan_devices_helper(dev, NULL);
diff --git a/drivers/base/core.c b/drivers/base/core.c
index b20d016..694e61b 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -764,7 +764,7 @@ class_dir_create_and_add(struct class *class, struct kobject *parent_kobj)
dir = kzalloc(sizeof(*dir), GFP_KERNEL);
if (!dir)
- return NULL;
+ return ERR_PTR(-ENOMEM);
dir->class = class;
kobject_init(&dir->kobj, &class_dir_ktype);
@@ -774,7 +774,7 @@ class_dir_create_and_add(struct class *class, struct kobject *parent_kobj)
retval = kobject_add(&dir->kobj, parent_kobj, "%s", class->name);
if (retval < 0) {
kobject_put(&dir->kobj);
- return NULL;
+ return ERR_PTR(retval);
}
return &dir->kobj;
}
@@ -1081,6 +1081,10 @@ int device_add(struct device *dev)
parent = get_device(dev->parent);
kobj = get_device_parent(dev, parent);
+ if (IS_ERR(kobj)) {
+ error = PTR_ERR(kobj);
+ goto parent_error;
+ }
if (kobj)
dev->kobj.parent = kobj;
@@ -1179,6 +1183,7 @@ int device_add(struct device *dev)
kobject_del(&dev->kobj);
Error:
cleanup_glue_dir(dev, glue_dir);
+parent_error:
put_device(parent);
name_error:
kfree(dev->p);
@@ -1996,6 +2001,11 @@ int device_move(struct device *dev, struct device *new_parent,
device_pm_lock();
new_parent = get_device(new_parent);
new_parent_kobj = get_device_parent(dev, new_parent);
+ if (IS_ERR(new_parent_kobj)) {
+ error = PTR_ERR(new_parent_kobj);
+ put_device(new_parent);
+ goto out;
+ }
pr_debug("device: '%s': %s: moving to '%s'\n", dev_name(dev),
__func__, new_parent ? dev_name(new_parent) : "<NULL>");
@@ -2067,6 +2077,9 @@ void device_shutdown(void)
{
struct device *dev, *parent;
+ wait_for_device_probe();
+ device_block_probing();
+
spin_lock(&devices_kset->list_lock);
/*
* Walk the devices list backward, shutting down each in turn.
diff --git a/drivers/base/cpu.c b/drivers/base/cpu.c
index 4fe86f7..7ff7ae0 100644
--- a/drivers/base/cpu.c
+++ b/drivers/base/cpu.c
@@ -610,14 +610,30 @@ ssize_t __weak cpu_show_spectre_v2(struct device *dev,
return sprintf(buf, "Not affected\n");
}
+ssize_t __weak cpu_show_spec_store_bypass(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ return sprintf(buf, "Not affected\n");
+}
+
+ssize_t __weak cpu_show_l1tf(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ return sprintf(buf, "Not affected\n");
+}
+
static DEVICE_ATTR(meltdown, 0444, cpu_show_meltdown, NULL);
static DEVICE_ATTR(spectre_v1, 0444, cpu_show_spectre_v1, NULL);
static DEVICE_ATTR(spectre_v2, 0444, cpu_show_spectre_v2, NULL);
+static DEVICE_ATTR(spec_store_bypass, 0444, cpu_show_spec_store_bypass, NULL);
+static DEVICE_ATTR(l1tf, 0444, cpu_show_l1tf, NULL);
static struct attribute *cpu_root_vulnerabilities_attrs[] = {
&dev_attr_meltdown.attr,
&dev_attr_spectre_v1.attr,
&dev_attr_spectre_v2.attr,
+ &dev_attr_spec_store_bypass.attr,
+ &dev_attr_l1tf.attr,
NULL
};
diff --git a/drivers/base/dd.c b/drivers/base/dd.c
index f95593a..e8ab9c3 100644
--- a/drivers/base/dd.c
+++ b/drivers/base/dd.c
@@ -19,6 +19,7 @@
#include <linux/device.h>
#include <linux/delay.h>
+#include <linux/init.h>
#include <linux/module.h>
#include <linux/kthread.h>
#include <linux/wait.h>
@@ -52,6 +53,7 @@ static DEFINE_MUTEX(deferred_probe_mutex);
static LIST_HEAD(deferred_probe_pending_list);
static LIST_HEAD(deferred_probe_active_list);
static atomic_t deferred_trigger_count = ATOMIC_INIT(0);
+static bool initcalls_done;
/*
* In some cases, like suspend to RAM or hibernation, It might be reasonable
@@ -61,6 +63,26 @@ static atomic_t deferred_trigger_count = ATOMIC_INIT(0);
static bool defer_all_probes;
/*
+ * For initcall_debug, show the deferred probes executed in late_initcall
+ * processing.
+ */
+static void deferred_probe_debug(struct device *dev)
+{
+ ktime_t calltime, delta, rettime;
+ unsigned long long duration;
+
+ printk(KERN_DEBUG "deferred probe %s @ %i\n", dev_name(dev),
+ task_pid_nr(current));
+ calltime = ktime_get();
+ bus_probe_device(dev);
+ rettime = ktime_get();
+ delta = ktime_sub(rettime, calltime);
+ duration = (unsigned long long) ktime_to_ns(delta) >> 10;
+ printk(KERN_DEBUG "deferred probe %s returned after %lld usecs\n",
+ dev_name(dev), duration);
+}
+
+/*
* deferred_probe_work_func() - Retry probing devices in the active list.
*/
static void deferred_probe_work_func(struct work_struct *work)
@@ -105,7 +127,10 @@ static void deferred_probe_work_func(struct work_struct *work)
device_pm_unlock();
dev_dbg(dev, "Retrying from deferred list\n");
- bus_probe_device(dev);
+ if (initcall_debug && !initcalls_done)
+ deferred_probe_debug(dev);
+ else
+ bus_probe_device(dev);
mutex_lock(&deferred_probe_mutex);
@@ -224,6 +249,7 @@ static int deferred_probe_initcall(void)
{
enable_trigger_defer_cycle();
driver_deferred_probe_enable = false;
+ initcalls_done = true;
return 0;
}
arch_initcall_sync(deferred_probe_initcall);
@@ -384,14 +410,6 @@ static int really_probe(struct device *dev, struct device_driver *drv)
goto probe_failed;
}
- /*
- * Ensure devices are listed in devices_kset in correct order
- * It's important to move Dev to the end of devices_kset before
- * calling .probe, because it could be recursive and parent Dev
- * should always go first
- */
- devices_kset_move_last(dev);
-
if (dev->bus->probe) {
ret = dev->bus->probe(dev);
if (ret)
@@ -762,13 +780,13 @@ static int __driver_attach(struct device *dev, void *data)
return ret;
} /* ret > 0 means positive match */
- if (dev->parent) /* Needed for USB */
+ if (dev->parent && dev->bus->need_parent_lock)
device_lock(dev->parent);
device_lock(dev);
if (!dev->driver)
driver_probe_device(drv, dev);
device_unlock(dev);
- if (dev->parent)
+ if (dev->parent && dev->bus->need_parent_lock)
device_unlock(dev->parent);
return 0;
@@ -875,13 +893,13 @@ void driver_detach(struct device_driver *drv)
get_device(dev);
spin_unlock(&drv->p->klist_devices.k_lock);
- if (dev->parent) /* Needed for USB */
+ if (dev->parent && dev->bus->need_parent_lock)
device_lock(dev->parent);
device_lock(dev);
if (dev->driver == drv)
__device_release_driver(dev);
device_unlock(dev);
- if (dev->parent)
+ if (dev->parent && dev->bus->need_parent_lock)
device_unlock(dev->parent);
put_device(dev);
}
diff --git a/drivers/base/firmware_class.c b/drivers/base/firmware_class.c
index 813a191..b59de4f 100644
--- a/drivers/base/firmware_class.c
+++ b/drivers/base/firmware_class.c
@@ -1022,7 +1022,7 @@ fw_load_from_user_helper(struct firmware *firmware, const char *name,
/* No abort during direct loading */
#define is_fw_load_aborted(buf) false
-#ifdef CONFIG_PM_SLEEP
+#ifdef CONFIG_FW_CACHE
static inline void kill_requests_without_uevent(void) { }
#endif
@@ -1428,6 +1428,7 @@ static int cache_firmware(const char *fw_name)
return ret;
}
+#ifdef CONFIG_FW_CACHE
static struct firmware_buf *fw_lookup_buf(const char *fw_name)
{
struct firmware_buf *tmp;
@@ -1439,7 +1440,7 @@ static struct firmware_buf *fw_lookup_buf(const char *fw_name)
return tmp;
}
-
+#endif
/**
* uncache_firmware - remove one cached firmware image
* @fw_name: the firmware image name
diff --git a/drivers/base/node.c b/drivers/base/node.c
index 5548f96..1d803c1 100644
--- a/drivers/base/node.c
+++ b/drivers/base/node.c
@@ -105,6 +105,9 @@ static ssize_t node_read_meminfo(struct device *dev,
"Node %d AnonPages: %8lu kB\n"
"Node %d Shmem: %8lu kB\n"
"Node %d KernelStack: %8lu kB\n"
+#ifdef CONFIG_SHADOW_CALL_STACK
+ "Node %d ShadowCallStack:%8lu kB\n"
+#endif
"Node %d PageTables: %8lu kB\n"
"Node %d NFS_Unstable: %8lu kB\n"
"Node %d Bounce: %8lu kB\n"
@@ -125,6 +128,9 @@ static ssize_t node_read_meminfo(struct device *dev,
nid, K(node_page_state(pgdat, NR_ANON_MAPPED)),
nid, K(i.sharedram),
nid, sum_zone_node_page_state(nid, NR_KERNEL_STACK_KB),
+#ifdef CONFIG_SHADOW_CALL_STACK
+ nid, sum_zone_node_page_state(nid, NR_KERNEL_SCS_BYTES) / 1024,
+#endif
nid, K(sum_zone_node_page_state(nid, NR_PAGETABLE)),
nid, K(node_page_state(pgdat, NR_UNSTABLE_NFS)),
nid, K(sum_zone_node_page_state(nid, NR_BOUNCE)),
diff --git a/drivers/base/platform-msi.c b/drivers/base/platform-msi.c
index be6a599..7ba1d73 100644
--- a/drivers/base/platform-msi.c
+++ b/drivers/base/platform-msi.c
@@ -375,14 +375,16 @@ void platform_msi_domain_free(struct irq_domain *domain, unsigned int virq,
unsigned int nvec)
{
struct platform_msi_priv_data *data = domain->host_data;
- struct msi_desc *desc;
- for_each_msi_entry(desc, data->dev) {
+ struct msi_desc *desc, *tmp;
+ for_each_msi_entry_safe(desc, tmp, data->dev) {
if (WARN_ON(!desc->irq || desc->nvec_used != 1))
return;
if (!(desc->irq >= virq && desc->irq < (virq + nvec)))
continue;
irq_domain_free_irqs_common(domain, desc->irq, 1);
+ list_del(&desc->list);
+ free_msi_entry(desc);
}
}
diff --git a/drivers/base/power/clock_ops.c b/drivers/base/power/clock_ops.c
index 8e2e475..5a42ae4 100644
--- a/drivers/base/power/clock_ops.c
+++ b/drivers/base/power/clock_ops.c
@@ -185,7 +185,7 @@ EXPORT_SYMBOL_GPL(of_pm_clk_add_clk);
int of_pm_clk_add_clks(struct device *dev)
{
struct clk **clks;
- unsigned int i, count;
+ int i, count;
int ret;
if (!dev || !dev->of_node)
diff --git a/drivers/base/power/main.c b/drivers/base/power/main.c
index dc259d2..765f3a4 100644
--- a/drivers/base/power/main.c
+++ b/drivers/base/power/main.c
@@ -1061,11 +1061,13 @@ static int __device_suspend_noirq(struct device *dev, pm_message_t state, bool a
}
error = dpm_run_callback(callback, dev, state, info);
- if (!error)
+ if (!error) {
dev->power.is_noirq_suspended = true;
- else
+ } else {
+ log_suspend_abort_reason("Callback failed on %s in %pF returned %d",
+ dev_name(dev), callback, error);
async_error = error;
-
+ }
Complete:
complete_all(&dev->power.completion);
TRACE_SUSPEND(error);
@@ -1208,10 +1210,13 @@ static int __device_suspend_late(struct device *dev, pm_message_t state, bool as
}
error = dpm_run_callback(callback, dev, state, info);
- if (!error)
+ if (!error) {
dev->power.is_late_suspended = true;
- else
+ } else {
+ log_suspend_abort_reason("Callback failed on %s in %pF returned %d",
+ dev_name(dev), callback, error);
async_error = error;
+ }
Complete:
TRACE_SUSPEND(error);
@@ -1354,7 +1359,6 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
pm_callback_t callback = NULL;
char *info = NULL;
int error = 0;
- char suspend_abort[MAX_SUSPEND_ABORT_LEN];
DECLARE_DPM_WATCHDOG_ON_STACK(wd);
TRACE_DEVICE(dev);
@@ -1362,8 +1366,10 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
dpm_wait_for_children(dev, async);
- if (async_error)
+ if (async_error) {
+ dev->power.direct_complete = false;
goto Complete;
+ }
/*
* If a device configured to wake up the system from sleep states
@@ -1375,9 +1381,7 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
pm_wakeup_event(dev, 0);
if (pm_wakeup_pending()) {
- pm_get_active_wakeup_sources(suspend_abort,
- MAX_SUSPEND_ABORT_LEN);
- log_suspend_abort_reason(suspend_abort);
+ dev->power.direct_complete = false;
async_error = -EBUSY;
goto Complete;
}
@@ -1459,6 +1463,9 @@ static int __device_suspend(struct device *dev, pm_message_t state, bool async)
spin_unlock_irq(&parent->power.lock);
}
+ } else {
+ log_suspend_abort_reason("Callback failed on %s in %pF returned %d",
+ dev_name(dev), callback, error);
}
device_unlock(dev);
@@ -1668,6 +1675,9 @@ int dpm_prepare(pm_message_t state)
printk(KERN_INFO "PM: Device %s not prepared "
"for power transition: code %d\n",
dev_name(dev), error);
+ log_suspend_abort_reason("Device %s not prepared "
+ "for power transition: code %d",
+ dev_name(dev), error);
put_device(dev);
break;
}
diff --git a/drivers/base/power/opp/core.c b/drivers/base/power/opp/core.c
index 94001aa..e494a93 100644
--- a/drivers/base/power/opp/core.c
+++ b/drivers/base/power/opp/core.c
@@ -651,7 +651,7 @@ int dev_pm_opp_set_rate(struct device *dev, unsigned long target_freq)
rcu_read_unlock();
/* Scaling up? Scale voltage before frequency */
- if (freq > old_freq) {
+ if (freq >= old_freq) {
ret = _set_opp_voltage(dev, reg, u_volt, u_volt_min,
u_volt_max);
if (ret)
diff --git a/drivers/base/power/wakeup.c b/drivers/base/power/wakeup.c
index b758633..4adbad6 100644
--- a/drivers/base/power/wakeup.c
+++ b/drivers/base/power/wakeup.c
@@ -16,6 +16,7 @@
#include <linux/debugfs.h>
#include <linux/pm_wakeirq.h>
#include <linux/types.h>
+#include <linux/wakeup_reason.h>
#include <trace/events/power.h>
#include <linux/irq.h>
#include <linux/irqdesc.h>
@@ -877,6 +878,7 @@ bool pm_wakeup_pending(void)
{
unsigned long flags;
bool ret = false;
+ char suspend_abort[MAX_SUSPEND_ABORT_LEN];
spin_lock_irqsave(&events_lock, flags);
if (events_check_enabled) {
@@ -890,7 +892,10 @@ bool pm_wakeup_pending(void)
if (ret) {
pr_info("PM: Wakeup pending, aborting suspend\n");
- pm_print_active_wakeup_sources();
+ pm_get_active_wakeup_sources(suspend_abort,
+ MAX_SUSPEND_ABORT_LEN);
+ log_suspend_abort_reason(suspend_abort);
+ pr_info("PM: %s\n", suspend_abort);
}
return ret || pm_abort_suspend;
@@ -922,9 +927,9 @@ void pm_system_irq_wakeup(unsigned int irq_number)
else if (desc->action && desc->action->name)
name = desc->action->name;
+ log_base_wakeup_reason(irq_number);
pr_warn("%s: %d triggered %s\n", __func__,
irq_number, name);
-
}
pm_wakeup_irq = irq_number;
pm_system_wakeup();
diff --git a/drivers/base/regmap/regmap.c b/drivers/base/regmap/regmap.c
index f74f3ca..18228ba 100644
--- a/drivers/base/regmap/regmap.c
+++ b/drivers/base/regmap/regmap.c
@@ -98,7 +98,7 @@ bool regmap_cached(struct regmap *map, unsigned int reg)
int ret;
unsigned int val;
- if (map->cache == REGCACHE_NONE)
+ if (map->cache_type == REGCACHE_NONE)
return false;
if (!map->cache_ops)
diff --git a/drivers/base/syscore.c b/drivers/base/syscore.c
index 96c34a9..eef576b 100644
--- a/drivers/base/syscore.c
+++ b/drivers/base/syscore.c
@@ -12,6 +12,7 @@
#include <linux/suspend.h>
#include <trace/events/power.h>
#include <linux/wakeup_reason.h>
+#include <linux/irq.h>
static LIST_HEAD(syscore_ops_list);
static DEFINE_MUTEX(syscore_ops_lock);
diff --git a/drivers/block/ataflop.c b/drivers/block/ataflop.c
index 2104b1b..9ab759b 100644
--- a/drivers/block/ataflop.c
+++ b/drivers/block/ataflop.c
@@ -1933,6 +1933,11 @@ static int __init atari_floppy_init (void)
unit[i].disk = alloc_disk(1);
if (!unit[i].disk)
goto Enomem;
+
+ unit[i].disk->queue = blk_init_queue(do_fd_request,
+ &ataflop_lock);
+ if (!unit[i].disk->queue)
+ goto Enomem;
}
if (UseTrackbuffer < 0)
@@ -1964,10 +1969,6 @@ static int __init atari_floppy_init (void)
sprintf(unit[i].disk->disk_name, "fd%d", i);
unit[i].disk->fops = &floppy_fops;
unit[i].disk->private_data = &unit[i];
- unit[i].disk->queue = blk_init_queue(do_fd_request,
- &ataflop_lock);
- if (!unit[i].disk->queue)
- goto Enomem;
set_capacity(unit[i].disk, MAX_DISK_SIZE * 2);
add_disk(unit[i].disk);
}
@@ -1982,13 +1983,17 @@ static int __init atari_floppy_init (void)
return 0;
Enomem:
- while (i--) {
- struct request_queue *q = unit[i].disk->queue;
+ do {
+ struct gendisk *disk = unit[i].disk;
- put_disk(unit[i].disk);
- if (q)
- blk_cleanup_queue(q);
- }
+ if (disk) {
+ if (disk->queue) {
+ blk_cleanup_queue(disk->queue);
+ disk->queue = NULL;
+ }
+ put_disk(unit[i].disk);
+ }
+ } while (i--);
unregister_blkdev(FLOPPY_MAJOR, "fd");
return -ENOMEM;
diff --git a/drivers/block/drbd/drbd_worker.c b/drivers/block/drbd/drbd_worker.c
index c6755c9..51c233c 100644
--- a/drivers/block/drbd/drbd_worker.c
+++ b/drivers/block/drbd/drbd_worker.c
@@ -269,8 +269,8 @@ void drbd_request_endio(struct bio *bio)
what = COMPLETED_OK;
}
- bio_put(req->private_bio);
req->private_bio = ERR_PTR(bio->bi_error);
+ bio_put(bio);
/* not req_mod(), we need irqsave here! */
spin_lock_irqsave(&device->resource->req_lock, flags);
diff --git a/drivers/block/floppy.c b/drivers/block/floppy.c
index e3d8e4c..326b9ba 100644
--- a/drivers/block/floppy.c
+++ b/drivers/block/floppy.c
@@ -3459,6 +3459,9 @@ static int fd_locked_ioctl(struct block_device *bdev, fmode_t mode, unsigned int
(struct floppy_struct **)&outparam);
if (ret)
return ret;
+ memcpy(&inparam.g, outparam,
+ offsetof(struct floppy_struct, name));
+ outparam = &inparam.g;
break;
case FDMSGON:
UDP->flags |= FTD_MSG;
@@ -3820,10 +3823,11 @@ static int __floppy_read_block_0(struct block_device *bdev, int drive)
bio.bi_end_io = floppy_rb0_cb;
bio_set_op_attrs(&bio, REQ_OP_READ, 0);
+ init_completion(&cbdata.complete);
+
submit_bio(&bio);
process_fd_request();
- init_completion(&cbdata.complete);
wait_for_completion(&cbdata.complete);
__free_page(page);
diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index ff1c4d7..b3e432a 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -640,6 +640,36 @@ static void loop_reread_partitions(struct loop_device *lo,
__func__, lo->lo_number, lo->lo_file_name, rc);
}
+static inline int is_loop_device(struct file *file)
+{
+ struct inode *i = file->f_mapping->host;
+
+ return i && S_ISBLK(i->i_mode) && MAJOR(i->i_rdev) == LOOP_MAJOR;
+}
+
+static int loop_validate_file(struct file *file, struct block_device *bdev)
+{
+ struct inode *inode = file->f_mapping->host;
+ struct file *f = file;
+
+ /* Avoid recursion */
+ while (is_loop_device(f)) {
+ struct loop_device *l;
+
+ if (f->f_mapping->host->i_bdev == bdev)
+ return -EBADF;
+
+ l = f->f_mapping->host->i_bdev->bd_disk->private_data;
+ if (l->lo_state == Lo_unbound) {
+ return -EINVAL;
+ }
+ f = l->lo_backing_file;
+ }
+ if (!S_ISREG(inode->i_mode) && !S_ISBLK(inode->i_mode))
+ return -EINVAL;
+ return 0;
+}
+
/*
* loop_change_fd switched the backing store of a loopback device to
* a new file. This is useful for operating system installers to free up
@@ -669,14 +699,15 @@ static int loop_change_fd(struct loop_device *lo, struct block_device *bdev,
if (!file)
goto out;
+ error = loop_validate_file(file, bdev);
+ if (error)
+ goto out_putf;
+
inode = file->f_mapping->host;
old_file = lo->lo_backing_file;
error = -EINVAL;
- if (!S_ISREG(inode->i_mode) && !S_ISBLK(inode->i_mode))
- goto out_putf;
-
/* size of the new backing store needs to be the same */
if (get_loop_size(lo, file) != get_loop_size(lo, old_file))
goto out_putf;
@@ -697,13 +728,6 @@ static int loop_change_fd(struct loop_device *lo, struct block_device *bdev,
return error;
}
-static inline int is_loop_device(struct file *file)
-{
- struct inode *i = file->f_mapping->host;
-
- return i && S_ISBLK(i->i_mode) && MAJOR(i->i_rdev) == LOOP_MAJOR;
-}
-
/* loop sysfs attributes */
static ssize_t loop_attr_show(struct device *dev, char *page,
@@ -800,16 +824,17 @@ static struct attribute_group loop_attribute_group = {
.attrs= loop_attrs,
};
-static int loop_sysfs_init(struct loop_device *lo)
+static void loop_sysfs_init(struct loop_device *lo)
{
- return sysfs_create_group(&disk_to_dev(lo->lo_disk)->kobj,
- &loop_attribute_group);
+ lo->sysfs_inited = !sysfs_create_group(&disk_to_dev(lo->lo_disk)->kobj,
+ &loop_attribute_group);
}
static void loop_sysfs_exit(struct loop_device *lo)
{
- sysfs_remove_group(&disk_to_dev(lo->lo_disk)->kobj,
- &loop_attribute_group);
+ if (lo->sysfs_inited)
+ sysfs_remove_group(&disk_to_dev(lo->lo_disk)->kobj,
+ &loop_attribute_group);
}
static void loop_config_discard(struct loop_device *lo)
@@ -861,7 +886,7 @@ static int loop_prepare_queue(struct loop_device *lo)
static int loop_set_fd(struct loop_device *lo, fmode_t mode,
struct block_device *bdev, unsigned int arg)
{
- struct file *file, *f;
+ struct file *file;
struct inode *inode;
struct address_space *mapping;
unsigned lo_blocksize;
@@ -881,29 +906,13 @@ static int loop_set_fd(struct loop_device *lo, fmode_t mode,
if (lo->lo_state != Lo_unbound)
goto out_putf;
- /* Avoid recursion */
- f = file;
- while (is_loop_device(f)) {
- struct loop_device *l;
-
- if (f->f_mapping->host->i_bdev == bdev)
- goto out_putf;
-
- l = f->f_mapping->host->i_bdev->bd_disk->private_data;
- if (l->lo_state == Lo_unbound) {
- error = -EINVAL;
- goto out_putf;
- }
- f = l->lo_backing_file;
- }
+ error = loop_validate_file(file, bdev);
+ if (error)
+ goto out_putf;
mapping = file->f_mapping;
inode = mapping->host;
- error = -EINVAL;
- if (!S_ISREG(inode->i_mode) && !S_ISBLK(inode->i_mode))
- goto out_putf;
-
if (!(file->f_mode & FMODE_WRITE) || !(mode & FMODE_WRITE) ||
!file->f_op->write_iter)
lo_flags |= LO_FLAGS_READ_ONLY;
@@ -1050,6 +1059,7 @@ static int loop_clr_fd(struct loop_device *lo)
memset(lo->lo_encrypt_key, 0, LO_KEY_SIZE);
memset(lo->lo_crypt_name, 0, LO_NAME_SIZE);
memset(lo->lo_file_name, 0, LO_NAME_SIZE);
+ blk_queue_logical_block_size(lo->lo_queue, 512);
if (bdev) {
bdput(bdev);
invalidate_bdev(bdev);
@@ -1100,6 +1110,12 @@ loop_set_status(struct loop_device *lo, const struct loop_info64 *info)
if ((unsigned int) info->lo_encrypt_key_size > LO_KEY_SIZE)
return -EINVAL;
+ if (lo->lo_offset != info->lo_offset ||
+ lo->lo_sizelimit != info->lo_sizelimit) {
+ sync_blockdev(lo->lo_device);
+ kill_bdev(lo->lo_device);
+ }
+
/* I/O need to be drained during transfer transition */
blk_mq_freeze_queue(lo->lo_queue);
@@ -1127,11 +1143,20 @@ loop_set_status(struct loop_device *lo, const struct loop_info64 *info)
goto exit;
if (lo->lo_offset != info->lo_offset ||
- lo->lo_sizelimit != info->lo_sizelimit)
+ lo->lo_sizelimit != info->lo_sizelimit) {
+ /* kill_bdev should have truncated all the pages */
+ if (lo->lo_device->bd_inode->i_mapping->nrpages) {
+ err = -EAGAIN;
+ pr_warn("%s: loop%d (%s) has still dirty pages (nrpages=%lu)\n",
+ __func__, lo->lo_number, lo->lo_file_name,
+ lo->lo_device->bd_inode->i_mapping->nrpages);
+ goto exit;
+ }
if (figure_loop_size(lo, info->lo_offset, info->lo_sizelimit)) {
err = -EFBIG;
goto exit;
}
+ }
loop_config_discard(lo);
@@ -1335,6 +1360,41 @@ static int loop_set_dio(struct loop_device *lo, unsigned long arg)
return error;
}
+static int loop_set_block_size(struct loop_device *lo, unsigned long arg)
+{
+ int err = 0;
+
+ if (lo->lo_state != Lo_bound)
+ return -ENXIO;
+
+ if (arg < 512 || arg > PAGE_SIZE || !is_power_of_2(arg))
+ return -EINVAL;
+
+ if (lo->lo_queue->limits.logical_block_size != arg) {
+ sync_blockdev(lo->lo_device);
+ kill_bdev(lo->lo_device);
+ }
+
+ blk_mq_freeze_queue(lo->lo_queue);
+
+ /* kill_bdev should have truncated all the pages */
+ if (lo->lo_queue->limits.logical_block_size != arg &&
+ lo->lo_device->bd_inode->i_mapping->nrpages) {
+ err = -EAGAIN;
+ pr_warn("%s: loop%d (%s) has still dirty pages (nrpages=%lu)\n",
+ __func__, lo->lo_number, lo->lo_file_name,
+ lo->lo_device->bd_inode->i_mapping->nrpages);
+ goto out_unfreeze;
+ }
+
+ blk_queue_logical_block_size(lo->lo_queue, arg);
+ loop_update_dio(lo);
+out_unfreeze:
+ blk_mq_unfreeze_queue(lo->lo_queue);
+
+ return err;
+}
+
static int lo_ioctl(struct block_device *bdev, fmode_t mode,
unsigned int cmd, unsigned long arg)
{
@@ -1383,6 +1443,11 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN))
err = loop_set_dio(lo, arg);
break;
+ case LOOP_SET_BLOCK_SIZE:
+ err = -EPERM;
+ if ((mode & FMODE_WRITE) || capable(CAP_SYS_ADMIN))
+ err = loop_set_block_size(lo, arg);
+ break;
default:
err = lo->ioctl ? lo->ioctl(lo, cmd, arg) : -EINVAL;
}
@@ -1537,6 +1602,7 @@ static int lo_compat_ioctl(struct block_device *bdev, fmode_t mode,
arg = (unsigned long) compat_ptr(arg);
case LOOP_SET_FD:
case LOOP_CHANGE_FD:
+ case LOOP_SET_BLOCK_SIZE:
err = lo_ioctl(bdev, mode, cmd, arg);
break;
default:
@@ -1772,6 +1838,7 @@ static int loop_add(struct loop_device **l, int i)
}
lo->lo_queue->queuedata = lo;
+ blk_queue_max_hw_sectors(lo->lo_queue, BLK_DEF_MAX_SECTORS);
/*
* It doesn't make sense to enable merge because the I/O
* submitted to backing file is handled page by page.
diff --git a/drivers/block/loop.h b/drivers/block/loop.h
index fb2237c..60f0fd2 100644
--- a/drivers/block/loop.h
+++ b/drivers/block/loop.h
@@ -59,6 +59,7 @@ struct loop_device {
struct kthread_worker worker;
struct task_struct *worker_task;
bool use_dio;
+ bool sysfs_inited;
struct request_queue *lo_queue;
struct blk_mq_tag_set tag_set;
diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index 4d30da2..42a5395 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -269,7 +269,7 @@ static inline int sock_send_bvec(struct nbd_device *nbd, struct bio_vec *bvec,
static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd)
{
struct request *req = blk_mq_rq_from_pdu(cmd);
- int result, flags;
+ int result;
struct nbd_request request;
unsigned long size = blk_rq_bytes(req);
struct bio *bio;
@@ -309,7 +309,6 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd)
if (type != NBD_CMD_WRITE)
return 0;
- flags = 0;
bio = req->bio;
while (bio) {
struct bio *next = bio->bi_next;
@@ -318,9 +317,8 @@ static int nbd_send_cmd(struct nbd_device *nbd, struct nbd_cmd *cmd)
bio_for_each_segment(bvec, bio, iter) {
bool is_last = !next && bio_iter_last(bvec, iter);
+ int flags = is_last ? 0 : MSG_MORE;
- if (is_last)
- flags = MSG_MORE;
dev_dbg(nbd_to_dev(nbd), "request %p: sending %d bytes data\n",
cmd, bvec.bv_len);
result = sock_send_bvec(nbd, &bvec, flags);
diff --git a/drivers/block/paride/pcd.c b/drivers/block/paride/pcd.c
index 9336236..8474a1b 100644
--- a/drivers/block/paride/pcd.c
+++ b/drivers/block/paride/pcd.c
@@ -230,6 +230,8 @@ static int pcd_block_open(struct block_device *bdev, fmode_t mode)
struct pcd_unit *cd = bdev->bd_disk->private_data;
int ret;
+ check_disk_change(bdev);
+
mutex_lock(&pcd_mutex);
ret = cdrom_open(&cd->info, bdev, mode);
mutex_unlock(&pcd_mutex);
diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c
index 194db0e2..8beee2a 100644
--- a/drivers/block/rbd.c
+++ b/drivers/block/rbd.c
@@ -3900,7 +3900,6 @@ static void cancel_tasks_sync(struct rbd_device *rbd_dev)
{
dout("%s rbd_dev %p\n", __func__, rbd_dev);
- cancel_delayed_work_sync(&rbd_dev->watch_dwork);
cancel_work_sync(&rbd_dev->acquired_lock_work);
cancel_work_sync(&rbd_dev->released_lock_work);
cancel_delayed_work_sync(&rbd_dev->lock_dwork);
@@ -3918,6 +3917,7 @@ static void rbd_unregister_watch(struct rbd_device *rbd_dev)
rbd_dev->watch_state = RBD_WATCH_STATE_UNREGISTERED;
mutex_unlock(&rbd_dev->watch_mutex);
+ cancel_delayed_work_sync(&rbd_dev->watch_dwork);
ceph_osdc_flush_notifies(&rbd_dev->rbd_client->client->osdc);
}
diff --git a/drivers/block/swim.c b/drivers/block/swim.c
index b5afd49..eec6e39 100644
--- a/drivers/block/swim.c
+++ b/drivers/block/swim.c
@@ -868,8 +868,17 @@ static int swim_floppy_init(struct swim_priv *swd)
exit_put_disks:
unregister_blkdev(FLOPPY_MAJOR, "fd");
- while (drive--)
- put_disk(swd->unit[drive].disk);
+ do {
+ struct gendisk *disk = swd->unit[drive].disk;
+
+ if (disk) {
+ if (disk->queue) {
+ blk_cleanup_queue(disk->queue);
+ disk->queue = NULL;
+ }
+ put_disk(disk);
+ }
+ } while (drive--);
return err;
}
diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c
index f11d62d..c08ee8c 100644
--- a/drivers/block/xen-blkfront.c
+++ b/drivers/block/xen-blkfront.c
@@ -2524,6 +2524,9 @@ static int blkfront_remove(struct xenbus_device *xbdev)
dev_dbg(&xbdev->dev, "%s removed", xbdev->nodename);
+ if (!info)
+ return 0;
+
blkif_free(info, 0);
mutex_lock(&info->mutex);
diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index b8ecba6..e8f2fa2 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -12,4 +12,29 @@
It has several use cases, for example: /tmp storage, use as swap
disks and maybe many more.
- See zram.txt for more information.
+ See Documentation/blockdev/zram.txt for more information.
+
+config ZRAM_WRITEBACK
+ bool "Write back incompressible or idle page to backing device"
+ depends on ZRAM
+ default n
+ help
+ With incompressible page, there is no memory saving to keep it
+ in memory. Instead, write it out to backing device.
+ For this feature, admin should set up backing device via
+ /sys/block/zramX/backing_dev.
+
+ With /sys/block/zramX/{idle,writeback}, application could ask
+ idle page's writeback to the backing device to save in memory.
+
+ See Documentation/blockdev/zram.txt for more information.
+
+config ZRAM_MEMORY_TRACKING
+ bool "Track zRam block status"
+ depends on ZRAM && DEBUG_FS
+ help
+ With this feature, admin can track the state of allocated blocks
+ of zRAM. Admin could see the information via
+ /sys/kernel/debug/zram/zramX/block_state.
+
+ See Documentation/blockdev/zram.txt for more information.
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 58f7c39..27dc9cc 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -31,6 +31,7 @@
#include <linux/err.h>
#include <linux/idr.h>
#include <linux/sysfs.h>
+#include <linux/debugfs.h>
#include "zram_drv.h"
@@ -50,26 +51,25 @@ static const char *default_compressor = "lzo";
/* Module params (documentation at end) */
static unsigned int num_devices = 1;
-static inline void deprecated_attr_warn(const char *name)
+static void zram_free_page(struct zram *zram, size_t index);
+static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
+ u32 index, int offset, struct bio *bio);
+
+
+static int zram_slot_trylock(struct zram *zram, u32 index)
{
- pr_warn_once("%d (%s) Attribute %s (and others) will be removed. %s\n",
- task_pid_nr(current),
- current->comm,
- name,
- "See zram documentation.");
+ return bit_spin_trylock(ZRAM_LOCK, &zram->table[index].flags);
}
-#define ZRAM_ATTR_RO(name) \
-static ssize_t name##_show(struct device *d, \
- struct device_attribute *attr, char *b) \
-{ \
- struct zram *zram = dev_to_zram(d); \
- \
- deprecated_attr_warn(__stringify(name)); \
- return scnprintf(b, PAGE_SIZE, "%llu\n", \
- (u64)atomic64_read(&zram->stats.name)); \
-} \
-static DEVICE_ATTR_RO(name);
+static void zram_slot_lock(struct zram *zram, u32 index)
+{
+ bit_spin_lock(ZRAM_LOCK, &zram->table[index].flags);
+}
+
+static void zram_slot_unlock(struct zram *zram, u32 index)
+{
+ bit_spin_unlock(ZRAM_LOCK, &zram->table[index].flags);
+}
static inline bool init_done(struct zram *zram)
{
@@ -81,50 +81,77 @@ static inline struct zram *dev_to_zram(struct device *dev)
return (struct zram *)dev_to_disk(dev)->private_data;
}
+static unsigned long zram_get_handle(struct zram *zram, u32 index)
+{
+ return zram->table[index].handle;
+}
+
+static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle)
+{
+ zram->table[index].handle = handle;
+}
+
/* flag operations require table entry bit_spin_lock() being held */
-static int zram_test_flag(struct zram_meta *meta, u32 index,
+static bool zram_test_flag(struct zram *zram, u32 index,
enum zram_pageflags flag)
{
- return meta->table[index].value & BIT(flag);
+ return zram->table[index].flags & BIT(flag);
}
-static void zram_set_flag(struct zram_meta *meta, u32 index,
+static void zram_set_flag(struct zram *zram, u32 index,
enum zram_pageflags flag)
{
- meta->table[index].value |= BIT(flag);
+ zram->table[index].flags |= BIT(flag);
}
-static void zram_clear_flag(struct zram_meta *meta, u32 index,
+static void zram_clear_flag(struct zram *zram, u32 index,
enum zram_pageflags flag)
{
- meta->table[index].value &= ~BIT(flag);
+ zram->table[index].flags &= ~BIT(flag);
}
-static size_t zram_get_obj_size(struct zram_meta *meta, u32 index)
+static inline void zram_set_element(struct zram *zram, u32 index,
+ unsigned long element)
{
- return meta->table[index].value & (BIT(ZRAM_FLAG_SHIFT) - 1);
+ zram->table[index].element = element;
}
-static void zram_set_obj_size(struct zram_meta *meta,
+static unsigned long zram_get_element(struct zram *zram, u32 index)
+{
+ return zram->table[index].element;
+}
+
+static size_t zram_get_obj_size(struct zram *zram, u32 index)
+{
+ return zram->table[index].flags & (BIT(ZRAM_FLAG_SHIFT) - 1);
+}
+
+static void zram_set_obj_size(struct zram *zram,
u32 index, size_t size)
{
- unsigned long flags = meta->table[index].value >> ZRAM_FLAG_SHIFT;
+ unsigned long flags = zram->table[index].flags >> ZRAM_FLAG_SHIFT;
- meta->table[index].value = (flags << ZRAM_FLAG_SHIFT) | size;
+ zram->table[index].flags = (flags << ZRAM_FLAG_SHIFT) | size;
}
+static inline bool zram_allocated(struct zram *zram, u32 index)
+{
+ return zram_get_obj_size(zram, index) ||
+ zram_test_flag(zram, index, ZRAM_SAME) ||
+ zram_test_flag(zram, index, ZRAM_WB);
+}
+
+#if PAGE_SIZE != 4096
static inline bool is_partial_io(struct bio_vec *bvec)
{
return bvec->bv_len != PAGE_SIZE;
}
-
-static void zram_revalidate_disk(struct zram *zram)
+#else
+static inline bool is_partial_io(struct bio_vec *bvec)
{
- revalidate_disk(zram->disk);
- /* revalidate_disk reset the BDI_CAP_STABLE_WRITES so set again */
- zram->disk->queue->backing_dev_info->capabilities |=
- BDI_CAP_STABLE_WRITES;
+ return false;
}
+#endif
/*
* Check if request is within bounds and aligned on zram logical blocks.
@@ -152,8 +179,7 @@ static inline bool valid_io_request(struct zram *zram,
static void update_position(u32 *index, int *offset, struct bio_vec *bvec)
{
- if (*offset + bvec->bv_len >= PAGE_SIZE)
- (*index)++;
+ *index += (*offset + bvec->bv_len) / PAGE_SIZE;
*offset = (*offset + bvec->bv_len) % PAGE_SIZE;
}
@@ -172,36 +198,41 @@ static inline void update_used_max(struct zram *zram,
} while (old_max != cur_max);
}
-static bool page_zero_filled(void *ptr)
+static inline void zram_fill_page(char *ptr, unsigned long len,
+ unsigned long value)
+{
+ int i;
+ unsigned long *page = (unsigned long *)ptr;
+
+ WARN_ON_ONCE(!IS_ALIGNED(len, sizeof(unsigned long)));
+
+ if (likely(value == 0)) {
+ memset(ptr, 0, len);
+ } else {
+ for (i = 0; i < len / sizeof(*page); i++)
+ page[i] = value;
+ }
+}
+
+static bool page_same_filled(void *ptr, unsigned long *element)
{
unsigned int pos;
unsigned long *page;
+ unsigned long val;
page = (unsigned long *)ptr;
+ val = page[0];
- for (pos = 0; pos != PAGE_SIZE / sizeof(*page); pos++) {
- if (page[pos])
+ for (pos = 1; pos < PAGE_SIZE / sizeof(*page); pos++) {
+ if (val != page[pos])
return false;
}
+ *element = val;
+
return true;
}
-static void handle_zero_page(struct bio_vec *bvec)
-{
- struct page *page = bvec->bv_page;
- void *user_mem;
-
- user_mem = kmap_atomic(page);
- if (is_partial_io(bvec))
- memset(user_mem + bvec->bv_offset, 0, bvec->bv_len);
- else
- clear_page(user_mem);
- kunmap_atomic(user_mem);
-
- flush_dcache_page(page);
-}
-
static ssize_t initstate_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
@@ -223,47 +254,6 @@ static ssize_t disksize_show(struct device *dev,
return scnprintf(buf, PAGE_SIZE, "%llu\n", zram->disksize);
}
-static ssize_t orig_data_size_show(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- struct zram *zram = dev_to_zram(dev);
-
- deprecated_attr_warn("orig_data_size");
- return scnprintf(buf, PAGE_SIZE, "%llu\n",
- (u64)(atomic64_read(&zram->stats.pages_stored)) << PAGE_SHIFT);
-}
-
-static ssize_t mem_used_total_show(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- u64 val = 0;
- struct zram *zram = dev_to_zram(dev);
-
- deprecated_attr_warn("mem_used_total");
- down_read(&zram->init_lock);
- if (init_done(zram)) {
- struct zram_meta *meta = zram->meta;
- val = zs_get_total_pages(meta->mem_pool);
- }
- up_read(&zram->init_lock);
-
- return scnprintf(buf, PAGE_SIZE, "%llu\n", val << PAGE_SHIFT);
-}
-
-static ssize_t mem_limit_show(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- u64 val;
- struct zram *zram = dev_to_zram(dev);
-
- deprecated_attr_warn("mem_limit");
- down_read(&zram->init_lock);
- val = zram->limit_pages;
- up_read(&zram->init_lock);
-
- return scnprintf(buf, PAGE_SIZE, "%llu\n", val << PAGE_SHIFT);
-}
-
static ssize_t mem_limit_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
@@ -282,21 +272,6 @@ static ssize_t mem_limit_store(struct device *dev,
return len;
}
-static ssize_t mem_used_max_show(struct device *dev,
- struct device_attribute *attr, char *buf)
-{
- u64 val = 0;
- struct zram *zram = dev_to_zram(dev);
-
- deprecated_attr_warn("mem_used_max");
- down_read(&zram->init_lock);
- if (init_done(zram))
- val = atomic_long_read(&zram->stats.max_used_pages);
- up_read(&zram->init_lock);
-
- return scnprintf(buf, PAGE_SIZE, "%llu\n", val << PAGE_SHIFT);
-}
-
static ssize_t mem_used_max_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
@@ -310,15 +285,704 @@ static ssize_t mem_used_max_store(struct device *dev,
down_read(&zram->init_lock);
if (init_done(zram)) {
- struct zram_meta *meta = zram->meta;
atomic_long_set(&zram->stats.max_used_pages,
- zs_get_total_pages(meta->mem_pool));
+ zs_get_total_pages(zram->mem_pool));
}
up_read(&zram->init_lock);
return len;
}
+static ssize_t idle_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t len)
+{
+ struct zram *zram = dev_to_zram(dev);
+ unsigned long nr_pages = zram->disksize >> PAGE_SHIFT;
+ int index;
+ char mode_buf[8];
+ ssize_t sz;
+
+ sz = strscpy(mode_buf, buf, sizeof(mode_buf));
+ if (sz <= 0)
+ return -EINVAL;
+
+ /* ignore trailing new line */
+ if (mode_buf[sz - 1] == '\n')
+ mode_buf[sz - 1] = 0x00;
+
+ if (strcmp(mode_buf, "all"))
+ return -EINVAL;
+
+ down_read(&zram->init_lock);
+ if (!init_done(zram)) {
+ up_read(&zram->init_lock);
+ return -EINVAL;
+ }
+
+ for (index = 0; index < nr_pages; index++) {
+ /*
+ * Do not mark ZRAM_UNDER_WB slot as ZRAM_IDLE to close race.
+ * See the comment in writeback_store.
+ */
+ zram_slot_lock(zram, index);
+ if (zram_allocated(zram, index) &&
+ !zram_test_flag(zram, index, ZRAM_UNDER_WB))
+ zram_set_flag(zram, index, ZRAM_IDLE);
+ zram_slot_unlock(zram, index);
+ }
+
+ up_read(&zram->init_lock);
+
+ return len;
+}
+
+#ifdef CONFIG_ZRAM_WRITEBACK
+static ssize_t writeback_limit_enable_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t len)
+{
+ struct zram *zram = dev_to_zram(dev);
+ u64 val;
+ ssize_t ret = -EINVAL;
+
+ if (kstrtoull(buf, 10, &val))
+ return ret;
+
+ down_read(&zram->init_lock);
+ spin_lock(&zram->wb_limit_lock);
+ zram->wb_limit_enable = val;
+ spin_unlock(&zram->wb_limit_lock);
+ up_read(&zram->init_lock);
+ ret = len;
+
+ return ret;
+}
+
+static ssize_t writeback_limit_enable_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ bool val;
+ struct zram *zram = dev_to_zram(dev);
+
+ down_read(&zram->init_lock);
+ spin_lock(&zram->wb_limit_lock);
+ val = zram->wb_limit_enable;
+ spin_unlock(&zram->wb_limit_lock);
+ up_read(&zram->init_lock);
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t writeback_limit_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t len)
+{
+ struct zram *zram = dev_to_zram(dev);
+ u64 val;
+ ssize_t ret = -EINVAL;
+
+ if (kstrtoull(buf, 10, &val))
+ return ret;
+
+ down_read(&zram->init_lock);
+ spin_lock(&zram->wb_limit_lock);
+ zram->bd_wb_limit = val;
+ spin_unlock(&zram->wb_limit_lock);
+ up_read(&zram->init_lock);
+ ret = len;
+
+ return ret;
+}
+
+static ssize_t writeback_limit_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ u64 val;
+ struct zram *zram = dev_to_zram(dev);
+
+ down_read(&zram->init_lock);
+ spin_lock(&zram->wb_limit_lock);
+ val = zram->bd_wb_limit;
+ spin_unlock(&zram->wb_limit_lock);
+ up_read(&zram->init_lock);
+
+ return scnprintf(buf, PAGE_SIZE, "%llu\n", val);
+}
+
+static void reset_bdev(struct zram *zram)
+{
+ struct block_device *bdev;
+
+ if (!zram->backing_dev)
+ return;
+
+ bdev = zram->bdev;
+ if (zram->old_block_size)
+ set_blocksize(bdev, zram->old_block_size);
+ blkdev_put(bdev, FMODE_READ|FMODE_WRITE|FMODE_EXCL);
+ /* hope filp_close flush all of IO */
+ filp_close(zram->backing_dev, NULL);
+ zram->backing_dev = NULL;
+ zram->old_block_size = 0;
+ zram->bdev = NULL;
+
+ kvfree(zram->bitmap);
+ zram->bitmap = NULL;
+}
+
+static ssize_t backing_dev_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct zram *zram = dev_to_zram(dev);
+ struct file *file = zram->backing_dev;
+ char *p;
+ ssize_t ret;
+
+ down_read(&zram->init_lock);
+ if (!zram->backing_dev) {
+ memcpy(buf, "none\n", 5);
+ up_read(&zram->init_lock);
+ return 5;
+ }
+
+ p = file_path(file, buf, PAGE_SIZE - 1);
+ if (IS_ERR(p)) {
+ ret = PTR_ERR(p);
+ goto out;
+ }
+
+ ret = strlen(p);
+ memmove(buf, p, ret);
+ buf[ret++] = '\n';
+out:
+ up_read(&zram->init_lock);
+ return ret;
+}
+
+static ssize_t backing_dev_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t len)
+{
+ char *file_name;
+ size_t sz;
+ struct file *backing_dev = NULL;
+ struct inode *inode;
+ struct address_space *mapping;
+ unsigned int bitmap_sz, old_block_size = 0;
+ unsigned long nr_pages, *bitmap = NULL;
+ struct block_device *bdev = NULL;
+ int err;
+ struct zram *zram = dev_to_zram(dev);
+ gfp_t kmalloc_flags;
+
+ file_name = kmalloc(PATH_MAX, GFP_KERNEL);
+ if (!file_name)
+ return -ENOMEM;
+
+ down_write(&zram->init_lock);
+ if (init_done(zram)) {
+ pr_info("Can't setup backing device for initialized device\n");
+ err = -EBUSY;
+ goto out;
+ }
+
+ strlcpy(file_name, buf, PATH_MAX);
+ /* ignore trailing newline */
+ sz = strlen(file_name);
+ if (sz > 0 && file_name[sz - 1] == '\n')
+ file_name[sz - 1] = 0x00;
+
+ backing_dev = filp_open(file_name, O_RDWR|O_LARGEFILE, 0);
+ if (IS_ERR(backing_dev)) {
+ err = PTR_ERR(backing_dev);
+ backing_dev = NULL;
+ goto out;
+ }
+
+ mapping = backing_dev->f_mapping;
+ inode = mapping->host;
+
+ /* Support only block device in this moment */
+ if (!S_ISBLK(inode->i_mode)) {
+ err = -ENOTBLK;
+ goto out;
+ }
+
+ bdev = bdgrab(I_BDEV(inode));
+ err = blkdev_get(bdev, FMODE_READ | FMODE_WRITE | FMODE_EXCL, zram);
+ if (err < 0) {
+ bdev = NULL;
+ goto out;
+ }
+
+ nr_pages = i_size_read(inode) >> PAGE_SHIFT;
+ bitmap_sz = BITS_TO_LONGS(nr_pages) * sizeof(long);
+ kmalloc_flags = GFP_KERNEL | __GFP_ZERO;
+ if (bitmap_sz > PAGE_SIZE)
+ kmalloc_flags |= __GFP_NOWARN | __GFP_NORETRY;
+
+ bitmap = kmalloc_node(bitmap_sz, kmalloc_flags, NUMA_NO_NODE);
+ if (!bitmap && bitmap_sz > PAGE_SIZE)
+ bitmap = vzalloc(bitmap_sz);
+
+ if (!bitmap) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ old_block_size = block_size(bdev);
+ err = set_blocksize(bdev, PAGE_SIZE);
+ if (err)
+ goto out;
+
+ reset_bdev(zram);
+
+ zram->old_block_size = old_block_size;
+ zram->bdev = bdev;
+ zram->backing_dev = backing_dev;
+ zram->bitmap = bitmap;
+ zram->nr_pages = nr_pages;
+ up_write(&zram->init_lock);
+
+ pr_info("setup backing device %s\n", file_name);
+ kfree(file_name);
+
+ return len;
+out:
+ if (bitmap)
+ kvfree(bitmap);
+
+ if (bdev)
+ blkdev_put(bdev, FMODE_READ | FMODE_WRITE | FMODE_EXCL);
+
+ if (backing_dev)
+ filp_close(backing_dev, NULL);
+
+ up_write(&zram->init_lock);
+
+ kfree(file_name);
+
+ return err;
+}
+
+static unsigned long alloc_block_bdev(struct zram *zram)
+{
+ unsigned long blk_idx = 1;
+retry:
+ /* skip 0 bit to confuse zram.handle = 0 */
+ blk_idx = find_next_zero_bit(zram->bitmap, zram->nr_pages, blk_idx);
+ if (blk_idx == zram->nr_pages)
+ return 0;
+
+ if (test_and_set_bit(blk_idx, zram->bitmap))
+ goto retry;
+
+ atomic64_inc(&zram->stats.bd_count);
+ return blk_idx;
+}
+
+static void free_block_bdev(struct zram *zram, unsigned long blk_idx)
+{
+ int was_set;
+
+ was_set = test_and_clear_bit(blk_idx, zram->bitmap);
+ WARN_ON_ONCE(!was_set);
+ atomic64_dec(&zram->stats.bd_count);
+}
+
+static void zram_page_end_io(struct bio *bio)
+{
+ struct page *page = bio->bi_io_vec[0].bv_page;
+
+ page_endio(page, op_is_write(bio_op(bio)), bio->bi_error);
+ bio_put(bio);
+}
+
+/*
+ * Returns 1 if the submission is successful.
+ */
+static int read_from_bdev_async(struct zram *zram, struct bio_vec *bvec,
+ unsigned long entry, struct bio *parent)
+{
+ struct bio *bio;
+
+ bio = bio_alloc(GFP_ATOMIC, 1);
+ if (!bio)
+ return -ENOMEM;
+
+ bio->bi_iter.bi_sector = entry * (PAGE_SIZE >> 9);
+ bio->bi_bdev = zram->bdev;
+ if (!bio_add_page(bio, bvec->bv_page, bvec->bv_len, bvec->bv_offset)) {
+ bio_put(bio);
+ return -EIO;
+ }
+
+ if (!parent) {
+ bio->bi_opf = REQ_OP_READ;
+ bio->bi_end_io = zram_page_end_io;
+ } else {
+ bio->bi_opf = parent->bi_opf;
+ bio_chain(bio, parent);
+ }
+
+ submit_bio(bio);
+ return 1;
+}
+
+#define HUGE_WRITEBACK 1
+#define IDLE_WRITEBACK 2
+
+static ssize_t writeback_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t len)
+{
+ struct zram *zram = dev_to_zram(dev);
+ unsigned long nr_pages = zram->disksize >> PAGE_SHIFT;
+ unsigned long index;
+ struct bio bio;
+ struct page *page;
+ ssize_t ret, sz;
+ char mode_buf[8];
+ int mode = -1;
+ unsigned long blk_idx = 0;
+
+ sz = strscpy(mode_buf, buf, sizeof(mode_buf));
+ if (sz <= 0)
+ return -EINVAL;
+
+ /* ignore trailing newline */
+ if (mode_buf[sz - 1] == '\n')
+ mode_buf[sz - 1] = 0x00;
+
+ if (!strcmp(mode_buf, "idle"))
+ mode = IDLE_WRITEBACK;
+ else if (!strcmp(mode_buf, "huge"))
+ mode = HUGE_WRITEBACK;
+
+ if (mode == -1)
+ return -EINVAL;
+
+ down_read(&zram->init_lock);
+ if (!init_done(zram)) {
+ ret = -EINVAL;
+ goto release_init_lock;
+ }
+
+ if (!zram->backing_dev) {
+ ret = -ENODEV;
+ goto release_init_lock;
+ }
+
+ page = alloc_page(GFP_KERNEL);
+ if (!page) {
+ ret = -ENOMEM;
+ goto release_init_lock;
+ }
+
+ for (index = 0; index < nr_pages; index++) {
+ struct bio_vec bvec;
+
+ bvec.bv_page = page;
+ bvec.bv_len = PAGE_SIZE;
+ bvec.bv_offset = 0;
+
+ spin_lock(&zram->wb_limit_lock);
+ if (zram->wb_limit_enable && !zram->bd_wb_limit) {
+ spin_unlock(&zram->wb_limit_lock);
+ ret = -EIO;
+ goto error;
+ }
+ spin_unlock(&zram->wb_limit_lock);
+
+ if (!blk_idx) {
+ blk_idx = alloc_block_bdev(zram);
+ if (!blk_idx) {
+ ret = -ENOSPC;
+ goto error;
+ }
+ }
+
+ zram_slot_lock(zram, index);
+ if (!zram_allocated(zram, index))
+ goto next;
+
+ if (zram_test_flag(zram, index, ZRAM_WB) ||
+ zram_test_flag(zram, index, ZRAM_SAME) ||
+ zram_test_flag(zram, index, ZRAM_UNDER_WB))
+ goto next;
+
+ if (mode == IDLE_WRITEBACK &&
+ !zram_test_flag(zram, index, ZRAM_IDLE))
+ goto next;
+
+ if (mode == HUGE_WRITEBACK &&
+ !zram_test_flag(zram, index, ZRAM_HUGE))
+ goto next;
+ /*
+ * Clearing ZRAM_UNDER_WB is duty of caller.
+ * IOW, zram_free_page never clear it.
+ */
+ zram_set_flag(zram, index, ZRAM_UNDER_WB);
+ /* Need for hugepage writeback racing */
+ zram_set_flag(zram, index, ZRAM_IDLE);
+ zram_slot_unlock(zram, index);
+ if (zram_bvec_read(zram, &bvec, index, 0, NULL)) {
+ zram_slot_lock(zram, index);
+ zram_clear_flag(zram, index, ZRAM_UNDER_WB);
+ zram_clear_flag(zram, index, ZRAM_IDLE);
+ zram_slot_unlock(zram, index);
+ continue;
+ }
+
+ bio_init(&bio);
+
+ bio.bi_max_vecs = 1;
+ bio.bi_io_vec = &bvec;
+ bio.bi_bdev = zram->bdev;
+
+ bio.bi_iter.bi_sector = blk_idx * (PAGE_SIZE >> 9);
+ bio_set_op_attrs(&bio, REQ_OP_WRITE, REQ_SYNC);
+ bio_add_page(&bio, bvec.bv_page, bvec.bv_len,
+ bvec.bv_offset);
+ /*
+ * XXX: A single page IO would be inefficient for write
+ * but it would be not bad as starter.
+ */
+ ret = submit_bio_wait(&bio);
+ if (ret) {
+ zram_slot_lock(zram, index);
+ zram_clear_flag(zram, index, ZRAM_UNDER_WB);
+ zram_clear_flag(zram, index, ZRAM_IDLE);
+ zram_slot_unlock(zram, index);
+ continue;
+ }
+
+ atomic64_inc(&zram->stats.bd_writes);
+ /*
+ * We released zram_slot_lock so need to check if the slot was
+ * changed. If there is freeing for the slot, we can catch it
+ * easily by zram_allocated.
+ * A subtle case is the slot is freed/reallocated/marked as
+ * ZRAM_IDLE again. To close the race, idle_store doesn't
+ * mark ZRAM_IDLE once it found the slot was ZRAM_UNDER_WB.
+ * Thus, we could close the race by checking ZRAM_IDLE bit.
+ */
+ zram_slot_lock(zram, index);
+ if (!zram_allocated(zram, index) ||
+ !zram_test_flag(zram, index, ZRAM_IDLE)) {
+ zram_clear_flag(zram, index, ZRAM_UNDER_WB);
+ zram_clear_flag(zram, index, ZRAM_IDLE);
+ goto next;
+ }
+
+ zram_free_page(zram, index);
+ zram_clear_flag(zram, index, ZRAM_UNDER_WB);
+ zram_set_flag(zram, index, ZRAM_WB);
+ zram_set_element(zram, index, blk_idx);
+ blk_idx = 0;
+ atomic64_inc(&zram->stats.pages_stored);
+
+ spin_lock(&zram->wb_limit_lock);
+ if (zram->wb_limit_enable && zram->bd_wb_limit > 0)
+ zram->bd_wb_limit -= min_t(u64,
+ 1UL << (PAGE_SHIFT - 12), zram->bd_wb_limit);
+ spin_unlock(&zram->wb_limit_lock);
+next:
+ zram_slot_unlock(zram, index);
+ }
+
+ ret = len;
+error:
+ if (blk_idx)
+ free_block_bdev(zram, blk_idx);
+ __free_page(page);
+release_init_lock:
+ up_read(&zram->init_lock);
+
+ return ret;
+}
+
+struct zram_work {
+ struct work_struct work;
+ struct zram *zram;
+ unsigned long entry;
+ struct bio *bio;
+};
+
+#if PAGE_SIZE != 4096
+static void zram_sync_read(struct work_struct *work)
+{
+ struct bio_vec bvec;
+ struct zram_work *zw = container_of(work, struct zram_work, work);
+ struct zram *zram = zw->zram;
+ unsigned long entry = zw->entry;
+ struct bio *bio = zw->bio;
+
+ read_from_bdev_async(zram, &bvec, entry, bio);
+}
+
+/*
+ * Block layer want one ->make_request_fn to be active at a time
+ * so if we use chained IO with parent IO in same context,
+ * it's a deadlock. To avoid, it, it uses worker thread context.
+ */
+static int read_from_bdev_sync(struct zram *zram, struct bio_vec *bvec,
+ unsigned long entry, struct bio *bio)
+{
+ struct zram_work work;
+
+ work.zram = zram;
+ work.entry = entry;
+ work.bio = bio;
+
+ INIT_WORK_ONSTACK(&work.work, zram_sync_read);
+ queue_work(system_unbound_wq, &work.work);
+ flush_work(&work.work);
+ destroy_work_on_stack(&work.work);
+
+ return 1;
+}
+#else
+static int read_from_bdev_sync(struct zram *zram, struct bio_vec *bvec,
+ unsigned long entry, struct bio *bio)
+{
+ WARN_ON(1);
+ return -EIO;
+}
+#endif
+
+static int read_from_bdev(struct zram *zram, struct bio_vec *bvec,
+ unsigned long entry, struct bio *parent, bool sync)
+{
+ atomic64_inc(&zram->stats.bd_reads);
+ if (sync)
+ return read_from_bdev_sync(zram, bvec, entry, parent);
+ else
+ return read_from_bdev_async(zram, bvec, entry, parent);
+}
+#else
+static inline void reset_bdev(struct zram *zram) {};
+static int read_from_bdev(struct zram *zram, struct bio_vec *bvec,
+ unsigned long entry, struct bio *parent, bool sync)
+{
+ return -EIO;
+}
+
+static void free_block_bdev(struct zram *zram, unsigned long blk_idx) {};
+#endif
+
+#ifdef CONFIG_ZRAM_MEMORY_TRACKING
+
+static struct dentry *zram_debugfs_root;
+
+static void zram_debugfs_create(void)
+{
+ zram_debugfs_root = debugfs_create_dir("zram", NULL);
+}
+
+static void zram_debugfs_destroy(void)
+{
+ debugfs_remove_recursive(zram_debugfs_root);
+}
+
+static void zram_accessed(struct zram *zram, u32 index)
+{
+ zram_clear_flag(zram, index, ZRAM_IDLE);
+ zram->table[index].ac_time = ktime_get_boottime();
+}
+
+static ssize_t read_block_state(struct file *file, char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ char *kbuf;
+ ssize_t index, written = 0;
+ struct zram *zram = file->private_data;
+ unsigned long nr_pages = zram->disksize >> PAGE_SHIFT;
+ struct timespec64 ts;
+
+ gfp_t kmalloc_flags;
+
+ kmalloc_flags = GFP_KERNEL;
+ if (count > PAGE_SIZE)
+ kmalloc_flags |= __GFP_NOWARN | __GFP_NORETRY;
+
+ kbuf = kmalloc_node(count, kmalloc_flags, NUMA_NO_NODE);
+ if (!kbuf && count > PAGE_SIZE)
+ kbuf = vmalloc(count);
+ if (!kbuf)
+ return -ENOMEM;
+
+ down_read(&zram->init_lock);
+ if (!init_done(zram)) {
+ up_read(&zram->init_lock);
+ kvfree(kbuf);
+ return -EINVAL;
+ }
+
+ for (index = *ppos; index < nr_pages; index++) {
+ int copied;
+
+ zram_slot_lock(zram, index);
+ if (!zram_allocated(zram, index))
+ goto next;
+
+ ts = ktime_to_timespec64(zram->table[index].ac_time);
+ copied = snprintf(kbuf + written, count,
+ "%12zd %12lld.%06lu %c%c%c%c\n",
+ index, (s64)ts.tv_sec,
+ ts.tv_nsec / NSEC_PER_USEC,
+ zram_test_flag(zram, index, ZRAM_SAME) ? 's' : '.',
+ zram_test_flag(zram, index, ZRAM_WB) ? 'w' : '.',
+ zram_test_flag(zram, index, ZRAM_HUGE) ? 'h' : '.',
+ zram_test_flag(zram, index, ZRAM_IDLE) ? 'i' : '.');
+
+ if (count < copied) {
+ zram_slot_unlock(zram, index);
+ break;
+ }
+ written += copied;
+ count -= copied;
+next:
+ zram_slot_unlock(zram, index);
+ *ppos += 1;
+ }
+
+ up_read(&zram->init_lock);
+ if (copy_to_user(buf, kbuf, written))
+ written = -EFAULT;
+ kvfree(kbuf);
+
+ return written;
+}
+
+static const struct file_operations proc_zram_block_state_op = {
+ .open = simple_open,
+ .read = read_block_state,
+ .llseek = default_llseek,
+};
+
+static void zram_debugfs_register(struct zram *zram)
+{
+ if (!zram_debugfs_root)
+ return;
+
+ zram->debugfs_dir = debugfs_create_dir(zram->disk->disk_name,
+ zram_debugfs_root);
+ debugfs_create_file("block_state", 0400, zram->debugfs_dir,
+ zram, &proc_zram_block_state_op);
+}
+
+static void zram_debugfs_unregister(struct zram *zram)
+{
+ debugfs_remove_recursive(zram->debugfs_dir);
+}
+#else
+static void zram_debugfs_create(void) {};
+static void zram_debugfs_destroy(void) {};
+static void zram_accessed(struct zram *zram, u32 index)
+{
+ zram_clear_flag(zram, index, ZRAM_IDLE);
+};
+static void zram_debugfs_register(struct zram *zram) {};
+static void zram_debugfs_unregister(struct zram *zram) {};
+#endif
+
/*
* We switched to per-cpu streams and this attr is not needed anymore.
* However, we will keep it around for some time, because:
@@ -357,7 +1021,7 @@ static ssize_t comp_algorithm_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
struct zram *zram = dev_to_zram(dev);
- char compressor[CRYPTO_MAX_ALG_NAME];
+ char compressor[ARRAY_SIZE(zram->compressor)];
size_t sz;
strlcpy(compressor, buf, sizeof(compressor));
@@ -376,7 +1040,7 @@ static ssize_t comp_algorithm_store(struct device *dev,
return -EBUSY;
}
- strlcpy(zram->compressor, compressor, sizeof(compressor));
+ strcpy(zram->compressor, compressor);
up_write(&zram->init_lock);
return len;
}
@@ -385,7 +1049,6 @@ static ssize_t compact_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
struct zram *zram = dev_to_zram(dev);
- struct zram_meta *meta;
down_read(&zram->init_lock);
if (!init_done(zram)) {
@@ -393,8 +1056,7 @@ static ssize_t compact_store(struct device *dev,
return -EINVAL;
}
- meta = zram->meta;
- zs_compact(meta->mem_pool);
+ zs_compact(zram->mem_pool);
up_read(&zram->init_lock);
return len;
@@ -431,27 +1093,48 @@ static ssize_t mm_stat_show(struct device *dev,
down_read(&zram->init_lock);
if (init_done(zram)) {
- mem_used = zs_get_total_pages(zram->meta->mem_pool);
- zs_pool_stats(zram->meta->mem_pool, &pool_stats);
+ mem_used = zs_get_total_pages(zram->mem_pool);
+ zs_pool_stats(zram->mem_pool, &pool_stats);
}
orig_size = atomic64_read(&zram->stats.pages_stored);
max_used = atomic_long_read(&zram->stats.max_used_pages);
ret = scnprintf(buf, PAGE_SIZE,
- "%8llu %8llu %8llu %8lu %8ld %8llu %8lu\n",
+ "%8llu %8llu %8llu %8lu %8ld %8llu %8lu %8llu\n",
orig_size << PAGE_SHIFT,
(u64)atomic64_read(&zram->stats.compr_data_size),
mem_used << PAGE_SHIFT,
zram->limit_pages << PAGE_SHIFT,
max_used << PAGE_SHIFT,
- (u64)atomic64_read(&zram->stats.zero_pages),
- pool_stats.pages_compacted);
+ (u64)atomic64_read(&zram->stats.same_pages),
+ pool_stats.pages_compacted,
+ (u64)atomic64_read(&zram->stats.huge_pages));
up_read(&zram->init_lock);
return ret;
}
+#ifdef CONFIG_ZRAM_WRITEBACK
+#define FOUR_K(x) ((x) * (1 << (PAGE_SHIFT - 12)))
+static ssize_t bd_stat_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct zram *zram = dev_to_zram(dev);
+ ssize_t ret;
+
+ down_read(&zram->init_lock);
+ ret = scnprintf(buf, PAGE_SIZE,
+ "%8llu %8llu %8llu\n",
+ FOUR_K((u64)atomic64_read(&zram->stats.bd_count)),
+ FOUR_K((u64)atomic64_read(&zram->stats.bd_reads)),
+ FOUR_K((u64)atomic64_read(&zram->stats.bd_writes)));
+ up_read(&zram->init_lock);
+
+ return ret;
+}
+#endif
+
static ssize_t debug_stat_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
@@ -461,9 +1144,10 @@ static ssize_t debug_stat_show(struct device *dev,
down_read(&zram->init_lock);
ret = scnprintf(buf, PAGE_SIZE,
- "version: %d\n%8llu\n",
+ "version: %d\n%8llu %8llu\n",
version,
- (u64)atomic64_read(&zram->stats.writestall));
+ (u64)atomic64_read(&zram->stats.writestall),
+ (u64)atomic64_read(&zram->stats.miss_free));
up_read(&zram->init_lock);
return ret;
@@ -471,75 +1155,40 @@ static ssize_t debug_stat_show(struct device *dev,
static DEVICE_ATTR_RO(io_stat);
static DEVICE_ATTR_RO(mm_stat);
+#ifdef CONFIG_ZRAM_WRITEBACK
+static DEVICE_ATTR_RO(bd_stat);
+#endif
static DEVICE_ATTR_RO(debug_stat);
-ZRAM_ATTR_RO(num_reads);
-ZRAM_ATTR_RO(num_writes);
-ZRAM_ATTR_RO(failed_reads);
-ZRAM_ATTR_RO(failed_writes);
-ZRAM_ATTR_RO(invalid_io);
-ZRAM_ATTR_RO(notify_free);
-ZRAM_ATTR_RO(zero_pages);
-ZRAM_ATTR_RO(compr_data_size);
-static inline bool zram_meta_get(struct zram *zram)
-{
- if (atomic_inc_not_zero(&zram->refcount))
- return true;
- return false;
-}
-
-static inline void zram_meta_put(struct zram *zram)
-{
- atomic_dec(&zram->refcount);
-}
-
-static void zram_meta_free(struct zram_meta *meta, u64 disksize)
+static void zram_meta_free(struct zram *zram, u64 disksize)
{
size_t num_pages = disksize >> PAGE_SHIFT;
size_t index;
/* Free all pages that are still in this zram device */
- for (index = 0; index < num_pages; index++) {
- unsigned long handle = meta->table[index].handle;
+ for (index = 0; index < num_pages; index++)
+ zram_free_page(zram, index);
- if (!handle)
- continue;
-
- zs_free(meta->mem_pool, handle);
- }
-
- zs_destroy_pool(meta->mem_pool);
- vfree(meta->table);
- kfree(meta);
+ zs_destroy_pool(zram->mem_pool);
+ vfree(zram->table);
}
-static struct zram_meta *zram_meta_alloc(char *pool_name, u64 disksize)
+static bool zram_meta_alloc(struct zram *zram, u64 disksize)
{
size_t num_pages;
- struct zram_meta *meta = kmalloc(sizeof(*meta), GFP_KERNEL);
-
- if (!meta)
- return NULL;
num_pages = disksize >> PAGE_SHIFT;
- meta->table = vzalloc(num_pages * sizeof(*meta->table));
- if (!meta->table) {
- pr_err("Error allocating zram address table\n");
- goto out_error;
+ zram->table = vzalloc(num_pages * sizeof(*zram->table));
+ if (!zram->table)
+ return false;
+
+ zram->mem_pool = zs_create_pool(zram->disk->disk_name);
+ if (!zram->mem_pool) {
+ vfree(zram->table);
+ return false;
}
- meta->mem_pool = zs_create_pool(pool_name);
- if (!meta->mem_pool) {
- pr_err("Error creating memory pool\n");
- goto out_error;
- }
-
- return meta;
-
-out_error:
- vfree(meta->table);
- kfree(meta);
- return NULL;
+ return true;
}
/*
@@ -549,194 +1198,183 @@ static struct zram_meta *zram_meta_alloc(char *pool_name, u64 disksize)
*/
static void zram_free_page(struct zram *zram, size_t index)
{
- struct zram_meta *meta = zram->meta;
- unsigned long handle = meta->table[index].handle;
+ unsigned long handle;
- if (unlikely(!handle)) {
- /*
- * No memory is allocated for zero filled pages.
- * Simply clear zero page flag.
- */
- if (zram_test_flag(meta, index, ZRAM_ZERO)) {
- zram_clear_flag(meta, index, ZRAM_ZERO);
- atomic64_dec(&zram->stats.zero_pages);
- }
- return;
+#ifdef CONFIG_ZRAM_MEMORY_TRACKING
+ zram->table[index].ac_time.tv64 = 0;
+#endif
+ if (zram_test_flag(zram, index, ZRAM_IDLE))
+ zram_clear_flag(zram, index, ZRAM_IDLE);
+
+ if (zram_test_flag(zram, index, ZRAM_HUGE)) {
+ zram_clear_flag(zram, index, ZRAM_HUGE);
+ atomic64_dec(&zram->stats.huge_pages);
}
- zs_free(meta->mem_pool, handle);
+ if (zram_test_flag(zram, index, ZRAM_WB)) {
+ zram_clear_flag(zram, index, ZRAM_WB);
+ free_block_bdev(zram, zram_get_element(zram, index));
+ goto out;
+ }
- atomic64_sub(zram_get_obj_size(meta, index),
+ /*
+ * No memory is allocated for same element filled pages.
+ * Simply clear same page flag.
+ */
+ if (zram_test_flag(zram, index, ZRAM_SAME)) {
+ zram_clear_flag(zram, index, ZRAM_SAME);
+ atomic64_dec(&zram->stats.same_pages);
+ goto out;
+ }
+
+ handle = zram_get_handle(zram, index);
+ if (!handle)
+ return;
+
+ zs_free(zram->mem_pool, handle);
+
+ atomic64_sub(zram_get_obj_size(zram, index),
&zram->stats.compr_data_size);
+out:
atomic64_dec(&zram->stats.pages_stored);
-
- meta->table[index].handle = 0;
- zram_set_obj_size(meta, index, 0);
+ zram_set_handle(zram, index, 0);
+ zram_set_obj_size(zram, index, 0);
+ WARN_ON_ONCE(zram->table[index].flags &
+ ~(1UL << ZRAM_LOCK | 1UL << ZRAM_UNDER_WB));
}
-static int zram_decompress_page(struct zram *zram, char *mem, u32 index)
+static int __zram_bvec_read(struct zram *zram, struct page *page, u32 index,
+ struct bio *bio, bool partial_io)
{
- int ret = 0;
- unsigned char *cmem;
- struct zram_meta *meta = zram->meta;
+ int ret;
unsigned long handle;
unsigned int size;
+ void *src, *dst;
- bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value);
- handle = meta->table[index].handle;
- size = zram_get_obj_size(meta, index);
+ zram_slot_lock(zram, index);
+ if (zram_test_flag(zram, index, ZRAM_WB)) {
+ struct bio_vec bvec;
- if (!handle || zram_test_flag(meta, index, ZRAM_ZERO)) {
- bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value);
- memset(mem, 0, PAGE_SIZE);
+ zram_slot_unlock(zram, index);
+
+ bvec.bv_page = page;
+ bvec.bv_len = PAGE_SIZE;
+ bvec.bv_offset = 0;
+ return read_from_bdev(zram, &bvec,
+ zram_get_element(zram, index),
+ bio, partial_io);
+ }
+
+ handle = zram_get_handle(zram, index);
+ if (!handle || zram_test_flag(zram, index, ZRAM_SAME)) {
+ unsigned long value;
+ void *mem;
+
+ value = handle ? zram_get_element(zram, index) : 0;
+ mem = kmap_atomic(page);
+ zram_fill_page(mem, PAGE_SIZE, value);
+ kunmap_atomic(mem);
+ zram_slot_unlock(zram, index);
return 0;
}
- cmem = zs_map_object(meta->mem_pool, handle, ZS_MM_RO);
+ size = zram_get_obj_size(zram, index);
+
+ src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO);
if (size == PAGE_SIZE) {
- memcpy(mem, cmem, PAGE_SIZE);
+ dst = kmap_atomic(page);
+ memcpy(dst, src, PAGE_SIZE);
+ kunmap_atomic(dst);
+ ret = 0;
} else {
struct zcomp_strm *zstrm = zcomp_stream_get(zram->comp);
- ret = zcomp_decompress(zstrm, cmem, size, mem);
+ dst = kmap_atomic(page);
+ ret = zcomp_decompress(zstrm, src, size, dst);
+ kunmap_atomic(dst);
zcomp_stream_put(zram->comp);
}
- zs_unmap_object(meta->mem_pool, handle);
- bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value);
+ zs_unmap_object(zram->mem_pool, handle);
+ zram_slot_unlock(zram, index);
/* Should NEVER happen. Return bio error if it does. */
- if (unlikely(ret)) {
- pr_err("Decompression failed! err=%d, page=%u\n", ret, index);
- return ret;
- }
-
- return 0;
-}
-
-static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
- u32 index, int offset)
-{
- int ret;
- struct page *page;
- unsigned char *user_mem, *uncmem = NULL;
- struct zram_meta *meta = zram->meta;
- page = bvec->bv_page;
-
- bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value);
- if (unlikely(!meta->table[index].handle) ||
- zram_test_flag(meta, index, ZRAM_ZERO)) {
- bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value);
- handle_zero_page(bvec);
- return 0;
- }
- bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value);
-
- if (is_partial_io(bvec))
- /* Use a temporary buffer to decompress the page */
- uncmem = kmalloc(PAGE_SIZE, GFP_NOIO);
-
- user_mem = kmap_atomic(page);
- if (!is_partial_io(bvec))
- uncmem = user_mem;
-
- if (!uncmem) {
- pr_err("Unable to allocate temp memory\n");
- ret = -ENOMEM;
- goto out_cleanup;
- }
-
- ret = zram_decompress_page(zram, uncmem, index);
- /* Should NEVER happen. Return bio error if it does. */
if (unlikely(ret))
- goto out_cleanup;
+ pr_err("Decompression failed! err=%d, page=%u\n", ret, index);
- if (is_partial_io(bvec))
- memcpy(user_mem + bvec->bv_offset, uncmem + offset,
- bvec->bv_len);
-
- flush_dcache_page(page);
- ret = 0;
-out_cleanup:
- kunmap_atomic(user_mem);
- if (is_partial_io(bvec))
- kfree(uncmem);
return ret;
}
-static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, u32 index,
- int offset)
+static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
+ u32 index, int offset, struct bio *bio)
{
- int ret = 0;
- unsigned int clen;
- unsigned long handle = 0;
+ int ret;
struct page *page;
- unsigned char *user_mem, *cmem, *src, *uncmem = NULL;
- struct zram_meta *meta = zram->meta;
- struct zcomp_strm *zstrm = NULL;
- unsigned long alloced_pages;
- static unsigned long zram_rs_time;
page = bvec->bv_page;
if (is_partial_io(bvec)) {
- /*
- * This is a partial IO. We need to read the full page
- * before to write the changes.
- */
- uncmem = kmalloc(PAGE_SIZE, GFP_NOIO);
- if (!uncmem) {
- ret = -ENOMEM;
- goto out;
- }
- ret = zram_decompress_page(zram, uncmem, index);
- if (ret)
- goto out;
+ /* Use a temporary buffer to decompress the page */
+ page = alloc_page(GFP_NOIO|__GFP_HIGHMEM);
+ if (!page)
+ return -ENOMEM;
}
+ ret = __zram_bvec_read(zram, page, index, bio, is_partial_io(bvec));
+ if (unlikely(ret))
+ goto out;
+
+ if (is_partial_io(bvec)) {
+ void *dst = kmap_atomic(bvec->bv_page);
+ void *src = kmap_atomic(page);
+
+ memcpy(dst + bvec->bv_offset, src + offset, bvec->bv_len);
+ kunmap_atomic(src);
+ kunmap_atomic(dst);
+ }
+out:
+ if (is_partial_io(bvec))
+ __free_page(page);
+
+ return ret;
+}
+
+static int __zram_bvec_write(struct zram *zram, struct bio_vec *bvec,
+ u32 index, struct bio *bio)
+{
+ int ret = 0;
+ unsigned long alloced_pages;
+ unsigned long handle = 0;
+ unsigned int comp_len = 0;
+ void *src, *dst, *mem;
+ struct zcomp_strm *zstrm;
+ struct page *page = bvec->bv_page;
+ unsigned long element = 0;
+ enum zram_pageflags flags = 0;
+
+ mem = kmap_atomic(page);
+ if (page_same_filled(mem, &element)) {
+ kunmap_atomic(mem);
+ /* Free memory associated with this sector now. */
+ flags = ZRAM_SAME;
+ atomic64_inc(&zram->stats.same_pages);
+ goto out;
+ }
+ kunmap_atomic(mem);
+
compress_again:
- user_mem = kmap_atomic(page);
- if (is_partial_io(bvec)) {
- memcpy(uncmem + offset, user_mem + bvec->bv_offset,
- bvec->bv_len);
- kunmap_atomic(user_mem);
- user_mem = NULL;
- } else {
- uncmem = user_mem;
- }
-
- if (page_zero_filled(uncmem)) {
- if (user_mem)
- kunmap_atomic(user_mem);
- /* Free memory associated with this sector now. */
- bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value);
- zram_free_page(zram, index);
- zram_set_flag(meta, index, ZRAM_ZERO);
- bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value);
-
- atomic64_inc(&zram->stats.zero_pages);
- ret = 0;
- goto out;
- }
-
zstrm = zcomp_stream_get(zram->comp);
- ret = zcomp_compress(zstrm, uncmem, &clen);
- if (!is_partial_io(bvec)) {
- kunmap_atomic(user_mem);
- user_mem = NULL;
- uncmem = NULL;
- }
+ src = kmap_atomic(page);
+ ret = zcomp_compress(zstrm, src, &comp_len);
+ kunmap_atomic(src);
if (unlikely(ret)) {
+ zcomp_stream_put(zram->comp);
pr_err("Compression failed! err=%d\n", ret);
- goto out;
+ zs_free(zram->mem_pool, handle);
+ return ret;
}
- src = zstrm->buffer;
- if (unlikely(clen > max_zpage_size)) {
- clen = PAGE_SIZE;
- if (is_partial_io(bvec))
- src = uncmem;
- }
-
+ if (comp_len > max_zpage_size)
+ comp_len = PAGE_SIZE;
/*
* handle allocation has 2 paths:
* a) fast path is executed with preemption disabled (for
@@ -751,73 +1389,108 @@ static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, u32 index,
* from the slow path and handle has already been allocated.
*/
if (!handle)
- handle = zs_malloc(meta->mem_pool, clen,
+ handle = zs_malloc(zram->mem_pool, comp_len,
__GFP_KSWAPD_RECLAIM |
__GFP_NOWARN |
__GFP_HIGHMEM |
__GFP_MOVABLE);
if (!handle) {
zcomp_stream_put(zram->comp);
- zstrm = NULL;
-
atomic64_inc(&zram->stats.writestall);
-
- handle = zs_malloc(meta->mem_pool, clen,
+ handle = zs_malloc(zram->mem_pool, comp_len,
GFP_NOIO | __GFP_HIGHMEM |
__GFP_MOVABLE);
if (handle)
goto compress_again;
-
- if (printk_timed_ratelimit(&zram_rs_time,
- ALLOC_ERROR_LOG_RATE_MS))
- pr_err("Error allocating memory for compressed page: %u, size=%u\n",
- index, clen);
- ret = -ENOMEM;
- goto out;
+ return -ENOMEM;
}
- alloced_pages = zs_get_total_pages(meta->mem_pool);
+ alloced_pages = zs_get_total_pages(zram->mem_pool);
update_used_max(zram, alloced_pages);
if (zram->limit_pages && alloced_pages > zram->limit_pages) {
- zs_free(meta->mem_pool, handle);
- ret = -ENOMEM;
- goto out;
+ zcomp_stream_put(zram->comp);
+ zs_free(zram->mem_pool, handle);
+ return -ENOMEM;
}
- cmem = zs_map_object(meta->mem_pool, handle, ZS_MM_WO);
+ dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO);
- if ((clen == PAGE_SIZE) && !is_partial_io(bvec)) {
+ src = zstrm->buffer;
+ if (comp_len == PAGE_SIZE)
src = kmap_atomic(page);
- memcpy(cmem, src, PAGE_SIZE);
+ memcpy(dst, src, comp_len);
+ if (comp_len == PAGE_SIZE)
kunmap_atomic(src);
- } else {
- memcpy(cmem, src, clen);
- }
zcomp_stream_put(zram->comp);
- zstrm = NULL;
- zs_unmap_object(meta->mem_pool, handle);
-
+ zs_unmap_object(zram->mem_pool, handle);
+ atomic64_add(comp_len, &zram->stats.compr_data_size);
+out:
/*
* Free memory associated with this sector
* before overwriting unused sectors.
*/
- bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value);
+ zram_slot_lock(zram, index);
zram_free_page(zram, index);
- meta->table[index].handle = handle;
- zram_set_obj_size(meta, index, clen);
- bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value);
+ if (comp_len == PAGE_SIZE) {
+ zram_set_flag(zram, index, ZRAM_HUGE);
+ atomic64_inc(&zram->stats.huge_pages);
+ }
+
+ if (flags) {
+ zram_set_flag(zram, index, flags);
+ zram_set_element(zram, index, element);
+ } else {
+ zram_set_handle(zram, index, handle);
+ zram_set_obj_size(zram, index, comp_len);
+ }
+ zram_slot_unlock(zram, index);
/* Update stats */
- atomic64_add(clen, &zram->stats.compr_data_size);
atomic64_inc(&zram->stats.pages_stored);
+ return ret;
+}
+
+static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec,
+ u32 index, int offset, struct bio *bio)
+{
+ int ret;
+ struct page *page = NULL;
+ void *src;
+ struct bio_vec vec;
+
+ vec = *bvec;
+ if (is_partial_io(bvec)) {
+ void *dst;
+ /*
+ * This is a partial IO. We need to read the full page
+ * before to write the changes.
+ */
+ page = alloc_page(GFP_NOIO|__GFP_HIGHMEM);
+ if (!page)
+ return -ENOMEM;
+
+ ret = __zram_bvec_read(zram, page, index, bio, true);
+ if (ret)
+ goto out;
+
+ src = kmap_atomic(bvec->bv_page);
+ dst = kmap_atomic(page);
+ memcpy(dst + offset, src + bvec->bv_offset, bvec->bv_len);
+ kunmap_atomic(dst);
+ kunmap_atomic(src);
+
+ vec.bv_page = page;
+ vec.bv_len = PAGE_SIZE;
+ vec.bv_offset = 0;
+ }
+
+ ret = __zram_bvec_write(zram, &vec, index, bio);
out:
- if (zstrm)
- zcomp_stream_put(zram->comp);
if (is_partial_io(bvec))
- kfree(uncmem);
+ __free_page(page);
return ret;
}
@@ -830,7 +1503,6 @@ static void zram_bio_discard(struct zram *zram, u32 index,
int offset, struct bio *bio)
{
size_t n = bio->bi_iter.bi_size;
- struct zram_meta *meta = zram->meta;
/*
* zram manages data in physical block size units. Because logical block
@@ -851,17 +1523,22 @@ static void zram_bio_discard(struct zram *zram, u32 index,
}
while (n >= PAGE_SIZE) {
- bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value);
+ zram_slot_lock(zram, index);
zram_free_page(zram, index);
- bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value);
+ zram_slot_unlock(zram, index);
atomic64_inc(&zram->stats.notify_free);
index++;
n -= PAGE_SIZE;
}
}
+/*
+ * Returns errno if it has some problem. Otherwise return 0 or 1.
+ * Returns 0 if IO request was done synchronously
+ * Returns 1 if IO request was successfully submitted.
+ */
static int zram_bvec_rw(struct zram *zram, struct bio_vec *bvec, u32 index,
- int offset, bool is_write)
+ int offset, bool is_write, struct bio *bio)
{
unsigned long start_time = jiffies;
int rw_acct = is_write ? REQ_OP_WRITE : REQ_OP_READ;
@@ -872,15 +1549,20 @@ static int zram_bvec_rw(struct zram *zram, struct bio_vec *bvec, u32 index,
if (!is_write) {
atomic64_inc(&zram->stats.num_reads);
- ret = zram_bvec_read(zram, bvec, index, offset);
+ ret = zram_bvec_read(zram, bvec, index, offset, bio);
+ flush_dcache_page(bvec->bv_page);
} else {
atomic64_inc(&zram->stats.num_writes);
- ret = zram_bvec_write(zram, bvec, index, offset);
+ ret = zram_bvec_write(zram, bvec, index, offset, bio);
}
generic_end_io_acct(rw_acct, &zram->disk->part0, start_time);
- if (unlikely(ret)) {
+ zram_slot_lock(zram, index);
+ zram_accessed(zram, index);
+ zram_slot_unlock(zram, index);
+
+ if (unlikely(ret < 0)) {
if (!is_write)
atomic64_inc(&zram->stats.failed_reads);
else
@@ -908,34 +1590,21 @@ static void __zram_make_request(struct zram *zram, struct bio *bio)
}
bio_for_each_segment(bvec, bio, iter) {
- int max_transfer_size = PAGE_SIZE - offset;
+ struct bio_vec bv = bvec;
+ unsigned int unwritten = bvec.bv_len;
- if (bvec.bv_len > max_transfer_size) {
- /*
- * zram_bvec_rw() can only make operation on a single
- * zram page. Split the bio vector.
- */
- struct bio_vec bv;
-
- bv.bv_page = bvec.bv_page;
- bv.bv_len = max_transfer_size;
- bv.bv_offset = bvec.bv_offset;
-
+ do {
+ bv.bv_len = min_t(unsigned int, PAGE_SIZE - offset,
+ unwritten);
if (zram_bvec_rw(zram, &bv, index, offset,
- op_is_write(bio_op(bio))) < 0)
+ op_is_write(bio_op(bio)), bio) < 0)
goto out;
- bv.bv_len = bvec.bv_len - max_transfer_size;
- bv.bv_offset += max_transfer_size;
- if (zram_bvec_rw(zram, &bv, index + 1, 0,
- op_is_write(bio_op(bio))) < 0)
- goto out;
- } else
- if (zram_bvec_rw(zram, &bvec, index, offset,
- op_is_write(bio_op(bio))) < 0)
- goto out;
+ bv.bv_offset += bv.bv_len;
+ unwritten -= bv.bv_len;
- update_position(&index, &offset, &bvec);
+ update_position(&index, &offset, &bv);
+ } while (unwritten);
}
bio_endio(bio);
@@ -952,22 +1621,15 @@ static blk_qc_t zram_make_request(struct request_queue *queue, struct bio *bio)
{
struct zram *zram = queue->queuedata;
- if (unlikely(!zram_meta_get(zram)))
- goto error;
-
- blk_queue_split(queue, &bio, queue->bio_split);
-
if (!valid_io_request(zram, bio->bi_iter.bi_sector,
bio->bi_iter.bi_size)) {
atomic64_inc(&zram->stats.invalid_io);
- goto put_zram;
+ goto error;
}
__zram_make_request(zram, bio);
- zram_meta_put(zram);
return BLK_QC_T_NONE;
-put_zram:
- zram_meta_put(zram);
+
error:
bio_io_error(bio);
return BLK_QC_T_NONE;
@@ -977,45 +1639,43 @@ static void zram_slot_free_notify(struct block_device *bdev,
unsigned long index)
{
struct zram *zram;
- struct zram_meta *meta;
zram = bdev->bd_disk->private_data;
- meta = zram->meta;
- bit_spin_lock(ZRAM_ACCESS, &meta->table[index].value);
- zram_free_page(zram, index);
- bit_spin_unlock(ZRAM_ACCESS, &meta->table[index].value);
atomic64_inc(&zram->stats.notify_free);
+ if (!zram_slot_trylock(zram, index)) {
+ atomic64_inc(&zram->stats.miss_free);
+ return;
+ }
+
+ zram_free_page(zram, index);
+ zram_slot_unlock(zram, index);
}
static int zram_rw_page(struct block_device *bdev, sector_t sector,
struct page *page, bool is_write)
{
- int offset, err = -EIO;
+ int offset, ret;
u32 index;
struct zram *zram;
struct bio_vec bv;
zram = bdev->bd_disk->private_data;
- if (unlikely(!zram_meta_get(zram)))
- goto out;
if (!valid_io_request(zram, sector, PAGE_SIZE)) {
atomic64_inc(&zram->stats.invalid_io);
- err = -EINVAL;
- goto put_zram;
+ ret = -EINVAL;
+ goto out;
}
index = sector >> SECTORS_PER_PAGE_SHIFT;
- offset = sector & (SECTORS_PER_PAGE - 1) << SECTOR_SHIFT;
+ offset = (sector & (SECTORS_PER_PAGE - 1)) << SECTOR_SHIFT;
bv.bv_page = page;
bv.bv_len = PAGE_SIZE;
bv.bv_offset = 0;
- err = zram_bvec_rw(zram, &bv, index, offset, is_write);
-put_zram:
- zram_meta_put(zram);
+ ret = zram_bvec_rw(zram, &bv, index, offset, is_write, NULL);
out:
/*
* If I/O fails, just return error(ie, non-zero) without
@@ -1025,14 +1685,24 @@ static int zram_rw_page(struct block_device *bdev, sector_t sector,
* bio->bi_end_io does things to handle the error
* (e.g., SetPageError, set_page_dirty and extra works).
*/
- if (err == 0)
+ if (unlikely(ret < 0))
+ return ret;
+
+ switch (ret) {
+ case 0:
page_endio(page, is_write, 0);
- return err;
+ break;
+ case 1:
+ ret = 0;
+ break;
+ default:
+ WARN_ON(1);
+ }
+ return ret;
}
static void zram_reset_device(struct zram *zram)
{
- struct zram_meta *meta;
struct zcomp *comp;
u64 disksize;
@@ -1045,23 +1715,8 @@ static void zram_reset_device(struct zram *zram)
return;
}
- meta = zram->meta;
comp = zram->comp;
disksize = zram->disksize;
- /*
- * Refcount will go down to 0 eventually and r/w handler
- * cannot handle further I/O so it will bail out by
- * check zram_meta_get.
- */
- zram_meta_put(zram);
- /*
- * We want to free zram_meta in process context to avoid
- * deadlock between reclaim path and any other locks.
- */
- wait_event(zram->io_done, atomic_read(&zram->refcount) == 0);
-
- /* Reset stats */
- memset(&zram->stats, 0, sizeof(zram->stats));
zram->disksize = 0;
set_capacity(zram->disk, 0);
@@ -1069,8 +1724,10 @@ static void zram_reset_device(struct zram *zram)
up_write(&zram->init_lock);
/* I/O operation under all of CPU are done so let's free */
- zram_meta_free(meta, disksize);
+ zram_meta_free(zram, disksize);
+ memset(&zram->stats, 0, sizeof(zram->stats));
zcomp_destroy(comp);
+ reset_bdev(zram);
}
static ssize_t disksize_store(struct device *dev,
@@ -1078,7 +1735,6 @@ static ssize_t disksize_store(struct device *dev,
{
u64 disksize;
struct zcomp *comp;
- struct zram_meta *meta;
struct zram *zram = dev_to_zram(dev);
int err;
@@ -1086,10 +1742,18 @@ static ssize_t disksize_store(struct device *dev,
if (!disksize)
return -EINVAL;
+ down_write(&zram->init_lock);
+ if (init_done(zram)) {
+ pr_info("Cannot change disksize for initialized device\n");
+ err = -EBUSY;
+ goto out_unlock;
+ }
+
disksize = PAGE_ALIGN(disksize);
- meta = zram_meta_alloc(zram->disk->disk_name, disksize);
- if (!meta)
- return -ENOMEM;
+ if (!zram_meta_alloc(zram, disksize)) {
+ err = -ENOMEM;
+ goto out_unlock;
+ }
comp = zcomp_create(zram->compressor);
if (IS_ERR(comp)) {
@@ -1099,29 +1763,19 @@ static ssize_t disksize_store(struct device *dev,
goto out_free_meta;
}
- down_write(&zram->init_lock);
- if (init_done(zram)) {
- pr_info("Cannot change disksize for initialized device\n");
- err = -EBUSY;
- goto out_destroy_comp;
- }
-
- init_waitqueue_head(&zram->io_done);
- atomic_set(&zram->refcount, 1);
- zram->meta = meta;
zram->comp = comp;
zram->disksize = disksize;
set_capacity(zram->disk, zram->disksize >> SECTOR_SHIFT);
- zram_revalidate_disk(zram);
+
+ revalidate_disk(zram->disk);
up_write(&zram->init_lock);
return len;
-out_destroy_comp:
- up_write(&zram->init_lock);
- zcomp_destroy(comp);
out_free_meta:
- zram_meta_free(meta, disksize);
+ zram_meta_free(zram, disksize);
+out_unlock:
+ up_write(&zram->init_lock);
return err;
}
@@ -1160,7 +1814,7 @@ static ssize_t reset_store(struct device *dev,
/* Make sure all the pending I/O are finished */
fsync_bdev(bdev);
zram_reset_device(zram);
- zram_revalidate_disk(zram);
+ revalidate_disk(zram->disk);
bdput(bdev);
mutex_lock(&bdev->bd_mutex);
@@ -1196,42 +1850,52 @@ static DEVICE_ATTR_WO(compact);
static DEVICE_ATTR_RW(disksize);
static DEVICE_ATTR_RO(initstate);
static DEVICE_ATTR_WO(reset);
-static DEVICE_ATTR_RO(orig_data_size);
-static DEVICE_ATTR_RO(mem_used_total);
-static DEVICE_ATTR_RW(mem_limit);
-static DEVICE_ATTR_RW(mem_used_max);
+static DEVICE_ATTR_WO(mem_limit);
+static DEVICE_ATTR_WO(mem_used_max);
+static DEVICE_ATTR_WO(idle);
static DEVICE_ATTR_RW(max_comp_streams);
static DEVICE_ATTR_RW(comp_algorithm);
+#ifdef CONFIG_ZRAM_WRITEBACK
+static DEVICE_ATTR_RW(backing_dev);
+static DEVICE_ATTR_WO(writeback);
+static DEVICE_ATTR_RW(writeback_limit);
+static DEVICE_ATTR_RW(writeback_limit_enable);
+#endif
static struct attribute *zram_disk_attrs[] = {
&dev_attr_disksize.attr,
&dev_attr_initstate.attr,
&dev_attr_reset.attr,
- &dev_attr_num_reads.attr,
- &dev_attr_num_writes.attr,
- &dev_attr_failed_reads.attr,
- &dev_attr_failed_writes.attr,
&dev_attr_compact.attr,
- &dev_attr_invalid_io.attr,
- &dev_attr_notify_free.attr,
- &dev_attr_zero_pages.attr,
- &dev_attr_orig_data_size.attr,
- &dev_attr_compr_data_size.attr,
- &dev_attr_mem_used_total.attr,
&dev_attr_mem_limit.attr,
&dev_attr_mem_used_max.attr,
+ &dev_attr_idle.attr,
&dev_attr_max_comp_streams.attr,
&dev_attr_comp_algorithm.attr,
+#ifdef CONFIG_ZRAM_WRITEBACK
+ &dev_attr_backing_dev.attr,
+ &dev_attr_writeback.attr,
+ &dev_attr_writeback_limit.attr,
+ &dev_attr_writeback_limit_enable.attr,
+#endif
&dev_attr_io_stat.attr,
&dev_attr_mm_stat.attr,
+#ifdef CONFIG_ZRAM_WRITEBACK
+ &dev_attr_bd_stat.attr,
+#endif
&dev_attr_debug_stat.attr,
NULL,
};
-static struct attribute_group zram_disk_attr_group = {
+static const struct attribute_group zram_disk_attr_group = {
.attrs = zram_disk_attrs,
};
+static const struct attribute_group *zram_disk_attr_groups[] = {
+ &zram_disk_attr_group,
+ NULL,
+};
+
/*
* Allocate and initialize new zram device. the function returns
* '>= 0' device_id upon success, and negative value otherwise.
@@ -1252,7 +1916,9 @@ static int zram_add(void)
device_id = ret;
init_rwsem(&zram->init_lock);
-
+#ifdef CONFIG_ZRAM_WRITEBACK
+ spin_lock_init(&zram->wb_limit_lock);
+#endif
queue = blk_alloc_queue(GFP_KERNEL);
if (!queue) {
pr_err("Error allocating disk queue for device %d\n",
@@ -1286,6 +1952,7 @@ static int zram_add(void)
/* zram devices sort of resembles non-rotational disks */
queue_flag_set_unlocked(QUEUE_FLAG_NONROT, zram->disk->queue);
queue_flag_clear_unlocked(QUEUE_FLAG_ADD_RANDOM, zram->disk->queue);
+
/*
* To ensure that we always get PAGE_SIZE aligned
* and n*PAGE_SIZED sized I/O requests.
@@ -1296,8 +1963,6 @@ static int zram_add(void)
blk_queue_io_min(zram->disk->queue, PAGE_SIZE);
blk_queue_io_opt(zram->disk->queue, PAGE_SIZE);
zram->disk->queue->limits.discard_granularity = PAGE_SIZE;
- zram->disk->queue->limits.max_sectors = SECTORS_PER_PAGE;
- zram->disk->queue->limits.chunk_sectors = 0;
blk_queue_max_discard_sectors(zram->disk->queue, UINT_MAX);
/*
* zram_bio_discard() will clear all logical blocks if logical block
@@ -1313,24 +1978,17 @@ static int zram_add(void)
zram->disk->queue->limits.discard_zeroes_data = 0;
queue_flag_set_unlocked(QUEUE_FLAG_DISCARD, zram->disk->queue);
+ zram->disk->queue->backing_dev_info->capabilities |=
+ BDI_CAP_STABLE_WRITES;
+ disk_to_dev(zram->disk)->groups = zram_disk_attr_groups;
add_disk(zram->disk);
- ret = sysfs_create_group(&disk_to_dev(zram->disk)->kobj,
- &zram_disk_attr_group);
- if (ret < 0) {
- pr_err("Error creating sysfs group for device %d\n",
- device_id);
- goto out_free_disk;
- }
strlcpy(zram->compressor, default_compressor, sizeof(zram->compressor));
- zram->meta = NULL;
+ zram_debugfs_register(zram);
pr_info("Added device: %s\n", zram->disk->disk_name);
return device_id;
-out_free_disk:
- del_gendisk(zram->disk);
- put_disk(zram->disk);
out_free_queue:
blk_cleanup_queue(queue);
out_free_idr:
@@ -1358,16 +2016,7 @@ static int zram_remove(struct zram *zram)
zram->claim = true;
mutex_unlock(&bdev->bd_mutex);
- /*
- * Remove sysfs first, so no one will perform a disksize
- * store while we destroy the devices. This also helps during
- * hot_remove -- zram_reset_device() is the last holder of
- * ->init_lock, no later/concurrent disksize_store() or any
- * other sysfs handlers are possible.
- */
- sysfs_remove_group(&disk_to_dev(zram->disk)->kobj,
- &zram_disk_attr_group);
-
+ zram_debugfs_unregister(zram);
/* Make sure all the pending I/O are finished */
fsync_bdev(bdev);
zram_reset_device(zram);
@@ -1375,8 +2024,8 @@ static int zram_remove(struct zram *zram)
pr_info("Removed device: %s\n", zram->disk->disk_name);
- blk_cleanup_queue(zram->disk->queue);
del_gendisk(zram->disk);
+ blk_cleanup_queue(zram->disk->queue);
put_disk(zram->disk);
kfree(zram);
return 0;
@@ -1456,6 +2105,7 @@ static void destroy_devices(void)
{
class_unregister(&zram_control_class);
idr_for_each(&zram_index_idr, &zram_remove_cb, NULL);
+ zram_debugfs_destroy();
idr_destroy(&zram_index_idr);
unregister_blkdev(zram_major, "zram");
}
@@ -1470,6 +2120,7 @@ static int __init zram_init(void)
return ret;
}
+ zram_debugfs_create();
zram_major = register_blkdev(0, "zram");
if (zram_major <= 0) {
pr_err("Unable to get major number\n");
diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h
index 74fcf10..6ef2d5d 100644
--- a/drivers/block/zram/zram_drv.h
+++ b/drivers/block/zram/zram_drv.h
@@ -47,7 +47,7 @@ static const size_t max_zpage_size = PAGE_SIZE / 4 * 3;
/*
- * The lower ZRAM_FLAG_SHIFT bits of table.value is for
+ * The lower ZRAM_FLAG_SHIFT bits of table.flags is for
* object size (excluding header), the higher bits is for
* zram_pageflags.
*
@@ -58,11 +58,15 @@ static const size_t max_zpage_size = PAGE_SIZE / 4 * 3;
*/
#define ZRAM_FLAG_SHIFT 24
-/* Flags for zram pages (table[page_no].value) */
+/* Flags for zram pages (table[page_no].flags) */
enum zram_pageflags {
- /* Page consists entirely of zeros */
- ZRAM_ZERO = ZRAM_FLAG_SHIFT,
- ZRAM_ACCESS, /* page is now accessed */
+ /* zram slot is locked */
+ ZRAM_LOCK = ZRAM_FLAG_SHIFT,
+ ZRAM_SAME, /* Page consists the same element */
+ ZRAM_WB, /* page is stored on backing_device */
+ ZRAM_UNDER_WB, /* page is under writeback */
+ ZRAM_HUGE, /* Incompressible page */
+ ZRAM_IDLE, /* not accessed page since last idle marking */
__NR_ZRAM_PAGEFLAGS,
};
@@ -71,8 +75,14 @@ enum zram_pageflags {
/* Allocated for each disk page */
struct zram_table_entry {
- unsigned long handle;
- unsigned long value;
+ union {
+ unsigned long handle;
+ unsigned long element;
+ };
+ unsigned long flags;
+#ifdef CONFIG_ZRAM_MEMORY_TRACKING
+ ktime_t ac_time;
+#endif
};
struct zram_stats {
@@ -83,19 +93,22 @@ struct zram_stats {
atomic64_t failed_writes; /* can happen when memory is too low */
atomic64_t invalid_io; /* non-page-aligned I/O requests */
atomic64_t notify_free; /* no. of swap slot free notifications */
- atomic64_t zero_pages; /* no. of zero filled pages */
+ atomic64_t same_pages; /* no. of same element filled pages */
+ atomic64_t huge_pages; /* no. of huge pages */
atomic64_t pages_stored; /* no. of pages currently stored */
atomic_long_t max_used_pages; /* no. of maximum pages stored */
atomic64_t writestall; /* no. of write slow paths */
-};
-
-struct zram_meta {
- struct zram_table_entry *table;
- struct zs_pool *mem_pool;
+ atomic64_t miss_free; /* no. of missed free */
+#ifdef CONFIG_ZRAM_WRITEBACK
+ atomic64_t bd_count; /* no. of pages in backing device */
+ atomic64_t bd_reads; /* no. of reads from backing device */
+ atomic64_t bd_writes; /* no. of writes from backing device */
+#endif
};
struct zram {
- struct zram_meta *meta;
+ struct zram_table_entry *table;
+ struct zs_pool *mem_pool;
struct zcomp *comp;
struct gendisk *disk;
/* Prevent concurrent execution of device init */
@@ -106,9 +119,6 @@ struct zram {
unsigned long limit_pages;
struct zram_stats stats;
- atomic_t refcount; /* refcount for zram_meta */
- /* wait all IO under all of cpu are done */
- wait_queue_head_t io_done;
/*
* This is the limit on amount of *uncompressed* worth of data
* we can store in a disk.
@@ -119,5 +129,18 @@ struct zram {
* zram is claimed so open request will be failed
*/
bool claim; /* Protected by bdev->bd_mutex */
+ struct file *backing_dev;
+#ifdef CONFIG_ZRAM_WRITEBACK
+ spinlock_t wb_limit_lock;
+ bool wb_limit_enable;
+ u64 bd_wb_limit;
+ struct block_device *bdev;
+ unsigned int old_block_size;
+ unsigned long *bitmap;
+ unsigned long nr_pages;
+#endif
+#ifdef CONFIG_ZRAM_MEMORY_TRACKING
+ struct dentry *debugfs_dir;
+#endif
};
#endif
diff --git a/drivers/bluetooth/btbcm.c b/drivers/bluetooth/btbcm.c
index fdb4482..475f25c 100644
--- a/drivers/bluetooth/btbcm.c
+++ b/drivers/bluetooth/btbcm.c
@@ -270,6 +270,7 @@ static const struct {
{ 0x4103, "BCM4330B1" }, /* 002.001.003 */
{ 0x410e, "BCM43341B0" }, /* 002.001.014 */
{ 0x4406, "BCM4324B3" }, /* 002.004.006 */
+ { 0x6109, "BCM4335C0" }, /* 003.001.009 */
{ 0x610c, "BCM4354" }, /* 003.001.012 */
{ }
};
diff --git a/drivers/bluetooth/btfm_slim.c b/drivers/bluetooth/btfm_slim.c
index 92ea0135..baab63a 100644
--- a/drivers/bluetooth/btfm_slim.c
+++ b/drivers/bluetooth/btfm_slim.c
@@ -23,8 +23,8 @@
#include <sound/soc.h>
#include <sound/soc-dapm.h>
#include <sound/tlv.h>
-#include <btfm_slim.h>
-#include <btfm_slim_wcn3990.h>
+#include "btfm_slim.h"
+#include "btfm_slim_wcn3990.h"
#include <linux/bluetooth-power.h>
int btfm_slim_write(struct btfmslim *btfmslim,
diff --git a/drivers/bluetooth/btfm_slim_codec.c b/drivers/bluetooth/btfm_slim_codec.c
index 4c6986a..35c1c74 100644
--- a/drivers/bluetooth/btfm_slim_codec.c
+++ b/drivers/bluetooth/btfm_slim_codec.c
@@ -24,7 +24,7 @@
#include <sound/soc.h>
#include <sound/soc-dapm.h>
#include <sound/tlv.h>
-#include <btfm_slim.h>
+#include "btfm_slim.h"
static int bt_soc_enable_status;
int btfm_feedback_ch_setting;
diff --git a/drivers/bluetooth/btfm_slim_wcn3990.c b/drivers/bluetooth/btfm_slim_wcn3990.c
index 6615189..511c0b6 100644
--- a/drivers/bluetooth/btfm_slim_wcn3990.c
+++ b/drivers/bluetooth/btfm_slim_wcn3990.c
@@ -10,8 +10,8 @@
* GNU General Public License for more details.
*/
#include <linux/slimbus/slimbus.h>
-#include <btfm_slim.h>
-#include <btfm_slim_wcn3990.h>
+#include "btfm_slim.h"
+#include "btfm_slim_wcn3990.h"
/* WCN3990 Port assignment */
struct btfmslim_ch wcn3990_rxport[] = {
diff --git a/drivers/bluetooth/btfm_slim_wcn3990.h b/drivers/bluetooth/btfm_slim_wcn3990.h
index b2723ff..f266007 100644
--- a/drivers/bluetooth/btfm_slim_wcn3990.h
+++ b/drivers/bluetooth/btfm_slim_wcn3990.h
@@ -12,7 +12,7 @@
#ifndef BTFM_SLIM_WCN3990_H
#define BTFM_SLIM_WCN3990_H
//#ifdef CONFIG_BTFM_SLIM_WCN3990
-#include <btfm_slim.h>
+#include "btfm_slim.h"
/* Registers Address */
#define CHRK_SB_COMP_TEST 0x00000000
diff --git a/drivers/bluetooth/btusb.c b/drivers/bluetooth/btusb.c
index f8ba5c7..8dce1a8 100644
--- a/drivers/bluetooth/btusb.c
+++ b/drivers/bluetooth/btusb.c
@@ -217,6 +217,7 @@ static const struct usb_device_id blacklist_table[] = {
{ USB_DEVICE(0x0930, 0x0227), .driver_info = BTUSB_ATH3012 },
{ USB_DEVICE(0x0b05, 0x17d0), .driver_info = BTUSB_ATH3012 },
{ USB_DEVICE(0x0cf3, 0x0036), .driver_info = BTUSB_ATH3012 },
+ { USB_DEVICE(0x0cf3, 0x3004), .driver_info = BTUSB_ATH3012 },
{ USB_DEVICE(0x0cf3, 0x3008), .driver_info = BTUSB_ATH3012 },
{ USB_DEVICE(0x0cf3, 0x311d), .driver_info = BTUSB_ATH3012 },
{ USB_DEVICE(0x0cf3, 0x311e), .driver_info = BTUSB_ATH3012 },
@@ -249,7 +250,6 @@ static const struct usb_device_id blacklist_table[] = {
{ USB_DEVICE(0x0489, 0xe03c), .driver_info = BTUSB_ATH3012 },
/* QCA ROME chipset */
- { USB_DEVICE(0x0cf3, 0x3004), .driver_info = BTUSB_QCA_ROME },
{ USB_DEVICE(0x0cf3, 0xe007), .driver_info = BTUSB_QCA_ROME },
{ USB_DEVICE(0x0cf3, 0xe009), .driver_info = BTUSB_QCA_ROME },
{ USB_DEVICE(0x0cf3, 0xe300), .driver_info = BTUSB_QCA_ROME },
@@ -345,6 +345,13 @@ static const struct usb_device_id blacklist_table[] = {
{ USB_DEVICE(0x13d3, 0x3459), .driver_info = BTUSB_REALTEK },
{ USB_DEVICE(0x13d3, 0x3494), .driver_info = BTUSB_REALTEK },
+ /* Additional Realtek 8723BU Bluetooth devices */
+ { USB_DEVICE(0x7392, 0xa611), .driver_info = BTUSB_REALTEK },
+
+ /* Additional Realtek 8723DE Bluetooth devices */
+ { USB_DEVICE(0x0bda, 0xb009), .driver_info = BTUSB_REALTEK },
+ { USB_DEVICE(0x2ff8, 0xb011), .driver_info = BTUSB_REALTEK },
+
/* Additional Realtek 8821AE Bluetooth devices */
{ USB_DEVICE(0x0b05, 0x17dc), .driver_info = BTUSB_REALTEK },
{ USB_DEVICE(0x13d3, 0x3414), .driver_info = BTUSB_REALTEK },
@@ -352,6 +359,9 @@ static const struct usb_device_id blacklist_table[] = {
{ USB_DEVICE(0x13d3, 0x3461), .driver_info = BTUSB_REALTEK },
{ USB_DEVICE(0x13d3, 0x3462), .driver_info = BTUSB_REALTEK },
+ /* Additional Realtek 8822BE Bluetooth devices */
+ { USB_DEVICE(0x0b05, 0x185c), .driver_info = BTUSB_REALTEK },
+
/* Silicon Wave based devices */
{ USB_DEVICE(0x0c10, 0x0000), .driver_info = BTUSB_SWAVE },
diff --git a/drivers/bluetooth/hci_qca.c b/drivers/bluetooth/hci_qca.c
index 74b2f4a..0986c32 100644
--- a/drivers/bluetooth/hci_qca.c
+++ b/drivers/bluetooth/hci_qca.c
@@ -884,7 +884,7 @@ static int qca_set_baudrate(struct hci_dev *hdev, uint8_t baudrate)
*/
set_current_state(TASK_UNINTERRUPTIBLE);
schedule_timeout(msecs_to_jiffies(BAUDRATE_SETTLE_TIMEOUT_MS));
- set_current_state(TASK_INTERRUPTIBLE);
+ set_current_state(TASK_RUNNING);
return 0;
}
@@ -939,6 +939,12 @@ static int qca_setup(struct hci_uart *hu)
} else if (ret == -ENOENT) {
/* No patch/nvm-config found, run with original fw/config */
ret = 0;
+ } else if (ret == -EAGAIN) {
+ /*
+ * Userspace firmware loader will return -EAGAIN in case no
+ * patch/nvm-config is found, so run with original fw/config.
+ */
+ ret = 0;
}
/* Setup bdaddr */
diff --git a/drivers/bus/arm-cci.c b/drivers/bus/arm-cci.c
index 10f5613..8e08cb4 100644
--- a/drivers/bus/arm-cci.c
+++ b/drivers/bus/arm-cci.c
@@ -2103,8 +2103,6 @@ asmlinkage void __naked cci_enable_port_for_self(void)
[sizeof_struct_cpu_port] "i" (sizeof(struct cpu_port)),
[sizeof_struct_ace_port] "i" (sizeof(struct cci_ace_port)),
[offsetof_port_phys] "i" (offsetof(struct cci_ace_port, phys)) );
-
- unreachable();
}
/**
diff --git a/drivers/bus/arm-ccn.c b/drivers/bus/arm-ccn.c
index 45d7ecc..4e9e9e6 100644
--- a/drivers/bus/arm-ccn.c
+++ b/drivers/bus/arm-ccn.c
@@ -736,7 +736,7 @@ static int arm_ccn_pmu_event_init(struct perf_event *event)
ccn = pmu_to_arm_ccn(event->pmu);
if (hw->sample_period) {
- dev_warn(ccn->dev, "Sampling not supported!\n");
+ dev_dbg(ccn->dev, "Sampling not supported!\n");
return -EOPNOTSUPP;
}
@@ -744,12 +744,12 @@ static int arm_ccn_pmu_event_init(struct perf_event *event)
event->attr.exclude_kernel || event->attr.exclude_hv ||
event->attr.exclude_idle || event->attr.exclude_host ||
event->attr.exclude_guest) {
- dev_warn(ccn->dev, "Can't exclude execution levels!\n");
+ dev_dbg(ccn->dev, "Can't exclude execution levels!\n");
return -EINVAL;
}
if (event->cpu < 0) {
- dev_warn(ccn->dev, "Can't provide per-task data!\n");
+ dev_dbg(ccn->dev, "Can't provide per-task data!\n");
return -EOPNOTSUPP;
}
/*
@@ -771,13 +771,13 @@ static int arm_ccn_pmu_event_init(struct perf_event *event)
switch (type) {
case CCN_TYPE_MN:
if (node_xp != ccn->mn_id) {
- dev_warn(ccn->dev, "Invalid MN ID %d!\n", node_xp);
+ dev_dbg(ccn->dev, "Invalid MN ID %d!\n", node_xp);
return -EINVAL;
}
break;
case CCN_TYPE_XP:
if (node_xp >= ccn->num_xps) {
- dev_warn(ccn->dev, "Invalid XP ID %d!\n", node_xp);
+ dev_dbg(ccn->dev, "Invalid XP ID %d!\n", node_xp);
return -EINVAL;
}
break;
@@ -785,11 +785,11 @@ static int arm_ccn_pmu_event_init(struct perf_event *event)
break;
default:
if (node_xp >= ccn->num_nodes) {
- dev_warn(ccn->dev, "Invalid node ID %d!\n", node_xp);
+ dev_dbg(ccn->dev, "Invalid node ID %d!\n", node_xp);
return -EINVAL;
}
if (!arm_ccn_pmu_type_eq(type, ccn->node[node_xp].type)) {
- dev_warn(ccn->dev, "Invalid type 0x%x for node %d!\n",
+ dev_dbg(ccn->dev, "Invalid type 0x%x for node %d!\n",
type, node_xp);
return -EINVAL;
}
@@ -808,19 +808,19 @@ static int arm_ccn_pmu_event_init(struct perf_event *event)
if (event_id != e->event)
continue;
if (e->num_ports && port >= e->num_ports) {
- dev_warn(ccn->dev, "Invalid port %d for node/XP %d!\n",
+ dev_dbg(ccn->dev, "Invalid port %d for node/XP %d!\n",
port, node_xp);
return -EINVAL;
}
if (e->num_vcs && vc >= e->num_vcs) {
- dev_warn(ccn->dev, "Invalid vc %d for node/XP %d!\n",
+ dev_dbg(ccn->dev, "Invalid vc %d for node/XP %d!\n",
vc, node_xp);
return -EINVAL;
}
valid = 1;
}
if (!valid) {
- dev_warn(ccn->dev, "Invalid event 0x%x for node/XP %d!\n",
+ dev_dbg(ccn->dev, "Invalid event 0x%x for node/XP %d!\n",
event_id, node_xp);
return -EINVAL;
}
diff --git a/drivers/cdrom/cdrom.c b/drivers/cdrom/cdrom.c
index 5d475b3..ff42808 100644
--- a/drivers/cdrom/cdrom.c
+++ b/drivers/cdrom/cdrom.c
@@ -1154,9 +1154,6 @@ int cdrom_open(struct cdrom_device_info *cdi, struct block_device *bdev,
cd_dbg(CD_OPEN, "entering cdrom_open\n");
- /* open is event synchronization point, check events first */
- check_disk_change(bdev);
-
/* if this was a O_NONBLOCK open and we should honor the flags,
* do a quick open without drive/disc integrity checks. */
cdi->use_count++;
@@ -2368,7 +2365,7 @@ static int cdrom_ioctl_media_changed(struct cdrom_device_info *cdi,
if (!CDROM_CAN(CDC_SELECT_DISC) || arg == CDSL_CURRENT)
return media_changed(cdi, 1);
- if ((unsigned int)arg >= cdi->capacity)
+ if (arg >= cdi->capacity)
return -EINVAL;
info = kmalloc(sizeof(*info), GFP_KERNEL);
@@ -2438,7 +2435,7 @@ static int cdrom_ioctl_select_disc(struct cdrom_device_info *cdi,
return -ENOSYS;
if (arg != CDSL_CURRENT && arg != CDSL_NONE) {
- if ((int)arg >= cdi->capacity)
+ if (arg >= cdi->capacity)
return -EINVAL;
}
@@ -2539,7 +2536,7 @@ static int cdrom_ioctl_drive_status(struct cdrom_device_info *cdi,
if (!CDROM_CAN(CDC_SELECT_DISC) ||
(arg == CDSL_CURRENT || arg == CDSL_NONE))
return cdi->ops->drive_status(cdi, CDSL_CURRENT);
- if (((int)arg >= cdi->capacity))
+ if (arg >= cdi->capacity)
return -EINVAL;
return cdrom_slot_status(cdi, arg);
}
diff --git a/drivers/cdrom/gdrom.c b/drivers/cdrom/gdrom.c
index 584bc31..e2808fe 100644
--- a/drivers/cdrom/gdrom.c
+++ b/drivers/cdrom/gdrom.c
@@ -497,6 +497,9 @@ static struct cdrom_device_ops gdrom_ops = {
static int gdrom_bdops_open(struct block_device *bdev, fmode_t mode)
{
int ret;
+
+ check_disk_change(bdev);
+
mutex_lock(&gdrom_mutex);
ret = cdrom_open(gd.cd_info, bdev, mode);
mutex_unlock(&gdrom_mutex);
diff --git a/drivers/char/adsprpc.c b/drivers/char/adsprpc.c
index c980fb4..6ee9e97 100644
--- a/drivers/char/adsprpc.c
+++ b/drivers/char/adsprpc.c
@@ -3271,8 +3271,8 @@ static ssize_t fastrpc_debugfs_read(struct file *filp, char __user *buffer,
}
len += scnprintf(fileinfo + len, DEBUGFS_SIZE - len,
- "\n%s %s %s\n", title,
- " LIST OF INTERRUPTED SMQCONTEXTS ", title);
+ "\n%s %s %s\n", title,
+ " LIST OF INTERRUPTED SMQCONTEXTS ", title);
len += scnprintf(fileinfo + len, DEBUGFS_SIZE - len,
"%-20s|%-10s|%-10s|%-10s|%-20s\n",
diff --git a/drivers/char/diag/Makefile b/drivers/char/diag/Makefile
index 897375e..4814282 100644
--- a/drivers/char/diag/Makefile
+++ b/drivers/char/diag/Makefile
@@ -5,3 +5,7 @@
obj-$(CONFIG_MSM_MHI) += diagfwd_mhi.o
obj-$(CONFIG_DIAG_USES_SMD) += diagfwd_smd.o
diagchar-objs := diagchar_core.o diagchar_hdlc.o diagfwd.o diagfwd_glink.o diagfwd_peripheral.o diagfwd_socket.o diag_mux.o diag_memorydevice.o diag_usb.o diagmem.o diagfwd_cntl.o diag_dci.o diag_masks.o diag_debugfs.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-strlcpy-strlcat-size \
diff --git a/drivers/char/diag/diagchar_core.c b/drivers/char/diag/diagchar_core.c
index 01732be..d02863e 100644
--- a/drivers/char/diag/diagchar_core.c
+++ b/drivers/char/diag/diagchar_core.c
@@ -46,6 +46,7 @@
#ifdef CONFIG_COMPAT
#include <linux/compat.h>
#endif
+#include <linux/miscdevice.h>
MODULE_DESCRIPTION("Diag Char Driver");
MODULE_LICENSE("GPL v2");
@@ -142,6 +143,8 @@ module_param(max_clients, uint, 0000);
static struct timer_list drain_timer;
static int timer_in_progress;
+static struct timer_list wake_timer;
+
/*
* Diag Mask clear variable
* Used for clearing masks upon
@@ -210,6 +213,11 @@ static void drain_timer_func(unsigned long data)
queue_work(driver->diag_wq, &(driver->diag_drain_work));
}
+static void wake_timer_func(unsigned long data)
+{
+ pm_relax(driver->diag_dev);
+}
+
static void diag_drain_apps_data(struct diag_apps_data_t *data)
{
int err = 0;
@@ -3634,6 +3642,131 @@ static ssize_t diagchar_write(struct file *file, const char __user *buf,
return err;
}
+static ssize_t diagtest_write(struct file *file, const char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ int err = 0;
+ int pkt_type = 0;
+ int payload_len = 0;
+ const char __user *payload_buf = NULL;
+ int i;
+ unsigned char *ks_payload_buf =
+ kzalloc(sizeof(unsigned char)*count*2+1, GFP_KERNEL);
+ char temp[3];
+
+ /* Print the buffer from ks_payload_buf, for debugging. */
+ if (!ks_payload_buf) {
+ /* no memory for debug, go ahead and try to process it */
+ goto exit_dbg_print;
+ }
+
+ err = copy_from_user((&ks_payload_buf[0]), buf, count);
+ if (!err) {
+ for (i = count-1; i >= 0; i--) {
+ snprintf(temp, sizeof(temp), "%02X", ks_payload_buf[i]);
+ ks_payload_buf[i*2] = temp[0];
+ ks_payload_buf[i*2+1] = temp[1];
+ }
+
+ ks_payload_buf[count*2] = 0x00;
+ pr_debug("count:%d, %s\n", (int)count, ks_payload_buf);
+ }
+ kfree(ks_payload_buf);
+exit_dbg_print:
+
+ /*
+ * The data coming from the user space should at least have the
+ * packet type header.
+ */
+ if (count < sizeof(int)) {
+ pr_err("diag: In %s, client is sending short data, len: %d\n",
+ __func__, (int)count);
+ return -EBADMSG;
+ }
+
+ err = copy_from_user((&pkt_type), buf, sizeof(int));
+ if (err)
+ return err;
+
+ payload_buf = buf + sizeof(int);
+ payload_len = count - sizeof(int);
+
+ if (pkt_type == DCI_PKT_TYPE)
+ return diag_user_process_dci_apps_data(payload_buf,
+ payload_len, pkt_type);
+ else if (pkt_type == DCI_DATA_TYPE)
+ return diag_user_process_dci_data(payload_buf, payload_len);
+ else if (pkt_type == USER_SPACE_RAW_DATA_TYPE)
+ return (err = diag_user_process_raw_data(payload_buf,
+ payload_len)) ?
+ err : count;
+ else if (pkt_type == USER_SPACE_DATA_TYPE)
+ return diag_user_process_userspace_data(payload_buf,
+ payload_len);
+
+ if (pkt_type & (DATA_TYPE_DCI_LOG | DATA_TYPE_DCI_EVENT)) {
+ err = diag_user_process_dci_apps_data(payload_buf,
+ payload_len, pkt_type);
+ if (err)
+ return err;
+ /* Strip the DCI_LOG / DCI_EVENT bits. */
+ if (pkt_type & DATA_TYPE_DCI_LOG)
+ pkt_type ^= DATA_TYPE_DCI_LOG;
+ if (pkt_type & DATA_TYPE_DCI_EVENT)
+ pkt_type ^= DATA_TYPE_DCI_EVENT;
+ }
+
+ switch (pkt_type) {
+ case DATA_TYPE_EVENT:
+ case DATA_TYPE_F3:
+ case DATA_TYPE_LOG:
+ case DATA_TYPE_DELAYED_RESPONSE:
+ case DATA_TYPE_RESPONSE:
+ return diag_user_process_apps_data(payload_buf, payload_len,
+ pkt_type);
+ }
+
+ pr_err_ratelimited("diag: In %s, invalid pkt_type: %d\n", __func__,
+ pkt_type);
+ return -EINVAL;
+}
+
+static ssize_t diagtest_read(struct file *file, char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ pr_debug("diagtest_read count");
+ return 0;
+}
+
+static int diagtest_open(struct inode *inode, struct file *file)
+{
+ pr_debug("diagtest_open driver->logging_mask = %x\n",
+ driver->logging_mask);
+ diagfwd_mux_open(0, DIAG_USB_MODE);
+ return 0;
+}
+
+static int diagtest_release(struct inode *inode, struct file *file)
+{
+ pr_debug("diagtest_release");
+ diagfwd_mux_close(0, DIAG_USB_MODE);
+ return 0;
+}
+
+static const struct file_operations diagsmdfops = {
+ .owner = THIS_MODULE,
+ .read = diagtest_read,
+ .write = diagtest_write,
+ .open = diagtest_open,
+ .release = diagtest_release
+};
+
+struct miscdevice diagtest = {
+ .minor = MISC_DYNAMIC_MINOR,
+ .name = "diagtest",
+ .fops = &diagsmdfops,
+};
+
void diag_ws_init(void)
{
driver->dci_ws.ref_count = 0;
@@ -3667,6 +3800,8 @@ void diag_ws_on_notify(void)
* interrupts.
*/
pm_stay_awake(driver->diag_dev);
+
+ mod_timer(&wake_timer, jiffies + msecs_to_jiffies(5000));
}
void diag_ws_on_read(int type, int pkt_len)
@@ -3932,6 +4067,7 @@ static int __init diagchar_init(void)
driver->hdlc_disabled = 0;
driver->dci_state = DIAG_DCI_NO_ERROR;
setup_timer(&drain_timer, drain_timer_func, 1234);
+ setup_timer(&wake_timer, wake_timer_func, 0);
driver->supports_sockets = 1;
driver->time_sync_enabled = 0;
driver->uses_time_api = 0;
@@ -4044,6 +4180,7 @@ static int __init diagchar_init(void)
ret = diagchar_setup_cdev(dev);
if (ret)
goto fail;
+ misc_register(&diagtest);
mutex_init(&driver->diag_id_mutex);
INIT_LIST_HEAD(&driver->diag_id_list);
diag_add_diag_id_to_list(DIAG_ID_APPS, "APPS", APPS_DATA, APPS_DATA);
diff --git a/drivers/char/diag/diagfwd.c b/drivers/char/diag/diagfwd.c
index 6247503..4924667 100644
--- a/drivers/char/diag/diagfwd.c
+++ b/drivers/char/diag/diagfwd.c
@@ -1383,7 +1383,7 @@ void diag_process_hdlc_pkt(void *data, unsigned int len, int pid)
mutex_unlock(&driver->diag_hdlc_mutex);
}
-static int diagfwd_mux_open(int id, int mode)
+int diagfwd_mux_open(int id, int mode)
{
uint8_t i;
unsigned long flags;
@@ -1417,7 +1417,7 @@ static int diagfwd_mux_open(int id, int mode)
return 0;
}
-static int diagfwd_mux_close(int id, int mode)
+int diagfwd_mux_close(int id, int mode)
{
uint8_t i;
diff --git a/drivers/char/diag/diagfwd.h b/drivers/char/diag/diagfwd.h
index 687aeb7..a939c75 100644
--- a/drivers/char/diag/diagfwd.h
+++ b/drivers/char/diag/diagfwd.h
@@ -49,4 +49,6 @@ void diag_update_pkt_buffer(unsigned char *buf, uint32_t len, int type);
int diag_process_stm_cmd(unsigned char *buf, unsigned char *dest_buf);
void diag_md_hdlc_reset_timer_func(unsigned long pid);
void diag_update_md_clients(unsigned int type);
+int diagfwd_mux_open(int id, int mode);
+int diagfwd_mux_close(int id, int mode);
#endif
diff --git a/drivers/char/hw_random/stm32-rng.c b/drivers/char/hw_random/stm32-rng.c
index 63d84e6..83c69593 100644
--- a/drivers/char/hw_random/stm32-rng.c
+++ b/drivers/char/hw_random/stm32-rng.c
@@ -21,6 +21,7 @@
#include <linux/of_address.h>
#include <linux/of_platform.h>
#include <linux/pm_runtime.h>
+#include <linux/reset.h>
#include <linux/slab.h>
#define RNG_CR 0x00
@@ -46,6 +47,7 @@ struct stm32_rng_private {
struct hwrng rng;
void __iomem *base;
struct clk *clk;
+ struct reset_control *rst;
};
static int stm32_rng_read(struct hwrng *rng, void *data, size_t max, bool wait)
@@ -140,6 +142,13 @@ static int stm32_rng_probe(struct platform_device *ofdev)
if (IS_ERR(priv->clk))
return PTR_ERR(priv->clk);
+ priv->rst = devm_reset_control_get(&ofdev->dev, NULL);
+ if (!IS_ERR(priv->rst)) {
+ reset_control_assert(priv->rst);
+ udelay(2);
+ reset_control_deassert(priv->rst);
+ }
+
dev_set_drvdata(dev, priv);
priv->rng.name = dev_driver_string(dev),
diff --git a/drivers/char/ipmi/ipmi_bt_sm.c b/drivers/char/ipmi/ipmi_bt_sm.c
index feafdab..4835b58 100644
--- a/drivers/char/ipmi/ipmi_bt_sm.c
+++ b/drivers/char/ipmi/ipmi_bt_sm.c
@@ -522,11 +522,12 @@ static enum si_sm_result bt_event(struct si_sm_data *bt, long time)
if (status & BT_H_BUSY) /* clear a leftover H_BUSY */
BT_CONTROL(BT_H_BUSY);
+ bt->timeout = bt->BT_CAP_req2rsp;
+
/* Read BT capabilities if it hasn't been done yet */
if (!bt->BT_CAP_outreqs)
BT_STATE_CHANGE(BT_STATE_CAPABILITIES_BEGIN,
SI_SM_CALL_WITHOUT_DELAY);
- bt->timeout = bt->BT_CAP_req2rsp;
BT_SI_SM_RETURN(SI_SM_IDLE);
case BT_STATE_XACTION_START:
diff --git a/drivers/char/ipmi/ipmi_powernv.c b/drivers/char/ipmi/ipmi_powernv.c
index 6e658aa..a70518a 100644
--- a/drivers/char/ipmi/ipmi_powernv.c
+++ b/drivers/char/ipmi/ipmi_powernv.c
@@ -251,8 +251,9 @@ static int ipmi_powernv_probe(struct platform_device *pdev)
ipmi->irq = opal_event_request(prop);
}
- if (request_irq(ipmi->irq, ipmi_opal_event, IRQ_TYPE_LEVEL_HIGH,
- "opal-ipmi", ipmi)) {
+ rc = request_irq(ipmi->irq, ipmi_opal_event, IRQ_TYPE_LEVEL_HIGH,
+ "opal-ipmi", ipmi);
+ if (rc) {
dev_warn(dev, "Unable to request irq\n");
goto err_dispose;
}
diff --git a/drivers/char/ipmi/ipmi_ssif.c b/drivers/char/ipmi/ipmi_ssif.c
index f11c1c7..a0bb52b 100644
--- a/drivers/char/ipmi/ipmi_ssif.c
+++ b/drivers/char/ipmi/ipmi_ssif.c
@@ -617,8 +617,9 @@ static void msg_done_handler(struct ssif_info *ssif_info, int result,
flags = ipmi_ssif_lock_cond(ssif_info, &oflags);
ssif_info->waiting_alert = true;
ssif_info->rtc_us_timer = SSIF_MSG_USEC;
- mod_timer(&ssif_info->retry_timer,
- jiffies + SSIF_MSG_JIFFIES);
+ if (!ssif_info->stopping)
+ mod_timer(&ssif_info->retry_timer,
+ jiffies + SSIF_MSG_JIFFIES);
ipmi_ssif_unlock_cond(ssif_info, flags);
return;
}
@@ -761,7 +762,7 @@ static void msg_done_handler(struct ssif_info *ssif_info, int result,
ssif_info->ssif_state = SSIF_NORMAL;
ipmi_ssif_unlock_cond(ssif_info, flags);
pr_warn(PFX "Error getting flags: %d %d, %x\n",
- result, len, data[2]);
+ result, len, (len >= 3) ? data[2] : 0);
} else if (data[0] != (IPMI_NETFN_APP_REQUEST | 1) << 2
|| data[1] != IPMI_GET_MSG_FLAGS_CMD) {
/*
@@ -783,7 +784,7 @@ static void msg_done_handler(struct ssif_info *ssif_info, int result,
if ((result < 0) || (len < 3) || (data[2] != 0)) {
/* Error clearing flags */
pr_warn(PFX "Error clearing flags: %d %d, %x\n",
- result, len, data[2]);
+ result, len, (len >= 3) ? data[2] : 0);
} else if (data[0] != (IPMI_NETFN_APP_REQUEST | 1) << 2
|| data[1] != IPMI_CLEAR_MSG_FLAGS_CMD) {
pr_warn(PFX "Invalid response clearing flags: %x %x\n",
@@ -950,8 +951,9 @@ static void msg_written_handler(struct ssif_info *ssif_info, int result,
ssif_info->waiting_alert = true;
ssif_info->retries_left = SSIF_RECV_RETRIES;
ssif_info->rtc_us_timer = SSIF_MSG_PART_USEC;
- mod_timer(&ssif_info->retry_timer,
- jiffies + SSIF_MSG_PART_JIFFIES);
+ if (!ssif_info->stopping)
+ mod_timer(&ssif_info->retry_timer,
+ jiffies + SSIF_MSG_PART_JIFFIES);
ipmi_ssif_unlock_cond(ssif_info, flags);
}
}
diff --git a/drivers/char/random.c b/drivers/char/random.c
index 8a167a6..464b95a 100644
--- a/drivers/char/random.c
+++ b/drivers/char/random.c
@@ -259,6 +259,7 @@
#include <linux/kmemcheck.h>
#include <linux/workqueue.h>
#include <linux/irq.h>
+#include <linux/ratelimit.h>
#include <linux/syscalls.h>
#include <linux/completion.h>
#include <linux/uuid.h>
@@ -444,6 +445,16 @@ static void _crng_backtrack_protect(struct crng_state *crng,
__u8 tmp[CHACHA20_BLOCK_SIZE], int used);
static void process_random_ready_list(void);
+static struct ratelimit_state unseeded_warning =
+ RATELIMIT_STATE_INIT("warn_unseeded_randomness", HZ, 3);
+static struct ratelimit_state urandom_warning =
+ RATELIMIT_STATE_INIT("warn_urandom_randomness", HZ, 3);
+
+static int ratelimit_disable __read_mostly;
+
+module_param_named(ratelimit_disable, ratelimit_disable, int, 0644);
+MODULE_PARM_DESC(ratelimit_disable, "Disable random ratelimit suppression");
+
/**********************************************************************
*
* OS independent entropy store. Here are the functions which handle
@@ -819,6 +830,39 @@ static int crng_fast_load(const char *cp, size_t len)
return 1;
}
+#ifdef CONFIG_NUMA
+static void do_numa_crng_init(struct work_struct *work)
+{
+ int i;
+ struct crng_state *crng;
+ struct crng_state **pool;
+
+ pool = kcalloc(nr_node_ids, sizeof(*pool), GFP_KERNEL|__GFP_NOFAIL);
+ for_each_online_node(i) {
+ crng = kmalloc_node(sizeof(struct crng_state),
+ GFP_KERNEL | __GFP_NOFAIL, i);
+ spin_lock_init(&crng->lock);
+ crng_initialize(crng);
+ pool[i] = crng;
+ }
+ mb();
+ if (cmpxchg(&crng_node_pool, NULL, pool)) {
+ for_each_node(i)
+ kfree(pool[i]);
+ kfree(pool);
+ }
+}
+
+static DECLARE_WORK(numa_crng_init_work, do_numa_crng_init);
+
+static void numa_crng_init(void)
+{
+ schedule_work(&numa_crng_init_work);
+}
+#else
+static void numa_crng_init(void) {}
+#endif
+
static void crng_reseed(struct crng_state *crng, struct entropy_store *r)
{
unsigned long flags;
@@ -848,10 +892,23 @@ static void crng_reseed(struct crng_state *crng, struct entropy_store *r)
memzero_explicit(&buf, sizeof(buf));
crng->init_time = jiffies;
if (crng == &primary_crng && crng_init < 2) {
+ numa_crng_init();
crng_init = 2;
process_random_ready_list();
wake_up_interruptible(&crng_init_wait);
pr_notice("random: crng init done\n");
+ if (unseeded_warning.missed) {
+ pr_notice("random: %d get_random_xx warning(s) missed "
+ "due to ratelimiting\n",
+ unseeded_warning.missed);
+ unseeded_warning.missed = 0;
+ }
+ if (urandom_warning.missed) {
+ pr_notice("random: %d urandom warning(s) missed "
+ "due to ratelimiting\n",
+ urandom_warning.missed);
+ urandom_warning.missed = 0;
+ }
}
spin_unlock_irqrestore(&crng->lock, flags);
}
@@ -1661,29 +1718,14 @@ static void init_std_data(struct entropy_store *r)
*/
static int rand_initialize(void)
{
-#ifdef CONFIG_NUMA
- int i;
- struct crng_state *crng;
- struct crng_state **pool;
-#endif
-
init_std_data(&input_pool);
init_std_data(&blocking_pool);
crng_initialize(&primary_crng);
crng_global_init_time = jiffies;
-
-#ifdef CONFIG_NUMA
- pool = kcalloc(nr_node_ids, sizeof(*pool), GFP_KERNEL|__GFP_NOFAIL);
- for_each_online_node(i) {
- crng = kmalloc_node(sizeof(struct crng_state),
- GFP_KERNEL | __GFP_NOFAIL, i);
- spin_lock_init(&crng->lock);
- crng_initialize(crng);
- pool[i] = crng;
+ if (ratelimit_disable) {
+ urandom_warning.interval = 0;
+ unseeded_warning.interval = 0;
}
- mb();
- crng_node_pool = pool;
-#endif
return 0;
}
early_initcall(rand_initialize);
@@ -1751,9 +1793,10 @@ urandom_read(struct file *file, char __user *buf, size_t nbytes, loff_t *ppos)
if (!crng_ready() && maxwarn > 0) {
maxwarn--;
- printk(KERN_NOTICE "random: %s: uninitialized urandom read "
- "(%zd bytes read)\n",
- current->comm, nbytes);
+ if (__ratelimit(&urandom_warning))
+ printk(KERN_NOTICE "random: %s: uninitialized "
+ "urandom read (%zd bytes read)\n",
+ current->comm, nbytes);
spin_lock_irqsave(&primary_crng.lock, flags);
crng_init_cnt = 0;
spin_unlock_irqrestore(&primary_crng.lock, flags);
@@ -1783,14 +1826,22 @@ static int
write_pool(struct entropy_store *r, const char __user *buffer, size_t count)
{
size_t bytes;
- __u32 buf[16];
+ __u32 t, buf[16];
const char __user *p = buffer;
while (count > 0) {
+ int b, i = 0;
+
bytes = min(count, sizeof(buf));
if (copy_from_user(&buf, p, bytes))
return -EFAULT;
+ for (b = bytes ; b > 0 ; b -= sizeof(__u32), i++) {
+ if (!arch_get_random_int(&t))
+ break;
+ buf[i] ^= t;
+ }
+
count -= bytes;
p += bytes;
diff --git a/drivers/char/tpm/tpm-chip.c b/drivers/char/tpm/tpm-chip.c
index 9ff8532..8d097d1 100644
--- a/drivers/char/tpm/tpm-chip.c
+++ b/drivers/char/tpm/tpm-chip.c
@@ -26,6 +26,7 @@
#include <linux/spinlock.h>
#include <linux/freezer.h>
#include <linux/major.h>
+#include <linux/of.h>
#include "tpm.h"
#include "tpm_eventlog.h"
@@ -388,8 +389,20 @@ static int tpm_add_legacy_sysfs(struct tpm_chip *chip)
*/
int tpm_chip_register(struct tpm_chip *chip)
{
+#ifdef CONFIG_OF
+ struct device_node *np;
+#endif
int rc;
+#ifdef CONFIG_OF
+ np = of_find_node_by_name(NULL, "vtpm");
+ if (np) {
+ if (of_property_read_bool(np, "powered-while-suspended"))
+ chip->flags |= TPM_CHIP_FLAG_ALWAYS_POWERED;
+ }
+ of_node_put(np);
+#endif
+
if (chip->ops->flags & TPM_OPS_AUTO_STARTUP) {
if (chip->flags & TPM_CHIP_FLAG_TPM2)
rc = tpm2_auto_startup(chip);
diff --git a/drivers/char/tpm/tpm-dev.c b/drivers/char/tpm/tpm-dev.c
index 65b8249..1662e46 100644
--- a/drivers/char/tpm/tpm-dev.c
+++ b/drivers/char/tpm/tpm-dev.c
@@ -25,7 +25,7 @@ struct file_priv {
struct tpm_chip *chip;
/* Data passed to and from the tpm via the read/write calls */
- atomic_t data_pending;
+ size_t data_pending;
struct mutex buffer_mutex;
struct timer_list user_read_timer; /* user needs to claim result */
@@ -46,7 +46,7 @@ static void timeout_work(struct work_struct *work)
struct file_priv *priv = container_of(work, struct file_priv, work);
mutex_lock(&priv->buffer_mutex);
- atomic_set(&priv->data_pending, 0);
+ priv->data_pending = 0;
memset(priv->data_buffer, 0, sizeof(priv->data_buffer));
mutex_unlock(&priv->buffer_mutex);
}
@@ -72,7 +72,6 @@ static int tpm_open(struct inode *inode, struct file *file)
}
priv->chip = chip;
- atomic_set(&priv->data_pending, 0);
mutex_init(&priv->buffer_mutex);
setup_timer(&priv->user_read_timer, user_reader_timeout,
(unsigned long)priv);
@@ -86,28 +85,24 @@ static ssize_t tpm_read(struct file *file, char __user *buf,
size_t size, loff_t *off)
{
struct file_priv *priv = file->private_data;
- ssize_t ret_size;
+ ssize_t ret_size = 0;
int rc;
del_singleshot_timer_sync(&priv->user_read_timer);
flush_work(&priv->work);
- ret_size = atomic_read(&priv->data_pending);
- if (ret_size > 0) { /* relay data */
- ssize_t orig_ret_size = ret_size;
- if (size < ret_size)
- ret_size = size;
+ mutex_lock(&priv->buffer_mutex);
- mutex_lock(&priv->buffer_mutex);
+ if (priv->data_pending) {
+ ret_size = min_t(ssize_t, size, priv->data_pending);
rc = copy_to_user(buf, priv->data_buffer, ret_size);
- memset(priv->data_buffer, 0, orig_ret_size);
+ memset(priv->data_buffer, 0, priv->data_pending);
if (rc)
ret_size = -EFAULT;
- mutex_unlock(&priv->buffer_mutex);
+ priv->data_pending = 0;
}
- atomic_set(&priv->data_pending, 0);
-
+ mutex_unlock(&priv->buffer_mutex);
return ret_size;
}
@@ -118,18 +113,20 @@ static ssize_t tpm_write(struct file *file, const char __user *buf,
size_t in_size = size;
ssize_t out_size;
- /* cannot perform a write until the read has cleared
- either via tpm_read or a user_read_timer timeout.
- This also prevents splitted buffered writes from blocking here.
- */
- if (atomic_read(&priv->data_pending) != 0)
- return -EBUSY;
-
if (in_size > TPM_BUFSIZE)
return -E2BIG;
mutex_lock(&priv->buffer_mutex);
+ /* Cannot perform a write until the read has cleared either via
+ * tpm_read or a user_read_timer timeout. This also prevents split
+ * buffered writes from blocking here.
+ */
+ if (priv->data_pending != 0) {
+ mutex_unlock(&priv->buffer_mutex);
+ return -EBUSY;
+ }
+
if (copy_from_user
(priv->data_buffer, (void __user *) buf, in_size)) {
mutex_unlock(&priv->buffer_mutex);
@@ -159,7 +156,7 @@ static ssize_t tpm_write(struct file *file, const char __user *buf,
return out_size;
}
- atomic_set(&priv->data_pending, out_size);
+ priv->data_pending = out_size;
mutex_unlock(&priv->buffer_mutex);
/* Set a timeout by which the reader must come claim the result */
@@ -178,7 +175,7 @@ static int tpm_release(struct inode *inode, struct file *file)
del_singleshot_timer_sync(&priv->user_read_timer);
flush_work(&priv->work);
file->private_data = NULL;
- atomic_set(&priv->data_pending, 0);
+ priv->data_pending = 0;
clear_bit(0, &priv->chip->is_open);
kfree(priv);
return 0;
diff --git a/drivers/char/tpm/tpm-interface.c b/drivers/char/tpm/tpm-interface.c
index 830d7e3..4f0b194 100644
--- a/drivers/char/tpm/tpm-interface.c
+++ b/drivers/char/tpm/tpm-interface.c
@@ -420,7 +420,8 @@ ssize_t tpm_transmit_cmd(struct tpm_chip *chip, const void *cmd,
header = cmd;
err = be32_to_cpu(header->return_code);
- if (err != 0 && desc)
+ if (err != 0 && err != TPM_ERR_DISABLED && err != TPM_ERR_DEACTIVATED
+ && desc)
dev_err(&chip->dev, "A TPM error (%d) occurred %s\n", err,
desc);
@@ -803,6 +804,10 @@ int tpm_do_selftest(struct tpm_chip *chip)
loops = jiffies_to_msecs(duration) / delay_msec;
rc = tpm_continue_selftest(chip);
+ if (rc == TPM_ERR_INVALID_POSTINIT) {
+ chip->flags |= TPM_CHIP_FLAG_ALWAYS_POWERED;
+ dev_info(&chip->dev, "TPM not ready (%d)\n", rc);
+ }
/* This may fail if there was no TPM driver during a suspend/resume
* cycle; some may return 10 (BAD_ORDINAL), others 28 (FAILEDSELFTEST)
*/
@@ -969,6 +974,9 @@ int tpm_pm_suspend(struct device *dev)
if (chip == NULL)
return -ENODEV;
+ if (chip->flags & TPM_CHIP_FLAG_ALWAYS_POWERED)
+ return 0;
+
if (chip->flags & TPM_CHIP_FLAG_TPM2) {
tpm2_shutdown(chip, TPM2_SU_STATE);
return 0;
diff --git a/drivers/char/tpm/tpm.h b/drivers/char/tpm/tpm.h
index aa4299c..a4fc2ba 100644
--- a/drivers/char/tpm/tpm.h
+++ b/drivers/char/tpm/tpm.h
@@ -143,6 +143,7 @@ enum tpm_chip_flags {
TPM_CHIP_FLAG_TPM2 = BIT(1),
TPM_CHIP_FLAG_IRQ = BIT(2),
TPM_CHIP_FLAG_VIRTUAL = BIT(3),
+ TPM_CHIP_FLAG_ALWAYS_POWERED = BIT(5),
};
struct tpm_chip {
diff --git a/drivers/char/tpm/tpm_i2c_infineon.c b/drivers/char/tpm/tpm_i2c_infineon.c
index da69dde..b15479a 100644
--- a/drivers/char/tpm/tpm_i2c_infineon.c
+++ b/drivers/char/tpm/tpm_i2c_infineon.c
@@ -115,7 +115,7 @@ static int iic_tpm_read(u8 addr, u8 *buffer, size_t len)
/* Lock the adapter for the duration of the whole sequence. */
if (!tpm_dev.client->adapter->algo->master_xfer)
return -EOPNOTSUPP;
- i2c_lock_adapter(tpm_dev.client->adapter);
+ i2c_lock_bus(tpm_dev.client->adapter, I2C_LOCK_SEGMENT);
if (tpm_dev.chip_type == SLB9645) {
/* use a combined read for newer chips
@@ -156,7 +156,7 @@ static int iic_tpm_read(u8 addr, u8 *buffer, size_t len)
}
out:
- i2c_unlock_adapter(tpm_dev.client->adapter);
+ i2c_unlock_bus(tpm_dev.client->adapter, I2C_LOCK_SEGMENT);
/* take care of 'guard time' */
usleep_range(SLEEP_DURATION_LOW, SLEEP_DURATION_HI);
@@ -188,7 +188,7 @@ static int iic_tpm_write_generic(u8 addr, u8 *buffer, size_t len,
if (!tpm_dev.client->adapter->algo->master_xfer)
return -EOPNOTSUPP;
- i2c_lock_adapter(tpm_dev.client->adapter);
+ i2c_lock_bus(tpm_dev.client->adapter, I2C_LOCK_SEGMENT);
/* prepend the 'register address' to the buffer */
tpm_dev.buf[0] = addr;
@@ -207,7 +207,7 @@ static int iic_tpm_write_generic(u8 addr, u8 *buffer, size_t len,
usleep_range(sleep_low, sleep_hi);
}
- i2c_unlock_adapter(tpm_dev.client->adapter);
+ i2c_unlock_bus(tpm_dev.client->adapter, I2C_LOCK_SEGMENT);
/* take care of 'guard time' */
usleep_range(SLEEP_DURATION_LOW, SLEEP_DURATION_HI);
diff --git a/drivers/char/tpm/tpm_i2c_nuvoton.c b/drivers/char/tpm/tpm_i2c_nuvoton.c
index caa86b1..f74f451 100644
--- a/drivers/char/tpm/tpm_i2c_nuvoton.c
+++ b/drivers/char/tpm/tpm_i2c_nuvoton.c
@@ -369,6 +369,7 @@ static int i2c_nuvoton_send(struct tpm_chip *chip, u8 *buf, size_t len)
struct device *dev = chip->dev.parent;
struct i2c_client *client = to_i2c_client(dev);
u32 ordinal;
+ unsigned long duration;
size_t count = 0;
int burst_count, bytes2write, retries, rc = -EIO;
@@ -455,10 +456,12 @@ static int i2c_nuvoton_send(struct tpm_chip *chip, u8 *buf, size_t len)
return rc;
}
ordinal = be32_to_cpu(*((__be32 *) (buf + 6)));
- rc = i2c_nuvoton_wait_for_data_avail(chip,
- tpm_calc_ordinal_duration(chip,
- ordinal),
- &priv->read_queue);
+ if (chip->flags & TPM_CHIP_FLAG_TPM2)
+ duration = tpm2_calc_ordinal_duration(chip, ordinal);
+ else
+ duration = tpm_calc_ordinal_duration(chip, ordinal);
+
+ rc = i2c_nuvoton_wait_for_data_avail(chip, duration, &priv->read_queue);
if (rc) {
dev_err(dev, "%s() timeout command duration\n", __func__);
i2c_nuvoton_ready(chip);
diff --git a/drivers/char/tpm/tpm_tis_spi.c b/drivers/char/tpm/tpm_tis_spi.c
index 01eccb1..950c2d2 100644
--- a/drivers/char/tpm/tpm_tis_spi.c
+++ b/drivers/char/tpm/tpm_tis_spi.c
@@ -189,6 +189,7 @@ static const struct tpm_tis_phy_ops tpm_spi_phy_ops = {
static int tpm_tis_spi_probe(struct spi_device *dev)
{
struct tpm_tis_spi_phy *phy;
+ int irq;
phy = devm_kzalloc(&dev->dev, sizeof(struct tpm_tis_spi_phy),
GFP_KERNEL);
@@ -201,7 +202,13 @@ static int tpm_tis_spi_probe(struct spi_device *dev)
if (!phy->iobuf)
return -ENOMEM;
- return tpm_tis_core_init(&dev->dev, &phy->priv, -1, &tpm_spi_phy_ops,
+ /* If the SPI device has an IRQ then use that */
+ if (dev->irq > 0)
+ irq = dev->irq;
+ else
+ irq = -1;
+
+ return tpm_tis_core_init(&dev->dev, &phy->priv, irq, &tpm_spi_phy_ops,
NULL);
}
diff --git a/drivers/char/tpm/xen-tpmfront.c b/drivers/char/tpm/xen-tpmfront.c
index a2ab008..97b3e31 100644
--- a/drivers/char/tpm/xen-tpmfront.c
+++ b/drivers/char/tpm/xen-tpmfront.c
@@ -203,7 +203,7 @@ static int setup_ring(struct xenbus_device *dev, struct tpm_private *priv)
return -ENOMEM;
}
- rv = xenbus_grant_ring(dev, &priv->shr, 1, &gref);
+ rv = xenbus_grant_ring(dev, priv->shr, 1, &gref);
if (rv < 0)
return rv;
diff --git a/drivers/char/virtio_console.c b/drivers/char/virtio_console.c
index 8f890c1..8c0017d 100644
--- a/drivers/char/virtio_console.c
+++ b/drivers/char/virtio_console.c
@@ -1405,7 +1405,6 @@ static int add_port(struct ports_device *portdev, u32 id)
{
char debugfs_name[16];
struct port *port;
- struct port_buffer *buf;
dev_t devt;
unsigned int nr_added_bufs;
int err;
@@ -1516,8 +1515,6 @@ static int add_port(struct ports_device *portdev, u32 id)
return 0;
free_inbufs:
- while ((buf = virtqueue_detach_unused_buf(port->in_vq)))
- free_buf(buf, true);
free_device:
device_destroy(pdrvdata.class, port->dev->devt);
free_cdev:
@@ -1542,34 +1539,14 @@ static void remove_port(struct kref *kref)
static void remove_port_data(struct port *port)
{
- struct port_buffer *buf;
-
spin_lock_irq(&port->inbuf_lock);
/* Remove unused data this port might have received. */
discard_port_data(port);
spin_unlock_irq(&port->inbuf_lock);
- /* Remove buffers we queued up for the Host to send us data in. */
- do {
- spin_lock_irq(&port->inbuf_lock);
- buf = virtqueue_detach_unused_buf(port->in_vq);
- spin_unlock_irq(&port->inbuf_lock);
- if (buf)
- free_buf(buf, true);
- } while (buf);
-
spin_lock_irq(&port->outvq_lock);
reclaim_consumed_buffers(port);
spin_unlock_irq(&port->outvq_lock);
-
- /* Free pending buffers from the out-queue. */
- do {
- spin_lock_irq(&port->outvq_lock);
- buf = virtqueue_detach_unused_buf(port->out_vq);
- spin_unlock_irq(&port->outvq_lock);
- if (buf)
- free_buf(buf, true);
- } while (buf);
}
/*
@@ -1794,13 +1771,24 @@ static void control_work_handler(struct work_struct *work)
spin_unlock(&portdev->c_ivq_lock);
}
+static void flush_bufs(struct virtqueue *vq, bool can_sleep)
+{
+ struct port_buffer *buf;
+ unsigned int len;
+
+ while ((buf = virtqueue_get_buf(vq, &len)))
+ free_buf(buf, can_sleep);
+}
+
static void out_intr(struct virtqueue *vq)
{
struct port *port;
port = find_port_by_vq(vq->vdev->priv, vq);
- if (!port)
+ if (!port) {
+ flush_bufs(vq, false);
return;
+ }
wake_up_interruptible(&port->waitqueue);
}
@@ -1811,8 +1799,10 @@ static void in_intr(struct virtqueue *vq)
unsigned long flags;
port = find_port_by_vq(vq->vdev->priv, vq);
- if (!port)
+ if (!port) {
+ flush_bufs(vq, false);
return;
+ }
spin_lock_irqsave(&port->inbuf_lock, flags);
port->inbuf = get_inbuf(port);
@@ -1987,6 +1977,15 @@ static const struct file_operations portdev_fops = {
static void remove_vqs(struct ports_device *portdev)
{
+ struct virtqueue *vq;
+
+ virtio_device_for_each_vq(portdev->vdev, vq) {
+ struct port_buffer *buf;
+
+ flush_bufs(vq, true);
+ while ((buf = virtqueue_detach_unused_buf(vq)))
+ free_buf(buf, true);
+ }
portdev->vdev->config->del_vqs(portdev->vdev);
kfree(portdev->in_vqs);
kfree(portdev->out_vqs);
diff --git a/drivers/clk/at91/clk-pll.c b/drivers/clk/at91/clk-pll.c
index 45ad168..a17a428 100644
--- a/drivers/clk/at91/clk-pll.c
+++ b/drivers/clk/at91/clk-pll.c
@@ -132,19 +132,11 @@ static unsigned long clk_pll_recalc_rate(struct clk_hw *hw,
unsigned long parent_rate)
{
struct clk_pll *pll = to_clk_pll(hw);
- unsigned int pllr;
- u16 mul;
- u8 div;
- regmap_read(pll->regmap, PLL_REG(pll->id), &pllr);
-
- div = PLL_DIV(pllr);
- mul = PLL_MUL(pllr, pll->layout);
-
- if (!div || !mul)
+ if (!pll->div || !pll->mul)
return 0;
- return (parent_rate / div) * (mul + 1);
+ return (parent_rate / pll->div) * (pll->mul + 1);
}
static long clk_pll_get_best_div_mul(struct clk_pll *pll, unsigned long rate,
diff --git a/drivers/clk/clk-fixed-factor.c b/drivers/clk/clk-fixed-factor.c
index fda42ab..88a9d94 100644
--- a/drivers/clk/clk-fixed-factor.c
+++ b/drivers/clk/clk-fixed-factor.c
@@ -177,8 +177,15 @@ static struct clk *_of_fixed_factor_clk_setup(struct device_node *node)
clk = clk_register_fixed_factor(NULL, clk_name, parent_name, flags,
mult, div);
- if (IS_ERR(clk))
+ if (IS_ERR(clk)) {
+ /*
+ * If parent clock is not registered, registration would fail.
+ * Clear OF_POPULATED flag so that clock registration can be
+ * attempted again from probe function.
+ */
+ of_node_clear_flag(node, OF_POPULATED);
return clk;
+ }
ret = of_clk_add_provider(node, of_clk_src_simple_get, clk);
if (ret) {
@@ -203,6 +210,7 @@ static int of_fixed_factor_clk_remove(struct platform_device *pdev)
{
struct clk *clk = platform_get_drvdata(pdev);
+ of_clk_del_provider(pdev->dev.of_node);
clk_unregister_fixed_factor(clk);
return 0;
diff --git a/drivers/clk/clk-fixed-rate.c b/drivers/clk/clk-fixed-rate.c
index 9933e1c..d176d2d 100644
--- a/drivers/clk/clk-fixed-rate.c
+++ b/drivers/clk/clk-fixed-rate.c
@@ -200,6 +200,7 @@ static int of_fixed_clk_remove(struct platform_device *pdev)
{
struct clk *clk = platform_get_drvdata(pdev);
+ of_clk_del_provider(pdev->dev.of_node);
clk_unregister_fixed_rate(clk);
return 0;
diff --git a/drivers/clk/clk-s2mps11.c b/drivers/clk/clk-s2mps11.c
index fbaa84a..14071a5 100644
--- a/drivers/clk/clk-s2mps11.c
+++ b/drivers/clk/clk-s2mps11.c
@@ -245,6 +245,36 @@ static const struct platform_device_id s2mps11_clk_id[] = {
};
MODULE_DEVICE_TABLE(platform, s2mps11_clk_id);
+#ifdef CONFIG_OF
+/*
+ * Device is instantiated through parent MFD device and device matching is done
+ * through platform_device_id.
+ *
+ * However if device's DT node contains proper clock compatible and driver is
+ * built as a module, then the *module* matching will be done trough DT aliases.
+ * This requires of_device_id table. In the same time this will not change the
+ * actual *device* matching so do not add .of_match_table.
+ */
+static const struct of_device_id s2mps11_dt_match[] = {
+ {
+ .compatible = "samsung,s2mps11-clk",
+ .data = (void *)S2MPS11X,
+ }, {
+ .compatible = "samsung,s2mps13-clk",
+ .data = (void *)S2MPS13X,
+ }, {
+ .compatible = "samsung,s2mps14-clk",
+ .data = (void *)S2MPS14X,
+ }, {
+ .compatible = "samsung,s5m8767-clk",
+ .data = (void *)S5M8767X,
+ }, {
+ /* Sentinel */
+ },
+};
+MODULE_DEVICE_TABLE(of, s2mps11_dt_match);
+#endif
+
static struct platform_driver s2mps11_clk_driver = {
.driver = {
.name = "s2mps11-clk",
diff --git a/drivers/clk/clk.c b/drivers/clk/clk.c
index f41307d..959922a 100644
--- a/drivers/clk/clk.c
+++ b/drivers/clk/clk.c
@@ -1144,7 +1144,7 @@ static int clk_core_round_rate_nolock(struct clk_core *core,
struct clk_core *parent;
long rate;
- lockdep_assert_held(&prepare_lock);
+ // lockdep_assert_held(&prepare_lock);
if (!core)
return 0;
@@ -2280,6 +2280,9 @@ static int clk_core_get_phase(struct clk_core *core)
int ret;
clk_prepare_lock();
+ /* Always try to update cached phase if possible */
+ if (core->ops->get_phase)
+ core->phase = core->ops->get_phase(core->hw);
ret = core->phase;
clk_prepare_unlock();
diff --git a/drivers/clk/hisilicon/reset.c b/drivers/clk/hisilicon/reset.c
index 2a5015c..43e82fa 100644
--- a/drivers/clk/hisilicon/reset.c
+++ b/drivers/clk/hisilicon/reset.c
@@ -109,9 +109,8 @@ struct hisi_reset_controller *hisi_reset_init(struct platform_device *pdev)
return NULL;
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- rstc->membase = devm_ioremap(&pdev->dev,
- res->start, resource_size(res));
- if (!rstc->membase)
+ rstc->membase = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(rstc->membase))
return NULL;
spin_lock_init(&rstc->lock);
diff --git a/drivers/clk/imx/clk-imx6ul.c b/drivers/clk/imx/clk-imx6ul.c
index d1d7787..db29016 100644
--- a/drivers/clk/imx/clk-imx6ul.c
+++ b/drivers/clk/imx/clk-imx6ul.c
@@ -120,6 +120,7 @@ static void __init imx6ul_clocks_init(struct device_node *ccm_node)
np = of_find_compatible_node(NULL, NULL, "fsl,imx6ul-anatop");
base = of_iomap(np, 0);
+ of_node_put(np);
WARN_ON(!base);
clks[IMX6UL_PLL1_BYPASS_SRC] = imx_clk_mux("pll1_bypass_src", base + 0x00, 14, 1, pll_bypass_src_sels, ARRAY_SIZE(pll_bypass_src_sels));
diff --git a/drivers/clk/mmp/clk.c b/drivers/clk/mmp/clk.c
index 61893fe..18b6c9b 100644
--- a/drivers/clk/mmp/clk.c
+++ b/drivers/clk/mmp/clk.c
@@ -182,7 +182,7 @@ void mmp_clk_add(struct mmp_clk_unit *unit, unsigned int id,
pr_err("CLK %d has invalid pointer %p\n", id, clk);
return;
}
- if (id > unit->nr_clks) {
+ if (id >= unit->nr_clks) {
pr_err("CLK %d is invalid\n", id);
return;
}
diff --git a/drivers/clk/mvebu/cp110-system-controller.c b/drivers/clk/mvebu/cp110-system-controller.c
index f2303da..465953c 100644
--- a/drivers/clk/mvebu/cp110-system-controller.c
+++ b/drivers/clk/mvebu/cp110-system-controller.c
@@ -172,11 +172,11 @@ static struct clk *cp110_of_clk_get(struct of_phandle_args *clkspec, void *data)
unsigned int idx = clkspec->args[1];
if (type == CP110_CLK_TYPE_CORE) {
- if (idx > CP110_MAX_CORE_CLOCKS)
+ if (idx >= CP110_MAX_CORE_CLOCKS)
return ERR_PTR(-EINVAL);
return clk_data->clks[idx];
} else if (type == CP110_CLK_TYPE_GATABLE) {
- if (idx > CP110_MAX_GATABLE_CLOCKS)
+ if (idx >= CP110_MAX_GATABLE_CLOCKS)
return ERR_PTR(-EINVAL);
return clk_data->clks[CP110_MAX_CORE_CLOCKS + idx];
}
diff --git a/drivers/clk/qcom/clk-debug.c b/drivers/clk/qcom/clk-debug.c
index d101536..3bd085e 100644
--- a/drivers/clk/qcom/clk-debug.c
+++ b/drivers/clk/qcom/clk-debug.c
@@ -251,12 +251,16 @@ static int clk_debug_measure_get(void *data, u64 *val)
meas_rate = clk_get_rate(hw->clk);
par = clk_hw_get_parent(measure);
- if (!par)
- return -EINVAL;
+ if (!par) {
+ ret = -EINVAL;
+ goto out;
+ }
sw_rate = clk_get_rate(par->clk);
if (sw_rate && meas_rate >= (sw_rate * 2))
*val *= DIV_ROUND_CLOSEST(meas_rate, sw_rate);
+
+out:
mutex_unlock(&clk_debug_lock);
return ret;
diff --git a/drivers/clk/qcom/gcc-sdm845.c b/drivers/clk/qcom/gcc-sdm845.c
index 555b8bd..f26bd27 100644
--- a/drivers/clk/qcom/gcc-sdm845.c
+++ b/drivers/clk/qcom/gcc-sdm845.c
@@ -2188,7 +2188,7 @@ static struct clk_branch gcc_pcie_0_mstr_axi_clk = {
static struct clk_branch gcc_pcie_0_pipe_clk = {
.halt_reg = 0x6b020,
- .halt_check = BRANCH_VOTED,
+ .halt_check = BRANCH_HALT_DELAY,
.clkr = {
.enable_reg = 0x5200c,
.enable_mask = BIT(4),
@@ -2288,7 +2288,7 @@ static struct clk_branch gcc_pcie_1_mstr_axi_clk = {
static struct clk_branch gcc_pcie_1_pipe_clk = {
.halt_reg = 0x8d020,
- .halt_check = BRANCH_VOTED,
+ .halt_check = BRANCH_HALT_DELAY,
.clkr = {
.enable_reg = 0x52004,
.enable_mask = BIT(30),
diff --git a/drivers/clk/qcom/mdss/Makefile b/drivers/clk/qcom/mdss/Makefile
index 87feee6..26de919 100644
--- a/drivers/clk/qcom/mdss/Makefile
+++ b/drivers/clk/qcom/mdss/Makefile
@@ -4,3 +4,6 @@
obj-$(CONFIG_QCOM_MDSS_PLL) += mdss-dp-pll-10nm.o
obj-$(CONFIG_QCOM_MDSS_PLL) += mdss-dp-pll-10nm-util.o
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-unused-function \
diff --git a/drivers/clk/renesas/renesas-cpg-mssr.c b/drivers/clk/renesas/renesas-cpg-mssr.c
index 25c41cd..7ecc5ea 100644
--- a/drivers/clk/renesas/renesas-cpg-mssr.c
+++ b/drivers/clk/renesas/renesas-cpg-mssr.c
@@ -243,8 +243,9 @@ struct clk *cpg_mssr_clk_src_twocell_get(struct of_phandle_args *clkspec,
dev_err(dev, "Cannot get %s clock %u: %ld", type, clkidx,
PTR_ERR(clk));
else
- dev_dbg(dev, "clock (%u, %u) is %pC at %pCr Hz\n",
- clkspec->args[0], clkspec->args[1], clk, clk);
+ dev_dbg(dev, "clock (%u, %u) is %pC at %lu Hz\n",
+ clkspec->args[0], clkspec->args[1], clk,
+ clk_get_rate(clk));
return clk;
}
@@ -304,7 +305,7 @@ static void __init cpg_mssr_register_core_clk(const struct cpg_core_clk *core,
if (IS_ERR_OR_NULL(clk))
goto fail;
- dev_dbg(dev, "Core clock %pC at %pCr Hz\n", clk, clk);
+ dev_dbg(dev, "Core clock %pC at %lu Hz\n", clk, clk_get_rate(clk));
priv->clks[id] = clk;
return;
@@ -372,7 +373,7 @@ static void __init cpg_mssr_register_mod_clk(const struct mssr_mod_clk *mod,
if (IS_ERR(clk))
goto fail;
- dev_dbg(dev, "Module clock %pC at %pCr Hz\n", clk, clk);
+ dev_dbg(dev, "Module clock %pC at %lu Hz\n", clk, clk_get_rate(clk));
priv->clks[id] = clk;
return;
diff --git a/drivers/clk/rockchip/clk-ddr.c b/drivers/clk/rockchip/clk-ddr.c
index e8075359..ebce526 100644
--- a/drivers/clk/rockchip/clk-ddr.c
+++ b/drivers/clk/rockchip/clk-ddr.c
@@ -80,16 +80,12 @@ static long rockchip_ddrclk_sip_round_rate(struct clk_hw *hw,
static u8 rockchip_ddrclk_get_parent(struct clk_hw *hw)
{
struct rockchip_ddrclk *ddrclk = to_rockchip_ddrclk_hw(hw);
- int num_parents = clk_hw_get_num_parents(hw);
u32 val;
val = clk_readl(ddrclk->reg_base +
ddrclk->mux_offset) >> ddrclk->mux_shift;
val &= GENMASK(ddrclk->mux_width - 1, 0);
- if (val >= num_parents)
- return -EINVAL;
-
return val;
}
diff --git a/drivers/clk/rockchip/clk-mmc-phase.c b/drivers/clk/rockchip/clk-mmc-phase.c
index 077fcdc..fe7d9ed 100644
--- a/drivers/clk/rockchip/clk-mmc-phase.c
+++ b/drivers/clk/rockchip/clk-mmc-phase.c
@@ -58,6 +58,12 @@ static int rockchip_mmc_get_phase(struct clk_hw *hw)
u16 degrees;
u32 delay_num = 0;
+ /* See the comment for rockchip_mmc_set_phase below */
+ if (!rate) {
+ pr_err("%s: invalid clk rate\n", __func__);
+ return -EINVAL;
+ }
+
raw_value = readl(mmc_clock->reg) >> (mmc_clock->shift);
degrees = (raw_value & ROCKCHIP_MMC_DEGREE_MASK) * 90;
@@ -84,6 +90,23 @@ static int rockchip_mmc_set_phase(struct clk_hw *hw, int degrees)
u32 raw_value;
u32 delay;
+ /*
+ * The below calculation is based on the output clock from
+ * MMC host to the card, which expects the phase clock inherits
+ * the clock rate from its parent, namely the output clock
+ * provider of MMC host. However, things may go wrong if
+ * (1) It is orphan.
+ * (2) It is assigned to the wrong parent.
+ *
+ * This check help debug the case (1), which seems to be the
+ * most likely problem we often face and which makes it difficult
+ * for people to debug unstable mmc tuning results.
+ */
+ if (!rate) {
+ pr_err("%s: invalid clk rate\n", __func__);
+ return -EINVAL;
+ }
+
nineties = degrees / 90;
remainder = (degrees % 90);
diff --git a/drivers/clk/rockchip/clk-rk3188.c b/drivers/clk/rockchip/clk-rk3188.c
index d0e722a..523378d 100644
--- a/drivers/clk/rockchip/clk-rk3188.c
+++ b/drivers/clk/rockchip/clk-rk3188.c
@@ -381,7 +381,7 @@ static struct rockchip_clk_branch common_clk_branches[] __initdata = {
COMPOSITE_NOMUX(0, "spdif_pre", "i2s_src", 0,
RK2928_CLKSEL_CON(5), 0, 7, DFLAGS,
RK2928_CLKGATE_CON(0), 13, GFLAGS),
- COMPOSITE_FRACMUX(0, "spdif_frac", "spdif_pll", CLK_SET_RATE_PARENT,
+ COMPOSITE_FRACMUX(0, "spdif_frac", "spdif_pre", CLK_SET_RATE_PARENT,
RK2928_CLKSEL_CON(9), 0,
RK2928_CLKGATE_CON(0), 14, GFLAGS,
&common_spdif_fracmux),
diff --git a/drivers/clk/rockchip/clk-rk3228.c b/drivers/clk/rockchip/clk-rk3228.c
index db6e5a9..53f16ef 100644
--- a/drivers/clk/rockchip/clk-rk3228.c
+++ b/drivers/clk/rockchip/clk-rk3228.c
@@ -369,7 +369,7 @@ static struct rockchip_clk_branch rk3228_clk_branches[] __initdata = {
RK2928_CLKSEL_CON(23), 5, 2, MFLAGS, 0, 6, DFLAGS,
RK2928_CLKGATE_CON(2), 15, GFLAGS),
- COMPOSITE(SCLK_SDMMC, "sclk_sdmmc0", mux_mmc_src_p, 0,
+ COMPOSITE(SCLK_SDMMC, "sclk_sdmmc", mux_mmc_src_p, 0,
RK2928_CLKSEL_CON(11), 8, 2, MFLAGS, 0, 8, DFLAGS,
RK2928_CLKGATE_CON(2), 11, GFLAGS),
diff --git a/drivers/clk/rockchip/clk-rk3399.c b/drivers/clk/rockchip/clk-rk3399.c
index 8387c7a..4109988 100644
--- a/drivers/clk/rockchip/clk-rk3399.c
+++ b/drivers/clk/rockchip/clk-rk3399.c
@@ -629,7 +629,7 @@ static struct rockchip_clk_branch rk3399_clk_branches[] __initdata = {
MUX(0, "clk_i2sout_src", mux_i2sch_p, CLK_SET_RATE_PARENT,
RK3399_CLKSEL_CON(31), 0, 2, MFLAGS),
COMPOSITE_NODIV(SCLK_I2S_8CH_OUT, "clk_i2sout", mux_i2sout_p, CLK_SET_RATE_PARENT,
- RK3399_CLKSEL_CON(30), 8, 2, MFLAGS,
+ RK3399_CLKSEL_CON(31), 2, 1, MFLAGS,
RK3399_CLKGATE_CON(8), 12, GFLAGS),
/* uart */
@@ -1521,6 +1521,7 @@ static const char *const rk3399_pmucru_critical_clocks[] __initconst = {
"pclk_pmu_src",
"fclk_cm0s_src_pmu",
"clk_timer_src_pmu",
+ "pclk_rkpwm_pmu",
};
static void __init rk3399_clk_init(struct device_node *np)
diff --git a/drivers/clk/samsung/clk-exynos3250.c b/drivers/clk/samsung/clk-exynos3250.c
index 1b81e28..ed36728 100644
--- a/drivers/clk/samsung/clk-exynos3250.c
+++ b/drivers/clk/samsung/clk-exynos3250.c
@@ -698,7 +698,7 @@ static const struct samsung_pll_rate_table exynos3250_epll_rates[] __initconst =
PLL_36XX_RATE(144000000, 96, 2, 3, 0),
PLL_36XX_RATE( 96000000, 128, 2, 4, 0),
PLL_36XX_RATE( 84000000, 112, 2, 4, 0),
- PLL_36XX_RATE( 80000004, 106, 2, 4, 43691),
+ PLL_36XX_RATE( 80000003, 106, 2, 4, 43691),
PLL_36XX_RATE( 73728000, 98, 2, 4, 19923),
PLL_36XX_RATE( 67737598, 270, 3, 5, 62285),
PLL_36XX_RATE( 65535999, 174, 2, 5, 49982),
@@ -734,7 +734,7 @@ static const struct samsung_pll_rate_table exynos3250_vpll_rates[] __initconst =
PLL_36XX_RATE(148352005, 98, 2, 3, 59070),
PLL_36XX_RATE(108000000, 144, 2, 4, 0),
PLL_36XX_RATE( 74250000, 99, 2, 4, 0),
- PLL_36XX_RATE( 74176002, 98, 3, 4, 59070),
+ PLL_36XX_RATE( 74176002, 98, 2, 4, 59070),
PLL_36XX_RATE( 54054000, 216, 3, 5, 14156),
PLL_36XX_RATE( 54000000, 144, 2, 5, 0),
{ /* sentinel */ }
diff --git a/drivers/clk/samsung/clk-exynos5250.c b/drivers/clk/samsung/clk-exynos5250.c
index 27a227d..6a0cb8a 100644
--- a/drivers/clk/samsung/clk-exynos5250.c
+++ b/drivers/clk/samsung/clk-exynos5250.c
@@ -711,13 +711,13 @@ static const struct samsung_pll_rate_table epll_24mhz_tbl[] __initconst = {
/* sorted in descending order */
/* PLL_36XX_RATE(rate, m, p, s, k) */
PLL_36XX_RATE(192000000, 64, 2, 2, 0),
- PLL_36XX_RATE(180633600, 90, 3, 2, 20762),
+ PLL_36XX_RATE(180633605, 90, 3, 2, 20762),
PLL_36XX_RATE(180000000, 90, 3, 2, 0),
PLL_36XX_RATE(73728000, 98, 2, 4, 19923),
- PLL_36XX_RATE(67737600, 90, 2, 4, 20762),
+ PLL_36XX_RATE(67737602, 90, 2, 4, 20762),
PLL_36XX_RATE(49152000, 98, 3, 4, 19923),
- PLL_36XX_RATE(45158400, 90, 3, 4, 20762),
- PLL_36XX_RATE(32768000, 131, 3, 5, 4719),
+ PLL_36XX_RATE(45158401, 90, 3, 4, 20762),
+ PLL_36XX_RATE(32768001, 131, 3, 5, 4719),
{ },
};
diff --git a/drivers/clk/samsung/clk-exynos5260.c b/drivers/clk/samsung/clk-exynos5260.c
index fd1d9bf..8eae175 100644
--- a/drivers/clk/samsung/clk-exynos5260.c
+++ b/drivers/clk/samsung/clk-exynos5260.c
@@ -65,7 +65,7 @@ static const struct samsung_pll_rate_table pll2650_24mhz_tbl[] __initconst = {
PLL_36XX_RATE(480000000, 160, 2, 2, 0),
PLL_36XX_RATE(432000000, 144, 2, 2, 0),
PLL_36XX_RATE(400000000, 200, 3, 2, 0),
- PLL_36XX_RATE(394073130, 459, 7, 2, 49282),
+ PLL_36XX_RATE(394073128, 459, 7, 2, 49282),
PLL_36XX_RATE(333000000, 111, 2, 2, 0),
PLL_36XX_RATE(300000000, 100, 2, 2, 0),
PLL_36XX_RATE(266000000, 266, 3, 3, 0),
diff --git a/drivers/clk/samsung/clk-exynos5420.c b/drivers/clk/samsung/clk-exynos5420.c
index cdc092a..13c09a7 100644
--- a/drivers/clk/samsung/clk-exynos5420.c
+++ b/drivers/clk/samsung/clk-exynos5420.c
@@ -280,6 +280,7 @@ static const struct samsung_clk_reg_dump exynos5420_set_clksrc[] = {
{ .offset = GATE_BUS_TOP, .value = 0xffffffff, },
{ .offset = GATE_BUS_DISP1, .value = 0xffffffff, },
{ .offset = GATE_IP_PERIC, .value = 0xffffffff, },
+ { .offset = GATE_IP_PERIS, .value = 0xffffffff, },
};
static int exynos5420_clk_suspend(void)
@@ -987,7 +988,7 @@ static const struct samsung_gate_clock exynos5x_gate_clks[] __initconst = {
GATE(0, "aclk400_isp", "mout_user_aclk400_isp",
GATE_BUS_TOP, 16, 0, 0),
GATE(0, "aclk400_mscl", "mout_user_aclk400_mscl",
- GATE_BUS_TOP, 17, 0, 0),
+ GATE_BUS_TOP, 17, CLK_IS_CRITICAL, 0),
GATE(0, "aclk200_disp1", "mout_user_aclk200_disp1",
GATE_BUS_TOP, 18, CLK_IS_CRITICAL, 0),
GATE(CLK_SCLK_MPHY_IXTAL24, "sclk_mphy_ixtal24", "mphy_refclk_ixtal24",
diff --git a/drivers/clk/samsung/clk-exynos5433.c b/drivers/clk/samsung/clk-exynos5433.c
index 2fe0573..09cdd35 100644
--- a/drivers/clk/samsung/clk-exynos5433.c
+++ b/drivers/clk/samsung/clk-exynos5433.c
@@ -725,7 +725,7 @@ static const struct samsung_pll_rate_table exynos5443_pll_rates[] __initconst =
PLL_35XX_RATE(800000000U, 400, 6, 1),
PLL_35XX_RATE(733000000U, 733, 12, 1),
PLL_35XX_RATE(700000000U, 175, 3, 1),
- PLL_35XX_RATE(667000000U, 222, 4, 1),
+ PLL_35XX_RATE(666000000U, 222, 4, 1),
PLL_35XX_RATE(633000000U, 211, 4, 1),
PLL_35XX_RATE(600000000U, 500, 5, 2),
PLL_35XX_RATE(552000000U, 460, 5, 2),
@@ -751,12 +751,12 @@ static const struct samsung_pll_rate_table exynos5443_pll_rates[] __initconst =
/* AUD_PLL */
static const struct samsung_pll_rate_table exynos5443_aud_pll_rates[] __initconst = {
PLL_36XX_RATE(400000000U, 200, 3, 2, 0),
- PLL_36XX_RATE(393216000U, 197, 3, 2, -25690),
+ PLL_36XX_RATE(393216003U, 197, 3, 2, -25690),
PLL_36XX_RATE(384000000U, 128, 2, 2, 0),
- PLL_36XX_RATE(368640000U, 246, 4, 2, -15729),
- PLL_36XX_RATE(361507200U, 181, 3, 2, -16148),
- PLL_36XX_RATE(338688000U, 113, 2, 2, -6816),
- PLL_36XX_RATE(294912000U, 98, 1, 3, 19923),
+ PLL_36XX_RATE(368639991U, 246, 4, 2, -15729),
+ PLL_36XX_RATE(361507202U, 181, 3, 2, -16148),
+ PLL_36XX_RATE(338687988U, 113, 2, 2, -6816),
+ PLL_36XX_RATE(294912002U, 98, 1, 3, 19923),
PLL_36XX_RATE(288000000U, 96, 1, 3, 0),
PLL_36XX_RATE(252000000U, 84, 1, 3, 0),
{ /* sentinel */ }
diff --git a/drivers/clk/samsung/clk-exynos7.c b/drivers/clk/samsung/clk-exynos7.c
index 5931a41..bbfa57b 100644
--- a/drivers/clk/samsung/clk-exynos7.c
+++ b/drivers/clk/samsung/clk-exynos7.c
@@ -140,7 +140,7 @@ static const struct samsung_div_clock topc_div_clks[] __initconst = {
};
static const struct samsung_pll_rate_table pll1460x_24mhz_tbl[] __initconst = {
- PLL_36XX_RATE(491520000, 20, 1, 0, 31457),
+ PLL_36XX_RATE(491519897, 20, 1, 0, 31457),
{},
};
diff --git a/drivers/clk/samsung/clk-s3c2410.c b/drivers/clk/samsung/clk-s3c2410.c
index d7a1e77..5f50037 100644
--- a/drivers/clk/samsung/clk-s3c2410.c
+++ b/drivers/clk/samsung/clk-s3c2410.c
@@ -168,7 +168,7 @@ static struct samsung_pll_rate_table pll_s3c2410_12mhz_tbl[] __initdata = {
PLL_35XX_RATE(226000000, 105, 1, 1),
PLL_35XX_RATE(210000000, 132, 2, 1),
/* 2410 common */
- PLL_35XX_RATE(203000000, 161, 3, 1),
+ PLL_35XX_RATE(202800000, 161, 3, 1),
PLL_35XX_RATE(192000000, 88, 1, 1),
PLL_35XX_RATE(186000000, 85, 1, 1),
PLL_35XX_RATE(180000000, 82, 1, 1),
@@ -178,18 +178,18 @@ static struct samsung_pll_rate_table pll_s3c2410_12mhz_tbl[] __initdata = {
PLL_35XX_RATE(147000000, 90, 2, 1),
PLL_35XX_RATE(135000000, 82, 2, 1),
PLL_35XX_RATE(124000000, 116, 1, 2),
- PLL_35XX_RATE(118000000, 150, 2, 2),
+ PLL_35XX_RATE(118500000, 150, 2, 2),
PLL_35XX_RATE(113000000, 105, 1, 2),
- PLL_35XX_RATE(101000000, 127, 2, 2),
+ PLL_35XX_RATE(101250000, 127, 2, 2),
PLL_35XX_RATE(90000000, 112, 2, 2),
- PLL_35XX_RATE(85000000, 105, 2, 2),
+ PLL_35XX_RATE(84750000, 105, 2, 2),
PLL_35XX_RATE(79000000, 71, 1, 2),
- PLL_35XX_RATE(68000000, 82, 2, 2),
- PLL_35XX_RATE(56000000, 142, 2, 3),
+ PLL_35XX_RATE(67500000, 82, 2, 2),
+ PLL_35XX_RATE(56250000, 142, 2, 3),
PLL_35XX_RATE(48000000, 120, 2, 3),
- PLL_35XX_RATE(51000000, 161, 3, 3),
+ PLL_35XX_RATE(50700000, 161, 3, 3),
PLL_35XX_RATE(45000000, 82, 1, 3),
- PLL_35XX_RATE(34000000, 82, 2, 3),
+ PLL_35XX_RATE(33750000, 82, 2, 3),
{ /* sentinel */ },
};
diff --git a/drivers/clk/tegra/clk-pll.c b/drivers/clk/tegra/clk-pll.c
index b385536..66d1fc7 100644
--- a/drivers/clk/tegra/clk-pll.c
+++ b/drivers/clk/tegra/clk-pll.c
@@ -1145,6 +1145,8 @@ static const struct clk_ops tegra_clk_pllu_ops = {
.enable = clk_pllu_enable,
.disable = clk_pll_disable,
.recalc_rate = clk_pll_recalc_rate,
+ .round_rate = clk_pll_round_rate,
+ .set_rate = clk_pll_set_rate,
};
static int _pll_fixed_mdiv(struct tegra_clk_pll_params *pll_params,
diff --git a/drivers/clocksource/fsl_ftm_timer.c b/drivers/clocksource/fsl_ftm_timer.c
index 738515b..a22c1d7 100644
--- a/drivers/clocksource/fsl_ftm_timer.c
+++ b/drivers/clocksource/fsl_ftm_timer.c
@@ -281,7 +281,7 @@ static int __init __ftm_clk_init(struct device_node *np, char *cnt_name,
static unsigned long __init ftm_clk_init(struct device_node *np)
{
- unsigned long freq;
+ long freq;
freq = __ftm_clk_init(np, "ftm-evt-counter-en", "ftm-evt");
if (freq <= 0)
diff --git a/drivers/clocksource/i8253.c b/drivers/clocksource/i8253.c
index 0efd36e..a2b545f 100644
--- a/drivers/clocksource/i8253.c
+++ b/drivers/clocksource/i8253.c
@@ -19,6 +19,13 @@
DEFINE_RAW_SPINLOCK(i8253_lock);
EXPORT_SYMBOL(i8253_lock);
+/*
+ * Handle PIT quirk in pit_shutdown() where zeroing the counter register
+ * restarts the PIT, negating the shutdown. On platforms with the quirk,
+ * platform specific code can set this to false.
+ */
+bool i8253_clear_counter_on_shutdown __ro_after_init = true;
+
#ifdef CONFIG_CLKSRC_I8253
/*
* Since the PIT overflows every tick, its not very useful
@@ -108,8 +115,11 @@ static int pit_shutdown(struct clock_event_device *evt)
raw_spin_lock(&i8253_lock);
outb_p(0x30, PIT_MODE);
- outb_p(0, PIT_CH0);
- outb_p(0, PIT_CH0);
+
+ if (i8253_clear_counter_on_shutdown) {
+ outb_p(0, PIT_CH0);
+ outb_p(0, PIT_CH0);
+ }
raw_spin_unlock(&i8253_lock);
return 0;
diff --git a/drivers/clocksource/timer-ti-32k.c b/drivers/clocksource/timer-ti-32k.c
index cf5b14e..e9ab92d 100644
--- a/drivers/clocksource/timer-ti-32k.c
+++ b/drivers/clocksource/timer-ti-32k.c
@@ -98,6 +98,9 @@ static int __init ti_32k_timer_init(struct device_node *np)
return -ENXIO;
}
+ if (!of_machine_is_compatible("ti,am43"))
+ ti_32k_timer.cs.flags |= CLOCK_SOURCE_SUSPEND_NONSTOP;
+
ti_32k_timer.counter = ti_32k_timer.base;
/*
diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c
index 4852d9e..9f09752 100644
--- a/drivers/cpufreq/cppc_cpufreq.c
+++ b/drivers/cpufreq/cppc_cpufreq.c
@@ -151,9 +151,19 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy)
policy->cpuinfo.transition_latency = cppc_get_transition_latency(cpu_num);
policy->shared_type = cpu->shared_type;
- if (policy->shared_type == CPUFREQ_SHARED_TYPE_ANY)
+ if (policy->shared_type == CPUFREQ_SHARED_TYPE_ANY) {
+ int i;
+
cpumask_copy(policy->cpus, cpu->shared_cpu_map);
- else if (policy->shared_type == CPUFREQ_SHARED_TYPE_ALL) {
+
+ for_each_cpu(i, policy->cpus) {
+ if (unlikely(i == policy->cpu))
+ continue;
+
+ memcpy(&all_cpu_data[i]->perf_caps, &cpu->perf_caps,
+ sizeof(cpu->perf_caps));
+ }
+ } else if (policy->shared_type == CPUFREQ_SHARED_TYPE_ALL) {
/* Support only SW_ANY for now. */
pr_debug("Unsupported CPU co-ord type\n");
return -EFAULT;
@@ -218,8 +228,13 @@ static int __init cppc_cpufreq_init(void)
return ret;
out:
- for_each_possible_cpu(i)
- kfree(all_cpu_data[i]);
+ for_each_possible_cpu(i) {
+ cpu = all_cpu_data[i];
+ if (!cpu)
+ break;
+ free_cpumask_var(cpu->shared_cpu_map);
+ kfree(cpu);
+ }
kfree(all_cpu_data);
return -ENODEV;
diff --git a/drivers/cpufreq/cpufreq-dt.c b/drivers/cpufreq/cpufreq-dt.c
index e4cbfe5..fbb278b 100644
--- a/drivers/cpufreq/cpufreq-dt.c
+++ b/drivers/cpufreq/cpufreq-dt.c
@@ -32,6 +32,7 @@ struct private_data {
struct device *cpu_dev;
struct thermal_cooling_device *cdev;
const char *reg_name;
+ bool have_static_opps;
};
static struct freq_attr *cpufreq_dt_attr[] = {
@@ -197,6 +198,15 @@ static int cpufreq_init(struct cpufreq_policy *policy)
}
}
+ priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+ if (!priv) {
+ ret = -ENOMEM;
+ goto out_put_regulator;
+ }
+
+ priv->reg_name = name;
+ priv->opp_table = opp_table;
+
/*
* Initialize OPP tables for all policy->cpus. They will be shared by
* all CPUs which have marked their CPUs shared with OPP bindings.
@@ -207,7 +217,8 @@ static int cpufreq_init(struct cpufreq_policy *policy)
*
* OPPs might be populated at runtime, don't check for error here
*/
- dev_pm_opp_of_cpumask_add_table(policy->cpus);
+ if (!dev_pm_opp_of_cpumask_add_table(policy->cpus))
+ priv->have_static_opps = true;
/*
* But we need OPP table to function so if it is not there let's
@@ -233,19 +244,10 @@ static int cpufreq_init(struct cpufreq_policy *policy)
__func__, ret);
}
- priv = kzalloc(sizeof(*priv), GFP_KERNEL);
- if (!priv) {
- ret = -ENOMEM;
- goto out_free_opp;
- }
-
- priv->reg_name = name;
- priv->opp_table = opp_table;
-
ret = dev_pm_opp_init_cpufreq_table(cpu_dev, &freq_table);
if (ret) {
dev_err(cpu_dev, "failed to init cpufreq table: %d\n", ret);
- goto out_free_priv;
+ goto out_free_opp;
}
priv->cpu_dev = cpu_dev;
@@ -291,10 +293,11 @@ static int cpufreq_init(struct cpufreq_policy *policy)
out_free_cpufreq_table:
dev_pm_opp_free_cpufreq_table(cpu_dev, &freq_table);
-out_free_priv:
- kfree(priv);
out_free_opp:
- dev_pm_opp_of_cpumask_remove_table(policy->cpus);
+ if (priv->have_static_opps)
+ dev_pm_opp_of_cpumask_remove_table(policy->cpus);
+ kfree(priv);
+out_put_regulator:
if (name)
dev_pm_opp_put_regulator(opp_table);
out_put_clk:
@@ -309,7 +312,8 @@ static int cpufreq_exit(struct cpufreq_policy *policy)
cpufreq_cooling_unregister(priv->cdev);
dev_pm_opp_free_cpufreq_table(priv->cpu_dev, &policy->freq_table);
- dev_pm_opp_of_cpumask_remove_table(policy->related_cpus);
+ if (priv->have_static_opps)
+ dev_pm_opp_of_cpumask_remove_table(policy->related_cpus);
if (priv->reg_name)
dev_pm_opp_put_regulator(priv->opp_table);
diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c
index d82ce73..8aa06ce 100644
--- a/drivers/cpufreq/cpufreq.c
+++ b/drivers/cpufreq/cpufreq.c
@@ -152,7 +152,7 @@ static inline u64 get_cpu_idle_time_jiffy(unsigned int cpu, u64 *wall)
u64 cur_wall_time;
u64 busy_time;
- cur_wall_time = jiffies64_to_cputime64(get_jiffies_64());
+ cur_wall_time = jiffies64_to_nsecs(get_jiffies_64());
busy_time = kcpustat_cpu(cpu).cpustat[CPUTIME_USER];
busy_time += kcpustat_cpu(cpu).cpustat[CPUTIME_SYSTEM];
@@ -163,9 +163,9 @@ static inline u64 get_cpu_idle_time_jiffy(unsigned int cpu, u64 *wall)
idle_time = cur_wall_time - busy_time;
if (wall)
- *wall = cputime_to_usecs(cur_wall_time);
+ *wall = div_u64(cur_wall_time, NSEC_PER_USEC);
- return cputime_to_usecs(idle_time);
+ return div_u64(idle_time, NSEC_PER_USEC);
}
u64 get_cpu_idle_time(unsigned int cpu, u64 *wall, int io_busy)
@@ -773,9 +773,8 @@ static ssize_t store_##file_name \
struct cpufreq_policy new_policy; \
\
memcpy(&new_policy, policy, sizeof(*policy)); \
- \
- new_policy.min = new_policy.user_policy.min; \
- new_policy.max = new_policy.user_policy.max; \
+ new_policy.min = policy->user_policy.min; \
+ new_policy.max = policy->user_policy.max; \
\
ret = sscanf(buf, "%u", &new_policy.object); \
if (ret != 1) \
@@ -1409,14 +1408,14 @@ static int cpufreq_online(unsigned int cpu)
return 0;
out_exit_policy:
+ for_each_cpu(j, policy->real_cpus)
+ remove_cpu_dev_symlink(policy, get_cpu_device(j));
+
up_write(&policy->rwsem);
if (cpufreq_driver->exit)
cpufreq_driver->exit(policy);
- for_each_cpu(j, policy->real_cpus)
- remove_cpu_dev_symlink(policy, get_cpu_device(j));
-
out_free_policy:
cpufreq_policy_free(policy, !new_policy);
return ret;
diff --git a/drivers/cpufreq/cpufreq_governor.c b/drivers/cpufreq/cpufreq_governor.c
index 642dd0f..38d1a82 100644
--- a/drivers/cpufreq/cpufreq_governor.c
+++ b/drivers/cpufreq/cpufreq_governor.c
@@ -152,7 +152,7 @@ unsigned int dbs_update(struct cpufreq_policy *policy)
if (ignore_nice) {
u64 cur_nice = kcpustat_cpu(j).cpustat[CPUTIME_NICE];
- idle_time += cputime_to_usecs(cur_nice - j_cdbs->prev_cpu_nice);
+ idle_time += div_u64(cur_nice - j_cdbs->prev_cpu_nice, NSEC_PER_USEC);
j_cdbs->prev_cpu_nice = cur_nice;
}
diff --git a/drivers/cpufreq/cpufreq_stats.c b/drivers/cpufreq/cpufreq_stats.c
index 06d3abd..b084708 100644
--- a/drivers/cpufreq/cpufreq_stats.c
+++ b/drivers/cpufreq/cpufreq_stats.c
@@ -13,7 +13,6 @@
#include <linux/cpufreq.h>
#include <linux/module.h>
#include <linux/slab.h>
-#include <linux/cputime.h>
static DEFINE_SPINLOCK(cpufreq_stats_lock);
diff --git a/drivers/cpufreq/cpufreq_times.c b/drivers/cpufreq/cpufreq_times.c
index 6254f45..34979e7 100644
--- a/drivers/cpufreq/cpufreq_times.c
+++ b/drivers/cpufreq/cpufreq_times.c
@@ -24,19 +24,30 @@
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/threads.h>
+#include <linux/uaccess.h>
#define UID_HASH_BITS 10
static DECLARE_HASHTABLE(uid_hash_table, UID_HASH_BITS);
static DEFINE_SPINLOCK(task_time_in_state_lock); /* task->time_in_state */
+static DEFINE_SPINLOCK(task_concurrent_active_time_lock);
+ /* task->concurrent_active_time */
+static DEFINE_SPINLOCK(task_concurrent_policy_time_lock);
+ /* task->concurrent_policy_time */
static DEFINE_SPINLOCK(uid_lock); /* uid_hash_table */
+struct concurrent_times {
+ atomic64_t active[NR_CPUS];
+ atomic64_t policy[NR_CPUS];
+};
+
struct uid_entry {
uid_t uid;
unsigned int max_state;
struct hlist_node hash;
struct rcu_head rcu;
+ struct concurrent_times *concurrent_times;
u64 time_in_state[0];
};
@@ -58,6 +69,28 @@ static struct cpu_freqs *all_freqs[NR_CPUS];
static unsigned int next_offset;
+/* uid_cpupower_enable is used to enable/disable concurrent_*_time. This
+ * variable will be used in P/H experiments.
+ *
+ * Because it is being used to test performance and power, it should have a
+ * minimum impact on both. For these performance reasons, it will not be guarded
+ * by a lock or protective barriers. This limits what it can safely
+ * enable/disable.
+ *
+ * It is safe to check it before updating any concurrent_*_time stats. If there
+ * are changes uid_cpupower_enable state while we are updating the stats, we
+ * will simply ignore the changes until the next attempt to update the stats.
+ * This may result in a couple ms where the uid_cpupower_enable is in one state
+ * and the code is acting in another. Since the P/H experiments are done over
+ * the course of many days, a couple ms delay should not be an issue.
+ *
+ * It is not safe to delete the associated proc files without additional locking
+ * mechanisms that would hurt performance. Leaving the files empty but intact
+ * will not have any impact on the P/H experiments provided that userspace does
+ * not attempt to read them. Since the P/H experiment will also disable the code
+ * that reads these files from userspace, this is not a concern.
+ */
+static char uid_cpupower_enable;
/* Caller must hold rcu_read_lock() */
static struct uid_entry *find_uid_entry_rcu(uid_t uid)
@@ -87,6 +120,7 @@ static struct uid_entry *find_uid_entry_locked(uid_t uid)
static struct uid_entry *find_or_register_uid_locked(uid_t uid)
{
struct uid_entry *uid_entry, *temp;
+ struct concurrent_times *times;
unsigned int max_state = READ_ONCE(next_offset);
size_t alloc_size = sizeof(*uid_entry) + max_state *
sizeof(uid_entry->time_in_state[0]);
@@ -115,9 +149,15 @@ static struct uid_entry *find_or_register_uid_locked(uid_t uid)
uid_entry = kzalloc(alloc_size, GFP_ATOMIC);
if (!uid_entry)
return NULL;
+ times = kzalloc(sizeof(*times), GFP_ATOMIC);
+ if (!times) {
+ kfree(uid_entry);
+ return NULL;
+ }
uid_entry->uid = uid;
uid_entry->max_state = max_state;
+ uid_entry->concurrent_times = times;
hash_add_rcu(uid_hash_table, &uid_entry->hash, uid);
@@ -232,18 +272,288 @@ static int uid_time_in_state_seq_show(struct seq_file *m, void *v)
return 0;
}
+/*
+ * time_in_state is an array of u64's in the following format:
+ * [n, uid0, time0a, time0b, ..., time0n,
+ * uid1, time1a, time1b, ..., time1n,
+ * uid2, time2a, time2b, ..., time2n, etc.]
+ * where n is the total number of frequencies
+ */
+static int time_in_state_seq_show(struct seq_file *m, void *v)
+{
+ struct uid_entry *uid_entry;
+ struct cpu_freqs *freqs, *last_freqs = NULL;
+ u32 cnt = 0;
+ u32 uid, time;
+ int i, cpu;
+
+ if (v == uid_hash_table) {
+ for_each_possible_cpu(cpu) {
+ freqs = all_freqs[cpu];
+ if (!freqs || freqs == last_freqs)
+ continue;
+ last_freqs = freqs;
+ for (i = 0; i < freqs->max_state; i++) {
+ if (freqs->freq_table[i] ==
+ CPUFREQ_ENTRY_INVALID)
+ continue;
+ cnt++;
+ }
+ }
+ seq_write(m, &cnt, sizeof(cnt));
+ }
+
+ rcu_read_lock();
+
+ hlist_for_each_entry_rcu(uid_entry, (struct hlist_head *)v, hash) {
+ if (uid_entry->max_state) {
+ uid = (u32) uid_entry->uid;
+ seq_write(m, &uid, sizeof(uid));
+ }
+
+ for (i = 0; i < uid_entry->max_state; ++i) {
+ if (freq_index_invalid(i))
+ continue;
+ time = (u32)
+ cputime_to_clock_t(uid_entry->time_in_state[i]);
+ seq_write(m, &time, sizeof(time));
+ }
+ }
+
+ rcu_read_unlock();
+ return 0;
+}
+
+/*
+ * concurrent_active_time is an array of u32's in the following format:
+ * [n, uid0, time0a, time0b, ..., time0n,
+ * uid1, time1a, time1b, ..., time1n,
+ * uid2, time2a, time2b, ..., time2n, etc.]
+ * where n is the total number of cpus (num_possible_cpus)
+ */
+static int concurrent_active_time_seq_show(struct seq_file *m, void *v)
+{
+ struct uid_entry *uid_entry;
+ u32 uid, time, num_possible_cpus = num_possible_cpus();
+ int i;
+
+ if (!uid_cpupower_enable)
+ return 0;
+
+ if (v == uid_hash_table)
+ seq_write(m, &num_possible_cpus, sizeof(num_possible_cpus));
+
+ rcu_read_lock();
+
+ hlist_for_each_entry_rcu(uid_entry, (struct hlist_head *)v, hash) {
+ uid = (u32) uid_entry->uid;
+ seq_write(m, &uid, sizeof(uid));
+
+ for (i = 0; i < num_possible_cpus; ++i) {
+ time = (u32) cputime_to_clock_t(
+ atomic64_read(
+ &uid_entry->concurrent_times->active[i])
+ );
+ seq_write(m, &time, sizeof(time));
+ }
+ }
+
+ rcu_read_unlock();
+ return 0;
+}
+
+/*
+ * concurrent_policy_time is an array of u32's in the following format:
+ * [n, x1, ..., xn, uid0, time0_1_1, ..., time0_1_x1,
+ * ...,
+ * time0_n_1, ..., time0_n_xn,
+ * uid1, time1_1_1, ..., time1_1_x1,
+ * ...,
+ * time1_n_1, ..., time1_n_xn, etc.]
+ * where n is the number of policies
+ * xi is the number cpus on a particular policy
+ * timea_b_c is uid a's time running on policy b as one of c concurrent tasks
+ */
+static int concurrent_policy_time_seq_show(struct seq_file *m, void *v)
+{
+ struct uid_entry *uid_entry;
+ struct cpu_freqs *freqs;
+ struct cpu_freqs *last_freqs = NULL;
+ u32 buf[num_possible_cpus() + 1];
+ u32 uid, time;
+ int i, cnt = 0, num_possible_cpus = num_possible_cpus();
+
+ if (!uid_cpupower_enable)
+ return 0;
+
+ if (v == uid_hash_table) {
+ for_each_possible_cpu(i) {
+ freqs = all_freqs[i];
+ if (!freqs)
+ continue;
+ if (freqs != last_freqs) {
+ cnt++;
+ last_freqs = freqs;
+ buf[cnt] = 0;
+ }
+ ++buf[cnt];
+ }
+ buf[0] = (u32) cnt;
+ seq_write(m, buf, (cnt + 1) * sizeof(*buf));
+ }
+ rcu_read_lock();
+ hlist_for_each_entry_rcu(uid_entry, (struct hlist_head *)v, hash) {
+ uid = (u32) uid_entry->uid;
+ seq_write(m, &uid, sizeof(uid));
+
+ for (i = 0; i < num_possible_cpus; ++i) {
+ time = (u32) cputime_to_clock_t(
+ atomic64_read(
+ &uid_entry->concurrent_times->policy[i])
+ );
+ seq_write(m, &time, sizeof(time));
+ }
+ }
+ rcu_read_unlock();
+ return 0;
+}
+
+static int concurrent_time_text_seq_show(struct seq_file *m, void *v,
+ atomic64_t *(*get_times)(struct concurrent_times *))
+{
+ struct uid_entry *uid_entry;
+ int i, num_possible_cpus = num_possible_cpus();
+
+ if (!uid_cpupower_enable)
+ return 0;
+
+ rcu_read_lock();
+
+ hlist_for_each_entry_rcu(uid_entry, (struct hlist_head *)v, hash) {
+ atomic64_t *times = get_times(uid_entry->concurrent_times);
+ seq_put_decimal_ull(m, "", (u64)uid_entry->uid);
+ seq_putc(m, ':');
+
+ for (i = 0; i < num_possible_cpus; ++i) {
+ u64 time = cputime_to_clock_t(atomic64_read(×[i]));
+ seq_put_decimal_ull(m, " ", time);
+ }
+ seq_putc(m, '\n');
+ }
+
+ rcu_read_unlock();
+
+ return 0;
+}
+
+static inline atomic64_t *get_active_times(struct concurrent_times *times)
+{
+ return times->active;
+}
+
+static int concurrent_active_time_text_seq_show(struct seq_file *m, void *v)
+{
+ if (!uid_cpupower_enable)
+ return 0;
+
+ if (v == uid_hash_table) {
+ seq_put_decimal_ull(m, "cpus: ", num_possible_cpus());
+ seq_putc(m, '\n');
+ }
+
+ return concurrent_time_text_seq_show(m, v, get_active_times);
+}
+
+static inline atomic64_t *get_policy_times(struct concurrent_times *times)
+{
+ return times->policy;
+}
+
+static int concurrent_policy_time_text_seq_show(struct seq_file *m, void *v)
+{
+ int i;
+ struct cpu_freqs *freqs, *last_freqs = NULL;
+
+ if (!uid_cpupower_enable)
+ return 0;
+
+ if (v == uid_hash_table) {
+ int cnt = 0;
+ for_each_possible_cpu(i) {
+ freqs = all_freqs[i];
+ if (!freqs)
+ continue;
+ if (freqs != last_freqs) {
+ if (last_freqs) {
+ seq_put_decimal_ull(m, ": ", cnt);
+ seq_putc(m, ' ');
+ cnt = 0;
+ }
+ seq_put_decimal_ull(m, "policy", i);
+
+ last_freqs = freqs;
+ }
+ cnt++;
+ }
+ if (last_freqs) {
+ seq_put_decimal_ull(m, ": ", cnt);
+ seq_putc(m, '\n');
+ }
+ }
+ return concurrent_time_text_seq_show(m, v, get_policy_times);
+}
+
+static int uid_cpupower_enable_show(struct seq_file *m, void *v)
+{
+ seq_putc(m, uid_cpupower_enable);
+ seq_putc(m, '\n');
+
+ return 0;
+}
+
+static ssize_t uid_cpupower_enable_write(struct file *file,
+ const char __user *buffer, size_t count, loff_t *ppos)
+{
+ char enable;
+
+ if (count >= sizeof(enable))
+ count = sizeof(enable);
+
+ if (copy_from_user(&enable, buffer, count))
+ return -EFAULT;
+
+ if (enable == '0')
+ uid_cpupower_enable = 0;
+ else if (enable == '1')
+ uid_cpupower_enable = 1;
+ else
+ return -EINVAL;
+
+ return count;
+}
+
void cpufreq_task_times_init(struct task_struct *p)
{
- void *temp;
unsigned long flags;
- unsigned int max_state;
-
spin_lock_irqsave(&task_time_in_state_lock, flags);
p->time_in_state = NULL;
spin_unlock_irqrestore(&task_time_in_state_lock, flags);
p->max_state = 0;
- max_state = READ_ONCE(next_offset);
+ spin_lock_irqsave(&task_concurrent_active_time_lock, flags);
+ p->concurrent_active_time = NULL;
+ spin_unlock_irqrestore(&task_concurrent_active_time_lock, flags);
+
+ spin_lock_irqsave(&task_concurrent_policy_time_lock, flags);
+ p->concurrent_policy_time = NULL;
+ spin_unlock_irqrestore(&task_concurrent_policy_time_lock, flags);
+}
+
+void cpufreq_task_times_alloc(struct task_struct *p)
+{
+ void *temp;
+ unsigned long flags;
+ unsigned int max_state = READ_ONCE(next_offset);
/* We use one array to avoid multiple allocs per task */
temp = kcalloc(max_state, sizeof(p->time_in_state[0]), GFP_ATOMIC);
@@ -254,6 +564,22 @@ void cpufreq_task_times_init(struct task_struct *p)
p->time_in_state = temp;
spin_unlock_irqrestore(&task_time_in_state_lock, flags);
p->max_state = max_state;
+
+ temp = kcalloc(num_possible_cpus(),
+ sizeof(p->concurrent_active_time[0]),
+ GFP_ATOMIC);
+
+ spin_lock_irqsave(&task_concurrent_active_time_lock, flags);
+ p->concurrent_active_time = temp;
+ spin_unlock_irqrestore(&task_concurrent_active_time_lock, flags);
+
+ temp = kcalloc(num_possible_cpus(),
+ sizeof(p->concurrent_policy_time[0]),
+ GFP_ATOMIC);
+
+ spin_lock_irqsave(&task_concurrent_policy_time_lock, flags);
+ p->concurrent_policy_time = temp;
+ spin_unlock_irqrestore(&task_concurrent_policy_time_lock, flags);
}
/* Caller must hold task_time_in_state_lock */
@@ -277,14 +603,23 @@ void cpufreq_task_times_exit(struct task_struct *p)
unsigned long flags;
void *temp;
- if (!p->time_in_state)
- return;
-
spin_lock_irqsave(&task_time_in_state_lock, flags);
temp = p->time_in_state;
p->time_in_state = NULL;
spin_unlock_irqrestore(&task_time_in_state_lock, flags);
kfree(temp);
+
+ spin_lock_irqsave(&task_concurrent_active_time_lock, flags);
+ temp = p->concurrent_active_time;
+ p->concurrent_active_time = NULL;
+ spin_unlock_irqrestore(&task_concurrent_active_time_lock, flags);
+ kfree(temp);
+
+ spin_lock_irqsave(&task_concurrent_policy_time_lock, flags);
+ temp = p->concurrent_policy_time;
+ p->concurrent_policy_time = NULL;
+ spin_unlock_irqrestore(&task_concurrent_policy_time_lock, flags);
+ kfree(temp);
}
int proc_time_in_state_show(struct seq_file *m, struct pid_namespace *ns,
@@ -319,13 +654,73 @@ int proc_time_in_state_show(struct seq_file *m, struct pid_namespace *ns,
return 0;
}
+int proc_concurrent_active_time_show(struct seq_file *m,
+ struct pid_namespace *ns, struct pid *pid, struct task_struct *p)
+{
+ int i;
+ cputime_t cputime;
+ unsigned long flags;
+
+ spin_lock_irqsave(&task_concurrent_active_time_lock, flags);
+ for (i = 0; i < num_possible_cpus(); ++i) {
+ cputime = 0;
+ if (p->concurrent_active_time)
+ cputime = p->concurrent_active_time[i];
+
+ seq_printf(m, "%d %lu\n", i,
+ (unsigned long)cputime_to_clock_t(cputime));
+ }
+ spin_unlock_irqrestore(&task_concurrent_active_time_lock, flags);
+
+ return 0;
+}
+
+int proc_concurrent_policy_time_show(struct seq_file *m,
+ struct pid_namespace *ns, struct pid *pid, struct task_struct *p)
+{
+ struct cpu_freqs *freqs;
+ struct cpu_freqs *last_freqs = NULL;
+ int cpu, cnt = 0;
+ cputime_t cputime;
+ unsigned long flags;
+
+ spin_lock_irqsave(&task_concurrent_policy_time_lock, flags);
+ for (cpu = 0; cpu < num_possible_cpus(); ++cpu) {
+ freqs = all_freqs[cpu];
+ if (!freqs)
+ continue;
+ if (freqs != last_freqs) {
+ cnt = 0;
+ last_freqs = freqs;
+ seq_printf(m, "policy%i\n", cpu);
+ }
+ cnt++;
+
+ cputime = 0;
+ if (p->concurrent_policy_time)
+ cputime = p->concurrent_policy_time[cpu];
+
+ seq_printf(m, "%d %lu\n", cnt,
+ (unsigned long)cputime_to_clock_t(cputime));
+ }
+ spin_unlock_irqrestore(&task_concurrent_policy_time_lock, flags);
+
+ return 0;
+}
+
void cpufreq_acct_update_power(struct task_struct *p, cputime_t cputime)
{
unsigned long flags;
unsigned int state;
+ unsigned int index;
+ unsigned int active_cpu_cnt = 0;
+ unsigned int policy_cpu_cnt = 0;
+ unsigned int policy_first_cpu;
struct uid_entry *uid_entry;
struct cpu_freqs *freqs = all_freqs[task_cpu(p)];
+ struct cpufreq_policy *policy;
uid_t uid = from_kuid_munged(current_user_ns(), task_uid(p));
+ int cpu = 0;
if (!freqs || p->flags & PF_EXITING)
return;
@@ -343,6 +738,58 @@ void cpufreq_acct_update_power(struct task_struct *p, cputime_t cputime)
if (uid_entry && state < uid_entry->max_state)
uid_entry->time_in_state[state] += cputime;
spin_unlock_irqrestore(&uid_lock, flags);
+
+ if (!uid_cpupower_enable)
+ return;
+
+ rcu_read_lock();
+ uid_entry = find_uid_entry_rcu(uid);
+
+ for_each_possible_cpu(cpu) {
+ if (!idle_cpu(cpu))
+ ++active_cpu_cnt;
+ }
+
+ index = active_cpu_cnt - 1;
+ spin_lock_irqsave(&task_concurrent_active_time_lock, flags);
+ if (p->concurrent_active_time)
+ p->concurrent_active_time[index] += cputime;
+ spin_unlock_irqrestore(&task_concurrent_active_time_lock, flags);
+
+ if (uid_entry) {
+ atomic64_add(cputime,
+ &uid_entry->concurrent_times->active[index]);
+ }
+
+ policy = cpufreq_cpu_get(task_cpu(p));
+ if (!policy) {
+ /*
+ * This CPU may have just come up and not have a cpufreq policy
+ * yet.
+ */
+ rcu_read_unlock();
+ return;
+ }
+
+ for_each_cpu(cpu, policy->related_cpus)
+ if (!idle_cpu(cpu))
+ ++policy_cpu_cnt;
+
+ policy_first_cpu = cpumask_first(policy->related_cpus);
+ cpufreq_cpu_put(policy);
+
+ index = policy_first_cpu + policy_cpu_cnt - 1;
+ spin_lock_irqsave(&task_concurrent_policy_time_lock, flags);
+ if (p->concurrent_policy_time)
+ p->concurrent_policy_time[index] += cputime;
+ spin_unlock_irqrestore(&task_concurrent_policy_time_lock, flags);
+
+ if (uid_entry) {
+ atomic64_add(cputime,
+ &uid_entry->concurrent_times->policy[index]);
+ }
+
+ rcu_read_unlock();
}
void cpufreq_times_create_policy(struct cpufreq_policy *policy)
@@ -384,6 +831,14 @@ void cpufreq_times_create_policy(struct cpufreq_policy *policy)
all_freqs[cpu] = freqs;
}
+static void uid_entry_reclaim(struct rcu_head *rcu)
+{
+ struct uid_entry *uid_entry = container_of(rcu, struct uid_entry, rcu);
+
+ kfree(uid_entry->concurrent_times);
+ kfree(uid_entry);
+}
+
void cpufreq_task_times_remove_uids(uid_t uid_start, uid_t uid_end)
{
struct uid_entry *uid_entry;
@@ -397,7 +852,7 @@ void cpufreq_task_times_remove_uids(uid_t uid_start, uid_t uid_end)
hash, uid_start) {
if (uid_start == uid_entry->uid) {
hash_del_rcu(&uid_entry->hash);
- kfree_rcu(uid_entry, rcu);
+ call_rcu(&uid_entry->rcu, uid_entry_reclaim);
}
}
}
@@ -450,11 +905,148 @@ static const struct file_operations uid_time_in_state_fops = {
.release = seq_release,
};
+static const struct seq_operations time_in_state_seq_ops = {
+ .start = uid_seq_start,
+ .next = uid_seq_next,
+ .stop = uid_seq_stop,
+ .show = time_in_state_seq_show,
+};
+
+int time_in_state_open(struct inode *inode, struct file *file)
+{
+ return seq_open(file, &time_in_state_seq_ops);
+}
+
+const struct file_operations time_in_state_fops = {
+ .open = time_in_state_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+
+static const struct seq_operations concurrent_active_time_seq_ops = {
+ .start = uid_seq_start,
+ .next = uid_seq_next,
+ .stop = uid_seq_stop,
+ .show = concurrent_active_time_seq_show,
+};
+
+static int concurrent_active_time_open(struct inode *inode, struct file *file)
+{
+ return seq_open(file, &concurrent_active_time_seq_ops);
+}
+
+static const struct file_operations concurrent_active_time_fops = {
+ .open = concurrent_active_time_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+
+static const struct seq_operations concurrent_active_time_text_seq_ops = {
+ .start = uid_seq_start,
+ .next = uid_seq_next,
+ .stop = uid_seq_stop,
+ .show = concurrent_active_time_text_seq_show,
+};
+
+static int concurrent_active_time_text_open(struct inode *inode,
+ struct file *file)
+{
+ return seq_open(file, &concurrent_active_time_text_seq_ops);
+}
+
+static const struct file_operations concurrent_active_time_text_fops = {
+ .open = concurrent_active_time_text_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+
+static const struct seq_operations concurrent_policy_time_seq_ops = {
+ .start = uid_seq_start,
+ .next = uid_seq_next,
+ .stop = uid_seq_stop,
+ .show = concurrent_policy_time_seq_show,
+};
+
+static int concurrent_policy_time_open(struct inode *inode, struct file *file)
+{
+ return seq_open(file, &concurrent_policy_time_seq_ops);
+}
+
+static const struct file_operations concurrent_policy_time_fops = {
+ .open = concurrent_policy_time_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+
+static const struct seq_operations concurrent_policy_time_text_seq_ops = {
+ .start = uid_seq_start,
+ .next = uid_seq_next,
+ .stop = uid_seq_stop,
+ .show = concurrent_policy_time_text_seq_show,
+};
+
+static int concurrent_policy_time_text_open(struct inode *inode,
+ struct file *file)
+{
+ return seq_open(file, &concurrent_policy_time_text_seq_ops);
+}
+
+static const struct file_operations concurrent_policy_time_text_fops = {
+ .open = concurrent_policy_time_text_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+
+static int uid_cpupower_enable_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, uid_cpupower_enable_show, PDE_DATA(inode));
+}
+
+static const struct file_operations uid_cpupower_enable_fops = {
+ .open = uid_cpupower_enable_open,
+ .read = seq_read,
+ .release = single_release,
+ .write = uid_cpupower_enable_write,
+};
+
static int __init cpufreq_times_init(void)
{
+ struct proc_dir_entry *uid_cpupower;
+
proc_create_data("uid_time_in_state", 0444, NULL,
&uid_time_in_state_fops, NULL);
+ proc_create_data("uid_concurrent_active_time", 0444, NULL,
+ &concurrent_active_time_text_fops, NULL);
+
+ proc_create_data("uid_concurrent_policy_time", 0444, NULL,
+ &concurrent_policy_time_text_fops, NULL);
+
+ uid_cpupower = proc_mkdir("uid_cpupower", NULL);
+ if (!uid_cpupower) {
+ pr_warn("%s: failed to create uid_cputime proc entry\n",
+ __func__);
+ } else {
+ proc_create_data("enable", 0666, uid_cpupower,
+ &uid_cpupower_enable_fops, NULL);
+
+ proc_create_data("time_in_state", 0444, uid_cpupower,
+ &time_in_state_fops, NULL);
+
+ proc_create_data("concurrent_active_time", 0444, uid_cpupower,
+ &concurrent_active_time_fops, NULL);
+
+ proc_create_data("concurrent_policy_time", 0444, uid_cpupower,
+ &concurrent_policy_time_fops, NULL);
+
+ uid_cpupower_enable = 1;
+ }
+
return 0;
}
diff --git a/drivers/cpufreq/imx6q-cpufreq.c b/drivers/cpufreq/imx6q-cpufreq.c
index ef1fa81..fa86946 100644
--- a/drivers/cpufreq/imx6q-cpufreq.c
+++ b/drivers/cpufreq/imx6q-cpufreq.c
@@ -130,8 +130,13 @@ static int imx6q_set_target(struct cpufreq_policy *policy, unsigned int index)
/* Ensure the arm clock divider is what we expect */
ret = clk_set_rate(arm_clk, new_freq * 1000);
if (ret) {
+ int ret1;
+
dev_err(cpu_dev, "failed to set clock rate: %d\n", ret);
- regulator_set_voltage_tol(arm_reg, volt_old, 0);
+ ret1 = regulator_set_voltage_tol(arm_reg, volt_old, 0);
+ if (ret1)
+ dev_warn(cpu_dev,
+ "failed to restore vddarm voltage: %d\n", ret1);
return ret;
}
diff --git a/drivers/cpufreq/powernv-cpufreq.c b/drivers/cpufreq/powernv-cpufreq.c
index 6fb3cd2..a1d7fa4 100644
--- a/drivers/cpufreq/powernv-cpufreq.c
+++ b/drivers/cpufreq/powernv-cpufreq.c
@@ -599,6 +599,16 @@ void gpstate_timer_handler(unsigned long data)
if (!spin_trylock(&gpstates->gpstate_lock))
return;
+ /*
+ * If the timer has migrated to the different cpu then bring
+ * it back to one of the policy->cpus
+ */
+ if (!cpumask_test_cpu(raw_smp_processor_id(), policy->cpus)) {
+ gpstates->timer.expires = jiffies + msecs_to_jiffies(1);
+ add_timer_on(&gpstates->timer, cpumask_first(policy->cpus));
+ spin_unlock(&gpstates->gpstate_lock);
+ return;
+ }
gpstates->last_sampled_time += time_diff;
gpstates->elapsed_time += time_diff;
@@ -626,10 +636,8 @@ void gpstate_timer_handler(unsigned long data)
gpstates->last_gpstate_idx = pstate_to_idx(freq_data.gpstate_id);
gpstates->last_lpstate_idx = pstate_to_idx(freq_data.pstate_id);
+ set_pstate(&freq_data);
spin_unlock(&gpstates->gpstate_lock);
-
- /* Timer may get migrated to a different cpu on cpu hot unplug */
- smp_call_function_any(policy->cpus, set_pstate, &freq_data, 1);
}
/*
diff --git a/drivers/cpuidle/cpuidle-powernv.c b/drivers/cpuidle/cpuidle-powernv.c
index 854a567..fd96af1 100644
--- a/drivers/cpuidle/cpuidle-powernv.c
+++ b/drivers/cpuidle/cpuidle-powernv.c
@@ -32,9 +32,31 @@ static struct cpuidle_state *cpuidle_state_table;
static u64 stop_psscr_table[CPUIDLE_STATE_MAX];
-static u64 snooze_timeout;
+static u64 default_snooze_timeout;
static bool snooze_timeout_en;
+static u64 get_snooze_timeout(struct cpuidle_device *dev,
+ struct cpuidle_driver *drv,
+ int index)
+{
+ int i;
+
+ if (unlikely(!snooze_timeout_en))
+ return default_snooze_timeout;
+
+ for (i = index + 1; i < drv->state_count; i++) {
+ struct cpuidle_state *s = &drv->states[i];
+ struct cpuidle_state_usage *su = &dev->states_usage[i];
+
+ if (s->disabled || su->disable)
+ continue;
+
+ return s->target_residency * tb_ticks_per_usec;
+ }
+
+ return default_snooze_timeout;
+}
+
static int snooze_loop(struct cpuidle_device *dev,
struct cpuidle_driver *drv,
int index)
@@ -44,7 +66,7 @@ static int snooze_loop(struct cpuidle_device *dev,
local_irq_enable();
set_thread_flag(TIF_POLLING_NRFLAG);
- snooze_exit_time = get_tb() + snooze_timeout;
+ snooze_exit_time = get_tb() + get_snooze_timeout(dev, drv, index);
ppc64_runlatch_off();
while (!need_resched()) {
HMT_low();
@@ -337,11 +359,9 @@ static int powernv_idle_probe(void)
cpuidle_state_table = powernv_states;
/* Device tree can indicate more idle states */
max_idle_state = powernv_add_idle_states();
- if (max_idle_state > 1) {
+ default_snooze_timeout = TICK_USEC * tb_ticks_per_usec;
+ if (max_idle_state > 1)
snooze_timeout_en = true;
- snooze_timeout = powernv_states[1].target_residency *
- tb_ticks_per_usec;
- }
} else
return -ENODEV;
diff --git a/drivers/cpuidle/governors/menu.c b/drivers/cpuidle/governors/menu.c
index 65bb6fd..72bc6e6 100644
--- a/drivers/cpuidle/governors/menu.c
+++ b/drivers/cpuidle/governors/menu.c
@@ -18,6 +18,7 @@
#include <linux/hrtimer.h>
#include <linux/tick.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/math64.h>
#include <linux/module.h>
@@ -130,10 +131,6 @@ struct menu_device {
int interval_ptr;
};
-
-#define LOAD_INT(x) ((x) >> FSHIFT)
-#define LOAD_FRAC(x) LOAD_INT(((x) & (FIXED_1-1)) * 100)
-
static inline int get_loadavg(unsigned long load)
{
return LOAD_INT(load) * 10 + LOAD_FRAC(load) / 10;
diff --git a/drivers/crypto/amcc/crypto4xx_core.c b/drivers/crypto/amcc/crypto4xx_core.c
index dae1e39..c7524bb 100644
--- a/drivers/crypto/amcc/crypto4xx_core.c
+++ b/drivers/crypto/amcc/crypto4xx_core.c
@@ -208,7 +208,7 @@ static u32 crypto4xx_build_pdr(struct crypto4xx_device *dev)
dev->pdr_pa);
return -ENOMEM;
}
- memset(dev->pdr, 0, sizeof(struct ce_pd) * PPC4XX_NUM_PD);
+ memset(dev->pdr, 0, sizeof(struct ce_pd) * PPC4XX_NUM_PD);
dev->shadow_sa_pool = dma_alloc_coherent(dev->core_dev->device,
256 * PPC4XX_NUM_PD,
&dev->shadow_sa_pool_pa,
@@ -241,13 +241,15 @@ static u32 crypto4xx_build_pdr(struct crypto4xx_device *dev)
static void crypto4xx_destroy_pdr(struct crypto4xx_device *dev)
{
- if (dev->pdr != NULL)
+ if (dev->pdr)
dma_free_coherent(dev->core_dev->device,
sizeof(struct ce_pd) * PPC4XX_NUM_PD,
dev->pdr, dev->pdr_pa);
+
if (dev->shadow_sa_pool)
dma_free_coherent(dev->core_dev->device, 256 * PPC4XX_NUM_PD,
dev->shadow_sa_pool, dev->shadow_sa_pool_pa);
+
if (dev->shadow_sr_pool)
dma_free_coherent(dev->core_dev->device,
sizeof(struct sa_state_record) * PPC4XX_NUM_PD,
@@ -417,12 +419,12 @@ static u32 crypto4xx_build_sdr(struct crypto4xx_device *dev)
static void crypto4xx_destroy_sdr(struct crypto4xx_device *dev)
{
- if (dev->sdr != NULL)
+ if (dev->sdr)
dma_free_coherent(dev->core_dev->device,
sizeof(struct ce_sd) * PPC4XX_NUM_SD,
dev->sdr, dev->sdr_pa);
- if (dev->scatter_buffer_va != NULL)
+ if (dev->scatter_buffer_va)
dma_free_coherent(dev->core_dev->device,
dev->scatter_buffer_size * PPC4XX_NUM_SD,
dev->scatter_buffer_va,
@@ -1034,12 +1036,10 @@ int crypto4xx_register_alg(struct crypto4xx_device *sec_dev,
break;
}
- if (rc) {
- list_del(&alg->entry);
+ if (rc)
kfree(alg);
- } else {
+ else
list_add_tail(&alg->entry, &sec_dev->alg_list);
- }
}
return 0;
@@ -1193,7 +1193,7 @@ static int crypto4xx_probe(struct platform_device *ofdev)
rc = crypto4xx_build_gdr(core_dev->dev);
if (rc)
- goto err_build_gdr;
+ goto err_build_pdr;
rc = crypto4xx_build_sdr(core_dev->dev);
if (rc)
@@ -1236,12 +1236,11 @@ static int crypto4xx_probe(struct platform_device *ofdev)
err_request_irq:
irq_dispose_mapping(core_dev->irq);
tasklet_kill(&core_dev->tasklet);
- crypto4xx_destroy_sdr(core_dev->dev);
err_build_sdr:
+ crypto4xx_destroy_sdr(core_dev->dev);
crypto4xx_destroy_gdr(core_dev->dev);
-err_build_gdr:
- crypto4xx_destroy_pdr(core_dev->dev);
err_build_pdr:
+ crypto4xx_destroy_pdr(core_dev->dev);
kfree(core_dev->dev);
err_alloc_dev:
kfree(core_dev);
diff --git a/drivers/crypto/caam/jr.c b/drivers/crypto/caam/jr.c
index 9e7f281..6d475a2 100644
--- a/drivers/crypto/caam/jr.c
+++ b/drivers/crypto/caam/jr.c
@@ -189,7 +189,8 @@ static void caam_jr_dequeue(unsigned long devarg)
BUG_ON(CIRC_CNT(head, tail + i, JOBR_DEPTH) <= 0);
/* Unmap just-run descriptor so we can post-process */
- dma_unmap_single(dev, jrp->outring[hw_idx].desc,
+ dma_unmap_single(dev,
+ caam_dma_to_cpu(jrp->outring[hw_idx].desc),
jrp->entinfo[sw_idx].desc_size,
DMA_TO_DEVICE);
diff --git a/drivers/crypto/caam/regs.h b/drivers/crypto/caam/regs.h
index 84d2f83..b03e6f5 100644
--- a/drivers/crypto/caam/regs.h
+++ b/drivers/crypto/caam/regs.h
@@ -68,22 +68,22 @@
extern bool caam_little_end;
-#define caam_to_cpu(len) \
-static inline u##len caam##len ## _to_cpu(u##len val) \
-{ \
- if (caam_little_end) \
- return le##len ## _to_cpu(val); \
- else \
- return be##len ## _to_cpu(val); \
+#define caam_to_cpu(len) \
+static inline u##len caam##len ## _to_cpu(u##len val) \
+{ \
+ if (caam_little_end) \
+ return le##len ## _to_cpu((__force __le##len)val); \
+ else \
+ return be##len ## _to_cpu((__force __be##len)val); \
}
-#define cpu_to_caam(len) \
-static inline u##len cpu_to_caam##len(u##len val) \
-{ \
- if (caam_little_end) \
- return cpu_to_le##len(val); \
- else \
- return cpu_to_be##len(val); \
+#define cpu_to_caam(len) \
+static inline u##len cpu_to_caam##len(u##len val) \
+{ \
+ if (caam_little_end) \
+ return (__force u##len)cpu_to_le##len(val); \
+ else \
+ return (__force u##len)cpu_to_be##len(val); \
}
caam_to_cpu(16)
diff --git a/drivers/crypto/msm/Makefile b/drivers/crypto/msm/Makefile
index b712fc1..89e03134 100644
--- a/drivers/crypto/msm/Makefile
+++ b/drivers/crypto/msm/Makefile
@@ -4,3 +4,8 @@
obj-$(CONFIG_CRYPTO_DEV_QCRYPTO) += qcrypto.o
obj-$(CONFIG_CRYPTO_DEV_OTA_CRYPTO) += ota_crypto.o
obj-$(CONFIG_CRYPTO_DEV_QCOM_ICE) += ice.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-ignored-attributes \
+ $(call cc-option,-Wno-frame-larger-than=) \
diff --git a/drivers/crypto/msm/ice.c b/drivers/crypto/msm/ice.c
index f15267e..571f7c6 100644
--- a/drivers/crypto/msm/ice.c
+++ b/drivers/crypto/msm/ice.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2014-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2014-2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -21,31 +21,12 @@
#include <linux/cdev.h>
#include <linux/regulator/consumer.h>
#include <linux/msm-bus.h>
+#include <linux/pfk.h>
#include <crypto/ice.h>
#include <soc/qcom/scm.h>
#include <soc/qcom/qseecomi.h>
#include "iceregs.h"
-#ifdef CONFIG_PFK
-#include <linux/pfk.h>
-#else
-#include <linux/bio.h>
-static inline int pfk_load_key_start(const struct bio *bio,
- struct ice_crypto_setting *ice_setting, bool *is_pfe, bool async)
-{
- return 0;
-}
-
-static inline int pfk_load_key_end(const struct bio *bio, bool *is_pfe)
-{
- return 0;
-}
-
-static inline void pfk_clear_on_reset(void)
-{
-}
-#endif
-
#define TZ_SYSCALL_CREATE_SMC_ID(o, s, f) \
((uint32_t)((((o & 0x3f) << 24) | (s & 0xff) << 8) | (f & 0xff)))
@@ -74,9 +55,6 @@ static inline void pfk_clear_on_reset(void)
#define QCOM_ICE_MAX_BIST_CHECK_COUNT 100
#define QCOM_ICE_UFS 10
#define QCOM_ICE_SDCC 20
-#define QCOM_ICE_ENCRYPT 0x1
-#define QCOM_ICE_DECRYPT 0x2
-#define QCOM_SECT_LEN_IN_BYTE 512
struct ice_clk_info {
struct list_head list;
@@ -126,11 +104,6 @@ struct ice_device {
ktime_t ice_reset_complete_time;
};
-static int ice_fde_flag;
-static unsigned long userdata_start;
-static unsigned long userdata_end;
-static struct ice_crypto_setting ice_data;
-
static int qti_ice_setting_config(struct request *req,
struct platform_device *pdev,
struct ice_crypto_setting *crypto_data,
@@ -160,50 +133,20 @@ static int qti_ice_setting_config(struct request *req,
memcpy(&setting->crypto_data, crypto_data,
sizeof(setting->crypto_data));
- if (rq_data_dir(req) == WRITE &&
- (ice_fde_flag & QCOM_ICE_ENCRYPT))
+ if (rq_data_dir(req) == WRITE)
setting->encr_bypass = false;
- else if (rq_data_dir(req) == READ &&
- (ice_fde_flag & QCOM_ICE_DECRYPT))
+ else if (rq_data_dir(req) == READ)
setting->decr_bypass = false;
else {
/* Should I say BUG_ON */
setting->encr_bypass = true;
setting->decr_bypass = true;
- pr_debug("%s direction unknown", __func__);
}
}
return 0;
}
-void qcom_ice_set_fde_flag(int flag)
-{
- ice_fde_flag = flag;
- pr_debug("%s read_write setting %d\n", __func__, ice_fde_flag);
-}
-EXPORT_SYMBOL(qcom_ice_set_fde_flag);
-
-int qcom_ice_set_fde_conf(sector_t s_sector, sector_t size,
- int index, int mode)
-{
- userdata_start = s_sector;
- userdata_end = s_sector + size;
- if (INT_MAX - s_sector < size) {
- WARN_ON(1);
- return -EINVAL;
- }
- ice_data.key_index = index;
- ice_data.algo_mode = mode;
- ice_data.key_size = ICE_CRYPTO_KEY_SIZE_256;
- ice_data.key_mode = ICE_CRYPTO_USE_LUT_SW_KEY;
-
- pr_debug("%s sector info set start %lu end %lu\n", __func__,
- userdata_start, userdata_end);
- return 0;
-}
-EXPORT_SYMBOL(qcom_ice_set_fde_conf);
-
static int qcom_ice_enable_clocks(struct ice_device *, bool);
#ifdef CONFIG_MSM_BUS_SCALING
@@ -1486,10 +1429,11 @@ static int qcom_ice_config_start(struct platform_device *pdev,
struct request *req,
struct ice_data_setting *setting, bool async)
{
+ struct ice_crypto_setting *crypto_data;
struct ice_crypto_setting pfk_crypto_data = {0};
+ union map_info *info;
int ret = 0;
bool is_pfe = false;
- sector_t data_size;
if (!pdev || !req) {
pr_err("%s: Invalid params passed\n", __func__);
@@ -1525,22 +1469,28 @@ static int qcom_ice_config_start(struct platform_device *pdev,
&pfk_crypto_data, setting);
}
- if (ice_fde_flag == 0)
- return 0;
-
- if ((req->__sector >= userdata_start) &&
- (req->__sector < userdata_end)) {
/*
- * Ugly hack to address non-block-size aligned userdata end address in
- * eMMC based devices.
+ * info field in req->end_io_data could be used by mulitple dm or
+ * non-dm entities. To ensure that we are running operation on dm
+ * based request, check BIO_DONT_FREE flag
*/
- data_size = req->__data_len/QCOM_SECT_LEN_IN_BYTE;
-
- if ((req->__sector + data_size) > userdata_end)
+ if (bio_flagged(req->bio, BIO_INLINECRYPT)) {
+ info = dm_get_rq_mapinfo(req);
+ if (!info) {
+ pr_debug("%s info not available in request\n",
+ __func__);
return 0;
- else
- return qti_ice_setting_config(req, pdev,
- &ice_data, setting);
+ }
+
+ crypto_data = (struct ice_crypto_setting *)info->ptr;
+ if (!crypto_data) {
+ pr_err("%s crypto_data not available in request\n",
+ __func__);
+ return -EINVAL;
+ }
+
+ return qti_ice_setting_config(req, pdev,
+ crypto_data, setting);
}
/*
diff --git a/drivers/crypto/mxs-dcp.c b/drivers/crypto/mxs-dcp.c
index 625ee50..decaed4 100644
--- a/drivers/crypto/mxs-dcp.c
+++ b/drivers/crypto/mxs-dcp.c
@@ -63,7 +63,7 @@ struct dcp {
struct dcp_coherent_block *coh;
struct completion completion[DCP_MAX_CHANS];
- struct mutex mutex[DCP_MAX_CHANS];
+ spinlock_t lock[DCP_MAX_CHANS];
struct task_struct *thread[DCP_MAX_CHANS];
struct crypto_queue queue[DCP_MAX_CHANS];
};
@@ -349,13 +349,20 @@ static int dcp_chan_thread_aes(void *data)
int ret;
- do {
- __set_current_state(TASK_INTERRUPTIBLE);
+ while (!kthread_should_stop()) {
+ set_current_state(TASK_INTERRUPTIBLE);
- mutex_lock(&sdcp->mutex[chan]);
+ spin_lock(&sdcp->lock[chan]);
backlog = crypto_get_backlog(&sdcp->queue[chan]);
arq = crypto_dequeue_request(&sdcp->queue[chan]);
- mutex_unlock(&sdcp->mutex[chan]);
+ spin_unlock(&sdcp->lock[chan]);
+
+ if (!backlog && !arq) {
+ schedule();
+ continue;
+ }
+
+ set_current_state(TASK_RUNNING);
if (backlog)
backlog->complete(backlog, -EINPROGRESS);
@@ -363,11 +370,8 @@ static int dcp_chan_thread_aes(void *data)
if (arq) {
ret = mxs_dcp_aes_block_crypt(arq);
arq->complete(arq, ret);
- continue;
}
-
- schedule();
- } while (!kthread_should_stop());
+ }
return 0;
}
@@ -409,9 +413,9 @@ static int mxs_dcp_aes_enqueue(struct ablkcipher_request *req, int enc, int ecb)
rctx->ecb = ecb;
actx->chan = DCP_CHAN_CRYPTO;
- mutex_lock(&sdcp->mutex[actx->chan]);
+ spin_lock(&sdcp->lock[actx->chan]);
ret = crypto_enqueue_request(&sdcp->queue[actx->chan], &req->base);
- mutex_unlock(&sdcp->mutex[actx->chan]);
+ spin_unlock(&sdcp->lock[actx->chan]);
wake_up_process(sdcp->thread[actx->chan]);
@@ -640,13 +644,20 @@ static int dcp_chan_thread_sha(void *data)
struct ahash_request *req;
int ret, fini;
- do {
- __set_current_state(TASK_INTERRUPTIBLE);
+ while (!kthread_should_stop()) {
+ set_current_state(TASK_INTERRUPTIBLE);
- mutex_lock(&sdcp->mutex[chan]);
+ spin_lock(&sdcp->lock[chan]);
backlog = crypto_get_backlog(&sdcp->queue[chan]);
arq = crypto_dequeue_request(&sdcp->queue[chan]);
- mutex_unlock(&sdcp->mutex[chan]);
+ spin_unlock(&sdcp->lock[chan]);
+
+ if (!backlog && !arq) {
+ schedule();
+ continue;
+ }
+
+ set_current_state(TASK_RUNNING);
if (backlog)
backlog->complete(backlog, -EINPROGRESS);
@@ -658,12 +669,8 @@ static int dcp_chan_thread_sha(void *data)
ret = dcp_sha_req_to_buf(arq);
fini = rctx->fini;
arq->complete(arq, ret);
- if (!fini)
- continue;
}
-
- schedule();
- } while (!kthread_should_stop());
+ }
return 0;
}
@@ -721,9 +728,9 @@ static int dcp_sha_update_fx(struct ahash_request *req, int fini)
rctx->init = 1;
}
- mutex_lock(&sdcp->mutex[actx->chan]);
+ spin_lock(&sdcp->lock[actx->chan]);
ret = crypto_enqueue_request(&sdcp->queue[actx->chan], &req->base);
- mutex_unlock(&sdcp->mutex[actx->chan]);
+ spin_unlock(&sdcp->lock[actx->chan]);
wake_up_process(sdcp->thread[actx->chan]);
mutex_unlock(&actx->mutex);
@@ -979,7 +986,7 @@ static int mxs_dcp_probe(struct platform_device *pdev)
platform_set_drvdata(pdev, sdcp);
for (i = 0; i < DCP_MAX_CHANS; i++) {
- mutex_init(&sdcp->mutex[i]);
+ spin_lock_init(&sdcp->lock[i]);
init_completion(&sdcp->completion[i]);
crypto_init_queue(&sdcp->queue[i], 50);
}
diff --git a/drivers/crypto/omap-sham.c b/drivers/crypto/omap-sham.c
index d8305dd..ff6ac4e 100644
--- a/drivers/crypto/omap-sham.c
+++ b/drivers/crypto/omap-sham.c
@@ -1081,7 +1081,7 @@ static void omap_sham_finish_req(struct ahash_request *req, int err)
if (test_bit(FLAGS_SGS_COPIED, &dd->flags))
free_pages((unsigned long)sg_virt(ctx->sg),
- get_order(ctx->sg->length));
+ get_order(ctx->sg->length + ctx->bufcnt));
if (test_bit(FLAGS_SGS_ALLOCED, &dd->flags))
kfree(ctx->sg);
diff --git a/drivers/crypto/padlock-aes.c b/drivers/crypto/padlock-aes.c
index 9126627..75f2bef 100644
--- a/drivers/crypto/padlock-aes.c
+++ b/drivers/crypto/padlock-aes.c
@@ -266,6 +266,8 @@ static inline void padlock_xcrypt_ecb(const u8 *input, u8 *output, void *key,
return;
}
+ count -= initial;
+
if (initial)
asm volatile (".byte 0xf3,0x0f,0xa7,0xc8" /* rep xcryptecb */
: "+S"(input), "+D"(output)
@@ -273,7 +275,7 @@ static inline void padlock_xcrypt_ecb(const u8 *input, u8 *output, void *key,
asm volatile (".byte 0xf3,0x0f,0xa7,0xc8" /* rep xcryptecb */
: "+S"(input), "+D"(output)
- : "d"(control_word), "b"(key), "c"(count - initial));
+ : "d"(control_word), "b"(key), "c"(count));
}
static inline u8 *padlock_xcrypt_cbc(const u8 *input, u8 *output, void *key,
@@ -284,6 +286,8 @@ static inline u8 *padlock_xcrypt_cbc(const u8 *input, u8 *output, void *key,
if (count < cbc_fetch_blocks)
return cbc_crypt(input, output, key, iv, control_word, count);
+ count -= initial;
+
if (initial)
asm volatile (".byte 0xf3,0x0f,0xa7,0xd0" /* rep xcryptcbc */
: "+S" (input), "+D" (output), "+a" (iv)
@@ -291,7 +295,7 @@ static inline u8 *padlock_xcrypt_cbc(const u8 *input, u8 *output, void *key,
asm volatile (".byte 0xf3,0x0f,0xa7,0xd0" /* rep xcryptcbc */
: "+S" (input), "+D" (output), "+a" (iv)
- : "d" (control_word), "b" (key), "c" (count-initial));
+ : "d" (control_word), "b" (key), "c" (count));
return iv;
}
diff --git a/drivers/crypto/qat/qat_c3xxx/adf_drv.c b/drivers/crypto/qat/qat_c3xxx/adf_drv.c
index 640c3fc..ad9d6fb 100644
--- a/drivers/crypto/qat/qat_c3xxx/adf_drv.c
+++ b/drivers/crypto/qat/qat_c3xxx/adf_drv.c
@@ -123,7 +123,8 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
struct adf_hw_device_data *hw_data;
char name[ADF_DEVICE_NAME_LENGTH];
unsigned int i, bar_nr;
- int ret, bar_mask;
+ unsigned long bar_mask;
+ int ret;
switch (ent->device) {
case ADF_C3XXX_PCI_DEVICE_ID:
@@ -235,8 +236,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
/* Find and map all the device's BARS */
i = 0;
bar_mask = pci_select_bars(pdev, IORESOURCE_MEM);
- for_each_set_bit(bar_nr, (const unsigned long *)&bar_mask,
- ADF_PCI_MAX_BARS * 2) {
+ for_each_set_bit(bar_nr, &bar_mask, ADF_PCI_MAX_BARS * 2) {
struct adf_bar *bar = &accel_pci_dev->pci_bars[i++];
bar->base_addr = pci_resource_start(pdev, bar_nr);
diff --git a/drivers/crypto/qat/qat_c3xxxvf/adf_drv.c b/drivers/crypto/qat/qat_c3xxxvf/adf_drv.c
index 949d77b..0dd8d2d 100644
--- a/drivers/crypto/qat/qat_c3xxxvf/adf_drv.c
+++ b/drivers/crypto/qat/qat_c3xxxvf/adf_drv.c
@@ -125,7 +125,8 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
struct adf_hw_device_data *hw_data;
char name[ADF_DEVICE_NAME_LENGTH];
unsigned int i, bar_nr;
- int ret, bar_mask;
+ unsigned long bar_mask;
+ int ret;
switch (ent->device) {
case ADF_C3XXXIOV_PCI_DEVICE_ID:
@@ -215,8 +216,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
/* Find and map all the device's BARS */
i = 0;
bar_mask = pci_select_bars(pdev, IORESOURCE_MEM);
- for_each_set_bit(bar_nr, (const unsigned long *)&bar_mask,
- ADF_PCI_MAX_BARS * 2) {
+ for_each_set_bit(bar_nr, &bar_mask, ADF_PCI_MAX_BARS * 2) {
struct adf_bar *bar = &accel_pci_dev->pci_bars[i++];
bar->base_addr = pci_resource_start(pdev, bar_nr);
diff --git a/drivers/crypto/qat/qat_c62x/adf_drv.c b/drivers/crypto/qat/qat_c62x/adf_drv.c
index 5b2d78a..dcdb94c 100644
--- a/drivers/crypto/qat/qat_c62x/adf_drv.c
+++ b/drivers/crypto/qat/qat_c62x/adf_drv.c
@@ -123,7 +123,8 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
struct adf_hw_device_data *hw_data;
char name[ADF_DEVICE_NAME_LENGTH];
unsigned int i, bar_nr;
- int ret, bar_mask;
+ unsigned long bar_mask;
+ int ret;
switch (ent->device) {
case ADF_C62X_PCI_DEVICE_ID:
@@ -235,8 +236,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
/* Find and map all the device's BARS */
i = (hw_data->fuses & ADF_DEVICE_FUSECTL_MASK) ? 1 : 0;
bar_mask = pci_select_bars(pdev, IORESOURCE_MEM);
- for_each_set_bit(bar_nr, (const unsigned long *)&bar_mask,
- ADF_PCI_MAX_BARS * 2) {
+ for_each_set_bit(bar_nr, &bar_mask, ADF_PCI_MAX_BARS * 2) {
struct adf_bar *bar = &accel_pci_dev->pci_bars[i++];
bar->base_addr = pci_resource_start(pdev, bar_nr);
diff --git a/drivers/crypto/qat/qat_c62xvf/adf_drv.c b/drivers/crypto/qat/qat_c62xvf/adf_drv.c
index 7540ce1..cd9e634 100644
--- a/drivers/crypto/qat/qat_c62xvf/adf_drv.c
+++ b/drivers/crypto/qat/qat_c62xvf/adf_drv.c
@@ -125,7 +125,8 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
struct adf_hw_device_data *hw_data;
char name[ADF_DEVICE_NAME_LENGTH];
unsigned int i, bar_nr;
- int ret, bar_mask;
+ unsigned long bar_mask;
+ int ret;
switch (ent->device) {
case ADF_C62XIOV_PCI_DEVICE_ID:
@@ -215,8 +216,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
/* Find and map all the device's BARS */
i = 0;
bar_mask = pci_select_bars(pdev, IORESOURCE_MEM);
- for_each_set_bit(bar_nr, (const unsigned long *)&bar_mask,
- ADF_PCI_MAX_BARS * 2) {
+ for_each_set_bit(bar_nr, &bar_mask, ADF_PCI_MAX_BARS * 2) {
struct adf_bar *bar = &accel_pci_dev->pci_bars[i++];
bar->base_addr = pci_resource_start(pdev, bar_nr);
diff --git a/drivers/crypto/qat/qat_dh895xcc/adf_drv.c b/drivers/crypto/qat/qat_dh895xcc/adf_drv.c
index 4d2de28..3417443 100644
--- a/drivers/crypto/qat/qat_dh895xcc/adf_drv.c
+++ b/drivers/crypto/qat/qat_dh895xcc/adf_drv.c
@@ -123,7 +123,8 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
struct adf_hw_device_data *hw_data;
char name[ADF_DEVICE_NAME_LENGTH];
unsigned int i, bar_nr;
- int ret, bar_mask;
+ unsigned long bar_mask;
+ int ret;
switch (ent->device) {
case ADF_DH895XCC_PCI_DEVICE_ID:
@@ -237,8 +238,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
/* Find and map all the device's BARS */
i = 0;
bar_mask = pci_select_bars(pdev, IORESOURCE_MEM);
- for_each_set_bit(bar_nr, (const unsigned long *)&bar_mask,
- ADF_PCI_MAX_BARS * 2) {
+ for_each_set_bit(bar_nr, &bar_mask, ADF_PCI_MAX_BARS * 2) {
struct adf_bar *bar = &accel_pci_dev->pci_bars[i++];
bar->base_addr = pci_resource_start(pdev, bar_nr);
diff --git a/drivers/crypto/qat/qat_dh895xccvf/adf_drv.c b/drivers/crypto/qat/qat_dh895xccvf/adf_drv.c
index 60df986..15de9cbe 100644
--- a/drivers/crypto/qat/qat_dh895xccvf/adf_drv.c
+++ b/drivers/crypto/qat/qat_dh895xccvf/adf_drv.c
@@ -125,7 +125,8 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
struct adf_hw_device_data *hw_data;
char name[ADF_DEVICE_NAME_LENGTH];
unsigned int i, bar_nr;
- int ret, bar_mask;
+ unsigned long bar_mask;
+ int ret;
switch (ent->device) {
case ADF_DH895XCCIOV_PCI_DEVICE_ID:
@@ -215,8 +216,7 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
/* Find and map all the device's BARS */
i = 0;
bar_mask = pci_select_bars(pdev, IORESOURCE_MEM);
- for_each_set_bit(bar_nr, (const unsigned long *)&bar_mask,
- ADF_PCI_MAX_BARS * 2) {
+ for_each_set_bit(bar_nr, &bar_mask, ADF_PCI_MAX_BARS * 2) {
struct adf_bar *bar = &accel_pci_dev->pci_bars[i++];
bar->base_addr = pci_resource_start(pdev, bar_nr);
diff --git a/drivers/crypto/sahara.c b/drivers/crypto/sahara.c
index 0c49956..4d0ec7b 100644
--- a/drivers/crypto/sahara.c
+++ b/drivers/crypto/sahara.c
@@ -1352,7 +1352,7 @@ static int sahara_register_algs(struct sahara_dev *dev)
err_sha_v3_algs:
for (j = 0; j < k; j++)
- crypto_unregister_ahash(&sha_v4_algs[j]);
+ crypto_unregister_ahash(&sha_v3_algs[j]);
err_aes_algs:
for (j = 0; j < i; j++)
@@ -1368,7 +1368,7 @@ static void sahara_unregister_algs(struct sahara_dev *dev)
for (i = 0; i < ARRAY_SIZE(aes_algs); i++)
crypto_unregister_alg(&aes_algs[i]);
- for (i = 0; i < ARRAY_SIZE(sha_v4_algs); i++)
+ for (i = 0; i < ARRAY_SIZE(sha_v3_algs); i++)
crypto_unregister_ahash(&sha_v3_algs[i]);
if (dev->version > SAHARA_VERSION_3)
diff --git a/drivers/crypto/sunxi-ss/sun4i-ss-core.c b/drivers/crypto/sunxi-ss/sun4i-ss-core.c
index 3ac6c6c..16bb660 100644
--- a/drivers/crypto/sunxi-ss/sun4i-ss-core.c
+++ b/drivers/crypto/sunxi-ss/sun4i-ss-core.c
@@ -422,6 +422,7 @@ static struct platform_driver sun4i_ss_driver = {
module_platform_driver(sun4i_ss_driver);
+MODULE_ALIAS("platform:sun4i-ss");
MODULE_DESCRIPTION("Allwinner Security System cryptographic accelerator");
MODULE_LICENSE("GPL");
MODULE_AUTHOR("Corentin LABBE <clabbe.montjoie@gmail.com>");
diff --git a/drivers/crypto/talitos.c b/drivers/crypto/talitos.c
index 42c060c..7c71722 100644
--- a/drivers/crypto/talitos.c
+++ b/drivers/crypto/talitos.c
@@ -1116,10 +1116,10 @@ static int sg_to_link_tbl_offset(struct scatterlist *sg, int sg_count,
return count;
}
-int talitos_sg_map(struct device *dev, struct scatterlist *src,
- unsigned int len, struct talitos_edesc *edesc,
- struct talitos_ptr *ptr,
- int sg_count, unsigned int offset, int tbl_off)
+static int talitos_sg_map_ext(struct device *dev, struct scatterlist *src,
+ unsigned int len, struct talitos_edesc *edesc,
+ struct talitos_ptr *ptr, int sg_count,
+ unsigned int offset, int tbl_off, int elen)
{
struct talitos_private *priv = dev_get_drvdata(dev);
bool is_sec1 = has_ftr_sec1(priv);
@@ -1130,7 +1130,7 @@ int talitos_sg_map(struct device *dev, struct scatterlist *src,
}
to_talitos_ptr_len(ptr, len, is_sec1);
- to_talitos_ptr_ext_set(ptr, 0, is_sec1);
+ to_talitos_ptr_ext_set(ptr, elen, is_sec1);
if (sg_count == 1) {
to_talitos_ptr(ptr, sg_dma_address(src) + offset, is_sec1);
@@ -1140,7 +1140,7 @@ int talitos_sg_map(struct device *dev, struct scatterlist *src,
to_talitos_ptr(ptr, edesc->dma_link_tbl + offset, is_sec1);
return sg_count;
}
- sg_count = sg_to_link_tbl_offset(src, sg_count, offset, len,
+ sg_count = sg_to_link_tbl_offset(src, sg_count, offset, len + elen,
&edesc->link_tbl[tbl_off]);
if (sg_count == 1) {
/* Only one segment now, so no link tbl needed*/
@@ -1154,6 +1154,15 @@ int talitos_sg_map(struct device *dev, struct scatterlist *src,
return sg_count;
}
+static int talitos_sg_map(struct device *dev, struct scatterlist *src,
+ unsigned int len, struct talitos_edesc *edesc,
+ struct talitos_ptr *ptr, int sg_count,
+ unsigned int offset, int tbl_off)
+{
+ return talitos_sg_map_ext(dev, src, len, edesc, ptr, sg_count, offset,
+ tbl_off, 0);
+}
+
/*
* fill in and submit ipsec_esp descriptor
*/
@@ -1171,7 +1180,7 @@ static int ipsec_esp(struct talitos_edesc *edesc, struct aead_request *areq,
unsigned int ivsize = crypto_aead_ivsize(aead);
int tbl_off = 0;
int sg_count, ret;
- int sg_link_tbl_len;
+ int elen = 0;
bool sync_needed = false;
struct talitos_private *priv = dev_get_drvdata(dev);
bool is_sec1 = has_ftr_sec1(priv);
@@ -1225,20 +1234,12 @@ static int ipsec_esp(struct talitos_edesc *edesc, struct aead_request *areq,
* extent is bytes of HMAC postpended to ciphertext,
* typically 12 for ipsec
*/
- to_talitos_ptr_len(&desc->ptr[4], cryptlen, is_sec1);
- to_talitos_ptr_ext_set(&desc->ptr[4], 0, is_sec1);
+ if ((desc->hdr & DESC_HDR_TYPE_IPSEC_ESP) &&
+ (desc->hdr & DESC_HDR_MODE1_MDEU_CICV))
+ elen = authsize;
- sg_link_tbl_len = cryptlen;
-
- if (desc->hdr & DESC_HDR_TYPE_IPSEC_ESP) {
- to_talitos_ptr_ext_set(&desc->ptr[4], authsize, is_sec1);
-
- if (edesc->desc.hdr & DESC_HDR_MODE1_MDEU_CICV)
- sg_link_tbl_len += authsize;
- }
-
- ret = talitos_sg_map(dev, areq->src, sg_link_tbl_len, edesc,
- &desc->ptr[4], sg_count, areq->assoclen, tbl_off);
+ ret = talitos_sg_map_ext(dev, areq->src, cryptlen, edesc, &desc->ptr[4],
+ sg_count, areq->assoclen, tbl_off, elen);
if (ret > 1) {
tbl_off += ret;
diff --git a/drivers/crypto/vmx/aes.c b/drivers/crypto/vmx/aes.c
index 022c7ab..b0cd5af 100644
--- a/drivers/crypto/vmx/aes.c
+++ b/drivers/crypto/vmx/aes.c
@@ -53,8 +53,6 @@ static int p8_aes_init(struct crypto_tfm *tfm)
alg, PTR_ERR(fallback));
return PTR_ERR(fallback);
}
- printk(KERN_INFO "Using '%s' as fallback implementation.\n",
- crypto_tfm_alg_driver_name((struct crypto_tfm *) fallback));
crypto_cipher_set_flags(fallback,
crypto_cipher_get_flags((struct
diff --git a/drivers/crypto/vmx/aes_cbc.c b/drivers/crypto/vmx/aes_cbc.c
index 94ad5c0..92e1163 100644
--- a/drivers/crypto/vmx/aes_cbc.c
+++ b/drivers/crypto/vmx/aes_cbc.c
@@ -55,8 +55,6 @@ static int p8_aes_cbc_init(struct crypto_tfm *tfm)
alg, PTR_ERR(fallback));
return PTR_ERR(fallback);
}
- printk(KERN_INFO "Using '%s' as fallback implementation.\n",
- crypto_tfm_alg_driver_name((struct crypto_tfm *) fallback));
crypto_blkcipher_set_flags(
fallback,
@@ -113,24 +111,23 @@ static int p8_aes_cbc_encrypt(struct blkcipher_desc *desc,
ret = crypto_blkcipher_encrypt(&fallback_desc, dst, src,
nbytes);
} else {
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
-
blkcipher_walk_init(&walk, dst, src, nbytes);
ret = blkcipher_walk_virt(desc, &walk);
while ((nbytes = walk.nbytes)) {
+ preempt_disable();
+ pagefault_disable();
+ enable_kernel_vsx();
aes_p8_cbc_encrypt(walk.src.virt.addr,
walk.dst.virt.addr,
nbytes & AES_BLOCK_MASK,
&ctx->enc_key, walk.iv, 1);
+ disable_kernel_vsx();
+ pagefault_enable();
+ preempt_enable();
+
nbytes &= AES_BLOCK_SIZE - 1;
ret = blkcipher_walk_done(desc, &walk, nbytes);
}
-
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
}
return ret;
@@ -154,24 +151,23 @@ static int p8_aes_cbc_decrypt(struct blkcipher_desc *desc,
ret = crypto_blkcipher_decrypt(&fallback_desc, dst, src,
nbytes);
} else {
- preempt_disable();
- pagefault_disable();
- enable_kernel_vsx();
-
blkcipher_walk_init(&walk, dst, src, nbytes);
ret = blkcipher_walk_virt(desc, &walk);
while ((nbytes = walk.nbytes)) {
+ preempt_disable();
+ pagefault_disable();
+ enable_kernel_vsx();
aes_p8_cbc_encrypt(walk.src.virt.addr,
walk.dst.virt.addr,
nbytes & AES_BLOCK_MASK,
&ctx->dec_key, walk.iv, 0);
+ disable_kernel_vsx();
+ pagefault_enable();
+ preempt_enable();
+
nbytes &= AES_BLOCK_SIZE - 1;
ret = blkcipher_walk_done(desc, &walk, nbytes);
}
-
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
}
return ret;
diff --git a/drivers/crypto/vmx/aes_ctr.c b/drivers/crypto/vmx/aes_ctr.c
index 7cf6d31..6ef7548 100644
--- a/drivers/crypto/vmx/aes_ctr.c
+++ b/drivers/crypto/vmx/aes_ctr.c
@@ -53,8 +53,6 @@ static int p8_aes_ctr_init(struct crypto_tfm *tfm)
alg, PTR_ERR(fallback));
return PTR_ERR(fallback);
}
- printk(KERN_INFO "Using '%s' as fallback implementation.\n",
- crypto_tfm_alg_driver_name((struct crypto_tfm *) fallback));
crypto_blkcipher_set_flags(
fallback,
diff --git a/drivers/crypto/vmx/aes_xts.c b/drivers/crypto/vmx/aes_xts.c
index 24353ec3..52e7ae0 100644
--- a/drivers/crypto/vmx/aes_xts.c
+++ b/drivers/crypto/vmx/aes_xts.c
@@ -123,32 +123,39 @@ static int p8_aes_xts_crypt(struct blkcipher_desc *desc,
ret = enc ? crypto_blkcipher_encrypt(&fallback_desc, dst, src, nbytes) :
crypto_blkcipher_decrypt(&fallback_desc, dst, src, nbytes);
} else {
+ blkcipher_walk_init(&walk, dst, src, nbytes);
+
+ ret = blkcipher_walk_virt(desc, &walk);
+
preempt_disable();
pagefault_disable();
enable_kernel_vsx();
- blkcipher_walk_init(&walk, dst, src, nbytes);
-
- ret = blkcipher_walk_virt(desc, &walk);
iv = walk.iv;
memset(tweak, 0, AES_BLOCK_SIZE);
aes_p8_encrypt(iv, tweak, &ctx->tweak_key);
+ disable_kernel_vsx();
+ pagefault_enable();
+ preempt_enable();
+
while ((nbytes = walk.nbytes)) {
+ preempt_disable();
+ pagefault_disable();
+ enable_kernel_vsx();
if (enc)
aes_p8_xts_encrypt(walk.src.virt.addr, walk.dst.virt.addr,
nbytes & AES_BLOCK_MASK, &ctx->enc_key, NULL, tweak);
else
aes_p8_xts_decrypt(walk.src.virt.addr, walk.dst.virt.addr,
nbytes & AES_BLOCK_MASK, &ctx->dec_key, NULL, tweak);
+ disable_kernel_vsx();
+ pagefault_enable();
+ preempt_enable();
nbytes &= AES_BLOCK_SIZE - 1;
ret = blkcipher_walk_done(desc, &walk, nbytes);
}
-
- disable_kernel_vsx();
- pagefault_enable();
- preempt_enable();
}
return ret;
}
diff --git a/drivers/crypto/vmx/ghash.c b/drivers/crypto/vmx/ghash.c
index 27a94a1..1c4b5b8 100644
--- a/drivers/crypto/vmx/ghash.c
+++ b/drivers/crypto/vmx/ghash.c
@@ -64,8 +64,6 @@ static int p8_ghash_init_tfm(struct crypto_tfm *tfm)
alg, PTR_ERR(fallback));
return PTR_ERR(fallback);
}
- printk(KERN_INFO "Using '%s' as fallback implementation.\n",
- crypto_tfm_alg_driver_name(crypto_shash_tfm(fallback)));
crypto_shash_set_flags(fallback,
crypto_shash_get_flags((struct crypto_shash
diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index 6b54e02..6b88de2 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -34,6 +34,7 @@
#include <linux/poll.h>
#include <linux/reservation.h>
#include <linux/mm.h>
+#include <linux/mount.h>
#include <uapi/linux/dma-buf.h>
@@ -46,6 +47,25 @@ struct dma_buf_list {
static struct dma_buf_list db_list;
+static const struct dentry_operations dma_buf_dentry_ops = {
+ .d_dname = simple_dname,
+};
+
+static struct vfsmount *dma_buf_mnt;
+
+static struct dentry *dma_buf_fs_mount(struct file_system_type *fs_type,
+ int flags, const char *name, void *data)
+{
+ return mount_pseudo(fs_type, "dmabuf:", NULL, &dma_buf_dentry_ops,
+ DMA_BUF_MAGIC);
+}
+
+static struct file_system_type dma_buf_fs_type = {
+ .name = "dmabuf",
+ .mount = dma_buf_fs_mount,
+ .kill_sb = kill_anon_super,
+};
+
static int dma_buf_release(struct inode *inode, struct file *file)
{
struct dma_buf *dmabuf;
@@ -254,6 +274,36 @@ static unsigned int dma_buf_poll(struct file *file, poll_table *poll)
return events;
}
+static long dma_buf_set_name(struct dma_buf *dmabuf, const char __user *buf)
+{
+ char *name = strndup_user(buf, DMA_BUF_NAME_LEN);
+
+ if (IS_ERR(name))
+ return PTR_ERR(name);
+
+ mutex_lock(&dmabuf->lock);
+ kfree(dmabuf->name);
+ dmabuf->name = name;
+ mutex_unlock(&dmabuf->lock);
+
+ return 0;
+}
+
+static long dma_buf_get_name(struct dma_buf *dmabuf, char __user *buf)
+{
+ const char *name = "";
+ long ret = 0;
+
+ mutex_lock(&dmabuf->lock);
+ if (dmabuf->name)
+ name = dmabuf->name;
+ if (copy_to_user(buf, name, strlen(name) + 1))
+ ret = -EFAULT;
+ mutex_unlock(&dmabuf->lock);
+
+ return ret;
+}
+
static long dma_buf_ioctl(struct file *file,
unsigned int cmd, unsigned long arg)
{
@@ -292,11 +342,32 @@ static long dma_buf_ioctl(struct file *file,
ret = dma_buf_begin_cpu_access(dmabuf, direction);
return ret;
+
+ case DMA_BUF_SET_NAME:
+ return dma_buf_set_name(dmabuf, (const char __user *)arg);
+
+ case DMA_BUF_GET_NAME:
+ return dma_buf_get_name(dmabuf, (char __user *)arg);
+
default:
return -ENOTTY;
}
}
+static void dma_buf_show_fdinfo(struct seq_file *m, struct file *file)
+{
+ struct dma_buf *dmabuf = file->private_data;
+
+ seq_printf(m, "size:\t%zu\n", dmabuf->size);
+ /* Don't count the temporary reference taken inside procfs seq_show */
+ seq_printf(m, "count:\t%ld\n", file_count(dmabuf->file) - 1);
+ seq_printf(m, "exp_name:\t%s\n", dmabuf->exp_name);
+ mutex_lock(&dmabuf->lock);
+ if (dmabuf->name)
+ seq_printf(m, "name:\t%s\n", dmabuf->name);
+ mutex_unlock(&dmabuf->lock);
+}
+
static const struct file_operations dma_buf_fops = {
.release = dma_buf_release,
.mmap = dma_buf_mmap_internal,
@@ -306,6 +377,7 @@ static const struct file_operations dma_buf_fops = {
#ifdef CONFIG_COMPAT
.compat_ioctl = dma_buf_ioctl,
#endif
+ .show_fdinfo = dma_buf_show_fdinfo,
};
/*
@@ -316,6 +388,43 @@ static inline int is_dma_buf_file(struct file *file)
return file->f_op == &dma_buf_fops;
}
+static struct file *dma_buf_getfile(struct dma_buf *dmabuf, int flags)
+{
+ static const struct qstr this = QSTR_INIT("dmabuf", 6);
+ struct path path;
+ struct file *file;
+ struct inode *inode = alloc_anon_inode(dma_buf_mnt->mnt_sb);
+
+ if (IS_ERR(inode))
+ return ERR_CAST(inode);
+
+ inode->i_size = dmabuf->size;
+ inode_set_bytes(inode, dmabuf->size);
+
+ path.dentry = d_alloc_pseudo(dma_buf_mnt->mnt_sb, &this);
+ if (!path.dentry) {
+ file = ERR_PTR(-ENOMEM);
+ goto err_d_alloc;
+ }
+ path.mnt = mntget(dma_buf_mnt);
+
+ d_instantiate(path.dentry, inode);
+ file = alloc_file(&path, OPEN_FMODE(flags) | FMODE_LSEEK,
+ &dma_buf_fops);
+ if (IS_ERR(file))
+ goto err_alloc_file;
+ file->f_flags = flags & (O_ACCMODE | O_NONBLOCK);
+ file->private_data = dmabuf;
+
+ return file;
+
+err_alloc_file:
+ path_put(&path);
+err_d_alloc:
+ iput(inode);
+ return file;
+}
+
/**
* dma_buf_export - Creates a new dma_buf, and associates an anon file
* with this buffer, so it can be exported.
@@ -359,6 +468,11 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
if (!try_module_get(exp_info->owner))
return ERR_PTR(-ENOENT);
+ /*
+ * dma_buf struct must be allocated with kzalloc to zero out all the
+ * fields. Otherwise dmabuf->name field may leak kernel data when the
+ * name is unspecified.
+ */
dmabuf = kzalloc(alloc_size, GFP_KERNEL);
if (!dmabuf) {
ret = -ENOMEM;
@@ -380,14 +494,11 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
}
dmabuf->resv = resv;
- file = anon_inode_getfile("dmabuf", &dma_buf_fops, dmabuf,
- exp_info->flags);
+ file = dma_buf_getfile(dmabuf, exp_info->flags);
if (IS_ERR(file)) {
ret = PTR_ERR(file);
goto err_dmabuf;
}
-
- file->f_mode |= FMODE_LSEEK;
dmabuf->file = file;
mutex_init(&dmabuf->lock);
@@ -551,7 +662,7 @@ EXPORT_SYMBOL_GPL(dma_buf_detach);
struct sg_table *dma_buf_map_attachment(struct dma_buf_attachment *attach,
enum dma_data_direction direction)
{
- struct sg_table *sg_table = ERR_PTR(-EINVAL);
+ struct sg_table *sg_table;
might_sleep();
@@ -872,6 +983,8 @@ static int dma_buf_debug_show(struct seq_file *s, void *unused)
seq_puts(s, "\nDma-buf Objects:\n");
seq_puts(s, "size\tflags\tmode\tcount\texp_name\n");
+ seq_printf(s, "%-8s\t%-8s\t%-8s\t%-8s\texp_name\t%-8s\n",
+ "size", "flags", "mode", "count", "ino");
list_for_each_entry(buf_obj, &db_list.head, list_node) {
ret = mutex_lock_interruptible(&buf_obj->lock);
@@ -882,11 +995,13 @@ static int dma_buf_debug_show(struct seq_file *s, void *unused)
continue;
}
- seq_printf(s, "%08zu\t%08x\t%08x\t%08ld\t%s\n",
+ seq_printf(s, "%08zu\t%08x\t%08x\t%08ld\t%s\t%08lu\t%s\n",
buf_obj->size,
buf_obj->file->f_flags, buf_obj->file->f_mode,
file_count(buf_obj->file),
- buf_obj->exp_name);
+ buf_obj->exp_name,
+ file_inode(buf_obj->file)->i_ino,
+ buf_obj->name ? : "");
seq_puts(s, "\tAttached Devices:\n");
attach_count = 0;
@@ -966,6 +1081,10 @@ static inline void dma_buf_uninit_debugfs(void)
static int __init dma_buf_init(void)
{
+ dma_buf_mnt = kern_mount(&dma_buf_fs_type);
+ if (IS_ERR(dma_buf_mnt))
+ return PTR_ERR(dma_buf_mnt);
+
mutex_init(&db_list.lock);
INIT_LIST_HEAD(&db_list.head);
dma_buf_init_debugfs();
@@ -976,5 +1095,6 @@ subsys_initcall(dma_buf_init);
static void __exit dma_buf_deinit(void)
{
dma_buf_uninit_debugfs();
+ kern_unmount(dma_buf_mnt);
}
__exitcall(dma_buf_deinit);
diff --git a/drivers/dma/at_hdmac.c b/drivers/dma/at_hdmac.c
index e2cec5b..a32cd71 100644
--- a/drivers/dma/at_hdmac.c
+++ b/drivers/dma/at_hdmac.c
@@ -1774,6 +1774,12 @@ static void atc_free_chan_resources(struct dma_chan *chan)
atchan->descs_allocated = 0;
atchan->status = 0;
+ /*
+ * Free atslave allocated in at_dma_xlate()
+ */
+ kfree(chan->private);
+ chan->private = NULL;
+
dev_vdbg(chan2dev(chan), "free_chan_resources: done\n");
}
@@ -1808,7 +1814,7 @@ static struct dma_chan *at_dma_xlate(struct of_phandle_args *dma_spec,
dma_cap_zero(mask);
dma_cap_set(DMA_SLAVE, mask);
- atslave = devm_kzalloc(&dmac_pdev->dev, sizeof(*atslave), GFP_KERNEL);
+ atslave = kzalloc(sizeof(*atslave), GFP_KERNEL);
if (!atslave)
return NULL;
@@ -2139,6 +2145,8 @@ static int at_dma_remove(struct platform_device *pdev)
struct resource *io;
at_dma_off(atdma);
+ if (pdev->dev.of_node)
+ of_dma_controller_free(pdev->dev.of_node);
dma_async_device_unregister(&atdma->dma_common);
dma_pool_destroy(atdma->memset_pool);
diff --git a/drivers/dma/cppi41.c b/drivers/dma/cppi41.c
index 55c1782..2ceb5a2 100644
--- a/drivers/dma/cppi41.c
+++ b/drivers/dma/cppi41.c
@@ -717,8 +717,22 @@ static int cppi41_stop_chan(struct dma_chan *chan)
desc_phys = lower_32_bits(c->desc_phys);
desc_num = (desc_phys - cdd->descs_phys) / sizeof(struct cppi41_desc);
- if (!cdd->chan_busy[desc_num])
+ if (!cdd->chan_busy[desc_num]) {
+ struct cppi41_channel *cc, *_ct;
+
+ /*
+ * channels might still be in the pendling list if
+ * cppi41_dma_issue_pending() is called after
+ * cppi41_runtime_suspend() is called
+ */
+ list_for_each_entry_safe(cc, _ct, &cdd->pending, node) {
+ if (cc != c)
+ continue;
+ list_del(&cc->node);
+ break;
+ }
return 0;
+ }
ret = cppi41_tear_down_chan(c);
if (ret)
diff --git a/drivers/dma/dma-jz4780.c b/drivers/dma/dma-jz4780.c
index 7373b7a..803cfb4 100644
--- a/drivers/dma/dma-jz4780.c
+++ b/drivers/dma/dma-jz4780.c
@@ -754,6 +754,11 @@ static int jz4780_dma_probe(struct platform_device *pdev)
struct resource *res;
int i, ret;
+ if (!dev->of_node) {
+ dev_err(dev, "This driver must be probed from devicetree\n");
+ return -EINVAL;
+ }
+
jzdma = devm_kzalloc(dev, sizeof(*jzdma), GFP_KERNEL);
if (!jzdma)
return -ENOMEM;
diff --git a/drivers/dma/ioat/init.c b/drivers/dma/ioat/init.c
index 84eb83e..d139706 100644
--- a/drivers/dma/ioat/init.c
+++ b/drivers/dma/ioat/init.c
@@ -1210,8 +1210,15 @@ static void ioat_shutdown(struct pci_dev *pdev)
spin_lock_bh(&ioat_chan->prep_lock);
set_bit(IOAT_CHAN_DOWN, &ioat_chan->state);
- del_timer_sync(&ioat_chan->timer);
spin_unlock_bh(&ioat_chan->prep_lock);
+ /*
+ * Synchronization rule for del_timer_sync():
+ * - The caller must not hold locks which would prevent
+ * completion of the timer's handler.
+ * So prep_lock cannot be held before calling it.
+ */
+ del_timer_sync(&ioat_chan->timer);
+
/* this should quiesce then reset */
ioat_reset_hw(ioat_chan);
}
diff --git a/drivers/dma/k3dma.c b/drivers/dma/k3dma.c
index aabcb79..cd7f67b 100644
--- a/drivers/dma/k3dma.c
+++ b/drivers/dma/k3dma.c
@@ -792,7 +792,7 @@ static struct dma_chan *k3_of_dma_simple_xlate(struct of_phandle_args *dma_spec,
struct k3_dma_dev *d = ofdma->of_dma_data;
unsigned int request = dma_spec->args[0];
- if (request > d->dma_requests)
+ if (request >= d->dma_requests)
return NULL;
return dma_get_slave_channel(&(d->chans[request].vc.chan));
diff --git a/drivers/dma/mv_xor_v2.c b/drivers/dma/mv_xor_v2.c
index f3e211f..be1f5c2 100644
--- a/drivers/dma/mv_xor_v2.c
+++ b/drivers/dma/mv_xor_v2.c
@@ -152,6 +152,7 @@ struct mv_xor_v2_device {
void __iomem *dma_base;
void __iomem *glob_base;
struct clk *clk;
+ struct clk *reg_clk;
struct tasklet_struct irq_tasklet;
struct list_head free_sw_desc;
struct dma_device dmadev;
@@ -697,13 +698,26 @@ static int mv_xor_v2_probe(struct platform_device *pdev)
if (ret)
return ret;
+ xor_dev->reg_clk = devm_clk_get(&pdev->dev, "reg");
+ if (PTR_ERR(xor_dev->reg_clk) != -ENOENT) {
+ if (!IS_ERR(xor_dev->reg_clk)) {
+ ret = clk_prepare_enable(xor_dev->reg_clk);
+ if (ret)
+ return ret;
+ } else {
+ return PTR_ERR(xor_dev->reg_clk);
+ }
+ }
+
xor_dev->clk = devm_clk_get(&pdev->dev, NULL);
- if (IS_ERR(xor_dev->clk) && PTR_ERR(xor_dev->clk) == -EPROBE_DEFER)
- return -EPROBE_DEFER;
+ if (IS_ERR(xor_dev->clk) && PTR_ERR(xor_dev->clk) == -EPROBE_DEFER) {
+ ret = EPROBE_DEFER;
+ goto disable_reg_clk;
+ }
if (!IS_ERR(xor_dev->clk)) {
ret = clk_prepare_enable(xor_dev->clk);
if (ret)
- return ret;
+ goto disable_reg_clk;
}
ret = platform_msi_domain_alloc_irqs(&pdev->dev, 1,
@@ -812,8 +826,9 @@ static int mv_xor_v2_probe(struct platform_device *pdev)
free_msi_irqs:
platform_msi_domain_free_irqs(&pdev->dev);
disable_clk:
- if (!IS_ERR(xor_dev->clk))
- clk_disable_unprepare(xor_dev->clk);
+ clk_disable_unprepare(xor_dev->clk);
+disable_reg_clk:
+ clk_disable_unprepare(xor_dev->reg_clk);
return ret;
}
@@ -829,6 +844,8 @@ static int mv_xor_v2_remove(struct platform_device *pdev)
platform_msi_domain_free_irqs(&pdev->dev);
+ tasklet_kill(&xor_dev->irq_tasklet);
+
clk_disable_unprepare(xor_dev->clk);
return 0;
diff --git a/drivers/dma/pl330.c b/drivers/dma/pl330.c
index fb2e747..6d7e3cd 100644
--- a/drivers/dma/pl330.c
+++ b/drivers/dma/pl330.c
@@ -1570,7 +1570,7 @@ static void pl330_dotask(unsigned long data)
/* Returns 1 if state was updated, 0 otherwise */
static int pl330_update(struct pl330_dmac *pl330)
{
- struct dma_pl330_desc *descdone, *tmp;
+ struct dma_pl330_desc *descdone;
unsigned long flags;
void __iomem *regs;
u32 val;
@@ -1648,7 +1648,9 @@ static int pl330_update(struct pl330_dmac *pl330)
}
/* Now that we are in no hurry, do the callbacks */
- list_for_each_entry_safe(descdone, tmp, &pl330->req_done, rqd) {
+ while (!list_empty(&pl330->req_done)) {
+ descdone = list_first_entry(&pl330->req_done,
+ struct dma_pl330_desc, rqd);
list_del(&descdone->rqd);
spin_unlock_irqrestore(&pl330->lock, flags);
dma_pl330_rqcb(descdone, PL330_ERR_NONE);
@@ -2165,13 +2167,14 @@ static int pl330_terminate_all(struct dma_chan *chan)
pm_runtime_get_sync(pl330->ddma.dev);
spin_lock_irqsave(&pch->lock, flags);
+
spin_lock(&pl330->lock);
_stop(pch->thread);
- spin_unlock(&pl330->lock);
-
pch->thread->req[0].desc = NULL;
pch->thread->req[1].desc = NULL;
pch->thread->req_running = -1;
+ spin_unlock(&pl330->lock);
+
power_down = pch->active;
pch->active = false;
@@ -2949,7 +2952,7 @@ pl330_probe(struct amba_device *adev, const struct amba_id *id)
pd->src_addr_widths = PL330_DMA_BUSWIDTHS;
pd->dst_addr_widths = PL330_DMA_BUSWIDTHS;
pd->directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
- pd->residue_granularity = DMA_RESIDUE_GRANULARITY_SEGMENT;
+ pd->residue_granularity = DMA_RESIDUE_GRANULARITY_BURST;
pd->max_burst = ((pl330->quirks & PL330_QUIRK_BROKEN_NO_FLUSHP) ?
1 : PL330_MAX_BURST);
diff --git a/drivers/dma/qcom/bam_dma.c b/drivers/dma/qcom/bam_dma.c
index 03c4eb3..6497f52 100644
--- a/drivers/dma/qcom/bam_dma.c
+++ b/drivers/dma/qcom/bam_dma.c
@@ -387,6 +387,7 @@ struct bam_device {
struct device_dma_parameters dma_parms;
struct bam_chan *channels;
u32 num_channels;
+ u32 num_ees;
/* execution environment ID, from DT */
u32 ee;
@@ -1076,15 +1077,19 @@ static int bam_init(struct bam_device *bdev)
u32 val;
/* read revision and configuration information */
- val = readl_relaxed(bam_addr(bdev, 0, BAM_REVISION)) >> NUM_EES_SHIFT;
- val &= NUM_EES_MASK;
+ if (!bdev->num_ees) {
+ val = readl_relaxed(bam_addr(bdev, 0, BAM_REVISION));
+ bdev->num_ees = (val >> NUM_EES_SHIFT) & NUM_EES_MASK;
+ }
/* check that configured EE is within range */
- if (bdev->ee >= val)
+ if (bdev->ee >= bdev->num_ees)
return -EINVAL;
- val = readl_relaxed(bam_addr(bdev, 0, BAM_NUM_PIPES));
- bdev->num_channels = val & BAM_NUM_PIPES_MASK;
+ if (!bdev->num_channels) {
+ val = readl_relaxed(bam_addr(bdev, 0, BAM_NUM_PIPES));
+ bdev->num_channels = val & BAM_NUM_PIPES_MASK;
+ }
if (bdev->controlled_remotely)
return 0;
@@ -1179,6 +1184,18 @@ static int bam_dma_probe(struct platform_device *pdev)
bdev->controlled_remotely = of_property_read_bool(pdev->dev.of_node,
"qcom,controlled-remotely");
+ if (bdev->controlled_remotely) {
+ ret = of_property_read_u32(pdev->dev.of_node, "num-channels",
+ &bdev->num_channels);
+ if (ret)
+ dev_err(bdev->dev, "num-channels unspecified in dt\n");
+
+ ret = of_property_read_u32(pdev->dev.of_node, "qcom,num-ees",
+ &bdev->num_ees);
+ if (ret)
+ dev_err(bdev->dev, "num-ees unspecified in dt\n");
+ }
+
bdev->bamclk = devm_clk_get(bdev->dev, "bam_clk");
if (IS_ERR(bdev->bamclk))
return PTR_ERR(bdev->bamclk);
diff --git a/drivers/dma/sh/rcar-dmac.c b/drivers/dma/sh/rcar-dmac.c
index 4c357d4..d032032 100644
--- a/drivers/dma/sh/rcar-dmac.c
+++ b/drivers/dma/sh/rcar-dmac.c
@@ -870,7 +870,7 @@ rcar_dmac_chan_prep_sg(struct rcar_dmac_chan *chan, struct scatterlist *sgl,
rcar_dmac_chan_configure_desc(chan, desc);
- max_chunk_size = (RCAR_DMATCR_MASK + 1) << desc->xfer_shift;
+ max_chunk_size = RCAR_DMATCR_MASK << desc->xfer_shift;
/*
* Allocate and fill the transfer chunk descriptors. We own the only
@@ -1246,8 +1246,17 @@ static unsigned int rcar_dmac_chan_get_residue(struct rcar_dmac_chan *chan,
* If the cookie doesn't correspond to the currently running transfer
* then the descriptor hasn't been processed yet, and the residue is
* equal to the full descriptor size.
+ * Also, a client driver is possible to call this function before
+ * rcar_dmac_isr_channel_thread() runs. In this case, the "desc.running"
+ * will be the next descriptor, and the done list will appear. So, if
+ * the argument cookie matches the done list's cookie, we can assume
+ * the residue is zero.
*/
if (cookie != desc->async_tx.cookie) {
+ list_for_each_entry(desc, &chan->desc.done, node) {
+ if (cookie == desc->async_tx.cookie)
+ return 0;
+ }
list_for_each_entry(desc, &chan->desc.pending, node) {
if (cookie == desc->async_tx.cookie)
return desc->size;
diff --git a/drivers/edac/altera_edac.c b/drivers/edac/altera_edac.c
index 61262a7..b0bd0f6 100644
--- a/drivers/edac/altera_edac.c
+++ b/drivers/edac/altera_edac.c
@@ -1111,7 +1111,7 @@ static void *ocram_alloc_mem(size_t size, void **other)
static void ocram_free_mem(void *p, size_t size, void *other)
{
- gen_pool_free((struct gen_pool *)other, (u32)p, size);
+ gen_pool_free((struct gen_pool *)other, (unsigned long)p, size);
}
static const struct edac_device_prv_data ocramecc_data = {
diff --git a/drivers/edac/edac_mc_sysfs.c b/drivers/edac/edac_mc_sysfs.c
index bea71fb..7335a86 100644
--- a/drivers/edac/edac_mc_sysfs.c
+++ b/drivers/edac/edac_mc_sysfs.c
@@ -1059,14 +1059,14 @@ int __init edac_mc_sysfs_init(void)
err = device_add(mci_pdev);
if (err < 0)
- goto out_dev_free;
+ goto out_put_device;
edac_dbg(0, "device %s created\n", dev_name(mci_pdev));
return 0;
- out_dev_free:
- kfree(mci_pdev);
+ out_put_device:
+ put_device(mci_pdev);
out:
return err;
}
diff --git a/drivers/edac/i7core_edac.c b/drivers/edac/i7core_edac.c
index 8a68a5e..f95d5b9 100644
--- a/drivers/edac/i7core_edac.c
+++ b/drivers/edac/i7core_edac.c
@@ -1177,15 +1177,14 @@ static int i7core_create_sysfs_devices(struct mem_ctl_info *mci)
rc = device_add(pvt->addrmatch_dev);
if (rc < 0)
- return rc;
+ goto err_put_addrmatch;
if (!pvt->is_registered) {
pvt->chancounts_dev = kzalloc(sizeof(*pvt->chancounts_dev),
GFP_KERNEL);
if (!pvt->chancounts_dev) {
- put_device(pvt->addrmatch_dev);
- device_del(pvt->addrmatch_dev);
- return -ENOMEM;
+ rc = -ENOMEM;
+ goto err_del_addrmatch;
}
pvt->chancounts_dev->type = &all_channel_counts_type;
@@ -1199,9 +1198,18 @@ static int i7core_create_sysfs_devices(struct mem_ctl_info *mci)
rc = device_add(pvt->chancounts_dev);
if (rc < 0)
- return rc;
+ goto err_put_chancounts;
}
return 0;
+
+err_put_chancounts:
+ put_device(pvt->chancounts_dev);
+err_del_addrmatch:
+ device_del(pvt->addrmatch_dev);
+err_put_addrmatch:
+ put_device(pvt->addrmatch_dev);
+
+ return rc;
}
static void i7core_delete_sysfs_devices(struct mem_ctl_info *mci)
@@ -1211,11 +1219,11 @@ static void i7core_delete_sysfs_devices(struct mem_ctl_info *mci)
edac_dbg(1, "\n");
if (!pvt->is_registered) {
- put_device(pvt->chancounts_dev);
device_del(pvt->chancounts_dev);
+ put_device(pvt->chancounts_dev);
}
- put_device(pvt->addrmatch_dev);
device_del(pvt->addrmatch_dev);
+ put_device(pvt->addrmatch_dev);
}
/****************************************************************************
@@ -1703,6 +1711,7 @@ static void i7core_mce_output_error(struct mem_ctl_info *mci,
u32 errnum = find_first_bit(&error, 32);
if (uncorrected_error) {
+ core_err_cnt = 1;
if (ripv)
tp_event = HW_EVENT_ERR_FATAL;
else
diff --git a/drivers/edac/kryo3xx_arm64_edac.c b/drivers/edac/kryo3xx_arm64_edac.c
index cf3fdde..d37f8a2 100644
--- a/drivers/edac/kryo3xx_arm64_edac.c
+++ b/drivers/edac/kryo3xx_arm64_edac.c
@@ -318,7 +318,7 @@ static void kryo3xx_check_l3_scu_error(struct edac_device_ctl_info *edev_ctl)
if (l3_is_bus_error(errxstatus)) {
if (edev_ctl->panic_on_ue)
panic("Causing panic due to Bus Error\n");
- return;
+ goto out;
}
if (KRYO3XX_ERRXSTATUS_UE(errxstatus)) {
edac_printk(KERN_CRIT, EDAC_CPU, "Detected L3 uncorrectable error\n");
@@ -332,6 +332,8 @@ static void kryo3xx_check_l3_scu_error(struct edac_device_ctl_info *edev_ctl)
clear_errxstatus_valid(errxstatus);
}
+
+out:
spin_unlock_irqrestore(&local_handler_lock, flags);
}
diff --git a/drivers/edac/sb_edac.c b/drivers/edac/sb_edac.c
index 3c47e63..e939195 100644
--- a/drivers/edac/sb_edac.c
+++ b/drivers/edac/sb_edac.c
@@ -2934,6 +2934,7 @@ static void sbridge_mce_output_error(struct mem_ctl_info *mci,
recoverable = GET_BITFIELD(m->status, 56, 56);
if (uncorrected_error) {
+ core_err_cnt = 1;
if (ripv) {
type = "FATAL";
tp_event = HW_EVENT_ERR_FATAL;
diff --git a/drivers/edac/skx_edac.c b/drivers/edac/skx_edac.c
index 0ff4878..321035a 100644
--- a/drivers/edac/skx_edac.c
+++ b/drivers/edac/skx_edac.c
@@ -606,7 +606,7 @@ static bool skx_sad_decode(struct decoded_addr *res)
break;
case 2:
lchan = (addr >> shift) % 2;
- lchan = (lchan << 1) | ~lchan;
+ lchan = (lchan << 1) | !lchan;
break;
case 3:
lchan = ((addr >> shift) % 2) << 1;
@@ -897,6 +897,7 @@ static void skx_mce_output_error(struct mem_ctl_info *mci,
recoverable = GET_BITFIELD(m->status, 56, 56);
if (uncorrected_error) {
+ core_err_cnt = 1;
if (ripv) {
type = "FATAL";
tp_event = HW_EVENT_ERR_FATAL;
diff --git a/drivers/firewire/ohci.c b/drivers/firewire/ohci.c
index 8bf8926..d731b41 100644
--- a/drivers/firewire/ohci.c
+++ b/drivers/firewire/ohci.c
@@ -1130,7 +1130,13 @@ static int context_add_buffer(struct context *ctx)
return -ENOMEM;
offset = (void *)&desc->buffer - (void *)desc;
- desc->buffer_size = PAGE_SIZE - offset;
+ /*
+ * Some controllers, like JMicron ones, always issue 0x20-byte DMA reads
+ * for descriptors, even 0x10-byte ones. This can cause page faults when
+ * an IOMMU is in use and the oversized read crosses a page boundary.
+ * Work around this by always leaving at least 0x10 bytes of padding.
+ */
+ desc->buffer_size = PAGE_SIZE - offset - 0x10;
desc->buffer_bus = bus_addr + offset;
desc->used = 0;
diff --git a/drivers/firmware/dmi_scan.c b/drivers/firmware/dmi_scan.c
index 88bebe1..42844c3 100644
--- a/drivers/firmware/dmi_scan.c
+++ b/drivers/firmware/dmi_scan.c
@@ -18,7 +18,7 @@ EXPORT_SYMBOL_GPL(dmi_kobj);
* of and an antecedent to, SMBIOS, which stands for System
* Management BIOS. See further: http://www.dmtf.org/standards
*/
-static const char dmi_empty_string[] = " ";
+static const char dmi_empty_string[] = "";
static u32 dmi_ver __initdata;
static u32 dmi_len;
@@ -44,25 +44,21 @@ static int dmi_memdev_nr;
static const char * __init dmi_string_nosave(const struct dmi_header *dm, u8 s)
{
const u8 *bp = ((u8 *) dm) + dm->length;
+ const u8 *nsp;
if (s) {
- s--;
- while (s > 0 && *bp) {
+ while (--s > 0 && *bp)
bp += strlen(bp) + 1;
- s--;
- }
- if (*bp != 0) {
- size_t len = strlen(bp)+1;
- size_t cmp_len = len > 8 ? 8 : len;
-
- if (!memcmp(bp, dmi_empty_string, cmp_len))
- return dmi_empty_string;
+ /* Strings containing only spaces are considered empty */
+ nsp = bp;
+ while (*nsp == ' ')
+ nsp++;
+ if (*nsp != '\0')
return bp;
- }
}
- return "";
+ return dmi_empty_string;
}
static const char * __init dmi_string(const struct dmi_header *dm, u8 s)
diff --git a/drivers/firmware/efi/arm-init.c b/drivers/firmware/efi/arm-init.c
index 8efe130..8ee9177 100644
--- a/drivers/firmware/efi/arm-init.c
+++ b/drivers/firmware/efi/arm-init.c
@@ -250,13 +250,16 @@ void __init efi_init(void)
reserve_regions();
efi_memattr_init();
efi_esrt_init();
- efi_memmap_unmap();
memblock_reserve(params.mmap & PAGE_MASK,
PAGE_ALIGN(params.mmap_size +
(params.mmap & ~PAGE_MASK)));
init_screen_info();
+
+ /* ARM does not permit early mappings to persist across paging_init() */
+ if (IS_ENABLED(CONFIG_ARM))
+ efi_memmap_unmap();
}
static int __init register_gop_device(void)
diff --git a/drivers/firmware/efi/arm-runtime.c b/drivers/firmware/efi/arm-runtime.c
index 6bdf39e..069c5a4 100644
--- a/drivers/firmware/efi/arm-runtime.c
+++ b/drivers/firmware/efi/arm-runtime.c
@@ -123,6 +123,8 @@ static int __init arm_enable_runtime_services(void)
return 0;
}
+ efi_memmap_unmap();
+
if (efi_runtime_disabled()) {
pr_info("EFI runtime services will be disabled.\n");
return 0;
diff --git a/drivers/firmware/efi/esrt.c b/drivers/firmware/efi/esrt.c
index 311c9d0..241dd7c 100644
--- a/drivers/firmware/efi/esrt.c
+++ b/drivers/firmware/efi/esrt.c
@@ -333,7 +333,8 @@ void __init efi_esrt_init(void)
end = esrt_data + size;
pr_info("Reserving ESRT space from %pa to %pa.\n", &esrt_data, &end);
- efi_mem_reserve(esrt_data, esrt_data_size);
+ if (md.type == EFI_BOOT_SERVICES_DATA)
+ efi_mem_reserve(esrt_data, esrt_data_size);
pr_debug("esrt-init: loaded.\n");
err_memunmap:
diff --git a/drivers/firmware/efi/libstub/Makefile b/drivers/firmware/efi/libstub/Makefile
index cf2ed6c..5c281a5 100644
--- a/drivers/firmware/efi/libstub/Makefile
+++ b/drivers/firmware/efi/libstub/Makefile
@@ -11,8 +11,9 @@
-mno-mmx -mno-sse
cflags-$(CONFIG_ARM64) := $(subst -pg,,$(KBUILD_CFLAGS)) -fpie
-cflags-$(CONFIG_ARM) := $(subst -pg,,$(KBUILD_CFLAGS)) -g0 \
- -fno-builtin -fpic -mno-single-pic-base
+cflags-$(CONFIG_ARM) := $(subst -pg,,$(KBUILD_CFLAGS)) \
+ -fno-builtin -fpic \
+ $(call cc-option,-mno-single-pic-base)
cflags-$(CONFIG_EFI_ARMSTUB) += -I$(srctree)/scripts/dtc/libfdt
@@ -62,7 +63,7 @@
extra-$(CONFIG_EFI_ARMSTUB) := $(lib-y)
lib-$(CONFIG_EFI_ARMSTUB) := $(patsubst %.o,%.stub.o,$(lib-y))
-STUBCOPY_FLAGS-y := -R .debug* -R *ksymtab* -R *kcrctab*
+STUBCOPY_RM-y := -R *ksymtab* -R *kcrctab*
STUBCOPY_FLAGS-$(CONFIG_ARM64) += --prefix-alloc-sections=.init \
--prefix-symbols=__efistub_
STUBCOPY_RELOC-$(CONFIG_ARM64) := R_AARCH64_ABS
@@ -70,17 +71,25 @@
$(obj)/%.stub.o: $(obj)/%.o FORCE
$(call if_changed,stubcopy)
+#
+# Strip debug sections and some other sections that may legally contain
+# absolute relocations, so that we can inspect the remaining sections for
+# such relocations. If none are found, regenerate the output object, but
+# this time, use objcopy and leave all sections in place.
+#
quiet_cmd_stubcopy = STUBCPY $@
- cmd_stubcopy = if $(OBJCOPY) $(STUBCOPY_FLAGS-y) $< $@; then \
- $(OBJDUMP) -r $@ | grep $(STUBCOPY_RELOC-y) \
- && (echo >&2 "$@: absolute symbol references not allowed in the EFI stub"; \
- rm -f $@; /bin/false); else /bin/false; fi
+ cmd_stubcopy = if $(STRIP) --strip-debug $(STUBCOPY_RM-y) -o $@ $<; \
+ then if $(OBJDUMP) -r $@ | grep $(STUBCOPY_RELOC-y); \
+ then (echo >&2 "$@: absolute symbol references not allowed in the EFI stub"; \
+ rm -f $@; /bin/false); \
+ else $(OBJCOPY) $(STUBCOPY_FLAGS-y) $< $@; fi \
+ else /bin/false; fi
#
# ARM discards the .data section because it disallows r/w data in the
# decompressor. So move our .data to .data.efistub, which is preserved
# explicitly by the decompressor linker script.
#
-STUBCOPY_FLAGS-$(CONFIG_ARM) += --rename-section .data=.data.efistub \
- -R ___ksymtab+sort -R ___kcrctab+sort
+STUBCOPY_FLAGS-$(CONFIG_ARM) += --rename-section .data=.data.efistub
+STUBCOPY_RM-$(CONFIG_ARM) += -R ___ksymtab+sort -R ___kcrctab+sort
STUBCOPY_RELOC-$(CONFIG_ARM) := R_ARM_ABS
diff --git a/drivers/firmware/efi/libstub/efi-stub-helper.c b/drivers/firmware/efi/libstub/efi-stub-helper.c
index 2db141c..d984332 100644
--- a/drivers/firmware/efi/libstub/efi-stub-helper.c
+++ b/drivers/firmware/efi/libstub/efi-stub-helper.c
@@ -367,6 +367,14 @@ efi_status_t efi_parse_options(char const *cmdline)
__nokaslr = 1;
/*
+ * Currently, the only efi= option we look for is 'nochunk', which
+ * is intended to work around known issues on certain x86 UEFI
+ * versions. So ignore for now on other architectures.
+ */
+ if (!IS_ENABLED(CONFIG_X86))
+ return EFI_SUCCESS;
+
+ /*
* If no EFI parameters were specified on the cmdline we've got
* nothing to do.
*/
@@ -539,7 +547,8 @@ efi_status_t handle_cmdline_files(efi_system_table_t *sys_table_arg,
size = files[j].size;
while (size) {
unsigned long chunksize;
- if (size > __chunk_size)
+
+ if (IS_ENABLED(CONFIG_X86) && size > __chunk_size)
chunksize = __chunk_size;
else
chunksize = size;
diff --git a/drivers/firmware/efi/memmap.c b/drivers/firmware/efi/memmap.c
index 7868644..3fd2b45 100644
--- a/drivers/firmware/efi/memmap.c
+++ b/drivers/firmware/efi/memmap.c
@@ -117,6 +117,9 @@ int __init efi_memmap_init_early(struct efi_memory_map_data *data)
void __init efi_memmap_unmap(void)
{
+ if (!efi_enabled(EFI_MEMMAP))
+ return;
+
if (!efi.memmap.late) {
unsigned long size;
diff --git a/drivers/firmware/qcom/Makefile b/drivers/firmware/qcom/Makefile
index 635f60c..8a275db 100644
--- a/drivers/firmware/qcom/Makefile
+++ b/drivers/firmware/qcom/Makefile
@@ -1 +1,5 @@
obj-$(CONFIG_MSM_TZ_LOG) += tz_log.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-ignored-attributes \
diff --git a/drivers/gpio/gpio-adp5588.c b/drivers/gpio/gpio-adp5588.c
index c0f718b..c85407a 100644
--- a/drivers/gpio/gpio-adp5588.c
+++ b/drivers/gpio/gpio-adp5588.c
@@ -41,6 +41,8 @@ struct adp5588_gpio {
uint8_t int_en[3];
uint8_t irq_mask[3];
uint8_t irq_stat[3];
+ uint8_t int_input_en[3];
+ uint8_t int_lvl_cached[3];
};
static int adp5588_gpio_read(struct i2c_client *client, u8 reg)
@@ -173,12 +175,28 @@ static void adp5588_irq_bus_sync_unlock(struct irq_data *d)
struct adp5588_gpio *dev = irq_data_get_irq_chip_data(d);
int i;
- for (i = 0; i <= ADP5588_BANK(ADP5588_MAXGPIO); i++)
+ for (i = 0; i <= ADP5588_BANK(ADP5588_MAXGPIO); i++) {
+ if (dev->int_input_en[i]) {
+ mutex_lock(&dev->lock);
+ dev->dir[i] &= ~dev->int_input_en[i];
+ dev->int_input_en[i] = 0;
+ adp5588_gpio_write(dev->client, GPIO_DIR1 + i,
+ dev->dir[i]);
+ mutex_unlock(&dev->lock);
+ }
+
+ if (dev->int_lvl_cached[i] != dev->int_lvl[i]) {
+ dev->int_lvl_cached[i] = dev->int_lvl[i];
+ adp5588_gpio_write(dev->client, GPIO_INT_LVL1 + i,
+ dev->int_lvl[i]);
+ }
+
if (dev->int_en[i] ^ dev->irq_mask[i]) {
dev->int_en[i] = dev->irq_mask[i];
adp5588_gpio_write(dev->client, GPIO_INT_EN1 + i,
dev->int_en[i]);
}
+ }
mutex_unlock(&dev->irq_lock);
}
@@ -221,9 +239,7 @@ static int adp5588_irq_set_type(struct irq_data *d, unsigned int type)
else
return -EINVAL;
- adp5588_gpio_direction_input(&dev->gpio_chip, gpio);
- adp5588_gpio_write(dev->client, GPIO_INT_LVL1 + bank,
- dev->int_lvl[bank]);
+ dev->int_input_en[bank] |= bit;
return 0;
}
diff --git a/drivers/gpio/gpio-aspeed.c b/drivers/gpio/gpio-aspeed.c
index 03a5925..a9daf71 100644
--- a/drivers/gpio/gpio-aspeed.c
+++ b/drivers/gpio/gpio-aspeed.c
@@ -256,7 +256,7 @@ static void aspeed_gpio_irq_set_mask(struct irq_data *d, bool set)
if (set)
reg |= bit;
else
- reg &= bit;
+ reg &= ~bit;
iowrite32(reg, addr);
spin_unlock_irqrestore(&gpio->lock, flags);
diff --git a/drivers/gpio/gpio-max7301.c b/drivers/gpio/gpio-max7301.c
index 05813fb..647dfbb 100644
--- a/drivers/gpio/gpio-max7301.c
+++ b/drivers/gpio/gpio-max7301.c
@@ -25,7 +25,7 @@ static int max7301_spi_write(struct device *dev, unsigned int reg,
struct spi_device *spi = to_spi_device(dev);
u16 word = ((reg & 0x7F) << 8) | (val & 0xFF);
- return spi_write(spi, (const u8 *)&word, sizeof(word));
+ return spi_write_then_read(spi, &word, sizeof(word), NULL, 0);
}
/* A read from the MAX7301 means two transfers; here, one message each */
@@ -37,14 +37,8 @@ static int max7301_spi_read(struct device *dev, unsigned int reg)
struct spi_device *spi = to_spi_device(dev);
word = 0x8000 | (reg << 8);
- ret = spi_write(spi, (const u8 *)&word, sizeof(word));
- if (ret)
- return ret;
- /*
- * This relies on the fact, that a transfer with NULL tx_buf shifts out
- * zero bytes (=NOOP for MAX7301)
- */
- ret = spi_read(spi, (u8 *)&word, sizeof(word));
+ ret = spi_write_then_read(spi, &word, sizeof(word), &word,
+ sizeof(word));
if (ret)
return ret;
return word & 0xff;
diff --git a/drivers/gpio/gpio-menz127.c b/drivers/gpio/gpio-menz127.c
index a1210e3..95061d2 100644
--- a/drivers/gpio/gpio-menz127.c
+++ b/drivers/gpio/gpio-menz127.c
@@ -56,9 +56,9 @@ static int men_z127_debounce(struct gpio_chip *gc, unsigned gpio,
rnd = fls(debounce) - 1;
if (rnd && (debounce & BIT(rnd - 1)))
- debounce = round_up(debounce, MEN_Z127_DB_MIN_US);
+ debounce = roundup(debounce, MEN_Z127_DB_MIN_US);
else
- debounce = round_down(debounce, MEN_Z127_DB_MIN_US);
+ debounce = rounddown(debounce, MEN_Z127_DB_MIN_US);
if (debounce > MEN_Z127_DB_MAX_US)
debounce = MEN_Z127_DB_MAX_US;
diff --git a/drivers/gpio/gpio-ml-ioh.c b/drivers/gpio/gpio-ml-ioh.c
index 796a5a4..e9b6074 100644
--- a/drivers/gpio/gpio-ml-ioh.c
+++ b/drivers/gpio/gpio-ml-ioh.c
@@ -495,9 +495,10 @@ static int ioh_gpio_probe(struct pci_dev *pdev,
chip = chip_save;
err_gpiochip_add:
+ chip = chip_save;
while (--i >= 0) {
- chip--;
gpiochip_remove(&chip->gpio);
+ chip++;
}
kfree(chip_save);
diff --git a/drivers/gpio/gpio-msm-smp2p-test.c b/drivers/gpio/gpio-msm-smp2p-test.c
new file mode 100644
index 0000000..c9a5cdf
--- /dev/null
+++ b/drivers/gpio/gpio-msm-smp2p-test.c
@@ -0,0 +1,763 @@
+/* drivers/gpio/gpio-msm-smp2p-test.c
+ *
+ * Copyright (c) 2013-2016, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/of_gpio.h>
+#include <linux/of_irq.h>
+#include <linux/gpio.h>
+#include <linux/debugfs.h>
+#include <linux/completion.h>
+#include <linux/interrupt.h>
+#include <linux/bitmap.h>
+#include "../soc/qcom/smp2p_private.h"
+#include "../soc/qcom/smp2p_test_common.h"
+
+/* Interrupt callback data */
+struct gpio_info {
+ int gpio_base_id;
+ int irq_base_id;
+
+ bool initialized;
+ struct completion cb_completion;
+ int cb_count;
+ DECLARE_BITMAP(triggered_irqs, SMP2P_BITS_PER_ENTRY);
+};
+
+/* GPIO Inbound/Outbound callback info */
+struct gpio_inout {
+ struct gpio_info in;
+ struct gpio_info out;
+};
+
+static struct gpio_inout gpio_info[SMP2P_NUM_PROCS];
+
+/**
+ * Init/reset the callback data.
+ *
+ * @info: Pointer to callback data
+ */
+static void cb_data_reset(struct gpio_info *info)
+{
+ int n;
+
+ if (!info)
+ return;
+
+ if (!info->initialized) {
+ init_completion(&info->cb_completion);
+ info->initialized = true;
+ }
+ info->cb_count = 0;
+
+ for (n = 0; n < SMP2P_BITS_PER_ENTRY; ++n)
+ clear_bit(n, info->triggered_irqs);
+
+ reinit_completion(&info->cb_completion);
+}
+
+static int smp2p_gpio_test_probe(struct platform_device *pdev)
+{
+ int id;
+ int cnt;
+ struct device_node *node = pdev->dev.of_node;
+ struct gpio_info *gpio_info_ptr = NULL;
+
+ /*
+ * NOTE: This does a string-lookup of the GPIO pin name and doesn't
+ * actually directly link to the SMP2P GPIO driver since all
+ * GPIO/Interrupt access must be through standard
+ * Linux GPIO / Interrupt APIs.
+ */
+ if (strcmp("qcom,smp2pgpio_test_smp2p_1_in", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_MODEM_PROC].in;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_1_out", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_MODEM_PROC].out;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_2_in", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_AUDIO_PROC].in;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_2_out", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_AUDIO_PROC].out;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_3_in", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_SENSOR_PROC].in;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_3_out", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_SENSOR_PROC].out;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_4_in", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_WIRELESS_PROC].in;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_4_out", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_WIRELESS_PROC].out;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_5_in", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_CDSP_PROC].in;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_5_out", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_CDSP_PROC].out;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_7_in", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_TZ_PROC].in;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_7_out", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_TZ_PROC].out;
+ } else if (strcmp("qcom,smp2pgpio_test_smp2p_15_in", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_REMOTE_MOCK_PROC].in;
+ } else if (
+ strcmp("qcom,smp2pgpio_test_smp2p_15_out", node->name) == 0) {
+ gpio_info_ptr = &gpio_info[SMP2P_REMOTE_MOCK_PROC].out;
+ } else {
+ pr_err("%s: unable to match device type '%s'\n",
+ __func__, node->name);
+ return -ENODEV;
+ }
+
+ /* retrieve the GPIO and interrupt ID's */
+ cnt = of_gpio_count(node);
+ if (cnt && gpio_info_ptr) {
+ /*
+ * Instead of looping through all 32-bits, we can just get the
+ * first pin to get the base IDs. This saves on the verbosity
+ * of the device tree nodes as well.
+ */
+ id = of_get_gpio(node, 0);
+ if (id == -EPROBE_DEFER)
+ return id;
+ gpio_info_ptr->gpio_base_id = id;
+ gpio_info_ptr->irq_base_id = gpio_to_irq(id);
+ }
+ return 0;
+}
+
+/*
+ * NOTE: Instead of match table and device driver, you may be able to just
+ * call of_find_compatible_node() in your init function.
+ */
+static const struct of_device_id msm_smp2p_match_table[] = {
+ /* modem */
+ {.compatible = "qcom,smp2pgpio_test_smp2p_1_out", },
+ {.compatible = "qcom,smp2pgpio_test_smp2p_1_in", },
+
+ /* audio (adsp) */
+ {.compatible = "qcom,smp2pgpio_test_smp2p_2_out", },
+ {.compatible = "qcom,smp2pgpio_test_smp2p_2_in", },
+
+ /* sensor */
+ {.compatible = "qcom,smp2pgpio_test_smp2p_3_out", },
+ {.compatible = "qcom,smp2pgpio_test_smp2p_3_in", },
+
+ /* wcnss */
+ {.compatible = "qcom,smp2pgpio_test_smp2p_4_out", },
+ {.compatible = "qcom,smp2pgpio_test_smp2p_4_in", },
+
+ /* CDSP */
+ {.compatible = "qcom,smp2pgpio_test_smp2p_5_out", },
+ {.compatible = "qcom,smp2pgpio_test_smp2p_5_in", },
+
+ /* TZ */
+ {.compatible = "qcom,smp2pgpio_test_smp2p_7_out", },
+ {.compatible = "qcom,smp2pgpio_test_smp2p_7_in", },
+
+ /* mock loopback */
+ {.compatible = "qcom,smp2pgpio_test_smp2p_15_out", },
+ {.compatible = "qcom,smp2pgpio_test_smp2p_15_in", },
+ {},
+};
+
+static struct platform_driver smp2p_gpio_driver = {
+ .probe = smp2p_gpio_test_probe,
+ .driver = {
+ .name = "smp2pgpio_test",
+ .owner = THIS_MODULE,
+ .of_match_table = msm_smp2p_match_table,
+ },
+};
+
+/**
+ * smp2p_ut_local_gpio_out - Verify outbound functionality.
+ *
+ * @s: pointer to output file
+ */
+static void smp2p_ut_local_gpio_out(struct seq_file *s)
+{
+ int failed = 0;
+ struct gpio_info *cb_info = &gpio_info[SMP2P_REMOTE_MOCK_PROC].out;
+ int ret;
+ int id;
+ struct msm_smp2p_remote_mock *mock;
+
+ seq_printf(s, "Running %s\n", __func__);
+ do {
+ /* initialize mock edge */
+ ret = smp2p_reset_mock_edge();
+ UT_ASSERT_INT(ret, ==, 0);
+
+ mock = msm_smp2p_get_remote_mock();
+ UT_ASSERT_PTR(mock, !=, NULL);
+
+ mock->rx_interrupt_count = 0;
+ memset(&mock->remote_item, 0,
+ sizeof(struct smp2p_smem_item));
+ smp2p_init_header((struct smp2p_smem *)&mock->remote_item,
+ SMP2P_REMOTE_MOCK_PROC, SMP2P_APPS_PROC,
+ 0, 1);
+ strlcpy(mock->remote_item.entries[0].name, "smp2p",
+ SMP2P_MAX_ENTRY_NAME);
+ SMP2P_SET_ENT_VALID(
+ &mock->remote_item.header, valid_total_ent, 1);
+ msm_smp2p_set_remote_mock_exists(true);
+ mock->tx_interrupt();
+
+ /* open GPIO entry */
+ smp2p_gpio_open_test_entry("smp2p",
+ SMP2P_REMOTE_MOCK_PROC, true);
+
+ /* verify set/get functions */
+ UT_ASSERT_INT(0, <, cb_info->gpio_base_id);
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY && !failed; ++id) {
+ int pin = cb_info->gpio_base_id + id;
+
+ mock->rx_interrupt_count = 0;
+ gpio_set_value(pin, 1);
+ UT_ASSERT_INT(1, ==, mock->rx_interrupt_count);
+ UT_ASSERT_INT(1, ==, gpio_get_value(pin));
+
+ gpio_set_value(pin, 0);
+ UT_ASSERT_INT(2, ==, mock->rx_interrupt_count);
+ UT_ASSERT_INT(0, ==, gpio_get_value(pin));
+ }
+ if (failed)
+ break;
+
+ seq_puts(s, "\tOK\n");
+ } while (0);
+
+ if (failed) {
+ pr_err("%s: Failed\n", __func__);
+ seq_puts(s, "\tFailed\n");
+ }
+
+ smp2p_gpio_open_test_entry("smp2p",
+ SMP2P_REMOTE_MOCK_PROC, false);
+}
+
+/**
+ * smp2p_gpio_irq - Interrupt handler for inbound entries.
+ *
+ * @irq: Virtual IRQ being triggered
+ * @data: Cookie data (struct gpio_info * in this case)
+ * @returns: Number of bytes written
+ */
+static irqreturn_t smp2p_gpio_irq(int irq, void *data)
+{
+ struct gpio_info *gpio_ptr = (struct gpio_info *)data;
+ int offset;
+
+ if (!gpio_ptr) {
+ pr_err("%s: gpio_ptr is NULL for irq %d\n", __func__, irq);
+ return IRQ_HANDLED;
+ }
+
+ offset = irq - gpio_ptr->irq_base_id;
+ if (offset >= 0 && offset < SMP2P_BITS_PER_ENTRY)
+ set_bit(offset, gpio_ptr->triggered_irqs);
+ else
+ pr_err("%s: invalid irq offset base %d; irq %d\n",
+ __func__, gpio_ptr->irq_base_id, irq);
+
+ ++gpio_ptr->cb_count;
+ complete(&gpio_ptr->cb_completion);
+ return IRQ_HANDLED;
+}
+
+/**
+ * smp2p_ut_local_gpio_in - Verify inbound functionality.
+ *
+ * @s: pointer to output file
+ */
+static void smp2p_ut_local_gpio_in(struct seq_file *s)
+{
+ int failed = 0;
+ struct gpio_info *cb_info = &gpio_info[SMP2P_REMOTE_MOCK_PROC].in;
+ int id;
+ int ret;
+ int virq;
+ struct msm_smp2p_remote_mock *mock;
+
+ seq_printf(s, "Running %s\n", __func__);
+
+ cb_data_reset(cb_info);
+ do {
+ /* initialize mock edge */
+ ret = smp2p_reset_mock_edge();
+ UT_ASSERT_INT(ret, ==, 0);
+
+ mock = msm_smp2p_get_remote_mock();
+ UT_ASSERT_PTR(mock, !=, NULL);
+
+ mock->rx_interrupt_count = 0;
+ memset(&mock->remote_item, 0,
+ sizeof(struct smp2p_smem_item));
+ smp2p_init_header((struct smp2p_smem *)&mock->remote_item,
+ SMP2P_REMOTE_MOCK_PROC, SMP2P_APPS_PROC,
+ 0, 1);
+ strlcpy(mock->remote_item.entries[0].name, "smp2p",
+ SMP2P_MAX_ENTRY_NAME);
+ SMP2P_SET_ENT_VALID(
+ &mock->remote_item.header, valid_total_ent, 1);
+ msm_smp2p_set_remote_mock_exists(true);
+ mock->tx_interrupt();
+
+ smp2p_gpio_open_test_entry("smp2p",
+ SMP2P_REMOTE_MOCK_PROC, true);
+
+ /* verify set/get functions locally */
+ UT_ASSERT_INT(0, <, cb_info->gpio_base_id);
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY && !failed; ++id) {
+ int pin;
+ int current_value;
+
+ /* verify pin value cannot be set */
+ pin = cb_info->gpio_base_id + id;
+ current_value = gpio_get_value(pin);
+
+ gpio_set_value(pin, 0);
+ UT_ASSERT_INT(current_value, ==, gpio_get_value(pin));
+ gpio_set_value(pin, 1);
+ UT_ASSERT_INT(current_value, ==, gpio_get_value(pin));
+
+ /* verify no interrupts */
+ UT_ASSERT_INT(0, ==, cb_info->cb_count);
+ }
+ if (failed)
+ break;
+
+ /* register for interrupts */
+ UT_ASSERT_INT(0, <, cb_info->irq_base_id);
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY && !failed; ++id) {
+ virq = cb_info->irq_base_id + id;
+ UT_ASSERT_PTR(NULL, !=, irq_to_desc(virq));
+ ret = request_irq(virq,
+ smp2p_gpio_irq, IRQF_TRIGGER_RISING,
+ "smp2p_test", cb_info);
+ UT_ASSERT_INT(0, ==, ret);
+ }
+ if (failed)
+ break;
+
+ /* verify both rising and falling edge interrupts */
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY && !failed; ++id) {
+ virq = cb_info->irq_base_id + id;
+ irq_set_irq_type(virq, IRQ_TYPE_EDGE_BOTH);
+ cb_data_reset(cb_info);
+
+ /* verify rising-edge interrupt */
+ mock->remote_item.entries[0].entry = 1 << id;
+ mock->tx_interrupt();
+ UT_ASSERT_INT(cb_info->cb_count, ==, 1);
+ UT_ASSERT_INT(0, <,
+ test_bit(id, cb_info->triggered_irqs));
+ test_bit(id, cb_info->triggered_irqs);
+
+ /* verify falling-edge interrupt */
+ mock->remote_item.entries[0].entry = 0;
+ mock->tx_interrupt();
+ UT_ASSERT_INT(cb_info->cb_count, ==, 2);
+ UT_ASSERT_INT(0, <,
+ test_bit(id, cb_info->triggered_irqs));
+ }
+ if (failed)
+ break;
+
+ /* verify rising-edge interrupts */
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY && !failed; ++id) {
+ virq = cb_info->irq_base_id + id;
+ irq_set_irq_type(virq, IRQ_TYPE_EDGE_RISING);
+ cb_data_reset(cb_info);
+
+ /* verify only rising-edge interrupt is triggered */
+ mock->remote_item.entries[0].entry = 1 << id;
+ mock->tx_interrupt();
+ UT_ASSERT_INT(cb_info->cb_count, ==, 1);
+ UT_ASSERT_INT(0, <,
+ test_bit(id, cb_info->triggered_irqs));
+ test_bit(id, cb_info->triggered_irqs);
+
+ mock->remote_item.entries[0].entry = 0;
+ mock->tx_interrupt();
+ UT_ASSERT_INT(cb_info->cb_count, ==, 1);
+ UT_ASSERT_INT(0, <,
+ test_bit(id, cb_info->triggered_irqs));
+ }
+ if (failed)
+ break;
+
+ /* verify falling-edge interrupts */
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY && !failed; ++id) {
+ virq = cb_info->irq_base_id + id;
+ irq_set_irq_type(virq, IRQ_TYPE_EDGE_FALLING);
+ cb_data_reset(cb_info);
+
+ /* verify only rising-edge interrupt is triggered */
+ mock->remote_item.entries[0].entry = 1 << id;
+ mock->tx_interrupt();
+ UT_ASSERT_INT(cb_info->cb_count, ==, 0);
+ UT_ASSERT_INT(0, ==,
+ test_bit(id, cb_info->triggered_irqs));
+
+ mock->remote_item.entries[0].entry = 0;
+ mock->tx_interrupt();
+ UT_ASSERT_INT(cb_info->cb_count, ==, 1);
+ UT_ASSERT_INT(0, <,
+ test_bit(id, cb_info->triggered_irqs));
+ }
+ if (failed)
+ break;
+
+ seq_puts(s, "\tOK\n");
+ } while (0);
+
+ if (failed) {
+ pr_err("%s: Failed\n", __func__);
+ seq_puts(s, "\tFailed\n");
+ }
+
+ /* unregister for interrupts */
+ if (cb_info->irq_base_id) {
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY; ++id)
+ free_irq(cb_info->irq_base_id + id, cb_info);
+ }
+
+ smp2p_gpio_open_test_entry("smp2p",
+ SMP2P_REMOTE_MOCK_PROC, false);
+}
+
+/**
+ * smp2p_ut_local_gpio_in_update_open - Verify combined open/update.
+ *
+ * @s: pointer to output file
+ *
+ * If the remote side updates the SMP2P bits and sends before negotiation is
+ * complete, then the UPDATE event will have to be delayed until negotiation is
+ * complete. This should result in both the OPEN and UPDATE events coming in
+ * right after each other and the behavior should be transparent to the clients
+ * of SMP2P GPIO.
+ */
+static void smp2p_ut_local_gpio_in_update_open(struct seq_file *s)
+{
+ int failed = 0;
+ struct gpio_info *cb_info = &gpio_info[SMP2P_REMOTE_MOCK_PROC].in;
+ int id;
+ int ret;
+ int virq;
+ struct msm_smp2p_remote_mock *mock;
+
+ seq_printf(s, "Running %s\n", __func__);
+
+ cb_data_reset(cb_info);
+ do {
+ /* initialize mock edge */
+ ret = smp2p_reset_mock_edge();
+ UT_ASSERT_INT(ret, ==, 0);
+
+ mock = msm_smp2p_get_remote_mock();
+ UT_ASSERT_PTR(mock, !=, NULL);
+
+ mock->rx_interrupt_count = 0;
+ memset(&mock->remote_item, 0,
+ sizeof(struct smp2p_smem_item));
+ smp2p_init_header((struct smp2p_smem *)&mock->remote_item,
+ SMP2P_REMOTE_MOCK_PROC, SMP2P_APPS_PROC,
+ 0, 1);
+ strlcpy(mock->remote_item.entries[0].name, "smp2p",
+ SMP2P_MAX_ENTRY_NAME);
+ SMP2P_SET_ENT_VALID(
+ &mock->remote_item.header, valid_total_ent, 1);
+
+ /* register for interrupts */
+ smp2p_gpio_open_test_entry("smp2p",
+ SMP2P_REMOTE_MOCK_PROC, true);
+
+ UT_ASSERT_INT(0, <, cb_info->irq_base_id);
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY && !failed; ++id) {
+ virq = cb_info->irq_base_id + id;
+ UT_ASSERT_PTR(NULL, !=, irq_to_desc(virq));
+ ret = request_irq(virq,
+ smp2p_gpio_irq, IRQ_TYPE_EDGE_BOTH,
+ "smp2p_test", cb_info);
+ UT_ASSERT_INT(0, ==, ret);
+ }
+ if (failed)
+ break;
+
+ /* update the state value and complete negotiation */
+ mock->remote_item.entries[0].entry = 0xDEADDEAD;
+ msm_smp2p_set_remote_mock_exists(true);
+ mock->tx_interrupt();
+
+ /* verify delayed state updates were processed */
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY && !failed; ++id) {
+ virq = cb_info->irq_base_id + id;
+
+ UT_ASSERT_INT(cb_info->cb_count, >, 0);
+ if (0x1 & (0xDEADDEAD >> id)) {
+ /* rising edge should have been triggered */
+ if (!test_bit(id, cb_info->triggered_irqs)) {
+ seq_printf(s, "%s:%d bit %d clear, ",
+ __func__, __LINE__, id);
+ seq_puts(s, "expected set\n");
+ failed = 1;
+ break;
+ }
+ } else {
+ /* edge should not have been triggered */
+ if (test_bit(id, cb_info->triggered_irqs)) {
+ seq_printf(s, "%s:%d bit %d set, ",
+ __func__, __LINE__, id);
+ seq_puts(s, "expected clear\n");
+ failed = 1;
+ break;
+ }
+ }
+ }
+ if (failed)
+ break;
+
+ seq_puts(s, "\tOK\n");
+ } while (0);
+
+ if (failed) {
+ pr_err("%s: Failed\n", __func__);
+ seq_puts(s, "\tFailed\n");
+ }
+
+ /* unregister for interrupts */
+ if (cb_info->irq_base_id) {
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY; ++id)
+ free_irq(cb_info->irq_base_id + id, cb_info);
+ }
+
+ smp2p_gpio_open_test_entry("smp2p",
+ SMP2P_REMOTE_MOCK_PROC, false);
+}
+
+/**
+ * smp2p_gpio_write_bits - writes value to each GPIO pin specified in mask.
+ *
+ * @gpio: gpio test structure
+ * @mask: 1 = write gpio_value to this GPIO pin
+ * @gpio_value: value to write to GPIO pin
+ */
+static void smp2p_gpio_write_bits(struct gpio_info *gpio, uint32_t mask,
+ int gpio_value)
+{
+ int n;
+
+ for (n = 0; n < SMP2P_BITS_PER_ENTRY; ++n) {
+ if (mask & 0x1)
+ gpio_set_value(gpio->gpio_base_id + n, gpio_value);
+ mask >>= 1;
+ }
+}
+
+static void smp2p_gpio_set_bits(struct gpio_info *gpio, uint32_t mask)
+{
+ smp2p_gpio_write_bits(gpio, mask, 1);
+}
+
+static void smp2p_gpio_clr_bits(struct gpio_info *gpio, uint32_t mask)
+{
+ smp2p_gpio_write_bits(gpio, mask, 0);
+}
+
+/**
+ * smp2p_gpio_get_value - reads entire 32-bits of GPIO
+ *
+ * @gpio: gpio structure
+ * @returns: 32 bit value of GPIO pins
+ */
+static uint32_t smp2p_gpio_get_value(struct gpio_info *gpio)
+{
+ int n;
+ uint32_t value = 0;
+
+ for (n = 0; n < SMP2P_BITS_PER_ENTRY; ++n) {
+ if (gpio_get_value(gpio->gpio_base_id + n))
+ value |= 1 << n;
+ }
+ return value;
+}
+
+/**
+ * smp2p_ut_remote_inout_core - Verify inbound/outbound functionality.
+ *
+ * @s: pointer to output file
+ * @remote_pid: Remote processor to test
+ * @name: Name of the test for reporting
+ *
+ * This test verifies inbound/outbound functionality for the remote processor.
+ */
+static void smp2p_ut_remote_inout_core(struct seq_file *s, int remote_pid,
+ const char *name)
+{
+ int failed = 0;
+ uint32_t request;
+ uint32_t response;
+ struct gpio_info *cb_in;
+ struct gpio_info *cb_out;
+ int id;
+ int ret;
+
+ seq_printf(s, "Running %s for '%s' remote pid %d\n",
+ __func__, smp2p_pid_to_name(remote_pid), remote_pid);
+
+ cb_in = &gpio_info[remote_pid].in;
+ cb_out = &gpio_info[remote_pid].out;
+ cb_data_reset(cb_in);
+ cb_data_reset(cb_out);
+ do {
+ /* open test entries */
+ msm_smp2p_deinit_rmt_lpb_proc(remote_pid);
+ smp2p_gpio_open_test_entry("smp2p", remote_pid, true);
+
+ /* register for interrupts */
+ UT_ASSERT_INT(0, <, cb_in->gpio_base_id);
+ UT_ASSERT_INT(0, <, cb_in->irq_base_id);
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY && !failed; ++id) {
+ int virq = cb_in->irq_base_id + id;
+
+ UT_ASSERT_PTR(NULL, !=, irq_to_desc(virq));
+ ret = request_irq(virq,
+ smp2p_gpio_irq,
+ IRQF_TRIGGER_RISING | IRQF_TRIGGER_FALLING,
+ "smp2p_test", cb_in);
+ UT_ASSERT_INT(0, ==, ret);
+ }
+ if (failed)
+ break;
+
+ /* write echo of data value 0 */
+ UT_ASSERT_INT(0, <, cb_out->gpio_base_id);
+ request = 0x0;
+ SMP2P_SET_RMT_CMD_TYPE(request, 1);
+ SMP2P_SET_RMT_CMD(request, SMP2P_LB_CMD_ECHO);
+ SMP2P_SET_RMT_DATA(request, 0x0);
+
+ smp2p_gpio_set_bits(cb_out, SMP2P_RMT_IGNORE_MASK);
+ smp2p_gpio_clr_bits(cb_out, ~SMP2P_RMT_IGNORE_MASK);
+ smp2p_gpio_set_bits(cb_out, request);
+
+ UT_ASSERT_INT(cb_in->cb_count, ==, 0);
+ smp2p_gpio_clr_bits(cb_out, SMP2P_RMT_IGNORE_MASK);
+
+ /* verify response */
+ do {
+ /* wait for up to 32 changes */
+ if (wait_for_completion_timeout(
+ &cb_in->cb_completion, HZ / 2) == 0)
+ break;
+ reinit_completion(&cb_in->cb_completion);
+ } while (cb_in->cb_count < 32);
+ UT_ASSERT_INT(cb_in->cb_count, >, 0);
+ response = smp2p_gpio_get_value(cb_in);
+ SMP2P_SET_RMT_CMD_TYPE(request, 0);
+ UT_ASSERT_HEX(request, ==, response);
+
+ /* write echo of data value of all 1's */
+ request = 0x0;
+ SMP2P_SET_RMT_CMD_TYPE(request, 1);
+ SMP2P_SET_RMT_CMD(request, SMP2P_LB_CMD_ECHO);
+ SMP2P_SET_RMT_DATA(request, ~0);
+
+ smp2p_gpio_set_bits(cb_out, SMP2P_RMT_IGNORE_MASK);
+ cb_data_reset(cb_in);
+ smp2p_gpio_clr_bits(cb_out, ~SMP2P_RMT_IGNORE_MASK);
+ smp2p_gpio_set_bits(cb_out, request);
+
+ UT_ASSERT_INT(cb_in->cb_count, ==, 0);
+ smp2p_gpio_clr_bits(cb_out, SMP2P_RMT_IGNORE_MASK);
+
+ /* verify response including 24 interrupts */
+ do {
+ UT_ASSERT_INT(
+ (int)wait_for_completion_timeout(
+ &cb_in->cb_completion, HZ / 2),
+ >, 0);
+ reinit_completion(&cb_in->cb_completion);
+ } while (cb_in->cb_count < 24);
+ response = smp2p_gpio_get_value(cb_in);
+ SMP2P_SET_RMT_CMD_TYPE(request, 0);
+ UT_ASSERT_HEX(request, ==, response);
+ UT_ASSERT_INT(24, ==, cb_in->cb_count);
+
+ seq_puts(s, "\tOK\n");
+ } while (0);
+
+ if (failed) {
+ pr_err("%s: Failed\n", name);
+ seq_puts(s, "\tFailed\n");
+ }
+
+ /* unregister for interrupts */
+ if (cb_in->irq_base_id) {
+ for (id = 0; id < SMP2P_BITS_PER_ENTRY; ++id)
+ free_irq(cb_in->irq_base_id + id, cb_in);
+ }
+
+ smp2p_gpio_open_test_entry("smp2p", remote_pid, false);
+ msm_smp2p_init_rmt_lpb_proc(remote_pid);
+}
+
+/**
+ * smp2p_ut_remote_inout - Verify inbound/outbound functionality for all.
+ *
+ * @s: pointer to output file
+ *
+ * This test verifies inbound and outbound functionality for all
+ * configured remote processor.
+ */
+static void smp2p_ut_remote_inout(struct seq_file *s)
+{
+ struct smp2p_interrupt_config *int_cfg;
+ int pid;
+
+ int_cfg = smp2p_get_interrupt_config();
+ if (!int_cfg) {
+ seq_puts(s, "Remote processor config unavailable\n");
+ return;
+ }
+
+ for (pid = 0; pid < SMP2P_NUM_PROCS; ++pid) {
+ if (!int_cfg[pid].is_configured)
+ continue;
+
+ smp2p_ut_remote_inout_core(s, pid, __func__);
+ }
+}
+
+static int __init smp2p_debugfs_init(void)
+{
+ /* register GPIO pins */
+ (void)platform_driver_register(&smp2p_gpio_driver);
+
+ /*
+ * Add Unit Test entries.
+ *
+ * The idea with unit tests is that you can run all of them
+ * from ADB shell by doing:
+ * adb shell
+ * cat ut*
+ *
+ * And if particular tests fail, you can then repeatedly run the
+ * failing tests as you debug and resolve the failing test.
+ */
+ smp2p_debug_create("ut_local_gpio_out", smp2p_ut_local_gpio_out);
+ smp2p_debug_create("ut_local_gpio_in", smp2p_ut_local_gpio_in);
+ smp2p_debug_create("ut_local_gpio_in_update_open",
+ smp2p_ut_local_gpio_in_update_open);
+ smp2p_debug_create("ut_remote_gpio_inout", smp2p_ut_remote_inout);
+ return 0;
+}
+late_initcall(smp2p_debugfs_init);
diff --git a/drivers/gpio/gpio-mxs.c b/drivers/gpio/gpio-mxs.c
index ee17248..ab8dcfe 100644
--- a/drivers/gpio/gpio-mxs.c
+++ b/drivers/gpio/gpio-mxs.c
@@ -32,8 +32,6 @@
#include <linux/platform_device.h>
#include <linux/slab.h>
#include <linux/gpio/driver.h>
-/* FIXME: for gpio_get_value(), replace this by direct register read */
-#include <linux/gpio.h>
#include <linux/module.h>
#define MXS_SET 0x4
@@ -94,7 +92,7 @@ static int mxs_gpio_set_irq_type(struct irq_data *d, unsigned int type)
port->both_edges &= ~pin_mask;
switch (type) {
case IRQ_TYPE_EDGE_BOTH:
- val = gpio_get_value(port->gc.base + d->hwirq);
+ val = port->gc.get(&port->gc, d->hwirq);
if (val)
edge = GPIO_INT_FALL_EDGE;
else
diff --git a/drivers/gpio/gpio-pxa.c b/drivers/gpio/gpio-pxa.c
index 76ac906..7a63058 100644
--- a/drivers/gpio/gpio-pxa.c
+++ b/drivers/gpio/gpio-pxa.c
@@ -660,6 +660,8 @@ static int pxa_gpio_probe(struct platform_device *pdev)
pchip->irq0 = irq0;
pchip->irq1 = irq1;
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (!res)
+ return -EINVAL;
gpio_reg_base = devm_ioremap(&pdev->dev, res->start,
resource_size(res));
if (!gpio_reg_base)
diff --git a/drivers/gpio/gpio-tegra.c b/drivers/gpio/gpio-tegra.c
index 661b0e3..05d3241 100644
--- a/drivers/gpio/gpio-tegra.c
+++ b/drivers/gpio/gpio-tegra.c
@@ -723,4 +723,4 @@ static int __init tegra_gpio_init(void)
{
return platform_driver_register(&tegra_gpio_driver);
}
-postcore_initcall(tegra_gpio_init);
+subsys_initcall(tegra_gpio_init);
diff --git a/drivers/gpio/gpiolib-of.c b/drivers/gpio/gpiolib-of.c
index 193f15d..aac8432 100644
--- a/drivers/gpio/gpiolib-of.c
+++ b/drivers/gpio/gpiolib-of.c
@@ -31,6 +31,7 @@ static int of_gpiochip_match_node_and_xlate(struct gpio_chip *chip, void *data)
struct of_phandle_args *gpiospec = data;
return chip->gpiodev->dev.of_node == gpiospec->np &&
+ chip->of_xlate &&
chip->of_xlate(chip, gpiospec, NULL) >= 0;
}
diff --git a/drivers/gpio/gpiolib.c b/drivers/gpio/gpiolib.c
index 4f54ff4..9e2fe12 100644
--- a/drivers/gpio/gpiolib.c
+++ b/drivers/gpio/gpiolib.c
@@ -425,7 +425,7 @@ static int linehandle_create(struct gpio_device *gdev, void __user *ip)
struct gpiohandle_request handlereq;
struct linehandle_state *lh;
struct file *file;
- int fd, i, ret;
+ int fd, i, count = 0, ret;
if (copy_from_user(&handlereq, ip, sizeof(handlereq)))
return -EFAULT;
@@ -471,6 +471,7 @@ static int linehandle_create(struct gpio_device *gdev, void __user *ip)
if (ret)
goto out_free_descs;
lh->descs[i] = desc;
+ count = i + 1;
if (lflags & GPIOHANDLE_REQUEST_ACTIVE_LOW)
set_bit(FLAG_ACTIVE_LOW, &desc->flags);
@@ -537,7 +538,7 @@ static int linehandle_create(struct gpio_device *gdev, void __user *ip)
out_put_unused_fd:
put_unused_fd(fd);
out_free_descs:
- for (; i >= 0; i--)
+ for (i = 0; i < count; i++)
gpiod_free(lh->descs[i]);
kfree(lh->label);
out_free_lh:
@@ -794,7 +795,7 @@ static int lineevent_create(struct gpio_device *gdev, void __user *ip)
desc = &gdev->descs[offset];
ret = gpiod_request(desc, le->label);
if (ret)
- goto out_free_desc;
+ goto out_free_label;
le->desc = desc;
le->eflags = eflags;
@@ -1152,7 +1153,7 @@ int gpiochip_add_data(struct gpio_chip *chip, void *data)
gdev->descs = kcalloc(chip->ngpio, sizeof(gdev->descs[0]), GFP_KERNEL);
if (!gdev->descs) {
status = -ENOMEM;
- goto err_free_gdev;
+ goto err_free_ida;
}
if (chip->ngpio == 0) {
@@ -1284,8 +1285,9 @@ int gpiochip_add_data(struct gpio_chip *chip, void *data)
kfree(gdev->label);
err_free_descs:
kfree(gdev->descs);
-err_free_gdev:
+err_free_ida:
ida_simple_remove(&gpio_ida, gdev->id);
+err_free_gdev:
/* failures here can mean systems won't boot... */
pr_err("%s: GPIOs %d..%d (%s) failed to register\n", __func__,
gdev->base, gdev->base + gdev->ngpio - 1,
@@ -3203,6 +3205,8 @@ struct gpio_desc *__must_check gpiod_get_index(struct device *dev,
struct gpio_desc *desc = NULL;
int status;
enum gpio_lookup_flags lookupflags = 0;
+ /* Maybe we have a device name, maybe not */
+ const char *devname = dev ? dev_name(dev) : "?";
dev_dbg(dev, "GPIO lookup for consumer %s\n", con_id);
@@ -3231,8 +3235,11 @@ struct gpio_desc *__must_check gpiod_get_index(struct device *dev,
return desc;
}
- /* If a connection label was passed use that, else use the device name as label */
- status = gpiod_request(desc, con_id ? con_id : dev_name(dev));
+ /*
+ * If a connection label was passed use that, else attempt to use
+ * the device name as label
+ */
+ status = gpiod_request(desc, con_id ? con_id : devname);
if (status < 0)
return ERR_PTR(status);
diff --git a/drivers/gpio/gpiolib.h b/drivers/gpio/gpiolib.h
index 346fbda..6c4d728 100644
--- a/drivers/gpio/gpiolib.h
+++ b/drivers/gpio/gpiolib.h
@@ -85,7 +85,7 @@ struct acpi_gpio_info {
};
/* gpio suffixes used for ACPI and device tree lookup */
-static const char * const gpio_suffixes[] = { "gpios", "gpio" };
+static __maybe_unused const char * const gpio_suffixes[] = { "gpios", "gpio" };
#ifdef CONFIG_OF_GPIO
struct gpio_desc *of_find_gpio(struct device *dev,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v7.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v7.c
index 47951f4..d47c32a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v7.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gfx_v7.c
@@ -505,7 +505,7 @@ static int kgd_hqd_sdma_destroy(struct kgd_dev *kgd, void *mqd,
while (true) {
temp = RREG32(sdma_base_addr + mmSDMA0_RLC0_CONTEXT_STATUS);
- if (temp & SDMA0_STATUS_REG__RB_CMD_IDLE__SHIFT)
+ if (temp & SDMA0_RLC0_CONTEXT_STATUS__IDLE_MASK)
break;
if (timeout <= 0)
return -ETIME;
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
index a88d365..c8a5cf5 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v8_0.c
@@ -1484,10 +1484,11 @@ static const u32 sgpr_init_compute_shader[] =
static const u32 vgpr_init_regs[] =
{
mmCOMPUTE_STATIC_THREAD_MGMT_SE0, 0xffffffff,
- mmCOMPUTE_RESOURCE_LIMITS, 0,
+ mmCOMPUTE_RESOURCE_LIMITS, 0x1000000, /* CU_GROUP_COUNT=1 */
mmCOMPUTE_NUM_THREAD_X, 256*4,
mmCOMPUTE_NUM_THREAD_Y, 1,
mmCOMPUTE_NUM_THREAD_Z, 1,
+ mmCOMPUTE_PGM_RSRC1, 0x100004f, /* VGPRS=15 (64 logical VGPRs), SGPRS=1 (16 SGPRs), BULKY=1 */
mmCOMPUTE_PGM_RSRC2, 20,
mmCOMPUTE_USER_DATA_0, 0xedcedc00,
mmCOMPUTE_USER_DATA_1, 0xedcedc01,
@@ -1504,10 +1505,11 @@ static const u32 vgpr_init_regs[] =
static const u32 sgpr1_init_regs[] =
{
mmCOMPUTE_STATIC_THREAD_MGMT_SE0, 0x0f,
- mmCOMPUTE_RESOURCE_LIMITS, 0x1000000,
+ mmCOMPUTE_RESOURCE_LIMITS, 0x1000000, /* CU_GROUP_COUNT=1 */
mmCOMPUTE_NUM_THREAD_X, 256*5,
mmCOMPUTE_NUM_THREAD_Y, 1,
mmCOMPUTE_NUM_THREAD_Z, 1,
+ mmCOMPUTE_PGM_RSRC1, 0x240, /* SGPRS=9 (80 GPRS) */
mmCOMPUTE_PGM_RSRC2, 20,
mmCOMPUTE_USER_DATA_0, 0xedcedc00,
mmCOMPUTE_USER_DATA_1, 0xedcedc01,
@@ -1528,6 +1530,7 @@ static const u32 sgpr2_init_regs[] =
mmCOMPUTE_NUM_THREAD_X, 256*5,
mmCOMPUTE_NUM_THREAD_Y, 1,
mmCOMPUTE_NUM_THREAD_Z, 1,
+ mmCOMPUTE_PGM_RSRC1, 0x240, /* SGPRS=9 (80 GPRS) */
mmCOMPUTE_PGM_RSRC2, 20,
mmCOMPUTE_USER_DATA_0, 0xedcedc00,
mmCOMPUTE_USER_DATA_1, 0xedcedc01,
@@ -5548,6 +5551,11 @@ static int gfx_v8_0_set_powergating_state(void *handle,
if (!(adev->pg_flags & AMD_PG_SUPPORT_GFX_PG))
return 0;
+ if (adev->pg_flags & (AMD_PG_SUPPORT_GFX_SMG |
+ AMD_PG_SUPPORT_RLC_SMU_HS |
+ AMD_PG_SUPPORT_CP |
+ AMD_PG_SUPPORT_GFX_DMG))
+ adev->gfx.rlc.funcs->enter_safe_mode(adev);
switch (adev->asic_type) {
case CHIP_CARRIZO:
case CHIP_STONEY:
@@ -5583,7 +5591,11 @@ static int gfx_v8_0_set_powergating_state(void *handle,
default:
break;
}
-
+ if (adev->pg_flags & (AMD_PG_SUPPORT_GFX_SMG |
+ AMD_PG_SUPPORT_RLC_SMU_HS |
+ AMD_PG_SUPPORT_CP |
+ AMD_PG_SUPPORT_GFX_DMG))
+ adev->gfx.rlc.funcs->exit_safe_mode(adev);
return 0;
}
diff --git a/drivers/gpu/drm/amd/amdgpu/kv_dpm.c b/drivers/gpu/drm/amd/amdgpu/kv_dpm.c
index 71d2856..f61c489 100644
--- a/drivers/gpu/drm/amd/amdgpu/kv_dpm.c
+++ b/drivers/gpu/drm/amd/amdgpu/kv_dpm.c
@@ -1350,8 +1350,6 @@ static int kv_dpm_enable(struct amdgpu_device *adev)
return ret;
}
- kv_update_current_ps(adev, adev->pm.dpm.boot_ps);
-
if (adev->irq.installed &&
amdgpu_is_internal_thermal_sensor(adev->pm.int_thermal_type)) {
ret = kv_set_thermal_temperature_range(adev, KV_TEMP_RANGE_MIN, KV_TEMP_RANGE_MAX);
@@ -3086,7 +3084,7 @@ static int kv_dpm_hw_init(void *handle)
else
adev->pm.dpm_enabled = true;
mutex_unlock(&adev->pm.mutex);
-
+ amdgpu_pm_compute_clocks(adev);
return ret;
}
diff --git a/drivers/gpu/drm/amd/amdgpu/si_dpm.c b/drivers/gpu/drm/amd/amdgpu/si_dpm.c
index 3fa8320..4826bef 100644
--- a/drivers/gpu/drm/amd/amdgpu/si_dpm.c
+++ b/drivers/gpu/drm/amd/amdgpu/si_dpm.c
@@ -6959,7 +6959,6 @@ static int si_dpm_enable(struct amdgpu_device *adev)
si_enable_auto_throttle_source(adev, AMDGPU_DPM_AUTO_THROTTLE_SRC_THERMAL, true);
si_thermal_start_thermal_controller(adev);
- ni_update_current_ps(adev, boot_ps);
return 0;
}
@@ -7836,7 +7835,7 @@ static int si_dpm_hw_init(void *handle)
else
adev->pm.dpm_enabled = true;
mutex_unlock(&adev->pm.mutex);
-
+ amdgpu_pm_compute_clocks(adev);
return ret;
}
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
index 171480b..6e7eb76 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c
@@ -124,6 +124,8 @@ struct kfd_process *kfd_get_process(const struct task_struct *thread)
return ERR_PTR(-EINVAL);
process = find_process(thread);
+ if (!process)
+ return ERR_PTR(-EINVAL);
return process;
}
diff --git a/drivers/gpu/drm/arm/malidp_drv.c b/drivers/gpu/drm/arm/malidp_drv.c
index 9280358..59d4847 100644
--- a/drivers/gpu/drm/arm/malidp_drv.c
+++ b/drivers/gpu/drm/arm/malidp_drv.c
@@ -378,6 +378,7 @@ static int malidp_bind(struct device *dev)
goto irq_init_fail;
ret = drm_vblank_init(drm, drm->mode_config.num_crtc);
+ drm_crtc_vblank_reset(&malidp->crtc);
if (ret < 0) {
DRM_ERROR("failed to initialise vblank\n");
goto vblank_fail;
diff --git a/drivers/gpu/drm/arm/malidp_hw.c b/drivers/gpu/drm/arm/malidp_hw.c
index a6132f1..5eee325 100644
--- a/drivers/gpu/drm/arm/malidp_hw.c
+++ b/drivers/gpu/drm/arm/malidp_hw.c
@@ -432,7 +432,8 @@ const struct malidp_hw_device malidp_device[MALIDP_MAX_DEVICES] = {
.vsync_irq = MALIDP500_DE_IRQ_VSYNC,
},
.se_irq_map = {
- .irq_mask = MALIDP500_SE_IRQ_CONF_MODE,
+ .irq_mask = MALIDP500_SE_IRQ_CONF_MODE |
+ MALIDP500_SE_IRQ_GLOBAL,
.vsync_irq = 0,
},
.dc_irq_map = {
diff --git a/drivers/gpu/drm/armada/armada_hw.h b/drivers/gpu/drm/armada/armada_hw.h
index 27319a8..345dc4d 100644
--- a/drivers/gpu/drm/armada/armada_hw.h
+++ b/drivers/gpu/drm/armada/armada_hw.h
@@ -160,6 +160,7 @@ enum {
CFG_ALPHAM_GRA = 0x1 << 16,
CFG_ALPHAM_CFG = 0x2 << 16,
CFG_ALPHA_MASK = 0xff << 8,
+#define CFG_ALPHA(x) ((x) << 8)
CFG_PIXCMD_MASK = 0xff,
};
diff --git a/drivers/gpu/drm/armada/armada_overlay.c b/drivers/gpu/drm/armada/armada_overlay.c
index 152b4e7..6a9bba7 100644
--- a/drivers/gpu/drm/armada/armada_overlay.c
+++ b/drivers/gpu/drm/armada/armada_overlay.c
@@ -27,6 +27,7 @@ struct armada_ovl_plane_properties {
uint16_t contrast;
uint16_t saturation;
uint32_t colorkey_mode;
+ uint32_t colorkey_enable;
};
struct armada_ovl_plane {
@@ -62,11 +63,13 @@ armada_ovl_update_attr(struct armada_ovl_plane_properties *prop,
writel_relaxed(0x00002000, dcrtc->base + LCD_SPU_CBSH_HUE);
spin_lock_irq(&dcrtc->irq_lock);
- armada_updatel(prop->colorkey_mode | CFG_ALPHAM_GRA,
- CFG_CKMODE_MASK | CFG_ALPHAM_MASK | CFG_ALPHA_MASK,
- dcrtc->base + LCD_SPU_DMA_CTRL1);
-
- armada_updatel(ADV_GRACOLORKEY, 0, dcrtc->base + LCD_SPU_ADV_REG);
+ armada_updatel(prop->colorkey_mode,
+ CFG_CKMODE_MASK | CFG_ALPHAM_MASK | CFG_ALPHA_MASK,
+ dcrtc->base + LCD_SPU_DMA_CTRL1);
+ if (dcrtc->variant->has_spu_adv_reg)
+ armada_updatel(prop->colorkey_enable,
+ ADV_GRACOLORKEY | ADV_VIDCOLORKEY,
+ dcrtc->base + LCD_SPU_ADV_REG);
spin_unlock_irq(&dcrtc->irq_lock);
}
@@ -340,8 +343,17 @@ static int armada_ovl_plane_set_property(struct drm_plane *plane,
dplane->prop.colorkey_vb |= K2B(val);
update_attr = true;
} else if (property == priv->colorkey_mode_prop) {
- dplane->prop.colorkey_mode &= ~CFG_CKMODE_MASK;
- dplane->prop.colorkey_mode |= CFG_CKMODE(val);
+ if (val == CKMODE_DISABLE) {
+ dplane->prop.colorkey_mode =
+ CFG_CKMODE(CKMODE_DISABLE) |
+ CFG_ALPHAM_CFG | CFG_ALPHA(255);
+ dplane->prop.colorkey_enable = 0;
+ } else {
+ dplane->prop.colorkey_mode =
+ CFG_CKMODE(val) |
+ CFG_ALPHAM_GRA | CFG_ALPHA(0);
+ dplane->prop.colorkey_enable = ADV_GRACOLORKEY;
+ }
update_attr = true;
} else if (property == priv->brightness_prop) {
dplane->prop.brightness = val - 256;
@@ -470,7 +482,9 @@ int armada_overlay_plane_create(struct drm_device *dev, unsigned long crtcs)
dplane->prop.colorkey_yr = 0xfefefe00;
dplane->prop.colorkey_ug = 0x01010100;
dplane->prop.colorkey_vb = 0x01010100;
- dplane->prop.colorkey_mode = CFG_CKMODE(CKMODE_RGB);
+ dplane->prop.colorkey_mode = CFG_CKMODE(CKMODE_RGB) |
+ CFG_ALPHAM_GRA | CFG_ALPHA(0);
+ dplane->prop.colorkey_enable = ADV_GRACOLORKEY;
dplane->prop.brightness = 0;
dplane->prop.contrast = 0x4000;
dplane->prop.saturation = 0x4000;
diff --git a/drivers/gpu/drm/ast/ast_drv.c b/drivers/gpu/drm/ast/ast_drv.c
index f54afd2..736133f 100644
--- a/drivers/gpu/drm/ast/ast_drv.c
+++ b/drivers/gpu/drm/ast/ast_drv.c
@@ -60,8 +60,29 @@ static const struct pci_device_id pciidlist[] = {
MODULE_DEVICE_TABLE(pci, pciidlist);
+static void ast_kick_out_firmware_fb(struct pci_dev *pdev)
+{
+ struct apertures_struct *ap;
+ bool primary = false;
+
+ ap = alloc_apertures(1);
+ if (!ap)
+ return;
+
+ ap->ranges[0].base = pci_resource_start(pdev, 0);
+ ap->ranges[0].size = pci_resource_len(pdev, 0);
+
+#ifdef CONFIG_X86
+ primary = pdev->resource[PCI_ROM_RESOURCE].flags & IORESOURCE_ROM_SHADOW;
+#endif
+ drm_fb_helper_remove_conflicting_framebuffers(ap, "astdrmfb", primary);
+ kfree(ap);
+}
+
static int ast_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
{
+ ast_kick_out_firmware_fb(pdev);
+
return drm_get_pci_dev(pdev, ent, &driver);
}
diff --git a/drivers/gpu/drm/ast/ast_fb.c b/drivers/gpu/drm/ast/ast_fb.c
index 7a86e24..5e0d3e5 100644
--- a/drivers/gpu/drm/ast/ast_fb.c
+++ b/drivers/gpu/drm/ast/ast_fb.c
@@ -286,6 +286,7 @@ static void ast_fbdev_destroy(struct drm_device *dev,
{
struct ast_framebuffer *afb = &afbdev->afb;
+ drm_crtc_force_disable_all(dev);
drm_fb_helper_unregister_fbi(&afbdev->helper);
drm_fb_helper_release_fbi(&afbdev->helper);
diff --git a/drivers/gpu/drm/ast/ast_main.c b/drivers/gpu/drm/ast/ast_main.c
index fb99762..fabfeeb 100644
--- a/drivers/gpu/drm/ast/ast_main.c
+++ b/drivers/gpu/drm/ast/ast_main.c
@@ -556,7 +556,8 @@ int ast_driver_unload(struct drm_device *dev)
drm_mode_config_cleanup(dev);
ast_mm_fini(ast);
- pci_iounmap(dev->pdev, ast->ioregs);
+ if (ast->ioregs != ast->regs + AST_IO_MM_OFFSET)
+ pci_iounmap(dev->pdev, ast->ioregs);
pci_iounmap(dev->pdev, ast->regs);
kfree(ast);
return 0;
diff --git a/drivers/gpu/drm/ast/ast_mode.c b/drivers/gpu/drm/ast/ast_mode.c
index 5957c3e..201874b 100644
--- a/drivers/gpu/drm/ast/ast_mode.c
+++ b/drivers/gpu/drm/ast/ast_mode.c
@@ -545,6 +545,7 @@ static int ast_crtc_do_set_base(struct drm_crtc *crtc,
}
ast_bo_unreserve(bo);
+ ast_set_offset_reg(crtc);
ast_set_start_address_crt1(crtc, (u32)gpu_addr);
return 0;
@@ -953,9 +954,21 @@ static int get_clock(void *i2c_priv)
{
struct ast_i2c_chan *i2c = i2c_priv;
struct ast_private *ast = i2c->dev->dev_private;
- uint32_t val;
+ uint32_t val, val2, count, pass;
- val = ast_get_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0x10) >> 4;
+ count = 0;
+ pass = 0;
+ val = (ast_get_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0x10) >> 4) & 0x01;
+ do {
+ val2 = (ast_get_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0x10) >> 4) & 0x01;
+ if (val == val2) {
+ pass++;
+ } else {
+ pass = 0;
+ val = (ast_get_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0x10) >> 4) & 0x01;
+ }
+ } while ((pass < 5) && (count++ < 0x10000));
+
return val & 1 ? 1 : 0;
}
@@ -963,9 +976,21 @@ static int get_data(void *i2c_priv)
{
struct ast_i2c_chan *i2c = i2c_priv;
struct ast_private *ast = i2c->dev->dev_private;
- uint32_t val;
+ uint32_t val, val2, count, pass;
- val = ast_get_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0x20) >> 5;
+ count = 0;
+ pass = 0;
+ val = (ast_get_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0x20) >> 5) & 0x01;
+ do {
+ val2 = (ast_get_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0x20) >> 5) & 0x01;
+ if (val == val2) {
+ pass++;
+ } else {
+ pass = 0;
+ val = (ast_get_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0x20) >> 5) & 0x01;
+ }
+ } while ((pass < 5) && (count++ < 0x10000));
+
return val & 1 ? 1 : 0;
}
@@ -978,7 +1003,7 @@ static void set_clock(void *i2c_priv, int clock)
for (i = 0; i < 0x10000; i++) {
ujcrb7 = ((clock & 0x01) ? 0 : 1);
- ast_set_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0xfe, ujcrb7);
+ ast_set_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0xf4, ujcrb7);
jtemp = ast_get_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0x01);
if (ujcrb7 == jtemp)
break;
@@ -994,7 +1019,7 @@ static void set_data(void *i2c_priv, int data)
for (i = 0; i < 0x10000; i++) {
ujcrb7 = ((data & 0x01) ? 0 : 1) << 2;
- ast_set_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0xfb, ujcrb7);
+ ast_set_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0xf1, ujcrb7);
jtemp = ast_get_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xb7, 0x04);
if (ujcrb7 == jtemp)
break;
@@ -1235,7 +1260,7 @@ static int ast_cursor_move(struct drm_crtc *crtc,
ast_set_index_reg(ast, AST_IO_CRTC_PORT, 0xc7, ((y >> 8) & 0x07));
/* dummy write to fire HWC */
- ast_set_index_reg_mask(ast, AST_IO_CRTC_PORT, 0xCB, 0xFF, 0x00);
+ ast_show_cursor(crtc);
return 0;
}
diff --git a/drivers/gpu/drm/bochs/bochs_fbdev.c b/drivers/gpu/drm/bochs/bochs_fbdev.c
index e1ec498..35f4025 100644
--- a/drivers/gpu/drm/bochs/bochs_fbdev.c
+++ b/drivers/gpu/drm/bochs/bochs_fbdev.c
@@ -138,6 +138,7 @@ static int bochsfb_create(struct drm_fb_helper *helper,
info->fix.smem_start = 0;
info->fix.smem_len = size;
+ bochs->fb.initialized = true;
return 0;
}
@@ -155,7 +156,6 @@ static int bochs_fbdev_destroy(struct bochs_device *bochs)
gfb->obj = NULL;
}
- drm_fb_helper_fini(&bochs->fb.helper);
drm_framebuffer_unregister_private(&gfb->base);
drm_framebuffer_cleanup(&gfb->base);
@@ -188,7 +188,6 @@ int bochs_fbdev_init(struct bochs_device *bochs)
if (ret)
goto fini;
- bochs->fb.initialized = true;
return 0;
fini:
@@ -198,9 +197,9 @@ int bochs_fbdev_init(struct bochs_device *bochs)
void bochs_fbdev_fini(struct bochs_device *bochs)
{
- if (!bochs->fb.initialized)
- return;
+ if (bochs->fb.initialized)
+ bochs_fbdev_destroy(bochs);
- bochs_fbdev_destroy(bochs);
+ drm_fb_helper_fini(&bochs->fb.helper);
bochs->fb.initialized = false;
}
diff --git a/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c b/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c
index a68f94d..32ab5c3 100644
--- a/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c
+++ b/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c
@@ -424,6 +424,18 @@ static void adv7511_hpd_work(struct work_struct *work)
else
status = connector_status_disconnected;
+ /*
+ * The bridge resets its registers on unplug. So when we get a plug
+ * event and we're already supposed to be powered, cycle the bridge to
+ * restore its state.
+ */
+ if (status == connector_status_connected &&
+ adv7511->connector.status == connector_status_disconnected &&
+ adv7511->powered) {
+ regcache_mark_dirty(adv7511->regmap);
+ adv7511_power_on(adv7511);
+ }
+
if (adv7511->connector.status != status) {
adv7511->connector.status = status;
drm_kms_helper_hotplug_event(adv7511->connector.dev);
diff --git a/drivers/gpu/drm/bridge/dumb-vga-dac.c b/drivers/gpu/drm/bridge/dumb-vga-dac.c
index afec232..cfd80bc 100644
--- a/drivers/gpu/drm/bridge/dumb-vga-dac.c
+++ b/drivers/gpu/drm/bridge/dumb-vga-dac.c
@@ -53,7 +53,9 @@ static int dumb_vga_get_modes(struct drm_connector *connector)
}
drm_mode_connector_update_edid_property(connector, edid);
- return drm_add_edid_modes(connector, edid);
+ ret = drm_add_edid_modes(connector, edid);
+ kfree(edid);
+ return ret;
fallback:
/*
diff --git a/drivers/gpu/drm/drm_atomic.c b/drivers/gpu/drm/drm_atomic.c
index 33778bf..4e4043f 100644
--- a/drivers/gpu/drm/drm_atomic.c
+++ b/drivers/gpu/drm/drm_atomic.c
@@ -1134,7 +1134,9 @@ drm_atomic_set_crtc_for_plane(struct drm_plane_state *plane_state,
{
struct drm_plane *plane = plane_state->plane;
struct drm_crtc_state *crtc_state;
-
+ /* Nothing to do for same crtc*/
+ if (plane_state->crtc == crtc)
+ return 0;
if (plane_state->crtc) {
crtc_state = drm_atomic_get_crtc_state(plane_state->state,
plane_state->crtc);
diff --git a/drivers/gpu/drm/drm_auth.c b/drivers/gpu/drm/drm_auth.c
index 6b14351..56b2dd9 100644
--- a/drivers/gpu/drm/drm_auth.c
+++ b/drivers/gpu/drm/drm_auth.c
@@ -133,6 +133,7 @@ static int drm_new_set_master(struct drm_device *dev, struct drm_file *fpriv)
lockdep_assert_held_once(&dev->master_mutex);
+ WARN_ON(fpriv->is_master);
old_master = fpriv->master;
fpriv->master = drm_master_create(dev);
if (!fpriv->master) {
@@ -161,6 +162,7 @@ static int drm_new_set_master(struct drm_device *dev, struct drm_file *fpriv)
/* drop references and restore old master on failure */
drm_master_put(&fpriv->master);
fpriv->master = old_master;
+ fpriv->is_master = 0;
return ret;
}
diff --git a/drivers/gpu/drm/drm_dp_dual_mode_helper.c b/drivers/gpu/drm/drm_dp_dual_mode_helper.c
index a7b2a75..cdb5358 100644
--- a/drivers/gpu/drm/drm_dp_dual_mode_helper.c
+++ b/drivers/gpu/drm/drm_dp_dual_mode_helper.c
@@ -322,19 +322,44 @@ int drm_dp_dual_mode_set_tmds_output(enum drm_dp_dual_mode_type type,
{
uint8_t tmds_oen = enable ? 0 : DP_DUAL_MODE_TMDS_DISABLE;
ssize_t ret;
+ int retry;
if (type < DRM_DP_DUAL_MODE_TYPE2_DVI)
return 0;
- ret = drm_dp_dual_mode_write(adapter, DP_DUAL_MODE_TMDS_OEN,
- &tmds_oen, sizeof(tmds_oen));
- if (ret) {
- DRM_DEBUG_KMS("Failed to %s TMDS output buffers\n",
- enable ? "enable" : "disable");
- return ret;
+ /*
+ * LSPCON adapters in low-power state may ignore the first write, so
+ * read back and verify the written value a few times.
+ */
+ for (retry = 0; retry < 3; retry++) {
+ uint8_t tmp;
+
+ ret = drm_dp_dual_mode_write(adapter, DP_DUAL_MODE_TMDS_OEN,
+ &tmds_oen, sizeof(tmds_oen));
+ if (ret) {
+ DRM_DEBUG_KMS("Failed to %s TMDS output buffers (%d attempts)\n",
+ enable ? "enable" : "disable",
+ retry + 1);
+ return ret;
+ }
+
+ ret = drm_dp_dual_mode_read(adapter, DP_DUAL_MODE_TMDS_OEN,
+ &tmp, sizeof(tmp));
+ if (ret) {
+ DRM_DEBUG_KMS("I2C read failed during TMDS output buffer %s (%d attempts)\n",
+ enable ? "enabling" : "disabling",
+ retry + 1);
+ return ret;
+ }
+
+ if (tmp == tmds_oen)
+ return 0;
}
- return 0;
+ DRM_DEBUG_KMS("I2C write value mismatch during TMDS output buffer %s\n",
+ enable ? "enabling" : "disabling");
+
+ return -EIO;
}
EXPORT_SYMBOL(drm_dp_dual_mode_set_tmds_output);
diff --git a/drivers/gpu/drm/drm_dp_helper.c b/drivers/gpu/drm/drm_dp_helper.c
index 6394109..876a2d9 100644
--- a/drivers/gpu/drm/drm_dp_helper.c
+++ b/drivers/gpu/drm/drm_dp_helper.c
@@ -1062,6 +1062,7 @@ int drm_dp_psr_setup_time(const u8 psr_cap[EDP_PSR_RECEIVER_CAP_SIZE])
static const u16 psr_setup_time_us[] = {
PSR_SETUP_TIME(330),
PSR_SETUP_TIME(275),
+ PSR_SETUP_TIME(220),
PSR_SETUP_TIME(165),
PSR_SETUP_TIME(110),
PSR_SETUP_TIME(55),
diff --git a/drivers/gpu/drm/drm_dp_mst_topology.c b/drivers/gpu/drm/drm_dp_mst_topology.c
index db7890c..b59441d 100644
--- a/drivers/gpu/drm/drm_dp_mst_topology.c
+++ b/drivers/gpu/drm/drm_dp_mst_topology.c
@@ -1230,6 +1230,9 @@ static struct drm_dp_mst_branch *drm_dp_get_mst_branch_device(struct drm_dp_mst_
mutex_lock(&mgr->lock);
mstb = mgr->mst_primary;
+ if (!mstb)
+ goto out;
+
for (i = 0; i < lct - 1; i++) {
int shift = (i % 2) ? 0 : 4;
int port_num = (rad[i / 2] >> shift) & 0xf;
diff --git a/drivers/gpu/drm/drm_edid.c b/drivers/gpu/drm/drm_edid.c
index 03a9f20..12a5c2e 100644
--- a/drivers/gpu/drm/drm_edid.c
+++ b/drivers/gpu/drm/drm_edid.c
@@ -115,9 +115,18 @@ static const struct edid_quirk {
/* AEO model 0 reports 8 bpc, but is a 6 bpc panel */
{ "AEO", 0, EDID_QUIRK_FORCE_6BPC },
+ /* BOE model on HP Pavilion 15-n233sl reports 8 bpc, but is a 6 bpc panel */
+ { "BOE", 0x78b, EDID_QUIRK_FORCE_6BPC },
+
/* CPT panel of Asus UX303LA reports 8 bpc, but is a 6 bpc panel */
{ "CPT", 0x17df, EDID_QUIRK_FORCE_6BPC },
+ /* SDC panel of Lenovo B50-80 reports 8 bpc, but is a 6 bpc panel */
+ { "SDC", 0x3652, EDID_QUIRK_FORCE_6BPC },
+
+ /* BOE model 0x0771 reports 8 bpc, but is a 6 bpc panel */
+ { "BOE", 0x0771, EDID_QUIRK_FORCE_6BPC },
+
/* Belinea 10 15 55 */
{ "MAX", 1516, EDID_QUIRK_PREFER_LARGE_60 },
{ "MAX", 0x77e, EDID_QUIRK_PREFER_LARGE_60 },
diff --git a/drivers/gpu/drm/drm_fops.c b/drivers/gpu/drm/drm_fops.c
index ca227e8..847ba40 100644
--- a/drivers/gpu/drm/drm_fops.c
+++ b/drivers/gpu/drm/drm_fops.c
@@ -205,6 +205,7 @@ static int drm_open_helper(struct file *filp, struct drm_minor *minor)
return -ENOMEM;
filp->private_data = priv;
+ filp->f_mode |= FMODE_UNSIGNED_OFFSET;
priv->filp = filp;
priv->pid = get_pid(task_pid(current));
priv->minor = minor;
diff --git a/drivers/gpu/drm/drm_ioctl.c b/drivers/gpu/drm/drm_ioctl.c
index 71c3473..04b26ca 100644
--- a/drivers/gpu/drm/drm_ioctl.c
+++ b/drivers/gpu/drm/drm_ioctl.c
@@ -36,6 +36,7 @@
#include <linux/pci.h>
#include <linux/export.h>
+#include <linux/nospec.h>
/**
* DOC: getunique and setversion story
@@ -668,13 +669,17 @@ long drm_ioctl(struct file *filp,
if (is_driver_ioctl) {
/* driver ioctl */
- if (nr - DRM_COMMAND_BASE >= dev->driver->num_ioctls)
+ unsigned int index = nr - DRM_COMMAND_BASE;
+
+ if (index >= dev->driver->num_ioctls)
goto err_i1;
- ioctl = &dev->driver->ioctls[nr - DRM_COMMAND_BASE];
+ index = array_index_nospec(index, dev->driver->num_ioctls);
+ ioctl = &dev->driver->ioctls[index];
} else {
/* core ioctl */
if (nr >= DRM_CORE_IOCTL_COUNT)
goto err_i1;
+ nr = array_index_nospec(nr, DRM_CORE_IOCTL_COUNT);
ioctl = &drm_ioctls[nr];
}
@@ -770,6 +775,7 @@ bool drm_ioctl_flags(unsigned int nr, unsigned int *flags)
if (nr >= DRM_CORE_IOCTL_COUNT)
return false;
+ nr = array_index_nospec(nr, DRM_CORE_IOCTL_COUNT);
*flags = drm_ioctls[nr].flags;
return true;
diff --git a/drivers/gpu/drm/drm_mipi_dsi.c b/drivers/gpu/drm/drm_mipi_dsi.c
index d9a5762..d518ff5 100644
--- a/drivers/gpu/drm/drm_mipi_dsi.c
+++ b/drivers/gpu/drm/drm_mipi_dsi.c
@@ -362,6 +362,8 @@ static ssize_t mipi_dsi_device_transfer(struct mipi_dsi_device *dsi,
msg->flags |= MIPI_DSI_MSG_USE_LPM;
msg->flags |= MIPI_DSI_MSG_LASTCOMMAND;
+ msg->flags |= MIPI_DSI_MSG_LASTCOMMAND;
+
return ops->transfer(dsi->host, msg);
}
@@ -1049,17 +1051,33 @@ EXPORT_SYMBOL(mipi_dsi_dcs_set_tear_scanline);
* display
* @dsi: DSI peripheral device
* @brightness: brightness value
+ * @num_params: Number of parameters (bytes) to encode brightness value in. The
+ * MIPI specification states that one parameter shall be sent for
+ * devices that support 8-bit brightness levels. For devices that
+ * support brightness levels wider than 8-bit, two parameters
+ * shall be sent.
*
* Return: 0 on success or a negative error code on failure.
*/
int mipi_dsi_dcs_set_display_brightness(struct mipi_dsi_device *dsi,
- u16 brightness)
+ u16 brightness, size_t num_params)
{
- u8 payload[2] = { brightness & 0xff, brightness >> 8 };
+ u8 payload[2];
ssize_t err;
+ switch (num_params) {
+ case 1:
+ payload[0] = brightness & 0xff;
+ break;
+ case 2:
+ payload[0] = brightness >> 8;
+ payload[1] = brightness & 0xff;
+ break;
+ default:
+ return -EINVAL;
+ }
err = mipi_dsi_dcs_write(dsi, MIPI_DCS_SET_DISPLAY_BRIGHTNESS,
- payload, sizeof(payload));
+ payload, num_params);
if (err < 0)
return err;
@@ -1072,16 +1090,25 @@ EXPORT_SYMBOL(mipi_dsi_dcs_set_display_brightness);
* of the display
* @dsi: DSI peripheral device
* @brightness: brightness value
+ * @num_params: Number of parameters (i.e. bytes) the brightness value is
+ * encoded in. The MIPI specification states that one parameter
+ * shall be returned from devices that support 8-bit brightness
+ * levels. Devices that support brightness levels wider than
+ * 8-bit return two parameters (i.e. bytes).
*
* Return: 0 on success or a negative error code on failure.
*/
int mipi_dsi_dcs_get_display_brightness(struct mipi_dsi_device *dsi,
- u16 *brightness)
+ u16 *brightness, size_t num_params)
{
+ u8 payload[2];
ssize_t err;
+ if (!(num_params == 1 || num_params == 2))
+ return -EINVAL;
+
err = mipi_dsi_dcs_read(dsi, MIPI_DCS_GET_DISPLAY_BRIGHTNESS,
- brightness, sizeof(*brightness));
+ payload, num_params);
if (err <= 0) {
if (err == 0)
err = -ENODATA;
@@ -1089,6 +1116,15 @@ int mipi_dsi_dcs_get_display_brightness(struct mipi_dsi_device *dsi,
return err;
}
+ switch (num_params) {
+ case 1:
+ *brightness = payload[0];
+ break;
+ case 2:
+ *brightness = payload[0] << 8 || payload[1];
+ break;
+ }
+
return 0;
}
EXPORT_SYMBOL(mipi_dsi_dcs_get_display_brightness);
diff --git a/drivers/gpu/drm/exynos/exynos5433_drm_decon.c b/drivers/gpu/drm/exynos/exynos5433_drm_decon.c
index 6dd09c3..bdcc6ec 100644
--- a/drivers/gpu/drm/exynos/exynos5433_drm_decon.c
+++ b/drivers/gpu/drm/exynos/exynos5433_drm_decon.c
@@ -199,7 +199,7 @@ static void decon_win_set_pixfmt(struct decon_context *ctx, unsigned int win,
unsigned long val;
val = readl(ctx->addr + DECON_WINCONx(win));
- val &= ~WINCONx_BPPMODE_MASK;
+ val &= WINCONx_ENWIN_F;
switch (fb->pixel_format) {
case DRM_FORMAT_XRGB1555:
@@ -291,8 +291,8 @@ static void decon_update_plane(struct exynos_drm_crtc *crtc,
COORDINATE_Y(state->crtc.y + state->crtc.h - 1);
writel(val, ctx->addr + DECON_VIDOSDxB(win));
- val = VIDOSD_Wx_ALPHA_R_F(0x0) | VIDOSD_Wx_ALPHA_G_F(0x0) |
- VIDOSD_Wx_ALPHA_B_F(0x0);
+ val = VIDOSD_Wx_ALPHA_R_F(0xff) | VIDOSD_Wx_ALPHA_G_F(0xff) |
+ VIDOSD_Wx_ALPHA_B_F(0xff);
writel(val, ctx->addr + DECON_VIDOSDxC(win));
val = VIDOSD_Wx_ALPHA_R_F(0x0) | VIDOSD_Wx_ALPHA_G_F(0x0) |
diff --git a/drivers/gpu/drm/exynos/exynos_drm_g2d.c b/drivers/gpu/drm/exynos/exynos_drm_g2d.c
index 603d842..699db13 100644
--- a/drivers/gpu/drm/exynos/exynos_drm_g2d.c
+++ b/drivers/gpu/drm/exynos/exynos_drm_g2d.c
@@ -926,7 +926,7 @@ static void g2d_finish_event(struct g2d_data *g2d, u32 cmdlist_no)
struct drm_device *drm_dev = g2d->subdrv.drm_dev;
struct g2d_runqueue_node *runqueue_node = g2d->runqueue_node;
struct drm_exynos_pending_g2d_event *e;
- struct timeval now;
+ struct timespec64 now;
if (list_empty(&runqueue_node->event_list))
return;
@@ -934,9 +934,9 @@ static void g2d_finish_event(struct g2d_data *g2d, u32 cmdlist_no)
e = list_first_entry(&runqueue_node->event_list,
struct drm_exynos_pending_g2d_event, base.link);
- do_gettimeofday(&now);
+ ktime_get_ts64(&now);
e->event.tv_sec = now.tv_sec;
- e->event.tv_usec = now.tv_usec;
+ e->event.tv_usec = now.tv_nsec / NSEC_PER_USEC;
e->event.cmdlist_no = cmdlist_no;
drm_send_event(drm_dev, &e->base);
diff --git a/drivers/gpu/drm/exynos/exynos_drm_gsc.c b/drivers/gpu/drm/exynos/exynos_drm_gsc.c
index 52a9d26..4c81c79 100644
--- a/drivers/gpu/drm/exynos/exynos_drm_gsc.c
+++ b/drivers/gpu/drm/exynos/exynos_drm_gsc.c
@@ -532,21 +532,25 @@ static int gsc_src_set_fmt(struct device *dev, u32 fmt)
GSC_IN_CHROMA_ORDER_CRCB);
break;
case DRM_FORMAT_NV21:
+ cfg |= (GSC_IN_CHROMA_ORDER_CRCB | GSC_IN_YUV420_2P);
+ break;
case DRM_FORMAT_NV61:
- cfg |= (GSC_IN_CHROMA_ORDER_CRCB |
- GSC_IN_YUV420_2P);
+ cfg |= (GSC_IN_CHROMA_ORDER_CRCB | GSC_IN_YUV422_2P);
break;
case DRM_FORMAT_YUV422:
cfg |= GSC_IN_YUV422_3P;
break;
case DRM_FORMAT_YUV420:
+ cfg |= (GSC_IN_CHROMA_ORDER_CBCR | GSC_IN_YUV420_3P);
+ break;
case DRM_FORMAT_YVU420:
- cfg |= GSC_IN_YUV420_3P;
+ cfg |= (GSC_IN_CHROMA_ORDER_CRCB | GSC_IN_YUV420_3P);
break;
case DRM_FORMAT_NV12:
+ cfg |= (GSC_IN_CHROMA_ORDER_CBCR | GSC_IN_YUV420_2P);
+ break;
case DRM_FORMAT_NV16:
- cfg |= (GSC_IN_CHROMA_ORDER_CBCR |
- GSC_IN_YUV420_2P);
+ cfg |= (GSC_IN_CHROMA_ORDER_CBCR | GSC_IN_YUV422_2P);
break;
default:
dev_err(ippdrv->dev, "invalid target yuv order 0x%x.\n", fmt);
@@ -806,18 +810,25 @@ static int gsc_dst_set_fmt(struct device *dev, u32 fmt)
GSC_OUT_CHROMA_ORDER_CRCB);
break;
case DRM_FORMAT_NV21:
- case DRM_FORMAT_NV61:
cfg |= (GSC_OUT_CHROMA_ORDER_CRCB | GSC_OUT_YUV420_2P);
break;
+ case DRM_FORMAT_NV61:
+ cfg |= (GSC_OUT_CHROMA_ORDER_CRCB | GSC_OUT_YUV422_2P);
+ break;
case DRM_FORMAT_YUV422:
+ cfg |= GSC_OUT_YUV422_3P;
+ break;
case DRM_FORMAT_YUV420:
+ cfg |= (GSC_OUT_CHROMA_ORDER_CBCR | GSC_OUT_YUV420_3P);
+ break;
case DRM_FORMAT_YVU420:
- cfg |= GSC_OUT_YUV420_3P;
+ cfg |= (GSC_OUT_CHROMA_ORDER_CRCB | GSC_OUT_YUV420_3P);
break;
case DRM_FORMAT_NV12:
+ cfg |= (GSC_OUT_CHROMA_ORDER_CBCR | GSC_OUT_YUV420_2P);
+ break;
case DRM_FORMAT_NV16:
- cfg |= (GSC_OUT_CHROMA_ORDER_CBCR |
- GSC_OUT_YUV420_2P);
+ cfg |= (GSC_OUT_CHROMA_ORDER_CBCR | GSC_OUT_YUV422_2P);
break;
default:
dev_err(ippdrv->dev, "invalid target yuv order 0x%x.\n", fmt);
diff --git a/drivers/gpu/drm/exynos/regs-fimc.h b/drivers/gpu/drm/exynos/regs-fimc.h
index 3049613..d7cbe53 100644
--- a/drivers/gpu/drm/exynos/regs-fimc.h
+++ b/drivers/gpu/drm/exynos/regs-fimc.h
@@ -569,7 +569,7 @@
#define EXYNOS_CIIMGEFF_FIN_EMBOSSING (4 << 26)
#define EXYNOS_CIIMGEFF_FIN_SILHOUETTE (5 << 26)
#define EXYNOS_CIIMGEFF_FIN_MASK (7 << 26)
-#define EXYNOS_CIIMGEFF_PAT_CBCR_MASK ((0xff < 13) | (0xff < 0))
+#define EXYNOS_CIIMGEFF_PAT_CBCR_MASK ((0xff << 13) | (0xff << 0))
/* Real input DMA size register */
#define EXYNOS_CIREAL_ISIZE_AUTOLOAD_ENABLE (1 << 31)
diff --git a/drivers/gpu/drm/exynos/regs-gsc.h b/drivers/gpu/drm/exynos/regs-gsc.h
index 4704a99..16b39734 100644
--- a/drivers/gpu/drm/exynos/regs-gsc.h
+++ b/drivers/gpu/drm/exynos/regs-gsc.h
@@ -138,6 +138,7 @@
#define GSC_OUT_YUV420_3P (3 << 4)
#define GSC_OUT_YUV422_1P (4 << 4)
#define GSC_OUT_YUV422_2P (5 << 4)
+#define GSC_OUT_YUV422_3P (6 << 4)
#define GSC_OUT_YUV444 (7 << 4)
#define GSC_OUT_TILE_TYPE_MASK (1 << 2)
#define GSC_OUT_TILE_C_16x8 (0 << 2)
diff --git a/drivers/gpu/drm/gma500/mdfld_intel_display.c b/drivers/gpu/drm/gma500/mdfld_intel_display.c
index 92e3f93e..06d61e6 100644
--- a/drivers/gpu/drm/gma500/mdfld_intel_display.c
+++ b/drivers/gpu/drm/gma500/mdfld_intel_display.c
@@ -99,7 +99,7 @@ void mdfldWaitForPipeEnable(struct drm_device *dev, int pipe)
/* Wait for for the pipe enable to take effect. */
for (count = 0; count < COUNT_MAX; count++) {
temp = REG_READ(map->conf);
- if ((temp & PIPEACONF_PIPE_STATE) == 1)
+ if (temp & PIPEACONF_PIPE_STATE)
break;
}
}
diff --git a/drivers/gpu/drm/gma500/psb_intel_drv.h b/drivers/gpu/drm/gma500/psb_intel_drv.h
index 2a3b7c6..fbd3fa3 100644
--- a/drivers/gpu/drm/gma500/psb_intel_drv.h
+++ b/drivers/gpu/drm/gma500/psb_intel_drv.h
@@ -255,7 +255,7 @@ extern int intelfb_remove(struct drm_device *dev,
extern bool psb_intel_lvds_mode_fixup(struct drm_encoder *encoder,
const struct drm_display_mode *mode,
struct drm_display_mode *adjusted_mode);
-extern int psb_intel_lvds_mode_valid(struct drm_connector *connector,
+extern enum drm_mode_status psb_intel_lvds_mode_valid(struct drm_connector *connector,
struct drm_display_mode *mode);
extern int psb_intel_lvds_set_property(struct drm_connector *connector,
struct drm_property *property,
diff --git a/drivers/gpu/drm/gma500/psb_intel_lvds.c b/drivers/gpu/drm/gma500/psb_intel_lvds.c
index 79e9d36..e2c6ba3 100644
--- a/drivers/gpu/drm/gma500/psb_intel_lvds.c
+++ b/drivers/gpu/drm/gma500/psb_intel_lvds.c
@@ -343,7 +343,7 @@ static void psb_intel_lvds_restore(struct drm_connector *connector)
}
}
-int psb_intel_lvds_mode_valid(struct drm_connector *connector,
+enum drm_mode_status psb_intel_lvds_mode_valid(struct drm_connector *connector,
struct drm_display_mode *mode)
{
struct drm_psb_private *dev_priv = connector->dev->dev_private;
diff --git a/drivers/gpu/drm/i915/i915_drv.h b/drivers/gpu/drm/i915/i915_drv.h
index 36a665f..e23748c 100644
--- a/drivers/gpu/drm/i915/i915_drv.h
+++ b/drivers/gpu/drm/i915/i915_drv.h
@@ -3681,7 +3681,11 @@ extern void intel_display_print_error_state(struct drm_i915_error_state_buf *e,
struct intel_display_error_state *error);
int sandybridge_pcode_read(struct drm_i915_private *dev_priv, u32 mbox, u32 *val);
-int sandybridge_pcode_write(struct drm_i915_private *dev_priv, u32 mbox, u32 val);
+int sandybridge_pcode_write_timeout(struct drm_i915_private *dev_priv, u32 mbox,
+ u32 val, int timeout_us);
+#define sandybridge_pcode_write(dev_priv, mbox, val) \
+ sandybridge_pcode_write_timeout(dev_priv, mbox, val, 500)
+
int skl_pcode_request(struct drm_i915_private *dev_priv, u32 mbox, u32 request,
u32 reply_mask, u32 reply, int timeout_base_ms);
diff --git a/drivers/gpu/drm/i915/i915_gem_userptr.c b/drivers/gpu/drm/i915/i915_gem_userptr.c
index c6f780f..555fd47 100644
--- a/drivers/gpu/drm/i915/i915_gem_userptr.c
+++ b/drivers/gpu/drm/i915/i915_gem_userptr.c
@@ -778,6 +778,9 @@ i915_gem_userptr_ioctl(struct drm_device *dev, void *data, struct drm_file *file
I915_USERPTR_UNSYNCHRONIZED))
return -EINVAL;
+ if (!args->user_size)
+ return -EINVAL;
+
if (offset_in_page(args->user_ptr | args->user_size))
return -EINVAL;
diff --git a/drivers/gpu/drm/i915/i915_irq.c b/drivers/gpu/drm/i915/i915_irq.c
index 02908e3..279d1e0 100644
--- a/drivers/gpu/drm/i915/i915_irq.c
+++ b/drivers/gpu/drm/i915/i915_irq.c
@@ -1684,10 +1684,38 @@ static void valleyview_pipestat_irq_handler(struct drm_i915_private *dev_priv,
static u32 i9xx_hpd_irq_ack(struct drm_i915_private *dev_priv)
{
- u32 hotplug_status = I915_READ(PORT_HOTPLUG_STAT);
+ u32 hotplug_status = 0, hotplug_status_mask;
+ int i;
- if (hotplug_status)
+ if (IS_G4X(dev_priv) ||
+ IS_VALLEYVIEW(dev_priv) || IS_CHERRYVIEW(dev_priv))
+ hotplug_status_mask = HOTPLUG_INT_STATUS_G4X |
+ DP_AUX_CHANNEL_MASK_INT_STATUS_G4X;
+ else
+ hotplug_status_mask = HOTPLUG_INT_STATUS_I915;
+
+ /*
+ * We absolutely have to clear all the pending interrupt
+ * bits in PORT_HOTPLUG_STAT. Otherwise the ISR port
+ * interrupt bit won't have an edge, and the i965/g4x
+ * edge triggered IIR will not notice that an interrupt
+ * is still pending. We can't use PORT_HOTPLUG_EN to
+ * guarantee the edge as the act of toggling the enable
+ * bits can itself generate a new hotplug interrupt :(
+ */
+ for (i = 0; i < 10; i++) {
+ u32 tmp = I915_READ(PORT_HOTPLUG_STAT) & hotplug_status_mask;
+
+ if (tmp == 0)
+ return hotplug_status;
+
+ hotplug_status |= tmp;
I915_WRITE(PORT_HOTPLUG_STAT, hotplug_status);
+ }
+
+ WARN_ONCE(1,
+ "PORT_HOTPLUG_STAT did not clear (0x%08x)\n",
+ I915_READ(PORT_HOTPLUG_STAT));
return hotplug_status;
}
diff --git a/drivers/gpu/drm/i915/intel_audio.c b/drivers/gpu/drm/i915/intel_audio.c
index 6c70a5b..8405228 100644
--- a/drivers/gpu/drm/i915/intel_audio.c
+++ b/drivers/gpu/drm/i915/intel_audio.c
@@ -76,6 +76,9 @@ static const struct {
/* HDMI N/CTS table */
#define TMDS_297M 297000
#define TMDS_296M 296703
+#define TMDS_594M 594000
+#define TMDS_593M 593407
+
static const struct {
int sample_rate;
int clock;
@@ -96,6 +99,20 @@ static const struct {
{ 176400, TMDS_297M, 18816, 247500 },
{ 192000, TMDS_296M, 23296, 281250 },
{ 192000, TMDS_297M, 20480, 247500 },
+ { 44100, TMDS_593M, 8918, 937500 },
+ { 44100, TMDS_594M, 9408, 990000 },
+ { 48000, TMDS_593M, 5824, 562500 },
+ { 48000, TMDS_594M, 6144, 594000 },
+ { 32000, TMDS_593M, 5824, 843750 },
+ { 32000, TMDS_594M, 3072, 445500 },
+ { 88200, TMDS_593M, 17836, 937500 },
+ { 88200, TMDS_594M, 18816, 990000 },
+ { 96000, TMDS_593M, 11648, 562500 },
+ { 96000, TMDS_594M, 12288, 594000 },
+ { 176400, TMDS_593M, 35672, 937500 },
+ { 176400, TMDS_594M, 37632, 990000 },
+ { 192000, TMDS_593M, 23296, 562500 },
+ { 192000, TMDS_594M, 24576, 594000 },
};
/* get AUD_CONFIG_PIXEL_CLOCK_HDMI_* value for mode */
diff --git a/drivers/gpu/drm/i915/intel_display.c b/drivers/gpu/drm/i915/intel_display.c
index ce32303..c185625 100644
--- a/drivers/gpu/drm/i915/intel_display.c
+++ b/drivers/gpu/drm/i915/intel_display.c
@@ -6012,8 +6012,8 @@ static void bxt_set_cdclk(struct drm_i915_private *dev_priv, int cdclk)
/* Inform power controller of upcoming frequency change */
mutex_lock(&dev_priv->rps.hw_lock);
- ret = sandybridge_pcode_write(dev_priv, HSW_PCODE_DE_WRITE_FREQ_REQ,
- 0x80000000);
+ ret = sandybridge_pcode_write_timeout(dev_priv, HSW_PCODE_DE_WRITE_FREQ_REQ,
+ 0x80000000, 2000);
mutex_unlock(&dev_priv->rps.hw_lock);
if (ret) {
@@ -6044,8 +6044,9 @@ static void bxt_set_cdclk(struct drm_i915_private *dev_priv, int cdclk)
I915_WRITE(CDCLK_CTL, val);
mutex_lock(&dev_priv->rps.hw_lock);
- ret = sandybridge_pcode_write(dev_priv, HSW_PCODE_DE_WRITE_FREQ_REQ,
- DIV_ROUND_UP(cdclk, 25000));
+ ret = sandybridge_pcode_write_timeout(dev_priv,
+ HSW_PCODE_DE_WRITE_FREQ_REQ,
+ DIV_ROUND_UP(cdclk, 25000), 2000);
mutex_unlock(&dev_priv->rps.hw_lock);
if (ret) {
diff --git a/drivers/gpu/drm/i915/intel_lrc.c b/drivers/gpu/drm/i915/intel_lrc.c
index 67db157..62bcc77 100644
--- a/drivers/gpu/drm/i915/intel_lrc.c
+++ b/drivers/gpu/drm/i915/intel_lrc.c
@@ -368,7 +368,8 @@ static u64 execlists_update_context(struct drm_i915_gem_request *rq)
reg_state[CTX_RING_TAIL+1] = intel_ring_offset(rq->ring, rq->tail);
- /* True 32b PPGTT with dynamic page allocation: update PDP
+ /*
+ * True 32b PPGTT with dynamic page allocation: update PDP
* registers and point the unallocated PDPs to scratch page.
* PML4 is allocated during ppgtt init, so this is not needed
* in 48-bit mode.
@@ -376,6 +377,22 @@ static u64 execlists_update_context(struct drm_i915_gem_request *rq)
if (ppgtt && !USES_FULL_48BIT_PPGTT(ppgtt->base.dev))
execlists_update_context_pdps(ppgtt, reg_state);
+ /*
+ * Make sure the context image is complete before we submit it to HW.
+ *
+ * Ostensibly, writes (including the WCB) should be flushed prior to
+ * an uncached write such as our mmio register access, the empirical
+ * evidence (esp. on Braswell) suggests that the WC write into memory
+ * may not be visible to the HW prior to the completion of the UC
+ * register write and that we may begin execution from the context
+ * before its image is complete leading to invalid PD chasing.
+ *
+ * Furthermore, Braswell, at least, wants a full mb to be sure that
+ * the writes are coherent in memory (visible to the GPU) prior to
+ * execution, and not just visible to other CPUs (as is the result of
+ * wmb).
+ */
+ mb();
return ce->lrc_desc;
}
diff --git a/drivers/gpu/drm/i915/intel_lvds.c b/drivers/gpu/drm/i915/intel_lvds.c
index e1d47d5..479d641 100644
--- a/drivers/gpu/drm/i915/intel_lvds.c
+++ b/drivers/gpu/drm/i915/intel_lvds.c
@@ -321,7 +321,8 @@ static void intel_enable_lvds(struct intel_encoder *encoder,
I915_WRITE(PP_CONTROL(0), I915_READ(PP_CONTROL(0)) | PANEL_POWER_ON);
POSTING_READ(lvds_encoder->reg);
- if (intel_wait_for_register(dev_priv, PP_STATUS(0), PP_ON, PP_ON, 1000))
+
+ if (intel_wait_for_register(dev_priv, PP_STATUS(0), PP_ON, PP_ON, 5000))
DRM_ERROR("timed out waiting for panel to power on\n");
intel_panel_enable_backlight(intel_connector);
@@ -863,6 +864,14 @@ static const struct dmi_system_id intel_no_lvds[] = {
DMI_EXACT_MATCH(DMI_BOARD_NAME, "D525MW"),
},
},
+ {
+ .callback = intel_no_lvds_dmi_callback,
+ .ident = "Radiant P845",
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Radiant Systems Inc"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "P845"),
+ },
+ },
{ } /* terminating entry */
};
diff --git a/drivers/gpu/drm/i915/intel_pm.c b/drivers/gpu/drm/i915/intel_pm.c
index 49de476..05427d2 100644
--- a/drivers/gpu/drm/i915/intel_pm.c
+++ b/drivers/gpu/drm/i915/intel_pm.c
@@ -7913,8 +7913,8 @@ int sandybridge_pcode_read(struct drm_i915_private *dev_priv, u32 mbox, u32 *val
return 0;
}
-int sandybridge_pcode_write(struct drm_i915_private *dev_priv,
- u32 mbox, u32 val)
+int sandybridge_pcode_write_timeout(struct drm_i915_private *dev_priv,
+ u32 mbox, u32 val, int timeout_us)
{
int status;
@@ -7935,7 +7935,7 @@ int sandybridge_pcode_write(struct drm_i915_private *dev_priv,
if (intel_wait_for_register_fw(dev_priv,
GEN6_PCODE_MAILBOX, GEN6_PCODE_READY, 0,
- 500)) {
+ timeout_us)) {
DRM_ERROR("timeout waiting for pcode write (%d) to finish\n", mbox);
return -ETIMEDOUT;
}
diff --git a/drivers/gpu/drm/imx/imx-ldb.c b/drivers/gpu/drm/imx/imx-ldb.c
index 3ce391c2..67881e5 100644
--- a/drivers/gpu/drm/imx/imx-ldb.c
+++ b/drivers/gpu/drm/imx/imx-ldb.c
@@ -634,6 +634,9 @@ static int imx_ldb_bind(struct device *dev, struct device *master, void *data)
return PTR_ERR(imx_ldb->regmap);
}
+ /* disable LDB by resetting the control register to POR default */
+ regmap_write(imx_ldb->regmap, IOMUXC_GPR2, 0);
+
imx_ldb->dev = dev;
if (of_id)
@@ -675,14 +678,14 @@ static int imx_ldb_bind(struct device *dev, struct device *master, void *data)
if (ret || i < 0 || i > 1)
return -EINVAL;
+ if (!of_device_is_available(child))
+ continue;
+
if (dual && i > 0) {
dev_warn(dev, "dual-channel mode, ignoring second output\n");
continue;
}
- if (!of_device_is_available(child))
- continue;
-
channel = &imx_ldb->channel[i];
channel->ldb = imx_ldb;
channel->chno = i;
diff --git a/drivers/gpu/drm/imx/ipuv3-crtc.c b/drivers/gpu/drm/imx/ipuv3-crtc.c
index 6be515a..8dbba61 100644
--- a/drivers/gpu/drm/imx/ipuv3-crtc.c
+++ b/drivers/gpu/drm/imx/ipuv3-crtc.c
@@ -189,7 +189,11 @@ static void ipu_crtc_atomic_begin(struct drm_crtc *crtc,
struct drm_crtc_state *old_crtc_state)
{
drm_crtc_vblank_on(crtc);
+}
+static void ipu_crtc_atomic_flush(struct drm_crtc *crtc,
+ struct drm_crtc_state *old_crtc_state)
+{
spin_lock_irq(&crtc->dev->event_lock);
if (crtc->state->event) {
WARN_ON(drm_crtc_vblank_get(crtc));
@@ -257,6 +261,7 @@ static const struct drm_crtc_helper_funcs ipu_helper_funcs = {
.mode_set_nofb = ipu_crtc_mode_set_nofb,
.atomic_check = ipu_crtc_atomic_check,
.atomic_begin = ipu_crtc_atomic_begin,
+ .atomic_flush = ipu_crtc_atomic_flush,
.atomic_disable = ipu_crtc_atomic_disable,
.enable = ipu_crtc_enable,
};
diff --git a/drivers/gpu/drm/mediatek/mtk_hdmi.c b/drivers/gpu/drm/mediatek/mtk_hdmi.c
index e097780..863d030 100644
--- a/drivers/gpu/drm/mediatek/mtk_hdmi.c
+++ b/drivers/gpu/drm/mediatek/mtk_hdmi.c
@@ -1446,8 +1446,7 @@ static int mtk_hdmi_dt_parse_pdata(struct mtk_hdmi *hdmi,
}
/* The CEC module handles HDMI hotplug detection */
- cec_np = of_find_compatible_node(np->parent, NULL,
- "mediatek,mt8173-cec");
+ cec_np = of_get_compatible_child(np->parent, "mediatek,mt8173-cec");
if (!cec_np) {
dev_err(dev, "Failed to find CEC node\n");
return -EINVAL;
@@ -1457,8 +1456,10 @@ static int mtk_hdmi_dt_parse_pdata(struct mtk_hdmi *hdmi,
if (!cec_pdev) {
dev_err(hdmi->dev, "Waiting for CEC device %s\n",
cec_np->full_name);
+ of_node_put(cec_np);
return -EPROBE_DEFER;
}
+ of_node_put(cec_np);
hdmi->cec_dev = &cec_pdev->dev;
/*
diff --git a/drivers/gpu/drm/msm/Makefile b/drivers/gpu/drm/msm/Makefile
index c81832a..c0b20a1 100644
--- a/drivers/gpu/drm/msm/Makefile
+++ b/drivers/gpu/drm/msm/Makefile
@@ -134,6 +134,7 @@
dsi-staging/dsi_drm.o \
dsi-staging/dsi_display.o \
dsi-staging/dsi_panel.o \
+ dsi-staging/dsi_backlight.o \
dsi-staging/dsi_clk_manager.o \
dsi-staging/dsi_display_test.o
@@ -187,3 +188,11 @@
msm_debugfs.o
obj-$(CONFIG_DRM_MSM) += msm_drm.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-header-guard \
+ -Wno-pointer-bool-conversion \
+ -Wno-duplicate-decl-specifier \
+
+CFLAGS_sde_crtc.o += -Wframe-larger-than=3072
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_backlight.c b/drivers/gpu/drm/msm/dsi-staging/dsi_backlight.c
new file mode 100644
index 0000000..391a2c9
--- /dev/null
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_backlight.c
@@ -0,0 +1,1526 @@
+/*
+ * Copyright (c) 2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#define pr_fmt(fmt) "%s:%d: " fmt, __func__, __LINE__
+#include <linux/backlight.h>
+#include <linux/of_gpio.h>
+#include <linux/sysfs.h>
+#include <linux/list.h>
+#include <linux/list_sort.h>
+#include <video/mipi_display.h>
+
+#include <dsi_drm.h>
+#include <sde_crtc.h>
+#include <sde_encoder.h>
+
+#include "dsi_display.h"
+#include "dsi_panel.h"
+
+#define BL_NODE_NAME_SIZE 32
+
+#define BL_STATE_STANDBY BL_CORE_FBBLANK
+#define BL_STATE_LP BL_CORE_DRIVER1
+#define BL_STATE_LP2 BL_CORE_DRIVER2
+
+struct dsi_backlight_pwm_config {
+ bool pwm_pmi_control;
+ u32 pwm_pmic_bank;
+ u32 pwm_period_usecs;
+ int pwm_gpio;
+};
+
+static void dsi_panel_bl_hbm_free(struct device *dev,
+ struct dsi_backlight_config *bl);
+
+static inline bool is_standby_mode(unsigned long state)
+{
+ return (state & BL_STATE_STANDBY) != 0;
+}
+
+static inline bool is_lp_mode(unsigned long state)
+{
+ return (state & (BL_STATE_LP | BL_STATE_LP2)) != 0;
+}
+
+static inline unsigned int regulator_mode_from_state(unsigned long state)
+{
+ if (is_standby_mode(state))
+ return REGULATOR_MODE_STANDBY;
+ else if (is_lp_mode(state))
+ return REGULATOR_MODE_IDLE;
+ else
+ return REGULATOR_MODE_NORMAL;
+}
+
+static int dsi_panel_pwm_bl_register(struct dsi_backlight_config *bl);
+
+static void dsi_panel_bl_free_unregister(struct dsi_backlight_config *bl)
+{
+ kfree(bl->priv);
+}
+
+static int dsi_backlight_update_dcs(struct dsi_backlight_config *bl, u32 bl_lvl)
+{
+ int rc = 0;
+ struct dsi_panel *panel;
+ struct mipi_dsi_device *dsi;
+ size_t num_params;
+
+ if (!bl || (bl_lvl > 0xffff)) {
+ pr_err("invalid params\n");
+ return -EINVAL;
+ }
+
+ panel = container_of(bl, struct dsi_panel, bl_config);
+ /* if no change in backlight, abort */
+ if (bl_lvl == bl->bl_actual)
+ return 0;
+
+ dsi = &panel->mipi_device;
+
+ num_params = bl->bl_max_level > 0xFF ? 2 : 1;
+ rc = mipi_dsi_dcs_set_display_brightness(dsi, bl_lvl, num_params);
+ if (rc < 0)
+ pr_err("failed to update dcs backlight:%d\n", bl_lvl);
+
+ return rc;
+}
+
+/* Linearly interpolate value x from range [x1, x2] to determine the
+ * corresponding value in range [y1, y2].
+ */
+static int dsi_backlight_lerp(u16 x1, u16 x2, u16 y1, u16 y2, u16 x, u32 *y)
+{
+ if ((x2 < x1) || (y2 < y1))
+ return -EINVAL;
+
+ if (((x2 - x1) == 0) || (x <= x1))
+ *y = y1;
+ else if (x >= x2)
+ *y = y2;
+ else
+ *y = DIV_ROUND_CLOSEST((x - x1) * (y2 - y1), x2 - x1) + y1;
+
+ return 0;
+}
+
+static u32 dsi_backlight_calculate_normal(struct dsi_backlight_config *bl,
+ int brightness)
+{
+ u32 bl_lvl = 0;
+ int rc = 0;
+
+ if (bl->lut) {
+ /*
+ * look up panel brightness; the first entry in the LUT
+ corresponds to userspace brightness level 1
+ */
+ if (WARN_ON(brightness > bl->brightness_max_level))
+ bl_lvl = bl->lut[bl->brightness_max_level];
+ else
+ bl_lvl = bl->lut[brightness];
+ } else {
+ /* map UI brightness into driver backlight level rounding it */
+ rc = dsi_backlight_lerp(
+ 1, bl->brightness_max_level,
+ bl->bl_min_level ? : 1, bl->bl_max_level,
+ brightness, &bl_lvl);
+ if (unlikely(rc))
+ pr_err("failed to linearly interpolate, brightness unmodified\n");
+ }
+
+ pr_debug("normal bl: bl_lut %sused\n", bl->lut ? "" : "un");
+
+ return bl_lvl;
+}
+
+int dsi_backlight_hbm_dimming_start(struct dsi_backlight_config *bl,
+ u32 num_frames, struct dsi_panel_cmd_set *stop_cmd)
+{
+ struct hbm_data *hbm = bl->hbm;
+
+ if (!hbm || !num_frames)
+ return 0;
+
+ if (unlikely(!hbm->dimming_workq)) {
+ pr_err("hbm: tried to start dimming, but missing worker thread\n");
+ return -EINVAL;
+ }
+
+ if (!hbm->dimming_active) {
+ struct dsi_display *display =
+ dev_get_drvdata(hbm->panel->parent);
+ int rc;
+
+ if (likely(display->bridge &&
+ display->bridge->base.encoder &&
+ display->bridge->base.encoder->crtc)) {
+ rc = drm_crtc_vblank_get(
+ display->bridge->base.encoder->crtc);
+ } else {
+ pr_err("hbm: missing CRTC during dimming start.\n");
+ return -EINVAL;
+ }
+
+ if (rc) {
+ pr_err("hbm: failed DRM request to get vblank\n: %d",
+ rc);
+ return rc;
+ }
+ }
+
+ hbm->dimming_frames_total = num_frames;
+ hbm->dimming_frames_left = num_frames;
+ hbm->dimming_stop_cmd = stop_cmd;
+ hbm->dimming_active = true;
+
+ pr_debug("HBM dimming starting\n");
+ queue_work(hbm->dimming_workq, &hbm->dimming_work);
+
+ return 0;
+}
+
+void dsi_backlight_hbm_dimming_stop(struct dsi_backlight_config *bl)
+{
+ struct dsi_display *display;
+ struct hbm_data *hbm = bl->hbm;
+
+ if (!hbm || !hbm->dimming_active)
+ return;
+
+ display = dev_get_drvdata(hbm->panel->parent);
+ if (likely(display->bridge &&
+ display->bridge->base.encoder &&
+ display->bridge->base.encoder->crtc)) {
+ drm_crtc_vblank_put(display->bridge->base.encoder->crtc);
+ } else {
+ pr_err("hbm: missing CRTC during dimming end.\n");
+ }
+
+ if (hbm->dimming_stop_cmd) {
+ int rc = dsi_panel_cmd_set_transfer(hbm->panel,
+ hbm->dimming_stop_cmd);
+
+ if (rc)
+ pr_err("hbm: failed to disable brightness dimming.\n");
+ }
+
+ hbm->dimming_frames_total = 0;
+ hbm->dimming_frames_left = 0;
+ hbm->dimming_stop_cmd = NULL;
+ hbm->dimming_active = false;
+ pr_debug("HBM dimming stopped\n");
+}
+
+static void dsi_backlight_hbm_dimming_restart(struct dsi_backlight_config *bl)
+{
+ struct hbm_data *hbm = bl->hbm;
+
+ if (!hbm || !hbm->dimming_active)
+ return;
+
+ hbm->dimming_frames_left = hbm->dimming_frames_total;
+ pr_debug("hbm: dimming restarted\n");
+}
+
+static int dsi_backlight_hbm_wait_frame(struct hbm_data *hbm)
+{
+ struct dsi_display *display = dev_get_drvdata(hbm->panel->parent);
+
+ if (likely(display->bridge && display->bridge->base.encoder)) {
+ int rc = sde_encoder_wait_for_event(
+ display->bridge->base.encoder, MSM_ENC_VBLANK);
+ if (rc)
+ return rc;
+ } else {
+ pr_err("hbm: missing SDE encoder, can't wait for vblank\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void dsi_backlight_hbm_dimming_work(struct work_struct *work)
+{
+ struct dsi_panel *panel;
+ struct hbm_data *hbm =
+ container_of(work, struct hbm_data, dimming_work);
+
+ if (!hbm)
+ return;
+
+ panel = hbm->panel;
+ while (hbm->dimming_active) {
+ int rc = dsi_backlight_hbm_wait_frame(hbm);
+
+ /*
+ * It's possible that this thread is running while the driver is
+ * attempting to shut down. If this is the case, the driver
+ * will signal for dimming to stop while holding panel_lock.
+ * So if we fail to acquire the lock, wait a bit, then check the
+ * state of dimming_active again.
+ */
+ if (!mutex_trylock(&panel->panel_lock)) {
+ usleep_range(1000, 2000);
+ continue;
+ }
+
+ pr_debug("hbm: dimming waited on frame %d of %d\n",
+ hbm->dimming_frames_left, hbm->dimming_frames_total);
+ if (!hbm->dimming_active) {
+ mutex_unlock(&panel->panel_lock);
+ break;
+ }
+
+ if (rc) {
+ pr_err("hbm: failed to wait for vblank, disabling dimming now\n");
+ hbm->dimming_frames_left = 0;
+ } else if (hbm->dimming_frames_left > 0) {
+ hbm->dimming_frames_left--;
+ }
+
+ if (!hbm->dimming_frames_left)
+ dsi_backlight_hbm_dimming_stop(&panel->bl_config);
+
+ mutex_unlock(&panel->panel_lock);
+ }
+}
+
+int dsi_backlight_hbm_find_range(struct dsi_backlight_config *bl,
+ int brightness, u32 *range)
+{
+ u32 i;
+
+ if (!bl || !bl->hbm || !range)
+ return -EINVAL;
+
+ for (i = 0; i < bl->hbm->num_ranges; i++) {
+ if (brightness <= bl->hbm->ranges[i].user_bri_end) {
+ *range = i;
+ return 0;
+ }
+ }
+
+ return -EINVAL;
+}
+
+static u32 dsi_backlight_calculate_hbm(struct dsi_backlight_config *bl,
+ int brightness)
+{
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ struct hbm_data *hbm = bl->hbm;
+ struct hbm_range *range = NULL;
+ u32 bl_lvl = 0;
+ int rc = 0;
+ /* It's unlikely that a brightness value of 0 will make it to this
+ * function, but if it does use the dimmest HBM range.
+ */
+ u32 target_range = 0;
+
+ if (likely(brightness)) {
+ rc = dsi_backlight_hbm_find_range(bl, brightness,
+ &target_range);
+ if (rc) {
+ pr_err("Did not find a matching HBM range for brightness %d\n",
+ brightness);
+ return bl->bl_actual;
+ }
+ }
+
+ range = hbm->ranges + target_range;
+ if (hbm->cur_range != target_range) {
+ rc = dsi_panel_cmd_set_transfer(panel, &range->entry_cmd);
+ if (rc) {
+ pr_err("Failed to send command for range %d\n",
+ target_range);
+ return bl->bl_actual;
+ }
+
+ dsi_backlight_hbm_dimming_start(bl, range->num_dimming_frames,
+ &range->dimming_stop_cmd);
+ pr_info("hbm: range %d -> %d\n", hbm->cur_range, target_range);
+ hbm->cur_range = target_range;
+ }
+
+ rc = dsi_backlight_lerp(
+ range->user_bri_start, range->user_bri_end,
+ range->panel_bri_start, range->panel_bri_end,
+ brightness, &bl_lvl);
+ if (unlikely(rc))
+ pr_err("hbm: failed to linearly interpolate, brightness unmodified\n");
+
+ pr_debug("hbm: user %d-%d, panel %d-%d\n",
+ range->user_bri_start, range->user_bri_end,
+ range->panel_bri_start, range->panel_bri_end);
+
+ return bl_lvl;
+}
+
+static u32 dsi_backlight_calculate(struct dsi_backlight_config *bl,
+ int brightness)
+{
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ u32 bl_lvl = 0;
+ u32 bl_temp;
+
+ if (brightness <= 0)
+ return 0;
+
+ /* scale backlight */
+ bl_temp = mult_frac(brightness, bl->bl_scale,
+ MAX_BL_SCALE_LEVEL);
+
+ bl_temp = mult_frac(bl_temp, bl->bl_scale_ad,
+ MAX_AD_BL_SCALE_LEVEL);
+
+ if (panel->hbm_mode)
+ bl_lvl = dsi_backlight_calculate_hbm(bl, bl_temp);
+ else
+ bl_lvl = dsi_backlight_calculate_normal(bl, bl_temp);
+
+ pr_debug("brightness=%d, bl_scale=%d, ad=%d, bl_lvl=%d, hbm = %d\n",
+ brightness, bl->bl_scale, bl->bl_scale_ad, bl_lvl,
+ panel->hbm_mode);
+
+ return bl_lvl;
+}
+
+static int dsi_backlight_update_status(struct backlight_device *bd)
+{
+ struct dsi_backlight_config *bl = bl_get_data(bd);
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ int brightness = bd->props.brightness;
+ int bl_lvl;
+ int rc = 0;
+
+ if ((bd->props.state & (BL_CORE_FBBLANK | BL_CORE_SUSPENDED)) ||
+ (bd->props.power != FB_BLANK_UNBLANK))
+ brightness = 0;
+
+ mutex_lock(&panel->panel_lock);
+ bl_lvl = dsi_backlight_calculate(bl, brightness);
+ if (bl_lvl == bl->bl_actual && bl->last_state == bd->props.state)
+ goto done;
+
+ dsi_backlight_hbm_dimming_restart(bl);
+
+ if (dsi_panel_initialized(panel) && bl->update_bl) {
+ /* VR brightness is set as part of the VR entry sequence.
+ * This safeguard is to protect against userspace reducing
+ * brightness below safe limits during VR mode. Normal
+ * brightness range is restored when exiting VR mode by
+ * reevaluating requested user brightness.
+ */
+ if (panel->vr_mode && (bl_lvl < bl->bl_vr_min_safe_level))
+ bl_lvl = bl->bl_vr_min_safe_level;
+
+ pr_info("req:%d bl:%d state:0x%x\n",
+ bd->props.brightness, bl_lvl, bd->props.state);
+
+ rc = bl->update_bl(bl, bl_lvl);
+ if (rc) {
+ pr_err("unable to set backlight (%d)\n", rc);
+ goto done;
+ }
+ }
+ bl->bl_actual = bl_lvl;
+ bl->last_state = bd->props.state;
+
+done:
+ mutex_unlock(&panel->panel_lock);
+ return rc;
+}
+
+static int dsi_backlight_get_brightness(struct backlight_device *bd)
+{
+ struct dsi_backlight_config *bl = bl_get_data(bd);
+
+ return bl->bl_actual;
+}
+
+static const struct backlight_ops dsi_backlight_ops = {
+ .update_status = dsi_backlight_update_status,
+ .get_brightness = dsi_backlight_get_brightness,
+};
+
+static ssize_t alpm_mode_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct backlight_device *bd = to_backlight_device(dev);
+ struct dsi_backlight_config *bl = bl_get_data(bd);
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ int rc, alpm_mode;
+ const unsigned int lp_state = bl->bl_device->props.state &
+ (BL_STATE_LP | BL_STATE_LP2);
+
+ rc = kstrtoint(buf, 0, &alpm_mode);
+ if (rc)
+ return rc;
+
+ if (bl->bl_device->props.state & BL_CORE_FBBLANK) {
+ return -EINVAL;
+ } else if ((alpm_mode == 1) && (lp_state != BL_STATE_LP)) {
+ pr_info("activating lp1 mode\n");
+ dsi_panel_set_lp1(panel);
+ } else if ((alpm_mode > 1) && !(lp_state & BL_STATE_LP2)) {
+ pr_info("activating lp2 mode\n");
+ dsi_panel_set_lp2(panel);
+ } else if (!alpm_mode && lp_state) {
+ pr_info("activating normal mode\n");
+ dsi_panel_set_nolp(panel);
+ }
+
+ return count;
+}
+
+static ssize_t alpm_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct backlight_device *bd = to_backlight_device(dev);
+ int alpm_mode;
+
+ if (bd->props.state & BL_STATE_LP2)
+ alpm_mode = 2;
+ else
+ alpm_mode = (bd->props.state & BL_STATE_LP) != 0;
+
+ return sprintf(buf, "%d\n", alpm_mode);
+}
+static DEVICE_ATTR_RW(alpm_mode);
+
+static ssize_t vr_mode_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct backlight_device *bd = NULL;
+ struct dsi_backlight_config *bl = NULL;
+ struct dsi_panel *panel = NULL;
+ int rc = 0;
+ bool vr_mode = false;
+
+ /* dev is non-NULL, enforced by sysfs_create_file_ns */
+ bd = to_backlight_device(dev);
+ bl = bl_get_data(bd);
+ panel = container_of(bl, struct dsi_panel, bl_config);
+
+ rc = kstrtobool(buf, &vr_mode);
+ if (rc)
+ return rc;
+
+ rc = dsi_panel_update_vr_mode(panel, vr_mode);
+
+ if (!rc && !vr_mode) {
+ bl->bl_actual = -1;
+ if (backlight_update_status(bd))
+ pr_warn("couldn't restore backlight after VR exit");
+ }
+
+ if (rc)
+ return rc;
+ else
+ return count;
+}
+
+static ssize_t vr_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct backlight_device *bd = NULL;
+ struct dsi_backlight_config *bl = NULL;
+ struct dsi_panel *panel = NULL;
+ bool vr_mode = false;
+
+ /* dev is non-NULL, enforced by sysfs_create_file_ns */
+ bd = to_backlight_device(dev);
+ bl = bl_get_data(bd);
+ panel = container_of(bl, struct dsi_panel, bl_config);
+
+ vr_mode = dsi_panel_get_vr_mode(panel);
+
+ return snprintf(buf, PAGE_SIZE, "%s\n", vr_mode ? "on" : "off");
+}
+
+static DEVICE_ATTR_RW(vr_mode);
+
+static ssize_t hbm_mode_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct backlight_device *bd = NULL;
+ struct dsi_backlight_config *bl = NULL;
+ struct dsi_panel *panel = NULL;
+ int rc = 0;
+ bool hbm_mode = false;
+
+ /* dev is non-NULL, enforced by sysfs_create_file_ns */
+ bd = to_backlight_device(dev);
+ bl = bl_get_data(bd);
+
+ if (!bl->hbm)
+ return -ENOTSUPP;
+
+ rc = kstrtobool(buf, &hbm_mode);
+ if (rc)
+ return rc;
+
+ panel = container_of(bl, struct dsi_panel, bl_config);
+ rc = dsi_panel_update_hbm(panel, hbm_mode);
+ if (rc)
+ return rc;
+
+ return count;
+}
+
+static ssize_t hbm_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct backlight_device *bd = NULL;
+ struct dsi_backlight_config *bl = NULL;
+ struct dsi_panel *panel = NULL;
+ bool hbm_mode = false;
+
+ /* dev is non-NULL, enforced by sysfs_create_file_ns */
+ bd = to_backlight_device(dev);
+ bl = bl_get_data(bd);
+
+ if (!bl->hbm)
+ return snprintf(buf, PAGE_SIZE, "unsupported\n");
+
+ panel = container_of(bl, struct dsi_panel, bl_config);
+ hbm_mode = dsi_panel_get_hbm(panel);
+
+ return snprintf(buf, PAGE_SIZE, "%s\n", hbm_mode ? "on" : "off");
+}
+
+static DEVICE_ATTR_RW(hbm_mode);
+
+static struct attribute *bl_device_attrs[] = {
+ &dev_attr_alpm_mode.attr,
+ &dev_attr_vr_mode.attr,
+ &dev_attr_hbm_mode.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(bl_device);
+
+#ifdef CONFIG_LEDS_TRIGGERS
+static int dsi_panel_update_backlight_led(struct dsi_backlight_config *bl,
+ u32 bl_lvl)
+{
+ struct led_trigger *wled = bl->priv;
+ if (!wled)
+ return -EINVAL;
+
+ if (bl_lvl != bl->bl_actual)
+ led_trigger_event(wled, bl_lvl);
+
+ return 0;
+}
+
+static void dsi_panel_led_bl_unregister(struct dsi_backlight_config *bl)
+{
+ struct led_trigger *wled = bl->priv;
+
+ if (wled) {
+ led_trigger_unregister_simple(wled);
+ bl->priv = NULL;
+ }
+}
+
+static int dsi_panel_led_bl_register(struct dsi_backlight_config *bl)
+{
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ struct led_trigger *wled;
+ int rc = 0;
+
+ led_trigger_register_simple("bkl-trigger", &wled);
+
+ /* LED APIs don't tell us directly whether a classdev has yet
+ * been registered to service this trigger. Until classdev is
+ * registered, calling led_trigger has no effect, and doesn't
+ * fail. Classdevs are associated with any registered triggers
+ * when they do register, but that is too late for FBCon.
+ * Check the cdev list directly and defer if appropriate.
+ */
+ if (!wled) {
+ pr_err("[%s] backlight registration failed\n", panel->name);
+ rc = -EINVAL;
+ } else {
+ read_lock(&wled->leddev_list_lock);
+ if (list_empty(&wled->led_cdevs))
+ rc = -EPROBE_DEFER;
+ read_unlock(&wled->leddev_list_lock);
+
+ if (rc) {
+ pr_info("[%s] backlight %s not ready, defer probe\n",
+ panel->name, wled->name);
+ led_trigger_unregister_simple(wled);
+ }
+
+ bl->priv = wled;
+ bl->update_bl = dsi_panel_update_backlight_led;
+ bl->unregister = dsi_panel_led_bl_unregister;
+ }
+
+ return rc;
+}
+#else
+static int dsi_panel_led_bl_register(struct dsi_backlight_config *bl)
+{
+ return 0;
+}
+#endif
+
+static int dsi_backlight_register(struct dsi_backlight_config *bl)
+{
+ static int display_count;
+ char bl_node_name[BL_NODE_NAME_SIZE];
+ struct backlight_properties props = {
+ .type = BACKLIGHT_RAW,
+ .power = FB_BLANK_UNBLANK,
+ };
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ struct regulator *reg;
+
+ props.max_brightness = bl->brightness_max_level;
+ props.brightness = bl->brightness_max_level / 2;
+
+ snprintf(bl_node_name, BL_NODE_NAME_SIZE, "panel%u-backlight",
+ display_count);
+ bl->bl_device = devm_backlight_device_register(panel->parent,
+ bl_node_name, panel->parent, bl,
+ &dsi_backlight_ops, &props);
+ if (IS_ERR_OR_NULL(bl->bl_device)) {
+ bl->bl_device = NULL;
+ return -ENODEV;
+ }
+
+ if (sysfs_create_groups(&bl->bl_device->dev.kobj, bl_device_groups))
+ pr_warn("unable to create device groups\n");
+
+ reg = regulator_get(panel->parent, "lab");
+ if (!PTR_ERR_OR_ZERO(reg)) {
+ pr_info("LAB regulator found\n");
+ panel->bl_config.lab_vreg = reg;
+ }
+
+ display_count++;
+ return 0;
+}
+
+static unsigned long get_state_after_dpms(struct dsi_backlight_config *bl,
+ int power_mode)
+{
+ struct backlight_device *bd = bl->bl_device;
+ unsigned long state = bd->props.state;
+
+ switch (power_mode) {
+ case SDE_MODE_DPMS_ON:
+ state &= ~(BL_CORE_FBBLANK | BL_STATE_LP | BL_STATE_LP2);
+ break;
+ case SDE_MODE_DPMS_OFF:
+ state &= ~(BL_STATE_LP | BL_STATE_LP2);
+ state |= BL_CORE_FBBLANK;
+ break;
+ case SDE_MODE_DPMS_LP1:
+ state |= BL_STATE_LP;
+ state &= ~BL_STATE_LP2;
+ break;
+ case SDE_MODE_DPMS_LP2:
+ state |= BL_STATE_LP | BL_STATE_LP2;
+ break;
+ }
+
+ return state;
+}
+
+int dsi_backlight_early_dpms(struct dsi_backlight_config *bl, int power_mode)
+{
+ struct backlight_device *bd = bl->bl_device;
+ unsigned long state;
+ unsigned int last_mode, mode;
+
+ if (!bd)
+ return 0;
+
+ pr_info("power_mode:%d state:0x%0x\n", power_mode, bd->props.state);
+
+ mutex_lock(&bd->ops_lock);
+ state = get_state_after_dpms(bl, power_mode);
+
+ if (bl->lab_vreg) {
+ last_mode = regulator_mode_from_state(bl->last_state);
+ mode = regulator_mode_from_state(state);
+
+ if (last_mode != mode)
+ regulator_set_mode(bl->lab_vreg, mode);
+ }
+
+ mutex_unlock(&bd->ops_lock);
+
+ return 0;
+}
+
+int dsi_backlight_late_dpms(struct dsi_backlight_config *bl, int power_mode)
+{
+ struct backlight_device *bd = bl->bl_device;
+ unsigned long state;
+
+ if (!bd)
+ return 0;
+
+ pr_debug("power_mode:%d state:0x%0x\n", power_mode, bd->props.state);
+
+ mutex_lock(&bd->ops_lock);
+ state = get_state_after_dpms(bl, power_mode);
+
+ bd->props.power = state & BL_CORE_FBBLANK ? FB_BLANK_POWERDOWN :
+ FB_BLANK_UNBLANK;
+ bd->props.state = state;
+
+ backlight_update_status(bd);
+ mutex_unlock(&bd->ops_lock);
+
+ return 0;
+}
+
+int dsi_backlight_get_dpms(struct dsi_backlight_config *bl)
+{
+ struct backlight_device *bd = bl->bl_device;
+ int power = 0;
+ int state = 0;
+
+ mutex_lock(&bd->ops_lock);
+ power = bd->props.power;
+ state = bd->props.state;
+ mutex_unlock(&bd->ops_lock);
+
+ if (power == FB_BLANK_POWERDOWN)
+ return SDE_MODE_DPMS_OFF;
+ else if (state & BL_STATE_LP2)
+ return SDE_MODE_DPMS_LP2;
+ else if (state & BL_STATE_LP)
+ return SDE_MODE_DPMS_LP1;
+ else
+ return SDE_MODE_DPMS_ON;
+}
+
+#define MAX_BINNED_BL_MODES 10
+#define DCS_BRIGHTNESS_COMPENSATION 0xC7
+enum dcs_brightness_comp_reg {
+ DCS_LPB_COMP_VALUE = 0,
+ DCS_LPB_BGR_TRIM = 2,
+ DCS_LPB_COMP_SIZE,
+};
+
+struct binned_lp_node {
+ struct list_head head;
+ const char *name;
+ u32 bl_threshold;
+ struct dsi_panel_cmd_set dsi_cmd;
+};
+
+enum lp_comp_state {
+ LP_COMP_DISABLED,
+ LP_COMP_UNINITIALIZED,
+ LP_COMP_INITIALIZED,
+};
+
+struct binned_lp_data {
+ struct list_head mode_list;
+ struct binned_lp_node *last_lp_mode;
+ struct binned_lp_node priv_pool[MAX_BINNED_BL_MODES];
+
+ struct {
+ enum lp_comp_state state;
+ u8 buf[DCS_LPB_COMP_SIZE];
+ u8 adder;
+ } lp_brightness_comp;
+};
+
+static void dsi_panel_lp_compensate(struct dsi_backlight_config *bl,
+ struct binned_lp_node *target_node)
+{
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ struct binned_lp_data *lp_data = bl->priv;
+ ssize_t read_size;
+ u8 buf[16];
+
+ switch (lp_data->lp_brightness_comp.state) {
+ case LP_COMP_UNINITIALIZED:
+ read_size = mipi_dsi_dcs_read(&panel->mipi_device,
+ DCS_BRIGHTNESS_COMPENSATION, buf,
+ DCS_LPB_COMP_SIZE);
+ if (unlikely(read_size != DCS_LPB_COMP_SIZE)) {
+ pr_warn("unable to read brightness comp");
+ return;
+ }
+ memcpy(lp_data->lp_brightness_comp.buf, buf, read_size);
+ lp_data->lp_brightness_comp.state = LP_COMP_INITIALIZED;
+ pr_debug("Read LP brightness comp: %02X\n", buf[0]);
+ break;
+ case LP_COMP_INITIALIZED:
+ memcpy(buf, lp_data->lp_brightness_comp.buf, DCS_LPB_COMP_SIZE);
+ break;
+ case LP_COMP_DISABLED:
+ default:
+ return;
+ }
+
+ /* add compensation in LP mode, otherwise restore original value */
+ if (target_node && target_node->bl_threshold > 0) {
+ buf[DCS_LPB_COMP_VALUE] += lp_data->lp_brightness_comp.adder;
+ buf[DCS_LPB_BGR_TRIM] = 0x00; /* lowest bgr trim current */
+ }
+
+ pr_debug("LP brightness compensation: %02X\n", buf[0]);
+
+ mipi_dsi_dcs_write(&panel->mipi_device, DCS_BRIGHTNESS_COMPENSATION,
+ buf, DCS_LPB_COMP_SIZE);
+}
+
+static int dsi_panel_binned_bl_update(struct dsi_backlight_config *bl,
+ u32 bl_lvl)
+{
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ struct binned_lp_data *lp_data = bl->priv;
+ struct binned_lp_node *node = NULL;
+ struct backlight_properties *props = &bl->bl_device->props;
+
+ if (is_lp_mode(props->state)) {
+ struct binned_lp_node *tmp;
+
+ list_for_each_entry(tmp, &lp_data->mode_list, head) {
+ if (props->brightness <= tmp->bl_threshold) {
+ node = tmp;
+ break;
+ }
+ }
+ WARN(!node, "unable to find lp node for bl_lvl: %d\n",
+ props->brightness);
+ }
+
+ if (node != lp_data->last_lp_mode) {
+ dsi_panel_lp_compensate(bl, node);
+
+ lp_data->last_lp_mode = node;
+ if (node) {
+ pr_debug("switching display lp mode: %s (%d)\n",
+ node->name, props->brightness);
+ dsi_panel_cmd_set_transfer(panel, &node->dsi_cmd);
+ } else {
+ /* ensure update after lpm */
+ bl->bl_actual = -1;
+ }
+ }
+
+ /* no need to send backlight command if HLPM active */
+ if (node)
+ return 0;
+
+ return dsi_backlight_update_dcs(bl, bl_lvl);
+}
+
+static int _dsi_panel_binned_lp_parse(struct device_node *np,
+ struct binned_lp_node *node)
+{
+ int rc;
+ u32 val = 0;
+
+ rc = of_property_read_u32(np, "google,dsi-lp-brightness-threshold",
+ &val);
+ /* treat lack of property as max threshold */
+ node->bl_threshold = !rc ? val : UINT_MAX;
+
+ rc = dsi_panel_parse_dt_cmd_set(np, "google,dsi-lp-command",
+ "google,dsi-lp-command-state",
+ &node->dsi_cmd);
+ if (rc) {
+ pr_err("Unable to parse dsi-lp-command\n");
+ return rc;
+ }
+
+ of_property_read_string(np, "label", &node->name);
+
+ pr_debug("Successfully parsed lp mode: %s threshold: %d\n",
+ node->name, node->bl_threshold);
+
+ return 0;
+}
+
+static int _dsi_panel_binned_bl_cmp(void *priv, struct list_head *lha,
+ struct list_head *lhb)
+{
+ struct binned_lp_node *a = list_entry(lha, struct binned_lp_node, head);
+ struct binned_lp_node *b = list_entry(lhb, struct binned_lp_node, head);
+
+ return a->bl_threshold - b->bl_threshold;
+}
+
+static int dsi_panel_binned_lp_register(struct dsi_backlight_config *bl)
+{
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ struct binned_lp_data *lp_data;
+ struct device_node *lp_modes_np, *child_np;
+ struct binned_lp_node *lp_node;
+ u32 val;
+ int num_modes;
+ int rc = -ENOTSUPP;
+
+ lp_data = kzalloc(sizeof(*lp_data), GFP_KERNEL);
+ if (!lp_data)
+ return -ENOMEM;
+
+ lp_modes_np = of_get_child_by_name(panel->panel_of_node,
+ "google,lp-modes");
+
+ if (!lp_modes_np) {
+ kfree(lp_data);
+ return rc;
+ }
+
+ num_modes = of_get_child_count(lp_modes_np);
+ if (!num_modes || (num_modes > MAX_BINNED_BL_MODES)) {
+ pr_err("Invalid binned brightness modes: %d\n", num_modes);
+ goto exit;
+ }
+
+ INIT_LIST_HEAD(&lp_data->mode_list);
+ lp_node = lp_data->priv_pool;
+
+ for_each_child_of_node(lp_modes_np, child_np) {
+ rc = _dsi_panel_binned_lp_parse(child_np, lp_node);
+ if (rc)
+ goto exit;
+
+ list_add_tail(&lp_node->head, &lp_data->mode_list);
+ lp_node++;
+ if (lp_node > &lp_data->priv_pool[MAX_BINNED_BL_MODES - 1]) {
+ pr_err("Too many LP modes\n");
+ rc = -ENOTSUPP;
+ goto exit;
+ }
+ }
+ list_sort(NULL, &lp_data->mode_list, _dsi_panel_binned_bl_cmp);
+
+ if (!of_property_read_u32(panel->panel_of_node,
+ "google,mdss-dsi-lp-brightness-compensation",
+ &val)) {
+ pr_debug("LP brightness compensation: %d\n", val);
+ lp_data->lp_brightness_comp.adder = (u8)val;
+ lp_data->lp_brightness_comp.state = LP_COMP_UNINITIALIZED;
+ } else {
+ lp_data->lp_brightness_comp.state = LP_COMP_DISABLED;
+ }
+
+ bl->update_bl = dsi_panel_binned_bl_update;
+ bl->unregister = dsi_panel_bl_free_unregister;
+ bl->priv = lp_data;
+
+exit:
+ of_node_put(lp_modes_np);
+ if (rc)
+ kfree(lp_data);
+
+ return rc;
+}
+
+static const struct of_device_id dsi_backlight_dt_match[] = {
+ {
+ .compatible = "google,dsi_binned_lp",
+ .data = dsi_panel_binned_lp_register,
+ },
+ {}
+};
+
+static int dsi_panel_bl_parse_hbm_node(struct device *parent,
+ struct dsi_backlight_config *bl, struct device_node *np,
+ struct hbm_range *range)
+{
+ int rc;
+ u32 val = 0;
+
+ rc = of_property_read_u32(np,
+ "google,dsi-hbm-range-brightness-threshold", &val);
+ if (rc) {
+ pr_err("Unable to parse dsi-hbm-range-brightness-threshold");
+ return rc;
+ }
+ if (val > bl->brightness_max_level) {
+ pr_err("hbm-brightness-threshold exceeds max userspace brightness\n");
+ return rc;
+ }
+ range->user_bri_start = val;
+
+ rc = of_property_read_u32(np, "google,dsi-hbm-range-bl-min-level",
+ &val);
+ if (rc) {
+ pr_err("dsi-hbm-range-bl-min-level unspecified\n");
+ return rc;
+ }
+ range->panel_bri_start = val;
+
+ rc = of_property_read_u32(np, "google,dsi-hbm-range-bl-max-level",
+ &val);
+ if (rc) {
+ pr_err("dsi-hbm-range-bl-max-level unspecified\n");
+ return rc;
+ }
+ if (val < range->panel_bri_start) {
+ pr_err("Invalid HBM panel brightness range: bl-hbm-max-level < bl-hbm-min-level\n");
+ return rc;
+ }
+ range->panel_bri_end = val;
+
+ rc = dsi_panel_parse_dt_cmd_set(np,
+ "google,dsi-hbm-range-entry-command",
+ "google,dsi-hbm-range-commands-state", &range->entry_cmd);
+ if (rc)
+ pr_info("Unable to parse optional dsi-hbm-range-entry-command\n");
+
+ rc = of_property_read_u32(np,
+ "google,dsi-hbm-range-num-dimming-frames", &val);
+ if (rc) {
+ pr_debug("Unable to parse optional hbm-range-entry-num-dimming-frames\n");
+ range->num_dimming_frames = 0;
+ } else {
+ range->num_dimming_frames = val;
+ }
+
+ rc = dsi_panel_parse_dt_cmd_set(np,
+ "google,dsi-hbm-range-dimming-stop-command",
+ "google,dsi-hbm-range-commands-state",
+ &range->dimming_stop_cmd);
+ if (rc)
+ pr_debug("Unable to parse optional dsi-hbm-range-dimming-stop-command\n");
+
+ if ((range->dimming_stop_cmd.count && !range->num_dimming_frames) ||
+ (!range->dimming_stop_cmd.count && range->num_dimming_frames)) {
+ pr_err("HBM dimming requires both stop command and number of frames.\n");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+int dsi_panel_bl_register(struct dsi_panel *panel)
+{
+ int rc = 0;
+ struct dsi_backlight_config *bl = &panel->bl_config;
+ const struct of_device_id *match;
+ int (*register_func)(struct dsi_backlight_config *) = NULL;
+
+ match = of_match_node(dsi_backlight_dt_match, panel->panel_of_node);
+ if (match && match->data) {
+ register_func = match->data;
+ rc = register_func(bl);
+ }
+
+ if (!register_func || (rc == -ENOTSUPP)) {
+ switch (bl->type) {
+ case DSI_BACKLIGHT_WLED:
+ register_func = dsi_panel_led_bl_register;
+ break;
+ case DSI_BACKLIGHT_DCS:
+ bl->update_bl = dsi_backlight_update_dcs;
+ break;
+ case DSI_BACKLIGHT_PWM:
+ register_func = dsi_panel_pwm_bl_register;
+ break;
+ default:
+ pr_err("Backlight type(%d) not supported\n", bl->type);
+ rc = -ENOTSUPP;
+ break;
+ }
+
+ if (register_func)
+ rc = register_func(bl);
+ }
+
+ if (!rc)
+ rc = dsi_backlight_register(bl);
+
+ return rc;
+}
+
+int dsi_panel_bl_unregister(struct dsi_panel *panel)
+{
+ struct dsi_backlight_config *bl = &panel->bl_config;
+
+ if (bl->unregister)
+ bl->unregister(bl);
+
+ if (bl->bl_device)
+ sysfs_remove_groups(&bl->bl_device->dev.kobj, bl_device_groups);
+
+ dsi_panel_bl_hbm_free(panel->parent, bl);
+
+ return 0;
+}
+
+static int dsi_panel_bl_parse_pwm_config(struct dsi_backlight_pwm_config *config,
+ struct device_node *of_node)
+{
+ int rc = 0;
+ u32 val;
+
+ rc = of_property_read_u32(of_node, "qcom,dsi-bl-pmic-bank-select",
+ &val);
+ if (rc) {
+ pr_err("bl-pmic-bank-select is not defined, rc=%d\n", rc);
+ goto error;
+ }
+ config->pwm_pmic_bank = val;
+
+ rc = of_property_read_u32(of_node, "qcom,dsi-bl-pmic-pwm-frequency",
+ &val);
+ if (rc) {
+ pr_err("bl-pmic-bank-select is not defined, rc=%d\n", rc);
+ goto error;
+ }
+ config->pwm_period_usecs = val;
+
+ config->pwm_pmi_control = of_property_read_bool(of_node,
+ "qcom,mdss-dsi-bl-pwm-pmi");
+
+ config->pwm_gpio = of_get_named_gpio(of_node,
+ "qcom,mdss-dsi-pwm-gpio",
+ 0);
+ if (!gpio_is_valid(config->pwm_gpio)) {
+ pr_err("pwm gpio is invalid\n");
+ rc = -EINVAL;
+ goto error;
+ }
+
+error:
+ return rc;
+}
+
+static int dsi_panel_pwm_bl_register(struct dsi_backlight_config *bl)
+{
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ struct dsi_backlight_pwm_config *pwm_cfg;
+ int rc;
+
+ pwm_cfg = kzalloc(sizeof(*pwm_cfg), GFP_KERNEL);
+ if (!pwm_cfg)
+ return -ENOMEM;
+
+
+ rc = dsi_panel_bl_parse_pwm_config(pwm_cfg, panel->panel_of_node);
+ if (rc) {
+ kfree(pwm_cfg);
+ return rc;
+ }
+
+ bl->priv = pwm_cfg;
+ bl->unregister = dsi_panel_bl_free_unregister;
+
+ return 0;
+}
+
+static int dsi_panel_bl_parse_lut(struct device *parent,
+ struct device_node *of_node, const char *bl_lut_prop_name,
+ u32 brightness_max_level, u16 **lut)
+{
+ u32 len = 0;
+ u32 i = 0;
+ u32 rc = 0;
+ const __be32 *val = 0;
+ struct property *prop = NULL;
+ u32 lut_length = brightness_max_level + 1;
+ u16 *bl_lut_tmp = NULL;
+
+ if (!of_node || !bl_lut_prop_name || !lut)
+ return -EINVAL;
+
+ if (*lut) {
+ pr_warn("LUT for %s already exists, freeing before reparsing\n",
+ bl_lut_prop_name);
+ devm_kfree(parent, *lut);
+ *lut = NULL;
+ }
+
+ prop = of_find_property(of_node, bl_lut_prop_name, &len);
+ if (!prop)
+ goto done; /* LUT is unspecified */
+
+ len /= sizeof(u32);
+ if (len != lut_length) {
+ pr_warn("%s length %d doesn't match brightness_max_level + 1 %d\n",
+ bl_lut_prop_name, len, lut_length);
+ goto done;
+ }
+
+ pr_debug("%s length %d\n", bl_lut_prop_name, lut_length);
+ bl_lut_tmp = devm_kmalloc(parent, sizeof(u16) * lut_length, GFP_KERNEL);
+ if (bl_lut_tmp == NULL) {
+ rc = -ENOMEM;
+ goto done;
+ }
+
+ val = prop->value;
+ for (i = 0; i < len; i++)
+ bl_lut_tmp[i] = (u16)(be32_to_cpup(val++) & 0xffff);
+
+ *lut = bl_lut_tmp;
+
+done:
+ return rc;
+}
+
+static void dsi_panel_bl_hbm_free(struct device *dev,
+ struct dsi_backlight_config *bl)
+{
+ u32 i = 0;
+ struct hbm_data *hbm = bl->hbm;
+
+ if (!hbm)
+ return;
+
+ if (hbm->dimming_workq) {
+ dsi_backlight_hbm_dimming_stop(bl);
+ flush_workqueue(hbm->dimming_workq);
+ destroy_workqueue(hbm->dimming_workq);
+ }
+
+ dsi_panel_destroy_cmd_packets(&hbm->exit_cmd);
+ dsi_panel_destroy_cmd_packets(&hbm->exit_dimming_stop_cmd);
+
+ for (i = 0; i < hbm->num_ranges; i++) {
+ dsi_panel_destroy_cmd_packets(&hbm->ranges[i].entry_cmd);
+ dsi_panel_destroy_cmd_packets(&hbm->ranges[i].dimming_stop_cmd);
+ }
+
+ devm_kfree(dev, hbm);
+ bl->hbm = NULL;
+}
+
+static int dsi_panel_bl_parse_hbm(struct device *parent,
+ struct dsi_backlight_config *bl, struct device_node *of_node)
+{
+ struct device_node *hbm_ranges_np;
+ struct device_node *child_np;
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ u32 rc = 0;
+ u32 i = 0;
+ u32 num_ranges = 0;
+ u32 val = 0;
+ bool dimming_used = false;
+
+ panel->hbm_mode = false;
+
+ if (bl->hbm) {
+ pr_warn("HBM data already parsed, freeing before reparsing\n");
+ dsi_panel_bl_hbm_free(parent, bl);
+ }
+
+ hbm_ranges_np = of_get_child_by_name(of_node, "google,hbm-ranges");
+ if (!hbm_ranges_np) {
+ pr_info("HBM modes list not found\n");
+ return 0;
+ }
+
+ num_ranges = of_get_child_count(hbm_ranges_np);
+ if (!num_ranges || (num_ranges > HBM_RANGE_MAX)) {
+ pr_err("Invalid number of HBM modes: %d\n", num_ranges);
+ return -EINVAL;
+ }
+
+ bl->hbm = devm_kzalloc(parent, sizeof(struct hbm_data), GFP_KERNEL);
+ if (bl->hbm == NULL) {
+ pr_err("Failed to allocate memory for HBM data\n");
+ return -ENOMEM;
+ }
+
+ rc = dsi_panel_parse_dt_cmd_set(hbm_ranges_np,
+ "google,dsi-hbm-exit-command",
+ "google,dsi-hbm-commands-state", &bl->hbm->exit_cmd);
+ if (rc)
+ pr_info("Unable to parse optional dsi-hbm-exit-command\n");
+
+ bl->hbm->num_ranges = num_ranges;
+
+ rc = of_property_read_u32(hbm_ranges_np,
+ "google,dsi-hbm-exit-num-dimming-frames", &val);
+ if (rc) {
+ pr_debug("Unable to parse optional num-dimming-frames\n");
+ bl->hbm->exit_num_dimming_frames = 0;
+ } else {
+ bl->hbm->exit_num_dimming_frames = val;
+ }
+
+ rc = dsi_panel_parse_dt_cmd_set(hbm_ranges_np,
+ "google,dsi-hbm-exit-dimming-stop-command",
+ "google,dsi-hbm-commands-state",
+ &bl->hbm->exit_dimming_stop_cmd);
+ if (rc)
+ pr_debug("Unable to parse optional dsi-hbm-exit-dimming-stop-command\n");
+
+ if ((bl->hbm->exit_dimming_stop_cmd.count &&
+ !bl->hbm->exit_num_dimming_frames) ||
+ (!bl->hbm->exit_dimming_stop_cmd.count &&
+ bl->hbm->exit_num_dimming_frames)) {
+ pr_err("HBM dimming requires both stop command and number of frames.\n");
+ goto exit_free;
+ }
+
+ for_each_child_of_node(hbm_ranges_np, child_np) {
+ rc = dsi_panel_bl_parse_hbm_node(parent, bl,
+ child_np, bl->hbm->ranges + i);
+ if (rc) {
+ pr_err("Failed to parse HBM range %d of %d\n",
+ i + 1, num_ranges);
+ goto exit_free;
+ }
+ i++;
+ }
+
+ for (i = 0; i < num_ranges - 1; i++) {
+ /* Make sure ranges are sorted and not overlapping */
+ if (bl->hbm->ranges[i].user_bri_start >=
+ bl->hbm->ranges[i + 1].user_bri_start) {
+ pr_err("HBM ranges must be sorted by hbm-brightness-threshold\n");
+ rc = -EINVAL;
+ goto exit_free;
+ }
+
+ if (bl->hbm->ranges[i].num_dimming_frames)
+ dimming_used = true;
+
+ /* Fill in user_bri_end for each range */
+ bl->hbm->ranges[i].user_bri_end =
+ bl->hbm->ranges[i + 1].user_bri_start - 1;
+ }
+
+ if (bl->hbm->ranges[num_ranges - 1].num_dimming_frames ||
+ bl->hbm->exit_num_dimming_frames)
+ dimming_used = true;
+
+
+ if (dimming_used) {
+ bl->hbm->dimming_workq =
+ create_singlethread_workqueue("dsi_dimming_workq");
+ if (!bl->hbm->dimming_workq)
+ pr_err("failed to create hbm dimming workq!\n");
+ else
+ INIT_WORK(&bl->hbm->dimming_work,
+ dsi_backlight_hbm_dimming_work);
+ }
+
+ bl->hbm->ranges[i].user_bri_end = bl->brightness_max_level;
+ bl->hbm->cur_range = HBM_RANGE_MAX;
+ bl->hbm->dimming_active = false;
+ bl->hbm->dimming_frames_total = 0;
+ bl->hbm->dimming_frames_left = 0;
+ bl->hbm->panel = panel;
+
+ return 0;
+
+exit_free:
+ dsi_panel_bl_hbm_free(parent, bl);
+ return rc;
+}
+
+int dsi_panel_bl_parse_config(struct device *parent,
+ struct dsi_backlight_config *bl, struct device_node *of_node)
+{
+ struct dsi_panel *panel = container_of(bl, struct dsi_panel, bl_config);
+ int rc = 0;
+ const char *bl_type;
+ u32 val = 0;
+
+ bl_type = of_get_property(of_node,
+ "qcom,mdss-dsi-bl-pmic-control-type",
+ NULL);
+ if (!bl_type) {
+ bl->type = DSI_BACKLIGHT_UNKNOWN;
+ } else if (!strcmp(bl_type, "bl_ctrl_pwm")) {
+ bl->type = DSI_BACKLIGHT_PWM;
+ } else if (!strcmp(bl_type, "bl_ctrl_wled")) {
+ bl->type = DSI_BACKLIGHT_WLED;
+ } else if (!strcmp(bl_type, "bl_ctrl_dcs")) {
+ bl->type = DSI_BACKLIGHT_DCS;
+ } else {
+ pr_debug("[%s] bl-pmic-control-type unknown-%s\n",
+ panel->name, bl_type);
+ bl->type = DSI_BACKLIGHT_UNKNOWN;
+ }
+
+ bl->bl_scale = MAX_BL_SCALE_LEVEL;
+ bl->bl_scale_ad = MAX_AD_BL_SCALE_LEVEL;
+
+ rc = of_property_read_u32(of_node, "qcom,mdss-dsi-bl-min-level", &val);
+ if (rc) {
+ pr_debug("[%s] bl-min-level unspecified, defaulting to zero\n",
+ panel->name);
+ bl->bl_min_level = 0;
+ } else {
+ bl->bl_min_level = val;
+ }
+
+ rc = of_property_read_u32(of_node, "qcom,mdss-dsi-bl-max-level", &val);
+ if (rc) {
+ pr_debug("[%s] bl-max-level unspecified, defaulting to max level\n",
+ panel->name);
+ bl->bl_max_level = MAX_BL_LEVEL;
+ } else {
+ bl->bl_max_level = val;
+ }
+
+ rc = of_property_read_u32(of_node, "qcom,mdss-brightness-max-level",
+ &val);
+ if (rc) {
+ pr_debug("[%s] brigheness-max-level unspecified, defaulting to 255\n",
+ panel->name);
+ bl->brightness_max_level = 255;
+ } else {
+ bl->brightness_max_level = val;
+ }
+
+ rc = dsi_panel_bl_parse_lut(parent, of_node, "qcom,mdss-dsi-bl-lut",
+ bl->brightness_max_level, &bl->lut);
+ if (rc) {
+ pr_err("[%s] failed to create backlight LUT, rc=%d\n",
+ panel->name, rc);
+ goto error;
+ }
+ pr_debug("[%s] bl-lut %sused\n", panel->name, bl->lut ? "" : "un");
+
+ rc = dsi_panel_bl_parse_hbm(parent, bl, of_node);
+ if (rc)
+ pr_err("[%s] error while parsing high brightness mode (hbm) details, rc=%d\n",
+ panel->name, rc);
+
+ rc = of_property_read_u32(of_node, "qcom,mdss-dsi-bl-vr-min-safe-level",
+ &val);
+ if (rc) {
+ bl->bl_vr_min_safe_level = 0;
+ } else {
+ pr_debug("[%s] bl-vr-min-safe-level is %d\n",
+ panel->name, val);
+ bl->bl_vr_min_safe_level = val;
+ }
+
+ bl->en_gpio = of_get_named_gpio(of_node,
+ "qcom,platform-bklight-en-gpio",
+ 0);
+ if (!gpio_is_valid(bl->en_gpio)) {
+ pr_debug("[%s] failed get bklt gpio, rc=%d\n", panel->name, rc);
+ rc = 0;
+ goto error;
+ }
+
+error:
+ return rc;
+}
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_catalog.c b/drivers/gpu/drm/msm/dsi-staging/dsi_catalog.c
index a457070..5012090 100644
--- a/drivers/gpu/drm/msm/dsi-staging/dsi_catalog.c
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_catalog.c
@@ -59,6 +59,8 @@ static void dsi_catalog_cmn_init(struct dsi_ctrl_hw *ctrl,
ctrl->ops.phy_reset_config = dsi_ctrl_hw_cmn_phy_reset_config;
ctrl->ops.setup_misr = dsi_ctrl_hw_cmn_setup_misr;
ctrl->ops.collect_misr = dsi_ctrl_hw_cmn_collect_misr;
+ ctrl->ops.get_cmd_read_data = dsi_ctrl_hw_cmn_get_cmd_read_data;
+ ctrl->ops.clear_rdbk_register = dsi_ctrl_hw_cmn_clear_rdbk_reg;
ctrl->ops.debug_bus = dsi_ctrl_hw_cmn_debug_bus;
ctrl->ops.get_cmd_read_data = dsi_ctrl_hw_cmn_get_cmd_read_data;
ctrl->ops.clear_rdbk_register = dsi_ctrl_hw_cmn_clear_rdbk_reg;
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_catalog.h b/drivers/gpu/drm/msm/dsi-staging/dsi_catalog.h
index 144ccd9..ede8149 100644
--- a/drivers/gpu/drm/msm/dsi-staging/dsi_catalog.h
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_catalog.h
@@ -175,10 +175,10 @@ void dsi_ctrl_hw_cmn_phy_reset_config(struct dsi_ctrl_hw *ctrl,
void dsi_ctrl_hw_22_phy_reset_config(struct dsi_ctrl_hw *ctrl,
bool enable);
u32 dsi_ctrl_hw_cmn_get_cmd_read_data(struct dsi_ctrl_hw *ctrl,
- u8 *rd_buf,
- u32 read_offset,
- u32 rx_byte,
- u32 pkt_size, u32 *hw_read_cnt);
+ u8 *rd_buf,
+ u32 read_offset,
+ u32 rx_byte,
+ u32 pkt_size, u32 *hw_read_cnt);
void dsi_ctrl_hw_cmn_clear_rdbk_reg(struct dsi_ctrl_hw *ctrl);
void dsi_ctrl_hw_22_schedule_dma_cmd(struct dsi_ctrl_hw *ctrl, int line_on);
int dsi_ctrl_hw_cmn_ctrl_reset(struct dsi_ctrl_hw *ctrl,
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_ctrl.c b/drivers/gpu/drm/msm/dsi-staging/dsi_ctrl.c
index 83ef9ae..6f606ea 100644
--- a/drivers/gpu/drm/msm/dsi-staging/dsi_ctrl.c
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_ctrl.c
@@ -1333,15 +1333,17 @@ static int dsi_message_rx(struct dsi_ctrl *dsi_ctrl,
const struct mipi_dsi_msg *msg,
u32 flags)
{
- int rc = 0;
+ int rc = 0, i = 0;
u32 rd_pkt_size, total_read_len, hw_read_cnt;
u32 current_read_len = 0, total_bytes_read = 0;
bool short_resp = false;
bool read_done = false;
u32 dlen, diff, rlen;
- unsigned char *buff;
+ unsigned char *buff = NULL;
char cmd;
struct dsi_cmd_desc *of_cmd;
+ u32 buffer_sz = 0, header_offset = 0;
+ u8 *head = NULL;
if (!msg) {
pr_err("Invalid msg\n");
@@ -1356,6 +1358,13 @@ static int dsi_message_rx(struct dsi_ctrl *dsi_ctrl,
short_resp = true;
rd_pkt_size = msg->rx_len;
total_read_len = 4;
+
+ /*
+ * buffer size: header + data
+ * No 32 bits alignment issue, thus offset is 0
+ */
+ buffer_sz = 4;
+
} else {
short_resp = false;
current_read_len = 10;
@@ -1365,8 +1374,30 @@ static int dsi_message_rx(struct dsi_ctrl *dsi_ctrl,
rd_pkt_size = current_read_len;
total_read_len = current_read_len + 6;
+ /*
+ * buffer size: header + data + footer, rounded up to 4 bytes
+ * Out of bound can occurs is rx_len is not aligned to size 4.
+ * We are reading 32 bits registers, and converting
+ * the data to CPU endianness, thus inserting garbage data
+ * at the beginning of buffer.
+ */
+ buffer_sz = ALIGN(4 + msg->rx_len + 2, 4);
+ if (buffer_sz < 16)
+ buffer_sz = 16;
}
- buff = msg->rx_buf;
+
+ pr_debug("short_resp %d, msg->rx_len %zd, rd_pkt_size %u\n",
+ short_resp, msg->rx_len, rd_pkt_size);
+
+ pr_debug("total_read_len %u, buffer_sz %u\n",
+ total_read_len, buffer_sz);
+
+ buff = kzalloc(buffer_sz, GFP_KERNEL);
+ if (!buff) {
+ rc = -ENOMEM;
+ goto error;
+ }
+ head = buff;
while (!read_done) {
rc = dsi_set_max_return_size(dsi_ctrl, msg, rd_pkt_size);
@@ -1423,13 +1454,22 @@ static int dsi_message_rx(struct dsi_ctrl *dsi_ctrl,
}
}
+ buff = head;
+ for (i = 0; i < buffer_sz; i += 4)
+ pr_debug("buffer[%d-%d] = %08x\n",
+ i, i + 3, *((u32 *)&buff[i]));
+
if (hw_read_cnt < 16 && !short_resp)
- buff = msg->rx_buf + (16 - hw_read_cnt);
+ header_offset = (16 - hw_read_cnt);
else
- buff = msg->rx_buf;
+ header_offset = 0;
+
+ pr_debug("hw_read_cnt %d, header_offset %d\n",
+ hw_read_cnt, header_offset);
/* parse the data read from panel */
- cmd = buff[0];
+ cmd = buff[header_offset];
+ pr_debug("response type %d\n", cmd);
switch (cmd) {
case MIPI_DSI_RX_ACKNOWLEDGE_AND_ERROR_REPORT:
pr_err("Rx ACK_ERROR\n");
@@ -1437,15 +1477,15 @@ static int dsi_message_rx(struct dsi_ctrl *dsi_ctrl,
break;
case MIPI_DSI_RX_GENERIC_SHORT_READ_RESPONSE_1BYTE:
case MIPI_DSI_RX_DCS_SHORT_READ_RESPONSE_1BYTE:
- rc = dsi_parse_short_read1_resp(msg, buff);
+ rc = dsi_parse_short_read1_resp(msg, &buff[header_offset]);
break;
case MIPI_DSI_RX_GENERIC_SHORT_READ_RESPONSE_2BYTE:
case MIPI_DSI_RX_DCS_SHORT_READ_RESPONSE_2BYTE:
- rc = dsi_parse_short_read2_resp(msg, buff);
+ rc = dsi_parse_short_read2_resp(msg, &buff[header_offset]);
break;
case MIPI_DSI_RX_GENERIC_LONG_READ_RESPONSE:
case MIPI_DSI_RX_DCS_LONG_READ_RESPONSE:
- rc = dsi_parse_long_read_resp(msg, buff);
+ rc = dsi_parse_long_read_resp(msg, &buff[header_offset]);
break;
default:
pr_warn("Invalid response\n");
@@ -1453,6 +1493,7 @@ static int dsi_message_rx(struct dsi_ctrl *dsi_ctrl,
}
error:
+ kfree(buff);
return rc;
}
@@ -2799,8 +2840,10 @@ int dsi_ctrl_cmd_transfer(struct dsi_ctrl *dsi_ctrl,
if (flags & DSI_CTRL_CMD_READ) {
rc = dsi_message_rx(dsi_ctrl, msg, flags);
- if (rc <= 0)
- pr_err("read message failed read length, rc=%d\n", rc);
+ if (rc < 0)
+ pr_err("read message failed read, rc=%d\n", rc);
+ else if (rc == 0)
+ pr_warn("empty message back\n");
} else {
rc = dsi_message_tx(dsi_ctrl, msg, flags);
if (rc)
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_ctrl_hw.h b/drivers/gpu/drm/msm/dsi-staging/dsi_ctrl_hw.h
index d6fab59..8cf4834 100644
--- a/drivers/gpu/drm/msm/dsi-staging/dsi_ctrl_hw.h
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_ctrl_hw.h
@@ -551,7 +551,7 @@ struct dsi_ctrl_hw_ops {
* @rd_buf: Buffer where data will be read into.
* @read_offset: Offset from where to read.
* @rx_byte: Number of bytes to be read.
- * @pkt_size: Size of response expected.
+ * @pkt_size: Size of response expected.
* @hw_read_cnt: Actual number of bytes read by HW.
*/
u32 (*get_cmd_read_data)(struct dsi_ctrl_hw *ctrl,
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_defs.h b/drivers/gpu/drm/msm/dsi-staging/dsi_defs.h
index 6540182..e133c4d 100644
--- a/drivers/gpu/drm/msm/dsi-staging/dsi_defs.h
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_defs.h
@@ -238,6 +238,8 @@ enum dsi_dfps_type {
* @DSI_CMD_SET_LP1: Low power mode 1
* @DSI_CMD_SET_LP2: Low power mode 2
* @DSI_CMD_SET_NOLP: Low power mode disable
+ * @DSI_CMD_SET_VR: VR mode (low persistence)
+ * @DSI_CMD_SET_NOVR: VR mode disable
* @DSI_CMD_SET_PPS: DSC PPS command
* @DSI_CMD_SET_ROI: Panel ROI update
* @DSI_CMD_SET_TIMING_SWITCH: Timing switch
@@ -262,6 +264,8 @@ enum dsi_cmd_set_type {
DSI_CMD_SET_LP1,
DSI_CMD_SET_LP2,
DSI_CMD_SET_NOLP,
+ DSI_CMD_SET_VR,
+ DSI_CMD_SET_NOVR,
DSI_CMD_SET_PPS,
DSI_CMD_SET_ROI,
DSI_CMD_SET_TIMING_SWITCH,
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_display.c b/drivers/gpu/drm/msm/dsi-staging/dsi_display.c
index b36ad82..474dc3e 100644
--- a/drivers/gpu/drm/msm/dsi-staging/dsi_display.c
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_display.c
@@ -114,62 +114,6 @@ void dsi_rect_intersect(const struct dsi_rect *r1,
}
}
-int dsi_display_set_backlight(void *display, u32 bl_lvl)
-{
- struct dsi_display *dsi_display = display;
- struct dsi_panel *panel;
- u32 bl_scale, bl_scale_ad;
- u64 bl_temp;
- int rc = 0;
-
- if (dsi_display == NULL || dsi_display->panel == NULL)
- return -EINVAL;
-
- panel = dsi_display->panel;
-
- mutex_lock(&panel->panel_lock);
- if (!dsi_panel_initialized(panel)) {
- rc = -EINVAL;
- goto error;
- }
-
- panel->bl_config.bl_level = bl_lvl;
-
- /* scale backlight */
- bl_scale = panel->bl_config.bl_scale;
- bl_temp = bl_lvl * bl_scale / MAX_BL_SCALE_LEVEL;
-
- bl_scale_ad = panel->bl_config.bl_scale_ad;
- bl_temp = (u32)bl_temp * bl_scale_ad / MAX_AD_BL_SCALE_LEVEL;
-
- pr_debug("bl_scale = %u, bl_scale_ad = %u, bl_lvl = %u\n",
- bl_scale, bl_scale_ad, (u32)bl_temp);
-
- rc = dsi_display_clk_ctrl(dsi_display->dsi_clk_handle,
- DSI_CORE_CLK, DSI_CLK_ON);
- if (rc) {
- pr_err("[%s] failed to enable DSI core clocks, rc=%d\n",
- dsi_display->name, rc);
- goto error;
- }
-
- rc = dsi_panel_set_backlight(panel, (u32)bl_temp);
- if (rc)
- pr_err("unable to set backlight\n");
-
- rc = dsi_display_clk_ctrl(dsi_display->dsi_clk_handle,
- DSI_CORE_CLK, DSI_CLK_OFF);
- if (rc) {
- pr_err("[%s] failed to disable DSI core clocks, rc=%d\n",
- dsi_display->name, rc);
- goto error;
- }
-
-error:
- mutex_unlock(&panel->panel_lock);
- return rc;
-}
-
static int dsi_display_cmd_engine_enable(struct dsi_display *display)
{
int rc = 0;
@@ -563,15 +507,15 @@ static int dsi_display_read_status(struct dsi_display_ctrl *ctrl,
lenp = config->status_valid_params ?: config->status_cmds_rlen;
count = config->status_cmd.count;
cmds = config->status_cmd.cmds;
+ if (cmds->last_command) {
+ cmds->msg.flags |= MIPI_DSI_MSG_LASTCOMMAND;
+ flags |= DSI_CTRL_CMD_LAST_COMMAND;
+ }
flags |= (DSI_CTRL_CMD_FETCH_MEMORY | DSI_CTRL_CMD_READ |
DSI_CTRL_CMD_CUSTOM_DMA_SCHED);
for (i = 0; i < count; ++i) {
memset(config->status_buf, 0x0, SZ_4K);
- if (cmds[i].last_command) {
- cmds[i].msg.flags |= MIPI_DSI_MSG_LASTCOMMAND;
- flags |= DSI_CTRL_CMD_LAST_COMMAND;
- }
cmds[i].msg.rx_buf = config->status_buf;
cmds[i].msg.rx_len = config->status_cmds_rlen[i];
rc = dsi_ctrl_cmd_transfer(ctrl->ctrl, &cmds[i].msg, flags);
@@ -652,7 +596,7 @@ static int dsi_display_status_reg_read(struct dsi_display *display)
rc = dsi_display_validate_status(ctrl, display->panel);
if (rc <= 0) {
- pr_err("[%s] read status failed on slave,rc=%d\n",
+ pr_err("[%s] read status failed on master,rc=%d\n",
display->name, rc);
goto exit;
}
@@ -864,6 +808,191 @@ enum dsi_pixel_format dsi_display_get_dst_format(void *display)
return format;
}
+int dsi_display_get_esd_mode(void *display)
+{
+ struct dsi_display *dsi_display = display;
+ struct dsi_panel *panel;
+ int esd_mode;
+
+ if (!dsi_display)
+ return -ENODEV;
+
+ panel = dsi_display->panel;
+ if (!panel)
+ return -ENODEV;
+
+ esd_mode = panel->esd_config.status_mode;
+ if (esd_mode < 0 || esd_mode >= ESD_MODE_MAX)
+ return -EINVAL;
+
+ return esd_mode;
+}
+
+static void dsi_display_esd_irq_work(struct work_struct *work)
+{
+ struct dsi_display *display;
+ struct drm_connector *connector;
+
+ display = container_of(work, struct dsi_display, esd_irq_work);
+ connector = display->drm_conn;
+ if (!connector) {
+ pr_err("invalid drm connector\n");
+ return;
+ }
+
+ sde_connector_report_panel_dead(connector);
+}
+
+static irqreturn_t dsi_display_esd_irq_handler(int irq, void *data)
+{
+ struct dsi_display *display = (struct dsi_display *)data;
+
+ if (!display)
+ pr_err("invalid dsi display\n");
+ else
+ schedule_work(&display->esd_irq_work);
+
+ return IRQ_HANDLED;
+}
+
+void dsi_display_register_esd_irq(struct dsi_display *display)
+{
+ struct platform_device *pdev;
+ struct dsi_panel *panel;
+ int rc;
+
+ if (!display) {
+ pr_err("invalid dsi display\n");
+ return;
+ }
+
+ pdev = display->pdev;
+ if (!pdev) {
+ pr_err("invalid platform device\n");
+ return;
+ }
+
+ panel = display->panel;
+ if (!panel) {
+ pr_err("invalid panel\n");
+ return;
+ }
+
+ if (gpio_is_valid(panel->esd_config.irq_gpio)) {
+ rc = devm_request_irq(&pdev->dev,
+ gpio_to_irq(panel->esd_config.irq_gpio),
+ dsi_display_esd_irq_handler,
+ panel->esd_config.irq_mode,
+ "esd_isr", display);
+ if (rc) {
+ pr_err("esd irq request failed\n");
+ return;
+ }
+ } else {
+ pr_err("esd irq gpio is invalid\n");
+ return;
+ }
+
+ disable_irq(gpio_to_irq(panel->esd_config.irq_gpio));
+ panel->esd_config.irq_enabled = false;
+
+ pr_info("register esd irq success\n");
+}
+
+void dsi_display_esd_irq_prepare(struct dsi_display *display)
+{
+ struct dsi_panel *panel;
+
+ if (!display) {
+ pr_err("invalid dsi display\n");
+ return;
+ }
+
+ panel = display->panel;
+ if (!panel) {
+ pr_err("invalid panel\n");
+ return;
+ }
+
+ INIT_WORK(&display->esd_irq_work,
+ dsi_display_esd_irq_work);
+ dsi_display_register_esd_irq(display);
+
+ panel->esd_config.irq_mode_prepared = true;
+}
+
+void dsi_display_esd_irq_configure(struct dsi_display *display,
+ bool enable)
+{
+ struct dsi_panel *panel;
+
+ if (!display) {
+ pr_err("invalid dsi display\n");
+ return;
+ }
+
+ panel = display->panel;
+ if (!panel) {
+ pr_err("invalid dsi panel\n");
+ return;
+ }
+
+ if (enable && !panel->esd_config.irq_enabled) {
+ enable_irq(gpio_to_irq(panel->esd_config.irq_gpio));
+ panel->esd_config.irq_enabled = true;
+ } else if (!enable && panel->esd_config.irq_enabled) {
+ disable_irq(gpio_to_irq(panel->esd_config.irq_gpio));
+ panel->esd_config.irq_enabled = false;
+ }
+}
+
+void dsi_display_esd_irq_mode_switch(struct dsi_display *display,
+ bool enable)
+{
+ struct dsi_panel *panel;
+ struct drm_connector *connector;
+ struct sde_connector *conn;
+
+ if (!display) {
+ pr_err("invalid dsi display\n");
+ return;
+ }
+
+ panel = display->panel;
+ if (!panel) {
+ pr_err("invalid dsi panel\n");
+ return;
+ }
+
+ connector = display->drm_conn;
+ if (!connector) {
+ pr_err("invalid drm connector\n");
+ return;
+ }
+ conn = to_sde_connector(connector);
+
+ if ((panel->esd_config.status_mode != ESD_MODE_IRQ_GPIO) &&
+ enable) {
+ /* Cancel any pending ESD status check */
+ cancel_delayed_work_sync(&conn->status_work);
+ conn->esd_status_check = false;
+
+ if (!panel->esd_config.irq_mode_prepared)
+ dsi_display_esd_irq_prepare(display);
+ dsi_display_esd_irq_configure(display, true);
+ } else if ((panel->esd_config.status_mode == ESD_MODE_IRQ_GPIO) &&
+ !enable) {
+ dsi_display_esd_irq_configure(display, false);
+ /* Cancel any pending ESD IRQ work */
+ cancel_work_sync(&display->esd_irq_work);
+
+ /* Schedule ESD status check */
+ schedule_delayed_work(&conn->status_work,
+ msecs_to_jiffies(STATUS_CHECK_INTERVAL_MS));
+ conn->esd_status_check = true;
+ }
+}
+
static void _dsi_display_setup_misr(struct dsi_display *display)
{
int i;
@@ -919,6 +1048,9 @@ int dsi_display_set_power(struct drm_connector *connector,
case SDE_MODE_DPMS_LP2:
rc = dsi_panel_set_lp2(display->panel);
break;
+ case SDE_MODE_DPMS_OFF:
+ /* nothing to do */
+ break;
default:
rc = dsi_panel_set_nolp(display->panel);
break;
@@ -1209,6 +1341,7 @@ static ssize_t debugfs_alter_esd_check_mode(struct file *file,
goto error;
if (!strcmp(buf, "te_signal_check\n")) {
+ dsi_display_esd_irq_mode_switch(display, false);
esd_config->status_mode = ESD_MODE_PANEL_TE;
dsi_display_change_te_irq_status(display, true);
}
@@ -1222,11 +1355,25 @@ static ssize_t debugfs_alter_esd_check_mode(struct file *file,
rc = user_len;
goto error;
}
+ dsi_display_esd_irq_mode_switch(display, false);
esd_config->status_mode = ESD_MODE_REG_READ;
if (dsi_display_is_te_based_esd(display))
dsi_display_change_te_irq_status(display, false);
}
+ if (!strcmp(buf, "irq_check\n")) {
+ rc = dsi_panel_parse_esd_irq_gpio_configs(display->panel,
+ display->panel_of);
+ if (rc) {
+ pr_err("failed to alter esd check mode,rc=%d\n",
+ rc);
+ rc = user_len;
+ goto error;
+ }
+ dsi_display_esd_irq_mode_switch(display, true);
+ esd_config->status_mode = ESD_MODE_IRQ_GPIO;
+ }
+
rc = len;
error:
kfree(buf);
@@ -1272,10 +1419,13 @@ static ssize_t debugfs_read_esd_check_mode(struct file *file,
}
if (esd_config->status_mode == ESD_MODE_REG_READ)
- rc = snprintf(buf, len, "reg_read");
+ rc = snprintf(buf, len, "reg_read\n");
if (esd_config->status_mode == ESD_MODE_PANEL_TE)
- rc = snprintf(buf, len, "te_signal_check");
+ rc = snprintf(buf, len, "te_signal_check\n");
+
+ if (esd_config->status_mode == ESD_MODE_IRQ_GPIO)
+ rc = snprintf(buf, len, "err_irq_check\n");
output_mode:
if (!rc) {
@@ -2773,10 +2923,14 @@ static ssize_t dsi_host_transfer(struct mipi_dsi_host *host,
} else {
int ctrl_idx = (msg->flags & MIPI_DSI_MSG_UNICAST) ?
msg->ctrl : 0;
+ u32 flags = DSI_CTRL_CMD_FETCH_MEMORY;
+
+ if (msg->rx_buf)
+ flags |= DSI_CTRL_CMD_READ;
rc = dsi_ctrl_cmd_transfer(display->ctrl[ctrl_idx].ctrl, msg,
- DSI_CTRL_CMD_FETCH_MEMORY);
- if (rc) {
+ flags);
+ if (rc < 0) {
pr_err("[%s] cmd transfer failed, rc=%d\n",
display->name, rc);
goto error_disable_cmd_engine;
@@ -4518,6 +4672,8 @@ static int dsi_display_bind(struct device *dev,
goto error_host_deinit;
}
+ dsi_panel_debugfs_init(display->panel, display->root);
+
pr_info("Successfully bind display panel '%s'\n", display->name);
display->drm_dev = drm;
@@ -4539,6 +4695,9 @@ static int dsi_display_bind(struct device *dev,
/* register te irq handler */
dsi_display_register_te_irq(display);
+ if (dsi_display_get_esd_mode(display) == ESD_MODE_IRQ_GPIO)
+ dsi_display_esd_irq_prepare(display);
+
goto error;
error_host_deinit:
@@ -5123,16 +5282,18 @@ int dsi_display_get_modes(struct dsi_display *display,
return -EINVAL;
}
- *out_modes = NULL;
-
mutex_lock(&display->display_lock);
+ if (display->modes) {
+ pr_debug("return existing mode list\n");
+ goto done;
+ }
+
+ *out_modes = NULL;
rc = dsi_display_get_mode_count_no_lock(display, &total_mode_count);
if (rc)
goto error;
- /* free any previously probed modes */
- kfree(display->modes);
display->modes = kcalloc(total_mode_count, sizeof(*display->modes),
GFP_KERNEL);
@@ -5212,6 +5373,7 @@ int dsi_display_get_modes(struct dsi_display *display,
}
}
+done:
*out_modes = display->modes;
rc = 0;
@@ -5832,7 +5994,8 @@ int dsi_display_prepare(struct dsi_display *display)
if (mode->dsi_mode_flags & DSI_MODE_FLAG_DMS) {
if (display->is_cont_splash_enabled) {
pr_err("DMS is not supposed to be set on first frame\n");
- return -EINVAL;
+ rc = -EINVAL;
+ goto error;
}
/* update dsi ctrl for new mode */
rc = dsi_display_pre_switch(display);
@@ -6272,7 +6435,7 @@ int dsi_display_enable(struct dsi_display *display)
int dsi_display_post_enable(struct dsi_display *display)
{
- int rc = 0;
+ int rc = 0, err;
if (!display) {
pr_err("Invalid params\n");
@@ -6286,11 +6449,22 @@ int dsi_display_post_enable(struct dsi_display *display)
pr_err("[%s] panel post-enable failed, rc=%d\n",
display->name, rc);
+ /* shouldn't read sn if post_enable fails */
+ if (!rc && !display->panel->vendor_info.is_sn) {
+ err = dsi_panel_get_sn(display->panel);
+ if (err)
+ pr_err("[%s] failed to get SN, err=%d\n",
+ display->name, err);
+ }
+
/* remove the clk vote for CMD mode panels */
if (display->config.panel_mode == DSI_OP_CMD_MODE)
dsi_display_clk_ctrl(display->dsi_clk_handle,
DSI_ALL_CLKS, DSI_CLK_OFF);
+ if (dsi_display_get_esd_mode(display) == ESD_MODE_IRQ_GPIO)
+ dsi_display_esd_irq_configure(display, true);
+
mutex_unlock(&display->display_lock);
return rc;
}
@@ -6306,6 +6480,12 @@ int dsi_display_pre_disable(struct dsi_display *display)
mutex_lock(&display->display_lock);
+ if (dsi_display_get_esd_mode(display) == ESD_MODE_IRQ_GPIO) {
+ dsi_display_esd_irq_configure(display, false);
+ /* Cancel any pending ESD IRQ work */
+ cancel_work_sync(&display->esd_irq_work);
+ }
+
/* enable the clk vote for CMD mode panels */
if (display->config.panel_mode == DSI_OP_CMD_MODE)
dsi_display_clk_ctrl(display->dsi_clk_handle,
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_display.h b/drivers/gpu/drm/msm/dsi-staging/dsi_display.h
index 5612016..889f9de 100644
--- a/drivers/gpu/drm/msm/dsi-staging/dsi_display.h
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_display.h
@@ -166,6 +166,7 @@ struct dsi_display_clk_info {
* @misr_enable Frame MISR enable/disable
* @misr_frame_count Number of frames to accumulate the MISR value
* @esd_trigger field indicating ESD trigger through debugfs
+ * @esd_irq_work: work object to perform panel recovery for ESD IRQ mode
*/
struct dsi_display {
struct platform_device *pdev;
@@ -188,6 +189,7 @@ struct dsi_display {
/* panel info */
struct dsi_panel *panel;
struct device_node *panel_of;
+ struct device *panel_info_dev;
struct dsi_display_mode *modes;
@@ -237,6 +239,9 @@ struct dsi_display {
struct work_struct fifo_underflow_work;
struct work_struct fifo_overflow_work;
struct work_struct lp_rx_timeout_work;
+
+ /* panel recovery */
+ struct work_struct esd_irq_work;
};
/**
@@ -665,4 +670,12 @@ int dsi_display_cont_splash_config(void *display);
int dsi_display_get_panel_vfp(void *display,
int h_active, int v_active);
+/**
+ * dsi_display_get_esd_mode() - get ESD mode from DSI display
+ * @display: Handle to display
+ *
+ * Return: ESD mode or error code
+ */
+int dsi_display_get_esd_mode(void *display);
+
#endif /* _DSI_DISPLAY_H_ */
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_drm.c b/drivers/gpu/drm/msm/dsi-staging/dsi_drm.c
index 6b5bfb4..2e9750e 100644
--- a/drivers/gpu/drm/msm/dsi-staging/dsi_drm.c
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_drm.c
@@ -180,6 +180,9 @@ static void dsi_bridge_pre_enable(struct drm_bridge *bridge)
if (rc)
pr_err("Continuous splash pipeline cleanup failed, rc=%d\n",
rc);
+
+ if (c_bridge->display && c_bridge->display->drm_conn)
+ sde_connector_helper_bridge_enable(c_bridge->display->drm_conn);
}
static void dsi_bridge_enable(struct drm_bridge *bridge)
@@ -204,9 +207,6 @@ static void dsi_bridge_enable(struct drm_bridge *bridge)
if (rc)
pr_err("[%d] DSI display post enabled failed, rc=%d\n",
c_bridge->id, rc);
-
- if (display && display->drm_conn)
- sde_connector_helper_bridge_enable(display->drm_conn);
}
static void dsi_bridge_disable(struct drm_bridge *bridge)
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_panel.c b/drivers/gpu/drm/msm/dsi-staging/dsi_panel.c
index cb9c1fa..4fc63ae 100644
--- a/drivers/gpu/drm/msm/dsi-staging/dsi_panel.c
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_panel.c
@@ -13,6 +13,7 @@
*/
#define pr_fmt(fmt) "msm-dsi-panel:[%s:%d] " fmt, __func__, __LINE__
+#include <linux/debugfs.h>
#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/gpio.h>
@@ -34,6 +35,7 @@
#define DSI_PANEL_DEFAULT_LABEL "Default dsi panel"
#define EXT_BRIDGE_DEFAULT_LABEL "Default ext bridge"
+#define DSI_PANEL_VENDOR_DEFAULT_LABEL "Undefined vendor"
#define DEFAULT_MDP_TRANSFER_TIME 14000
@@ -43,6 +45,9 @@
#define MAX_PANEL_JITTER 10
#define DEFAULT_PANEL_PREFILL_LINES 25
+#define LONG_READ_RESPONSE_BUFFER_SIZE 16
+#define LONG_READ_RESPONSE_HEADER_CRC_SIZE 6
+
enum dsi_dsc_ratio_type {
DSC_8BPC_8BPP,
DSC_10BPC_8BPP,
@@ -100,6 +105,14 @@ static char dsi_dsc_rc_range_max_qp_1_1_scr1[][15] = {
static char dsi_dsc_rc_range_bpg_offset[] = {2, 0, 0, -2, -4, -6, -8, -8,
-8, -10, -10, -12, -12, -12, -12};
+/*
+ * VR utility function. Forward-declared so that other commands
+ * can clear VR mode if necessary.
+ */
+
+static int dsi_panel_clear_vr_locked(struct dsi_panel *panel);
+static int dsi_panel_update_hbm_locked(struct dsi_panel *panel, bool enable);
+
int dsi_dsc_create_pps_buf_cmd(struct msm_display_dsc_info *dsc, char *buf,
int pps_id)
{
@@ -431,12 +444,6 @@ static int dsi_panel_power_on(struct dsi_panel *panel)
{
int rc = 0;
- rc = dsi_pwr_enable_regulator(&panel->power_info, true);
- if (rc) {
- pr_err("[%s] failed to enable vregs, rc=%d\n", panel->name, rc);
- goto exit;
- }
-
rc = dsi_panel_set_pinctrl_state(panel, true);
if (rc) {
pr_err("[%s] failed to set pinctrl, rc=%d\n", panel->name, rc);
@@ -492,37 +499,24 @@ static int dsi_panel_power_off(struct dsi_panel *panel)
return rc;
}
-static int dsi_panel_tx_cmd_set(struct dsi_panel *panel,
- enum dsi_cmd_set_type type)
+
+int dsi_panel_cmd_set_transfer(struct dsi_panel *panel,
+ struct dsi_panel_cmd_set *cmd)
{
int rc = 0, i = 0;
ssize_t len;
struct dsi_cmd_desc *cmds;
- u32 count;
- enum dsi_cmd_set_state state;
- struct dsi_display_mode *mode;
const struct mipi_dsi_host_ops *ops = panel->host->ops;
- if (!panel || !panel->cur_mode)
- return -EINVAL;
+ cmds = cmd->cmds;
- if (panel->type == EXT_BRIDGE)
- return 0;
-
- mode = panel->cur_mode;
-
- cmds = mode->priv_info->cmd_sets[type].cmds;
- count = mode->priv_info->cmd_sets[type].count;
- state = mode->priv_info->cmd_sets[type].state;
-
- if (count == 0) {
- pr_debug("[%s] No commands to be sent for state(%d)\n",
- panel->name, type);
+ if (cmd->count == 0) {
+ pr_debug("[%s] No commands to be sent\n", panel->name);
goto error;
}
- for (i = 0; i < count; i++) {
- if (state == DSI_CMD_SET_STATE_LP)
+ for (i = 0; i < cmd->count; i++) {
+ if (cmd->state == DSI_CMD_SET_STATE_LP)
cmds->msg.flags |= MIPI_DSI_MSG_USE_LPM;
if (cmds->last_command)
@@ -531,7 +525,7 @@ static int dsi_panel_tx_cmd_set(struct dsi_panel *panel,
len = ops->transfer(panel->host, &cmds->msg);
if (len < 0) {
rc = len;
- pr_err("failed to set cmds(%d), rc=%d\n", type, rc);
+ pr_err("failed to set cmds, rc=%d\n", rc);
goto error;
}
if (cmds->post_wait_ms)
@@ -543,6 +537,23 @@ static int dsi_panel_tx_cmd_set(struct dsi_panel *panel,
return rc;
}
+static int dsi_panel_tx_cmd_set(struct dsi_panel *panel,
+ enum dsi_cmd_set_type type)
+{
+ struct dsi_display_mode *mode;
+
+ if (!panel || !panel->cur_mode)
+ return -EINVAL;
+
+ if (panel->type == EXT_BRIDGE)
+ return 0;
+
+ mode = panel->cur_mode;
+
+ return dsi_panel_cmd_set_transfer(panel,
+ &mode->priv_info->cmd_sets[type]);
+}
+
static int dsi_panel_pinctrl_deinit(struct dsi_panel *panel)
{
int rc = 0;
@@ -585,133 +596,6 @@ static int dsi_panel_pinctrl_init(struct dsi_panel *panel)
return rc;
}
-#ifdef CONFIG_LEDS_TRIGGERS
-static int dsi_panel_led_bl_register(struct dsi_panel *panel,
- struct dsi_backlight_config *bl)
-{
- int rc = 0;
-
- led_trigger_register_simple("bkl-trigger", &bl->wled);
-
- /* LED APIs don't tell us directly whether a classdev has yet
- * been registered to service this trigger. Until classdev is
- * registered, calling led_trigger has no effect, and doesn't
- * fail. Classdevs are associated with any registered triggers
- * when they do register, but that is too late for FBCon.
- * Check the cdev list directly and defer if appropriate.
- */
- if (!bl->wled) {
- pr_err("[%s] backlight registration failed\n", panel->name);
- rc = -EINVAL;
- } else {
- read_lock(&bl->wled->leddev_list_lock);
- if (list_empty(&bl->wled->led_cdevs))
- rc = -EPROBE_DEFER;
- read_unlock(&bl->wled->leddev_list_lock);
-
- if (rc) {
- pr_info("[%s] backlight %s not ready, defer probe\n",
- panel->name, bl->wled->name);
- led_trigger_unregister_simple(bl->wled);
- }
- }
-
- return rc;
-}
-#else
-static int dsi_panel_led_bl_register(struct dsi_panel *panel,
- struct dsi_backlight_config *bl)
-{
- return 0;
-}
-#endif
-
-static int dsi_panel_update_backlight(struct dsi_panel *panel,
- u32 bl_lvl)
-{
- int rc = 0;
- struct mipi_dsi_device *dsi;
-
- if (!panel || (bl_lvl > 0xffff)) {
- pr_err("invalid params\n");
- return -EINVAL;
- }
-
- dsi = &panel->mipi_device;
-
- rc = mipi_dsi_dcs_set_display_brightness(dsi, bl_lvl);
- if (rc < 0)
- pr_err("failed to update dcs backlight:%d\n", bl_lvl);
-
- return rc;
-}
-
-int dsi_panel_set_backlight(struct dsi_panel *panel, u32 bl_lvl)
-{
- int rc = 0;
- struct dsi_backlight_config *bl = &panel->bl_config;
-
- if (panel->type == EXT_BRIDGE)
- return 0;
-
- pr_debug("backlight type:%d lvl:%d\n", bl->type, bl_lvl);
- switch (bl->type) {
- case DSI_BACKLIGHT_WLED:
- led_trigger_event(bl->wled, bl_lvl);
- break;
- case DSI_BACKLIGHT_DCS:
- rc = dsi_panel_update_backlight(panel, bl_lvl);
- break;
- default:
- pr_err("Backlight type(%d) not supported\n", bl->type);
- rc = -ENOTSUPP;
- }
-
- return rc;
-}
-
-static int dsi_panel_bl_register(struct dsi_panel *panel)
-{
- int rc = 0;
- struct dsi_backlight_config *bl = &panel->bl_config;
-
- switch (bl->type) {
- case DSI_BACKLIGHT_WLED:
- rc = dsi_panel_led_bl_register(panel, bl);
- break;
- case DSI_BACKLIGHT_DCS:
- break;
- default:
- pr_err("Backlight type(%d) not supported\n", bl->type);
- rc = -ENOTSUPP;
- goto error;
- }
-
-error:
- return rc;
-}
-
-static int dsi_panel_bl_unregister(struct dsi_panel *panel)
-{
- int rc = 0;
- struct dsi_backlight_config *bl = &panel->bl_config;
-
- switch (bl->type) {
- case DSI_BACKLIGHT_WLED:
- led_trigger_unregister_simple(bl->wled);
- break;
- case DSI_BACKLIGHT_DCS:
- break;
- default:
- pr_err("Backlight type(%d) not supported\n", bl->type);
- rc = -ENOTSUPP;
- goto error;
- }
-
-error:
- return rc;
-}
-
static int dsi_panel_fw_parse(const struct firmware *fw_entry,
char *id_match, u32 *param_value)
{
@@ -1079,6 +963,170 @@ static int dsi_panel_parse_misc_host_config(struct dsi_host_common_cfg *host,
return 0;
}
+static int dsi_panel_create_sn_buf(struct dsi_panel *panel)
+{
+ struct dsi_panel_vendor_info *const vendor_info = &panel->vendor_info;
+ struct dsi_panel_sn_location *const location = &vendor_info->location;
+ ssize_t rc = 0;
+ u32 sn_str_size;
+ u8 *tmp_sn_buf;
+
+ if (!panel)
+ return -EINVAL;
+
+ if (!location->addr || !location->sn_length) {
+ pr_err("[%s] invalid location\n", __func__);
+ return -EINVAL;
+ }
+
+ // prepare buffer for bin2hex()
+ // e.g. buf[0] = 0x01 -> sn[0] = '0' and sn[1] = '1'
+ sn_str_size = location->sn_length * 2;
+ tmp_sn_buf = kmalloc(sn_str_size + 1, GFP_KERNEL);
+ if (!tmp_sn_buf)
+ return -ENOMEM;
+
+ mutex_lock(&panel->panel_lock);
+ if (!vendor_info->sn) {
+ vendor_info->is_sn = false;
+ vendor_info->sn = tmp_sn_buf;
+ } else
+ kfree(tmp_sn_buf);
+
+ mutex_unlock(&panel->panel_lock);
+ return rc;
+}
+
+static int dsi_panel_release_sn_buf(struct dsi_panel *panel)
+{
+ struct dsi_panel_vendor_info *const vendor_info = &panel->vendor_info;
+ ssize_t rc = 0;
+
+ if (!panel)
+ return -EINVAL;
+
+ mutex_lock(&panel->panel_lock);
+ kfree(vendor_info->sn);
+ vendor_info->sn = NULL;
+ vendor_info->is_sn = false;
+ mutex_unlock(&panel->panel_lock);
+ return rc;
+}
+
+int dsi_panel_get_sn(struct dsi_panel *panel)
+{
+ struct dsi_panel_vendor_info *const vendor_info = &panel->vendor_info;
+ struct dsi_panel_sn_location *const location = &vendor_info->location;
+ ssize_t rc = 0;
+ u32 read_size, read_back_size, sn_str_size, buf_size;
+ u8 *buf;
+
+ if (!panel || !panel->panel_initialized) {
+ pr_err("panel is not ready\n");
+ return -EINVAL;
+ }
+
+ read_size = location->start_byte + location->sn_length;
+ /*
+ * Note:
+ * the buffer size needs read_size + 6 >= 16 for long read response.
+ */
+ buf_size = max_t(u32, read_size + LONG_READ_RESPONSE_HEADER_CRC_SIZE,
+ LONG_READ_RESPONSE_BUFFER_SIZE);
+ buf = kmalloc(buf_size, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ mutex_lock(&panel->panel_lock);
+
+ if (!panel->vendor_info.sn) {
+ rc = -EINVAL;
+ goto out_mutex;
+ }
+
+ read_back_size = mipi_dsi_dcs_read(&panel->mipi_device, location->addr,
+ buf, read_size);
+ if (read_back_size == read_size) {
+ bin2hex(vendor_info->sn, &buf[location->start_byte],
+ location->sn_length);
+ // e.g. buf[0] = 0x01 -> sn[0] = '0' and sn[1] = '1'
+ sn_str_size = location->sn_length * 2;
+ vendor_info->sn[sn_str_size] = '\0';
+ vendor_info->is_sn = true;
+ } else {
+ rc = -EINVAL;
+ pr_err("failed to read: addr=0x%X, read_back_size=%d, read_size=%d\n",
+ location->addr, read_back_size, read_size);
+ }
+out_mutex:
+ mutex_unlock(&panel->panel_lock);
+ kfree(buf);
+ return rc;
+}
+
+static int dsi_panel_parse_sn_location(struct dsi_panel *panel,
+ struct device_node *of_node)
+{
+ struct dsi_panel_vendor_info *const vendor_info = &panel->vendor_info;
+ int rc = 0, len = 0;
+ u32 array[3] = {0};
+
+ if (!panel)
+ return -EINVAL;
+
+ len = of_property_count_u32_elems(of_node,
+ "google,mdss-dsi-panel-sn-location");
+ if (len != 3) {
+ pr_err("[%s] invalid format\n", __func__);
+ rc = -EINVAL;
+ goto error;
+ }
+
+ rc = of_property_read_u32_array(of_node,
+ "google,mdss-dsi-panel-sn-location", array, len);
+ if (rc || !array[0] || !array[2]) {
+ pr_err("[%s] invalid format\n", __func__);
+ rc = -EINVAL;
+ goto error;
+ }
+
+ vendor_info->location.addr = array[0];
+ vendor_info->location.start_byte = array[1];
+ vendor_info->location.sn_length = array[2];
+
+ pr_debug("addr=0x%x, start=%d, length=%d",
+ vendor_info->location.addr, vendor_info->location.start_byte,
+ vendor_info->location.sn_length);
+error:
+ return rc;
+}
+
+static int dsi_panel_parse_vendor_info(struct dsi_panel *panel,
+ struct device_node *of_node)
+{
+ int rc = 0;
+ struct dsi_panel_vendor_info *const vendor_info = &panel->vendor_info;
+
+ if (!panel)
+ return -EINVAL;
+
+ vendor_info->name = of_get_property(of_node,
+ "google,mdss-dsi-panel-vendor", NULL);
+
+ if (!vendor_info->name) {
+ vendor_info->name = DSI_PANEL_VENDOR_DEFAULT_LABEL;
+ rc = -EINVAL;
+ goto error;
+ }
+ rc = dsi_panel_parse_sn_location(panel, of_node);
+ if (rc) {
+ pr_err("[%s] failed to parse the parameter for SN, rc=%d\n",
+ panel->name, rc);
+ }
+error:
+ return rc;
+}
+
static int dsi_panel_parse_host_config(struct dsi_panel *panel,
struct device_node *of_node)
{
@@ -1465,6 +1513,8 @@ const char *cmd_set_prop_map[DSI_CMD_SET_MAX] = {
"qcom,mdss-dsi-lp1-command",
"qcom,mdss-dsi-lp2-command",
"qcom,mdss-dsi-nolp-command",
+ "qcom,mdss-dsi-vr-command",
+ "qcom,mdss-dsi-novr-command",
"PPS not parsed from DTSI, generated dynamically",
"ROI not parsed from DTSI, generated dynamically",
"qcom,mdss-dsi-timing-switch-command",
@@ -1489,6 +1539,8 @@ const char *cmd_set_state_map[DSI_CMD_SET_MAX] = {
"qcom,mdss-dsi-lp1-command-state",
"qcom,mdss-dsi-lp2-command-state",
"qcom,mdss-dsi-nolp-command-state",
+ "qcom,mdss-dsi-vr-command-state",
+ "qcom,mdss-dsi-novr-command-state",
"PPS not parsed from DTSI, generated dynamically",
"ROI not parsed from DTSI, generated dynamically",
"qcom,mdss-dsi-timing-switch-command-state",
@@ -1557,14 +1609,15 @@ static int dsi_panel_create_cmd_packets(const char *data,
return rc;
error_free_payloads:
for (i = i - 1; i >= 0; i--) {
- cmd--;
- kfree(cmd->msg.tx_buf);
+ kfree(cmd[i].msg.tx_buf);
+ cmd[i].msg.tx_buf = NULL;
}
return rc;
}
-void dsi_panel_destroy_cmd_packets(struct dsi_panel_cmd_set *set)
+
+static void dsi_panel_destroy_cmds_packets_buf(struct dsi_panel_cmd_set *set)
{
u32 i = 0;
struct dsi_cmd_desc *cmd;
@@ -1572,8 +1625,14 @@ void dsi_panel_destroy_cmd_packets(struct dsi_panel_cmd_set *set)
for (i = 0; i < set->count; i++) {
cmd = &set->cmds[i];
kfree(cmd->msg.tx_buf);
+ cmd->msg.tx_buf = NULL;
}
+}
+void dsi_panel_destroy_cmd_packets(struct dsi_panel_cmd_set *set)
+{
+
+ dsi_panel_destroy_cmds_packets_buf(set);
kfree(set->cmds);
}
@@ -1592,29 +1651,17 @@ static int dsi_panel_alloc_cmd_packets(struct dsi_panel_cmd_set *cmd,
}
static int dsi_panel_parse_cmd_sets_sub(struct dsi_panel_cmd_set *cmd,
- enum dsi_cmd_set_type type,
- struct device_node *of_node)
+ const char *data,
+ size_t length)
{
int rc = 0;
- u32 length = 0;
- const char *data;
- const char *state;
u32 packet_count = 0;
- data = of_get_property(of_node, cmd_set_prop_map[type], &length);
- if (!data) {
- pr_debug("%s commands not defined\n", cmd_set_prop_map[type]);
- rc = -ENOTSUPP;
- goto error;
- }
-
rc = dsi_panel_get_cmd_pkt_count(data, length, &packet_count);
if (rc) {
pr_err("commands failed, rc=%d\n", rc);
goto error;
}
- pr_debug("[%s] packet-count=%d, %d\n", cmd_set_prop_map[type],
- packet_count, length);
rc = dsi_panel_alloc_cmd_packets(cmd, packet_count);
if (rc) {
@@ -1629,17 +1676,6 @@ static int dsi_panel_parse_cmd_sets_sub(struct dsi_panel_cmd_set *cmd,
goto error_free_mem;
}
- state = of_get_property(of_node, cmd_set_state_map[type], NULL);
- if (!state || !strcmp(state, "dsi_lp_mode")) {
- cmd->state = DSI_CMD_SET_STATE_LP;
- } else if (!strcmp(state, "dsi_hs_mode")) {
- cmd->state = DSI_CMD_SET_STATE_HS;
- } else {
- pr_err("[%s] command state unrecognized-%s\n",
- cmd_set_state_map[type], state);
- goto error_free_mem;
- }
-
return rc;
error_free_mem:
kfree(cmd->cmds);
@@ -1649,9 +1685,56 @@ static int dsi_panel_parse_cmd_sets_sub(struct dsi_panel_cmd_set *cmd,
}
+int dsi_panel_parse_dt_cmd_set(struct device_node *of_node,
+ const char *cmd_str,
+ const char *cmd_state_str,
+ struct dsi_panel_cmd_set *cmd)
+{
+ const char *data;
+ const char *state;
+ enum dsi_cmd_set_state st;
+ u32 length = 0;
+ int rc;
+
+ data = of_get_property(of_node, cmd_str, &length);
+ if (!data) {
+ pr_debug("%s commands not defined\n", cmd_str);
+ return -ENOTSUPP;
+ }
+
+ state = of_get_property(of_node, cmd_state_str, NULL);
+ if (!state || !strcmp(state, "dsi_lp_mode")) {
+ st = DSI_CMD_SET_STATE_LP;
+ } else if (!strcmp(state, "dsi_hs_mode")) {
+ st = DSI_CMD_SET_STATE_HS;
+ } else {
+ pr_err("[%s] command state unrecognized-%s\n",
+ cmd_state_str, state);
+ return -ENOTSUPP;
+ }
+
+ rc = dsi_panel_parse_cmd_sets_sub(cmd, data, length);
+ if (rc)
+ return rc;
+
+ cmd->state = st;
+
+ return 0;
+}
+
+
+static int dsi_panel_parse_cmd_sets_dt(struct dsi_panel_cmd_set *cmd,
+ enum dsi_cmd_set_type type,
+ struct device_node *of_node)
+{
+ return dsi_panel_parse_dt_cmd_set(of_node, cmd_set_prop_map[type],
+ cmd_set_state_map[type], cmd);
+}
+
static int dsi_panel_parse_cmd_sets(
struct dsi_display_mode_priv_info *priv_info,
- struct device_node *of_node)
+ struct device_node *of_node,
+ struct dsi_panel *panel)
{
int rc = 0;
struct dsi_panel_cmd_set *set;
@@ -1672,9 +1755,10 @@ static int dsi_panel_parse_cmd_sets(
if (rc)
pr_err("failed to allocate cmd set %d, rc = %d\n",
i, rc);
- set->state = DSI_CMD_SET_STATE_LP;
+ set->state = panel->hs_pps ? DSI_CMD_SET_STATE_HS :
+ DSI_CMD_SET_STATE_LP;
} else {
- rc = dsi_panel_parse_cmd_sets_sub(set, i, of_node);
+ rc = dsi_panel_parse_cmd_sets_dt(set, i, of_node);
if (rc)
pr_debug("failed to parse set %d\n", i);
}
@@ -1754,6 +1838,8 @@ static int dsi_panel_parse_reset_sequence(struct dsi_panel *panel,
static int dsi_panel_parse_misc_features(struct dsi_panel *panel,
struct device_node *of_node)
{
+ u32 val;
+
panel->ulps_enabled =
of_property_read_bool(of_node, "qcom,ulps-enabled");
@@ -1768,6 +1854,18 @@ static int dsi_panel_parse_misc_features(struct dsi_panel *panel,
panel->te_using_watchdog_timer = of_property_read_bool(of_node,
"qcom,mdss-dsi-te-using-wd");
+ panel->lp11_init =
+ of_property_read_bool(of_node, "qcom,mdss-dsi-lp11-init");
+ if (!of_property_read_u32(of_node,
+ "qcom,mdss-dsi-init-delay-us",
+ &val)) {
+ pr_debug("Panel init delay specified: %d\n", val);
+ panel->init_delay_us = val;
+ } else {
+ panel->init_delay_us = 0;
+ }
+ panel->hs_pps =
+ of_property_read_bool(of_node, "qcom,mdss-dsi-hs-pps");
panel->sync_broadcast_en = of_property_read_bool(of_node,
"qcom,cmd-sync-wait-broadcast");
@@ -1907,121 +2005,6 @@ static int dsi_panel_parse_gpios(struct dsi_panel *panel,
return rc;
}
-static int dsi_panel_parse_bl_pwm_config(struct dsi_backlight_config *config,
- struct device_node *of_node)
-{
- int rc = 0;
- u32 val;
-
- rc = of_property_read_u32(of_node, "qcom,dsi-bl-pmic-bank-select",
- &val);
- if (rc) {
- pr_err("bl-pmic-bank-select is not defined, rc=%d\n", rc);
- goto error;
- }
- config->pwm_pmic_bank = val;
-
- rc = of_property_read_u32(of_node, "qcom,dsi-bl-pmic-pwm-frequency",
- &val);
- if (rc) {
- pr_err("bl-pmic-bank-select is not defined, rc=%d\n", rc);
- goto error;
- }
- config->pwm_period_usecs = val;
-
- config->pwm_pmi_control = of_property_read_bool(of_node,
- "qcom,mdss-dsi-bl-pwm-pmi");
-
- config->pwm_gpio = of_get_named_gpio(of_node,
- "qcom,mdss-dsi-pwm-gpio",
- 0);
- if (!gpio_is_valid(config->pwm_gpio)) {
- pr_err("pwm gpio is invalid\n");
- rc = -EINVAL;
- goto error;
- }
-
-error:
- return rc;
-}
-
-static int dsi_panel_parse_bl_config(struct dsi_panel *panel,
- struct device_node *of_node)
-{
- int rc = 0;
- const char *bl_type;
- u32 val = 0;
-
- bl_type = of_get_property(of_node,
- "qcom,mdss-dsi-bl-pmic-control-type",
- NULL);
- if (!bl_type) {
- panel->bl_config.type = DSI_BACKLIGHT_UNKNOWN;
- } else if (!strcmp(bl_type, "bl_ctrl_pwm")) {
- panel->bl_config.type = DSI_BACKLIGHT_PWM;
- } else if (!strcmp(bl_type, "bl_ctrl_wled")) {
- panel->bl_config.type = DSI_BACKLIGHT_WLED;
- } else if (!strcmp(bl_type, "bl_ctrl_dcs")) {
- panel->bl_config.type = DSI_BACKLIGHT_DCS;
- } else {
- pr_debug("[%s] bl-pmic-control-type unknown-%s\n",
- panel->name, bl_type);
- panel->bl_config.type = DSI_BACKLIGHT_UNKNOWN;
- }
-
- panel->bl_config.bl_scale = MAX_BL_SCALE_LEVEL;
- panel->bl_config.bl_scale_ad = MAX_AD_BL_SCALE_LEVEL;
-
- rc = of_property_read_u32(of_node, "qcom,mdss-dsi-bl-min-level", &val);
- if (rc) {
- pr_debug("[%s] bl-min-level unspecified, defaulting to zero\n",
- panel->name);
- panel->bl_config.bl_min_level = 0;
- } else {
- panel->bl_config.bl_min_level = val;
- }
-
- rc = of_property_read_u32(of_node, "qcom,mdss-dsi-bl-max-level", &val);
- if (rc) {
- pr_debug("[%s] bl-max-level unspecified, defaulting to max level\n",
- panel->name);
- panel->bl_config.bl_max_level = MAX_BL_LEVEL;
- } else {
- panel->bl_config.bl_max_level = val;
- }
-
- rc = of_property_read_u32(of_node, "qcom,mdss-brightness-max-level",
- &val);
- if (rc) {
- pr_debug("[%s] brigheness-max-level unspecified, defaulting to 255\n",
- panel->name);
- panel->bl_config.brightness_max_level = 255;
- } else {
- panel->bl_config.brightness_max_level = val;
- }
-
- if (panel->bl_config.type == DSI_BACKLIGHT_PWM) {
- rc = dsi_panel_parse_bl_pwm_config(&panel->bl_config, of_node);
- if (rc) {
- pr_err("[%s] failed to parse pwm config, rc=%d\n",
- panel->name, rc);
- goto error;
- }
- }
-
- panel->bl_config.en_gpio = of_get_named_gpio(of_node,
- "qcom,platform-bklight-en-gpio",
- 0);
- if (!gpio_is_valid(panel->bl_config.en_gpio)) {
- pr_debug("[%s] failed get bklt gpio, rc=%d\n", panel->name, rc);
- rc = 0;
- goto error;
- }
-
-error:
- return rc;
-}
-
void dsi_dsc_pclk_param_calc(struct msm_display_dsc_info *dsc, int intf_width)
{
int slice_per_pkt, slice_per_intf;
@@ -2689,7 +2672,7 @@ int dsi_panel_parse_esd_reg_read_configs(struct dsi_panel *panel,
if (!esd_config)
return -EINVAL;
- dsi_panel_parse_cmd_sets_sub(&esd_config->status_cmd,
+ dsi_panel_parse_cmd_sets_dt(&esd_config->status_cmd,
DSI_CMD_SET_PANEL_STATUS, of_node);
if (!esd_config->status_cmd.count) {
pr_err("panel status command parsing failed\n");
@@ -2791,6 +2774,57 @@ int dsi_panel_parse_esd_reg_read_configs(struct dsi_panel *panel,
return rc;
}
+int dsi_panel_parse_esd_irq_gpio_configs(struct dsi_panel *panel,
+ struct device_node *of_node)
+{
+ struct drm_panel_esd_config *esd_config;
+ struct device_node *esd_of_node;
+ const char *esd_irq_mode;
+ int rc = 0;
+
+ if (!panel || !of_node) {
+ pr_err("Invalid panel or node\n");
+ return -EINVAL;
+ }
+
+ esd_of_node = of_parse_phandle(of_node,
+ "google,dsi-esd-irq", 0);
+ if (!esd_of_node) {
+ pr_err("google,dsi-esd-irq is not found\n");
+ return -EINVAL;
+ }
+
+ esd_config = &panel->esd_config;
+ esd_config->irq_gpio = of_get_named_gpio(esd_of_node,
+ "google,esd-irq-gpio",
+ 0);
+
+ rc = of_property_read_string(esd_of_node,
+ "google,esd-irq-mode",
+ &esd_irq_mode);
+ if (rc) {
+ pr_err("esd irq mode is not defined\n");
+ return -EINVAL;
+ }
+
+ if (!strcmp(esd_irq_mode, "trigger_rising")) {
+ esd_config->irq_mode = ESD_IRQ_MODE_TRIGGER_RISING;
+ } else if (!strcmp(esd_irq_mode, "trigger_falling")) {
+ esd_config->irq_mode = ESD_IRQ_MODE_TRIGGER_FALLING;
+ } else if (!strcmp(esd_irq_mode, "trigger_high")) {
+ esd_config->irq_mode = ESD_IRQ_MODE_TRIGGER_HIGH;
+ } else if (!strcmp(esd_irq_mode, "trigger_low")) {
+ esd_config->irq_mode = ESD_IRQ_MODE_TRIGGER_LOW;
+ } else {
+ pr_err("No valid string for google,esd-irq-mode\n");
+ return -EINVAL;
+ }
+
+ pr_info("esd irq mode is %s\n", esd_irq_mode);
+
+ return 0;
+}
+
static int dsi_panel_parse_esd_config(struct dsi_panel *panel,
struct device_node *of_node)
{
@@ -2822,6 +2856,8 @@ static int dsi_panel_parse_esd_config(struct dsi_panel *panel,
rc = -EINVAL;
goto error;
}
+ } else if (!strcmp(string, "irq_check")) {
+ esd_config->status_mode = ESD_MODE_IRQ_GPIO;
} else {
pr_err("No valid panel-status-check-mode string\n");
rc = -EINVAL;
@@ -2845,6 +2881,14 @@ static int dsi_panel_parse_esd_config(struct dsi_panel *panel,
esd_mode = "bta_trigger";
} else if (panel->esd_config.status_mode == ESD_MODE_PANEL_TE) {
esd_mode = "te_check";
+ } else if (panel->esd_config.status_mode == ESD_MODE_IRQ_GPIO) {
+ rc = dsi_panel_parse_esd_irq_gpio_configs(panel, of_node);
+ if (rc) {
+ pr_err("failed to parse esd irq gpio mode params, rc=%d\n",
+ rc);
+ goto error;
+ }
+ esd_mode = "irq_check";
}
pr_info("ESD enabled with mode: %s\n", esd_mode);
@@ -2874,6 +2918,11 @@ struct dsi_panel *dsi_panel_get(struct device *parent,
if (!panel->name)
panel->name = DSI_PANEL_DEFAULT_LABEL;
+ rc = dsi_panel_parse_vendor_info(panel, of_node);
+ if (rc)
+ pr_err("failed to parse vendor information, rc=%d\n",
+ rc);
+
rc = dsi_panel_parse_host_config(panel, of_node);
if (rc) {
pr_err("failed to parse host configuration, rc=%d\n",
@@ -2910,7 +2959,7 @@ struct dsi_panel *dsi_panel_get(struct device *parent,
if (rc)
pr_err("failed to parse panel gpios, rc=%d\n", rc);
- rc = dsi_panel_parse_bl_config(panel, of_node);
+ rc = dsi_panel_bl_parse_config(parent, &panel->bl_config, of_node);
if (rc)
pr_err("failed to parse backlight config, rc=%d\n", rc);
@@ -2951,6 +3000,11 @@ struct dsi_panel *dsi_panel_get(struct device *parent,
drm_panel_init(&panel->drm_panel);
mutex_init(&panel->panel_lock);
panel->parent = parent;
+
+ rc = dsi_panel_create_sn_buf(panel);
+ if (rc)
+ pr_err("failed to create buffer for SN, rc=%d\n", rc);
+
return panel;
error:
kfree(panel);
@@ -2959,6 +3013,8 @@ struct dsi_panel *dsi_panel_get(struct device *parent,
void dsi_panel_put(struct dsi_panel *panel)
{
+ dsi_panel_release_sn_buf(panel);
+
/* free resources allocated for ESD check */
if (panel->type == DSI_PANEL)
dsi_panel_esd_config_deinit(&panel->esd_config);
@@ -2966,6 +3022,337 @@ void dsi_panel_put(struct dsi_panel *panel)
kfree(panel);
}
+static ssize_t parse_byte_buf(u8 *out, size_t len, char *src)
+{
+ const char *skip = "\n ";
+ size_t i = 0;
+ int rc = 0;
+ char *s;
+
+ while (src && !rc && i < len) {
+ s = strsep(&src, skip);
+ if (*s != '\0') {
+ rc = kstrtou8(s, 16, out + i);
+ i++;
+ }
+ }
+
+ return rc ? : i;
+}
+
+ssize_t dsi_panel_debugfs_write_reg(struct file *file,
+ const char __user *user_buf,
+ size_t count, loff_t *ppos)
+{
+ struct seq_file *seq = file->private_data;
+ struct dsi_panel *panel = seq->private;
+ char *buf;
+ char *payload;
+ size_t len;
+ int rc = 0;
+
+ if (!panel || !panel->panel_initialized)
+ return -EPERM;
+
+ /* calculate length for worst case (1 digit per byte + whitespace) */
+ len = (count + 1) / 2;
+
+ buf = kmalloc(count + 1, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ payload = kmalloc(len, GFP_KERNEL);
+ if (!payload) {
+ kfree(buf);
+ return -ENOMEM;
+ }
+
+ if (copy_from_user(buf, user_buf, count)) {
+ rc = -EFAULT;
+ goto done;
+ }
+
+ buf[count] = 0; /* terminate end of string */
+
+ rc = parse_byte_buf(payload, len, buf);
+ if (rc <= 0) {
+ rc = -EINVAL;
+ goto done;
+ }
+ len = rc;
+
+ pr_debug("writing cmd=%x len=%zu\n", payload[0], len);
+
+ mutex_lock(&panel->panel_lock);
+ rc = mipi_dsi_dcs_write_buffer(&panel->mipi_device, payload, len);
+ mutex_unlock(&panel->panel_lock);
+
+done:
+ kfree(buf);
+ kfree(payload);
+
+ return rc ? : count;
+}
+
+int dsi_panel_debugfs_read_reg(struct seq_file *seq, void *data)
+{
+ struct dsi_panel *panel = seq->private;
+ char *buf;
+ ssize_t rc;
+ size_t len;
+ u8 cmd;
+
+ if (!panel || !panel->panel_initialized)
+ return -EPERM;
+
+ len = panel->debug.reg_read_len;
+ cmd = panel->debug.reg_read_cmd;
+
+ if (len == 0)
+ return -EINVAL;
+
+ buf = kmalloc(max(PAGE_SIZE, len), GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ mutex_lock(&panel->panel_lock);
+ rc = mipi_dsi_dcs_read(&panel->mipi_device, cmd, buf, len);
+ mutex_unlock(&panel->panel_lock);
+
+ if (rc > 0) {
+ int i;
+
+ pr_debug("got %zd bytes back, first: 0x%x\n", rc, buf[0]);
+ for (i = 0; i < rc; i++) {
+ if ((i % 8) > 0)
+ seq_puts(seq, " ");
+ else if (i)
+ seq_puts(seq, "\n");
+ seq_printf(seq, "%02X", buf[i]);
+ }
+ seq_puts(seq, "\n");
+ rc = 0;
+ } else if (rc == 0) {
+ pr_debug("no response back\n");
+ }
+ kfree(buf);
+
+ return rc;
+}
+
+static int dsi_panel_debugfs_open_reg(struct inode *inode, struct file *f)
+{
+ return single_open(f, dsi_panel_debugfs_read_reg, inode->i_private);
+}
+
+static const struct file_operations panel_reg_fops = {
+ .owner = THIS_MODULE,
+ .open = dsi_panel_debugfs_open_reg,
+ .write = dsi_panel_debugfs_write_reg,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+struct debugfs_cmdset_entry {
+ struct dsi_panel *panel;
+ enum dsi_cmd_set_type type;
+};
+
+struct {
+ const char *label;
+ enum dsi_cmd_set_type type;
+} cmdset_list[] = {
+ { "pre_on", DSI_CMD_SET_PRE_ON },
+ { "on", DSI_CMD_SET_ON },
+ { "post_on", DSI_CMD_SET_POST_ON },
+ { "pre_off", DSI_CMD_SET_PRE_OFF },
+ { "off", DSI_CMD_SET_OFF },
+ { "post_off", DSI_CMD_SET_POST_OFF },
+ { "panel_status", DSI_CMD_SET_PANEL_STATUS },
+ { "pps", DSI_CMD_SET_PPS },
+ { "lp1", DSI_CMD_SET_LP1 },
+ { "lp2", DSI_CMD_SET_LP2 },
+ { "no_lp", DSI_CMD_SET_NOLP },
+ { "vr", DSI_CMD_SET_VR },
+ { "novr", DSI_CMD_SET_NOVR },
+};
+
+static inline ssize_t parse_cmdset(struct dsi_panel_cmd_set *set, char *buf,
+ size_t count)
+{
+ char *tmp;
+ size_t len;
+ ssize_t rc;
+
+ /* calculate length for worst case (1 digit per byte + whitespace) */
+ len = (count + 1) / 2;
+ tmp = kmalloc(len, GFP_KERNEL);
+ if (!tmp)
+ return -ENOMEM;
+
+ rc = parse_byte_buf(tmp, len, buf);
+ if (rc <= 0) {
+ rc = -EINVAL;
+ goto done;
+ }
+
+ rc = dsi_panel_parse_cmd_sets_sub(set, tmp, rc);
+done:
+ kfree(tmp);
+
+ return rc;
+}
+
+ssize_t dsi_panel_debugfs_write_cmdset(struct file *file,
+ const char __user *user_buf,
+ size_t count, loff_t *ppos)
+{
+ struct seq_file *seq = file->private_data;
+ struct debugfs_cmdset_entry *entry = seq->private;
+ struct dsi_panel *panel = entry->panel;
+ struct dsi_panel_cmd_set tmp_set;
+ struct dsi_panel_cmd_set *set;
+ char *buf;
+ int rc = 0;
+ int i;
+
+ mutex_lock(&panel->panel_lock);
+ if (!panel->cur_mode || !panel->cur_mode->priv_info) {
+ pr_err("Invalid mode for panel [%s]\n", panel->name);
+ mutex_unlock(&panel->panel_lock);
+ return -EINVAL;
+ }
+
+ set = &panel->cur_mode->priv_info->cmd_sets[entry->type];
+ tmp_set = *set;
+
+ buf = kmalloc(count + 1, GFP_KERNEL);
+ if (!buf) {
+ rc = -ENOMEM;
+ goto done;
+ }
+
+ if (copy_from_user(buf, user_buf, count)) {
+ rc = -EFAULT;
+ goto done;
+ }
+ buf[count] = '\0';
+
+ rc = parse_cmdset(&tmp_set, buf, count);
+ if (rc)
+ goto done;
+
+ for (i = 0; i < set->count; i++) {
+ struct dsi_cmd_desc *cmd = set->cmds + i;
+
+ kfree(cmd->msg.tx_buf);
+ kfree(cmd->msg.rx_buf);
+ }
+ kfree(set->cmds);
+ *set = tmp_set;
+
+done:
+ kfree(buf);
+ mutex_unlock(&panel->panel_lock);
+
+ return rc ? : count;
+}
+
+int dsi_panel_debugfs_read_cmdset(struct seq_file *seq, void *data)
+{
+ struct debugfs_cmdset_entry *entry = seq->private;
+ struct dsi_panel *panel = entry->panel;
+ struct dsi_panel_cmd_set *set;
+ int i, j;
+
+ mutex_lock(&panel->panel_lock);
+ if (!panel->cur_mode || !panel->cur_mode->priv_info) {
+ pr_err("Invalid mode for panel [%s]\n", panel->name);
+ mutex_unlock(&panel->panel_lock);
+ return -EINVAL;
+ }
+ set = &panel->cur_mode->priv_info->cmd_sets[entry->type];
+
+ for (i = 0; i < set->count; i++) {
+ struct dsi_cmd_desc *cmd = set->cmds + i;
+ const char *txbuf = cmd->msg.tx_buf;
+
+ seq_printf(seq, "%02X %02X %02X %02X %02X %02zX %02zX",
+ cmd->msg.type, cmd->last_command, cmd->msg.channel,
+ cmd->msg.flags & MIPI_DSI_MSG_REQ_ACK ? 1 : 0,
+ cmd->post_wait_ms,
+ cmd->msg.tx_len >> 8, cmd->msg.tx_len & 0xff);
+
+ for (j = 0; j < cmd->msg.tx_len; j++)
+ seq_printf(seq, " %02X", txbuf[j]);
+ seq_puts(seq, "\n");
+ }
+ mutex_unlock(&panel->panel_lock);
+
+ return 0;
+}
+
+static int dsi_panel_debugfs_open_cmdset(struct inode *inode, struct file *f)
+{
+ return single_open(f, dsi_panel_debugfs_read_cmdset, inode->i_private);
+}
+
+static const struct file_operations panel_cmdset_fops = {
+ .owner = THIS_MODULE,
+ .open = dsi_panel_debugfs_open_cmdset,
+ .write = dsi_panel_debugfs_write_cmdset,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static void dsi_panel_debugfs_create_cmdsets(struct dentry *parent,
+ struct dsi_panel *panel)
+{
+ struct device *dev = panel->parent;
+ struct debugfs_cmdset_entry *entry;
+ const size_t cmds_size = ARRAY_SIZE(cmdset_list);
+ struct dentry *r;
+ int i;
+
+ r = debugfs_create_dir("cmd_sets", parent);
+ if (IS_ERR(r))
+ return;
+
+ entry = devm_kzalloc(dev, cmds_size * sizeof(*entry), GFP_KERNEL);
+ if (!entry)
+ return;
+
+ for (i = 0; i < cmds_size; i++, entry++) {
+ const char *name = cmdset_list[i].label;
+
+ entry->type = cmdset_list[i].type;
+ entry->panel = panel;
+
+ debugfs_create_file(name, 0600, r, entry, &panel_cmdset_fops);
+ }
+}
+
+void dsi_panel_debugfs_init(struct dsi_panel *panel, struct dentry *dir)
+{
+ struct dentry *r;
+ struct dsi_panel_debug *pdbg = &panel->debug;
+
+ r = debugfs_create_dir("panel_reg", dir);
+ if (IS_ERR(r))
+ return;
+
+ /* default read of 2 bytes */
+ pdbg->reg_read_len = 2;
+
+ debugfs_create_u8("addr", 0600, r, &pdbg->reg_read_cmd);
+ debugfs_create_size_t("len", 0600, r, &pdbg->reg_read_len);
+ debugfs_create_file("payload", 0600, r, panel, &panel_reg_fops);
+
+ dsi_panel_debugfs_create_cmdsets(dir, panel);
+}
+
int dsi_panel_drv_init(struct dsi_panel *panel,
struct mipi_dsi_host *host)
{
@@ -3154,11 +3541,12 @@ void dsi_panel_put_mode(struct dsi_display_mode *mode)
if (!mode->priv_info)
return;
-
for (i = 0; i < DSI_CMD_SET_MAX; i++)
dsi_panel_destroy_cmd_packets(&mode->priv_info->cmd_sets[i]);
+ kfree(mode->priv_info->phy_timing_val);
kfree(mode->priv_info);
+ mode->priv_info = NULL;
}
int dsi_panel_get_mode(struct dsi_panel *panel,
@@ -3227,7 +3615,7 @@ int dsi_panel_get_mode(struct dsi_panel *panel,
goto parse_fail;
}
- rc = dsi_panel_parse_cmd_sets(prv_info, child_np);
+ rc = dsi_panel_parse_cmd_sets(prv_info, child_np, panel);
if (rc) {
pr_err("failed to parse command sets, rc=%d\n", rc);
goto parse_fail;
@@ -3311,6 +3699,12 @@ int dsi_panel_pre_prepare(struct dsi_panel *panel)
mutex_lock(&panel->panel_lock);
+ rc = dsi_pwr_enable_regulator(&panel->power_info, true);
+ if (rc) {
+ pr_err("[%s] failed to enable vregs, rc=%d\n", panel->name, rc);
+ goto error;
+ }
+
/* If LP11_INIT is set, panel will be powered up during prepare() */
if (panel->lp11_init)
goto error;
@@ -3355,12 +3749,11 @@ int dsi_panel_update_pps(struct dsi_panel *panel)
}
rc = dsi_panel_tx_cmd_set(panel, DSI_CMD_SET_PPS);
- if (rc) {
+ if (rc)
pr_err("[%s] failed to send DSI_CMD_SET_PPS cmds, rc=%d\n",
panel->name, rc);
- goto error;
- }
+ dsi_panel_destroy_cmds_packets_buf(set);
error:
mutex_unlock(&panel->panel_lock);
return rc;
@@ -3378,12 +3771,39 @@ int dsi_panel_set_lp1(struct dsi_panel *panel)
if (panel->type == EXT_BRIDGE)
return 0;
+ dsi_backlight_early_dpms(&panel->bl_config, SDE_MODE_DPMS_LP1);
+
mutex_lock(&panel->panel_lock);
+
+ if (panel->vr_mode) {
+ rc = dsi_panel_clear_vr_locked(panel);
+ if (rc) {
+ pr_err("[%s] couldn't disable VR mode for LP1 transition\n",
+ panel->name);
+ mutex_unlock(&panel->panel_lock);
+ return rc;
+ }
+ }
+
+ rc = dsi_panel_update_hbm_locked(panel, false);
+ if (rc) {
+ pr_err("[%s] couldn't disable HBM mode for LP1 transition\n",
+ panel->name);
+ mutex_unlock(&panel->panel_lock);
+ return rc;
+ }
+ dsi_backlight_hbm_dimming_stop(&panel->bl_config);
+
rc = dsi_panel_tx_cmd_set(panel, DSI_CMD_SET_LP1);
if (rc)
pr_err("[%s] failed to send DSI_CMD_SET_LP1 cmd, rc=%d\n",
panel->name, rc);
mutex_unlock(&panel->panel_lock);
+
+ if (!rc)
+ rc = dsi_backlight_late_dpms(&panel->bl_config,
+ SDE_MODE_DPMS_LP1);
+
return rc;
}
@@ -3399,12 +3819,39 @@ int dsi_panel_set_lp2(struct dsi_panel *panel)
if (panel->type == EXT_BRIDGE)
return 0;
+ dsi_backlight_early_dpms(&panel->bl_config, SDE_MODE_DPMS_LP2);
+
mutex_lock(&panel->panel_lock);
+
+ if (panel->vr_mode) {
+ rc = dsi_panel_clear_vr_locked(panel);
+ if (rc) {
+ pr_err("[%s] couldn't disable VR mode for LP2 transition\n",
+ panel->name);
+ mutex_unlock(&panel->panel_lock);
+ return rc;
+ }
+ }
+
+ rc = dsi_panel_update_hbm_locked(panel, false);
+ if (rc) {
+ pr_err("[%s] couldn't disable HBM mode for LP2 transition\n",
+ panel->name);
+ mutex_unlock(&panel->panel_lock);
+ return rc;
+ }
+ dsi_backlight_hbm_dimming_stop(&panel->bl_config);
+
rc = dsi_panel_tx_cmd_set(panel, DSI_CMD_SET_LP2);
if (rc)
pr_err("[%s] failed to send DSI_CMD_SET_LP2 cmd, rc=%d\n",
panel->name, rc);
mutex_unlock(&panel->panel_lock);
+
+ if (!rc)
+ rc = dsi_backlight_late_dpms(&panel->bl_config,
+ SDE_MODE_DPMS_LP2);
+
return rc;
}
@@ -3420,15 +3867,192 @@ int dsi_panel_set_nolp(struct dsi_panel *panel)
if (panel->type == EXT_BRIDGE)
return 0;
+ dsi_backlight_early_dpms(&panel->bl_config, SDE_MODE_DPMS_ON);
+
mutex_lock(&panel->panel_lock);
rc = dsi_panel_tx_cmd_set(panel, DSI_CMD_SET_NOLP);
if (rc)
pr_err("[%s] failed to send DSI_CMD_SET_NOLP cmd, rc=%d\n",
panel->name, rc);
mutex_unlock(&panel->panel_lock);
+
+ if (!rc)
+ rc = dsi_backlight_late_dpms(&panel->bl_config,
+ SDE_MODE_DPMS_ON);
return rc;
}
+static int dsi_panel_set_vr_locked(struct dsi_panel *panel)
+{
+ int rc = 0;
+
+ if (dsi_backlight_get_dpms(&panel->bl_config) != SDE_MODE_DPMS_ON) {
+ pr_err("[%s] DSI_CMD_SET_VR wasn't allowed\n", panel->name);
+ return -EINVAL;
+ }
+
+ rc = dsi_panel_update_hbm_locked(panel, false);
+ if (rc) {
+ pr_err("[%s] couldn't disable HBM mode before VR entry, aborting VR entry\n",
+ panel->name);
+ return -EINVAL;
+ }
+ dsi_backlight_hbm_dimming_stop(&panel->bl_config);
+
+ rc = dsi_panel_tx_cmd_set(panel, DSI_CMD_SET_VR);
+
+ if (rc)
+ pr_err("[%s] failed to send DSI_CMD_SET_VR cmd, rc=%d\n",
+ panel->name, rc);
+ else
+ panel->vr_mode = true;
+
+ return rc;
+}
+
+static int dsi_panel_clear_vr_locked(struct dsi_panel *panel)
+{
+ int rc = 0;
+
+ if (!panel->vr_mode) {
+ pr_err("[%s] tried to exit, but not in VR mode\n", panel->name);
+ return -EINVAL;
+ }
+
+ rc = dsi_panel_tx_cmd_set(panel, DSI_CMD_SET_NOVR);
+
+ if (rc)
+ pr_err("[%s] failed to send DSI_CMD_SET_NOVR cmd, rc=%d\n",
+ panel->name, rc);
+ else
+ panel->vr_mode = false;
+
+ return rc;
+}
+
+int dsi_panel_set_vr(struct dsi_panel *panel)
+{
+ int rc = 0;
+
+ if (!panel) {
+ pr_err("invalid params\n");
+ return -EINVAL;
+ }
+
+ if (panel->type == EXT_BRIDGE)
+ return 0;
+
+ mutex_lock(&panel->panel_lock);
+ rc = dsi_panel_set_vr_locked(panel);
+ mutex_unlock(&panel->panel_lock);
+
+ return rc;
+}
+
+int dsi_panel_clear_vr(struct dsi_panel *panel)
+{
+ int rc = 0;
+
+ if (!panel) {
+ pr_err("invalid params\n");
+ return -EINVAL;
+ }
+
+ if (panel->type == EXT_BRIDGE)
+ return 0;
+
+ mutex_lock(&panel->panel_lock);
+ rc = dsi_panel_clear_vr_locked(panel);
+ mutex_unlock(&panel->panel_lock);
+
+ return rc;
+}
+
+int dsi_panel_update_vr_mode(struct dsi_panel *panel, bool enable)
+{
+ if (enable)
+ return dsi_panel_set_vr(panel);
+ else
+ return dsi_panel_clear_vr(panel);
+}
+
+bool dsi_panel_get_vr_mode(struct dsi_panel *panel)
+{
+ bool vr_mode;
+
+ if (!panel) {
+ pr_err("invalid params\n");
+ return false;
+ }
+
+ return vr_mode = panel->vr_mode;
+}
+
+static int dsi_panel_update_hbm_locked(struct dsi_panel *panel, bool enable)
+{
+ struct dsi_backlight_config *bl = &panel->bl_config;
+ struct hbm_data *hbm = bl->hbm;
+
+ if (!hbm || (panel->hbm_mode == enable))
+ return 0;
+
+ if ((dsi_backlight_get_dpms(bl) != SDE_MODE_DPMS_ON) ||
+ panel->vr_mode) {
+ pr_err("[%s] Backlight in incompatible state, HBM changes not allowed\n",
+ panel->name);
+ return -EINVAL;
+ }
+
+ /* When HBM exit is requested, send HBM exit commands
+ * immediately to avoid conflict with subsequent backlight ops.
+ */
+ if (!enable) {
+ int rc = dsi_panel_cmd_set_transfer(panel, &hbm->exit_cmd);
+
+ if (rc)
+ pr_err("[%s] failed to send HBM exit cmd, rc=%d\n",
+ panel->name, rc);
+
+ dsi_backlight_hbm_dimming_start(bl,
+ hbm->exit_num_dimming_frames,
+ &hbm->exit_dimming_stop_cmd);
+ }
+
+ panel->hbm_mode = enable;
+ hbm->cur_range = HBM_RANGE_MAX;
+
+ return 0;
+}
+
+int dsi_panel_update_hbm(struct dsi_panel *panel, bool enable)
+{
+ int rc = 0;
+
+ if (!panel)
+ return -EINVAL;
+
+ if ((panel->type == EXT_BRIDGE) || !panel->bl_config.hbm)
+ return 0;
+
+ mutex_lock(&panel->panel_lock);
+ rc = dsi_panel_update_hbm_locked(panel, enable);
+ mutex_unlock(&panel->panel_lock);
+ if (rc)
+ return rc;
+
+ return backlight_update_status(panel->bl_config.bl_device);
+}
+
+bool dsi_panel_get_hbm(struct dsi_panel *panel)
+{
+ if (!panel) {
+ pr_err("invalid params\n");
+ return false;
+ }
+
+ return panel->hbm_mode;
+}
+
int dsi_panel_prepare(struct dsi_panel *panel)
{
int rc = 0;
@@ -3441,8 +4065,13 @@ int dsi_panel_prepare(struct dsi_panel *panel)
if (panel->type == EXT_BRIDGE)
return 0;
+ dsi_backlight_early_dpms(&panel->bl_config, SDE_MODE_DPMS_ON);
+
mutex_lock(&panel->panel_lock);
+ if (panel->init_delay_us)
+ usleep_range(panel->init_delay_us, panel->init_delay_us);
+
if (panel->lp11_init) {
rc = dsi_panel_power_on(panel);
if (rc) {
@@ -3649,6 +4278,11 @@ int dsi_panel_enable(struct dsi_panel *panel)
}
panel->panel_initialized = true;
mutex_unlock(&panel->panel_lock);
+
+ if (!rc)
+ rc = dsi_backlight_late_dpms(&panel->bl_config,
+ SDE_MODE_DPMS_ON);
+
return rc;
}
@@ -3689,17 +4323,17 @@ int dsi_panel_pre_disable(struct dsi_panel *panel)
if (panel->type == EXT_BRIDGE)
return 0;
+ dsi_backlight_early_dpms(&panel->bl_config, SDE_MODE_DPMS_OFF);
+
mutex_lock(&panel->panel_lock);
rc = dsi_panel_tx_cmd_set(panel, DSI_CMD_SET_PRE_OFF);
- if (rc) {
+ if (rc)
pr_err("[%s] failed to send DSI_CMD_SET_PRE_OFF cmds, rc=%d\n",
panel->name, rc);
- goto error;
- }
-error:
mutex_unlock(&panel->panel_lock);
+
return rc;
}
@@ -3717,6 +4351,20 @@ int dsi_panel_disable(struct dsi_panel *panel)
mutex_lock(&panel->panel_lock);
+ if (panel->vr_mode) {
+ rc = dsi_panel_clear_vr_locked(panel);
+ if (rc) {
+ pr_warn("[%s] couldn't disable VR mode to unprepare display\n",
+ panel->name);
+ }
+ }
+
+ rc = dsi_panel_update_hbm_locked(panel, false);
+ if (rc)
+ pr_warn("[%s] couldn't disable HBM mode to unprepare display\n",
+ panel->name);
+ dsi_backlight_hbm_dimming_stop(&panel->bl_config);
+
rc = dsi_panel_tx_cmd_set(panel, DSI_CMD_SET_OFF);
if (rc) {
pr_err("[%s] failed to send DSI_CMD_SET_OFF cmds, rc=%d\n",
@@ -3727,6 +4375,11 @@ int dsi_panel_disable(struct dsi_panel *panel)
error:
mutex_unlock(&panel->panel_lock);
+
+ if (!rc)
+ rc = dsi_backlight_late_dpms(&panel->bl_config,
+ SDE_MODE_DPMS_OFF);
+
return rc;
}
diff --git a/drivers/gpu/drm/msm/dsi-staging/dsi_panel.h b/drivers/gpu/drm/msm/dsi-staging/dsi_panel.h
index f8b65ab..d3f66e4f 100644
--- a/drivers/gpu/drm/msm/dsi-staging/dsi_panel.h
+++ b/drivers/gpu/drm/msm/dsi-staging/dsi_panel.h
@@ -35,6 +35,7 @@
#define DSI_CMD_PPS_SIZE 135
#define DSI_MODE_MAX 5
+#define HBM_RANGE_MAX 4
enum dsi_panel_rotation {
DSI_PANEL_ROTATE_NONE = 0,
@@ -83,26 +84,92 @@ struct dsi_panel_phy_props {
enum dsi_panel_rotation rotation;
};
+struct hbm_range {
+ /* Userspace brightness range (inclusive) for this HBM range */
+ u32 user_bri_start;
+ u32 user_bri_end;
+
+ /* Panel brightness range (inclusive) for this HBM range */
+ u32 panel_bri_start;
+ u32 panel_bri_end;
+
+ /* Command to be sent to the panel when entering this HBM range */
+ struct dsi_panel_cmd_set entry_cmd;
+ /*
+ * Command to be sent to the panel to stop brightness dimming while
+ * in this HBM range.
+ */
+ struct dsi_panel_cmd_set dimming_stop_cmd;
+ /* Number of frames dimming will take. */
+ u32 num_dimming_frames;
+};
+
+struct hbm_data {
+ /* Command to be sent to the panel when exiting HBM */
+ struct dsi_panel_cmd_set exit_cmd;
+ /* Command to be sent to the panel to stop brightness dimming */
+ struct dsi_panel_cmd_set exit_dimming_stop_cmd;
+ /* Number of frames dimming will take */
+ u32 exit_num_dimming_frames;
+
+ struct hbm_range ranges[HBM_RANGE_MAX];
+ u32 num_ranges;
+ u32 cur_range;
+
+ /* Brightness dimming currently active */
+ bool dimming_active;
+ /* Total number of frames brightness dimming takes */
+ u32 dimming_frames_total;
+ /* Number of frames remaining until brightness settles */
+ u32 dimming_frames_left;
+ /* DSI command to send once brightness dimming settles */
+ struct dsi_panel_cmd_set *dimming_stop_cmd;
+
+ /* Work queue used to count frames during dimming */
+ struct workqueue_struct *dimming_workq;
+ struct work_struct dimming_work;
+ struct dsi_panel *panel;
+};
+
struct dsi_backlight_config {
enum dsi_backlight_type type;
u32 bl_min_level;
u32 bl_max_level;
u32 brightness_max_level;
- u32 bl_level;
u32 bl_scale;
u32 bl_scale_ad;
+ u32 bl_actual;
+ u16 *lut;
+ unsigned int last_state;
+
+ /* Minimum safe brightness level during VR mode */
+ u32 bl_vr_min_safe_level;
+
+ struct hbm_data *hbm;
int en_gpio;
- /* PWM params */
- bool pwm_pmi_control;
- u32 pwm_pmic_bank;
- u32 pwm_period_usecs;
- int pwm_gpio;
- /* WLED params */
- struct led_trigger *wled;
- struct backlight_device *bd;
+ struct backlight_device *bl_device;
+ struct regulator *lab_vreg;
+
+ void *priv;
+
+ /**
+ * update_bl - function used to update backlight. Called with panel_lock
+ * locked.
+ * @bl_cfg - ptr to backlight config struct
+ * @bl_lvl - backlight level set
+ *
+ * return: non-zero on success otherwise errno
+ */
+ int (*update_bl)(struct dsi_backlight_config *bl_cfg, u32 bl_lvl);
+
+ /**
+ * unregister - unregisters and frees any backlight data
+ * @bl_cfg - ptr to backlight config struct
+ */
+ void (*unregister)(struct dsi_backlight_config *bl_cfg);
};
struct dsi_reset_seq {
@@ -120,13 +187,27 @@ struct dsi_panel_reset_config {
u32 mode_sel_state;
};
+struct dsi_panel_debug {
+ u8 reg_read_cmd;
+ size_t reg_read_len;
+};
+
enum esd_check_status_mode {
ESD_MODE_REG_READ,
ESD_MODE_SW_BTA,
ESD_MODE_PANEL_TE,
+ ESD_MODE_IRQ_GPIO,
ESD_MODE_MAX
};
+enum esd_irq_mode {
+ ESD_IRQ_MODE_TRIGGER_RISING = 1,
+ ESD_IRQ_MODE_TRIGGER_FALLING = 2,
+ ESD_IRQ_MODE_TRIGGER_HIGH = 4,
+ ESD_IRQ_MODE_TRIGGER_LOW = 8,
+ ESD_IRQ_MODE_MAX
+};
+
struct drm_panel_esd_config {
bool esd_enabled;
@@ -138,6 +219,11 @@ struct drm_panel_esd_config {
u8 *return_buf;
u8 *status_buf;
u32 groups;
+
+ enum esd_irq_mode irq_mode;
+ int irq_gpio;
+ bool irq_enabled;
+ bool irq_mode_prepared;
};
enum dsi_panel_type {
@@ -146,6 +232,19 @@ enum dsi_panel_type {
DSI_PANEL_TYPE_MAX,
};
+struct dsi_panel_sn_location {
+ u32 start_byte;
+ u32 sn_length;
+ u8 addr;
+};
+
+struct dsi_panel_vendor_info {
+ struct dsi_panel_sn_location location;
+ bool is_sn;
+ u8 *sn;
+ const char *name;
+};
+
struct dsi_panel {
const char *name;
enum dsi_panel_type type;
@@ -173,8 +272,12 @@ struct dsi_panel {
struct dsi_panel_reset_config reset_config;
struct dsi_pinctrl_info pinctrl;
struct drm_panel_hdr_properties hdr_props;
+ struct dsi_panel_debug debug;
struct drm_panel_esd_config esd_config;
+ struct dsi_panel_vendor_info vendor_info;
+ u32 init_delay_us;
+ bool hs_pps;
bool lp11_init;
bool ulps_enabled;
bool ulps_suspend_enabled;
@@ -187,6 +290,10 @@ struct dsi_panel {
enum dsi_dms_mode dms_mode;
bool sync_broadcast_en;
+
+ /* the following set of members are guarded by panel_lock */
+ bool vr_mode;
+ bool hbm_mode;
};
static inline bool dsi_panel_ulps_feature_enabled(struct dsi_panel *panel)
@@ -222,6 +329,8 @@ int dsi_panel_drv_init(struct dsi_panel *panel, struct mipi_dsi_host *host);
int dsi_panel_drv_deinit(struct dsi_panel *panel);
+void dsi_panel_debugfs_init(struct dsi_panel *panel, struct dentry *dir);
+
int dsi_panel_get_mode_count(struct dsi_panel *panel,
struct device_node *of_node);
@@ -252,6 +361,10 @@ int dsi_panel_set_lp2(struct dsi_panel *panel);
int dsi_panel_set_nolp(struct dsi_panel *panel);
+int dsi_panel_set_vr(struct dsi_panel *panel);
+
+int dsi_panel_set_novr(struct dsi_panel *panel);
+
int dsi_panel_prepare(struct dsi_panel *panel);
int dsi_panel_enable(struct dsi_panel *panel);
@@ -266,8 +379,6 @@ int dsi_panel_unprepare(struct dsi_panel *panel);
int dsi_panel_post_unprepare(struct dsi_panel *panel);
-int dsi_panel_set_backlight(struct dsi_panel *panel, u32 bl_lvl);
-
int dsi_panel_update_pps(struct dsi_panel *panel);
int dsi_panel_send_roi_dcs(struct dsi_panel *panel, int ctrl_idx,
@@ -285,7 +396,40 @@ struct dsi_panel *dsi_panel_ext_bridge_get(struct device *parent,
int dsi_panel_parse_esd_reg_read_configs(struct dsi_panel *panel,
struct device_node *of_node);
+int dsi_panel_parse_esd_irq_gpio_configs(struct dsi_panel *panel,
+ struct device_node *of_node);
void dsi_panel_ext_bridge_put(struct dsi_panel *panel);
+int dsi_panel_cmd_set_transfer(struct dsi_panel *panel,
+ struct dsi_panel_cmd_set *cmd);
+int dsi_panel_parse_dt_cmd_set(struct device_node *of_node,
+ const char *cmd_str,
+ const char *cmd_state_str,
+ struct dsi_panel_cmd_set *cmd);
+void dsi_panel_destroy_cmd_packets(struct dsi_panel_cmd_set *set);
+
+int dsi_backlight_early_dpms(struct dsi_backlight_config *bl, int power_state);
+int dsi_backlight_late_dpms(struct dsi_backlight_config *bl, int power_state);
+
+int dsi_backlight_get_dpms(struct dsi_backlight_config *bl);
+
+int dsi_backlight_hbm_dimming_start(struct dsi_backlight_config *bl,
+ u32 num_frames, struct dsi_panel_cmd_set *stop_cmd);
+void dsi_backlight_hbm_dimming_stop(struct dsi_backlight_config *bl);
+
+int dsi_panel_bl_register(struct dsi_panel *panel);
+int dsi_panel_bl_unregister(struct dsi_panel *panel);
+int dsi_panel_bl_parse_config(struct device *parent,
+ struct dsi_backlight_config *bl, struct device_node *of_node);
+
+int dsi_panel_update_vr_mode(struct dsi_panel *panel, bool enable);
+bool dsi_panel_get_vr_mode(struct dsi_panel *panel);
+
+int dsi_panel_get_sn(struct dsi_panel *panel);
+
+/* Set/get high brightness mode */
+int dsi_panel_update_hbm(struct dsi_panel *panel, bool enable);
+bool dsi_panel_get_hbm(struct dsi_panel *panel);
+
#endif /* _DSI_PANEL_H_ */
diff --git a/drivers/gpu/drm/msm/msm_atomic.c b/drivers/gpu/drm/msm/msm_atomic.c
index 8ffe044..45348f8 100644
--- a/drivers/gpu/drm/msm/msm_atomic.c
+++ b/drivers/gpu/drm/msm/msm_atomic.c
@@ -50,6 +50,7 @@ int msm_drm_register_client(struct notifier_block *nb)
return blocking_notifier_chain_register(&msm_drm_notifier_list,
nb);
}
+EXPORT_SYMBOL_GPL(msm_drm_register_client);
/**
* msm_drm_unregister_client - unregister a client notifier
@@ -63,6 +64,7 @@ int msm_drm_unregister_client(struct notifier_block *nb)
return blocking_notifier_chain_unregister(&msm_drm_notifier_list,
nb);
}
+EXPORT_SYMBOL_GPL(msm_drm_unregister_client);
/**
* msm_drm_notifier_call_chain - notify clients of drm_events
@@ -70,7 +72,7 @@ int msm_drm_unregister_client(struct notifier_block *nb)
* @v: notifier data, inculde display id and display blank
* event(unblank or power down).
*/
-static int msm_drm_notifier_call_chain(unsigned long val, void *v)
+int msm_drm_notifier_call_chain(unsigned long val, void *v)
{
return blocking_notifier_call_chain(&msm_drm_notifier_list, val,
v);
@@ -192,7 +194,12 @@ static void msm_atomic_wait_for_commit_done(
if (old_state->legacy_cursor_update)
continue;
+ if (drm_crtc_vblank_get(crtc))
+ continue;
+
kms->funcs->wait_for_crtc_commit_done(kms, crtc);
+
+ drm_crtc_vblank_put(crtc);
}
}
@@ -203,8 +210,7 @@ msm_disable_outputs(struct drm_device *dev, struct drm_atomic_state *old_state)
struct drm_connector_state *old_conn_state;
struct drm_crtc *crtc;
struct drm_crtc_state *old_crtc_state;
- struct msm_drm_notifier notifier_data;
- int i, blank;
+ int i;
SDE_ATRACE_BEGIN("msm_disable");
for_each_connector_in_state(old_state, connector, old_conn_state, i) {
@@ -244,11 +250,6 @@ msm_disable_outputs(struct drm_device *dev, struct drm_atomic_state *old_state)
DRM_DEBUG_ATOMIC("disabling [ENCODER:%d:%s]\n",
encoder->base.id, encoder->name);
- blank = MSM_DRM_BLANK_POWERDOWN;
- notifier_data.data = ␣
- notifier_data.id = crtc_idx;
- msm_drm_notifier_call_chain(MSM_DRM_EARLY_EVENT_BLANK,
- ¬ifier_data);
/*
* Each encoder has at most one connector (since we always steal
* it away), so we won't call disable hooks twice.
@@ -264,8 +265,6 @@ msm_disable_outputs(struct drm_device *dev, struct drm_atomic_state *old_state)
funcs->dpms(encoder, DRM_MODE_DPMS_OFF);
drm_bridge_post_disable(encoder->bridge);
- msm_drm_notifier_call_chain(MSM_DRM_EVENT_BLANK,
- ¬ifier_data);
}
for_each_crtc_in_state(old_state, crtc, old_crtc_state, i) {
@@ -405,11 +404,10 @@ static void msm_atomic_helper_commit_modeset_enables(struct drm_device *dev,
struct drm_crtc_state *old_crtc_state;
struct drm_connector *connector;
struct drm_connector_state *old_conn_state;
- struct msm_drm_notifier notifier_data;
struct msm_drm_private *priv = dev->dev_private;
struct msm_kms *kms = priv->kms;
int bridge_enable_count = 0;
- int i, blank;
+ int i;
SDE_ATRACE_BEGIN("msm_enable");
for_each_crtc_in_state(old_state, crtc, old_crtc_state, i) {
@@ -462,12 +460,6 @@ static void msm_atomic_helper_commit_modeset_enables(struct drm_device *dev,
DRM_DEBUG_ATOMIC("enabling [ENCODER:%d:%s]\n",
encoder->base.id, encoder->name);
- blank = MSM_DRM_BLANK_UNBLANK;
- notifier_data.data = ␣
- notifier_data.id =
- connector->state->crtc->index;
- msm_drm_notifier_call_chain(MSM_DRM_EARLY_EVENT_BLANK,
- ¬ifier_data);
/*
* Each encoder has at most one connector (since we always steal
* it away), so we won't call enable hooks twice.
@@ -512,8 +504,6 @@ static void msm_atomic_helper_commit_modeset_enables(struct drm_device *dev,
encoder->base.id, encoder->name);
drm_bridge_enable(encoder->bridge);
- msm_drm_notifier_call_chain(MSM_DRM_EVENT_BLANK,
- ¬ifier_data);
}
SDE_ATRACE_END("msm_enable");
}
diff --git a/drivers/gpu/drm/msm/msm_drv.c b/drivers/gpu/drm/msm/msm_drv.c
index 9eb62fe..5ec09762 100644
--- a/drivers/gpu/drm/msm/msm_drv.c
+++ b/drivers/gpu/drm/msm/msm_drv.c
@@ -39,6 +39,7 @@
#include <linux/of_address.h>
#include <linux/kthread.h>
+#include <linux/workqueue.h>
#include "msm_drv.h"
#include "msm_debugfs.h"
#include "msm_fence.h"
@@ -484,6 +485,156 @@ static int msm_power_enable_wrapper(void *handle, void *client, bool enable)
return sde_power_resource_enable(handle, client, enable);
}
+static ssize_t idle_encoder_mask_store(struct device *device,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct drm_device *ddev = dev_get_drvdata(device);
+ struct msm_drm_private *priv = ddev->dev_private;
+ struct msm_idle *idle = &priv->idle;
+ u32 encoder_mask = 0;
+ int rc;
+ unsigned long flags;
+
+ rc = kstrtouint(buf, 10, &encoder_mask);
+ if (rc)
+ return rc;
+
+ spin_lock_irqsave(&idle->lock, flags);
+ idle->encoder_mask = encoder_mask;
+ idle->active_mask &= encoder_mask;
+ spin_unlock_irqrestore(&idle->lock, flags);
+
+ return count;
+}
+
+static ssize_t idle_encoder_mask_show(struct device *device,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct drm_device *ddev = dev_get_drvdata(device);
+ struct msm_drm_private *priv = ddev->dev_private;
+ struct msm_idle *idle = &priv->idle;
+
+ return snprintf(buf, PAGE_SIZE, "0x%x\n", idle->encoder_mask);
+}
+
+static ssize_t idle_timeout_ms_store(struct device *device,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct drm_device *ddev = dev_get_drvdata(device);
+ struct msm_drm_private *priv = ddev->dev_private;
+ struct msm_idle *idle = &priv->idle;
+ u32 timeout_ms = 0;
+ int rc;
+ unsigned long flags;
+
+ rc = kstrtouint(buf, 10, &timeout_ms);
+ if (rc)
+ return rc;
+
+ spin_lock_irqsave(&idle->lock, flags);
+ idle->timeout_ms = timeout_ms;
+ spin_unlock_irqrestore(&idle->lock, flags);
+
+ return count;
+}
+
+static ssize_t idle_timeout_ms_show(struct device *device,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct drm_device *ddev = dev_get_drvdata(device);
+ struct msm_drm_private *priv = ddev->dev_private;
+ struct msm_idle *idle = &priv->idle;
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n", idle->timeout_ms);
+}
+
+static ssize_t idle_state_show(struct device *device,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct drm_device *ddev = dev_get_drvdata(device);
+ struct msm_drm_private *priv = ddev->dev_private;
+ struct msm_idle *idle = &priv->idle;
+ const char *state;
+ unsigned long flags;
+
+ spin_lock_irqsave(&idle->lock, flags);
+ if (idle->active_mask) {
+ state = "active";
+ spin_unlock_irqrestore(&idle->lock, flags);
+ return scnprintf(buf, PAGE_SIZE, "%s (0x%x)\n",
+ state, idle->active_mask);
+ } else if (delayed_work_pending(&idle->work))
+ state = "pending";
+ else
+ state = "idle";
+ spin_unlock_irqrestore(&idle->lock, flags);
+
+ return scnprintf(buf, PAGE_SIZE, "%s\n", state);
+}
+
+static DEVICE_ATTR_RW(idle_encoder_mask);
+static DEVICE_ATTR_RW(idle_timeout_ms);
+static DEVICE_ATTR_RO(idle_state);
+
+static const struct attribute *msm_idle_attrs[] = {
+ &dev_attr_idle_encoder_mask.attr,
+ &dev_attr_idle_timeout_ms.attr,
+ &dev_attr_idle_state.attr,
+ NULL
+};
+
+static void msm_idle_work(struct work_struct *work)
+{
+ struct delayed_work *dw = to_delayed_work(work);
+ struct msm_idle *idle = container_of(dw, struct msm_idle, work);
+ struct msm_drm_private *priv = container_of(idle,
+ struct msm_drm_private, idle);
+
+ if (!idle->active_mask)
+ sysfs_notify(&priv->dev->dev->kobj, NULL, "idle_state");
+}
+
+void msm_idle_set_state(struct drm_encoder *encoder, bool active)
+{
+ struct drm_device *ddev = encoder->dev;
+ struct msm_drm_private *priv = ddev->dev_private;
+ struct msm_idle *idle = &priv->idle;
+ unsigned int mask = 1 << drm_encoder_index(encoder);
+ unsigned long flags;
+
+ spin_lock_irqsave(&idle->lock, flags);
+ if (mask & idle->encoder_mask) {
+ if (active)
+ idle->active_mask |= mask;
+ else
+ idle->active_mask &= ~mask;
+
+ if (idle->timeout_ms && !idle->active_mask)
+ mod_delayed_work(system_wq, &idle->work,
+ msecs_to_jiffies(idle->timeout_ms));
+ else
+ cancel_delayed_work(&idle->work);
+ }
+ spin_unlock_irqrestore(&idle->lock, flags);
+}
+
+static void msm_idle_init(struct drm_device *ddev)
+{
+ struct msm_drm_private *priv = ddev->dev_private;
+ struct msm_idle *idle = &priv->idle;
+
+ if (sysfs_create_files(&ddev->dev->kobj, msm_idle_attrs) < 0)
+ pr_warn("failed to create idle state file");
+
+ INIT_DELAYED_WORK(&idle->work, msm_idle_work);
+ spin_lock_init(&idle->lock);
+}
+
static int msm_drm_init(struct device *dev, struct drm_driver *drv)
{
struct platform_device *pdev = to_platform_device(dev);
@@ -548,6 +699,8 @@ static int msm_drm_init(struct device *dev, struct drm_driver *drv)
goto dbg_init_fail;
}
+ msm_idle_init(ddev);
+
/* Bind all our sub-components: */
ret = msm_component_bind_all(dev, ddev);
if (ret)
@@ -1994,6 +2147,7 @@ static struct platform_driver msm_platform_driver = {
.name = "msm_drm",
.of_match_table = dt_match,
.pm = &msm_pm_ops,
+ .probe_type = PROBE_PREFER_ASYNCHRONOUS,
},
};
diff --git a/drivers/gpu/drm/msm/msm_drv.h b/drivers/gpu/drm/msm/msm_drv.h
index 978aba2..2071206 100644
--- a/drivers/gpu/drm/msm/msm_drv.h
+++ b/drivers/gpu/drm/msm/msm_drv.h
@@ -518,6 +518,15 @@ struct msm_drm_thread {
struct kthread_worker worker;
};
+struct msm_idle {
+ u32 timeout_ms;
+ u32 encoder_mask;
+ u32 active_mask;
+
+ spinlock_t lock;
+ struct delayed_work work;
+};
+
struct msm_drm_private {
struct drm_device *dev;
@@ -634,6 +643,8 @@ struct msm_drm_private {
/* update the flag when msm driver receives shutdown notification */
bool shutdown_in_progress;
+
+ struct msm_idle idle;
};
/* get struct msm_kms * from drm_device * */
@@ -660,6 +671,7 @@ void __msm_fence_worker(struct work_struct *work);
int msm_atomic_commit(struct drm_device *dev,
struct drm_atomic_state *state, bool nonblock);
+int msm_drm_notifier_call_chain(unsigned long val, void *v);
void msm_gem_submit_free(struct msm_gem_submit *submit);
void msm_gem_unmap_vma(struct msm_gem_address_space *aspace,
@@ -873,6 +885,7 @@ static inline int msm_dsi_modeset_init(struct msm_dsi *msm_dsi,
void __init msm_mdp_register(void);
void __exit msm_mdp_unregister(void);
+void msm_idle_set_state(struct drm_encoder *encoder, bool active);
#ifdef CONFIG_DEBUG_FS
void msm_gem_describe(struct drm_gem_object *obj, struct seq_file *m);
void msm_gem_describe_objects(struct list_head *list, struct seq_file *m);
diff --git a/drivers/gpu/drm/msm/msm_gem.c b/drivers/gpu/drm/msm/msm_gem.c
index ddd4607..6c01111 100644
--- a/drivers/gpu/drm/msm/msm_gem.c
+++ b/drivers/gpu/drm/msm/msm_gem.c
@@ -124,10 +124,19 @@ static void put_pages(struct drm_gem_object *obj)
struct msm_gem_object *msm_obj = to_msm_bo(obj);
if (msm_obj->pages) {
- if (msm_obj->sgt)
- sg_free_table(msm_obj->sgt);
+ if (msm_obj->sgt) {
+ /* For non-cached buffers, ensure the new
+ * pages are clean because display controller,
+ * GPU, etc. are not coherent:
+ */
+ if (msm_obj->flags & (MSM_BO_WC|MSM_BO_UNCACHED))
+ dma_unmap_sg(obj->dev->dev, msm_obj->sgt->sgl,
+ msm_obj->sgt->nents,
+ DMA_BIDIRECTIONAL);
- kfree(msm_obj->sgt);
+ sg_free_table(msm_obj->sgt);
+ kfree(msm_obj->sgt);
+ }
if (use_pages(obj))
drm_gem_put_pages(obj, msm_obj->pages, true, false);
diff --git a/drivers/gpu/drm/msm/sde/sde_connector.c b/drivers/gpu/drm/msm/sde/sde_connector.c
index 07d1ad7..d998344 100644
--- a/drivers/gpu/drm/msm/sde/sde_connector.c
+++ b/drivers/gpu/drm/msm/sde/sde_connector.c
@@ -24,8 +24,6 @@
#include "sde_crtc.h"
#include "sde_rm.h"
-#define BL_NODE_NAME_SIZE 32
-
/* Autorefresh will occur after FRAME_CNT frames. Large values are unlikely */
#define AUTOREFRESH_MAX_FRAME_CNT 6
@@ -61,94 +59,6 @@ static const struct drm_prop_enum_list e_power_mode[] = {
{SDE_MODE_DPMS_OFF, "OFF"},
};
-static int sde_backlight_device_update_status(struct backlight_device *bd)
-{
- int brightness;
- struct dsi_display *display;
- struct sde_connector *c_conn;
- int bl_lvl;
- struct drm_event event;
- int rc = 0;
-
- brightness = bd->props.brightness;
-
- if ((bd->props.power != FB_BLANK_UNBLANK) ||
- (bd->props.state & BL_CORE_FBBLANK) ||
- (bd->props.state & BL_CORE_SUSPENDED))
- brightness = 0;
-
- c_conn = bl_get_data(bd);
- display = (struct dsi_display *) c_conn->display;
- if (brightness > display->panel->bl_config.bl_max_level)
- brightness = display->panel->bl_config.bl_max_level;
-
- /* map UI brightness into driver backlight level with rounding */
- bl_lvl = mult_frac(brightness, display->panel->bl_config.bl_max_level,
- display->panel->bl_config.brightness_max_level);
-
- if (!bl_lvl && brightness)
- bl_lvl = 1;
-
- if (c_conn->ops.set_backlight) {
- event.type = DRM_EVENT_SYS_BACKLIGHT;
- event.length = sizeof(u32);
- msm_mode_object_event_notify(&c_conn->base.base,
- c_conn->base.dev, &event, (u8 *)&brightness);
- rc = c_conn->ops.set_backlight(c_conn->display, bl_lvl);
- }
-
- return rc;
-}
-
-static int sde_backlight_device_get_brightness(struct backlight_device *bd)
-{
- return 0;
-}
-
-static const struct backlight_ops sde_backlight_device_ops = {
- .update_status = sde_backlight_device_update_status,
- .get_brightness = sde_backlight_device_get_brightness,
-};
-
-static int sde_backlight_setup(struct sde_connector *c_conn,
- struct drm_device *dev)
-{
- struct backlight_properties props;
- struct dsi_display *display;
- struct dsi_backlight_config *bl_config;
- static int display_count;
- char bl_node_name[BL_NODE_NAME_SIZE];
-
- if (!c_conn || !dev || !dev->dev) {
- SDE_ERROR("invalid param\n");
- return -EINVAL;
- } else if (c_conn->connector_type != DRM_MODE_CONNECTOR_DSI) {
- return 0;
- }
-
- memset(&props, 0, sizeof(props));
- props.type = BACKLIGHT_RAW;
- props.power = FB_BLANK_UNBLANK;
-
- display = (struct dsi_display *) c_conn->display;
- bl_config = &display->panel->bl_config;
- props.max_brightness = bl_config->brightness_max_level;
- props.brightness = bl_config->brightness_max_level;
- snprintf(bl_node_name, BL_NODE_NAME_SIZE, "panel%u-backlight",
- display_count);
- c_conn->bl_device = backlight_device_register(bl_node_name, dev->dev,
- c_conn, &sde_backlight_device_ops, &props);
- if (IS_ERR_OR_NULL(c_conn->bl_device)) {
- SDE_ERROR("Failed to register backlight: %ld\n",
- PTR_ERR(c_conn->bl_device));
- c_conn->bl_device = NULL;
- return -ENODEV;
- }
- display_count++;
-
- return 0;
-}
-
int sde_connector_trigger_event(void *drm_connector,
uint32_t event_idx, uint32_t instance_idx,
uint32_t data0, uint32_t data1,
@@ -418,6 +328,13 @@ void sde_connector_schedule_status_work(struct drm_connector *connector,
if (en == c_conn->esd_status_check)
return;
+ /* IRQ GPIO mode has specific work in DSI */
+ if (c_conn->ops.get_esd_mode) {
+ if (c_conn->ops.get_esd_mode(c_conn->display) ==
+ ESD_MODE_IRQ_GPIO)
+ return;
+ }
+
sde_connector_get_info(connector, &info);
if (c_conn->ops.check_status &&
(info.capabilities & MSM_DISPLAY_ESD_ENABLED)) {
@@ -529,10 +446,10 @@ static int _sde_connector_update_bl_scale(struct sde_connector *c_conn)
else
bl_config->bl_scale_ad = c_conn->bl_scale_ad;
- SDE_DEBUG("bl_scale = %u, bl_scale_ad = %u, bl_level = %u\n",
- bl_config->bl_scale, bl_config->bl_scale_ad,
- bl_config->bl_level);
- rc = c_conn->ops.set_backlight(dsi_display, bl_config->bl_level);
+ SDE_DEBUG("bl_scale = %u, bl_scale_ad = %u\n",
+ bl_config->bl_scale, bl_config->bl_scale_ad);
+ if (bl_config->bl_device)
+ backlight_update_status(bl_config->bl_device);
return rc;
}
@@ -639,11 +556,7 @@ void sde_connector_helper_bridge_disable(struct drm_connector *connector)
sde_connector_schedule_status_work(connector, false);
c_conn = to_sde_connector(connector);
- if (c_conn->panel_dead) {
- c_conn->bl_device->props.power = FB_BLANK_POWERDOWN;
- c_conn->bl_device->props.state |= BL_CORE_FBBLANK;
- backlight_update_status(c_conn->bl_device);
- }
+ c_conn->last_panel_power_mode = SDE_MODE_DPMS_OFF;
}
void sde_connector_helper_bridge_enable(struct drm_connector *connector)
@@ -654,14 +567,7 @@ void sde_connector_helper_bridge_enable(struct drm_connector *connector)
return;
c_conn = to_sde_connector(connector);
-
- /* Special handling for ESD recovery case */
- if (c_conn->panel_dead) {
- c_conn->bl_device->props.power = FB_BLANK_UNBLANK;
- c_conn->bl_device->props.state &= ~BL_CORE_FBBLANK;
- backlight_update_status(c_conn->bl_device);
- c_conn->panel_dead = false;
- }
+ c_conn->last_panel_power_mode = SDE_MODE_DPMS_ON;
}
int sde_connector_clk_ctrl(struct drm_connector *connector, bool enable)
@@ -715,8 +621,6 @@ static void sde_connector_destroy(struct drm_connector *connector)
drm_property_unreference_blob(c_conn->blob_ext_hdr);
msm_property_destroy(&c_conn->property_info);
- if (c_conn->bl_device)
- backlight_device_unregister(c_conn->bl_device);
drm_connector_unregister(connector);
mutex_destroy(&c_conn->lock);
sde_fence_deinit(&c_conn->retire_fence);
@@ -1593,6 +1497,53 @@ static const struct file_operations conn_cmd_tx_fops = {
.write = _sde_debugfs_conn_cmd_tx_write,
};
+static ssize_t _sde_debugfs_force_panel_dead_write(struct file *file,
+ const char __user *user_buf, size_t count, loff_t *ppos)
+{
+ struct drm_connector *connector = file->private_data;
+ struct sde_connector *c_conn;
+ struct dsi_display *display;
+ char buf[1];
+ int esd_mode = 0;
+
+ if (*ppos || !connector)
+ return 0;
+
+ c_conn = to_sde_connector(connector);
+ /* only support DSI type */
+ if (c_conn->connector_type != DRM_MODE_CONNECTOR_DSI)
+ return 0;
+
+ if (c_conn->ops.get_esd_mode) {
+ esd_mode = c_conn->ops.get_esd_mode(c_conn->display);
+ if (esd_mode < 0)
+ return 0;
+ }
+
+ display = c_conn->display;
+ if (!display)
+ return 0;
+
+ if (!count || copy_from_user(buf, user_buf, 1))
+ return -EINVAL;
+
+ if (buf[0] == '1') {
+ pr_info("force panel dead triggered\n");
+ if (esd_mode != ESD_MODE_IRQ_GPIO)
+ c_conn->force_panel_dead = 1;
+ else
+ schedule_work(&display->esd_irq_work);
+ }
+
+ return count;
+}
+
+static const struct file_operations force_panel_dead_fops = {
+ .open = simple_open,
+ .llseek = noop_llseek,
+ .write = _sde_debugfs_force_panel_dead_write,
+};
+
#ifdef CONFIG_DEBUG_FS
/**
* sde_connector_init_debugfs - initialize connector debugfs
@@ -1613,12 +1564,15 @@ static int sde_connector_init_debugfs(struct drm_connector *connector)
sde_connector_get_info(connector, &info);
if (sde_connector->ops.check_status &&
(info.capabilities & MSM_DISPLAY_ESD_ENABLED)) {
- debugfs_create_u32("force_panel_dead", 0600,
- connector->debugfs_entry,
- &sde_connector->force_panel_dead);
+ if (!debugfs_create_file("force_panel_dead", 0600,
+ connector->debugfs_entry,
+ connector, &force_panel_dead_fops)) {
+ pr_err("failed to create force_panel_dead\n");
+ return -ENOMEM;
+ }
debugfs_create_u32("esd_status_interval", 0600,
- connector->debugfs_entry,
- &sde_connector->esd_status_interval);
+ connector->debugfs_entry,
+ &sde_connector->esd_status_interval);
}
if (!debugfs_create_bool("fb_kmap", 0600, connector->debugfs_entry,
@@ -1762,22 +1716,23 @@ sde_connector_best_encoder(struct drm_connector *connector)
return c_conn->encoder;
}
-static void _sde_connector_report_panel_dead(struct sde_connector *conn)
+void sde_connector_report_panel_dead(struct drm_connector *connector)
{
+ struct sde_connector *c_conn = to_sde_connector(connector);
struct drm_event event;
- if (!conn)
+ if (!c_conn)
return;
- conn->panel_dead = true;
+ c_conn->panel_dead = true;
event.type = DRM_EVENT_PANEL_DEAD;
event.length = sizeof(bool);
- msm_mode_object_event_notify(&conn->base.base,
- conn->base.dev, &event, (u8 *)&conn->panel_dead);
- sde_encoder_display_failure_notification(conn->encoder);
+ msm_mode_object_event_notify(&c_conn->base.base,
+ c_conn->base.dev, &event, (u8 *)&c_conn->panel_dead);
+ sde_encoder_display_failure_notification(c_conn->encoder);
SDE_EVT32(SDE_EVTLOG_ERROR);
SDE_ERROR("esd check failed report PANEL_DEAD conn_id: %d enc_id: %d\n",
- conn->base.base.id, conn->encoder->base.id);
+ c_conn->base.base.id, c_conn->encoder->base.id);
}
int sde_connector_esd_status(struct drm_connector *conn)
@@ -1800,7 +1755,7 @@ int sde_connector_esd_status(struct drm_connector *conn)
if (ret <= 0) {
/* cancel if any pending esd work */
sde_connector_schedule_status_work(conn, false);
- _sde_connector_report_panel_dead(sde_conn);
+ sde_connector_report_panel_dead(conn);
ret = -ETIMEDOUT;
} else {
SDE_DEBUG("Successfully received TE from panel\n");
@@ -1855,7 +1810,7 @@ static void sde_connector_check_status_work(struct work_struct *work)
}
status_dead:
- _sde_connector_report_panel_dead(conn);
+ sde_connector_report_panel_dead(&conn->base);
}
static const struct drm_connector_helper_funcs sde_connector_helper_ops = {
@@ -2113,12 +2068,6 @@ struct drm_connector *sde_connector_init(struct drm_device *dev,
goto error_cleanup_fence;
}
- rc = sde_backlight_setup(c_conn, dev);
- if (rc) {
- SDE_ERROR("failed to setup backlight, rc=%d\n", rc);
- goto error_cleanup_fence;
- }
-
/* create properties */
msm_property_init(&c_conn->property_info, &c_conn->base.base, dev,
priv->conn_property, c_conn->property_data,
diff --git a/drivers/gpu/drm/msm/sde/sde_connector.h b/drivers/gpu/drm/msm/sde/sde_connector.h
index 51dc92d..fb30c88 100644
--- a/drivers/gpu/drm/msm/sde/sde_connector.h
+++ b/drivers/gpu/drm/msm/sde/sde_connector.h
@@ -162,8 +162,6 @@ struct sde_connector_ops {
void (*enable_event)(struct drm_connector *connector,
uint32_t event_idx, bool enable, void *display);
- int (*set_backlight)(void *display, u32 bl_lvl);
-
/**
* soft_reset - perform a soft reset on the connector
* @display: Pointer to private display structure
@@ -267,6 +265,13 @@ struct sde_connector_ops {
* Returns: v_front_porch on success error-code on failure
*/
int (*get_panel_vfp)(void *display, int h_active, int v_active);
+
+ /**
+ * get_esd_mode - get ESD mode of display panel
+ * @display: Pointer to private display handle
+ * Returns: Positive value for ESD mode, negetive for failure
+ */
+ int (*get_esd_mode)(void *display);
};
/**
@@ -788,4 +793,10 @@ int sde_connector_get_panel_vfp(struct drm_connector *connector,
*/
int sde_connector_esd_status(struct drm_connector *connector);
+/**
+ * sde_connector_report_panel_dead - helper function for panel recovery
+ * @c_conn: Pointer to DRM connector object
+ */
+void sde_connector_report_panel_dead(struct drm_connector *connector);
+
#endif /* _SDE_CONNECTOR_H_ */
diff --git a/drivers/gpu/drm/msm/sde/sde_crtc.c b/drivers/gpu/drm/msm/sde/sde_crtc.c
index 339b9dd..60a78b3 100644
--- a/drivers/gpu/drm/msm/sde/sde_crtc.c
+++ b/drivers/gpu/drm/msm/sde/sde_crtc.c
@@ -2198,7 +2198,7 @@ void sde_crtc_prepare_commit(struct drm_crtc *crtc,
}
/**
- * _sde_crtc_complete_flip - signal pending page_flip events
+ * sde_crtc_complete_flip - signal pending page_flip events
* Any pending vblank events are added to the vblank_event_list
* so that the next vblank interrupt shall signal them.
* However PAGE_FLIP events are not handled through the vblank_event_list.
@@ -2208,7 +2208,7 @@ void sde_crtc_prepare_commit(struct drm_crtc *crtc,
* @crtc: Pointer to drm crtc structure
* @file: Pointer to drm file
*/
-static void _sde_crtc_complete_flip(struct drm_crtc *crtc,
+void sde_crtc_complete_flip(struct drm_crtc *crtc,
struct drm_file *file)
{
struct sde_crtc *sde_crtc = to_sde_crtc(crtc);
@@ -2218,19 +2218,23 @@ static void _sde_crtc_complete_flip(struct drm_crtc *crtc,
spin_lock_irqsave(&dev->event_lock, flags);
event = sde_crtc->event;
- if (event) {
- /* if regular vblank case (!file) or if cancel-flip from
- * preclose on file that requested flip, then send the
- * event:
- */
- if (!file || (event->base.file_priv == file)) {
- sde_crtc->event = NULL;
- DRM_DEBUG_VBL("%s: send event: %pK\n",
- sde_crtc->name, event);
- SDE_EVT32_VERBOSE(DRMID(crtc));
- drm_crtc_send_vblank_event(crtc, event);
- }
+ if (!event)
+ goto end;
+
+ /*
+ * if regular vblank case (!file) or if cancel-flip from
+ * preclose on file that requested flip, then send the
+ * event:
+ */
+ if (!file || (event->base.file_priv == file)) {
+ sde_crtc->event = NULL;
+ DRM_DEBUG_VBL("%s: send event: %pK\n",
+ sde_crtc->name, event);
+ SDE_EVT32_VERBOSE(DRMID(crtc));
+ drm_crtc_send_vblank_event(crtc, event);
}
+
+end:
spin_unlock_irqrestore(&dev->event_lock, flags);
}
@@ -2271,7 +2275,6 @@ static void sde_crtc_vblank_cb(void *data)
sde_crtc->vblank_last_cb_time = ktime_get();
sysfs_notify_dirent(sde_crtc->vsync_event_sf);
- _sde_crtc_complete_flip(crtc, NULL);
drm_crtc_handle_vblank(crtc);
DRM_DEBUG_VBL("crtc%d\n", crtc->base.id);
SDE_EVT32_VERBOSE(DRMID(crtc));
@@ -2298,7 +2301,6 @@ static void sde_crtc_frame_event_work(struct kthread_work *work)
struct sde_crtc *sde_crtc;
struct sde_kms *sde_kms;
unsigned long flags;
- bool frame_done = false;
bool in_clone_mode = false;
if (!work) {
@@ -2353,10 +2355,6 @@ static void sde_crtc_frame_event_work(struct kthread_work *work)
SDE_EVT32_VERBOSE(DRMID(crtc), fevent->event,
SDE_EVTLOG_FUNC_CASE3);
}
-
- if (fevent->event & (SDE_ENCODER_FRAME_EVENT_DONE
- | SDE_ENCODER_FRAME_EVENT_ERROR))
- frame_done = true;
}
if (fevent->event & SDE_ENCODER_FRAME_EVENT_SIGNAL_RELEASE_FENCE) {
@@ -2377,9 +2375,6 @@ static void sde_crtc_frame_event_work(struct kthread_work *work)
SDE_ERROR("crtc%d ts:%lld received panel dead event\n",
crtc->base.id, ktime_to_ns(fevent->ts));
- if (frame_done)
- complete_all(&sde_crtc->frame_done_comp);
-
spin_lock_irqsave(&sde_crtc->spin_lock, flags);
list_add_tail(&fevent->list, &sde_crtc->frame_event_list);
spin_unlock_irqrestore(&sde_crtc->spin_lock, flags);
@@ -3027,7 +3022,6 @@ static void sde_crtc_atomic_begin(struct drm_crtc *crtc,
struct sde_crtc *sde_crtc;
struct drm_encoder *encoder;
struct drm_device *dev;
- unsigned long flags;
struct sde_kms *sde_kms;
if (!crtc) {
@@ -3061,14 +3055,6 @@ static void sde_crtc_atomic_begin(struct drm_crtc *crtc,
_sde_crtc_setup_lm_bounds(crtc, crtc->state);
}
- if (sde_crtc->event) {
- WARN_ON(sde_crtc->event);
- } else {
- spin_lock_irqsave(&dev->event_lock, flags);
- sde_crtc->event = crtc->state->event;
- spin_unlock_irqrestore(&dev->event_lock, flags);
- }
-
list_for_each_entry(encoder, &dev->mode_config.encoder_list, head) {
if (encoder->crtc != crtc)
continue;
@@ -3100,7 +3086,8 @@ static void sde_crtc_atomic_begin(struct drm_crtc *crtc,
* apply color processing properties only if
* smmu state is attached,
*/
- if (!sde_kms_is_secure_session_inprogress(sde_kms) && sde_crtc->enabled)
+ if (!sde_kms_is_secure_session_inprogress(sde_kms) &&
+ (sde_kms->splash_data.cont_splash_en || sde_crtc->enabled))
sde_cp_crtc_apply_properties(crtc);
/*
@@ -3121,7 +3108,6 @@ static void sde_crtc_atomic_flush(struct drm_crtc *crtc,
struct drm_plane *plane;
struct msm_drm_private *priv;
struct msm_drm_thread *event_thread;
- unsigned long flags;
struct sde_crtc_state *cstate;
struct sde_kms *sde_kms;
int idle_time = 0;
@@ -3163,14 +3149,6 @@ static void sde_crtc_atomic_flush(struct drm_crtc *crtc,
event_thread = &priv->event_thread[crtc->index];
idle_time = sde_crtc_get_property(cstate, CRTC_PROP_IDLE_TIMEOUT);
- if (sde_crtc->event) {
- SDE_DEBUG("already received sde_crtc->event\n");
- } else {
- spin_lock_irqsave(&dev->event_lock, flags);
- sde_crtc->event = crtc->state->event;
- spin_unlock_irqrestore(&dev->event_lock, flags);
- }
-
/*
* If no mixers has been allocated in sde_crtc_atomic_check(),
* it means we are trying to flush a CRTC whose state is disabled:
@@ -3259,10 +3237,10 @@ static void sde_crtc_destroy_state(struct drm_crtc *crtc,
&cstate->property_state);
}
-static int _sde_crtc_wait_for_frame_done(struct drm_crtc *crtc)
+static int _sde_crtc_flush_event_thread(struct drm_crtc *crtc)
{
struct sde_crtc *sde_crtc;
- int ret, rc = 0, i;
+ int i;
if (!crtc) {
SDE_ERROR("invalid argument\n");
@@ -3286,17 +3264,9 @@ static int _sde_crtc_wait_for_frame_done(struct drm_crtc *crtc)
kthread_flush_work(&sde_crtc->frame_events[i].work);
}
- ret = wait_for_completion_timeout(&sde_crtc->frame_done_comp,
- msecs_to_jiffies(SDE_FRAME_DONE_TIMEOUT));
- if (!ret) {
- SDE_ERROR("frame done completion wait timed out, ret:%d\n",
- ret);
- SDE_EVT32(DRMID(crtc), SDE_EVTLOG_FATAL);
- rc = -ETIMEDOUT;
- }
SDE_EVT32_VERBOSE(DRMID(crtc), SDE_EVTLOG_FUNC_EXIT);
- return rc;
+ return 0;
}
static int _sde_crtc_commit_kickoff_rot(struct drm_crtc *crtc,
@@ -3623,7 +3593,7 @@ void sde_crtc_commit_kickoff(struct drm_crtc *crtc,
struct sde_kms *sde_kms;
struct sde_crtc_state *cstate;
bool is_error, reset_req;
- int ret;
+ unsigned long flags;
enum sde_crtc_idle_pc_state idle_pc_state;
if (!crtc) {
@@ -3689,20 +3659,9 @@ void sde_crtc_commit_kickoff(struct drm_crtc *crtc,
}
sde_crtc->reset_request = reset_req;
- /* wait for frame_event_done completion */
- SDE_ATRACE_BEGIN("wait_for_frame_done_event");
- ret = _sde_crtc_wait_for_frame_done(crtc);
- SDE_ATRACE_END("wait_for_frame_done_event");
- if (ret) {
- SDE_ERROR("crtc%d wait for frame done failed;frame_pending%d\n",
- crtc->base.id,
- atomic_read(&sde_crtc->frame_pending));
-
- is_error = true;
-
- /* force offline rotation mode since the commit has no pipes */
- cstate->sbuf_cfg.rot_op_mode = SDE_CTL_ROT_OP_MODE_OFFLINE;
- }
+ SDE_ATRACE_BEGIN("flush_event_thread");
+ _sde_crtc_flush_event_thread(crtc);
+ SDE_ATRACE_END("flush_event_thread");
if (atomic_inc_return(&sde_crtc->frame_pending) == 1) {
/* acquire bandwidth and other resources */
@@ -3741,7 +3700,15 @@ void sde_crtc_commit_kickoff(struct drm_crtc *crtc,
sde_encoder_kickoff(encoder, false);
}
- reinit_completion(&sde_crtc->frame_done_comp);
+ /* store the event after frame trigger */
+ if (sde_crtc->event) {
+ WARN_ON(sde_crtc->event);
+ } else {
+ spin_lock_irqsave(&dev->event_lock, flags);
+ sde_crtc->event = crtc->state->event;
+ spin_unlock_irqrestore(&dev->event_lock, flags);
+ }
+
SDE_ATRACE_END("crtc_commit");
return;
}
@@ -4142,11 +4109,7 @@ static void sde_crtc_disable(struct drm_crtc *crtc)
if (cstate->num_ds_enabled)
sde_crtc->ds_reconfig = true;
- /* wait for frame_event_done completion */
- if (_sde_crtc_wait_for_frame_done(crtc))
- SDE_ERROR("crtc%d wait for frame done failed;frame_pending%d\n",
- crtc->base.id,
- atomic_read(&sde_crtc->frame_pending));
+ _sde_crtc_flush_event_thread(crtc);
SDE_EVT32(DRMID(crtc), sde_crtc->enabled, sde_crtc->suspend,
sde_crtc->vblank_requested,
@@ -4161,6 +4124,8 @@ static void sde_crtc_disable(struct drm_crtc *crtc)
sde_crtc->enabled = false;
if (atomic_read(&sde_crtc->frame_pending)) {
+ SDE_ERROR("crtc%d frame_pending%d\n", crtc->base.id,
+ atomic_read(&sde_crtc->frame_pending));
SDE_EVT32(DRMID(crtc), atomic_read(&sde_crtc->frame_pending),
SDE_EVTLOG_FUNC_CASE2);
sde_core_perf_crtc_release_bw(crtc);
@@ -4857,14 +4822,6 @@ int sde_crtc_vblank(struct drm_crtc *crtc, bool en)
return 0;
}
-void sde_crtc_cancel_pending_flip(struct drm_crtc *crtc, struct drm_file *file)
-{
- struct sde_crtc *sde_crtc = to_sde_crtc(crtc);
-
- SDE_DEBUG("%s: cancel: %pK\n", sde_crtc->name, file);
- _sde_crtc_complete_flip(crtc, file);
-}
-
int sde_crtc_helper_reset_custom_properties(struct drm_crtc *crtc,
struct drm_crtc_state *crtc_state)
{
@@ -5518,16 +5475,16 @@ static int _sde_debugfs_status_show(struct seq_file *s, void *data)
sde_crtc->vblank_cb_count * 1000, diff_ms) : 0;
seq_printf(s,
- "vblank fps:%lld count:%u total:%llums total_framecount:%llu\n",
+ "vblank fps:%lld count:%u total:%llums\n",
fps, sde_crtc->vblank_cb_count,
- ktime_to_ms(diff), sde_crtc->play_count);
+ ktime_to_ms(diff));
/* reset time & count for next measurement */
sde_crtc->vblank_cb_count = 0;
sde_crtc->vblank_cb_time = ktime_set(0, 0);
}
-
seq_printf(s, "vblank_enable:%d\n", sde_crtc->vblank_requested);
+ seq_printf(s, "total_framecount:%llu\n", sde_crtc->play_count);
mutex_unlock(&sde_crtc->crtc_lock);
@@ -5960,7 +5917,6 @@ struct drm_crtc *sde_crtc_init(struct drm_device *dev, struct drm_plane *plane)
mutex_init(&sde_crtc->rp_lock);
INIT_LIST_HEAD(&sde_crtc->rp_head);
- init_completion(&sde_crtc->frame_done_comp);
sde_crtc->enabled = false;
INIT_LIST_HEAD(&sde_crtc->frame_event_list);
diff --git a/drivers/gpu/drm/msm/sde/sde_crtc.h b/drivers/gpu/drm/msm/sde/sde_crtc.h
index 022c77e..aa1c88c 100644
--- a/drivers/gpu/drm/msm/sde/sde_crtc.h
+++ b/drivers/gpu/drm/msm/sde/sde_crtc.h
@@ -196,7 +196,6 @@ struct sde_crtc_event {
* @frame_events : static allocation of in-flight frame events
* @frame_event_list : available frame event list
* @spin_lock : spin lock for frame event, transaction status, etc...
- * @frame_done_comp : for frame_event_done synchronization
* @event_thread : Pointer to event handler thread
* @event_worker : Event worker queue
* @event_cache : Local cache of event worker structures
@@ -267,7 +266,6 @@ struct sde_crtc {
struct sde_crtc_frame_event frame_events[SDE_CRTC_FRAME_EVENT_SIZE];
struct list_head frame_event_list;
spinlock_t spin_lock;
- struct completion frame_done_comp;
/* for handling internal event thread */
struct sde_crtc_event event_cache[SDE_CRTC_MAX_EVENT_COUNT];
@@ -562,11 +560,11 @@ struct drm_crtc *sde_crtc_init(struct drm_device *dev, struct drm_plane *plane);
int sde_crtc_post_init(struct drm_device *dev, struct drm_crtc *crtc);
/**
- * sde_crtc_cancel_pending_flip - complete flip for clients on lastclose
+ * sde_crtc_complete_flip - complete flip for clients
* @crtc: Pointer to drm crtc object
* @file: client to cancel's file handle
*/
-void sde_crtc_cancel_pending_flip(struct drm_crtc *crtc, struct drm_file *file);
+void sde_crtc_complete_flip(struct drm_crtc *crtc, struct drm_file *file);
/**
* sde_crtc_register_custom_event - api for enabling/disabling crtc event
diff --git a/drivers/gpu/drm/msm/sde/sde_encoder.c b/drivers/gpu/drm/msm/sde/sde_encoder.c
index 1f30378..5feb5b6 100644
--- a/drivers/gpu/drm/msm/sde/sde_encoder.c
+++ b/drivers/gpu/drm/msm/sde/sde_encoder.c
@@ -168,6 +168,7 @@ enum sde_enc_rc_states {
* Virtual encoder defers as much as possible to the physical encoders.
* Virtual encoder registers itself with the DRM Framework as the encoder.
* @base: drm_encoder base class for registration with DRM
+ * @vsync_source Source of vsync for connected display
* @enc_spin_lock: Virtual-Encoder-Wide Spin Lock for IRQ purposes
* @bus_scaling_client: Client handle to the bus scaling interface
* @num_phys_encs: Actual number of physical encoders contained.
@@ -220,6 +221,7 @@ enum sde_enc_rc_states {
*/
struct sde_encoder_virt {
struct drm_encoder base;
+ uint32_t vsync_source;
spinlock_t enc_spinlock;
struct mutex vblank_ctl_lock;
uint32_t bus_scaling_client;
@@ -266,6 +268,8 @@ struct sde_encoder_virt {
struct sde_rect cur_conn_roi;
struct sde_rect prv_conn_roi;
struct drm_crtc *crtc;
+
+ bool elevated_ahb_vote;
};
#define to_sde_encoder_virt(x) container_of(x, struct sde_encoder_virt, base)
@@ -1495,7 +1499,12 @@ static void _sde_encoder_update_vsync_source(struct sde_encoder_virt *sde_enc,
else if (disp_info->is_te_using_watchdog_timer)
vsync_cfg.vsync_source = SDE_VSYNC_SOURCE_WD_TIMER_0;
else
- vsync_cfg.vsync_source = SDE_VSYNC0_SOURCE_GPIO;
+ /* TODO(b/68321698) Revert this:
+ * drm: msm: Add vsync-source device tree config...
+ * and replace with QC implementation of moving
+ * vsync_source into device tree.
+ */
+ vsync_cfg.vsync_source = sde_enc->vsync_source;
vsync_cfg.is_dummy = is_dummy;
hw_mdptop->ops.setup_vsync_source(hw_mdptop, &vsync_cfg);
@@ -1849,6 +1858,7 @@ static int _sde_encoder_resource_control_helper(struct drm_encoder *drm_enc,
return rc;
}
+ sde_enc->elevated_ahb_vote = true;
/* enable DSI clks */
rc = sde_connector_clk_ctrl(sde_enc->cur_master->connector,
true);
@@ -1988,6 +1998,8 @@ static int sde_encoder_resource_control(struct drm_encoder *drm_enc,
SDE_DEBUG_ENC(sde_enc, "sw_event:%d, work cancelled\n",
sw_event);
+ msm_idle_set_state(drm_enc, true);
+
mutex_lock(&sde_enc->rc_lock);
/* return if the resource control is already in ON state */
@@ -2092,6 +2104,8 @@ static int sde_encoder_resource_control(struct drm_encoder *drm_enc,
else
idle_pc_duration = IDLE_POWERCOLLAPSE_DURATION;
+ msm_idle_set_state(drm_enc, false);
+
if (!autorefresh_enabled)
kthread_mod_delayed_work(
&disp_thread->worker,
@@ -2839,11 +2853,10 @@ static void sde_encoder_virt_disable(struct drm_encoder *drm_enc)
/* wait for idle */
sde_encoder_wait_for_event(drm_enc, MSM_ENC_TX_COMPLETE);
- kthread_flush_work(&sde_enc->input_event_work);
-
if (sde_enc->input_handler)
input_unregister_handler(sde_enc->input_handler);
+ kthread_cancel_work_sync(&sde_enc->input_event_work);
/*
* For primary command mode encoders, execute the resource control
* pre-stop operations before the physical encoders are disabled, to
@@ -3317,6 +3330,8 @@ static void _sde_encoder_kickoff_phys(struct sde_encoder_virt *sde_enc)
struct sde_hw_ctl *ctl;
uint32_t i, pending_flush;
unsigned long lock_flags;
+ struct msm_drm_private *priv = NULL;
+ struct sde_kms *sde_kms = NULL;
if (!sde_enc) {
SDE_ERROR("invalid encoder\n");
@@ -3394,6 +3409,16 @@ static void _sde_encoder_kickoff_phys(struct sde_encoder_virt *sde_enc)
_sde_encoder_trigger_start(sde_enc->cur_master);
spin_unlock_irqrestore(&sde_enc->enc_spinlock, lock_flags);
+
+ if (sde_enc->elevated_ahb_vote) {
+ priv = sde_enc->base.dev->dev_private;
+ sde_kms = to_sde_kms(priv->kms);
+ if (priv != NULL && sde_kms != NULL) {
+ sde_power_scale_reg_bus(&priv->phandle,
+ sde_kms->core_client, VOTE_INDEX_LOW, false);
+ }
+ sde_enc->elevated_ahb_vote = false;
+ }
}
static void _sde_encoder_ppsplit_swap_intf_for_right_only_update(
@@ -4600,6 +4625,7 @@ struct drm_encoder *sde_encoder_init(
int drm_enc_mode = DRM_MODE_ENCODER_NONE;
char name[SDE_NAME_SIZE];
int ret = 0;
+ struct device_node *np = dev->dev->of_node;
sde_enc = kzalloc(sizeof(*sde_enc), GFP_KERNEL);
if (!sde_enc) {
@@ -4658,6 +4684,10 @@ struct drm_encoder *sde_encoder_init(
memcpy(&sde_enc->disp_info, disp_info, sizeof(*disp_info));
+ if (of_property_read_u32(np, "qcom,sde-vsync-source",
+ &sde_enc->vsync_source))
+ SDE_ERROR("No vsync-source found in DT\n");
+
SDE_DEBUG_ENC(sde_enc, "created\n");
return drm_enc;
diff --git a/drivers/gpu/drm/msm/sde/sde_kms.c b/drivers/gpu/drm/msm/sde/sde_kms.c
index c2fffef..c307c7f 100644
--- a/drivers/gpu/drm/msm/sde/sde_kms.c
+++ b/drivers/gpu/drm/msm/sde/sde_kms.c
@@ -25,6 +25,7 @@
#include <linux/dma-buf.h>
#include <linux/memblock.h>
#include <linux/bootmem.h>
+#include <linux/msm_drm_notify.h>
#include "msm_drv.h"
#include "msm_mmu.h"
@@ -59,6 +60,8 @@
#define MEM_PROTECT_SD_CTRL_SWITCH 0x18
#define MDP_DEVICE_ID 0x1A
+#define PANEL_INFO_CLASS_NAME "panel_info"
+
static const char * const iommu_ports[] = {
"mdp_0",
};
@@ -100,6 +103,11 @@ static int _sde_kms_mmu_destroy(struct sde_kms *sde_kms);
static int _sde_kms_mmu_init(struct sde_kms *sde_kms);
static int _sde_kms_register_events(struct msm_kms *kms,
struct drm_mode_object *obj, u32 event, bool en);
+
+static int panel_info_dev_create(struct sde_kms *sde_kms);
+static void panel_info_dev_release(struct sde_kms *sde_kms);
+static struct class *panel_info_class;
+
bool sde_is_custom_client(void)
{
return sdecustom;
@@ -898,6 +906,71 @@ static int _sde_kms_splash_smmu_unmap(struct sde_kms *sde_kms)
return rc;
}
+static int _sde_kms_get_blank(struct drm_crtc_state *crtc_state,
+ struct drm_connector_state *conn_state)
+{
+ int lp_mode, blank;
+
+ if (crtc_state->active)
+ lp_mode = sde_connector_get_property(conn_state,
+ CONNECTOR_PROP_LP);
+ else
+ lp_mode = SDE_MODE_DPMS_OFF;
+
+ switch (lp_mode) {
+ case SDE_MODE_DPMS_ON:
+ blank = MSM_DRM_BLANK_UNBLANK;
+ break;
+ case SDE_MODE_DPMS_LP1:
+ case SDE_MODE_DPMS_LP2:
+ blank = MSM_DRM_BLANK_LP;
+ break;
+ case SDE_MODE_DPMS_OFF:
+ default:
+ blank = MSM_DRM_BLANK_POWERDOWN;
+ break;
+ }
+
+ return blank;
+}
+
+static void _sde_kms_drm_check_dpms(struct drm_atomic_state *old_state,
+ unsigned long event)
+{
+ struct drm_connector *connector;
+ struct drm_connector_state *old_conn_state;
+ struct drm_crtc_state *old_crtc_state;
+ int i, old_mode, new_mode;
+
+ for_each_connector_in_state(old_state, connector, old_conn_state, i) {
+ if (!connector->state->crtc)
+ continue;
+
+ new_mode = _sde_kms_get_blank(connector->state->crtc->state,
+ connector->state);
+ if (old_conn_state->crtc) {
+ old_crtc_state = drm_atomic_get_existing_crtc_state(
+ old_state, old_conn_state->crtc);
+ old_mode = _sde_kms_get_blank(old_crtc_state,
+ old_conn_state);
+ } else {
+ old_mode = MSM_DRM_BLANK_POWERDOWN;
+ }
+
+ if (old_mode != new_mode) {
+ struct msm_drm_notifier notifier_data;
+
+ pr_debug("power mode change detected %d->%d\n",
+ old_mode, new_mode);
+
+ notifier_data.data = &new_mode;
+ notifier_data.id = connector->state->crtc->index;
+
+ msm_drm_notifier_call_chain(event, ¬ifier_data);
+ }
+ }
+}
+
static void sde_kms_prepare_commit(struct msm_kms *kms,
struct drm_atomic_state *state)
{
@@ -926,6 +999,12 @@ static void sde_kms_prepare_commit(struct msm_kms *kms,
return;
}
+ if (sde_kms->first_kickoff) {
+ sde_power_scale_reg_bus(&priv->phandle, sde_kms->core_client,
+ VOTE_INDEX_HIGH, false);
+ sde_kms->first_kickoff = false;
+ }
+
for_each_crtc_in_state(state, crtc, crtc_state, i) {
list_for_each_entry(encoder, &dev->mode_config.encoder_list,
head) {
@@ -942,6 +1021,8 @@ static void sde_kms_prepare_commit(struct msm_kms *kms,
* transitions prepare below if any transtions is required.
*/
sde_kms_prepare_secure_transition(kms, state);
+
+ _sde_kms_drm_check_dpms(state, MSM_DRM_EARLY_EVENT_BLANK);
}
static void sde_kms_commit(struct msm_kms *kms,
@@ -1070,6 +1151,8 @@ static void sde_kms_complete_commit(struct msm_kms *kms,
}
}
+ _sde_kms_drm_check_dpms(old_state, MSM_DRM_EVENT_BLANK);
+
sde_power_resource_enable(&priv->phandle, sde_kms->core_client, false);
_sde_kms_release_splash_resource(sde_kms, old_state);
@@ -1115,6 +1198,8 @@ static void sde_kms_wait_for_commit_done(struct msm_kms *kms,
SDE_ERROR("wait for commit done returned %d\n", ret);
break;
}
+
+ sde_crtc_complete_flip(crtc, NULL);
}
}
@@ -1241,11 +1326,125 @@ static void _sde_kms_release_displays(struct sde_kms *sde_kms)
sde_kms->wb_displays = NULL;
sde_kms->wb_display_count = 0;
+ panel_info_dev_release(sde_kms);
kfree(sde_kms->dsi_displays);
sde_kms->dsi_displays = NULL;
sde_kms->dsi_display_count = 0;
}
+static ssize_t panel_vendor_name_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct dsi_display *display;
+ struct dsi_panel *panel;
+
+ display = dev_get_drvdata(dev);
+ panel = display->panel;
+
+ if (!display || !panel || !panel->vendor_info.name || !buf) {
+ pr_err("Failed to show vendor name\n");
+ return 0;
+ }
+
+ return scnprintf(buf, PAGE_SIZE, "%s\n", panel->vendor_info.name);
+}
+
+static ssize_t serial_number_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct dsi_display *display;
+ struct dsi_panel *panel;
+
+ display = dev_get_drvdata(dev);
+ panel = display->panel;
+
+ if (!display || !panel || !panel->vendor_info.is_sn || !buf) {
+ pr_err("Failed to show SN\n");
+ return 0;
+ }
+
+ return scnprintf(buf, PAGE_SIZE, "%s\n", panel->vendor_info.sn);
+}
+
+struct device_attribute dev_attr_panel_vendor_name =
+ __ATTR_RO_MODE(panel_vendor_name, 0400);
+struct device_attribute dev_attr_serial_number =
+ __ATTR_RO_MODE(serial_number, 0400);
+
+static struct attribute *panel_info_dev_attrs[] = {
+ &dev_attr_panel_vendor_name.attr,
+ &dev_attr_serial_number.attr,
+ NULL
+};
+
+
+static const struct attribute_group panel_info_dev_group = {
+ .attrs = panel_info_dev_attrs,
+};
+
+static const struct attribute_group *panel_info_dev_groups[] = {
+ &panel_info_dev_group,
+ NULL
+};
+
+static int panel_info_dev_create(struct sde_kms *sde_kms)
+{
+ struct dsi_display *display;
+ int i;
+
+ if (panel_info_class && !IS_ERR(panel_info_class))
+ return 0;
+
+ panel_info_class = class_create(THIS_MODULE, PANEL_INFO_CLASS_NAME);
+ if (!panel_info_class || IS_ERR(panel_info_class))
+ return -EINVAL;
+
+ for (i = 0; i < sde_kms->dsi_display_count; ++i) {
+ display = (struct dsi_display *)sde_kms->dsi_displays[i];
+ display->panel_info_dev =
+ device_create_with_groups(panel_info_class,
+ &display->pdev->dev,
+ 0,
+ display,
+ panel_info_dev_groups,
+ "panel%d",
+ i);
+ if (!display->panel_info_dev || IS_ERR(display->panel_info_dev))
+ goto error;
+ }
+ return 0;
+error:
+ while (--i >= 0) {
+ display = (struct dsi_display *)sde_kms->dsi_displays[i];
+ device_unregister(display->panel_info_dev);
+ display->panel_info_dev = NULL;
+ }
+
+ class_destroy(panel_info_class);
+ panel_info_class = NULL;
+
+ return -EINVAL;
+}
+
+static void panel_info_dev_release(struct sde_kms *sde_kms)
+{
+ struct dsi_display *display;
+ int i;
+
+ if (!panel_info_class || IS_ERR(panel_info_class))
+ return;
+
+ for (i = 0; i < sde_kms->dsi_display_count; ++i) {
+ display = (struct dsi_display *)sde_kms->dsi_displays[i];
+ device_unregister(display->panel_info_dev);
+ display->panel_info_dev = NULL;
+ }
+ class_destroy(panel_info_class);
+ panel_info_class = NULL;
+}
+
/**
* _sde_kms_setup_displays - create encoders, bridges and connectors
* for underlying displays
@@ -1265,7 +1464,6 @@ static int _sde_kms_setup_displays(struct drm_device *dev,
.put_modes = dsi_connector_put_modes,
.mode_valid = dsi_conn_mode_valid,
.get_info = dsi_display_get_info,
- .set_backlight = dsi_display_set_backlight,
.soft_reset = dsi_display_soft_reset,
.pre_kickoff = dsi_conn_pre_kickoff,
.clk_ctrl = dsi_display_clk_ctrl,
@@ -1278,6 +1476,7 @@ static int _sde_kms_setup_displays(struct drm_device *dev,
.cmd_transfer = dsi_display_cmd_transfer,
.cont_splash_config = dsi_display_cont_splash_config,
.get_panel_vfp = dsi_display_get_panel_vfp,
+ .get_esd_mode = dsi_display_get_esd_mode,
};
static const struct sde_connector_ops wb_ops = {
.post_init = sde_wb_connector_post_init,
@@ -1339,6 +1538,7 @@ static int _sde_kms_setup_displays(struct drm_device *dev,
}
/* dsi */
+ panel_info_dev_create(sde_kms);
for (i = 0; i < sde_kms->dsi_display_count &&
priv->num_encoders < max_encoders; ++i) {
display = sde_kms->dsi_displays[i];
@@ -2279,8 +2479,9 @@ static void sde_kms_preclose(struct msm_kms *kms, struct drm_file *file)
struct drm_atomic_state *state = NULL;
int ret = 0;
+ /* cancel pending flip event */
for (i = 0; i < priv->num_crtcs; i++)
- sde_crtc_cancel_pending_flip(priv->crtcs[i], file);
+ sde_crtc_complete_flip(priv->crtcs[i], file);
drm_modeset_lock_all(dev);
state = drm_atomic_state_alloc(dev);
@@ -3073,8 +3274,10 @@ static void sde_kms_handle_power_event(u32 event_type, void *usr)
if (event_type == SDE_POWER_EVENT_POST_ENABLE) {
sde_irq_update(msm_kms, true);
sde_vbif_init_memtypes(sde_kms);
+ sde_kms->first_kickoff = true;
} else if (event_type == SDE_POWER_EVENT_PRE_DISABLE) {
sde_irq_update(msm_kms, false);
+ sde_kms->first_kickoff = false;
}
}
@@ -3293,6 +3496,22 @@ static int sde_kms_hw_init(struct msm_kms *kms)
goto power_error;
}
+ sde_kms->splash_data.resource_handoff_pending = true;
+
+ rc = _sde_kms_mmu_init(sde_kms);
+ if (rc) {
+ SDE_ERROR("sde_kms_mmu_init failed: %d\n", rc);
+ goto power_error;
+ }
+
+ /* Initialize reg dma block which is a singleton */
+ rc = sde_reg_dma_init(sde_kms->reg_dma, sde_kms->catalog,
+ sde_kms->dev);
+ if (rc) {
+ SDE_ERROR("failed: reg dma init failed\n");
+ goto power_error;
+ }
+
sde_dbg_init_dbg_buses(sde_kms->core_rev);
rm = &sde_kms->rm;
@@ -3322,21 +3541,6 @@ static int sde_kms_hw_init(struct msm_kms *kms)
&sde_kms->splash_data,
sde_kms->catalog);
- sde_kms->splash_data.resource_handoff_pending = true;
-
- /* Initialize reg dma block which is a singleton */
- rc = sde_reg_dma_init(sde_kms->reg_dma, sde_kms->catalog,
- sde_kms->dev);
- if (rc) {
- SDE_ERROR("failed: reg dma init failed\n");
- goto power_error;
- }
-
- rc = _sde_kms_mmu_init(sde_kms);
- if (rc) {
- SDE_ERROR("sde_kms_mmu_init failed: %d\n", rc);
- goto power_error;
- }
sde_kms->hw_mdp = sde_rm_get_mdp(&sde_kms->rm);
if (IS_ERR_OR_NULL(sde_kms->hw_mdp)) {
rc = PTR_ERR(sde_kms->hw_mdp);
diff --git a/drivers/gpu/drm/msm/sde/sde_kms.h b/drivers/gpu/drm/msm/sde/sde_kms.h
index 8758683..49171e56 100644
--- a/drivers/gpu/drm/msm/sde/sde_kms.h
+++ b/drivers/gpu/drm/msm/sde/sde_kms.h
@@ -288,6 +288,8 @@ struct sde_kms {
atomic_t detach_sec_cb;
atomic_t detach_all_cb;
struct mutex secure_transition_lock;
+
+ bool first_kickoff;
};
struct vsync_info {
diff --git a/drivers/gpu/drm/nouveau/nouveau_connector.c b/drivers/gpu/drm/nouveau/nouveau_connector.c
index 56c288f..5bfae1f 100644
--- a/drivers/gpu/drm/nouveau/nouveau_connector.c
+++ b/drivers/gpu/drm/nouveau/nouveau_connector.c
@@ -271,12 +271,16 @@ nouveau_connector_detect(struct drm_connector *connector, bool force)
nv_connector->edid = NULL;
}
- /* Outputs are only polled while runtime active, so acquiring a
- * runtime PM ref here is unnecessary (and would deadlock upon
- * runtime suspend because it waits for polling to finish).
+ /* Outputs are only polled while runtime active, so resuming the
+ * device here is unnecessary (and would deadlock upon runtime suspend
+ * because it waits for polling to finish). We do however, want to
+ * prevent the autosuspend timer from elapsing during this operation
+ * if possible.
*/
- if (!drm_kms_helper_is_poll_worker()) {
- ret = pm_runtime_get_sync(connector->dev->dev);
+ if (drm_kms_helper_is_poll_worker()) {
+ pm_runtime_get_noresume(dev->dev);
+ } else {
+ ret = pm_runtime_get_sync(dev->dev);
if (ret < 0 && ret != -EACCES)
return conn_status;
}
@@ -354,10 +358,8 @@ nouveau_connector_detect(struct drm_connector *connector, bool force)
out:
- if (!drm_kms_helper_is_poll_worker()) {
- pm_runtime_mark_last_busy(connector->dev->dev);
- pm_runtime_put_autosuspend(connector->dev->dev);
- }
+ pm_runtime_mark_last_busy(dev->dev);
+ pm_runtime_put_autosuspend(dev->dev);
return conn_status;
}
diff --git a/drivers/gpu/drm/nouveau/nouveau_display.c b/drivers/gpu/drm/nouveau/nouveau_display.c
index 6526a33..3ddd409 100644
--- a/drivers/gpu/drm/nouveau/nouveau_display.c
+++ b/drivers/gpu/drm/nouveau/nouveau_display.c
@@ -367,8 +367,6 @@ nouveau_display_hpd_work(struct work_struct *work)
pm_runtime_get_sync(drm->dev->dev);
drm_helper_hpd_irq_event(drm->dev);
- /* enable polling for external displays */
- drm_kms_helper_poll_enable(drm->dev);
pm_runtime_mark_last_busy(drm->dev->dev);
pm_runtime_put_sync(drm->dev->dev);
@@ -391,15 +389,29 @@ nouveau_display_acpi_ntfy(struct notifier_block *nb, unsigned long val,
{
struct nouveau_drm *drm = container_of(nb, typeof(*drm), acpi_nb);
struct acpi_bus_event *info = data;
+ int ret;
if (!strcmp(info->device_class, ACPI_VIDEO_CLASS)) {
if (info->type == ACPI_VIDEO_NOTIFY_PROBE) {
- /*
- * This may be the only indication we receive of a
- * connector hotplug on a runtime suspended GPU,
- * schedule hpd_work to check.
- */
- schedule_work(&drm->hpd_work);
+ ret = pm_runtime_get(drm->dev->dev);
+ if (ret == 1 || ret == -EACCES) {
+ /* If the GPU is already awake, or in a state
+ * where we can't wake it up, it can handle
+ * it's own hotplug events.
+ */
+ pm_runtime_put_autosuspend(drm->dev->dev);
+ } else if (ret == 0) {
+ /* This may be the only indication we receive
+ * of a connector hotplug on a runtime
+ * suspended GPU, schedule hpd_work to check.
+ */
+ NV_DEBUG(drm, "ACPI requested connector reprobe\n");
+ schedule_work(&drm->hpd_work);
+ pm_runtime_put_noidle(drm->dev->dev);
+ } else {
+ NV_WARN(drm, "Dropped ACPI reprobe event due to RPM error: %d\n",
+ ret);
+ }
/* acpi-video should not generate keypresses for this */
return NOTIFY_BAD;
@@ -422,6 +434,11 @@ nouveau_display_init(struct drm_device *dev)
if (ret)
return ret;
+ /* enable connector detection and polling for connectors without HPD
+ * support
+ */
+ drm_kms_helper_poll_enable(dev);
+
/* enable hotplug interrupts */
list_for_each_entry(connector, &dev->mode_config.connector_list, head) {
struct nouveau_connector *conn = nouveau_connector(connector);
diff --git a/drivers/gpu/drm/nouveau/nouveau_gem.c b/drivers/gpu/drm/nouveau/nouveau_gem.c
index 909f69a..505dca4 100644
--- a/drivers/gpu/drm/nouveau/nouveau_gem.c
+++ b/drivers/gpu/drm/nouveau/nouveau_gem.c
@@ -601,7 +601,7 @@ nouveau_gem_pushbuf_reloc_apply(struct nouveau_cli *cli,
struct nouveau_bo *nvbo;
uint32_t data;
- if (unlikely(r->bo_index > req->nr_buffers)) {
+ if (unlikely(r->bo_index >= req->nr_buffers)) {
NV_PRINTK(err, cli, "reloc bo index invalid\n");
ret = -EINVAL;
break;
@@ -611,7 +611,7 @@ nouveau_gem_pushbuf_reloc_apply(struct nouveau_cli *cli,
if (b->presumed.valid)
continue;
- if (unlikely(r->reloc_bo_index > req->nr_buffers)) {
+ if (unlikely(r->reloc_bo_index >= req->nr_buffers)) {
NV_PRINTK(err, cli, "reloc container bo index invalid\n");
ret = -EINVAL;
break;
diff --git a/drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c b/drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c
index 9b638bd..d370bf8 100644
--- a/drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c
+++ b/drivers/gpu/drm/nouveau/nvkm/engine/device/tegra.c
@@ -23,6 +23,10 @@
#ifdef CONFIG_NOUVEAU_PLATFORM_DRIVER
#include "priv.h"
+#if IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU)
+#include <asm/dma-iommu.h>
+#endif
+
static int
nvkm_device_tegra_power_up(struct nvkm_device_tegra *tdev)
{
@@ -95,6 +99,15 @@ nvkm_device_tegra_probe_iommu(struct nvkm_device_tegra *tdev)
unsigned long pgsize_bitmap;
int ret;
+#if IS_ENABLED(CONFIG_ARM_DMA_USE_IOMMU)
+ if (dev->archdata.mapping) {
+ struct dma_iommu_mapping *mapping = to_dma_iommu_mapping(dev);
+
+ arm_iommu_detach_device(dev);
+ arm_iommu_release_mapping(mapping);
+ }
+#endif
+
if (!tdev->func->iommu_bit)
return;
diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/devinit/gm200.c b/drivers/gpu/drm/nouveau/nvkm/subdev/devinit/gm200.c
index a410c0d..6a1b81e 100644
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/devinit/gm200.c
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/devinit/gm200.c
@@ -161,7 +161,8 @@ gm200_devinit_post(struct nvkm_devinit *base, bool post)
}
/* load and execute some other ucode image (bios therm?) */
- return pmu_load(init, 0x01, post, NULL, NULL);
+ pmu_load(init, 0x01, post, NULL, NULL);
+ return 0;
}
static const struct nvkm_devinit_func
diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gf100.fuc3.h b/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gf100.fuc3.h
index e2faccf..d66e0e7 100644
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gf100.fuc3.h
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gf100.fuc3.h
@@ -46,8 +46,8 @@
0x00000000,
0x00000000,
0x584d454d,
- 0x00000756,
- 0x00000748,
+ 0x00000754,
+ 0x00000746,
0x00000000,
0x00000000,
0x00000000,
@@ -68,8 +68,8 @@
0x00000000,
0x00000000,
0x46524550,
- 0x0000075a,
0x00000758,
+ 0x00000756,
0x00000000,
0x00000000,
0x00000000,
@@ -90,8 +90,8 @@
0x00000000,
0x00000000,
0x5f433249,
- 0x00000b8a,
- 0x00000a2d,
+ 0x00000b88,
+ 0x00000a2b,
0x00000000,
0x00000000,
0x00000000,
@@ -112,8 +112,8 @@
0x00000000,
0x00000000,
0x54534554,
- 0x00000bb3,
- 0x00000b8c,
+ 0x00000bb1,
+ 0x00000b8a,
0x00000000,
0x00000000,
0x00000000,
@@ -134,8 +134,8 @@
0x00000000,
0x00000000,
0x454c4449,
- 0x00000bbf,
0x00000bbd,
+ 0x00000bbb,
0x00000000,
0x00000000,
0x00000000,
@@ -236,19 +236,19 @@
0x000005d3,
0x00000003,
0x00000002,
- 0x0000069d,
+ 0x0000069b,
0x00040004,
0x00000000,
- 0x000006b9,
+ 0x000006b7,
0x00010005,
0x00000000,
- 0x000006d6,
+ 0x000006d4,
0x00010006,
0x00000000,
0x0000065b,
0x00000007,
0x00000000,
- 0x000006e1,
+ 0x000006df,
/* 0x03c4: memx_func_tail */
/* 0x03c4: memx_ts_start */
0x00000000,
@@ -1372,432 +1372,432 @@ uint32_t gf100_pmu_code[] = {
/* 0x065b: memx_func_wait_vblank */
0x9800f840,
0x66b00016,
- 0x130bf400,
+ 0x120bf400,
0xf40166b0,
0x0ef4060b,
/* 0x066d: memx_func_wait_vblank_head1 */
- 0x2077f12e,
- 0x070ef400,
-/* 0x0674: memx_func_wait_vblank_head0 */
- 0x000877f1,
-/* 0x0678: memx_func_wait_vblank_0 */
- 0x07c467f1,
- 0xcf0664b6,
- 0x67fd0066,
- 0xf31bf404,
-/* 0x0688: memx_func_wait_vblank_1 */
- 0x07c467f1,
- 0xcf0664b6,
- 0x67fd0066,
- 0xf30bf404,
-/* 0x0698: memx_func_wait_vblank_fini */
- 0xf80410b6,
-/* 0x069d: memx_func_wr32 */
- 0x00169800,
- 0xb6011598,
- 0x60f90810,
- 0xd0fc50f9,
- 0x21f4e0fc,
- 0x0242b640,
- 0xf8e91bf4,
-/* 0x06b9: memx_func_wait */
- 0x2c87f000,
- 0xcf0684b6,
- 0x1e980088,
- 0x011d9800,
- 0x98021c98,
- 0x10b6031b,
- 0xa321f410,
-/* 0x06d6: memx_func_delay */
- 0x1e9800f8,
- 0x0410b600,
- 0xf87e21f4,
-/* 0x06e1: memx_func_train */
-/* 0x06e3: memx_exec */
- 0xf900f800,
- 0xb9d0f9e0,
- 0xb2b902c1,
-/* 0x06ed: memx_exec_next */
- 0x00139802,
- 0xe70410b6,
- 0xe701f034,
- 0xb601e033,
- 0x30f00132,
- 0xde35980c,
- 0x12b855f9,
- 0xe41ef406,
- 0x98f10b98,
- 0xcbbbf20c,
- 0xc4b7f102,
- 0x06b4b607,
- 0xfc00bbcf,
- 0xf5e0fcd0,
- 0xf8033621,
-/* 0x0729: memx_info */
- 0x01c67000,
-/* 0x072f: memx_info_data */
- 0xf10e0bf4,
- 0xf103ccc7,
- 0xf40800b7,
-/* 0x073a: memx_info_train */
- 0xc7f10b0e,
- 0xb7f10bcc,
-/* 0x0742: memx_info_send */
- 0x21f50100,
- 0x00f80336,
-/* 0x0748: memx_recv */
- 0xf401d6b0,
- 0xd6b0980b,
- 0xd80bf400,
-/* 0x0756: memx_init */
+ 0x2077f02c,
+/* 0x0673: memx_func_wait_vblank_head0 */
+ 0xf0060ef4,
+/* 0x0676: memx_func_wait_vblank_0 */
+ 0x67f10877,
+ 0x64b607c4,
+ 0x0066cf06,
+ 0xf40467fd,
+/* 0x0686: memx_func_wait_vblank_1 */
+ 0x67f1f31b,
+ 0x64b607c4,
+ 0x0066cf06,
+ 0xf40467fd,
+/* 0x0696: memx_func_wait_vblank_fini */
+ 0x10b6f30b,
+/* 0x069b: memx_func_wr32 */
+ 0x9800f804,
+ 0x15980016,
+ 0x0810b601,
+ 0x50f960f9,
+ 0xe0fcd0fc,
+ 0xb64021f4,
+ 0x1bf40242,
+/* 0x06b7: memx_func_wait */
+ 0xf000f8e9,
+ 0x84b62c87,
+ 0x0088cf06,
+ 0x98001e98,
+ 0x1c98011d,
+ 0x031b9802,
+ 0xf41010b6,
+ 0x00f8a321,
+/* 0x06d4: memx_func_delay */
+ 0xb6001e98,
+ 0x21f40410,
+/* 0x06df: memx_func_train */
+ 0xf800f87e,
+/* 0x06e1: memx_exec */
+ 0xf9e0f900,
+ 0x02c1b9d0,
+/* 0x06eb: memx_exec_next */
+ 0x9802b2b9,
+ 0x10b60013,
+ 0xf034e704,
+ 0xe033e701,
+ 0x0132b601,
+ 0x980c30f0,
+ 0x55f9de35,
+ 0xf40612b8,
+ 0x0b98e41e,
+ 0xf20c98f1,
+ 0xf102cbbb,
+ 0xb607c4b7,
+ 0xbbcf06b4,
+ 0xfcd0fc00,
+ 0x3621f5e0,
+/* 0x0727: memx_info */
+ 0x7000f803,
+ 0x0bf401c6,
+/* 0x072d: memx_info_data */
+ 0xccc7f10e,
+ 0x00b7f103,
+ 0x0b0ef408,
+/* 0x0738: memx_info_train */
+ 0x0bccc7f1,
+ 0x0100b7f1,
+/* 0x0740: memx_info_send */
+ 0x033621f5,
+/* 0x0746: memx_recv */
+ 0xd6b000f8,
+ 0x980bf401,
+ 0xf400d6b0,
+ 0x00f8d80b,
+/* 0x0754: memx_init */
+/* 0x0756: perf_recv */
0x00f800f8,
-/* 0x0758: perf_recv */
-/* 0x075a: perf_init */
- 0x00f800f8,
-/* 0x075c: i2c_drive_scl */
- 0xf40036b0,
- 0x07f1110b,
- 0x04b607e0,
- 0x0001d006,
- 0x00f804bd,
-/* 0x0770: i2c_drive_scl_lo */
- 0x07e407f1,
- 0xd00604b6,
- 0x04bd0001,
-/* 0x077e: i2c_drive_sda */
+/* 0x0758: perf_init */
+/* 0x075a: i2c_drive_scl */
0x36b000f8,
0x110bf400,
0x07e007f1,
0xd00604b6,
- 0x04bd0002,
-/* 0x0792: i2c_drive_sda_lo */
+ 0x04bd0001,
+/* 0x076e: i2c_drive_scl_lo */
0x07f100f8,
0x04b607e4,
+ 0x0001d006,
+ 0x00f804bd,
+/* 0x077c: i2c_drive_sda */
+ 0xf40036b0,
+ 0x07f1110b,
+ 0x04b607e0,
0x0002d006,
0x00f804bd,
-/* 0x07a0: i2c_sense_scl */
- 0xf10132f4,
- 0xb607c437,
- 0x33cf0634,
- 0x0431fd00,
- 0xf4060bf4,
-/* 0x07b6: i2c_sense_scl_done */
- 0x00f80131,
-/* 0x07b8: i2c_sense_sda */
- 0xf10132f4,
- 0xb607c437,
- 0x33cf0634,
- 0x0432fd00,
- 0xf4060bf4,
-/* 0x07ce: i2c_sense_sda_done */
- 0x00f80131,
-/* 0x07d0: i2c_raise_scl */
- 0x47f140f9,
- 0x37f00898,
- 0x5c21f501,
-/* 0x07dd: i2c_raise_scl_wait */
+/* 0x0790: i2c_drive_sda_lo */
+ 0x07e407f1,
+ 0xd00604b6,
+ 0x04bd0002,
+/* 0x079e: i2c_sense_scl */
+ 0x32f400f8,
+ 0xc437f101,
+ 0x0634b607,
+ 0xfd0033cf,
+ 0x0bf40431,
+ 0x0131f406,
+/* 0x07b4: i2c_sense_scl_done */
+/* 0x07b6: i2c_sense_sda */
+ 0x32f400f8,
+ 0xc437f101,
+ 0x0634b607,
+ 0xfd0033cf,
+ 0x0bf40432,
+ 0x0131f406,
+/* 0x07cc: i2c_sense_sda_done */
+/* 0x07ce: i2c_raise_scl */
+ 0x40f900f8,
+ 0x089847f1,
+ 0xf50137f0,
+/* 0x07db: i2c_raise_scl_wait */
+ 0xf1075a21,
+ 0xf403e8e7,
+ 0x21f57e21,
+ 0x01f4079e,
+ 0x0142b609,
+/* 0x07ef: i2c_raise_scl_done */
+ 0xfcef1bf4,
+/* 0x07f3: i2c_start */
+ 0xf500f840,
+ 0xf4079e21,
+ 0x21f50d11,
+ 0x11f407b6,
+ 0x300ef406,
+/* 0x0804: i2c_start_rep */
+ 0xf50037f0,
+ 0xf0075a21,
+ 0x21f50137,
+ 0x76bb077c,
+ 0x0465b600,
+ 0x659450f9,
+ 0x0256bb04,
+ 0x75fd50bd,
+ 0xf550fc04,
+ 0xb607ce21,
+ 0x11f40464,
+/* 0x0831: i2c_start_send */
+ 0x0037f01f,
+ 0x077c21f5,
+ 0x1388e7f1,
+ 0xf07e21f4,
+ 0x21f50037,
+ 0xe7f1075a,
+ 0x21f41388,
+/* 0x084d: i2c_start_out */
+/* 0x084f: i2c_stop */
+ 0xf000f87e,
+ 0x21f50037,
+ 0x37f0075a,
+ 0x7c21f500,
0xe8e7f107,
0x7e21f403,
- 0x07a021f5,
- 0xb60901f4,
- 0x1bf40142,
-/* 0x07f1: i2c_raise_scl_done */
- 0xf840fcef,
-/* 0x07f5: i2c_start */
- 0xa021f500,
- 0x0d11f407,
- 0x07b821f5,
- 0xf40611f4,
-/* 0x0806: i2c_start_rep */
- 0x37f0300e,
- 0x5c21f500,
- 0x0137f007,
- 0x077e21f5,
- 0xb60076bb,
- 0x50f90465,
- 0xbb046594,
- 0x50bd0256,
- 0xfc0475fd,
- 0xd021f550,
- 0x0464b607,
-/* 0x0833: i2c_start_send */
- 0xf01f11f4,
- 0x21f50037,
- 0xe7f1077e,
- 0x21f41388,
- 0x0037f07e,
- 0x075c21f5,
- 0x1388e7f1,
-/* 0x084f: i2c_start_out */
- 0xf87e21f4,
-/* 0x0851: i2c_stop */
- 0x0037f000,
- 0x075c21f5,
- 0xf50037f0,
- 0xf1077e21,
- 0xf403e8e7,
- 0x37f07e21,
- 0x5c21f501,
- 0x88e7f107,
- 0x7e21f413,
0xf50137f0,
- 0xf1077e21,
+ 0xf1075a21,
0xf41388e7,
- 0x00f87e21,
-/* 0x0884: i2c_bitw */
- 0x077e21f5,
- 0x03e8e7f1,
- 0xbb7e21f4,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x07d021f5,
- 0xf40464b6,
- 0xe7f11811,
- 0x21f41388,
- 0x0037f07e,
- 0x075c21f5,
- 0x1388e7f1,
-/* 0x08c3: i2c_bitw_out */
- 0xf87e21f4,
-/* 0x08c5: i2c_bitr */
- 0x0137f000,
- 0x077e21f5,
- 0x03e8e7f1,
- 0xbb7e21f4,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x07d021f5,
- 0xf40464b6,
- 0x21f51b11,
- 0x37f007b8,
- 0x5c21f500,
+ 0x37f07e21,
+ 0x7c21f501,
0x88e7f107,
0x7e21f413,
- 0xf4013cf0,
-/* 0x090a: i2c_bitr_done */
- 0x00f80131,
-/* 0x090c: i2c_get_byte */
- 0xf00057f0,
-/* 0x0912: i2c_get_byte_next */
- 0x54b60847,
- 0x0076bb01,
- 0xf90465b6,
- 0x04659450,
- 0xbd0256bb,
- 0x0475fd50,
- 0x21f550fc,
- 0x64b608c5,
- 0x2b11f404,
- 0xb60553fd,
- 0x1bf40142,
- 0x0137f0d8,
- 0xb60076bb,
- 0x50f90465,
- 0xbb046594,
- 0x50bd0256,
- 0xfc0475fd,
- 0x8421f550,
- 0x0464b608,
-/* 0x095c: i2c_get_byte_done */
-/* 0x095e: i2c_put_byte */
- 0x47f000f8,
-/* 0x0961: i2c_put_byte_next */
- 0x0142b608,
- 0xbb3854ff,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x088421f5,
- 0xf40464b6,
- 0x46b03411,
- 0xd81bf400,
- 0xb60076bb,
- 0x50f90465,
- 0xbb046594,
- 0x50bd0256,
- 0xfc0475fd,
- 0xc521f550,
- 0x0464b608,
- 0xbb0f11f4,
- 0x36b00076,
- 0x061bf401,
-/* 0x09b7: i2c_put_byte_done */
- 0xf80132f4,
-/* 0x09b9: i2c_addr */
- 0x0076bb00,
- 0xf90465b6,
- 0x04659450,
- 0xbd0256bb,
- 0x0475fd50,
- 0x21f550fc,
- 0x64b607f5,
- 0x2911f404,
- 0x012ec3e7,
- 0xfd0134b6,
- 0x76bb0553,
- 0x0465b600,
- 0x659450f9,
- 0x0256bb04,
- 0x75fd50bd,
- 0xf550fc04,
- 0xb6095e21,
-/* 0x09fe: i2c_addr_done */
- 0x00f80464,
-/* 0x0a00: i2c_acquire_addr */
- 0xb6f8cec7,
- 0xe0b702e4,
- 0xee980d1c,
-/* 0x0a0f: i2c_acquire */
- 0xf500f800,
- 0xf40a0021,
- 0xd9f00421,
- 0x4021f403,
-/* 0x0a1e: i2c_release */
+/* 0x0882: i2c_bitw */
0x21f500f8,
- 0x21f40a00,
- 0x03daf004,
- 0xf84021f4,
-/* 0x0a2d: i2c_recv */
- 0x0132f400,
- 0xb6f8c1c7,
- 0x16b00214,
- 0x3a1ff528,
- 0xf413a001,
- 0x0032980c,
- 0x0ccc13a0,
- 0xf4003198,
- 0xd0f90231,
- 0xd0f9e0f9,
- 0x000067f1,
- 0x100063f1,
- 0xbb016792,
+ 0xe7f1077c,
+ 0x21f403e8,
+ 0x0076bb7e,
+ 0xf90465b6,
+ 0x04659450,
+ 0xbd0256bb,
+ 0x0475fd50,
+ 0x21f550fc,
+ 0x64b607ce,
+ 0x1811f404,
+ 0x1388e7f1,
+ 0xf07e21f4,
+ 0x21f50037,
+ 0xe7f1075a,
+ 0x21f41388,
+/* 0x08c1: i2c_bitw_out */
+/* 0x08c3: i2c_bitr */
+ 0xf000f87e,
+ 0x21f50137,
+ 0xe7f1077c,
+ 0x21f403e8,
+ 0x0076bb7e,
+ 0xf90465b6,
+ 0x04659450,
+ 0xbd0256bb,
+ 0x0475fd50,
+ 0x21f550fc,
+ 0x64b607ce,
+ 0x1b11f404,
+ 0x07b621f5,
+ 0xf50037f0,
+ 0xf1075a21,
+ 0xf41388e7,
+ 0x3cf07e21,
+ 0x0131f401,
+/* 0x0908: i2c_bitr_done */
+/* 0x090a: i2c_get_byte */
+ 0x57f000f8,
+ 0x0847f000,
+/* 0x0910: i2c_get_byte_next */
+ 0xbb0154b6,
0x65b60076,
0x9450f904,
0x56bb0465,
0xfd50bd02,
0x50fc0475,
- 0x0a0f21f5,
- 0xfc0464b6,
- 0x00d6b0d0,
- 0x00b31bf5,
- 0xbb0057f0,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x09b921f5,
- 0xf50464b6,
- 0xc700d011,
- 0x76bbe0c5,
+ 0x08c321f5,
+ 0xf40464b6,
+ 0x53fd2b11,
+ 0x0142b605,
+ 0xf0d81bf4,
+ 0x76bb0137,
0x0465b600,
0x659450f9,
0x0256bb04,
0x75fd50bd,
0xf550fc04,
- 0xb6095e21,
- 0x11f50464,
- 0x57f000ad,
+ 0xb6088221,
+/* 0x095a: i2c_get_byte_done */
+ 0x00f80464,
+/* 0x095c: i2c_put_byte */
+/* 0x095f: i2c_put_byte_next */
+ 0xb60847f0,
+ 0x54ff0142,
+ 0x0076bb38,
+ 0xf90465b6,
+ 0x04659450,
+ 0xbd0256bb,
+ 0x0475fd50,
+ 0x21f550fc,
+ 0x64b60882,
+ 0x3411f404,
+ 0xf40046b0,
+ 0x76bbd81b,
+ 0x0465b600,
+ 0x659450f9,
+ 0x0256bb04,
+ 0x75fd50bd,
+ 0xf550fc04,
+ 0xb608c321,
+ 0x11f40464,
+ 0x0076bb0f,
+ 0xf40136b0,
+ 0x32f4061b,
+/* 0x09b5: i2c_put_byte_done */
+/* 0x09b7: i2c_addr */
+ 0xbb00f801,
+ 0x65b60076,
+ 0x9450f904,
+ 0x56bb0465,
+ 0xfd50bd02,
+ 0x50fc0475,
+ 0x07f321f5,
+ 0xf40464b6,
+ 0xc3e72911,
+ 0x34b6012e,
+ 0x0553fd01,
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0x5c21f550,
+ 0x0464b609,
+/* 0x09fc: i2c_addr_done */
+/* 0x09fe: i2c_acquire_addr */
+ 0xcec700f8,
+ 0x02e4b6f8,
+ 0x0d1ce0b7,
+ 0xf800ee98,
+/* 0x0a0d: i2c_acquire */
+ 0xfe21f500,
+ 0x0421f409,
+ 0xf403d9f0,
+ 0x00f84021,
+/* 0x0a1c: i2c_release */
+ 0x09fe21f5,
+ 0xf00421f4,
+ 0x21f403da,
+/* 0x0a2b: i2c_recv */
+ 0xf400f840,
+ 0xc1c70132,
+ 0x0214b6f8,
+ 0xf52816b0,
+ 0xa0013a1f,
+ 0x980cf413,
+ 0x13a00032,
+ 0x31980ccc,
+ 0x0231f400,
+ 0xe0f9d0f9,
+ 0x67f1d0f9,
+ 0x63f10000,
+ 0x67921000,
0x0076bb01,
0xf90465b6,
0x04659450,
0xbd0256bb,
0x0475fd50,
0x21f550fc,
- 0x64b609b9,
- 0x8a11f504,
+ 0x64b60a0d,
+ 0xb0d0fc04,
+ 0x1bf500d6,
+ 0x57f000b3,
0x0076bb00,
0xf90465b6,
0x04659450,
0xbd0256bb,
0x0475fd50,
0x21f550fc,
- 0x64b6090c,
- 0x6a11f404,
- 0xbbe05bcb,
+ 0x64b609b7,
+ 0xd011f504,
+ 0xe0c5c700,
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0x5c21f550,
+ 0x0464b609,
+ 0x00ad11f5,
+ 0xbb0157f0,
0x65b60076,
0x9450f904,
0x56bb0465,
0xfd50bd02,
0x50fc0475,
- 0x085121f5,
- 0xb90464b6,
- 0x74bd025b,
-/* 0x0b33: i2c_recv_not_rd08 */
- 0xb0430ef4,
- 0x1bf401d6,
- 0x0057f03d,
- 0x09b921f5,
- 0xc73311f4,
- 0x21f5e0c5,
- 0x11f4095e,
- 0x0057f029,
- 0x09b921f5,
- 0xc71f11f4,
- 0x21f5e0b5,
- 0x11f4095e,
- 0x5121f515,
- 0xc774bd08,
- 0x1bf408c5,
- 0x0232f409,
-/* 0x0b73: i2c_recv_not_wr08 */
-/* 0x0b73: i2c_recv_done */
- 0xc7030ef4,
- 0x21f5f8ce,
- 0xe0fc0a1e,
- 0x12f4d0fc,
- 0x027cb90a,
- 0x033621f5,
-/* 0x0b88: i2c_recv_exit */
-/* 0x0b8a: i2c_init */
- 0x00f800f8,
-/* 0x0b8c: test_recv */
- 0x05d817f1,
- 0xcf0614b6,
- 0x10b60011,
- 0xd807f101,
- 0x0604b605,
- 0xbd0001d0,
- 0x00e7f104,
- 0x4fe3f1d9,
- 0x5621f513,
-/* 0x0bb3: test_init */
- 0xf100f802,
- 0xf50800e7,
- 0xf8025621,
-/* 0x0bbd: idle_recv */
-/* 0x0bbf: idle */
- 0xf400f800,
- 0x17f10031,
- 0x14b605d4,
+ 0x09b721f5,
+ 0xf50464b6,
+ 0xbb008a11,
+ 0x65b60076,
+ 0x9450f904,
+ 0x56bb0465,
+ 0xfd50bd02,
+ 0x50fc0475,
+ 0x090a21f5,
+ 0xf40464b6,
+ 0x5bcb6a11,
+ 0x0076bbe0,
+ 0xf90465b6,
+ 0x04659450,
+ 0xbd0256bb,
+ 0x0475fd50,
+ 0x21f550fc,
+ 0x64b6084f,
+ 0x025bb904,
+ 0x0ef474bd,
+/* 0x0b31: i2c_recv_not_rd08 */
+ 0x01d6b043,
+ 0xf03d1bf4,
+ 0x21f50057,
+ 0x11f409b7,
+ 0xe0c5c733,
+ 0x095c21f5,
+ 0xf02911f4,
+ 0x21f50057,
+ 0x11f409b7,
+ 0xe0b5c71f,
+ 0x095c21f5,
+ 0xf51511f4,
+ 0xbd084f21,
+ 0x08c5c774,
+ 0xf4091bf4,
+ 0x0ef40232,
+/* 0x0b71: i2c_recv_not_wr08 */
+/* 0x0b71: i2c_recv_done */
+ 0xf8cec703,
+ 0x0a1c21f5,
+ 0xd0fce0fc,
+ 0xb90a12f4,
+ 0x21f5027c,
+/* 0x0b86: i2c_recv_exit */
+ 0x00f80336,
+/* 0x0b88: i2c_init */
+/* 0x0b8a: test_recv */
+ 0x17f100f8,
+ 0x14b605d8,
0x0011cf06,
0xf10110b6,
- 0xb605d407,
+ 0xb605d807,
0x01d00604,
-/* 0x0bdb: idle_loop */
- 0xf004bd00,
- 0x32f45817,
-/* 0x0be1: idle_proc */
-/* 0x0be1: idle_proc_exec */
- 0xb910f902,
- 0x21f5021e,
- 0x10fc033f,
- 0xf40911f4,
- 0x0ef40231,
-/* 0x0bf5: idle_proc_next */
- 0x5810b6ef,
- 0xf4061fb8,
- 0x02f4e61b,
- 0x0028f4dd,
- 0x00bb0ef4,
+ 0xf104bd00,
+ 0xf1d900e7,
+ 0xf5134fe3,
+ 0xf8025621,
+/* 0x0bb1: test_init */
+ 0x00e7f100,
+ 0x5621f508,
+/* 0x0bbb: idle_recv */
+ 0xf800f802,
+/* 0x0bbd: idle */
+ 0x0031f400,
+ 0x05d417f1,
+ 0xcf0614b6,
+ 0x10b60011,
+ 0xd407f101,
+ 0x0604b605,
+ 0xbd0001d0,
+/* 0x0bd9: idle_loop */
+ 0x5817f004,
+/* 0x0bdf: idle_proc */
+/* 0x0bdf: idle_proc_exec */
+ 0xf90232f4,
+ 0x021eb910,
+ 0x033f21f5,
+ 0x11f410fc,
+ 0x0231f409,
+/* 0x0bf3: idle_proc_next */
+ 0xb6ef0ef4,
+ 0x1fb85810,
+ 0xe61bf406,
+ 0xf4dd02f4,
+ 0x0ef40028,
+ 0x000000bb,
0x00000000,
0x00000000,
0x00000000,
diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gk208.fuc5.h b/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gk208.fuc5.h
index 3c731ff..9582224 100644
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gk208.fuc5.h
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gk208.fuc5.h
@@ -46,8 +46,8 @@
0x00000000,
0x00000000,
0x584d454d,
- 0x000005f3,
- 0x000005e5,
+ 0x000005ee,
+ 0x000005e0,
0x00000000,
0x00000000,
0x00000000,
@@ -68,8 +68,8 @@
0x00000000,
0x00000000,
0x46524550,
- 0x000005f7,
- 0x000005f5,
+ 0x000005f2,
+ 0x000005f0,
0x00000000,
0x00000000,
0x00000000,
@@ -90,8 +90,8 @@
0x00000000,
0x00000000,
0x5f433249,
- 0x000009f8,
- 0x000008a2,
+ 0x000009f3,
+ 0x0000089d,
0x00000000,
0x00000000,
0x00000000,
@@ -112,8 +112,8 @@
0x00000000,
0x00000000,
0x54534554,
- 0x00000a16,
- 0x000009fa,
+ 0x00000a11,
+ 0x000009f5,
0x00000000,
0x00000000,
0x00000000,
@@ -134,8 +134,8 @@
0x00000000,
0x00000000,
0x454c4449,
- 0x00000a21,
- 0x00000a1f,
+ 0x00000a1c,
+ 0x00000a1a,
0x00000000,
0x00000000,
0x00000000,
@@ -233,22 +233,22 @@
/* 0x037c: memx_func_next */
0x00000002,
0x00000000,
- 0x000004cf,
+ 0x000004cc,
0x00000003,
0x00000002,
- 0x00000546,
+ 0x00000541,
0x00040004,
0x00000000,
- 0x00000563,
+ 0x0000055e,
0x00010005,
0x00000000,
- 0x0000057d,
+ 0x00000578,
0x00010006,
0x00000000,
- 0x00000541,
+ 0x0000053c,
0x00000007,
0x00000000,
- 0x00000589,
+ 0x00000584,
/* 0x03c4: memx_func_tail */
/* 0x03c4: memx_ts_start */
0x00000000,
@@ -1238,454 +1238,454 @@ uint32_t gk208_pmu_code[] = {
0x0001f604,
0x00f804bd,
/* 0x045c: memx_func_enter */
- 0x162067f1,
- 0xf55d77f1,
- 0x047e6eb2,
- 0xd8b20000,
- 0xf90487fd,
- 0xfc80f960,
- 0x7ee0fcd0,
- 0x0700002d,
- 0x7e6eb2fe,
+ 0x47162046,
+ 0x6eb2f55d,
+ 0x0000047e,
+ 0x87fdd8b2,
+ 0xf960f904,
+ 0xfcd0fc80,
+ 0x002d7ee0,
+ 0xb2fe0700,
+ 0x00047e6e,
+ 0xfdd8b200,
+ 0x60f90487,
+ 0xd0fc80f9,
+ 0x2d7ee0fc,
+ 0xf0460000,
+ 0x7e6eb226,
0xb2000004,
0x0487fdd8,
0x80f960f9,
0xe0fcd0fc,
0x00002d7e,
- 0x26f067f1,
- 0x047e6eb2,
- 0xd8b20000,
- 0xf90487fd,
- 0xfc80f960,
- 0x7ee0fcd0,
- 0x0600002d,
- 0x07e04004,
- 0xbd0006f6,
-/* 0x04b9: memx_func_enter_wait */
- 0x07c04604,
- 0xf00066cf,
- 0x0bf40464,
- 0xcf2c06f7,
- 0x06b50066,
-/* 0x04cf: memx_func_leave */
- 0x0600f8f1,
- 0x0066cf2c,
- 0x06f206b5,
- 0x07e44004,
- 0xbd0006f6,
-/* 0x04e1: memx_func_leave_wait */
- 0x07c04604,
- 0xf00066cf,
- 0x1bf40464,
- 0xf067f1f7,
+ 0xe0400406,
+ 0x0006f607,
+/* 0x04b6: memx_func_enter_wait */
+ 0xc04604bd,
+ 0x0066cf07,
+ 0xf40464f0,
+ 0x2c06f70b,
+ 0xb50066cf,
+ 0x00f8f106,
+/* 0x04cc: memx_func_leave */
+ 0x66cf2c06,
+ 0xf206b500,
+ 0xe4400406,
+ 0x0006f607,
+/* 0x04de: memx_func_leave_wait */
+ 0xc04604bd,
+ 0x0066cf07,
+ 0xf40464f0,
+ 0xf046f71b,
0xb2010726,
0x00047e6e,
0xfdd8b200,
0x60f90587,
0xd0fc80f9,
0x2d7ee0fc,
- 0x67f10000,
- 0x6eb21620,
- 0x0000047e,
- 0x87fdd8b2,
- 0xf960f905,
- 0xfcd0fc80,
- 0x002d7ee0,
- 0x0aa24700,
- 0x047e6eb2,
- 0xd8b20000,
- 0xf90587fd,
- 0xfc80f960,
- 0x7ee0fcd0,
- 0xf800002d,
-/* 0x0541: memx_func_wait_vblank */
- 0x0410b600,
-/* 0x0546: memx_func_wr32 */
- 0x169800f8,
- 0x01159800,
- 0xf90810b6,
- 0xfc50f960,
- 0x7ee0fcd0,
- 0xb600002d,
- 0x1bf40242,
-/* 0x0563: memx_func_wait */
- 0x0800f8e8,
- 0x0088cf2c,
- 0x98001e98,
- 0x1c98011d,
- 0x031b9802,
- 0x7e1010b6,
- 0xf8000074,
-/* 0x057d: memx_func_delay */
- 0x001e9800,
- 0x7e0410b6,
- 0xf8000058,
-/* 0x0589: memx_func_train */
-/* 0x058b: memx_exec */
- 0xf900f800,
- 0xb2d0f9e0,
-/* 0x0593: memx_exec_next */
- 0x98b2b2c1,
- 0x10b60013,
- 0xf034e704,
- 0xe033e701,
- 0x0132b601,
- 0x980c30f0,
- 0x55f9de35,
- 0x1ef412a6,
- 0xf10b98e5,
- 0xbbf20c98,
- 0xc44b02cb,
- 0x00bbcf07,
+ 0x20460000,
+ 0x7e6eb216,
+ 0xb2000004,
+ 0x0587fdd8,
+ 0x80f960f9,
0xe0fcd0fc,
- 0x00029f7e,
-/* 0x05ca: memx_info */
- 0xc67000f8,
- 0x0c0bf401,
-/* 0x05d0: memx_info_data */
- 0x4b03cc4c,
- 0x0ef40800,
-/* 0x05d9: memx_info_train */
- 0x0bcc4c09,
-/* 0x05df: memx_info_send */
- 0x7e01004b,
- 0xf800029f,
-/* 0x05e5: memx_recv */
- 0x01d6b000,
- 0xb0a30bf4,
- 0x0bf400d6,
-/* 0x05f3: memx_init */
- 0xf800f8dc,
-/* 0x05f5: perf_recv */
-/* 0x05f7: perf_init */
- 0xf800f800,
-/* 0x05f9: i2c_drive_scl */
- 0x0036b000,
- 0x400d0bf4,
- 0x01f607e0,
- 0xf804bd00,
-/* 0x0609: i2c_drive_scl_lo */
- 0x07e44000,
- 0xbd0001f6,
-/* 0x0613: i2c_drive_sda */
- 0xb000f804,
- 0x0bf40036,
- 0x07e0400d,
- 0xbd0002f6,
-/* 0x0623: i2c_drive_sda_lo */
- 0x4000f804,
- 0x02f607e4,
- 0xf804bd00,
-/* 0x062d: i2c_sense_scl */
- 0x0132f400,
- 0xcf07c443,
- 0x31fd0033,
- 0x060bf404,
-/* 0x063f: i2c_sense_scl_done */
- 0xf80131f4,
-/* 0x0641: i2c_sense_sda */
- 0x0132f400,
- 0xcf07c443,
- 0x32fd0033,
- 0x060bf404,
-/* 0x0653: i2c_sense_sda_done */
- 0xf80131f4,
-/* 0x0655: i2c_raise_scl */
- 0x4440f900,
- 0x01030898,
- 0x0005f97e,
-/* 0x0660: i2c_raise_scl_wait */
- 0x7e03e84e,
- 0x7e000058,
- 0xf400062d,
- 0x42b60901,
- 0xef1bf401,
-/* 0x0674: i2c_raise_scl_done */
- 0x00f840fc,
-/* 0x0678: i2c_start */
- 0x00062d7e,
- 0x7e0d11f4,
- 0xf4000641,
- 0x0ef40611,
-/* 0x0689: i2c_start_rep */
- 0x7e00032e,
- 0x030005f9,
- 0x06137e01,
- 0x0076bb00,
- 0xf90465b6,
- 0x04659450,
- 0xbd0256bb,
- 0x0475fd50,
- 0x557e50fc,
- 0x64b60006,
- 0x1d11f404,
-/* 0x06b4: i2c_start_send */
- 0x137e0003,
- 0x884e0006,
- 0x00587e13,
- 0x7e000300,
- 0x4e0005f9,
- 0x587e1388,
-/* 0x06ce: i2c_start_out */
+ 0x00002d7e,
+ 0xb20aa247,
+ 0x00047e6e,
+ 0xfdd8b200,
+ 0x60f90587,
+ 0xd0fc80f9,
+ 0x2d7ee0fc,
0x00f80000,
-/* 0x06d0: i2c_stop */
- 0xf97e0003,
- 0x00030005,
- 0x0006137e,
- 0x7e03e84e,
- 0x03000058,
- 0x05f97e01,
+/* 0x053c: memx_func_wait_vblank */
+ 0xf80410b6,
+/* 0x0541: memx_func_wr32 */
+ 0x00169800,
+ 0xb6011598,
+ 0x60f90810,
+ 0xd0fc50f9,
+ 0x2d7ee0fc,
+ 0x42b60000,
+ 0xe81bf402,
+/* 0x055e: memx_func_wait */
+ 0x2c0800f8,
+ 0x980088cf,
+ 0x1d98001e,
+ 0x021c9801,
+ 0xb6031b98,
+ 0x747e1010,
+ 0x00f80000,
+/* 0x0578: memx_func_delay */
+ 0xb6001e98,
+ 0x587e0410,
+ 0x00f80000,
+/* 0x0584: memx_func_train */
+/* 0x0586: memx_exec */
+ 0xe0f900f8,
+ 0xc1b2d0f9,
+/* 0x058e: memx_exec_next */
+ 0x1398b2b2,
+ 0x0410b600,
+ 0x01f034e7,
+ 0x01e033e7,
+ 0xf00132b6,
+ 0x35980c30,
+ 0xa655f9de,
+ 0xe51ef412,
+ 0x98f10b98,
+ 0xcbbbf20c,
+ 0x07c44b02,
+ 0xfc00bbcf,
+ 0x7ee0fcd0,
+ 0xf800029f,
+/* 0x05c5: memx_info */
+ 0x01c67000,
+/* 0x05cb: memx_info_data */
+ 0x4c0c0bf4,
+ 0x004b03cc,
+ 0x090ef408,
+/* 0x05d4: memx_info_train */
+ 0x4b0bcc4c,
+/* 0x05da: memx_info_send */
+ 0x9f7e0100,
+ 0x00f80002,
+/* 0x05e0: memx_recv */
+ 0xf401d6b0,
+ 0xd6b0a30b,
+ 0xdc0bf400,
+/* 0x05ee: memx_init */
+ 0x00f800f8,
+/* 0x05f0: perf_recv */
+/* 0x05f2: perf_init */
+ 0x00f800f8,
+/* 0x05f4: i2c_drive_scl */
+ 0xf40036b0,
+ 0xe0400d0b,
+ 0x0001f607,
+ 0x00f804bd,
+/* 0x0604: i2c_drive_scl_lo */
+ 0xf607e440,
+ 0x04bd0001,
+/* 0x060e: i2c_drive_sda */
+ 0x36b000f8,
+ 0x0d0bf400,
+ 0xf607e040,
+ 0x04bd0002,
+/* 0x061e: i2c_drive_sda_lo */
+ 0xe44000f8,
+ 0x0002f607,
+ 0x00f804bd,
+/* 0x0628: i2c_sense_scl */
+ 0x430132f4,
+ 0x33cf07c4,
+ 0x0431fd00,
+ 0xf4060bf4,
+/* 0x063a: i2c_sense_scl_done */
+ 0x00f80131,
+/* 0x063c: i2c_sense_sda */
+ 0x430132f4,
+ 0x33cf07c4,
+ 0x0432fd00,
+ 0xf4060bf4,
+/* 0x064e: i2c_sense_sda_done */
+ 0x00f80131,
+/* 0x0650: i2c_raise_scl */
+ 0x984440f9,
+ 0x7e010308,
+/* 0x065b: i2c_raise_scl_wait */
+ 0x4e0005f4,
+ 0x587e03e8,
+ 0x287e0000,
+ 0x01f40006,
+ 0x0142b609,
+/* 0x066f: i2c_raise_scl_done */
+ 0xfcef1bf4,
+/* 0x0673: i2c_start */
+ 0x7e00f840,
+ 0xf4000628,
+ 0x3c7e0d11,
+ 0x11f40006,
+ 0x2e0ef406,
+/* 0x0684: i2c_start_rep */
+ 0xf47e0003,
+ 0x01030005,
+ 0x00060e7e,
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0x06507e50,
+ 0x0464b600,
+/* 0x06af: i2c_start_send */
+ 0x031d11f4,
+ 0x060e7e00,
0x13884e00,
0x0000587e,
- 0x137e0103,
- 0x884e0006,
- 0x00587e13,
-/* 0x06ff: i2c_bitw */
- 0x7e00f800,
- 0x4e000613,
- 0x587e03e8,
- 0x76bb0000,
- 0x0465b600,
- 0x659450f9,
- 0x0256bb04,
- 0x75fd50bd,
- 0x7e50fc04,
- 0xb6000655,
- 0x11f40464,
- 0x13884e17,
- 0x0000587e,
- 0xf97e0003,
+ 0xf47e0003,
0x884e0005,
0x00587e13,
-/* 0x073d: i2c_bitw_out */
-/* 0x073f: i2c_bitr */
+/* 0x06c9: i2c_start_out */
+/* 0x06cb: i2c_stop */
0x0300f800,
- 0x06137e01,
- 0x03e84e00,
- 0x0000587e,
- 0xb60076bb,
- 0x50f90465,
- 0xbb046594,
- 0x50bd0256,
- 0xfc0475fd,
- 0x06557e50,
- 0x0464b600,
- 0x7e1a11f4,
- 0x03000641,
- 0x05f97e00,
+ 0x05f47e00,
+ 0x7e000300,
+ 0x4e00060e,
+ 0x587e03e8,
+ 0x01030000,
+ 0x0005f47e,
+ 0x7e13884e,
+ 0x03000058,
+ 0x060e7e01,
0x13884e00,
0x0000587e,
- 0xf4013cf0,
-/* 0x0782: i2c_bitr_done */
- 0x00f80131,
-/* 0x0784: i2c_get_byte */
- 0x08040005,
-/* 0x0788: i2c_get_byte_next */
- 0xbb0154b6,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x00073f7e,
- 0xf40464b6,
- 0x53fd2a11,
- 0x0142b605,
- 0x03d81bf4,
- 0x0076bb01,
- 0xf90465b6,
- 0x04659450,
- 0xbd0256bb,
- 0x0475fd50,
- 0xff7e50fc,
- 0x64b60006,
-/* 0x07d1: i2c_get_byte_done */
-/* 0x07d3: i2c_put_byte */
- 0x0400f804,
-/* 0x07d5: i2c_put_byte_next */
- 0x0142b608,
- 0xbb3854ff,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x0006ff7e,
- 0xf40464b6,
- 0x46b03411,
- 0xd81bf400,
- 0xb60076bb,
- 0x50f90465,
- 0xbb046594,
- 0x50bd0256,
- 0xfc0475fd,
- 0x073f7e50,
- 0x0464b600,
- 0xbb0f11f4,
- 0x36b00076,
- 0x061bf401,
-/* 0x082b: i2c_put_byte_done */
- 0xf80132f4,
-/* 0x082d: i2c_addr */
+/* 0x06fa: i2c_bitw */
+ 0x0e7e00f8,
+ 0xe84e0006,
+ 0x00587e03,
0x0076bb00,
0xf90465b6,
0x04659450,
0xbd0256bb,
0x0475fd50,
- 0x787e50fc,
+ 0x507e50fc,
0x64b60006,
- 0x2911f404,
- 0x012ec3e7,
- 0xfd0134b6,
- 0x76bb0553,
- 0x0465b600,
- 0x659450f9,
- 0x0256bb04,
- 0x75fd50bd,
- 0x7e50fc04,
- 0xb60007d3,
-/* 0x0872: i2c_addr_done */
- 0x00f80464,
-/* 0x0874: i2c_acquire_addr */
- 0xb6f8cec7,
- 0xe0b705e4,
- 0x00f8d014,
-/* 0x0880: i2c_acquire */
- 0x0008747e,
- 0x0000047e,
- 0x7e03d9f0,
- 0xf800002d,
-/* 0x0891: i2c_release */
- 0x08747e00,
- 0x00047e00,
- 0x03daf000,
- 0x00002d7e,
-/* 0x08a2: i2c_recv */
- 0x32f400f8,
- 0xf8c1c701,
- 0xb00214b6,
- 0x1ff52816,
- 0x13b80134,
- 0x98000cf4,
- 0x13b80032,
- 0x98000ccc,
- 0x31f40031,
- 0xf9d0f902,
- 0xd6d0f9e0,
- 0x10000000,
- 0xbb016792,
+ 0x1711f404,
+ 0x7e13884e,
+ 0x03000058,
+ 0x05f47e00,
+ 0x13884e00,
+ 0x0000587e,
+/* 0x0738: i2c_bitw_out */
+/* 0x073a: i2c_bitr */
+ 0x010300f8,
+ 0x00060e7e,
+ 0x7e03e84e,
+ 0xbb000058,
0x65b60076,
0x9450f904,
0x56bb0465,
0xfd50bd02,
0x50fc0475,
- 0x0008807e,
- 0xfc0464b6,
- 0x00d6b0d0,
- 0x00b01bf5,
- 0x76bb0005,
- 0x0465b600,
- 0x659450f9,
- 0x0256bb04,
- 0x75fd50bd,
- 0x7e50fc04,
- 0xb600082d,
- 0x11f50464,
- 0xc5c700cc,
- 0x0076bbe0,
- 0xf90465b6,
- 0x04659450,
- 0xbd0256bb,
- 0x0475fd50,
- 0xd37e50fc,
- 0x64b60007,
- 0xa911f504,
- 0xbb010500,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x00082d7e,
- 0xf50464b6,
- 0xbb008711,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x0007847e,
+ 0x0006507e,
0xf40464b6,
- 0x5bcb6711,
- 0x0076bbe0,
+ 0x3c7e1a11,
+ 0x00030006,
+ 0x0005f47e,
+ 0x7e13884e,
+ 0xf0000058,
+ 0x31f4013c,
+/* 0x077d: i2c_bitr_done */
+/* 0x077f: i2c_get_byte */
+ 0x0500f801,
+/* 0x0783: i2c_get_byte_next */
+ 0xb6080400,
+ 0x76bb0154,
+ 0x0465b600,
+ 0x659450f9,
+ 0x0256bb04,
+ 0x75fd50bd,
+ 0x7e50fc04,
+ 0xb600073a,
+ 0x11f40464,
+ 0x0553fd2a,
+ 0xf40142b6,
+ 0x0103d81b,
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0x06fa7e50,
+ 0x0464b600,
+/* 0x07cc: i2c_get_byte_done */
+/* 0x07ce: i2c_put_byte */
+ 0x080400f8,
+/* 0x07d0: i2c_put_byte_next */
+ 0xff0142b6,
+ 0x76bb3854,
+ 0x0465b600,
+ 0x659450f9,
+ 0x0256bb04,
+ 0x75fd50bd,
+ 0x7e50fc04,
+ 0xb60006fa,
+ 0x11f40464,
+ 0x0046b034,
+ 0xbbd81bf4,
+ 0x65b60076,
+ 0x9450f904,
+ 0x56bb0465,
+ 0xfd50bd02,
+ 0x50fc0475,
+ 0x00073a7e,
+ 0xf40464b6,
+ 0x76bb0f11,
+ 0x0136b000,
+ 0xf4061bf4,
+/* 0x0826: i2c_put_byte_done */
+ 0x00f80132,
+/* 0x0828: i2c_addr */
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0x06737e50,
+ 0x0464b600,
+ 0xe72911f4,
+ 0xb6012ec3,
+ 0x53fd0134,
+ 0x0076bb05,
0xf90465b6,
0x04659450,
0xbd0256bb,
0x0475fd50,
- 0xd07e50fc,
- 0x64b60006,
- 0xbd5bb204,
- 0x410ef474,
-/* 0x09a4: i2c_recv_not_rd08 */
- 0xf401d6b0,
- 0x00053b1b,
- 0x00082d7e,
- 0xc73211f4,
- 0xd37ee0c5,
- 0x11f40007,
- 0x7e000528,
- 0xf400082d,
- 0xb5c71f11,
- 0x07d37ee0,
- 0x1511f400,
- 0x0006d07e,
- 0xc5c774bd,
- 0x091bf408,
- 0xf40232f4,
-/* 0x09e2: i2c_recv_not_wr08 */
-/* 0x09e2: i2c_recv_done */
- 0xcec7030e,
- 0x08917ef8,
- 0xfce0fc00,
- 0x0912f4d0,
- 0x9f7e7cb2,
-/* 0x09f6: i2c_recv_exit */
- 0x00f80002,
-/* 0x09f8: i2c_init */
-/* 0x09fa: test_recv */
- 0x584100f8,
- 0x0011cf04,
- 0x400110b6,
- 0x01f60458,
- 0xde04bd00,
- 0x134fd900,
- 0x0001de7e,
-/* 0x0a16: test_init */
- 0x004e00f8,
- 0x01de7e08,
-/* 0x0a1f: idle_recv */
+ 0xce7e50fc,
+ 0x64b60007,
+/* 0x086d: i2c_addr_done */
+/* 0x086f: i2c_acquire_addr */
+ 0xc700f804,
+ 0xe4b6f8ce,
+ 0x14e0b705,
+/* 0x087b: i2c_acquire */
+ 0x7e00f8d0,
+ 0x7e00086f,
+ 0xf0000004,
+ 0x2d7e03d9,
+ 0x00f80000,
+/* 0x088c: i2c_release */
+ 0x00086f7e,
+ 0x0000047e,
+ 0x7e03daf0,
+ 0xf800002d,
+/* 0x089d: i2c_recv */
+ 0x0132f400,
+ 0xb6f8c1c7,
+ 0x16b00214,
+ 0x341ff528,
+ 0xf413b801,
+ 0x3298000c,
+ 0xcc13b800,
+ 0x3198000c,
+ 0x0231f400,
+ 0xe0f9d0f9,
+ 0x00d6d0f9,
+ 0x92100000,
+ 0x76bb0167,
+ 0x0465b600,
+ 0x659450f9,
+ 0x0256bb04,
+ 0x75fd50bd,
+ 0x7e50fc04,
+ 0xb600087b,
+ 0xd0fc0464,
+ 0xf500d6b0,
+ 0x0500b01b,
+ 0x0076bb00,
+ 0xf90465b6,
+ 0x04659450,
+ 0xbd0256bb,
+ 0x0475fd50,
+ 0x287e50fc,
+ 0x64b60008,
+ 0xcc11f504,
+ 0xe0c5c700,
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0x07ce7e50,
+ 0x0464b600,
+ 0x00a911f5,
+ 0x76bb0105,
+ 0x0465b600,
+ 0x659450f9,
+ 0x0256bb04,
+ 0x75fd50bd,
+ 0x7e50fc04,
+ 0xb6000828,
+ 0x11f50464,
+ 0x76bb0087,
+ 0x0465b600,
+ 0x659450f9,
+ 0x0256bb04,
+ 0x75fd50bd,
+ 0x7e50fc04,
+ 0xb600077f,
+ 0x11f40464,
+ 0xe05bcb67,
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0x06cb7e50,
+ 0x0464b600,
+ 0x74bd5bb2,
+/* 0x099f: i2c_recv_not_rd08 */
+ 0xb0410ef4,
+ 0x1bf401d6,
+ 0x7e00053b,
+ 0xf4000828,
+ 0xc5c73211,
+ 0x07ce7ee0,
+ 0x2811f400,
+ 0x287e0005,
+ 0x11f40008,
+ 0xe0b5c71f,
+ 0x0007ce7e,
+ 0x7e1511f4,
+ 0xbd0006cb,
+ 0x08c5c774,
+ 0xf4091bf4,
+ 0x0ef40232,
+/* 0x09dd: i2c_recv_not_wr08 */
+/* 0x09dd: i2c_recv_done */
+ 0xf8cec703,
+ 0x00088c7e,
+ 0xd0fce0fc,
+ 0xb20912f4,
+ 0x029f7e7c,
+/* 0x09f1: i2c_recv_exit */
+/* 0x09f3: i2c_init */
0xf800f800,
-/* 0x0a21: idle */
- 0x0031f400,
- 0xcf045441,
- 0x10b60011,
- 0x04544001,
- 0xbd0001f6,
-/* 0x0a35: idle_loop */
- 0xf4580104,
-/* 0x0a3a: idle_proc */
-/* 0x0a3a: idle_proc_exec */
- 0x10f90232,
- 0xa87e1eb2,
- 0x10fc0002,
- 0xf40911f4,
- 0x0ef40231,
-/* 0x0a4d: idle_proc_next */
- 0x5810b6f0,
- 0x1bf41fa6,
- 0xe002f4e8,
- 0xf40028f4,
- 0x0000c60e,
+/* 0x09f5: test_recv */
+ 0x04584100,
+ 0xb60011cf,
+ 0x58400110,
+ 0x0001f604,
+ 0x00de04bd,
+ 0x7e134fd9,
+ 0xf80001de,
+/* 0x0a11: test_init */
+ 0x08004e00,
+ 0x0001de7e,
+/* 0x0a1a: idle_recv */
+ 0x00f800f8,
+/* 0x0a1c: idle */
+ 0x410031f4,
+ 0x11cf0454,
+ 0x0110b600,
+ 0xf6045440,
+ 0x04bd0001,
+/* 0x0a30: idle_loop */
+ 0x32f45801,
+/* 0x0a35: idle_proc */
+/* 0x0a35: idle_proc_exec */
+ 0xb210f902,
+ 0x02a87e1e,
+ 0xf410fc00,
+ 0x31f40911,
+ 0xf00ef402,
+/* 0x0a48: idle_proc_next */
+ 0xa65810b6,
+ 0xe81bf41f,
+ 0xf4e002f4,
+ 0x0ef40028,
+ 0x000000c6,
+ 0x00000000,
0x00000000,
0x00000000,
0x00000000,
diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gt215.fuc3.h b/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gt215.fuc3.h
index e833418..e29b785 100644
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gt215.fuc3.h
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/gt215.fuc3.h
@@ -46,8 +46,8 @@
0x00000000,
0x00000000,
0x584d454d,
- 0x0000083a,
- 0x0000082c,
+ 0x00000833,
+ 0x00000825,
0x00000000,
0x00000000,
0x00000000,
@@ -68,8 +68,8 @@
0x00000000,
0x00000000,
0x46524550,
- 0x0000083e,
- 0x0000083c,
+ 0x00000837,
+ 0x00000835,
0x00000000,
0x00000000,
0x00000000,
@@ -90,8 +90,8 @@
0x00000000,
0x00000000,
0x5f433249,
- 0x00000c6e,
- 0x00000b11,
+ 0x00000c67,
+ 0x00000b0a,
0x00000000,
0x00000000,
0x00000000,
@@ -112,8 +112,8 @@
0x00000000,
0x00000000,
0x54534554,
- 0x00000c97,
- 0x00000c70,
+ 0x00000c90,
+ 0x00000c69,
0x00000000,
0x00000000,
0x00000000,
@@ -134,8 +134,8 @@
0x00000000,
0x00000000,
0x454c4449,
- 0x00000ca3,
- 0x00000ca1,
+ 0x00000c9c,
+ 0x00000c9a,
0x00000000,
0x00000000,
0x00000000,
@@ -233,22 +233,22 @@
/* 0x037c: memx_func_next */
0x00000002,
0x00000000,
- 0x000005a0,
+ 0x0000059f,
0x00000003,
0x00000002,
- 0x00000632,
+ 0x0000062f,
0x00040004,
0x00000000,
- 0x0000064e,
+ 0x0000064b,
0x00010005,
0x00000000,
- 0x0000066b,
+ 0x00000668,
0x00010006,
0x00000000,
- 0x000005f0,
+ 0x000005ef,
0x00000007,
0x00000000,
- 0x00000676,
+ 0x00000673,
/* 0x03c4: memx_func_tail */
/* 0x03c4: memx_ts_start */
0x00000000,
@@ -1304,560 +1304,560 @@ uint32_t gt215_pmu_code[] = {
0x67f102d7,
0x63f1fffc,
0x76fdffff,
- 0x0267f104,
- 0x0576fd00,
- 0x70f980f9,
- 0xe0fcd0fc,
- 0xf04021f4,
+ 0x0267f004,
+ 0xf90576fd,
+ 0xfc70f980,
+ 0xf4e0fcd0,
+ 0x67f04021,
+ 0xe007f104,
+ 0x0604b607,
+ 0xbd0006d0,
+/* 0x0581: memx_func_enter_wait */
+ 0xc067f104,
+ 0x0664b607,
+ 0xf00066cf,
+ 0x0bf40464,
+ 0x2c67f0f3,
+ 0xcf0664b6,
+ 0x06800066,
+/* 0x059f: memx_func_leave */
+ 0xf000f8f1,
+ 0x64b62c67,
+ 0x0066cf06,
+ 0xf0f20680,
0x07f10467,
- 0x04b607e0,
+ 0x04b607e4,
0x0006d006,
-/* 0x0582: memx_func_enter_wait */
+/* 0x05ba: memx_func_leave_wait */
0x67f104bd,
0x64b607c0,
0x0066cf06,
0xf40464f0,
- 0x67f0f30b,
- 0x0664b62c,
- 0x800066cf,
- 0x00f8f106,
-/* 0x05a0: memx_func_leave */
- 0xb62c67f0,
- 0x66cf0664,
- 0xf2068000,
- 0xf10467f0,
- 0xb607e407,
- 0x06d00604,
-/* 0x05bb: memx_func_leave_wait */
- 0xf104bd00,
- 0xb607c067,
- 0x66cf0664,
- 0x0464f000,
- 0xf1f31bf4,
- 0xb9161087,
- 0x21f4028e,
- 0x02d7b904,
- 0xffcc67f1,
- 0xffff63f1,
- 0xf90476fd,
- 0xfc70f980,
- 0xf4e0fcd0,
- 0x00f84021,
-/* 0x05f0: memx_func_wait_vblank */
- 0xb0001698,
- 0x0bf40066,
- 0x0166b013,
- 0xf4060bf4,
-/* 0x0602: memx_func_wait_vblank_head1 */
- 0x77f12e0e,
- 0x0ef40020,
-/* 0x0609: memx_func_wait_vblank_head0 */
- 0x0877f107,
-/* 0x060d: memx_func_wait_vblank_0 */
- 0xc467f100,
- 0x0664b607,
- 0xfd0066cf,
- 0x1bf40467,
-/* 0x061d: memx_func_wait_vblank_1 */
- 0xc467f1f3,
- 0x0664b607,
- 0xfd0066cf,
- 0x0bf40467,
-/* 0x062d: memx_func_wait_vblank_fini */
- 0x0410b6f3,
-/* 0x0632: memx_func_wr32 */
- 0x169800f8,
- 0x01159800,
- 0xf90810b6,
- 0xfc50f960,
- 0xf4e0fcd0,
- 0x42b64021,
- 0xe91bf402,
-/* 0x064e: memx_func_wait */
- 0x87f000f8,
- 0x0684b62c,
- 0x980088cf,
- 0x1d98001e,
- 0x021c9801,
- 0xb6031b98,
- 0x21f41010,
-/* 0x066b: memx_func_delay */
- 0x9800f8a3,
- 0x10b6001e,
- 0x7e21f404,
-/* 0x0676: memx_func_train */
- 0x57f100f8,
- 0x77f10003,
- 0x97f10000,
- 0x93f00000,
- 0x029eb970,
- 0xb90421f4,
- 0xe7f102d8,
- 0x21f42710,
-/* 0x0695: memx_func_train_loop_outer */
- 0x0158e07e,
- 0x0083f101,
- 0xe097f102,
- 0x1193f011,
- 0x80f990f9,
+ 0x87f1f31b,
+ 0x8eb91610,
+ 0x0421f402,
+ 0xf102d7b9,
+ 0xf1ffcc67,
+ 0xfdffff63,
+ 0x80f90476,
+ 0xd0fc70f9,
+ 0x21f4e0fc,
+/* 0x05ef: memx_func_wait_vblank */
+ 0x9800f840,
+ 0x66b00016,
+ 0x120bf400,
+ 0xf40166b0,
+ 0x0ef4060b,
+/* 0x0601: memx_func_wait_vblank_head1 */
+ 0x2077f02c,
+/* 0x0607: memx_func_wait_vblank_head0 */
+ 0xf0060ef4,
+/* 0x060a: memx_func_wait_vblank_0 */
+ 0x67f10877,
+ 0x64b607c4,
+ 0x0066cf06,
+ 0xf40467fd,
+/* 0x061a: memx_func_wait_vblank_1 */
+ 0x67f1f31b,
+ 0x64b607c4,
+ 0x0066cf06,
+ 0xf40467fd,
+/* 0x062a: memx_func_wait_vblank_fini */
+ 0x10b6f30b,
+/* 0x062f: memx_func_wr32 */
+ 0x9800f804,
+ 0x15980016,
+ 0x0810b601,
+ 0x50f960f9,
0xe0fcd0fc,
- 0xf94021f4,
- 0x0067f150,
-/* 0x06b5: memx_func_train_loop_inner */
- 0x1187f100,
- 0x9068ff11,
- 0xfd109894,
- 0x97f10589,
- 0x93f00720,
- 0xf990f910,
- 0xfcd0fc80,
- 0x4021f4e0,
- 0x008097f1,
- 0xb91093f0,
- 0x21f4029e,
- 0x02d8b904,
- 0xf92088c5,
+ 0xb64021f4,
+ 0x1bf40242,
+/* 0x064b: memx_func_wait */
+ 0xf000f8e9,
+ 0x84b62c87,
+ 0x0088cf06,
+ 0x98001e98,
+ 0x1c98011d,
+ 0x031b9802,
+ 0xf41010b6,
+ 0x00f8a321,
+/* 0x0668: memx_func_delay */
+ 0xb6001e98,
+ 0x21f40410,
+/* 0x0673: memx_func_train */
+ 0xf000f87e,
+ 0x77f00357,
+ 0x0097f100,
+ 0x7093f000,
+ 0xf4029eb9,
+ 0xd8b90421,
+ 0x10e7f102,
+ 0x7e21f427,
+/* 0x0690: memx_func_train_loop_outer */
+ 0x010158e0,
+ 0x020083f1,
+ 0x11e097f1,
+ 0xf91193f0,
+ 0xfc80f990,
+ 0xf4e0fcd0,
+ 0x50f94021,
+/* 0x06af: memx_func_train_loop_inner */
+ 0xf10067f0,
+ 0xff111187,
+ 0x98949068,
+ 0x0589fd10,
+ 0x072097f1,
+ 0xf91093f0,
0xfc80f990,
0xf4e0fcd0,
0x97f14021,
- 0x93f0053c,
- 0x0287f110,
- 0x0083f130,
- 0xf990f980,
+ 0x93f00080,
+ 0x029eb910,
+ 0xb90421f4,
+ 0x88c502d8,
+ 0xf990f920,
0xfcd0fc80,
0x4021f4e0,
- 0x0560e7f1,
- 0xf110e3f0,
- 0xf10000d7,
- 0x908000d3,
- 0xb7f100dc,
- 0xb3f08480,
- 0xa321f41e,
- 0x000057f1,
- 0xffff97f1,
- 0x830093f1,
-/* 0x0734: memx_func_train_loop_4x */
- 0x0080a7f1,
- 0xb910a3f0,
- 0x21f402ae,
- 0x02d8b904,
- 0xffdfb7f1,
- 0xffffb3f1,
- 0xf9048bfd,
- 0xfc80f9a0,
+ 0x053c97f1,
+ 0xf11093f0,
+ 0xf1300287,
+ 0xf9800083,
+ 0xfc80f990,
0xf4e0fcd0,
- 0xa7f14021,
- 0xa3f0053c,
- 0x0287f110,
- 0x0083f130,
- 0xf9a0f980,
- 0xfcd0fc80,
- 0x4021f4e0,
- 0x0560e7f1,
- 0xf110e3f0,
- 0xf10000d7,
- 0xb98000d3,
- 0xb7f102dc,
- 0xb3f02710,
- 0xa321f400,
- 0xf402eeb9,
- 0xddb90421,
- 0x949dff02,
+ 0xe7f14021,
+ 0xe3f00560,
+ 0x00d7f110,
+ 0x00d3f100,
+ 0x00dc9080,
+ 0x8480b7f1,
+ 0xf41eb3f0,
+ 0x57f0a321,
+ 0xff97f100,
+ 0x0093f1ff,
+/* 0x072d: memx_func_train_loop_4x */
+ 0x80a7f183,
+ 0x10a3f000,
+ 0xf402aeb9,
+ 0xd8b90421,
+ 0xdfb7f102,
+ 0xffb3f1ff,
+ 0x048bfdff,
+ 0x80f9a0f9,
+ 0xe0fcd0fc,
+ 0xf14021f4,
+ 0xf0053ca7,
+ 0x87f110a3,
+ 0x83f13002,
+ 0xa0f98000,
+ 0xd0fc80f9,
+ 0x21f4e0fc,
+ 0x60e7f140,
+ 0x10e3f005,
+ 0x0000d7f1,
+ 0x8000d3f1,
+ 0xf102dcb9,
+ 0xf02710b7,
+ 0x21f400b3,
+ 0x02eeb9a3,
+ 0xb90421f4,
+ 0x9dff02dd,
+ 0x0150b694,
+ 0xf4045670,
+ 0x7aa0921e,
+ 0xa9800bcc,
+ 0x0160b600,
+ 0x700470b6,
+ 0x1ef51066,
+ 0x50fcff01,
0x700150b6,
- 0x1ef40456,
- 0xcc7aa092,
- 0x00a9800b,
- 0xb60160b6,
- 0x66700470,
- 0x001ef510,
- 0xb650fcff,
- 0x56700150,
- 0xd41ef507,
-/* 0x07c7: memx_exec */
- 0xf900f8fe,
- 0xb9d0f9e0,
- 0xb2b902c1,
-/* 0x07d1: memx_exec_next */
- 0x00139802,
- 0xe70410b6,
- 0xe701f034,
- 0xb601e033,
- 0x30f00132,
- 0xde35980c,
- 0x12b855f9,
- 0xe41ef406,
- 0x98f10b98,
- 0xcbbbf20c,
- 0xc4b7f102,
- 0x06b4b607,
- 0xfc00bbcf,
- 0xf5e0fcd0,
+ 0x1ef50756,
+ 0x00f8fed6,
+/* 0x07c0: memx_exec */
+ 0xd0f9e0f9,
+ 0xb902c1b9,
+/* 0x07ca: memx_exec_next */
+ 0x139802b2,
+ 0x0410b600,
+ 0x01f034e7,
+ 0x01e033e7,
+ 0xf00132b6,
+ 0x35980c30,
+ 0xb855f9de,
+ 0x1ef40612,
+ 0xf10b98e4,
+ 0xbbf20c98,
+ 0xb7f102cb,
+ 0xb4b607c4,
+ 0x00bbcf06,
+ 0xe0fcd0fc,
+ 0x033621f5,
+/* 0x0806: memx_info */
+ 0xc67000f8,
+ 0x0e0bf401,
+/* 0x080c: memx_info_data */
+ 0x03ccc7f1,
+ 0x0800b7f1,
+/* 0x0817: memx_info_train */
+ 0xf10b0ef4,
+ 0xf10bccc7,
+/* 0x081f: memx_info_send */
+ 0xf50100b7,
0xf8033621,
-/* 0x080d: memx_info */
- 0x01c67000,
-/* 0x0813: memx_info_data */
- 0xf10e0bf4,
- 0xf103ccc7,
- 0xf40800b7,
-/* 0x081e: memx_info_train */
- 0xc7f10b0e,
- 0xb7f10bcc,
-/* 0x0826: memx_info_send */
- 0x21f50100,
- 0x00f80336,
-/* 0x082c: memx_recv */
- 0xf401d6b0,
- 0xd6b0980b,
- 0xd80bf400,
-/* 0x083a: memx_init */
- 0x00f800f8,
-/* 0x083c: perf_recv */
-/* 0x083e: perf_init */
- 0x00f800f8,
-/* 0x0840: i2c_drive_scl */
- 0xf40036b0,
- 0x07f1110b,
- 0x04b607e0,
- 0x0001d006,
- 0x00f804bd,
-/* 0x0854: i2c_drive_scl_lo */
- 0x07e407f1,
- 0xd00604b6,
- 0x04bd0001,
-/* 0x0862: i2c_drive_sda */
- 0x36b000f8,
- 0x110bf400,
- 0x07e007f1,
- 0xd00604b6,
- 0x04bd0002,
-/* 0x0876: i2c_drive_sda_lo */
- 0x07f100f8,
- 0x04b607e4,
- 0x0002d006,
- 0x00f804bd,
-/* 0x0884: i2c_sense_scl */
- 0xf10132f4,
- 0xb607c437,
- 0x33cf0634,
- 0x0431fd00,
- 0xf4060bf4,
-/* 0x089a: i2c_sense_scl_done */
- 0x00f80131,
-/* 0x089c: i2c_sense_sda */
- 0xf10132f4,
- 0xb607c437,
- 0x33cf0634,
- 0x0432fd00,
- 0xf4060bf4,
-/* 0x08b2: i2c_sense_sda_done */
- 0x00f80131,
-/* 0x08b4: i2c_raise_scl */
- 0x47f140f9,
- 0x37f00898,
- 0x4021f501,
-/* 0x08c1: i2c_raise_scl_wait */
+/* 0x0825: memx_recv */
+ 0x01d6b000,
+ 0xb0980bf4,
+ 0x0bf400d6,
+/* 0x0833: memx_init */
+ 0xf800f8d8,
+/* 0x0835: perf_recv */
+/* 0x0837: perf_init */
+ 0xf800f800,
+/* 0x0839: i2c_drive_scl */
+ 0x0036b000,
+ 0xf1110bf4,
+ 0xb607e007,
+ 0x01d00604,
+ 0xf804bd00,
+/* 0x084d: i2c_drive_scl_lo */
+ 0xe407f100,
+ 0x0604b607,
+ 0xbd0001d0,
+/* 0x085b: i2c_drive_sda */
+ 0xb000f804,
+ 0x0bf40036,
+ 0xe007f111,
+ 0x0604b607,
+ 0xbd0002d0,
+/* 0x086f: i2c_drive_sda_lo */
+ 0xf100f804,
+ 0xb607e407,
+ 0x02d00604,
+ 0xf804bd00,
+/* 0x087d: i2c_sense_scl */
+ 0x0132f400,
+ 0x07c437f1,
+ 0xcf0634b6,
+ 0x31fd0033,
+ 0x060bf404,
+/* 0x0893: i2c_sense_scl_done */
+ 0xf80131f4,
+/* 0x0895: i2c_sense_sda */
+ 0x0132f400,
+ 0x07c437f1,
+ 0xcf0634b6,
+ 0x32fd0033,
+ 0x060bf404,
+/* 0x08ab: i2c_sense_sda_done */
+ 0xf80131f4,
+/* 0x08ad: i2c_raise_scl */
+ 0xf140f900,
+ 0xf0089847,
+ 0x21f50137,
+/* 0x08ba: i2c_raise_scl_wait */
+ 0xe7f10839,
+ 0x21f403e8,
+ 0x7d21f57e,
+ 0x0901f408,
+ 0xf40142b6,
+/* 0x08ce: i2c_raise_scl_done */
+ 0x40fcef1b,
+/* 0x08d2: i2c_start */
+ 0x21f500f8,
+ 0x11f4087d,
+ 0x9521f50d,
+ 0x0611f408,
+/* 0x08e3: i2c_start_rep */
+ 0xf0300ef4,
+ 0x21f50037,
+ 0x37f00839,
+ 0x5b21f501,
+ 0x0076bb08,
+ 0xf90465b6,
+ 0x04659450,
+ 0xbd0256bb,
+ 0x0475fd50,
+ 0x21f550fc,
+ 0x64b608ad,
+ 0x1f11f404,
+/* 0x0910: i2c_start_send */
+ 0xf50037f0,
+ 0xf1085b21,
+ 0xf41388e7,
+ 0x37f07e21,
+ 0x3921f500,
+ 0x88e7f108,
+ 0x7e21f413,
+/* 0x092c: i2c_start_out */
+/* 0x092e: i2c_stop */
+ 0x37f000f8,
+ 0x3921f500,
+ 0x0037f008,
+ 0x085b21f5,
+ 0x03e8e7f1,
+ 0xf07e21f4,
+ 0x21f50137,
+ 0xe7f10839,
+ 0x21f41388,
+ 0x0137f07e,
+ 0x085b21f5,
+ 0x1388e7f1,
+ 0xf87e21f4,
+/* 0x0961: i2c_bitw */
+ 0x5b21f500,
0xe8e7f108,
0x7e21f403,
- 0x088421f5,
- 0xb60901f4,
- 0x1bf40142,
-/* 0x08d5: i2c_raise_scl_done */
- 0xf840fcef,
-/* 0x08d9: i2c_start */
- 0x8421f500,
- 0x0d11f408,
- 0x089c21f5,
- 0xf40611f4,
-/* 0x08ea: i2c_start_rep */
- 0x37f0300e,
- 0x4021f500,
- 0x0137f008,
- 0x086221f5,
0xb60076bb,
0x50f90465,
0xbb046594,
0x50bd0256,
0xfc0475fd,
- 0xb421f550,
+ 0xad21f550,
0x0464b608,
-/* 0x0917: i2c_start_send */
- 0xf01f11f4,
- 0x21f50037,
- 0xe7f10862,
- 0x21f41388,
- 0x0037f07e,
- 0x084021f5,
- 0x1388e7f1,
-/* 0x0933: i2c_start_out */
- 0xf87e21f4,
-/* 0x0935: i2c_stop */
- 0x0037f000,
- 0x084021f5,
- 0xf50037f0,
- 0xf1086221,
- 0xf403e8e7,
- 0x37f07e21,
- 0x4021f501,
- 0x88e7f108,
- 0x7e21f413,
- 0xf50137f0,
- 0xf1086221,
+ 0xf11811f4,
0xf41388e7,
- 0x00f87e21,
-/* 0x0968: i2c_bitw */
- 0x086221f5,
- 0x03e8e7f1,
- 0xbb7e21f4,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x08b421f5,
- 0xf40464b6,
- 0xe7f11811,
- 0x21f41388,
- 0x0037f07e,
- 0x084021f5,
- 0x1388e7f1,
-/* 0x09a7: i2c_bitw_out */
- 0xf87e21f4,
-/* 0x09a9: i2c_bitr */
- 0x0137f000,
- 0x086221f5,
- 0x03e8e7f1,
- 0xbb7e21f4,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x08b421f5,
- 0xf40464b6,
- 0x21f51b11,
- 0x37f0089c,
- 0x4021f500,
+ 0x37f07e21,
+ 0x3921f500,
0x88e7f108,
0x7e21f413,
- 0xf4013cf0,
-/* 0x09ee: i2c_bitr_done */
- 0x00f80131,
-/* 0x09f0: i2c_get_byte */
- 0xf00057f0,
-/* 0x09f6: i2c_get_byte_next */
- 0x54b60847,
+/* 0x09a0: i2c_bitw_out */
+/* 0x09a2: i2c_bitr */
+ 0x37f000f8,
+ 0x5b21f501,
+ 0xe8e7f108,
+ 0x7e21f403,
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0xad21f550,
+ 0x0464b608,
+ 0xf51b11f4,
+ 0xf0089521,
+ 0x21f50037,
+ 0xe7f10839,
+ 0x21f41388,
+ 0x013cf07e,
+/* 0x09e7: i2c_bitr_done */
+ 0xf80131f4,
+/* 0x09e9: i2c_get_byte */
+ 0x0057f000,
+/* 0x09ef: i2c_get_byte_next */
+ 0xb60847f0,
+ 0x76bb0154,
+ 0x0465b600,
+ 0x659450f9,
+ 0x0256bb04,
+ 0x75fd50bd,
+ 0xf550fc04,
+ 0xb609a221,
+ 0x11f40464,
+ 0x0553fd2b,
+ 0xf40142b6,
+ 0x37f0d81b,
0x0076bb01,
0xf90465b6,
0x04659450,
0xbd0256bb,
0x0475fd50,
0x21f550fc,
- 0x64b609a9,
- 0x2b11f404,
- 0xb60553fd,
- 0x1bf40142,
- 0x0137f0d8,
+ 0x64b60961,
+/* 0x0a39: i2c_get_byte_done */
+/* 0x0a3b: i2c_put_byte */
+ 0xf000f804,
+/* 0x0a3e: i2c_put_byte_next */
+ 0x42b60847,
+ 0x3854ff01,
0xb60076bb,
0x50f90465,
0xbb046594,
0x50bd0256,
0xfc0475fd,
- 0x6821f550,
+ 0x6121f550,
0x0464b609,
-/* 0x0a40: i2c_get_byte_done */
-/* 0x0a42: i2c_put_byte */
- 0x47f000f8,
-/* 0x0a45: i2c_put_byte_next */
- 0x0142b608,
- 0xbb3854ff,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x096821f5,
- 0xf40464b6,
- 0x46b03411,
- 0xd81bf400,
- 0xb60076bb,
- 0x50f90465,
- 0xbb046594,
- 0x50bd0256,
- 0xfc0475fd,
- 0xa921f550,
- 0x0464b609,
- 0xbb0f11f4,
- 0x36b00076,
- 0x061bf401,
-/* 0x0a9b: i2c_put_byte_done */
- 0xf80132f4,
-/* 0x0a9d: i2c_addr */
- 0x0076bb00,
+ 0xb03411f4,
+ 0x1bf40046,
+ 0x0076bbd8,
0xf90465b6,
0x04659450,
0xbd0256bb,
0x0475fd50,
0x21f550fc,
- 0x64b608d9,
- 0x2911f404,
- 0x012ec3e7,
- 0xfd0134b6,
- 0x76bb0553,
+ 0x64b609a2,
+ 0x0f11f404,
+ 0xb00076bb,
+ 0x1bf40136,
+ 0x0132f406,
+/* 0x0a94: i2c_put_byte_done */
+/* 0x0a96: i2c_addr */
+ 0x76bb00f8,
0x0465b600,
0x659450f9,
0x0256bb04,
0x75fd50bd,
0xf550fc04,
- 0xb60a4221,
-/* 0x0ae2: i2c_addr_done */
- 0x00f80464,
-/* 0x0ae4: i2c_acquire_addr */
- 0xb6f8cec7,
- 0xe0b702e4,
- 0xee980d1c,
-/* 0x0af3: i2c_acquire */
- 0xf500f800,
- 0xf40ae421,
- 0xd9f00421,
+ 0xb608d221,
+ 0x11f40464,
+ 0x2ec3e729,
+ 0x0134b601,
+ 0xbb0553fd,
+ 0x65b60076,
+ 0x9450f904,
+ 0x56bb0465,
+ 0xfd50bd02,
+ 0x50fc0475,
+ 0x0a3b21f5,
+/* 0x0adb: i2c_addr_done */
+ 0xf80464b6,
+/* 0x0add: i2c_acquire_addr */
+ 0xf8cec700,
+ 0xb702e4b6,
+ 0x980d1ce0,
+ 0x00f800ee,
+/* 0x0aec: i2c_acquire */
+ 0x0add21f5,
+ 0xf00421f4,
+ 0x21f403d9,
+/* 0x0afb: i2c_release */
+ 0xf500f840,
+ 0xf40add21,
+ 0xdaf00421,
0x4021f403,
-/* 0x0b02: i2c_release */
- 0x21f500f8,
- 0x21f40ae4,
- 0x03daf004,
- 0xf84021f4,
-/* 0x0b11: i2c_recv */
- 0x0132f400,
- 0xb6f8c1c7,
- 0x16b00214,
- 0x3a1ff528,
- 0xf413a001,
- 0x0032980c,
- 0x0ccc13a0,
- 0xf4003198,
- 0xd0f90231,
- 0xd0f9e0f9,
- 0x000067f1,
- 0x100063f1,
- 0xbb016792,
+/* 0x0b0a: i2c_recv */
+ 0x32f400f8,
+ 0xf8c1c701,
+ 0xb00214b6,
+ 0x1ff52816,
+ 0x13a0013a,
+ 0x32980cf4,
+ 0xcc13a000,
+ 0x0031980c,
+ 0xf90231f4,
+ 0xf9e0f9d0,
+ 0x0067f1d0,
+ 0x0063f100,
+ 0x01679210,
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0xec21f550,
+ 0x0464b60a,
+ 0xd6b0d0fc,
+ 0xb31bf500,
+ 0x0057f000,
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0x9621f550,
+ 0x0464b60a,
+ 0x00d011f5,
+ 0xbbe0c5c7,
0x65b60076,
0x9450f904,
0x56bb0465,
0xfd50bd02,
0x50fc0475,
- 0x0af321f5,
- 0xfc0464b6,
- 0x00d6b0d0,
- 0x00b31bf5,
- 0xbb0057f0,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x0a9d21f5,
+ 0x0a3b21f5,
0xf50464b6,
- 0xc700d011,
- 0x76bbe0c5,
+ 0xf000ad11,
+ 0x76bb0157,
0x0465b600,
0x659450f9,
0x0256bb04,
0x75fd50bd,
0xf550fc04,
- 0xb60a4221,
+ 0xb60a9621,
0x11f50464,
- 0x57f000ad,
- 0x0076bb01,
- 0xf90465b6,
- 0x04659450,
- 0xbd0256bb,
- 0x0475fd50,
- 0x21f550fc,
- 0x64b60a9d,
- 0x8a11f504,
- 0x0076bb00,
- 0xf90465b6,
- 0x04659450,
- 0xbd0256bb,
- 0x0475fd50,
- 0x21f550fc,
- 0x64b609f0,
- 0x6a11f404,
- 0xbbe05bcb,
- 0x65b60076,
- 0x9450f904,
- 0x56bb0465,
- 0xfd50bd02,
- 0x50fc0475,
- 0x093521f5,
- 0xb90464b6,
- 0x74bd025b,
-/* 0x0c17: i2c_recv_not_rd08 */
- 0xb0430ef4,
- 0x1bf401d6,
- 0x0057f03d,
- 0x0a9d21f5,
- 0xc73311f4,
- 0x21f5e0c5,
- 0x11f40a42,
- 0x0057f029,
- 0x0a9d21f5,
- 0xc71f11f4,
- 0x21f5e0b5,
- 0x11f40a42,
- 0x3521f515,
- 0xc774bd09,
- 0x1bf408c5,
- 0x0232f409,
-/* 0x0c57: i2c_recv_not_wr08 */
-/* 0x0c57: i2c_recv_done */
- 0xc7030ef4,
- 0x21f5f8ce,
- 0xe0fc0b02,
- 0x12f4d0fc,
- 0x027cb90a,
- 0x033621f5,
-/* 0x0c6c: i2c_recv_exit */
-/* 0x0c6e: i2c_init */
+ 0x76bb008a,
+ 0x0465b600,
+ 0x659450f9,
+ 0x0256bb04,
+ 0x75fd50bd,
+ 0xf550fc04,
+ 0xb609e921,
+ 0x11f40464,
+ 0xe05bcb6a,
+ 0xb60076bb,
+ 0x50f90465,
+ 0xbb046594,
+ 0x50bd0256,
+ 0xfc0475fd,
+ 0x2e21f550,
+ 0x0464b609,
+ 0xbd025bb9,
+ 0x430ef474,
+/* 0x0c10: i2c_recv_not_rd08 */
+ 0xf401d6b0,
+ 0x57f03d1b,
+ 0x9621f500,
+ 0x3311f40a,
+ 0xf5e0c5c7,
+ 0xf40a3b21,
+ 0x57f02911,
+ 0x9621f500,
+ 0x1f11f40a,
+ 0xf5e0b5c7,
+ 0xf40a3b21,
+ 0x21f51511,
+ 0x74bd092e,
+ 0xf408c5c7,
+ 0x32f4091b,
+ 0x030ef402,
+/* 0x0c50: i2c_recv_not_wr08 */
+/* 0x0c50: i2c_recv_done */
+ 0xf5f8cec7,
+ 0xfc0afb21,
+ 0xf4d0fce0,
+ 0x7cb90a12,
+ 0x3621f502,
+/* 0x0c65: i2c_recv_exit */
+/* 0x0c67: i2c_init */
+ 0xf800f803,
+/* 0x0c69: test_recv */
+ 0xd817f100,
+ 0x0614b605,
+ 0xb60011cf,
+ 0x07f10110,
+ 0x04b605d8,
+ 0x0001d006,
+ 0xe7f104bd,
+ 0xe3f1d900,
+ 0x21f5134f,
+ 0x00f80256,
+/* 0x0c90: test_init */
+ 0x0800e7f1,
+ 0x025621f5,
+/* 0x0c9a: idle_recv */
0x00f800f8,
-/* 0x0c70: test_recv */
- 0x05d817f1,
- 0xcf0614b6,
- 0x10b60011,
- 0xd807f101,
- 0x0604b605,
- 0xbd0001d0,
- 0x00e7f104,
- 0x4fe3f1d9,
- 0x5621f513,
-/* 0x0c97: test_init */
- 0xf100f802,
- 0xf50800e7,
- 0xf8025621,
-/* 0x0ca1: idle_recv */
-/* 0x0ca3: idle */
- 0xf400f800,
- 0x17f10031,
- 0x14b605d4,
- 0x0011cf06,
- 0xf10110b6,
- 0xb605d407,
- 0x01d00604,
-/* 0x0cbf: idle_loop */
- 0xf004bd00,
- 0x32f45817,
-/* 0x0cc5: idle_proc */
-/* 0x0cc5: idle_proc_exec */
- 0xb910f902,
- 0x21f5021e,
- 0x10fc033f,
- 0xf40911f4,
- 0x0ef40231,
-/* 0x0cd9: idle_proc_next */
- 0x5810b6ef,
- 0xf4061fb8,
- 0x02f4e61b,
- 0x0028f4dd,
- 0x00bb0ef4,
+/* 0x0c9c: idle */
+ 0xf10031f4,
+ 0xb605d417,
+ 0x11cf0614,
+ 0x0110b600,
+ 0x05d407f1,
+ 0xd00604b6,
+ 0x04bd0001,
+/* 0x0cb8: idle_loop */
+ 0xf45817f0,
+/* 0x0cbe: idle_proc */
+/* 0x0cbe: idle_proc_exec */
+ 0x10f90232,
+ 0xf5021eb9,
+ 0xfc033f21,
+ 0x0911f410,
+ 0xf40231f4,
+/* 0x0cd2: idle_proc_next */
+ 0x10b6ef0e,
+ 0x061fb858,
+ 0xf4e61bf4,
+ 0x28f4dd02,
+ 0xbb0ef400,
+ 0x00000000,
+ 0x00000000,
0x00000000,
0x00000000,
0x00000000,
diff --git a/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/memx.fuc b/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/memx.fuc
index ec03f9a..1663bf9 100644
--- a/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/memx.fuc
+++ b/drivers/gpu/drm/nouveau/nvkm/subdev/pmu/fuc/memx.fuc
@@ -82,15 +82,15 @@
// $r0 - zero
memx_func_enter:
#if NVKM_PPWR_CHIPSET == GT215
- movw $r8 0x1610
+ mov $r8 0x1610
nv_rd32($r7, $r8)
imm32($r6, 0xfffffffc)
and $r7 $r6
- movw $r6 0x2
+ mov $r6 0x2
or $r7 $r6
nv_wr32($r8, $r7)
#else
- movw $r6 0x001620
+ mov $r6 0x001620
imm32($r7, ~0x00000aa2);
nv_rd32($r8, $r6)
and $r8 $r7
@@ -101,7 +101,7 @@
and $r8 $r7
nv_wr32($r6, $r8)
- movw $r6 0x0026f0
+ mov $r6 0x0026f0
nv_rd32($r8, $r6)
and $r8 $r7
nv_wr32($r6, $r8)
@@ -136,19 +136,19 @@
bra nz #memx_func_leave_wait
#if NVKM_PPWR_CHIPSET == GT215
- movw $r8 0x1610
+ mov $r8 0x1610
nv_rd32($r7, $r8)
imm32($r6, 0xffffffcc)
and $r7 $r6
nv_wr32($r8, $r7)
#else
- movw $r6 0x0026f0
+ mov $r6 0x0026f0
imm32($r7, 0x00000001)
nv_rd32($r8, $r6)
or $r8 $r7
nv_wr32($r6, $r8)
- movw $r6 0x001620
+ mov $r6 0x001620
nv_rd32($r8, $r6)
or $r8 $r7
nv_wr32($r6, $r8)
@@ -177,11 +177,11 @@
bra #memx_func_wait_vblank_fini
memx_func_wait_vblank_head1:
- movw $r7 0x20
+ mov $r7 0x20
bra #memx_func_wait_vblank_0
memx_func_wait_vblank_head0:
- movw $r7 0x8
+ mov $r7 0x8
memx_func_wait_vblank_0:
nv_iord($r6, NV_PPWR_INPUT)
@@ -273,13 +273,13 @@
// $r5 - outer loop counter
// $r6 - inner loop counter
// $r7 - entry counter (#memx_train_head + $r7)
- movw $r5 0x3
- movw $r7 0x0
+ mov $r5 0x3
+ mov $r7 0x0
// Read random memory to wake up... things
imm32($r9, 0x700000)
nv_rd32($r8,$r9)
- movw $r14 0x2710
+ mov $r14 0x2710
call(nsec)
memx_func_train_loop_outer:
@@ -289,9 +289,9 @@
nv_wr32($r9, $r8)
push $r5
- movw $r6 0x0
+ mov $r6 0x0
memx_func_train_loop_inner:
- movw $r8 0x1111
+ mov $r8 0x1111
mulu $r9 $r6 $r8
shl b32 $r8 $r9 0x10
or $r8 $r9
@@ -315,7 +315,7 @@
// $r5 - inner inner loop counter
// $r9 - result
- movw $r5 0
+ mov $r5 0
imm32($r9, 0x8300ffff)
memx_func_train_loop_4x:
imm32($r10, 0x100080)
diff --git a/drivers/gpu/drm/omapdrm/omap_dmm_tiler.c b/drivers/gpu/drm/omapdrm/omap_dmm_tiler.c
index 7def040..6a0b25e 100644
--- a/drivers/gpu/drm/omapdrm/omap_dmm_tiler.c
+++ b/drivers/gpu/drm/omapdrm/omap_dmm_tiler.c
@@ -273,6 +273,17 @@ static int dmm_txn_commit(struct dmm_txn *txn, bool wait)
}
txn->last_pat->next_pa = 0;
+ /* ensure that the written descriptors are visible to DMM */
+ wmb();
+
+ /*
+ * NOTE: the wmb() above should be enough, but there seems to be a bug
+ * in OMAP's memory barrier implementation, which in some rare cases may
+ * cause the writes not to be observable after wmb().
+ */
+
+ /* read back to ensure the data is in RAM */
+ readl(&txn->last_pat->next_pa);
/* write to PAT_DESCR to clear out any pending transaction */
dmm_write(dmm, 0x0, reg[PAT_DESCR][engine->id]);
diff --git a/drivers/gpu/drm/panel/panel-jdi-lt070me05000.c b/drivers/gpu/drm/panel/panel-jdi-lt070me05000.c
index 5b2340e..0c08add 100644
--- a/drivers/gpu/drm/panel/panel-jdi-lt070me05000.c
+++ b/drivers/gpu/drm/panel/panel-jdi-lt070me05000.c
@@ -343,7 +343,7 @@ static int dsi_dcs_bl_get_brightness(struct backlight_device *bl)
dsi->mode_flags &= ~MIPI_DSI_MODE_LPM;
- ret = mipi_dsi_dcs_get_display_brightness(dsi, &brightness);
+ ret = mipi_dsi_dcs_get_display_brightness(dsi, &brightness, 1);
if (ret < 0)
return ret;
@@ -359,7 +359,7 @@ static int dsi_dcs_bl_update_status(struct backlight_device *bl)
dsi->mode_flags &= ~MIPI_DSI_MODE_LPM;
- ret = mipi_dsi_dcs_set_display_brightness(dsi, bl->props.brightness);
+ ret = mipi_dsi_dcs_set_display_brightness(dsi, bl->props.brightness, 1);
if (ret < 0)
return ret;
diff --git a/drivers/gpu/drm/panel/panel-samsung-s6e8aa0.c b/drivers/gpu/drm/panel/panel-samsung-s6e8aa0.c
index a188a39..6ad827b 100644
--- a/drivers/gpu/drm/panel/panel-samsung-s6e8aa0.c
+++ b/drivers/gpu/drm/panel/panel-samsung-s6e8aa0.c
@@ -823,7 +823,7 @@ static void s6e8aa0_read_mtp_id(struct s6e8aa0 *ctx)
int ret, i;
ret = s6e8aa0_dcs_read(ctx, 0xd1, id, ARRAY_SIZE(id));
- if (ret < ARRAY_SIZE(id) || id[0] == 0x00) {
+ if (ret < 0 || ret < ARRAY_SIZE(id) || id[0] == 0x00) {
dev_err(ctx->dev, "read id failed\n");
ctx->error = -EIO;
return;
diff --git a/drivers/gpu/drm/panel/panel-simple.c b/drivers/gpu/drm/panel/panel-simple.c
index 6f65846..5b2a9f9 100644
--- a/drivers/gpu/drm/panel/panel-simple.c
+++ b/drivers/gpu/drm/panel/panel-simple.c
@@ -1250,7 +1250,7 @@ static const struct panel_desc ontat_yx700wv03 = {
.width = 154,
.height = 83,
},
- .bus_format = MEDIA_BUS_FMT_RGB888_1X24,
+ .bus_format = MEDIA_BUS_FMT_RGB666_1X18,
};
static const struct drm_display_mode ortustech_com43h4m85ulc_mode = {
diff --git a/drivers/gpu/drm/radeon/radeon_connectors.c b/drivers/gpu/drm/radeon/radeon_connectors.c
index f416f5c2..c5e1aa5 100644
--- a/drivers/gpu/drm/radeon/radeon_connectors.c
+++ b/drivers/gpu/drm/radeon/radeon_connectors.c
@@ -850,7 +850,7 @@ static int radeon_lvds_get_modes(struct drm_connector *connector)
return ret;
}
-static int radeon_lvds_mode_valid(struct drm_connector *connector,
+static enum drm_mode_status radeon_lvds_mode_valid(struct drm_connector *connector,
struct drm_display_mode *mode)
{
struct drm_encoder *encoder = radeon_best_single_encoder(connector);
@@ -1010,7 +1010,7 @@ static int radeon_vga_get_modes(struct drm_connector *connector)
return ret;
}
-static int radeon_vga_mode_valid(struct drm_connector *connector,
+static enum drm_mode_status radeon_vga_mode_valid(struct drm_connector *connector,
struct drm_display_mode *mode)
{
struct drm_device *dev = connector->dev;
@@ -1154,7 +1154,7 @@ static int radeon_tv_get_modes(struct drm_connector *connector)
return 1;
}
-static int radeon_tv_mode_valid(struct drm_connector *connector,
+static enum drm_mode_status radeon_tv_mode_valid(struct drm_connector *connector,
struct drm_display_mode *mode)
{
if ((mode->hdisplay > 1024) || (mode->vdisplay > 768))
@@ -1496,7 +1496,7 @@ static void radeon_dvi_force(struct drm_connector *connector)
radeon_connector->use_digital = true;
}
-static int radeon_dvi_mode_valid(struct drm_connector *connector,
+static enum drm_mode_status radeon_dvi_mode_valid(struct drm_connector *connector,
struct drm_display_mode *mode)
{
struct drm_device *dev = connector->dev;
@@ -1798,7 +1798,7 @@ radeon_dp_detect(struct drm_connector *connector, bool force)
return ret;
}
-static int radeon_dp_mode_valid(struct drm_connector *connector,
+static enum drm_mode_status radeon_dp_mode_valid(struct drm_connector *connector,
struct drm_display_mode *mode)
{
struct drm_device *dev = connector->dev;
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_gem.c b/drivers/gpu/drm/rockchip/rockchip_drm_gem.c
index b70f942..cab4d60 100644
--- a/drivers/gpu/drm/rockchip/rockchip_drm_gem.c
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_gem.c
@@ -64,7 +64,6 @@ static int rockchip_drm_gem_object_mmap(struct drm_gem_object *obj,
* VM_PFNMAP flag that was set by drm_gem_mmap_obj()/drm_gem_mmap().
*/
vma->vm_flags &= ~VM_PFNMAP;
- vma->vm_pgoff = 0;
ret = dma_mmap_attrs(drm->dev, vma, rk_obj->kvaddr, rk_obj->dma_addr,
obj->size, rk_obj->dma_attrs);
@@ -96,6 +95,12 @@ int rockchip_gem_mmap(struct file *filp, struct vm_area_struct *vma)
if (ret)
return ret;
+ /*
+ * Set vm_pgoff (used as a fake buffer offset by DRM) to 0 and map the
+ * whole buffer from the start.
+ */
+ vma->vm_pgoff = 0;
+
obj = vma->vm_private_data;
return rockchip_drm_gem_object_mmap(obj, vma);
diff --git a/drivers/gpu/drm/sun4i/sun4i_dotclock.c b/drivers/gpu/drm/sun4i/sun4i_dotclock.c
index d401156..4460ca4 100644
--- a/drivers/gpu/drm/sun4i/sun4i_dotclock.c
+++ b/drivers/gpu/drm/sun4i/sun4i_dotclock.c
@@ -129,10 +129,13 @@ static int sun4i_dclk_get_phase(struct clk_hw *hw)
static int sun4i_dclk_set_phase(struct clk_hw *hw, int degrees)
{
struct sun4i_dclk *dclk = hw_to_dclk(hw);
+ u32 val = degrees / 120;
+
+ val <<= 28;
regmap_update_bits(dclk->regmap, SUN4I_TCON0_IO_POL_REG,
GENMASK(29, 28),
- degrees / 120);
+ val);
return 0;
}
diff --git a/drivers/gpu/drm/sun4i/sun4i_drv.c b/drivers/gpu/drm/sun4i/sun4i_drv.c
index aad2f4a..97828fa 100644
--- a/drivers/gpu/drm/sun4i/sun4i_drv.c
+++ b/drivers/gpu/drm/sun4i/sun4i_drv.c
@@ -283,7 +283,6 @@ static int sun4i_drv_add_endpoints(struct device *dev,
remote = of_graph_get_remote_port_parent(ep);
if (!remote) {
DRM_DEBUG_DRIVER("Error retrieving the output node\n");
- of_node_put(remote);
continue;
}
@@ -297,11 +296,13 @@ static int sun4i_drv_add_endpoints(struct device *dev,
if (of_graph_parse_endpoint(ep, &endpoint)) {
DRM_DEBUG_DRIVER("Couldn't parse endpoint\n");
+ of_node_put(remote);
continue;
}
if (!endpoint.id) {
DRM_DEBUG_DRIVER("Endpoint is our panel... skipping\n");
+ of_node_put(remote);
continue;
}
}
diff --git a/drivers/gpu/drm/udl/udl_fb.c b/drivers/gpu/drm/udl/udl_fb.c
index 67ea2ce..6a7994a 100644
--- a/drivers/gpu/drm/udl/udl_fb.c
+++ b/drivers/gpu/drm/udl/udl_fb.c
@@ -136,7 +136,10 @@ int udl_handle_damage(struct udl_framebuffer *fb, int x, int y,
if (cmd > (char *) urb->transfer_buffer) {
/* Send partial buffer remaining before exiting */
- int len = cmd - (char *) urb->transfer_buffer;
+ int len;
+ if (cmd < (char *) urb->transfer_buffer + urb->transfer_buffer_length)
+ *cmd++ = 0xAF;
+ len = cmd - (char *) urb->transfer_buffer;
ret = udl_submit_urb(dev, urb, len);
bytes_sent += len;
} else
@@ -214,7 +217,7 @@ static int udl_fb_open(struct fb_info *info, int user)
struct fb_deferred_io *fbdefio;
- fbdefio = kmalloc(sizeof(struct fb_deferred_io), GFP_KERNEL);
+ fbdefio = kzalloc(sizeof(struct fb_deferred_io), GFP_KERNEL);
if (fbdefio) {
fbdefio->delay = DL_DEFIO_WRITE_DELAY;
diff --git a/drivers/gpu/drm/udl/udl_main.c b/drivers/gpu/drm/udl/udl_main.c
index 873f010..10e2c19 100644
--- a/drivers/gpu/drm/udl/udl_main.c
+++ b/drivers/gpu/drm/udl/udl_main.c
@@ -169,18 +169,13 @@ static void udl_free_urb_list(struct drm_device *dev)
struct list_head *node;
struct urb_node *unode;
struct urb *urb;
- int ret;
unsigned long flags;
DRM_DEBUG("Waiting for completes and freeing all render urbs\n");
/* keep waiting and freeing, until we've got 'em all */
while (count--) {
-
- /* Getting interrupted means a leak, but ok at shutdown*/
- ret = down_interruptible(&udl->urbs.limit_sem);
- if (ret)
- break;
+ down(&udl->urbs.limit_sem);
spin_lock_irqsave(&udl->urbs.lock, flags);
@@ -204,17 +199,22 @@ static void udl_free_urb_list(struct drm_device *dev)
static int udl_alloc_urb_list(struct drm_device *dev, int count, size_t size)
{
struct udl_device *udl = dev->dev_private;
- int i = 0;
struct urb *urb;
struct urb_node *unode;
char *buf;
+ size_t wanted_size = count * size;
spin_lock_init(&udl->urbs.lock);
+retry:
udl->urbs.size = size;
INIT_LIST_HEAD(&udl->urbs.list);
- while (i < count) {
+ sema_init(&udl->urbs.limit_sem, 0);
+ udl->urbs.count = 0;
+ udl->urbs.available = 0;
+
+ while (udl->urbs.count * size < wanted_size) {
unode = kzalloc(sizeof(struct urb_node), GFP_KERNEL);
if (!unode)
break;
@@ -230,11 +230,16 @@ static int udl_alloc_urb_list(struct drm_device *dev, int count, size_t size)
}
unode->urb = urb;
- buf = usb_alloc_coherent(udl->udev, MAX_TRANSFER, GFP_KERNEL,
+ buf = usb_alloc_coherent(udl->udev, size, GFP_KERNEL,
&urb->transfer_dma);
if (!buf) {
kfree(unode);
usb_free_urb(urb);
+ if (size > PAGE_SIZE) {
+ size /= 2;
+ udl_free_urb_list(dev);
+ goto retry;
+ }
break;
}
@@ -245,16 +250,14 @@ static int udl_alloc_urb_list(struct drm_device *dev, int count, size_t size)
list_add_tail(&unode->entry, &udl->urbs.list);
- i++;
+ up(&udl->urbs.limit_sem);
+ udl->urbs.count++;
+ udl->urbs.available++;
}
- sema_init(&udl->urbs.limit_sem, i);
- udl->urbs.count = i;
- udl->urbs.available = i;
+ DRM_DEBUG("allocated %d %d byte urbs\n", udl->urbs.count, (int) size);
- DRM_DEBUG("allocated %d %d byte urbs\n", i, (int) size);
-
- return i;
+ return udl->urbs.count;
}
struct urb *udl_get_urb(struct drm_device *dev)
diff --git a/drivers/gpu/drm/udl/udl_transfer.c b/drivers/gpu/drm/udl/udl_transfer.c
index 917dcb9..9259a2f 100644
--- a/drivers/gpu/drm/udl/udl_transfer.c
+++ b/drivers/gpu/drm/udl/udl_transfer.c
@@ -152,11 +152,11 @@ static void udl_compress_hline16(
raw_pixels_count_byte = cmd++; /* we'll know this later */
raw_pixel_start = pixel;
- cmd_pixel_end = pixel + (min(MAX_CMD_PIXELS + 1,
- min((int)(pixel_end - pixel) / bpp,
- (int)(cmd_buffer_end - cmd) / 2))) * bpp;
+ cmd_pixel_end = pixel + min3(MAX_CMD_PIXELS + 1UL,
+ (unsigned long)(pixel_end - pixel) / bpp,
+ (unsigned long)(cmd_buffer_end - 1 - cmd) / 2) * bpp;
- prefetch_range((void *) pixel, (cmd_pixel_end - pixel) * bpp);
+ prefetch_range((void *) pixel, cmd_pixel_end - pixel);
pixel_val16 = get_pixel_val16(pixel, bpp);
while (pixel < cmd_pixel_end) {
@@ -192,6 +192,9 @@ static void udl_compress_hline16(
if (pixel > raw_pixel_start) {
/* finalize last RAW span */
*raw_pixels_count_byte = ((pixel-raw_pixel_start) / bpp) & 0xFF;
+ } else {
+ /* undo unused byte */
+ cmd--;
}
*cmd_pixels_count_byte = ((pixel - cmd_pixel_start) / bpp) & 0xFF;
diff --git a/drivers/gpu/drm/vc4/vc4_bo.c b/drivers/gpu/drm/vc4/vc4_bo.c
index ec9023b..d53e805 100644
--- a/drivers/gpu/drm/vc4/vc4_bo.c
+++ b/drivers/gpu/drm/vc4/vc4_bo.c
@@ -80,6 +80,7 @@ static void vc4_bo_destroy(struct vc4_bo *bo)
struct vc4_dev *vc4 = to_vc4_dev(obj->dev);
if (bo->validated_shader) {
+ kfree(bo->validated_shader->uniform_addr_offsets);
kfree(bo->validated_shader->texture_samples);
kfree(bo->validated_shader);
bo->validated_shader = NULL;
@@ -328,6 +329,7 @@ void vc4_free_object(struct drm_gem_object *gem_bo)
}
if (bo->validated_shader) {
+ kfree(bo->validated_shader->uniform_addr_offsets);
kfree(bo->validated_shader->texture_samples);
kfree(bo->validated_shader);
bo->validated_shader = NULL;
diff --git a/drivers/gpu/drm/vc4/vc4_plane.c b/drivers/gpu/drm/vc4/vc4_plane.c
index 881bf48..70051bf 100644
--- a/drivers/gpu/drm/vc4/vc4_plane.c
+++ b/drivers/gpu/drm/vc4/vc4_plane.c
@@ -327,6 +327,9 @@ static int vc4_plane_setup_clipping_and_scaling(struct drm_plane_state *state)
vc4_state->y_scaling[0] = vc4_get_scaling_mode(vc4_state->src_h[0],
vc4_state->crtc_h);
+ vc4_state->is_unity = (vc4_state->x_scaling[0] == VC4_SCALING_NONE &&
+ vc4_state->y_scaling[0] == VC4_SCALING_NONE);
+
if (num_planes > 1) {
vc4_state->is_yuv = true;
@@ -342,21 +345,18 @@ static int vc4_plane_setup_clipping_and_scaling(struct drm_plane_state *state)
vc4_get_scaling_mode(vc4_state->src_h[1],
vc4_state->crtc_h);
- /* YUV conversion requires that scaling be enabled,
- * even on a plane that's otherwise 1:1. Choose TPZ
- * for simplicity.
+ /* YUV conversion requires that horizontal scaling be enabled,
+ * even on a plane that's otherwise 1:1. Looks like only PPF
+ * works in that case, so let's pick that one.
*/
- if (vc4_state->x_scaling[0] == VC4_SCALING_NONE)
- vc4_state->x_scaling[0] = VC4_SCALING_TPZ;
- if (vc4_state->y_scaling[0] == VC4_SCALING_NONE)
- vc4_state->y_scaling[0] = VC4_SCALING_TPZ;
+ if (vc4_state->is_unity)
+ vc4_state->x_scaling[0] = VC4_SCALING_PPF;
+ } else {
+ vc4_state->is_yuv = false;
+ vc4_state->x_scaling[1] = VC4_SCALING_NONE;
+ vc4_state->y_scaling[1] = VC4_SCALING_NONE;
}
- vc4_state->is_unity = (vc4_state->x_scaling[0] == VC4_SCALING_NONE &&
- vc4_state->y_scaling[0] == VC4_SCALING_NONE &&
- vc4_state->x_scaling[1] == VC4_SCALING_NONE &&
- vc4_state->y_scaling[1] == VC4_SCALING_NONE);
-
/* No configuring scaling on the cursor plane, since it gets
non-vblank-synced updates, and scaling requires requires
LBM changes which have to be vblank-synced.
@@ -533,7 +533,7 @@ static int vc4_plane_mode_set(struct drm_plane *plane,
* the scl fields here.
*/
if (num_planes == 1) {
- scl0 = vc4_get_scl_field(state, 1);
+ scl0 = vc4_get_scl_field(state, 0);
scl1 = scl0;
} else {
scl0 = vc4_get_scl_field(state, 1);
@@ -611,7 +611,10 @@ static int vc4_plane_mode_set(struct drm_plane *plane,
vc4_dlist_write(vc4_state, SCALER_CSC2_ITR_R_601_5);
}
- if (!vc4_state->is_unity) {
+ if (vc4_state->x_scaling[0] != VC4_SCALING_NONE ||
+ vc4_state->x_scaling[1] != VC4_SCALING_NONE ||
+ vc4_state->y_scaling[0] != VC4_SCALING_NONE ||
+ vc4_state->y_scaling[1] != VC4_SCALING_NONE) {
/* LBM Base Address. */
if (vc4_state->y_scaling[0] != VC4_SCALING_NONE ||
vc4_state->y_scaling[1] != VC4_SCALING_NONE) {
diff --git a/drivers/gpu/drm/vc4/vc4_validate_shaders.c b/drivers/gpu/drm/vc4/vc4_validate_shaders.c
index 917321c..19a5bde8 100644
--- a/drivers/gpu/drm/vc4/vc4_validate_shaders.c
+++ b/drivers/gpu/drm/vc4/vc4_validate_shaders.c
@@ -874,6 +874,7 @@ vc4_validate_shader(struct drm_gem_cma_object *shader_obj)
fail:
kfree(validation_state.branch_targets);
if (validated_shader) {
+ kfree(validated_shader->uniform_addr_offsets);
kfree(validated_shader->texture_samples);
kfree(validated_shader);
}
diff --git a/drivers/gpu/drm/virtio/virtgpu_ioctl.c b/drivers/gpu/drm/virtio/virtgpu_ioctl.c
index 818478b..5463939 100644
--- a/drivers/gpu/drm/virtio/virtgpu_ioctl.c
+++ b/drivers/gpu/drm/virtio/virtgpu_ioctl.c
@@ -194,6 +194,9 @@ static int virtio_gpu_getparam_ioctl(struct drm_device *dev, void *data,
case VIRTGPU_PARAM_3D_FEATURES:
value = vgdev->has_virgl_3d == true ? 1 : 0;
break;
+ case VIRTGPU_PARAM_CAPSET_QUERY_FIX:
+ value = 1;
+ break;
default:
return -EINVAL;
}
@@ -469,7 +472,7 @@ static int virtio_gpu_get_caps_ioctl(struct drm_device *dev,
{
struct virtio_gpu_device *vgdev = dev->dev_private;
struct drm_virtgpu_get_caps *args = data;
- int size;
+ unsigned size, host_caps_size;
int i;
int found_valid = -1;
int ret;
@@ -478,6 +481,10 @@ static int virtio_gpu_get_caps_ioctl(struct drm_device *dev,
if (vgdev->num_capsets == 0)
return -ENOSYS;
+ /* don't allow userspace to pass 0 */
+ if (args->size == 0)
+ return -EINVAL;
+
spin_lock(&vgdev->display_info_lock);
for (i = 0; i < vgdev->num_capsets; i++) {
if (vgdev->capsets[i].id == args->cap_set_id) {
@@ -493,11 +500,9 @@ static int virtio_gpu_get_caps_ioctl(struct drm_device *dev,
return -EINVAL;
}
- size = vgdev->capsets[found_valid].max_size;
- if (args->size > size) {
- spin_unlock(&vgdev->display_info_lock);
- return -EINVAL;
- }
+ host_caps_size = vgdev->capsets[found_valid].max_size;
+ /* only copy to user the minimum of the host caps size or the guest caps size */
+ size = min(args->size, host_caps_size);
list_for_each_entry(cache_ent, &vgdev->cap_cache, head) {
if (cache_ent->id == args->cap_set_id &&
diff --git a/drivers/gpu/drm/virtio/virtgpu_vq.c b/drivers/gpu/drm/virtio/virtgpu_vq.c
index 5a0f8a7..52436b3c 100644
--- a/drivers/gpu/drm/virtio/virtgpu_vq.c
+++ b/drivers/gpu/drm/virtio/virtgpu_vq.c
@@ -324,7 +324,7 @@ static int virtio_gpu_queue_ctrl_buffer_locked(struct virtio_gpu_device *vgdev,
ret = virtqueue_add_sgs(vq, sgs, outcnt, incnt, vbuf, GFP_ATOMIC);
if (ret == -ENOSPC) {
spin_unlock(&vgdev->ctrlq.qlock);
- wait_event(vgdev->ctrlq.ack_queue, vq->num_free);
+ wait_event(vgdev->ctrlq.ack_queue, vq->num_free >= outcnt + incnt);
spin_lock(&vgdev->ctrlq.qlock);
goto retry;
} else {
@@ -399,7 +399,7 @@ static int virtio_gpu_queue_cursor(struct virtio_gpu_device *vgdev,
ret = virtqueue_add_sgs(vq, sgs, outcnt, 0, vbuf, GFP_ATOMIC);
if (ret == -ENOSPC) {
spin_unlock(&vgdev->cursorq.qlock);
- wait_event(vgdev->cursorq.ack_queue, vq->num_free);
+ wait_event(vgdev->cursorq.ack_queue, vq->num_free >= outcnt);
spin_lock(&vgdev->cursorq.qlock);
goto retry;
} else {
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
index 87086af..33ca24a 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
@@ -2014,6 +2014,7 @@ void vmw_kms_helper_resource_finish(struct vmw_validation_ctx *ctx,
vmw_kms_helper_buffer_finish(res->dev_priv, NULL, ctx->buf,
out_fence, NULL);
+ vmw_dmabuf_unreference(&ctx->buf);
vmw_resource_unreserve(res, false, NULL, 0);
mutex_unlock(&res->dev_priv->cmdbuf_mutex);
}
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_msg.h b/drivers/gpu/drm/vmwgfx/vmwgfx_msg.h
index 557a033..8545488 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_msg.h
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_msg.h
@@ -135,17 +135,24 @@
#else
-/* In the 32-bit version of this macro, we use "m" because there is no
- * more register left for bp
+/*
+ * In the 32-bit version of this macro, we store bp in a memory location
+ * because we've ran out of registers.
+ * Now we can't reference that memory location while we've modified
+ * %esp or %ebp, so we first push it on the stack, just before we push
+ * %ebp, and then when we need it we read it from the stack where we
+ * just pushed it.
*/
#define VMW_PORT_HB_OUT(cmd, in_ecx, in_si, in_di, \
port_num, magic, bp, \
eax, ebx, ecx, edx, si, di) \
({ \
- asm volatile ("push %%ebp;" \
- "mov %12, %%ebp;" \
+ asm volatile ("push %12;" \
+ "push %%ebp;" \
+ "mov 0x04(%%esp), %%ebp;" \
"rep outsb;" \
- "pop %%ebp;" : \
+ "pop %%ebp;" \
+ "add $0x04, %%esp;" : \
"=a"(eax), \
"=b"(ebx), \
"=c"(ecx), \
@@ -167,10 +174,12 @@
port_num, magic, bp, \
eax, ebx, ecx, edx, si, di) \
({ \
- asm volatile ("push %%ebp;" \
- "mov %12, %%ebp;" \
+ asm volatile ("push %12;" \
+ "push %%ebp;" \
+ "mov 0x04(%%esp), %%ebp;" \
"rep insb;" \
- "pop %%ebp" : \
+ "pop %%ebp;" \
+ "add $0x04, %%esp;" : \
"=a"(eax), \
"=b"(ebx), \
"=c"(ecx), \
diff --git a/drivers/gpu/ipu-v3/ipu-common.c b/drivers/gpu/ipu-v3/ipu-common.c
index b9539f7..99c813a 100644
--- a/drivers/gpu/ipu-v3/ipu-common.c
+++ b/drivers/gpu/ipu-v3/ipu-common.c
@@ -715,15 +715,16 @@ void ipu_set_ic_src_mux(struct ipu_soc *ipu, int csi_id, bool vdi)
spin_lock_irqsave(&ipu->lock, flags);
val = ipu_cm_read(ipu, IPU_CONF);
- if (vdi) {
+ if (vdi)
val |= IPU_CONF_IC_INPUT;
- } else {
+ else
val &= ~IPU_CONF_IC_INPUT;
- if (csi_id == 1)
- val |= IPU_CONF_CSI_SEL;
- else
- val &= ~IPU_CONF_CSI_SEL;
- }
+
+ if (csi_id == 1)
+ val |= IPU_CONF_CSI_SEL;
+ else
+ val &= ~IPU_CONF_CSI_SEL;
+
ipu_cm_write(ipu, val, IPU_CONF);
spin_unlock_irqrestore(&ipu->lock, flags);
diff --git a/drivers/gpu/ipu-v3/ipu-csi.c b/drivers/gpu/ipu-v3/ipu-csi.c
index d6e5ded..8774bf1 100644
--- a/drivers/gpu/ipu-v3/ipu-csi.c
+++ b/drivers/gpu/ipu-v3/ipu-csi.c
@@ -316,13 +316,17 @@ static int mbus_code_to_bus_cfg(struct ipu_csi_bus_config *cfg, u32 mbus_code)
/*
* Fill a CSI bus config struct from mbus_config and mbus_framefmt.
*/
-static void fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
+static int fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
struct v4l2_mbus_config *mbus_cfg,
struct v4l2_mbus_framefmt *mbus_fmt)
{
+ int ret;
+
memset(csicfg, 0, sizeof(*csicfg));
- mbus_code_to_bus_cfg(csicfg, mbus_fmt->code);
+ ret = mbus_code_to_bus_cfg(csicfg, mbus_fmt->code);
+ if (ret < 0)
+ return ret;
switch (mbus_cfg->type) {
case V4L2_MBUS_PARALLEL:
@@ -353,6 +357,8 @@ static void fill_csi_bus_cfg(struct ipu_csi_bus_config *csicfg,
/* will never get here, keep compiler quiet */
break;
}
+
+ return 0;
}
int ipu_csi_init_interface(struct ipu_csi *csi,
@@ -362,8 +368,11 @@ int ipu_csi_init_interface(struct ipu_csi *csi,
struct ipu_csi_bus_config cfg;
unsigned long flags;
u32 width, height, data = 0;
+ int ret;
- fill_csi_bus_cfg(&cfg, mbus_cfg, mbus_fmt);
+ ret = fill_csi_bus_cfg(&cfg, mbus_cfg, mbus_fmt);
+ if (ret < 0)
+ return ret;
/* set default sensor frame width and height */
width = mbus_fmt->width;
@@ -567,11 +576,14 @@ int ipu_csi_set_mipi_datatype(struct ipu_csi *csi, u32 vc,
struct ipu_csi_bus_config cfg;
unsigned long flags;
u32 temp;
+ int ret;
if (vc > 3)
return -EINVAL;
- mbus_code_to_bus_cfg(&cfg, mbus_fmt->code);
+ ret = mbus_code_to_bus_cfg(&cfg, mbus_fmt->code);
+ if (ret < 0)
+ return ret;
spin_lock_irqsave(&csi->lock, flags);
diff --git a/drivers/gpu/msm/adreno.c b/drivers/gpu/msm/adreno.c
index 59dbdc4..15f2c34 100644
--- a/drivers/gpu/msm/adreno.c
+++ b/drivers/gpu/msm/adreno.c
@@ -3738,6 +3738,7 @@ static struct platform_driver adreno_platform_driver = {
.name = DEVICE_3D_NAME,
.pm = &kgsl_pm_ops,
.of_match_table = adreno_match_table,
+ .probe_type = PROBE_PREFER_ASYNCHRONOUS,
}
};
diff --git a/drivers/gpu/msm/adreno_trace.h b/drivers/gpu/msm/adreno_trace.h
index de028fa..7774ca9 100644
--- a/drivers/gpu/msm/adreno_trace.h
+++ b/drivers/gpu/msm/adreno_trace.h
@@ -17,7 +17,7 @@
#undef TRACE_SYSTEM
#define TRACE_SYSTEM kgsl
#undef TRACE_INCLUDE_PATH
-#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_PATH ../../drivers/gpu/msm
#undef TRACE_INCLUDE_FILE
#define TRACE_INCLUDE_FILE adreno_trace
diff --git a/drivers/gpu/msm/kgsl_events.c b/drivers/gpu/msm/kgsl_events.c
index 759a966..9dfefec 100644
--- a/drivers/gpu/msm/kgsl_events.c
+++ b/drivers/gpu/msm/kgsl_events.c
@@ -15,8 +15,8 @@
#include <linux/list.h>
#include <linux/workqueue.h>
#include <linux/debugfs.h>
-#include <kgsl_device.h>
+#include "kgsl_device.h"
#include "kgsl_debugfs.h"
#include "kgsl_trace.h"
diff --git a/drivers/gpu/msm/kgsl_pool.c b/drivers/gpu/msm/kgsl_pool.c
index 5da8c1d..cd5f8e3 100644
--- a/drivers/gpu/msm/kgsl_pool.c
+++ b/drivers/gpu/msm/kgsl_pool.c
@@ -467,12 +467,16 @@ kgsl_pool_shrink_scan_objects(struct shrinker *shrinker,
/* nr represents number of pages to be removed*/
int nr = sc->nr_to_scan;
int total_pages = kgsl_pool_size_total();
+ unsigned long ret;
/* Target pages represents new pool size */
int target_pages = (nr > total_pages) ? 0 : (total_pages - nr);
/* Reduce pool size to target_pages */
- return kgsl_pool_reduce(target_pages, false);
+ ret = kgsl_pool_reduce(target_pages, false);
+
+ /* If we are unable to shrink more, stop trying */
+ return (ret == 0) ? SHRINK_STOP : ret;
}
static unsigned long
diff --git a/drivers/gpu/msm/kgsl_pwrctrl.c b/drivers/gpu/msm/kgsl_pwrctrl.c
index 2a149ac..1b4a44c 100644
--- a/drivers/gpu/msm/kgsl_pwrctrl.c
+++ b/drivers/gpu/msm/kgsl_pwrctrl.c
@@ -127,7 +127,7 @@ static void _record_pwrevent(struct kgsl_device *device,
/**
* kgsl_get_bw() - Return latest msm bus IB vote
*/
-static unsigned int kgsl_get_bw(void)
+static unsigned long kgsl_get_bw(void)
{
return ib_votes[last_vote_buslevel];
}
diff --git a/drivers/gpu/msm/kgsl_trace.h b/drivers/gpu/msm/kgsl_trace.h
index c7690a1..919fd04 100644
--- a/drivers/gpu/msm/kgsl_trace.h
+++ b/drivers/gpu/msm/kgsl_trace.h
@@ -17,7 +17,7 @@
#undef TRACE_SYSTEM
#define TRACE_SYSTEM kgsl
#undef TRACE_INCLUDE_PATH
-#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_PATH ../../drivers/gpu/msm
#undef TRACE_INCLUDE_FILE
#define TRACE_INCLUDE_FILE kgsl_trace
diff --git a/drivers/hid/hid-apple.c b/drivers/hid/hid-apple.c
index cb2e85c..a8b8058 100644
--- a/drivers/hid/hid-apple.c
+++ b/drivers/hid/hid-apple.c
@@ -333,7 +333,8 @@ static int apple_input_mapping(struct hid_device *hdev, struct hid_input *hi,
struct hid_field *field, struct hid_usage *usage,
unsigned long **bit, int *max)
{
- if (usage->hid == (HID_UP_CUSTOM | 0x0003)) {
+ if (usage->hid == (HID_UP_CUSTOM | 0x0003) ||
+ usage->hid == (HID_UP_MSVENDOR | 0x0003)) {
/* The fn key on Apple USB keyboards */
set_bit(EV_REP, hi->input->evbit);
hid_map_usage_clear(hi, usage, bit, max, EV_KEY, KEY_FN);
@@ -479,6 +480,12 @@ static const struct hid_device_id apple_devices[] = {
.driver_data = APPLE_NUMLOCK_EMULATION | APPLE_HAS_FN },
{ HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_ANSI),
.driver_data = APPLE_HAS_FN },
+ { HID_BLUETOOTH_DEVICE(BT_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_ANSI),
+ .driver_data = APPLE_HAS_FN },
+ { HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_NUMPAD_ANSI),
+ .driver_data = APPLE_HAS_FN },
+ { HID_BLUETOOTH_DEVICE(BT_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_NUMPAD_ANSI),
+ .driver_data = APPLE_HAS_FN },
{ HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_WELLSPRING_ANSI),
.driver_data = APPLE_HAS_FN },
{ HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_WELLSPRING_ISO),
diff --git a/drivers/hid/hid-core.c b/drivers/hid/hid-core.c
index aea6267..d988e3f 100644
--- a/drivers/hid/hid-core.c
+++ b/drivers/hid/hid-core.c
@@ -1855,6 +1855,9 @@ static const struct hid_device_id hid_have_special_driver[] = {
{ HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_ALU_WIRELESS_2011_ISO) },
{ HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_ALU_WIRELESS_2011_JIS) },
{ HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_ANSI) },
+ { HID_BLUETOOTH_DEVICE(BT_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_ANSI) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_NUMPAD_ANSI) },
+ { HID_BLUETOOTH_DEVICE(BT_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_NUMPAD_ANSI) },
{ HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_FOUNTAIN_TP_ONLY) },
{ HID_USB_DEVICE(USB_VENDOR_ID_APPLE, USB_DEVICE_ID_APPLE_GEYSER1_TP_ONLY) },
{ HID_I2C_DEVICE(USB_VENDOR_ID_ASUSTEK, USB_DEVICE_ID_ASUSTEK_NOTEBOOK_KEYBOARD) },
@@ -2060,6 +2063,10 @@ static const struct hid_device_id hid_have_special_driver[] = {
{ HID_USB_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS3_CONTROLLER) },
{ HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS3_CONTROLLER) },
{ HID_USB_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER) },
+ { HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER_2) },
+ { HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER_2) },
+ { HID_USB_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER_DONGLE) },
{ HID_USB_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_VAIO_VGX_MOUSE) },
{ HID_USB_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_VAIO_VGP_MOUSE) },
{ HID_USB_DEVICE(USB_VENDOR_ID_SINO_LITE, USB_DEVICE_ID_SINO_LITE_CONTROLLER) },
diff --git a/drivers/hid/hid-debug.c b/drivers/hid/hid-debug.c
index acfb522..2942369 100644
--- a/drivers/hid/hid-debug.c
+++ b/drivers/hid/hid-debug.c
@@ -1152,6 +1152,8 @@ static ssize_t hid_debug_events_read(struct file *file, char __user *buffer,
goto out;
if (list->tail > list->head) {
len = list->tail - list->head;
+ if (len > count)
+ len = count;
if (copy_to_user(buffer + ret, &list->hid_debug_buf[list->head], len)) {
ret = -EFAULT;
@@ -1161,6 +1163,8 @@ static ssize_t hid_debug_events_read(struct file *file, char __user *buffer,
list->head += len;
} else {
len = HID_DEBUG_BUFSIZE - list->head;
+ if (len > count)
+ len = count;
if (copy_to_user(buffer, &list->hid_debug_buf[list->head], len)) {
ret = -EFAULT;
@@ -1168,7 +1172,9 @@ static ssize_t hid_debug_events_read(struct file *file, char __user *buffer,
}
list->head = 0;
ret += len;
- goto copy_rest;
+ count -= len;
+ if (count > 0)
+ goto copy_rest;
}
}
diff --git a/drivers/hid/hid-ids.h b/drivers/hid/hid-ids.h
index 9347b37..6f4c84d 100644
--- a/drivers/hid/hid-ids.h
+++ b/drivers/hid/hid-ids.h
@@ -83,6 +83,7 @@
#define USB_DEVICE_ID_ANTON_TOUCH_PAD 0x3101
#define USB_VENDOR_ID_APPLE 0x05ac
+#define BT_VENDOR_ID_APPLE 0x004c
#define USB_DEVICE_ID_APPLE_MIGHTYMOUSE 0x0304
#define USB_DEVICE_ID_APPLE_MAGICMOUSE 0x030d
#define USB_DEVICE_ID_APPLE_MAGICTRACKPAD 0x030e
@@ -152,6 +153,7 @@
#define USB_DEVICE_ID_APPLE_ALU_WIRELESS_2011_ISO 0x0256
#define USB_DEVICE_ID_APPLE_ALU_WIRELESS_2011_JIS 0x0257
#define USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_ANSI 0x0267
+#define USB_DEVICE_ID_APPLE_MAGIC_KEYBOARD_NUMPAD_ANSI 0x026c
#define USB_DEVICE_ID_APPLE_WELLSPRING8_ANSI 0x0290
#define USB_DEVICE_ID_APPLE_WELLSPRING8_ISO 0x0291
#define USB_DEVICE_ID_APPLE_WELLSPRING8_JIS 0x0292
@@ -549,6 +551,9 @@
#define USB_VENDOR_ID_IRTOUCHSYSTEMS 0x6615
#define USB_DEVICE_ID_IRTOUCH_INFRARED_USB 0x0070
+#define USB_VENDOR_ID_INNOMEDIA 0x1292
+#define USB_DEVICE_ID_INNEX_GENESIS_ATARI 0x4745
+
#define USB_VENDOR_ID_ITE 0x048d
#define USB_DEVICE_ID_ITE_LENOVO_YOGA 0x8386
#define USB_DEVICE_ID_ITE_LENOVO_YOGA2 0x8350
@@ -885,6 +890,7 @@
#define USB_DEVICE_ID_SAITEK_RUMBLEPAD 0xff17
#define USB_DEVICE_ID_SAITEK_PS1000 0x0621
#define USB_DEVICE_ID_SAITEK_RAT7_OLD 0x0ccb
+#define USB_DEVICE_ID_SAITEK_RAT7_CONTAGION 0x0ccd
#define USB_DEVICE_ID_SAITEK_RAT7 0x0cd7
#define USB_DEVICE_ID_SAITEK_RAT9 0x0cfa
#define USB_DEVICE_ID_SAITEK_MMO7 0x0cd0
@@ -924,6 +930,8 @@
#define USB_DEVICE_ID_SONY_PS3_BDREMOTE 0x0306
#define USB_DEVICE_ID_SONY_PS3_CONTROLLER 0x0268
#define USB_DEVICE_ID_SONY_PS4_CONTROLLER 0x05c4
+#define USB_DEVICE_ID_SONY_PS4_CONTROLLER_2 0x09cc
+#define USB_DEVICE_ID_SONY_PS4_CONTROLLER_DONGLE 0x0ba0
#define USB_DEVICE_ID_SONY_MOTION_CONTROLLER 0x03d5
#define USB_DEVICE_ID_SONY_NAVIGATION_CONTROLLER 0x042f
#define USB_DEVICE_ID_SONY_BUZZ_CONTROLLER 0x0002
@@ -957,6 +965,7 @@
#define USB_VENDOR_ID_SYMBOL 0x05e0
#define USB_DEVICE_ID_SYMBOL_SCANNER_1 0x0800
#define USB_DEVICE_ID_SYMBOL_SCANNER_2 0x1300
+#define USB_DEVICE_ID_SYMBOL_SCANNER_3 0x1200
#define USB_VENDOR_ID_SYNAPTICS 0x06cb
#define USB_DEVICE_ID_SYNAPTICS_TP 0x0001
diff --git a/drivers/hid/hid-input.c b/drivers/hid/hid-input.c
index 5ff6dd8..fc7ada2 100644
--- a/drivers/hid/hid-input.c
+++ b/drivers/hid/hid-input.c
@@ -324,6 +324,9 @@ static const struct hid_device_id hid_battery_quirks[] = {
{ HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_ELECOM,
USB_DEVICE_ID_ELECOM_BM084),
HID_BATTERY_QUIRK_IGNORE },
+ { HID_USB_DEVICE(USB_VENDOR_ID_SYMBOL,
+ USB_DEVICE_ID_SYMBOL_SCANNER_3),
+ HID_BATTERY_QUIRK_IGNORE },
{}
};
diff --git a/drivers/hid/hid-ntrig.c b/drivers/hid/hid-ntrig.c
index 1b0084d..28373da 100644
--- a/drivers/hid/hid-ntrig.c
+++ b/drivers/hid/hid-ntrig.c
@@ -955,6 +955,8 @@ static int ntrig_probe(struct hid_device *hdev, const struct hid_device_id *id)
ret = sysfs_create_group(&hdev->dev.kobj,
&ntrig_attribute_group);
+ if (ret)
+ hid_err(hdev, "cannot create sysfs group\n");
return 0;
err_free:
diff --git a/drivers/hid/hid-plantronics.c b/drivers/hid/hid-plantronics.c
index febb21e..584b10d 100644
--- a/drivers/hid/hid-plantronics.c
+++ b/drivers/hid/hid-plantronics.c
@@ -2,7 +2,7 @@
* Plantronics USB HID Driver
*
* Copyright (c) 2014 JD Cole <jd.cole@plantronics.com>
- * Copyright (c) 2015 Terry Junge <terry.junge@plantronics.com>
+ * Copyright (c) 2015-2018 Terry Junge <terry.junge@plantronics.com>
*/
/*
@@ -48,6 +48,10 @@ static int plantronics_input_mapping(struct hid_device *hdev,
unsigned short mapped_key;
unsigned long plt_type = (unsigned long)hid_get_drvdata(hdev);
+ /* special case for PTT products */
+ if (field->application == HID_GD_JOYSTICK)
+ goto defaulted;
+
/* handle volume up/down mapping */
/* non-standard types or multi-HID interfaces - plt_type is PID */
if (!(plt_type & HID_USAGE_PAGE)) {
diff --git a/drivers/hid/hid-roccat-kovaplus.c b/drivers/hid/hid-roccat-kovaplus.c
index 43617fb..317c9c2 100644
--- a/drivers/hid/hid-roccat-kovaplus.c
+++ b/drivers/hid/hid-roccat-kovaplus.c
@@ -37,6 +37,8 @@ static uint kovaplus_convert_event_cpi(uint value)
static void kovaplus_profile_activated(struct kovaplus_device *kovaplus,
uint new_profile_index)
{
+ if (new_profile_index >= ARRAY_SIZE(kovaplus->profile_settings))
+ return;
kovaplus->actual_profile = new_profile_index;
kovaplus->actual_cpi = kovaplus->profile_settings[new_profile_index].cpi_startup_level;
kovaplus->actual_x_sensitivity = kovaplus->profile_settings[new_profile_index].sensitivity_x;
diff --git a/drivers/hid/hid-saitek.c b/drivers/hid/hid-saitek.c
index 39e6426..683861f 100644
--- a/drivers/hid/hid-saitek.c
+++ b/drivers/hid/hid-saitek.c
@@ -183,6 +183,8 @@ static const struct hid_device_id saitek_devices[] = {
.driver_data = SAITEK_RELEASE_MODE_RAT7 },
{ HID_USB_DEVICE(USB_VENDOR_ID_SAITEK, USB_DEVICE_ID_SAITEK_RAT7),
.driver_data = SAITEK_RELEASE_MODE_RAT7 },
+ { HID_USB_DEVICE(USB_VENDOR_ID_SAITEK, USB_DEVICE_ID_SAITEK_RAT7_CONTAGION),
+ .driver_data = SAITEK_RELEASE_MODE_RAT7 },
{ HID_USB_DEVICE(USB_VENDOR_ID_SAITEK, USB_DEVICE_ID_SAITEK_RAT9),
.driver_data = SAITEK_RELEASE_MODE_RAT7 },
{ HID_USB_DEVICE(USB_VENDOR_ID_MADCATZ, USB_DEVICE_ID_MADCATZ_RAT9),
diff --git a/drivers/hid/hid-sony.c b/drivers/hid/hid-sony.c
index 1b1dccd..d03203a 100644
--- a/drivers/hid/hid-sony.c
+++ b/drivers/hid/hid-sony.c
@@ -36,6 +36,8 @@
#include <linux/list.h>
#include <linux/idr.h>
#include <linux/input/mt.h>
+#include <linux/crc32.h>
+#include <asm/unaligned.h>
#include "hid-ids.h"
@@ -46,19 +48,21 @@
#define PS3REMOTE BIT(4)
#define DUALSHOCK4_CONTROLLER_USB BIT(5)
#define DUALSHOCK4_CONTROLLER_BT BIT(6)
-#define MOTION_CONTROLLER_USB BIT(7)
-#define MOTION_CONTROLLER_BT BIT(8)
-#define NAVIGATION_CONTROLLER_USB BIT(9)
-#define NAVIGATION_CONTROLLER_BT BIT(10)
-#define SINO_LITE_CONTROLLER BIT(11)
-#define FUTUREMAX_DANCE_MAT BIT(12)
+#define DUALSHOCK4_DONGLE BIT(7)
+#define MOTION_CONTROLLER_USB BIT(8)
+#define MOTION_CONTROLLER_BT BIT(9)
+#define NAVIGATION_CONTROLLER_USB BIT(10)
+#define NAVIGATION_CONTROLLER_BT BIT(11)
+#define SINO_LITE_CONTROLLER BIT(12)
+#define FUTUREMAX_DANCE_MAT BIT(13)
#define SIXAXIS_CONTROLLER (SIXAXIS_CONTROLLER_USB | SIXAXIS_CONTROLLER_BT)
#define MOTION_CONTROLLER (MOTION_CONTROLLER_USB | MOTION_CONTROLLER_BT)
#define NAVIGATION_CONTROLLER (NAVIGATION_CONTROLLER_USB |\
NAVIGATION_CONTROLLER_BT)
#define DUALSHOCK4_CONTROLLER (DUALSHOCK4_CONTROLLER_USB |\
- DUALSHOCK4_CONTROLLER_BT)
+ DUALSHOCK4_CONTROLLER_BT | \
+ DUALSHOCK4_DONGLE)
#define SONY_LED_SUPPORT (SIXAXIS_CONTROLLER | BUZZ_CONTROLLER |\
DUALSHOCK4_CONTROLLER | MOTION_CONTROLLER |\
NAVIGATION_CONTROLLER)
@@ -71,89 +75,6 @@
#define MAX_LEDS 4
-/*
- * The Sixaxis reports both digital and analog values for each button on the
- * controller except for Start, Select and the PS button. The controller ends
- * up reporting 27 axes which causes them to spill over into the multi-touch
- * axis values. Additionally, the controller only has 20 actual, physical axes
- * so there are several unused axes in between the used ones.
- */
-static u8 sixaxis_rdesc[] = {
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x04, /* Usage (Joystick), */
- 0xA1, 0x01, /* Collection (Application), */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0x01, /* Report ID (1), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x01, /* Report Count (1), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x81, 0x03, /* Input (Constant, Variable), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x13, /* Report Count (19), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x01, /* Logical Maximum (1), */
- 0x35, 0x00, /* Physical Minimum (0), */
- 0x45, 0x01, /* Physical Maximum (1), */
- 0x05, 0x09, /* Usage Page (Button), */
- 0x19, 0x01, /* Usage Minimum (01h), */
- 0x29, 0x13, /* Usage Maximum (13h), */
- 0x81, 0x02, /* Input (Variable), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x0D, /* Report Count (13), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x81, 0x03, /* Input (Constant, Variable), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xA1, 0x00, /* Collection (Physical), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x04, /* Report Count (4), */
- 0x35, 0x00, /* Physical Minimum (0), */
- 0x46, 0xFF, 0x00, /* Physical Maximum (255), */
- 0x09, 0x30, /* Usage (X), */
- 0x09, 0x31, /* Usage (Y), */
- 0x09, 0x32, /* Usage (Z), */
- 0x09, 0x35, /* Usage (Rz), */
- 0x81, 0x02, /* Input (Variable), */
- 0xC0, /* End Collection, */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x95, 0x13, /* Report Count (19), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0x81, 0x02, /* Input (Variable), */
- 0x95, 0x0C, /* Report Count (12), */
- 0x81, 0x01, /* Input (Constant), */
- 0x75, 0x10, /* Report Size (16), */
- 0x95, 0x04, /* Report Count (4), */
- 0x26, 0xFF, 0x03, /* Logical Maximum (1023), */
- 0x46, 0xFF, 0x03, /* Physical Maximum (1023), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0x81, 0x02, /* Input (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0x02, /* Report ID (2), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0xEE, /* Report ID (238), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0xEF, /* Report ID (239), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xC0 /* End Collection */
-};
/* PS/3 Motion controller */
static u8 motion_rdesc[] = {
@@ -252,567 +173,6 @@ static u8 motion_rdesc[] = {
0xC0 /* End Collection */
};
-/* PS/3 Navigation controller */
-static u8 navigation_rdesc[] = {
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x04, /* Usage (Joystick), */
- 0xA1, 0x01, /* Collection (Application), */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0x01, /* Report ID (1), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x01, /* Report Count (1), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x81, 0x03, /* Input (Constant, Variable), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x13, /* Report Count (19), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x01, /* Logical Maximum (1), */
- 0x35, 0x00, /* Physical Minimum (0), */
- 0x45, 0x01, /* Physical Maximum (1), */
- 0x05, 0x09, /* Usage Page (Button), */
- 0x19, 0x01, /* Usage Minimum (01h), */
- 0x29, 0x13, /* Usage Maximum (13h), */
- 0x81, 0x02, /* Input (Variable), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x0D, /* Report Count (13), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x81, 0x03, /* Input (Constant, Variable), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xA1, 0x00, /* Collection (Physical), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x02, /* Report Count (2), */
- 0x35, 0x00, /* Physical Minimum (0), */
- 0x46, 0xFF, 0x00, /* Physical Maximum (255), */
- 0x09, 0x30, /* Usage (X), */
- 0x09, 0x31, /* Usage (Y), */
- 0x81, 0x02, /* Input (Variable), */
- 0xC0, /* End Collection, */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x95, 0x06, /* Report Count (6), */
- 0x81, 0x03, /* Input (Constant, Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x05, /* Report Count (5), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x95, 0x01, /* Report Count (1), */
- 0x81, 0x02, /* Input (Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x95, 0x01, /* Report Count (1), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x95, 0x1E, /* Report Count (24), */
- 0x81, 0x02, /* Input (Variable), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0x91, 0x02, /* Output (Variable), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0x02, /* Report ID (2), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0xEE, /* Report ID (238), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xA1, 0x02, /* Collection (Logical), */
- 0x85, 0xEF, /* Report ID (239), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x30, /* Report Count (48), */
- 0x09, 0x01, /* Usage (Pointer), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0, /* End Collection, */
- 0xC0 /* End Collection */
-};
-
-/*
- * The default descriptor doesn't provide mapping for the accelerometers
- * or orientation sensors. This fixed descriptor maps the accelerometers
- * to usage values 0x40, 0x41 and 0x42 and maps the orientation sensors
- * to usage values 0x43, 0x44 and 0x45.
- */
-static u8 dualshock4_usb_rdesc[] = {
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x05, /* Usage (Gamepad), */
- 0xA1, 0x01, /* Collection (Application), */
- 0x85, 0x01, /* Report ID (1), */
- 0x09, 0x30, /* Usage (X), */
- 0x09, 0x31, /* Usage (Y), */
- 0x09, 0x32, /* Usage (Z), */
- 0x09, 0x35, /* Usage (Rz), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x04, /* Report Count (4), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x39, /* Usage (Hat Switch), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x07, /* Logical Maximum (7), */
- 0x35, 0x00, /* Physical Minimum (0), */
- 0x46, 0x3B, 0x01, /* Physical Maximum (315), */
- 0x65, 0x14, /* Unit (Degrees), */
- 0x75, 0x04, /* Report Size (4), */
- 0x95, 0x01, /* Report Count (1), */
- 0x81, 0x42, /* Input (Variable, Null State), */
- 0x65, 0x00, /* Unit, */
- 0x05, 0x09, /* Usage Page (Button), */
- 0x19, 0x01, /* Usage Minimum (01h), */
- 0x29, 0x0E, /* Usage Maximum (0Eh), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x01, /* Logical Maximum (1), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x0E, /* Report Count (14), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x20, /* Usage (20h), */
- 0x75, 0x06, /* Report Size (6), */
- 0x95, 0x01, /* Report Count (1), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x3F, /* Logical Maximum (63), */
- 0x81, 0x02, /* Input (Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x33, /* Usage (Rx), */
- 0x09, 0x34, /* Usage (Ry), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x02, /* Report Count (2), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x21, /* Usage (21h), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x19, 0x40, /* Usage Minimum (40h), */
- 0x29, 0x42, /* Usage Maximum (42h), */
- 0x16, 0x00, 0x80, /* Logical Minimum (-32768), */
- 0x26, 0x00, 0x7F, /* Logical Maximum (32767), */
- 0x75, 0x10, /* Report Size (16), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x19, 0x43, /* Usage Minimum (43h), */
- 0x29, 0x45, /* Usage Maximum (45h), */
- 0x16, 0x00, 0xE0, /* Logical Minimum (-8192), */
- 0x26, 0xFF, 0x1F, /* Logical Maximum (8191), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x21, /* Usage (21h), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x27, /* Report Count (39), */
- 0x81, 0x02, /* Input (Variable), */
- 0x85, 0x05, /* Report ID (5), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x1F, /* Report Count (31), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x04, /* Report ID (4), */
- 0x09, 0x23, /* Usage (23h), */
- 0x95, 0x24, /* Report Count (36), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x02, /* Report ID (2), */
- 0x09, 0x24, /* Usage (24h), */
- 0x95, 0x24, /* Report Count (36), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x08, /* Report ID (8), */
- 0x09, 0x25, /* Usage (25h), */
- 0x95, 0x03, /* Report Count (3), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x10, /* Report ID (16), */
- 0x09, 0x26, /* Usage (26h), */
- 0x95, 0x04, /* Report Count (4), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x11, /* Report ID (17), */
- 0x09, 0x27, /* Usage (27h), */
- 0x95, 0x02, /* Report Count (2), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x12, /* Report ID (18), */
- 0x06, 0x02, 0xFF, /* Usage Page (FF02h), */
- 0x09, 0x21, /* Usage (21h), */
- 0x95, 0x0F, /* Report Count (15), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x13, /* Report ID (19), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x16, /* Report Count (22), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x14, /* Report ID (20), */
- 0x06, 0x05, 0xFF, /* Usage Page (FF05h), */
- 0x09, 0x20, /* Usage (20h), */
- 0x95, 0x10, /* Report Count (16), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x15, /* Report ID (21), */
- 0x09, 0x21, /* Usage (21h), */
- 0x95, 0x2C, /* Report Count (44), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x06, 0x80, 0xFF, /* Usage Page (FF80h), */
- 0x85, 0x80, /* Report ID (128), */
- 0x09, 0x20, /* Usage (20h), */
- 0x95, 0x06, /* Report Count (6), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x81, /* Report ID (129), */
- 0x09, 0x21, /* Usage (21h), */
- 0x95, 0x06, /* Report Count (6), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x82, /* Report ID (130), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x05, /* Report Count (5), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x83, /* Report ID (131), */
- 0x09, 0x23, /* Usage (23h), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x84, /* Report ID (132), */
- 0x09, 0x24, /* Usage (24h), */
- 0x95, 0x04, /* Report Count (4), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x85, /* Report ID (133), */
- 0x09, 0x25, /* Usage (25h), */
- 0x95, 0x06, /* Report Count (6), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x86, /* Report ID (134), */
- 0x09, 0x26, /* Usage (26h), */
- 0x95, 0x06, /* Report Count (6), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x87, /* Report ID (135), */
- 0x09, 0x27, /* Usage (27h), */
- 0x95, 0x23, /* Report Count (35), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x88, /* Report ID (136), */
- 0x09, 0x28, /* Usage (28h), */
- 0x95, 0x22, /* Report Count (34), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x89, /* Report ID (137), */
- 0x09, 0x29, /* Usage (29h), */
- 0x95, 0x02, /* Report Count (2), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x90, /* Report ID (144), */
- 0x09, 0x30, /* Usage (30h), */
- 0x95, 0x05, /* Report Count (5), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x91, /* Report ID (145), */
- 0x09, 0x31, /* Usage (31h), */
- 0x95, 0x03, /* Report Count (3), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x92, /* Report ID (146), */
- 0x09, 0x32, /* Usage (32h), */
- 0x95, 0x03, /* Report Count (3), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x93, /* Report ID (147), */
- 0x09, 0x33, /* Usage (33h), */
- 0x95, 0x0C, /* Report Count (12), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA0, /* Report ID (160), */
- 0x09, 0x40, /* Usage (40h), */
- 0x95, 0x06, /* Report Count (6), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA1, /* Report ID (161), */
- 0x09, 0x41, /* Usage (41h), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA2, /* Report ID (162), */
- 0x09, 0x42, /* Usage (42h), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA3, /* Report ID (163), */
- 0x09, 0x43, /* Usage (43h), */
- 0x95, 0x30, /* Report Count (48), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA4, /* Report ID (164), */
- 0x09, 0x44, /* Usage (44h), */
- 0x95, 0x0D, /* Report Count (13), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA5, /* Report ID (165), */
- 0x09, 0x45, /* Usage (45h), */
- 0x95, 0x15, /* Report Count (21), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA6, /* Report ID (166), */
- 0x09, 0x46, /* Usage (46h), */
- 0x95, 0x15, /* Report Count (21), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF0, /* Report ID (240), */
- 0x09, 0x47, /* Usage (47h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF1, /* Report ID (241), */
- 0x09, 0x48, /* Usage (48h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF2, /* Report ID (242), */
- 0x09, 0x49, /* Usage (49h), */
- 0x95, 0x0F, /* Report Count (15), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA7, /* Report ID (167), */
- 0x09, 0x4A, /* Usage (4Ah), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA8, /* Report ID (168), */
- 0x09, 0x4B, /* Usage (4Bh), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA9, /* Report ID (169), */
- 0x09, 0x4C, /* Usage (4Ch), */
- 0x95, 0x08, /* Report Count (8), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAA, /* Report ID (170), */
- 0x09, 0x4E, /* Usage (4Eh), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAB, /* Report ID (171), */
- 0x09, 0x4F, /* Usage (4Fh), */
- 0x95, 0x39, /* Report Count (57), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAC, /* Report ID (172), */
- 0x09, 0x50, /* Usage (50h), */
- 0x95, 0x39, /* Report Count (57), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAD, /* Report ID (173), */
- 0x09, 0x51, /* Usage (51h), */
- 0x95, 0x0B, /* Report Count (11), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAE, /* Report ID (174), */
- 0x09, 0x52, /* Usage (52h), */
- 0x95, 0x01, /* Report Count (1), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xAF, /* Report ID (175), */
- 0x09, 0x53, /* Usage (53h), */
- 0x95, 0x02, /* Report Count (2), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xB0, /* Report ID (176), */
- 0x09, 0x54, /* Usage (54h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0 /* End Collection */
-};
-
-/*
- * The default behavior of the Dualshock 4 is to send reports using report
- * type 1 when running over Bluetooth. However, when feature report 2 is
- * requested during the controller initialization it starts sending input
- * reports in report 17. Since report 17 is undefined in the default HID
- * descriptor the button and axis definitions must be moved to report 17 or
- * the HID layer won't process the received input.
- */
-static u8 dualshock4_bt_rdesc[] = {
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x05, /* Usage (Gamepad), */
- 0xA1, 0x01, /* Collection (Application), */
- 0x85, 0x01, /* Report ID (1), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x0A, /* Report Count (9), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x04, 0xFF, /* Usage Page (FF04h), */
- 0x85, 0x02, /* Report ID (2), */
- 0x09, 0x24, /* Usage (24h), */
- 0x95, 0x24, /* Report Count (36), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA3, /* Report ID (163), */
- 0x09, 0x25, /* Usage (25h), */
- 0x95, 0x30, /* Report Count (48), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x05, /* Report ID (5), */
- 0x09, 0x26, /* Usage (26h), */
- 0x95, 0x28, /* Report Count (40), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x06, /* Report ID (6), */
- 0x09, 0x27, /* Usage (27h), */
- 0x95, 0x34, /* Report Count (52), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x07, /* Report ID (7), */
- 0x09, 0x28, /* Usage (28h), */
- 0x95, 0x30, /* Report Count (48), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x08, /* Report ID (8), */
- 0x09, 0x29, /* Usage (29h), */
- 0x95, 0x2F, /* Report Count (47), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x06, 0x03, 0xFF, /* Usage Page (FF03h), */
- 0x85, 0x03, /* Report ID (3), */
- 0x09, 0x21, /* Usage (21h), */
- 0x95, 0x26, /* Report Count (38), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x04, /* Report ID (4), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x2E, /* Report Count (46), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF0, /* Report ID (240), */
- 0x09, 0x47, /* Usage (47h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF1, /* Report ID (241), */
- 0x09, 0x48, /* Usage (48h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xF2, /* Report ID (242), */
- 0x09, 0x49, /* Usage (49h), */
- 0x95, 0x0F, /* Report Count (15), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x11, /* Report ID (17), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x20, /* Usage (20h), */
- 0x95, 0x02, /* Report Count (2), */
- 0x81, 0x02, /* Input (Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x30, /* Usage (X), */
- 0x09, 0x31, /* Usage (Y), */
- 0x09, 0x32, /* Usage (Z), */
- 0x09, 0x35, /* Usage (Rz), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x04, /* Report Count (4), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x39, /* Usage (Hat Switch), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x07, /* Logical Maximum (7), */
- 0x75, 0x04, /* Report Size (4), */
- 0x95, 0x01, /* Report Count (1), */
- 0x81, 0x42, /* Input (Variable, Null State), */
- 0x05, 0x09, /* Usage Page (Button), */
- 0x19, 0x01, /* Usage Minimum (01h), */
- 0x29, 0x0E, /* Usage Maximum (0Eh), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x25, 0x01, /* Logical Maximum (1), */
- 0x75, 0x01, /* Report Size (1), */
- 0x95, 0x0E, /* Report Count (14), */
- 0x81, 0x02, /* Input (Variable), */
- 0x75, 0x06, /* Report Size (6), */
- 0x95, 0x01, /* Report Count (1), */
- 0x81, 0x01, /* Input (Constant), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x09, 0x33, /* Usage (Rx), */
- 0x09, 0x34, /* Usage (Ry), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x02, /* Report Count (2), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x20, /* Usage (20h), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x05, 0x01, /* Usage Page (Desktop), */
- 0x19, 0x40, /* Usage Minimum (40h), */
- 0x29, 0x42, /* Usage Maximum (42h), */
- 0x16, 0x00, 0x80, /* Logical Minimum (-32768), */
- 0x26, 0x00, 0x7F, /* Logical Maximum (32767), */
- 0x75, 0x10, /* Report Size (16), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x19, 0x43, /* Usage Minimum (43h), */
- 0x29, 0x45, /* Usage Maximum (45h), */
- 0x16, 0x00, 0xE0, /* Logical Minimum (-8192), */
- 0x26, 0xFF, 0x1F, /* Logical Maximum (8191), */
- 0x95, 0x03, /* Report Count (3), */
- 0x81, 0x02, /* Input (Variable), */
- 0x06, 0x00, 0xFF, /* Usage Page (FF00h), */
- 0x09, 0x20, /* Usage (20h), */
- 0x15, 0x00, /* Logical Minimum (0), */
- 0x26, 0xFF, 0x00, /* Logical Maximum (255), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x31, /* Report Count (51), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x21, /* Usage (21h), */
- 0x75, 0x08, /* Report Size (8), */
- 0x95, 0x4D, /* Report Count (77), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x12, /* Report ID (18), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x8D, /* Report Count (141), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x23, /* Usage (23h), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x13, /* Report ID (19), */
- 0x09, 0x24, /* Usage (24h), */
- 0x95, 0xCD, /* Report Count (205), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x25, /* Usage (25h), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x14, /* Report ID (20), */
- 0x09, 0x26, /* Usage (26h), */
- 0x96, 0x0D, 0x01, /* Report Count (269), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x27, /* Usage (27h), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x15, /* Report ID (21), */
- 0x09, 0x28, /* Usage (28h), */
- 0x96, 0x4D, 0x01, /* Report Count (333), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x29, /* Usage (29h), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x16, /* Report ID (22), */
- 0x09, 0x2A, /* Usage (2Ah), */
- 0x96, 0x8D, 0x01, /* Report Count (397), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x2B, /* Usage (2Bh), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x17, /* Report ID (23), */
- 0x09, 0x2C, /* Usage (2Ch), */
- 0x96, 0xCD, 0x01, /* Report Count (461), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x2D, /* Usage (2Dh), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x18, /* Report ID (24), */
- 0x09, 0x2E, /* Usage (2Eh), */
- 0x96, 0x0D, 0x02, /* Report Count (525), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x2F, /* Usage (2Fh), */
- 0x91, 0x02, /* Output (Variable), */
- 0x85, 0x19, /* Report ID (25), */
- 0x09, 0x30, /* Usage (30h), */
- 0x96, 0x22, 0x02, /* Report Count (546), */
- 0x81, 0x02, /* Input (Variable), */
- 0x09, 0x31, /* Usage (31h), */
- 0x91, 0x02, /* Output (Variable), */
- 0x06, 0x80, 0xFF, /* Usage Page (FF80h), */
- 0x85, 0x82, /* Report ID (130), */
- 0x09, 0x22, /* Usage (22h), */
- 0x95, 0x3F, /* Report Count (63), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x83, /* Report ID (131), */
- 0x09, 0x23, /* Usage (23h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x84, /* Report ID (132), */
- 0x09, 0x24, /* Usage (24h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x90, /* Report ID (144), */
- 0x09, 0x30, /* Usage (30h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x91, /* Report ID (145), */
- 0x09, 0x31, /* Usage (31h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x92, /* Report ID (146), */
- 0x09, 0x32, /* Usage (32h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0x93, /* Report ID (147), */
- 0x09, 0x33, /* Usage (33h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA0, /* Report ID (160), */
- 0x09, 0x40, /* Usage (40h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0x85, 0xA4, /* Report ID (164), */
- 0x09, 0x44, /* Usage (44h), */
- 0xB1, 0x02, /* Feature (Variable), */
- 0xC0 /* End Collection */
-};
-
static u8 ps3remote_rdesc[] = {
0x05, 0x01, /* GUsagePage Generic Desktop */
0x09, 0x05, /* LUsage 0x05 [Game Pad] */
@@ -975,6 +335,97 @@ static const unsigned int buzz_keymap[] = {
[20] = BTN_TRIGGER_HAPPY20,
};
+/* The Navigation controller is a partial DS3 and uses the same HID report
+ * and hence the same keymap indices, however not not all axes/buttons
+ * are physically present. We use the same axis and button mapping as
+ * the DS3, which uses the Linux gamepad spec.
+ */
+static const unsigned int navigation_absmap[] = {
+ [0x30] = ABS_X,
+ [0x31] = ABS_Y,
+ [0x33] = ABS_Z, /* L2 */
+};
+
+/* Buttons not physically available on the device, but still available
+ * in the reports are explicitly set to 0 for documentation purposes.
+ */
+static const unsigned int navigation_keymap[] = {
+ [0x01] = 0, /* Select */
+ [0x02] = BTN_THUMBL, /* L3 */
+ [0x03] = 0, /* R3 */
+ [0x04] = 0, /* Start */
+ [0x05] = BTN_DPAD_UP, /* Up */
+ [0x06] = BTN_DPAD_RIGHT, /* Right */
+ [0x07] = BTN_DPAD_DOWN, /* Down */
+ [0x08] = BTN_DPAD_LEFT, /* Left */
+ [0x09] = BTN_TL2, /* L2 */
+ [0x0a] = 0, /* R2 */
+ [0x0b] = BTN_TL, /* L1 */
+ [0x0c] = 0, /* R1 */
+ [0x0d] = BTN_NORTH, /* Triangle */
+ [0x0e] = BTN_EAST, /* Circle */
+ [0x0f] = BTN_SOUTH, /* Cross */
+ [0x10] = BTN_WEST, /* Square */
+ [0x11] = BTN_MODE, /* PS */
+};
+
+static const unsigned int sixaxis_absmap[] = {
+ [0x30] = ABS_X,
+ [0x31] = ABS_Y,
+ [0x32] = ABS_RX, /* right stick X */
+ [0x35] = ABS_RY, /* right stick Y */
+};
+
+static const unsigned int sixaxis_keymap[] = {
+ [0x01] = BTN_SELECT, /* Select */
+ [0x02] = BTN_THUMBL, /* L3 */
+ [0x03] = BTN_THUMBR, /* R3 */
+ [0x04] = BTN_START, /* Start */
+ [0x05] = BTN_DPAD_UP, /* Up */
+ [0x06] = BTN_DPAD_RIGHT, /* Right */
+ [0x07] = BTN_DPAD_DOWN, /* Down */
+ [0x08] = BTN_DPAD_LEFT, /* Left */
+ [0x09] = BTN_TL2, /* L2 */
+ [0x0a] = BTN_TR2, /* R2 */
+ [0x0b] = BTN_TL, /* L1 */
+ [0x0c] = BTN_TR, /* R1 */
+ [0x0d] = BTN_NORTH, /* Triangle */
+ [0x0e] = BTN_EAST, /* Circle */
+ [0x0f] = BTN_SOUTH, /* Cross */
+ [0x10] = BTN_WEST, /* Square */
+ [0x11] = BTN_MODE, /* PS */
+};
+
+static const unsigned int ds4_absmap[] = {
+ [0x30] = ABS_X,
+ [0x31] = ABS_Y,
+ [0x32] = ABS_RX, /* right stick X */
+ [0x33] = ABS_Z, /* L2 */
+ [0x34] = ABS_RZ, /* R2 */
+ [0x35] = ABS_RY, /* right stick Y */
+};
+
+static const unsigned int ds4_keymap[] = {
+ [0x1] = BTN_WEST, /* Square */
+ [0x2] = BTN_SOUTH, /* Cross */
+ [0x3] = BTN_EAST, /* Circle */
+ [0x4] = BTN_NORTH, /* Triangle */
+ [0x5] = BTN_TL, /* L1 */
+ [0x6] = BTN_TR, /* R1 */
+ [0x7] = BTN_TL2, /* L2 */
+ [0x8] = BTN_TR2, /* R2 */
+ [0x9] = BTN_SELECT, /* Share */
+ [0xa] = BTN_START, /* Options */
+ [0xb] = BTN_THUMBL, /* L3 */
+ [0xc] = BTN_THUMBR, /* R3 */
+ [0xd] = BTN_MODE, /* PS */
+};
+
+static const struct {int x; int y; } ds4_hat_mapping[] = {
+ {0, -1}, {1, -1}, {1, 0}, {1, 1}, {0, 1}, {-1, 1}, {-1, 0}, {-1, -1},
+ {0, 0}
+};
+
static enum power_supply_property sony_battery_props[] = {
POWER_SUPPLY_PROP_PRESENT,
POWER_SUPPLY_PROP_CAPACITY,
@@ -1019,24 +470,75 @@ struct motion_output_report_02 {
u8 rumble;
};
-#define DS4_REPORT_0x02_SIZE 37
-#define DS4_REPORT_0x05_SIZE 32
-#define DS4_REPORT_0x11_SIZE 78
-#define DS4_REPORT_0x81_SIZE 7
+#define DS4_FEATURE_REPORT_0x02_SIZE 37
+#define DS4_FEATURE_REPORT_0x05_SIZE 41
+#define DS4_FEATURE_REPORT_0x81_SIZE 7
+#define DS4_INPUT_REPORT_0x11_SIZE 78
+#define DS4_OUTPUT_REPORT_0x05_SIZE 32
+#define DS4_OUTPUT_REPORT_0x11_SIZE 78
#define SIXAXIS_REPORT_0xF2_SIZE 17
#define SIXAXIS_REPORT_0xF5_SIZE 8
#define MOTION_REPORT_0x02_SIZE 49
+/* Offsets relative to USB input report (0x1). Bluetooth (0x11) requires an
+ * additional +2.
+ */
+#define DS4_INPUT_REPORT_AXIS_OFFSET 1
+#define DS4_INPUT_REPORT_BUTTON_OFFSET 5
+#define DS4_INPUT_REPORT_TIMESTAMP_OFFSET 10
+#define DS4_INPUT_REPORT_GYRO_X_OFFSET 13
+#define DS4_INPUT_REPORT_BATTERY_OFFSET 30
+#define DS4_INPUT_REPORT_TOUCHPAD_OFFSET 33
+
+#define SENSOR_SUFFIX " Motion Sensors"
+#define DS4_TOUCHPAD_SUFFIX " Touchpad"
+
+/* Default to 4ms poll interval, which is same as USB (not adjustable). */
+#define DS4_BT_DEFAULT_POLL_INTERVAL_MS 4
+#define DS4_BT_MAX_POLL_INTERVAL_MS 62
+#define DS4_GYRO_RES_PER_DEG_S 1024
+#define DS4_ACC_RES_PER_G 8192
+
+#define SIXAXIS_INPUT_REPORT_ACC_X_OFFSET 41
+#define SIXAXIS_ACC_RES_PER_G 113
+
static DEFINE_SPINLOCK(sony_dev_list_lock);
static LIST_HEAD(sony_device_list);
static DEFINE_IDA(sony_device_id_allocator);
+/* Used for calibration of DS4 accelerometer and gyro. */
+struct ds4_calibration_data {
+ int abs_code;
+ short bias;
+ /* Calibration requires scaling against a sensitivity value, which is a
+ * float. Store sensitivity as a fraction to limit floating point
+ * calculations until final calibration.
+ */
+ int sens_numer;
+ int sens_denom;
+};
+
+enum ds4_dongle_state {
+ DONGLE_DISCONNECTED,
+ DONGLE_CALIBRATING,
+ DONGLE_CONNECTED,
+ DONGLE_DISABLED
+};
+
+enum sony_worker {
+ SONY_WORKER_STATE,
+ SONY_WORKER_HOTPLUG
+};
+
struct sony_sc {
spinlock_t lock;
struct list_head list_node;
struct hid_device *hdev;
+ struct input_dev *touchpad;
+ struct input_dev *sensor_dev;
struct led_classdev *leds[MAX_LEDS];
unsigned long quirks;
+ struct work_struct hotplug_worker;
struct work_struct state_worker;
void (*send_output_report)(struct sony_sc *);
struct power_supply *battery;
@@ -1050,7 +552,8 @@ struct sony_sc {
#endif
u8 mac_address[6];
- u8 worker_initialized;
+ u8 hotplug_worker_initialized;
+ u8 state_worker_initialized;
u8 defer_initialization;
u8 cable_state;
u8 battery_charging;
@@ -1059,21 +562,72 @@ struct sony_sc {
u8 led_delay_on[MAX_LEDS];
u8 led_delay_off[MAX_LEDS];
u8 led_count;
+
+ bool timestamp_initialized;
+ u16 prev_timestamp;
+ unsigned int timestamp_us;
+
+ u8 ds4_bt_poll_interval;
+ enum ds4_dongle_state ds4_dongle_state;
+ /* DS4 calibration data */
+ struct ds4_calibration_data ds4_calib_data[6];
};
-static inline void sony_schedule_work(struct sony_sc *sc)
+static void sony_set_leds(struct sony_sc *sc);
+
+static inline void sony_schedule_work(struct sony_sc *sc,
+ enum sony_worker which)
{
- if (!sc->defer_initialization)
- schedule_work(&sc->state_worker);
+ switch (which) {
+ case SONY_WORKER_STATE:
+ if (!sc->defer_initialization)
+ schedule_work(&sc->state_worker);
+ break;
+ case SONY_WORKER_HOTPLUG:
+ if (sc->hotplug_worker_initialized)
+ schedule_work(&sc->hotplug_worker);
+ break;
+ }
}
-static u8 *sixaxis_fixup(struct hid_device *hdev, u8 *rdesc,
- unsigned int *rsize)
+static ssize_t ds4_show_poll_interval(struct device *dev,
+ struct device_attribute
+ *attr, char *buf)
{
- *rsize = sizeof(sixaxis_rdesc);
- return sixaxis_rdesc;
+ struct hid_device *hdev = to_hid_device(dev);
+ struct sony_sc *sc = hid_get_drvdata(hdev);
+
+ return snprintf(buf, PAGE_SIZE, "%i\n", sc->ds4_bt_poll_interval);
}
+static ssize_t ds4_store_poll_interval(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct hid_device *hdev = to_hid_device(dev);
+ struct sony_sc *sc = hid_get_drvdata(hdev);
+ unsigned long flags;
+ u8 interval;
+
+ if (kstrtou8(buf, 0, &interval))
+ return -EINVAL;
+
+ if (interval > DS4_BT_MAX_POLL_INTERVAL_MS)
+ return -EINVAL;
+
+ spin_lock_irqsave(&sc->lock, flags);
+ sc->ds4_bt_poll_interval = interval;
+ spin_unlock_irqrestore(&sc->lock, flags);
+
+ sony_schedule_work(sc, SONY_WORKER_STATE);
+
+ return count;
+}
+
+static DEVICE_ATTR(bt_poll_interval, 0644, ds4_show_poll_interval,
+ ds4_store_poll_interval);
+
+
static u8 *motion_fixup(struct hid_device *hdev, u8 *rdesc,
unsigned int *rsize)
{
@@ -1081,13 +635,6 @@ static u8 *motion_fixup(struct hid_device *hdev, u8 *rdesc,
return motion_rdesc;
}
-static u8 *navigation_fixup(struct hid_device *hdev, u8 *rdesc,
- unsigned int *rsize)
-{
- *rsize = sizeof(navigation_rdesc);
- return navigation_rdesc;
-}
-
static u8 *ps3remote_fixup(struct hid_device *hdev, u8 *rdesc,
unsigned int *rsize)
{
@@ -1129,6 +676,133 @@ static int ps3remote_mapping(struct hid_device *hdev, struct hid_input *hi,
return 1;
}
+static int navigation_mapping(struct hid_device *hdev, struct hid_input *hi,
+ struct hid_field *field, struct hid_usage *usage,
+ unsigned long **bit, int *max)
+{
+ if ((usage->hid & HID_USAGE_PAGE) == HID_UP_BUTTON) {
+ unsigned int key = usage->hid & HID_USAGE;
+
+ if (key >= ARRAY_SIZE(sixaxis_keymap))
+ return -1;
+
+ key = navigation_keymap[key];
+ if (!key)
+ return -1;
+
+ hid_map_usage_clear(hi, usage, bit, max, EV_KEY, key);
+ return 1;
+ } else if (usage->hid == HID_GD_POINTER) {
+ /* See comment in sixaxis_mapping, basically the L2 (and R2)
+ * triggers are reported through GD Pointer.
+ * In addition we ignore any analog button 'axes' and only
+ * support digital buttons.
+ */
+ switch (usage->usage_index) {
+ case 8: /* L2 */
+ usage->hid = HID_GD_Z;
+ break;
+ default:
+ return -1;
+ }
+
+ hid_map_usage_clear(hi, usage, bit, max, EV_ABS, usage->hid & 0xf);
+ return 1;
+ } else if ((usage->hid & HID_USAGE_PAGE) == HID_UP_GENDESK) {
+ unsigned int abs = usage->hid & HID_USAGE;
+
+ if (abs >= ARRAY_SIZE(navigation_absmap))
+ return -1;
+
+ abs = navigation_absmap[abs];
+
+ hid_map_usage_clear(hi, usage, bit, max, EV_ABS, abs);
+ return 1;
+ }
+
+ return -1;
+}
+
+
+static int sixaxis_mapping(struct hid_device *hdev, struct hid_input *hi,
+ struct hid_field *field, struct hid_usage *usage,
+ unsigned long **bit, int *max)
+{
+ if ((usage->hid & HID_USAGE_PAGE) == HID_UP_BUTTON) {
+ unsigned int key = usage->hid & HID_USAGE;
+
+ if (key >= ARRAY_SIZE(sixaxis_keymap))
+ return -1;
+
+ key = sixaxis_keymap[key];
+ hid_map_usage_clear(hi, usage, bit, max, EV_KEY, key);
+ return 1;
+ } else if (usage->hid == HID_GD_POINTER) {
+ /* The DS3 provides analog values for most buttons and even
+ * for HAT axes through GD Pointer. L2 and R2 are reported
+ * among these as well instead of as GD Z / RZ. Remap L2
+ * and R2 and ignore other analog 'button axes' as there is
+ * no good way for reporting them.
+ */
+ switch (usage->usage_index) {
+ case 8: /* L2 */
+ usage->hid = HID_GD_Z;
+ break;
+ case 9: /* R2 */
+ usage->hid = HID_GD_RZ;
+ break;
+ default:
+ return -1;
+ }
+
+ hid_map_usage_clear(hi, usage, bit, max, EV_ABS, usage->hid & 0xf);
+ return 1;
+ } else if ((usage->hid & HID_USAGE_PAGE) == HID_UP_GENDESK) {
+ unsigned int abs = usage->hid & HID_USAGE;
+
+ if (abs >= ARRAY_SIZE(sixaxis_absmap))
+ return -1;
+
+ abs = sixaxis_absmap[abs];
+
+ hid_map_usage_clear(hi, usage, bit, max, EV_ABS, abs);
+ return 1;
+ }
+
+ return -1;
+}
+
+static int ds4_mapping(struct hid_device *hdev, struct hid_input *hi,
+ struct hid_field *field, struct hid_usage *usage,
+ unsigned long **bit, int *max)
+{
+ if ((usage->hid & HID_USAGE_PAGE) == HID_UP_BUTTON) {
+ unsigned int key = usage->hid & HID_USAGE;
+
+ if (key >= ARRAY_SIZE(ds4_keymap))
+ return -1;
+
+ key = ds4_keymap[key];
+ hid_map_usage_clear(hi, usage, bit, max, EV_KEY, key);
+ return 1;
+ } else if ((usage->hid & HID_USAGE_PAGE) == HID_UP_GENDESK) {
+ unsigned int abs = usage->hid & HID_USAGE;
+
+ /* Let the HID parser deal with the HAT. */
+ if (usage->hid == HID_GD_HATSWITCH)
+ return 0;
+
+ if (abs >= ARRAY_SIZE(ds4_absmap))
+ return -1;
+
+ abs = ds4_absmap[abs];
+ hid_map_usage_clear(hi, usage, bit, max, EV_ABS, abs);
+ return 1;
+ }
+
+ return 0;
+}
+
static u8 *sony_report_fixup(struct hid_device *hdev, u8 *rdesc,
unsigned int *rsize)
{
@@ -1153,30 +827,9 @@ static u8 *sony_report_fixup(struct hid_device *hdev, u8 *rdesc,
rdesc[55] = 0x06;
}
- /*
- * The default Dualshock 4 USB descriptor doesn't assign
- * the gyroscope values to corresponding axes so we need a
- * modified one.
- */
- if (sc->quirks & DUALSHOCK4_CONTROLLER_USB) {
- hid_info(hdev, "Using modified Dualshock 4 report descriptor with gyroscope axes\n");
- rdesc = dualshock4_usb_rdesc;
- *rsize = sizeof(dualshock4_usb_rdesc);
- } else if (sc->quirks & DUALSHOCK4_CONTROLLER_BT) {
- hid_info(hdev, "Using modified Dualshock 4 Bluetooth report descriptor\n");
- rdesc = dualshock4_bt_rdesc;
- *rsize = sizeof(dualshock4_bt_rdesc);
- }
-
- if (sc->quirks & SIXAXIS_CONTROLLER)
- return sixaxis_fixup(hdev, rdesc, rsize);
-
if (sc->quirks & MOTION_CONTROLLER)
return motion_fixup(hdev, rdesc, rsize);
- if (sc->quirks & NAVIGATION_CONTROLLER)
- return navigation_fixup(hdev, rdesc, rsize);
-
if (sc->quirks & PS3REMOTE)
return ps3remote_fixup(hdev, rdesc, rsize);
@@ -1214,6 +867,23 @@ static void sixaxis_parse_report(struct sony_sc *sc, u8 *rd, int size)
sc->battery_capacity = battery_capacity;
sc->battery_charging = battery_charging;
spin_unlock_irqrestore(&sc->lock, flags);
+
+ if (sc->quirks & SIXAXIS_CONTROLLER) {
+ int val;
+
+ offset = SIXAXIS_INPUT_REPORT_ACC_X_OFFSET;
+ val = ((rd[offset+1] << 8) | rd[offset]) - 511;
+ input_report_abs(sc->sensor_dev, ABS_X, val);
+
+ /* Y and Z are swapped and inversed */
+ val = 511 - ((rd[offset+5] << 8) | rd[offset+4]);
+ input_report_abs(sc->sensor_dev, ABS_Y, val);
+
+ val = 511 - ((rd[offset+3] << 8) | rd[offset+2]);
+ input_report_abs(sc->sensor_dev, ABS_Z, val);
+
+ input_sync(sc->sensor_dev);
+ }
}
static void dualshock4_parse_report(struct sony_sc *sc, u8 *rd, int size)
@@ -1222,19 +892,111 @@ static void dualshock4_parse_report(struct sony_sc *sc, u8 *rd, int size)
struct hid_input, list);
struct input_dev *input_dev = hidinput->input;
unsigned long flags;
- int n, offset;
+ int n, m, offset, num_touch_data, max_touch_data;
u8 cable_state, battery_capacity, battery_charging;
+ u16 timestamp;
+
+ /* When using Bluetooth the header is 2 bytes longer, so skip these. */
+ int data_offset = (sc->quirks & DUALSHOCK4_CONTROLLER_BT) ? 2 : 0;
+
+ /* Second bit of third button byte is for the touchpad button. */
+ offset = data_offset + DS4_INPUT_REPORT_BUTTON_OFFSET;
+ input_report_key(sc->touchpad, BTN_LEFT, rd[offset+2] & 0x2);
/*
- * Battery and touchpad data starts at byte 30 in the USB report and
- * 32 in Bluetooth report.
+ * The default behavior of the Dualshock 4 is to send reports using
+ * report type 1 when running over Bluetooth. However, when feature
+ * report 2 is requested during the controller initialization it starts
+ * sending input reports in report 17. Since report 17 is undefined
+ * in the default HID descriptor, the HID layer won't generate events.
+ * While it is possible (and this was done before) to fixup the HID
+ * descriptor to add this mapping, it was better to do this manually.
+ * The reason is there were various pieces software both open and closed
+ * source, relying on the descriptors to be the same across various
+ * operating systems. If the descriptors wouldn't match some
+ * applications e.g. games on Wine would not be able to function due
+ * to different descriptors, which such applications are not parsing.
*/
- offset = (sc->quirks & DUALSHOCK4_CONTROLLER_USB) ? 30 : 32;
+ if (rd[0] == 17) {
+ int value;
+
+ offset = data_offset + DS4_INPUT_REPORT_AXIS_OFFSET;
+ input_report_abs(input_dev, ABS_X, rd[offset]);
+ input_report_abs(input_dev, ABS_Y, rd[offset+1]);
+ input_report_abs(input_dev, ABS_RX, rd[offset+2]);
+ input_report_abs(input_dev, ABS_RY, rd[offset+3]);
+
+ value = rd[offset+4] & 0xf;
+ if (value > 7)
+ value = 8; /* Center 0, 0 */
+ input_report_abs(input_dev, ABS_HAT0X, ds4_hat_mapping[value].x);
+ input_report_abs(input_dev, ABS_HAT0Y, ds4_hat_mapping[value].y);
+
+ input_report_key(input_dev, BTN_WEST, rd[offset+4] & 0x10);
+ input_report_key(input_dev, BTN_SOUTH, rd[offset+4] & 0x20);
+ input_report_key(input_dev, BTN_EAST, rd[offset+4] & 0x40);
+ input_report_key(input_dev, BTN_NORTH, rd[offset+4] & 0x80);
+
+ input_report_key(input_dev, BTN_TL, rd[offset+5] & 0x1);
+ input_report_key(input_dev, BTN_TR, rd[offset+5] & 0x2);
+ input_report_key(input_dev, BTN_TL2, rd[offset+5] & 0x4);
+ input_report_key(input_dev, BTN_TR2, rd[offset+5] & 0x8);
+ input_report_key(input_dev, BTN_SELECT, rd[offset+5] & 0x10);
+ input_report_key(input_dev, BTN_START, rd[offset+5] & 0x20);
+ input_report_key(input_dev, BTN_THUMBL, rd[offset+5] & 0x40);
+ input_report_key(input_dev, BTN_THUMBR, rd[offset+5] & 0x80);
+
+ input_report_key(input_dev, BTN_MODE, rd[offset+6] & 0x1);
+
+ input_report_abs(input_dev, ABS_Z, rd[offset+7]);
+ input_report_abs(input_dev, ABS_RZ, rd[offset+8]);
+
+ input_sync(input_dev);
+ }
+
+ /* Convert timestamp (in 5.33us unit) to timestamp_us */
+ offset = data_offset + DS4_INPUT_REPORT_TIMESTAMP_OFFSET;
+ timestamp = get_unaligned_le16(&rd[offset]);
+ if (!sc->timestamp_initialized) {
+ sc->timestamp_us = ((unsigned int)timestamp * 16) / 3;
+ sc->timestamp_initialized = true;
+ } else {
+ u16 delta;
+
+ if (sc->prev_timestamp > timestamp)
+ delta = (U16_MAX - sc->prev_timestamp + timestamp + 1);
+ else
+ delta = timestamp - sc->prev_timestamp;
+ sc->timestamp_us += (delta * 16) / 3;
+ }
+ sc->prev_timestamp = timestamp;
+ input_event(sc->sensor_dev, EV_MSC, MSC_TIMESTAMP, sc->timestamp_us);
+
+ offset = data_offset + DS4_INPUT_REPORT_GYRO_X_OFFSET;
+ for (n = 0; n < 6; n++) {
+ /* Store data in int for more precision during mult_frac. */
+ int raw_data = (short)((rd[offset+1] << 8) | rd[offset]);
+ struct ds4_calibration_data *calib = &sc->ds4_calib_data[n];
+
+ /* High precision is needed during calibration, but the
+ * calibrated values are within 32-bit.
+ * Note: we swap numerator 'x' and 'numer' in mult_frac for
+ * precision reasons so we don't need 64-bit.
+ */
+ int calib_data = mult_frac(calib->sens_numer,
+ raw_data - calib->bias,
+ calib->sens_denom);
+
+ input_report_abs(sc->sensor_dev, calib->abs_code, calib_data);
+ offset += 2;
+ }
+ input_sync(sc->sensor_dev);
/*
- * The lower 4 bits of byte 30 contain the battery level
+ * The lower 4 bits of byte 30 (or 32 for BT) contain the battery level
* and the 5th bit contains the USB cable state.
*/
+ offset = data_offset + DS4_INPUT_REPORT_BATTERY_OFFSET;
cable_state = (rd[offset] >> 4) & 0x01;
battery_capacity = rd[offset] & 0x0F;
@@ -1261,30 +1023,52 @@ static void dualshock4_parse_report(struct sony_sc *sc, u8 *rd, int size)
sc->battery_charging = battery_charging;
spin_unlock_irqrestore(&sc->lock, flags);
- offset += 5;
-
/*
- * The Dualshock 4 multi-touch trackpad data starts at offset 35 on USB
- * and 37 on Bluetooth.
- * The first 7 bits of the first byte is a counter and bit 8 is a touch
- * indicator that is 0 when pressed and 1 when not pressed.
- * The next 3 bytes are two 12 bit touch coordinates, X and Y.
- * The data for the second touch is in the same format and immediatly
- * follows the data for the first.
+ * The Dualshock 4 multi-touch trackpad data starts at offset 33 on USB
+ * and 35 on Bluetooth.
+ * The first byte indicates the number of touch data in the report.
+ * Trackpad data starts 2 bytes later (e.g. 35 for USB).
*/
- for (n = 0; n < 2; n++) {
- u16 x, y;
+ offset = data_offset + DS4_INPUT_REPORT_TOUCHPAD_OFFSET;
+ max_touch_data = (sc->quirks & DUALSHOCK4_CONTROLLER_BT) ? 4 : 3;
+ if (rd[offset] > 0 && rd[offset] <= max_touch_data)
+ num_touch_data = rd[offset];
+ else
+ num_touch_data = 1;
+ offset += 1;
- x = rd[offset+1] | ((rd[offset+2] & 0xF) << 8);
- y = ((rd[offset+2] & 0xF0) >> 4) | (rd[offset+3] << 4);
+ for (m = 0; m < num_touch_data; m++) {
+ /* Skip past timestamp */
+ offset += 1;
- input_mt_slot(input_dev, n);
- input_mt_report_slot_state(input_dev, MT_TOOL_FINGER,
- !(rd[offset] >> 7));
- input_report_abs(input_dev, ABS_MT_POSITION_X, x);
- input_report_abs(input_dev, ABS_MT_POSITION_Y, y);
+ /*
+ * The first 7 bits of the first byte is a counter and bit 8 is
+ * a touch indicator that is 0 when pressed and 1 when not
+ * pressed.
+ * The next 3 bytes are two 12 bit touch coordinates, X and Y.
+ * The data for the second touch is in the same format and
+ * immediately follows the data for the first.
+ */
+ for (n = 0; n < 2; n++) {
+ u16 x, y;
+ bool active;
- offset += 4;
+ x = rd[offset+1] | ((rd[offset+2] & 0xF) << 8);
+ y = ((rd[offset+2] & 0xF0) >> 4) | (rd[offset+3] << 4);
+
+ active = !(rd[offset] >> 7);
+ input_mt_slot(sc->touchpad, n);
+ input_mt_report_slot_state(sc->touchpad, MT_TOOL_FINGER, active);
+
+ if (active) {
+ input_report_abs(sc->touchpad, ABS_MT_POSITION_X, x);
+ input_report_abs(sc->touchpad, ABS_MT_POSITION_Y, y);
+ }
+
+ offset += 4;
+ }
+ input_mt_sync_frame(sc->touchpad);
+ input_sync(sc->touchpad);
}
}
@@ -1320,15 +1104,87 @@ static int sony_raw_event(struct hid_device *hdev, struct hid_report *report,
} else if ((sc->quirks & NAVIGATION_CONTROLLER) && rd[0] == 0x01 &&
size == 49) {
sixaxis_parse_report(sc, rd, size);
- } else if (((sc->quirks & DUALSHOCK4_CONTROLLER_USB) && rd[0] == 0x01 &&
- size == 64) || ((sc->quirks & DUALSHOCK4_CONTROLLER_BT)
- && rd[0] == 0x11 && size == 78)) {
+ } else if ((sc->quirks & DUALSHOCK4_CONTROLLER_USB) && rd[0] == 0x01 &&
+ size == 64) {
+ dualshock4_parse_report(sc, rd, size);
+ } else if (((sc->quirks & DUALSHOCK4_CONTROLLER_BT) && rd[0] == 0x11 &&
+ size == 78)) {
+ /* CRC check */
+ u8 bthdr = 0xA1;
+ u32 crc;
+ u32 report_crc;
+
+ crc = crc32_le(0xFFFFFFFF, &bthdr, 1);
+ crc = ~crc32_le(crc, rd, DS4_INPUT_REPORT_0x11_SIZE-4);
+ report_crc = get_unaligned_le32(&rd[DS4_INPUT_REPORT_0x11_SIZE-4]);
+ if (crc != report_crc) {
+ hid_dbg(sc->hdev, "DualShock 4 input report's CRC check failed, received crc 0x%0x != 0x%0x\n",
+ report_crc, crc);
+ return -EILSEQ;
+ }
+
+ dualshock4_parse_report(sc, rd, size);
+ } else if ((sc->quirks & DUALSHOCK4_DONGLE) && rd[0] == 0x01 &&
+ size == 64) {
+ unsigned long flags;
+ enum ds4_dongle_state dongle_state;
+
+ /*
+ * In the case of a DS4 USB dongle, bit[2] of byte 31 indicates
+ * if a DS4 is actually connected (indicated by '0').
+ * For non-dongle, this bit is always 0 (connected).
+ */
+ bool connected = (rd[31] & 0x04) ? false : true;
+
+ spin_lock_irqsave(&sc->lock, flags);
+ dongle_state = sc->ds4_dongle_state;
+ spin_unlock_irqrestore(&sc->lock, flags);
+
+ /*
+ * The dongle always sends input reports even when no
+ * DS4 is attached. When a DS4 is connected, we need to
+ * obtain calibration data before we can use it.
+ * The code below tracks dongle state and kicks of
+ * calibration when needed and only allows us to process
+ * input if a DS4 is actually connected.
+ */
+ if (dongle_state == DONGLE_DISCONNECTED && connected) {
+ hid_info(sc->hdev, "DualShock 4 USB dongle: controller connected\n");
+ sony_set_leds(sc);
+
+ spin_lock_irqsave(&sc->lock, flags);
+ sc->ds4_dongle_state = DONGLE_CALIBRATING;
+ spin_unlock_irqrestore(&sc->lock, flags);
+
+ sony_schedule_work(sc, SONY_WORKER_HOTPLUG);
+
+ /* Don't process the report since we don't have
+ * calibration data, but let hidraw have it anyway.
+ */
+ return 0;
+ } else if ((dongle_state == DONGLE_CONNECTED ||
+ dongle_state == DONGLE_DISABLED) && !connected) {
+ hid_info(sc->hdev, "DualShock 4 USB dongle: controller disconnected\n");
+
+ spin_lock_irqsave(&sc->lock, flags);
+ sc->ds4_dongle_state = DONGLE_DISCONNECTED;
+ spin_unlock_irqrestore(&sc->lock, flags);
+
+ /* Return 0, so hidraw can get the report. */
+ return 0;
+ } else if (dongle_state == DONGLE_CALIBRATING ||
+ dongle_state == DONGLE_DISABLED ||
+ dongle_state == DONGLE_DISCONNECTED) {
+ /* Return 0, so hidraw can get the report. */
+ return 0;
+ }
+
dualshock4_parse_report(sc, rd, size);
}
if (sc->defer_initialization) {
sc->defer_initialization = 0;
- sony_schedule_work(sc);
+ sony_schedule_work(sc, SONY_WORKER_STATE);
}
return 0;
@@ -1366,49 +1222,189 @@ static int sony_mapping(struct hid_device *hdev, struct hid_input *hi,
if (sc->quirks & PS3REMOTE)
return ps3remote_mapping(hdev, hi, field, usage, bit, max);
+ if (sc->quirks & NAVIGATION_CONTROLLER)
+ return navigation_mapping(hdev, hi, field, usage, bit, max);
+
+ if (sc->quirks & SIXAXIS_CONTROLLER)
+ return sixaxis_mapping(hdev, hi, field, usage, bit, max);
+
+ if (sc->quirks & DUALSHOCK4_CONTROLLER)
+ return ds4_mapping(hdev, hi, field, usage, bit, max);
+
+
/* Let hid-core decide for the others */
return 0;
}
-static int sony_register_touchpad(struct hid_input *hi, int touch_count,
+static int sony_register_touchpad(struct sony_sc *sc, int touch_count,
int w, int h)
{
- struct input_dev *input_dev = hi->input;
+ size_t name_sz;
+ char *name;
int ret;
- ret = input_mt_init_slots(input_dev, touch_count, 0);
- if (ret < 0)
- return ret;
+ sc->touchpad = input_allocate_device();
+ if (!sc->touchpad)
+ return -ENOMEM;
- input_set_abs_params(input_dev, ABS_MT_POSITION_X, 0, w, 0, 0);
- input_set_abs_params(input_dev, ABS_MT_POSITION_Y, 0, h, 0, 0);
+ input_set_drvdata(sc->touchpad, sc);
+ sc->touchpad->dev.parent = &sc->hdev->dev;
+ sc->touchpad->phys = sc->hdev->phys;
+ sc->touchpad->uniq = sc->hdev->uniq;
+ sc->touchpad->id.bustype = sc->hdev->bus;
+ sc->touchpad->id.vendor = sc->hdev->vendor;
+ sc->touchpad->id.product = sc->hdev->product;
+ sc->touchpad->id.version = sc->hdev->version;
+
+ /* Append a suffix to the controller name as there are various
+ * DS4 compatible non-Sony devices with different names.
+ */
+ name_sz = strlen(sc->hdev->name) + sizeof(DS4_TOUCHPAD_SUFFIX);
+ name = kzalloc(name_sz, GFP_KERNEL);
+ if (!name) {
+ ret = -ENOMEM;
+ goto err;
+ }
+ snprintf(name, name_sz, "%s" DS4_TOUCHPAD_SUFFIX, sc->hdev->name);
+ sc->touchpad->name = name;
+
+ ret = input_mt_init_slots(sc->touchpad, touch_count, INPUT_MT_POINTER);
+ if (ret < 0)
+ goto err;
+
+ /* We map the button underneath the touchpad to BTN_LEFT. */
+ __set_bit(EV_KEY, sc->touchpad->evbit);
+ __set_bit(BTN_LEFT, sc->touchpad->keybit);
+ __set_bit(INPUT_PROP_BUTTONPAD, sc->touchpad->propbit);
+
+ input_set_abs_params(sc->touchpad, ABS_MT_POSITION_X, 0, w, 0, 0);
+ input_set_abs_params(sc->touchpad, ABS_MT_POSITION_Y, 0, h, 0, 0);
+
+ ret = input_register_device(sc->touchpad);
+ if (ret < 0)
+ goto err;
return 0;
+
+err:
+ kfree(sc->touchpad->name);
+ sc->touchpad->name = NULL;
+
+ input_free_device(sc->touchpad);
+ sc->touchpad = NULL;
+
+ return ret;
}
-static int sony_input_configured(struct hid_device *hdev,
- struct hid_input *hidinput)
+static void sony_unregister_touchpad(struct sony_sc *sc)
{
- struct sony_sc *sc = hid_get_drvdata(hdev);
- int ret;
+ if (!sc->touchpad)
+ return;
- /*
- * The Dualshock 4 touchpad supports 2 touches and has a
- * resolution of 1920x942 (44.86 dots/mm).
+ kfree(sc->touchpad->name);
+ sc->touchpad->name = NULL;
+
+ input_unregister_device(sc->touchpad);
+ sc->touchpad = NULL;
+}
+
+static int sony_register_sensors(struct sony_sc *sc)
+{
+ size_t name_sz;
+ char *name;
+ int ret;
+ int range;
+
+ sc->sensor_dev = input_allocate_device();
+ if (!sc->sensor_dev)
+ return -ENOMEM;
+
+ input_set_drvdata(sc->sensor_dev, sc);
+ sc->sensor_dev->dev.parent = &sc->hdev->dev;
+ sc->sensor_dev->phys = sc->hdev->phys;
+ sc->sensor_dev->uniq = sc->hdev->uniq;
+ sc->sensor_dev->id.bustype = sc->hdev->bus;
+ sc->sensor_dev->id.vendor = sc->hdev->vendor;
+ sc->sensor_dev->id.product = sc->hdev->product;
+ sc->sensor_dev->id.version = sc->hdev->version;
+
+ /* Append a suffix to the controller name as there are various
+ * DS4 compatible non-Sony devices with different names.
*/
- if (sc->quirks & DUALSHOCK4_CONTROLLER) {
- ret = sony_register_touchpad(hidinput, 2, 1920, 942);
- if (ret) {
- hid_err(sc->hdev,
- "Unable to initialize multi-touch slots: %d\n",
- ret);
- return ret;
- }
+ name_sz = strlen(sc->hdev->name) + sizeof(SENSOR_SUFFIX);
+ name = kzalloc(name_sz, GFP_KERNEL);
+ if (!name) {
+ ret = -ENOMEM;
+ goto err;
+ }
+ snprintf(name, name_sz, "%s" SENSOR_SUFFIX, sc->hdev->name);
+ sc->sensor_dev->name = name;
+
+ if (sc->quirks & SIXAXIS_CONTROLLER) {
+ /* For the DS3 we only support the accelerometer, which works
+ * quite well even without calibration. The device also has
+ * a 1-axis gyro, but it is very difficult to manage from within
+ * the driver even to get data, the sensor is inaccurate and
+ * the behavior is very different between hardware revisions.
+ */
+ input_set_abs_params(sc->sensor_dev, ABS_X, -512, 511, 4, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_Y, -512, 511, 4, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_Z, -512, 511, 4, 0);
+ input_abs_set_res(sc->sensor_dev, ABS_X, SIXAXIS_ACC_RES_PER_G);
+ input_abs_set_res(sc->sensor_dev, ABS_Y, SIXAXIS_ACC_RES_PER_G);
+ input_abs_set_res(sc->sensor_dev, ABS_Z, SIXAXIS_ACC_RES_PER_G);
+ } else if (sc->quirks & DUALSHOCK4_CONTROLLER) {
+ range = DS4_ACC_RES_PER_G*4;
+ input_set_abs_params(sc->sensor_dev, ABS_X, -range, range, 16, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_Y, -range, range, 16, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_Z, -range, range, 16, 0);
+ input_abs_set_res(sc->sensor_dev, ABS_X, DS4_ACC_RES_PER_G);
+ input_abs_set_res(sc->sensor_dev, ABS_Y, DS4_ACC_RES_PER_G);
+ input_abs_set_res(sc->sensor_dev, ABS_Z, DS4_ACC_RES_PER_G);
+
+ range = DS4_GYRO_RES_PER_DEG_S*2048;
+ input_set_abs_params(sc->sensor_dev, ABS_RX, -range, range, 16, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_RY, -range, range, 16, 0);
+ input_set_abs_params(sc->sensor_dev, ABS_RZ, -range, range, 16, 0);
+ input_abs_set_res(sc->sensor_dev, ABS_RX, DS4_GYRO_RES_PER_DEG_S);
+ input_abs_set_res(sc->sensor_dev, ABS_RY, DS4_GYRO_RES_PER_DEG_S);
+ input_abs_set_res(sc->sensor_dev, ABS_RZ, DS4_GYRO_RES_PER_DEG_S);
+
+ __set_bit(EV_MSC, sc->sensor_dev->evbit);
+ __set_bit(MSC_TIMESTAMP, sc->sensor_dev->mscbit);
}
+ __set_bit(INPUT_PROP_ACCELEROMETER, sc->sensor_dev->propbit);
+
+ ret = input_register_device(sc->sensor_dev);
+ if (ret < 0)
+ goto err;
+
return 0;
+
+err:
+ kfree(sc->sensor_dev->name);
+ sc->sensor_dev->name = NULL;
+
+ input_free_device(sc->sensor_dev);
+ sc->sensor_dev = NULL;
+
+ return ret;
}
+static void sony_unregister_sensors(struct sony_sc *sc)
+{
+ if (!sc->sensor_dev)
+ return;
+
+ kfree(sc->sensor_dev->name);
+ sc->sensor_dev->name = NULL;
+
+ input_unregister_device(sc->sensor_dev);
+ sc->sensor_dev = NULL;
+}
+
+
/*
* Sending HID_REQ_GET_REPORT changes the operation mode of the ps3 controller
* to "operational". Without this, the ps3 controller will not report any
@@ -1474,26 +1470,176 @@ static int sixaxis_set_operational_bt(struct hid_device *hdev)
}
/*
- * Requesting feature report 0x02 in Bluetooth mode changes the state of the
- * controller so that it sends full input reports of type 0x11.
+ * Request DS4 calibration data for the motion sensors.
+ * For Bluetooth this also affects the operating mode (see below).
*/
-static int dualshock4_set_operational_bt(struct hid_device *hdev)
+static int dualshock4_get_calibration_data(struct sony_sc *sc)
{
u8 *buf;
int ret;
+ short gyro_pitch_bias, gyro_pitch_plus, gyro_pitch_minus;
+ short gyro_yaw_bias, gyro_yaw_plus, gyro_yaw_minus;
+ short gyro_roll_bias, gyro_roll_plus, gyro_roll_minus;
+ short gyro_speed_plus, gyro_speed_minus;
+ short acc_x_plus, acc_x_minus;
+ short acc_y_plus, acc_y_minus;
+ short acc_z_plus, acc_z_minus;
+ int speed_2x;
+ int range_2g;
- buf = kmalloc(DS4_REPORT_0x02_SIZE, GFP_KERNEL);
- if (!buf)
- return -ENOMEM;
+ /* For Bluetooth we use a different request, which supports CRC.
+ * Note: in Bluetooth mode feature report 0x02 also changes the state
+ * of the controller, so that it sends input reports of type 0x11.
+ */
+ if (sc->quirks & (DUALSHOCK4_CONTROLLER_USB | DUALSHOCK4_DONGLE)) {
+ buf = kmalloc(DS4_FEATURE_REPORT_0x02_SIZE, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
- ret = hid_hw_raw_request(hdev, 0x02, buf, DS4_REPORT_0x02_SIZE,
- HID_FEATURE_REPORT, HID_REQ_GET_REPORT);
+ ret = hid_hw_raw_request(sc->hdev, 0x02, buf,
+ DS4_FEATURE_REPORT_0x02_SIZE,
+ HID_FEATURE_REPORT,
+ HID_REQ_GET_REPORT);
+ if (ret < 0)
+ goto err_stop;
+ } else {
+ u8 bthdr = 0xA3;
+ u32 crc;
+ u32 report_crc;
+ int retries;
+ buf = kmalloc(DS4_FEATURE_REPORT_0x05_SIZE, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ for (retries = 0; retries < 3; retries++) {
+ ret = hid_hw_raw_request(sc->hdev, 0x05, buf,
+ DS4_FEATURE_REPORT_0x05_SIZE,
+ HID_FEATURE_REPORT,
+ HID_REQ_GET_REPORT);
+ if (ret < 0)
+ goto err_stop;
+
+ /* CRC check */
+ crc = crc32_le(0xFFFFFFFF, &bthdr, 1);
+ crc = ~crc32_le(crc, buf, DS4_FEATURE_REPORT_0x05_SIZE-4);
+ report_crc = get_unaligned_le32(&buf[DS4_FEATURE_REPORT_0x05_SIZE-4]);
+ if (crc != report_crc) {
+ hid_warn(sc->hdev, "DualShock 4 calibration report's CRC check failed, received crc 0x%0x != 0x%0x\n",
+ report_crc, crc);
+ if (retries < 2) {
+ hid_warn(sc->hdev, "Retrying DualShock 4 get calibration report request\n");
+ continue;
+ } else {
+ ret = -EILSEQ;
+ goto err_stop;
+ }
+ } else {
+ break;
+ }
+ }
+ }
+
+ gyro_pitch_bias = get_unaligned_le16(&buf[1]);
+ gyro_yaw_bias = get_unaligned_le16(&buf[3]);
+ gyro_roll_bias = get_unaligned_le16(&buf[5]);
+ if (sc->quirks & DUALSHOCK4_CONTROLLER_USB) {
+ gyro_pitch_plus = get_unaligned_le16(&buf[7]);
+ gyro_pitch_minus = get_unaligned_le16(&buf[9]);
+ gyro_yaw_plus = get_unaligned_le16(&buf[11]);
+ gyro_yaw_minus = get_unaligned_le16(&buf[13]);
+ gyro_roll_plus = get_unaligned_le16(&buf[15]);
+ gyro_roll_minus = get_unaligned_le16(&buf[17]);
+ } else {
+ /* BT + Dongle */
+ gyro_pitch_plus = get_unaligned_le16(&buf[7]);
+ gyro_yaw_plus = get_unaligned_le16(&buf[9]);
+ gyro_roll_plus = get_unaligned_le16(&buf[11]);
+ gyro_pitch_minus = get_unaligned_le16(&buf[13]);
+ gyro_yaw_minus = get_unaligned_le16(&buf[15]);
+ gyro_roll_minus = get_unaligned_le16(&buf[17]);
+ }
+ gyro_speed_plus = get_unaligned_le16(&buf[19]);
+ gyro_speed_minus = get_unaligned_le16(&buf[21]);
+ acc_x_plus = get_unaligned_le16(&buf[23]);
+ acc_x_minus = get_unaligned_le16(&buf[25]);
+ acc_y_plus = get_unaligned_le16(&buf[27]);
+ acc_y_minus = get_unaligned_le16(&buf[29]);
+ acc_z_plus = get_unaligned_le16(&buf[31]);
+ acc_z_minus = get_unaligned_le16(&buf[33]);
+
+ /* Set gyroscope calibration and normalization parameters.
+ * Data values will be normalized to 1/DS4_GYRO_RES_PER_DEG_S degree/s.
+ */
+ speed_2x = (gyro_speed_plus + gyro_speed_minus);
+ sc->ds4_calib_data[0].abs_code = ABS_RX;
+ sc->ds4_calib_data[0].bias = gyro_pitch_bias;
+ sc->ds4_calib_data[0].sens_numer = speed_2x*DS4_GYRO_RES_PER_DEG_S;
+ sc->ds4_calib_data[0].sens_denom = gyro_pitch_plus - gyro_pitch_minus;
+
+ sc->ds4_calib_data[1].abs_code = ABS_RY;
+ sc->ds4_calib_data[1].bias = gyro_yaw_bias;
+ sc->ds4_calib_data[1].sens_numer = speed_2x*DS4_GYRO_RES_PER_DEG_S;
+ sc->ds4_calib_data[1].sens_denom = gyro_yaw_plus - gyro_yaw_minus;
+
+ sc->ds4_calib_data[2].abs_code = ABS_RZ;
+ sc->ds4_calib_data[2].bias = gyro_roll_bias;
+ sc->ds4_calib_data[2].sens_numer = speed_2x*DS4_GYRO_RES_PER_DEG_S;
+ sc->ds4_calib_data[2].sens_denom = gyro_roll_plus - gyro_roll_minus;
+
+ /* Set accelerometer calibration and normalization parameters.
+ * Data values will be normalized to 1/DS4_ACC_RES_PER_G G.
+ */
+ range_2g = acc_x_plus - acc_x_minus;
+ sc->ds4_calib_data[3].abs_code = ABS_X;
+ sc->ds4_calib_data[3].bias = acc_x_plus - range_2g / 2;
+ sc->ds4_calib_data[3].sens_numer = 2*DS4_ACC_RES_PER_G;
+ sc->ds4_calib_data[3].sens_denom = range_2g;
+
+ range_2g = acc_y_plus - acc_y_minus;
+ sc->ds4_calib_data[4].abs_code = ABS_Y;
+ sc->ds4_calib_data[4].bias = acc_y_plus - range_2g / 2;
+ sc->ds4_calib_data[4].sens_numer = 2*DS4_ACC_RES_PER_G;
+ sc->ds4_calib_data[4].sens_denom = range_2g;
+
+ range_2g = acc_z_plus - acc_z_minus;
+ sc->ds4_calib_data[5].abs_code = ABS_Z;
+ sc->ds4_calib_data[5].bias = acc_z_plus - range_2g / 2;
+ sc->ds4_calib_data[5].sens_numer = 2*DS4_ACC_RES_PER_G;
+ sc->ds4_calib_data[5].sens_denom = range_2g;
+
+err_stop:
kfree(buf);
-
return ret;
}
+static void dualshock4_calibration_work(struct work_struct *work)
+{
+ struct sony_sc *sc = container_of(work, struct sony_sc, hotplug_worker);
+ unsigned long flags;
+ enum ds4_dongle_state dongle_state;
+ int ret;
+
+ ret = dualshock4_get_calibration_data(sc);
+ if (ret < 0) {
+ /* This call is very unlikely to fail for the dongle. When it
+ * fails we are probably in a very bad state, so mark the
+ * dongle as disabled. We will re-enable the dongle if a new
+ * DS4 hotplug is detect from sony_raw_event as any issues
+ * are likely resolved then (the dongle is quite stupid).
+ */
+ hid_err(sc->hdev, "DualShock 4 USB dongle: calibration failed, disabling device\n");
+ dongle_state = DONGLE_DISABLED;
+ } else {
+ hid_info(sc->hdev, "DualShock 4 USB dongle: calibration completed\n");
+ dongle_state = DONGLE_CONNECTED;
+ }
+
+ spin_lock_irqsave(&sc->lock, flags);
+ sc->ds4_dongle_state = dongle_state;
+ spin_unlock_irqrestore(&sc->lock, flags);
+}
+
static void sixaxis_set_leds_from_id(struct sony_sc *sc)
{
static const u8 sixaxis_leds[10][4] = {
@@ -1524,10 +1670,10 @@ static void dualshock4_set_leds_from_id(struct sony_sc *sc)
{
/* The first 4 color/index entries match what the PS4 assigns */
static const u8 color_code[7][3] = {
- /* Blue */ { 0x00, 0x00, 0x01 },
- /* Red */ { 0x01, 0x00, 0x00 },
- /* Green */ { 0x00, 0x01, 0x00 },
- /* Pink */ { 0x02, 0x00, 0x01 },
+ /* Blue */ { 0x00, 0x00, 0x40 },
+ /* Red */ { 0x40, 0x00, 0x00 },
+ /* Green */ { 0x00, 0x40, 0x00 },
+ /* Pink */ { 0x20, 0x00, 0x20 },
/* Orange */ { 0x02, 0x01, 0x00 },
/* Teal */ { 0x00, 0x01, 0x01 },
/* White */ { 0x01, 0x01, 0x01 }
@@ -1568,7 +1714,7 @@ static void buzz_set_leds(struct sony_sc *sc)
static void sony_set_leds(struct sony_sc *sc)
{
if (!(sc->quirks & BUZZ_CONTROLLER))
- sony_schedule_work(sc);
+ sony_schedule_work(sc, SONY_WORKER_STATE);
else
buzz_set_leds(sc);
}
@@ -1679,7 +1825,7 @@ static int sony_led_blink_set(struct led_classdev *led, unsigned long *delay_on,
new_off != drv_data->led_delay_off[n]) {
drv_data->led_delay_on[n] = new_on;
drv_data->led_delay_off[n] = new_off;
- sony_schedule_work(drv_data);
+ sony_schedule_work(drv_data, SONY_WORKER_STATE);
}
return 0;
@@ -1881,26 +2027,24 @@ static void dualshock4_send_output_report(struct sony_sc *sc)
int offset;
/*
- * NOTE: The buf[1] field of the Bluetooth report controls
- * the Dualshock 4 reporting rate.
- *
- * Known values include:
- *
- * 0x80 - 1000hz (full speed)
- * 0xA0 - 31hz
- * 0xB0 - 20hz
- * 0xD0 - 66hz
+ * NOTE: The lower 6 bits of buf[1] field of the Bluetooth report
+ * control the interval at which Dualshock 4 reports data:
+ * 0x00 - 1ms
+ * 0x01 - 1ms
+ * 0x02 - 2ms
+ * 0x3E - 62ms
+ * 0x3F - disabled
*/
- if (sc->quirks & DUALSHOCK4_CONTROLLER_USB) {
- memset(buf, 0, DS4_REPORT_0x05_SIZE);
+ if (sc->quirks & (DUALSHOCK4_CONTROLLER_USB | DUALSHOCK4_DONGLE)) {
+ memset(buf, 0, DS4_OUTPUT_REPORT_0x05_SIZE);
buf[0] = 0x05;
- buf[1] = 0xFF;
+ buf[1] = 0x07; /* blink + LEDs + motor */
offset = 4;
} else {
- memset(buf, 0, DS4_REPORT_0x11_SIZE);
+ memset(buf, 0, DS4_OUTPUT_REPORT_0x11_SIZE);
buf[0] = 0x11;
- buf[1] = 0x80;
- buf[3] = 0x0F;
+ buf[1] = 0xC0 /* HID + CRC */ | sc->ds4_bt_poll_interval;
+ buf[3] = 0x07; /* blink + LEDs + motor */
offset = 6;
}
@@ -1924,11 +2068,18 @@ static void dualshock4_send_output_report(struct sony_sc *sc)
buf[offset++] = sc->led_delay_on[3];
buf[offset++] = sc->led_delay_off[3];
- if (sc->quirks & DUALSHOCK4_CONTROLLER_USB)
- hid_hw_output_report(hdev, buf, DS4_REPORT_0x05_SIZE);
- else
- hid_hw_raw_request(hdev, 0x11, buf, DS4_REPORT_0x11_SIZE,
- HID_OUTPUT_REPORT, HID_REQ_SET_REPORT);
+ if (sc->quirks & (DUALSHOCK4_CONTROLLER_USB | DUALSHOCK4_DONGLE))
+ hid_hw_output_report(hdev, buf, DS4_OUTPUT_REPORT_0x05_SIZE);
+ else {
+ /* CRC generation */
+ u8 bthdr = 0xA2;
+ u32 crc;
+
+ crc = crc32_le(0xFFFFFFFF, &bthdr, 1);
+ crc = ~crc32_le(crc, buf, DS4_OUTPUT_REPORT_0x11_SIZE-4);
+ put_unaligned_le32(crc, &buf[74]);
+ hid_hw_output_report(hdev, buf, DS4_OUTPUT_REPORT_0x11_SIZE);
+ }
}
static void motion_send_output_report(struct sony_sc *sc)
@@ -1972,10 +2123,10 @@ static int sony_allocate_output_report(struct sony_sc *sc)
kmalloc(sizeof(union sixaxis_output_report_01),
GFP_KERNEL);
else if (sc->quirks & DUALSHOCK4_CONTROLLER_BT)
- sc->output_report_dmabuf = kmalloc(DS4_REPORT_0x11_SIZE,
+ sc->output_report_dmabuf = kmalloc(DS4_OUTPUT_REPORT_0x11_SIZE,
GFP_KERNEL);
- else if (sc->quirks & DUALSHOCK4_CONTROLLER_USB)
- sc->output_report_dmabuf = kmalloc(DS4_REPORT_0x05_SIZE,
+ else if (sc->quirks & (DUALSHOCK4_CONTROLLER_USB | DUALSHOCK4_DONGLE))
+ sc->output_report_dmabuf = kmalloc(DS4_OUTPUT_REPORT_0x05_SIZE,
GFP_KERNEL);
else if (sc->quirks & MOTION_CONTROLLER)
sc->output_report_dmabuf = kmalloc(MOTION_REPORT_0x02_SIZE,
@@ -2002,7 +2153,7 @@ static int sony_play_effect(struct input_dev *dev, void *data,
sc->left = effect->u.rumble.strong_magnitude / 256;
sc->right = effect->u.rumble.weak_magnitude / 256;
- sony_schedule_work(sc);
+ sony_schedule_work(sc, SONY_WORKER_STATE);
return 0;
}
@@ -2219,8 +2370,8 @@ static int sony_check_add(struct sony_sc *sc)
hid_warn(sc->hdev, "UNIQ does not contain a MAC address; duplicate check skipped\n");
return 0;
}
- } else if (sc->quirks & DUALSHOCK4_CONTROLLER_USB) {
- buf = kmalloc(DS4_REPORT_0x81_SIZE, GFP_KERNEL);
+ } else if (sc->quirks & (DUALSHOCK4_CONTROLLER_USB | DUALSHOCK4_DONGLE)) {
+ buf = kmalloc(DS4_FEATURE_REPORT_0x81_SIZE, GFP_KERNEL);
if (!buf)
return -ENOMEM;
@@ -2230,16 +2381,22 @@ static int sony_check_add(struct sony_sc *sc)
* offset 1.
*/
ret = hid_hw_raw_request(sc->hdev, 0x81, buf,
- DS4_REPORT_0x81_SIZE, HID_FEATURE_REPORT,
+ DS4_FEATURE_REPORT_0x81_SIZE, HID_FEATURE_REPORT,
HID_REQ_GET_REPORT);
- if (ret != DS4_REPORT_0x81_SIZE) {
+ if (ret != DS4_FEATURE_REPORT_0x81_SIZE) {
hid_err(sc->hdev, "failed to retrieve feature report 0x81 with the DualShock 4 MAC address\n");
ret = ret < 0 ? ret : -EINVAL;
goto out_free;
}
memcpy(sc->mac_address, &buf[1], sizeof(sc->mac_address));
+
+ snprintf(sc->hdev->uniq, sizeof(sc->hdev->uniq),
+ "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx",
+ sc->mac_address[5], sc->mac_address[4],
+ sc->mac_address[3], sc->mac_address[2],
+ sc->mac_address[1], sc->mac_address[0]);
} else if ((sc->quirks & SIXAXIS_CONTROLLER_USB) ||
(sc->quirks & NAVIGATION_CONTROLLER_USB)) {
buf = kmalloc(SIXAXIS_REPORT_0xF2_SIZE, GFP_KERNEL);
@@ -2267,6 +2424,12 @@ static int sony_check_add(struct sony_sc *sc)
*/
for (n = 0; n < 6; n++)
sc->mac_address[5-n] = buf[4+n];
+
+ snprintf(sc->hdev->uniq, sizeof(sc->hdev->uniq),
+ "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx",
+ sc->mac_address[5], sc->mac_address[4],
+ sc->mac_address[3], sc->mac_address[2],
+ sc->mac_address[1], sc->mac_address[0]);
} else {
return 0;
}
@@ -2317,57 +2480,27 @@ static inline void sony_init_output_report(struct sony_sc *sc,
{
sc->send_output_report = send_output_report;
- if (!sc->worker_initialized)
+ if (!sc->state_worker_initialized)
INIT_WORK(&sc->state_worker, sony_state_worker);
- sc->worker_initialized = 1;
+ sc->state_worker_initialized = 1;
}
static inline void sony_cancel_work_sync(struct sony_sc *sc)
{
- if (sc->worker_initialized)
+ if (sc->hotplug_worker_initialized)
+ cancel_work_sync(&sc->hotplug_worker);
+ if (sc->state_worker_initialized)
cancel_work_sync(&sc->state_worker);
}
-static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id)
+
+static int sony_input_configured(struct hid_device *hdev,
+ struct hid_input *hidinput)
{
- int ret;
+ struct sony_sc *sc = hid_get_drvdata(hdev);
int append_dev_id;
- unsigned long quirks = id->driver_data;
- struct sony_sc *sc;
- unsigned int connect_mask = HID_CONNECT_DEFAULT;
-
- if (!strcmp(hdev->name, "FutureMax Dance Mat"))
- quirks |= FUTUREMAX_DANCE_MAT;
-
- sc = devm_kzalloc(&hdev->dev, sizeof(*sc), GFP_KERNEL);
- if (sc == NULL) {
- hid_err(hdev, "can't alloc sony descriptor\n");
- return -ENOMEM;
- }
-
- spin_lock_init(&sc->lock);
-
- sc->quirks = quirks;
- hid_set_drvdata(hdev, sc);
- sc->hdev = hdev;
-
- ret = hid_parse(hdev);
- if (ret) {
- hid_err(hdev, "parse failed\n");
- return ret;
- }
-
- if (sc->quirks & VAIO_RDESC_CONSTANT)
- connect_mask |= HID_CONNECT_HIDDEV_FORCE;
- else if (sc->quirks & SIXAXIS_CONTROLLER)
- connect_mask |= HID_CONNECT_HIDDEV_FORCE;
-
- ret = hid_hw_start(hdev, connect_mask);
- if (ret) {
- hid_err(hdev, "hw start failed\n");
- return ret;
- }
+ int ret;
ret = sony_set_device_id(sc);
if (ret < 0) {
@@ -2375,14 +2508,17 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id)
goto err_stop;
}
+ ret = append_dev_id = sony_check_add(sc);
+ if (ret < 0)
+ goto err_stop;
+
ret = sony_allocate_output_report(sc);
if (ret < 0) {
hid_err(hdev, "failed to allocate the output report buffer\n");
goto err_stop;
}
- if ((sc->quirks & SIXAXIS_CONTROLLER_USB) ||
- (sc->quirks & NAVIGATION_CONTROLLER_USB)) {
+ if (sc->quirks & NAVIGATION_CONTROLLER_USB) {
/*
* The Sony Sixaxis does not handle HID Output Reports on the
* Interrupt EP like it could, so we need to force HID Output
@@ -2402,29 +2538,113 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id)
hdev->quirks |= HID_QUIRK_NO_OUTPUT_REPORTS_ON_INTR_EP;
hdev->quirks |= HID_QUIRK_SKIP_OUTPUT_REPORT_ID;
sc->defer_initialization = 1;
+
ret = sixaxis_set_operational_usb(hdev);
+ if (ret < 0) {
+ hid_err(hdev, "Failed to set controller into operational mode\n");
+ goto err_stop;
+ }
+
sony_init_output_report(sc, sixaxis_send_output_report);
- } else if ((sc->quirks & SIXAXIS_CONTROLLER_BT) ||
- (sc->quirks & NAVIGATION_CONTROLLER_BT)) {
+ } else if (sc->quirks & NAVIGATION_CONTROLLER_BT) {
+ /*
+ * The Navigation controller wants output reports sent on the ctrl
+ * endpoint when connected via Bluetooth.
+ */
+ hdev->quirks |= HID_QUIRK_NO_OUTPUT_REPORTS_ON_INTR_EP;
+
+ ret = sixaxis_set_operational_bt(hdev);
+ if (ret < 0) {
+ hid_err(hdev, "Failed to set controller into operational mode\n");
+ goto err_stop;
+ }
+
+ sony_init_output_report(sc, sixaxis_send_output_report);
+ } else if (sc->quirks & SIXAXIS_CONTROLLER_USB) {
+ /*
+ * The Sony Sixaxis does not handle HID Output Reports on the
+ * Interrupt EP and the device only becomes active when the
+ * PS button is pressed. See comment for Navigation controller
+ * above for more details.
+ */
+ hdev->quirks |= HID_QUIRK_NO_OUTPUT_REPORTS_ON_INTR_EP;
+ hdev->quirks |= HID_QUIRK_SKIP_OUTPUT_REPORT_ID;
+ sc->defer_initialization = 1;
+
+ ret = sixaxis_set_operational_usb(hdev);
+ if (ret < 0) {
+ hid_err(hdev, "Failed to set controller into operational mode\n");
+ goto err_stop;
+ }
+
+ ret = sony_register_sensors(sc);
+ if (ret) {
+ hid_err(sc->hdev,
+ "Unable to initialize motion sensors: %d\n", ret);
+ goto err_stop;
+ }
+
+ sony_init_output_report(sc, sixaxis_send_output_report);
+ } else if (sc->quirks & SIXAXIS_CONTROLLER_BT) {
/*
* The Sixaxis wants output reports sent on the ctrl endpoint
* when connected via Bluetooth.
*/
hdev->quirks |= HID_QUIRK_NO_OUTPUT_REPORTS_ON_INTR_EP;
+
ret = sixaxis_set_operational_bt(hdev);
+ if (ret < 0) {
+ hid_err(hdev, "Failed to set controller into operational mode\n");
+ goto err_stop;
+ }
+
+ ret = sony_register_sensors(sc);
+ if (ret) {
+ hid_err(sc->hdev,
+ "Unable to initialize motion sensors: %d\n", ret);
+ goto err_stop;
+ }
+
sony_init_output_report(sc, sixaxis_send_output_report);
} else if (sc->quirks & DUALSHOCK4_CONTROLLER) {
+ ret = dualshock4_get_calibration_data(sc);
+ if (ret < 0) {
+ hid_err(hdev, "Failed to get calibration data from Dualshock 4\n");
+ goto err_stop;
+ }
+
+ /*
+ * The Dualshock 4 touchpad supports 2 touches and has a
+ * resolution of 1920x942 (44.86 dots/mm).
+ */
+ ret = sony_register_touchpad(sc, 2, 1920, 942);
+ if (ret) {
+ hid_err(sc->hdev,
+ "Unable to initialize multi-touch slots: %d\n",
+ ret);
+ goto err_stop;
+ }
+
+ ret = sony_register_sensors(sc);
+ if (ret) {
+ hid_err(sc->hdev,
+ "Unable to initialize motion sensors: %d\n", ret);
+ goto err_stop;
+ }
+
if (sc->quirks & DUALSHOCK4_CONTROLLER_BT) {
- /*
- * The DualShock 4 wants output reports sent on the ctrl
- * endpoint when connected via Bluetooth.
- */
- hdev->quirks |= HID_QUIRK_NO_OUTPUT_REPORTS_ON_INTR_EP;
- ret = dualshock4_set_operational_bt(hdev);
- if (ret < 0) {
- hid_err(hdev, "failed to set the Dualshock 4 operational mode\n");
- goto err_stop;
- }
+ sc->ds4_bt_poll_interval = DS4_BT_DEFAULT_POLL_INTERVAL_MS;
+ ret = device_create_file(&sc->hdev->dev, &dev_attr_bt_poll_interval);
+ if (ret)
+ hid_warn(sc->hdev,
+ "can't create sysfs bt_poll_interval attribute err: %d\n",
+ ret);
+ }
+
+ if (sc->quirks & DUALSHOCK4_DONGLE) {
+ INIT_WORK(&sc->hotplug_worker, dualshock4_calibration_work);
+ sc->hotplug_worker_initialized = 1;
+ sc->ds4_dongle_state = DONGLE_DISCONNECTED;
}
sony_init_output_report(sc, dualshock4_send_output_report);
@@ -2434,13 +2654,6 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id)
ret = 0;
}
- if (ret < 0)
- goto err_stop;
-
- ret = append_dev_id = sony_check_add(sc);
- if (ret < 0)
- goto err_stop;
-
if (sc->quirks & SONY_LED_SUPPORT) {
ret = sony_leds_init(sc);
if (ret < 0)
@@ -2470,10 +2683,20 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id)
err_close:
hid_hw_close(hdev);
err_stop:
+ /* Piggy back on the default ds4_bt_ poll_interval to determine
+ * if we need to remove the file as we don't know for sure if we
+ * executed that logic.
+ */
+ if (sc->ds4_bt_poll_interval)
+ device_remove_file(&sc->hdev->dev, &dev_attr_bt_poll_interval);
if (sc->quirks & SONY_LED_SUPPORT)
sony_leds_remove(sc);
if (sc->quirks & SONY_BATTERY_SUPPORT)
sony_battery_remove(sc);
+ if (sc->touchpad)
+ sony_unregister_touchpad(sc);
+ if (sc->sensor_dev)
+ sony_unregister_sensors(sc);
sony_cancel_work_sync(sc);
kfree(sc->output_report_dmabuf);
sony_remove_dev_list(sc);
@@ -2482,17 +2705,90 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id)
return ret;
}
+static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id)
+{
+ int ret;
+ unsigned long quirks = id->driver_data;
+ struct sony_sc *sc;
+ unsigned int connect_mask = HID_CONNECT_DEFAULT;
+
+ if (!strcmp(hdev->name, "FutureMax Dance Mat"))
+ quirks |= FUTUREMAX_DANCE_MAT;
+
+ sc = devm_kzalloc(&hdev->dev, sizeof(*sc), GFP_KERNEL);
+ if (sc == NULL) {
+ hid_err(hdev, "can't alloc sony descriptor\n");
+ return -ENOMEM;
+ }
+
+ spin_lock_init(&sc->lock);
+
+ sc->quirks = quirks;
+ hid_set_drvdata(hdev, sc);
+ sc->hdev = hdev;
+
+ ret = hid_parse(hdev);
+ if (ret) {
+ hid_err(hdev, "parse failed\n");
+ return ret;
+ }
+
+ if (sc->quirks & VAIO_RDESC_CONSTANT)
+ connect_mask |= HID_CONNECT_HIDDEV_FORCE;
+ else if (sc->quirks & SIXAXIS_CONTROLLER)
+ connect_mask |= HID_CONNECT_HIDDEV_FORCE;
+
+ /* Patch the hw version on DS3/4 compatible devices, so applications can
+ * distinguish between the default HID mappings and the mappings defined
+ * by the Linux game controller spec. This is important for the SDL2
+ * library, which has a game controller database, which uses device ids
+ * in combination with version as a key.
+ */
+ if (sc->quirks & (SIXAXIS_CONTROLLER | DUALSHOCK4_CONTROLLER))
+ hdev->version |= 0x8000;
+
+ ret = hid_hw_start(hdev, connect_mask);
+ if (ret) {
+ hid_err(hdev, "hw start failed\n");
+ return ret;
+ }
+
+ /* sony_input_configured can fail, but this doesn't result
+ * in hid_hw_start failures (intended). Check whether
+ * the HID layer claimed the device else fail.
+ * We don't know the actual reason for the failure, most
+ * likely it is due to EEXIST in case of double connection
+ * of USB and Bluetooth, but could have been due to ENOMEM
+ * or other reasons as well.
+ */
+ if (!(hdev->claimed & HID_CLAIMED_INPUT)) {
+ hid_err(hdev, "failed to claim input\n");
+ return -ENODEV;
+ }
+
+ return ret;
+}
+
static void sony_remove(struct hid_device *hdev)
{
struct sony_sc *sc = hid_get_drvdata(hdev);
+ hid_hw_close(hdev);
+
if (sc->quirks & SONY_LED_SUPPORT)
sony_leds_remove(sc);
- if (sc->quirks & SONY_BATTERY_SUPPORT) {
- hid_hw_close(hdev);
+ if (sc->quirks & SONY_BATTERY_SUPPORT)
sony_battery_remove(sc);
- }
+
+ if (sc->touchpad)
+ sony_unregister_touchpad(sc);
+
+ if (sc->sensor_dev)
+ sony_unregister_sensors(sc);
+
+ if (sc->quirks & DUALSHOCK4_CONTROLLER_BT)
+ device_remove_file(&sc->hdev->dev, &dev_attr_bt_poll_interval);
sony_cancel_work_sync(sc);
@@ -2581,6 +2877,12 @@ static const struct hid_device_id sony_devices[] = {
.driver_data = DUALSHOCK4_CONTROLLER_USB },
{ HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER),
.driver_data = DUALSHOCK4_CONTROLLER_BT },
+ { HID_USB_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER_2),
+ .driver_data = DUALSHOCK4_CONTROLLER_USB },
+ { HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER_2),
+ .driver_data = DUALSHOCK4_CONTROLLER_BT },
+ { HID_USB_DEVICE(USB_VENDOR_ID_SONY, USB_DEVICE_ID_SONY_PS4_CONTROLLER_DONGLE),
+ .driver_data = DUALSHOCK4_DONGLE },
/* Nyko Core Controller for PS3 */
{ HID_USB_DEVICE(USB_VENDOR_ID_SINO_LITE, USB_DEVICE_ID_SINO_LITE_CONTROLLER),
.driver_data = SIXAXIS_CONTROLLER_USB | SINO_LITE_CONTROLLER },
diff --git a/drivers/hid/i2c-hid/i2c-hid.c b/drivers/hid/i2c-hid/i2c-hid.c
index 2548c5d..ce2b800 100644
--- a/drivers/hid/i2c-hid/i2c-hid.c
+++ b/drivers/hid/i2c-hid/i2c-hid.c
@@ -477,7 +477,7 @@ static void i2c_hid_get_input(struct i2c_hid *ihid)
return;
}
- if ((ret_size > size) || (ret_size <= 2)) {
+ if ((ret_size > size) || (ret_size < 2)) {
dev_err(&ihid->client->dev, "%s: incomplete report (%d/%d)\n",
__func__, size, ret_size);
return;
@@ -1101,6 +1101,14 @@ static int i2c_hid_probe(struct i2c_client *client,
pm_runtime_enable(&client->dev);
device_enable_async_suspend(&client->dev);
+ /* Make sure there is something at this address */
+ ret = i2c_smbus_read_byte(client);
+ if (ret < 0) {
+ dev_dbg(&client->dev, "nothing at this address: %d\n", ret);
+ ret = -ENXIO;
+ goto err_pm;
+ }
+
ret = i2c_hid_fetch_hid_descriptor(ihid);
if (ret < 0)
goto err_pm;
diff --git a/drivers/hid/intel-ish-hid/ipc/pci-ish.c b/drivers/hid/intel-ish-hid/ipc/pci-ish.c
index 20d647d..00aafe0 100644
--- a/drivers/hid/intel-ish-hid/ipc/pci-ish.c
+++ b/drivers/hid/intel-ish-hid/ipc/pci-ish.c
@@ -202,8 +202,7 @@ static void ish_remove(struct pci_dev *pdev)
kfree(ishtp_dev);
}
-#ifdef CONFIG_PM
-static struct device *ish_resume_device;
+static struct device __maybe_unused *ish_resume_device;
/**
* ish_resume_handler() - Work function to complete resume
@@ -214,7 +213,7 @@ static struct device *ish_resume_device;
* in that case a simple resume message is enough, others we need
* a reset sequence.
*/
-static void ish_resume_handler(struct work_struct *work)
+static void __maybe_unused ish_resume_handler(struct work_struct *work)
{
struct pci_dev *pdev = to_pci_dev(ish_resume_device);
struct ishtp_device *dev = pci_get_drvdata(pdev);
@@ -245,7 +244,7 @@ static void ish_resume_handler(struct work_struct *work)
*
* Return: 0 to the pm core
*/
-static int ish_suspend(struct device *device)
+static int __maybe_unused ish_suspend(struct device *device)
{
struct pci_dev *pdev = to_pci_dev(device);
struct ishtp_device *dev = pci_get_drvdata(pdev);
@@ -271,7 +270,7 @@ static int ish_suspend(struct device *device)
return 0;
}
-static DECLARE_WORK(resume_work, ish_resume_handler);
+static __maybe_unused DECLARE_WORK(resume_work, ish_resume_handler);
/**
* ish_resume() - ISH resume callback
* @device: device pointer
@@ -280,7 +279,7 @@ static DECLARE_WORK(resume_work, ish_resume_handler);
*
* Return: 0 to the pm core
*/
-static int ish_resume(struct device *device)
+static int __maybe_unused ish_resume(struct device *device)
{
struct pci_dev *pdev = to_pci_dev(device);
struct ishtp_device *dev = pci_get_drvdata(pdev);
@@ -294,21 +293,14 @@ static int ish_resume(struct device *device)
return 0;
}
-static const struct dev_pm_ops ish_pm_ops = {
- .suspend = ish_suspend,
- .resume = ish_resume,
-};
-#define ISHTP_ISH_PM_OPS (&ish_pm_ops)
-#else
-#define ISHTP_ISH_PM_OPS NULL
-#endif /* CONFIG_PM */
+static SIMPLE_DEV_PM_OPS(ish_pm_ops, ish_suspend, ish_resume);
static struct pci_driver ish_driver = {
.name = KBUILD_MODNAME,
.id_table = ish_pci_tbl,
.probe = ish_probe,
.remove = ish_remove,
- .driver.pm = ISHTP_ISH_PM_OPS,
+ .driver.pm = &ish_pm_ops,
};
module_pci_driver(ish_driver);
diff --git a/drivers/hid/uhid.c b/drivers/hid/uhid.c
index e46f656..7256647 100644
--- a/drivers/hid/uhid.c
+++ b/drivers/hid/uhid.c
@@ -12,6 +12,7 @@
#include <linux/atomic.h>
#include <linux/compat.h>
+#include <linux/cred.h>
#include <linux/device.h>
#include <linux/fs.h>
#include <linux/hid.h>
@@ -24,6 +25,7 @@
#include <linux/spinlock.h>
#include <linux/uhid.h>
#include <linux/wait.h>
+#include <linux/uaccess.h>
#define UHID_NAME "uhid"
#define UHID_BUFSIZE 32
@@ -734,6 +736,17 @@ static ssize_t uhid_char_write(struct file *file, const char __user *buffer,
switch (uhid->input_buf.type) {
case UHID_CREATE:
+ /*
+ * 'struct uhid_create_req' contains a __user pointer which is
+ * copied from, so it's unsafe to allow this with elevated
+ * privileges (e.g. from a setuid binary) or via kernel_write().
+ */
+ if (file->f_cred != current_cred() || uaccess_kernel()) {
+ pr_err_once("UHID_CREATE from different security context by process %d (%s), this is not allowed.\n",
+ task_tgid_vnr(current), current->comm);
+ ret = -EACCES;
+ goto unlock;
+ }
ret = uhid_dev_create(uhid, &uhid->input_buf);
break;
case UHID_CREATE2:
diff --git a/drivers/hid/usbhid/hid-quirks.c b/drivers/hid/usbhid/hid-quirks.c
index 1916f80..617ae29 100644
--- a/drivers/hid/usbhid/hid-quirks.c
+++ b/drivers/hid/usbhid/hid-quirks.c
@@ -170,6 +170,7 @@ static const struct hid_blacklist {
{ USB_VENDOR_ID_MULTIPLE_1781, USB_DEVICE_ID_RAPHNET_4NES4SNES_OLD, HID_QUIRK_MULTI_INPUT },
{ USB_VENDOR_ID_DRACAL_RAPHNET, USB_DEVICE_ID_RAPHNET_2NES2SNES, HID_QUIRK_MULTI_INPUT },
{ USB_VENDOR_ID_DRACAL_RAPHNET, USB_DEVICE_ID_RAPHNET_4NES4SNES, HID_QUIRK_MULTI_INPUT },
+ { USB_VENDOR_ID_INNOMEDIA, USB_DEVICE_ID_INNEX_GENESIS_ATARI, HID_QUIRK_MULTI_INPUT },
{ 0, 0 }
};
diff --git a/drivers/hid/usbhid/hiddev.c b/drivers/hid/usbhid/hiddev.c
index 700145b..308d843 100644
--- a/drivers/hid/usbhid/hiddev.c
+++ b/drivers/hid/usbhid/hiddev.c
@@ -35,6 +35,7 @@
#include <linux/hiddev.h>
#include <linux/compat.h>
#include <linux/vmalloc.h>
+#include <linux/nospec.h>
#include "usbhid.h"
#ifdef CONFIG_USB_DYNAMIC_MINORS
@@ -478,10 +479,14 @@ static noinline int hiddev_ioctl_usage(struct hiddev *hiddev, unsigned int cmd,
if (uref->field_index >= report->maxfield)
goto inval;
+ uref->field_index = array_index_nospec(uref->field_index,
+ report->maxfield);
field = report->field[uref->field_index];
if (uref->usage_index >= field->maxusage)
goto inval;
+ uref->usage_index = array_index_nospec(uref->usage_index,
+ field->maxusage);
uref->usage_code = field->usage[uref->usage_index].hid;
@@ -508,20 +513,32 @@ static noinline int hiddev_ioctl_usage(struct hiddev *hiddev, unsigned int cmd,
if (uref->field_index >= report->maxfield)
goto inval;
+ uref->field_index = array_index_nospec(uref->field_index,
+ report->maxfield);
field = report->field[uref->field_index];
if (cmd == HIDIOCGCOLLECTIONINDEX) {
if (uref->usage_index >= field->maxusage)
goto inval;
+ uref->usage_index =
+ array_index_nospec(uref->usage_index,
+ field->maxusage);
} else if (uref->usage_index >= field->report_count)
goto inval;
}
- if ((cmd == HIDIOCGUSAGES || cmd == HIDIOCSUSAGES) &&
- (uref_multi->num_values > HID_MAX_MULTI_USAGES ||
- uref->usage_index + uref_multi->num_values > field->report_count))
- goto inval;
+ if (cmd == HIDIOCGUSAGES || cmd == HIDIOCSUSAGES) {
+ if (uref_multi->num_values > HID_MAX_MULTI_USAGES ||
+ uref->usage_index + uref_multi->num_values >
+ field->report_count)
+ goto inval;
+
+ uref->usage_index =
+ array_index_nospec(uref->usage_index,
+ field->report_count -
+ uref_multi->num_values);
+ }
switch (cmd) {
case HIDIOCGUSAGE:
@@ -761,6 +778,8 @@ static long hiddev_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
if (finfo.field_index >= report->maxfield)
break;
+ finfo.field_index = array_index_nospec(finfo.field_index,
+ report->maxfield);
field = report->field[finfo.field_index];
memset(&finfo, 0, sizeof(finfo));
@@ -801,6 +820,8 @@ static long hiddev_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
if (cinfo.index >= hid->maxcollection)
break;
+ cinfo.index = array_index_nospec(cinfo.index,
+ hid->maxcollection);
cinfo.type = hid->collection[cinfo.index].type;
cinfo.usage = hid->collection[cinfo.index].usage;
diff --git a/drivers/hid/wacom_wac.c b/drivers/hid/wacom_wac.c
index db951c4..b1ad378 100644
--- a/drivers/hid/wacom_wac.c
+++ b/drivers/hid/wacom_wac.c
@@ -2429,8 +2429,14 @@ void wacom_setup_device_quirks(struct wacom *wacom)
if (features->type >= INTUOSHT && features->type <= BAMBOO_PT)
features->device_type |= WACOM_DEVICETYPE_PAD;
- features->x_max = 4096;
- features->y_max = 4096;
+ if (features->type == INTUOSHT2) {
+ features->x_max = features->x_max / 10;
+ features->y_max = features->y_max / 10;
+ }
+ else {
+ features->x_max = 4096;
+ features->y_max = 4096;
+ }
}
else if (features->pktlen == WACOM_PKGLEN_BBTOUCH) {
features->device_type |= WACOM_DEVICETYPE_PAD;
diff --git a/drivers/hv/channel.c b/drivers/hv/channel.c
index 1606e7f..784c454 100644
--- a/drivers/hv/channel.c
+++ b/drivers/hv/channel.c
@@ -448,6 +448,14 @@ int vmbus_establish_gpadl(struct vmbus_channel *channel, void *kbuffer,
}
wait_for_completion(&msginfo->waitevent);
+ if (msginfo->response.gpadl_created.creation_status != 0) {
+ pr_err("Failed to establish GPADL: err = 0x%x\n",
+ msginfo->response.gpadl_created.creation_status);
+
+ ret = -EDQUOT;
+ goto cleanup;
+ }
+
if (channel->rescind) {
ret = -ENODEV;
goto cleanup;
diff --git a/drivers/hv/hv_kvp.c b/drivers/hv/hv_kvp.c
index 5e1fdc8..2fd0f119 100644
--- a/drivers/hv/hv_kvp.c
+++ b/drivers/hv/hv_kvp.c
@@ -616,21 +616,22 @@ void hv_kvp_onchannelcallback(void *context)
NEGO_IN_PROGRESS,
NEGO_FINISHED} host_negotiatied = NEGO_NOT_STARTED;
- if (host_negotiatied == NEGO_NOT_STARTED &&
- kvp_transaction.state < HVUTIL_READY) {
+ if (kvp_transaction.state < HVUTIL_READY) {
/*
* If userspace daemon is not connected and host is asking
* us to negotiate we need to delay to not lose messages.
* This is important for Failover IP setting.
*/
- host_negotiatied = NEGO_IN_PROGRESS;
- schedule_delayed_work(&kvp_host_handshake_work,
+ if (host_negotiatied == NEGO_NOT_STARTED) {
+ host_negotiatied = NEGO_IN_PROGRESS;
+ schedule_delayed_work(&kvp_host_handshake_work,
HV_UTIL_NEGO_TIMEOUT * HZ);
+ }
return;
}
if (kvp_transaction.state > HVUTIL_READY)
return;
-
+recheck:
vmbus_recvpacket(channel, recv_buffer, PAGE_SIZE * 4, &recvlen,
&requestid);
@@ -707,6 +708,8 @@ void hv_kvp_onchannelcallback(void *context)
VM_PKT_DATA_INBAND, 0);
host_negotiatied = NEGO_FINISHED;
+
+ goto recheck;
}
}
diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 0276d2e..9a0d0d0 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -317,6 +317,8 @@ static ssize_t out_intr_mask_show(struct device *dev,
if (!hv_dev->channel)
return -ENODEV;
+ if (hv_dev->channel->state != CHANNEL_OPENED_STATE)
+ return -EINVAL;
hv_ringbuffer_get_debuginfo(&hv_dev->channel->outbound, &outbound);
return sprintf(buf, "%d\n", outbound.current_interrupt_mask);
}
@@ -330,6 +332,8 @@ static ssize_t out_read_index_show(struct device *dev,
if (!hv_dev->channel)
return -ENODEV;
+ if (hv_dev->channel->state != CHANNEL_OPENED_STATE)
+ return -EINVAL;
hv_ringbuffer_get_debuginfo(&hv_dev->channel->outbound, &outbound);
return sprintf(buf, "%d\n", outbound.current_read_index);
}
@@ -344,6 +348,8 @@ static ssize_t out_write_index_show(struct device *dev,
if (!hv_dev->channel)
return -ENODEV;
+ if (hv_dev->channel->state != CHANNEL_OPENED_STATE)
+ return -EINVAL;
hv_ringbuffer_get_debuginfo(&hv_dev->channel->outbound, &outbound);
return sprintf(buf, "%d\n", outbound.current_write_index);
}
@@ -358,6 +364,8 @@ static ssize_t out_read_bytes_avail_show(struct device *dev,
if (!hv_dev->channel)
return -ENODEV;
+ if (hv_dev->channel->state != CHANNEL_OPENED_STATE)
+ return -EINVAL;
hv_ringbuffer_get_debuginfo(&hv_dev->channel->outbound, &outbound);
return sprintf(buf, "%d\n", outbound.bytes_avail_toread);
}
@@ -372,6 +380,8 @@ static ssize_t out_write_bytes_avail_show(struct device *dev,
if (!hv_dev->channel)
return -ENODEV;
+ if (hv_dev->channel->state != CHANNEL_OPENED_STATE)
+ return -EINVAL;
hv_ringbuffer_get_debuginfo(&hv_dev->channel->outbound, &outbound);
return sprintf(buf, "%d\n", outbound.bytes_avail_towrite);
}
@@ -385,6 +395,8 @@ static ssize_t in_intr_mask_show(struct device *dev,
if (!hv_dev->channel)
return -ENODEV;
+ if (hv_dev->channel->state != CHANNEL_OPENED_STATE)
+ return -EINVAL;
hv_ringbuffer_get_debuginfo(&hv_dev->channel->inbound, &inbound);
return sprintf(buf, "%d\n", inbound.current_interrupt_mask);
}
@@ -398,6 +410,8 @@ static ssize_t in_read_index_show(struct device *dev,
if (!hv_dev->channel)
return -ENODEV;
+ if (hv_dev->channel->state != CHANNEL_OPENED_STATE)
+ return -EINVAL;
hv_ringbuffer_get_debuginfo(&hv_dev->channel->inbound, &inbound);
return sprintf(buf, "%d\n", inbound.current_read_index);
}
@@ -411,6 +425,8 @@ static ssize_t in_write_index_show(struct device *dev,
if (!hv_dev->channel)
return -ENODEV;
+ if (hv_dev->channel->state != CHANNEL_OPENED_STATE)
+ return -EINVAL;
hv_ringbuffer_get_debuginfo(&hv_dev->channel->inbound, &inbound);
return sprintf(buf, "%d\n", inbound.current_write_index);
}
@@ -425,6 +441,8 @@ static ssize_t in_read_bytes_avail_show(struct device *dev,
if (!hv_dev->channel)
return -ENODEV;
+ if (hv_dev->channel->state != CHANNEL_OPENED_STATE)
+ return -EINVAL;
hv_ringbuffer_get_debuginfo(&hv_dev->channel->inbound, &inbound);
return sprintf(buf, "%d\n", inbound.bytes_avail_toread);
}
@@ -439,6 +457,8 @@ static ssize_t in_write_bytes_avail_show(struct device *dev,
if (!hv_dev->channel)
return -ENODEV;
+ if (hv_dev->channel->state != CHANNEL_OPENED_STATE)
+ return -EINVAL;
hv_ringbuffer_get_debuginfo(&hv_dev->channel->inbound, &inbound);
return sprintf(buf, "%d\n", inbound.bytes_avail_towrite);
}
diff --git a/drivers/hwmon/Makefile b/drivers/hwmon/Makefile
index 7b9a113..7d16ccd 100644
--- a/drivers/hwmon/Makefile
+++ b/drivers/hwmon/Makefile
@@ -173,3 +173,7 @@
ccflags-$(CONFIG_HWMON_DEBUG_CHIP) := -DDEBUG
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-logical-not-parentheses \
+ -Wno-enum-conversion \
diff --git a/drivers/hwmon/adt7475.c b/drivers/hwmon/adt7475.c
index 3cefd1a..9c262d9 100644
--- a/drivers/hwmon/adt7475.c
+++ b/drivers/hwmon/adt7475.c
@@ -274,14 +274,18 @@ static inline u16 volt2reg(int channel, long volt, u8 bypass_attn)
return clamp_val(reg, 0, 1023) & (0xff << 2);
}
-static u16 adt7475_read_word(struct i2c_client *client, int reg)
+static int adt7475_read_word(struct i2c_client *client, int reg)
{
- u16 val;
+ int val1, val2;
- val = i2c_smbus_read_byte_data(client, reg);
- val |= (i2c_smbus_read_byte_data(client, reg + 1) << 8);
+ val1 = i2c_smbus_read_byte_data(client, reg);
+ if (val1 < 0)
+ return val1;
+ val2 = i2c_smbus_read_byte_data(client, reg + 1);
+ if (val2 < 0)
+ return val2;
- return val;
+ return val1 | (val2 << 8);
}
static void adt7475_write_word(struct i2c_client *client, int reg, u16 val)
diff --git a/drivers/hwmon/ibmpowernv.c b/drivers/hwmon/ibmpowernv.c
index 6d2e660..18b3c8f 100644
--- a/drivers/hwmon/ibmpowernv.c
+++ b/drivers/hwmon/ibmpowernv.c
@@ -114,7 +114,7 @@ static ssize_t show_label(struct device *dev, struct device_attribute *devattr,
return sprintf(buf, "%s\n", sdata->label);
}
-static int __init get_logical_cpu(int hwcpu)
+static int get_logical_cpu(int hwcpu)
{
int cpu;
@@ -125,9 +125,8 @@ static int __init get_logical_cpu(int hwcpu)
return -ENOENT;
}
-static void __init make_sensor_label(struct device_node *np,
- struct sensor_data *sdata,
- const char *label)
+static void make_sensor_label(struct device_node *np,
+ struct sensor_data *sdata, const char *label)
{
u32 id;
size_t n;
diff --git a/drivers/hwmon/ina2xx.c b/drivers/hwmon/ina2xx.c
index ac63e56..1f291b3 100644
--- a/drivers/hwmon/ina2xx.c
+++ b/drivers/hwmon/ina2xx.c
@@ -17,7 +17,7 @@
* Bi-directional Current/Power Monitor with I2C Interface
* Datasheet: http://www.ti.com/product/ina230
*
- * Copyright (C) 2012 Lothar Felten <l-felten@ti.com>
+ * Copyright (C) 2012 Lothar Felten <lothar.felten@gmail.com>
* Thanks to Jan Volkering
*
* This program is free software; you can redistribute it and/or modify
@@ -273,7 +273,7 @@ static int ina2xx_get_value(struct ina2xx_data *data, u8 reg,
break;
case INA2XX_CURRENT:
/* signed register, result in mA */
- val = regval * data->current_lsb_uA;
+ val = (s16)regval * data->current_lsb_uA;
val = DIV_ROUND_CLOSEST(val, 1000);
break;
case INA2XX_CALIBRATION:
@@ -328,6 +328,15 @@ static int ina2xx_set_shunt(struct ina2xx_data *data, long val)
return 0;
}
+static ssize_t ina2xx_show_shunt(struct device *dev,
+ struct device_attribute *da,
+ char *buf)
+{
+ struct ina2xx_data *data = dev_get_drvdata(dev);
+
+ return snprintf(buf, PAGE_SIZE, "%li\n", data->rshunt);
+}
+
static ssize_t ina2xx_store_shunt(struct device *dev,
struct device_attribute *da,
const char *buf, size_t count)
@@ -402,7 +411,7 @@ static SENSOR_DEVICE_ATTR(power1_input, S_IRUGO, ina2xx_show_value, NULL,
/* shunt resistance */
static SENSOR_DEVICE_ATTR(shunt_resistor, S_IRUGO | S_IWUSR,
- ina2xx_show_value, ina2xx_store_shunt,
+ ina2xx_show_shunt, ina2xx_store_shunt,
INA2XX_CALIBRATION);
/* update interval (ina226 only) */
diff --git a/drivers/hwmon/nct6775.c b/drivers/hwmon/nct6775.c
index ce75dd4..2b31b84 100644
--- a/drivers/hwmon/nct6775.c
+++ b/drivers/hwmon/nct6775.c
@@ -1393,7 +1393,7 @@ static void nct6775_update_pwm(struct device *dev)
duty_is_dc = data->REG_PWM_MODE[i] &&
(nct6775_read_value(data, data->REG_PWM_MODE[i])
& data->PWM_MODE_MASK[i]);
- data->pwm_mode[i] = duty_is_dc;
+ data->pwm_mode[i] = !duty_is_dc;
fanmodecfg = nct6775_read_value(data, data->REG_FAN_MODE[i]);
for (j = 0; j < ARRAY_SIZE(data->REG_PWM); j++) {
@@ -2270,7 +2270,7 @@ show_pwm_mode(struct device *dev, struct device_attribute *attr, char *buf)
struct nct6775_data *data = nct6775_update_device(dev);
struct sensor_device_attribute *sattr = to_sensor_dev_attr(attr);
- return sprintf(buf, "%d\n", !data->pwm_mode[sattr->index]);
+ return sprintf(buf, "%d\n", data->pwm_mode[sattr->index]);
}
static ssize_t
@@ -2291,9 +2291,9 @@ store_pwm_mode(struct device *dev, struct device_attribute *attr,
if (val > 1)
return -EINVAL;
- /* Setting DC mode is not supported for all chips/channels */
+ /* Setting DC mode (0) is not supported for all chips/channels */
if (data->REG_PWM_MODE[nr] == 0) {
- if (val)
+ if (!val)
return -EINVAL;
return count;
}
@@ -2302,7 +2302,7 @@ store_pwm_mode(struct device *dev, struct device_attribute *attr,
data->pwm_mode[nr] = val;
reg = nct6775_read_value(data, data->REG_PWM_MODE[nr]);
reg &= ~data->PWM_MODE_MASK[nr];
- if (val)
+ if (!val)
reg |= data->PWM_MODE_MASK[nr];
nct6775_write_value(data, data->REG_PWM_MODE[nr], reg);
mutex_unlock(&data->update_lock);
diff --git a/drivers/hwmon/pmbus/adm1275.c b/drivers/hwmon/pmbus/adm1275.c
index d659a02..c3a8f68 100644
--- a/drivers/hwmon/pmbus/adm1275.c
+++ b/drivers/hwmon/pmbus/adm1275.c
@@ -154,7 +154,7 @@ static int adm1275_read_word_data(struct i2c_client *client, int page, int reg)
const struct adm1275_data *data = to_adm1275_data(info);
int ret = 0;
- if (page)
+ if (page > 0)
return -ENXIO;
switch (reg) {
@@ -240,7 +240,7 @@ static int adm1275_write_word_data(struct i2c_client *client, int page, int reg,
const struct adm1275_data *data = to_adm1275_data(info);
int ret;
- if (page)
+ if (page > 0)
return -ENXIO;
switch (reg) {
diff --git a/drivers/hwmon/pmbus/max8688.c b/drivers/hwmon/pmbus/max8688.c
index dd4883a..e951f9b 100644
--- a/drivers/hwmon/pmbus/max8688.c
+++ b/drivers/hwmon/pmbus/max8688.c
@@ -45,7 +45,7 @@ static int max8688_read_word_data(struct i2c_client *client, int page, int reg)
{
int ret;
- if (page)
+ if (page > 0)
return -ENXIO;
switch (reg) {
diff --git a/drivers/hwmon/pmbus/pmbus.c b/drivers/hwmon/pmbus/pmbus.c
index 44ca8a9..2fe5a99 100644
--- a/drivers/hwmon/pmbus/pmbus.c
+++ b/drivers/hwmon/pmbus/pmbus.c
@@ -118,6 +118,8 @@ static int pmbus_identify(struct i2c_client *client,
} else {
info->pages = 1;
}
+
+ pmbus_clear_faults(client);
}
if (pmbus_check_byte_register(client, 0, PMBUS_VOUT_MODE)) {
diff --git a/drivers/hwmon/pmbus/pmbus_core.c b/drivers/hwmon/pmbus/pmbus_core.c
index d013acf..c00bad0 100644
--- a/drivers/hwmon/pmbus/pmbus_core.c
+++ b/drivers/hwmon/pmbus/pmbus_core.c
@@ -1759,7 +1759,10 @@ static int pmbus_init_common(struct i2c_client *client, struct pmbus_data *data,
if (ret >= 0 && (ret & PB_CAPABILITY_ERROR_CHECK))
client->flags |= I2C_CLIENT_PEC;
- pmbus_clear_faults(client);
+ if (data->info->pages)
+ pmbus_clear_faults(client);
+ else
+ pmbus_clear_fault_page(client, -1);
if (info->identify) {
ret = (*info->identify)(client, info);
diff --git a/drivers/hwmon/pwm-fan.c b/drivers/hwmon/pwm-fan.c
index f9af393..fb03449 100644
--- a/drivers/hwmon/pwm-fan.c
+++ b/drivers/hwmon/pwm-fan.c
@@ -306,9 +306,19 @@ static int pwm_fan_remove(struct platform_device *pdev)
static int pwm_fan_suspend(struct device *dev)
{
struct pwm_fan_ctx *ctx = dev_get_drvdata(dev);
+ struct pwm_args args;
+ int ret;
- if (ctx->pwm_value)
+ pwm_get_args(ctx->pwm, &args);
+
+ if (ctx->pwm_value) {
+ ret = pwm_config(ctx->pwm, 0, args.period);
+ if (ret < 0)
+ return ret;
+
pwm_disable(ctx->pwm);
+ }
+
return 0;
}
diff --git a/drivers/hwmon/w83795.c b/drivers/hwmon/w83795.c
index 49276bb..1bb80f9 100644
--- a/drivers/hwmon/w83795.c
+++ b/drivers/hwmon/w83795.c
@@ -1691,7 +1691,7 @@ store_sf_setup(struct device *dev, struct device_attribute *attr,
* somewhere else in the code
*/
#define SENSOR_ATTR_TEMP(index) { \
- SENSOR_ATTR_2(temp##index##_type, S_IRUGO | (index < 4 ? S_IWUSR : 0), \
+ SENSOR_ATTR_2(temp##index##_type, S_IRUGO | (index < 5 ? S_IWUSR : 0), \
show_temp_mode, store_temp_mode, NOT_USED, index - 1), \
SENSOR_ATTR_2(temp##index##_input, S_IRUGO, show_temp, \
NULL, TEMP_READ, index - 1), \
diff --git a/drivers/hwtracing/coresight/Makefile b/drivers/hwtracing/coresight/Makefile
index 157ec9c..d2ca188 100644
--- a/drivers/hwtracing/coresight/Makefile
+++ b/drivers/hwtracing/coresight/Makefile
@@ -28,3 +28,6 @@
obj-$(CONFIG_CORESIGHT_HWEVENT) += coresight-hwevent.o
obj-$(CONFIG_CORESIGHT_DUMMY) += coresight-dummy.o
obj-$(CONFIG_CORESIGHT_REMOTE_ETM) += coresight-remote-etm.o
+
+subdir-ccflags-y += \
+ -Wno-pointer-bool-conversion \
diff --git a/drivers/hwtracing/coresight/coresight-etb10.c b/drivers/hwtracing/coresight/coresight-etb10.c
index 7e041d9..b9e985f 100644
--- a/drivers/hwtracing/coresight/coresight-etb10.c
+++ b/drivers/hwtracing/coresight/coresight-etb10.c
@@ -155,6 +155,10 @@ static int etb_enable(struct coresight_device *csdev, u32 mode)
if (val == CS_MODE_PERF)
return -EBUSY;
+ /* Don't let perf disturb sysFS sessions */
+ if (val == CS_MODE_SYSFS && mode == CS_MODE_PERF)
+ return -EBUSY;
+
/* Nothing to do, the tracer is already enabled. */
if (val == CS_MODE_SYSFS)
goto out;
diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c b/drivers/hwtracing/coresight/coresight-tmc-etr.c
index 9d2ab01..938dfd0 100644
--- a/drivers/hwtracing/coresight/coresight-tmc-etr.c
+++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c
@@ -910,6 +910,7 @@ static void tmc_disable_etr_sink(struct coresight_device *csdev)
spin_lock_irqsave(&drvdata->spinlock, flags);
if (drvdata->reading) {
spin_unlock_irqrestore(&drvdata->spinlock, flags);
+ mutex_unlock(&drvdata->mem_lock);
return;
}
diff --git a/drivers/hwtracing/coresight/coresight-tpiu.c b/drivers/hwtracing/coresight/coresight-tpiu.c
index ff579a7..7473c6e 100644
--- a/drivers/hwtracing/coresight/coresight-tpiu.c
+++ b/drivers/hwtracing/coresight/coresight-tpiu.c
@@ -47,8 +47,9 @@
/** register definition **/
/* FFSR - 0x300 */
-#define FFSR_FT_STOPPED BIT(1)
+#define FFSR_FT_STOPPED_BIT 1
/* FFCR - 0x304 */
+#define FFCR_FON_MAN_BIT 6
#define FFCR_FON_MAN BIT(6)
#define FFCR_STOP_FI BIT(12)
@@ -93,9 +94,9 @@ static void tpiu_disable_hw(struct tpiu_drvdata *drvdata)
/* Generate manual flush */
writel_relaxed(FFCR_STOP_FI | FFCR_FON_MAN, drvdata->base + TPIU_FFCR);
/* Wait for flush to complete */
- coresight_timeout(drvdata->base, TPIU_FFCR, FFCR_FON_MAN, 0);
+ coresight_timeout(drvdata->base, TPIU_FFCR, FFCR_FON_MAN_BIT, 0);
/* Wait for formatter to stop */
- coresight_timeout(drvdata->base, TPIU_FFSR, FFSR_FT_STOPPED, 1);
+ coresight_timeout(drvdata->base, TPIU_FFSR, FFSR_FT_STOPPED_BIT, 1);
CS_LOCK(drvdata->base);
}
diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
index 85a16b13..f48dc11 100644
--- a/drivers/hwtracing/coresight/coresight.c
+++ b/drivers/hwtracing/coresight/coresight.c
@@ -106,7 +106,7 @@ static int coresight_find_link_inport(struct coresight_device *csdev,
dev_err(&csdev->dev, "couldn't find inport, parent: %s, child: %s\n",
dev_name(&parent->dev), dev_name(&csdev->dev));
- return 0;
+ return -ENODEV;
}
static int coresight_find_link_outport(struct coresight_device *csdev,
@@ -124,7 +124,7 @@ static int coresight_find_link_outport(struct coresight_device *csdev,
dev_err(&csdev->dev, "couldn't find outport, parent: %s, child: %s\n",
dev_name(&csdev->dev), dev_name(&child->dev));
- return 0;
+ return -ENODEV;
}
static int coresight_enable_sink(struct coresight_device *csdev, u32 mode)
@@ -178,6 +178,9 @@ static int coresight_enable_link(struct coresight_device *csdev,
else
refport = 0;
+ if (refport < 0)
+ return refport;
+
if (atomic_inc_return(&csdev->refcnt[refport]) == 1) {
if (link_ops(csdev)->enable) {
ret = link_ops(csdev)->enable(csdev, inport, outport);
diff --git a/drivers/hwtracing/intel_th/msu.c b/drivers/hwtracing/intel_th/msu.c
index e8d55a1..f91d9fa 100644
--- a/drivers/hwtracing/intel_th/msu.c
+++ b/drivers/hwtracing/intel_th/msu.c
@@ -1429,7 +1429,8 @@ nr_pages_store(struct device *dev, struct device_attribute *attr,
if (!end)
break;
- len -= end - p;
+ /* consume the number and the following comma, hence +1 */
+ len -= end - p + 1;
p = end + 1;
} while (len);
diff --git a/drivers/hwtracing/stm/core.c b/drivers/hwtracing/stm/core.c
index 2dd60ee..d89101d 100644
--- a/drivers/hwtracing/stm/core.c
+++ b/drivers/hwtracing/stm/core.c
@@ -27,6 +27,7 @@
#include <linux/stm.h>
#include <linux/fs.h>
#include <linux/mm.h>
+#include <linux/vmalloc.h>
#include "stm.h"
#include <uapi/linux/stm.h>
@@ -688,7 +689,7 @@ static void stm_device_release(struct device *dev)
{
struct stm_device *stm = to_stm_device(dev);
- kfree(stm);
+ vfree(stm);
}
int stm_register_device(struct device *parent, struct stm_data *stm_data,
@@ -705,7 +706,7 @@ int stm_register_device(struct device *parent, struct stm_data *stm_data,
return -EINVAL;
nmasters = stm_data->sw_end - stm_data->sw_start + 1;
- stm = kzalloc(sizeof(*stm) + nmasters * sizeof(void *), GFP_KERNEL);
+ stm = vzalloc(sizeof(*stm) + nmasters * sizeof(void *));
if (!stm)
return -ENOMEM;
@@ -758,7 +759,7 @@ int stm_register_device(struct device *parent, struct stm_data *stm_data,
/* matches device_initialize() above */
put_device(&stm->dev);
err_free:
- kfree(stm);
+ vfree(stm);
return err;
}
diff --git a/drivers/i2c/busses/i2c-axxia.c b/drivers/i2c/busses/i2c-axxia.c
index 4351a93..96a6d5d 100644
--- a/drivers/i2c/busses/i2c-axxia.c
+++ b/drivers/i2c/busses/i2c-axxia.c
@@ -74,8 +74,7 @@
MST_STATUS_ND)
#define MST_STATUS_ERR (MST_STATUS_NAK | \
MST_STATUS_AL | \
- MST_STATUS_IP | \
- MST_STATUS_TSS)
+ MST_STATUS_IP)
#define MST_TX_BYTES_XFRD 0x50
#define MST_RX_BYTES_XFRD 0x54
#define SCL_HIGH_PERIOD 0x80
@@ -241,7 +240,7 @@ static int axxia_i2c_empty_rx_fifo(struct axxia_i2c_dev *idev)
*/
if (c <= 0 || c > I2C_SMBUS_BLOCK_MAX) {
idev->msg_err = -EPROTO;
- i2c_int_disable(idev, ~0);
+ i2c_int_disable(idev, ~MST_STATUS_TSS);
complete(&idev->msg_complete);
break;
}
@@ -299,14 +298,19 @@ static irqreturn_t axxia_i2c_isr(int irq, void *_dev)
if (status & MST_STATUS_SCC) {
/* Stop completed */
- i2c_int_disable(idev, ~0);
+ i2c_int_disable(idev, ~MST_STATUS_TSS);
complete(&idev->msg_complete);
} else if (status & MST_STATUS_SNS) {
/* Transfer done */
- i2c_int_disable(idev, ~0);
+ i2c_int_disable(idev, ~MST_STATUS_TSS);
if (i2c_m_rd(idev->msg) && idev->msg_xfrd < idev->msg->len)
axxia_i2c_empty_rx_fifo(idev);
complete(&idev->msg_complete);
+ } else if (status & MST_STATUS_TSS) {
+ /* Transfer timeout */
+ idev->msg_err = -ETIMEDOUT;
+ i2c_int_disable(idev, ~MST_STATUS_TSS);
+ complete(&idev->msg_complete);
} else if (unlikely(status & MST_STATUS_ERR)) {
/* Transfer error */
i2c_int_disable(idev, ~0);
@@ -339,10 +343,10 @@ static int axxia_i2c_xfer_msg(struct axxia_i2c_dev *idev, struct i2c_msg *msg)
u32 rx_xfer, tx_xfer;
u32 addr_1, addr_2;
unsigned long time_left;
+ unsigned int wt_value;
idev->msg = msg;
idev->msg_xfrd = 0;
- idev->msg_err = 0;
reinit_completion(&idev->msg_complete);
if (i2c_m_ten(msg)) {
@@ -382,9 +386,18 @@ static int axxia_i2c_xfer_msg(struct axxia_i2c_dev *idev, struct i2c_msg *msg)
else if (axxia_i2c_fill_tx_fifo(idev) != 0)
int_mask |= MST_STATUS_TFL;
+ wt_value = WT_VALUE(readl(idev->base + WAIT_TIMER_CONTROL));
+ /* Disable wait timer temporarly */
+ writel(wt_value, idev->base + WAIT_TIMER_CONTROL);
+ /* Check if timeout error happened */
+ if (idev->msg_err)
+ goto out;
+
/* Start manual mode */
writel(CMD_MANUAL, idev->base + MST_COMMAND);
+ writel(WT_EN | wt_value, idev->base + WAIT_TIMER_CONTROL);
+
i2c_int_enable(idev, int_mask);
time_left = wait_for_completion_timeout(&idev->msg_complete,
@@ -395,13 +408,15 @@ static int axxia_i2c_xfer_msg(struct axxia_i2c_dev *idev, struct i2c_msg *msg)
if (readl(idev->base + MST_COMMAND) & CMD_BUSY)
dev_warn(idev->dev, "busy after xfer\n");
- if (time_left == 0)
+ if (time_left == 0) {
idev->msg_err = -ETIMEDOUT;
-
- if (idev->msg_err == -ETIMEDOUT)
i2c_recover_bus(&idev->adapter);
+ axxia_i2c_init(idev);
+ }
- if (unlikely(idev->msg_err) && idev->msg_err != -ENXIO)
+out:
+ if (unlikely(idev->msg_err) && idev->msg_err != -ENXIO &&
+ idev->msg_err != -ETIMEDOUT)
axxia_i2c_init(idev);
return idev->msg_err;
@@ -409,7 +424,7 @@ static int axxia_i2c_xfer_msg(struct axxia_i2c_dev *idev, struct i2c_msg *msg)
static int axxia_i2c_stop(struct axxia_i2c_dev *idev)
{
- u32 int_mask = MST_STATUS_ERR | MST_STATUS_SCC;
+ u32 int_mask = MST_STATUS_ERR | MST_STATUS_SCC | MST_STATUS_TSS;
unsigned long time_left;
reinit_completion(&idev->msg_complete);
@@ -436,6 +451,9 @@ axxia_i2c_xfer(struct i2c_adapter *adap, struct i2c_msg msgs[], int num)
int i;
int ret = 0;
+ idev->msg_err = 0;
+ i2c_int_enable(idev, MST_STATUS_TSS);
+
for (i = 0; ret == 0 && i < num; ++i)
ret = axxia_i2c_xfer_msg(idev, &msgs[i]);
diff --git a/drivers/i2c/busses/i2c-bcm2835.c b/drivers/i2c/busses/i2c-bcm2835.c
index f283b71..7ed0986 100644
--- a/drivers/i2c/busses/i2c-bcm2835.c
+++ b/drivers/i2c/busses/i2c-bcm2835.c
@@ -128,7 +128,9 @@ static irqreturn_t bcm2835_i2c_isr(int this_irq, void *data)
}
if (val & BCM2835_I2C_S_DONE) {
- if (i2c_dev->curr_msg->flags & I2C_M_RD) {
+ if (!i2c_dev->curr_msg) {
+ dev_err(i2c_dev->dev, "Got unexpected interrupt (from firmware?)\n");
+ } else if (i2c_dev->curr_msg->flags & I2C_M_RD) {
bcm2835_drain_rxfifo(i2c_dev);
val = bcm2835_i2c_readl(i2c_dev, BCM2835_I2C_S);
}
diff --git a/drivers/i2c/busses/i2c-davinci.c b/drivers/i2c/busses/i2c-davinci.c
index 9e7ef5c..b2d8b63 100644
--- a/drivers/i2c/busses/i2c-davinci.c
+++ b/drivers/i2c/busses/i2c-davinci.c
@@ -234,12 +234,16 @@ static void i2c_davinci_calc_clk_dividers(struct davinci_i2c_dev *dev)
/*
* It's not always possible to have 1 to 2 ratio when d=7, so fall back
* to minimal possible clkh in this case.
+ *
+ * Note:
+ * CLKH is not allowed to be 0, in this case I2C clock is not generated
+ * at all
*/
- if (clk >= clkl + d) {
+ if (clk > clkl + d) {
clkh = clk - clkl - d;
clkl -= d;
} else {
- clkh = 0;
+ clkh = 1;
clkl = clk - (d << 1);
}
diff --git a/drivers/i2c/busses/i2c-designware-core.c b/drivers/i2c/busses/i2c-designware-core.c
index 340e037..884c1ec 100644
--- a/drivers/i2c/busses/i2c-designware-core.c
+++ b/drivers/i2c/busses/i2c-designware-core.c
@@ -507,7 +507,10 @@ static void i2c_dw_xfer_init(struct dw_i2c_dev *dev)
i2c_dw_disable_int(dev);
/* Enable the adapter */
- __i2c_dw_enable_and_wait(dev, true);
+ __i2c_dw_enable(dev, true);
+
+ /* Dummy read to avoid the register getting stuck on Bay Trail */
+ dw_readl(dev, DW_IC_ENABLE_STATUS);
/* Clear and enable interrupts */
dw_readl(dev, DW_IC_CLR_INTR);
diff --git a/drivers/i2c/busses/i2c-i801.c b/drivers/i2c/busses/i2c-i801.c
index e6fe21a..26f1691 100644
--- a/drivers/i2c/busses/i2c-i801.c
+++ b/drivers/i2c/busses/i2c-i801.c
@@ -135,6 +135,7 @@
#define SBREG_BAR 0x10
#define SBREG_SMBCTRL 0xc6000c
+#define SBREG_SMBCTRL_DNV 0xcf000c
/* Host status bits for SMBPCISTS */
#define SMBPCISTS_INTS 0x08
@@ -243,6 +244,7 @@ struct i801_priv {
struct i2c_adapter adapter;
unsigned long smba;
unsigned char original_hstcfg;
+ unsigned char original_slvcmd;
struct pci_dev *pci_dev;
unsigned int features;
@@ -962,13 +964,24 @@ static int i801_enable_host_notify(struct i2c_adapter *adapter)
if (!priv->host_notify)
return -ENOMEM;
- outb_p(SMBSLVCMD_HST_NTFY_INTREN, SMBSLVCMD(priv));
+ if (!(SMBSLVCMD_HST_NTFY_INTREN & priv->original_slvcmd))
+ outb_p(SMBSLVCMD_HST_NTFY_INTREN | priv->original_slvcmd,
+ SMBSLVCMD(priv));
+
/* clear Host Notify bit to allow a new notification */
outb_p(SMBSLVSTS_HST_NTFY_STS, SMBSLVSTS(priv));
return 0;
}
+static void i801_disable_host_notify(struct i801_priv *priv)
+{
+ if (!(priv->features & FEATURE_HOST_NOTIFY))
+ return;
+
+ outb_p(priv->original_slvcmd, SMBSLVCMD(priv));
+}
+
static const struct i2c_algorithm smbus_algorithm = {
.smbus_xfer = i801_access,
.functionality = i801_func,
@@ -1375,7 +1388,11 @@ static void i801_add_tco(struct i801_priv *priv)
spin_unlock(&p2sb_spinlock);
res = &tco_res[ICH_RES_MEM_OFF];
- res->start = (resource_size_t)base64_addr + SBREG_SMBCTRL;
+ if (pci_dev->device == PCI_DEVICE_ID_INTEL_DNV_SMBUS)
+ res->start = (resource_size_t)base64_addr + SBREG_SMBCTRL_DNV;
+ else
+ res->start = (resource_size_t)base64_addr + SBREG_SMBCTRL;
+
res->end = res->start + 3;
res->flags = IORESOURCE_MEM;
@@ -1391,6 +1408,13 @@ static void i801_add_tco(struct i801_priv *priv)
}
#ifdef CONFIG_ACPI
+static bool i801_acpi_is_smbus_ioport(const struct i801_priv *priv,
+ acpi_physical_address address)
+{
+ return address >= priv->smba &&
+ address <= pci_resource_end(priv->pci_dev, SMBBAR);
+}
+
static acpi_status
i801_acpi_io_handler(u32 function, acpi_physical_address address, u32 bits,
u64 *value, void *handler_context, void *region_context)
@@ -1406,7 +1430,7 @@ i801_acpi_io_handler(u32 function, acpi_physical_address address, u32 bits,
*/
mutex_lock(&priv->acpi_lock);
- if (!priv->acpi_reserved) {
+ if (!priv->acpi_reserved && i801_acpi_is_smbus_ioport(priv, address)) {
priv->acpi_reserved = true;
dev_warn(&pdev->dev, "BIOS is accessing SMBus registers\n");
@@ -1589,6 +1613,10 @@ static int i801_probe(struct pci_dev *dev, const struct pci_device_id *id)
outb_p(inb_p(SMBAUXCTL(priv)) &
~(SMBAUXCTL_CRC | SMBAUXCTL_E32B), SMBAUXCTL(priv));
+ /* Remember original Host Notify setting */
+ if (priv->features & FEATURE_HOST_NOTIFY)
+ priv->original_slvcmd = inb_p(SMBSLVCMD(priv));
+
/* Default timeout in interrupt mode: 200 ms */
priv->adapter.timeout = HZ / 5;
@@ -1666,6 +1694,7 @@ static void i801_remove(struct pci_dev *dev)
pm_runtime_forbid(&dev->dev);
pm_runtime_get_noresume(&dev->dev);
+ i801_disable_host_notify(priv);
i801_del_mux(priv);
i2c_del_adapter(&priv->adapter);
i801_acpi_remove(priv);
@@ -1679,6 +1708,15 @@ static void i801_remove(struct pci_dev *dev)
*/
}
+static void i801_shutdown(struct pci_dev *dev)
+{
+ struct i801_priv *priv = pci_get_drvdata(dev);
+
+ /* Restore config registers to avoid hard hang on some systems */
+ i801_disable_host_notify(priv);
+ pci_write_config_byte(dev, SMBHSTCFG, priv->original_hstcfg);
+}
+
#ifdef CONFIG_PM
static int i801_suspend(struct device *dev)
{
@@ -1711,6 +1749,7 @@ static struct pci_driver i801_driver = {
.id_table = i801_ids,
.probe = i801_probe,
.remove = i801_remove,
+ .shutdown = i801_shutdown,
.driver = {
.pm = &i801_pm_ops,
},
diff --git a/drivers/i2c/busses/i2c-imx.c b/drivers/i2c/busses/i2c-imx.c
index 47fc1f1..c4188308 100644
--- a/drivers/i2c/busses/i2c-imx.c
+++ b/drivers/i2c/busses/i2c-imx.c
@@ -376,6 +376,7 @@ static int i2c_imx_dma_xfer(struct imx_i2c_struct *i2c_imx,
goto err_desc;
}
+ reinit_completion(&dma->cmd_complete);
txdesc->callback = i2c_imx_dma_callback;
txdesc->callback_param = i2c_imx;
if (dma_submit_error(dmaengine_submit(txdesc))) {
@@ -619,7 +620,6 @@ static int i2c_imx_dma_write(struct imx_i2c_struct *i2c_imx,
* The first byte must be transmitted by the CPU.
*/
imx_i2c_write_reg(msgs->addr << 1, i2c_imx, IMX_I2C_I2DR);
- reinit_completion(&i2c_imx->dma->cmd_complete);
time_left = wait_for_completion_timeout(
&i2c_imx->dma->cmd_complete,
msecs_to_jiffies(DMA_TIMEOUT));
@@ -665,9 +665,6 @@ static int i2c_imx_dma_read(struct imx_i2c_struct *i2c_imx,
struct imx_i2c_dma *dma = i2c_imx->dma;
struct device *dev = &i2c_imx->adapter.dev;
- temp = imx_i2c_read_reg(i2c_imx, IMX_I2C_I2CR);
- temp |= I2CR_DMAEN;
- imx_i2c_write_reg(temp, i2c_imx, IMX_I2C_I2CR);
dma->chan_using = dma->chan_rx;
dma->dma_transfer_dir = DMA_DEV_TO_MEM;
@@ -678,7 +675,6 @@ static int i2c_imx_dma_read(struct imx_i2c_struct *i2c_imx,
if (result)
return result;
- reinit_completion(&i2c_imx->dma->cmd_complete);
time_left = wait_for_completion_timeout(
&i2c_imx->dma->cmd_complete,
msecs_to_jiffies(DMA_TIMEOUT));
@@ -781,6 +777,7 @@ static int i2c_imx_read(struct imx_i2c_struct *i2c_imx, struct i2c_msg *msgs, bo
int i, result;
unsigned int temp;
int block_data = msgs->flags & I2C_M_RECV_LEN;
+ int use_dma = i2c_imx->dma && msgs->len >= DMA_THRESHOLD && !block_data;
dev_dbg(&i2c_imx->adapter.dev,
"<%s> write slave address: addr=0x%x\n",
@@ -807,12 +804,14 @@ static int i2c_imx_read(struct imx_i2c_struct *i2c_imx, struct i2c_msg *msgs, bo
*/
if ((msgs->len - 1) || block_data)
temp &= ~I2CR_TXAK;
+ if (use_dma)
+ temp |= I2CR_DMAEN;
imx_i2c_write_reg(temp, i2c_imx, IMX_I2C_I2CR);
imx_i2c_read_reg(i2c_imx, IMX_I2C_I2DR); /* dummy read */
dev_dbg(&i2c_imx->adapter.dev, "<%s> read data\n", __func__);
- if (i2c_imx->dma && msgs->len >= DMA_THRESHOLD && !block_data)
+ if (use_dma)
return i2c_imx_dma_read(i2c_imx, msgs, is_lastmsg);
/* read data */
diff --git a/drivers/i2c/busses/i2c-ismt.c b/drivers/i2c/busses/i2c-ismt.c
index 7aea288..b51adff 100644
--- a/drivers/i2c/busses/i2c-ismt.c
+++ b/drivers/i2c/busses/i2c-ismt.c
@@ -589,7 +589,7 @@ static int ismt_access(struct i2c_adapter *adap, u16 addr,
/* unmap the data buffer */
if (dma_size != 0)
- dma_unmap_single(&adap->dev, dma_addr, dma_size, dma_direction);
+ dma_unmap_single(dev, dma_addr, dma_size, dma_direction);
if (unlikely(!time_left)) {
dev_err(dev, "completion wait timed out\n");
diff --git a/drivers/i2c/busses/i2c-mv64xxx.c b/drivers/i2c/busses/i2c-mv64xxx.c
index b4dec08..5c9dea7 100644
--- a/drivers/i2c/busses/i2c-mv64xxx.c
+++ b/drivers/i2c/busses/i2c-mv64xxx.c
@@ -848,12 +848,16 @@ mv64xxx_of_config(struct mv64xxx_i2c_data *drv_data,
*/
if (of_device_is_compatible(np, "marvell,mv78230-i2c")) {
drv_data->offload_enabled = true;
- drv_data->errata_delay = true;
+ /* The delay is only needed in standard mode (100kHz) */
+ if (bus_freq <= 100000)
+ drv_data->errata_delay = true;
}
if (of_device_is_compatible(np, "marvell,mv78230-a0-i2c")) {
drv_data->offload_enabled = false;
- drv_data->errata_delay = true;
+ /* The delay is only needed in standard mode (100kHz) */
+ if (bus_freq <= 100000)
+ drv_data->errata_delay = true;
}
if (of_device_is_compatible(np, "allwinner,sun6i-a31-i2c"))
diff --git a/drivers/i2c/busses/i2c-qcom-geni.c b/drivers/i2c/busses/i2c-qcom-geni.c
index c712866..8a1543b 100644
--- a/drivers/i2c/busses/i2c-qcom-geni.c
+++ b/drivers/i2c/busses/i2c-qcom-geni.c
@@ -1040,6 +1040,7 @@ static struct platform_driver geni_i2c_driver = {
.name = "i2c_geni",
.pm = &geni_i2c_pm_ops,
.of_match_table = geni_i2c_dt_match,
+ .probe_type = PROBE_PREFER_ASYNCHRONOUS,
},
};
diff --git a/drivers/i2c/busses/i2c-rcar.c b/drivers/i2c/busses/i2c-rcar.c
index 726615e..93b8069 100644
--- a/drivers/i2c/busses/i2c-rcar.c
+++ b/drivers/i2c/busses/i2c-rcar.c
@@ -700,6 +700,8 @@ static int rcar_i2c_master_xfer(struct i2c_adapter *adap,
pm_runtime_get_sync(dev);
+ rcar_i2c_init(priv);
+
ret = rcar_i2c_bus_barrier(priv);
if (ret < 0)
goto out;
@@ -721,8 +723,12 @@ static int rcar_i2c_master_xfer(struct i2c_adapter *adap,
time_left = wait_event_timeout(priv->wait, priv->flags & ID_DONE,
num * adap->timeout);
- if (!time_left) {
+
+ /* cleanup DMA if it couldn't complete properly due to an error */
+ if (priv->dma_direction != DMA_NONE)
rcar_i2c_cleanup_dma(priv);
+
+ if (!time_left) {
rcar_i2c_init(priv);
ret = -ETIMEDOUT;
} else if (priv->flags & ID_NACK) {
@@ -857,8 +863,6 @@ static int rcar_i2c_probe(struct platform_device *pdev)
if (ret < 0)
goto out_pm_put;
- rcar_i2c_init(priv);
-
/* Don't suspend when multi-master to keep arbitration working */
if (of_property_read_bool(dev->of_node, "multi-master"))
priv->flags |= ID_P_PM_BLOCKED;
diff --git a/drivers/i2c/busses/i2c-scmi.c b/drivers/i2c/busses/i2c-scmi.c
index 7aa7b9c..d2178f7 100644
--- a/drivers/i2c/busses/i2c-scmi.c
+++ b/drivers/i2c/busses/i2c-scmi.c
@@ -152,6 +152,7 @@ acpi_smbus_cmi_access(struct i2c_adapter *adap, u16 addr, unsigned short flags,
mt_params[3].type = ACPI_TYPE_INTEGER;
mt_params[3].integer.value = len;
mt_params[4].type = ACPI_TYPE_BUFFER;
+ mt_params[4].buffer.length = len;
mt_params[4].buffer.pointer = data->block + 1;
}
break;
@@ -363,6 +364,7 @@ static int acpi_smbus_cmi_add(struct acpi_device *device)
{
struct acpi_smbus_cmi *smbus_cmi;
const struct acpi_device_id *id;
+ int ret;
smbus_cmi = kzalloc(sizeof(struct acpi_smbus_cmi), GFP_KERNEL);
if (!smbus_cmi)
@@ -384,8 +386,10 @@ static int acpi_smbus_cmi_add(struct acpi_device *device)
acpi_walk_namespace(ACPI_TYPE_METHOD, smbus_cmi->handle, 1,
acpi_smbus_cmi_query_methods, NULL, smbus_cmi, NULL);
- if (smbus_cmi->cap_info == 0)
+ if (smbus_cmi->cap_info == 0) {
+ ret = -ENODEV;
goto err;
+ }
snprintf(smbus_cmi->adapter.name, sizeof(smbus_cmi->adapter.name),
"SMBus CMI adapter %s",
@@ -396,7 +400,8 @@ static int acpi_smbus_cmi_add(struct acpi_device *device)
smbus_cmi->adapter.class = I2C_CLASS_HWMON | I2C_CLASS_SPD;
smbus_cmi->adapter.dev.parent = &device->dev;
- if (i2c_add_adapter(&smbus_cmi->adapter)) {
+ ret = i2c_add_adapter(&smbus_cmi->adapter);
+ if (ret) {
dev_err(&device->dev, "Couldn't register adapter!\n");
goto err;
}
@@ -406,7 +411,7 @@ static int acpi_smbus_cmi_add(struct acpi_device *device)
err:
kfree(smbus_cmi);
device->driver_data = NULL;
- return -EIO;
+ return ret;
}
static int acpi_smbus_cmi_remove(struct acpi_device *device)
diff --git a/drivers/i2c/busses/i2c-tegra.c b/drivers/i2c/busses/i2c-tegra.c
index 4af9bba..586e557 100644
--- a/drivers/i2c/busses/i2c-tegra.c
+++ b/drivers/i2c/busses/i2c-tegra.c
@@ -547,6 +547,14 @@ static int tegra_i2c_disable_packet_mode(struct tegra_i2c_dev *i2c_dev)
{
u32 cnfg;
+ /*
+ * NACK interrupt is generated before the I2C controller generates
+ * the STOP condition on the bus. So wait for 2 clock periods
+ * before disabling the controller so that the STOP condition has
+ * been delivered properly.
+ */
+ udelay(DIV_ROUND_UP(2 * 1000000, i2c_dev->bus_clk_rate));
+
cnfg = i2c_readl(i2c_dev, I2C_CNFG);
if (cnfg & I2C_CNFG_PACKET_MODE_EN)
i2c_writel(i2c_dev, cnfg & ~I2C_CNFG_PACKET_MODE_EN, I2C_CNFG);
@@ -708,15 +716,6 @@ static int tegra_i2c_xfer_msg(struct tegra_i2c_dev *i2c_dev,
if (likely(i2c_dev->msg_err == I2C_ERR_NONE))
return 0;
- /*
- * NACK interrupt is generated before the I2C controller generates
- * the STOP condition on the bus. So wait for 2 clock periods
- * before resetting the controller so that the STOP condition has
- * been delivered properly.
- */
- if (i2c_dev->msg_err == I2C_ERR_NO_ACK)
- udelay(DIV_ROUND_UP(2 * 1000000, i2c_dev->bus_clk_rate));
-
tegra_i2c_init(i2c_dev);
if (i2c_dev->msg_err == I2C_ERR_NO_ACK) {
if (msg->flags & I2C_M_IGNORE_NAK)
diff --git a/drivers/i2c/busses/i2c-uniphier-f.c b/drivers/i2c/busses/i2c-uniphier-f.c
index db9105e..0da4991 100644
--- a/drivers/i2c/busses/i2c-uniphier-f.c
+++ b/drivers/i2c/busses/i2c-uniphier-f.c
@@ -400,11 +400,8 @@ static int uniphier_fi2c_master_xfer(struct i2c_adapter *adap,
return ret;
for (msg = msgs; msg < emsg; msg++) {
- /* If next message is read, skip the stop condition */
- bool stop = !(msg + 1 < emsg && msg[1].flags & I2C_M_RD);
- /* but, force it if I2C_M_STOP is set */
- if (msg->flags & I2C_M_STOP)
- stop = true;
+ /* Emit STOP if it is the last message or I2C_M_STOP is set. */
+ bool stop = (msg + 1 == emsg) || (msg->flags & I2C_M_STOP);
ret = uniphier_fi2c_master_xfer_one(adap, msg, stop);
if (ret)
diff --git a/drivers/i2c/busses/i2c-uniphier.c b/drivers/i2c/busses/i2c-uniphier.c
index 56e92af..fdfcee92 100644
--- a/drivers/i2c/busses/i2c-uniphier.c
+++ b/drivers/i2c/busses/i2c-uniphier.c
@@ -247,11 +247,8 @@ static int uniphier_i2c_master_xfer(struct i2c_adapter *adap,
return ret;
for (msg = msgs; msg < emsg; msg++) {
- /* If next message is read, skip the stop condition */
- bool stop = !(msg + 1 < emsg && msg[1].flags & I2C_M_RD);
- /* but, force it if I2C_M_STOP is set */
- if (msg->flags & I2C_M_STOP)
- stop = true;
+ /* Emit STOP if it is the last message or I2C_M_STOP is set. */
+ bool stop = (msg + 1 == emsg) || (msg->flags & I2C_M_STOP);
ret = uniphier_i2c_master_xfer_one(adap, msg, stop);
if (ret)
diff --git a/drivers/i2c/busses/i2c-xiic.c b/drivers/i2c/busses/i2c-xiic.c
index 66bce3b..b72cf2f 100644
--- a/drivers/i2c/busses/i2c-xiic.c
+++ b/drivers/i2c/busses/i2c-xiic.c
@@ -538,6 +538,7 @@ static void xiic_start_recv(struct xiic_i2c *i2c)
{
u8 rx_watermark;
struct i2c_msg *msg = i2c->rx_msg = i2c->tx_msg;
+ unsigned long flags;
/* Clear and enable Rx full interrupt. */
xiic_irq_clr_en(i2c, XIIC_INTR_RX_FULL_MASK | XIIC_INTR_TX_ERROR_MASK);
@@ -553,6 +554,7 @@ static void xiic_start_recv(struct xiic_i2c *i2c)
rx_watermark = IIC_RX_FIFO_DEPTH;
xiic_setreg8(i2c, XIIC_RFD_REG_OFFSET, rx_watermark - 1);
+ local_irq_save(flags);
if (!(msg->flags & I2C_M_NOSTART))
/* write the address */
xiic_setreg16(i2c, XIIC_DTR_REG_OFFSET,
@@ -563,6 +565,8 @@ static void xiic_start_recv(struct xiic_i2c *i2c)
xiic_setreg16(i2c, XIIC_DTR_REG_OFFSET,
msg->len | ((i2c->nmsgs == 1) ? XIIC_TX_DYN_STOP_MASK : 0));
+ local_irq_restore(flags);
+
if (i2c->nmsgs == 1)
/* very last, enable bus not busy as well */
xiic_irq_clr_en(i2c, XIIC_INTR_BNB_MASK);
diff --git a/drivers/ide/ide-cd.c b/drivers/ide/ide-cd.c
index bf9a2ad..883fe2c 100644
--- a/drivers/ide/ide-cd.c
+++ b/drivers/ide/ide-cd.c
@@ -1593,6 +1593,8 @@ static int idecd_open(struct block_device *bdev, fmode_t mode)
struct cdrom_info *info;
int rc = -ENXIO;
+ check_disk_change(bdev);
+
mutex_lock(&ide_cd_mutex);
info = ide_cd_get(bdev->bd_disk);
if (!info)
diff --git a/drivers/ide/pmac.c b/drivers/ide/pmac.c
index 0c5d3a9..b20025a 100644
--- a/drivers/ide/pmac.c
+++ b/drivers/ide/pmac.c
@@ -920,6 +920,7 @@ static u8 pmac_ide_cable_detect(ide_hwif_t *hwif)
struct device_node *root = of_find_node_by_path("/");
const char *model = of_get_property(root, "model", NULL);
+ of_node_put(root);
/* Get cable type from device-tree. */
if (cable && !strncmp(cable, "80-", 3)) {
/* Some drives fail to detect 80c cable in PowerBook */
diff --git a/drivers/iio/adc/at91_adc.c b/drivers/iio/adc/at91_adc.c
index bbdac07..e3e2155 100644
--- a/drivers/iio/adc/at91_adc.c
+++ b/drivers/iio/adc/at91_adc.c
@@ -247,12 +247,14 @@ static irqreturn_t at91_adc_trigger_handler(int irq, void *p)
struct iio_poll_func *pf = p;
struct iio_dev *idev = pf->indio_dev;
struct at91_adc_state *st = iio_priv(idev);
+ struct iio_chan_spec const *chan;
int i, j = 0;
for (i = 0; i < idev->masklength; i++) {
if (!test_bit(i, idev->active_scan_mask))
continue;
- st->buffer[j] = at91_adc_readl(st, AT91_ADC_CHAN(st, i));
+ chan = idev->channels + i;
+ st->buffer[j] = at91_adc_readl(st, AT91_ADC_CHAN(st, chan->channel));
j++;
}
@@ -278,6 +280,8 @@ static void handle_adc_eoc_trigger(int irq, struct iio_dev *idev)
iio_trigger_poll(idev->trig);
} else {
st->last_value = at91_adc_readl(st, AT91_ADC_CHAN(st, st->chnb));
+ /* Needed to ACK the DRDY interruption */
+ at91_adc_readl(st, AT91_ADC_LCDR);
st->done = true;
wake_up_interruptible(&st->wq_data_avail);
}
diff --git a/drivers/iio/adc/axp288_adc.c b/drivers/iio/adc/axp288_adc.c
index 64799ad..7fd2494 100644
--- a/drivers/iio/adc/axp288_adc.c
+++ b/drivers/iio/adc/axp288_adc.c
@@ -28,6 +28,8 @@
#include <linux/iio/driver.h>
#define AXP288_ADC_EN_MASK 0xF1
+#define AXP288_ADC_TS_PIN_GPADC 0xF2
+#define AXP288_ADC_TS_PIN_ON 0xF3
enum axp288_adc_id {
AXP288_ADC_TS,
@@ -121,6 +123,16 @@ static int axp288_adc_read_channel(int *val, unsigned long address,
return IIO_VAL_INT;
}
+static int axp288_adc_set_ts(struct regmap *regmap, unsigned int mode,
+ unsigned long address)
+{
+ /* channels other than GPADC do not need to switch TS pin */
+ if (address != AXP288_GP_ADC_H)
+ return 0;
+
+ return regmap_write(regmap, AXP288_ADC_TS_PIN_CTRL, mode);
+}
+
static int axp288_adc_read_raw(struct iio_dev *indio_dev,
struct iio_chan_spec const *chan,
int *val, int *val2, long mask)
@@ -131,7 +143,16 @@ static int axp288_adc_read_raw(struct iio_dev *indio_dev,
mutex_lock(&indio_dev->mlock);
switch (mask) {
case IIO_CHAN_INFO_RAW:
+ if (axp288_adc_set_ts(info->regmap, AXP288_ADC_TS_PIN_GPADC,
+ chan->address)) {
+ dev_err(&indio_dev->dev, "GPADC mode\n");
+ ret = -EINVAL;
+ break;
+ }
ret = axp288_adc_read_channel(val, chan->address, info->regmap);
+ if (axp288_adc_set_ts(info->regmap, AXP288_ADC_TS_PIN_ON,
+ chan->address))
+ dev_err(&indio_dev->dev, "TS pin restore\n");
break;
default:
ret = -EINVAL;
@@ -141,6 +162,15 @@ static int axp288_adc_read_raw(struct iio_dev *indio_dev,
return ret;
}
+static int axp288_adc_set_state(struct regmap *regmap)
+{
+ /* ADC should be always enabled for internal FG to function */
+ if (regmap_write(regmap, AXP288_ADC_TS_PIN_CTRL, AXP288_ADC_TS_PIN_ON))
+ return -EIO;
+
+ return regmap_write(regmap, AXP20X_ADC_EN1, AXP288_ADC_EN_MASK);
+}
+
static const struct iio_info axp288_adc_iio_info = {
.read_raw = &axp288_adc_read_raw,
.driver_module = THIS_MODULE,
@@ -169,7 +199,7 @@ static int axp288_adc_probe(struct platform_device *pdev)
* Set ADC to enabled state at all time, including system suspend.
* otherwise internal fuel gauge functionality may be affected.
*/
- ret = regmap_write(info->regmap, AXP20X_ADC_EN1, AXP288_ADC_EN_MASK);
+ ret = axp288_adc_set_state(axp20x->regmap);
if (ret) {
dev_err(&pdev->dev, "unable to enable ADC device\n");
return ret;
diff --git a/drivers/iio/adc/fsl-imx25-gcq.c b/drivers/iio/adc/fsl-imx25-gcq.c
index ea264fa..929c617 100644
--- a/drivers/iio/adc/fsl-imx25-gcq.c
+++ b/drivers/iio/adc/fsl-imx25-gcq.c
@@ -209,12 +209,14 @@ static int mx25_gcq_setup_cfgs(struct platform_device *pdev,
ret = of_property_read_u32(child, "reg", ®);
if (ret) {
dev_err(dev, "Failed to get reg property\n");
+ of_node_put(child);
return ret;
}
if (reg >= MX25_NUM_CFGS) {
dev_err(dev,
"reg value is greater than the number of available configuration registers\n");
+ of_node_put(child);
return -EINVAL;
}
@@ -228,6 +230,7 @@ static int mx25_gcq_setup_cfgs(struct platform_device *pdev,
if (IS_ERR(priv->vref[refp])) {
dev_err(dev, "Error, trying to use external voltage reference without a vref-%s regulator.",
mx25_gcq_refp_names[refp]);
+ of_node_put(child);
return PTR_ERR(priv->vref[refp]);
}
priv->channel_vref_mv[reg] =
@@ -240,6 +243,7 @@ static int mx25_gcq_setup_cfgs(struct platform_device *pdev,
break;
default:
dev_err(dev, "Invalid positive reference %d\n", refp);
+ of_node_put(child);
return -EINVAL;
}
@@ -254,10 +258,12 @@ static int mx25_gcq_setup_cfgs(struct platform_device *pdev,
if ((refp & MX25_ADCQ_CFG_REFP_MASK) != refp) {
dev_err(dev, "Invalid fsl,adc-refp property value\n");
+ of_node_put(child);
return -EINVAL;
}
if ((refn & MX25_ADCQ_CFG_REFN_MASK) != refn) {
dev_err(dev, "Invalid fsl,adc-refn property value\n");
+ of_node_put(child);
return -EINVAL;
}
diff --git a/drivers/iio/adc/qcom-rradc.c b/drivers/iio/adc/qcom-rradc.c
index 02dfbf8..313aa0c 100644
--- a/drivers/iio/adc/qcom-rradc.c
+++ b/drivers/iio/adc/qcom-rradc.c
@@ -61,6 +61,8 @@
#define FG_ADC_RR_AUX_THERM_CTRL 0x80
#define FG_ADC_RR_AUX_THERM_TRIGGER 0x81
+#define FG_ADC_RR_AUX_THERM_EVERY_CYCLE_MASK 0x80
+#define FG_ADC_RR_AUX_THERM_EVERY_CYCLE BIT(7)
#define FG_ADC_RR_AUX_THERM_STS 0x82
#define FG_ADC_RR_AUX_THERM_CFG 0x83
#define FG_ADC_RR_AUX_THERM_LSB 0x84
@@ -766,7 +768,6 @@ static int rradc_check_status_ready_with_retry(struct rradc_chip *chip,
rradc_chans[prop->channel].datasheet_name, buf[0]);
if (((prop->channel == RR_ADC_CHG_TEMP) ||
- (prop->channel == RR_ADC_SKIN_TEMP) ||
(prop->channel == RR_ADC_USBIN_I) ||
(prop->channel == RR_ADC_DIE_TEMP)) &&
((!rradc_is_usb_present(chip)))) {
@@ -915,6 +916,12 @@ static int rradc_do_conversion(struct rradc_chip *chip,
}
break;
case RR_ADC_USBIN_V:
+ /* Don't waste time reporting V BUS on boot */
+ if (ktime_get_seconds() <= 10) {
+ rc = -EAGAIN;
+ goto fail;
+ }
+
/* Force conversion every cycle */
rc = rradc_masked_write(chip, FG_ADC_RR_USB_IN_V_TRIGGER,
FG_ADC_RR_USB_IN_V_EVERY_CYCLE_MASK,
@@ -1187,6 +1194,12 @@ static int rradc_probe(struct platform_device *pdev)
if (!chip->usb_trig)
pr_debug("Error obtaining usb power supply\n");
+ rc = rradc_masked_write(chip, FG_ADC_RR_AUX_THERM_TRIGGER,
+ FG_ADC_RR_AUX_THERM_EVERY_CYCLE_MASK,
+ FG_ADC_RR_AUX_THERM_EVERY_CYCLE);
+ if (rc < 0)
+ pr_err("Failed to set FG_ADC_RR_AUX_THERM_EVERY_CYCLE");
+
return devm_iio_device_register(dev, indio_dev);
}
diff --git a/drivers/iio/buffer/kfifo_buf.c b/drivers/iio/buffer/kfifo_buf.c
index c5b999f..e44181f 100644
--- a/drivers/iio/buffer/kfifo_buf.c
+++ b/drivers/iio/buffer/kfifo_buf.c
@@ -19,11 +19,18 @@ struct iio_kfifo {
#define iio_to_kfifo(r) container_of(r, struct iio_kfifo, buffer)
static inline int __iio_allocate_kfifo(struct iio_kfifo *buf,
- int bytes_per_datum, int length)
+ size_t bytes_per_datum, unsigned int length)
{
if ((length == 0) || (bytes_per_datum == 0))
return -EINVAL;
+ /*
+ * Make sure we don't overflow an unsigned int after kfifo rounds up to
+ * the next power of 2.
+ */
+ if (roundup_pow_of_two(length) > UINT_MAX / bytes_per_datum)
+ return -EINVAL;
+
return __kfifo_alloc((struct __kfifo *)&buf->kf, length,
bytes_per_datum, GFP_KERNEL);
}
@@ -64,7 +71,7 @@ static int iio_set_bytes_per_datum_kfifo(struct iio_buffer *r, size_t bpd)
return 0;
}
-static int iio_set_length_kfifo(struct iio_buffer *r, int length)
+static int iio_set_length_kfifo(struct iio_buffer *r, unsigned int length)
{
/* Avoid an invalid state */
if (length < 2)
diff --git a/drivers/iio/dac/ad5064.c b/drivers/iio/dac/ad5064.c
index 6803e4a..94d4677 100644
--- a/drivers/iio/dac/ad5064.c
+++ b/drivers/iio/dac/ad5064.c
@@ -760,6 +760,40 @@ static int ad5064_set_config(struct ad5064_state *st, unsigned int val)
return ad5064_write(st, cmd, 0, val, 0);
}
+static int ad5064_request_vref(struct ad5064_state *st, struct device *dev)
+{
+ unsigned int i;
+ int ret;
+
+ for (i = 0; i < ad5064_num_vref(st); ++i)
+ st->vref_reg[i].supply = ad5064_vref_name(st, i);
+
+ if (!st->chip_info->internal_vref)
+ return devm_regulator_bulk_get(dev, ad5064_num_vref(st),
+ st->vref_reg);
+
+ /*
+ * This assumes that when the regulator has an internal VREF
+ * there is only one external VREF connection, which is
+ * currently the case for all supported devices.
+ */
+ st->vref_reg[0].consumer = devm_regulator_get_optional(dev, "vref");
+ if (!IS_ERR(st->vref_reg[0].consumer))
+ return 0;
+
+ ret = PTR_ERR(st->vref_reg[0].consumer);
+ if (ret != -ENODEV)
+ return ret;
+
+ /* If no external regulator was supplied use the internal VREF */
+ st->use_internal_vref = true;
+ ret = ad5064_set_config(st, AD5064_CONFIG_INT_VREF_ENABLE);
+ if (ret)
+ dev_err(dev, "Failed to enable internal vref: %d\n", ret);
+
+ return ret;
+}
+
static int ad5064_probe(struct device *dev, enum ad5064_type type,
const char *name, ad5064_write_func write)
{
@@ -780,22 +814,11 @@ static int ad5064_probe(struct device *dev, enum ad5064_type type,
st->dev = dev;
st->write = write;
- for (i = 0; i < ad5064_num_vref(st); ++i)
- st->vref_reg[i].supply = ad5064_vref_name(st, i);
+ ret = ad5064_request_vref(st, dev);
+ if (ret)
+ return ret;
- ret = devm_regulator_bulk_get(dev, ad5064_num_vref(st),
- st->vref_reg);
- if (ret) {
- if (!st->chip_info->internal_vref)
- return ret;
- st->use_internal_vref = true;
- ret = ad5064_set_config(st, AD5064_CONFIG_INT_VREF_ENABLE);
- if (ret) {
- dev_err(dev, "Failed to enable internal vref: %d\n",
- ret);
- return ret;
- }
- } else {
+ if (!st->use_internal_vref) {
ret = regulator_bulk_enable(ad5064_num_vref(st), st->vref_reg);
if (ret)
return ret;
diff --git a/drivers/iio/frequency/ad9523.c b/drivers/iio/frequency/ad9523.c
index 99eba52..1642b55 100644
--- a/drivers/iio/frequency/ad9523.c
+++ b/drivers/iio/frequency/ad9523.c
@@ -508,7 +508,7 @@ static ssize_t ad9523_store(struct device *dev,
return ret;
if (!state)
- return 0;
+ return len;
mutex_lock(&indio_dev->mlock);
switch ((u32)this_attr->address) {
@@ -642,7 +642,7 @@ static int ad9523_read_raw(struct iio_dev *indio_dev,
code = (AD9523_CLK_DIST_DIV_PHASE_REV(ret) * 3141592) /
AD9523_CLK_DIST_DIV_REV(ret);
*val = code / 1000000;
- *val2 = (code % 1000000) * 10;
+ *val2 = code % 1000000;
return IIO_VAL_INT_PLUS_MICRO;
default:
return -EINVAL;
diff --git a/drivers/iio/magnetometer/st_magn_buffer.c b/drivers/iio/magnetometer/st_magn_buffer.c
index 0a9e8fa..37ab305 100644
--- a/drivers/iio/magnetometer/st_magn_buffer.c
+++ b/drivers/iio/magnetometer/st_magn_buffer.c
@@ -30,11 +30,6 @@ int st_magn_trig_set_state(struct iio_trigger *trig, bool state)
return st_sensors_set_dataready_irq(indio_dev, state);
}
-static int st_magn_buffer_preenable(struct iio_dev *indio_dev)
-{
- return st_sensors_set_enable(indio_dev, true);
-}
-
static int st_magn_buffer_postenable(struct iio_dev *indio_dev)
{
int err;
@@ -50,7 +45,7 @@ static int st_magn_buffer_postenable(struct iio_dev *indio_dev)
if (err < 0)
goto st_magn_buffer_postenable_error;
- return err;
+ return st_sensors_set_enable(indio_dev, true);
st_magn_buffer_postenable_error:
kfree(mdata->buffer_data);
@@ -63,11 +58,11 @@ static int st_magn_buffer_predisable(struct iio_dev *indio_dev)
int err;
struct st_sensor_data *mdata = iio_priv(indio_dev);
- err = iio_triggered_buffer_predisable(indio_dev);
+ err = st_sensors_set_enable(indio_dev, false);
if (err < 0)
goto st_magn_buffer_predisable_error;
- err = st_sensors_set_enable(indio_dev, false);
+ err = iio_triggered_buffer_predisable(indio_dev);
st_magn_buffer_predisable_error:
kfree(mdata->buffer_data);
@@ -75,7 +70,6 @@ static int st_magn_buffer_predisable(struct iio_dev *indio_dev)
}
static const struct iio_buffer_setup_ops st_magn_buffer_setup_ops = {
- .preenable = &st_magn_buffer_preenable,
.postenable = &st_magn_buffer_postenable,
.predisable = &st_magn_buffer_predisable,
};
diff --git a/drivers/iio/pressure/bmp280-core.c b/drivers/iio/pressure/bmp280-core.c
index 19aa957..c9263ac 100644
--- a/drivers/iio/pressure/bmp280-core.c
+++ b/drivers/iio/pressure/bmp280-core.c
@@ -347,10 +347,9 @@ static int bmp280_read_humid(struct bmp280_data *data, int *val, int *val2)
adc_humidity = be16_to_cpu(tmp);
comp_humidity = bmp280_compensate_humidity(data, adc_humidity);
- *val = comp_humidity;
- *val2 = 1024;
+ *val = comp_humidity * 1000 / 1024;
- return IIO_VAL_FRACTIONAL;
+ return IIO_VAL_INT;
}
static int bmp280_read_raw(struct iio_dev *indio_dev,
diff --git a/drivers/iio/pressure/zpa2326.c b/drivers/iio/pressure/zpa2326.c
index 2a4a62e..cc002b9 100644
--- a/drivers/iio/pressure/zpa2326.c
+++ b/drivers/iio/pressure/zpa2326.c
@@ -869,7 +869,6 @@ static irqreturn_t zpa2326_handle_threaded_irq(int irq, void *data)
static int zpa2326_wait_oneshot_completion(const struct iio_dev *indio_dev,
struct zpa2326_private *private)
{
- int ret;
unsigned int val;
long timeout;
@@ -891,14 +890,11 @@ static int zpa2326_wait_oneshot_completion(const struct iio_dev *indio_dev,
/* Timed out. */
zpa2326_warn(indio_dev, "no one shot interrupt occurred (%ld)",
timeout);
- ret = -ETIME;
- } else if (timeout < 0) {
- zpa2326_warn(indio_dev,
- "wait for one shot interrupt cancelled");
- ret = -ERESTARTSYS;
+ return -ETIME;
}
- return ret;
+ zpa2326_warn(indio_dev, "wait for one shot interrupt cancelled");
+ return -ERESTARTSYS;
}
static int zpa2326_init_managed_irq(struct device *parent,
diff --git a/drivers/infiniband/Kconfig b/drivers/infiniband/Kconfig
index fb3fb89..5d5368a 100644
--- a/drivers/infiniband/Kconfig
+++ b/drivers/infiniband/Kconfig
@@ -34,6 +34,18 @@
libibverbs, libibcm and a hardware driver library from
<http://www.openfabrics.org/git/>.
+config INFINIBAND_USER_ACCESS_UCM
+ bool "Userspace CM (UCM, DEPRECATED)"
+ depends on BROKEN
+ depends on INFINIBAND_USER_ACCESS
+ help
+ The UCM module has known security flaws, which no one is
+ interested to fix. The user-space part of this code was
+ dropped from the upstream a long time ago.
+
+ This option is DEPRECATED and planned to be removed.
+
+
config INFINIBAND_USER_MEM
bool
depends on INFINIBAND_USER_ACCESS != n
diff --git a/drivers/infiniband/core/Makefile b/drivers/infiniband/core/Makefile
index edaae9f..33dc00c 100644
--- a/drivers/infiniband/core/Makefile
+++ b/drivers/infiniband/core/Makefile
@@ -4,8 +4,8 @@
obj-$(CONFIG_INFINIBAND) += ib_core.o ib_cm.o iw_cm.o \
$(infiniband-y)
obj-$(CONFIG_INFINIBAND_USER_MAD) += ib_umad.o
-obj-$(CONFIG_INFINIBAND_USER_ACCESS) += ib_uverbs.o ib_ucm.o \
- $(user_access-y)
+obj-$(CONFIG_INFINIBAND_USER_ACCESS) += ib_uverbs.o $(user_access-y)
+obj-$(CONFIG_INFINIBAND_USER_ACCESS_UCM) += ib_ucm.o $(user_access-y)
ib_core-y := packer.o ud_header.o verbs.o cq.o rw.o sysfs.o \
device.o fmr_pool.o cache.o netlink.o \
diff --git a/drivers/infiniband/core/cache.c b/drivers/infiniband/core/cache.c
index ae04826..a32dd85 100644
--- a/drivers/infiniband/core/cache.c
+++ b/drivers/infiniband/core/cache.c
@@ -437,7 +437,7 @@ static int __ib_cache_gid_get(struct ib_device *ib_dev, u8 port, int index,
return -EINVAL;
if (table->data_vec[index].props & GID_TABLE_ENTRY_INVALID)
- return -EAGAIN;
+ return -EINVAL;
memcpy(gid, &table->data_vec[index].gid, sizeof(*gid));
if (attr) {
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index cbe5324..85d4ef3 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -673,6 +673,7 @@ static int cma_resolve_ib_dev(struct rdma_id_private *id_priv)
dgid = (union ib_gid *) &addr->sib_addr;
pkey = ntohs(addr->sib_pkey);
+ mutex_lock(&lock);
list_for_each_entry(cur_dev, &dev_list, list) {
for (p = 1; p <= cur_dev->device->phys_port_cnt; ++p) {
if (!rdma_cap_af_ib(cur_dev->device, p))
@@ -696,18 +697,19 @@ static int cma_resolve_ib_dev(struct rdma_id_private *id_priv)
cma_dev = cur_dev;
sgid = gid;
id_priv->id.port_num = p;
+ goto found;
}
}
}
}
-
- if (!cma_dev)
- return -ENODEV;
+ mutex_unlock(&lock);
+ return -ENODEV;
found:
cma_attach_to_dev(id_priv, cma_dev);
- addr = (struct sockaddr_ib *) cma_src_addr(id_priv);
- memcpy(&addr->sib_addr, &sgid, sizeof sgid);
+ mutex_unlock(&lock);
+ addr = (struct sockaddr_ib *)cma_src_addr(id_priv);
+ memcpy(&addr->sib_addr, &sgid, sizeof(sgid));
cma_translate_ib(addr, &id_priv->id.route.addr.dev_addr);
return 0;
}
@@ -1409,9 +1411,16 @@ static bool cma_match_net_dev(const struct rdma_cm_id *id,
(addr->src_addr.ss_family == AF_IB ||
cma_protocol_roce_dev_port(id->device, port_num));
- return !addr->dev_addr.bound_dev_if ||
- (net_eq(dev_net(net_dev), addr->dev_addr.net) &&
- addr->dev_addr.bound_dev_if == net_dev->ifindex);
+ /*
+ * Net namespaces must match, and if the listner is listening
+ * on a specific netdevice than netdevice must match as well.
+ */
+ if (net_eq(dev_net(net_dev), addr->dev_addr.net) &&
+ (!!addr->dev_addr.bound_dev_if ==
+ (addr->dev_addr.bound_dev_if == net_dev->ifindex)))
+ return true;
+ else
+ return false;
}
static struct rdma_id_private *cma_find_listener(
diff --git a/drivers/infiniband/core/device.c b/drivers/infiniband/core/device.c
index 760ef60..15f4bdf 100644
--- a/drivers/infiniband/core/device.c
+++ b/drivers/infiniband/core/device.c
@@ -999,8 +999,7 @@ static int __init ib_core_init(void)
return -ENOMEM;
ib_comp_wq = alloc_workqueue("ib-comp-wq",
- WQ_UNBOUND | WQ_HIGHPRI | WQ_MEM_RECLAIM,
- WQ_UNBOUND_MAX_ACTIVE);
+ WQ_HIGHPRI | WQ_MEM_RECLAIM | WQ_SYSFS, 0);
if (!ib_comp_wq) {
ret = -ENOMEM;
goto err;
diff --git a/drivers/infiniband/core/mad.c b/drivers/infiniband/core/mad.c
index 2395fe2..3e2ab04 100644
--- a/drivers/infiniband/core/mad.c
+++ b/drivers/infiniband/core/mad.c
@@ -1549,7 +1549,8 @@ static int add_oui_reg_req(struct ib_mad_reg_req *mad_reg_req,
mad_reg_req->oui, 3)) {
method = &(*vendor_table)->vendor_class[
vclass]->method_table[i];
- BUG_ON(!*method);
+ if (!*method)
+ goto error3;
goto check_in_use;
}
}
@@ -1559,10 +1560,12 @@ static int add_oui_reg_req(struct ib_mad_reg_req *mad_reg_req,
vclass]->oui[i])) {
method = &(*vendor_table)->vendor_class[
vclass]->method_table[i];
- BUG_ON(*method);
/* Allocate method table for this OUI */
- if ((ret = allocate_method_table(method)))
- goto error3;
+ if (!*method) {
+ ret = allocate_method_table(method);
+ if (ret)
+ goto error3;
+ }
memcpy((*vendor_table)->vendor_class[vclass]->oui[i],
mad_reg_req->oui, 3);
goto check_in_use;
diff --git a/drivers/infiniband/core/multicast.c b/drivers/infiniband/core/multicast.c
index 322cb67..28d1845 100644
--- a/drivers/infiniband/core/multicast.c
+++ b/drivers/infiniband/core/multicast.c
@@ -724,21 +724,19 @@ int ib_init_ah_from_mcmember(struct ib_device *device, u8 port_num,
{
int ret;
u16 gid_index;
- u8 p;
- if (rdma_protocol_roce(device, port_num)) {
- ret = ib_find_cached_gid_by_port(device, &rec->port_gid,
- gid_type, port_num,
- ndev,
- &gid_index);
- } else if (rdma_protocol_ib(device, port_num)) {
- ret = ib_find_cached_gid(device, &rec->port_gid,
- IB_GID_TYPE_IB, NULL, &p,
+ /* GID table is not based on the netdevice for IB link layer,
+ * so ignore ndev during search.
+ */
+ if (rdma_protocol_ib(device, port_num))
+ ndev = NULL;
+ else if (!rdma_protocol_roce(device, port_num))
+ return -EINVAL;
+
+ ret = ib_find_cached_gid_by_port(device, &rec->port_gid,
+ gid_type, port_num,
+ ndev,
&gid_index);
- } else {
- ret = -EINVAL;
- }
-
if (ret)
return ret;
diff --git a/drivers/infiniband/core/rw.c b/drivers/infiniband/core/rw.c
index dbfd854..1d90a12 100644
--- a/drivers/infiniband/core/rw.c
+++ b/drivers/infiniband/core/rw.c
@@ -87,7 +87,7 @@ static int rdma_rw_init_one_mr(struct ib_qp *qp, u8 port_num,
}
ret = ib_map_mr_sg(reg->mr, sg, nents, &offset, PAGE_SIZE);
- if (ret < nents) {
+ if (ret < 0 || ret < nents) {
ib_mr_pool_put(qp, &qp->rdma_mrs, reg->mr);
return -EINVAL;
}
diff --git a/drivers/infiniband/core/sa_query.c b/drivers/infiniband/core/sa_query.c
index 81b742c..4baf3b8 100644
--- a/drivers/infiniband/core/sa_query.c
+++ b/drivers/infiniband/core/sa_query.c
@@ -1137,10 +1137,9 @@ int ib_init_ah_from_path(struct ib_device *device, u8 port_num,
resolved_dev = dev_get_by_index(dev_addr.net,
dev_addr.bound_dev_if);
- if (resolved_dev->flags & IFF_LOOPBACK) {
- dev_put(resolved_dev);
- resolved_dev = idev;
- dev_hold(resolved_dev);
+ if (!resolved_dev) {
+ dev_put(idev);
+ return -ENODEV;
}
ndev = ib_get_ndev_from_path(rec);
rcu_read_lock();
diff --git a/drivers/infiniband/core/sysfs.c b/drivers/infiniband/core/sysfs.c
index 42de5f2..a1240dd 100644
--- a/drivers/infiniband/core/sysfs.c
+++ b/drivers/infiniband/core/sysfs.c
@@ -485,7 +485,7 @@ static ssize_t show_pma_counter(struct ib_port *p, struct port_attribute *attr,
ret = get_perf_mad(p->ibdev, p->port_num, tab_attr->attr_id, &data,
40 + offset / 8, sizeof(data));
if (ret < 0)
- return sprintf(buf, "N/A (no PMA)\n");
+ return ret;
switch (width) {
case 4:
@@ -1008,10 +1008,12 @@ static int add_port(struct ib_device *device, int port_num,
goto err_put;
}
- p->pma_table = get_counter_table(device, port_num);
- ret = sysfs_create_group(&p->kobj, p->pma_table);
- if (ret)
- goto err_put_gid_attrs;
+ if (device->process_mad) {
+ p->pma_table = get_counter_table(device, port_num);
+ ret = sysfs_create_group(&p->kobj, p->pma_table);
+ if (ret)
+ goto err_put_gid_attrs;
+ }
p->gid_group.name = "gids";
p->gid_group.attrs = alloc_group_attrs(show_port_gid, attr.gid_tbl_len);
@@ -1124,7 +1126,8 @@ static int add_port(struct ib_device *device, int port_num,
p->gid_group.attrs = NULL;
err_remove_pma:
- sysfs_remove_group(&p->kobj, p->pma_table);
+ if (p->pma_table)
+ sysfs_remove_group(&p->kobj, p->pma_table);
err_put_gid_attrs:
kobject_put(&p->gid_attr_group->kobj);
@@ -1236,7 +1239,9 @@ static void free_port_list_attributes(struct ib_device *device)
kfree(port->hw_stats);
free_hsag(&port->kobj, port->hw_stats_ag);
}
- sysfs_remove_group(p, port->pma_table);
+
+ if (port->pma_table)
+ sysfs_remove_group(p, port->pma_table);
sysfs_remove_group(p, &port->pkey_group);
sysfs_remove_group(p, &port->gid_group);
sysfs_remove_group(&port->gid_attr_group->kobj,
diff --git a/drivers/infiniband/core/ucm.c b/drivers/infiniband/core/ucm.c
index 7713ef0..85be45e 100644
--- a/drivers/infiniband/core/ucm.c
+++ b/drivers/infiniband/core/ucm.c
@@ -46,6 +46,8 @@
#include <linux/mutex.h>
#include <linux/slab.h>
+#include <linux/nospec.h>
+
#include <asm/uaccess.h>
#include <rdma/ib.h>
@@ -1115,6 +1117,7 @@ static ssize_t ib_ucm_write(struct file *filp, const char __user *buf,
if (hdr.cmd >= ARRAY_SIZE(ucm_cmd_table))
return -EINVAL;
+ hdr.cmd = array_index_nospec(hdr.cmd, ARRAY_SIZE(ucm_cmd_table));
if (hdr.in + sizeof(hdr) > len)
return -EINVAL;
diff --git a/drivers/infiniband/core/ucma.c b/drivers/infiniband/core/ucma.c
index cb79d17..a4f4cd4 100644
--- a/drivers/infiniband/core/ucma.c
+++ b/drivers/infiniband/core/ucma.c
@@ -44,6 +44,8 @@
#include <linux/module.h>
#include <linux/nsproxy.h>
+#include <linux/nospec.h>
+
#include <rdma/rdma_user_cm.h>
#include <rdma/ib_marshall.h>
#include <rdma/rdma_cm.h>
@@ -124,6 +126,8 @@ static DEFINE_MUTEX(mut);
static DEFINE_IDR(ctx_idr);
static DEFINE_IDR(multicast_idr);
+static const struct file_operations ucma_fops;
+
static inline struct ucma_context *_ucma_find_context(int id,
struct ucma_file *file)
{
@@ -218,7 +222,7 @@ static struct ucma_multicast* ucma_alloc_multicast(struct ucma_context *ctx)
return NULL;
mutex_lock(&mut);
- mc->id = idr_alloc(&multicast_idr, mc, 0, 0, GFP_KERNEL);
+ mc->id = idr_alloc(&multicast_idr, NULL, 0, 0, GFP_KERNEL);
mutex_unlock(&mut);
if (mc->id < 0)
goto error;
@@ -676,7 +680,7 @@ static ssize_t ucma_resolve_ip(struct ucma_file *file,
if (copy_from_user(&cmd, inbuf, sizeof(cmd)))
return -EFAULT;
- if (!rdma_addr_size_in6(&cmd.src_addr) ||
+ if ((cmd.src_addr.sin6_family && !rdma_addr_size_in6(&cmd.src_addr)) ||
!rdma_addr_size_in6(&cmd.dst_addr))
return -EINVAL;
@@ -1296,7 +1300,7 @@ static ssize_t ucma_set_option(struct ucma_file *file, const char __user *inbuf,
if (IS_ERR(ctx))
return PTR_ERR(ctx);
- if (unlikely(cmd.optval > KMALLOC_MAX_SIZE))
+ if (unlikely(cmd.optlen > KMALLOC_MAX_SIZE))
return -EINVAL;
optval = memdup_user((void __user *) (unsigned long) cmd.optval,
@@ -1385,6 +1389,10 @@ static ssize_t ucma_process_join(struct ucma_file *file,
goto err3;
}
+ mutex_lock(&mut);
+ idr_replace(&multicast_idr, mc, mc->id);
+ mutex_unlock(&mut);
+
mutex_unlock(&file->mut);
ucma_put_ctx(ctx);
return 0;
@@ -1541,6 +1549,10 @@ static ssize_t ucma_migrate_id(struct ucma_file *new_file,
f = fdget(cmd.fd);
if (!f.file)
return -ENOENT;
+ if (f.file->f_op != &ucma_fops) {
+ ret = -EINVAL;
+ goto file_put;
+ }
/* Validate current fd and prevent destruction of id. */
ctx = ucma_get_ctx(f.file->private_data, cmd.id);
@@ -1627,6 +1639,7 @@ static ssize_t ucma_write(struct file *filp, const char __user *buf,
if (hdr.cmd >= ARRAY_SIZE(ucma_cmd_table))
return -EINVAL;
+ hdr.cmd = array_index_nospec(hdr.cmd, ARRAY_SIZE(ucma_cmd_table));
if (hdr.in + sizeof(hdr) > len)
return -EINVAL;
@@ -1710,6 +1723,8 @@ static int ucma_close(struct inode *inode, struct file *filp)
mutex_lock(&mut);
if (!ctx->closing) {
mutex_unlock(&mut);
+ ucma_put_ctx(ctx);
+ wait_for_completion(&ctx->comp);
/* rdma_destroy_id ensures that no event handlers are
* inflight for that id before releasing it.
*/
diff --git a/drivers/infiniband/core/umem.c b/drivers/infiniband/core/umem.c
index e74aa1d..99cebf3 100644
--- a/drivers/infiniband/core/umem.c
+++ b/drivers/infiniband/core/umem.c
@@ -122,16 +122,7 @@ struct ib_umem *ib_umem_get(struct ib_ucontext *context, unsigned long addr,
umem->address = addr;
umem->page_size = PAGE_SIZE;
umem->pid = get_task_pid(current, PIDTYPE_PID);
- /*
- * We ask for writable memory if any of the following
- * access flags are set. "Local write" and "remote write"
- * obviously require write access. "Remote atomic" can do
- * things like fetch and add, which will modify memory, and
- * "MW bind" can change permissions by binding a window.
- */
- umem->writable = !!(access &
- (IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_WRITE |
- IB_ACCESS_REMOTE_ATOMIC | IB_ACCESS_MW_BIND));
+ umem->writable = ib_access_writable(access);
if (access & IB_ACCESS_ON_DEMAND) {
put_pid(umem->pid);
diff --git a/drivers/infiniband/core/uverbs_cmd.c b/drivers/infiniband/core/uverbs_cmd.c
index 4b717cf..6f875bf 100644
--- a/drivers/infiniband/core/uverbs_cmd.c
+++ b/drivers/infiniband/core/uverbs_cmd.c
@@ -3725,6 +3725,11 @@ int ib_uverbs_ex_create_flow(struct ib_uverbs_file *file,
goto err_uobj;
}
+ if (qp->qp_type != IB_QPT_UD && qp->qp_type != IB_QPT_RAW_PACKET) {
+ err = -EINVAL;
+ goto err_put;
+ }
+
flow_attr = kzalloc(sizeof(*flow_attr) + cmd.flow_attr.num_of_specs *
sizeof(union ib_flow_spec), GFP_KERNEL);
if (!flow_attr) {
diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c
index 0e64b52..dc92dc4 100644
--- a/drivers/infiniband/core/verbs.c
+++ b/drivers/infiniband/core/verbs.c
@@ -1510,6 +1510,44 @@ EXPORT_SYMBOL(ib_dealloc_fmr);
/* Multicast groups */
+static bool is_valid_mcast_lid(struct ib_qp *qp, u16 lid)
+{
+ struct ib_qp_init_attr init_attr = {};
+ struct ib_qp_attr attr = {};
+ int num_eth_ports = 0;
+ int port;
+
+ /* If QP state >= init, it is assigned to a port and we can check this
+ * port only.
+ */
+ if (!ib_query_qp(qp, &attr, IB_QP_STATE | IB_QP_PORT, &init_attr)) {
+ if (attr.qp_state >= IB_QPS_INIT) {
+ if (rdma_port_get_link_layer(qp->device, attr.port_num) !=
+ IB_LINK_LAYER_INFINIBAND)
+ return true;
+ goto lid_check;
+ }
+ }
+
+ /* Can't get a quick answer, iterate over all ports */
+ for (port = 0; port < qp->device->phys_port_cnt; port++)
+ if (rdma_port_get_link_layer(qp->device, port) !=
+ IB_LINK_LAYER_INFINIBAND)
+ num_eth_ports++;
+
+ /* If we have at lease one Ethernet port, RoCE annex declares that
+ * multicast LID should be ignored. We can't tell at this step if the
+ * QP belongs to an IB or Ethernet port.
+ */
+ if (num_eth_ports)
+ return true;
+
+ /* If all the ports are IB, we can check according to IB spec. */
+lid_check:
+ return !(lid < be16_to_cpu(IB_MULTICAST_LID_BASE) ||
+ lid == be16_to_cpu(IB_LID_PERMISSIVE));
+}
+
int ib_attach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)
{
int ret;
@@ -1517,8 +1555,7 @@ int ib_attach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)
if (!qp->device->attach_mcast)
return -ENOSYS;
if (gid->raw[0] != 0xff || qp->qp_type != IB_QPT_UD ||
- lid < be16_to_cpu(IB_MULTICAST_LID_BASE) ||
- lid == be16_to_cpu(IB_LID_PERMISSIVE))
+ !is_valid_mcast_lid(qp, lid))
return -EINVAL;
ret = qp->device->attach_mcast(qp, gid, lid);
@@ -1535,8 +1572,7 @@ int ib_detach_mcast(struct ib_qp *qp, union ib_gid *gid, u16 lid)
if (!qp->device->detach_mcast)
return -ENOSYS;
if (gid->raw[0] != 0xff || qp->qp_type != IB_QPT_UD ||
- lid < be16_to_cpu(IB_MULTICAST_LID_BASE) ||
- lid == be16_to_cpu(IB_LID_PERMISSIVE))
+ !is_valid_mcast_lid(qp, lid))
return -EINVAL;
ret = qp->device->detach_mcast(qp, gid, lid);
diff --git a/drivers/infiniband/hw/cxgb4/device.c b/drivers/infiniband/hw/cxgb4/device.c
index b85a1a9..9e0f2cc 100644
--- a/drivers/infiniband/hw/cxgb4/device.c
+++ b/drivers/infiniband/hw/cxgb4/device.c
@@ -856,6 +856,11 @@ static int c4iw_rdev_open(struct c4iw_rdev *rdev)
rdev->status_page->db_off = 0;
+ init_completion(&rdev->rqt_compl);
+ init_completion(&rdev->pbl_compl);
+ kref_init(&rdev->rqt_kref);
+ kref_init(&rdev->pbl_kref);
+
return 0;
err_free_status_page:
free_page((unsigned long)rdev->status_page);
@@ -872,12 +877,14 @@ static int c4iw_rdev_open(struct c4iw_rdev *rdev)
static void c4iw_rdev_close(struct c4iw_rdev *rdev)
{
- destroy_workqueue(rdev->free_workq);
kfree(rdev->wr_log);
free_page((unsigned long)rdev->status_page);
c4iw_pblpool_destroy(rdev);
c4iw_rqtpool_destroy(rdev);
+ wait_for_completion(&rdev->pbl_compl);
+ wait_for_completion(&rdev->rqt_compl);
c4iw_destroy_resource(&rdev->resource);
+ destroy_workqueue(rdev->free_workq);
}
static void c4iw_dealloc(struct uld_ctx *ctx)
diff --git a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
index 7d54066..896dff7 100644
--- a/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
+++ b/drivers/infiniband/hw/cxgb4/iw_cxgb4.h
@@ -186,6 +186,10 @@ struct c4iw_rdev {
struct wr_log_entry *wr_log;
int wr_log_size;
struct workqueue_struct *free_workq;
+ struct completion rqt_compl;
+ struct completion pbl_compl;
+ struct kref rqt_kref;
+ struct kref pbl_kref;
};
static inline int c4iw_fatal_error(struct c4iw_rdev *rdev)
diff --git a/drivers/infiniband/hw/cxgb4/mem.c b/drivers/infiniband/hw/cxgb4/mem.c
index 410408f..0c21535 100644
--- a/drivers/infiniband/hw/cxgb4/mem.c
+++ b/drivers/infiniband/hw/cxgb4/mem.c
@@ -724,7 +724,7 @@ static int c4iw_set_page(struct ib_mr *ibmr, u64 addr)
{
struct c4iw_mr *mhp = to_c4iw_mr(ibmr);
- if (unlikely(mhp->mpl_len == mhp->max_mpl_len))
+ if (unlikely(mhp->mpl_len == mhp->attr.pbl_size))
return -ENOMEM;
mhp->mpl[mhp->mpl_len++] = addr;
diff --git a/drivers/infiniband/hw/cxgb4/qp.c b/drivers/infiniband/hw/cxgb4/qp.c
index cc2243f..bb45eb2 100644
--- a/drivers/infiniband/hw/cxgb4/qp.c
+++ b/drivers/infiniband/hw/cxgb4/qp.c
@@ -1258,6 +1258,12 @@ static void flush_qp(struct c4iw_qp *qhp)
t4_set_wq_in_error(&qhp->wq);
if (qhp->ibqp.uobject) {
+
+ /* for user qps, qhp->wq.flushed is protected by qhp->mutex */
+ if (qhp->wq.flushed)
+ return;
+
+ qhp->wq.flushed = 1;
t4_set_cq_in_error(&rchp->cq);
spin_lock_irqsave(&rchp->comp_handler_lock, flag);
(*rchp->ibcq.comp_handler)(&rchp->ibcq, rchp->ibcq.cq_context);
diff --git a/drivers/infiniband/hw/cxgb4/resource.c b/drivers/infiniband/hw/cxgb4/resource.c
index 67df71a..803c677 100644
--- a/drivers/infiniband/hw/cxgb4/resource.c
+++ b/drivers/infiniband/hw/cxgb4/resource.c
@@ -260,12 +260,22 @@ u32 c4iw_pblpool_alloc(struct c4iw_rdev *rdev, int size)
rdev->stats.pbl.cur += roundup(size, 1 << MIN_PBL_SHIFT);
if (rdev->stats.pbl.cur > rdev->stats.pbl.max)
rdev->stats.pbl.max = rdev->stats.pbl.cur;
+ kref_get(&rdev->pbl_kref);
} else
rdev->stats.pbl.fail++;
mutex_unlock(&rdev->stats.lock);
return (u32)addr;
}
+static void destroy_pblpool(struct kref *kref)
+{
+ struct c4iw_rdev *rdev;
+
+ rdev = container_of(kref, struct c4iw_rdev, pbl_kref);
+ gen_pool_destroy(rdev->pbl_pool);
+ complete(&rdev->pbl_compl);
+}
+
void c4iw_pblpool_free(struct c4iw_rdev *rdev, u32 addr, int size)
{
PDBG("%s addr 0x%x size %d\n", __func__, addr, size);
@@ -273,6 +283,7 @@ void c4iw_pblpool_free(struct c4iw_rdev *rdev, u32 addr, int size)
rdev->stats.pbl.cur -= roundup(size, 1 << MIN_PBL_SHIFT);
mutex_unlock(&rdev->stats.lock);
gen_pool_free(rdev->pbl_pool, (unsigned long)addr, size);
+ kref_put(&rdev->pbl_kref, destroy_pblpool);
}
int c4iw_pblpool_create(struct c4iw_rdev *rdev)
@@ -312,7 +323,7 @@ int c4iw_pblpool_create(struct c4iw_rdev *rdev)
void c4iw_pblpool_destroy(struct c4iw_rdev *rdev)
{
- gen_pool_destroy(rdev->pbl_pool);
+ kref_put(&rdev->pbl_kref, destroy_pblpool);
}
/*
@@ -333,12 +344,22 @@ u32 c4iw_rqtpool_alloc(struct c4iw_rdev *rdev, int size)
rdev->stats.rqt.cur += roundup(size << 6, 1 << MIN_RQT_SHIFT);
if (rdev->stats.rqt.cur > rdev->stats.rqt.max)
rdev->stats.rqt.max = rdev->stats.rqt.cur;
+ kref_get(&rdev->rqt_kref);
} else
rdev->stats.rqt.fail++;
mutex_unlock(&rdev->stats.lock);
return (u32)addr;
}
+static void destroy_rqtpool(struct kref *kref)
+{
+ struct c4iw_rdev *rdev;
+
+ rdev = container_of(kref, struct c4iw_rdev, rqt_kref);
+ gen_pool_destroy(rdev->rqt_pool);
+ complete(&rdev->rqt_compl);
+}
+
void c4iw_rqtpool_free(struct c4iw_rdev *rdev, u32 addr, int size)
{
PDBG("%s addr 0x%x size %d\n", __func__, addr, size << 6);
@@ -346,6 +367,7 @@ void c4iw_rqtpool_free(struct c4iw_rdev *rdev, u32 addr, int size)
rdev->stats.rqt.cur -= roundup(size << 6, 1 << MIN_RQT_SHIFT);
mutex_unlock(&rdev->stats.lock);
gen_pool_free(rdev->rqt_pool, (unsigned long)addr, size << 6);
+ kref_put(&rdev->rqt_kref, destroy_rqtpool);
}
int c4iw_rqtpool_create(struct c4iw_rdev *rdev)
@@ -383,7 +405,7 @@ int c4iw_rqtpool_create(struct c4iw_rdev *rdev)
void c4iw_rqtpool_destroy(struct c4iw_rdev *rdev)
{
- gen_pool_destroy(rdev->rqt_pool);
+ kref_put(&rdev->rqt_kref, destroy_rqtpool);
}
/*
diff --git a/drivers/infiniband/hw/hfi1/chip.c b/drivers/infiniband/hw/hfi1/chip.c
index 7853b0c..d30b3b9 100644
--- a/drivers/infiniband/hw/hfi1/chip.c
+++ b/drivers/infiniband/hw/hfi1/chip.c
@@ -5860,6 +5860,7 @@ static void is_sendctxt_err_int(struct hfi1_devdata *dd,
u64 status;
u32 sw_index;
int i = 0;
+ unsigned long irq_flags;
sw_index = dd->hw_to_sw[hw_context];
if (sw_index >= dd->num_send_contexts) {
@@ -5869,10 +5870,12 @@ static void is_sendctxt_err_int(struct hfi1_devdata *dd,
return;
}
sci = &dd->send_contexts[sw_index];
+ spin_lock_irqsave(&dd->sc_lock, irq_flags);
sc = sci->sc;
if (!sc) {
dd_dev_err(dd, "%s: context %u(%u): no sc?\n", __func__,
sw_index, hw_context);
+ spin_unlock_irqrestore(&dd->sc_lock, irq_flags);
return;
}
@@ -5894,6 +5897,7 @@ static void is_sendctxt_err_int(struct hfi1_devdata *dd,
*/
if (sc->type != SC_USER)
queue_work(dd->pport->hfi1_wq, &sc->halt_work);
+ spin_unlock_irqrestore(&dd->sc_lock, irq_flags);
/*
* Update the counters for the corresponding status bits.
@@ -6713,7 +6717,7 @@ static void rxe_kernel_unfreeze(struct hfi1_devdata *dd)
for (i = 0; i < dd->n_krcv_queues; i++) {
rcvmask = HFI1_RCVCTRL_CTXT_ENB;
/* HFI1_RCVCTRL_TAILUPD_[ENB|DIS] needs to be set explicitly */
- rcvmask |= HFI1_CAP_KGET_MASK(dd->rcd[i]->flags, DMA_RTAIL) ?
+ rcvmask |= dd->rcd[i]->rcvhdrtail_kvaddr ?
HFI1_RCVCTRL_TAILUPD_ENB : HFI1_RCVCTRL_TAILUPD_DIS;
hfi1_rcvctrl(dd, rcvmask, i);
}
@@ -8207,7 +8211,7 @@ static inline int check_packet_present(struct hfi1_ctxtdata *rcd)
u32 tail;
int present;
- if (!HFI1_CAP_IS_KSET(DMA_RTAIL))
+ if (!rcd->rcvhdrtail_kvaddr)
present = (rcd->seq_cnt ==
rhf_rcv_seq(rhf_to_cpu(get_rhf_addr(rcd))));
else /* is RDMA rtail */
@@ -11546,7 +11550,7 @@ void hfi1_rcvctrl(struct hfi1_devdata *dd, unsigned int op, int ctxt)
/* reset the tail and hdr addresses, and sequence count */
write_kctxt_csr(dd, ctxt, RCV_HDR_ADDR,
rcd->rcvhdrq_dma);
- if (HFI1_CAP_KGET_MASK(rcd->flags, DMA_RTAIL))
+ if (rcd->rcvhdrtail_kvaddr)
write_kctxt_csr(dd, ctxt, RCV_HDR_TAIL_ADDR,
rcd->rcvhdrqtailaddr_dma);
rcd->seq_cnt = 1;
@@ -11626,7 +11630,7 @@ void hfi1_rcvctrl(struct hfi1_devdata *dd, unsigned int op, int ctxt)
rcvctrl |= RCV_CTXT_CTRL_INTR_AVAIL_SMASK;
if (op & HFI1_RCVCTRL_INTRAVAIL_DIS)
rcvctrl &= ~RCV_CTXT_CTRL_INTR_AVAIL_SMASK;
- if (op & HFI1_RCVCTRL_TAILUPD_ENB && rcd->rcvhdrqtailaddr_dma)
+ if ((op & HFI1_RCVCTRL_TAILUPD_ENB) && rcd->rcvhdrtail_kvaddr)
rcvctrl |= RCV_CTXT_CTRL_TAIL_UPD_SMASK;
if (op & HFI1_RCVCTRL_TAILUPD_DIS) {
/* See comment on RcvCtxtCtrl.TailUpd above */
diff --git a/drivers/infiniband/hw/hfi1/file_ops.c b/drivers/infiniband/hw/hfi1/file_ops.c
index bb72976..d612f9d 100644
--- a/drivers/infiniband/hw/hfi1/file_ops.c
+++ b/drivers/infiniband/hw/hfi1/file_ops.c
@@ -609,7 +609,7 @@ static int hfi1_file_mmap(struct file *fp, struct vm_area_struct *vma)
ret = -EINVAL;
goto done;
}
- if (flags & VM_WRITE) {
+ if ((flags & VM_WRITE) || !uctxt->rcvhdrtail_kvaddr) {
ret = -EPERM;
goto done;
}
diff --git a/drivers/infiniband/hw/hfi1/hfi.h b/drivers/infiniband/hw/hfi1/hfi.h
index a3279f3d..a79d9b3 100644
--- a/drivers/infiniband/hw/hfi1/hfi.h
+++ b/drivers/infiniband/hw/hfi1/hfi.h
@@ -1631,6 +1631,7 @@ struct cc_state *get_cc_state_protected(struct hfi1_pportdata *ppd)
#define HFI1_HAS_SDMA_TIMEOUT 0x8
#define HFI1_HAS_SEND_DMA 0x10 /* Supports Send DMA */
#define HFI1_FORCED_FREEZE 0x80 /* driver forced freeze mode */
+#define HFI1_SHUTDOWN 0x100 /* device is shutting down */
/* IB dword length mask in PBC (lower 11 bits); same for all chips */
#define HFI1_PBC_LENGTH_MASK ((1 << 11) - 1)
diff --git a/drivers/infiniband/hw/hfi1/init.c b/drivers/infiniband/hw/hfi1/init.c
index 84a97f3..9dc8cf09 100644
--- a/drivers/infiniband/hw/hfi1/init.c
+++ b/drivers/infiniband/hw/hfi1/init.c
@@ -857,6 +857,10 @@ static void shutdown_device(struct hfi1_devdata *dd)
unsigned pidx;
int i;
+ if (dd->flags & HFI1_SHUTDOWN)
+ return;
+ dd->flags |= HFI1_SHUTDOWN;
+
for (pidx = 0; pidx < dd->num_pports; ++pidx) {
ppd = dd->pport + pidx;
@@ -1049,6 +1053,8 @@ struct hfi1_devdata *hfi1_alloc_devdata(struct pci_dev *pdev, size_t extra)
return ERR_PTR(-ENOMEM);
dd->num_pports = nports;
dd->pport = (struct hfi1_pportdata *)(dd + 1);
+ dd->pcidev = pdev;
+ pci_set_drvdata(pdev, dd);
INIT_LIST_HEAD(&dd->list);
idr_preload(GFP_KERNEL);
@@ -1166,6 +1172,7 @@ void hfi1_disable_after_error(struct hfi1_devdata *dd)
static void remove_one(struct pci_dev *);
static int init_one(struct pci_dev *, const struct pci_device_id *);
+static void shutdown_one(struct pci_dev *);
#define DRIVER_LOAD_MSG "Intel " DRIVER_NAME " loaded: "
#define PFX DRIVER_NAME ": "
@@ -1182,6 +1189,7 @@ static struct pci_driver hfi1_pci_driver = {
.name = DRIVER_NAME,
.probe = init_one,
.remove = remove_one,
+ .shutdown = shutdown_one,
.id_table = hfi1_pci_tbl,
.err_handler = &hfi1_pci_err_handler,
};
@@ -1588,6 +1596,13 @@ static void remove_one(struct pci_dev *pdev)
postinit_cleanup(dd);
}
+static void shutdown_one(struct pci_dev *pdev)
+{
+ struct hfi1_devdata *dd = pci_get_drvdata(pdev);
+
+ shutdown_device(dd);
+}
+
/**
* hfi1_create_rcvhdrq - create a receive header queue
* @dd: the hfi1_ib device
@@ -1603,7 +1618,6 @@ int hfi1_create_rcvhdrq(struct hfi1_devdata *dd, struct hfi1_ctxtdata *rcd)
u64 reg;
if (!rcd->rcvhdrq) {
- dma_addr_t dma_hdrqtail;
gfp_t gfp_flags;
/*
@@ -1626,13 +1640,13 @@ int hfi1_create_rcvhdrq(struct hfi1_devdata *dd, struct hfi1_ctxtdata *rcd)
goto bail;
}
- if (HFI1_CAP_KGET_MASK(rcd->flags, DMA_RTAIL)) {
+ if (HFI1_CAP_KGET_MASK(rcd->flags, DMA_RTAIL) ||
+ HFI1_CAP_UGET_MASK(rcd->flags, DMA_RTAIL)) {
rcd->rcvhdrtail_kvaddr = dma_zalloc_coherent(
- &dd->pcidev->dev, PAGE_SIZE, &dma_hdrqtail,
- gfp_flags);
+ &dd->pcidev->dev, PAGE_SIZE,
+ &rcd->rcvhdrqtailaddr_dma, gfp_flags);
if (!rcd->rcvhdrtail_kvaddr)
goto bail_free;
- rcd->rcvhdrqtailaddr_dma = dma_hdrqtail;
}
rcd->rcvhdrq_size = amt;
diff --git a/drivers/infiniband/hw/hfi1/pcie.c b/drivers/infiniband/hw/hfi1/pcie.c
index 335613a1..7176260 100644
--- a/drivers/infiniband/hw/hfi1/pcie.c
+++ b/drivers/infiniband/hw/hfi1/pcie.c
@@ -162,9 +162,6 @@ int hfi1_pcie_ddinit(struct hfi1_devdata *dd, struct pci_dev *pdev)
unsigned long len;
resource_size_t addr;
- dd->pcidev = pdev;
- pci_set_drvdata(pdev, dd);
-
addr = pci_resource_start(pdev, 0);
len = pci_resource_len(pdev, 0);
diff --git a/drivers/infiniband/hw/hfi1/pio.c b/drivers/infiniband/hw/hfi1/pio.c
index d89b874..c2982bb 100644
--- a/drivers/infiniband/hw/hfi1/pio.c
+++ b/drivers/infiniband/hw/hfi1/pio.c
@@ -88,6 +88,7 @@ void pio_send_control(struct hfi1_devdata *dd, int op)
unsigned long flags;
int write = 1; /* write sendctrl back */
int flush = 0; /* re-read sendctrl to make sure it is flushed */
+ int i;
spin_lock_irqsave(&dd->sendctrl_lock, flags);
@@ -97,9 +98,13 @@ void pio_send_control(struct hfi1_devdata *dd, int op)
reg |= SEND_CTRL_SEND_ENABLE_SMASK;
/* Fall through */
case PSC_DATA_VL_ENABLE:
+ mask = 0;
+ for (i = 0; i < ARRAY_SIZE(dd->vld); i++)
+ if (!dd->vld[i].mtu)
+ mask |= BIT_ULL(i);
/* Disallow sending on VLs not enabled */
- mask = (((~0ull) << num_vls) & SEND_CTRL_UNSUPPORTED_VL_MASK) <<
- SEND_CTRL_UNSUPPORTED_VL_SHIFT;
+ mask = (mask & SEND_CTRL_UNSUPPORTED_VL_MASK) <<
+ SEND_CTRL_UNSUPPORTED_VL_SHIFT;
reg = (reg & ~SEND_CTRL_UNSUPPORTED_VL_SMASK) | mask;
break;
case PSC_GLOBAL_DISABLE:
diff --git a/drivers/infiniband/hw/hfi1/rc.c b/drivers/infiniband/hw/hfi1/rc.c
index 613074e9..e8e0fa5 100644
--- a/drivers/infiniband/hw/hfi1/rc.c
+++ b/drivers/infiniband/hw/hfi1/rc.c
@@ -397,7 +397,7 @@ int hfi1_make_rc_req(struct rvt_qp *qp, struct hfi1_pkt_state *ps)
lockdep_assert_held(&qp->s_lock);
ps->s_txreq = get_txreq(ps->dev, qp);
- if (IS_ERR(ps->s_txreq))
+ if (!ps->s_txreq)
goto bail_no_tx;
ohdr = &ps->s_txreq->phdr.hdr.u.oth;
diff --git a/drivers/infiniband/hw/hfi1/uc.c b/drivers/infiniband/hw/hfi1/uc.c
index 5e6d1ba..de21128 100644
--- a/drivers/infiniband/hw/hfi1/uc.c
+++ b/drivers/infiniband/hw/hfi1/uc.c
@@ -1,5 +1,5 @@
/*
- * Copyright(c) 2015, 2016 Intel Corporation.
+ * Copyright(c) 2015 - 2018 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
@@ -72,7 +72,7 @@ int hfi1_make_uc_req(struct rvt_qp *qp, struct hfi1_pkt_state *ps)
int middle = 0;
ps->s_txreq = get_txreq(ps->dev, qp);
- if (IS_ERR(ps->s_txreq))
+ if (!ps->s_txreq)
goto bail_no_tx;
if (!(ib_rvt_state_ops[qp->state] & RVT_PROCESS_SEND_OK)) {
diff --git a/drivers/infiniband/hw/hfi1/ud.c b/drivers/infiniband/hw/hfi1/ud.c
index 97ae24b..1a7ce1d 100644
--- a/drivers/infiniband/hw/hfi1/ud.c
+++ b/drivers/infiniband/hw/hfi1/ud.c
@@ -1,5 +1,5 @@
/*
- * Copyright(c) 2015, 2016 Intel Corporation.
+ * Copyright(c) 2015 - 2018 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
@@ -285,7 +285,7 @@ int hfi1_make_ud_req(struct rvt_qp *qp, struct hfi1_pkt_state *ps)
u8 sc5;
ps->s_txreq = get_txreq(ps->dev, qp);
- if (IS_ERR(ps->s_txreq))
+ if (!ps->s_txreq)
goto bail_no_tx;
if (!(ib_rvt_state_ops[qp->state] & RVT_PROCESS_NEXT_SEND_OK)) {
diff --git a/drivers/infiniband/hw/hfi1/user_sdma.c b/drivers/infiniband/hw/hfi1/user_sdma.c
index 77697d6..4c11116 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.c
+++ b/drivers/infiniband/hw/hfi1/user_sdma.c
@@ -148,15 +148,8 @@ MODULE_PARM_DESC(sdma_comp_size, "Size of User SDMA completion ring. Default: 12
#define TXREQ_FLAGS_REQ_LAST_PKT BIT(0)
/* SDMA request flag bits */
-#define SDMA_REQ_FOR_THREAD 1
-#define SDMA_REQ_SEND_DONE 2
-#define SDMA_REQ_HAVE_AHG 3
-#define SDMA_REQ_HAS_ERROR 4
-#define SDMA_REQ_DONE_ERROR 5
-
-#define SDMA_PKT_Q_INACTIVE BIT(0)
-#define SDMA_PKT_Q_ACTIVE BIT(1)
-#define SDMA_PKT_Q_DEFERRED BIT(2)
+#define SDMA_REQ_HAVE_AHG 1
+#define SDMA_REQ_HAS_ERROR 2
/*
* Maximum retry attempts to submit a TX request
@@ -252,8 +245,6 @@ struct user_sdma_request {
u64 seqsubmitted;
struct list_head txps;
unsigned long flags;
- /* status of the last txreq completed */
- int status;
};
/*
@@ -413,7 +404,6 @@ int hfi1_user_sdma_alloc_queues(struct hfi1_ctxtdata *uctxt, struct file *fp)
pq->ctxt = uctxt->ctxt;
pq->subctxt = fd->subctxt;
pq->n_max_reqs = hfi1_sdma_comp_ring_size;
- pq->state = SDMA_PKT_Q_INACTIVE;
atomic_set(&pq->n_reqs, 0);
init_waitqueue_head(&pq->wait);
atomic_set(&pq->n_locked, 0);
@@ -496,7 +486,7 @@ int hfi1_user_sdma_free_queues(struct hfi1_filedata *fd)
/* Wait until all requests have been freed. */
wait_event_interruptible(
pq->wait,
- (ACCESS_ONCE(pq->state) == SDMA_PKT_Q_INACTIVE));
+ !atomic_read(&pq->n_reqs));
kfree(pq->reqs);
kfree(pq->req_in_use);
kmem_cache_destroy(pq->txreq_cache);
@@ -532,6 +522,13 @@ static u8 dlid_to_selector(u16 dlid)
return mapping[hash];
}
+/**
+ * hfi1_user_sdma_process_request() - Process and start a user sdma request
+ * @fp: valid file pointer
+ * @iovec: array of io vectors to process
+ * @dim: overall iovec array size
+ * @count: number of io vector array entries processed
+ */
int hfi1_user_sdma_process_request(struct file *fp, struct iovec *iovec,
unsigned long dim, unsigned long *count)
{
@@ -546,7 +543,6 @@ int hfi1_user_sdma_process_request(struct file *fp, struct iovec *iovec,
struct sdma_req_info info;
struct user_sdma_request *req;
u8 opcode, sc, vl;
- int req_queued = 0;
u16 dlid;
u32 selector;
@@ -611,11 +607,13 @@ int hfi1_user_sdma_process_request(struct file *fp, struct iovec *iovec,
req->data_iovs = req_iovcnt(info.ctrl) - 1; /* subtract header vector */
req->pq = pq;
req->cq = cq;
- req->status = -1;
INIT_LIST_HEAD(&req->txps);
memcpy(&req->info, &info, sizeof(info));
+ /* The request is initialized, count it */
+ atomic_inc(&pq->n_reqs);
+
if (req_opcode(info.ctrl) == EXPECTED) {
/* expected must have a TID info and at least one data vector */
if (req->data_iovs < 2) {
@@ -704,7 +702,7 @@ int hfi1_user_sdma_process_request(struct file *fp, struct iovec *iovec,
memcpy(&req->iovs[i].iov, iovec + idx++, sizeof(struct iovec));
ret = pin_vector_pages(req, &req->iovs[i]);
if (ret) {
- req->status = ret;
+ req->data_iovs = i;
goto free_req;
}
req->data_len += req->iovs[i].iov.iov_len;
@@ -772,23 +770,11 @@ int hfi1_user_sdma_process_request(struct file *fp, struct iovec *iovec,
}
set_comp_state(pq, cq, info.comp_idx, QUEUED, 0);
- atomic_inc(&pq->n_reqs);
- req_queued = 1;
+ pq->state = SDMA_PKT_Q_ACTIVE;
/* Send the first N packets in the request to buy us some time */
ret = user_sdma_send_pkts(req, pcount);
- if (unlikely(ret < 0 && ret != -EBUSY)) {
- req->status = ret;
+ if (unlikely(ret < 0 && ret != -EBUSY))
goto free_req;
- }
-
- /*
- * It is possible that the SDMA engine would have processed all the
- * submitted packets by the time we get here. Therefore, only set
- * packet queue state to ACTIVE if there are still uncompleted
- * requests.
- */
- if (atomic_read(&pq->n_reqs))
- xchg(&pq->state, SDMA_PKT_Q_ACTIVE);
/*
* This is a somewhat blocking send implementation.
@@ -796,17 +782,11 @@ int hfi1_user_sdma_process_request(struct file *fp, struct iovec *iovec,
* request have been submitted to the SDMA engine. However, it
* will not wait for send completions.
*/
- while (!test_bit(SDMA_REQ_SEND_DONE, &req->flags)) {
+ while (req->seqsubmitted != req->info.npkts) {
ret = user_sdma_send_pkts(req, pcount);
if (ret < 0) {
- if (ret != -EBUSY) {
- req->status = ret;
- set_bit(SDMA_REQ_DONE_ERROR, &req->flags);
- if (ACCESS_ONCE(req->seqcomp) ==
- req->seqsubmitted - 1)
- goto free_req;
- return ret;
- }
+ if (ret != -EBUSY)
+ goto free_req;
wait_event_interruptible_timeout(
pq->busy.wait_dma,
(pq->state == SDMA_PKT_Q_ACTIVE),
@@ -817,10 +797,19 @@ int hfi1_user_sdma_process_request(struct file *fp, struct iovec *iovec,
*count += idx;
return 0;
free_req:
- user_sdma_free_request(req, true);
- if (req_queued)
+ /*
+ * If the submitted seqsubmitted == npkts, the completion routine
+ * controls the final state. If sequbmitted < npkts, wait for any
+ * outstanding packets to finish before cleaning up.
+ */
+ if (req->seqsubmitted < req->info.npkts) {
+ if (req->seqsubmitted)
+ wait_event(pq->busy.wait_dma,
+ (req->seqcomp == req->seqsubmitted - 1));
+ user_sdma_free_request(req, true);
pq_update(pq);
- set_comp_state(pq, cq, info.comp_idx, ERROR, req->status);
+ set_comp_state(pq, cq, info.comp_idx, ERROR, ret);
+ }
return ret;
}
@@ -903,10 +892,8 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
pq = req->pq;
/* If tx completion has reported an error, we are done. */
- if (test_bit(SDMA_REQ_HAS_ERROR, &req->flags)) {
- set_bit(SDMA_REQ_DONE_ERROR, &req->flags);
+ if (test_bit(SDMA_REQ_HAS_ERROR, &req->flags))
return -EFAULT;
- }
/*
* Check if we might have sent the entire request already
@@ -929,10 +916,8 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
* with errors. If so, we are not going to process any
* more packets from this request.
*/
- if (test_bit(SDMA_REQ_HAS_ERROR, &req->flags)) {
- set_bit(SDMA_REQ_DONE_ERROR, &req->flags);
+ if (test_bit(SDMA_REQ_HAS_ERROR, &req->flags))
return -EFAULT;
- }
tx = kmem_cache_alloc(pq->txreq_cache, GFP_KERNEL);
if (!tx)
@@ -956,7 +941,7 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
if (ACCESS_ONCE(iovec->offset) == iovec->iov.iov_len) {
if (++req->iov_idx == req->data_iovs) {
ret = -EFAULT;
- goto free_txreq;
+ goto free_tx;
}
iovec = &req->iovs[req->iov_idx];
WARN_ON(iovec->offset);
@@ -1090,7 +1075,6 @@ static int user_sdma_send_pkts(struct user_sdma_request *req, unsigned maxpkts)
ret = sdma_send_txlist(req->sde, &pq->busy, &req->txps, &count);
req->seqsubmitted += count;
if (req->seqsubmitted == req->info.npkts) {
- set_bit(SDMA_REQ_SEND_DONE, &req->flags);
/*
* The txreq has already been submitted to the HW queue
* so we can free the AHG entry now. Corruption will not
@@ -1489,11 +1473,15 @@ static int set_txreq_header_ahg(struct user_sdma_request *req,
return diff;
}
-/*
- * SDMA tx request completion callback. Called when the SDMA progress
- * state machine gets notification that the SDMA descriptors for this
- * tx request have been processed by the DMA engine. Called in
- * interrupt context.
+/**
+ * user_sdma_txreq_cb() - SDMA tx request completion callback.
+ * @txreq: valid sdma tx request
+ * @status: success/failure of request
+ *
+ * Called when the SDMA progress state machine gets notification that
+ * the SDMA descriptors for this tx request have been processed by the
+ * DMA engine. Called in interrupt context.
+ * Only do work on completed sequences.
*/
static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status)
{
@@ -1502,7 +1490,7 @@ static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status)
struct user_sdma_request *req;
struct hfi1_user_sdma_pkt_q *pq;
struct hfi1_user_sdma_comp_q *cq;
- u16 idx;
+ enum hfi1_sdma_comp_state state = COMPLETE;
if (!tx->req)
return;
@@ -1515,39 +1503,25 @@ static void user_sdma_txreq_cb(struct sdma_txreq *txreq, int status)
SDMA_DBG(req, "SDMA completion with error %d",
status);
set_bit(SDMA_REQ_HAS_ERROR, &req->flags);
+ state = ERROR;
}
req->seqcomp = tx->seqnum;
kmem_cache_free(pq->txreq_cache, tx);
- tx = NULL;
- idx = req->info.comp_idx;
- if (req->status == -1 && status == SDMA_TXREQ_S_OK) {
- if (req->seqcomp == req->info.npkts - 1) {
- req->status = 0;
- user_sdma_free_request(req, false);
- pq_update(pq);
- set_comp_state(pq, cq, idx, COMPLETE, 0);
- }
- } else {
- if (status != SDMA_TXREQ_S_OK)
- req->status = status;
- if (req->seqcomp == (ACCESS_ONCE(req->seqsubmitted) - 1) &&
- (test_bit(SDMA_REQ_SEND_DONE, &req->flags) ||
- test_bit(SDMA_REQ_DONE_ERROR, &req->flags))) {
- user_sdma_free_request(req, false);
- pq_update(pq);
- set_comp_state(pq, cq, idx, ERROR, req->status);
- }
- }
+ /* sequence isn't complete? We are done */
+ if (req->seqcomp != req->info.npkts - 1)
+ return;
+
+ user_sdma_free_request(req, false);
+ set_comp_state(pq, cq, req->info.comp_idx, state, status);
+ pq_update(pq);
}
static inline void pq_update(struct hfi1_user_sdma_pkt_q *pq)
{
- if (atomic_dec_and_test(&pq->n_reqs)) {
- xchg(&pq->state, SDMA_PKT_Q_INACTIVE);
+ if (atomic_dec_and_test(&pq->n_reqs))
wake_up(&pq->wait);
- }
}
static void user_sdma_free_request(struct user_sdma_request *req, bool unpin)
@@ -1572,6 +1546,8 @@ static void user_sdma_free_request(struct user_sdma_request *req, bool unpin)
if (!node)
continue;
+ req->iovs[i].node = NULL;
+
if (unpin)
hfi1_mmu_rb_remove(req->pq->handler,
&node->rb);
diff --git a/drivers/infiniband/hw/hfi1/user_sdma.h b/drivers/infiniband/hw/hfi1/user_sdma.h
index 3900171..09dd843 100644
--- a/drivers/infiniband/hw/hfi1/user_sdma.h
+++ b/drivers/infiniband/hw/hfi1/user_sdma.h
@@ -53,6 +53,11 @@
extern uint extended_psn;
+enum pkt_q_sdma_state {
+ SDMA_PKT_Q_ACTIVE,
+ SDMA_PKT_Q_DEFERRED,
+};
+
struct hfi1_user_sdma_pkt_q {
struct list_head list;
unsigned ctxt;
@@ -65,7 +70,7 @@ struct hfi1_user_sdma_pkt_q {
struct user_sdma_request *reqs;
unsigned long *req_in_use;
struct iowait busy;
- unsigned state;
+ enum pkt_q_sdma_state state;
wait_queue_head_t wait;
unsigned long unpinned;
struct mmu_rb_handler *handler;
diff --git a/drivers/infiniband/hw/hfi1/verbs.c b/drivers/infiniband/hw/hfi1/verbs.c
index 01a380e..d9c7175 100644
--- a/drivers/infiniband/hw/hfi1/verbs.c
+++ b/drivers/infiniband/hw/hfi1/verbs.c
@@ -1088,6 +1088,8 @@ int hfi1_verbs_send_pio(struct rvt_qp *qp, struct hfi1_pkt_state *ps,
if (slen > len)
slen = len;
+ if (slen > ss->sge.sge_length)
+ slen = ss->sge.sge_length;
update_sge(ss, slen);
seg_pio_copy_mid(pbuf, addr, slen);
len -= slen;
@@ -1511,12 +1513,18 @@ static int hfi1_check_ah(struct ib_device *ibdev, struct ib_ah_attr *ah_attr)
struct hfi1_pportdata *ppd;
struct hfi1_devdata *dd;
u8 sc5;
+ u8 sl;
/* test the mapping for validity */
ibp = to_iport(ibdev, ah_attr->port_num);
ppd = ppd_from_ibp(ibp);
- sc5 = ibp->sl_to_sc[ah_attr->sl];
dd = dd_from_ppd(ppd);
+
+ sl = ah_attr->sl;
+ if (sl >= ARRAY_SIZE(ibp->sl_to_sc))
+ return -EINVAL;
+
+ sc5 = ibp->sl_to_sc[sl];
if (sc_to_vlt(dd, sc5) > num_vls && sc_to_vlt(dd, sc5) != 0xf)
return -EINVAL;
return 0;
diff --git a/drivers/infiniband/hw/hfi1/verbs_txreq.c b/drivers/infiniband/hw/hfi1/verbs_txreq.c
index 094ab82..d8a5bad 100644
--- a/drivers/infiniband/hw/hfi1/verbs_txreq.c
+++ b/drivers/infiniband/hw/hfi1/verbs_txreq.c
@@ -1,5 +1,5 @@
/*
- * Copyright(c) 2016 Intel Corporation.
+ * Copyright(c) 2016 - 2018 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
@@ -94,7 +94,7 @@ struct verbs_txreq *__get_txreq(struct hfi1_ibdev *dev,
struct rvt_qp *qp)
__must_hold(&qp->s_lock)
{
- struct verbs_txreq *tx = ERR_PTR(-EBUSY);
+ struct verbs_txreq *tx = NULL;
write_seqlock(&dev->iowait_lock);
if (ib_rvt_state_ops[qp->state] & RVT_PROCESS_RECV_OK) {
diff --git a/drivers/infiniband/hw/hfi1/verbs_txreq.h b/drivers/infiniband/hw/hfi1/verbs_txreq.h
index 5660897..31ded57 100644
--- a/drivers/infiniband/hw/hfi1/verbs_txreq.h
+++ b/drivers/infiniband/hw/hfi1/verbs_txreq.h
@@ -1,5 +1,5 @@
/*
- * Copyright(c) 2016 Intel Corporation.
+ * Copyright(c) 2016 - 2018 Intel Corporation.
*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
@@ -82,7 +82,7 @@ static inline struct verbs_txreq *get_txreq(struct hfi1_ibdev *dev,
if (unlikely(!tx)) {
/* call slow path to get the lock */
tx = __get_txreq(dev, qp);
- if (IS_ERR(tx))
+ if (!tx)
return tx;
}
tx->qp = qp;
diff --git a/drivers/infiniband/hw/hns/hns_roce_pd.c b/drivers/infiniband/hw/hns/hns_roce_pd.c
index 05db7d5..da61ce8 100644
--- a/drivers/infiniband/hw/hns/hns_roce_pd.c
+++ b/drivers/infiniband/hw/hns/hns_roce_pd.c
@@ -35,7 +35,7 @@
static int hns_roce_pd_alloc(struct hns_roce_dev *hr_dev, unsigned long *pdn)
{
- return hns_roce_bitmap_alloc(&hr_dev->pd_bitmap, pdn);
+ return hns_roce_bitmap_alloc(&hr_dev->pd_bitmap, pdn) ? -ENOMEM : 0;
}
static void hns_roce_pd_free(struct hns_roce_dev *hr_dev, unsigned long pdn)
diff --git a/drivers/infiniband/hw/hns/hns_roce_qp.c b/drivers/infiniband/hw/hns/hns_roce_qp.c
index e86dd8d..33cf103 100644
--- a/drivers/infiniband/hw/hns/hns_roce_qp.c
+++ b/drivers/infiniband/hw/hns/hns_roce_qp.c
@@ -114,7 +114,10 @@ static int hns_roce_reserve_range_qp(struct hns_roce_dev *hr_dev, int cnt,
{
struct hns_roce_qp_table *qp_table = &hr_dev->qp_table;
- return hns_roce_bitmap_alloc_range(&qp_table->bitmap, cnt, align, base);
+ return hns_roce_bitmap_alloc_range(&qp_table->bitmap, cnt, align,
+ base) ?
+ -ENOMEM :
+ 0;
}
enum hns_roce_qp_state to_hns_roce_state(enum ib_qp_state state)
diff --git a/drivers/infiniband/hw/i40iw/i40iw_verbs.c b/drivers/infiniband/hw/i40iw/i40iw_verbs.c
index 4b892ca..095912fb 100644
--- a/drivers/infiniband/hw/i40iw/i40iw_verbs.c
+++ b/drivers/infiniband/hw/i40iw/i40iw_verbs.c
@@ -1515,6 +1515,7 @@ static struct ib_mr *i40iw_alloc_mr(struct ib_pd *pd,
err_code = -EOVERFLOW;
goto err;
}
+ stag &= ~I40IW_CQPSQ_STAG_KEY_MASK;
iwmr->stag = stag;
iwmr->ibmr.rkey = stag;
iwmr->ibmr.lkey = stag;
diff --git a/drivers/infiniband/hw/mlx4/mad.c b/drivers/infiniband/hw/mlx4/mad.c
index 18d309e..d9323d7 100644
--- a/drivers/infiniband/hw/mlx4/mad.c
+++ b/drivers/infiniband/hw/mlx4/mad.c
@@ -1897,7 +1897,6 @@ static void mlx4_ib_sqp_comp_worker(struct work_struct *work)
"buf:%lld\n", wc.wr_id);
break;
default:
- BUG_ON(1);
break;
}
} else {
diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index 19bc1c2..8d59a59 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -216,8 +216,6 @@ static int mlx4_ib_update_gids_v1_v2(struct gid_entry *gids,
gid_tbl[i].version = 2;
if (!ipv6_addr_v4mapped((struct in6_addr *)&gids[i].gid))
gid_tbl[i].type = 1;
- else
- memset(&gid_tbl[i].gid, 0, 12);
}
}
@@ -363,8 +361,13 @@ static int mlx4_ib_del_gid(struct ib_device *device,
if (!gids) {
ret = -ENOMEM;
} else {
- for (i = 0; i < MLX4_MAX_PORT_GIDS; i++)
- memcpy(&gids[i].gid, &port_gid_table->gids[i].gid, sizeof(union ib_gid));
+ for (i = 0; i < MLX4_MAX_PORT_GIDS; i++) {
+ memcpy(&gids[i].gid,
+ &port_gid_table->gids[i].gid,
+ sizeof(union ib_gid));
+ gids[i].gid_type =
+ port_gid_table->gids[i].gid_type;
+ }
}
}
spin_unlock_bh(&iboe->lock);
diff --git a/drivers/infiniband/hw/mlx4/mr.c b/drivers/infiniband/hw/mlx4/mr.c
index ae41623..ddd3182 100644
--- a/drivers/infiniband/hw/mlx4/mr.c
+++ b/drivers/infiniband/hw/mlx4/mr.c
@@ -131,6 +131,40 @@ int mlx4_ib_umem_write_mtt(struct mlx4_ib_dev *dev, struct mlx4_mtt *mtt,
return err;
}
+static struct ib_umem *mlx4_get_umem_mr(struct ib_ucontext *context, u64 start,
+ u64 length, u64 virt_addr,
+ int access_flags)
+{
+ /*
+ * Force registering the memory as writable if the underlying pages
+ * are writable. This is so rereg can change the access permissions
+ * from readable to writable without having to run through ib_umem_get
+ * again
+ */
+ if (!ib_access_writable(access_flags)) {
+ struct vm_area_struct *vma;
+
+ down_read(¤t->mm->mmap_sem);
+ /*
+ * FIXME: Ideally this would iterate over all the vmas that
+ * cover the memory, but for now it requires a single vma to
+ * entirely cover the MR to support RO mappings.
+ */
+ vma = find_vma(current->mm, start);
+ if (vma && vma->vm_end >= start + length &&
+ vma->vm_start <= start) {
+ if (vma->vm_flags & VM_WRITE)
+ access_flags |= IB_ACCESS_LOCAL_WRITE;
+ } else {
+ access_flags |= IB_ACCESS_LOCAL_WRITE;
+ }
+
+ up_read(¤t->mm->mmap_sem);
+ }
+
+ return ib_umem_get(context, start, length, access_flags, 0);
+}
+
struct ib_mr *mlx4_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
u64 virt_addr, int access_flags,
struct ib_udata *udata)
@@ -145,10 +179,8 @@ struct ib_mr *mlx4_ib_reg_user_mr(struct ib_pd *pd, u64 start, u64 length,
if (!mr)
return ERR_PTR(-ENOMEM);
- /* Force registering the memory as writable. */
- /* Used for memory re-registeration. HCA protects the access */
- mr->umem = ib_umem_get(pd->uobject->context, start, length,
- access_flags | IB_ACCESS_LOCAL_WRITE, 0);
+ mr->umem = mlx4_get_umem_mr(pd->uobject->context, start, length,
+ virt_addr, access_flags);
if (IS_ERR(mr->umem)) {
err = PTR_ERR(mr->umem);
goto err_free;
@@ -215,6 +247,12 @@ int mlx4_ib_rereg_user_mr(struct ib_mr *mr, int flags,
}
if (flags & IB_MR_REREG_ACCESS) {
+ if (ib_access_writable(mr_access_flags) &&
+ !mmr->umem->writable) {
+ err = -EPERM;
+ goto release_mpt_entry;
+ }
+
err = mlx4_mr_hw_change_access(dev->dev, *pmpt_entry,
convert_access(mr_access_flags));
@@ -228,10 +266,9 @@ int mlx4_ib_rereg_user_mr(struct ib_mr *mr, int flags,
mlx4_mr_rereg_mem_cleanup(dev->dev, &mmr->mmr);
ib_umem_release(mmr->umem);
- mmr->umem = ib_umem_get(mr->uobject->context, start, length,
- mr_access_flags |
- IB_ACCESS_LOCAL_WRITE,
- 0);
+ mmr->umem =
+ mlx4_get_umem_mr(mr->uobject->context, start, length,
+ virt_addr, mr_access_flags);
if (IS_ERR(mmr->umem)) {
err = PTR_ERR(mmr->umem);
/* Prevent mlx4_ib_dereg_mr from free'ing invalid pointer */
diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index fc62a7d..a19ebb1 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -645,7 +645,7 @@ static int mlx5_poll_one(struct mlx5_ib_cq *cq,
}
static int poll_soft_wc(struct mlx5_ib_cq *cq, int num_entries,
- struct ib_wc *wc)
+ struct ib_wc *wc, bool is_fatal_err)
{
struct mlx5_ib_dev *dev = to_mdev(cq->ibcq.device);
struct mlx5_ib_wc *soft_wc, *next;
@@ -658,6 +658,10 @@ static int poll_soft_wc(struct mlx5_ib_cq *cq, int num_entries,
mlx5_ib_dbg(dev, "polled software generated completion on CQ 0x%x\n",
cq->mcq.cqn);
+ if (unlikely(is_fatal_err)) {
+ soft_wc->wc.status = IB_WC_WR_FLUSH_ERR;
+ soft_wc->wc.vendor_err = MLX5_CQE_SYNDROME_WR_FLUSH_ERR;
+ }
wc[npolled++] = soft_wc->wc;
list_del(&soft_wc->list);
kfree(soft_wc);
@@ -678,12 +682,17 @@ int mlx5_ib_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
spin_lock_irqsave(&cq->lock, flags);
if (mdev->state == MLX5_DEVICE_STATE_INTERNAL_ERROR) {
- mlx5_ib_poll_sw_comp(cq, num_entries, wc, &npolled);
+ /* make sure no soft wqe's are waiting */
+ if (unlikely(!list_empty(&cq->wc_list)))
+ soft_polled = poll_soft_wc(cq, num_entries, wc, true);
+
+ mlx5_ib_poll_sw_comp(cq, num_entries - soft_polled,
+ wc + soft_polled, &npolled);
goto out;
}
if (unlikely(!list_empty(&cq->wc_list)))
- soft_polled = poll_soft_wc(cq, num_entries, wc);
+ soft_polled = poll_soft_wc(cq, num_entries, wc, false);
for (npolled = 0; npolled < num_entries - soft_polled; npolled++) {
if (mlx5_poll_one(cq, &cur_qp, wc + soft_polled + npolled))
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index d7da1dc..b1daf5c 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -710,31 +710,26 @@ enum mlx5_ib_width {
MLX5_IB_WIDTH_12X = 1 << 4
};
-static int translate_active_width(struct ib_device *ibdev, u8 active_width,
+static void translate_active_width(struct ib_device *ibdev, u8 active_width,
u8 *ib_width)
{
struct mlx5_ib_dev *dev = to_mdev(ibdev);
- int err = 0;
- if (active_width & MLX5_IB_WIDTH_1X) {
+ if (active_width & MLX5_IB_WIDTH_1X)
*ib_width = IB_WIDTH_1X;
- } else if (active_width & MLX5_IB_WIDTH_2X) {
- mlx5_ib_dbg(dev, "active_width %d is not supported by IB spec\n",
- (int)active_width);
- err = -EINVAL;
- } else if (active_width & MLX5_IB_WIDTH_4X) {
+ else if (active_width & MLX5_IB_WIDTH_4X)
*ib_width = IB_WIDTH_4X;
- } else if (active_width & MLX5_IB_WIDTH_8X) {
+ else if (active_width & MLX5_IB_WIDTH_8X)
*ib_width = IB_WIDTH_8X;
- } else if (active_width & MLX5_IB_WIDTH_12X) {
+ else if (active_width & MLX5_IB_WIDTH_12X)
*ib_width = IB_WIDTH_12X;
- } else {
- mlx5_ib_dbg(dev, "Invalid active_width %d\n",
+ else {
+ mlx5_ib_dbg(dev, "Invalid active_width %d, setting width to default value: 4x\n",
(int)active_width);
- err = -EINVAL;
+ *ib_width = IB_WIDTH_4X;
}
- return err;
+ return;
}
static int mlx5_mtu_to_ib_mtu(int mtu)
@@ -842,10 +837,8 @@ static int mlx5_query_hca_port(struct ib_device *ibdev, u8 port,
if (err)
goto out;
- err = translate_active_width(ibdev, ib_link_width_oper,
- &props->active_width);
- if (err)
- goto out;
+ translate_active_width(ibdev, ib_link_width_oper, &props->active_width);
+
err = mlx5_query_port_ib_proto_oper(mdev, &props->active_speed, port);
if (err)
goto out;
diff --git a/drivers/infiniband/hw/mlx5/qp.c b/drivers/infiniband/hw/mlx5/qp.c
index 403df35..f89489b 100644
--- a/drivers/infiniband/hw/mlx5/qp.c
+++ b/drivers/infiniband/hw/mlx5/qp.c
@@ -253,7 +253,11 @@ static int set_rq_size(struct mlx5_ib_dev *dev, struct ib_qp_cap *cap,
} else {
if (ucmd) {
qp->rq.wqe_cnt = ucmd->rq_wqe_count;
+ if (ucmd->rq_wqe_shift > BITS_PER_BYTE * sizeof(ucmd->rq_wqe_shift))
+ return -EINVAL;
qp->rq.wqe_shift = ucmd->rq_wqe_shift;
+ if ((1 << qp->rq.wqe_shift) / sizeof(struct mlx5_wqe_data_seg) < qp->wq_sig)
+ return -EINVAL;
qp->rq.max_gs = (1 << qp->rq.wqe_shift) / sizeof(struct mlx5_wqe_data_seg) - qp->wq_sig;
qp->rq.max_post = qp->rq.wqe_cnt;
} else {
@@ -1519,6 +1523,7 @@ static int create_qp_common(struct mlx5_ib_dev *dev, struct ib_pd *pd,
u32 uidx = MLX5_IB_DEFAULT_UIDX;
struct mlx5_ib_create_qp ucmd;
struct mlx5_ib_qp_base *base;
+ int mlx5_st;
void *qpc;
u32 *in;
int err;
@@ -1534,6 +1539,10 @@ static int create_qp_common(struct mlx5_ib_dev *dev, struct ib_pd *pd,
spin_lock_init(&qp->sq.lock);
spin_lock_init(&qp->rq.lock);
+ mlx5_st = to_mlx5_st(init_attr->qp_type);
+ if (mlx5_st < 0)
+ return -EINVAL;
+
if (init_attr->rwq_ind_tbl) {
if (!udata)
return -ENOSYS;
@@ -1661,7 +1670,7 @@ static int create_qp_common(struct mlx5_ib_dev *dev, struct ib_pd *pd,
qpc = MLX5_ADDR_OF(create_qp_in, in, qpc);
- MLX5_SET(qpc, qpc, st, to_mlx5_st(init_attr->qp_type));
+ MLX5_SET(qpc, qpc, st, mlx5_st);
MLX5_SET(qpc, qpc, pm_state, MLX5_QP_PM_MIGRATED);
if (init_attr->qp_type != MLX5_IB_QPT_REG_UMR)
@@ -2164,18 +2173,18 @@ enum {
static int ib_rate_to_mlx5(struct mlx5_ib_dev *dev, u8 rate)
{
- if (rate == IB_RATE_PORT_CURRENT) {
+ if (rate == IB_RATE_PORT_CURRENT)
return 0;
- } else if (rate < IB_RATE_2_5_GBPS || rate > IB_RATE_300_GBPS) {
- return -EINVAL;
- } else {
- while (rate != IB_RATE_2_5_GBPS &&
- !(1 << (rate + MLX5_STAT_RATE_OFFSET) &
- MLX5_CAP_GEN(dev->mdev, stat_rate_support)))
- --rate;
- }
- return rate + MLX5_STAT_RATE_OFFSET;
+ if (rate < IB_RATE_2_5_GBPS || rate > IB_RATE_300_GBPS)
+ return -EINVAL;
+
+ while (rate != IB_RATE_PORT_CURRENT &&
+ !(1 << (rate + MLX5_STAT_RATE_OFFSET) &
+ MLX5_CAP_GEN(dev->mdev, stat_rate_support)))
+ --rate;
+
+ return rate ? rate + MLX5_STAT_RATE_OFFSET : rate;
}
static int modify_raw_packet_eth_prio(struct mlx5_core_dev *dev,
@@ -2805,8 +2814,10 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp,
mlx5_ib_qp_disable_pagefaults(qp);
if (mlx5_cur >= MLX5_QP_NUM_STATE || mlx5_new >= MLX5_QP_NUM_STATE ||
- !optab[mlx5_cur][mlx5_new])
+ !optab[mlx5_cur][mlx5_new]) {
+ err = -EINVAL;
goto out;
+ }
op = optab[mlx5_cur][mlx5_new];
optpar = ib_mask_to_mlx5_opt(attr_mask);
@@ -2848,7 +2859,8 @@ static int __mlx5_ib_modify_qp(struct ib_qp *ibqp,
* If we moved a kernel QP to RESET, clean up all old CQ
* entries and reinitialize the QP.
*/
- if (new_state == IB_QPS_RESET && !ibqp->uobject) {
+ if (new_state == IB_QPS_RESET &&
+ !ibqp->uobject && ibqp->qp_type != IB_QPT_XRC_TGT) {
mlx5_ib_cq_clean(recv_cq, base->mqp.qpn,
ibqp->srq ? to_msrq(ibqp->srq) : NULL);
if (send_cq != recv_cq)
@@ -3876,17 +3888,18 @@ int mlx5_ib_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
goto out;
}
- if (wr->opcode == IB_WR_LOCAL_INV ||
- wr->opcode == IB_WR_REG_MR) {
+ if (wr->opcode == IB_WR_REG_MR) {
fence = dev->umr_fence;
next_fence = MLX5_FENCE_MODE_INITIATOR_SMALL;
- } else if (wr->send_flags & IB_SEND_FENCE) {
- if (qp->next_fence)
- fence = MLX5_FENCE_MODE_SMALL_AND_FENCE;
- else
- fence = MLX5_FENCE_MODE_FENCE;
- } else {
- fence = qp->next_fence;
+ } else {
+ if (wr->send_flags & IB_SEND_FENCE) {
+ if (qp->next_fence)
+ fence = MLX5_FENCE_MODE_SMALL_AND_FENCE;
+ else
+ fence = MLX5_FENCE_MODE_FENCE;
+ } else {
+ fence = qp->next_fence;
+ }
}
switch (ibqp->qp_type) {
@@ -4605,13 +4618,10 @@ int mlx5_ib_dealloc_xrcd(struct ib_xrcd *xrcd)
int err;
err = mlx5_core_xrcd_dealloc(dev->mdev, xrcdn);
- if (err) {
+ if (err)
mlx5_ib_warn(dev, "failed to dealloc xrcdn 0x%x\n", xrcdn);
- return err;
- }
kfree(xrcd);
-
return 0;
}
diff --git a/drivers/infiniband/hw/mlx5/srq.c b/drivers/infiniband/hw/mlx5/srq.c
index 5c1dbe2..d6e5002 100644
--- a/drivers/infiniband/hw/mlx5/srq.c
+++ b/drivers/infiniband/hw/mlx5/srq.c
@@ -268,18 +268,24 @@ struct ib_srq *mlx5_ib_create_srq(struct ib_pd *pd,
desc_size = sizeof(struct mlx5_wqe_srq_next_seg) +
srq->msrq.max_gs * sizeof(struct mlx5_wqe_data_seg);
- if (desc_size == 0 || srq->msrq.max_gs > desc_size)
- return ERR_PTR(-EINVAL);
+ if (desc_size == 0 || srq->msrq.max_gs > desc_size) {
+ err = -EINVAL;
+ goto err_srq;
+ }
desc_size = roundup_pow_of_two(desc_size);
desc_size = max_t(size_t, 32, desc_size);
- if (desc_size < sizeof(struct mlx5_wqe_srq_next_seg))
- return ERR_PTR(-EINVAL);
+ if (desc_size < sizeof(struct mlx5_wqe_srq_next_seg)) {
+ err = -EINVAL;
+ goto err_srq;
+ }
srq->msrq.max_avail_gather = (desc_size - sizeof(struct mlx5_wqe_srq_next_seg)) /
sizeof(struct mlx5_wqe_data_seg);
srq->msrq.wqe_shift = ilog2(desc_size);
buf_size = srq->msrq.max * desc_size;
- if (buf_size < desc_size)
- return ERR_PTR(-EINVAL);
+ if (buf_size < desc_size) {
+ err = -EINVAL;
+ goto err_srq;
+ }
in.type = init_attr->srq_type;
if (pd->uobject)
diff --git a/drivers/infiniband/hw/ocrdma/ocrdma_stats.c b/drivers/infiniband/hw/ocrdma/ocrdma_stats.c
index 2659430..84349d9 100644
--- a/drivers/infiniband/hw/ocrdma/ocrdma_stats.c
+++ b/drivers/infiniband/hw/ocrdma/ocrdma_stats.c
@@ -645,7 +645,7 @@ static ssize_t ocrdma_dbgfs_ops_write(struct file *filp,
struct ocrdma_stats *pstats = filp->private_data;
struct ocrdma_dev *dev = pstats->dev;
- if (count > 32)
+ if (*ppos != 0 || count == 0 || count > sizeof(tmp_str))
goto err;
if (copy_from_user(tmp_str, buffer, count))
diff --git a/drivers/infiniband/hw/qedr/main.c b/drivers/infiniband/hw/qedr/main.c
index 58e92bc..f937873e 100644
--- a/drivers/infiniband/hw/qedr/main.c
+++ b/drivers/infiniband/hw/qedr/main.c
@@ -762,7 +762,8 @@ static struct qedr_dev *qedr_add(struct qed_dev *cdev, struct pci_dev *pdev,
dev->num_cnq = dev->ops->rdma_get_min_cnq_msix(cdev);
if (!dev->num_cnq) {
- DP_ERR(dev, "not enough CNQ resources.\n");
+ DP_ERR(dev, "Failed. At least one CNQ is required.\n");
+ rc = -ENOMEM;
goto init_err;
}
diff --git a/drivers/infiniband/hw/qedr/verbs.c b/drivers/infiniband/hw/qedr/verbs.c
index 35d5b89..cd0408c 100644
--- a/drivers/infiniband/hw/qedr/verbs.c
+++ b/drivers/infiniband/hw/qedr/verbs.c
@@ -1888,18 +1888,23 @@ int qedr_modify_qp(struct ib_qp *ibqp, struct ib_qp_attr *attr,
SET_FIELD(qp_params.modify_flags,
QED_ROCE_MODIFY_QP_VALID_ACK_TIMEOUT, 1);
- qp_params.ack_timeout = attr->timeout;
- if (attr->timeout) {
- u32 temp;
-
- temp = 4096 * (1UL << attr->timeout) / 1000 / 1000;
- /* FW requires [msec] */
- qp_params.ack_timeout = temp;
- } else {
- /* Infinite */
+ /* The received timeout value is an exponent used like this:
+ * "12.7.34 LOCAL ACK TIMEOUT
+ * Value representing the transport (ACK) timeout for use by
+ * the remote, expressed as: 4.096 * 2^timeout [usec]"
+ * The FW expects timeout in msec so we need to divide the usec
+ * result by 1000. We'll approximate 1000~2^10, and 4.096 ~ 2^2,
+ * so we get: 2^2 * 2^timeout / 2^10 = 2^(timeout - 8).
+ * The value of zero means infinite so we use a 'max_t' to make
+ * sure that sub 1 msec values will be configured as 1 msec.
+ */
+ if (attr->timeout)
+ qp_params.ack_timeout =
+ 1 << max_t(int, attr->timeout - 8, 0);
+ else
qp_params.ack_timeout = 0;
- }
}
+
if (attr_mask & IB_QP_RETRY_CNT) {
SET_FIELD(qp_params.modify_flags,
QED_ROCE_MODIFY_QP_VALID_RETRY_CNT, 1);
@@ -2807,6 +2812,11 @@ int __qedr_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
switch (wr->opcode) {
case IB_WR_SEND_WITH_IMM:
+ if (unlikely(rdma_protocol_iwarp(&dev->ibdev, 1))) {
+ rc = -EINVAL;
+ *bad_wr = wr;
+ break;
+ }
wqe->req_type = RDMA_SQ_REQ_TYPE_SEND_WITH_IMM;
swqe = (struct rdma_sq_send_wqe_1st *)wqe;
swqe->wqe_size = 2;
@@ -2848,6 +2858,11 @@ int __qedr_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
break;
case IB_WR_RDMA_WRITE_WITH_IMM:
+ if (unlikely(rdma_protocol_iwarp(&dev->ibdev, 1))) {
+ rc = -EINVAL;
+ *bad_wr = wr;
+ break;
+ }
wqe->req_type = RDMA_SQ_REQ_TYPE_RDMA_WR_WITH_IMM;
rwqe = (struct rdma_sq_rdma_wqe_1st *)wqe;
@@ -3467,7 +3482,7 @@ int qedr_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
{
struct qedr_dev *dev = get_qedr_dev(ibcq->device);
struct qedr_cq *cq = get_qedr_cq(ibcq);
- union rdma_cqe *cqe = cq->latest_cqe;
+ union rdma_cqe *cqe;
u32 old_cons, new_cons;
unsigned long flags;
int update = 0;
@@ -3477,6 +3492,7 @@ int qedr_poll_cq(struct ib_cq *ibcq, int num_entries, struct ib_wc *wc)
return qedr_gsi_poll_cq(ibcq, num_entries, wc);
spin_lock_irqsave(&cq->cq_lock, flags);
+ cqe = cq->latest_cqe;
old_cons = qed_chain_get_cons_idx_u32(&cq->pbl);
while (num_entries && is_valid_cqe(cq, cqe)) {
struct qedr_qp *qp;
diff --git a/drivers/infiniband/hw/qib/qib.h b/drivers/infiniband/hw/qib/qib.h
index a3e21a2..ef092cc 100644
--- a/drivers/infiniband/hw/qib/qib.h
+++ b/drivers/infiniband/hw/qib/qib.h
@@ -1250,6 +1250,7 @@ static inline struct qib_ibport *to_iport(struct ib_device *ibdev, u8 port)
#define QIB_BADINTR 0x8000 /* severe interrupt problems */
#define QIB_DCA_ENABLED 0x10000 /* Direct Cache Access enabled */
#define QIB_HAS_QSFP 0x20000 /* device (card instance) has QSFP */
+#define QIB_SHUTDOWN 0x40000 /* device is shutting down */
/*
* values for ppd->lflags (_ib_port_ related flags)
@@ -1448,8 +1449,7 @@ u64 qib_sps_ints(void);
/*
* dma_addr wrappers - all 0's invalid for hw
*/
-dma_addr_t qib_map_page(struct pci_dev *, struct page *, unsigned long,
- size_t, int);
+int qib_map_page(struct pci_dev *d, struct page *p, dma_addr_t *daddr);
const char *qib_get_unit_name(int unit);
const char *qib_get_card_name(struct rvt_dev_info *rdi);
struct pci_dev *qib_get_pci_dev(struct rvt_dev_info *rdi);
diff --git a/drivers/infiniband/hw/qib/qib_file_ops.c b/drivers/infiniband/hw/qib/qib_file_ops.c
index 382466a..cc6a923 100644
--- a/drivers/infiniband/hw/qib/qib_file_ops.c
+++ b/drivers/infiniband/hw/qib/qib_file_ops.c
@@ -364,6 +364,8 @@ static int qib_tid_update(struct qib_ctxtdata *rcd, struct file *fp,
goto done;
}
for (i = 0; i < cnt; i++, vaddr += PAGE_SIZE) {
+ dma_addr_t daddr;
+
for (; ntids--; tid++) {
if (tid == tidcnt)
tid = 0;
@@ -380,12 +382,14 @@ static int qib_tid_update(struct qib_ctxtdata *rcd, struct file *fp,
ret = -ENOMEM;
break;
}
+ ret = qib_map_page(dd->pcidev, pagep[i], &daddr);
+ if (ret)
+ break;
+
tidlist[i] = tid + tidoff;
/* we "know" system pages and TID pages are same size */
dd->pageshadow[ctxttid + tid] = pagep[i];
- dd->physshadow[ctxttid + tid] =
- qib_map_page(dd->pcidev, pagep[i], 0, PAGE_SIZE,
- PCI_DMA_FROMDEVICE);
+ dd->physshadow[ctxttid + tid] = daddr;
/*
* don't need atomic or it's overhead
*/
diff --git a/drivers/infiniband/hw/qib/qib_init.c b/drivers/infiniband/hw/qib/qib_init.c
index 1730aa8..caf7c51 100644
--- a/drivers/infiniband/hw/qib/qib_init.c
+++ b/drivers/infiniband/hw/qib/qib_init.c
@@ -878,6 +878,10 @@ static void qib_shutdown_device(struct qib_devdata *dd)
struct qib_pportdata *ppd;
unsigned pidx;
+ if (dd->flags & QIB_SHUTDOWN)
+ return;
+ dd->flags |= QIB_SHUTDOWN;
+
for (pidx = 0; pidx < dd->num_pports; ++pidx) {
ppd = dd->pport + pidx;
@@ -1223,6 +1227,7 @@ void qib_disable_after_error(struct qib_devdata *dd)
static void qib_remove_one(struct pci_dev *);
static int qib_init_one(struct pci_dev *, const struct pci_device_id *);
+static void qib_shutdown_one(struct pci_dev *);
#define DRIVER_LOAD_MSG "Intel " QIB_DRV_NAME " loaded: "
#define PFX QIB_DRV_NAME ": "
@@ -1240,6 +1245,7 @@ static struct pci_driver qib_driver = {
.name = QIB_DRV_NAME,
.probe = qib_init_one,
.remove = qib_remove_one,
+ .shutdown = qib_shutdown_one,
.id_table = qib_pci_tbl,
.err_handler = &qib_pci_err_handler,
};
@@ -1591,6 +1597,13 @@ static void qib_remove_one(struct pci_dev *pdev)
qib_postinit_cleanup(dd);
}
+static void qib_shutdown_one(struct pci_dev *pdev)
+{
+ struct qib_devdata *dd = pci_get_drvdata(pdev);
+
+ qib_shutdown_device(dd);
+}
+
/**
* qib_create_rcvhdrq - create a receive header queue
* @dd: the qlogic_ib device
diff --git a/drivers/infiniband/hw/qib/qib_user_pages.c b/drivers/infiniband/hw/qib/qib_user_pages.c
index 75f0862..4715edf 100644
--- a/drivers/infiniband/hw/qib/qib_user_pages.c
+++ b/drivers/infiniband/hw/qib/qib_user_pages.c
@@ -98,23 +98,27 @@ static int __qib_get_user_pages(unsigned long start_page, size_t num_pages,
*
* I'm sure we won't be so lucky with other iommu's, so FIXME.
*/
-dma_addr_t qib_map_page(struct pci_dev *hwdev, struct page *page,
- unsigned long offset, size_t size, int direction)
+int qib_map_page(struct pci_dev *hwdev, struct page *page, dma_addr_t *daddr)
{
dma_addr_t phys;
- phys = pci_map_page(hwdev, page, offset, size, direction);
+ phys = pci_map_page(hwdev, page, 0, PAGE_SIZE, PCI_DMA_FROMDEVICE);
+ if (pci_dma_mapping_error(hwdev, phys))
+ return -ENOMEM;
- if (phys == 0) {
- pci_unmap_page(hwdev, phys, size, direction);
- phys = pci_map_page(hwdev, page, offset, size, direction);
+ if (!phys) {
+ pci_unmap_page(hwdev, phys, PAGE_SIZE, PCI_DMA_FROMDEVICE);
+ phys = pci_map_page(hwdev, page, 0, PAGE_SIZE,
+ PCI_DMA_FROMDEVICE);
+ if (pci_dma_mapping_error(hwdev, phys))
+ return -ENOMEM;
/*
* FIXME: If we get 0 again, we should keep this page,
* map another, then free the 0 page.
*/
}
-
- return phys;
+ *daddr = phys;
+ return 0;
}
/**
diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c
index 6c5e29d..df15b6d 100644
--- a/drivers/infiniband/sw/rxe/rxe_comp.c
+++ b/drivers/infiniband/sw/rxe/rxe_comp.c
@@ -273,6 +273,7 @@ static inline enum comp_state check_ack(struct rxe_qp *qp,
case IB_OPCODE_RC_RDMA_READ_RESPONSE_MIDDLE:
if (wqe->wr.opcode != IB_WR_RDMA_READ &&
wqe->wr.opcode != IB_WR_RDMA_READ_WITH_INV) {
+ wqe->status = IB_WC_FATAL_ERR;
return COMPST_ERROR;
}
reset_retry_counters(qp);
diff --git a/drivers/infiniband/sw/rxe/rxe_pool.c b/drivers/infiniband/sw/rxe/rxe_pool.c
index 1c4e5b2..527ca66 100644
--- a/drivers/infiniband/sw/rxe/rxe_pool.c
+++ b/drivers/infiniband/sw/rxe/rxe_pool.c
@@ -402,23 +402,25 @@ void *rxe_alloc(struct rxe_pool *pool)
kref_get(&pool->rxe->ref_cnt);
- if (atomic_inc_return(&pool->num_elem) > pool->max_elem) {
- atomic_dec(&pool->num_elem);
- rxe_dev_put(pool->rxe);
- rxe_pool_put(pool);
- return NULL;
- }
+ if (atomic_inc_return(&pool->num_elem) > pool->max_elem)
+ goto out_put_pool;
elem = kmem_cache_zalloc(pool_cache(pool),
(pool->flags & RXE_POOL_ATOMIC) ?
GFP_ATOMIC : GFP_KERNEL);
if (!elem)
- return NULL;
+ goto out_put_pool;
elem->pool = pool;
kref_init(&elem->ref_cnt);
return elem;
+
+out_put_pool:
+ atomic_dec(&pool->num_elem);
+ rxe_dev_put(pool->rxe);
+ rxe_pool_put(pool);
+ return NULL;
}
void rxe_elem_release(struct kref *kref)
diff --git a/drivers/infiniband/sw/rxe/rxe_recv.c b/drivers/infiniband/sw/rxe/rxe_recv.c
index 46f0628..db6bb02 100644
--- a/drivers/infiniband/sw/rxe/rxe_recv.c
+++ b/drivers/infiniband/sw/rxe/rxe_recv.c
@@ -225,9 +225,14 @@ static int hdr_check(struct rxe_pkt_info *pkt)
goto err1;
}
+ if (unlikely(qpn == 0)) {
+ pr_warn_once("QP 0 not supported");
+ goto err1;
+ }
+
if (qpn != IB_MULTICAST_QPN) {
- index = (qpn == 0) ? port->qp_smi_index :
- ((qpn == 1) ? port->qp_gsi_index : qpn);
+ index = (qpn == 1) ? port->qp_gsi_index : qpn;
+
qp = rxe_pool_get_index(&rxe->qp_pool, index);
if (unlikely(!qp)) {
pr_warn_ratelimited("no qp matches qpn 0x%x\n", qpn);
diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c
index 5b0ca35..47219eb 100644
--- a/drivers/infiniband/sw/rxe/rxe_req.c
+++ b/drivers/infiniband/sw/rxe/rxe_req.c
@@ -648,6 +648,9 @@ int rxe_requester(void *arg)
} else {
goto exit;
}
+ if ((wqe->wr.send_flags & IB_SEND_SIGNALED) ||
+ qp->sq_sig_type == IB_SIGNAL_ALL_WR)
+ rxe_run_task(&qp->comp.task, 1);
qp->req.wqe_index = next_index(qp->sq.queue,
qp->req.wqe_index);
goto next_wqe;
diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c
index 2152c71..297653a 100644
--- a/drivers/infiniband/sw/rxe/rxe_resp.c
+++ b/drivers/infiniband/sw/rxe/rxe_resp.c
@@ -826,11 +826,16 @@ static enum resp_states do_complete(struct rxe_qp *qp,
memset(&cqe, 0, sizeof(cqe));
- wc->wr_id = wqe->wr_id;
- wc->status = qp->resp.status;
- wc->qp = &qp->ibqp;
+ if (qp->rcq->is_user) {
+ uwc->status = qp->resp.status;
+ uwc->qp_num = qp->ibqp.qp_num;
+ uwc->wr_id = wqe->wr_id;
+ } else {
+ wc->status = qp->resp.status;
+ wc->qp = &qp->ibqp;
+ wc->wr_id = wqe->wr_id;
+ }
- /* fields after status are not required for errors */
if (wc->status == IB_WC_SUCCESS) {
wc->opcode = (pkt->mask & RXE_IMMDT_MASK &&
pkt->mask & RXE_WRITE_MASK) ?
diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c
index ced416f5..ef13082 100644
--- a/drivers/infiniband/sw/rxe/rxe_verbs.c
+++ b/drivers/infiniband/sw/rxe/rxe_verbs.c
@@ -729,13 +729,8 @@ static int init_send_wqe(struct rxe_qp *qp, struct ib_send_wr *ibwr,
sge = ibwr->sg_list;
for (i = 0; i < num_sge; i++, sge++) {
- if (qp->is_user && copy_from_user(p, (__user void *)
- (uintptr_t)sge->addr, sge->length))
- return -EFAULT;
-
- else if (!qp->is_user)
- memcpy(p, (void *)(uintptr_t)sge->addr,
- sge->length);
+ memcpy(p, (void *)(uintptr_t)sge->addr,
+ sge->length);
p += sge->length;
}
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
index 7576166..95a3e0a 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
@@ -1009,12 +1009,14 @@ static int ipoib_cm_rep_handler(struct ib_cm_id *cm_id, struct ib_cm_event *even
skb_queue_head_init(&skqueue);
+ netif_tx_lock_bh(p->dev);
spin_lock_irq(&priv->lock);
set_bit(IPOIB_FLAG_OPER_UP, &p->flags);
if (p->neigh)
while ((skb = __skb_dequeue(&p->neigh->queue)))
__skb_queue_tail(&skqueue, skb);
spin_unlock_irq(&priv->lock);
+ netif_tx_unlock_bh(p->dev);
while ((skb = __skb_dequeue(&skqueue))) {
skb->dev = p->dev;
@@ -1420,11 +1422,15 @@ static void ipoib_cm_skb_reap(struct work_struct *work)
spin_unlock_irqrestore(&priv->lock, flags);
netif_tx_unlock_bh(dev);
- if (skb->protocol == htons(ETH_P_IP))
+ if (skb->protocol == htons(ETH_P_IP)) {
+ memset(IPCB(skb), 0, sizeof(*IPCB(skb)));
icmp_send(skb, ICMP_DEST_UNREACH, ICMP_FRAG_NEEDED, htonl(mtu));
+ }
#if IS_ENABLED(CONFIG_IPV6)
- else if (skb->protocol == htons(ETH_P_IPV6))
+ else if (skb->protocol == htons(ETH_P_IPV6)) {
+ memset(IP6CB(skb), 0, sizeof(*IP6CB(skb)));
icmpv6_send(skb, ICMPV6_PKT_TOOBIG, 0, mtu);
+ }
#endif
dev_kfree_skb_any(skb);
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_fs.c b/drivers/infiniband/ulp/ipoib/ipoib_fs.c
index 09396bd..63be3bc 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_fs.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_fs.c
@@ -281,8 +281,6 @@ void ipoib_delete_debug_files(struct net_device *dev)
{
struct ipoib_dev_priv *priv = netdev_priv(dev);
- WARN_ONCE(!priv->mcg_dentry, "null mcg debug file\n");
- WARN_ONCE(!priv->path_dentry, "null path debug file\n");
debugfs_remove(priv->mcg_dentry);
debugfs_remove(priv->path_dentry);
priv->mcg_dentry = priv->path_dentry = NULL;
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_ib.c b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
index 34122c9..ad3089c 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_ib.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_ib.c
@@ -974,19 +974,6 @@ static inline int update_parent_pkey(struct ipoib_dev_priv *priv)
*/
priv->dev->broadcast[8] = priv->pkey >> 8;
priv->dev->broadcast[9] = priv->pkey & 0xff;
-
- /*
- * Update the broadcast address in the priv->broadcast object,
- * in case it already exists, otherwise no one will do that.
- */
- if (priv->broadcast) {
- spin_lock_irq(&priv->lock);
- memcpy(priv->broadcast->mcmember.mgid.raw,
- priv->dev->broadcast + 4,
- sizeof(union ib_gid));
- spin_unlock_irq(&priv->lock);
- }
-
return 0;
}
@@ -1190,13 +1177,10 @@ static void __ipoib_ib_dev_flush(struct ipoib_dev_priv *priv,
ipoib_ib_dev_down(dev);
if (level == IPOIB_FLUSH_HEAVY) {
- rtnl_lock();
if (test_bit(IPOIB_FLAG_INITIALIZED, &priv->flags))
ipoib_ib_dev_stop(dev);
- result = ipoib_ib_dev_open(dev);
- rtnl_unlock();
- if (result)
+ if (ipoib_ib_dev_open(dev))
return;
if (netif_queue_stopped(dev))
@@ -1236,7 +1220,9 @@ void ipoib_ib_dev_flush_heavy(struct work_struct *work)
struct ipoib_dev_priv *priv =
container_of(work, struct ipoib_dev_priv, flush_heavy);
+ rtnl_lock();
__ipoib_ib_dev_flush(priv, IPOIB_FLUSH_HEAVY, 0);
+ rtnl_unlock();
}
void ipoib_ib_dev_cleanup(struct net_device *dev)
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_main.c b/drivers/infiniband/ulp/ipoib/ipoib_main.c
index 0df7d45..17c5bc7 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_main.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c
@@ -2119,6 +2119,9 @@ static struct net_device *ipoib_add_port(const char *format,
goto event_failed;
}
+ /* call event handler to ensure pkey in sync */
+ queue_work(ipoib_workqueue, &priv->flush_heavy);
+
result = register_netdev(priv->dev);
if (result) {
printk(KERN_WARNING "%s: couldn't register ipoib port %d; error %d\n",
diff --git a/drivers/infiniband/ulp/iser/iser_verbs.c b/drivers/infiniband/ulp/iser/iser_verbs.c
index bc6f5bb..d46424d 100644
--- a/drivers/infiniband/ulp/iser/iser_verbs.c
+++ b/drivers/infiniband/ulp/iser/iser_verbs.c
@@ -1110,7 +1110,9 @@ u8 iser_check_task_pi_status(struct iscsi_iser_task *iser_task,
IB_MR_CHECK_SIG_STATUS, &mr_status);
if (ret) {
pr_err("ib_check_mr_status failed, ret %d\n", ret);
- goto err;
+ /* Not a lot we can do, return ambiguous guard error */
+ *sector = 0;
+ return 0x1;
}
if (mr_status.fail_status & IB_MR_CHECK_SIG_STATUS) {
@@ -1138,9 +1140,6 @@ u8 iser_check_task_pi_status(struct iscsi_iser_task *iser_task,
}
return 0;
-err:
- /* Not alot we can do here, return ambiguous guard error */
- return 0x1;
}
void iser_err_comp(struct ib_wc *wc, const char *type)
diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c
index b879d21..02a5e2d 100644
--- a/drivers/infiniband/ulp/isert/ib_isert.c
+++ b/drivers/infiniband/ulp/isert/ib_isert.c
@@ -879,15 +879,9 @@ isert_login_post_send(struct isert_conn *isert_conn, struct iser_tx_desc *tx_des
}
static void
-isert_create_send_desc(struct isert_conn *isert_conn,
- struct isert_cmd *isert_cmd,
- struct iser_tx_desc *tx_desc)
+__isert_create_send_desc(struct isert_device *device,
+ struct iser_tx_desc *tx_desc)
{
- struct isert_device *device = isert_conn->device;
- struct ib_device *ib_dev = device->ib_device;
-
- ib_dma_sync_single_for_cpu(ib_dev, tx_desc->dma_addr,
- ISER_HEADERS_LEN, DMA_TO_DEVICE);
memset(&tx_desc->iser_header, 0, sizeof(struct iser_ctrl));
tx_desc->iser_header.flags = ISCSI_CTRL;
@@ -900,6 +894,20 @@ isert_create_send_desc(struct isert_conn *isert_conn,
}
}
+static void
+isert_create_send_desc(struct isert_conn *isert_conn,
+ struct isert_cmd *isert_cmd,
+ struct iser_tx_desc *tx_desc)
+{
+ struct isert_device *device = isert_conn->device;
+ struct ib_device *ib_dev = device->ib_device;
+
+ ib_dma_sync_single_for_cpu(ib_dev, tx_desc->dma_addr,
+ ISER_HEADERS_LEN, DMA_TO_DEVICE);
+
+ __isert_create_send_desc(device, tx_desc);
+}
+
static int
isert_init_tx_hdrs(struct isert_conn *isert_conn,
struct iser_tx_desc *tx_desc)
@@ -987,7 +995,7 @@ isert_put_login_tx(struct iscsi_conn *conn, struct iscsi_login *login,
struct iser_tx_desc *tx_desc = &isert_conn->login_tx_desc;
int ret;
- isert_create_send_desc(isert_conn, NULL, tx_desc);
+ __isert_create_send_desc(device, tx_desc);
memcpy(&tx_desc->iscsi_header, &login->rsp[0],
sizeof(struct iscsi_hdr));
@@ -2082,7 +2090,7 @@ isert_set_sig_attrs(struct se_cmd *se_cmd, struct ib_sig_attrs *sig_attrs)
sig_attrs->check_mask =
(se_cmd->prot_checks & TARGET_DIF_CHECK_GUARD ? 0xc0 : 0) |
- (se_cmd->prot_checks & TARGET_DIF_CHECK_REFTAG ? 0x30 : 0) |
+ (se_cmd->prot_checks & TARGET_DIF_CHECK_APPTAG ? 0x30 : 0) |
(se_cmd->prot_checks & TARGET_DIF_CHECK_REFTAG ? 0x0f : 0);
return 0;
}
diff --git a/drivers/infiniband/ulp/srp/ib_srp.c b/drivers/infiniband/ulp/srp/ib_srp.c
index 463ea59..6463590 100644
--- a/drivers/infiniband/ulp/srp/ib_srp.c
+++ b/drivers/infiniband/ulp/srp/ib_srp.c
@@ -2639,7 +2639,7 @@ static int srp_reset_device(struct scsi_cmnd *scmnd)
{
struct srp_target_port *target = host_to_target(scmnd->device->host);
struct srp_rdma_ch *ch;
- int i;
+ int i, j;
u8 status;
shost_printk(KERN_ERR, target->scsi_host, "SRP reset_device called\n");
@@ -2653,8 +2653,8 @@ static int srp_reset_device(struct scsi_cmnd *scmnd)
for (i = 0; i < target->ch_count; i++) {
ch = &target->ch[i];
- for (i = 0; i < target->req_ring_size; ++i) {
- struct srp_request *req = &ch->req_ring[i];
+ for (j = 0; j < target->req_ring_size; ++j) {
+ struct srp_request *req = &ch->req_ring[j];
srp_finish_req(ch, req, scmnd->device, DID_RESET << 16);
}
diff --git a/drivers/infiniband/ulp/srpt/ib_srpt.c b/drivers/infiniband/ulp/srpt/ib_srpt.c
index 9888c9b..1c206df 100644
--- a/drivers/infiniband/ulp/srpt/ib_srpt.c
+++ b/drivers/infiniband/ulp/srpt/ib_srpt.c
@@ -1701,8 +1701,7 @@ static bool srpt_close_ch(struct srpt_rdma_ch *ch)
int ret;
if (!srpt_set_ch_state(ch, CH_DRAINING)) {
- pr_debug("%s-%d: already closed\n", ch->sess_name,
- ch->qp->qp_num);
+ pr_debug("%s: already closed\n", ch->sess_name);
return false;
}
@@ -1764,8 +1763,8 @@ static void __srpt_close_all_ch(struct srpt_device *sdev)
list_for_each_entry(ch, &sdev->rch_list, list) {
if (srpt_disconnect_ch(ch) >= 0)
- pr_info("Closing channel %s-%d because target %s has been disabled\n",
- ch->sess_name, ch->qp->qp_num,
+ pr_info("Closing channel %s because target %s has been disabled\n",
+ ch->sess_name,
sdev->device->name);
srpt_close_ch(ch);
}
diff --git a/drivers/input/Kconfig b/drivers/input/Kconfig
index 34ffa02..8eeddf6 100644
--- a/drivers/input/Kconfig
+++ b/drivers/input/Kconfig
@@ -194,6 +194,14 @@
---help---
Say Y here if you want to reboot when some keys are pressed;
+config INPUT_KEYDEBUG
+ bool "Long Press Key Debugging"
+ depends on INPUT
+ select INPUT_KEYCOMBO
+ ---help---
+ Say Y here if you want to hook debug function to dump information
+ to debug device hang when some keys are long pressed.
+
config INPUT_KEYCOMBO
bool "Key combo"
depends on INPUT
diff --git a/drivers/input/Makefile b/drivers/input/Makefile
index d01a5b1..cfbce86 100644
--- a/drivers/input/Makefile
+++ b/drivers/input/Makefile
@@ -33,4 +33,5 @@
obj-$(CONFIG_INPUT_KEYRESET) += keyreset.o
obj-$(CONFIG_INPUT_KEYCOMBO) += keycombo.o
-
+obj-$(CONFIG_INPUT_KEYDEBUG) += keydebug.o
+keydebug-y := keydebug-core.o keydebug-func.o
diff --git a/drivers/input/input-leds.c b/drivers/input/input-leds.c
index 766bf26..5f04b2d 100644
--- a/drivers/input/input-leds.c
+++ b/drivers/input/input-leds.c
@@ -88,6 +88,7 @@ static int input_leds_connect(struct input_handler *handler,
const struct input_device_id *id)
{
struct input_leds *leds;
+ struct input_led *led;
unsigned int num_leds;
unsigned int led_code;
int led_no;
@@ -119,14 +120,13 @@ static int input_leds_connect(struct input_handler *handler,
led_no = 0;
for_each_set_bit(led_code, dev->ledbit, LED_CNT) {
- struct input_led *led = &leds->leds[led_no];
-
- led->handle = &leds->handle;
- led->code = led_code;
-
if (!input_led_info[led_code].name)
continue;
+ led = &leds->leds[led_no];
+ led->handle = &leds->handle;
+ led->code = led_code;
+
led->cdev.name = kasprintf(GFP_KERNEL, "%s::%s",
dev_name(&dev->dev),
input_led_info[led_code].name);
diff --git a/drivers/input/joystick/xpad.c b/drivers/input/joystick/xpad.c
index f397a5b..f55dcdf 100644
--- a/drivers/input/joystick/xpad.c
+++ b/drivers/input/joystick/xpad.c
@@ -89,8 +89,10 @@
#define XPAD_PKT_LEN 64
-/* xbox d-pads should map to buttons, as is required for DDR pads
- but we map them to axes when possible to simplify things */
+/*
+ * xbox d-pads should map to buttons, as is required for DDR pads
+ * but we map them to axes when possible to simplify things
+ */
#define MAP_DPAD_TO_BUTTONS (1 << 0)
#define MAP_TRIGGERS_TO_BUTTONS (1 << 1)
#define MAP_STICKS_TO_NULL (1 << 2)
@@ -126,45 +128,77 @@ static const struct xpad_device {
u8 mapping;
u8 xtype;
} xpad_device[] = {
+ { 0x0079, 0x18d4, "GPD Win 2 X-Box Controller", 0, XTYPE_XBOX360 },
+ { 0x044f, 0x0f00, "Thrustmaster Wheel", 0, XTYPE_XBOX },
+ { 0x044f, 0x0f03, "Thrustmaster Wheel", 0, XTYPE_XBOX },
+ { 0x044f, 0x0f07, "Thrustmaster, Inc. Controller", 0, XTYPE_XBOX },
+ { 0x044f, 0x0f10, "Thrustmaster Modena GT Wheel", 0, XTYPE_XBOX },
+ { 0x044f, 0xb326, "Thrustmaster Gamepad GP XID", 0, XTYPE_XBOX360 },
{ 0x045e, 0x0202, "Microsoft X-Box pad v1 (US)", 0, XTYPE_XBOX },
{ 0x045e, 0x0285, "Microsoft X-Box pad (Japan)", 0, XTYPE_XBOX },
{ 0x045e, 0x0287, "Microsoft Xbox Controller S", 0, XTYPE_XBOX },
+ { 0x045e, 0x0288, "Microsoft Xbox Controller S v2", 0, XTYPE_XBOX },
{ 0x045e, 0x0289, "Microsoft X-Box pad v2 (US)", 0, XTYPE_XBOX },
{ 0x045e, 0x028e, "Microsoft X-Box 360 pad", 0, XTYPE_XBOX360 },
+ { 0x045e, 0x028f, "Microsoft X-Box 360 pad v2", 0, XTYPE_XBOX360 },
+ { 0x045e, 0x0291, "Xbox 360 Wireless Receiver (XBOX)", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360W },
{ 0x045e, 0x02d1, "Microsoft X-Box One pad", 0, XTYPE_XBOXONE },
{ 0x045e, 0x02dd, "Microsoft X-Box One pad (Firmware 2015)", 0, XTYPE_XBOXONE },
{ 0x045e, 0x02e3, "Microsoft X-Box One Elite pad", 0, XTYPE_XBOXONE },
- { 0x045e, 0x0291, "Xbox 360 Wireless Receiver (XBOX)", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360W },
+ { 0x045e, 0x02ea, "Microsoft X-Box One S pad", 0, XTYPE_XBOXONE },
{ 0x045e, 0x0719, "Xbox 360 Wireless Receiver", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360W },
- { 0x044f, 0x0f07, "Thrustmaster, Inc. Controller", 0, XTYPE_XBOX },
- { 0x044f, 0xb326, "Thrustmaster Gamepad GP XID", 0, XTYPE_XBOX360 },
{ 0x046d, 0xc21d, "Logitech Gamepad F310", 0, XTYPE_XBOX360 },
{ 0x046d, 0xc21e, "Logitech Gamepad F510", 0, XTYPE_XBOX360 },
{ 0x046d, 0xc21f, "Logitech Gamepad F710", 0, XTYPE_XBOX360 },
{ 0x046d, 0xc242, "Logitech Chillstream Controller", 0, XTYPE_XBOX360 },
{ 0x046d, 0xca84, "Logitech Xbox Cordless Controller", 0, XTYPE_XBOX },
{ 0x046d, 0xca88, "Logitech Compact Controller for Xbox", 0, XTYPE_XBOX },
+ { 0x046d, 0xca8a, "Logitech Precision Vibration Feedback Wheel", 0, XTYPE_XBOX },
+ { 0x046d, 0xcaa3, "Logitech DriveFx Racing Wheel", 0, XTYPE_XBOX360 },
+ { 0x056e, 0x2004, "Elecom JC-U3613M", 0, XTYPE_XBOX360 },
{ 0x05fd, 0x1007, "Mad Catz Controller (unverified)", 0, XTYPE_XBOX },
{ 0x05fd, 0x107a, "InterAct 'PowerPad Pro' X-Box pad (Germany)", 0, XTYPE_XBOX },
+ { 0x05fe, 0x3030, "Chic Controller", 0, XTYPE_XBOX },
+ { 0x05fe, 0x3031, "Chic Controller", 0, XTYPE_XBOX },
+ { 0x062a, 0x0020, "Logic3 Xbox GamePad", 0, XTYPE_XBOX },
+ { 0x062a, 0x0033, "Competition Pro Steering Wheel", 0, XTYPE_XBOX },
+ { 0x06a3, 0x0200, "Saitek Racing Wheel", 0, XTYPE_XBOX },
+ { 0x06a3, 0x0201, "Saitek Adrenalin", 0, XTYPE_XBOX },
+ { 0x06a3, 0xf51a, "Saitek P3600", 0, XTYPE_XBOX360 },
+ { 0x0738, 0x4506, "Mad Catz 4506 Wireless Controller", 0, XTYPE_XBOX },
{ 0x0738, 0x4516, "Mad Catz Control Pad", 0, XTYPE_XBOX },
+ { 0x0738, 0x4520, "Mad Catz Control Pad Pro", 0, XTYPE_XBOX },
{ 0x0738, 0x4522, "Mad Catz LumiCON", 0, XTYPE_XBOX },
{ 0x0738, 0x4526, "Mad Catz Control Pad Pro", 0, XTYPE_XBOX },
+ { 0x0738, 0x4530, "Mad Catz Universal MC2 Racing Wheel and Pedals", 0, XTYPE_XBOX },
{ 0x0738, 0x4536, "Mad Catz MicroCON", 0, XTYPE_XBOX },
{ 0x0738, 0x4540, "Mad Catz Beat Pad", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX },
{ 0x0738, 0x4556, "Mad Catz Lynx Wireless Controller", 0, XTYPE_XBOX },
+ { 0x0738, 0x4586, "Mad Catz MicroCon Wireless Controller", 0, XTYPE_XBOX },
+ { 0x0738, 0x4588, "Mad Catz Blaster", 0, XTYPE_XBOX },
+ { 0x0738, 0x45ff, "Mad Catz Beat Pad (w/ Handle)", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX },
{ 0x0738, 0x4716, "Mad Catz Wired Xbox 360 Controller", 0, XTYPE_XBOX360 },
{ 0x0738, 0x4718, "Mad Catz Street Fighter IV FightStick SE", 0, XTYPE_XBOX360 },
{ 0x0738, 0x4726, "Mad Catz Xbox 360 Controller", 0, XTYPE_XBOX360 },
{ 0x0738, 0x4728, "Mad Catz Street Fighter IV FightPad", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x0738, 0x4736, "Mad Catz MicroCon Gamepad", 0, XTYPE_XBOX360 },
{ 0x0738, 0x4738, "Mad Catz Wired Xbox 360 Controller (SFIV)", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x0738, 0x4740, "Mad Catz Beat Pad", 0, XTYPE_XBOX360 },
+ { 0x0738, 0x4743, "Mad Catz Beat Pad Pro", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX },
+ { 0x0738, 0x4758, "Mad Catz Arcade Game Stick", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x0738, 0x4a01, "Mad Catz FightStick TE 2", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOXONE },
{ 0x0738, 0x6040, "Mad Catz Beat Pad Pro", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX },
+ { 0x0738, 0x9871, "Mad Catz Portable Drum", 0, XTYPE_XBOX360 },
{ 0x0738, 0xb726, "Mad Catz Xbox controller - MW2", 0, XTYPE_XBOX360 },
+ { 0x0738, 0xb738, "Mad Catz MVC2TE Stick 2", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x0738, 0xbeef, "Mad Catz JOYTECH NEO SE Advanced GamePad", XTYPE_XBOX360 },
{ 0x0738, 0xcb02, "Saitek Cyborg Rumble Pad - PC/Xbox 360", 0, XTYPE_XBOX360 },
{ 0x0738, 0xcb03, "Saitek P3200 Rumble Pad - PC/Xbox 360", 0, XTYPE_XBOX360 },
+ { 0x0738, 0xcb29, "Saitek Aviator Stick AV8R02", 0, XTYPE_XBOX360 },
{ 0x0738, 0xf738, "Super SFIV FightStick TE S", 0, XTYPE_XBOX360 },
+ { 0x07ff, 0xffff, "Mad Catz GamePad", 0, XTYPE_XBOX360 },
+ { 0x0c12, 0x0005, "Intec wireless", 0, XTYPE_XBOX },
+ { 0x0c12, 0x8801, "Nyko Xbox Controller", 0, XTYPE_XBOX },
{ 0x0c12, 0x8802, "Zeroplus Xbox Controller", 0, XTYPE_XBOX },
{ 0x0c12, 0x8809, "RedOctane Xbox Dance Pad", DANCEPAD_MAP_CONFIG, XTYPE_XBOX },
{ 0x0c12, 0x880a, "Pelican Eclipse PL-2023", 0, XTYPE_XBOX },
@@ -172,35 +206,66 @@ static const struct xpad_device {
{ 0x0c12, 0x9902, "HAMA VibraX - *FAULTY HARDWARE*", 0, XTYPE_XBOX },
{ 0x0d2f, 0x0002, "Andamiro Pump It Up pad", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX },
{ 0x0e4c, 0x1097, "Radica Gamester Controller", 0, XTYPE_XBOX },
+ { 0x0e4c, 0x1103, "Radica Gamester Reflex", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX },
{ 0x0e4c, 0x2390, "Radica Games Jtech Controller", 0, XTYPE_XBOX },
+ { 0x0e4c, 0x3510, "Radica Gamester", 0, XTYPE_XBOX },
{ 0x0e6f, 0x0003, "Logic3 Freebird wireless Controller", 0, XTYPE_XBOX },
{ 0x0e6f, 0x0005, "Eclipse wireless Controller", 0, XTYPE_XBOX },
{ 0x0e6f, 0x0006, "Edge wireless Controller", 0, XTYPE_XBOX },
+ { 0x0e6f, 0x0008, "After Glow Pro Controller", 0, XTYPE_XBOX },
{ 0x0e6f, 0x0105, "HSM3 Xbox360 dancepad", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x0e6f, 0x0113, "Afterglow AX.1 Gamepad for Xbox 360", 0, XTYPE_XBOX360 },
+ { 0x0e6f, 0x011f, "Rock Candy Gamepad Wired Controller", 0, XTYPE_XBOX360 },
+ { 0x0e6f, 0x0131, "PDP EA Sports Controller", 0, XTYPE_XBOX360 },
+ { 0x0e6f, 0x0133, "Xbox 360 Wired Controller", 0, XTYPE_XBOX360 },
{ 0x0e6f, 0x0139, "Afterglow Prismatic Wired Controller", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x013a, "PDP Xbox One Controller", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x0146, "Rock Candy Wired Controller for Xbox One", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x0147, "PDP Marvel Xbox One Controller", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x015c, "PDP Xbox One Arcade Stick", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOXONE },
+ { 0x0e6f, 0x0161, "PDP Xbox One Controller", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x0162, "PDP Xbox One Controller", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x0163, "PDP Xbox One Controller", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x0164, "PDP Battlefield One", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x0165, "PDP Titanfall 2", 0, XTYPE_XBOXONE },
{ 0x0e6f, 0x0201, "Pelican PL-3601 'TSZ' Wired Xbox 360 Controller", 0, XTYPE_XBOX360 },
{ 0x0e6f, 0x0213, "Afterglow Gamepad for Xbox 360", 0, XTYPE_XBOX360 },
{ 0x0e6f, 0x021f, "Rock Candy Gamepad for Xbox 360", 0, XTYPE_XBOX360 },
- { 0x0e6f, 0x0146, "Rock Candy Wired Controller for Xbox One", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x0246, "Rock Candy Gamepad for Xbox One 2015", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x02ab, "PDP Controller for Xbox One", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x02a4, "PDP Wired Controller for Xbox One - Stealth Series", 0, XTYPE_XBOXONE },
+ { 0x0e6f, 0x02a6, "PDP Wired Controller for Xbox One - Camo Series", 0, XTYPE_XBOXONE },
{ 0x0e6f, 0x0301, "Logic3 Controller", 0, XTYPE_XBOX360 },
+ { 0x0e6f, 0x0346, "Rock Candy Gamepad for Xbox One 2016", 0, XTYPE_XBOXONE },
{ 0x0e6f, 0x0401, "Logic3 Controller", 0, XTYPE_XBOX360 },
+ { 0x0e6f, 0x0413, "Afterglow AX.1 Gamepad for Xbox 360", 0, XTYPE_XBOX360 },
+ { 0x0e6f, 0x0501, "PDP Xbox 360 Controller", 0, XTYPE_XBOX360 },
+ { 0x0e6f, 0xf900, "PDP Afterglow AX.1", 0, XTYPE_XBOX360 },
{ 0x0e8f, 0x0201, "SmartJoy Frag Xpad/PS2 adaptor", 0, XTYPE_XBOX },
{ 0x0e8f, 0x3008, "Generic xbox control (dealextreme)", 0, XTYPE_XBOX },
{ 0x0f0d, 0x000a, "Hori Co. DOA4 FightStick", 0, XTYPE_XBOX360 },
+ { 0x0f0d, 0x000c, "Hori PadEX Turbo", 0, XTYPE_XBOX360 },
{ 0x0f0d, 0x000d, "Hori Fighting Stick EX2", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x0f0d, 0x0016, "Hori Real Arcade Pro.EX", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x0f0d, 0x001b, "Hori Real Arcade Pro VX", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x0f0d, 0x0063, "Hori Real Arcade Pro Hayabusa (USA) Xbox One", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOXONE },
{ 0x0f0d, 0x0067, "HORIPAD ONE", 0, XTYPE_XBOXONE },
+ { 0x0f0d, 0x0078, "Hori Real Arcade Pro V Kai Xbox One", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOXONE },
+ { 0x0f30, 0x010b, "Philips Recoil", 0, XTYPE_XBOX },
{ 0x0f30, 0x0202, "Joytech Advanced Controller", 0, XTYPE_XBOX },
{ 0x0f30, 0x8888, "BigBen XBMiniPad Controller", 0, XTYPE_XBOX },
{ 0x102c, 0xff0c, "Joytech Wireless Advanced Controller", 0, XTYPE_XBOX },
+ { 0x11c9, 0x55f0, "Nacon GC-100XF", 0, XTYPE_XBOX360 },
{ 0x12ab, 0x0004, "Honey Bee Xbox360 dancepad", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x12ab, 0x0301, "PDP AFTERGLOW AX.1", 0, XTYPE_XBOX360 },
+ { 0x12ab, 0x0303, "Mortal Kombat Klassic FightStick", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x12ab, 0x8809, "Xbox DDR dancepad", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX },
{ 0x1430, 0x4748, "RedOctane Guitar Hero X-plorer", 0, XTYPE_XBOX360 },
{ 0x1430, 0x8888, "TX6500+ Dance Pad (first generation)", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX },
+ { 0x1430, 0xf801, "RedOctane Controller", 0, XTYPE_XBOX360 },
{ 0x146b, 0x0601, "BigBen Interactive XBOX 360 Controller", 0, XTYPE_XBOX360 },
{ 0x1532, 0x0037, "Razer Sabertooth", 0, XTYPE_XBOX360 },
+ { 0x1532, 0x0a00, "Razer Atrox Arcade Stick", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOXONE },
{ 0x1532, 0x0a03, "Razer Wildcat", 0, XTYPE_XBOXONE },
{ 0x15e4, 0x3f00, "Power A Mini Pro Elite", 0, XTYPE_XBOX360 },
{ 0x15e4, 0x3f0a, "Xbox Airflo wired controller", 0, XTYPE_XBOX360 },
@@ -208,27 +273,67 @@ static const struct xpad_device {
{ 0x162e, 0xbeef, "Joytech Neo-Se Take2", 0, XTYPE_XBOX360 },
{ 0x1689, 0xfd00, "Razer Onza Tournament Edition", 0, XTYPE_XBOX360 },
{ 0x1689, 0xfd01, "Razer Onza Classic Edition", 0, XTYPE_XBOX360 },
- { 0x24c6, 0x542a, "Xbox ONE spectra", 0, XTYPE_XBOXONE },
- { 0x24c6, 0x5d04, "Razer Sabertooth", 0, XTYPE_XBOX360 },
+ { 0x1689, 0xfe00, "Razer Sabertooth", 0, XTYPE_XBOX360 },
{ 0x1bad, 0x0002, "Harmonix Rock Band Guitar", 0, XTYPE_XBOX360 },
{ 0x1bad, 0x0003, "Harmonix Rock Band Drumkit", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0x0130, "Ion Drum Rocker", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x1bad, 0xf016, "Mad Catz Xbox 360 Controller", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xf018, "Mad Catz Street Fighter IV SE Fighting Stick", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf019, "Mad Catz Brawlstick for Xbox 360", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf021, "Mad Cats Ghost Recon FS GamePad", 0, XTYPE_XBOX360 },
{ 0x1bad, 0xf023, "MLG Pro Circuit Controller (Xbox)", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xf025, "Mad Catz Call Of Duty", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xf027, "Mad Catz FPS Pro", 0, XTYPE_XBOX360 },
{ 0x1bad, 0xf028, "Street Fighter IV FightPad", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xf02e, "Mad Catz Fightpad", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf030, "Mad Catz Xbox 360 MC2 MicroCon Racing Wheel", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xf036, "Mad Catz MicroCon GamePad Pro", 0, XTYPE_XBOX360 },
{ 0x1bad, 0xf038, "Street Fighter IV FightStick TE", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xf039, "Mad Catz MvC2 TE", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf03a, "Mad Catz SFxT Fightstick Pro", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf03d, "Street Fighter IV Arcade Stick TE - Chun Li", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf03e, "Mad Catz MLG FightStick TE", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf03f, "Mad Catz FightStick SoulCaliber", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf042, "Mad Catz FightStick TES+", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf080, "Mad Catz FightStick TE2", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf501, "HoriPad EX2 Turbo", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xf502, "Hori Real Arcade Pro.VX SA", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf503, "Hori Fighting Stick VX", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf504, "Hori Real Arcade Pro. EX", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf505, "Hori Fighting Stick EX2B", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xf506, "Hori Real Arcade Pro.EX Premium VLX", 0, XTYPE_XBOX360 },
{ 0x1bad, 0xf900, "Harmonix Xbox 360 Controller", 0, XTYPE_XBOX360 },
{ 0x1bad, 0xf901, "Gamestop Xbox 360 Controller", 0, XTYPE_XBOX360 },
{ 0x1bad, 0xf903, "Tron Xbox 360 controller", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xf904, "PDP Versus Fighting Pad", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xf906, "MortalKombat FightStick", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x1bad, 0xfa01, "MadCatz GamePad", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xfd00, "Razer Onza TE", 0, XTYPE_XBOX360 },
+ { 0x1bad, 0xfd01, "Razer Onza", 0, XTYPE_XBOX360 },
{ 0x24c6, 0x5000, "Razer Atrox Arcade Stick", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x24c6, 0x5300, "PowerA MINI PROEX Controller", 0, XTYPE_XBOX360 },
{ 0x24c6, 0x5303, "Xbox Airflo wired controller", 0, XTYPE_XBOX360 },
+ { 0x24c6, 0x530a, "Xbox 360 Pro EX Controller", 0, XTYPE_XBOX360 },
+ { 0x24c6, 0x531a, "PowerA Pro Ex", 0, XTYPE_XBOX360 },
+ { 0x24c6, 0x5397, "FUS1ON Tournament Controller", 0, XTYPE_XBOX360 },
{ 0x24c6, 0x541a, "PowerA Xbox One Mini Wired Controller", 0, XTYPE_XBOXONE },
+ { 0x24c6, 0x542a, "Xbox ONE spectra", 0, XTYPE_XBOXONE },
{ 0x24c6, 0x543a, "PowerA Xbox One wired controller", 0, XTYPE_XBOXONE },
{ 0x24c6, 0x5500, "Hori XBOX 360 EX 2 with Turbo", 0, XTYPE_XBOX360 },
{ 0x24c6, 0x5501, "Hori Real Arcade Pro VX-SA", 0, XTYPE_XBOX360 },
+ { 0x24c6, 0x5502, "Hori Fighting Stick VX Alt", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x24c6, 0x5503, "Hori Fighting Edge", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
{ 0x24c6, 0x5506, "Hori SOULCALIBUR V Stick", 0, XTYPE_XBOX360 },
+ { 0x24c6, 0x550d, "Hori GEM Xbox controller", 0, XTYPE_XBOX360 },
+ { 0x24c6, 0x550e, "Hori Real Arcade Pro V Kai 360", MAP_TRIGGERS_TO_BUTTONS, XTYPE_XBOX360 },
+ { 0x24c6, 0x551a, "PowerA FUSION Pro Controller", 0, XTYPE_XBOXONE },
+ { 0x24c6, 0x561a, "PowerA FUSION Controller", 0, XTYPE_XBOXONE },
+ { 0x24c6, 0x5b00, "ThrustMaster Ferrari 458 Racing Wheel", 0, XTYPE_XBOX360 },
{ 0x24c6, 0x5b02, "Thrustmaster, Inc. GPX Controller", 0, XTYPE_XBOX360 },
{ 0x24c6, 0x5b03, "Thrustmaster Ferrari 458 Racing Wheel", 0, XTYPE_XBOX360 },
+ { 0x24c6, 0x5d04, "Razer Sabertooth", 0, XTYPE_XBOX360 },
+ { 0x24c6, 0xfafe, "Rock Candy Gamepad for Xbox 360", 0, XTYPE_XBOX360 },
+ { 0x3767, 0x0101, "Fanatec Speedster 3 Forceshock Wheel", 0, XTYPE_XBOX },
{ 0xffff, 0xffff, "Chinese-made Xbox Controller", 0, XTYPE_XBOX },
{ 0x0000, 0x0000, "Generic X-Box pad", 0, XTYPE_UNKNOWN }
};
@@ -289,15 +394,15 @@ static const signed short xpad_abs_triggers[] = {
* match against vendor id as well. Wired Xbox 360 devices have protocol 1,
* wireless controllers have protocol 129.
*/
-#define XPAD_XBOX360_VENDOR_PROTOCOL(vend,pr) \
+#define XPAD_XBOX360_VENDOR_PROTOCOL(vend, pr) \
.match_flags = USB_DEVICE_ID_MATCH_VENDOR | USB_DEVICE_ID_MATCH_INT_INFO, \
.idVendor = (vend), \
.bInterfaceClass = USB_CLASS_VENDOR_SPEC, \
.bInterfaceSubClass = 93, \
.bInterfaceProtocol = (pr)
#define XPAD_XBOX360_VENDOR(vend) \
- { XPAD_XBOX360_VENDOR_PROTOCOL(vend,1) }, \
- { XPAD_XBOX360_VENDOR_PROTOCOL(vend,129) }
+ { XPAD_XBOX360_VENDOR_PROTOCOL((vend), 1) }, \
+ { XPAD_XBOX360_VENDOR_PROTOCOL((vend), 129) }
/* The Xbox One controller uses subclass 71 and protocol 208. */
#define XPAD_XBOXONE_VENDOR_PROTOCOL(vend, pr) \
@@ -307,37 +412,134 @@ static const signed short xpad_abs_triggers[] = {
.bInterfaceSubClass = 71, \
.bInterfaceProtocol = (pr)
#define XPAD_XBOXONE_VENDOR(vend) \
- { XPAD_XBOXONE_VENDOR_PROTOCOL(vend, 208) }
+ { XPAD_XBOXONE_VENDOR_PROTOCOL((vend), 208) }
-static struct usb_device_id xpad_table[] = {
+static const struct usb_device_id xpad_table[] = {
{ USB_INTERFACE_INFO('X', 'B', 0) }, /* X-Box USB-IF not approved class */
+ XPAD_XBOX360_VENDOR(0x0079), /* GPD Win 2 Controller */
XPAD_XBOX360_VENDOR(0x044f), /* Thrustmaster X-Box 360 controllers */
XPAD_XBOX360_VENDOR(0x045e), /* Microsoft X-Box 360 controllers */
XPAD_XBOXONE_VENDOR(0x045e), /* Microsoft X-Box One controllers */
XPAD_XBOX360_VENDOR(0x046d), /* Logitech X-Box 360 style controllers */
+ XPAD_XBOX360_VENDOR(0x056e), /* Elecom JC-U3613M */
+ XPAD_XBOX360_VENDOR(0x06a3), /* Saitek P3600 */
XPAD_XBOX360_VENDOR(0x0738), /* Mad Catz X-Box 360 controllers */
{ USB_DEVICE(0x0738, 0x4540) }, /* Mad Catz Beat Pad */
XPAD_XBOXONE_VENDOR(0x0738), /* Mad Catz FightStick TE 2 */
+ XPAD_XBOX360_VENDOR(0x07ff), /* Mad Catz GamePad */
XPAD_XBOX360_VENDOR(0x0e6f), /* 0x0e6f X-Box 360 controllers */
XPAD_XBOXONE_VENDOR(0x0e6f), /* 0x0e6f X-Box One controllers */
+ XPAD_XBOX360_VENDOR(0x0f0d), /* Hori Controllers */
+ XPAD_XBOXONE_VENDOR(0x0f0d), /* Hori Controllers */
+ XPAD_XBOX360_VENDOR(0x11c9), /* Nacon GC100XF */
XPAD_XBOX360_VENDOR(0x12ab), /* X-Box 360 dance pads */
XPAD_XBOX360_VENDOR(0x1430), /* RedOctane X-Box 360 controllers */
XPAD_XBOX360_VENDOR(0x146b), /* BigBen Interactive Controllers */
- XPAD_XBOX360_VENDOR(0x1bad), /* Harminix Rock Band Guitar and Drums */
- XPAD_XBOX360_VENDOR(0x0f0d), /* Hori Controllers */
- XPAD_XBOXONE_VENDOR(0x0f0d), /* Hori Controllers */
- XPAD_XBOX360_VENDOR(0x1689), /* Razer Onza */
- XPAD_XBOX360_VENDOR(0x24c6), /* PowerA Controllers */
- XPAD_XBOXONE_VENDOR(0x24c6), /* PowerA Controllers */
XPAD_XBOX360_VENDOR(0x1532), /* Razer Sabertooth */
XPAD_XBOXONE_VENDOR(0x1532), /* Razer Wildcat */
XPAD_XBOX360_VENDOR(0x15e4), /* Numark X-Box 360 controllers */
XPAD_XBOX360_VENDOR(0x162e), /* Joytech X-Box 360 controllers */
+ XPAD_XBOX360_VENDOR(0x1689), /* Razer Onza */
+ XPAD_XBOX360_VENDOR(0x1bad), /* Harminix Rock Band Guitar and Drums */
+ XPAD_XBOX360_VENDOR(0x24c6), /* PowerA Controllers */
+ XPAD_XBOXONE_VENDOR(0x24c6), /* PowerA Controllers */
{ }
};
MODULE_DEVICE_TABLE(usb, xpad_table);
+struct xboxone_init_packet {
+ u16 idVendor;
+ u16 idProduct;
+ const u8 *data;
+ u8 len;
+};
+
+#define XBOXONE_INIT_PKT(_vid, _pid, _data) \
+ { \
+ .idVendor = (_vid), \
+ .idProduct = (_pid), \
+ .data = (_data), \
+ .len = ARRAY_SIZE(_data), \
+ }
+
+
+/*
+ * This packet is required for all Xbox One pads with 2015
+ * or later firmware installed (or present from the factory).
+ */
+static const u8 xboxone_fw2015_init[] = {
+ 0x05, 0x20, 0x00, 0x01, 0x00
+};
+
+/*
+ * This packet is required for the Titanfall 2 Xbox One pads
+ * (0x0e6f:0x0165) to finish initialization and for Hori pads
+ * (0x0f0d:0x0067) to make the analog sticks work.
+ */
+static const u8 xboxone_hori_init[] = {
+ 0x01, 0x20, 0x00, 0x09, 0x00, 0x04, 0x20, 0x3a,
+ 0x00, 0x00, 0x00, 0x80, 0x00
+};
+
+/*
+ * This packet is required for most (all?) of the PDP pads to start
+ * sending input reports. These pads include: (0x0e6f:0x02ab),
+ * (0x0e6f:0x02a4), (0x0e6f:0x02a6).
+ */
+static const u8 xboxone_pdp_init1[] = {
+ 0x0a, 0x20, 0x00, 0x03, 0x00, 0x01, 0x14
+};
+
+/*
+ * This packet is required for most (all?) of the PDP pads to start
+ * sending input reports. These pads include: (0x0e6f:0x02ab),
+ * (0x0e6f:0x02a4), (0x0e6f:0x02a6).
+ */
+static const u8 xboxone_pdp_init2[] = {
+ 0x06, 0x20, 0x00, 0x02, 0x01, 0x00
+};
+
+/*
+ * A specific rumble packet is required for some PowerA pads to start
+ * sending input reports. One of those pads is (0x24c6:0x543a).
+ */
+static const u8 xboxone_rumblebegin_init[] = {
+ 0x09, 0x00, 0x00, 0x09, 0x00, 0x0F, 0x00, 0x00,
+ 0x1D, 0x1D, 0xFF, 0x00, 0x00
+};
+
+/*
+ * A rumble packet with zero FF intensity will immediately
+ * terminate the rumbling required to init PowerA pads.
+ * This should happen fast enough that the motors don't
+ * spin up to enough speed to actually vibrate the gamepad.
+ */
+static const u8 xboxone_rumbleend_init[] = {
+ 0x09, 0x00, 0x00, 0x09, 0x00, 0x0F, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00
+};
+
+/*
+ * This specifies the selection of init packets that a gamepad
+ * will be sent on init *and* the order in which they will be
+ * sent. The correct sequence number will be added when the
+ * packet is going to be sent.
+ */
+static const struct xboxone_init_packet xboxone_init_packets[] = {
+ XBOXONE_INIT_PKT(0x0e6f, 0x0165, xboxone_hori_init),
+ XBOXONE_INIT_PKT(0x0f0d, 0x0067, xboxone_hori_init),
+ XBOXONE_INIT_PKT(0x0000, 0x0000, xboxone_fw2015_init),
+ XBOXONE_INIT_PKT(0x0e6f, 0x0000, xboxone_pdp_init1),
+ XBOXONE_INIT_PKT(0x0e6f, 0x0000, xboxone_pdp_init2),
+ XBOXONE_INIT_PKT(0x24c6, 0x541a, xboxone_rumblebegin_init),
+ XBOXONE_INIT_PKT(0x24c6, 0x542a, xboxone_rumblebegin_init),
+ XBOXONE_INIT_PKT(0x24c6, 0x543a, xboxone_rumblebegin_init),
+ XBOXONE_INIT_PKT(0x24c6, 0x541a, xboxone_rumbleend_init),
+ XBOXONE_INIT_PKT(0x24c6, 0x542a, xboxone_rumbleend_init),
+ XBOXONE_INIT_PKT(0x24c6, 0x543a, xboxone_rumbleend_init),
+};
+
struct xpad_output_packet {
u8 data[XPAD_PKT_LEN];
u8 len;
@@ -374,6 +576,7 @@ struct usb_xpad {
struct xpad_output_packet out_packets[XPAD_NUM_OUT_PACKETS];
int last_out_packet;
+ int init_seq;
#if defined(CONFIG_JOYSTICK_XPAD_LEDS)
struct xpad_led *led;
@@ -390,6 +593,7 @@ struct usb_xpad {
static int xpad_init_input(struct usb_xpad *xpad);
static void xpad_deinit_input(struct usb_xpad *xpad);
+static void xpadone_ack_mode_report(struct usb_xpad *xpad, u8 seq_num);
/*
* xpad_process_packet
@@ -609,14 +813,36 @@ static void xpad360w_process_packet(struct usb_xpad *xpad, u16 cmd, unsigned cha
}
/*
- * xpadone_process_buttons
+ * xpadone_process_packet
*
- * Process a button update packet from an Xbox one controller.
+ * Completes a request by converting the data into events for the
+ * input subsystem. This version is for the Xbox One controller.
+ *
+ * The report format was gleaned from
+ * https://github.com/kylelemons/xbox/blob/master/xbox.go
*/
-static void xpadone_process_buttons(struct usb_xpad *xpad,
- struct input_dev *dev,
- unsigned char *data)
+static void xpadone_process_packet(struct usb_xpad *xpad, u16 cmd, unsigned char *data)
{
+ struct input_dev *dev = xpad->dev;
+
+ /* the xbox button has its own special report */
+ if (data[0] == 0X07) {
+ /*
+ * The Xbox One S controller requires these reports to be
+ * acked otherwise it continues sending them forever and
+ * won't report further mode button events.
+ */
+ if (data[1] == 0x30)
+ xpadone_ack_mode_report(xpad, data[2]);
+
+ input_report_key(dev, BTN_MODE, data[4] & 0x01);
+ input_sync(dev);
+ return;
+ }
+ /* check invalid packet */
+ else if (data[0] != 0X20)
+ return;
+
/* menu/view buttons */
input_report_key(dev, BTN_START, data[4] & 0x04);
input_report_key(dev, BTN_SELECT, data[4] & 0x08);
@@ -679,34 +905,6 @@ static void xpadone_process_buttons(struct usb_xpad *xpad,
input_sync(dev);
}
-/*
- * xpadone_process_packet
- *
- * Completes a request by converting the data into events for the
- * input subsystem. This version is for the Xbox One controller.
- *
- * The report format was gleaned from
- * https://github.com/kylelemons/xbox/blob/master/xbox.go
- */
-
-static void xpadone_process_packet(struct usb_xpad *xpad,
- u16 cmd, unsigned char *data)
-{
- struct input_dev *dev = xpad->dev;
-
- switch (data[0]) {
- case 0x20:
- xpadone_process_buttons(xpad, dev, data);
- break;
-
- case 0x07:
- /* the xbox button has its own special report */
- input_report_key(dev, BTN_MODE, data[4] & 0x01);
- input_sync(dev);
- break;
- }
-}
-
static void xpad_irq_in(struct urb *urb)
{
struct usb_xpad *xpad = urb->context;
@@ -754,11 +952,47 @@ static void xpad_irq_in(struct urb *urb)
}
/* Callers must hold xpad->odata_lock spinlock */
+static bool xpad_prepare_next_init_packet(struct usb_xpad *xpad)
+{
+ const struct xboxone_init_packet *init_packet;
+
+ if (xpad->xtype != XTYPE_XBOXONE)
+ return false;
+
+ /* Perform initialization sequence for Xbox One pads that require it */
+ while (xpad->init_seq < ARRAY_SIZE(xboxone_init_packets)) {
+ init_packet = &xboxone_init_packets[xpad->init_seq++];
+
+ if (init_packet->idVendor != 0 &&
+ init_packet->idVendor != xpad->dev->id.vendor)
+ continue;
+
+ if (init_packet->idProduct != 0 &&
+ init_packet->idProduct != xpad->dev->id.product)
+ continue;
+
+ /* This packet applies to our device, so prepare to send it */
+ memcpy(xpad->odata, init_packet->data, init_packet->len);
+ xpad->irq_out->transfer_buffer_length = init_packet->len;
+
+ /* Update packet with current sequence number */
+ xpad->odata[2] = xpad->odata_serial++;
+ return true;
+ }
+
+ return false;
+}
+
+/* Callers must hold xpad->odata_lock spinlock */
static bool xpad_prepare_next_out_packet(struct usb_xpad *xpad)
{
struct xpad_output_packet *pkt, *packet = NULL;
int i;
+ /* We may have init packets to send before we can send user commands */
+ if (xpad_prepare_next_init_packet(xpad))
+ return true;
+
for (i = 0; i < XPAD_NUM_OUT_PACKETS; i++) {
if (++xpad->last_out_packet >= XPAD_NUM_OUT_PACKETS)
xpad->last_out_packet = 0;
@@ -851,10 +1085,9 @@ static void xpad_irq_out(struct urb *urb)
spin_unlock_irqrestore(&xpad->odata_lock, flags);
}
-static int xpad_init_output(struct usb_interface *intf, struct usb_xpad *xpad)
+static int xpad_init_output(struct usb_interface *intf, struct usb_xpad *xpad,
+ struct usb_endpoint_descriptor *ep_irq_out)
{
- struct usb_endpoint_descriptor *ep_irq_out;
- int ep_irq_out_idx;
int error;
if (xpad->xtype == XTYPE_UNKNOWN)
@@ -864,23 +1097,17 @@ static int xpad_init_output(struct usb_interface *intf, struct usb_xpad *xpad)
xpad->odata = usb_alloc_coherent(xpad->udev, XPAD_PKT_LEN,
GFP_KERNEL, &xpad->odata_dma);
- if (!xpad->odata) {
- error = -ENOMEM;
- goto fail1;
- }
+ if (!xpad->odata)
+ return -ENOMEM;
spin_lock_init(&xpad->odata_lock);
xpad->irq_out = usb_alloc_urb(0, GFP_KERNEL);
if (!xpad->irq_out) {
error = -ENOMEM;
- goto fail2;
+ goto err_free_coherent;
}
- /* Xbox One controller has in/out endpoints swapped. */
- ep_irq_out_idx = xpad->xtype == XTYPE_XBOXONE ? 0 : 1;
- ep_irq_out = &intf->cur_altsetting->endpoint[ep_irq_out_idx].desc;
-
usb_fill_int_urb(xpad->irq_out, xpad->udev,
usb_sndintpipe(xpad->udev, ep_irq_out->bEndpointAddress),
xpad->odata, XPAD_PKT_LEN,
@@ -890,8 +1117,9 @@ static int xpad_init_output(struct usb_interface *intf, struct usb_xpad *xpad)
return 0;
- fail2: usb_free_coherent(xpad->udev, XPAD_PKT_LEN, xpad->odata, xpad->odata_dma);
- fail1: return error;
+err_free_coherent:
+ usb_free_coherent(xpad->udev, XPAD_PKT_LEN, xpad->odata, xpad->odata_dma);
+ return error;
}
static void xpad_stop_output(struct usb_xpad *xpad)
@@ -950,24 +1178,17 @@ static int xpad_inquiry_pad_presence(struct usb_xpad *xpad)
static int xpad_start_xbox_one(struct usb_xpad *xpad)
{
- struct xpad_output_packet *packet =
- &xpad->out_packets[XPAD_OUT_CMD_IDX];
unsigned long flags;
int retval;
spin_lock_irqsave(&xpad->odata_lock, flags);
- /* Xbox one controller needs to be initialized. */
- packet->data[0] = 0x05;
- packet->data[1] = 0x20;
- packet->data[2] = xpad->odata_serial++; /* packet serial */
- packet->data[3] = 0x01; /* rumble bit enable? */
- packet->data[4] = 0x00;
- packet->len = 5;
- packet->pending = true;
-
- /* Reset the sequence so we send out start packet first */
- xpad->last_out_packet = -1;
+ /*
+ * Begin the init sequence by attempting to send a packet.
+ * We will cycle through the init packet sequence before
+ * sending any packets from the output ring.
+ */
+ xpad->init_seq = 0;
retval = xpad_try_sending_next_out_packet(xpad);
spin_unlock_irqrestore(&xpad->odata_lock, flags);
@@ -975,6 +1196,30 @@ static int xpad_start_xbox_one(struct usb_xpad *xpad)
return retval;
}
+static void xpadone_ack_mode_report(struct usb_xpad *xpad, u8 seq_num)
+{
+ unsigned long flags;
+ struct xpad_output_packet *packet =
+ &xpad->out_packets[XPAD_OUT_CMD_IDX];
+ static const u8 mode_report_ack[] = {
+ 0x01, 0x20, 0x00, 0x09, 0x00, 0x07, 0x20, 0x02,
+ 0x00, 0x00, 0x00, 0x00, 0x00
+ };
+
+ spin_lock_irqsave(&xpad->odata_lock, flags);
+
+ packet->len = sizeof(mode_report_ack);
+ memcpy(packet->data, mode_report_ack, packet->len);
+ packet->data[2] = seq_num;
+ packet->pending = true;
+
+ /* Reset the sequence so we send out the ack now */
+ xpad->last_out_packet = -1;
+ xpad_try_sending_next_out_packet(xpad);
+
+ spin_unlock_irqrestore(&xpad->odata_lock, flags);
+}
+
#ifdef CONFIG_JOYSTICK_XPAD_FF
static int xpad_play_effect(struct input_dev *dev, void *data, struct ff_effect *effect)
{
@@ -1046,9 +1291,9 @@ static int xpad_play_effect(struct input_dev *dev, void *data, struct ff_effect
packet->data[7] = 0x00;
packet->data[8] = strong / 512; /* left actuator */
packet->data[9] = weak / 512; /* right actuator */
- packet->data[10] = 0xFF;
- packet->data[11] = 0x00;
- packet->data[12] = 0x00;
+ packet->data[10] = 0xFF; /* on period */
+ packet->data[11] = 0x00; /* off period */
+ packet->data[12] = 0xFF; /* repeat count */
packet->len = 13;
packet->pending = true;
break;
@@ -1199,6 +1444,7 @@ static int xpad_led_probe(struct usb_xpad *xpad)
led_cdev = &led->led_cdev;
led_cdev->name = led->name;
led_cdev->brightness_set = xpad_led_set;
+ led_cdev->flags = LED_CORE_SUSPENDRESUME;
error = led_classdev_register(&xpad->udev->dev, led_cdev);
if (error)
@@ -1333,7 +1579,6 @@ static void xpad_close(struct input_dev *dev)
static void xpad_set_up_abs(struct input_dev *input_dev, signed short abs)
{
struct usb_xpad *xpad = input_get_drvdata(input_dev);
- set_bit(abs, input_dev->absbit);
switch (abs) {
case ABS_X:
@@ -1353,6 +1598,9 @@ static void xpad_set_up_abs(struct input_dev *input_dev, signed short abs)
case ABS_HAT0Y: /* the d-pad (only if dpad is mapped to axes */
input_set_abs_params(input_dev, abs, -1, 1, 0, 0);
break;
+ default:
+ input_set_abs_params(input_dev, abs, 0, 0, 0, 0);
+ break;
}
}
@@ -1393,10 +1641,7 @@ static int xpad_init_input(struct usb_xpad *xpad)
input_dev->close = xpad_close;
}
- __set_bit(EV_KEY, input_dev->evbit);
-
if (!(xpad->mapping & MAP_STICKS_TO_NULL)) {
- __set_bit(EV_ABS, input_dev->evbit);
/* set up axes */
for (i = 0; xpad_abs[i] >= 0; i++)
xpad_set_up_abs(input_dev, xpad_abs[i]);
@@ -1404,21 +1649,22 @@ static int xpad_init_input(struct usb_xpad *xpad)
/* set up standard buttons */
for (i = 0; xpad_common_btn[i] >= 0; i++)
- __set_bit(xpad_common_btn[i], input_dev->keybit);
+ input_set_capability(input_dev, EV_KEY, xpad_common_btn[i]);
/* set up model-specific ones */
if (xpad->xtype == XTYPE_XBOX360 || xpad->xtype == XTYPE_XBOX360W ||
xpad->xtype == XTYPE_XBOXONE) {
for (i = 0; xpad360_btn[i] >= 0; i++)
- __set_bit(xpad360_btn[i], input_dev->keybit);
+ input_set_capability(input_dev, EV_KEY, xpad360_btn[i]);
} else {
for (i = 0; xpad_btn[i] >= 0; i++)
- __set_bit(xpad_btn[i], input_dev->keybit);
+ input_set_capability(input_dev, EV_KEY, xpad_btn[i]);
}
if (xpad->mapping & MAP_DPAD_TO_BUTTONS) {
for (i = 0; xpad_btn_pad[i] >= 0; i++)
- __set_bit(xpad_btn_pad[i], input_dev->keybit);
+ input_set_capability(input_dev, EV_KEY,
+ xpad_btn_pad[i]);
}
/*
@@ -1435,7 +1681,8 @@ static int xpad_init_input(struct usb_xpad *xpad)
if (xpad->mapping & MAP_TRIGGERS_TO_BUTTONS) {
for (i = 0; xpad_btn_triggers[i] >= 0; i++)
- __set_bit(xpad_btn_triggers[i], input_dev->keybit);
+ input_set_capability(input_dev, EV_KEY,
+ xpad_btn_triggers[i]);
} else {
for (i = 0; xpad_abs_triggers[i] >= 0; i++)
xpad_set_up_abs(input_dev, xpad_abs_triggers[i]);
@@ -1469,8 +1716,7 @@ static int xpad_probe(struct usb_interface *intf, const struct usb_device_id *id
{
struct usb_device *udev = interface_to_usbdev(intf);
struct usb_xpad *xpad;
- struct usb_endpoint_descriptor *ep_irq_in;
- int ep_irq_in_idx;
+ struct usb_endpoint_descriptor *ep_irq_in, *ep_irq_out;
int i, error;
if (intf->cur_altsetting->desc.bNumEndpoints != 2)
@@ -1540,14 +1786,29 @@ static int xpad_probe(struct usb_interface *intf, const struct usb_device_id *id
goto err_free_in_urb;
}
- error = xpad_init_output(intf, xpad);
+ ep_irq_in = ep_irq_out = NULL;
+
+ for (i = 0; i < 2; i++) {
+ struct usb_endpoint_descriptor *ep =
+ &intf->cur_altsetting->endpoint[i].desc;
+
+ if (usb_endpoint_xfer_int(ep)) {
+ if (usb_endpoint_dir_in(ep))
+ ep_irq_in = ep;
+ else
+ ep_irq_out = ep;
+ }
+ }
+
+ if (!ep_irq_in || !ep_irq_out) {
+ error = -ENODEV;
+ goto err_free_in_urb;
+ }
+
+ error = xpad_init_output(intf, xpad, ep_irq_out);
if (error)
goto err_free_in_urb;
- /* Xbox One controller has in/out endpoints swapped. */
- ep_irq_in_idx = xpad->xtype == XTYPE_XBOXONE ? 1 : 0;
- ep_irq_in = &intf->cur_altsetting->endpoint[ep_irq_in_idx].desc;
-
usb_fill_int_urb(xpad->irq_in, udev,
usb_rcvintpipe(udev, ep_irq_in->bEndpointAddress),
xpad->idata, XPAD_PKT_LEN, xpad_irq_in,
@@ -1663,8 +1924,16 @@ static int xpad_resume(struct usb_interface *intf)
retval = xpad360w_start_input(xpad);
} else {
mutex_lock(&input->mutex);
- if (input->users)
+ if (input->users) {
retval = xpad_start_input(xpad);
+ } else if (xpad->xtype == XTYPE_XBOXONE) {
+ /*
+ * Even if there are no users, we'll send Xbox One pads
+ * the startup sequence so they don't sit there and
+ * blink until somebody opens the input device again.
+ */
+ retval = xpad_start_xbox_one(xpad);
+ }
mutex_unlock(&input->mutex);
}
diff --git a/drivers/input/keyboard/atakbd.c b/drivers/input/keyboard/atakbd.c
index f123583..fdeda0b 100644
--- a/drivers/input/keyboard/atakbd.c
+++ b/drivers/input/keyboard/atakbd.c
@@ -79,8 +79,7 @@ MODULE_LICENSE("GPL");
*/
-static unsigned char atakbd_keycode[0x72] = { /* American layout */
- [0] = KEY_GRAVE,
+static unsigned char atakbd_keycode[0x73] = { /* American layout */
[1] = KEY_ESC,
[2] = KEY_1,
[3] = KEY_2,
@@ -121,9 +120,9 @@ static unsigned char atakbd_keycode[0x72] = { /* American layout */
[38] = KEY_L,
[39] = KEY_SEMICOLON,
[40] = KEY_APOSTROPHE,
- [41] = KEY_BACKSLASH, /* FIXME, '#' */
+ [41] = KEY_GRAVE,
[42] = KEY_LEFTSHIFT,
- [43] = KEY_GRAVE, /* FIXME: '~' */
+ [43] = KEY_BACKSLASH,
[44] = KEY_Z,
[45] = KEY_X,
[46] = KEY_C,
@@ -149,45 +148,34 @@ static unsigned char atakbd_keycode[0x72] = { /* American layout */
[66] = KEY_F8,
[67] = KEY_F9,
[68] = KEY_F10,
- [69] = KEY_ESC,
- [70] = KEY_DELETE,
- [71] = KEY_KP7,
- [72] = KEY_KP8,
- [73] = KEY_KP9,
+ [71] = KEY_HOME,
+ [72] = KEY_UP,
[74] = KEY_KPMINUS,
- [75] = KEY_KP4,
- [76] = KEY_KP5,
- [77] = KEY_KP6,
+ [75] = KEY_LEFT,
+ [77] = KEY_RIGHT,
[78] = KEY_KPPLUS,
- [79] = KEY_KP1,
- [80] = KEY_KP2,
- [81] = KEY_KP3,
- [82] = KEY_KP0,
- [83] = KEY_KPDOT,
- [90] = KEY_KPLEFTPAREN,
- [91] = KEY_KPRIGHTPAREN,
- [92] = KEY_KPASTERISK, /* FIXME */
- [93] = KEY_KPASTERISK,
- [94] = KEY_KPPLUS,
- [95] = KEY_HELP,
+ [80] = KEY_DOWN,
+ [82] = KEY_INSERT,
+ [83] = KEY_DELETE,
[96] = KEY_102ND,
- [97] = KEY_KPASTERISK, /* FIXME */
- [98] = KEY_KPSLASH,
+ [97] = KEY_UNDO,
+ [98] = KEY_HELP,
[99] = KEY_KPLEFTPAREN,
[100] = KEY_KPRIGHTPAREN,
[101] = KEY_KPSLASH,
[102] = KEY_KPASTERISK,
- [103] = KEY_UP,
- [104] = KEY_KPASTERISK, /* FIXME */
- [105] = KEY_LEFT,
- [106] = KEY_RIGHT,
- [107] = KEY_KPASTERISK, /* FIXME */
- [108] = KEY_DOWN,
- [109] = KEY_KPASTERISK, /* FIXME */
- [110] = KEY_KPASTERISK, /* FIXME */
- [111] = KEY_KPASTERISK, /* FIXME */
- [112] = KEY_KPASTERISK, /* FIXME */
- [113] = KEY_KPASTERISK /* FIXME */
+ [103] = KEY_KP7,
+ [104] = KEY_KP8,
+ [105] = KEY_KP9,
+ [106] = KEY_KP4,
+ [107] = KEY_KP5,
+ [108] = KEY_KP6,
+ [109] = KEY_KP1,
+ [110] = KEY_KP2,
+ [111] = KEY_KP3,
+ [112] = KEY_KP0,
+ [113] = KEY_KPDOT,
+ [114] = KEY_KPENTER,
};
static struct input_dev *atakbd_dev;
@@ -195,21 +183,15 @@ static struct input_dev *atakbd_dev;
static void atakbd_interrupt(unsigned char scancode, char down)
{
- if (scancode < 0x72) { /* scancodes < 0xf2 are keys */
+ if (scancode < 0x73) { /* scancodes < 0xf3 are keys */
// report raw events here?
scancode = atakbd_keycode[scancode];
- if (scancode == KEY_CAPSLOCK) { /* CapsLock is a toggle switch key on Amiga */
- input_report_key(atakbd_dev, scancode, 1);
- input_report_key(atakbd_dev, scancode, 0);
- input_sync(atakbd_dev);
- } else {
- input_report_key(atakbd_dev, scancode, down);
- input_sync(atakbd_dev);
- }
- } else /* scancodes >= 0xf2 are mouse data, most likely */
+ input_report_key(atakbd_dev, scancode, down);
+ input_sync(atakbd_dev);
+ } else /* scancodes >= 0xf3 are mouse data, most likely */
printk(KERN_INFO "atakbd: unhandled scancode %x\n", scancode);
return;
diff --git a/drivers/input/keyboard/matrix_keypad.c b/drivers/input/keyboard/matrix_keypad.c
index 795fa35..c64d874 100644
--- a/drivers/input/keyboard/matrix_keypad.c
+++ b/drivers/input/keyboard/matrix_keypad.c
@@ -405,7 +405,7 @@ matrix_keypad_parse_dt(struct device *dev)
struct matrix_keypad_platform_data *pdata;
struct device_node *np = dev->of_node;
unsigned int *gpios;
- int i, nrow, ncol;
+ int ret, i, nrow, ncol;
if (!np) {
dev_err(dev, "device lacks DT data\n");
@@ -447,12 +447,19 @@ matrix_keypad_parse_dt(struct device *dev)
return ERR_PTR(-ENOMEM);
}
- for (i = 0; i < pdata->num_row_gpios; i++)
- gpios[i] = of_get_named_gpio(np, "row-gpios", i);
+ for (i = 0; i < nrow; i++) {
+ ret = of_get_named_gpio(np, "row-gpios", i);
+ if (ret < 0)
+ return ERR_PTR(ret);
+ gpios[i] = ret;
+ }
- for (i = 0; i < pdata->num_col_gpios; i++)
- gpios[pdata->num_row_gpios + i] =
- of_get_named_gpio(np, "col-gpios", i);
+ for (i = 0; i < ncol; i++) {
+ ret = of_get_named_gpio(np, "col-gpios", i);
+ if (ret < 0)
+ return ERR_PTR(ret);
+ gpios[nrow + i] = ret;
+ }
pdata->row_gpios = gpios;
pdata->col_gpios = &gpios[pdata->num_row_gpios];
@@ -479,10 +486,8 @@ static int matrix_keypad_probe(struct platform_device *pdev)
pdata = dev_get_platdata(&pdev->dev);
if (!pdata) {
pdata = matrix_keypad_parse_dt(&pdev->dev);
- if (IS_ERR(pdata)) {
- dev_err(&pdev->dev, "no platform data defined\n");
+ if (IS_ERR(pdata))
return PTR_ERR(pdata);
- }
} else if (!pdata->keymap_data) {
dev_err(&pdev->dev, "no keymap data defined\n");
return -EINVAL;
diff --git a/drivers/input/keyboard/omap4-keypad.c b/drivers/input/keyboard/omap4-keypad.c
index 6639b2b..3d2c60c 100644
--- a/drivers/input/keyboard/omap4-keypad.c
+++ b/drivers/input/keyboard/omap4-keypad.c
@@ -60,8 +60,18 @@
/* OMAP4 values */
#define OMAP4_VAL_IRQDISABLE 0x0
-#define OMAP4_VAL_DEBOUNCINGTIME 0x7
-#define OMAP4_VAL_PVT 0x7
+
+/*
+ * Errata i689: If a key is released for a time shorter than debounce time,
+ * the keyboard will idle and never detect the key release. The workaround
+ * is to use at least a 12ms debounce time. See omap5432 TRM chapter
+ * "26.4.6.2 Keyboard Controller Timer" for more information.
+ */
+#define OMAP4_KEYPAD_PTV_DIV_128 0x6
+#define OMAP4_KEYPAD_DEBOUNCINGTIME_MS(dbms, ptv) \
+ ((((dbms) * 1000) / ((1 << ((ptv) + 1)) * (1000000 / 32768))) - 1)
+#define OMAP4_VAL_DEBOUNCINGTIME_16MS \
+ OMAP4_KEYPAD_DEBOUNCINGTIME_MS(16, OMAP4_KEYPAD_PTV_DIV_128)
enum {
KBD_REVISION_OMAP4 = 0,
@@ -116,12 +126,8 @@ static irqreturn_t omap4_keypad_irq_handler(int irq, void *dev_id)
{
struct omap4_keypad *keypad_data = dev_id;
- if (kbd_read_irqreg(keypad_data, OMAP4_KBD_IRQSTATUS)) {
- /* Disable interrupts */
- kbd_write_irqreg(keypad_data, OMAP4_KBD_IRQENABLE,
- OMAP4_VAL_IRQDISABLE);
+ if (kbd_read_irqreg(keypad_data, OMAP4_KBD_IRQSTATUS))
return IRQ_WAKE_THREAD;
- }
return IRQ_NONE;
}
@@ -163,11 +169,6 @@ static irqreturn_t omap4_keypad_irq_thread_fn(int irq, void *dev_id)
kbd_write_irqreg(keypad_data, OMAP4_KBD_IRQSTATUS,
kbd_read_irqreg(keypad_data, OMAP4_KBD_IRQSTATUS));
- /* enable interrupts */
- kbd_write_irqreg(keypad_data, OMAP4_KBD_IRQENABLE,
- OMAP4_DEF_IRQENABLE_EVENTEN |
- OMAP4_DEF_IRQENABLE_LONGKEY);
-
return IRQ_HANDLED;
}
@@ -181,9 +182,9 @@ static int omap4_keypad_open(struct input_dev *input)
kbd_writel(keypad_data, OMAP4_KBD_CTRL,
OMAP4_DEF_CTRL_NOSOFTMODE |
- (OMAP4_VAL_PVT << OMAP4_DEF_CTRL_PTV_SHIFT));
+ (OMAP4_KEYPAD_PTV_DIV_128 << OMAP4_DEF_CTRL_PTV_SHIFT));
kbd_writel(keypad_data, OMAP4_KBD_DEBOUNCINGTIME,
- OMAP4_VAL_DEBOUNCINGTIME);
+ OMAP4_VAL_DEBOUNCINGTIME_16MS);
/* clear pending interrupts */
kbd_write_irqreg(keypad_data, OMAP4_KBD_IRQSTATUS,
kbd_read_irqreg(keypad_data, OMAP4_KBD_IRQSTATUS));
@@ -204,9 +205,10 @@ static void omap4_keypad_close(struct input_dev *input)
disable_irq(keypad_data->irq);
- /* Disable interrupts */
+ /* Disable interrupts and wake-up events */
kbd_write_irqreg(keypad_data, OMAP4_KBD_IRQENABLE,
OMAP4_VAL_IRQDISABLE);
+ kbd_writel(keypad_data, OMAP4_KBD_WAKEUPENABLE, 0);
/* clear pending interrupts */
kbd_write_irqreg(keypad_data, OMAP4_KBD_IRQSTATUS,
@@ -354,7 +356,7 @@ static int omap4_keypad_probe(struct platform_device *pdev)
}
error = request_threaded_irq(keypad_data->irq, omap4_keypad_irq_handler,
- omap4_keypad_irq_thread_fn, 0,
+ omap4_keypad_irq_thread_fn, IRQF_ONESHOT,
"omap4-keypad", keypad_data);
if (error) {
dev_err(&pdev->dev, "failed to register interrupt\n");
diff --git a/drivers/input/keydebug-core.c b/drivers/input/keydebug-core.c
new file mode 100644
index 0000000..a5a7c04
--- /dev/null
+++ b/drivers/input/keydebug-core.c
@@ -0,0 +1,288 @@
+/* drivers/input/keydebug-core.c
+ *
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/input.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/syscalls.h>
+#include <linux/of.h>
+#include <linux/time.h>
+#include <linux/keycombo.h>
+#include <linux/keydebug.h>
+#include <linux/keydebug-func.h>
+
+/*
+ * On the kernel command line specify
+ * keydebug.kernel_top_enable=1 to enable the kernel_top
+ * By default kernel_top is turned on
+ */
+static int kernel_top_enable = 1;
+module_param(kernel_top_enable, int, 0644);
+
+/*
+ * On the kernel command line specify
+ * keydebug.show_dstate_enable=1 to enable the show_dstate
+ * By default show_dstate is turned on
+ */
+static int show_dstate_enable = 1;
+module_param(show_dstate_enable, int, 0644);
+
+/*
+ * On the kernel command line specify
+ * keydebug.showallcpus_enable=1 to enable the showallcpus
+ * By default showallcpus is turned on
+ */
+static int showallcpus_enable = 1;
+module_param(showallcpus_enable, int, 0644);
+
+#define DEFAULT_DBG_DELAY 3000 /* millisecond */
+static int probe_cnt;
+static struct workqueue_struct *kdbg_wq;
+
+void do_keydebug(struct work_struct *this)
+{
+ struct delayed_work *dwork = container_of(this, struct delayed_work,
+ work);
+ struct keydebug_platform_data *pdata = container_of(dwork,
+ struct keydebug_platform_data, delayed_work);
+
+ if (kernel_top_enable)
+ kernel_top_monitor();
+
+ if (show_dstate_enable) {
+ pr_info("======= Show D state tasks++ =======\n");
+ show_state_filter(TASK_UNINTERRUPTIBLE);
+ pr_info("======= Show D state tasks-- =======\n");
+ }
+
+ if (showallcpus_enable)
+ keydebug_showallcpus();
+
+ pdata->keydebug_requested = false;
+}
+
+static void keydebug_event(void *priv)
+{
+ struct keydebug_platform_data *pdata = priv;
+ uint32_t msecs = DEFAULT_DBG_DELAY;
+
+ if (pdata->keydebug_requested) {
+ pr_info("%s: request is running\n", __func__);
+ return;
+ } else {
+ if (pdata->dbg_fn_delay)
+ msecs = pdata->dbg_fn_delay;
+
+ if (kernel_top_enable)
+ kernel_top_init();
+
+ queue_delayed_work(kdbg_wq, &pdata->delayed_work,
+ msecs_to_jiffies(msecs));
+ pdata->keydebug_requested = true;
+ }
+}
+
+static int keydebug_parse_dt(struct device *dev,
+ struct keydebug_platform_data *pdata)
+{
+ int ret = 0, cnt = 0, num_keys;
+ struct device_node *dt = dev_of_node(dev);
+ struct property *prop;
+
+ /* Parse key_down_delay */
+ if (of_property_read_u32(dt, "key_down_delay", &pdata->key_down_delay))
+ pr_info("%s: DT:key_down_delay property not found\n", __func__);
+
+ /* Parse dbg_delay */
+ if (of_property_read_u32(dt, "dbg_fn_delay", &pdata->dbg_fn_delay))
+ pr_info("%s: DT:dbg_fn_delay property not found\n", __func__);
+
+ /* Must have keys_down property */
+ prop = of_find_property(dt, "keys_down", NULL);
+ if (!prop) {
+ pr_err("%s: DT:keys_down property not found\n", __func__);
+ ret = -EINVAL;
+ goto err_parse_keys_down_failed;
+ } else {
+ num_keys = prop->length / sizeof(uint32_t);
+ }
+
+ /* Allocate down_size + 1 to assure 0 terminated */
+ pdata->keys_down = devm_kzalloc(dev,
+ (num_keys + 1) * sizeof(uint32_t), GFP_KERNEL);
+ if (!pdata->keys_down) {
+ pr_err("%s: DT:keys_down fail to allocate memory\n", __func__);
+ ret = -ENOMEM;
+ goto err_parse_keys_down_failed;
+ }
+
+ if (of_property_read_u32_array(dt, "keys_down",
+ pdata->keys_down, num_keys)) {
+ pr_err("%s: DT:keys_down parse err\n", __func__);
+ ret = -EINVAL;
+ goto err_keys_down_failed;
+ }
+
+ pr_info("%s: DT:key_down_delay=%d dbg_fn_delay=%d"
+ " keys_down num_keys=%d\n", __func__, pdata->key_down_delay,
+ pdata->dbg_fn_delay, num_keys);
+
+ for (cnt = 0; cnt < num_keys; cnt++)
+ pr_info("%s: DT:keys_down=%d\n", __func__, pdata->keys_down[cnt]);
+
+ return 0;
+
+err_keys_down_failed:
+ devm_kfree(dev, pdata->keys_down);
+err_parse_keys_down_failed:
+ return ret;
+}
+
+static int keydebug_probe(struct platform_device *pdev)
+{
+ int ret = -ENOMEM;
+ struct device *dev = &pdev->dev;
+ struct keycombo_platform_data *pdata_child;
+ struct keydebug_platform_data *pdata = dev_get_platdata(dev);
+ int down_size = 0, size;
+ uint32_t key, *keyp;
+ int *ckeyp;
+
+ /* only allow one instance at a time*/
+ if (probe_cnt)
+ return 0;
+ else
+ probe_cnt++;
+
+ if (dev_of_node(dev) && !pdata) {
+ pdata = devm_kzalloc(dev, sizeof(*pdata), GFP_KERNEL);
+ if (!pdata) {
+ pr_err("%s: fail to allocate "
+ "keydebug_platform_data\n", __func__);
+ ret = -ENOMEM;
+ goto err_get_pdata_fail;
+ }
+
+ ret = keydebug_parse_dt(dev, pdata);
+ if (ret < 0) {
+ pr_err("%s: keydebug_parse_dt fail\n", __func__);
+ goto err_parse_fail;
+ }
+ } else if (!pdata)
+ return -EINVAL;
+
+ kdbg_wq = alloc_workqueue("kdbgd", WQ_MEM_RECLAIM | WQ_HIGHPRI, 0);
+
+ INIT_DELAYED_WORK(&pdata->delayed_work, do_keydebug);
+
+ pdata->pdev_child = platform_device_alloc(KEYCOMBO_NAME,
+ PLATFORM_DEVID_AUTO);
+ if (!pdata->pdev_child)
+ return -ENOMEM;
+
+ pdata->pdev_child->dev.parent = dev;
+
+ /* Calculate valid keys down size*/
+ keyp = pdata->keys_down;
+ while ((key = *keyp++)) {
+ if (key >= KEY_MAX)
+ continue;
+ down_size++;
+ }
+
+ /* Allocate down_size + 1 to assure 0 terminated */
+ size = sizeof(struct keycombo_platform_data)
+ + sizeof(int) * (down_size + 1);
+ pdata_child = devm_kzalloc(dev, size, GFP_KERNEL);
+ if (!pdata_child)
+ goto error;
+
+ /* Copy valid key code*/
+ keyp = pdata->keys_down;
+ ckeyp = pdata_child->keys_down;
+ while ((*ckeyp = *keyp++)) {
+ if (*ckeyp >= KEY_MAX)
+ continue;
+ ckeyp++;
+ }
+
+ pdata_child->priv = pdata;
+ pdata_child->key_down_fn = keydebug_event;
+ pdata_child->key_down_delay = pdata->key_down_delay;
+ ret = platform_device_add_data(pdata->pdev_child, pdata_child, size);
+ if (ret)
+ goto error;
+
+ return platform_device_add(pdata->pdev_child);
+error:
+ platform_device_put(pdata->pdev_child);
+err_parse_fail:
+ devm_kfree(dev, pdata);
+err_get_pdata_fail:
+ if (kdbg_wq)
+ destroy_workqueue(kdbg_wq);
+ probe_cnt--;
+ return ret;
+}
+
+static int keydebug_remove(struct platform_device *pdev)
+{
+ struct keydebug_platform_data *pdata = dev_get_platdata(&pdev->dev);
+
+ if (kernel_top_enable)
+ kernel_top_exit();
+
+ platform_device_put(pdata->pdev_child);
+ if (kdbg_wq)
+ destroy_workqueue(kdbg_wq);
+ probe_cnt = 0;
+ return 0;
+}
+
+#ifdef CONFIG_OF
+static const struct of_device_id keydebug_match_table[] = {
+ { .compatible = KEYDEBUG_NAME},
+ {},
+};
+#else
+#define keydebug_match_table NULL
+#endif
+
+struct platform_driver keydebug_driver = {
+ .probe = keydebug_probe,
+ .remove = keydebug_remove,
+ .driver = {
+ .name = KEYDEBUG_NAME,
+ .owner = THIS_MODULE,
+ .of_match_table = keydebug_match_table,
+ },
+};
+
+static int __init keydebug_init(void)
+{
+ return platform_driver_register(&keydebug_driver);
+}
+
+static void __exit keydebug_exit(void)
+{
+ return platform_driver_unregister(&keydebug_driver);
+}
+
+module_init(keydebug_init);
+module_exit(keydebug_exit);
+MODULE_DESCRIPTION("keydebug Driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/input/keydebug-func.c b/drivers/input/keydebug-func.c
new file mode 100644
index 0000000..1239b4e
--- /dev/null
+++ b/drivers/input/keydebug-func.c
@@ -0,0 +1,422 @@
+/* drivers/input/keydebug-func.c
+ *
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/sched.h>
+#include <linux/slab.h>
+#include <linux/kernel_stat.h>
+#include <linux/time.h>
+#include <linux/tick.h>
+#include <linux/rtc.h>
+#include <linux/threads.h>
+#include <linux/nmi.h>
+#include <asm/irq_regs.h>
+#include <linux/keydebug-func.h>
+
+#define NUM_BUSY_TASK_CHECK 5
+
+struct kernel_top_context {
+ u64 *prev_tasktics_array;
+ u64 *frame_tasktics_array;
+ pid_t *curr_task_pid_array;
+ pid_t top_task_pid_array[NUM_BUSY_TASK_CHECK];
+ struct task_struct **task_ptr_array;
+ struct kernel_cpustat curr_all_cpustat;
+ struct kernel_cpustat prev_all_cpustat;
+ u64 frame_cpustat_total;
+ bool kernel_top_alloc_done;
+};
+
+static struct kernel_top_context ktop_cxt;
+static DEFINE_MUTEX(kernel_top_mutex);
+
+#ifdef arch_idle_time
+
+static cputime64_t get_idle_time(int cpu)
+{
+ cputime64_t idle;
+
+ idle = kcpustat_cpu(cpu).cpustat[CPUTIME_IDLE];
+ if (cpu_online(cpu) && !nr_iowait_cpu(cpu))
+ idle += arch_idle_time(cpu);
+ return idle;
+}
+
+static cputime64_t get_iowait_time(int cpu)
+{
+ cputime64_t iowait;
+
+ iowait = kcpustat_cpu(cpu).cpustat[CPUTIME_IOWAIT];
+ if (cpu_online(cpu) && nr_iowait_cpu(cpu))
+ iowait += arch_idle_time(cpu);
+ return iowait;
+}
+
+#else
+
+static u64 get_idle_time(int cpu)
+{
+ u64 idle, idle_time = -1ULL;
+
+ if (cpu_online(cpu))
+ idle_time = get_cpu_idle_time_us(cpu, NULL);
+
+ if (idle_time == -1ULL)
+ /* !NO_HZ or cpu offline so we can rely on cpustat.idle */
+ idle = kcpustat_cpu(cpu).cpustat[CPUTIME_IDLE];
+ else
+ idle = usecs_to_cputime64(idle_time);
+
+ return idle;
+}
+
+static u64 get_iowait_time(int cpu)
+{
+ u64 iowait, iowait_time = -1ULL;
+
+ if (cpu_online(cpu))
+ iowait_time = get_cpu_iowait_time_us(cpu, NULL);
+
+ if (iowait_time == -1ULL)
+ /* !NO_HZ or cpu offline so we can rely on cpustat.iowait */
+ iowait = kcpustat_cpu(cpu).cpustat[CPUTIME_IOWAIT];
+ else
+ iowait = usecs_to_cputime64(iowait_time);
+
+ return iowait;
+}
+
+#endif
+
+static void get_all_cpustat(struct kernel_cpustat *cpu_stat)
+{
+ int cpu;
+
+ if (!cpu_stat)
+ return;
+
+ memset(cpu_stat, 0, sizeof(struct kernel_cpustat));
+
+#ifdef CONFIG_SMP
+ for_each_possible_cpu(cpu) {
+ cpu_stat->cpustat[CPUTIME_USER] +=
+ kcpustat_cpu(cpu).cpustat[CPUTIME_USER];
+ cpu_stat->cpustat[CPUTIME_NICE] +=
+ kcpustat_cpu(cpu).cpustat[CPUTIME_NICE];
+ cpu_stat->cpustat[CPUTIME_SYSTEM] +=
+ kcpustat_cpu(cpu).cpustat[CPUTIME_SYSTEM];
+ cpu_stat->cpustat[CPUTIME_IDLE] +=
+ get_idle_time(cpu);
+ cpu_stat->cpustat[CPUTIME_IOWAIT] +=
+ get_iowait_time(cpu);
+ cpu_stat->cpustat[CPUTIME_IRQ] +=
+ kcpustat_cpu(cpu).cpustat[CPUTIME_IRQ];
+ cpu_stat->cpustat[CPUTIME_SOFTIRQ] +=
+ kcpustat_cpu(cpu).cpustat[CPUTIME_SOFTIRQ];
+ cpu_stat->cpustat[CPUTIME_STEAL] +=
+ kcpustat_cpu(cpu).cpustat[CPUTIME_STEAL];
+ cpu_stat->cpustat[CPUTIME_GUEST] +=
+ kcpustat_cpu(cpu).cpustat[CPUTIME_GUEST];
+ cpu_stat->cpustat[CPUTIME_GUEST_NICE] +=
+ kcpustat_cpu(cpu).cpustat[CPUTIME_GUEST_NICE];
+ }
+#endif
+}
+
+static void sort_top_tasks(u64 *frame_tasktics_array,
+ pid_t *curr_task_pid_array, int task_count, pid_t *result)
+{
+ int i = 0, j = 0, k = 0;
+ int pid_checked = 0;
+ pid_t p = 0;
+
+ for (i = 0; i < NUM_BUSY_TASK_CHECK; i++) {
+ result[i] = 0;
+ /* Find the task which has the largest cputime in this frame */
+ if (i == 0) {
+ for (j = 0; j < task_count; j++) {
+ p = curr_task_pid_array[j];
+
+ if (frame_tasktics_array[result[i]] <
+ frame_tasktics_array[p])
+ result[i] = p;
+ }
+ } else {
+ for (j = 0; j < task_count; j++) {
+ p = curr_task_pid_array[j];
+ for (k = 0; k < i; k++) {
+ if (result[k] == p) {
+ pid_checked = 1;
+ break;
+ }
+ }
+ if (pid_checked) {
+ pid_checked = 0;
+ continue;
+ }
+ if (frame_tasktics_array[result[i]] <
+ frame_tasktics_array[p])
+ result[i] = p;
+ }
+ }
+ }
+}
+
+static u64 cal_frame_cpustat_total(struct kernel_cpustat curr_all_cpustat,
+ struct kernel_cpustat prev_all_cpustat)
+{
+ u64 user_time = 0, system_time = 0, io_time = 0;
+ u64 irq_time = 0, idle_time = 0;
+
+ user_time = ((curr_all_cpustat.cpustat[CPUTIME_USER] +
+ curr_all_cpustat.cpustat[CPUTIME_NICE]) -
+ (prev_all_cpustat.cpustat[CPUTIME_USER] +
+ prev_all_cpustat.cpustat[CPUTIME_NICE]));
+ system_time = (curr_all_cpustat.cpustat[CPUTIME_SYSTEM] -
+ prev_all_cpustat.cpustat[CPUTIME_SYSTEM]);
+ io_time = (curr_all_cpustat.cpustat[CPUTIME_IOWAIT] -
+ prev_all_cpustat.cpustat[CPUTIME_IOWAIT]);
+ irq_time = ((curr_all_cpustat.cpustat[CPUTIME_IRQ] +
+ curr_all_cpustat.cpustat[CPUTIME_SOFTIRQ]) -
+ (prev_all_cpustat.cpustat[CPUTIME_IRQ] +
+ prev_all_cpustat.cpustat[CPUTIME_SOFTIRQ]));
+ idle_time = ((curr_all_cpustat.cpustat[CPUTIME_IDLE] >
+ prev_all_cpustat.cpustat[CPUTIME_IDLE]) ?
+ curr_all_cpustat.cpustat[CPUTIME_IDLE] -
+ prev_all_cpustat.cpustat[CPUTIME_IDLE] : 0);
+ idle_time += ((curr_all_cpustat.cpustat[CPUTIME_STEAL] +
+ curr_all_cpustat.cpustat[CPUTIME_GUEST]) -
+ (prev_all_cpustat.cpustat[CPUTIME_STEAL] +
+ prev_all_cpustat.cpustat[CPUTIME_GUEST]));
+
+ return (user_time + system_time + io_time + irq_time + idle_time);
+}
+
+static void kernel_top_cal(void)
+{
+ int task_count = 0;
+ struct task_struct *tsk;
+ struct task_cputime cputime;
+ struct kernel_top_context *cxt = &ktop_cxt;
+
+ /* Calculate each tasks tics in this time frame*/
+ rcu_read_lock();
+ for_each_process(tsk) {
+ thread_group_cputime(tsk, &cputime);
+ if (tsk->pid < PID_MAX_DEFAULT) {
+ u64 cur_tasktics = (cputime.utime + cputime.stime);
+
+ cxt->frame_tasktics_array[tsk->pid] =
+ cur_tasktics -
+ cxt->prev_tasktics_array[tsk->pid];
+
+ cxt->prev_tasktics_array[tsk->pid] = cur_tasktics;
+ cxt->task_ptr_array[tsk->pid] = tsk;
+
+ if (cxt->frame_tasktics_array[tsk->pid] > 0) {
+ cxt->curr_task_pid_array[task_count] = tsk->pid;
+ task_count++;
+ }
+ }
+ }
+ rcu_read_unlock();
+
+ get_all_cpustat(&cxt->curr_all_cpustat);
+
+ sort_top_tasks(cxt->frame_tasktics_array,
+ cxt->curr_task_pid_array, task_count, cxt->top_task_pid_array);
+
+ cxt->frame_cpustat_total =
+ cal_frame_cpustat_total(cxt->curr_all_cpustat,
+ cxt->prev_all_cpustat);
+ memcpy(&cxt->prev_all_cpustat,
+ &cxt->curr_all_cpustat, sizeof(struct kernel_cpustat));
+
+}
+
+static void kernel_top_show(void)
+{
+ pid_t top_n_pid = 0;
+ int i;
+ struct kernel_top_context *cxt = &ktop_cxt;
+
+ pr_info("%s: CPU Usage PID Name\n", __func__);
+ for (i = 0; i < NUM_BUSY_TASK_CHECK; i++) {
+ if (cxt->frame_cpustat_total > 0) {
+ top_n_pid = cxt->top_task_pid_array[i];
+ pr_info("%s: %8llu%%%8d %s%10llu\n", __func__,
+ cxt->frame_tasktics_array[top_n_pid] * 100 /
+ cxt->frame_cpustat_total,
+ top_n_pid,
+ cxt->task_ptr_array[top_n_pid]->comm,
+ cxt->frame_tasktics_array[top_n_pid]);
+ }
+ }
+
+ memset(cxt->frame_tasktics_array, 0, sizeof(u64) * PID_MAX_DEFAULT);
+ memset(cxt->task_ptr_array, 0,
+ sizeof(struct task_struct *) * PID_MAX_DEFAULT);
+ memset(cxt->curr_task_pid_array, 0, sizeof(pid_t) * PID_MAX_DEFAULT);
+}
+
+void kernel_top_monitor(void)
+{
+ struct timespec ts;
+ struct rtc_time tm;
+ struct kernel_top_context *cxt = &ktop_cxt;
+
+ mutex_lock(&kernel_top_mutex);
+ if (cxt->kernel_top_alloc_done == false)
+ goto done;
+
+ kernel_top_cal();
+ kernel_top_show();
+
+ getnstimeofday(&ts);
+ rtc_time_to_tm(ts.tv_sec - (sys_tz.tz_minuteswest * 60), &tm);
+ pr_info("%s: Kernel Top Statistic done"
+ "(%02d-%02d %02d:%02d:%02d)\n", __func__,
+ tm.tm_mon + 1, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec);
+
+done:
+ mutex_unlock(&kernel_top_mutex);
+}
+EXPORT_SYMBOL_GPL(kernel_top_monitor);
+
+void kernel_top_init(void)
+{
+ struct task_struct *tsk;
+ struct task_cputime cputime;
+ struct timespec ts;
+ struct rtc_time tm;
+ struct kernel_top_context *cxt = &ktop_cxt;
+
+ mutex_lock(&kernel_top_mutex);
+ if (cxt->kernel_top_alloc_done == false) {
+
+ cxt->prev_tasktics_array =
+ vmalloc(sizeof(u64) * PID_MAX_DEFAULT);
+ if (cxt->prev_tasktics_array == NULL)
+ goto err_alloc_prev_tasktics;
+ cxt->frame_tasktics_array =
+ vmalloc(sizeof(u64) * PID_MAX_DEFAULT);
+ if (cxt->frame_tasktics_array == NULL)
+ goto err_alloc_frame_tasktics;
+ cxt->task_ptr_array =
+ vmalloc(sizeof(struct task_struct *) * PID_MAX_DEFAULT);
+ if (cxt->task_ptr_array == NULL)
+ goto err_alloc_task_ptr;
+ cxt->curr_task_pid_array =
+ vmalloc(sizeof(pid_t) * PID_MAX_DEFAULT);
+ if (cxt->curr_task_pid_array == NULL)
+ goto err_alloc_curr_task_pid;
+
+ cxt->kernel_top_alloc_done = true;
+ }
+
+ memset(cxt->prev_tasktics_array, 0, sizeof(u64) * PID_MAX_DEFAULT);
+ memset(cxt->frame_tasktics_array, 0, sizeof(u64) * PID_MAX_DEFAULT);
+ memset(cxt->task_ptr_array, 0,
+ sizeof(struct task_struct *) * PID_MAX_DEFAULT);
+ memset(cxt->curr_task_pid_array, 0, sizeof(pid_t) * PID_MAX_DEFAULT);
+
+ getnstimeofday(&ts);
+ rtc_time_to_tm(ts.tv_sec - (sys_tz.tz_minuteswest * 60), &tm);
+ pr_info("%s: Kernel Top Statistic start"
+ "(%02d-%02d %02d:%02d:%02d)\n", __func__,
+ tm.tm_mon + 1, tm.tm_mday, tm.tm_hour, tm.tm_min, tm.tm_sec);
+
+ get_all_cpustat(&cxt->curr_all_cpustat);
+ memcpy(&cxt->prev_all_cpustat,
+ &cxt->curr_all_cpustat, sizeof(struct kernel_cpustat));
+
+ /* Calculate time in a process;
+ * the sum of user time (utime) and system time (stime)*/
+ rcu_read_lock();
+ for_each_process(tsk) {
+ if (tsk->pid < PID_MAX_DEFAULT) {
+ thread_group_cputime(tsk, &cputime);
+ cxt->prev_tasktics_array[tsk->pid] =
+ cputime.stime + cputime.utime;
+ }
+ }
+ rcu_read_unlock();
+ goto done;
+
+err_alloc_curr_task_pid:
+ vfree(cxt->curr_task_pid_array);
+err_alloc_task_ptr:
+ vfree(cxt->task_ptr_array);
+err_alloc_frame_tasktics:
+ vfree(cxt->frame_tasktics_array);
+err_alloc_prev_tasktics:
+ vfree(cxt->prev_tasktics_array);
+
+ cxt->kernel_top_alloc_done = false;
+ pr_info("%s: memory allocate failed", __func__);
+done:
+ mutex_unlock(&kernel_top_mutex);
+}
+
+void kernel_top_exit(void)
+{
+ struct kernel_top_context *cxt = &ktop_cxt;
+
+ mutex_lock(&kernel_top_mutex);
+ if (cxt->kernel_top_alloc_done) {
+ vfree(cxt->curr_task_pid_array);
+ vfree(cxt->task_ptr_array);
+ vfree(cxt->frame_tasktics_array);
+ vfree(cxt->prev_tasktics_array);
+ memset(cxt, 0, sizeof(*cxt));
+ }
+ mutex_unlock(&kernel_top_mutex);
+}
+
+#ifdef CONFIG_SMP
+static DEFINE_SPINLOCK(show_lock);
+
+static void keydebug_showacpu(void *dummy)
+{
+ unsigned long flags;
+
+ /* Idle CPUs have no interesting backtrace. */
+ if (idle_cpu(smp_processor_id()))
+ return;
+
+ spin_lock_irqsave(&show_lock, flags);
+ dump_stack();
+ spin_unlock_irqrestore(&show_lock, flags);
+}
+
+void keydebug_showallcpus(void)
+{
+ if(!trigger_all_cpu_backtrace()) {
+ struct pt_regs *regs = NULL;
+
+ if (in_irq())
+ regs = get_irq_regs();
+ if (regs) {
+ pr_info("CPU%d:\n", smp_processor_id());
+ show_regs(regs);
+ }
+ dump_stack();
+ smp_call_function(keydebug_showacpu, NULL, 0);
+ }
+}
+#else
+void keydebug_showallcpus(void)
+{
+}
+#endif
diff --git a/drivers/input/misc/Kconfig b/drivers/input/misc/Kconfig
index f8a987a..e0de873 100644
--- a/drivers/input/misc/Kconfig
+++ b/drivers/input/misc/Kconfig
@@ -848,6 +848,17 @@
To compile this driver as a module, choose M here: the
module will be called drv260x-haptics.
+config INPUT_DRV2624_HAPTICS
+ tristate "TI DRV2624 haptics support"
+ depends on INPUT && I2C
+ select INPUT_FF_MEMLESS
+ select REGMAP_I2C
+ help
+ Say Y to enable support for the TI DRV2624 haptics driver.
+
+ To compile this driver as a module, choose M here: the
+ module will be called drv2624-haptics.
+
config INPUT_DRV2665_HAPTICS
tristate "TI DRV2665 haptics support"
depends on INPUT && I2C
@@ -880,3 +891,6 @@
module will be called hisi_powerkey.
endif
+
+source "drivers/input/misc/vd6281/Kconfig"
+source "drivers/input/misc/fpr_FingerprintCard/Kconfig"
diff --git a/drivers/input/misc/Makefile b/drivers/input/misc/Makefile
index 149273c..e2030a3d 100644
--- a/drivers/input/misc/Makefile
+++ b/drivers/input/misc/Makefile
@@ -30,6 +30,7 @@
obj-$(CONFIG_INPUT_DM355EVM) += dm355evm_keys.o
obj-$(CONFIG_INPUT_E3X0_BUTTON) += e3x0-button.o
obj-$(CONFIG_INPUT_DRV260X_HAPTICS) += drv260x.o
+obj-$(CONFIG_INPUT_DRV2624_HAPTICS) += drv2624.o
obj-$(CONFIG_INPUT_DRV2665_HAPTICS) += drv2665.o
obj-$(CONFIG_INPUT_DRV2667_HAPTICS) += drv2667.o
obj-$(CONFIG_INPUT_GP2A) += gp2ap002a00f.o
@@ -84,3 +85,5 @@
obj-$(CONFIG_INPUT_IDEAPAD_SLIDEBAR) += ideapad_slidebar.o
obj-$(CONFIG_INPUT_PIXART_OTS_PAT9125_SWITCH) += ots_pat9125/
obj-$(CONFIG_STMVL53L0X) += vl53l0x/
+obj-$(CONFIG_INPUT_VD6281) += vd6281/
+obj-$(CONFIG_FPR_FPC) += fpr_FingerprintCard/
\ No newline at end of file
diff --git a/drivers/input/misc/drv260x.c b/drivers/input/misc/drv260x.c
index 930424e..251d64c 100644
--- a/drivers/input/misc/drv260x.c
+++ b/drivers/input/misc/drv260x.c
@@ -521,7 +521,7 @@ static int drv260x_probe(struct i2c_client *client,
if (!haptics)
return -ENOMEM;
- haptics->rated_voltage = DRV260X_DEF_OD_CLAMP_VOLT;
+ haptics->overdrive_voltage = DRV260X_DEF_OD_CLAMP_VOLT;
haptics->rated_voltage = DRV260X_DEF_RATED_VOLT;
if (pdata) {
diff --git a/drivers/input/misc/drv2624.c b/drivers/input/misc/drv2624.c
new file mode 100644
index 0000000..b15e951
--- /dev/null
+++ b/drivers/input/misc/drv2624.c
@@ -0,0 +1,1708 @@
+/*
+** =============================================================================
+** Copyright (c) 2016 Texas Instruments Inc.
+**
+** This program is free software; you can redistribute it and/or
+** modify it under the terms of the GNU General Public License
+** as published by the Free Software Foundation; either version 2
+** of the License, or (at your option) any later version.
+**
+** This program is distributed in the hope that it will be useful,
+** but WITHOUT ANY WARRANTY; without even the implied warranty of
+** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+** GNU General Public License for more details.
+**
+** File:
+** drv2624.c
+**
+** Description:
+** DRV2624 chip driver
+**
+** =============================================================================
+*/
+
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/fs.h>
+#include <linux/i2c.h>
+#include <linux/semaphore.h>
+#include <linux/device.h>
+#include <linux/syscalls.h>
+#include <linux/uaccess.h>
+#include <linux/gpio.h>
+#include <linux/sched.h>
+#include <linux/spinlock_types.h>
+#include <linux/spinlock.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/of.h>
+#include <linux/of_gpio.h>
+#include <linux/jiffies.h>
+#include <linux/err.h>
+#include <linux/clk.h>
+#include <linux/sysfs.h>
+
+#include "drv2624.h"
+
+static struct drv2624_data *drv2624_plat_data;
+
+static bool drv2624_is_volatile_reg(struct device *dev, unsigned int reg);
+
+static int drv2624_reg_read(struct drv2624_data *drv2624, unsigned char reg)
+{
+ unsigned int val;
+ int ret;
+
+ ret = regmap_read(drv2624->regmap, reg, &val);
+ if (ret < 0) {
+ dev_err(drv2624->dev,
+ "%s reg=0x%x error %d\n", __func__, reg, ret);
+ return ret;
+ }
+
+ dev_dbg(drv2624->dev, "%s, Reg[0x%x]=0x%x\n", __func__, reg, val);
+
+ return val;
+}
+
+static int drv2624_reg_write(struct drv2624_data *drv2624,
+ unsigned char reg, unsigned char val)
+{
+ int ret;
+
+ ret = regmap_write(drv2624->regmap, reg, val);
+ if (ret < 0) {
+ dev_err(drv2624->dev,
+ "%s reg=0x%x, value=0%x error %d\n",
+ __func__, reg, val, ret);
+ } else {
+ dev_dbg(drv2624->dev, "%s, Reg[0x%x]=0x%x\n",
+ __func__, reg, val);
+ }
+
+ return ret;
+}
+
+static int drv2624_bulk_write(struct drv2624_data *drv2624,
+ unsigned char reg, const u8 *buf,
+ unsigned int count)
+{
+ int ret, i;
+
+ ret = regmap_bulk_write(drv2624->regmap, reg, buf, count);
+ if (ret < 0) {
+ dev_err(drv2624->dev,
+ "%s reg=0%x, count=%d error %d\n",
+ __func__, reg, count, ret);
+ } else {
+ for (i = 0; i < count; i++)
+ dev_dbg(drv2624->dev, "%s, Reg[0x%x]=0x%x\n",
+ __func__, reg + i, buf[i]);
+ }
+ return ret;
+}
+
+static int drv2624_set_bits(struct drv2624_data *drv2624,
+ unsigned char reg, unsigned char mask,
+ unsigned char val)
+{
+ int ret;
+
+ if (drv2624_is_volatile_reg(drv2624->dev, reg))
+ ret = regmap_write_bits(drv2624->regmap, reg, mask, val);
+ else
+ ret = regmap_update_bits(drv2624->regmap, reg, mask, val);
+ if (ret < 0) {
+ dev_err(drv2624->dev,
+ "%s reg=%x, mask=0x%x, value=0x%x error %d\n",
+ __func__, reg, mask, val, ret);
+ } else {
+ dev_dbg(drv2624->dev, "%s, Reg[0x%x]:M=0x%x, V=0x%x\n",
+ __func__, reg, mask, val);
+ }
+
+ return ret;
+}
+
+static void drv2624_enable_irq(struct drv2624_data *drv2624, bool rtp)
+{
+ unsigned char mask;
+
+ if (!drv2624->irq)
+ return;
+
+ mask = INT_ENABLE_CRITICAL;
+
+ if (rtp)
+ mask = INT_ENABLE_ALL;
+
+ drv2624_reg_read(drv2624, DRV2624_REG_STATUS);
+ drv2624_reg_write(drv2624, DRV2624_REG_INT_ENABLE, mask);
+
+ enable_irq(drv2624->irq);
+}
+
+static void drv2624_disable_irq(struct drv2624_data *drv2624)
+{
+ if (!drv2624->irq)
+ return;
+
+ disable_irq(drv2624->irq);
+ drv2624_reg_write(drv2624,
+ DRV2624_REG_INT_ENABLE, INT_MASK_ALL);
+}
+
+static void drv2624_reset(struct drv2624_data *drv2624)
+{
+ int ret;
+
+ gpio_set_value(drv2624->plat_data.gpio_nrst, 0);
+ usleep_range(1000, 2000);
+ gpio_set_value(drv2624->plat_data.gpio_nrst, 1);
+ usleep_range(1000, 2000);
+
+ regcache_mark_dirty(drv2624->regmap);
+ ret = regcache_sync(drv2624->regmap);
+ if (ret) {
+ dev_err(drv2624->dev, "Failed to sync cache: %d\n", ret);
+ gpio_direction_output(drv2624->plat_data.gpio_nrst, 0);
+ }
+}
+
+static int drv2624_poll_go_bit_stop(struct drv2624_data *drv2624)
+{
+ int ret, value;
+ int poll_ready = POLL_GO_BIT_RETRY; /* to finish auto-brake */
+
+ do {
+ ret = drv2624_reg_read(drv2624, DRV2624_REG_GO);
+ if (ret >= 0) {
+ value = ret & 0x01;
+ if (!value)
+ return ret;
+ }
+ usleep_range(8000, 10000);
+ } while (poll_ready--);
+
+ dev_err(drv2624->dev, "%s, ERROR: failed to clear GO\n", __func__);
+
+ return -EINVAL;
+}
+
+static int drv2624_set_go_bit(struct drv2624_data *drv2624, unsigned char val)
+{
+ int ret;
+ int retry = POLL_GO_BIT_RETRY;
+
+ val &= 0x01;
+
+ do {
+ ret = drv2624_set_bits(drv2624, DRV2624_REG_GO, 0x01, val);
+ if (ret >= 0) {
+ usleep_range(1000, 1100);
+ /* Only poll GO bit for STOP */
+ if (!val)
+ ret = drv2624_poll_go_bit_stop(drv2624);
+ return ret;
+ }
+ usleep_range(8000, 10000);
+ } while (retry--);
+
+ drv2624_reset(drv2624);
+
+ return ret;
+}
+
+static void drv2624_change_mode(struct drv2624_data *drv2624,
+ unsigned char work_mode)
+{
+ drv2624_set_bits(drv2624, DRV2624_REG_MODE, WORKMODE_MASK, work_mode);
+}
+
+static int drv2624_get_mode(struct drv2624_data *drv2624)
+{
+ return drv2624_reg_read(drv2624, DRV2624_REG_MODE) & WORKMODE_MASK;
+}
+
+static void drv2624_stop(struct drv2624_data *drv2624)
+{
+ if (drv2624->vibrator_playing) {
+ dev_dbg(drv2624->dev, "%s\n", __func__);
+ drv2624_disable_irq(drv2624);
+ drv2624_set_go_bit(drv2624, STOP);
+ drv2624->work_mode = WORK_IDLE;
+ drv2624->vibrator_playing = false;
+ pm_relax(drv2624->dev);
+ }
+}
+
+static void drv2624_haptics_stopwork(struct work_struct *work)
+{
+ struct drv2624_data *drv2624 =
+ container_of(work, struct drv2624_data, stop_work);
+
+ mutex_lock(&drv2624->lock);
+
+ drv2624_stop(drv2624);
+ drv2624->led_dev.brightness = LED_OFF;
+
+ mutex_unlock(&drv2624->lock);
+}
+
+static void drv2624_haptics_work(struct work_struct *work)
+{
+ struct drv2624_data *drv2624 =
+ container_of(work, struct drv2624_data, work);
+ int ret = 0;
+
+ mutex_lock(&drv2624->lock);
+
+ drv2624_stop(drv2624);
+
+ pm_stay_awake(drv2624->dev);
+ drv2624->vibrator_playing = true;
+ drv2624_enable_irq(drv2624, true);
+
+ ret = drv2624_set_go_bit(drv2624, GO);
+ if (ret < 0) {
+ dev_warn(drv2624->dev, "Start playback failed\n");
+ drv2624->vibrator_playing = false;
+ drv2624_disable_irq(drv2624);
+ pm_relax(drv2624->dev);
+ } else {
+ drv2624->led_dev.brightness = LED_FULL;
+ drv2624->work_mode |= WORK_VIBRATOR;
+ }
+
+ mutex_unlock(&drv2624->lock);
+}
+
+static void vibrator_enable(struct led_classdev *led_cdev,
+ enum led_brightness value)
+{
+ struct drv2624_data *drv2624 =
+ container_of(led_cdev, struct drv2624_data, led_dev);
+
+ if (value == LED_OFF)
+ queue_work(drv2624->drv2624_wq, &drv2624->stop_work);
+ else
+ queue_work(drv2624->drv2624_wq, &drv2624->work);
+
+}
+
+static void vibrator_work_routine(struct work_struct *work)
+{
+ struct drv2624_data *drv2624 =
+ container_of(work, struct drv2624_data, vibrator_work);
+ unsigned char mode = MODE_RTP;
+ unsigned char status;
+ int ret = 0;
+
+ mutex_lock(&drv2624->lock);
+
+ dev_dbg(drv2624->dev, "%s, afer mnWorkMode=0x%x\n",
+ __func__, drv2624->work_mode);
+
+ if (drv2624->work_mode & WORK_IRQ) {
+ ret = drv2624_reg_read(drv2624, DRV2624_REG_STATUS);
+ if (ret >= 0)
+ drv2624->int_status = ret;
+
+ drv2624_disable_irq(drv2624);
+
+ if (ret < 0) {
+ dev_err(drv2624->dev,
+ "%s, reg read error\n", __func__);
+ goto err;
+ }
+
+ status = drv2624->int_status;
+ dev_dbg(drv2624->dev, "%s, status=0x%x\n",
+ __func__, drv2624->int_status);
+
+ if (status & OVERCURRENT_MASK) {
+ dev_err(drv2624->dev,
+ "ERROR, Over Current detected!!\n");
+ }
+
+ if (status & OVERTEMPRATURE_MASK) {
+ dev_err(drv2624->dev,
+ "ERROR, Over Temperature detected!!\n");
+ }
+
+ if (status & ULVO_MASK)
+ dev_err(drv2624->dev, "ERROR, VDD drop observed!!\n");
+
+ if (status & PRG_ERR_MASK)
+ dev_err(drv2624->dev, "ERROR, PRG error!!\n");
+
+ if (status & PROCESS_DONE_MASK) {
+ ret = drv2624_reg_read(drv2624, DRV2624_REG_MODE);
+ if (ret < 0) {
+ dev_err(drv2624->dev,
+ "%s, reg read error\n", __func__);
+ goto err;
+ }
+
+ mode = ret & WORKMODE_MASK;
+ if (mode == MODE_CALIBRATION) {
+ drv2624->auto_cal_result.result = status;
+ if ((status & DIAG_MASK) != DIAG_SUCCESS) {
+ dev_err(drv2624->dev,
+ "Calibration fail\n");
+ } else {
+ drv2624->auto_cal_result.cal_comp =
+ drv2624_reg_read(drv2624,
+ DRV2624_REG_CAL_COMP);
+ drv2624->auto_cal_result.cal_bemf =
+ drv2624_reg_read(drv2624,
+ DRV2624_REG_CAL_BEMF);
+ drv2624->auto_cal_result.cal_gain =
+ drv2624_reg_read(drv2624,
+ DRV2624_REG_CAL_COMP)
+ & BEMFGAIN_MASK;
+ dev_dbg(drv2624->dev,
+ "AutoCal : Comp=0x%x, Bemf=0x%x, Gain=0x%x\n",
+ drv2624->auto_cal_result.cal_comp,
+ drv2624->auto_cal_result.cal_bemf,
+ drv2624->auto_cal_result.cal_gain);
+ }
+ } else if (mode == MODE_DIAGNOSTIC) {
+ drv2624->diag_result.result = status;
+ if ((status & DIAG_MASK) != DIAG_SUCCESS) {
+ dev_err(drv2624->dev,
+ "Diagnostic fail\n");
+ } else {
+ drv2624->diag_result.diagz =
+ drv2624_reg_read(drv2624,
+ DRV2624_REG_DIAG_Z);
+ drv2624->diag_result.diagk =
+ drv2624_reg_read(drv2624,
+ DRV2624_REG_DIAG_K);
+ dev_dbg(drv2624->dev,
+ "Diag : ZResult=0x%x, CurrentK=0x%x\n",
+ drv2624->diag_result.diagz,
+ drv2624->diag_result.diagk);
+ }
+ } else if (mode == MODE_WAVEFORM_SEQUENCER) {
+ dev_dbg(drv2624->dev,
+ "Waveform Sequencer Playback finished\n");
+ } else if (mode == MODE_RTP) {
+ dev_dbg(drv2624->dev, "RTP IRQ\n");
+ }
+ }
+
+ if ((mode != MODE_RTP) && drv2624->vibrator_playing) {
+ dev_info(drv2624->dev, "release wklock\n");
+ drv2624->vibrator_playing = false;
+ pm_relax(drv2624->dev);
+ }
+
+ drv2624->work_mode &= ~WORK_IRQ;
+ }
+
+ if (drv2624->work_mode & WORK_VIBRATOR) {
+ drv2624_stop(drv2624);
+ drv2624->work_mode &= ~WORK_VIBRATOR;
+ }
+err:
+
+ mutex_unlock(&drv2624->lock);
+}
+
+
+static int drv2624_set_waveform(struct drv2624_data *drv2624,
+ struct drv2624_waveform_sequencer *sequencer)
+{
+ int ret = 0;
+ int i = 0;
+ unsigned char loop[2] = { 0 };
+ unsigned char effects[DRV2624_SEQUENCER_SIZE] = { 0 };
+ unsigned char len = 0;
+
+ for (i = 0; i < DRV2624_SEQUENCER_SIZE; i++) {
+ len++;
+ if (sequencer->waveform[i].effect != 0) {
+ if (i < 4)
+ loop[0] |=
+ (sequencer->waveform[i].loop << (2 * i));
+ else
+ loop[1] |=
+ (sequencer->waveform[i].loop << (2 * (i - 4)));
+
+ effects[i] = sequencer->waveform[i].effect;
+ } else {
+ break;
+ }
+ }
+
+ if (len == 1)
+ ret = drv2624_reg_write(drv2624, DRV2624_REG_SEQUENCER_1, 0);
+ else
+ ret =
+ drv2624_bulk_write(drv2624, DRV2624_REG_SEQUENCER_1,
+ effects, len);
+
+ if (ret < 0)
+ dev_err(drv2624->dev, "sequence error\n");
+
+ if ((ret >= 0) && (len > 1)) {
+ if ((len - 1) <= 4)
+ drv2624_reg_write(drv2624,
+ DRV2624_REG_SEQ_LOOP_1, loop[0]);
+ else
+ drv2624_bulk_write(drv2624,
+ DRV2624_REG_SEQ_LOOP_1, loop, 2);
+ }
+
+ return ret;
+}
+
+static int fw_chksum(const struct firmware *fw)
+{
+ int sum = 0;
+ int i = 0;
+ int size = fw->size;
+ const unsigned char *buffer = fw->data;
+
+ for (i = 0; i < size; i++) {
+ if (!((i > 11) && (i < 16)))
+ sum += buffer[i];
+ }
+
+ return sum;
+}
+
+/* drv2624_firmware_load: This function is called by the
+ * request_firmware_nowait function as soon
+ * as the firmware has been loaded from the file.
+ * The firmware structure contains the data and$
+ * the size of the firmware loaded.
+ * @fw: pointer to firmware file to be dowloaded
+ * @context: pointer variable to drv2624 data
+ *
+ *
+ */
+static void drv2624_firmware_load(const struct firmware *fw, void *context)
+{
+ struct drv2624_data *drv2624 = context;
+ int size = 0, fwsize = 0, i = 0;
+ const unsigned char *buffer = NULL;
+
+ if (!context)
+ return;
+
+ mutex_lock(&drv2624->lock);
+
+ if (fw) {
+ buffer = fw->data;
+ size = fw->size;
+ if (size > 1024) {
+ dev_err(drv2624->dev,
+ "%s, ERROR!! firmware size %d too big\n",
+ __func__, size);
+ } else {
+ memcpy(&drv2624->fw_header, buffer,
+ sizeof(struct drv2624_fw_header));
+ if ((drv2624->fw_header.fw_magic != DRV2624_MAGIC) ||
+ (drv2624->fw_header.fw_size != size) ||
+ (drv2624->fw_header.fw_chksum != fw_chksum(fw))) {
+ dev_err(drv2624->dev,
+ "%s, ERROR!! firmware not right:Magic=0x%x,Size=%d,chksum=0x%x\n",
+ __func__,
+ drv2624->fw_header.fw_magic,
+ drv2624->fw_header.fw_size,
+ drv2624->fw_header.fw_chksum);
+ } else {
+ dev_info(drv2624->dev, "%s, firmware good\n",
+ __func__);
+
+ buffer += sizeof(struct drv2624_fw_header);
+
+ drv2624_reg_write(drv2624,
+ DRV2624_REG_RAM_ADDR_UPPER,
+ 0);
+ drv2624_reg_write(drv2624,
+ DRV2624_REG_RAM_ADDR_LOWER,
+ 0);
+
+ fwsize =
+ size - sizeof(struct drv2624_fw_header);
+
+ for (i = 0; i < fwsize; i++) {
+ drv2624_reg_write(drv2624,
+ DRV2624_REG_RAM_DATA,
+ buffer[i]);
+ }
+ }
+ }
+ } else {
+ dev_err(drv2624->dev, "%s, ERROR!! firmware not found\n",
+ __func__);
+ }
+
+ release_firmware(fw);
+
+ mutex_unlock(&drv2624->lock);
+}
+
+static int haptics_init(struct drv2624_data *drv2624)
+{
+ int ret;
+
+ drv2624->vibrator_playing = false;
+ drv2624->led_dev.name = "vibrator";
+ drv2624->led_dev.max_brightness = LED_FULL;
+ drv2624->led_dev.brightness_set = vibrator_enable;
+ drv2624->led_dev.flags = LED_BRIGHTNESS_FAST;
+
+ ret = led_classdev_register(drv2624->dev, &drv2624->led_dev);
+ if (ret) {
+ dev_err(drv2624->dev,
+ "drv2624: fail to create led classdev\n");
+ return ret;
+ }
+
+ device_init_wakeup(drv2624->dev, false);
+ mutex_init(&drv2624->lock);
+
+ drv2624->drv2624_wq =
+ alloc_ordered_workqueue("drv2624_wq", WQ_HIGHPRI);
+ if (!drv2624->drv2624_wq) {
+ dev_err(drv2624->dev,
+ "drv2624: fail to alloc_workqueue for drv2624_wq\n");
+ return -ENOMEM;
+ }
+
+ INIT_WORK(&drv2624->vibrator_work, vibrator_work_routine);
+ INIT_WORK(&drv2624->work, drv2624_haptics_work);
+ INIT_WORK(&drv2624->stop_work, drv2624_haptics_stopwork);
+
+ return 0;
+}
+
+static void dev_init_platform_data(struct drv2624_data *drv2624)
+{
+ struct drv2624_platform_data *pdata = &drv2624->plat_data;
+ struct actuator_data actuator = pdata->actuator;
+ unsigned char value_temp = 0;
+ unsigned char mask_temp = 0;
+
+ drv2624_set_bits(drv2624,
+ DRV2624_REG_MODE, PINFUNC_MASK,
+ (PINFUNC_INT << PINFUNC_SHIFT));
+
+ if ((actuator.actuator_type == ERM) ||
+ (actuator.actuator_type == LRA)) {
+ mask_temp |= ACTUATOR_MASK;
+ value_temp |= (actuator.actuator_type << ACTUATOR_SHIFT);
+ }
+
+ if ((pdata->loop == CLOSE_LOOP) ||
+ (pdata->loop == OPEN_LOOP)) {
+ mask_temp |= LOOP_MASK;
+ value_temp |= (pdata->loop << LOOP_SHIFT);
+ }
+
+ if (value_temp != 0) {
+ drv2624_set_bits(drv2624,
+ DRV2624_REG_CONTROL1,
+ mask_temp | AUTOBRK_OK_MASK,
+ value_temp | AUTOBRK_OK_ENABLE);
+ }
+
+ value_temp = 0;
+ if (actuator.actuator_type == ERM)
+ value_temp = LIB_ERM;
+ else if (actuator.actuator_type == LRA)
+ value_temp = LIB_LRA;
+ if (value_temp != 0) {
+ drv2624_set_bits(drv2624,
+ DRV2624_REG_CONTROL2, LIB_MASK,
+ value_temp << LIB_SHIFT);
+ }
+
+ if (actuator.rated_voltage != 0) {
+ drv2624_reg_write(drv2624,
+ DRV2624_REG_RATED_VOLTAGE,
+ actuator.rated_voltage);
+ } else {
+ dev_err(drv2624->dev, "%s, ERROR Rated ZERO\n", __func__);
+ }
+
+ if (actuator.over_drive_clamp_voltage != 0) {
+ drv2624_reg_write(drv2624,
+ DRV2624_REG_OVERDRIVE_CLAMP,
+ actuator.over_drive_clamp_voltage);
+ } else {
+ dev_err(drv2624->dev,
+ "%s, ERROR OverDriveVol ZERO\n", __func__);
+ }
+
+ if (actuator.actuator_type == LRA) {
+ unsigned char drive_time =
+ 5 * (1000 - actuator.lra_freq) / actuator.lra_freq;
+ unsigned short open_loop_period =
+ (unsigned short)((unsigned int)1000000000 /
+ (24619 * actuator.lra_freq));
+
+ if (actuator.lra_freq < 125)
+ drive_time |= (MINFREQ_SEL_45HZ << MINFREQ_SEL_SHIFT);
+
+ drv2624_set_bits(drv2624,
+ DRV2624_REG_DRIVE_TIME,
+ DRIVE_TIME_MASK | MINFREQ_SEL_MASK, drive_time);
+
+ if (actuator.ol_lra_freq > -1)
+ open_loop_period =
+ (unsigned short)((unsigned int)1000000000 /
+ (24619 * actuator.ol_lra_freq));
+
+ drv2624_set_bits(drv2624,
+ DRV2624_REG_OL_PERIOD_H, 0x03,
+ (open_loop_period & 0x0300) >> 8);
+ drv2624_reg_write(drv2624, DRV2624_REG_OL_PERIOD_L,
+ (open_loop_period & 0x00ff));
+
+ dev_info(drv2624->dev,
+ "%s, LRA = %d, drive_time=0x%x\n",
+ __func__, actuator.lra_freq, drive_time);
+
+ if (actuator.lra_wave_shape > -1)
+ drv2624_set_bits(drv2624,
+ DRV2624_REG_LRA_OL_CTRL,
+ LRA_WAVE_SHAPE_MASK,
+ actuator.lra_wave_shape);
+ }
+
+ if (actuator.voltage_comp > -1)
+ drv2624_reg_write(drv2624, DRV2624_REG_CAL_COMP,
+ actuator.voltage_comp);
+
+ if (actuator.bemf_factor > -1)
+ drv2624_reg_write(drv2624, DRV2624_REG_CAL_BEMF,
+ actuator.bemf_factor);
+
+ if (actuator.bemf_gain > -1)
+ drv2624_set_bits(drv2624, DRV2624_REG_LOOP_CONTROL,
+ BEMFGAIN_MASK, actuator.bemf_gain);
+
+ if (actuator.blanking_time > -1)
+ drv2624_set_bits(drv2624, DRV2624_REG_BLK_IDISS_TIME,
+ BLANKING_TIME_MASK,
+ actuator.blanking_time << BLANKING_TIME_SHIFT);
+
+ if (actuator.idiss_time > -1)
+ drv2624_set_bits(drv2624, DRV2624_REG_BLK_IDISS_TIME,
+ IDISS_TIME_MASK, actuator.idiss_time);
+
+ if (actuator.zc_det_time > -1)
+ drv2624_set_bits(drv2624, DRV2624_REG_ZC_OD_TIME,
+ ZC_DET_TIME_MASK, actuator.zc_det_time);
+
+ if (actuator.waveform_interval > -1)
+ drv2624_set_bits(drv2624, DRV2624_REG_CONTROL2, INTERVAL_MASK,
+ actuator.waveform_interval << INTERVAL_SHIFT);
+
+}
+
+static irqreturn_t drv2624_irq_handler(int irq, void *dev_id)
+{
+ struct drv2624_data *drv2624 = (struct drv2624_data *)dev_id;
+
+ drv2624->work_mode |= WORK_IRQ;
+
+ schedule_work(&drv2624->vibrator_work);
+
+ return IRQ_HANDLED;
+}
+
+static int drv2624_parse_dt(struct device *dev, struct drv2624_data *drv2624)
+{
+ struct device_node *np = dev->of_node;
+ struct drv2624_platform_data *pdata = &drv2624->plat_data;
+ int ret = 0;
+ unsigned int value;
+
+ pdata->gpio_nrst = of_get_named_gpio(np, "ti,reset-gpio", 0);
+ if (!gpio_is_valid(pdata->gpio_nrst)) {
+ dev_err(drv2624->dev,
+ "Looking up %s property in node %s failed %d\n",
+ "ti,reset-gpio", np->full_name, pdata->gpio_nrst);
+ ret = -EINVAL;
+ goto drv2624_parse_dt_out;
+ }
+
+ /* irq-gpio is optional */
+ pdata->gpio_int = of_get_named_gpio(np, "ti,irq-gpio", 0);
+ if (!gpio_is_valid(pdata->gpio_int)) {
+ dev_warn(drv2624->dev,
+ "Looking up %s property in node %s failed %d\n",
+ "ti,irq-gpio", np->full_name, pdata->gpio_int);
+ }
+
+ ret = of_property_read_u32(np, "ti,smart-loop", &value);
+ if (ret) {
+ dev_err(drv2624->dev,
+ "Looking up %s property in node %s failed %d\n",
+ "ti,smart-loop", np->full_name, ret);
+ ret = -EINVAL;
+ goto drv2624_parse_dt_out;
+ }
+
+ pdata->loop = value & 0x01;
+ dev_dbg(drv2624->dev, "ti,smart-loop=%d\n", pdata->loop);
+
+ ret = of_property_read_u32(np, "ti,actuator", &value);
+ if (ret) {
+ dev_err(drv2624->dev,
+ "Looking up %s property in node %s failed %d\n",
+ "ti,actuator", np->full_name, ret);
+ ret = -EINVAL;
+ goto drv2624_parse_dt_out;
+ }
+
+ pdata->actuator.actuator_type = value & 0x01;
+ dev_dbg(drv2624->dev, "ti,actuator=%d\n",
+ pdata->actuator.actuator_type);
+
+ ret = of_property_read_u32(np, "ti,rated-voltage", &value);
+ if (ret) {
+ dev_err(drv2624->dev,
+ "Looking up %s property in node %s failed %d\n",
+ "ti,rated-voltage", np->full_name, ret);
+ ret = -EINVAL;
+ goto drv2624_parse_dt_out;
+ }
+
+ pdata->actuator.rated_voltage = value;
+ dev_dbg(drv2624->dev, "ti,rated-voltage=0x%x\n",
+ pdata->actuator.rated_voltage);
+
+ ret = of_property_read_u32(np, "ti,odclamp-voltage", &value);
+ if (ret) {
+ dev_err(drv2624->dev,
+ "Looking up %s property in node %s failed %d\n",
+ "ti,odclamp-voltage", np->full_name, ret);
+ ret = -EINVAL;
+ goto drv2624_parse_dt_out;
+ }
+
+ pdata->actuator.over_drive_clamp_voltage = value;
+ dev_dbg(drv2624->dev, "ti,odclamp-voltage=0x%x\n",
+ pdata->actuator.over_drive_clamp_voltage);
+
+ if (pdata->actuator.actuator_type == LRA) {
+ ret = of_property_read_u32(np, "ti,lra-frequency", &value);
+ if (ret) {
+ dev_err(drv2624->dev,
+ "Looking up %s property in node %s failed %d\n",
+ "ti,lra-frequency", np->full_name, ret);
+ ret = -EINVAL;
+ goto drv2624_parse_dt_out;
+ } else {
+ if ((value >= 45) && (value <= 300)) {
+ pdata->actuator.lra_freq = value;
+ dev_dbg(drv2624->dev,
+ "ti,lra-frequency=%d\n",
+ pdata->actuator.lra_freq);
+ } else {
+ dev_err(drv2624->dev,
+ "ERROR, ti,lra-frequency=%d, out of range\n",
+ value);
+ ret = -EINVAL;
+ goto drv2624_parse_dt_out;
+ }
+ }
+ }
+
+ /* actuator properties below are optional to have */
+
+ if (!of_property_read_u32(np, "ti,voltage-comp", &value))
+ pdata->actuator.voltage_comp = value;
+ else
+ pdata->actuator.voltage_comp = -1;
+
+ dev_dbg(drv2624->dev, "ti,voltage-comp=%d\n",
+ pdata->actuator.voltage_comp);
+
+ if (!of_property_read_u32(np, "ti,ol-lra-frequency", &value)) {
+ if ((value >= 45) && (value <= 300)) {
+ pdata->actuator.ol_lra_freq = value;
+ } else {
+ pdata->actuator.ol_lra_freq = -1;
+ dev_err(drv2624->dev,
+ "ERROR, ti,ol-lra-frequency=%d, out of range\n",
+ value);
+ }
+ } else {
+ pdata->actuator.ol_lra_freq = -1;
+ }
+
+ dev_dbg(drv2624->dev, "ti,ol-lra-frequency=%d\n",
+ pdata->actuator.ol_lra_freq);
+
+ if (!of_property_read_u32(np, "ti,bemf-factor", &value))
+ pdata->actuator.bemf_factor = value;
+ else
+ pdata->actuator.bemf_factor = -1;
+
+ dev_dbg(drv2624->dev, "ti,bemf-factor=%d\n",
+ pdata->actuator.bemf_factor);
+
+ if (!of_property_read_u32(np, "ti,bemf-gain", &value))
+ pdata->actuator.bemf_gain = value;
+ else
+ pdata->actuator.bemf_gain = -1;
+
+ dev_dbg(drv2624->dev, "ti,bemf-gain=%d\n",
+ pdata->actuator.bemf_gain);
+
+ if (!of_property_read_u32(np, "ti,blanking-time", &value))
+ pdata->actuator.blanking_time = value;
+ else
+ pdata->actuator.blanking_time = -1;
+
+ dev_dbg(drv2624->dev, "ti,blanking-time=%d\n",
+ pdata->actuator.ol_lra_freq);
+
+ if (!of_property_read_u32(np, "ti,idiss-time", &value))
+ pdata->actuator.idiss_time = value;
+ else
+ pdata->actuator.idiss_time = -1;
+
+ dev_dbg(drv2624->dev, "ti,idiss-time=%d\n",
+ pdata->actuator.idiss_time);
+
+ if (!of_property_read_u32(np, "ti,zc-det-time", &value))
+ pdata->actuator.zc_det_time = value;
+ else
+ pdata->actuator.zc_det_time = -1;
+
+ dev_dbg(drv2624->dev, "ti,zc-det-time=%d\n",
+ pdata->actuator.zc_det_time);
+
+ if (!of_property_read_u32(np, "ti,lra-wave-shape", &value))
+ pdata->actuator.lra_wave_shape = value;
+ else
+ pdata->actuator.lra_wave_shape = -1;
+
+ dev_dbg(drv2624->dev, "ti,lra-wave-shape=%d\n",
+ pdata->actuator.lra_wave_shape);
+
+ if (!of_property_read_u32(np, "ti,waveform-interval", &value))
+ pdata->actuator.waveform_interval = value;
+ else
+ pdata->actuator.waveform_interval = -1;
+
+ dev_dbg(drv2624->dev, "ti,waveform-interval=%d\n",
+ pdata->actuator.waveform_interval);
+
+drv2624_parse_dt_out:
+ return ret;
+}
+
+static const struct reg_default drv2624_reg_defaults[] = {
+ { DRV2624_REG_ID, 0x03 },
+ { DRV2624_REG_STATUS, 0x00 },
+ { DRV2624_REG_INT_ENABLE, INT_MASK_ALL },
+ { DRV2624_REG_DIAG_Z, 0x00 },
+ { DRV2624_REG_MODE, 0x44 },
+ { DRV2624_REG_LRA_PERIOD_H, 0x00 },
+ { DRV2624_REG_LRA_PERIOD_L, 0x00 },
+ { DRV2624_REG_CONTROL1, 0x88 },
+ { DRV2624_REG_GO, 0x00 },
+ { DRV2624_REG_CONTROL2, 0x00 },
+ { DRV2624_REG_RTP_INPUT, 0x7F },
+ { DRV2624_REG_SEQUENCER_1, 0x01 },
+ { DRV2624_REG_SEQ_LOOP_1, 0x00 },
+ { DRV2624_REG_SEQ_LOOP_2, 0x00 },
+ { DRV2624_REG_MAIN_LOOP, 0x00 },
+ { DRV2624_REG_RATED_VOLTAGE, 0x3F },
+ { DRV2624_REG_OVERDRIVE_CLAMP, 0x89 },
+ { DRV2624_REG_CAL_COMP, 0x0D },
+ { DRV2624_REG_CAL_BEMF, 0x6D },
+ { DRV2624_REG_LOOP_CONTROL, 0x36 },
+ { DRV2624_REG_DRIVE_TIME, 0x10 },
+ { DRV2624_REG_BLK_IDISS_TIME, 0x11 },
+ { DRV2624_REG_ZC_OD_TIME, 0x0C },
+ { DRV2624_REG_LRA_OL_CTRL, 0x00 },
+ { DRV2624_REG_OL_PERIOD_H, 0x00 },
+ { DRV2624_REG_OL_PERIOD_L, 0xC6 },
+ { DRV2624_REG_DIAG_K, 0x55 },
+ { DRV2624_REG_RAM_ADDR_UPPER, 0x00 },
+ { DRV2624_REG_RAM_ADDR_LOWER, 0x00 },
+ { DRV2624_REG_RAM_DATA, 0x32 },
+};
+
+static bool drv2624_is_volatile_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case DRV2624_REG_GO:
+ case DRV2624_REG_LRA_PERIOD_H:
+ case DRV2624_REG_LRA_PERIOD_L:
+ case DRV2624_REG_LOOP_CONTROL:
+ case DRV2624_REG_DIAG_Z:
+ case DRV2624_REG_DIAG_K:
+ case DRV2624_REG_CAL_COMP:
+ case DRV2624_REG_CAL_BEMF:
+ case DRV2624_REG_RAM_ADDR_UPPER:
+ case DRV2624_REG_RAM_ADDR_LOWER:
+ case DRV2624_REG_RAM_DATA:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static bool drv2624_is_precious_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case DRV2624_REG_STATUS:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static bool drv2624_is_writeable_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case DRV2624_REG_ID:
+ case DRV2624_REG_STATUS:
+ case DRV2624_REG_DIAG_Z:
+ case DRV2624_REG_DIAG_K:
+ case DRV2624_REG_LRA_PERIOD_H:
+ case DRV2624_REG_LRA_PERIOD_L:
+ return false;
+ default:
+ return true;
+ }
+}
+
+static struct regmap_config drv2624_i2c_regmap = {
+ .reg_bits = 8,
+ .val_bits = 8,
+ .reg_defaults = drv2624_reg_defaults,
+ .num_reg_defaults = ARRAY_SIZE(drv2624_reg_defaults),
+ .volatile_reg = drv2624_is_volatile_reg,
+ .precious_reg = drv2624_is_precious_reg,
+ .writeable_reg = drv2624_is_writeable_reg,
+ .max_register = DRV2624_REG_RAM_DATA,
+ .cache_type = REGCACHE_RBTREE,
+ .can_multi_write = true,
+};
+
+static ssize_t rtp_input_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int rtp_input;
+
+ mutex_lock(&drv2624->lock);
+ rtp_input = drv2624_reg_read(drv2624, DRV2624_REG_RTP_INPUT);
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", rtp_input);
+}
+
+static ssize_t rtp_input_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int ret;
+ char rtp_input;
+
+ ret = kstrtos8(buf, 10, &rtp_input);
+ if (ret) {
+ pr_err("Invalid input for rtp_input: ret = %d\n", ret);
+ return ret;
+ }
+
+ mutex_lock(&drv2624->lock);
+ drv2624_reg_write(drv2624, DRV2624_REG_RTP_INPUT, rtp_input);
+ mutex_unlock(&drv2624->lock);
+
+ return count;
+}
+
+static ssize_t mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int mode;
+
+ mutex_lock(&drv2624->lock);
+ mode = drv2624_get_mode(drv2624);
+ mutex_unlock(&drv2624->lock);
+
+ if (mode >= ARRAY_SIZE(drv2624_modes) || mode < 0) {
+ pr_err("Invalid mode: mode = %d\n", mode);
+ return snprintf(buf, PAGE_SIZE, "%d\n", mode);
+ }
+
+ return sprintf(buf, "%s\n", drv2624_modes[mode]);
+}
+
+static ssize_t mode_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ char mode_name[25];
+ size_t len;
+ unsigned char new_mode;
+
+ mode_name[sizeof(mode_name) - 1] = '\0';
+ strlcpy(mode_name, buf, sizeof(mode_name) - 1);
+ len = strlen(mode_name);
+
+ if (len && mode_name[len - 1] == '\n')
+ mode_name[len - 1] = '\0';
+
+ for (new_mode = 0; new_mode < ARRAY_SIZE(drv2624_modes); new_mode++) {
+ if (!strcmp(mode_name, drv2624_modes[new_mode])) {
+ mutex_lock(&drv2624->lock);
+ drv2624_change_mode(drv2624, new_mode);
+ mutex_unlock(&drv2624->lock);
+ break;
+ }
+ }
+
+ return count;
+}
+
+static ssize_t loop_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int loop;
+
+ mutex_lock(&drv2624->lock);
+ loop = drv2624_reg_read(drv2624, DRV2624_REG_MAIN_LOOP) &
+ MAIN_LOOP_MASK;
+ mutex_unlock(&drv2624->lock);
+
+ if (loop == MAIN_LOOP_MASK)
+ loop = -1; /* infinite */
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", loop);
+}
+
+static ssize_t loop_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int ret;
+ int loop;
+
+ ret = kstrtoint(buf, 10, &loop);
+ if (ret) {
+ pr_err("Invalid input for loop: ret = %d\n", ret);
+ return ret;
+ }
+
+ loop = max(min(loop, MAIN_LOOP_MASK), -1);
+
+ if (loop == -1)
+ loop = MAIN_LOOP_MASK; /*infinite */
+
+ mutex_lock(&drv2624->lock);
+ drv2624_reg_write(drv2624, DRV2624_REG_MAIN_LOOP, loop);
+ mutex_unlock(&drv2624->lock);
+
+ return count;
+}
+
+static ssize_t interval_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int interval;
+
+ mutex_lock(&drv2624->lock);
+ interval = drv2624_reg_read(drv2624, DRV2624_REG_CONTROL2);
+ interval = ((interval & INTERVAL_MASK) >> INTERVAL_SHIFT);
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", interval);
+}
+
+static ssize_t interval_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int ret;
+ int interval;
+
+ ret = kstrtoint(buf, 10, &interval);
+ if (ret) {
+ pr_err("Invalid input for loop: ret = %d\n", ret);
+ return ret;
+ }
+
+ interval = max(min(interval, 1), 0);
+
+ mutex_lock(&drv2624->lock);
+ drv2624_set_bits(drv2624, DRV2624_REG_CONTROL2, INTERVAL_MASK,
+ interval << INTERVAL_SHIFT);
+ mutex_unlock(&drv2624->lock);
+
+ return count;
+}
+
+static ssize_t scale_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int interval;
+
+ mutex_lock(&drv2624->lock);
+ interval = drv2624_reg_read(drv2624, DRV2624_REG_CONTROL2);
+ interval = interval & SCALE_MASK;
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", interval);
+}
+
+static ssize_t scale_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int ret;
+ int interval;
+
+ ret = kstrtoint(buf, 10, &interval);
+ if (ret) {
+ pr_err("Invalid input for loop: ret = %d\n", ret);
+ return ret;
+ }
+
+ interval = max(min(interval, 3), 0);
+
+ mutex_lock(&drv2624->lock);
+ drv2624_set_bits(drv2624, DRV2624_REG_CONTROL2, SCALE_MASK, interval);
+ mutex_unlock(&drv2624->lock);
+
+ return count;
+}
+
+static ssize_t ctrl_loop_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int ctrl_loop;
+
+ mutex_lock(&drv2624->lock);
+ ctrl_loop = drv2624_reg_read(drv2624, DRV2624_REG_CONTROL1);
+ ctrl_loop = (ctrl_loop & LOOP_MASK) >> LOOP_SHIFT;
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", ctrl_loop);
+}
+
+static ssize_t ctrl_loop_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int ret;
+ int ctrl_loop;
+
+ ret = kstrtoint(buf, 10, &ctrl_loop);
+ if (ret) {
+ pr_err("Invalid input for loop: ret = %d\n", ret);
+ return ret;
+ }
+
+ ctrl_loop = max(min(ctrl_loop, 1), 0);
+
+ mutex_lock(&drv2624->lock);
+ drv2624_set_bits(drv2624, DRV2624_REG_CONTROL1, LOOP_MASK,
+ ctrl_loop << LOOP_SHIFT);
+ mutex_unlock(&drv2624->lock);
+
+ return count;
+}
+
+static ssize_t set_sequencer_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ struct drv2624_waveform_sequencer sequencer;
+ int n;
+
+ memset(&sequencer, 0, sizeof(sequencer));
+
+ n = sscanf(buf, "%hhu %hhu %hhu %hhu %hhu %hhu %hhu %hhu %hhu %hhu %hhu %hhu %hhu %hhu %hhu %hhu",
+ &sequencer.waveform[0].effect, &sequencer.waveform[0].loop,
+ &sequencer.waveform[1].effect, &sequencer.waveform[1].loop,
+ &sequencer.waveform[2].effect, &sequencer.waveform[2].loop,
+ &sequencer.waveform[3].effect, &sequencer.waveform[3].loop,
+ &sequencer.waveform[4].effect, &sequencer.waveform[4].loop,
+ &sequencer.waveform[5].effect, &sequencer.waveform[5].loop,
+ &sequencer.waveform[6].effect, &sequencer.waveform[6].loop,
+ &sequencer.waveform[7].effect, &sequencer.waveform[7].loop);
+ if (n > DRV2624_SEQUENCER_SIZE * 2)
+ return -EINVAL;
+
+ mutex_lock(&drv2624->lock);
+ drv2624_set_waveform(drv2624, &sequencer);
+ mutex_unlock(&drv2624->lock);
+
+ return count;
+}
+
+static ssize_t od_clamp_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int od_clamp;
+
+ mutex_lock(&drv2624->lock);
+ od_clamp = drv2624_reg_read(drv2624, DRV2624_REG_OVERDRIVE_CLAMP);
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", od_clamp);
+}
+
+static ssize_t od_clamp_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int ret;
+ unsigned char od_clamp;
+
+ ret = kstrtou8(buf, 10, &od_clamp);
+ if (ret) {
+ pr_err("Invalid input for rtp_input: ret = %d\n", ret);
+ return ret;
+ }
+
+ mutex_lock(&drv2624->lock);
+ drv2624_reg_write(drv2624, DRV2624_REG_OVERDRIVE_CLAMP, od_clamp);
+ mutex_unlock(&drv2624->lock);
+
+ return count;
+}
+
+static ssize_t diag_result_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int diag_z, diag_k;
+
+ mutex_lock(&drv2624->lock);
+ diag_z = drv2624_reg_read(drv2624, DRV2624_REG_DIAG_Z);
+ diag_k = drv2624_reg_read(drv2624, DRV2624_REG_DIAG_K);
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "z=%d k=%d\n", diag_z, diag_k);
+}
+
+static ssize_t autocal_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ u32 autocal_comp, autocal_bemf, autocal_gain;
+
+ mutex_lock(&drv2624->lock);
+ autocal_comp = drv2624_reg_read(drv2624, DRV2624_REG_CAL_COMP);
+ autocal_bemf = drv2624_reg_read(drv2624, DRV2624_REG_CAL_BEMF);
+ autocal_gain =
+ drv2624_reg_read(drv2624, DRV2624_REG_LOOP_CONTROL) &
+ BEMFGAIN_MASK;
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "%d %d %d\n",
+ autocal_comp, autocal_bemf, autocal_gain);
+}
+
+static ssize_t autocal_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int n;
+ unsigned char comp, bemf, bemfgain;
+
+ n = sscanf(buf, "%hhu %hhu %hhu", &comp, &bemf, &bemfgain);
+ if (n != 3)
+ return -EINVAL;
+
+ mutex_lock(&drv2624->lock);
+ drv2624_reg_write(drv2624, DRV2624_REG_CAL_COMP, comp);
+ drv2624_reg_write(drv2624, DRV2624_REG_CAL_BEMF, bemf);
+ drv2624_set_bits(drv2624, DRV2624_REG_LOOP_CONTROL,
+ BEMFGAIN_MASK, bemfgain);
+ mutex_unlock(&drv2624->lock);
+
+ return count;
+}
+
+static ssize_t lra_period_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ u32 lra_period;
+
+ mutex_lock(&drv2624->lock);
+ lra_period = drv2624_reg_read(drv2624, DRV2624_REG_LRA_PERIOD_H) & 0x03;
+ lra_period = (lra_period << 8) |
+ drv2624_reg_read(drv2624, DRV2624_REG_LRA_PERIOD_L);
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", lra_period);
+}
+
+static ssize_t status_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ u32 status;
+
+ mutex_lock(&drv2624->lock);
+ status = drv2624_reg_read(drv2624, DRV2624_REG_STATUS);
+ /* Ignore Bit 6-5 reserved, Bit 4 PRG_ERR and BIT 3 PROC_DONE */
+ status &= ~((1 << 6) | (1 << 5) | (1 << 4) | (1 << 3));
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", status);
+}
+
+static ssize_t ol_lra_period_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ u32 ol_lra_period;
+
+ mutex_lock(&drv2624->lock);
+ ol_lra_period =
+ drv2624_reg_read(drv2624, DRV2624_REG_OL_PERIOD_H) & 0x03;
+ ol_lra_period = (ol_lra_period << 8) |
+ drv2624_reg_read(drv2624, DRV2624_REG_OL_PERIOD_L);
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", ol_lra_period);
+}
+
+static ssize_t ol_lra_period_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int ret;
+ u32 ol_lra_period;
+
+ ret = kstrtou32(buf, 10, &ol_lra_period);
+ if (ret) {
+ pr_err("Invalid input for ol_lra_period: ret = %d\n", ret);
+ return ret;
+ }
+
+ mutex_lock(&drv2624->lock);
+ drv2624_reg_write(drv2624, DRV2624_REG_OL_PERIOD_H,
+ (ol_lra_period >> 8) & 0x03);
+ drv2624_reg_write(drv2624, DRV2624_REG_OL_PERIOD_L,
+ ol_lra_period & 0xFF);
+ mutex_unlock(&drv2624->lock);
+
+ return count;
+}
+
+static ssize_t lra_wave_shape_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ u32 lra_wave_shape;
+
+ mutex_lock(&drv2624->lock);
+ lra_wave_shape =
+ drv2624_reg_read(drv2624,
+ DRV2624_REG_LRA_OL_CTRL) & LRA_WAVE_SHAPE_MASK;
+ mutex_unlock(&drv2624->lock);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", lra_wave_shape);
+}
+
+static ssize_t lra_wave_shape_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+ int ret;
+ u32 lra_wave_shape;
+
+ ret = kstrtou32(buf, 10, &lra_wave_shape);
+ if (ret) {
+ dev_err(dev,
+ "Invalid input for ol_lra_period: ret = %d\n", ret);
+ return ret;
+ }
+
+ mutex_lock(&drv2624->lock);
+ drv2624_set_bits(drv2624, DRV2624_REG_LRA_OL_CTRL,
+ LRA_WAVE_SHAPE_MASK, lra_wave_shape);
+ mutex_unlock(&drv2624->lock);
+
+ return count;
+}
+
+static DEVICE_ATTR(rtp_input, 0660, rtp_input_show, rtp_input_store);
+static DEVICE_ATTR(mode, 0660, mode_show, mode_store);
+static DEVICE_ATTR(loop, 0660, loop_show, loop_store);
+static DEVICE_ATTR(interval, 0660, interval_show, interval_store);
+static DEVICE_ATTR(scale, 0660, scale_show, scale_store);
+static DEVICE_ATTR(ctrl_loop, 0660, ctrl_loop_show, ctrl_loop_store);
+static DEVICE_ATTR(set_sequencer, 0660, NULL, set_sequencer_store);
+static DEVICE_ATTR(od_clamp, 0660, od_clamp_show, od_clamp_store);
+static DEVICE_ATTR(diag_result, 0600, diag_result_show, NULL);
+static DEVICE_ATTR(autocal, 0660, autocal_show, autocal_store);
+static DEVICE_ATTR(lra_period, 0600, lra_period_show, NULL);
+static DEVICE_ATTR(status, 0600, status_show, NULL);
+static DEVICE_ATTR(ol_lra_period, 0660, ol_lra_period_show,
+ ol_lra_period_store);
+static DEVICE_ATTR(lra_wave_shape, 0660, lra_wave_shape_show,
+ lra_wave_shape_store);
+
+static struct attribute *drv2624_fs_attrs[] = {
+ &dev_attr_rtp_input.attr,
+ &dev_attr_mode.attr,
+ &dev_attr_loop.attr,
+ &dev_attr_interval.attr,
+ &dev_attr_scale.attr,
+ &dev_attr_ctrl_loop.attr,
+ &dev_attr_set_sequencer.attr,
+ &dev_attr_od_clamp.attr,
+ &dev_attr_diag_result.attr,
+ &dev_attr_autocal.attr,
+ &dev_attr_lra_period.attr,
+ &dev_attr_status.attr,
+ &dev_attr_ol_lra_period.attr,
+ &dev_attr_lra_wave_shape.attr,
+ NULL,
+};
+
+static struct attribute_group drv2624_fs_attr_group = {
+ .attrs = drv2624_fs_attrs,
+};
+
+static int drv2624_i2c_probe(struct i2c_client *client,
+ const struct i2c_device_id *id)
+{
+ struct drv2624_data *drv2624;
+ int err = 0;
+
+ dev_info(&client->dev, "%s enter\n", __func__);
+
+ if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
+ dev_err(&client->dev, "%s:I2C check failed\n", __func__);
+ return -ENODEV;
+ }
+
+ drv2624 =
+ devm_kzalloc(&client->dev, sizeof(struct drv2624_data), GFP_KERNEL);
+ if (!drv2624) {
+ dev_err(&client->dev, "%s:no memory\n", __func__);
+ return -ENOMEM;
+ }
+
+ drv2624->dev = &client->dev;
+ i2c_set_clientdata(client, drv2624);
+ dev_set_drvdata(&client->dev, drv2624);
+
+ drv2624->regmap = devm_regmap_init_i2c(client, &drv2624_i2c_regmap);
+ if (IS_ERR(drv2624->regmap)) {
+ err = PTR_ERR(drv2624->regmap);
+ dev_err(drv2624->dev,
+ "%s:Failed to allocate register map: %d\n",
+ __func__, err);
+ return err;
+ }
+
+ if (client->dev.of_node) {
+ dev_dbg(drv2624->dev, "of node parse\n");
+ err = drv2624_parse_dt(&client->dev, drv2624);
+ if (err) {
+ dev_err(drv2624->dev,
+ "%s: platform data error\n", __func__);
+ return -ENODEV;
+ }
+ } else if (client->dev.platform_data) {
+ dev_dbg(drv2624->dev, "platform data parse\n");
+ memcpy(&drv2624->plat_data,
+ client->dev.platform_data,
+ sizeof(drv2624->plat_data));
+ } else {
+ dev_err(drv2624->dev, "%s: ERROR no platform data\n", __func__);
+ return -ENODEV;
+ }
+
+ if (gpio_is_valid(drv2624->plat_data.gpio_nrst)) {
+ err = devm_gpio_request(&client->dev,
+ drv2624->plat_data.gpio_nrst,
+ "DRV2624-NRST");
+ if (err < 0) {
+ dev_err(drv2624->dev,
+ "%s: GPIO %d request NRST error\n",
+ __func__, drv2624->plat_data.gpio_nrst);
+ return err;
+ }
+
+ gpio_direction_output(drv2624->plat_data.gpio_nrst, 0);
+ usleep_range(1000, 2000);
+ gpio_direction_output(drv2624->plat_data.gpio_nrst, 1);
+ usleep_range(1000, 2000); /* t(on) = 1ms */
+ }
+
+ err = drv2624_reg_read(drv2624, DRV2624_REG_ID);
+ if (err < 0) {
+ dev_err(drv2624->dev, "%s, i2c bus fail (%d)\n", __func__, err);
+ goto drv2624_i2c_probe_err;
+ } else {
+ dev_info(drv2624->dev, "%s, ID status (0x%x)\n", __func__, err);
+ drv2624->device_id = err;
+ }
+
+ if ((drv2624->device_id & 0xf0) != DRV2624_ID) {
+ dev_err(drv2624->dev, "%s, device_id(0x%x) fail\n",
+ __func__, drv2624->device_id);
+ goto drv2624_i2c_probe_err;
+ }
+
+ dev_init_platform_data(drv2624);
+
+ if (gpio_is_valid(drv2624->plat_data.gpio_int)) {
+ err = devm_gpio_request(&client->dev,
+ drv2624->plat_data.gpio_int,
+ "DRV2624-IRQ");
+ if (err < 0) {
+ dev_err(drv2624->dev,
+ "%s: GPIO %d request INT error\n",
+ __func__, drv2624->plat_data.gpio_int);
+ goto drv2624_i2c_probe_err;
+ }
+
+ gpio_direction_input(drv2624->plat_data.gpio_int);
+
+ drv2624->irq = gpio_to_irq(drv2624->plat_data.gpio_int);
+ dev_dbg(drv2624->dev, "irq = %d\n", drv2624->irq);
+
+ err = devm_request_threaded_irq(&client->dev, drv2624->irq,
+ drv2624_irq_handler, NULL,
+ IRQF_TRIGGER_FALLING |
+ IRQF_ONESHOT,
+ client->name, drv2624);
+
+ if (err < 0) {
+ dev_err(drv2624->dev, "request_irq failed, %d\n", err);
+ goto drv2624_i2c_probe_err;
+ }
+ drv2624_disable_irq(drv2624);
+ }
+
+ drv2624_plat_data = drv2624;
+
+ err = haptics_init(drv2624);
+ if (err)
+ goto drv2624_i2c_probe_err;
+
+ err = sysfs_create_group(&drv2624->dev->kobj, &drv2624_fs_attr_group);
+ if (err)
+ goto drv2624_i2c_probe_err;
+
+ request_firmware_nowait(THIS_MODULE, FW_ACTION_HOTPLUG, "drv2624.bin",
+ &client->dev, GFP_KERNEL, drv2624,
+ drv2624_firmware_load);
+
+ dev_info(drv2624->dev, "drv2624 probe succeeded\n");
+
+ return 0;
+
+drv2624_i2c_probe_err:
+
+ dev_err(drv2624->dev, "%s failed, err=%d\n", __func__, err);
+ return err;
+}
+
+static int drv2624_i2c_remove(struct i2c_client *client)
+{
+ struct drv2624_data *drv2624 = i2c_get_clientdata(client);
+
+ cancel_work_sync(&drv2624->vibrator_work);
+ cancel_work_sync(&drv2624->work);
+ destroy_workqueue(drv2624->drv2624_wq);
+
+ led_classdev_unregister(&drv2624->led_dev);
+
+ mutex_destroy(&drv2624->lock);
+
+ return 0;
+}
+
+static const struct i2c_device_id drv2624_i2c_id[] = {
+ {"drv2624", 0},
+ {}
+};
+
+MODULE_DEVICE_TABLE(i2c, drv2624_i2c_id);
+
+#if defined(CONFIG_OF)
+static const struct of_device_id drv2624_of_match[] = {
+ {.compatible = "ti,drv2624"},
+ {},
+};
+
+MODULE_DEVICE_TABLE(of, drv2624_of_match);
+#endif
+
+#ifdef CONFIG_PM
+static int drv2624_suspend(struct device *dev)
+{
+ struct drv2624_data *drv2624 = dev_get_drvdata(dev);
+
+ cancel_work_sync(&drv2624->vibrator_work);
+ cancel_work_sync(&drv2624->work);
+ cancel_work_sync(&drv2624->stop_work);
+ drv2624_stop(drv2624);
+
+ return 0;
+}
+
+static int drv2624_resume(struct device *dev)
+{
+ return 0;
+}
+
+#endif
+
+static SIMPLE_DEV_PM_OPS(drv2624_pm_ops, drv2624_suspend, drv2624_resume);
+
+static struct i2c_driver drv2624_i2c_driver = {
+ .driver = {
+ .name = "drv2624",
+ .owner = THIS_MODULE,
+#if defined(CONFIG_OF)
+ .of_match_table = of_match_ptr(drv2624_of_match),
+#endif
+ .pm = &drv2624_pm_ops,
+ },
+ .probe = drv2624_i2c_probe,
+ .remove = drv2624_i2c_remove,
+ .id_table = drv2624_i2c_id,
+};
+
+module_i2c_driver(drv2624_i2c_driver);
+
+MODULE_AUTHOR("Texas Instruments Inc.");
+MODULE_DESCRIPTION("DRV2624 I2C Smart Haptics driver");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/input/misc/drv2624.h b/drivers/input/misc/drv2624.h
new file mode 100644
index 0000000..6c1d8b32
--- /dev/null
+++ b/drivers/input/misc/drv2624.h
@@ -0,0 +1,328 @@
+#ifndef __DRV2624_H__
+#define __DRV2624_H__
+/*
+** =============================================================================
+** Copyright (c)2016 Texas Instruments Inc.
+**
+** This program is free software; you can redistribute it and/or
+** modify it under the terms of the GNU General Public License
+** as published by the Free Software Foundation; either version 2
+** of the License, or (at your option) any later version.
+**
+** This program is distributed in the hope that it will be useful,
+** but WITHOUT ANY WARRANTY; without even the implied warranty of
+** MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+** GNU General Public License for more details.
+**
+** File:
+** drv2624.h
+**
+** Description:
+** Header file for drv2624.c
+**
+** =============================================================================
+*/
+
+#include <linux/regmap.h>
+#include <linux/timer.h>
+#include <linux/workqueue.h>
+#include <linux/hrtimer.h>
+#include <linux/mutex.h>
+#include <linux/cdev.h>
+#include <linux/firmware.h>
+#include <linux/leds.h>
+
+#define HAPTICS_DEVICE_NAME "drv2624"
+
+#define DRV2624_REG_ID 0x00
+#define DRV2624_ID (0x02 & 0xf0)
+
+#define DRV2624_REG_STATUS 0x01
+#define DIAG_MASK 0x80
+#define DIAG_SUCCESS 0x00
+#define DIAG_SHIFT 0x07
+#define INT_MASK 0x1f
+#define PRG_ERR_MASK 0x10
+#define PROCESS_DONE_MASK 0x08
+#define ULVO_MASK 0x04
+#define OVERTEMPRATURE_MASK 0x02
+#define OVERCURRENT_MASK 0x01
+
+#define DRV2624_REG_INT_ENABLE 0x02
+#define INT_MASK_ALL 0x1f
+#define INT_ENABLE_ALL 0x00
+#define INT_ENABLE_CRITICAL 0x08
+
+#define DRV2624_REG_DIAG_Z 0x03
+
+#define DRV2624_REG_MODE 0x07
+#define WORKMODE_MASK 0x03
+#define MODE_RTP 0x00
+#define MODE_WAVEFORM_SEQUENCER 0x01
+#define MODE_DIAGNOSTIC 0x02
+#define MODE_CALIBRATION 0x03
+#define PINFUNC_MASK 0x0c
+#define PINFUNC_INT 0x02
+#define PINFUNC_SHIFT 0x02
+
+#define DRV2624_REG_LRA_PERIOD_H 0x05
+#define DRV2624_REG_LRA_PERIOD_L 0x06
+
+#define DRV2624_REG_CONTROL1 0x08
+#define ACTUATOR_MASK 0x80
+#define ACTUATOR_SHIFT 7
+#define LOOP_MASK 0x40
+#define LOOP_SHIFT 6
+#define AUTOBRK_OK_MASK 0x10
+#define AUTOBRK_OK_ENABLE 0x10
+
+#define DRV2624_REG_GO 0x0c
+
+#define DRV2624_REG_CONTROL2 0x0d
+#define LIB_LRA 0x00
+#define LIB_ERM 0x01
+#define LIB_MASK 0x80
+#define LIB_SHIFT 0x07
+#define SCALE_MASK 0x03
+#define INTERVAL_MASK 0x20
+#define INTERVAL_SHIFT 0x05
+
+#define DRV2624_REG_RTP_INPUT 0x0e
+
+#define DRV2624_REG_SEQUENCER_1 0x0f
+
+#define DRV2624_REG_SEQ_LOOP_1 0x17
+
+#define DRV2624_REG_SEQ_LOOP_2 0x18
+
+#define DRV2624_REG_MAIN_LOOP 0x19
+#define MAIN_LOOP_MASK 0x07
+
+#define DRV2624_REG_RATED_VOLTAGE 0x1f
+
+#define DRV2624_REG_OVERDRIVE_CLAMP 0x20
+
+#define DRV2624_REG_CAL_COMP 0x21
+
+#define DRV2624_REG_CAL_BEMF 0x22
+
+#define DRV2624_REG_LOOP_CONTROL 0x23
+#define BEMFGAIN_MASK 0x03
+
+#define DRV2624_REG_DRIVE_TIME 0x27
+#define DRIVE_TIME_MASK 0x1f
+#define MINFREQ_SEL_45HZ 0x01
+#define MINFREQ_SEL_MASK 0x80
+#define MINFREQ_SEL_SHIFT 0x07
+
+#define DRV2624_REG_BLK_IDISS_TIME 0x28
+#define IDISS_TIME_MASK 0x0f
+#define BLANKING_TIME_MASK 0xf0
+#define BLANKING_TIME_SHIFT 0x04
+
+#define DRV2624_REG_ZC_OD_TIME 0x29
+#define ZC_DET_TIME_MASK 0x03
+
+#define DRV2624_REG_LRA_OL_CTRL 0x2c
+#define LRA_WAVE_SHAPE_MASK 0x01
+
+#define DRV2624_REG_OL_PERIOD_H 0x2e
+
+#define DRV2624_REG_OL_PERIOD_L 0x2f
+
+#define DRV2624_REG_DIAG_K 0x30
+
+#define GO_BIT_POLL_INTERVAL 15
+#define STANDBY_WAKE_DELAY 1
+#define WAKE_STANDBY_DELAY 3
+
+#define DRV2624_REG_RAM_ADDR_UPPER 0xfd
+#define DRV2624_REG_RAM_ADDR_LOWER 0xfe
+#define DRV2624_REG_RAM_DATA 0xff
+
+/* Commands */
+#define HAPTIC_CMDID_PLAY_SINGLE_EFFECT 0x01
+#define HAPTIC_CMDID_PLAY_EFFECT_SEQUENCE 0x02
+#define HAPTIC_CMDID_PLAY_TIMED_EFFECT 0x03
+#define HAPTIC_CMDID_GET_DEV_ID 0x04
+#define HAPTIC_CMDID_RUN_DIAG 0x05
+#define HAPTIC_CMDID_AUDIOHAPTIC_ENABLE 0x06
+#define HAPTIC_CMDID_AUDIOHAPTIC_DISABLE 0x07
+#define HAPTIC_CMDID_AUDIOHAPTIC_GETSTATUS 0x08
+
+#define HAPTIC_CMDID_REG_WRITE 0x09
+#define HAPTIC_CMDID_REG_READ 0x0a
+#define HAPTIC_CMDID_REG_SETBIT 0x0b
+
+#define HAPTIC_CMDID_PATTERN_RTP 0x0c
+#define HAPTIC_CMDID_RTP_SEQUENCE 0x0d
+#define HAPTIC_CMDID_GET_EFFECT_COUNT 0x10
+#define HAPTIC_CMDID_UPDATE_FIRMWARE 0x11
+#define HAPTIC_CMDID_READ_FIRMWARE 0x12
+#define HAPTIC_CMDID_RUN_CALIBRATION 0x13
+#define HAPTIC_CMDID_CONFIG_WAVEFORM 0x14
+#define HAPTIC_CMDID_SET_SEQUENCER 0x15
+
+#define HAPTIC_CMDID_STOP 0xFF
+
+#define MAX_TIMEOUT 10000 /* 10s */
+#define MAX_READ_BYTES 0xff
+#define DRV2624_SEQUENCER_SIZE 8
+
+#define WORK_IDLE 0
+#define WORK_VIBRATOR 0x01
+#define WORK_IRQ 0x02
+
+#define GO 1
+#define STOP 0
+
+#define POLL_GO_BIT_RETRY 10
+
+enum actuator_type {
+ ERM,
+ LRA
+};
+
+enum loop_type {
+ CLOSE_LOOP,
+ OPEN_LOOP
+};
+
+struct actuator_data {
+ unsigned char actuator_type;
+ unsigned char rated_voltage;
+ unsigned char over_drive_clamp_voltage;
+ int lra_freq;
+ int ol_lra_freq;
+ int voltage_comp;
+ int bemf_factor;
+ int bemf_gain;
+ int blanking_time;
+ int idiss_time;
+ int zc_det_time;
+ int lra_wave_shape;
+ int waveform_interval;
+};
+
+enum wave_seq_loop {
+ SEQ_NO_LOOP,
+ SEQ_LOOP_ONCE,
+ SEQ_LOOP_TWICE,
+ SEQ_LOOP_TRIPPLE
+};
+
+enum wave_main_loop {
+ MAIN_NO_LOOP,
+ MAIN_LOOP_ONCE,
+ MAIN_LOOP_TWICE,
+ MAIN_LOOP_3_TIMES,
+ MAIN_LOOP_4_TIMES,
+ MAIN_LOOP_5_TIMES,
+ MAIN_LOOP_6_TIMES,
+ MAIN_LOOP_INFINITELY
+};
+
+enum wave_main_scale {
+ PERCENTAGE_100,
+ PERCENTAGE_75,
+ PERCENTAGE_50,
+ PERCENTAGE_25
+};
+
+enum wave_main_interval {
+ INTERVAL_5MS,
+ INTERVAL_1MS
+};
+
+struct drv2624_waveform {
+ unsigned char effect;
+ unsigned char loop;
+};
+
+struct drv2624_waveform_sequencer {
+ struct drv2624_waveform waveform[DRV2624_SEQUENCER_SIZE];
+};
+
+struct drv2624_wave_setting {
+ unsigned char loop;
+ unsigned char interval;
+ unsigned char scale;
+};
+
+struct drv2624_autocal_result {
+ int finished;
+ unsigned char result;
+ unsigned char cal_comp;
+ unsigned char cal_bemf;
+ unsigned char cal_gain;
+};
+
+struct drv2624_diag_result {
+ int finished;
+ unsigned char result;
+ unsigned char diagz;
+ unsigned char diagk;
+};
+
+struct drv2624_platform_data {
+ int gpio_nrst;
+ int gpio_int;
+ unsigned char loop;
+ struct actuator_data actuator;
+};
+
+struct drv2624_fw_header {
+ int fw_magic;
+ int fw_size;
+ int fw_date;
+ int fw_chksum;
+ int fw_effcount;
+};
+
+static const char * const drv2624_modes[] = {
+ "rtp",
+ "waveform",
+ "diag",
+ "autocal",
+};
+
+#define DRV2624_MAGIC 0x2624
+
+struct drv2624_data {
+ struct drv2624_platform_data plat_data;
+ unsigned char device_id;
+ struct device *dev;
+ struct regmap *regmap;
+ unsigned int irq;
+ unsigned char int_status;
+ struct drv2624_wave_setting waveform_setting;
+ struct drv2624_waveform_sequencer waveform_sequencer;
+ unsigned char file_cmd;
+ bool vibrator_playing;
+ char work_mode;
+ unsigned char current_reg;
+ struct mutex lock; /* protect mode switching */
+ struct workqueue_struct *drv2624_wq;
+ struct work_struct vibrator_work;
+ struct work_struct work;
+ struct work_struct stop_work;
+ struct led_classdev led_dev;
+
+ struct drv2624_autocal_result auto_cal_result;
+ struct drv2624_diag_result diag_result;
+
+ struct drv2624_fw_header fw_header;
+ unsigned char ram_lsb;
+ unsigned char ram_msb;
+};
+
+#define DRV2624_MAGIC_NUMBER 0x32363234 /* '2624' */
+
+#define DRV2624_WAVSEQ_PLAY _IOWR(DRV2624_MAGIC_NUMBER, 4, unsigned long)
+#define DRV2624_STOP _IOWR(DRV2624_MAGIC_NUMBER, 5, unsigned long)
+#define DRV2624_RUN_DIAGNOSTIC _IOWR(DRV2624_MAGIC_NUMBER, 6, unsigned long)
+#define DRV2624_GET_DIAGRESULT _IOWR(DRV2624_MAGIC_NUMBER, 7, struct drv2624_diag_result *)
+#define DRV2624_RUN_AUTOCAL _IOWR(DRV2624_MAGIC_NUMBER, 8, unsigned long)
+#define DRV2624_GET_CALRESULT _IOWR(DRV2624_MAGIC_NUMBER, 9, struct drv2624_autocal_result *)
+
+#endif
diff --git a/drivers/input/misc/fpr_FingerprintCard/Kconfig b/drivers/input/misc/fpr_FingerprintCard/Kconfig
new file mode 100644
index 0000000..c9599e6
--- /dev/null
+++ b/drivers/input/misc/fpr_FingerprintCard/Kconfig
@@ -0,0 +1,10 @@
+#
+# FingerprintCard fingerprint driver
+#
+menu "FingerprintCard fingerprint driver"
+config FPR_FPC
+ default n
+ tristate "FPC_BTP fingerprint sensor support"
+ depends on SPI_MASTER
+
+endmenu
diff --git a/drivers/input/misc/fpr_FingerprintCard/Makefile b/drivers/input/misc/fpr_FingerprintCard/Makefile
new file mode 100644
index 0000000..9a102ff
--- /dev/null
+++ b/drivers/input/misc/fpr_FingerprintCard/Makefile
@@ -0,0 +1,10 @@
+#
+# Makefile for FingerprintCard fingerprint driver
+#
+
+fpc1020-objs := fpc1020_platform_tee.o
+obj-$(CONFIG_FPR_FPC) += fpc1020.o
+
+#TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-duplicate-decl-specifier
diff --git a/drivers/input/misc/fpr_FingerprintCard/fpc1020_platform_tee.c b/drivers/input/misc/fpr_FingerprintCard/fpc1020_platform_tee.c
new file mode 100644
index 0000000..b0a3172c
--- /dev/null
+++ b/drivers/input/misc/fpr_FingerprintCard/fpc1020_platform_tee.c
@@ -0,0 +1,657 @@
+/*
+ * FPC1020 Fingerprint sensor device driver
+ *
+ * This driver will control the platform resources that the FPC fingerprint
+ * sensor needs to operate. The major things are probing the sensor to check
+ * that it is actually connected and let the Kernel know this and with that also
+ * enabling and disabling of regulators, controlling GPIOs such as sensor reset
+ * line, sensor IRQ line.
+ *
+ * The driver will expose most of its available functionality in sysfs which
+ * enables dynamic control of these features from eg. a user space process.
+ *
+ * The sensor's IRQ events will be pushed to Kernel's event handling system and
+ * are exposed in the drivers event node.
+ *
+ * This driver will NOT send any commands to the sensor it only controls the
+ * electrical parts.
+ *
+ *
+ * Copyright (c) 2015 Fingerprint Cards AB <tech@fingerprints.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License Version 2
+ * as published by the Free Software Foundation.
+ */
+
+#include <linux/atomic.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/gpio.h>
+#include <linux/interrupt.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/of.h>
+#include <linux/of_gpio.h>
+#include <linux/platform_device.h>
+#include <linux/regulator/consumer.h>
+
+#define FPC_TTW_HOLD_TIME 1000
+
+#define RESET_LOW_SLEEP_MIN_US 5000
+#define RESET_LOW_SLEEP_MAX_US (RESET_LOW_SLEEP_MIN_US + 100)
+#define RESET_HIGH_SLEEP1_MIN_US 100
+#define RESET_HIGH_SLEEP1_MAX_US (RESET_HIGH_SLEEP1_MIN_US + 100)
+#define RESET_HIGH_SLEEP2_MIN_US 5000
+#define RESET_HIGH_SLEEP2_MAX_US (RESET_HIGH_SLEEP2_MIN_US + 100)
+#define PWR_ON_SLEEP_MIN_US 100
+#define PWR_ON_SLEEP_MAX_US (PWR_ON_SLEEP_MIN_US + 900)
+
+#define NUM_PARAMS_REG_ENABLE_SET 2
+
+static const char * const pctl_names[] = {
+ "fpc1020_reset_reset",
+ "fpc1020_reset_active",
+ "fpc1020_irq_active",
+};
+
+struct vreg_config {
+ char *name;
+ unsigned long vmin;
+ unsigned long vmax;
+ int ua_load;
+};
+
+static const struct vreg_config const vreg_conf[] = {
+};
+
+struct fpc1020_data {
+ struct device *dev;
+
+ struct pinctrl *fingerprint_pinctrl;
+ struct pinctrl_state *pinctrl_state[ARRAY_SIZE(pctl_names)];
+ struct regulator *vreg[ARRAY_SIZE(vreg_conf)];
+
+ struct wakeup_source ttw_wakesrc;
+ int irq_gpio;
+ int rst_gpio;
+ struct mutex lock; /* To set/get exported values in sysfs */
+ bool prepared;
+ atomic_t wakeup_enabled; /* Used both in ISR and non-ISR */
+};
+
+static int vreg_setup(struct fpc1020_data *fpc1020, const char *name,
+ bool enable)
+{
+ size_t i;
+ int rc;
+ struct regulator *vreg;
+ struct device *dev = fpc1020->dev;
+
+ for (i = 0; i < ARRAY_SIZE(fpc1020->vreg); i++) {
+ const char *n = vreg_conf[i].name;
+
+ if (!strncmp(n, name, strlen(n)))
+ goto found;
+ }
+
+ dev_err(dev, "Regulator %s not found\n", name);
+
+ return -EINVAL;
+
+found:
+ vreg = fpc1020->vreg[i];
+ if (enable) {
+ if (!vreg) {
+ vreg = regulator_get(dev, name);
+ if (IS_ERR(vreg)) {
+ dev_err(dev, "Unable to get %s\n", name);
+ return PTR_ERR(vreg);
+ }
+ }
+
+ if (regulator_count_voltages(vreg) > 0) {
+ rc = regulator_set_voltage(vreg, vreg_conf[i].vmin,
+ vreg_conf[i].vmax);
+ if (rc)
+ dev_err(dev,
+ "Unable to set voltage on %s, %d\n",
+ name, rc);
+ }
+
+ rc = regulator_enable(vreg);
+ if (rc) {
+ dev_err(dev, "error enabling %s: %d\n", name, rc);
+ regulator_put(vreg);
+ vreg = NULL;
+ }
+ fpc1020->vreg[i] = vreg;
+ } else {
+ if (vreg) {
+ if (regulator_is_enabled(vreg)) {
+ regulator_disable(vreg);
+ dev_dbg(dev, "disabled %s\n", name);
+ }
+ regulator_put(vreg);
+ fpc1020->vreg[i] = NULL;
+ }
+ rc = 0;
+ }
+
+ return rc;
+}
+
+/**
+ * sysfs node for controlling clocks.
+ *
+ * This is disabled in platform variant of this driver but kept for
+ * backwards compatibility. Only prints a debug print that it is
+ * disabled.
+ */
+static ssize_t clk_enable_set(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ dev_dbg(dev,
+ "clk_enable sysfs node not enabled in platform driver\n");
+
+ return count;
+}
+static DEVICE_ATTR(clk_enable, S_IWUSR, NULL, clk_enable_set);
+
+/**
+ * Will try to select the set of pins (GPIOS) defined in a pin control node of
+ * the device tree named @p name.
+ *
+ * The node can contain several eg. GPIOs that is controlled when selecting it.
+ * The node may activate or deactivate the pins it contains, the action is
+ * defined in the device tree node itself and not here. The states used
+ * internally is fetched at probe time.
+ *
+ * @see pctl_names
+ * @see fpc1020_probe
+ */
+static int select_pin_ctl(struct fpc1020_data *fpc1020, const char *name)
+{
+ size_t i;
+ int rc;
+ struct device *dev = fpc1020->dev;
+
+ for (i = 0; i < ARRAY_SIZE(fpc1020->pinctrl_state); i++) {
+ const char *n = pctl_names[i];
+
+ if (!strncmp(n, name, strlen(n))) {
+ rc = pinctrl_select_state(fpc1020->fingerprint_pinctrl,
+ fpc1020->pinctrl_state[i]);
+ if (rc)
+ dev_err(dev, "cannot select '%s'\n", name);
+ else
+ dev_info(dev, "Selected '%s'\n", name);
+ goto exit;
+ }
+ }
+
+ rc = -EINVAL;
+ dev_err(dev, "%s:'%s' not found\n", __func__, name);
+
+exit:
+ return rc;
+}
+
+static ssize_t pinctl_set(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct fpc1020_data *fpc1020 = dev_get_drvdata(dev);
+ int rc;
+
+ mutex_lock(&fpc1020->lock);
+ rc = select_pin_ctl(fpc1020, buf);
+ mutex_unlock(&fpc1020->lock);
+
+ return rc ? rc : count;
+}
+static DEVICE_ATTR(pinctl_set, S_IWUSR, NULL, pinctl_set);
+
+static ssize_t regulator_enable_set(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct fpc1020_data *fpc1020 = dev_get_drvdata(dev);
+ char op;
+ char name[16];
+ int rc;
+ bool enable;
+
+ if (sscanf(buf, "%15[^,],%c", name, &op) != NUM_PARAMS_REG_ENABLE_SET)
+ return -EINVAL;
+ if (op == 'e')
+ enable = true;
+ else if (op == 'd')
+ enable = false;
+ else
+ return -EINVAL;
+
+ mutex_lock(&fpc1020->lock);
+ rc = vreg_setup(fpc1020, name, enable);
+ mutex_unlock(&fpc1020->lock);
+
+ return rc ? rc : count;
+}
+static DEVICE_ATTR(regulator_enable, S_IWUSR, NULL, regulator_enable_set);
+
+static int hw_reset(struct fpc1020_data *fpc1020)
+{
+ int irq_gpio;
+ struct device *dev = fpc1020->dev;
+ int rc = select_pin_ctl(fpc1020, "fpc1020_reset_active");
+
+ if (rc)
+ goto exit;
+ usleep_range(RESET_HIGH_SLEEP1_MIN_US, RESET_HIGH_SLEEP1_MAX_US);
+
+ rc = select_pin_ctl(fpc1020, "fpc1020_reset_reset");
+ if (rc)
+ goto exit;
+ usleep_range(RESET_LOW_SLEEP_MIN_US, RESET_LOW_SLEEP_MAX_US);
+
+ rc = select_pin_ctl(fpc1020, "fpc1020_reset_active");
+ if (rc)
+ goto exit;
+ usleep_range(RESET_HIGH_SLEEP2_MIN_US, RESET_HIGH_SLEEP2_MAX_US);
+
+ irq_gpio = gpio_get_value(fpc1020->irq_gpio);
+ dev_info(dev, "IRQ after reset %d\n", irq_gpio);
+
+exit:
+ return rc;
+}
+
+static ssize_t hw_reset_set(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ int rc;
+ struct fpc1020_data *fpc1020 = dev_get_drvdata(dev);
+
+ if (!strncmp(buf, "reset", strlen("reset"))) {
+ mutex_lock(&fpc1020->lock);
+ rc = hw_reset(fpc1020);
+ mutex_unlock(&fpc1020->lock);
+ } else {
+ return -EINVAL;
+ }
+
+ return rc ? rc : count;
+}
+static DEVICE_ATTR(hw_reset, S_IWUSR, NULL, hw_reset_set);
+
+/**
+ * Will setup GPIOs, and regulators to correctly initialize the touch sensor to
+ * be ready for work.
+ *
+ * In the correct order according to the sensor spec this function will
+ * enable/disable regulators, and reset line, all to set the sensor in a
+ * correct power on or off state "electrical" wise.
+ *
+ * @see device_prepare_set
+ * @note This function will not send any commands to the sensor it will only
+ * control it "electrically".
+ */
+static int device_prepare(struct fpc1020_data *fpc1020, bool enable)
+{
+ int rc;
+
+ mutex_lock(&fpc1020->lock);
+ if (enable && !fpc1020->prepared) {
+ fpc1020->prepared = true;
+ select_pin_ctl(fpc1020, "fpc1020_reset_reset");
+
+ rc = vreg_setup(fpc1020, "vcc_spi", true);
+ if (rc)
+ goto exit;
+
+ rc = vreg_setup(fpc1020, "vdd_io", true);
+ if (rc)
+ goto exit_1;
+
+ rc = vreg_setup(fpc1020, "vdd_ana", true);
+ if (rc)
+ goto exit_2;
+
+ usleep_range(PWR_ON_SLEEP_MIN_US, PWR_ON_SLEEP_MAX_US);
+
+ /* As we can't control chip select here the other part of the
+ * sensor driver eg. the TEE driver needs to do a _SOFT_ reset
+ * on the sensor after power up to be sure that the sensor is
+ * in a good state after power up. Okeyed by ASIC.
+ */
+
+ (void)select_pin_ctl(fpc1020, "fpc1020_reset_active");
+ } else if (!enable && fpc1020->prepared) {
+ rc = 0;
+ (void)select_pin_ctl(fpc1020, "fpc1020_reset_reset");
+
+ usleep_range(PWR_ON_SLEEP_MIN_US, PWR_ON_SLEEP_MAX_US);
+
+ (void)vreg_setup(fpc1020, "vdd_ana", false);
+exit_2:
+ (void)vreg_setup(fpc1020, "vdd_io", false);
+exit_1:
+ (void)vreg_setup(fpc1020, "vcc_spi", false);
+exit:
+ fpc1020->prepared = false;
+ } else {
+ rc = 0;
+ }
+ mutex_unlock(&fpc1020->lock);
+
+ return rc;
+}
+
+/**
+ * sysfs node to enable/disable (power up/power down) the touch sensor
+ *
+ * @see device_prepare
+ */
+static ssize_t device_prepare_set(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ int rc;
+ struct fpc1020_data *fpc1020 = dev_get_drvdata(dev);
+
+ if (!strncmp(buf, "enable", strlen("enable")))
+ rc = device_prepare(fpc1020, true);
+ else if (!strncmp(buf, "disable", strlen("disable")))
+ rc = device_prepare(fpc1020, false);
+ else
+ return -EINVAL;
+
+ return rc ? rc : count;
+}
+static DEVICE_ATTR(device_prepare, S_IWUSR, NULL, device_prepare_set);
+
+/**
+ * sysfs node for controlling whether the driver is allowed
+ * to wake up the platform on interrupt.
+ */
+static ssize_t wakeup_enable_set(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct fpc1020_data *fpc1020 = dev_get_drvdata(dev);
+ ssize_t ret = count;
+
+ mutex_lock(&fpc1020->lock);
+ if (!strncmp(buf, "enable", strlen("enable")))
+ atomic_set(&fpc1020->wakeup_enabled, 1);
+ else if (!strncmp(buf, "disable", strlen("disable")))
+ atomic_set(&fpc1020->wakeup_enabled, 0);
+ else
+ ret = -EINVAL;
+ mutex_unlock(&fpc1020->lock);
+
+ return ret;
+}
+static DEVICE_ATTR(wakeup_enable, S_IWUSR, NULL, wakeup_enable_set);
+
+/**
+ * sysf node to check the interrupt status of the sensor, the interrupt
+ * handler should perform sysf_notify to allow userland to poll the node.
+ */
+static ssize_t irq_get(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct fpc1020_data *fpc1020 = dev_get_drvdata(dev);
+ int irq = gpio_get_value(fpc1020->irq_gpio);
+
+ return scnprintf(buf, PAGE_SIZE, "%i\n", irq);
+}
+
+/**
+ * writing to the irq node will just drop a printk message
+ * and return success, used for latency measurement.
+ */
+static ssize_t irq_ack(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct fpc1020_data *fpc1020 = dev_get_drvdata(dev);
+
+ dev_dbg(fpc1020->dev, "%s\n", __func__);
+
+ return count;
+}
+static DEVICE_ATTR(irq, S_IRUSR | S_IWUSR, irq_get, irq_ack);
+
+static struct attribute *attributes[] = {
+ &dev_attr_pinctl_set.attr,
+ &dev_attr_device_prepare.attr,
+ &dev_attr_regulator_enable.attr,
+ &dev_attr_hw_reset.attr,
+ &dev_attr_wakeup_enable.attr,
+ &dev_attr_clk_enable.attr,
+ &dev_attr_irq.attr,
+ NULL
+};
+
+static const struct attribute_group attribute_group = {
+ .attrs = attributes,
+};
+
+static irqreturn_t fpc1020_irq_handler(int irq, void *handle)
+{
+ struct fpc1020_data *fpc1020 = handle;
+
+ dev_dbg(fpc1020->dev, "%s\n", __func__);
+
+ if (atomic_read(&fpc1020->wakeup_enabled)) {
+ __pm_wakeup_event(&fpc1020->ttw_wakesrc, FPC_TTW_HOLD_TIME);
+ }
+
+ sysfs_notify(&fpc1020->dev->kobj, NULL, dev_attr_irq.attr.name);
+
+ return IRQ_HANDLED;
+}
+
+static int fpc1020_request_named_gpio(struct fpc1020_data *fpc1020,
+ const char *label, int *gpio)
+{
+ struct device *dev = fpc1020->dev;
+ struct device_node *np = dev->of_node;
+ int rc = of_get_named_gpio(np, label, 0);
+
+ if (rc < 0) {
+ dev_err(dev, "failed to get '%s'\n", label);
+ return rc;
+ }
+ *gpio = rc;
+
+ rc = devm_gpio_request(dev, *gpio, label);
+ if (rc) {
+ dev_err(dev, "failed to request gpio %d\n", *gpio);
+ return rc;
+ }
+ dev_dbg(dev, "%s %d\n", label, *gpio);
+
+ return 0;
+}
+
+static int fpc1020_probe(struct platform_device *pdev)
+{
+ struct device *dev = &pdev->dev;
+ int rc = 0;
+ size_t i;
+ int irqf;
+ struct device_node *np = dev->of_node;
+ struct fpc1020_data *fpc1020 = devm_kzalloc(dev, sizeof(*fpc1020),
+ GFP_KERNEL);
+
+ if (!fpc1020) {
+ dev_err(dev,
+ "failed to allocate memory for struct fpc1020_data\n");
+ rc = -ENOMEM;
+ goto exit;
+ }
+
+ fpc1020->dev = dev;
+ platform_set_drvdata(pdev, fpc1020);
+
+ if (!np) {
+ dev_err(dev, "no of node found\n");
+ rc = -EINVAL;
+ goto exit;
+ }
+
+ rc = fpc1020_request_named_gpio(fpc1020, "fpc,gpio_irq",
+ &fpc1020->irq_gpio);
+ if (rc) {
+ dev_err(fpc1020->dev,
+ "fpc1020_request_named_gpio fpc,gpio_irqfailed.\n");
+ goto exit;
+ }
+ rc = fpc1020_request_named_gpio(fpc1020, "fpc,gpio_rst",
+ &fpc1020->rst_gpio);
+ if (rc) {
+ dev_err(fpc1020->dev,
+ "fpc1020_request_named_gpio fpc,gpio_rst failed.\n");
+ goto exit;
+ }
+
+ fpc1020->fingerprint_pinctrl = devm_pinctrl_get(dev);
+ if (IS_ERR(fpc1020->fingerprint_pinctrl)) {
+ if (PTR_ERR(fpc1020->fingerprint_pinctrl) == -EPROBE_DEFER) {
+ dev_info(dev, "pinctrl not ready\n");
+ rc = -EPROBE_DEFER;
+ goto exit;
+ }
+ dev_err(dev, "Target does not use pinctrl\n");
+ fpc1020->fingerprint_pinctrl = NULL;
+ rc = -EINVAL;
+ goto exit;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(fpc1020->pinctrl_state); i++) {
+ const char *n = pctl_names[i];
+ struct pinctrl_state *state =
+ pinctrl_lookup_state(fpc1020->fingerprint_pinctrl, n);
+ if (IS_ERR(state)) {
+ dev_err(dev, "cannot find '%s'\n", n);
+ rc = -EINVAL;
+ goto exit;
+ }
+ dev_info(dev, "found pin control %s\n", n);
+ fpc1020->pinctrl_state[i] = state;
+ }
+
+ rc = select_pin_ctl(fpc1020, "fpc1020_reset_reset");
+ if (rc) {
+ dev_err(fpc1020->dev,
+ "select_pin_ctl fpc1020_reset_reset failed.\n");
+ goto exit;
+ }
+ rc = select_pin_ctl(fpc1020, "fpc1020_irq_active");
+ if (rc) {
+ dev_err(fpc1020->dev,
+ "select_pin_ctl fpc1020_irq_active failed.\n");
+ goto exit;
+ }
+
+ atomic_set(&fpc1020->wakeup_enabled, 0);
+
+ irqf = IRQF_TRIGGER_RISING | IRQF_ONESHOT;
+ if (of_property_read_bool(dev->of_node, "fpc,enable-wakeup")) {
+ irqf |= IRQF_NO_SUSPEND;
+ device_init_wakeup(dev, 1);
+ }
+
+ mutex_init(&fpc1020->lock);
+ rc = devm_request_threaded_irq(dev, gpio_to_irq(fpc1020->irq_gpio),
+ NULL, fpc1020_irq_handler, irqf,
+ dev_name(dev), fpc1020);
+ if (rc) {
+ dev_err(dev, "could not request irq %d\n",
+ gpio_to_irq(fpc1020->irq_gpio));
+ goto exit;
+ }
+
+ dev_dbg(dev, "requested irq %d\n", gpio_to_irq(fpc1020->irq_gpio));
+
+ /* Request that the interrupt should be wakeable */
+ enable_irq_wake(gpio_to_irq(fpc1020->irq_gpio));
+
+ wakeup_source_init(&fpc1020->ttw_wakesrc, "fpc_ttw_wl");
+
+ rc = sysfs_create_group(&dev->kobj, &attribute_group);
+ if (rc) {
+ dev_err(dev, "could not create sysfs\n");
+ goto exit;
+ }
+
+ if (of_property_read_bool(dev->of_node, "fpc,enable-on-boot")) {
+ dev_info(dev, "Enabling hardware\n");
+ (void)device_prepare(fpc1020, true);
+ }
+
+ rc = hw_reset(fpc1020);
+
+ dev_info(dev, "%s: ok\n", __func__);
+
+exit:
+ return rc;
+}
+
+static int fpc1020_remove(struct platform_device *pdev)
+{
+ struct fpc1020_data *fpc1020 = platform_get_drvdata(pdev);
+
+ sysfs_remove_group(&pdev->dev.kobj, &attribute_group);
+ mutex_destroy(&fpc1020->lock);
+ wakeup_source_trash(&fpc1020->ttw_wakesrc);
+ (void)vreg_setup(fpc1020, "vdd_ana", false);
+ (void)vreg_setup(fpc1020, "vdd_io", false);
+ (void)vreg_setup(fpc1020, "vcc_spi", false);
+ dev_info(&pdev->dev, "%s\n", __func__);
+
+ return 0;
+}
+
+static const struct of_device_id fpc1020_of_match[] = {
+ { .compatible = "fpc,fpc1020", },
+ {}
+};
+MODULE_DEVICE_TABLE(of, fpc1020_of_match);
+
+static struct platform_driver fpc1020_driver = {
+ .driver = {
+ .name = "fpc1020",
+ .owner = THIS_MODULE,
+ .of_match_table = fpc1020_of_match,
+ },
+ .probe = fpc1020_probe,
+ .remove = fpc1020_remove,
+};
+
+static int __init fpc1020_init(void)
+{
+ int rc = platform_driver_register(&fpc1020_driver);
+
+ if (!rc)
+ pr_info("%s OK\n", __func__);
+ else
+ pr_err("%s %d\n", __func__, rc);
+
+ return rc;
+}
+
+static void __exit fpc1020_exit(void)
+{
+ pr_info("%s\n", __func__);
+ platform_driver_unregister(&fpc1020_driver);
+}
+
+module_init(fpc1020_init);
+module_exit(fpc1020_exit);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Aleksej Makarov");
+MODULE_AUTHOR("Henrik Tillman <henrik.tillman@fingerprints.com>");
+MODULE_DESCRIPTION("FPC1020 Fingerprint sensor device driver.");
diff --git a/drivers/input/misc/qpnp-power-on.c b/drivers/input/misc/qpnp-power-on.c
index 93c28ef..04ff156 100644
--- a/drivers/input/misc/qpnp-power-on.c
+++ b/drivers/input/misc/qpnp-power-on.c
@@ -34,6 +34,7 @@
#include <linux/qpnp/qpnp-pbs.h>
#include <linux/qpnp/qpnp-misc.h>
#include <linux/power_supply.h>
+#include <soc/qcom/restart.h>
#define PMIC_VER_8941 0x01
#define PMIC_VERSION_REG 0x0105
@@ -965,8 +966,9 @@ qpnp_pon_input_dispatch(struct qpnp_pon *pon, u32 pon_type)
return -EINVAL;
}
- pr_debug("PMIC input: code=%d, sts=0x%hhx\n",
- cfg->key_code, pon_rt_sts);
+ pr_info("PMIC input: code=%d, sts=0x%hhx, bit=0x%hhx, type=%d, os=%d\n",
+ cfg->key_code, pon_rt_sts, pon_rt_bit, cfg->pon_type,
+ cfg->old_state);
key_status = pon_rt_sts & pon_rt_bit;
if (pon->kpdpwr_dbc_enable && cfg->pon_type == PON_KPDPWR) {
@@ -1005,6 +1007,8 @@ static irqreturn_t qpnp_kpdpwr_irq(int irq, void *_pon)
static irqreturn_t qpnp_kpdpwr_bark_irq(int irq, void *_pon)
{
+ set_restart_msg("reboot_longkey");
+ qpnp_pon_set_restart_reason(PON_RESTART_REASON_REBOOT_LONGKEY);
return IRQ_HANDLED;
}
diff --git a/drivers/input/misc/vd6281/Kconfig b/drivers/input/misc/vd6281/Kconfig
new file mode 100644
index 0000000..7dd34e0
--- /dev/null
+++ b/drivers/input/misc/vd6281/Kconfig
@@ -0,0 +1,11 @@
+#
+# Miscellaneous I2C chip drivers configuration
+#
+menu "STMicro rainbow sensor support"
+
+config INPUT_VD6281
+ tristate "STMicro vd6281 rainbow sensor support"
+ help
+ If you say yes here you get support for STMicro vd6281 rainbow sensor.
+
+endmenu
diff --git a/drivers/input/misc/vd6281/Makefile b/drivers/input/misc/vd6281/Makefile
new file mode 100644
index 0000000..1dec98d
--- /dev/null
+++ b/drivers/input/misc/vd6281/Makefile
@@ -0,0 +1,15 @@
+#
+# Makefile for the vd6281 drivers.
+#
+
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_req_mgr
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_smmu/
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_sync
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_utils
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_sensor_module/cam_sensor
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_sensor_module/cam_cci
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_cpas/include
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io
+
+obj-$(CONFIG_INPUT_VD6281) += vd6281_module.o
diff --git a/drivers/input/misc/vd6281/vd6281_module.c b/drivers/input/misc/vd6281/vd6281_module.c
new file mode 100644
index 0000000..a1fc90e
--- /dev/null
+++ b/drivers/input/misc/vd6281/vd6281_module.c
@@ -0,0 +1,522 @@
+/*
+ * vd6281_module.c - Linux kernel module for rainbow sensor
+ *
+ * Copyright (C) 2017-2018 Google, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 2 as published
+ * by the Free Software Foundation.
+ */
+#include <linux/module.h>
+#include <linux/init.h>
+#include <linux/i2c.h>
+#include <linux/mutex.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/irq.h>
+#include <linux/gpio.h>
+#include <linux/input.h>
+#include <linux/kernel.h>
+#include <linux/kobject.h>
+#include <linux/regulator/consumer.h>
+#include <linux/pinctrl/consumer.h>
+#include <linux/of_gpio.h>
+#include <linux/slab.h>
+#include <linux/cdev.h>
+#include <linux/device.h>
+#include <media/cam_sensor.h>
+#include "cam_sensor_dev.h"
+
+#define VD6281_DEV_NAME "vd6281"
+
+struct rainbow_ctrl_t {
+ struct platform_device *pdev;
+ struct cam_hw_soc_info soc_info;
+ struct mutex cam_sensor_mutex;
+ enum cci_i2c_master_t cci_i2c_master;
+ struct camera_io_master io_master_info;
+ uint32_t id;
+ struct device_node *of_node;
+ struct cam_subdev v4l2_dev_str;
+ int is_power_up;
+ int hw_version;
+ struct regulator *vdd;
+ dev_t dev;
+ struct cdev c_dev;
+ struct class *cl;
+};
+
+static struct rainbow_ctrl_t *ctrl;
+static const struct file_operations vd6281_fops;
+
+int vd6281_write_data(struct rainbow_ctrl_t *ctrl,
+ uint32_t addr, uint32_t data)
+{
+ int rc = 0;
+ struct cam_sensor_i2c_reg_setting write_setting;
+ struct cam_sensor_i2c_reg_array i2c_reg_array;
+
+ i2c_reg_array.reg_addr = addr;
+ i2c_reg_array.reg_data = data;
+ i2c_reg_array.delay = 1;
+ write_setting.reg_setting = &i2c_reg_array;
+ write_setting.addr_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ write_setting.data_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ write_setting.size = 1;
+ write_setting.delay = 0;
+
+ rc = camera_io_dev_write(&(ctrl->io_master_info), &write_setting);
+ return rc;
+}
+
+static int vd6281_power_up(struct rainbow_ctrl_t *ctrl)
+{
+ int rc;
+
+ rc = regulator_enable(ctrl->vdd);
+ if (rc < 0)
+ pr_err("%s regulator_enable failed: rc: %d", __func__, rc);
+ rc = camera_io_init(&(ctrl->io_master_info));
+ if (rc < 0)
+ pr_err("%s cci_release failed: rc: %d", __func__, rc);
+
+ return rc;
+}
+
+static int vd6281_power_down(struct rainbow_ctrl_t *ctrl)
+{
+ int rc;
+
+ rc = camera_io_release(&(ctrl->io_master_info));
+ if (rc < 0)
+ pr_err("%s cci_release failed: rc: %d", __func__, rc);
+ rc = regulator_disable(ctrl->vdd);
+ if (rc < 0)
+ pr_err("%s regulator_disable failed: rc: %d", __func__, rc);
+
+ return rc;
+}
+
+static ssize_t rainbow_enable_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct rainbow_ctrl_t *ctrl = dev_get_drvdata(dev);
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n", ctrl->is_power_up);
+}
+
+static ssize_t rainbow_enable_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct rainbow_ctrl_t *ctrl = dev_get_drvdata(dev);
+ int rc;
+ bool value;
+
+ rc = kstrtobool(buf, &value);
+ if (rc)
+ return rc;
+
+ mutex_lock(&ctrl->cam_sensor_mutex);
+
+ if (value == 1)
+ rc = vd6281_power_up(ctrl);
+ else
+ rc = vd6281_power_down(ctrl);
+
+ if (rc == 0)
+ ctrl->is_power_up = (int)value;
+
+ mutex_unlock(&ctrl->cam_sensor_mutex);
+
+ return rc;
+}
+
+static ssize_t rainbow_read_byte_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct rainbow_ctrl_t *ctrl = dev_get_drvdata(dev);
+ unsigned long value;
+ uint32_t read_data = 0;
+ int rc;
+
+ rc = kstrtoul(buf, 0, &value);
+ if (rc)
+ return rc;
+
+ mutex_lock(&ctrl->cam_sensor_mutex);
+
+ rc = camera_io_dev_read(&(ctrl->io_master_info), value & 0xFF,
+ &read_data, CAMERA_SENSOR_I2C_TYPE_BYTE,
+ CAMERA_SENSOR_I2C_TYPE_BYTE);
+
+ mutex_unlock(&ctrl->cam_sensor_mutex);
+
+ if (rc < 0) {
+ dev_err(dev, "%s I2C read failed: %d.\n", __func__, rc);
+ return rc;
+ }
+
+ dev_dbg(dev, "I2C read addr: 0x%lx, data: 0x%x\n", value, read_data);
+
+ return read_data;
+}
+
+static ssize_t rainbow_write_byte_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct rainbow_ctrl_t *ctrl = dev_get_drvdata(dev);
+ unsigned long value;
+ int rc;
+ u8 addr;
+ u8 data;
+
+ rc = kstrtoul(buf, 0, &value);
+
+ if (rc)
+ return rc;
+
+ addr = (value >> 8) & 0xFF;
+ data = value & 0xFF;
+
+ mutex_lock(&ctrl->cam_sensor_mutex);
+
+ rc = vd6281_write_data(ctrl, addr, data);
+
+ mutex_unlock(&ctrl->cam_sensor_mutex);
+
+ if (rc < 0)
+ dev_err(dev, "%s I2C write failed: %d.", __func__, rc);
+
+ dev_dbg(dev, "I2C write addr: 0x%x, data: 0x%x", addr, data);
+
+ return rc;
+}
+
+static DEVICE_ATTR_RW(rainbow_enable);
+static DEVICE_ATTR_WO(rainbow_read_byte);
+static DEVICE_ATTR_WO(rainbow_write_byte);
+
+static struct attribute *rainbow_dev_attrs[] = {
+ &dev_attr_rainbow_enable.attr,
+ &dev_attr_rainbow_read_byte.attr,
+ &dev_attr_rainbow_write_byte.attr,
+ NULL
+};
+
+ATTRIBUTE_GROUPS(rainbow_dev);
+
+int32_t vd6281_update_i2c_info(struct rainbow_ctrl_t *ctrl)
+{
+ int32_t rc = 0;
+
+ ctrl->cci_i2c_master = MASTER_0;
+
+ if (ctrl->hw_version == 20)
+ ctrl->io_master_info.cci_client->sid = 0x7F;
+ else
+ ctrl->io_master_info.cci_client->sid = 0x20;
+
+ ctrl->io_master_info.cci_client->retries = 1;
+ ctrl->io_master_info.cci_client->id_map = 0;
+ ctrl->io_master_info.cci_client->i2c_freq_mode = I2C_FAST_PLUS_MODE;
+
+ return rc;
+}
+
+static int vd6281_parse_dt(struct device *dev)
+{
+ struct rainbow_ctrl_t *ctrl = dev_get_drvdata(dev);
+ int rc = 0;
+
+ /* Initialize mutex */
+ mutex_init(&ctrl->cam_sensor_mutex);
+
+ ctrl->vdd = devm_regulator_get(dev, "vdd");
+ if (IS_ERR(ctrl->vdd)) {
+ ctrl->vdd = NULL;
+ dev_err(dev, "unable to get vdd\n");
+ rc = -ENOENT;
+ }
+
+ if (of_property_read_u32(dev->of_node, "hw-version",
+ &(ctrl->hw_version)))
+ dev_warn(dev, "hw-version not specified in dt\n");
+
+ return rc;
+}
+
+static int32_t vd6281_platform_remove(struct platform_device *pdev)
+{
+ struct rainbow_ctrl_t *ctrl;
+ int32_t rc = 0;
+
+ ctrl = platform_get_drvdata(pdev);
+ if (!ctrl) {
+ pr_err("rainbow device is NULL");
+ return 0;
+ }
+
+ device_destroy(ctrl->cl, ctrl->dev);
+ class_destroy(ctrl->cl);
+ cdev_del(&ctrl->c_dev);
+ unregister_chrdev_region(ctrl->dev, 1);
+
+ mutex_destroy(&ctrl->cam_sensor_mutex);
+ sysfs_remove_groups(&pdev->dev.kobj, rainbow_dev_groups);
+ kfree(ctrl->io_master_info.cci_client);
+ ctrl->io_master_info.cci_client = NULL;
+ devm_kfree(&pdev->dev, ctrl);
+
+ return rc;
+}
+
+static int32_t vd6281_driver_platform_probe(
+ struct platform_device *pdev)
+{
+ int32_t rc = 0;
+ struct device *dev_ret;
+
+ /* Create sensor control structure */
+ ctrl = devm_kzalloc(&pdev->dev,
+ sizeof(struct rainbow_ctrl_t), GFP_KERNEL);
+ if (!ctrl)
+ return -ENOMEM;
+
+ if (cam_cci_get_subdev() == NULL) {
+ dev_err(&pdev->dev, "wait cci driver probe\n");
+ return -EPROBE_DEFER;
+ }
+
+ /*fill in platform device*/
+ ctrl->v4l2_dev_str.pdev = pdev;
+ ctrl->soc_info.pdev = pdev;
+ ctrl->soc_info.dev = &pdev->dev;
+ ctrl->soc_info.dev_name = pdev->name;
+ ctrl->io_master_info.master_type = CCI_MASTER;
+
+ ctrl->io_master_info.cci_client = kzalloc(sizeof(
+ struct cam_sensor_cci_client), GFP_KERNEL);
+ if (!(ctrl->io_master_info.cci_client))
+ return -ENOMEM;
+
+ platform_set_drvdata(pdev, ctrl);
+ v4l2_set_subdevdata(&ctrl->v4l2_dev_str.sd, ctrl);
+ dev_set_drvdata(&pdev->dev, ctrl);
+
+ rc = vd6281_parse_dt(&(pdev->dev));
+ if (rc)
+ dev_err(&pdev->dev, "paring rainbow dt failed rc %d", rc);
+
+ vd6281_update_i2c_info(ctrl);
+
+ /* Fill platform device id*/
+ pdev->id = ctrl->soc_info.index;
+
+ rc = sysfs_create_groups(&pdev->dev.kobj, rainbow_dev_groups);
+ if (rc)
+ dev_err(&pdev->dev, "failed to create sysfs files");
+
+ if (rc)
+ return -EINVAL;
+
+ rc = alloc_chrdev_region(&ctrl->dev, 0, 1, "vd6281_ioctl");
+
+ if (rc)
+ return rc;
+
+ cdev_init(&ctrl->c_dev, &vd6281_fops);
+
+ rc = cdev_add(&ctrl->c_dev, ctrl->dev, 1);
+ if (rc)
+ return rc;
+
+ rc = IS_ERR(ctrl->cl = class_create(THIS_MODULE, "char"));
+ if (rc) {
+ cdev_del(&ctrl->c_dev);
+ unregister_chrdev_region(ctrl->dev, 1);
+ return PTR_ERR(ctrl->cl);
+ }
+
+ rc = IS_ERR(dev_ret =
+ device_create(ctrl->cl, NULL,
+ ctrl->dev, NULL, "vd6281"));
+ if (rc) {
+ class_destroy(ctrl->cl);
+ cdev_del(&ctrl->c_dev);
+ unregister_chrdev_region(ctrl->dev, 1);
+ return PTR_ERR(dev_ret);
+ }
+
+ return rc;
+}
+
+static const struct of_device_id vd6281_driver_dt_match[] = {
+ {.compatible = "qcom,rainbow"},
+ {}
+};
+
+MODULE_DEVICE_TABLE(of, vd6281_driver_dt_match);
+
+static struct platform_driver vd6281_platform_driver = {
+ .probe = vd6281_driver_platform_probe,
+ .driver = {
+ .name = VD6281_DEV_NAME,
+ .owner = THIS_MODULE,
+ .of_match_table = vd6281_driver_dt_match,
+ },
+ .remove = vd6281_platform_remove,
+};
+
+static int vd6281_open(struct inode *inode, struct file *file)
+{
+ return 0;
+}
+
+static int vd6281_release(struct inode *inode, struct file *file)
+{
+ return 0;
+}
+
+static long vd6281_ioctl_handler(struct file *file, unsigned int cmd,
+ unsigned long arg, void __user *p)
+{
+ int rc = 0;
+ int i = 0;
+ struct rainbow_config config;
+
+ mutex_lock(&ctrl->cam_sensor_mutex);
+
+ switch (cmd) {
+ case RAINBOW_CONFIG:
+ if (copy_from_user(&config, p, sizeof(config))) {
+ rc = -EFAULT;
+ goto out;
+ }
+
+ if (config.size == 0 || config.size > MAX_RAINBOW_CONFIG_SIZE) {
+ rc = -EFAULT;
+ goto out;
+ }
+
+ if (config.operation == RAINBOW_RANDOM_READ) {
+ for (i = 0; i < config.size; i++) {
+ rc = camera_io_dev_read(&(ctrl->io_master_info),
+ config.reg_addr[i],
+ &config.reg_data[i],
+ CAMERA_SENSOR_I2C_TYPE_BYTE,
+ CAMERA_SENSOR_I2C_TYPE_BYTE);
+ if (rc) {
+ pr_err("%s i2c random read failed: %d\n",
+ __func__, rc);
+ goto out;
+ }
+ }
+ rc = copy_to_user(p, &config, sizeof(config));
+ } else if (config.operation == RAINBOW_SEQ_READ) {
+ uint8_t data[MAX_RAINBOW_CONFIG_SIZE];
+
+ rc = camera_io_dev_read_seq(&(ctrl->io_master_info),
+ config.reg_addr[0],
+ data,
+ CAMERA_SENSOR_I2C_TYPE_BYTE,
+ config.size);
+ if (rc) {
+ pr_err("%s i2c cont read failed: %d\n",
+ __func__, rc);
+ goto out;
+ }
+ for (i = 0; i < config.size; i++)
+ config.reg_data[i] = data[i];
+
+ rc = copy_to_user(p, &config, sizeof(config));
+ } else if (config.operation == RAINBOW_RANDOM_WRITE ||
+ config.operation == RAINBOW_SEQ_WRITE) {
+ struct cam_sensor_i2c_reg_setting write_setting;
+ struct cam_sensor_i2c_reg_array
+ reg_setting[MAX_RAINBOW_CONFIG_SIZE] = { {0} };
+
+ write_setting.reg_setting = reg_setting;
+ write_setting.size = config.size;
+ write_setting.addr_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ write_setting.data_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ write_setting.delay = 0;
+
+ for (i = 0; i < config.size; i++) {
+ reg_setting[i].reg_addr = config.reg_addr[0];
+ reg_setting[i].reg_data = config.reg_data[0];
+ reg_setting[i].delay = 0;
+ reg_setting[i].data_mask = 0;
+ }
+
+ if (config.operation == RAINBOW_RANDOM_WRITE)
+ rc = camera_io_dev_write(
+ &(ctrl->io_master_info),
+ &write_setting);
+ else
+ rc = camera_io_dev_write_continuous(
+ &(ctrl->io_master_info),
+ &write_setting, 0);
+
+ } else
+ pr_err("%s: Unsupported opertion type\n", __func__);
+
+ break;
+ default:
+ pr_err("%s: Unsupported ioctl command %u\n", __func__, cmd);
+ rc = -EINVAL;
+ break;
+ }
+
+out:
+ mutex_unlock(&ctrl->cam_sensor_mutex);
+ return rc;
+}
+
+static long vd6281_ioctl(struct file *file, unsigned int cmd,
+ unsigned long arg)
+{
+ return vd6281_ioctl_handler(file, cmd, arg, (void __user *)arg);
+}
+
+#ifdef CONFIG_COMPAT
+static long vd6281_compat_ioctl(struct file *file, unsigned int cmd,
+ unsigned long arg)
+{
+ return vd6281_ioctl_handler(file, cmd, arg, compat_ptr(arg));
+}
+#endif
+
+static const struct file_operations vd6281_fops = {
+ .owner = THIS_MODULE,
+ .open = vd6281_open,
+ .release = vd6281_release,
+ .unlocked_ioctl = vd6281_ioctl,
+#ifdef CONFIG_COMPAT
+ .compat_ioctl = vd6281_compat_ioctl,
+#endif
+};
+
+static int __init vd6281_init(void)
+{
+ int rc = 0;
+
+ rc = platform_driver_register(&vd6281_platform_driver);
+
+ return rc;
+}
+
+static void __exit vd6281_exit(void)
+{
+ platform_driver_unregister(&vd6281_platform_driver);
+}
+
+MODULE_DESCRIPTION("ST rainbow sensor");
+MODULE_LICENSE("GPL");
+
+module_init(vd6281_init);
+module_exit(vd6281_exit);
+
diff --git a/drivers/input/mouse/elan_i2c.h b/drivers/input/mouse/elan_i2c.h
index c0ec261..83dd0ce 100644
--- a/drivers/input/mouse/elan_i2c.h
+++ b/drivers/input/mouse/elan_i2c.h
@@ -27,6 +27,8 @@
#define ETP_DISABLE_POWER 0x0001
#define ETP_PRESSURE_OFFSET 25
+#define ETP_CALIBRATE_MAX_LEN 3
+
/* IAP Firmware handling */
#define ETP_PRODUCT_ID_FORMAT_STRING "%d.0"
#define ETP_FW_NAME "elan_i2c_" ETP_PRODUCT_ID_FORMAT_STRING ".bin"
diff --git a/drivers/input/mouse/elan_i2c_core.c b/drivers/input/mouse/elan_i2c_core.c
index 3851d57..30adc57 100644
--- a/drivers/input/mouse/elan_i2c_core.c
+++ b/drivers/input/mouse/elan_i2c_core.c
@@ -595,7 +595,7 @@ static ssize_t calibrate_store(struct device *dev,
int tries = 20;
int retval;
int error;
- u8 val[3];
+ u8 val[ETP_CALIBRATE_MAX_LEN];
retval = mutex_lock_interruptible(&data->sysfs_mutex);
if (retval)
@@ -1240,6 +1240,7 @@ MODULE_DEVICE_TABLE(i2c, elan_id);
static const struct acpi_device_id elan_acpi_id[] = {
{ "ELAN0000", 0 },
{ "ELAN0100", 0 },
+ { "ELAN0501", 0 },
{ "ELAN0600", 0 },
{ "ELAN0602", 0 },
{ "ELAN0605", 0 },
@@ -1249,6 +1250,14 @@ static const struct acpi_device_id elan_acpi_id[] = {
{ "ELAN060B", 0 },
{ "ELAN060C", 0 },
{ "ELAN0611", 0 },
+ { "ELAN0612", 0 },
+ { "ELAN0618", 0 },
+ { "ELAN061C", 0 },
+ { "ELAN061D", 0 },
+ { "ELAN061E", 0 },
+ { "ELAN0620", 0 },
+ { "ELAN0621", 0 },
+ { "ELAN0622", 0 },
{ "ELAN1000", 0 },
{ }
};
diff --git a/drivers/input/mouse/elan_i2c_smbus.c b/drivers/input/mouse/elan_i2c_smbus.c
index e23b249..d21bd55 100644
--- a/drivers/input/mouse/elan_i2c_smbus.c
+++ b/drivers/input/mouse/elan_i2c_smbus.c
@@ -56,7 +56,7 @@
static int elan_smbus_initialize(struct i2c_client *client)
{
u8 check[ETP_SMBUS_HELLOPACKET_LEN] = { 0x55, 0x55, 0x55, 0x55, 0x55 };
- u8 values[ETP_SMBUS_HELLOPACKET_LEN] = { 0, 0, 0, 0, 0 };
+ u8 values[I2C_SMBUS_BLOCK_MAX] = {0};
int len, error;
/* Get hello packet */
@@ -117,12 +117,16 @@ static int elan_smbus_calibrate(struct i2c_client *client)
static int elan_smbus_calibrate_result(struct i2c_client *client, u8 *val)
{
int error;
+ u8 buf[I2C_SMBUS_BLOCK_MAX] = {0};
+
+ BUILD_BUG_ON(ETP_CALIBRATE_MAX_LEN > sizeof(buf));
error = i2c_smbus_read_block_data(client,
- ETP_SMBUS_CALIBRATE_QUERY, val);
+ ETP_SMBUS_CALIBRATE_QUERY, buf);
if (error < 0)
return error;
+ memcpy(val, buf, ETP_CALIBRATE_MAX_LEN);
return 0;
}
@@ -130,7 +134,7 @@ static int elan_smbus_get_baseline_data(struct i2c_client *client,
bool max_baseline, u8 *value)
{
int error;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
error = i2c_smbus_read_block_data(client,
max_baseline ?
@@ -149,7 +153,7 @@ static int elan_smbus_get_version(struct i2c_client *client,
bool iap, u8 *version)
{
int error;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
error = i2c_smbus_read_block_data(client,
iap ? ETP_SMBUS_IAP_VERSION_CMD :
@@ -169,7 +173,7 @@ static int elan_smbus_get_sm_version(struct i2c_client *client,
u8 *ic_type, u8 *version)
{
int error;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
error = i2c_smbus_read_block_data(client,
ETP_SMBUS_SM_VERSION_CMD, val);
@@ -186,7 +190,7 @@ static int elan_smbus_get_sm_version(struct i2c_client *client,
static int elan_smbus_get_product_id(struct i2c_client *client, u16 *id)
{
int error;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
error = i2c_smbus_read_block_data(client,
ETP_SMBUS_UNIQUEID_CMD, val);
@@ -203,7 +207,7 @@ static int elan_smbus_get_checksum(struct i2c_client *client,
bool iap, u16 *csum)
{
int error;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
error = i2c_smbus_read_block_data(client,
iap ? ETP_SMBUS_FW_CHECKSUM_CMD :
@@ -224,7 +228,7 @@ static int elan_smbus_get_max(struct i2c_client *client,
{
int ret;
int error;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
ret = i2c_smbus_read_block_data(client, ETP_SMBUS_RANGE_CMD, val);
if (ret != 3) {
@@ -244,7 +248,7 @@ static int elan_smbus_get_resolution(struct i2c_client *client,
{
int ret;
int error;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
ret = i2c_smbus_read_block_data(client, ETP_SMBUS_RESOLUTION_CMD, val);
if (ret != 3) {
@@ -265,7 +269,7 @@ static int elan_smbus_get_num_traces(struct i2c_client *client,
{
int ret;
int error;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
ret = i2c_smbus_read_block_data(client, ETP_SMBUS_XY_TRACENUM_CMD, val);
if (ret != 3) {
@@ -292,7 +296,7 @@ static int elan_smbus_iap_get_mode(struct i2c_client *client,
{
int error;
u16 constant;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
error = i2c_smbus_read_block_data(client, ETP_SMBUS_IAP_CTRL_CMD, val);
if (error < 0) {
@@ -343,7 +347,7 @@ static int elan_smbus_prepare_fw_update(struct i2c_client *client)
int len;
int error;
enum tp_mode mode;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
u8 cmd[4] = {0x0F, 0x78, 0x00, 0x06};
u16 password;
@@ -417,7 +421,7 @@ static int elan_smbus_write_fw_block(struct i2c_client *client,
struct device *dev = &client->dev;
int error;
u16 result;
- u8 val[3];
+ u8 val[I2C_SMBUS_BLOCK_MAX] = {0};
/*
* Due to the limitation of smbus protocol limiting
@@ -470,6 +474,8 @@ static int elan_smbus_get_report(struct i2c_client *client, u8 *report)
{
int len;
+ BUILD_BUG_ON(I2C_SMBUS_BLOCK_MAX > ETP_SMBUS_REPORT_LEN);
+
len = i2c_smbus_read_block_data(client,
ETP_SMBUS_PACKET_QUERY,
&report[ETP_SMBUS_REPORT_OFFSET]);
diff --git a/drivers/input/mouse/elantech.c b/drivers/input/mouse/elantech.c
index c519c0b..c120afd 100644
--- a/drivers/input/mouse/elantech.c
+++ b/drivers/input/mouse/elantech.c
@@ -800,7 +800,7 @@ static int elantech_packet_check_v4(struct psmouse *psmouse)
else if (ic_version == 7 && etd->samples[1] == 0x2A)
sanity_check = ((packet[3] & 0x1c) == 0x10);
else
- sanity_check = ((packet[0] & 0x0c) == 0x04 &&
+ sanity_check = ((packet[0] & 0x08) == 0x00 &&
(packet[3] & 0x1c) == 0x10);
if (!sanity_check)
@@ -1173,6 +1173,14 @@ static const struct dmi_system_id elantech_dmi_has_middle_button[] = {
{ }
};
+static const char * const middle_button_pnp_ids[] = {
+ "LEN2131", /* ThinkPad P52 w/ NFC */
+ "LEN2132", /* ThinkPad P52 */
+ "LEN2133", /* ThinkPad P72 w/ NFC */
+ "LEN2134", /* ThinkPad P72 */
+ NULL
+};
+
/*
* Set the appropriate event bits for the input subsystem
*/
@@ -1192,7 +1200,8 @@ static int elantech_set_input_params(struct psmouse *psmouse)
__clear_bit(EV_REL, dev->evbit);
__set_bit(BTN_LEFT, dev->keybit);
- if (dmi_check_system(elantech_dmi_has_middle_button))
+ if (dmi_check_system(elantech_dmi_has_middle_button) ||
+ psmouse_matches_pnp_id(psmouse, middle_button_pnp_ids))
__set_bit(BTN_MIDDLE, dev->keybit);
__set_bit(BTN_RIGHT, dev->keybit);
diff --git a/drivers/input/mouse/psmouse-base.c b/drivers/input/mouse/psmouse-base.c
index bee2674..5cbf17a 100644
--- a/drivers/input/mouse/psmouse-base.c
+++ b/drivers/input/mouse/psmouse-base.c
@@ -937,6 +937,21 @@ static void psmouse_apply_defaults(struct psmouse *psmouse)
psmouse->pt_deactivate = NULL;
}
+static bool psmouse_do_detect(int (*detect)(struct psmouse *, bool),
+ struct psmouse *psmouse, bool allow_passthrough,
+ bool set_properties)
+{
+ if (psmouse->ps2dev.serio->id.type == SERIO_PS_PSTHRU &&
+ !allow_passthrough) {
+ return false;
+ }
+
+ if (set_properties)
+ psmouse_apply_defaults(psmouse);
+
+ return detect(psmouse, set_properties) == 0;
+}
+
static bool psmouse_try_protocol(struct psmouse *psmouse,
enum psmouse_type type,
unsigned int *max_proto,
@@ -948,15 +963,8 @@ static bool psmouse_try_protocol(struct psmouse *psmouse,
if (!proto)
return false;
- if (psmouse->ps2dev.serio->id.type == SERIO_PS_PSTHRU &&
- !proto->try_passthru) {
- return false;
- }
-
- if (set_properties)
- psmouse_apply_defaults(psmouse);
-
- if (proto->detect(psmouse, set_properties) != 0)
+ if (!psmouse_do_detect(proto->detect, psmouse, proto->try_passthru,
+ set_properties))
return false;
if (set_properties && proto->init && init_allowed) {
@@ -988,8 +996,8 @@ static int psmouse_extensions(struct psmouse *psmouse,
* Always check for focaltech, this is safe as it uses pnp-id
* matching.
*/
- if (psmouse_try_protocol(psmouse, PSMOUSE_FOCALTECH,
- &max_proto, set_properties, false)) {
+ if (psmouse_do_detect(focaltech_detect,
+ psmouse, false, set_properties)) {
if (max_proto > PSMOUSE_IMEX &&
IS_ENABLED(CONFIG_MOUSE_PS2_FOCALTECH) &&
(!set_properties || focaltech_init(psmouse) == 0)) {
@@ -1035,8 +1043,8 @@ static int psmouse_extensions(struct psmouse *psmouse,
* probing for IntelliMouse.
*/
if (max_proto > PSMOUSE_PS2 &&
- psmouse_try_protocol(psmouse, PSMOUSE_SYNAPTICS, &max_proto,
- set_properties, false)) {
+ psmouse_do_detect(synaptics_detect,
+ psmouse, false, set_properties)) {
synaptics_hardware = true;
if (max_proto > PSMOUSE_IMEX) {
diff --git a/drivers/input/serio/i8042-x86ia64io.h b/drivers/input/serio/i8042-x86ia64io.h
index e484ea2..34be096 100644
--- a/drivers/input/serio/i8042-x86ia64io.h
+++ b/drivers/input/serio/i8042-x86ia64io.h
@@ -527,6 +527,13 @@ static const struct dmi_system_id __initconst i8042_dmi_nomux_table[] = {
DMI_MATCH(DMI_PRODUCT_NAME, "N24_25BU"),
},
},
+ {
+ /* Lenovo LaVie Z */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+ DMI_MATCH(DMI_PRODUCT_VERSION, "Lenovo LaVie Z"),
+ },
+ },
{ }
};
diff --git a/drivers/input/touchscreen/Kconfig b/drivers/input/touchscreen/Kconfig
index 6c4156a..cd25bd9 100644
--- a/drivers/input/touchscreen/Kconfig
+++ b/drivers/input/touchscreen/Kconfig
@@ -12,11 +12,27 @@
if INPUT_TOUCHSCREEN
source "drivers/input/touchscreen/synaptics_dsx_2.6/Kconfig"
+source "drivers/input/touchscreen/stm/Kconfig"
+source "drivers/input/touchscreen/sec_ts/Kconfig"
config TOUCHSCREEN_PROPERTIES
def_tristate INPUT
depends on INPUT
+config TOUCHSCREEN_TBN
+ tristate "Touch Bus Negotiator for Pixel 2018"
+ depends on MSM_QMI_INTERFACE
+ help
+ Say Y here if you have a Google Pixel 2018 device. This module enables
+ support for kernel touch drivers to coordinate bus access with remote
+ processors (e.g. sensor DSP processor), which is required to enable
+ low power touch gesture detection in processors other than AP.
+
+ If unsure, say N.
+
+ To compile this driver as a module, choose M here: the module will be
+ called touch_bus_negotiator.
+
config TOUCHSCREEN_88PM860X
tristate "Marvell 88PM860x touchscreen"
depends on MFD_88PM860X
@@ -287,6 +303,13 @@
To compile this driver as a module, choose M here: the
module will be called hampshire.
+config TOUCHSCREEN_HEATMAP
+ tristate "Heatmap support for touchscreen"
+ select VIDEOBUF2_VMALLOC
+ help
+ Do not explicitly select this option. The drivers that utilize
+ this module must select this in their Kconfig.
+
config TOUCHSCREEN_EETI
tristate "EETI touchscreen panel support"
depends on I2C
@@ -1243,7 +1266,11 @@
If unsure, say N.
source "drivers/input/touchscreen/synaptics_dsx/Kconfig"
-source "drivers/input/touchscreen/focaltech_touch/Kconfig"
+
+# Hide focaltech_touch for now. It appears as if the Kconfig symbol
+# TOUCHSCREEN_FTS is overloaded: It is used for different drivers in
+# stm/Kconfig and focaltech_touch_Kconfig
+# source "drivers/input/touchscreen/focaltech_touch/Kconfig"
config TOUCHSCREEN_FT5X06
tristate "FocalTech touchscreens"
diff --git a/drivers/input/touchscreen/Makefile b/drivers/input/touchscreen/Makefile
index a5952ca..620524b 100644
--- a/drivers/input/touchscreen/Makefile
+++ b/drivers/input/touchscreen/Makefile
@@ -103,5 +103,12 @@
obj-$(CONFIG_TOUCHSCREEN_ZFORCE) += zforce_ts.o
obj-$(CONFIG_TOUCHSCREEN_COLIBRI_VF50) += colibri-vf50-ts.o
obj-$(CONFIG_TOUCHSCREEN_ROHM_BU21023) += rohm_bu21023.o
-obj-$(CONFIG_TOUCHSCREEN_FTS) += focaltech_touch/
+# Hide focaltech_touch for now. It appears as if the Kconfig symbol
+# TOUCHSCREEN_FTS is overloaded: It is used for different drivers in
+# stm/Kconfig and focaltech_touch_Kconfig
+# obj-$(CONFIG_TOUCHSCREEN_FTS) += focaltech_touch/
obj-$(CONFIG_TOUCHSCREEN_HIMAX_CHIPSET) += hxchipset/
+obj-$(CONFIG_TOUCHSCREEN_HEATMAP) += heatmap.o
+obj-$(CONFIG_TOUCHSCREEN_FTM4) += stm/
+obj-$(CONFIG_TOUCHSCREEN_SEC_TS) += sec_ts/
+obj-$(CONFIG_TOUCHSCREEN_TBN) += qmi_tbn_v01.o touch_bus_negotiator.o
\ No newline at end of file
diff --git a/drivers/input/touchscreen/atmel_mxt_ts.c b/drivers/input/touchscreen/atmel_mxt_ts.c
index e5d185f..c2fb023 100644
--- a/drivers/input/touchscreen/atmel_mxt_ts.c
+++ b/drivers/input/touchscreen/atmel_mxt_ts.c
@@ -1671,10 +1671,11 @@ static int mxt_get_object_table(struct mxt_data *data)
break;
case MXT_TOUCH_MULTI_T9:
data->multitouch = MXT_TOUCH_MULTI_T9;
+ /* Only handle messages from first T9 instance */
data->T9_reportid_min = min_id;
- data->T9_reportid_max = max_id;
- data->num_touchids = object->num_report_ids
- * mxt_obj_instances(object);
+ data->T9_reportid_max = min_id +
+ object->num_report_ids - 1;
+ data->num_touchids = object->num_report_ids;
break;
case MXT_SPT_MESSAGECOUNT_T44:
data->T44_address = object->start_address;
@@ -3028,6 +3029,15 @@ static const struct dmi_system_id mxt_dmi_table[] = {
.driver_data = samus_platform_data,
},
{
+ /* Samsung Chromebook Pro */
+ .ident = "Samsung Chromebook Pro",
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Google"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "Caroline"),
+ },
+ .driver_data = samus_platform_data,
+ },
+ {
/* Other Google Chromebooks */
.ident = "Chromebook",
.matches = {
diff --git a/drivers/input/touchscreen/goodix.c b/drivers/input/touchscreen/goodix.c
index 5907fdd..c599b5a 100644
--- a/drivers/input/touchscreen/goodix.c
+++ b/drivers/input/touchscreen/goodix.c
@@ -858,6 +858,7 @@ MODULE_DEVICE_TABLE(i2c, goodix_ts_id);
#ifdef CONFIG_ACPI
static const struct acpi_device_id goodix_acpi_match[] = {
{ "GDIX1001", 0 },
+ { "GDIX1002", 0 },
{ }
};
MODULE_DEVICE_TABLE(acpi, goodix_acpi_match);
diff --git a/drivers/input/touchscreen/heatmap.c b/drivers/input/touchscreen/heatmap.c
new file mode 100644
index 0000000..d7e08fb
--- /dev/null
+++ b/drivers/input/touchscreen/heatmap.c
@@ -0,0 +1,392 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/module.h>
+#include <linux/input.h>
+#include <linux/input/heatmap.h>
+#include <media/videobuf2-vmalloc.h>
+
+static const int FIRST_FREE_NODE = -1;
+
+/**
+ * Optimization: keep track of how many consecutive frames
+ * have been dropped due to not having any buffers available.
+ * If too many have been recently dropped, and still no free buffers
+ * are available, then skip the bus read.
+ * This situation could happen if an app have opened the video device,
+ * but went into paused state, and did not close the video device in
+ * onPause. With this optimization, we would avoid the wasteful bus reads
+ * when no one is likely to consume the buffers.
+ */
+static const unsigned int NUM_BUFFERS_BEFORE_DROP = 3;
+static unsigned int consecutive_frames_dropped;
+
+struct heatmap_vb2_buffer {
+ struct vb2_v4l2_buffer v4l2_vb;
+ struct list_head list;
+};
+
+static int heatmap_set_input(
+ struct v4l2_heatmap *v4l2, unsigned int input_index)
+{
+ struct v4l2_pix_format *fmt = &v4l2->format;
+
+ if (input_index != 0)
+ return -EINVAL;
+
+ /*
+ * Changing the input implies a format change, which is not allowed
+ * while buffers for use with streaming have already been allocated.
+ */
+ if (vb2_is_busy(&v4l2->queue))
+ return -EBUSY;
+
+ v4l2->input_index = input_index;
+
+ fmt->width = v4l2->width;
+ fmt->height = v4l2->height;
+ fmt->pixelformat = V4L2_TCH_FMT_DELTA_TD16;
+ fmt->field = V4L2_FIELD_NONE;
+ fmt->colorspace = V4L2_COLORSPACE_RAW;
+ fmt->bytesperline = fmt->width * sizeof(strength_t);
+ fmt->sizeimage = fmt->width * fmt->height * sizeof(strength_t);
+
+ return 0;
+}
+
+static inline struct heatmap_vb2_buffer *to_heatmap_vb2_buffer(
+ struct vb2_buffer *vb2)
+{
+ return container_of(to_vb2_v4l2_buffer(vb2), struct heatmap_vb2_buffer,
+ v4l2_vb);
+}
+
+static const struct v4l2_file_operations heatmap_video_fops = {
+ .owner = THIS_MODULE,
+ .open = v4l2_fh_open,
+ .release = vb2_fop_release,
+ .unlocked_ioctl = video_ioctl2,
+ .read = vb2_fop_read,
+ .mmap = vb2_fop_mmap,
+ .poll = vb2_fop_poll,
+};
+
+void heatmap_read(struct v4l2_heatmap *v4l2, uint64_t timestamp)
+{
+ struct heatmap_vb2_buffer *new_buf;
+ struct vb2_buffer *vb2_buf;
+ strength_t *data;
+ int total_bytes = v4l2->format.sizeimage;
+ strength_t temp_buffer[total_bytes];
+ bool read_success;
+
+ if (!vb2_is_streaming(&v4l2->queue)) {
+ /* No need to read, no one is viewing the video */
+ return;
+ }
+
+ /* Optimization */
+ if (consecutive_frames_dropped >= NUM_BUFFERS_BEFORE_DROP) {
+ spin_lock(&v4l2->heatmap_lock);
+ if (list_empty(&v4l2->heatmap_buffer_list)) {
+ /*
+ * Already dropped some frames, and still don't have
+ * any free buffers. A buffer could become available
+ * during read_frame(..), but given that we already
+ * dropped some frames, this is unlikely.
+ */
+ spin_unlock(&v4l2->heatmap_lock);
+ return; /* Drop the frame */
+ }
+ spin_unlock(&v4l2->heatmap_lock);
+ }
+
+ /* This is a potentially slow operation */
+ read_success = v4l2->read_frame(v4l2, temp_buffer);
+ if (!read_success)
+ return;
+
+ /* Copy the data into the buffer */
+ spin_lock(&v4l2->heatmap_lock);
+ if (list_empty(&v4l2->heatmap_buffer_list)) {
+ /*
+ * If streaming is off, then there would
+ * be no queued buffers. This is expected.
+ * On the other hand, if there is a consumer, but there
+ * aren't any available buffers, then this indicates
+ * slowness in the userspace for reading or
+ * processing buffers.
+ */
+ dev_warn(v4l2->parent_dev, "heatmap: No buffers available, dropping frame\n");
+ consecutive_frames_dropped++;
+ spin_unlock(&v4l2->heatmap_lock);
+ return;
+ }
+ consecutive_frames_dropped = 0;
+ new_buf = list_entry(v4l2->heatmap_buffer_list.next,
+ struct heatmap_vb2_buffer, list);
+ list_del(&new_buf->list);
+
+ vb2_buf = &new_buf->v4l2_vb.vb2_buf;
+ data = vb2_plane_vaddr(vb2_buf, 0);
+ if (!data) {
+ dev_err(v4l2->parent_dev, "heatmap: Error acquiring frame pointer\n");
+ vb2_buffer_done(vb2_buf, VB2_BUF_STATE_ERROR);
+ spin_unlock(&v4l2->heatmap_lock);
+ return;
+ }
+
+ memcpy(data, temp_buffer, total_bytes);
+ vb2_set_plane_payload(vb2_buf, /* plane number */ 0, total_bytes);
+ vb2_buf->timestamp = timestamp;
+ vb2_buffer_done(vb2_buf, VB2_BUF_STATE_DONE);
+ spin_unlock(&v4l2->heatmap_lock);
+}
+EXPORT_SYMBOL(heatmap_read);
+
+static void heatmap_buffer_queue(struct vb2_buffer *vb)
+{
+ struct v4l2_heatmap *v4l2 = vb2_get_drv_priv(vb->vb2_queue);
+ struct heatmap_vb2_buffer *heatmap_buffer = to_heatmap_vb2_buffer(vb);
+
+ spin_lock(&v4l2->heatmap_lock);
+ list_add_tail(&heatmap_buffer->list, &v4l2->heatmap_buffer_list);
+ spin_unlock(&v4l2->heatmap_lock);
+}
+
+static int heatmap_queue_setup(struct vb2_queue *vq,
+ unsigned int *num_buffers, unsigned int *num_planes,
+ unsigned int sizes[], struct device *alloc_devs[]) {
+ struct v4l2_heatmap *v4l2 = vb2_get_drv_priv(vq);
+ size_t size = v4l2->format.sizeimage;
+
+ if (*num_planes != 0)
+ return sizes[0] < size ? -EINVAL : 0;
+
+ *num_planes = 1;
+ sizes[0] = size;
+
+ return 0;
+}
+
+static void return_all_buffers(struct v4l2_heatmap *v4l2,
+ enum vb2_buffer_state state)
+{
+ struct heatmap_vb2_buffer *buf, *node;
+
+ spin_lock(&v4l2->heatmap_lock);
+ list_for_each_entry_safe(buf, node, &v4l2->heatmap_buffer_list, list) {
+ vb2_buffer_done(&buf->v4l2_vb.vb2_buf, state);
+ list_del(&buf->list);
+ }
+ spin_unlock(&v4l2->heatmap_lock);
+}
+
+/*
+ * Stop the DMA engine. Any remaining buffers in the DMA queue are dequeued
+ * and passed on to the vb2 framework marked as STATE_ERROR.
+ */
+static void stop_streaming(struct vb2_queue *vq)
+{
+ struct v4l2_heatmap *v4l2 = vb2_get_drv_priv(vq);
+ /* Release all active buffers */
+ return_all_buffers(v4l2, VB2_BUF_STATE_ERROR);
+}
+
+/* V4L2 structures */
+static const struct vb2_ops heatmap_queue_ops = {
+ .queue_setup = heatmap_queue_setup,
+ .buf_queue = heatmap_buffer_queue,
+ .stop_streaming = stop_streaming,
+ .wait_prepare = vb2_ops_wait_prepare,
+ .wait_finish = vb2_ops_wait_finish,
+};
+
+static const struct vb2_queue heatmap_queue = {
+ .type = V4L2_BUF_TYPE_VIDEO_CAPTURE,
+ .io_modes = VB2_MMAP | VB2_USERPTR | VB2_DMABUF | VB2_READ,
+ .buf_struct_size = sizeof(struct heatmap_vb2_buffer),
+ .ops = &heatmap_queue_ops,
+ .mem_ops = &vb2_vmalloc_memops,
+ .timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC,
+ .min_buffers_needed = 1,
+};
+
+static int heatmap_vidioc_querycap(struct file *file, void *priv,
+ struct v4l2_capability *cap) {
+ struct v4l2_heatmap *v4l2 = video_drvdata(file);
+ strlcpy(cap->driver, v4l2->parent_dev->driver->name,
+ sizeof(cap->driver));
+ if (v4l2->input_dev != NULL) {
+ strlcpy(cap->card, v4l2->input_dev->name, sizeof(cap->card));
+ } else {
+ strlcpy(cap->card, KBUILD_MODNAME, sizeof(cap->card));
+ }
+
+ strlcpy(cap->bus_info, dev_name(v4l2->parent_dev),
+ sizeof(cap->bus_info));
+ cap->capabilities = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_TOUCH |
+ V4L2_CAP_READWRITE | V4L2_CAP_STREAMING | V4L2_CAP_DEVICE_CAPS;
+ return 0;
+}
+
+static int heatmap_vidioc_enum_input(struct file *file, void *priv,
+ struct v4l2_input *video_input) {
+ if (video_input->index != 0)
+ return -EINVAL;
+
+ video_input->type = V4L2_INPUT_TYPE_TOUCH;
+ strlcpy(video_input->name, "strength", sizeof(video_input->name));
+ return 0;
+}
+
+static int heatmap_vidioc_s_input(
+ struct file *file, void *priv, unsigned int input_index)
+{
+ struct v4l2_heatmap *v4l2 = video_drvdata(file);
+ return heatmap_set_input(v4l2, input_index);
+}
+
+static int heatmap_vidioc_g_input(struct file *file, void *priv,
+ unsigned int *input_index)
+{
+ struct v4l2_heatmap *v4l2 = video_drvdata(file);
+ *input_index = v4l2->input_index;
+ return 0;
+}
+
+static int heatmap_vidioc_fmt(struct file *file, void *priv,
+ struct v4l2_format *fmt)
+{
+ struct v4l2_heatmap *v4l2 = video_drvdata(file);
+
+ fmt->type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+ fmt->fmt.pix = v4l2->format;
+
+ return 0;
+}
+
+static int heatmap_vidioc_enum_fmt(struct file *file, void *priv,
+ struct v4l2_fmtdesc *fmt)
+{
+ if (fmt->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+
+ if (fmt->index != 0)
+ return -EINVAL;
+
+ fmt->pixelformat = V4L2_TCH_FMT_DELTA_TD16;
+ return 0;
+}
+
+static int heatmap_vidioc_g_parm(struct file *file, void *fh,
+ struct v4l2_streamparm *streamparm)
+{
+ struct v4l2_heatmap *v4l2 = video_drvdata(file);
+
+ if (streamparm->type != V4L2_BUF_TYPE_VIDEO_CAPTURE)
+ return -EINVAL;
+
+ streamparm->parm.capture.capability = V4L2_CAP_TIMEPERFRAME;
+ streamparm->parm.capture.readbuffers = 1;
+ streamparm->parm.capture.timeperframe.numerator =
+ v4l2->timeperframe.numerator;
+ streamparm->parm.capture.timeperframe.denominator =
+ v4l2->timeperframe.denominator;
+ return 0;
+}
+
+static const struct v4l2_ioctl_ops heatmap_video_ioctl_ops = {
+ .vidioc_querycap = heatmap_vidioc_querycap,
+
+ .vidioc_enum_fmt_vid_cap = heatmap_vidioc_enum_fmt,
+ .vidioc_s_fmt_vid_cap = heatmap_vidioc_fmt,
+ .vidioc_g_fmt_vid_cap = heatmap_vidioc_fmt,
+ .vidioc_try_fmt_vid_cap = heatmap_vidioc_fmt,
+ .vidioc_g_parm = heatmap_vidioc_g_parm,
+
+ .vidioc_enum_input = heatmap_vidioc_enum_input,
+ .vidioc_g_input = heatmap_vidioc_g_input,
+ .vidioc_s_input = heatmap_vidioc_s_input,
+
+ .vidioc_reqbufs = vb2_ioctl_reqbufs,
+ .vidioc_create_bufs = vb2_ioctl_create_bufs,
+ .vidioc_querybuf = vb2_ioctl_querybuf,
+ .vidioc_qbuf = vb2_ioctl_qbuf,
+ .vidioc_dqbuf = vb2_ioctl_dqbuf,
+ .vidioc_expbuf = vb2_ioctl_expbuf,
+
+ .vidioc_streamon = vb2_ioctl_streamon,
+ .vidioc_streamoff = vb2_ioctl_streamoff,
+};
+
+static const struct video_device heatmap_video_device = {
+ .name = "heatmap",
+ .fops = &heatmap_video_fops,
+ .ioctl_ops = &heatmap_video_ioctl_ops,
+ .release = video_device_release_empty,
+ .device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_TOUCH |
+ V4L2_CAP_READWRITE | V4L2_CAP_STREAMING,
+};
+
+int heatmap_probe(struct v4l2_heatmap *v4l2)
+{
+ int error;
+
+ /* init channel to zero */
+ heatmap_set_input(v4l2, 0);
+
+ /* register video device */
+ strlcpy(v4l2->device.name, dev_name(v4l2->parent_dev),
+ V4L2_DEVICE_NAME_SIZE);
+ error = v4l2_device_register(v4l2->parent_dev, &v4l2->device);
+ if (error)
+ goto err_probe;
+
+ INIT_LIST_HEAD(&v4l2->heatmap_buffer_list);
+
+ /* initialize the queue */
+ spin_lock_init(&v4l2->heatmap_lock);
+ mutex_init(&v4l2->lock);
+
+ v4l2->queue = heatmap_queue;
+ v4l2->queue.drv_priv = v4l2;
+ v4l2->queue.lock = &v4l2->lock;
+
+ error = vb2_queue_init(&v4l2->queue);
+ if (error)
+ goto err_unreg_v4l2;
+
+ v4l2->vdev = heatmap_video_device;
+
+ v4l2->vdev.v4l2_dev = &v4l2->device;
+ v4l2->vdev.lock = &v4l2->lock;
+ v4l2->vdev.vfl_dir = VFL_DIR_RX;
+ v4l2->vdev.queue = &v4l2->queue;
+ video_set_drvdata(&v4l2->vdev, v4l2);
+
+ error = video_register_device(&v4l2->vdev, VFL_TYPE_TOUCH,
+ FIRST_FREE_NODE);
+ if (error)
+ goto err_video_device_release;
+ return 0;
+
+err_video_device_release:
+ video_device_release(&v4l2->vdev);
+
+err_unreg_v4l2:
+ v4l2_device_unregister(&v4l2->device);
+err_probe:
+ return error;
+}
+EXPORT_SYMBOL(heatmap_probe);
+
+void heatmap_remove(struct v4l2_heatmap *v4l2)
+{
+ video_unregister_device(&v4l2->vdev);
+ v4l2_device_unregister(&v4l2->device);
+}
+EXPORT_SYMBOL(heatmap_remove);
+
+MODULE_DESCRIPTION("Touchscreen heatmap video device");
+MODULE_AUTHOR("Siarhei Vishniakou <svv@google.com>");
+MODULE_LICENSE("GPL v2");
\ No newline at end of file
diff --git a/drivers/input/touchscreen/qmi_tbn_v01.c b/drivers/input/touchscreen/qmi_tbn_v01.c
new file mode 100644
index 0000000..a92287e
--- /dev/null
+++ b/drivers/input/touchscreen/qmi_tbn_v01.c
@@ -0,0 +1,69 @@
+/* drivers/input/touchscreen/qmi_tbn_v01.c
+ *
+ * QMI message interface definition for Touch Bus Negotiator.
+ *
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/qmi_encdec.h>
+#include "qmi_tbn_v01.h"
+
+struct elem_info tbn_kernel_request_bus_v01_ei[] = {
+ {
+ .data_type = QMI_EOTI,
+ .is_array = NO_ARRAY,
+ .tlv_type = QMI_COMMON_TLV_TYPE,
+ },
+};
+
+struct elem_info tbn_kernel_release_bus_v01_ei[] = {
+ {
+ .data_type = QMI_EOTI,
+ .is_array = NO_ARRAY,
+ .tlv_type = QMI_COMMON_TLV_TYPE,
+ },
+};
+
+struct elem_info tbn_ssc_release_bus_v01_ei[] = {
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size = sizeof(struct qmi_response_type_v01),
+ .is_array = NO_ARRAY,
+ .tlv_type = 0x02,
+ .offset = offsetof(struct tbn_ssc_release_bus_v01, resp),
+ .ei_array = get_qmi_response_type_v01_ei(),
+ },
+ {
+ .data_type = QMI_EOTI,
+ .is_array = NO_ARRAY,
+ .tlv_type = QMI_COMMON_TLV_TYPE,
+ },
+};
+
+struct elem_info tbn_ssc_acquire_bus_v01_ei[] = {
+ {
+ .data_type = QMI_STRUCT,
+ .elem_len = 1,
+ .elem_size = sizeof(struct qmi_response_type_v01),
+ .is_array = NO_ARRAY,
+ .tlv_type = 0x02,
+ .offset = offsetof(struct tbn_ssc_acquire_bus_v01, resp),
+ .ei_array = get_qmi_response_type_v01_ei(),
+ },
+ {
+ .data_type = QMI_EOTI,
+ .is_array = NO_ARRAY,
+ .tlv_type = QMI_COMMON_TLV_TYPE,
+ },
+};
diff --git a/drivers/input/touchscreen/qmi_tbn_v01.h b/drivers/input/touchscreen/qmi_tbn_v01.h
new file mode 100644
index 0000000..c8a481c
--- /dev/null
+++ b/drivers/input/touchscreen/qmi_tbn_v01.h
@@ -0,0 +1,61 @@
+/* drivers/input/touchscreen/qmi_tbn_v01.h
+ *
+ * QMI message interface definition for Touch Bus Negotiator.
+ *
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#ifndef QMI_TBN_V01_H
+#define QMI_TBN_V01_H
+
+#include <soc/qcom/msm_qmi_interface.h>
+
+#define TBN_SERVICE_ID_V01 232
+#define TBN_SERVICE_VERS_V01 1
+
+#define QMI_TBN_KERNEL_REQUEST_BUS_V01 0x0020
+#define QMI_TBN_SSC_RELEASE_BUS_V01 0x0020
+
+#define QMI_TBN_KERNEL_RELEASE_BUS_V01 0X0021
+#define QMI_TBN_SSC_ACQUIRE_BUS_V01 0x0021
+
+
+struct tbn_kernel_request_bus_v01 {
+ char placeholder;
+};
+#define TBN_KERNEL_REQUEST_BUS_V01_MAX_MSG_LEN 0
+extern struct elem_info tbn_kernel_request_bus_v01_ei[];
+
+
+struct tbn_kernel_release_bus_v01 {
+ char placeholder;
+};
+#define TBN_KERNEL_RELEASE_BUS_V01_MAX_MSG_LEN 0
+extern struct elem_info tbn_kernel_release_bus_v01_ei[];
+
+
+struct tbn_ssc_release_bus_v01 {
+ struct qmi_response_type_v01 resp;
+};
+#define TBN_SSC_RELEASE_BUS_V01_MAX_MSG_LEN 7
+extern struct elem_info tbn_ssc_release_bus_v01_ei[];
+
+
+struct tbn_ssc_acquire_bus_v01 {
+ struct qmi_response_type_v01 resp;
+};
+#define TBN_SSC_ACQUIRE_BUS_V01_MAX_MSG_LEN 7
+extern struct elem_info tbn_ssc_acquire_bus_v01_ei[];
+
+
+#endif // QMI_TBN_V01_H
diff --git a/drivers/input/touchscreen/rohm_bu21023.c b/drivers/input/touchscreen/rohm_bu21023.c
index 611156a..be29984 100644
--- a/drivers/input/touchscreen/rohm_bu21023.c
+++ b/drivers/input/touchscreen/rohm_bu21023.c
@@ -304,7 +304,7 @@ static int rohm_i2c_burst_read(struct i2c_client *client, u8 start, void *buf,
msg[1].len = len;
msg[1].buf = buf;
- i2c_lock_adapter(adap);
+ i2c_lock_bus(adap, I2C_LOCK_SEGMENT);
for (i = 0; i < 2; i++) {
if (__i2c_transfer(adap, &msg[i], 1) < 0) {
@@ -313,7 +313,7 @@ static int rohm_i2c_burst_read(struct i2c_client *client, u8 start, void *buf,
}
}
- i2c_unlock_adapter(adap);
+ i2c_unlock_bus(adap, I2C_LOCK_SEGMENT);
return ret;
}
diff --git a/drivers/input/touchscreen/sec_ts/Kconfig b/drivers/input/touchscreen/sec_ts/Kconfig
new file mode 100644
index 0000000..5193e28
--- /dev/null
+++ b/drivers/input/touchscreen/sec_ts/Kconfig
@@ -0,0 +1,23 @@
+#
+# Samsung Electronics TOUCH driver configuration
+#
+
+config TOUCHSCREEN_SEC_TS
+ tristate "Samsung Electronics Touchscreen"
+ depends on I2C
+ select TOUCHSCREEN_HEATMAP
+ help
+ Say Y here if you want support for SEC touchscreen controllers.
+ If unsure, say N.
+
+ To compile this driver as a module, choose M here: the
+ module will be called sec_ts.
+
+config TOUCHSCREEN_SEC_TS_GLOVEMODE
+ tristate "Samsung Electronics Touchscreen"
+ depends on I2C
+ help
+ Say Y here if you have a Samsung Electronics Touchscreen and want to enable
+ support for the built-in touchscreen.
+
+ If unsure, say N.
diff --git a/drivers/input/touchscreen/sec_ts/Makefile b/drivers/input/touchscreen/sec_ts/Makefile
new file mode 100644
index 0000000..04453d7
--- /dev/null
+++ b/drivers/input/touchscreen/sec_ts/Makefile
@@ -0,0 +1,3 @@
+obj-$(CONFIG_TOUCHSCREEN_SEC_TS) += sec_touch.o
+sec_touch-objs += sec_cmd.o sec_ts.o sec_ts_fw.o sec_ts_fn.o \
+ sec_ts_only_vendor.o
diff --git a/drivers/input/touchscreen/sec_ts/sec_cmd.c b/drivers/input/touchscreen/sec_ts/sec_cmd.c
new file mode 100644
index 0000000..3ac058a
--- /dev/null
+++ b/drivers/input/touchscreen/sec_ts/sec_cmd.c
@@ -0,0 +1,524 @@
+/*
+ * sec_cmd.c - samsung factory command driver
+ *
+ * Copyright (C) 2014 Samsung Electronics
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/input/sec_cmd.h>
+
+#if defined USE_SEC_CMD_QUEUE
+static void sec_cmd_store_function(struct sec_cmd_data *data);
+#endif
+
+void sec_cmd_set_cmd_exit(struct sec_cmd_data *data)
+{
+ atomic_set(&data->cmd_is_running, 0);
+
+#ifdef USE_SEC_CMD_QUEUE
+ mutex_lock(&data->fifo_lock);
+ if (kfifo_len(&data->cmd_queue)) {
+ pr_info("%s %s: do next cmd, left cmd[%d]\n", SECLOG, __func__,
+ (int)(kfifo_len(&data->cmd_queue) / sizeof(struct command)));
+ mutex_unlock(&data->fifo_lock);
+
+ atomic_set(&data->cmd_is_running, 1);
+
+ data->cmd_state = SEC_CMD_STATUS_RUNNING;
+ sec_cmd_store_function(data);
+
+ } else {
+ mutex_unlock(&data->fifo_lock);
+ }
+#endif
+}
+
+void sec_cmd_set_default_result(struct sec_cmd_data *data)
+{
+ char delim = ':';
+ memset(data->cmd_result, 0x00, SEC_CMD_RESULT_STR_LEN);
+ memcpy(data->cmd_result, data->cmd, SEC_CMD_STR_LEN);
+ strncat(data->cmd_result, &delim, 1);
+}
+
+void sec_cmd_set_cmd_result(struct sec_cmd_data *data, char *buff, int len)
+{
+ strlcat(data->cmd_result, buff, SEC_CMD_RESULT_STR_LEN);
+}
+
+#ifndef USE_SEC_CMD_QUEUE
+static ssize_t sec_cmd_store(struct device *dev,
+ struct device_attribute *devattr, const char *buf, size_t count)
+{
+ struct sec_cmd_data *data = dev_get_drvdata(dev);
+ char *cur, *start, *end;
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ size_t len;
+ struct sec_cmd *sec_cmd_ptr = NULL;
+ char delim = ',';
+ bool cmd_found = false;
+ unsigned int i, param_cnt = 0;
+
+ if (!data) {
+ pr_err("%s %s: No platform data found\n", SECLOG, __func__);
+ return -EINVAL;
+ }
+
+ if (count >= SEC_CMD_STR_LEN) {
+ pr_err("%s %s: cmd length is over (%s,%d)!!\n",
+ SECLOG, __func__, buf, (int)count);
+ return -EINVAL;
+ }
+
+ if (atomic_cmpxchg(&data->cmd_is_running, 0 , 1)) {
+ pr_err("%s %s: other cmd is running.\n", SECLOG, __func__);
+ return -EBUSY;
+ }
+
+ data->cmd_state = SEC_CMD_STATUS_RUNNING;
+ for (i = 0; i < ARRAY_SIZE(data->cmd_param); i++)
+ data->cmd_param[i] = 0;
+
+ len = count;
+ if (*(buf + len - 1) == '\n')
+ len--;
+
+ memset(data->cmd, 0x00, ARRAY_SIZE(data->cmd));
+ memcpy(data->cmd, buf, len);
+
+ cur = strchr(buf, (int)delim);
+ if (cur)
+ memcpy(buff, buf, cur - buf);
+ else
+ memcpy(buff, buf, len);
+
+ pr_debug("%s %s: COMMAND = %s\n", SECLOG, __func__, buff);
+
+ /* find command */
+ list_for_each_entry(sec_cmd_ptr, &data->cmd_list_head, list) {
+ if (!strncmp(buff, sec_cmd_ptr->cmd_name, SEC_CMD_STR_LEN)) {
+ cmd_found = true;
+ break;
+ }
+ }
+
+ /* set not_support_cmd */
+ if (!cmd_found) {
+ list_for_each_entry(sec_cmd_ptr, &data->cmd_list_head, list) {
+ if (!strncmp("not_support_cmd", sec_cmd_ptr->cmd_name,
+ SEC_CMD_STR_LEN))
+ break;
+ }
+ }
+
+ /* parsing parameters */
+ if (cur && cmd_found) {
+ cur++;
+ start = cur;
+ memset(buff, 0x00, ARRAY_SIZE(buff));
+
+ do {
+ if (*cur == delim || cur - buf == len) {
+ end = cur;
+ memcpy(buff, start, end - start);
+ *(buff + strnlen(buff, ARRAY_SIZE(buff))) = '\0';
+ if (kstrtoint(buff, 10, data->cmd_param + param_cnt) < 0)
+ goto err_out;
+ start = cur + 1;
+ memset(buff, 0x00, ARRAY_SIZE(buff));
+ param_cnt++;
+ }
+ cur++;
+ } while ((cur - buf <= len) && (param_cnt < SEC_CMD_PARAM_NUM));
+ }
+
+ if (cmd_found) {
+ pr_info("%s %s: cmd = %s", SECLOG, __func__, sec_cmd_ptr->cmd_name);
+ for (i = 0; i < param_cnt; i++) {
+ if (i == 0)
+ pr_cont(" param =");
+ pr_cont(" %d", data->cmd_param[i]);
+ }
+ pr_cont("\n");
+ } else {
+ pr_info("%s %s: cmd = %s(%s)\n", SECLOG, __func__, buff, sec_cmd_ptr->cmd_name);
+ }
+
+ sec_cmd_ptr->cmd_func(data);
+ sec_cmd_set_cmd_exit(data);
+
+err_out:
+ return count;
+}
+
+#else /* defined USE_SEC_CMD_QUEUE */
+static void sec_cmd_store_function(struct sec_cmd_data *data)
+{
+ char *cur, *start, *end;
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ int len, i;
+ struct sec_cmd *sec_cmd_ptr = NULL;
+ char delim = ',';
+ bool cmd_found = false;
+ int param_cnt = 0;
+ int ret;
+ const char *buf;
+ size_t count;
+ struct command cmd = {{0}};
+
+ if (!data) {
+ pr_err("%s %s: No platform data found\n", SECLOG, __func__);
+ return;
+ }
+
+ mutex_lock(&data->fifo_lock);
+ if (kfifo_len(&data->cmd_queue)) {
+ ret = kfifo_out(&data->cmd_queue, &cmd, sizeof(struct command));
+ if (!ret) {
+ pr_err("%s %s: kfifo_out failed, it seems empty, ret=%d\n", SECLOG, __func__, ret);
+ mutex_unlock(&data->fifo_lock);
+ return;
+ }
+ } else {
+ pr_err("%s %s: left cmd is nothing\n", SECLOG, __func__);
+ mutex_unlock(&data->fifo_lock);
+ return;
+ }
+ mutex_unlock(&data->fifo_lock);
+
+ buf = cmd.cmd;
+ count = strlen(buf);
+
+ for (i = 0; i < (int)ARRAY_SIZE(data->cmd_param); i++)
+ data->cmd_param[i] = 0;
+
+ len = (int)count;
+ if (*(buf + len - 1) == '\n')
+ len--;
+
+ memset(data->cmd, 0x00, ARRAY_SIZE(data->cmd));
+ memcpy(data->cmd, buf, len);
+
+ cur = strchr(buf, (int)delim);
+ if (cur)
+ memcpy(buff, buf, cur - buf);
+ else
+ memcpy(buff, buf, len);
+
+ pr_debug("%s %s: COMMAND : %s\n", SECLOG, __func__, buff);
+
+ /* find command */
+ list_for_each_entry(sec_cmd_ptr, &data->cmd_list_head, list) {
+ if (!strncmp(buff, sec_cmd_ptr->cmd_name, SEC_CMD_STR_LEN)) {
+ cmd_found = true;
+ break;
+ }
+ }
+
+ /* set not_support_cmd */
+ if (!cmd_found) {
+ list_for_each_entry(sec_cmd_ptr, &data->cmd_list_head, list) {
+ if (!strncmp("not_support_cmd", sec_cmd_ptr->cmd_name,
+ SEC_CMD_STR_LEN))
+ break;
+ }
+ }
+
+ /* parsing parameters */
+ if (cur && cmd_found) {
+ cur++;
+ start = cur;
+ memset(buff, 0x00, ARRAY_SIZE(buff));
+
+ do {
+ if (*cur == delim || cur - buf == len) {
+ end = cur;
+ memcpy(buff, start, end - start);
+ *(buff + strnlen(buff, ARRAY_SIZE(buff))) = '\0';
+ if (kstrtoint(buff, 10, data->cmd_param + param_cnt) < 0)
+ return;
+ start = cur + 1;
+ memset(buff, 0x00, ARRAY_SIZE(buff));
+ param_cnt++;
+ }
+ cur++;
+ } while ((cur - buf <= len) && (param_cnt < SEC_CMD_PARAM_NUM));
+ }
+
+ if (cmd_found) {
+ pr_info("%s %s: cmd = %s", SECLOG, __func__, sec_cmd_ptr->cmd_name);
+ for (i = 0; i < param_cnt; i++) {
+ if (i == 0)
+ pr_cont(" param =");
+ pr_cont(" %d", data->cmd_param[i]);
+ }
+ pr_cont("\n");
+ } else {
+ pr_info("%s %s: cmd = %s(%s)\n", SECLOG, __func__, buff, sec_cmd_ptr->cmd_name);
+ }
+
+ sec_cmd_ptr->cmd_func(data);
+
+}
+
+static ssize_t sec_cmd_store(struct device *dev, struct device_attribute *devattr,
+ const char *buf, size_t count)
+{
+ struct sec_cmd_data *data = dev_get_drvdata(dev);
+ struct command cmd = {{0}};
+ int queue_size;
+
+ if (!data) {
+ pr_err("%s %s: No platform data found\n", SECLOG, __func__);
+ return -EINVAL;
+ }
+
+ if (count >= SEC_CMD_STR_LEN) {
+ pr_err("%s %s: cmd length is over (%s,%d)!!\n", SECLOG,
+ __func__, buf, (int)count);
+ return -EINVAL;
+ }
+
+ strlcpy(cmd.cmd, buf, sizeof(cmd.cmd));
+
+ mutex_lock(&data->fifo_lock);
+ queue_size = (kfifo_len(&data->cmd_queue) / sizeof(struct command));
+
+ if (kfifo_avail(&data->cmd_queue) && (queue_size < SEC_CMD_MAX_QUEUE)) {
+ kfifo_in(&data->cmd_queue, &cmd, sizeof(struct command));
+ pr_info("%s %s: push cmd: %s\n", SECLOG, __func__, cmd.cmd);
+ } else {
+ pr_err("%s %s: cmd_queue is full!!\n", SECLOG, __func__);
+
+ kfifo_reset(&data->cmd_queue);
+ pr_err("%s %s: cmd_queue is reset!!\n", SECLOG, __func__);
+ mutex_unlock(&data->fifo_lock);
+
+ atomic_set(&data->cmd_is_running, 0);
+
+ return -ENOSPC;
+ }
+
+ if (atomic_cmpxchg(&data->cmd_is_running, 0 , 1)) {
+ pr_err("%s %s: other cmd is running. Wait until previous cmd is done[%d]\n",
+ SECLOG, __func__, (int)(kfifo_len(&data->cmd_queue) / sizeof(struct command)));
+ mutex_unlock(&data->fifo_lock);
+
+ return -EBUSY;
+ }
+
+ mutex_unlock(&data->fifo_lock);
+
+ data->cmd_state = SEC_CMD_STATUS_RUNNING;
+ sec_cmd_store_function(data);
+
+ return count;
+}
+#endif
+
+static ssize_t sec_cmd_show_status(struct device *dev,
+ struct device_attribute *devattr, char *buf)
+{
+ struct sec_cmd_data *data = dev_get_drvdata(dev);
+ char buff[16] = { 0 };
+
+ if (!data) {
+ pr_err("%s %s: No platform data found\n", SECLOG, __func__);
+ return -EINVAL;
+ }
+
+ if (data->cmd_state == SEC_CMD_STATUS_WAITING)
+ snprintf(buff, sizeof(buff), "WAITING");
+
+ else if (data->cmd_state == SEC_CMD_STATUS_RUNNING)
+ snprintf(buff, sizeof(buff), "RUNNING");
+
+ else if (data->cmd_state == SEC_CMD_STATUS_OK)
+ snprintf(buff, sizeof(buff), "OK");
+
+ else if (data->cmd_state == SEC_CMD_STATUS_FAIL)
+ snprintf(buff, sizeof(buff), "FAIL");
+
+ else if (data->cmd_state == SEC_CMD_STATUS_NOT_APPLICABLE)
+ snprintf(buff, sizeof(buff), "NOT_APPLICABLE");
+
+ pr_debug("%s %s: %d, %s\n", SECLOG, __func__, data->cmd_state, buff);
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%s\n", buff);
+}
+
+static ssize_t sec_cmd_show_result(struct device *dev,
+ struct device_attribute *devattr, char *buf)
+{
+ struct sec_cmd_data *data = dev_get_drvdata(dev);
+ int size;
+
+ if (!data) {
+ pr_err("%s %s: No platform data found\n", SECLOG, __func__);
+ return -EINVAL;
+ }
+
+ data->cmd_state = SEC_CMD_STATUS_WAITING;
+ pr_info("%s %s: %s\n", SECLOG, __func__, data->cmd_result);
+ size = snprintf(buf, SEC_CMD_RESULT_STR_LEN, "%s\n", data->cmd_result);
+
+ sec_cmd_set_cmd_exit(data);
+
+ return size;
+}
+
+static ssize_t sec_cmd_list_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *data = dev_get_drvdata(dev);
+ struct sec_cmd *sec_cmd_ptr = NULL;
+ char buffer[data->cmd_buffer_size + 30];
+ char buffer_name[SEC_CMD_STR_LEN];
+
+ snprintf(buffer, 30, "++factory command list++\n");
+
+ list_for_each_entry(sec_cmd_ptr, &data->cmd_list_head, list) {
+ if (strncmp(sec_cmd_ptr->cmd_name, "not_support_cmd", 15)) {
+ snprintf(buffer_name, SEC_CMD_STR_LEN, "%s\n", sec_cmd_ptr->cmd_name);
+ strncat(buffer, buffer_name, SEC_CMD_STR_LEN);
+ }
+ }
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%s\n", buffer);
+}
+
+static DEVICE_ATTR(cmd, S_IWUSR | S_IWGRP, NULL, sec_cmd_store);
+static DEVICE_ATTR(cmd_status, S_IRUGO, sec_cmd_show_status, NULL);
+static DEVICE_ATTR(cmd_result, S_IRUGO, sec_cmd_show_result, NULL);
+static DEVICE_ATTR(cmd_list, S_IRUGO, sec_cmd_list_show, NULL);
+
+static struct attribute *sec_fac_attrs[] = {
+ &dev_attr_cmd.attr,
+ &dev_attr_cmd_status.attr,
+ &dev_attr_cmd_result.attr,
+ &dev_attr_cmd_list.attr,
+ NULL,
+};
+
+static struct attribute_group sec_fac_attr_group = {
+ .attrs = sec_fac_attrs,
+};
+
+
+int sec_cmd_init(struct sec_cmd_data *data, struct sec_cmd *cmds,
+ int len, int devt)
+{
+ const char *dev_name;
+ int ret, i;
+
+ INIT_LIST_HEAD(&data->cmd_list_head);
+
+ data->cmd_buffer_size = 0;
+ for (i = 0; i < len; i++) {
+ list_add_tail(&cmds[i].list, &data->cmd_list_head);
+ if (cmds[i].cmd_name)
+ data->cmd_buffer_size += strlen(cmds[i].cmd_name) + 1;
+ }
+
+ atomic_set(&data->cmd_is_running, 0);
+
+#ifdef USE_SEC_CMD_QUEUE
+ if (kfifo_alloc(&data->cmd_queue,
+ SEC_CMD_MAX_QUEUE * sizeof(struct command), GFP_KERNEL)) {
+ pr_err("%s %s: failed to alloc queue for cmd\n", SECLOG, __func__);
+ goto err_alloc_queue;
+ }
+ mutex_init(&data->fifo_lock);
+#endif
+
+ if (devt == SEC_CLASS_DEVT_TSP) {
+ dev_name = SEC_CLASS_DEV_NAME_TSP;
+
+ } else if (devt == SEC_CLASS_DEVT_TKEY) {
+ dev_name = SEC_CLASS_DEV_NAME_TKEY;
+
+ } else if (devt == SEC_CLASS_DEVT_WACOM) {
+ dev_name = SEC_CLASS_DEV_NAME_WACOM;
+
+ } else {
+ pr_err("%s %s: not defined devt=%d\n", SECLOG, __func__, devt);
+ goto err_get_dev_name;
+ }
+
+#ifdef CONFIG_SEC_SYSFS
+ data->fac_dev = sec_device_create(data, dev_name);
+#else
+ data->fac_dev = device_create(sec_class, NULL, devt, data, dev_name);
+#endif
+ if (IS_ERR(data->fac_dev)) {
+ pr_err("%s %s: failed to create device for the sysfs\n", SECLOG, __func__);
+ goto err_sysfs_device;
+ }
+
+ dev_set_drvdata(data->fac_dev, data);
+
+ ret = sysfs_create_group(&data->fac_dev->kobj, &sec_fac_attr_group);
+ if (ret < 0) {
+ pr_err("%s %s: failed to create sysfs group\n", SECLOG, __func__);
+ goto err_sysfs_group;
+ }
+
+ return 0;
+
+ sysfs_remove_group(&data->fac_dev->kobj, &sec_fac_attr_group);
+err_sysfs_group:
+#ifdef CONFIG_SEC_SYSFS
+ sec_device_destroy(data->fac_dev->devt);
+#else
+ device_destroy(sec_class, devt);
+#endif
+err_sysfs_device:
+err_get_dev_name:
+#ifdef USE_SEC_CMD_QUEUE
+ mutex_destroy(&data->fifo_lock);
+ kfifo_free(&data->cmd_queue);
+err_alloc_queue:
+#endif
+ list_del(&data->cmd_list_head);
+ return -ENODEV;
+}
+
+void sec_cmd_exit(struct sec_cmd_data *data, int devt)
+{
+#ifdef USE_SEC_CMD_QUEUE
+ struct command cmd = {{0}};
+ int ret;
+#endif
+
+ pr_info("%s %s", SECLOG, __func__);
+ sysfs_remove_group(&data->fac_dev->kobj, &sec_fac_attr_group);
+ dev_set_drvdata(data->fac_dev, NULL);
+#ifdef CONFIG_SEC_SYSFS
+ sec_device_destroy(data->fac_dev->devt);
+#else
+ device_destroy(sec_class, devt);
+#endif
+#ifdef USE_SEC_CMD_QUEUE
+ mutex_lock(&data->fifo_lock);
+ while (kfifo_len(&data->cmd_queue)) {
+ ret = kfifo_out(&data->cmd_queue, &cmd, sizeof(struct command));
+ if (!ret) {
+ pr_err("%s %s: kfifo_out failed, it seems empty, ret=%d\n", SECLOG, __func__, ret);
+ }
+ pr_info("%s %s: remove pending commands: %s", SECLOG, __func__, cmd.cmd);
+ }
+ mutex_unlock(&data->fifo_lock);
+ mutex_destroy(&data->fifo_lock);
+ kfifo_free(&data->cmd_queue);
+#endif
+ list_del(&data->cmd_list_head);
+}
+
+MODULE_DESCRIPTION("Samsung factory command");
+MODULE_LICENSE("GPL");
+
+
diff --git a/drivers/input/touchscreen/sec_ts/sec_ts.c b/drivers/input/touchscreen/sec_ts/sec_ts.c
new file mode 100644
index 0000000..d3f788c
--- /dev/null
+++ b/drivers/input/touchscreen/sec_ts/sec_ts.c
@@ -0,0 +1,3249 @@
+/* drivers/input/touchscreen/sec_ts.c
+ *
+ * Copyright (C) 2011 Samsung Electronics Co., Ltd.
+ * http://www.samsungsemi.com/
+ *
+ * Core file for Samsung TSC driver
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+struct sec_ts_data *tsp_info;
+
+#include "sec_ts.h"
+
+/* Switch GPIO values */
+#define SEC_SWITCH_GPIO_VALUE_SLPI_MASTER 0
+#define SEC_SWITCH_GPIO_VALUE_AP_MASTER 1
+
+struct sec_ts_data *ts_dup;
+
+#ifndef CONFIG_SEC_SYSFS
+/* Declare extern sec_class */
+struct class *sec_class;
+#endif
+
+#ifdef USE_POWER_RESET_WORK
+static void sec_ts_reset_work(struct work_struct *work);
+#endif
+static void sec_ts_read_info_work(struct work_struct *work);
+static void sec_ts_fw_update_work(struct work_struct *work);
+static void sec_ts_suspend_work(struct work_struct *work);
+static void sec_ts_resume_work(struct work_struct *work);
+
+#ifdef USE_OPEN_CLOSE
+static int sec_ts_input_open(struct input_dev *dev);
+static void sec_ts_input_close(struct input_dev *dev);
+#endif
+
+int sec_ts_read_information(struct sec_ts_data *ts);
+
+
+int sec_ts_i2c_write(struct sec_ts_data *ts, u8 reg, u8 *data, int len)
+{
+ u8 *buf;
+ int ret;
+ unsigned char retry;
+ struct i2c_msg msg;
+
+ if (len + 1 > sizeof(ts->i2c_write_buf)) {
+ input_err(true, &ts->client->dev, "%s: len is larger than buffer size\n", __func__);
+ return -EINVAL;
+ }
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: POWER_STATUS : OFF\n", __func__);
+ goto err;
+ }
+
+ mutex_lock(&ts->i2c_mutex);
+
+ buf = ts->i2c_write_buf;
+ buf[0] = reg;
+ memcpy(buf + 1, data, len);
+
+ msg.addr = ts->client->addr;
+ msg.flags = 0;
+ msg.len = len + 1;
+ msg.buf = buf;
+
+ for (retry = 0; retry < SEC_TS_I2C_RETRY_CNT; retry++) {
+ if ((ret = i2c_transfer(ts->client->adapter, &msg, 1)) == 1)
+ break;
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: POWER_STATUS : OFF, retry:%d\n", __func__, retry);
+ mutex_unlock(&ts->i2c_mutex);
+ goto err;
+ }
+
+ usleep_range(1 * 1000, 1 * 1000);
+
+ if (retry > 1) {
+ input_err(true, &ts->client->dev, "%s: I2C retry %d\n", __func__, retry + 1);
+ ts->comm_err_count++;
+ }
+ }
+
+ mutex_unlock(&ts->i2c_mutex);
+
+ if (retry == SEC_TS_I2C_RETRY_CNT) {
+ input_err(true, &ts->client->dev, "%s: I2C write over retry limit\n", __func__);
+ ret = -EIO;
+#ifdef USE_POR_AFTER_I2C_RETRY
+ if (ts->probe_done && !ts->reset_is_on_going)
+ schedule_delayed_work(&ts->reset_work, msecs_to_jiffies(TOUCH_RESET_DWORK_TIME));
+#endif
+ }
+
+ if (ret == 1)
+ return 0;
+err:
+ return -EIO;
+}
+
+static int sec_ts_i2c_read_internal(struct sec_ts_data *ts, u8 reg,
+ u8 *data, int len, bool dma_safe)
+{
+ u8 *buf;
+ int ret;
+ unsigned char retry;
+ struct i2c_msg msg[2];
+ int remain = len;
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: POWER_STATUS : OFF\n", __func__);
+ goto err;
+ }
+
+ if (len > sizeof(ts->i2c_read_buf) && dma_safe == false) {
+ input_err(true, &ts->client->dev, "%s: len %d over pre-allocated size %d\n",
+ __func__, len, I2C_PREALLOC_READ_BUF_SZ);
+ return -ENOSPC;
+ }
+
+ mutex_lock(&ts->i2c_mutex);
+
+ buf = ts->i2c_write_buf;
+ buf[0] = reg;
+
+ msg[0].addr = ts->client->addr;
+ msg[0].flags = 0;
+ msg[0].len = 1;
+ msg[0].buf = buf;
+
+ msg[1].addr = ts->client->addr;
+ msg[1].flags = I2C_M_RD;
+ msg[1].len = len;
+ if (dma_safe == false)
+ msg[1].buf = ts->i2c_read_buf;
+ else
+ msg[1].buf = data;
+
+ if (len <= ts->i2c_burstmax) {
+
+ for (retry = 0; retry < SEC_TS_I2C_RETRY_CNT; retry++) {
+ ret = i2c_transfer(ts->client->adapter, msg, 2);
+ if (ret == 2)
+ break;
+ usleep_range(1 * 1000, 1 * 1000);
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: POWER_STATUS : OFF, retry:%d\n", __func__, retry);
+ mutex_unlock(&ts->i2c_mutex);
+ goto err;
+ }
+
+ if (retry > 1) {
+ input_err(true, &ts->client->dev, "%s: I2C retry %d\n", __func__, retry + 1);
+ ts->comm_err_count++;
+ }
+ }
+
+ if (ret == 2 && dma_safe == false)
+ memcpy(data, ts->i2c_read_buf, len);
+
+ } else {
+ /*
+ * I2C read buffer is 256 byte. do not support long buffer over than 256.
+ * So, try to seperate reading data about 256 bytes.
+ */
+
+ for (retry = 0; retry < SEC_TS_I2C_RETRY_CNT; retry++) {
+ ret = i2c_transfer(ts->client->adapter, msg, 1);
+ if (ret == 1)
+ break;
+ usleep_range(1 * 1000, 1 * 1000);
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: POWER_STATUS : OFF, retry:%d\n", __func__, retry);
+ mutex_unlock(&ts->i2c_mutex);
+ goto err;
+ }
+
+ if (retry > 1) {
+ input_err(true, &ts->client->dev, "%s: I2C retry %d\n", __func__, retry + 1);
+ ts->comm_err_count++;
+ }
+ }
+
+ do {
+ if (remain > ts->i2c_burstmax)
+ msg[1].len = ts->i2c_burstmax;
+ else
+ msg[1].len = remain;
+
+ remain -= ts->i2c_burstmax;
+
+ for (retry = 0; retry < SEC_TS_I2C_RETRY_CNT; retry++) {
+ ret = i2c_transfer(ts->client->adapter, &msg[1], 1);
+ if (ret == 1)
+ break;
+ usleep_range(1 * 1000, 1 * 1000);
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: POWER_STATUS : OFF, retry:%d\n", __func__, retry);
+ mutex_unlock(&ts->i2c_mutex);
+ goto err;
+ }
+
+ if (retry > 1) {
+ input_err(true, &ts->client->dev, "%s: I2C retry %d\n", __func__, retry + 1);
+ ts->comm_err_count++;
+ }
+ }
+
+ msg[1].buf += msg[1].len;
+
+ } while (remain > 0);
+
+ if (ret == 1 && dma_safe == false)
+ memcpy(data, ts->i2c_read_buf, len);
+ }
+
+ mutex_unlock(&ts->i2c_mutex);
+
+ if (retry == SEC_TS_I2C_RETRY_CNT) {
+ input_err(true, &ts->client->dev, "%s: I2C read over retry limit\n", __func__);
+ ret = -EIO;
+#ifdef USE_POR_AFTER_I2C_RETRY
+ if (ts->probe_done && !ts->reset_is_on_going)
+ schedule_delayed_work(&ts->reset_work, msecs_to_jiffies(TOUCH_RESET_DWORK_TIME));
+#endif
+
+ }
+
+ return ret;
+
+err:
+ return -EIO;
+}
+
+static int sec_ts_i2c_write_burst_internal(struct sec_ts_data *ts,
+ u8 *data, int len, bool dma_safe)
+{
+ int ret;
+ int retry;
+
+ if (len > sizeof(ts->i2c_write_buf) && dma_safe == false) {
+ input_err(true, &ts->client->dev, "%s: len %d over pre-allocated size %d\n",
+ __func__, len, sizeof(ts->i2c_write_buf));
+ return -ENOSPC;
+ }
+
+ mutex_lock(&ts->i2c_mutex);
+
+ if (dma_safe == false) {
+ memcpy(ts->i2c_write_buf, data, len);
+ data = ts->i2c_write_buf;
+ }
+
+ for (retry = 0; retry < SEC_TS_I2C_RETRY_CNT; retry++) {
+ if ((ret = i2c_master_send(ts->client, data, len)) == len)
+ break;
+
+ usleep_range(1 * 1000, 1 * 1000);
+
+ if (retry > 1) {
+ input_err(true, &ts->client->dev, "%s: I2C retry %d\n", __func__, retry + 1);
+ ts->comm_err_count++;
+ }
+ }
+
+ mutex_unlock(&ts->i2c_mutex);
+ if (retry == SEC_TS_I2C_RETRY_CNT) {
+ input_err(true, &ts->client->dev, "%s: I2C write over retry limit\n", __func__);
+ ret = -EIO;
+ }
+
+ return ret;
+}
+
+static int sec_ts_i2c_read_bulk_internal(struct sec_ts_data *ts,
+ u8 *data, int len, bool dma_safe)
+{
+ int ret;
+ unsigned char retry;
+ int remain = len;
+ struct i2c_msg msg;
+
+ if (len > sizeof(ts->i2c_read_buf) && dma_safe == false) {
+ input_err(true, &ts->client->dev,
+ "%s: len %d over pre-allocated size %d\n", __func__,
+ len, sizeof(ts->i2c_read_buf));
+ return -ENOSPC;
+ }
+
+ mutex_lock(&ts->i2c_mutex);
+
+ msg.addr = ts->client->addr;
+ msg.flags = I2C_M_RD;
+ msg.len = len;
+ if (dma_safe == false)
+ msg.buf = ts->i2c_read_buf;
+ else
+ msg.buf = data;
+
+ do {
+ if (remain > ts->i2c_burstmax)
+ msg.len = ts->i2c_burstmax;
+ else
+ msg.len = remain;
+
+ remain -= ts->i2c_burstmax;
+
+ for (retry = 0; retry < SEC_TS_I2C_RETRY_CNT; retry++) {
+ ret = i2c_transfer(ts->client->adapter, &msg, 1);
+ if (ret == 1)
+ break;
+ usleep_range(1 * 1000, 1 * 1000);
+
+ if (retry > 1) {
+ input_err(true, &ts->client->dev, "%s: I2C retry %d\n", __func__, retry + 1);
+ ts->comm_err_count++;
+ }
+ }
+
+ if (retry == SEC_TS_I2C_RETRY_CNT) {
+ input_err(true, &ts->client->dev,
+ "%s: I2C read over retry limit\n", __func__);
+ ret = -EIO;
+ break;
+ }
+
+ msg.buf += msg.len;
+
+ } while (remain > 0);
+
+ if (ret == 1 && dma_safe == false)
+ memcpy(data, ts->i2c_read_buf, len);
+
+ mutex_unlock(&ts->i2c_mutex);
+
+ if (ret == 1)
+ return 0;
+
+ return -EIO;
+}
+
+/* Wrapper API for i2c read and write */
+int sec_ts_i2c_read(struct sec_ts_data *ts, u8 reg, u8 *data, int len)
+{
+ return sec_ts_i2c_read_internal(ts, reg, data, len, false);
+}
+
+int sec_ts_i2c_read_heap(struct sec_ts_data *ts, u8 reg, u8 *data, int len)
+{
+ return sec_ts_i2c_read_internal(ts, reg, data, len, true);
+}
+
+int sec_ts_i2c_write_burst(struct sec_ts_data *ts, u8 *data, int len)
+{
+ return sec_ts_i2c_write_burst_internal(ts, data, len, false);
+}
+
+int sec_ts_i2c_write_burst_heap(struct sec_ts_data *ts, u8 *data, int len)
+{
+ return sec_ts_i2c_write_burst_internal(ts, data, len, true);
+}
+
+int sec_ts_i2c_read_bulk(struct sec_ts_data *ts, u8 *data, int len)
+{
+ return sec_ts_i2c_read_bulk_internal(ts, data, len, false);
+}
+
+int sec_ts_i2c_read_bulk_heap(struct sec_ts_data *ts, u8 *data, int len)
+{
+ return sec_ts_i2c_read_bulk_internal(ts, data, len, true);
+}
+
+static int sec_ts_read_from_customlib(struct sec_ts_data *ts, u8 *data, int len)
+{
+ int ret;
+
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_READ_PARAM, data, 2);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to read custom library command\n", __func__);
+
+ ret = sec_ts_i2c_read(ts, SEC_TS_CMD_CUSTOMLIB_READ_PARAM, (u8 *)data, len);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to read custom library command\n", __func__);
+
+ return ret;
+}
+
+#if defined(CONFIG_TOUCHSCREEN_DUMP_MODE)
+#include <linux/sec_debug.h>
+extern struct tsp_dump_callbacks dump_callbacks;
+static struct delayed_work *p_ghost_check;
+
+static void sec_ts_check_rawdata(struct work_struct *work)
+{
+ struct sec_ts_data *ts = container_of(work, struct sec_ts_data, ghost_check.work);
+
+ if (ts->tsp_dump_lock == 1) {
+ input_err(true, &ts->client->dev, "%s: ignored ## already checking..\n", __func__);
+ return;
+ }
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: ignored ## IC is power off\n", __func__);
+ return;
+ }
+
+ ts->tsp_dump_lock = 1;
+ input_info(true, &ts->client->dev, "%s: start ##\n", __func__);
+ sec_ts_run_rawdata_all((void *)ts, false);
+ msleep(100);
+
+ input_info(true, &ts->client->dev, "%s: done ##\n", __func__);
+ ts->tsp_dump_lock = 0;
+
+}
+
+static void dump_tsp_log(void)
+{
+ pr_info("%s: %s %s: start\n", SEC_TS_I2C_NAME, SECLOG, __func__);
+
+#ifdef CONFIG_BATTERY_SAMSUNG
+ if (lpcharge == 1) {
+ pr_err("%s: %s %s: ignored ## lpm charging Mode!!\n", SEC_TS_I2C_NAME, SECLOG, __func__);
+ return;
+ }
+#endif
+
+ if (p_ghost_check == NULL) {
+ pr_err("%s: %s %s: ignored ## tsp probe fail!!\n", SEC_TS_I2C_NAME, SECLOG, __func__);
+ return;
+ }
+ schedule_delayed_work(p_ghost_check, msecs_to_jiffies(100));
+}
+#endif
+
+
+void sec_ts_delay(unsigned int ms)
+{
+ if (ms < 20)
+ usleep_range(ms * 1000, ms * 1000);
+ else
+ msleep(ms);
+}
+
+int sec_ts_wait_for_ready(struct sec_ts_data *ts, unsigned int ack)
+{
+ return sec_ts_wait_for_ready_with_count(ts, ack,
+ SEC_TS_WAIT_RETRY_CNT);
+}
+
+int sec_ts_wait_for_ready_with_count(struct sec_ts_data *ts, unsigned int ack,
+ unsigned int count)
+{
+ int rc = -1;
+ int retry = 0;
+ u8 tBuff[SEC_TS_EVENT_BUFF_SIZE] = {0,};
+
+ while (sec_ts_i2c_read(ts, SEC_TS_READ_ONE_EVENT, tBuff,
+ SEC_TS_EVENT_BUFF_SIZE)) {
+ if (((tBuff[0] >> 2) & 0xF) == TYPE_STATUS_EVENT_INFO) {
+ if (tBuff[1] == ack) {
+ rc = 0;
+ break;
+ }
+ } else if (((tBuff[0] >> 2) & 0xF)
+ == TYPE_STATUS_EVENT_VENDOR_INFO) {
+ if (tBuff[1] == ack) {
+ rc = 0;
+ break;
+ }
+ }
+
+ if (retry++ > count) {
+ input_err(true, &ts->client->dev, "%s: Time Over\n",
+ __func__);
+ break;
+ }
+ sec_ts_delay(20);
+ }
+
+ input_info(true, &ts->client->dev,
+ "%s: %02X, %02X, %02X, %02X, %02X, %02X, %02X, %02X [%d]\n",
+ __func__, tBuff[0], tBuff[1], tBuff[2], tBuff[3],
+ tBuff[4], tBuff[5], tBuff[6], tBuff[7], retry);
+
+ return rc;
+}
+
+int sec_ts_read_calibration_report(struct sec_ts_data *ts)
+{
+ int ret;
+ u8 buf[5] = { 0 };
+
+ buf[0] = SEC_TS_READ_CALIBRATION_REPORT;
+
+ ret = sec_ts_i2c_read(ts, buf[0], &buf[1], 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: failed to read, %d\n", __func__, ret);
+ return ret;
+ }
+
+ input_info(true, &ts->client->dev, "%s: count:%d, pass count:%d, fail count:%d, status:0x%X\n",
+ __func__, buf[1], buf[2], buf[3], buf[4]);
+
+ return buf[4];
+}
+
+static void sec_ts_reinit(struct sec_ts_data *ts)
+{
+ u8 w_data[2] = {0x00, 0x00};
+ int ret = 0;
+
+ input_info(true, &ts->client->dev,
+ "%s : charger=0x%x, Cover=0x%x, Power mode=0x%x\n",
+ __func__, ts->charger_mode, ts->touch_functions, ts->lowpower_status);
+
+ /* charger mode */
+ if (ts->charger_mode != SEC_TS_BIT_CHARGER_MODE_NO) {
+ w_data[0] = ts->charger_mode;
+ ret = ts->sec_ts_i2c_write(ts, SET_TS_CMD_SET_CHARGER_MODE, (u8 *)&w_data[0], 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to send command(0x%x)",
+ __func__, SET_TS_CMD_SET_CHARGER_MODE);
+ }
+
+ /* Cover mode */
+ if (ts->touch_functions & SEC_TS_BIT_SETFUNC_COVER) {
+ w_data[0] = ts->cover_cmd;
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_COVERTYPE, (u8 *)&w_data[0], 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to send command(0x%x)",
+ __func__, SEC_TS_CMD_SET_COVERTYPE);
+
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCHFUNCTION, (u8 *)&(ts->touch_functions), 2);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to send command(0x%x)",
+ __func__, SEC_TS_CMD_SET_TOUCHFUNCTION);
+ }
+
+ #ifdef SEC_TS_SUPPORT_CUSTOMLIB
+ if (ts->use_customlib)
+ sec_ts_set_custom_library(ts);
+ #endif
+
+ /* Power mode */
+ if (ts->lowpower_status == TO_LOWPOWER_MODE) {
+ w_data[0] = (ts->lowpower_mode & SEC_TS_MODE_LOWPOWER_FLAG) >> 1;
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_WAKEUP_GESTURE_MODE, (u8 *)&w_data[0], 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to send command(0x%x)",
+ __func__, SEC_TS_CMD_WAKEUP_GESTURE_MODE);
+
+ w_data[0] = TO_LOWPOWER_MODE;
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_POWER_MODE, (u8 *)&w_data[0], 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to send command(0x%x)",
+ __func__, SEC_TS_CMD_SET_POWER_MODE);
+
+ sec_ts_delay(50);
+
+ if (ts->lowpower_mode & SEC_TS_MODE_CUSTOMLIB_AOD) {
+ int i, ret;
+ u8 data[10] = {0x02, 0};
+
+ for (i = 0; i < 4; i++) {
+ data[i * 2 + 2] = ts->rect_data[i] & 0xFF;
+ data[i * 2 + 3] = (ts->rect_data[i] >> 8) & 0xFF;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_WRITE_PARAM, &data[0], 10);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to write offset\n", __func__);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_NOTIFY_PACKET, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to send notify\n", __func__);
+
+ }
+
+ } else {
+
+ sec_ts_set_grip_type(ts, ONLY_EDGE_HANDLER);
+
+ if (ts->dex_mode) {
+ input_info(true, &ts->client->dev, "%s: set dex mode\n", __func__);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_DEX_MODE, &ts->dex_mode, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to set dex mode %x\n", __func__, ts->dex_mode);
+ }
+
+ if (ts->brush_mode) {
+ input_info(true, &ts->client->dev, "%s: set brush mode\n", __func__);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_BRUSH_MODE, &ts->brush_mode, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to set brush mode\n", __func__);
+ }
+
+ if (ts->touchable_area) {
+ input_info(true, &ts->client->dev, "%s: set 16:9 mode\n", __func__);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCHABLE_AREA, &ts->touchable_area, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to set 16:9 mode\n", __func__);
+ }
+
+ }
+ return;
+}
+
+static bool read_heatmap_raw(struct v4l2_heatmap *v4l2, strength_t *data)
+{
+ struct sec_ts_data *ts = container_of(v4l2, struct sec_ts_data, v4l2);
+
+ unsigned int num_elements;
+ /* index for looping through the heatmap buffer read over the bus */
+ unsigned int local_i;
+
+ int result;
+
+ strength_t heatmap_value;
+ /* final position of the heatmap value in the full heatmap frame */
+ unsigned int frame_i;
+ int heatmap_x, heatmap_y;
+ int max_x = v4l2->format.width;
+ int max_y = v4l2->format.height;
+
+ struct heatmap_report report = {0};
+
+ result = sec_ts_i2c_read(ts, SEC_TS_CMD_HEATMAP_READ,
+ (uint8_t *) &report, sizeof(report));
+ if (result < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: i2c read failed, sec_ts_i2c_read returned %i\n",
+ __func__, result);
+ return false;
+ }
+
+ num_elements = report.size_x * report.size_y;
+ if (num_elements > LOCAL_HEATMAP_WIDTH * LOCAL_HEATMAP_HEIGHT) {
+ input_err(true, &ts->client->dev,
+ "Unexpected heatmap size: %i x %i",
+ report.size_x, report.size_y);
+ return false;
+ }
+
+ /* set all to zero, will only write to non-zero locations in the loop */
+ memset(data, 0, max_x * max_y * sizeof(data[0]));
+ /* populate the data buffer, rearranging into final locations */
+ for (local_i = 0; local_i < num_elements; local_i++) {
+ /* enforce big-endian order */
+ be16_to_cpus(&report.data[local_i]);
+ heatmap_value = report.data[local_i];
+
+ if (heatmap_value == 0) {
+ /*
+ * Already initialized to zero. More importantly,
+ * samples around edges may go out of bounds.
+ * If their value is zero, this is ok.
+ */
+ continue;
+ }
+ heatmap_x = report.offset_x + (local_i % report.size_x);
+ heatmap_y = report.offset_y + (local_i / report.size_x);
+
+ if (heatmap_x < 0 || heatmap_x >= max_x ||
+ heatmap_y < 0 || heatmap_y >= max_y) {
+ input_err(true, &ts->client->dev,
+ "Invalid x or y: (%i, %i), value=%i, ending loop\n",
+ heatmap_x, heatmap_y, heatmap_value);
+ return false;
+ }
+ frame_i = heatmap_y * max_x + heatmap_x;
+ data[frame_i] = heatmap_value;
+ };
+ return true;
+}
+
+#define MAX_EVENT_COUNT 32
+static void sec_ts_read_event(struct sec_ts_data *ts)
+{
+ int ret;
+ u8 t_id;
+ u8 event_id;
+ u8 left_event_count;
+ u8 read_event_buff[MAX_EVENT_COUNT][SEC_TS_EVENT_BUFF_SIZE] = { { 0 } };
+ u8 *event_buff;
+ struct sec_ts_event_coordinate *p_event_coord;
+ struct sec_ts_gesture_status *p_gesture_status;
+ struct sec_ts_event_status *p_event_status;
+ int curr_pos;
+ int remain_event_count = 0;
+
+ if (ts->power_status == SEC_TS_STATE_LPM) {
+
+ pm_wakeup_event(&ts->client->dev, 3 * MSEC_PER_SEC);
+ /* waiting for blsp block resuming, if not occurs i2c error */
+ ret = wait_for_completion_interruptible_timeout(&ts->resume_done, msecs_to_jiffies(3 * MSEC_PER_SEC));
+ if (ret == 0) {
+ input_err(true, &ts->client->dev, "%s: LPM: pm resume is not handled\n", __func__);
+ return;
+ }
+
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: LPM: -ERESTARTSYS if interrupted, %d\n", __func__, ret);
+ return;
+ }
+
+ input_info(true, &ts->client->dev, "%s: run LPM interrupt handler, %d\n", __func__, ret);
+ /* run lpm interrupt handler */
+ }
+
+ ret = t_id = event_id = curr_pos = remain_event_count = 0;
+ /* repeat READ_ONE_EVENT until buffer is empty(No event) */
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_ONE_EVENT, (u8 *)read_event_buff[0], SEC_TS_EVENT_BUFF_SIZE);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: i2c read one event failed\n", __func__);
+ return;
+ }
+
+ if (ts->temp == 0x01)
+ input_info(true, &ts->client->dev, "ONE: %02X %02X %02X %02X %02X %02X %02X %02X\n",
+ read_event_buff[0][0], read_event_buff[0][1], read_event_buff[0][2], read_event_buff[0][3],
+ read_event_buff[0][4], read_event_buff[0][5], read_event_buff[0][6], read_event_buff[0][7]);
+
+ if (read_event_buff[0][0] == 0) {
+ input_info(true, &ts->client->dev, "%s: event buffer is empty\n", __func__);
+ return;
+ }
+
+ left_event_count = read_event_buff[0][7] & 0x3F;
+ remain_event_count = left_event_count;
+
+ if (left_event_count > MAX_EVENT_COUNT - 1 || left_event_count == 0xFF) {
+ input_err(true, &ts->client->dev, "%s: event buffer overflow\n", __func__);
+
+ /* write clear event stack command when read_event_count > MAX_EVENT_COUNT */
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_CLEAR_EVENT_STACK, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: i2c write clear event failed\n", __func__);
+ return;
+ }
+
+ if (left_event_count > 0) {
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_ALL_EVENT, (u8 *)read_event_buff[1],
+ sizeof(u8) * (SEC_TS_EVENT_BUFF_SIZE) * (left_event_count));
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: i2c read one event failed\n", __func__);
+ return;
+ }
+ }
+
+ do {
+ s16 max_force_p = 0;
+ event_buff = read_event_buff[curr_pos];
+ event_id = event_buff[0] & 0x3;
+
+ if (ts->temp == 0x01)
+ input_info(true, &ts->client->dev, "ALL: %02X %02X %02X %02X %02X %02X %02X %02X\n",
+ event_buff[0], event_buff[1], event_buff[2], event_buff[3],
+ event_buff[4], event_buff[5], event_buff[6], event_buff[7]);
+
+ switch (event_id) {
+ case SEC_TS_STATUS_EVENT:
+ p_event_status = (struct sec_ts_event_status *)event_buff;
+
+ /* tchsta == 0 && ttype == 0 && eid == 0 : buffer empty */
+ if (p_event_status->stype > 0) {
+ /* Demote 'vendor' messages */
+ if (p_event_status->stype ==
+ TYPE_STATUS_EVENT_VENDOR_INFO)
+ input_dbg(true, &ts->client->dev,
+ "%s: STATUS %x %x %x %x %x %x %x %x\n",
+ __func__, event_buff[0],
+ event_buff[1], event_buff[2],
+ event_buff[3], event_buff[4],
+ event_buff[5], event_buff[6],
+ event_buff[7]);
+ else
+ input_info(true, &ts->client->dev,
+ "%s: STATUS %x %x %x %x %x %x %x %x\n",
+ __func__, event_buff[0],
+ event_buff[1], event_buff[2],
+ event_buff[3], event_buff[4],
+ event_buff[5], event_buff[6],
+ event_buff[7]);
+ }
+
+ /* watchdog reset -> send SENSEON command */ /*=>?????*/
+ if ((p_event_status->stype == TYPE_STATUS_EVENT_INFO) &&
+ (p_event_status->status_id == SEC_TS_ACK_BOOT_COMPLETE) &&
+ (p_event_status->status_data_1 == 0x20)) {
+
+ sec_ts_unlocked_release_all_finger(ts);
+
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_ON, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to write Sense_on\n", __func__);
+
+ sec_ts_reinit(ts);
+ }
+
+ /* event queue full-> all finger release */
+ if ((p_event_status->stype == TYPE_STATUS_EVENT_ERR) &&
+ (p_event_status->status_id == SEC_TS_ERR_EVENT_QUEUE_FULL)) {
+ input_err(true, &ts->client->dev, "%s: IC Event Queue is full\n", __func__);
+ sec_ts_unlocked_release_all_finger(ts);
+ }
+
+ if ((p_event_status->stype == TYPE_STATUS_EVENT_ERR) &&
+ (p_event_status->status_id == SEC_TS_ERR_EVENT_ESD)) {
+ input_err(true, &ts->client->dev, "%s: ESD detected. run reset\n", __func__);
+#ifdef USE_RESET_DURING_POWER_ON
+ schedule_work(&ts->reset_work.work);
+#endif
+ }
+
+ if ((p_event_status->stype == TYPE_STATUS_EVENT_INFO) &&
+ (p_event_status->status_id == SEC_TS_ACK_WET_MODE)) {
+ ts->wet_mode = p_event_status->status_data_1;
+ input_info(true, &ts->client->dev, "%s: water wet mode %d\n",
+ __func__, ts->wet_mode);
+ if (ts->wet_mode)
+ ts->wet_count++;
+
+ }
+
+ if ((p_event_status->stype == TYPE_STATUS_EVENT_CUSTOMLIB_INFO) &&
+ (p_event_status->status_id == SEC_TS_EVENT_CUSTOMLIB_FORCE_KEY)) {
+ if (ts->power_status == SEC_TS_STATE_POWER_ON) {
+ if (p_event_status->status_data_1 & SEC_TS_CUSTOMLIB_EVENT_PRESSURE_TOUCHED) {
+ ts->all_force_count++;
+ ts->scrub_id = CUSTOMLIB_EVENT_TYPE_PRESSURE_TOUCHED;
+ } else {
+ if (ts->scrub_id == CUSTOMLIB_EVENT_TYPE_AOD_HOMEKEY_PRESS) {
+ input_report_key(ts->input_dev, KEY_HOMEPAGE, 0);
+ ts->scrub_id = CUSTOMLIB_EVENT_TYPE_AOD_HOMEKEY_RELEASE;
+ } else {
+ ts->scrub_id = CUSTOMLIB_EVENT_TYPE_PRESSURE_RELEASED;
+ }
+ }
+
+ input_report_key(ts->input_dev, KEY_BLACK_UI_GESTURE, 1);
+ } else {
+ if (p_event_status->status_data_1 & SEC_TS_CUSTOMLIB_EVENT_PRESSURE_RELEASED) {
+ input_report_key(ts->input_dev, KEY_HOMEPAGE, 0);
+ input_report_key(ts->input_dev, KEY_BLACK_UI_GESTURE, 1);
+ ts->scrub_id = CUSTOMLIB_EVENT_TYPE_AOD_HOMEKEY_RELEASE_NO_HAPTIC;
+ input_sync(ts->input_dev);
+
+ haptic_homekey_release();
+ } else {
+ input_report_key(ts->input_dev, KEY_HOMEPAGE, 1);
+ input_sync(ts->input_dev);
+
+ ts->scrub_id = CUSTOMLIB_EVENT_TYPE_AOD_HOMEKEY_PRESS;
+ haptic_homekey_press();
+ ts->all_force_count++;
+ }
+ }
+
+ ts->scrub_x = ((p_event_status->status_data_4 >> 4) & 0xF) << 8 | (p_event_status->status_data_3 & 0xFF);
+ ts->scrub_y = ((p_event_status->status_data_4 >> 0) & 0xF) << 8 | (p_event_status->status_data_2 & 0xFF);
+
+ input_info(true, &ts->client->dev, "%s: PRESSURE[%d]\n", __func__, ts->scrub_id);
+
+ input_sync(ts->input_dev);
+ input_report_key(ts->input_dev, KEY_BLACK_UI_GESTURE, 0);
+ }
+
+ break;
+
+ case SEC_TS_COORDINATE_EVENT:
+ if (ts->input_closed) {
+ input_err(true, &ts->client->dev, "%s: device is closed\n", __func__);
+ break;
+ }
+ p_event_coord = (struct sec_ts_event_coordinate *)event_buff;
+
+ t_id = (p_event_coord->tid - 1);
+
+ if (t_id < MAX_SUPPORT_TOUCH_COUNT + MAX_SUPPORT_HOVER_COUNT) {
+ ts->coord[t_id].id = t_id;
+ ts->coord[t_id].action = p_event_coord->tchsta;
+ ts->coord[t_id].x = (p_event_coord->x_11_4 << 4) | (p_event_coord->x_3_0);
+ ts->coord[t_id].y = (p_event_coord->y_11_4 << 4) | (p_event_coord->y_3_0);
+ ts->coord[t_id].z = p_event_coord->z &
+ SEC_TS_PRESSURE_MAX;
+ ts->coord[t_id].ttype = p_event_coord->ttype_3_2 << 2 | p_event_coord->ttype_1_0 << 0;
+ ts->coord[t_id].major = p_event_coord->major;
+ ts->coord[t_id].minor = p_event_coord->minor;
+
+ if (!ts->coord[t_id].palm && (ts->coord[t_id].ttype == SEC_TS_TOUCHTYPE_PALM))
+ ts->coord[t_id].palm_count++;
+
+ ts->coord[t_id].palm = (ts->coord[t_id].ttype == SEC_TS_TOUCHTYPE_PALM);
+ ts->coord[t_id].left_event = p_event_coord->left_event;
+
+ if (ts->coord[t_id].z <= 0)
+ ts->coord[t_id].z = 1;
+
+ if ((ts->coord[t_id].ttype ==
+ SEC_TS_TOUCHTYPE_NORMAL) ||
+ (ts->coord[t_id].ttype ==
+ SEC_TS_TOUCHTYPE_PALM) ||
+ (ts->coord[t_id].ttype ==
+ SEC_TS_TOUCHTYPE_WET) ||
+ (ts->coord[t_id].ttype ==
+ SEC_TS_TOUCHTYPE_GLOVE)) {
+
+ if (ts->coord[t_id].action == SEC_TS_COORDINATE_ACTION_RELEASE) {
+ u8 rbuf[2] = {0, };
+
+ do_gettimeofday(&ts->time_released[t_id]);
+
+ if (ts->time_longest < (ts->time_released[t_id].tv_sec - ts->time_pressed[t_id].tv_sec))
+ ts->time_longest = (ts->time_released[t_id].tv_sec - ts->time_pressed[t_id].tv_sec);
+
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_FORCE_SIG_MAX_VAL, rbuf, 2);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: fail to read max_pressure data\n",
+ __func__);
+ else
+ max_force_p = (rbuf[0] & 0xFF) << 8 | (rbuf[1] & 0xFF);
+
+ input_mt_slot(ts->input_dev, t_id);
+ if (ts->plat_data->support_mt_pressure)
+ input_report_abs(ts->input_dev, ABS_MT_PRESSURE, 0);
+ input_mt_report_slot_state(ts->input_dev, MT_TOOL_FINGER, 0);
+
+ if (ts->touch_count > 0)
+ ts->touch_count--;
+ if (ts->touch_count == 0) {
+ input_report_key(ts->input_dev, BTN_TOUCH, 0);
+ input_report_key(ts->input_dev, BTN_TOOL_FINGER, 0);
+ ts->check_multi = 0;
+ }
+
+ } else if (ts->coord[t_id].action == SEC_TS_COORDINATE_ACTION_PRESS) {
+ do_gettimeofday(&ts->time_pressed[t_id]);
+
+ ts->touch_count++;
+ ts->all_finger_count++;
+
+ ts->max_z_value = max((unsigned int)ts->coord[t_id].z, ts->max_z_value);
+ ts->min_z_value = min((unsigned int)ts->coord[t_id].z, ts->min_z_value);
+ ts->sum_z_value += (unsigned int)ts->coord[t_id].z;
+
+ input_mt_slot(ts->input_dev, t_id);
+ input_mt_report_slot_state(ts->input_dev, MT_TOOL_FINGER, 1);
+ input_report_key(ts->input_dev, BTN_TOUCH, 1);
+ input_report_key(ts->input_dev, BTN_TOOL_FINGER, 1);
+
+ input_report_abs(ts->input_dev, ABS_MT_POSITION_X, ts->coord[t_id].x);
+ input_report_abs(ts->input_dev, ABS_MT_POSITION_Y, ts->coord[t_id].y);
+ input_report_abs(ts->input_dev, ABS_MT_TOUCH_MAJOR, ts->coord[t_id].major);
+ input_report_abs(ts->input_dev, ABS_MT_TOUCH_MINOR, ts->coord[t_id].minor);
+ if (ts->brush_mode)
+ input_report_abs(ts->input_dev, ABS_MT_CUSTOM, (ts->coord[t_id].z << 1) | ts->coord[t_id].palm);
+ else
+ input_report_abs(ts->input_dev, ABS_MT_CUSTOM, (BRUSH_Z_DATA << 1) | ts->coord[t_id].palm);
+ if (ts->plat_data->support_mt_pressure)
+ input_report_abs(ts->input_dev, ABS_MT_PRESSURE, ts->coord[t_id].z);
+
+ if ((ts->touch_count > 4) && (ts->check_multi == 0)) {
+ ts->check_multi = 1;
+ ts->multi_count++;
+ }
+
+ } else if (ts->coord[t_id].action == SEC_TS_COORDINATE_ACTION_MOVE) {
+ if ((ts->coord[t_id].ttype == SEC_TS_TOUCHTYPE_GLOVE) && !ts->touchkey_glove_mode_status) {
+ ts->touchkey_glove_mode_status = true;
+ input_report_switch(ts->input_dev, SW_GLOVE, 1);
+ } else if ((ts->coord[t_id].ttype != SEC_TS_TOUCHTYPE_GLOVE) && ts->touchkey_glove_mode_status) {
+ ts->touchkey_glove_mode_status = false;
+ input_report_switch(ts->input_dev, SW_GLOVE, 0);
+ }
+
+ input_mt_slot(ts->input_dev, t_id);
+ input_mt_report_slot_state(ts->input_dev, MT_TOOL_FINGER, 1);
+ input_report_key(ts->input_dev, BTN_TOUCH, 1);
+ input_report_key(ts->input_dev, BTN_TOOL_FINGER, 1);
+
+ input_report_abs(ts->input_dev, ABS_MT_POSITION_X, ts->coord[t_id].x);
+ input_report_abs(ts->input_dev, ABS_MT_POSITION_Y, ts->coord[t_id].y);
+ input_report_abs(ts->input_dev, ABS_MT_TOUCH_MAJOR, ts->coord[t_id].major);
+ input_report_abs(ts->input_dev, ABS_MT_TOUCH_MINOR, ts->coord[t_id].minor);
+ if (ts->brush_mode)
+ input_report_abs(ts->input_dev, ABS_MT_CUSTOM, (ts->coord[t_id].z << 1) | ts->coord[t_id].palm);
+ else
+ input_report_abs(ts->input_dev, ABS_MT_CUSTOM, (BRUSH_Z_DATA << 1) | ts->coord[t_id].palm);
+
+ if (ts->plat_data->support_mt_pressure)
+ input_report_abs(ts->input_dev, ABS_MT_PRESSURE, ts->coord[t_id].z);
+ ts->coord[t_id].mcount++;
+ } else {
+ input_dbg(true, &ts->client->dev,
+ "%s: do not support coordinate action(%d)\n", __func__, ts->coord[t_id].action);
+ }
+ } else {
+ input_dbg(true, &ts->client->dev,
+ "%s: do not support coordinate type(%d)\n", __func__, ts->coord[t_id].ttype);
+ }
+ } else {
+ input_err(true, &ts->client->dev, "%s: tid(%d) is out of range\n", __func__, t_id);
+ }
+ break;
+
+ case SEC_TS_GESTURE_EVENT:
+ p_gesture_status = (struct sec_ts_gesture_status *)event_buff;
+ if ((p_gesture_status->eid == 0x02) && (p_gesture_status->stype == 0x00)) {
+ u8 customlib[3] = { 0 };
+
+ ret = sec_ts_read_from_customlib(ts, customlib, 3);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to read custom library data\n", __func__);
+
+ input_info(true, &ts->client->dev, "%s: Custom Library, %x, %x, %x\n",
+ __func__, customlib[0], customlib[1], customlib[2]);
+
+ if (p_gesture_status->gesture_id == SEC_TS_GESTURE_CODE_SPAY ||
+ p_gesture_status->gesture_id == SEC_TS_GESTURE_CODE_DOUBLE_TAP) {
+ /* will be fixed to data structure */
+ if (customlib[1] & SEC_TS_MODE_CUSTOMLIB_AOD) {
+ u8 data[5] = { 0x0A, 0x00, 0x00, 0x00, 0x00 };
+
+ ret = sec_ts_read_from_customlib(ts, data, 5);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to read custom library data\n", __func__);
+
+ if (data[4] & SEC_TS_AOD_GESTURE_DOUBLETAB)
+ ts->scrub_id = CUSTOMLIB_EVENT_TYPE_AOD_DOUBLETAB;
+
+ ts->scrub_x = (data[1] & 0xFF) << 8 | (data[0] & 0xFF);
+ ts->scrub_y = (data[3] & 0xFF) << 8 | (data[2] & 0xFF);
+ input_info(true, &ts->client->dev, "%s: aod: %d\n",
+ __func__, ts->scrub_id);
+ ts->all_aod_tap_count++;
+ }
+ if (customlib[1] & SEC_TS_MODE_CUSTOMLIB_SPAY) {
+ ts->scrub_id = CUSTOMLIB_EVENT_TYPE_SPAY;
+ input_info(true, &ts->client->dev, "%s: SPAY: %d\n",
+ __func__, ts->scrub_id);
+ ts->all_spay_count++;
+ }
+ input_report_key(ts->input_dev, KEY_BLACK_UI_GESTURE, 1);
+ input_sync(ts->input_dev);
+ input_report_key(ts->input_dev, KEY_BLACK_UI_GESTURE, 0);
+ }
+ }
+ break;
+
+ default:
+ input_err(true, &ts->client->dev, "%s: unknown event %x %x %x %x %x %x\n", __func__,
+ event_buff[0], event_buff[1], event_buff[2],
+ event_buff[3], event_buff[4], event_buff[5]);
+ break;
+ }
+
+ if (t_id < MAX_SUPPORT_TOUCH_COUNT + MAX_SUPPORT_HOVER_COUNT) {
+ if (ts->coord[t_id].action == SEC_TS_COORDINATE_ACTION_PRESS) {
+ input_dbg(false, &ts->client->dev,
+ "%s[P] tID:%d x:%d y:%d z:%d major:%d minor:%d tc:%d type:%X\n",
+ ts->dex_name,
+ t_id, ts->coord[t_id].x, ts->coord[t_id].y, ts->coord[t_id].z,
+ ts->coord[t_id].major, ts->coord[t_id].minor, ts->touch_count,
+ ts->coord[t_id].ttype);
+
+ } else if (ts->coord[t_id].action == SEC_TS_COORDINATE_ACTION_RELEASE) {
+ input_dbg(false, &ts->client->dev,
+ "%s[R] tID:%d mc:%d tc:%d lx:%d ly:%d f:%d v:%02X%02X cal:%02X(%02X) id(%d,%d) p:%d P%02XT%04X\n",
+ ts->dex_name,
+ t_id, ts->coord[t_id].mcount, ts->touch_count,
+ ts->coord[t_id].x, ts->coord[t_id].y, max_force_p,
+ ts->plat_data->img_version_of_ic[2],
+ ts->plat_data->img_version_of_ic[3],
+ ts->cal_status, ts->nv, ts->tspid_val,
+ ts->tspicid_val, ts->coord[t_id].palm_count,
+ ts->cal_count, ts->tune_fix_ver );
+
+ ts->coord[t_id].action = SEC_TS_COORDINATE_ACTION_NONE;
+ ts->coord[t_id].mcount = 0;
+ ts->coord[t_id].palm_count = 0;
+ max_force_p = 0;
+ }
+ }
+
+ curr_pos++;
+ remain_event_count--;
+ } while (remain_event_count >= 0);
+
+ input_event(ts->input_dev, EV_MSC, MSC_TIMESTAMP,
+ ts->timestamp / 1000);
+ input_sync(ts->input_dev);
+
+ heatmap_read(&ts->v4l2, ts->timestamp);
+}
+
+static irqreturn_t sec_ts_isr(int irq, void *handle)
+{
+ struct sec_ts_data *ts = (struct sec_ts_data *)handle;
+
+ ts->timestamp = ktime_get_ns();
+
+ return IRQ_WAKE_THREAD;
+}
+
+static irqreturn_t sec_ts_irq_thread(int irq, void *ptr)
+{
+ struct sec_ts_data *ts = (struct sec_ts_data *)ptr;
+
+ if (sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_IRQ, true) < 0) {
+ /* Interrupt during bus suspend */
+ input_info(true, &ts->client->dev,
+ "%s: Skipping stray interrupt since bus is suspended(power_status: %d)\n",
+ __func__, ts->power_status);
+ return IRQ_HANDLED;
+ }
+
+ /* prevent CPU from entering deep sleep */
+ pm_qos_update_request(&ts->pm_qos_req, 100);
+
+ mutex_lock(&ts->eventlock);
+
+ sec_ts_read_event(ts);
+
+ mutex_unlock(&ts->eventlock);
+
+ pm_qos_update_request(&ts->pm_qos_req, PM_QOS_DEFAULT_VALUE);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_IRQ, false);
+
+ return IRQ_HANDLED;
+}
+
+int get_tsp_status(void)
+{
+ return 0;
+}
+EXPORT_SYMBOL(get_tsp_status);
+
+void sec_ts_set_charger(bool enable)
+{
+ return;
+/*
+ int ret;
+ u8 noise_mode_on[] = {0x01};
+ u8 noise_mode_off[] = {0x00};
+
+ if (enable) {
+ input_info(true, &ts->client->dev, "sec_ts_set_charger : charger CONNECTED!!\n");
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_NOISE_MODE, noise_mode_on, sizeof(noise_mode_on));
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "sec_ts_set_charger: fail to write NOISE_ON\n");
+ } else {
+ input_info(true, &ts->client->dev, "sec_ts_set_charger : charger DISCONNECTED!!\n");
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_NOISE_MODE, noise_mode_off, sizeof(noise_mode_off));
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "sec_ts_set_charger: fail to write NOISE_OFF\n");
+ }
+ */
+}
+EXPORT_SYMBOL(sec_ts_set_charger);
+
+int sec_ts_glove_mode_enables(struct sec_ts_data *ts, int mode)
+{
+ int ret;
+
+ if (mode)
+ ts->touch_functions = (ts->touch_functions | SEC_TS_BIT_SETFUNC_GLOVE | SEC_TS_DEFAULT_ENABLE_BIT_SETFUNC);
+ else
+ ts->touch_functions = ((ts->touch_functions & (~SEC_TS_BIT_SETFUNC_GLOVE)) | SEC_TS_DEFAULT_ENABLE_BIT_SETFUNC);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: pwr off, glove:%d, status:%x\n", __func__,
+ mode, ts->touch_functions);
+ goto glove_enable_err;
+ }
+
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCHFUNCTION, (u8 *)&ts->touch_functions, 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Failed to send command", __func__);
+ goto glove_enable_err;
+ }
+
+ input_info(true, &ts->client->dev, "%s: glove:%d, status:%x\n", __func__,
+ mode, ts->touch_functions);
+
+ return 0;
+
+glove_enable_err:
+ return -EIO;
+}
+EXPORT_SYMBOL(sec_ts_glove_mode_enables);
+
+int sec_ts_set_cover_type(struct sec_ts_data *ts, bool enable)
+{
+ int ret;
+
+ input_info(true, &ts->client->dev, "%s: %d\n", __func__, ts->cover_type);
+
+
+ switch (ts->cover_type) {
+ case SEC_TS_VIEW_WIRELESS:
+ case SEC_TS_VIEW_COVER:
+ case SEC_TS_VIEW_WALLET:
+ case SEC_TS_FLIP_WALLET:
+ case SEC_TS_LED_COVER:
+ case SEC_TS_MONTBLANC_COVER:
+ case SEC_TS_CLEAR_FLIP_COVER:
+ case SEC_TS_QWERTY_KEYBOARD_EUR:
+ case SEC_TS_QWERTY_KEYBOARD_KOR:
+ ts->cover_cmd = (u8)ts->cover_type;
+ break;
+ case SEC_TS_CHARGER_COVER:
+ case SEC_TS_COVER_NOTHING1:
+ case SEC_TS_COVER_NOTHING2:
+ default:
+ ts->cover_cmd = 0;
+ input_err(true, &ts->client->dev, "%s: not chage touch state, %d\n",
+ __func__, ts->cover_type);
+ break;
+ }
+
+ if (enable)
+ ts->touch_functions = (ts->touch_functions | SEC_TS_BIT_SETFUNC_COVER | SEC_TS_DEFAULT_ENABLE_BIT_SETFUNC);
+ else
+ ts->touch_functions = ((ts->touch_functions & (~SEC_TS_BIT_SETFUNC_COVER)) | SEC_TS_DEFAULT_ENABLE_BIT_SETFUNC);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: pwr off, close:%d, status:%x\n", __func__,
+ enable, ts->touch_functions);
+ goto cover_enable_err;
+ }
+
+ if (enable) {
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_COVERTYPE, &ts->cover_cmd, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Failed to send covertype command: %d", __func__, ts->cover_cmd);
+ goto cover_enable_err;
+ }
+ }
+
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCHFUNCTION, (u8 *)&(ts->touch_functions), 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Failed to send command", __func__);
+ goto cover_enable_err;
+ }
+
+ input_info(true, &ts->client->dev, "%s: close:%d, status:%x\n", __func__,
+ enable, ts->touch_functions);
+
+ return 0;
+
+cover_enable_err:
+ return -EIO;
+
+
+}
+EXPORT_SYMBOL(sec_ts_set_cover_type);
+
+void sec_ts_set_grip_type(struct sec_ts_data *ts, u8 set_type)
+{
+ u8 mode = G_NONE;
+
+ input_info(true, &ts->client->dev, "%s: re-init grip(%d), edh:%d, edg:%d, lan:%d\n", __func__,
+ set_type, ts->grip_edgehandler_direction, ts->grip_edge_range, ts->grip_landscape_mode);
+
+ /* edge handler */
+ if (ts->grip_edgehandler_direction != 0)
+ mode |= G_SET_EDGE_HANDLER;
+
+ if (set_type == GRIP_ALL_DATA) {
+ /* edge */
+ if (ts->grip_edge_range != 60)
+ mode |= G_SET_EDGE_ZONE;
+
+ /* dead zone */
+ if (ts->grip_landscape_mode == 1) /* default 0 mode, 32 */
+ mode |= G_SET_LANDSCAPE_MODE;
+ else
+ mode |= G_SET_NORMAL_MODE;
+ }
+
+ if (mode)
+ set_grip_data_to_ic(ts, mode);
+
+}
+
+/* for debugging--------------------------------------------------------------------------------------*/
+
+static int sec_ts_pinctrl_configure(struct sec_ts_data *ts, bool enable)
+{
+ struct pinctrl_state *state;
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, enable ? "ACTIVE" : "SUSPEND");
+
+ if (enable) {
+ state = pinctrl_lookup_state(ts->plat_data->pinctrl, "on_state");
+ if (IS_ERR(ts->plat_data->pinctrl))
+ input_err(true, &ts->client->dev, "%s: could not get active pinstate\n", __func__);
+ } else {
+ state = pinctrl_lookup_state(ts->plat_data->pinctrl, "off_state");
+ if (IS_ERR(ts->plat_data->pinctrl))
+ input_err(true, &ts->client->dev, "%s: could not get suspend pinstate\n", __func__);
+ }
+
+ if (!IS_ERR_OR_NULL(state))
+ return pinctrl_select_state(ts->plat_data->pinctrl, state);
+
+ return 0;
+
+}
+
+static int sec_ts_power(void *data, bool on)
+{
+ struct sec_ts_data *ts = (struct sec_ts_data *)data;
+ const struct sec_ts_plat_data *pdata = ts->plat_data;
+ struct regulator *regulator_dvdd = NULL;
+ struct regulator *regulator_avdd = NULL;
+ static bool enabled;
+ int ret = 0;
+
+ if (enabled == on)
+ return ret;
+
+ regulator_dvdd = regulator_get(NULL, pdata->regulator_dvdd);
+ if (IS_ERR_OR_NULL(regulator_dvdd)) {
+ input_err(true, &ts->client->dev, "%s: Failed to get %s regulator.\n",
+ __func__, pdata->regulator_dvdd);
+ ret = PTR_ERR(regulator_dvdd);
+ goto error;
+ }
+
+ regulator_avdd = regulator_get(NULL, pdata->regulator_avdd);
+ if (IS_ERR_OR_NULL(regulator_avdd)) {
+ input_err(true, &ts->client->dev, "%s: Failed to get %s regulator.\n",
+ __func__, pdata->regulator_avdd);
+ ret = PTR_ERR(regulator_avdd);
+ goto error;
+ }
+
+ if (on) {
+ ret = regulator_enable(regulator_dvdd);
+ if (ret) {
+ input_err(true, &ts->client->dev, "%s: Failed to enable avdd: %d\n", __func__, ret);
+ goto out;
+ }
+
+ sec_ts_delay(1);
+
+ ret = regulator_enable(regulator_avdd);
+ if (ret) {
+ input_err(true, &ts->client->dev, "%s: Failed to enable vdd: %d\n", __func__, ret);
+ goto out;
+ }
+ } else {
+ regulator_disable(regulator_dvdd);
+ regulator_disable(regulator_avdd);
+ }
+
+ enabled = on;
+
+out:
+ input_err(true, &ts->client->dev, "%s: %s: avdd:%s, dvdd:%s\n", __func__, on ? "on" : "off",
+ regulator_is_enabled(regulator_avdd) ? "on" : "off",
+ regulator_is_enabled(regulator_dvdd) ? "on" : "off");
+
+error:
+ regulator_put(regulator_dvdd);
+ regulator_put(regulator_avdd);
+
+ return ret;
+}
+
+static int sec_ts_parse_dt(struct i2c_client *client)
+{
+ struct device *dev = &client->dev;
+ struct sec_ts_plat_data *pdata = dev->platform_data;
+ struct device_node *np = dev->of_node;
+ u32 coords[2];
+ int ret = 0;
+ int count = 0;
+ u32 ic_match_value;
+ int lcdtype = 0;
+#if defined(CONFIG_EXYNOS_DECON_FB)
+ int connected;
+#endif
+
+ pdata->tsp_icid = of_get_named_gpio(np, "sec,tsp-icid_gpio", 0);
+ if (gpio_is_valid(pdata->tsp_icid)) {
+ input_info(true, dev, "%s: TSP_ICID : %d\n", __func__, gpio_get_value(pdata->tsp_icid));
+ if (of_property_read_u32(np, "sec,icid_match_value", &ic_match_value)) {
+ input_err(true, dev, "%s: Failed to get icid match value\n", __func__);
+ return -EINVAL;
+ }
+
+ if (gpio_get_value(pdata->tsp_icid) != ic_match_value) {
+ input_err(true, dev, "%s: Do not match TSP_ICID\n", __func__);
+ return -EINVAL;
+ }
+ } else {
+ input_err(true, dev, "%s: Failed to get tsp-icid gpio\n", __func__);
+ }
+
+ pdata->tsp_vsync = of_get_named_gpio(np, "sec,tsp_vsync_gpio", 0);
+ if (gpio_is_valid(pdata->tsp_vsync))
+ input_info(true, &client->dev, "%s: vsync %s\n", __func__,
+ gpio_get_value(pdata->tsp_vsync) ? "disable" : "enable");
+
+ pdata->irq_gpio = of_get_named_gpio(np, "sec,irq_gpio", 0);
+ if (gpio_is_valid(pdata->irq_gpio)) {
+ ret = gpio_request_one(pdata->irq_gpio, GPIOF_DIR_IN, "sec,tsp_int");
+ if (ret) {
+ input_err(true, &client->dev, "%s: Unable to request tsp_int [%d]\n", __func__, pdata->irq_gpio);
+ return -EINVAL;
+ }
+ } else {
+ input_err(true, &client->dev, "%s: Failed to get irq gpio\n", __func__);
+ return -EINVAL;
+ }
+
+ client->irq = gpio_to_irq(pdata->irq_gpio);
+
+ if (of_property_read_u32(np, "sec,irq_type", &pdata->irq_type)) {
+ input_err(true, dev, "%s: Failed to get irq_type property\n", __func__);
+ pdata->irq_type = IRQF_TRIGGER_LOW | IRQF_ONESHOT;
+ }
+
+ if (of_property_read_u32(np, "sec,i2c-burstmax", &pdata->i2c_burstmax)) {
+ input_dbg(false, &client->dev, "%s: Failed to get i2c_burstmax property\n", __func__);
+ pdata->i2c_burstmax = 256;
+ }
+ if (pdata->i2c_burstmax > I2C_PREALLOC_READ_BUF_SZ ||
+ pdata->i2c_burstmax > I2C_PREALLOC_WRITE_BUF_SZ) {
+ input_err(true, &client->dev,
+ "%s: i2c_burstmax is larger than i2c_read_buf and/or i2c_write_buf.\n",
+ __func__);
+ return -EINVAL;
+ }
+
+ if (of_property_read_u32_array(np, "sec,max_coords", coords, 2)) {
+ input_err(true, &client->dev, "%s: Failed to get max_coords property\n", __func__);
+ return -EINVAL;
+ }
+ pdata->max_x = coords[0] - 1;
+ pdata->max_y = coords[1] - 1;
+
+#ifdef PAT_CONTROL
+ if (of_property_read_u32(np, "sec,pat_function", &pdata->pat_function) < 0) {
+ pdata->pat_function = 0;
+ input_err(true, dev, "%s: Failed to get pat_function property\n", __func__);
+ }
+
+ if (of_property_read_u32(np, "sec,afe_base", &pdata->afe_base) < 0) {
+ pdata->afe_base = 0;
+ input_err(true, dev, "%s: Failed to get afe_base property\n", __func__);
+ }
+#endif
+
+ pdata->tsp_id = of_get_named_gpio(np, "sec,tsp-id_gpio", 0);
+ if (gpio_is_valid(pdata->tsp_id))
+ input_info(true, dev, "%s: TSP_ID : %d\n", __func__, gpio_get_value(pdata->tsp_id));
+ else
+ input_err(true, dev, "%s: Failed to get tsp-id gpio\n", __func__);
+
+ pdata->switch_gpio = of_get_named_gpio(np,
+ "sec,switch_gpio", 0);
+ if (gpio_is_valid(pdata->switch_gpio)) {
+ ret = gpio_request_one(pdata->switch_gpio,
+ GPIOF_OUT_INIT_HIGH,
+ "sec,touch_i2c_switch");
+ if (ret) {
+ input_err(true, dev,
+ "%s: Failed to request gpio %d\n",
+ __func__, pdata->switch_gpio);
+ return -EINVAL;
+ }
+
+ ret = gpio_direction_output(pdata->switch_gpio, 1);
+ if (ret) {
+ input_err(true, dev,
+ "%s: Failed to set gpio %d direction\n",
+ __func__, pdata->switch_gpio);
+ return -EINVAL;
+ }
+ } else {
+ input_err(true, dev, "%s: Failed to get switch_gpio\n",
+ __func__);
+ }
+
+ pdata->reset_gpio = of_get_named_gpio(np, "sec,reset_gpio", 0);
+ if (gpio_is_valid(pdata->reset_gpio)) {
+ ret = gpio_request_one(pdata->reset_gpio,
+ GPIOF_OUT_INIT_HIGH,
+ "sec,touch_reset_gpio");
+ if (ret) {
+ input_err(true, dev,
+ "%s: Failed to request gpio %d, ret %d\n",
+ __func__, pdata->reset_gpio, ret);
+ pdata->reset_gpio = -1;
+ }
+
+ } else
+ input_err(true, dev, "%s: Failed to get reset_gpio\n",
+ __func__);
+
+ count = of_property_count_strings(np, "sec,firmware_name");
+ if (count <= 0) {
+ pdata->firmware_name = NULL;
+ } else {
+ if (gpio_is_valid(pdata->tsp_id))
+ of_property_read_string_index(np, "sec,firmware_name", gpio_get_value(pdata->tsp_id), &pdata->firmware_name);
+ else
+ of_property_read_string_index(np, "sec,firmware_name", 0, &pdata->firmware_name);
+ }
+
+ if (of_property_read_string_index(np, "sec,project_name", 0, &pdata->project_name))
+ input_err(true, &client->dev, "%s: skipped to get project_name property\n", __func__);
+ if (of_property_read_string_index(np, "sec,project_name", 1, &pdata->model_name))
+ input_err(true, &client->dev, "%s: skipped to get model_name property\n", __func__);
+
+#if defined(CONFIG_FB_MSM_MDSS_SAMSUNG)
+ lcdtype = get_lcd_attached("GET");
+ if (lcdtype < 0) {
+ input_err(true, &client->dev, "%s: lcd is not attached\n", __func__);
+ return -ENODEV;
+ }
+#endif
+
+#if defined(CONFIG_EXYNOS_DECON_FB)
+ connected = get_lcd_info("connected");
+ if (connected < 0) {
+ input_err(true, dev, "%s: Failed to get lcd info\n", __func__);
+ return -EINVAL;
+ }
+
+ if (!connected) {
+ input_err(true, &client->dev, "%s: lcd is disconnected\n", __func__);
+ return -ENODEV;
+ }
+
+ input_info(true, &client->dev, "%s: lcd is connected\n", __func__);
+
+ lcdtype = get_lcd_info("id");
+ if (lcdtype < 0) {
+ input_err(true, dev, "%s: Failed to get lcd info\n", __func__);
+ return -EINVAL;
+ }
+#endif
+
+ input_info(true, &client->dev, "%s: lcdtype 0x%08X\n", __func__, lcdtype);
+
+ if (pdata->model_name && strncmp(pdata->model_name, "G950", 4) == 0)
+ pdata->panel_revision = 0;
+ else
+ pdata->panel_revision = ((lcdtype >> 8) & 0xFF) >> 4;
+
+ if (of_property_read_string(np, "sec,regulator_dvdd", &pdata->regulator_dvdd)) {
+ input_err(true, dev, "%s: Failed to get regulator_dvdd name property\n", __func__);
+ return -EINVAL;
+ }
+
+ if (of_property_read_string(np, "sec,regulator_avdd", &pdata->regulator_avdd)) {
+ input_err(true, dev, "%s: Failed to get regulator_avdd name property\n", __func__);
+ return -EINVAL;
+ }
+
+ pdata->power = sec_ts_power;
+
+ if (of_property_read_u32(np, "sec,always_lpmode", &pdata->always_lpmode) < 0)
+ pdata->always_lpmode = 0;
+
+ if (of_property_read_u32(np, "sec,bringup", &pdata->bringup) < 0)
+ pdata->bringup = 0;
+
+ if (of_property_read_u32(np, "sec,mis_cal_check", &pdata->mis_cal_check) < 0)
+ pdata->mis_cal_check = 0;
+
+ pdata->regulator_boot_on = of_property_read_bool(np, "sec,regulator_boot_on");
+ pdata->support_sidegesture = of_property_read_bool(np, "sec,support_sidegesture");
+ pdata->support_dex = of_property_read_bool(np, "support_dex_mode");
+
+ pdata->support_mt_pressure = true;
+
+#ifdef PAT_CONTROL
+ input_err(true, &client->dev, "%s: i2c buffer limit: %d, lcd_id:%06X, bringup:%d, FW:%s(%d), id:%d,%d, pat_function:%d mis_cal:%d dex:%d, gesture:%d\n",
+ __func__, pdata->i2c_burstmax, lcdtype, pdata->bringup, pdata->firmware_name,
+ count, pdata->tsp_id, pdata->tsp_icid, pdata->pat_function,
+ pdata->mis_cal_check, pdata->support_dex, pdata->support_sidegesture);
+#else
+ input_err(true, &client->dev, "%s: i2c buffer limit: %d, lcd_id:%06X, bringup:%d, FW:%s(%d), id:%d,%d, dex:%d, gesture:%d\n",
+ __func__, pdata->i2c_burstmax, lcdtype, pdata->bringup, pdata->firmware_name,
+ count, pdata->tsp_id, pdata->tsp_icid, pdata->support_dex, pdata->support_sidegesture);
+#endif
+ return ret;
+}
+
+int sec_ts_read_information(struct sec_ts_data *ts)
+{
+ unsigned char data[13] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_READ_INFO, true);
+
+ memset(data, 0x0, 3);
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_ID, data, 3);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to read device id(%d)\n",
+ __func__, ret);
+ goto out;
+ }
+
+ input_info(true, &ts->client->dev,
+ "%s: %X, %X, %X\n",
+ __func__, data[0], data[1], data[2]);
+ memset(data, 0x0, 11);
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_PANEL_INFO, data, 11);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to read sub id(%d)\n",
+ __func__, ret);
+ goto out;
+ }
+
+ input_info(true, &ts->client->dev,
+ "%s: nTX:%X, nRX:%X, rY:%d, rX:%d\n",
+ __func__, data[8], data[9],
+ (data[2] << 8) | data[3], (data[0] << 8) | data[1]);
+
+ /* Set X,Y Resolution from IC information. */
+ if (((data[0] << 8) | data[1]) > 0)
+ ts->plat_data->max_x = ((data[0] << 8) | data[1]) - 1;
+
+ if (((data[2] << 8) | data[3]) > 0)
+ ts->plat_data->max_y = ((data[2] << 8) | data[3]) - 1;
+
+ ts->tx_count = data[8];
+ ts->rx_count = data[9];
+
+ data[0] = 0;
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_BOOT_STATUS, data, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to read sub id(%d)\n",
+ __func__, ret);
+ goto out;
+ }
+
+ input_info(true, &ts->client->dev,
+ "%s: STATUS : %X\n",
+ __func__, data[0]);
+
+ memset(data, 0x0, 4);
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_TS_STATUS, data, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to read sub id(%d)\n",
+ __func__, ret);
+ goto out;
+ }
+
+ input_info(true, &ts->client->dev,
+ "%s: TOUCH STATUS : %02X, %02X, %02X, %02X\n",
+ __func__, data[0], data[1], data[2], data[3]);
+ ret = sec_ts_i2c_read(ts, SEC_TS_CMD_SET_TOUCHFUNCTION, (u8 *)&(ts->touch_functions), 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to read touch functions(%d)\n",
+ __func__, ret);
+ goto out;
+ }
+
+ input_info(true, &ts->client->dev,
+ "%s: Functions : %02X\n",
+ __func__, ts->touch_functions);
+
+out:
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_READ_INFO, false);
+ return ret;
+}
+
+#ifdef SEC_TS_SUPPORT_CUSTOMLIB
+int sec_ts_set_custom_library(struct sec_ts_data *ts)
+{
+ u8 data[3] = { 0 };
+ int ret;
+
+ input_err(true, &ts->client->dev, "%s: Custom Library (0x%02x)\n",
+ __func__, ts->lowpower_mode);
+
+ data[2] = ts->lowpower_mode;
+
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_WRITE_PARAM, &data[0], 3);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to Custom Library\n", __func__);
+
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_NOTIFY_PACKET, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to send NOTIFY Custom Library\n", __func__);
+
+ return ret;
+}
+
+int sec_ts_check_custom_library(struct sec_ts_data *ts)
+{
+ u8 data[10] = { 0 };
+ int ret = -1;
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_CUSTOMLIB_GET_INFO, &data[0], 10);
+
+ input_info(true, &ts->client->dev,
+ "%s: (%d) %c%c%c%c, || %02X, %02X, %02X, %02X, || %02X, %02X\n",
+ __func__, ret, data[0], data[1], data[2], data[3], data[4],
+ data[5], data[6], data[7], data[8], data[9]);
+
+ /* compare model name with device tree */
+ if (ts->plat_data->model_name)
+ ret = strncmp(data, ts->plat_data->model_name, 4);
+
+ if (ret == 0)
+ ts->use_customlib= true;
+ else
+ ts->use_customlib= false;
+
+ input_err(true, &ts->client->dev, "%s: use %s\n",
+ __func__, ts->use_customlib? "CUSTOMLIB" : "VENDOR");
+
+ return ret;
+}
+#endif
+
+static void sec_ts_set_input_prop(struct sec_ts_data *ts, struct input_dev *dev, u8 propbit)
+{
+ static char sec_ts_phys[64] = { 0 };
+
+ snprintf(sec_ts_phys, sizeof(sec_ts_phys), "%s/input1",
+ dev->name);
+ dev->phys = sec_ts_phys;
+ dev->id.bustype = BUS_I2C;
+ dev->dev.parent = &ts->client->dev;
+
+ set_bit(EV_SYN, dev->evbit);
+ set_bit(EV_KEY, dev->evbit);
+ set_bit(EV_ABS, dev->evbit);
+ set_bit(EV_SW, dev->evbit);
+ set_bit(BTN_TOUCH, dev->keybit);
+ set_bit(BTN_TOOL_FINGER, dev->keybit);
+ set_bit(KEY_BLACK_UI_GESTURE, dev->keybit);
+#ifdef SEC_TS_SUPPORT_TOUCH_KEY
+ if (ts->plat_data->support_mskey) {
+ int i;
+
+ for (i = 0 ; i < ts->plat_data->num_touchkey ; i++)
+ set_bit(ts->plat_data->touchkey[i].keycode, dev->keybit);
+
+ set_bit(EV_LED, dev->evbit);
+ set_bit(LED_MISC, dev->ledbit);
+ }
+#endif
+ if (ts->plat_data->support_sidegesture) {
+ set_bit(KEY_SIDE_GESTURE, dev->keybit);
+ set_bit(KEY_SIDE_GESTURE_RIGHT, dev->keybit);
+ set_bit(KEY_SIDE_GESTURE_LEFT, dev->keybit);
+ }
+ set_bit(propbit, dev->propbit);
+ set_bit(KEY_HOMEPAGE, dev->keybit);
+
+ input_set_capability(dev, EV_SW, SW_GLOVE);
+
+ input_set_abs_params(dev, ABS_MT_POSITION_X, 0, ts->plat_data->max_x, 0, 0);
+ input_set_abs_params(dev, ABS_MT_POSITION_Y, 0, ts->plat_data->max_y, 0, 0);
+ input_set_abs_params(dev, ABS_MT_TOUCH_MAJOR, 0, 255, 0, 0);
+ input_set_abs_params(dev, ABS_MT_TOUCH_MINOR, 0, 255, 0, 0);
+ input_set_abs_params(dev, ABS_MT_CUSTOM, 0, 0xFFFF, 0, 0);
+ if (ts->plat_data->support_mt_pressure)
+ input_set_abs_params(dev, ABS_MT_PRESSURE, 0,
+ SEC_TS_PRESSURE_MAX, 0, 0);
+
+ input_set_capability(ts->input_dev, EV_MSC, MSC_TIMESTAMP);
+
+ if (propbit == INPUT_PROP_POINTER)
+ input_mt_init_slots(dev, MAX_SUPPORT_TOUCH_COUNT, INPUT_MT_POINTER);
+ else
+ input_mt_init_slots(dev, MAX_SUPPORT_TOUCH_COUNT, INPUT_MT_DIRECT);
+
+ input_set_drvdata(dev, ts);
+}
+
+static int sec_ts_fw_init(struct sec_ts_data *ts)
+{
+ int ret = SEC_TS_ERR_NA;
+ bool force_update = false;
+ bool valid_firmware_integrity = false;
+ unsigned char data[5] = { 0 };
+ unsigned char deviceID[5] = { 0 };
+ unsigned char result = 0;
+
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_DEVICE_ID, deviceID, 5);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: failed to read device ID(%d)\n",
+ __func__, ret);
+ else
+ input_info(true, &ts->client->dev,
+ "%s: TOUCH DEVICE ID : %02X, %02X, %02X, %02X, %02X\n",
+ __func__, deviceID[0], deviceID[1], deviceID[2],
+ deviceID[3], deviceID[4]);
+
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_FIRMWARE_INTEGRITY, &result, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: failed to integrity check (%d)\n",
+ __func__, ret);
+ } else {
+ if (result & 0x80)
+ valid_firmware_integrity = true;
+ else
+ input_err(true, &ts->client->dev, "%s: invalid integrity result (0x%x)\n",
+ __func__, result);
+ }
+
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_BOOT_STATUS, &data[0], 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to read sub id(%d)\n", __func__, ret);
+ } else {
+ ret = sec_ts_i2c_read(ts, SEC_TS_READ_TS_STATUS, &data[1], 4);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to touch status(%d)\n",
+ __func__, ret);
+ }
+ input_info(true, &ts->client->dev,
+ "%s: TOUCH STATUS : %02X || %02X, %02X, %02X, %02X\n",
+ __func__, data[0], data[1], data[2], data[3], data[4]);
+
+ if (data[0] == SEC_TS_STATUS_BOOT_MODE)
+ ts->checksum_result = 1;
+
+ if (((data[0] == SEC_TS_STATUS_APP_MODE &&
+ data[2] == TOUCH_SYSTEM_MODE_FLASH) || ret < 0) &&
+ (valid_firmware_integrity == false))
+ force_update = true;
+
+ ret = sec_ts_read_information(ts);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to read information 0x%x\n",
+ __func__, ret);
+ return SEC_TS_ERR_INIT;
+ }
+
+ ts->touch_functions |= SEC_TS_DEFAULT_ENABLE_BIT_SETFUNC;
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCHFUNCTION,
+ (u8 *)&ts->touch_functions, 2);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to send touch func_mode command",
+ __func__);
+
+ /* Sense_on */
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_ON, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to write Sense_on 0x%x\n",
+ __func__, ret);
+ return SEC_TS_ERR_INIT;
+ }
+
+ ts->pFrame = kzalloc(ts->tx_count * ts->rx_count * 2, GFP_KERNEL);
+ if (!ts->pFrame)
+ return SEC_TS_ERR_ALLOC_FRAME;
+
+ ts->gainTable = kzalloc(ts->tx_count * ts->rx_count, GFP_KERNEL);
+ if (!ts->gainTable) {
+ kfree(ts->pFrame);
+ ts->pFrame = NULL;
+ return SEC_TS_ERR_ALLOC_GAINTABLE;
+ }
+
+ if (ts->plat_data->support_dex) {
+ ts->input_dev_pad->name = "sec_touchpad";
+ sec_ts_set_input_prop(ts, ts->input_dev_pad,
+ INPUT_PROP_POINTER);
+ }
+ ts->dex_name = "";
+
+ ts->input_dev->name = "sec_touchscreen";
+ sec_ts_set_input_prop(ts, ts->input_dev, INPUT_PROP_DIRECT);
+#ifdef USE_OPEN_CLOSE
+ ts->input_dev->open = sec_ts_input_open;
+ ts->input_dev->close = sec_ts_input_close;
+#endif
+ ts->input_dev_touch = ts->input_dev;
+
+ ret = input_register_device(ts->input_dev);
+ if (ret) {
+ input_err(true, &ts->client->dev, "%s: Unable to register %s input device 0x%x\n",
+ __func__, ts->input_dev->name, ret);
+ return SEC_TS_ERR_REG_INPUT_DEV;
+ }
+
+ if (ts->plat_data->support_dex) {
+ ret = input_register_device(ts->input_dev_pad);
+ if (ret) {
+ input_err(true, &ts->client->dev, "%s: Unable to register %s input device 0x%x\n",
+ __func__, ts->input_dev_pad->name, ret);
+ return SEC_TS_ERR_REG_INPUT_PAD_DEV;
+ }
+ }
+
+ return SEC_TS_ERR_NA;
+}
+
+static void sec_ts_device_init(struct sec_ts_data *ts)
+{
+#if (1) //!defined(CONFIG_SAMSUNG_PRODUCT_SHIP)
+ sec_ts_raw_device_init(ts);
+#endif
+ sec_ts_fn_init(ts);
+
+#ifdef SEC_TS_SUPPORT_CUSTOMLIB
+ sec_ts_check_custom_library(ts);
+ if (ts->use_customlib)
+ sec_ts_set_custom_library(ts);
+#endif
+}
+
+static struct notifier_block sec_ts_screen_nb;
+
+static int sec_ts_probe(struct i2c_client *client,
+ const struct i2c_device_id *id)
+{
+ struct sec_ts_data *ts;
+ struct sec_ts_plat_data *pdata;
+ int ret = 0;
+
+ input_info(true, &client->dev, "%s\n", __func__);
+
+ if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
+ input_err(true, &client->dev, "%s: EIO err!\n", __func__);
+ return -EIO;
+ }
+
+ /* parse dt */
+ if (client->dev.of_node) {
+ pdata = devm_kzalloc(&client->dev,
+ sizeof(struct sec_ts_plat_data), GFP_KERNEL);
+
+ if (!pdata) {
+ input_err(true, &client->dev, "%s: Failed to allocate platform data\n", __func__);
+ goto error_allocate_pdata;
+ }
+
+ client->dev.platform_data = pdata;
+
+ ret = sec_ts_parse_dt(client);
+ if (ret) {
+ input_err(true, &client->dev, "%s: Failed to parse dt\n", __func__);
+ goto error_allocate_mem;
+ }
+ } else {
+ pdata = client->dev.platform_data;
+ if (!pdata) {
+ input_err(true, &client->dev, "%s: No platform data found\n", __func__);
+ goto error_allocate_pdata;
+ }
+ }
+
+ if (!pdata->power) {
+ input_err(true, &client->dev, "%s: No power contorl found\n", __func__);
+ goto error_allocate_mem;
+ }
+
+ pdata->pinctrl = devm_pinctrl_get(&client->dev);
+ if (IS_ERR(pdata->pinctrl))
+ input_err(true, &client->dev, "%s: could not get pinctrl\n", __func__);
+
+ ts = kzalloc(sizeof(struct sec_ts_data), GFP_KERNEL);
+ if (!ts)
+ goto error_allocate_mem;
+
+ ts->client = client;
+ ts->plat_data = pdata;
+ ts->crc_addr = 0x0001FE00;
+ ts->fw_addr = 0x00002000;
+ ts->para_addr = 0x18000;
+ ts->flash_page_size = SEC_TS_FW_BLK_SIZE_DEFAULT;
+ ts->sec_ts_i2c_read = sec_ts_i2c_read;
+ ts->sec_ts_i2c_read_heap = sec_ts_i2c_read_heap;
+ ts->sec_ts_i2c_write = sec_ts_i2c_write;
+ ts->sec_ts_i2c_write_burst = sec_ts_i2c_write_burst;
+ ts->sec_ts_i2c_write_burst_heap = sec_ts_i2c_write_burst_heap;
+ ts->sec_ts_i2c_read_bulk = sec_ts_i2c_read_bulk;
+ ts->sec_ts_i2c_read_bulk_heap = sec_ts_i2c_read_bulk_heap;
+ ts->i2c_burstmax = pdata->i2c_burstmax;
+#ifdef USE_POWER_RESET_WORK
+ INIT_DELAYED_WORK(&ts->reset_work, sec_ts_reset_work);
+#endif
+ INIT_DELAYED_WORK(&ts->work_read_info, sec_ts_read_info_work);
+ INIT_WORK(&ts->suspend_work, sec_ts_suspend_work);
+ INIT_WORK(&ts->resume_work, sec_ts_resume_work);
+ ts->event_wq = alloc_workqueue("sec_ts-event-queue", WQ_UNBOUND |
+ WQ_HIGHPRI | WQ_CPU_INTENSIVE, 1);
+ if (!ts->event_wq) {
+ input_err(true, &ts->client->dev,
+ "%s: Cannot create work thread\n", __func__);
+ ret = -ENOMEM;
+ goto error_alloc_workqueue;
+ }
+
+ init_completion(&ts->bus_resumed);
+ complete_all(&ts->bus_resumed);
+
+#ifdef SEC_TS_FW_UPDATE_ON_PROBE
+ INIT_WORK(&ts->work_fw_update, sec_ts_fw_update_work);
+#else
+ input_info(true, &ts->client->dev, "%s: fw update on probe disabled!\n",
+ __func__);
+#endif
+
+ ts->is_fw_corrupted = false;
+
+ /* Assume screen is on throughout probe */
+ ts->bus_refmask = SEC_TS_BUS_REF_SCREEN_ON;
+
+ i2c_set_clientdata(client, ts);
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ ts->tbn = tbn_init(&ts->client->dev);
+ if (!ts->tbn) {
+ input_err(true, &ts->client->dev,
+ "%s: TBN initialization error\n", __func__);
+ ret = -ENODEV;
+ goto err_init_tbn;
+ }
+#endif
+
+ if (gpio_is_valid(ts->plat_data->tsp_id))
+ ts->tspid_val = gpio_get_value(ts->plat_data->tsp_id);
+
+ if (gpio_is_valid(ts->plat_data->tsp_icid))
+ ts->tspicid_val = gpio_get_value(ts->plat_data->tsp_icid);
+
+ ts->input_dev = input_allocate_device();
+ if (!ts->input_dev) {
+ input_err(true, &ts->client->dev, "%s: allocate device err!\n", __func__);
+ ret = -ENOMEM;
+ goto err_allocate_input_dev;
+ }
+
+ if (ts->plat_data->support_dex) {
+ ts->input_dev_pad = input_allocate_device();
+ if (!ts->input_dev_pad) {
+ input_err(true, &ts->client->dev, "%s: allocate device err!\n", __func__);
+ ret = -ENOMEM;
+ goto err_allocate_input_dev_pad;
+ }
+ }
+
+ ts->touch_count = 0;
+ ts->sec_ts_i2c_write = sec_ts_i2c_write;
+ ts->sec_ts_i2c_read = sec_ts_i2c_read;
+ ts->sec_ts_i2c_read_heap = sec_ts_i2c_read_heap;
+ ts->sec_ts_read_customlib = sec_ts_read_from_customlib;
+
+ ts->max_z_value = 0;
+ ts->min_z_value = 0xFFFFFFFF;
+ ts->sum_z_value = 0;
+
+ mutex_init(&ts->bus_mutex);
+ mutex_init(&ts->lock);
+ mutex_init(&ts->device_mutex);
+ mutex_init(&ts->i2c_mutex);
+ mutex_init(&ts->eventlock);
+
+ init_completion(&ts->resume_done);
+ complete_all(&ts->resume_done);
+
+ if (pdata->always_lpmode)
+ ts->lowpower_mode |= SEC_TS_MODE_CUSTOMLIB_FORCE_KEY;
+ else
+ ts->lowpower_mode &= ~SEC_TS_MODE_CUSTOMLIB_FORCE_KEY;
+
+ input_info(true, &client->dev, "%s: init resource\n", __func__);
+
+ sec_ts_pinctrl_configure(ts, true);
+
+ /* power enable */
+ sec_ts_power(ts, true);
+ if (!pdata->regulator_boot_on)
+ sec_ts_delay(70);
+ ts->power_status = SEC_TS_STATE_POWER_ON;
+ ts->external_factory = false;
+
+ ret = sec_ts_wait_for_ready(ts, SEC_TS_ACK_BOOT_COMPLETE);
+ if (ret < 0) {
+ u8 boot_status;
+ /* Read the boot status in case device is in bootloader mode */
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_BOOT_STATUS,
+ &boot_status, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: could not read boot status. Assuming no device connected.\n",
+ __func__);
+ goto err_init;
+ }
+
+ input_info(true, &ts->client->dev,
+ "%s: Attempting to reflash the firmware. Boot status = 0x%02X\n",
+ __func__, boot_status);
+ if (boot_status != SEC_TS_STATUS_BOOT_MODE)
+ input_err(true, &ts->client->dev,
+ "%s: device is not in bootloader mode!\n",
+ __func__);
+
+ ts->is_fw_corrupted = true;
+ }
+
+ input_info(true, &client->dev, "%s: power enable\n", __func__);
+
+ if (ts->is_fw_corrupted == false) {
+ switch (sec_ts_fw_init(ts)) {
+ case SEC_TS_ERR_INIT:
+ goto err_init;
+ case SEC_TS_ERR_ALLOC_FRAME:
+ goto err_allocate_frame;
+ case SEC_TS_ERR_ALLOC_GAINTABLE:
+ goto err_allocate_gaintable;
+ case SEC_TS_ERR_REG_INPUT_DEV:
+ goto err_input_register_device;
+ case SEC_TS_ERR_REG_INPUT_PAD_DEV:
+ goto err_input_pad_register_device;
+ }
+ }
+
+ pm_qos_add_request(&ts->pm_qos_req, PM_QOS_CPU_DMA_LATENCY,
+ PM_QOS_DEFAULT_VALUE);
+
+ /*
+ * Heatmap_probe must be called before irq routine is registered,
+ * because heatmap_read is called from the irq context.
+ * If the ISR runs before heatmap_probe is finished, it will invoke
+ * heatmap_read and cause NPE, since read_frame would not yet be set.
+ */
+ ts->v4l2.parent_dev = &ts->client->dev;
+ ts->v4l2.input_dev = ts->input_dev;
+ ts->v4l2.read_frame = read_heatmap_raw;
+ ts->v4l2.width = ts->tx_count;
+ ts->v4l2.height = ts->rx_count;
+ /* 120 Hz operation */
+ ts->v4l2.timeperframe.numerator = 1;
+ ts->v4l2.timeperframe.denominator = 120;
+ ret = heatmap_probe(&ts->v4l2);
+ if (ret) {
+ input_err(true, &ts->client->dev,
+ "%s: Heatmap probe failed\n", __func__);
+ goto err_irq;
+ }
+
+ input_info(true, &ts->client->dev, "%s: request_irq = %d\n", __func__,
+ client->irq);
+
+ ret = request_threaded_irq(client->irq, sec_ts_isr, sec_ts_irq_thread,
+ ts->plat_data->irq_type, SEC_TS_I2C_NAME, ts);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: Unable to request threaded irq\n", __func__);
+ goto err_heatmap;
+ }
+
+ ts->notifier = sec_ts_screen_nb;
+ ret = msm_drm_register_client(&ts->notifier);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: msm_drm_register_client failed. ret = 0x%08X\n",
+ __func__, ret);
+ goto err_register_drm_client;
+ }
+
+#ifndef CONFIG_SEC_SYSFS
+ sec_class = class_create(THIS_MODULE, "sec");
+#endif
+
+ device_init_wakeup(&client->dev, true);
+
+ if (ts->is_fw_corrupted == false)
+ sec_ts_device_init(ts);
+
+#ifdef SEC_TS_FW_UPDATE_ON_PROBE
+ schedule_work(&ts->work_fw_update);
+
+ /* Do not finish probe without checking and flashing the firmware */
+ flush_work(&ts->work_fw_update);
+#endif
+
+ if (ts->is_fw_corrupted == false)
+ schedule_delayed_work(&ts->work_read_info,
+ msecs_to_jiffies(5000));
+
+#if defined(CONFIG_TOUCHSCREEN_DUMP_MODE)
+ dump_callbacks.inform_dump = dump_tsp_log;
+ INIT_DELAYED_WORK(&ts->ghost_check, sec_ts_check_rawdata);
+ p_ghost_check = &ts->ghost_check;
+#endif
+
+ ts_dup = ts;
+ ts->probe_done = true;
+
+ input_err(true, &ts->client->dev, "%s: done\n", __func__);
+ input_log_fix();
+
+ return 0;
+
+ /* need to be enabled when new goto statement is added */
+/*
+ sec_ts_fn_remove(ts);
+ free_irq(client->irq, ts);
+*/
+err_register_drm_client:
+ free_irq(client->irq, ts);
+err_heatmap:
+ heatmap_remove(&ts->v4l2);
+err_irq:
+ pm_qos_remove_request(&ts->pm_qos_req);
+ if (ts->plat_data->support_dex) {
+ input_unregister_device(ts->input_dev_pad);
+ ts->input_dev_pad = NULL;
+ }
+err_input_pad_register_device:
+ input_unregister_device(ts->input_dev);
+ ts->input_dev = NULL;
+ ts->input_dev_touch = NULL;
+err_input_register_device:
+ kfree(ts->gainTable);
+err_allocate_gaintable:
+ kfree(ts->pFrame);
+err_allocate_frame:
+err_init:
+ sec_ts_power(ts, false);
+ if (ts->plat_data->support_dex) {
+ if (ts->input_dev_pad)
+ input_free_device(ts->input_dev_pad);
+ }
+err_allocate_input_dev_pad:
+ if (ts->input_dev)
+ input_free_device(ts->input_dev);
+err_allocate_input_dev:
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ tbn_cleanup(ts->tbn);
+err_init_tbn:
+#endif
+
+ if (ts->event_wq)
+ destroy_workqueue(ts->event_wq);
+error_alloc_workqueue:
+ kfree(ts);
+
+error_allocate_mem:
+ if (gpio_is_valid(pdata->irq_gpio))
+ gpio_free(pdata->irq_gpio);
+ if (gpio_is_valid(pdata->tsp_id))
+ gpio_free(pdata->tsp_id);
+ if (gpio_is_valid(pdata->tsp_icid))
+ gpio_free(pdata->tsp_icid);
+ if (gpio_is_valid(pdata->switch_gpio))
+ gpio_free(pdata->switch_gpio);
+ if (gpio_is_valid(pdata->reset_gpio))
+ gpio_free(pdata->reset_gpio);
+
+error_allocate_pdata:
+ if (ret == -ECONNREFUSED)
+ sec_ts_delay(100);
+ ret = -ENODEV;
+#ifdef CONFIG_TOUCHSCREEN_DUMP_MODE
+ p_ghost_check = NULL;
+#endif
+ ts_dup = NULL;
+ input_err(true, &client->dev, "%s: failed(%d)\n", __func__, ret);
+ input_log_fix();
+ return ret;
+}
+
+void sec_ts_unlocked_release_all_finger(struct sec_ts_data *ts)
+{
+ int i;
+
+ for (i = 0; i < MAX_SUPPORT_TOUCH_COUNT; i++) {
+ input_mt_slot(ts->input_dev, i);
+ input_mt_report_slot_state(ts->input_dev, MT_TOOL_FINGER, false);
+
+ if ((ts->coord[i].action == SEC_TS_COORDINATE_ACTION_PRESS) ||
+ (ts->coord[i].action == SEC_TS_COORDINATE_ACTION_MOVE)) {
+
+ ts->coord[i].action = SEC_TS_COORDINATE_ACTION_RELEASE;
+ input_info(true, &ts->client->dev,
+ "%s: [RA] tID:%d mc:%d tc:%d v:%02X%02X cal:%02X(%02X) id(%d,%d) p:%d\n",
+ __func__, i, ts->coord[i].mcount, ts->touch_count,
+ ts->plat_data->img_version_of_ic[2],
+ ts->plat_data->img_version_of_ic[3],
+ ts->cal_status, ts->nv, ts->tspid_val,
+ ts->tspicid_val, ts->coord[i].palm_count);
+
+ do_gettimeofday(&ts->time_released[i]);
+
+ if (ts->time_longest < (ts->time_released[i].tv_sec - ts->time_pressed[i].tv_sec))
+ ts->time_longest = (ts->time_released[i].tv_sec - ts->time_pressed[i].tv_sec);
+ }
+
+ ts->coord[i].mcount = 0;
+ ts->coord[i].palm_count = 0;
+
+ }
+
+ input_mt_slot(ts->input_dev, 0);
+
+ input_report_key(ts->input_dev, BTN_TOUCH, false);
+ input_report_key(ts->input_dev, BTN_TOOL_FINGER, false);
+ input_report_switch(ts->input_dev, SW_GLOVE, false);
+ ts->touchkey_glove_mode_status = false;
+ ts->touch_count = 0;
+ ts->check_multi = 0;
+
+ if (ts->plat_data->support_sidegesture) {
+ input_report_key(ts->input_dev, KEY_SIDE_GESTURE, 0);
+ input_report_key(ts->input_dev, KEY_SIDE_GESTURE_LEFT, 0);
+ input_report_key(ts->input_dev, KEY_SIDE_GESTURE_RIGHT, 0);
+ }
+
+ input_report_key(ts->input_dev, KEY_HOMEPAGE, 0);
+ input_event(ts->input_dev, EV_MSC, MSC_TIMESTAMP,
+ ts->timestamp / 1000);
+ input_sync(ts->input_dev);
+
+}
+
+void sec_ts_locked_release_all_finger(struct sec_ts_data *ts)
+{
+ int i;
+
+ mutex_lock(&ts->eventlock);
+
+ for (i = 0; i < MAX_SUPPORT_TOUCH_COUNT; i++) {
+ input_mt_slot(ts->input_dev, i);
+ input_mt_report_slot_state(ts->input_dev, MT_TOOL_FINGER, false);
+
+ if ((ts->coord[i].action == SEC_TS_COORDINATE_ACTION_PRESS) ||
+ (ts->coord[i].action == SEC_TS_COORDINATE_ACTION_MOVE)) {
+
+ ts->coord[i].action = SEC_TS_COORDINATE_ACTION_RELEASE;
+ input_info(true, &ts->client->dev,
+ "%s: [RA] tID:%d mc: %d tc:%d, v:%02X%02X, cal:%X(%X|%X), id(%d,%d), p:%d\n",
+ __func__, i, ts->coord[i].mcount, ts->touch_count,
+ ts->plat_data->img_version_of_ic[2],
+ ts->plat_data->img_version_of_ic[3],
+ ts->cal_status, ts->nv, ts->cal_count, ts->tspid_val,
+ ts->tspicid_val, ts->coord[i].palm_count);
+
+ do_gettimeofday(&ts->time_released[i]);
+
+ if (ts->time_longest < (ts->time_released[i].tv_sec - ts->time_pressed[i].tv_sec))
+ ts->time_longest = (ts->time_released[i].tv_sec - ts->time_pressed[i].tv_sec);
+ }
+
+ ts->coord[i].mcount = 0;
+ ts->coord[i].palm_count = 0;
+
+ }
+
+ input_mt_slot(ts->input_dev, 0);
+
+ input_report_key(ts->input_dev, BTN_TOUCH, false);
+ input_report_key(ts->input_dev, BTN_TOOL_FINGER, false);
+ input_report_switch(ts->input_dev, SW_GLOVE, false);
+ ts->touchkey_glove_mode_status = false;
+ ts->touch_count = 0;
+ ts->check_multi = 0;
+
+ if (ts->plat_data->support_sidegesture) {
+ input_report_key(ts->input_dev, KEY_SIDE_GESTURE, 0);
+ input_report_key(ts->input_dev, KEY_SIDE_GESTURE_LEFT, 0);
+ input_report_key(ts->input_dev, KEY_SIDE_GESTURE_RIGHT, 0);
+ }
+
+ input_report_key(ts->input_dev, KEY_HOMEPAGE, 0);
+ input_sync(ts->input_dev);
+
+ mutex_unlock(&ts->eventlock);
+
+}
+
+#ifdef USE_POWER_RESET_WORK
+static void sec_ts_reset_work(struct work_struct *work)
+{
+ struct sec_ts_data *ts = container_of(work, struct sec_ts_data,
+ reset_work.work);
+
+ ts->reset_is_on_going = true;
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_RESET, true);
+
+ sec_ts_stop_device(ts);
+
+ sec_ts_delay(30);
+
+ sec_ts_start_device(ts);
+
+ if (ts->input_dev_touch->disabled) {
+ input_err(true, &ts->client->dev , "%s: call input_close\n", __func__);
+
+ sec_ts_input_close(ts->input_dev);
+
+ if ((ts->lowpower_mode & SEC_TS_MODE_CUSTOMLIB_AOD) && ts->use_customlib) {
+ int i, ret;
+ u8 data[10] = {0x02, 0};
+
+ for (i = 0; i < 4; i++) {
+ data[i * 2 + 2] = ts->rect_data[i] & 0xFF;
+ data[i * 2 + 3] = (ts->rect_data[i] >> 8) & 0xFF;
+ }
+
+ disable_irq(ts->client->irq);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_WRITE_PARAM, &data[0], 10);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to write offset\n", __func__);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_NOTIFY_PACKET, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to send notify\n", __func__);
+ enable_irq(ts->client->irq);
+ }
+ }
+ ts->reset_is_on_going = false;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_RESET, false);
+}
+#endif
+
+static void sec_ts_read_info_work(struct work_struct *work)
+{
+ struct sec_ts_data *ts = container_of(work, struct sec_ts_data,
+ work_read_info.work);
+#ifndef CONFIG_SEC_FACTORY
+ struct sec_ts_test_mode mode;
+ char para = TO_TOUCH_MODE;
+#endif
+#ifdef USE_PRESSURE_SENSOR
+ int ret;
+ unsigned char data[18] = { 0 };
+#endif
+
+ ts->nv = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_FAC_RESULT);
+ ts->cal_count = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+ ts->pressure_cal_base = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_PRESSURE_BASE_CAL_COUNT);
+ ts->pressure_cal_delta = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_PRESSURE_DELTA_CAL_COUNT);
+
+ input_info(true, &ts->client->dev, "%s: fac_nv:%02X, cal_count:%02X\n", __func__, ts->nv, ts->cal_count);
+
+#ifdef PAT_CONTROL
+ ts->tune_fix_ver = (get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION) << 8) | get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION+1);
+ input_info(true, &ts->client->dev, "%s: tune_fix_ver [%04X]\n", __func__, ts->tune_fix_ver);
+#endif
+
+#ifdef USE_PRESSURE_SENSOR
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_SET_GET_PRESSURE, data, 18);
+ if (ret < 0)
+ return;
+
+ ts->pressure_left = ((data[16] << 8) | data[17]);
+ ts->pressure_center = ((data[8] << 8) | data[9]);
+ ts->pressure_right = ((data[0] << 8) | data[1]);
+ input_info(true, &ts->client->dev, "%s: left: %d, center: %d, right: %d\n",
+ __func__, ts->pressure_left, ts->pressure_center, ts->pressure_right);
+#endif
+
+#ifndef CONFIG_SEC_FACTORY
+ /* run self-test */
+ disable_irq(ts->client->irq);
+ execute_selftest(ts, false);
+ enable_irq(ts->client->irq);
+
+ input_info(true, &ts->client->dev, "%s: %02X %02X %02X %02X\n",
+ __func__, ts->ito_test[0], ts->ito_test[1]
+ , ts->ito_test[2], ts->ito_test[3]);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_POWER_MODE, ¶, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to set\n", __func__);
+
+ sec_ts_delay(350);
+
+ /* run ambient read */
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_AMBIENT_DATA;
+ mode.allnode = TEST_MODE_ALL_NODE;
+
+ sec_ts_read_raw_data(ts, NULL, &mode);
+#endif
+
+ input_log_fix();
+}
+
+static void sec_ts_fw_update_work(struct work_struct *work)
+{
+ struct sec_ts_data *ts = container_of(work, struct sec_ts_data,
+ work_fw_update);
+ int ret;
+
+ input_info(true, &ts->client->dev,
+ "%s: Beginning firmware update after probe.\n", __func__);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_FW_UPDATE, true);
+
+ ret = sec_ts_firmware_update_on_probe(ts, false);
+ if (ret < 0)
+ input_info(true, &ts->client->dev,
+ "%s: firmware update was unsuccessful.\n",
+ __func__);
+
+ if (ts->is_fw_corrupted == true && ret == 0) {
+ ret = sec_ts_fw_init(ts);
+ if (ret == SEC_TS_ERR_NA) {
+ ts->is_fw_corrupted = false;
+ sec_ts_device_init(ts);
+ sec_ts_read_info_work(&ts->work_read_info.work);
+ } else
+ input_info(true, &ts->client->dev,
+ "%s: fail to sec_ts_fw_init 0x%x\n",
+ __func__, ret);
+ }
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_FW_UPDATE, false);
+}
+
+int sec_ts_set_lowpowermode(struct sec_ts_data *ts, u8 mode)
+{
+ int ret;
+ int retrycnt = 0;
+ u8 data;
+ char para = 0;
+
+ input_err(true, &ts->client->dev, "%s: %s(%X)\n", __func__,
+ mode == TO_LOWPOWER_MODE ? "ENTER" : "EXIT", ts->lowpower_mode);
+
+ if (mode) {
+ #ifdef SEC_TS_SUPPORT_CUSTOMLIB
+ if (ts->use_customlib)
+ sec_ts_set_custom_library(ts);
+ #endif
+
+ data = (ts->lowpower_mode & SEC_TS_MODE_LOWPOWER_FLAG) >> 1;
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_WAKEUP_GESTURE_MODE, &data, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to set\n", __func__);
+ }
+
+retry_pmode:
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_POWER_MODE, &mode, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed\n", __func__);
+ sec_ts_delay(50);
+
+ /* read data */
+
+ ret = sec_ts_i2c_read(ts, SEC_TS_CMD_SET_POWER_MODE, ¶, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: read power mode failed!\n", __func__);
+ else
+ input_info(true, &ts->client->dev, "%s: power mode - write(%d) read(%d)\n", __func__, mode, para);
+
+ if (mode != para) {
+ retrycnt++;
+ if (retrycnt < 5)
+ goto retry_pmode;
+ }
+
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_CLEAR_EVENT_STACK, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: i2c write clear event failed\n", __func__);
+
+
+ sec_ts_locked_release_all_finger(ts);
+
+ if (device_may_wakeup(&ts->client->dev)) {
+ if (mode)
+ enable_irq_wake(ts->client->irq);
+ else
+ disable_irq_wake(ts->client->irq);
+ }
+
+ ts->lowpower_status = mode;
+ input_info(true, &ts->client->dev, "%s: end\n", __func__);
+
+ return ret;
+}
+
+#ifdef USE_OPEN_CLOSE
+static int sec_ts_input_open(struct input_dev *dev)
+{
+ struct sec_ts_data *ts = input_get_drvdata(dev);
+ int ret;
+
+ ts->input_closed = false;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_INPUT_DEV, true);
+
+ if (ts->lowpower_status) {
+#ifdef USE_RESET_EXIT_LPM
+ schedule_delayed_work(&ts->reset_work, msecs_to_jiffies(TOUCH_RESET_DWORK_TIME));
+#else
+ sec_ts_set_lowpowermode(ts, TO_TOUCH_MODE);
+#endif
+ ts->power_status = SEC_TS_STATE_POWER_ON;
+ } else {
+ ret = sec_ts_start_device(ts);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Failed to start device\n", __func__);
+ }
+
+ /* because edge and dead zone will recover soon */
+ sec_ts_set_grip_type(ts, ONLY_EDGE_HANDLER);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_INPUT_DEV, false);
+
+ return 0;
+}
+
+static void sec_ts_input_close(struct input_dev *dev)
+{
+ struct sec_ts_data *ts = input_get_drvdata(dev);
+
+ ts->input_closed = true;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_INPUT_DEV, true);
+
+ cancel_work_sync(&ts->suspend_work);
+ cancel_work_sync(&ts->resume_work);
+
+#ifdef USE_POWER_RESET_WORK
+ cancel_delayed_work(&ts->reset_work);
+#endif
+
+#ifndef CONFIG_SEC_FACTORY
+ ts->lowpower_mode |= SEC_TS_MODE_CUSTOMLIB_FORCE_KEY;
+#endif
+ if (ts->lowpower_mode) {
+ sec_ts_set_lowpowermode(ts, TO_LOWPOWER_MODE);
+ ts->power_status = SEC_TS_STATE_LPM;
+ } else {
+ sec_ts_stop_device(ts);
+ }
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_INPUT_DEV, false);
+}
+#endif
+
+static int sec_ts_remove(struct i2c_client *client)
+{
+ struct sec_ts_data *ts = i2c_get_clientdata(client);
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ /* Force the bus active throughout removal of the client */
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_FORCE_ACTIVE, true);
+
+ msm_drm_unregister_client(&ts->notifier);
+
+ cancel_work_sync(&ts->suspend_work);
+ cancel_work_sync(&ts->resume_work);
+ destroy_workqueue(ts->event_wq);
+
+#ifdef SEC_TS_FW_UPDATE_ON_PROBE
+ cancel_work_sync(&ts->work_fw_update);
+#endif
+
+ cancel_delayed_work_sync(&ts->work_read_info);
+ flush_delayed_work(&ts->work_read_info);
+
+ disable_irq_nosync(ts->client->irq);
+ free_irq(ts->client->irq, ts);
+ input_info(true, &ts->client->dev, "%s: irq disabled\n", __func__);
+
+ heatmap_remove(&ts->v4l2);
+
+ pm_qos_remove_request(&ts->pm_qos_req);
+
+#ifdef USE_POWER_RESET_WORK
+ cancel_delayed_work_sync(&ts->reset_work);
+ flush_delayed_work(&ts->reset_work);
+
+ input_info(true, &ts->client->dev, "%s: flush queue\n", __func__);
+
+#endif
+
+ sec_ts_fn_remove(ts);
+
+#ifdef CONFIG_TOUCHSCREEN_DUMP_MODE
+ p_ghost_check = NULL;
+#endif
+ device_init_wakeup(&client->dev, false);
+
+ ts->lowpower_mode = false;
+ ts->probe_done = false;
+
+ if (ts->plat_data->support_dex) {
+ input_mt_destroy_slots(ts->input_dev_pad);
+ input_unregister_device(ts->input_dev_pad);
+ }
+
+ ts->input_dev = ts->input_dev_touch;
+ input_mt_destroy_slots(ts->input_dev);
+ input_unregister_device(ts->input_dev);
+
+ ts->input_dev_pad = NULL;
+ ts->input_dev = NULL;
+ ts->input_dev_touch = NULL;
+ ts_dup = NULL;
+
+ /* need to do software reset for next sec_ts_probe() without error */
+ ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SW_RESET, NULL, 0);
+
+ ts->plat_data->power(ts, false);
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ tbn_cleanup(ts->tbn);
+#endif
+
+ if (gpio_is_valid(ts->plat_data->irq_gpio))
+ gpio_free(ts->plat_data->irq_gpio);
+ if (gpio_is_valid(ts->plat_data->switch_gpio))
+ gpio_free(ts->plat_data->switch_gpio);
+ if (gpio_is_valid(ts->plat_data->reset_gpio))
+ gpio_free(ts->plat_data->reset_gpio);
+
+ sec_ts_raw_device_exit(ts);
+#ifndef CONFIG_SEC_SYSFS
+ class_destroy(sec_class);
+#endif
+
+ kfree(ts->gainTable);
+ kfree(ts->pFrame);
+ kfree(ts);
+ return 0;
+}
+
+static void sec_ts_shutdown(struct i2c_client *client)
+{
+ struct sec_ts_data *ts = i2c_get_clientdata(client);
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ sec_ts_remove(client);
+}
+
+int sec_ts_stop_device(struct sec_ts_data *ts)
+{
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ mutex_lock(&ts->device_mutex);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: already power off\n", __func__);
+ goto out;
+ }
+
+ ts->power_status = SEC_TS_STATE_POWER_OFF;
+
+ disable_irq(ts->client->irq);
+ sec_ts_locked_release_all_finger(ts);
+
+ ts->plat_data->power(ts, false);
+
+ if (ts->plat_data->enable_sync)
+ ts->plat_data->enable_sync(false);
+
+ sec_ts_pinctrl_configure(ts, false);
+
+out:
+ mutex_unlock(&ts->device_mutex);
+ return 0;
+}
+
+int sec_ts_start_device(struct sec_ts_data *ts)
+{
+ int ret;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ sec_ts_pinctrl_configure(ts, true);
+
+ mutex_lock(&ts->device_mutex);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_ON) {
+ input_err(true, &ts->client->dev, "%s: already power on\n", __func__);
+ goto out;
+ }
+
+ sec_ts_locked_release_all_finger(ts);
+
+ ts->plat_data->power(ts, true);
+ sec_ts_delay(70);
+ ts->power_status = SEC_TS_STATE_POWER_ON;
+ sec_ts_wait_for_ready(ts, SEC_TS_ACK_BOOT_COMPLETE);
+
+ if (ts->plat_data->enable_sync)
+ ts->plat_data->enable_sync(true);
+
+ if (ts->flip_enable) {
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_COVERTYPE, &ts->cover_cmd, 1);
+
+ ts->touch_functions = ts->touch_functions | SEC_TS_BIT_SETFUNC_COVER;
+ input_info(true, &ts->client->dev,
+ "%s: cover cmd write type:%d, mode:%x, ret:%d", __func__, ts->touch_functions, ts->cover_cmd, ret);
+ } else {
+ ts->touch_functions = (ts->touch_functions & (~SEC_TS_BIT_SETFUNC_COVER));
+ input_info(true, &ts->client->dev,
+ "%s: cover open, not send cmd", __func__);
+ }
+
+ ts->touch_functions = ts->touch_functions | SEC_TS_DEFAULT_ENABLE_BIT_SETFUNC;
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCHFUNCTION, (u8 *)&ts->touch_functions, 2);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: Failed to send touch function command", __func__);
+
+ #ifdef SEC_TS_SUPPORT_CUSTOMLIB
+ if (ts->use_customlib)
+ sec_ts_set_custom_library(ts);
+ #endif
+
+ sec_ts_set_grip_type(ts, ONLY_EDGE_HANDLER);
+
+ if (ts->dex_mode) {
+ input_info(true, &ts->client->dev, "%s: set dex mode\n", __func__);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_DEX_MODE, &ts->dex_mode, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to set dex mode %x\n", __func__, ts->dex_mode);
+ }
+
+ if (ts->brush_mode) {
+ input_info(true, &ts->client->dev, "%s: set brush mode\n", __func__);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_BRUSH_MODE, &ts->brush_mode, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to set brush mode\n", __func__);
+ }
+
+ if (ts->touchable_area) {
+ input_info(true, &ts->client->dev, "%s: set 16:9 mode\n", __func__);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCHABLE_AREA, &ts->touchable_area, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to set 16:9 mode\n", __func__);
+ }
+
+ /* Sense_on */
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_ON, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to write Sense_on\n", __func__);
+
+ enable_irq(ts->client->irq);
+
+out:
+ mutex_unlock(&ts->device_mutex);
+ return 0;
+}
+
+#ifdef CONFIG_PM
+static int sec_ts_pm_suspend(struct device *dev)
+{
+ struct sec_ts_data *ts = dev_get_drvdata(dev);
+
+ if (ts->lowpower_mode)
+ reinit_completion(&ts->resume_done);
+
+ return 0;
+}
+
+static int sec_ts_pm_resume(struct device *dev)
+{
+ struct sec_ts_data *ts = dev_get_drvdata(dev);
+
+ if (ts->lowpower_mode)
+ complete_all(&ts->resume_done);
+
+ return 0;
+}
+#endif
+
+static const struct i2c_device_id sec_ts_id[] = {
+ { SEC_TS_I2C_NAME, 0 },
+ { },
+};
+
+#ifdef CONFIG_PM
+static const struct dev_pm_ops sec_ts_dev_pm_ops = {
+ .suspend = sec_ts_pm_suspend,
+ .resume = sec_ts_pm_resume,
+};
+#endif
+
+/*
+ * Configure the switch GPIO to toggle bus master between AP and SLPI.
+ * gpio_value takes one of
+ * { SEC_SWITCH_GPIO_VALUE_SLPI_MASTER, SEC_SWITCH_GPIO_VALUE_AP_MASTER }
+ */
+static void sec_set_switch_gpio(struct sec_ts_data *ts, int gpio_value)
+{
+ int retval;
+ unsigned int gpio = ts->plat_data->switch_gpio;
+
+ if (!gpio_is_valid(gpio))
+ return;
+
+ input_info(true, &ts->client->dev, "%s: toggling i2c switch to %s\n",
+ __func__, gpio_value == SEC_SWITCH_GPIO_VALUE_AP_MASTER ?
+ "AP" : "SLPI");
+
+ retval = gpio_direction_output(gpio, gpio_value);
+ if (retval < 0)
+ input_err(true, &ts->client->dev,
+ "%s: Failed to toggle switch_gpio, err = %d\n",
+ __func__, retval);
+}
+
+static void sec_ts_suspend_work(struct work_struct *work)
+{
+ struct sec_ts_data *ts = container_of(work, struct sec_ts_data,
+ suspend_work);
+ int ret = 0;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ mutex_lock(&ts->device_mutex);
+
+ reinit_completion(&ts->bus_resumed);
+
+ if (ts->power_status == SEC_TS_STATE_SUSPEND) {
+ input_err(true, &ts->client->dev, "%s: already suspended.\n",
+ __func__);
+ mutex_unlock(&ts->device_mutex);
+ return;
+ }
+
+ /* Sense_off */
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_OFF, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to write Sense_off.\n", __func__);
+
+ disable_irq_nosync(ts->client->irq);
+ sec_ts_locked_release_all_finger(ts);
+
+ if (ts->plat_data->enable_sync)
+ ts->plat_data->enable_sync(false);
+
+ ts->power_status = SEC_TS_STATE_SUSPEND;
+
+ sec_ts_pinctrl_configure(ts, false);
+
+ sec_set_switch_gpio(ts, SEC_SWITCH_GPIO_VALUE_SLPI_MASTER);
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ if (ts->tbn)
+ tbn_release_bus(ts->tbn);
+#endif
+
+ mutex_unlock(&ts->device_mutex);
+}
+
+static void sec_ts_resume_work(struct work_struct *work)
+{
+ struct sec_ts_data *ts = container_of(work, struct sec_ts_data,
+ resume_work);
+ int ret = 0;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ mutex_lock(&ts->device_mutex);
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ if (ts->tbn)
+ tbn_request_bus(ts->tbn);
+#endif
+
+ sec_set_switch_gpio(ts, SEC_SWITCH_GPIO_VALUE_AP_MASTER);
+
+ sec_ts_pinctrl_configure(ts, true);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_ON) {
+ input_err(true, &ts->client->dev, "%s: already resumed.\n",
+ __func__);
+ mutex_unlock(&ts->device_mutex);
+ return;
+ }
+
+ sec_ts_locked_release_all_finger(ts);
+
+ ts->power_status = SEC_TS_STATE_POWER_ON;
+
+ ret = sec_ts_system_reset(ts);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: reset failed! ret %d\n", __func__, ret);
+
+ if (ts->plat_data->enable_sync)
+ ts->plat_data->enable_sync(true);
+
+ ts->touch_functions =
+ ts->touch_functions | SEC_TS_DEFAULT_ENABLE_BIT_SETFUNC;
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCHFUNCTION,
+ (u8 *)&ts->touch_functions, 2);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: Failed to send touch function command.",
+ __func__);
+
+#ifdef SEC_TS_SUPPORT_CUSTOMLIB
+ if (ts->use_customlib)
+ sec_ts_set_custom_library(ts);
+#endif
+
+ sec_ts_set_grip_type(ts, ONLY_EDGE_HANDLER);
+
+ if (ts->dex_mode) {
+ input_info(true, &ts->client->dev, "%s: set dex mode.\n",
+ __func__);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_DEX_MODE,
+ &ts->dex_mode, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to set dex mode %x.\n", __func__,
+ ts->dex_mode);
+ }
+
+ if (ts->brush_mode) {
+ input_info(true, &ts->client->dev, "%s: set brush mode.\n",
+ __func__);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_BRUSH_MODE,
+ &ts->brush_mode, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to set brush mode.\n", __func__);
+ }
+
+ if (ts->touchable_area) {
+ input_info(true, &ts->client->dev, "%s: set 16:9 mode.\n",
+ __func__);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCHABLE_AREA,
+ &ts->touchable_area, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to set 16:9 mode.\n", __func__);
+ }
+
+ /* Sense_on */
+ ret = sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_ON, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed to write Sense_on.\n", __func__);
+
+ enable_irq(ts->client->irq);
+
+ complete_all(&ts->bus_resumed);
+
+ mutex_unlock(&ts->device_mutex);
+}
+
+static void sec_ts_aggregate_bus_state(struct sec_ts_data *ts)
+{
+ input_dbg(true, &ts->client->dev, "%s: bus_refmask = 0x%02X.\n",
+ __func__, ts->bus_refmask);
+
+ /* Complete or cancel any outstanding transitions */
+ cancel_work_sync(&ts->suspend_work);
+ cancel_work_sync(&ts->resume_work);
+
+ if ((ts->bus_refmask == 0 &&
+ ts->power_status == SEC_TS_STATE_SUSPEND) ||
+ (ts->bus_refmask != 0 &&
+ ts->power_status != SEC_TS_STATE_SUSPEND))
+ return;
+
+ if (ts->bus_refmask == 0)
+ queue_work(ts->event_wq, &ts->suspend_work);
+ else
+ queue_work(ts->event_wq, &ts->resume_work);
+}
+
+int sec_ts_set_bus_ref(struct sec_ts_data *ts, u16 ref, bool enable)
+{
+ int result = 0;
+
+ mutex_lock(&ts->bus_mutex);
+
+ input_dbg(true, &ts->client->dev, "%s: bus_refmask = 0x%02X.\n",
+ __func__, ref);
+
+ if ((enable && (ts->bus_refmask & ref)) ||
+ (!enable && !(ts->bus_refmask & ref))) {
+ input_info(true, &ts->client->dev,
+ "%s: reference is unexpectedly set: mask=0x%04X, ref=0x%04X, enable=%d.\n",
+ __func__, ts->bus_refmask, ref, enable);
+ mutex_unlock(&ts->bus_mutex);
+ return -EINVAL;
+ }
+
+ if (enable) {
+ /* IRQs can only keep the bus active. IRQs received while the
+ * bus is transferred to SLPI should be ignored.
+ */
+ if (ref == SEC_TS_BUS_REF_IRQ && ts->bus_refmask == 0)
+ result = -EAGAIN;
+ else
+ ts->bus_refmask |= ref;
+ } else
+ ts->bus_refmask &= ~ref;
+ sec_ts_aggregate_bus_state(ts);
+
+ mutex_unlock(&ts->bus_mutex);
+
+ /* When triggering a wake, wait up to one second to resume. SCREEN_ON
+ * and IRQ references do not need to wait.
+ */
+ if (enable &&
+ ref != SEC_TS_BUS_REF_SCREEN_ON && ref != SEC_TS_BUS_REF_IRQ) {
+ wait_for_completion_timeout(&ts->bus_resumed, HZ);
+ if (ts->power_status != SEC_TS_STATE_POWER_ON) {
+ input_info(true, &ts->client->dev,
+ "%s: Failed to wake the touch bus.\n",
+ __func__);
+ result = -ETIMEDOUT;
+ }
+ }
+
+ return result;
+}
+
+static int sec_ts_screen_state_chg_callback(struct notifier_block *nb,
+ unsigned long val, void *data)
+{
+ struct sec_ts_data *ts = container_of(nb, struct sec_ts_data,
+ notifier);
+ struct msm_drm_notifier *evdata = (struct msm_drm_notifier *)data;
+ unsigned int blank;
+
+ input_dbg(true, &ts->client->dev, "%s: enter.\n", __func__);
+
+ if (val != MSM_DRM_EVENT_BLANK && val != MSM_DRM_EARLY_EVENT_BLANK)
+ return NOTIFY_DONE;
+
+ if (!ts || !evdata || !evdata->data) {
+ input_err(true, &ts->client->dev,
+ "%s: Bad screen state change notifier call.\n",
+ __func__);
+ return NOTIFY_DONE;
+ }
+
+ /* finish processing any events on queue */
+ flush_workqueue(ts->event_wq);
+
+ blank = *((unsigned int *)evdata->data);
+ switch (blank) {
+ case MSM_DRM_BLANK_POWERDOWN:
+ case MSM_DRM_BLANK_LP:
+ if (val == MSM_DRM_EARLY_EVENT_BLANK) {
+ input_dbg(true, &ts->client->dev,
+ "%s: MSM_DRM_BLANK_POWERDOWN.\n", __func__);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SCREEN_ON, false);
+ }
+ break;
+ case MSM_DRM_BLANK_UNBLANK:
+ if (val == MSM_DRM_EVENT_BLANK) {
+ input_dbg(true, &ts->client->dev,
+ "%s: MSM_DRM_BLANK_UNBLANK.\n", __func__);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SCREEN_ON, true);
+ }
+ break;
+ }
+
+ return NOTIFY_OK;
+}
+
+static struct notifier_block sec_ts_screen_nb = {
+ .notifier_call = sec_ts_screen_state_chg_callback,
+};
+
+#ifdef CONFIG_OF
+static const struct of_device_id sec_ts_match_table[] = {
+ { .compatible = "sec,sec_ts",},
+ { },
+};
+#else
+#define sec_ts_match_table NULL
+#endif
+
+static struct i2c_driver sec_ts_driver = {
+ .probe = sec_ts_probe,
+ .remove = sec_ts_remove,
+ .shutdown = sec_ts_shutdown,
+ .id_table = sec_ts_id,
+ .driver = {
+ .owner = THIS_MODULE,
+ .name = SEC_TS_I2C_NAME,
+#ifdef CONFIG_OF
+ .of_match_table = sec_ts_match_table,
+#endif
+#ifdef CONFIG_PM
+ .pm = &sec_ts_dev_pm_ops,
+#endif
+ },
+};
+
+static int __init sec_ts_init(void)
+{
+#ifdef CONFIG_BATTERY_SAMSUNG
+ if (lpcharge == 1) {
+ pr_err("%s %s: Do not load driver due to : lpm %d\n",
+ SECLOG, __func__, lpcharge);
+ return -ENODEV;
+ }
+#endif
+ pr_err("%s %s\n", SECLOG, __func__);
+
+ return i2c_add_driver(&sec_ts_driver);
+}
+
+static void __exit sec_ts_exit(void)
+{
+ i2c_del_driver(&sec_ts_driver);
+}
+
+MODULE_AUTHOR("Hyobae, Ahn<hyobae.ahn@samsung.com>");
+MODULE_DESCRIPTION("Samsung Electronics TouchScreen driver");
+MODULE_LICENSE("GPL");
+
+module_init(sec_ts_init);
+module_exit(sec_ts_exit);
diff --git a/drivers/input/touchscreen/sec_ts/sec_ts.h b/drivers/input/touchscreen/sec_ts/sec_ts.h
new file mode 100644
index 0000000..1117593
--- /dev/null
+++ b/drivers/input/touchscreen/sec_ts/sec_ts.h
@@ -0,0 +1,981 @@
+/* drivers/input/touchscreen/sec_ts.h
+ *
+ * Copyright (C) 2015 Samsung Electronics Co., Ltd.
+ * http://www.samsungsemi.com/
+ *
+ * Core file for Samsung TSC driver
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __SEC_TS_H__
+#define __SEC_TS_H__
+
+#include <asm/unaligned.h>
+#include <linux/completion.h>
+#include <linux/ctype.h>
+#include <linux/delay.h>
+#include <linux/firmware.h>
+#include <linux/gpio.h>
+#include <linux/hrtimer.h>
+#include <linux/i2c.h>
+#include <linux/input.h>
+#include <linux/input/heatmap.h>
+#include <linux/input/mt.h>
+#include <linux/input/sec_cmd.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/irq.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/msm_drm_notify.h>
+#include <linux/notifier.h>
+#include <linux/of_gpio.h>
+#include <linux/platform_device.h>
+#include <linux/pm_qos.h>
+#include <linux/regulator/consumer.h>
+#include <linux/slab.h>
+#include <linux/time.h>
+#include <linux/uaccess.h>
+#include <linux/vmalloc.h>
+#include <linux/workqueue.h>
+
+#ifdef CONFIG_SEC_SYSFS
+#include <linux/sec_sysfs.h>
+#endif
+
+#ifdef CONFIG_INPUT_BOOSTER
+#include <linux/input/input_booster.h>
+#endif
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+#include "../touch_bus_negotiator.h"
+#endif
+
+#define SEC_TS_I2C_NAME "sec_ts"
+#define SEC_TS_DEVICE_NAME "SEC_TS"
+
+#define USE_OPEN_CLOSE
+#undef USE_RESET_DURING_POWER_ON
+#undef USE_RESET_EXIT_LPM
+#undef USE_POR_AFTER_I2C_RETRY
+#undef USER_OPEN_DWORK
+#define USE_PRESSURE_SENSOR
+#define PAT_CONTROL
+
+#if defined(USE_RESET_DURING_POWER_ON) || defined(USE_POR_AFTER_I2C_RETRY) || defined(USE_RESET_EXIT_LPM)
+#define USE_POWER_RESET_WORK
+#endif
+
+#define TOUCH_RESET_DWORK_TIME 10
+#define BRUSH_Z_DATA 63 /* for ArtCanvas */
+
+#define MASK_1_BITS 0x0001
+#define MASK_2_BITS 0x0003
+#define MASK_3_BITS 0x0007
+#define MASK_4_BITS 0x000F
+#define MASK_5_BITS 0x001F
+#define MASK_6_BITS 0x003F
+#define MASK_7_BITS 0x007F
+#define MASK_8_BITS 0x00FF
+
+/* support feature */
+//#define SEC_TS_SUPPORT_CUSTOMLIB /* support user defined library */
+
+#define TYPE_STATUS_EVENT_CMD_DRIVEN 0
+#define TYPE_STATUS_EVENT_ERR 1
+#define TYPE_STATUS_EVENT_INFO 2
+#define TYPE_STATUS_EVENT_USER_INPUT 3
+#define TYPE_STATUS_EVENT_CUSTOMLIB_INFO 6
+#define TYPE_STATUS_EVENT_VENDOR_INFO 7
+#define TYPE_STATUS_CODE_SAR 0x28
+
+#define BIT_STATUS_EVENT_CMD_DRIVEN(a) (a << TYPE_STATUS_EVENT_CMD_DRIVEN)
+#define BIT_STATUS_EVENT_ERR(a) (a << TYPE_STATUS_EVENT_ERR)
+#define BIT_STATUS_EVENT_INFO(a) (a << TYPE_STATUS_EVENT_INFO)
+#define BIT_STATUS_EVENT_USER_INPUT(a) (a << TYPE_STATUS_EVENT_USER_INPUT)
+#define BIT_STATUS_EVENT_VENDOR_INFO(a) (a << TYPE_STATUS_EVENT_VENDOR_INFO)
+
+#define DO_FW_CHECKSUM (1 << 0)
+#define DO_PARA_CHECKSUM (1 << 1)
+#define MAX_SUPPORT_TOUCH_COUNT 10
+#define MAX_SUPPORT_HOVER_COUNT 1
+
+#define SEC_TS_EVENTID_HOVER 10
+
+#define SEC_TS_DEFAULT_FW_NAME "tsp_sec/sec_hero.fw"
+#define SEC_TS_DEFAULT_BL_NAME "tsp_sec/s6smc41_blupdate_img_REL.bin"
+#define SEC_TS_DEFAULT_PARA_NAME "tsp_sec/s6smc41_para_REL_DGA0_V0106_150114_193317.bin"
+#define SEC_TS_DEFAULT_UMS_FW "/sdcard/Firmware/TSP/lsi.bin"
+#define SEC_TS_DEFAULT_FFU_FW "ffu_tsp.bin"
+#define SEC_TS_MAX_FW_PATH 64
+#define SEC_TS_FW_BLK_SIZE_MAX (512)
+#define SEC_TS_FW_BLK_SIZE_DEFAULT (256)
+#define SEC_TS_SELFTEST_REPORT_SIZE 80
+#define SEC_TS_PRESSURE_MAX 0x3f
+
+#define I2C_WRITE_BUFFER_SIZE (256 - 1)//10
+
+/* max read size: from sec_ts_read_event() at sec_ts.c */
+#define I2C_PREALLOC_READ_BUF_SZ (32 * SEC_TS_EVENT_BUFF_SIZE)
+/* max write size: from sec_ts_flashpagewrite() at sec_ts_fw.c */
+#define I2C_PREALLOC_WRITE_BUF_SZ (1 + 2 + SEC_TS_FW_BLK_SIZE_MAX + 1)
+
+#define SEC_TS_FW_HEADER_SIGN 0x53494654
+#define SEC_TS_FW_CHUNK_SIGN 0x53434654
+
+#define SEC_TS_FW_UPDATE_ON_PROBE
+
+#define AMBIENT_CAL 0
+#define OFFSET_CAL_SDC 1
+#define OFFSET_CAL_SEC 2
+#define PRESSURE_CAL 3
+
+#define SEC_TS_SKIPTSP_DUTY 100
+
+#define SEC_TS_NVM_OFFSET_FAC_RESULT 0
+#define SEC_TS_NVM_OFFSET_CAL_COUNT 1
+#define SEC_TS_NVM_OFFSET_DISASSEMBLE_COUNT 2
+#define SEC_TS_NVM_OFFSET_TUNE_VERSION 3
+#define SEC_TS_NVM_OFFSET_TUNE_VERSION_LENGTH 2
+
+#define SEC_TS_NVM_OFFSET_PRESSURE_INDEX 5
+
+#define SEC_TS_NVM_OFFSET_PRESSURE_STRENGTH 6
+#define SEC_TS_NVM_OFFSET_PRESSURE_STRENGTH_1 6
+#define SEC_TS_NVM_OFFSET_PRESSURE_STRENGTH_2 12
+#define SEC_TS_NVM_OFFSET_PRESSURE_STRENGTH_3 18
+#define SEC_TS_NVM_OFFSET_PRESSURE_STRENGTH_4 24
+
+#define SEC_TS_NVM_OFFSET_PRESSURE_RAWDATA 30
+#define SEC_TS_NVM_OFFSET_PRESSURE_RAWDATA_1 30
+#define SEC_TS_NVM_OFFSET_PRESSURE_RAWDATA_2 36
+#define SEC_TS_NVM_OFFSET_PRESSURE_RAWDATA_3 42
+#define SEC_TS_NVM_OFFSET_PRESSURE_RAWDATA_4 48
+#define SEC_TS_NVM_SIZE_PRESSURE_BLOCK 6
+
+#define SEC_TS_NVM_OFFSET_PRESSURE_BASE_CAL_COUNT 54
+#define SEC_TS_NVM_OFFSET_PRESSURE_DELTA_CAL_COUNT 55
+#define SEC_TS_NVM_SIZE_PRESSURE_CAL_BLOCK 1
+
+#define SEC_TS_NVM_LAST_BLOCK_OFFSET SEC_TS_NVM_OFFSET_PRESSURE_DELTA_CAL_COUNT
+#define SEC_TS_NVM_LAST_BLOCK_SIZE SEC_TS_NVM_SIZE_PRESSURE_CAL_BLOCK
+
+#define SEC_TS_NVM_OFFSET_LENGTH (SEC_TS_NVM_LAST_BLOCK_OFFSET + SEC_TS_NVM_LAST_BLOCK_SIZE + 1)
+
+/* SEC_TS READ REGISTER ADDRESS */
+#define SEC_TS_CMD_SENSE_ON 0x10
+#define SEC_TS_CMD_SENSE_OFF 0x11
+#define SEC_TS_CMD_SW_RESET 0x12
+#define SEC_TS_CMD_CALIBRATION_SEC 0x13 // send it to touch ic, but toucu ic works nothing.
+#define SEC_TS_CMD_FACTORY_PANELCALIBRATION 0x14
+
+#define SEC_TS_READ_GPIO_STATUS 0x20 // not support
+#define SEC_TS_READ_FIRMWARE_INTEGRITY 0x21
+#define SEC_TS_READ_DEVICE_ID 0x22
+#define SEC_TS_READ_PANEL_INFO 0x23
+#define SEC_TS_READ_CORE_CONFIG_VERSION 0x24
+#define SEC_TS_CMD_DISABLE_GAIN_LIMIT 0x2A
+
+#define SEC_TS_CMD_SET_TOUCHFUNCTION 0x30
+#define SEC_TS_CMD_SET_TSC_MODE 0x31
+#define SET_TS_CMD_SET_CHARGER_MODE 0x32
+#define SET_TS_CMD_SET_NOISE_MODE 0x33
+#define SET_TS_CMD_SET_REPORT_RATE 0x34
+#define SEC_TS_CMD_TOUCH_MODE_FOR_THRESHOLD 0x35
+#define SEC_TS_CMD_TOUCH_THRESHOLD 0x36
+#define SET_TS_CMD_KEY_THRESHOLD 0x37
+#define SEC_TS_CMD_SET_COVERTYPE 0x38
+#define SEC_TS_CMD_WAKEUP_GESTURE_MODE 0x39
+#define SEC_TS_WRITE_POSITION_FILTER 0x3A
+#define SEC_TS_CMD_WET_MODE 0x3B
+#define SEC_TS_CMD_DISABLE_NORM_TABLE 0x40
+#define SEC_TS_CMD_READ_NORM_TABLE 0x41
+#define SEC_TS_CMD_DISABLE_BASELINE_ADAPT 0x43
+#define SEC_TS_CMD_DISABLE_DF 0x44
+#define SEC_TS_CMD_ERASE_FLASH 0x45
+#define SEC_TS_CMD_RESET_BASELINE 0x47
+#define SEC_TS_CMD_WRITE_NORM_TABLE 0x49
+#define SEC_TS_CMD_HEATMAP_READ 0x4A
+#define SEC_TS_CMD_HEATMAP_ENABLE 0x4B
+#define SEC_TS_READ_ID 0x52
+#define SEC_TS_READ_BOOT_STATUS 0x55
+#define SEC_TS_CMD_ENTER_FW_MODE 0x57
+#define SEC_TS_READ_ONE_EVENT 0x60
+#define SEC_TS_READ_ALL_EVENT 0x61
+#define SEC_TS_CMD_CLEAR_EVENT_STACK 0x62
+#define SEC_TS_CMD_MUTU_RAW_TYPE 0x70
+#define SEC_TS_CMD_SELF_RAW_TYPE 0x71
+#define SEC_TS_READ_TOUCH_RAWDATA 0x72
+#define SEC_TS_READ_TOUCH_SELF_RAWDATA 0x73
+#define SEC_TS_READ_SELFTEST_RESULT 0x80
+#define SEC_TS_CMD_CALIBRATION_AMBIENT 0x81
+#define SEC_TS_CMD_P2PTEST 0x82
+#define SEC_TS_CMD_SET_P2PTEST_MODE 0x83
+#define SEC_TS_CMD_NVM 0x85
+#define SEC_TS_CMD_STATEMANAGE_ON 0x8E
+#define SEC_TS_CMD_CALIBRATION_OFFSET_SDC 0x8F
+
+/* SEC_TS CUSTOMLIB OPCODE COMMAND */
+#define SEC_TS_CMD_CUSTOMLIB_GET_INFO 0x90
+#define SEC_TS_CMD_CUSTOMLIB_WRITE_PARAM 0x91
+#define SEC_TS_CMD_CUSTOMLIB_READ_PARAM 0x92
+#define SEC_TS_CMD_CUSTOMLIB_NOTIFY_PACKET 0x93
+#define SEC_TS_CMD_CUSTOMLIB_OFFSET_PRESSURE_LEVEL 0x5E
+#define SEC_TS_CMD_CUSTOMLIB_OFFSET_PRESSURE_THD_HIGH 0x84
+#define SEC_TS_CMD_CUSTOMLIB_OFFSET_PRESSURE_THD_LOW 0x86
+#define SEC_TS_CMD_CUSTOMLIB_LP_DUMP 0x01F0
+
+#define SEC_TS_CMD_STATUS_EVENT_TYPE 0xA0
+#define SEC_TS_READ_FW_INFO 0xA2
+#define SEC_TS_READ_FW_VERSION 0xA3
+#define SEC_TS_READ_PARA_VERSION 0xA4
+#define SEC_TS_READ_IMG_VERSION 0xA5
+#define SEC_TS_CMD_GET_CHECKSUM 0xA6
+#define SEC_TS_CMD_MIS_CAL_CHECK 0xA7
+#define SEC_TS_CMD_MIS_CAL_READ 0xA8
+#define SEC_TS_CMD_MIS_CAL_SPEC 0xA9
+#define SEC_TS_CMD_DEADZONE_RANGE 0xAA
+#define SEC_TS_CMD_LONGPRESSZONE_RANGE 0xAB
+#define SEC_TS_CMD_LONGPRESS_DROP_AREA 0xAC
+#define SEC_TS_CMD_LONGPRESS_DROP_DIFF 0xAD
+#define SEC_TS_READ_TS_STATUS 0xAF
+#define SEC_TS_CMD_SELFTEST 0xAE
+#define SEC_TS_READ_FORCE_RECAL_COUNT 0xB0
+#define SEC_TS_READ_FORCE_SIG_MAX_VAL 0xB1
+
+/* SEC_TS FLASH COMMAND */
+#define SEC_TS_CMD_FLASH_READ_ADDR 0xD0
+#define SEC_TS_CMD_FLASH_READ_SIZE 0xD1
+#define SEC_TS_CMD_FLASH_READ_DATA 0xD2
+#define SEC_TS_CMD_CHG_SYSMODE 0xD7
+#define SEC_TS_CMD_FLASH_ERASE 0xD8
+#define SEC_TS_CMD_FLASH_WRITE 0xD9
+#define SEC_TS_CMD_FLASH_PADDING 0xDA
+
+#define SEC_TS_READ_BL_UPDATE_STATUS 0xDB
+#define SEC_TS_CMD_SET_TOUCH_ENGINE_MODE 0xE1
+#define SEC_TS_CMD_SET_POWER_MODE 0xE4
+#define SEC_TS_CMD_EDGE_DEADZONE 0xE5
+#define SEC_TS_CMD_SET_DEX_MODE 0xE7
+#define SEC_TS_CMD_CALIBRATION_PRESSURE 0xE9
+/* Have to need delay 30msec after writing 0xEA command */
+/* Do not write Zero with 0xEA command */
+#define SEC_TS_CMD_SET_GET_PRESSURE 0xEA
+#define SEC_TS_CMD_SET_USER_PRESSURE 0xEB
+#define SEC_TS_CMD_SET_TEMPERATURE_COMP_MODE 0xEC
+#define SEC_TS_CMD_SET_TOUCHABLE_AREA 0xED
+#define SEC_TS_CMD_SET_BRUSH_MODE 0xEF
+
+#define SEC_TS_READ_CALIBRATION_REPORT 0xF1
+#define SEC_TS_CMD_SET_VENDOR_EVENT_LEVEL 0xF2
+#define SEC_TS_CMD_SET_SPENMODE 0xF3
+#define SEC_TS_CMD_SELECT_PRESSURE_TYPE 0xF5
+#define SEC_TS_CMD_READ_PRESSURE_DATA 0xF6
+
+#define SEC_TS_FLASH_SIZE_64 64
+#define SEC_TS_FLASH_SIZE_128 128
+#define SEC_TS_FLASH_SIZE_256 256
+
+#define SEC_TS_FLASH_SIZE_CMD 1
+#define SEC_TS_FLASH_SIZE_ADDR 2
+#define SEC_TS_FLASH_SIZE_CHECKSUM 1
+
+#define SEC_TS_STATUS_BOOT_MODE 0x10
+#define SEC_TS_STATUS_APP_MODE 0x20
+
+#define SEC_TS_FIRMWARE_PAGE_SIZE_256 256
+#define SEC_TS_FIRMWARE_PAGE_SIZE_128 128
+
+/* SEC status event id */
+#define SEC_TS_COORDINATE_EVENT 0
+#define SEC_TS_STATUS_EVENT 1
+#define SEC_TS_GESTURE_EVENT 2
+#define SEC_TS_EMPTY_EVENT 3
+
+#define SEC_TS_EVENT_BUFF_SIZE 8
+#define SEC_TS_SID_GESTURE 0x14
+#define SEC_TS_GESTURE_CODE_SPAY 0x00
+#define SEC_TS_GESTURE_CODE_DOUBLE_TAP 0x01
+
+#define SEC_TS_COORDINATE_ACTION_NONE 0
+#define SEC_TS_COORDINATE_ACTION_PRESS 1
+#define SEC_TS_COORDINATE_ACTION_MOVE 2
+#define SEC_TS_COORDINATE_ACTION_RELEASE 3
+
+#define SEC_TS_TOUCHTYPE_NORMAL 0
+#define SEC_TS_TOUCHTYPE_HOVER 1
+#define SEC_TS_TOUCHTYPE_FLIPCOVER 2
+#define SEC_TS_TOUCHTYPE_GLOVE 3
+#define SEC_TS_TOUCHTYPE_STYLUS 4
+#define SEC_TS_TOUCHTYPE_PALM 5
+#define SEC_TS_TOUCHTYPE_WET 6
+#define SEC_TS_TOUCHTYPE_PROXIMITY 7
+#define SEC_TS_TOUCHTYPE_JIG 8
+
+/* SEC_TS_INFO : Info acknowledge event */
+#define SEC_TS_ACK_BOOT_COMPLETE 0x00
+#define SEC_TS_ACK_WET_MODE 0x1
+
+/* SEC_TS_VENDOR_INFO : Vendor acknowledge event */
+#define SEC_TS_VENDOR_ACK_OFFSET_CAL_DONE 0x40
+#define SEC_TS_VENDOR_ACK_SELF_TEST_DONE 0x41
+#define SEC_TS_VENDOR_ACK_P2P_TEST_DONE 0x42
+
+/* SEC_TS_STATUS_EVENT_USER_INPUT */
+#define SEC_TS_EVENT_FORCE_KEY 0x1
+
+/* SEC_TS_STATUS_EVENT_CUSTOMLIB_INFO */
+#define SEC_TS_EVENT_CUSTOMLIB_FORCE_KEY 0x00
+
+/* SEC_TS_ERROR : Error event */
+#define SEC_TS_ERR_EVNET_CORE_ERR 0x0
+#define SEC_TS_ERR_EVENT_QUEUE_FULL 0x01
+#define SEC_TS_ERR_EVENT_ESD 0x2
+
+#define SEC_TS_BIT_SETFUNC_TOUCH (1 << 0)
+#define SEC_TS_BIT_SETFUNC_MUTUAL (1 << 0)
+#define SEC_TS_BIT_SETFUNC_HOVER (1 << 1)
+#define SEC_TS_BIT_SETFUNC_COVER (1 << 2)
+#define SEC_TS_BIT_SETFUNC_GLOVE (1 << 3)
+#define SEC_TS_BIT_SETFUNC_STYLUS (1 << 4)
+#define SEC_TS_BIT_SETFUNC_PALM (1 << 5)
+#define SEC_TS_BIT_SETFUNC_WET (1 << 6)
+#define SEC_TS_BIT_SETFUNC_PROXIMITY (1 << 7)
+
+#define SEC_TS_DEFAULT_ENABLE_BIT_SETFUNC (SEC_TS_BIT_SETFUNC_TOUCH | SEC_TS_BIT_SETFUNC_PALM | SEC_TS_BIT_SETFUNC_WET)
+
+#define SEC_TS_BIT_CHARGER_MODE_NO (0x1 << 0)
+#define SEC_TS_BIT_CHARGER_MODE_WIRE_CHARGER (0x1 << 1)
+#define SEC_TS_BIT_CHARGER_MODE_WIRELESS_CHARGER (0x1 << 2)
+#define SEC_TS_BIT_CHARGER_MODE_WIRELESS_BATTERY_PACK (0x1 << 3)
+
+#ifdef PAT_CONTROL
+/*
+ * <<< apply to server >>>
+ * 0x00 : no action
+ * 0x01 : clear nv
+ * 0x02 : pat magic
+ * 0x03 : rfu
+ *
+ * <<< use for temp bin >>>
+ * 0x05 : forced clear nv & f/w update before pat magic, eventhough same f/w
+ * 0x06 : rfu
+ */
+#define PAT_CONTROL_NONE 0x00
+#define PAT_CONTROL_CLEAR_NV 0x01
+#define PAT_CONTROL_PAT_MAGIC 0x02
+#define PAT_CONTROL_FORCE_UPDATE 0x05
+
+#define PAT_COUNT_ZERO 0x00
+#define PAT_MAX_LCIA 0x80
+#define PAT_MAGIC_NUMBER 0x83
+#define PAT_MAX_MAGIC 0xC5
+#define PAT_EXT_FACT 0xE0
+#define PAT_MAX_EXT 0xF5
+#endif
+
+#define STATE_MANAGE_ON 1
+#define STATE_MANAGE_OFF 0
+
+#define SEC_TS_STATUS_NOT_CALIBRATION 0x50
+#define SEC_TS_STATUS_CALIBRATION_SDC 0xA1
+#define SEC_TS_STATUS_CALIBRATION_SEC 0xA2
+
+#define SEC_TS_CMD_EDGE_HANDLER 0xAA
+#define SEC_TS_CMD_EDGE_AREA 0xAB
+#define SEC_TS_CMD_DEAD_ZONE 0xAC
+#define SEC_TS_CMD_LANDSCAPE_MODE 0xAD
+
+enum spec_check_type {
+ SPEC_NO_CHECK = 0,
+ SPEC_CHECK = 1,
+ SPEC_PASS = 2,
+ SPEC_FAIL = 3,
+};
+
+enum region_type {
+ REGION_NORMAL = 0,
+ REGION_EDGE = 1,
+ REGION_CORNER = 2,
+ REGION_NOTCH = 3,
+ REGION_TYPE_COUNT = 4,
+ /* REGION type should be continuous number start from 0,
+ * since REGION_TYPE_COUNT is used for type count
+ */
+};
+
+enum grip_write_mode {
+ G_NONE = 0,
+ G_SET_EDGE_HANDLER = 1,
+ G_SET_EDGE_ZONE = 2,
+ G_SET_NORMAL_MODE = 4,
+ G_SET_LANDSCAPE_MODE = 8,
+ G_CLR_LANDSCAPE_MODE = 16,
+};
+enum grip_set_data {
+ ONLY_EDGE_HANDLER = 0,
+ GRIP_ALL_DATA = 1,
+};
+
+typedef enum {
+ SEC_TS_STATE_POWER_OFF = 0,
+ SEC_TS_STATE_SUSPEND,
+ SEC_TS_STATE_LPM,
+ SEC_TS_STATE_POWER_ON
+} TOUCH_POWER_MODE;
+
+typedef enum {
+ TOUCH_SYSTEM_MODE_BOOT = 0,
+ TOUCH_SYSTEM_MODE_CALIBRATION = 1,
+ TOUCH_SYSTEM_MODE_TOUCH = 2,
+ TOUCH_SYSTEM_MODE_SELFTEST = 3,
+ TOUCH_SYSTEM_MODE_FLASH = 4,
+ TOUCH_SYSTEM_MODE_LOWPOWER = 5,
+ TOUCH_SYSTEM_MODE_LISTEN
+} TOUCH_SYSTEM_MODE;
+
+typedef enum {
+ TOUCH_MODE_STATE_IDLE = 0,
+ TOUCH_MODE_STATE_HOVER = 1,
+ TOUCH_MODE_STATE_TOUCH = 2,
+ TOUCH_MODE_STATE_NOISY = 3,
+ TOUCH_MODE_STATE_CAL = 4,
+ TOUCH_MODE_STATE_CAL2 = 5,
+ TOUCH_MODE_STATE_WAKEUP = 10
+} TOUCH_MODE_STATE;
+
+enum switch_system_mode {
+ TO_TOUCH_MODE = 0,
+ TO_LOWPOWER_MODE = 1,
+ TO_SELFTEST_MODE = 2,
+ TO_FLASH_MODE = 3,
+};
+
+enum {
+ TYPE_RAW_DATA = 0, /* Total - Offset : delta data */
+ TYPE_SIGNAL_DATA = 1, /* Signal - Filtering & Normalization */
+ TYPE_AMBIENT_BASELINE = 2, /* Cap Baseline */
+ TYPE_AMBIENT_DATA = 3, /* Cap Ambient */
+ TYPE_REMV_BASELINE_DATA = 4,
+ TYPE_DECODED_DATA = 5, /* Raw */
+ TYPE_REMV_AMB_DATA = 6, /* TYPE_RAW_DATA - TYPE_AMBIENT_DATA */
+ TYPE_NORM2_DATA = 15, /* After fs norm. data */
+ TYPE_OFFSET_DATA_SEC = 19, /* Cap Offset in SEC Manufacturing Line */
+ TYPE_OFFSET_DATA_SDC = 29, /* Cap Offset in SDC Manufacturing Line */
+ TYPE_NOI_P2P_MIN = 30, /* Peak-to-peak noise Min */
+ TYPE_NOI_P2P_MAX = 31, /* Peak-to-peak noise Max */
+ TYPE_INVALID_DATA = 0xFF, /* Invalid data type for release factory mode */
+};
+
+typedef enum {
+ CUSTOMLIB_EVENT_TYPE_SPAY = 0x04,
+ CUSTOMLIB_EVENT_TYPE_PRESSURE_TOUCHED = 0x05,
+ CUSTOMLIB_EVENT_TYPE_PRESSURE_RELEASED = 0x06,
+ CUSTOMLIB_EVENT_TYPE_AOD = 0x08,
+ CUSTOMLIB_EVENT_TYPE_AOD_PRESS = 0x09,
+ CUSTOMLIB_EVENT_TYPE_AOD_LONGPRESS = 0x0A,
+ CUSTOMLIB_EVENT_TYPE_AOD_DOUBLETAB = 0x0B,
+ CUSTOMLIB_EVENT_TYPE_AOD_HOMEKEY_PRESS = 0x0C,
+ CUSTOMLIB_EVENT_TYPE_AOD_HOMEKEY_RELEASE = 0x0D,
+ CUSTOMLIB_EVENT_TYPE_AOD_HOMEKEY_RELEASE_NO_HAPTIC = 0x0E
+} CUSTOMLIB_EVENT_TYPE;
+
+enum {
+ SEC_TS_BUS_REF_SCREEN_ON = 0x01,
+ SEC_TS_BUS_REF_IRQ = 0x02,
+ SEC_TS_BUS_REF_RESET = 0x04,
+ SEC_TS_BUS_REF_FW_UPDATE = 0x08,
+ SEC_TS_BUS_REF_INPUT_DEV = 0x10,
+ SEC_TS_BUS_REF_READ_INFO = 0x20,
+ SEC_TS_BUS_REF_SYSFS = 0x40,
+ SEC_TS_BUS_REF_FORCE_ACTIVE = 0x80
+};
+
+enum {
+ SEC_TS_ERR_NA = 0,
+ SEC_TS_ERR_INIT,
+ SEC_TS_ERR_ALLOC_FRAME,
+ SEC_TS_ERR_ALLOC_GAINTABLE,
+ SEC_TS_ERR_REG_INPUT_DEV,
+ SEC_TS_ERR_REG_INPUT_PAD_DEV
+};
+
+#define CMD_RESULT_WORD_LEN 10
+
+#define SEC_TS_I2C_RETRY_CNT 3
+#define SEC_TS_WAIT_RETRY_CNT 100
+
+#define SEC_TS_MODE_CUSTOMLIB_SPAY (1 << 1)
+#define SEC_TS_MODE_CUSTOMLIB_AOD (1 << 2)
+#define SEC_TS_MODE_CUSTOMLIB_FORCE_KEY (1 << 6)
+
+#define SEC_TS_MODE_LOWPOWER_FLAG (SEC_TS_MODE_CUSTOMLIB_SPAY | SEC_TS_MODE_CUSTOMLIB_AOD \
+ | SEC_TS_MODE_CUSTOMLIB_FORCE_KEY)
+
+#define SEC_TS_AOD_GESTURE_PRESS (1 << 7)
+#define SEC_TS_AOD_GESTURE_LONGPRESS (1 << 6)
+#define SEC_TS_AOD_GESTURE_DOUBLETAB (1 << 5)
+
+#define SEC_TS_CUSTOMLIB_EVENT_PRESSURE_TOUCHED (1 << 6)
+#define SEC_TS_CUSTOMLIB_EVENT_PRESSURE_RELEASED (1 << 7)
+
+enum sec_ts_cover_id {
+ SEC_TS_FLIP_WALLET = 0,
+ SEC_TS_VIEW_COVER,
+ SEC_TS_COVER_NOTHING1,
+ SEC_TS_VIEW_WIRELESS,
+ SEC_TS_COVER_NOTHING2,
+ SEC_TS_CHARGER_COVER,
+ SEC_TS_VIEW_WALLET,
+ SEC_TS_LED_COVER,
+ SEC_TS_CLEAR_FLIP_COVER,
+ SEC_TS_QWERTY_KEYBOARD_EUR,
+ SEC_TS_QWERTY_KEYBOARD_KOR,
+ SEC_TS_MONTBLANC_COVER = 100,
+};
+
+enum sec_fw_update_status {
+ SEC_NOT_UPDATE = 0,
+ SEC_NEED_FW_UPDATE,
+ SEC_NEED_CALIBRATION_ONLY,
+ SEC_NEED_FW_UPDATE_N_CALIBRATION,
+};
+
+enum tsp_hw_parameter {
+ TSP_ITO_CHECK = 1,
+ TSP_RAW_CHECK = 2,
+ TSP_MULTI_COUNT = 3,
+ TSP_WET_MODE = 4,
+ TSP_COMM_ERR_COUNT = 5,
+ TSP_MODULE_ID = 6,
+};
+
+/* Local heatmap */
+#define LOCAL_HEATMAP_WIDTH 7
+#define LOCAL_HEATMAP_HEIGHT 7
+
+struct heatmap_report {
+ int8_t offset_x;
+ uint8_t size_x;
+ int8_t offset_y;
+ uint8_t size_y;
+ /* data is in BE order; order should be enforced after data is read */
+ strength_t data[LOCAL_HEATMAP_WIDTH * LOCAL_HEATMAP_HEIGHT];
+} __attribute__((packed));
+
+#define TEST_MODE_MIN_MAX false
+#define TEST_MODE_ALL_NODE true
+#define TEST_MODE_READ_FRAME false
+#define TEST_MODE_READ_CHANNEL true
+
+/* factory test mode */
+struct sec_ts_test_mode {
+ u8 type;
+ short min[REGION_TYPE_COUNT];
+ short max[REGION_TYPE_COUNT];
+ bool allnode;
+ bool frame_channel;
+ enum spec_check_type spec_check;
+};
+
+struct sec_ts_fw_file {
+ u8 *data;
+ u32 pos;
+ size_t size;
+};
+
+/*
+ * write 0xE4 [ 11 | 10 | 01 | 00 ]
+ * MSB <-------------------> LSB
+ * read 0xE4
+ * mapping sequnce : LSB -> MSB
+ * struct sec_ts_test_result {
+ * * assy : front + OCTA assay
+ * * module : only OCTA
+ * union {
+ * struct {
+ * u8 assy_count:2; -> 00
+ * u8 assy_result:2; -> 01
+ * u8 module_count:2; -> 10
+ * u8 module_result:2; -> 11
+ * } __attribute__ ((packed));
+ * unsigned char data[1];
+ * };
+ *};
+ */
+struct sec_ts_test_result {
+ union {
+ struct {
+ u8 assy_count:2;
+ u8 assy_result:2;
+ u8 module_count:2;
+ u8 module_result:2;
+ } __attribute__ ((packed));
+ unsigned char data[1];
+ };
+};
+
+/* 8 byte */
+struct sec_ts_gesture_status {
+ u8 eid:2;
+ u8 stype:4;
+ u8 sf:2;
+ u8 gesture_id;
+ u8 gesture_data_1;
+ u8 gesture_data_2;
+ u8 gesture_data_3;
+ u8 gesture_data_4;
+ u8 reserved_1;
+ u8 left_event_5_0:6;
+ u8 reserved_2:2;
+} __attribute__ ((packed));
+
+/* 8 byte */
+struct sec_ts_event_status {
+ u8 eid:2;
+ u8 stype:4;
+ u8 sf:2;
+ u8 status_id;
+ u8 status_data_1;
+ u8 status_data_2;
+ u8 status_data_3;
+ u8 status_data_4;
+ u8 status_data_5;
+ u8 left_event_5_0:6;
+ u8 reserved_2:2;
+} __attribute__ ((packed));
+
+/* 8 byte */
+struct sec_ts_event_coordinate {
+ u8 eid:2;
+ u8 tid:4;
+ u8 tchsta:2;
+ u8 x_11_4;
+ u8 y_11_4;
+ u8 y_3_0:4;
+ u8 x_3_0:4;
+ u8 major;
+ u8 minor;
+ u8 z:6;
+ u8 ttype_3_2:2;
+ u8 left_event:6;
+ u8 ttype_1_0:2;
+} __attribute__ ((packed));
+
+/* not fixed */
+struct sec_ts_coordinate {
+ u8 id;
+ u8 ttype;
+ u8 action;
+ u16 x;
+ u16 y;
+ u8 z;
+ u8 hover_flag;
+ u8 glove_flag;
+ u8 touch_height;
+ u16 mcount;
+ u8 major;
+ u8 minor;
+ bool palm;
+ int palm_count;
+ u8 left_event;
+};
+
+struct sec_ts_data {
+ u32 isr_pin;
+
+ u32 crc_addr;
+ u32 fw_addr;
+ u32 para_addr;
+ u32 flash_page_size;
+ u8 boot_ver[3];
+
+ struct device *dev;
+ struct i2c_client *client;
+ struct input_dev *input_dev;
+ struct input_dev *input_dev_pad;
+ struct input_dev *input_dev_touch;
+ struct sec_ts_plat_data *plat_data;
+ struct sec_ts_coordinate coord[MAX_SUPPORT_TOUCH_COUNT + MAX_SUPPORT_HOVER_COUNT];
+
+ u64 timestamp; /* nanoseconds, acquired during hard interrupt */
+
+ struct timeval time_pressed[MAX_SUPPORT_TOUCH_COUNT + MAX_SUPPORT_HOVER_COUNT];
+ struct timeval time_released[MAX_SUPPORT_TOUCH_COUNT + MAX_SUPPORT_HOVER_COUNT];
+ long time_longest;
+
+ u8 lowpower_mode;
+ u8 lowpower_status;
+ u8 dex_mode;
+ char *dex_name;
+ u8 brush_mode;
+ u8 touchable_area;
+ volatile bool input_closed;
+
+ struct mutex bus_mutex;
+ u16 bus_refmask;
+ struct completion bus_resumed;
+
+ int touch_count;
+ int tx_count;
+ int rx_count;
+ int i2c_burstmax;
+ int ta_status;
+ volatile int power_status;
+ int raw_status;
+ int touchkey_glove_mode_status;
+ u16 touch_functions;
+ u8 charger_mode;
+ struct sec_ts_event_coordinate touchtype;
+ bool touched[11];
+ u8 gesture_status[6];
+ u8 cal_status;
+ struct mutex lock;
+ struct mutex device_mutex;
+ struct mutex i2c_mutex;
+ struct mutex eventlock;
+
+ struct notifier_block notifier;
+
+ struct pm_qos_request pm_qos_req;
+
+ struct v4l2_heatmap v4l2;
+
+ struct delayed_work work_read_info;
+#ifdef USE_POWER_RESET_WORK
+ struct delayed_work reset_work;
+ volatile bool reset_is_on_going;
+#endif
+ struct work_struct work_fw_update;
+ struct work_struct suspend_work;
+ struct work_struct resume_work;
+ struct workqueue_struct *event_wq; /* Used for event handler,
+ * suspend, resume threads */
+ struct completion resume_done;
+ struct sec_cmd_data sec;
+ short *pFrame;
+ u8 *gainTable;
+
+ bool probe_done;
+ bool reinit_done;
+ bool flip_enable;
+ int cover_type;
+ u8 cover_cmd;
+ u16 rect_data[4];
+
+ int tspid_val;
+ int tspicid_val;
+
+ bool use_customlib;
+ unsigned int scrub_id;
+ unsigned int scrub_x;
+ unsigned int scrub_y;
+
+ u8 grip_edgehandler_direction;
+ int grip_edgehandler_start_y;
+ int grip_edgehandler_end_y;
+ u16 grip_edge_range;
+ u8 grip_deadzone_up_x;
+ u8 grip_deadzone_dn_x;
+ int grip_deadzone_y;
+ u8 grip_landscape_mode;
+ int grip_landscape_edge;
+ u16 grip_landscape_deadzone;
+
+#ifdef CONFIG_TOUCHSCREEN_DUMP_MODE
+ struct delayed_work ghost_check;
+#endif
+ u8 tsp_dump_lock;
+
+ int nv;
+ int cal_count;
+ int tune_fix_ver;
+ bool external_factory;
+
+ int wet_mode;
+
+ unsigned char ito_test[4]; /* ito panel tx/rx chanel */
+ unsigned char check_multi;
+ unsigned int multi_count; /* multi touch count */
+ unsigned int wet_count; /* wet mode count */
+ unsigned int dive_count; /* dive mode count */
+ unsigned int comm_err_count; /* i2c comm error count */
+ unsigned int checksum_result; /* checksum result */
+ unsigned char module_id[4];
+ unsigned int all_finger_count;
+ unsigned int all_force_count;
+ unsigned int all_aod_tap_count;
+ unsigned int all_spay_count;
+ unsigned int max_z_value;
+ unsigned int min_z_value;
+ unsigned int sum_z_value;
+ unsigned char pressure_cal_base;
+ unsigned char pressure_cal_delta;
+
+#ifdef USE_PRESSURE_SENSOR
+ short pressure_left;
+ short pressure_center;
+ short pressure_right;
+ u8 pressure_user_level;
+#endif
+ int temp;
+
+ int fs_postcal_mean;
+
+ bool is_fw_corrupted;
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ struct tbn_context *tbn;
+#endif
+
+ int (*sec_ts_i2c_write)(struct sec_ts_data *ts, u8 reg,
+ u8 *data, int len);
+
+ int (*sec_ts_i2c_read)(struct sec_ts_data *ts, u8 reg,
+ u8 *data, int len);
+ int (*sec_ts_i2c_read_heap)(struct sec_ts_data *ts, u8 reg,
+ u8 *data, int len);
+
+ int (*sec_ts_i2c_write_burst)(struct sec_ts_data *ts,
+ u8 *data, int len);
+ int (*sec_ts_i2c_write_burst_heap)(struct sec_ts_data *ts,
+ u8 *data, int len);
+
+ int (*sec_ts_i2c_read_bulk)(struct sec_ts_data *ts,
+ u8 *data, int len);
+ int (*sec_ts_i2c_read_bulk_heap)(struct sec_ts_data *ts,
+ u8 *data, int len);
+
+ int (*sec_ts_read_customlib)(struct sec_ts_data *ts,
+ u8 *data, int len);
+
+ /* alloc for i2c read buffer */
+ u8 i2c_read_buf[I2C_PREALLOC_READ_BUF_SZ];
+ /* alloc for i2c write buffer */
+ u8 i2c_write_buf[I2C_PREALLOC_WRITE_BUF_SZ];
+};
+
+struct sec_ts_plat_data {
+ int max_x;
+ int max_y;
+ unsigned irq_gpio;
+ int irq_type;
+ int i2c_burstmax;
+ int always_lpmode;
+ int bringup;
+ int mis_cal_check;
+#ifdef PAT_CONTROL
+ int pat_function;
+ int afe_base;
+#endif
+ const char *firmware_name;
+ const char *model_name;
+ const char *project_name;
+ const char *regulator_dvdd;
+ const char *regulator_avdd;
+
+ u32 panel_revision;
+ u8 core_version_of_ic[4];
+ u8 core_version_of_bin[4];
+ u8 config_version_of_ic[4];
+ u8 config_version_of_bin[4];
+ u8 img_version_of_ic[4];
+ u8 img_version_of_bin[4];
+
+ struct pinctrl *pinctrl;
+
+ int (*power)(void *data, bool on);
+ void (*enable_sync)(bool on);
+ int tsp_icid;
+ int tsp_id;
+ int tsp_vsync;
+ int switch_gpio;
+ int reset_gpio;
+
+ bool regulator_boot_on;
+ bool support_mt_pressure;
+ bool support_dex;
+ bool support_sidegesture;
+};
+
+int sec_ts_stop_device(struct sec_ts_data *ts);
+int sec_ts_start_device(struct sec_ts_data *ts);
+int sec_ts_hw_reset(struct sec_ts_data *ts);
+int sec_ts_sw_reset(struct sec_ts_data *ts);
+int sec_ts_system_reset(struct sec_ts_data *ts);
+int sec_ts_set_lowpowermode(struct sec_ts_data *ts, u8 mode);
+int sec_ts_firmware_update_on_probe(struct sec_ts_data *ts, bool force_update);
+int sec_ts_firmware_update_on_hidden_menu(struct sec_ts_data *ts, int update_type);
+int sec_ts_glove_mode_enables(struct sec_ts_data *ts, int mode);
+int sec_ts_set_cover_type(struct sec_ts_data *ts, bool enable);
+int sec_ts_wait_for_ready(struct sec_ts_data *ts, unsigned int ack);
+int sec_ts_wait_for_ready_with_count(struct sec_ts_data *ts, unsigned int ack,
+ unsigned int count);
+int sec_ts_try_wake(struct sec_ts_data *ts, bool wake_setting);
+int sec_ts_set_bus_ref(struct sec_ts_data *ts, u16 ref, bool enable);
+
+int sec_ts_function(int (*func_init)(void *device_data), void (*func_remove)(void));
+int sec_ts_fn_init(struct sec_ts_data *ts);
+int sec_ts_read_calibration_report(struct sec_ts_data *ts);
+int sec_ts_execute_force_calibration(struct sec_ts_data *ts, int cal_mode);
+int sec_ts_fix_tmode(struct sec_ts_data *ts, u8 mode, u8 state);
+int sec_ts_release_tmode(struct sec_ts_data *ts);
+int get_tsp_nvm_data(struct sec_ts_data *ts, u8 offset);
+void set_tsp_nvm_data_clear(struct sec_ts_data *ts, u8 offset);
+#ifdef SEC_TS_SUPPORT_CUSTOMLIB
+int sec_ts_set_custom_library(struct sec_ts_data *ts);
+int sec_ts_check_custom_library(struct sec_ts_data *ts);
+#endif
+void sec_ts_unlocked_release_all_finger(struct sec_ts_data *ts);
+void sec_ts_locked_release_all_finger(struct sec_ts_data *ts);
+void sec_ts_fn_remove(struct sec_ts_data *ts);
+void sec_ts_delay(unsigned int ms);
+int sec_ts_read_information(struct sec_ts_data *ts);
+#ifdef PAT_CONTROL
+void set_pat_magic_number(struct sec_ts_data *ts);
+#endif
+void sec_ts_run_rawdata_all(struct sec_ts_data *ts, bool full_read);
+int execute_selftest(struct sec_ts_data *ts, bool save_result);
+int execute_p2ptest(struct sec_ts_data *ts);
+int sec_ts_read_raw_data(struct sec_ts_data *ts,
+ struct sec_cmd_data *sec, struct sec_ts_test_mode *mode);
+
+#if (1)//!defined(CONFIG_SAMSUNG_PRODUCT_SHIP)
+int sec_ts_raw_device_init(struct sec_ts_data *ts);
+#endif
+void sec_ts_raw_device_exit(struct sec_ts_data *ts);
+
+extern struct class *sec_class;
+
+#if defined(CONFIG_FB_MSM_MDSS_SAMSUNG)
+extern int get_lcd_attached(char *mode);
+#endif
+
+#if defined(CONFIG_EXYNOS_DECON_FB)
+extern int get_lcd_info(char *arg);
+#endif
+
+#ifdef CONFIG_MOTOR_DRV_MAX77865
+extern int haptic_homekey_press(void);
+extern int haptic_homekey_release(void);
+#else
+#define haptic_homekey_press() {}
+#define haptic_homekey_release() {}
+#endif
+
+extern bool tsp_init_done;
+
+extern struct sec_ts_data *ts_dup;
+
+#ifdef CONFIG_BATTERY_SAMSUNG
+extern unsigned int lpcharge;
+#endif
+
+extern void set_grip_data_to_ic(struct sec_ts_data *ts, u8 flag);
+extern void sec_ts_set_grip_type(struct sec_ts_data *ts, u8 set_type);
+
+
+#endif
diff --git a/drivers/input/touchscreen/sec_ts/sec_ts_fac_spec.h b/drivers/input/touchscreen/sec_ts/sec_ts_fac_spec.h
new file mode 100644
index 0000000..b5e0ebd
--- /dev/null
+++ b/drivers/input/touchscreen/sec_ts/sec_ts_fac_spec.h
@@ -0,0 +1,325 @@
+/* GC1 rawcap gap spec */
+
+/* Cm region definition table
+ * 0 : Normal
+ * 1 : Edge
+ * 2 : Corner
+ * 3 : Notch
+ */
+const int cm_region[37][18] = {
+ {2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1},
+ {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},
+ {1, 1, 1, 1, 1, 1, 3, 3, 3, 3, 3, 3, 1, 1, 1, 1, 1, 1},
+ {2, 1, 1, 1, 1, 2, 3, 3, 3, 3, 3, 3, 2, 1, 1, 1, 1, 2},
+};
+
+/* Cm spec by region
+ * region = {NORMAL, EDGE, CORNER, NOTCH}
+ * CM_MAX = max, CM_MIN = min, CM_MM = max - min
+ */
+const short cm_max[4] = { 701, 713, 664, 0 };
+const short cm_min[4] = { 415, 410, 336, 0 };
+const short cm_mm[4] = { 200, 303, 328, 0 };
+
+const int cm_gap[37][18] = {
+ {40, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 40, 40},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 20, 40, 40, 40, 40, 40, 40, 20, 20, 20, 20, 20, 20},
+ {20, 20, 20, 20, 20, 90, 90, 90, 90, 90, 90, 90, 90, 20, 20, 20, 20, 20},
+ {40, 20, 20, 20, 20, 90, 90, 90, 90, 90, 90, 90, 90, 20, 20, 20, 20, 40},
+ {40, 40, 20, 20, 90, 90, 90, 90, 90, 90, 90, 90, 90, 20, 20, 20, 40, 40},
+};
+
+const int noi_min[37][18] = {
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+ {-30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30, -30},
+};
+
+const int noi_max[37][18] = {
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+ {30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30, 30},
+};
+
+const int noi_mm = 60;
+
+const int cm_stdev_max = 15000;
+
+const int cs_tx_max = -12692;
+
+const int cs_tx_min = -22897;
+
+const int cs_tx_mm = 10205;
+
+const int cs_rx_max = -9638;
+
+const int cs_rx_min = -25296;
+
+const int cs_rx_mm = 15658;
+
+/* fs_precal high limit : +20% of fs target */
+const int fs_precal_h[37][18] = {
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 336, 366, 366, 366, 366, 336, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 0, 0, 0, 0, 0, 0, 336, 336, 336, 336, 336, 336},
+ {336, 336, 336, 336, 336, 336, 0, 0, 0, 0, 0, 0, 336, 336, 336, 336, 336, 336},
+};
+
+/* fs_mean high limit : +4% of fs_mean target */
+const int fs_mean_target_h = 269;
+
+/* fs_mean low limit : -4% of fs_mean target */
+const int fs_mean_target_l = 291;
+
+const int fs_postcal_uniform_spec = 4;
+
+/* fs_precal low limit : -20% of fs target */
+const int fs_precal_l[37][18] = {
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 224, 244, 244, 244, 244, 224, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 0, 0, 0, 0, 0, 0, 224, 224, 224, 224, 224, 224},
+ {224, 224, 224, 224, 224, 224, 0, 0, 0, 0, 0, 0, 224, 224, 224, 224, 224, 224},
+};
+
+const int fs_target[37][18] = {
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 0, 0, 0, 0, 0, 0, 280, 280, 280, 280, 280, 280},
+ {280, 280, 280, 280, 280, 280, 0, 0, 0, 0, 0, 0, 280, 280, 280, 280, 280, 280},
+};
+
+const int cs_tx_gap = 2000;
+
+const int cs_rx_gap = 7000;
diff --git a/drivers/input/touchscreen/sec_ts/sec_ts_fn.c b/drivers/input/touchscreen/sec_ts/sec_ts_fn.c
new file mode 100644
index 0000000..cd61486
--- /dev/null
+++ b/drivers/input/touchscreen/sec_ts/sec_ts_fn.c
@@ -0,0 +1,6824 @@
+/* drivers/input/touchscreen/sec_ts_fn.c
+ *
+ * Copyright (C) 2015 Samsung Electronics Co., Ltd.
+ * http://www.samsungsemi.com/
+ *
+ * Core file for Samsung TSC driver
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include "sec_ts.h"
+#include "sec_ts_fac_spec.h"
+
+static void fw_update(void *device_data);
+static void get_fw_ver_bin(void *device_data);
+static void get_fw_ver_ic(void *device_data);
+static void get_config_ver(void *device_data);
+#ifdef PAT_CONTROL
+static void get_pat_information(void *device_data);
+static void set_external_factory(void *device_data);
+#endif
+static void get_threshold(void *device_data);
+static void module_off_master(void *device_data);
+static void module_on_master(void *device_data);
+static void get_chip_vendor(void *device_data);
+static void get_chip_name(void *device_data);
+static void set_mis_cal_spec(void *device_data);
+static void get_mis_cal_info(void *device_data);
+static void get_wet_mode(void *device_data);
+static void get_x_num(void *device_data);
+static void get_y_num(void *device_data);
+static void get_x_cross_routing(void *device_data);
+static void get_y_cross_routing(void *device_data);
+static void get_checksum_data(void *device_data);
+static void run_reference_read(void *device_data);
+static void run_reference_read_all(void *device_data);
+static void get_reference(void *device_data);
+static void run_rawcap_read(void *device_data);
+static void run_rawcap_read_all(void *device_data);
+static void get_rawcap(void *device_data);
+static void run_rawcap_gap_read_all(void *device_data);
+static void run_delta_read(void *device_data);
+static void run_delta_read_all(void *device_data);
+static void get_delta(void *device_data);
+static void run_rawdata_stdev_read(void *device_data);
+static void run_rawdata_p2p_read_all(void *device_data);
+static void run_rawdata_read_all(void *device_data);
+static void run_self_reference_read(void *device_data);
+static void run_self_reference_read_all(void *device_data);
+static void run_self_rawcap_read(void *device_data);
+static void run_self_rawcap_read_all(void *device_data);
+static void run_self_rawcap_gap_read_all(void *device_data);
+static void run_self_delta_read(void *device_data);
+static void run_self_delta_read_all(void *device_data);
+static void run_force_calibration(void *device_data);
+static void get_force_calibration(void *device_data);
+#ifdef USE_PRESSURE_SENSOR
+static void run_force_pressure_calibration(void *device_data);
+static void set_pressure_test_mode(void *device_data);
+static void run_pressure_filtered_strength_read_all(void *device_data);
+static void run_pressure_strength_read_all(void *device_data);
+static void run_pressure_rawdata_read_all(void *device_data);
+static void run_pressure_offset_read_all(void *device_data);
+static void set_pressure_strength(void *device_data);
+static void set_pressure_rawdata(void *device_data);
+static void set_pressure_data_index(void *device_data);
+static void get_pressure_strength(void *device_data);
+static void get_pressure_rawdata(void *device_data);
+static void get_pressure_data_index(void *device_data);
+static void set_pressure_strength_clear(void *device_data);
+static void get_pressure_threshold(void *device_data);
+static void set_pressure_user_level(void *device_data);
+static void get_pressure_user_level(void *device_data);
+#endif
+static void run_fs_cal_pre_press(void *device_data);
+static void run_fs_cal_get_data(void *device_data);
+static void run_fs_cal_post_press(void *device_data);
+static void enable_fs_cal_table(void *device_data);
+static void enable_coordinate_report(void *device_data);
+static void enable_gain_limit(void *device_data);
+static void run_trx_short_test(void *device_data);
+static void set_tsp_test_result(void *device_data);
+static void get_tsp_test_result(void *device_data);
+static void increase_disassemble_count(void *device_data);
+static void get_disassemble_count(void *device_data);
+static void glove_mode(void *device_data);
+static void clear_cover_mode(void *device_data);
+static void dead_zone_enable(void *device_data);
+static void drawing_test_enable(void *device_data);
+static void set_lowpower_mode(void *device_data);
+static void set_wirelesscharger_mode(void *device_data);
+static void spay_enable(void *device_data);
+static void set_aod_rect(void *device_data);
+static void get_aod_rect(void *device_data);
+static void aod_enable(void *device_data);
+static void set_grip_data(void *device_data);
+static void dex_enable(void *device_data);
+static void brush_enable(void *device_data);
+static void force_touch_active(void *device_data);
+static void set_touchable_area(void *device_data);
+static void set_log_level(void *device_data);
+static void debug(void *device_data);
+static void not_support_cmd(void *device_data);
+
+
+static struct sec_cmd sec_cmds[] = {
+ {SEC_CMD("fw_update", fw_update),},
+ {SEC_CMD("get_fw_ver_bin", get_fw_ver_bin),},
+ {SEC_CMD("get_fw_ver_ic", get_fw_ver_ic),},
+ {SEC_CMD("get_config_ver", get_config_ver),},
+#ifdef PAT_CONTROL
+ {SEC_CMD("get_pat_information", get_pat_information),},
+ {SEC_CMD("set_external_factory", set_external_factory),},
+#endif
+ {SEC_CMD("get_threshold", get_threshold),},
+ {SEC_CMD("module_off_master", module_off_master),},
+ {SEC_CMD("module_on_master", module_on_master),},
+ {SEC_CMD("get_chip_vendor", get_chip_vendor),},
+ {SEC_CMD("get_chip_name", get_chip_name),},
+ {SEC_CMD("set_mis_cal_spec", set_mis_cal_spec),},
+ {SEC_CMD("get_mis_cal_info", get_mis_cal_info),},
+ {SEC_CMD("get_wet_mode", get_wet_mode),},
+ {SEC_CMD("get_x_num", get_x_num),},
+ {SEC_CMD("get_y_num", get_y_num),},
+ {SEC_CMD("get_x_cross_routing", get_x_cross_routing),},
+ {SEC_CMD("get_y_cross_routing", get_y_cross_routing),},
+ {SEC_CMD("get_checksum_data", get_checksum_data),},
+ {SEC_CMD("run_reference_read", run_reference_read),},
+ {SEC_CMD("run_reference_read_all", run_reference_read_all),},
+ {SEC_CMD("get_reference", get_reference),},
+ {SEC_CMD("run_rawcap_read", run_rawcap_read),},
+ {SEC_CMD("run_rawcap_read_all", run_rawcap_read_all),},
+ {SEC_CMD("get_rawcap", get_rawcap),},
+ {SEC_CMD("run_rawcap_gap_read_all", run_rawcap_gap_read_all),},
+ {SEC_CMD("run_delta_read", run_delta_read),},
+ {SEC_CMD("run_delta_read_all", run_delta_read_all),},
+ {SEC_CMD("get_delta", get_delta),},
+ {SEC_CMD("run_rawdata_stdev_read", run_rawdata_stdev_read),},
+ {SEC_CMD("run_rawdata_p2p_read_all", run_rawdata_p2p_read_all),},
+ {SEC_CMD("run_rawdata_read_all", run_rawdata_read_all),},
+ {SEC_CMD("run_self_reference_read", run_self_reference_read),},
+ {SEC_CMD("run_self_reference_read_all", run_self_reference_read_all),},
+ {SEC_CMD("run_self_rawcap_read", run_self_rawcap_read),},
+ {SEC_CMD("run_self_rawcap_read_all", run_self_rawcap_read_all),},
+ {SEC_CMD("run_self_rawcap_gap_read_all",
+ run_self_rawcap_gap_read_all),},
+ {SEC_CMD("run_self_delta_read", run_self_delta_read),},
+ {SEC_CMD("run_self_delta_read_all", run_self_delta_read_all),},
+ {SEC_CMD("run_force_calibration", run_force_calibration),},
+ {SEC_CMD("get_force_calibration", get_force_calibration),},
+#ifdef USE_PRESSURE_SENSOR
+ {SEC_CMD("run_force_pressure_calibration", run_force_pressure_calibration),},
+ {SEC_CMD("set_pressure_test_mode", set_pressure_test_mode),},
+ {SEC_CMD("run_pressure_filtered_strength_read_all", run_pressure_filtered_strength_read_all),},
+ {SEC_CMD("run_pressure_strength_read_all", run_pressure_strength_read_all),},
+ {SEC_CMD("run_pressure_rawdata_read_all", run_pressure_rawdata_read_all),},
+ {SEC_CMD("run_pressure_offset_read_all", run_pressure_offset_read_all),},
+ {SEC_CMD("set_pressure_strength", set_pressure_strength),},
+ {SEC_CMD("set_pressure_rawdata", set_pressure_rawdata),},
+ {SEC_CMD("set_pressure_data_index", set_pressure_data_index),},
+ {SEC_CMD("get_pressure_strength", get_pressure_strength),},
+ {SEC_CMD("get_pressure_rawdata", get_pressure_rawdata),},
+ {SEC_CMD("get_pressure_data_index", get_pressure_data_index),},
+ {SEC_CMD("set_pressure_strength_clear", set_pressure_strength_clear),},
+ {SEC_CMD("get_pressure_threshold", get_pressure_threshold),},
+ {SEC_CMD("set_pressure_user_level", set_pressure_user_level),},
+ {SEC_CMD("get_pressure_user_level", get_pressure_user_level),},
+#endif
+ {SEC_CMD("run_fs_cal_pre_press", run_fs_cal_pre_press),},
+ {SEC_CMD("run_fs_cal_get_data", run_fs_cal_get_data),},
+ {SEC_CMD("run_fs_cal_post_press", run_fs_cal_post_press),},
+ {SEC_CMD("enable_fs_cal_table", enable_fs_cal_table),},
+ {SEC_CMD("enable_coordinate_report", enable_coordinate_report),},
+ {SEC_CMD("enable_gain_limit", enable_gain_limit),},
+ {SEC_CMD("run_trx_short_test", run_trx_short_test),},
+ {SEC_CMD("set_tsp_test_result", set_tsp_test_result),},
+ {SEC_CMD("get_tsp_test_result", get_tsp_test_result),},
+ {SEC_CMD("increase_disassemble_count", increase_disassemble_count),},
+ {SEC_CMD("get_disassemble_count", get_disassemble_count),},
+ {SEC_CMD("glove_mode", glove_mode),},
+ {SEC_CMD("clear_cover_mode", clear_cover_mode),},
+ {SEC_CMD("dead_zone_enable", dead_zone_enable),},
+ {SEC_CMD("drawing_test_enable", drawing_test_enable),},
+ {SEC_CMD("set_lowpower_mode", set_lowpower_mode),},
+ {SEC_CMD("set_wirelesscharger_mode", set_wirelesscharger_mode),},
+ {SEC_CMD("spay_enable", spay_enable),},
+ {SEC_CMD("set_aod_rect", set_aod_rect),},
+ {SEC_CMD("get_aod_rect", get_aod_rect),},
+ {SEC_CMD("aod_enable", aod_enable),},
+ {SEC_CMD("set_grip_data", set_grip_data),},
+ {SEC_CMD("dex_enable", dex_enable),},
+ {SEC_CMD("brush_enable", brush_enable),},
+ {SEC_CMD("force_touch_active", force_touch_active),},
+ {SEC_CMD("set_touchable_area", set_touchable_area),},
+ {SEC_CMD("set_log_level", set_log_level),},
+ {SEC_CMD("debug", debug),},
+ {SEC_CMD("not_support_cmd", not_support_cmd),},
+};
+
+static ssize_t scrub_position_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[256] = { 0 };
+
+#ifdef CONFIG_SAMSUNG_PRODUCT_SHIP
+ input_info(true, &ts->client->dev,
+ "%s: scrub_id: %d\n", __func__, ts->scrub_id);
+#else
+ input_info(true, &ts->client->dev,
+ "%s: scrub_id: %d, X:%d, Y:%d\n", __func__,
+ ts->scrub_id, ts->scrub_x, ts->scrub_y);
+#endif
+ snprintf(buff, sizeof(buff), "%d %d %d", ts->scrub_id, ts->scrub_x, ts->scrub_y);
+
+ ts->scrub_x = 0;
+ ts->scrub_y = 0;
+
+ return snprintf(buf, PAGE_SIZE, "%s", buff);
+}
+
+static DEVICE_ATTR(scrub_pos, S_IRUGO, scrub_position_show, NULL);
+
+static ssize_t read_ito_check_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[256] = { 0 };
+
+ input_info(true, &ts->client->dev, "%s: %02X%02X%02X%02X\n", __func__,
+ ts->ito_test[0], ts->ito_test[1],
+ ts->ito_test[2], ts->ito_test[3]);
+
+ snprintf(buff, sizeof(buff), "%02X%02X%02X%02X",
+ ts->ito_test[0], ts->ito_test[1],
+ ts->ito_test[2], ts->ito_test[3]);
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%s", buff);
+}
+
+static ssize_t read_raw_check_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ int ii, ret = 0;
+ char *buffer = NULL;
+ char temp[CMD_RESULT_WORD_LEN] = { 0 };
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ buffer = vzalloc(ts->rx_count * ts->tx_count * 6);
+ if (!buffer) {
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return -ENOMEM;
+ }
+
+ memset(buffer, 0x00, ts->rx_count * ts->tx_count * 6);
+
+ for (ii = 0; ii < (ts->rx_count * ts->tx_count - 1); ii++) {
+ snprintf(temp, CMD_RESULT_WORD_LEN, "%d ", ts->pFrame[ii]);
+ strncat(buffer, temp, CMD_RESULT_WORD_LEN);
+
+ memset(temp, 0x00, CMD_RESULT_WORD_LEN);
+ }
+
+ snprintf(temp, CMD_RESULT_WORD_LEN, "%d", ts->pFrame[ii]);
+ strncat(buffer, temp, CMD_RESULT_WORD_LEN);
+
+ ret = snprintf(buf, ts->rx_count * ts->tx_count * 6, buffer);
+ vfree(buffer);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return ret;
+}
+
+static ssize_t read_multi_count_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ input_info(true, &ts->client->dev, "%s: %d\n", __func__,
+ ts->multi_count);
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%d", ts->multi_count);
+}
+
+static ssize_t clear_multi_count_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ ts->multi_count = 0;
+
+ input_info(true, &ts->client->dev, "%s: clear\n", __func__);
+
+ return count;
+}
+
+static ssize_t read_wet_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ input_info(true, &ts->client->dev, "%s: %d, %d\n", __func__,
+ ts->wet_count, ts->dive_count);
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%d", ts->wet_count);
+}
+
+static ssize_t clear_wet_mode_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ ts->wet_count = 0;
+ ts->dive_count= 0;
+
+ input_info(true, &ts->client->dev, "%s: clear\n", __func__);
+
+ return count;
+}
+
+static ssize_t read_comm_err_count_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ input_info(true, &ts->client->dev, "%s: %d\n", __func__,
+ ts->multi_count);
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%d", ts->comm_err_count);
+}
+
+static ssize_t clear_comm_err_count_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ ts->comm_err_count = 0;
+
+ input_info(true, &ts->client->dev, "%s: clear\n", __func__);
+
+ return count;
+}
+
+static ssize_t read_module_id_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[256] = { 0 };
+
+ input_info(true, &ts->client->dev, "%s: %d\n", __func__,
+ ts->multi_count);
+
+ snprintf(buff, sizeof(buff), "SE%02X%02X%02X%02X%02X%02X%02X",
+ ts->plat_data->panel_revision, ts->plat_data->img_version_of_bin[2],
+ ts->plat_data->img_version_of_bin[3], ts->nv, ts->cal_count,
+ ts->pressure_cal_base, ts->pressure_cal_delta);
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%s", buff);
+}
+
+static ssize_t read_vendor_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ unsigned char buffer[10] = { 0 };
+
+ snprintf(buffer, 5, ts->plat_data->firmware_name + 8);
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "LSI_%s", buffer);
+}
+
+static ssize_t clear_checksum_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ ts->checksum_result = 0;
+
+ input_info(true, &ts->client->dev, "%s: clear\n", __func__);
+
+ return count;
+}
+
+static ssize_t read_checksum_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ input_info(true, &ts->client->dev, "%s: %d\n", __func__,
+ ts->checksum_result);
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%d", ts->checksum_result);
+}
+
+static ssize_t clear_holding_time_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ ts->time_longest = 0;
+
+ input_info(true, &ts->client->dev, "%s: clear\n", __func__);
+
+ return count;
+}
+
+static ssize_t read_holding_time_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ input_info(true, &ts->client->dev, "%s: %ld\n", __func__,
+ ts->time_longest);
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%ld", ts->time_longest);
+}
+
+static ssize_t read_all_touch_count_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ input_info(true, &ts->client->dev, "%s: touch:%d, force:%d, aod:%d, spay:%d\n", __func__,
+ ts->all_finger_count, ts->all_force_count,
+ ts->all_aod_tap_count, ts->all_spay_count);
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE,
+ "\"TTCN\":\"%d\",\"TFCN\":\"%d\",\"TACN\":\"%d\",\"TSCN\":\"%d\"",
+ ts->all_finger_count, ts->all_force_count,
+ ts->all_aod_tap_count, ts->all_spay_count);
+}
+
+static ssize_t clear_all_touch_count_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ ts->all_force_count = 0;
+ ts->all_aod_tap_count = 0;
+ ts->all_spay_count = 0;
+
+ input_info(true, &ts->client->dev, "%s: clear\n", __func__);
+
+ return count;
+}
+
+static ssize_t read_z_value_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ input_info(true, &ts->client->dev, "%s: max:%d, min:%d, avg:%d\n", __func__,
+ ts->max_z_value, ts->min_z_value,
+ ts->sum_z_value);
+
+ if (ts->all_finger_count)
+ return snprintf(buf, SEC_CMD_BUF_SIZE,
+ "\"TMXZ\":\"%d\",\"TMNZ\":\"%d\",\"TAVZ\":\"%d\"",
+ ts->max_z_value, ts->min_z_value,
+ ts->sum_z_value / ts->all_finger_count);
+ else
+ return snprintf(buf, SEC_CMD_BUF_SIZE,
+ "\"TMXZ\":\"%d\",\"TMNZ\":\"%d\"",
+ ts->max_z_value, ts->min_z_value);
+
+}
+
+static ssize_t clear_z_value_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+
+ ts->max_z_value= 0;
+ ts->min_z_value= 0xFFFFFFFF;
+ ts->sum_z_value= 0;
+ ts->all_finger_count = 0;
+
+ input_info(true, &ts->client->dev, "%s: clear\n", __func__);
+
+ return count;
+}
+
+static ssize_t pressure_enable_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[256] = { 0 };
+
+ if (ts->lowpower_mode & SEC_TS_MODE_CUSTOMLIB_FORCE_KEY)
+ snprintf(buff, sizeof(buff), "1");
+ else
+ snprintf(buff, sizeof(buff), "0");
+
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%s\n", buff);
+}
+
+static ssize_t pressure_enable_strore(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ int ret;
+ unsigned long value = 0;
+
+ if (count > 2)
+ return -EINVAL;
+
+ ret = kstrtoul(buf, 10, &value);
+ if (ret != 0)
+ return ret;
+
+ if (!ts->use_customlib)
+ return -EINVAL;
+
+ if (value == 1)
+ ts->lowpower_mode |= SEC_TS_MODE_CUSTOMLIB_FORCE_KEY;
+ else
+ ts->lowpower_mode &= ~SEC_TS_MODE_CUSTOMLIB_FORCE_KEY;
+
+ #ifdef SEC_TS_SUPPORT_CUSTOMLIB
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+ sec_ts_set_custom_library(ts);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ #endif
+
+ return count;
+}
+
+static ssize_t get_lp_dump(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ u8 string_data[8] = {0, };
+ u16 current_index;
+ int i, ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "TSP turned off");
+ }
+
+ string_data[0] = SEC_TS_CMD_CUSTOMLIB_LP_DUMP & 0xFF;
+ string_data[1] = (SEC_TS_CMD_CUSTOMLIB_LP_DUMP & 0xFF00) >> 8;
+
+ disable_irq(ts->client->irq);
+
+ ret = ts->sec_ts_read_customlib(ts, string_data, 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Failed to read rect\n", __func__);
+ snprintf(buf, SEC_CMD_BUF_SIZE, "NG, Failed to read rect");
+ goto out;
+ }
+
+ current_index = (string_data[1] & 0xFF) << 8 | (string_data[0] & 0xFF);
+ if (current_index > 1000 || current_index < 500) {
+ input_err(true, &ts->client->dev,
+ "Failed to Custom Library LP log %d\n", current_index);
+ snprintf(buf, SEC_CMD_BUF_SIZE,
+ "NG, Failed to Custom Library LP log, current_index=%d",
+ current_index);
+ goto out;
+ }
+
+ input_info(true, &ts->client->dev,
+ "%s: DEBUG current_index = %d\n", __func__, current_index);
+
+ /* Custom Library has 62 stacks for LP dump */
+ for (i = 61; i >= 0; i--) {
+ u16 data0, data1, data2, data3;
+ char buff[30] = {0, };
+ u16 string_addr;
+
+ string_addr = current_index - (8 * i);
+ if (string_addr < 500)
+ string_addr += SEC_TS_CMD_CUSTOMLIB_LP_DUMP;
+ string_data[0] = string_addr & 0xFF;
+ string_data[1] = (string_addr & 0xFF00) >> 8;
+
+ ret = ts->sec_ts_read_customlib(ts, string_data, 8);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: Failed to read rect\n", __func__);
+ snprintf(buf, SEC_CMD_BUF_SIZE,
+ "NG, Failed to read rect, addr=%d",
+ string_addr);
+ goto out;
+ }
+
+ data0 = (string_data[1] & 0xFF) << 8 | (string_data[0] & 0xFF);
+ data1 = (string_data[3] & 0xFF) << 8 | (string_data[2] & 0xFF);
+ data2 = (string_data[5] & 0xFF) << 8 | (string_data[4] & 0xFF);
+ data3 = (string_data[7] & 0xFF) << 8 | (string_data[6] & 0xFF);
+ if (data0 || data1 || data2 || data3) {
+ snprintf(buff, sizeof(buff),
+ "%d: %04x%04x%04x%04x\n",
+ string_addr, data0, data1, data2, data3);
+ strncat(buf, buff, SEC_CMD_BUF_SIZE);
+ }
+ }
+
+out:
+ enable_irq(ts->client->irq);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return strlen(buf);
+}
+
+static ssize_t get_force_recal_count(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ u8 rbuf[4] = {0, };
+ u32 recal_count;
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%d", -ENODEV);
+ }
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_FORCE_RECAL_COUNT, rbuf, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: Failed to read\n", __func__);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%d", -EIO);
+ }
+
+ recal_count = (rbuf[0] & 0xFF) << 24 | (rbuf[1] & 0xFF) << 16 |
+ (rbuf[2] & 0xFF) << 8 | (rbuf[3] & 0xFF);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return snprintf(buf, SEC_CMD_BUF_SIZE, "%d", recal_count);
+}
+
+static ssize_t fw_version_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ int written = 0;
+
+ written = scnprintf(buf, PAGE_SIZE, "SE-V%02X.%02X.%02X\n",
+ ts->plat_data->panel_revision,
+ ts->plat_data->img_version_of_ic[2],
+ ts->plat_data->img_version_of_ic[3]);
+ return written;
+}
+
+static ssize_t status_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_cmd_data *sec = dev_get_drvdata(dev);
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ int written = 0;
+ unsigned char data[4] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ data[0] = 0;
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_BOOT_STATUS, data, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to read boot status(%d)\n",
+ __func__, ret);
+ goto out;
+ }
+ written += scnprintf(buf + written, PAGE_SIZE - written,
+ "BOOT STATUS: 0x%02X\n", data[0]);
+
+ memset(data, 0x0, 4);
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_TS_STATUS, data, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to touch status(%d)\n",
+ __func__, ret);
+ goto out;
+ }
+ written += scnprintf(buf + written, PAGE_SIZE - written,
+ "TOUCH STATUS: 0x%02X, 0x%02X, 0x%02X, 0x%02X\n",
+ data[0], data[1], data[2], data[3]);
+
+ memset(data, 0x0, 2);
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_SET_TOUCHFUNCTION, data, 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to read touch functions(%d)\n",
+ __func__, ret);
+ goto out;
+ }
+ written += scnprintf(buf + written, PAGE_SIZE - written,
+ "Functions: 0x%02X, 0x%02X\n", data[0], data[1]);
+
+out:
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return written;
+}
+
+static DEVICE_ATTR(ito_check, S_IRUGO, read_ito_check_show, NULL);
+static DEVICE_ATTR(raw_check, S_IRUGO, read_raw_check_show, NULL);
+static DEVICE_ATTR(multi_count, S_IRUGO | S_IWUSR | S_IWGRP, read_multi_count_show, clear_multi_count_store);
+static DEVICE_ATTR(wet_mode, S_IRUGO | S_IWUSR | S_IWGRP, read_wet_mode_show, clear_wet_mode_store);
+static DEVICE_ATTR(comm_err_count, S_IRUGO | S_IWUSR | S_IWGRP, read_comm_err_count_show, clear_comm_err_count_store);
+static DEVICE_ATTR(checksum, S_IRUGO | S_IWUSR | S_IWGRP, read_checksum_show, clear_checksum_store);
+static DEVICE_ATTR(holding_time, S_IRUGO | S_IWUSR | S_IWGRP, read_holding_time_show, clear_holding_time_store);
+static DEVICE_ATTR(all_touch_count, S_IRUGO | S_IWUSR | S_IWGRP, read_all_touch_count_show, clear_all_touch_count_store);
+static DEVICE_ATTR(z_value, S_IRUGO | S_IWUSR | S_IWGRP, read_z_value_show, clear_z_value_store);
+static DEVICE_ATTR(module_id, S_IRUGO, read_module_id_show, NULL);
+static DEVICE_ATTR(vendor, S_IRUGO, read_vendor_show, NULL);
+static DEVICE_ATTR(pressure_enable, S_IRUGO | S_IWUSR | S_IWGRP, pressure_enable_show, pressure_enable_strore);
+static DEVICE_ATTR(get_lp_dump, S_IRUGO, get_lp_dump, NULL);
+static DEVICE_ATTR(force_recal_count, S_IRUGO, get_force_recal_count, NULL);
+static DEVICE_ATTR(fw_version, 0444, fw_version_show, NULL);
+static DEVICE_ATTR(status, 0444, status_show, NULL);
+
+static struct attribute *cmd_attributes[] = {
+ &dev_attr_scrub_pos.attr,
+ &dev_attr_ito_check.attr,
+ &dev_attr_raw_check.attr,
+ &dev_attr_multi_count.attr,
+ &dev_attr_wet_mode.attr,
+ &dev_attr_comm_err_count.attr,
+ &dev_attr_checksum.attr,
+ &dev_attr_holding_time.attr,
+ &dev_attr_all_touch_count.attr,
+ &dev_attr_z_value.attr,
+ &dev_attr_module_id.attr,
+ &dev_attr_vendor.attr,
+ &dev_attr_pressure_enable.attr,
+ &dev_attr_get_lp_dump.attr,
+ &dev_attr_force_recal_count.attr,
+ &dev_attr_fw_version.attr,
+ &dev_attr_status.attr,
+ NULL,
+};
+
+static struct attribute_group cmd_attr_group = {
+ .attrs = cmd_attributes,
+};
+
+static int sec_ts_check_index(struct sec_ts_data *ts)
+{
+ struct sec_cmd_data *sec = &ts->sec;
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ int node;
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > ts->tx_count
+ || sec->cmd_param[1] < 0 || sec->cmd_param[1] > ts->rx_count) {
+
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ input_info(true, &ts->client->dev, "%s: parameter error: %u, %u\n",
+ __func__, sec->cmd_param[0], sec->cmd_param[0]);
+ node = -1;
+ return node;
+ }
+ node = sec->cmd_param[1] * ts->tx_count + sec->cmd_param[0];
+ input_info(true, &ts->client->dev, "%s: node = %d\n", __func__, node);
+
+ return node;
+}
+static void fw_update(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[64] = { 0 };
+ int retval = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ retval = sec_ts_firmware_update_on_hidden_menu(ts, sec->cmd_param[0]);
+ if (retval < 0) {
+ snprintf(buff, sizeof(buff), "%s", "NA");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ input_err(true, &ts->client->dev, "%s: failed [%d]\n", __func__, retval);
+ } else {
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: success [%d]\n", __func__, retval);
+ }
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+int sec_ts_fix_tmode(struct sec_ts_data *ts, u8 mode, u8 state)
+{
+ int ret;
+ u8 onoff[1] = {STATE_MANAGE_OFF};
+ u8 tBuff[2] = { mode, state };
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_STATEMANAGE_ON, onoff, 1);
+ sec_ts_delay(20);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CHG_SYSMODE, tBuff, sizeof(tBuff));
+ sec_ts_delay(20);
+
+ return ret;
+}
+
+int sec_ts_release_tmode(struct sec_ts_data *ts)
+{
+ int ret;
+ u8 onoff[1] = {STATE_MANAGE_ON};
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_STATEMANAGE_ON, onoff, 1);
+ sec_ts_delay(20);
+
+ return ret;
+}
+
+/* sec_ts_cm_spec_over_check : apply gap calculation with ts->pFrame data
+ * gap = abs(N1 - N2) / MAX(N1, N2) * 100 (%)
+ */
+static int sec_ts_cm_spec_over_check(struct sec_ts_data *ts, short *gap,
+ bool gap_dir)
+{
+ int i = 0;
+ int j = 0;
+ int gapx, gapy, pos1, pos2;
+ short dpos1, dpos2;
+ int specover_count = 0;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ /* Get x-direction cm gap */
+ if (!gap_dir) {
+ input_info(true, &ts->client->dev, "gapX TX\n");
+
+ for (i = 0; i < ts->rx_count; i++) {
+ for (j = 0; j < ts->tx_count - 1; j++) {
+ /* Exclude last line to get gap between two
+ * lines.
+ */
+ pos1 = (i * ts->tx_count) + j;
+ pos2 = (i * ts->tx_count) + (j + 1);
+
+ dpos1 = ts->pFrame[pos1];
+ dpos2 = ts->pFrame[pos2];
+
+ if (dpos1 > dpos2)
+ gapx = 100 - (dpos2 * 100 / dpos1);
+ else
+ gapx = 100 - (dpos1 * 100 / dpos2);
+
+ gap[pos1] = gapx;
+
+ if (gapx > cm_gap[i][j])
+ specover_count++;
+ }
+ }
+ }
+
+ /* get y-direction cm gap */
+ else {
+ input_info(true, &ts->client->dev, "gapY RX\n");
+
+ for (i = 0; i < ts->rx_count - 1; i++) {
+ for (j = 0; j < ts->tx_count; j++) {
+ pos1 = (i * ts->tx_count) + j;
+ pos2 = ((i + 1) * ts->tx_count) + j;
+
+ dpos1 = ts->pFrame[pos1];
+ dpos2 = ts->pFrame[pos2];
+
+ if (dpos1 > dpos2)
+ gapy = 100 - (dpos2 * 100 / dpos1);
+ else
+ gapy = 100 - (dpos1 * 100 / dpos2);
+
+ gap[pos1] = gapy;
+
+ if (gapy > cm_gap[i][j])
+ specover_count++;
+ }
+ }
+ }
+
+ input_info(true, &ts->client->dev, "%s: Gap NG for %d node(s)\n",
+ gap_dir == 0 ? "gapX" : "gapY", specover_count);
+
+ return specover_count;
+}
+
+static int sec_ts_cs_spec_over_check(struct sec_ts_data *ts, short *gap)
+{
+ int i;
+ int specover_count = 0;
+ short dTmp;
+
+ for (i = 0; i < ts->tx_count - 1; i++) {
+ dTmp = ts->pFrame[i] - ts->pFrame[i + 1];
+ if (dTmp < 0)
+ dTmp *= -1;
+
+ gap[i] = dTmp;
+
+ if (dTmp > cs_tx_gap)
+ specover_count++;
+ }
+
+ for (i = ts->tx_count; i < ts->tx_count + ts->rx_count - 1; i++) {
+ dTmp = ts->pFrame[i] - ts->pFrame[i + 1];
+ if (dTmp < 0)
+ dTmp *= -1;
+
+ gap[i] = dTmp;
+
+ if (dTmp > cs_rx_gap)
+ specover_count++;
+ }
+
+ input_info(true, &ts->client->dev, "%s: Gap NG for %d node(s)\n",
+ __func__, specover_count);
+
+ return specover_count;
+}
+
+static int sec_ts_get_gain_table(struct sec_ts_data *ts)
+{
+ int i, j;
+ int temp;
+ int tmp_dv;
+ unsigned int str_size, str_len = 0;
+ unsigned char *pStr = NULL;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ for (i = 0; i < ts->rx_count; i++) {
+ for (j = 0; j < ts->tx_count; j++) {
+ tmp_dv = ts->pFrame[i * ts->tx_count + j];
+
+ /* skip notch area */
+ if (cm_region[i][j] == REGION_NOTCH) {
+ ts->gainTable[j * ts->rx_count + i] = 0;
+ continue;
+ }
+
+ if (tmp_dv <= 0) {
+ input_info(true, &ts->client->dev,
+ "%s: node[%d,%d] == 0\n", __func__, i,
+ j);
+ tmp_dv = 1;
+ }
+
+ temp = (fs_target[i][j] * 1000) / (tmp_dv) * 64;
+ /* Add 500 to round the result */
+ temp = (temp + 500) / 1000;
+ if (temp > 255)
+ temp = 255;
+ ts->gainTable[j * ts->rx_count + i] = (temp & 0xFF);
+ }
+ }
+
+ str_size = 6 * (ts->tx_count + 1);
+ pStr = kzalloc(str_size, GFP_KERNEL);
+ if (pStr == NULL)
+ return -ENOMEM;
+
+ input_info(true, &ts->client->dev, "%s: Gain Table\n", __func__);
+
+ for (i = 0; i < ts->rx_count; i++) {
+ pStr[0] = 0;
+ str_len = 0;
+ for (j = 0; j < ts->tx_count; j++) {
+ str_len += scnprintf(pStr + str_len, str_size - str_len,
+ " %3d",
+ ts->gainTable[(j * ts->rx_count) + i]);
+ }
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+ }
+
+ kfree(pStr);
+
+ return 0;
+}
+
+static int sec_ts_write_gain_table(struct sec_ts_data *ts)
+{
+ int node_cnt = ts->tx_count * ts->rx_count;
+ u8 *gainTable = NULL;
+ u8 *tCmd = NULL;
+ int copy_max, copy_left, copy_size, copy_cur;
+ int ret = -1;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ /* Write norm table to ic
+ * divide data into 256 bytes:
+ * i2c buffer size limit 256 bytes
+ */
+ gainTable = ts->gainTable;
+
+ copy_max = ts->i2c_burstmax - 3;
+ copy_left = node_cnt;
+ copy_size = 0;
+ copy_cur = (copy_left > copy_max) ? copy_max : copy_left;
+
+ tCmd = kzalloc(copy_cur + 3, GFP_KERNEL);
+ if (!tCmd)
+ goto ErrorAlloc;
+
+ while (copy_left > 0) {
+ tCmd[0] = SEC_TS_CMD_WRITE_NORM_TABLE;
+ tCmd[1] = (copy_size >> 8) & 0xFF;
+ tCmd[2] = (copy_size >> 0) & 0xFF;
+
+ memcpy(&tCmd[3], &gainTable[copy_size], copy_cur);
+
+ input_info(true, &ts->client->dev,
+ "%s: left = %d, cur = %d, size = %d\n",
+ __func__, copy_left, copy_cur, copy_size);
+
+ ret = ts->sec_ts_i2c_write_burst_heap(ts, tCmd, 3 + copy_cur);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: table write failed\n", __func__);
+
+ copy_size += copy_cur;
+ copy_left -= copy_cur;
+ copy_cur = (copy_left > copy_max) ? copy_max : copy_left;
+ }
+
+ErrorAlloc:
+ kfree(tCmd);
+
+ return ret;
+}
+
+/* sec_ts_get_postcal_mean : get mean value for all nodes */
+static int sec_ts_get_postcal_mean(struct sec_ts_data *ts)
+{
+ int i, j;
+ int sum = 0;
+ int nCnt = 0;
+
+ for (i = 0; i < ts->rx_count; i++) {
+ for (j = 0; j < ts->tx_count; j++) {
+ if (cm_region[i][j] == REGION_NOTCH) {
+ /* Count notch nodes, where fs target is 0 */
+ nCnt++;
+ continue;
+ }
+ sum += ts->pFrame[(i * ts->tx_count) + j];
+ }
+ }
+
+ /* exclude notch area from average */
+ sum = sum / (ts->tx_count * ts->rx_count - nCnt);
+
+ return sum;
+}
+
+static int sec_ts_get_postcal_uniformity(struct sec_ts_data *ts, short *diff)
+{
+ int pos1, pos2;
+ short dpos1, dpos2, gap;
+ int i = 0;
+ int j = 0;
+ int specover_cnt = 0;
+
+ for (i = 0; i < ts->rx_count; i++) {
+ for (j = 0; j < ts->tx_count - 1; j++) {
+ /* At the notch boundary, skip (leave gap as 0)
+ * if node[row][col] or node[row][col+1] is 0,
+ * it is notch boundary for column direction
+ */
+ if ((cm_region[i][j] == REGION_NOTCH) ||
+ (cm_region[i][j + 1] == REGION_NOTCH))
+ continue;
+
+ pos1 = (i * ts->tx_count) + j;
+ pos2 = (i * ts->tx_count) + (j + 1);
+
+ dpos1 = ts->pFrame[pos1];
+ dpos2 = ts->pFrame[pos2];
+
+ gap = (dpos1 > dpos2) ? (dpos1 - dpos2) :
+ (dpos2 - dpos1);
+
+ diff[pos1] = gap;
+ }
+ }
+
+ for (i = 0; i < ts->rx_count - 1; i++) {
+ for (j = 0; j < ts->tx_count; j++) {
+ /* At the notch boundary, skip (leave gap as 0)
+ * if node[row][col] or node[row+1][col] is 0,
+ * it is notch boundary for row direction
+ */
+ if ((cm_region[i][j] == REGION_NOTCH) ||
+ (cm_region[i + 1][j] == REGION_NOTCH))
+ continue;
+
+ pos1 = (i * ts->tx_count) + j;
+ pos2 = ((i + 1) * ts->tx_count) + j;
+
+ dpos1 = ts->pFrame[pos1];
+ dpos2 = ts->pFrame[pos2];
+
+ gap = (dpos1 > dpos2) ? (dpos1 - dpos2) :
+ (dpos2 - dpos1);
+
+ /* find max gap between x and y direction */
+ if (diff[pos1] < gap)
+ diff[pos1] = gap;
+ }
+ }
+
+ for (i = 0; i < ts->rx_count * ts->tx_count; i++) {
+ /* since spec is in % unit, multiply 100 */
+ diff[i] *= 100;
+ diff[i] /= (ts->fs_postcal_mean);
+ if (diff[i] > fs_postcal_uniform_spec)
+ specover_cnt++;
+ }
+
+ return specover_cnt;
+}
+
+static void sec_ts_print_frame(struct sec_ts_data *ts, short *min, short *max)
+{
+ int i = 0;
+ int j = 0;
+ unsigned char *pStr = NULL;
+ unsigned char pTmp[16] = { 0 };
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ pStr = kzalloc(6 * (ts->tx_count + 1), GFP_KERNEL);
+ if (pStr == NULL)
+ return;
+
+ memset(pStr, 0x0, 6 * (ts->tx_count + 1));
+ snprintf(pTmp, sizeof(pTmp), " TX");
+ strncat(pStr, pTmp, 6 * ts->tx_count);
+
+ for (i = 0; i < ts->tx_count; i++) {
+ snprintf(pTmp, sizeof(pTmp), " %02d ", i);
+ strncat(pStr, pTmp, 6 * ts->tx_count);
+ }
+
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+ memset(pStr, 0x0, 6 * (ts->tx_count + 1));
+ snprintf(pTmp, sizeof(pTmp), " +");
+ strncat(pStr, pTmp, 6 * ts->tx_count);
+
+ for (i = 0; i < ts->tx_count; i++) {
+ snprintf(pTmp, sizeof(pTmp), "----");
+ strncat(pStr, pTmp, 6 * ts->rx_count);
+ }
+
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+
+ for (i = 0; i < ts->rx_count; i++) {
+ memset(pStr, 0x0, 6 * (ts->tx_count + 1));
+ snprintf(pTmp, sizeof(pTmp), "Rx%02d | ", i);
+ strncat(pStr, pTmp, 6 * ts->tx_count);
+
+ for (j = 0; j < ts->tx_count; j++) {
+ snprintf(pTmp, sizeof(pTmp), " %3d", ts->pFrame[(j * ts->rx_count) + i]);
+
+ if (i > 0) {
+ if (ts->pFrame[(j * ts->rx_count) + i] < *min)
+ *min = ts->pFrame[(j * ts->rx_count) + i];
+
+ if (ts->pFrame[(j * ts->rx_count) + i] > *max)
+ *max = ts->pFrame[(j * ts->rx_count) + i];
+ }
+ strncat(pStr, pTmp, 6 * ts->rx_count);
+ }
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+ }
+ kfree(pStr);
+}
+
+static int sec_ts_read_frame(struct sec_ts_data *ts, u8 type, short *min,
+ short *max, enum spec_check_type *spec_check)
+{
+ unsigned int readbytes = 0xFF;
+ unsigned char *pRead = NULL;
+ u8 mode = TYPE_INVALID_DATA;
+ int ret = 0;
+ int i = 0;
+ int j = 0;
+ short dTmp = 0;
+ short *temp = NULL;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ /* set data length, allocation buffer memory */
+ readbytes = ts->rx_count * ts->tx_count * 2;
+
+ pRead = kzalloc(readbytes, GFP_KERNEL);
+ if (!pRead)
+ return -ENOMEM;
+
+ /* set OPCODE and data type */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_MUTU_RAW_TYPE, &type, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Set rawdata type failed\n", __func__);
+ goto ErrorExit;
+ }
+
+ sec_ts_delay(50);
+
+ if (type == TYPE_OFFSET_DATA_SDC) {
+ /* excute selftest for real cap offset data, because real cap data is not memory data in normal touch. */
+ char para = TO_TOUCH_MODE;
+
+ disable_irq(ts->client->irq);
+
+ execute_selftest(ts, true);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_POWER_MODE, ¶, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: Set powermode failed\n", __func__);
+ enable_irq(ts->client->irq);
+ goto ErrorRelease;
+ }
+
+ enable_irq(ts->client->irq);
+ }
+
+ /* read data */
+ ret = ts->sec_ts_i2c_read_heap(ts, SEC_TS_READ_TOUCH_RAWDATA, pRead,
+ readbytes);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: read rawdata failed!\n", __func__);
+ goto ErrorRelease;
+ }
+
+ memset(ts->pFrame, 0x00, readbytes);
+
+ for (i = 0; i < readbytes; i += 2)
+ ts->pFrame[i / 2] = pRead[i + 1] + (pRead[i] << 8);
+
+#ifdef DEBUG_MSG
+ input_info(true, &ts->client->dev, "%s: 02X%02X%02X readbytes=%d\n", __func__,
+ pRead[0], pRead[1], pRead[2], readbytes);
+#endif
+ sec_ts_print_frame(ts, min, max);
+
+ temp = kzalloc(readbytes, GFP_KERNEL);
+ if (!temp)
+ goto ErrorRelease;
+
+ memcpy(temp, ts->pFrame, ts->tx_count * ts->rx_count * 2);
+ memset(ts->pFrame, 0x00, ts->tx_count * ts->rx_count * 2);
+
+ for (i = 0; i < ts->tx_count; i++) {
+ for (j = 0; j < ts->rx_count; j++)
+ ts->pFrame[(j * ts->tx_count) + i] = temp[(i * ts->rx_count) + j];
+ }
+
+ /* spec check */
+ if (*spec_check == SPEC_CHECK) {
+ int specover_count = 0;
+
+ if (type == TYPE_OFFSET_DATA_SDC) {
+ unsigned int region = 0;
+ /* set initial value for min, max */
+ for (i = 0; i < REGION_TYPE_COUNT; i++) {
+ min[i] = SHRT_MAX;
+ max[i] = SHRT_MIN;
+ }
+
+ for (i = 0; i < ts->rx_count; i++) {
+ for (j = 0; j < ts->tx_count; j++) {
+ dTmp = ts->pFrame[i * ts->tx_count + j];
+ region = cm_region[i][j];
+
+ if (region == REGION_NOTCH)
+ continue;
+
+ min[region] = min(min[region], dTmp);
+ max[region] = max(max[region], dTmp);
+
+ if (dTmp > cm_max[region])
+ specover_count++;
+ if (dTmp < cm_min[region])
+ specover_count++;
+ }
+ }
+ input_info(true, &ts->client->dev, "%s: type = %d, specover = %d\n",
+ __func__, type, specover_count);
+
+ if (specover_count == 0 &&
+ (max[REGION_NORMAL] - min[REGION_NORMAL] <
+ cm_mm[REGION_NORMAL]) &&
+ (max[REGION_EDGE] - min[REGION_EDGE] <
+ cm_mm[REGION_EDGE]) &&
+ (max[REGION_CORNER] - min[REGION_CORNER] <
+ cm_mm[REGION_CORNER]))
+ *spec_check = SPEC_PASS;
+ else
+ *spec_check = SPEC_FAIL;
+ } else if (type == TYPE_NOI_P2P_MIN) {
+ for (i = 0; i < ts->rx_count; i++) {
+ for (j = 0; j < ts->tx_count; j++) {
+ dTmp = ts->pFrame[i * ts->tx_count + j];
+ if (cm_region[i][j] != REGION_NOTCH &&
+ dTmp < noi_min[i][j])
+ specover_count++;
+ }
+ }
+ input_info(true, &ts->client->dev, "%s: type = %d, specover = %d\n",
+ __func__, type, specover_count);
+
+ if (specover_count == 0)
+ *spec_check = SPEC_PASS;
+ else
+ *spec_check = SPEC_FAIL;
+ } else if (type == TYPE_NOI_P2P_MAX) {
+ for (i = 0; i < ts->rx_count; i++) {
+ for (j = 0; j < ts->tx_count; j++) {
+ dTmp = ts->pFrame[i * ts->tx_count + j];
+ if (cm_region[i][j] != REGION_NOTCH &&
+ dTmp > noi_max[i][j])
+ specover_count++;
+ }
+ }
+ input_info(true, &ts->client->dev, "%s: type = %d, specover = %d\n",
+ __func__, type, specover_count);
+
+ if (specover_count == 0)
+ *spec_check = SPEC_PASS;
+ else
+ *spec_check = SPEC_FAIL;
+ }
+ }
+
+ kfree(temp);
+
+ErrorRelease:
+ /* release data monitory (unprepare AFE data memory) */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_MUTU_RAW_TYPE, &mode, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Set rawdata type failed\n", __func__);
+
+ErrorExit:
+ kfree(pRead);
+
+ return ret;
+}
+
+static void sec_ts_print_channel(struct sec_ts_data *ts)
+{
+ unsigned char *pStr = NULL;
+ unsigned int str_size, str_len = 0;
+ int i = 0, j = 0, k = 0;
+
+ if (!ts->tx_count)
+ return;
+
+ str_size = 7 * (ts->tx_count + 1);
+ pStr = vzalloc(str_size);
+ if (!pStr)
+ return;
+
+ str_len = scnprintf(pStr, str_size, " TX");
+
+ for (k = 0; k < ts->tx_count; k++) {
+ str_len += scnprintf(pStr + str_len, str_size - str_len,
+ " %02d", k);
+ }
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+
+ str_len = scnprintf(pStr, str_size, " +");
+
+ for (k = 0; k < ts->tx_count; k++) {
+ str_len += scnprintf(pStr + str_len, str_size - str_len,
+ "------");
+ }
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+
+ str_len = scnprintf(pStr, str_size, " | ");
+
+ for (i = 0; i < (ts->tx_count + ts->rx_count) * 2; i += 2) {
+ if (j == ts->tx_count) {
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+ input_info(true, &ts->client->dev, "\n");
+ str_len = scnprintf(pStr, str_size, " RX");
+
+ for (k = 0; k < ts->tx_count; k++) {
+ str_len += scnprintf(pStr + str_len,
+ str_size - str_len,
+ " %02d", k);
+ }
+
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+
+ str_len = scnprintf(pStr, str_size, " +");
+
+ for (k = 0; k < ts->tx_count; k++) {
+ str_len += scnprintf(pStr + str_len,
+ str_size - str_len,
+ "------");
+ }
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+
+ str_len = scnprintf(pStr, str_size, " | ");
+ } else if (j && !(j % ts->tx_count)) {
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+ str_len = scnprintf(pStr, str_size, " | ");
+ }
+
+ str_len += scnprintf(pStr + str_len, str_size - str_len, " %5d",
+ ts->pFrame[j]);
+
+ j++;
+ }
+ input_info(true, &ts->client->dev, "%s\n", pStr);
+ vfree(pStr);
+}
+
+static int sec_ts_read_channel(struct sec_ts_data *ts, u8 type, short *min,
+ short *max, enum spec_check_type *spec_check)
+{
+ unsigned char *pRead = NULL;
+ u8 mode = TYPE_INVALID_DATA;
+ int ret = 0;
+ int ii = 0;
+ int jj = 0;
+ unsigned int data_length = (ts->tx_count + ts->rx_count) * 2;
+ u8 w_data;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ pRead = kzalloc(data_length, GFP_KERNEL);
+ if (!pRead)
+ return -ENOMEM;
+
+ /* set OPCODE and data type */
+ w_data = type;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SELF_RAW_TYPE, &w_data, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Set rawdata type failed\n", __func__);
+ goto out_read_channel;
+ }
+
+ sec_ts_delay(50);
+
+ if (type == TYPE_OFFSET_DATA_SDC) {
+ /* excute selftest for real cap offset data, because real cap data is not memory data in normal touch. */
+ char para = TO_TOUCH_MODE;
+ disable_irq(ts->client->irq);
+ execute_selftest(ts, true);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_POWER_MODE, ¶, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: set rawdata type failed!\n", __func__);
+ enable_irq(ts->client->irq);
+ goto err_read_data;
+ }
+ enable_irq(ts->client->irq);
+ /* end */
+ }
+
+ /* read data */
+ ret = ts->sec_ts_i2c_read_heap(ts, SEC_TS_READ_TOUCH_SELF_RAWDATA,
+ pRead, data_length);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: read rawdata failed!\n", __func__);
+ goto err_read_data;
+ }
+
+ /* clear all pFrame data */
+ memset(ts->pFrame, 0x00, data_length);
+
+/* d[00] ~ d[14] : TX channel
+ * d[15] ~ d[51] : none
+ * d[52] ~ d[77] : RX channel
+ * d[78] ~ d[103] : none
+ */
+ for (ii = 0; ii < data_length; ii += 2) {
+ ts->pFrame[jj] = ((pRead[ii] << 8) | pRead[ii + 1]);
+ jj++;
+ }
+
+ sec_ts_print_channel(ts);
+
+ if (*spec_check == SPEC_CHECK) {
+ int specover_count = 0;
+
+ if (type == TYPE_OFFSET_DATA_SDC) {
+ min[0] = min[1] = SHRT_MAX;
+ max[0] = max[1] = SHRT_MIN;
+
+ for (ii = 0; ii < ts->tx_count; ii++) {
+ if (ts->pFrame[ii] > cs_tx_max)
+ specover_count++;
+ if (ts->pFrame[ii] < cs_tx_min)
+ specover_count++;
+
+ min[0] = min(min[0], ts->pFrame[ii]);
+ max[0] = max(max[0], ts->pFrame[ii]);
+ }
+ for (ii = ts->tx_count;
+ ii < ts->tx_count + ts->rx_count; ii++) {
+ if (ts->pFrame[ii] > cs_rx_max)
+ specover_count++;
+ if (ts->pFrame[ii] < cs_rx_min)
+ specover_count++;
+
+ min[1] = min(min[1], ts->pFrame[ii]);
+ max[1] = max(max[1], ts->pFrame[ii]);
+ }
+ }
+
+ input_info(true, &ts->client->dev, "%s: type : %d, specover = %d\n",
+ __func__, type, specover_count);
+
+ if (specover_count == 0 &&
+ (max[0] - min[0]) < cs_tx_mm &&
+ (max[1] - min[1]) < cs_rx_mm)
+ *spec_check = SPEC_PASS;
+ else
+ *spec_check = SPEC_FAIL;
+ }
+
+err_read_data:
+ /* release data monitory (unprepare AFE data memory) */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SELF_RAW_TYPE, &mode, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Set rawdata type failed\n", __func__);
+
+out_read_channel:
+ kfree(pRead);
+
+ return ret;
+}
+
+static int sec_ts_read_gain_table(struct sec_ts_data *ts)
+{
+ int readbytes = ts->tx_count * ts->rx_count;
+ unsigned char *pRead = NULL;
+ short min = 0;
+ short max = 0;
+ int ret;
+ int i;
+
+ /* readbytes : 1 byte for enable/disable info + 1 byte per node */
+ pRead = kzalloc(1 + readbytes, GFP_KERNEL);
+ if (!pRead)
+ return -ENOMEM;
+
+ ret = ts->sec_ts_i2c_read_heap(ts, SEC_TS_CMD_READ_NORM_TABLE, pRead,
+ 1 + readbytes);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: read rawdata failed!\n",
+ __func__);
+ goto ErrorRead;
+ }
+
+ input_info(true, &ts->client->dev, "%s: gain table is %s\n",
+ __func__, pRead[0] ? "On." : "Off.");
+
+ for (i = 0; i < readbytes; i++)
+ ts->pFrame[i] = (short)pRead[i + 1];
+
+ sec_ts_print_frame(ts, &min, &max);
+
+ErrorRead:
+ kfree(pRead);
+
+ return ret;
+}
+
+int sec_ts_read_raw_data(struct sec_ts_data *ts,
+ struct sec_cmd_data *sec, struct sec_ts_test_mode *mode)
+{
+ int ii;
+ int ret = 0;
+ const unsigned int buff_size = ts->tx_count * ts->rx_count *
+ CMD_RESULT_WORD_LEN;
+ unsigned int buff_len = 0;
+ char *buff;
+
+ buff = kzalloc(buff_size, GFP_KERNEL);
+ if (!buff)
+ goto error_alloc_mem;
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev,
+ "%s: [ERROR] Touch is stopped\n", __func__);
+ goto error_power_state;
+ }
+
+ input_info(true, &ts->client->dev, "%s: %d, %s\n",
+ __func__, mode->type, mode->allnode ? "ALL" : "");
+
+ ret = sec_ts_fix_tmode(ts, TOUCH_SYSTEM_MODE_TOUCH,
+ TOUCH_MODE_STATE_TOUCH);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: failed to fix tmode\n",
+ __func__);
+ goto error_test_fail;
+ }
+
+ if (mode->frame_channel)
+ ret = sec_ts_read_channel(ts, mode->type, mode->min,
+ mode->max, &mode->spec_check);
+ else
+ ret = sec_ts_read_frame(ts, mode->type, mode->min,
+ mode->max, &mode->spec_check);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: failed to read frame\n",
+ __func__);
+ goto error_test_fail;
+ }
+
+ if (mode->allnode) {
+ if (mode->frame_channel) {
+ if (mode->spec_check == SPEC_PASS) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "OK %d %d",
+ ts->rx_count, ts->tx_count);
+ } else if (mode->spec_check == SPEC_FAIL) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "NG %d %d",
+ ts->rx_count, ts->tx_count);
+ }
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "\n ");
+ for (ii = 0; ii < (ts->rx_count + ts->tx_count); ii++) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "%3d,", ts->pFrame[ii]);
+ if (ii >= ts->tx_count - 1)
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "\n");
+ }
+ } else {
+ if (mode->spec_check == SPEC_NO_CHECK)
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "\n");
+ else if (mode->spec_check == SPEC_PASS) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "OK %d %d\n",
+ ts->rx_count, ts->tx_count);
+ } else { /* mode->spec_check == SPEC_FAIL) */
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "NG %d %d\n", ts->rx_count,
+ ts->tx_count);
+ }
+ for (ii = 0; ii < (ts->rx_count * ts->tx_count); ii++) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "%3d,", ts->pFrame[ii]);
+ if (ii % ts->tx_count == (ts->tx_count - 1))
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "\n");
+ }
+ }
+ } else {
+ buff_len += scnprintf(buff + buff_len, buff_size - buff_len,
+ "%3d,%3d", mode->min[0], mode->max[0]);
+ }
+
+ ret = sec_ts_release_tmode(ts);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to release tmode\n", __func__);
+ goto error_test_fail;
+ }
+
+ if (!sec)
+ goto out_rawdata;
+ sec_cmd_set_cmd_result(sec, buff, buff_len);
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+out_rawdata:
+ kfree(buff);
+
+ sec_ts_locked_release_all_finger(ts);
+
+ return ret;
+
+error_test_fail:
+error_power_state:
+ kfree(buff);
+error_alloc_mem:
+ if (!sec)
+ return ret;
+
+ sec_cmd_set_cmd_result(sec, "FAIL", 4);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+
+ sec_ts_locked_release_all_finger(ts);
+
+ return ret;
+}
+
+int sec_ts_check_fs_precal(struct sec_ts_data *ts)
+{
+ int i, j;
+ int fail_count = 0;
+ short temp;
+
+ for (i = 0; i < ts->rx_count; i++) {
+ for (j = 0; j < ts->tx_count; j++) {
+ temp = ts->pFrame[i * ts->tx_count + j];
+ if (cm_region[i][j] == REGION_NOTCH)
+ continue;
+ /* check whether fs_precal data is within range */
+ if ((temp > fs_precal_h[i][j]) ||
+ (temp < fs_precal_l[i][j]))
+ fail_count++;
+ }
+ }
+
+ return fail_count;
+}
+
+static void get_fw_ver_bin(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+
+ sec_cmd_set_default_result(sec);
+
+ snprintf(buff, sizeof(buff), "SE-V%02X.%02X.%02X",
+ ts->plat_data->panel_revision, ts->plat_data->img_version_of_bin[2],
+ ts->plat_data->img_version_of_bin[3]);
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_fw_ver_ic(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+ int ret;
+ u8 fw_ver[4];
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_IMG_VERSION, fw_ver, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: firmware version read error\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ snprintf(buff, sizeof(buff), "SE-V%02X.%02X.%02X",
+ ts->plat_data->panel_revision, fw_ver[2], fw_ver[3]);
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_config_ver(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[22] = { 0 };
+
+ sec_cmd_set_default_result(sec);
+
+ snprintf(buff, sizeof(buff), "%s_SE_%02X%02X",
+ ts->plat_data->model_name,
+ ts->plat_data->config_version_of_ic[2], ts->plat_data->config_version_of_ic[3]);
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+}
+
+#ifdef PAT_CONTROL
+static void get_pat_information(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[22] = { 0 };
+
+ sec_cmd_set_default_result(sec);
+
+ /* fixed tune version will be saved at excute autotune */
+ snprintf(buff, sizeof(buff), "P%02XT%04X",
+ ts->cal_count, ts->tune_fix_ver);
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void set_external_factory(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[22] = { 0 };
+
+ sec_cmd_set_default_result(sec);
+
+ ts->external_factory = true;
+ snprintf(buff, sizeof(buff), "OK");
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+}
+#endif
+
+static void get_threshold(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[20] = { 0 };
+ char threshold[2] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ goto err;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_TOUCH_MODE_FOR_THRESHOLD, threshold, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: threshold write type failed. ret: %d\n", __func__, ret);
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ goto err;
+ }
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_TOUCH_THRESHOLD, threshold, 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: read threshold fail!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ goto err;
+ }
+
+ input_info(true, &ts->client->dev, "0x%02X, 0x%02X\n",
+ threshold[0], threshold[1]);
+
+ snprintf(buff, sizeof(buff), "%d", (threshold[0] << 8) | threshold[1]);
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+err:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+}
+
+static void module_off_master(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[3] = { 0 };
+ int ret = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ ret = sec_ts_stop_device(ts);
+
+ if (ret == 0)
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ else
+ snprintf(buff, sizeof(buff), "%s", "NG");
+
+ sec_cmd_set_default_result(sec);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ if (strncmp(buff, "OK", 2) == 0)
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ else
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void module_on_master(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[3] = { 0 };
+ int ret = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ ret = sec_ts_start_device(ts);
+
+ if (ts->input_dev->disabled) {
+ sec_ts_set_lowpowermode(ts, TO_LOWPOWER_MODE);
+ ts->power_status = SEC_TS_STATE_LPM;
+ }
+
+ if (ret == 0)
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ else
+ snprintf(buff, sizeof(buff), "%s", "NG");
+
+ sec_cmd_set_default_result(sec);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ if (strncmp(buff, "OK", 2) == 0)
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ else
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_chip_vendor(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+
+ strncpy(buff, "SEC", sizeof(buff));
+ sec_cmd_set_default_result(sec);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_chip_name(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+
+ if (ts->plat_data->img_version_of_ic[0] == 0x02)
+ strncpy(buff, "MC44", sizeof(buff));
+ else if (ts->plat_data->img_version_of_ic[0] == 0x05)
+ strncpy(buff, "A552", sizeof(buff));
+ else if (ts->plat_data->img_version_of_ic[0] == 0x09)
+ strncpy(buff, "Y661", sizeof(buff));
+ else if (ts->plat_data->img_version_of_ic[0] == 0x10)
+ strncpy(buff, "Y761", sizeof(buff));
+ else
+ strncpy(buff, "N/A", sizeof(buff));
+
+ sec_cmd_set_default_result(sec);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void set_mis_cal_spec(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+ char wreg[5] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->plat_data->mis_cal_check == 0) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] not support, %d\n", __func__);
+ goto NG;
+ } else if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n", __func__);
+ goto NG;
+ } else {
+ if ((sec->cmd_param[0] < 0 || sec->cmd_param[0] > 255) ||
+ (sec->cmd_param[1] < 0 || sec->cmd_param[1] > 255) ||
+ (sec->cmd_param[2] < 0 || sec->cmd_param[2] > 255)) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ goto NG;
+ } else {
+ wreg[0] = sec->cmd_param[0];
+ wreg[1] = sec->cmd_param[1];
+ wreg[2] = sec->cmd_param[2];
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_MIS_CAL_SPEC, wreg, 3);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm write failed. ret: %d\n", __func__, ret);
+ goto NG;
+ } else {
+ input_info(true, &ts->client->dev, "%s: tx gap=%d, rx gap=%d, peak=%d\n", __func__, wreg[0], wreg[1], wreg[2]);
+ sec_ts_delay(20);
+ }
+ }
+ }
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+NG:
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+
+}
+
+/*
+ * ## Mis Cal result ##
+ * FF : initial value in Firmware.
+ * FD : Cal fail case
+ * F4 : i2c fail case (5F)
+ * F3 : i2c fail case (5E)
+ * F2 : power off state
+ * F1 : not support mis cal concept
+ * F0 : initial value in fucntion
+ * 08 : Ambient Ambient condition check(PEAK) result 0 (PASS), 1(FAIL)
+ * 04 : Ambient Ambient condition check(DIFF MAX TX) result 0 (PASS), 1(FAIL)
+ * 02 : Ambient Ambient condition check(DIFF MAX RX) result 0 (PASS), 1(FAIL)
+ * 01 : Wet Wet mode result 0 (PASS), 1(FAIL)
+ * 00 : Pass
+ */
+static void get_mis_cal_info(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+ char mis_cal_data = 0xF0;
+ char wreg[5] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->plat_data->mis_cal_check == 0) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] not support, %d\n", __func__);
+ mis_cal_data = 0xF1;
+ goto NG;
+ } else if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n", __func__);
+ mis_cal_data = 0xF2;
+ goto NG;
+ } else {
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_MIS_CAL_READ, &mis_cal_data, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: i2c fail!, %d\n", __func__, ret);
+ mis_cal_data = 0xF3;
+ goto NG;
+ } else {
+ input_info(true, &ts->client->dev, "%s: miss cal data : %d\n", __func__, mis_cal_data);
+ }
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_MIS_CAL_SPEC, wreg, 3);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: i2c fail!, %d\n", __func__, ret);
+ mis_cal_data = 0xF4;
+ goto NG;
+ } else {
+ input_info(true, &ts->client->dev, "%s: miss cal spec : %d,%d,%d\n", __func__,
+ wreg[0], wreg[1], wreg[2]);
+ }
+ }
+
+ snprintf(buff, sizeof(buff), "%d,%d,%d,%d", mis_cal_data, wreg[0], wreg[1], wreg[2]);
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+NG:
+ snprintf(buff, sizeof(buff), "%d,%d,%d,%d", mis_cal_data, 0, 0, 0);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_wet_mode(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+ char wet_mode_info = 0;
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_WET_MODE, &wet_mode_info, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: i2c fail!, %d\n", __func__, ret);
+ goto NG;
+ }
+
+ snprintf(buff, sizeof(buff), "%d", wet_mode_info);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+NG:
+ snprintf(buff, sizeof(buff), "NG");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_x_num(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+
+ sec_cmd_set_default_result(sec);
+ snprintf(buff, sizeof(buff), "%d", ts->tx_count);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_y_num(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+
+ sec_cmd_set_default_result(sec);
+ snprintf(buff, sizeof(buff), "%d", ts->rx_count);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_x_cross_routing(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+
+ sec_cmd_set_default_result(sec);
+ snprintf(buff, sizeof(buff), "NG");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_y_cross_routing(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+ int ret;
+
+ sec_cmd_set_default_result(sec);
+
+ ret = strncmp(ts->plat_data->model_name, "G935", 4)
+ && strncmp(ts->plat_data->model_name, "N930", 4);
+ if (ret == 0)
+ snprintf(buff, sizeof(buff), "13,14");
+ else
+ snprintf(buff, sizeof(buff), "NG");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_checksum_data(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+ char csum_result[4] = { 0 };
+ u8 cal_result;
+ u8 nv_result;
+ u8 temp;
+ u8 csum = 0;
+ int ret, i;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ goto err;
+ }
+
+ temp = DO_FW_CHECKSUM | DO_PARA_CHECKSUM;
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_GET_CHECKSUM, &temp, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: send get_checksum_cmd fail!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "SendCMDfail");
+ goto err;
+ }
+
+ sec_ts_delay(20);
+
+ ret = ts->sec_ts_i2c_read_bulk(ts, csum_result, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: read get_checksum result fail!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "ReadCSUMfail");
+ goto err;
+ }
+
+ nv_result = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_FAC_RESULT);
+ nv_result += get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+
+ cal_result = sec_ts_read_calibration_report(ts);
+
+ for (i = 0; i < 4; i++)
+ csum += csum_result[i];
+
+ csum += nv_result;
+ csum += cal_result;
+
+ csum = ~csum;
+
+ input_info(true, &ts->client->dev, "%s: checksum = %02X\n", __func__, csum);
+ snprintf(buff, sizeof(buff), "%02X", csum);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+err:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_reference_read(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_OFFSET_DATA_SEC;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_reference_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_OFFSET_DATA_SEC;
+ mode.allnode = TEST_MODE_ALL_NODE;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_reference(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ short val = 0;
+ int node = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ node = sec_ts_check_index(ts);
+ if (node < 0) {
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ val = ts->pFrame[node];
+ snprintf(buff, sizeof(buff), "%d", val);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_rawcap_read(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_OFFSET_DATA_SDC;
+ mode.spec_check = SPEC_CHECK;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_rawcap_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_OFFSET_DATA_SDC;
+ mode.allnode = TEST_MODE_ALL_NODE;
+ mode.spec_check = SPEC_CHECK;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_rawcap(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ short val = 0;
+ int node = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ node = sec_ts_check_index(ts);
+ if (node < 0) {
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ val = ts->pFrame[node];
+ snprintf(buff, sizeof(buff), "%d", val);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_rawcap_gap_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+ int ret_x, ret_y;
+ int i;
+ short *gap_x, *gap_y;
+ short dTmp;
+ char *buff;
+ char temp[SEC_CMD_STR_LEN] = { 0 };
+ const unsigned int buff_size = ts->tx_count * ts->rx_count * 2
+ * CMD_RESULT_WORD_LEN + 4 * CMD_RESULT_WORD_LEN;
+ const unsigned int readbytes = ts->tx_count * ts->rx_count * 2;
+ const unsigned int X_DIR = 0;
+ const unsigned int Y_DIR = 1;
+
+ if (!sec)
+ return;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+
+ gap_x = kzalloc(readbytes, GFP_KERNEL);
+ gap_y = kzalloc(readbytes, GFP_KERNEL);
+ buff = kzalloc(buff_size, GFP_KERNEL);
+ if (!gap_x || !gap_y || !buff) {
+ snprintf(temp, sizeof(temp), "FAIL");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, temp, sizeof(temp));
+ goto ErrorAlloc;
+ }
+
+ mode.type = TYPE_OFFSET_DATA_SDC;
+ mode.allnode = TEST_MODE_ALL_NODE;
+ mode.spec_check = SPEC_NO_CHECK;
+
+ sec_ts_read_frame(ts, mode.type, mode.min, mode.max,
+ &mode.spec_check);
+
+ ret_x = sec_ts_cm_spec_over_check(ts, gap_x, X_DIR);
+
+ ret_y = sec_ts_cm_spec_over_check(ts, gap_y, Y_DIR);
+
+ if (0 == (ret_x + ret_y)) {
+ strlcat(buff, "OK", buff_size);
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ } else {
+ strlcat(buff, "NG", buff_size);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ }
+ strlcat(buff, "\n", buff_size);
+ for (i = 0; i < (ts->tx_count * ts->rx_count); i++) {
+ dTmp = (gap_x[i] > gap_y[i]) ? gap_x[i] : gap_y[i];
+ snprintf(temp, sizeof(temp), "%3d,", dTmp);
+ strlcat(buff, temp, buff_size);
+ if (i % ts->tx_count == (ts->tx_count - 1))
+ strlcat(buff, "\n", buff_size);
+ memset(temp, 0x00, sizeof(temp));
+ }
+ strlcat(buff, "\n", buff_size);
+
+ sec_cmd_set_cmd_result(sec, buff, buff_size);
+
+ErrorAlloc:
+ kfree(buff);
+ kfree(gap_y);
+ kfree(gap_x);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_delta_read(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_SIGNAL_DATA;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_delta_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_SIGNAL_DATA;
+ mode.allnode = TEST_MODE_ALL_NODE;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_delta(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ short val = 0;
+ int node = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ node = sec_ts_check_index(ts);
+ if (node < 0) {
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ val = ts->pFrame[node];
+ snprintf(buff, sizeof(buff), "%d", val);
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static int sec_ts_read_frame_stdev(struct sec_ts_data *ts,
+ struct sec_cmd_data *sec, u8 type, short *min, short *max,
+ enum spec_check_type *spec_check, bool get_average_only)
+{
+ unsigned char *pRead = NULL;
+ short *pFrameAll = NULL;
+ int *pFrameAvg = NULL;
+ u64 *pFrameStd = NULL;
+ u8 inval_type = TYPE_INVALID_DATA;
+ int node_tot = 0;
+ int ret = 0;
+ int i = 0;
+ int j = 0;
+ int frame_len_byte = 0;
+ int frame_cnt = 0;
+ int frame_tot = 0;
+ int tmp = 0;
+
+ const unsigned int buff_size = ts->tx_count * ts->rx_count *
+ CMD_RESULT_WORD_LEN;
+ unsigned int buff_len = 0;
+ char *pBuff;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ frame_tot = 100;
+
+ /* set data length, allocation buffer memory */
+
+ ret = -ENOMEM;
+ pBuff = kzalloc(buff_size, GFP_KERNEL);
+ if (!pBuff)
+ goto ErrorAlloc;
+
+ /* each node data 2bytes : 1frame bytes = node_tot * 2 */
+ node_tot = ts->rx_count * ts->tx_count;
+ frame_len_byte = node_tot * 2;
+
+ pRead = kzalloc(frame_len_byte, GFP_KERNEL);
+ if (!pRead)
+ goto ErrorAlloc;
+
+ /* memory whole frame data : 1frame bytes * total frame */
+ pFrameAll = kzalloc(frame_len_byte * frame_tot, GFP_KERNEL);
+ if (!pFrameAll)
+ goto ErrorAlloc;
+
+ /* float type : type size is double */
+ pFrameAvg = kzalloc(frame_len_byte * 2, GFP_KERNEL);
+ if (!pFrameAvg)
+ goto ErrorAlloc;
+
+ /* 64-bit to prevent overflow */
+ pFrameStd = kzalloc(frame_len_byte * 4, GFP_KERNEL);
+ if (!pFrameStd)
+ goto ErrorAlloc;
+
+ /* fix touch mode */
+ ret = sec_ts_fix_tmode(ts, TOUCH_SYSTEM_MODE_TOUCH,
+ TOUCH_MODE_STATE_TOUCH);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: failed to fix tmode\n",
+ __func__);
+ goto ErrorAlloc;
+ }
+
+ /* set OPCODE and data type */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_MUTU_RAW_TYPE, &type, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: Set rawdata type failed\n", __func__);
+ goto ErrorDataType;
+ }
+
+ sec_ts_delay(50);
+
+ for (frame_cnt = 0; frame_cnt < frame_tot; frame_cnt++) {
+ /* read data */
+ ret = ts->sec_ts_i2c_read_heap(ts, SEC_TS_READ_TOUCH_RAWDATA,
+ pRead, frame_len_byte);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: read rawdata failed!\n", __func__);
+ goto ErrorRelease;
+ }
+
+ memset(ts->pFrame, 0x00, frame_len_byte);
+
+ for (i = 0; i < frame_len_byte; i += 2) {
+ ts->pFrame[i / 2] = pRead[i + 1] + (pRead[i] << 8);
+ pFrameAvg[i / 2] += ts->pFrame[i / 2];
+ }
+
+ memcpy(pFrameAll + (frame_len_byte * frame_cnt) / sizeof(short),
+ ts->pFrame, frame_len_byte);
+ }
+
+ /* get total frame average of each node */
+ /* in the case of getting only average, *1000 not needed */
+ for (j = 0; j < node_tot; j++) {
+ if (!get_average_only)
+ pFrameAvg[j] = pFrameAvg[j] * 1000;
+ pFrameAvg[j] = pFrameAvg[j] / frame_tot;
+ }
+
+ input_info(true, &ts->client->dev, "%s: FrameAvg x 1000\n", __func__);
+
+ /* print frame average x 1000 of each node */
+ for (i = 0; i < ts->rx_count; i++) {
+ buff_len = scnprintf(pBuff, buff_size, "Rx%02d | ", i);
+
+ for (j = 0; j < ts->tx_count; j++) {
+ buff_len += scnprintf(pBuff + buff_len,
+ buff_size - buff_len,
+ " %6d",
+ pFrameAvg[(j * ts->rx_count) + i]);
+ }
+ input_info(true, &ts->client->dev, "%s\n", pBuff);
+ }
+
+ /* when only getting average, put average in
+ * ts->pFrame and goto set cmd_result
+ */
+ if (get_average_only) {
+ for (i = 0; i < ts->tx_count; i++) {
+ for (j = 0; j < ts->rx_count; j++) {
+ ts->pFrame[(j * ts->tx_count) + i] =
+ (short)(pFrameAvg[(i * ts->rx_count) +
+ j]);
+ }
+ }
+ goto OnlyAverage;
+ }
+
+ /* get standard deviation */
+ for (i = 0; i < frame_tot; i++) {
+ for (j = 0; j < node_tot; j++) {
+ tmp = pFrameAll[node_tot * i + j] * 1000;
+ pFrameStd[j] = pFrameStd[j] +
+ (tmp - pFrameAvg[j]) * (tmp - pFrameAvg[j]);
+ }
+ }
+
+ for (j = 0; j < node_tot; j++)
+ pFrameStd[j] = int_sqrt(pFrameStd[j] / frame_tot);
+
+ /* print standard deviation x 1000 of each node */
+ input_info(true, &ts->client->dev, "%s: FrameStd x 1000\n", __func__);
+
+ *min = *max = pFrameStd[0];
+
+ for (i = 0; i < ts->rx_count; i++) {
+ buff_len = scnprintf(pBuff, buff_size, "Rx%02d | ", i);
+
+ for (j = 0; j < ts->tx_count; j++) {
+ if (i > 0) {
+ if (pFrameStd[(j * ts->rx_count) + i] < *min)
+ *min =
+ pFrameStd[(j * ts->rx_count) + i];
+
+ if (pFrameStd[(j * ts->rx_count) + i] > *max)
+ *max =
+ pFrameStd[(j * ts->rx_count) + i];
+ }
+ buff_len += scnprintf(pBuff + buff_len,
+ buff_size - buff_len,
+ " %6d",
+ (int)pFrameStd[(j * ts->rx_count) + i]);
+ }
+ input_info(true, &ts->client->dev, "%s\n", pBuff);
+ }
+ // SQRT(VAR)
+
+ /* Rotate 90 degrees for readability */
+ for (i = 0; i < ts->tx_count; i++) {
+ for (j = 0; j < ts->rx_count; j++) {
+ if (pFrameStd[(i * ts->rx_count) + j] > 32767)
+ /* Reduce to short data type and allow high
+ * values to saturate.
+ */
+ ts->pFrame[(j * ts->tx_count) + i] = 32767;
+ else {
+ ts->pFrame[(j * ts->tx_count) + i] =
+ (short)(pFrameStd[(i * ts->rx_count) +
+ j]);
+ }
+ }
+ }
+
+ if (*spec_check == SPEC_CHECK) {
+ int specover_count = 0;
+
+ for (i = 0; i < ts->tx_count; i++) {
+ for (j = 0; j < ts->rx_count; j++) {
+ if (ts->pFrame[(j * ts->tx_count) + i] >
+ cm_stdev_max)
+ specover_count++;
+ }
+ }
+
+ if (specover_count == 0)
+ *spec_check = SPEC_PASS;
+ else
+ *spec_check = SPEC_FAIL;
+ }
+
+ if (*spec_check == SPEC_PASS)
+ buff_len = scnprintf(pBuff, buff_size, "OK %d %d\n",
+ ts->rx_count, ts->tx_count);
+ else if (*spec_check == SPEC_FAIL)
+ buff_len = scnprintf(pBuff, buff_size, "NG %d %d\n",
+ ts->rx_count, ts->tx_count);
+ else
+ buff_len = scnprintf(pBuff, buff_size, "\n");
+
+ for (i = 0; i < node_tot; i++) {
+ buff_len += scnprintf(pBuff + buff_len, buff_size - buff_len,
+ "%4d,", ts->pFrame[i]);
+
+ if (i % ts->tx_count == ts->tx_count - 1)
+ buff_len += scnprintf(pBuff + buff_len,
+ buff_size - buff_len, "\n");
+ }
+
+ if (!sec)
+ goto ErrorRelease;
+
+ sec_cmd_set_cmd_result(sec, pBuff, buff_len);
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ErrorRelease:
+OnlyAverage:
+ /* release data monitory (unprepare AFE data memory) */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_MUTU_RAW_TYPE, &inval_type,
+ 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: Set rawdata type failed\n", __func__);
+ goto ErrorAlloc;
+ }
+
+ErrorDataType:
+ /* release mode fix */
+ ret = sec_ts_release_tmode(ts);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: failed to release tmode\n",
+ __func__);
+ }
+
+ErrorAlloc:
+ kfree(pFrameStd);
+ kfree(pFrameAvg);
+ kfree(pFrameAll);
+ kfree(pRead);
+ kfree(pBuff);
+
+ return ret;
+}
+
+static void run_rawdata_stdev_read(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_REMV_AMB_DATA;
+ mode.spec_check = SPEC_CHECK;
+
+ sec_ts_read_frame_stdev(ts, sec, mode.type, mode.min, mode.max,
+ &mode.spec_check, false);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static int sec_ts_read_frame_p2p(struct sec_ts_data *ts,
+ struct sec_cmd_data *sec, struct sec_ts_test_mode mode)
+{
+ const unsigned int frame_size = ts->rx_count * ts->tx_count * 2;
+ short *temp = NULL;
+ unsigned short readbytes;
+ int i;
+ int ret = -1;
+ char para = TO_TOUCH_MODE;
+ const unsigned int buff_size = ts->tx_count * ts->rx_count *
+ CMD_RESULT_WORD_LEN;
+ unsigned int buff_len = 0;
+ char *buff;
+ u8 result = 0x0;
+
+ buff = kzalloc(buff_size, GFP_KERNEL);
+ if (!buff)
+ goto ErrorAllocbuff;
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev,
+ "%s: [ERROR] Touch is stopped\n", __func__);
+ goto ErrorPowerState;
+ }
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+
+ disable_irq(ts->client->irq);
+
+ ret = execute_p2ptest(ts);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: P2P test failed\n",
+ __func__);
+ goto ErrorP2PTest;
+ }
+
+ /* get min data */
+ mode.type = TYPE_NOI_P2P_MIN;
+ mode.spec_check = SPEC_CHECK;
+ sec_ts_read_frame(ts, mode.type, mode.min, mode.max,
+ &mode.spec_check);
+ if (mode.spec_check == SPEC_FAIL)
+ result |= 0x1;
+
+ readbytes = ts->rx_count * ts->tx_count;
+
+ /* 2 bytes for each node data */
+ temp = kzalloc(frame_size, GFP_KERNEL);
+ if (!temp)
+ goto ErrorAlloctemp;
+
+ memcpy(temp, ts->pFrame, frame_size);
+ memset(ts->pFrame, 0x00, frame_size);
+
+ /* get max data */
+ mode.type = TYPE_NOI_P2P_MAX;
+ mode.spec_check = SPEC_CHECK;
+ sec_ts_read_frame(ts, mode.type, mode.min, mode.max,
+ &mode.spec_check);
+ if (mode.spec_check == SPEC_FAIL)
+ result |= 0x2;
+
+ for (i = 0; i < readbytes; i++) {
+ /* get p2p by subtract min from max data */
+ ts->pFrame[i] = ts->pFrame[i] - temp[i];
+ if (ts->pFrame[i] > noi_mm)
+ result |= 0x4;
+ }
+
+ if (result != 0x0)
+ buff_len += scnprintf(buff + buff_len, buff_size - buff_len,
+ "NG\n");
+ else
+ buff_len += scnprintf(buff + buff_len, buff_size - buff_len,
+ "OK\n");
+
+ for (i = 0; i < readbytes; i++) {
+ buff_len += scnprintf(buff + buff_len, buff_size - buff_len,
+ "%3d,", ts->pFrame[i]);
+ if (i % ts->tx_count == (ts->tx_count - 1))
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "\n");
+ }
+
+ if (sec) {
+ sec_cmd_set_cmd_result(sec, buff, buff_len);
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+
+ kfree(temp);
+ErrorAlloctemp:
+ErrorP2PTest:
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_POWER_MODE, ¶, 1);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Set powermode failed\n",
+ __func__);
+
+ enable_irq(ts->client->irq);
+ErrorPowerState:
+ kfree(buff);
+ErrorAllocbuff:
+
+ if (sec && ret < 0) {
+ sec_cmd_set_cmd_result(sec, "NG", 3);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ }
+
+ return ret;
+}
+
+static void run_rawdata_p2p_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+
+ sec_ts_read_frame_p2p(ts, sec, mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+/* self reference : send TX power in TX channel, receive in TX channel */
+static void run_self_reference_read(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_OFFSET_DATA_SEC;
+ mode.frame_channel = TEST_MODE_READ_CHANNEL;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_self_reference_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_OFFSET_DATA_SEC;
+ mode.frame_channel = TEST_MODE_READ_CHANNEL;
+ mode.allnode = TEST_MODE_ALL_NODE;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_self_rawcap_read(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_OFFSET_DATA_SDC;
+ mode.frame_channel = TEST_MODE_READ_CHANNEL;
+ mode.spec_check = SPEC_CHECK;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_self_rawcap_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_OFFSET_DATA_SDC;
+ mode.frame_channel = TEST_MODE_READ_CHANNEL;
+ mode.allnode = TEST_MODE_ALL_NODE;
+ mode.spec_check = SPEC_CHECK;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_self_rawcap_gap_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+ int i;
+ int ret = 0;
+ char *buff = NULL;
+ short *gap = NULL;
+ const int gap_buff_size = (ts->tx_count - 1) + (ts->rx_count - 1);
+ const int buff_size = gap_buff_size * CMD_RESULT_WORD_LEN + 4;
+ unsigned int buff_len = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_OFFSET_DATA_SDC;
+ mode.frame_channel = TEST_MODE_READ_CHANNEL;
+ mode.allnode = TEST_MODE_ALL_NODE;
+
+ gap = kzalloc(gap_buff_size, GFP_KERNEL);
+ buff = kzalloc(buff_size, GFP_KERNEL);
+ if (!gap || !buff) {
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, "FAIL", 4);
+ goto ErrorAlloc;
+ }
+
+ sec_ts_read_channel(ts, mode.type, mode.min, mode.max,
+ &mode.spec_check);
+
+ /* ret is number of spec over channel */
+ ret = sec_ts_cs_spec_over_check(ts, gap);
+
+ if (ret == 0) {
+ buff_len = scnprintf(buff, buff_size, "OK\n ");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ } else {
+ buff_len = scnprintf(buff, buff_size, "NG\n ");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ }
+
+ for (i = 0; i < (ts->tx_count - 1); i++) {
+ buff_len += scnprintf(buff + buff_len, buff_size - buff_len,
+ "%6d,", gap[i]);
+ }
+ buff_len += scnprintf(buff + buff_len, buff_size - buff_len, "\n");
+
+ for (i = ts->tx_count; i < ts->tx_count + (ts->rx_count - 1); i++) {
+ buff_len += scnprintf(buff + buff_len, buff_size - buff_len,
+ "%6d,\n", gap[i]);
+ }
+
+ sec_cmd_set_cmd_result(sec, buff, buff_len);
+
+ErrorAlloc:
+ kfree(buff);
+ kfree(gap);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_self_delta_read(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_SIGNAL_DATA;
+ mode.frame_channel = TEST_MODE_READ_CHANNEL;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_self_delta_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_SIGNAL_DATA;
+ mode.frame_channel = TEST_MODE_READ_CHANNEL;
+ mode.allnode = TEST_MODE_ALL_NODE;
+
+ sec_ts_read_raw_data(ts, sec, &mode);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+/* Use TSP NV area
+ * buff[0] : offset from user NVM storage
+ * buff[1] : length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * buff[2] : write data
+ * buff[..] : cont.
+ */
+void set_tsp_nvm_data_clear(struct sec_ts_data *ts, u8 offset)
+{
+ char buff[4] = { 0 };
+ int ret;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ buff[0] = offset;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, buff, 3);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: nvm write failed. ret: %d\n", __func__, ret);
+
+ sec_ts_delay(20);
+}
+
+int get_tsp_nvm_data(struct sec_ts_data *ts, u8 offset)
+{
+ char buff[2] = { 0 };
+ int ret;
+
+ /* SENSE OFF -> CELAR EVENT STACK -> READ NV -> SENSE ON */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_OFF, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to write Sense_off\n", __func__);
+ goto out_nvm;
+ }
+
+ input_dbg(true, &ts->client->dev, "%s: SENSE OFF\n", __func__);
+
+ sec_ts_delay(100);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CLEAR_EVENT_STACK, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: i2c write clear event failed\n", __func__);
+ goto out_nvm;
+ }
+
+ input_dbg(true, &ts->client->dev, "%s: CLEAR EVENT STACK\n", __func__);
+
+ sec_ts_delay(100);
+
+ sec_ts_locked_release_all_finger(ts);
+
+ /* send NV data using command
+ * Use TSP NV area : in this model, use only one byte
+ * buff[0] : offset from user NVM storage
+ * buff[1] : length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ */
+ memset(buff, 0x00, 2);
+ buff[0] = offset;
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, buff, 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm send command failed. ret: %d\n", __func__, ret);
+ goto out_nvm;
+ }
+
+ sec_ts_delay(20);
+
+ /* read NV data
+ * Use TSP NV area : in this model, use only one byte
+ */
+ ret = ts->sec_ts_i2c_read_bulk(ts, buff, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm send command failed. ret: %d\n", __func__, ret);
+ goto out_nvm;
+ }
+
+ input_info(true, &ts->client->dev, "%s: offset:%u data:%02X\n", __func__,offset, buff[0]);
+
+out_nvm:
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_ON, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to write Sense_on\n", __func__);
+
+ input_dbg(true, &ts->client->dev, "%s: SENSE ON\n", __func__);
+
+ return buff[0];
+}
+
+int get_tsp_nvm_data_by_size(struct sec_ts_data *ts, u8 offset, int length, u8 *data)
+{
+ char *buff = NULL;
+ int ret;
+
+ buff = kzalloc(length, GFP_KERNEL);
+ if (!buff)
+ return -ENOMEM;
+
+ input_info(true, &ts->client->dev, "%s: offset:%u, length:%d, size:%d\n", __func__, offset, length, sizeof(data));
+
+ /* SENSE OFF -> CELAR EVENT STACK -> READ NV -> SENSE ON */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_OFF, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to write Sense_off\n", __func__);
+ goto out_nvm;
+ }
+
+ input_dbg(true, &ts->client->dev, "%s: SENSE OFF\n", __func__);
+
+ sec_ts_delay(100);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CLEAR_EVENT_STACK, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: i2c write clear event failed\n", __func__);
+ goto out_nvm;
+ }
+
+ input_dbg(true, &ts->client->dev, "%s: CLEAR EVENT STACK\n", __func__);
+
+ sec_ts_delay(100);
+
+ sec_ts_locked_release_all_finger(ts);
+
+ /* send NV data using command
+ * Use TSP NV area : in this model, use only one byte
+ * buff[0] : offset from user NVM storage
+ * buff[1] : length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ */
+ memset(buff, 0x00, 2);
+ buff[0] = offset;
+ buff[1] = length - 1;
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, buff, 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm send command failed. ret: %d\n", __func__, ret);
+ goto out_nvm;
+ }
+
+ sec_ts_delay(20);
+
+ /* read NV data
+ * Use TSP NV area : in this model, use only one byte
+ */
+ ret = ts->sec_ts_i2c_read_bulk_heap(ts, buff, length);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm send command failed. ret: %d\n", __func__, ret);
+ goto out_nvm;
+ }
+
+ memcpy(data, buff, length);
+
+out_nvm:
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_ON, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to write Sense_on\n", __func__);
+
+ input_dbg(true, &ts->client->dev, "%s: SENSE ON\n", __func__);
+
+ kfree(buff);
+
+ return ret;
+}
+
+#ifdef PAT_CONTROL
+void set_pat_magic_number(struct sec_ts_data *ts)
+{
+ char buff[4] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+ buff[0] = SEC_TS_NVM_OFFSET_CAL_COUNT;
+ buff[1] = 0;
+ buff[2] = PAT_MAGIC_NUMBER;
+
+ input_info(true, &ts->client->dev, "%s: %02X\n", __func__, buff[2]);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, buff, 3);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: nvm write failed. ret: %d\n", __func__, ret);
+ }
+ sec_ts_delay(20);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+#endif
+
+
+/* FACTORY TEST RESULT SAVING FUNCTION
+ * bit 3 ~ 0 : OCTA Assy
+ * bit 7 ~ 4 : OCTA module
+ * param[0] : OCTA modue(1) / OCTA Assy(2)
+ * param[1] : TEST NONE(0) / TEST FAIL(1) / TEST PASS(2) : 2 bit
+ */
+
+#define TEST_OCTA_MODULE 1
+#define TEST_OCTA_ASSAY 2
+
+#define TEST_OCTA_NONE 0
+#define TEST_OCTA_FAIL 1
+#define TEST_OCTA_PASS 2
+
+static void set_tsp_test_result(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_result *result;
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ char r_data[1] = { 0 };
+ int ret = 0;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP_truned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ r_data[0] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_FAC_RESULT);
+ if (r_data[0] == 0xFF)
+ r_data[0] = 0;
+
+ result = (struct sec_ts_test_result *)r_data;
+
+ if (sec->cmd_param[0] == TEST_OCTA_ASSAY) {
+ result->assy_result = sec->cmd_param[1];
+ if (result->assy_count < 3)
+ result->assy_count++;
+ }
+
+ if (sec->cmd_param[0] == TEST_OCTA_MODULE) {
+ result->module_result = sec->cmd_param[1];
+ if (result->module_count < 3)
+ result->module_count++;
+ }
+
+ input_info(true, &ts->client->dev, "%s: %d, %d, %d, %d, 0x%X\n", __func__,
+ result->module_result, result->module_count,
+ result->assy_result, result->assy_count, result->data[0]);
+
+ /* Use TSP NV area : in this model, use only one byte
+ * buff[0] : offset from user NVM storage
+ * buff[1] : length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * buff[2] : write data
+ */
+ memset(buff, 0x00, SEC_CMD_STR_LEN);
+ buff[2] = *result->data;
+
+ input_info(true, &ts->client->dev, "%s: command (1)%X, (2)%X: %X\n",
+ __func__, sec->cmd_param[0], sec->cmd_param[1], buff[2]);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, buff, 3);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: nvm write failed. ret: %d\n", __func__, ret);
+
+ sec_ts_delay(20);
+
+ ts->nv = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_FAC_RESULT);
+
+ snprintf(buff, sizeof(buff), "OK");
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_tsp_test_result(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ struct sec_ts_test_result *result;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev,
+ "%s: [ERROR] Touch is stopped\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP_truned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ memset(buff, 0x00, SEC_CMD_STR_LEN);
+ buff[0] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_FAC_RESULT);
+ if (buff[0] == 0xFF) {
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_FAC_RESULT);
+ buff[0] = 0;
+ }
+
+ ts->nv = buff[0];
+
+ result = (struct sec_ts_test_result *)buff;
+
+ input_info(true, &ts->client->dev, "%s: [0x%X][0x%X] M:%d, M:%d, A:%d, A:%d\n",
+ __func__, *result->data, buff[0],
+ result->module_result, result->module_count,
+ result->assy_result, result->assy_count);
+
+ snprintf(buff, sizeof(buff), "M:%s, M:%d, A:%s, A:%d",
+ result->module_result == 0 ? "NONE" :
+ result->module_result == 1 ? "FAIL" : "PASS", result->module_count,
+ result->assy_result == 0 ? "NONE" :
+ result->assy_result == 1 ? "FAIL" : "PASS", result->assy_count);
+
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void increase_disassemble_count(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[3] = { 0 };
+ int ret = 0;
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP_truned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ buff[2] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_DISASSEMBLE_COUNT);
+
+ input_info(true, &ts->client->dev, "%s: disassemble count is #1 %d\n", __func__, buff[2]);
+
+ if (buff[2] == 0xFF)
+ buff[2] = 0;
+
+ if (buff[2] < 0xFE)
+ buff[2]++;
+
+ /* Use TSP NV area : in this model, use only one byte
+ * buff[0] : offset from user NVM storage
+ * buff[1] : length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * buff[2] : write data
+ */
+ buff[0] = SEC_TS_NVM_OFFSET_DISASSEMBLE_COUNT;
+ buff[1] = 0;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, buff, 3);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: nvm write failed. ret: %d\n", __func__, ret);
+
+ sec_ts_delay(20);
+
+ memset(buff, 0x00, 3);
+ buff[0] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_DISASSEMBLE_COUNT);
+ input_info(true, &ts->client->dev, "%s: check disassemble count: %d\n", __func__, buff[0]);
+
+ snprintf(buff, sizeof(buff), "OK");
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_disassemble_count(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+
+ input_info(true, &ts->client->dev, "%s\n", __func__);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP_truned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ memset(buff, 0x00, SEC_CMD_STR_LEN);
+ buff[0] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_DISASSEMBLE_COUNT);
+ if (buff[0] == 0xFF) {
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_DISASSEMBLE_COUNT);
+ buff[0] = 0;
+ }
+
+ input_info(true, &ts->client->dev, "%s: read disassemble count: %d\n", __func__, buff[0]);
+
+ snprintf(buff, sizeof(buff), "%d", buff[0]);
+
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+#define GLOVE_MODE_EN (1 << 0)
+#define CLEAR_COVER_EN (1 << 1)
+#define FAST_GLOVE_MODE_EN (1 << 2)
+
+static void glove_mode(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ int glove_mode_enables = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) {
+ snprintf(buff, sizeof(buff), "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ int retval;
+
+ if (sec->cmd_param[0])
+ glove_mode_enables |= GLOVE_MODE_EN;
+ else
+ glove_mode_enables &= ~(GLOVE_MODE_EN);
+
+ retval = sec_ts_glove_mode_enables(ts, glove_mode_enables);
+
+ if (retval < 0) {
+ input_err(true, &ts->client->dev, "%s: failed, retval = %d\n", __func__, retval);
+ snprintf(buff, sizeof(buff), "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ snprintf(buff, sizeof(buff), "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+ }
+
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+ sec->cmd_state = SEC_CMD_STATUS_WAITING;
+ sec_cmd_set_cmd_exit(sec);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void clear_cover_mode(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+
+ input_info(true, &ts->client->dev, "%s: start clear_cover_mode %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 3) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ if (sec->cmd_param[0] > 1) {
+ ts->flip_enable = true;
+ ts->cover_type = sec->cmd_param[1];
+ ts->cover_cmd = (u8)ts->cover_type;
+ } else {
+ ts->flip_enable = false;
+ }
+
+ if (!(ts->power_status == SEC_TS_STATE_POWER_OFF) && ts->reinit_done) {
+ if (ts->flip_enable)
+ sec_ts_set_cover_type(ts, true);
+ else
+ sec_ts_set_cover_type(ts, false);
+ }
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_WAITING;
+ sec_cmd_set_cmd_exit(sec);
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+};
+
+static void dead_zone_enable(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ int ret;
+ char data = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ data = sec->cmd_param[0];
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_EDGE_DEADZONE, &data, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to set deadzone\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto err_set_dead_zone;
+ }
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+
+err_set_dead_zone:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+};
+
+
+static void drawing_test_enable(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ if (ts->use_customlib) {
+ if (sec->cmd_param[0])
+ ts->lowpower_mode &= ~SEC_TS_MODE_CUSTOMLIB_FORCE_KEY;
+ else
+ ts->lowpower_mode |= SEC_TS_MODE_CUSTOMLIB_FORCE_KEY;
+
+ #ifdef SEC_TS_SUPPORT_CUSTOMLIB
+ ret = sec_ts_set_custom_library(ts);
+ if (ret < 0) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+ #endif
+
+ } else {
+ snprintf(buff, sizeof(buff), "%s", "NA");
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ }
+ }
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+};
+
+static void sec_ts_swap(u8 *a, u8 *b)
+{
+ u8 temp = *a;
+
+ *a = *b;
+ *b = temp;
+}
+
+static void rearrange_sft_result(u8 *data, int length)
+{
+ int i;
+
+ for(i = 0; i < length; i += 4) {
+ sec_ts_swap(&data[i], &data[i + 3]);
+ sec_ts_swap(&data[i + 1], &data[i + 2]);
+ }
+}
+
+int execute_p2ptest(struct sec_ts_data *ts)
+{
+ int rc;
+ u8 tpara[2] = {0x0F, 0x11};
+
+ input_info(true, &ts->client->dev, "%s: P2P test start!\n", __func__);
+ rc = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_P2PTEST_MODE, tpara, 2);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: Send P2Ptest Mode cmd failed!\n", __func__);
+ goto err_exit;
+ }
+
+ sec_ts_delay(15);
+
+ tpara[0] = 0x00;
+ tpara[1] = 0x64;
+ rc = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_P2PTEST, tpara, 2);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: Send P2Ptest cmd failed!\n", __func__);
+ goto err_exit;
+ }
+
+ sec_ts_delay(1000);
+
+ rc = sec_ts_wait_for_ready(ts, SEC_TS_VENDOR_ACK_P2P_TEST_DONE);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: P2Ptest execution time out!\n", __func__);
+ goto err_exit;
+ }
+
+ input_info(true, &ts->client->dev, "%s: P2P test done!\n", __func__);
+
+err_exit:
+ return rc;
+}
+
+/* execute_selftest options
+ * bit[7] : Do NOT save
+ * bit[6] : Load self-test configuration only
+ * bit[5] : Get Self capacitance
+ * bit[4] : Enable/Disable Force test
+ * bit[3] : Reserved
+ * bit[2] : Enable/disable the short test
+ * bit[1] : Enable/disable the node variance test
+ * bit[0] : Enable/disable the open test
+ */
+int execute_selftest(struct sec_ts_data *ts, bool save_result)
+{
+ int rc;
+ /* Selftest setting
+ * Get self capacitance
+ * Enable/disable the short test
+ * Enable/disable the node variance test
+ * Enable/disable the open test
+ */
+ u8 tpara[2] = {0x27, 0x40};
+ u8 *rBuff;
+ int i;
+ int result_size = SEC_TS_SELFTEST_REPORT_SIZE + ts->tx_count * ts->rx_count * 2;
+
+ /* don't save selftest result in flash */
+ if (!save_result) {
+ tpara[0] = 0xA7;
+ tpara[1] = 0x00;
+ }
+
+ rBuff = kzalloc(result_size, GFP_KERNEL);
+ if (!rBuff)
+ return -ENOMEM;
+
+ input_info(true, &ts->client->dev, "%s: Self test start!\n", __func__);
+ rc = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SELFTEST, tpara, 2);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev, "%s: Send selftest cmd failed!\n", __func__);
+ goto err_exit;
+ }
+
+ sec_ts_delay(350);
+
+ rc = sec_ts_wait_for_ready(ts, SEC_TS_VENDOR_ACK_SELF_TEST_DONE);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev, "%s: Selftest execution time out!\n", __func__);
+ goto err_exit;
+ }
+
+ input_info(true, &ts->client->dev, "%s: Self test done!\n", __func__);
+
+ rc = ts->sec_ts_i2c_read_heap(ts, SEC_TS_READ_SELFTEST_RESULT, rBuff,
+ result_size);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev, "%s: Selftest execution time out!\n", __func__);
+ goto err_exit;
+ }
+ rearrange_sft_result(rBuff, result_size);
+
+ for (i = 0; i < 80; i += 4) {
+ if (i % 8 == 0) pr_cont("\n");
+ if (i % 4 == 0) pr_cont("%s sec_ts : ", SECLOG);
+
+ if (i / 4 == 0) pr_cont("SIG");
+ else if (i / 4 == 1) pr_cont("VER");
+ else if (i / 4 == 2) pr_cont("SIZ");
+ else if (i / 4 == 3) pr_cont("CRC");
+ else if (i / 4 == 4) pr_cont("RES");
+ else if (i / 4 == 5) pr_cont("COU");
+ else if (i / 4 == 6) pr_cont("PAS");
+ else if (i / 4 == 7) pr_cont("FAI");
+ else if (i / 4 == 8) pr_cont("CHA");
+ else if (i / 4 == 9) pr_cont("AMB");
+ else if (i / 4 == 10) pr_cont("RXS");
+ else if (i / 4 == 11) pr_cont("TXS");
+ else if (i / 4 == 12) pr_cont("RXO");
+ else if (i / 4 == 13) pr_cont("TXO");
+ else if (i / 4 == 14) pr_cont("RXG");
+ else if (i / 4 == 15) pr_cont("TXG");
+ else if (i / 4 == 16) pr_cont("RXR");
+ else if (i / 4 == 17) pr_cont("TXT");
+ else if (i / 4 == 18) pr_cont("RXT");
+ else if (i / 4 == 19) pr_cont("TXR");
+
+ pr_cont(" %2X, %2X, %2X, %2X ", rBuff[i], rBuff[i + 1], rBuff[i + 2], rBuff[i + 3]);
+
+
+ if (i / 4 == 4) {
+ /* RX, RX open check. */
+ if ((rBuff[i + 3] & 0x30) != 0)
+ rc = 0;
+ /* TX, RX GND(VDD) short check. */
+ else if ((rBuff[i + 3] & 0xC0) != 0)
+ rc = 0;
+ /* RX-RX, TX-TX short check. */
+ else if ((rBuff[i + 2] & 0x03) != 0)
+ rc = 0;
+ /* TX-RX short check. */
+ else if ((rBuff[i + 2] & 0x04) != 0)
+ rc = 0;
+ else
+ rc = 1;
+
+ ts->ito_test[0] = rBuff[i];
+ ts->ito_test[1] = rBuff[i + 1];
+ ts->ito_test[2] = rBuff[i + 2];
+ ts->ito_test[3] = rBuff[i + 3];
+ }
+
+ }
+
+err_exit:
+ kfree(rBuff);
+ return rc;
+}
+
+static void run_fs_cal_pre_press(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int ret = 0;
+ u8 off[1] = {STATE_MANAGE_OFF};
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ input_info(true, &ts->client->dev, "%s: initial sequence for fs cal\n",
+ __func__);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n",
+ __func__);
+ goto ErrorPowerOff;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_DISABLE_GAIN_LIMIT, off, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: fail to disable gain limit\n", __func__);
+ goto ErrorSendingCmd;
+ }
+
+ ret = sec_ts_fix_tmode(ts, TOUCH_SYSTEM_MODE_TOUCH,
+ TOUCH_MODE_STATE_TOUCH);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to fix tmode\n",
+ __func__);
+ goto ErrorSendingCmd;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_DISABLE_BASELINE_ADAPT, off,
+ 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: fail to disable baselineAdapt\n", __func__);
+ goto ErrorSendingCmd;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_DISABLE_DF, off, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to disable df\n",
+ __func__);
+ goto ErrorSendingCmd;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCH_ENGINE_MODE,
+ off, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: fail to disable touch engine\n", __func__);
+ goto ErrorSendingCmd;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_RESET_BASELINE, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to fix tmode\n",
+ __func__);
+ goto ErrorSendingCmd;
+ }
+
+ sec_ts_delay(50);
+
+ input_info(true, &ts->client->dev, "%s: ready to press\n", __func__);
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+ErrorSendingCmd:
+ErrorPowerOff:
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_fs_cal_get_data(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ struct sec_ts_test_mode mode;
+ short *diff_table = NULL;
+ const bool only_average = true;
+ char *buff;
+ int ret;
+ int i, j;
+ const unsigned int buff_size = ts->tx_count * ts->rx_count *
+ CMD_RESULT_WORD_LEN;
+ unsigned int buff_len = 0;
+
+ input_info(true, &ts->client->dev, "%s: fs cal with stim pad\n",
+ __func__);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n",
+ __func__);
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 3) {
+ input_err(true, &ts->client->dev,
+ "%s: Parameter Error\n", __func__);
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ buff = kzalloc(buff_size, GFP_KERNEL);
+ if (!buff) {
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+
+ if (sec->cmd_param[0] == 0) {
+ mode.type = TYPE_REMV_AMB_DATA;
+ mode.spec_check = SPEC_NO_CHECK;
+ ret = sec_ts_read_frame_stdev(ts, sec, mode.type, mode.min,
+ mode.max, &mode.spec_check, only_average);
+ if (ret < 0) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto SetCmdResult;
+ }
+ ret = sec_ts_check_fs_precal(ts);
+
+ /* ret = NG node count */
+ if (ret > 0) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "NG\n");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "OK\n");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+
+ for (i = 0; i < ts->tx_count * ts->rx_count; i++) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "%4d,",
+ ts->pFrame[i]);
+
+ if (i % ts->tx_count == ts->tx_count - 1)
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "\n");
+ }
+ /* calculate gain table, and store it to ts->gainTable */
+ sec_ts_get_gain_table(ts);
+
+ } else if (sec->cmd_param[0] == 1) {
+ /* write gaintable(ts->gainTable) to ic */
+ ret = sec_ts_write_gain_table(ts);
+ if (ret < 0) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "NG %d %d\n",
+ ts->rx_count, ts->tx_count);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "OK %d %d\n",
+ ts->rx_count, ts->tx_count);
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+
+ for (j = 0; j < ts->rx_count; j++) {
+ for (i = 0; i < ts->tx_count; i++) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "%4d,",
+ ts->gainTable[i*ts->rx_count
+ + j]);
+ }
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "\n");
+ }
+ } else if (sec->cmd_param[0] == 2) {
+ int mean;
+ /* for stim pad fixture 2 */
+ mode.type = TYPE_NORM2_DATA;
+ mode.spec_check = SPEC_NO_CHECK;
+
+ ret = sec_ts_read_frame_stdev(ts, sec, mode.type, mode.min,
+ mode.max, &mode.spec_check, only_average);
+ if (ret < 0) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto SetCmdResult;
+ }
+
+ mean = sec_ts_get_postcal_mean(ts);
+ ts->fs_postcal_mean = mean;
+
+ input_info(true, &ts->client->dev,
+ "%s : FS mean = %d\n", __func__, mean);
+
+ if ((mean > fs_mean_target_h) || (mean < fs_mean_target_l)) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "NG %d %d\n%d\n",
+ ts->rx_count, ts->tx_count,
+ ts->fs_postcal_mean);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "OK %d %d\n%d\n",
+ ts->rx_count, ts->tx_count,
+ ts->fs_postcal_mean);
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+
+ for (i = 0; i < ts->tx_count * ts->rx_count; i++) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "%4d,",
+ ts->pFrame[i]);
+
+ if (i % ts->tx_count == ts->tx_count - 1)
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "\n");
+ }
+ } else { //(sec->cmd_param[0] == 3)
+ /* get fs_uniformity */
+
+ diff_table = kzalloc(ts->tx_count * ts->rx_count * 2,
+ GFP_KERNEL);
+
+ if ((diff_table == NULL) || (ts->fs_postcal_mean == 0)) {
+ input_err(true, &ts->client->dev,
+ "%s: fail to alloc diffTable, postcal mean = %d\n",
+ __func__, ts->fs_postcal_mean);
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "NG %d %d",
+ ts->rx_count, ts->tx_count);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto SetCmdResult;
+ }
+
+ ret = sec_ts_get_postcal_uniformity(ts, diff_table);
+
+ if (ret == 0) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "OK %d %d\n",
+ ts->rx_count, ts->tx_count);
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ } else {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "NG %d %d\n",
+ ts->rx_count, ts->tx_count);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ }
+
+ for (i = 0; i < ts->tx_count * ts->rx_count; i++) {
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len, "%4d,",
+ diff_table[i]);
+
+ if (i % ts->tx_count == ts->tx_count - 1)
+ buff_len += scnprintf(buff + buff_len,
+ buff_size - buff_len,
+ "\n");
+ }
+ }
+
+SetCmdResult:
+
+ sec_cmd_set_cmd_result(sec, buff, buff_len);
+
+ kfree(diff_table);
+ kfree(buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_fs_cal_post_press(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int ret = 0;
+ u8 on[1] = {STATE_MANAGE_ON};
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n",
+ __func__);
+ goto ErrorPowerOff;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCH_ENGINE_MODE, on, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: fail to enable touch engine\n", __func__);
+ goto ErrorSendingCmd;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_DISABLE_DF, on, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to enable df\n",
+ __func__);
+ goto ErrorSendingCmd;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_DISABLE_BASELINE_ADAPT, on,
+ 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: fail to enable baselineAdapt\n", __func__);
+ goto ErrorSendingCmd;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_DISABLE_GAIN_LIMIT, on, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: fail to enable gain limit\n", __func__);
+ goto ErrorSendingCmd;
+ }
+
+ ret = sec_ts_release_tmode(ts);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to release tmode\n",
+ __func__);
+ goto ErrorSendingCmd;
+ }
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+ErrorSendingCmd:
+ErrorPowerOff:
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+/* enable_fs_cal_table : enable or disable fs cal table
+ * cmd_param : 0 to disable, 1 to enable
+ * touch mode and state should be fixed before enable or disable
+ */
+static void enable_fs_cal_table(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ u8 tPara;
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) {
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ ret = sec_ts_fix_tmode(ts, TOUCH_SYSTEM_MODE_TOUCH,
+ TOUCH_MODE_STATE_TOUCH);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to fix tmode\n",
+ __func__);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ tPara = sec->cmd_param[0];
+
+ input_info(true, &ts->client->dev, "%s: fs cal table %s\n",
+ __func__, ((tPara == 0) ? "disable" : "enable"));
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_DISABLE_NORM_TABLE,
+ &tPara, 1);
+
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: cmd write failed\n", __func__);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ } else {
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, "OK", 2);
+ }
+
+ ret = sec_ts_release_tmode(ts);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to release tmode\n",
+ __func__);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void enable_coordinate_report(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ int ret = 0;
+ u8 tPara;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n",
+ __func__);
+ sec_cmd_set_cmd_result(sec, "TSP turned off", 14);
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) {
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ tPara = sec->cmd_param[0];
+
+ input_info(true, &ts->client->dev, "%s: coordinate report %s\n",
+ __func__, ((tPara == 0) ? "disable" : "enable"));
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCH_ENGINE_MODE,
+ &tPara, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: cmd write failed\n", __func__);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ } else {
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, "OK", 2);
+ }
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void enable_gain_limit(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ int ret = 0;
+ u8 tPara;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n",
+ __func__);
+ sec_cmd_set_cmd_result(sec, "TSP turned off", 14);
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) {
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ tPara = sec->cmd_param[0];
+
+ input_info(true, &ts->client->dev, "%s: gain limit %s\n",
+ __func__, ((tPara == 0) ? "disable" : "enable"));
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_DISABLE_GAIN_LIMIT, &tPara,
+ 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: cmd write failed\n", __func__);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ } else {
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, "OK", 2);
+ }
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_trx_short_test(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int rc;
+ char para = TO_TOUCH_MODE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ disable_irq(ts->client->irq);
+
+ rc = execute_selftest(ts, true);
+ if (rc > 0) {
+ ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_POWER_MODE, ¶, 1);
+ enable_irq(ts->client->irq);
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_POWER_MODE, ¶, 1);
+ enable_irq(ts->client->irq);
+
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+
+int sec_ts_execute_force_calibration(struct sec_ts_data *ts, int cal_mode)
+{
+ int rc = -1;
+ u8 cmd;
+
+ input_info(true, &ts->client->dev, "%s: %d\n", __func__, cal_mode);
+
+ if (cal_mode == OFFSET_CAL_SEC)
+ cmd = SEC_TS_CMD_FACTORY_PANELCALIBRATION;
+ else if (cal_mode == AMBIENT_CAL)
+ cmd = SEC_TS_CMD_CALIBRATION_AMBIENT;
+#ifdef USE_PRESSURE_SENSOR
+ else if (cal_mode == PRESSURE_CAL)
+ cmd = SEC_TS_CMD_CALIBRATION_PRESSURE;
+#endif
+ else
+ return rc;
+
+ if (ts->sec_ts_i2c_write(ts, cmd, NULL, 0) < 0) {
+ input_err(true, &ts->client->dev, "%s: Write Cal commend failed!\n", __func__);
+ return rc;
+ }
+
+ sec_ts_delay(1000);
+
+ rc = sec_ts_wait_for_ready(ts, SEC_TS_VENDOR_ACK_OFFSET_CAL_DONE);
+
+ return rc;
+}
+
+static void run_force_calibration(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int rc;
+#ifdef PAT_CONTROL
+ u8 img_ver[4];
+#endif
+ struct sec_ts_test_mode mode;
+ char mis_cal_data = 0xF0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ sec_ts_read_calibration_report(ts);
+
+ if (ts->touch_count > 0) {
+ snprintf(buff, sizeof(buff), "%s", "NG_FINGER_ON");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out_force_cal;
+ }
+
+ disable_irq(ts->client->irq);
+
+ rc = sec_ts_execute_force_calibration(ts, OFFSET_CAL_SEC);
+ if (rc < 0) {
+ snprintf(buff, sizeof(buff), "%s", "FAIL");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+#ifdef USE_PRESSURE_SENSOR
+ rc = sec_ts_execute_force_calibration(ts, PRESSURE_CAL);
+ if (rc < 0)
+ input_err(true, &ts->client->dev, "%s: fail to write PRESSURE CAL!\n", __func__);
+#endif
+
+ if (ts->plat_data->mis_cal_check) {
+ buff[0] = 0;
+ rc = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_STATEMANAGE_ON, buff, 1);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: mis_cal_check error[1] ret: %d\n", __func__, rc);
+ }
+
+ buff[0] = 0x2;
+ buff[1] = 0x2;
+ rc = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CHG_SYSMODE, buff, 2);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: mis_cal_check error[2] ret: %d\n", __func__, rc);
+ }
+
+ input_err(true, &ts->client->dev, "%s: try mis Cal. check\n", __func__);
+ rc = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_MIS_CAL_CHECK, NULL, 0);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: mis_cal_check error[3] ret: %d\n", __func__, rc);
+ }
+ sec_ts_delay(200);
+
+ rc = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_MIS_CAL_READ, &mis_cal_data, 1);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev, "%s: i2c fail!, %d\n", __func__, rc);
+ mis_cal_data = 0xF3;
+ } else {
+ input_info(true, &ts->client->dev, "%s: miss cal data : %d\n", __func__, mis_cal_data);
+ }
+
+ buff[0] = 1;
+ rc = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_STATEMANAGE_ON, buff, 1);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: mis_cal_check error[4] ret: %d\n", __func__, rc);
+ }
+
+ if (mis_cal_data) {
+ memset(&mode, 0x00, sizeof(struct sec_ts_test_mode));
+ mode.type = TYPE_AMBIENT_DATA;
+ mode.allnode = TEST_MODE_ALL_NODE;
+
+ sec_ts_read_raw_data(ts, NULL, &mode);
+ snprintf(buff, sizeof(buff), "%s", "MIS CAL");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+
+ enable_irq(ts->client->irq);
+
+ goto out_force_cal;
+ }
+ }
+
+#ifdef PAT_CONTROL
+ ts->cal_count = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+
+ if(ts->external_factory == true) {
+ /* for external factory mode */
+ if (ts->cal_count == PAT_MAX_EXT)
+ ts->cal_count = PAT_MAX_EXT;
+ else if (PAT_EXT_FACT <= ts->cal_count && ts->cal_count < PAT_MAX_EXT)
+ ts->cal_count++;
+ else
+ ts->cal_count = PAT_EXT_FACT;
+
+ /* not to enter external factory mode without setting everytime */
+ ts->external_factory = false;
+ } else {
+ /*change from ( virtual pat or vpat by external fatory ) to real pat by forced calibarion by LCIA */
+ if (ts->cal_count >= PAT_MAGIC_NUMBER)
+ ts->cal_count = 1;
+ else if (ts->cal_count == PAT_MAX_LCIA)
+ ts->cal_count = PAT_MAX_LCIA;
+ else
+ ts->cal_count++;
+ }
+
+ /* Use TSP NV area : in this model, use only one byte
+ * buff[0] : offset from user NVM storage
+ * buff[1] : length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * buff[2] : write data
+ */
+ buff[0] = SEC_TS_NVM_OFFSET_CAL_COUNT;
+ buff[1] = 0;
+ buff[2] = ts->cal_count;
+ input_info(true, &ts->client->dev, "%s: write to nvm cal_count(%2X)\n",
+ __func__, buff[2]);
+
+ rc = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, buff, 3);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: nvm write failed. ret: %d\n", __func__, rc);
+ }
+
+ sec_ts_delay(20);
+
+ ts->cal_count = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+
+ rc = ts->sec_ts_i2c_read(ts, SEC_TS_READ_IMG_VERSION, img_ver, 4);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev, "%s: Image version read error\n", __func__);
+ } else {
+ memset(buff, 0x00, SEC_CMD_STR_LEN);
+ buff[0] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION);
+ if (buff[0] == 0xFF) {
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION);
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION+1);
+ }
+
+ ts->tune_fix_ver = (img_ver[2]<<8 | img_ver[3]);
+ buff[0] = SEC_TS_NVM_OFFSET_TUNE_VERSION;
+ buff[1] = 1;// 2bytes
+ buff[2] = img_ver[2];
+ buff[3] = img_ver[3];
+ input_info(true, &ts->client->dev, "%s: write tune_ver to nvm (%2X %2X)\n", __func__, buff[2], buff[3]);
+
+ rc = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, buff, 4);
+ if (rc < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: nvm write failed. ret: %d\n", __func__, rc);
+ }
+ sec_ts_delay(20);
+
+ buff[0] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION);
+ buff[1] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION+1);
+ ts->tune_fix_ver = buff[0]<<8 | buff[1];
+ input_info(true, &ts->client->dev,
+ "%s: cal_count [%2X] tune_fix_ver [%04X]\n", __func__, ts->cal_count, ts->tune_fix_ver);
+ }
+#endif
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+
+ enable_irq(ts->client->irq);
+
+out_force_cal:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_force_calibration(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int rc;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ rc = sec_ts_read_calibration_report(ts);
+ if (rc < 0) {
+ snprintf(buff, sizeof(buff), "%s", "FAIL");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else if (rc == SEC_TS_STATUS_CALIBRATION_SEC) {
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ } else {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ }
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+#ifdef USE_PRESSURE_SENSOR
+static void run_force_pressure_calibration(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int rc;
+ char data[3] = { 0 };
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ if (ts->touch_count > 0) {
+ snprintf(buff, sizeof(buff), "%s", "NG_FINGER_ON");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out_force_pressure_cal;
+ }
+
+ disable_irq(ts->client->irq);
+
+ rc = sec_ts_execute_force_calibration(ts, PRESSURE_CAL);
+ if (rc < 0) {
+ snprintf(buff, sizeof(buff), "%s", "FAIL");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+
+ ts->pressure_cal_base = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_PRESSURE_BASE_CAL_COUNT);
+ if (ts->pressure_cal_base == 0xFF)
+ ts->pressure_cal_base = 0;
+ if (ts->pressure_cal_base > 0xFD)
+ ts->pressure_cal_base = 0xFD;
+
+ /* Use TSP NV area : in this model, use only one byte
+ * data[0] : offset from user NVM storage
+ * data[1] : length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * data[2] : write data
+ */
+ data[0] = SEC_TS_NVM_OFFSET_PRESSURE_BASE_CAL_COUNT;
+ data[1] = 0;
+ data[2] = ts->pressure_cal_base + 1;
+
+ rc = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, data, 3);
+ if (rc < 0)
+ input_err(true, &ts->client->dev,
+ "%s: nvm write failed. ret: %d\n", __func__, rc);
+
+ ts->pressure_cal_base = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_PRESSURE_BASE_CAL_COUNT);
+
+ input_info(true, &ts->client->dev, "%s: count:%d\n", __func__, ts->pressure_cal_base);
+
+ enable_irq(ts->client->irq);
+
+out_force_pressure_cal:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void set_pressure_test_mode(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int ret;
+ unsigned char data = TYPE_INVALID_DATA;
+ unsigned char enable = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ if (sec->cmd_param[0] == 1) {
+ enable = 0x1;
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TEMPERATURE_COMP_MODE, &enable, 1);
+ if (ret < 0) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out_test_mode;
+ }
+
+ ret = sec_ts_fix_tmode(ts, TOUCH_SYSTEM_MODE_TOUCH, TOUCH_MODE_STATE_TOUCH);
+ if (ret < 0) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out_test_mode;
+ }
+
+ } else {
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SELECT_PRESSURE_TYPE, &data, 1);
+ if (ret < 0) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out_test_mode;
+ }
+
+ ret = sec_ts_release_tmode(ts);
+ if (ret < 0) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out_test_mode;
+ }
+
+ enable = 0x0;
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TEMPERATURE_COMP_MODE, &enable, 1);
+ if (ret < 0) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out_test_mode;
+ }
+ }
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+out_test_mode:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static int read_pressure_data(struct sec_ts_data *ts, u8 type, short *value)
+{
+ unsigned char data[6] = { 0 };
+ short pressure[3] = { 0 };
+ int ret;
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF)
+ return -ENODEV;
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_SELECT_PRESSURE_TYPE, data, 1);
+ if (ret < 0)
+ return -EIO;
+
+ if (data[0] != type) {
+ input_info(true, &ts->client->dev, "%s: type change to %02X\n", __func__, type);
+
+ data[1] = type;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SELECT_PRESSURE_TYPE, &data[1], 1);
+ if (ret < 0)
+ return -EIO;
+
+ sec_ts_delay(30);
+ }
+
+ memset(data, 0x00, 6);
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_READ_PRESSURE_DATA, data, 6);
+ if (ret < 0)
+ return -EIO;
+
+ pressure[0] = (data[0] << 8 | data[1]);
+ pressure[1] = (data[2] << 8 | data[3]);
+ pressure[2] = (data[4] << 8 | data[5]);
+
+ input_info(true, &ts->client->dev, "%s: Left: %d, Center: %d, Rignt: %d\n",
+ __func__, pressure[2], pressure[1], pressure[0]);
+
+ memcpy(value, pressure, 3 * 2);
+
+ return ret;
+}
+
+static void run_pressure_filtered_strength_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ short pressure[3] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ ret = read_pressure_data(ts, TYPE_SIGNAL_DATA, pressure);
+ if (ret < 0) {
+ snprintf(buff, sizeof(buff), "%s", "WRITE FAILED");
+ goto error_read_str;
+ }
+
+ snprintf(buff, sizeof(buff), "%d,%d,%d", pressure[2], pressure[1], pressure[0]);
+
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+error_read_str:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_pressure_strength_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ short pressure[3] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ ret = read_pressure_data(ts, TYPE_REMV_AMB_DATA, pressure);
+ if (ret < 0) {
+ snprintf(buff, sizeof(buff), "%s", "WRITE FAILED");
+ goto error_read_str;
+ }
+
+ snprintf(buff, sizeof(buff), "%d,%d,%d", pressure[2], pressure[1], pressure[0]);
+
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+error_read_str:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_pressure_rawdata_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ short pressure[3] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ ret = read_pressure_data(ts, TYPE_RAW_DATA, pressure);
+ if (ret < 0) {
+ snprintf(buff, sizeof(buff), "%s", "WRITE FAILED");
+ goto error_read_rawdata;
+ }
+
+ snprintf(buff, sizeof(buff), "%d,%d,%d", pressure[2], pressure[1], pressure[0]);
+
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+error_read_rawdata:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void run_pressure_offset_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ short pressure[3] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ ret = read_pressure_data(ts, TYPE_OFFSET_DATA_SEC, pressure);
+ if (ret < 0) {
+ snprintf(buff, sizeof(buff), "%s", "WRITE FAILED");
+ goto error_read_str;
+ }
+
+ snprintf(buff, sizeof(buff), "%d,%d,%d", pressure[2], pressure[1], pressure[0]);
+
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+error_read_str:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void set_pressure_strength(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ u8 data[8] = { 0 };
+ u8 cal_data[18] = { 0 };
+ int index;
+ int ret;
+ short pressure[3] = { 0 };
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if ((sec->cmd_param[0] < 1) || (sec->cmd_param[0] > 4))
+ goto err_cmd_param_str;
+
+ index = sec->cmd_param[0] - 1;
+
+ /* RIGHT */
+ cal_data[0] = (sec->cmd_param[3] >> 8);
+ cal_data[1] = (sec->cmd_param[3] & 0xFF);
+ /* CENTER */
+ cal_data[8] = (sec->cmd_param[2] >> 8);
+ cal_data[9] = (sec->cmd_param[2] & 0xFF);
+ /* LEFT */
+ cal_data[16] = (sec->cmd_param[1] >> 8);
+ cal_data[17] = (sec->cmd_param[1] & 0xFF);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_GET_PRESSURE, cal_data, 18);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: cmd write failed. ret: %d\n", __func__, ret);
+ goto err_comm_str;
+ }
+
+ sec_ts_delay(30);
+
+ memset(cal_data, 0x00, 18);
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_SET_GET_PRESSURE, cal_data, 18);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: cmd write failed. ret: %d\n", __func__, ret);
+ goto err_comm_str;
+ }
+
+ pressure[0] = ((cal_data[16] << 8) | cal_data[17]);
+ pressure[1] = ((cal_data[8] << 8) | cal_data[9]);
+ pressure[2] = ((cal_data[0] << 8) | cal_data[1]);
+
+ input_info(true, &ts->client->dev, "%s: [%d] : %d, %d, %d\n", __func__, pressure[0], pressure[1], pressure[2]);
+
+ /* Use TSP NV area : in this model, use only one byte
+ * buff[0] : offset from user NVM storage
+ * buff[1] : [n] length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * buff[2] ... [n] : write data ...
+ */
+ data[0] = SEC_TS_NVM_OFFSET_PRESSURE_STRENGTH + (index * SEC_TS_NVM_SIZE_PRESSURE_BLOCK);
+ data[1] = SEC_TS_NVM_SIZE_PRESSURE_BLOCK - 1;
+ /* RIGHT */
+ data[2] = (sec->cmd_param[3] >> 8);
+ data[3] = (sec->cmd_param[3] & 0xFF);
+ /* CENTER */
+ data[4] = (sec->cmd_param[2] >> 8);
+ data[5] = (sec->cmd_param[2] & 0xFF);
+ /*LEFT */
+ data[6] = (sec->cmd_param[1] >> 8);
+ data[7] = (sec->cmd_param[1] & 0xFF);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, data, 8);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm write failed. ret: %d\n", __func__, ret);
+ goto err_comm_str;
+ }
+
+ sec_ts_delay(20);
+
+ input_info(true, &ts->client->dev, "%s: [%d] : %d, %d, %d\n",
+ __func__, index, (data[6] << 8) + data[7], (data[4] << 8) + data[5], (data[0] << 8) + data[1]);
+
+ memset(data, 0x00, 8);
+
+ data[0] = SEC_TS_NVM_OFFSET_PRESSURE_INDEX;
+ data[1] = 0;
+ data[2] = (u8)(index & 0xFF);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, data, 3);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm write failed. ret: %d\n", __func__, ret);
+ goto err_comm_str;
+ }
+
+ sec_ts_delay(20);
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ ts->pressure_cal_delta = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_PRESSURE_DELTA_CAL_COUNT);
+ if (ts->pressure_cal_delta == 0xFF)
+ ts->pressure_cal_delta = 0;
+
+ if (ts->pressure_cal_delta > 0xFD)
+ ts->pressure_cal_delta = 0xFD;
+
+ /* Use TSP NV area : in this model, use only one byte
+ * data[0] : offset from user NVM storage
+ * data[1] : length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * data[2] : write data
+ */
+ data[0] = SEC_TS_NVM_OFFSET_PRESSURE_DELTA_CAL_COUNT;
+ data[1] = 0;
+ data[2] = ts->pressure_cal_delta + 1;
+
+ ret= ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, data, 3);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: nvm write failed. ret: %d\n", __func__, ret);
+
+ ts->pressure_cal_delta = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_PRESSURE_DELTA_CAL_COUNT);
+
+ input_info(true, &ts->client->dev, "%s: count:%d\n", __func__, ts->pressure_cal_delta);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+err_cmd_param_str:
+ input_info(true, &ts->client->dev, "%s: parameter error: %u\n",
+ __func__, sec->cmd_param[0]);
+err_comm_str:
+ snprintf(buff, sizeof(buff), "NG");
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void set_pressure_rawdata(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ u8 data[8] = { 0 };
+ int index;
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if ((sec->cmd_param[0] < 1) || (sec->cmd_param[0] > 4))
+ goto err_cmd_param_raw;
+
+ index = sec->cmd_param[0] - 1;
+
+ /* Use TSP NV area : in this model, use only one byte
+ * buff[0] : offset from user NVM storage
+ * buff[1] : [n] length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * buff[2] ... [n] : write data ...
+ */
+ data[0] = SEC_TS_NVM_OFFSET_PRESSURE_RAWDATA + (index * SEC_TS_NVM_SIZE_PRESSURE_BLOCK);
+ data[1] = SEC_TS_NVM_SIZE_PRESSURE_BLOCK - 1;
+ data[2] = (sec->cmd_param[3] >> 8);
+ data[3] = (sec->cmd_param[3] & 0xFF);
+ data[4] = (sec->cmd_param[2] >> 8);
+ data[5] = (sec->cmd_param[2] & 0xFF);
+ data[6] = (sec->cmd_param[1] >> 8);
+ data[7] = (sec->cmd_param[1] & 0xFF);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, data, 8);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm write failed. ret: %d\n", __func__, ret);
+ goto err_comm_raw;
+ }
+ sec_ts_delay(20);
+
+ memset(data, 0x00, 8);
+
+ ret = get_tsp_nvm_data_by_size(ts, SEC_TS_NVM_OFFSET_PRESSURE_RAWDATA + (index * SEC_TS_NVM_SIZE_PRESSURE_BLOCK), SEC_TS_NVM_SIZE_PRESSURE_BLOCK, data);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm read failed. ret: %d\n", __func__, ret);
+ goto err_comm_raw;
+ }
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ input_info(true, &ts->client->dev, "%s: [%d] : %d, %d, %d\n",
+ __func__, index, (data[4] << 8) + data[5], (data[2] << 8) + data[3], (data[1] << 8) + data[0]);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+err_cmd_param_raw:
+ input_info(true, &ts->client->dev, "%s: parameter error: %u\n",
+ __func__, sec->cmd_param[0]);
+err_comm_raw:
+
+ snprintf(buff, sizeof(buff), "NG");
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void set_pressure_data_index(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ u8 data[8] = { 0 };
+ u8 cal_data[18] = { 0 };
+ int index;
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if ((sec->cmd_param[0] < 0) || (sec->cmd_param[0] > 4))
+ goto err_set_cmd_param_index;
+
+ if (sec->cmd_param[0] == 0) {
+ input_info(true, &ts->client->dev, "%s: clear calibration result\n", __func__);
+ /* clear pressure calibrated data */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_GET_PRESSURE, cal_data, 18);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: cmd write failed. ret: %d\n", __func__, ret);
+ goto err_set_comm_index;
+ }
+
+ sec_ts_delay(30);
+
+ goto clear_index;
+ }
+
+ index = sec->cmd_param[0] - 1;
+
+ ret = get_tsp_nvm_data_by_size(ts, SEC_TS_NVM_OFFSET_PRESSURE_STRENGTH, 24, data);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm read failed. ret: %d\n", __func__, ret);
+ goto err_set_comm_index;
+ }
+
+ cal_data[16] = data[6 * index + 4];
+ cal_data[17] = data[6 * index + 5]; /* LEFT */
+
+ cal_data[8] = data[6 * index + 2];
+ cal_data[9] = data[6 * index + 3]; /* CENTER */
+
+ cal_data[0] = data[6 * index + 0];
+ cal_data[1] = data[6 * index + 1]; /* RIGHT */
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_GET_PRESSURE, cal_data, 18);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: cmd write failed. ret: %d\n", __func__, ret);
+ goto err_set_comm_index;
+ }
+
+ sec_ts_delay(30);
+
+ /* Use TSP NV area : in this model, use only one byte
+ * buff[0] : offset from user NVM storage
+ * buff[1] : length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * buff[2] : write data
+ */
+ memset(data, 0x00, 8);
+ data[0] = SEC_TS_NVM_OFFSET_PRESSURE_INDEX;
+ data[1] = 0;
+ data[2] = (u8)(index & 0xFF);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, data, 3);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm write failed. ret: %d\n", __func__, ret);
+ goto err_set_comm_index;
+ }
+
+ sec_ts_delay(20);
+
+clear_index:
+ snprintf(buff, sizeof(buff), "%s", "OK");
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+err_set_cmd_param_index:
+ input_info(true, &ts->client->dev, "%s: parameter error: %u\n",
+ __func__, sec->cmd_param[0]);
+err_set_comm_index:
+
+ snprintf(buff, sizeof(buff), "NG");
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_pressure_strength(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int index;
+ u8 data[24] = { 0 };
+ short pressure[3] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if ((sec->cmd_param[0] < 1) || (sec->cmd_param[0] > 4))
+ goto err_get_cmd_param_str;
+
+ index = sec->cmd_param[0] - 1;
+
+ ret = get_tsp_nvm_data_by_size(ts, SEC_TS_NVM_OFFSET_PRESSURE_STRENGTH, 24, data);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm read failed. ret: %d\n", __func__, ret);
+ goto err_get_comm_str;
+ }
+
+ pressure[0] = ((data[6 * index + 4] << 8) + data[6 * index + 5]);
+ pressure[1] = ((data[6 * index + 2] << 8) + data[6 * index + 3]);
+ pressure[2] = ((data[6 * index + 0] << 8) + data[6 * index + 1]);
+
+ snprintf(buff, sizeof(buff), "%d,%d,%d", pressure[0], pressure[1], pressure[2]);
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ input_info(true, &ts->client->dev, "%s: [%d] : %d, %d, %d\n",
+ __func__, index, pressure[0], pressure[1], pressure[2]);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+err_get_comm_str:
+ input_info(true, &ts->client->dev, "%s: parameter error: %u\n",
+ __func__, sec->cmd_param[0]);
+err_get_cmd_param_str:
+
+ snprintf(buff, sizeof(buff), "NG");
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_pressure_rawdata(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int index;
+ u8 data[24] = { 0 };
+ short pressure[3] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if ((sec->cmd_param[0] < 1) || (sec->cmd_param[0] > 4))
+ goto err_get_cmd_param_raw;
+
+ index = sec->cmd_param[0] - 1;
+
+ ret = get_tsp_nvm_data_by_size(ts, SEC_TS_NVM_OFFSET_PRESSURE_RAWDATA, 24, data);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm read failed. ret: %d\n", __func__, ret);
+ goto err_get_comm_raw;
+ }
+
+ pressure[0] = ((data[6 * index + 4] << 8) + data[6 * index + 5]);
+ pressure[1] = ((data[6 * index + 2] << 8) + data[6 * index + 3]);
+ pressure[2] = ((data[6 * index + 0] << 8) + data[6 * index + 1]);
+
+ snprintf(buff, sizeof(buff), "%d,%d,%d", pressure[0], pressure[1], pressure[2]);
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ input_info(true, &ts->client->dev, "%s: [%d] : %d, %d, %d\n",
+ __func__, index, pressure[0], pressure[1], pressure[2]);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+err_get_cmd_param_raw:
+ input_info(true, &ts->client->dev, "%s: parameter error: %u\n",
+ __func__, sec->cmd_param[0]);
+err_get_comm_raw:
+
+ snprintf(buff, sizeof(buff), "NG");
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_pressure_data_index(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int index = 0;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ index = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_PRESSURE_INDEX);
+ if (index < 0) {
+ goto err_get_index;
+ } else {
+ if (index == 0xFF)
+ snprintf(buff, sizeof(buff), "%d", 0);
+ else
+ snprintf(buff, sizeof(buff), "%d", index + 1);
+ }
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ input_info(true, &ts->client->dev, "%s: %d\n",
+ __func__, index);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+err_get_index:
+
+ snprintf(buff, sizeof(buff), "NG");
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+}
+static void set_pressure_strength_clear(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ u8 *data;
+ u8 cal_data[18] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ /* clear pressure calibrated data */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_GET_PRESSURE, cal_data, 18);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: cmd write failed. ret: %d\n", __func__, ret);
+ goto err_comm_str;
+ }
+
+ sec_ts_delay(30);
+
+ /* Use TSP NV area : in this model, use only one byte
+ * buff[0] : offset from user NVM storage
+ * buff[1] : length of stroed data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * buff[2] : write data
+ */
+
+ /* strength 6 * 4, rawdata 6 * 4, buff[0], buff[1] */
+ data = kzalloc(50, GFP_KERNEL);
+ if (!data) {
+ input_err(true, &ts->client->dev, "%s failed to allocate memory. ret: %d\n", __func__, ret);
+ goto err_comm_str;
+ }
+
+ data[0] = SEC_TS_NVM_OFFSET_PRESSURE_INDEX;
+ data[1] = (SEC_TS_NVM_SIZE_PRESSURE_BLOCK * 8) - 1;
+
+ /* remove calicated strength, rawdata in NVM */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, data, 50);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: nvm write failed. ret: %d\n", __func__, ret);
+ goto err_mem_str;
+ }
+
+ sec_ts_delay(20);
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+
+ kfree(data);
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+err_mem_str:
+ kfree(data);
+err_comm_str:
+ snprintf(buff, sizeof(buff), "NG");
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_pressure_threshold(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ char buff[SEC_CMD_STR_LEN] = {0};
+
+ sec_cmd_set_default_result(sec);
+
+ snprintf(buff, sizeof(buff), "300");
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+}
+
+/* low level is more sensitivity, except level-0(value 0) */
+static void set_pressure_user_level(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ int ret;
+ char addr[3] = { 0 };
+ char data[2] = { 0 };
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ if ((sec->cmd_param[0] < 1) || (sec->cmd_param[0] > 5))
+ goto out_set_user_level;
+
+ /*
+ * byte[0]: m_customlib_ifpacket_addr[7:0]
+ * byte[1]: m_customlib_ifpacket_addr[15:8]
+ * byte[n] : user data (max 32 bytes)
+ */
+ addr[0] = SEC_TS_CMD_CUSTOMLIB_OFFSET_PRESSURE_LEVEL;
+ addr[1] = 0x00;
+ addr[2] = sec->cmd_param[0];
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_WRITE_PARAM, addr, 3);
+ if (ret < 0)
+ goto out_set_user_level;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_NOTIFY_PACKET, NULL, 0);
+ if (ret < 0)
+ goto out_set_user_level;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_READ_PARAM, addr, 2);
+ if (ret < 0)
+ goto out_set_user_level;
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_CUSTOMLIB_READ_PARAM, data, 1);
+ if (ret < 0)
+ goto out_set_user_level;
+
+ input_info(true, &ts->client->dev, "%s: set user level: %d\n", __func__, data[0]);
+
+ ts->pressure_user_level = data[0];
+
+ addr[0] = SEC_TS_CMD_CUSTOMLIB_OFFSET_PRESSURE_THD_HIGH;
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_READ_PARAM, addr, 2);
+ if (ret < 0)
+ goto out_set_user_level;
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_CUSTOMLIB_READ_PARAM, data, 1);
+ if (ret < 0)
+ goto out_set_user_level;
+
+ input_info(true, &ts->client->dev, "%s: HIGH THD: %d\n", __func__, data[0]);
+
+ addr[0] = SEC_TS_CMD_CUSTOMLIB_OFFSET_PRESSURE_THD_LOW;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_READ_PARAM, addr, 2);
+ if (ret < 0)
+ goto out_set_user_level;
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_CUSTOMLIB_READ_PARAM, data, 1);
+ if (ret < 0)
+ goto out_set_user_level;
+
+ input_info(true, &ts->client->dev, "%s: LOW THD: %d\n", __func__, data[0]);
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+out_set_user_level:
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void get_pressure_user_level(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = {0};
+ char addr[3] = { 0 };
+ char data[2] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ snprintf(buff, sizeof(buff), "%d", ts->pressure_user_level);
+
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+
+ addr[0] = SEC_TS_CMD_CUSTOMLIB_OFFSET_PRESSURE_LEVEL;
+ addr[1] = 0x00;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_READ_PARAM, addr, 2);
+ if (ret < 0)
+ goto out_get_user_level;
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_CUSTOMLIB_READ_PARAM, data, 1);
+ if (ret < 0)
+ goto out_get_user_level;
+
+ input_err(true, &ts->client->dev, "%s: set user level: %d\n", __func__, data[0]);
+ ts->pressure_user_level = data[0];
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+out_get_user_level:
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+#endif
+
+static void set_lowpower_mode(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ } else {
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ }
+
+/* set lowpower mode by spay, edge_swipe function.
+ ts->lowpower_mode = sec->cmd_param[0];
+*/
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+
+ sec_cmd_set_cmd_exit(sec);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void set_wirelesscharger_mode(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ int ret;
+ bool mode;
+ u8 w_data[1] = {0x00};
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 3)
+ goto OUT;
+
+ if (sec->cmd_param[0] == 0) {
+ ts->charger_mode |= SEC_TS_BIT_CHARGER_MODE_NO;
+ mode = false;
+ } else {
+ ts->charger_mode &= (~SEC_TS_BIT_CHARGER_MODE_NO);
+ mode = true;
+ }
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: fail to enable w-charger status, POWER_STATUS=OFF\n", __func__);
+ goto NG;
+ }
+
+ if (sec->cmd_param[0] == 1)
+ ts->charger_mode = ts->charger_mode | SEC_TS_BIT_CHARGER_MODE_WIRELESS_CHARGER;
+ else if (sec->cmd_param[0] == 3)
+ ts->charger_mode = ts->charger_mode | SEC_TS_BIT_CHARGER_MODE_WIRELESS_BATTERY_PACK;
+ else if (mode == false)
+ ts->charger_mode = ts->charger_mode & (~SEC_TS_BIT_CHARGER_MODE_WIRELESS_CHARGER) & (~SEC_TS_BIT_CHARGER_MODE_WIRELESS_BATTERY_PACK);
+
+ w_data[0] = ts->charger_mode;
+ ret = ts->sec_ts_i2c_write(ts, SET_TS_CMD_SET_CHARGER_MODE, w_data, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Failed to send command 74\n", __func__);
+ goto NG;
+ }
+
+ input_err(true, &ts->client->dev, "%s: %s, status =%x\n",
+ __func__, (mode) ? "wireless enable" : "wireless disable", ts->charger_mode);
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+NG:
+ input_err(true, &ts->client->dev, "%s: %s, status =%x\n",
+ __func__, (mode) ? "wireless enable" : "wireless disable", ts->charger_mode);
+
+OUT:
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void spay_enable(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1)
+ goto NG;
+
+ if (sec->cmd_param[0]) {
+ if (ts->use_customlib)
+ ts->lowpower_mode |= SEC_TS_MODE_CUSTOMLIB_SPAY;
+ } else {
+ if (ts->use_customlib)
+ ts->lowpower_mode &= ~SEC_TS_MODE_CUSTOMLIB_SPAY;
+ }
+
+ input_info(true, &ts->client->dev, "%s: %02X\n", __func__, ts->lowpower_mode);
+
+ #ifdef SEC_TS_SUPPORT_CUSTOMLIB
+ if (ts->use_customlib)
+ sec_ts_set_custom_library(ts);
+ #endif
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+NG:
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void set_aod_rect(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ u8 data[10] = {0x02, 0};
+ int ret, i;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ input_info(true, &ts->client->dev, "%s: w:%d, h:%d, x:%d, y:%d\n",
+ __func__, sec->cmd_param[0], sec->cmd_param[1],
+ sec->cmd_param[2], sec->cmd_param[3]);
+
+ for (i = 0; i < 4; i++) {
+ data[i * 2 + 2] = sec->cmd_param[i] & 0xFF;
+ data[i * 2 + 3] = (sec->cmd_param[i] >> 8) & 0xFF;
+ ts->rect_data[i] = sec->cmd_param[i];
+ }
+
+ if (ts->use_customlib) {
+ disable_irq(ts->client->irq);
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_WRITE_PARAM, &data[0], 10);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Failed to write offset\n", __func__);
+ goto NG;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CUSTOMLIB_NOTIFY_PACKET, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Failed to send notify\n", __func__);
+ goto NG;
+ }
+ enable_irq(ts->client->irq);
+ }
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+NG:
+ enable_irq(ts->client->irq);
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+
+static void get_aod_rect(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ u8 data[8] = {0x02, 0};
+ u16 rect_data[4] = {0, };
+ int ret, i;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->use_customlib) {
+ disable_irq(ts->client->irq);
+ ret = ts->sec_ts_read_customlib(ts, data, 8);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Failed to read rect\n", __func__);
+ goto NG;
+ }
+ enable_irq(ts->client->irq);
+ }
+
+ for (i = 0; i < 4; i++)
+ rect_data[i] = (data[i * 2 + 1] & 0xFF) << 8 | (data[i * 2] & 0xFF);
+
+ input_info(true, &ts->client->dev, "%s: w:%d, h:%d, x:%d, y:%d\n",
+ __func__, rect_data[0], rect_data[1], rect_data[2], rect_data[3]);
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+NG:
+ enable_irq(ts->client->irq);
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void aod_enable(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1)
+ goto NG;
+
+ if (sec->cmd_param[0]) {
+ if (ts->use_customlib)
+ ts->lowpower_mode |= SEC_TS_MODE_CUSTOMLIB_AOD;
+ } else {
+ if (ts->use_customlib)
+ ts->lowpower_mode &= ~SEC_TS_MODE_CUSTOMLIB_AOD;
+ }
+
+ input_info(true, &ts->client->dev, "%s: %02X\n", __func__, ts->lowpower_mode);
+
+ #ifdef SEC_TS_SUPPORT_CUSTOMLIB
+ if (ts->use_customlib)
+ sec_ts_set_custom_library(ts);
+ #endif
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+NG:
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+/*
+ * flag 1 : set edge handler
+ * 2 : set (portrait, normal) edge zone data
+ * 4 : set (portrait, normal) dead zone data
+ * 8 : set landscape mode data
+ * 16 : mode clear
+ * data
+ * 0x30, FFF (y start), FFF (y end), FF(direction)
+ * 0x31, FFFF (edge zone)
+ * 0x32, FF (up x), FF (down x), FFFF (y)
+ * 0x33, FF (mode), FFF (edge), FFF (dead zone)
+ * case
+ * edge handler set : 0x30....
+ * booting time : 0x30... + 0x31...
+ * normal mode : 0x32... (+0x31...)
+ * landscape mode : 0x33...
+ * landscape -> normal (if same with old data) : 0x33, 0
+ * landscape -> normal (etc) : 0x32.... + 0x33, 0
+ */
+
+void set_grip_data_to_ic(struct sec_ts_data *ts, u8 flag)
+{
+ u8 data[8] = { 0 };
+
+ input_info(true, &ts->client->dev, "%s: flag: %02X (clr,lan,nor,edg,han)\n", __func__, flag);
+
+ if (flag & G_SET_EDGE_HANDLER) {
+ if (ts->grip_edgehandler_direction == 0) {
+ data[0] = 0x0;
+ data[1] = 0x0;
+ data[2] = 0x0;
+ data[3] = 0x0;
+ } else {
+ data[0] = (ts->grip_edgehandler_start_y >> 4) & 0xFF;
+ data[1] = (ts->grip_edgehandler_start_y << 4 & 0xF0) | ((ts->grip_edgehandler_end_y >> 8) & 0xF);
+ data[2] = ts->grip_edgehandler_end_y & 0xFF;
+ data[3] = ts->grip_edgehandler_direction & 0x3;
+ }
+ ts->sec_ts_i2c_write(ts, SEC_TS_CMD_EDGE_HANDLER, data, 4);
+ input_info(true, &ts->client->dev, "%s: 0x%02X %02X,%02X,%02X,%02X\n",
+ __func__, SEC_TS_CMD_EDGE_HANDLER, data[0], data[1], data[2], data[3]);
+ }
+
+ if (flag & G_SET_EDGE_ZONE) {
+ data[0] = (ts->grip_edge_range >> 8) & 0xFF;
+ data[1] = ts->grip_edge_range & 0xFF;
+ ts->sec_ts_i2c_write(ts, SEC_TS_CMD_EDGE_AREA, data, 2);
+ input_info(true, &ts->client->dev, "%s: 0x%02X %02X,%02X\n",
+ __func__, SEC_TS_CMD_EDGE_AREA, data[0], data[1]);
+ }
+
+ if (flag & G_SET_NORMAL_MODE) {
+ data[0] = ts->grip_deadzone_up_x & 0xFF;
+ data[1] = ts->grip_deadzone_dn_x & 0xFF;
+ data[2] = (ts->grip_deadzone_y >> 8) & 0xFF;
+ data[3] = ts->grip_deadzone_y & 0xFF;
+ ts->sec_ts_i2c_write(ts, SEC_TS_CMD_DEAD_ZONE, data, 4);
+ input_info(true, &ts->client->dev, "%s: 0x%02X %02X,%02X,%02X,%02X\n",
+ __func__, SEC_TS_CMD_DEAD_ZONE, data[0], data[1], data[2], data[3]);
+ }
+
+ if (flag & G_SET_LANDSCAPE_MODE) {
+ data[0] = ts->grip_landscape_mode & 0x1;
+ data[1] = (ts->grip_landscape_edge >> 4) & 0xFF;
+ data[2] = (ts->grip_landscape_edge << 4 & 0xF0) | ((ts->grip_landscape_deadzone >> 8) & 0xF);
+ data[3] = ts->grip_landscape_deadzone & 0xFF;
+ ts->sec_ts_i2c_write(ts, SEC_TS_CMD_LANDSCAPE_MODE, data, 4);
+ input_info(true, &ts->client->dev, "%s: 0x%02X %02X,%02X,%02X,%02X\n",
+ __func__, SEC_TS_CMD_LANDSCAPE_MODE, data[0], data[1], data[2], data[3]);
+ }
+
+ if (flag & G_CLR_LANDSCAPE_MODE) {
+ data[0] = ts->grip_landscape_mode;
+ ts->sec_ts_i2c_write(ts, SEC_TS_CMD_LANDSCAPE_MODE, data, 1);
+ input_info(true, &ts->client->dev, "%s: 0x%02X %02X\n",
+ __func__, SEC_TS_CMD_LANDSCAPE_MODE, data[0]);
+ }
+}
+
+/*
+ * index 0 : set edge handler
+ * 1 : portrait (normal) mode
+ * 2 : landscape mode
+ *
+ * data
+ * 0, X (direction), X (y start), X (y end)
+ * direction : 0 (off), 1 (left), 2 (right)
+ * ex) echo set_grip_data,0,2,600,900 > cmd
+ *
+ * 1, X (edge zone), X (dead zone up x), X (dead zone down x), X (dead zone y)
+ * ex) echo set_grip_data,1,200,10,50,1500 > cmd
+ *
+ * 2, 1 (landscape mode), X (edge zone), X (dead zone)
+ * ex) echo set_grip_data,2,1,200,100 > cmd
+ *
+ * 2, 0 (portrait mode)
+ * ex) echo set_grip_data,2,0 > cmd
+ */
+
+static void set_grip_data(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ /* u8 mode = G_NONE; */
+ u8 tPara[2] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ memset(buff, 0, sizeof(buff));
+
+ mutex_lock(&ts->device_mutex);
+
+ tPara[0] = sec->cmd_param[0] & 0xFF;
+ tPara[1] = (sec->cmd_param[0] >> 8) & 0xFF;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_DEADZONE_RANGE, tPara, 2);
+ if (ret < 0)
+ goto err_grip_data;
+ /*
+ if (sec->cmd_param[0] == 0) { // edge handler
+ if (sec->cmd_param[1] == 0) { // clear
+ ts->grip_edgehandler_direction = 0;
+ } else if (sec->cmd_param[1] < 3) {
+ ts->grip_edgehandler_direction = sec->cmd_param[1];
+ ts->grip_edgehandler_start_y = sec->cmd_param[2];
+ ts->grip_edgehandler_end_y = sec->cmd_param[3];
+ } else {
+ input_err(true, &ts->client->dev, "%s: cmd1 is abnormal, %d (%d)\n",
+ __func__, sec->cmd_param[1], __LINE__);
+ goto err_grip_data;
+ }
+
+ mode = mode | G_SET_EDGE_HANDLER;
+ set_grip_data_to_ic(ts, mode);
+
+ } else if (sec->cmd_param[0] == 1) { // normal mode
+ if (ts->grip_edge_range != sec->cmd_param[1])
+ mode = mode | G_SET_EDGE_ZONE;
+
+ ts->grip_edge_range = sec->cmd_param[1];
+ ts->grip_deadzone_up_x = sec->cmd_param[2];
+ ts->grip_deadzone_dn_x = sec->cmd_param[3];
+ ts->grip_deadzone_y = sec->cmd_param[4];
+ mode = mode | G_SET_NORMAL_MODE;
+
+ if (ts->grip_landscape_mode == 1) {
+ ts->grip_landscape_mode = 0;
+ mode = mode | G_CLR_LANDSCAPE_MODE;
+ }
+ set_grip_data_to_ic(ts, mode);
+ } else if (sec->cmd_param[0] == 2) { // landscape mode
+ if (sec->cmd_param[1] == 0) { // normal mode
+ ts->grip_landscape_mode = 0;
+ mode = mode | G_CLR_LANDSCAPE_MODE;
+ } else if (sec->cmd_param[1] == 1) {
+ ts->grip_landscape_mode = 1;
+ ts->grip_landscape_edge = sec->cmd_param[2];
+ ts->grip_landscape_deadzone = sec->cmd_param[3];
+ mode = mode | G_SET_LANDSCAPE_MODE;
+ } else {
+ input_err(true, &ts->client->dev, "%s: cmd1 is abnormal, %d (%d)\n",
+ __func__, sec->cmd_param[1], __LINE__);
+ goto err_grip_data;
+ }
+ set_grip_data_to_ic(ts, mode);
+ } else {
+ input_err(true, &ts->client->dev, "%s: cmd0 is abnormal, %d", __func__, sec->cmd_param[0]);
+ goto err_grip_data;
+ }
+ */
+
+ mutex_unlock(&ts->device_mutex);
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+err_grip_data:
+ mutex_unlock(&ts->device_mutex);
+
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+/*
+ * Set/Get Dex Mode 0xE7
+ * 0: Disable dex mode
+ * 1: Full screen mode
+ * 2: Iris mode
+ */
+static void dex_enable(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (!ts->plat_data->support_dex) {
+ input_err(true, &ts->client->dev, "%s: not support DeX mode\n", __func__);
+ goto NG;
+ }
+
+ if ((sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) &&
+ (sec->cmd_param[1] < 0 || sec->cmd_param[1] > 1)) {
+ input_err(true, &ts->client->dev, "%s: not support param\n", __func__);
+ goto NG;
+ }
+
+ ts->dex_mode = sec->cmd_param[0];
+ if (ts->dex_mode) {
+ input_err(true, &ts->client->dev, "%s: set DeX touch_pad mode%s\n",
+ __func__, sec->cmd_param[1] ? " & Iris mode" : "");
+ ts->input_dev = ts->input_dev_pad;
+ if (sec->cmd_param[1]) {
+ /* Iris mode */
+ ts->dex_mode = 0x02;
+ ts->dex_name = "[DeXI]";
+ } else {
+ ts->dex_name = "[DeX]";
+ }
+ } else {
+ input_err(true, &ts->client->dev, "%s: set touch mode\n", __func__);
+ ts->input_dev = ts->input_dev_touch;
+ ts->dex_name = "";
+ }
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ goto NG;
+ }
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_DEX_MODE, &ts->dex_mode, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to set dex %smode\n", __func__,
+ sec->cmd_param[1] ? "iris " : "");
+ goto NG;
+ }
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+
+NG:
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void brush_enable(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out;
+ }
+
+ ts->brush_mode = sec->cmd_param[0];
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ goto out;
+ }
+
+ input_info(true, &ts->client->dev,
+ "%s: set brush mode %s\n", __func__, ts->brush_mode ? "enable" : "disable");
+
+ /* - 0: Disable Artcanvas min phi mode
+ - 1: Enable Artcanvas min phi mode */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_BRUSH_MODE, &ts->brush_mode, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to set brush mode\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out;
+ }
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+out:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void force_touch_active(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ int active;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) {
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ sec_cmd_set_cmd_exit(sec);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ active = sec->cmd_param[0];
+ if (sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_FORCE_ACTIVE, active) == 0) {
+ sec_cmd_set_cmd_result(sec, "OK", 2);
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ } else {
+ sec_cmd_set_cmd_result(sec, "NG", 2);
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ }
+ sec_cmd_set_cmd_exit(sec);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void set_touchable_area(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out;
+ }
+
+ ts->touchable_area = sec->cmd_param[0];
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ goto out;
+ }
+
+ input_info(true, &ts->client->dev,
+ "%s: set 16:9 mode %s\n", __func__, ts->touchable_area ? "enable" : "disable");
+
+ /* - 0: Disable 16:9 mode
+ * - 1: Enable 16:9 mode */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_TOUCHABLE_AREA, &ts->touchable_area, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to set 16:9 mode\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out;
+ }
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+out:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec_cmd_set_cmd_exit(sec);
+
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void set_log_level(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ char tBuff[2] = { 0 };
+ u8 w_data[1] = {0x00};
+ int ret;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Touch is stopped!\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ if ((sec->cmd_param[0] < 0 || sec->cmd_param[0] > 1) ||
+ (sec->cmd_param[1] < 0 || sec->cmd_param[1] > 1) ||
+ (sec->cmd_param[2] < 0 || sec->cmd_param[2] > 1) ||
+ (sec->cmd_param[3] < 0 || sec->cmd_param[3] > 1) ||
+ (sec->cmd_param[4] < 0 || sec->cmd_param[4] > 1) ||
+ (sec->cmd_param[5] < 0 || sec->cmd_param[5] > 1) ||
+ (sec->cmd_param[6] < 0 || sec->cmd_param[6] > 1) ||
+ (sec->cmd_param[7] < 0 || sec->cmd_param[7] > 1)) {
+ input_err(true, &ts->client->dev, "%s: para out of range\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "Para out of range");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+ }
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_STATUS_EVENT_TYPE, tBuff, 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Read Event type enable status fail\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "Read Stat Fail");
+ goto err;
+ }
+
+ input_info(true, &ts->client->dev, "%s: STATUS_EVENT enable = 0x%02X, 0x%02X\n",
+ __func__, tBuff[0], tBuff[1]);
+
+ tBuff[0] = BIT_STATUS_EVENT_VENDOR_INFO(sec->cmd_param[6]);
+ tBuff[1] = BIT_STATUS_EVENT_ERR(sec->cmd_param[0]) |
+ BIT_STATUS_EVENT_INFO(sec->cmd_param[1]) |
+ BIT_STATUS_EVENT_USER_INPUT(sec->cmd_param[2]);
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_STATUS_EVENT_TYPE, tBuff, 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Write Event type enable status fail\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "Write Stat Fail");
+ goto err;
+ }
+
+ if (sec->cmd_param[0] == 1 && sec->cmd_param[1] == 1 && sec->cmd_param[2] == 1 && sec->cmd_param[3] == 1 &&
+ sec->cmd_param[4] == 1 && sec->cmd_param[5] == 1 && sec->cmd_param[6] == 1 && sec->cmd_param[7] == 1) {
+ w_data[0] = 0x1;
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_VENDOR_EVENT_LEVEL, w_data, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Write Vendor Event Level fail\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "Write Stat Fail");
+ goto err;
+ }
+ } else {
+ w_data[0] = 0x0;
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SET_VENDOR_EVENT_LEVEL, w_data, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Write Vendor Event Level fail\n", __func__);
+ snprintf(buff, sizeof(buff), "%s", "Write Stat Fail");
+ goto err;
+ }
+ }
+
+ input_info(true, &ts->client->dev, "%s: ERROR : %d, INFO : %d, USER_INPUT : %d, INFO_CUSTOMLIB : %d, VENDOR_INFO : %d, VENDOR_EVENT_LEVEL : %d\n",
+ __func__, sec->cmd_param[0], sec->cmd_param[1], sec->cmd_param[2], sec->cmd_param[5], sec->cmd_param[6], w_data[0]);
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+ return;
+err:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
+
+static void debug(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+
+ sec_cmd_set_default_result(sec);
+
+ ts->temp = sec->cmd_param[0];
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+}
+
+static void not_support_cmd(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+
+ sec_cmd_set_default_result(sec);
+ snprintf(buff, sizeof(buff), "%s", "NA");
+
+ sec_cmd_set_cmd_result(sec, buff, strlen(buff));
+ sec->cmd_state = SEC_CMD_STATUS_NOT_APPLICABLE;
+ sec_cmd_set_cmd_exit(sec);
+}
+
+int sec_ts_fn_init(struct sec_ts_data *ts)
+{
+ int retval;
+
+ retval = sec_cmd_init(&ts->sec, sec_cmds,
+ ARRAY_SIZE(sec_cmds), SEC_CLASS_DEVT_TSP);
+ if (retval < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: Failed to sec_cmd_init\n", __func__);
+ goto exit;
+ }
+
+ retval = sysfs_create_group(&ts->sec.fac_dev->kobj,
+ &cmd_attr_group);
+ if (retval < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: FTS Failed to create sysfs attributes\n", __func__);
+ goto exit;
+ }
+
+ retval = sysfs_create_link(&ts->sec.fac_dev->kobj,
+ &ts->input_dev->dev.kobj, "input");
+ if (retval < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: Failed to create input symbolic link\n",
+ __func__);
+ goto exit;
+ }
+
+ ts->reinit_done = true;
+
+ return 0;
+
+exit:
+ return retval;
+}
+
+void sec_ts_fn_remove(struct sec_ts_data *ts)
+{
+ input_err(true, &ts->client->dev, "%s\n", __func__);
+
+ sysfs_remove_link(&ts->sec.fac_dev->kobj, "input");
+
+ sysfs_remove_group(&ts->sec.fac_dev->kobj,
+ &cmd_attr_group);
+
+ sec_cmd_exit(&ts->sec, SEC_CLASS_DEVT_TSP);
+
+}
+
+/*
+ * sec_ts_run_rawdata_all : read all raw data
+ *
+ * when you want to read full raw data (full_read : true)
+ * "mutual/self 3, 5, 29, 1, 19" data will be saved in log
+ *
+ * otherwise, (full_read : false, especially on boot time)
+ * only "mutual 3, 5, 29" data will be saved in log
+ */
+void sec_ts_run_rawdata_all(struct sec_ts_data *ts, bool full_read)
+{
+ short min[REGION_TYPE_COUNT], max[REGION_TYPE_COUNT];
+ enum spec_check_type spec_check = SPEC_NO_CHECK;
+ int ret, i, read_num;
+ u8 test_type[5] = {TYPE_AMBIENT_DATA, TYPE_DECODED_DATA,
+ TYPE_SIGNAL_DATA, TYPE_OFFSET_DATA_SEC, TYPE_OFFSET_DATA_SDC};
+#ifdef USE_PRESSURE_SENSOR
+ short pressure[3] = { 0 };
+ u8 cal_data[18] = { 0 };
+#endif
+
+ ts->tsp_dump_lock = 1;
+ input_info(true, &ts->client->dev,
+ "%s: start (wet:%d)##\n",
+ __func__, ts->wet_mode);
+
+ ret = sec_ts_fix_tmode(ts, TOUCH_SYSTEM_MODE_TOUCH,
+ TOUCH_MODE_STATE_TOUCH);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: failed to fix tmode\n",
+ __func__);
+ goto out;
+ }
+
+ if (full_read) {
+ read_num = 5;
+ } else {
+ read_num = 3;
+ test_type[read_num - 1] = TYPE_OFFSET_DATA_SDC;
+ }
+
+ for (i = 0; i < read_num; i++) {
+ ret = sec_ts_read_frame(ts, test_type[i], min, max,
+ &spec_check);
+ if (ret < 0)
+ input_info(true, &ts->client->dev,
+ "%s: mutual %d : error ## ret:%d\n",
+ __func__, test_type[i], ret);
+ else
+ input_info(true, &ts->client->dev,
+ "%s: mutual %d : Max/Min %d,%d ##\n",
+ __func__, test_type[i], max[0], min[0]);
+ sec_ts_delay(20);
+
+ if (full_read) {
+ ret = sec_ts_read_channel(ts, test_type[i], min,
+ max, &spec_check);
+ if (ret < 0)
+ input_info(true, &ts->client->dev,
+ "%s: self %d : error ## ret:%d\n",
+ __func__, test_type[i], ret);
+ else
+ input_info(true, &ts->client->dev,
+ "%s: self %d : Max/Min %d,%d ##\n",
+ __func__, test_type[i], max[0],
+ min[0]);
+ sec_ts_delay(20);
+ }
+ }
+
+#ifdef USE_PRESSURE_SENSOR
+ ret = sec_ts_fix_tmode(ts, TOUCH_SYSTEM_MODE_TOUCH,
+ TOUCH_MODE_STATE_TOUCH);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: failed to fix tmode\n",
+ __func__);
+ goto out;
+ }
+
+ /* run pressure offset data read */
+ read_pressure_data(ts, TYPE_OFFSET_DATA_SEC, pressure);
+ sec_ts_delay(20);
+
+ /* run pressure rawdata read */
+ read_pressure_data(ts, TYPE_RAW_DATA, pressure);
+ sec_ts_delay(20);
+
+ /* run pressure raw delta read */
+ read_pressure_data(ts, TYPE_REMV_AMB_DATA, pressure);
+ sec_ts_delay(20);
+
+ /* run pressure sigdata read */
+ read_pressure_data(ts, TYPE_SIGNAL_DATA, pressure);
+ sec_ts_delay(20);
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_CMD_SET_GET_PRESSURE, cal_data,
+ 18);
+ ts->pressure_left = ((cal_data[16] << 8) | cal_data[17]);
+ ts->pressure_center = ((cal_data[8] << 8) | cal_data[9]);
+ ts->pressure_right = ((cal_data[0] << 8) | cal_data[1]);
+ input_info(true, &ts->client->dev, "%s: pressure cal data - Left: %d, Center: %d, Right: %d\n",
+ __func__, ts->pressure_left, ts->pressure_center,
+ ts->pressure_right);
+#endif
+ sec_ts_release_tmode(ts);
+
+ ret = sec_ts_fix_tmode(ts, TOUCH_SYSTEM_MODE_TOUCH,
+ TOUCH_MODE_STATE_TOUCH);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: failed to fix tmode.\n",
+ __func__);
+ goto out;
+ }
+
+ sec_ts_read_gain_table(ts);
+
+ sec_ts_release_tmode(ts);
+out:
+ input_info(true, &ts->client->dev, "%s: ito : %02X %02X %02X %02X\n",
+ __func__, ts->ito_test[0], ts->ito_test[1]
+ , ts->ito_test[2], ts->ito_test[3]);
+
+ input_info(true, &ts->client->dev, "%s: done (wet:%d)##\n",
+ __func__, ts->wet_mode);
+ ts->tsp_dump_lock = 0;
+
+ sec_ts_locked_release_all_finger(ts);
+}
+
+static void run_rawdata_read_all(void *device_data)
+{
+ struct sec_cmd_data *sec = (struct sec_cmd_data *)device_data;
+ struct sec_ts_data *ts = container_of(sec, struct sec_ts_data, sec);
+ char buff[16] = { 0 };
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, true);
+
+ sec_cmd_set_default_result(sec);
+
+ if (ts->tsp_dump_lock == 1) {
+ input_err(true, &ts->client->dev, "%s: already checking now\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out;
+ }
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: IC is power off\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "NG");
+ sec->cmd_state = SEC_CMD_STATUS_FAIL;
+ goto out;
+ }
+
+ sec_ts_run_rawdata_all(ts, true);
+
+ snprintf(buff, sizeof(buff), "OK");
+ sec->cmd_state = SEC_CMD_STATUS_OK;
+out:
+ sec_cmd_set_cmd_result(sec, buff, strnlen(buff, sizeof(buff)));
+ input_info(true, &ts->client->dev, "%s: %s\n", __func__, buff);
+
+ sec_ts_set_bus_ref(ts, SEC_TS_BUS_REF_SYSFS, false);
+}
diff --git a/drivers/input/touchscreen/sec_ts/sec_ts_fw.c b/drivers/input/touchscreen/sec_ts/sec_ts_fw.c
new file mode 100644
index 0000000..efd9152
--- /dev/null
+++ b/drivers/input/touchscreen/sec_ts/sec_ts_fw.c
@@ -0,0 +1,1264 @@
+/* drivers/input/touchscreen/sec_ts_fw.c
+ *
+ * Copyright (C) 2015 Samsung Electronics Co., Ltd.
+ * http://www.samsungsemi.com/
+ *
+ * Core file for Samsung TSC driver
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include "sec_ts.h"
+
+#define SEC_TS_FW_BLK_SIZE 256
+
+enum {
+ BUILT_IN = 0,
+ UMS,
+ BL,
+ FFU,
+};
+
+typedef struct {
+ u32 signature; /* signature */
+ u32 version; /* version */
+ u32 totalsize; /* total size */
+ u32 checksum; /* checksum */
+ u32 img_ver; /* image file version */
+ u32 img_date; /* image file date */
+ u32 img_description; /* image file description */
+ u32 fw_ver; /* firmware version */
+ u32 fw_date; /* firmware date */
+ u32 fw_description; /* firmware description */
+ u32 para_ver; /* parameter version */
+ u32 para_date; /* parameter date */
+ u32 para_description; /* parameter description */
+ u32 num_chunk; /* number of chunk */
+ u32 reserved1;
+ u32 reserved2;
+} fw_header;
+
+typedef struct {
+ u32 signature;
+ u32 addr;
+ u32 size;
+ u32 reserved;
+} fw_chunk;
+
+static int sec_ts_enter_fw_mode(struct sec_ts_data *ts)
+{
+ int ret;
+ u8 fw_update_mode_passwd[] = {0x55, 0xAC};
+ u8 fw_status;
+ u8 id[3];
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_ENTER_FW_MODE, fw_update_mode_passwd, sizeof(fw_update_mode_passwd));
+ sec_ts_delay(20);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: write fail, enter_fw_mode\n", __func__);
+ return 0;
+ }
+
+ input_info(true, &ts->client->dev, "%s: write ok, enter_fw_mode - 0x%x 0x%x 0x%x\n",
+ __func__, SEC_TS_CMD_ENTER_FW_MODE, fw_update_mode_passwd[0], fw_update_mode_passwd[1]);
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_BOOT_STATUS, &fw_status, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: read fail, read_boot_status\n", __func__);
+ return 0;
+ }
+ if (fw_status != SEC_TS_STATUS_BOOT_MODE) {
+ input_err(true, &ts->client->dev, "%s: enter fail! read_boot_status = 0x%x\n", __func__, fw_status);
+ return 0;
+ }
+
+ input_info(true, &ts->client->dev, "%s: Success! read_boot_status = 0x%x\n", __func__, fw_status);
+
+ sec_ts_delay(10);
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_ID, id, 3);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: read id fail\n", __func__);
+ return 0;
+ }
+
+ ts->boot_ver[0] = id[0];
+ ts->boot_ver[1] = id[1];
+ ts->boot_ver[2] = id[2];
+
+ ts->flash_page_size = SEC_TS_FW_BLK_SIZE_DEFAULT;
+ if ((ts->boot_ver[1] == 0x37) && (ts->boot_ver[2] == 0x61))
+ ts->flash_page_size = 512;
+
+ input_info(true, &ts->client->dev, "%s: read_boot_id = %02X%02X%02X\n", __func__, id[0], id[1], id[2]);
+
+ return 1;
+}
+
+int sec_ts_hw_reset(struct sec_ts_data *ts)
+{
+ int reset_gpio = ts->plat_data->reset_gpio;
+
+ if (!gpio_is_valid(reset_gpio)) {
+ input_err(true, &ts->client->dev, "%s: invalid gpio %d\n",
+ __func__, reset_gpio);
+ return -EINVAL;
+ }
+
+ gpio_set_value(reset_gpio, 0);
+ sec_ts_delay(10);
+ gpio_set_value(reset_gpio, 1);
+
+ return 0;
+}
+
+int sec_ts_sw_reset(struct sec_ts_data *ts)
+{
+ int ret;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SW_RESET, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: write fail, sw_reset\n", __func__);
+ return 0;
+ }
+
+ sec_ts_delay(100);
+
+ ret = sec_ts_wait_for_ready(ts, SEC_TS_ACK_BOOT_COMPLETE);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: time out\n", __func__);
+ return 0;
+ }
+
+ input_info(true, &ts->client->dev, "%s: sw_reset\n", __func__);
+
+ /* Sense_on */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_ON, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: write fail, Sense_on\n", __func__);
+ return 0;
+ }
+
+ return ret;
+}
+
+int sec_ts_system_reset(struct sec_ts_data *ts)
+{
+ int ret = -1;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SW_RESET, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: write fail, sw_reset\n",
+ __func__);
+ else {
+ sec_ts_delay(100);
+ /* Normally it should not happen with any retry.
+ * But, if happened, retry less time to wait ack
+ */
+ ret = sec_ts_wait_for_ready_with_count(ts,
+ SEC_TS_ACK_BOOT_COMPLETE, 10);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: sw_reset time out!\n", __func__);
+ else
+ input_info(true,
+ &ts->client->dev, "%s: sw_reset done\n",
+ __func__);
+ }
+
+ if (ret < 0) {
+ if (!gpio_is_valid(ts->plat_data->reset_gpio)) {
+ input_err(true, &ts->client->dev,
+ "%s: reset gpio is unavailable!\n", __func__);
+ return ret;
+ }
+
+ input_err(true, &ts->client->dev,
+ "%s: sw_reset failed or time out, try hw_reset to recover!\n",
+ __func__);
+ ret = sec_ts_hw_reset(ts);
+ if (ret) {
+ input_err(true, &ts->client->dev,
+ "%s: hw_reset failed\n", __func__);
+ return ret;
+ }
+
+ ret = sec_ts_wait_for_ready_with_count(ts,
+ SEC_TS_ACK_BOOT_COMPLETE, 10);
+ if (ret < 0) {
+ input_err(true,
+ &ts->client->dev, "%s: hw_reset time out\n",
+ __func__);
+ return ret;
+ }
+ input_info(true, &ts->client->dev, "%s: hw_reset done\n",
+ __func__);
+ }
+
+ /* Sense_on */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_ON, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: write fail, Sense_on\n",
+ __func__);
+ return ret;
+ }
+
+ return 0;
+}
+
+static void sec_ts_save_version_of_bin(struct sec_ts_data *ts, const fw_header *fw_hd)
+{
+ ts->plat_data->img_version_of_bin[3] = ((fw_hd->img_ver >> 24) & 0xff);
+ ts->plat_data->img_version_of_bin[2] = ((fw_hd->img_ver >> 16) & 0xff);
+ ts->plat_data->img_version_of_bin[1] = ((fw_hd->img_ver >> 8) & 0xff);
+ ts->plat_data->img_version_of_bin[0] = ((fw_hd->img_ver >> 0) & 0xff);
+
+ ts->plat_data->core_version_of_bin[3] = ((fw_hd->fw_ver >> 24) & 0xff);
+ ts->plat_data->core_version_of_bin[2] = ((fw_hd->fw_ver >> 16) & 0xff);
+ ts->plat_data->core_version_of_bin[1] = ((fw_hd->fw_ver >> 8) & 0xff);
+ ts->plat_data->core_version_of_bin[0] = ((fw_hd->fw_ver >> 0) & 0xff);
+
+ ts->plat_data->config_version_of_bin[3] = ((fw_hd->para_ver >> 24) & 0xff);
+ ts->plat_data->config_version_of_bin[2] = ((fw_hd->para_ver >> 16) & 0xff);
+ ts->plat_data->config_version_of_bin[1] = ((fw_hd->para_ver >> 8) & 0xff);
+ ts->plat_data->config_version_of_bin[0] = ((fw_hd->para_ver >> 0) & 0xff);
+
+ input_info(true, &ts->client->dev, "%s: img_ver of bin = %x.%x.%x.%x\n", __func__,
+ ts->plat_data->img_version_of_bin[0],
+ ts->plat_data->img_version_of_bin[1],
+ ts->plat_data->img_version_of_bin[2],
+ ts->plat_data->img_version_of_bin[3]);
+
+ input_info(true, &ts->client->dev, "%s: core_ver of bin = %x.%x.%x.%x\n", __func__,
+ ts->plat_data->core_version_of_bin[0],
+ ts->plat_data->core_version_of_bin[1],
+ ts->plat_data->core_version_of_bin[2],
+ ts->plat_data->core_version_of_bin[3]);
+
+ input_info(true, &ts->client->dev, "%s: config_ver of bin = %x.%x.%x.%x\n", __func__,
+ ts->plat_data->config_version_of_bin[0],
+ ts->plat_data->config_version_of_bin[1],
+ ts->plat_data->config_version_of_bin[2],
+ ts->plat_data->config_version_of_bin[3]);
+}
+
+static int sec_ts_save_version_of_ic(struct sec_ts_data *ts)
+{
+ u8 img_ver[4] = {0,};
+ u8 core_ver[4] = {0,};
+ u8 config_ver[4] = {0,};
+ int ret;
+
+ /* Image ver */
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_IMG_VERSION, img_ver, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Image version read error\n", __func__);
+ return -EIO;
+ }
+ input_info(true, &ts->client->dev, "%s: IC Image version info : %x.%x.%x.%x\n",
+ __func__, img_ver[0], img_ver[1], img_ver[2], img_ver[3]);
+
+ ts->plat_data->img_version_of_ic[0] = img_ver[0];
+ ts->plat_data->img_version_of_ic[1] = img_ver[1];
+ ts->plat_data->img_version_of_ic[2] = img_ver[2];
+ ts->plat_data->img_version_of_ic[3] = img_ver[3];
+
+ /* Core ver */
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_FW_VERSION, core_ver, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: core version read error\n", __func__);
+ return -EIO;
+ }
+ input_info(true, &ts->client->dev, "%s: IC Core version info : %x.%x.%x.%x,\n",
+ __func__, core_ver[0], core_ver[1], core_ver[2], core_ver[3]);
+
+ ts->plat_data->core_version_of_ic[0] = core_ver[0];
+ ts->plat_data->core_version_of_ic[1] = core_ver[1];
+ ts->plat_data->core_version_of_ic[2] = core_ver[2];
+ ts->plat_data->core_version_of_ic[3] = core_ver[3];
+
+ /* Config ver */
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_PARA_VERSION, config_ver, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: config version read error\n", __func__);
+ return -EIO;
+ }
+ input_info(true, &ts->client->dev, "%s: IC config version info : %x.%x.%x.%x\n",
+ __func__, config_ver[0], config_ver[1], config_ver[2], config_ver[3]);
+
+ ts->plat_data->config_version_of_ic[0] = config_ver[0];
+ ts->plat_data->config_version_of_ic[1] = config_ver[1];
+ ts->plat_data->config_version_of_ic[2] = config_ver[2];
+ ts->plat_data->config_version_of_ic[3] = config_ver[3];
+
+ return 1;
+}
+
+static int sec_ts_check_firmware_version(struct sec_ts_data *ts, const u8 *fw_info)
+{
+ fw_header *fw_hd;
+ u8 buff[1];
+ int i;
+ int ret;
+ /*
+ * sec_ts_check_firmware_version
+ * return value = 1 : firmware download needed,
+ * return value = 0 : skip firmware download
+ */
+
+ fw_hd = (fw_header *)fw_info;
+
+ sec_ts_save_version_of_bin(ts, fw_hd);
+
+ /* firmware download if READ_BOOT_STATUS = 0x10 */
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_BOOT_STATUS, buff, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: fail to read BootStatus\n", __func__);
+ return -EIO;
+ }
+
+ if (buff[0] == SEC_TS_STATUS_BOOT_MODE) {
+ input_err(true, &ts->client->dev,
+ "%s: ReadBootStatus = 0x%x, Firmware download Start!\n",
+ __func__, buff[0]);
+ return 1;
+ }
+
+ ret = sec_ts_save_version_of_ic(ts);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail to read ic version\n", __func__);
+ return -EIO;
+ }
+
+ /* check f/w version
+ * ver[0] : IC version
+ * ver[1] : Project version
+ */
+ for (i = 0; i < 2; i++) {
+ if (ts->plat_data->img_version_of_ic[i] != ts->plat_data->img_version_of_bin[i]) {
+ input_err(true, &ts->client->dev, "%s: do not matched version info\n", __func__);
+ return 0;
+ }
+ }
+
+ for (i = 2; i < 4; i++) {
+ if (ts->plat_data->img_version_of_ic[i] !=
+ ts->plat_data->img_version_of_bin[i])
+ return 1;
+ }
+
+ return 0;
+}
+
+static u8 sec_ts_checksum(u8 *data, int offset, int size)
+{
+ int i;
+ u8 checksum = 0;
+
+ for (i = 0; i < size; i++)
+ checksum += data[i + offset];
+
+ return checksum;
+}
+
+static int sec_ts_flashpageerase(struct sec_ts_data *ts, u32 page_idx, u32 page_num)
+{
+ int ret;
+ u8 tCmd[6];
+
+ tCmd[0] = SEC_TS_CMD_FLASH_ERASE;
+ tCmd[1] = (u8)((page_idx >> 8) & 0xFF);
+ tCmd[2] = (u8)((page_idx >> 0) & 0xFF);
+ tCmd[3] = (u8)((page_num >> 8) & 0xFF);
+ tCmd[4] = (u8)((page_num >> 0) & 0xFF);
+ tCmd[5] = sec_ts_checksum(tCmd, 1, 4);
+
+ ret = ts->sec_ts_i2c_write_burst(ts, tCmd, 6);
+
+ return ret;
+}
+
+static int sec_ts_flashpagewrite(struct sec_ts_data *ts, u32 page_idx, u8 *page_data)
+{
+ int ret;
+ u8 tCmd[1 + 2 + SEC_TS_FW_BLK_SIZE_MAX + 1];
+ int flash_page_size = (int)ts->flash_page_size;
+
+ tCmd[0] = 0xD9;
+ tCmd[1] = (u8)((page_idx >> 8) & 0xFF);
+ tCmd[2] = (u8)((page_idx >> 0) & 0xFF);
+
+ memcpy(&tCmd[3], page_data, flash_page_size);
+ tCmd[1 + 2 + flash_page_size] = sec_ts_checksum(tCmd, 1, 2 + flash_page_size);
+
+ ret = ts->sec_ts_i2c_write_burst(ts, tCmd, 1 + 2 + flash_page_size + 1);
+ return ret;
+}
+
+static bool sec_ts_limited_flashpagewrite(struct sec_ts_data *ts, u32 page_idx, u8 *page_data)
+{
+ int ret = 0;
+ u8 *tCmd;
+ u8 copy_data[3 + SEC_TS_FW_BLK_SIZE_MAX];
+ int copy_left = (int)ts->flash_page_size + 3;
+ int copy_size = 0;
+ int copy_max = ts->i2c_burstmax - 1;
+ int flash_page_size = (int)ts->flash_page_size;
+
+ copy_data[0] = (u8)((page_idx >> 8) & 0xFF); /* addH */
+ copy_data[1] = (u8)((page_idx >> 0) & 0xFF); /* addL */
+
+ memcpy(©_data[2], page_data, flash_page_size); /* DATA */
+ copy_data[2 + flash_page_size] = sec_ts_checksum(copy_data, 0, 2 + flash_page_size); /* CS */
+
+ while (copy_left > 0) {
+ int copy_cur = (copy_left > copy_max) ? copy_max : copy_left;
+
+ tCmd = kzalloc(copy_cur + 1, GFP_KERNEL);
+ if (!tCmd)
+ goto err_write;
+
+ if (copy_size == 0)
+ tCmd[0] = SEC_TS_CMD_FLASH_WRITE;
+ else
+ tCmd[0] = SEC_TS_CMD_FLASH_PADDING;
+
+ memcpy(&tCmd[1], ©_data[copy_size], copy_cur);
+
+ ret = ts->sec_ts_i2c_write_burst_heap(ts, tCmd, 1 + copy_cur);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: failed, ret:%d\n", __func__, ret);
+
+ copy_size += copy_cur;
+ copy_left -= copy_cur;
+ kfree(tCmd);
+ }
+ return ret;
+
+err_write:
+ input_err(true, &ts->client->dev,
+ "%s: failed to alloc.\n", __func__);
+ return -ENOMEM;
+
+}
+
+static int sec_ts_flashwrite(struct sec_ts_data *ts, u32 mem_addr, u8 *mem_data, u32 mem_size, int retry)
+{
+ int ret;
+ u32 page_idx;
+ u32 size_copy;
+ u32 flash_page_size;
+ u32 page_idx_start;
+ u32 page_idx_end;
+ u32 page_num;
+ u8 page_buf[SEC_TS_FW_BLK_SIZE_MAX];
+
+ if (mem_size == 0)
+ return 0;
+
+ flash_page_size = ts->flash_page_size;
+ page_idx_start = mem_addr / flash_page_size;
+ page_idx_end = (mem_addr + mem_size - 1) / flash_page_size;
+ page_num = page_idx_end - page_idx_start + 1;
+
+ ret = sec_ts_flashpageerase(ts, page_idx_start, page_num);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: fw erase failed, mem_addr= %08X, pagenum = %d\n",
+ __func__, mem_addr, page_num);
+ return -EIO;
+ }
+
+ sec_ts_delay(page_num + 10);
+
+ size_copy = mem_size % flash_page_size;
+ if (size_copy == 0)
+ size_copy = flash_page_size;
+
+ memset(page_buf, 0, flash_page_size);
+
+ for (page_idx = page_num - 1;; page_idx--) {
+ memcpy(page_buf, mem_data + (page_idx * flash_page_size), size_copy);
+ if (ts->boot_ver[0] == 0xB2) {
+ ret = sec_ts_flashpagewrite(ts, (page_idx + page_idx_start), page_buf);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fw write failed, page_idx = %u\n", __func__, page_idx);
+ goto err;
+ }
+
+ if (retry) {
+ sec_ts_delay(50);
+ ret = sec_ts_flashpagewrite(ts, (page_idx + page_idx_start), page_buf);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fw write failed, page_idx = %u\n", __func__, page_idx);
+ goto err;
+ }
+ }
+ } else {
+ ret = sec_ts_limited_flashpagewrite(ts, (page_idx + page_idx_start), page_buf);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fw write failed, page_idx = %u\n", __func__, page_idx);
+ goto err;
+ }
+
+ if (retry) {
+ sec_ts_delay(50);
+ ret = sec_ts_limited_flashpagewrite(ts, (page_idx + page_idx_start), page_buf);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fw write failed, page_idx = %u\n", __func__, page_idx);
+ goto err;
+ }
+ }
+
+ }
+
+ size_copy = flash_page_size;
+ sec_ts_delay(5);
+
+ if (page_idx == 0) /* end condition (page_idx >= 0) page_idx type unsinged int */
+ break;
+ }
+
+ return mem_size;
+err:
+ return -EIO;
+}
+
+static int sec_ts_memoryblockread(struct sec_ts_data *ts, u32 mem_addr, int mem_size, u8 *buf)
+{
+ int ret;
+ u8 cmd[5];
+ u8 *data;
+
+ if (mem_size >= 64 * 1024) {
+ input_err(true, &ts->client->dev,
+ "%s: mem size over 64K\n", __func__);
+ return -EIO;
+ }
+
+ cmd[0] = (u8)SEC_TS_CMD_FLASH_READ_ADDR;
+ cmd[1] = (u8)((mem_addr >> 24) & 0xff);
+ cmd[2] = (u8)((mem_addr >> 16) & 0xff);
+ cmd[3] = (u8)((mem_addr >> 8) & 0xff);
+ cmd[4] = (u8)((mem_addr >> 0) & 0xff);
+
+ ret = ts->sec_ts_i2c_write_burst(ts, cmd, 5);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: send command failed, %02X\n", __func__, cmd[0]);
+ return -EIO;
+ }
+
+ udelay(10);
+ cmd[0] = (u8)SEC_TS_CMD_FLASH_READ_SIZE;
+ cmd[1] = (u8)((mem_size >> 8) & 0xff);
+ cmd[2] = (u8)((mem_size >> 0) & 0xff);
+
+ ret = ts->sec_ts_i2c_write_burst(ts, cmd, 3);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: send command failed, %02X\n", __func__, cmd[0]);
+ return -EIO;
+ }
+
+ udelay(10);
+ cmd[0] = (u8)SEC_TS_CMD_FLASH_READ_DATA;
+
+ data = buf;
+
+
+ ret = ts->sec_ts_i2c_read_heap(ts, cmd[0], data, mem_size);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: memory read failed\n", __func__);
+ return -EIO;
+ }
+/*
+ ret = ts->sec_ts_i2c_write(ts, cmd[0], NULL, 0);
+ ret = ts->sec_ts_i2c_read_bulk_heap(ts, data, mem_size);
+*/
+ return 0;
+}
+
+static int sec_ts_memoryread(struct sec_ts_data *ts, u32 mem_addr, u8 *mem_data, u32 mem_size)
+{
+ int ret;
+ int retry = 3;
+ int read_size = 0;
+ int unit_size;
+ int max_size = 1024;
+ int read_left = (int)mem_size;
+ u8 *tmp_data;
+
+ tmp_data = kmalloc(max_size, GFP_KERNEL);
+ if (!tmp_data) {
+ input_err(true, &ts->client->dev,
+ "%s: failed to kmalloc\n", __func__);
+ return -ENOMEM;
+ }
+
+ while (read_left > 0) {
+ unit_size = (read_left > max_size) ? max_size : read_left;
+ retry = 3;
+ do {
+ ret = sec_ts_memoryblockread(ts, mem_addr, unit_size, tmp_data);
+ if (retry-- == 0) {
+ input_err(true, &ts->client->dev,
+ "%s: fw read fail mem_addr=%08X,unit_size=%d\n",
+ __func__, mem_addr, unit_size);
+ kfree(tmp_data);
+ return -1;
+ }
+
+ memcpy(mem_data + read_size, tmp_data, unit_size);
+ } while (ret < 0);
+
+ mem_addr += unit_size;
+ read_size += unit_size;
+ read_left -= unit_size;
+ }
+
+ kfree(tmp_data);
+
+ return read_size;
+}
+
+static int sec_ts_chunk_update(struct sec_ts_data *ts, u32 addr, u32 size, u8 *data, int retry)
+{
+ u32 fw_size;
+ u32 write_size;
+ u8 *mem_rb;
+ int ret = 0;
+
+ fw_size = size;
+
+ write_size = sec_ts_flashwrite(ts, addr, data, fw_size, retry);
+ if (write_size != fw_size) {
+ input_err(true, &ts->client->dev, "%s: fw write failed\n", __func__);
+ ret = -1;
+ goto err_write_fail;
+ }
+
+ mem_rb = vzalloc(fw_size);
+ if (!mem_rb) {
+ input_err(true, &ts->client->dev, "%s: vzalloc failed\n", __func__);
+ ret = -1;
+ goto err_write_fail;
+ }
+
+ if (sec_ts_memoryread(ts, addr, mem_rb, fw_size) >= 0) {
+ u32 ii;
+
+ for (ii = 0; ii < fw_size; ii++) {
+ if (data[ii] != mem_rb[ii])
+ break;
+ }
+
+ if (fw_size != ii) {
+ input_err(true, &ts->client->dev, "%s: fw verify fail\n", __func__);
+ ret = -1;
+ goto out;
+ }
+ } else {
+ ret = -1;
+ goto out;
+ }
+
+ input_info(true, &ts->client->dev, "%s: verify done(%d)\n", __func__, ret);
+
+out:
+ vfree(mem_rb);
+err_write_fail:
+ sec_ts_delay(10);
+
+ return ret;
+}
+
+static int sec_ts_firmware_update(struct sec_ts_data *ts, const u8 *data, size_t size, int bl_update, int restore_cal, int retry)
+{
+ int i;
+ int ret;
+ fw_header *fw_hd;
+ fw_chunk *fw_ch;
+ u8 fw_status = 0;
+ u8 *fd = (u8 *)data;
+ u8 tBuff[3];
+#ifdef PAT_CONTROL
+ char buff[SEC_CMD_STR_LEN] = {0};
+ u8 img_ver[4];
+ bool magic_cal = false;
+#endif
+
+ /* Check whether CRC is appended or not.
+ * Enter Firmware Update Mode
+ */
+ if (!sec_ts_enter_fw_mode(ts)) {
+ input_err(true, &ts->client->dev, "%s: firmware mode failed\n", __func__);
+ return -1;
+ }
+
+ if (bl_update && (ts->boot_ver[0] == 0xB4)) {
+ input_info(true, &ts->client->dev, "%s: bootloader is up to date\n", __func__);
+ return 0;
+ }
+
+ input_info(true, &ts->client->dev, "%s: firmware update retry :%d\n", __func__, retry);
+
+ fw_hd = (fw_header *)fd;
+ fd += sizeof(fw_header);
+
+ if (fw_hd->signature != SEC_TS_FW_HEADER_SIGN) {
+ input_err(true, &ts->client->dev, "%s: firmware header error = %08X\n", __func__, fw_hd->signature);
+ return -1;
+ }
+
+ input_err(true, &ts->client->dev, "%s: num_chunk : %d\n", __func__, fw_hd->num_chunk);
+
+ for (i = 0; i < fw_hd->num_chunk; i++) {
+ fw_ch = (fw_chunk *)fd;
+
+ input_err(true, &ts->client->dev, "%s: [%d] 0x%08X, 0x%08X, 0x%08X, 0x%08X\n", __func__, i,
+ fw_ch->signature, fw_ch->addr, fw_ch->size, fw_ch->reserved);
+
+ if (fw_ch->signature != SEC_TS_FW_CHUNK_SIGN) {
+ input_err(true, &ts->client->dev, "%s: firmware chunk error = %08X\n", __func__, fw_ch->signature);
+ return -1;
+ }
+ fd += sizeof(fw_chunk);
+ ret = sec_ts_chunk_update(ts, fw_ch->addr, fw_ch->size, fd, retry);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: firmware chunk write failed, addr=%08X, size = %d\n", __func__, fw_ch->addr, fw_ch->size);
+ return -1;
+ }
+ fd += fw_ch->size;
+ }
+
+ sec_ts_sw_reset(ts);
+
+#ifdef PAT_CONTROL
+ if (restore_cal) {
+ if(ts->plat_data->pat_function == PAT_CONTROL_PAT_MAGIC) {
+ /* NOT to control cal count that was marked on external factory ( E0~E5 )*/
+ if ((ts->cal_count >= PAT_MAGIC_NUMBER) && (ts->cal_count < PAT_MAX_MAGIC))
+ magic_cal = true;
+ }
+ }
+ input_info(true, &ts->client->dev, "%s: cal_count(0x%02X) pat_function dt(%d) restore_cal(%d) magic_cal(%d)\n",
+ __func__,ts->cal_count,ts->plat_data->pat_function,restore_cal,magic_cal);
+#endif
+
+ if (!bl_update) {
+#ifdef PAT_CONTROL
+ if ((ts->cal_count == 0) || (ts->cal_count == 0xFF) || (magic_cal == true)) {
+ input_err(true, &ts->client->dev, "%s: RUN OFFSET CALIBRATION(0x%02X)\n", __func__, ts->cal_count);
+
+ ret = sec_ts_execute_force_calibration(ts, OFFSET_CAL_SEC);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to write OFFSET CAL SEC!\n", __func__);
+
+#ifdef USE_PRESSURE_SENSOR
+ ret = sec_ts_execute_force_calibration(ts, PRESSURE_CAL);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to write PRESSURE CAL!\n", __func__);
+#endif
+ if (ret >= 0 && magic_cal) {
+
+ ts->cal_count = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+ if (ts->cal_count == 0x00 || ts->cal_count == 0xFF)
+ ts->cal_count = PAT_MAGIC_NUMBER;
+ else if (PAT_MAGIC_NUMBER <= ts->cal_count && ts->cal_count < PAT_MAX_MAGIC)
+ ts->cal_count++;
+
+ /* Use TSP NV area : in this model, use only one byte
+ * buff[0] : offset from user NVM storage
+ * buff[1] : length of stored data - 1 (ex. using 1byte, value is 1 - 1 = 0)
+ * buff[2] : write data
+ */
+ buff[0] = SEC_TS_NVM_OFFSET_CAL_COUNT;
+ buff[1] = 0;
+ buff[2] = ts->cal_count;
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, buff, 3);
+ if (ret < 0)
+ input_err(true, &ts->client->dev,
+ "%s: nvm write failed. ret: %d\n", __func__, ret);
+
+ sec_ts_delay(20);
+ ts->cal_count = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+ input_info(true, &ts->client->dev, "%s: cal_count = [%02X]\n",
+ __func__, ts->cal_count);
+ }
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_IMG_VERSION, img_ver, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: Image version read error\n", __func__);
+ } else {
+ memset(buff, 0x00, SEC_CMD_STR_LEN);
+ buff[0] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION);
+ if (buff[0] == 0xFF) {
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION);
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION+1);
+ }
+
+ ts->tune_fix_ver = (img_ver[2] << 8 | img_ver[3]);
+ buff[0] = SEC_TS_NVM_OFFSET_TUNE_VERSION;
+ buff[1] = 1;// 2bytes
+ buff[2] = img_ver[2];
+ buff[3] = img_ver[3];
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_NVM, buff, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev,
+ "%s: nvm write failed. ret: %d\n", __func__, ret);
+ }
+ sec_ts_delay(20);
+
+ buff[0] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION);
+ buff[1] = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION+1);
+ ts->tune_fix_ver = buff[0]<<8 | buff[1];
+ input_info(true, &ts->client->dev, "%s: tune_fix_ver [%02X %02X]\n", __func__, buff[0], buff[1]);
+ }
+ } else {
+ input_err(true, &ts->client->dev, "%s: DO NOT CALIBRATION(0x%02X)\n", __func__, ts->cal_count);
+ }
+#else
+ /* always calibration after fw update */
+ input_err(true, &ts->client->dev, "%s: RUN OFFSET CALIBRATION\n", __func__);
+
+ ret = sec_ts_execute_force_calibration(ts, OFFSET_CAL_SEC);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to write OFFSET CAL SEC!\n", __func__);
+
+#ifdef USE_PRESSURE_SENSOR
+ ret = sec_ts_execute_force_calibration(ts, PRESSURE_CAL);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to write PRESSURE CAL!\n", __func__);
+#endif
+#endif
+
+ /* Sense_on */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_SENSE_ON, NULL, 0);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: write fail, Sense_on\n", __func__);
+ return -EIO;
+ }
+
+ if (ts->sec_ts_i2c_read(ts, SEC_TS_READ_BOOT_STATUS, &fw_status, 1) < 0) {
+ input_err(true, &ts->client->dev, "%s: read fail, read_boot_status = 0x%x\n", __func__, fw_status);
+ return -EIO;
+ }
+
+ if (fw_status != SEC_TS_STATUS_APP_MODE) {
+ input_err(true, &ts->client->dev, "%s: fw update sequence done, BUT read_boot_status = 0x%x\n", __func__, fw_status);
+ return -EIO;
+ }
+
+ input_info(true, &ts->client->dev, "%s: fw update Success! read_boot_status = 0x%x\n", __func__, fw_status);
+
+ return 1;
+ } else {
+
+ if (ts->sec_ts_i2c_read(ts, SEC_TS_READ_ID, tBuff, 3) < 0) {
+ input_err(true, &ts->client->dev, "%s: read device id fail after bl fw download\n", __func__);
+ return -EIO;
+ }
+
+ if (tBuff[0] == 0xA0) {
+ input_info(true, &ts->client->dev, "%s: bl fw download success - device id = %02X\n", __func__, tBuff[0]);
+ return -EIO;
+ } else {
+ input_err(true, &ts->client->dev, "%s: bl fw id does not match - device id = %02X\n", __func__, tBuff[0]);
+ return -EIO;
+ }
+ }
+
+}
+
+int sec_ts_firmware_update_bl(struct sec_ts_data *ts)
+{
+ const struct firmware *fw_entry;
+ char fw_path[SEC_TS_MAX_FW_PATH];
+ int result = -1;
+
+ disable_irq(ts->client->irq);
+
+ snprintf(fw_path, SEC_TS_MAX_FW_PATH, "%s", SEC_TS_DEFAULT_BL_NAME);
+
+ input_info(true, &ts->client->dev, "%s: initial bl update %s\n", __func__, fw_path);
+
+ /* Loading Firmware------------------------------------------ */
+ if (request_firmware(&fw_entry, fw_path, &ts->client->dev) != 0) {
+ input_err(true, &ts->client->dev, "%s: bt is not available\n", __func__);
+ goto err_request_fw;
+ }
+ input_info(true, &ts->client->dev, "%s: request bt done! size = %d\n", __func__, (int)fw_entry->size);
+
+ result = sec_ts_firmware_update(ts, fw_entry->data, fw_entry->size, 1, 0, 0);
+
+err_request_fw:
+ release_firmware(fw_entry);
+ enable_irq(ts->client->irq);
+
+ return result;
+}
+
+int sec_ts_bl_update(struct sec_ts_data *ts)
+{
+ int ret;
+ u8 tCmd[5] = { 0xDE, 0xAD, 0xBE, 0xEF };
+ u8 tBuff[3];
+
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_READ_BL_UPDATE_STATUS, tCmd, 4);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: bl update command send fail!\n", __func__);
+ goto err;
+ }
+ sec_ts_delay(10);
+
+ do {
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_BL_UPDATE_STATUS, tBuff, 1);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: read bl update status fail!\n", __func__);
+ goto err;
+ }
+ sec_ts_delay(2);
+
+ } while (tBuff[0] == 0x1);
+
+ tCmd[0] = 0x55;
+ tCmd[1] = 0xAC;
+ ret = ts->sec_ts_i2c_write(ts, 0x57, tCmd, 2);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: write passwd fail!\n", __func__);
+ goto err;
+ }
+
+ ret = ts->sec_ts_i2c_read(ts, SEC_TS_READ_ID, tBuff, 3);
+
+ if (tBuff[0] == 0xB4) {
+ input_info(true, &ts->client->dev, "%s: bl update completed!\n", __func__);
+ ret = 1;
+ } else {
+ input_info(true, &ts->client->dev, "%s: bl updated but bl version not matching, ver=%02X\n", __func__, tBuff[0]);
+ goto err;
+ }
+
+ return ret;
+err:
+ return -EIO;
+}
+
+int sec_ts_firmware_update_on_probe(struct sec_ts_data *ts, bool force_update)
+{
+ const struct firmware *fw_entry;
+ char fw_path[SEC_TS_MAX_FW_PATH];
+ int result = -1, restore_cal = 0;
+ int ii = 0;
+ int ret = 0;
+
+ if (ts->plat_data->bringup == 1) {
+ input_err(true, &ts->client->dev, "%s: bringup. do not update\n", __func__);
+ return 0;
+ }
+
+ if (ts->plat_data->firmware_name)
+ snprintf(fw_path, SEC_TS_MAX_FW_PATH, "%s", ts->plat_data->firmware_name);
+ else
+ return 0;
+
+ disable_irq(ts->client->irq);
+
+ /* read cal status */
+ ts->cal_status = sec_ts_read_calibration_report(ts);
+
+ input_info(true, &ts->client->dev, "%s: initial firmware update %s, cal:%X\n",
+ __func__, fw_path, ts->cal_status);
+
+ /* Loading Firmware */
+ if (request_firmware(&fw_entry, fw_path, &ts->client->dev) != 0) {
+ input_err(true, &ts->client->dev, "%s: firmware is not available\n", __func__);
+ goto err_request_fw;
+ }
+ input_info(true, &ts->client->dev, "%s: request firmware done! size = %d\n", __func__, (int)fw_entry->size);
+
+ result = sec_ts_check_firmware_version(ts, fw_entry->data);
+
+ if (ts->plat_data->bringup == 2) {
+ input_err(true, &ts->client->dev, "%s: bringup. do not update\n", __func__);
+ result = 0;
+ goto err_request_fw;
+ }
+
+#ifdef PAT_CONTROL
+ /* ic fw ver > bin fw ver && force is false*/
+ if ((result <= 0) && (!force_update)) {
+ /* clear nv, forced f/w update eventhough same f/w, then apply pat magic */
+ if (ts->plat_data->pat_function == PAT_CONTROL_FORCE_UPDATE) {
+ input_info(true, &ts->client->dev, "%s: run forced f/w update and excute autotune\n", __func__);
+ } else {
+ input_info(true, &ts->client->dev, "%s: skip - fw update & nv read\n", __func__);
+ goto err_request_fw;
+ }
+ }
+
+ ts->cal_count = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+ input_info(true, &ts->client->dev, "%s: cal_count [%02X]\n", __func__, ts->cal_count);
+
+ /* initialize nv default value from 0xff to 0x00 */
+ if (ts->cal_count == 0xFF) {
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION);
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION+1);
+ input_info(true, &ts->client->dev, "%s: initialize nv as default value & excute autotune\n", __func__);
+ }
+
+ ts->tune_fix_ver = (get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION) << 8) | get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_TUNE_VERSION+1);
+ input_info(true, &ts->client->dev, "%s: tune_fix_ver [%04X] afe_base [%04X]\n", __func__, ts->tune_fix_ver, ts->plat_data->afe_base);
+
+ /* check dt to clear pat */
+ if (ts->plat_data->pat_function == PAT_CONTROL_CLEAR_NV || ts->plat_data->pat_function == PAT_CONTROL_FORCE_UPDATE)
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+ ts->cal_count = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+
+ /* mismatch calibration - ic has too old calibration data after pat enabled*/
+ if (ts->plat_data->afe_base > ts->tune_fix_ver) {
+ restore_cal = 1;
+ set_tsp_nvm_data_clear(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+ ts->cal_count = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+ }
+
+ input_info(true, &ts->client->dev, "%s: cal_count [%02X]\n", __func__, ts->cal_count);
+#else
+ /* ic firmware version >= binary firmare version
+ * && forced is FALSE */
+ /* ic fw ver > bin fw ver && force is false*/
+ if ((result <= 0) && (!force_update)) {
+ input_info(true, &ts->client->dev, "%s: skip fw update\n", __func__);
+ goto err_request_fw;
+ }
+#endif
+
+ for (ii = 0; ii < 3; ii++) {
+ ret = sec_ts_firmware_update(ts, fw_entry->data, fw_entry->size, 0, restore_cal, ii);
+ if (ret >= 0)
+ break;
+ }
+
+ if (ret < 0) {
+ result = -1;
+ } else {
+ result = 0;
+#ifdef PAT_CONTROL
+ /* change cal_count from 0 to magic number to make virtual pure auto tune */
+ if ((ts->cal_count == 0 && ts->plat_data->pat_function == PAT_CONTROL_PAT_MAGIC)||
+ (ts->plat_data->pat_function == PAT_CONTROL_FORCE_UPDATE)) {
+ set_pat_magic_number(ts);
+ ts->cal_count = get_tsp_nvm_data(ts, SEC_TS_NVM_OFFSET_CAL_COUNT);
+ }
+#endif
+ }
+
+ sec_ts_save_version_of_ic(ts);
+
+err_request_fw:
+ release_firmware(fw_entry);
+ enable_irq(ts->client->irq);
+ return result;
+}
+
+static int sec_ts_load_fw_from_bin(struct sec_ts_data *ts)
+{
+ const struct firmware *fw_entry;
+ char fw_path[SEC_TS_MAX_FW_PATH];
+ int error = 0;
+
+ if (ts->client->irq)
+ disable_irq(ts->client->irq);
+
+ if (!ts->plat_data->firmware_name)
+ snprintf(fw_path, SEC_TS_MAX_FW_PATH, "%s", SEC_TS_DEFAULT_FW_NAME);
+ else
+ snprintf(fw_path, SEC_TS_MAX_FW_PATH, "%s", ts->plat_data->firmware_name);
+
+ input_info(true, &ts->client->dev, "%s: initial firmware update %s\n", __func__, fw_path);
+
+ /* Loading Firmware */
+ if (request_firmware(&fw_entry, fw_path, &ts->client->dev) != 0) {
+ input_err(true, &ts->client->dev, "%s: firmware is not available\n", __func__);
+ error = -1;
+ goto err_request_fw;
+ }
+ input_info(true, &ts->client->dev, "%s: request firmware done! size = %d\n", __func__, (int)fw_entry->size);
+
+ /* use virtual pat_control - magic cal 1 */
+ if (sec_ts_firmware_update(ts, fw_entry->data, fw_entry->size, 0, 1, 0) < 0)
+ error = -1;
+ else
+ error = 0;
+
+ sec_ts_save_version_of_ic(ts);
+
+err_request_fw:
+ release_firmware(fw_entry);
+ if (ts->client->irq)
+ enable_irq(ts->client->irq);
+
+ return error;
+}
+
+static int sec_ts_load_fw_from_ums(struct sec_ts_data *ts)
+{
+ fw_header *fw_hd;
+ struct file *fp;
+ mm_segment_t old_fs;
+ long fw_size, nread;
+ int error = 0;
+
+ old_fs = get_fs();
+ set_fs(KERNEL_DS);
+
+ fp = filp_open(SEC_TS_DEFAULT_UMS_FW, O_RDONLY, S_IRUSR);
+ if (IS_ERR(fp)) {
+ input_err(true, ts->dev, "%s: failed to open %s.\n", __func__,
+ SEC_TS_DEFAULT_UMS_FW);
+ error = -ENOENT;
+ goto open_err;
+ }
+
+ fw_size = fp->f_path.dentry->d_inode->i_size;
+
+ if (fw_size > 0) {
+ unsigned char *fw_data;
+
+ fw_data = kzalloc(fw_size, GFP_KERNEL);
+ nread = vfs_read(fp, (char __user *)fw_data,
+ fw_size, &fp->f_pos);
+
+ input_info(true, ts->dev,
+ "%s: start, file path %s, size %ld Bytes\n",
+ __func__, SEC_TS_DEFAULT_UMS_FW, fw_size);
+
+ if (nread != fw_size) {
+ input_err(true, ts->dev,
+ "%s: failed to read firmware file, nread %ld Bytes\n",
+ __func__, nread);
+ error = -EIO;
+ } else {
+ fw_hd = (fw_header *)fw_data;
+/*
+ sec_ts_check_firmware_version(ts, fw_data);
+*/
+ input_info(true, &ts->client->dev, "%s: firmware version %08X\n", __func__, fw_hd->fw_ver);
+ input_info(true, &ts->client->dev, "%s: parameter version %08X\n", __func__, fw_hd->para_ver);
+
+ if (ts->client->irq)
+ disable_irq(ts->client->irq);
+ /* use virtual pat_control - magic cal 1 */
+ if (sec_ts_firmware_update(ts, fw_data, fw_size, 0, 1, 0) < 0)
+ goto done;
+
+ sec_ts_save_version_of_ic(ts);
+ }
+
+ if (error < 0)
+ input_err(true, ts->dev, "%s: failed update firmware\n",
+ __func__);
+
+done:
+ if (ts->client->irq)
+ enable_irq(ts->client->irq);
+ kfree(fw_data);
+ }
+
+ filp_close(fp, NULL);
+
+open_err:
+ set_fs(old_fs);
+ return error;
+}
+
+static int sec_ts_load_fw_from_ffu(struct sec_ts_data *ts)
+{
+ const struct firmware *fw_entry;
+ const char *fw_path = SEC_TS_DEFAULT_FFU_FW;
+ int result = -1;
+
+ disable_irq(ts->client->irq);
+
+ input_info(true, ts->dev, "%s: Load firmware : %s\n", __func__, fw_path);
+
+ /* Loading Firmware */
+ if (request_firmware(&fw_entry, fw_path, &ts->client->dev) != 0) {
+ input_err(true, &ts->client->dev, "%s: firmware is not available\n", __func__);
+ goto err_request_fw;
+ }
+ input_info(true, &ts->client->dev, "%s: request firmware done! size = %d\n", __func__, (int)fw_entry->size);
+
+ sec_ts_check_firmware_version(ts, fw_entry->data);
+
+ if (sec_ts_firmware_update(ts, fw_entry->data, fw_entry->size, 0, 0, 0) < 0)
+ result = -1;
+ else
+ result = 0;
+
+ sec_ts_save_version_of_ic(ts);
+
+err_request_fw:
+ release_firmware(fw_entry);
+ enable_irq(ts->client->irq);
+ return result;
+}
+
+int sec_ts_firmware_update_on_hidden_menu(struct sec_ts_data *ts, int update_type)
+{
+ int ret = 0;
+
+ /* Factory cmd for firmware update
+ * argument represent what is source of firmware like below.
+ *
+ * 0 : [BUILT_IN] Getting firmware which is for user.
+ * 1 : [UMS] Getting firmware from sd card.
+ * 2 : none
+ * 3 : [FFU] Getting firmware from air.
+ */
+
+ switch (update_type) {
+ case BUILT_IN:
+ ret = sec_ts_load_fw_from_bin(ts);
+ break;
+ case UMS:
+ ret = sec_ts_load_fw_from_ums(ts);
+ break;
+ case FFU:
+ ret = sec_ts_load_fw_from_ffu(ts);
+ break;
+ case BL:
+ ret = sec_ts_firmware_update_bl(ts);
+ if (ret < 0) {
+ break;
+ } else if (!ret) {
+ ret = sec_ts_firmware_update_on_probe(ts, false);
+ break;
+ } else {
+ ret = sec_ts_bl_update(ts);
+ if (ret < 0)
+ break;
+ ret = sec_ts_firmware_update_on_probe(ts, false);
+ if (ret < 0)
+ break;
+ }
+ break;
+ default:
+ input_err(true, ts->dev, "%s: Not support command[%d]\n",
+ __func__, update_type);
+ break;
+ }
+
+#ifdef SEC_TS_SUPPORT_CUSTOMLIB
+ sec_ts_check_custom_library(ts);
+#endif
+
+ return ret;
+}
+EXPORT_SYMBOL(sec_ts_firmware_update_on_hidden_menu);
diff --git a/drivers/input/touchscreen/sec_ts/sec_ts_only_vendor.c b/drivers/input/touchscreen/sec_ts/sec_ts_only_vendor.c
new file mode 100644
index 0000000..3c866cf
--- /dev/null
+++ b/drivers/input/touchscreen/sec_ts/sec_ts_only_vendor.c
@@ -0,0 +1,302 @@
+/* drivers/input/touchscreen/sec_ts_fw.c
+ *
+ * Copyright (C) 2015 Samsung Electronics Co., Ltd.
+ * http://www.samsungsemi.com/
+ *
+ * Core file for Samsung TSC driver
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/delay.h>
+#include <linux/slab.h>
+#include <linux/i2c.h>
+#include <linux/input.h>
+#include <linux/input/mt.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/platform_device.h>
+#include <linux/firmware.h>
+#include <linux/gpio.h>
+#include <linux/regulator/consumer.h>
+#include <linux/irq.h>
+#include <linux/of_gpio.h>
+#include <linux/time.h>
+#include <linux/vmalloc.h>
+
+#include <linux/uaccess.h>
+/*#include <asm/gpio.h>*/
+
+#include "sec_ts.h"
+
+u8 lv1cmd;
+u8 *read_lv1_buff;
+static int lv1_readsize;
+static int lv1_readremain;
+static int lv1_readoffset;
+
+static ssize_t sec_ts_reg_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t size);
+static ssize_t sec_ts_regreadsize_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t size);
+static inline ssize_t sec_ts_store_error(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count);
+static ssize_t sec_ts_enter_recovery_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t size);
+static ssize_t sec_ts_regread_show(struct device *dev,
+ struct device_attribute *attr, char *buf);
+static ssize_t sec_ts_gesture_status_show(struct device *dev,
+ struct device_attribute *attr, char *buf);
+static inline ssize_t sec_ts_show_error(struct device *dev,
+ struct device_attribute *attr, char *buf);
+
+static DEVICE_ATTR(sec_ts_reg, (S_IWUSR | S_IWGRP), NULL, sec_ts_reg_store);
+static DEVICE_ATTR(sec_ts_regreadsize, (S_IWUSR | S_IWGRP), NULL, sec_ts_regreadsize_store);
+static DEVICE_ATTR(sec_ts_enter_recovery, (S_IWUSR | S_IWGRP), NULL, sec_ts_enter_recovery_store);
+static DEVICE_ATTR(sec_ts_regread, S_IRUGO, sec_ts_regread_show, NULL);
+static DEVICE_ATTR(sec_ts_gesture_status, S_IRUGO, sec_ts_gesture_status_show, NULL);
+
+static struct attribute *cmd_attributes[] = {
+ &dev_attr_sec_ts_reg.attr,
+ &dev_attr_sec_ts_regreadsize.attr,
+ &dev_attr_sec_ts_enter_recovery.attr,
+ &dev_attr_sec_ts_regread.attr,
+ &dev_attr_sec_ts_gesture_status.attr,
+ NULL,
+};
+
+static struct attribute_group cmd_attr_group = {
+ .attrs = cmd_attributes,
+};
+
+/* for debugging--------------------------------------------------------------------------------------*/
+static ssize_t sec_ts_reg_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t size)
+{
+ struct sec_ts_data *ts = dev_get_drvdata(dev);
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_info(true, &ts->client->dev, "%s: Power off state\n", __func__);
+ return -EIO;
+ }
+
+ if (size > 0)
+ ts->sec_ts_i2c_write_burst(ts, (u8 *)buf, size);
+
+ input_info(true, &ts->client->dev, "%s: 0x%x, 0x%x, size %d\n", __func__, buf[0], buf[1], (int)size);
+ return size;
+}
+
+static ssize_t sec_ts_regread_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct sec_ts_data *ts = dev_get_drvdata(dev);
+ int ret;
+ int length;
+ int remain;
+ int offset;
+
+ if (ts->power_status == SEC_TS_STATE_POWER_OFF) {
+ input_err(true, &ts->client->dev, "%s: Power off state\n", __func__);
+ return -EIO;
+ }
+
+ disable_irq(ts->client->irq);
+
+ mutex_lock(&ts->device_mutex);
+
+ read_lv1_buff = kzalloc(lv1_readsize, GFP_KERNEL);
+ if (!read_lv1_buff)
+ goto malloc_err;
+
+ remain = lv1_readsize;
+ offset = 0;
+ do {
+ if (remain >= ts->i2c_burstmax)
+ length = ts->i2c_burstmax;
+ else
+ length = remain;
+
+ if (offset == 0)
+ ret = ts->sec_ts_i2c_read_heap(ts, lv1cmd,
+ &read_lv1_buff[offset], length);
+ else
+ ret = ts->sec_ts_i2c_read_bulk_heap(ts,
+ &read_lv1_buff[offset], length);
+
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: i2c read %x command, remain =%d\n", __func__, lv1cmd, remain);
+ goto i2c_err;
+ }
+
+ remain -= length;
+ offset += length;
+ } while (remain > 0);
+
+ input_info(true, &ts->client->dev, "%s: lv1_readsize = %d\n", __func__, lv1_readsize);
+ memcpy(buf, read_lv1_buff + lv1_readoffset, lv1_readsize);
+
+i2c_err:
+ kfree(read_lv1_buff);
+malloc_err:
+ mutex_unlock(&ts->device_mutex);
+ lv1_readremain = 0;
+ enable_irq(ts->client->irq);
+
+ return lv1_readsize;
+}
+
+static ssize_t sec_ts_gesture_status_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct sec_ts_data *ts = dev_get_drvdata(dev);
+
+ mutex_lock(&ts->device_mutex);
+ memcpy(buf, ts->gesture_status, sizeof(ts->gesture_status));
+ input_info(true, &ts->client->dev,
+ "%s: GESTURE STATUS %x %x %x %x %x %x\n", __func__,
+ ts->gesture_status[0], ts->gesture_status[1], ts->gesture_status[2],
+ ts->gesture_status[3], ts->gesture_status[4], ts->gesture_status[5]);
+ mutex_unlock(&ts->device_mutex);
+
+ return sizeof(ts->gesture_status);
+}
+
+static ssize_t sec_ts_regreadsize_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t size)
+{
+ struct sec_ts_data *ts = dev_get_drvdata(dev);
+
+ mutex_lock(&ts->device_mutex);
+
+ lv1cmd = buf[0];
+ lv1_readsize = ((unsigned int)buf[4] << 24) |
+ ((unsigned int)buf[3] << 16) | ((unsigned int) buf[2] << 8) | ((unsigned int)buf[1] << 0);
+ lv1_readoffset = 0;
+ lv1_readremain = 0;
+
+ mutex_unlock(&ts->device_mutex);
+
+ return size;
+}
+
+static ssize_t sec_ts_enter_recovery_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t size)
+{
+ struct sec_ts_data *ts = dev_get_drvdata(dev);
+ struct sec_ts_plat_data *pdata = ts->plat_data;
+ int ret;
+ unsigned long on;
+
+ ret = kstrtoul(buf, 10, &on);
+ if (ret != 0) {
+ input_err(true, &ts->client->dev, "%s: failed to read:%d\n",
+ __func__, ret);
+ return -EINVAL;
+ }
+
+ if (on == 1) {
+ disable_irq(ts->client->irq);
+ gpio_free(pdata->irq_gpio);
+
+ input_info(true, &ts->client->dev, "%s: gpio free\n", __func__);
+ if (gpio_is_valid(pdata->irq_gpio)) {
+ ret = gpio_request_one(pdata->irq_gpio, GPIOF_OUT_INIT_LOW, "sec,tsp_int");
+ input_info(true, &ts->client->dev, "%s: gpio request one\n", __func__);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: Unable to request tsp_int [%d]: %d\n", __func__, pdata->irq_gpio, ret);
+ } else {
+ input_err(true, &ts->client->dev, "%s: Failed to get irq gpio\n", __func__);
+ return -EINVAL;
+ }
+
+ pdata->power(ts, false);
+ sec_ts_delay(100);
+ pdata->power(ts, true);
+ } else {
+ gpio_free(pdata->irq_gpio);
+
+ if (gpio_is_valid(pdata->irq_gpio)) {
+ ret = gpio_request_one(pdata->irq_gpio, GPIOF_DIR_IN, "sec,tsp_int");
+ if (ret) {
+ input_err(true, &ts->client->dev, "%s: Unable to request tsp_int [%d]\n", __func__, pdata->irq_gpio);
+ return -EINVAL;
+ }
+ } else {
+ input_err(true, &ts->client->dev, "%s: Failed to get irq gpio\n", __func__);
+ return -EINVAL;
+ }
+
+ pdata->power(ts, false);
+ sec_ts_delay(500);
+ pdata->power(ts, true);
+ sec_ts_delay(500);
+
+ /* AFE Calibration */
+ ret = ts->sec_ts_i2c_write(ts, SEC_TS_CMD_CALIBRATION_AMBIENT, NULL, 0);
+ if (ret < 0)
+ input_err(true, &ts->client->dev, "%s: fail to write AFE_CAL\n", __func__);
+
+ sec_ts_delay(1000);
+ enable_irq(ts->client->irq);
+ }
+
+ sec_ts_read_information(ts);
+
+ return size;
+}
+
+static inline ssize_t sec_ts_show_error(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct sec_ts_data *ts = dev_get_drvdata(dev);
+
+ input_err(true, &ts->client->dev, "%s: read only function, %s\n", __func__, attr->attr.name);
+ return -EPERM;
+}
+
+static inline ssize_t sec_ts_store_error(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct sec_ts_data *ts = dev_get_drvdata(dev);
+
+ input_err(true, &ts->client->dev, "%s: write only function, %s\n", __func__, attr->attr.name);
+ return -EPERM;
+}
+
+int sec_ts_raw_device_init(struct sec_ts_data *ts)
+{
+ int ret;
+
+#ifdef CONFIG_SEC_SYSFS
+ ts->dev = sec_device_create(ts, "sec_ts");
+#else
+ ts->dev = device_create(sec_class, NULL, 0, ts, "sec_ts");
+#endif
+ ret = IS_ERR(ts->dev);
+ if (ret) {
+ input_err(true, &ts->client->dev, "%s: fail - device_create\n", __func__);
+ return ret;
+ }
+
+ ret = sysfs_create_group(&ts->dev->kobj, &cmd_attr_group);
+ if (ret < 0) {
+ input_err(true, &ts->client->dev, "%s: fail - sysfs_create_group\n", __func__);
+ goto err_sysfs;
+ }
+
+ return ret;
+err_sysfs:
+ input_err(true, &ts->client->dev, "%s: fail\n", __func__);
+ return ret;
+}
+
+void sec_ts_raw_device_exit(struct sec_ts_data *ts)
+{
+ sysfs_remove_group(&ts->dev->kobj, &cmd_attr_group);
+#ifdef CONFIG_SEC_SYSFS
+ sec_device_destroy(ts->dev->devt)
+#else
+ device_destroy(sec_class, 0);
+#endif
+}
+
diff --git a/drivers/input/touchscreen/stm/Kconfig b/drivers/input/touchscreen/stm/Kconfig
new file mode 100644
index 0000000..d73e75b
--- /dev/null
+++ b/drivers/input/touchscreen/stm/Kconfig
@@ -0,0 +1,30 @@
+#
+# STMicroelectronics TOUCH driver configuration
+#
+
+config TOUCHSCREEN_FTS
+ tristate "STMicroelectronics i2c multitouch touchscreen - FingerTipS (FTM5)"
+ depends on I2C
+ select TOUCHSCREEN_HEATMAP
+ help
+ Say Y here to enable STMicroelectronics (FTM5) touchscreen support.
+ If unsure, say N.
+ To compile this driver as a module, choose M here: the module
+ will be called fts_touch.
+
+
+config TOUCHSCREEN_FTM4
+ tristate "STMicroelectronics i2c multitouch touchscreen - FTM4 with FingerTipS"
+ depends on I2C
+ help
+ Say Y here to enable STMicroelectronics touchscreen support.
+ If unsure, say N.
+ To compile this driver as a module, choose M here: the module
+ will be called fts_touch.
+
+config TOUCHSCREEN_FTM4_SHOW_EVENTS
+ boolean "Enable debug of FTM4 events"
+ depends on TOUCHSCREEN_FTM4
+ help
+ Say Y here to enable dumping of touch driver events. This should
+ only be enabled for debugging and not for shipping code.
diff --git a/drivers/input/touchscreen/stm/Makefile b/drivers/input/touchscreen/stm/Makefile
new file mode 100644
index 0000000..9d20024
--- /dev/null
+++ b/drivers/input/touchscreen/stm/Makefile
@@ -0,0 +1,11 @@
+# FTM5 support
+obj-$(CONFIG_TOUCHSCREEN_FTS) += ftm5.o
+ftm5-objs = fts.o fts_proc.o
+ftm5-objs += \
+ fts_lib/ftsCompensation.o fts_lib/ftsCore.o fts_lib/ftsError.o \
+ fts_lib/ftsFrame.o fts_lib/ftsIO.o fts_lib/ftsTest.o fts_lib/ftsTime.o \
+ fts_lib/ftsTool.o fts_lib/ftsFlash.o fts_lib/ftsGesture.o
+
+# Legacy driver
+obj-$(CONFIG_TOUCHSCREEN_FTM4) += ftm4.o
+ftm4-objs += ftm4_ts.o ftm4_fwu.o ftm4_pdc.o
diff --git a/drivers/input/touchscreen/stm/ftm4_fwu.c b/drivers/input/touchscreen/stm/ftm4_fwu.c
new file mode 100644
index 0000000..76e8db5
--- /dev/null
+++ b/drivers/input/touchscreen/stm/ftm4_fwu.c
@@ -0,0 +1,960 @@
+/******************** (C) COPYRIGHT 2012 STMicroelectronics ********************
+*
+* File Name : ftm4_fwu.c
+* Authors : AMS(Analog Mems Sensor) Team
+* Description : FTS Capacitive touch screen controller (FingerTipS)
+*
+********************************************************************************
+*
+* This program is free software; you can redistribute it and/or modify
+* it under the terms of the GNU General Public License version 2 as
+* published by the Free Software Foundation.
+*
+* THE PRESENT SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES
+* OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, FOR THE SOLE
+* PURPOSE TO SUPPORT YOUR APPLICATION DEVELOPMENT.
+* AS A RESULT, STMICROELECTRONICS SHALL NOT BE HELD LIABLE FOR ANY DIRECT,
+* INDIRECT OR CONSEQUENTIAL DAMAGES WITH RESPECT TO ANY CLAIMS ARISING FROM THE
+* CONTENT OF SUCH SOFTWARE AND/OR THE USE MADE BY CUSTOMERS OF THE CODING
+* INFORMATION CONTAINED HEREIN IN CONNECTION WITH THEIR PRODUCTS.
+*
+* THIS SOFTWARE IS SPECIFICALLY DESIGNED FOR EXCLUSIVE USE WITH ST PARTS.
+*******************************************************************************/
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/delay.h>
+#include <linux/input.h>
+#include <linux/firmware.h>
+#include <linux/slab.h>
+#include <asm/uaccess.h>
+#include <linux/hrtimer.h>
+#include <linux/interrupt.h>
+#include <linux/gpio.h>
+
+#include "ftm4_ts.h"
+
+#define FTS64FILE_SIGNATURE 0xaaaa5555
+
+enum {
+ BUILT_IN = 0,
+ UMS,
+ NONE,
+ FFU,
+};
+
+struct fts64_header {
+ unsigned int signature;
+ unsigned short fw_ver;
+ unsigned char fw_id;
+ unsigned char reserved1;
+ unsigned char internal_ver[8];
+ unsigned char released_ver[8];
+ unsigned int reserved2;
+ unsigned int checksum;
+};
+
+bool get_pure_autotune_status(struct fts_ts_info *info)
+{
+ int rc = 0;
+ unsigned char addrs[3];
+ unsigned char buf[5];
+ bool ret = false;
+ int doffset = 1;
+
+ if (info->digital_rev == FTS_DIGITAL_REV_1)
+ doffset = 0;
+
+ addrs[0] = 0xd0;
+ addrs[1] = 0x00;
+ addrs[2] = 0x4E;
+
+ rc = info->fts_read_reg(info, addrs, 3, buf, 4);
+ if (rc < 0) {
+ tsp_debug_err(info->dev,
+ "%s: PureAutotune Information Read Fail!!"
+ "[Data : %2X%2X]\n",
+ __func__, buf[0 + doffset],
+ buf[1 + doffset]);
+ } else {
+ if ((buf[0 + doffset] == 0xA5) && (buf[1 + doffset] == 0x96))
+ ret = 1;
+ tsp_debug_info(info->dev,
+ "%s: PureAutotune Information !! "
+ "[Data : %2X%2X]\n", __func__,
+ buf[0 + doffset],
+ buf[1 + doffset]);
+ }
+ return ret;
+}
+EXPORT_SYMBOL(get_pure_autotune_status);
+
+static bool get_afe_status(struct fts_ts_info *info)
+{
+ int rc = 0;
+ unsigned char addrs[3];
+ unsigned char buf[5];
+ bool ret = false;
+ int doffset = 1;
+
+ if (info->digital_rev == FTS_DIGITAL_REV_1)
+ doffset = 0;
+
+ addrs[0] = 0xd0;
+ addrs[1] = 0x00;
+ addrs[2] = 0x52;
+
+ rc = info->fts_read_reg(info, addrs, 3, buf, 4);
+ if (rc < 0) {
+ tsp_debug_err(info->dev,
+ "%s: Read Fail - Final AFE [Data :"
+ " %2X] AFE Ver [Data : %2X]\n",
+ __func__,
+ buf[0 + doffset],
+ buf[1 + doffset]);
+ return rc;
+ }
+
+ if (buf[0 + doffset])
+ ret = true;
+
+ tsp_debug_info(info->dev,
+ "%s: Final AFE [Data : %2X] AFE Ver "
+ "[Data : %2X]\n",
+ __func__,
+ buf[0 + doffset],
+ buf[1 + doffset]);
+
+ return ret;
+}
+
+int fts_fw_wait_for_specific_event(struct fts_ts_info *info,
+ unsigned char eid0, unsigned char eid1, unsigned char eid2)
+{
+ int rc = 0;
+ unsigned char addrs;
+ unsigned char data[FTS_EVENT_SIZE];
+ int retry = 0;
+
+ memset(data, 0x0, FTS_EVENT_SIZE);
+
+ addrs = READ_ONE_EVENT;
+
+ while (info->fts_read_reg(info, &addrs, 1, (unsigned char *)data,
+ FTS_EVENT_SIZE)) {
+ if (data[0]) {
+ if ((data[0] == eid0) && (data[1] == eid1) &&
+ (data[2] == eid2)) {
+ rc = 0;
+ break;
+ } else {
+ tsp_debug_dbg(info->dev,
+ "%s: %2X, %2X, %2X, %2X\n",
+ __func__, data[0], data[1],
+ data[2], data[3]);
+ }
+ }
+ if (retry++ > FTS_RETRY_COUNT * 15) {
+ rc = -1;
+ tsp_debug_err(info->dev,
+ "%s: Time Over ( %2X, %2X, %2X, %2X )\n",
+ __func__, data[0], data[1],
+ data[2], data[3]);
+ break;
+ }
+ fts_delay(20);
+ }
+
+ return rc;
+}
+
+int fts_fw_wait_for_event(struct fts_ts_info *info, unsigned char eid0,
+ unsigned char eid1)
+{
+ int rc = 0;
+ unsigned char addrs;
+ unsigned char data[FTS_EVENT_SIZE];
+ int retry = 0;
+
+ memset(data, 0x0, FTS_EVENT_SIZE);
+
+ addrs = READ_ONE_EVENT;
+
+ while (info->fts_read_reg(info, &addrs, 1, (unsigned char *)data,
+ FTS_EVENT_SIZE)) {
+ if ((data[0] == EVENTID_STATUS_EVENT) ||
+ (data[0] == EVENTID_ERROR)) {
+ if ((data[0] == EVENTID_STATUS_EVENT) &&
+ (data[1] == eid0) && (data[2] == eid1)) {
+ break;
+ } else if ((data[0] == EVENTID_STATUS_EVENT) &&
+ (data[1] == STATUS_EVENT_FORCE_CAL_DONE)) {
+ break;
+ } else {
+ tsp_debug_dbg(info->dev,
+ "%s: %2X,%2X,%2X,%2X\n",
+ __func__,
+ data[0],
+ data[1],
+ data[2],
+ data[3]);
+ }
+ }
+
+ if (retry++ > FTS_RETRY_COUNT * 15) {
+ rc = -1;
+ tsp_debug_err(info->dev,
+ "%s: Time Over (%2X,%2X,%2X,%2X)\n",
+ __func__,
+ data[0],
+ data[1],
+ data[2],
+ data[3]);
+ break;
+ }
+ fts_delay(20);
+ }
+
+ return rc;
+}
+
+void fts_execute_autotune(struct fts_ts_info *info)
+{
+ int ret = 0;
+ unsigned char regData[4]; /* {0xC1, 0x0E}; */
+ bool bFinalAFE = false;
+ bool NoNeedAutoTune = false; /* default for factory */
+
+ bFinalAFE = get_afe_status(info);
+
+ /* Check flag and decide cx_tune */
+ NoNeedAutoTune = get_pure_autotune_status(info);
+
+ tsp_debug_info(info->dev,
+ "%s: AFE(%d), NoNeedAutoTune(%d)\n", __func__,
+ bFinalAFE, NoNeedAutoTune);
+
+ if ((!NoNeedAutoTune) || (info->o_afe_ver != info->afe_ver)) {
+ info->fts_command(info, CX_TUNNING);
+ fts_delay(300);
+ fts_fw_wait_for_event(info, STATUS_EVENT_MUTUAL_AUTOTUNE_DONE,
+ 0x00);
+
+ info->fts_command(info, SELF_AUTO_TUNE);
+ fts_delay(300);
+ fts_fw_wait_for_event(info, STATUS_EVENT_SELF_AUTOTUNE_DONE,
+ 0x00);
+
+ if (NoNeedAutoTune) {
+ tsp_debug_info(info->dev,
+ "%s: AFE_status(%d) write ( C8 01 )\n",
+ __func__, bFinalAFE);
+
+ regData[0] = 0xC8;
+ regData[1] = 0x01;
+ ret = info->fts_write_reg(info, regData, 2);
+ if (ret < 0) {
+ tsp_debug_err(info->dev,
+ "%s: Flash Back up PureAutotune"
+ "Fail (Clear)\n", __func__);
+ }
+
+ fts_delay(20);
+ fts_fw_wait_for_event(info,
+ STATUS_EVENT_PURE_AUTOTUNE_FLAG_CLEAR_FINISH,
+ 0x00);
+ }
+
+ info->fts_command(info, FTS_CMD_SAVE_CX_TUNING);
+ fts_delay(230);
+ fts_fw_wait_for_event(info,
+ STATUS_EVENT_FLASH_WRITE_CXTUNE_VALUE, 0x00);
+
+ info->fts_command(info, FTS_CMD_SAVE_FWCONFIG);
+ fts_delay(230);
+ fts_fw_wait_for_event(info, STATUS_EVENT_FLASH_WRITE_CONFIG,
+ 0x00);
+
+ /* Reset FTS */
+ info->fts_systemreset(info);
+ fts_delay(20);
+ /* wait for ready event */
+ info->fts_wait_for_ready(info);
+ }
+}
+
+#define FW_IMAGE_NAME_D2_TB_INTEG "tsp_stm/stm_tb_integ.fw"
+#define FW_IMAGE_NAME_D2_Z2A "tsp_stm/stm_z2a.fw"
+#define FW_IMAGE_NAME_D2_Z2I "tsp_stm/stm_z2i.fw"
+#define CONFIG_ID_D1_S 0x2C
+#define CONFIG_ID_D2_TR 0x2E
+#define CONFIG_ID_D2_TB 0x30
+#define CONFIG_OFFSET_BIN_D1 0xf822
+#define CONFIG_OFFSET_BIN_D2 0x1E822
+#define RX_OFFSET_BIN_D2 0x1E834
+#define TX_OFFSET_BIN_D2 0x1E835
+
+#define FW_IMAGE_SIZE_D3 (256 * 1024)
+#define SIGNEDKEY_SIZE (256)
+
+static int wait_for_flash_ready(struct fts_ts_info *info, uint8_t type)
+{
+ uint8_t cmd[2] = {FLASH_CMD_READ_REGISTER, type};
+ uint8_t readData;
+ int i, res = -1;
+
+ tsp_debug_info(info->dev, "[wait_for_flash_ready"
+ " Waiting for flash ready\n");
+
+ for (i = 0; i < 1000 && res != 0; i++) {
+ info->fts_read_reg(info, cmd, sizeof(cmd), &readData, 1);
+ res = readData & 0x80;
+ fts_delay(50);
+ }
+
+ if (i >= 1000 && res != 0) {
+ tsp_debug_err(info->dev, "[wait_for_flash_ready]"
+ " Wait for flash TIMEOUT! ERROR\n");
+ return 0;
+ }
+
+ tsp_debug_info(info->dev, "[wait_for_flash_ready]"
+ " Flash READY!\n");
+
+ return 1;
+}
+
+static int start_flash_dma(struct fts_ts_info *info)
+{
+ int status;
+ uint8_t cmd[3] = {FLASH_CMD_WRITE_REGISTER,
+ FLASH_DMA_CODE0, FLASH_DMA_CODE1};
+
+ tsp_debug_info(info->dev,
+ "[start_flash_dma] Command flash DMA ...\n");
+ info->fts_write_reg(info, cmd, sizeof(cmd));
+
+ status = wait_for_flash_ready(info, FLASH_DMA_CODE0);
+
+ if (status != true) {
+ tsp_debug_err(info->dev,
+ "[start_flash_dma] start_flash_dma: ERROR\n");
+ return false;
+ }
+ tsp_debug_info(info->dev, "[start_flash_dma] flash DMA DONE!\n");
+
+ return true;
+}
+
+static int fillFlash(struct fts_ts_info *info, uint32_t address, uint8_t *data,
+ int size)
+{
+ int remaining;
+ int toWrite = 0;
+ int byteBlock = 0;
+ int wheel = 0;
+ uint32_t addr = 0;
+ int res;
+ int delta;
+
+ uint8_t buff[DMA_CHUNK + 3] = {0};
+
+ remaining = size;
+ while (remaining > 0) {
+ byteBlock = 0;
+ addr = 0;
+ tsp_debug_info(info->dev,
+ "[fillFlash] [%d] Write data to memory.\n",
+ wheel);
+ while (byteBlock < FLASH_CHUNK && remaining > 0) {
+ buff[0] = FLASH_CMD_WRITE_64K;
+ if (remaining >= DMA_CHUNK) {
+ if ((byteBlock + DMA_CHUNK) <= FLASH_CHUNK) {
+ toWrite = DMA_CHUNK;
+ remaining -= DMA_CHUNK;
+ byteBlock += DMA_CHUNK;
+ } else {
+ delta = FLASH_CHUNK - byteBlock;
+ toWrite = delta;
+ remaining -= delta;
+ byteBlock += delta;
+ }
+ } else {
+ if ((byteBlock + remaining) <= FLASH_CHUNK) {
+ toWrite = remaining;
+ byteBlock += remaining;
+ remaining = 0;
+
+ } else {
+ delta = FLASH_CHUNK - byteBlock;
+ toWrite = delta;
+ remaining -= delta;
+ byteBlock += delta;
+ }
+ }
+
+ buff[1] = (uint8_t) ((addr & 0x0000FF00) >> 8);
+ buff[2] = (uint8_t) (addr & 0x000000FF);
+ memcpy(&buff[3], data, toWrite);
+ info->fts_write_reg(info, buff, 3 + toWrite);
+
+ addr += toWrite;
+ data += toWrite;
+ }
+
+ /* configuring the DMA */
+ tsp_debug_info(info->dev,
+ "[fillFlash] [%d] Configure DMA\n", wheel);
+ byteBlock = byteBlock / 4 - 1;
+
+ buff[0] = FLASH_CMD_WRITE_REGISTER;
+ buff[1] = FLASH_DMA_CONFIG;
+ buff[2] = 0x00;
+ buff[3] = 0x00;
+
+ addr = address + ((wheel * FLASH_CHUNK)/4);
+ buff[4] = (uint8_t) ((addr & 0x000000FF));
+ buff[5] = (uint8_t) ((addr & 0x0000FF00) >> 8);
+ buff[6] = (uint8_t) (byteBlock & 0x000000FF);
+ buff[7] = (uint8_t) ((byteBlock & 0x0000FF00) >> 8);
+ buff[8] = 0x00;
+
+ info->fts_write_reg(info, buff, 9);
+ fts_delay(10);
+
+ tsp_debug_info(info->dev,
+ "[fillFlash] [%d] Start flash DMA\n", wheel);
+ res = start_flash_dma(info);
+ if (res < true) {
+ tsp_debug_err(info->dev,
+ "[fillFlash] Error during flashing DMA! ERROR\n");
+ return false;
+ }
+ tsp_debug_info(info->dev,
+ "[fillFlash] [%d] DMA done\n", wheel);
+
+ wheel++;
+ }
+ return true;
+}
+
+uint32_t convU8toU32(uint8_t *src)
+{
+ uint32_t tmpData;
+
+ tmpData = (uint32_t) (((src[3] & 0xFF) << 24) +
+ ((src[2] & 0xFF) << 16) +
+ ((src[1] & 0xFF) << 8) +
+ (src[0] & 0xFF));
+
+ return tmpData;
+}
+
+static int parseBinFile(struct fts_ts_info *info, uint8_t *data,
+ int fw_size,
+ struct FW_FTB_HEADER *fw_header,
+ int keep_cx)
+{
+ int dimension, index;
+ uint32_t temp;
+ int file_type;
+
+ /* start the parsing */
+ index = 0;
+ fw_header->signature = convU8toU32(&data[index]);
+ if (fw_header->signature == FW_HEADER_FTB_SIGNATURE) {
+ tsp_debug_info(info->dev,
+ "[parseBinFile] FW Signature - ftb file\n");
+ file_type = BIN_FTB;
+ } else {
+ tsp_debug_info(info->dev,
+ "[parseBinFile] FW Signature - ftsxxx file. %08X\n",
+ fw_header->signature);
+ file_type = BIN_FTS256;
+ return file_type;
+ }
+
+ index += FW_BYTES_ALLIGN;
+#ifdef FTS_USE_FTB_1
+ fw_header->ftb_ver = convU8toU32(&data[index]);
+ if (fw_header->ftb_ver != FW_FTB_VER) {
+ tsp_debug_err(info->dev,
+ "[parseBinFile] Wrong" " ftb_version %08X ... ERROR\n",
+ fw_header->ftb_ver);
+ return false;
+ }
+#endif /* FTS_USE_FTB_1 */
+
+ index += FW_BYTES_ALLIGN;
+ fw_header->target = convU8toU32(&data[index]);
+ if (fw_header->target != 0x00007036) {
+ tsp_debug_err(info->dev,
+ "[parseBinFile] Wrong target version %08X ... ERROR\n",
+ fw_header->target);
+ return false;
+ }
+
+ index += FW_BYTES_ALLIGN;
+ fw_header->fw_id = convU8toU32(&data[index]);
+
+ index += FW_BYTES_ALLIGN;
+ fw_header->fw_ver = convU8toU32(&data[index]);
+ info->fw_version_of_bin = fw_header->fw_ver;
+
+ index += FW_BYTES_ALLIGN;
+ fw_header->cfg_id = convU8toU32(&data[index]);
+
+ index += FW_BYTES_ALLIGN;
+ fw_header->cfg_ver = convU8toU32(&data[index]);
+ info->config_version_of_bin = fw_header->cfg_ver;
+
+ index += FW_BYTES_ALLIGN * 3; /* skip 2 reserved data */
+ fw_header->bl_fw_ver = convU8toU32(&data[index]);
+ index += FW_BYTES_ALLIGN;
+
+ fw_header->ext_ver = convU8toU32(&data[index]);
+
+ tsp_debug_info(info->dev,
+ "[parseBinFile] Version : External"
+ " = %04X, FW = %04X, CFG = %04X\n",
+ fw_header->ext_ver,
+ fw_header->fw_ver,
+ fw_header->cfg_ver);
+
+ index += FW_BYTES_ALLIGN;
+ fw_header->sec0_size = convU8toU32(&data[index]);
+
+ index += FW_BYTES_ALLIGN;
+ fw_header->sec1_size = convU8toU32(&data[index]);
+
+ tsp_debug_info(info->dev,
+ "[parseBinFile] sec0_size = %08X"
+ " (%d bytes), sec1_size = %08X (%d bytes)\n",
+ fw_header->sec0_size,
+ fw_header->sec0_size,
+ fw_header->sec1_size,
+ fw_header->sec1_size);
+
+ index += FW_BYTES_ALLIGN;
+ fw_header->sec2_size = convU8toU32(&data[index]);
+
+ index += FW_BYTES_ALLIGN;
+ fw_header->sec3_size = convU8toU32(&data[index]);
+
+ tsp_debug_info(info->dev,
+ "[parseBinFile] sec2_size = %08X"
+ " (%d bytes), sec3_size = %08X (%d bytes)\n",
+ fw_header->sec2_size,
+ fw_header->sec2_size,
+ fw_header->sec3_size,
+ fw_header->sec3_size);
+
+ index += FW_BYTES_ALLIGN;
+ fw_header->hdr_crc = convU8toU32(&data[index]);
+
+ if (!keep_cx) {
+ dimension = fw_header->sec0_size + fw_header->sec1_size
+ + fw_header->sec2_size + fw_header->sec3_size;
+
+ temp = fw_size;
+ } else {
+ /* sec2 may contain cx data (future implementation)
+ * sec3 atm not used */
+ dimension = fw_header->sec0_size + fw_header->sec1_size;
+ temp = fw_size - fw_header->sec2_size - fw_header->sec3_size;
+ }
+
+ if (dimension + FW_HEADER_SIZE + FW_BYTES_ALLIGN != temp) {
+ tsp_debug_info(info->dev,
+ "[parseBinFile] Read only %d"
+ " instead of %d... ERROR\n",
+ fw_size, dimension
+ + FW_HEADER_SIZE
+ + FW_BYTES_ALLIGN);
+
+ return false;
+ }
+
+ return file_type;
+}
+
+static int fts_check_erase_done(struct fts_ts_info *info)
+{
+ int timeout = 60; /* 3 sec timeout */
+ unsigned char addrs[2] = {0xF9, 0x02};
+ unsigned char val[1];
+ int rc = 0;
+
+ do {
+ info->fts_read_reg(info, &addrs[0], 2, (unsigned char *)val, 1);
+
+ if ((val[0] & 0x80) != 0x80)
+ break;
+
+ fts_delay(50);
+ timeout--;
+ } while (timeout != 0);
+
+ if (timeout == 0)
+ rc = -1;
+
+ return rc;
+}
+
+int fw_download(struct fts_ts_info *info, uint8_t *pFilename,
+ struct FW_FTB_HEADER *fw_Header, int8_t block_type)
+{
+ uint32_t FTS_TOTAL_SIZE = (256 * 1024); /* Total 256kB */
+ int HEADER_DATA_SIZE = 32;
+
+ int res = 0, rc = 0, i = 0;
+ uint8_t addrs[8] = {0};
+
+ /* System reset */
+ /* System Reset ==> F7 52 34 */
+
+ addrs[0] = 0xF7;
+ addrs[1] = 0x52;
+ addrs[2] = 0x34;
+ info->fts_write_reg(info, &addrs[0], 3);
+ fts_delay(30);
+
+ /* Unlock Flash */
+ /* Unlock Flash Command ==> F7 74 45 */
+ addrs[0] = 0xF7;
+ addrs[1] = 0x74;
+ addrs[2] = 0x45;
+ info->fts_write_reg(info, &addrs[0], 3);
+ fts_delay(100);
+
+ /* Unlock Erase Operation */
+ addrs[0] = 0xFA;
+ addrs[1] = 0x72;
+ addrs[2] = 0x01;
+ info->fts_write_reg(info, &addrs[0], 3);
+ fts_delay(30);
+
+ /* Erase Partial Flash */
+ for (i = 0; i < 64; i++) {
+ /* skip CX2 area (page 61 and page 62) */
+ if ((i == 61) || (i == 62))
+ continue;
+
+ addrs[0] = 0xFA;
+ addrs[1] = 0x02;
+ addrs[2] = (0x80 + i) & 0xFF;
+ info->fts_write_reg(info, &addrs[0], 3);
+ rc = fts_check_erase_done(info);
+ if (rc < 0)
+ return rc;
+ }
+
+ /* Unlock Programming operation */
+ addrs[0] = 0xFA;
+ addrs[1] = 0x72;
+ addrs[2] = 0x02;
+ info->fts_write_reg(info, &addrs[0], 3);
+ fts_delay(100);
+
+ /* Write to FLASH */
+ if (block_type == BIN_FTB) {
+ tsp_debug_info(info->dev,
+ "[fw_download] Start sec0 program\n");
+
+ res = fillFlash(info,
+ FLASH_ADDR_CODE,
+ &pFilename[FW_HEADER_SIZE],
+ fw_Header->sec0_size);
+
+ if (res != true) {
+ tsp_debug_err(info->dev,
+ "[fw_download] Error - load sec0 program\n");
+ return false;
+ }
+ tsp_debug_info(info->dev,
+ "[fw_download] load sec0 program DONE!\n");
+ tsp_debug_info(info->dev,
+ "[fw_download] Start sec1 program\n");
+
+ res = fillFlash(info,
+ FLASH_ADDR_CONFIG,
+ &pFilename[FW_HEADER_SIZE +
+ fw_Header->sec0_size],
+ fw_Header->sec1_size);
+
+ if (res != true) {
+ tsp_debug_err(info->dev,
+ "[fw_download] Error - load sec1 program\n");
+ return false;
+ }
+ tsp_debug_info(info->dev,
+ "[fw_download] load sec1 program DONE!\n");
+
+ tsp_debug_info(info->dev,
+ "[fw_download] Flash burn COMPLETED!\n");
+ } else {
+ tsp_debug_info(info->dev,
+ "[fw_download] Start firmware downloading\n");
+ res = fillFlash(info, FLASH_ADDR_CODE,
+ &pFilename[HEADER_DATA_SIZE], FTS_TOTAL_SIZE);
+ if (res != true) {
+ tsp_debug_err(info->dev,
+ "[fw_download] Error - load sec0 program\n");
+ return false;
+ }
+ }
+
+ /* System reset */
+ addrs[0] = 0xF7;
+ addrs[1] = 0x52;
+ addrs[2] = 0x34;
+ info->fts_write_reg(info, &addrs[0], 3);
+ if (fts_cmd_completion_check(info, 0x10, 0x00, 0x00) < 0) {
+ tsp_debug_err(info->dev,
+ "[fw_download] Error - System Reset FAILED\n");
+ return false;
+ }
+
+ return true;
+}
+
+static int fts_fw_compare(struct fts_ts_info *info, const struct firmware *fw)
+{
+ u32 bin_fw_ver_addr_1 = 0;
+ u32 bin_fw_ver_addr_2 = 0;
+ u32 bin_fw_ver_offset = 24;
+ u8 buf[2] = {0};
+ struct fts_version *binary = NULL;
+ struct fts_version *device = &info->ic_fw_ver;
+ int update = 0;
+
+ if ((u32)fw->size < bin_fw_ver_offset) {
+ tsp_debug_err(info->dev,
+ " fw->size(0x%08X) < bin_fw_ver_offset(0x%08X)\n",
+ (u32)fw->size, bin_fw_ver_offset);
+ update = 0;
+ goto error;
+ }
+
+ bin_fw_ver_addr_1 = (u32)fw->size - bin_fw_ver_offset;
+ bin_fw_ver_addr_2 = bin_fw_ver_addr_1 + 1;
+ tsp_debug_info(info->dev,
+ "%s: bin_fw_ver_addr_1 = 0x%08X , bin_fw_ver_addr_2 = 0x%08X\n",
+ __func__, bin_fw_ver_addr_1, bin_fw_ver_addr_2);
+
+ binary = kzalloc(sizeof(struct fts_version), GFP_KERNEL);
+ if (binary == NULL) {
+ tsp_debug_err(info->dev, "failed to kzalloc binary\n");
+ update = 0;
+ goto error;
+ }
+
+ buf[0] = fw->data[bin_fw_ver_addr_1];
+ buf[1] = fw->data[bin_fw_ver_addr_2];
+
+ binary->build = (buf[0] >> 4) & 0x0F;
+ binary->major = buf[0] & 0x0F;
+ binary->minor = buf[1];
+
+ if (binary->major != device->major) {
+ update = 1;
+ } else {
+ if (binary->minor != device->minor)
+ update = 1;
+ else if (binary->build > device->build)
+ update = 1;
+ }
+
+ tsp_debug_info(info->dev,
+ "%s : binary[%d.%02d.%d] device[%d.%02d.%d]"
+ " -> update: %d\n", __func__,
+ binary->major, binary->minor, binary->build,
+ device->major, device->minor, device->build,
+ update);
+
+error:
+ if (binary)
+ kfree(binary);
+
+ return update;
+}
+
+void fts_fw_init(struct fts_ts_info *info)
+{
+ tsp_debug_info(info->dev, "%s\n", __func__);
+
+ info->fts_command(info, FTS_CMD_TRIM_LOW_POWER_OSCILLATOR);
+ fts_delay(200);
+ info->fts_command(info, FTS_CMD_SAVE_CX_TUNING);
+ fts_delay(230);
+ fts_fw_wait_for_event(info, STATUS_EVENT_FLASH_WRITE_CXTUNE_VALUE, 0x00);
+
+ fts_get_afe_info(info);
+
+ fts_execute_autotune(info);
+
+ info->fts_command(info, SENSEON);
+
+ fts_fw_wait_for_event(info, STATUS_EVENT_FORCE_CAL_DONE, 0x00);
+
+ info->fts_interrupt_set(info, INT_ENABLE);
+}
+
+static int fts_fw_check(struct fts_ts_info *info)
+{
+ int retval = 0;
+
+ retval = fts_systemreset(info);
+ if (retval < 0)
+ return retval;
+
+ retval = fts_wait_for_ready(info);
+ if (retval < 0)
+ return retval;
+
+ retval = fts_read_chip_id(info);
+ if (retval < 0)
+ return retval;
+
+ return retval;
+}
+
+static int fts_fw_update(struct fts_ts_info *info)
+{
+ const struct firmware *fw_entry = NULL;
+ unsigned char *fw_data = NULL;
+ char fw_path[FTS_MAX_FW_PATH];
+ const struct FW_FTB_HEADER *header;
+ int fw_size;
+ int fw_type;
+ int keep_cx;
+ int retval = 0;
+ struct FW_FTB_HEADER fw_ftbHeader;
+
+ if (info->fts_power_state != FTS_POWER_STATE_ACTIVE) {
+ tsp_debug_err(info->dev,
+ "%s : FTS_POWER_STATE is not ACTIVE\n", __func__);
+ return -EPERM;
+ }
+
+ if (info->test_fwpath[0]) {
+ strlcpy(fw_path, &info->test_fwpath[0], sizeof(fw_path));
+ } else if(info->board->firmware_name) {
+ /* A pointer and size of buffer for binary file */
+ strlcpy(fw_path, &info->board->firmware_name[0], sizeof(fw_path));
+ } else {
+ tsp_debug_err(info->dev, "%s : no firmware file\n", __func__);
+ return -EPERM;
+ }
+
+ tsp_debug_info(info->dev,
+ "%s : firmware name : %s\n", __func__, fw_path);
+
+ retval = request_firmware(&fw_entry, fw_path, info->dev);
+ if (retval) {
+ tsp_debug_err(info->dev,
+ "%s : Firmware image %s not available\n", __func__,
+ fw_path);
+ return retval;
+ }
+
+ if (!fts_fw_compare(info, fw_entry)) {
+ tsp_debug_info(info->dev,
+ "%s : skip fw_upgrade(ic_fw_ver == bin_fw_ver)\n",
+ __func__);
+ goto out;
+ }
+
+ fw_size = fw_entry->size;
+ fw_data = (unsigned char *)fw_entry->data;
+ header = (struct FW_FTB_HEADER *)fw_data;
+
+ info->fw_version_of_bin = header->fw_ver;
+ info->config_version_of_bin = header->cfg_ver;
+ /* saver previous afe version before downloading */
+ info->o_afe_ver = info->afe_ver;
+#ifdef FTS_FTB_STYLE_2
+ info->fw_main_version_of_bin =
+ ((header->ext_ver & 0xff)<<8) +
+ ((header->ext_ver >> 8) & 0xff);
+
+ tsp_debug_info(info->dev,
+ "Bin Firmware Version : 0x%04X "
+ "Bin Config Version : 0x%04X "
+ "Bin Main Firmware Version : 0x%04X ",
+ info->fw_version_of_bin,
+ info->config_version_of_bin,
+ info->fw_main_version_of_bin);
+#else /* FTS_FTB_STYLE_2 */
+ tsp_debug_info(info->dev,
+ "Bin Firmware Version : 0x%04X "
+ "Bin Config Version : 0x%04X ",
+ info->fw_version_of_bin,
+ info->config_version_of_bin);
+#endif
+ memcpy(&fw_ftbHeader, fw_data, sizeof(struct FW_FTB_HEADER));
+
+ tsp_debug_info(info->dev,
+ "[flashProcedure] Firmware size : %d\n", fw_size);
+ keep_cx = 0;
+
+ fw_type = parseBinFile(info, fw_data, fw_size, &fw_ftbHeader, keep_cx);
+ if (fw_type == false) {
+ tsp_debug_err(info->dev,
+ "[flashProcedure] Error - FW is not appreciate\n");
+ retval = -EINVAL;
+ goto out;
+ }
+
+ retval = fw_download(info, fw_data, &fw_ftbHeader, fw_type);
+ if (retval == 0) {
+ tsp_debug_err(info->dev,
+ "[flashProcedure] Error - Firmware update is not completed.\n");
+ retval = -EIO;
+ goto out;
+ }
+
+ fts_fw_init(info);
+ retval = fts_fw_check(info);
+ if (retval < 0 ||
+ info->flash_corruption_info.fw_broken ||
+ info->flash_corruption_info.cfg_broken ||
+ info->flash_corruption_info.cx_broken) {
+ retval = -EIO;
+ goto out;
+ }
+
+ fts_get_version_info(info);
+ if (fts_fw_compare(info, fw_entry)) {
+ tsp_debug_err(info->dev,
+ "[flashProcedure] Firmware update failed\n");
+ retval = -EIO;
+ goto out;
+ }
+
+ tsp_debug_info(info->dev,
+ "[flashProcedure] Firmware update is done successfully.\n");
+ retval = 0;
+out:
+ if (fw_entry)
+ release_firmware(fw_entry);
+ return retval;
+}
+
+int fts_fw_verify_update(struct fts_ts_info *info)
+{
+ int retry = 0;
+
+ info->fts_irq_enable(info, false);
+ while (retry++ < FTS_FW_UPDATE_RETRY) {
+ tsp_debug_info(info->dev,
+ "[fw_update] try:%d\n", retry);
+ if (0 == fts_fw_update(info)) {
+ info->fts_irq_enable(info, true);
+ return 0;
+ }
+ }
+ info->fts_irq_enable(info, true);
+ return -EIO;
+}
+EXPORT_SYMBOL(fts_fw_verify_update);
diff --git a/drivers/input/touchscreen/stm/ftm4_pdc.c b/drivers/input/touchscreen/stm/ftm4_pdc.c
new file mode 100644
index 0000000..83daa21
--- /dev/null
+++ b/drivers/input/touchscreen/stm/ftm4_pdc.c
@@ -0,0 +1,2265 @@
+/******************** (C) COPYRIGHT 2012 STMicroelectronics ********************
+*
+* File Name : ftm4_pdc.c
+* Authors : AMS(Analog Mems Sensor) Team
+* Description : FTS Capacitive touch screen controller (FingerTipS)
+*
+********************************************************************************
+*
+* This program is free software; you can redistribute it and/or modify
+* it under the terms of the GNU General Public License version 2 as
+* published by the Free Software Foundation.
+*
+* THE PRESENT SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES
+* OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, FOR THE SOLE
+* PURPOSE TO SUPPORT YOUR APPLICATION DEVELOPMENT.
+* AS A RESULT, STMICROELECTRONICS SHALL NOT BE HELD LIABLE FOR ANY DIRECT,
+* INDIRECT OR CONSEQUENTIAL DAMAGES WITH RESPECT TO ANY CLAIMS ARISING FROM THE
+* CONTENT OF SUCH SOFTWARE AND/OR THE USE MADE BY CUSTOMERS OF THE CODING
+* INFORMATION CONTAINED HEREIN IN CONNECTION WITH THEIR PRODUCTS.
+*
+* THIS SOFTWARE IS SPECIFICALLY DESIGNED FOR EXCLUSIVE USE WITH ST PARTS.
+*******************************************************************************/
+
+#include <linux/init.h>
+#include <linux/errno.h>
+#include <linux/platform_device.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/input.h>
+#include <linux/interrupt.h>
+#include <linux/serio.h>
+#include <linux/init.h>
+#include <linux/pm.h>
+#include <linux/delay.h>
+#include <linux/ctype.h>
+#include <linux/gpio.h>
+#include <linux/i2c.h>
+#include <linux/i2c-dev.h>
+#include <linux/fs.h>
+#include <linux/uaccess.h>
+#include <linux/power_supply.h>
+#include <linux/firmware.h>
+#include <linux/regulator/consumer.h>
+#include <linux/of_gpio.h>
+#include <linux/device.h>
+#include <linux/list.h>
+#include "ftm4_ts.h"
+
+#define TSP_FACTEST_RESULT_PASS 2
+#define TSP_FACTEST_RESULT_FAIL 1
+#define TSP_FACTEST_RESULT_NONE 0
+
+#define BUFFER_MAX ((256 * 1024) - 16)
+#define READ_CHUNK_SIZE 128
+
+#define FTS_F_WIX1_ADDR 0x1FE7
+#define FTS_S_WIX1_ADDR 0x1FE8
+#define FTS_F_WIX2_ADDR 0x18FD
+#define FTS_S_WIX2_ADDR 0x1929
+#define FTS_WATER_SELF_RAW_ADDR 0x1E
+
+#define FTS_MAX_TX_LENGTH 44
+#define FTS_MAX_RX_LENGTH 64
+
+#define FTS_CX2_READ_LENGTH 4
+#define FTS_CX2_ADDR_OFFSET 3
+#define FTS_CX2_TX_START 0
+#define FTS_CX2_BASE_ADDR 0x1000
+#define SEC_CMD_STR_LEN 12
+
+#define DEBUG_MSG 1
+enum {
+ TYPE_RAW_DATA = 0,
+ TYPE_FILTERED_DATA = 2,
+ TYPE_STRENGTH_DATA = 4,
+ TYPE_BASELINE_DATA = 6
+};
+
+enum {
+ BUILT_IN = 0,
+ UMS,
+};
+
+enum CMD_STATUS {
+ CMD_STATUS_WAITING = 0,
+ CMD_STATUS_RUNNING,
+ CMD_STATUS_OK,
+ CMD_STATUS_FAIL,
+ CMD_STATUS_NOT_APPLICABLE,
+};
+
+static void get_fw_ver_bin(void *device_data);
+static void get_fw_ver_ic(void *device_data);
+static void get_config_ver(void *device_data);
+static void get_threshold(void *device_data);
+static void get_x_num(void *device_data);
+static void get_y_num(void *device_data);
+static void run_rawcap_read(void *device_data);
+static void get_rawcap(void *device_data);
+static void run_ix_data_read(void *device_data);
+static void run_ix_data_read_all(void *device_data);
+static void run_self_raw_read(void *device_data);
+static void run_self_raw_read_all(void *device_data);
+static void get_cx_data(void *device_data);
+static void run_cx_data_read(void *device_data);
+static void get_cx_all_data(void *device_data);
+static void get_strength_all_data(void *device_data);
+static void set_tsp_test_result(void *device_data);
+static void get_tsp_test_result(void *device_data);
+static void run_trx_short_test(void *device_data);
+static void report_rate(void *device_data);
+static void delay(void *device_data);
+static void debug(void *device_data);
+static void run_autotune_enable(void *device_data);
+static void run_autotune(void *device_data);
+static void not_support_cmd(void *device_data);
+
+static ssize_t store_cmd(struct device *dev, struct device_attribute *devattr,
+ const char *buf, size_t count);
+static ssize_t show_cmd_status(struct device *dev,
+ struct device_attribute *devattr, char *buf);
+static ssize_t show_cmd_result(struct device *dev,
+ struct device_attribute *devattr, char *buf);
+static ssize_t cmd_list_show(struct device *dev,
+ struct device_attribute *attr, char *buf);
+static ssize_t store_upgrade(struct device *dev,
+ struct device_attribute *devattr, const char *buf, size_t count);
+static ssize_t store_check_fw(struct device *dev,
+ struct device_attribute *devattr, const char *buf, size_t count);
+static ssize_t show_version_info(struct device *dev,
+ struct device_attribute *devattr, char *buf);
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+static void tui_mode_cmd(struct fts_ts_info *info);
+#endif
+
+struct fts_cmd fts_commands[] = {
+ {FTS_CMD("get_fw_ver_bin", get_fw_ver_bin),},
+ {FTS_CMD("get_fw_ver_ic", get_fw_ver_ic),},
+ {FTS_CMD("get_config_ver", get_config_ver),},
+ {FTS_CMD("get_threshold", get_threshold),},
+ {FTS_CMD("get_x_num", get_x_num),},
+ {FTS_CMD("get_y_num", get_y_num),},
+ {FTS_CMD("run_rawcap_read", run_rawcap_read),},
+ {FTS_CMD("get_rawcap", get_rawcap),},
+ {FTS_CMD("run_ix_data_read", run_ix_data_read),},
+ {FTS_CMD("run_ix_data_read_all", run_ix_data_read_all),},
+ {FTS_CMD("run_self_raw_read", run_self_raw_read),},
+ {FTS_CMD("run_self_raw_read_all", run_self_raw_read_all),},
+ {FTS_CMD("get_cx_data", get_cx_data),},
+ {FTS_CMD("run_cx_data_read", run_cx_data_read),},
+ {FTS_CMD("get_cx_all_data", get_cx_all_data),},
+ {FTS_CMD("get_strength_all_data", get_strength_all_data),},
+ {FTS_CMD("set_tsp_test_result", set_tsp_test_result),},
+ {FTS_CMD("get_tsp_test_result", get_tsp_test_result),},
+ {FTS_CMD("report_rate", report_rate),},
+ {FTS_CMD("delay", delay),},
+ {FTS_CMD("debug", debug),},
+ {FTS_CMD("run_autotune_enable", run_autotune_enable),},
+ {FTS_CMD("run_autotune", run_autotune),},
+ {FTS_CMD("run_trx_short_test", run_trx_short_test),},
+ {FTS_CMD("not_support_cmd", not_support_cmd),},
+};
+
+static DEVICE_ATTR(cmd, S_IWUSR | S_IWGRP, NULL, store_cmd);
+static DEVICE_ATTR(cmd_status, S_IRUGO, show_cmd_status, NULL);
+static DEVICE_ATTR(cmd_result, S_IRUGO, show_cmd_result, NULL);
+static DEVICE_ATTR(cmd_list, S_IRUGO, cmd_list_show, NULL);
+static DEVICE_ATTR(fw_upgrade, S_IWUSR | S_IWGRP, NULL, store_upgrade);
+static DEVICE_ATTR(check_fw, S_IWUSR | S_IWGRP, NULL, store_check_fw);
+static DEVICE_ATTR(version, S_IRUGO, show_version_info, NULL);
+
+static struct attribute *touch_pdc_attributes[] = {
+ &dev_attr_cmd.attr,
+ &dev_attr_cmd_status.attr,
+ &dev_attr_cmd_result.attr,
+ &dev_attr_cmd_list.attr,
+ &dev_attr_fw_upgrade.attr,
+ &dev_attr_check_fw.attr,
+ &dev_attr_version.attr,
+ NULL,
+};
+
+static struct attribute_group touch_pdc_attr_group = {
+ .attrs = touch_pdc_attributes,
+};
+
+static ssize_t store_check_fw(struct device *dev, struct device_attribute *devattr, const char *buf, size_t count)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+ unsigned int input = 0;
+ int ret = 0;
+
+ if (sscanf(buf, "%u", &input) != 1) {
+ tsp_debug_err(&info->client->dev, "%s: Invalid argument\n", __func__);
+ return -EINVAL;
+ }
+
+ if (input) {
+ mutex_lock(&info->device_mutex);
+
+ info->test_fwpath[0] = '\0';
+ ret = fts_fw_verify_update(info);
+
+ mutex_unlock(&info->device_mutex);
+ }
+
+ if (ret)
+ return ret;
+ else
+ return count;
+}
+
+static ssize_t store_upgrade(struct device *dev,
+ struct device_attribute *devattr, const char *buf, size_t count)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+ int ret = 0;
+
+ if (strlcpy(&info->test_fwpath[0], buf, count) <= 0) {
+ tsp_debug_err(&info->client->dev, "%s: invalid firmware name\n", __func__);
+ return -EINVAL;
+ }
+
+ mutex_lock(&info->device_mutex);
+
+ ret = fts_fw_verify_update(info);
+ info->test_fwpath[0] = '\0';
+
+ mutex_unlock(&info->device_mutex);
+
+ if (ret)
+ return ret;
+ else
+ return count;
+}
+
+static ssize_t show_version_info(struct device *dev,
+ struct device_attribute *devattr, char *buf)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+ int offset = 0;
+ char str[16] = {0};
+ int ret = 0;
+
+ mutex_lock(&info->device_mutex);
+
+ if (fts_get_version_info(info) < 0) {
+ mutex_unlock(&info->device_mutex);
+ return -EINVAL;
+ }
+
+ mutex_unlock(&info->device_mutex);
+
+ ret += snprintf(str + ret, sizeof(str) - ret,
+ "v%d.%02d", info->ic_fw_ver.major, info->ic_fw_ver.minor);
+
+ if (info->ic_fw_ver.build) {
+ ret += snprintf(str + ret, sizeof(str) - ret,
+ ".%d", info->ic_fw_ver.build);
+ }
+
+ offset += snprintf(buf + offset, PAGE_SIZE - offset, "%s\n", str);
+
+ return offset;
+}
+
+static int fts_check_index(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+ int node;
+
+ if (info->cmd_param[0] < 0 ||
+ info->cmd_param[0] >= info->SenseChannelLength ||
+ info->cmd_param[1] < 0 ||
+ info->cmd_param[1] >= info->ForceChannelLength) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ strncat(info->cmd_result, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_FAIL;
+ tsp_debug_info(&info->client->dev,
+ "%s: parameter error: %u,%u\n",
+ __func__, info->cmd_param[0], info->cmd_param[1]);
+ node = -1;
+ return node;
+ }
+ node = info->cmd_param[1] * info->SenseChannelLength +
+ info->cmd_param[0];
+ tsp_debug_info(&info->client->dev, "%s: node = %d\n", __func__, node);
+ return node;
+}
+
+static ssize_t store_cmd(struct device *dev, struct device_attribute *devattr,
+ const char *buf, size_t count)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+ char *cur, *start, *end;
+ char buff[CMD_STR_LEN] = { 0 };
+ int len, i;
+ struct fts_cmd *ft_cmd_ptr = NULL;
+ char delim = ',';
+ bool cmd_found = false;
+ int param_cnt = 0;
+
+ if (!info) {
+ tsp_debug_err(&info->client->dev,
+ "%s: No platform data found\n", __func__);
+ return -EINVAL;
+ }
+
+ if (!info->input_dev) {
+ tsp_debug_err(&info->client->dev,
+ "%s: No input_dev data found\n", __func__);
+ return -EINVAL;
+ }
+
+ if (count > CMD_STR_LEN) {
+ tsp_debug_err(&info->client->dev,
+ "%s: overflow command length\n", __func__);
+ return -EINVAL;
+ }
+
+ if (info->cmd_is_running == true) {
+ tsp_debug_err(&info->client->dev,
+ "ft_cmd: other cmd is running.\n");
+ if (strncmp("clear_cover_mode", buf, 16) == 0) {
+ cancel_delayed_work(&info->cover_cmd_work);
+ tsp_debug_err(&info->client->dev,
+ "[cmd is delayed] %d, param = %d, %d\n",
+ __LINE__, buf[17]-'0', buf[19]-'0');
+ info->delayed_cmd_param[0] = buf[17]-'0';
+ if (info->delayed_cmd_param[0] > 1)
+ info->delayed_cmd_param[1] = buf[19]-'0';
+
+ schedule_delayed_work(&info->cover_cmd_work,
+ msecs_to_jiffies(10));
+ }
+ return -EBUSY;
+ } else if (info->reinit_done == false) {
+ tsp_debug_err(&info->client->dev,
+ "ft_cmd: reinit is working\n");
+ if (strncmp("clear_cover_mode", buf, 16) == 0) {
+ cancel_delayed_work(&info->cover_cmd_work);
+ tsp_debug_err(&info->client->dev,
+ "[cmd is delayed] %d, param = %d, %d\n",
+ __LINE__, buf[17]-'0', buf[19]-'0');
+ info->delayed_cmd_param[0] = buf[17]-'0';
+ if (info->delayed_cmd_param[0] > 1)
+ info->delayed_cmd_param[1] = buf[19]-'0';
+
+ if (info->delayed_cmd_param[0] == 0)
+ schedule_delayed_work(&info->cover_cmd_work,
+ msecs_to_jiffies(300));
+ }
+ }
+
+ /* check lock */
+ mutex_lock(&info->cmd_lock);
+ info->cmd_is_running = true;
+ mutex_unlock(&info->cmd_lock);
+ info->cmd_state = 1;
+ memset(info->cmd_param, 0x00, ARRAY_SIZE(info->cmd_param));
+
+ len = (int)count;
+ if (*(buf + len - 1) == '\n')
+ len--;
+ memset(info->cmd, 0x00, ARRAY_SIZE(info->cmd));
+ memcpy(info->cmd, buf, len);
+ cur = strchr(buf, (int)delim);
+ if (cur)
+ memcpy(buff, buf, cur - buf);
+
+ else
+ memcpy(buff, buf, len);
+ tsp_debug_info(&info->client->dev, "COMMAND : %s\n", buff);
+
+ /* find command */
+ list_for_each_entry(ft_cmd_ptr, &info->cmd_list_head, list) {
+ if (!strncmp(buff, ft_cmd_ptr->cmd_name, CMD_STR_LEN)) {
+ cmd_found = true;
+ break;
+ }
+ }
+
+ /* set not_support_cmd */
+ if (!cmd_found) {
+ list_for_each_entry(ft_cmd_ptr, &info->cmd_list_head, list) {
+ if (!strncmp
+ ("not_support_cmd", ft_cmd_ptr->cmd_name,
+ CMD_STR_LEN))
+ break;
+ }
+ }
+
+ /* parsing parameters */
+ if (cur && cmd_found) {
+ cur++;
+ start = cur;
+ memset(buff, 0x00, ARRAY_SIZE(buff));
+
+ do {
+ if (*cur == delim || cur - buf == len) {
+ end = cur;
+ memcpy(buff, start, end - start);
+ *(buff + strnlen(buff, ARRAY_SIZE(buff))) =
+ '\0';
+ if (kstrtoint
+ (buff, 10,
+ info->cmd_param + param_cnt) < 0)
+ goto err_out;
+ start = cur + 1;
+ memset(buff, 0x00, ARRAY_SIZE(buff));
+ param_cnt++;
+ }
+ cur++;
+ } while (cur - buf <= len);
+ }
+ tsp_debug_info(&info->client->dev, "cmd = %s\n", ft_cmd_ptr->cmd_name);
+ for (i = 0; i < param_cnt; i++)
+ tsp_debug_info(&info->client->dev, "cmd param %d= %d\n", i,
+ info->cmd_param[i]);
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+ if (TRUSTEDUI_MODE_INPUT_SECURED & trustedui_get_current_mode())
+ tui_mode_cmd(info);
+ else
+#endif
+ ft_cmd_ptr->cmd_func(info);
+
+err_out:
+ return count;
+}
+
+static ssize_t show_cmd_status(struct device *dev,
+ struct device_attribute *devattr, char *buf)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+ char buff[16] = {0};
+
+ if (!info) {
+ tsp_debug_err(&info->client->dev,
+ "%s: No platform data found\n", __func__);
+ return -EINVAL;
+ }
+
+ if (!info->input_dev) {
+ tsp_debug_err(&info->client->dev,
+ "%s: No input_dev data found\n", __func__);
+ return -EINVAL;
+ }
+
+ tsp_debug_info(&info->client->dev, "tsp cmd: status:%d\n", info->cmd_state);
+ if (info->cmd_state == CMD_STATUS_WAITING)
+ snprintf(buff, sizeof(buff), "WAITING");
+
+ else if (info->cmd_state == CMD_STATUS_RUNNING)
+ snprintf(buff, sizeof(buff), "RUNNING");
+
+ else if (info->cmd_state == CMD_STATUS_OK)
+ snprintf(buff, sizeof(buff), "OK");
+
+ else if (info->cmd_state == CMD_STATUS_FAIL)
+ snprintf(buff, sizeof(buff), "FAIL");
+
+ else if (info->cmd_state == CMD_STATUS_NOT_APPLICABLE)
+ snprintf(buff, sizeof(buff), "NOT_APPLICABLE");
+ return snprintf(buf, TSP_BUF_SIZE, "%s\n", buff);
+}
+
+static ssize_t show_cmd_result(struct device *dev,
+ struct device_attribute *devattr, char *buf)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ if (!info) {
+ tsp_debug_err(&info->client->dev,
+ "%s: No platform data found\n", __func__);
+ return -EINVAL;
+ }
+
+ if (!info->input_dev) {
+ tsp_debug_err(&info->client->dev,
+ "%s: No input_dev data found\n", __func__);
+ return -EINVAL;
+ }
+
+ tsp_debug_info(&info->client->dev, "tsp cmd: result: %s\n",
+ info->cmd_result);
+ mutex_lock(&info->cmd_lock);
+ info->cmd_is_running = false;
+ mutex_unlock(&info->cmd_lock);
+ info->cmd_state = 0;
+ return snprintf(buf, TSP_BUF_SIZE, "%s\n", info->cmd_result);
+}
+
+static ssize_t cmd_list_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+ int ii = 0;
+ char buffer[info->cmd_buf_size+CMD_STR_LEN];
+ char buffer_name[CMD_STR_LEN];
+
+ snprintf(buffer, CMD_STR_LEN, "++factory command list++\n");
+ while (strncmp(fts_commands[ii].cmd_name, "not_support_cmd", 16) != 0) {
+ snprintf(buffer_name, CMD_STR_LEN,
+ "%s\n", fts_commands[ii].cmd_name);
+ strcat(buffer, buffer_name);
+ ii++;
+ }
+
+ tsp_debug_info(&info->client->dev,
+ "%s: length : %u / %d\n", __func__,
+ (unsigned int)strlen(buffer), info->cmd_buf_size+CMD_STR_LEN);
+
+ return snprintf(buf, TSP_BUF_SIZE, "%s\n", buffer);
+}
+
+static void set_default_result(struct fts_ts_info *info)
+{
+ char delim = ':';
+
+ memset(info->cmd_result, 0x00, ARRAY_SIZE(info->cmd_result));
+ memcpy(info->cmd_result, info->cmd, strnlen(info->cmd, CMD_STR_LEN));
+ strncat(info->cmd_result, &delim, 1);
+}
+
+static void set_cmd_result(struct fts_ts_info *info, char *buff, int len)
+{
+ strncat(info->cmd_result, buff, len);
+}
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+static void tui_mode_cmd(struct fts_ts_info *info)
+{
+ char buff[16] = "TUImode:FAIL";
+
+ set_default_result(info);
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+
+ mutex_lock(&info->cmd_lock);
+ info->cmd_is_running = false;
+ mutex_unlock(&info->cmd_lock);
+
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+#endif
+
+static void not_support_cmd(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[16] = {0};
+
+ set_default_result(info);
+ snprintf(buff, sizeof(buff), "%s", "NA");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+
+ mutex_lock(&info->cmd_lock);
+ info->cmd_is_running = false;
+ mutex_unlock(&info->cmd_lock);
+
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ tsp_debug_info(&info->client->dev, "%s: \"%s\"\n", __func__, buff);
+}
+
+void procedure_cmd_event(struct fts_ts_info *info, unsigned char *data)
+{
+ char buff[16] = {0};
+
+ if ((data[1] == 0x00) && (data[2] == 0x62)) {
+ snprintf(buff, sizeof(buff), "%d",
+ *(unsigned short *)&data[3]);
+ tsp_debug_info(&info->client->dev, "%s: %s\n", "get_threshold", buff);
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_OK;
+
+ } else if ((data[1] == 0x07) && (data[2] == 0xE7)) {
+ if (data[3] <= TSP_FACTEST_RESULT_PASS) {
+ sprintf(buff, "%s",
+ data[3] == TSP_FACTEST_RESULT_PASS ? "PASS" :
+ data[3] == TSP_FACTEST_RESULT_FAIL ? "FAIL" : "NONE");
+ tsp_debug_info(&info->client->dev,
+ "%s: success [%s][%d]", "get_tsp_test_result",
+ data[3] == TSP_FACTEST_RESULT_PASS ? "PASS" :
+ data[3] == TSP_FACTEST_RESULT_FAIL ? "FAIL" :
+ "NONE", data[3]);
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_OK;
+ } else {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_FAIL;
+ tsp_debug_info(&info->client->dev, "%s: %s\n",
+ "get_tsp_test_result",
+ buff);
+ }
+ }
+}
+EXPORT_SYMBOL(procedure_cmd_event);
+
+void fts_print_frame(struct fts_ts_info *info, short *min, short *max)
+{
+ int i = 0;
+ int j = 0;
+ unsigned char *pStr = NULL;
+ unsigned char pTmp[16] = {0};
+
+ pStr = kzalloc(6 * (info->SenseChannelLength + 1), GFP_KERNEL);
+ if (pStr == NULL) {
+ tsp_debug_info(&info->client->dev, "FTS pStr kzalloc failed\n");
+ return;
+ }
+
+ memset(pStr, 0x0, 6 * (info->SenseChannelLength + 1));
+ snprintf(pTmp, sizeof(pTmp), " ");
+ strncat(pStr, pTmp, 6 * info->SenseChannelLength);
+
+ for (i = 0; i < info->SenseChannelLength; i++) {
+ snprintf(pTmp, sizeof(pTmp), "Rx%02d ", i);
+ strncat(pStr, pTmp, 6 * info->SenseChannelLength);
+ }
+
+ tsp_debug_info(&info->client->dev, "FTS %s\n", pStr);
+ memset(pStr, 0x0, 6 * (info->SenseChannelLength + 1));
+ snprintf(pTmp, sizeof(pTmp), " +");
+ strncat(pStr, pTmp, 6 * info->SenseChannelLength);
+
+ for (i = 0; i < info->SenseChannelLength; i++) {
+ snprintf(pTmp, sizeof(pTmp), "------");
+ strncat(pStr, pTmp, 6 * info->SenseChannelLength);
+ }
+
+ tsp_debug_info(&info->client->dev, "FTS %s\n", pStr);
+
+ for (i = 0; i < info->ForceChannelLength; i++) {
+ memset(pStr, 0x0, 6 * (info->SenseChannelLength + 1));
+ snprintf(pTmp, sizeof(pTmp), "Tx%02d | ", i);
+ strncat(pStr, pTmp, 6 * info->SenseChannelLength);
+
+ for (j = 0; j < info->SenseChannelLength; j++) {
+ snprintf(pTmp, sizeof(pTmp), "%5d ",
+ info->pFrame[(i * info->SenseChannelLength) + j]);
+
+ if (i > 0) {
+ if (info->pFrame[(i * info->SenseChannelLength) + j] < *min)
+ *min = info->pFrame[(i * info->SenseChannelLength) + j];
+
+ if (info->pFrame[(i * info->SenseChannelLength) + j] > *max)
+ *max = info->pFrame[(i * info->SenseChannelLength) + j];
+ }
+ strncat(pStr, pTmp, 6 * info->SenseChannelLength);
+ }
+ tsp_debug_info(&info->client->dev, "FTS %s\n", pStr);
+ }
+
+ kfree(pStr);
+}
+
+static int fts_panel_ito_test(struct fts_ts_info *info)
+{
+ unsigned char cmd = READ_ONE_EVENT;
+ unsigned char data[FTS_EVENT_SIZE];
+ unsigned char regAdd[4] = {0xB0, 0x03, 0x60, 0xFB};
+ unsigned char wregAdd[3] = {0xA7, 0x01, 0x00};
+ uint8_t *errortypes[16] = {
+ "F open", "S open", "F2G short", "S2G short", "F2V short",
+ "S2V short", "F2F short", "S2S short", "F2S short",
+ "FPC F open", "FPC S open", "Key F open", "Key S open",
+ "Reserved", "Reserved", "Reserved"};
+ int retry = 0;
+ int result = -1;
+
+ info->fts_systemreset(info);
+ info->fts_wait_for_ready(info);
+ info->fts_irq_enable(info, false);
+ info->fts_interrupt_set(info, INT_DISABLE);
+ info->fts_write_reg(info, ®Add[0], 4);
+ info->fts_command(info, FLUSHBUFFER);
+ info->fts_write_reg(info, &wregAdd[0], 3);
+ fts_delay(200);
+ memset(data, 0x0, FTS_EVENT_SIZE);
+ while (info->fts_read_reg
+ (info, &cmd, 1, (unsigned char *)data, FTS_EVENT_SIZE)) {
+ if ((data[0] == 0x0F) && (data[1] == 0x05)) {
+ switch (data[2]) {
+ case NO_ERROR:
+ if (data[3] == 0x00) {
+ tsp_debug_info(
+ &info->client->dev,
+ "ITO open / short test PASS!!\n");
+ return 1;
+ }
+ break;
+ case ITO_FORCE_OPEN:
+ case ITO_SENSE_OPEN:
+ case ITO_FORCE_SHRT_GND:
+ case ITO_SENSE_SHRT_GND:
+ case ITO_FORCE_SHRT_VCM:
+ case ITO_SENSE_SHRT_VCM:
+ case ITO_FORCE_SHRT_FORCE:
+ case ITO_SENSE_SHRT_SENSE:
+ case ITO_F2E_SENSE:
+ case ITO_FPC_FORCE_OPEN:
+ case ITO_FPC_SENSE_OPEN:
+ case ITO_KEY_FORCE_OPEN:
+ case ITO_KEY_SENSE_OPEN:
+ case ITO_RESERVED0:
+ case ITO_RESERVED1:
+ case ITO_RESERVED2:
+ case ITO_MAX_ERR_REACHED:
+ tsp_debug_info(
+ &info->client->dev,
+ "ITO open / short test FAIL!! Error Type : %s, Channel : %d\n",
+ errortypes[data[2]], data[3]);
+ break;
+ }
+ break;
+ }
+ if (retry++ > 30) {
+ tsp_debug_info(&info->client->dev,
+ "Time over - wait for result of ITO test\n");
+ break;
+ }
+ fts_delay(10);
+ }
+ info->fts_systemreset(info);
+ info->fts_wait_for_ready(info);
+ info->fts_command(info, SENSEON);
+ info->touch_count = 0;
+ info->fts_command(info, FLUSHBUFFER);
+ info->fts_interrupt_set(info, INT_ENABLE);
+ info->fts_irq_enable(info, true);
+ return result;
+}
+int fts_read_frame(struct fts_ts_info *info, unsigned char type, short *min,
+ short *max)
+{
+ unsigned char pFrameAddress[8] = {
+ 0xD0, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00};
+ unsigned int FrameAddress = 0;
+ unsigned int writeAddr = 0;
+ unsigned int start_addr = 0;
+ unsigned int end_addr = 0;
+ unsigned int totalbytes = 0;
+ unsigned int remained = 0;
+ unsigned int readbytes = 0xFF;
+ unsigned int dataposition = 0;
+ unsigned char *pRead = NULL;
+ int rc = 0;
+ int ret = 0;
+ int i = 0;
+
+ tsp_debug_info(&info->client->dev,
+ "===> fts_read_frame digital rev( %d ) sense ( %d ) force (%d)\n",
+ info->digital_rev, info->SenseChannelLength, info->ForceChannelLength);
+
+ pRead = kzalloc(BUFFER_MAX, GFP_KERNEL);
+ if (pRead == NULL) {
+ tsp_debug_info(&info->client->dev,
+ "FTS pRead kzalloc failed\n");
+ rc = 1;
+ goto ErrorExit;
+ }
+
+ pFrameAddress[2] = type;
+ totalbytes = info->SenseChannelLength * info->ForceChannelLength * 2;
+ ret = info->fts_read_reg(info, &pFrameAddress[0], 3, pRead, pFrameAddress[3]);
+
+ if (ret > 0) {
+ if (info->digital_rev == FTS_DIGITAL_REV_1)
+ FrameAddress = pRead[0] + (pRead[1] << 8);
+ else if (info->digital_rev == FTS_DIGITAL_REV_2)
+ FrameAddress = pRead[1] + (pRead[2] << 8);
+
+ start_addr = FrameAddress+info->SenseChannelLength * 2;
+ end_addr = start_addr + totalbytes;
+ } else {
+ tsp_debug_info(&info->client->dev,
+ "FTS read failed rc = %d\n", ret);
+ rc = 2;
+ goto ErrorExit;
+ }
+
+#ifdef DEBUG_MSG
+ tsp_debug_info(&info->client->dev,
+ "FTS FrameAddress = %X\n", FrameAddress);
+ tsp_debug_info(&info->client->dev,
+ "FTS start_addr = %X, end_addr = %X\n",
+ start_addr, end_addr);
+#endif
+
+ remained = totalbytes;
+ for (writeAddr = start_addr; writeAddr < end_addr;
+ writeAddr += READ_CHUNK_SIZE) {
+ pFrameAddress[1] = (writeAddr >> 8) & 0xFF;
+ pFrameAddress[2] = writeAddr & 0xFF;
+
+ if (remained >= READ_CHUNK_SIZE)
+ readbytes = READ_CHUNK_SIZE;
+ else
+ readbytes = remained;
+
+ memset(pRead, 0x0, readbytes);
+
+#ifdef DEBUG_MSG
+ tsp_debug_info(&info->client->dev,
+ "FTS %02X%02X%02X readbytes=%d\n",
+ pFrameAddress[0], pFrameAddress[1],
+ pFrameAddress[2], readbytes);
+
+#endif
+ if (info->digital_rev == FTS_DIGITAL_REV_1) {
+ info->fts_read_reg(info, &pFrameAddress[0],
+ 3, pRead, readbytes);
+ remained -= readbytes;
+
+ for (i = 0; i < readbytes; i += 2) {
+ info->pFrame[dataposition++] =
+ pRead[i] + (pRead[i + 1] << 8);
+ }
+ } else if (info->digital_rev == FTS_DIGITAL_REV_2) {
+ info->fts_read_reg(info, &pFrameAddress[0],
+ 3, pRead, readbytes + 1);
+ remained -= readbytes;
+
+ for (i = 1; i < (readbytes+1); i += 2) {
+ info->pFrame[dataposition++] =
+ pRead[i] + (pRead[i + 1] << 8);
+ }
+ }
+ }
+ kfree(pRead);
+
+#ifdef DEBUG_MSG
+ tsp_debug_info(&info->client->dev,
+ "FTS writeAddr = %X, start_addr = %X, end_addr = %X\n",
+ writeAddr, start_addr, end_addr);
+#endif
+
+ switch (type) {
+ case TYPE_RAW_DATA:
+ tsp_debug_info(&info->client->dev,
+ "FTS [Raw Data : 0x%X%X]\n", pFrameAddress[0],
+ FrameAddress);
+ break;
+ case TYPE_FILTERED_DATA:
+ tsp_debug_info(&info->client->dev,
+ "FTS [Filtered Data : 0x%X%X]\n",
+ pFrameAddress[0], FrameAddress);
+ break;
+ case TYPE_STRENGTH_DATA:
+ tsp_debug_info(&info->client->dev,
+ "FTS [Strength Data : 0x%X%X]\n",
+ pFrameAddress[0], FrameAddress);
+ break;
+ case TYPE_BASELINE_DATA:
+ tsp_debug_info(&info->client->dev,
+ "FTS [Baseline Data : 0x%X%X]\n",
+ pFrameAddress[0], FrameAddress);
+ break;
+ }
+ fts_print_frame(info, min, max);
+
+ErrorExit:
+ return rc;
+}
+
+static void get_fw_ver_bin(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[16] = {0};
+
+ set_default_result(info);
+
+ if (strncmp(info->board->model_name, "G925", 4) == 0) {
+ info->tspid_val = gpio_get_value(info->board->tspid);
+ info->tspid2_val = gpio_get_value(info->board->tspid2);
+
+ sprintf(buff, "ST%01X%01X%04X",
+ info->tspid_val, info->tspid2_val,
+ info->fw_main_version_of_bin);
+ } else {
+ sprintf(buff, "ST%02X%04X",
+ info->panel_revision,
+ info->fw_main_version_of_bin);
+ }
+
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_OK;
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_fw_ver_ic(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[16] = {0};
+
+ set_default_result(info);
+
+ if (strncmp(info->board->model_name, "G925", 4) == 0) {
+ info->tspid_val = gpio_get_value(info->board->tspid);
+ info->tspid2_val = gpio_get_value(info->board->tspid2);
+
+ sprintf(buff, "ST%01X%01X%04X",
+ info->tspid_val, info->tspid2_val,
+ info->fw_main_version_of_ic);
+ } else {
+ sprintf(buff, "ST%02X%04X",
+ info->panel_revision,
+ info->fw_main_version_of_ic);
+ }
+
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_OK;
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_config_ver(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[20] = {0};
+ const char *name = NULL;
+
+ if (info->board->model_name)
+ name = info->board->model_name;
+ else if (info->board->project_name)
+ name = info->board->project_name;
+
+ snprintf(buff, sizeof(buff), "%s_ST_%04X",
+ name? name : STM_DEVICE_NAME,
+ info->config_version_of_ic);
+
+ set_default_result(info);
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_OK;
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_threshold(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ unsigned char cmd[4] = {0xB2, 0x00, 0x62, 0x02};
+ char buff[CMD_STR_LEN] = {0};
+ int timeout = 0;
+
+ set_default_result(info);
+
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ info->fts_write_reg(info, &cmd[0], 4);
+ info->cmd_state = CMD_STATUS_RUNNING;
+
+ while (info->cmd_state == CMD_STATUS_RUNNING) {
+ if (timeout++ > 30) {
+ info->cmd_state = CMD_STATUS_FAIL;
+ break;
+ }
+ fts_delay(10);
+ }
+}
+
+static void get_x_num(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[16] = { 0 };
+
+ set_default_result(info);
+ snprintf(buff, sizeof(buff), "%d", info->SenseChannelLength);
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = 2;
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_y_num(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[16] = { 0 };
+
+ set_default_result(info);
+ snprintf(buff, sizeof(buff), "%d", info->ForceChannelLength);
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_OK;
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void run_rawcap_read(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+ short min = 0x7FFF;
+ short max = 0x8000;
+
+ set_default_result(info);
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ fts_read_frame(info, TYPE_FILTERED_DATA, &min, &max);
+ snprintf(buff, sizeof(buff), "%d,%d", min, max);
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_OK;
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+
+}
+
+static void get_rawcap(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+ short val = 0;
+ int node = 0;
+
+ set_default_result(info);
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ node = fts_check_index(info);
+ if (node < 0)
+ return;
+
+ val = info->pFrame[node];
+ snprintf(buff, sizeof(buff), "%d", val);
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_OK;
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_strength_all_data(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+ short min = 0x7FFF;
+ short max = 0x8000;
+ const int str_size = info->ForceChannelLength *
+ info->SenseChannelLength * 5;
+ char all_strbuff[str_size];
+ int i, j;
+
+ memset(all_strbuff, 0, sizeof(char)*(str_size)); /* size 5 ex(1125,) */
+
+ set_default_result(info);
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ fts_read_frame(info, TYPE_STRENGTH_DATA, &min, &max);
+
+
+ for (i = 0; i < info->ForceChannelLength; i++) {
+ for (j = 0; j < info->SenseChannelLength; j++) {
+ sprintf(buff, "%d,",
+ info->pFrame[
+ (i * info->SenseChannelLength) + j]);
+ strcat(all_strbuff, buff);
+ }
+ }
+
+ info->cmd_state = CMD_STATUS_OK;
+
+ set_cmd_result(info, all_strbuff,
+ strnlen(all_strbuff, sizeof(all_strbuff)));
+ tsp_debug_info(&info->client->dev,
+ "%ld (%ld)\n", strnlen(all_strbuff,
+ sizeof(all_strbuff)), sizeof(all_strbuff));
+}
+
+void fts_read_self_frame(struct fts_ts_info *info, unsigned short oAddr)
+{
+ char buff[66] = {0, };
+ short *data = 0;
+ char temp[9] = {0, };
+ char temp2[512] = {0, };
+ int i = 0;
+ int rc = 0;
+ unsigned char regAdd[6] = {0xD0, 0x00, 0x00, 0xD0, 0x00, 0x00};
+
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ regAdd[1] = (oAddr >> 8) & 0xff;
+ regAdd[2] = oAddr & 0xff;
+ rc = info->fts_read_reg(info, ®Add[0], 3, (unsigned char *)&buff[0], 5);
+ if (rc < 0) {
+ info->cmd_state = CMD_STATUS_FAIL;
+ return;
+ }
+
+ if (info->digital_rev == FTS_DIGITAL_REV_1) {
+ tsp_debug_info(&info->client->dev,
+ "%s: Force Address : %02x%02x\n",
+ __func__, buff[1], buff[0]);
+ tsp_debug_info(&info->client->dev,
+ "%s: Sense Address : %02x%02x\n",
+ __func__, buff[3], buff[2]);
+ regAdd[1] = buff[3];
+ regAdd[2] = buff[2];
+ regAdd[4] = buff[1];
+ regAdd[5] = buff[0];
+ } else if (info->digital_rev == FTS_DIGITAL_REV_2) {
+ tsp_debug_info(&info->client->dev,
+ "%s: Force Address : %02x%02x\n",
+ __func__, buff[2], buff[1]);
+ tsp_debug_info(&info->client->dev,
+ "%s: Sense Address : %02x%02x\n",
+ __func__, buff[4], buff[3]);
+ regAdd[1] = buff[4];
+ regAdd[2] = buff[3];
+ regAdd[4] = buff[2];
+ regAdd[5] = buff[1];
+ }
+
+ rc = info->fts_read_reg(info, ®Add[0], 3,
+ (unsigned char *)&buff[0],
+ info->SenseChannelLength * 2 + 1);
+ if (rc < 0) {
+ info->cmd_state = CMD_STATUS_FAIL;
+ return;
+ }
+
+ if (info->digital_rev == FTS_DIGITAL_REV_1)
+ data = (short *)&buff[0];
+ else
+ data = (short *)&buff[1];
+
+ memset(temp, 0x00, ARRAY_SIZE(temp));
+ memset(temp2, 0x00, ARRAY_SIZE(temp2));
+
+ for (i = 0; i < info->SenseChannelLength; i++) {
+ tsp_debug_info(&info->client->dev,
+ "%s: Rx [%d] = %d\n", __func__,
+ i,
+ *data);
+ sprintf(temp, "%d,", *data);
+ strncat(temp2, temp, 9);
+ data++;
+ }
+
+ rc = info->fts_read_reg(info, ®Add[3], 3,
+ (unsigned char *)&buff[0],
+ info->ForceChannelLength * 2 + 1);
+ if (rc < 0) {
+ info->cmd_state = CMD_STATUS_FAIL;
+ return;
+ }
+
+ if (info->digital_rev == FTS_DIGITAL_REV_1)
+ data = (short *)&buff[0];
+ else
+ data = (short *)&buff[1];
+
+ for (i = 0; i < info->ForceChannelLength; i++) {
+ tsp_debug_info(&info->client->dev,
+ "%s: Tx [%d] = %d\n", __func__, i, *data);
+ sprintf(temp, "%d,", *data);
+ strncat(temp2, temp, 9);
+ data++;
+ }
+
+ set_cmd_result(info, temp2, strnlen(temp2, sizeof(temp2)));
+
+ info->cmd_state = CMD_STATUS_OK;
+}
+
+static void fts_read_ix_data(struct fts_ts_info *info, bool allnode)
+{
+ char buff[33] = { 0 };
+
+ unsigned short max_tx_ix_sum = 0;
+ unsigned short min_tx_ix_sum = 0xFFFF;
+
+ unsigned short max_rx_ix_sum = 0;
+ unsigned short min_rx_ix_sum = 0xFFFF;
+
+ unsigned char tx_ix2[info->ForceChannelLength + 4];
+ unsigned char rx_ix2[info->SenseChannelLength + 4];
+
+ unsigned char regAdd[FTS_EVENT_SIZE];
+ unsigned short tx_ix1 = 0, rx_ix1 = 0;
+
+ unsigned short force_ix_data[info->ForceChannelLength * 2 + 1];
+ unsigned short sense_ix_data[info->SenseChannelLength * 2 + 1];
+ int buff_size, j;
+ char *mbuff = NULL;
+ int num, n, a, fzero;
+ char cnum;
+ int i = 0;
+ int comp_header_addr, comp_start_tx_addr, comp_start_rx_addr;
+ unsigned int rx_num, tx_num;
+
+ set_default_result(info);
+
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ info->fts_irq_enable(info, false);
+ info->fts_interrupt_set(info, INT_DISABLE);
+
+ info->fts_command(info, SENSEOFF);
+
+ info->fts_command(info, FLUSHBUFFER); /* Clear FIFO */
+ fts_delay(50);
+
+ info->fts_release_all_finger(info);
+
+ /* Request compensation data */
+ regAdd[0] = 0xB8;
+ regAdd[1] = 0x20; /* SELF IX */
+ regAdd[2] = 0x00;
+ info->fts_write_reg(info, ®Add[0], 3);
+ fts_fw_wait_for_specific_event(info,
+ EVENTID_STATUS_REQUEST_COMP, 0x20, 0x00);
+
+ /* Read an address of compensation data */
+ regAdd[0] = 0xD0;
+ regAdd[1] = 0x00;
+ regAdd[2] = FTS_SI_COMPENSATION_OFFSET_ADDR;
+ info->fts_read_reg(info, regAdd, 3, &buff[0], 3);
+ comp_header_addr = buff[1] + (buff[2] << 8);
+
+ /* Read header of compensation area */
+ regAdd[0] = 0xD0;
+ regAdd[1] = (comp_header_addr >> 8) & 0xFF;
+ regAdd[2] = comp_header_addr & 0xFF;
+ info->fts_read_reg(info, regAdd, 3, &buff[0], 16 + 1);
+ tx_num = buff[5];
+ rx_num = buff[6];
+ tsp_debug_info(&info->client->dev,
+ "%s: [FTS] tx : %d, rx : %d",
+ __func__, tx_num, rx_num);
+
+ tx_ix1 = (short) buff[10];
+ rx_ix1 = (short) buff[11];
+
+ comp_start_tx_addr = comp_header_addr + 0x10;
+ comp_start_rx_addr = comp_start_tx_addr + tx_num;
+
+ memset(tx_ix2, 0x0, tx_num);
+ memset(rx_ix2, 0x0, rx_num);
+
+ /* Read Self TX Ix2 */
+ regAdd[0] = 0xD0;
+ regAdd[1] = (comp_start_tx_addr >> 8) & 0xFF;
+ regAdd[2] = comp_start_tx_addr & 0xFF;
+ info->fts_read_reg(info, regAdd, 3, &tx_ix2[0], tx_num + 1);
+
+ /* Read Self RX Ix2 */
+ regAdd[0] = 0xD0;
+ regAdd[1] = (comp_start_rx_addr >> 8) & 0xFF;
+ regAdd[2] = comp_start_rx_addr & 0xFF;
+ info->fts_read_reg(info, regAdd, 3, &rx_ix2[0], rx_num + 1);
+
+ for (i = 0; i < info->ForceChannelLength; i++) {
+ force_ix_data[i] = tx_ix1 + tx_ix2[i + 1];
+ if (max_tx_ix_sum < force_ix_data[i])
+ max_tx_ix_sum = force_ix_data[i];
+ if (min_tx_ix_sum > force_ix_data[i])
+ min_tx_ix_sum = force_ix_data[i];
+ }
+
+ for (i = 0; i < info->SenseChannelLength; i++) {
+ sense_ix_data[i] = rx_ix1 + rx_ix2[i + 1];
+ if (max_rx_ix_sum < sense_ix_data[i])
+ max_rx_ix_sum = sense_ix_data[i];
+ if (min_rx_ix_sum > sense_ix_data[i])
+ min_rx_ix_sum = sense_ix_data[i];
+ }
+
+ tsp_debug_info(&info->client->dev,
+ "%s: MIN_TX_IX_SUM : %d MAX_TX_IX_SUM : %d\n",
+ __func__, min_tx_ix_sum, max_tx_ix_sum);
+ tsp_debug_info(&info->client->dev,
+ "%s: MIN_RX_IX_SUM : %d MAX_RX_IX_SUM : %d\n",
+ __func__, min_rx_ix_sum, max_rx_ix_sum);
+
+ info->fts_command(info, SENSEON);
+
+ info->fts_irq_enable(info, true);
+ info->fts_interrupt_set(info, INT_ENABLE);
+
+ if (allnode == true) {
+ buff_size = (info->ForceChannelLength +
+ info->SenseChannelLength + 2) * 5;
+ mbuff = kzalloc(buff_size, GFP_KERNEL);
+ }
+ if (mbuff != NULL) {
+ char *pBuf = mbuff;
+
+ for (i = 0; i < info->ForceChannelLength; i++) {
+ num = force_ix_data[i];
+ n = 100000;
+ fzero = 0;
+ for (j = 5; j > 0; j--) {
+ n = n / 10;
+ a = num / n;
+ if (a)
+ fzero = 1;
+ cnum = a + '0';
+ num = num - a*n;
+ if (fzero)
+ *pBuf++ = cnum;
+ }
+ if (!fzero)
+ *pBuf++ = '0';
+ *pBuf++ = ',';
+ tsp_debug_info(&info->client->dev,
+ "Force[%d] %d\n", i, force_ix_data[i]);
+ }
+ for (i = 0; i < info->SenseChannelLength; i++) {
+ num = sense_ix_data[i];
+ n = 100000;
+ fzero = 0;
+ for (j = 5; j > 0; j--) {
+ n = n / 10;
+ a = num / n;
+ if (a)
+ fzero = 1;
+ cnum = a + '0';
+ num = num - a * n;
+ if (fzero)
+ *pBuf++ = cnum;
+ }
+ if (!fzero)
+ *pBuf++ = '0';
+ if (i < (info->SenseChannelLength - 1))
+ *pBuf++ = ',';
+ tsp_debug_info(&info->client->dev,
+ "Sense[%d] %d\n", i, sense_ix_data[i]);
+ }
+
+ set_cmd_result(info, mbuff, buff_size);
+ info->cmd_state = CMD_STATUS_OK;
+ kfree(mbuff);
+ } else {
+ if (allnode == true) {
+ snprintf(buff, sizeof(buff), "%s", "kzalloc failed");
+ info->cmd_state = CMD_STATUS_FAIL;
+ } else{
+ snprintf(buff, sizeof(buff), "%d,%d,%d,%d",
+ min_tx_ix_sum, max_tx_ix_sum,
+ min_rx_ix_sum, max_rx_ix_sum);
+ info->cmd_state = CMD_STATUS_OK;
+ }
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ tsp_debug_info(&info->client->dev,
+ "%s: %s\n", __func__, buff);
+ }
+}
+
+static void run_ix_data_read(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+
+ set_default_result(info);
+ fts_read_ix_data(info, false);
+}
+
+static void run_ix_data_read_all(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+
+ set_default_result(info);
+ fts_read_ix_data(info, true);
+}
+
+static void fts_read_self_raw_frame(struct fts_ts_info *info,
+ unsigned short oAddr, bool allnode)
+{
+ char buff[32 * 2 + 1] = { 0 };
+ unsigned char D0_offset = 1;
+ unsigned char regAdd[3] = {0xD0, 0x00, 0x00};
+ unsigned char ReadData[info->SenseChannelLength * 2 + 1];
+ unsigned short self_force_raw_data[info->ForceChannelLength * 2 + 1];
+ unsigned short self_sense_raw_data[info->SenseChannelLength * 2 + 1];
+ unsigned int FrameAddress = 0;
+ unsigned char count = 0;
+ int buff_size, i, j;
+ char *mbuff = NULL;
+ int num, n, a, fzero;
+ char cnum;
+ unsigned short min_tx_self_raw_data = 0xFFFF;
+ unsigned short max_tx_self_raw_data = 0;
+ unsigned short min_rx_self_raw_data = 0xFFFF;
+ unsigned short max_rx_self_raw_data = 0;
+
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ info->fts_irq_enable(info, false);
+ info->fts_interrupt_set(info, INT_DISABLE);
+ info->fts_command(info, SENSEOFF);
+
+ fts_delay(50);
+
+ info->fts_command(info, FLUSHBUFFER); /* Clear FIFO */
+ fts_delay(50);
+
+ regAdd[1] = 0x00;
+ regAdd[2] = oAddr;
+ info->fts_read_reg(info, regAdd, 3, &ReadData[0], 4);
+
+ /* D1 : DOFFSET = 0, D2 : DOFFSET : 1 */
+ FrameAddress = ReadData[D0_offset] + (ReadData[D0_offset + 1] << 8);
+
+ regAdd[1] = (FrameAddress >> 8) & 0xFF;
+ regAdd[2] = FrameAddress & 0xFF;
+
+ info->fts_read_reg(info, regAdd, 3, &ReadData[0], info->ForceChannelLength * 2 + 1);
+
+ for (count = 0; count < info->ForceChannelLength; count++) {
+ self_force_raw_data[count] = ReadData[count*2+D0_offset] + (ReadData[count*2+D0_offset+1]<<8);
+
+ if (max_tx_self_raw_data < self_force_raw_data[count])
+ max_tx_self_raw_data = self_force_raw_data[count];
+ if (min_tx_self_raw_data > self_force_raw_data[count])
+ min_tx_self_raw_data = self_force_raw_data[count];
+ }
+
+ regAdd[1] = 0x00;
+ regAdd[2] = oAddr + 2;
+ info->fts_read_reg(info, regAdd, 3, &ReadData[0], 4);
+
+ FrameAddress = ReadData[D0_offset] + (ReadData[D0_offset + 1] << 8); /* D1 : DOFFSET = 0, D2 : DOFFSET : 1 */
+
+ regAdd[1] = (FrameAddress >> 8) & 0xFF;
+ regAdd[2] = FrameAddress & 0xFF;
+
+ info->fts_read_reg(info, regAdd, 3, &ReadData[0],
+ info->SenseChannelLength * 2 + 1);
+
+ for (count = 0; count < info->SenseChannelLength; count++) {
+ self_sense_raw_data[count] = ReadData[count*2+D0_offset] +
+ (ReadData[count*2+D0_offset+1]<<8);
+
+ if (max_rx_self_raw_data < self_sense_raw_data[count])
+ max_rx_self_raw_data = self_sense_raw_data[count];
+ if (min_rx_self_raw_data > self_sense_raw_data[count])
+ min_rx_self_raw_data = self_sense_raw_data[count];
+ }
+
+ tsp_debug_info(&info->client->dev,
+ "%s MIN_TX_SELF_RAW: %d MAX_TX_SELF_RAW : %d\n",
+ __func__, min_tx_self_raw_data, max_tx_self_raw_data);
+ tsp_debug_info(&info->client->dev,
+ "%s MIN_RX_SELF_RAW : %d MIN_RX_SELF_RAW : %d\n",
+ __func__, min_rx_self_raw_data, max_rx_self_raw_data);
+
+ fts_delay(1);
+ info->fts_command(info, SENSEON);
+
+ info->fts_irq_enable(info, true);
+ info->fts_interrupt_set(info, INT_ENABLE);
+
+ if (allnode == true) {
+ buff_size = (info->ForceChannelLength +
+ info->SenseChannelLength + 2)*10;
+ mbuff = kzalloc(buff_size, GFP_KERNEL);
+ }
+ if (mbuff != NULL) {
+ char *pBuf = mbuff;
+
+ for (i = 0; i < info->ForceChannelLength; i++) {
+ num = self_force_raw_data[i];
+ n = 100000;
+ fzero = 0;
+ for (j = 5; j > 0; j--) {
+ n = n/10;
+ a = num/n;
+ if (a)
+ fzero = 1;
+ cnum = a + '0';
+ num = num - a*n;
+ if (fzero)
+ *pBuf++ = cnum;
+ }
+ if (!fzero)
+ *pBuf++ = '0';
+ *pBuf++ = ',';
+ tsp_debug_info(&info->client->dev, "%d ", self_force_raw_data[i]);
+ }
+ for (i = 0; i < info->SenseChannelLength; i++) {
+ num = self_sense_raw_data[i];
+ n = 100000;
+ fzero = 0;
+ for (j = 5; j > 0; j--) {
+ n = n/10;
+ a = num/n;
+ if (a)
+ fzero = 1;
+ cnum = a + '0';
+ num = num - a*n;
+ if (fzero)
+ *pBuf++ = cnum;
+ }
+ if (!fzero)
+ *pBuf++ = '0';
+ if (i < (info->SenseChannelLength-1))
+ *pBuf++ = ',';
+ tsp_debug_info(&info->client->dev, "%d ", self_sense_raw_data[i]);
+ }
+
+
+ set_cmd_result(info, mbuff, buff_size);
+ info->cmd_state = CMD_STATUS_OK;
+ kfree(mbuff);
+ } else {
+ if (allnode == true) {
+ snprintf(buff, sizeof(buff), "%s", "kzalloc failed");
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ } else{
+ snprintf(buff, sizeof(buff), "%d,%d,%d,%d",
+ min_tx_self_raw_data,
+ max_tx_self_raw_data,
+ min_rx_self_raw_data,
+ max_rx_self_raw_data);
+ info->cmd_state = CMD_STATUS_OK;
+ }
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ tsp_debug_info(&info->client->dev, "%s: %s\n",
+ __func__, buff);
+ }
+}
+
+static void run_self_raw_read(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+
+ set_default_result(info);
+ fts_read_self_raw_frame(info, FTS_WATER_SELF_RAW_ADDR, false);
+}
+
+static void run_self_raw_read_all(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+
+ set_default_result(info);
+ fts_read_self_raw_frame(info, FTS_WATER_SELF_RAW_ADDR, true);
+}
+
+static void get_cx_data(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+ short val = 0;
+ int node = 0;
+
+ set_default_result(info);
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ node = fts_check_index(info);
+ if (node < 0)
+ return;
+
+ val = info->cx_data[node];
+ snprintf(buff, sizeof(buff), "%d", val);
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_OK;
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+
+}
+
+static void run_cx_data_read(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ unsigned char ReadData[info->ForceChannelLength]
+ [info->SenseChannelLength + FTS_CX2_READ_LENGTH];
+ int cxdiffData_rx[info->ForceChannelLength *
+ (info->SenseChannelLength + FTS_CX2_READ_LENGTH)];
+ int cxdiffData_tx[info->ForceChannelLength *
+ (info->SenseChannelLength + FTS_CX2_READ_LENGTH)];
+ int Max_cxdiffData_rx = 0;
+ int Low_cxdiffData_rx = 0;
+ int Max_cxdiffData_tx = 0;
+ int Low_cxdiffData_tx = 0;
+ unsigned char regAdd[8];
+ unsigned int addr, rx_num, tx_num;
+ int i, j;
+ unsigned char *pStr = NULL;
+ unsigned char pTmp[16] = { 0 };
+
+ int comp_header_addr, comp_start_addr;
+
+ set_default_result(info);
+
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ pStr = kzalloc(4 * (info->SenseChannelLength + 1), GFP_KERNEL);
+ if (pStr == NULL) {
+ tsp_debug_info(&info->client->dev,
+ "%s: pStr kzalloc failed\n", __func__);
+ return;
+ }
+
+ tsp_debug_info(&info->client->dev, "%s: start\n", __func__);
+
+ info->fts_irq_enable(info, false);
+ tsp_debug_info(&info->client->dev, "%s: disable_irq\n", __func__);
+ info->fts_interrupt_set(info, INT_DISABLE);
+ tsp_debug_info(&info->client->dev,
+ "%s: fts_interrupt_set\n", __func__);
+ info->fts_command(info, SENSEOFF);
+ tsp_debug_info(&info->client->dev, "%s: fts_command\n", __func__);
+ fts_delay(50);
+ tsp_debug_info(&info->client->dev, "%s: senseoff\n", __func__);
+
+ info->fts_command(info, FLUSHBUFFER);
+ fts_delay(50);
+
+ info->fts_release_all_finger(info);
+
+ /* Request compensation data */
+ regAdd[0] = 0xB8;
+ regAdd[1] = 0x04; /* MUTUAL CX (LPA) */
+ regAdd[2] = 0x00;
+ info->fts_write_reg(info, ®Add[0], 3);
+ tsp_debug_info(&info->client->dev,
+ "%s: Writing Request compensation data\n", __func__);
+ fts_cmd_completion_check(info, EVENTID_STATUS_REQUEST_COMP,
+ regAdd[1], regAdd[2]);
+
+ /* Read an address of compensation data */
+ regAdd[0] = 0xD0;
+ regAdd[1] = 0x00;
+ regAdd[2] = 0x50;
+ info->fts_read_reg(info, regAdd, 3, &buff[0], 4);
+ comp_header_addr = buff[1] + (buff[2] << 8);
+ tsp_debug_info(&info->client->dev,
+ "%s:Read an address of compensation data\n", __func__);
+
+ /* Read header of compensation area */
+ regAdd[0] = 0xD0;
+ regAdd[1] = (comp_header_addr >> 8) & 0xFF;
+ regAdd[2] = comp_header_addr & 0xFF;
+ info->fts_read_reg(info, regAdd, 3, &buff[0], 16 + 1);
+ tx_num = buff[5];
+ rx_num = buff[6];
+ comp_start_addr = comp_header_addr + 0x10;
+
+ tsp_debug_info(&info->client->dev,
+ "%s:Read header of compensation area data\n", __func__);
+ tsp_debug_info(&info->client->dev,
+ "%s: Tx num ( %d ) Rx num ( %d )\n", __func__,
+ tx_num, rx_num);
+ tsp_debug_info(&info->client->dev,
+ "%s: comp_stat_addr : 0x%x\n", __func__,
+ comp_start_addr);
+ /* Read compensation data */
+ for (j = 0; j < tx_num; j++) {
+ memset(&ReadData[j], 0x0, rx_num);
+ memset(pStr, 0x0, 4 * (rx_num + 1));
+ snprintf(pTmp, sizeof(pTmp), "Tx%02d | ", j);
+ strncat(pStr, pTmp, 4 * rx_num);
+
+ addr = comp_start_addr + (rx_num * j);
+ regAdd[0] = 0xD0;
+ regAdd[1] = (addr >> 8) & 0xFF;
+ regAdd[2] = addr & 0xFF;
+ info->fts_read_reg(info, regAdd, 3, &ReadData[j][0], rx_num + 1);
+ for (i = 1; i < rx_num + 1; i++) {
+ snprintf(pTmp, sizeof(pTmp), "%3d", ReadData[j][i]);
+ strncat(pStr, pTmp, 4 * rx_num);
+ }
+ tsp_debug_info(&info->client->dev, "%s\n", pStr);
+ }
+
+ tsp_debug_info(&info->client->dev,
+ "%s:Read compensation data\n", __func__);
+ if (info->cx_data) {
+ for (j = 0; j < tx_num; j++) {
+ for (i = 1; i < rx_num + 1; i++)
+ info->cx_data[(j * rx_num) + i - 1] =
+ ReadData[j][i];
+ }
+ }
+ tsp_debug_err(&info->client->dev,
+ "===================> %s : Rx diff\n", __func__);
+ for (j = 0; j < tx_num; j++) {
+ memset(pStr, 0x0, 4 * (rx_num + 1));
+ snprintf(pTmp, sizeof(pTmp), "Tx%02d | ", j);
+ strncat(pStr, pTmp, 16);
+ for (i = 0; i < (rx_num-1); i++) {
+ cxdiffData_rx[(j*rx_num)+i] =
+ info->cx_data[(j*rx_num)+i] -
+ info->cx_data[(j*rx_num)+i+1];
+ if ((j == 0) && (i == 0)) {
+ Max_cxdiffData_rx = cxdiffData_rx[(j*rx_num)+i];
+ Low_cxdiffData_rx = cxdiffData_rx[(j*rx_num)+i];
+ }
+ if (cxdiffData_rx[(j*rx_num)+i] > Max_cxdiffData_rx)
+ Max_cxdiffData_rx = cxdiffData_rx[(j*rx_num)+i];
+ if (cxdiffData_rx[(j*rx_num)+i] < Low_cxdiffData_rx)
+ Low_cxdiffData_rx = cxdiffData_rx[(j*rx_num)+i];
+ snprintf(pTmp, sizeof(pTmp), "%4d", cxdiffData_rx[(j*rx_num)+i]);
+ strcat(pStr, pTmp);
+ }
+ tsp_debug_info(&info->client->dev, "FTS %s\n", pStr);
+ }
+ tsp_debug_err(&info->client->dev,
+ "======> Max : %d / Low : %d <=========\n",
+ Max_cxdiffData_rx, Low_cxdiffData_rx);
+ tsp_debug_err(&info->client->dev,
+ " %s : Tx diff\n <=======================", __func__);
+ for (j = 0; j < (tx_num - 1); j++) {
+ memset(pStr, 0x0, 4 * (rx_num + 1));
+ snprintf(pTmp, sizeof(pTmp), "Tx%02d | ", j);
+ strncat(pStr, pTmp, 16);
+ for (i = 0; i < rx_num; i++) {
+ cxdiffData_tx[(j*rx_num)+i] =
+ info->cx_data[(j*rx_num)+i] -
+ info->cx_data[((j+1)*rx_num)+i];
+ if ((j == 0) && (i == 0)) {
+ Max_cxdiffData_tx = cxdiffData_rx[(j*rx_num)+i];
+ Low_cxdiffData_tx = cxdiffData_rx[(j*rx_num)+i];
+ }
+ if (cxdiffData_tx[(j*rx_num)+i] > Max_cxdiffData_tx)
+ Max_cxdiffData_tx = cxdiffData_tx[(j*rx_num)+i];
+ if (cxdiffData_tx[(j*rx_num)+i] < Low_cxdiffData_tx)
+ Low_cxdiffData_tx = cxdiffData_tx[(j*rx_num)+i];
+ snprintf(pTmp, sizeof(pTmp), "%4d",
+ cxdiffData_tx[(j*rx_num)+i]);
+ strcat(pStr, pTmp);
+ }
+ tsp_debug_info(&info->client->dev, "FTS %s\n", pStr);
+ }
+ tsp_debug_err(&info->client->dev,
+ "======> Max : %d / Low : %d <=========\n",
+ Max_cxdiffData_tx, Low_cxdiffData_tx);
+
+ kfree(pStr);
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ info->fts_irq_enable(info, true);
+ info->fts_interrupt_set(info, INT_ENABLE);
+ info->fts_command(info, SENSEON);
+ info->cmd_state = CMD_STATUS_OK;
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+
+}
+
+static void get_cx_all_data(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[SEC_CMD_STR_LEN] = { 0 };
+ unsigned char ReadData[info->ForceChannelLength]
+ [info->SenseChannelLength + FTS_CX2_READ_LENGTH];
+ unsigned char regAdd[8];
+ unsigned int addr, rx_num, tx_num;
+ int i, j;
+ unsigned char *pStr = NULL;
+ unsigned char pTmp[16] = { 0 };
+ char all_strbuff[(info->ForceChannelLength)*
+ (info->SenseChannelLength)*3];
+
+ int comp_header_addr, comp_start_addr;
+
+ set_default_result(info);
+
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ tsp_debug_info(&info->client->dev, "%s: start\n", __func__);
+
+ info->fts_command(info, SENSEOFF);
+ info->fts_irq_enable(info, false);
+ info->fts_command(info, FLUSHBUFFER);
+ fts_delay(50);
+
+ info->fts_release_all_finger(info);
+
+ tx_num = info->ForceChannelLength;
+ rx_num = info->SenseChannelLength;
+
+ pStr = kzalloc(4 * (info->SenseChannelLength + 1), GFP_KERNEL);
+ if (pStr == NULL) {
+ tsp_debug_info(&info->client->dev,
+ "%s: pStr kzalloc failed\n", __func__);
+ goto out;
+ }
+
+ /* size 3 ex(45,) */
+ memset(all_strbuff, 0, sizeof(char) * (tx_num*rx_num*3));
+
+ /* Request compensation data */
+ regAdd[0] = 0xB8;
+ regAdd[1] = 0x04; /* MUTUAL CX (LPA) */
+ regAdd[2] = 0x00;
+ info->fts_write_reg(info, ®Add[0], 3);
+ fts_fw_wait_for_specific_event(info, EVENTID_STATUS_REQUEST_COMP, 0x04, 0x00);
+
+ /* Read an address of compensation data */
+ regAdd[0] = 0xD0;
+ regAdd[1] = 0x00;
+ regAdd[2] = FTS_SI_COMPENSATION_OFFSET_ADDR;
+ info->fts_read_reg(info, regAdd, 3, &buff[0], 4);
+ comp_header_addr = buff[1] + (buff[2] << 8);
+
+ /* Read header of compensation area */
+ regAdd[0] = 0xD0;
+ regAdd[1] = (comp_header_addr >> 8) & 0xFF;
+ regAdd[2] = comp_header_addr & 0xFF;
+ info->fts_read_reg(info, regAdd, 3, &buff[0], 16 + 1);
+ tx_num = buff[5];
+ rx_num = buff[6];
+ comp_start_addr = comp_header_addr + 0x10;
+
+ /* Read compensation data */
+ for (j = 0; j < tx_num; j++) {
+ memset(&ReadData[j], 0x0, rx_num);
+ memset(pStr, 0x0, 4 * (rx_num + 1));
+ snprintf(pTmp, sizeof(pTmp), "Tx%02d | ", j);
+ strlcat(pStr, pTmp, 4 * (rx_num + 1));
+
+ addr = comp_start_addr + (rx_num * j);
+ regAdd[0] = 0xD0;
+ regAdd[1] = (addr >> 8) & 0xFF;
+ regAdd[2] = addr & 0xFF;
+ info->fts_read_reg(info, regAdd, 3, &ReadData[j][0], rx_num + 1);
+ for (i = 0; i < rx_num; i++) {
+ snprintf(pTmp, sizeof(pTmp), "%3d", ReadData[j][i]);
+ strlcat(pStr, pTmp, 4 * (rx_num + 1));
+ }
+ tsp_debug_info(&info->client->dev, "%s\n", pStr);
+ }
+
+ if (info->cx_data) {
+ for (j = 0; j < tx_num; j++) {
+ for (i = 0; i < rx_num; i++) {
+ info->cx_data[(j * rx_num) + i] =
+ ReadData[j][i];
+ snprintf(buff, sizeof(buff),
+ "%d,", ReadData[j][i]);
+ strlcat(all_strbuff, buff, sizeof(all_strbuff));
+ }
+ }
+ }
+
+ kfree(pStr);
+
+out:
+ info->fts_irq_enable(info, true);
+ info->fts_command(info, SENSEON);
+ info->cmd_state = CMD_STATUS_OK;
+ set_cmd_result(info, all_strbuff,
+ strnlen(all_strbuff, sizeof(all_strbuff)));
+ tsp_debug_info(&info->client->dev,
+ "%s: %ld (%ld)\n", __func__,
+ strnlen(all_strbuff, sizeof(all_strbuff)),
+ sizeof(all_strbuff));
+}
+
+
+static void set_tsp_test_result(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+ unsigned char regAdd[4] = {0xB0, 0x07, 0xE7, 0x00};
+
+ set_default_result(info);
+
+ if (info->cmd_param[0] < TSP_FACTEST_RESULT_NONE
+ || info->cmd_param[0] > TSP_FACTEST_RESULT_PASS) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ info->cmd_state = CMD_STATUS_FAIL;
+ return;
+ }
+
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ info->fts_irq_enable(info, false);
+ info->fts_interrupt_set(info, INT_DISABLE);
+
+ regAdd[3] = info->cmd_param[0];
+ info->fts_write_reg(info, ®Add[0], 4);
+ fts_delay(100);
+ info->fts_command(info, FTS_CMD_SAVE_FWCONFIG);
+
+ fts_delay(230);
+ fts_fw_wait_for_event(info, STATUS_EVENT_FLASH_WRITE_CONFIG, 0x00);
+
+ info->fts_irq_enable(info, true);
+ info->fts_interrupt_set(info, INT_ENABLE);
+
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ info->cmd_state = CMD_STATUS_OK;
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void get_tsp_test_result(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ unsigned char cmd[4] = {0xB2, 0x07, 0xE7, 0x01};
+ char buff[CMD_STR_LEN] = { 0 };
+ int timeout = 0;
+
+ set_default_result(info);
+
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ info->fts_command(info, FLUSHBUFFER);
+ info->fts_write_reg(info, &cmd[0], 4);
+ info->cmd_state = CMD_STATUS_RUNNING;
+
+ while (info->cmd_state == CMD_STATUS_RUNNING) {
+ if (timeout++ > 30) {
+ info->cmd_state = CMD_STATUS_FAIL;
+ break;
+ }
+ fts_delay(10);
+ }
+}
+
+static void run_trx_short_test(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+ int ret = 0;
+
+ set_default_result(info);
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+ info->fts_irq_enable(info, false);
+ ret = fts_panel_ito_test(info);
+ if (ret == 1)
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ else
+ snprintf(buff, sizeof(buff), "%s", "FAIL");
+ info->fts_irq_enable(info, true);
+ info->cmd_state = CMD_STATUS_OK;
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void report_rate(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+
+ set_default_result(info);
+
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ goto out;
+ }
+
+ if (info->cmd_param[0] < 0 || info->cmd_param[0] > 2) {
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ info->cmd_state = CMD_STATUS_FAIL;
+ } else {
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ info->cmd_state = CMD_STATUS_OK;
+ }
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_WAITING;
+
+out:
+ mutex_lock(&info->cmd_lock);
+ info->cmd_is_running = false;
+ mutex_unlock(&info->cmd_lock);
+
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void delay(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+
+ set_default_result(info);
+
+ info->delay_time = info->cmd_param[0];
+
+ tsp_debug_info(&info->client->dev,
+ "%s: delay time is %d\n", __func__, info->delay_time);
+ snprintf(buff, sizeof(buff), "%d", info->delay_time);
+
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+
+ mutex_lock(&info->cmd_lock);
+ info->cmd_is_running = false;
+ mutex_unlock(&info->cmd_lock);
+ info->cmd_state = CMD_STATUS_WAITING;
+
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void debug(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+
+ set_default_result(info);
+
+ info->debug_string = info->cmd_param[0];
+
+ tsp_debug_info(&info->client->dev,
+ "%s: command is %d\n", __func__, info->debug_string);
+
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+
+ mutex_lock(&info->cmd_lock);
+ info->cmd_is_running = false;
+ mutex_unlock(&info->cmd_lock);
+ info->cmd_state = CMD_STATUS_WAITING;
+
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void run_autotune_enable(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+
+ set_default_result(info);
+
+ info->run_autotune = info->cmd_param[0];
+
+ tsp_debug_info(&info->client->dev, "%s: command is %s\n",
+ __func__, info->run_autotune ? "ENABLE" : "DISABLE");
+
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+
+ mutex_lock(&info->cmd_lock);
+ info->cmd_is_running = false;
+ mutex_unlock(&info->cmd_lock);
+ info->cmd_state = CMD_STATUS_WAITING;
+
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+}
+
+static void run_autotune(void *device_data)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)device_data;
+ char buff[CMD_STR_LEN] = { 0 };
+
+ set_default_result(info);
+
+ if (info->touch_stopped) {
+ dev_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n", __func__);
+ }
+
+ if (info->touch_stopped) {
+ tsp_debug_info(&info->client->dev,
+ "%s: [ERROR] Touch is stopped\n",
+ __func__);
+ snprintf(buff, sizeof(buff), "%s", "TSP turned off");
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+ info->cmd_state = CMD_STATUS_NOT_APPLICABLE;
+ return;
+ }
+
+ if (!info->run_autotune) {
+ tsp_debug_info(&info->client->dev,
+ "%s: autotune is disabled, %d\n",
+ __func__, info->run_autotune);
+ goto autotune_fail;
+ }
+
+ info->fts_irq_enable(info, false);
+
+ if (info->digital_rev == FTS_DIGITAL_REV_2) {
+ info->fts_interrupt_set(info, INT_DISABLE);
+
+ info->fts_command(info, SENSEOFF);
+ fts_delay(50);
+
+ info->fts_command(info, FTS_CMD_TRIM_LOW_POWER_OSCILLATOR);
+ fts_delay(200);
+
+ info->fts_command(info, FLUSHBUFFER);
+
+ info->fts_release_all_finger(info);
+ fts_execute_autotune(info);
+
+ info->fts_command(info, SENSEON);
+
+ info->fts_interrupt_set(info, INT_ENABLE);
+ } else {
+ tsp_debug_info(&info->client->dev,
+ "%s: digital_rev not matched, %d\n",
+ __func__, info->digital_rev);
+ goto autotune_fail;
+ }
+
+ info->fts_irq_enable(info, true);
+ snprintf(buff, sizeof(buff), "%s", "OK");
+ info->cmd_state = CMD_STATUS_OK;
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+
+ mutex_lock(&info->cmd_lock);
+ info->cmd_is_running = false;
+ mutex_unlock(&info->cmd_lock);
+
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+ return;
+
+autotune_fail:
+ snprintf(buff, sizeof(buff), "%s", "NG");
+ info->cmd_state = CMD_STATUS_FAIL;
+ set_cmd_result(info, buff, strnlen(buff, sizeof(buff)));
+
+ mutex_lock(&info->cmd_lock);
+ info->cmd_is_running = false;
+ mutex_unlock(&info->cmd_lock);
+
+ tsp_debug_info(&info->client->dev, "%s: %s\n", __func__, buff);
+ return;
+}
+
+void fts_production_init(void *device_info)
+{
+ char pdc_dir_name[20] = {0, };
+ int j = 0;
+ struct fts_ts_info *info = (struct fts_ts_info *)device_info;
+ int retval = 0;
+
+ INIT_LIST_HEAD(&info->cmd_list_head);
+
+ info->cmd_buf_size = 0;
+ for (j = 0; j < ARRAY_SIZE(fts_commands); j++) {
+ list_add_tail(&fts_commands[j].list, &info->cmd_list_head);
+ if (fts_commands[j].cmd_name)
+ info->cmd_buf_size += strlen(fts_commands[j].cmd_name) + 1;
+ }
+
+ mutex_init(&info->cmd_lock);
+ info->cmd_is_running = false;
+
+ sprintf(pdc_dir_name, "ftm4_touch");
+
+ info->pdc_dev_ts = device_create(info->input_dev->dev.class,
+ NULL, 0, NULL, pdc_dir_name);
+ if (IS_ERR(info->pdc_dev_ts)) {
+ tsp_debug_err(&info->client->dev,
+ "FTS Failed to create device for the sysfs\n");
+ retval = -ENOENT;
+ goto err_sysfs;
+ }
+
+ dev_set_drvdata(info->pdc_dev_ts, info);
+
+ retval = sysfs_create_group(&info->pdc_dev_ts->kobj,
+ &touch_pdc_attr_group);
+ if (retval < 0) {
+ tsp_debug_err(&info->client->dev,
+ "FTS Failed to create sysfs group\n");
+ goto err_sysfs;
+ }
+
+
+ if (retval < 0) {
+ tsp_debug_err(&info->client->dev,
+ "%s: Failed to create link\n", __func__);
+ goto err_sysfs;
+ }
+
+ return;
+err_sysfs:
+ mutex_destroy(&info->cmd_lock);
+}
+EXPORT_SYMBOL(fts_production_init);
diff --git a/drivers/input/touchscreen/stm/ftm4_ts.c b/drivers/input/touchscreen/stm/ftm4_ts.c
new file mode 100644
index 0000000..d5a9f94
--- /dev/null
+++ b/drivers/input/touchscreen/stm/ftm4_ts.c
@@ -0,0 +1,2277 @@
+/******************** (C) COPYRIGHT 2012 STMicroelectronics ********************
+*
+* File Name : fts.c
+* Authors : AMS(Analog Mems Sensor) Team
+* Description : FTS Capacitive touch screen controller (FingerTipS)
+*
+********************************************************************************
+*
+* This program is free software; you can redistribute it and/or modify
+* it under the terms of the GNU General Public License version 2 as
+* published by the Free Software Foundation.
+*
+* THE PRESENT SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES
+* OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, FOR THE SOLE
+* PURPOSE TO SUPPORT YOUR APPLICATION DEVELOPMENT.
+* AS A RESULT, STMICROELECTRONICS SHALL NOT BE HELD LIABLE FOR ANY DIRECT,
+* INDIRECT OR CONSEQUENTIAL DAMAGES WITH RESPECT TO ANY CLAIMS ARISING FROM THE
+* CONTENT OF SUCH SOFTWARE AND/OR THE USE MADE BY CUSTOMERS OF THE CODING
+* INFORMATION CONTAINED HEREIN IN CONNECTION WITH THEIR PRODUCTS.
+*
+* THIS SOFTWARE IS SPECIFICALLY DESIGNED FOR EXCLUSIVE USE WITH ST PARTS.
+********************************************************************************
+* REVISON HISTORY
+* DATE | DESCRIPTION
+* 03/09/2012| First Release
+* 08/11/2012| Code migration
+* 09/04/2013| Support Blob Information
+*******************************************************************************/
+
+#include <linux/init.h>
+#include <linux/errno.h>
+#include <linux/platform_device.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/input.h>
+#include <linux/interrupt.h>
+#include <linux/serio.h>
+#include <linux/init.h>
+#include <linux/pm.h>
+#include <linux/delay.h>
+#include <linux/ctype.h>
+#include <linux/gpio.h>
+#include <linux/i2c.h>
+#include <linux/i2c-dev.h>
+#include <linux/fs.h>
+#include <linux/uaccess.h>
+#include <linux/power_supply.h>
+#include <linux/firmware.h>
+#include <linux/regulator/consumer.h>
+#include <linux/of_gpio.h>
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+#include <linux/trustedui.h>
+#endif
+
+#ifdef CONFIG_OF
+#ifndef USE_OPEN_CLOSE
+#define USE_OPEN_CLOSE
+#undef CONFIG_PM
+#endif
+#endif
+
+#include <linux/input/mt.h>
+#include "ftm4_ts.h"
+
+static struct i2c_driver fts_i2c_driver;
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+extern int tui_force_close(uint32_t arg);
+struct fts_ts_info *tui_tsp_info;
+#endif
+
+#ifdef USE_OPEN_CLOSE
+static int fts_input_open(struct input_dev *dev);
+static void fts_input_close(struct input_dev *dev);
+#ifdef USE_OPEN_DWORK
+static void fts_open_work(struct work_struct *work);
+#endif
+#endif
+
+static int fts_stop_device(struct fts_ts_info *info);
+static int fts_start_device(struct fts_ts_info *info);
+static void fts_irq_enable(struct fts_ts_info *info, bool enable);
+static void fts_reset_work(struct work_struct *work);
+void fts_recovery_cx(struct fts_ts_info *info);
+void fts_release_all_finger(struct fts_ts_info *info);
+static int fts_suspend(struct i2c_client *client, pm_message_t mesg);
+static int fts_resume(struct i2c_client *client);
+
+#if defined(CONFIG_FB)
+static int touch_fb_notifier_callback(struct notifier_block *self,
+ unsigned long event, void *data);
+#endif
+
+int fts_write_reg(struct fts_ts_info *info,
+ unsigned char *reg, unsigned short num_com)
+{
+ struct i2c_msg xfer_msg[2];
+ int ret = 0;
+
+ if (info->touch_stopped) {
+ tsp_debug_err(&info->client->dev,
+ "%s: Sensor stopped\n", __func__);
+ goto exit;
+ }
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+ if (TRUSTEDUI_MODE_INPUT_SECURED & trustedui_get_current_mode()) {
+ tsp_debug_err(&info->client->dev,
+ "%s TSP no accessible from Linux,"
+ "TUI is enabled!\n", __func__);
+ return -EIO;
+ }
+#endif
+
+ mutex_lock(&info->i2c_mutex);
+
+ xfer_msg[0].addr = info->client->addr;
+ xfer_msg[0].len = num_com;
+ xfer_msg[0].flags = 0;
+ xfer_msg[0].buf = reg;
+
+ ret = i2c_transfer(info->client->adapter, xfer_msg, 1);
+
+ mutex_unlock(&info->i2c_mutex);
+ return ret;
+
+ exit:
+ return -EIO;
+}
+
+int fts_read_reg(struct fts_ts_info *info, unsigned char *reg, int cnum,
+ unsigned char *buf, int num)
+{
+ struct i2c_msg xfer_msg[2];
+ int ret = 0;
+
+ if (info->touch_stopped) {
+ tsp_debug_err(&info->client->dev,
+ "%s: Sensor stopped\n", __func__);
+ goto exit;
+ }
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+ if (TRUSTEDUI_MODE_INPUT_SECURED & trustedui_get_current_mode()) {
+ tsp_debug_err(&info->client->dev,
+ "%s TSP no accessible from Linux,"
+ " TUI is enabled!\n", __func__);
+ return -EIO;
+ }
+#endif
+
+ mutex_lock(&info->i2c_mutex);
+
+ xfer_msg[0].addr = info->client->addr;
+ xfer_msg[0].len = cnum;
+ xfer_msg[0].flags = 0;
+ xfer_msg[0].buf = reg;
+
+ xfer_msg[1].addr = info->client->addr;
+ xfer_msg[1].len = num;
+ xfer_msg[1].flags = I2C_M_RD;
+ xfer_msg[1].buf = buf;
+
+ ret = i2c_transfer(info->client->adapter, xfer_msg, 2);
+
+ mutex_unlock(&info->i2c_mutex);
+
+ return ret;
+
+ exit:
+ return -EIO;
+}
+
+void fts_delay(unsigned int ms)
+{
+ if (ms < 20)
+ usleep_range(ms * 1000, ms * 1000);
+ else
+ msleep(ms);
+}
+
+void fts_command(struct fts_ts_info *info, unsigned char cmd)
+{
+ unsigned char regAdd = 0;
+ int ret = 0;
+
+ regAdd = cmd;
+ ret = fts_write_reg(info, ®Add, 1);
+ tsp_debug_dbg(&info->client->dev,
+ "FTS Command (%02X) , "
+ "ret = %d\n", cmd, ret);
+}
+
+void fts_change_scan_rate(struct fts_ts_info *info, unsigned char cmd)
+{
+ unsigned char regAdd[2] = {0xC3, 0x00};
+ int ret = 0;
+
+ regAdd[1] = cmd;
+ ret = fts_write_reg(info, ®Add[0], 2);
+
+ tsp_debug_dbg(&info->client->dev,
+ "FTS %s Scan Rate (%02X %02X) , ret = %d\n",
+ (cmd == FTS_CMD_FAST_SCAN) ? "90Hz" :
+ (cmd == FTS_CMD_SLOW_SCAN) ? "60Hz" : "30Hz",
+ regAdd[0], regAdd[1], ret);
+}
+
+int fts_systemreset(struct fts_ts_info *info)
+{
+ int ret = 0;
+ unsigned char addr[4] = {0xB6, 0x00, 0x28, 0x80};
+ unsigned char addr_wbcrc[4] = {0xB6, 0x00, 0x1E, 0x20};
+
+ tsp_debug_info(&info->client->dev, "FTS Enable WBCRC\n");
+ ret = fts_write_reg(info, &addr_wbcrc[0], 4);
+ fts_delay(10);
+
+ tsp_debug_dbg(&info->client->dev, "FTS SystemReset\n");
+ ret = fts_write_reg(info, &addr[0], 4);
+ fts_delay(10);
+
+ return ret;
+}
+
+static void fts_interrupt_set(struct fts_ts_info *info, int enable)
+{
+ unsigned char regAdd[4] = {0xB6, 0x00, 0x2C, INT_ENABLE};
+
+ if (enable== INT_ENABLE) {
+ tsp_debug_dbg(&info->client->dev, "FTS INT Enable\n");
+ } else {
+ regAdd[3] = INT_DISABLE;
+ tsp_debug_dbg(&info->client->dev, "FTS INT Disable\n");
+ }
+
+ fts_write_reg(info, ®Add[0], 4);
+}
+
+void fts_get_afe_info(struct fts_ts_info *info)
+{
+ int rc = 0;
+ unsigned char regAdd[3];
+ unsigned char data[FTS_EVENT_SIZE];
+
+ /* Reading the final AFE version */
+ regAdd[0] = 0xd0;
+ regAdd[1] = 0x00;
+ regAdd[2] = 0x52;
+
+ rc = fts_read_reg(info, regAdd, 3, (unsigned char *)data, 3);
+ if (rc < 0) {
+ info->afe_ver = 0;
+ tsp_debug_err(info->dev,
+ "%s: Read Fail - Final AFE [Data : "
+ "%2X] AFE Ver [Data : %2X] \n",
+ __func__,
+ data[1],
+ data[2]);
+ } else
+ info->afe_ver = data[2];
+}
+
+static int fts_product_info_read(struct fts_ts_info *info)
+{
+ unsigned char data[FTS_EVENT_SIZE] = {0,};
+ unsigned char prd_info[FTS_LOCKDOWNCODE_SIZE] ={0x0,};
+ static u8 addr[2] = {READ_ONE_EVENT, 0};
+ int retry = (FTS_RETRY_COUNT * 5);
+ int total_length = 0;
+ int offset = 0;
+ int ret = 0;
+ int i = 0;
+
+ memset(&info->prd_info, 0, sizeof(struct fts_prd_info));
+
+ info->fts_interrupt_set(info, INT_DISABLE);
+ info->fts_command(info, SENSEOFF);
+
+ fts_command(info,LOCKDOWN_READ);
+
+ while (retry--) {
+ fts_delay(5);
+
+ ret = fts_read_reg(info, &addr[0], 1, &data[0], FTS_EVENT_SIZE);
+ if (ret < 0) {
+ tsp_debug_err(info->dev, "ftm4_reg_read fail\n");
+ goto error;
+ }
+
+ if (data[0] == EVENTID_LOCKDOWN_CODE) {
+ total_length = data[1];
+ offset = data[2];
+
+ tsp_debug_dbg(info->dev, "Total length : %d | offset : %d\n", total_length, offset);
+
+ if (total_length == FTS_LOCKDOWNCODE_SIZE) {
+ for (i = 0; i < 4; i++) {
+ if (offset+i >= FTS_LOCKDOWNCODE_SIZE) {
+ strncpy(&info->prd_info.product_id[0], &prd_info[0], 3);
+ info->prd_info.chip_rev = (prd_info[3] >> 4) & 0xF;
+ info->prd_info.fpc_rev = prd_info[3] & 0xF;
+ info->prd_info.t_sensor_rev = prd_info[4];
+ info->prd_info.site = prd_info[5];
+ info->prd_info.inspector_no = prd_info[6];
+ strncpy(&info->prd_info.date[0], &prd_info[7], 6);
+
+ info->fts_command(info, SENSEON);
+ info->fts_interrupt_set(info, INT_ENABLE);
+ return 0;
+ }
+ prd_info[offset+i] = data[i+3];
+ tsp_debug_dbg(info->dev, "[fts_lockdown_read] code [0x%02X]\n", prd_info[offset+i]);
+ }
+ }
+ } else if ((data[0] == EVENTID_ERROR) && (data[1] == EVENTID_ERROR_LOCKDOWN)) {
+ switch (data[2] & 0x0F) {
+ case 0x01:
+ tsp_debug_err(info->dev, "[fts_lockdown_read] Error - no lockdown code");
+ goto error;
+ case 0x02:
+ tsp_debug_err(info->dev, "[fts_lockdown_read] Error - Data Corrupted");
+ goto error;
+ case 0x03:
+ tsp_debug_err(info->dev, "[fts_lockdown_read] Error - Command format invalid");
+ goto error;
+ }
+ }
+ }
+
+ tsp_debug_err(info->dev, "[fts_lockdown_read] Error - Time over, retry =%d", retry);
+error:
+ info->fts_command(info, SENSEON);
+ info->fts_interrupt_set(info, INT_ENABLE);
+
+ return -EINVAL;
+}
+
+int fts_get_version_info(struct fts_ts_info *info)
+{
+ int rc = 0;
+ unsigned char addr[3] = {0xD0, 0x00, 0x56};
+ unsigned char buff[7] = {0};
+ char str[16] = {0};
+ int str_ret = 0;
+
+ rc = fts_read_reg(info, &addr[0], 3, &buff[0], 7);
+ if (rc < 0) {
+ tsp_debug_err(&info->client->dev, "FTS get version info fail!\n");
+ goto error;
+ }
+
+ info->fw_version_of_ic = buff[1] + (buff[2] << 8);
+ info->config_version_of_ic = buff[3] + (buff[4] << 8);
+ info->fw_main_version_of_ic = buff[6] + (buff[5] << 8);
+ info->ic_fw_ver.build = ((buff[5] >> 4) & 0x0F);
+ info->ic_fw_ver.major = (buff[5] & 0x0F);
+ info->ic_fw_ver.minor = buff[6];
+
+ str_ret += snprintf(str + str_ret, sizeof(str) - str_ret,
+ "v%d.%02d",
+ info->ic_fw_ver.major,
+ info->ic_fw_ver.minor);
+
+ if (info->ic_fw_ver.build) {
+ str_ret += snprintf(str + str_ret, sizeof(str) - str_ret,
+ ".%d",
+ info->ic_fw_ver.build);
+ }
+
+ fts_get_afe_info(info);
+ fts_product_info_read(info);
+
+ tsp_debug_info(&info->client->dev,
+ "IC Firmware Version : 0x%04X [%s] "
+ "IC Config Version : 0x%04X "
+ "IC Main Version : 0x%04X "
+ "AFE Version : 0x%02X\n",
+ info->fw_version_of_ic, str,
+ info->config_version_of_ic,
+ info->fw_main_version_of_ic,
+ info->afe_ver);
+ tsp_debug_info(&info->client->dev,
+ "product id : [%02x %02x %02x]\n",
+ info->prd_info.product_id[0],
+ info->prd_info.product_id[1],
+ info->prd_info.product_id[2]);
+ tsp_debug_info(&info->client->dev,
+ "Chip reveion : %d, fpc : %d, t_sensor: %d site : %d , inspector_no : %d\n",
+ info->prd_info.chip_rev, info->prd_info.fpc_rev, info->prd_info.t_sensor_rev,
+ info->prd_info.site, info->prd_info.inspector_no);
+ tsp_debug_info(&info->client->dev,
+ "date : %02d.%02d.%02d %02d:%02d:%02d\n",
+ info->prd_info.date[0], info->prd_info.date[1], info->prd_info.date[2],
+ info->prd_info.date[3], info->prd_info.date[4], info->prd_info.date[5]);
+
+error:
+ return rc;
+}
+
+int fts_read_chip_id(struct fts_ts_info *info)
+{
+ unsigned char regAdd[3] = {0xB6, 0x00, 0x04};
+ unsigned char val[7] = {0};
+ int ret = 0;
+
+ ret = fts_read_reg(info, regAdd, 3, (unsigned char *)val, 7);
+ if (ret < 0) {
+ tsp_debug_err(&info->client->dev, "%s failed. ret: %d\n",
+ __func__, ret);
+ return ret;
+ }
+
+ tsp_debug_dbg(&info->client->dev,
+ "FTS %02X%02X%02X = %02X %02X %02X %02X %02X %02X\n",
+ regAdd[0], regAdd[1], regAdd[2],
+ val[1], val[2], val[3], val[4],
+ val[5], val[6]);
+
+ if ((val[1] == FTS_ID0) && (val[2] == FTS_ID1)) {
+ if ((val[5] == 0x00) && (val[6] == 0x00)) {
+ tsp_debug_err(&info->client->dev,
+ "\n\r[fts_read_chip_id] Error - No FW : %02X %02X",
+ val[5], val[6]);
+ info->flash_corruption_info.fw_broken = true;
+ } else {
+ tsp_debug_info(&info->client->dev,
+ "FTS Chip ID : %02X %02X\n",
+ val[1], val[2]);
+ }
+ } else
+ return -FTS_ERROR_INVALID_CHIP_ID;
+
+ return ret;
+}
+
+int fts_wait_for_ready(struct fts_ts_info *info)
+{
+ int rc = 0;
+ unsigned char addr;
+ unsigned char data[FTS_EVENT_SIZE];
+ int retry = 0;
+ int err_cnt = 0;
+
+ memset(data, 0x0, FTS_EVENT_SIZE);
+
+ addr = READ_ONE_EVENT;
+
+ while (fts_read_reg(info, &addr, 1,
+ (unsigned char *)data, FTS_EVENT_SIZE)) {
+ if (data[0] == EVENTID_CONTROLLER_READY) {
+ rc = 0;
+ break;
+ }
+
+ if (data[0] == EVENTID_ERROR) {
+ if (data[1] == EVENTID_ERROR_FLASH_CORRUPTION) {
+ rc = -FTS_ERROR_EVENT_ID;
+
+ tsp_debug_err(&info->client->dev,
+ "%s: flash corruption:%02X,%02X,%02X\n",
+ __func__, data[0],
+ data[1], data[2]);
+
+ switch (data[2]) {
+ case EVENTID_ERROR_CONFIG_FLASH_CORRUPTION_1:
+ info->flash_corruption_info.cfg_broken = true;
+ break;
+ case EVENTID_ERROR_CONFIG_FLASH_CORRUPTION_2:
+ info->flash_corruption_info.cfg_broken = true;
+ break;
+ case EVENTID_ERROR_CX_FLASH_CORRUPTION:
+ info->flash_corruption_info.cx_broken = true;
+ break;
+ default:
+ break;
+ }
+ }
+
+ if (err_cnt++ > 32) {
+ rc = -FTS_ERROR_EVENT_ID;
+ break;
+ }
+ continue;
+ }
+
+ if (retry++ > FTS_RETRY_COUNT) {
+ rc = -FTS_ERROR_TIMEOUT;
+ tsp_debug_err(&info->client->dev, "%s: Time Over\n",
+ __func__);
+
+ if (info->lowpower_mode) {
+ schedule_delayed_work(&info->reset_work,
+ msecs_to_jiffies(10));
+ }
+ break;
+ }
+ fts_delay(20);
+ }
+
+ tsp_debug_dbg(&info->client->dev,
+ "%s: %02X, %02X, %02X, %02X, %02X, %02X, %02X, %02X\n",
+ __func__, data[0], data[1], data[2], data[3],
+ data[4], data[5], data[6], data[7]);
+
+ return rc;
+}
+
+int fts_get_channel_info(struct fts_ts_info *info)
+{
+ int rc = 0;
+ unsigned char cmd[4] = { 0xB2, 0x00, 0x14, 0x02 };
+ unsigned char data[FTS_EVENT_SIZE];
+ int retry = 0;
+
+ memset(data, 0x0, FTS_EVENT_SIZE);
+
+ fts_write_reg(info, &cmd[0], 4);
+ cmd[0] = READ_ONE_EVENT;
+ while (fts_read_reg
+ (info, &cmd[0], 1, (unsigned char *)data, FTS_EVENT_SIZE)) {
+ if (data[0] == EVENTID_RESULT_READ_REGISTER) {
+ if ((data[1] == cmd[1]) && (data[2] == cmd[2])) {
+ info->SenseChannelLength = data[3];
+ info->ForceChannelLength = data[4];
+ rc = 0;
+ break;
+ }
+ }
+ if (retry++ > 30) {
+ rc = -1;
+ tsp_debug_err(&info->client->dev,
+ "Time over - wait for channel info\n");
+ break;
+ }
+ fts_delay(5);
+ }
+ return rc;
+}
+
+#ifdef FTS_SUPPORT_NOISE_PARAM
+int fts_get_noise_param_address(struct fts_ts_info *info)
+{
+ int rc = 0;
+ unsigned char regAdd[3];
+ unsigned char rData[3];
+ struct fts_noise_param *noise_param;
+ int i;
+
+ noise_param = (struct fts_noise_param *)&info->noise_param;
+
+ regAdd[0] = 0xd0;
+ regAdd[1] = 0x00;
+ regAdd[2] = 32 * 2;
+
+ if (info->digital_rev == FTS_DIGITAL_REV_1)
+ rc = fts_read_reg(info, regAdd, 3,
+ (unsigned char *)noise_param->pAddr, 2);
+ else {
+ rc = fts_read_reg(info, regAdd, 3, (unsigned char *)rData, 3);
+ noise_param->pAddr[0] = rData[1] + (rData[2]<<8);
+ }
+
+ for (i = 1; i < MAX_NOISE_PARAM; i++)
+ noise_param->pAddr[i] = noise_param->pAddr[0] + i * 2;
+
+ for (i = 0; i < MAX_NOISE_PARAM; i++)
+ tsp_debug_dbg(&info->client->dev,
+ "Get Noise Param%d Address = 0x%4x\n",
+ i, noise_param->pAddr[i]);
+
+ return rc;
+}
+
+static int fts_get_noise_param(struct fts_ts_info *info)
+{
+ int rc = 0;
+ unsigned char regAdd[3];
+ unsigned char data[MAX_NOISE_PARAM * 2];
+ struct fts_noise_param *noise_param;
+ int i;
+ unsigned char buf[3];
+
+ noise_param = (struct fts_noise_param *)&info->noise_param;
+ memset(data, 0x0, MAX_NOISE_PARAM * 2);
+
+ for (i = 0; i < MAX_NOISE_PARAM; i++) {
+ regAdd[0] = 0xb3;
+ regAdd[1] = 0x00;
+ regAdd[2] = 0x10;
+ fts_write_reg(info, regAdd, 3);
+
+ regAdd[0] = 0xb1;
+ regAdd[1] = (noise_param->pAddr[i] >> 8) & 0xff;
+ regAdd[2] = noise_param->pAddr[i] & 0xff;
+ rc = fts_read_reg(info, regAdd, 3, &buf[0], 3);
+
+ noise_param->pData[i] = buf[1]+(buf[2]<<8);
+ }
+
+ for (i = 0; i < MAX_NOISE_PARAM; i++) {
+ tsp_debug_dbg(&info->client->dev,
+ "Get Noise Param%d Address "
+ "[ 0x%04x ] = 0x%04x\n", i,
+ noise_param->pAddr[i],
+ noise_param->pData[i]);
+ }
+
+ return rc;
+}
+
+static int fts_set_noise_param(struct fts_ts_info *info)
+{
+ int i;
+ unsigned char regAdd[5];
+ struct fts_noise_param *noise_param;
+
+ noise_param = (struct fts_noise_param *)&info->noise_param;
+
+ for (i = 0; i < MAX_NOISE_PARAM; i++) {
+ regAdd[0] = 0xb3;
+ regAdd[1] = 0x00;
+ regAdd[2] = 0x10;
+ fts_write_reg(info, regAdd, 3);
+
+ regAdd[0] = 0xb1;
+ regAdd[1] = (noise_param->pAddr[i] >> 8) & 0xff;
+ regAdd[2] = noise_param->pAddr[i] & 0xff;
+ regAdd[3] = noise_param->pData[i] & 0xff;
+ regAdd[4] = (noise_param->pData[i] >> 8) & 0xff;
+ fts_write_reg(info, regAdd, 5);
+ }
+
+ for (i = 0; i < MAX_NOISE_PARAM; i++) {
+ tsp_debug_dbg(&info->client->dev,
+ "Set Noise Param%d Address "
+ "[ 0x%04x ] = 0x%04x\n",
+ i,
+ noise_param->pAddr[i],
+ noise_param->pData[i]);
+ }
+
+ return 0;
+}
+#endif
+
+int fts_cmd_completion_check(struct fts_ts_info *info, uint8_t event1, uint8_t event2, uint8_t event3)
+{
+ unsigned char val[8];
+ unsigned char reg[2] = {0x85, 0};
+ int retry = 100;
+ int rc = 0;
+
+ while (retry--) {
+ fts_delay(10);
+ info->fts_read_reg(info, ®[0], 1, &val[0], FTS_EVENT_SIZE);
+ if ((val[0] == event1) && (val[1] == event2) && (val[2] == event3)) {
+ tsp_debug_info(&info->client->dev,
+ "\n\r[fts_cmd_completion_check] OK [%02x][%02x][%02x]", val[0], val[1], val[2]);
+ return rc;
+ } else if (val[0] == 0x0F) {
+ tsp_debug_err(&info->client->dev,
+ "\n\r[fts_cmd_completion_check] Error - [%02x][%02x][%02x]", val[0], val[1], val[2]);
+ }
+ }
+
+ rc = -1;
+ if (retry <= 0)
+ tsp_debug_err(&info->client->dev,
+ "\n\r[fts_cmd_completion_check] Error - Time Over [%02x][%02x][%02x]", event1, event2, event3);
+ return rc;
+}
+
+static int fts_init(struct fts_ts_info *info)
+{
+ unsigned char val[16];
+ unsigned char regAdd[8];
+ int rc = 0;
+
+ fts_systemreset(info);
+
+ rc = fts_wait_for_ready(info);
+ if (rc == -FTS_ERROR_EVENT_ID) {
+ info->fw_version_of_ic = 0;
+ info->config_version_of_ic = 0;
+ info->fw_main_version_of_ic = 0;
+ } else {
+ fts_get_version_info(info);
+ }
+
+ rc = fts_read_chip_id(info);
+ if (rc < 0)
+ tsp_debug_err(&info->client->dev, "%s: Failed to fts_read_chip_id\n",
+ __func__);
+
+/*
+ rc = fts_fw_update(info);
+ if (rc < 0)
+ tsp_debug_err(&info->client->dev, "%s: Failed to firmware update\n",
+ __func__);
+*/
+
+#ifdef FEATURE_FTS_PRODUCTION_CODE
+ info->digital_rev = FTS_DIGITAL_REV_2;
+ rc = info->fts_get_channel_info(info);
+ if (rc == 0) {
+ tsp_debug_dbg(&info->client->dev,
+ "FTS Sense(%02d) Force(%02d)\n",
+ info->SenseChannelLength,
+ info->ForceChannelLength);
+ } else {
+ tsp_debug_err(&info->client->dev,
+ "FTS read failed rc = %d\n", rc);
+ tsp_debug_err(&info->client->dev,
+ "FTS Initialise Failed\n");
+ }
+ info->pFrame =
+ kzalloc(info->SenseChannelLength * info->ForceChannelLength * 2,
+ GFP_KERNEL);
+ if (info->pFrame == NULL) {
+ tsp_debug_err(&info->client->dev,
+ "FTS pFrame kzalloc Failed\n");
+ return -ENOMEM;
+ }
+ info->cx_data = kzalloc(info->SenseChannelLength *
+ info->ForceChannelLength, GFP_KERNEL);
+ if (!info->cx_data)
+ tsp_debug_err(&info->client->dev,
+ "%s: cx_data kzalloc Failed\n", __func__);
+#endif
+ fts_command(info, SENSEON);
+
+#ifdef FTS_SUPPORT_NOISE_PARAM
+ fts_get_noise_param_address(info);
+#endif
+ /* fts driver set functional feature */
+ info->touch_count = 0;
+ info->palm_pressed = false;
+ info->flip_enable = false;
+ info->mainscr_disable = false;
+
+ info->deepsleep_mode = false;
+ info->wirelesscharger_mode = false;
+ info->lowpower_mode = false;
+ info->lowpower_flag = 0x00;
+ info->fts_power_state = FTS_POWER_STATE_ACTIVE;
+
+ fts_command(info, FORCECALIBRATION);
+
+ fts_interrupt_set(info, INT_ENABLE);
+
+ memset(val, 0x0, 4);
+ regAdd[0] = READ_STATUS;
+ fts_read_reg(info, regAdd, 1, (unsigned char *)val, 4);
+ tsp_debug_dbg(&info->client->dev,
+ "FTS ReadStatus(0x84) : "
+ "%02X %02X %02X %02X\n",
+ val[0], val[1], val[2], val[3]);
+
+ tsp_debug_info(&info->client->dev,
+ "FTS Initialized\n");
+
+ return 0;
+}
+
+static void fts_debug_msg_event_handler(struct fts_ts_info *info,
+ unsigned char data[])
+{
+ tsp_debug_dbg(&info->client->dev,
+ "%s: %02X %02X %02X %02X "
+ "%02X %02X %02X %02X\n", __func__,
+ data[0], data[1], data[2], data[3],
+ data[4], data[5], data[6], data[7]);
+}
+
+static unsigned char fts_event_handler_type_b(struct fts_ts_info *info,
+ unsigned char data[],
+ unsigned char LeftEvent)
+{
+ unsigned char EventNum = 0;
+ unsigned char NumTouches = 0;
+ unsigned char TouchID = 0, EventID = 0, status = 0;
+ unsigned char LastLeftEvent = 0;
+ int x = 0, y = 0, z = 0;
+ int bw = 0, bh = 0, palm = 0;
+ int orient = 0;
+
+#if defined(CONFIG_INPUT_BOOSTER)
+ bool booster_restart = false;
+#endif
+
+ for (EventNum = 0; EventNum < LeftEvent; EventNum++) {
+#ifdef DEBUG
+ tsp_debug_dbg(&info->client->dev,
+ "%d %2x %2x %2x %2x %2x %2x %2x %2x\n",
+ EventNum,
+ data[EventNum * FTS_EVENT_SIZE],
+ data[EventNum * FTS_EVENT_SIZE+1],
+ data[EventNum * FTS_EVENT_SIZE+2],
+ data[EventNum * FTS_EVENT_SIZE+3],
+ data[EventNum * FTS_EVENT_SIZE+4],
+ data[EventNum * FTS_EVENT_SIZE+5],
+ data[EventNum * FTS_EVENT_SIZE+6],
+ data[EventNum * FTS_EVENT_SIZE+7]);
+ tsp_debug_dbg(&info->client->dev,
+ "fts_power_state (%d)\n",
+ info->fts_power_state );
+#endif
+
+ if (info->fts_power_state == FTS_POWER_STATE_LOWPOWER)
+ EventID = data[EventNum * FTS_EVENT_SIZE] & 0xFF;
+ else
+ EventID = data[EventNum * FTS_EVENT_SIZE] & 0x0F;
+
+ if ((EventID >= 3) && (EventID <= 5)) {
+ LastLeftEvent = 0;
+ NumTouches = 1;
+ TouchID = (data[EventNum * FTS_EVENT_SIZE] >> 4) & 0x0F;
+ } else {
+ LastLeftEvent =
+ data[7 + EventNum * FTS_EVENT_SIZE] & 0x0F;
+ NumTouches =
+ (data[1 + EventNum * FTS_EVENT_SIZE] & 0xF0) >> 4;
+ TouchID = data[1 + EventNum * FTS_EVENT_SIZE] & 0x0F;
+ EventID = data[EventNum * FTS_EVENT_SIZE] & 0xFF;
+ status = data[1 + EventNum * FTS_EVENT_SIZE] & 0xFF;
+ }
+
+ switch (EventID) {
+ case EVENTID_NO_EVENT:
+ break;
+
+ case EVENTID_ERROR:
+ if (data[1 + EventNum *
+ FTS_EVENT_SIZE] == 0x08) {
+ /* Get Auto tune fail event */
+ if (data[2 + EventNum *
+ FTS_EVENT_SIZE] == 0x00) {
+ tsp_debug_err(&info->client->dev,
+ "[FTS] Fail Mutual Auto tune\n");
+ } else if (data[2 + EventNum *
+ FTS_EVENT_SIZE] == 0x01) {
+ tsp_debug_err(&info->client->dev,
+ "[FTS] Fail Self Auto tune\n");
+ }
+ } else if (data[1 + EventNum *
+ FTS_EVENT_SIZE] == 0x09)
+ /* Get detect SYNC fail event */
+ tsp_debug_err(&info->client->dev,
+ "[FTS] Fail detect SYNC\n");
+ break;
+
+ case EVENTID_HOVER_ENTER_POINTER:
+ case EVENTID_HOVER_MOTION_POINTER:
+ x = ((data[4 + EventNum * FTS_EVENT_SIZE] & 0xF0) >> 4)
+ | ((data[2 + EventNum * FTS_EVENT_SIZE]) << 4);
+ y = ((data[4 + EventNum * FTS_EVENT_SIZE] & 0x0F) |
+ ((data[3 + EventNum * FTS_EVENT_SIZE]) << 4));
+
+ z = data[5 + EventNum * FTS_EVENT_SIZE];
+
+ input_mt_slot(info->input_dev, 0);
+ input_mt_report_slot_state(info->input_dev,
+ MT_TOOL_FINGER, 1);
+
+ input_report_key(info->input_dev, BTN_TOUCH, 0);
+ input_report_key(info->input_dev, BTN_TOOL_FINGER, 1);
+
+ input_report_abs(info->input_dev, ABS_MT_POSITION_X, x);
+ input_report_abs(info->input_dev, ABS_MT_POSITION_Y, y);
+ input_report_abs(info->input_dev, ABS_MT_DISTANCE, 255 - z);
+ break;
+
+ case EVENTID_HOVER_LEAVE_POINTER:
+ input_mt_slot(info->input_dev, 0);
+ input_mt_report_slot_state(info->input_dev,
+ MT_TOOL_FINGER, 0);
+ break;
+
+ case EVENTID_ENTER_POINTER:
+ if (info->fts_power_state == FTS_POWER_STATE_LOWPOWER)
+ break;
+
+ info->touch_count++;
+#if defined(CONFIG_INPUT_BOOSTER)
+ booster_restart = true;
+#endif
+ case EVENTID_MOTION_POINTER:
+ if (info->fts_power_state == FTS_POWER_STATE_LOWPOWER) {
+ tsp_debug_info(&info->client->dev,
+ "%s: low power mode\n", __func__);
+ fts_release_all_finger(info);
+ break;
+ }
+
+ if (info->touch_count == 0) {
+ tsp_debug_info(&info->client->dev,
+ "%s: count 0\n", __func__);
+ fts_release_all_finger(info);
+ break;
+ }
+
+ if ((EventID == EVENTID_MOTION_POINTER) &&
+ (info->finger[TouchID].state ==
+ EVENTID_LEAVE_POINTER)) {
+ tsp_debug_info(&info->client->dev,
+ "%s: state leave but point is moved.\n", __func__);
+ break;
+ }
+
+ if (info->fts_power_state == FTS_POWER_STATE_LOWPOWER)
+ break;
+
+ x = ((data[1 + EventNum * FTS_EVENT_SIZE]
+ & 0xFF) << 4) +
+ ((data[3 + EventNum * FTS_EVENT_SIZE]
+ & 0xF0) >> 4);
+ y = ((data[2 + EventNum * FTS_EVENT_SIZE]
+ & 0xFF) << 4) +
+ (data[3 + EventNum * FTS_EVENT_SIZE]
+ & 0xF);
+
+ z = data[4 + EventNum * FTS_EVENT_SIZE];
+
+ bw = data[6 + EventNum * FTS_EVENT_SIZE];
+ bh = data[7 + EventNum * FTS_EVENT_SIZE];
+
+ orient = data[5 + EventNum * FTS_EVENT_SIZE];
+
+ if (z == 255) {
+ tsp_debug_info(&info->client->dev,
+ "%s: Palm Detected\n", __func__);
+ tsp_debug_info(&info->client->dev, "%s: "
+ "[ID:%2d X:%4d Y:%4d Z:%4d "
+ " WM:%4d Wm:%4d Orient:%2d "
+ "tc:%2d]\n", __func__,
+ TouchID, x, y, z,
+ max(bw, bh), min(bw, bh),
+ orient, info->touch_count);
+ info->palm_pressed = true;
+ fts_release_all_finger(info);
+ return 0;
+ }
+
+ input_mt_slot(info->input_dev, TouchID);
+ input_mt_report_slot_state(info->input_dev,
+ MT_TOOL_FINGER,
+ 1 + (palm << 1));
+
+ input_report_key(info->input_dev, BTN_TOUCH, 1);
+ input_report_key(info->input_dev,
+ BTN_TOOL_FINGER, 1);
+ input_report_abs(info->input_dev,
+ ABS_MT_POSITION_X, x);
+ input_report_abs(info->input_dev,
+ ABS_MT_POSITION_Y, y);
+
+ input_report_abs(info->input_dev,
+ ABS_MT_TOUCH_MAJOR, max(bw,
+ bh));
+
+ input_report_abs(info->input_dev,
+ ABS_MT_TOUCH_MINOR, min(bw,
+ bh));
+
+ input_report_abs(info->input_dev,
+ ABS_MT_PRESSURE, z);
+
+ input_report_abs(info->input_dev,
+ ABS_MT_ORIENTATION, orient);
+
+ info->finger[TouchID].lx = x;
+ info->finger[TouchID].ly = y;
+
+ break;
+
+ case EVENTID_LEAVE_POINTER:
+ if (info->fts_power_state == FTS_POWER_STATE_LOWPOWER)
+ break;
+
+ if (info->palm_pressed) {
+ tsp_debug_info(&info->client->dev,
+ "%s: Palm Released\n", __func__);
+ info->palm_pressed = false;
+ return 0;
+ }
+
+ if (info->touch_count <= 0) {
+ tsp_debug_info(&info->client->dev,
+ "%s: count 0\n", __func__);
+ fts_release_all_finger(info);
+ break;
+ }
+
+ info->touch_count--;
+
+ input_mt_slot(info->input_dev, TouchID);
+
+ input_mt_report_slot_state(info->input_dev,
+ MT_TOOL_FINGER, 0);
+
+ if (info->touch_count == 0) {
+ /* Clear BTN_TOUCH when All touch are released */
+ input_report_key(info->input_dev, BTN_TOUCH, 0);
+ input_report_key(info->input_dev, BTN_TOOL_FINGER, 0);
+
+ }
+ break;
+ case EVENTID_STATUS_EVENT:
+ if (status == STATUS_EVENT_GLOVE_MODE) {
+
+ } else if (status == STATUS_EVENT_RAW_DATA_READY) {
+ unsigned char regAdd[4] = {0xB0, 0x01, 0x29, 0x01};
+
+ fts_write_reg(info, ®Add[0], 4);
+
+ tsp_debug_dbg(&info->client->dev, "[FTS] Received the Raw Data Ready Event\n");
+ } else if (status == STATUS_EVENT_FORCE_CAL_MUTUAL) {
+ tsp_debug_dbg(&info->client->dev, "[FTS] Received Force Calibration Mutual only Event\n");
+ } else if (status == STATUS_EVENT_FORCE_CAL_SELF) {
+ tsp_debug_dbg(&info->client->dev, "[FTS] Received Force Calibration Self only Event\n");
+ } else if (status == STATUS_EVENT_WATERMODE_ON) {
+ tsp_debug_dbg(&info->client->dev, "[FTS] Received Water Mode On Event\n");
+ } else if (status == STATUS_EVENT_WATERMODE_OFF) {
+ tsp_debug_dbg(&info->client->dev, "[FTS] Received Water Mode Off Event\n");
+ } else if (status == STATUS_EVENT_MUTUAL_CAL_FRAME_CHECK) {
+ tsp_debug_dbg(&info->client->dev, "[FTS] Received Mutual Calib Frame Check Event\n");
+ } else if (status == STATUS_EVENT_SELF_CAL_FRAME_CHECK) {
+ tsp_debug_dbg(&info->client->dev, "[FTS] Received Self Calib Frame Check Event\n");
+ } else {
+ fts_debug_msg_event_handler(info,
+ &data[EventNum *
+ FTS_EVENT_SIZE]);
+ }
+ break;
+
+#ifdef FEATURE_FTS_PRODUCTION_CODE
+ case EVENTID_RESULT_READ_REGISTER:
+ procedure_cmd_event(info, &data[EventNum * FTS_EVENT_SIZE]);
+ break;
+#endif /* FEATURE_FTS_PRODUCTION_CODE */
+
+ default:
+ fts_debug_msg_event_handler(info,
+ &data[EventNum *
+ FTS_EVENT_SIZE]);
+ continue;
+ }
+
+ if (EventID == EVENTID_ENTER_POINTER)
+ tsp_debug_event(&info->client->dev,
+ "[P] tID:%d x:%d y:%d w:%d "
+ "h:%d z:%d p:%d tc:%d tm:%d\n",
+ TouchID, x, y, bw,
+ bh, z, palm, info->touch_count, info->touch_mode);
+ else if (EventID == EVENTID_HOVER_ENTER_POINTER)
+ tsp_debug_event(&info->client->dev,
+ "[HP] tID:%d x:%d y:%d z:%d\n",
+ TouchID, x, y, z);
+
+ else if (EventID == EVENTID_LEAVE_POINTER) {
+ tsp_debug_event(&info->client->dev,
+ "[R] tID:%d mc: %d tc:%d lx: %d ly: %d "
+ "Ver[%02X%04X%01X%01X]\n",
+ TouchID, info->finger[TouchID].mcount, info->touch_count,
+ info->finger[TouchID].lx,
+ info->finger[TouchID].ly,
+ info->panel_revision,
+ info->fw_main_version_of_ic,
+ info->flip_enable,
+ info->mainscr_disable);
+
+ info->finger[TouchID].mcount = 0;
+ } else if (EventID == EVENTID_HOVER_LEAVE_POINTER) {
+ tsp_debug_event(&info->client->dev,
+ "[HR] tID:%d Ver[%02X%04X%01X]\n",
+ TouchID,
+ info->panel_revision,
+ info->fw_main_version_of_ic,
+ info->flip_enable);
+ info->finger[TouchID].mcount = 0;
+ } else if (EventID == EVENTID_MOTION_POINTER)
+ info->finger[TouchID].mcount++;
+
+ if ((EventID == EVENTID_ENTER_POINTER) ||
+ (EventID == EVENTID_MOTION_POINTER) ||
+ (EventID == EVENTID_LEAVE_POINTER))
+ info->finger[TouchID].state = EventID;
+ }
+
+ input_sync(info->input_dev);
+
+#if defined(CONFIG_INPUT_BOOSTER)
+ if ((EventID == EVENTID_ENTER_POINTER)
+ || (EventID == EVENTID_LEAVE_POINTER)) {
+ if (booster_restart)
+ input_booster_send_event(BOOSTER_DEVICE_TOUCH, BOOSTER_MODE_ON);
+ if (!info->touch_count)
+ input_booster_send_event(BOOSTER_DEVICE_TOUCH, BOOSTER_MODE_OFF);
+ }
+#endif
+
+ return LastLeftEvent;
+}
+
+#ifdef FTS_SUPPORT_TA_MODE
+static void fts_ta_cb(struct fts_callbacks *cb, int ta_status)
+{
+ struct fts_ts_info *info =
+ container_of(cb, struct fts_ts_info, callbacks);
+
+ if (ta_status == 0x01 || ta_status == 0x03) {
+ fts_command(info, FTS_CMD_CHARGER_PLUGGED);
+ info->TA_Pluged = true;
+ tsp_debug_dbg(&info->client->dev,
+ "%s: device_control : CHARGER CONNECTED, ta_status : %x\n",
+ __func__, ta_status);
+ } else {
+ fts_command(info, FTS_CMD_CHARGER_UNPLUGGED);
+ info->TA_Pluged = false;
+ tsp_debug_dbg(&info->client->dev,
+ "%s: device_control : CHARGER DISCONNECTED, ta_status : %x\n",
+ __func__, ta_status);
+ }
+}
+#endif
+
+/**
+ * fts_interrupt_handler()
+ *
+ * Called by the kernel when an interrupt occurs (when the sensor
+ * asserts the attention irq).
+ *
+ * This function is the ISR thread and handles the acquisition
+ * and the reporting of finger data when the presence of fingers
+ * is detected.
+ */
+static irqreturn_t fts_interrupt_handler(int irq, void *handle)
+{
+ struct fts_ts_info *info = handle;
+ unsigned char regAdd[4] = {0xB6, 0x00, 0x23, READ_ALL_EVENT};
+ unsigned short evtcount = 0;
+
+ evtcount = 0;
+
+ fts_read_reg(info, ®Add[0], 3, (unsigned char *)&evtcount, 2);
+ evtcount = evtcount >> 8;
+ evtcount = evtcount / 2;
+
+ if (evtcount > FTS_FIFO_MAX)
+ evtcount = FTS_FIFO_MAX;
+
+ if (evtcount > 0) {
+ memset(info->data, 0x0, FTS_EVENT_SIZE * evtcount);
+ fts_read_reg(info, ®Add[3], 1, (unsigned char *)info->data,
+ FTS_EVENT_SIZE * evtcount);
+ fts_event_handler_type_b(info, info->data, evtcount);
+ }
+ return IRQ_HANDLED;
+}
+
+static void fts_irq_enable(struct fts_ts_info *info,
+ bool enable)
+{
+ spin_lock(&info->lock);
+
+ if (enable) {
+ if (atomic_cmpxchg(&info->irq_enabled, 0, 1) == 0) {
+ tsp_debug_dbg(info->dev, "enable_irq\n");
+ enable_irq(info->irq);
+ }
+ } else {
+ if (atomic_cmpxchg(&info->irq_enabled, 1, 0) == 1) {
+ tsp_debug_dbg(info->dev, "disable_irq\n");
+ disable_irq_nosync(info->irq);
+ }
+ }
+
+ spin_unlock(&info->lock);
+}
+
+#ifdef CONFIG_OF
+#ifdef FTS_SUPPORT_TA_MODE
+struct fts_callbacks *fts_charger_callbacks;
+void tsp_charger_infom(bool en)
+{
+ pr_err("[TSP]%s: ta:%d\n", __func__, en);
+
+ if (fts_charger_callbacks && fts_charger_callbacks->inform_charger)
+ fts_charger_callbacks->inform_charger(fts_charger_callbacks, en);
+}
+static void fts_tsp_register_callback(void *cb)
+{
+ fts_charger_callbacks = cb;
+}
+#endif
+static int fts_power_ctrl(void *data, bool on)
+{
+ struct fts_ts_info *info = (struct fts_ts_info *)data;
+ const struct fts_i2c_platform_data *pdata = info->board;
+ struct device *dev = &info->client->dev;
+ struct regulator *regulator_dvdd = NULL;
+ struct regulator *regulator_avdd = NULL;
+ static bool enabled;
+ int retval = 0;
+
+ if (enabled == on)
+ return retval;
+
+ /* touch power init */
+ if (gpio_is_valid(pdata->vdd_gpio)) {
+ gpio_request(pdata->vdd_gpio, "touch-vdd");
+ } else {
+ regulator_avdd = regulator_get(NULL, pdata->regulator_avdd);
+ if (IS_ERR_OR_NULL(regulator_avdd)) {
+ tsp_debug_err(dev, "%s: "
+ "Failed to get %s regulator.\n",
+ __func__, pdata->regulator_avdd);
+ goto out;
+ }
+ }
+ if (gpio_is_valid(pdata->vio_gpio)) {
+ gpio_request(pdata->vio_gpio, "touch-vio");
+ } else {
+ regulator_dvdd = regulator_get(NULL, pdata->regulator_dvdd);
+ if (IS_ERR_OR_NULL(regulator_dvdd)) {
+ tsp_debug_err(dev, "%s: "
+ "Failed to get %s regulator.\n",
+ __func__, pdata->regulator_dvdd);
+ goto out;
+ }
+ }
+
+ tsp_debug_info(dev, "%s: %s\n", __func__, on ? "on" : "off");
+
+ if (on) {
+ if (gpio_is_valid(pdata->vdd_gpio)) {
+ retval = gpio_direction_output(pdata->vdd_gpio, 1);
+ if (retval) {
+ tsp_debug_err(dev, "%s: "
+ "Failed to enable vdd: %d\n",
+ __func__, retval);
+ }
+ } else if (!IS_ERR_OR_NULL(regulator_avdd)) {
+ retval = regulator_enable(regulator_avdd);
+ if (retval) {
+ tsp_debug_err(dev, "%s: "
+ "Failed to enable avdd: %d\n",
+ __func__, retval);
+ goto out;
+ }
+ }
+
+ if (gpio_is_valid(pdata->vio_gpio)) {
+ retval = gpio_direction_output(pdata->vio_gpio, 1);
+ if (retval) {
+ tsp_debug_err(dev, "%s: "
+ "Failed to enable vio: %d\n",
+ __func__, retval);
+ }
+ } else if (!IS_ERR_OR_NULL(regulator_dvdd)) {
+ retval = regulator_enable(regulator_dvdd);
+ if (retval) {
+ tsp_debug_err(dev, "%s: "
+ "Failed to enable dvdd: %d\n",
+ __func__, retval);
+ goto out;
+ }
+ }
+
+ retval = pinctrl_select_state(pdata->pinctrl, pdata->pins_default);
+ if (retval < 0)
+ tsp_debug_err(dev, "%s: "
+ "Failed to configure tsp_attn pin\n",
+ __func__);
+
+ fts_delay(5);
+ } else {
+ if (gpio_is_valid(pdata->vio_gpio)) {
+ retval = gpio_direction_output(pdata->vio_gpio, 0);
+ if (retval) {
+ tsp_debug_err(dev, "%s: "
+ "Failed to enable vio: %d\n",
+ __func__, retval);
+ }
+ } else if (!IS_ERR_OR_NULL(regulator_dvdd)) {
+ retval = regulator_disable(regulator_dvdd);
+ if (retval) {
+ tsp_debug_err(dev, "%s: "
+ "Failed to enable dvdd: %d\n",
+ __func__, retval);
+ goto out;
+ }
+ }
+
+ if (gpio_is_valid(pdata->vdd_gpio)) {
+ retval = gpio_direction_output(pdata->vdd_gpio, 0);
+ if (retval)
+ tsp_debug_err(dev, "%s: "
+ "Failed to enable vdd: %d\n",
+ __func__, retval);
+ } else if (!IS_ERR_OR_NULL(regulator_avdd)) {
+ retval = regulator_disable(regulator_avdd);
+ if (retval) {
+ tsp_debug_err(dev, "%s: "
+ "Failed to enable avdd: %d\n",
+ __func__, retval);
+ goto out;
+ }
+ }
+
+ retval = pinctrl_select_state(pdata->pinctrl, pdata->pins_sleep);
+ if (retval < 0)
+ tsp_debug_err(dev, "%s: Failed to "
+ "configure tsp_attn pin\n", __func__);
+ }
+
+ enabled = on;
+
+out:
+ regulator_put(regulator_dvdd);
+ regulator_put(regulator_avdd);
+
+ return retval;
+}
+
+static int fts_parse_dt(struct i2c_client *client)
+{
+ struct device *dev = &client->dev;
+ struct fts_i2c_platform_data *pdata = dev->platform_data;
+ struct device_node *np = dev->of_node;
+ u32 coords[2], lines[2];
+ int retval = 0;
+
+ pdata->tspid = of_get_named_gpio(np, "stm,tspid_gpio", 0);
+ if (gpio_is_valid(pdata->tspid))
+ tsp_debug_dbg(dev,
+ "TSP_ID : %d\n",
+ gpio_get_value(pdata->tspid));
+ else
+ tsp_debug_err(dev,
+ "Failed to get tspid gpio\n");
+
+ pdata->tspid2 = of_get_named_gpio(np, "stm,tspid2_gpio", 0);
+ if (gpio_is_valid(pdata->tspid2))
+ tsp_debug_dbg(dev,
+ "TSP_ID2 : %d\n",
+ gpio_get_value(pdata->tspid2));
+ else
+ tsp_debug_err(dev,
+ "Failed to get tspid2 gpio\n");
+
+ pdata->gpio = of_get_named_gpio(np, "stm,irq_gpio", 0);
+ if (gpio_is_valid(pdata->gpio)) {
+ retval = gpio_request_one(pdata->gpio,
+ GPIOF_DIR_IN, "stm,tsp_int");
+ if (retval) {
+ tsp_debug_err(dev,
+ "Unable to request tsp_int [%d]\n",
+ pdata->gpio);
+ return -EINVAL;
+ }
+ } else {
+ tsp_debug_err(dev,
+ "Failed to get irq gpio\n");
+ return -EINVAL;
+ }
+ client->irq = gpio_to_irq(pdata->gpio);
+
+ if (of_property_read_u32(np, "stm,irq_type", &pdata->irq_type)) {
+ tsp_debug_err(dev, "Failed to get irq_type property\n");
+ return -EINVAL;
+ }
+
+ if (of_property_read_u32(np, "stm,grip_area", &pdata->grip_area))
+ tsp_debug_err(dev, "Failed to get grip_area property\n");
+
+ if (of_property_read_u32_array(np, "stm,max_coords", coords, 2)) {
+ tsp_debug_err(dev, "Failed to get max_coords property\n");
+ return -EINVAL;
+ }
+ pdata->max_x = coords[0];
+ pdata->max_y = coords[1];
+
+ if (of_property_read_u32_array(np, "stm,num_lines", lines, 2))
+ tsp_debug_dbg(dev, "skipped to get num_lines property\n");
+ else {
+ pdata->SenseChannelLength = lines[0];
+ pdata->ForceChannelLength = lines[1];
+ tsp_debug_dbg(dev, "num_of[rx,tx]: [%d,%d]\n",
+ pdata->SenseChannelLength, pdata->ForceChannelLength);
+ }
+
+ if (of_property_read_string(np, "stm,regulator_dvdd", &pdata->regulator_dvdd)) {
+ tsp_debug_err(dev,
+ "Failed to get regulator_dvdd name property\n");
+ }
+ if (of_property_read_string(np, "stm,regulator_avdd", &pdata->regulator_avdd)) {
+ tsp_debug_err(dev,
+ "Failed to get regulator_avdd name property\n");
+ }
+
+ pdata->vdd_gpio = of_get_named_gpio(np, "stm,vdd-gpio", 0);
+ if (gpio_is_valid(pdata->vdd_gpio))
+ tsp_debug_dbg(dev, "vdd_gpio : %d\n",
+ gpio_get_value(pdata->vdd_gpio));
+ else
+ tsp_debug_err(dev, "Failed to get vdd_gpio gpio\n");
+
+ pdata->vio_gpio = of_get_named_gpio(np, "stm,vio-gpio", 0);
+ if (gpio_is_valid(pdata->vio_gpio))
+ tsp_debug_dbg(dev, "vio_gpio :"
+ "%d\n", gpio_get_value(pdata->vio_gpio));
+ else
+ tsp_debug_err(dev, "Failed to get vio_gpio gpio\n");
+
+ pdata->power = fts_power_ctrl;
+
+ /* Optional parmeters(those values are not mandatory)
+ * do not return error value even if fail to get the value
+ */
+ of_property_read_string(np, "stm,firmware_name", &pdata->firmware_name);
+
+ if (of_property_read_string_index(np, "stm,project_name", 0, &pdata->project_name))
+ tsp_debug_dbg(dev,
+ "skipped to get project_name property\n");
+ if (of_property_read_string_index(np, "stm,project_name", 1, &pdata->model_name))
+ tsp_debug_dbg(dev,
+ "skipped to get model_name property\n");
+
+ pdata->max_width = 28;
+ pdata->support_hover = true;
+ pdata->support_mshover = true;
+#ifdef FTS_SUPPORT_TA_MODE
+ pdata->register_cb = fts_tsp_register_callback;
+#endif
+
+ return retval;
+}
+#endif
+
+static int fts_setup_drv_data(struct i2c_client *client)
+{
+ int retval = 0;
+ struct fts_i2c_platform_data *pdata;
+ struct fts_ts_info *info = NULL;
+
+ /* parse dt */
+ if (client->dev.of_node) {
+ pdata = devm_kzalloc(&client->dev,
+ sizeof(struct fts_i2c_platform_data), GFP_KERNEL);
+
+ if (!pdata) {
+ tsp_debug_err(&client->dev, "Failed to allocate platform data\n");
+ return -ENOMEM;
+ }
+
+ client->dev.platform_data = pdata;
+ retval = fts_parse_dt(client);
+ if (retval) {
+ tsp_debug_err(&client->dev, "Failed to parse dt\n");
+ goto error;
+ }
+ } else {
+ pdata = client->dev.platform_data;
+ }
+
+ if (!pdata) {
+ tsp_debug_err(&client->dev, "No platform data found\n");
+ return -EINVAL;
+ }
+ if (!pdata->power) {
+ tsp_debug_err(&client->dev, "No power contorl found\n");
+ retval = -EINVAL;
+ goto error;
+ }
+
+ pdata->pinctrl = devm_pinctrl_get(&client->dev);
+ if (IS_ERR(pdata->pinctrl)) {
+ tsp_debug_err(&client->dev, "could not get pinctrl\n");
+ retval = PTR_ERR(pdata->pinctrl);
+ goto error;
+ }
+
+ pdata->pins_default = pinctrl_lookup_state(pdata->pinctrl, "on_state");
+ if (IS_ERR(pdata->pins_default))
+ tsp_debug_err(&client->dev, "could not get default pinstate\n");
+
+ pdata->pins_sleep = pinctrl_lookup_state(pdata->pinctrl, "off_state");
+ if (IS_ERR(pdata->pins_sleep))
+ tsp_debug_err(&client->dev, "could not get sleep pinstate\n");
+
+ info = kzalloc(sizeof(struct fts_ts_info), GFP_KERNEL);
+ if (!info) {
+ tsp_debug_err(&client->dev,
+ "%s: Failed to alloc mem for info\n",
+ __func__);
+ retval = -ENOMEM;
+ goto error;
+ }
+
+ info->client = client;
+ info->board = pdata;
+ info->irq = client->irq;
+ info->irq_type = info->board->irq_type;
+ atomic_set(&info->irq_enabled, 0);
+ info->touch_stopped = false;
+ info->panel_revision = info->board->panel_revision;
+ info->stop_device = fts_stop_device;
+ info->start_device = fts_start_device;
+ info->fts_command = fts_command;
+ info->fts_read_reg = fts_read_reg;
+ info->fts_write_reg = fts_write_reg;
+ info->fts_systemreset = fts_systemreset;
+ info->fts_get_version_info = fts_get_version_info;
+ info->fts_wait_for_ready = fts_wait_for_ready;
+#ifdef FEATURE_FTS_PRODUCTION_CODE
+ info->fts_get_channel_info = fts_get_channel_info;
+ info->fts_interrupt_set = fts_interrupt_set;
+ info->fts_irq_enable = fts_irq_enable;
+ info->fts_release_all_finger = fts_release_all_finger;
+#endif
+
+#ifdef FTS_SUPPORT_NOISE_PARAM
+ info->fts_get_noise_param_address = fts_get_noise_param_address;
+#endif
+
+#ifdef USE_OPEN_DWORK
+ INIT_DELAYED_WORK(&info->open_work, fts_open_work);
+#endif
+ info->delay_time = 300;
+ INIT_DELAYED_WORK(&info->reset_work, fts_reset_work);
+
+ if (info->board->support_hover)
+ tsp_debug_info(&info->client->dev, "FTS Support Hover Event\n");
+ else
+ tsp_debug_info(&info->client->dev, "FTS Not support Hover Event\n");
+
+ i2c_set_clientdata(client, info);
+
+ if (pdata->get_ddi_type) {
+ info->ddi_type = pdata->get_ddi_type();
+ tsp_debug_info(&client->dev,
+ "%s: DDI Type is %s[%d]\n",
+ __func__, info->ddi_type ?
+ "MAGNA" : "SDC", info->ddi_type);
+ }
+
+ pdata->switch_gpio = of_get_named_gpio(client->dev.of_node,
+ "stm,switch_gpio", 0);
+
+ if (gpio_is_valid(pdata->switch_gpio)) {
+ retval = gpio_request_one(pdata->switch_gpio,
+ GPIOF_OUT_INIT_LOW,
+ "stm,tsp_i2c_switch");
+ if (retval) {
+ tsp_debug_err(&client->dev,
+ "Can't req GPIO%d: rev %d\n",
+ pdata->switch_gpio, retval);
+ goto error;
+ }
+ } else {
+ tsp_debug_info(&client->dev, "Didn't find switch_gpio = %d\n",
+ pdata->switch_gpio);
+ }
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+ tui_tsp_info = info;
+#endif
+ return 0;
+error:
+ kfree(info);
+ if (client->dev.of_node) {
+ kfree(pdata);
+ client->dev.platform_data = NULL;
+ }
+ return retval;
+}
+
+static int fts_probe(struct i2c_client *client, const struct i2c_device_id *idp)
+{
+ int retval = 0;
+ struct fts_ts_info *info = NULL;
+ static char fts_ts_phys[64] = { 0 };
+ int i = 0;
+
+/*
+ tsp_debug_info(&client->dev, "FTS Driver [12%s] %s %s\n",
+ FTS_TS_DRV_VERSION, __DATE__, __TIME__);
+*/
+
+ if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
+ tsp_debug_err(&client->dev, "FTS err = EIO!\n");
+ return -EIO;
+ }
+
+ /* Build up driver data */
+ retval = fts_setup_drv_data(client);
+ if (retval < 0) {
+ tsp_debug_err(&client->dev,
+ "%s: Failed to set up driver data\n",
+ __func__);
+ goto err_setup_drv_data;
+ }
+
+ info = (struct fts_ts_info *)i2c_get_clientdata(client);
+ if (!info) {
+ tsp_debug_err(&client->dev,
+ "%s: Failed to get driver data\n", __func__);
+ retval = -ENODEV;
+ goto err_get_drv_data;
+ }
+
+ if (info->board->power)
+ info->board->power(info, true);
+
+ info->dev = &info->client->dev;
+ info->input_dev = input_allocate_device();
+ if (!info->input_dev) {
+ tsp_debug_err(&info->client->dev, "FTS err = ENOMEM!\n");
+ retval = -ENOMEM;
+ goto err_input_allocate_device;
+ }
+
+ info->input_dev->dev.parent = &client->dev;
+ info->input_dev->name = "touchscreen";
+ snprintf(fts_ts_phys, sizeof(fts_ts_phys), "%s/input1",
+ info->input_dev->name);
+ info->input_dev->phys = fts_ts_phys;
+ info->input_dev->id.bustype = BUS_I2C;
+
+#ifdef USE_OPEN_CLOSE
+ info->input_dev->open = fts_input_open;
+ info->input_dev->close = fts_input_close;
+#endif
+
+ set_bit(EV_SYN, info->input_dev->evbit);
+ set_bit(EV_KEY, info->input_dev->evbit);
+ set_bit(EV_ABS, info->input_dev->evbit);
+#ifdef INPUT_PROP_DIRECT
+ set_bit(INPUT_PROP_DIRECT, info->input_dev->propbit);
+#endif
+ set_bit(BTN_TOUCH, info->input_dev->keybit);
+ set_bit(BTN_TOOL_FINGER, info->input_dev->keybit);
+
+ input_mt_init_slots(info->input_dev, FINGER_MAX, INPUT_MT_DIRECT);
+ input_set_abs_params(info->input_dev, ABS_MT_POSITION_X,
+ 0, info->board->max_x, 0, 0);
+ input_set_abs_params(info->input_dev, ABS_MT_POSITION_Y,
+ 0, info->board->max_y, 0, 0);
+ input_set_abs_params(info->input_dev, ABS_MT_PRESSURE,
+ 0, 255, 0, 0);
+
+ mutex_init(&info->device_mutex);
+ mutex_init(&info->i2c_mutex);
+ spin_lock_init(&info->lock);
+
+ info->enabled = false;
+ mutex_lock(&info->device_mutex);
+ retval = fts_init(info);
+ info->reinit_done = true;
+ mutex_unlock(&info->device_mutex);
+ if (retval < 0) {
+ tsp_debug_err(&info->client->dev, "FTS fts_init fail!\n");
+ goto err_fts_init;
+ }
+
+ input_set_abs_params(info->input_dev, ABS_MT_TOUCH_MAJOR,
+ 0, 255, 0, 0);
+ input_set_abs_params(info->input_dev, ABS_MT_TOUCH_MINOR,
+ 0, 255, 0, 0);
+ input_set_abs_params(info->input_dev, ABS_MT_DISTANCE,
+ 0, 255, 0, 0);
+ input_set_abs_params(info->input_dev, ABS_MT_ORIENTATION,
+ 0, 255, 0, 0);
+
+ input_set_drvdata(info->input_dev, info);
+ i2c_set_clientdata(client, info);
+
+ retval = input_register_device(info->input_dev);
+ if (retval) {
+ tsp_debug_err(&info->client->dev, "FTS input_register_device fail!\n");
+ goto err_register_input;
+ }
+
+ for (i = 0; i < FINGER_MAX; i++) {
+ info->finger[i].state = EVENTID_LEAVE_POINTER;
+ info->finger[i].mcount = 0;
+ }
+
+ info->enabled = true;
+
+ retval = request_threaded_irq(info->irq, NULL,
+ fts_interrupt_handler, info->board->irq_type,
+ FTS_TS_DRV_NAME, info);
+
+ if (retval < 0) {
+ tsp_debug_err(&info->client->dev,
+ "%s: Failed to enable attention interrupt\n",
+ __func__);
+ goto err_enable_irq;
+ }
+ atomic_set(&info->irq_enabled, 1);
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+ trustedui_set_tsp_irq(info->irq);
+ tsp_debug_info(&client->dev, "%s[%d] called!\n",
+ __func__, info->irq);
+#endif
+
+#if defined(CONFIG_FB)
+ info->fb_notif.notifier_call = touch_fb_notifier_callback;
+ retval = fb_register_client(&info->fb_notif);
+#endif
+
+
+#ifdef FTS_SUPPORT_TA_MODE
+ info->register_cb = info->board->register_cb;
+
+ info->callbacks.inform_charger = fts_ta_cb;
+ if (info->register_cb)
+ info->register_cb(&info->callbacks);
+#endif
+
+#ifdef FEATURE_FTS_PRODUCTION_CODE
+ fts_production_init(info);
+#endif /* FEATURE_FTS_PRODUCTION_CODE */
+ device_init_wakeup(&client->dev, true);
+ if (device_may_wakeup(&info->client->dev))
+ enable_irq_wake(info->irq);
+ info->lowpower_mode = true;
+
+ return 0;
+
+err_enable_irq:
+ input_unregister_device(info->input_dev);
+ info->input_dev = NULL;
+
+err_register_input:
+ if (info->input_dev)
+ input_free_device(info->input_dev);
+
+err_fts_init:
+ mutex_destroy(&info->device_mutex);
+ mutex_destroy(&info->i2c_mutex);
+err_input_allocate_device:
+ info->board->power(info, false);
+ kfree(info);
+err_get_drv_data:
+err_setup_drv_data:
+ return retval;
+}
+
+static int fts_remove(struct i2c_client *client)
+{
+ struct fts_ts_info *info = i2c_get_clientdata(client);
+
+ tsp_debug_info(&info->client->dev, "FTS removed\n");
+
+#if defined(CONFIG_FB)
+ if (fb_unregister_client(&info->fb_notif))
+ tsp_debug_err(&info->client->dev,
+ "%s: Error occured while unregistering fb_notifier.\n", __func__);
+#endif
+
+ fts_interrupt_set(info, INT_DISABLE);
+ fts_command(info, FLUSHBUFFER);
+
+ fts_irq_enable(info, false);
+ free_irq(info->irq, info);
+
+ input_mt_destroy_slots(info->input_dev);
+
+ input_unregister_device(info->input_dev);
+ info->input_dev = NULL;
+
+ info->board->power(info, false);
+
+ kfree(info);
+
+ return 0;
+}
+
+#ifdef USE_OPEN_CLOSE
+#ifdef USE_OPEN_DWORK
+static void fts_open_work(struct work_struct *work)
+{
+ int retval = 0;
+ struct fts_ts_info *info = container_of(work, struct fts_ts_info,
+ open_work.work);
+
+ tsp_debug_info(&info->client->dev, "%s\n", __func__);
+
+ retval = fts_start_device(info);
+ if (retval < 0)
+ tsp_debug_err(&info->client->dev,
+ "%s: Failed to start device\n", __func__);
+}
+#endif
+static int fts_input_open(struct input_dev *dev)
+{
+ struct fts_ts_info *info = input_get_drvdata(dev);
+ unsigned char regAdd[4] = {0xB0, 0x01, 0x29, 0x41};
+ int retval = 0;
+
+ tsp_debug_info(&info->client->dev, "%s\n", __func__);
+
+#ifdef USE_OPEN_DWORK
+ schedule_delayed_work(&info->open_work,
+ msecs_to_jiffies(TOUCH_OPEN_DWORK_TIME));
+#else
+ retval = fts_start_device(info);
+ if (retval < 0) {
+ tsp_debug_err(&info->client->dev,
+ "%s: Failed to start device\n", __func__);
+ goto out;
+ }
+#endif
+
+ tsp_debug_info(&info->client->dev,
+ "FTS cmd after wakeup : h%d\n",
+ info->retry_hover_enable_after_wakeup);
+
+ if (info->retry_hover_enable_after_wakeup == 1) {
+ fts_write_reg(info, ®Add[0], 4);
+ fts_command(info, FTS_CMD_HOVER_ON);
+ }
+
+out:
+ return 0;
+}
+
+static void fts_input_close(struct input_dev *dev)
+{
+ struct fts_ts_info *info = input_get_drvdata(dev);
+
+ tsp_debug_info(&info->client->dev, "%s\n", __func__);
+
+#ifdef USE_OPEN_DWORK
+ cancel_delayed_work(&info->open_work);
+#endif
+
+ fts_stop_device(info);
+
+ info->retry_hover_enable_after_wakeup = 0;
+}
+#endif
+
+static void fts_reinit(struct fts_ts_info *info)
+{
+ fts_systemreset(info);
+
+ fts_wait_for_ready(info);
+
+#ifdef FTS_SUPPORT_NOISE_PARAM
+ fts_set_noise_param(info);
+#endif
+
+ fts_command(info, SENSEON);
+ fts_delay(50);
+
+#ifdef FTS_SUPPORT_TA_MODE
+ if (info->TA_Pluged)
+ fts_command(info, FTS_CMD_CHARGER_PLUGGED);
+#endif
+
+ info->touch_count = 0;
+ info->palm_pressed = false;
+
+ fts_command(info, FLUSHBUFFER);
+ fts_interrupt_set(info, INT_ENABLE);
+}
+
+void fts_release_all_finger(struct fts_ts_info *info)
+{
+ int i;
+
+ for (i = 0; i < FINGER_MAX; i++) {
+ input_mt_slot(info->input_dev, i);
+ input_mt_report_slot_state(info->input_dev, MT_TOOL_FINGER, 0);
+
+ if ((info->finger[i].state == EVENTID_ENTER_POINTER) ||
+ (info->finger[i].state == EVENTID_MOTION_POINTER)) {
+ info->touch_count--;
+ if (info->touch_count < 0)
+ info->touch_count = 0;
+
+ tsp_debug_info(&info->client->dev,
+ "[RA] tID:%d mc: %d tc:%d Ver[%02X%04X%01X%01X]\n",
+ i, info->finger[i].mcount, info->touch_count,
+ info->panel_revision, info->fw_main_version_of_ic,
+ info->flip_enable, info->mainscr_disable);
+ }
+
+ info->finger[i].state = EVENTID_LEAVE_POINTER;
+ info->finger[i].mcount = 0;
+ }
+
+ input_report_key(info->input_dev, BTN_TOUCH, 0);
+ input_report_key(info->input_dev, BTN_TOOL_FINGER, 0);
+
+#ifdef CONFIG_INPUT_BOOSTER
+ input_booster_send_event(BOOSTER_DEVICE_TOUCH, BOOSTER_MODE_FORCE_OFF);
+#endif
+
+ input_sync(info->input_dev);
+}
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+void trustedui_mode_on(void)
+{
+ tsp_debug_info(&tui_tsp_info->client->dev,
+ "%s, release all finger..", __func__);
+ fts_release_all_finger(tui_tsp_info);
+}
+#endif
+
+static void fts_reset_work(struct work_struct *work)
+{
+ struct fts_ts_info *info = container_of(work, struct fts_ts_info,
+ reset_work.work);
+ bool temp_lpm;
+
+ temp_lpm = info->lowpower_mode;
+ /* Reset-routine must go to power off state */
+ info->lowpower_mode = 0;
+
+ tsp_debug_info(&info->client->dev, "%s, Call Power-Off to recover IC, lpm:%d\n", __func__, temp_lpm);
+ fts_stop_device(info);
+
+ fts_delay(100); /* Delay to discharge the IC from ESD or On-state.*/
+ if (fts_start_device(info) < 0)
+ tsp_debug_err(&info->client->dev, "%s: Failed to start device\n", __func__);
+
+ info->lowpower_mode = temp_lpm;
+}
+
+
+static int fts_stop_device(struct fts_ts_info *info)
+{
+ tsp_debug_dbg(&info->client->dev, "%s\n", __func__);
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+ if (TRUSTEDUI_MODE_TUI_SESSION & trustedui_get_current_mode()) {
+ tsp_debug_err(&info->client->dev,
+ "%s TUI cancel event call!\n", __func__);
+ fts_delay(100);
+ tui_force_close(1);
+ fts_delay(200);
+ if (TRUSTEDUI_MODE_TUI_SESSION & trustedui_get_current_mode()) {
+ tsp_debug_err(&info->client->dev,
+ "%s TUI flag force clear!\n", __func__);
+ trustedui_clear_mask(TRUSTEDUI_MODE_VIDEO_SECURED|
+ TRUSTEDUI_MODE_INPUT_SECURED);
+ trustedui_set_mode(TRUSTEDUI_MODE_OFF);
+ }
+ }
+#endif
+
+ mutex_lock(&info->device_mutex);
+
+ if (info->touch_stopped) {
+ tsp_debug_err(&info->client->dev,
+ "%s already power off\n", __func__);
+ goto out;
+ }
+
+ if (info->lowpower_mode) {
+ tsp_debug_info(&info->client->dev,
+ "%s lowpower flag:%d\n",
+ __func__, info->lowpower_flag);
+
+ info->fts_power_state = FTS_POWER_STATE_LOWPOWER;
+
+ fts_command(info, FLUSHBUFFER);
+
+ fts_command(info, FTS_CMD_LOWPOWER_MODE);
+
+ if (device_may_wakeup(&info->client->dev))
+ enable_irq_wake(info->irq);
+
+ fts_command(info, FLUSHBUFFER);
+
+ fts_release_all_finger(info);
+#ifdef FTS_SUPPORT_NOISE_PARAM
+ fts_get_noise_param(info);
+#endif
+
+ } else {
+ fts_interrupt_set(info, INT_DISABLE);
+ fts_irq_enable(info, false);
+
+ fts_command(info, FLUSHBUFFER);
+ fts_release_all_finger(info);
+#ifdef FTS_SUPPORT_NOISE_PARAM
+ fts_get_noise_param(info);
+#endif
+ info->touch_stopped = true;
+
+ if (info->board->power)
+ info->board->power(info, false);
+
+ info->fts_power_state = FTS_POWER_STATE_POWERDOWN;
+ }
+ out:
+ mutex_unlock(&info->device_mutex);
+ return 0;
+}
+
+static int fts_start_device(struct fts_ts_info *info)
+{
+ tsp_debug_dbg(&info->client->dev, "%s %s\n",
+ __func__,
+ info->lowpower_mode ?
+ "exit low power mode" : "");
+
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+ if (TRUSTEDUI_MODE_TUI_SESSION & trustedui_get_current_mode()) {
+ tsp_debug_err(&info->client->dev,
+ "%s TUI cancel event call!\n", __func__);
+ fts_delay(100);
+ tui_force_close(1);
+ fts_delay(200);
+ if (TRUSTEDUI_MODE_TUI_SESSION & trustedui_get_current_mode()) {
+ tsp_debug_err(&info->client->dev,
+ "%s TUI flag force clear!\n", __func__);
+ trustedui_clear_mask(TRUSTEDUI_MODE_VIDEO_SECURED|
+ TRUSTEDUI_MODE_INPUT_SECURED);
+ trustedui_set_mode(TRUSTEDUI_MODE_OFF);
+ }
+ }
+#endif
+
+ mutex_lock(&info->device_mutex);
+
+ if (!info->touch_stopped && !info->lowpower_mode) {
+ tsp_debug_err(&info->client->dev,
+ "%s already power on\n", __func__);
+ goto out;
+ }
+
+ fts_release_all_finger(info);
+ if (info->lowpower_mode) {
+ /* low power mode command is sent after LCD OFF. */
+ /* turn on touch power @ LCD ON */
+ if (info->touch_stopped)
+ goto tsp_power_on;
+
+ disable_irq(info->irq);
+
+ info->reinit_done = false;
+ fts_reinit(info);
+ info->reinit_done = true;
+
+ fts_irq_enable(info, true);
+
+ if (device_may_wakeup(&info->client->dev))
+ disable_irq_wake(info->irq);
+ } else {
+tsp_power_on:
+ if (info->board->power)
+ info->board->power(info, true);
+ info->touch_stopped = false;
+ info->reinit_done = false;
+
+ fts_reinit(info);
+ info->reinit_done = true;
+
+ fts_irq_enable(info, true);
+ }
+
+ out:
+ mutex_unlock(&info->device_mutex);
+
+ info->fts_power_state = FTS_POWER_STATE_ACTIVE;
+
+ return 0;
+}
+
+static void fts_shutdown(struct i2c_client *client)
+{
+ struct fts_ts_info *info = i2c_get_clientdata(client);
+
+ tsp_debug_info(&info->client->dev, "FTS %s called!\n", __func__);
+
+ if (info->lowpower_mode) {
+ info->lowpower_mode = 0;
+ tsp_debug_info(&info->client->dev, "FTS lowpower_mode off!\n");
+ }
+
+ fts_stop_device(info);
+}
+
+void fts_recovery_cx(struct fts_ts_info *info)
+{
+ unsigned char regAdd[4] = {0};
+ unsigned char buf[8] = {0};
+ unsigned char cnt = 100;
+ int ret = 0;
+
+ regAdd[0] = 0xB6;
+ regAdd[1] = 0x00;
+ regAdd[2] = 0x1E;
+ regAdd[3] = 0x08;
+ fts_write_reg(info, ®Add[0], 4); /* Loading FW to PRAM without CRC Check */
+ fts_delay(30);
+
+
+ fts_command(info, CX_TUNNING);
+ fts_delay(300);
+
+ fts_command(info, FTS_CMD_SAVE_CX_TUNING);
+ fts_delay(200);
+
+ do {
+ regAdd[0] = READ_ONE_EVENT;
+ ret = fts_read_reg(info, regAdd, 1, &buf[0], FTS_EVENT_SIZE);
+
+ fts_delay(10);
+
+ if (cnt-- == 0)
+ break;
+ } while (buf[0] != 0x16 || buf[1] != 0x04);
+
+ fts_command(info, SENSEON);
+ fts_delay(50);
+
+ fts_command(info, FLUSHBUFFER);
+}
+
+#ifdef CONFIG_PM
+static int fts_pm_suspend(struct device *dev)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ tsp_debug_info(&info->client->dev, "%s\n", __func__);
+
+ mutex_lock(&info->input_dev->mutex);
+
+ if (info->input_dev->users)
+ fts_stop_device(info);
+
+ mutex_unlock(&info->input_dev->mutex);
+
+ return 0;
+}
+
+static int fts_pm_resume(struct device *dev)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ tsp_debug_info(&info->client->dev, "%s\n", __func__);
+
+ mutex_lock(&info->input_dev->mutex);
+
+ if (info->input_dev->users)
+ fts_start_device(info);
+
+ mutex_unlock(&info->input_dev->mutex);
+
+ return 0;
+}
+#endif
+
+static int fts_suspend(struct i2c_client *client, pm_message_t mesg)
+{
+ struct fts_ts_info *info = i2c_get_clientdata(client);
+
+ tsp_debug_info(&info->client->dev, "%s power state : %d\n",
+ __func__, info->fts_power_state);
+
+ fts_stop_device(info);
+
+ return 0;
+}
+
+static int fts_resume(struct i2c_client *client)
+{
+ struct fts_ts_info *info = i2c_get_clientdata(client);
+
+ tsp_debug_info(&info->client->dev, "%s power state : %d\n",
+ __func__, info->fts_power_state);
+ /* if resume is called from active state, the i2c bus is not
+ * switched to AP, skipping resume routine */
+ if (info->fts_power_state == FTS_POWER_STATE_ACTIVE) {
+ tsp_debug_info(&info->client->dev,
+ "%s: calling resume from active state, "
+ "skipping\n", __func__);
+ return 0;
+ }
+
+ fts_start_device(info);
+
+ return 0;
+}
+
+#if defined(CONFIG_FB)
+static int touch_fb_notifier_callback(struct notifier_block *self,
+ unsigned long event, void *data)
+{
+ struct fts_ts_info *info =
+ container_of(self, struct fts_ts_info, fb_notif);
+ struct fb_event *ev = (struct fb_event *)data;
+
+ if (ev && ev->data && event == FB_EVENT_BLANK) {
+ int *blank = (int *)ev->data;
+
+ if (*blank == FB_BLANK_UNBLANK)
+ fts_resume(info->client);
+ else if (*blank == FB_BLANK_POWERDOWN)
+ fts_suspend(info->client, PMSG_SUSPEND);
+ }
+
+ return 0;
+}
+#endif
+
+static const struct i2c_device_id fts_device_id[] = {
+ {FTS_TS_DRV_NAME, 0},
+ {}
+};
+
+#ifdef CONFIG_PM
+static const struct dev_pm_ops fts_dev_pm_ops = {
+ .suspend = fts_pm_suspend,
+ .resume = fts_pm_resume,
+};
+#endif
+
+#ifdef CONFIG_OF
+static struct of_device_id fts_match_table[] = {
+ { .compatible = "stm,ftm4_fts",},
+ { },
+};
+#else
+#define fts_match_table NULL
+#endif
+
+static struct i2c_driver fts_i2c_driver = {
+ .driver = {
+ .name = FTS_TS_DRV_NAME,
+ .owner = THIS_MODULE,
+#ifdef CONFIG_OF
+ .of_match_table = fts_match_table,
+#endif
+#ifdef CONFIG_PM
+ .pm = &fts_dev_pm_ops,
+#endif
+ },
+ .probe = fts_probe,
+ .remove = fts_remove,
+ .shutdown = fts_shutdown,
+#if (!defined(CONFIG_FB))
+ .suspend = fts_suspend,
+ .resume = fts_resume,
+#endif
+ .id_table = fts_device_id,
+};
+
+static int __init fts_driver_init(void)
+{
+ return i2c_add_driver(&fts_i2c_driver);
+}
+
+static void __exit fts_driver_exit(void)
+{
+ i2c_del_driver(&fts_i2c_driver);
+}
+
+MODULE_DESCRIPTION("STMicroelectronics MultiTouch IC Driver");
+MODULE_AUTHOR("STMicroelectronics, Inc.");
+MODULE_LICENSE("GPL v2");
+
+module_init(fts_driver_init);
+module_exit(fts_driver_exit);
diff --git a/drivers/input/touchscreen/stm/ftm4_ts.h b/drivers/input/touchscreen/stm/ftm4_ts.h
new file mode 100644
index 0000000..fbb3311
--- /dev/null
+++ b/drivers/input/touchscreen/stm/ftm4_ts.h
@@ -0,0 +1,568 @@
+#ifndef _LINUX_FTM4_TS_H_
+#define _LINUX_FTM4_TS_H_
+
+#include <linux/device.h>
+#include <linux/hrtimer.h>
+#include <linux/i2c/fts.h>
+#if defined(CONFIG_FB)
+#include <linux/notifier.h>
+#include <linux/fb.h>
+#endif
+#ifdef CONFIG_INPUT_BOOSTER
+#include <linux/input/input_booster.h>
+#endif
+#include <linux/atomic.h>
+
+#include <linux/printk.h>
+#define tsp_debug_dbg(dev, fmt, ...) dev_dbg(dev, fmt, ## __VA_ARGS__)
+#define tsp_debug_info(dev, fmt, ...) dev_info(dev, fmt, ## __VA_ARGS__)
+#define tsp_debug_err(dev, fmt, ...) dev_err(dev, fmt, ## __VA_ARGS__)
+#ifdef CONFIG_TOUCHSCREEN_FTM4_SHOW_EVENTS
+#define tsp_debug_event(dev, fmt, ...) dev_dbg(dev, fmt, ## __VA_ARGS__)
+#else
+#define tsp_debug_event(dev, fmt, ...)
+#endif
+
+#define USE_OPEN_CLOSE
+#define FEATURE_FTS_PRODUCTION_CODE 1
+
+#ifdef USE_OPEN_DWORK
+#define TOUCH_OPEN_DWORK_TIME 10
+#endif
+
+#define FIRMWARE_IC "fts_ic"
+
+#define FTS_MAX_FW_PATH 64
+
+#define FTS_TS_DRV_NAME "fts_touch"
+#define FTS_TS_DRV_VERSION "0132"
+
+#define STM_DEVICE_NAME "STM"
+
+#define FTS_ID0 0x36
+#define FTS_ID1 0x70
+
+#define FTS_SEC_IX1_TX_MULTIPLIER (4)
+#define FTS_SEC_IX1_RX_MULTIPLIER (2)
+
+#define FTS_DIGITAL_REV_1 0x01
+#define FTS_DIGITAL_REV_2 0x02
+#define FTS_FIFO_MAX 32
+#define FTS_EVENT_SIZE 8
+#define FTS_FW_UPDATE_RETRY 3
+#define FTS_LOCKDOWNCODE_SIZE 13
+
+#define PRESSURE_MIN 0
+#define PRESSURE_MAX 127
+#define P70_PATCH_ADDR_START 0x00420000
+#define FINGER_MAX 10
+#define AREA_MIN PRESSURE_MIN
+#define AREA_MAX PRESSURE_MAX
+
+#define EVENTID_NO_EVENT 0x00
+#define EVENTID_ENTER_POINTER 0x03
+#define EVENTID_LEAVE_POINTER 0x04
+#define EVENTID_MOTION_POINTER 0x05
+#define EVENTID_HOVER_ENTER_POINTER 0x07
+#define EVENTID_HOVER_LEAVE_POINTER 0x08
+#define EVENTID_HOVER_MOTION_POINTER 0x09
+#define EVENTID_PROXIMITY_IN 0x0B
+#define EVENTID_PROXIMITY_OUT 0x0C
+#define EVENTID_MSKEY 0x0E
+#define EVENTID_ERROR 0x0F
+#define EVENTID_CONTROLLER_READY 0x10
+#define EVENTID_SLEEPOUT_CONTROLLER_READY 0x11
+#define EVENTID_RESULT_READ_REGISTER 0x12
+#define EVENTID_STATUS_REQUEST_COMP 0x13
+
+#define EVENTID_STATUS_EVENT 0x16
+#define EVENTID_INTERNAL_RELEASE_INFO 0x14
+#define EVENTID_EXTERNAL_RELEASE_INFO 0x15
+
+#define EVENTID_FROM_STRING 0x80
+#define EVENTID_GESTURE 0x20
+
+#define EVENTID_SIDE_SCROLL 0x40
+/* side touch event-id for debug, remove after f/w fixed */
+#define EVENTID_SIDE_TOUCH_DEBUG 0xDB
+#define EVENTID_SIDE_TOUCH 0x0B
+
+#define EVENTID_ERROR_FLASH_CORRUPTION 0x03
+
+/* define flash corruption type */
+#define EVENTID_ERROR_CONFIG_FLASH_CORRUPTION_1 0x01
+#define EVENTID_ERROR_CONFIG_FLASH_CORRUPTION_2 0x02
+#define EVENTID_ERROR_CX_FLASH_CORRUPTION 0x03
+
+#define EVENTID_LOCKDOWN_CODE 0x1E
+#define EVENTID_ERROR_LOCKDOWN 0x0B
+
+#define STATUS_EVENT_MUTUAL_AUTOTUNE_DONE 0x01
+#define STATUS_EVENT_SELF_AUTOTUNE_DONE 0x02
+#define STATUS_EVENT_WATER_SELF_AUTOTUNE_DONE 0x4E
+#ifdef FTS_SUPPORT_WATER_MODE
+#define STATUS_EVENT_WATER_SELF_DONE 0x17
+#endif
+#define STATUS_EVENT_FLASH_WRITE_CONFIG 0x03
+#define STATUS_EVENT_FLASH_WRITE_CXTUNE_VALUE 0x04
+#define STATUS_EVENT_FORCE_CAL_MUTUAL_SELF 0x05
+#define STATUS_EVENT_FORCE_CAL_DONE 0x06
+
+#define STATUS_EVENT_FORCE_CAL_MUTUAL 0x15
+#define STATUS_EVENT_FORCE_CAL_SELF 0x06
+#define STATUS_EVENT_PARAM1_FCAL_MS_SS_DONE 0x23
+#define STATUS_EVENT_WATERMODE_ON 0x07
+#define STATUS_EVENT_WATERMODE_OFF 0x08
+#define STATUS_EVENT_RTUNE_MUTUAL 0x09
+#define STATUS_EVENT_RTUNE_SELF 0x0A
+#define STATUS_EVENT_PANEL_TEST_RESULT 0x0B
+#define STATUS_EVENT_GLOVE_MODE 0x0C
+#define STATUS_EVENT_RAW_DATA_READY 0x0D
+#define STATUS_EVENT_MUTUAL_CAL_FRAME_CHECK 0xC1
+#define STATUS_EVENT_SELF_CAL_FRAME_CHECK 0xC2
+#define STATUS_EVENT_CHARGER_CONNECTED 0xCC
+#define STATUS_EVENT_CHARGER_DISCONNECTED 0xCD
+#define STATUS_EVENT_PURE_AUTOTUNE_FLAG_WRITE_FINISH 0x10
+#define STATUS_EVENT_PURE_AUTOTUNE_FLAG_CLEAR_FINISH 0x11
+
+#define INT_ENABLE 0x48
+#define INT_DISABLE 0x08
+
+#define READ_STATUS 0x84
+#define READ_ONE_EVENT 0x85
+#define READ_ALL_EVENT 0x86
+
+#define SENSEOFF 0x92
+#define SENSEON 0x93
+#define FTS_CMD_HOVER_OFF 0x94
+#define FTS_CMD_HOVER_ON 0x95
+
+#define FTS_CMD_MSKEY_AUTOTUNE 0x96
+#define FTS_CMD_TRIM_LOW_POWER_OSCILLATOR 0x97
+
+#define FTS_CMD_KEY_SENSE_OFF 0x9A
+#define FTS_CMD_KEY_SENSE_ON 0x9B
+#define FTS_CMD_SET_FAST_GLOVE_MODE 0x9D
+
+#define FTS_CMD_MSHOVER_OFF 0x9E
+#define FTS_CMD_MSHOVER_ON 0x9F
+#define FTS_CMD_SET_NOR_GLOVE_MODE 0x9F
+
+#define FLUSHBUFFER 0xA1
+#define FORCECALIBRATION 0xA2
+#define CX_TUNNING 0xA3
+#define SELF_AUTO_TUNE 0xA4
+
+#define FTS_CMD_CHARGER_PLUGGED 0xA8
+#define FTS_CMD_CHARGER_UNPLUGGED 0xAB
+
+#define FTS_CMD_RELEASEINFO 0xAA
+#define FTS_CMD_STYLUS_OFF 0xAB
+#define FTS_CMD_STYLUS_ON 0xAC
+#define FTS_CMD_LOWPOWER_MODE 0xAD
+
+#define FTS_CMS_ENABLE_FEATURE 0xC1
+#define FTS_CMS_DISABLE_FEATURE 0xC2
+
+#define LOCKDOWN_READ 0xC4
+
+#define FTS_CMD_WRITE_PRAM 0xF0
+#define FTS_CMD_BURN_PROG_FLASH 0xF2
+#define FTS_CMD_ERASE_PROG_FLASH 0xF3
+#define FTS_CMD_READ_FLASH_STAT 0xF4
+#define FTS_CMD_UNLOCK_FLASH 0xF7
+#define FTS_CMD_SAVE_FWCONFIG 0xFB
+#define FTS_CMD_SAVE_CX_TUNING 0xFC
+
+#define FTS_CMD_FAST_SCAN 0x01
+#define FTS_CMD_SLOW_SCAN 0x02
+#define FTS_CMD_USLOW_SCAN 0x03
+
+#define REPORT_RATE_90HZ 0
+#define REPORT_RATE_60HZ 1
+#define REPORT_RATE_30HZ 2
+
+#define FTS_CMD_STRING_ACCESS 0xEC00
+#define FTS_CMD_NOTIFY 0xC0
+
+#define FTS_RETRY_COUNT 10
+
+/* QUICK SHOT : Quick Camera Launching */
+#define FTS_STRING_EVENT_REAR_CAM (1 << 0)
+#define FTS_STRING_EVENT_FRONT_CAM (1 << 1)
+
+/* SCRUB : Display Watch, Event Status / Fast Access Event */
+#define FTS_STRING_EVENT_WATCH_STATUS (1 << 2)
+#define FTS_STRING_EVENT_FAST_ACCESS (1 << 3)
+#define FTS_STRING_EVENT_DIRECT_INDICATOR ((1 << 3) | (1 << 2))
+#define FTS_STRING_EVENT_SPAY (1 << 4)
+#define FTS_STRING_EVENT_SPAY1 (1 << 5)
+#define FTS_STRING_EVENT_SPAY2 ((1 << 4) | (1 << 5))
+
+#define FTS_SIDEGESTURE_EVENT_SINGLE_STROKE 0xE0
+#define FTS_SIDEGESTURE_EVENT_DOUBLE_STROKE 0xE1
+#define FTS_SIDEGESTURE_EVENT_INNER_STROKE 0xE3
+
+#define FTS_SIDETOUCH_EVENT_LONG_PRESS 0xBB
+#define FTS_SIDETOUCH_EVENT_REBOOT_BY_ESD 0xED
+
+#define FTS_ENABLE 1
+#define FTS_DISABLE 0
+
+#define FTS_MODE_QUICK_SHOT (1 << 0)
+#define FTS_MODE_SCRUB (1 << 1)
+#define FTS_MODE_SPAY (1 << 1)
+#define FTS_MODE_QUICK_APP_ACCESS (1 << 2)
+#define FTS_MODE_DIRECT_INDICATOR (1 << 3)
+
+#define TSP_BUF_SIZE 2048
+#define CMD_STR_LEN 32
+#define CMD_RESULT_STR_LEN 2048
+#define CMD_PARAM_NUM 8
+
+#define FTS_LOWP_FLAG_QUICK_CAM (1 << 0)
+#define FTS_LOWP_FLAG_2ND_SCREEN (1 << 1)
+#define FTS_LOWP_FLAG_BLACK_UI (1 << 2)
+#define FTS_LOWP_FLAG_QUICK_APP_ACCESS (1 << 3)
+#define FTS_LOWP_FLAG_DIRECT_INDICATOR (1 << 4)
+#define FTS_LOWP_FLAG_SPAY (1 << 5)
+#define FTS_LOWP_FLAG_TEMP_CMD (1 << 6)
+
+enum fts_error_return {
+ FTS_NOT_ERROR = 0,
+ FTS_ERROR_INVALID_CHIP_ID,
+ FTS_ERROR_INVALID_CHIP_VERSION_ID,
+ FTS_ERROR_INVALID_SW_VERSION,
+ FTS_ERROR_EVENT_ID,
+ FTS_ERROR_TIMEOUT,
+ FTS_ERROR_FW_UPDATE_FAIL,
+};
+#define RAW_MAX 3750
+/**
+ * struct fts_finger - Represents fingers.
+ * @ state: finger status (Event ID).
+ * @ mcount: moving counter for debug.
+ */
+struct fts_finger {
+ unsigned char state;
+ unsigned short mcount;
+ int lx;
+ int ly;
+};
+
+enum tsp_power_mode {
+ FTS_POWER_STATE_ACTIVE = 0,
+ FTS_POWER_STATE_LOWPOWER,
+ FTS_POWER_STATE_POWERDOWN,
+ FTS_POWER_STATE_DEEPSLEEP,
+};
+
+enum fts_cover_id {
+ FTS_FLIP_WALLET = 0,
+ FTS_VIEW_COVER,
+ FTS_COVER_NOTHING1,
+ FTS_VIEW_WIRELESS,
+ FTS_COVER_NOTHING2,
+ FTS_CHARGER_COVER,
+ FTS_VIEW_WALLET,
+ FTS_LED_COVER,
+ FTS_CLEAR_FLIP_COVER,
+ FTS_QWERTY_KEYBOARD_EUR,
+ FTS_QWERTY_KEYBOARD_KOR,
+ FTS_MONTBLANC_COVER = 100,
+};
+
+enum fts_customer_feature {
+ FTS_FEATURE_ORIENTATION_GESTURE = 1,
+ FTS_FEATURE_STYLUS,
+ FTS_FEATURE_QUICK_SHORT_CAMERA_ACCESS,
+ FTS_FEATURE_SIDE_GUSTURE,
+ FTS_FEATURE_COVER_GLASS,
+ FTS_FEATURE_COVER_WALLET,
+ FTS_FEATURE_COVER_LED,
+ FTS_FEATURE_COVER_CLEAR_FLIP,
+ FTS_FEATURE_DUAL_SIDE_GUSTURE,
+ FTS_FEATURE_CUSTOM_COVER_GLASS_ON,
+};
+
+enum ftsito_error_type {
+ NO_ERROR = 0,
+ ITO_FORCE_OPEN,
+ ITO_SENSE_OPEN,
+ ITO_FORCE_SHRT_GND,
+ ITO_SENSE_SHRT_GND,
+ ITO_FORCE_SHRT_VCM,
+ ITO_SENSE_SHRT_VCM,
+ ITO_FORCE_SHRT_FORCE,
+ ITO_SENSE_SHRT_SENSE,
+ ITO_F2E_SENSE,
+ ITO_FPC_FORCE_OPEN,
+ ITO_FPC_SENSE_OPEN,
+ ITO_KEY_FORCE_OPEN,
+ ITO_KEY_SENSE_OPEN,
+ ITO_RESERVED0,
+ ITO_RESERVED1,
+ ITO_RESERVED2,
+ ITO_MAX_ERR_REACHED = 0xFF
+};
+
+struct fts_version {
+ u8 build: 4;
+ u8 major: 4;
+ u8 minor;
+};
+
+struct fts_prd_info {
+ u8 product_id[3];
+ u8 chip_rev:4;
+ u8 fpc_rev:4;
+ u8 t_sensor_rev;
+ u8 site;
+ u8 inspector_no;
+ u8 date[6];
+};
+
+struct fts_flash_corruption_info {
+ bool fw_broken;
+ bool cfg_broken;
+ bool cx_broken;
+};
+
+struct fts_ts_info {
+ struct device *dev;
+ struct i2c_client *client;
+ struct input_dev *input_dev;
+ struct hrtimer timer;
+ struct timer_list timer_charger;
+ struct timer_list timer_firmware;
+ struct work_struct work;
+
+ int irq;
+ int irq_type;
+ atomic_t irq_enabled;
+ struct fts_i2c_platform_data *board;
+ void (*register_cb)(void *);
+ struct fts_callbacks callbacks;
+ bool enabled;
+#if defined(CONFIG_FB)
+ struct notifier_block fb_notif;
+#endif
+#ifdef FEATURE_FTS_PRODUCTION_CODE
+ struct device *pdc_dev_ts;
+ struct list_head cmd_list_head;
+ u8 cmd_state;
+ char cmd[CMD_STR_LEN];
+ int cmd_param[CMD_PARAM_NUM];
+ char cmd_result[CMD_RESULT_STR_LEN];
+ int cmd_buf_size;
+ struct mutex cmd_lock;
+ bool cmd_is_running;
+ int SenseChannelLength;
+ int ForceChannelLength;
+ short *pFrame;
+ unsigned char *cx_data;
+ struct delayed_work cover_cmd_work;
+ int delayed_cmd_param[2];
+#endif /* FEATURE_FTS_PRODUCTION_CODE */
+ bool flip_enable;
+ bool run_autotune;
+ bool mainscr_disable;
+ unsigned int cover_type;
+
+ unsigned char lowpower_flag;
+ bool lowpower_mode;
+ bool deepsleep_mode;
+ bool wirelesscharger_mode;
+ int fts_power_state;
+#ifdef FTS_SUPPORT_TA_MODE
+ bool TA_Pluged;
+#endif
+ int digital_rev;
+ int touch_count;
+ struct fts_finger finger[FINGER_MAX];
+ bool palm_pressed;
+
+ int touch_mode;
+ int retry_hover_enable_after_wakeup;
+
+ struct fts_prd_info prd_info;
+ int fw_version_of_ic; /* firmware version of IC */
+ int fw_version_of_bin; /* firmware version of binary */
+ int config_version_of_ic; /* Config release data from IC */
+ int config_version_of_bin; /* Config release data from IC */
+ unsigned short fw_main_version_of_ic; /* firmware main version of IC */
+ unsigned short fw_main_version_of_bin; /* firmware main version of binary */
+ int panel_revision; /* Octa panel revision */
+ int tspid_val;
+ int tspid2_val;
+
+#ifdef USE_OPEN_DWORK
+ struct delayed_work open_work;
+#endif
+
+#ifdef FTS_SUPPORT_NOISE_PARAM
+ struct fts_noise_param noise_param;
+ int (*fts_get_noise_param_address)(struct fts_ts_info *info);
+#endif
+ unsigned int delay_time;
+ unsigned int debug_string;
+ struct delayed_work reset_work;
+
+ unsigned int scrub_id;
+ unsigned int scrub_x;
+ unsigned int scrub_y;
+
+ struct mutex i2c_mutex;
+ struct mutex device_mutex;
+ spinlock_t lock;
+ bool touch_stopped;
+ bool reinit_done;
+
+ unsigned char data[FTS_EVENT_SIZE * FTS_FIFO_MAX];
+ unsigned char ddi_type;
+
+ char test_fwpath[256];
+ struct fts_version ic_fw_ver;
+
+ unsigned char o_afe_ver;
+ unsigned char afe_ver;
+
+ struct fts_flash_corruption_info flash_corruption_info;
+
+ unsigned int checksum_error;
+
+ int (*stop_device)(struct fts_ts_info *info);
+ int (*start_device)(struct fts_ts_info *info);
+
+ int (*fts_write_reg)(struct fts_ts_info *info, unsigned char *reg,
+ unsigned short num_com);
+ int (*fts_read_reg)(struct fts_ts_info *info, unsigned char *reg,
+ int cnum, unsigned char *buf, int num);
+ int (*fts_systemreset)(struct fts_ts_info *info);
+ int (*fts_wait_for_ready)(struct fts_ts_info *info);
+ void (*fts_command)(struct fts_ts_info *info, unsigned char cmd);
+ void (*fts_enable_feature)(struct fts_ts_info *info, unsigned char cmd,
+ int enable);
+#ifdef FEATURE_FTS_PRODUCTION_CODE
+ int (*fts_get_channel_info)(struct fts_ts_info *info);
+ int (*fts_get_version_info)(struct fts_ts_info *info);
+ void (*fts_interrupt_set)(struct fts_ts_info *info, int enable);
+ void (*fts_irq_enable)(struct fts_ts_info *info, bool enable);
+ void (*fts_release_all_finger)(struct fts_ts_info *info);
+#endif /* FEATURE_FTS_PRODUCTION_CODE */
+};
+
+#ifdef FEATURE_FTS_PRODUCTION_CODE
+#define FTS_CMD(name, func) .cmd_name = name, .cmd_func = func
+struct fts_cmd {
+ struct list_head list;
+ const char *cmd_name;
+ void (*cmd_func)(void *device_data);
+};
+
+extern struct fts_cmd fts_commands[];
+#endif /* FEATURE_FTS_PRODUCTION_CODE */
+
+#define WRITE_CHUNK_SIZE 32
+#define FLASH_CHUNK (64 * 1024)
+#define DMA_CHUNK 32
+
+#define FW_HEADER_SIZE 64
+#define FW_HEADER_FTB_SIGNATURE 0xAA55AA55
+#define FW_FTB_VER 0x00000001
+#define FW_BYTES_ALLIGN 4
+#define FW_BIN_VER_OFFSET 16
+#define FW_BIN_CONFIG_VER_OFFSET 20
+
+/* Command for flash */
+#define FLASH_CMD_UNLOCK 0xF7
+#define FLASH_CMD_WRITE_64K 0xF8
+#define FLASH_CMD_READ_REGISTER 0xF9
+#define FLASH_CMD_WRITE_REGISTER 0xFA
+
+/* Parameters for commands */
+#define ADDR_WARM_BOOT 0x001E
+#define WARM_BOOT_VALUE 0x38
+#define FLASH_ADDR_CODE 0x00000000
+#define FLASH_ADDR_CONFIG 0x0000FC00
+
+#define FLASH_UNLOCK_CODE0 0x74
+#define FLASH_UNLOCK_CODE1 0x45
+
+#define FLASH_ERASE_UNLOCK_CODE0 0x72
+#define FLASH_ERASE_UNLOCK_CODE1 0x03
+#define FLASH_ERASE_UNLOCK_CODE2 0x02
+#define FLASH_ERASE_CODE0 0x02
+#define FLASH_ERASE_CODE1 0xC0
+#define FLASH_DMA_CODE0 0x05
+#define FLASH_DMA_CODE1 0xC0
+#define FLASH_DMA_CONFIG 0x06
+
+enum binfile_type {
+ BIN_FTS128 = 1,
+ BIN_FTS256 = 2,
+ BIN_FTB = 3
+};
+
+struct FW_FTB_HEADER {
+ uint32_t signature;
+ uint32_t ftb_ver;
+ uint32_t target;
+ uint32_t fw_id;
+ uint32_t fw_ver;
+ uint32_t cfg_id;
+ uint32_t cfg_ver;
+ uint32_t reserved[2];
+ uint32_t bl_fw_ver;
+ uint32_t ext_ver;
+ uint32_t sec0_size;
+ uint32_t sec1_size;
+ uint32_t sec2_size;
+ uint32_t sec3_size;
+ uint32_t hdr_crc;
+};
+
+#ifdef FEATURE_FTS_PRODUCTION_CODE
+enum fts_system_information_address {
+ FTS_SI_FILTERED_RAW_ADDR = 0x02,
+ FTS_SI_STRENGTH_ADDR = 0x04,
+ FTS_SI_SELF_FILTERED_FORCE_RAW_ADDR = 0x1E,
+ FTS_SI_SELF_FILTERED_SENSE_RAW_ADDR = 0x20,
+ FTS_SI_NOISE_PARAM_ADDR = 0x40,
+ FTS_SI_PURE_AUTOTUNE_FLAG = 0x4E,
+ FTS_SI_COMPENSATION_OFFSET_ADDR = 0x50,
+ FTS_SI_PURE_AUTOTUNE_CONFIG = 0x52,
+ FTS_SI_FACTORY_RESULT_FLAG = 0x56,
+ FTS_SI_AUTOTUNE_CNT = 0x58,
+ FTS_SI_SENSE_CH_LENGTH = 0x5A, /* 2 bytes */
+ FTS_SI_FORCE_CH_LENGTH = 0x5C, /* 2 bytes */
+ FTS_SI_FINGER_THRESHOLD = 0x60, /* 2 bytes */
+ FTS_SI_AUTOTUNE_PROTECTION_CONFIG = 0x62, /* 2 bytes */
+ FTS_SI_REPORT_PRESSURE_RAW_DATA = 0x64, /* 2 bytes */
+ FTS_SI_SS_KEY_THRESHOLD = 0x66, /* 2 bytes */
+ FTS_SI_MS_TUNE_VERSION = 0x68, /* 2 bytes */
+ FTS_SI_CONFIG_CHECKSUM = 0x6A, /* 4 bytes */
+ FTS_SI_PRESSURE_FILTERED_RAW_ADDR = 0x70,
+ FTS_SI_PRESSURE_STRENGTH_ADDR = 0x72,
+ FTS_SI_PRESSURE_THRESHOLD = 0x76,
+};
+#endif
+
+void fts_delay(unsigned int ms);
+int fts_cmd_completion_check(struct fts_ts_info *info, uint8_t event1, uint8_t event2, uint8_t event3);
+int fts_fw_verify_update(struct fts_ts_info *info);
+int fts_get_version_info(struct fts_ts_info *info);
+void fts_get_afe_info(struct fts_ts_info *info);
+void fts_execute_autotune(struct fts_ts_info *info);
+int fts_fw_wait_for_event(struct fts_ts_info *info, unsigned char eid1, unsigned char eid2);
+int fts_systemreset(struct fts_ts_info *info);
+int fts_wait_for_ready(struct fts_ts_info *info);
+int fts_read_chip_id(struct fts_ts_info *info);
+
+#ifdef FEATURE_FTS_PRODUCTION_CODE
+int fts_fw_wait_for_specific_event(struct fts_ts_info *info,
+ unsigned char eid0, unsigned char eid1, unsigned char eid2);
+void procedure_cmd_event(struct fts_ts_info *info, unsigned char *data);
+void fts_production_init(void *device_info);
+#endif
+#endif /* _LINUX_FTM4_TS_H_ */
diff --git a/drivers/input/touchscreen/stm/fts.c b/drivers/input/touchscreen/stm/fts.c
new file mode 100644
index 0000000..d026339
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts.c
@@ -0,0 +1,4458 @@
+/**
+ * fts.c
+ *
+ * FTS Capacitive touch screen controller (FingerTipS)
+ *
+ * Copyright (C) 2016, STMicroelectronics Limited.
+ * Authors: AMG(Analog Mems Group)
+ *
+ * marco.cali@st.com
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * THE PRESENT SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, FOR THE SOLE
+ * PURPOSE TO SUPPORT YOUR APPLICATION DEVELOPMENT.
+ * AS A RESULT, STMICROELECTRONICS SHALL NOT BE HELD LIABLE FOR ANY DIRECT,
+ * INDIRECT OR CONSEQUENTIAL DAMAGES WITH RESPECT TO ANY CLAIMS ARISING FROM
+ * THE CONTENT OF SUCH SOFTWARE AND/OR THE USE MADE BY CUSTOMERS OF THE
+ * CODING INFORMATION CONTAINED HEREIN IN CONNECTION WITH THEIR PRODUCTS.
+ *
+ * THIS SOFTWARE IS SPECIFICALLY DESIGNED FOR EXCLUSIVE USE WITH ST PARTS.
+ */
+
+
+/*!
+ * \file fts.c
+ * \brief It is the main file which contains all the most important functions
+ * generally used by a device driver the driver
+ */
+#include <linux/device.h>
+
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/input.h>
+#include <linux/input/mt.h>
+#include <linux/interrupt.h>
+#include <linux/hrtimer.h>
+#include <linux/delay.h>
+#include <linux/firmware.h>
+#include <linux/i2c.h>
+#include <linux/i2c-dev.h>
+#include <linux/spi/spi.h>
+#include <linux/completion.h>
+#include <linux/device.h>
+
+#include <linux/gpio.h>
+#include <linux/of_gpio.h>
+#include <linux/regulator/consumer.h>
+
+#include <linux/notifier.h>
+#include <linux/msm_drm_notify.h>
+
+#ifdef KERNEL_ABOVE_2_6_38
+#include <linux/input/mt.h>
+#endif
+
+
+#include "fts.h"
+#include "fts_lib/ftsCompensation.h"
+#include "fts_lib/ftsCore.h"
+#include "fts_lib/ftsIO.h"
+#include "fts_lib/ftsError.h"
+#include "fts_lib/ftsFlash.h"
+#include "fts_lib/ftsFrame.h"
+#include "fts_lib/ftsGesture.h"
+#include "fts_lib/ftsTest.h"
+#include "fts_lib/ftsTime.h"
+#include "fts_lib/ftsTool.h"
+
+
+/* Switch GPIO values */
+#define FTS_SWITCH_GPIO_VALUE_SLPI_MASTER 0
+#define FTS_SWITCH_GPIO_VALUE_AP_MASTER 1
+
+/**
+ * Event handler installer helpers
+ */
+#define event_id(_e) (EVT_ID_##_e >> 4)
+#define handler_name(_h) fts_##_h##_event_handler
+
+#define install_handler(_i, _evt, _hnd) \
+ do { \
+ _i->event_dispatch_table[event_id(_evt)] = handler_name(_hnd); \
+ } while (0)
+
+
+/* Use decimal-formatted raw data */
+#define RAW_DATA_FORMAT_DEC
+
+#ifdef KERNEL_ABOVE_2_6_38
+#define TYPE_B_PROTOCOL
+#endif
+
+
+extern SysInfo systemInfo;
+extern TestToDo tests;
+#ifdef GESTURE_MODE
+extern struct mutex gestureMask_mutex;
+#endif
+
+char fts_ts_phys[64]; /* /< buffer which store the input device name
+ * assigned by the kernel */
+
+static u32 typeOfComand[CMD_STR_LEN] = { 0 }; /* /< buffer used to store the
+ * command sent from the MP
+ * device file node */
+static int numberParameters; /* /< number of parameter passed through the MP
+ * device file node */
+#ifdef USE_ONE_FILE_NODE
+static int feature_feasibility = ERROR_OP_NOT_ALLOW;
+#endif
+#ifdef GESTURE_MODE
+static u8 mask[GESTURE_MASK_SIZE + 2];
+extern u16 gesture_coordinates_x[GESTURE_MAX_COORDS_PAIRS_REPORT];
+extern u16 gesture_coordinates_y[GESTURE_MAX_COORDS_PAIRS_REPORT];
+extern int gesture_coords_reported;
+extern struct mutex gestureMask_mutex;
+#endif
+
+#ifdef PHONE_KEY
+static u8 key_mask; /* /< store the last update of the key mask
+ * published by the IC */
+#endif
+
+extern spinlock_t fts_int;
+
+static int fts_init_sensing(struct fts_ts_info *info);
+static int fts_mode_handler(struct fts_ts_info *info, int force);
+
+
+static int fts_chip_initialization(struct fts_ts_info *info, int init_type);
+
+static void fts_report_timestamp(struct fts_ts_info *info)
+{
+ input_event(info->input_dev, EV_MSC, MSC_TIMESTAMP,
+ info->timestamp / 1000);
+}
+
+/**
+ * Release all the touches in the linux input subsystem
+ * @param info pointer to fts_ts_info which contains info about device/hw setup
+ */
+void release_all_touches(struct fts_ts_info *info)
+{
+ unsigned int type = MT_TOOL_FINGER;
+ int i;
+
+ for (i = 0; i < TOUCH_ID_MAX; i++) {
+#ifdef STYLUS_MODE
+ if (test_bit(i, &info->stylus_id))
+ type = MT_TOOL_PEN;
+ else
+ type = MT_TOOL_FINGER;
+#endif
+ input_mt_slot(info->input_dev, i);
+ input_report_abs(info->input_dev, ABS_MT_PRESSURE, 0);
+ input_mt_report_slot_state(info->input_dev, type, 0);
+ input_report_abs(info->input_dev, ABS_MT_TRACKING_ID, -1);
+ }
+ input_report_key(info->input_dev, BTN_TOUCH, 0);
+ fts_report_timestamp(info);
+ input_sync(info->input_dev);
+ info->touch_id = 0;
+#ifdef STYLUS_MODE
+ info->stylus_id = 0;
+#endif
+}
+
+
+/**
+ * @defgroup file_nodes Driver File Nodes
+ * Driver publish a series of file nodes used to provide several utilities
+ * to the host and give him access to different API.
+ * @{
+ */
+
+/**
+ * @defgroup device_file_nodes Device File Nodes
+ * @ingroup file_nodes
+ * Device File Nodes \n
+ * There are several file nodes that are associated to the device and which
+ * are designed to be used by the host to enable/disable features or trigger
+ * some system specific actions \n
+ * Usually their final path depend on the definition of device tree node of
+ * the IC (e.g /sys/devices/soc.0/f9928000.i2c/i2c-6/6-0049)
+ * @{
+ */
+/***************************************** FW UPGGRADE
+ * ***************************************************/
+
+/**
+ * File node function to Update firmware from shell \n
+ * echo path_to_fw X Y > fwupdate perform a fw update \n
+ * where: \n
+ * path_to_fw = file name or path of the the FW to burn, if "NULL" the default
+ * approach selected in the driver will be used\n
+ * X = 0/1 to force the FW update whichever fw_version and config_id;
+ * 0=perform a fw update only if the fw in the file is newer than the fw in the
+ * chip \n
+ * Y = 0/1 keep the initialization data; 0 = will erase the initialization data
+ * from flash, 1 = will keep the initialization data
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which represent an error code (00000000 no
+ * error) \n
+ * } = end byte
+ */
+static ssize_t fts_fwupdate_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ret, mode[2];
+ char path[100];
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ /* default(if not specified by user) set force = 0 and keep_cx to 1 */
+ mode[0] = 0;
+ mode[1] = 1;
+
+ /* reading out firmware upgrade parameters */
+ sscanf(buf, "%100s %d %d", path, &mode[0], &mode[1]);
+ pr_info("fts_fwupdate_store: path = %s\n", path);
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true);
+
+ if (info->sensor_sleep)
+ ret = ERROR_BUS_WR;
+ else
+ ret = flashProcedure(path, mode[0], mode[1]);
+
+ info->fwupdate_stat = ret;
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+
+ if (ret == ERROR_BUS_WR)
+ pr_err("%s: bus is not accessible. ERROR %08X\n",
+ __func__, ret);
+ else if (ret < OK)
+ pr_err("%s Unable to upgrade firmware! ERROR %08X\n",
+ __func__, ret);
+
+ return count;
+}
+
+static ssize_t fts_fwupdate_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ /* fwupdate_stat: ERROR code Returned by flashProcedure. */
+ return scnprintf(buf, PAGE_SIZE, "{ %08X }\n", info->fwupdate_stat);
+}
+
+
+/***************************************** UTILITIES
+ * (current fw_ver/conf_id, active mode, file fw_ver/conf_id)
+ ***************************************************/
+/**
+ * File node to show on terminal external release version in Little Endian \n
+ * (first the less significant byte) \n
+ * cat appid show the external release version of the FW running in the IC
+ */
+static ssize_t fts_appid_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ int error;
+ char temp[100];
+
+ error = scnprintf(buf,
+ PAGE_SIZE,
+ "%s\n",
+ printHex("ST-V",
+ systemInfo.u8_releaseInfo,
+ EXTERNAL_RELEASE_INFO_SIZE,
+ temp,
+ sizeof(temp)));
+
+ return error;
+}
+
+/**
+ * File node to show on terminal the mode that is active on the IC \n
+ * cat mode_active to show the bitmask which indicate
+ * the modes/features which are running on the IC in a specific instant of time
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1 = 1 byte in HEX format which represent the actual running scan mode
+ * (@link scan_opt Scan Mode Options @endlink) \n
+ * X2 = 1 byte in HEX format which represent the bitmask on which is running
+ * the actual scan mode \n
+ * X3X4 = 2 bytes in HEX format which represent a bitmask of the features that
+ * are enabled at this moment (@link feat_opt Feature Selection Options
+ * @endlink) \n
+ * } = end byte
+ * @see fts_mode_handler()
+ */
+static ssize_t fts_mode_active_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ pr_info("Current mode active = %08X\n", info->mode);
+ return scnprintf(buf, PAGE_SIZE, "{ %08X }\n", info->mode);
+}
+
+/**
+ * File node to show the fw_ver and config_id of the FW file
+ * cat fw_file_test show on the kernel log external release
+ * of the FW stored in the fw file/header file
+ */
+static ssize_t fts_fw_test_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+ Firmware fw;
+ int ret;
+ char temp[100] = { 0 };
+
+ fw.data = NULL;
+ ret = readFwFile(info->board->fw_name, &fw, 0);
+
+ if (ret < OK)
+ pr_err("Error during reading FW file! ERROR %08X\n", ret);
+ else
+ pr_info("%s, size = %d bytes\n",
+ printHex("EXT Release = ",
+ systemInfo.u8_releaseInfo,
+ EXTERNAL_RELEASE_INFO_SIZE,
+ temp,
+ sizeof(temp)),
+ fw.data_size);
+ kfree(fw.data);
+ return 0;
+}
+
+static ssize_t fts_status_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+ u8 *dump = NULL;
+ int dumpSize = ERROR_DUMP_ROW_SIZE * ERROR_DUMP_COL_SIZE;
+ u8 reg;
+ int written = 0;
+ int res;
+ int i;
+
+ if (fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true) < 0) {
+ pr_err("%s: bus is not accessible.\n", __func__);
+ written += scnprintf(buf, PAGE_SIZE,
+ "Bus is not accessible.\n");
+ goto exit;
+ }
+
+ written += scnprintf(buf + written, PAGE_SIZE - written,
+ "Mode: 0x%08X\n", info->mode);
+
+ res = fts_writeReadU8UX(FTS_CMD_HW_REG_R, ADDR_SIZE_HW_REG, ADDR_ICR,
+ ®, 1, DUMMY_HW_REG);
+ if (res < 0)
+ pr_err("%s: failed to read ICR.\n", __func__);
+ else
+ written += scnprintf(buf + written, PAGE_SIZE - written,
+ "ICR: 0x%02X\n", reg);
+
+ dump = kzalloc(dumpSize, GFP_KERNEL);
+ if (!dump) {
+ written += strlcat(buf + written, "Buffer allocation failed!\n",
+ PAGE_SIZE - written);
+ goto exit;
+ }
+
+ res = dumpErrorInfo(dump, ERROR_DUMP_ROW_SIZE * ERROR_DUMP_COL_SIZE);
+ if (res >= 0) {
+ written += strlcat(buf + written, "Error dump:",
+ PAGE_SIZE - written);
+ for (i = 0; i < dumpSize; i++) {
+ if (i % 8 == 0)
+ written += scnprintf(buf + written,
+ PAGE_SIZE - written,
+ "\n%02X: ", i);
+ written += scnprintf(buf + written,
+ PAGE_SIZE - written,
+ "%02X ", dump[i]);
+ }
+ written += strlcat(buf + written, "\n", PAGE_SIZE - written);
+ }
+
+exit:
+ kfree(dump);
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return written;
+}
+
+#if 0
+/**
+ * File node to obtain and show strength frame
+ * cat strength_frame to obtain strength data \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which represent an error code (00000000 no
+ *error) \n
+ * **** if error code is all 0s **** \n
+ * FF = 1 byte in HEX format number of rows \n
+ * SS = 1 byte in HEX format number of columns \n
+ * N1, ... = the decimal value of each node separated by a coma \n
+ * ********************************* \n
+ * } = end byte
+ */
+static ssize_t fts_strength_frame_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ MutualSenseFrame frame;
+ int res, count, j, size = (6 * 2) + 1, index = 0;
+ char *all_strbuff = NULL;
+ /* char buff[CMD_STR_LEN] = {0}; */
+ /* struct i2c_client *client = to_i2c_client(dev); */
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ frame.node_data = NULL;
+
+ res = fts_disableInterrupt();
+ if (res < OK)
+ goto END;
+
+ res = senseOn();
+ if (res < OK) {
+ pr_err("%s: could not start scanning! ERROR %08X\n",
+ __func__, res);
+ goto END;
+ }
+ mdelay(WAIT_FOR_FRESH_FRAMES);
+ res = senseOff();
+ if (res < OK) {
+ pr_err("%s: could not finish scanning! ERROR %08X\n",
+ __func__, res);
+ goto END;
+ }
+
+ mdelay(WAIT_AFTER_SENSEOFF);
+ flushFIFO();
+
+ res = getMSFrame3(MS_STRENGTH, &frame);
+ if (res < OK) {
+ pr_err("%s: could not get the frame! ERROR %08X\n",
+ __func__, res);
+ goto END;
+ } else {
+ size += (res * 6);
+ pr_info("The frame size is %d words\n", res);
+ res = OK;
+ print_frame_short("MS Strength frame =", array1dTo2d_short(
+ frame.node_data, frame.node_data_size,
+ frame.header.sense_node),
+ frame.header.force_node,
+ frame.header.sense_node);
+ }
+
+END:
+ flushFIFO();
+ release_all_touches(info);
+ fts_mode_handler(info, 1);
+
+ all_strbuff = (char *)kzalloc(size * sizeof(char), GFP_KERNEL);
+
+ if (all_strbuff != NULL) {
+ snprintf(&all_strbuff[index], 11, "{ %08X", res);
+
+ index += 10;
+
+ if (res >= OK) {
+ snprintf(&all_strbuff[index], 3, "%02X",
+ (u8)frame.header.force_node);
+ index += 2;
+ snprintf(&all_strbuff[index], 3, "%02X",
+ (u8)frame.header.sense_node);
+
+ index += 2;
+
+ for (j = 0; j < frame.node_data_size; j++) {
+ snprintf(&all_strbuff[index], 10, "%d,%n",
+ frame.node_data[j], &count);
+ index += count;
+ }
+
+ kfree(frame.node_data);
+ }
+
+ snprintf(&all_strbuff[index], 3, " }");
+ index += 2;
+
+ count = snprintf(buf, TSP_BUF_SIZE, "%s\n", all_strbuff);
+ kfree(all_strbuff);
+ } else
+ pr_err("%s: Unable to allocate all_strbuff! ERROR %08X\n",
+ __func__, ERROR_ALLOC);
+
+ fts_enableInterrupt();
+ return count;
+}
+#endif
+
+/***************************************** FEATURES
+ ***************************************************/
+
+/* TODO: edit this function according to the features policy to allow during
+ * the screen on/off, following is shown an example but check always with ST
+ * for more details */
+/**
+ * Check if there is any conflict in enable/disable a particular feature
+ * considering the features already enabled and running
+ * @param info pointer to fts_ts_info which contains info about the device
+ * and its hw setup
+ * @param feature code of the feature that want to be tested
+ * @return OK if is possible to enable/disable feature, ERROR_OP_NOT_ALLOW
+ * in case of any other conflict
+ */
+int check_feature_feasibility(struct fts_ts_info *info, unsigned int feature)
+{
+ int res = OK;
+
+/* Example based on the status of the screen and on the feature
+ * that is trying to enable */
+ /*res=ERROR_OP_NOT_ALLOW;
+ * if(info->resume_bit ==0){
+ * switch(feature){
+ #ifdef GESTURE_MODE
+ * case FEAT_SEL_GESTURE:
+ * res = OK;
+ * break;
+ #endif
+ * default:
+ * pr_err("%s: Feature not allowed in this
+ * operating mode! ERROR %08X\n", __func__, res);
+ * break;
+ *
+ * }
+ * }else{
+ * switch(feature){
+ #ifdef GESTURE_MODE
+ * case FEAT_SEL_GESTURE:
+ #endif
+ * case FEAT__SEL_GLOVE: // glove mode can only activate
+ *during sense on
+ * res = OK;
+ * break;
+ *
+ * default:
+ * pr_err("%s: Feature not allowed in this
+ * operating mode! ERROR %08X\n", __func__, res);
+ * break;
+ *
+ * }
+ * }*/
+
+
+/* Example based only on the feature that is going to be activated */
+ switch (feature) {
+ case FEAT_SEL_GESTURE:
+ if (info->cover_enabled == 1) {
+ res = ERROR_OP_NOT_ALLOW;
+ pr_err("%s: Feature not allowed when in Cover mode! ERROR %08X\n",
+ __func__, res);
+ /* for example here can be placed a code for disabling
+ * the cover mode when gesture is activated */
+ }
+ break;
+
+ case FEAT_SEL_GLOVE:
+ if (info->gesture_enabled == 1) {
+ res = ERROR_OP_NOT_ALLOW;
+ pr_err("%s: Feature not allowed when Gestures enabled! ERROR %08X\n",
+ __func__, res);
+ /* for example here can be placed a code for disabling
+ * the gesture mode when cover is activated
+ * (that means that cover mode has
+ * an higher priority on gesture mode) */
+ }
+ break;
+
+ default:
+ pr_info("%s: Feature Allowed!\n", __func__);
+ }
+
+ return res;
+}
+
+#ifdef USE_ONE_FILE_NODE
+/**
+ * File node to enable some feature
+ * echo XX 00/01 > feature_enable to enable/disable XX
+ * (possible values @link feat_opt Feature Selection Options @endlink) feature
+ * cat feature_enable to show the result of enabling/disabling process
+ * echo 01/00 > feature_enable; cat feature_enable to perform
+ * both actions stated before in just one call \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which represent an error code (00000000 =
+ * no error) \n
+ * } = end byte
+ */
+static ssize_t fts_feature_enable_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+ char *p = (char *)buf;
+ unsigned int temp;
+ int res = OK;
+
+ if (fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true) < 0) {
+ res = ERROR_BUS_WR;
+ pr_err("%s: bus is not accessible.", __func__);
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+ }
+
+ if ((count - 2 + 1) / 3 != 1)
+ pr_err("fts_feature_enable: Number of parameter wrong! %d > %d\n",
+ (count - 2 + 1) / 3, 1);
+ else {
+ sscanf(p, "%02X ", &temp);
+ p += 3;
+ res = check_feature_feasibility(info, temp);
+ if (res >= OK) {
+ switch (temp) {
+ #ifdef GESTURE_MODE
+ case FEAT_SEL_GESTURE:
+ sscanf(p, "%02X ", &info->gesture_enabled);
+ pr_info("fts_feature_enable: Gesture Enabled = %d\n",
+ info->gesture_enabled);
+ break;
+ #endif
+
+ #ifdef GLOVE_MODE
+ case FEAT_SEL_GLOVE:
+ sscanf(p, "%02X ", &info->glove_enabled);
+ pr_info("fts_feature_enable: Glove Enabled = %d\n",
+ info->glove_enabled);
+ break;
+ #endif
+
+ #ifdef STYLUS_MODE
+ case FEAT_SEL_STYLUS:
+ sscanf(p, "%02X ", &info->stylus_enabled);
+ pr_info("fts_feature_enable: Stylus Enabled = %d\n",
+ info->stylus_enabled);
+ break;
+ #endif
+
+ #ifdef COVER_MODE
+ case FEAT_SEL_COVER:
+ sscanf(p, "%02X ", &info->cover_enabled);
+ pr_info("fts_feature_enable: Cover Enabled = %d\n",
+ info->cover_enabled);
+ break;
+ #endif
+
+ #ifdef CHARGER_MODE
+ case FEAT_SEL_CHARGER:
+ sscanf(p, "%02X ", &info->charger_enabled);
+ pr_info("fts_feature_enable: Charger Enabled = %d\n",
+ info->charger_enabled);
+ break;
+ #endif
+
+ #ifdef GRIP_MODE
+ case FEAT_SEL_GRIP:
+ sscanf(p, "%02X ", &info->grip_enabled);
+ pr_info("fts_feature_enable: Grip Enabled = %d\n",
+ info->grip_enabled);
+ break;
+ #endif
+
+
+
+ default:
+ pr_err("fts_feature_enable: Feature %08X not valid! ERROR %08X\n",
+ temp, ERROR_OP_NOT_ALLOW);
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ feature_feasibility = res;
+ }
+ if (feature_feasibility >= OK)
+ feature_feasibility = fts_mode_handler(info, 1);
+ else
+ pr_err("%s: Call echo XX 00/01 > feature_enable with a correct feature value (XX)! ERROR %08X\n",
+ __func__, res);
+ }
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+}
+
+
+
+static ssize_t fts_feature_enable_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ int count = 0;
+
+ if (feature_feasibility < OK)
+ pr_err("%s: Call before echo XX 00/01 > feature_enable with a correct feature value (XX)! ERROR %08X\n",
+ __func__, feature_feasibility);
+
+ count += scnprintf(buf + count,
+ PAGE_SIZE - count, "{ %08X }\n",
+ feature_feasibility);
+
+ feature_feasibility = ERROR_OP_NOT_ALLOW;
+ return count;
+}
+
+#else
+
+
+#ifdef GRIP_MODE
+/**
+ * File node to set the grip mode
+ * echo 01/00 > grip_mode to enable/disable glove mode \n
+ * cat grip_mode to show the status of the grip_enabled switch \n
+ * echo 01/00 > grip_mode; cat grip_mode to enable/disable grip
+ *mode
+ * and see the switch status in just one call \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which represent the value
+ * info->grip_enabled (1 = enabled; 0= disabled) \n
+ * } = end byte
+ */
+static ssize_t fts_grip_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ int count = 0;
+
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ pr_info("%s: grip_enabled = %d\n", __func__,
+ info->grip_enabled);
+
+ count += scnprintf(buf + count,
+ PAGE_SIZE - count, "{ %08X }\n",
+ info->grip_enabled);
+
+ return count;
+}
+
+
+static ssize_t fts_grip_mode_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ char *p = (char *)buf;
+ unsigned int temp;
+ int res;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ if (fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true) < 0) {
+ res = ERROR_BUS_WR;
+ pr_err("%s: bus is not accessible.", __func__);
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+ }
+
+ /* in case of a different elaboration of the input, just modify
+ * this initial part of the code according to customer needs */
+ if ((count + 1) / 3 != 1)
+ pr_err("%s: Number of bytes of parameter wrong! %zu != 1 byte\n",
+ __func__, (count + 1) / 3);
+ else {
+ sscanf(p, "%02X ", &temp);
+ p += 3;
+
+/* standard code that should be always used when a feature is enabled! */
+/* first step : check if the wanted feature can be enabled */
+/* second step: call fts_mode_handler to actually enable it */
+/* NOTE: Disabling a feature is always allowed by default */
+ res = check_feature_feasibility(info, FEAT_SEL_GRIP);
+ if (res >= OK || temp == FEAT_DISABLE) {
+ info->grip_enabled = temp;
+ res = fts_mode_handler(info, 1);
+ if (res < OK)
+ pr_err("%s: Error during fts_mode_handler! ERROR %08X\n",
+ __func__, res);
+ }
+ }
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+}
+#endif
+
+#ifdef CHARGER_MODE
+/**
+ * File node to set the glove mode
+ * echo XX/00 > charger_mode to value >0 to enable
+ * (possible values: @link charger_opt Charger Options @endlink),
+ * 00 to disable charger mode \n
+ * cat charger_mode to show the status of the charger_enabled switch \n
+ * echo 01/00 > charger_mode; cat charger_mode to enable/disable
+ * charger mode and see the switch status in just one call \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which represent the value
+ * info->charger_enabled (>0 = enabled; 0= disabled) \n
+ * } = end byte
+ */
+static ssize_t fts_charger_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ int count = 0;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ pr_info("%s: charger_enabled = %d\n", __func__,
+ info->charger_enabled);
+
+ count += scnprintf(buf + count,
+ PAGE_SIZE - count, "{ %08X }\n",
+ info->charger_enabled);
+ return count;
+}
+
+
+static ssize_t fts_charger_mode_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ char *p = (char *)buf;
+ unsigned int temp;
+ int res;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ if (fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true) < 0) {
+ res = ERROR_BUS_WR;
+ pr_err("%s: bus is not accessible.\n", __func__);
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+ }
+
+/* in case of a different elaboration of the input, just modify this
+ * initial part of the code according to customer needs */
+ if ((count + 1) / 3 != 1)
+ pr_err("%s: Number of bytes of parameter wrong! %zu != 1 byte\n",
+ __func__, (count + 1) / 3);
+ else {
+ sscanf(p, "%02X ", &temp);
+ p += 3;
+
+/** standard code that should be always used when a feature is enabled!
+ * first step : check if the wanted feature can be enabled
+ * second step: call fts_mode_handler to actually enable it
+ * NOTE: Disabling a feature is always allowed by default
+ */
+ res = check_feature_feasibility(info, FEAT_SEL_CHARGER);
+ if (res >= OK || temp == FEAT_DISABLE) {
+ info->charger_enabled = temp;
+ res = fts_mode_handler(info, 1);
+ if (res < OK)
+ pr_err("%s: Error during fts_mode_handler! ERROR %08X\n",
+ __func__, res);
+ }
+ }
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+}
+#endif
+
+#ifdef GLOVE_MODE
+/**
+ * File node to set the glove mode
+ * echo 01/00 > glove_mode to enable/disable glove mode \n
+ * cat glove_mode to show the status of the glove_enabled switch \n
+ * echo 01/00 > glove_mode; cat glove_mode to enable/disable glove mode and
+ * see the switch status in just one call \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which represent the of value
+ * info->glove_enabled (1 = enabled; 0= disabled) \n
+ * } = end byte
+ */
+static ssize_t fts_glove_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ int count = 0;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ pr_info("%s: glove_enabled = %d\n", __func__, info->glove_enabled);
+
+ count += scnprintf(buf + count,
+ PAGE_SIZE - count, "{ %08X }\n",
+ info->glove_enabled);
+
+ return count;
+}
+
+
+static ssize_t fts_glove_mode_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ char *p = (char *)buf;
+ unsigned int temp;
+ int res;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ if (fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true) < 0) {
+ res = ERROR_BUS_WR;
+ pr_err("%s: bus is not accessible.\n", __func__);
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+ }
+
+/* in case of a different elaboration of the input, just modify this
+ * initial part of the code according to customer needs */
+ if ((count + 1) / 3 != 1)
+ pr_err("%s: Number of bytes of parameter wrong! %zu != 1 byte\n",
+ __func__, (count + 1) / 3);
+ else {
+ sscanf(p, "%02X ", &temp);
+ p += 3;
+
+/* standard code that should be always used when a feature is enabled! */
+/* first step : check if the wanted feature can be enabled */
+/* second step: call fts_mode_handler to actually enable it */
+/* NOTE: Disabling a feature is always allowed by default */
+ res = check_feature_feasibility(info, FEAT_SEL_GLOVE);
+ if (res >= OK || temp == FEAT_DISABLE) {
+ info->glove_enabled = temp;
+ res = fts_mode_handler(info, 1);
+ if (res < OK)
+ pr_err("%s: Error during fts_mode_handler! ERROR %08X\n",
+ __func__, res);
+ }
+ }
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+}
+#endif
+
+
+#ifdef COVER_MODE
+/* echo 01/00 > cover_mode to enable/disable cover mode */
+/* cat cover_mode to show the status of the cover_enabled switch
+ * (example output in the terminal = "AA00000001BB" if the switch is enabled) */
+/* echo 01/00 > cover_mode; cat cover_mode to enable/disable cover mode and
+ * see the switch status in just one call */
+/* NOTE: the cover can be handled also using a notifier, in this case the body
+ * of these functions should be copied in the notifier callback */
+/**
+ * File node to set the cover mode
+ * echo 01/00 > cover_mode to enable/disable cover mode \n
+ * cat cover_mode to show the status of the cover_enabled switch \n
+ * echo 01/00 > cover_mode; cat cover_mode to enable/disable cover mode
+ * and see the switch status in just one call \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which is the value of info->cover_enabled
+ * (1 = enabled; 0= disabled)\n
+ * } = end byte \n
+ * NOTE: \n
+ * the cover can be handled also using a notifier, in this case the body of
+ * these functions should be copied in the notifier callback
+ */
+static ssize_t fts_cover_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ int count = 0;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ pr_info("%s: cover_enabled = %d\n", __func__, info->cover_enabled);
+
+ count += scnprintf(buf + count,
+ PAGE_SIZE - count, "{ %08X }\n",
+ info->cover_enabled);
+
+ return count;
+}
+
+
+static ssize_t fts_cover_mode_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ char *p = (char *)buf;
+ unsigned int temp;
+ int res;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ if (fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true) < 0) {
+ res = ERROR_BUS_WR;
+ pr_err("%s: bus is not accessible.\n", __func__);
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+ }
+
+/* in case of a different elaboration of the input, just modify this
+ * initial part of the code according to customer needs */
+ if ((count + 1) / 3 != 1)
+ pr_err("%s: Number of bytes of parameter wrong! %zu != 1 byte\n",
+ __func__, (count + 1) / 3);
+ else {
+ sscanf(p, "%02X ", &temp);
+ p += 3;
+
+/* standard code that should be always used when a feature is enabled! */
+/* first step : check if the wanted feature can be enabled */
+/* second step: call fts_mode_handler to actually enable it */
+/* NOTE: Disabling a feature is always allowed by default */
+ res = check_feature_feasibility(info, FEAT_SEL_COVER);
+ if (res >= OK || temp == FEAT_DISABLE) {
+ info->cover_enabled = temp;
+ res = fts_mode_handler(info, 1);
+ if (res < OK)
+ pr_err("%s: Error during fts_mode_handler! ERROR %08X\n",
+ __func__, res);
+ }
+ }
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+}
+#endif
+
+#ifdef STYLUS_MODE
+/**
+ * File node to enable the stylus report
+ * echo 01/00 > stylus_mode to enable/disable stylus mode \n
+ * cat stylus_mode to show the status of the stylus_enabled switch \n
+ * echo 01/00 > stylus_mode; cat stylus_mode to enable/disable stylus mode
+ * and see the switch status in just one call \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which is the value of info->stylus_enabled
+ * (1 = enabled; 0= disabled)\n
+ * } = end byte
+ */
+static ssize_t fts_stylus_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ int count = 0;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ pr_info("%s: stylus_enabled = %d\n", __func__, info->stylus_enabled);
+
+ count += scnprintf(buf + count,
+ PAGE_SIZE - count, "{ %08X }\n",
+ info->stylus_enabled);
+
+ return count;
+}
+
+
+static ssize_t fts_stylus_mode_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ char *p = (char *)buf;
+ unsigned int temp;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+
+/* in case of a different elaboration of the input, just modify this
+ * initial part of the code according to customer needs */
+ if ((count + 1) / 3 != 1)
+ pr_err("%s: Number of bytes of parameter wrong! %zu != 1 byte\n",
+ __func__, (count + 1) / 3);
+ else {
+ sscanf(p, "%02X ", &temp);
+ p += 3;
+
+
+ info->stylus_enabled = temp;
+ }
+
+ return count;
+}
+#endif
+
+#endif
+
+/***************************************** GESTURES
+ ***************************************************/
+#ifdef GESTURE_MODE
+#ifdef USE_GESTURE_MASK /* if this define is used, a gesture bit mask
+ * is used as method to select the gestures to
+ * enable/disable */
+
+/**
+ * File node used by the host to set the gesture mask to enable or disable
+ * echo EE X1 X2 ~~ > gesture_mask set the gesture to disable/enable;
+ * EE = 00(disable) or 01(enable) \n
+ * X1 ~~ = gesture mask (example 06 00 ~~ 00 this gesture mask represents
+ * the gestures with ID = 1 and 2) can be specified
+ * from 1 to GESTURE_MASK_SIZE bytes, \n
+ * if less than GESTURE_MASK_SIZE bytes are passed as arguments,
+ * the omit bytes of the mask maintain the previous settings \n
+ * if one or more gestures is enabled the driver will automatically
+ * enable the gesture mode, If all the gestures are disabled the driver
+ * automatically will disable the gesture mode \n
+ * cat gesture_mask set inside the specified mask and return an error code
+ * for the operation \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which represent an error code for enabling
+ * the mask (00000000 = no error)\n
+ * } = end byte \n\n
+ * if USE_GESTURE_MASK is not define the usage of the function become: \n\n
+ * echo EE X1 X2 ~~ > gesture_mask set the gesture to disable/enable;
+ * EE = 00(disable) or 01(enable) \n
+ * X1 ~~ = gesture IDs (example 01 02 05 represent the gestures with ID = 1, 2
+ * and 5)
+ * there is no limit of the IDs passed as arguments, (@link gesture_opt Gesture
+ * IDs @endlink) \n
+ * if one or more gestures is enabled the driver will automatically enable
+ * the gesture mode. If all the gestures are disabled the driver automatically
+ * will disable the gesture mode. \n
+ * cat gesture_mask to show the status of the gesture enabled switch \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which is the value of info->gesture_enabled
+ * (1 = enabled; 0= disabled)\n
+ * } = end byte
+ */
+static ssize_t fts_gesture_mask_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ int count = 0, res, temp;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ if (fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true) < 0) {
+ res = ERROR_BUS_WR;
+ pr_err("%s: bus is not accessible.\n", __func__);
+ scnprintf(buf, PAGE_SIZE, "{ %08X }\n", res);
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+ }
+
+ if (mask[0] == 0) {
+ res = ERROR_OP_NOT_ALLOW;
+ pr_err("%s: Call before echo enable/disable xx xx .... > gesture_mask with a correct number of parameters! ERROR %08X\n",
+ __func__, res);
+ } else {
+ if (mask[1] == FEAT_ENABLE || mask[1] == FEAT_DISABLE)
+ res = updateGestureMask(&mask[2], mask[0], mask[1]);
+ else
+ res = ERROR_OP_NOT_ALLOW;
+
+ if (res < OK)
+ pr_err("fts_gesture_mask_store: ERROR %08X\n", res);
+ }
+ res |= check_feature_feasibility(info, FEAT_SEL_GESTURE);
+ temp = isAnyGestureActive();
+ if (res >= OK || temp == FEAT_DISABLE)
+ info->gesture_enabled = temp;
+
+ pr_info("fts_gesture_mask_store: Gesture Enabled = %d\n",
+ info->gesture_enabled);
+
+ count += scnprintf(buf + count,
+ PAGE_SIZE - count, "{ %08X }\n", res);
+ mask[0] = 0;
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+}
+
+
+static ssize_t fts_gesture_mask_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ char *p = (char *)buf;
+ int n;
+ unsigned int temp;
+
+ if ((count + 1) / 3 > GESTURE_MASK_SIZE + 1) {
+ pr_err("fts_gesture_mask_store: Number of bytes of parameter wrong! %zu > (enable/disable + %d )\n",
+ (count + 1) / 3, GESTURE_MASK_SIZE);
+ mask[0] = 0;
+ } else {
+ mask[0] = ((count + 1) / 3) - 1;
+ for (n = 1; n <= (count + 1) / 3; n++) {
+ sscanf(p, "%02X ", &temp);
+ p += 3;
+ mask[n] = (u8)temp;
+ pr_info("mask[%d] = %02X\n", n, mask[n]);
+ }
+ }
+
+ return count;
+}
+
+#else /* if this define is not used, to select the gestures to enable/disable
+ * are used the IDs of the gestures */
+/* echo EE X1 X2 ... > gesture_mask set the gesture to disable/enable;
+ * EE = 00(disable) or 01(enable); X1 ... = gesture IDs
+ * (example 01 02 05... represent the gestures with ID = 1, 2 and 5)
+ * there is no limit of the parameters that can be passed,
+ * of course the gesture IDs should be valid (all the valid IDs are listed in
+ * ftsGesture.h) */
+/* cat gesture_mask enable/disable the given gestures, if one or more
+ * gestures is enabled the driver will automatically enable the gesture mode.
+ * If all the gestures are disabled the driver automatically will disable the
+ * gesture mode.
+ * At the end an error code will be printed
+ * (example output in the terminal = "AA00000000BB" if there are no errors) */
+/* echo EE X1 X2 ... > gesture_mask; cat gesture_mask perform in one command
+ * both actions stated before */
+/**
+ * File node used by the host to set the gesture mask to enable or disable
+ * echo EE X1 X2 ~~ > gesture_mask set the gesture to disable/enable;
+ * EE = 00(disable) or 01(enable) \n
+ * X1 ~ = gesture IDs (example 01 02 05 represent the gestures with ID = 1, 2
+ * and 5)
+ * there is no limit of the IDs passed as arguments, (@link gesture_opt Gesture
+ * IDs @endlink) \n
+ * if one or more gestures is enabled the driver will automatically enable
+ * the gesture mode, If all the gestures are disabled the driver automatically
+ * will disable the gesture mode \n
+ * cat gesture_mask to show the status of the gesture enabled switch \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which is the value of info->gesture_enabled
+ * (1 = enabled; 0= disabled)\n
+ * } = end byte
+ */
+static ssize_t fts_gesture_mask_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ int count = 0;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ pr_info("fts_gesture_mask_show: gesture_enabled = %d\n",
+ info->gesture_enabled);
+
+ count += scnprintf(buf + count,
+ PAGE_SIZE - count, "{ %08X }\n",
+ info->gesture_enabled);
+
+
+ return count;
+}
+
+
+static ssize_t fts_gesture_mask_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ char *p = (char *)buf;
+ int n;
+ unsigned int temp;
+ int res;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ if (fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true) < 0) {
+ res = ERROR_BUS_WR;
+ pr_err("%s: bus is not accessible.\n", __func__);
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+ }
+
+ if ((count + 1) / 3 < 2 || (count + 1) / 3 > GESTURE_MASK_SIZE + 1) {
+ pr_err("fts_gesture_mask_store: Number of bytes of parameter wrong! %d < or > (enable/disable + at least one gestureID or max %d bytes)\n",
+ (count + 1) / 3, GESTURE_MASK_SIZE);
+ mask[0] = 0;
+ } else {
+ memset(mask, 0, GESTURE_MASK_SIZE + 2);
+ mask[0] = ((count + 1) / 3) - 1;
+ sscanf(p, "%02X ", &temp);
+ p += 3;
+ mask[1] = (u8)temp;
+ for (n = 1; n < (count + 1) / 3; n++) {
+ sscanf(p, "%02X ", &temp);
+ p += 3;
+ fromIDtoMask((u8)temp, &mask[2], GESTURE_MASK_SIZE);
+ }
+
+ for (n = 0; n < GESTURE_MASK_SIZE + 2; n++)
+ pr_info("mask[%d] = %02X\n", n, mask[n]);
+ }
+
+
+ if (mask[0] == 0) {
+ res = ERROR_OP_NOT_ALLOW;
+ pr_err("%s: Call before echo enable/disable xx xx .... > gesture_mask with a correct number of parameters! ERROR %08X\n",
+ __func__, res);
+ } else {
+ if (mask[1] == FEAT_ENABLE || mask[1] == FEAT_DISABLE)
+ res = updateGestureMask(&mask[2], mask[0], mask[1]);
+ else
+ res = ERROR_OP_NOT_ALLOW;
+
+ if (res < OK)
+ pr_err("fts_gesture_mask_store: ERROR %08X\n", res);
+ }
+
+ res = check_feature_feasibility(info, FEAT_SEL_GESTURE);
+ temp = isAnyGestureActive();
+ if (res >= OK || temp == FEAT_DISABLE)
+ info->gesture_enabled = temp;
+ res = fts_mode_handler(info, 0);
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return count;
+}
+
+
+#endif
+
+
+/**
+ * File node to read the coordinates of the last gesture drawn by the user \n
+ * cat gesture_coordinates to obtain the gesture coordinates \n
+ * the string returned in the shell follow this up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which represent an error code (00000000 =
+ *OK) \n
+ * \n if error code = 00000000 \n
+ * CC = 1 byte in HEX format number of coords (pair of x,y) returned \n
+ * XXiYYi ... = XXi 2 bytes in HEX format for x[i] and
+ * YYi 2 bytes in HEX format for y[i] (big endian) \n
+ * \n
+ * } = end byte
+ */
+static ssize_t fts_gesture_coordinates_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ int size = PAGE_SIZE;
+ int count = 0, res, i = 0;
+
+ pr_info("%s: Getting gestures coordinates...\n", __func__);
+
+ if (gesture_coords_reported < OK) {
+ pr_err("%s: invalid coordinates! ERROR %08X\n",
+ __func__, gesture_coords_reported);
+ res = gesture_coords_reported;
+ } else {
+ size += gesture_coords_reported * 2 * 4 + 2;
+ /* coords are pairs of x,y (*2) where each coord is
+ * short(2bytes=4char)(*4) + 1 byte(2char) num of coords (+2)
+ **/
+ res = OK; /* set error code to OK */
+ }
+
+
+ count += scnprintf(buf + count,
+ size - count, "{ %08X", res);
+
+ if (res >= OK) {
+ count += scnprintf(buf + count,
+ size - count, "%02X",
+ gesture_coords_reported);
+
+ for (i = 0; i < gesture_coords_reported; i++) {
+ count += scnprintf(buf + count,
+ size - count,
+ "%04X",
+ gesture_coordinates_x[i]);
+ count += scnprintf(buf + count,
+ size - count,
+ "%04X",
+ gesture_coordinates_y[i]);
+ }
+ }
+
+ count += scnprintf(buf + count, size - count, " }\n");
+ pr_info("%s: Getting gestures coordinates FINISHED!\n", __func__);
+
+ return count;
+}
+#endif
+
+
+
+/***************************************** PRODUCTION TEST
+ ***************************************************/
+
+/**
+ * File node to execute the Mass Production Test or to get data from the IC
+ * (raw or ms/ss init data)
+ * echo cmd > stm_fts_cmd to execute a command \n
+ * cat stm_fts_cmd to show the result of the command \n
+ * echo cmd > stm_fts_cmd; cat stm_fts_cmd to execute and show the result
+ * in just one call \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * X1X2X3X4 = 4 bytes in HEX format which represent an error_code (00000000 =
+ * OK)\n
+ * (optional) data = data coming from the command executed represented as HEX
+ * string \n
+ * Not all the command return additional data \n
+ * } = end byte \n
+ * \n
+ * Possible commands (cmd): \n
+ * - 00 = MP Test -> return error_code \n
+ * - 01 = ITO Test -> return error_code \n
+ * - 03 = MS Raw Test -> return error_code \n
+ * - 04 = MS Init Data Test -> return error_code \n
+ * - 05 = SS Raw Test -> return error_code \n
+ * - 06 = SS Init Data Test -> return error_code \n
+ * - 13 = Read 1 MS Raw Frame -> return additional data: MS frame row after row
+ * \n
+ * - 14 = Read MS Init Data -> return additional data: MS init data row after
+ * row \n
+ * - 15 = Read 1 SS Raw Frame -> return additional data: SS frame,
+ * force channels followed by sense channels \n
+ * - 16 = Read SS Init Data -> return additional data: SS Init data,
+ * first IX for force and sense channels and then CX for force and sense
+ * channels \n
+ * - F0 = Perform a system reset -> return error_code \n
+ * - F1 = Perform a system reset and reenable the sensing and the interrupt
+ */
+static ssize_t stm_fts_cmd_store(struct device *dev,
+ struct device_attribute *attr, const char *buf,
+ size_t count)
+{
+ int n;
+ char *p = (char *)buf;
+
+ memset(typeOfComand, 0, CMD_STR_LEN * sizeof(u32));
+
+ pr_info("%s:\n", __func__);
+ for (n = 0; n < (count + 1) / 3; n++) {
+ sscanf(p, "%02X ", &typeOfComand[n]);
+ p += 3;
+ pr_info("typeOfComand[%d] = %02X\n", n, typeOfComand[n]);
+ }
+
+ numberParameters = n;
+ pr_info("Number of Parameters = %d\n", numberParameters);
+ return count;
+}
+
+static ssize_t stm_fts_cmd_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ int res, j, doClean = 0, index = 0;
+ int size = (6 * 2) + 1;
+ int nodes = 0;
+ int init_type = SPECIAL_PANEL_INIT;
+ u8 *all_strbuff = buf;
+ struct fts_ts_info *info = dev_get_drvdata(dev);
+
+ MutualSenseData compData;
+ SelfSenseData comData;
+ MutualSenseFrame frameMS;
+ SelfSenseFrame frameSS;
+
+ if (fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true) < 0) {
+ res = ERROR_BUS_WR;
+ pr_err("%s: bus is not accessible.\n", __func__);
+ scnprintf(buf, PAGE_SIZE, "{ %08X }\n", res);
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return 0;
+ }
+
+ if (numberParameters >= 1) {
+ res = fts_disableInterrupt();
+ if (res < 0) {
+ pr_err("fts_disableInterrupt: ERROR %08X\n", res);
+ res = (res | ERROR_DISABLE_INTER);
+ goto END;
+ }
+
+ switch (typeOfComand[0]) {
+ /*ITO TEST*/
+ case 0x01:
+ res = production_test_ito(LIMITS_FILE, &tests);
+ break;
+
+ /*PRODUCTION TEST*/
+ case 0x02:
+ if (systemInfo.u8_cfgAfeVer != systemInfo.u8_cxAfeVer) {
+ res = ERROR_OP_NOT_ALLOW;
+ pr_err("Miss match in CX version! MP test not allowed with wrong CX memory! ERROR %08X\n",
+ res);
+ break;
+ }
+ res = production_test_initialization(init_type);
+ break;
+
+ case 0x00:
+ if (systemInfo.u8_cfgAfeVer != systemInfo.u8_cxAfeVer) {
+ res = ERROR_OP_NOT_ALLOW;
+ pr_err("Miss match in CX version! MP test not allowed with wrong CX memory! ERROR %08X\n",
+ res);
+ break;
+ }
+
+ res = production_test_main(LIMITS_FILE, 1, init_type,
+ &tests);
+ break;
+
+ /*read mutual raw*/
+ case 0x13:
+ if (numberParameters > 1) {
+ pr_info("Get 1 MS Frame\n");
+ setScanMode(SCAN_MODE_LOCKED, typeOfComand[1]);
+ mdelay(WAIT_FOR_FRESH_FRAMES);
+ setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ mdelay(WAIT_AFTER_SENSEOFF);
+ /* Delete the events related to some touch
+ * (allow to call this function while touching
+ * the screen without having a flooding of the
+ * FIFO)
+ */
+ flushFIFO();
+ res = getMSFrame3(MS_RAW, &frameMS);
+ if (res < 0) {
+ pr_err("Error while taking the MS frame... ERROR %08X\n",
+ res);
+ } else {
+ pr_info("The frame size is %d words\n",
+ res);
+#ifdef RAW_DATA_FORMAT_DEC
+ size += 3 * 2 +
+ (7 * frameMS.header.sense_node + 1)
+ * frameMS.header.force_node;
+#else
+ size += (res * sizeof(short) + 2) * 2;
+#endif
+ /* set res to OK because if getMSFrame
+ * is successful res = number of words
+ * read
+ */
+ res = OK;
+ print_frame_short(
+ "MS frame =",
+ array1dTo2d_short(
+ frameMS.node_data,
+ frameMS.node_data_size,
+ frameMS.header.
+ sense_node),
+ frameMS.header.force_node,
+ frameMS.header.sense_node);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+ /*read self raw*/
+ case 0x15:
+ if (numberParameters > 1) {
+ pr_info("Get 1 SS Frame\n");
+ setScanMode(SCAN_MODE_LOCKED, typeOfComand[1]);
+ mdelay(WAIT_FOR_FRESH_FRAMES);
+ setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ mdelay(WAIT_AFTER_SENSEOFF);
+ flushFIFO();
+ /* delete the events related to some touch
+ * (allow to call this function while touching
+ * the screen without having a flooding of the
+ * FIFO) */
+ res = getSSFrame3(SS_RAW, &frameSS);
+
+ if (res < OK) {
+ pr_err("Error while taking the SS frame... ERROR %08X\n",
+ res);
+ } else {
+ pr_info("The frame size is %d words\n",
+ res);
+#ifdef RAW_DATA_FORMAT_DEC
+ size += 3 * 2 + 5 +
+ (frameSS.header.sense_node +
+ frameSS.header.force_node) * 7;
+#else
+ size += (res * sizeof(short) + 2) * 2;
+#endif
+ /* set res to OK because if getMSFrame
+ * is successful res = number of words
+ * read
+ */
+ res = OK;
+ print_frame_short(
+ "SS force frame =",
+ array1dTo2d_short(
+ frameSS.force_data,
+ frameSS.header.
+ force_node, 1),
+ frameSS.header.force_node, 1);
+ print_frame_short(
+ "SS sense frame =",
+ array1dTo2d_short(
+ frameSS.sense_data,
+ frameSS.header.
+ sense_node,
+ frameSS.header.
+ sense_node),
+ 1, frameSS.header.sense_node);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+ break;
+
+ case 0x14: /* read mutual comp data */
+ pr_info("Get MS Compensation Data\n");
+ res = readMutualSenseCompensationData(LOAD_CX_MS_TOUCH,
+ &compData);
+
+ if (res < 0)
+ pr_err("Error reading MS compensation data ERROR %08X\n",
+ res);
+ else {
+ pr_info("MS Compensation Data Reading Finished!\n");
+ size += ((compData.node_data_size + 3) *
+ sizeof(u8)) * 2;
+ print_frame_i8("MS Data (Cx2) =",
+ array1dTo2d_i8(
+ compData.node_data,
+ compData.
+ node_data_size,
+ compData.header.
+ sense_node),
+ compData.header.force_node,
+ compData.header.sense_node);
+ }
+ break;
+
+ case 0x16: /* read self comp data */
+ pr_info("Get SS Compensation Data...\n");
+ res = readSelfSenseCompensationData(LOAD_CX_SS_TOUCH,
+ &comData);
+ if (res < 0)
+ pr_err("Error reading SS compensation data ERROR %08X\n",
+ res);
+ else {
+ pr_info("SS Compensation Data Reading Finished!\n");
+ size += ((comData.header.force_node +
+ comData.header.sense_node) * 2 + 6) *
+ sizeof(u8) * 2;
+ print_frame_u8("SS Data Ix2_fm = ",
+ array1dTo2d_u8(comData.ix2_fm,
+ comData.header.
+ force_node, 1),
+ comData.header.force_node, 1);
+ print_frame_i8("SS Data Cx2_fm = ",
+ array1dTo2d_i8(comData.cx2_fm,
+ comData.header.
+ force_node, 1),
+ comData.header.force_node, 1);
+ print_frame_u8("SS Data Ix2_sn = ",
+ array1dTo2d_u8(comData.ix2_sn,
+ comData.header.
+ sense_node,
+ comData.header.
+ sense_node), 1,
+ comData.header.sense_node);
+ print_frame_i8("SS Data Cx2_sn = ",
+ array1dTo2d_i8(comData.cx2_sn,
+ comData.header.
+ sense_node,
+ comData.header.
+ sense_node), 1,
+ comData.header.sense_node);
+ }
+ break;
+ case 0x17: /* Read mutual strength */
+ pr_info("Get 1 MS Strength\n");
+ setScanMode(SCAN_MODE_ACTIVE, 0xFF);
+ msleep(WAIT_FOR_FRESH_FRAMES);
+ setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ msleep(WAIT_AFTER_SENSEOFF);
+ /* Flush outstanding touch events */
+ flushFIFO();
+ nodes = getMSFrame3(MS_STRENGTH, &frameMS);
+ if (nodes < 0) {
+ res = nodes;
+ pr_err("Error while taking the MS strength... ERROR %08X\n",
+ res);
+ } else {
+ pr_info("The frame size is %d words\n", nodes);
+#ifdef RAW_DATA_FORMAT_DEC
+ size += 3 * 2 +
+ (7 * frameMS.header.sense_node + 1)
+ * frameMS.header.force_node;
+#else
+ size += (nodes * sizeof(short) + 2) * 2;
+#endif
+ print_frame_short("MS strength =",
+ array1dTo2d_short(frameMS.node_data,
+ frameMS.node_data_size,
+ frameMS.header.sense_node),
+ frameMS.header.force_node,
+ frameMS.header.sense_node);
+ res = OK;
+ }
+ break;
+ case 0x03: /* MS Raw DATA TEST */
+ res = fts_system_reset();
+ if (res >= OK)
+ res = production_test_ms_raw(LIMITS_FILE, 1,
+ &tests);
+ break;
+
+ case 0x04: /* MS CX DATA TEST */
+ res = fts_system_reset();
+ if (res >= OK)
+ res = production_test_ms_cx(LIMITS_FILE, 1,
+ &tests);
+ break;
+
+ case 0x05: /* SS RAW DATA TEST */
+ res = fts_system_reset();
+ if (res >= OK)
+ res = production_test_ss_raw(LIMITS_FILE, 1,
+ &tests);
+ break;
+
+ case 0x06: /* SS IX CX DATA TEST */
+ res = fts_system_reset();
+ if (res >= OK)
+ res = production_test_ss_ix_cx(LIMITS_FILE, 1,
+ &tests);
+ break;
+
+
+ case 0xF0:
+ case 0xF1: /* TOUCH ENABLE/DISABLE */
+ doClean = (int)(typeOfComand[0] & 0x01);
+ res = cleanUp(doClean);
+ break;
+
+ default:
+ pr_err("COMMAND NOT VALID!! Insert a proper value ...\n");
+ res = ERROR_OP_NOT_ALLOW;
+ break;
+ }
+
+ doClean = fts_mode_handler(info, 1);
+ if (typeOfComand[0] != 0xF0)
+ doClean |= fts_enableInterrupt();
+ if (doClean < 0)
+ pr_err("%s: ERROR %08X\n", __func__,
+ (doClean | ERROR_ENABLE_INTER));
+ } else {
+ pr_err("NO COMMAND SPECIFIED!!! do: 'echo [cmd_code] [args] > stm_fts_cmd' before looking for result!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+END:
+ /* here start the reporting phase, assembling the data
+ * to send in the file node */
+ size = PAGE_SIZE;
+ index = 0;
+ index += scnprintf(all_strbuff + index, size - index, "{ %08X", res);
+
+ if (res >= OK) {
+ /*all the other cases are already fine printing only the res.*/
+ switch (typeOfComand[0]) {
+ case 0x13:
+ case 0x17:
+#ifdef RAW_DATA_FORMAT_DEC
+ index += scnprintf(all_strbuff + index, size - index,
+ "%3d",
+ (u8)frameMS.header.force_node);
+ index += scnprintf(all_strbuff + index, size - index,
+ "%3d",
+ (u8)frameMS.header.sense_node);
+#else
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (u8)frameMS.header.force_node);
+
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (u8)frameMS.header.sense_node);
+#endif
+
+ for (j = 0; j < frameMS.node_data_size; j++) {
+#ifdef RAW_DATA_FORMAT_DEC
+ if (j % frameMS.header.sense_node == 0)
+ index += scnprintf(all_strbuff + index,
+ size - index, "\n");
+ index += scnprintf(all_strbuff + index,
+ size - index, "%7d",
+ frameMS.node_data[j]);
+#else
+ index += scnprintf(all_strbuff + index,
+ size - index,
+ "%02X%02X",
+ (frameMS.node_data[j] & 0xFF00) >> 8,
+ frameMS.node_data[j] & 0xFF);
+#endif
+ }
+
+ kfree(frameMS.node_data);
+ break;
+
+ case 0x15:
+#ifdef RAW_DATA_FORMAT_DEC
+ index += scnprintf(all_strbuff + index, size - index,
+ "%3d",
+ (u8)frameSS.header.force_node);
+ index += scnprintf(all_strbuff + index, size - index,
+ "%3d",
+ (u8)frameSS.header.sense_node);
+ index += scnprintf(all_strbuff + index, size - index,
+ "\n");
+#else
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (u8)frameSS.header.force_node);
+
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (u8)frameSS.header.sense_node);
+#endif
+
+ /* Copying self raw data Force */
+ for (j = 0; j < frameSS.header.force_node; j++) {
+#ifdef RAW_DATA_FORMAT_DEC
+ index += scnprintf(all_strbuff + index,
+ size - index,
+ "%7d",
+ frameSS.force_data[j]);
+#else
+ index += scnprintf(all_strbuff + index,
+ size - index,
+ "%02X%02X",
+ (frameSS.force_data[j] & 0xFF00) >> 8,
+ frameSS.force_data[j] & 0xFF);
+#endif
+ }
+
+
+
+#ifdef RAW_DATA_FORMAT_DEC
+ index += scnprintf(all_strbuff + index, size - index,
+ "\n");
+#endif
+
+ /* Copying self raw data Sense */
+ for (j = 0; j < frameSS.header.sense_node; j++) {
+#ifdef RAW_DATA_FORMAT_DEC
+ index += scnprintf(all_strbuff + index,
+ size - index, "%7d",
+ frameSS.sense_data[j]);
+#else
+ index += scnprintf(all_strbuff + index,
+ size - index,
+ "%02X%02X",
+ (frameSS.sense_data[j] & 0xFF00) >> 8,
+ frameSS.sense_data[j] & 0xFF);
+#endif
+ }
+
+ kfree(frameSS.force_data);
+ kfree(frameSS.sense_data);
+ break;
+
+ case 0x14:
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (u8)compData.header.force_node);
+
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (u8)compData.header.sense_node);
+
+ /* Cpying CX1 value */
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (compData.cx1) & 0xFF);
+
+ /* Copying CX2 values */
+ for (j = 0; j < compData.node_data_size; j++) {
+ index += scnprintf(all_strbuff + index,
+ size - index,
+ "%02X",
+ (compData.node_data[j]) & 0xFF);
+ }
+
+ kfree(compData.node_data);
+ break;
+
+ case 0x16:
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ comData.header.force_node);
+
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ comData.header.sense_node);
+
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (comData.f_ix1) & 0xFF);
+
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (comData.s_ix1) & 0xFF);
+
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (comData.f_cx1) & 0xFF);
+
+ index += scnprintf(all_strbuff + index,
+ size - index, "%02X",
+ (comData.s_cx1) & 0xFF);
+
+ /* Copying IX2 Force */
+ for (j = 0; j < comData.header.force_node; j++) {
+ index += scnprintf(all_strbuff + index,
+ size - index,
+ "%02X",
+ comData.ix2_fm[j] & 0xFF);
+ }
+
+ /* Copying IX2 Sense */
+ for (j = 0; j < comData.header.sense_node; j++) {
+ index += scnprintf(all_strbuff + index,
+ size - index,
+ "%02X",
+ comData.ix2_sn[j] & 0xFF);
+ }
+
+ /* Copying CX2 Force */
+ for (j = 0; j < comData.header.force_node; j++) {
+ index += scnprintf(all_strbuff + index,
+ size - index,
+ "%02X",
+ comData.cx2_fm[j] & 0xFF);
+ }
+
+ /* Copying CX2 Sense */
+ for (j = 0; j < comData.header.sense_node; j++) {
+ index += scnprintf(all_strbuff + index,
+ size - index,
+ "%02X",
+ comData.cx2_sn[j] & 0xFF);
+ }
+
+ kfree(comData.ix2_fm);
+ kfree(comData.ix2_sn);
+ kfree(comData.cx2_fm);
+ kfree(comData.cx2_sn);
+ break;
+
+ default:
+ break;
+ }
+ }
+
+ index += scnprintf(all_strbuff + index, size - index, " }\n");
+ numberParameters = 0;
+ /* need to reset the number of parameters in order to wait the
+ * next command, comment if you want to repeat the last command sent
+ * just doing a cat */
+ /* pr_err("numberParameters = %d\n", numberParameters); */
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ return index;
+}
+
+static DEVICE_ATTR(fwupdate, 0664, fts_fwupdate_show,
+ fts_fwupdate_store);
+static DEVICE_ATTR(appid, 0444, fts_appid_show, NULL);
+static DEVICE_ATTR(mode_active, 0444, fts_mode_active_show, NULL);
+static DEVICE_ATTR(fw_file_test, 0444, fts_fw_test_show, NULL);
+static DEVICE_ATTR(status, 0444, fts_status_show, NULL);
+static DEVICE_ATTR(stm_fts_cmd, 0664, stm_fts_cmd_show,
+ stm_fts_cmd_store);
+#ifdef USE_ONE_FILE_NODE
+static DEVICE_ATTR(feature_enable, 0664,
+ fts_feature_enable_show, fts_feature_enable_store);
+#else
+
+
+#ifdef GRIP_MODE
+static DEVICE_ATTR(grip_mode, 0664, fts_grip_mode_show,
+ fts_grip_mode_store);
+#endif
+
+#ifdef CHARGER_MODE
+static DEVICE_ATTR(charger_mode, 0664,
+ fts_charger_mode_show, fts_charger_mode_store);
+#endif
+
+#ifdef GLOVE_MODE
+static DEVICE_ATTR(glove_mode, 0664,
+ fts_glove_mode_show, fts_glove_mode_store);
+#endif
+
+#ifdef COVER_MODE
+static DEVICE_ATTR(cover_mode, 0664,
+ fts_cover_mode_show, fts_cover_mode_store);
+#endif
+
+#ifdef STYLUS_MODE
+static DEVICE_ATTR(stylus_mode, 0664,
+ fts_stylus_mode_show, fts_stylus_mode_store);
+#endif
+
+#endif
+
+#ifdef GESTURE_MODE
+static DEVICE_ATTR(gesture_mask, 0664,
+ fts_gesture_mask_show, fts_gesture_mask_store);
+static DEVICE_ATTR(gesture_coordinates, 0664,
+ fts_gesture_coordinates_show, NULL);
+#endif
+
+/* /sys/devices/soc.0/f9928000.i2c/i2c-6/6-0049 */
+static struct attribute *fts_attr_group[] = {
+ &dev_attr_fwupdate.attr,
+ &dev_attr_appid.attr,
+ &dev_attr_mode_active.attr,
+ &dev_attr_fw_file_test.attr,
+ &dev_attr_status.attr,
+ &dev_attr_stm_fts_cmd.attr,
+#ifdef USE_ONE_FILE_NODE
+ &dev_attr_feature_enable.attr,
+#else
+
+#ifdef GRIP_MODE
+ &dev_attr_grip_mode.attr,
+#endif
+#ifdef CHARGER_MODE
+ &dev_attr_charger_mode.attr,
+#endif
+#ifdef GLOVE_MODE
+ &dev_attr_glove_mode.attr,
+#endif
+#ifdef COVER_MODE
+ &dev_attr_cover_mode.attr,
+#endif
+#ifdef STYLUS_MODE
+ &dev_attr_stylus_mode.attr,
+#endif
+
+#endif
+
+#ifdef GESTURE_MODE
+ &dev_attr_gesture_mask.attr,
+ &dev_attr_gesture_coordinates.attr,
+#endif
+ NULL,
+};
+
+/** @}*/
+/** @}*/
+
+
+/**
+ * @defgroup isr Interrupt Service Routine (Event Handler)
+ * The most important part of the driver is the ISR (Interrupt Service Routine)
+ * called also as Event Handler \n
+ * As soon as the interrupt pin goes low, fts_interrupt_handler() is called and
+ * the chain to read and parse the event read from the FIFO start.\n
+ * For any different kind of EVT_ID there is a specific event handler
+ * which will take the correct action to report the proper info to the host. \n
+ * The most important events are the one related to touch information, status
+ * update or user report.
+ * @{
+ */
+
+/**
+ * Report to the linux input system the pressure and release of a button
+ * handling concurrency
+ * @param info pointer to fts_ts_info which contains info about the device
+ * and its hw setup
+ * @param key_code button value
+ */
+void fts_input_report_key(struct fts_ts_info *info, int key_code)
+{
+ mutex_lock(&info->input_report_mutex);
+ input_report_key(info->input_dev, key_code, 1);
+ input_sync(info->input_dev);
+ input_report_key(info->input_dev, key_code, 0);
+ input_sync(info->input_dev);
+ mutex_unlock(&info->input_report_mutex);
+}
+
+
+
+/**
+ * Event Handler for no events (EVT_ID_NOEVENT)
+ */
+static bool fts_nop_event_handler(struct fts_ts_info *info, unsigned
+ char *event)
+{
+ pr_info("%s: Doing nothing for event = %02X %02X %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[0], event[1], event[2], event[3],
+ event[4],
+ event[5], event[6], event[7]);
+ return false;
+}
+
+/**
+ * Event handler for enter and motion events (EVT_ID_ENTER_POINT,
+ * EVT_ID_MOTION_POINT )
+ * report touch coordinates and additional information
+ * to the linux input system
+ */
+static bool fts_enter_pointer_event_handler(struct fts_ts_info *info, unsigned
+ char *event)
+{
+ unsigned char touchId;
+ unsigned int touch_condition = 1, tool = MT_TOOL_FINGER;
+ int x, y, z, major, minor, distance;
+ u8 touchType;
+
+ if (!info->resume_bit)
+ goto no_report;
+
+ touchType = event[1] & 0x0F;
+ touchId = (event[1] & 0xF0) >> 4;
+
+ x = (((int)event[3] & 0x0F) << 8) | (event[2]);
+ y = ((int)event[4] << 4) | ((event[3] & 0xF0) >> 4);
+ z = (int)event[5];
+ if (z <= 0) {
+ /* Should not happen, because zero pressure implies contact has
+ * left, so this function should not be invoked. For safety, to
+ * prevent this touch from being dropped, set to smallest
+ * pressure value instead
+ */
+ pr_err("%s: Pressure is %i, but pointer is not leaving\n",
+ __func__, z);
+ z = 1; /* smallest non-zero pressure value */
+ }
+ major = (int)(((event[0] & 0x0C) << 2) | ((event[6] & 0xF0) >> 4));
+ minor = (int)(((event[7] & 0xC0) >> 2) | (event[6] & 0x0F));
+ /* TODO: check with fw how they will report distance */
+ distance = 0; /* if the tool is touching the display
+ * the distance should be 0 */
+
+ if (x == info->board->x_axis_max)
+ x--;
+
+ if (y == info->board->y_axis_max)
+ y--;
+
+ input_mt_slot(info->input_dev, touchId);
+ switch (touchType) {
+#ifdef STYLUS_MODE
+ case TOUCH_TYPE_STYLUS:
+ pr_info("%s : It is a stylus!\n", __func__);
+ if (info->stylus_enabled == 1) {
+ /* if stylus_enabled is not ==1
+ * it will be reported as normal touch */
+ tool = MT_TOOL_PEN;
+ touch_condition = 1;
+ __set_bit(touchId, &info->stylus_id);
+ break;
+ }
+#endif
+ /* TODO: customer can implement a different strategy for each kind of
+ * touch */
+ case TOUCH_TYPE_FINGER:
+ case TOUCH_TYPE_GLOVE:
+ case TOUCH_TYPE_PALM:
+ pr_debug("%s : It is a touch type %d!\n", __func__, touchType);
+ tool = MT_TOOL_FINGER;
+ touch_condition = 1;
+ __set_bit(touchId, &info->touch_id);
+ break;
+
+
+ case TOUCH_TYPE_HOVER:
+ tool = MT_TOOL_FINGER;
+ touch_condition = 0; /* need to hover */
+ z = 0; /* no pressure */
+ __set_bit(touchId, &info->touch_id);
+ distance = DISTANCE_MAX;/* check with fw report the hovering
+ * distance */
+ break;
+
+ case TOUCH_TYPE_INVALID:
+ default:
+ pr_err("%s : Invalid touch type = %d ! No Report...\n",
+ __func__, touchType);
+ goto no_report;
+ }
+
+ input_report_key(info->input_dev, BTN_TOUCH, touch_condition);
+ input_mt_report_slot_state(info->input_dev, tool, 1);
+
+ /* pr_info("%s : TouchID = %d,Touchcount = %d\n", __func__,
+ * touchId,touchcount); */
+
+ input_report_abs(info->input_dev, ABS_MT_POSITION_X, x);
+ input_report_abs(info->input_dev, ABS_MT_POSITION_Y, y);
+ input_report_abs(info->input_dev, ABS_MT_TOUCH_MAJOR, major);
+ input_report_abs(info->input_dev, ABS_MT_TOUCH_MINOR, minor);
+ input_report_abs(info->input_dev, ABS_MT_PRESSURE, z);
+
+#ifndef SKIP_DISTANCE
+ input_report_abs(info->input_dev, ABS_MT_DISTANCE, distance);
+#endif
+ /* pr_info("%s : Event 0x%02x - ID[%d], (x, y) = (%3d, %3d)
+ * Size = %d\n",
+ * __func__, *event, touchId, x, y, touchType); */
+
+ return true;
+no_report:
+ return false;
+}
+
+/**
+ * Event handler for leave event (EVT_ID_LEAVE_POINT )
+ * Report to the linux input system that one touch left the display
+ */
+static bool fts_leave_pointer_event_handler(struct fts_ts_info *info, unsigned
+ char *event)
+{
+ unsigned char touchId;
+ unsigned int tool = MT_TOOL_FINGER;
+ u8 touchType;
+
+ touchType = event[1] & 0x0F;
+ touchId = (event[1] & 0xF0) >> 4;
+
+ input_mt_slot(info->input_dev, touchId);
+
+ input_report_abs(info->input_dev, ABS_MT_PRESSURE, 0);
+ switch (touchType) {
+#ifdef STYLUS_MODE
+ case TOUCH_TYPE_STYLUS:
+ pr_info("%s : It is a stylus!\n", __func__);
+ if (info->stylus_enabled == 1) {
+ /* if stylus_enabled is not ==1 it will be reported as
+ * normal touch */
+ tool = MT_TOOL_PEN;
+ __clear_bit(touchId, &info->stylus_id);
+ break;
+ }
+#endif
+
+ case TOUCH_TYPE_FINGER:
+ /* pr_info("%s : It is a finger!\n", __func__); */
+ case TOUCH_TYPE_GLOVE:
+ /* pr_info("%s : It is a glove!\n", __func__); */
+ case TOUCH_TYPE_PALM:
+ /* pr_info("%s : It is a palm!\n", __func__); */
+ case TOUCH_TYPE_HOVER:
+ tool = MT_TOOL_FINGER;
+ __clear_bit(touchId, &info->touch_id);
+ break;
+
+ case TOUCH_TYPE_INVALID:
+ default:
+ pr_err("%s : Invalid touch type = %d ! No Report...\n",
+ __func__, touchType);
+ return false;
+ }
+
+ input_mt_report_slot_state(info->input_dev, tool, 0);
+
+ /* pr_info("%s : TouchID = %d, Touchcount = %d\n", __func__,
+ * touchId,touchcount); */
+
+
+ input_report_abs(info->input_dev, ABS_MT_TRACKING_ID, -1);
+ /* pr_info("%s : Event 0x%02x - release ID[%d]\n", __func__,
+ * event[0], touchId); */
+ return true;
+}
+
+/* EventId : EVT_ID_MOTION_POINT */
+#define fts_motion_pointer_event_handler fts_enter_pointer_event_handler
+/* remap the motion event handler to the same function which handle the enter
+ * event */
+
+/**
+ * Event handler for error events (EVT_ID_ERROR)
+ * Handle unexpected error events implementing recovery strategy and
+ * restoring the sensing status that the IC had before the error occurred
+ */
+static bool fts_error_event_handler(struct fts_ts_info *info, unsigned
+ char *event)
+{
+ int error = 0;
+
+ pr_info("%s: Received event %02X %02X %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[0], event[1], event[2], event[3], event[4],
+ event[5],
+ event[6], event[7]);
+
+ switch (event[1]) {
+ case EVT_TYPE_ERROR_ESD:/* esd */
+ {/* before reset clear all slot */
+ release_all_touches(info);
+
+ fts_chip_powercycle(info);
+
+ error = fts_system_reset();
+ error |= fts_mode_handler(info, 0);
+ error |= fts_enableInterrupt();
+ if (error < OK)
+ pr_err("%s Cannot restore the device ERROR %08X\n",
+ __func__, error);
+ }
+ break;
+ case EVT_TYPE_ERROR_WATCHDOG: /* watch dog timer */
+ {
+ dumpErrorInfo(NULL, 0);
+ /* before reset clear all slots */
+ release_all_touches(info);
+ error = fts_system_reset();
+ error |= fts_mode_handler(info, 0);
+ error |= fts_enableInterrupt();
+ if (error < OK)
+ pr_err("%s Cannot reset the device ERROR %08X\n",
+ __func__, error);
+ }
+ break;
+ }
+ return false;
+}
+
+/**
+ * Event handler for controller ready event (EVT_ID_CONTROLLER_READY)
+ * Handle controller events received after unexpected reset of the IC updating
+ * the resets flag and restoring the proper sensing status
+ */
+static bool fts_controller_ready_event_handler(struct fts_ts_info *info,
+ unsigned char *event)
+{
+ int error;
+
+ pr_info("%s: Received event %02X %02X %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[0], event[1], event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ release_all_touches(info);
+ setSystemResetedUp(1);
+ setSystemResetedDown(1);
+ error = fts_mode_handler(info, 0);
+ if (error < OK)
+ pr_err("%s Cannot restore the device status ERROR %08X\n",
+ __func__, error);
+ return false;
+}
+
+/**
+ * Event handler for status events (EVT_ID_STATUS_UPDATE)
+ * Handle status update events
+ */
+static bool fts_status_event_handler(struct fts_ts_info *info, unsigned
+ char *event)
+{
+ switch (event[1]) {
+ case EVT_TYPE_STATUS_ECHO:
+ pr_debug("%s: Echo event of command = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4], event[5],
+ event[6], event[7]);
+ break;
+
+ case EVT_TYPE_STATUS_FORCE_CAL:
+ switch (event[2]) {
+ case 0x00:
+ pr_info("%s: Continuous frame drop Force cal = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x01:
+ pr_info("%s: Mutual negative detect Force cal = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x02:
+ pr_info("%s: Mutual calib deviation Force cal = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x11:
+ pr_info("%s: SS negative detect Force cal = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x12:
+ pr_info("%s: SS negative detect Force cal in Low Power mode = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x13:
+ pr_info("%s: SS negative detect Force cal in Idle mode = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x20:
+ pr_info("%s: SS invalid Mutual Strength soft Force cal = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x21:
+ pr_info("%s: SS invalid Self Strength soft Force cal = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x22:
+ pr_info("%s: SS invalid Self Island soft Force cal = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x30:
+ pr_info("%s: MS invalid Mutual Strength soft Force cal = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x31:
+ pr_info("%s: MS invalid Self Strength soft Force cal = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ default:
+ pr_info("%s: Force cal = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ }
+ break;
+
+ case EVT_TYPE_STATUS_FRAME_DROP:
+ switch (event[2]) {
+ case 0x01:
+ pr_info("%s: Frame drop noisy frame = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x02:
+ pr_info("%s: Frame drop bad R = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case 0x03:
+ pr_info("%s: Frame drop invalid processing state = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ default:
+ pr_info("%s: Frame drop = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ }
+ break;
+
+ case EVT_TYPE_STATUS_SS_RAW_SAT:
+ if (event[2] == 1)
+ pr_info("%s: SS Raw Saturated = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ else
+ pr_info("%s: SS Raw No more Saturated = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ case EVT_TYPE_STATUS_WATER:
+ if (event[2] == 1)
+ pr_info("%s: Enter Water mode = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ else
+ pr_info("%s: Exit Water mode = %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+ break;
+
+ default:
+ pr_err("%s: Received unhandled status event = %02X %02X %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[0], event[1], event[2], event[3],
+ event[4], event[5], event[6], event[7]);
+ break;
+ }
+ return false;
+}
+
+
+/* key events reported in the user report */
+#ifdef PHONE_KEY
+/* TODO: the customer should handle the events coming from the keys according
+ * his needs
+ * (this is just an sample code that report the click of a button after a
+ * press->release action) */
+/**
+ * Event handler for status events (EVT_TYPE_USER_KEY)
+ * Handle keys update events, the third byte of the event is a bitmask,
+ * if the bit set means that the corresponding key is pressed.
+ */
+static void fts_key_event_handler(struct fts_ts_info *info,
+ unsigned char *event)
+{
+ /* int value; */
+ pr_info("%s: Received event %02X %02X %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[0], event[1], event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+
+ if (event[0] == EVT_ID_USER_REPORT && event[1] == EVT_TYPE_USER_KEY) {
+ /* event[2] contain the bitmask of the keys that are actually
+ * pressed */
+
+ if ((event[2] & FTS_KEY_0) == 0 && (key_mask & FTS_KEY_0) > 0) {
+ pr_info("%s: Button HOME pressed and released!\n",
+ __func__);
+ fts_input_report_key(info, KEY_HOMEPAGE);
+ }
+
+ if ((event[2] & FTS_KEY_1) == 0 && (key_mask & FTS_KEY_1) > 0) {
+ pr_info("%s: Button Back pressed and released!\n",
+ __func__);
+ fts_input_report_key(info, KEY_BACK);
+ }
+
+ if ((event[2] & FTS_KEY_2) == 0 && (key_mask & FTS_KEY_2) > 0) {
+ pr_info("%s: Button Menu pressed!\n", __func__);
+ fts_input_report_key(info, KEY_MENU);
+ }
+
+ key_mask = event[2];
+ } else
+ pr_err("%s: Invalid event passed as argument!\n", __func__);
+}
+#endif
+
+/* gesture event must be handled in the user event handler */
+#ifdef GESTURE_MODE
+/* TODO: Customer should implement their own actions in respond of a gesture
+ * event.
+ * This is an example that simply print the gesture received and simulate
+ * the click on a different button for each gesture. */
+/**
+ * Event handler for gesture events (EVT_TYPE_USER_GESTURE)
+ * Handle gesture events and simulate the click on a different button
+ * for any gesture detected (@link gesture_opt Gesture IDs @endlink)
+ */
+static void fts_gesture_event_handler(struct fts_ts_info *info, unsigned
+ char *event)
+{
+ int value;
+ int needCoords = 0;
+
+ pr_info("gesture event data: %02X %02X %02X %02X %02X %02X %02X %02X\n",
+ event[0], event[1], event[2], event[3], event[4],
+ event[5], event[6], event[7]);
+
+ if (event[0] == EVT_ID_USER_REPORT && event[1] ==
+ EVT_TYPE_USER_GESTURE) {
+ needCoords = 1;
+ /* default read the coordinates for all gestures excluding
+ * double tap */
+
+ switch (event[2]) {
+ case GEST_ID_DBLTAP:
+ value = KEY_WAKEUP;
+ pr_info("%s: double tap !\n", __func__);
+ needCoords = 0;
+ break;
+
+ case GEST_ID_AT:
+ value = KEY_WWW;
+ pr_info("%s: @ !\n", __func__);
+ break;
+
+ case GEST_ID_C:
+ value = KEY_C;
+ pr_info("%s: C !\n", __func__);
+ break;
+
+ case GEST_ID_E:
+ value = KEY_E;
+ pr_info("%s: e !\n", __func__);
+ break;
+
+ case GEST_ID_F:
+ value = KEY_F;
+ pr_info("%s: F !\n", __func__);
+ break;
+
+ case GEST_ID_L:
+ value = KEY_L;
+ pr_info("%s: L !\n", __func__);
+ break;
+
+ case GEST_ID_M:
+ value = KEY_M;
+ pr_info("%s: M !\n", __func__);
+ break;
+
+ case GEST_ID_O:
+ value = KEY_O;
+ pr_info("%s: O !\n", __func__);
+ break;
+
+ case GEST_ID_S:
+ value = KEY_S;
+ pr_info("%s: S !\n", __func__);
+ break;
+
+ case GEST_ID_V:
+ value = KEY_V;
+ pr_info("%s: V !\n", __func__);
+ break;
+
+ case GEST_ID_W:
+ value = KEY_W;
+ pr_info("%s: W !\n", __func__);
+ break;
+
+ case GEST_ID_Z:
+ value = KEY_Z;
+ pr_info("%s: Z !\n", __func__);
+ break;
+
+ case GEST_ID_RIGHT_1F:
+ value = KEY_RIGHT;
+ pr_info("%s: -> !\n", __func__);
+ break;
+
+ case GEST_ID_LEFT_1F:
+ value = KEY_LEFT;
+ pr_info("%s: <- !\n", __func__);
+ break;
+
+ case GEST_ID_UP_1F:
+ value = KEY_UP;
+ pr_info("%s: UP !\n", __func__);
+ break;
+
+ case GEST_ID_DOWN_1F:
+ value = KEY_DOWN;
+ pr_info("%s: DOWN !\n", __func__);
+ break;
+
+ case GEST_ID_CARET:
+ value = KEY_APOSTROPHE;
+ pr_info("%s: ^ !\n", __func__);
+ break;
+
+ case GEST_ID_LEFTBRACE:
+ value = KEY_LEFTBRACE;
+ pr_info("%s: < !\n", __func__);
+ break;
+
+ case GEST_ID_RIGHTBRACE:
+ value = KEY_RIGHTBRACE;
+ pr_info("%s: > !\n", __func__);
+ break;
+
+ default:
+ pr_err("%s: No valid GestureID!\n", __func__);
+ goto gesture_done;
+ }
+
+ if (needCoords == 1)
+ readGestureCoords(event);
+
+ fts_input_report_key(info, value);
+
+gesture_done:
+ return;
+ } else
+ pr_err("%s: Invalid event passed as argument!\n", __func__);
+}
+#endif
+
+
+/**
+ * Event handler for user report events (EVT_ID_USER_REPORT)
+ * Handle user events reported by the FW due to some interaction triggered
+ * by an external user (press keys, perform gestures, etc.)
+ */
+static bool fts_user_report_event_handler(struct fts_ts_info *info, unsigned
+ char *event)
+{
+ switch (event[1]) {
+#ifdef PHONE_KEY
+ case EVT_TYPE_USER_KEY:
+ fts_key_event_handler(info, event);
+ break;
+#endif
+
+ case EVT_TYPE_USER_PROXIMITY:
+ if (event[2] == 0)
+ pr_err("%s No proximity!\n", __func__);
+ else
+ pr_err("%s Proximity Detected!\n", __func__);
+ break;
+
+#ifdef GESTURE_MODE
+ case EVT_TYPE_USER_GESTURE:
+ fts_gesture_event_handler(info, event);
+ break;
+#endif
+ default:
+ pr_err("%s: Received unhandled user report event = %02X %02X %02X %02X %02X %02X %02X %02X\n",
+ __func__, event[0], event[1], event[2], event[3],
+ event[4], event[5], event[6], event[7]);
+ break;
+ }
+ return false;
+}
+
+static void heatmap_enable(void)
+{
+ unsigned char command[] = {0xA4, 0x06, LOCAL_HEATMAP_MODE};
+ fts_write(command, 3);
+}
+
+static bool read_heatmap_raw(struct v4l2_heatmap *v4l2, strength_t *data)
+{
+ unsigned char heatmap_read_command[] = {0xA6, 0x00, 0x00};
+
+ unsigned int num_elements;
+ /* index for looping through the heatmap buffer read over the bus */
+ unsigned int local_i;
+
+ int result;
+
+ /* old value of the counter, for comparison */
+ static uint16_t counter;
+
+ strength_t heatmap_value;
+ /* final position of the heatmap value in the full heatmap frame */
+ unsigned int frame_i;
+
+ int heatmap_x, heatmap_y;
+ int max_x = v4l2->format.width;
+ int max_y = v4l2->format.height;
+
+ struct heatmap_report report = {0};
+
+ result = fts_writeRead(heatmap_read_command, 3,
+ (uint8_t *) &report, sizeof(report));
+ if (result != OK) {
+ pr_err("%s: i2c read failed, fts_writeRead returned %i",
+ __func__, result);
+ return false;
+ }
+ if (report.mode != LOCAL_HEATMAP_MODE) {
+ pr_err("Touch IC not in local heatmap mode: %X %X %i",
+ report.prefix, report.mode, report.counter);
+ return false;
+ }
+
+ le16_to_cpus(&report.counter); /* enforce little-endian order */
+ if (report.counter == counter && counter != 0) {
+ /*
+ * We shouldn't make ordered comparisons because of
+ * potential overflow, but at least the value
+ * should have changed. If the value remains the same,
+ * but we are processing a new interrupt,
+ * this could indicate slowness in the interrupt handler.
+ */
+ pr_warn("Heatmap frame has stale counter value %i",
+ counter);
+ }
+ counter = report.counter;
+ num_elements = report.size_x * report.size_y;
+ if (num_elements > LOCAL_HEATMAP_WIDTH * LOCAL_HEATMAP_HEIGHT) {
+ pr_err("Unexpected heatmap size: %i x %i",
+ report.size_x, report.size_y);
+ return false;
+ }
+
+ /* set all to zero, will only write to non-zero locations in the loop */
+ memset(data, 0, max_x * max_y * sizeof(data[0]));
+ /* populate the data buffer, rearranging into final locations */
+ for (local_i = 0; local_i < num_elements; local_i++) {
+ /* enforce little-endian order */
+ le16_to_cpus(&report.data[local_i]);
+ heatmap_value = report.data[local_i];
+
+ if (heatmap_value == 0) {
+ /* Already set to zero. Nothing to do */
+ continue;
+ }
+
+ heatmap_x = report.offset_x + (local_i % report.size_x);
+ heatmap_y = report.offset_y + (local_i / report.size_x);
+
+ if (heatmap_x < 0 || heatmap_x >= max_x ||
+ heatmap_y < 0 || heatmap_y >= max_y) {
+ pr_err("Invalid x or y: (%i, %i), value=%i, ending loop\n",
+ heatmap_x, heatmap_y, heatmap_value);
+ return false;
+ }
+ frame_i = heatmap_y * max_x + heatmap_x;
+ data[frame_i] = heatmap_value;
+ }
+ return true;
+}
+
+/**
+ * Bottom Half Interrupt Handler function
+ * This handler is called each time there is at least one new event in the FIFO
+ * and the interrupt pin of the IC goes low. It will read all the events from
+ * the FIFO and dispatch them to the proper event handler according the event
+ * ID
+ */
+static irqreturn_t fts_interrupt_handler(int irq, void *handle)
+{
+ struct fts_ts_info *info = handle;
+ int error = 0, count = 0;
+ unsigned char regAdd = FIFO_CMD_READALL;
+ unsigned char data[FIFO_EVENT_SIZE * FIFO_DEPTH];
+ unsigned char eventId;
+ const unsigned char EVENTS_REMAINING_POS = 7;
+ const unsigned char EVENTS_REMAINING_MASK = 0x1F;
+ unsigned char events_remaining = 0;
+ unsigned char *evt_data;
+ event_dispatch_handler_t event_handler;
+ bool processed_pointer_event = false;
+
+ /* It is possible that interrupts were disabled while the handler is
+ * executing, before acquiring the mutex. If so, simply return.
+ */
+ if (fts_set_bus_ref(info, FTS_BUS_REF_IRQ, true) < 0) {
+ fts_set_bus_ref(info, FTS_BUS_REF_IRQ, false);
+ return IRQ_HANDLED;
+ }
+
+ /* prevent CPU from entering deep sleep */
+ pm_qos_update_request(&info->pm_qos_req, 100);
+
+ __pm_wakeup_event(&info->wakesrc, jiffies_to_msecs(HZ));
+
+ /* Read the first FIFO event and the number of events remaining */
+ error = fts_writeReadU8UX(regAdd, 0, 0, data, FIFO_EVENT_SIZE,
+ DUMMY_FIFO);
+ events_remaining = data[EVENTS_REMAINING_POS] & EVENTS_REMAINING_MASK;
+ events_remaining = (events_remaining > FIFO_DEPTH - 1) ?
+ FIFO_DEPTH - 1 : events_remaining;
+
+ /* Drain the rest of the FIFO, up to 31 events */
+ if (error == OK && events_remaining > 0) {
+ error = fts_writeReadU8UX(regAdd, 0, 0, &data[FIFO_EVENT_SIZE],
+ FIFO_EVENT_SIZE * events_remaining,
+ DUMMY_FIFO);
+ }
+ if (error != OK) {
+ pr_err("Error (%08X) while reading from FIFO in fts_event_handler\n",
+ error);
+ } else {
+ for (count = 0; count < events_remaining + 1; count++) {
+ evt_data = &data[count * FIFO_EVENT_SIZE];
+
+ if (evt_data[0] == EVT_ID_NOEVENT)
+ break;
+
+ eventId = evt_data[0] >> 4;
+
+ /* Ensure event ID is within bounds */
+ if (eventId < NUM_EVT_ID) {
+ event_handler =
+ info->event_dispatch_table[eventId];
+ processed_pointer_event =
+ event_handler(info, evt_data);
+ }
+ }
+ }
+
+ if (info->touch_id == 0)
+ input_report_key(info->input_dev, BTN_TOUCH, 0);
+
+ /*
+ * Only report timestamp for pointer events and ignore events
+ * like errors, status updates, etc.
+ * Otherwise, we will generate events that only consist of timestamps.
+ */
+ if (processed_pointer_event) {
+ fts_report_timestamp(info);
+ }
+ input_sync(info->input_dev);
+
+ heatmap_read(&info->v4l2, info->timestamp);
+
+ pm_qos_update_request(&info->pm_qos_req, PM_QOS_DEFAULT_VALUE);
+ fts_set_bus_ref(info, FTS_BUS_REF_IRQ, false);
+ return IRQ_HANDLED;
+}
+/** @}*/
+
+
+
+/**
+ * Implement the fw update and initialization flow of the IC that should
+ * be executed at every boot up. The function perform a fw update of the
+ * IC in case of crc error or a new fw version and then understand if the
+ * IC need to be re-initialized again.
+ *
+ * @return OK if success or an error code which specify the type of error
+ * encountered
+ */
+static int fts_fw_update(struct fts_ts_info *info)
+{
+ u8 error_to_search[4] = { EVT_TYPE_ERROR_CRC_CX_HEAD,
+ EVT_TYPE_ERROR_CRC_CX,
+ EVT_TYPE_ERROR_CRC_CX_SUB_HEAD,
+ EVT_TYPE_ERROR_CRC_CX_SUB };
+ int ret = 0;
+ int init_type = NO_INIT;
+
+#ifdef PRE_SAVED_METHOD
+ int keep_cx = 1;
+#else
+ int keep_cx = 0;
+#endif
+
+
+ pr_info("Fw Auto Update is starting...\n");
+
+ /* Check CRC status */
+ ret = fts_crc_check();
+ if (ret > OK) {
+ pr_err("%s: CRC Error or NO FW!\n", __func__);
+ info->reflash_fw = 1;
+ } else {
+ pr_info("%s: NO CRC Error or Impossible to read CRC register!\n",
+ __func__);
+ }
+ ret = flashProcedure(info->board->fw_name, info->reflash_fw, keep_cx);
+ if ((ret & 0xF000000F) == ERROR_FILE_NOT_FOUND) {
+ pr_err("%s: firmware file not found. Bypassing update.\n",
+ __func__);
+ ret = 0;
+ goto out;
+ } else if ((ret & 0xFF000000) == ERROR_FLASH_PROCEDURE) {
+ pr_err("%s: firmware update failed; retrying. ERROR %08X\n",
+ __func__, ret);
+ /* Power cycle the touch IC */
+ fts_chip_powercycle(info);
+ ret = flashProcedure(info->board->fw_name, info->reflash_fw,
+ keep_cx);
+ if ((ret & 0xFF000000) == ERROR_FLASH_PROCEDURE) {
+ pr_err("%s: firmware update failed again! ERROR %08X\n",
+ __func__, ret);
+ pr_err("Fw Auto Update Failed!\n");
+ return ret;
+ }
+ }
+ info->reflash_fw = 0;
+
+ pr_info("%s: Verifying if CX CRC Error...\n", __func__);
+ ret = fts_system_reset();
+ if (ret >= OK) {
+ ret = pollForErrorType(error_to_search, 4);
+ if (ret < OK) {
+ pr_info("%s: No Cx CRC Error Found!\n", __func__);
+ pr_info("%s: Verifying if Panel CRC Error...\n",
+ __func__);
+ error_to_search[0] = EVT_TYPE_ERROR_CRC_PANEL_HEAD;
+ error_to_search[1] = EVT_TYPE_ERROR_CRC_PANEL;
+ ret = pollForErrorType(error_to_search, 2);
+ if (ret < OK) {
+ pr_info("%s: No Panel CRC Error Found!\n",
+ __func__);
+ init_type = NO_INIT;
+ } else {
+ pr_err("%s: Panel CRC Error FOUND! CRC ERROR = %02X\n",
+ __func__, ret);
+ init_type = SPECIAL_PANEL_INIT;
+ }
+ } else {
+ pr_err("%s: Cx CRC Error FOUND! CRC ERROR = %02X\n",
+ __func__, ret);
+
+ /** This path of the code is used only in case there is
+ * a CRC error in code or config which not allow the fw
+ * to compute the CRC in the CX before
+ */
+ pr_info("%s: Try to recovery with CX in fw file...\n",
+ __func__);
+ ret = flashProcedure(info->board->fw_name, CRC_CX, 0);
+ pr_info("%s: Refresh panel init data", __func__);
+ }
+ } else {
+ /* Skip initialization because the real state is unknown */
+ pr_err("%s: Error while executing system reset! ERROR %08X\n",
+ __func__, ret);
+ }
+
+ if (init_type == NO_INIT) {
+#ifdef PRE_SAVED_METHOD
+ if (systemInfo.u8_cfgAfeVer != systemInfo.u8_cxAfeVer) {
+ init_type = SPECIAL_FULL_PANEL_INIT;
+ pr_err("%s: Different CX AFE Ver: %02X != %02X... Execute FULL Panel Init!\n",
+ __func__, systemInfo.u8_cfgAfeVer,
+ systemInfo.u8_cxAfeVer);
+ } else
+#endif
+ if (systemInfo.u8_cfgAfeVer != systemInfo.u8_panelCfgAfeVer) {
+ init_type = SPECIAL_PANEL_INIT;
+ pr_err("%s: Different Panel AFE Ver: %02X != %02X... Execute Panel Init!\n",
+ __func__, systemInfo.u8_cfgAfeVer,
+ systemInfo.u8_panelCfgAfeVer);
+ } else
+ init_type = NO_INIT;
+ }
+
+out:
+ /** Reinitialize after a complete FW update or if the initialization
+ * status is not correct.
+ */
+ if (init_type != NO_INIT) {
+ ret = fts_chip_initialization(info, init_type);
+ if (ret < OK) {
+ pr_err("%s: Cannot initialize the chip ERROR %08X\n",
+ __func__, ret);
+ }
+
+ /* Reset after initialization */
+ ret = fts_system_reset();
+ if (ret < OK) {
+ pr_err("%s: Reset failed, ERROR %08X\n", __func__,
+ ret);
+ }
+ }
+
+ ret = fts_init_sensing(info);
+ if (ret < OK) {
+ pr_err("Cannot initialize the hardware device ERROR %08X\n",
+ ret);
+ }
+
+ pr_err("Fw Update Finished! error = %08X\n", ret);
+ return ret;
+}
+
+#ifndef FW_UPDATE_ON_PROBE
+/**
+ * Function called by the delayed workthread executed after the probe in
+ * order to perform the fw update flow
+ * @see fts_fw_update()
+ */
+static void fts_fw_update_auto(struct work_struct *work)
+{
+ struct delayed_work *fwu_work = container_of(work, struct delayed_work,
+ work);
+ struct fts_ts_info *info = container_of(fwu_work, struct fts_ts_info,
+ fwu_work);
+
+ fts_set_bus_ref(info, FTS_BUS_REF_FW_UPDATE, true);
+ fts_fw_update(info);
+ fts_set_bus_ref(info, FTS_BUS_REF_FW_UPDATE, false);
+}
+#endif
+
+/* TODO: define if need to do the full mp at the boot */
+/**
+ * Execute the initialization of the IC (supporting a retry mechanism),
+ * checking also the resulting data
+ * @see production_test_main()
+ */
+static int fts_chip_initialization(struct fts_ts_info *info, int init_type)
+{
+ int ret2 = 0;
+ int retry;
+ int initretrycnt = 0;
+
+ /* initialization error, retry initialization */
+ for (retry = 0; retry <= RETRY_INIT_BOOT; retry++) {
+ ret2 = production_test_initialization(init_type);
+ if (ret2 == OK)
+ break;
+ initretrycnt++;
+ pr_err("initialization cycle count = %04d - ERROR %08X\n",
+ initretrycnt, ret2);
+ fts_chip_powercycle(info);
+ }
+
+ if (ret2 < OK) /* initialization error */
+ pr_err("fts initialization failed 3 times\n");
+
+
+ return ret2;
+}
+
+
+static irqreturn_t fts_isr(int irq, void *handle)
+{
+ struct fts_ts_info *info = handle;
+
+ info->timestamp = ktime_get_ns();
+
+ return IRQ_WAKE_THREAD;
+}
+
+/**
+ * Initialize the dispatch table with the event handlers for any possible event
+ * ID
+ * Set IRQ pin behavior (level triggered low)
+ * Register top half interrupt handler function.
+ * @see fts_interrupt_handler()
+ */
+static int fts_interrupt_install(struct fts_ts_info *info)
+{
+ int i, error = 0;
+
+ info->event_dispatch_table = kzalloc(sizeof(event_dispatch_handler_t) *
+ NUM_EVT_ID, GFP_KERNEL);
+
+ if (!info->event_dispatch_table) {
+ pr_err("OOM allocating event dispatch table\n");
+ return -ENOMEM;
+ }
+
+ for (i = 0; i < NUM_EVT_ID; i++)
+ info->event_dispatch_table[i] = fts_nop_event_handler;
+
+ install_handler(info, ENTER_POINT, enter_pointer);
+ install_handler(info, LEAVE_POINT, leave_pointer);
+ install_handler(info, MOTION_POINT, motion_pointer);
+ install_handler(info, ERROR, error);
+ install_handler(info, CONTROLLER_READY, controller_ready);
+ install_handler(info, STATUS_UPDATE, status);
+ install_handler(info, USER_REPORT, user_report);
+
+ /* disable interrupts in any case */
+ error = fts_disableInterrupt();
+ if (error) {
+ return error;
+ }
+
+ error = request_threaded_irq(info->client->irq, fts_isr,
+ fts_interrupt_handler, IRQF_ONESHOT | IRQF_TRIGGER_LOW,
+ FTS_TS_DRV_NAME, info);
+
+ if (error) {
+ pr_err("Request irq failed\n");
+ kfree(info->event_dispatch_table);
+ }
+
+ return error;
+}
+
+/**
+ * Clean the dispatch table and the free the IRQ.
+ * This function is called when the driver need to be removed
+ */
+static void fts_interrupt_uninstall(struct fts_ts_info *info)
+{
+ fts_disableInterrupt();
+
+ kfree(info->event_dispatch_table);
+
+ free_irq(info->client->irq, info);
+}
+
+/**@}*/
+
+/**
+ * This function try to attempt to communicate with the IC for the first time
+ * during the boot up process in order to read the necessary info for the
+ * following stages.
+ * The function execute a system reset, read fundamental info (system info)
+ * @return OK if success or an error code which specify the type of error
+ */
+static int fts_init(struct fts_ts_info *info)
+{
+ int error;
+
+
+ error = fts_system_reset();
+ if (error < OK && isI2cError(error)) {
+ pr_err("Cannot reset the device! ERROR %08X\n", error);
+ return error;
+ } else {
+ if (error == (ERROR_TIMEOUT | ERROR_SYSTEM_RESET_FAIL)) {
+ pr_err("Setting default Sys INFO!\n");
+ error = defaultSysInfo(0);
+ } else {
+ error = readSysInfo(0); /* system reset OK */
+ if (error < OK) {
+ if (!isI2cError(error))
+ error = OK;
+ pr_err("Cannot read Sys Info! ERROR %08X\n",
+ error);
+ }
+ }
+ }
+
+ return error;
+}
+
+/**
+ * Execute a power cycle in the IC, toggling the power lines (AVDD and DVDD)
+ * @param info pointer to fts_ts_info struct which contain information of the
+ * regulators
+ * @return 0 if success or another value if fail
+ */
+int fts_chip_powercycle(struct fts_ts_info *info)
+{
+ int error = 0;
+
+ pr_info("%s: Power Cycle Starting...\n", __func__);
+ pr_info("%s: Disabling IRQ...\n", __func__);
+ /** if IRQ pin is short with DVDD a call to the ISR will triggered when
+ * the regulator is turned off if IRQ not disabled */
+ fts_disableInterrupt();
+
+ if (info->vdd_reg) {
+ error = regulator_disable(info->vdd_reg);
+ if (error < 0)
+ pr_err("%s: Failed to disable DVDD regulator\n",
+ __func__);
+ }
+
+ if (info->avdd_reg) {
+ error = regulator_disable(info->avdd_reg);
+ if (error < 0)
+ pr_err("%s: Failed to disable AVDD regulator\n",
+ __func__);
+ }
+
+ if (info->board->reset_gpio != GPIO_NOT_DEFINED)
+ gpio_set_value(info->board->reset_gpio, 0);
+ else
+ mdelay(300);
+
+ /* in FTI power up first the digital and then the analog */
+ if (info->vdd_reg) {
+ error = regulator_enable(info->vdd_reg);
+ if (error < 0)
+ pr_err("%s: Failed to enable DVDD regulator\n",
+ __func__);
+ }
+
+ mdelay(1);
+
+ if (info->avdd_reg) {
+ error = regulator_enable(info->avdd_reg);
+ if (error < 0)
+ pr_err("%s: Failed to enable AVDD regulator\n",
+ __func__);
+ }
+
+ mdelay(5); /* time needed by the regulators for reaching the regime
+ * values */
+
+
+ if (info->board->reset_gpio != GPIO_NOT_DEFINED) {
+ mdelay(10); /* time to wait before bring up the reset
+ * gpio after the power up of the regulators */
+ gpio_set_value(info->board->reset_gpio, 1);
+ }
+
+ release_all_touches(info);
+
+ pr_info("%s: Power Cycle Finished! ERROR CODE = %08x\n",
+ __func__, error);
+ setSystemResetedUp(1);
+ setSystemResetedDown(1);
+ return error;
+}
+
+
+/**
+ * Complete the boot up process, initializing the sensing of the IC according
+ * to the current setting chosen by the host
+ * Register the notifier for the suspend/resume actions and the event handler
+ * @return OK if success or an error code which specify the type of error
+ */
+static int fts_init_sensing(struct fts_ts_info *info)
+{
+ int error = 0;
+
+ error |= msm_drm_register_client(&info->notifier);/* register the
+ * suspend/resume
+ * function */
+ error |= fts_interrupt_install(info); /* register event handler */
+ error |= fts_mode_handler(info, 0); /* enable the features and
+ * sensing */
+ /* error |= fts_enableInterrupt(); */ /* enable the interrupt */
+ error |= fts_resetDisableIrqCount();
+
+ if (error < OK)
+ pr_err("%s Init after Probe error (ERROR = %08X)\n",
+ __func__, error);
+
+ heatmap_enable();
+
+ return error;
+}
+
+/* TODO: change this function according with the needs of customer in terms
+ * of feature to enable/disable */
+
+/**
+ * @ingroup mode_section
+ * @{
+ */
+/**
+ * The function handle the switching of the mode in the IC enabling/disabling
+ * the sensing and the features set from the host
+ * @param info pointer to fts_ts_info which contains info about the device and
+ * its hw setup
+ * @param force if 1, the enabling/disabling command will be send even
+ * if the feature was already enabled/disabled otherwise it will judge if
+ * the feature changed status or the IC had a system reset
+ * @return OK if success or an error code which specify the type of error
+ */
+static int fts_mode_handler(struct fts_ts_info *info, int force)
+{
+ int res = OK;
+ int ret = OK;
+ u8 settings[4] = { 0 };
+
+ /* disable irq wake because resuming from gesture mode */
+ if (IS_POWER_MODE(info->mode, SCAN_MODE_LOW_POWER) &&
+ (info->resume_bit == 1))
+ disable_irq_wake(info->client->irq);
+
+ info->mode = MODE_NOTHING; /* initialize the mode to nothing
+ * in order to be updated depending
+ * on the features enabled */
+
+ pr_debug("%s: Mode Handler starting...\n", __func__);
+ switch (info->resume_bit) {
+ case 0: /* screen down */
+ pr_debug("%s: Screen OFF...\n", __func__);
+ /* do sense off in order to avoid the flooding of the fifo with
+ * touch events if someone is touching the panel during suspend
+ **/
+ pr_info("%s: Sense OFF!\n", __func__);
+ /* for speed reason (no need to check echo in this case and
+ * interrupt can be enabled) */
+ ret = setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ res |= ret; /* to avoid warning unsused ret variable when a
+ * ll the features are disabled */
+
+#ifdef GESTURE_MODE
+ if (info->gesture_enabled == 1) {
+ pr_info("%s: enter in gesture mode !\n",
+ __func__);
+ res = enterGestureMode(isSystemResettedDown());
+ if (res >= OK) {
+ enable_irq_wake(info->client->irq);
+ fromIDtoMask(FEAT_SEL_GESTURE,
+ (u8 *)&info->mode,
+ sizeof(info->mode));
+ MODE_LOW_POWER(info->mode, 0);
+ } else
+ pr_err("%s: enterGestureMode failed! ERROR %08X recovery in senseOff...\n",
+ __func__, res);
+ }
+#endif
+
+ setSystemResetedDown(0);
+ break;
+
+ case 1: /* screen up */
+ pr_debug("%s: Screen ON...\n", __func__);
+
+#ifdef GLOVE_MODE
+ if ((info->glove_enabled == FEAT_ENABLE &&
+ isSystemResettedUp()) || force == 1) {
+ pr_info("%s: Glove Mode setting...\n", __func__);
+ settings[0] = info->glove_enabled;
+ /* required to satisfy also the disable case */
+ ret = setFeatures(FEAT_SEL_GLOVE, settings, 1);
+ if (ret < OK)
+ pr_err("%s: error during setting GLOVE_MODE! ERROR %08X\n",
+ __func__, ret);
+ res |= ret;
+
+ if (ret >= OK && info->glove_enabled == FEAT_ENABLE) {
+ fromIDtoMask(FEAT_SEL_GLOVE, (u8 *)&info->mode,
+ sizeof(info->mode));
+ pr_info("%s: GLOVE_MODE Enabled!\n", __func__);
+ } else
+ pr_info("%s: GLOVE_MODE Disabled!\n", __func__);
+ }
+
+#endif
+
+#ifdef COVER_MODE
+ if ((info->cover_enabled == FEAT_ENABLE &&
+ isSystemResettedUp()) || force == 1) {
+ pr_info("%s: Cover Mode setting...\n", __func__);
+ settings[0] = info->cover_enabled;
+ ret = setFeatures(FEAT_SEL_COVER, settings, 1);
+ if (ret < OK)
+ pr_err("%s: error during setting COVER_MODE! ERROR %08X\n",
+ __func__, ret);
+ res |= ret;
+
+ if (ret >= OK && info->cover_enabled == FEAT_ENABLE) {
+ fromIDtoMask(FEAT_SEL_COVER, (u8 *)&info->mode,
+ sizeof(info->mode));
+ pr_info("%s: COVER_MODE Enabled!\n", __func__);
+ } else
+ pr_info("%s: COVER_MODE Disabled!\n", __func__);
+ }
+#endif
+#ifdef CHARGER_MODE
+ if ((info->charger_enabled > 0 && isSystemResettedUp()) ||
+ force == 1) {
+ pr_info("%s: Charger Mode setting...\n", __func__);
+
+ settings[0] = info->charger_enabled;
+ ret = setFeatures(FEAT_SEL_CHARGER, settings, 1);
+ if (ret < OK)
+ pr_err("%s: error during setting CHARGER_MODE! ERROR %08X\n",
+ __func__, ret);
+ res |= ret;
+
+ if (ret >= OK && info->charger_enabled == FEAT_ENABLE) {
+ fromIDtoMask(FEAT_SEL_CHARGER,
+ (u8 *)&info->mode,
+ sizeof(info->mode));
+ pr_info("%s: CHARGER_MODE Enabled!\n",
+ __func__);
+ } else
+ pr_info("%s: CHARGER_MODE Disabled!\n",
+ __func__);
+ }
+#endif
+
+
+#ifdef GRIP_MODE
+ if ((info->grip_enabled == FEAT_ENABLE &&
+ isSystemResettedUp()) || force == 1) {
+ pr_info("%s: Grip Mode setting...\n", __func__);
+ settings[0] = info->grip_enabled;
+ ret = setFeatures(FEAT_SEL_GRIP, settings, 1);
+ if (ret < OK)
+ pr_err("%s: error during setting GRIP_MODE! ERROR %08X\n",
+ __func__, ret);
+ res |= ret;
+
+ if (ret >= OK && info->grip_enabled == FEAT_ENABLE) {
+ fromIDtoMask(FEAT_SEL_GRIP, (u8 *)&info->mode,
+ sizeof(info->mode));
+ pr_info("%s: GRIP_MODE Enabled!\n", __func__);
+ } else
+ pr_info("%s: GRIP_MODE Disabled!\n", __func__);
+ }
+#endif
+ /* If some selective scan want to be enabled can be done
+ * an or of the following options
+ */
+ /* settings[0] = ACTIVE_MULTI_TOUCH | ACTIVE_KEY | */
+ /* ACTIVE_HOVER | ACTIVE_PROXIMITY | */
+ /* ACTIVE_FORCE; */
+ settings[0] = 0xFF; /* enable all the possible scans mode
+ * supported by the config */
+ pr_info("%s: Sense ON!\n", __func__);
+ res |= setScanMode(SCAN_MODE_ACTIVE, settings[0]);
+ info->mode |= (SCAN_MODE_ACTIVE << 24);
+ MODE_ACTIVE(info->mode, settings[0]);
+
+
+ setSystemResetedUp(0);
+ break;
+
+ default:
+ pr_err("%s: invalid resume_bit value = %d! ERROR %08X\n",
+ __func__, info->resume_bit, ERROR_OP_NOT_ALLOW);
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+
+ pr_debug("%s: Mode Handler finished! res = %08X mode = %08X\n",
+ __func__, res, info->mode);
+ return res;
+}
+
+/**
+ * Configure the switch GPIO to toggle bus master between AP and SLPI.
+ * gpio_value takes one of
+ * { FTS_SWITCH_GPIO_VALUE_SLPI_MASTER, FTS_SWITCH_GPIO_VALUE_AP_MASTER }
+ */
+static void fts_set_switch_gpio(struct fts_ts_info *info, int gpio_value)
+{
+ int retval;
+ unsigned int gpio = info->board->switch_gpio;
+
+ if (!gpio_is_valid(gpio))
+ return;
+
+ pr_debug("%s: toggling i2c switch to %s\n", __func__,
+ gpio_value == FTS_SWITCH_GPIO_VALUE_AP_MASTER ? "AP" : "SLPI");
+
+ retval = gpio_direction_output(gpio, gpio_value);
+ if (retval < 0)
+ pr_err("%s: Failed to toggle switch_gpio, err = %d\n",
+ __func__, retval);
+}
+
+/**
+ * Resume work function which perform a system reset, clean all the touches
+ * from the linux input system and prepare the ground for enabling the sensing
+ */
+static void fts_resume_work(struct work_struct *work)
+{
+ struct fts_ts_info *info;
+
+ info = container_of(work, struct fts_ts_info, resume_work);
+
+ if (!info->sensor_sleep)
+ return;
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ if (info->tbn)
+ tbn_request_bus(info->tbn);
+#endif
+
+ fts_set_switch_gpio(info, FTS_SWITCH_GPIO_VALUE_AP_MASTER);
+
+ __pm_wakeup_event(&info->wakesrc, jiffies_to_msecs(HZ));
+
+ info->resume_bit = 1;
+
+ fts_system_reset();
+
+ release_all_touches(info);
+
+ fts_mode_handler(info, 0);
+
+ info->sensor_sleep = false;
+
+ /* heatmap must be enabled after every chip reset (fts_system_reset) */
+ heatmap_enable();
+
+ fts_enableInterrupt();
+
+ complete_all(&info->bus_resumed);
+}
+
+/**
+ * Suspend work function which clean all the touches from Linux input system
+ * and prepare the ground to disabling the sensing or enter in gesture mode
+ */
+static void fts_suspend_work(struct work_struct *work)
+{
+ struct fts_ts_info *info;
+
+ info = container_of(work, struct fts_ts_info, suspend_work);
+
+ if (info->sensor_sleep)
+ return;
+
+ reinit_completion(&info->bus_resumed);
+
+ __pm_wakeup_event(&info->wakesrc, jiffies_to_msecs(HZ));
+
+ info->resume_bit = 0;
+
+ fts_mode_handler(info, 0);
+
+ release_all_touches(info);
+
+ info->sensor_sleep = true;
+
+ fts_disableInterrupt();
+
+ fts_set_switch_gpio(info, FTS_SWITCH_GPIO_VALUE_SLPI_MASTER);
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ if (info->tbn)
+ tbn_release_bus(info->tbn);
+#endif
+}
+/** @}*/
+
+
+static void fts_aggregate_bus_state(struct fts_ts_info *info)
+{
+ pr_debug("%s: bus_refmask = 0x%02X.\n", __func__,
+ info->bus_refmask);
+
+ /* Complete or cancel any outstanding transitions */
+ cancel_work_sync(&info->suspend_work);
+ cancel_work_sync(&info->resume_work);
+
+ if ((info->bus_refmask == 0 && info->sensor_sleep) ||
+ (info->bus_refmask != 0 && !info->sensor_sleep))
+ return;
+
+ if (info->bus_refmask == 0)
+ queue_work(info->event_wq, &info->suspend_work);
+ else
+ queue_work(info->event_wq, &info->resume_work);
+}
+
+int fts_set_bus_ref(struct fts_ts_info *info, u16 ref, bool enable)
+{
+ int result = OK;
+
+ mutex_lock(&info->bus_mutex);
+
+ if ((enable && (info->bus_refmask & ref)) ||
+ (!enable && !(info->bus_refmask & ref))) {
+ pr_err("%s: reference is unexpectedly set: mask=0x%04X, ref=0x%04X, enable=%d.\n",
+ __func__, info->bus_refmask, ref, enable);
+ mutex_unlock(&info->bus_mutex);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ if (enable) {
+ /* IRQs can only keep the bus active. IRQs received while the
+ * bus is transferred to SLPI should be ignored.
+ */
+ if (ref == FTS_BUS_REF_IRQ && info->bus_refmask == 0)
+ result = ERROR_OP_NOT_ALLOW;
+ else
+ info->bus_refmask |= ref;
+ } else
+ info->bus_refmask &= ~ref;
+ fts_aggregate_bus_state(info);
+
+ mutex_unlock(&info->bus_mutex);
+
+ /* When triggering a wake, wait up to one second to resume. SCREEN_ON
+ * and IRQ references do not need to wait.
+ */
+ if (enable && ref != FTS_BUS_REF_SCREEN_ON && ref != FTS_BUS_REF_IRQ) {
+ wait_for_completion_timeout(&info->bus_resumed, HZ);
+ if (info->sensor_sleep) {
+ pr_err("%s: Failed to wake the touch bus: mask=0x%04X, ref=0x%04X, enable=%d.\n",
+ __func__, info->bus_refmask, ref, enable);
+ result = ERROR_TIMEOUT;
+ }
+ }
+
+ return result;
+}
+
+/**
+ * Callback function used to detect the suspend/resume events generated by
+ * clicking the power button.
+ * This function schedule a suspend or resume work according to the event
+ * received.
+ */
+static int fts_screen_state_chg_callback(struct notifier_block *nb,
+ unsigned long val, void *data)
+{
+ struct fts_ts_info *info = container_of(nb, struct fts_ts_info,
+ notifier);
+ struct msm_drm_notifier *evdata = data;
+ unsigned int blank;
+
+ if (val != MSM_DRM_EVENT_BLANK && val != MSM_DRM_EARLY_EVENT_BLANK)
+ return NOTIFY_DONE;
+
+ if (!info || !evdata || !evdata->data) {
+ pr_info("%s: Bad fts notifier call!\n", __func__);
+ return NOTIFY_DONE;
+ }
+
+ pr_debug("%s: fts notifier begin!\n", __func__);
+
+ /* finish processing any events on queue */
+ flush_workqueue(info->event_wq);
+
+ blank = *(int *) (evdata->data);
+ switch (blank) {
+ case MSM_DRM_BLANK_POWERDOWN:
+ case MSM_DRM_BLANK_LP:
+ if (val == MSM_DRM_EARLY_EVENT_BLANK) {
+ pr_info("%s: BLANK\n", __func__);
+ fts_set_bus_ref(info, FTS_BUS_REF_SCREEN_ON, false);
+ }
+ break;
+ case MSM_DRM_BLANK_UNBLANK:
+ if (val == MSM_DRM_EVENT_BLANK) {
+ pr_info("%s: UNBLANK\n", __func__);
+ fts_set_bus_ref(info, FTS_BUS_REF_SCREEN_ON, true);
+ }
+ break;
+ }
+ return NOTIFY_OK;
+}
+
+static struct notifier_block fts_noti_block = {
+ .notifier_call = fts_screen_state_chg_callback,
+};
+
+/**
+ * From the name of the power regulator get/put the actual regulator structs
+ * (copying their references into fts_ts_info variable)
+ * @param info pointer to fts_ts_info which contains info about the device and
+ * its hw setup
+ * @param get if 1, the regulators are get otherwise they are put (released)
+ * back to the system
+ * @return OK if success or an error code which specify the type of error
+ */
+static int fts_get_reg(struct fts_ts_info *info, bool get)
+{
+ int retval;
+ const struct fts_hw_platform_data *bdata = info->board;
+
+ if (!get) {
+ retval = 0;
+ goto regulator_put;
+ }
+
+ if ((bdata->vdd_reg_name != NULL) && (*bdata->vdd_reg_name != 0)) {
+ info->vdd_reg = regulator_get(info->dev, bdata->vdd_reg_name);
+ if (IS_ERR(info->vdd_reg)) {
+ pr_err("%s: Failed to get power regulator\n", __func__);
+ retval = PTR_ERR(info->vdd_reg);
+ goto regulator_put;
+ }
+ }
+
+ if ((bdata->avdd_reg_name != NULL) && (*bdata->avdd_reg_name != 0)) {
+ info->avdd_reg = regulator_get(info->dev, bdata->avdd_reg_name);
+ if (IS_ERR(info->avdd_reg)) {
+ pr_err("%s: Failed to get bus pullup regulator\n",
+ __func__);
+ retval = PTR_ERR(info->avdd_reg);
+ goto regulator_put;
+ }
+ }
+
+ return OK;
+
+regulator_put:
+ if (info->vdd_reg) {
+ regulator_put(info->vdd_reg);
+ info->vdd_reg = NULL;
+ }
+
+ if (info->avdd_reg) {
+ regulator_put(info->avdd_reg);
+ info->avdd_reg = NULL;
+ }
+
+ return retval;
+}
+
+
+/**
+ * Enable or disable the power regulators
+ * @param info pointer to fts_ts_info which contains info about the device and
+ * its hw setup
+ * @param enable if 1, the power regulators are turned on otherwise they are
+ * turned off
+ * @return OK if success or an error code which specify the type of error
+ */
+static int fts_enable_reg(struct fts_ts_info *info, bool enable)
+{
+ int retval;
+
+ if (!enable) {
+ retval = 0;
+ goto disable_pwr_reg;
+ }
+
+ if (info->vdd_reg) {
+ retval = regulator_enable(info->vdd_reg);
+ if (retval < 0) {
+ pr_err("%s: Failed to enable bus regulator\n",
+ __func__);
+ goto exit;
+ }
+ }
+
+ if (info->avdd_reg) {
+ retval = regulator_enable(info->avdd_reg);
+ if (retval < 0) {
+ pr_err("%s: Failed to enable power regulator\n",
+ __func__);
+ goto disable_bus_reg;
+ }
+ }
+
+ return OK;
+
+disable_pwr_reg:
+ if (info->avdd_reg)
+ regulator_disable(info->avdd_reg);
+
+disable_bus_reg:
+ if (info->vdd_reg)
+ regulator_disable(info->vdd_reg);
+
+exit:
+ return retval;
+}
+
+/**
+ * Configure a GPIO according to the parameters
+ * @param gpio gpio number
+ * @param config if true, the gpio is set up otherwise it is free
+ * @param dir direction of the gpio, 0 = in, 1 = out
+ * @param state initial value (if the direction is in, this parameter is
+ * ignored)
+ * return error code
+ */
+static int fts_gpio_setup(int gpio, bool config, int dir, int state)
+{
+ int retval = 0;
+ unsigned char buf[16];
+
+ if (config) {
+ scnprintf(buf, sizeof(buf), "fts_gpio_%u\n", gpio);
+
+ retval = gpio_request(gpio, buf);
+ if (retval) {
+ pr_err("%s: Failed to get gpio %d (code: %d)",
+ __func__, gpio, retval);
+ return retval;
+ }
+
+ if (dir == 0)
+ retval = gpio_direction_input(gpio);
+ else
+ retval = gpio_direction_output(gpio, state);
+ if (retval) {
+ pr_err("%s: Failed to set gpio %d direction",
+ __func__, gpio);
+ return retval;
+ }
+ } else
+ gpio_free(gpio);
+
+ return retval;
+}
+
+/**
+ * Setup the IRQ and RESET (if present) gpios.
+ * If the Reset Gpio is present it will perform a cycle HIGH-LOW-HIGH in order
+ * to assure that the IC has been reset properly
+ */
+static int fts_set_gpio(struct fts_ts_info *info)
+{
+ int retval;
+ struct fts_hw_platform_data *bdata =
+ info->board;
+
+ retval = fts_gpio_setup(bdata->irq_gpio, true, 0, 0);
+ if (retval < 0) {
+ pr_err("%s: Failed to configure irq GPIO\n", __func__);
+ goto err_gpio_irq;
+ }
+
+ if (gpio_is_valid(bdata->switch_gpio)) {
+ retval = fts_gpio_setup(bdata->switch_gpio, true, 1, 1);
+ if (retval < 0)
+ pr_err("%s: Failed to configure I2C switch\n",
+ __func__);
+ }
+
+ if (bdata->reset_gpio >= 0) {
+ retval = fts_gpio_setup(bdata->reset_gpio, true, 1, 0);
+ if (retval < 0) {
+ pr_err("%s: Failed to configure reset GPIO\n",
+ __func__);
+ goto err_gpio_reset;
+ }
+ }
+ if (bdata->reset_gpio >= 0) {
+ gpio_set_value(bdata->reset_gpio, 0);
+ mdelay(10);
+ gpio_set_value(bdata->reset_gpio, 1);
+ }
+
+ return OK;
+
+err_gpio_reset:
+ fts_gpio_setup(bdata->irq_gpio, false, 0, 0);
+ bdata->reset_gpio = GPIO_NOT_DEFINED;
+err_gpio_irq:
+ return retval;
+}
+
+/**
+ * Retrieve and parse the hw information from the device tree node defined in
+ * the system.
+ * the most important information to obtain are: IRQ and RESET gpio numbers,
+ * power regulator names
+ * In the device file node is possible to define additional optional
+ * information
+ * that can be parsed here.
+ */
+static int parse_dt(struct device *dev, struct fts_hw_platform_data *bdata)
+{
+ int retval;
+ const char *name;
+ struct device_node *np = dev->of_node;
+ u32 coords[2];
+
+ bdata->switch_gpio = of_get_named_gpio(np, "st,switch_gpio", 0);
+ pr_info("switch_gpio = %d\n", bdata->switch_gpio);
+
+ bdata->irq_gpio = of_get_named_gpio_flags(np, "st,irq-gpio", 0, NULL);
+
+ pr_info("irq_gpio = %d\n", bdata->irq_gpio);
+
+
+ retval = of_property_read_string(np, "st,regulator_dvdd", &name);
+ if (retval == -EINVAL)
+ bdata->vdd_reg_name = NULL;
+ else if (retval < 0)
+ return retval;
+ else {
+ bdata->vdd_reg_name = name;
+ pr_info("pwr_reg_name = %s\n", name);
+ }
+
+ retval = of_property_read_string(np, "st,regulator_avdd", &name);
+ if (retval == -EINVAL)
+ bdata->avdd_reg_name = NULL;
+ else if (retval < 0)
+ return retval;
+ else {
+ bdata->avdd_reg_name = name;
+ pr_info("bus_reg_name = %s\n", name);
+ }
+
+ if (of_property_read_bool(np, "st,reset-gpio")) {
+ bdata->reset_gpio = of_get_named_gpio_flags(np,
+ "st,reset-gpio", 0,
+ NULL);
+ pr_info("reset_gpio =%d\n", bdata->reset_gpio);
+ } else
+ bdata->reset_gpio = GPIO_NOT_DEFINED;
+
+ retval = of_property_read_string(np, "st,firmware_name", &name);
+ if (retval == -EINVAL)
+ bdata->fw_name = PATH_FILE_FW;
+ else if (retval >= 0)
+ bdata->fw_name = name;
+ pr_info("firmware name = %s\n", bdata->fw_name);
+
+ if (of_property_read_u32_array(np, "st,max-coords", coords, 2)) {
+ pr_err("st,max-coords not found, using 1440x2560\n");
+ coords[0] = 1440;
+ coords[1] = 2560;
+ }
+ bdata->x_axis_max = coords[0];
+ bdata->y_axis_max = coords[1];
+
+ return OK;
+}
+
+/**
+ * Probe function, called when the driver it is matched with a device
+ * with the same name compatible name
+ * This function allocate, initialize all the most important functions and flow
+ * those are used by the driver to operate with the IC.
+ * It allocates device variables, initialize queues and schedule works,
+ * registers the IRQ handler, suspend/resume callbacks, registers the device
+ * to the linux input subsystem etc.
+ */
+#ifdef I2C_INTERFACE
+static int fts_probe(struct i2c_client *client, const struct i2c_device_id *idp)
+{
+#else
+static int fts_probe(struct spi_device *client)
+{
+#endif
+
+ struct fts_ts_info *info = NULL;
+ int error = 0;
+ struct device_node *dp = client->dev.of_node;
+ int retval;
+ int skip_5_1 = 0;
+ u16 bus_type;
+
+ pr_info("%s: driver probe begin!\n", __func__);
+ pr_info("driver ver. %s\n", FTS_TS_DRV_VERSION);
+
+ pr_info("SET Bus Functionality :\n");
+#ifdef I2C_INTERFACE
+ pr_info("I2C interface...\n");
+ if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
+ pr_err("Unsupported I2C functionality\n");
+ error = -EIO;
+ goto ProbeErrorExit_0;
+ }
+
+ pr_info("i2c address: %x\n", client->addr);
+ bus_type = BUS_I2C;
+#else
+ pr_info("SPI interface...\n");
+ client->mode = SPI_MODE_0;
+#ifndef SPI4_WIRE
+ client->mode |= SPI_3WIRE;
+#endif
+
+ client->max_speed_hz = SPI_CLOCK_FREQ;
+ client->bits_per_word = 8;
+ if (spi_setup(client) < 0) {
+ pr_err("Unsupported SPI functionality\n");
+ error = -EIO;
+ goto ProbeErrorExit_0;
+ }
+ bus_type = BUS_SPI;
+#endif
+
+
+ pr_info("SET Device driver INFO:\n");
+
+
+ info = kzalloc(sizeof(struct fts_ts_info), GFP_KERNEL);
+ if (!info) {
+ pr_err("Out of memory... Impossible to allocate struct info!\n");
+ error = -ENOMEM;
+ goto ProbeErrorExit_0;
+ }
+
+ info->client = client;
+ info->dev = &info->client->dev;
+
+ dev_set_drvdata(info->dev, info);
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ info->tbn = tbn_init(info->dev);
+ if (!info->tbn) {
+ pr_err("ERROR: failed to init tbn context\n");
+ error = -ENODEV;
+ goto ProbeErrorExit_1;
+ }
+#endif
+
+ if (dp) {
+ info->board = devm_kzalloc(&client->dev,
+ sizeof(struct fts_hw_platform_data),
+ GFP_KERNEL);
+ if (!info->board) {
+ pr_err("ERROR:info.board kzalloc failed\n");
+ goto ProbeErrorExit_1;
+ }
+ parse_dt(&client->dev, info->board);
+ }
+
+ pr_info("SET Regulators:\n");
+ retval = fts_get_reg(info, true);
+ if (retval < 0) {
+ pr_err("ERROR: %s: Failed to get regulators\n", __func__);
+ goto ProbeErrorExit_1;
+ }
+
+ retval = fts_enable_reg(info, true);
+ if (retval < 0) {
+ pr_err("%s: ERROR Failed to enable regulators\n", __func__);
+ goto ProbeErrorExit_2;
+ }
+
+ pr_info("SET GPIOS:\n");
+ retval = fts_set_gpio(info);
+ if (retval < 0) {
+ pr_err("%s: ERROR Failed to set up GPIO's\n", __func__);
+ goto ProbeErrorExit_2;
+ }
+ info->client->irq = gpio_to_irq(info->board->irq_gpio);
+
+ pr_info("SET Event Handler:\n");
+
+ wakeup_source_init(&info->wakesrc, "fts_tp");
+ info->event_wq = alloc_workqueue("fts-event-queue", WQ_UNBOUND |
+ WQ_HIGHPRI | WQ_CPU_INTENSIVE, 1);
+ if (!info->event_wq) {
+ pr_err("ERROR: Cannot create work thread\n");
+ error = -ENOMEM;
+ goto ProbeErrorExit_4;
+ }
+
+ INIT_WORK(&info->resume_work, fts_resume_work);
+ INIT_WORK(&info->suspend_work, fts_suspend_work);
+
+ init_completion(&info->bus_resumed);
+ complete_all(&info->bus_resumed);
+
+ pr_info("SET Input Device Property:\n");
+ info->dev = &info->client->dev;
+ info->input_dev = input_allocate_device();
+ if (!info->input_dev) {
+ pr_err("ERROR: No such input device defined!\n");
+ error = -ENODEV;
+ goto ProbeErrorExit_5;
+ }
+ info->input_dev->dev.parent = &client->dev;
+ info->input_dev->name = FTS_TS_DRV_NAME;
+ scnprintf(fts_ts_phys, sizeof(fts_ts_phys), "%s/input0",
+ info->input_dev->name);
+ info->input_dev->phys = fts_ts_phys;
+ info->input_dev->id.bustype = bus_type;
+ info->input_dev->id.vendor = 0x0001;
+ info->input_dev->id.product = 0x0002;
+ info->input_dev->id.version = 0x0100;
+
+ __set_bit(EV_SYN, info->input_dev->evbit);
+ __set_bit(EV_KEY, info->input_dev->evbit);
+ __set_bit(EV_ABS, info->input_dev->evbit);
+ __set_bit(BTN_TOUCH, info->input_dev->keybit);
+ /* __set_bit(BTN_TOOL_FINGER, info->input_dev->keybit); */
+ /* __set_bit(BTN_TOOL_PEN, info->input_dev->keybit); */
+
+ input_mt_init_slots(info->input_dev, TOUCH_ID_MAX, INPUT_MT_DIRECT);
+
+ /* input_mt_init_slots(info->input_dev, TOUCH_ID_MAX); */
+
+ input_set_abs_params(info->input_dev, ABS_MT_POSITION_X, X_AXIS_MIN,
+ info->board->x_axis_max, 0, 0);
+ input_set_abs_params(info->input_dev, ABS_MT_POSITION_Y, Y_AXIS_MIN,
+ info->board->y_axis_max, 0, 0);
+ input_set_abs_params(info->input_dev, ABS_MT_TOUCH_MAJOR, AREA_MIN,
+ AREA_MAX, 0, 0);
+ input_set_abs_params(info->input_dev, ABS_MT_TOUCH_MINOR, AREA_MIN,
+ AREA_MAX, 0, 0);
+ input_set_abs_params(info->input_dev, ABS_MT_PRESSURE, PRESSURE_MIN,
+ PRESSURE_MAX, 0, 0);
+#ifndef SKIP_DISTANCE
+ input_set_abs_params(info->input_dev, ABS_MT_DISTANCE, DISTANCE_MIN,
+ DISTANCE_MAX, 0, 0);
+#endif
+ input_set_capability(info->input_dev, EV_MSC, MSC_TIMESTAMP);
+
+#ifdef GESTURE_MODE
+ input_set_capability(info->input_dev, EV_KEY, KEY_WAKEUP);
+
+ input_set_capability(info->input_dev, EV_KEY, KEY_M);
+ input_set_capability(info->input_dev, EV_KEY, KEY_O);
+ input_set_capability(info->input_dev, EV_KEY, KEY_E);
+ input_set_capability(info->input_dev, EV_KEY, KEY_W);
+ input_set_capability(info->input_dev, EV_KEY, KEY_C);
+ input_set_capability(info->input_dev, EV_KEY, KEY_L);
+ input_set_capability(info->input_dev, EV_KEY, KEY_F);
+ input_set_capability(info->input_dev, EV_KEY, KEY_V);
+ input_set_capability(info->input_dev, EV_KEY, KEY_S);
+ input_set_capability(info->input_dev, EV_KEY, KEY_Z);
+ input_set_capability(info->input_dev, EV_KEY, KEY_WWW);
+
+ input_set_capability(info->input_dev, EV_KEY, KEY_LEFT);
+ input_set_capability(info->input_dev, EV_KEY, KEY_RIGHT);
+ input_set_capability(info->input_dev, EV_KEY, KEY_UP);
+ input_set_capability(info->input_dev, EV_KEY, KEY_DOWN);
+
+ input_set_capability(info->input_dev, EV_KEY, KEY_F1);
+ input_set_capability(info->input_dev, EV_KEY, KEY_F2);
+ input_set_capability(info->input_dev, EV_KEY, KEY_F3);
+ input_set_capability(info->input_dev, EV_KEY, KEY_F4);
+ input_set_capability(info->input_dev, EV_KEY, KEY_F5);
+
+ input_set_capability(info->input_dev, EV_KEY, KEY_LEFTBRACE);
+ input_set_capability(info->input_dev, EV_KEY, KEY_RIGHTBRACE);
+#endif
+
+#ifdef PHONE_KEY
+ /* KEY associated to the touch screen buttons */
+ input_set_capability(info->input_dev, EV_KEY, KEY_HOMEPAGE);
+ input_set_capability(info->input_dev, EV_KEY, KEY_BACK);
+ input_set_capability(info->input_dev, EV_KEY, KEY_MENU);
+#endif
+
+ mutex_init(&(info->input_report_mutex));
+ mutex_init(&info->bus_mutex);
+
+ /* Assume screen is on throughout probe */
+ info->bus_refmask = FTS_BUS_REF_SCREEN_ON;
+
+#ifdef GESTURE_MODE
+ mutex_init(&gestureMask_mutex);
+#endif
+
+ spin_lock_init(&fts_int);
+
+ /* register the multi-touch input device */
+ error = input_register_device(info->input_dev);
+ if (error) {
+ pr_err("ERROR: No such input device\n");
+ error = -ENODEV;
+ goto ProbeErrorExit_5_1;
+ }
+
+ skip_5_1 = 1;
+ /* track slots */
+ info->touch_id = 0;
+#ifdef STYLUS_MODE
+ info->stylus_id = 0;
+#endif
+
+
+ /* init feature switches (by default all the features are disable,
+ * if one feature want to be enabled from the start,
+ * set the corresponding value to 1)*/
+ info->gesture_enabled = 0;
+ info->glove_enabled = 0;
+ info->charger_enabled = 0;
+ info->cover_enabled = 0;
+ info->grip_enabled = 0;
+
+ info->resume_bit = 1;
+ info->notifier = fts_noti_block;
+
+ /*
+ * This *must* be done before request_threaded_irq is called.
+ * Otherwise, if an interrupt is received before request is added,
+ * but after the interrupt has been subscribed to, pm_qos_req
+ * may be accessed before initialization in the interrupt handler.
+ */
+ pm_qos_add_request(&info->pm_qos_req, PM_QOS_CPU_DMA_LATENCY,
+ PM_QOS_DEFAULT_VALUE);
+
+ pr_info("Init Core Lib:\n");
+ initCore(info);
+ /* init hardware device */
+ pr_info("Device Initialization:\n");
+ error = fts_init(info);
+ if (error < OK) {
+ pr_err("Cannot initialize the device ERROR %08X\n", error);
+ error = -ENODEV;
+ goto ProbeErrorExit_6;
+ }
+
+ /*
+ * Heatmap_probe must be called before irq routine is registered,
+ * because heatmap_read is called from interrupt context.
+ * This is done as part of fwu_work.
+ * At the same time, heatmap_probe must be done after fts_init(..) has
+ * completed, because getForceLen() and getSenseLen() require
+ * the chip to be initialized.
+ */
+ info->v4l2.parent_dev = info->dev;
+ info->v4l2.input_dev = info->input_dev;
+ info->v4l2.read_frame = read_heatmap_raw;
+ info->v4l2.width = getForceLen();
+ info->v4l2.height = getSenseLen();
+ /* 120 Hz operation */
+ info->v4l2.timeperframe.numerator = 1;
+ info->v4l2.timeperframe.denominator = 120;
+ error = heatmap_probe(&info->v4l2);
+ if (error < OK)
+ goto ProbeErrorExit_6;
+
+#if defined(FW_UPDATE_ON_PROBE) && defined(FW_H_FILE)
+ pr_info("FW Update and Sensing Initialization:\n");
+ error = fts_fw_update(info);
+ if (error < OK) {
+ pr_err("Cannot execute fw upgrade the device ERROR %08X\n",
+ error);
+ error = -ENODEV;
+ goto ProbeErrorExit_7;
+ }
+
+#else
+ pr_info("SET Auto Fw Update:\n");
+ info->fwu_workqueue = alloc_workqueue("fts-fwu-queue", WQ_UNBOUND |
+ WQ_HIGHPRI | WQ_CPU_INTENSIVE, 1);
+ if (!info->fwu_workqueue) {
+ pr_err("ERROR: Cannot create fwu work thread\n");
+ goto ProbeErrorExit_7;
+ }
+ INIT_DELAYED_WORK(&info->fwu_work, fts_fw_update_auto);
+#endif
+
+ pr_info("SET Device File Nodes:\n");
+ /* sysfs stuff */
+ info->attrs.attrs = fts_attr_group;
+ error = sysfs_create_group(&client->dev.kobj, &info->attrs);
+ if (error) {
+ pr_err("ERROR: Cannot create sysfs structure!\n");
+ error = -ENODEV;
+ goto ProbeErrorExit_7;
+ }
+
+ error = fts_proc_init();
+ if (error < OK)
+ pr_err("Error: can not create /proc file!\n");
+
+#ifndef FW_UPDATE_ON_PROBE
+ queue_delayed_work(info->fwu_workqueue, &info->fwu_work,
+ msecs_to_jiffies(EXP_FN_WORK_DELAY_MS));
+#endif
+
+ pr_info("Probe Finished!\n");
+
+ return OK;
+
+
+ProbeErrorExit_7:
+#ifdef FW_UPDATE_ON_PROBE
+ msm_drm_unregister_client(&info->notifier);
+#endif
+
+ heatmap_remove(&info->v4l2);
+
+ProbeErrorExit_6:
+ pm_qos_remove_request(&info->pm_qos_req);
+ input_unregister_device(info->input_dev);
+
+ProbeErrorExit_5_1:
+ if (skip_5_1 != 1)
+ input_free_device(info->input_dev);
+
+ProbeErrorExit_5:
+ destroy_workqueue(info->event_wq);
+
+ProbeErrorExit_4:
+ /* destroy_workqueue(info->fwu_workqueue); */
+ wakeup_source_trash(&info->wakesrc);
+
+ fts_enable_reg(info, false);
+
+ProbeErrorExit_2:
+ fts_get_reg(info, false);
+
+ProbeErrorExit_1:
+ kfree(info);
+
+ProbeErrorExit_0:
+ pr_err("Probe Failed!\n");
+
+ return error;
+}
+
+
+/**
+ * Clear and free all the resources associated to the driver.
+ * This function is called when the driver need to be removed.
+ */
+#ifdef I2C_INTERFACE
+static int fts_remove(struct i2c_client *client)
+{
+#else
+static int fts_remove(struct spi_device *client)
+{
+#endif
+
+ struct fts_ts_info *info = dev_get_drvdata(&(client->dev));
+
+ /* Force the bus active throughout removal of the client */
+ fts_set_bus_ref(info, FTS_BUS_REF_FORCE_ACTIVE, true);
+
+ pr_info("%s\n", __func__);
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ tbn_cleanup(info->tbn);
+#endif
+
+ fts_proc_remove();
+
+ /* sysfs stuff */
+ sysfs_remove_group(&client->dev.kobj, &info->attrs);
+
+ /* remove interrupt and event handlers */
+ fts_interrupt_uninstall(info);
+
+ heatmap_remove(&info->v4l2);
+
+ pm_qos_remove_request(&info->pm_qos_req);
+
+ msm_drm_unregister_client(&info->notifier);
+
+ /* unregister the device */
+ input_unregister_device(info->input_dev);
+
+ /* input_free_device(info->input_dev ); */
+
+ /* Remove the work thread */
+ destroy_workqueue(info->event_wq);
+ wakeup_source_trash(&info->wakesrc);
+#ifndef FW_UPDATE_ON_PROBE
+ destroy_workqueue(info->fwu_workqueue);
+#endif
+
+ fts_enable_reg(info, false);
+ fts_get_reg(info, false);
+
+ /* free gpio */
+ if (gpio_is_valid(info->board->irq_gpio))
+ gpio_free(info->board->irq_gpio);
+ if (gpio_is_valid(info->board->switch_gpio))
+ gpio_free(info->board->switch_gpio);
+ if (gpio_is_valid(info->board->reset_gpio))
+ gpio_free(info->board->reset_gpio);
+
+ /* free all */
+ kfree(info);
+
+ return OK;
+}
+
+/**
+ * Struct which contains the compatible names that need to match with
+ * the definition of the device in the device tree node
+ */
+static struct of_device_id fts_of_match_table[] = {
+ {
+ .compatible = "st,fts",
+ },
+ {},
+};
+
+#ifdef I2C_INTERFACE
+static const struct i2c_device_id fts_device_id[] = {
+ { FTS_TS_DRV_NAME, 0 },
+ {}
+};
+
+static struct i2c_driver fts_i2c_driver = {
+ .driver = {
+ .name = FTS_TS_DRV_NAME,
+ .of_match_table = fts_of_match_table,
+ },
+ .probe = fts_probe,
+ .remove = fts_remove,
+ .id_table = fts_device_id,
+};
+#else
+static struct spi_driver fts_spi_driver = {
+ .driver = {
+ .name = FTS_TS_DRV_NAME,
+ .of_match_table = fts_of_match_table,
+ .owner = THIS_MODULE,
+ },
+ .probe = fts_probe,
+ .remove = fts_remove,
+};
+#endif
+
+
+
+
+static int __init fts_driver_init(void)
+{
+#ifdef I2C_INTERFACE
+ return i2c_add_driver(&fts_i2c_driver);
+#else
+ return spi_register_driver(&fts_spi_driver);
+#endif
+}
+
+static void __exit fts_driver_exit(void)
+{
+ pr_info("%s\n", __func__);
+#ifdef I2C_INTERFACE
+ i2c_del_driver(&fts_i2c_driver);
+#else
+ spi_unregister_driver(&fts_spi_driver);
+#endif
+}
+
+
+MODULE_DESCRIPTION("STMicroelectronics MultiTouch IC Driver");
+MODULE_AUTHOR("STMicroelectronics");
+MODULE_LICENSE("GPL");
+
+late_initcall(fts_driver_init);
+module_exit(fts_driver_exit);
diff --git a/drivers/input/touchscreen/stm/fts.h b/drivers/input/touchscreen/stm/fts.h
new file mode 100644
index 0000000..cb741d3
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts.h
@@ -0,0 +1,404 @@
+/*
+ * fts.h
+ *
+ * FTS Capacitive touch screen controller (FingerTipS)
+ *
+ * Copyright (C) 2017, STMicroelectronics
+ * Authors: AMG(Analog Mems Group)
+ *
+ * marco.cali@st.com
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * THE PRESENT SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, FOR THE SOLE
+ * PURPOSE TO SUPPORT YOUR APPLICATION DEVELOPMENT.
+ * AS A RESULT, STMICROELECTRONICS SHALL NOT BE HELD LIABLE FOR ANY DIRECT,
+ * INDIRECT OR CONSEQUENTIAL DAMAGES WITH RESPECT TO ANY CLAIMS ARISING FROM
+ * THE
+ * CONTENT OF SUCH SOFTWARE AND/OR THE USE MADE BY CUSTOMERS OF THE CODING
+ * INFORMATION CONTAINED HEREIN IN CONNECTION WITH THEIR PRODUCTS.
+ *
+ * THIS SOFTWARE IS SPECIFICALLY DESIGNED FOR EXCLUSIVE USE WITH ST PARTS.
+ */
+
+/*!
+ * \file fts.h
+ * \brief Contains all the definitions and structs used generally by the driver
+ */
+
+#ifndef _LINUX_FTS_I2C_H_
+#define _LINUX_FTS_I2C_H_
+
+#include <linux/device.h>
+#include <linux/input/heatmap.h>
+#include <linux/pm_qos.h>
+#include "fts_lib/ftsSoftware.h"
+#include "fts_lib/ftsHardware.h"
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+#include "../touch_bus_negotiator.h"
+#endif
+
+
+/****************** CONFIGURATION SECTION ******************/
+/** @defgroup conf_section Driver Configuration Section
+ * Settings of the driver code in order to suit the HW set up and the
+ *application behavior
+ * @{
+ */
+/* **** CODE CONFIGURATION **** */
+#define FTS_TS_DRV_NAME "fts" /* driver name */
+#define FTS_TS_DRV_VERSION "5.2.10_Google_B1" /* driver version string
+ * */
+#define FTS_TS_DRV_VER 0x05020A00 /* driver version u32 format */
+
+/* #define DEBUG */ /* /< define to print more logs in the kernel log
+ * and better follow the code flow */
+#ifdef pr_fmt
+#undef pr_fmt
+#define pr_fmt(fmt) "[ FTS ] " fmt
+#endif
+
+#define DRIVER_TEST /* /< if defined allow to use and test special functions
+ * of the driver and fts_lib from command shell
+ * (useful for enginering/debug operations) */
+
+/* Comment 2 flags to disable auto-tune in MP test and device boot-up */
+/* if defined allow to have some procedures at the boot or from file node to
+ * assure that touch works under any condition that usually are disabled in the
+ * MP stage of the project
+ */
+/* #define ENGINEERING_CODE */
+/* Initialization of CX memory allowed on the phone */
+/* #define COMPUTE_CX_ON_PHONE */
+/* #define PRE_SAVED_METHOD */
+
+/*#define FW_H_FILE*/ /* include the FW data as header file */
+#ifdef FW_H_FILE
+#define FW_SIZE_NAME myArray_size /* FW data array size */
+#define FW_ARRAY_NAME myArray /* FW data array name */
+/*#define FW_UPDATE_ON_PROBE*/ /* No delay updating FW */
+#endif
+
+#ifndef FW_UPDATE_ON_PROBE
+/* Include the Production Limit File as header file, can be commented to use a
+ * .csv file instead */
+/* #define LIMITS_H_FILE */
+#ifdef LIMITS_H_FILE
+ #define LIMITS_SIZE_NAME myArray2_size /* /< name of the
+ * variable
+ * in the limits header
+ *file which
+ * specified the
+ *dimension of
+ * the limits data
+ *array */
+ #define LIMITS_ARRAY_NAME myArray2 /* /< name of the
+ * variable in
+ * the limits header
+ *file which
+ * specified the limits
+ *data array */
+#endif
+#else
+/* if execute fw update in the probe the limit file must be a .h */
+#define LIMITS_H_FILE /* /< include the Production Limit File as header file,
+ * DO NOT COMMENT! */
+#define LIMITS_SIZE_NAME myArray2_size /* /< name of the
+ * variable
+ * in the limits header
+ *file
+ * which specified the
+ *dimension
+ * of the limits data
+ *array */
+#define LIMITS_ARRAY_NAME myArray2 /* /< name of the
+ * variable in the
+ * limits header file
+ *which specified
+ * the limits data
+ *array */
+#endif
+
+/* #define USE_ONE_FILE_NODE */ /* /< allow to enable/disable all the features
+ * just using one file node */
+
+#ifndef FW_UPDATE_ON_PROBE
+#define EXP_FN_WORK_DELAY_MS 1000 /* /< time in ms elapsed after the probe
+ * to start the work which execute FW
+ *update
+ * and the Initialization of the IC */
+#endif
+
+/* **** END **** */
+
+
+/* **** FEATURES USED IN THE IC **** */
+/* Enable the support of keys */
+/* #define PHONE_KEY */
+
+#define GESTURE_MODE /* /< enable the support of the gestures */
+#ifdef GESTURE_MODE
+ #define USE_GESTURE_MASK /* /< the gestures to select are
+ * referred using
+ * a gesture bitmask instead of their
+ *gesture IDs */
+#endif
+
+
+#define CHARGER_MODE /* /< enable the support to charger mode feature
+ * (comment to disable) */
+
+#define GLOVE_MODE /* /< enable the support to glove mode feature (comment
+ * to disable) */
+
+#define COVER_MODE /* /< enable the support to cover mode feature (comment
+ * to disable) */
+
+#define STYLUS_MODE /* /< enable the support to stylus mode feature (comment
+ * to disable) */
+
+#define GRIP_MODE /* /< enable the support to grip mode feature (comment
+ * to disable) */
+
+
+/* **** END **** */
+
+
+/* **** PANEL SPECIFICATION **** */
+#define X_AXIS_MIN 0 /* /< min X coordinate of the display */
+#define Y_AXIS_MIN 0 /* /< min Y coordinate of the display */
+#define Y_AXIS_MAX 2959 /* /< Max Y coordinate of the display */
+#define X_AXIS_MAX 1440 /* /< Max X coordinate of the display */
+
+#define PRESSURE_MIN 0 /* /< min value of pressure reported */
+#define PRESSURE_MAX 127 /* /< Max value of pressure reported */
+
+#define DISTANCE_MIN 0 /* /< min distance between the tool and the
+ * display */
+#define DISTANCE_MAX 127 /* /< Max distance between the tool and the
+ * display */
+
+#define TOUCH_ID_MAX 10 /* /< Max number of simoultaneous touches
+ * reported */
+
+#define AREA_MIN PRESSURE_MIN /* /< min value of Major/minor axis
+ * reported */
+#define AREA_MAX PRESSURE_MAX /* /< Man value of Major/minor axis
+ * reported */
+/* **** END **** */
+
+/**@}*/
+/*********************************************************/
+
+/* **** LOCAL HEATMAP FEATURE *** */
+#define LOCAL_HEATMAP_WIDTH 7
+#define LOCAL_HEATMAP_HEIGHT 7
+#define LOCAL_HEATMAP_MODE 0xC1
+
+struct heatmap_report {
+ uint8_t prefix; /* always should be 0xA0 */
+ uint8_t mode; /* mode should be 0xC1 for heatmap */
+
+ uint16_t counter; /* LE order, should increment on each heatmap read */
+ int8_t offset_x;
+ uint8_t size_x;
+ int8_t offset_y;
+ uint8_t size_y;
+ /* data is in LE order; order should be enforced after data is read */
+ strength_t data[LOCAL_HEATMAP_WIDTH * LOCAL_HEATMAP_HEIGHT];
+} __attribute__((packed));
+/* **** END **** */
+
+/*
+ * Configuration mode
+ *
+ * bitmask which can assume the value defined as features in ftsSoftware.h or
+ * the following values
+ */
+
+/** @defgroup mode_section IC Status Mode
+ * Bitmask which keeps track of the features and working mode enabled in the
+ * IC.
+ * The meaning of the the LSB of the bitmask must be interpreted considering
+ * that the value defined in @link feat_opt Feature Selection Option @endlink
+ * correspond to the position of the corresponding bit in the mask
+ * @{
+ */
+#define MODE_NOTHING 0x00000000 /* /< nothing enabled (sense off) */
+#define MODE_ACTIVE(_mask, _sett) \
+ (_mask |= (SCAN_MODE_ACTIVE << 24) | (_sett << 16))
+/* /< store the status of scan mode active and its setting */
+#define MODE_LOW_POWER(_mask, _sett) \
+ (_mask |= (SCAN_MODE_LOW_POWER << 24) | (_sett << 16))
+/* /< store the status of scan mode low power and its setting */
+#define IS_POWER_MODE(_mask, _mode) ((_mask&(_mode<<24)) != 0x00)
+/* /< check the current mode of the IC */
+
+/** @}*/
+
+#define CMD_STR_LEN 32 /* /< max number of parameters that can accept
+ * the
+ * MP file node (stm_fts_cmd) */
+
+#define TSP_BUF_SIZE PAGE_SIZE /* /< max number of bytes printable on
+ * the shell in the normal file nodes
+ **/
+
+
+/**
+ * Struct which contains information about the HW platform and set up
+ */
+struct fts_hw_platform_data {
+ int (*power) (bool on);
+ int switch_gpio;/* (optional) I2C switch */
+ int irq_gpio; /* /< number of the gpio associated to the interrupt pin
+ * */
+ int reset_gpio; /* /< number of the gpio associated to the reset pin */
+ const char *vdd_reg_name; /* /< name of the VDD regulator */
+ const char *avdd_reg_name; /* /< name of the AVDD regulator */
+ const char *fw_name;
+ int x_axis_max;
+ int y_axis_max;
+};
+
+/* Bits for the bus reference mask */
+enum {
+ FTS_BUS_REF_SCREEN_ON = 0x01,
+ FTS_BUS_REF_IRQ = 0x02,
+ FTS_BUS_REF_FW_UPDATE = 0x04,
+ FTS_BUS_REF_SYSFS = 0x08,
+ FTS_BUS_REF_FORCE_ACTIVE = 0x10
+};
+
+/*
+ * Forward declaration
+ */
+struct fts_ts_info;
+
+/*
+ * Dispatch event handler
+ * Return true if the handler has processed a pointer event
+ */
+typedef bool (*event_dispatch_handler_t)
+ (struct fts_ts_info *info, unsigned char *data);
+
+/**
+ * FTS capacitive touch screen device information
+ * - dev Pointer to the structure device \n
+ * - client client structure \n
+ * - input_dev Input device structure \n
+ * - work Work thread \n
+ * - event_wq Event queue for work thread \n
+ * - event_dispatch_table Event dispatch table handlers \n
+ * - attrs SysFS attributes \n
+ * - mode Device operating mode (bitmask) \n
+ * - touch_id Bitmask for touch id (mapped to input slots) \n
+ * - stylus_id Bitmask for tracking the stylus touches (mapped using the
+ * touchId) \n
+ * - timer Timer when operating in polling mode \n
+ * - power Power on/off routine \n
+ * - board HW info retrieved from device tree \n
+ * - vdd_reg DVDD power regulator \n
+ * - avdd_reg AVDD power regulator \n
+ * - resume_bit Indicate if screen off/on \n
+ * - fwupdate_stat Store the result of a fw update triggered by the host \n
+ * - notifier Used for be notified from a suspend/resume event \n
+ * - sensor_sleep true suspend was called, false resume was called \n
+ * - wakesrc Wakeup Source struct \n
+ * - input_report_mutex mutex for handling the pressure of keys \n
+ * - series_of_switches to store the enabling status of a particular feature
+ * from the host \n
+ * - tbn Touch Bus Negotiator context
+ */
+struct fts_ts_info {
+ struct device *dev; /* Pointer to the device */
+#ifdef I2C_INTERFACE
+ struct i2c_client *client; /* I2C client structure */
+#else
+ struct spi_device *client; /* SPI client structure */
+#endif
+ struct input_dev *input_dev; /* Input device structure */
+
+ struct work_struct suspend_work; /* Suspend work thread */
+ struct work_struct resume_work; /* Resume work thread */
+ struct workqueue_struct *event_wq; /* Used for event handler, */
+ /* suspend, resume threads */
+
+ struct completion bus_resumed; /* resume_work complete */
+
+ struct pm_qos_request pm_qos_req;
+
+ struct v4l2_heatmap v4l2;
+
+#ifndef FW_UPDATE_ON_PROBE
+ struct delayed_work fwu_work; /* Work for fw update */
+ struct workqueue_struct *fwu_workqueue; /* Fw update work queue */
+#endif
+ event_dispatch_handler_t *event_dispatch_table; /* Dispatch table */
+
+ struct attribute_group attrs; /* SysFS attributes */
+
+ unsigned int mode; /* Device operating mode */
+ /* MSB - active or lpm */
+ unsigned long touch_id; /* Bitmask for touch id */
+#ifdef STYLUS_MODE
+ unsigned long stylus_id; /* Bitmask for the stylus */
+#endif
+
+ u64 timestamp; /* nanoseconds, acquired during hard interrupt */
+
+ struct fts_hw_platform_data *board; /* HW info from device tree */
+ struct regulator *vdd_reg; /* DVDD power regulator */
+ struct regulator *avdd_reg; /* AVDD power regulator */
+
+ struct mutex bus_mutex; /* Protect access to the bus */
+ unsigned int bus_refmask; /* References to the bus */
+
+ int resume_bit; /* Indicate if screen off/on */
+ int fwupdate_stat; /* Result of a fw update */
+ int reflash_fw; /* Attempt to reflash fw */
+
+ struct notifier_block notifier; /* Notify on suspend/resume */
+ bool sensor_sleep; /* True if suspend called */
+ struct wakeup_source wakesrc; /* Wake Lock struct */
+
+ /* input lock */
+ struct mutex input_report_mutex; /* Mutex for pressure report */
+
+ /* switches for features */
+ int gesture_enabled; /* Gesture during suspend */
+ int glove_enabled; /* Glove mode */
+ int charger_enabled; /* Charger mode */
+ int stylus_enabled; /* Stylus mode */
+ int cover_enabled; /* Cover mode */
+ int grip_enabled; /* Grip mode */
+
+#ifdef CONFIG_TOUCHSCREEN_TBN
+ struct tbn_context *tbn;
+#endif
+
+ /* Preallocated i/o read buffer */
+ u8 io_read_buf[READ_CHUNK + DUMMY_FIFO];
+ /* Preallocated i/o write buffer */
+ u8 io_write_buf[WRITE_CHUNK + BITS_64 + DUMMY_FIFO];
+ /* Preallocated i/o extra write buffer */
+ u8 io_extra_write_buf[WRITE_CHUNK + BITS_64 + DUMMY_FIFO];
+
+};
+
+int fts_chip_powercycle(struct fts_ts_info *info);
+extern int input_register_notifier_client(struct notifier_block *nb);
+extern int input_unregister_notifier_client(struct notifier_block *nb);
+
+/* export declaration of functions in fts_proc.c */
+extern int fts_proc_init(void);
+extern int fts_proc_remove(void);
+
+/* Bus reference tracking */
+int fts_set_bus_ref(struct fts_ts_info *info, u16 ref, bool enable);
+
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_fw.h b/drivers/input/touchscreen/stm/fts_fw.h
new file mode 100644
index 0000000..b1d7c8d
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_fw.h
@@ -0,0 +1,6146 @@
+#ifndef FTS_FW_V5_H
+#define FTS_FW_V5_H
+//This is an auto generated header file
+//--->Remember to change the name of the two variables!<---
+const uint32_t myArray_size = 98168;
+
+const uint8_t myArray[] = {
+ 0x55, 0xAA, 0x55, 0xAA, 0x01, 0x00, 0x00, 0x00, 0x36, 0x39, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00,
+ 0x10, 0x00, 0x00, 0x00, 0x10, 0x00, 0x10, 0x00, 0x10, 0x00, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xD4, 0x74, 0x01, 0x00,
+ 0x60, 0x0A, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x44, 0x13, 0x78, 0xF8,
+ 0x65, 0xFF, 0xBA, 0x4A, 0x34, 0x5D, 0x00, 0x00, 0x9A, 0x00, 0x45, 0xB5, 0xCB, 0xA2, 0xFF, 0xFF,
+ 0x00, 0x04, 0x10, 0x00, 0xE1, 0x01, 0x00, 0x00, 0x39, 0x02, 0x00, 0x00, 0x53, 0x02, 0x00, 0x00,
+ 0x6D, 0x02, 0x00, 0x00, 0x87, 0x02, 0x00, 0x00, 0xA1, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF7, 0x43, 0x01, 0x00,
+ 0xFF, 0x43, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0xBB, 0x02, 0x00, 0x00, 0x07, 0x44, 0x01, 0x00,
+ 0x0F, 0x44, 0x01, 0x00, 0x1D, 0x44, 0x01, 0x00, 0x2B, 0x44, 0x01, 0x00, 0x39, 0x44, 0x01, 0x00,
+ 0x47, 0x44, 0x01, 0x00, 0x55, 0x44, 0x01, 0x00, 0x63, 0x44, 0x01, 0x00, 0x71, 0x44, 0x01, 0x00,
+ 0x7F, 0x44, 0x01, 0x00, 0xC9, 0x02, 0x00, 0x00, 0x8D, 0x44, 0x01, 0x00, 0x9B, 0x44, 0x01, 0x00,
+ 0xA9, 0x44, 0x01, 0x00, 0xB7, 0x44, 0x01, 0x00, 0xC5, 0x44, 0x01, 0x00, 0xCD, 0x44, 0x01, 0x00,
+ 0xDB, 0x44, 0x01, 0x00, 0xE3, 0x44, 0x01, 0x00, 0xEB, 0x44, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xF3, 0x44, 0x01, 0x00,
+ 0xFB, 0x44, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x45, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x10, 0x00, 0x32, 0x02, 0x00, 0x00, 0xFF, 0xFF, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0xF0, 0x02, 0xF8, 0x00, 0xF0, 0x68, 0xF8, 0x0A, 0xA0, 0x90, 0xE8, 0x00, 0x0C, 0x82, 0x44,
+ 0x83, 0x44, 0xAA, 0xF1, 0x01, 0x07, 0xDA, 0x45, 0x01, 0xD1, 0x00, 0xF0, 0x5D, 0xF8, 0xAF, 0xF2,
+ 0x09, 0x0E, 0xBA, 0xE8, 0x0F, 0x00, 0x13, 0xF0, 0x01, 0x0F, 0x18, 0xBF, 0xFB, 0x1A, 0x43, 0xF0,
+ 0x01, 0x03, 0x18, 0x47, 0x08, 0x73, 0x01, 0x00, 0x38, 0x73, 0x01, 0x00, 0x0A, 0x44, 0x4F, 0xF0,
+ 0x00, 0x0C, 0x10, 0xF8, 0x01, 0x3B, 0x13, 0xF0, 0x07, 0x04, 0x08, 0xBF, 0x10, 0xF8, 0x01, 0x4B,
+ 0x1D, 0x11, 0x08, 0xBF, 0x10, 0xF8, 0x01, 0x5B, 0x64, 0x1E, 0x05, 0xD0, 0x10, 0xF8, 0x01, 0x6B,
+ 0x64, 0x1E, 0x01, 0xF8, 0x01, 0x6B, 0xF9, 0xD1, 0x13, 0xF0, 0x08, 0x0F, 0x1E, 0xBF, 0x10, 0xF8,
+ 0x01, 0x4B, 0xAD, 0x1C, 0x0C, 0x1B, 0x09, 0xD1, 0x6D, 0x1E, 0x58, 0xBF, 0x01, 0xF8, 0x01, 0xCB,
+ 0xFA, 0xD5, 0x05, 0xE0, 0x14, 0xF8, 0x01, 0x6B, 0x01, 0xF8, 0x01, 0x6B, 0x6D, 0x1E, 0xF9, 0xD5,
+ 0x91, 0x42, 0xD6, 0xD3, 0x70, 0x47, 0x00, 0x00, 0x10, 0x3A, 0x24, 0xBF, 0x78, 0xC8, 0x78, 0xC1,
+ 0xFA, 0xD8, 0x52, 0x07, 0x24, 0xBF, 0x30, 0xC8, 0x30, 0xC1, 0x44, 0xBF, 0x04, 0x68, 0x0C, 0x60,
+ 0x70, 0x47, 0x00, 0x00, 0x00, 0x23, 0x00, 0x24, 0x00, 0x25, 0x00, 0x26, 0x10, 0x3A, 0x28, 0xBF,
+ 0x78, 0xC1, 0xFB, 0xD8, 0x52, 0x07, 0x28, 0xBF, 0x30, 0xC1, 0x48, 0xBF, 0x0B, 0x60, 0x70, 0x47,
+ 0x1F, 0xB5, 0x1F, 0xBD, 0x10, 0xB5, 0x10, 0xBD, 0xDF, 0xF8, 0x0C, 0xD0, 0xFF, 0xF7, 0xF8, 0xFF,
+ 0x14, 0xF0, 0xA2, 0xF9, 0x16, 0xF0, 0xC6, 0xFB, 0x00, 0x04, 0x10, 0x00, 0x03, 0xB4, 0xFF, 0xF7,
+ 0xF1, 0xFF, 0x03, 0xBC, 0x16, 0xF0, 0xC4, 0xFB, 0x40, 0xBA, 0x70, 0x47, 0xC0, 0xBA, 0x70, 0x47,
+ 0x12, 0x48, 0x13, 0x49, 0xA1, 0xEB, 0x00, 0x01, 0x4F, 0xF0, 0x20, 0x03, 0x91, 0xFB, 0xF3, 0xF2,
+ 0x02, 0xFB, 0x13, 0x11, 0x4F, 0xF0, 0xCC, 0x33, 0x4F, 0xF0, 0xCC, 0x34, 0x4F, 0xF0, 0xCC, 0x35,
+ 0x4F, 0xF0, 0xCC, 0x36, 0x4F, 0xF0, 0xCC, 0x37, 0x4F, 0xF0, 0xCC, 0x38, 0x4F, 0xF0, 0xCC, 0x39,
+ 0x4F, 0xF0, 0xCC, 0x3A, 0xA0, 0xE8, 0xF8, 0x07, 0x01, 0x3A, 0xFB, 0xD1, 0x19, 0xB1, 0x40, 0xF8,
+ 0x04, 0xAB, 0x04, 0x39, 0xFB, 0xD1, 0x03, 0x48, 0x00, 0x47, 0x00, 0x00, 0x00, 0x00, 0x10, 0x00,
+ 0x00, 0x04, 0x10, 0x00, 0xE1, 0x00, 0x00, 0x00, 0x1E, 0xF0, 0x04, 0x0F, 0x0C, 0xBF, 0xEF, 0xF3,
+ 0x08, 0x80, 0xEF, 0xF3, 0x09, 0x80, 0x4F, 0xF0, 0x01, 0x01, 0x4F, 0xF0, 0x00, 0x02, 0x10, 0xF0,
+ 0x0B, 0xB8, 0x1E, 0xF0, 0x04, 0x0F, 0x0C, 0xBF, 0xEF, 0xF3, 0x08, 0x80, 0xEF, 0xF3, 0x09, 0x80,
+ 0x4F, 0xF0, 0x02, 0x01, 0x4F, 0xF0, 0x00, 0x02, 0x0F, 0xF0, 0xFE, 0xBF, 0x1E, 0xF0, 0x04, 0x0F,
+ 0x0C, 0xBF, 0xEF, 0xF3, 0x08, 0x80, 0xEF, 0xF3, 0x09, 0x80, 0x4F, 0xF0, 0x03, 0x01, 0x4F, 0xF0,
+ 0x00, 0x02, 0x0F, 0xF0, 0xF1, 0xBF, 0x1E, 0xF0, 0x04, 0x0F, 0x0C, 0xBF, 0xEF, 0xF3, 0x08, 0x80,
+ 0xEF, 0xF3, 0x09, 0x80, 0x4F, 0xF0, 0x04, 0x01, 0x4F, 0xF0, 0x00, 0x02, 0x0F, 0xF0, 0xE4, 0xBF,
+ 0x1E, 0xF0, 0x04, 0x0F, 0x0C, 0xBF, 0xEF, 0xF3, 0x08, 0x80, 0xEF, 0xF3, 0x09, 0x80, 0x4F, 0xF0,
+ 0x05, 0x01, 0x4F, 0xF0, 0x00, 0x02, 0x0F, 0xF0, 0xD7, 0xBF, 0x00, 0xB5, 0x14, 0xF0, 0x94, 0xF8,
+ 0x5D, 0xF8, 0x04, 0xEB, 0x00, 0xF0, 0x32, 0xB8, 0x1E, 0xF0, 0x04, 0x0F, 0x0C, 0xBF, 0xEF, 0xF3,
+ 0x08, 0x80, 0xEF, 0xF3, 0x09, 0x80, 0x4F, 0xF0, 0x06, 0x01, 0x4F, 0xF0, 0x00, 0x02, 0x0F, 0xF0,
+ 0xC3, 0xBF, 0x00, 0x00, 0x0A, 0x48, 0x4F, 0xF0, 0xFF, 0x01, 0x01, 0x70, 0x4F, 0xF0, 0x00, 0x00,
+ 0x80, 0xF3, 0x11, 0x88, 0x4F, 0xF0, 0x00, 0x00, 0x80, 0xF3, 0x09, 0x88, 0x05, 0x48, 0x4F, 0xF0,
+ 0x02, 0x01, 0x01, 0x70, 0x62, 0xB6, 0x04, 0x48, 0x4F, 0xF0, 0x80, 0x51, 0x01, 0x60, 0x70, 0x47,
+ 0x22, 0xED, 0x00, 0xE0, 0x18, 0x08, 0x10, 0x00, 0x04, 0xED, 0x00, 0xE0, 0x02, 0x48, 0x4F, 0xF0,
+ 0x80, 0x51, 0x01, 0x60, 0x70, 0x47, 0x00, 0x00, 0x04, 0xED, 0x00, 0xE0, 0x4F, 0xF0, 0x20, 0x00,
+ 0x80, 0xF3, 0x11, 0x88, 0xEF, 0xF3, 0x09, 0x80, 0x98, 0xB1, 0x12, 0x48, 0x00, 0x68, 0x12, 0x49,
+ 0x09, 0x68, 0x88, 0x42, 0x17, 0xD0, 0xEF, 0xF3, 0x09, 0x80, 0x0E, 0x49, 0x09, 0x68, 0x20, 0xE9,
+ 0xF0, 0x0F, 0x88, 0x60, 0x00, 0xB5, 0x0B, 0x48, 0x00, 0x68, 0x14, 0xF0, 0x6B, 0xF9, 0x5D, 0xF8,
+ 0x04, 0xEB, 0x08, 0x48, 0x08, 0x49, 0x0A, 0x68, 0x02, 0x60, 0x00, 0x68, 0x81, 0x68, 0xB1, 0xE8,
+ 0xF0, 0x0F, 0x81, 0xF3, 0x09, 0x88, 0x4E, 0xF0, 0x04, 0x0E, 0x4F, 0xF0, 0x00, 0x00, 0x80, 0xF3,
+ 0x11, 0x88, 0x70, 0x47, 0x1C, 0x08, 0x10, 0x00, 0x20, 0x08, 0x10, 0x00, 0xEF, 0xF3, 0x11, 0x80,
+ 0x4F, 0xF0, 0x20, 0x01, 0x81, 0xF3, 0x11, 0x88, 0x70, 0x47, 0x00, 0x00, 0x4F, 0xF0, 0x00, 0x00,
+ 0x80, 0xF3, 0x11, 0x88, 0x70, 0x47, 0x00, 0x00, 0x62, 0xB6, 0x70, 0x47, 0x72, 0xB6, 0x70, 0x47,
+ 0x40, 0x1E, 0xFD, 0xD1, 0x70, 0x47, 0x00, 0x00, 0x10, 0xB5, 0x01, 0x46, 0x34, 0x22, 0xFD, 0x48,
+ 0x16, 0xF0, 0x45, 0xFA, 0x01, 0x21, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0x20, 0x14, 0xF0, 0x6B, 0xB9,
+ 0x38, 0xB5, 0x04, 0x46, 0xF8, 0x48, 0xF7, 0x4D, 0x20, 0x60, 0x20, 0x46, 0x13, 0xF0, 0xD3, 0xFC,
+ 0x00, 0x22, 0x6B, 0x46, 0x01, 0x21, 0x10, 0x46, 0x14, 0xF0, 0xA6, 0xF9, 0x01, 0x21, 0x00, 0x20,
+ 0x14, 0xF0, 0x85, 0xF9, 0x28, 0x78, 0x00, 0x28, 0x02, 0xD0, 0x0F, 0xF0, 0x8C, 0xFF, 0xEC, 0xE7,
+ 0x38, 0xBD, 0x2D, 0xE9, 0xF0, 0x47, 0xDF, 0xF8, 0xB4, 0x83, 0x82, 0x46, 0x01, 0x27, 0xD8, 0xF8,
+ 0x00, 0x00, 0xEB, 0x4D, 0x46, 0x46, 0x90, 0xF8, 0x31, 0x40, 0x10, 0xF8, 0x30, 0x1F, 0x90, 0xF8,
+ 0x31, 0x00, 0x4C, 0x43, 0x00, 0xF0, 0x03, 0x00, 0x87, 0x40, 0xA1, 0x00, 0x28, 0x46, 0x16, 0xF0,
+ 0x6A, 0xFA, 0x30, 0x68, 0x90, 0xF8, 0xB0, 0x00, 0xC0, 0x07, 0x01, 0xD0, 0x00, 0xF0, 0xE5, 0xF9,
+ 0x00, 0x26, 0xDF, 0xF8, 0x70, 0x93, 0x12, 0xE0, 0xDA, 0x48, 0x34, 0x30, 0xFF, 0xF7, 0xC0, 0xFF,
+ 0x00, 0x20, 0xD9, 0xF8, 0x08, 0x10, 0x07, 0xE0, 0x55, 0xF8, 0x20, 0x20, 0x31, 0xF9, 0x10, 0x30,
+ 0x1A, 0x44, 0x45, 0xF8, 0x20, 0x20, 0x40, 0x1C, 0xA0, 0x42, 0xF5, 0xDB, 0x76, 0x1C, 0xBE, 0x42,
+ 0xEA, 0xDB, 0x40, 0x46, 0x0A, 0xE0, 0x00, 0xBF, 0x02, 0x68, 0x55, 0xF8, 0x24, 0x10, 0x92, 0xF8,
+ 0x61, 0x20, 0x02, 0xF0, 0x03, 0x02, 0x11, 0x41, 0x2A, 0xF8, 0x14, 0x10, 0x64, 0x1E, 0xF3, 0xD2,
+ 0xBD, 0xE8, 0xF0, 0x87, 0x2D, 0xE9, 0xF0, 0x4F, 0xDF, 0xF8, 0x20, 0xB3, 0x4F, 0xF0, 0x01, 0x09,
+ 0xC7, 0x4E, 0xDB, 0xF8, 0x00, 0x00, 0x9F, 0xB0, 0x90, 0xF8, 0x61, 0x10, 0x90, 0xF8, 0x31, 0x40,
+ 0xC1, 0xF3, 0x81, 0x08, 0x90, 0xF8, 0x30, 0x50, 0x09, 0xFA, 0x08, 0xFA, 0x60, 0x19, 0x81, 0x00,
+ 0x06, 0xEB, 0x84, 0x07, 0x30, 0x46, 0x16, 0xF0, 0x1E, 0xFA, 0xBA, 0x48, 0x34, 0x30, 0x13, 0xF0,
+ 0x35, 0xFC, 0xB8, 0x49, 0x34, 0x31, 0x81, 0xF8, 0x06, 0x90, 0xDB, 0xF8, 0x00, 0x00, 0x90, 0xF8,
+ 0x60, 0x20, 0x92, 0x07, 0x01, 0xD5, 0x81, 0xF8, 0x04, 0x90, 0x90, 0xF8, 0xB0, 0x00, 0xC0, 0x07,
+ 0x01, 0xD0, 0x00, 0xF0, 0x8A, 0xF9, 0xDB, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x60, 0x00, 0xC0, 0x07,
+ 0x03, 0xD0, 0xAC, 0x48, 0x34, 0x30, 0xFF, 0xF7, 0x63, 0xFF, 0x00, 0x20, 0x2B, 0xE0, 0xA9, 0x48,
+ 0x34, 0x30, 0xFF, 0xF7, 0x5D, 0xFF, 0xDF, 0xF8, 0x9C, 0xB2, 0x62, 0x00, 0x13, 0xA8, 0xDB, 0xF8,
+ 0x18, 0x10, 0x16, 0xF0, 0x4F, 0xF9, 0x6A, 0x00, 0x68, 0x46, 0xDB, 0xF8, 0x1C, 0x10, 0x16, 0xF0,
+ 0x49, 0xF9, 0x20, 0x46, 0x13, 0xAA, 0x06, 0xE0, 0x56, 0xF8, 0x20, 0x10, 0x32, 0xF9, 0x10, 0x30,
+ 0x19, 0x44, 0x46, 0xF8, 0x20, 0x10, 0x40, 0x1E, 0xF6, 0xD2, 0x28, 0x46, 0x69, 0x46, 0x06, 0xE0,
+ 0x57, 0xF8, 0x20, 0x20, 0x31, 0xF9, 0x10, 0x30, 0x1A, 0x44, 0x47, 0xF8, 0x20, 0x20, 0x40, 0x1E,
+ 0xF6, 0xD2, 0x09, 0xF1, 0x01, 0x00, 0x81, 0x46, 0x50, 0x45, 0xD0, 0xDB, 0x95, 0x49, 0x05, 0xE0,
+ 0x56, 0xF8, 0x24, 0x00, 0x40, 0xFA, 0x08, 0xF0, 0x21, 0xF8, 0x14, 0x00, 0x64, 0x1E, 0xF7, 0xD2,
+ 0x91, 0x49, 0x05, 0xE0, 0x57, 0xF8, 0x25, 0x00, 0x40, 0xFA, 0x08, 0xF0, 0x21, 0xF8, 0x15, 0x00,
+ 0x6D, 0x1E, 0xF7, 0xD2, 0x1F, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F, 0x2D, 0xE9, 0xFF, 0x5F, 0xDF, 0xF8,
+ 0x1C, 0x82, 0x87, 0x4F, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x31, 0x40, 0x90, 0xF8, 0x30, 0x50,
+ 0x07, 0xEB, 0x84, 0x0A, 0x60, 0x19, 0x81, 0x00, 0x04, 0xFB, 0x05, 0xFB, 0x0A, 0xEB, 0x85, 0x06,
+ 0x38, 0x46, 0x16, 0xF0, 0xA0, 0xF9, 0x4F, 0xEA, 0x8B, 0x01, 0x30, 0x46, 0x16, 0xF0, 0x9B, 0xF9,
+ 0x78, 0x48, 0x34, 0x30, 0x13, 0xF0, 0xB2, 0xFB, 0x76, 0x48, 0x4F, 0xF0, 0x01, 0x09, 0x34, 0x30,
+ 0x80, 0xF8, 0x06, 0x90, 0x80, 0xF8, 0x04, 0x90, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0xB0, 0x00,
+ 0xC0, 0x07, 0x01, 0xD0, 0x00, 0xF0, 0x09, 0xF9, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x60, 0x00,
+ 0xC0, 0x07, 0x03, 0xD0, 0x6B, 0x48, 0x34, 0x30, 0xFF, 0xF7, 0xE2, 0xFE, 0xD8, 0xF8, 0x00, 0x00,
+ 0x90, 0xF8, 0x61, 0x00, 0xC0, 0xF3, 0x81, 0x01, 0x02, 0x91, 0x00, 0xF0, 0x03, 0x01, 0x03, 0x91,
+ 0x02, 0x99, 0x48, 0x46, 0x09, 0xFA, 0x01, 0xF9, 0x03, 0x99, 0x88, 0x40, 0x01, 0x90, 0x00, 0x20,
+ 0x31, 0xE0, 0x60, 0x48, 0x34, 0x30, 0xFF, 0xF7, 0xCB, 0xFE, 0x5E, 0x48, 0xD0, 0xE9, 0x06, 0x32,
+ 0x81, 0x68, 0x20, 0x46, 0x07, 0xE0, 0x00, 0xBF, 0x57, 0xF8, 0x20, 0xC0, 0x33, 0xF9, 0x10, 0x80,
+ 0xC4, 0x44, 0x47, 0xF8, 0x20, 0xC0, 0x40, 0x1E, 0xF6, 0xD2, 0x28, 0x46, 0x07, 0xE0, 0x00, 0xBF,
+ 0x5A, 0xF8, 0x20, 0x30, 0x32, 0xF9, 0x10, 0xC0, 0x9C, 0x44, 0x4A, 0xF8, 0x20, 0xC0, 0x40, 0x1E,
+ 0xF6, 0xD2, 0xDD, 0xE9, 0x00, 0x02, 0x90, 0x42, 0x0B, 0xDA, 0x58, 0x46, 0x07, 0xE0, 0x00, 0xBF,
+ 0x56, 0xF8, 0x20, 0x20, 0x31, 0xF9, 0x10, 0x30, 0x1A, 0x44, 0x46, 0xF8, 0x20, 0x20, 0x40, 0x1E,
+ 0xF6, 0xD2, 0x00, 0x98, 0x40, 0x1C, 0x00, 0x90, 0x48, 0x45, 0xCA, 0xDB, 0x01, 0x98, 0x48, 0x45,
+ 0x1D, 0xDD, 0x44, 0x48, 0x00, 0x21, 0x34, 0x30, 0x88, 0x46, 0x81, 0x71, 0x01, 0x99, 0xA1, 0xEB,
+ 0x09, 0x09, 0x12, 0xE0, 0x3F, 0x48, 0x34, 0x30, 0xFF, 0xF7, 0x8A, 0xFE, 0x3D, 0x48, 0x81, 0x68,
+ 0x58, 0x46, 0x06, 0xE0, 0x56, 0xF8, 0x20, 0x20, 0x31, 0xF9, 0x10, 0x30, 0x1A, 0x44, 0x46, 0xF8,
+ 0x20, 0x20, 0x40, 0x1E, 0xF6, 0xD2, 0x08, 0xF1, 0x01, 0x08, 0xC1, 0x45, 0xEA, 0xDC, 0x39, 0x49,
+ 0x06, 0xE0, 0x00, 0xBF, 0x57, 0xF8, 0x24, 0x00, 0x02, 0x9A, 0x10, 0x41, 0x21, 0xF8, 0x14, 0x00,
+ 0x64, 0x1E, 0xF7, 0xD2, 0x34, 0x49, 0x05, 0xE0, 0x5A, 0xF8, 0x25, 0x00, 0x02, 0x9A, 0x10, 0x41,
+ 0x21, 0xF8, 0x15, 0x00, 0x6D, 0x1E, 0xF7, 0xD2, 0x58, 0x46, 0x30, 0x4A, 0x06, 0xE0, 0x00, 0xBF,
+ 0x56, 0xF8, 0x20, 0x10, 0x03, 0x9B, 0x19, 0x41, 0x22, 0xF8, 0x10, 0x10, 0x40, 0x1E, 0xF7, 0xD2,
+ 0xBD, 0xE8, 0xFF, 0x9F, 0x2D, 0xE9, 0xF0, 0x41, 0x05, 0x46, 0x00, 0x20, 0x14, 0xF0, 0x5B, 0xF8,
+ 0x02, 0x20, 0x14, 0xF0, 0x58, 0xF8, 0x03, 0x20, 0x14, 0xF0, 0x55, 0xF8, 0x24, 0x4E, 0x00, 0x20,
+ 0x30, 0x70, 0x1C, 0x48, 0x34, 0x30, 0x13, 0xF0, 0xF9, 0xFA, 0x1C, 0x48, 0x21, 0x4F, 0xDF, 0xF8,
+ 0x88, 0x80, 0x00, 0x68, 0x4F, 0xF0, 0x01, 0x04, 0x90, 0xF8, 0x60, 0x00, 0x80, 0x07, 0x4F, 0xEA,
+ 0x05, 0x70, 0x0A, 0xD5, 0x00, 0x28, 0x0F, 0xDA, 0xA8, 0x07, 0x08, 0xD5, 0x34, 0x70, 0xFF, 0xF7,
+ 0x14, 0xFF, 0x3C, 0x70, 0x88, 0xF8, 0x00, 0x40, 0x01, 0xE0, 0x00, 0x28, 0x04, 0xDA, 0x30, 0x78,
+ 0x10, 0xB9, 0xFF, 0xF7, 0x87, 0xFE, 0x3C, 0x70, 0x0A, 0x4F, 0xA8, 0x07, 0x07, 0xF1, 0x34, 0x07,
+ 0x26, 0xD5, 0x30, 0x78, 0x70, 0xBB, 0x38, 0x46, 0x13, 0xF0, 0xD0, 0xFA, 0x3C, 0x71, 0x0B, 0x48,
+ 0xFF, 0xF7, 0x2F, 0xFE, 0x09, 0x48, 0x00, 0xF0, 0x3C, 0xF8, 0x0C, 0x49, 0x08, 0x70, 0x88, 0xF8,
+ 0x00, 0x40, 0x1F, 0xE0, 0x38, 0x08, 0x10, 0x00, 0xB9, 0x03, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x84, 0x5A, 0x10, 0x00, 0x6A, 0x4A, 0x01, 0x20, 0x94, 0x4A, 0x01, 0x20, 0x2E, 0x44, 0x01, 0x20,
+ 0x00, 0x04, 0x10, 0x00, 0x51, 0x07, 0x10, 0x00, 0x50, 0x07, 0x10, 0x00, 0x52, 0x07, 0x10, 0x00,
+ 0x68, 0x07, 0x07, 0xD5, 0x38, 0x48, 0x13, 0xF0, 0xA9, 0xFA, 0x04, 0x20, 0x38, 0x71, 0x37, 0x48,
+ 0xFF, 0xF7, 0x07, 0xFE, 0x03, 0x20, 0x14, 0xF0, 0x11, 0xF8, 0x02, 0x20, 0x14, 0xF0, 0x0E, 0xF8,
+ 0xBD, 0xE8, 0xF0, 0x41, 0x00, 0x20, 0x14, 0xF0, 0x09, 0xB8, 0x31, 0x48, 0x00, 0x68, 0x90, 0xF8,
+ 0xB0, 0x00, 0x81, 0x07, 0x2C, 0x48, 0x01, 0xD5, 0x02, 0x21, 0x00, 0xE0, 0x01, 0x21, 0xC1, 0x72,
+ 0x70, 0x47, 0x2D, 0xE9, 0xF0, 0x43, 0x2A, 0x49, 0x4F, 0xF0, 0x00, 0x09, 0xCC, 0x46, 0x0F, 0x68,
+ 0x4B, 0x46, 0x97, 0xF8, 0x31, 0x20, 0x97, 0xF8, 0x30, 0x60, 0x97, 0xF8, 0x60, 0x40, 0x02, 0xFB,
+ 0x06, 0xF1, 0x09, 0xB2, 0x64, 0x07, 0x37, 0xD5, 0x22, 0x4C, 0x24, 0x68, 0x25, 0x8E, 0x8D, 0x42,
+ 0x32, 0xD1, 0x32, 0x34, 0x28, 0xE0, 0x35, 0x46, 0x20, 0xE0, 0x00, 0xBF, 0x34, 0xF8, 0x15, 0x10,
+ 0x30, 0xF8, 0x15, 0x80, 0xA1, 0xEB, 0x08, 0x01, 0x09, 0xB2, 0x00, 0x29, 0x01, 0xDB, 0x88, 0x46,
+ 0x01, 0xE0, 0xC1, 0xF1, 0x00, 0x08, 0xE0, 0x45, 0x07, 0xDD, 0x00, 0x29, 0x01, 0xDB, 0x8C, 0x46,
+ 0x01, 0xE0, 0xC1, 0xF1, 0x00, 0x0C, 0x0F, 0xFA, 0x8C, 0xFC, 0x00, 0x29, 0x00, 0xDA, 0x49, 0x42,
+ 0xB7, 0xF8, 0x63, 0x80, 0x41, 0x45, 0x01, 0xDD, 0x5B, 0x1C, 0x9B, 0xB2, 0x6D, 0x1E, 0xDD, 0xD1,
+ 0x04, 0xEB, 0x46, 0x04, 0x00, 0xEB, 0x46, 0x00, 0x52, 0x1E, 0xD4, 0xD2, 0x97, 0xF8, 0x65, 0x00,
+ 0x98, 0x42, 0x01, 0xD2, 0x4F, 0xF0, 0x01, 0x09, 0x07, 0x48, 0xA0, 0xF8, 0x00, 0xC0, 0x07, 0x48,
+ 0x03, 0x80, 0x48, 0x46, 0xBD, 0xE8, 0xF0, 0x83, 0x6C, 0x08, 0x10, 0x00, 0x2E, 0x44, 0x01, 0x20,
+ 0x20, 0x07, 0x10, 0x00, 0x24, 0x07, 0x10, 0x00, 0x54, 0x07, 0x10, 0x00, 0x56, 0x07, 0x10, 0x00,
+ 0x10, 0xB5, 0x01, 0x46, 0x34, 0x22, 0xFE, 0x48, 0x15, 0xF0, 0xB1, 0xFF, 0x01, 0x21, 0xBD, 0xE8,
+ 0x10, 0x40, 0x00, 0x20, 0x13, 0xF0, 0xD7, 0xBE, 0x38, 0xB5, 0x04, 0x46, 0xF9, 0x48, 0x20, 0x60,
+ 0xF9, 0x48, 0x00, 0x68, 0x90, 0xF8, 0xB0, 0x00, 0xC0, 0x07, 0x01, 0xD0, 0x01, 0x20, 0xE0, 0x72,
+ 0xF3, 0x4D, 0x20, 0x46, 0x13, 0xF0, 0x37, 0xFA, 0x00, 0x22, 0x6B, 0x46, 0x01, 0x21, 0x10, 0x46,
+ 0x13, 0xF0, 0x0A, 0xFF, 0x01, 0x21, 0x00, 0x20, 0x13, 0xF0, 0xE9, 0xFE, 0x28, 0x78, 0x00, 0x28,
+ 0x02, 0xD0, 0x0F, 0xF0, 0xF0, 0xFC, 0xEC, 0xE7, 0x38, 0xBD, 0x2D, 0xE9, 0xFF, 0x41, 0x00, 0x24,
+ 0xDD, 0xE9, 0x0A, 0x76, 0x08, 0x21, 0x68, 0x46, 0x14, 0xF0, 0x7C, 0xFC, 0x05, 0x46, 0x08, 0x21,
+ 0x02, 0xA8, 0x14, 0xF0, 0x77, 0xFC, 0x02, 0x46, 0x00, 0x20, 0x6F, 0xF0, 0x7E, 0x0E, 0x14, 0xE0,
+ 0x00, 0x21, 0x00, 0xFB, 0x02, 0xF3, 0x0D, 0xE0, 0x07, 0xEB, 0x01, 0x0C, 0x13, 0xF9, 0x0C, 0xC0,
+ 0xB4, 0x44, 0xF4, 0x45, 0x01, 0xDA, 0x01, 0x24, 0x03, 0xE0, 0xBC, 0xF1, 0x7F, 0x0F, 0x00, 0xDD,
+ 0x02, 0x24, 0x49, 0x1C, 0x91, 0x42, 0xEF, 0xDB, 0x40, 0x1C, 0xA8, 0x42, 0xE8, 0xDB, 0x20, 0x46,
+ 0x04, 0xB0, 0xBD, 0xE8, 0xF0, 0x81, 0x2D, 0xE9, 0xFF, 0x41, 0x0A, 0xAC, 0x08, 0x21, 0x94, 0xE8,
+ 0xC0, 0x01, 0x00, 0x24, 0x68, 0x46, 0x14, 0xF0, 0x4D, 0xFC, 0x05, 0x46, 0x08, 0x21, 0x02, 0xA8,
+ 0x14, 0xF0, 0x48, 0xFC, 0x01, 0x46, 0x00, 0x20, 0x08, 0xE0, 0x32, 0x56, 0xD2, 0x19, 0x01, 0xD5,
+ 0x01, 0x24, 0x02, 0xE0, 0xFF, 0x2A, 0x00, 0xDD, 0x02, 0x24, 0x40, 0x1C, 0xA8, 0x42, 0xF4, 0xDB,
+ 0x00, 0x20, 0x0A, 0xE0, 0x2A, 0x18, 0xB2, 0x56, 0x12, 0xEB, 0x08, 0x02, 0x01, 0xD5, 0x01, 0x24,
+ 0x02, 0xE0, 0xFF, 0x2A, 0x00, 0xDD, 0x02, 0x24, 0x40, 0x1C, 0x88, 0x42, 0xF2, 0xDB, 0x20, 0x46,
+ 0xCE, 0xE7, 0x2D, 0xE9, 0xF0, 0x4F, 0x87, 0xB0, 0x13, 0xF0, 0x3F, 0xFA, 0x04, 0x46, 0x0D, 0x46,
+ 0x13, 0xF0, 0x3D, 0xFA, 0xDF, 0xF8, 0xE0, 0x82, 0x06, 0x46, 0x0F, 0x46, 0xD8, 0xF8, 0x00, 0x00,
+ 0x90, 0xF8, 0xE7, 0x00, 0x10, 0xF0, 0x60, 0x0F, 0x05, 0xD0, 0x20, 0x46, 0x29, 0x46, 0x10, 0xF0,
+ 0x79, 0xFD, 0x04, 0x46, 0x0D, 0x46, 0xB0, 0x48, 0x01, 0x68, 0x91, 0xF8, 0x31, 0x80, 0x91, 0xF8,
+ 0x30, 0xA0, 0x01, 0x21, 0x08, 0xFB, 0x0A, 0xF0, 0x05, 0x90, 0x06, 0xA8, 0x0D, 0xF0, 0x4A, 0xFF,
+ 0xDF, 0xF8, 0xA8, 0x92, 0x00, 0x20, 0xDF, 0xF8, 0xA8, 0xB2, 0xD9, 0xF8, 0x00, 0x10, 0x06, 0x9B,
+ 0x32, 0x46, 0x91, 0xF9, 0x1B, 0x10, 0x03, 0x90, 0xCD, 0xF8, 0x00, 0xB0, 0xCD, 0xE9, 0x01, 0x31,
+ 0x3B, 0x46, 0x20, 0x46, 0x29, 0x46, 0x10, 0xF0, 0xD4, 0xFD, 0x01, 0x20, 0x03, 0x90, 0x00, 0x21,
+ 0x06, 0x9B, 0xCD, 0xF8, 0x00, 0xB0, 0xCD, 0xE9, 0x01, 0x31, 0x32, 0x46, 0x3B, 0x46, 0x20, 0x46,
+ 0x29, 0x46, 0x10, 0xF0, 0xC6, 0xFD, 0x96, 0x48, 0x34, 0x30, 0x13, 0xF0, 0x57, 0xF9, 0x94, 0x49,
+ 0x05, 0x20, 0x34, 0x31, 0x08, 0x71, 0x08, 0x46, 0xFF, 0xF7, 0x2E, 0xFF, 0x90, 0x48, 0x02, 0x21,
+ 0x80, 0x68, 0x00, 0x90, 0x04, 0xA8, 0x0D, 0xF0, 0x15, 0xFF, 0x8F, 0x48, 0x00, 0x68, 0xB0, 0xF9,
+ 0xC2, 0x11, 0x01, 0x91, 0xB0, 0xF9, 0xC4, 0xB1, 0x59, 0x44, 0x49, 0x10, 0x02, 0x91, 0x90, 0xF8,
+ 0xE8, 0x00, 0x80, 0x07, 0x0A, 0xD4, 0xAA, 0xF1, 0x01, 0x01, 0x08, 0xFB, 0x01, 0xF0, 0x05, 0x90,
+ 0x00, 0x99, 0x53, 0x46, 0x42, 0x46, 0x08, 0x46, 0x00, 0xF0, 0xF2, 0xFA, 0xDD, 0xE9, 0x04, 0x12,
+ 0x00, 0x98, 0x11, 0xF0, 0x90, 0xFA, 0x82, 0x46, 0x02, 0x21, 0x04, 0xA8, 0x0D, 0xF0, 0x55, 0xFF,
+ 0x01, 0x99, 0x4F, 0xF0, 0x00, 0x08, 0x8A, 0x45, 0x01, 0xDC, 0xDA, 0x45, 0x0A, 0xDA, 0x7A, 0x48,
+ 0x02, 0x99, 0x00, 0x68, 0xAA, 0xEB, 0x01, 0x01, 0xB0, 0xF8, 0xC6, 0x01, 0x91, 0xFB, 0xF0, 0xF0,
+ 0x4F, 0xFA, 0x80, 0xF8, 0x06, 0x99, 0xCD, 0xE9, 0x00, 0x18, 0x32, 0x46, 0x3B, 0x46, 0x20, 0x46,
+ 0x29, 0x46, 0xFF, 0xF7, 0x0A, 0xFF, 0x01, 0x28, 0x1C, 0xD0, 0x02, 0x28, 0x21, 0xD0, 0xD9, 0xF8,
+ 0x00, 0x10, 0x01, 0x20, 0x6E, 0x4A, 0x81, 0xF8, 0x1B, 0x80, 0x03, 0x90, 0x06, 0x9B, 0x8D, 0xE8,
+ 0x0C, 0x01, 0x32, 0x46, 0x3B, 0x46, 0x20, 0x46, 0x29, 0x46, 0x10, 0xF0, 0x62, 0xFD, 0xD9, 0xF8,
+ 0x00, 0x00, 0xA5, 0x21, 0x81, 0x76, 0x01, 0x21, 0x06, 0xA8, 0x0D, 0xF0, 0x1E, 0xFF, 0x07, 0xB0,
+ 0xBD, 0xE8, 0xF0, 0x8F, 0xD9, 0xF8, 0x00, 0x10, 0x00, 0x20, 0xC8, 0x76, 0x01, 0x21, 0xB0, 0x20,
+ 0x05, 0xE0, 0xD9, 0xF8, 0x00, 0x10, 0x00, 0x20, 0xC8, 0x76, 0x01, 0x21, 0xB1, 0x20, 0x0F, 0xF0,
+ 0xBB, 0xFB, 0xE4, 0xE7, 0x2D, 0xE9, 0xF0, 0x4F, 0xDF, 0xF8, 0x5C, 0x91, 0x8D, 0xB0, 0xD9, 0xF8,
+ 0x00, 0x00, 0x90, 0xF8, 0x31, 0x80, 0x90, 0xF8, 0x30, 0xA0, 0x13, 0xF0, 0x6E, 0xF9, 0x04, 0x46,
+ 0x0D, 0x46, 0x13, 0xF0, 0x6C, 0xF9, 0x06, 0x46, 0xD9, 0xF8, 0x00, 0x00, 0x0F, 0x46, 0x90, 0xF8,
+ 0xE7, 0x00, 0x10, 0xF0, 0x60, 0x0F, 0x05, 0xD0, 0x20, 0x46, 0x29, 0x46, 0x10, 0xF0, 0xAA, 0xFC,
+ 0x04, 0x46, 0x0D, 0x46, 0x01, 0x21, 0x04, 0xA8, 0x0D, 0xF0, 0x84, 0xFE, 0xDF, 0xF8, 0x1C, 0x91,
+ 0x00, 0x21, 0x04, 0x9B, 0xD9, 0xF8, 0x00, 0x00, 0xDF, 0xF8, 0x18, 0xB1, 0x90, 0xF9, 0x1D, 0x20,
+ 0xCD, 0xE9, 0x02, 0x21, 0xCD, 0xE9, 0x00, 0xB3, 0x32, 0x46, 0x3B, 0x46, 0x20, 0x46, 0x29, 0x46,
+ 0x10, 0xF0, 0x0F, 0xFD, 0x01, 0x21, 0x00, 0x22, 0x04, 0x9B, 0xCD, 0xE9, 0x02, 0x21, 0xCD, 0xE9,
+ 0x00, 0xB3, 0x32, 0x46, 0x3B, 0x46, 0x20, 0x46, 0x29, 0x46, 0x10, 0xF0, 0x02, 0xFD, 0x34, 0x48,
+ 0x34, 0x30, 0x13, 0xF0, 0x93, 0xF8, 0x32, 0x49, 0x06, 0x20, 0x34, 0x31, 0x08, 0x71, 0x08, 0x46,
+ 0xFF, 0xF7, 0x6A, 0xFE, 0x2E, 0x48, 0x02, 0x21, 0x80, 0x68, 0x00, 0x90, 0x0C, 0xA8, 0x0D, 0xF0,
+ 0x51, 0xFE, 0x2D, 0x48, 0x00, 0x68, 0xB0, 0xF9, 0xC8, 0x11, 0xB0, 0xF9, 0xCA, 0x21, 0x11, 0x44,
+ 0x49, 0x10, 0x03, 0x91, 0x08, 0xFB, 0x0A, 0xF1, 0x01, 0x91, 0x90, 0xF8, 0xE8, 0x00, 0x80, 0x07,
+ 0x0A, 0xD4, 0xAA, 0xF1, 0x01, 0x01, 0x08, 0xFB, 0x01, 0xF0, 0x00, 0x99, 0x01, 0x90, 0x53, 0x46,
+ 0x42, 0x46, 0x08, 0x46, 0x00, 0xF0, 0x2C, 0xFA, 0xDD, 0xE9, 0x00, 0x02, 0x0C, 0x99, 0x11, 0xF0,
+ 0xCA, 0xF9, 0x82, 0x46, 0x02, 0x21, 0x0C, 0xA8, 0x0D, 0xF0, 0x8F, 0xFE, 0x1A, 0x48, 0x4F, 0xF0,
+ 0x00, 0x08, 0x00, 0x68, 0xB0, 0xF9, 0xC8, 0x21, 0x52, 0x45, 0x03, 0xDB, 0xB0, 0xF9, 0xCA, 0x21,
+ 0x52, 0x45, 0x08, 0xDD, 0x03, 0x9A, 0xB0, 0xF8, 0xCC, 0x01, 0xAA, 0xEB, 0x02, 0x01, 0x91, 0xFB,
+ 0xF0, 0xF0, 0x4F, 0xFA, 0x80, 0xF8, 0x04, 0x99, 0xCD, 0xE9, 0x00, 0x18, 0x32, 0x46, 0x3B, 0x46,
+ 0x20, 0x46, 0x29, 0x46, 0xFF, 0xF7, 0x41, 0xFE, 0x4F, 0xF4, 0x80, 0x71, 0x01, 0x28, 0x29, 0xD0,
+ 0x02, 0x28, 0x2D, 0xD0, 0xD9, 0xF8, 0x00, 0x10, 0x32, 0x46, 0x81, 0xF8, 0x1D, 0x80, 0x01, 0x21,
+ 0x04, 0x9B, 0xCD, 0xF8, 0x00, 0xB0, 0x03, 0x91, 0xCD, 0xE9, 0x01, 0x38, 0x3B, 0x46, 0x0B, 0xE0,
+ 0x7C, 0x08, 0x10, 0x00, 0xE1, 0x08, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00, 0x24, 0x07, 0x10, 0x00,
+ 0xFC, 0x4C, 0x01, 0x20, 0x24, 0x52, 0x01, 0x20, 0x20, 0x46, 0x29, 0x46, 0x10, 0xF0, 0x89, 0xFC,
+ 0xD9, 0xF8, 0x00, 0x00, 0xA5, 0x21, 0x01, 0x77, 0x01, 0x21, 0x04, 0xA8, 0x0D, 0xF0, 0x45, 0xFE,
+ 0x0D, 0xB0, 0x25, 0xE7, 0xD9, 0xF8, 0x00, 0x20, 0x00, 0x20, 0x50, 0x77, 0xB0, 0x20, 0x04, 0xE0,
+ 0xD9, 0xF8, 0x00, 0x20, 0x00, 0x20, 0x50, 0x77, 0xB1, 0x20, 0x0F, 0xF0, 0xE5, 0xFA, 0xE7, 0xE7,
+ 0x2D, 0xE9, 0xF0, 0x4F, 0xF3, 0x48, 0x8F, 0xB0, 0x00, 0x68, 0x90, 0xF8, 0x31, 0x60, 0x90, 0xF8,
+ 0x30, 0x00, 0x06, 0x90, 0x13, 0xF0, 0x99, 0xF8, 0x80, 0x46, 0x89, 0x46, 0x13, 0xF0, 0x97, 0xF8,
+ 0x04, 0x46, 0x0D, 0x46, 0x01, 0x21, 0x08, 0xA8, 0x0D, 0xF0, 0xBC, 0xFD, 0xEA, 0x48, 0x02, 0xAB,
+ 0x00, 0x27, 0x00, 0x68, 0x08, 0x9A, 0x90, 0xF9, 0x20, 0x10, 0x90, 0xF9, 0x1F, 0x00, 0x83, 0xE8,
+ 0x83, 0x00, 0xE6, 0x48, 0xCD, 0xE9, 0x00, 0x02, 0x0E, 0x90, 0x22, 0x46, 0x2B, 0x46, 0x40, 0x46,
+ 0x49, 0x46, 0x10, 0xF0, 0x97, 0xFC, 0x08, 0x99, 0xCD, 0xE9, 0x01, 0x17, 0x01, 0x20, 0xCD, 0xE9,
+ 0x03, 0x70, 0x0E, 0x98, 0x00, 0x90, 0x22, 0x46, 0x2B, 0x46, 0x40, 0x46, 0x49, 0x46, 0x10, 0xF0,
+ 0x89, 0xFC, 0xDB, 0x48, 0x12, 0xF0, 0xCA, 0xFF, 0xD9, 0x49, 0x03, 0x20, 0x88, 0x71, 0x08, 0x46,
+ 0xFF, 0xF7, 0xA2, 0xFD, 0xD6, 0x48, 0x02, 0x21, 0x34, 0x38, 0xD0, 0xE9, 0x06, 0x70, 0x00, 0x90,
+ 0xD0, 0x48, 0x00, 0x68, 0xB0, 0xF9, 0xCE, 0xA1, 0xB0, 0xF9, 0xD0, 0xB1, 0x05, 0xA8, 0x0D, 0xF0,
+ 0x81, 0xFD, 0x0A, 0xEB, 0x0B, 0x00, 0x40, 0x10, 0x0D, 0x90, 0xCA, 0x48, 0x00, 0x68, 0x90, 0xF8,
+ 0x34, 0x01, 0x80, 0x07, 0x05, 0xD4, 0x39, 0x46, 0x32, 0x46, 0x08, 0x46, 0x00, 0xF0, 0x7C, 0xF9,
+ 0x76, 0x1E, 0x32, 0x46, 0x38, 0x46, 0x05, 0x99, 0x11, 0xF0, 0x05, 0xF9, 0x00, 0x27, 0x50, 0x45,
+ 0x01, 0xDC, 0x58, 0x45, 0x08, 0xDA, 0x0D, 0x99, 0x09, 0x1A, 0xBE, 0x48, 0x00, 0x68, 0xB0, 0xF8,
+ 0xD2, 0x01, 0x91, 0xFB, 0xF0, 0xF0, 0x47, 0xB2, 0xBA, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x34, 0x01,
+ 0x80, 0x07, 0x07, 0xD4, 0x00, 0x99, 0x06, 0x9A, 0x08, 0x46, 0x00, 0xF0, 0x5D, 0xF9, 0x06, 0x98,
+ 0x40, 0x1E, 0x06, 0x90, 0xDD, 0xE9, 0x05, 0x12, 0x00, 0x98, 0x11, 0xF0, 0xE4, 0xF8, 0x00, 0x26,
+ 0x50, 0x45, 0x01, 0xDC, 0x58, 0x45, 0x08, 0xDA, 0x0D, 0x99, 0x09, 0x1A, 0xAD, 0x48, 0x00, 0x68,
+ 0xB0, 0xF8, 0xD4, 0x01, 0x91, 0xFB, 0xF0, 0xF0, 0x46, 0xB2, 0x02, 0x21, 0x05, 0xA8, 0x0D, 0xF0,
+ 0x9C, 0xFD, 0x08, 0x98, 0xCD, 0xE9, 0x00, 0x07, 0x22, 0x46, 0x2B, 0x46, 0x02, 0x96, 0x40, 0x46,
+ 0x49, 0x46, 0xFF, 0xF7, 0x90, 0xFD, 0x01, 0x28, 0x1E, 0xD0, 0x02, 0x28, 0x25, 0xD0, 0xA2, 0x48,
+ 0x22, 0x46, 0x2B, 0x46, 0x00, 0x68, 0xC7, 0x77, 0x80, 0xF8, 0x20, 0x60, 0x01, 0x20, 0x08, 0x99,
+ 0xCD, 0xE9, 0x03, 0x60, 0xCD, 0xE9, 0x01, 0x17, 0x0E, 0x98, 0x00, 0x90, 0x40, 0x46, 0x49, 0x46,
+ 0x10, 0xF0, 0x08, 0xFC, 0x98, 0x48, 0xA5, 0x21, 0x00, 0x68, 0x81, 0x77, 0x01, 0x21, 0x08, 0xA8,
+ 0x0D, 0xF0, 0x73, 0xFD, 0x0F, 0xB0, 0x53, 0xE6, 0x93, 0x49, 0x00, 0x20, 0x09, 0x68, 0xC8, 0x77,
+ 0x81, 0xF8, 0x20, 0x00, 0x10, 0x21, 0xB2, 0x20, 0x07, 0xE0, 0x8F, 0x49, 0x00, 0x20, 0x09, 0x68,
+ 0xC8, 0x77, 0x81, 0xF8, 0x20, 0x00, 0x10, 0x21, 0xB3, 0x20, 0x0F, 0xF0, 0x0D, 0xFA, 0xE1, 0xE7,
+ 0x2D, 0xE9, 0xF0, 0x4F, 0x87, 0xB0, 0x12, 0xF0, 0xC8, 0xFF, 0x06, 0x46, 0x0F, 0x46, 0x12, 0xF0,
+ 0xC6, 0xFF, 0xDF, 0xF8, 0x10, 0xB2, 0x01, 0x21, 0xDA, 0x46, 0xDB, 0xF8, 0x00, 0x00, 0x90, 0xF8,
+ 0x31, 0x50, 0x05, 0xA8, 0x0D, 0xF0, 0xE6, 0xFC, 0x7F, 0x4C, 0x02, 0xA9, 0x4F, 0xF0, 0x00, 0x08,
+ 0x20, 0x68, 0x05, 0x9B, 0xDF, 0xF8, 0xF4, 0x91, 0x90, 0xF9, 0x23, 0x20, 0x90, 0xF9, 0x22, 0x00,
+ 0x09, 0xF1, 0x98, 0x09, 0x81, 0xE8, 0x05, 0x01, 0xCD, 0xE9, 0x00, 0x93, 0x42, 0x46, 0x43, 0x46,
+ 0x30, 0x46, 0x39, 0x46, 0x10, 0xF0, 0xBE, 0xFB, 0x05, 0x9A, 0xCD, 0xE9, 0x01, 0x28, 0x01, 0x21,
+ 0xCD, 0xF8, 0x00, 0x90, 0xCD, 0xE9, 0x03, 0x81, 0x00, 0x22, 0x13, 0x46, 0x30, 0x46, 0x39, 0x46,
+ 0x10, 0xF0, 0xB0, 0xFB, 0x6E, 0x48, 0x12, 0xF0, 0xF1, 0xFE, 0x6D, 0x49, 0x03, 0x20, 0x08, 0x72,
+ 0x08, 0x46, 0xFF, 0xF7, 0xC9, 0xFC, 0x6A, 0x48, 0x02, 0x21, 0x34, 0x38, 0xD0, 0xF8, 0x28, 0x90,
+ 0x06, 0xA8, 0x0D, 0xF0, 0xAF, 0xFC, 0xDA, 0xF8, 0x00, 0x00, 0xB0, 0xF9, 0xD6, 0x11, 0xB0, 0xF9,
+ 0xD8, 0x21, 0x90, 0xF8, 0x84, 0x09, 0x11, 0x44, 0x01, 0xEB, 0xD1, 0x71, 0x4F, 0xEA, 0x61, 0x0A,
+ 0x80, 0x07, 0x05, 0xD4, 0x6D, 0x1E, 0x49, 0x46, 0x2A, 0x46, 0x08, 0x46, 0x00, 0xF0, 0xA4, 0xF8,
+ 0x2A, 0x46, 0x48, 0x46, 0x06, 0x99, 0x11, 0xF0, 0x2E, 0xF8, 0xDB, 0xF8, 0x00, 0x10, 0x00, 0x25,
+ 0xA9, 0x46, 0xB1, 0xF9, 0xD6, 0x21, 0x82, 0x42, 0x03, 0xDB, 0xB1, 0xF9, 0xD8, 0x21, 0x82, 0x42,
+ 0x06, 0xDD, 0xB1, 0xF8, 0xDA, 0x11, 0xAA, 0xEB, 0x00, 0x00, 0x90, 0xFB, 0xF1, 0xF0, 0x45, 0xB2,
+ 0x02, 0x21, 0x06, 0xA8, 0x0D, 0xF0, 0xE1, 0xFC, 0x05, 0x99, 0x00, 0x22, 0x8D, 0xE8, 0x22, 0x02,
+ 0x13, 0x46, 0x30, 0x46, 0x39, 0x46, 0xFF, 0xF7, 0xD6, 0xFC, 0x01, 0x28, 0x1C, 0xD0, 0x02, 0x28,
+ 0x22, 0xD0, 0x21, 0x68, 0x45, 0x48, 0x01, 0xF8, 0x22, 0x5F, 0x81, 0xF8, 0x01, 0x80, 0x01, 0x21,
+ 0x05, 0x9A, 0xCD, 0xE9, 0x03, 0x81, 0x8D, 0xE8, 0x25, 0x00, 0x00, 0x22, 0x13, 0x46, 0x30, 0x46,
+ 0x39, 0x46, 0x10, 0xF0, 0x4F, 0xFB, 0x21, 0x68, 0xA5, 0x20, 0x81, 0xF8, 0x21, 0x00, 0x01, 0x21,
+ 0x05, 0xA8, 0x0D, 0xF0, 0xBA, 0xFC, 0x9A, 0xE5, 0x21, 0x68, 0x01, 0xF8, 0x22, 0x8F, 0x81, 0xF8,
+ 0x01, 0x80, 0x40, 0x21, 0xB2, 0x20, 0x06, 0xE0, 0x21, 0x68, 0xB3, 0x20, 0x01, 0xF8, 0x22, 0x8F,
+ 0x81, 0xF8, 0x01, 0x80, 0x40, 0x21, 0x0F, 0xF0, 0x57, 0xF9, 0xE4, 0xE7, 0x70, 0xB5, 0x04, 0x46,
+ 0x00, 0x20, 0x13, 0xF0, 0xC8, 0xFB, 0x02, 0x20, 0x13, 0xF0, 0xC5, 0xFB, 0x29, 0x4D, 0x28, 0x68,
+ 0x90, 0xF8, 0xC0, 0x01, 0x20, 0x40, 0xC0, 0x07, 0x01, 0xD0, 0xFF, 0xF7, 0xC2, 0xFC, 0x28, 0x68,
+ 0x90, 0xF8, 0xC0, 0x01, 0x80, 0x07, 0x03, 0xD5, 0xA0, 0x07, 0x01, 0xD5, 0xFF, 0xF7, 0x82, 0xFD,
+ 0x28, 0x68, 0x90, 0xF8, 0xC0, 0x01, 0x40, 0x07, 0x03, 0xD5, 0x60, 0x07, 0x01, 0xD5, 0xFF, 0xF7,
+ 0x4F, 0xFE, 0x28, 0x68, 0x90, 0xF8, 0xC0, 0x01, 0x00, 0x07, 0x03, 0xD5, 0x20, 0x07, 0x01, 0xD5,
+ 0xFF, 0xF7, 0x1E, 0xFF, 0x02, 0x20, 0x13, 0xF0, 0xB9, 0xFB, 0x00, 0x20, 0x13, 0xF0, 0xB6, 0xFB,
+ 0xE0, 0x07, 0x04, 0xD0, 0xBD, 0xE8, 0x70, 0x40, 0x01, 0x20, 0x01, 0xF0, 0xC9, 0xB9, 0x70, 0xBD,
+ 0xF0, 0xB5, 0x00, 0x26, 0x35, 0x46, 0x0C, 0xE0, 0x01, 0x24, 0x07, 0xE0, 0x05, 0xFB, 0x03, 0x47,
+ 0x64, 0x1C, 0x31, 0xF8, 0x17, 0x70, 0x20, 0xF8, 0x16, 0x70, 0x76, 0x1C, 0x9C, 0x42, 0xF5, 0xDB,
+ 0x6D, 0x1C, 0x95, 0x42, 0xF0, 0xDB, 0xF0, 0xBD, 0x30, 0xB5, 0x00, 0x24, 0x01, 0x23, 0x05, 0xE0,
+ 0x31, 0xF8, 0x13, 0x50, 0x20, 0xF8, 0x14, 0x50, 0x5B, 0x1C, 0x64, 0x1C, 0x93, 0x42, 0xF7, 0xDB,
+ 0x30, 0xBD, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00, 0x24, 0x07, 0x10, 0x00, 0xEC, 0x57, 0x01, 0x20,
+ 0xB0, 0x08, 0x10, 0x00, 0x70, 0xB5, 0x6D, 0x4D, 0x4F, 0xF4, 0x26, 0x74, 0x21, 0x46, 0x28, 0x68,
+ 0x14, 0xF0, 0xCA, 0xF8, 0x00, 0x21, 0x1F, 0x20, 0x0E, 0xF1, 0x6A, 0xFD, 0x22, 0x46, 0x29, 0x68,
+ 0xBD, 0xE8, 0x70, 0x40, 0x1F, 0x23, 0x66, 0x48, 0x14, 0xF0, 0xE5, 0xB8, 0x63, 0x48, 0x10, 0xB5,
+ 0x00, 0x68, 0x64, 0x4C, 0x90, 0xF8, 0x3F, 0x10, 0x20, 0x68, 0x01, 0x76, 0x4F, 0xF4, 0x8C, 0x61,
+ 0x14, 0xF0, 0xB2, 0xF8, 0x00, 0x21, 0x1A, 0x20, 0x0E, 0xF1, 0x52, 0xFD, 0x00, 0x21, 0x1B, 0x20,
+ 0x0E, 0xF1, 0x4E, 0xFD, 0x1A, 0x23, 0x4F, 0xF4, 0x80, 0x62, 0x5B, 0x48, 0x21, 0x68, 0x14, 0xF0,
+ 0xCA, 0xF8, 0x21, 0x68, 0x1B, 0x23, 0xBD, 0xE8, 0x10, 0x40, 0x60, 0x22, 0x57, 0x48, 0x01, 0xF5,
+ 0x80, 0x51, 0x14, 0xF0, 0xC0, 0xB8, 0x2D, 0xE9, 0xF0, 0x41, 0x55, 0x4D, 0x04, 0x1D, 0x0E, 0x46,
+ 0x21, 0x46, 0x28, 0x46, 0x14, 0xF0, 0x90, 0xF8, 0x30, 0x68, 0x52, 0x49, 0x00, 0xEB, 0x84, 0x03,
+ 0x8B, 0x42, 0x01, 0xD2, 0x1C, 0x23, 0x12, 0xE0, 0x4F, 0x4A, 0x93, 0x42, 0x12, 0xD2, 0x88, 0x42,
+ 0x0C, 0xD2, 0x88, 0x46, 0x09, 0x1A, 0x8F, 0x08, 0x1C, 0x23, 0x3A, 0x46, 0x29, 0x46, 0x14, 0xF0,
+ 0xA2, 0xF8, 0xE2, 0x1B, 0x05, 0xEB, 0x87, 0x01, 0x1D, 0x23, 0x11, 0xE0, 0x1D, 0x23, 0x22, 0x46,
+ 0x29, 0x46, 0x0E, 0xE0, 0x90, 0x42, 0x14, 0xD2, 0x11, 0x1A, 0x8F, 0x08, 0x90, 0x46, 0x1D, 0x23,
+ 0x3A, 0x46, 0x29, 0x46, 0x14, 0xF0, 0x8F, 0xF8, 0xE2, 0x1B, 0x05, 0xEB, 0x87, 0x01, 0x1E, 0x23,
+ 0x40, 0x46, 0x14, 0xF0, 0x88, 0xF8, 0x30, 0x68, 0x00, 0xEB, 0x84, 0x00, 0x30, 0x60, 0xBD, 0xE8,
+ 0xF0, 0x81, 0x1E, 0x23, 0xE3, 0xE7, 0xF8, 0xB5, 0x35, 0x4C, 0x38, 0x48, 0x00, 0x90, 0x10, 0x34,
+ 0x02, 0x20, 0x13, 0xF0, 0xE8, 0xFA, 0x00, 0x21, 0x1C, 0x20, 0x0E, 0xF1, 0xF1, 0xFC, 0x00, 0x21,
+ 0x1D, 0x20, 0x0E, 0xF1, 0xED, 0xFC, 0x00, 0x21, 0x1E, 0x20, 0x0E, 0xF1, 0xE9, 0xFC, 0x10, 0x21,
+ 0x20, 0x46, 0x15, 0xF0, 0x46, 0xFB, 0x00, 0x20, 0x20, 0x80, 0xA0, 0x70, 0x23, 0x4D, 0x01, 0x20,
+ 0xE0, 0x70, 0x28, 0x68, 0xC1, 0x8A, 0xA1, 0x80, 0x81, 0x8A, 0xE1, 0x80, 0x28, 0x49, 0xC2, 0x8A,
+ 0xA1, 0xF8, 0xFA, 0x2E, 0x80, 0x8A, 0xA1, 0xF8, 0xF8, 0x0E, 0x04, 0x20, 0x69, 0x46, 0xFF, 0xF7,
+ 0x92, 0xFF, 0x01, 0x23, 0x00, 0x22, 0x21, 0x46, 0x10, 0x20, 0x10, 0xF0, 0x46, 0xFC, 0x06, 0x46,
+ 0x69, 0x46, 0xFF, 0xF7, 0x88, 0xFF, 0x01, 0x23, 0x00, 0x22, 0x21, 0x46, 0x11, 0x20, 0x10, 0xF0,
+ 0x3C, 0xFC, 0x69, 0x46, 0x30, 0x46, 0xFF, 0xF7, 0x7E, 0xFF, 0x29, 0x68, 0x91, 0xF8, 0x50, 0x02,
+ 0xC0, 0x07, 0x03, 0xD1, 0x91, 0xF8, 0x80, 0x02, 0xC0, 0x07, 0x09, 0xD0, 0x01, 0x23, 0x00, 0x22,
+ 0x21, 0x46, 0x12, 0x20, 0x10, 0xF0, 0x29, 0xFC, 0x69, 0x46, 0x30, 0x46, 0xFF, 0xF7, 0x6B, 0xFF,
+ 0x01, 0x23, 0x00, 0x22, 0x21, 0x46, 0x13, 0x20, 0x10, 0xF0, 0x1F, 0xFC, 0x69, 0x46, 0x30, 0x46,
+ 0xFF, 0xF7, 0x61, 0xFF, 0x02, 0x20, 0x13, 0xF0, 0xA9, 0xFA, 0xF8, 0xBD, 0x20, 0x07, 0x10, 0x00,
+ 0xF0, 0xEF, 0x01, 0x00, 0x24, 0x07, 0x10, 0x00, 0xF0, 0x9F, 0x01, 0x00, 0xF0, 0xAF, 0x01, 0x00,
+ 0x84, 0x5A, 0x10, 0x00, 0xF0, 0xCF, 0x01, 0x00, 0xF0, 0xDF, 0x01, 0x00, 0xF0, 0xBF, 0x01, 0x00,
+ 0xFC, 0x4C, 0x01, 0x20, 0x0B, 0x48, 0x01, 0x78, 0x5A, 0x29, 0x11, 0xD1, 0x81, 0x88, 0xB1, 0xF5,
+ 0x7C, 0x7F, 0x0D, 0xD8, 0x41, 0x88, 0x00, 0x29, 0x0A, 0xD1, 0x06, 0x49, 0x40, 0x78, 0x10, 0x31,
+ 0x02, 0x28, 0x05, 0xD0, 0xA0, 0xF1, 0x70, 0x02, 0x07, 0x2A, 0x01, 0xD8, 0x0F, 0xF0, 0xBF, 0xB9,
+ 0x70, 0x47, 0x00, 0x00, 0x00, 0x00, 0x01, 0x20, 0x10, 0xB5, 0x01, 0x46, 0x34, 0x22, 0xFE, 0x48,
+ 0x15, 0xF0, 0x8D, 0xFA, 0x01, 0x21, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0x20, 0x13, 0xF0, 0xB3, 0xB9,
+ 0x2D, 0xE9, 0xF0, 0x41, 0xF9, 0x4C, 0xA0, 0xF1, 0x60, 0x06, 0x09, 0x2E, 0xD4, 0xE9, 0x12, 0x05,
+ 0xD4, 0xE9, 0x14, 0x23, 0x16, 0xD2, 0xDF, 0xE8, 0x06, 0xF0, 0x05, 0x16, 0x26, 0x37, 0x46, 0x58,
+ 0x67, 0x78, 0x87, 0x00, 0x02, 0x46, 0x2B, 0x46, 0x08, 0x46, 0x00, 0xF0, 0x9D, 0xF9, 0x02, 0x46,
+ 0x01, 0x20, 0x00, 0x21, 0x15, 0xF0, 0xEF, 0xF9, 0xD4, 0xE9, 0x02, 0x23, 0x10, 0x43, 0x19, 0x43,
+ 0xC4, 0xE9, 0x02, 0x01, 0x75, 0xE0, 0x08, 0x46, 0x00, 0xF0, 0x8E, 0xF9, 0x01, 0x23, 0x02, 0x46,
+ 0x00, 0x21, 0x18, 0x46, 0x15, 0xF0, 0xDF, 0xF9, 0xD4, 0xE9, 0x0A, 0x23, 0x10, 0x43, 0x19, 0x43,
+ 0xC4, 0xE9, 0x0A, 0x01, 0x65, 0xE0, 0x02, 0x46, 0x2B, 0x46, 0x08, 0x46, 0x00, 0xF0, 0x7C, 0xF9,
+ 0x02, 0x46, 0x01, 0x20, 0x00, 0x21, 0x15, 0xF0, 0xCE, 0xF9, 0xD4, 0xE9, 0x06, 0x23, 0x10, 0x43,
+ 0x19, 0x43, 0xC4, 0xE9, 0x06, 0x01, 0x54, 0xE0, 0x08, 0x46, 0x00, 0xF0, 0x6D, 0xF9, 0x02, 0x46,
+ 0x01, 0x20, 0x00, 0x21, 0x15, 0xF0, 0xBF, 0xF9, 0xD4, 0xE9, 0x0E, 0x23, 0x10, 0x43, 0x19, 0x43,
+ 0xC4, 0xE9, 0x0E, 0x01, 0x45, 0xE0, 0x02, 0x46, 0x2B, 0x46, 0x08, 0x46, 0x00, 0xF0, 0x5C, 0xF9,
+ 0x01, 0x23, 0x02, 0x46, 0x00, 0x21, 0x18, 0x46, 0x15, 0xF0, 0xAD, 0xF9, 0xD4, 0xE9, 0x04, 0x23,
+ 0x10, 0x43, 0x19, 0x43, 0xC4, 0xE9, 0x04, 0x01, 0x33, 0xE0, 0x08, 0x46, 0x00, 0xF0, 0x4C, 0xF9,
+ 0x02, 0x46, 0x01, 0x20, 0x00, 0x21, 0x15, 0xF0, 0x9E, 0xF9, 0xD4, 0xE9, 0x0C, 0x23, 0x10, 0x43,
+ 0x19, 0x43, 0xC4, 0xE9, 0x0C, 0x01, 0x24, 0xE0, 0x02, 0x46, 0x2B, 0x46, 0x08, 0x46, 0x00, 0xF0,
+ 0x3B, 0xF9, 0x02, 0x46, 0x01, 0x20, 0x00, 0x21, 0x15, 0xF0, 0x8D, 0xF9, 0xD4, 0xE9, 0x08, 0x23,
+ 0x10, 0x43, 0x19, 0x43, 0xC4, 0xE9, 0x08, 0x01, 0x13, 0xE0, 0x08, 0x46, 0x00, 0xF0, 0x2C, 0xF9,
+ 0x02, 0x46, 0x01, 0x20, 0x00, 0x21, 0x15, 0xF0, 0x7E, 0xF9, 0xD4, 0xE9, 0x10, 0x23, 0x10, 0x43,
+ 0x19, 0x43, 0xC4, 0xE9, 0x10, 0x01, 0x04, 0xE0, 0x01, 0x20, 0x88, 0x40, 0x61, 0x68, 0x08, 0x43,
+ 0x60, 0x60, 0x20, 0x68, 0x40, 0x1C, 0x20, 0x60, 0xBD, 0xE8, 0xF0, 0x81, 0x2D, 0xE9, 0xF0, 0x41,
+ 0x17, 0x46, 0x04, 0x46, 0x0D, 0x46, 0x00, 0x26, 0x09, 0xE0, 0xE0, 0x07, 0x03, 0xD0, 0x31, 0x46,
+ 0x38, 0x46, 0x0E, 0xF0, 0x31, 0xFF, 0x76, 0x1C, 0x6D, 0x08, 0x4F, 0xEA, 0x34, 0x04, 0x54, 0xEA,
+ 0x05, 0x00, 0xF2, 0xD1, 0xE8, 0xE7, 0x2D, 0xE9, 0xF0, 0x41, 0xA0, 0x4C, 0xD4, 0xE9, 0x02, 0x23,
+ 0xD4, 0xE9, 0x06, 0x56, 0x10, 0x46, 0x19, 0x46, 0x2A, 0x43, 0x33, 0x43, 0xD4, 0xE9, 0x0A, 0x56,
+ 0xD4, 0xE9, 0x0E, 0xC7, 0x45, 0xEA, 0x0C, 0x05, 0x3E, 0x43, 0xD4, 0xE9, 0x08, 0x7C, 0x97, 0x43,
+ 0x2C, 0xEA, 0x03, 0x0C, 0xC4, 0xE9, 0x08, 0x7C, 0xD4, 0xE9, 0x10, 0x23, 0xAA, 0x43, 0xB3, 0x43,
+ 0xC4, 0xE9, 0x10, 0x23, 0x22, 0x68, 0x00, 0x2A, 0xC6, 0xD0, 0x60, 0x22, 0xFF, 0xF7, 0xC6, 0xFF,
+ 0xD4, 0xE9, 0x0A, 0x01, 0x61, 0x22, 0xFF, 0xF7, 0xC1, 0xFF, 0xD4, 0xE9, 0x06, 0x01, 0x62, 0x22,
+ 0xFF, 0xF7, 0xBC, 0xFF, 0xD4, 0xE9, 0x0E, 0x01, 0x63, 0x22, 0xFF, 0xF7, 0xB7, 0xFF, 0xD4, 0xE9,
+ 0x04, 0x01, 0x64, 0x22, 0xFF, 0xF7, 0xB2, 0xFF, 0xD4, 0xE9, 0x0C, 0x01, 0x65, 0x22, 0xFF, 0xF7,
+ 0xAD, 0xFF, 0xD4, 0xE9, 0x08, 0x01, 0x66, 0x22, 0xFF, 0xF7, 0xA8, 0xFF, 0xD4, 0xE9, 0x10, 0x01,
+ 0x67, 0x22, 0xFF, 0xF7, 0xA3, 0xFF, 0x60, 0x68, 0xBD, 0xE8, 0xF0, 0x41, 0x68, 0x22, 0x00, 0x21,
+ 0x9C, 0xE7, 0x99, 0xE7, 0x2D, 0xE9, 0xF1, 0x4F, 0x4F, 0xF0, 0x00, 0x09, 0x12, 0xF0, 0x95, 0xFC,
+ 0x06, 0x46, 0x88, 0x46, 0x12, 0xF0, 0x93, 0xFC, 0x74, 0x4A, 0x04, 0x46, 0x0D, 0x46, 0xC2, 0xE9,
+ 0x14, 0x01, 0xC2, 0xE9, 0x12, 0x68, 0x4F, 0xF0, 0x1D, 0x0A, 0x4F, 0xF0, 0x26, 0x0B, 0x13, 0xF0,
+ 0x8C, 0xFB, 0x00, 0x27, 0x3A, 0x46, 0x01, 0x20, 0x00, 0x21, 0x15, 0xF0, 0xEC, 0xF8, 0x30, 0x40,
+ 0x01, 0xEA, 0x08, 0x01, 0x08, 0x43, 0x22, 0xD0, 0xF8, 0xB2, 0x00, 0x21, 0x13, 0xF0, 0x85, 0xFB,
+ 0x41, 0x07, 0x05, 0xD5, 0x00, 0x99, 0xC9, 0x06, 0x02, 0xD5, 0x49, 0x46, 0x62, 0x20, 0x10, 0xE0,
+ 0xC1, 0x07, 0x05, 0xD0, 0x00, 0x99, 0x49, 0x07, 0x02, 0xD5, 0x49, 0x46, 0x60, 0x20, 0x08, 0xE0,
+ 0x00, 0x99, 0x11, 0xF0, 0xC0, 0x0F, 0x06, 0xD0, 0x10, 0xF0, 0x0A, 0x0F, 0x03, 0xD0, 0x49, 0x46,
+ 0x64, 0x20, 0xFF, 0xF7, 0xBD, 0xFE, 0x09, 0xF1, 0x01, 0x00, 0x00, 0xF0, 0xFF, 0x09, 0x7F, 0x1C,
+ 0x57, 0x45, 0xCF, 0xDB, 0x13, 0xF0, 0x59, 0xFB, 0x00, 0x27, 0x3E, 0x46, 0x32, 0x46, 0x01, 0x20,
+ 0x00, 0x21, 0x15, 0xF0, 0xB8, 0xF8, 0x20, 0x40, 0x29, 0x40, 0x08, 0x43, 0x20, 0xD0, 0xF0, 0xB2,
+ 0x01, 0x21, 0x13, 0xF0, 0x52, 0xFB, 0x41, 0x07, 0x05, 0xD5, 0x00, 0x99, 0x89, 0x06, 0x02, 0xD5,
+ 0x39, 0x46, 0x63, 0x20, 0x10, 0xE0, 0xC1, 0x07, 0x05, 0xD0, 0x00, 0x99, 0x09, 0x07, 0x02, 0xD5,
+ 0x39, 0x46, 0x61, 0x20, 0x08, 0xE0, 0x00, 0x99, 0x11, 0xF4, 0xC0, 0x7F, 0x06, 0xD0, 0x10, 0xF0,
+ 0x0A, 0x0F, 0x03, 0xD0, 0x39, 0x46, 0x65, 0x20, 0xFF, 0xF7, 0x8A, 0xFE, 0x7F, 0x1C, 0xFF, 0xB2,
+ 0x76, 0x1C, 0x5E, 0x45, 0xD2, 0xDD, 0xBD, 0xE8, 0xF8, 0x4F, 0x13, 0xF0, 0x26, 0xBB, 0x10, 0xB5,
+ 0x04, 0x46, 0x00, 0x20, 0x13, 0xF0, 0xCF, 0xF8, 0x38, 0x49, 0x00, 0x20, 0x14, 0xF0, 0x0C, 0x0F,
+ 0xC1, 0xE9, 0x08, 0x00, 0xC1, 0xE9, 0x02, 0x00, 0xC1, 0xE9, 0x04, 0x00, 0xC1, 0xE9, 0x06, 0x00,
+ 0xC1, 0xE9, 0x10, 0x00, 0xC1, 0xE9, 0x0A, 0x00, 0xC1, 0xE9, 0x0C, 0x00, 0xC1, 0xE9, 0x0E, 0x00,
+ 0x48, 0x60, 0x08, 0x60, 0x09, 0xD1, 0xE0, 0x06, 0x07, 0xD4, 0xA0, 0x06, 0x05, 0xD4, 0x60, 0x06,
+ 0x03, 0xD4, 0xE0, 0x05, 0x01, 0xD4, 0x20, 0x06, 0x02, 0xD5, 0x20, 0x46, 0xFF, 0xF7, 0x5A, 0xFF,
+ 0xA0, 0x07, 0x02, 0xD0, 0x20, 0x46, 0x00, 0xF0, 0x22, 0xF8, 0xFF, 0xF7, 0x04, 0xFF, 0xBD, 0xE8,
+ 0x10, 0x40, 0x00, 0x20, 0x13, 0xF0, 0xBA, 0xB8, 0x2D, 0xE9, 0xF0, 0x41, 0x16, 0x46, 0x1F, 0x46,
+ 0x80, 0x46, 0x4F, 0xF0, 0xFF, 0x35, 0x00, 0x24, 0x22, 0x46, 0x01, 0x20, 0x00, 0x21, 0x15, 0xF0,
+ 0x4A, 0xF8, 0x30, 0x40, 0x39, 0x40, 0x08, 0x43, 0x04, 0xD0, 0x6D, 0x1C, 0x45, 0x45, 0x01, 0xD1,
+ 0x20, 0x46, 0xD1, 0xE6, 0x64, 0x1C, 0x40, 0x2C, 0xEE, 0xD3, 0x00, 0x20, 0xCC, 0xE6, 0x2D, 0xE9,
+ 0xFF, 0x41, 0x04, 0x46, 0x01, 0x20, 0x00, 0xF0, 0x31, 0xF8, 0x80, 0x46, 0x12, 0xF0, 0xC5, 0xFB,
+ 0xCD, 0xE9, 0x00, 0x01, 0x12, 0xF0, 0xC3, 0xFB, 0xCD, 0xE9, 0x02, 0x01, 0x08, 0x21, 0x68, 0x46,
+ 0x13, 0xF0, 0xA0, 0xFD, 0x07, 0x46, 0x08, 0x21, 0x02, 0xA8, 0x13, 0xF0, 0x9B, 0xFD, 0x05, 0x46,
+ 0x07, 0x4E, 0xE0, 0x07, 0x07, 0xD0, 0x30, 0x68, 0x42, 0x46, 0x29, 0x46, 0xB0, 0xF8, 0xDE, 0x30,
+ 0x38, 0x46, 0x00, 0xF0, 0x4C, 0xF8, 0x05, 0xE0, 0xC0, 0x08, 0x10, 0x00, 0x08, 0x04, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0xA0, 0x07, 0x07, 0xD5, 0x30, 0x68, 0x42, 0x46, 0x29, 0x46, 0xB0, 0xF8,
+ 0xDE, 0x30, 0x38, 0x46, 0x00, 0xF0, 0x8B, 0xF8, 0xBD, 0xE8, 0xFF, 0x81, 0x30, 0xB5, 0x85, 0xB0,
+ 0x04, 0x46, 0x68, 0x46, 0x12, 0xF0, 0xF2, 0xFA, 0x67, 0x48, 0x00, 0x90, 0x67, 0x48, 0x00, 0x68,
+ 0x90, 0xF8, 0xB0, 0x00, 0xC0, 0x07, 0x02, 0xD0, 0x01, 0x20, 0x8D, 0xF8, 0x0B, 0x00, 0x02, 0x2C,
+ 0x15, 0xD0, 0x07, 0x20, 0x8D, 0xF8, 0x04, 0x00, 0x61, 0x4C, 0x68, 0x46, 0x12, 0xF0, 0x03, 0xFB,
+ 0x00, 0x22, 0x04, 0xAB, 0x01, 0x21, 0x10, 0x46, 0x12, 0xF0, 0xD6, 0xFF, 0x01, 0x21, 0x00, 0x20,
+ 0x12, 0xF0, 0xB5, 0xFF, 0x20, 0x78, 0x30, 0xB1, 0x0E, 0xF0, 0xBD, 0xFD, 0xED, 0xE7, 0x03, 0x20,
+ 0x8D, 0xF8, 0x09, 0x00, 0xE8, 0xE7, 0x12, 0xF0, 0x68, 0xFB, 0x56, 0x4D, 0xC5, 0xE9, 0x12, 0x01,
+ 0x12, 0xF0, 0x65, 0xFB, 0xC5, 0xE9, 0x14, 0x01, 0xA0, 0x68, 0x05, 0xB0, 0x30, 0xBD, 0x2D, 0xE9,
+ 0xFF, 0x5F, 0x9B, 0x46, 0x92, 0x46, 0x0F, 0x46, 0x00, 0x25, 0x44, 0xE0, 0x00, 0x98, 0x4F, 0xF0,
+ 0x00, 0x09, 0x4C, 0x46, 0xA0, 0xF1, 0x01, 0x08, 0x3A, 0xE0, 0x00, 0xBF, 0x04, 0xFB, 0x07, 0x50,
+ 0x81, 0xB2, 0x60, 0x1C, 0x00, 0xFB, 0x07, 0x50, 0x80, 0xB2, 0x3A, 0xF8, 0x11, 0x20, 0x3A, 0xF8,
+ 0x10, 0x00, 0x10, 0x1A, 0x00, 0xB2, 0x58, 0x45, 0x02, 0xDD, 0xB9, 0xF1, 0x00, 0x0F, 0x0A, 0xD0,
+ 0xCB, 0xF1, 0x00, 0x01, 0x88, 0x42, 0x13, 0xDA, 0xB9, 0xF1, 0x00, 0x0F, 0x06, 0xD0, 0x4F, 0xF0,
+ 0x00, 0x09, 0x21, 0x46, 0x18, 0xE0, 0x4F, 0xF0, 0x01, 0x09, 0x14, 0xE0, 0x00, 0x26, 0x04, 0xE0,
+ 0x31, 0x46, 0x66, 0x20, 0xFF, 0xF7, 0x8C, 0xFD, 0x76, 0x1C, 0xA6, 0x42, 0xF8, 0xD9, 0x0E, 0xE0,
+ 0xB9, 0xF1, 0x00, 0x0F, 0x0B, 0xD0, 0x21, 0x46, 0x66, 0x20, 0xFF, 0xF7, 0x81, 0xFD, 0x00, 0x98,
+ 0x80, 0x1E, 0x84, 0x42, 0x03, 0xD1, 0x61, 0x1C, 0x66, 0x20, 0xFF, 0xF7, 0x79, 0xFD, 0x64, 0x1C,
+ 0x44, 0x45, 0xC3, 0xD3, 0x6D, 0x1C, 0xBD, 0x42, 0xB8, 0xD3, 0xBD, 0xE8, 0xFF, 0x9F, 0x2D, 0xE9,
+ 0xFF, 0x5F, 0x9B, 0x46, 0x92, 0x46, 0x0D, 0x46, 0x00, 0x26, 0x41, 0xE0, 0x4F, 0xF0, 0x00, 0x09,
+ 0x4C, 0x46, 0xA5, 0xF1, 0x01, 0x08, 0x38, 0xE0, 0x06, 0xFB, 0x05, 0x40, 0x61, 0x1C, 0x06, 0xFB,
+ 0x05, 0x11, 0x80, 0xB2, 0x89, 0xB2, 0x3A, 0xF8, 0x10, 0x20, 0x3A, 0xF8, 0x11, 0x00, 0x10, 0x1A,
+ 0x00, 0xB2, 0x58, 0x45, 0x02, 0xDD, 0xB9, 0xF1, 0x00, 0x0F, 0x0A, 0xD0, 0xCB, 0xF1, 0x00, 0x01,
+ 0x88, 0x42, 0x13, 0xDA, 0xB9, 0xF1, 0x00, 0x0F, 0x06, 0xD0, 0x4F, 0xF0, 0x00, 0x09, 0x21, 0x46,
+ 0x17, 0xE0, 0x4F, 0xF0, 0x01, 0x09, 0x13, 0xE0, 0x00, 0x27, 0x04, 0xE0, 0x39, 0x46, 0x67, 0x20,
+ 0xFF, 0xF7, 0x3E, 0xFD, 0x7F, 0x1C, 0xA7, 0x42, 0xF8, 0xD9, 0x0D, 0xE0, 0xB9, 0xF1, 0x00, 0x0F,
+ 0x0A, 0xD0, 0x21, 0x46, 0x67, 0x20, 0xFF, 0xF7, 0x33, 0xFD, 0xA8, 0x1E, 0x84, 0x42, 0x03, 0xD1,
+ 0x61, 0x1C, 0x67, 0x20, 0xFF, 0xF7, 0x2C, 0xFD, 0x64, 0x1C, 0x44, 0x45, 0xC4, 0xD3, 0x76, 0x1C,
+ 0x00, 0x98, 0x86, 0x42, 0xBA, 0xD3, 0xB0, 0xE7, 0x29, 0x13, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0xC0, 0x08, 0x10, 0x00, 0x08, 0x04, 0x10, 0x00, 0x11, 0x49, 0x08, 0x60, 0x4F, 0xF4, 0x00, 0x51,
+ 0x00, 0x20, 0x12, 0xF0, 0xC8, 0xBE, 0x38, 0xB5, 0xAF, 0xF2, 0x13, 0x00, 0x14, 0xF0, 0xC2, 0xFD,
+ 0x00, 0x22, 0x4F, 0xF4, 0x00, 0x54, 0x6B, 0x46, 0x21, 0x46, 0x10, 0x46, 0x12, 0xF0, 0x04, 0xFF,
+ 0x21, 0x46, 0x00, 0x20, 0x12, 0xF0, 0xE3, 0xFE, 0x06, 0x48, 0xA5, 0x22, 0x01, 0x68, 0x04, 0x48,
+ 0x0A, 0x74, 0x00, 0x68, 0x20, 0xF0, 0x7F, 0x42, 0x4A, 0x61, 0x14, 0xF0, 0xC9, 0xFD, 0x38, 0xBD,
+ 0x60, 0x04, 0x10, 0x00, 0x24, 0x07, 0x10, 0x00, 0x02, 0x00, 0x04, 0xD0, 0x01, 0x2A, 0x04, 0xD0,
+ 0x02, 0x28, 0x03, 0xD1, 0x04, 0xE0, 0x01, 0x20, 0x00, 0xE0, 0x04, 0x20, 0x12, 0xF0, 0x8F, 0xB9,
+ 0x00, 0x20, 0x70, 0x47, 0x10, 0xB5, 0x01, 0x46, 0x34, 0x22, 0xFE, 0x48, 0x14, 0xF0, 0x67, 0xFF,
+ 0x01, 0x21, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0x20, 0x12, 0xF0, 0x8D, 0xBE, 0x30, 0xB5, 0x85, 0xB0,
+ 0x04, 0x46, 0x68, 0x46, 0x12, 0xF0, 0xD2, 0xF9, 0xF7, 0x48, 0x00, 0x90, 0x02, 0x20, 0x6C, 0xB3,
+ 0x01, 0x2C, 0x2D, 0xD0, 0x02, 0x2C, 0x2D, 0xD0, 0x03, 0x2C, 0x25, 0xD1, 0x5F, 0xF0, 0x08, 0x01,
+ 0x8D, 0xF8, 0x04, 0x10, 0xF1, 0x49, 0x09, 0x68, 0x91, 0xF8, 0xB0, 0x10, 0xCA, 0x07, 0x04, 0xD0,
+ 0x89, 0x07, 0x00, 0xD4, 0x01, 0x20, 0x8D, 0xF8, 0x0B, 0x00, 0xEA, 0x4D, 0x03, 0x24, 0x68, 0x46,
+ 0x12, 0xF0, 0xD9, 0xF9, 0x00, 0x22, 0x04, 0xAB, 0x01, 0x21, 0x10, 0x46, 0x12, 0xF0, 0xAC, 0xFE,
+ 0x01, 0x21, 0x00, 0x20, 0x12, 0xF0, 0x8B, 0xFE, 0x28, 0x78, 0x00, 0x28, 0x04, 0xD0, 0x0E, 0xF0,
+ 0x92, 0xFC, 0x64, 0x1E, 0x00, 0x2C, 0xEA, 0xDC, 0x05, 0xB0, 0x30, 0xBD, 0x05, 0x21, 0xD7, 0xE7,
+ 0x06, 0x21, 0xD5, 0xE7, 0x8D, 0xF8, 0x09, 0x00, 0xD4, 0xE7, 0x2D, 0xE9, 0xF1, 0x4F, 0x88, 0xB0,
+ 0x12, 0xF0, 0x33, 0xFA, 0x83, 0x46, 0x02, 0x91, 0x12, 0xF0, 0x31, 0xFA, 0x81, 0x46, 0xD7, 0x4C,
+ 0x08, 0x98, 0x8A, 0x46, 0x00, 0x28, 0x20, 0x68, 0x07, 0xD0, 0xB0, 0xF8, 0x7A, 0x11, 0x08, 0x98,
+ 0xFF, 0xF7, 0x92, 0xFF, 0x04, 0x90, 0xD2, 0x48, 0x12, 0xE0, 0x90, 0xF8, 0xE7, 0x00, 0x10, 0xF0,
+ 0x60, 0x0F, 0x05, 0xD0, 0x58, 0x46, 0x02, 0x99, 0x0F, 0xF0, 0x64, 0xFD, 0x83, 0x46, 0x02, 0x91,
+ 0x20, 0x68, 0xB0, 0xF8, 0x72, 0x11, 0x08, 0x98, 0xFF, 0xF7, 0x7E, 0xFF, 0x04, 0x90, 0xC9, 0x48,
+ 0x03, 0x90, 0x40, 0x20, 0x07, 0x90, 0x06, 0x90, 0xC2, 0x48, 0xB8, 0x21, 0xB8, 0x38, 0x14, 0xF0,
+ 0x4A, 0xFF, 0x00, 0x20, 0x00, 0x90, 0x08, 0x98, 0xFF, 0xF7, 0x88, 0xFF, 0xBD, 0x48, 0x00, 0x27,
+ 0x3E, 0x46, 0x80, 0x68, 0x05, 0x90, 0x01, 0x20, 0x3D, 0x46, 0x01, 0x90, 0x2A, 0x46, 0x01, 0x20,
+ 0x00, 0x21, 0x14, 0xF0, 0x60, 0xFE, 0x02, 0x9A, 0x00, 0xEA, 0x0B, 0x00, 0x11, 0x40, 0x08, 0x43,
+ 0x41, 0xD0, 0x00, 0x24, 0x22, 0x46, 0x01, 0x20, 0x00, 0x21, 0x14, 0xF0, 0x54, 0xFE, 0x02, 0x46,
+ 0x0B, 0x46, 0x00, 0xEA, 0x09, 0x00, 0x01, 0xEA, 0x0A, 0x01, 0x08, 0x43, 0x20, 0xD0, 0xAD, 0x48,
+ 0xB8, 0x38, 0x00, 0xEB, 0xC6, 0x01, 0x8E, 0x46, 0x03, 0xC9, 0x00, 0xEA, 0x02, 0x0C, 0x01, 0xEA,
+ 0x03, 0x08, 0x5C, 0xEA, 0x08, 0x0C, 0x12, 0xD1, 0xDD, 0xF8, 0x14, 0xC0, 0x3C, 0xF9, 0x17, 0x80,
+ 0xDD, 0xF8, 0x10, 0xC0, 0xE0, 0x45, 0x04, 0xDC, 0x10, 0x43, 0x19, 0x43, 0xCE, 0xE9, 0x00, 0x01,
+ 0x05, 0xE0, 0x22, 0x46, 0x31, 0x46, 0x00, 0x9B, 0x03, 0x98, 0x13, 0xF0, 0x1E, 0xF8, 0x7F, 0x1C,
+ 0x06, 0x98, 0x64, 0x1C, 0x84, 0x42, 0xCD, 0xDB, 0x9A, 0x48, 0xB8, 0x38, 0x00, 0xEB, 0xC6, 0x01,
+ 0xD1, 0xE9, 0x00, 0x02, 0x80, 0xEA, 0x09, 0x00, 0x82, 0xEA, 0x0A, 0x02, 0x10, 0x43, 0x01, 0xD0,
+ 0x00, 0x20, 0x01, 0x90, 0x76, 0x1C, 0x07, 0x98, 0x6D, 0x1C, 0x85, 0x42, 0xAE, 0xDB, 0x01, 0x98,
+ 0x01, 0x28, 0x04, 0xD0, 0x00, 0x98, 0x40, 0x1C, 0x00, 0x90, 0x7F, 0x28, 0x9B, 0xDD, 0x01, 0x98,
+ 0x09, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F, 0x2D, 0xE9, 0xF1, 0x4F, 0x98, 0xB0, 0x12, 0xF0, 0x95, 0xF9,
+ 0xCD, 0xE9, 0x0C, 0x01, 0x12, 0xF0, 0x93, 0xF9, 0xCD, 0xE9, 0x10, 0x01, 0x18, 0x98, 0x87, 0x4C,
+ 0x00, 0x28, 0x5C, 0xD0, 0x20, 0x68, 0xB0, 0xF8, 0x7A, 0x11, 0x18, 0x98, 0xFF, 0xF7, 0xF4, 0xFE,
+ 0x0F, 0x90, 0x83, 0x48, 0x06, 0x90, 0x40, 0x20, 0x82, 0x46, 0x0E, 0x90, 0x08, 0x21, 0x0C, 0xA8,
+ 0x13, 0xF0, 0x60, 0xFB, 0x04, 0x46, 0x08, 0x21, 0x10, 0xA8, 0x13, 0xF0, 0x5B, 0xFB, 0x44, 0x43,
+ 0x7D, 0x48, 0x01, 0x90, 0x00, 0xF5, 0xE6, 0x60, 0x21, 0x46, 0x02, 0x90, 0x14, 0xF0, 0x91, 0xFE,
+ 0x61, 0x00, 0x01, 0x98, 0x14, 0xF0, 0x8D, 0xFE, 0x72, 0x48, 0xB8, 0x21, 0xB8, 0x38, 0x14, 0xF0,
+ 0xAA, 0xFE, 0x4F, 0xF0, 0x00, 0x09, 0x18, 0x98, 0xFF, 0xF7, 0xE8, 0xFE, 0x6D, 0x48, 0x00, 0x27,
+ 0x3C, 0x46, 0x80, 0x68, 0x0A, 0x90, 0xC9, 0xF1, 0x00, 0x00, 0x4F, 0xF0, 0x01, 0x0B, 0x3E, 0x46,
+ 0x13, 0x90, 0x32, 0x46, 0x01, 0x20, 0x00, 0x21, 0x14, 0xF0, 0xBD, 0xFD, 0xDD, 0xE9, 0x0C, 0x23,
+ 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x6E, 0xD0, 0x00, 0x25, 0x2A, 0x46, 0x01, 0x20, 0x00, 0x21,
+ 0x14, 0xF0, 0xB1, 0xFD, 0xDD, 0xE9, 0x10, 0x8C, 0x02, 0x46, 0x0B, 0x46, 0x00, 0xEA, 0x08, 0x00,
+ 0x01, 0xEA, 0x0C, 0x01, 0x08, 0x43, 0x73, 0xD0, 0x5A, 0x48, 0xB8, 0x38, 0x00, 0xEB, 0xC7, 0x0C,
+ 0xDC, 0xE9, 0x00, 0x01, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x0F, 0xD0, 0x93, 0xE0, 0xDD, 0xE9,
+ 0x0C, 0x01, 0x0F, 0xF0, 0x77, 0xFC, 0xCD, 0xE9, 0x0C, 0x01, 0x20, 0x68, 0xB0, 0xF8, 0x72, 0x11,
+ 0x18, 0x98, 0xFF, 0xF7, 0x91, 0xFE, 0x0F, 0x90, 0x52, 0x48, 0x9B, 0xE7, 0xB9, 0xF1, 0x00, 0x0F,
+ 0x0A, 0xDA, 0x0A, 0x98, 0x30, 0xF9, 0x14, 0x10, 0x0F, 0x98, 0x81, 0x42, 0x01, 0xDB, 0x00, 0x21,
+ 0x00, 0xE0, 0x01, 0x21, 0x02, 0x98, 0x01, 0x55, 0x0A, 0x98, 0x0F, 0x99, 0x30, 0xF9, 0x14, 0x00,
+ 0x88, 0x42, 0x31, 0xDD, 0x02, 0x99, 0x09, 0x5D, 0x11, 0xB1, 0x01, 0x29, 0x09, 0xD0, 0x6A, 0xE0,
+ 0xB9, 0xF1, 0x7F, 0x0F, 0x67, 0xDA, 0x01, 0x99, 0x09, 0xF1, 0x01, 0x03, 0x21, 0xF8, 0x14, 0x00,
+ 0x51, 0xE0, 0xDC, 0xE9, 0x00, 0x10, 0x11, 0x43, 0x18, 0x43, 0xCC, 0xE9, 0x00, 0x10, 0x01, 0x98,
+ 0x30, 0xF9, 0x14, 0x10, 0x0F, 0x98, 0x09, 0x1A, 0x0A, 0x98, 0x30, 0xF9, 0x14, 0x20, 0x0F, 0x98,
+ 0xA2, 0xEB, 0x00, 0x00, 0x00, 0xD5, 0x49, 0x42, 0x00, 0x28, 0x00, 0xDA, 0x40, 0x42, 0x81, 0x42,
+ 0x05, 0xDC, 0x13, 0x9B, 0x2A, 0x46, 0x39, 0x46, 0x5B, 0x1C, 0x06, 0x98, 0x36, 0xE0, 0x01, 0x98,
+ 0x20, 0xF8, 0x14, 0x20, 0x3F, 0xE0, 0x52, 0xE0, 0x37, 0xDA, 0x02, 0x99, 0x09, 0x5D, 0x01, 0x29,
+ 0x01, 0xD0, 0x71, 0xB1, 0x37, 0xE0, 0x13, 0x99, 0x11, 0xF1, 0x7F, 0x0F, 0x33, 0xDD, 0x01, 0x99,
+ 0x2A, 0x46, 0x21, 0xF8, 0x14, 0x00, 0x13, 0x9B, 0x39, 0x46, 0x5B, 0x1E, 0x06, 0x98, 0x1D, 0xE0,
+ 0x2A, 0xE0, 0xDC, 0xE9, 0x00, 0x10, 0x11, 0x43, 0x18, 0x43, 0xCC, 0xE9, 0x00, 0x10, 0x01, 0x98,
+ 0x0F, 0x9A, 0x30, 0xF9, 0x14, 0x10, 0x0F, 0x98, 0x08, 0x1A, 0x0A, 0x99, 0x31, 0xF9, 0x14, 0x10,
+ 0xA1, 0xEB, 0x02, 0x02, 0x00, 0xD5, 0x40, 0x42, 0x00, 0x2A, 0x00, 0xDA, 0x52, 0x42, 0x90, 0x42,
+ 0x07, 0xDC, 0xA9, 0xF1, 0x01, 0x03, 0x2A, 0x46, 0x39, 0x46, 0x06, 0x98, 0x12, 0xF0, 0x05, 0xFF,
+ 0x09, 0xE0, 0x01, 0x98, 0x20, 0xF8, 0x14, 0x10, 0x05, 0xE0, 0xDC, 0xE9, 0x00, 0x10, 0x11, 0x43,
+ 0x18, 0x43, 0xCC, 0xE9, 0x00, 0x10, 0x64, 0x1C, 0x6D, 0x1C, 0x55, 0x45, 0xFF, 0xF6, 0x4D, 0xAF,
+ 0x08, 0x48, 0xB8, 0x38, 0x00, 0xEB, 0xC7, 0x01, 0xD1, 0xE9, 0x00, 0x01, 0xDD, 0xE9, 0x10, 0x23,
+ 0x50, 0x40, 0x59, 0x40, 0x08, 0x43, 0x01, 0xD0, 0x4F, 0xF0, 0x00, 0x0B, 0x7F, 0x1C, 0x76, 0x1C,
+ 0x0E, 0x98, 0x0B, 0xE0, 0xB0, 0x09, 0x10, 0x00, 0x75, 0x19, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x24, 0x52, 0x01, 0x20, 0xFC, 0x4C, 0x01, 0x20, 0x84, 0x5A, 0x10, 0x00, 0x86, 0x42, 0xFF, 0xF6,
+ 0x20, 0xAF, 0xBB, 0xF1, 0x01, 0x0F, 0x05, 0xD0, 0x09, 0xF1, 0x01, 0x09, 0xB9, 0xF1, 0x7F, 0x0F,
+ 0x7F, 0xF7, 0x09, 0xAF, 0x58, 0x46, 0x19, 0xB0, 0xCB, 0xE6, 0x2D, 0xE9, 0xF1, 0x4F, 0x98, 0xB0,
+ 0x4F, 0xF0, 0x00, 0x0A, 0x12, 0xF0, 0x61, 0xF8, 0xCD, 0xE9, 0x0E, 0x01, 0x12, 0xF0, 0x5F, 0xF8,
+ 0xCD, 0xE9, 0x10, 0x01, 0x18, 0x98, 0x48, 0xB1, 0xFD, 0x48, 0x00, 0x68, 0xB0, 0xF8, 0x7A, 0x11,
+ 0x18, 0x98, 0xFF, 0xF7, 0xC1, 0xFD, 0x13, 0x90, 0xFA, 0x48, 0x14, 0xE0, 0xF8, 0x4C, 0x20, 0x68,
+ 0x90, 0xF8, 0xE7, 0x00, 0x10, 0xF0, 0x60, 0x0F, 0x05, 0xD0, 0xDD, 0xE9, 0x0E, 0x01, 0x0F, 0xF0,
+ 0x91, 0xFB, 0xCD, 0xE9, 0x0E, 0x01, 0x20, 0x68, 0xB0, 0xF8, 0x72, 0x11, 0x18, 0x98, 0xFF, 0xF7,
+ 0xAB, 0xFD, 0x13, 0x90, 0xF0, 0x48, 0x40, 0x25, 0xCD, 0xE9, 0x04, 0x05, 0x08, 0x21, 0x0E, 0xA8,
+ 0x13, 0xF0, 0x18, 0xFA, 0x03, 0x90, 0x08, 0x21, 0x10, 0xA8, 0x13, 0xF0, 0x13, 0xFA, 0x0A, 0x90,
+ 0xB8, 0x21, 0xEA, 0x48, 0x14, 0xF0, 0x6F, 0xFD, 0x01, 0x20, 0x12, 0x90, 0x00, 0x20, 0x04, 0x46,
+ 0x08, 0x90, 0x22, 0x46, 0x01, 0x20, 0x00, 0x21, 0x14, 0xF0, 0x8D, 0xFC, 0xDD, 0xE9, 0x10, 0x23,
+ 0x81, 0x46, 0x8B, 0x46, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x7D, 0xD0, 0xDC, 0x48, 0x00, 0x68,
+ 0x90, 0xF8, 0x70, 0x01, 0x80, 0x07, 0x13, 0xD5, 0x08, 0x98, 0x80, 0xB1, 0x00, 0x26, 0x0B, 0xE0,
+ 0x31, 0x46, 0x02, 0x9A, 0x04, 0x98, 0x12, 0xF0, 0x5D, 0xFE, 0x03, 0x46, 0x22, 0x46, 0x31, 0x46,
+ 0x04, 0x98, 0x12, 0xF0, 0x62, 0xFE, 0x76, 0x1C, 0x03, 0x98, 0x86, 0x42, 0xF0, 0xDB, 0x02, 0x94,
+ 0x18, 0x98, 0xFF, 0xF7, 0x83, 0xFD, 0xD1, 0x48, 0x4F, 0xF0, 0x01, 0x08, 0xB8, 0x30, 0x00, 0x27,
+ 0x80, 0x68, 0x3E, 0x46, 0x16, 0x90, 0x32, 0x46, 0x01, 0x20, 0x00, 0x21, 0x14, 0xF0, 0x5B, 0xFC,
+ 0xDD, 0xE9, 0x0E, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x45, 0xD0, 0xC7, 0x48, 0x00, 0xEB,
+ 0xC7, 0x01, 0x0D, 0x91, 0xD1, 0xE9, 0x00, 0x01, 0x00, 0xEA, 0x09, 0x00, 0x01, 0xEA, 0x0B, 0x01,
+ 0x08, 0x43, 0x2E, 0xD1, 0x22, 0x46, 0x39, 0x46, 0x04, 0x98, 0x12, 0xF0, 0x2B, 0xFE, 0x0A, 0x9A,
+ 0x08, 0x99, 0x07, 0xFB, 0x02, 0x12, 0x16, 0x99, 0x31, 0xF9, 0x12, 0x20, 0x13, 0x99, 0x8A, 0x42,
+ 0x0A, 0xDA, 0x0D, 0x98, 0x0D, 0x99, 0xD0, 0xE9, 0x00, 0x30, 0x43, 0xEA, 0x09, 0x03, 0x40, 0xEA,
+ 0x0B, 0x00, 0xC1, 0xE9, 0x00, 0x30, 0x14, 0xE0, 0x7F, 0x28, 0x06, 0xDA, 0x43, 0x1C, 0x22, 0x46,
+ 0x39, 0x46, 0x04, 0x98, 0x12, 0xF0, 0x19, 0xFE, 0x0B, 0xE0, 0x0D, 0x98, 0x0D, 0x9A, 0xD0, 0xE9,
+ 0x00, 0x10, 0x41, 0xEA, 0x09, 0x01, 0x40, 0xEA, 0x0B, 0x00, 0xC2, 0xE9, 0x00, 0x10, 0x00, 0x20,
+ 0x12, 0x90, 0x0D, 0x99, 0xD1, 0xE9, 0x00, 0x01, 0x00, 0xEA, 0x09, 0x00, 0x01, 0xEA, 0x0B, 0x01,
+ 0x08, 0x43, 0x00, 0xD1, 0x80, 0x46, 0x7F, 0x1C, 0x76, 0x1C, 0xAE, 0x42, 0xAB, 0xDB, 0xB8, 0xF1,
+ 0x00, 0x0F, 0x9D, 0xD0, 0x08, 0x98, 0x00, 0xE0, 0x01, 0xE0, 0x40, 0x1C, 0x08, 0x90, 0x05, 0x98,
+ 0x64, 0x1C, 0x84, 0x42, 0xFF, 0xF6, 0x6D, 0xAF, 0x00, 0x20, 0xB9, 0xE0, 0x4F, 0xF0, 0x00, 0x08,
+ 0x46, 0x46, 0x32, 0x46, 0x01, 0x20, 0x00, 0x21, 0x14, 0xF0, 0xF5, 0xFB, 0xDD, 0xE9, 0x10, 0x23,
+ 0x83, 0x46, 0x03, 0x91, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x76, 0xD0, 0xB8, 0xF1, 0x00, 0x0F,
+ 0x01, 0xD1, 0x4F, 0xF0, 0x01, 0x0A, 0x18, 0x98, 0xFF, 0xF7, 0x00, 0xFD, 0x8F, 0x48, 0x4F, 0xF0,
+ 0x01, 0x09, 0xB8, 0x30, 0x00, 0x24, 0x80, 0x68, 0x27, 0x46, 0x16, 0x90, 0x3A, 0x46, 0x01, 0x20,
+ 0x00, 0x21, 0x14, 0xF0, 0xD8, 0xFB, 0xDD, 0xE9, 0x0E, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43,
+ 0x78, 0xD0, 0xB8, 0xF1, 0x00, 0x0F, 0x0E, 0xDD, 0x32, 0x46, 0x21, 0x46, 0x04, 0x98, 0x12, 0xF0,
+ 0xB1, 0xFD, 0x82, 0x49, 0x32, 0x46, 0xEC, 0x31, 0x09, 0x57, 0x0B, 0x18, 0x21, 0x46, 0x04, 0x98,
+ 0x12, 0xF0, 0xB3, 0xFD, 0x65, 0xE0, 0xBA, 0xF1, 0x01, 0x0F, 0x12, 0xD1, 0x32, 0x46, 0x21, 0x46,
+ 0x04, 0x98, 0x12, 0xF0, 0x9F, 0xFD, 0x79, 0x49, 0xEC, 0x31, 0x08, 0x55, 0x77, 0x48, 0x00, 0xEB,
+ 0xC4, 0x02, 0xD2, 0xE9, 0x00, 0x01, 0x03, 0x9B, 0x20, 0xEA, 0x0B, 0x00, 0x99, 0x43, 0xC2, 0xE9,
+ 0x00, 0x01, 0x72, 0x48, 0x00, 0xEB, 0xC4, 0x01, 0x0B, 0x91, 0xD1, 0xE9, 0x00, 0x01, 0x03, 0x9A,
+ 0x00, 0xEA, 0x0B, 0x00, 0x11, 0x40, 0x08, 0x43, 0x39, 0xD1, 0x32, 0x46, 0x21, 0x46, 0x04, 0x98,
+ 0x12, 0xF0, 0x80, 0xFD, 0x0A, 0x9A, 0x16, 0x99, 0x04, 0xFB, 0x02, 0x82, 0x31, 0xF9, 0x12, 0x20,
+ 0x13, 0x99, 0x8A, 0x42, 0x10, 0xDA, 0x0B, 0x9A, 0xDD, 0xF8, 0x0C, 0xC0, 0xD2, 0xE9, 0x00, 0x13,
+ 0x41, 0xEA, 0x0B, 0x01, 0x43, 0xEA, 0x0C, 0x03, 0xC2, 0xE9, 0x00, 0x13, 0x5F, 0x49, 0xEC, 0x31,
+ 0x0A, 0x5D, 0x80, 0x1A, 0x08, 0x55, 0x1A, 0xE0, 0x7F, 0x28, 0x07, 0xDA, 0x43, 0x1C, 0x32, 0x46,
+ 0x21, 0x46, 0x04, 0x98, 0x12, 0xF0, 0x69, 0xFD, 0x11, 0xE0, 0x27, 0xE0, 0x0B, 0x99, 0xD1, 0xE9,
+ 0x00, 0x32, 0x03, 0x99, 0x43, 0xEA, 0x0B, 0x03, 0x0A, 0x43, 0x0B, 0x99, 0xC1, 0xE9, 0x00, 0x32,
+ 0x52, 0x49, 0xEC, 0x31, 0x0A, 0x5D, 0x80, 0x1A, 0x08, 0x55, 0x00, 0x20, 0x12, 0x90, 0x0B, 0x98,
+ 0x03, 0x9A, 0xD0, 0xE9, 0x00, 0x10, 0x01, 0xEA, 0x0B, 0x01, 0x10, 0x40, 0x01, 0x43, 0x00, 0xD1,
+ 0x89, 0x46, 0x64, 0x1C, 0x7F, 0x1C, 0xAF, 0x42, 0xFF, 0xF6, 0x78, 0xAF, 0x4F, 0xF0, 0x00, 0x0A,
+ 0xB9, 0xF1, 0x00, 0x0F, 0x3F, 0xF4, 0x67, 0xAF, 0x08, 0xF1, 0x01, 0x08, 0x05, 0x98, 0x76, 0x1C,
+ 0x86, 0x42, 0xFF, 0xF6, 0x4E, 0xAF, 0x18, 0x98, 0xFF, 0xF7, 0x60, 0xFC, 0x0C, 0x98, 0x40, 0x1C,
+ 0x0C, 0x90, 0x3B, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x70, 0x11, 0x0C, 0x98, 0xB0, 0xEB, 0x11, 0x1F,
+ 0xFF, 0xF6, 0x3C, 0xAF, 0x12, 0x98, 0x5E, 0xE6, 0x2D, 0xE9, 0xF0, 0x4F, 0x04, 0x46, 0x85, 0xB0,
+ 0x00, 0x20, 0x12, 0xF0, 0x78, 0xFB, 0x02, 0x20, 0x12, 0xF0, 0x75, 0xFB, 0x11, 0xF0, 0xBF, 0xFE,
+ 0xCD, 0xE9, 0x02, 0x01, 0x32, 0x48, 0x04, 0x90, 0x40, 0x27, 0x08, 0x21, 0x02, 0xA8, 0x13, 0xF0,
+ 0x99, 0xF8, 0x80, 0x46, 0x01, 0x2C, 0x7F, 0xD1, 0xDD, 0xE9, 0x02, 0x01, 0xCD, 0xE9, 0x00, 0x01,
+ 0x01, 0x22, 0x00, 0x23, 0x04, 0x98, 0x12, 0xF0, 0x17, 0xFD, 0xDF, 0xF8, 0xA8, 0x90, 0x00, 0x20,
+ 0x6F, 0xF0, 0x7E, 0x05, 0x09, 0xEB, 0x48, 0x0B, 0x69, 0x0C, 0x04, 0xE0, 0x29, 0xF8, 0x10, 0x10,
+ 0x0B, 0xF8, 0x00, 0x50, 0x40, 0x1C, 0x40, 0x45, 0xF8, 0xDB, 0x03, 0x20, 0xFF, 0xF7, 0x1E, 0xFC,
+ 0x1E, 0x48, 0xB8, 0x30, 0x00, 0x24, 0xD0, 0xF8, 0x08, 0xA0, 0x26, 0x46, 0x32, 0x46, 0x01, 0x20,
+ 0x00, 0x21, 0x14, 0xF0, 0xF8, 0xFA, 0xDD, 0xE9, 0x02, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43,
+ 0x1D, 0xD0, 0x3A, 0xF9, 0x14, 0x10, 0x39, 0xF9, 0x14, 0x00, 0x00, 0x29, 0x00, 0xDA, 0x49, 0x42,
+ 0x00, 0x28, 0x00, 0xDA, 0x40, 0x42, 0x81, 0x42, 0x0A, 0xDA, 0x32, 0x46, 0x00, 0x21, 0x04, 0x98,
+ 0x12, 0xF0, 0xC8, 0xFC, 0x0B, 0xF8, 0x04, 0x00, 0x3A, 0xF8, 0x14, 0x10, 0x29, 0xF8, 0x14, 0x10,
+ 0x2B, 0x46, 0x32, 0x46, 0x00, 0x21, 0x04, 0x98, 0x12, 0xF0, 0xC7, 0xFC, 0x64, 0x1C, 0x76, 0x1C,
+ 0xBE, 0x42, 0xD3, 0xDB, 0x6D, 0x1C, 0x7F, 0x2D, 0xC7, 0xDD, 0x00, 0x25, 0x2C, 0x46, 0x0B, 0xE0,
+ 0x20, 0x07, 0x10, 0x00, 0x24, 0x52, 0x01, 0x20, 0xFC, 0x4C, 0x01, 0x20, 0xF8, 0x08, 0x10, 0x00,
+ 0xF8, 0x5B, 0x01, 0x20, 0x84, 0x5A, 0x10, 0x00, 0x22, 0x46, 0x01, 0x20, 0x00, 0x21, 0x14, 0xF0,
+ 0xBA, 0xFA, 0xDD, 0xE9, 0x02, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x07, 0xD0, 0x1B, 0xF9,
+ 0x05, 0x30, 0x22, 0x46, 0x00, 0x21, 0x04, 0x98, 0x12, 0xF0, 0x9F, 0xFC, 0x6D, 0x1C, 0x64, 0x1C,
+ 0xBC, 0x42, 0xDC, 0xDB, 0x76, 0x48, 0x26, 0x22, 0x04, 0x99, 0x00, 0x68, 0x00, 0xF5, 0x86, 0x50,
+ 0x00, 0xE0, 0x01, 0xE0, 0x14, 0xF0, 0xD6, 0xFA, 0x00, 0x26, 0x72, 0x4D, 0x34, 0x46, 0x22, 0x46,
+ 0x01, 0x20, 0x00, 0x21, 0x14, 0xF0, 0x97, 0xFA, 0xDD, 0xE9, 0x02, 0x23, 0x10, 0x40, 0x19, 0x40,
+ 0x08, 0x43, 0x07, 0xD0, 0x22, 0x46, 0x00, 0x21, 0x04, 0x98, 0x12, 0xF0, 0x73, 0xFC, 0x25, 0xF8,
+ 0x16, 0x00, 0x76, 0x1C, 0x64, 0x1C, 0xBC, 0x42, 0xE9, 0xDB, 0x03, 0x20, 0xFF, 0xF7, 0x9E, 0xFB,
+ 0x65, 0x48, 0x81, 0x68, 0x00, 0x20, 0x06, 0xE0, 0x31, 0xF8, 0x10, 0x30, 0x08, 0xEB, 0x00, 0x02,
+ 0x40, 0x1C, 0x25, 0xF8, 0x12, 0x30, 0x40, 0x45, 0xF6, 0xDB, 0x02, 0x20, 0x12, 0xF0, 0xD6, 0xFA,
+ 0x00, 0x20, 0x12, 0xF0, 0xD3, 0xFA, 0x05, 0xB0, 0x63, 0xE4, 0x2D, 0xE9, 0xF0, 0x41, 0x05, 0x46,
+ 0x01, 0x24, 0x00, 0x20, 0x12, 0xF0, 0xAF, 0xFA, 0x02, 0x20, 0x12, 0xF0, 0xAC, 0xFA, 0x54, 0x4E,
+ 0xE8, 0x07, 0x4F, 0xF0, 0x00, 0x07, 0x4F, 0xF0, 0xA5, 0x08, 0x08, 0xD0, 0x00, 0xF0, 0x20, 0xF8,
+ 0x10, 0xF0, 0x01, 0x04, 0x03, 0xD0, 0x30, 0x68, 0xC7, 0x76, 0x80, 0xF8, 0x1A, 0x80, 0xA8, 0x07,
+ 0x08, 0xD5, 0x00, 0xF0, 0x63, 0xF8, 0x20, 0x40, 0x04, 0x00, 0x03, 0xD0, 0x30, 0x68, 0x47, 0x77,
+ 0x80, 0xF8, 0x1C, 0x80, 0x00, 0x20, 0x12, 0xF0, 0xA9, 0xFA, 0x02, 0x20, 0x12, 0xF0, 0xA6, 0xFA,
+ 0xE8, 0x07, 0x02, 0xD0, 0x01, 0x20, 0x00, 0xF0, 0xBB, 0xF8, 0x20, 0x46, 0xBD, 0xE8, 0xF0, 0x81,
+ 0x2D, 0xE9, 0xFC, 0x47, 0x01, 0x26, 0x11, 0xF0, 0xC8, 0xFD, 0x04, 0x46, 0x0D, 0x46, 0x11, 0xF0,
+ 0xC6, 0xFD, 0xDF, 0xF8, 0xF8, 0x90, 0x07, 0x46, 0x88, 0x46, 0xD9, 0xF8, 0x00, 0x00, 0x90, 0xF8,
+ 0xE7, 0x00, 0x10, 0xF0, 0x60, 0x0F, 0x05, 0xD0, 0x20, 0x46, 0x29, 0x46, 0x0F, 0xF0, 0x02, 0xF9,
+ 0x04, 0x46, 0x0D, 0x46, 0xD9, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x70, 0x01, 0xC0, 0x07, 0x27, 0xD0,
+ 0xCD, 0xE9, 0x00, 0x78, 0x22, 0x46, 0x2B, 0x46, 0x31, 0x48, 0x12, 0xF0, 0x15, 0xFC, 0xD9, 0xF8,
+ 0x00, 0x00, 0x90, 0xF8, 0x70, 0x11, 0xC9, 0x07, 0x1A, 0xD0, 0x90, 0xF8, 0xE8, 0x10, 0x89, 0x07,
+ 0x03, 0xD4, 0x00, 0x20, 0xFF, 0xF7, 0x31, 0xFD, 0x0C, 0xE0, 0x90, 0xF8, 0xE7, 0x00, 0xC0, 0xF3,
+ 0x41, 0x10, 0x02, 0x28, 0x4F, 0xF0, 0x00, 0x00, 0x02, 0xD3, 0xFF, 0xF7, 0xF4, 0xFB, 0x01, 0xE0,
+ 0xFF, 0xF7, 0x53, 0xFB, 0x06, 0x00, 0x03, 0xD1, 0x01, 0x21, 0x74, 0x20, 0x0D, 0xF0, 0xC4, 0xFF,
+ 0x00, 0x20, 0xFF, 0xF7, 0x0B, 0xFB, 0x30, 0x46, 0xBD, 0xE8, 0xFC, 0x87, 0x2D, 0xE9, 0xFC, 0x41,
+ 0x01, 0x24, 0x11, 0xF0, 0x7A, 0xFD, 0x05, 0x46, 0x0E, 0x46, 0x11, 0xF0, 0x78, 0xFD, 0x17, 0x4F,
+ 0x3A, 0x68, 0x92, 0xF8, 0x78, 0x21, 0xD2, 0x07, 0x1C, 0xD0, 0xCD, 0xE9, 0x00, 0x01, 0x2A, 0x46,
+ 0x33, 0x46, 0x14, 0x48, 0x12, 0xF0, 0xD8, 0xFB, 0x38, 0x68, 0x90, 0xF8, 0x78, 0x11, 0xC9, 0x07,
+ 0x10, 0xD0, 0x90, 0xF8, 0xE8, 0x00, 0x80, 0x07, 0x20, 0x46, 0x02, 0xD4, 0xFF, 0xF7, 0xF5, 0xFC,
+ 0x01, 0xE0, 0xFF, 0xF7, 0x22, 0xFB, 0x04, 0x00, 0x04, 0xD1, 0x4F, 0xF4, 0x80, 0x71, 0x74, 0x20,
+ 0x0D, 0xF0, 0x92, 0xFF, 0x01, 0x20, 0xFF, 0xF7, 0xD9, 0xFA, 0x20, 0x46, 0xBD, 0xE8, 0xFC, 0x81,
+ 0x24, 0x07, 0x10, 0x00, 0xE0, 0x4A, 0x01, 0x20, 0xB0, 0x09, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0xFC, 0x4C, 0x01, 0x20, 0x24, 0x52, 0x01, 0x20, 0x10, 0xB5, 0xFF, 0xF7, 0x8C, 0xFA, 0x01, 0x20,
+ 0xFF, 0xF7, 0x72, 0xFE, 0x01, 0x20, 0x00, 0xF0, 0xA4, 0xF9, 0x53, 0x20, 0xFF, 0xF7, 0x35, 0xFF,
+ 0x00, 0x28, 0x0B, 0xD0, 0x4F, 0xF4, 0xD6, 0x70, 0x00, 0xF0, 0xF7, 0xFF, 0x00, 0x28, 0x05, 0xD0,
+ 0xFE, 0xF7, 0xF1, 0xFE, 0xBD, 0xE8, 0x10, 0x40, 0xFE, 0xF7, 0x88, 0xBE, 0x10, 0xBD, 0x10, 0xB5,
+ 0xFF, 0xF7, 0x71, 0xFA, 0x01, 0x20, 0xFF, 0xF7, 0x57, 0xFE, 0x01, 0x20, 0x00, 0xF0, 0x89, 0xF9,
+ 0x40, 0xF2, 0xFF, 0x10, 0xFE, 0xF7, 0x02, 0xFE, 0xBD, 0xE8, 0x10, 0x40, 0xFE, 0xF7, 0x76, 0xBE,
+ 0x2D, 0xE9, 0xF0, 0x41, 0x12, 0x49, 0x09, 0x68, 0x91, 0xF8, 0xC1, 0x21, 0xD2, 0x07, 0x1D, 0xD0,
+ 0x01, 0x28, 0x1B, 0xD1, 0x10, 0x4F, 0x91, 0xF8, 0x31, 0x20, 0x91, 0xF8, 0x30, 0x10, 0x3D, 0x68,
+ 0x4A, 0x43, 0x02, 0x20, 0x0B, 0x4E, 0x14, 0xB2, 0x32, 0x35, 0xFE, 0xF7, 0x53, 0xF9, 0x03, 0x20,
+ 0x12, 0xF0, 0xB1, 0xF9, 0x38, 0x68, 0x62, 0x00, 0x31, 0x46, 0x04, 0x86, 0x28, 0x46, 0x14, 0xF0,
+ 0x99, 0xF9, 0xBD, 0xE8, 0xF0, 0x41, 0x03, 0x20, 0x12, 0xF0, 0xC0, 0xB9, 0xBD, 0xE8, 0xF0, 0x81,
+ 0x20, 0x07, 0x10, 0x00, 0x2E, 0x44, 0x01, 0x20, 0x24, 0x07, 0x10, 0x00, 0x10, 0xB5, 0x01, 0x46,
+ 0x34, 0x22, 0xFE, 0x48, 0x14, 0xF0, 0xCB, 0xF9, 0x01, 0x21, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0x20,
+ 0x12, 0xF0, 0xF1, 0xB8, 0x30, 0xB5, 0x85, 0xB0, 0x01, 0x24, 0x68, 0x46, 0x11, 0xF0, 0x36, 0xFC,
+ 0xF7, 0x4B, 0x02, 0x22, 0x03, 0x21, 0x18, 0x78, 0xE0, 0xB1, 0x01, 0x28, 0x0D, 0xD0, 0x03, 0x28,
+ 0x0E, 0xD0, 0x02, 0x28, 0x19, 0xD0, 0x04, 0x22, 0x05, 0x28, 0x0C, 0xD0, 0x04, 0x28, 0x0D, 0xD0,
+ 0x06, 0x28, 0x0F, 0xD0, 0x07, 0x28, 0x10, 0xD0, 0x11, 0xE0, 0x8D, 0xF8, 0x06, 0x10, 0x0E, 0xE0,
+ 0x8D, 0xF8, 0x08, 0x10, 0x0B, 0xE0, 0x8D, 0xF8, 0x0A, 0x20, 0x01, 0xE0, 0x8D, 0xF8, 0x0A, 0x10,
+ 0x00, 0x24, 0x04, 0xE0, 0x8D, 0xF8, 0x06, 0x20, 0x01, 0xE0, 0x8D, 0xF8, 0x08, 0x20, 0xE5, 0x48,
+ 0x00, 0x90, 0xE5, 0x48, 0x00, 0x68, 0x90, 0xF8, 0xB0, 0x00, 0xC0, 0x07, 0x21, 0xD0, 0x04, 0xB3,
+ 0xD8, 0x79, 0x10, 0xB1, 0x8D, 0xF8, 0x0B, 0x10, 0x02, 0xE0, 0x01, 0x20, 0x8D, 0xF8, 0x0B, 0x00,
+ 0xDA, 0x4D, 0x03, 0x24, 0x68, 0x46, 0x11, 0xF0, 0x1E, 0xFC, 0x00, 0x22, 0x04, 0xAB, 0x01, 0x21,
+ 0x10, 0x46, 0x12, 0xF0, 0xF1, 0xF8, 0x01, 0x21, 0x00, 0x20, 0x12, 0xF0, 0xD0, 0xF8, 0x28, 0x78,
+ 0x00, 0x28, 0x04, 0xD0, 0x0D, 0xF0, 0xD7, 0xFE, 0x64, 0x1E, 0x00, 0x2C, 0xEA, 0xDC, 0x05, 0xB0,
+ 0x30, 0xBD, 0x00, 0x20, 0xE2, 0xE7, 0x2D, 0xE9, 0xF3, 0x4F, 0xCD, 0x4D, 0x87, 0xB0, 0x06, 0x20,
+ 0x0E, 0x46, 0x28, 0x70, 0x11, 0xF0, 0x79, 0xFC, 0xCD, 0xE9, 0x04, 0x01, 0x36, 0x20, 0x02, 0x90,
+ 0xCA, 0x4C, 0x07, 0x98, 0x4F, 0xF0, 0x40, 0x0A, 0xC0, 0xB1, 0x01, 0x28, 0x21, 0xD0, 0x02, 0x28,
+ 0x7E, 0xD1, 0x07, 0x20, 0x28, 0x70, 0xC6, 0x48, 0xDF, 0xF8, 0x00, 0x83, 0x00, 0x90, 0x20, 0x68,
+ 0x08, 0xF1, 0x28, 0x08, 0x41, 0xF2, 0x42, 0x11, 0x08, 0x44, 0x01, 0x90, 0x08, 0x21, 0x04, 0xA8,
+ 0x12, 0xF0, 0x40, 0xFE, 0x04, 0x46, 0x01, 0x2E, 0x1E, 0xD0, 0x8A, 0xE0, 0xBC, 0x48, 0xDF, 0xF8,
+ 0xDC, 0x82, 0x48, 0x38, 0x00, 0x90, 0x08, 0xF1, 0x18, 0x08, 0x41, 0xF2, 0xE6, 0x01, 0x20, 0x68,
+ 0xEA, 0xE7, 0x11, 0xF0, 0x4C, 0xFC, 0xCD, 0xE9, 0x04, 0x01, 0xB5, 0x48, 0x41, 0xF2, 0x1C, 0x11,
+ 0x28, 0x38, 0x00, 0x90, 0x20, 0x68, 0xDF, 0xF8, 0xB4, 0x82, 0x08, 0x44, 0x01, 0x90, 0x26, 0x20,
+ 0x08, 0xF1, 0x1C, 0x08, 0x02, 0x90, 0xD9, 0xE7, 0x02, 0x46, 0x04, 0xA9, 0x00, 0x98, 0x12, 0xF0,
+ 0xE3, 0xFA, 0xDF, 0xF8, 0xB0, 0x92, 0x6F, 0xF0, 0x7E, 0x06, 0x09, 0xEB, 0x44, 0x00, 0x03, 0x90,
+ 0x00, 0x20, 0x71, 0x0C, 0x05, 0xE0, 0x00, 0xBF, 0x29, 0xF8, 0x10, 0x10, 0x03, 0x9A, 0x16, 0x54,
+ 0x40, 0x1C, 0xA0, 0x42, 0xF8, 0xDB, 0x00, 0xBF, 0xFF, 0xF7, 0x44, 0xFF, 0x00, 0x25, 0xD8, 0xF8,
+ 0x00, 0xB0, 0x2F, 0x46, 0x3A, 0x46, 0x01, 0x20, 0x00, 0x21, 0x14, 0xF0, 0x84, 0xF8, 0xDD, 0xE9,
+ 0x04, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x1B, 0xD0, 0x3B, 0xF9, 0x15, 0x10, 0x39, 0xF9,
+ 0x15, 0x00, 0x00, 0x29, 0x00, 0xDA, 0x49, 0x42, 0x00, 0x28, 0x00, 0xDA, 0x40, 0x42, 0x81, 0x42,
+ 0x09, 0xDA, 0x39, 0x46, 0x00, 0x98, 0x12, 0xF0, 0x9D, 0xFA, 0x03, 0x99, 0x48, 0x55, 0x3B, 0xF8,
+ 0x15, 0x10, 0x29, 0xF8, 0x15, 0x10, 0x32, 0x46, 0x39, 0x46, 0x00, 0x98, 0x12, 0xF0, 0x99, 0xFA,
+ 0x6D, 0x1C, 0xA5, 0x42, 0x02, 0xD0, 0x7F, 0x1C, 0x57, 0x45, 0xD3, 0xDB, 0x76, 0x1C, 0x00, 0xE0,
+ 0x54, 0xE0, 0x7F, 0x2E, 0xC8, 0xDD, 0x00, 0x26, 0x35, 0x46, 0x2A, 0x46, 0x01, 0x20, 0x00, 0x21,
+ 0x14, 0xF0, 0x51, 0xF8, 0xDD, 0xE9, 0x04, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x06, 0xD0,
+ 0x03, 0x98, 0x29, 0x46, 0x82, 0x57, 0x00, 0x98, 0x12, 0xF0, 0x7B, 0xFA, 0x76, 0x1C, 0xA6, 0x42,
+ 0x02, 0xD0, 0x6D, 0x1C, 0x55, 0x45, 0xE8, 0xDB, 0xDD, 0xE9, 0x00, 0x10, 0x02, 0x9A, 0x14, 0xF0,
+ 0x71, 0xF8, 0x07, 0x98, 0x30, 0xB1, 0x01, 0x28, 0x06, 0xD0, 0x02, 0x28, 0x06, 0xD0, 0x03, 0x28,
+ 0x05, 0xD1, 0x03, 0xE0, 0x74, 0x4E, 0x02, 0xE0, 0x74, 0x4E, 0x00, 0xE0, 0x74, 0x4E, 0x00, 0x27,
+ 0x3D, 0x46, 0x2A, 0x46, 0x01, 0x20, 0x00, 0x21, 0x14, 0xF0, 0x25, 0xF8, 0xDD, 0xE9, 0x04, 0x23,
+ 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x06, 0xD0, 0x29, 0x46, 0x00, 0x98, 0x12, 0xF0, 0x4A, 0xFA,
+ 0x26, 0xF8, 0x17, 0x00, 0x7F, 0x1C, 0xA7, 0x42, 0x02, 0xD0, 0x6D, 0x1C, 0x55, 0x45, 0xE8, 0xDB,
+ 0xFF, 0xF7, 0xC8, 0xFE, 0x00, 0x20, 0xD8, 0xF8, 0x00, 0x10, 0x05, 0xE0, 0x31, 0xF8, 0x10, 0x20,
+ 0x23, 0x18, 0x40, 0x1C, 0x26, 0xF8, 0x13, 0x20, 0xA0, 0x42, 0xF7, 0xDB, 0x09, 0xB0, 0xBD, 0xE8,
+ 0xF0, 0x8F, 0x10, 0xB5, 0x04, 0x46, 0x00, 0x20, 0x12, 0xF0, 0x45, 0xF8, 0x02, 0x20, 0x12, 0xF0,
+ 0x42, 0xF8, 0x21, 0x46, 0x00, 0x20, 0xFF, 0xF7, 0x06, 0xFF, 0x21, 0x46, 0x01, 0x20, 0xFF, 0xF7,
+ 0x02, 0xFF, 0x21, 0x46, 0x02, 0x20, 0xFF, 0xF7, 0xFE, 0xFE, 0x02, 0x20, 0x12, 0xF0, 0x4E, 0xF8,
+ 0xBD, 0xE8, 0x10, 0x40, 0x00, 0x20, 0x12, 0xF0, 0x49, 0xB8, 0x2D, 0xE9, 0xF0, 0x47, 0x48, 0x48,
+ 0x00, 0x25, 0x2C, 0x46, 0xD0, 0xE9, 0x14, 0x78, 0xD0, 0xF8, 0x38, 0x90, 0x2E, 0x46, 0x82, 0x46,
+ 0x32, 0x46, 0x01, 0x20, 0x00, 0x21, 0x13, 0xF0, 0xD6, 0xFF, 0x38, 0x40, 0x01, 0xEA, 0x08, 0x01,
+ 0x08, 0x43, 0x04, 0xD0, 0x19, 0xF8, 0x06, 0x00, 0xA8, 0x42, 0x00, 0xD9, 0x05, 0x46, 0x76, 0x1C,
+ 0x40, 0x2E, 0xED, 0xD3, 0xDA, 0xE9, 0x16, 0x78, 0xDA, 0xF8, 0x3C, 0x90, 0x00, 0x26, 0x32, 0x46,
+ 0x01, 0x20, 0x00, 0x21, 0x13, 0xF0, 0xBF, 0xFF, 0x38, 0x40, 0x01, 0xEA, 0x08, 0x01, 0x08, 0x43,
+ 0x04, 0xD0, 0x19, 0xF8, 0x06, 0x00, 0xA0, 0x42, 0x00, 0xD9, 0x04, 0x46, 0x76, 0x1C, 0x40, 0x2E,
+ 0xED, 0xD3, 0xDA, 0xF8, 0x48, 0x20, 0x50, 0x46, 0x15, 0x70, 0xDA, 0xF8, 0x4C, 0x10, 0x0C, 0x70,
+ 0xDA, 0xF8, 0x30, 0x30, 0xDA, 0xF8, 0x28, 0x70, 0xDA, 0xF8, 0x34, 0x60, 0x1B, 0x78, 0x3F, 0x78,
+ 0x2B, 0x44, 0x3B, 0x44, 0xDA, 0xF8, 0x2C, 0x70, 0x36, 0x78, 0x3F, 0x78, 0x26, 0x44, 0x3E, 0x44,
+ 0xD0, 0xE9, 0x14, 0x7C, 0x57, 0xEA, 0x0C, 0x07, 0x0D, 0xD0, 0xD0, 0xE9, 0x16, 0x7C, 0x57, 0xEA,
+ 0x0C, 0x07, 0x0A, 0xD0, 0xFF, 0x27, 0xB3, 0x42, 0x13, 0xD2, 0xF3, 0x1A, 0x2B, 0x44, 0xFF, 0x2B,
+ 0x07, 0xD8, 0x13, 0x70, 0x0B, 0xE0, 0x14, 0x70, 0x09, 0xE0, 0x15, 0x70, 0x0D, 0x70, 0xBD, 0xE8,
+ 0xF0, 0x87, 0x17, 0x70, 0xC2, 0x69, 0x42, 0xF4, 0x00, 0x72, 0xCA, 0xF8, 0x1C, 0x20, 0x0C, 0x70,
+ 0xF5, 0xE7, 0x9E, 0x42, 0x0D, 0xD2, 0x9B, 0x1B, 0x23, 0x44, 0xFF, 0x2B, 0x01, 0xD8, 0x0B, 0x70,
+ 0x05, 0xE0, 0x0F, 0x70, 0xC1, 0x69, 0x41, 0xF4, 0x00, 0x71, 0xCA, 0xF8, 0x1C, 0x10, 0x15, 0x70,
+ 0xE5, 0xE7, 0x15, 0x70, 0xEB, 0xE7, 0x2D, 0xE9, 0xF0, 0x41, 0x1F, 0x46, 0x0D, 0x46, 0x00, 0x2A,
+ 0x34, 0xDD, 0x07, 0x4C, 0x1F, 0xB3, 0xB4, 0xF9, 0x0C, 0x60, 0x11, 0x46, 0x0F, 0xF0, 0x01, 0xFC,
+ 0xB0, 0x42, 0x28, 0x78, 0x27, 0xDA, 0xFF, 0x28, 0x1C, 0xD2, 0x13, 0xE0, 0xFC, 0x09, 0x10, 0x00,
+ 0x68, 0x04, 0x10, 0x00, 0xAD, 0x24, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00, 0x24, 0x07, 0x10, 0x00,
+ 0x68, 0x5C, 0x01, 0x20, 0x84, 0x5A, 0x10, 0x00, 0x78, 0x4B, 0x01, 0x20, 0xCC, 0x4B, 0x01, 0x20,
+ 0x64, 0x4C, 0x01, 0x20, 0x40, 0x1C, 0x28, 0x70, 0x00, 0x20, 0xBD, 0xE8, 0xF0, 0x81, 0xB4, 0xF9,
+ 0x0A, 0x60, 0xDA, 0xE7, 0xE0, 0x69, 0x1F, 0xB1, 0x40, 0xF0, 0x02, 0x00, 0xE0, 0x61, 0x05, 0xE0,
+ 0x40, 0xF4, 0x00, 0x60, 0xFA, 0xE7, 0x08, 0xB1, 0x40, 0x1E, 0x28, 0x70, 0x01, 0x20, 0xEC, 0xE7,
+ 0x2D, 0xE9, 0xF0, 0x41, 0x1E, 0x46, 0x0D, 0x46, 0x00, 0x2A, 0x1A, 0xDD, 0xFE, 0x4C, 0x66, 0xB1,
+ 0xB4, 0xF9, 0x0C, 0x70, 0x11, 0x46, 0x0F, 0xF0, 0xC4, 0xFB, 0xB8, 0x42, 0x11, 0xDD, 0x28, 0x78,
+ 0x50, 0xB1, 0x10, 0x28, 0x04, 0xD3, 0x10, 0x38, 0x03, 0xE0, 0xB4, 0xF9, 0x0A, 0x70, 0xF1, 0xE7,
+ 0x00, 0x20, 0x28, 0x70, 0x00, 0x20, 0xD0, 0xE7, 0xE0, 0x69, 0x26, 0xB1, 0x40, 0xF0, 0x01, 0x00,
+ 0xE0, 0x61, 0x01, 0x20, 0xC9, 0xE7, 0x40, 0xF4, 0x80, 0x60, 0xF9, 0xE7, 0x2D, 0xE9, 0xF1, 0x4F,
+ 0x82, 0xB0, 0x00, 0x25, 0xEC, 0x48, 0x02, 0x9A, 0x2C, 0x46, 0xFF, 0x21, 0x02, 0xB3, 0xD0, 0xF8,
+ 0x30, 0x90, 0x02, 0x26, 0x4F, 0xF0, 0x18, 0x0B, 0x89, 0xF8, 0x00, 0x10, 0xD0, 0xF8, 0x34, 0x80,
+ 0x88, 0xF8, 0x00, 0x10, 0xB0, 0xF9, 0x0C, 0x70, 0x31, 0x02, 0x00, 0x91, 0x41, 0x69, 0x00, 0x29,
+ 0x00, 0xDC, 0x01, 0x25, 0xE0, 0x48, 0x80, 0x69, 0x00, 0x28, 0x00, 0xDC, 0x01, 0x24, 0xFF, 0xF7,
+ 0xFC, 0xFE, 0xFF, 0xF7, 0x97, 0xFD, 0x00, 0x98, 0x0D, 0xF0, 0x49, 0xFC, 0x85, 0xB1, 0x21, 0xE0,
+ 0xD0, 0xF8, 0x28, 0x90, 0x4F, 0xF4, 0x00, 0x66, 0x4F, 0xF0, 0x18, 0x0B, 0x89, 0xF8, 0x00, 0x10,
+ 0xD0, 0xF8, 0x2C, 0x80, 0x88, 0xF8, 0x00, 0x10, 0xB0, 0xF9, 0x0A, 0x70, 0xF1, 0x10, 0xDC, 0xE7,
+ 0xDF, 0xF8, 0x44, 0xA3, 0x3A, 0x46, 0xDA, 0xF8, 0x20, 0x00, 0xDA, 0xF8, 0x14, 0x10, 0x00, 0x68,
+ 0x00, 0xF0, 0x54, 0xFD, 0x01, 0x28, 0x05, 0xD1, 0xDA, 0xF8, 0x1C, 0x10, 0x31, 0x43, 0x01, 0x25,
+ 0xCA, 0xF8, 0x1C, 0x10, 0x8C, 0xB9, 0xDF, 0xF8, 0x20, 0xA3, 0x3A, 0x46, 0xDA, 0xF8, 0x24, 0x00,
+ 0xDA, 0xF8, 0x18, 0x10, 0x00, 0x68, 0x00, 0xF0, 0x41, 0xFD, 0x01, 0x28, 0x05, 0xD1, 0xDA, 0xF8,
+ 0x1C, 0x10, 0x31, 0x43, 0x01, 0x24, 0xCA, 0xF8, 0x1C, 0x10, 0x0D, 0xB1, 0x00, 0x2C, 0x4E, 0xD1,
+ 0x2F, 0x46, 0x26, 0x46, 0x4F, 0xF0, 0x17, 0x0A, 0x4F, 0xB9, 0xBB, 0x48, 0x02, 0x9B, 0x01, 0x6A,
+ 0x42, 0x69, 0x0F, 0x68, 0x49, 0x46, 0x38, 0x46, 0xFF, 0xF7, 0x6A, 0xFF, 0x07, 0x46, 0x4E, 0xB9,
+ 0xB5, 0x48, 0x02, 0x9B, 0x41, 0x6A, 0x82, 0x69, 0x0E, 0x68, 0x41, 0x46, 0x30, 0x46, 0xFF, 0xF7,
+ 0x5F, 0xFF, 0x06, 0x46, 0xFF, 0xF7, 0xA1, 0xFE, 0xFF, 0xF7, 0x3C, 0xFD, 0x00, 0x98, 0x0D, 0xF0,
+ 0xEE, 0xFB, 0x07, 0xB1, 0x16, 0xB9, 0xBA, 0xF1, 0x01, 0x0A, 0xDD, 0xD2, 0x2F, 0x46, 0xAA, 0x4D,
+ 0xAB, 0xF1, 0x01, 0x06, 0x3F, 0xB9, 0x28, 0x6A, 0x02, 0x9B, 0x49, 0x46, 0x00, 0x68, 0x6A, 0x69,
+ 0xFF, 0xF7, 0x09, 0xFF, 0x07, 0x46, 0x3C, 0xB9, 0x68, 0x6A, 0x02, 0x9B, 0x41, 0x46, 0x00, 0x68,
+ 0xAA, 0x69, 0xFF, 0xF7, 0x00, 0xFF, 0x04, 0x46, 0xFF, 0xF7, 0x7F, 0xFE, 0xFF, 0xF7, 0x1A, 0xFD,
+ 0x00, 0x98, 0x0D, 0xF0, 0xCC, 0xFB, 0x07, 0xB1, 0x0C, 0xB9, 0x76, 0x1E, 0xE2, 0xD2, 0x01, 0x2F,
+ 0x05, 0xD1, 0x01, 0x2C, 0x03, 0xD1, 0xE8, 0x69, 0x20, 0xF4, 0x00, 0x70, 0xE8, 0x61, 0xBD, 0xE8,
+ 0xFE, 0x8F, 0x2D, 0xE9, 0xF0, 0x4F, 0x9A, 0x46, 0x91, 0x46, 0x00, 0x22, 0x92, 0x4B, 0x83, 0x46,
+ 0x14, 0x46, 0xB3, 0xF9, 0x0E, 0x70, 0x10, 0x46, 0x01, 0x23, 0x90, 0x46, 0x15, 0xE0, 0x00, 0xBF,
+ 0x09, 0xEB, 0xE4, 0x05, 0x04, 0xF0, 0x07, 0x0C, 0x2E, 0x78, 0x03, 0xFA, 0x0C, 0xF5, 0x35, 0x42,
+ 0x0A, 0xD0, 0x3B, 0xF9, 0x12, 0x50, 0xBD, 0x42, 0x03, 0xDC, 0x0A, 0xF8, 0x02, 0x30, 0x01, 0x20,
+ 0x01, 0xE0, 0x0A, 0xF8, 0x02, 0x80, 0x52, 0x1C, 0x64, 0x1C, 0x8A, 0x42, 0xE8, 0xDB, 0x26, 0xE6,
+ 0x2D, 0xE9, 0xFF, 0x5F, 0x00, 0x24, 0x4F, 0xF0, 0x01, 0x08, 0x9A, 0x46, 0x17, 0x46, 0x25, 0x46,
+ 0xDF, 0xF8, 0xF4, 0x91, 0x46, 0x46, 0xDD, 0xF8, 0x3C, 0xB0, 0x4E, 0xE0, 0x0A, 0xEB, 0xE5, 0x00,
+ 0x05, 0xF0, 0x07, 0x01, 0x00, 0x78, 0x06, 0xFA, 0x01, 0xF2, 0x02, 0x42, 0x44, 0xD0, 0x1B, 0xF8,
+ 0x04, 0x00, 0x01, 0x28, 0x3F, 0xD0, 0x00, 0x98, 0xB9, 0xF9, 0x0E, 0x10, 0x30, 0xF9, 0x14, 0x20,
+ 0x8A, 0x42, 0x1D, 0xDA, 0x29, 0x46, 0x01, 0x98, 0x12, 0xF0, 0x61, 0xF8, 0xFF, 0x28, 0x0E, 0xDA,
+ 0x40, 0x1C, 0xC2, 0xB2, 0x4F, 0xF0, 0x00, 0x08, 0x29, 0x46, 0x01, 0x98, 0x12, 0xF0, 0x59, 0xF8,
+ 0x00, 0x98, 0x30, 0xF8, 0x14, 0x10, 0x0E, 0x98, 0x20, 0xF8, 0x14, 0x10, 0x23, 0xE0, 0x0B, 0xF8,
+ 0x04, 0x60, 0xD9, 0xF8, 0x1C, 0x10, 0x41, 0xF0, 0x08, 0x01, 0xC9, 0xF8, 0x1C, 0x10, 0x1A, 0xE0,
+ 0x0E, 0x98, 0x30, 0xF9, 0x14, 0x00, 0x00, 0x28, 0x13, 0xDD, 0x40, 0x1A, 0xA2, 0xEB, 0x01, 0x01,
+ 0x00, 0xD5, 0x40, 0x42, 0x00, 0x29, 0x00, 0xDA, 0x49, 0x42, 0x88, 0x42, 0x09, 0xDC, 0x29, 0x46,
+ 0x01, 0x98, 0x12, 0xF0, 0x34, 0xF8, 0x40, 0x1E, 0xC2, 0xB2, 0x29, 0x46, 0x01, 0x98, 0x12, 0xF0,
+ 0x30, 0xF8, 0x0B, 0xF8, 0x04, 0x60, 0x64, 0x1C, 0x6D, 0x1C, 0xBC, 0x42, 0xAE, 0xDB, 0x40, 0x46,
+ 0x04, 0xB0, 0xBD, 0xE8, 0xF0, 0x9F, 0x2D, 0xE9, 0xFF, 0x5F, 0x00, 0x24, 0x4F, 0xF0, 0x01, 0x08,
+ 0x9A, 0x46, 0x17, 0x46, 0x25, 0x46, 0x46, 0x46, 0xDF, 0xF8, 0x2C, 0x91, 0xDD, 0xF8, 0x38, 0xB0,
+ 0x31, 0xE0, 0x00, 0xBF, 0x0A, 0xEB, 0xE5, 0x00, 0x05, 0xF0, 0x07, 0x02, 0x01, 0x78, 0x06, 0xFA,
+ 0x02, 0xF0, 0x08, 0x42, 0x26, 0xD0, 0x1B, 0xF8, 0x04, 0x00, 0x01, 0x28, 0x21, 0xD0, 0x00, 0x98,
+ 0x30, 0xF9, 0x14, 0x10, 0xB9, 0xF9, 0x0E, 0x00, 0x81, 0x42, 0x1A, 0xDD, 0x29, 0x46, 0x01, 0x98,
+ 0x11, 0xF0, 0xFD, 0xFF, 0x00, 0x28, 0x0C, 0xDD, 0x4F, 0xF0, 0x00, 0x08, 0x04, 0x28, 0x01, 0xDB,
+ 0x00, 0x1F, 0x00, 0xE0, 0x00, 0x20, 0xC2, 0xB2, 0x29, 0x46, 0x01, 0x98, 0x11, 0xF0, 0xF1, 0xFF,
+ 0x07, 0xE0, 0x0B, 0xF8, 0x04, 0x60, 0xD9, 0xF8, 0x1C, 0x10, 0x41, 0xF0, 0x04, 0x01, 0xC9, 0xF8,
+ 0x1C, 0x10, 0x64, 0x1C, 0x6D, 0x1C, 0xBC, 0x42, 0xCC, 0xDB, 0x40, 0x46, 0xB8, 0xE7, 0x2D, 0xE9,
+ 0xF0, 0x4F, 0x93, 0xB0, 0x00, 0x24, 0x25, 0x46, 0x18, 0x21, 0x0C, 0xA8, 0x13, 0xF0, 0x53, 0xFE,
+ 0x28, 0x21, 0x02, 0xA8, 0x13, 0xF0, 0x4F, 0xFE, 0x27, 0x48, 0x41, 0x69, 0x00, 0x29, 0x00, 0xDC,
+ 0x01, 0x24, 0x82, 0x69, 0x00, 0x2A, 0x00, 0xDC, 0x01, 0x25, 0x23, 0x4A, 0x81, 0x46, 0x06, 0x46,
+ 0xFF, 0x23, 0x50, 0x32, 0x80, 0x6B, 0x00, 0xF0, 0x03, 0xFC, 0x1F, 0x4A, 0xFF, 0x23, 0x58, 0x32,
+ 0xB1, 0x69, 0xF0, 0x6B, 0x00, 0xF0, 0xFC, 0xFB, 0xFF, 0xF7, 0x77, 0xFD, 0xFF, 0xF7, 0x12, 0xFC,
+ 0x4F, 0xF4, 0x80, 0x6A, 0x50, 0x46, 0x0D, 0xF0, 0xC2, 0xFA, 0x8C, 0xB1, 0xD9, 0xF8, 0x20, 0x00,
+ 0x15, 0x4A, 0x4E, 0x46, 0x00, 0x68, 0x0C, 0xAB, 0x50, 0x32, 0xD9, 0xF8, 0x14, 0x10, 0xFF, 0xF7,
+ 0xF8, 0xFE, 0x01, 0x28, 0x04, 0xD1, 0xF0, 0x69, 0x01, 0x24, 0x40, 0xF0, 0x08, 0x00, 0xF0, 0x61,
+ 0x8D, 0xB1, 0xD9, 0xF8, 0x24, 0x00, 0x0C, 0x4A, 0x4E, 0x46, 0x00, 0x68, 0x02, 0xAB, 0x58, 0x32,
+ 0xD9, 0xF8, 0x18, 0x10, 0xFF, 0xF7, 0xE5, 0xFE, 0x01, 0x28, 0x04, 0xD1, 0xF0, 0x69, 0x01, 0x25,
+ 0x40, 0xF0, 0x08, 0x00, 0xF0, 0x61, 0x0C, 0xB1, 0x00, 0x2D, 0x79, 0xD1, 0x26, 0x46, 0xA8, 0x46,
+ 0x00, 0x27, 0x7E, 0xB9, 0x0C, 0xA8, 0x01, 0xE0, 0x68, 0x04, 0x10, 0x00, 0x00, 0x90, 0xD9, 0xF8,
+ 0x20, 0x00, 0xFD, 0x4B, 0xD9, 0xF8, 0x14, 0x20, 0x00, 0x68, 0xD9, 0xF8, 0x38, 0x10, 0xFF, 0xF7,
+ 0x52, 0xFF, 0x06, 0x46, 0xB8, 0xF1, 0x00, 0x0F, 0x0D, 0xD1, 0x02, 0xA8, 0x00, 0x90, 0xD9, 0xF8,
+ 0x24, 0x10, 0xF5, 0x4B, 0xD9, 0xF8, 0x18, 0x20, 0x08, 0x68, 0x08, 0x33, 0xD9, 0xF8, 0x3C, 0x10,
+ 0xFF, 0xF7, 0x41, 0xFF, 0x80, 0x46, 0xFF, 0xF7, 0x20, 0xFD, 0xFF, 0xF7, 0xBB, 0xFB, 0x50, 0x46,
+ 0x0D, 0xF0, 0x6D, 0xFA, 0x16, 0xB1, 0xB8, 0xF1, 0x00, 0x0F, 0x02, 0xD1, 0x7F, 0x1C, 0x60, 0x2F,
+ 0xCF, 0xDB, 0xDF, 0xF8, 0xA8, 0xB3, 0xD9, 0xF8, 0x14, 0x00, 0x26, 0x46, 0x41, 0x00, 0x0B, 0xF5,
+ 0xE6, 0x68, 0x4C, 0x46, 0x58, 0x46, 0x13, 0xF0, 0xA4, 0xFD, 0xA0, 0x69, 0x41, 0x00, 0x40, 0x46,
+ 0x13, 0xF0, 0x9F, 0xFD, 0x00, 0x27, 0xD1, 0x46, 0xFF, 0xF7, 0x9C, 0xFB, 0x48, 0x46, 0x0D, 0xF0,
+ 0x4E, 0xFA, 0x56, 0xB9, 0x0C, 0xA8, 0xCD, 0xE9, 0x00, 0xB0, 0x20, 0x6A, 0xDA, 0x4B, 0x62, 0x69,
+ 0x00, 0x68, 0xA1, 0x6B, 0xFF, 0xF7, 0xAC, 0xFE, 0x06, 0x46, 0x5D, 0xB9, 0x02, 0xA9, 0xCD, 0xE9,
+ 0x00, 0x81, 0x60, 0x6A, 0xD4, 0x4B, 0xA2, 0x69, 0x00, 0x68, 0x08, 0x33, 0xE1, 0x6B, 0xFF, 0xF7,
+ 0x9F, 0xFE, 0x05, 0x46, 0xFF, 0xF7, 0xE1, 0xFC, 0xFF, 0xF7, 0x7C, 0xFB, 0x48, 0x46, 0x0D, 0xF0,
+ 0x2E, 0xFA, 0x06, 0xB1, 0x15, 0xB9, 0x7F, 0x1C, 0x60, 0x2F, 0xD5, 0xDB, 0x01, 0x2E, 0x07, 0xD1,
+ 0x01, 0x2D, 0x05, 0xD1, 0xE0, 0x69, 0x81, 0x05, 0x02, 0xD5, 0x20, 0xF4, 0x00, 0x70, 0xE0, 0x61,
+ 0x13, 0xB0, 0xAC, 0xE4, 0x2D, 0xE9, 0xF0, 0x41, 0x0F, 0x46, 0xC3, 0x49, 0x16, 0x46, 0x02, 0x46,
+ 0x50, 0x39, 0xDD, 0xE9, 0x06, 0x05, 0xB1, 0xF9, 0x10, 0xC0, 0x01, 0x24, 0x62, 0x45, 0x11, 0xDD,
+ 0x1D, 0x68, 0xA0, 0xF1, 0x7F, 0x02, 0x95, 0x42, 0x07, 0xDD, 0x2A, 0x1A, 0x31, 0x46, 0x38, 0x46,
+ 0x1A, 0x60, 0x11, 0xF0, 0xCE, 0xFE, 0x00, 0x24, 0x11, 0xE0, 0xC8, 0x69, 0x40, 0xF0, 0x40, 0x00,
+ 0xC8, 0x61, 0x0C, 0xE0, 0x01, 0x21, 0x29, 0x70, 0x19, 0x68, 0xC0, 0xF1, 0x7F, 0x02, 0x91, 0x42,
+ 0x05, 0xDA, 0x0A, 0x18, 0x31, 0x46, 0x38, 0x46, 0x1A, 0x60, 0x11, 0xF0, 0xBA, 0xFE, 0x20, 0x46,
+ 0x43, 0xE5, 0x2D, 0xE9, 0xFF, 0x4F, 0x00, 0x24, 0x85, 0xB0, 0x15, 0x46, 0x06, 0x46, 0xA0, 0x46,
+ 0xA3, 0x46, 0x8B, 0xE0, 0x08, 0x98, 0x08, 0xF0, 0x07, 0x01, 0x00, 0xEB, 0xE8, 0x00, 0x7F, 0x22,
+ 0x03, 0x78, 0x01, 0x20, 0x88, 0x40, 0x18, 0x42, 0x03, 0x92, 0x7D, 0xD0, 0x41, 0x46, 0x06, 0x98,
+ 0x11, 0xF0, 0x9F, 0xFE, 0x00, 0x20, 0xDF, 0xF8, 0x80, 0xA2, 0x8D, 0xF8, 0x08, 0x00, 0x81, 0x46,
+ 0xAA, 0xF1, 0x50, 0x0A, 0xFF, 0xF7, 0x16, 0xFB, 0xDA, 0xF8, 0x14, 0x00, 0x85, 0x42, 0x03, 0xD1,
+ 0x4F, 0xF4, 0x00, 0x60, 0x0D, 0xF0, 0xC3, 0xF9, 0x02, 0xAF, 0x10, 0x20, 0xCD, 0xE9, 0x00, 0x07,
+ 0x36, 0xF9, 0x14, 0x00, 0x03, 0xAB, 0x42, 0x46, 0x06, 0x99, 0xFF, 0xF7, 0x9B, 0xFF, 0x20, 0xB9,
+ 0x09, 0xF1, 0x01, 0x09, 0xB9, 0xF1, 0x18, 0x0F, 0xE4, 0xDB, 0x9D, 0xF8, 0x08, 0x00, 0xE8, 0xB3,
+ 0x00, 0x20, 0x8D, 0xF8, 0x08, 0x00, 0x81, 0x46, 0xFF, 0xF7, 0xF4, 0xFA, 0xDA, 0xF8, 0x14, 0x00,
+ 0x85, 0x42, 0x03, 0xD1, 0x4F, 0xF4, 0x00, 0x60, 0x0D, 0xF0, 0xA1, 0xF9, 0x04, 0x20, 0xCD, 0xE9,
+ 0x00, 0x07, 0x36, 0xF9, 0x14, 0x00, 0x03, 0xAB, 0x42, 0x46, 0x06, 0x99, 0xFF, 0xF7, 0x7A, 0xFF,
+ 0x20, 0xB9, 0x09, 0xF1, 0x01, 0x09, 0xB9, 0xF1, 0x18, 0x0F, 0xE5, 0xDB, 0x9D, 0xF8, 0x08, 0x00,
+ 0xE0, 0xB1, 0x00, 0x20, 0x8D, 0xF8, 0x08, 0x00, 0x81, 0x46, 0x4F, 0xF0, 0x01, 0x0A, 0x00, 0xBF,
+ 0xFF, 0xF7, 0xD0, 0xFA, 0x78, 0x48, 0x50, 0x38, 0x40, 0x69, 0x85, 0x42, 0x03, 0xD1, 0x4F, 0xF4,
+ 0x00, 0x60, 0x0D, 0xF0, 0x7C, 0xF9, 0xCD, 0xE9, 0x00, 0xA7, 0x36, 0xF9, 0x14, 0x00, 0x03, 0xAB,
+ 0x42, 0x46, 0x06, 0x99, 0xFF, 0xF7, 0x56, 0xFF, 0x80, 0xB1, 0x01, 0xE0, 0x0D, 0xE0, 0x13, 0xE0,
+ 0x9D, 0xF8, 0x08, 0x00, 0x01, 0x28, 0x0E, 0xD0, 0x03, 0x9A, 0x41, 0x46, 0x52, 0x1E, 0x03, 0x92,
+ 0x06, 0x98, 0x11, 0xF0, 0x2E, 0xFE, 0x0B, 0xF1, 0x01, 0x0B, 0x04, 0xE0, 0x09, 0xF1, 0x01, 0x09,
+ 0xB9, 0xF1, 0x18, 0x0F, 0xD4, 0xDB, 0x64, 0x1C, 0x08, 0xF1, 0x01, 0x08, 0xAC, 0x42, 0xFF, 0xF6,
+ 0x71, 0xAF, 0xAB, 0x45, 0x02, 0xD0, 0x00, 0x20, 0x09, 0xB0, 0xB8, 0xE5, 0x01, 0x20, 0xFB, 0xE7,
+ 0x2D, 0xE9, 0xFF, 0x5F, 0xDF, 0xF8, 0x70, 0x91, 0x4F, 0xF0, 0x01, 0x08, 0x00, 0x25, 0x9A, 0x46,
+ 0x17, 0x46, 0x2C, 0x46, 0xA9, 0xF1, 0x50, 0x09, 0x46, 0x46, 0xDD, 0xF8, 0x3C, 0xB0, 0x39, 0xE0,
+ 0x0A, 0xEB, 0xE4, 0x00, 0x04, 0xF0, 0x07, 0x02, 0x01, 0x78, 0x06, 0xFA, 0x02, 0xF0, 0x08, 0x42,
+ 0x2F, 0xD0, 0x1B, 0xF8, 0x04, 0x00, 0x58, 0xBB, 0x21, 0x46, 0x01, 0x98, 0x11, 0xF0, 0xF2, 0xFD,
+ 0x00, 0x99, 0xB9, 0xF9, 0x10, 0x20, 0x31, 0xF9, 0x15, 0x30, 0x49, 0x46, 0x93, 0x42, 0x12, 0xDD,
+ 0x0E, 0x9A, 0x7F, 0x3A, 0x90, 0x42, 0x08, 0xDD, 0x0E, 0x99, 0x42, 0x1A, 0x21, 0x46, 0x01, 0x98,
+ 0x11, 0xF0, 0xE7, 0xFD, 0x4F, 0xF0, 0x00, 0x08, 0x12, 0xE0, 0xC8, 0x69, 0x40, 0xF0, 0x40, 0x00,
+ 0xC9, 0xF8, 0x1C, 0x00, 0x0C, 0xE0, 0x0B, 0xF8, 0x04, 0x60, 0x0E, 0x99, 0xC1, 0xF1, 0x7F, 0x01,
+ 0x81, 0x42, 0x05, 0xDD, 0x0E, 0x99, 0x42, 0x18, 0x21, 0x46, 0x01, 0x98, 0x11, 0xF0, 0xD1, 0xFD,
+ 0x6D, 0x1C, 0x64, 0x1C, 0xBD, 0x42, 0xC3, 0xDB, 0x40, 0x46, 0xC9, 0xE5, 0x2D, 0xE9, 0xF0, 0x4F,
+ 0x35, 0x4C, 0x9D, 0xB0, 0x50, 0x3C, 0x61, 0x69, 0xB1, 0xB3, 0x00, 0x20, 0x83, 0x46, 0xA0, 0x69,
+ 0x98, 0xB3, 0x00, 0x20, 0x82, 0x46, 0xBB, 0xF1, 0x00, 0x0F, 0x02, 0xD0, 0xBA, 0xF1, 0x00, 0x0F,
+ 0x58, 0xD1, 0x2F, 0x4B, 0x04, 0xAD, 0x0D, 0xCB, 0x85, 0xE8, 0x0D, 0x00, 0x7F, 0x23, 0x2A, 0x4A,
+ 0x20, 0x6C, 0x00, 0xF0, 0x15, 0xFA, 0x28, 0x4A, 0x7F, 0x23, 0x08, 0x32, 0xA1, 0x69, 0x60, 0x6C,
+ 0x00, 0xF0, 0x0E, 0xFA, 0x00, 0x26, 0xA1, 0x46, 0x04, 0xA8, 0x5C, 0x46, 0x50, 0xF8, 0x26, 0x80,
+ 0x55, 0x46, 0x18, 0x21, 0x11, 0xA8, 0x13, 0xF0, 0x3E, 0xFC, 0x28, 0x21, 0x07, 0xA8, 0x13, 0xF0,
+ 0x3A, 0xFC, 0xD9, 0xF8, 0x1C, 0x10, 0x00, 0x27, 0x21, 0xF0, 0x40, 0x01, 0xC9, 0xF8, 0x1C, 0x10,
+ 0xFF, 0xF7, 0x10, 0xFA, 0x34, 0xB1, 0x13, 0xE0, 0x00, 0xE0, 0x01, 0xE0, 0x01, 0x20, 0xC5, 0xE7,
+ 0x01, 0x20, 0xC7, 0xE7, 0x11, 0xA9, 0xCD, 0xE9, 0x00, 0x81, 0xD9, 0xF8, 0x20, 0x10, 0x12, 0x4B,
+ 0xD9, 0xF8, 0x14, 0x20, 0x08, 0x68, 0xD9, 0xF8, 0x40, 0x10, 0xFF, 0xF7, 0x61, 0xFF, 0x04, 0x46,
+ 0x75, 0xB9, 0x07, 0xA9, 0xCD, 0xE9, 0x00, 0x81, 0xD9, 0xF8, 0x24, 0x10, 0x0A, 0x4B, 0xD9, 0xF8,
+ 0x18, 0x20, 0x08, 0x68, 0x08, 0x33, 0xD9, 0xF8, 0x44, 0x10, 0xFF, 0xF7, 0x51, 0xFF, 0x05, 0x46,
+ 0xFF, 0xF7, 0x4B, 0xFB, 0x04, 0xB1, 0x15, 0xB9, 0x7F, 0x1C, 0x18, 0x2F, 0xD0, 0xDB, 0x76, 0x1C,
+ 0x03, 0x2E, 0xB9, 0xD3, 0x1D, 0xB0, 0xFA, 0xE4, 0xB8, 0x04, 0x10, 0x00, 0x84, 0x5A, 0x10, 0x00,
+ 0x0C, 0x6B, 0x01, 0x00, 0x70, 0xB5, 0xFF, 0x4C, 0x00, 0x26, 0x05, 0x46, 0xE6, 0x61, 0x01, 0x28,
+ 0x5E, 0xD0, 0x02, 0x2D, 0x5E, 0xD0, 0x01, 0x20, 0x20, 0x70, 0x60, 0x79, 0x68, 0xB1, 0xF9, 0x4A,
+ 0x00, 0x23, 0x50, 0x32, 0x61, 0x69, 0xA0, 0x6B, 0x00, 0xF0, 0xAA, 0xF9, 0xF5, 0x4A, 0x00, 0x23,
+ 0x58, 0x32, 0xA1, 0x69, 0xE0, 0x6B, 0x00, 0xF0, 0xA3, 0xF9, 0xA0, 0x79, 0x58, 0xB1, 0xF1, 0x49,
+ 0x62, 0x69, 0x50, 0x31, 0x20, 0x6C, 0x11, 0xF0, 0x3F, 0xFD, 0xEE, 0x49, 0xA2, 0x69, 0x58, 0x31,
+ 0x60, 0x6C, 0x11, 0xF0, 0x39, 0xFD, 0xE0, 0x78, 0x10, 0xB1, 0x00, 0x20, 0xFF, 0xF7, 0xEE, 0xFB,
+ 0x20, 0x79, 0x60, 0xB1, 0xE0, 0x78, 0x38, 0xB1, 0x61, 0x78, 0xA0, 0x6A, 0x00, 0xF0, 0x76, 0xF9,
+ 0x61, 0x78, 0xE0, 0x6A, 0x00, 0xF0, 0x72, 0xF9, 0x01, 0x20, 0xFF, 0xF7, 0xDF, 0xFB, 0x60, 0x79,
+ 0x08, 0xB1, 0xFF, 0xF7, 0x5C, 0xFD, 0x01, 0x2D, 0x26, 0xD0, 0x02, 0x2D, 0x26, 0xD0, 0x26, 0x70,
+ 0xFF, 0xF7, 0x90, 0xF9, 0xA0, 0x79, 0x68, 0xB1, 0x20, 0x79, 0x38, 0xB1, 0xA1, 0x78, 0x20, 0x6B,
+ 0x00, 0xF0, 0x5C, 0xF9, 0xA1, 0x78, 0x60, 0x6B, 0x00, 0xF0, 0x58, 0xF9, 0x20, 0x7A, 0xC0, 0xB1,
+ 0xFF, 0xF7, 0x34, 0xFF, 0xFF, 0xF7, 0x7E, 0xF9, 0x01, 0x2D, 0x27, 0xD0, 0x02, 0x2D, 0x27, 0xD0,
+ 0x10, 0x21, 0xE0, 0x69, 0x00, 0xF0, 0x72, 0xF9, 0xE0, 0x69, 0x20, 0xB3, 0x00, 0x20, 0x70, 0xBD,
+ 0x03, 0x20, 0xA1, 0xE7, 0x05, 0x20, 0x9F, 0xE7, 0x02, 0x20, 0x00, 0xE0, 0x04, 0x20, 0x20, 0x70,
+ 0xD6, 0xE7, 0xA2, 0x69, 0x00, 0x2A, 0x06, 0xDD, 0x60, 0x6A, 0xC6, 0x4B, 0x61, 0x6C, 0x00, 0x68,
+ 0x58, 0x33, 0xFF, 0xF7, 0x26, 0xFE, 0x62, 0x69, 0x00, 0x2A, 0xDB, 0xDD, 0x20, 0x6A, 0xC1, 0x4B,
+ 0x21, 0x6C, 0x00, 0x68, 0x50, 0x33, 0xFF, 0xF7, 0x1C, 0xFE, 0xD3, 0xE7, 0x40, 0x21, 0xD8, 0xE7,
+ 0x4F, 0xF4, 0x00, 0x71, 0xD5, 0xE7, 0x01, 0x20, 0x70, 0xBD, 0x70, 0xB5, 0xB9, 0x4D, 0x0C, 0x46,
+ 0x02, 0x28, 0x28, 0xD0, 0x10, 0xF0, 0x21, 0xFE, 0xC5, 0xE9, 0x14, 0x01, 0x10, 0xF0, 0x1F, 0xFE,
+ 0xC5, 0xE9, 0x16, 0x01, 0xB3, 0x48, 0x08, 0x21, 0x50, 0x30, 0x11, 0xF0, 0xFB, 0xFF, 0x68, 0x61,
+ 0xB0, 0x48, 0x08, 0x21, 0x58, 0x30, 0x11, 0xF0, 0xF5, 0xFF, 0xA8, 0x61, 0x04, 0xF1, 0x90, 0x00,
+ 0xA8, 0x62, 0x40, 0x1C, 0xE8, 0x62, 0x40, 0x1C, 0x28, 0x63, 0x40, 0x1C, 0x68, 0x63, 0x4B, 0x38,
+ 0xA8, 0x63, 0x20, 0x30, 0xC5, 0xE9, 0x0F, 0x04, 0x48, 0x38, 0x68, 0x64, 0x74, 0x30, 0x95, 0x34,
+ 0xC5, 0xE9, 0x12, 0x04, 0x70, 0xBD, 0x10, 0xF0, 0xFC, 0xFD, 0xC5, 0xE9, 0x14, 0x01, 0x10, 0xF0,
+ 0xFA, 0xFD, 0xD5, 0xE7, 0x10, 0xB5, 0xA0, 0x49, 0x00, 0x20, 0xFF, 0xF7, 0xC6, 0xFF, 0x9D, 0x48,
+ 0x9E, 0x49, 0x01, 0x62, 0x09, 0x1D, 0x41, 0x62, 0x9D, 0x49, 0x0B, 0x68, 0x93, 0xF8, 0x88, 0x21,
+ 0x12, 0xF0, 0x01, 0x04, 0xC4, 0x70, 0x4F, 0xF0, 0x00, 0x02, 0x04, 0xD0, 0x83, 0x6A, 0x1A, 0x70,
+ 0xC3, 0x6A, 0x1A, 0x70, 0x08, 0xE0, 0x84, 0x6A, 0x93, 0xF8, 0x92, 0x31, 0x23, 0x70, 0x0B, 0x68,
+ 0xC4, 0x6A, 0x93, 0xF8, 0x93, 0x31, 0x23, 0x70, 0x0C, 0x68, 0x94, 0xF8, 0x88, 0x31, 0xC3, 0xF3,
+ 0x40, 0x03, 0x03, 0x71, 0x13, 0xB1, 0x03, 0x6B, 0x1A, 0x70, 0x06, 0xE0, 0x03, 0x6B, 0x94, 0xF8,
+ 0x94, 0x21, 0x1A, 0x70, 0x0A, 0x68, 0x92, 0xF8, 0x95, 0x21, 0x43, 0x6B, 0x1A, 0x70, 0x09, 0x68,
+ 0x91, 0xF8, 0x88, 0x21, 0xC2, 0xF3, 0x80, 0x03, 0x43, 0x71, 0xC2, 0xF3, 0xC0, 0x03, 0x83, 0x71,
+ 0xB1, 0xF8, 0x8A, 0x31, 0x43, 0x81, 0xB1, 0xF8, 0x8C, 0x31, 0x83, 0x81, 0xB1, 0xF8, 0x8E, 0x31,
+ 0xC3, 0x81, 0xB1, 0xF8, 0x90, 0x31, 0x03, 0x82, 0x91, 0xF8, 0x96, 0x31, 0x43, 0x70, 0x91, 0xF8,
+ 0x97, 0x31, 0x83, 0x70, 0xC2, 0xF3, 0x00, 0x12, 0xC2, 0x71, 0x91, 0xF8, 0x34, 0x11, 0xC1, 0xF3,
+ 0x40, 0x01, 0x01, 0x72, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0x20, 0xE3, 0xE6, 0x72, 0x49, 0x10, 0xB5,
+ 0x98, 0x31, 0x01, 0x20, 0xFF, 0xF7, 0x69, 0xFF, 0x70, 0x49, 0x6E, 0x48, 0x10, 0x31, 0x01, 0x62,
+ 0x00, 0x21, 0xC0, 0xE9, 0x16, 0x11, 0x0A, 0x46, 0x81, 0x61, 0x6D, 0x49, 0x0B, 0x68, 0x93, 0xF8,
+ 0xA0, 0x41, 0x14, 0xF0, 0x01, 0x04, 0xC4, 0x70, 0x04, 0xD0, 0x83, 0x6A, 0x1A, 0x70, 0xC3, 0x6A,
+ 0x1A, 0x70, 0x08, 0xE0, 0x84, 0x6A, 0x93, 0xF8, 0xAA, 0x31, 0x23, 0x70, 0x0B, 0x68, 0xC4, 0x6A,
+ 0x93, 0xF8, 0xAB, 0x31, 0x23, 0x70, 0x0B, 0x68, 0x93, 0xF8, 0xA0, 0x41, 0xC4, 0xF3, 0x40, 0x04,
+ 0x04, 0x71, 0x14, 0xB1, 0x03, 0x6B, 0x1A, 0x70, 0x06, 0xE0, 0x93, 0xF8, 0xAC, 0x21, 0x03, 0x6B,
+ 0x1A, 0x70, 0x0A, 0x68, 0x92, 0xF8, 0xAD, 0x21, 0x43, 0x6B, 0x1A, 0x70, 0x09, 0x68, 0x91, 0xF8,
+ 0xA0, 0x21, 0xC2, 0xF3, 0x80, 0x03, 0x43, 0x71, 0xC2, 0xF3, 0xC0, 0x03, 0x83, 0x71, 0xB1, 0xF8,
+ 0xA2, 0x31, 0x43, 0x81, 0xB1, 0xF8, 0xA4, 0x31, 0x83, 0x81, 0xB1, 0xF8, 0xA6, 0x31, 0xC3, 0x81,
+ 0xB1, 0xF8, 0xA8, 0x31, 0x03, 0x82, 0x91, 0xF8, 0xAE, 0x31, 0x43, 0x70, 0x91, 0xF8, 0xAF, 0x31,
+ 0x83, 0x70, 0xC2, 0xF3, 0x00, 0x12, 0xC2, 0x71, 0x91, 0xF8, 0x84, 0x19, 0xC1, 0xF3, 0x40, 0x01,
+ 0x01, 0x72, 0xBD, 0xE8, 0x10, 0x40, 0x01, 0x20, 0x84, 0xE6, 0x2D, 0xE9, 0xF0, 0x41, 0x07, 0x46,
+ 0x01, 0x24, 0x00, 0x20, 0x11, 0xF0, 0xE7, 0xF9, 0x02, 0x20, 0x11, 0xF0, 0xE4, 0xF9, 0x41, 0x4E,
+ 0x78, 0x07, 0x4F, 0xF0, 0x00, 0x05, 0x4F, 0xF0, 0xA5, 0x08, 0x0A, 0xD5, 0xFF, 0xF7, 0x32, 0xFF,
+ 0x10, 0xF0, 0x01, 0x04, 0x05, 0xD0, 0x30, 0x68, 0xC5, 0x77, 0x80, 0xF8, 0x20, 0x50, 0x80, 0xF8,
+ 0x1E, 0x80, 0x38, 0x07, 0x0A, 0xD5, 0xFF, 0xF7, 0x81, 0xFF, 0x20, 0x40, 0x04, 0x00, 0x05, 0xD0,
+ 0x30, 0x68, 0x00, 0xF8, 0x22, 0x5F, 0x45, 0x70, 0x00, 0xF8, 0x01, 0x8C, 0x02, 0x20, 0x11, 0xF0,
+ 0xDD, 0xF9, 0x00, 0x20, 0x11, 0xF0, 0xDA, 0xF9, 0x20, 0x46, 0xF1, 0xE4, 0x02, 0x78, 0x8A, 0x42,
+ 0x01, 0xD9, 0x51, 0x1A, 0x00, 0xE0, 0x00, 0x21, 0x01, 0x70, 0x70, 0x47, 0x10, 0xB5, 0x14, 0x46,
+ 0x0E, 0xF0, 0x0F, 0xFE, 0xA0, 0x42, 0x01, 0xDC, 0x01, 0x20, 0x10, 0xBD, 0x00, 0x20, 0x10, 0xBD,
+ 0xF0, 0xB5, 0x00, 0x25, 0x2C, 0x46, 0x01, 0x26, 0x0D, 0xE0, 0x00, 0xBF, 0x02, 0xEB, 0xE4, 0x07,
+ 0x04, 0xF0, 0x07, 0x0C, 0x3F, 0x78, 0x06, 0xFA, 0x0C, 0xFE, 0x1E, 0xEA, 0x07, 0x0F, 0x01, 0xD0,
+ 0x03, 0x55, 0x6D, 0x1C, 0x64, 0x1C, 0x8D, 0x42, 0xF0, 0xDB, 0xF0, 0xBD, 0x70, 0xB5, 0x04, 0x46,
+ 0x0D, 0x46, 0xC0, 0x07, 0x02, 0xD0, 0x83, 0x20, 0x0C, 0xF0, 0x16, 0xFF, 0xA0, 0x07, 0x03, 0xD5,
+ 0x29, 0x46, 0x84, 0x20, 0x0C, 0xF0, 0x10, 0xFF, 0x60, 0x07, 0x03, 0xD5, 0x29, 0x46, 0x85, 0x20,
+ 0x0C, 0xF0, 0x0A, 0xFF, 0x20, 0x07, 0x03, 0xD5, 0x29, 0x46, 0x86, 0x20, 0x0C, 0xF0, 0x04, 0xFF,
+ 0x60, 0x06, 0x03, 0xD5, 0x29, 0x46, 0x8B, 0x20, 0x0C, 0xF0, 0xFE, 0xFE, 0x20, 0x06, 0x03, 0xD5,
+ 0x29, 0x46, 0x8C, 0x20, 0x0C, 0xF0, 0xF8, 0xFE, 0xE0, 0x05, 0x0F, 0xD5, 0x29, 0x46, 0x87, 0x20,
+ 0x0A, 0xE0, 0x00, 0x00, 0x68, 0x04, 0x10, 0x00, 0xEC, 0x57, 0x01, 0x20, 0x14, 0x0A, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0x24, 0x07, 0x10, 0x00, 0x0C, 0xF0, 0xE6, 0xFE, 0xA0, 0x05, 0x05, 0xD5,
+ 0x29, 0x46, 0xBD, 0xE8, 0x70, 0x40, 0x88, 0x20, 0x0C, 0xF0, 0xDE, 0xBE, 0x70, 0xBD, 0x00, 0x00,
+ 0x5C, 0x4E, 0x08, 0xB5, 0x5C, 0x4D, 0x00, 0x24, 0xDF, 0xF8, 0x70, 0x91, 0x34, 0x60, 0xDF, 0xF8,
+ 0x70, 0x81, 0x2C, 0x60, 0xDF, 0xF8, 0x6C, 0xB1, 0xC9, 0xF8, 0x00, 0x40, 0xDF, 0xF8, 0x68, 0xA1,
+ 0xC8, 0xF8, 0x00, 0x40, 0xCB, 0xF8, 0x00, 0x40, 0xCA, 0xF8, 0x00, 0x40, 0x00, 0x22, 0x6B, 0x46,
+ 0x56, 0x49, 0x10, 0x46, 0x11, 0xF0, 0xE0, 0xF8, 0x00, 0x98, 0x81, 0x07, 0x13, 0xD5, 0xFC, 0xF7,
+ 0x15, 0xFF, 0x2F, 0x68, 0x2C, 0x60, 0xFC, 0xF7, 0x0F, 0xFF, 0x17, 0xF0, 0x53, 0x0F, 0x02, 0xD0,
+ 0x38, 0x46, 0xFE, 0xF7, 0x72, 0xFE, 0x17, 0xF4, 0xD6, 0x7F, 0x02, 0xD0, 0x38, 0x46, 0xFF, 0xF7,
+ 0x34, 0xFF, 0x02, 0x21, 0x82, 0xE0, 0xC1, 0x05, 0x0B, 0xD5, 0xFC, 0xF7, 0xFF, 0xFE, 0x37, 0x68,
+ 0x34, 0x60, 0xFC, 0xF7, 0xF9, 0xFE, 0x38, 0x46, 0xFD, 0xF7, 0xB4, 0xF8, 0x4F, 0xF4, 0x80, 0x71,
+ 0x74, 0xE0, 0x81, 0x05, 0x04, 0xD5, 0xFE, 0xF7, 0xA6, 0xF9, 0x4F, 0xF4, 0x00, 0x71, 0x6D, 0xE0,
+ 0x41, 0x05, 0x16, 0xD5, 0xFC, 0xF7, 0xEA, 0xFE, 0xD8, 0xF8, 0x00, 0x70, 0xC8, 0xF8, 0x00, 0x40,
+ 0xFC, 0xF7, 0xE2, 0xFE, 0xF8, 0x07, 0x01, 0xD0, 0xFD, 0xF7, 0x94, 0xFD, 0xB8, 0x07, 0x01, 0xD5,
+ 0xFD, 0xF7, 0x09, 0xFE, 0x78, 0x07, 0x01, 0xD5, 0xFD, 0xF7, 0xA0, 0xFD, 0x4F, 0xF4, 0x80, 0x61,
+ 0x54, 0xE0, 0x01, 0x06, 0x0C, 0xD5, 0xFC, 0xF7, 0xD1, 0xFE, 0xDB, 0xF8, 0x00, 0x70, 0xCB, 0xF8,
+ 0x00, 0x40, 0xFC, 0xF7, 0xC9, 0xFE, 0x38, 0x46, 0xFE, 0xF7, 0x11, 0xF8, 0x80, 0x21, 0x45, 0xE0,
+ 0x81, 0x02, 0x0D, 0xD5, 0xFC, 0xF7, 0xC2, 0xFE, 0xD9, 0xF8, 0x00, 0x70, 0xC9, 0xF8, 0x00, 0x40,
+ 0xFC, 0xF7, 0xBA, 0xFE, 0x38, 0x46, 0xFD, 0xF7, 0x09, 0xFD, 0x4F, 0xF4, 0x00, 0x11, 0x35, 0xE0,
+ 0xC1, 0x04, 0x04, 0xD5, 0xFE, 0xF7, 0xD8, 0xFE, 0x4F, 0xF4, 0x80, 0x51, 0x2E, 0xE0, 0x01, 0x02,
+ 0x04, 0xD5, 0xFD, 0xF7, 0x4F, 0xFE, 0x4F, 0xF4, 0x00, 0x01, 0x27, 0xE0, 0x01, 0x05, 0x04, 0xD5,
+ 0xFE, 0xF7, 0xE5, 0xFE, 0x4F, 0xF4, 0x00, 0x61, 0x20, 0xE0, 0x40, 0x02, 0x7F, 0xF5, 0x7E, 0xAF,
+ 0xFC, 0xF7, 0x9C, 0xFE, 0xDA, 0xF8, 0x00, 0x70, 0xCA, 0xF8, 0x00, 0x40, 0xFC, 0xF7, 0x94, 0xFE,
+ 0xF8, 0x07, 0x01, 0xD0, 0xFE, 0xF7, 0x47, 0xF9, 0xB8, 0x07, 0x05, 0xD5, 0x01, 0x20, 0xFE, 0xF7,
+ 0x2B, 0xFD, 0x01, 0x20, 0xFF, 0xF7, 0x5D, 0xF8, 0x78, 0x07, 0x05, 0xD5, 0x00, 0x20, 0xFE, 0xF7,
+ 0x23, 0xFD, 0x00, 0x20, 0xFF, 0xF7, 0x55, 0xF8, 0x4F, 0xF4, 0x80, 0x01, 0x00, 0x20, 0x11, 0xF0,
+ 0x26, 0xF8, 0x5B, 0xE7, 0x28, 0x07, 0x10, 0x00, 0x2C, 0x07, 0x10, 0x00, 0x30, 0x07, 0x10, 0x00,
+ 0x34, 0x07, 0x10, 0x00, 0x38, 0x07, 0x10, 0x00, 0x3C, 0x07, 0x10, 0x00, 0x83, 0x9F, 0xF0, 0x00,
+ 0x20, 0x48, 0x10, 0xB5, 0x01, 0x68, 0x20, 0x48, 0x02, 0x68, 0x13, 0x46, 0x92, 0xEA, 0x01, 0x0F,
+ 0x28, 0xD0, 0x1E, 0x4A, 0xCC, 0x07, 0x12, 0x68, 0x25, 0xD0, 0x92, 0xF8, 0x21, 0x43, 0xE4, 0x07,
+ 0x02, 0xD0, 0x43, 0xF0, 0x01, 0x03, 0x03, 0x60, 0x8B, 0x07, 0x1F, 0xD5, 0x92, 0xF8, 0x40, 0x33,
+ 0xDB, 0x07, 0x03, 0xD0, 0x03, 0x68, 0x43, 0xF0, 0x02, 0x03, 0x03, 0x60, 0x0B, 0x07, 0x19, 0xD5,
+ 0x92, 0xF8, 0x60, 0x33, 0xDB, 0x07, 0x03, 0xD0, 0x03, 0x68, 0x43, 0xF0, 0x08, 0x03, 0x03, 0x60,
+ 0x49, 0x06, 0x13, 0xD5, 0x92, 0xF8, 0x60, 0x13, 0xC9, 0x07, 0x03, 0xD0, 0x01, 0x68, 0x41, 0xF0,
+ 0x40, 0x01, 0x01, 0x60, 0x10, 0xBD, 0x23, 0xF0, 0x01, 0x03, 0xDC, 0xE7, 0x03, 0x68, 0x23, 0xF0,
+ 0x02, 0x03, 0xE2, 0xE7, 0x03, 0x68, 0x23, 0xF0, 0x08, 0x03, 0xE8, 0xE7, 0x01, 0x68, 0x21, 0xF0,
+ 0x40, 0x01, 0xEE, 0xE7, 0x48, 0x07, 0x10, 0x00, 0xF0, 0x04, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x2D, 0xE9, 0xF0, 0x47, 0xDF, 0xF8, 0x10, 0x92, 0x84, 0x4C, 0x00, 0x25, 0xD9, 0xF8, 0x00, 0x10,
+ 0x91, 0xF8, 0x31, 0x00, 0x84, 0xF8, 0x30, 0x00, 0x91, 0xF8, 0x30, 0x20, 0x80, 0x49, 0x84, 0xF8,
+ 0x31, 0x20, 0x50, 0x43, 0x09, 0x78, 0x84, 0xF8, 0x33, 0x10, 0x7E, 0x49, 0x21, 0x60, 0x34, 0x39,
+ 0x06, 0xB2, 0x8B, 0x7B, 0x84, 0xF8, 0x2D, 0x30, 0x88, 0x46, 0x77, 0x00, 0x0F, 0xE0, 0x2A, 0x46,
+ 0x31, 0x46, 0xD8, 0xF8, 0x08, 0x00, 0x0D, 0xF0, 0xD6, 0xFD, 0x01, 0x46, 0x22, 0x68, 0x06, 0xFB,
+ 0x05, 0xF0, 0x02, 0xEB, 0x40, 0x00, 0x3A, 0x46, 0x13, 0xF0, 0x04, 0xF8, 0x6D, 0x1C, 0x94, 0xF8,
+ 0x2D, 0x00, 0xA8, 0x42, 0xEB, 0xDC, 0x70, 0x48, 0x70, 0x49, 0x00, 0x78, 0x09, 0x78, 0x08, 0x43,
+ 0x40, 0x07, 0x11, 0xD5, 0x6E, 0x48, 0x60, 0x60, 0xD9, 0xF8, 0x00, 0x10, 0x91, 0xF8, 0xF1, 0x11,
+ 0x89, 0x07, 0x4A, 0x0F, 0x94, 0xF8, 0x31, 0x10, 0x92, 0x1C, 0x4A, 0x43, 0xD8, 0xF8, 0x10, 0x10,
+ 0xBD, 0xE8, 0xF0, 0x47, 0x12, 0xF0, 0xE6, 0xBF, 0xBD, 0xE8, 0xF0, 0x87, 0x2D, 0xE9, 0xF0, 0x41,
+ 0x61, 0x48, 0x62, 0x4F, 0x00, 0x26, 0x05, 0x68, 0x38, 0x68, 0x05, 0x43, 0x01, 0x20, 0x10, 0xF0,
+ 0xEA, 0xFF, 0x38, 0x21, 0x59, 0x48, 0x13, 0xF0, 0x76, 0xF8, 0xE8, 0x07, 0x06, 0xD0, 0xFF, 0xF7,
+ 0x9F, 0xFF, 0xE8, 0x06, 0x01, 0xD5, 0x00, 0xF0, 0x77, 0xF8, 0x01, 0x26, 0x59, 0x48, 0x53, 0x4C,
+ 0x00, 0x68, 0xC4, 0xE9, 0x07, 0x05, 0x38, 0x68, 0x60, 0x62, 0x52, 0x48, 0x56, 0x4F, 0x34, 0x38,
+ 0x41, 0x68, 0xA1, 0x62, 0xC1, 0x7B, 0x84, 0xF8, 0x35, 0x10, 0x90, 0xF8, 0x30, 0x00, 0x84, 0xF8,
+ 0x36, 0x00, 0x38, 0x78, 0x84, 0xF8, 0x2C, 0x00, 0xE8, 0x07, 0x1A, 0xD0, 0xE8, 0x06, 0x18, 0xD5,
+ 0x45, 0x4D, 0x28, 0x68, 0x90, 0xF8, 0x50, 0x02, 0xC0, 0x07, 0x06, 0xD0, 0x00, 0xF0, 0x75, 0xFD,
+ 0x18, 0xB9, 0xE0, 0x69, 0x40, 0xF0, 0x10, 0x00, 0xE0, 0x61, 0x28, 0x68, 0x90, 0xF8, 0x80, 0x02,
+ 0xC0, 0x07, 0x06, 0xD0, 0x00, 0xF0, 0x69, 0xFD, 0x18, 0xB9, 0xE0, 0x69, 0x40, 0xF0, 0x20, 0x00,
+ 0xE0, 0x61, 0x01, 0x20, 0x10, 0xF0, 0xC2, 0xFF, 0x00, 0x2E, 0x0C, 0xD0, 0x38, 0x78, 0x05, 0x28,
+ 0x07, 0xD0, 0x06, 0x28, 0x05, 0xD0, 0x01, 0x21, 0xBD, 0xE8, 0xF0, 0x41, 0x02, 0x20, 0x10, 0xF0,
+ 0xFA, 0xBE, 0x02, 0x21, 0xF8, 0xE7, 0xBD, 0xE8, 0xF0, 0x81, 0x38, 0x48, 0x10, 0xB5, 0x00, 0x78,
+ 0x00, 0x28, 0x28, 0xD1, 0x2F, 0x4C, 0x34, 0x3C, 0xD4, 0xE9, 0x06, 0x01, 0x00, 0xF0, 0x48, 0xFD,
+ 0x94, 0xF8, 0x31, 0x10, 0xA0, 0x6A, 0x00, 0xF0, 0x40, 0xFC, 0xFF, 0xF7, 0x8F, 0xFF, 0x26, 0x48,
+ 0x00, 0x68, 0x90, 0xF8, 0x60, 0x03, 0xC1, 0x07, 0x15, 0xD0, 0x81, 0x07, 0x13, 0xD5, 0x00, 0x24,
+ 0x40, 0x09, 0x13, 0xF0, 0x8B, 0xF8, 0x00, 0xB1, 0x01, 0x24, 0x29, 0x48, 0x01, 0x78, 0x8C, 0x42,
+ 0x09, 0xD0, 0x04, 0x70, 0x0C, 0xB1, 0x08, 0x21, 0x00, 0xE0, 0x10, 0x21, 0xBD, 0xE8, 0x10, 0x40,
+ 0x03, 0x20, 0x10, 0xF0, 0xC8, 0xBE, 0x10, 0xBD, 0x2D, 0xE9, 0xF0, 0x41, 0x16, 0x48, 0x19, 0x4E,
+ 0x16, 0x4C, 0x00, 0x68, 0x34, 0x3E, 0x90, 0xF8, 0x31, 0x50, 0x90, 0xF8, 0x30, 0x70, 0xB1, 0x69,
+ 0x39, 0xB1, 0x17, 0x48, 0x84, 0xF8, 0x30, 0x50, 0xC2, 0x38, 0x6A, 0x00, 0x20, 0x61, 0x12, 0xF0,
+ 0x41, 0xFF, 0xF1, 0x69, 0x39, 0xB1, 0x12, 0x48, 0x84, 0xF8, 0x31, 0x70, 0x98, 0x38, 0x7A, 0x00,
+ 0x60, 0x61, 0x12, 0xF0, 0x37, 0xFF, 0x0C, 0x48, 0x0C, 0x49, 0x00, 0x78, 0x09, 0x78, 0x08, 0x43,
+ 0x80, 0x06, 0xA8, 0xD5, 0x0A, 0x48, 0x4C, 0x38, 0xE0, 0x60, 0x6A, 0x00, 0x71, 0x6A, 0xBD, 0xE8,
+ 0xF0, 0x41, 0x12, 0xF0, 0x27, 0xBF, 0x9E, 0xE7, 0x20, 0x07, 0x10, 0x00, 0x7C, 0x6C, 0x10, 0x00,
+ 0xFC, 0x04, 0x10, 0x00, 0x64, 0x0A, 0x10, 0x00, 0xF4, 0x04, 0x10, 0x00, 0xF8, 0x04, 0x10, 0x00,
+ 0xD6, 0x07, 0x01, 0x20, 0xF0, 0x04, 0x10, 0x00, 0xE8, 0x04, 0x10, 0x00, 0xE9, 0x04, 0x10, 0x00,
+ 0xC8, 0x04, 0x10, 0x00, 0x12, 0x48, 0x10, 0xB5, 0x12, 0x49, 0x04, 0x78, 0x00, 0x20, 0x08, 0x60,
+ 0x00, 0xF0, 0x47, 0xF8, 0x00, 0xF0, 0xEC, 0xFA, 0x05, 0x2C, 0x11, 0xD0, 0x06, 0x2C, 0x0F, 0xD0,
+ 0x0D, 0x48, 0x01, 0x68, 0x21, 0xF0, 0x02, 0x01, 0x41, 0xF0, 0x04, 0x01, 0x01, 0x60, 0x01, 0x21,
+ 0x05, 0x20, 0x0C, 0xF0, 0x6F, 0xFF, 0x4F, 0xF4, 0x80, 0x71, 0x00, 0x20, 0x10, 0xF0, 0x63, 0xFE,
+ 0x00, 0xF0, 0x7D, 0xFD, 0x02, 0x21, 0xBD, 0xE8, 0x10, 0x40, 0x01, 0x20, 0x10, 0xF0, 0x5B, 0xBE,
+ 0xE8, 0x04, 0x10, 0x00, 0xEC, 0x04, 0x10, 0x00, 0x28, 0x07, 0x10, 0x00, 0x10, 0xB5, 0xC0, 0x07,
+ 0x1E, 0xD0, 0x38, 0x48, 0x03, 0x24, 0x00, 0x68, 0x90, 0xF8, 0x50, 0x12, 0xC9, 0x07, 0x03, 0xD1,
+ 0x90, 0xF8, 0x80, 0x02, 0xC0, 0x07, 0x00, 0xD0, 0x0B, 0x24, 0xFC, 0xF7, 0xC7, 0xFC, 0x32, 0x48,
+ 0x01, 0x68, 0x21, 0x43, 0x01, 0x60, 0xFC, 0xF7, 0xBF, 0xFC, 0x01, 0x21, 0x05, 0x20, 0x0C, 0xF0,
+ 0x41, 0xFF, 0xBD, 0xE8, 0x10, 0x40, 0x4F, 0xF4, 0x80, 0x71, 0x00, 0x20, 0x10, 0xF0, 0x33, 0xBE,
+ 0x10, 0xBD, 0x10, 0xB5, 0x00, 0xF0, 0x4B, 0xFD, 0x02, 0x21, 0x01, 0x20, 0x10, 0xF0, 0x57, 0xFE,
+ 0x01, 0x20, 0x10, 0xF0, 0xC8, 0xFE, 0x38, 0x21, 0x24, 0x48, 0x12, 0xF0, 0x54, 0xFF, 0x23, 0x49,
+ 0x01, 0x20, 0x81, 0xF8, 0x2C, 0x00, 0x10, 0xF0, 0xD9, 0xFE, 0x01, 0x21, 0x02, 0x20, 0x10, 0xF0,
+ 0x1A, 0xFE, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0xF0, 0x84, 0xBA, 0x70, 0xB5, 0x1C, 0x48, 0x1D, 0x4E,
+ 0x05, 0x68, 0x30, 0x68, 0x90, 0xEA, 0x05, 0x04, 0x1F, 0xD0, 0x35, 0xB1, 0x40, 0xB1, 0x04, 0xEA,
+ 0x05, 0x00, 0xFF, 0xF7, 0xB3, 0xFF, 0x35, 0x60, 0x70, 0xBD, 0xFF, 0xF7, 0xD2, 0xFF, 0xFA, 0xE7,
+ 0x00, 0xF0, 0x1D, 0xFD, 0x02, 0x21, 0x01, 0x20, 0x10, 0xF0, 0x29, 0xFE, 0x00, 0xF0, 0xDB, 0xF8,
+ 0x00, 0xF0, 0x5F, 0xFA, 0x04, 0xEA, 0x05, 0x00, 0xFF, 0xF7, 0xA0, 0xFF, 0x00, 0xF0, 0x48, 0xFA,
+ 0x02, 0x21, 0x01, 0x20, 0x10, 0xF0, 0xEF, 0xFD, 0xE5, 0xE7, 0x00, 0x2D, 0xE4, 0xD1, 0x0A, 0x48,
+ 0x00, 0x78, 0x05, 0x28, 0x01, 0xD0, 0x06, 0x28, 0xDE, 0xD1, 0x35, 0x60, 0xBD, 0xE8, 0x70, 0x40,
+ 0xAF, 0xE7, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00, 0x28, 0x07, 0x10, 0x00, 0x7C, 0x6C, 0x10, 0x00,
+ 0x44, 0x07, 0x10, 0x00, 0xEC, 0x04, 0x10, 0x00, 0xE8, 0x04, 0x10, 0x00, 0x70, 0xB5, 0xFB, 0x49,
+ 0x00, 0x20, 0x03, 0x46, 0x09, 0x68, 0x91, 0xF8, 0x40, 0x20, 0xD2, 0x07, 0x33, 0xD0, 0xF8, 0x4A,
+ 0x15, 0x78, 0x02, 0x2D, 0x16, 0xD0, 0x04, 0x2D, 0x03, 0xD0, 0x05, 0x2D, 0x0F, 0xD0, 0x06, 0x2D,
+ 0x0D, 0xD0, 0x91, 0xF8, 0x41, 0x20, 0xF3, 0x49, 0x0C, 0x68, 0x64, 0x1C, 0x0C, 0x60, 0x94, 0x42,
+ 0x21, 0xDD, 0xF1, 0x4A, 0x00, 0x20, 0x14, 0x78, 0x64, 0xB1, 0x08, 0x60, 0x0C, 0xE0, 0x91, 0xF8,
+ 0x43, 0x20, 0xF0, 0xE7, 0x91, 0xF8, 0x42, 0x20, 0x91, 0xF8, 0x90, 0x10, 0xC9, 0x06, 0xEA, 0xD5,
+ 0x01, 0x23, 0xE8, 0xE7, 0x13, 0xB1, 0x01, 0x20, 0x10, 0x70, 0x00, 0xE0, 0x08, 0x60, 0x05, 0x2D,
+ 0x0A, 0xD0, 0x00, 0xF0, 0xB3, 0xF8, 0x04, 0x46, 0x00, 0xF0, 0xB6, 0xF8, 0x40, 0xEA, 0x04, 0x05,
+ 0x00, 0xF0, 0xC0, 0xF8, 0x28, 0x43, 0x70, 0xBD, 0x01, 0x20, 0x70, 0xBD, 0x10, 0xB5, 0x01, 0x46,
+ 0x34, 0x22, 0xDE, 0x48, 0x12, 0xF0, 0x63, 0xFE, 0x01, 0x21, 0xBD, 0xE8, 0x10, 0x40, 0x08, 0x46,
+ 0x10, 0xF0, 0x89, 0xBD, 0x2D, 0xE9, 0xFF, 0x47, 0x81, 0x46, 0xD5, 0x48, 0x01, 0x24, 0x00, 0x78,
+ 0x02, 0x28, 0x22, 0xD0, 0x03, 0x28, 0x1C, 0xD0, 0x04, 0x25, 0xD5, 0x4E, 0xDF, 0xF8, 0x54, 0x83,
+ 0x05, 0x28, 0x1E, 0xD0, 0x06, 0x28, 0x68, 0x46, 0x2F, 0xD0, 0x00, 0xF0, 0x11, 0xF9, 0xD2, 0x48,
+ 0x00, 0x90, 0xCA, 0x48, 0x00, 0x68, 0x90, 0xF8, 0xB0, 0x10, 0xC8, 0x07, 0x38, 0xD0, 0x89, 0x07,
+ 0x30, 0xD5, 0xCE, 0x49, 0x0A, 0x78, 0x52, 0xB3, 0x00, 0x20, 0x8D, 0xF8, 0x0B, 0x00, 0x08, 0x70,
+ 0x2E, 0xE0, 0x68, 0x46, 0x00, 0xF0, 0xCD, 0xF8, 0xE9, 0xE7, 0x68, 0x46, 0x00, 0xF0, 0x91, 0xF8,
+ 0xE5, 0xE7, 0x68, 0x46, 0x10, 0xF0, 0xA2, 0xF8, 0xFF, 0xF7, 0x80, 0xFF, 0x40, 0x21, 0xC2, 0x07,
+ 0x01, 0xD0, 0x8D, 0xF8, 0x04, 0x50, 0xC2, 0x06, 0x01, 0xD5, 0x8D, 0xF8, 0x06, 0x40, 0x8D, 0xF8,
+ 0x08, 0x40, 0x30, 0x60, 0xC8, 0xF8, 0x00, 0x10, 0xD1, 0xE7, 0x00, 0x27, 0x10, 0xF0, 0x8E, 0xF8,
+ 0x8D, 0xF8, 0x04, 0x50, 0x01, 0x20, 0x37, 0x60, 0xC8, 0xF8, 0x00, 0x00, 0xC7, 0xE7, 0xB9, 0xF1,
+ 0x08, 0x0F, 0x02, 0xD9, 0x8D, 0xF8, 0x0B, 0x40, 0x02, 0xE0, 0x02, 0x20, 0x8D, 0xF8, 0x0B, 0x00,
+ 0x68, 0x46, 0x10, 0xF0, 0xA0, 0xF8, 0xB2, 0x49, 0xB2, 0x4A, 0xC9, 0x78, 0x11, 0x70, 0x04, 0xB0,
+ 0xBD, 0xE8, 0xF0, 0x87, 0x10, 0xB5, 0x00, 0xF0, 0x42, 0xFC, 0x02, 0x21, 0xBD, 0xE8, 0x10, 0x40,
+ 0x01, 0x20, 0x10, 0xF0, 0x20, 0xBD, 0xA3, 0x49, 0x00, 0x20, 0x08, 0x60, 0x70, 0x47, 0xA1, 0x49,
+ 0x00, 0x20, 0x08, 0x60, 0x70, 0x47, 0x38, 0xB5, 0x00, 0x20, 0x10, 0xF0, 0xB4, 0xFD, 0x00, 0xF0,
+ 0x94, 0xF9, 0x9B, 0x48, 0x00, 0x78, 0x00, 0xF0, 0xDA, 0xF8, 0x04, 0x46, 0xAF, 0xF2, 0x3B, 0x01,
+ 0x00, 0xF0, 0x1D, 0xFC, 0x20, 0x46, 0xFF, 0xF7, 0x7D, 0xFF, 0x01, 0x28, 0x0E, 0xD1, 0x01, 0x21,
+ 0x6B, 0x46, 0x00, 0x22, 0x08, 0x46, 0x10, 0xF0, 0x47, 0xFD, 0x01, 0x21, 0x08, 0x46, 0x10, 0xF0,
+ 0x26, 0xFD, 0x92, 0x48, 0x00, 0x78, 0x30, 0xB1, 0x0C, 0xF0, 0x2D, 0xFB, 0xBD, 0xE8, 0x38, 0x40,
+ 0x00, 0x20, 0x10, 0xF0, 0xAB, 0xBD, 0xFF, 0xF7, 0xF8, 0xFD, 0xF7, 0xE7, 0x92, 0x48, 0x00, 0x78,
+ 0xC0, 0x07, 0x00, 0xD0, 0x01, 0x20, 0x70, 0x47, 0x8F, 0x48, 0x00, 0x78, 0xC0, 0x07, 0x07, 0xD0,
+ 0x82, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x50, 0x02, 0xC0, 0x07, 0x01, 0xD0, 0x10, 0x20, 0x70, 0x47,
+ 0x00, 0x20, 0x70, 0x47, 0x88, 0x48, 0x00, 0x78, 0xC1, 0x07, 0x7C, 0x48, 0x00, 0x68, 0x90, 0xF8,
+ 0x80, 0x02, 0x03, 0xD0, 0xC1, 0x07, 0x01, 0xD0, 0x10, 0x20, 0x70, 0x47, 0x00, 0xF0, 0x01, 0x00,
+ 0x70, 0x47, 0x2D, 0xE9, 0xF0, 0x41, 0x06, 0x46, 0x00, 0x24, 0x10, 0xF0, 0x0F, 0xF8, 0xFF, 0xF7,
+ 0xED, 0xFE, 0x05, 0x46, 0x7C, 0x48, 0x00, 0x78, 0xC0, 0x07, 0x00, 0xD0, 0x40, 0x24, 0x6D, 0xB1,
+ 0x6E, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x90, 0x00, 0x00, 0x07, 0x07, 0xD5, 0x00, 0xF0, 0xAE, 0xF8,
+ 0x40, 0xEA, 0x04, 0x07, 0x00, 0xF0, 0xB8, 0xF8, 0x40, 0xEA, 0x07, 0x04, 0x45, 0xEA, 0x04, 0x00,
+ 0xC0, 0x07, 0x02, 0xD0, 0x00, 0xF0, 0xBE, 0xF8, 0x30, 0x71, 0x61, 0x07, 0x4F, 0xF0, 0x01, 0x00,
+ 0x00, 0xD5, 0x70, 0x71, 0xE9, 0x06, 0x00, 0xD5, 0xB0, 0x71, 0x61, 0x06, 0x00, 0xD5, 0x30, 0x72,
+ 0xA1, 0x06, 0x00, 0xD5, 0xF0, 0x71, 0x62, 0x48, 0x05, 0x60, 0x62, 0x48, 0x04, 0x60, 0xBD, 0xE8,
+ 0xF0, 0x81, 0x2D, 0xE9, 0xF0, 0x41, 0x06, 0x46, 0x0F, 0xF0, 0xD8, 0xFF, 0xFF, 0xF7, 0xB6, 0xFE,
+ 0x07, 0x46, 0xFF, 0xF7, 0x9B, 0xFF, 0x04, 0x46, 0xFF, 0xF7, 0x9E, 0xFF, 0x04, 0x43, 0xFF, 0xF7,
+ 0xA9, 0xFF, 0x04, 0x43, 0x00, 0xF0, 0x7A, 0xF8, 0x04, 0x43, 0x00, 0xF0, 0x85, 0xF8, 0x40, 0xEA,
+ 0x04, 0x05, 0x47, 0xEA, 0x05, 0x04, 0xE0, 0x07, 0x02, 0xD0, 0x00, 0xF0, 0x8B, 0xF8, 0x30, 0x71,
+ 0x69, 0x07, 0x4F, 0xF0, 0x01, 0x00, 0x00, 0xD5, 0x70, 0x71, 0xE1, 0x06, 0x00, 0xD5, 0xB0, 0x71,
+ 0xA9, 0x06, 0x00, 0xD5, 0xF0, 0x71, 0x4A, 0x48, 0x07, 0x60, 0x4A, 0x48, 0x05, 0x60, 0xCE, 0xE7,
+ 0x2D, 0xE9, 0xF0, 0x41, 0x07, 0x46, 0x0F, 0xF0, 0xA9, 0xFF, 0xFF, 0xF7, 0x87, 0xFE, 0x05, 0x46,
+ 0xFF, 0xF7, 0x6C, 0xFF, 0x04, 0x46, 0xFF, 0xF7, 0x6F, 0xFF, 0x04, 0x43, 0xFF, 0xF7, 0x7A, 0xFF,
+ 0x04, 0x43, 0x00, 0xF0, 0x4B, 0xF8, 0x04, 0x43, 0x00, 0xF0, 0x56, 0xF8, 0x40, 0xEA, 0x04, 0x06,
+ 0x45, 0xEA, 0x06, 0x04, 0xE0, 0x07, 0x02, 0xD0, 0x00, 0xF0, 0x5C, 0xF8, 0x38, 0x71, 0x61, 0x07,
+ 0x4F, 0xF0, 0x01, 0x00, 0x00, 0xD5, 0x78, 0x71, 0xE1, 0x06, 0x00, 0xD5, 0xB8, 0x71, 0xB1, 0x06,
+ 0x00, 0xD5, 0xF8, 0x71, 0x32, 0x48, 0x05, 0x60, 0x32, 0x48, 0x06, 0x60, 0x9F, 0xE7, 0x38, 0xB5,
+ 0x2A, 0x49, 0x80, 0x1E, 0x05, 0x28, 0x09, 0x68, 0x13, 0xD2, 0xDF, 0xE8, 0x00, 0xF0, 0x03, 0x06,
+ 0x09, 0x0C, 0x0F, 0x00, 0x91, 0xF8, 0x73, 0x40, 0x0C, 0xE0, 0x91, 0xF8, 0x72, 0x40, 0x09, 0xE0,
+ 0x91, 0xF8, 0x71, 0x40, 0x06, 0xE0, 0x91, 0xF8, 0x74, 0x40, 0x03, 0xE0, 0x91, 0xF8, 0x75, 0x40,
+ 0x00, 0xE0, 0x01, 0x24, 0x00, 0x25, 0x0B, 0xF0, 0x6B, 0xFE, 0x00, 0xB1, 0x01, 0x25, 0x69, 0x46,
+ 0x03, 0x20, 0x10, 0xF0, 0x4A, 0xFC, 0x00, 0x98, 0x00, 0xB9, 0x1D, 0xB1, 0x14, 0x2C, 0x01, 0xD2,
+ 0x14, 0x24, 0x01, 0xE0, 0x04, 0xB9, 0x01, 0x24, 0x20, 0x46, 0x38, 0xBD, 0x1E, 0x48, 0x00, 0x78,
+ 0xC0, 0x07, 0x07, 0xD0, 0x11, 0x48, 0x00, 0x68, 0x90, 0xF8, 0xF0, 0x01, 0xC0, 0x07, 0x01, 0xD0,
+ 0x04, 0x20, 0x70, 0x47, 0x00, 0x20, 0x70, 0x47, 0x17, 0x48, 0x00, 0x78, 0xC0, 0x07, 0x07, 0xD0,
+ 0x0A, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x10, 0x02, 0xC0, 0x07, 0x01, 0xD0, 0x20, 0x20, 0x70, 0x47,
+ 0x00, 0x20, 0x70, 0x47, 0x05, 0x49, 0x01, 0x20, 0x09, 0x68, 0x91, 0xF8, 0xE7, 0x10, 0x09, 0x06,
+ 0x03, 0xD5, 0x0B, 0x48, 0x40, 0x78, 0xC8, 0xB1, 0x03, 0x20, 0x70, 0x47, 0x20, 0x07, 0x10, 0x00,
+ 0xE8, 0x04, 0x10, 0x00, 0xCC, 0x04, 0x10, 0x00, 0xE9, 0x04, 0x10, 0x00, 0x30, 0x0A, 0x10, 0x00,
+ 0xF8, 0x04, 0x10, 0x00, 0xF4, 0x04, 0x10, 0x00, 0x7D, 0x3B, 0x00, 0x00, 0xDB, 0x04, 0x10, 0x00,
+ 0x4C, 0x07, 0x10, 0x00, 0xFC, 0x04, 0x10, 0x00, 0xEC, 0x04, 0x10, 0x00, 0x02, 0x20, 0xE4, 0xE7,
+ 0x10, 0xB5, 0x00, 0xF0, 0x9C, 0xF9, 0x00, 0xF0, 0x12, 0xFA, 0x62, 0x49, 0x00, 0x20, 0x48, 0x60,
+ 0x61, 0x49, 0x02, 0x20, 0x08, 0x70, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0xF0, 0x90, 0xF9, 0x00, 0xF0,
+ 0x06, 0xFA, 0x5C, 0x49, 0x00, 0x20, 0x48, 0x60, 0x5B, 0x49, 0x03, 0x20, 0x08, 0x70, 0x10, 0xBD,
+ 0x10, 0xB5, 0x00, 0xF0, 0x84, 0xF9, 0x00, 0xF0, 0xFA, 0xF9, 0x56, 0x49, 0x00, 0x20, 0x48, 0x60,
+ 0x55, 0x49, 0x04, 0x20, 0x08, 0x70, 0x55, 0x49, 0x01, 0x20, 0x08, 0x70, 0x54, 0x49, 0x08, 0x70,
+ 0x10, 0xBD, 0x50, 0x49, 0x00, 0x20, 0x48, 0x60, 0x70, 0x47, 0x4E, 0x49, 0x00, 0x20, 0x48, 0x60,
+ 0x70, 0x47, 0x4C, 0x49, 0x00, 0x20, 0x48, 0x60, 0x4B, 0x49, 0x01, 0x20, 0x08, 0x70, 0x70, 0x47,
+ 0x10, 0xB5, 0x00, 0xF0, 0x61, 0xF9, 0x47, 0x49, 0x08, 0x70, 0x00, 0xF0, 0x60, 0xF9, 0x00, 0xF0,
+ 0xD6, 0xF9, 0x45, 0x49, 0x05, 0x20, 0x08, 0x70, 0x10, 0xBD, 0x70, 0xB5, 0x42, 0x4D, 0x00, 0x26,
+ 0x40, 0x4C, 0x28, 0x78, 0x80, 0x1E, 0x04, 0x28, 0x10, 0xD2, 0xDF, 0xE8, 0x00, 0xF0, 0x30, 0x1F,
+ 0x02, 0x3E, 0x00, 0xF0, 0x45, 0xF8, 0x01, 0x28, 0x07, 0xD0, 0x60, 0x68, 0x40, 0x1C, 0x60, 0x60,
+ 0x00, 0xF0, 0x46, 0xF8, 0x01, 0x28, 0x02, 0xD0, 0x0B, 0xE0, 0x66, 0x60, 0x70, 0xBD, 0x39, 0x48,
+ 0x61, 0x68, 0x00, 0x68, 0x90, 0xF8, 0x77, 0x00, 0x88, 0x42, 0x02, 0xD2, 0xBD, 0xE8, 0x70, 0x40,
+ 0xA2, 0xE7, 0x00, 0xF0, 0x47, 0xF8, 0x01, 0x28, 0x0E, 0xD0, 0x70, 0xBD, 0x00, 0xF0, 0x28, 0xF8,
+ 0x01, 0x28, 0x17, 0xD0, 0x60, 0x68, 0x40, 0x1C, 0x60, 0x60, 0x00, 0xF0, 0x3B, 0xF8, 0x01, 0x28,
+ 0x02, 0xD0, 0x00, 0xF0, 0x25, 0xF8, 0x0B, 0xE0, 0xBD, 0xE8, 0x70, 0x40, 0x80, 0xE7, 0x00, 0xF0,
+ 0x17, 0xF8, 0x01, 0x28, 0x06, 0xD0, 0x60, 0x68, 0x40, 0x1C, 0x60, 0x60, 0x00, 0xF0, 0x22, 0xF8,
+ 0x01, 0x28, 0xE2, 0xD0, 0xBD, 0xE8, 0x70, 0x40, 0x8A, 0xE7, 0x00, 0xF0, 0x0D, 0xF9, 0x08, 0xB1,
+ 0x21, 0x78, 0x09, 0xB1, 0x20, 0x70, 0x70, 0xBD, 0x20, 0x70, 0x06, 0x20, 0x28, 0x70, 0x70, 0xBD,
+ 0x10, 0xB5, 0x00, 0xF0, 0x01, 0xF9, 0x04, 0x46, 0x0B, 0xF0, 0x4F, 0xF8, 0x20, 0x43, 0x10, 0xBD,
+ 0x18, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x70, 0x00, 0x80, 0x07, 0x01, 0xD5, 0x01, 0x20, 0x70, 0x47,
+ 0x00, 0x20, 0x70, 0x47, 0x13, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x70, 0x00, 0xC0, 0x07, 0x00, 0xD0,
+ 0x01, 0x20, 0x70, 0x47, 0x10, 0xB5, 0xFF, 0xF7, 0xF5, 0xFF, 0x01, 0x28, 0x0F, 0xD1, 0x00, 0xF0,
+ 0x64, 0xF9, 0x60, 0xB9, 0x0B, 0x48, 0x07, 0x49, 0x00, 0x68, 0x49, 0x68, 0x90, 0xF8, 0x78, 0x00,
+ 0x00, 0xEB, 0x80, 0x00, 0xB1, 0xEB, 0x40, 0x0F, 0x01, 0xD9, 0x01, 0x20, 0x10, 0xBD, 0x00, 0x20,
+ 0x10, 0xBD, 0x00, 0x00, 0xD0, 0x04, 0x10, 0x00, 0xE8, 0x04, 0x10, 0x00, 0x50, 0x07, 0x10, 0x00,
+ 0x51, 0x07, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0x2D, 0xE9, 0xF0, 0x41, 0x90, 0x4C, 0x07, 0x46,
+ 0x0E, 0x46, 0x20, 0x68, 0x00, 0x25, 0x90, 0xF8, 0x98, 0x20, 0x8E, 0x48, 0x0D, 0xF0, 0x07, 0xFF,
+ 0x02, 0x00, 0x8D, 0x48, 0x29, 0x46, 0x34, 0xD0, 0x02, 0x78, 0x01, 0x25, 0x52, 0x1C, 0xD2, 0xB2,
+ 0x02, 0x70, 0x23, 0x68, 0x93, 0xF8, 0x99, 0x30, 0x93, 0x42, 0x2B, 0xD2, 0x01, 0x70, 0x85, 0x48,
+ 0x76, 0x00, 0x32, 0x46, 0x39, 0x46, 0x54, 0x38, 0x12, 0xF0, 0x5C, 0xFB, 0x81, 0x48, 0x32, 0x46,
+ 0x39, 0x46, 0x2A, 0x38, 0x12, 0xF0, 0x56, 0xFB, 0x20, 0x68, 0x03, 0x26, 0x90, 0xF8, 0x50, 0x12,
+ 0xC9, 0x07, 0x03, 0xD1, 0x90, 0xF8, 0x80, 0x02, 0xC0, 0x07, 0x00, 0xD0, 0x0B, 0x26, 0xFC, 0xF7,
+ 0x3D, 0xF9, 0x7A, 0x48, 0x01, 0x68, 0x31, 0x43, 0x01, 0x60, 0xFC, 0xF7, 0x35, 0xF9, 0x20, 0x21,
+ 0x05, 0x20, 0x0C, 0xF0, 0xB7, 0xFB, 0x4F, 0xF4, 0x80, 0x71, 0x00, 0x20, 0x10, 0xF0, 0xAB, 0xFA,
+ 0x00, 0xE0, 0x01, 0x70, 0x28, 0x46, 0xBD, 0xE8, 0xF0, 0x81, 0x2D, 0xE9, 0xF0, 0x5F, 0x8A, 0x46,
+ 0x00, 0x28, 0x6E, 0xD0, 0x6C, 0x4D, 0x00, 0x21, 0xDF, 0xF8, 0xB4, 0xB1, 0xE9, 0x70, 0x89, 0x46,
+ 0x9B, 0xF8, 0x00, 0x20, 0x0C, 0x46, 0x02, 0x2A, 0x03, 0xD0, 0x03, 0x2A, 0x01, 0xD0, 0x05, 0x2A,
+ 0x5E, 0xD1, 0x00, 0x21, 0xBA, 0xF1, 0x00, 0x0F, 0x00, 0xD0, 0x01, 0x21, 0xAA, 0x88, 0x00, 0xEB,
+ 0x41, 0x00, 0x52, 0x1A, 0x16, 0xB2, 0x80, 0x46, 0xAA, 0x78, 0x31, 0x46, 0x00, 0xF0, 0x72, 0xF8,
+ 0x5B, 0x4F, 0xA8, 0x70, 0x38, 0x68, 0x90, 0xF8, 0x90, 0x00, 0x80, 0x07, 0x04, 0xD5, 0x40, 0x46,
+ 0x31, 0x46, 0xFF, 0xF7, 0x89, 0xFF, 0x81, 0x46, 0x01, 0x20, 0xDF, 0xF8, 0x58, 0x81, 0xBA, 0xF1,
+ 0x00, 0x0F, 0x17, 0xD0, 0x00, 0x23, 0x19, 0x46, 0x3F, 0x68, 0x0F, 0xE0, 0x38, 0xF9, 0x11, 0x20,
+ 0x00, 0x2A, 0x00, 0xDA, 0x52, 0x42, 0xB7, 0xF8, 0x92, 0xC0, 0x62, 0x45, 0x04, 0xDD, 0x97, 0xF8,
+ 0x91, 0x20, 0x5B, 0x1C, 0x9A, 0x42, 0x14, 0xDD, 0x49, 0x1C, 0xC9, 0xB2, 0xB1, 0x42, 0xED, 0xDB,
+ 0x21, 0x46, 0x25, 0xE0, 0x00, 0x22, 0x11, 0x46, 0x3B, 0x68, 0x16, 0xE0, 0x38, 0xF9, 0x11, 0x70,
+ 0xB3, 0xF8, 0x92, 0xC0, 0x67, 0x45, 0x0E, 0xDD, 0x93, 0xF8, 0x91, 0x70, 0x52, 0x1C, 0x97, 0x42,
+ 0x09, 0xDC, 0x9B, 0xF8, 0x00, 0x20, 0x01, 0x46, 0x02, 0x2A, 0x09, 0xD0, 0x03, 0x2A, 0x07, 0xD0,
+ 0x05, 0x2A, 0x08, 0xD0, 0x0C, 0xE0, 0x49, 0x1C, 0xC9, 0xB2, 0xB1, 0x42, 0xE6, 0xDB, 0xDF, 0xE7,
+ 0x68, 0x70, 0xE8, 0x70, 0x08, 0xE0, 0x68, 0x78, 0x10, 0xB9, 0xB9, 0xF1, 0x00, 0x0F, 0x02, 0xD0,
+ 0x69, 0x70, 0xBD, 0xE8, 0xF0, 0x9F, 0x69, 0x70, 0x00, 0xF0, 0x49, 0xF9, 0xBD, 0xE8, 0xF0, 0x5F,
+ 0x02, 0x21, 0x01, 0x20, 0x10, 0xF0, 0x27, 0xBA, 0x2F, 0x48, 0x40, 0x78, 0x70, 0x47, 0x2E, 0x48,
+ 0x00, 0x21, 0x01, 0x22, 0x01, 0x70, 0x82, 0x70, 0x29, 0x4A, 0x12, 0x68, 0x92, 0xF8, 0x31, 0x20,
+ 0x82, 0x80, 0x2B, 0x4A, 0x12, 0x78, 0x05, 0x2A, 0x02, 0xD0, 0x06, 0x2A, 0x00, 0xD0, 0x41, 0x70,
+ 0x70, 0x47, 0xEC, 0xE7, 0x2D, 0xE9, 0xFC, 0x41, 0x80, 0x46, 0x23, 0x48, 0x14, 0x46, 0x0D, 0x46,
+ 0x40, 0x78, 0x16, 0x46, 0x40, 0xB9, 0x1F, 0x48, 0x0A, 0x46, 0x7F, 0x23, 0x41, 0x46, 0x54, 0x38,
+ 0x00, 0x94, 0x0D, 0xF0, 0x79, 0xFE, 0x00, 0x26, 0x19, 0x4F, 0x38, 0x68, 0x90, 0xF8, 0x90, 0x10,
+ 0xC9, 0x07, 0x00, 0xD0, 0xEC, 0xB1, 0x17, 0x48, 0x6A, 0x00, 0x41, 0x46, 0x2A, 0x38, 0x12, 0xF0,
+ 0x81, 0xFA, 0x14, 0x4A, 0x2B, 0x46, 0x54, 0x3A, 0x02, 0xF1, 0x2A, 0x01, 0x11, 0x48, 0x0D, 0xF0,
+ 0x2C, 0xFE, 0x38, 0x68, 0x90, 0xF8, 0x90, 0x10, 0x49, 0x07, 0x07, 0xD5, 0x90, 0xF8, 0x9A, 0x30,
+ 0x23, 0xB1, 0x0C, 0x49, 0x2A, 0x46, 0x08, 0x46, 0x0D, 0xF0, 0x6F, 0xFE, 0x30, 0x46, 0xBD, 0xE8,
+ 0xFC, 0x81, 0xB0, 0xF9, 0x96, 0x10, 0xCD, 0xE9, 0x00, 0x15, 0x06, 0x4A, 0xB0, 0xF9, 0x94, 0x30,
+ 0x54, 0x3A, 0x41, 0x46, 0x02, 0xF1, 0x2A, 0x00, 0x0D, 0xF0, 0x96, 0xFD, 0xD9, 0xE7, 0x00, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0x5A, 0x09, 0x01, 0x20, 0xD8, 0x04, 0x10, 0x00, 0x28, 0x07, 0x10, 0x00,
+ 0xE8, 0x04, 0x10, 0x00, 0x70, 0xB5, 0x00, 0x28, 0x04, 0x9E, 0x0D, 0xD0, 0x00, 0x24, 0x06, 0xE0,
+ 0x30, 0xF9, 0x11, 0x50, 0x9D, 0x42, 0x01, 0xDD, 0x95, 0x42, 0x00, 0xDB, 0x64, 0x1C, 0x49, 0x1E,
+ 0xF6, 0xD2, 0xB4, 0x42, 0x01, 0xDD, 0x01, 0x20, 0x70, 0xBD, 0x00, 0x20, 0x70, 0xBD, 0x3F, 0x48,
+ 0x00, 0x21, 0x01, 0x70, 0x41, 0x60, 0x70, 0x47, 0xF9, 0xE7, 0x3C, 0x48, 0x00, 0x78, 0x70, 0x47,
+ 0x2D, 0xE9, 0xF3, 0x4F, 0xDF, 0xF8, 0xE8, 0xA0, 0x84, 0x46, 0x4F, 0xF0, 0x00, 0x08, 0xDA, 0xF8,
+ 0x00, 0x00, 0xDF, 0xF8, 0xD8, 0xB0, 0x81, 0xB0, 0x90, 0xF8, 0x80, 0x10, 0x47, 0x46, 0xC9, 0x07,
+ 0xC1, 0x46, 0x2B, 0xD0, 0x33, 0x49, 0x09, 0x78, 0x03, 0x29, 0x2D, 0xD0, 0x04, 0x29, 0x2B, 0xD0,
+ 0x05, 0x29, 0x36, 0xD1, 0xB0, 0xF8, 0x88, 0x60, 0xB0, 0xF8, 0x8A, 0x50, 0x90, 0xF8, 0x87, 0x40,
+ 0xBC, 0xF1, 0x00, 0x0F, 0x08, 0xD0, 0x00, 0x94, 0x90, 0xF8, 0x31, 0x10, 0x2B, 0xB2, 0x32, 0xB2,
+ 0x60, 0x46, 0xFF, 0xF7, 0xB7, 0xFF, 0x80, 0x46, 0x02, 0x98, 0x50, 0xB1, 0x00, 0x94, 0xDA, 0xF8,
+ 0x00, 0x00, 0x2B, 0xB2, 0x32, 0xB2, 0x90, 0xF8, 0x30, 0x10, 0x02, 0x98, 0xFF, 0xF7, 0xAA, 0xFF,
+ 0x07, 0x46, 0xB8, 0xF1, 0x01, 0x0F, 0x0E, 0xD0, 0x01, 0x2F, 0x0C, 0xD0, 0x9B, 0xF8, 0x00, 0x10,
+ 0x58, 0x46, 0x01, 0x29, 0x1D, 0xD0, 0x0A, 0xE0, 0xB0, 0xF8, 0x82, 0x60, 0xB0, 0xF8, 0x84, 0x50,
+ 0x90, 0xF8, 0x86, 0x40, 0xD4, 0xE7, 0x9B, 0xF8, 0x00, 0x10, 0x58, 0x46, 0x19, 0xB1, 0xCB, 0xF8,
+ 0x04, 0x90, 0xBD, 0xE8, 0xFE, 0x8F, 0x41, 0x68, 0x49, 0x1C, 0x41, 0x60, 0xDA, 0xF8, 0x00, 0x20,
+ 0x92, 0xF8, 0x81, 0x20, 0x8A, 0x42, 0xF4, 0xDA, 0x01, 0x21, 0x01, 0x70, 0xCB, 0xF8, 0x04, 0x90,
+ 0x0D, 0xE0, 0x41, 0x68, 0x49, 0x1C, 0x41, 0x60, 0xDA, 0xF8, 0x00, 0x20, 0x92, 0xF8, 0x81, 0x20,
+ 0x8A, 0x42, 0xE6, 0xDA, 0x80, 0xF8, 0x00, 0x90, 0x00, 0x21, 0xCB, 0xF8, 0x04, 0x90, 0x03, 0xB0,
+ 0x07, 0x20, 0xBD, 0xE8, 0xF0, 0x4F, 0x0C, 0xF0, 0x35, 0xBA, 0x00, 0x00, 0xE0, 0x04, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0xE8, 0x04, 0x10, 0x00, 0x22, 0x48, 0x08, 0xB5, 0x00, 0x21, 0x41, 0x60,
+ 0x01, 0x22, 0x81, 0x60, 0x02, 0x70, 0x41, 0x70, 0x1F, 0x48, 0x01, 0x70, 0xFF, 0xF7, 0x09, 0xFF,
+ 0xFF, 0xF7, 0x72, 0xFF, 0xFF, 0xF7, 0xFB, 0xFB, 0xFF, 0xF7, 0x7F, 0xFD, 0x6B, 0x46, 0x00, 0x22,
+ 0x5E, 0x21, 0x01, 0x20, 0x10, 0xF0, 0x58, 0xF9, 0x00, 0x98, 0x41, 0x07, 0x03, 0xD5, 0xFF, 0xF7,
+ 0xF4, 0xFA, 0x04, 0x21, 0x0E, 0xE0, 0x01, 0x07, 0x03, 0xD5, 0xFF, 0xF7, 0x19, 0xF9, 0x08, 0x21,
+ 0x08, 0xE0, 0xC1, 0x06, 0x03, 0xD5, 0xFF, 0xF7, 0x7D, 0xFA, 0x10, 0x21, 0x02, 0xE0, 0x41, 0x06,
+ 0x04, 0xD5, 0x40, 0x21, 0x01, 0x20, 0x10, 0xF0, 0x22, 0xF9, 0xDF, 0xE7, 0x80, 0x07, 0xDD, 0xD5,
+ 0x02, 0x21, 0x01, 0x20, 0x10, 0xF0, 0x1B, 0xF9, 0xFF, 0xF7, 0xD5, 0xFB, 0xD6, 0xE7, 0x00, 0xEB,
+ 0x80, 0x00, 0x4F, 0xF6, 0xFF, 0x72, 0x02, 0xEA, 0x40, 0x00, 0x11, 0xF0, 0x85, 0xBE, 0x11, 0xF0,
+ 0xA5, 0xBE, 0x00, 0x00, 0xE8, 0x04, 0x10, 0x00, 0x52, 0x07, 0x10, 0x00, 0x00, 0xF0, 0xA3, 0xBF,
+ 0x5E, 0x48, 0x40, 0x78, 0x00, 0x28, 0x0B, 0xD0, 0x5D, 0x48, 0x5E, 0x49, 0x00, 0x7E, 0x09, 0x7E,
+ 0xC8, 0x42, 0x05, 0xD1, 0x01, 0x23, 0x5C, 0x4A, 0x5A, 0x49, 0x59, 0x48, 0x00, 0xF0, 0xE0, 0xBB,
+ 0x70, 0x47, 0x3C, 0xB5, 0x00, 0x23, 0x00, 0x93, 0x01, 0x93, 0x6B, 0x46, 0x07, 0xE0, 0x44, 0x18,
+ 0x64, 0x78, 0x14, 0xB1, 0x1D, 0x5D, 0x6D, 0x1C, 0x1D, 0x55, 0x49, 0x1C, 0xC9, 0xB2, 0x91, 0x42,
+ 0xF5, 0xD9, 0x00, 0x20, 0x01, 0x21, 0x1C, 0x5C, 0x5A, 0x5C, 0xA2, 0x42, 0x00, 0xD9, 0x08, 0x46,
+ 0x49, 0x1C, 0xC9, 0xB2, 0x06, 0x29, 0xF6, 0xD3, 0x3C, 0xBD, 0x70, 0xB5, 0x04, 0x46, 0x00, 0x20,
+ 0x08, 0x70, 0x0E, 0x46, 0x10, 0x70, 0x46, 0x48, 0x46, 0x49, 0x15, 0x46, 0x00, 0x7E, 0x09, 0x7E,
+ 0xC8, 0x42, 0x0B, 0xD0, 0xE2, 0x78, 0xA1, 0x78, 0x44, 0x48, 0xFF, 0xF7, 0xD2, 0xFF, 0x30, 0x70,
+ 0x62, 0x78, 0x21, 0x78, 0x42, 0x48, 0xFF, 0xF7, 0xCC, 0xFF, 0x28, 0x70, 0x70, 0xBD, 0x70, 0xB5,
+ 0x00, 0x24, 0x01, 0x23, 0x05, 0x7E, 0x08, 0xE0, 0xC6, 0x18, 0x96, 0xF9, 0x12, 0x60, 0x8E, 0x42,
+ 0x02, 0xDB, 0x96, 0x42, 0x00, 0xDC, 0x64, 0x1C, 0x5B, 0x1C, 0x9D, 0x42, 0xF4, 0xD2, 0x01, 0x2C,
+ 0x01, 0xDC, 0x01, 0x20, 0x70, 0xBD, 0x00, 0x20, 0x70, 0xBD, 0x2D, 0xE9, 0xF0, 0x43, 0x99, 0x46,
+ 0x00, 0x23, 0x1F, 0x46, 0x47, 0xF6, 0xFF, 0x76, 0x4F, 0xF0, 0xFF, 0x3C, 0x01, 0x25, 0x04, 0x7E,
+ 0x0D, 0xE0, 0x00, 0xBF, 0x00, 0xEB, 0x05, 0x08, 0x98, 0xF9, 0x12, 0x80, 0x88, 0x45, 0x05, 0xDB,
+ 0x90, 0x45, 0x03, 0xDC, 0x43, 0x19, 0x93, 0xF9, 0x12, 0x30, 0x02, 0xE0, 0x6D, 0x1C, 0xAC, 0x42,
+ 0xF0, 0xD2, 0x64, 0x1C, 0x0A, 0xE0, 0x05, 0x19, 0x95, 0xF9, 0x12, 0x50, 0x8D, 0x42, 0x05, 0xDB,
+ 0x95, 0x42, 0x03, 0xDC, 0x20, 0x44, 0x90, 0xF9, 0x12, 0x70, 0x01, 0xE0, 0x64, 0x1E, 0xF2, 0xD1,
+ 0xBB, 0x42, 0x09, 0xD1, 0x0A, 0xE0, 0x00, 0xBF, 0x39, 0xF9, 0x13, 0x00, 0xB0, 0x42, 0x02, 0xDA,
+ 0x06, 0x46, 0x4F, 0xFA, 0x83, 0xFC, 0x5B, 0x1C, 0xBB, 0x42, 0xF5, 0xDD, 0x60, 0x46, 0xBD, 0xE8,
+ 0xF0, 0x83, 0x2D, 0xE9, 0xF0, 0x41, 0x17, 0x46, 0x0E, 0x46, 0x04, 0x46, 0xC2, 0x78, 0x81, 0x78,
+ 0x0F, 0x48, 0xFF, 0xF7, 0xA4, 0xFF, 0x05, 0x00, 0x06, 0xD1, 0xE2, 0x78, 0xA1, 0x78, 0x11, 0x4B,
+ 0x0B, 0x48, 0xFF, 0xF7, 0xB2, 0xFF, 0x30, 0x70, 0x62, 0x78, 0x21, 0x78, 0x09, 0x48, 0xFF, 0xF7,
+ 0x96, 0xFF, 0x06, 0x00, 0x06, 0xD1, 0x62, 0x78, 0x21, 0x78, 0x0B, 0x4B, 0x05, 0x48, 0xFF, 0xF7,
+ 0xA4, 0xFF, 0x38, 0x70, 0x05, 0xEA, 0x06, 0x00, 0xBD, 0xE8, 0xF0, 0x81, 0xEC, 0x06, 0x10, 0x00,
+ 0x58, 0x23, 0x10, 0x00, 0x98, 0x23, 0x10, 0x00, 0x80, 0x24, 0x10, 0x00, 0xFC, 0x0B, 0x01, 0x20,
+ 0x13, 0x0C, 0x01, 0x20, 0x36, 0x0B, 0x01, 0x20, 0x60, 0x0B, 0x01, 0x20, 0x2D, 0xE9, 0xF0, 0x47,
+ 0x00, 0x25, 0x82, 0x46, 0xA6, 0xB0, 0x50, 0x1E, 0xC7, 0xB2, 0x2C, 0x46, 0x28, 0x46, 0x6B, 0x46,
+ 0x0D, 0xF1, 0x4C, 0x09, 0x12, 0xE0, 0x00, 0xBF, 0x31, 0xF9, 0x10, 0x60, 0x35, 0x44, 0x46, 0x1C,
+ 0xB4, 0x46, 0x95, 0xFB, 0xF6, 0xF6, 0x23, 0xF8, 0x10, 0x60, 0x3E, 0x1A, 0x40, 0x1C, 0x31, 0xF9,
+ 0x16, 0x80, 0x44, 0x44, 0x94, 0xFB, 0xFC, 0xFC, 0x29, 0xF8, 0x16, 0xC0, 0x90, 0x42, 0xEB, 0xDB,
+ 0x48, 0x46, 0x0C, 0xE0, 0x33, 0xF9, 0x12, 0x40, 0x30, 0xF9, 0x12, 0x50, 0x2C, 0x44, 0x31, 0xF8,
+ 0x12, 0x50, 0x04, 0xEB, 0xD4, 0x74, 0xA5, 0xEB, 0x64, 0x04, 0x2A, 0xF8, 0x12, 0x40, 0x52, 0x1E,
+ 0xF0, 0xD2, 0x26, 0xB0, 0xBD, 0xE8, 0xF0, 0x87, 0x2D, 0xE9, 0xFF, 0x4F, 0x4F, 0xF0, 0x00, 0x0B,
+ 0xA7, 0xB0, 0xC0, 0xF8, 0x00, 0xB0, 0x81, 0x46, 0xC0, 0xF8, 0x04, 0xB0, 0x98, 0x46, 0x14, 0x46,
+ 0x5D, 0x46, 0x5E, 0x46, 0xDA, 0x46, 0x5F, 0x46, 0x28, 0x21, 0x01, 0xA8, 0x12, 0xF0, 0xF3, 0xF8,
+ 0x4C, 0x21, 0x0B, 0xA8, 0x12, 0xF0, 0xEF, 0xF8, 0xCD, 0xF8, 0x00, 0x80, 0x02, 0x23, 0x22, 0x46,
+ 0x0B, 0xA8, 0x28, 0x99, 0x0D, 0xF0, 0x16, 0xFC, 0x8E, 0x49, 0x01, 0x20, 0x02, 0x46, 0x00, 0x23,
+ 0x0D, 0xF1, 0x04, 0x0E, 0xD1, 0xF8, 0x00, 0x80, 0x17, 0xE0, 0x0B, 0xA9, 0x31, 0xF9, 0x10, 0xC0,
+ 0xB8, 0xF9, 0x4E, 0x12, 0x8C, 0x45, 0x04, 0xDD, 0x4F, 0xF0, 0x01, 0x0B, 0x0E, 0xF8, 0x00, 0x20,
+ 0x0A, 0xE0, 0x49, 0x42, 0x8C, 0x45, 0x05, 0xDA, 0xFF, 0x21, 0x4F, 0xF0, 0x01, 0x0B, 0x0E, 0xF8,
+ 0x00, 0x10, 0x01, 0xE0, 0x0E, 0xF8, 0x00, 0x30, 0x40, 0x1C, 0xA0, 0x42, 0xE5, 0xDB, 0xBB, 0xF1,
+ 0x01, 0x0F, 0x42, 0xD1, 0x01, 0x20, 0x71, 0x46, 0x09, 0xE0, 0x0A, 0x5C, 0x32, 0xB1, 0xC5, 0xB2,
+ 0x08, 0x56, 0x00, 0x28, 0x05, 0xDD, 0x4F, 0xF0, 0x01, 0x0A, 0x02, 0xE0, 0x40, 0x1C, 0xA0, 0x42,
+ 0xF3, 0xDB, 0x60, 0x1E, 0x07, 0xE0, 0x0A, 0x5C, 0x2A, 0xB1, 0xC6, 0xB2, 0x08, 0x56, 0x00, 0x28,
+ 0x03, 0xDA, 0x01, 0x27, 0x01, 0xE0, 0x40, 0x1E, 0xF5, 0xD2, 0xA4, 0xF1, 0x01, 0x08, 0xBA, 0xF1,
+ 0x00, 0x0F, 0x01, 0xD0, 0x97, 0xB9, 0x02, 0xE0, 0x4F, 0xF0, 0x01, 0x05, 0x77, 0xB9, 0x08, 0xF0,
+ 0xFF, 0x06, 0x0B, 0xE0, 0xD9, 0xE9, 0x00, 0x47, 0x2A, 0x46, 0x01, 0x20, 0x00, 0x21, 0x11, 0xF0,
+ 0xBA, 0xFF, 0x04, 0x43, 0x0F, 0x43, 0xC9, 0xE9, 0x00, 0x47, 0x6D, 0x1C, 0xB5, 0x42, 0xF1, 0xDD,
+ 0xD9, 0xE9, 0x00, 0x04, 0x40, 0xF0, 0x01, 0x05, 0x42, 0x46, 0x01, 0x20, 0x00, 0x21, 0x11, 0xF0,
+ 0xAA, 0xFF, 0x05, 0x43, 0x0C, 0x43, 0xC9, 0xE9, 0x00, 0x54, 0x2B, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F,
+ 0x2D, 0xE9, 0xF8, 0x43, 0x05, 0x46, 0x16, 0x46, 0x0F, 0x46, 0x03, 0x20, 0x0F, 0xF0, 0xE3, 0xFF,
+ 0x55, 0x4C, 0x56, 0x4A, 0x29, 0x46, 0x23, 0x78, 0x55, 0x48, 0x0D, 0xF0, 0x7E, 0xFB, 0x55, 0x4D,
+ 0x53, 0x48, 0x55, 0x4A, 0x2B, 0x78, 0x39, 0x46, 0x2A, 0x30, 0x0D, 0xF0, 0x76, 0xFB, 0x03, 0x20,
+ 0x0F, 0xF0, 0xEC, 0xFF, 0xDF, 0xF8, 0x2C, 0x81, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0xE0, 0x13,
+ 0x89, 0x07, 0x12, 0xD5, 0x90, 0xF8, 0xF4, 0x03, 0x00, 0x96, 0x00, 0xF0, 0x0F, 0x07, 0x3B, 0x46,
+ 0x22, 0x78, 0x47, 0x49, 0x49, 0x48, 0x0D, 0xF0, 0x7D, 0xFB, 0x45, 0x49, 0x00, 0x96, 0x3B, 0x46,
+ 0x2A, 0x78, 0x2A, 0x31, 0x46, 0x48, 0x0D, 0xF0, 0x75, 0xFB, 0x22, 0x78, 0x40, 0x49, 0x45, 0x48,
+ 0x00, 0xF0, 0x5B, 0xF8, 0x3E, 0x49, 0x2A, 0x78, 0x2A, 0x31, 0x43, 0x48, 0x00, 0xF0, 0x55, 0xF8,
+ 0x42, 0x4F, 0x00, 0x96, 0x22, 0x78, 0x3F, 0x49, 0x41, 0x48, 0x3B, 0x68, 0x0D, 0xF0, 0x62, 0xFB,
+ 0x00, 0x96, 0x2A, 0x78, 0x3C, 0x49, 0x3F, 0x48, 0x3B, 0x68, 0x0D, 0xF0, 0x5B, 0xFB, 0xD8, 0xF8,
+ 0x00, 0x00, 0x47, 0x46, 0x90, 0xF8, 0x30, 0x02, 0xC0, 0x06, 0x0A, 0xD5, 0x22, 0x78, 0x38, 0x49,
+ 0x39, 0x48, 0xFF, 0xF7, 0xE3, 0xFE, 0x2A, 0x78, 0x36, 0x49, 0x38, 0x48, 0xFF, 0xF7, 0xDE, 0xFE,
+ 0x0B, 0xE0, 0x20, 0x78, 0x32, 0x49, 0x42, 0x00, 0x33, 0x48, 0x11, 0xF0, 0x73, 0xFF, 0x28, 0x78,
+ 0x30, 0x49, 0x42, 0x00, 0x31, 0x48, 0x11, 0xF0, 0x6D, 0xFF, 0x38, 0x68, 0x90, 0xF8, 0x30, 0x02,
+ 0x80, 0x06, 0x04, 0xD5, 0x2C, 0x49, 0x22, 0x78, 0x08, 0x46, 0x00, 0xF0, 0x30, 0xF8, 0x38, 0x68,
+ 0x90, 0xF8, 0x30, 0x02, 0x40, 0x06, 0x04, 0xD5, 0x28, 0x49, 0x2A, 0x78, 0x08, 0x46, 0x00, 0xF0,
+ 0x26, 0xF8, 0x33, 0x46, 0x22, 0x78, 0x1A, 0x49, 0x25, 0x48, 0xFF, 0xF7, 0xED, 0xFE, 0x2A, 0x78,
+ 0x01, 0xB0, 0x33, 0x46, 0xBD, 0xE8, 0xF0, 0x43, 0x15, 0x49, 0x21, 0x48, 0x2A, 0x31, 0x08, 0x30,
+ 0xE2, 0xE6, 0x20, 0x49, 0x00, 0x20, 0x08, 0x70, 0x70, 0x47, 0x30, 0xB5, 0x00, 0x23, 0x03, 0x80,
+ 0x01, 0x23, 0x09, 0xE0, 0x00, 0xEB, 0x43, 0x04, 0x31, 0xF8, 0x13, 0x50, 0x34, 0xF8, 0x02, 0x4C,
+ 0x2C, 0x44, 0x20, 0xF8, 0x13, 0x40, 0x5B, 0x1C, 0x93, 0x42, 0xF3, 0xDB, 0x30, 0xBD, 0x08, 0xB5,
+ 0x00, 0x92, 0x04, 0x4A, 0x12, 0x68, 0xB2, 0xF9, 0x4C, 0x32, 0xB2, 0xF9, 0x4A, 0x22, 0x01, 0xF0,
+ 0xED, 0xFC, 0x08, 0xBD, 0x20, 0x07, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0x6A, 0x4A, 0x01, 0x20,
+ 0x84, 0x09, 0x01, 0x20, 0xBA, 0x06, 0x10, 0x00, 0x94, 0x4A, 0x01, 0x20, 0x08, 0x24, 0x10, 0x00,
+ 0x32, 0x24, 0x10, 0x00, 0x92, 0x0A, 0x01, 0x20, 0xAC, 0x0B, 0x01, 0x20, 0x78, 0x06, 0x10, 0x00,
+ 0xC0, 0x0A, 0x01, 0x20, 0xEA, 0x0A, 0x01, 0x20, 0x36, 0x0B, 0x01, 0x20, 0x60, 0x0B, 0x01, 0x20,
+ 0x90, 0x24, 0x10, 0x00, 0xC0, 0x06, 0x10, 0x00, 0x10, 0xB5, 0x40, 0x21, 0xFA, 0x48, 0x11, 0xF0,
+ 0xA2, 0xFF, 0xF9, 0x48, 0x40, 0x21, 0x40, 0x30, 0x11, 0xF0, 0x9D, 0xFF, 0xF6, 0x48, 0x00, 0x21,
+ 0x80, 0x30, 0xC0, 0xE9, 0x00, 0x11, 0xC0, 0xE9, 0x02, 0x11, 0x10, 0x30, 0xC0, 0xE9, 0x00, 0x11,
+ 0xC0, 0xE9, 0x02, 0x11, 0x10, 0x30, 0xC0, 0xE9, 0x00, 0x11, 0xC0, 0xE9, 0x02, 0x11, 0x08, 0x46,
+ 0xEE, 0x49, 0x08, 0x60, 0x10, 0xBD, 0x2D, 0xE9, 0xFF, 0x5F, 0x06, 0x46, 0x89, 0x46, 0xEC, 0x48,
+ 0xEC, 0x49, 0x98, 0x46, 0x00, 0x78, 0x09, 0x68, 0x14, 0x46, 0x01, 0x28, 0x50, 0xD0, 0xB1, 0xF9,
+ 0x86, 0xA2, 0x91, 0xF8, 0x88, 0x52, 0xB1, 0xF9, 0x82, 0x72, 0x11, 0x46, 0x48, 0x46, 0x0D, 0xF0,
+ 0x26, 0xFB, 0x83, 0x46, 0x21, 0x46, 0x48, 0x46, 0x0D, 0xF0, 0x2C, 0xFB, 0x00, 0x90, 0x39, 0x46,
+ 0x58, 0x46, 0x0C, 0xF0, 0xFB, 0xFB, 0x0F, 0xFA, 0x80, 0xFB, 0x39, 0x46, 0x00, 0x98, 0x0C, 0xF0,
+ 0xF5, 0xFB, 0x00, 0xB2, 0xAB, 0xEB, 0x00, 0x00, 0x68, 0x43, 0x64, 0x21, 0x90, 0xFB, 0xF1, 0xF0,
+ 0x38, 0x44, 0x05, 0xB2, 0xA8, 0xF8, 0x08, 0x50, 0x70, 0x1C, 0x00, 0x21, 0x03, 0x90, 0xC8, 0xF8,
+ 0x00, 0x10, 0xC8, 0xF8, 0x04, 0x10, 0xB9, 0xF9, 0x00, 0x00, 0xA8, 0x42, 0x02, 0xDD, 0x03, 0x99,
+ 0x01, 0x20, 0x08, 0x70, 0x09, 0xEB, 0x44, 0x00, 0x76, 0x1C, 0x30, 0xF9, 0x02, 0x2C, 0x31, 0x19,
+ 0xAA, 0x42, 0x02, 0xDD, 0x01, 0x22, 0x01, 0xF8, 0x01, 0x2C, 0xB9, 0xF9, 0x02, 0x30, 0x03, 0xEB,
+ 0x0A, 0x02, 0x12, 0xB2, 0xAB, 0x42, 0x18, 0xDD, 0xB9, 0xF9, 0x00, 0x30, 0x93, 0x42, 0x03, 0xDD,
+ 0xB9, 0xF9, 0x04, 0x30, 0x93, 0x42, 0x0A, 0xDC, 0x03, 0x9B, 0x01, 0x22, 0x5A, 0x70, 0x0C, 0xE0,
+ 0xB1, 0xF9, 0x6A, 0xA2, 0x91, 0xF8, 0x65, 0x52, 0xB1, 0xF9, 0x66, 0x72, 0xAD, 0xE7, 0xD8, 0xF8,
+ 0x00, 0x60, 0x46, 0xF0, 0x02, 0x06, 0xC8, 0xF8, 0x00, 0x60, 0x30, 0xF9, 0x04, 0x3C, 0xA4, 0xF1,
+ 0x02, 0x0B, 0x03, 0xEB, 0x0A, 0x02, 0x12, 0xB2, 0xAB, 0x42, 0x16, 0xDD, 0x30, 0xF9, 0x06, 0x3C,
+ 0x93, 0x42, 0x03, 0xDD, 0x30, 0xF9, 0x02, 0x0C, 0x90, 0x42, 0x03, 0xDC, 0x01, 0x20, 0x01, 0xF8,
+ 0x02, 0x0C, 0x0A, 0xE0, 0xD8, 0xE9, 0x00, 0x64, 0x5A, 0x46, 0x01, 0x20, 0x00, 0x21, 0x11, 0xF0,
+ 0x2A, 0xFE, 0x06, 0x43, 0x0C, 0x43, 0xC8, 0xE9, 0x00, 0x64, 0x02, 0x24, 0x58, 0xE0, 0x00, 0xBF,
+ 0x39, 0xF9, 0x14, 0x00, 0xA8, 0x42, 0x52, 0xDD, 0x09, 0xEB, 0x44, 0x01, 0x02, 0x91, 0x50, 0x44,
+ 0x31, 0xF9, 0x02, 0x1C, 0x00, 0xB2, 0x81, 0x42, 0x04, 0xDD, 0x02, 0x9A, 0xB2, 0xF9, 0x02, 0x20,
+ 0x82, 0x42, 0x10, 0xDC, 0x02, 0x9A, 0x32, 0xF9, 0x04, 0x2C, 0x82, 0x42, 0x04, 0xDD, 0x02, 0x9A,
+ 0xB2, 0xF9, 0x02, 0x20, 0x82, 0x42, 0x06, 0xDC, 0x81, 0x42, 0x35, 0xDD, 0x02, 0x99, 0xB1, 0xF9,
+ 0x04, 0x10, 0x81, 0x42, 0x30, 0xDD, 0xD8, 0xE9, 0x00, 0x67, 0x22, 0x46, 0x01, 0x20, 0x00, 0x21,
+ 0x11, 0xF0, 0xF9, 0xFD, 0xCD, 0xE9, 0x00, 0x01, 0x06, 0x43, 0x0F, 0x43, 0x62, 0x1E, 0x01, 0x20,
+ 0x00, 0x21, 0xC8, 0xE9, 0x00, 0x67, 0x11, 0xF0, 0xEE, 0xFD, 0x02, 0x46, 0x0B, 0x46, 0x06, 0xEA,
+ 0x02, 0x00, 0x07, 0xEA, 0x03, 0x01, 0x08, 0x43, 0x19, 0xD0, 0x02, 0x98, 0x39, 0xF9, 0x14, 0x10,
+ 0x30, 0xF9, 0x02, 0x0C, 0x81, 0x42, 0x09, 0xDA, 0x96, 0x43, 0x9F, 0x43, 0xC8, 0xE9, 0x00, 0x67,
+ 0x03, 0x99, 0x01, 0x20, 0x21, 0x44, 0x01, 0xF8, 0x01, 0x0C, 0x08, 0xE0, 0xDD, 0xE9, 0x00, 0x10,
+ 0x8E, 0x43, 0x87, 0x43, 0xC8, 0xE9, 0x00, 0x67, 0x03, 0x99, 0x01, 0x20, 0x08, 0x55, 0x64, 0x1C,
+ 0x5C, 0x45, 0xA5, 0xD3, 0xBD, 0xE8, 0xFF, 0x9F, 0x2D, 0xE9, 0xFF, 0x4F, 0x81, 0xB0, 0x82, 0x46,
+ 0x00, 0x21, 0x1D, 0x46, 0x19, 0x76, 0x06, 0x20, 0x90, 0x46, 0xC3, 0x1F, 0x76, 0x4C, 0x07, 0xE0,
+ 0x00, 0x22, 0x29, 0x18, 0x0A, 0x73, 0x8B, 0x74, 0x05, 0xEB, 0x40, 0x01, 0x22, 0x88, 0x0A, 0x80,
+ 0x40, 0x1E, 0xF5, 0xD2, 0x6E, 0x48, 0x01, 0x78, 0x6E, 0x48, 0x01, 0x29, 0x00, 0x68, 0x06, 0xD0,
+ 0xB0, 0xF9, 0x84, 0x22, 0x4F, 0xF0, 0x00, 0x0B, 0x59, 0x46, 0x58, 0x46, 0x16, 0xE0, 0xB0, 0xF9,
+ 0x68, 0x22, 0xF7, 0xE7, 0x0A, 0xEB, 0x00, 0x03, 0x5C, 0x78, 0x74, 0xB1, 0x1A, 0xF8, 0x00, 0x40,
+ 0x0C, 0xB9, 0x49, 0x1C, 0xC9, 0xB2, 0x59, 0x70, 0x02, 0x9B, 0x33, 0xF9, 0x10, 0x30, 0x93, 0x42,
+ 0x03, 0xDD, 0x01, 0x23, 0x8B, 0x40, 0x43, 0xEA, 0x0B, 0x0B, 0x40, 0x1C, 0x40, 0x45, 0xE9, 0xD3,
+ 0x5E, 0x48, 0x00, 0x90, 0x0E, 0x98, 0x4F, 0xF0, 0x00, 0x09, 0x4C, 0x46, 0xC0, 0xE9, 0x00, 0x99,
+ 0x37, 0xE0, 0x00, 0xBF, 0x0A, 0xEB, 0x04, 0x00, 0x42, 0x78, 0x8A, 0xB3, 0x01, 0x21, 0x91, 0x40,
+ 0x11, 0xEA, 0x0B, 0x0F, 0x02, 0xD1, 0x00, 0x21, 0x41, 0x70, 0x29, 0xE0, 0x1A, 0xF8, 0x04, 0x10,
+ 0x39, 0xB9, 0x09, 0xF1, 0x01, 0x01, 0x01, 0xF0, 0xFF, 0x09, 0x50, 0x49, 0x00, 0x91, 0x05, 0xF8,
+ 0x19, 0x40, 0x80, 0xF8, 0x01, 0x90, 0x0E, 0x98, 0x22, 0x46, 0x00, 0x21, 0xD0, 0xE9, 0x00, 0x67,
+ 0x01, 0x20, 0x11, 0xF0, 0x60, 0xFD, 0x06, 0x43, 0x0E, 0x98, 0x0F, 0x43, 0xC0, 0xE9, 0x00, 0x67,
+ 0x05, 0xEB, 0x09, 0x00, 0x01, 0x7B, 0x49, 0x1C, 0x01, 0x73, 0x05, 0xEB, 0x49, 0x01, 0x4C, 0x70,
+ 0x02, 0x99, 0x00, 0x9A, 0x31, 0xF9, 0x14, 0x10, 0x91, 0x42, 0x01, 0xDD, 0x00, 0x91, 0x84, 0x74,
+ 0x64, 0x1C, 0x44, 0x45, 0xC6, 0xD3, 0x85, 0xF8, 0x18, 0x90, 0x05, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F,
+ 0x2D, 0xE9, 0xF8, 0x4F, 0x0C, 0x46, 0x05, 0x46, 0x1E, 0x46, 0x91, 0x46, 0x17, 0x21, 0x38, 0x48,
+ 0x11, 0xF0, 0xEF, 0xFD, 0x28, 0x21, 0x37, 0x48, 0x11, 0xF0, 0xEB, 0xFD, 0xDF, 0xF8, 0xD8, 0xA0,
+ 0x36, 0x4F, 0x09, 0xF1, 0x08, 0x08, 0x9A, 0xF8, 0x00, 0x10, 0x76, 0xB1, 0x0A, 0x46, 0x05, 0xF1,
+ 0x20, 0x03, 0x33, 0x49, 0x2E, 0x48, 0xFF, 0xF7, 0x76, 0xFE, 0x04, 0xF1, 0x20, 0x03, 0x3A, 0x78,
+ 0x30, 0x49, 0x2C, 0x48, 0xFF, 0xF7, 0x6F, 0xFE, 0x11, 0xE0, 0x4A, 0x46, 0x28, 0x48, 0x00, 0xF0,
+ 0x24, 0xF9, 0x42, 0x46, 0x39, 0x78, 0x27, 0x48, 0x00, 0xF0, 0x1F, 0xF9, 0x00, 0x20, 0xC5, 0xE9,
+ 0x08, 0x00, 0xC4, 0xE9, 0x08, 0x00, 0x28, 0x48, 0x00, 0x88, 0x28, 0x85, 0x20, 0x85, 0x27, 0x4E,
+ 0x28, 0x8D, 0x2B, 0x46, 0x26, 0xF8, 0x11, 0x0F, 0x20, 0x8D, 0x70, 0x80, 0xCD, 0xF8, 0x00, 0x90,
+ 0x9A, 0xF8, 0x00, 0x20, 0x1E, 0x49, 0x1A, 0x48, 0xFF, 0xF7, 0x36, 0xFF, 0xCD, 0xF8, 0x00, 0x80,
+ 0x23, 0x46, 0x3A, 0x78, 0x1B, 0x49, 0x17, 0x48, 0xFF, 0xF7, 0x2E, 0xFF, 0x29, 0x7E, 0x30, 0x79,
+ 0x61, 0xF3, 0x03, 0x00, 0x30, 0x71, 0x21, 0x7E, 0x61, 0xF3, 0x07, 0x10, 0x30, 0x71, 0xBD, 0xE8,
+ 0xF8, 0x8F, 0x70, 0xB5, 0x0B, 0x4C, 0x20, 0x68, 0x90, 0xF8, 0x30, 0x12, 0x09, 0x07, 0x42, 0xD5,
+ 0x13, 0x49, 0x0A, 0x78, 0x0A, 0xB9, 0x49, 0x78, 0xE9, 0xB3, 0x05, 0x49, 0x09, 0x78, 0x01, 0x29,
+ 0x20, 0xD0, 0xB0, 0xF8, 0x42, 0x12, 0x1F, 0xE0, 0x58, 0x23, 0x10, 0x00, 0x00, 0x05, 0x10, 0x00,
+ 0x0E, 0x05, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0x18, 0x6B, 0x01, 0x00, 0x00, 0x80, 0xFF, 0xFF,
+ 0xFC, 0x0B, 0x01, 0x20, 0x13, 0x0C, 0x01, 0x20, 0xB9, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00,
+ 0x36, 0x0B, 0x01, 0x20, 0x60, 0x0B, 0x01, 0x20, 0x10, 0x05, 0x10, 0x00, 0xE6, 0x43, 0x01, 0x20,
+ 0xEC, 0x06, 0x10, 0x00, 0xB0, 0xF8, 0x48, 0x12, 0x01, 0x22, 0x0D, 0xB2, 0xB0, 0xF9, 0x44, 0x12,
+ 0x6A, 0x48, 0x00, 0xF0, 0x94, 0xFA, 0x20, 0x68, 0x01, 0x22, 0xB0, 0xF9, 0x46, 0x12, 0x67, 0x48,
+ 0x10, 0x30, 0x00, 0xF0, 0x8C, 0xFA, 0x29, 0x46, 0xBD, 0xE8, 0x70, 0x40, 0x63, 0x48, 0x01, 0x22,
+ 0x10, 0x38, 0x00, 0xF0, 0x84, 0xBA, 0x61, 0x48, 0x4F, 0xF0, 0xFF, 0x31, 0xC0, 0xE9, 0x00, 0x11,
+ 0xC0, 0xE9, 0x02, 0x11, 0x10, 0x38, 0xC0, 0xE9, 0x00, 0x11, 0xC0, 0xE9, 0x02, 0x11, 0x20, 0x30,
+ 0xC0, 0xE9, 0x00, 0x11, 0xC0, 0xE9, 0x02, 0x11, 0x70, 0xBD, 0x2D, 0xE9, 0xFF, 0x4F, 0x83, 0xB0,
+ 0x15, 0x46, 0x06, 0x99, 0x91, 0xF8, 0x40, 0x10, 0x02, 0x29, 0x0C, 0xD0, 0x01, 0x29, 0x0C, 0xD0,
+ 0x52, 0x49, 0x10, 0x39, 0x8A, 0x46, 0x4F, 0xF0, 0x00, 0x0B, 0xA9, 0x78, 0x11, 0xF0, 0x48, 0xFE,
+ 0x81, 0x46, 0xA8, 0x78, 0x32, 0xE0, 0x4D, 0x49, 0xF4, 0xE7, 0x4C, 0x49, 0x10, 0x31, 0xF1, 0xE7,
+ 0x02, 0x46, 0x2C, 0x78, 0x01, 0x20, 0x00, 0x21, 0x11, 0xF0, 0x6D, 0xFC, 0xCD, 0xE9, 0x00, 0x01,
+ 0x1D, 0xE0, 0x00, 0xBF, 0x19, 0xF8, 0x04, 0x10, 0x04, 0x98, 0x81, 0x42, 0x16, 0xD1, 0xDD, 0xE9,
+ 0x00, 0x23, 0xDA, 0xE9, 0x00, 0x01, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x23, 0xD0, 0xDA, 0xE9,
+ 0x02, 0x67, 0x22, 0x46, 0x01, 0x20, 0x00, 0x21, 0x11, 0xF0, 0x55, 0xFC, 0x06, 0x40, 0x0F, 0x40,
+ 0x3E, 0x43, 0x18, 0xD0, 0x0B, 0xF1, 0x01, 0x00, 0x1F, 0xFA, 0x80, 0xFB, 0x64, 0x1C, 0x68, 0x78,
+ 0xA0, 0x42, 0xDF, 0xDA, 0x09, 0xF1, 0x28, 0x09, 0x08, 0xF1, 0x01, 0x00, 0xE9, 0x78, 0x80, 0x46,
+ 0x41, 0x45, 0xCD, 0xDA, 0x06, 0x99, 0x5F, 0xEA, 0x0B, 0x00, 0xA1, 0xF8, 0x26, 0xB0, 0x00, 0xD0,
+ 0x01, 0x20, 0x07, 0xB0, 0xF2, 0xE6, 0x00, 0x20, 0x09, 0xF8, 0x04, 0x00, 0xE6, 0xE7, 0x2D, 0xE9,
+ 0xFC, 0x5F, 0x80, 0x46, 0x4F, 0xF0, 0x00, 0x0B, 0x87, 0x78, 0x90, 0xF8, 0x03, 0xA0, 0x2A, 0xE0,
+ 0x98, 0xF8, 0x00, 0x40, 0x3A, 0x46, 0x01, 0x20, 0x00, 0x21, 0x11, 0xF0, 0x24, 0xFC, 0x24, 0x4E,
+ 0xD6, 0xE9, 0x00, 0x32, 0x03, 0x40, 0x0A, 0x40, 0xCD, 0xE9, 0x00, 0x23, 0xD6, 0xE9, 0x02, 0x56,
+ 0x98, 0xF8, 0x01, 0x90, 0x14, 0xE0, 0x00, 0xBF, 0xDD, 0xE9, 0x00, 0x10, 0x08, 0x43, 0x0E, 0xD0,
+ 0x22, 0x46, 0x01, 0x20, 0x00, 0x21, 0x11, 0xF0, 0x0E, 0xFC, 0x05, 0xEA, 0x00, 0x02, 0x06, 0xEA,
+ 0x01, 0x00, 0x02, 0x43, 0x03, 0xD0, 0x0B, 0xF1, 0x01, 0x00, 0x1F, 0xFA, 0x80, 0xFB, 0x64, 0x1C,
+ 0xA1, 0x45, 0xE9, 0xDA, 0x7F, 0x1C, 0xBA, 0x45, 0xD2, 0xDA, 0xBB, 0xF1, 0x00, 0x0F, 0x02, 0xD0,
+ 0x01, 0x20, 0xBD, 0xE8, 0xFC, 0x9F, 0x00, 0x20, 0xFB, 0xE7, 0x2D, 0xE9, 0xF0, 0x47, 0x91, 0x46,
+ 0x45, 0x1C, 0x0C, 0x46, 0x4F, 0xF0, 0x01, 0x08, 0x0C, 0xE0, 0xD9, 0xE9, 0x00, 0x67, 0x22, 0x46,
+ 0x01, 0x20, 0x00, 0x21, 0x11, 0xF0, 0xE7, 0xFB, 0x06, 0x40, 0x0F, 0x40, 0x3E, 0x43, 0x01, 0xD0,
+ 0x05, 0xF8, 0x04, 0x80, 0x64, 0x1E, 0xF0, 0xD2, 0xBD, 0xE8, 0xF0, 0x87, 0xE8, 0x23, 0x10, 0x00,
+ 0x90, 0x24, 0x10, 0x00, 0xA1, 0x4A, 0x10, 0xB5, 0x13, 0x68, 0x00, 0x21, 0xB3, 0xF8, 0x14, 0x22,
+ 0x82, 0x42, 0x05, 0xDA, 0x93, 0xF8, 0x10, 0x22, 0x52, 0x07, 0x01, 0xD5, 0x02, 0x21, 0x08, 0xE0,
+ 0xB3, 0xF8, 0x12, 0x22, 0x82, 0x42, 0x04, 0xDA, 0x93, 0xF8, 0x10, 0x02, 0x80, 0x07, 0x00, 0xD5,
+ 0x01, 0x21, 0x97, 0x48, 0x97, 0x4C, 0x42, 0x78, 0x91, 0x42, 0x14, 0xD1, 0xA1, 0x78, 0x91, 0x42,
+ 0x0D, 0xD0, 0x95, 0x4B, 0x1B, 0x78, 0x1A, 0x43, 0x94, 0x4B, 0x1B, 0x78, 0x1A, 0x43, 0x07, 0xD0,
+ 0x41, 0x68, 0x49, 0x1E, 0x41, 0x60, 0x00, 0x29, 0x01, 0xDC, 0x40, 0x78, 0xA0, 0x70, 0x10, 0xBD,
+ 0x41, 0x70, 0x41, 0x68, 0xF7, 0xE7, 0x59, 0xB1, 0x02, 0x29, 0x01, 0xD1, 0x01, 0x2A, 0x07, 0xD0,
+ 0x93, 0xF8, 0x18, 0x22, 0x42, 0x60, 0x41, 0x70, 0x00, 0x2A, 0xF0, 0xDC, 0xA1, 0x70, 0x10, 0xBD,
+ 0x93, 0xF8, 0x19, 0x22, 0xF6, 0xE7, 0x70, 0xB5, 0x85, 0x4A, 0x86, 0x4B, 0x81, 0x4D, 0x16, 0x78,
+ 0x7E, 0x4A, 0x9C, 0x79, 0x00, 0x21, 0x12, 0x68, 0xAB, 0x78, 0x01, 0x2E, 0x02, 0xD1, 0x82, 0x4E,
+ 0x36, 0x78, 0x4E, 0xB1, 0xB2, 0xF8, 0x14, 0x62, 0x86, 0x42, 0x17, 0xDA, 0x92, 0xF8, 0x10, 0x62,
+ 0x76, 0x07, 0x13, 0xD5, 0x02, 0x21, 0x25, 0xE0, 0xB2, 0xF8, 0x1C, 0x62, 0x86, 0x42, 0x03, 0xDA,
+ 0x92, 0xF8, 0x10, 0x62, 0x76, 0x07, 0xF5, 0xD4, 0xB2, 0xF8, 0x1A, 0x62, 0x08, 0xE0, 0x00, 0xBF,
+ 0x92, 0xF8, 0x10, 0x02, 0x80, 0x07, 0x05, 0xD5, 0x01, 0x21, 0x13, 0xE0, 0xB2, 0xF8, 0x12, 0x62,
+ 0x86, 0x42, 0xF5, 0xDB, 0x73, 0xB1, 0x6C, 0x48, 0x00, 0x78, 0x01, 0x28, 0x03, 0xD0, 0x6B, 0x48,
+ 0x00, 0x78, 0x01, 0x28, 0x03, 0xD1, 0x92, 0xF8, 0x10, 0x02, 0x00, 0x07, 0x01, 0xD5, 0x01, 0x2C,
+ 0x00, 0xD1, 0x19, 0x46, 0x62, 0x48, 0x34, 0xB1, 0x19, 0xB1, 0x00, 0xBF, 0x92, 0xF8, 0x19, 0x22,
+ 0x42, 0x60, 0xA9, 0x70, 0x70, 0xBD, 0x00, 0x29, 0xF8, 0xD1, 0x00, 0x2B, 0xF6, 0xD0, 0x44, 0x68,
+ 0x00, 0x2C, 0xF6, 0xDD, 0x92, 0xF8, 0x10, 0x12, 0x0A, 0x07, 0x59, 0x49, 0x49, 0x68, 0xA1, 0xF1,
+ 0x01, 0x01, 0x02, 0xD5, 0x5D, 0x4A, 0x12, 0x78, 0x02, 0xB1, 0x41, 0x60, 0x19, 0x46, 0xE8, 0xE7,
+ 0x52, 0x48, 0x53, 0x49, 0x00, 0x68, 0x90, 0xF8, 0x10, 0x02, 0xC2, 0x06, 0x4F, 0xF0, 0x00, 0x00,
+ 0x03, 0xD4, 0x50, 0x4A, 0x90, 0x70, 0x48, 0x70, 0x48, 0x60, 0x08, 0x70, 0x70, 0x47, 0x10, 0xB5,
+ 0x04, 0x46, 0x4A, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x10, 0x02, 0xC0, 0x06, 0x20, 0x46, 0x02, 0xD5,
+ 0xFF, 0xF7, 0x89, 0xFF, 0x01, 0xE0, 0xFF, 0xF7, 0x45, 0xFF, 0x46, 0x48, 0x82, 0x78, 0x4C, 0x48,
+ 0x41, 0x7B, 0x62, 0xF3, 0x07, 0x11, 0x41, 0x73, 0xA0, 0xF8, 0x2F, 0x40, 0x10, 0xBD, 0x2D, 0xE9,
+ 0xFC, 0x5F, 0x83, 0x46, 0x47, 0x48, 0x00, 0x25, 0x0A, 0x46, 0x00, 0x78, 0x2F, 0x46, 0x2C, 0x46,
+ 0x01, 0x28, 0x00, 0xD1, 0x01, 0x24, 0x44, 0x48, 0x06, 0x78, 0x00, 0x78, 0x41, 0x00, 0x01, 0x2A,
+ 0x1B, 0xD0, 0xDF, 0xF8, 0xD8, 0x80, 0xDF, 0xF8, 0x04, 0xA1, 0xDF, 0xF8, 0x04, 0x91, 0xD8, 0xF8,
+ 0x00, 0x00, 0x90, 0xF8, 0x10, 0x22, 0x92, 0x06, 0x14, 0xD5, 0x32, 0x1B, 0xB0, 0xF9, 0x20, 0x12,
+ 0xCD, 0xE9, 0x00, 0x12, 0xB0, 0xF9, 0x1E, 0x32, 0x0A, 0xEB, 0x44, 0x02, 0x0B, 0xEB, 0x44, 0x01,
+ 0x09, 0xEB, 0x44, 0x00, 0x0C, 0xF0, 0x58, 0xFE, 0x09, 0xE0, 0x34, 0x48, 0x4C, 0x38, 0x11, 0xF0,
+ 0x98, 0xFB, 0x41, 0xE0, 0x0A, 0x46, 0x59, 0x46, 0x31, 0x48, 0x11, 0xF0, 0x13, 0xFB, 0x2F, 0x48,
+ 0x4C, 0x38, 0x34, 0xE0, 0x3A, 0xF9, 0x14, 0x20, 0x39, 0xF9, 0x14, 0x10, 0x51, 0x1A, 0x00, 0xD5,
+ 0x49, 0x42, 0xD8, 0xF8, 0x00, 0x30, 0x09, 0xB2, 0x93, 0xF8, 0x10, 0x22, 0x52, 0x06, 0x0B, 0xD5,
+ 0x20, 0x4A, 0x92, 0x79, 0x01, 0x2A, 0x07, 0xD1, 0x26, 0x4A, 0x12, 0x78, 0x22, 0xB9, 0xB3, 0xF8,
+ 0x22, 0x22, 0x8A, 0x42, 0x00, 0xDA, 0x01, 0x27, 0x30, 0xF9, 0x14, 0x20, 0x8A, 0x42, 0x02, 0xDA,
+ 0x93, 0xF8, 0x16, 0x32, 0x01, 0xE0, 0x93, 0xF8, 0x17, 0x32, 0x5A, 0x43, 0xC3, 0xF5, 0x80, 0x73,
+ 0x01, 0xFB, 0x03, 0x21, 0xCA, 0x17, 0x01, 0xEB, 0x12, 0x61, 0x09, 0x12, 0x20, 0xF8, 0x14, 0x10,
+ 0x30, 0xF9, 0x14, 0x10, 0xA9, 0x42, 0x00, 0xDD, 0x0D, 0x46, 0x64, 0x1C, 0x24, 0xB2, 0xB4, 0x42,
+ 0xC8, 0xDB, 0x28, 0x46, 0xFF, 0xF7, 0x73, 0xFF, 0x72, 0x00, 0x59, 0x46, 0x0F, 0x48, 0x11, 0xF0,
+ 0xD1, 0xFA, 0x03, 0x48, 0x07, 0x70, 0xBD, 0xE8, 0xFC, 0x9F, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x04, 0x05, 0x10, 0x00, 0x4C, 0x07, 0x10, 0x00, 0xC1, 0x06, 0x10, 0x00, 0x0C, 0x05, 0x10, 0x00,
+ 0xC2, 0x06, 0x10, 0x00, 0xEC, 0x06, 0x10, 0x00, 0xC3, 0x06, 0x10, 0x00, 0xC6, 0x06, 0x10, 0x00,
+ 0xE6, 0x43, 0x01, 0x20, 0xC0, 0x06, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0x46, 0x0A, 0x01, 0x20,
+ 0x92, 0x0A, 0x01, 0x20, 0x1C, 0x05, 0x10, 0x00, 0x2D, 0xE9, 0xF0, 0x4F, 0x81, 0x46, 0xDD, 0xE9,
+ 0x09, 0xB5, 0x01, 0x20, 0x47, 0xF6, 0xFF, 0x77, 0x9A, 0x46, 0x84, 0x46, 0xFE, 0x43, 0x2C, 0x46,
+ 0x15, 0xE0, 0x00, 0xBF, 0x32, 0xF8, 0x14, 0x30, 0x31, 0xF8, 0x14, 0x80, 0xA3, 0xEB, 0x08, 0x03,
+ 0x1B, 0xB2, 0xBB, 0x42, 0x00, 0xDA, 0x1F, 0x46, 0xB3, 0x42, 0x00, 0xDD, 0x1E, 0x46, 0x00, 0x2B,
+ 0x02, 0xDD, 0x4F, 0xF0, 0x00, 0x0C, 0x02, 0xE0, 0x01, 0xDA, 0x4F, 0xF0, 0x00, 0x00, 0x64, 0x1E,
+ 0xE8, 0xD2, 0x20, 0xB1, 0xBC, 0xF1, 0x00, 0x0F, 0x01, 0xD0, 0x00, 0x23, 0x09, 0xE0, 0x50, 0xEA,
+ 0x0C, 0x03, 0x06, 0xD0, 0x20, 0xB9, 0xBC, 0xF1, 0x00, 0x0F, 0x01, 0xD0, 0x33, 0x46, 0x00, 0xE0,
+ 0x3B, 0x46, 0x00, 0x27, 0x3C, 0x46, 0x28, 0x46, 0xCB, 0xF1, 0x00, 0x0C, 0x0E, 0xE0, 0x00, 0xBF,
+ 0x32, 0xF8, 0x10, 0x60, 0x31, 0xF8, 0x10, 0x80, 0xA6, 0xEB, 0x08, 0x06, 0xF6, 0x1A, 0x36, 0xB2,
+ 0x56, 0x45, 0x03, 0xDA, 0x66, 0x45, 0x01, 0xDD, 0x37, 0x44, 0x64, 0x1C, 0x40, 0x1E, 0xEF, 0xD2,
+ 0x04, 0xB9, 0x01, 0x24, 0x97, 0xFB, 0xF4, 0xF0, 0x18, 0x44, 0x04, 0xE0, 0x31, 0xF8, 0x15, 0x20,
+ 0x02, 0x44, 0x29, 0xF8, 0x15, 0x20, 0x6D, 0x1E, 0xF8, 0xD2, 0xBD, 0xE8, 0xF0, 0x8F, 0x2D, 0xE9,
+ 0xF0, 0x41, 0x04, 0x46, 0x03, 0x46, 0xFE, 0x48, 0x15, 0x46, 0x0E, 0x46, 0x0A, 0x46, 0x01, 0x78,
+ 0xFC, 0x48, 0x00, 0xF0, 0x57, 0xFA, 0xFC, 0x48, 0x04, 0xF1, 0x08, 0x03, 0x1F, 0x46, 0x01, 0x78,
+ 0xF8, 0x48, 0x32, 0x46, 0x2A, 0x30, 0x00, 0xF0, 0x4D, 0xFA, 0x00, 0x2D, 0x07, 0xD0, 0x20, 0x46,
+ 0x11, 0xF0, 0xB0, 0xFB, 0x38, 0x46, 0xBD, 0xE8, 0xF0, 0x41, 0x11, 0xF0, 0xAB, 0xBB, 0xBD, 0xE8,
+ 0xF0, 0x81, 0x2D, 0xE9, 0xFC, 0x41, 0x07, 0x46, 0x14, 0x46, 0x88, 0x46, 0x03, 0x20, 0x0F, 0xF0,
+ 0x3A, 0xFA, 0xEE, 0x4D, 0xEA, 0x4E, 0x00, 0x94, 0xEA, 0x48, 0x32, 0x78, 0x39, 0x46, 0x76, 0x38,
+ 0x2B, 0x68, 0x0C, 0xF0, 0xEF, 0xFD, 0x00, 0x94, 0xE7, 0x4C, 0xE6, 0x48, 0x41, 0x46, 0x22, 0x78,
+ 0x4C, 0x38, 0x2B, 0x68, 0x0C, 0xF0, 0xE6, 0xFD, 0xE5, 0x4D, 0x28, 0x68, 0x90, 0xF8, 0x30, 0x12,
+ 0xC9, 0x07, 0x1A, 0xD0, 0x32, 0x78, 0xB0, 0xF9, 0x34, 0x12, 0xCD, 0xE9, 0x00, 0x12, 0xDD, 0x49,
+ 0xB0, 0xF9, 0x32, 0x32, 0x76, 0x39, 0xDF, 0x4A, 0x08, 0x46, 0xFF, 0xF7, 0x55, 0xFF, 0x28, 0x68,
+ 0x22, 0x78, 0xB0, 0xF9, 0x38, 0x12, 0xCD, 0xE9, 0x00, 0x12, 0xD6, 0x49, 0xB0, 0xF9, 0x36, 0x32,
+ 0x4C, 0x39, 0xD9, 0x4A, 0x08, 0x46, 0xFF, 0xF7, 0x47, 0xFF, 0xD2, 0x49, 0x33, 0x78, 0xD5, 0x4A,
+ 0x76, 0x39, 0xD0, 0x48, 0x0C, 0xF0, 0xA1, 0xFD, 0xCE, 0x49, 0x23, 0x78, 0x4C, 0x39, 0xD2, 0x4A,
+ 0x01, 0xF1, 0x76, 0x00, 0x0C, 0xF0, 0x99, 0xFD, 0x28, 0x68, 0x90, 0xF8, 0x30, 0x12, 0x8A, 0x07,
+ 0x04, 0xD4, 0xCE, 0x4A, 0x12, 0x78, 0x8A, 0xB1, 0x49, 0x07, 0x0F, 0xD5, 0xC5, 0x49, 0x90, 0xF8,
+ 0x41, 0x32, 0x32, 0x78, 0x08, 0x46, 0x0C, 0xF0, 0xD8, 0xFD, 0x28, 0x68, 0xC1, 0x49, 0x22, 0x78,
+ 0x90, 0xF8, 0x41, 0x32, 0x2A, 0x31, 0x08, 0x46, 0x0C, 0xF0, 0xCF, 0xFD, 0xBD, 0xE8, 0xFC, 0x41,
+ 0x03, 0x20, 0x0F, 0xF0, 0xF3, 0xB9, 0x70, 0xB5, 0xC1, 0x4C, 0xC2, 0x49, 0x00, 0x25, 0xA5, 0x70,
+ 0x01, 0x20, 0x08, 0x70, 0xBA, 0x48, 0x25, 0x70, 0x65, 0x70, 0x00, 0x68, 0xB0, 0xF8, 0x3A, 0x02,
+ 0xA0, 0x80, 0x00, 0xF0, 0xE9, 0xF9, 0xFF, 0xF7, 0x6C, 0xFA, 0xA5, 0x60, 0xE5, 0x60, 0x70, 0xBD,
+ 0x2D, 0xE9, 0xF0, 0x41, 0x0E, 0x46, 0xB7, 0x49, 0x07, 0x46, 0xB5, 0x4C, 0x0D, 0x78, 0x00, 0x20,
+ 0x08, 0x70, 0x60, 0x70, 0x00, 0xF0, 0xD8, 0xF9, 0xB3, 0x48, 0x2A, 0x46, 0x31, 0x46, 0x00, 0x78,
+ 0x00, 0x28, 0x38, 0x46, 0x02, 0xD0, 0xFF, 0xF7, 0xC3, 0xF9, 0x01, 0xE0, 0xFF, 0xF7, 0x69, 0xFF,
+ 0xAE, 0x48, 0x41, 0x78, 0xA6, 0x48, 0xA9, 0xB1, 0xA1, 0x78, 0x01, 0x29, 0x12, 0xD0, 0x00, 0x68,
+ 0xB0, 0xF9, 0x3C, 0x12, 0xAA, 0x4D, 0xA1, 0x80, 0x01, 0x22, 0x28, 0x46, 0xFF, 0xF7, 0x37, 0xFF,
+ 0xD5, 0xE9, 0x00, 0x01, 0x08, 0x43, 0x09, 0xD0, 0xD5, 0xE9, 0x02, 0x01, 0x08, 0x43, 0x05, 0xD0,
+ 0x01, 0x20, 0x04, 0xE0, 0x00, 0x68, 0xB0, 0xF9, 0x3A, 0x12, 0xEB, 0xE7, 0x00, 0x20, 0x20, 0x70,
+ 0x45, 0xE7, 0x2D, 0xE9, 0xF0, 0x4F, 0x04, 0x46, 0x9E, 0x48, 0xA7, 0xB0, 0x4F, 0xF0, 0x01, 0x0A,
+ 0x01, 0x78, 0x93, 0x48, 0x01, 0x29, 0x00, 0x68, 0x18, 0xD0, 0x94, 0x49, 0x09, 0x78, 0xC1, 0xB1,
+ 0xB0, 0xF9, 0x54, 0x62, 0x8A, 0x48, 0x01, 0x21, 0x00, 0x27, 0x00, 0x78, 0x80, 0x1E, 0x00, 0xF0,
+ 0xFF, 0x09, 0x89, 0x48, 0x03, 0x78, 0x5A, 0x1E, 0x02, 0xF0, 0xFF, 0x08, 0x92, 0x4A, 0x12, 0x78,
+ 0x02, 0xB1, 0x01, 0x27, 0x91, 0x4D, 0x22, 0x46, 0x83, 0x46, 0x12, 0xE0, 0xB0, 0xF9, 0x59, 0x62,
+ 0xE8, 0xE7, 0xB0, 0xF9, 0x52, 0x62, 0xE5, 0xE7, 0x38, 0x46, 0x05, 0xE0, 0x32, 0xF9, 0x10, 0xC0,
+ 0xAC, 0x45, 0x00, 0xDD, 0x65, 0x46, 0x40, 0x1C, 0x40, 0x45, 0xF7, 0xDD, 0x02, 0xEB, 0x43, 0x02,
+ 0x49, 0x1C, 0x49, 0x45, 0xF0, 0xDD, 0x5F, 0x46, 0x5A, 0x00, 0x04, 0xEB, 0x43, 0x01, 0x68, 0x46,
+ 0x11, 0xF0, 0x38, 0xF9, 0x82, 0x4A, 0x38, 0x78, 0x6B, 0x46, 0x12, 0x88, 0x41, 0x42, 0x02, 0xEB,
+ 0x41, 0x01, 0x09, 0xB2, 0x42, 0x00, 0x04, 0xEB, 0x41, 0x01, 0x03, 0xEB, 0x40, 0x00, 0x11, 0xF0,
+ 0x29, 0xF9, 0x38, 0x78, 0x69, 0x46, 0x42, 0x00, 0x08, 0x46, 0x0C, 0xF0, 0x44, 0xFD, 0x28, 0x1A,
+ 0x00, 0xB2, 0xB0, 0x42, 0x01, 0xDD, 0x4F, 0xF0, 0x00, 0x0A, 0x50, 0x46, 0x27, 0xB0, 0xC4, 0xE6,
+ 0x2D, 0xE9, 0xF0, 0x4F, 0x73, 0x49, 0x72, 0x4A, 0x00, 0x27, 0x0C, 0x68, 0xB2, 0xF9, 0x00, 0x20,
+ 0x4F, 0xF0, 0x01, 0x0B, 0x21, 0x8E, 0x3B, 0x46, 0x91, 0x42, 0x4F, 0xD1, 0x63, 0x49, 0x0A, 0x78,
+ 0x5F, 0x49, 0x0E, 0x68, 0xB2, 0xB1, 0xB6, 0xF9, 0x54, 0x12, 0x02, 0x25, 0x89, 0x46, 0x58, 0x49,
+ 0x4F, 0xF0, 0x00, 0x0A, 0x09, 0x78, 0xC9, 0x1E, 0x01, 0xF0, 0xFF, 0x0E, 0x56, 0x49, 0x09, 0x78,
+ 0x49, 0x1E, 0x01, 0xF0, 0xFF, 0x0C, 0x60, 0x49, 0x09, 0x78, 0x09, 0xB1, 0x4F, 0xF0, 0x01, 0x0A,
+ 0x32, 0x34, 0x2B, 0xE0, 0xB6, 0xF9, 0x52, 0x12, 0xE7, 0xE7, 0x52, 0x46, 0x1D, 0xE0, 0x00, 0xBF,
+ 0x34, 0xF8, 0x12, 0x10, 0x30, 0xF8, 0x12, 0x80, 0xA1, 0xEB, 0x08, 0x01, 0x09, 0xB2, 0x00, 0x29,
+ 0x01, 0xDB, 0x88, 0x46, 0x01, 0xE0, 0xC1, 0xF1, 0x00, 0x08, 0xB8, 0x45, 0x05, 0xDD, 0x00, 0x29,
+ 0x01, 0xDB, 0x0F, 0x46, 0x00, 0xE0, 0x4F, 0x42, 0x3F, 0xB2, 0x00, 0x29, 0x00, 0xDA, 0x49, 0x42,
+ 0x49, 0x45, 0x01, 0xDD, 0x5B, 0x1C, 0x9B, 0xB2, 0x52, 0x1C, 0x62, 0x45, 0xE0, 0xDD, 0x96, 0xF8,
+ 0x30, 0x10, 0x6D, 0x1C, 0x04, 0xEB, 0x41, 0x04, 0x00, 0xEB, 0x41, 0x00, 0x75, 0x45, 0xD4, 0xDD,
+ 0x96, 0xF8, 0x56, 0x02, 0x98, 0x42, 0x01, 0xD2, 0x4F, 0xF0, 0x00, 0x0B, 0x46, 0x48, 0x20, 0xF8,
+ 0x33, 0x7F, 0x83, 0x70, 0x58, 0x46, 0x60, 0xE6, 0x2D, 0xE9, 0xF0, 0x41, 0x06, 0x46, 0x43, 0x48,
+ 0x00, 0x24, 0x00, 0x78, 0x00, 0x28, 0x5A, 0xD0, 0x3A, 0x48, 0x34, 0x4D, 0x00, 0x78, 0x88, 0xB1,
+ 0x3F, 0x48, 0x2F, 0x4B, 0x32, 0x49, 0x02, 0x68, 0x18, 0x68, 0x08, 0x31, 0xB0, 0xF8, 0x5D, 0x02,
+ 0x40, 0x42, 0x07, 0xB2, 0x00, 0x20, 0xC8, 0x80, 0x88, 0x80, 0x08, 0x60, 0x34, 0x48, 0xB0, 0xF9,
+ 0x00, 0x00, 0x25, 0xE0, 0x2C, 0x78, 0x01, 0x29, 0x01, 0xD0, 0x11, 0xB1, 0x3B, 0xE0, 0x01, 0x24,
+ 0x3A, 0xE0, 0x00, 0x24, 0x69, 0xE0, 0x00, 0xBF, 0x32, 0xF9, 0x10, 0xC0, 0xBC, 0xF1, 0x00, 0x0F,
+ 0x16, 0xDA, 0xB1, 0xF8, 0x04, 0xC0, 0x0C, 0xF1, 0x01, 0x0C, 0xA1, 0xF8, 0x04, 0xC0, 0x32, 0xF9,
+ 0x10, 0xE0, 0xD1, 0xF8, 0x00, 0xC0, 0xF4, 0x44, 0xC1, 0xF8, 0x00, 0xC0, 0x32, 0xF9, 0x10, 0xC0,
+ 0xBC, 0x45, 0x05, 0xDA, 0xB1, 0xF8, 0x06, 0xC0, 0x0C, 0xF1, 0x01, 0x0C, 0xA1, 0xF8, 0x06, 0xC0,
+ 0x40, 0x1E, 0xE1, 0xD2, 0x08, 0x68, 0x40, 0x42, 0x08, 0x60, 0x15, 0x48, 0x19, 0x68, 0x08, 0x30,
+ 0x02, 0x68, 0xB1, 0xF8, 0x5F, 0x32, 0x9A, 0x42, 0x3F, 0xDD, 0xB0, 0xF9, 0x06, 0x00, 0x91, 0xF8,
+ 0x61, 0x12, 0x88, 0x42, 0x39, 0xDD, 0x1B, 0x48, 0x11, 0x4A, 0xB0, 0xF9, 0x00, 0x10, 0x30, 0x46,
+ 0x00, 0xF0, 0x1E, 0xF9, 0x04, 0x46, 0x7C, 0xB3, 0x00, 0xF0, 0xF0, 0xF8, 0x2F, 0xE0, 0x3F, 0xE0,
+ 0xB9, 0x06, 0x10, 0x00, 0x36, 0x0B, 0x01, 0x20, 0xBA, 0x06, 0x10, 0x00, 0x78, 0x06, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0x6A, 0x4A, 0x01, 0x20, 0x94, 0x4A, 0x01, 0x20, 0x1C, 0x05, 0x10, 0x00,
+ 0x0C, 0x05, 0x10, 0x00, 0x51, 0x07, 0x10, 0x00, 0xC0, 0x06, 0x10, 0x00, 0x4C, 0x07, 0x10, 0x00,
+ 0x80, 0x24, 0x10, 0x00, 0x21, 0x05, 0x10, 0x00, 0xBF, 0x06, 0x10, 0x00, 0x00, 0x80, 0xFF, 0xFF,
+ 0xCA, 0x06, 0x10, 0x00, 0x24, 0x07, 0x10, 0x00, 0xE6, 0x43, 0x01, 0x20, 0xEC, 0x06, 0x10, 0x00,
+ 0xE0, 0x06, 0x10, 0x00, 0x74, 0x06, 0x10, 0x00, 0xFF, 0xE7, 0x00, 0xF0, 0xD5, 0xF8, 0x50, 0x4E,
+ 0x28, 0x78, 0x31, 0x78, 0x88, 0x42, 0x06, 0xD0, 0x08, 0xB1, 0x01, 0x21, 0x00, 0xE0, 0x00, 0x21,
+ 0x06, 0x20, 0x0B, 0xF0, 0x8F, 0xF8, 0x28, 0x78, 0x30, 0x70, 0x0C, 0xB1, 0x01, 0x20, 0xDE, 0xE5,
+ 0x00, 0x20, 0xDC, 0xE5, 0x70, 0xB5, 0x03, 0x20, 0x0F, 0xF0, 0x1D, 0xF8, 0x00, 0xF0, 0x48, 0xF8,
+ 0x04, 0x46, 0x03, 0x20, 0x0F, 0xF0, 0x32, 0xF8, 0x42, 0x49, 0xC8, 0x7B, 0x64, 0xF3, 0x05, 0x10,
+ 0xC8, 0x73, 0x41, 0x48, 0x01, 0x78, 0x21, 0xB1, 0x00, 0x21, 0x01, 0x70, 0x20, 0x46, 0x00, 0xF0,
+ 0x35, 0xF9, 0x3E, 0x4D, 0x3E, 0x48, 0x21, 0x46, 0x2A, 0x78, 0x00, 0x68, 0xFF, 0xF7, 0x3C, 0xFF,
+ 0xA8, 0x70, 0x70, 0xBD, 0x2D, 0xE9, 0xF0, 0x47, 0x1D, 0x46, 0x4F, 0xF0, 0x00, 0x08, 0xC5, 0xE9,
+ 0x00, 0x88, 0x91, 0x46, 0x82, 0x46, 0x0C, 0x46, 0x11, 0xE0, 0x00, 0xBF, 0x3A, 0xF9, 0x14, 0x10,
+ 0x49, 0x45, 0x0C, 0xDD, 0xD5, 0xE9, 0x00, 0x67, 0x4F, 0xF0, 0x01, 0x08, 0x22, 0x46, 0x40, 0x46,
+ 0x00, 0x21, 0x10, 0xF0, 0xA0, 0xFF, 0x06, 0x43, 0x0F, 0x43, 0xC5, 0xE9, 0x00, 0x67, 0x64, 0x1E,
+ 0xEC, 0xD2, 0x40, 0x46, 0xBD, 0xE8, 0xF0, 0x87, 0x2A, 0x48, 0x00, 0x21, 0xC0, 0xE9, 0x00, 0x11,
+ 0xC0, 0xE9, 0x02, 0x11, 0x08, 0x46, 0x28, 0x49, 0x08, 0x76, 0x28, 0x49, 0x08, 0x76, 0x70, 0x47,
+ 0x2D, 0xE9, 0xF0, 0x41, 0x26, 0x4C, 0x20, 0x78, 0x02, 0x28, 0x33, 0xD9, 0x25, 0x4E, 0x30, 0x68,
+ 0x90, 0xF8, 0x50, 0x02, 0x41, 0x07, 0x24, 0x48, 0x00, 0x68, 0x02, 0xD5, 0x23, 0x49, 0x09, 0x78,
+ 0x79, 0xB1, 0xFF, 0xF7, 0x36, 0xFE, 0x05, 0x46, 0x30, 0x68, 0x90, 0xF8, 0x50, 0x02, 0x00, 0x07,
+ 0x1F, 0xD5, 0x22, 0x78, 0x1E, 0x4B, 0x1F, 0x49, 0x02, 0x2A, 0x1A, 0xDB, 0x1E, 0x4C, 0x50, 0x1E,
+ 0x07, 0xE0, 0xFF, 0xF7, 0x85, 0xFE, 0xEE, 0xE7, 0x33, 0xF9, 0x10, 0x70, 0xA7, 0x42, 0x00, 0xDD,
+ 0x3C, 0x46, 0x40, 0x1E, 0xF8, 0xD1, 0x98, 0x1C, 0x92, 0x1E, 0x0C, 0xF0, 0xB4, 0xFB, 0x20, 0x1A,
+ 0x01, 0xB2, 0x30, 0x68, 0xB0, 0xF9, 0x57, 0x02, 0x81, 0x42, 0x02, 0xDD, 0x35, 0xB1, 0x02, 0x20,
+ 0x55, 0xE5, 0x0D, 0xB1, 0x00, 0x20, 0x52, 0xE5, 0x01, 0x20, 0x50, 0xE5, 0x03, 0x20, 0x4E, 0xE5,
+ 0x1D, 0x05, 0x10, 0x00, 0xE6, 0x43, 0x01, 0x20, 0x20, 0x05, 0x10, 0x00, 0x0C, 0x05, 0x10, 0x00,
+ 0xE0, 0x06, 0x10, 0x00, 0x80, 0x24, 0x10, 0x00, 0x58, 0x23, 0x10, 0x00, 0x98, 0x23, 0x10, 0x00,
+ 0xB9, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0xE8, 0x06, 0x10, 0x00, 0x21, 0x05, 0x10, 0x00,
+ 0x6A, 0x4A, 0x01, 0x20, 0x92, 0x0A, 0x01, 0x20, 0x00, 0x80, 0xFF, 0xFF, 0x53, 0x48, 0x00, 0x21,
+ 0xC1, 0x70, 0x02, 0x78, 0x00, 0x2A, 0x0E, 0xD1, 0x82, 0x78, 0x51, 0x4B, 0x52, 0x1C, 0xD2, 0xB2,
+ 0x82, 0x70, 0x1B, 0x68, 0x93, 0xF8, 0x62, 0x32, 0x03, 0xF0, 0x0F, 0x03, 0x93, 0x42, 0x02, 0xD2,
+ 0x01, 0x22, 0x02, 0x70, 0x81, 0x70, 0x70, 0x47, 0x48, 0x48, 0x00, 0x21, 0x81, 0x70, 0x02, 0x78,
+ 0x00, 0x2A, 0x0C, 0xD0, 0xC2, 0x78, 0x46, 0x4B, 0x52, 0x1C, 0xD2, 0xB2, 0xC2, 0x70, 0x1B, 0x68,
+ 0x93, 0xF8, 0x62, 0x32, 0xB2, 0xEB, 0x13, 0x1F, 0x01, 0xD9, 0x01, 0x70, 0xC1, 0x70, 0x70, 0x47,
+ 0x2D, 0xE9, 0xF7, 0x4F, 0x86, 0xB0, 0x8B, 0x46, 0xD2, 0xE9, 0x00, 0x01, 0xCD, 0xE9, 0x02, 0x01,
+ 0xD2, 0xE9, 0x02, 0x01, 0xCD, 0xE9, 0x04, 0x01, 0x02, 0xA8, 0x11, 0xF0, 0xA3, 0xF8, 0x04, 0xA8,
+ 0x11, 0xF0, 0xA0, 0xF8, 0x37, 0x48, 0x00, 0x27, 0xB9, 0x46, 0x06, 0x78, 0x31, 0xE0, 0xF1, 0xB2,
+ 0x06, 0x98, 0x11, 0xF0, 0xAD, 0xF8, 0x82, 0x46, 0x33, 0x48, 0x32, 0x46, 0x00, 0x21, 0x05, 0x78,
+ 0x01, 0x20, 0x10, 0xF0, 0xE0, 0xFE, 0x02, 0x9A, 0x00, 0xEA, 0x02, 0x04, 0x03, 0x98, 0x01, 0xEA,
+ 0x00, 0x08, 0xDD, 0xE9, 0x04, 0x01, 0xCD, 0xE9, 0x00, 0x01, 0x18, 0xE0, 0x3A, 0xF9, 0x15, 0x10,
+ 0x59, 0x45, 0x14, 0xDD, 0x54, 0xEA, 0x08, 0x00, 0x0F, 0xD0, 0x2A, 0x46, 0x01, 0x20, 0x00, 0x21,
+ 0x10, 0xF0, 0xC9, 0xFE, 0xDD, 0xE9, 0x00, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x04, 0xD0,
+ 0x09, 0xF1, 0x01, 0x00, 0x1F, 0xFA, 0x80, 0xF9, 0x01, 0xE0, 0x7F, 0x1C, 0xBF, 0xB2, 0x6D, 0x1E,
+ 0xE4, 0xD2, 0x76, 0x1E, 0xCB, 0xD2, 0x1A, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x5B, 0x12, 0xB9, 0x42,
+ 0x07, 0xD8, 0x90, 0xF8, 0x5C, 0x12, 0x49, 0x45, 0x03, 0xD3, 0x01, 0x20, 0x09, 0xB0, 0xBD, 0xE8,
+ 0xF0, 0x8F, 0x00, 0x20, 0xFA, 0xE7, 0x11, 0x48, 0x10, 0xB5, 0x41, 0x78, 0x01, 0x70, 0x00, 0x21,
+ 0x81, 0x70, 0xC1, 0x70, 0xFF, 0xF7, 0x0F, 0xFD, 0xBD, 0xE8, 0x10, 0x40, 0xFE, 0xF7, 0xCC, 0xBF,
+ 0x0A, 0x48, 0x00, 0x21, 0x0A, 0x4A, 0x01, 0x70, 0x41, 0x70, 0x12, 0x68, 0x92, 0xF8, 0x50, 0x22,
+ 0x92, 0x07, 0x00, 0xD5, 0x01, 0x21, 0x41, 0x71, 0x01, 0x71, 0x70, 0x47, 0x03, 0x49, 0x4A, 0x79,
+ 0x01, 0x2A, 0x02, 0xD1, 0x00, 0x28, 0x00, 0xD1, 0x48, 0x71, 0x70, 0x47, 0x1C, 0x05, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0x2D, 0xE9, 0xFF, 0x4F,
+ 0xCE, 0x4B, 0x83, 0xB0, 0x1B, 0x68, 0xB3, 0xF9, 0xD2, 0x42, 0xB3, 0xF9, 0xD4, 0xE2, 0x99, 0xE0,
+ 0x47, 0xF6, 0xFF, 0x76, 0xF7, 0x43, 0x06, 0x9D, 0x08, 0xE0, 0x00, 0xBF, 0x31, 0xF9, 0x15, 0x30,
+ 0xB3, 0x42, 0x00, 0xDA, 0x1E, 0x46, 0xBB, 0x42, 0x00, 0xDD, 0x1F, 0x46, 0x6D, 0x1E, 0xF5, 0xD1,
+ 0xAB, 0x46, 0x00, 0x2F, 0x01, 0xDA, 0xBB, 0x46, 0x02, 0xE0, 0x00, 0x2E, 0x00, 0xDD, 0xB3, 0x46,
+ 0x00, 0x26, 0x37, 0x46, 0xB2, 0x46, 0x35, 0x46, 0xB1, 0x46, 0x06, 0x9B, 0x13, 0xE0, 0x00, 0xBF,
+ 0x31, 0xF9, 0x13, 0x80, 0xA8, 0xEB, 0x0B, 0x0C, 0xA4, 0x45, 0x0C, 0xDA, 0x0C, 0xEB, 0x04, 0x08,
+ 0xB8, 0xF1, 0x00, 0x0F, 0x07, 0xDD, 0x03, 0xFB, 0x03, 0x77, 0x03, 0xFB, 0x0C, 0x55, 0x1E, 0x44,
+ 0xE2, 0x44, 0x09, 0xF1, 0x01, 0x09, 0x5B, 0x1E, 0xEA, 0xD1, 0x07, 0xFB, 0x09, 0xF3, 0x06, 0xFB,
+ 0x16, 0x33, 0x02, 0x93, 0x05, 0xFB, 0x09, 0xF3, 0x06, 0xFB, 0x1A, 0x33, 0x07, 0xFB, 0x0A, 0xF7,
+ 0x01, 0x93, 0x05, 0xFB, 0x16, 0x73, 0x4F, 0xF0, 0x00, 0x0C, 0x00, 0x93, 0x67, 0x46, 0x66, 0x46,
+ 0xE2, 0x46, 0x06, 0x9B, 0x28, 0xE0, 0xDD, 0xF8, 0x08, 0x80, 0x00, 0x25, 0xB8, 0xF1, 0x00, 0x0F,
+ 0x0D, 0xD0, 0xDD, 0xE9, 0x00, 0x58, 0x08, 0xFB, 0x03, 0x58, 0x02, 0x9D, 0x98, 0xFB, 0xF5, 0xF5,
+ 0xA5, 0x42, 0x01, 0xDD, 0x25, 0x46, 0x02, 0xE0, 0xE5, 0x42, 0x00, 0xD5, 0x65, 0x42, 0x31, 0xF8,
+ 0x13, 0x80, 0x5D, 0x44, 0xA8, 0xEB, 0x05, 0x05, 0x2D, 0xB2, 0x7F, 0x1C, 0xA9, 0x46, 0xAC, 0x44,
+ 0x75, 0x45, 0x07, 0xDA, 0x05, 0xEB, 0x0E, 0x08, 0xB8, 0xF1, 0x00, 0x0F, 0x02, 0xDD, 0x4E, 0x44,
+ 0x0A, 0xF1, 0x01, 0x0A, 0x20, 0xF8, 0x13, 0x90, 0x5B, 0x1E, 0xD4, 0xD1, 0x9C, 0xFB, 0xF7, 0xF5,
+ 0xBA, 0xF1, 0x00, 0x0F, 0x01, 0xD1, 0x4F, 0xF0, 0x01, 0x0A, 0x96, 0xFB, 0xFA, 0xF3, 0x00, 0x2D,
+ 0x01, 0xDB, 0x2F, 0x46, 0x00, 0xE0, 0x6F, 0x42, 0x1E, 0x1E, 0x00, 0xDA, 0x5E, 0x42, 0xB7, 0xEB,
+ 0x46, 0x0F, 0x00, 0xDD, 0x1D, 0x46, 0x06, 0x9B, 0x05, 0xE0, 0x00, 0xBF, 0x30, 0xF8, 0x13, 0x60,
+ 0x76, 0x1B, 0x20, 0xF8, 0x13, 0x60, 0x5B, 0x1E, 0xF8, 0xD1, 0x06, 0x9B, 0x01, 0xEB, 0x43, 0x01,
+ 0x00, 0xEB, 0x43, 0x00, 0x52, 0x1E, 0xBF, 0xF4, 0x63, 0xAF, 0x07, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F,
+ 0x2D, 0xE9, 0xF7, 0x4F, 0x79, 0x4D, 0x7A, 0x4C, 0x7A, 0x4F, 0x28, 0x68, 0x0E, 0x46, 0x90, 0xF8,
+ 0xD0, 0x02, 0xC0, 0x07, 0x00, 0xD0, 0x3C, 0x68, 0x03, 0x20, 0x0E, 0xF0, 0x0C, 0xFE, 0x77, 0x48,
+ 0xDF, 0xF8, 0xD4, 0xB1, 0x31, 0x46, 0x02, 0x68, 0xBB, 0xF9, 0x00, 0x30, 0x20, 0x46, 0x0C, 0xF0,
+ 0xA4, 0xF9, 0x03, 0x20, 0x0E, 0xF0, 0x1A, 0xFE, 0x28, 0x68, 0xDF, 0xF8, 0xC4, 0xA1, 0x71, 0x4E,
+ 0x90, 0xF8, 0xD0, 0x02, 0xC0, 0x07, 0x06, 0xD0, 0x9A, 0xF8, 0x00, 0x30, 0x32, 0x78, 0x21, 0x46,
+ 0x67, 0x48, 0xFF, 0xF7, 0x2B, 0xFF, 0xB9, 0x46, 0x65, 0x4A, 0x33, 0x78, 0x4F, 0xF0, 0x00, 0x08,
+ 0x54, 0x46, 0x38, 0x68, 0x16, 0xE0, 0x00, 0xBF, 0xA0, 0xF8, 0x00, 0x80, 0x01, 0x21, 0x09, 0xE0,
+ 0x00, 0xEB, 0x41, 0x07, 0x32, 0xF8, 0x11, 0xC0, 0x37, 0xF8, 0x02, 0x7C, 0x67, 0x44, 0x20, 0xF8,
+ 0x11, 0x70, 0x49, 0x1C, 0x27, 0x78, 0xB9, 0x42, 0xF2, 0xDB, 0xF9, 0xB2, 0x02, 0xEB, 0x41, 0x02,
+ 0x00, 0xEB, 0x41, 0x00, 0x5B, 0x1E, 0xE7, 0xD2, 0x5B, 0x48, 0x5C, 0x4F, 0x02, 0x9B, 0x02, 0x68,
+ 0xD9, 0xF8, 0x00, 0x10, 0x38, 0x68, 0x00, 0xF0, 0x66, 0xFA, 0x28, 0x68, 0x90, 0xF8, 0xD0, 0x02,
+ 0x80, 0x07, 0x06, 0xD5, 0x23, 0x78, 0x32, 0x78, 0x39, 0x68, 0x00, 0x98, 0x00, 0xF0, 0x1C, 0xF8,
+ 0x06, 0xE0, 0xBB, 0xF9, 0x00, 0x00, 0x39, 0x68, 0x42, 0x00, 0x00, 0x98, 0x10, 0xF0, 0xA2, 0xFD,
+ 0x28, 0x68, 0x90, 0xF8, 0xD0, 0x02, 0x00, 0x07, 0x08, 0xD5, 0x00, 0x99, 0x23, 0x78, 0x32, 0x78,
+ 0x03, 0xB0, 0x08, 0x46, 0xBD, 0xE8, 0xF0, 0x4F, 0x00, 0xF0, 0x03, 0xBC, 0xBD, 0xE8, 0xFE, 0x8F,
+ 0x47, 0x49, 0x00, 0x20, 0x08, 0x70, 0x70, 0x47, 0x2D, 0xE9, 0xF0, 0x4F, 0xA7, 0xB0, 0x1C, 0x46,
+ 0x0D, 0x46, 0x06, 0x46, 0x17, 0x46, 0xEA, 0x46, 0x0D, 0xF1, 0x4C, 0x0B, 0x3C, 0xE0, 0x00, 0x22,
+ 0x60, 0x1E, 0x00, 0xF0, 0xFF, 0x09, 0x11, 0x46, 0x10, 0x46, 0x11, 0xE0, 0x35, 0xF9, 0x10, 0x30,
+ 0x1A, 0x44, 0x43, 0x1C, 0x92, 0xFB, 0xF3, 0xFC, 0x2A, 0xF8, 0x10, 0xC0, 0xA9, 0xEB, 0x00, 0x0C,
+ 0x40, 0x1C, 0x35, 0xF9, 0x1C, 0x80, 0x41, 0x44, 0x91, 0xFB, 0xF3, 0xF3, 0x2B, 0xF8, 0x1C, 0x30,
+ 0xA0, 0x42, 0xEB, 0xDB, 0x20, 0x46, 0x52, 0x46, 0x5B, 0x46, 0x0C, 0xE0, 0x32, 0xF9, 0x10, 0x10,
+ 0x33, 0xF9, 0x10, 0xC0, 0x61, 0x44, 0x35, 0xF8, 0x10, 0xC0, 0x01, 0xEB, 0xD1, 0x71, 0xAC, 0xEB,
+ 0x61, 0x01, 0x26, 0xF8, 0x10, 0x10, 0x40, 0x1E, 0xF0, 0xD2, 0x20, 0x48, 0x00, 0x68, 0x90, 0xF8,
+ 0xD0, 0x02, 0x40, 0x07, 0x04, 0xD5, 0x31, 0x46, 0x22, 0x46, 0x08, 0x46, 0x00, 0xF0, 0x08, 0xF8,
+ 0x05, 0xEB, 0x44, 0x05, 0x06, 0xEB, 0x44, 0x06, 0x7F, 0x1E, 0xC0, 0xD2, 0x27, 0xB0, 0x35, 0xE7,
+ 0xF0, 0xB5, 0x16, 0x4B, 0x00, 0x26, 0x35, 0x46, 0xD3, 0xF8, 0x00, 0xE0, 0x34, 0x46, 0x13, 0x46,
+ 0xBE, 0xF9, 0xD6, 0x72, 0xC7, 0xF1, 0x00, 0x0C, 0x06, 0xE0, 0x00, 0xBF, 0x31, 0xF9, 0x13, 0x70,
+ 0x67, 0x45, 0x01, 0xDA, 0x3E, 0x44, 0x6D, 0x1C, 0x5B, 0x1E, 0xF7, 0xD2, 0x00, 0x2D, 0x01, 0xDD,
+ 0x96, 0xFB, 0xF5, 0xF4, 0xBE, 0xF8, 0xD8, 0x32, 0x5E, 0x42, 0xA6, 0x42, 0x00, 0xDD, 0x1C, 0x44,
+ 0x53, 0x1B, 0x63, 0x43, 0x93, 0xFB, 0xF2, 0xF3, 0x05, 0xE0, 0x00, 0xBF, 0x31, 0xF8, 0x12, 0x40,
+ 0xE4, 0x1A, 0x20, 0xF8, 0x12, 0x40, 0x52, 0x1E, 0xF8, 0xD2, 0xF0, 0xBD, 0x20, 0x07, 0x10, 0x00,
+ 0x3C, 0x0C, 0x01, 0x20, 0x2C, 0x05, 0x10, 0x00, 0xCA, 0x06, 0x10, 0x00, 0xE8, 0x06, 0x10, 0x00,
+ 0xBA, 0x06, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0x78, 0x06, 0x10, 0x00, 0xE4, 0x06, 0x10, 0x00,
+ 0xBF, 0x06, 0x10, 0x00, 0x2D, 0xE9, 0xF8, 0x4F, 0x1E, 0x46, 0x15, 0x46, 0x88, 0x46, 0x83, 0x46,
+ 0x00, 0x24, 0x4F, 0xF0, 0xAA, 0x09, 0x0A, 0x9F, 0x4F, 0xE0, 0x30, 0x5D, 0x08, 0xB9, 0x28, 0x5D,
+ 0xF0, 0xB3, 0x06, 0xF8, 0x04, 0x90, 0x38, 0xF9, 0x14, 0x00, 0x00, 0xF0, 0xCE, 0xF8, 0x28, 0xF8,
+ 0x14, 0x00, 0x28, 0x5D, 0xA0, 0xB3, 0x05, 0xEB, 0x04, 0x0A, 0x1A, 0xF8, 0x01, 0x1C, 0x81, 0x42,
+ 0x1C, 0xD0, 0xAA, 0x28, 0x1A, 0xD0, 0x0A, 0xF8, 0x01, 0x9C, 0x0B, 0xEB, 0x44, 0x01, 0x00, 0x91,
+ 0x31, 0xF9, 0x02, 0x0C, 0x00, 0xF0, 0xB9, 0xF8, 0x00, 0x99, 0x21, 0xF8, 0x02, 0x0C, 0x31, 0x19,
+ 0x01, 0xF8, 0x01, 0x9C, 0x08, 0xEB, 0x44, 0x01, 0x00, 0x91, 0x31, 0xF9, 0x02, 0x0C, 0x00, 0xF0,
+ 0xAC, 0xF8, 0x00, 0x99, 0x21, 0xF8, 0x02, 0x0C, 0x28, 0x5D, 0x88, 0xB1, 0x9A, 0xF8, 0x01, 0x10,
+ 0x28, 0x5D, 0x81, 0x42, 0x18, 0xD0, 0xAA, 0x28, 0x16, 0xD0, 0x8A, 0xF8, 0x01, 0x90, 0x0B, 0xEB,
+ 0x44, 0x0A, 0xBA, 0xF9, 0x02, 0x00, 0x00, 0xF0, 0x98, 0xF8, 0xAA, 0xF8, 0x02, 0x00, 0x00, 0xE0,
+ 0x0A, 0xE0, 0x31, 0x19, 0x08, 0xEB, 0x44, 0x0A, 0x81, 0xF8, 0x01, 0x90, 0xBA, 0xF9, 0x02, 0x00,
+ 0x00, 0xF0, 0x8B, 0xF8, 0xAA, 0xF8, 0x02, 0x00, 0x64, 0x1C, 0xBC, 0x42, 0xAD, 0xD3, 0xBD, 0xE8,
+ 0xF8, 0x8F, 0x4A, 0x49, 0x00, 0x20, 0x08, 0x60, 0x70, 0x47, 0x48, 0x49, 0x08, 0x60, 0x70, 0x47,
+ 0x2D, 0xE9, 0xF7, 0x4F, 0x46, 0x48, 0x8A, 0xB0, 0x90, 0x46, 0x00, 0x78, 0x0C, 0x46, 0x00, 0x28,
+ 0x70, 0xD0, 0x44, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x00, 0x04, 0x80, 0x07, 0x6A, 0xD5, 0x42, 0x48,
+ 0x04, 0x90, 0x42, 0x48, 0xDF, 0xF8, 0xF4, 0xA0, 0x01, 0x25, 0x07, 0x78, 0x4F, 0xF0, 0x00, 0x0B,
+ 0xDA, 0xF8, 0x00, 0x90, 0x20, 0xE0, 0x01, 0x20, 0xA8, 0x40, 0x10, 0xEA, 0x09, 0x0F, 0x1A, 0xD1,
+ 0x3B, 0x48, 0x00, 0xEB, 0x85, 0x06, 0x04, 0x98, 0xB1, 0x78, 0x10, 0xF0, 0x09, 0xFE, 0xB2, 0x78,
+ 0x5B, 0x46, 0x0D, 0xE0, 0x31, 0x78, 0x05, 0xE0, 0x10, 0xF8, 0x01, 0xC0, 0xAC, 0x45, 0x00, 0xD1,
+ 0x43, 0x54, 0x49, 0x1C, 0x96, 0xF8, 0x01, 0xC0, 0x8C, 0x45, 0xF5, 0xDA, 0x28, 0x30, 0x52, 0x1C,
+ 0xF1, 0x78, 0x91, 0x42, 0xEE, 0xDA, 0x6D, 0x1C, 0xBD, 0x42, 0xDC, 0xD9, 0x00, 0x21, 0x0A, 0x98,
+ 0x10, 0xF0, 0xF4, 0xFD, 0x01, 0x90, 0x01, 0x21, 0x0A, 0x98, 0x10, 0xF0, 0xEF, 0xFD, 0x08, 0x90,
+ 0x00, 0x21, 0x25, 0x48, 0x10, 0xF0, 0xE4, 0xFD, 0x81, 0x46, 0x01, 0x21, 0x22, 0x48, 0x10, 0xF0,
+ 0xDF, 0xFD, 0x06, 0x46, 0x65, 0x1E, 0xE9, 0xB2, 0x0A, 0x98, 0x10, 0xF0, 0xDF, 0xFD, 0xA4, 0x1E,
+ 0x06, 0x90, 0xE1, 0xB2, 0x0A, 0x98, 0x10, 0xF0, 0xD9, 0xFD, 0x07, 0x46, 0xE9, 0xB2, 0x1A, 0x48,
+ 0x10, 0xF0, 0xCE, 0xFD, 0x05, 0x46, 0xE1, 0xB2, 0x17, 0x48, 0x10, 0xF0, 0xC9, 0xFD, 0x04, 0x46,
+ 0x33, 0x46, 0x4A, 0x46, 0xCD, 0xF8, 0x00, 0x80, 0x08, 0x99, 0x01, 0x98, 0xFF, 0xF7, 0x2A, 0xFF,
+ 0x23, 0x46, 0x2A, 0x46, 0x39, 0x46, 0xCD, 0xF8, 0x00, 0x80, 0x06, 0x98, 0xFF, 0xF7, 0x22, 0xFF,
+ 0xCA, 0xF8, 0x00, 0xB0, 0x0D, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F, 0x0A, 0x49, 0x09, 0x68, 0x91, 0xF8,
+ 0x08, 0x24, 0x42, 0x43, 0x64, 0x20, 0x92, 0xFB, 0xF0, 0xF0, 0xB1, 0xF8, 0x06, 0x24, 0x00, 0xB2,
+ 0x82, 0x42, 0x01, 0xDA, 0xB1, 0xF9, 0x06, 0x04, 0x70, 0x47, 0x00, 0x00, 0x24, 0x05, 0x10, 0x00,
+ 0x80, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0xB4, 0x18, 0x01, 0x20, 0x15, 0x06, 0x10, 0x00,
+ 0x4C, 0x1C, 0x01, 0x20, 0x2D, 0xE9, 0xF0, 0x4F, 0xFE, 0x4C, 0x80, 0x25, 0xFE, 0x4E, 0x23, 0x68,
+ 0x93, 0xF8, 0x31, 0x10, 0x93, 0xF8, 0x30, 0x00, 0x01, 0xFB, 0x00, 0xF2, 0x02, 0xE0, 0x00, 0xBF,
+ 0x26, 0xF8, 0x12, 0x50, 0x52, 0x1E, 0xFB, 0xD2, 0x93, 0xF8, 0xAA, 0x22, 0xA6, 0x46, 0x12, 0x07,
+ 0x4F, 0xF0, 0x64, 0x08, 0x31, 0xD5, 0x4A, 0x1E, 0x02, 0xFB, 0x00, 0xFA, 0xDF, 0xF8, 0xC8, 0xB3,
+ 0x03, 0xF5, 0x94, 0x69, 0x02, 0x46, 0x26, 0xE0, 0x19, 0xF8, 0x02, 0x30, 0x0A, 0xEB, 0x02, 0x04,
+ 0xDB, 0x01, 0xB3, 0xFB, 0xF8, 0xF3, 0x0B, 0xEB, 0x44, 0x05, 0x0C, 0x46, 0x19, 0xE0, 0x80, 0x2B,
+ 0x05, 0xDD, 0xC3, 0xF1, 0x80, 0x06, 0x66, 0x43, 0x96, 0xFB, 0xF1, 0xF6, 0x06, 0xE0, 0xA3, 0xF1,
+ 0x80, 0x06, 0x06, 0xFB, 0x04, 0xF7, 0x97, 0xFB, 0xF1, 0xF7, 0xF6, 0x1B, 0xB5, 0xF9, 0x00, 0x70,
+ 0x80, 0x36, 0x77, 0x43, 0xFE, 0x17, 0x07, 0xEB, 0x56, 0x66, 0xF6, 0x11, 0x2E, 0x80, 0xA5, 0xEB,
+ 0x40, 0x05, 0x64, 0x1E, 0xE3, 0xD2, 0x52, 0x1E, 0xD6, 0xD2, 0xDE, 0xF8, 0x00, 0x20, 0xF2, 0x46,
+ 0x92, 0xF8, 0xAA, 0x32, 0xDB, 0x06, 0x2F, 0xD5, 0x02, 0xF5, 0x98, 0x69, 0xD6, 0x4B, 0x4A, 0x1E,
+ 0x42, 0x43, 0x03, 0xEB, 0x42, 0x05, 0x0A, 0x46, 0x24, 0xE0, 0x00, 0xBF, 0x19, 0xF8, 0x02, 0x30,
+ 0xDB, 0x01, 0xB3, 0xFB, 0xF8, 0xF6, 0x03, 0x46, 0x18, 0xE0, 0x80, 0x2E, 0x05, 0xDD, 0xC6, 0xF1,
+ 0x80, 0x04, 0x5C, 0x43, 0x94, 0xFB, 0xF0, 0xF4, 0x06, 0xE0, 0xA6, 0xF1, 0x80, 0x04, 0x04, 0xFB,
+ 0x03, 0xF7, 0x97, 0xFB, 0xF0, 0xF7, 0xE4, 0x1B, 0x35, 0xF9, 0x13, 0x70, 0x80, 0x34, 0x67, 0x43,
+ 0xFC, 0x17, 0x07, 0xEB, 0x54, 0x64, 0xE4, 0x11, 0x25, 0xF8, 0x13, 0x40, 0x5B, 0x1E, 0xE4, 0xD2,
+ 0xA5, 0xEB, 0x40, 0x05, 0x52, 0x1E, 0xD9, 0xD2, 0xDA, 0xF8, 0x00, 0x40, 0x94, 0xF8, 0xAB, 0x22,
+ 0xD2, 0x07, 0x20, 0xD0, 0xBC, 0x4E, 0x00, 0x23, 0x04, 0xF5, 0x9C, 0x64, 0x19, 0xE0, 0x00, 0xBF,
+ 0x03, 0xFB, 0x00, 0xF2, 0x06, 0xEB, 0x42, 0x07, 0x00, 0x22, 0x0F, 0xE0, 0xA5, 0x5C, 0xED, 0x01,
+ 0xB5, 0xFB, 0xF8, 0xFC, 0x37, 0xF9, 0x12, 0x50, 0x05, 0xFB, 0x0C, 0xF5, 0x4F, 0xEA, 0xE5, 0x7C,
+ 0x05, 0xEB, 0x5C, 0x65, 0xED, 0x11, 0x27, 0xF8, 0x12, 0x50, 0x52, 0x1C, 0x82, 0x42, 0xED, 0xD3,
+ 0x5B, 0x1C, 0x8B, 0x42, 0xE4, 0xD3, 0xBD, 0xE8, 0xF0, 0x8F, 0x08, 0xB5, 0x00, 0x93, 0x80, 0x23,
+ 0x0B, 0xF0, 0x10, 0xFF, 0x08, 0xBD, 0x08, 0xB5, 0x00, 0x93, 0x13, 0x46, 0xA7, 0x4A, 0xB2, 0xF9,
+ 0x00, 0x20, 0x0B, 0xF0, 0x17, 0xFF, 0x08, 0xBD, 0x2D, 0xE9, 0xF0, 0x5F, 0x00, 0xF0, 0x36, 0xFD,
+ 0xA0, 0x4F, 0xDF, 0xF8, 0x8C, 0xA2, 0x38, 0x68, 0x9A, 0xF8, 0x01, 0x40, 0xB0, 0xF9, 0x00, 0x13,
+ 0xB0, 0xF9, 0x22, 0x33, 0xB0, 0xF9, 0x42, 0x23, 0x01, 0x2C, 0x50, 0xD0, 0x02, 0x2C, 0x04, 0xD3,
+ 0xB0, 0xF8, 0x04, 0x43, 0x0C, 0xB1, 0xB0, 0xF9, 0x04, 0x13, 0x9A, 0x4C, 0xA5, 0x79, 0x9D, 0xB1,
+ 0x90, 0xF8, 0x69, 0x53, 0x64, 0x26, 0x05, 0xFB, 0x01, 0xFC, 0x9C, 0xFB, 0xF6, 0xFC, 0x61, 0x44,
+ 0x05, 0xFB, 0x03, 0xFC, 0x55, 0x43, 0x9C, 0xFB, 0xF6, 0xFC, 0x95, 0xFB, 0xF6, 0xF5, 0x63, 0x44,
+ 0x2A, 0x44, 0x09, 0xB2, 0x1B, 0xB2, 0x12, 0xB2, 0x8F, 0x4E, 0x90, 0x4D, 0xDF, 0xF8, 0x40, 0xC2,
+ 0x31, 0x80, 0x2B, 0x80, 0xAC, 0xF8, 0x00, 0x20, 0x90, 0xF8, 0x02, 0x73, 0xC9, 0x1B, 0x8D, 0x4F,
+ 0x39, 0x80, 0x90, 0xF8, 0x24, 0x13, 0x5B, 0x1A, 0x8B, 0x49, 0x0B, 0x80, 0x90, 0xF8, 0x44, 0x33,
+ 0xD3, 0x1A, 0x8A, 0x4A, 0x13, 0x80, 0x23, 0x78, 0x53, 0xB1, 0x89, 0x4B, 0x1B, 0x78, 0x3B, 0xB1,
+ 0xB0, 0xF8, 0x6C, 0x32, 0x33, 0x80, 0x90, 0xF8, 0x6E, 0x82, 0xA3, 0xEB, 0x08, 0x03, 0x3B, 0x80,
+ 0xDF, 0xF8, 0x10, 0xB2, 0x9B, 0xF8, 0x00, 0x30, 0x33, 0xB1, 0xB0, 0xF8, 0x6C, 0x32, 0x33, 0x80,
+ 0x90, 0xF8, 0x6E, 0x02, 0x18, 0x1A, 0x38, 0x80, 0xA0, 0x78, 0x68, 0xB1, 0x10, 0xE0, 0x90, 0xF8,
+ 0x03, 0x43, 0x21, 0x44, 0x90, 0xF8, 0x25, 0x43, 0x09, 0xB2, 0x23, 0x44, 0x90, 0xF8, 0x45, 0x43,
+ 0x1B, 0xB2, 0x22, 0x44, 0x12, 0xB2, 0xA8, 0xE7, 0x30, 0x88, 0x28, 0x80, 0x38, 0x88, 0x08, 0x80,
+ 0xE0, 0x78, 0xA1, 0x46, 0x20, 0xB9, 0x30, 0x88, 0xAC, 0xF8, 0x00, 0x00, 0x38, 0x88, 0x10, 0x80,
+ 0xB2, 0xF9, 0x00, 0x20, 0xB1, 0xF9, 0x00, 0x10, 0xB7, 0xF9, 0x00, 0x00, 0x0A, 0xF0, 0xE4, 0xFF,
+ 0x6D, 0x4D, 0x00, 0xB2, 0x6D, 0x4A, 0x28, 0x80, 0x31, 0x88, 0x6D, 0x4B, 0x11, 0x80, 0x5D, 0x4C,
+ 0x39, 0x88, 0x19, 0x80, 0x21, 0x68, 0x91, 0xF8, 0x00, 0xC4, 0x5F, 0xEA, 0xCC, 0x7C, 0x14, 0xD0,
+ 0x9A, 0xF8, 0x01, 0xC0, 0xBC, 0xF1, 0x00, 0x0F, 0x01, 0xD0, 0x91, 0xF8, 0x05, 0xC4, 0xB1, 0xF8,
+ 0x02, 0x84, 0xC4, 0x44, 0xA2, 0xF8, 0x00, 0xC0, 0x91, 0xF8, 0x04, 0x14, 0xAC, 0xEB, 0x01, 0x01,
+ 0x09, 0xB2, 0x19, 0x80, 0x0A, 0xF0, 0xCE, 0xFF, 0x28, 0x80, 0xB2, 0x46, 0x30, 0x88, 0x5D, 0x4E,
+ 0x5D, 0x49, 0x5E, 0x4A, 0x30, 0x80, 0x08, 0x80, 0x38, 0x88, 0x10, 0x80, 0x99, 0xF8, 0x04, 0x00,
+ 0xC8, 0x46, 0x00, 0xB3, 0x20, 0x68, 0xB0, 0xF8, 0x44, 0x34, 0x2B, 0xB1, 0x98, 0xF8, 0x06, 0x30,
+ 0x13, 0xB1, 0xB0, 0xF8, 0x44, 0x34, 0x04, 0xE0, 0xB0, 0xF8, 0x42, 0x34, 0x13, 0xB1, 0xB0, 0xF8,
+ 0x42, 0x34, 0x33, 0x80, 0xB0, 0xF8, 0x46, 0x34, 0x33, 0xB1, 0xB0, 0xF8, 0x46, 0x34, 0x0B, 0x80,
+ 0x90, 0xF8, 0x48, 0x04, 0x18, 0x1A, 0x10, 0x80, 0xB2, 0xF9, 0x00, 0x10, 0xB5, 0xF9, 0x00, 0x00,
+ 0x0A, 0xF0, 0xA0, 0xFF, 0x28, 0x80, 0x4A, 0x4F, 0xBA, 0xF8, 0x00, 0x00, 0x38, 0x80, 0x98, 0xF8,
+ 0x05, 0x00, 0x30, 0xB1, 0x20, 0x68, 0xB0, 0xF8, 0x21, 0x14, 0x11, 0xB1, 0xB0, 0xF8, 0x21, 0x04,
+ 0x38, 0x80, 0x3B, 0x48, 0x43, 0x4D, 0x00, 0x78, 0x18, 0xB1, 0x20, 0x68, 0xB0, 0xF8, 0xBE, 0x02,
+ 0x19, 0xE0, 0x9B, 0xF8, 0x00, 0x00, 0x00, 0x28, 0xF7, 0xD1, 0x20, 0x68, 0xB0, 0xF9, 0xBC, 0x02,
+ 0x28, 0x80, 0x98, 0xF8, 0x04, 0x10, 0x21, 0xB1, 0xB6, 0xF9, 0x00, 0x10, 0x0A, 0xF0, 0x7A, 0xFF,
+ 0x28, 0x80, 0x98, 0xF8, 0x05, 0x00, 0x00, 0x28, 0x06, 0xD0, 0xB7, 0xF9, 0x00, 0x10, 0xB5, 0xF9,
+ 0x00, 0x00, 0x0A, 0xF0, 0x6F, 0xFF, 0x28, 0x80, 0xBD, 0xE8, 0xF0, 0x9F, 0x2D, 0xE9, 0xFF, 0x4F,
+ 0x8A, 0x46, 0x81, 0x46, 0x30, 0x49, 0x0D, 0x98, 0x93, 0x46, 0x4A, 0x0C, 0x04, 0x46, 0x07, 0xE0,
+ 0x3A, 0xF9, 0x14, 0x30, 0x93, 0x42, 0x00, 0xDA, 0x1A, 0x46, 0x8B, 0x42, 0x00, 0xDD, 0x19, 0x46,
+ 0x64, 0x1E, 0xF5, 0xD2, 0x00, 0x25, 0x00, 0x29, 0x01, 0xDA, 0x0D, 0x46, 0x02, 0xE0, 0x00, 0x2A,
+ 0x00, 0xDD, 0x15, 0x46, 0x00, 0x23, 0x1C, 0x46, 0x1E, 0x46, 0x19, 0x46, 0x1F, 0x46, 0x02, 0x46,
+ 0x12, 0xE0, 0x00, 0xBF, 0x3A, 0xF9, 0x12, 0xC0, 0xAC, 0xEB, 0x05, 0x0C, 0xDC, 0x45, 0x0B, 0xDA,
+ 0x0C, 0xEB, 0x0B, 0x08, 0xB8, 0xF1, 0x00, 0x0F, 0x06, 0xDD, 0x02, 0xFB, 0x02, 0x44, 0x02, 0xFB,
+ 0x0C, 0x11, 0x13, 0x44, 0x66, 0x44, 0x7F, 0x1C, 0x52, 0x1E, 0xEB, 0xD2, 0x04, 0xFB, 0x07, 0xF2,
+ 0x2C, 0xE0, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00, 0x78, 0x12, 0x01, 0x20, 0xCA, 0x06, 0x10, 0x00,
+ 0x4C, 0x07, 0x10, 0x00, 0xEC, 0x06, 0x10, 0x00, 0x68, 0x06, 0x10, 0x00, 0x6A, 0x06, 0x10, 0x00,
+ 0x6C, 0x06, 0x10, 0x00, 0x6E, 0x06, 0x10, 0x00, 0x70, 0x06, 0x10, 0x00, 0x72, 0x06, 0x10, 0x00,
+ 0x1C, 0x05, 0x10, 0x00, 0x28, 0x05, 0x10, 0x00, 0x74, 0x06, 0x10, 0x00, 0x7C, 0x06, 0x10, 0x00,
+ 0x7E, 0x06, 0x10, 0x00, 0x84, 0x06, 0x10, 0x00, 0x86, 0x06, 0x10, 0x00, 0x88, 0x06, 0x10, 0x00,
+ 0x82, 0x06, 0x10, 0x00, 0x76, 0x06, 0x10, 0x00, 0x00, 0x80, 0xFF, 0xFF, 0x03, 0xFB, 0x13, 0x2E,
+ 0x01, 0xFB, 0x07, 0xF2, 0x03, 0xFB, 0x16, 0x27, 0x74, 0x43, 0x01, 0xFB, 0x13, 0x46, 0x00, 0x24,
+ 0x21, 0x46, 0x02, 0x46, 0x25, 0xE0, 0x00, 0x23, 0xBE, 0xF1, 0x00, 0x0F, 0x0F, 0xD0, 0x07, 0xFB,
+ 0x02, 0x6C, 0x9C, 0xFB, 0xFE, 0xF3, 0xDC, 0x46, 0x5B, 0x45, 0x01, 0xDD, 0x63, 0x46, 0x06, 0xE0,
+ 0xDD, 0xF8, 0x0C, 0xC0, 0x13, 0xEB, 0x0C, 0x0F, 0x01, 0xD5, 0x03, 0x9B, 0x5B, 0x42, 0x3A, 0xF8,
+ 0x12, 0xC0, 0x2B, 0x44, 0xAC, 0xEB, 0x03, 0x03, 0xDD, 0xF8, 0x0C, 0xC0, 0x1B, 0xB2, 0x63, 0x45,
+ 0x05, 0xDA, 0x9C, 0x44, 0xBC, 0xF1, 0x00, 0x0F, 0x01, 0xDD, 0x1C, 0x44, 0x49, 0x1C, 0x29, 0xF8,
+ 0x12, 0x30, 0x52, 0x1E, 0xD7, 0xD2, 0x01, 0xB9, 0x01, 0x21, 0x94, 0xFB, 0xF1, 0xF1, 0x04, 0xE0,
+ 0x39, 0xF8, 0x10, 0x20, 0x52, 0x1A, 0x29, 0xF8, 0x10, 0x20, 0x40, 0x1E, 0xF8, 0xD2, 0xBD, 0xE8,
+ 0xFF, 0x8F, 0x2D, 0xE9, 0xF8, 0x43, 0x1F, 0x46, 0xFE, 0x4B, 0x0E, 0x46, 0x05, 0x46, 0x1B, 0x68,
+ 0x14, 0x46, 0xB3, 0xF9, 0xDA, 0x92, 0xB3, 0xF9, 0xDC, 0x82, 0x0A, 0xE0, 0x43, 0x46, 0x4A, 0x46,
+ 0x31, 0x46, 0x28, 0x46, 0x00, 0x97, 0xFF, 0xF7, 0x41, 0xFF, 0x06, 0xEB, 0x47, 0x06, 0x05, 0xEB,
+ 0x47, 0x05, 0x64, 0x1E, 0xF2, 0xD2, 0xBD, 0xE8, 0xF8, 0x83, 0x2D, 0xE9, 0xFF, 0x5F, 0xF2, 0x4C,
+ 0x88, 0x46, 0x00, 0x26, 0x07, 0x46, 0x92, 0x46, 0x35, 0x46, 0x4F, 0xF4, 0x66, 0x71, 0x40, 0x46,
+ 0x26, 0x60, 0x10, 0xF0, 0xE6, 0xF9, 0x00, 0x21, 0x40, 0x46, 0x10, 0xF0, 0xF9, 0xFA, 0x81, 0x46,
+ 0x00, 0x21, 0x38, 0x46, 0x10, 0xF0, 0xEC, 0xFA, 0x00, 0x22, 0xE8, 0x4B, 0xE8, 0x4F, 0x29, 0xE0,
+ 0x00, 0x21, 0x4F, 0xF0, 0x01, 0x0E, 0xDF, 0xF8, 0x8C, 0xB3, 0x18, 0xE0, 0x30, 0xF9, 0x11, 0x80,
+ 0xD0, 0x45, 0x05, 0xDD, 0x09, 0xF8, 0x01, 0xE0, 0x30, 0xF9, 0x11, 0xC0, 0x76, 0x1C, 0x65, 0x44,
+ 0xDB, 0xF8, 0x00, 0xC0, 0x30, 0xF9, 0x11, 0x80, 0xBC, 0xF8, 0x55, 0xC3, 0xE0, 0x45, 0x05, 0xDD,
+ 0xD4, 0xF8, 0x00, 0xC0, 0x0C, 0xF1, 0x01, 0x0C, 0xC4, 0xF8, 0x00, 0xC0, 0x49, 0x1C, 0x93, 0xF8,
+ 0x00, 0xC0, 0x61, 0x45, 0xE2, 0xDB, 0x0C, 0xF0, 0xFF, 0x01, 0x09, 0xF1, 0x28, 0x09, 0x00, 0xEB,
+ 0x41, 0x00, 0x52, 0x1C, 0x39, 0x78, 0x8A, 0x42, 0xD2, 0xDB, 0x03, 0x98, 0x05, 0x60, 0x30, 0x46,
+ 0x04, 0xB0, 0xE9, 0xE6, 0x06, 0xE0, 0x00, 0xBF, 0x30, 0xF9, 0x12, 0x30, 0x8B, 0x42, 0x01, 0xDD,
+ 0x01, 0x20, 0x70, 0x47, 0x52, 0x1E, 0xF7, 0xD2, 0x00, 0x20, 0x70, 0x47, 0x2D, 0xE9, 0xFC, 0x47,
+ 0x04, 0x46, 0xC8, 0x48, 0xC8, 0x4F, 0x13, 0x46, 0x02, 0x68, 0x38, 0x68, 0xFF, 0xF7, 0xC3, 0xFD,
+ 0x03, 0x20, 0x0E, 0xF0, 0x20, 0xF9, 0xDF, 0xF8, 0xFC, 0x92, 0xDF, 0xF8, 0x10, 0x83, 0xBF, 0x4D,
+ 0xD9, 0xF8, 0x00, 0x00, 0xBE, 0x4E, 0x90, 0xF8, 0xAA, 0x02, 0x80, 0x07, 0x15, 0xD5, 0xC0, 0x48,
+ 0xB0, 0xF9, 0x00, 0x20, 0xBF, 0x48, 0xB0, 0xF9, 0x00, 0x10, 0xBF, 0x48, 0xB0, 0xF9, 0x00, 0x00,
+ 0x0A, 0xF0, 0x1A, 0xFE, 0x03, 0xB2, 0x29, 0x78, 0x30, 0x78, 0xCD, 0xE9, 0x00, 0x01, 0x39, 0x68,
+ 0xD8, 0xF8, 0x00, 0x20, 0x08, 0x46, 0x00, 0xF0, 0xF6, 0xFA, 0xB8, 0x48, 0xD8, 0xF8, 0x00, 0x20,
+ 0x39, 0x68, 0xB0, 0xF9, 0x00, 0x30, 0x20, 0x46, 0x0B, 0xF0, 0x97, 0xFC, 0x03, 0x20, 0x0E, 0xF0,
+ 0x0D, 0xF9, 0xD9, 0xF8, 0x00, 0x00, 0x4F, 0x46, 0x90, 0xF8, 0xAA, 0x02, 0x00, 0x06, 0x05, 0xD5,
+ 0x21, 0x46, 0x2B, 0x78, 0x32, 0x78, 0x08, 0x46, 0xFF, 0xF7, 0x43, 0xFF, 0x38, 0x68, 0x90, 0xF8,
+ 0xAA, 0x02, 0x40, 0x07, 0x25, 0xD5, 0x30, 0x78, 0x2E, 0x78, 0x32, 0x46, 0x1F, 0xE0, 0x00, 0x21,
+ 0x03, 0x46, 0x0C, 0xE0, 0x03, 0xFB, 0x06, 0x27, 0x34, 0xF9, 0x17, 0x50, 0x00, 0x2D, 0x04, 0xDD,
+ 0x09, 0xB1, 0x8D, 0x42, 0x03, 0xDA, 0x29, 0x46, 0x01, 0xE0, 0x00, 0x21, 0x43, 0x1C, 0x5B, 0x1E,
+ 0xF0, 0xD2, 0x00, 0x29, 0x0B, 0xDD, 0x03, 0x46, 0x07, 0xE0, 0x00, 0xBF, 0x03, 0xFB, 0x06, 0x25,
+ 0x34, 0xF8, 0x15, 0x70, 0x7F, 0x1A, 0x24, 0xF8, 0x15, 0x70, 0x5B, 0x1E, 0xF6, 0xD2, 0x52, 0x1E,
+ 0xDD, 0xD2, 0xBD, 0xE8, 0xFC, 0x87, 0x2D, 0xE9, 0xF0, 0x43, 0x91, 0x46, 0x1C, 0x46, 0x36, 0xE0,
+ 0x93, 0x4D, 0x4F, 0xF0, 0x01, 0x0C, 0x6A, 0x0C, 0xE0, 0x46, 0x4F, 0x46, 0x12, 0xE0, 0x00, 0xBF,
+ 0x07, 0xFB, 0x03, 0x46, 0x31, 0xF9, 0x16, 0x60, 0x96, 0x42, 0x00, 0xDA, 0x32, 0x46, 0xAE, 0x42,
+ 0x00, 0xDD, 0x35, 0x46, 0x00, 0x2E, 0x02, 0xDD, 0x4F, 0xF0, 0x00, 0x08, 0x02, 0xE0, 0x01, 0xDA,
+ 0x4F, 0xF0, 0x00, 0x0C, 0x7F, 0x1E, 0xEB, 0xD2, 0xBC, 0xF1, 0x00, 0x0F, 0x02, 0xD0, 0xB8, 0xF1,
+ 0x00, 0x0F, 0x14, 0xD1, 0x5C, 0xEA, 0x08, 0x06, 0x11, 0xD0, 0xBC, 0xF1, 0x00, 0x0F, 0x03, 0xD1,
+ 0xB8, 0xF1, 0x00, 0x0F, 0x00, 0xD0, 0x2A, 0x46, 0x4E, 0x46, 0x06, 0xE0, 0x06, 0xFB, 0x03, 0x45,
+ 0x31, 0xF8, 0x15, 0x70, 0xBF, 0x1A, 0x20, 0xF8, 0x15, 0x70, 0x76, 0x1E, 0xF6, 0xD2, 0x64, 0x1E,
+ 0xC6, 0xD2, 0xBD, 0xE8, 0xF0, 0x83, 0x2D, 0xE9, 0xF3, 0x4F, 0x76, 0x49, 0x89, 0xB0, 0x09, 0x78,
+ 0x00, 0x29, 0x75, 0xD1, 0xCD, 0xE9, 0x00, 0x10, 0x68, 0x48, 0x07, 0x91, 0x8A, 0x46, 0x90, 0xF8,
+ 0x00, 0x90, 0x67, 0x48, 0x88, 0x46, 0x00, 0x78, 0x08, 0x90, 0x57, 0xE0, 0xDF, 0xF8, 0xB0, 0xB1,
+ 0x00, 0x20, 0xCD, 0xE9, 0x05, 0x0B, 0x05, 0x46, 0x04, 0x90, 0x42, 0x46, 0x01, 0x20, 0x00, 0x21,
+ 0x10, 0xF0, 0x11, 0xF8, 0x0A, 0x9A, 0xD2, 0xE9, 0x00, 0x32, 0x03, 0x40, 0x0A, 0x40, 0xCD, 0xE9,
+ 0x02, 0x23, 0x22, 0xE0, 0x01, 0x98, 0x30, 0xF9, 0x15, 0x40, 0x5C, 0x45, 0x00, 0xDD, 0xA3, 0x46,
+ 0x05, 0x98, 0x84, 0x42, 0x00, 0xDA, 0x05, 0x94, 0xDD, 0xE9, 0x02, 0x10, 0x08, 0x43, 0x13, 0xD1,
+ 0x0A, 0x98, 0x2A, 0x46, 0x00, 0x21, 0xD0, 0xE9, 0x02, 0x67, 0x01, 0x20, 0x0F, 0xF0, 0xF3, 0xFF,
+ 0x06, 0x40, 0x0F, 0x40, 0x3E, 0x43, 0x07, 0xD1, 0x06, 0x98, 0x84, 0x42, 0x00, 0xDD, 0x06, 0x94,
+ 0x04, 0x98, 0x84, 0x42, 0x00, 0xDA, 0x04, 0x94, 0x6D, 0x1C, 0x4D, 0x45, 0xDA, 0xDB, 0x05, 0x98,
+ 0x00, 0x28, 0x00, 0xDA, 0x40, 0x42, 0xAB, 0xEB, 0x00, 0x00, 0x01, 0xB2, 0x04, 0x98, 0x00, 0x28,
+ 0x00, 0xDA, 0x40, 0x42, 0x06, 0x9A, 0x10, 0x1A, 0x02, 0xB2, 0x51, 0x45, 0x00, 0xDD, 0x8A, 0x46,
+ 0xDD, 0xE9, 0x02, 0x10, 0x08, 0x43, 0x03, 0xD1, 0x00, 0x98, 0x82, 0x42, 0x00, 0xDA, 0x00, 0x92,
+ 0x01, 0x99, 0x08, 0xF1, 0x01, 0x08, 0x01, 0xEB, 0x49, 0x00, 0x01, 0x90, 0x08, 0x99, 0x88, 0x45,
+ 0xA4, 0xDB, 0x34, 0x48, 0x00, 0x68, 0xB0, 0xF9, 0xC0, 0x22, 0xB0, 0xF9, 0xC2, 0x12, 0x92, 0x45,
+ 0x07, 0xDD, 0x00, 0x9A, 0x49, 0x42, 0x8A, 0x42, 0x03, 0xDA, 0x01, 0x21, 0x07, 0x91, 0x03, 0xE0,
+ 0x18, 0xE0, 0x38, 0x49, 0x00, 0x22, 0x4A, 0x70, 0x90, 0xF8, 0xC4, 0x12, 0x35, 0x48, 0x07, 0x9A,
+ 0x40, 0x1C, 0x0A, 0xF0, 0x98, 0xFD, 0x33, 0x49, 0x08, 0x70, 0x28, 0xB1, 0xFF, 0xF7, 0x94, 0xFC,
+ 0x01, 0x21, 0x08, 0x20, 0x0A, 0xF0, 0x4E, 0xF8, 0x2F, 0x48, 0xA0, 0xF8, 0x40, 0xA0, 0x00, 0x99,
+ 0xA0, 0xF8, 0x42, 0x10, 0x2B, 0x48, 0x02, 0x78, 0x2B, 0x48, 0x01, 0x7C, 0x62, 0xF3, 0xC7, 0x11,
+ 0x01, 0x74, 0x0B, 0xB0, 0x6F, 0xE4, 0x2D, 0xE9, 0xF7, 0x4F, 0x93, 0x46, 0x27, 0x4A, 0x4F, 0xF0,
+ 0x00, 0x09, 0x06, 0x46, 0x15, 0x78, 0x82, 0xF8, 0x00, 0x90, 0xFF, 0xF7, 0x75, 0xFC, 0x21, 0x48,
+ 0x02, 0x21, 0x00, 0x1D, 0x08, 0xF0, 0x3E, 0xF8, 0x21, 0x48, 0x1E, 0x4C, 0x2A, 0x46, 0x00, 0x78,
+ 0x31, 0x46, 0x01, 0x28, 0x60, 0x68, 0x5B, 0xD0, 0xFF, 0xF7, 0x90, 0xFE, 0x0D, 0x4D, 0xDF, 0xF8,
+ 0x3C, 0x80, 0x0F, 0x4E, 0x28, 0x68, 0x90, 0xF8, 0xAB, 0x02, 0x80, 0x07, 0x06, 0xD5, 0x61, 0x68,
+ 0x98, 0xF8, 0x00, 0x30, 0x32, 0x78, 0x08, 0x46, 0xFF, 0xF7, 0xF5, 0xFE, 0x29, 0x68, 0x0F, 0x4F,
+ 0x91, 0xF8, 0xAA, 0x02, 0x02, 0x07, 0x26, 0xD4, 0xC0, 0x06, 0x24, 0xD4, 0x91, 0xF8, 0xAB, 0x02,
+ 0xC0, 0x07, 0x1F, 0xE0, 0x20, 0x07, 0x10, 0x00, 0x5C, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00,
+ 0xB9, 0x06, 0x10, 0x00, 0x78, 0x06, 0x10, 0x00, 0xE4, 0x06, 0x10, 0x00, 0xE8, 0x06, 0x10, 0x00,
+ 0x6C, 0x06, 0x10, 0x00, 0x6A, 0x06, 0x10, 0x00, 0x68, 0x06, 0x10, 0x00, 0xCA, 0x06, 0x10, 0x00,
+ 0x00, 0x80, 0xFF, 0xFF, 0x28, 0x05, 0x10, 0x00, 0xE6, 0x43, 0x01, 0x20, 0x50, 0x07, 0x10, 0x00,
+ 0xBF, 0x06, 0x10, 0x00, 0x06, 0xD0, 0x61, 0x68, 0xB7, 0xF9, 0x00, 0x30, 0xF8, 0x4A, 0x08, 0x46,
+ 0xFF, 0xF7, 0x13, 0xFC, 0xB7, 0xF9, 0x00, 0x00, 0x61, 0x68, 0x42, 0x00, 0xF5, 0x48, 0x00, 0x68,
+ 0x0F, 0xF0, 0x60, 0xFF, 0x98, 0xF8, 0x00, 0x20, 0x31, 0x78, 0x60, 0x68, 0xFF, 0xF7, 0xC0, 0xFA,
+ 0xDF, 0xF8, 0xC4, 0x83, 0xDF, 0xF8, 0xC4, 0xA3, 0x98, 0xF8, 0x00, 0x00, 0x30, 0xB9, 0x02, 0xE0,
+ 0xFF, 0xF7, 0x46, 0xF9, 0xA2, 0xE7, 0x98, 0xF8, 0x01, 0x00, 0x28, 0xB1, 0x59, 0x46, 0x62, 0x68,
+ 0x01, 0x98, 0xFE, 0xF7, 0x95, 0xFD, 0x01, 0xE0, 0x8A, 0xF8, 0x00, 0x90, 0xE8, 0x48, 0xE9, 0x4E,
+ 0x40, 0x78, 0x02, 0x28, 0x28, 0x68, 0x90, 0xF8, 0xAA, 0x12, 0x4F, 0xEA, 0x81, 0x61, 0x16, 0xD3,
+ 0x00, 0x29, 0x07, 0xDA, 0x61, 0x68, 0xB0, 0xF9, 0xBA, 0x22, 0xB7, 0xF9, 0x00, 0x30, 0x08, 0x46,
+ 0x00, 0xF0, 0xA9, 0xF9, 0x28, 0x68, 0x90, 0xF8, 0xAA, 0x12, 0x49, 0x06, 0x12, 0xD5, 0xB0, 0xF9,
+ 0xB8, 0x32, 0xDD, 0x4A, 0x61, 0x68, 0x30, 0x68, 0x06, 0xF0, 0x70, 0xFB, 0x11, 0xE0, 0x00, 0x29,
+ 0x08, 0xDA, 0xB0, 0xF9, 0xBA, 0x22, 0xB7, 0xF9, 0x00, 0x30, 0x61, 0x68, 0x30, 0x68, 0x00, 0xF0,
+ 0x92, 0xF9, 0x06, 0xE0, 0xB7, 0xF9, 0x00, 0x00, 0x61, 0x68, 0x42, 0x00, 0x30, 0x68, 0x0F, 0xF0,
+ 0x11, 0xFF, 0x02, 0x21, 0xD1, 0x48, 0x07, 0xF0, 0xF8, 0xFF, 0x9A, 0xF8, 0x00, 0x00, 0x4F, 0xF0,
+ 0x01, 0x0B, 0xA8, 0xB1, 0x98, 0xF8, 0x00, 0x00, 0x90, 0xB1, 0xCD, 0x48, 0x00, 0x78, 0x01, 0x28,
+ 0x0E, 0xD1, 0x28, 0x68, 0x90, 0xF8, 0xAB, 0x02, 0x40, 0x07, 0x24, 0xD5, 0x20, 0x78, 0x60, 0xB9,
+ 0x84, 0xF8, 0x00, 0xB0, 0xFF, 0xF7, 0xB0, 0xFB, 0x01, 0x21, 0x08, 0x20, 0x09, 0xF0, 0x6A, 0xFF,
+ 0x28, 0x68, 0x90, 0xF8, 0xAB, 0x02, 0x40, 0x07, 0x12, 0xD5, 0xC2, 0x49, 0x30, 0x68, 0xFF, 0xF7,
+ 0x82, 0xFE, 0x28, 0x68, 0x90, 0xF8, 0xAB, 0x12, 0x09, 0x07, 0x09, 0xD5, 0x21, 0x78, 0x39, 0xB1,
+ 0xB0, 0xF9, 0xC5, 0x32, 0xB3, 0x48, 0xB8, 0x4A, 0x01, 0x68, 0x30, 0x68, 0x06, 0xF0, 0x26, 0xFB,
+ 0x9A, 0xF8, 0x00, 0x00, 0x30, 0xB1, 0x98, 0xF8, 0x01, 0x00, 0x18, 0xB1, 0xB4, 0x48, 0x00, 0x78,
+ 0x02, 0x28, 0x1C, 0xD0, 0xB4, 0x48, 0xB7, 0xF9, 0x00, 0x20, 0xB0, 0xF9, 0x00, 0x10, 0x30, 0x68,
+ 0xFF, 0xF7, 0xA0, 0xFD, 0xB1, 0x49, 0xB2, 0x4D, 0xA0, 0xB1, 0x81, 0xF8, 0x00, 0xB0, 0x02, 0x24,
+ 0x01, 0x21, 0xB0, 0x48, 0x07, 0xF0, 0x46, 0xFF, 0xB0, 0x48, 0xAF, 0x4B, 0xB0, 0xF9, 0x00, 0x20,
+ 0xAC, 0x48, 0x01, 0x68, 0x30, 0x68, 0xFF, 0xF7, 0x40, 0xFD, 0x28, 0x60, 0x0D, 0xE0, 0x04, 0x20,
+ 0xBD, 0xE8, 0xFE, 0x8F, 0xAA, 0x4A, 0xC5, 0xF8, 0x00, 0x90, 0x01, 0x24, 0xC2, 0xF8, 0x00, 0x90,
+ 0xA5, 0x4A, 0xC2, 0xF8, 0x00, 0x90, 0x81, 0xF8, 0x00, 0x90, 0x20, 0x46, 0xF0, 0xE7, 0x2D, 0xE9,
+ 0xF0, 0x41, 0x07, 0x46, 0xA3, 0x48, 0x00, 0x26, 0x05, 0x78, 0x06, 0x70, 0xFF, 0xF7, 0x54, 0xFB,
+ 0x02, 0x21, 0x96, 0x48, 0x07, 0xF0, 0x1E, 0xFF, 0x9F, 0x48, 0x94, 0x4C, 0x2A, 0x46, 0x00, 0x78,
+ 0x24, 0x1F, 0x01, 0x28, 0x60, 0x68, 0x39, 0x46, 0x4B, 0xD0, 0xFF, 0xF7, 0x6F, 0xFD, 0x9B, 0x4D,
+ 0x28, 0x68, 0x90, 0xF8, 0xAB, 0x02, 0x80, 0x07, 0x07, 0xD5, 0x99, 0x48, 0x61, 0x68, 0x03, 0x78,
+ 0x98, 0x48, 0x02, 0x78, 0x08, 0x46, 0xFF, 0xF7, 0xD6, 0xFD, 0x29, 0x68, 0x96, 0x4D, 0x91, 0xF8,
+ 0xAA, 0x02, 0x02, 0x07, 0x05, 0xD4, 0xC0, 0x06, 0x03, 0xD4, 0x91, 0xF8, 0xAB, 0x02, 0xC0, 0x07,
+ 0x06, 0xD0, 0x61, 0x68, 0xB5, 0xF9, 0x00, 0x30, 0x79, 0x4A, 0x08, 0x46, 0xFF, 0xF7, 0x15, 0xFB,
+ 0x61, 0x68, 0xB5, 0xF9, 0x00, 0x00, 0x7B, 0x4C, 0x42, 0x00, 0x20, 0x68, 0x0F, 0xF0, 0x62, 0xFE,
+ 0x02, 0x21, 0x7A, 0x48, 0x07, 0xF0, 0x49, 0xFF, 0x7B, 0x48, 0xB5, 0xF9, 0x00, 0x20, 0xB0, 0xF9,
+ 0x00, 0x10, 0x20, 0x68, 0xFF, 0xF7, 0x2E, 0xFD, 0x78, 0x49, 0xA8, 0xB1, 0x01, 0x20, 0x08, 0x70,
+ 0x01, 0x46, 0x02, 0x25, 0x77, 0x48, 0x07, 0xF0, 0xD5, 0xFE, 0x78, 0x48, 0x76, 0x4B, 0xB0, 0xF9,
+ 0x00, 0x20, 0x74, 0x48, 0x01, 0x68, 0x20, 0x68, 0xFF, 0xF7, 0xCF, 0xFC, 0x70, 0x49, 0x08, 0x60,
+ 0x04, 0xE0, 0xFF, 0xF7, 0x35, 0xF8, 0xB2, 0xE7, 0x0E, 0x70, 0x01, 0x25, 0x28, 0x46, 0xBD, 0xE8,
+ 0xF0, 0x81, 0x66, 0x49, 0x10, 0xB5, 0x09, 0x1F, 0x68, 0x4A, 0x00, 0x20, 0x48, 0x60, 0x10, 0x70,
+ 0x08, 0x70, 0x48, 0x70, 0xFF, 0xF7, 0x9C, 0xF8, 0xFF, 0xF7, 0x8B, 0xF9, 0xBD, 0xE8, 0x10, 0x40,
+ 0x00, 0xF0, 0x02, 0xB8, 0xFF, 0xF7, 0x26, 0xBA, 0x68, 0x48, 0x10, 0xB5, 0x00, 0x68, 0x6B, 0x49,
+ 0x6B, 0x4B, 0xB0, 0xF8, 0x00, 0x23, 0x0A, 0x80, 0xB0, 0xF8, 0x22, 0x13, 0x69, 0x4C, 0x19, 0x80,
+ 0xB0, 0xF8, 0x42, 0x33, 0x23, 0x80, 0x68, 0x4C, 0x22, 0x80, 0x68, 0x4A, 0x11, 0x80, 0x68, 0x49,
+ 0x0B, 0x80, 0x5A, 0x49, 0xB0, 0xF8, 0xBC, 0x02, 0x08, 0x80, 0x10, 0xBD, 0x4C, 0x48, 0x00, 0x22,
+ 0x43, 0x78, 0x4E, 0x48, 0x00, 0x1F, 0x02, 0x2B, 0x01, 0x78, 0x47, 0x48, 0x00, 0xD2, 0x21, 0xB1,
+ 0x82, 0x70, 0x02, 0x2B, 0x00, 0xD2, 0x01, 0xB1, 0xC2, 0x70, 0x83, 0x79, 0x0B, 0x43, 0x01, 0xD0,
+ 0x02, 0x71, 0x42, 0x71, 0x70, 0x47, 0x2D, 0xE9, 0xFF, 0x4F, 0x9B, 0x46, 0xDD, 0xE9, 0x0D, 0x19,
+ 0x86, 0x46, 0x64, 0xE0, 0x01, 0x9A, 0x01, 0xFB, 0x09, 0xF0, 0x02, 0xEB, 0x40, 0x04, 0x02, 0x9A,
+ 0x54, 0x4F, 0x02, 0xEB, 0x40, 0x06, 0x4F, 0xF0, 0x01, 0x0C, 0x0E, 0xEB, 0x40, 0x05, 0x7B, 0x0C,
+ 0xE2, 0x46, 0x4A, 0x46, 0x15, 0xE0, 0x00, 0xBF, 0x36, 0xF8, 0x12, 0x00, 0x34, 0xF8, 0x12, 0x80,
+ 0xA0, 0xEB, 0x08, 0x00, 0x00, 0xB2, 0x98, 0x42, 0x00, 0xDA, 0x03, 0x46, 0xB8, 0x42, 0x00, 0xDD,
+ 0x07, 0x46, 0x00, 0x28, 0x02, 0xDD, 0x4F, 0xF0, 0x00, 0x0A, 0x02, 0xE0, 0x01, 0xDA, 0x4F, 0xF0,
+ 0x00, 0x0C, 0x52, 0x1E, 0xE8, 0xD2, 0xBC, 0xF1, 0x00, 0x0F, 0x04, 0xD0, 0xBA, 0xF1, 0x00, 0x0F,
+ 0x01, 0xD0, 0x00, 0x20, 0x0B, 0xE0, 0x5C, 0xEA, 0x0A, 0x00, 0x08, 0xD0, 0xBC, 0xF1, 0x00, 0x0F,
+ 0x04, 0xD1, 0xBA, 0xF1, 0x00, 0x0F, 0x01, 0xD0, 0x38, 0x46, 0x00, 0xE0, 0x18, 0x46, 0x00, 0x27,
+ 0x3B, 0x46, 0x4A, 0x46, 0xCB, 0xF1, 0x00, 0x0A, 0x10, 0xE0, 0x00, 0xBF, 0x36, 0xF8, 0x12, 0xC0,
+ 0x34, 0xF8, 0x12, 0x80, 0xAC, 0xEB, 0x08, 0x0C, 0xAC, 0xEB, 0x00, 0x0C, 0x0F, 0xFA, 0x8C, 0xFC,
+ 0xDC, 0x45, 0x03, 0xDA, 0xD4, 0x45, 0x01, 0xDD, 0x67, 0x44, 0x5B, 0x1C, 0x52, 0x1E, 0xED, 0xD2,
+ 0x03, 0xB9, 0x01, 0x23, 0x97, 0xFB, 0xF3, 0xF2, 0x02, 0x44, 0x48, 0x46, 0x05, 0xE0, 0x00, 0xBF,
+ 0x34, 0xF8, 0x10, 0x30, 0x13, 0x44, 0x25, 0xF8, 0x10, 0x30, 0x40, 0x1E, 0xF8, 0xD2, 0x49, 0x1E,
+ 0x98, 0xD2, 0xBD, 0xE8, 0xFF, 0x8F, 0x70, 0xB5, 0x00, 0x24, 0x26, 0x46, 0x47, 0xE0, 0x00, 0xBF,
+ 0x31, 0xF9, 0x14, 0x50, 0x95, 0x42, 0x3F, 0xDA, 0x20, 0xF8, 0x14, 0x60, 0x3E, 0xE0, 0x00, 0x00,
+ 0x78, 0x12, 0x01, 0x20, 0xE0, 0x06, 0x10, 0x00, 0xEC, 0x06, 0x10, 0x00, 0x0C, 0x05, 0x10, 0x00,
+ 0x4C, 0x07, 0x10, 0x00, 0xDC, 0x06, 0x10, 0x00, 0x2A, 0x6C, 0x01, 0x00, 0x2C, 0x05, 0x10, 0x00,
+ 0x0E, 0x05, 0x10, 0x00, 0x80, 0x24, 0x10, 0x00, 0x74, 0x06, 0x10, 0x00, 0xC1, 0x06, 0x10, 0x00,
+ 0x58, 0x06, 0x10, 0x00, 0x2C, 0x06, 0x10, 0x00, 0x60, 0x06, 0x10, 0x00, 0x76, 0x06, 0x10, 0x00,
+ 0x5C, 0x06, 0x10, 0x00, 0x50, 0x07, 0x10, 0x00, 0xBF, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0xBA, 0x06, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0xCA, 0x06, 0x10, 0x00, 0x68, 0x06, 0x10, 0x00,
+ 0x6A, 0x06, 0x10, 0x00, 0x6C, 0x06, 0x10, 0x00, 0x6E, 0x06, 0x10, 0x00, 0x70, 0x06, 0x10, 0x00,
+ 0x72, 0x06, 0x10, 0x00, 0x00, 0x80, 0xFF, 0xFF, 0x20, 0xF8, 0x14, 0x50, 0x64, 0x1C, 0x9C, 0x42,
+ 0xB6, 0xDB, 0x70, 0xBD, 0x7F, 0x48, 0x00, 0x21, 0x01, 0x70, 0x41, 0x80, 0x70, 0x47, 0x30, 0xB5,
+ 0x7D, 0x49, 0x02, 0x46, 0x01, 0x20, 0x0C, 0x68, 0x94, 0xF8, 0xC6, 0x39, 0x4B, 0xB1, 0x79, 0x4D,
+ 0xB2, 0xF8, 0x44, 0x10, 0x6D, 0x68, 0x49, 0x1B, 0x00, 0xD5, 0x49, 0x42, 0x8B, 0x42, 0x00, 0xDA,
+ 0x00, 0x20, 0x94, 0xF8, 0xC7, 0x19, 0x00, 0x29, 0x12, 0xD0, 0x72, 0x4C, 0xB2, 0xF9, 0x1A, 0x50,
+ 0x08, 0x34, 0xB2, 0xF9, 0x18, 0x20, 0xB4, 0xF9, 0x02, 0x30, 0xB4, 0xF9, 0x00, 0x40, 0x5B, 0x1B,
+ 0xA2, 0x1A, 0x5B, 0x43, 0x02, 0xFB, 0x02, 0x32, 0x49, 0x43, 0x8A, 0x42, 0x00, 0xDD, 0x00, 0x20,
+ 0x30, 0xBD, 0x2D, 0xE9, 0xF0, 0x47, 0x68, 0x4F, 0xDF, 0xF8, 0x98, 0x81, 0x06, 0x00, 0x4F, 0xF0,
+ 0x05, 0x04, 0x4F, 0xF0, 0x06, 0x05, 0x10, 0xD0, 0x96, 0xF8, 0x3E, 0x01, 0x00, 0xF0, 0x74, 0xF9,
+ 0x04, 0x46, 0x38, 0x68, 0x90, 0xF8, 0xC5, 0x09, 0x18, 0xB1, 0xB6, 0xF8, 0x44, 0x10, 0x81, 0x42,
+ 0x2B, 0xD8, 0xB0, 0x69, 0x00, 0xF0, 0x8A, 0xF8, 0x38, 0xB3, 0x98, 0xF8, 0x00, 0x00, 0x4F, 0xF0,
+ 0x00, 0x09, 0x07, 0x28, 0x0E, 0xD2, 0xDF, 0xE8, 0x00, 0xF0, 0x04, 0x04, 0x23, 0x3B, 0x54, 0x04,
+ 0x04, 0x00, 0x05, 0x2C, 0x04, 0xD0, 0x04, 0x2C, 0x06, 0xD0, 0x01, 0x2C, 0x06, 0xD0, 0x14, 0xE0,
+ 0x88, 0xF8, 0x00, 0x90, 0xBD, 0xE8, 0xF0, 0x87, 0x01, 0x20, 0x51, 0xE0, 0xFF, 0xF7, 0x9A, 0xFF,
+ 0x02, 0x20, 0x88, 0xF8, 0x00, 0x00, 0xA8, 0xF8, 0x02, 0x90, 0xB6, 0xF8, 0x44, 0x00, 0xC8, 0xF8,
+ 0x04, 0x00, 0xB0, 0x69, 0xC8, 0xF8, 0x08, 0x00, 0xEC, 0xE7, 0x88, 0xF8, 0x00, 0x50, 0xE9, 0xE7,
+ 0xB8, 0xF8, 0x02, 0x00, 0x40, 0x1C, 0x80, 0xB2, 0xA8, 0xF8, 0x02, 0x00, 0x03, 0x2C, 0x03, 0xD0,
+ 0x39, 0x68, 0x91, 0xF8, 0xC0, 0x19, 0x2F, 0xE0, 0x03, 0x20, 0x88, 0xF8, 0x00, 0x00, 0xA8, 0xF8,
+ 0x02, 0x90, 0xD7, 0xE7, 0x05, 0x2C, 0xD5, 0xD0, 0x30, 0x46, 0xFF, 0xF7, 0x78, 0xFF, 0x15, 0xE0,
+ 0xB8, 0xF8, 0x02, 0x10, 0x49, 0x1C, 0x89, 0xB2, 0xA8, 0xF8, 0x02, 0x10, 0x01, 0x2C, 0x05, 0xD0,
+ 0x38, 0x68, 0x90, 0xF8, 0xC1, 0x29, 0x8A, 0x42, 0xD7, 0xD9, 0xC3, 0xE7, 0x30, 0x46, 0xFF, 0xF7,
+ 0x66, 0xFF, 0x04, 0x22, 0x88, 0xF8, 0x00, 0x20, 0xA8, 0xF8, 0x02, 0x90, 0x00, 0x28, 0xCC, 0xD0,
+ 0xB8, 0xE7, 0xB8, 0xF8, 0x02, 0x00, 0x40, 0x1C, 0x80, 0xB2, 0xA8, 0xF8, 0x02, 0x00, 0x03, 0x2C,
+ 0x05, 0xD0, 0x39, 0x68, 0x91, 0xF8, 0xC2, 0x19, 0x81, 0x42, 0xD3, 0xD8, 0xBD, 0xE7, 0x05, 0x20,
+ 0x88, 0xF8, 0x00, 0x00, 0xA6, 0xE7, 0x38, 0xB5, 0x00, 0x21, 0x00, 0x91, 0x69, 0x46, 0x00, 0xF0,
+ 0xCE, 0xF8, 0x20, 0x4C, 0x01, 0x28, 0x02, 0xD9, 0x06, 0x20, 0x20, 0x70, 0x05, 0xE0, 0x00, 0x98,
+ 0xFF, 0xF7, 0x67, 0xFF, 0x20, 0x78, 0x05, 0x28, 0x01, 0xD0, 0x00, 0x20, 0x38, 0xBD, 0x1B, 0x48,
+ 0x01, 0x21, 0x41, 0x70, 0x05, 0x21, 0x81, 0x70, 0x01, 0x20, 0x38, 0xBD, 0xF0, 0xB5, 0x16, 0x49,
+ 0x04, 0x46, 0x01, 0x20, 0x0A, 0x68, 0x92, 0xF8, 0xC3, 0x19, 0x92, 0xF8, 0xC3, 0x39, 0x8E, 0x00,
+ 0x92, 0xF8, 0xC4, 0x19, 0x8D, 0x00, 0x92, 0xF8, 0x36, 0x10, 0xCF, 0x07, 0x51, 0x8E, 0xA1, 0xEB,
+ 0x83, 0x01, 0x93, 0x8E, 0x92, 0xF8, 0xC4, 0x29, 0x09, 0xB2, 0xA3, 0xEB, 0x82, 0x02, 0x12, 0xB2,
+ 0x0C, 0xD0, 0x13, 0x46, 0x22, 0x14, 0xB2, 0x42, 0x06, 0xDB, 0x9A, 0x42, 0x04, 0xDC, 0x22, 0xB2,
+ 0xAA, 0x42, 0x01, 0xDB, 0x8A, 0x42, 0x00, 0xDD, 0x00, 0x20, 0xF0, 0xBD, 0x0B, 0x46, 0x11, 0x46,
+ 0xF0, 0xE7, 0x00, 0x00, 0x30, 0x05, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0xA0, 0x24, 0x10, 0x00,
+ 0x75, 0x48, 0x90, 0xF8, 0x2C, 0x00, 0x06, 0x28, 0x01, 0xD0, 0x05, 0x28, 0x07, 0xD1, 0x73, 0x48,
+ 0x73, 0x49, 0x00, 0x68, 0x90, 0xF8, 0xB4, 0x02, 0x00, 0xF0, 0x0F, 0x00, 0x08, 0x60, 0x71, 0x49,
+ 0x00, 0x20, 0x08, 0x70, 0x70, 0x47, 0x2D, 0xE9, 0xF0, 0x41, 0xDF, 0xF8, 0xB8, 0x81, 0x04, 0x46,
+ 0x98, 0xF8, 0x00, 0x00, 0x10, 0xB9, 0x0A, 0x21, 0x88, 0xF8, 0x00, 0x10, 0x00, 0x27, 0x04, 0xF5,
+ 0x80, 0x56, 0x01, 0x25, 0xD6, 0xF8, 0xB0, 0x03, 0x05, 0xFA, 0x07, 0xF1, 0x08, 0x42, 0x0C, 0xD0,
+ 0xC7, 0xEB, 0xC7, 0x00, 0x00, 0xEB, 0x40, 0x01, 0x04, 0xEB, 0x01, 0x11, 0x91, 0xF8, 0x3E, 0x01,
+ 0x03, 0x28, 0x02, 0xD1, 0x20, 0x46, 0x03, 0xF0, 0x83, 0xF9, 0x7F, 0x1C, 0xFF, 0xB2, 0x0F, 0x2F,
+ 0xE8, 0xD3, 0x00, 0x26, 0x04, 0xF5, 0x80, 0x57, 0x05, 0xFA, 0x06, 0xF1, 0xD7, 0xF8, 0xB0, 0x03,
+ 0x08, 0x42, 0x0A, 0xD0, 0xC6, 0xEB, 0xC6, 0x00, 0x00, 0xEB, 0x40, 0x01, 0x04, 0xEB, 0x01, 0x10,
+ 0x00, 0xF1, 0x18, 0x01, 0x0C, 0x30, 0x0A, 0xF0, 0x48, 0xFA, 0x76, 0x1C, 0xF6, 0xB2, 0x0F, 0x2E,
+ 0xEA, 0xD3, 0x51, 0x48, 0x00, 0x21, 0x01, 0x60, 0x41, 0x60, 0x98, 0xF8, 0x08, 0x00, 0x80, 0x06,
+ 0x08, 0xD5, 0x20, 0x46, 0xFF, 0xF7, 0x57, 0xFF, 0x20, 0xB1, 0x4B, 0x48, 0x09, 0xF0, 0x55, 0xFC,
+ 0xFF, 0xF7, 0x98, 0xFE, 0x00, 0x27, 0xA8, 0x46, 0x04, 0xF5, 0x80, 0x56, 0x08, 0xFA, 0x07, 0xF0,
+ 0xD6, 0xF8, 0xB0, 0x13, 0x01, 0x42, 0x11, 0xD0, 0xC7, 0xEB, 0xC7, 0x00, 0x00, 0xEB, 0x40, 0x01,
+ 0x04, 0xEB, 0x01, 0x15, 0x95, 0xF8, 0x3E, 0x01, 0x08, 0x28, 0x07, 0xD1, 0x29, 0x46, 0x20, 0x46,
+ 0x03, 0xF0, 0x59, 0xF9, 0x29, 0x46, 0x20, 0x46, 0x03, 0xF0, 0x1C, 0xF9, 0x7F, 0x1C, 0xFF, 0xB2,
+ 0x0F, 0x2F, 0xE3, 0xD3, 0x20, 0x46, 0xBD, 0xE8, 0xF0, 0x41, 0x00, 0xF0, 0x38, 0xB8, 0x30, 0xB5,
+ 0x05, 0x46, 0x00, 0x20, 0x4F, 0xEA, 0x00, 0x03, 0xC3, 0xEB, 0xC3, 0x02, 0x02, 0xEB, 0x42, 0x02,
+ 0x05, 0xEB, 0x02, 0x14, 0x94, 0xF8, 0x3E, 0x21, 0x01, 0x2A, 0x0D, 0xD0, 0x02, 0x2A, 0x0B, 0xD0,
+ 0x03, 0x2A, 0x09, 0xD0, 0x04, 0x2A, 0x07, 0xD0, 0x05, 0x2A, 0x05, 0xD0, 0x06, 0x2A, 0x03, 0xD0,
+ 0x07, 0x2A, 0x01, 0xD0, 0x08, 0x2A, 0x02, 0xD1, 0x40, 0x1C, 0xC0, 0xB2, 0x0C, 0x60, 0x5B, 0x1C,
+ 0xDB, 0xB2, 0x0F, 0x2B, 0xE0, 0xD3, 0x30, 0xBD, 0x09, 0x28, 0x0E, 0xD2, 0xDF, 0xE8, 0x00, 0xF0,
+ 0x0D, 0x05, 0x05, 0x07, 0x09, 0x09, 0x09, 0x0B, 0x0B, 0x00, 0x04, 0x20, 0x70, 0x47, 0x01, 0x20,
+ 0x70, 0x47, 0x02, 0x20, 0x70, 0x47, 0x03, 0x20, 0x70, 0x47, 0x05, 0x20, 0x70, 0x47, 0x70, 0xB5,
+ 0x1A, 0x49, 0x0A, 0x78, 0x17, 0x49, 0x06, 0x2A, 0x05, 0xD0, 0x22, 0xB1, 0x05, 0x2A, 0x02, 0xD0,
+ 0x0A, 0x20, 0x08, 0x70, 0x70, 0xBD, 0x0A, 0x78, 0x22, 0xB1, 0x52, 0x1E, 0x12, 0xF0, 0xFF, 0x02,
+ 0x0A, 0x70, 0xF7, 0xD1, 0x05, 0x46, 0x00, 0x24, 0xC4, 0xEB, 0xC4, 0x00, 0x00, 0xEB, 0x40, 0x00,
+ 0x05, 0xEB, 0x00, 0x16, 0x31, 0x46, 0x28, 0x46, 0x03, 0xF0, 0xBC, 0xF8, 0x31, 0x46, 0x28, 0x46,
+ 0x03, 0xF0, 0xF1, 0xF8, 0x64, 0x1C, 0xE4, 0xB2, 0x0F, 0x2C, 0xED, 0xD3, 0xBD, 0xE8, 0x70, 0x40,
+ 0x10, 0x21, 0x01, 0x20, 0x0D, 0xF0, 0xB7, 0xBA, 0x7C, 0x6C, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x78, 0x06, 0x10, 0x00, 0x40, 0x05, 0x10, 0x00, 0xA0, 0x24, 0x10, 0x00, 0x30, 0x05, 0x10, 0x00,
+ 0x70, 0x47, 0xA5, 0x4A, 0x30, 0xB4, 0x12, 0x68, 0x92, 0xF8, 0xA4, 0x23, 0x12, 0x09, 0x80, 0xF8,
+ 0x28, 0x20, 0x0A, 0x68, 0x02, 0x62, 0x4A, 0x68, 0x42, 0x62, 0xB1, 0xF9, 0x02, 0x20, 0xD3, 0x17,
+ 0x1B, 0x04, 0x43, 0xEA, 0x12, 0x43, 0x12, 0x04, 0x82, 0x60, 0xC3, 0x60, 0xB1, 0xF9, 0x00, 0x10,
+ 0xCA, 0x17, 0x12, 0x04, 0x42, 0xEA, 0x11, 0x42, 0x09, 0x04, 0x01, 0x60, 0x42, 0x60, 0x10, 0x22,
+ 0x01, 0x46, 0x30, 0xBC, 0x10, 0x30, 0x0F, 0xF0, 0x15, 0xBB, 0x2D, 0xE9, 0xF0, 0x4F, 0x92, 0x4F,
+ 0x81, 0x46, 0x8A, 0x46, 0x38, 0x68, 0x8F, 0xB0, 0x16, 0x46, 0xB0, 0xF8, 0xA6, 0x13, 0x90, 0xF8,
+ 0xAA, 0x53, 0x01, 0xFB, 0x01, 0xFB, 0xB0, 0xF8, 0xA8, 0x13, 0x01, 0xFB, 0x01, 0xF8, 0x0F, 0xF0,
+ 0x53, 0xFC, 0x00, 0x28, 0x38, 0x68, 0x02, 0xD0, 0x90, 0xF8, 0xAC, 0x43, 0x01, 0xE0, 0x90, 0xF8,
+ 0xAB, 0x43, 0xAC, 0x42, 0x00, 0xD2, 0x25, 0x46, 0x00, 0x2E, 0x02, 0xDB, 0x89, 0xF8, 0x28, 0x60,
+ 0x19, 0xE0, 0x09, 0xF1, 0x24, 0x01, 0x99, 0xF8, 0x28, 0x60, 0x08, 0x1F, 0x0A, 0xF0, 0x11, 0xF9,
+ 0x58, 0x45, 0x01, 0xDB, 0x26, 0x46, 0x02, 0xE0, 0x40, 0x45, 0x00, 0xDC, 0x2E, 0x46, 0x38, 0x68,
+ 0x99, 0xF8, 0x28, 0x10, 0x90, 0xF8, 0xA4, 0x03, 0x00, 0xF0, 0x0F, 0x02, 0x30, 0x46, 0x0A, 0xF0,
+ 0x23, 0xF9, 0x89, 0xF8, 0x28, 0x00, 0x99, 0xF8, 0x28, 0x10, 0x74, 0x48, 0x00, 0xEB, 0x81, 0x00,
+ 0x01, 0x68, 0xCA, 0x17, 0xCD, 0xE9, 0x07, 0x21, 0x41, 0x6C, 0xCA, 0x17, 0xCD, 0xE9, 0x04, 0x12,
+ 0xD0, 0xF8, 0xCC, 0x20, 0xD0, 0xF8, 0x88, 0x50, 0xD4, 0x17, 0x01, 0x94, 0x93, 0x46, 0xDD, 0xE9,
+ 0x07, 0x21, 0xD9, 0xF8, 0x08, 0x30, 0xD9, 0xF8, 0x0C, 0x40, 0xA3, 0xFB, 0x01, 0x07, 0x04, 0xFB,
+ 0x01, 0x71, 0x03, 0xFB, 0x02, 0x11, 0xEE, 0x17, 0x4F, 0xF4, 0x80, 0x32, 0x00, 0x23, 0x0F, 0xF0,
+ 0x8D, 0xFA, 0xBA, 0xF9, 0x02, 0x30, 0xB9, 0xF9, 0x26, 0x20, 0x01, 0x9C, 0x9F, 0x18, 0xA7, 0xFB,
+ 0x0B, 0x38, 0x4F, 0xEA, 0xE7, 0x7C, 0x0C, 0xFB, 0x0B, 0x82, 0x07, 0xFB, 0x04, 0x24, 0xB9, 0xF9,
+ 0x22, 0x20, 0xA2, 0xFB, 0x05, 0xC8, 0xD7, 0x17, 0x07, 0xFB, 0x05, 0x87, 0x13, 0xEB, 0x0C, 0x03,
+ 0x02, 0xFB, 0x06, 0x72, 0x54, 0x41, 0x1F, 0x1A, 0x64, 0xEB, 0x01, 0x04, 0xDD, 0xE9, 0x04, 0x12,
+ 0xD9, 0xF8, 0x18, 0x30, 0xD9, 0xF8, 0x1C, 0xC0, 0xA3, 0xFB, 0x01, 0x08, 0x0C, 0xFB, 0x01, 0x81,
+ 0x03, 0xFB, 0x02, 0x11, 0x4F, 0xF4, 0x80, 0x32, 0x00, 0x23, 0x0F, 0xF0, 0x5F, 0xFA, 0x38, 0x1A,
+ 0x64, 0xEB, 0x01, 0x04, 0xCD, 0xE9, 0x0D, 0x04, 0x06, 0x94, 0x0A, 0x90, 0xDD, 0xE9, 0x07, 0x21,
+ 0xD9, 0xF8, 0x00, 0x30, 0xD9, 0xF8, 0x04, 0x40, 0xA3, 0xFB, 0x01, 0x07, 0x04, 0xFB, 0x01, 0x71,
+ 0x03, 0xFB, 0x02, 0x11, 0x4F, 0xF4, 0x80, 0x32, 0x00, 0x23, 0x0F, 0xF0, 0x47, 0xFA, 0xBA, 0xF9,
+ 0x00, 0x30, 0xB9, 0xF9, 0x24, 0x20, 0x01, 0x9C, 0x13, 0x44, 0xA3, 0xFB, 0x0B, 0xC8, 0xDF, 0x17,
+ 0x07, 0xFB, 0x0B, 0x82, 0x03, 0xFB, 0x04, 0x27, 0xB9, 0xF9, 0x20, 0x20, 0xA2, 0xFB, 0x05, 0x48,
+ 0xD3, 0x17, 0x03, 0xFB, 0x05, 0x83, 0x02, 0xFB, 0x06, 0x32, 0x1C, 0xEB, 0x04, 0x03, 0x57, 0x41,
+ 0x1C, 0x1A, 0x67, 0xEB, 0x01, 0x07, 0xDD, 0xE9, 0x04, 0x12, 0xD9, 0xF8, 0x10, 0x30, 0xD9, 0xF8,
+ 0x14, 0x50, 0xA3, 0xFB, 0x01, 0x06, 0x05, 0xFB, 0x01, 0x61, 0x03, 0xFB, 0x02, 0x11, 0x4F, 0xF4,
+ 0x80, 0x32, 0x00, 0x23, 0x0F, 0xF0, 0x1A, 0xFA, 0x25, 0x1A, 0x67, 0xEB, 0x01, 0x07, 0xCD, 0xE9,
+ 0x0B, 0x57, 0xD9, 0xF8, 0x20, 0x10, 0xC9, 0xF8, 0x24, 0x10, 0xDA, 0xF8, 0x00, 0x10, 0xC9, 0xF8,
+ 0x20, 0x10, 0x10, 0x22, 0x49, 0x46, 0x09, 0xF1, 0x10, 0x00, 0x0F, 0xF0, 0x2B, 0xFA, 0x10, 0x22,
+ 0x0B, 0xA9, 0x48, 0x46, 0x0F, 0xF0, 0x26, 0xFA, 0x0A, 0x98, 0x4F, 0xF4, 0x00, 0x41, 0x06, 0x9C,
+ 0x40, 0x18, 0x44, 0xF1, 0x00, 0x01, 0x4F, 0xF4, 0x80, 0x32, 0x00, 0x23, 0x0F, 0xF0, 0xF6, 0xF9,
+ 0xAA, 0xF8, 0x02, 0x00, 0x4F, 0xF4, 0x00, 0x40, 0x28, 0x18, 0x47, 0xF1, 0x00, 0x01, 0x4F, 0xF4,
+ 0x80, 0x32, 0x00, 0x23, 0x0F, 0xF0, 0xEA, 0xF9, 0xAA, 0xF8, 0x00, 0x00, 0x0F, 0xB0, 0xBD, 0xE8,
+ 0xF0, 0x8F, 0x70, 0xB5, 0x0D, 0x46, 0x04, 0x46, 0x01, 0x46, 0x10, 0x22, 0x10, 0x30, 0x0F, 0xF0,
+ 0x01, 0xFA, 0xB5, 0xF9, 0x02, 0x00, 0xC1, 0x17, 0x09, 0x04, 0x41, 0xEA, 0x10, 0x41, 0x00, 0x04,
+ 0xA0, 0x60, 0xE1, 0x60, 0xB5, 0xF9, 0x00, 0x00, 0xC1, 0x17, 0x09, 0x04, 0x41, 0xEA, 0x10, 0x41,
+ 0x00, 0x04, 0x20, 0x60, 0x61, 0x60, 0x70, 0xBD, 0x20, 0x07, 0x10, 0x00, 0x1A, 0x6B, 0x01, 0x00,
+ 0x30, 0xB5, 0x2A, 0x49, 0x0B, 0x68, 0x93, 0xF8, 0x14, 0x14, 0x89, 0x07, 0x17, 0xD5, 0x41, 0xF2,
+ 0xB0, 0x34, 0x0E, 0x21, 0x24, 0x58, 0x01, 0x22, 0x02, 0xFA, 0x01, 0xF5, 0x2C, 0x42, 0x0F, 0xD0,
+ 0xC1, 0xEB, 0xC1, 0x05, 0x05, 0xEB, 0x45, 0x05, 0x00, 0xEB, 0x05, 0x15, 0x95, 0xF8, 0x3F, 0x51,
+ 0x6D, 0x07, 0x05, 0xD5, 0x1E, 0x48, 0x93, 0xF8, 0x15, 0x14, 0x01, 0x70, 0x42, 0x70, 0x30, 0xBD,
+ 0x49, 0x1E, 0xE9, 0xD2, 0x30, 0xBD, 0x19, 0x49, 0x09, 0x68, 0x91, 0xF8, 0x14, 0x14, 0x89, 0x07,
+ 0x09, 0xD5, 0x17, 0x49, 0x10, 0xB1, 0x00, 0x20, 0x48, 0x70, 0x03, 0xE0, 0x08, 0x78, 0x00, 0x28,
+ 0x01, 0xD0, 0x40, 0x1E, 0x08, 0x70, 0x70, 0x47, 0x10, 0x49, 0x09, 0x68, 0x91, 0xF8, 0x14, 0x14,
+ 0x8A, 0x07, 0x06, 0xD5, 0x0E, 0x49, 0x4A, 0x78, 0x01, 0x2A, 0x08, 0xD0, 0x09, 0x78, 0x31, 0xB9,
+ 0x70, 0x47, 0xC9, 0x07, 0xFC, 0xD0, 0x90, 0xF8, 0x3F, 0x11, 0x04, 0x29, 0xF8, 0xD1, 0x90, 0xF8,
+ 0x3E, 0x11, 0x03, 0x29, 0x05, 0xD0, 0x04, 0x29, 0x01, 0xD0, 0x06, 0x29, 0xF0, 0xD1, 0x08, 0x21,
+ 0x00, 0xE0, 0x02, 0x21, 0x80, 0xF8, 0x3E, 0x11, 0x70, 0x47, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x58, 0x05, 0x10, 0x00, 0x70, 0xB5, 0x04, 0x46, 0x00, 0x20, 0x84, 0xF8, 0x42, 0x01, 0xE9, 0x48,
+ 0xE9, 0x4D, 0x0E, 0x46, 0x01, 0x78, 0x28, 0x68, 0x11, 0xB9, 0xE8, 0x49, 0x09, 0x78, 0x41, 0xB1,
+ 0x94, 0xF9, 0x4A, 0x11, 0x90, 0xF8, 0x63, 0x02, 0xB1, 0x42, 0x0D, 0xD1, 0x00, 0xF0, 0x0F, 0x00,
+ 0x0B, 0xE0, 0x0F, 0xF0, 0xC9, 0xFA, 0x00, 0x28, 0x94, 0xF9, 0x4A, 0x01, 0x0D, 0xD0, 0xB0, 0x42,
+ 0x28, 0x68, 0x90, 0xF8, 0x9B, 0x03, 0xF1, 0xD0, 0x00, 0x09, 0x84, 0xF8, 0x42, 0x01, 0x20, 0x46,
+ 0x00, 0xF0, 0x2A, 0xF9, 0x01, 0x28, 0x06, 0xD0, 0x0F, 0xE0, 0xB0, 0x42, 0x28, 0x68, 0x90, 0xF8,
+ 0x9A, 0x03, 0xE3, 0xD0, 0xF0, 0xE7, 0x29, 0x68, 0x94, 0xF8, 0x42, 0x01, 0x91, 0xF8, 0x9C, 0x13,
+ 0x01, 0xF0, 0x0F, 0x01, 0x08, 0x44, 0x84, 0xF8, 0x42, 0x01, 0xD1, 0x48, 0xD1, 0x49, 0x00, 0x78,
+ 0x09, 0x78, 0x08, 0x43, 0x07, 0xD0, 0x29, 0x68, 0x94, 0xF8, 0x42, 0x01, 0x91, 0xF8, 0x9D, 0x13,
+ 0x08, 0x44, 0x84, 0xF8, 0x42, 0x01, 0x70, 0xBD, 0xCB, 0x49, 0x00, 0x20, 0x08, 0x70, 0xCB, 0x49,
+ 0x08, 0x70, 0xCB, 0x49, 0x08, 0x70, 0x70, 0x47, 0x2D, 0xE9, 0xF8, 0x4F, 0x00, 0xF5, 0x80, 0x55,
+ 0x04, 0x46, 0xB5, 0xF8, 0xB0, 0x03, 0x4F, 0xF0, 0x00, 0x0A, 0x00, 0x04, 0xC2, 0x4E, 0xD1, 0x46,
+ 0x00, 0x0C, 0x0D, 0xD0, 0x20, 0x46, 0xFF, 0xF7, 0x43, 0xFF, 0x30, 0x46, 0x4F, 0xF0, 0xFF, 0x37,
+ 0x00, 0x78, 0x39, 0x46, 0x4F, 0xF0, 0x00, 0x48, 0x4F, 0xF0, 0x01, 0x0B, 0x18, 0xBB, 0x08, 0xE0,
+ 0xBA, 0x49, 0x00, 0x20, 0x30, 0x70, 0x08, 0x70, 0xBD, 0xE8, 0xF8, 0x4F, 0x01, 0x20, 0xFF, 0xF7,
+ 0x52, 0xBF, 0x04, 0xF5, 0x80, 0x52, 0x00, 0x20, 0x92, 0xF9, 0xB8, 0x63, 0x22, 0xE0, 0x00, 0xBF,
+ 0xD2, 0xF8, 0xB0, 0xC3, 0x0B, 0xFA, 0x00, 0xF3, 0x1C, 0xEA, 0x03, 0x0F, 0x18, 0xD0, 0xC0, 0xEB,
+ 0xC0, 0x03, 0x03, 0xEB, 0x43, 0x03, 0x04, 0xEB, 0x03, 0x13, 0x93, 0xF8, 0x3E, 0xC1, 0xBC, 0xF1,
+ 0x01, 0x0F, 0x01, 0xD0, 0x39, 0x46, 0x0F, 0xE0, 0xB3, 0xF8, 0x44, 0xC0, 0xBC, 0xF1, 0x00, 0x0F,
+ 0x06, 0xD0, 0x5B, 0x6B, 0x93, 0xFB, 0xFC, 0xF3, 0x98, 0x45, 0x01, 0xDA, 0x98, 0x46, 0x01, 0x46,
+ 0x40, 0x1C, 0x40, 0xB2, 0x86, 0x42, 0xDB, 0xDA, 0x00, 0x27, 0xDF, 0xF8, 0x6C, 0x82, 0x00, 0x91,
+ 0x9A, 0xE0, 0x00, 0xBF, 0xD5, 0xF8, 0xB0, 0x13, 0x0B, 0xFA, 0x07, 0xF0, 0x01, 0x42, 0x72, 0xD0,
+ 0xC7, 0xEB, 0xC7, 0x00, 0x00, 0xEB, 0x40, 0x00, 0x04, 0xEB, 0x00, 0x16, 0x96, 0xF8, 0x3E, 0x01,
+ 0x01, 0x28, 0x02, 0xD0, 0x05, 0x28, 0x08, 0xD0, 0x10, 0xE0, 0x30, 0x46, 0x00, 0x99, 0xFF, 0xF7,
+ 0x41, 0xFF, 0x02, 0x20, 0x86, 0xF8, 0x3E, 0x01, 0x11, 0xE0, 0x30, 0x46, 0x00, 0xF0, 0xA8, 0xF8,
+ 0x06, 0x20, 0x86, 0xF8, 0x3E, 0x01, 0x30, 0x46, 0x00, 0xF0, 0xDB, 0xF8, 0x96, 0xF8, 0x3E, 0x01,
+ 0x02, 0x28, 0x04, 0xD0, 0x06, 0x28, 0x30, 0xD0, 0x07, 0x28, 0x39, 0xD0, 0x3E, 0xE0, 0x96, 0xF8,
+ 0x42, 0x00, 0xC0, 0x07, 0x12, 0xD0, 0x30, 0x46, 0x00, 0xF0, 0x76, 0xF8, 0x70, 0xB9, 0xD8, 0xF8,
+ 0x00, 0x00, 0x96, 0xF8, 0x42, 0x11, 0x90, 0xF8, 0x9C, 0x03, 0x00, 0xF0, 0x0F, 0x00, 0x81, 0x42,
+ 0x01, 0xD9, 0x08, 0x1A, 0x00, 0xE0, 0x00, 0x20, 0x86, 0xF8, 0x42, 0x01, 0x78, 0x48, 0x00, 0x78,
+ 0x40, 0xB1, 0x96, 0xF8, 0x42, 0x01, 0x08, 0xBB, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x9D, 0x03,
+ 0x86, 0xF8, 0x42, 0x01, 0x96, 0xF8, 0x42, 0x01, 0xC0, 0xB9, 0x03, 0x20, 0x86, 0xF8, 0x3E, 0x01,
+ 0x71, 0x48, 0x01, 0x78, 0x49, 0x1C, 0x01, 0x70, 0x10, 0xE0, 0x96, 0xF8, 0x43, 0x01, 0x68, 0xB9,
+ 0x96, 0xF8, 0x42, 0x00, 0x80, 0x07, 0x03, 0xD5, 0x07, 0x20, 0x86, 0xF8, 0x3E, 0x01, 0x05, 0xE0,
+ 0x08, 0x20, 0x86, 0xF8, 0x3E, 0x01, 0x30, 0x46, 0x00, 0xF0, 0x8C, 0xF8, 0x30, 0x46, 0xFF, 0xF7,
+ 0xC3, 0xFE, 0x96, 0xF8, 0x3E, 0x01, 0x02, 0x28, 0x06, 0xD0, 0x06, 0x28, 0x0B, 0xD0, 0x03, 0x28,
+ 0x10, 0xD0, 0x04, 0x28, 0x0E, 0xD0, 0x1D, 0xE0, 0x96, 0xF8, 0x42, 0x01, 0xD0, 0xB1, 0x40, 0x1E,
+ 0x86, 0xF8, 0x42, 0x01, 0x16, 0xE0, 0x96, 0xF8, 0x43, 0x01, 0x98, 0xB1, 0x40, 0x1E, 0x86, 0xF8,
+ 0x43, 0x01, 0x0F, 0xE0, 0x96, 0xF8, 0x3F, 0x01, 0x01, 0x28, 0x02, 0xD0, 0x03, 0x28, 0x05, 0xD0,
+ 0x08, 0xE0, 0x09, 0xF1, 0x01, 0x00, 0x00, 0xF0, 0xFF, 0x09, 0x03, 0xE0, 0x0A, 0xF1, 0x01, 0x00,
+ 0x00, 0xF0, 0xFF, 0x0A, 0x7F, 0x1C, 0x7F, 0xB2, 0x95, 0xF9, 0xB8, 0x03, 0xB8, 0x42, 0xBF, 0xF6,
+ 0x61, 0xAF, 0x00, 0x20, 0xFF, 0xF7, 0x7F, 0xFE, 0x4E, 0x49, 0x81, 0xF8, 0x00, 0xA0, 0x4E, 0x49,
+ 0x81, 0xF8, 0x00, 0x90, 0xBD, 0xE8, 0xF8, 0x8F, 0x90, 0xF8, 0x5B, 0x10, 0x90, 0xF8, 0x5A, 0x20,
+ 0x8A, 0x1A, 0xB0, 0xF8, 0x42, 0x10, 0x52, 0x1C, 0x02, 0x2A, 0x0B, 0xD9, 0x90, 0xF8, 0x5D, 0x20,
+ 0x90, 0xF8, 0x5C, 0x30, 0x9A, 0x42, 0x05, 0xD1, 0x41, 0xF0, 0x01, 0x01, 0xA0, 0xF8, 0x42, 0x10,
+ 0x01, 0x20, 0x70, 0x47, 0x21, 0xF0, 0x01, 0x01, 0xA0, 0xF8, 0x42, 0x10, 0x00, 0x20, 0x70, 0x47,
+ 0x70, 0xB5, 0x04, 0x46, 0x00, 0x20, 0x84, 0xF8, 0x43, 0x01, 0x32, 0x48, 0x32, 0x4D, 0x01, 0x78,
+ 0x28, 0x68, 0x51, 0xB1, 0x90, 0xF8, 0x64, 0x02, 0x00, 0xF0, 0x0F, 0x00, 0x84, 0xF8, 0x43, 0x01,
+ 0x94, 0xF8, 0x42, 0x00, 0x80, 0x07, 0x0E, 0xD5, 0x70, 0xBD, 0x2C, 0x49, 0x09, 0x78, 0x00, 0x29,
+ 0xF0, 0xD1, 0x0F, 0xF0, 0x59, 0xF9, 0x00, 0x28, 0x28, 0x68, 0x90, 0xF8, 0x9E, 0x03, 0xEB, 0xD0,
+ 0x00, 0x09, 0x84, 0xF8, 0x43, 0x01, 0x28, 0x68, 0xE1, 0x6A, 0x90, 0xF8, 0x9F, 0x03, 0x00, 0xF0,
+ 0x0F, 0x02, 0x91, 0x42, 0xE8, 0xD9, 0x94, 0xF8, 0x43, 0x11, 0x01, 0xEB, 0x10, 0x10, 0x84, 0xF8,
+ 0x43, 0x01, 0x70, 0xBD, 0x20, 0x49, 0x08, 0x78, 0x00, 0x28, 0x01, 0xD0, 0x40, 0x1E, 0x08, 0x70,
+ 0x70, 0x47, 0x2D, 0xE9, 0xF0, 0x41, 0x18, 0x4D, 0x04, 0x46, 0x28, 0x68, 0x90, 0xF8, 0x81, 0x03,
+ 0xC0, 0x07, 0x24, 0xD0, 0x21, 0x46, 0x20, 0x1D, 0x09, 0xF0, 0x03, 0xFE, 0x07, 0x46, 0x21, 0x46,
+ 0x04, 0xF1, 0x20, 0x00, 0x09, 0xF0, 0xFD, 0xFD, 0x06, 0x46, 0x0F, 0xF0, 0x25, 0xF9, 0x00, 0x28,
+ 0x28, 0x68, 0x05, 0xD0, 0xB0, 0xF8, 0x93, 0x13, 0xB0, 0xF8, 0x8F, 0x03, 0x49, 0x43, 0x04, 0xE0,
+ 0xB0, 0xF8, 0x91, 0x13, 0xB0, 0xF8, 0x8D, 0x03, 0x49, 0x43, 0x40, 0x43, 0x86, 0x42, 0x06, 0xDD,
+ 0x8F, 0x42, 0x04, 0xDD, 0x34, 0xF8, 0x42, 0x0F, 0x40, 0xF0, 0x02, 0x00, 0x20, 0x80, 0xBD, 0xE8,
+ 0xF0, 0x81, 0x00, 0x00, 0x1C, 0x05, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0x28, 0x05, 0x10, 0x00,
+ 0x52, 0x06, 0x10, 0x00, 0x52, 0x07, 0x10, 0x00, 0xC6, 0x06, 0x10, 0x00, 0xF3, 0x06, 0x10, 0x00,
+ 0xC5, 0x06, 0x10, 0x00, 0xC7, 0x06, 0x10, 0x00, 0xC8, 0x06, 0x10, 0x00, 0xF8, 0xB5, 0x07, 0x46,
+ 0xFD, 0x48, 0x0E, 0x46, 0xFE, 0x4C, 0x01, 0x78, 0xFC, 0x48, 0x15, 0x46, 0x00, 0x78, 0x01, 0xB9,
+ 0x88, 0xB1, 0x20, 0x68, 0x90, 0xF8, 0xBE, 0x13, 0xB0, 0xF8, 0xC9, 0x23, 0x01, 0xF0, 0x0F, 0x03,
+ 0x0C, 0x09, 0xB0, 0xF8, 0xC7, 0x13, 0x49, 0x43, 0x52, 0x43, 0x30, 0x46, 0x00, 0x94, 0x09, 0xF0,
+ 0xBE, 0xFD, 0x2C, 0x60, 0xF8, 0xBD, 0x0F, 0xF0, 0xD7, 0xF8, 0x58, 0xB1, 0x20, 0x68, 0x90, 0xF8,
+ 0xB4, 0x13, 0xB0, 0xF8, 0xB7, 0x23, 0x01, 0xF0, 0x0F, 0x03, 0x0C, 0x09, 0xB0, 0xF8, 0xB5, 0x13,
+ 0x49, 0x43, 0xE9, 0xE7, 0x97, 0xF8, 0x3F, 0x01, 0x02, 0x28, 0x0E, 0xD0, 0x01, 0x28, 0x20, 0x68,
+ 0x0F, 0xD0, 0x90, 0xF8, 0xB9, 0x13, 0x00, 0xBF, 0x01, 0xF0, 0x0F, 0x03, 0x0C, 0x09, 0xB0, 0xF8,
+ 0xBF, 0x13, 0xB0, 0xF8, 0xC1, 0x23, 0x49, 0x43, 0xD6, 0xE7, 0x20, 0x68, 0x90, 0xF8, 0xBA, 0x13,
+ 0xF2, 0xE7, 0x90, 0xF8, 0xBB, 0x13, 0xB0, 0xF8, 0xC5, 0x23, 0x01, 0xF0, 0x0F, 0x03, 0x0C, 0x09,
+ 0xB0, 0xF8, 0xC3, 0x13, 0x49, 0x43, 0xC7, 0xE7, 0x2D, 0xE9, 0xF8, 0x43, 0x04, 0x46, 0x90, 0xF8,
+ 0x3E, 0x01, 0x88, 0x46, 0x04, 0xF1, 0x78, 0x07, 0x03, 0x28, 0x0B, 0xD0, 0xD4, 0x4E, 0x04, 0x28,
+ 0x0D, 0xD0, 0x07, 0x28, 0x3B, 0xD0, 0x08, 0x28, 0x37, 0xD1, 0x38, 0x46, 0xBD, 0xE8, 0xF8, 0x43,
+ 0x00, 0xF0, 0xCB, 0xBB, 0x38, 0x46, 0xBD, 0xE8, 0xF8, 0x43, 0x00, 0xF0, 0x94, 0xBB, 0x20, 0x46,
+ 0x0F, 0xF0, 0x9F, 0xF8, 0x18, 0xB1, 0x20, 0x46, 0x0F, 0xF0, 0x96, 0xF8, 0x1D, 0xE0, 0x04, 0xF1,
+ 0x0C, 0x01, 0x08, 0x1D, 0x09, 0xF0, 0x55, 0xFD, 0x01, 0x46, 0x6A, 0x46, 0x20, 0x46, 0xFF, 0xF7,
+ 0x85, 0xFF, 0x05, 0x46, 0x94, 0xF8, 0x41, 0x01, 0x80, 0xB9, 0x20, 0x46, 0x0F, 0xF0, 0x86, 0xF8,
+ 0x10, 0xB1, 0x00, 0x98, 0x85, 0x42, 0x09, 0xDB, 0x30, 0x68, 0xD4, 0xF8, 0x80, 0x10, 0x90, 0xF8,
+ 0xBC, 0x03, 0x02, 0x09, 0x28, 0x46, 0x09, 0xF0, 0x5F, 0xFD, 0x05, 0x46, 0x2A, 0x46, 0x41, 0x46,
+ 0x38, 0x46, 0x00, 0xF0, 0x6A, 0xFB, 0xC4, 0xF8, 0x80, 0x50, 0xBD, 0xE8, 0xF8, 0x83, 0x30, 0x68,
+ 0x00, 0x24, 0x90, 0xF8, 0x81, 0x13, 0x89, 0x07, 0x0A, 0xD5, 0x90, 0xF8, 0xBD, 0x03, 0x00, 0xF0,
+ 0x0F, 0x04, 0x0F, 0xF0, 0x51, 0xF8, 0x18, 0xB1, 0x30, 0x68, 0x90, 0xF8, 0xBD, 0x03, 0x04, 0x09,
+ 0x22, 0x46, 0x41, 0x46, 0x38, 0x46, 0xBD, 0xE8, 0xF8, 0x43, 0x00, 0xF0, 0x66, 0xBB, 0x2D, 0xE9,
+ 0xF0, 0x43, 0xDF, 0xF8, 0x9C, 0x92, 0x4F, 0xF0, 0x00, 0x0C, 0xB0, 0xF9, 0x02, 0x30, 0xD9, 0xF8,
+ 0x00, 0x40, 0x67, 0x46, 0x66, 0x46, 0x65, 0x46, 0x00, 0x2B, 0x02, 0xDA, 0x4F, 0xF0, 0x01, 0x0C,
+ 0x04, 0xE0, 0xB4, 0xF8, 0x32, 0x80, 0x98, 0x45, 0x00, 0xDA, 0x01, 0x27, 0xB0, 0xF9, 0x00, 0x30,
+ 0x00, 0x2B, 0x01, 0xDA, 0x01, 0x26, 0x04, 0xE0, 0xB4, 0xF8, 0x34, 0x80, 0x98, 0x45, 0x00, 0xDA,
+ 0x01, 0x25, 0x00, 0x23, 0xBC, 0xF1, 0x00, 0x0F, 0x02, 0xD0, 0x0E, 0xB1, 0x43, 0x80, 0x03, 0xE0,
+ 0x4F, 0xB1, 0x26, 0xB1, 0x61, 0x8E, 0x38, 0xE0, 0x03, 0x80, 0xBD, 0xE8, 0xF0, 0x83, 0x15, 0xB1,
+ 0x62, 0x8E, 0x42, 0x80, 0x04, 0xE0, 0xBC, 0xF1, 0x00, 0x0F, 0x30, 0xD0, 0x2D, 0xB1, 0x43, 0x80,
+ 0xD9, 0xF8, 0x00, 0x10, 0x89, 0x8E, 0x01, 0x80, 0xEF, 0xE7, 0xB2, 0xF8, 0x02, 0xC0, 0xBC, 0xF1,
+ 0x00, 0x0F, 0x24, 0xD0, 0x43, 0x80, 0xB1, 0xF9, 0x02, 0x40, 0x67, 0x42, 0x2C, 0xE0, 0x00, 0xBF,
+ 0x0C, 0xFB, 0x07, 0xFC, 0x00, 0x2C, 0x00, 0xDA, 0x64, 0x42, 0x9C, 0xFB, 0xF4, 0xF4, 0x0F, 0x88,
+ 0x3C, 0x44, 0x04, 0x80, 0x46, 0xB3, 0xB2, 0xF9, 0x00, 0x40, 0x2C, 0xB3, 0xB1, 0xF9, 0x00, 0x50,
+ 0xB2, 0xF9, 0x02, 0x20, 0x6D, 0x42, 0x00, 0xD5, 0x6D, 0x42, 0x6A, 0x43, 0x00, 0x2C, 0x00, 0xDA,
+ 0x64, 0x42, 0x92, 0xFB, 0xF4, 0xF2, 0x49, 0x88, 0x11, 0x44, 0x41, 0x80, 0xC4, 0xE7, 0x00, 0x2F,
+ 0xE8, 0xD0, 0x57, 0x88, 0x00, 0x2F, 0xE5, 0xD0, 0x64, 0x8E, 0x44, 0x80, 0xD9, 0xF8, 0x00, 0x40,
+ 0xB1, 0xF9, 0x02, 0x70, 0x64, 0x8E, 0xE7, 0x1B, 0xB2, 0xF9, 0x02, 0x40, 0xB2, 0xF9, 0x00, 0xC0,
+ 0x00, 0x2F, 0xCD, 0xDA, 0x7F, 0x42, 0xCB, 0xE7, 0x00, 0x2D, 0xAE, 0xD0, 0xB2, 0xF9, 0x00, 0x30,
+ 0x00, 0x2B, 0xAA, 0xD0, 0xD9, 0xF8, 0x00, 0x40, 0xB1, 0xF9, 0x00, 0x60, 0xB2, 0xF9, 0x02, 0x20,
+ 0xA4, 0x8E, 0xA4, 0x1B, 0x00, 0xD5, 0x64, 0x42, 0x62, 0x43, 0x00, 0x2B, 0x00, 0xDA, 0x5B, 0x42,
+ 0x92, 0xFB, 0xF3, 0xF2, 0x49, 0x88, 0x11, 0x44, 0x41, 0x80, 0xA1, 0xE7, 0x2D, 0xE9, 0xF0, 0x5F,
+ 0x0D, 0x46, 0x5C, 0x49, 0x04, 0x46, 0x00, 0x20, 0x48, 0x80, 0x08, 0x80, 0xA1, 0xF1, 0x10, 0x08,
+ 0x28, 0x68, 0xC8, 0xF8, 0x00, 0x00, 0x17, 0x46, 0x4F, 0xF0, 0x01, 0x0A, 0x46, 0x46, 0x29, 0x46,
+ 0x20, 0x46, 0x00, 0xF0, 0x6D, 0xFF, 0xDF, 0xF8, 0x48, 0x91, 0x28, 0x68, 0x70, 0x60, 0xD9, 0xF8,
+ 0x00, 0x00, 0x04, 0xF1, 0x0C, 0x0B, 0x90, 0xF8, 0x80, 0x03, 0x41, 0x07, 0x22, 0xD5, 0x94, 0xF8,
+ 0x3E, 0x01, 0x04, 0xF1, 0x84, 0x06, 0x02, 0x28, 0x0A, 0xD0, 0x03, 0x28, 0x08, 0xD0, 0x04, 0x28,
+ 0x0B, 0xD0, 0x06, 0x28, 0x12, 0xD0, 0x07, 0x28, 0x10, 0xD0, 0x08, 0x28, 0x0E, 0xD0, 0x11, 0xE0,
+ 0x59, 0x46, 0x30, 0x46, 0xFF, 0xF7, 0xED, 0xFA, 0x0C, 0xE0, 0x20, 0x46, 0x00, 0xF0, 0xD9, 0xF9,
+ 0x02, 0x46, 0x29, 0x46, 0x30, 0x46, 0xFF, 0xF7, 0x08, 0xFB, 0x03, 0xE0, 0x29, 0x46, 0x30, 0x46,
+ 0xFF, 0xF7, 0x0F, 0xFC, 0x29, 0x68, 0xC8, 0xF8, 0x08, 0x10, 0xD9, 0xF8, 0x00, 0x00, 0x4E, 0x46,
+ 0x90, 0xF8, 0x80, 0x03, 0x00, 0x07, 0x14, 0xD5, 0x94, 0xF8, 0x42, 0x00, 0x10, 0xF0, 0xC8, 0x0F,
+ 0x05, 0xD0, 0x32, 0x48, 0x00, 0x78, 0x10, 0xB9, 0x2F, 0x48, 0x00, 0x78, 0x20, 0xB1, 0x29, 0x46,
+ 0x20, 0x46, 0x00, 0xF0, 0xE6, 0xF9, 0x04, 0xE0, 0x59, 0x46, 0x04, 0xF1, 0xB0, 0x00, 0x00, 0xF0,
+ 0xEE, 0xFD, 0x29, 0x68, 0xC8, 0xF8, 0x0C, 0x10, 0x30, 0x68, 0x90, 0xF8, 0x80, 0x13, 0x49, 0x06,
+ 0x0F, 0xD4, 0x90, 0xF8, 0x40, 0x14, 0x49, 0x06, 0x03, 0xD5, 0x94, 0xF8, 0x42, 0x10, 0x09, 0x07,
+ 0x07, 0xD4, 0x90, 0xF8, 0x20, 0x04, 0x80, 0x07, 0x08, 0xD5, 0x94, 0xF8, 0x42, 0x00, 0x00, 0x06,
+ 0x04, 0xD5, 0x29, 0x46, 0x20, 0x46, 0x01, 0xF0, 0x37, 0xFE, 0x82, 0x46, 0x30, 0x68, 0x90, 0xF8,
+ 0x80, 0x03, 0x80, 0x06, 0x03, 0xD5, 0x29, 0x46, 0x20, 0x46, 0xFF, 0xF7, 0x7D, 0xFE, 0x30, 0x68,
+ 0x90, 0xF8, 0x80, 0x03, 0x01, 0x07, 0x0B, 0xD5, 0xC0, 0x06, 0x09, 0xD5, 0x94, 0xF8, 0x3E, 0x01,
+ 0x07, 0x28, 0x05, 0xD1, 0x13, 0x4A, 0x04, 0xF1, 0xF4, 0x01, 0x28, 0x46, 0xFF, 0xF7, 0xCF, 0xFE,
+ 0x30, 0x68, 0xB0, 0xF9, 0x32, 0x10, 0xB5, 0xF9, 0x02, 0x00, 0x09, 0xF0, 0xAB, 0xFB, 0x68, 0x80,
+ 0x30, 0x68, 0xB0, 0xF9, 0x34, 0x10, 0xB5, 0xF9, 0x00, 0x00, 0x09, 0xF0, 0xA3, 0xFB, 0x28, 0x80,
+ 0x94, 0xF8, 0x3E, 0x01, 0x03, 0x28, 0x0F, 0xD0, 0x04, 0x28, 0x11, 0xD0, 0x07, 0x28, 0x1F, 0xD0,
+ 0x08, 0x28, 0x1D, 0xD0, 0x1F, 0xE0, 0x00, 0x00, 0x1C, 0x05, 0x10, 0x00, 0x28, 0x05, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0x6A, 0x05, 0x10, 0x00, 0x38, 0x88, 0xA4, 0xF8, 0x50, 0x00, 0x12, 0xE0,
+ 0x30, 0x68, 0x34, 0xF8, 0x50, 0x2F, 0x90, 0xF8, 0xBC, 0x03, 0x00, 0xF0, 0x0F, 0x01, 0xC1, 0xF1,
+ 0x10, 0x00, 0x4A, 0x43, 0x39, 0x88, 0x01, 0xFB, 0x00, 0x20, 0x00, 0x09, 0x20, 0x80, 0x01, 0xE0,
+ 0xB4, 0xF8, 0x50, 0x00, 0x38, 0x80, 0x50, 0x46, 0xBD, 0xE8, 0xF0, 0x9F, 0x2D, 0xE9, 0xF3, 0x4F,
+ 0xE1, 0x49, 0x83, 0xB0, 0x4F, 0xF0, 0x00, 0x08, 0xC1, 0xF8, 0x00, 0x80, 0xDF, 0x49, 0xE0, 0x4D,
+ 0x83, 0x46, 0xC1, 0xF8, 0x00, 0x80, 0x85, 0xF8, 0x00, 0x80, 0x46, 0x46, 0xDF, 0xF8, 0x74, 0xA3,
+ 0x4F, 0xF0, 0x01, 0x09, 0x00, 0xF5, 0x80, 0x57, 0x6A, 0xE0, 0x00, 0xBF, 0xD7, 0xF8, 0xB0, 0x13,
+ 0x09, 0xFA, 0x06, 0xF0, 0x01, 0x42, 0x62, 0xD0, 0xC6, 0xEB, 0xC6, 0x00, 0x00, 0xEB, 0x40, 0x00,
+ 0x0B, 0xEB, 0x00, 0x14, 0x94, 0xF8, 0x3E, 0x01, 0x03, 0x28, 0x05, 0xD0, 0x04, 0x28, 0x03, 0xD0,
+ 0x07, 0x28, 0x01, 0xD0, 0x08, 0x28, 0x52, 0xD1, 0xE0, 0x68, 0x00, 0x90, 0xB4, 0xF8, 0x4E, 0x00,
+ 0xAD, 0xF8, 0x04, 0x00, 0x04, 0xF1, 0x0C, 0x01, 0xC7, 0x4A, 0x08, 0x1D, 0x00, 0xF0, 0xF8, 0xF8,
+ 0x01, 0xAA, 0x69, 0x46, 0x20, 0x46, 0xFF, 0xF7, 0xE9, 0xFE, 0x02, 0x90, 0xC3, 0x4A, 0x69, 0x46,
+ 0x04, 0xF1, 0x24, 0x00, 0x00, 0xF0, 0xEC, 0xF8, 0xDA, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x00, 0x04,
+ 0xC0, 0x07, 0x0E, 0xD0, 0x94, 0xF8, 0x42, 0x00, 0x40, 0x07, 0x02, 0xD5, 0x00, 0x20, 0x02, 0x90,
+ 0x07, 0xE0, 0x94, 0xF8, 0x34, 0x01, 0x01, 0x28, 0x03, 0xD1, 0x20, 0x6A, 0x00, 0x90, 0x84, 0xF8,
+ 0x34, 0x81, 0x94, 0xF8, 0x3F, 0x01, 0x03, 0x28, 0x09, 0xD1, 0x28, 0x78, 0x40, 0x1C, 0xC0, 0xB2,
+ 0x28, 0x70, 0x94, 0xF8, 0x42, 0x10, 0x49, 0x07, 0x01, 0xD5, 0x40, 0x1E, 0x28, 0x70, 0x00, 0x98,
+ 0x60, 0x62, 0xBD, 0xF8, 0x04, 0x00, 0xA4, 0xF8, 0x50, 0x00, 0x02, 0x98, 0x01, 0x28, 0x0E, 0xD1,
+ 0x00, 0x98, 0xA0, 0x61, 0xBD, 0xF8, 0x04, 0x00, 0xA4, 0xF8, 0x52, 0x00, 0x04, 0x98, 0x94, 0xF8,
+ 0x4A, 0x21, 0x01, 0x68, 0x09, 0xFA, 0x02, 0xF0, 0x01, 0x43, 0x04, 0x98, 0x01, 0x60, 0x76, 0x1C,
+ 0x97, 0xF9, 0xB8, 0x03, 0xB0, 0x42, 0x91, 0xDA, 0xDA, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x00, 0x04,
+ 0x00, 0x07, 0x0E, 0xD5, 0x29, 0x78, 0xA0, 0x48, 0x29, 0xB1, 0xA0, 0x49, 0x09, 0x78, 0x01, 0x29,
+ 0x01, 0xD1, 0x80, 0xF8, 0x00, 0x90, 0xD7, 0xF8, 0xB4, 0x13, 0x00, 0x29, 0x01, 0xD1, 0x80, 0xF8,
+ 0x00, 0x80, 0x05, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F, 0x2D, 0xE9, 0xF0, 0x47, 0x89, 0x46, 0x06, 0x46,
+ 0x00, 0x27, 0x4F, 0xF0, 0x01, 0x08, 0x00, 0xF5, 0x80, 0x55, 0x3C, 0xE0, 0xD5, 0xF8, 0xB0, 0x13,
+ 0x08, 0xFA, 0x07, 0xF2, 0x11, 0x42, 0x35, 0xD0, 0xC7, 0xEB, 0xC7, 0x01, 0x01, 0xEB, 0x41, 0x01,
+ 0x06, 0xEB, 0x01, 0x14, 0x94, 0xF8, 0x3E, 0x11, 0x03, 0x29, 0x05, 0xD0, 0x04, 0x29, 0x03, 0xD0,
+ 0x07, 0x29, 0x01, 0xD0, 0x08, 0x29, 0x25, 0xD1, 0x94, 0xF8, 0x4A, 0x31, 0xD9, 0xF8, 0x00, 0x20,
+ 0x08, 0xFA, 0x03, 0xF0, 0x02, 0x42, 0x1D, 0xD0, 0x03, 0x29, 0x03, 0xD1, 0x21, 0x46, 0x30, 0x46,
+ 0x02, 0xF0, 0x3E, 0xFA, 0x94, 0xF9, 0x4B, 0x01, 0x00, 0x28, 0x13, 0xDB, 0x94, 0xF8, 0x3F, 0x01,
+ 0x00, 0x21, 0x01, 0x28, 0x04, 0xD0, 0x02, 0x28, 0x04, 0xD0, 0x04, 0x28, 0x04, 0xD0, 0x04, 0xE0,
+ 0x03, 0x21, 0x02, 0xE0, 0x04, 0x21, 0x00, 0xE0, 0x05, 0x21, 0x84, 0xF8, 0x40, 0x11, 0x20, 0x46,
+ 0x08, 0xF0, 0x04, 0xFD, 0x7F, 0x1C, 0x95, 0xF9, 0xB8, 0x03, 0xB8, 0x42, 0xBE, 0xDA, 0x00, 0x24,
+ 0x47, 0x46, 0x24, 0xE0, 0xD5, 0xF8, 0xB0, 0x03, 0x07, 0xFA, 0x04, 0xF1, 0x08, 0x42, 0x1D, 0xD0,
+ 0xC4, 0xEB, 0xC4, 0x00, 0x00, 0xEB, 0x40, 0x00, 0x06, 0xEB, 0x00, 0x18, 0x98, 0xF8, 0x3E, 0x01,
+ 0x08, 0x28, 0x13, 0xD1, 0x98, 0xF8, 0x4A, 0x21, 0xD9, 0xF8, 0x00, 0x10, 0x07, 0xFA, 0x02, 0xF0,
+ 0x01, 0x42, 0x07, 0xD0, 0x98, 0xF9, 0x4B, 0x01, 0x00, 0x28, 0x03, 0xDB, 0x41, 0x46, 0x30, 0x46,
+ 0x02, 0xF0, 0x19, 0xFA, 0x41, 0x46, 0x30, 0x46, 0x02, 0xF0, 0xDC, 0xF9, 0x64, 0x1C, 0x95, 0xF9,
+ 0xB8, 0x03, 0xA0, 0x42, 0xD6, 0xDA, 0xBD, 0xE8, 0xF0, 0x87, 0x57, 0x49, 0x10, 0xB5, 0x00, 0x20,
+ 0x08, 0x60, 0x56, 0x49, 0x08, 0x60, 0xFF, 0xF7, 0x2B, 0xF9, 0x00, 0xF0, 0x39, 0xFC, 0xBD, 0xE8,
+ 0x10, 0x40, 0x00, 0xF0, 0xB7, 0xB8, 0x38, 0xB5, 0x04, 0x46, 0x00, 0x20, 0x00, 0x90, 0x69, 0x46,
+ 0x20, 0x46, 0xFF, 0xF7, 0xD3, 0xFE, 0x69, 0x46, 0x20, 0x46, 0xFF, 0xF7, 0x6D, 0xFF, 0x38, 0xBD,
+ 0x10, 0xB5, 0x14, 0x46, 0x09, 0xF0, 0x75, 0xFA, 0x21, 0x68, 0x81, 0x42, 0x00, 0xDA, 0x20, 0x60,
+ 0x10, 0xBD, 0x4B, 0x49, 0x10, 0xB5, 0x0B, 0x78, 0x46, 0x49, 0x02, 0x46, 0x4F, 0xF0, 0xFF, 0x30,
+ 0x09, 0x68, 0x13, 0xB1, 0x91, 0xF9, 0xAA, 0x03, 0x10, 0xBD, 0x46, 0x4B, 0x1B, 0x78, 0x00, 0x2B,
+ 0xF8, 0xD1, 0xB2, 0xF8, 0x42, 0x30, 0x5C, 0x06, 0x03, 0xD5, 0x91, 0xF8, 0xA5, 0x03, 0x00, 0x09,
+ 0x10, 0xBD, 0x1C, 0x07, 0x02, 0xD5, 0x91, 0xF8, 0x51, 0x04, 0x03, 0xE0, 0x1B, 0x06, 0x04, 0xD5,
+ 0x91, 0xF8, 0x2A, 0x04, 0x00, 0xF0, 0x0F, 0x00, 0x10, 0xBD, 0x92, 0xF8, 0x3F, 0x21, 0x01, 0x2A,
+ 0x04, 0xD0, 0x39, 0x4A, 0x12, 0x78, 0x01, 0x2A, 0xDC, 0xD8, 0x10, 0xBD, 0x91, 0xF8, 0xA5, 0x03,
+ 0xF0, 0xE7, 0x30, 0xB4, 0x0C, 0x46, 0x90, 0xF8, 0x3E, 0x11, 0x00, 0xF1, 0xB0, 0x03, 0x03, 0x29,
+ 0x23, 0xD0, 0x04, 0x29, 0x03, 0xD0, 0x06, 0x29, 0x01, 0xD0, 0x07, 0x29, 0x41, 0xD1, 0x29, 0x4A,
+ 0x12, 0x68, 0x92, 0xF8, 0x80, 0x23, 0xD2, 0x06, 0x2D, 0xD5, 0x90, 0xF8, 0x5B, 0x20, 0x90, 0xF8,
+ 0x5A, 0x50, 0x52, 0x1B, 0xC0, 0xF8, 0x24, 0x21, 0x90, 0xF8, 0x5D, 0x20, 0x90, 0xF8, 0x5C, 0x50,
+ 0x52, 0x1B, 0xC0, 0xF8, 0x28, 0x21, 0x82, 0x6B, 0xC0, 0xF8, 0x20, 0x21, 0x23, 0x4A, 0x07, 0x29,
+ 0x12, 0x88, 0xC0, 0xF8, 0x2C, 0x21, 0x06, 0xD0, 0x0F, 0xE0, 0x00, 0xF1, 0x0C, 0x01, 0x30, 0xBC,
+ 0x18, 0x46, 0x00, 0xF0, 0xDC, 0xBB, 0x90, 0xF8, 0x31, 0x11, 0x03, 0x29, 0x05, 0xD9, 0x21, 0x46,
+ 0x30, 0xBC, 0x1B, 0x4A, 0x18, 0x46, 0x00, 0xF0, 0x6A, 0xBC, 0x90, 0xF8, 0x31, 0x11, 0x49, 0x1C,
+ 0x80, 0xF8, 0x31, 0x11, 0x03, 0xE0, 0x90, 0xF8, 0x31, 0x21, 0x03, 0x2A, 0x04, 0xD9, 0x21, 0x46,
+ 0x30, 0xBC, 0x18, 0x46, 0x00, 0xF0, 0x28, 0xBC, 0x04, 0x29, 0x04, 0xD0, 0x00, 0x21, 0x80, 0xF8,
+ 0x31, 0x11, 0x30, 0xBC, 0x70, 0x47, 0x52, 0x1C, 0x80, 0xF8, 0x31, 0x21, 0x21, 0x46, 0x30, 0xBC,
+ 0x18, 0x46, 0x00, 0xF0, 0x02, 0xBC, 0x00, 0x00, 0xF4, 0x06, 0x10, 0x00, 0xF8, 0x06, 0x10, 0x00,
+ 0xF3, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0x81, 0x06, 0x10, 0x00, 0x80, 0x06, 0x10, 0x00,
+ 0x1C, 0x05, 0x10, 0x00, 0x28, 0x05, 0x10, 0x00, 0xC6, 0x06, 0x10, 0x00, 0x54, 0x06, 0x10, 0x00,
+ 0x6A, 0x05, 0x10, 0x00, 0x70, 0x47, 0x00, 0xF0, 0x35, 0xB8, 0x7C, 0xB5, 0x04, 0x46, 0x15, 0x46,
+ 0x0E, 0x46, 0x68, 0x46, 0x00, 0xF0, 0x2E, 0xF8, 0x01, 0x99, 0x2A, 0x46, 0x60, 0x68, 0x00, 0xF0,
+ 0x3B, 0xF8, 0x60, 0x60, 0x00, 0x99, 0x2A, 0x46, 0x20, 0x68, 0x00, 0xF0, 0x35, 0xF8, 0x20, 0x60,
+ 0x21, 0x46, 0x30, 0x46, 0x00, 0xF0, 0x27, 0xF8, 0x7C, 0xBD, 0x7C, 0xB5, 0x04, 0x46, 0x15, 0x46,
+ 0x0E, 0x46, 0x68, 0x46, 0x00, 0xF0, 0x16, 0xF8, 0x01, 0x99, 0x2A, 0x46, 0x60, 0x68, 0x00, 0xF0,
+ 0x23, 0xF8, 0x60, 0x60, 0x00, 0x99, 0x2A, 0x46, 0x20, 0x68, 0x00, 0xF0, 0x1D, 0xF8, 0x20, 0x60,
+ 0x21, 0x46, 0x30, 0x46, 0x00, 0xF0, 0x0F, 0xF8, 0x7C, 0xBD, 0x0A, 0x46, 0x01, 0x46, 0x10, 0x46,
+ 0x00, 0xF0, 0x09, 0xB8, 0xB1, 0xF9, 0x02, 0x20, 0x12, 0x02, 0x42, 0x60, 0xB1, 0xF9, 0x00, 0x10,
+ 0x09, 0x02, 0x01, 0x60, 0x70, 0x47, 0x4A, 0x68, 0x80, 0x32, 0x12, 0x12, 0x42, 0x80, 0x09, 0x68,
+ 0x80, 0x31, 0x09, 0x12, 0x01, 0x80, 0x70, 0x47, 0xC2, 0xF1, 0x10, 0x03, 0x50, 0x43, 0x01, 0xFB,
+ 0x03, 0x00, 0x00, 0x11, 0x70, 0x47, 0x00, 0x00, 0x2D, 0xE9, 0xF0, 0x41, 0xFE, 0x4A, 0x12, 0x68,
+ 0x51, 0xB3, 0x51, 0x8E, 0x92, 0xF8, 0x30, 0x30, 0x82, 0x6F, 0x04, 0x6F, 0xC0, 0x6F, 0x64, 0x26,
+ 0x40, 0x00, 0x00, 0xFB, 0x02, 0x40, 0x48, 0x43, 0xA0, 0xFB, 0x00, 0x25, 0x00, 0x21, 0x01, 0xFB,
+ 0x00, 0x55, 0x00, 0xFB, 0x01, 0x51, 0xA2, 0xFB, 0x06, 0x07, 0x00, 0x25, 0x01, 0xFB, 0x06, 0x71,
+ 0x02, 0xFB, 0x05, 0x11, 0x5C, 0x43, 0x22, 0x46, 0x2B, 0x46, 0x0E, 0xF0, 0x85, 0xFA, 0x22, 0x46,
+ 0x00, 0x23, 0x0E, 0xF0, 0x81, 0xFA, 0xED, 0x4B, 0x00, 0x22, 0x1B, 0x1A, 0x8A, 0x41, 0x01, 0xD2,
+ 0xEA, 0x48, 0x00, 0x21, 0xBD, 0xE8, 0xF0, 0x81, 0x91, 0x8E, 0x92, 0xF8, 0x31, 0x30, 0x42, 0x6F,
+ 0xD3, 0xE7, 0x7C, 0xB5, 0x04, 0x46, 0xE6, 0x48, 0xB4, 0xF9, 0x02, 0x10, 0x00, 0x78, 0x01, 0x28,
+ 0x30, 0xD0, 0xB4, 0xF9, 0x06, 0x00, 0xB4, 0xF9, 0x0A, 0x20, 0x09, 0x1A, 0x80, 0x1A, 0x08, 0x1A,
+ 0x01, 0x90, 0xB4, 0xF9, 0x00, 0x10, 0xB4, 0xF9, 0x04, 0x00, 0xB4, 0xF9, 0x08, 0x20, 0x09, 0x1A,
+ 0x80, 0x1A, 0x08, 0x1A, 0xD8, 0x49, 0x00, 0x90, 0x20, 0x6C, 0x09, 0x68, 0x01, 0x9B, 0x91, 0xF8,
+ 0xB3, 0x23, 0x50, 0x43, 0xC2, 0xF1, 0x80, 0x02, 0x53, 0x43, 0xC2, 0x17, 0x00, 0xEB, 0x52, 0x60,
+ 0x03, 0xEB, 0xE0, 0x10, 0x20, 0x64, 0x91, 0xF8, 0xB3, 0x13, 0xE0, 0x6B, 0x00, 0x9A, 0x48, 0x43,
+ 0xC1, 0xF1, 0x80, 0x01, 0x4A, 0x43, 0xC1, 0x17, 0x00, 0xEB, 0x51, 0x60, 0x02, 0xEB, 0xE0, 0x10,
+ 0xE0, 0x63, 0x7C, 0xBD, 0xB4, 0xF9, 0x46, 0x00, 0xB4, 0xF9, 0x4A, 0x20, 0x09, 0x1A, 0x80, 0x1A,
+ 0x08, 0x1A, 0x01, 0x90, 0xB4, 0xF9, 0x00, 0x10, 0xB4, 0xF9, 0x44, 0x00, 0xB4, 0xF9, 0x48, 0x20,
+ 0x09, 0x1A, 0x80, 0x1A, 0x08, 0x1A, 0x00, 0x90, 0x04, 0xF1, 0x50, 0x00, 0x03, 0x22, 0x69, 0x46,
+ 0x05, 0x46, 0x09, 0xF0, 0xDC, 0xF8, 0x94, 0xF8, 0x80, 0x00, 0x03, 0x28, 0x02, 0xD2, 0x40, 0x1C,
+ 0x84, 0xF8, 0x80, 0x00, 0xC2, 0xB2, 0x29, 0x46, 0x04, 0xF1, 0x68, 0x00, 0x00, 0xF0, 0x98, 0xFB,
+ 0x7C, 0xBD, 0x2D, 0xE9, 0xF0, 0x4F, 0x89, 0xB0, 0x00, 0x26, 0x05, 0x96, 0x06, 0x96, 0x07, 0x96,
+ 0x01, 0x24, 0x0D, 0xF1, 0x14, 0x08, 0x08, 0x96, 0x02, 0x25, 0xCD, 0xE9, 0x01, 0x48, 0x07, 0x46,
+ 0xCD, 0xE9, 0x03, 0x54, 0x00, 0xF1, 0x0C, 0x03, 0xAD, 0x48, 0x9B, 0x46, 0x2A, 0x46, 0x29, 0x46,
+ 0x20, 0x30, 0x00, 0x95, 0x09, 0xF0, 0x2C, 0xF8, 0xDF, 0xF8, 0xA4, 0x92, 0x4F, 0xF4, 0x80, 0x7A,
+ 0x99, 0xF8, 0x00, 0x00, 0x01, 0x28, 0x48, 0xD0, 0x38, 0x6C, 0x00, 0x90, 0x01, 0x23, 0x02, 0x22,
+ 0x07, 0xA9, 0x09, 0xF1, 0x08, 0x00, 0x08, 0xF0, 0xF2, 0xFF, 0x07, 0xA9, 0x01, 0x23, 0x02, 0x22,
+ 0x08, 0x46, 0xCD, 0xF8, 0x00, 0xA0, 0x08, 0xF0, 0xDC, 0xFF, 0x02, 0x23, 0x5A, 0x46, 0x07, 0xA9,
+ 0x05, 0xA8, 0x00, 0x94, 0x08, 0xF0, 0xF0, 0xFF, 0x05, 0x96, 0x06, 0x96, 0x07, 0x96, 0x08, 0x96,
+ 0xCD, 0xE9, 0x01, 0x48, 0xCD, 0xE9, 0x03, 0x54, 0x95, 0x48, 0x07, 0xF1, 0x14, 0x03, 0x00, 0x95,
+ 0x02, 0x22, 0x1D, 0x46, 0x11, 0x46, 0x20, 0x30, 0x08, 0xF0, 0xFA, 0xFF, 0x99, 0xF8, 0x00, 0x00,
+ 0x01, 0x28, 0x1F, 0xD0, 0xF8, 0x6B, 0x00, 0x90, 0x8D, 0x48, 0x01, 0x23, 0x02, 0x22, 0x07, 0xA9,
+ 0x08, 0x30, 0x08, 0xF0, 0xC4, 0xFF, 0x07, 0xA9, 0x01, 0x23, 0x02, 0x22, 0x08, 0x46, 0xCD, 0xF8,
+ 0x00, 0xA0, 0x08, 0xF0, 0xAE, 0xFF, 0x02, 0x23, 0x2A, 0x46, 0x07, 0xA9, 0x05, 0xA8, 0x00, 0x94,
+ 0x08, 0xF0, 0xC2, 0xFF, 0x09, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F, 0xDD, 0xE9, 0x05, 0x01, 0xC7, 0xE9,
+ 0x03, 0x01, 0xC9, 0xE7, 0xDD, 0xE9, 0x05, 0x01, 0xC7, 0xE9, 0x05, 0x01, 0xF2, 0xE7, 0x2D, 0xE9,
+ 0xF0, 0x4F, 0x8D, 0xB0, 0x00, 0x26, 0x02, 0x24, 0x05, 0x96, 0x06, 0x96, 0x07, 0x96, 0x08, 0x96,
+ 0x09, 0x96, 0x0A, 0x96, 0x0B, 0x96, 0x05, 0xAF, 0x0C, 0x96, 0xCD, 0xE9, 0x01, 0x47, 0x05, 0x46,
+ 0x73, 0x48, 0x05, 0xF1, 0x1C, 0x03, 0x03, 0x94, 0x04, 0x94, 0x9B, 0x46, 0x22, 0x46, 0x21, 0x46,
+ 0x20, 0x30, 0x00, 0x94, 0x08, 0xF0, 0xB4, 0xFF, 0x0D, 0xF1, 0x24, 0x0A, 0xCD, 0xE9, 0x01, 0x4A,
+ 0x6B, 0x4B, 0x02, 0x22, 0x03, 0x94, 0x04, 0x94, 0x30, 0x33, 0x11, 0x46, 0x05, 0xA8, 0x00, 0x94,
+ 0x08, 0xF0, 0xA6, 0xFF, 0xDF, 0xF8, 0x98, 0x81, 0x4F, 0xF0, 0x04, 0x09, 0x98, 0xF8, 0x00, 0x00,
+ 0x01, 0x28, 0x12, 0xD1, 0xE8, 0x6E, 0x02, 0x23, 0x08, 0xF1, 0x60, 0x01, 0x00, 0x90, 0x1A, 0x46,
+ 0xA1, 0xF1, 0x10, 0x00, 0x08, 0xF0, 0x6B, 0xFF, 0x02, 0x23, 0x08, 0xF1, 0x60, 0x01, 0x1A, 0x46,
+ 0x08, 0x46, 0xCD, 0xF8, 0x00, 0x90, 0x08, 0xF0, 0x54, 0xFF, 0x59, 0x49, 0x02, 0x23, 0x5A, 0x46,
+ 0x60, 0x31, 0x09, 0xA8, 0x00, 0x94, 0x08, 0xF0, 0x67, 0xFF, 0x05, 0x96, 0x06, 0x96, 0x07, 0x96,
+ 0x08, 0x96, 0x09, 0x96, 0x0A, 0x96, 0x0B, 0x96, 0x0C, 0x96, 0xCD, 0xE9, 0x01, 0x47, 0x50, 0x48,
+ 0x05, 0xF1, 0x2C, 0x03, 0x02, 0x22, 0x03, 0x94, 0x04, 0x94, 0x1E, 0x46, 0x11, 0x46, 0x20, 0x30,
+ 0x00, 0x94, 0x08, 0xF0, 0x6D, 0xFF, 0xCD, 0xE9, 0x01, 0x4A, 0x49, 0x4B, 0x02, 0x22, 0x03, 0x94,
+ 0x04, 0x94, 0x30, 0x33, 0x11, 0x46, 0x05, 0xA8, 0x00, 0x94, 0x08, 0xF0, 0x61, 0xFF, 0x98, 0xF8,
+ 0x00, 0x00, 0x01, 0x28, 0x12, 0xD1, 0x42, 0x49, 0xA8, 0x6E, 0x02, 0x23, 0x60, 0x31, 0x00, 0x90,
+ 0x1A, 0x46, 0xA1, 0xF1, 0x10, 0x00, 0x08, 0xF0, 0x2A, 0xFF, 0x3D, 0x49, 0x02, 0x23, 0x60, 0x31,
+ 0x1A, 0x46, 0x08, 0x46, 0xCD, 0xF8, 0x00, 0x90, 0x08, 0xF0, 0x13, 0xFF, 0x38, 0x49, 0x02, 0x23,
+ 0x32, 0x46, 0x60, 0x31, 0x09, 0xA8, 0x00, 0x94, 0x08, 0xF0, 0x26, 0xFF, 0x0D, 0xB0, 0x62, 0xE7,
+ 0x2D, 0xE9, 0xF0, 0x4F, 0x89, 0xB0, 0x00, 0x26, 0x01, 0x24, 0x05, 0x96, 0x06, 0x96, 0x0D, 0xF1,
+ 0x14, 0x0A, 0x07, 0x96, 0xCD, 0xE9, 0x01, 0x4A, 0x05, 0x46, 0x00, 0xF1, 0x0C, 0x03, 0x4F, 0xF0,
+ 0x02, 0x09, 0x2B, 0x48, 0x93, 0x46, 0x0F, 0x46, 0x03, 0x94, 0x98, 0x46, 0x4A, 0x46, 0x21, 0x46,
+ 0x10, 0x30, 0xCD, 0xF8, 0x00, 0x90, 0x04, 0x94, 0x08, 0xF0, 0x22, 0xFF, 0xB5, 0xF9, 0x02, 0x00,
+ 0x05, 0x99, 0x01, 0x23, 0x40, 0x1A, 0x05, 0x90, 0x00, 0x90, 0x02, 0x22, 0x06, 0xA9, 0x38, 0x46,
+ 0x08, 0xF0, 0xED, 0xFE, 0x64, 0x27, 0x41, 0x46, 0x01, 0x23, 0x02, 0x22, 0x08, 0x46, 0x00, 0x97,
+ 0x08, 0xF0, 0xE5, 0xFE, 0x42, 0x46, 0x02, 0x23, 0x06, 0xA9, 0x10, 0x46, 0x00, 0x94, 0x08, 0xF0,
+ 0xEB, 0xFE, 0x41, 0x46, 0x01, 0x23, 0x02, 0x22, 0x08, 0x46, 0x00, 0x97, 0x08, 0xF0, 0xC9, 0xFE,
+ 0x05, 0x96, 0x06, 0x96, 0x07, 0x96, 0xCD, 0xE9, 0x01, 0x4A, 0x11, 0x48, 0x05, 0xF1, 0x14, 0x03,
+ 0x03, 0x94, 0x1E, 0x46, 0x02, 0x22, 0x01, 0x21, 0x10, 0x30, 0xCD, 0xF8, 0x00, 0x90, 0x04, 0x94,
+ 0x08, 0xF0, 0xEE, 0xFE, 0xB5, 0xF9, 0x00, 0x00, 0x05, 0x99, 0x01, 0x23, 0x40, 0x1A, 0x05, 0x90,
+ 0x00, 0x90, 0x02, 0x22, 0x06, 0xA9, 0x58, 0x46, 0x08, 0xF0, 0xB9, 0xFE, 0x01, 0x23, 0x31, 0x46,
+ 0x02, 0x22, 0x08, 0x46, 0x00, 0x97, 0x05, 0xE0, 0x20, 0x07, 0x10, 0x00, 0x40, 0x42, 0x0F, 0x00,
+ 0x70, 0x05, 0x10, 0x00, 0x08, 0xF0, 0xAB, 0xFE, 0x32, 0x46, 0x02, 0x23, 0x06, 0xA9, 0x10, 0x46,
+ 0x00, 0x94, 0x08, 0xF0, 0xB1, 0xFE, 0x31, 0x46, 0x01, 0x23, 0x02, 0x22, 0x08, 0x46, 0x00, 0x97,
+ 0x08, 0xF0, 0x8F, 0xFE, 0xE6, 0xE6, 0x2D, 0xE9, 0xF7, 0x4F, 0x8A, 0xB0, 0x00, 0x26, 0x05, 0x96,
+ 0x06, 0x96, 0x07, 0x96, 0x08, 0x96, 0x01, 0x24, 0x4F, 0xF0, 0x02, 0x09, 0x0D, 0xF1, 0x14, 0x0A,
+ 0x09, 0x96, 0xCD, 0xE9, 0x01, 0x4A, 0xCD, 0xE9, 0x03, 0x94, 0x07, 0x46, 0x07, 0xF1, 0x1C, 0x00,
+ 0x4D, 0x46, 0xFD, 0x4B, 0x4A, 0x46, 0x49, 0x46, 0x80, 0x46, 0xCD, 0xF8, 0x00, 0x90, 0x08, 0xF0,
+ 0xA7, 0xFE, 0x07, 0xA8, 0xCD, 0xE9, 0x01, 0x50, 0xF7, 0x48, 0xCD, 0xE9, 0x03, 0x45, 0x43, 0x46,
+ 0x02, 0x22, 0x01, 0x21, 0x08, 0x38, 0x00, 0x95, 0x08, 0xF0, 0x9A, 0xFE, 0x0D, 0xF1, 0x24, 0x0B,
+ 0xCD, 0xE9, 0x01, 0x4B, 0x03, 0x94, 0xF0, 0x4B, 0x02, 0x22, 0x01, 0x21, 0x07, 0xA8, 0x00, 0x95,
+ 0x04, 0x94, 0x08, 0xF0, 0x8D, 0xFE, 0xEC, 0x4D, 0x18, 0x3D, 0x28, 0x78, 0x01, 0x28, 0x04, 0xD1,
+ 0x01, 0x21, 0x38, 0x46, 0xFF, 0xF7, 0xA8, 0xFD, 0x68, 0x60, 0x09, 0x98, 0x69, 0x68, 0x4F, 0xF0,
+ 0x64, 0x08, 0x08, 0x44, 0x05, 0xA9, 0x09, 0x90, 0x01, 0x23, 0x02, 0x22, 0xCD, 0xF8, 0x00, 0x80,
+ 0x08, 0x46, 0x08, 0xF0, 0x4C, 0xFE, 0x09, 0x98, 0x00, 0x90, 0x01, 0x23, 0x02, 0x22, 0x05, 0xA8,
+ 0x0B, 0x99, 0x08, 0xF0, 0x36, 0xFE, 0x07, 0x96, 0x08, 0x96, 0x05, 0x96, 0x06, 0x96, 0x09, 0x96,
+ 0xCD, 0xE9, 0x01, 0x4A, 0xCD, 0xE9, 0x03, 0x94, 0x4E, 0x46, 0x02, 0x22, 0x07, 0xF1, 0x2C, 0x00,
+ 0xCD, 0xF8, 0x00, 0x90, 0xD4, 0x4B, 0x11, 0x46, 0x81, 0x46, 0x08, 0xF0, 0x59, 0xFE, 0x07, 0xA8,
+ 0xCD, 0xE9, 0x01, 0x60, 0xCD, 0xE9, 0x03, 0x46, 0xCF, 0x48, 0x4B, 0x46, 0x02, 0x22, 0x01, 0x21,
+ 0x08, 0x38, 0x00, 0x96, 0x08, 0xF0, 0x4C, 0xFE, 0xCD, 0xE9, 0x01, 0x4B, 0x03, 0x94, 0xCA, 0x4B,
+ 0x02, 0x22, 0x01, 0x21, 0x07, 0xA8, 0x00, 0x96, 0x04, 0x94, 0x08, 0xF0, 0x41, 0xFE, 0x28, 0x78,
+ 0x01, 0x28, 0x04, 0xD1, 0x00, 0x21, 0x38, 0x46, 0xFF, 0xF7, 0x5E, 0xFD, 0x68, 0x60, 0x09, 0x98,
+ 0x69, 0x68, 0x01, 0x23, 0x08, 0x44, 0x05, 0xA9, 0x09, 0x90, 0x02, 0x22, 0xCD, 0xF8, 0x00, 0x80,
+ 0x08, 0x46, 0x08, 0xF0, 0x04, 0xFE, 0x09, 0x98, 0x00, 0x90, 0x01, 0x23, 0x02, 0x22, 0x05, 0xA8,
+ 0x0C, 0x99, 0x08, 0xF0, 0xEE, 0xFD, 0xE1, 0xE6, 0x2D, 0xE9, 0xF7, 0x4F, 0x8E, 0xB0, 0x00, 0x25,
+ 0x05, 0x95, 0x06, 0x95, 0x07, 0x95, 0x02, 0x24, 0x08, 0x95, 0x09, 0x95, 0x0A, 0x95, 0x0B, 0x95,
+ 0x05, 0xAF, 0x0C, 0x95, 0xCD, 0xE9, 0x01, 0x47, 0x8C, 0x46, 0x4F, 0xF0, 0x01, 0x0A, 0xAE, 0x4B,
+ 0x06, 0x46, 0x03, 0x94, 0x08, 0x3B, 0x52, 0x46, 0x21, 0x46, 0x60, 0x46, 0xCD, 0xF8, 0x00, 0xA0,
+ 0x04, 0x94, 0x08, 0xF0, 0x05, 0xFE, 0xA8, 0x48, 0x05, 0xAA, 0x02, 0x23, 0x11, 0x46, 0x28, 0x30,
+ 0x00, 0x94, 0x08, 0xF0, 0xEF, 0xFD, 0x0D, 0xF1, 0x24, 0x08, 0xCD, 0xE9, 0x01, 0x48, 0x06, 0xF1,
+ 0x1C, 0x03, 0x02, 0x22, 0x03, 0x94, 0x04, 0x94, 0x9B, 0x46, 0x11, 0x46, 0x05, 0xA8, 0x00, 0x94,
+ 0x08, 0xF0, 0xEE, 0xFD, 0x4F, 0xF0, 0x64, 0x09, 0x02, 0x23, 0x1A, 0x46, 0x59, 0x46, 0x09, 0xA8,
+ 0xCD, 0xF8, 0x00, 0x90, 0x08, 0xF0, 0xAD, 0xFD, 0x05, 0x95, 0x06, 0x95, 0x07, 0x95, 0x08, 0x95,
+ 0x09, 0x95, 0x0A, 0x95, 0x0B, 0x95, 0x0C, 0x95, 0xCD, 0xE9, 0x01, 0x47, 0x92, 0x4B, 0x03, 0x94,
+ 0x08, 0x3B, 0x01, 0x22, 0x02, 0x21, 0xCD, 0xF8, 0x00, 0xA0, 0x04, 0x94, 0x10, 0x98, 0x08, 0xF0,
+ 0xCF, 0xFD, 0x8D, 0x48, 0x05, 0xAA, 0x02, 0x23, 0x11, 0x46, 0x28, 0x30, 0x00, 0x94, 0x08, 0xF0,
+ 0xB9, 0xFD, 0xCD, 0xE9, 0x01, 0x48, 0x03, 0x94, 0x04, 0x94, 0x06, 0xF1, 0x2C, 0x03, 0x00, 0x94,
+ 0x02, 0x22, 0x1C, 0x46, 0x11, 0x46, 0x05, 0xA8, 0x08, 0xF0, 0xBA, 0xFD, 0x02, 0x23, 0x1A, 0x46,
+ 0x21, 0x46, 0x09, 0xA8, 0xCD, 0xF8, 0x00, 0x90, 0x08, 0xF0, 0x7B, 0xFD, 0x11, 0xB0, 0xD2, 0xE5,
+ 0x38, 0xB5, 0x7E, 0x48, 0x7C, 0x4C, 0x01, 0x68, 0x18, 0x3C, 0x91, 0xF8, 0x80, 0x03, 0xC0, 0xF3,
+ 0x00, 0x12, 0x22, 0x70, 0xB1, 0xF8, 0xB1, 0x03, 0x1A, 0xB1, 0x64, 0x21, 0x48, 0x43, 0x0D, 0x46,
+ 0x01, 0xE0, 0xD1, 0xF8, 0xAD, 0x53, 0x74, 0x49, 0x80, 0x08, 0x02, 0x23, 0x48, 0x31, 0x00, 0x90,
+ 0x1A, 0x46, 0xA1, 0xF1, 0x10, 0x00, 0x08, 0xF0, 0x6A, 0xFD, 0x65, 0x60, 0x38, 0xBD, 0xF8, 0xB5,
+ 0x04, 0x46, 0x08, 0x68, 0x20, 0x60, 0x48, 0x68, 0x60, 0x60, 0x88, 0x68, 0xA0, 0x60, 0x00, 0x25,
+ 0x25, 0x64, 0xE5, 0x63, 0xB1, 0xF9, 0x02, 0x00, 0xC4, 0xE9, 0x03, 0x05, 0xB1, 0xF9, 0x00, 0x00,
+ 0xC4, 0xE9, 0x05, 0x05, 0x64, 0x48, 0x18, 0x38, 0x01, 0x78, 0x01, 0x29, 0x09, 0xD0, 0x60, 0x30,
+ 0x1C, 0x34, 0x0F, 0xC8, 0x0F, 0xC4, 0x84, 0xE8, 0x0F, 0x00, 0x2C, 0x3C, 0x84, 0xF8, 0x81, 0x50,
+ 0xF8, 0xBD, 0x50, 0x30, 0x1C, 0x34, 0x0F, 0xC8, 0x0F, 0xC4, 0x5C, 0x4E, 0x0F, 0xC4, 0x30, 0x68,
+ 0x64, 0x21, 0x02, 0x23, 0xB0, 0xF8, 0xB1, 0x03, 0x1A, 0x46, 0x48, 0x43, 0x80, 0x08, 0xA4, 0xF1,
+ 0x20, 0x01, 0x00, 0x90, 0x08, 0x46, 0x3C, 0x3C, 0x08, 0xF0, 0x31, 0xFD, 0x30, 0x68, 0x64, 0x21,
+ 0x02, 0x23, 0xB0, 0xF8, 0xB1, 0x03, 0x1A, 0x46, 0x48, 0x43, 0x80, 0x08, 0x04, 0xF1, 0x2C, 0x01,
+ 0x00, 0x90, 0x08, 0x46, 0x08, 0xF0, 0x23, 0xFD, 0x94, 0xE8, 0x07, 0x00, 0x04, 0xF1, 0x44, 0x03,
+ 0x83, 0xE8, 0x07, 0x00, 0x84, 0xF8, 0x80, 0x50, 0xD0, 0xE7, 0x10, 0xB5, 0x04, 0x46, 0x03, 0x22,
+ 0x08, 0xF0, 0xB0, 0xFD, 0x20, 0x46, 0xFF, 0xF7, 0x94, 0xFC, 0xB4, 0xF9, 0x02, 0x00, 0xE0, 0x60,
+ 0xB4, 0xF9, 0x06, 0x10, 0x40, 0x1A, 0x20, 0x61, 0xB4, 0xF9, 0x00, 0x00, 0x60, 0x61, 0xB4, 0xF9,
+ 0x04, 0x10, 0x40, 0x1A, 0xA0, 0x61, 0x10, 0xBD, 0x7F, 0xB5, 0x0D, 0x46, 0x04, 0x46, 0x03, 0x22,
+ 0x08, 0xF0, 0x98, 0xFD, 0x20, 0x46, 0xFF, 0xF7, 0x7C, 0xFC, 0x20, 0x46, 0xFF, 0xF7, 0xD9, 0xFC,
+ 0x20, 0x46, 0xFF, 0xF7, 0x4C, 0xFD, 0x00, 0x20, 0x00, 0x90, 0x01, 0x90, 0x02, 0x90, 0x03, 0x90,
+ 0x02, 0xAA, 0x69, 0x46, 0x20, 0x46, 0xFF, 0xF7, 0x4E, 0xFE, 0x02, 0xAA, 0x69, 0x46, 0x20, 0x46,
+ 0xFF, 0xF7, 0xCE, 0xFD, 0x02, 0xAA, 0x69, 0x46, 0x20, 0x46, 0xFF, 0xF7, 0xE5, 0xFE, 0xA0, 0x89,
+ 0x68, 0x80, 0xA0, 0x8A, 0x28, 0x80, 0x28, 0x48, 0x18, 0x38, 0x00, 0x78, 0x01, 0x28, 0x05, 0xD1,
+ 0x03, 0x22, 0x29, 0x46, 0x04, 0xF1, 0x44, 0x00, 0x08, 0xF0, 0x6C, 0xFD, 0x7F, 0xBD, 0x2D, 0xE9,
+ 0xF0, 0x47, 0x06, 0x46, 0x48, 0x88, 0xB6, 0xF8, 0x4A, 0x30, 0x89, 0x46, 0xC0, 0x1A, 0xB6, 0xF8,
+ 0x46, 0x40, 0x92, 0x46, 0x01, 0xB2, 0xB9, 0xF8, 0x00, 0x00, 0xB6, 0xF8, 0x48, 0x20, 0xE3, 0x1A,
+ 0x1C, 0xB2, 0x80, 0x1A, 0xB6, 0xF8, 0x44, 0x30, 0x00, 0xB2, 0x9A, 0x1A, 0x15, 0xB2, 0x0C, 0xB9,
+ 0x00, 0x2D, 0x27, 0xD0, 0x49, 0x43, 0x00, 0xFB, 0x00, 0x10, 0x64, 0x21, 0x48, 0x43, 0x04, 0xFB,
+ 0x04, 0xF1, 0x05, 0xFB, 0x05, 0x11, 0xB0, 0xFB, 0xF1, 0xF0, 0x08, 0xF0, 0x83, 0xFC, 0x44, 0x43,
+ 0x0A, 0x22, 0x45, 0x43, 0x21, 0xB2, 0x28, 0xB2, 0x91, 0xFB, 0xF2, 0xF1, 0x90, 0xFB, 0xF2, 0xF0,
+ 0xB6, 0xF8, 0x46, 0x20, 0x61, 0xF3, 0x1F, 0x48, 0x11, 0x44, 0x61, 0xF3, 0x1F, 0x47, 0xB6, 0xF8,
+ 0x44, 0x10, 0x60, 0xF3, 0x0F, 0x08, 0x08, 0x44, 0x60, 0xF3, 0x0F, 0x07, 0xC9, 0xF8, 0x00, 0x70,
+ 0xCA, 0xF8, 0x00, 0x80, 0xBD, 0xE8, 0xF0, 0x87, 0x88, 0x05, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0xF0, 0xB5, 0x4F, 0xF0, 0x00, 0x0C, 0x63, 0x46, 0x65, 0x46, 0x14, 0x46, 0x07, 0xE0, 0x00, 0xBF,
+ 0x01, 0xEB, 0xC4, 0x06, 0x76, 0x68, 0x35, 0x44, 0x51, 0xF8, 0x34, 0x60, 0x33, 0x44, 0x64, 0x1E,
+ 0xF6, 0xD2, 0x95, 0xFB, 0xF2, 0xF6, 0x93, 0xFB, 0xF2, 0xF7, 0xC0, 0xF8, 0x04, 0xC0, 0x15, 0x46,
+ 0x42, 0xF2, 0x10, 0x74, 0xC0, 0xF8, 0x00, 0xC0, 0x17, 0xE0, 0x00, 0xBF, 0x01, 0xEB, 0xC5, 0x03,
+ 0x5B, 0x68, 0x9B, 0x1B, 0x5B, 0x43, 0xA3, 0x42, 0x00, 0xDD, 0x23, 0x46, 0xD0, 0xF8, 0x04, 0xC0,
+ 0x63, 0x44, 0x43, 0x60, 0x51, 0xF8, 0x35, 0x30, 0xDB, 0x1B, 0x5B, 0x43, 0xA3, 0x42, 0x00, 0xDD,
+ 0x23, 0x46, 0xD0, 0xF8, 0x00, 0xC0, 0x63, 0x44, 0x03, 0x60, 0x6D, 0x1E, 0xE6, 0xD2, 0x41, 0x68,
+ 0x64, 0x23, 0x59, 0x43, 0x91, 0xFB, 0xF2, 0xF1, 0x41, 0x60, 0x01, 0x68, 0x59, 0x43, 0x91, 0xFB,
+ 0xF2, 0xF1, 0x01, 0x60, 0xF0, 0xBD, 0x00, 0x00, 0x10, 0xB5, 0x14, 0x21, 0x53, 0x48, 0x0D, 0xF0,
+ 0x82, 0xFF, 0x0D, 0xF0, 0x24, 0xFD, 0x64, 0x21, 0x48, 0x43, 0x50, 0x49, 0x08, 0x60, 0x10, 0xBD,
+ 0x2D, 0xE9, 0xF0, 0x41, 0xDF, 0xF8, 0x38, 0x81, 0x04, 0x46, 0x0D, 0x46, 0xD8, 0xF8, 0x00, 0x00,
+ 0x90, 0xF8, 0xA0, 0x03, 0x00, 0xF0, 0x0F, 0x06, 0xC6, 0xF1, 0x10, 0x07, 0x20, 0x46, 0x00, 0xF0,
+ 0x5C, 0xF8, 0xF0, 0xB1, 0xE0, 0x89, 0x61, 0x8A, 0x22, 0x8A, 0x40, 0x1A, 0xA1, 0x89, 0x00, 0xB2,
+ 0x89, 0x1A, 0xB4, 0xF9, 0x2A, 0x20, 0x78, 0x43, 0x02, 0xFB, 0x06, 0x00, 0x00, 0x11, 0x60, 0x85,
+ 0x09, 0xB2, 0xB4, 0xF9, 0x28, 0x00, 0x79, 0x43, 0x00, 0xFB, 0x06, 0x10, 0x00, 0x11, 0x20, 0x85,
+ 0x0D, 0xF0, 0xDF, 0xFB, 0x39, 0x4A, 0x91, 0x89, 0x08, 0x1A, 0x11, 0x7C, 0x01, 0x29, 0x0E, 0xD0,
+ 0x0F, 0xE0, 0x20, 0x6B, 0x02, 0x28, 0x07, 0xD1, 0xE0, 0x89, 0x61, 0x8A, 0x40, 0x1A, 0x60, 0x85,
+ 0xA0, 0x89, 0x21, 0x8A, 0x40, 0x1A, 0x20, 0x85, 0x00, 0x20, 0xBD, 0xE8, 0xF0, 0x81, 0x51, 0x68,
+ 0x08, 0x44, 0x00, 0x28, 0x00, 0xDA, 0x00, 0x20, 0x00, 0xEB, 0x80, 0x01, 0x16, 0x68, 0x49, 0x00,
+ 0xB1, 0xFB, 0xF6, 0xF7, 0xD8, 0xF8, 0x00, 0x10, 0xB1, 0xF8, 0xA1, 0x33, 0x9F, 0x42, 0x03, 0xD9,
+ 0x73, 0x43, 0x0A, 0x20, 0xB3, 0xFB, 0xF0, 0xF0, 0xB4, 0xF9, 0x2A, 0x60, 0x91, 0xF8, 0xA3, 0x33,
+ 0x92, 0x68, 0x5E, 0x43, 0x64, 0x21, 0x46, 0x43, 0x4A, 0x43, 0xB4, 0xF9, 0x28, 0x40, 0x96, 0xFB,
+ 0xF2, 0xF1, 0x5C, 0x43, 0x44, 0x43, 0x94, 0xFB, 0xF2, 0xF0, 0x6A, 0x88, 0x11, 0x44, 0x69, 0x80,
+ 0x29, 0x88, 0x08, 0x44, 0x28, 0x80, 0x01, 0x20, 0xCF, 0xE7, 0x70, 0xB5, 0x04, 0x46, 0x18, 0x48,
+ 0x01, 0x25, 0x00, 0x68, 0x90, 0xF8, 0x80, 0x03, 0x80, 0x07, 0x12, 0xD5, 0xB4, 0xF8, 0x42, 0x00,
+ 0x01, 0x07, 0x0E, 0xD4, 0x00, 0x06, 0x0C, 0xD4, 0x0D, 0xF0, 0xAE, 0xFF, 0x48, 0xB9, 0x11, 0x48,
+ 0x00, 0x78, 0x30, 0xB9, 0x10, 0x48, 0x00, 0x78, 0x18, 0xB9, 0x10, 0x48, 0x00, 0x78, 0x01, 0x28,
+ 0x02, 0xD9, 0x00, 0x20, 0x20, 0x63, 0x0E, 0xE0, 0x20, 0x6B, 0x02, 0x28, 0x0C, 0xD8, 0x40, 0x1C,
+ 0x20, 0x63, 0x02, 0x28, 0x07, 0xD9, 0x94, 0xF8, 0x3E, 0x01, 0x04, 0x28, 0x04, 0xD0, 0x06, 0x28,
+ 0x02, 0xD0, 0x07, 0x28, 0x00, 0xD0, 0x00, 0x25, 0x28, 0x46, 0x70, 0xBD, 0xC8, 0x24, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0x1C, 0x05, 0x10, 0x00, 0x28, 0x05, 0x10, 0x00, 0xF3, 0x06, 0x10, 0x00,
+ 0xB0, 0xF8, 0x44, 0x20, 0xB1, 0xF8, 0x44, 0x30, 0x1A, 0x44, 0xA0, 0xF8, 0x44, 0x20, 0xB0, 0xF8,
+ 0x46, 0x20, 0xB1, 0xF8, 0x46, 0x30, 0x1A, 0x44, 0xA0, 0xF8, 0x46, 0x20, 0x42, 0x6B, 0x4B, 0x6B,
+ 0x1A, 0x44, 0x42, 0x63, 0x82, 0x6B, 0x8B, 0x6B, 0x1A, 0x44, 0x82, 0x63, 0xB1, 0xF9, 0x54, 0x20,
+ 0xB0, 0xF9, 0x54, 0x30, 0x9A, 0x42, 0x01, 0xDD, 0xA0, 0xF8, 0x54, 0x20, 0xB1, 0xF9, 0x56, 0x20,
+ 0xB0, 0xF9, 0x56, 0x30, 0x9A, 0x42, 0x01, 0xDD, 0xA0, 0xF8, 0x56, 0x20, 0x90, 0xF8, 0x44, 0x21,
+ 0x91, 0xF8, 0x44, 0x31, 0x9A, 0x42, 0x0A, 0xD0, 0x30, 0xF8, 0x48, 0x2F, 0xB1, 0xF8, 0x48, 0x30,
+ 0x1A, 0x44, 0x20, 0xF8, 0x0C, 0x29, 0xC9, 0x6B, 0x02, 0x68, 0x11, 0x44, 0x01, 0x60, 0x70, 0x47,
+ 0x2D, 0xE9, 0xF0, 0x4F, 0x97, 0xB0, 0x80, 0x46, 0x00, 0xF5, 0x80, 0x50, 0x00, 0x26, 0x11, 0x90,
+ 0xBF, 0xE0, 0x00, 0x20, 0x08, 0x90, 0x05, 0x90, 0x10, 0x90, 0x12, 0x90, 0x81, 0x46, 0x82, 0x46,
+ 0x83, 0x46, 0x05, 0x46, 0x07, 0x46, 0x0A, 0x90, 0x0B, 0x90, 0x0C, 0x90, 0x0D, 0x90, 0x08, 0xEB,
+ 0x86, 0x00, 0x00, 0xF5, 0x80, 0x50, 0x14, 0x90, 0xD0, 0xF8, 0x2C, 0x04, 0x0A, 0xAA, 0x04, 0x46,
+ 0x17, 0xE0, 0x00, 0xBF, 0x90, 0xF8, 0x44, 0x11, 0x08, 0xEB, 0x01, 0x0C, 0x53, 0x5C, 0x0C, 0xF5,
+ 0x80, 0x5C, 0x5B, 0x1C, 0xDB, 0xB2, 0x53, 0x54, 0x9C, 0xF8, 0x1A, 0xC4, 0x63, 0x45, 0x06, 0xD1,
+ 0x01, 0x2B, 0x04, 0xD9, 0x01, 0x23, 0x8B, 0x40, 0x12, 0x99, 0x0B, 0x43, 0x12, 0x93, 0xD0, 0xF8,
+ 0x4C, 0x01, 0x00, 0x28, 0xE6, 0xD1, 0x40, 0xE0, 0x94, 0xF8, 0x44, 0x11, 0x01, 0x20, 0x00, 0xFA,
+ 0x01, 0xFC, 0x12, 0x99, 0x1C, 0xEA, 0x01, 0x0F, 0x12, 0xD1, 0xA0, 0x6B, 0xB4, 0xF9, 0x0E, 0x30,
+ 0x05, 0x9A, 0x49, 0x46, 0xC0, 0xFB, 0x03, 0x12, 0x05, 0x92, 0x89, 0x46, 0xB4, 0xF9, 0x0C, 0x30,
+ 0x51, 0x46, 0x5A, 0x46, 0xC0, 0xFB, 0x03, 0x12, 0x8A, 0x46, 0x93, 0x46, 0x05, 0x44, 0x19, 0xE0,
+ 0x08, 0x99, 0x1C, 0xEA, 0x01, 0x0F, 0x15, 0xD1, 0xE0, 0x6B, 0xB4, 0xF9, 0x1E, 0x30, 0x05, 0x9A,
+ 0x49, 0x46, 0xC0, 0xFB, 0x03, 0x12, 0x05, 0x92, 0x89, 0x46, 0xB4, 0xF9, 0x1C, 0x30, 0x51, 0x46,
+ 0x5A, 0x46, 0xC0, 0xFB, 0x03, 0x12, 0x8A, 0x46, 0x08, 0x99, 0x05, 0x44, 0x4C, 0xEA, 0x01, 0x00,
+ 0x93, 0x46, 0x08, 0x90, 0x20, 0x46, 0x00, 0xF0, 0xFA, 0xFC, 0x01, 0x28, 0x02, 0xD1, 0x10, 0x98,
+ 0x40, 0x1C, 0x10, 0x90, 0xD4, 0xF8, 0x4C, 0x41, 0x7F, 0x1C, 0x00, 0x2C, 0xBC, 0xD1, 0x00, 0x2F,
+ 0x45, 0xDD, 0xE8, 0x17, 0x04, 0x46, 0x03, 0x46, 0x2A, 0x46, 0x48, 0x46, 0x05, 0x99, 0x0D, 0xF0,
+ 0x4D, 0xFD, 0xAD, 0xF8, 0x1E, 0x00, 0x2A, 0x46, 0x23, 0x46, 0x50, 0x46, 0x59, 0x46, 0x0D, 0xF0,
+ 0x45, 0xFD, 0xAD, 0xF8, 0x1C, 0x00, 0x14, 0x98, 0x4F, 0xF0, 0xFF, 0x37, 0x00, 0x25, 0xD0, 0xF8,
+ 0x2C, 0x44, 0x23, 0xE0, 0x20, 0x46, 0x00, 0xF0, 0xD2, 0xFC, 0x01, 0x28, 0x01, 0xD0, 0x10, 0x98,
+ 0x90, 0xB9, 0x04, 0xF1, 0x10, 0x01, 0x07, 0xA8, 0x08, 0xF0, 0x7B, 0xFB, 0xB8, 0x42, 0x0B, 0xD2,
+ 0x07, 0x46, 0x3D, 0xB1, 0x29, 0x46, 0x20, 0x46, 0xFF, 0xF7, 0x1A, 0xFF, 0x29, 0x46, 0x40, 0x46,
+ 0x00, 0xF0, 0xD4, 0xFC, 0x25, 0x46, 0x07, 0xE0, 0x21, 0x46, 0x28, 0x46, 0xFF, 0xF7, 0x10, 0xFF,
+ 0x21, 0x46, 0x40, 0x46, 0x00, 0xF0, 0xCA, 0xFC, 0xD4, 0xF8, 0x4C, 0x41, 0x00, 0x2C, 0xD9, 0xD1,
+ 0x07, 0x98, 0x45, 0xF8, 0x0C, 0x0F, 0xE8, 0x8E, 0x40, 0xF0, 0x40, 0x00, 0xE8, 0x86, 0x11, 0x98,
+ 0x76, 0x1C, 0x90, 0xF8, 0xA4, 0x04, 0xB0, 0x42, 0x3F, 0xF7, 0x3B, 0xAF, 0x17, 0xB0, 0xBD, 0xE8,
+ 0xF0, 0x8F, 0x2D, 0xE9, 0xF7, 0x4F, 0x05, 0x46, 0x0E, 0x46, 0xC1, 0xEB, 0xC6, 0x00, 0x00, 0xEB,
+ 0x40, 0x00, 0x82, 0xB0, 0x05, 0xEB, 0x00, 0x1A, 0x00, 0x24, 0x4F, 0xF0, 0x01, 0x08, 0xDF, 0xF8,
+ 0x98, 0xB5, 0x05, 0xF5, 0x80, 0x57, 0x7F, 0xE0, 0xD7, 0xF8, 0xB0, 0x23, 0x08, 0xFA, 0x04, 0xF9,
+ 0x12, 0xEA, 0x09, 0x0F, 0x77, 0xD0, 0xB4, 0x42, 0x75, 0xD0, 0xDB, 0xF8, 0x0C, 0x10, 0x31, 0xF8,
+ 0x16, 0x10, 0x11, 0xEA, 0x09, 0x0F, 0x6E, 0xD1, 0xC4, 0xEB, 0xC4, 0x00, 0x00, 0xEB, 0x40, 0x00,
+ 0x05, 0xEB, 0x00, 0x10, 0x00, 0x90, 0x00, 0xF0, 0x6C, 0xFC, 0x01, 0x28, 0x04, 0xD0, 0x00, 0x98,
+ 0x00, 0xF0, 0x6D, 0xFC, 0x01, 0x28, 0x5E, 0xD1, 0x00, 0x98, 0x00, 0xF0, 0x5A, 0xFC, 0xE8, 0xBB,
+ 0x00, 0x99, 0x0A, 0xF1, 0x0C, 0x00, 0x0C, 0x31, 0x08, 0xF0, 0x13, 0xFB, 0x01, 0x90, 0xDB, 0xF8,
+ 0x0C, 0x00, 0x08, 0xFA, 0x06, 0xF2, 0x30, 0xF8, 0x16, 0x30, 0x43, 0xEA, 0x09, 0x03, 0x20, 0xF8,
+ 0x16, 0x30, 0x30, 0xF8, 0x14, 0x30, 0x13, 0x43, 0x20, 0xF8, 0x14, 0x30, 0x50, 0x46, 0xDB, 0xF8,
+ 0x04, 0x90, 0x00, 0xF0, 0x58, 0xFC, 0x01, 0x28, 0x04, 0xD1, 0x00, 0x98, 0x00, 0xF0, 0x41, 0xFC,
+ 0x01, 0x28, 0x1D, 0xD0, 0x50, 0x46, 0x00, 0xF0, 0x3C, 0xFC, 0x01, 0x28, 0x04, 0xD1, 0x00, 0x98,
+ 0x00, 0xF0, 0x49, 0xFC, 0x01, 0x28, 0x13, 0xD0, 0x50, 0x46, 0x00, 0xF0, 0x32, 0xFC, 0x01, 0x28,
+ 0x1E, 0xD1, 0x00, 0x98, 0x00, 0xF0, 0x2D, 0xFC, 0x01, 0x28, 0x19, 0xD1, 0x00, 0x99, 0x9A, 0xF8,
+ 0x42, 0x00, 0x91, 0xF8, 0x42, 0x10, 0x08, 0x43, 0x80, 0x06, 0x00, 0xE0, 0x1B, 0xE0, 0x0F, 0xD5,
+ 0xBA, 0xF8, 0x42, 0x00, 0xDB, 0xF8, 0x08, 0x90, 0x40, 0xF0, 0x20, 0x01, 0xAA, 0xF8, 0x42, 0x10,
+ 0x00, 0x98, 0xB0, 0xF8, 0x42, 0x00, 0x40, 0xF0, 0x20, 0x01, 0x00, 0x98, 0xA0, 0xF8, 0x42, 0x10,
+ 0x01, 0x98, 0x48, 0x45, 0x07, 0xD2, 0x41, 0xF2, 0x2C, 0x40, 0x51, 0x46, 0x28, 0x44, 0x04, 0x9B,
+ 0x00, 0x9A, 0x00, 0xF0, 0x32, 0xFC, 0x64, 0x1C, 0x97, 0xF9, 0xB8, 0x03, 0xA0, 0x42, 0xBF, 0xF6,
+ 0x7B, 0xAF, 0x05, 0xB0, 0x63, 0xE7, 0x2D, 0xE9, 0xF8, 0x4F, 0x88, 0x46, 0x06, 0x46, 0xC1, 0xEB,
+ 0xC8, 0x00, 0x00, 0xEB, 0x40, 0x00, 0x06, 0xEB, 0x00, 0x14, 0xDF, 0xF8, 0x6C, 0xA4, 0x94, 0xF8,
+ 0x42, 0x00, 0x4F, 0xF0, 0x04, 0x0B, 0x40, 0x06, 0x4F, 0xF0, 0x01, 0x09, 0x06, 0xF5, 0x80, 0x55,
+ 0x2E, 0xD5, 0x00, 0x27, 0x04, 0xF1, 0x0C, 0x0B, 0x24, 0xE0, 0x00, 0xBF, 0xD5, 0xF8, 0xB0, 0x13,
+ 0x09, 0xFA, 0x07, 0xF0, 0x01, 0x42, 0x1C, 0xD0, 0x47, 0x45, 0x1A, 0xD0, 0xC7, 0xEB, 0xC7, 0x00,
+ 0x00, 0xEB, 0x40, 0x00, 0x06, 0xEB, 0x00, 0x10, 0x00, 0x90, 0x00, 0xF0, 0xD2, 0xFB, 0x01, 0x28,
+ 0x0F, 0xD1, 0x00, 0x98, 0x00, 0xF0, 0xC5, 0xFB, 0x58, 0xB9, 0x00, 0x99, 0x58, 0x46, 0x10, 0x31,
+ 0x08, 0xF0, 0x7F, 0xFA, 0xDA, 0xF8, 0x08, 0x10, 0x88, 0x42, 0x02, 0xD2, 0x04, 0x20, 0x84, 0xF8,
+ 0x3E, 0x01, 0x7F, 0x1C, 0x95, 0xF9, 0xB8, 0x03, 0xB8, 0x42, 0xD7, 0xDA, 0xBD, 0xE8, 0xF8, 0x8F,
+ 0xFE, 0x49, 0xE0, 0x6A, 0x09, 0x68, 0x91, 0xF8, 0x95, 0x13, 0x88, 0x42, 0xF6, 0xD9, 0x00, 0x27,
+ 0x31, 0xE0, 0x00, 0xBF, 0xD5, 0xF8, 0xB0, 0x13, 0x09, 0xFA, 0x07, 0xF0, 0x01, 0x42, 0x29, 0xD0,
+ 0x47, 0x45, 0x27, 0xD0, 0xC7, 0xEB, 0xC7, 0x00, 0x00, 0xEB, 0x40, 0x00, 0x06, 0xEB, 0x00, 0x10,
+ 0x00, 0x90, 0x00, 0xF0, 0x9E, 0xFB, 0x01, 0x28, 0x1C, 0xD1, 0x00, 0x98, 0x94, 0xF8, 0x44, 0x11,
+ 0x90, 0xF8, 0x45, 0x01, 0x81, 0x42, 0x15, 0xD1, 0x00, 0x98, 0x00, 0xF0, 0x8A, 0xFB, 0x88, 0xB9,
+ 0x00, 0x99, 0x04, 0xF1, 0x0C, 0x00, 0x10, 0x31, 0x08, 0xF0, 0x43, 0xFA, 0xDA, 0xF8, 0x08, 0x10,
+ 0x88, 0x42, 0x07, 0xD2, 0x84, 0xF8, 0x3E, 0xB1, 0xB4, 0xF8, 0x42, 0x00, 0x40, 0xF0, 0x40, 0x00,
+ 0xA4, 0xF8, 0x42, 0x00, 0x7F, 0x1C, 0x95, 0xF9, 0xB8, 0x03, 0xB8, 0x42, 0xCA, 0xDA, 0x00, 0x27,
+ 0x28, 0xE0, 0x00, 0xBF, 0xD5, 0xF8, 0xB0, 0x13, 0x09, 0xFA, 0x07, 0xF0, 0x01, 0x42, 0x20, 0xD0,
+ 0x47, 0x45, 0x1E, 0xD0, 0xC7, 0xEB, 0xC7, 0x00, 0x00, 0xEB, 0x40, 0x00, 0x06, 0xEB, 0x00, 0x10,
+ 0x83, 0x46, 0x00, 0xF0, 0x6C, 0xFB, 0x01, 0x28, 0x13, 0xD1, 0x58, 0x46, 0x00, 0xF0, 0x59, 0xFB,
+ 0x78, 0xB9, 0x0B, 0xF1, 0x10, 0x01, 0x04, 0xF1, 0x0C, 0x00, 0x08, 0xF0, 0x12, 0xFA, 0xDA, 0xF8,
+ 0x08, 0x10, 0x88, 0x42, 0x05, 0xD2, 0xBB, 0xF8, 0x42, 0x00, 0x40, 0xF0, 0x40, 0x01, 0xAB, 0xF8,
+ 0x42, 0x10, 0x7F, 0x1C, 0x95, 0xF9, 0xB8, 0x03, 0xB8, 0x42, 0xD3, 0xDA, 0x8E, 0xE7, 0x2D, 0xE9,
+ 0xF8, 0x43, 0x06, 0x46, 0xFF, 0x20, 0x00, 0x25, 0x8D, 0xF8, 0x00, 0x00, 0x01, 0x27, 0x06, 0xF5,
+ 0x80, 0x54, 0x19, 0xE0, 0xC5, 0xEB, 0xC5, 0x00, 0x00, 0xEB, 0x40, 0x00, 0x06, 0xEB, 0x00, 0x18,
+ 0xD4, 0xF8, 0xB0, 0x03, 0x07, 0xFA, 0x05, 0xF1, 0x08, 0x42, 0x0C, 0xD0, 0x40, 0x46, 0x00, 0xF0,
+ 0x28, 0xFB, 0x01, 0x28, 0x07, 0xD0, 0x98, 0xF8, 0x3E, 0x01, 0x05, 0x28, 0x03, 0xD1, 0x29, 0x46,
+ 0x30, 0x46, 0xFF, 0xF7, 0x28, 0xFF, 0x6D, 0x1C, 0x94, 0xF9, 0xB8, 0x03, 0xA8, 0x42, 0xE1, 0xDA,
+ 0x00, 0x25, 0x1B, 0xE0, 0xC5, 0xEB, 0xC5, 0x00, 0x00, 0xEB, 0x40, 0x00, 0x06, 0xEB, 0x00, 0x18,
+ 0xD4, 0xF8, 0xB0, 0x03, 0x07, 0xFA, 0x05, 0xF1, 0x08, 0x42, 0x0E, 0xD0, 0x40, 0x46, 0x00, 0xF0,
+ 0x08, 0xFB, 0x01, 0x28, 0x09, 0xD0, 0x40, 0x46, 0x00, 0xF0, 0x11, 0xFB, 0x01, 0x28, 0x04, 0xD1,
+ 0x6A, 0x46, 0x29, 0x46, 0x30, 0x46, 0xFF, 0xF7, 0x6C, 0xFE, 0x6D, 0x1C, 0x94, 0xF9, 0xB8, 0x03,
+ 0xA8, 0x42, 0xDF, 0xDA, 0x00, 0x25, 0xB8, 0x46, 0x21, 0xE0, 0x00, 0xBF, 0xC5, 0xEB, 0xC5, 0x00,
+ 0x00, 0xEB, 0x40, 0x00, 0x06, 0xEB, 0x00, 0x17, 0xD4, 0xF8, 0xB0, 0x13, 0x08, 0xFA, 0x05, 0xF0,
+ 0x01, 0x42, 0x13, 0xD0, 0x38, 0x46, 0x00, 0xF0, 0xE4, 0xFA, 0x01, 0x28, 0x0E, 0xD0, 0x38, 0x46,
+ 0x00, 0xF0, 0xE7, 0xFA, 0x01, 0x28, 0x04, 0xD1, 0x6A, 0x46, 0x29, 0x46, 0x30, 0x46, 0xFF, 0xF7,
+ 0x48, 0xFE, 0x37, 0xF8, 0x42, 0x0F, 0x20, 0xF0, 0x20, 0x00, 0x38, 0x80, 0x6D, 0x1C, 0x94, 0xF9,
+ 0xB8, 0x03, 0xA8, 0x42, 0xDA, 0xDA, 0x9D, 0xF8, 0x00, 0x00, 0x40, 0x1C, 0x84, 0xF8, 0xA4, 0x04,
+ 0xBD, 0xE8, 0xF8, 0x83, 0x10, 0xB5, 0x04, 0x46, 0x87, 0x48, 0x03, 0x21, 0x0C, 0x30, 0x05, 0xF0,
+ 0x97, 0xFB, 0x85, 0x48, 0x1E, 0x21, 0xC0, 0x68, 0x0D, 0xF0, 0xD3, 0xFB, 0x41, 0xF2, 0x2C, 0x40,
+ 0x7C, 0x21, 0x20, 0x44, 0x0D, 0xF0, 0xEF, 0xFB, 0x80, 0x48, 0x02, 0x68, 0x80, 0x48, 0xB2, 0xF8,
+ 0x96, 0x13, 0x80, 0x79, 0x49, 0x43, 0x10, 0xB1, 0xB2, 0xF8, 0x6C, 0x13, 0x49, 0x43, 0xB2, 0xF8,
+ 0x98, 0x23, 0x02, 0xFB, 0x02, 0x10, 0x78, 0x4A, 0x12, 0x1D, 0x11, 0x60, 0x11, 0x1D, 0x08, 0x60,
+ 0x20, 0x46, 0xFF, 0xF7, 0x5C, 0xFF, 0x20, 0x46, 0xFF, 0xF7, 0x3A, 0xFD, 0x72, 0x48, 0x03, 0x21,
+ 0xBD, 0xE8, 0x10, 0x40, 0x0C, 0x30, 0x05, 0xF0, 0x83, 0xBB, 0xF0, 0xB4, 0x84, 0x6B, 0xC2, 0x68,
+ 0x8B, 0x6B, 0xCE, 0x68, 0x17, 0x14, 0x67, 0x43, 0x4F, 0xEA, 0x26, 0x4C, 0x03, 0xFB, 0x0C, 0x77,
+ 0xE5, 0x18, 0x97, 0xFB, 0xF5, 0xF7, 0x67, 0xF3, 0x1F, 0x42, 0x17, 0xB2, 0x67, 0x43, 0x34, 0xB2,
+ 0x03, 0xFB, 0x04, 0x73, 0x93, 0xFB, 0xF5, 0xF3, 0x63, 0xF3, 0x0F, 0x02, 0xC2, 0x60, 0xF0, 0xBC,
+ 0xDE, 0xE4, 0x62, 0x4A, 0x12, 0x68, 0x92, 0xF8, 0x40, 0x24, 0x92, 0x06, 0x11, 0xD5, 0x90, 0xF8,
+ 0x47, 0x31, 0x23, 0xB1, 0x91, 0xF8, 0x47, 0x21, 0x0A, 0xB1, 0x93, 0x42, 0x07, 0xD1, 0x90, 0xF8,
+ 0x48, 0x01, 0x30, 0xB1, 0x91, 0xF8, 0x48, 0x11, 0x19, 0xB1, 0x88, 0x42, 0x01, 0xD0, 0x00, 0x20,
+ 0x70, 0x47, 0x01, 0x20, 0x70, 0x47, 0x2D, 0xE9, 0xFC, 0x5F, 0x00, 0xF5, 0x80, 0x54, 0x05, 0x46,
+ 0xD4, 0xF8, 0xBC, 0x13, 0xD4, 0xF8, 0xB0, 0x03, 0x01, 0x40, 0x00, 0x28, 0xC4, 0xF8, 0xBC, 0x13,
+ 0x4F, 0xF0, 0x00, 0x00, 0x0A, 0xD0, 0x81, 0x46, 0x0F, 0x21, 0x4E, 0x48, 0x0D, 0xF0, 0x83, 0xFB,
+ 0x00, 0x26, 0xDF, 0xF8, 0x30, 0xA1, 0x4F, 0xF0, 0x01, 0x08, 0x5B, 0xE0, 0xC4, 0xF8, 0xBC, 0x03,
+ 0xBD, 0xE8, 0xFC, 0x9F, 0xD4, 0xF8, 0xB0, 0x13, 0x08, 0xFA, 0x06, 0xF2, 0x40, 0x46, 0x11, 0x42,
+ 0x4F, 0xD0, 0xC6, 0xEB, 0xC6, 0x01, 0x01, 0xEB, 0x41, 0x01, 0x05, 0xEB, 0x01, 0x11, 0x91, 0xF8,
+ 0x40, 0x30, 0x1B, 0x06, 0xD4, 0xF8, 0xBC, 0x33, 0x11, 0xD5, 0x13, 0x43, 0x05, 0xEB, 0x86, 0x02,
+ 0xC4, 0xF8, 0xBC, 0x33, 0x02, 0xF5, 0x9C, 0x52, 0xD1, 0xF8, 0x5A, 0x30, 0x13, 0x64, 0x91, 0xF8,
+ 0x44, 0x11, 0x88, 0x40, 0x40, 0xEA, 0x09, 0x09, 0x0A, 0xF8, 0x01, 0x60, 0x31, 0xE0, 0x13, 0x42,
+ 0x2F, 0xD0, 0x91, 0xF8, 0x3E, 0x31, 0x06, 0x2B, 0x15, 0xD0, 0x05, 0x2B, 0x13, 0xD0, 0x91, 0xF8,
+ 0x44, 0x21, 0x08, 0xFA, 0x02, 0xF0, 0x40, 0xEA, 0x09, 0x09, 0x0A, 0xF8, 0x02, 0x60, 0x00, 0xBF,
+ 0x05, 0xEB, 0x86, 0x07, 0x5A, 0x31, 0x07, 0xF5, 0x9E, 0x57, 0x8B, 0x46, 0x38, 0x46, 0x03, 0xF0,
+ 0x15, 0xFA, 0x98, 0xB1, 0x15, 0xE0, 0xB1, 0xF8, 0x42, 0x00, 0x20, 0xF0, 0x08, 0x00, 0xA1, 0xF8,
+ 0x42, 0x00, 0xD4, 0xF8, 0xBC, 0x03, 0x90, 0x43, 0xC4, 0xF8, 0xBC, 0x03, 0x22, 0x4A, 0x05, 0xEB,
+ 0x86, 0x00, 0x00, 0xF5, 0x9C, 0x50, 0x12, 0x68, 0x02, 0x64, 0xE1, 0xE7, 0xDB, 0xF8, 0x00, 0x00,
+ 0x38, 0x60, 0x76, 0x1C, 0x94, 0xF9, 0xB8, 0x03, 0xB0, 0x42, 0xA3, 0xDA, 0x00, 0x27, 0x71, 0xE0,
+ 0xD4, 0xF8, 0xB0, 0x23, 0x08, 0xFA, 0x07, 0xF1, 0x40, 0x46, 0x0A, 0x42, 0x69, 0xD0, 0xD4, 0xF8,
+ 0xBC, 0x23, 0x0A, 0x42, 0x65, 0xD1, 0xC7, 0xEB, 0xC7, 0x01, 0x01, 0xEB, 0x41, 0x01, 0x05, 0xEB,
+ 0x01, 0x1B, 0x9B, 0xF8, 0x3E, 0x11, 0x02, 0x29, 0x01, 0xD0, 0x01, 0x29, 0x59, 0xD1, 0x9B, 0xF8,
+ 0x44, 0x11, 0x00, 0xFA, 0x01, 0xF2, 0x12, 0xEA, 0x09, 0x0F, 0x24, 0xD0, 0x1A, 0xF8, 0x01, 0x00,
+ 0xB8, 0x42, 0x4E, 0xD0, 0xC0, 0xEB, 0xC0, 0x00, 0x00, 0xEB, 0x40, 0x00, 0x05, 0xEB, 0x00, 0x16,
+ 0x30, 0x46, 0x59, 0x46, 0x0A, 0xE0, 0x00, 0x00, 0xE0, 0x05, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0xEC, 0x06, 0x10, 0x00, 0x70, 0x25, 0x10, 0x00, 0x52, 0x6C, 0x01, 0x00, 0xFF, 0xF7, 0x31, 0xFF,
+ 0x01, 0x28, 0x36, 0xD1, 0x30, 0x46, 0x59, 0x46, 0xFF, 0xF7, 0x0F, 0xFF, 0x59, 0x46, 0x28, 0x46,
+ 0x00, 0xF0, 0xA0, 0xFF, 0x2D, 0xE0, 0x0B, 0xF1, 0x5A, 0x01, 0x00, 0x26, 0x01, 0x91, 0x24, 0xE0,
+ 0xD4, 0xF8, 0xBC, 0x13, 0x08, 0xFA, 0x06, 0xF0, 0x01, 0x42, 0x1D, 0xD0, 0xC6, 0xEB, 0xC6, 0x00,
+ 0x00, 0xEB, 0x40, 0x00, 0x05, 0xEB, 0x00, 0x10, 0x05, 0xEB, 0x86, 0x01, 0x00, 0x90, 0x01, 0xF5,
+ 0x9E, 0x51, 0x01, 0x98, 0x03, 0xF0, 0x9A, 0xF9, 0x01, 0x28, 0x0D, 0xD1, 0x59, 0x46, 0x00, 0x98,
+ 0xFF, 0xF7, 0x07, 0xFF, 0x01, 0x28, 0x07, 0xD1, 0x59, 0x46, 0x00, 0x98, 0xFF, 0xF7, 0xE5, 0xFE,
+ 0x59, 0x46, 0x28, 0x46, 0x00, 0xF0, 0x76, 0xFF, 0x76, 0x1C, 0x94, 0xF9, 0xB8, 0x03, 0xB0, 0x42,
+ 0xD6, 0xDA, 0x7F, 0x1C, 0x94, 0xF9, 0xB8, 0x03, 0xB8, 0x42, 0x89, 0xDA, 0x28, 0xE7, 0x2D, 0xE9,
+ 0xF0, 0x47, 0x04, 0x46, 0x91, 0x42, 0x56, 0xD0, 0x07, 0xD9, 0x01, 0xEB, 0x02, 0x00, 0xA0, 0xEB,
+ 0x02, 0x01, 0xCA, 0xB2, 0xA0, 0xEB, 0x02, 0x00, 0xC1, 0xB2, 0x01, 0xEB, 0xC1, 0x00, 0x04, 0xEB,
+ 0xC0, 0x01, 0x02, 0xEB, 0xC2, 0x00, 0x04, 0xEB, 0xC0, 0x00, 0xB1, 0xF9, 0x32, 0x50, 0xB0, 0xF9,
+ 0x32, 0x30, 0x81, 0x46, 0x9D, 0x42, 0x00, 0xDA, 0x4B, 0x86, 0xCB, 0x8C, 0xC5, 0x8C, 0x2B, 0x44,
+ 0xCB, 0x84, 0x0B, 0x8D, 0x05, 0x8D, 0x2B, 0x44, 0x0B, 0x85, 0x8B, 0x69, 0x85, 0x69, 0x2B, 0x44,
+ 0x8B, 0x61, 0xC7, 0x69, 0xCE, 0x69, 0xCB, 0x68, 0xD9, 0xF8, 0x0C, 0x00, 0x4F, 0xEA, 0x23, 0x4C,
+ 0x4F, 0xEA, 0x20, 0x48, 0x0C, 0xFB, 0x06, 0xFC, 0xF5, 0x19, 0x07, 0xFB, 0x08, 0xCC, 0x9C, 0xFB,
+ 0xF5, 0xFC, 0x6C, 0xF3, 0x1F, 0x43, 0x0F, 0xFA, 0x83, 0xFC, 0x0C, 0xFB, 0x06, 0xFC, 0x00, 0xB2,
+ 0x07, 0xFB, 0x00, 0xC0, 0x90, 0xFB, 0xF5, 0xF0, 0x60, 0xF3, 0x0F, 0x03, 0xCD, 0x61, 0xCB, 0x60,
+ 0x94, 0xF8, 0x38, 0x04, 0x41, 0x1E, 0x91, 0x42, 0x08, 0xD0, 0x00, 0xEB, 0xC0, 0x00, 0x04, 0xEB,
+ 0xC0, 0x01, 0x48, 0x46, 0x48, 0x22, 0x48, 0x39, 0x0D, 0xF0, 0xF1, 0xF9, 0x94, 0xF8, 0x38, 0x04,
+ 0x40, 0x1E, 0x84, 0xF8, 0x38, 0x04, 0xBD, 0xE8, 0xF0, 0x87, 0x2D, 0xE9, 0xFF, 0x5F, 0x16, 0x46,
+ 0x0F, 0x46, 0x82, 0x46, 0x01, 0x2B, 0x2C, 0xD1, 0x01, 0x20, 0xB2, 0x4A, 0x10, 0xE0, 0x00, 0xBF,
+ 0x17, 0xF8, 0x10, 0x10, 0x11, 0xB1, 0x49, 0x1E, 0x07, 0xF8, 0x10, 0x10, 0x07, 0xEB, 0x40, 0x04,
+ 0x11, 0x78, 0x63, 0x78, 0x49, 0x1E, 0x8B, 0x42, 0x01, 0xDA, 0x5B, 0x1C, 0x63, 0x70, 0x40, 0x1C,
+ 0x39, 0x7E, 0x81, 0x42, 0xEC, 0xD2, 0x01, 0x20, 0xA7, 0x4B, 0x0F, 0xE0, 0x16, 0xF8, 0x10, 0x10,
+ 0x11, 0xB1, 0x49, 0x1E, 0x06, 0xF8, 0x10, 0x10, 0x06, 0xEB, 0x40, 0x02, 0x1C, 0x78, 0x51, 0x78,
+ 0x64, 0x1E, 0xA1, 0x42, 0x01, 0xDA, 0x49, 0x1C, 0x51, 0x70, 0x40, 0x1C, 0x31, 0x7E, 0x81, 0x42,
+ 0xEC, 0xD2, 0x9E, 0x48, 0x4F, 0xF0, 0x01, 0x09, 0xCB, 0x46, 0x00, 0x68, 0xB0, 0xF8, 0x8E, 0x02,
+ 0x40, 0x43, 0x03, 0x90, 0x00, 0x20, 0x01, 0x90, 0x62, 0xE0, 0x01, 0x20, 0x5A, 0xE0, 0x00, 0x20,
+ 0x00, 0x90, 0x9A, 0xF8, 0x38, 0x04, 0x06, 0xEB, 0x48, 0x03, 0x07, 0xEB, 0x49, 0x02, 0x19, 0xE0,
+ 0x00, 0xEB, 0xC0, 0x04, 0x0A, 0xEB, 0xC4, 0x01, 0x16, 0xF8, 0x18, 0x50, 0x11, 0xF8, 0x3E, 0x4F,
+ 0xAC, 0x42, 0x0F, 0xD3, 0x5D, 0x78, 0xAC, 0x42, 0x0C, 0xD8, 0x49, 0x78, 0x17, 0xF8, 0x19, 0x40,
+ 0xA1, 0x42, 0x07, 0xD3, 0x54, 0x78, 0xA1, 0x42, 0x04, 0xD8, 0x00, 0x9C, 0x0B, 0xFA, 0x00, 0xF1,
+ 0x21, 0x43, 0x00, 0x91, 0x40, 0x1E, 0xE3, 0xD2, 0x04, 0x21, 0x68, 0x46, 0x0B, 0xF0, 0x7A, 0xFE,
+ 0x01, 0x28, 0x2D, 0xDD, 0x00, 0x24, 0x58, 0x46, 0x00, 0x99, 0x01, 0xE0, 0x64, 0x1C, 0xE4, 0xB2,
+ 0x00, 0xFA, 0x04, 0xF2, 0x0A, 0x42, 0xF9, 0xD0, 0x04, 0xEB, 0xC4, 0x00, 0x0A, 0xEB, 0xC0, 0x00,
+ 0x9A, 0xF8, 0x38, 0x54, 0x0C, 0x30, 0x02, 0x90, 0x17, 0xE0, 0x00, 0xBF, 0x00, 0x99, 0x0B, 0xFA,
+ 0x05, 0xF0, 0x08, 0x42, 0x11, 0xD0, 0x05, 0xEB, 0xC5, 0x02, 0x0A, 0xEB, 0xC2, 0x01, 0x0C, 0x31,
+ 0x02, 0x98, 0x07, 0xF0, 0x3E, 0xFF, 0x03, 0x99, 0x88, 0x42, 0x06, 0xD2, 0xEA, 0xB2, 0x21, 0x46,
+ 0x50, 0x46, 0xFF, 0xF7, 0x0C, 0xFF, 0x01, 0x20, 0x01, 0x90, 0x6D, 0x1E, 0xA5, 0x42, 0xE5, 0xD8,
+ 0x08, 0xF1, 0x01, 0x00, 0x31, 0x7E, 0x80, 0x46, 0x41, 0x45, 0xA0, 0xD2, 0x09, 0xF1, 0x01, 0x09,
+ 0x39, 0x7E, 0x49, 0x45, 0x99, 0xD2, 0x01, 0x98, 0x04, 0xB0, 0xBD, 0xE8, 0xF0, 0x9F, 0x64, 0x48,
+ 0x10, 0xB5, 0x40, 0x78, 0x0C, 0x46, 0x00, 0x28, 0x52, 0xD0, 0x62, 0x48, 0x00, 0x78, 0x00, 0x28,
+ 0x4E, 0xD1, 0x61, 0x48, 0x00, 0x78, 0x00, 0x28, 0x4A, 0xD1, 0x5C, 0x48, 0x94, 0xF8, 0x38, 0x24,
+ 0x00, 0x68, 0x90, 0xF8, 0x89, 0x12, 0x8A, 0x42, 0x42, 0xD8, 0x00, 0x23, 0x09, 0xE0, 0x00, 0xBF,
+ 0x02, 0xEB, 0xC2, 0x01, 0x04, 0xEB, 0xC1, 0x01, 0xB1, 0xF9, 0x32, 0x10, 0x99, 0x42, 0x00, 0xDD,
+ 0x0B, 0x46, 0x52, 0x1E, 0xF4, 0xD2, 0x55, 0x49, 0x0B, 0x80, 0xB0, 0xF8, 0x8A, 0x12, 0x99, 0x42,
+ 0x2E, 0xDB, 0xB0, 0xF9, 0x8C, 0x12, 0x00, 0x22, 0x51, 0x48, 0xFB, 0xF7, 0x78, 0xFE, 0x50, 0x4A,
+ 0xD2, 0xE9, 0x00, 0x10, 0x01, 0x43, 0x23, 0xD0, 0xD2, 0xE9, 0x02, 0x01, 0x08, 0x43, 0x1F, 0xD0,
+ 0xA2, 0xF1, 0x40, 0x01, 0x01, 0x23, 0xA1, 0xF1, 0x40, 0x00, 0xFB, 0xF7, 0x41, 0xFB, 0x48, 0x48,
+ 0x80, 0x38, 0x00, 0x7E, 0x00, 0x28, 0x13, 0xD0, 0x45, 0x49, 0x40, 0x39, 0x09, 0x7E, 0x00, 0x29,
+ 0x0E, 0xD0, 0x48, 0x43, 0x94, 0xF8, 0x38, 0x14, 0x00, 0x23, 0x88, 0x42, 0x00, 0xD2, 0x01, 0x23,
+ 0x3F, 0x4A, 0x20, 0x46, 0x40, 0x3A, 0xBD, 0xE8, 0x10, 0x40, 0xA2, 0xF1, 0x40, 0x01, 0xFC, 0xE6,
+ 0x10, 0xBD, 0x3A, 0x49, 0x00, 0x20, 0xC8, 0x60, 0x39, 0x48, 0x0F, 0x21, 0x10, 0x30, 0x0D, 0xF0,
+ 0x3A, 0xB9, 0x90, 0xF8, 0x42, 0x00, 0x40, 0x07, 0x01, 0xD5, 0x01, 0x20, 0x70, 0x47, 0x00, 0x20,
+ 0x70, 0x47, 0x90, 0xF8, 0x3E, 0x01, 0x01, 0x28, 0x00, 0xD0, 0x00, 0x20, 0x70, 0x47, 0x90, 0xF8,
+ 0x3E, 0x01, 0x02, 0x28, 0x05, 0xD0, 0x03, 0x28, 0x03, 0xD0, 0x04, 0x28, 0x01, 0xD0, 0x00, 0x20,
+ 0x70, 0x47, 0x01, 0x20, 0x70, 0x47, 0x10, 0xB5, 0x90, 0xF8, 0x42, 0x10, 0x49, 0x06, 0x03, 0xD5,
+ 0xFF, 0xF7, 0xED, 0xFF, 0x01, 0x28, 0x00, 0xD0, 0x00, 0x20, 0x10, 0xBD, 0x91, 0xF8, 0x3E, 0x21,
+ 0x01, 0x2A, 0x08, 0xD0, 0x02, 0x2A, 0x06, 0xD0, 0x06, 0x20, 0x81, 0xF8, 0x3E, 0x01, 0x00, 0x20,
+ 0x81, 0xF8, 0x43, 0x01, 0x70, 0x47, 0x00, 0xF0, 0xCD, 0xBD, 0xF0, 0xB5, 0x91, 0xF9, 0x49, 0x61,
+ 0x93, 0xF9, 0x00, 0x40, 0x92, 0xF9, 0x49, 0x51, 0x77, 0x1C, 0x02, 0xD0, 0x6D, 0x1C, 0x15, 0xD0,
+ 0x0B, 0xE0, 0x6E, 0x1C, 0x0B, 0xD0, 0x00, 0xEB, 0x85, 0x02, 0x00, 0xEB, 0x84, 0x00, 0xD2, 0x6B,
+ 0xC2, 0xF8, 0x4C, 0x11, 0xC1, 0x63, 0x81, 0xF8, 0x49, 0x41, 0x1C, 0x70, 0xF0, 0xBD, 0x64, 0x1C,
+ 0x64, 0xB2, 0x40, 0xF8, 0x24, 0x10, 0x81, 0xF8, 0x49, 0x41, 0x02, 0xE0, 0x00, 0xEB, 0x86, 0x01,
+ 0xC9, 0x6B, 0x00, 0xEB, 0x84, 0x00, 0xC1, 0xF8, 0x4C, 0x21, 0xC2, 0x63, 0x82, 0xF8, 0x49, 0x41,
+ 0xEB, 0xE7, 0x00, 0x00, 0xB9, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0xEC, 0x06, 0x10, 0x00, 0x1C, 0x05, 0x10, 0x00, 0x28, 0x05, 0x10, 0x00, 0xE0, 0x05, 0x10, 0x00,
+ 0x60, 0x25, 0x10, 0x00, 0xF0, 0xB5, 0x44, 0x88, 0x4E, 0x88, 0x01, 0x25, 0xA4, 0x1B, 0x24, 0xB2,
+ 0x00, 0x2C, 0x01, 0xDB, 0x26, 0x46, 0x00, 0xE0, 0x66, 0x42, 0x5E, 0x80, 0x56, 0x42, 0x94, 0x42,
+ 0x02, 0xDD, 0x44, 0x88, 0xA4, 0x1A, 0x03, 0xE0, 0xB4, 0x42, 0x03, 0xDA, 0x44, 0x88, 0x14, 0x44,
+ 0x4C, 0x80, 0x00, 0x25, 0x04, 0x88, 0x0F, 0x88, 0xE4, 0x1B, 0x24, 0xB2, 0x00, 0x2C, 0x01, 0xDB,
+ 0x27, 0x46, 0x00, 0xE0, 0x67, 0x42, 0x1F, 0x80, 0x94, 0x42, 0x02, 0xDD, 0x00, 0x88, 0x80, 0x1A,
+ 0x03, 0xE0, 0xB4, 0x42, 0x03, 0xDA, 0x00, 0x88, 0x10, 0x44, 0x08, 0x80, 0x00, 0x25, 0x28, 0x46,
+ 0xF0, 0xBD, 0x2D, 0xE9, 0xF0, 0x41, 0x96, 0x4D, 0x0F, 0x46, 0x06, 0x46, 0x81, 0x89, 0x28, 0x68,
+ 0x90, 0x46, 0xF9, 0xB1, 0x90, 0xF8, 0xCC, 0x43, 0x0D, 0xF0, 0x3E, 0xF9, 0x18, 0xB1, 0x28, 0x68,
+ 0x90, 0xF8, 0xCE, 0x43, 0x12, 0xE0, 0xB0, 0x7D, 0x02, 0x28, 0x04, 0xD0, 0x01, 0x28, 0x06, 0xD0,
+ 0x03, 0x28, 0x08, 0xD0, 0x0A, 0xE0, 0x28, 0x68, 0x90, 0xF8, 0xD3, 0x43, 0x06, 0xE0, 0x28, 0x68,
+ 0x90, 0xF8, 0x53, 0x44, 0x02, 0xE0, 0x28, 0x68, 0x90, 0xF8, 0x2C, 0x44, 0xB0, 0x89, 0x40, 0x1E,
+ 0xB0, 0x81, 0x11, 0xE0, 0xB1, 0x7D, 0x90, 0xF8, 0xD0, 0x43, 0x02, 0x29, 0x04, 0xD0, 0x01, 0x29,
+ 0x05, 0xD0, 0x03, 0x29, 0x06, 0xD0, 0x07, 0xE0, 0x90, 0xF8, 0xD3, 0x43, 0x04, 0xE0, 0x90, 0xF8,
+ 0x54, 0x44, 0x01, 0xE0, 0x90, 0xF8, 0x2D, 0x44, 0x29, 0x68, 0xB2, 0x7D, 0x91, 0xF8, 0xCF, 0x03,
+ 0x02, 0x2A, 0x04, 0xD0, 0x01, 0x2A, 0x05, 0xD0, 0x03, 0x2A, 0x06, 0xD0, 0x07, 0xE0, 0x91, 0xF8,
+ 0xD2, 0x03, 0x04, 0xE0, 0x91, 0xF8, 0x55, 0x04, 0x01, 0xE0, 0x91, 0xF8, 0x2E, 0x04, 0x38, 0x80,
+ 0xA8, 0xF8, 0x00, 0x40, 0xBD, 0xE8, 0xF0, 0x81, 0x2D, 0xE9, 0xF0, 0x4F, 0x00, 0xF1, 0x5E, 0x04,
+ 0x85, 0xB0, 0x4F, 0xF0, 0x00, 0x0B, 0x84, 0xF8, 0x15, 0xB0, 0xDF, 0xF8, 0xA4, 0x91, 0x0E, 0x46,
+ 0x80, 0xF8, 0x74, 0xB0, 0x05, 0x46, 0xB0, 0xF8, 0x42, 0x10, 0x4F, 0xF0, 0x01, 0x08, 0xD9, 0xF8,
+ 0x00, 0x00, 0x4A, 0x06, 0xC2, 0x46, 0x01, 0xD5, 0x02, 0x21, 0x0F, 0xE0, 0x0A, 0x07, 0x06, 0xD5,
+ 0x90, 0xF8, 0x40, 0x24, 0x52, 0x06, 0x02, 0xD5, 0x84, 0xF8, 0x16, 0xA0, 0x07, 0xE0, 0x90, 0xF8,
+ 0x20, 0x24, 0x92, 0x07, 0x03, 0xD5, 0x09, 0x06, 0x01, 0xD5, 0x03, 0x21, 0xA1, 0x75, 0x95, 0xF8,
+ 0x3E, 0x11, 0x03, 0x29, 0x09, 0xD0, 0xA2, 0x7C, 0x01, 0x2A, 0x06, 0xD0, 0x04, 0x29, 0x2D, 0xD0,
+ 0x07, 0x29, 0x2B, 0xD0, 0x08, 0x29, 0x76, 0xD0, 0x9C, 0xE0, 0x90, 0xF8, 0xCB, 0x73, 0x0D, 0xF0,
+ 0xBB, 0xF8, 0x18, 0xB1, 0xD9, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0xCD, 0x73, 0x95, 0xF8, 0x3E, 0x01,
+ 0x03, 0x28, 0x00, 0xD0, 0x00, 0x27, 0xA0, 0x7D, 0x01, 0x28, 0x02, 0xD0, 0x03, 0x28, 0x05, 0xD0,
+ 0x08, 0xE0, 0xD9, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x52, 0x74, 0x03, 0xE0, 0xD9, 0xF8, 0x00, 0x00,
+ 0x90, 0xF8, 0x2B, 0x74, 0xA7, 0x81, 0x31, 0x68, 0x21, 0x60, 0x31, 0x68, 0x61, 0x60, 0x84, 0xF8,
+ 0x14, 0xA0, 0x84, 0xF8, 0x13, 0xA0, 0x84, 0xF8, 0x12, 0xB0, 0x73, 0xE0, 0x02, 0xAA, 0x01, 0xA9,
+ 0x20, 0x46, 0xFF, 0xF7, 0x46, 0xFF, 0x03, 0xAB, 0xBD, 0xF9, 0x08, 0x20, 0x21, 0x46, 0x30, 0x46,
+ 0xFF, 0xF7, 0x10, 0xFF, 0x83, 0x46, 0x04, 0xAB, 0xBD, 0xF9, 0x04, 0x20, 0x21, 0x1D, 0x30, 0x46,
+ 0xFF, 0xF7, 0x08, 0xFF, 0x07, 0x46, 0xBB, 0xF1, 0x01, 0x0F, 0x03, 0xD0, 0x20, 0x7D, 0x01, 0x28,
+ 0x4D, 0xD0, 0x4E, 0xE0, 0xBD, 0xF9, 0x0C, 0x00, 0xBD, 0xF9, 0x0E, 0x10, 0x81, 0x42, 0x00, 0xDD,
+ 0x08, 0x46, 0xD9, 0xF8, 0x00, 0x10, 0x91, 0xF8, 0xD1, 0x23, 0x02, 0xF0, 0x0F, 0x03, 0x4F, 0xEA,
+ 0x12, 0x1C, 0xA2, 0x7D, 0x02, 0x2A, 0x04, 0xD0, 0x01, 0x2A, 0x05, 0xD0, 0x03, 0x2A, 0x06, 0xD0,
+ 0x0B, 0xE0, 0x91, 0xF8, 0xD4, 0x13, 0x04, 0xE0, 0x91, 0xF8, 0x56, 0x14, 0x01, 0xE0, 0x91, 0xF8,
+ 0x2F, 0x14, 0x01, 0xF0, 0x0F, 0x03, 0x4F, 0xEA, 0x11, 0x1C, 0x01, 0x2F, 0x0C, 0xD0, 0xCD, 0xF8,
+ 0x00, 0xC0, 0xBD, 0xF9, 0x08, 0x20, 0xBD, 0xF9, 0x04, 0x10, 0x07, 0xF0, 0x30, 0xFD, 0x21, 0x7D,
+ 0x01, 0x29, 0x0D, 0xD0, 0x18, 0xE0, 0x23, 0xE0, 0xA0, 0x68, 0x30, 0x60, 0xD9, 0xF8, 0x00, 0x00,
+ 0x90, 0xF8, 0x80, 0x03, 0x00, 0x06, 0x01, 0xD4, 0x4F, 0xF0, 0x00, 0x08, 0x60, 0x46, 0xEE, 0xE7,
+ 0x95, 0xF8, 0x41, 0x11, 0x41, 0xB9, 0xD9, 0xF8, 0x00, 0x10, 0x91, 0xF8, 0xBC, 0x13, 0x0A, 0x09,
+ 0xD4, 0xF8, 0x0E, 0x10, 0x07, 0xF0, 0x28, 0xFD, 0xC4, 0xF8, 0x0E, 0x00, 0x01, 0xE0, 0x84, 0xF8,
+ 0x15, 0xA0, 0x84, 0xF8, 0x14, 0xB0, 0xE7, 0x74, 0xB8, 0xF1, 0x00, 0x0F, 0x02, 0xD1, 0x03, 0xE0,
+ 0xA0, 0x68, 0x30, 0x60, 0x30, 0x68, 0xA0, 0x60, 0x05, 0xB0, 0x40, 0x46, 0xBD, 0xE8, 0xF0, 0x8F,
+ 0x20, 0x07, 0x10, 0x00, 0x2D, 0xE9, 0xF0, 0x41, 0x17, 0x46, 0x06, 0x46, 0xDD, 0xE9, 0x06, 0xC2,
+ 0xC8, 0x78, 0x00, 0x25, 0x0C, 0x46, 0x03, 0x28, 0x30, 0xD2, 0xFE, 0x48, 0xDF, 0xF8, 0xF8, 0xE3,
+ 0x01, 0x68, 0x91, 0xF8, 0x0A, 0x14, 0xA1, 0x70, 0xDE, 0xF8, 0x04, 0xE0, 0xBE, 0xF1, 0x00, 0x0F,
+ 0x06, 0xDD, 0xD0, 0xF8, 0x00, 0xE0, 0x9E, 0xF8, 0x0B, 0xE4, 0xA1, 0xEB, 0x0E, 0x01, 0xA1, 0x70,
+ 0x00, 0x68, 0x71, 0x6B, 0xB0, 0xF8, 0x0C, 0x04, 0x81, 0x42, 0x15, 0xDD, 0xDB, 0xB2, 0x39, 0x46,
+ 0x60, 0x46, 0x00, 0xF0, 0xE8, 0xFA, 0xE0, 0x78, 0x40, 0x1C, 0xC0, 0xB2, 0xE0, 0x70, 0x03, 0x28,
+ 0x06, 0xD0, 0xEE, 0x49, 0xB6, 0xF9, 0x56, 0x00, 0xB1, 0xF9, 0x00, 0x10, 0x88, 0x42, 0x05, 0xDD,
+ 0x03, 0x20, 0xE0, 0x70, 0x01, 0x25, 0x01, 0xE0, 0x00, 0x20, 0xE0, 0x70, 0x28, 0x46, 0xBD, 0xE8,
+ 0xF0, 0x81, 0x2D, 0xE9, 0xFC, 0x5F, 0x4F, 0xF0, 0x00, 0x0B, 0x0C, 0x46, 0x81, 0x46, 0x5D, 0x46,
+ 0x4F, 0xF0, 0x02, 0x0A, 0x01, 0x26, 0x00, 0xF5, 0x80, 0x57, 0x7C, 0xE0, 0xD7, 0xF8, 0xB0, 0x03,
+ 0x06, 0xFA, 0x05, 0xFE, 0x10, 0xEA, 0x0E, 0x0F, 0x74, 0xD0, 0xC5, 0xEB, 0xC5, 0x00, 0x00, 0xEB,
+ 0x40, 0x01, 0x09, 0xEB, 0x01, 0x10, 0x00, 0xF5, 0x9A, 0x71, 0x90, 0xF8, 0x3E, 0x81, 0xB0, 0xF9,
+ 0x0C, 0xC0, 0xB8, 0xF1, 0x01, 0x0F, 0x0F, 0xD0, 0xB8, 0xF1, 0x02, 0x0F, 0x0C, 0xD0, 0x90, 0xF8,
+ 0x42, 0x00, 0x40, 0x07, 0x51, 0xD5, 0xD2, 0x48, 0xEB, 0xB2, 0x61, 0x46, 0xB0, 0xF9, 0x08, 0x20,
+ 0x20, 0x46, 0x00, 0xF0, 0xA0, 0xFA, 0x55, 0xE0, 0xCE, 0x4B, 0xB0, 0xF9, 0x54, 0x20, 0xB3, 0xF9,
+ 0x00, 0x30, 0x9A, 0x42, 0x4E, 0xDB, 0xB8, 0xF1, 0x01, 0x0F, 0x03, 0xD1, 0x00, 0x22, 0x0A, 0x60,
+ 0x4A, 0x60, 0x0A, 0x81, 0xC6, 0x4A, 0xB2, 0xF9, 0x00, 0x30, 0x63, 0x45, 0x03, 0xDC, 0xB2, 0xF9,
+ 0x02, 0x30, 0x63, 0x45, 0x0C, 0xDA, 0xB0, 0xF8, 0x42, 0x30, 0x43, 0xF0, 0x04, 0x03, 0xA0, 0xF8,
+ 0x42, 0x30, 0xC3, 0x68, 0x4B, 0x60, 0x81, 0xF8, 0x08, 0xA0, 0xB2, 0xF9, 0x00, 0x20, 0x19, 0xE0,
+ 0xB2, 0xF9, 0x08, 0x30, 0x63, 0x45, 0x03, 0xDC, 0xB2, 0xF9, 0x0A, 0x30, 0x63, 0x45, 0x19, 0xDA,
+ 0xB0, 0xF8, 0x42, 0x30, 0x43, 0xF0, 0x04, 0x03, 0xA0, 0xF8, 0x42, 0x30, 0x90, 0xF8, 0x3E, 0x31,
+ 0x01, 0x2B, 0x02, 0xD0, 0x0B, 0x7A, 0x02, 0x2B, 0x01, 0xD1, 0xC3, 0x68, 0x4B, 0x60, 0x0E, 0x72,
+ 0xB2, 0xF9, 0x08, 0x20, 0x2B, 0x46, 0xCD, 0xE9, 0x00, 0x42, 0x62, 0x46, 0xFF, 0xF7, 0x4A, 0xFF,
+ 0x83, 0x46, 0x0F, 0xE0, 0x00, 0x20, 0x08, 0x72, 0x08, 0xE0, 0xA9, 0x48, 0xB0, 0xF9, 0x08, 0x10,
+ 0x61, 0x45, 0x07, 0xDA, 0xB0, 0xF9, 0x0A, 0x00, 0x60, 0x45, 0x03, 0xDD, 0xA0, 0x68, 0x40, 0xEA,
+ 0x0E, 0x00, 0xA0, 0x60, 0x6D, 0x1C, 0x97, 0xF9, 0xB8, 0x03, 0xA8, 0x42, 0xBF, 0xF6, 0x7E, 0xAF,
+ 0x20, 0x68, 0x61, 0x68, 0x9C, 0x4A, 0x50, 0xEA, 0x01, 0x03, 0xA3, 0x68, 0x01, 0xD0, 0x23, 0xB1,
+ 0x04, 0xE0, 0x1B, 0xB1, 0x82, 0xF8, 0x00, 0xA0, 0x00, 0xE0, 0x16, 0x70, 0x98, 0x4A, 0x18, 0x32,
+ 0x13, 0x68, 0x98, 0x43, 0x20, 0x60, 0x50, 0x68, 0x81, 0x43, 0x61, 0x60, 0x96, 0x49, 0xA0, 0x68,
+ 0x09, 0x78, 0x08, 0x43, 0x05, 0xD1, 0x92, 0x49, 0x18, 0x22, 0x18, 0x39, 0x90, 0x48, 0x0C, 0xF0,
+ 0x1E, 0xFE, 0x58, 0x46, 0xBD, 0xE8, 0xFC, 0x9F, 0x8B, 0x4A, 0x10, 0xB5, 0x12, 0x78, 0x00, 0x21,
+ 0x01, 0x2A, 0x01, 0xD1, 0x03, 0xB1, 0x01, 0x21, 0x90, 0xF8, 0x3E, 0x21, 0x04, 0x2A, 0x18, 0xD1,
+ 0x84, 0x4A, 0x12, 0x68, 0x92, 0xF8, 0x00, 0x34, 0x5B, 0x07, 0x12, 0xD5, 0xB0, 0xF8, 0x4A, 0x30,
+ 0xB0, 0xF8, 0x4C, 0x40, 0x1B, 0x1B, 0x82, 0x4C, 0xA4, 0x8A, 0xA3, 0x42, 0x09, 0xDD, 0x90, 0xF8,
+ 0x5B, 0x30, 0x90, 0xF8, 0x5A, 0x00, 0x92, 0xF8, 0x09, 0x24, 0x18, 0x1A, 0x90, 0x42, 0x00, 0xDD,
+ 0x01, 0x21, 0x08, 0x46, 0x10, 0xBD, 0x30, 0xB5, 0x79, 0x4D, 0x00, 0x24, 0x18, 0x35, 0x09, 0xB1,
+ 0x29, 0x68, 0x11, 0xB9, 0x12, 0xB1, 0x69, 0x68, 0x01, 0xB1, 0x01, 0x24, 0x72, 0x49, 0x09, 0x78,
+ 0x01, 0x29, 0x01, 0xD1, 0x03, 0xB1, 0x01, 0x24, 0x6E, 0x49, 0x09, 0x68, 0x91, 0xF8, 0x00, 0x24,
+ 0x52, 0x07, 0x16, 0xD5, 0x90, 0xF8, 0x3E, 0x21, 0x04, 0x2A, 0x12, 0xD1, 0xB0, 0xF8, 0x4A, 0x20,
+ 0xB0, 0xF8, 0x4C, 0x30, 0xD2, 0x1A, 0x6A, 0x4B, 0x9B, 0x8A, 0x9A, 0x42, 0x09, 0xDD, 0x90, 0xF8,
+ 0x5B, 0x20, 0x90, 0xF8, 0x5A, 0x00, 0x91, 0xF8, 0x09, 0x14, 0x10, 0x1A, 0x88, 0x42, 0x00, 0xDD,
+ 0x01, 0x24, 0x20, 0x46, 0x30, 0xBD, 0x2D, 0xE9, 0xF3, 0x4F, 0x8D, 0xB0, 0x5E, 0x48, 0x00, 0x25,
+ 0x8B, 0x46, 0x0A, 0x95, 0x01, 0x78, 0x5E, 0x48, 0x18, 0x38, 0x01, 0x29, 0x06, 0xD1, 0xDB, 0xF8,
+ 0x08, 0x10, 0x19, 0xB1, 0xD0, 0xF8, 0x1C, 0x80, 0x47, 0x6A, 0x02, 0xE0, 0xD0, 0xF8, 0x18, 0x80,
+ 0x07, 0x6A, 0x0D, 0x9E, 0x00, 0x24, 0x4F, 0xF0, 0x01, 0x09, 0x06, 0xF5, 0x80, 0x56, 0x0F, 0xE1,
+ 0x09, 0xFA, 0x04, 0xF0, 0xD6, 0xF8, 0xB0, 0x13, 0x0B, 0x90, 0x01, 0x42, 0x1A, 0xD0, 0xC4, 0xEB,
+ 0xC4, 0x00, 0x00, 0xEB, 0x40, 0x01, 0x0D, 0x98, 0x00, 0xEB, 0x01, 0x1A, 0x9A, 0xF8, 0x42, 0x00,
+ 0x40, 0x07, 0x0F, 0xD5, 0x0A, 0xF5, 0x9A, 0x70, 0x01, 0x90, 0xBA, 0xF9, 0x0C, 0x00, 0x02, 0x90,
+ 0x9A, 0xF8, 0x3E, 0x01, 0x01, 0x28, 0x06, 0xD0, 0x02, 0x28, 0x04, 0xD0, 0x04, 0x28, 0x1F, 0xD0,
+ 0x05, 0x28, 0x7D, 0xD0, 0xEB, 0xE0, 0xDB, 0xF8, 0x00, 0x10, 0xDB, 0xF8, 0x04, 0x20, 0xDB, 0xF8,
+ 0x08, 0x30, 0x50, 0x46, 0xFF, 0xF7, 0x60, 0xFF, 0x01, 0x28, 0x02, 0xD1, 0x01, 0x99, 0x81, 0xF8,
+ 0x01, 0x90, 0xDB, 0xF8, 0x00, 0x10, 0xDB, 0xF8, 0x04, 0x20, 0xDB, 0xF8, 0x08, 0x30, 0x50, 0x46,
+ 0xFF, 0xF7, 0x79, 0xFF, 0x01, 0x28, 0xE5, 0xD1, 0x01, 0x99, 0x00, 0x20, 0x88, 0x70, 0xCE, 0xE0,
+ 0xDB, 0xF8, 0x00, 0x10, 0xDB, 0xF8, 0x04, 0x20, 0xDB, 0xF8, 0x08, 0x30, 0x50, 0x46, 0xFF, 0xF7,
+ 0x6A, 0xFF, 0x01, 0x28, 0x09, 0xD0, 0x01, 0x98, 0x80, 0x78, 0x48, 0xB1, 0x01, 0x99, 0x40, 0x1E,
+ 0x10, 0xF0, 0xFF, 0x00, 0x88, 0x70, 0x03, 0xD0, 0x07, 0xE0, 0x01, 0x99, 0x00, 0x20, 0x88, 0x70,
+ 0x9A, 0xF8, 0x46, 0x11, 0x09, 0xFA, 0x01, 0xF0, 0x05, 0x43, 0xDB, 0xF8, 0x00, 0x10, 0xDB, 0xF8,
+ 0x04, 0x20, 0xDB, 0xF8, 0x08, 0x30, 0x50, 0x46, 0xFF, 0xF7, 0x26, 0xFF, 0x01, 0x28, 0x02, 0xD1,
+ 0x01, 0x99, 0x81, 0xF8, 0x01, 0x90, 0x01, 0x98, 0xC0, 0x78, 0x03, 0x28, 0x1D, 0xD2, 0x01, 0x99,
+ 0x89, 0x78, 0xD1, 0xB1, 0x17, 0x49, 0xDA, 0xF8, 0x34, 0x20, 0x09, 0x68, 0xB1, 0xF8, 0x0C, 0x14,
+ 0x8A, 0x42, 0x0F, 0xDD, 0x40, 0x1C, 0x01, 0x99, 0xC0, 0xB2, 0x03, 0x28, 0xC8, 0x70, 0x06, 0xD0,
+ 0x12, 0x49, 0xBA, 0xF9, 0x56, 0x00, 0xB1, 0xF9, 0x00, 0x10, 0x88, 0x42, 0x05, 0xDD, 0x01, 0x20,
+ 0x0A, 0x90, 0x02, 0xE0, 0x01, 0x99, 0x00, 0x20, 0xC8, 0x70, 0x02, 0x98, 0x0F, 0xFA, 0x88, 0xF2,
+ 0x82, 0x42, 0x04, 0xDC, 0x02, 0x98, 0x4F, 0xEA, 0x28, 0x41, 0x81, 0x42, 0x16, 0xDA, 0x01, 0x98,
+ 0x02, 0x21, 0xE3, 0xB2, 0x01, 0x72, 0x01, 0x98, 0xDA, 0xF8, 0x0C, 0x10, 0x41, 0x60, 0x37, 0xE0,
+ 0x56, 0xE0, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00, 0xF0, 0x05, 0x10, 0x00, 0x68, 0x06, 0x10, 0x00,
+ 0x98, 0x25, 0x10, 0x00, 0x7C, 0x06, 0x10, 0x00, 0xF3, 0x06, 0x10, 0x00, 0x38, 0xB2, 0x02, 0x99,
+ 0x04, 0x90, 0x88, 0x42, 0x03, 0xDC, 0x02, 0x98, 0x39, 0x14, 0x81, 0x42, 0x25, 0xDA, 0x01, 0x98,
+ 0x00, 0x7A, 0x02, 0x28, 0x06, 0xD1, 0x01, 0x99, 0x81, 0xF8, 0x08, 0x90, 0x01, 0x98, 0xDA, 0xF8,
+ 0x0C, 0x10, 0x41, 0x60, 0x01, 0x98, 0x40, 0x78, 0x80, 0xB9, 0x01, 0x98, 0x53, 0x46, 0x80, 0x78,
+ 0x00, 0x28, 0x82, 0x48, 0x02, 0x69, 0x04, 0xD0, 0x01, 0x98, 0x0A, 0xF1, 0x0C, 0x01, 0x00, 0x1D,
+ 0x02, 0xE0, 0x0A, 0xF1, 0x10, 0x01, 0x08, 0x1F, 0x00, 0xF0, 0xE2, 0xF8, 0x04, 0x9A, 0xE3, 0xB2,
+ 0x58, 0x46, 0x02, 0x99, 0x00, 0xF0, 0xCF, 0xF8, 0x31, 0xE0, 0xDB, 0xF8, 0x08, 0x00, 0x0B, 0x99,
+ 0x08, 0x43, 0xCB, 0xF8, 0x08, 0x00, 0x01, 0x98, 0x80, 0x78, 0x20, 0xB1, 0x01, 0x99, 0x00, 0x22,
+ 0x50, 0x46, 0x09, 0x1D, 0x03, 0xE0, 0x01, 0x22, 0x0A, 0xF1, 0x0C, 0x01, 0x50, 0x46, 0x00, 0xF0,
+ 0xA8, 0xF8, 0xBA, 0xF8, 0x42, 0x00, 0x20, 0xF0, 0x04, 0x01, 0xAA, 0xF8, 0x42, 0x10, 0x16, 0xE0,
+ 0x01, 0x98, 0x00, 0x7A, 0x01, 0x28, 0x12, 0xD1, 0x01, 0x98, 0x80, 0x78, 0x78, 0xB1, 0x01, 0x99,
+ 0x01, 0x22, 0x50, 0x46, 0x09, 0x1D, 0x00, 0xF0, 0x94, 0xF8, 0xBA, 0xF8, 0x42, 0x00, 0x20, 0xF0,
+ 0x04, 0x01, 0xAA, 0xF8, 0x42, 0x10, 0xBA, 0xF8, 0x56, 0x10, 0xAA, 0xF8, 0x54, 0x10, 0x64, 0x1C,
+ 0x96, 0xF9, 0xB8, 0x03, 0xA0, 0x42, 0xBF, 0xF6, 0xEB, 0xAE, 0x5D, 0x48, 0x00, 0x68, 0x90, 0xF8,
+ 0x00, 0x04, 0x80, 0x07, 0x02, 0xD5, 0x28, 0x46, 0xFB, 0xF7, 0xA7, 0xFF, 0x59, 0x48, 0x15, 0xB1,
+ 0x80, 0xF8, 0x00, 0x90, 0x01, 0xE0, 0x00, 0x21, 0x01, 0x70, 0x0A, 0x98, 0x0F, 0xB0, 0xBD, 0xE8,
+ 0xF0, 0x8F, 0x53, 0x49, 0x0B, 0x68, 0x51, 0x49, 0x9A, 0x8E, 0x18, 0x39, 0x12, 0x1A, 0x08, 0x80,
+ 0x4A, 0x80, 0x93, 0xF8, 0x0F, 0x34, 0x18, 0x44, 0x88, 0x80, 0xD0, 0x1A, 0xC8, 0x80, 0x70, 0x47,
+ 0x4B, 0x49, 0x0B, 0x68, 0x49, 0x49, 0x9A, 0x8E, 0x18, 0x39, 0x12, 0x1A, 0x08, 0x81, 0x4A, 0x81,
+ 0x93, 0xF8, 0x12, 0x34, 0x18, 0x44, 0x88, 0x81, 0xD0, 0x1A, 0xC8, 0x81, 0x70, 0x47, 0x10, 0xB5,
+ 0x43, 0x4C, 0x20, 0x68, 0x90, 0xF8, 0x0E, 0x04, 0xFF, 0xF7, 0xDB, 0xFF, 0x20, 0x68, 0x90, 0xF8,
+ 0x0E, 0x04, 0xFF, 0xF7, 0xE5, 0xFF, 0x22, 0x68, 0x3C, 0x49, 0x92, 0xF8, 0x10, 0x04, 0x18, 0x39,
+ 0x40, 0x43, 0x08, 0x61, 0x92, 0xF8, 0x11, 0x04, 0x88, 0x82, 0x18, 0x22, 0x37, 0x48, 0x0C, 0xF0,
+ 0x46, 0xFC, 0x38, 0x49, 0x00, 0x20, 0x35, 0x4A, 0x08, 0x70, 0x37, 0x49, 0x18, 0x3A, 0x48, 0x60,
+ 0x10, 0x63, 0x50, 0x63, 0x90, 0x63, 0x35, 0x4A, 0x10, 0x70, 0x08, 0x70, 0x10, 0xBD, 0xFE, 0xB5,
+ 0x00, 0x24, 0x06, 0x46, 0x00, 0x94, 0x01, 0x94, 0x69, 0x46, 0x02, 0x94, 0xFF, 0xF7, 0x61, 0xFD,
+ 0x05, 0x46, 0x69, 0x46, 0x30, 0x46, 0xFF, 0xF7, 0x66, 0xFE, 0x40, 0xEA, 0x05, 0x01, 0x2A, 0x48,
+ 0x01, 0x29, 0x01, 0xD0, 0x41, 0x68, 0x05, 0xE0, 0x25, 0x49, 0x09, 0x68, 0x91, 0xF8, 0x0A, 0x14,
+ 0x49, 0x1F, 0x41, 0x60, 0x00, 0x29, 0x04, 0xDD, 0x49, 0x1E, 0x41, 0x60, 0x21, 0x49, 0x09, 0x78,
+ 0x01, 0xB9, 0x44, 0x60, 0x1D, 0x48, 0x9D, 0xE8, 0x0E, 0x00, 0x18, 0x30, 0x80, 0xE8, 0x0E, 0x00,
+ 0xFE, 0xBD, 0x02, 0x23, 0x80, 0xF8, 0x3E, 0x31, 0x00, 0x23, 0x80, 0xF8, 0x42, 0x31, 0x01, 0x2A,
+ 0x05, 0xD0, 0x09, 0x68, 0x01, 0x62, 0x01, 0x21, 0x80, 0xF8, 0x34, 0x11, 0x70, 0x47, 0x03, 0x22,
+ 0x0C, 0x30, 0x07, 0xF0, 0xBB, 0xB9, 0x10, 0xB5, 0x01, 0x24, 0x9C, 0x40, 0x91, 0x42, 0x03, 0xDA,
+ 0x01, 0x68, 0x21, 0x43, 0x01, 0x60, 0x10, 0xBD, 0x41, 0x68, 0x21, 0x43, 0x41, 0x60, 0x10, 0xBD,
+ 0x70, 0xB5, 0x15, 0x46, 0x0A, 0x46, 0x06, 0x46, 0x01, 0x46, 0x1C, 0x46, 0x10, 0x46, 0x07, 0xF0,
+ 0xC8, 0xF9, 0xA8, 0x42, 0x09, 0xDD, 0x00, 0x22, 0x31, 0x46, 0x20, 0x46, 0xFF, 0xF7, 0xD1, 0xFF,
+ 0x34, 0xF8, 0x42, 0x0F, 0x20, 0xF0, 0x04, 0x00, 0x20, 0x80, 0x70, 0xBD, 0x98, 0x25, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0x80, 0x06, 0x10, 0x00, 0xF0, 0x05, 0x10, 0x00, 0x81, 0x06, 0x10, 0x00,
+ 0x10, 0xB5, 0x41, 0xF2, 0xA8, 0x41, 0x9F, 0x48, 0x0C, 0xF0, 0x25, 0xFC, 0x40, 0xF2, 0x3C, 0x41,
+ 0x9D, 0x48, 0x0C, 0xF0, 0x20, 0xFC, 0x9D, 0x48, 0xFF, 0x21, 0x80, 0xF8, 0xB8, 0x13, 0x00, 0xF0,
+ 0xFC, 0xFB, 0xFF, 0xF7, 0xD6, 0xFA, 0xFF, 0xF7, 0x5A, 0xFF, 0x00, 0xF0, 0x37, 0xF9, 0xFD, 0xF7,
+ 0x2B, 0xFA, 0xFD, 0xF7, 0x02, 0xFF, 0xFE, 0xF7, 0x87, 0xFC, 0xBD, 0xE8, 0x10, 0x40, 0xFC, 0xF7,
+ 0x37, 0xBF, 0x70, 0xB5, 0x92, 0x4D, 0x00, 0x24, 0x8E, 0x48, 0xAC, 0x71, 0x00, 0xF0, 0xEB, 0xFB,
+ 0x90, 0x4E, 0x30, 0x68, 0x90, 0xF8, 0x40, 0x04, 0xC0, 0x06, 0x05, 0xD5, 0x8E, 0x48, 0x00, 0x79,
+ 0x10, 0xB1, 0x88, 0x48, 0xFF, 0xF7, 0x5F, 0xF8, 0x30, 0x68, 0x90, 0xF8, 0x00, 0x04, 0xC0, 0x07,
+ 0x05, 0xD0, 0x84, 0x48, 0xFF, 0xF7, 0x5B, 0xFF, 0x82, 0x48, 0x00, 0xF0, 0x10, 0xF9, 0x81, 0x48,
+ 0xFD, 0xF7, 0x0A, 0xFA, 0x7F, 0x48, 0xFD, 0xF7, 0xE6, 0xFE, 0x7F, 0x48, 0x80, 0xF8, 0x38, 0x44,
+ 0x7E, 0x48, 0xB0, 0xF8, 0xBC, 0x03, 0xA5, 0xF8, 0x1F, 0x00, 0x70, 0xBD, 0x70, 0xB5, 0x7A, 0x4D,
+ 0x7B, 0x4E, 0x95, 0xF8, 0x38, 0x04, 0xD8, 0xB3, 0x7A, 0x4C, 0xB0, 0x71, 0x20, 0x68, 0x90, 0xF8,
+ 0x80, 0x02, 0x80, 0x07, 0x03, 0xD5, 0x29, 0x46, 0x72, 0x48, 0xFF, 0xF7, 0x30, 0xFA, 0x72, 0x49,
+ 0x70, 0x48, 0x00, 0xF0, 0xE3, 0xFB, 0x20, 0x68, 0x90, 0xF8, 0x40, 0x04, 0xC0, 0x06, 0x05, 0xD5,
+ 0x71, 0x48, 0x00, 0x79, 0x10, 0xB1, 0x6B, 0x48, 0xFF, 0xF7, 0x25, 0xF8, 0x20, 0x68, 0x90, 0xF8,
+ 0x00, 0x04, 0xC0, 0x07, 0x02, 0xD0, 0x67, 0x48, 0xFF, 0xF7, 0x21, 0xFF, 0x20, 0x68, 0x90, 0xF8,
+ 0x80, 0x03, 0xC0, 0x07, 0x02, 0xD0, 0x63, 0x48, 0xFE, 0xF7, 0xAC, 0xFF, 0x61, 0x48, 0x00, 0xF0,
+ 0xCE, 0xF8, 0x60, 0x48, 0xFD, 0xF7, 0xC8, 0xF9, 0x5E, 0x48, 0xFD, 0xF7, 0xA4, 0xFE, 0x00, 0x20,
+ 0x85, 0xF8, 0x38, 0x04, 0x5D, 0x48, 0xB0, 0xF8, 0xBC, 0x03, 0xA6, 0xF8, 0x1F, 0x00, 0x70, 0xBD,
+ 0xFF, 0xE7, 0xFF, 0xF7, 0x8E, 0xFF, 0xF5, 0xE7, 0x10, 0xB5, 0x59, 0x48, 0x00, 0x24, 0x84, 0x71,
+ 0x54, 0x48, 0x00, 0xF0, 0x78, 0xFB, 0x53, 0x48, 0xFD, 0xF7, 0xAE, 0xF9, 0x51, 0x48, 0xFC, 0xF7,
+ 0xCA, 0xFE, 0x51, 0x48, 0x80, 0xF8, 0x38, 0x44, 0x10, 0xBD, 0x10, 0xB5, 0x4E, 0x4C, 0x94, 0xF8,
+ 0x38, 0x04, 0x90, 0xB1, 0x4E, 0x49, 0x88, 0x71, 0x21, 0x46, 0x4A, 0x48, 0x00, 0xF0, 0x96, 0xFB,
+ 0x48, 0x48, 0x00, 0xF0, 0x9C, 0xF8, 0x47, 0x48, 0xFD, 0xF7, 0x96, 0xF9, 0x45, 0x48, 0xFC, 0xF7,
+ 0xB2, 0xFE, 0x00, 0x20, 0x84, 0xF8, 0x38, 0x04, 0x10, 0xBD, 0xBD, 0xE8, 0x10, 0x40, 0xD3, 0xE7,
+ 0x10, 0xB5, 0x41, 0x4C, 0x01, 0x46, 0x94, 0xF8, 0x38, 0x04, 0x0F, 0x28, 0x0B, 0xD2, 0x00, 0xEB,
+ 0xC0, 0x00, 0x04, 0xEB, 0xC0, 0x00, 0x48, 0x22, 0x0C, 0xF0, 0x01, 0xFB, 0x94, 0xF8, 0x38, 0x04,
+ 0x40, 0x1C, 0x84, 0xF8, 0x38, 0x04, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0xF5, 0x80, 0x50, 0x00, 0x21,
+ 0xD0, 0xF8, 0xB0, 0x33, 0x01, 0x22, 0x00, 0xBF, 0x02, 0xFA, 0x01, 0xF4, 0x23, 0x42, 0x0B, 0xD1,
+ 0x8A, 0x40, 0x13, 0x43, 0xC0, 0xF8, 0xB0, 0x33, 0x90, 0xF9, 0xB8, 0x23, 0x8A, 0x42, 0x01, 0xDA,
+ 0x80, 0xF8, 0xB8, 0x13, 0x48, 0xB2, 0x10, 0xBD, 0x49, 0x1C, 0x0F, 0x29, 0xEC, 0xDB, 0x4F, 0xF0,
+ 0xFF, 0x30, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0xF5, 0x80, 0x50, 0x91, 0xF8, 0x4A, 0x41, 0x01, 0x23,
+ 0xD0, 0xF8, 0xB0, 0x23, 0xA3, 0x40, 0x9A, 0x43, 0xC0, 0xF8, 0xB0, 0x23, 0x00, 0x22, 0x81, 0xF8,
+ 0x3E, 0x21, 0x90, 0xF9, 0xB8, 0x23, 0x91, 0xF9, 0x4A, 0x11, 0x8A, 0x42, 0x07, 0xD1, 0xD0, 0xF8,
+ 0xB0, 0x13, 0xB1, 0xFA, 0x81, 0xF1, 0xC1, 0xF1, 0x1F, 0x01, 0x80, 0xF8, 0xB8, 0x13, 0x10, 0xBD,
+ 0x30, 0xB5, 0x00, 0xF5, 0x80, 0x50, 0x00, 0x22, 0xD0, 0xF8, 0xB4, 0x43, 0x01, 0x23, 0x00, 0xBF,
+ 0x03, 0xFA, 0x02, 0xF5, 0x2C, 0x42, 0x06, 0xD1, 0x93, 0x40, 0x1C, 0x43, 0xC0, 0xF8, 0xB4, 0x43,
+ 0x81, 0xF8, 0x4B, 0x21, 0x30, 0xBD, 0x52, 0x1C, 0x52, 0xB2, 0x0A, 0x2A, 0xF0, 0xDB, 0xFF, 0x20,
+ 0x81, 0xF8, 0x4B, 0x01, 0x30, 0xBD, 0x10, 0xB5, 0x00, 0xF5, 0x80, 0x50, 0x91, 0xF8, 0x4B, 0x41,
+ 0x01, 0x23, 0xD0, 0xF8, 0xB4, 0x23, 0xA3, 0x40, 0x9A, 0x43, 0xC0, 0xF8, 0xB4, 0x23, 0xFF, 0x20,
+ 0x81, 0xF8, 0x4B, 0x01, 0x10, 0xBD, 0x04, 0x48, 0x70, 0x47, 0x03, 0x48, 0x90, 0xF8, 0x38, 0x04,
+ 0x70, 0x47, 0x00, 0x00, 0xBC, 0x25, 0x10, 0x00, 0x64, 0x3A, 0x10, 0x00, 0xBC, 0x35, 0x10, 0x00,
+ 0xE6, 0x43, 0x01, 0x20, 0x20, 0x07, 0x10, 0x00, 0xEC, 0x06, 0x10, 0x00, 0x70, 0x47, 0x2D, 0xE9,
+ 0xF0, 0x47, 0x00, 0x24, 0x06, 0x46, 0x01, 0x27, 0xDF, 0xF8, 0xE0, 0x80, 0x4F, 0xF0, 0x05, 0x09,
+ 0x00, 0xF5, 0x80, 0x55, 0x64, 0xE0, 0x00, 0xBF, 0xD5, 0xF8, 0xB0, 0x03, 0x07, 0xFA, 0x04, 0xF1,
+ 0x08, 0x42, 0x5B, 0xD0, 0xC4, 0xEB, 0xC4, 0x00, 0x00, 0xEB, 0x40, 0x00, 0x06, 0xEB, 0x00, 0x11,
+ 0x91, 0xF8, 0x3E, 0x01, 0x01, 0x28, 0x04, 0xD0, 0x02, 0x28, 0x02, 0xD0, 0x04, 0x28, 0x27, 0xD0,
+ 0x4C, 0xE0, 0x91, 0xF8, 0x3F, 0x01, 0x03, 0x28, 0x01, 0xD0, 0x04, 0x28, 0x46, 0xD1, 0x28, 0x48,
+ 0xB1, 0xF8, 0x42, 0x20, 0xB0, 0xF9, 0x00, 0x00, 0x53, 0x07, 0x02, 0xD5, 0x25, 0x48, 0xB0, 0xF9,
+ 0x00, 0x00, 0x13, 0x07, 0x02, 0xD5, 0x24, 0x48, 0xB0, 0xF9, 0x00, 0x00, 0xB1, 0xF9, 0x54, 0x30,
+ 0x83, 0x42, 0x33, 0xDA, 0xD0, 0x05, 0x07, 0xD5, 0x20, 0x48, 0xB0, 0xF9, 0x00, 0x00, 0x83, 0x42,
+ 0x02, 0xDD, 0x81, 0xF8, 0x3F, 0x71, 0x29, 0xE0, 0x30, 0x46, 0xFF, 0xF7, 0x5B, 0xFF, 0x25, 0xE0,
+ 0x91, 0xF8, 0x40, 0x00, 0x10, 0xF0, 0x06, 0x0F, 0x20, 0xD0, 0x91, 0xF8, 0x42, 0x00, 0x40, 0x07,
+ 0x1C, 0xD4, 0x17, 0x48, 0xB0, 0xF9, 0x00, 0x20, 0x16, 0x48, 0x00, 0x78, 0x01, 0x28, 0x01, 0xD1,
+ 0xB8, 0xF9, 0x00, 0x20, 0x91, 0xF8, 0x3F, 0x01, 0x03, 0x28, 0x02, 0xD0, 0x02, 0x28, 0x07, 0xD0,
+ 0x0C, 0xE0, 0xB1, 0xF9, 0x54, 0x20, 0xB8, 0xF9, 0x00, 0x00, 0x82, 0x42, 0x04, 0xDB, 0x05, 0xE0,
+ 0xB1, 0xF9, 0x54, 0x00, 0x90, 0x42, 0x01, 0xDA, 0x81, 0xF8, 0x3E, 0x91, 0x64, 0x1C, 0x64, 0xB2,
+ 0x95, 0xF9, 0xB8, 0x03, 0xA0, 0x42, 0x97, 0xDA, 0xBD, 0xE8, 0xF0, 0x87, 0x6E, 0x06, 0x10, 0x00,
+ 0x68, 0x06, 0x10, 0x00, 0x7C, 0x06, 0x10, 0x00, 0x86, 0x06, 0x10, 0x00, 0x6C, 0x06, 0x10, 0x00,
+ 0x70, 0x06, 0x10, 0x00, 0x81, 0x06, 0x10, 0x00, 0x90, 0xF8, 0x40, 0x10, 0x09, 0x06, 0x14, 0xD5,
+ 0xB0, 0xF8, 0x42, 0x10, 0x0A, 0x07, 0x0C, 0xD4, 0xF8, 0x4A, 0x12, 0x68, 0x92, 0xF8, 0x40, 0x34,
+ 0x5B, 0x06, 0x06, 0xD5, 0x92, 0xF8, 0x80, 0x23, 0x52, 0x06, 0x02, 0xD4, 0x01, 0x22, 0x80, 0xF8,
+ 0x70, 0x20, 0x41, 0xF0, 0x08, 0x01, 0xA0, 0xF8, 0x42, 0x10, 0x70, 0x47, 0xB0, 0xF8, 0x40, 0x10,
+ 0xC9, 0x05, 0x14, 0xD5, 0xB0, 0xF8, 0x42, 0x10, 0x0A, 0x06, 0x0C, 0xD4, 0xEB, 0x4A, 0x12, 0x68,
+ 0x92, 0xF8, 0x20, 0x34, 0x9B, 0x07, 0x06, 0xD5, 0x92, 0xF8, 0x80, 0x23, 0x52, 0x06, 0x02, 0xD4,
+ 0x01, 0x22, 0x80, 0xF8, 0x70, 0x20, 0x41, 0xF0, 0x80, 0x01, 0xA0, 0xF8, 0x42, 0x10, 0x70, 0x47,
+ 0x2D, 0xE9, 0xF0, 0x47, 0x15, 0x46, 0x0C, 0x46, 0x82, 0x46, 0x01, 0xF1, 0x0C, 0x07, 0x4F, 0xF0,
+ 0x00, 0x08, 0x4F, 0xF0, 0xFF, 0x39, 0x02, 0xF1, 0x0C, 0x06, 0x53, 0xB3, 0x01, 0x20, 0x84, 0xF8,
+ 0x3E, 0x01, 0x95, 0xF8, 0x40, 0x00, 0x84, 0xF8, 0x3F, 0x01, 0x84, 0xF8, 0x4B, 0x91, 0x84, 0xF8,
+ 0x42, 0x81, 0x84, 0xF8, 0x43, 0x81, 0xA4, 0xF8, 0x42, 0x80, 0xC4, 0xF8, 0x2C, 0x80, 0x84, 0xF8,
+ 0x41, 0x81, 0x95, 0xF8, 0x42, 0x00, 0x84, 0xF8, 0x45, 0x01, 0x68, 0x8E, 0xA4, 0xF8, 0x56, 0x00,
+ 0x95, 0xF8, 0x41, 0x00, 0x84, 0xF8, 0x46, 0x01, 0xE8, 0x68, 0x20, 0x62, 0x03, 0x22, 0x31, 0x46,
+ 0x20, 0x46, 0x06, 0xF0, 0x53, 0xFF, 0x03, 0x22, 0x31, 0x46, 0x38, 0x46, 0x06, 0xF0, 0x4E, 0xFF,
+ 0x10, 0xE0, 0xE0, 0x6A, 0x03, 0x22, 0x40, 0x1C, 0xE0, 0x62, 0x31, 0x46, 0x20, 0x46, 0x06, 0xF0,
+ 0x39, 0xFF, 0x03, 0x22, 0x31, 0x46, 0x38, 0x46, 0x06, 0xF0, 0x34, 0xFF, 0x94, 0xF8, 0x44, 0x01,
+ 0x84, 0xF8, 0x45, 0x01, 0x95, 0xF8, 0x40, 0x00, 0x84, 0xF8, 0x3F, 0x01, 0x95, 0xF8, 0x42, 0x00,
+ 0x84, 0xF8, 0x44, 0x01, 0xE8, 0x8C, 0xA4, 0xF8, 0x44, 0x00, 0x28, 0x8D, 0xA4, 0xF8, 0x46, 0x00,
+ 0x68, 0x8D, 0xA4, 0xF8, 0x48, 0x00, 0xE8, 0x8D, 0xA4, 0xF8, 0x4A, 0x00, 0x28, 0x8E, 0xA4, 0xF8,
+ 0x4C, 0x00, 0x28, 0x8B, 0xA4, 0xF8, 0x4E, 0x00, 0xB5, 0xF9, 0x32, 0x00, 0xA4, 0xF8, 0x54, 0x00,
+ 0xA9, 0x69, 0x61, 0x63, 0xE9, 0x69, 0xA1, 0x63, 0x29, 0x6A, 0xE1, 0x63, 0xA9, 0x8E, 0xA4, 0xF8,
+ 0x58, 0x10, 0xA9, 0x8C, 0xA4, 0xF8, 0x40, 0x10, 0xD5, 0xF8, 0x36, 0x10, 0xC4, 0xF8, 0x5A, 0x10,
+ 0x69, 0x69, 0xE1, 0x61, 0x95, 0xF8, 0x43, 0x10, 0x84, 0xF8, 0x47, 0x11, 0x95, 0xF8, 0x44, 0x10,
+ 0x84, 0xF8, 0x48, 0x11, 0xB4, 0xF9, 0x56, 0x10, 0x81, 0x42, 0x01, 0xDA, 0xA4, 0xF8, 0x56, 0x00,
+ 0xC4, 0xF8, 0x4C, 0x81, 0x84, 0xF8, 0x49, 0x91, 0xA8, 0x8C, 0x80, 0x05, 0xB4, 0xF8, 0x42, 0x00,
+ 0x02, 0xD5, 0x40, 0xF4, 0x80, 0x70, 0x01, 0xE0, 0x20, 0xF4, 0x80, 0x70, 0xA4, 0xF8, 0x42, 0x00,
+ 0x21, 0x46, 0x50, 0x46, 0x00, 0xF0, 0x21, 0xFA, 0x20, 0x46, 0xFF, 0xF7, 0x2D, 0xFF, 0x20, 0x46,
+ 0xBD, 0xE8, 0xF0, 0x47, 0x42, 0xE7, 0xF0, 0xB5, 0xB1, 0xF9, 0x02, 0x60, 0xB2, 0xF9, 0x02, 0x50,
+ 0x78, 0x24, 0x06, 0xFB, 0x05, 0xF7, 0x00, 0x2F, 0x2A, 0xDD, 0xB1, 0xF9, 0x00, 0x70, 0xB2, 0xF9,
+ 0x00, 0xC0, 0x07, 0xFB, 0x0C, 0xF7, 0x00, 0x2F, 0x22, 0xDD, 0x76, 0x43, 0x5D, 0x43, 0x96, 0xFB,
+ 0xF5, 0xF5, 0x2D, 0xB2, 0x45, 0x80, 0xB1, 0xF9, 0x00, 0x10, 0xB2, 0xF9, 0x00, 0x20, 0x49, 0x43,
+ 0x5A, 0x43, 0x91, 0xFB, 0xF2, 0xF1, 0x0A, 0xB2, 0x02, 0x80, 0x78, 0x2D, 0x00, 0xDD, 0x44, 0x80,
+ 0xB0, 0xF9, 0x02, 0x30, 0x6F, 0xF0, 0x77, 0x01, 0x8B, 0x42, 0x00, 0xDA, 0x41, 0x80, 0x78, 0x2A,
+ 0x00, 0xDD, 0x04, 0x80, 0xB0, 0xF9, 0x00, 0x20, 0x8A, 0x42, 0x00, 0xDA, 0x01, 0x80, 0xF0, 0xBD,
+ 0x00, 0x21, 0x41, 0x80, 0xFA, 0xE7, 0x2D, 0xE9, 0xFF, 0x4F, 0xDF, 0xF8, 0xD4, 0x91, 0x83, 0x46,
+ 0x0F, 0x46, 0xD9, 0xF8, 0x04, 0x00, 0x71, 0x4D, 0x8F, 0xB0, 0x00, 0x21, 0x01, 0x60, 0x28, 0x68,
+ 0xB0, 0xF8, 0x82, 0x43, 0x0C, 0xF0, 0xF2, 0xF9, 0x01, 0x28, 0x04, 0xD1, 0x28, 0x68, 0xB0, 0xF8,
+ 0x84, 0x03, 0x00, 0xB1, 0x04, 0x46, 0x0C, 0xF0, 0xDF, 0xF9, 0x80, 0xB1, 0x28, 0x68, 0x69, 0x49,
+ 0xB0, 0xF8, 0x86, 0x43, 0xB0, 0xF8, 0x8A, 0x03, 0x09, 0x68, 0x88, 0x42, 0x07, 0xDA, 0x0C, 0xF0,
+ 0xDD, 0xF9, 0x01, 0x28, 0x03, 0xD1, 0x28, 0x68, 0xB0, 0xF8, 0x88, 0x03, 0x04, 0x44, 0x64, 0x43,
+ 0x02, 0x94, 0x28, 0x68, 0x00, 0x25, 0x0B, 0xF5, 0x80, 0x56, 0x90, 0xF8, 0x8C, 0x03, 0x4F, 0xF0,
+ 0x01, 0x0A, 0x05, 0x90, 0x65, 0xE0, 0x00, 0xBF, 0xD6, 0xF8, 0xB0, 0x13, 0x0A, 0xFA, 0x05, 0xF0,
+ 0x01, 0x42, 0x5C, 0xD0, 0xC5, 0xEB, 0xC5, 0x00, 0x00, 0xEB, 0x40, 0x01, 0x0B, 0xEB, 0x01, 0x14,
+ 0x94, 0xF8, 0x3E, 0x01, 0x07, 0x28, 0x52, 0xD0, 0x21, 0x46, 0x20, 0x1D, 0x06, 0xF0, 0x81, 0xFE,
+ 0x05, 0x99, 0x48, 0x43, 0x00, 0xFB, 0x01, 0xF8, 0x02, 0x99, 0x88, 0x45, 0x00, 0xDA, 0x88, 0x46,
+ 0x22, 0x46, 0x21, 0x1D, 0x08, 0xA8, 0x06, 0xF0, 0x6A, 0xFE, 0x22, 0x1D, 0x11, 0x1D, 0x0A, 0xA8,
+ 0x06, 0xF0, 0x65, 0xFE, 0xBD, 0xF9, 0x20, 0x00, 0x64, 0x30, 0xC8, 0x28, 0x04, 0xD8, 0xBD, 0xF9,
+ 0x22, 0x00, 0x64, 0x30, 0xC8, 0x28, 0x01, 0xD9, 0x01, 0x23, 0x00, 0xE0, 0x03, 0x23, 0x0A, 0xAA,
+ 0x08, 0xA9, 0x09, 0xA8, 0xFF, 0xF7, 0x57, 0xFF, 0x60, 0x88, 0xBD, 0xF8, 0x26, 0x10, 0x08, 0x44,
+ 0xAD, 0xF8, 0x2E, 0x00, 0x20, 0x88, 0xBD, 0xF8, 0x24, 0x10, 0x00, 0x24, 0x08, 0x44, 0xAD, 0xF8,
+ 0x2C, 0x00, 0x18, 0xE0, 0x04, 0xEB, 0xC4, 0x00, 0x07, 0xEB, 0xC0, 0x01, 0x0B, 0xA8, 0x0C, 0x31,
+ 0x06, 0xF0, 0x47, 0xFE, 0x40, 0x45, 0x0C, 0xDA, 0xD9, 0xF8, 0x04, 0x10, 0x0A, 0x68, 0x02, 0xEB,
+ 0x42, 0x02, 0x01, 0xEB, 0x42, 0x02, 0x50, 0x60, 0x54, 0x72, 0x15, 0x72, 0x08, 0x68, 0x40, 0x1C,
+ 0x08, 0x60, 0x64, 0x1C, 0xE4, 0xB2, 0x97, 0xF8, 0x38, 0x04, 0xA0, 0x42, 0xE2, 0xD8, 0x6D, 0x1C,
+ 0x6D, 0xB2, 0x96, 0xF9, 0xB8, 0x03, 0xA8, 0x42, 0x96, 0xDA, 0xD9, 0xF8, 0x04, 0x00, 0x4D, 0x46,
+ 0x01, 0x68, 0x00, 0x29, 0x7E, 0xD0, 0x00, 0xF0, 0x61, 0xF9, 0x00, 0x26, 0x4F, 0xF0, 0x04, 0x09,
+ 0x74, 0xE0, 0x00, 0xBF, 0x06, 0xEB, 0x46, 0x01, 0x00, 0xEB, 0x41, 0x00, 0x51, 0x46, 0x90, 0xF9,
+ 0x08, 0x40, 0x90, 0xF9, 0x09, 0xC0, 0x11, 0x98, 0x03, 0x68, 0x0A, 0xFA, 0x04, 0xF0, 0x03, 0x42,
+ 0x63, 0xD1, 0x12, 0x9A, 0xD2, 0xF8, 0x00, 0x80, 0x01, 0xFA, 0x0C, 0xF2, 0x18, 0xEA, 0x02, 0x0F,
+ 0x5B, 0xD1, 0x03, 0x43, 0x11, 0x98, 0x88, 0x46, 0x03, 0x60, 0x12, 0x98, 0x12, 0x99, 0x00, 0x23,
+ 0x00, 0x68, 0x10, 0x43, 0x08, 0x60, 0xC4, 0xEB, 0xC4, 0x00, 0x00, 0xEB, 0x40, 0x01, 0x0C, 0xEB,
+ 0xCC, 0x00, 0x0B, 0xEB, 0x01, 0x14, 0x07, 0xEB, 0xC0, 0x02, 0x21, 0x46, 0x58, 0x46, 0xFF, 0xF7,
+ 0x47, 0xFE, 0x94, 0xF8, 0x3E, 0x01, 0x03, 0x28, 0x04, 0xD0, 0x04, 0x28, 0x0C, 0xD0, 0x06, 0x28,
+ 0x10, 0xD0, 0x3A, 0xE0, 0x84, 0xF8, 0x41, 0x81, 0x09, 0xE0, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0xF8, 0x05, 0x10, 0x00, 0xF4, 0x06, 0x10, 0x00, 0x00, 0x20, 0x84, 0xF8, 0x41, 0x01, 0x84, 0xF8,
+ 0x3E, 0x91, 0x2A, 0xE0, 0x84, 0xF8, 0x3E, 0x91, 0x94, 0xF8, 0x40, 0x00, 0x10, 0xF0, 0x06, 0x0F,
+ 0x23, 0xD0, 0x94, 0xF8, 0x42, 0x00, 0x40, 0x07, 0x1F, 0xD4, 0x97, 0x49, 0x95, 0x48, 0x09, 0x78,
+ 0xB0, 0xF9, 0x00, 0x00, 0x01, 0x29, 0x02, 0xD1, 0x94, 0x48, 0xB0, 0xF9, 0x00, 0x00, 0x94, 0xF8,
+ 0x3F, 0x11, 0x03, 0x29, 0x02, 0xD0, 0x02, 0x29, 0x0B, 0xD0, 0x0E, 0xE0, 0x8F, 0x48, 0xB4, 0xF9,
+ 0x54, 0x10, 0xB0, 0xF9, 0x00, 0x00, 0x06, 0xE0, 0x0C, 0xE0, 0x06, 0x20, 0x84, 0xF8, 0x3E, 0x01,
+ 0x03, 0xE0, 0xB4, 0xF9, 0x54, 0x10, 0x81, 0x42, 0xF7, 0xDB, 0x76, 0x1C, 0x68, 0x68, 0x01, 0x68,
+ 0xB1, 0x42, 0x87, 0xDC, 0x13, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F, 0x85, 0x48, 0x00, 0x21, 0x01, 0x70,
+ 0x41, 0x60, 0x81, 0x60, 0x70, 0x47, 0x2D, 0xE9, 0xF0, 0x41, 0x05, 0x46, 0x41, 0xF2, 0xFC, 0x30,
+ 0x2E, 0x21, 0x28, 0x44, 0x0C, 0xF0, 0x0F, 0xF8, 0x00, 0x24, 0x05, 0xF5, 0x80, 0x56, 0x01, 0x27,
+ 0x4F, 0xF0, 0x05, 0x08, 0x1C, 0xE0, 0x00, 0xBF, 0xD6, 0xF8, 0xB0, 0x03, 0x07, 0xFA, 0x04, 0xF1,
+ 0x08, 0x42, 0x14, 0xD0, 0xC4, 0xEB, 0xC4, 0x00, 0x00, 0xEB, 0x40, 0x00, 0x05, 0xEB, 0x00, 0x11,
+ 0x91, 0xF8, 0x3E, 0x01, 0x03, 0x28, 0x04, 0xD0, 0x04, 0x28, 0x02, 0xD0, 0x02, 0x28, 0x03, 0xD0,
+ 0x05, 0xE0, 0x81, 0xF8, 0x3E, 0x81, 0x02, 0xE0, 0x28, 0x46, 0xFF, 0xF7, 0xB3, 0xFC, 0x64, 0x1C,
+ 0x96, 0xF9, 0xB8, 0x03, 0xA0, 0x42, 0xDF, 0xDA, 0xBD, 0xE8, 0xF0, 0x81, 0x2D, 0xE9, 0xFC, 0x47,
+ 0x04, 0x46, 0x00, 0x25, 0x66, 0x48, 0x0E, 0x46, 0x00, 0x95, 0x01, 0x21, 0x00, 0x1D, 0x01, 0x95,
+ 0x03, 0xF0, 0x76, 0xFF, 0x62, 0x48, 0x05, 0x21, 0x08, 0x30, 0x03, 0xF0, 0x71, 0xFF, 0x41, 0xF2,
+ 0xFC, 0x30, 0x2E, 0x21, 0x20, 0x44, 0x0B, 0xF0, 0xCE, 0xFF, 0x5D, 0x4F, 0xFF, 0x22, 0x2D, 0x21,
+ 0xB8, 0x68, 0x0B, 0xF0, 0x9E, 0xFF, 0x3D, 0x70, 0x04, 0xF5, 0x80, 0x57, 0x4F, 0xF0, 0x01, 0x08,
+ 0xD7, 0xF8, 0xB0, 0x03, 0x58, 0xB3, 0x6B, 0x46, 0x01, 0xAA, 0x31, 0x46, 0x20, 0x46, 0xFF, 0xF7,
+ 0x72, 0xFE, 0x4F, 0xF0, 0x05, 0x09, 0x1E, 0xE0, 0xD7, 0xF8, 0xB0, 0x13, 0x08, 0xFA, 0x05, 0xF0,
+ 0x01, 0x42, 0x17, 0xD0, 0x01, 0x99, 0x08, 0x42, 0x14, 0xD1, 0xC5, 0xEB, 0xC5, 0x00, 0x00, 0xEB,
+ 0x40, 0x00, 0x04, 0xEB, 0x00, 0x11, 0x91, 0xF8, 0x3E, 0x01, 0x03, 0x28, 0x04, 0xD0, 0x04, 0x28,
+ 0x02, 0xD0, 0x02, 0x28, 0x03, 0xD0, 0x05, 0xE0, 0x81, 0xF8, 0x3E, 0x91, 0x02, 0xE0, 0x20, 0x46,
+ 0xFF, 0xF7, 0x60, 0xFC, 0x6D, 0x1C, 0x97, 0xF9, 0xB8, 0x03, 0xA8, 0x42, 0xDC, 0xDA, 0x00, 0x25,
+ 0x47, 0x46, 0x1A, 0xE0, 0x00, 0x99, 0x07, 0xFA, 0x05, 0xF0, 0x08, 0x42, 0x14, 0xD1, 0x20, 0x46,
+ 0xFF, 0xF7, 0x32, 0xFC, 0x00, 0x28, 0x0F, 0xDB, 0xC0, 0xEB, 0xC0, 0x01, 0x01, 0xEB, 0x41, 0x01,
+ 0x04, 0xEB, 0x01, 0x11, 0x05, 0xEB, 0xC5, 0x02, 0x81, 0xF8, 0x4A, 0x01, 0x06, 0xEB, 0xC2, 0x02,
+ 0x01, 0x23, 0x20, 0x46, 0xFF, 0xF7, 0x54, 0xFD, 0x6D, 0x1C, 0x96, 0xF8, 0x38, 0x04, 0xA8, 0x42,
+ 0xE0, 0xD8, 0x2F, 0x48, 0x01, 0x21, 0x00, 0x1D, 0x03, 0xF0, 0x22, 0xFF, 0x2C, 0x48, 0x05, 0x21,
+ 0x08, 0x30, 0x03, 0xF0, 0x1D, 0xFF, 0xBD, 0xE8, 0xFC, 0x87, 0x30, 0xB5, 0x28, 0x4A, 0x91, 0xF8,
+ 0x44, 0x51, 0x94, 0x68, 0x63, 0x5D, 0xFF, 0x2B, 0x05, 0xD1, 0x13, 0x78, 0x63, 0x55, 0x0F, 0x2B,
+ 0x01, 0xD2, 0x5B, 0x1C, 0x13, 0x70, 0x91, 0xF8, 0x44, 0x21, 0xA2, 0x5C, 0x81, 0xF8, 0x44, 0x21,
+ 0x00, 0xEB, 0x42, 0x02, 0x02, 0xF5, 0x80, 0x52, 0x91, 0xF8, 0x4A, 0x51, 0xB2, 0xF8, 0xFC, 0x33,
+ 0x01, 0x24, 0xAC, 0x40, 0x23, 0x43, 0xA2, 0xF8, 0xFC, 0x33, 0x91, 0xF8, 0x44, 0x11, 0x41, 0xF2,
+ 0x1A, 0x42, 0x10, 0x44, 0x0A, 0x5C, 0x52, 0x1C, 0x0A, 0x54, 0x30, 0xBD, 0x7C, 0xB5, 0x00, 0x21,
+ 0x1C, 0xE0, 0x01, 0xEB, 0x41, 0x03, 0x4A, 0x1C, 0x00, 0xEB, 0x43, 0x03, 0x12, 0xE0, 0x02, 0xEB,
+ 0x42, 0x04, 0x00, 0xEB, 0x44, 0x04, 0x5E, 0x68, 0x65, 0x68, 0xAE, 0x42, 0x09, 0xDD, 0x00, 0x96,
+ 0x1E, 0x89, 0xAD, 0xF8, 0x04, 0x60, 0x5D, 0x60, 0x25, 0x89, 0x1D, 0x81, 0x00, 0x9D, 0x65, 0x60,
+ 0x26, 0x81, 0x52, 0x1C, 0x04, 0x68, 0x94, 0x42, 0xE9, 0xDC, 0x49, 0x1C, 0x02, 0x68, 0x8A, 0x42,
+ 0xDF, 0xDC, 0x7C, 0xBD, 0x70, 0x06, 0x10, 0x00, 0x81, 0x06, 0x10, 0x00, 0x6E, 0x06, 0x10, 0x00,
+ 0xF8, 0x05, 0x10, 0x00, 0x2D, 0xE9, 0xF8, 0x4F, 0x00, 0x22, 0x91, 0x46, 0x15, 0x46, 0x14, 0x46,
+ 0x00, 0x92, 0xFE, 0x4A, 0xFE, 0x4B, 0x12, 0x68, 0xB3, 0xF9, 0x00, 0xC0, 0xFD, 0x4B, 0x17, 0x8E,
+ 0x1E, 0x68, 0x67, 0x45, 0x60, 0xD1, 0x86, 0x46, 0xFB, 0x48, 0x02, 0xF1, 0x32, 0x0A, 0x8B, 0x46,
+ 0x90, 0xF8, 0x00, 0xC0, 0xF9, 0x48, 0x07, 0x78, 0x4D, 0xE0, 0x3A, 0x46, 0x43, 0xE0, 0x00, 0xBF,
+ 0x3A, 0xF8, 0x12, 0x10, 0x3E, 0xF8, 0x12, 0x00, 0x3B, 0xF8, 0x12, 0x30, 0x08, 0x1A, 0xC9, 0x1A,
+ 0x00, 0xB2, 0x0B, 0xB2, 0x00, 0x28, 0x01, 0xDB, 0x01, 0x46, 0x00, 0xE0, 0x41, 0x42, 0x49, 0x45,
+ 0x06, 0xDD, 0x00, 0x28, 0x01, 0xDB, 0x01, 0x46, 0x00, 0xE0, 0x41, 0x42, 0x0F, 0xFA, 0x81, 0xF9,
+ 0x19, 0x1A, 0x01, 0xD4, 0x88, 0x46, 0x01, 0xE0, 0xC1, 0xF1, 0x00, 0x08, 0xA8, 0x45, 0x05, 0xDD,
+ 0x00, 0x29, 0x01, 0xDB, 0x0D, 0x46, 0x00, 0xE0, 0x4D, 0x42, 0x2D, 0xB2, 0x00, 0x29, 0x00, 0xDA,
+ 0x49, 0x42, 0xB6, 0xF8, 0xE2, 0x83, 0x41, 0x45, 0x15, 0xDD, 0x00, 0x2B, 0x00, 0xDA, 0x5B, 0x42,
+ 0x01, 0x1E, 0x00, 0xDA, 0x41, 0x42, 0x8B, 0x42, 0x02, 0xDD, 0x64, 0x1C, 0xA4, 0xB2, 0x0A, 0xE0,
+ 0x00, 0x28, 0x00, 0xDA, 0x40, 0x42, 0xB6, 0xF8, 0xE4, 0x13, 0x88, 0x42, 0x03, 0xDD, 0x4F, 0xF0,
+ 0x00, 0x0C, 0x64, 0x46, 0x01, 0xE0, 0x52, 0x1E, 0xBA, 0xD1, 0x0A, 0xEB, 0x47, 0x0A, 0x0E, 0xEB,
+ 0x47, 0x0E, 0x0B, 0xEB, 0x47, 0x0B, 0xBC, 0xF1, 0x01, 0x0C, 0xAE, 0xD2, 0x96, 0xF8, 0xE6, 0x03,
+ 0xA0, 0x42, 0x01, 0xD2, 0x01, 0x20, 0x00, 0x90, 0x96, 0xF8, 0xE7, 0x13, 0xCC, 0x48, 0x00, 0x9A,
+ 0x06, 0xF0, 0xB1, 0xFB, 0xCB, 0x49, 0x21, 0xF8, 0x25, 0x5F, 0x21, 0xF8, 0x02, 0x9C, 0x8C, 0x70,
+ 0xBD, 0xE8, 0xF8, 0x8F, 0x10, 0xB5, 0x0C, 0x46, 0xC3, 0x49, 0x09, 0x78, 0x4A, 0x00, 0x01, 0x46,
+ 0xC5, 0x48, 0x0B, 0xF0, 0xE7, 0xFD, 0xC1, 0x48, 0x21, 0x46, 0x00, 0x78, 0x42, 0x00, 0xC3, 0x48,
+ 0x0B, 0xF0, 0xE0, 0xFD, 0xC2, 0x49, 0x01, 0x20, 0x08, 0x70, 0x10, 0xBD, 0xB8, 0x49, 0x10, 0xB5,
+ 0xB1, 0xF9, 0x00, 0x10, 0x4A, 0x00, 0x01, 0x46, 0xBE, 0x48, 0x00, 0x68, 0x0B, 0xF0, 0xD2, 0xFD,
+ 0xBD, 0x49, 0x01, 0x20, 0x08, 0x70, 0x10, 0xBD, 0x10, 0xB5, 0x01, 0x23, 0x0A, 0xE0, 0x00, 0xBF,
+ 0x30, 0xF9, 0x13, 0x40, 0x00, 0x2C, 0x00, 0xDA, 0x64, 0x42, 0x8C, 0x42, 0x01, 0xDD, 0x01, 0x20,
+ 0x10, 0xBD, 0x5B, 0x1C, 0x93, 0x42, 0xF3, 0xDB, 0x00, 0x20, 0x10, 0xBD, 0x2D, 0xE9, 0xFF, 0x5F,
+ 0xAB, 0x48, 0x00, 0x24, 0x08, 0x38, 0xDF, 0xF8, 0x9C, 0x82, 0x44, 0x60, 0x04, 0x70, 0xD8, 0xF8,
+ 0x00, 0x00, 0x9A, 0x46, 0x27, 0x46, 0x90, 0xF8, 0xAB, 0x02, 0x26, 0x46, 0x25, 0x46, 0x40, 0x07,
+ 0x03, 0xD5, 0xAA, 0x48, 0x00, 0x68, 0x00, 0xF0, 0xAB, 0xF9, 0xD8, 0xF8, 0x00, 0x00, 0xDF, 0xF8,
+ 0x80, 0x92, 0x90, 0xF8, 0xE0, 0x03, 0xC0, 0x07, 0x0E, 0xD0, 0x03, 0x20, 0x09, 0xF0, 0xAB, 0xFD,
+ 0xA0, 0x48, 0x01, 0x68, 0x00, 0x98, 0xFF, 0xF7, 0x25, 0xFF, 0x04, 0x46, 0x11, 0x26, 0x03, 0x20,
+ 0x09, 0xF0, 0xBC, 0xFD, 0x14, 0xB1, 0x74, 0xE0, 0x89, 0xF8, 0x00, 0x50, 0x9C, 0x48, 0x01, 0x78,
+ 0x11, 0xB9, 0x40, 0x78, 0x00, 0x28, 0x6C, 0xD0, 0xFA, 0xF7, 0x74, 0xFD, 0xD8, 0xF8, 0x00, 0x00,
+ 0xDF, 0xF8, 0x60, 0xB2, 0x90, 0xF8, 0xE0, 0x03, 0x80, 0x07, 0x14, 0xD5, 0x96, 0x48, 0x00, 0x78,
+ 0x88, 0xB9, 0x96, 0x48, 0x00, 0x78, 0x70, 0xB9, 0x0B, 0xF0, 0xC6, 0xFE, 0x58, 0xB9, 0x94, 0x4A,
+ 0x94, 0x49, 0xDB, 0xF8, 0x00, 0x00, 0x00, 0xF0, 0x35, 0xF9, 0x04, 0x00, 0x4F, 0xF0, 0x30, 0x06,
+ 0x4F, 0xF0, 0x01, 0x07, 0x0F, 0xD1, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0xE1, 0x03, 0xC0, 0x07,
+ 0x09, 0xD0, 0x8D, 0x4A, 0x8D, 0x49, 0xDB, 0xF8, 0x00, 0x00, 0x00, 0xF0, 0x7D, 0xF8, 0x30, 0x26,
+ 0x04, 0x46, 0x01, 0x27, 0x01, 0xE0, 0x89, 0xF8, 0x01, 0x50, 0x7C, 0xB9, 0xD8, 0xF8, 0x00, 0x00,
+ 0x90, 0xF8, 0xE1, 0x03, 0x80, 0x07, 0x09, 0xD5, 0x83, 0x4A, 0x84, 0x49, 0xDB, 0xF8, 0x00, 0x00,
+ 0x00, 0xF0, 0xA5, 0xF8, 0x31, 0x26, 0x04, 0x46, 0x01, 0x27, 0x01, 0xE0, 0x89, 0xF8, 0x02, 0x50,
+ 0x2C, 0xBB, 0xD8, 0xF8, 0x00, 0x10, 0x91, 0xF8, 0xE0, 0x03, 0x40, 0x07, 0x1F, 0xD5, 0xB1, 0xF9,
+ 0xFD, 0x23, 0x69, 0x49, 0x79, 0x48, 0x78, 0x4C, 0x09, 0x78, 0x07, 0xF0, 0xC8, 0xF8, 0x06, 0x46,
+ 0xD8, 0xF8, 0x00, 0x00, 0xB0, 0xF9, 0xFD, 0x23, 0x64, 0x48, 0x01, 0x78, 0x20, 0x46, 0x07, 0xF0,
+ 0xBE, 0xF8, 0x40, 0xEA, 0x06, 0x02, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0xFF, 0x13, 0x09, 0xF1,
+ 0x03, 0x00, 0x06, 0xF0, 0xD8, 0xFA, 0x21, 0x26, 0x04, 0x46, 0x01, 0x27, 0x01, 0xE0, 0x89, 0xF8,
+ 0x03, 0x50, 0x0B, 0xF0, 0x81, 0xFE, 0x01, 0x46, 0x69, 0x48, 0x01, 0x70, 0x4C, 0xB3, 0x6F, 0xB1,
+ 0x8A, 0xF8, 0x00, 0x50, 0x31, 0x46, 0x05, 0x20, 0x05, 0xF0, 0x84, 0xFD, 0x00, 0x98, 0xFF, 0xF7,
+ 0x2D, 0xFF, 0xDD, 0xE9, 0x01, 0x01, 0xFF, 0xF7, 0x15, 0xFF, 0x16, 0xE0, 0x01, 0x20, 0x8A, 0xF8,
+ 0x00, 0x00, 0x4F, 0x48, 0x08, 0x38, 0x06, 0x70, 0x41, 0x68, 0x41, 0xF0, 0x02, 0x01, 0x41, 0x60,
+ 0xD8, 0xF8, 0x00, 0x20, 0x92, 0xF8, 0x50, 0x32, 0xDB, 0x07, 0x03, 0xD1, 0x92, 0xF8, 0x80, 0x22,
+ 0xD2, 0x07, 0x02, 0xD0, 0x41, 0xF0, 0x08, 0x01, 0x41, 0x60, 0x01, 0x20, 0x04, 0xB0, 0xBD, 0xE8,
+ 0xF0, 0x9F, 0x8A, 0xF8, 0x00, 0x50, 0x00, 0x78, 0xF8, 0xE7, 0x41, 0x48, 0x00, 0x21, 0x08, 0x38,
+ 0x41, 0x60, 0x01, 0x70, 0x81, 0x60, 0x70, 0x47, 0x2D, 0xE9, 0xF7, 0x4F, 0xDF, 0xF8, 0xE4, 0x80,
+ 0x93, 0x46, 0x0B, 0x46, 0xD8, 0xF8, 0x00, 0x00, 0x00, 0x25, 0x2C, 0x46, 0x90, 0xF8, 0xEC, 0x73,
+ 0xB0, 0xF9, 0xE8, 0x63, 0xB0, 0xF9, 0xEA, 0xA3, 0x33, 0x48, 0x31, 0x46, 0x02, 0x78, 0x18, 0x46,
+ 0x00, 0xF0, 0x07, 0xF9, 0x81, 0x46, 0x31, 0x48, 0x31, 0x46, 0x02, 0x78, 0x58, 0x46, 0x00, 0xF0,
+ 0x00, 0xF9, 0x01, 0x46, 0xB9, 0xF1, 0x00, 0x0F, 0x00, 0xD0, 0x51, 0xB9, 0x28, 0x48, 0x51, 0x46,
+ 0xB0, 0xF9, 0x00, 0x20, 0x00, 0x98, 0x00, 0xF0, 0x0C, 0xF9, 0x04, 0x46, 0xB8, 0x42, 0x00, 0xD9,
+ 0x01, 0x25, 0xD8, 0xF8, 0x00, 0x00, 0x2A, 0x46, 0x90, 0xF8, 0xED, 0x13, 0x24, 0x48, 0x40, 0x1C,
+ 0x06, 0xF0, 0x61, 0xFA, 0x23, 0x49, 0x81, 0xF8, 0x31, 0x40, 0xBD, 0xE8, 0xFE, 0x8F, 0x2D, 0xE9,
+ 0xF0, 0x47, 0xDF, 0xF8, 0x70, 0x80, 0x84, 0x46, 0x0B, 0x46, 0xD8, 0xF8, 0x00, 0x00, 0x18, 0x49,
+ 0x00, 0x25, 0x90, 0xF8, 0xF2, 0x73, 0xB0, 0xF9, 0xF0, 0x63, 0x91, 0x46, 0x2C, 0x46, 0xB0, 0xF9,
+ 0xEE, 0x03, 0xB1, 0xF9, 0x00, 0x10, 0x06, 0xE0, 0x3C, 0xF9, 0x11, 0x20, 0x00, 0x2A, 0x00, 0xDA,
+ 0x52, 0x42, 0x82, 0x42, 0x13, 0xDC, 0x49, 0x1E, 0xF6, 0xD2, 0x0F, 0x48, 0x31, 0x46, 0x02, 0x78,
+ 0x18, 0x46, 0x00, 0xF0, 0xC9, 0xF8, 0x04, 0x46, 0x0C, 0x48, 0x31, 0x46, 0x02, 0x78, 0x48, 0x46,
+ 0x00, 0xF0, 0xC2, 0xF8, 0x20, 0x44, 0xC4, 0xB2, 0xBC, 0x42, 0x00, 0xD9, 0x01, 0x25, 0xD8, 0xF8,
+ 0x00, 0x00, 0x2A, 0x46, 0x90, 0xF8, 0xF3, 0x13, 0x05, 0x48, 0x2B, 0xE0, 0x24, 0x07, 0x10, 0x00,
+ 0xCA, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00,
+ 0x0C, 0x06, 0x10, 0x00, 0xE6, 0x43, 0x01, 0x20, 0x6A, 0x4A, 0x01, 0x20, 0x94, 0x4A, 0x01, 0x20,
+ 0x51, 0x07, 0x10, 0x00, 0xE8, 0x06, 0x10, 0x00, 0x50, 0x07, 0x10, 0x00, 0xE0, 0x06, 0x10, 0x00,
+ 0xEC, 0x06, 0x10, 0x00, 0xDC, 0x06, 0x10, 0x00, 0x28, 0x05, 0x10, 0x00, 0xC5, 0x06, 0x10, 0x00,
+ 0x32, 0x24, 0x10, 0x00, 0x08, 0x24, 0x10, 0x00, 0x60, 0x0B, 0x01, 0x20, 0x36, 0x0B, 0x01, 0x20,
+ 0x52, 0x06, 0x10, 0x00, 0x80, 0x1C, 0x06, 0xF0, 0xF6, 0xF9, 0x52, 0x49, 0x81, 0xF8, 0x32, 0x40,
+ 0xBD, 0xE8, 0xF0, 0x87, 0x2D, 0xE9, 0xF0, 0x5F, 0x4F, 0x4E, 0x83, 0x46, 0x00, 0x25, 0x30, 0x68,
+ 0x92, 0x46, 0x0B, 0x46, 0x90, 0xF8, 0xF5, 0x73, 0xB0, 0xF9, 0xF6, 0xC3, 0xB0, 0xF9, 0xFA, 0x93,
+ 0x4A, 0x48, 0x2C, 0x46, 0x00, 0x78, 0x18, 0xB1, 0x49, 0x48, 0x00, 0x78, 0x90, 0xB1, 0x1C, 0xE0,
+ 0x48, 0x48, 0x61, 0x46, 0x02, 0x78, 0x18, 0x46, 0xFF, 0xF7, 0x4E, 0xFE, 0x80, 0x46, 0x46, 0x48,
+ 0x61, 0x46, 0x02, 0x78, 0x50, 0x46, 0xFF, 0xF7, 0x47, 0xFE, 0x01, 0x46, 0xB8, 0xF1, 0x00, 0x0F,
+ 0x00, 0xD0, 0x51, 0xB9, 0x41, 0x48, 0x49, 0x46, 0xB0, 0xF9, 0x00, 0x20, 0x58, 0x46, 0x00, 0xF0,
+ 0x60, 0xF8, 0x04, 0x46, 0xB8, 0x42, 0x00, 0xD9, 0x01, 0x25, 0x30, 0x68, 0x2A, 0x46, 0x90, 0xF8,
+ 0xFC, 0x13, 0x3B, 0x48, 0x06, 0xF0, 0xB7, 0xF9, 0x32, 0x49, 0x81, 0xF8, 0x31, 0x40, 0x0E, 0xE7,
+ 0x2D, 0xE9, 0xF0, 0x41, 0x37, 0x4F, 0x39, 0x78, 0x00, 0x29, 0x30, 0xD0, 0x36, 0x49, 0x09, 0x78,
+ 0x00, 0x29, 0x2C, 0xD1, 0x2F, 0x49, 0x35, 0x4D, 0x0B, 0x78, 0x2F, 0x49, 0x6C, 0x0C, 0x0E, 0x78,
+ 0x0E, 0xE0, 0x31, 0x46, 0x08, 0xE0, 0x00, 0xBF, 0x30, 0xF9, 0x11, 0x20, 0xAA, 0x42, 0x00, 0xDD,
+ 0x15, 0x46, 0xA2, 0x42, 0x00, 0xDA, 0x14, 0x46, 0x49, 0x1E, 0xF5, 0xD2, 0x00, 0xEB, 0x46, 0x00,
+ 0x5B, 0x1E, 0xEE, 0xD2, 0x20, 0x48, 0x01, 0x68, 0xB1, 0xF9, 0xC7, 0x02, 0xB1, 0xF9, 0xC9, 0x12,
+ 0x85, 0x42, 0x08, 0xDA, 0x48, 0x42, 0x84, 0x42, 0x05, 0xDD, 0x00, 0x20, 0x38, 0x70, 0x01, 0x46,
+ 0x08, 0x20, 0x05, 0xF0, 0x3F, 0xFC, 0x17, 0x48, 0x20, 0xF8, 0x44, 0x5F, 0x44, 0x80, 0xBD, 0xE8,
+ 0xF0, 0x81, 0x05, 0xE0, 0x30, 0xF9, 0x12, 0x30, 0x8B, 0x42, 0x01, 0xDD, 0x01, 0x20, 0x70, 0x47,
+ 0x52, 0x1E, 0xF7, 0xD2, 0x00, 0x20, 0x70, 0x47, 0x10, 0xB5, 0x03, 0x46, 0x00, 0x20, 0x05, 0xE0,
+ 0x33, 0xF9, 0x12, 0x40, 0x8C, 0x42, 0x01, 0xDD, 0x40, 0x1C, 0xC0, 0xB2, 0x52, 0x1E, 0xF7, 0xD2,
+ 0x10, 0xBD, 0x10, 0xB5, 0x04, 0x46, 0x00, 0x20, 0x09, 0xE0, 0x00, 0xBF, 0x34, 0xF9, 0x12, 0x30,
+ 0x00, 0x2B, 0x00, 0xDA, 0x5B, 0x42, 0x8B, 0x42, 0x01, 0xDD, 0x40, 0x1C, 0xC0, 0xB2, 0x52, 0x1E,
+ 0xF4, 0xD2, 0x10, 0xBD, 0xE6, 0x43, 0x01, 0x20, 0x20, 0x07, 0x10, 0x00, 0x1C, 0x05, 0x10, 0x00,
+ 0x0C, 0x05, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0xCA, 0x06, 0x10, 0x00,
+ 0x0D, 0x06, 0x10, 0x00, 0x28, 0x05, 0x10, 0x00, 0xC6, 0x06, 0x10, 0x00, 0x00, 0x80, 0xFF, 0xFF,
+ 0x2D, 0xE9, 0xFF, 0x4F, 0x87, 0xB0, 0x00, 0x20, 0x81, 0x46, 0x03, 0x90, 0x02, 0x90, 0x01, 0x90,
+ 0x6C, 0x48, 0x88, 0x46, 0x93, 0x46, 0x00, 0x68, 0xB0, 0xF8, 0xBA, 0x02, 0x06, 0x90, 0x89, 0x78,
+ 0x50, 0x68, 0x0B, 0xF0, 0xFD, 0xFC, 0x00, 0x90, 0x98, 0xF8, 0x02, 0x50, 0x98, 0xF8, 0x03, 0x00,
+ 0x05, 0x90, 0x32, 0xE0, 0x07, 0x98, 0x98, 0xF8, 0x00, 0x40, 0x00, 0xEB, 0xC5, 0x00, 0x04, 0x90,
+ 0x98, 0xF8, 0x01, 0xA0, 0x1F, 0xE0, 0x04, 0x98, 0x22, 0x46, 0x00, 0x21, 0xD0, 0xE9, 0x00, 0x67,
+ 0x01, 0x20, 0x0B, 0xF0, 0x20, 0xFB, 0x06, 0x40, 0x0F, 0x40, 0x3E, 0x43, 0x12, 0xD0, 0x00, 0x98,
+ 0x06, 0x99, 0x30, 0xF9, 0x14, 0x00, 0x88, 0x42, 0x0C, 0xDD, 0x02, 0x99, 0x81, 0x44, 0x04, 0xFB,
+ 0x00, 0x11, 0x02, 0x91, 0x01, 0x99, 0x05, 0xFB, 0x00, 0x10, 0x01, 0x90, 0x03, 0x98, 0x40, 0x1C,
+ 0x80, 0xB2, 0x03, 0x90, 0x64, 0x1C, 0xA2, 0x45, 0xDD, 0xDA, 0x4F, 0x49, 0x00, 0x98, 0x6D, 0x1C,
+ 0x09, 0x78, 0x00, 0xEB, 0x41, 0x00, 0x00, 0x90, 0x05, 0x98, 0xA8, 0x42, 0xCA, 0xDA, 0x49, 0x48,
+ 0x00, 0x25, 0x04, 0x68, 0x01, 0x98, 0xA1, 0x8E, 0xC2, 0x17, 0xA1, 0xFB, 0x00, 0x36, 0x05, 0xFB,
+ 0x00, 0x60, 0x01, 0xFB, 0x02, 0x02, 0x4F, 0xF6, 0xFF, 0x76, 0xA3, 0xFB, 0x06, 0x07, 0x02, 0xFB,
+ 0x06, 0x72, 0x03, 0xFB, 0x05, 0x21, 0x94, 0xF8, 0x31, 0x30, 0x30, 0x34, 0x5B, 0x1E, 0x89, 0xFB,
+ 0x03, 0x23, 0x0B, 0xF0, 0xF3, 0xFA, 0x72, 0x10, 0x80, 0x18, 0xB0, 0x46, 0x41, 0xF1, 0x00, 0x01,
+ 0x42, 0x46, 0x00, 0x23, 0x0B, 0xF0, 0xEA, 0xFA, 0x07, 0x46, 0x02, 0x98, 0x61, 0x88, 0xC2, 0x17,
+ 0xA1, 0xFB, 0x00, 0x36, 0x05, 0xFB, 0x00, 0x60, 0x01, 0xFB, 0x02, 0x01, 0x42, 0x46, 0xA3, 0xFB,
+ 0x02, 0x06, 0x01, 0xFB, 0x02, 0x61, 0x03, 0xFB, 0x05, 0x11, 0x23, 0x78, 0x5B, 0x1E, 0x89, 0xFB,
+ 0x03, 0x23, 0x0B, 0xF0, 0xD3, 0xFA, 0x47, 0xF6, 0xFF, 0x72, 0x80, 0x18, 0x41, 0xF1, 0x00, 0x01,
+ 0x42, 0x46, 0x00, 0x23, 0x0B, 0xF0, 0xCA, 0xFA, 0xCB, 0xF8, 0x1C, 0x90, 0x03, 0x99, 0xAB, 0xF8,
+ 0x28, 0x10, 0x0A, 0x99, 0x48, 0x60, 0x0A, 0x99, 0x0F, 0x60, 0x0B, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F,
+ 0x10, 0xB5, 0x20, 0x49, 0x43, 0x68, 0x00, 0x22, 0x09, 0x68, 0x00, 0x2B, 0x01, 0xDA, 0x42, 0x60,
+ 0x03, 0xE0, 0x4C, 0x8E, 0xA3, 0x42, 0x00, 0xDD, 0x44, 0x60, 0x03, 0x68, 0x00, 0x2B, 0x01, 0xDA,
+ 0x02, 0x60, 0x10, 0xBD, 0x89, 0x8E, 0x8B, 0x42, 0xFB, 0xDD, 0x01, 0x60, 0x10, 0xBD, 0x7C, 0xB5,
+ 0x14, 0x46, 0x00, 0x25, 0x6B, 0x46, 0xFF, 0xF7, 0x43, 0xFF, 0x14, 0x48, 0x01, 0x78, 0x11, 0x48,
+ 0x29, 0xB1, 0x01, 0x68, 0x91, 0xF8, 0x07, 0x15, 0x09, 0x07, 0x00, 0xD5, 0x04, 0x25, 0x94, 0xF8,
+ 0x40, 0x10, 0x03, 0x29, 0x08, 0xD1, 0x00, 0x68, 0x90, 0xF8, 0x07, 0x05, 0xC0, 0x07, 0x03, 0xD0,
+ 0x29, 0x46, 0x68, 0x46, 0x01, 0xF0, 0xD2, 0xFC, 0x68, 0x46, 0xFF, 0xF7, 0xC9, 0xFF, 0x01, 0x98,
+ 0xE0, 0x81, 0x00, 0x98, 0xA0, 0x81, 0x94, 0xF8, 0x24, 0x00, 0xC0, 0x06, 0x01, 0xD4, 0xE0, 0x68,
+ 0x60, 0x61, 0x7C, 0xBD, 0x20, 0x07, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0x80, 0x06, 0x10, 0x00,
+ 0x2D, 0xE9, 0xF0, 0x47, 0x05, 0x46, 0x90, 0xF8, 0x02, 0x80, 0x0F, 0x46, 0x88, 0x68, 0x14, 0x46,
+ 0x01, 0x26, 0x41, 0x46, 0x0B, 0xF0, 0x1C, 0xFC, 0x81, 0x46, 0xA8, 0xF1, 0x01, 0x00, 0xC1, 0xB2,
+ 0xB8, 0x68, 0x0B, 0xF0, 0x15, 0xFC, 0x29, 0x78, 0x09, 0xEB, 0x41, 0x02, 0x32, 0xF9, 0x02, 0x2C,
+ 0xA2, 0x42, 0x03, 0xDB, 0x30, 0xF9, 0x11, 0x10, 0xA1, 0x42, 0x00, 0xDA, 0x00, 0x26, 0x69, 0x78,
+ 0x09, 0xEB, 0x41, 0x02, 0xB2, 0xF9, 0x02, 0x20, 0xA2, 0x42, 0x03, 0xDB, 0x30, 0xF9, 0x11, 0x00,
+ 0xA0, 0x42, 0x00, 0xDA, 0x00, 0x26, 0x95, 0xF8, 0x03, 0x80, 0xB8, 0x68, 0x41, 0x46, 0x0B, 0xF0,
+ 0xF7, 0xFB, 0x81, 0x46, 0x08, 0xF1, 0x01, 0x00, 0xC1, 0xB2, 0xB8, 0x68, 0x0B, 0xF0, 0xF0, 0xFB,
+ 0x29, 0x78, 0x09, 0xEB, 0x41, 0x02, 0x32, 0xF9, 0x02, 0x2C, 0xA2, 0x42, 0x03, 0xDB, 0x30, 0xF9,
+ 0x11, 0x10, 0xA1, 0x42, 0x00, 0xDA, 0x00, 0x26, 0x69, 0x78, 0x09, 0xEB, 0x41, 0x02, 0xB2, 0xF9,
+ 0x02, 0x20, 0xA2, 0x42, 0x03, 0xDB, 0x30, 0xF9, 0x11, 0x00, 0xA0, 0x42, 0x00, 0xDA, 0x00, 0x26,
+ 0x30, 0x46, 0xBD, 0xE8, 0xF0, 0x87, 0x2D, 0xE9, 0xFF, 0x4F, 0xFB, 0x48, 0xDF, 0xF8, 0xEC, 0x83,
+ 0x92, 0x46, 0x04, 0x68, 0x97, 0xB0, 0x99, 0x46, 0x01, 0x22, 0x51, 0x46, 0xD8, 0xF8, 0x00, 0x00,
+ 0x01, 0xF0, 0xC8, 0xF9, 0xF6, 0x48, 0x00, 0x68, 0xB0, 0xF9, 0x4D, 0x14, 0x11, 0x91, 0x90, 0xF8,
+ 0x4C, 0x04, 0x0A, 0x90, 0xF3, 0x48, 0x9A, 0xF8, 0x02, 0x70, 0x9A, 0xF8, 0x03, 0x60, 0x00, 0x78,
+ 0x9A, 0xF8, 0x00, 0xB0, 0x9A, 0xF8, 0x01, 0x50, 0x40, 0x1E, 0x87, 0x42, 0x01, 0xDA, 0x7F, 0x1C,
+ 0xFF, 0xB2, 0x0E, 0xB1, 0x76, 0x1E, 0xF6, 0xB2, 0xEB, 0x49, 0x09, 0x78, 0x49, 0x1E, 0x8B, 0x45,
+ 0x03, 0xDA, 0x0B, 0xF1, 0x01, 0x00, 0x00, 0xF0, 0xFF, 0x0B, 0x0D, 0xB1, 0x6D, 0x1E, 0xED, 0xB2,
+ 0xB8, 0x1E, 0xC1, 0xB2, 0xD9, 0xF8, 0x08, 0x00, 0x0B, 0xF0, 0x9A, 0xFB, 0x0B, 0x90, 0x78, 0x1E,
+ 0xC1, 0xB2, 0xD9, 0xF8, 0x08, 0x00, 0x0B, 0xF0, 0x93, 0xFB, 0x04, 0x90, 0x39, 0x46, 0xD9, 0xF8,
+ 0x08, 0x00, 0x0B, 0xF0, 0x8D, 0xFB, 0x01, 0x90, 0x78, 0x1C, 0xC1, 0xB2, 0xD9, 0xF8, 0x08, 0x00,
+ 0x0B, 0xF0, 0x86, 0xFB, 0x02, 0x90, 0xB8, 0x1C, 0xC1, 0xB2, 0xD9, 0xF8, 0x08, 0x00, 0x0B, 0xF0,
+ 0x7F, 0xFB, 0x03, 0x90, 0x39, 0x46, 0x17, 0x98, 0x0B, 0xF0, 0x74, 0xFB, 0x08, 0x90, 0xCF, 0x48,
+ 0x39, 0x46, 0x00, 0x68, 0x0B, 0xF0, 0x7C, 0xFB, 0x86, 0x46, 0x00, 0x20, 0x7A, 0xB2, 0xCF, 0x4F,
+ 0x0C, 0x90, 0xB7, 0xE0, 0x4F, 0xFA, 0x8B, 0xF0, 0x94, 0xE0, 0x00, 0x21, 0x0E, 0xF8, 0x00, 0x10,
+ 0x08, 0x99, 0x18, 0x9B, 0x09, 0x5C, 0x99, 0x42, 0x75, 0xD1, 0xDF, 0xF8, 0x14, 0xC3, 0x01, 0x9B,
+ 0xDC, 0xF8, 0x00, 0xC0, 0x33, 0xF9, 0x10, 0x30, 0xBC, 0xF8, 0x4F, 0xC4, 0x63, 0x45, 0x6A, 0xDA,
+ 0x00, 0x21, 0x14, 0x91, 0x13, 0x91, 0x11, 0x99, 0x19, 0x44, 0x04, 0x9B, 0x09, 0xB2, 0x33, 0xF9,
+ 0x10, 0x30, 0x10, 0x93, 0x8B, 0x42, 0x06, 0xDD, 0x02, 0x9B, 0x33, 0xF9, 0x10, 0x30, 0x8B, 0x42,
+ 0x01, 0xDD, 0x01, 0x23, 0x13, 0x93, 0x01, 0x9B, 0x03, 0xEB, 0x40, 0x03, 0x33, 0xF9, 0x02, 0x8C,
+ 0x88, 0x45, 0x07, 0xDD, 0xB3, 0xF9, 0x02, 0xC0, 0x8C, 0x45, 0x03, 0xDD, 0x4F, 0xF0, 0x01, 0x0C,
+ 0xCD, 0xF8, 0x50, 0xC0, 0x01, 0x2A, 0x0F, 0xDD, 0xDD, 0xF8, 0x2C, 0xC0, 0x3C, 0xF9, 0x10, 0xC0,
+ 0x8C, 0x45, 0x09, 0xDD, 0xDD, 0xF8, 0x08, 0xC0, 0x3C, 0xF9, 0x10, 0xC0, 0x8C, 0x45, 0x03, 0xDD,
+ 0x4F, 0xF0, 0x01, 0x0C, 0xCD, 0xF8, 0x4C, 0xC0, 0xDF, 0xF8, 0x98, 0xC2, 0x9C, 0xF8, 0x00, 0xC0,
+ 0xAC, 0xF1, 0x02, 0x0C, 0x62, 0x45, 0x0D, 0xDA, 0xDD, 0xF8, 0x40, 0xC0, 0x8C, 0x45, 0x09, 0xDD,
+ 0xDD, 0xF8, 0x0C, 0xC0, 0x3C, 0xF9, 0x10, 0xC0, 0x8C, 0x45, 0x03, 0xDD, 0x4F, 0xF0, 0x01, 0x0C,
+ 0xCD, 0xF8, 0x4C, 0xC0, 0x01, 0x28, 0x0B, 0xDD, 0x33, 0xF9, 0x04, 0xCC, 0x8C, 0x45, 0x07, 0xDD,
+ 0xB3, 0xF9, 0x02, 0xC0, 0x8C, 0x45, 0x03, 0xDD, 0x4F, 0xF0, 0x01, 0x0C, 0xCD, 0xF8, 0x50, 0xC0,
+ 0xDF, 0xF8, 0x54, 0xC2, 0x9C, 0xF8, 0x00, 0xC0, 0xAC, 0xF1, 0x02, 0x0C, 0x60, 0x45, 0x05, 0xDA,
+ 0x88, 0x45, 0x03, 0xDD, 0xB3, 0xF9, 0x04, 0x30, 0x8B, 0x42, 0x12, 0xDC, 0x14, 0x99, 0x81, 0xB9,
+ 0x13, 0x99, 0x0D, 0xE0, 0xFF, 0xE7, 0x99, 0xB9, 0x01, 0x99, 0xB7, 0xF9, 0x00, 0x30, 0x31, 0xF9,
+ 0x10, 0x10, 0x99, 0x42, 0x0C, 0xDA, 0x86, 0x49, 0x09, 0x68, 0x91, 0xF8, 0x41, 0x14, 0xC9, 0x07,
+ 0x31, 0xB1, 0x01, 0x21, 0x0E, 0xF8, 0x00, 0x10, 0x0C, 0x99, 0x49, 0x1C, 0x89, 0xB2, 0x0C, 0x91,
+ 0x40, 0x1C, 0x40, 0xB2, 0xA8, 0x42, 0x7F, 0xF7, 0x68, 0xAF, 0x7F, 0x48, 0x0B, 0x99, 0x0E, 0xF1,
+ 0x28, 0x0E, 0x00, 0x78, 0x01, 0xEB, 0x40, 0x01, 0x0B, 0x91, 0x04, 0x99, 0x01, 0xEB, 0x40, 0x01,
+ 0x04, 0x91, 0x01, 0x99, 0x01, 0xEB, 0x40, 0x01, 0x01, 0x91, 0x02, 0x99, 0x01, 0xEB, 0x40, 0x01,
+ 0x02, 0x91, 0x03, 0x99, 0x01, 0xEB, 0x40, 0x00, 0x03, 0x90, 0x08, 0x98, 0x28, 0x30, 0x52, 0x1C,
+ 0x52, 0xB2, 0x08, 0x90, 0xB2, 0x42, 0x7F, 0xF7, 0x45, 0xAF, 0x0A, 0x99, 0x0C, 0x98, 0x88, 0x42,
+ 0x2A, 0xD9, 0x00, 0x20, 0x84, 0xF8, 0x0E, 0x01, 0xDA, 0xF8, 0x00, 0x00, 0x20, 0x60, 0x67, 0x48,
+ 0x00, 0x23, 0x1A, 0x46, 0x21, 0x46, 0x00, 0x68, 0x01, 0xF0, 0xA6, 0xF9, 0x01, 0x25, 0x17, 0xE0,
+ 0x04, 0xEB, 0x85, 0x00, 0x06, 0x46, 0x51, 0x46, 0x00, 0xF0, 0x3F, 0xF9, 0x70, 0xB1, 0x04, 0xEB,
+ 0x45, 0x00, 0xB0, 0xF8, 0xB4, 0x10, 0x0A, 0x98, 0x81, 0x42, 0x07, 0xD9, 0x30, 0x46, 0xB7, 0xF9,
+ 0x00, 0x20, 0x49, 0x46, 0xFF, 0xF7, 0x64, 0xFE, 0x01, 0x28, 0x06, 0xD0, 0x6D, 0x1C, 0xED, 0xB2,
+ 0x94, 0xF8, 0x0E, 0x01, 0xA8, 0x42, 0xE3, 0xD2, 0x00, 0x20, 0x1B, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F,
+ 0x2D, 0xE9, 0xF8, 0x4F, 0x04, 0x46, 0x50, 0x48, 0xDF, 0xF8, 0x40, 0x81, 0x91, 0x46, 0x8B, 0x46,
+ 0x05, 0x68, 0x01, 0x22, 0x21, 0x46, 0xD8, 0xF8, 0x00, 0x00, 0x01, 0xF0, 0x73, 0xF8, 0x00, 0x27,
+ 0xA6, 0x78, 0x4F, 0xF0, 0x01, 0x0A, 0x1D, 0xE0, 0x31, 0x46, 0xD8, 0xF8, 0x00, 0x00, 0x0B, 0xF0,
+ 0x6F, 0xFA, 0x00, 0x90, 0x31, 0x46, 0xDB, 0xF8, 0x08, 0x00, 0x0B, 0xF0, 0x61, 0xFA, 0x01, 0x46,
+ 0x20, 0x78, 0x52, 0x46, 0x09, 0xE0, 0x00, 0xBF, 0x31, 0xF9, 0x10, 0xC0, 0xCC, 0x45, 0x02, 0xDA,
+ 0x00, 0x9B, 0x01, 0x27, 0x1A, 0x54, 0x40, 0x1C, 0xC0, 0xB2, 0x63, 0x78, 0x83, 0x42, 0xF3, 0xD2,
+ 0x76, 0x1C, 0xF6, 0xB2, 0xE0, 0x78, 0xB0, 0x42, 0xDE, 0xD2, 0x01, 0x2F, 0x2E, 0xD1, 0x00, 0x20,
+ 0x85, 0xF8, 0x0E, 0x01, 0x20, 0x68, 0x28, 0x60, 0x00, 0x23, 0x1A, 0x46, 0x29, 0x46, 0xD8, 0xF8,
+ 0x00, 0x00, 0x01, 0xF0, 0x41, 0xF9, 0x01, 0x26, 0xDF, 0xF8, 0xC4, 0x80, 0x1A, 0xE0, 0x00, 0xBF,
+ 0x05, 0xEB, 0x86, 0x07, 0x21, 0x46, 0x38, 0x46, 0x00, 0xF0, 0xD7, 0xF8, 0x80, 0xB1, 0x05, 0xEB,
+ 0x46, 0x00, 0xB0, 0xF8, 0xB4, 0x10, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x4C, 0x04, 0x81, 0x42,
+ 0x06, 0xD9, 0x38, 0x46, 0x4A, 0x46, 0x59, 0x46, 0xFF, 0xF7, 0xFA, 0xFD, 0x01, 0x28, 0x06, 0xD0,
+ 0x76, 0x1C, 0xF6, 0xB2, 0x95, 0xF8, 0x0E, 0x01, 0xB0, 0x42, 0xE1, 0xD2, 0x00, 0x20, 0xBD, 0xE8,
+ 0xF8, 0x8F, 0x2D, 0xE9, 0xF0, 0x41, 0x1E, 0x4E, 0x0F, 0x46, 0x80, 0x46, 0x31, 0x68, 0x14, 0x46,
+ 0x1F, 0x48, 0x1D, 0x46, 0x9A, 0x69, 0xB1, 0xF8, 0x49, 0x34, 0x00, 0x78, 0x9A, 0x42, 0x10, 0xDC,
+ 0xEA, 0x8C, 0x04, 0x2A, 0x0D, 0xD9, 0x62, 0x78, 0x23, 0x78, 0x91, 0xF8, 0x4B, 0x14, 0xD2, 0x1A,
+ 0x52, 0x1C, 0x8A, 0x42, 0x05, 0xDC, 0xE2, 0x78, 0xA3, 0x78, 0xD2, 0x1A, 0x52, 0x1C, 0x8A, 0x42,
+ 0x02, 0xDD, 0x00, 0x20, 0xBD, 0xE8, 0xF0, 0x81, 0x00, 0x28, 0xFB, 0xD1, 0x0F, 0x48, 0x29, 0x46,
+ 0xB0, 0xF9, 0x00, 0x20, 0x20, 0x46, 0xFF, 0xF7, 0x6B, 0xFF, 0x00, 0x28, 0xF2, 0xD1, 0x31, 0x68,
+ 0x91, 0xF8, 0x40, 0x14, 0x89, 0x07, 0xED, 0xD5, 0x2B, 0x46, 0x22, 0x46, 0x39, 0x46, 0x40, 0x46,
+ 0xBD, 0xE8, 0xF0, 0x41, 0x07, 0xE6, 0x00, 0x00, 0x48, 0x06, 0x10, 0x00, 0x30, 0x06, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0x84, 0x06, 0x10, 0x00,
+ 0x10, 0x06, 0x10, 0x00, 0x2D, 0xE9, 0xFF, 0x4F, 0x81, 0xB0, 0xDF, 0xF8, 0x08, 0xB1, 0xDD, 0xF8,
+ 0x38, 0x90, 0x5F, 0xEA, 0x03, 0x08, 0x15, 0x46, 0x1D, 0xD4, 0x2C, 0x78, 0x0B, 0xEB, 0xC8, 0x0A,
+ 0x16, 0xE0, 0x00, 0xBF, 0x08, 0xF0, 0xFF, 0x01, 0x01, 0x98, 0x0B, 0xF0, 0xC1, 0xF9, 0x00, 0x90,
+ 0xDA, 0xE9, 0x00, 0x67, 0x22, 0x46, 0x01, 0x20, 0x00, 0x21, 0x0A, 0xF0, 0xEC, 0xFF, 0x06, 0x40,
+ 0x0F, 0x40, 0x3E, 0x43, 0x02, 0xD1, 0x00, 0x98, 0x00, 0x21, 0x01, 0x55, 0x64, 0x1C, 0xE4, 0xB2,
+ 0x68, 0x78, 0xA0, 0x42, 0xE6, 0xD2, 0xB9, 0xF1, 0x00, 0x00, 0x1E, 0xDB, 0x02, 0x46, 0xAC, 0x78,
+ 0x01, 0x20, 0x00, 0x21, 0x0A, 0xF0, 0xD7, 0xFF, 0x06, 0x46, 0x0F, 0x46, 0x4F, 0xF0, 0x00, 0x08,
+ 0x10, 0xE0, 0x21, 0x46, 0x01, 0x98, 0x0B, 0xF0, 0x9B, 0xF9, 0x0B, 0xEB, 0xC4, 0x01, 0x02, 0x46,
+ 0xD1, 0xE9, 0x00, 0x01, 0x30, 0x40, 0x39, 0x40, 0x08, 0x43, 0x01, 0xD1, 0x02, 0xF8, 0x09, 0x80,
+ 0x64, 0x1C, 0xE4, 0xB2, 0xE8, 0x78, 0xA0, 0x42, 0xEB, 0xD2, 0x05, 0xB0, 0xFE, 0xE6, 0x70, 0xB5,
+ 0x1E, 0x4B, 0x1D, 0x4A, 0x8C, 0x69, 0x1B, 0x68, 0x12, 0x78, 0xB3, 0xF8, 0x49, 0x54, 0xAC, 0x42,
+ 0x10, 0xDC, 0xCC, 0x8C, 0x04, 0x2C, 0x0D, 0xD9, 0x44, 0x78, 0x05, 0x78, 0x93, 0xF8, 0x4B, 0x34,
+ 0x64, 0x1B, 0x64, 0x1C, 0x9C, 0x42, 0x05, 0xDC, 0xC4, 0x78, 0x85, 0x78, 0x64, 0x1B, 0x64, 0x1C,
+ 0x9C, 0x42, 0x01, 0xDD, 0x00, 0x22, 0x06, 0xE0, 0x2A, 0xB9, 0x11, 0x4A, 0xB2, 0xF9, 0x00, 0x20,
+ 0xFF, 0xF7, 0xDE, 0xFE, 0x02, 0x46, 0x10, 0x46, 0x70, 0xBD, 0x02, 0x78, 0x0B, 0x78, 0x9A, 0x42,
+ 0x0B, 0xD9, 0x42, 0x78, 0x4B, 0x78, 0x9A, 0x42, 0x07, 0xD2, 0x82, 0x78, 0x8B, 0x78, 0x9A, 0x42,
+ 0x03, 0xD9, 0xC0, 0x78, 0xC9, 0x78, 0x88, 0x42, 0x01, 0xD3, 0x00, 0x20, 0x70, 0x47, 0x01, 0x20,
+ 0x70, 0x47, 0x00, 0x00, 0x48, 0x3F, 0x10, 0x00, 0x10, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x6E, 0x06, 0x10, 0x00, 0x70, 0xB5, 0xFE, 0x4C, 0x00, 0xEB, 0x43, 0x05, 0x04, 0x9E, 0x00, 0xEB,
+ 0x83, 0x03, 0x86, 0xB1, 0x8E, 0x69, 0xA6, 0x60, 0xCE, 0x8C, 0x66, 0x80, 0x09, 0x68, 0x61, 0x61,
+ 0xB5, 0xF8, 0xB4, 0x10, 0xA1, 0x80, 0x19, 0x68, 0xE1, 0x60, 0x90, 0xF8, 0x0E, 0x01, 0x20, 0x70,
+ 0x10, 0x68, 0x20, 0x61, 0x70, 0xBD, 0xA6, 0x68, 0x8E, 0x61, 0x66, 0x88, 0xCE, 0x84, 0x66, 0x69,
+ 0x0E, 0x60, 0xA1, 0x88, 0xA5, 0xF8, 0xB4, 0x10, 0xE1, 0x68, 0x19, 0x60, 0x21, 0x78, 0x80, 0xF8,
+ 0x0E, 0x11, 0x20, 0x69, 0x10, 0x60, 0x70, 0xBD, 0x2D, 0xE9, 0xF0, 0x41, 0x80, 0x46, 0xE9, 0x48,
+ 0x0F, 0x46, 0xE9, 0x4D, 0x01, 0x78, 0x1C, 0x46, 0x16, 0x46, 0x09, 0xB9, 0x40, 0x78, 0xD8, 0xB1,
+ 0x28, 0x68, 0x90, 0xF8, 0x30, 0x02, 0x00, 0x07, 0x05, 0xD5, 0x39, 0x46, 0x40, 0x46, 0xF9, 0xF7,
+ 0xAC, 0xFA, 0x00, 0x28, 0x41, 0xD0, 0x28, 0x68, 0x90, 0xF8, 0x30, 0x02, 0x00, 0x06, 0x0B, 0xD5,
+ 0xDE, 0x48, 0x00, 0x78, 0x01, 0x28, 0x07, 0xD0, 0xDD, 0x48, 0x00, 0x78, 0x20, 0xB1, 0x30, 0x46,
+ 0xF9, 0xF7, 0xF5, 0xFA, 0x00, 0x28, 0x30, 0xD0, 0xDA, 0x4B, 0x32, 0x46, 0x39, 0x46, 0x40, 0x46,
+ 0x00, 0xF0, 0x9F, 0xFE, 0x00, 0x28, 0x28, 0xD0, 0x94, 0xF8, 0x40, 0x00, 0x03, 0x28, 0x08, 0xD1,
+ 0x29, 0x68, 0xE0, 0x8C, 0x91, 0xF8, 0x07, 0x13, 0x88, 0x42, 0x02, 0xD9, 0x04, 0x20, 0x84, 0xF8,
+ 0x40, 0x00, 0x30, 0x68, 0xC4, 0xF8, 0x36, 0x00, 0x04, 0xF1, 0x44, 0x02, 0x51, 0x1E, 0x30, 0x46,
+ 0xF8, 0xF7, 0x03, 0xFE, 0x22, 0x46, 0x31, 0x46, 0xCA, 0x48, 0x00, 0xF0, 0x9C, 0xFD, 0x31, 0x46,
+ 0xC8, 0x48, 0x00, 0xF0, 0x17, 0xFE, 0x22, 0x46, 0x31, 0x46, 0xC6, 0x48, 0xFF, 0xF7, 0x67, 0xFC,
+ 0x20, 0x46, 0xBD, 0xE8, 0xF0, 0x41, 0xFE, 0xF7, 0x63, 0xBC, 0xBD, 0xE8, 0xF0, 0x81, 0x2D, 0xE9,
+ 0xF0, 0x41, 0x0D, 0x46, 0x1F, 0x46, 0x16, 0x46, 0x04, 0x46, 0x48, 0x21, 0x0A, 0xF0, 0xC3, 0xFF,
+ 0x00, 0x20, 0x84, 0xF8, 0x40, 0x00, 0xBC, 0x48, 0x00, 0x68, 0x60, 0x60, 0xA0, 0x60, 0x06, 0xEB,
+ 0x45, 0x00, 0x84, 0xF8, 0x42, 0x50, 0x30, 0xF8, 0xB4, 0x1F, 0x61, 0x85, 0x00, 0x88, 0xE0, 0x84,
+ 0x06, 0xEB, 0x85, 0x00, 0x00, 0x68, 0xC4, 0xF8, 0x3A, 0x00, 0xAD, 0x48, 0x81, 0x69, 0x21, 0x60,
+ 0x01, 0x21, 0xA9, 0x40, 0x39, 0x42, 0xD8, 0xD0, 0x40, 0x6A, 0x20, 0x60, 0xD5, 0xE7, 0x2D, 0xE9,
+ 0xF0, 0x47, 0x89, 0x46, 0x1C, 0x46, 0x16, 0x46, 0xAC, 0x4D, 0x03, 0xF1, 0x3E, 0x08, 0x91, 0x78,
+ 0x0B, 0xF0, 0x80, 0xF8, 0x07, 0x46, 0xB1, 0x78, 0x60, 0x68, 0x0B, 0xF0, 0x81, 0xF8, 0x02, 0x46,
+ 0x00, 0x20, 0xA0, 0x61, 0xB1, 0x78, 0x1F, 0xE0, 0x30, 0x78, 0x13, 0xE0, 0x17, 0xF8, 0x00, 0xC0,
+ 0xCC, 0x45, 0x0D, 0xD1, 0x32, 0xF9, 0x10, 0x30, 0xD4, 0xF8, 0x18, 0xC0, 0xAB, 0x42, 0x9C, 0x44,
+ 0xC4, 0xF8, 0x18, 0xC0, 0x04, 0xDD, 0x88, 0xF8, 0x00, 0x00, 0x1D, 0x46, 0x88, 0xF8, 0x01, 0x10,
+ 0x40, 0x1C, 0xC0, 0xB2, 0x73, 0x78, 0x83, 0x42, 0xE8, 0xD2, 0x99, 0x48, 0x49, 0x1C, 0xC9, 0xB2,
+ 0x00, 0x78, 0x28, 0x37, 0x02, 0xEB, 0x40, 0x02, 0xF0, 0x78, 0x88, 0x42, 0xDC, 0xD2, 0x65, 0x86,
+ 0xBD, 0xE8, 0xF0, 0x87, 0x2D, 0xE9, 0xFF, 0x4F, 0x81, 0xB0, 0x0C, 0x46, 0x1E, 0x46, 0x99, 0x78,
+ 0xDD, 0xF8, 0x40, 0x80, 0x0E, 0x9D, 0x0B, 0xF0, 0x53, 0xF8, 0x00, 0x90, 0xB1, 0x78, 0x20, 0x46,
+ 0x0B, 0xF0, 0x40, 0xF8, 0x83, 0x46, 0xB1, 0x78, 0xD8, 0xF8, 0x04, 0x00, 0x0B, 0xF0, 0x40, 0xF8,
+ 0x81, 0x46, 0x88, 0x48, 0x4F, 0xF0, 0x00, 0x0A, 0xC8, 0xF8, 0x18, 0xA0, 0x00, 0x68, 0x28, 0x60,
+ 0xB7, 0x78, 0x41, 0xE0, 0x34, 0x78, 0x31, 0xE0, 0x1B, 0xF8, 0x04, 0x10, 0x03, 0x98, 0x81, 0x42,
+ 0x2A, 0xD1, 0x39, 0xF9, 0x14, 0x10, 0x0F, 0x98, 0x81, 0x42, 0x22, 0xDD, 0x00, 0x99, 0x03, 0x98,
+ 0x08, 0x55, 0x29, 0x78, 0x20, 0x46, 0x05, 0xF0, 0xC5, 0xFB, 0x28, 0x70, 0x69, 0x78, 0x20, 0x46,
+ 0x05, 0xF0, 0xC4, 0xFB, 0x68, 0x70, 0xA9, 0x78, 0x38, 0x46, 0x05, 0xF0, 0xBB, 0xFB, 0xA8, 0x70,
+ 0xE9, 0x78, 0x38, 0x46, 0x05, 0xF0, 0xBA, 0xFB, 0xE8, 0x70, 0x0A, 0xF1, 0x01, 0x00, 0x1F, 0xFA,
+ 0x80, 0xFA, 0x39, 0xF9, 0x14, 0x00, 0xD8, 0xF8, 0x18, 0x10, 0x01, 0x44, 0xC8, 0xF8, 0x18, 0x10,
+ 0x02, 0xE0, 0x00, 0x99, 0x00, 0x20, 0x08, 0x55, 0x64, 0x1C, 0xE4, 0xB2, 0x70, 0x78, 0xA0, 0x42,
+ 0xCA, 0xD2, 0x00, 0x98, 0x66, 0x49, 0x28, 0x30, 0x00, 0x90, 0x09, 0x78, 0x7F, 0x1C, 0x0B, 0xF1,
+ 0x28, 0x0B, 0x09, 0xEB, 0x41, 0x09, 0xFF, 0xB2, 0xF0, 0x78, 0xB8, 0x42, 0xBA, 0xD2, 0xA8, 0xF8,
+ 0x26, 0xA0, 0x05, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F, 0x2D, 0xE9, 0xFF, 0x41, 0x17, 0x46, 0x0A, 0xAA,
+ 0x80, 0x46, 0x92, 0xE8, 0x31, 0x00, 0x5C, 0x4A, 0x1E, 0x46, 0xB2, 0xF9, 0x00, 0x20, 0x82, 0x42,
+ 0x02, 0xDB, 0x00, 0x20, 0x04, 0xB0, 0x20, 0xE7, 0x03, 0xAA, 0xCD, 0xE9, 0x00, 0x20, 0x02, 0x94,
+ 0x3A, 0x46, 0x40, 0x46, 0xFF, 0xF7, 0x7E, 0xFF, 0x01, 0x21, 0x28, 0x68, 0xB9, 0x40, 0x08, 0x43,
+ 0x28, 0x60, 0xC4, 0xF8, 0x00, 0x80, 0xE0, 0x8C, 0x10, 0xB1, 0x03, 0x98, 0x30, 0x60, 0xE8, 0xE7,
+ 0x01, 0x20, 0xE7, 0xE7, 0x2D, 0xE9, 0xFF, 0x4F, 0x81, 0xB0, 0x90, 0x46, 0x0E, 0x9C, 0x8B, 0x46,
+ 0x04, 0xEB, 0x88, 0x00, 0x94, 0xF8, 0x0E, 0x71, 0xDD, 0xF8, 0x40, 0xA0, 0x00, 0x68, 0x1E, 0x46,
+ 0x20, 0x60, 0x01, 0x23, 0x21, 0x46, 0x58, 0x46, 0x00, 0xF0, 0xB6, 0xFE, 0x01, 0x98, 0x58, 0x45,
+ 0x1F, 0xD0, 0xB5, 0x78, 0x1A, 0xE0, 0x29, 0x46, 0x01, 0x98, 0x0A, 0xF0, 0xB1, 0xFF, 0x81, 0x46,
+ 0x29, 0x46, 0x58, 0x46, 0x0A, 0xF0, 0xAC, 0xFF, 0x01, 0x46, 0x30, 0x78, 0x09, 0xE0, 0x00, 0xBF,
+ 0x19, 0xF8, 0x00, 0x30, 0x43, 0x45, 0x02, 0xD1, 0x0B, 0x5C, 0x09, 0xF8, 0x00, 0x30, 0x40, 0x1C,
+ 0xC0, 0xB2, 0x72, 0x78, 0x82, 0x42, 0xF3, 0xD2, 0x6D, 0x1C, 0xED, 0xB2, 0xF0, 0x78, 0xA8, 0x42,
+ 0xE1, 0xD2, 0x94, 0xF8, 0x0E, 0x01, 0x00, 0x25, 0x01, 0x21, 0xB8, 0x42, 0x12, 0xD0, 0x11, 0x98,
+ 0x78, 0xB1, 0x7F, 0x1C, 0xF8, 0xB2, 0x08, 0xE0, 0xDA, 0xF8, 0x00, 0x20, 0x01, 0xFA, 0x00, 0xF3,
+ 0x1A, 0x43, 0x40, 0x1C, 0xC0, 0xB2, 0xCA, 0xF8, 0x00, 0x20, 0x94, 0xF8, 0x0E, 0x21, 0x82, 0x42,
+ 0xF2, 0xD2, 0x01, 0x25, 0x00, 0x91, 0x43, 0x46, 0x52, 0x46, 0x20, 0x46, 0x0F, 0x99, 0xFF, 0xF7,
+ 0x31, 0xFE, 0x28, 0x46, 0x7D, 0xE7, 0x10, 0xB5, 0x17, 0x4A, 0x20, 0x4C, 0x13, 0x68, 0x64, 0x78,
+ 0x93, 0xF8, 0x06, 0x23, 0x4C, 0xB1, 0x91, 0xF8, 0x24, 0x10, 0x09, 0x07, 0x02, 0xD5, 0x93, 0xF8,
+ 0x09, 0x13, 0x01, 0xE0, 0x93, 0xF8, 0x08, 0x13, 0x0A, 0x44, 0x90, 0x42, 0x01, 0xD9, 0x01, 0x20,
+ 0x10, 0xBD, 0x00, 0x20, 0x10, 0xBD, 0x2D, 0xE9, 0xF0, 0x47, 0x1C, 0x46, 0x90, 0x46, 0x89, 0x46,
+ 0x82, 0x46, 0x01, 0xF0, 0x41, 0xF8, 0x08, 0x4F, 0xE1, 0x8C, 0x38, 0x68, 0x90, 0xF8, 0x0E, 0x23,
+ 0x91, 0x42, 0x7C, 0xD8, 0x03, 0x21, 0x84, 0xF8, 0x40, 0x10, 0x00, 0x25, 0x18, 0xE0, 0x00, 0x00,
+ 0x14, 0x06, 0x10, 0x00, 0xEC, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0x0E, 0x05, 0x10, 0x00,
+ 0xC0, 0x06, 0x10, 0x00, 0xA0, 0x3E, 0x10, 0x00, 0xDC, 0x06, 0x10, 0x00, 0x00, 0x80, 0xFF, 0xFF,
+ 0xBA, 0x06, 0x10, 0x00, 0x52, 0x6C, 0x01, 0x00, 0x76, 0x06, 0x10, 0x00, 0x4C, 0x07, 0x10, 0x00,
+ 0x90, 0xF8, 0xE0, 0x12, 0x8A, 0x07, 0x01, 0xD4, 0x49, 0x07, 0x00, 0xD5, 0x01, 0x25, 0xFF, 0x4A,
+ 0xB0, 0xF8, 0x0A, 0x13, 0xFE, 0x4E, 0x52, 0x78, 0x1A, 0xB1, 0xB2, 0x79, 0x0A, 0xB9, 0xB0, 0xF8,
+ 0x0C, 0x13, 0xA2, 0x8D, 0x8A, 0x42, 0x02, 0xD2, 0xFA, 0x49, 0x09, 0x78, 0x21, 0xB1, 0x61, 0x8D,
+ 0x03, 0x29, 0x06, 0xD2, 0x00, 0x25, 0x29, 0xE0, 0x61, 0x8D, 0x00, 0x25, 0x03, 0x29, 0xF9, 0xD3,
+ 0x24, 0xE0, 0x1D, 0xB3, 0x31, 0x79, 0x79, 0xB1, 0x90, 0xF8, 0x40, 0x04, 0x40, 0x07, 0x0B, 0xD5,
+ 0x21, 0x46, 0x40, 0x46, 0xFF, 0xF7, 0x7B, 0xFD, 0xEF, 0x49, 0x08, 0x70, 0x20, 0xB1, 0xA0, 0x8C,
+ 0x40, 0xF0, 0x80, 0x00, 0xA0, 0x84, 0xE5, 0xE7, 0x70, 0x79, 0x78, 0xB1, 0xEB, 0x48, 0xEC, 0x49,
+ 0xB0, 0xF9, 0x00, 0x00, 0xB1, 0xF9, 0x00, 0x10, 0x88, 0x42, 0x07, 0xDB, 0x23, 0x46, 0x42, 0x46,
+ 0x49, 0x46, 0x50, 0x46, 0x02, 0xF0, 0xD3, 0xF9, 0x01, 0x28, 0x04, 0xD0, 0xA0, 0x8C, 0x10, 0xF4,
+ 0xC0, 0x7F, 0x19, 0xD1, 0x04, 0xE0, 0xA0, 0x8C, 0x40, 0xF4, 0x80, 0x70, 0xA0, 0x84, 0x13, 0xE0,
+ 0x95, 0xB1, 0x38, 0x68, 0x90, 0xF8, 0xE0, 0x02, 0xC0, 0x07, 0x03, 0xD0, 0x21, 0x46, 0x40, 0x46,
+ 0x01, 0xF0, 0x95, 0xF9, 0x23, 0x46, 0x42, 0x46, 0x49, 0x46, 0x50, 0x46, 0x00, 0xE0, 0x0A, 0xE0,
+ 0xBD, 0xE8, 0xF0, 0x47, 0x02, 0xF0, 0xA8, 0xB8, 0x23, 0x46, 0x42, 0x46, 0x49, 0x46, 0x50, 0x46,
+ 0xBD, 0xE8, 0xF0, 0x47, 0xA8, 0xE5, 0x04, 0x20, 0x84, 0xF8, 0x40, 0x00, 0xF4, 0xE7, 0x2D, 0xE9,
+ 0xFF, 0x4F, 0xD0, 0x48, 0x87, 0xB0, 0xDF, 0xF8, 0x28, 0xB3, 0xB0, 0xF9, 0x00, 0x80, 0x0C, 0x46,
+ 0x00, 0x26, 0x14, 0x98, 0x9B, 0xF8, 0x04, 0x10, 0xDF, 0xF8, 0x2C, 0xA3, 0x01, 0x27, 0x15, 0x46,
+ 0x87, 0x40, 0xB1, 0x46, 0xD1, 0xB3, 0xC9, 0x48, 0xC5, 0x4A, 0xB0, 0xF9, 0x00, 0x10, 0xB2, 0xF9,
+ 0x00, 0x20, 0x91, 0x42, 0x32, 0xDA, 0x2A, 0x68, 0x3A, 0x42, 0x2F, 0xD1, 0xE2, 0x8C, 0x04, 0x2A,
+ 0x2C, 0xD9, 0x8D, 0xE8, 0x32, 0x00, 0x0A, 0x9B, 0x14, 0x9A, 0xDA, 0xF8, 0x18, 0x10, 0xDA, 0xF8,
+ 0x24, 0x00, 0xFF, 0xF7, 0xA1, 0xFE, 0x06, 0x00, 0x20, 0xD1, 0x23, 0x46, 0x20, 0x68, 0x0A, 0x9A,
+ 0x14, 0x99, 0xFF, 0xF7, 0x6E, 0xFC, 0xB4, 0x49, 0x08, 0x70, 0xA0, 0xB1, 0x70, 0x1E, 0x8D, 0xF8,
+ 0x10, 0x00, 0x8D, 0xF8, 0x14, 0x00, 0xB6, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x41, 0x04, 0x80, 0x07,
+ 0x05, 0xD5, 0x05, 0xAA, 0x04, 0xA9, 0x0A, 0x98, 0xF8, 0xF7, 0x13, 0xFC, 0x90, 0xB3, 0xA0, 0x8C,
+ 0x40, 0xF0, 0x80, 0x00, 0xA0, 0x84, 0xAD, 0x48, 0xB0, 0xF9, 0x00, 0x80, 0x9B, 0xF8, 0x05, 0x00,
+ 0x38, 0xB3, 0x94, 0xF8, 0x24, 0x00, 0x00, 0x06, 0x4F, 0xD4, 0xA4, 0x48, 0xA4, 0x4A, 0xB0, 0xF9,
+ 0x00, 0x10, 0xB2, 0xF9, 0x00, 0x20, 0x91, 0x42, 0x47, 0xDA, 0x83, 0x46, 0x41, 0x45, 0x17, 0xD0,
+ 0x28, 0x68, 0x38, 0x42, 0x07, 0xD0, 0xCD, 0xF8, 0x00, 0x90, 0x2A, 0x46, 0x21, 0x46, 0x14, 0x9B,
+ 0x07, 0x98, 0xFF, 0xF7, 0x0F, 0xFD, 0xBB, 0xF9, 0x00, 0x00, 0x8D, 0xE8, 0x31, 0x00, 0x0A, 0x9B,
+ 0x14, 0x9A, 0xDA, 0xF8, 0x18, 0x10, 0xDA, 0xF8, 0x24, 0x00, 0xFF, 0xF7, 0x55, 0xFE, 0x06, 0x46,
+ 0xFE, 0xB1, 0x2A, 0xE0, 0xFF, 0xE7, 0x97, 0x49, 0x9D, 0xF9, 0x10, 0x30, 0x01, 0xF8, 0x36, 0x3F,
+ 0x9D, 0xF9, 0x14, 0x00, 0x48, 0x70, 0x00, 0x90, 0x20, 0x68, 0x0A, 0x9A, 0x07, 0x99, 0xFF, 0xF7,
+ 0x61, 0xFC, 0x07, 0x99, 0x04, 0xF1, 0x3A, 0x03, 0x8D, 0xE8, 0x32, 0x02, 0x21, 0x68, 0x14, 0x9A,
+ 0xDA, 0xF8, 0x18, 0x00, 0xFF, 0xF7, 0x5E, 0xFE, 0x00, 0x28, 0xB4, 0xD0, 0x01, 0x20, 0x0B, 0xB0,
+ 0x30, 0xE6, 0x23, 0x46, 0x20, 0x68, 0x0A, 0x9A, 0x14, 0x99, 0x02, 0xF0, 0x08, 0xF9, 0x01, 0x28,
+ 0x03, 0xD1, 0xA0, 0x8C, 0x40, 0xF4, 0x80, 0x70, 0xA0, 0x84, 0xA0, 0x8C, 0x10, 0xF4, 0xC0, 0x7F,
+ 0x0A, 0xD1, 0x28, 0x68, 0x38, 0x42, 0x07, 0xD0, 0xCD, 0xF8, 0x00, 0x90, 0x2A, 0x46, 0x21, 0x46,
+ 0x14, 0x9B, 0x07, 0x98, 0xFF, 0xF7, 0xC6, 0xFC, 0x30, 0x46, 0xE0, 0xE7, 0x2D, 0xE9, 0xF0, 0x4F,
+ 0x79, 0x49, 0xDF, 0xF8, 0xCC, 0x81, 0xE5, 0xB0, 0xB1, 0xF9, 0x00, 0x10, 0xB8, 0xF9, 0x00, 0x00,
+ 0x00, 0x27, 0xCD, 0xE9, 0x60, 0x01, 0x70, 0x48, 0xBA, 0x46, 0xB9, 0x46, 0x3D, 0x46, 0x01, 0x21,
+ 0x1C, 0x30, 0x02, 0xF0, 0xE7, 0xFC, 0x6C, 0x48, 0x01, 0x21, 0x24, 0x30, 0x02, 0xF0, 0xE2, 0xFC,
+ 0x69, 0x48, 0x02, 0x21, 0x20, 0x30, 0x02, 0xF0, 0xDD, 0xFC, 0x02, 0x21, 0x6B, 0x48, 0x02, 0xF0,
+ 0x8F, 0xFC, 0x04, 0x21, 0x6A, 0x48, 0x02, 0xF0, 0x8B, 0xFC, 0x63, 0x4C, 0x4F, 0xF4, 0x66, 0x76,
+ 0x31, 0x46, 0xE0, 0x69, 0x0A, 0xF0, 0xC5, 0xFC, 0x31, 0x46, 0x60, 0x6A, 0x0A, 0xF0, 0xC1, 0xFC,
+ 0x64, 0x48, 0xB0, 0xF9, 0x00, 0x00, 0x41, 0x00, 0x20, 0x6A, 0x0A, 0xF0, 0xBA, 0xFC, 0x31, 0x46,
+ 0x61, 0x48, 0x0A, 0xF0, 0xB6, 0xFC, 0x4F, 0xF0, 0x00, 0x0B, 0x5E, 0x46, 0x84, 0xF8, 0x01, 0xB0,
+ 0xF8, 0xF7, 0x0F, 0xFF, 0xF8, 0xF7, 0xC4, 0xFA, 0x5C, 0x48, 0x8D, 0xF8, 0x72, 0x61, 0x01, 0x23,
+ 0x00, 0x68, 0x19, 0x90, 0x00, 0x22, 0x19, 0xA9, 0xA0, 0x69, 0x00, 0xF0, 0xB5, 0xFC, 0x58, 0x48,
+ 0x5E, 0x96, 0x00, 0x24, 0xB0, 0xF9, 0x00, 0x10, 0xB8, 0xF9, 0x00, 0x00, 0x05, 0xF0, 0x4A, 0xF9,
+ 0x0F, 0xFA, 0x80, 0xF8, 0x9D, 0xF8, 0x72, 0x01, 0x01, 0x28, 0x02, 0xD9, 0x42, 0x49, 0x81, 0xF8,
+ 0x00, 0xB0, 0x01, 0x26, 0xF1, 0xE0, 0x19, 0xA8, 0x00, 0xEB, 0x86, 0x00, 0x05, 0x90, 0x19, 0xAA,
+ 0x31, 0x46, 0x07, 0xA8, 0x5E, 0x9B, 0xFF, 0xF7, 0xDA, 0xFC, 0x07, 0xAB, 0x31, 0x46, 0x05, 0x9A,
+ 0x07, 0x98, 0xFF, 0xF7, 0xFC, 0xFC, 0x01, 0x20, 0x00, 0x90, 0x33, 0x46, 0x5E, 0xAA, 0x07, 0xA9,
+ 0x19, 0xA8, 0xFF, 0xF7, 0x47, 0xFC, 0x0D, 0xF1, 0x5A, 0x00, 0x00, 0xF0, 0xAD, 0xFA, 0x78, 0xB1,
+ 0xBD, 0xF8, 0x40, 0x00, 0x40, 0xF0, 0x01, 0x00, 0xAD, 0xF8, 0x40, 0x00, 0x07, 0xA8, 0x00, 0xF0,
+ 0x18, 0xFC, 0x28, 0xB1, 0x3B, 0x48, 0x34, 0x49, 0x00, 0x88, 0x08, 0x80, 0x2C, 0x49, 0x08, 0x80,
+ 0x28, 0x48, 0x00, 0x78, 0x38, 0xB1, 0xBD, 0xF8, 0x40, 0x00, 0x10, 0xF0, 0x06, 0x0F, 0x02, 0xD1,
+ 0x35, 0x48, 0x00, 0x68, 0x08, 0x90, 0x05, 0x98, 0x00, 0xF0, 0xA0, 0xFA, 0x28, 0xB1, 0xBD, 0xF8,
+ 0x40, 0x00, 0x40, 0xF0, 0x08, 0x00, 0xAD, 0xF8, 0x40, 0x00, 0xBD, 0xF9, 0x4E, 0x10, 0x41, 0x45,
+ 0x7E, 0xDD, 0x00, 0x96, 0x5E, 0xAA, 0x07, 0xA9, 0x19, 0xA8, 0x05, 0x9B, 0xFF, 0xF7, 0x97, 0xFE,
+ 0x00, 0x28, 0x76, 0xD1, 0x01, 0x20, 0xB0, 0x40, 0x5E, 0x99, 0x03, 0x90, 0x08, 0x42, 0x67, 0xD1,
+ 0xBD, 0xF8, 0x40, 0x00, 0x10, 0xF4, 0xC0, 0x7F, 0x62, 0xD1, 0x15, 0x48, 0x5E, 0xAA, 0x07, 0xA9,
+ 0xB0, 0xF9, 0x00, 0x00, 0x8D, 0xE8, 0x07, 0x00, 0x13, 0x48, 0x32, 0x46, 0x05, 0x9B, 0x81, 0x69,
+ 0x40, 0x6A, 0xFF, 0xF7, 0x49, 0xFD, 0x5E, 0x9A, 0x03, 0x99, 0x01, 0x90, 0x11, 0x42, 0x4D, 0xD0,
+ 0x0F, 0x49, 0xBD, 0xF8, 0x42, 0x20, 0x09, 0x68, 0x91, 0xF8, 0x0E, 0x33, 0x9A, 0x42, 0x2E, 0xD8,
+ 0x91, 0xF8, 0xE0, 0x12, 0x8A, 0x07, 0x41, 0xD4, 0x49, 0x07, 0x27, 0xE0, 0x4C, 0x07, 0x10, 0x00,
+ 0xEC, 0x06, 0x10, 0x00, 0x80, 0x06, 0x10, 0x00, 0x10, 0x06, 0x10, 0x00, 0x82, 0x06, 0x10, 0x00,
+ 0x6E, 0x06, 0x10, 0x00, 0x76, 0x06, 0x10, 0x00, 0x14, 0x06, 0x10, 0x00, 0x84, 0x06, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0xE6, 0x43, 0x01, 0x20, 0x70, 0x06, 0x10, 0x00, 0x3C, 0x06, 0x10, 0x00,
+ 0x48, 0x06, 0x10, 0x00, 0xCA, 0x06, 0x10, 0x00, 0xB4, 0x18, 0x01, 0x20, 0x64, 0x06, 0x10, 0x00,
+ 0x88, 0x06, 0x10, 0x00, 0x7E, 0x06, 0x10, 0x00, 0xE0, 0x06, 0x10, 0x00, 0x16, 0xD4, 0x01, 0x23,
+ 0x5E, 0xAA, 0x07, 0xA9, 0x19, 0xA8, 0x8D, 0xE8, 0x0F, 0x00, 0xFF, 0x48, 0x0D, 0xF1, 0x56, 0x03,
+ 0x32, 0x46, 0x41, 0x6A, 0x80, 0x69, 0xFF, 0xF7, 0x25, 0xFD, 0x01, 0x90, 0x01, 0x20, 0x00, 0x90,
+ 0x33, 0x46, 0x5E, 0xAA, 0x07, 0xA9, 0x19, 0xA8, 0xFF, 0xF7, 0xA4, 0xFB, 0x01, 0x98, 0x80, 0xB9,
+ 0xBD, 0xF8, 0x42, 0x00, 0x07, 0xA9, 0xFF, 0xF7, 0x6E, 0xFD, 0x50, 0xB1, 0x07, 0xAB, 0x01, 0xE0,
+ 0x0F, 0xE0, 0x28, 0xE0, 0x31, 0x46, 0x05, 0x9A, 0x07, 0x98, 0xFF, 0xF7, 0x7C, 0xFD, 0x01, 0x27,
+ 0x07, 0xE0, 0x33, 0x46, 0x5E, 0xAA, 0x07, 0xA9, 0x19, 0xA8, 0xCD, 0xF8, 0x00, 0xB0, 0xFF, 0xF7,
+ 0x89, 0xFB, 0x01, 0x20, 0x5E, 0x99, 0xB0, 0x40, 0x01, 0x43, 0xE8, 0x4A, 0x5E, 0x91, 0x61, 0x99,
+ 0x11, 0x80, 0xE7, 0x4A, 0x60, 0x99, 0x11, 0x80, 0xE6, 0x49, 0x0A, 0x78, 0x32, 0xB1, 0x9D, 0xF8,
+ 0x72, 0x21, 0x04, 0x43, 0x01, 0x2A, 0x01, 0xD9, 0x81, 0xF8, 0x00, 0xB0, 0xBD, 0xF8, 0x40, 0x10,
+ 0xC9, 0x05, 0x00, 0xD5, 0x05, 0x43, 0x76, 0x1C, 0xF6, 0xB2, 0x9D, 0xF8, 0x72, 0x01, 0xB0, 0x42,
+ 0xBF, 0xF4, 0x09, 0xAF, 0xD8, 0x48, 0xDD, 0x4A, 0xDB, 0x4B, 0x41, 0x78, 0xA2, 0xF5, 0x66, 0x70,
+ 0x01, 0xF0, 0x0F, 0xFE, 0xDA, 0x48, 0x9D, 0xF8, 0x72, 0x11, 0x00, 0xF8, 0x05, 0x1F, 0xD9, 0x49,
+ 0x04, 0x83, 0x85, 0x83, 0xD8, 0x48, 0x81, 0xF8, 0x00, 0xA0, 0xD8, 0x49, 0x07, 0x70, 0xD8, 0x48,
+ 0x81, 0xF8, 0x00, 0x90, 0x02, 0x21, 0x02, 0xF0, 0x53, 0xFB, 0x04, 0x21, 0xD5, 0x48, 0x02, 0xF0,
+ 0x4F, 0xFB, 0xC9, 0x48, 0x01, 0x21, 0x18, 0x30, 0x02, 0xF0, 0xDF, 0xFB, 0xC6, 0x48, 0x01, 0x21,
+ 0x1C, 0x30, 0x02, 0xF0, 0xDA, 0xFB, 0xC4, 0x48, 0x01, 0x21, 0x24, 0x30, 0x02, 0xF0, 0xD5, 0xFB,
+ 0xC1, 0x48, 0x02, 0x21, 0x20, 0x30, 0x02, 0xF0, 0xD0, 0xFB, 0x65, 0xB0, 0x82, 0xE4, 0x2D, 0xE9,
+ 0xF0, 0x4F, 0xBF, 0x48, 0x00, 0x25, 0xE3, 0xB0, 0xB0, 0xF9, 0x00, 0x60, 0xBA, 0x48, 0xA9, 0x46,
+ 0xA8, 0x46, 0x01, 0x21, 0x1C, 0x30, 0x02, 0xF0, 0x5D, 0xFB, 0xB7, 0x48, 0x01, 0x21, 0x24, 0x30,
+ 0x02, 0xF0, 0x58, 0xFB, 0xB4, 0x48, 0x02, 0x21, 0x20, 0x30, 0x02, 0xF0, 0x53, 0xFB, 0x02, 0x21,
+ 0xBB, 0x48, 0x02, 0xF0, 0x05, 0xFB, 0x04, 0x21, 0xBA, 0x48, 0x02, 0xF0, 0x01, 0xFB, 0xDF, 0xF8,
+ 0xB8, 0xA2, 0x4F, 0xF4, 0x66, 0x77, 0x39, 0x46, 0x54, 0x46, 0xDA, 0xF8, 0x1C, 0x00, 0x0A, 0xF0,
+ 0x38, 0xFB, 0x39, 0x46, 0x60, 0x6A, 0x0A, 0xF0, 0x34, 0xFB, 0xB3, 0x48, 0xB0, 0xF9, 0x00, 0x00,
+ 0x41, 0x00, 0x20, 0x6A, 0x0A, 0xF0, 0x2D, 0xFB, 0x39, 0x46, 0xB0, 0x48, 0x0A, 0xF0, 0x29, 0xFB,
+ 0xAB, 0x46, 0xAF, 0x48, 0x84, 0xF8, 0x01, 0xB0, 0x8D, 0xF8, 0x72, 0xB1, 0x00, 0x68, 0x19, 0x90,
+ 0x5F, 0x46, 0x01, 0x23, 0x5A, 0x46, 0x19, 0xA9, 0xA0, 0x69, 0x00, 0xF0, 0x2D, 0xFB, 0x01, 0x24,
+ 0x05, 0x97, 0x8C, 0xE0, 0x19, 0xA8, 0x00, 0xEB, 0x84, 0x00, 0x5D, 0x90, 0x19, 0xAA, 0x21, 0x46,
+ 0x07, 0xA8, 0x05, 0x9B, 0xFF, 0xF7, 0x63, 0xFB, 0x07, 0xAB, 0x21, 0x46, 0x5D, 0x9A, 0x07, 0x98,
+ 0xFF, 0xF7, 0x85, 0xFB, 0x01, 0x27, 0x23, 0x46, 0x05, 0xAA, 0x07, 0xA9, 0x19, 0xA8, 0x00, 0x97,
+ 0xFF, 0xF7, 0xD0, 0xFA, 0x0D, 0xF1, 0x5A, 0x00, 0x00, 0xF0, 0x36, 0xF9, 0x28, 0xB1, 0xBD, 0xF8,
+ 0x40, 0x00, 0x40, 0xF0, 0x01, 0x00, 0xAD, 0xF8, 0x40, 0x00, 0x5D, 0x98, 0x00, 0xF0, 0x3E, 0xF9,
+ 0x28, 0xB1, 0xBD, 0xF8, 0x40, 0x00, 0x40, 0xF0, 0x08, 0x00, 0xAD, 0xF8, 0x40, 0x00, 0xBD, 0xF9,
+ 0x4E, 0x00, 0xB0, 0x42, 0x55, 0xDD, 0x07, 0xFA, 0x04, 0xF0, 0x05, 0x99, 0x03, 0x90, 0x08, 0x42,
+ 0x39, 0xD1, 0xBD, 0xF8, 0x40, 0x00, 0x10, 0xF4, 0xC0, 0x7F, 0x34, 0xD1, 0x7C, 0x48, 0x05, 0xAA,
+ 0x07, 0xA9, 0xB0, 0xF9, 0x00, 0x00, 0x8D, 0xE8, 0x07, 0x00, 0x22, 0x46, 0x5D, 0x9B, 0xDA, 0xF8,
+ 0x18, 0x10, 0xDA, 0xF8, 0x24, 0x00, 0xFF, 0xF7, 0xEF, 0xFB, 0x5F, 0x90, 0x05, 0x99, 0x03, 0x98,
+ 0x08, 0x42, 0x1E, 0xD0, 0x7F, 0x48, 0xBD, 0xF8, 0x42, 0x10, 0x00, 0x68, 0x90, 0xF8, 0x0E, 0x23,
+ 0x91, 0x42, 0x05, 0xD8, 0x90, 0xF8, 0xE0, 0x02, 0x81, 0x07, 0x12, 0xD4, 0x40, 0x07, 0x10, 0xD4,
+ 0x07, 0xA9, 0x19, 0xAA, 0x05, 0xA8, 0xCD, 0xE9, 0x00, 0x21, 0xCD, 0xE9, 0x02, 0x07, 0x0D, 0xF1,
+ 0x56, 0x03, 0x22, 0x46, 0xDA, 0xF8, 0x24, 0x10, 0xDA, 0xF8, 0x18, 0x00, 0xFF, 0xF7, 0xF2, 0xFB,
+ 0xD8, 0xB9, 0x5F, 0x98, 0x68, 0xB9, 0xBD, 0xF8, 0x42, 0x00, 0x07, 0xA9, 0xFF, 0xF7, 0x43, 0xFC,
+ 0x38, 0xB1, 0x07, 0xAB, 0x21, 0x46, 0x5D, 0x9A, 0x07, 0x98, 0xFF, 0xF7, 0x54, 0xFC, 0x01, 0x25,
+ 0x07, 0xE0, 0x23, 0x46, 0x05, 0xAA, 0x07, 0xA9, 0x19, 0xA8, 0xCD, 0xF8, 0x00, 0xB0, 0xFF, 0xF7,
+ 0x61, 0xFA, 0x05, 0x98, 0xA7, 0x40, 0x07, 0x43, 0x05, 0x97, 0x64, 0x1C, 0xE4, 0xB2, 0x9D, 0xF8,
+ 0x72, 0x01, 0xA0, 0x42, 0xBF, 0xF4, 0x6E, 0xAF, 0x53, 0x4B, 0x54, 0x4A, 0x9A, 0xF8, 0x01, 0x10,
+ 0x5A, 0x48, 0x01, 0xF0, 0xFE, 0xFC, 0x52, 0x49, 0x9D, 0xF8, 0x72, 0x01, 0x48, 0x71, 0x51, 0x49,
+ 0x51, 0x48, 0x81, 0xF8, 0x00, 0x90, 0x51, 0x49, 0x05, 0x70, 0x51, 0x48, 0x81, 0xF8, 0x00, 0x80,
+ 0x02, 0x21, 0x02, 0xF0, 0x45, 0xFA, 0x04, 0x21, 0x4E, 0x48, 0x02, 0xF0, 0x41, 0xFA, 0x42, 0x48,
+ 0x01, 0x21, 0x18, 0x30, 0x02, 0xF0, 0xD1, 0xFA, 0x3F, 0x48, 0x01, 0x21, 0x1C, 0x30, 0x02, 0xF0,
+ 0xCC, 0xFA, 0x3D, 0x48, 0x01, 0x21, 0x24, 0x30, 0x02, 0xF0, 0xC7, 0xFA, 0x3A, 0x48, 0x02, 0x21,
+ 0x20, 0x30, 0x02, 0xF0, 0xC2, 0xFA, 0x63, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F, 0x3D, 0x49, 0x00, 0x20,
+ 0x08, 0x70, 0x3D, 0x49, 0x08, 0x70, 0x3D, 0x49, 0x08, 0x70, 0x36, 0x49, 0x08, 0x70, 0xA8, 0x21,
+ 0x35, 0x48, 0x0A, 0xF0, 0x68, 0xBA, 0xF1, 0xE7, 0x2F, 0x49, 0x00, 0x20, 0x38, 0x4A, 0x08, 0x62,
+ 0x88, 0x61, 0xC8, 0x61, 0x10, 0x60, 0x37, 0x4A, 0x10, 0x60, 0x32, 0x4A, 0x10, 0x70, 0x32, 0x4A,
+ 0x10, 0x70, 0x32, 0x4A, 0x10, 0x70, 0x2B, 0x4A, 0x10, 0x70, 0x48, 0x70, 0x70, 0x47, 0x26, 0x49,
+ 0x10, 0xB5, 0x00, 0x20, 0x48, 0x70, 0x4F, 0xF4, 0x66, 0x71, 0x30, 0x48, 0x0A, 0xF0, 0x29, 0xFA,
+ 0x25, 0x48, 0xA8, 0x21, 0xA8, 0x38, 0x0A, 0xF0, 0x46, 0xFA, 0xBD, 0xE8, 0x10, 0x40, 0xA8, 0x21,
+ 0x21, 0x48, 0x0A, 0xF0, 0x40, 0xBA, 0x2D, 0xE9, 0xF7, 0x4F, 0x1B, 0x48, 0x0D, 0x46, 0x41, 0x78,
+ 0x0F, 0x29, 0x01, 0xD2, 0x49, 0x1C, 0x41, 0x70, 0x01, 0xF0, 0xFF, 0x09, 0x1B, 0x49, 0x82, 0xF8,
+ 0x41, 0x90, 0x01, 0xEB, 0x89, 0x00, 0x29, 0x68, 0x01, 0x60, 0xA9, 0x78, 0x1F, 0x48, 0x0A, 0xF0,
+ 0x1F, 0xFB, 0x82, 0x46, 0xA8, 0x78, 0x19, 0xE0, 0x00, 0x98, 0x2C, 0x78, 0x00, 0xEB, 0xC8, 0x0B,
+ 0x0D, 0xE0, 0xDB, 0xE9, 0x00, 0x67, 0x22, 0x46, 0x01, 0x20, 0x00, 0x21, 0x0A, 0xF0, 0x43, 0xF9,
+ 0x06, 0x40, 0x0F, 0x40, 0x3E, 0x43, 0x01, 0xD0, 0x0A, 0xF8, 0x04, 0x90, 0x64, 0x1C, 0x68, 0x78,
+ 0xA0, 0x42, 0xEE, 0xDA, 0x0A, 0xF1, 0x28, 0x0A, 0x08, 0xF1, 0x01, 0x00, 0xE9, 0x78, 0x80, 0x46,
+ 0x41, 0x45, 0xE1, 0xDA, 0xBD, 0xE8, 0xFE, 0x8F, 0x14, 0x06, 0x10, 0x00, 0x70, 0x06, 0x10, 0x00,
+ 0x6E, 0x06, 0x10, 0x00, 0x10, 0x06, 0x10, 0x00, 0x48, 0x3F, 0x10, 0x00, 0x4C, 0x1C, 0x01, 0x20,
+ 0xE6, 0x43, 0x01, 0x20, 0xC2, 0x06, 0x10, 0x00, 0xC3, 0x06, 0x10, 0x00, 0xC4, 0x06, 0x10, 0x00,
+ 0x3C, 0x06, 0x10, 0x00, 0x48, 0x06, 0x10, 0x00, 0xCA, 0x06, 0x10, 0x00, 0xB4, 0x18, 0x01, 0x20,
+ 0x64, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0x01, 0x78, 0x59, 0xB1, 0x95, 0x4A, 0x12, 0x78,
+ 0x52, 0x1E, 0x91, 0x42, 0x06, 0xD0, 0x40, 0x78, 0x20, 0xB1, 0x93, 0x49, 0x09, 0x78, 0x49, 0x1E,
+ 0x88, 0x42, 0x01, 0xD1, 0x01, 0x20, 0x70, 0x47, 0x00, 0x20, 0x70, 0x47, 0x01, 0x78, 0x29, 0xB1,
+ 0x8C, 0x49, 0x42, 0x78, 0x09, 0x78, 0x49, 0x1E, 0x8A, 0x42, 0x09, 0xD1, 0x81, 0x78, 0x29, 0xB1,
+ 0xC1, 0x78, 0x89, 0x48, 0x00, 0x78, 0x40, 0x1E, 0x81, 0x42, 0x01, 0xD1, 0x01, 0x20, 0x70, 0x47,
+ 0x00, 0x20, 0x70, 0x47, 0xF0, 0xB5, 0x85, 0x4D, 0x00, 0x22, 0x05, 0xEB, 0xC2, 0x06, 0x00, 0x23,
+ 0xC6, 0xE9, 0x00, 0x33, 0x52, 0x1C, 0xD2, 0xB2, 0x17, 0x2A, 0xF6, 0xD3, 0x8A, 0x78, 0x2C, 0xE0,
+ 0x00, 0xEB, 0xC2, 0x03, 0xD3, 0xE9, 0x00, 0x76, 0x57, 0xEA, 0x06, 0x03, 0x23, 0xD0, 0x74, 0x08,
+ 0x4F, 0xEA, 0x37, 0x03, 0x17, 0xEB, 0x07, 0x0E, 0x46, 0xEB, 0x06, 0x0C, 0x44, 0xEA, 0x0C, 0x04,
+ 0x34, 0x43, 0x43, 0xEA, 0x0E, 0x03, 0x05, 0xEB, 0xC2, 0x06, 0x3B, 0x43, 0xD6, 0xE9, 0x00, 0x7C,
+ 0x1F, 0x43, 0x4C, 0xEA, 0x04, 0x0C, 0xC6, 0xE9, 0x00, 0x7C, 0xD6, 0xE9, 0x02, 0x7C, 0x1F, 0x43,
+ 0x4C, 0xEA, 0x04, 0x0C, 0xC6, 0xE9, 0x02, 0x7C, 0xD6, 0xE9, 0x04, 0x7C, 0x1F, 0x43, 0x4C, 0xEA,
+ 0x04, 0x0C, 0xC6, 0xE9, 0x04, 0x7C, 0x52, 0x1C, 0xD2, 0xB2, 0xCB, 0x78, 0x93, 0x42, 0xCF, 0xD2,
+ 0x00, 0x22, 0x65, 0x4B, 0x0A, 0xE0, 0x00, 0xBF, 0x05, 0xEB, 0xC2, 0x04, 0x00, 0xEB, 0xC2, 0x07,
+ 0xD4, 0xE9, 0x02, 0x64, 0xC7, 0xE9, 0x00, 0x64, 0x52, 0x1C, 0xD2, 0xB2, 0x1C, 0x78, 0xA2, 0x42,
+ 0xF2, 0xD3, 0x08, 0x78, 0x08, 0xB1, 0x40, 0x1E, 0x08, 0x70, 0x5A, 0x4A, 0x48, 0x78, 0x12, 0x78,
+ 0x52, 0x1E, 0x90, 0x42, 0x01, 0xDA, 0x40, 0x1C, 0x48, 0x70, 0x88, 0x78, 0x08, 0xB1, 0x40, 0x1E,
+ 0x88, 0x70, 0x1A, 0x78, 0xC8, 0x78, 0x52, 0x1E, 0x90, 0x42, 0x01, 0xDA, 0x40, 0x1C, 0xC8, 0x70,
+ 0xF0, 0xBD, 0x2D, 0xE9, 0xFF, 0x5F, 0x94, 0x78, 0x16, 0x46, 0x04, 0xEB, 0x84, 0x02, 0x8B, 0x46,
+ 0x00, 0xEB, 0xC2, 0x00, 0x00, 0x21, 0x00, 0xF1, 0x29, 0x08, 0x20, 0xE0, 0x00, 0x20, 0x81, 0x46,
+ 0x82, 0x46, 0x35, 0x78, 0x77, 0x78, 0x0F, 0xE0, 0x18, 0xF8, 0x05, 0x20, 0x5A, 0x45, 0x09, 0xD1,
+ 0x2A, 0x46, 0x01, 0x20, 0x00, 0x21, 0x0A, 0xF0, 0x66, 0xF8, 0x41, 0xEA, 0x0A, 0x0A, 0x40, 0xEA,
+ 0x09, 0x09, 0x01, 0x21, 0x6D, 0x1C, 0xED, 0xB2, 0xAF, 0x42, 0xED, 0xD2, 0x03, 0x9B, 0x08, 0xF1,
+ 0x28, 0x08, 0x03, 0xEB, 0xC4, 0x03, 0x64, 0x1C, 0xC3, 0xE9, 0x00, 0x9A, 0xE4, 0xB2, 0xF0, 0x78,
+ 0xA0, 0x42, 0xDB, 0xD2, 0x04, 0xB0, 0x08, 0x46, 0xBD, 0xE8, 0xF0, 0x9F, 0x0A, 0x78, 0x43, 0x78,
+ 0x9A, 0x42, 0x0B, 0xD8, 0x4A, 0x78, 0x03, 0x78, 0x9A, 0x42, 0x07, 0xD3, 0x8A, 0x78, 0xC3, 0x78,
+ 0x9A, 0x42, 0x03, 0xD8, 0xC9, 0x78, 0x80, 0x78, 0x81, 0x42, 0x01, 0xD2, 0x00, 0x20, 0x70, 0x47,
+ 0x01, 0x20, 0x70, 0x47, 0x70, 0xB5, 0x0C, 0x78, 0x8B, 0x78, 0x4D, 0x78, 0xC9, 0x78, 0x9A, 0xB1,
+ 0x0C, 0xB1, 0x64, 0x1E, 0xE4, 0xB2, 0x27, 0x4A, 0x12, 0x78, 0x52, 0x1E, 0x95, 0x42, 0x01, 0xDA,
+ 0x6D, 0x1C, 0xED, 0xB2, 0x0B, 0xB1, 0x5B, 0x1E, 0xDB, 0xB2, 0x23, 0x4A, 0x12, 0x78, 0x52, 0x1E,
+ 0x91, 0x42, 0x01, 0xDA, 0x49, 0x1C, 0xC9, 0xB2, 0x03, 0xEB, 0x83, 0x02, 0x00, 0xEB, 0xC2, 0x02,
+ 0x00, 0x26, 0x29, 0x32, 0x09, 0xE0, 0x20, 0x46, 0x02, 0xE0, 0x16, 0x54, 0x40, 0x1C, 0xC0, 0xB2,
+ 0xA8, 0x42, 0xFA, 0xD9, 0x5B, 0x1C, 0xDB, 0xB2, 0x28, 0x32, 0x8B, 0x42, 0xF3, 0xD9, 0x70, 0xBD,
+ 0x2D, 0xE9, 0xF0, 0x43, 0x9C, 0x78, 0x93, 0xF8, 0x03, 0xC0, 0x04, 0xEB, 0x84, 0x05, 0x00, 0xEB,
+ 0xC5, 0x05, 0x10, 0x48, 0x90, 0xF8, 0x00, 0x90, 0x04, 0xFB, 0x09, 0xF0, 0x01, 0xEB, 0x40, 0x06,
+ 0x00, 0x20, 0x29, 0x35, 0x12, 0xE0, 0x19, 0x78, 0x5F, 0x78, 0x08, 0xE0, 0x15, 0xF8, 0x01, 0x80,
+ 0x90, 0x45, 0x02, 0xD1, 0x36, 0xF9, 0x11, 0x80, 0x40, 0x44, 0x49, 0x1C, 0xC9, 0xB2, 0x8F, 0x42,
+ 0xF4, 0xD2, 0x64, 0x1C, 0x06, 0xEB, 0x49, 0x06, 0xE4, 0xB2, 0x28, 0x35, 0xA4, 0x45, 0xEA, 0xD2,
+ 0xBD, 0xE8, 0xF0, 0x83, 0xBA, 0x06, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0xF0, 0x3F, 0x10, 0x00,
+ 0x2D, 0xE9, 0xF0, 0x47, 0x80, 0x46, 0x27, 0x48, 0x8C, 0x78, 0x0D, 0x46, 0x00, 0x78, 0x80, 0x1E,
+ 0x01, 0x2C, 0x05, 0xD9, 0xE9, 0x78, 0x81, 0x42, 0x02, 0xDA, 0x00, 0x20, 0xBD, 0xE8, 0xF0, 0x87,
+ 0x02, 0x2C, 0x01, 0xD2, 0x02, 0x27, 0x01, 0xE0, 0xEF, 0x78, 0xC4, 0xB2, 0x4F, 0xF0, 0x00, 0x09,
+ 0x21, 0x46, 0xD8, 0xF8, 0x08, 0x00, 0x0A, 0xF0, 0x83, 0xF9, 0x06, 0x46, 0x21, 0x46, 0xD8, 0xF8,
+ 0x04, 0x00, 0x0A, 0xF0, 0x7D, 0xF9, 0x01, 0x46, 0x17, 0x48, 0x02, 0x78, 0x15, 0xE0, 0x28, 0x78,
+ 0x6B, 0x78, 0x0A, 0xE0, 0x36, 0xF9, 0x10, 0xC0, 0x31, 0xF9, 0x10, 0x80, 0xC4, 0x45, 0x02, 0xD0,
+ 0x4F, 0xF0, 0x01, 0x09, 0x03, 0xE0, 0x40, 0x1C, 0xC0, 0xB2, 0x83, 0x42, 0xF2, 0xD2, 0x64, 0x1C,
+ 0x06, 0xEB, 0x42, 0x06, 0x01, 0xEB, 0x42, 0x01, 0xE4, 0xB2, 0xBC, 0x42, 0xE7, 0xD9, 0x48, 0x46,
+ 0xCC, 0xE7, 0x08, 0x49, 0x09, 0x78, 0x49, 0x1E, 0xCA, 0xB2, 0x90, 0xF8, 0x3F, 0x10, 0x09, 0xB1,
+ 0x91, 0x42, 0x05, 0xD1, 0x81, 0x8C, 0x41, 0xF0, 0x02, 0x01, 0x81, 0x84, 0x01, 0x20, 0x70, 0x47,
+ 0x00, 0x20, 0x70, 0x47, 0xB9, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0x30, 0xB5, 0xB0, 0xB1,
+ 0x88, 0x42, 0x14, 0xD0, 0x02, 0xD2, 0x0A, 0x46, 0x01, 0x46, 0x10, 0x46, 0xC9, 0x4A, 0x53, 0x68,
+ 0x03, 0xEB, 0x40, 0x03, 0x1C, 0x78, 0x8C, 0x42, 0x09, 0xD0, 0x5C, 0x78, 0x8C, 0x42, 0x06, 0xD0,
+ 0x95, 0x68, 0x2A, 0x5C, 0x02, 0x2A, 0x02, 0xD2, 0x54, 0x1C, 0x2C, 0x54, 0x99, 0x54, 0x08, 0x46,
+ 0x30, 0xBD, 0xF0, 0xB5, 0x03, 0xEB, 0x82, 0x04, 0xA5, 0x78, 0x05, 0xEB, 0x85, 0x06, 0x00, 0xEB,
+ 0xC6, 0x06, 0xE0, 0x78, 0x28, 0x36, 0xA8, 0x42, 0x0F, 0xD3, 0x20, 0x78, 0x07, 0xE0, 0x37, 0x18,
+ 0x97, 0xF8, 0x01, 0xC0, 0x94, 0x45, 0x00, 0xD1, 0x79, 0x70, 0x40, 0x1C, 0xC0, 0xB2, 0x67, 0x78,
+ 0x87, 0x42, 0xF4, 0xD2, 0x6D, 0x1C, 0xED, 0xB2, 0xEB, 0xE7, 0x03, 0xEB, 0x81, 0x00, 0x25, 0x68,
+ 0x05, 0x60, 0x03, 0xEB, 0x42, 0x00, 0x03, 0xEB, 0x41, 0x01, 0x30, 0xF8, 0xB4, 0x2F, 0xA1, 0xF8,
+ 0xB4, 0x20, 0xAD, 0x49, 0x09, 0x68, 0x21, 0x60, 0x00, 0x21, 0x01, 0x80, 0x93, 0xF8, 0x0E, 0x01,
+ 0x40, 0x1E, 0x83, 0xF8, 0x0E, 0x01, 0xF0, 0xBD, 0x2D, 0xE9, 0xFF, 0x4F, 0x88, 0x46, 0x08, 0x46,
+ 0x09, 0x78, 0x8F, 0xB0, 0xFF, 0x29, 0x13, 0xD0, 0x8A, 0x46, 0xA4, 0x49, 0x42, 0x78, 0x09, 0x78,
+ 0x8A, 0x42, 0x00, 0xD2, 0x51, 0x1C, 0xCE, 0xB2, 0xA1, 0x49, 0x90, 0xF8, 0x02, 0x90, 0xC0, 0x78,
+ 0x09, 0x78, 0x88, 0x42, 0x01, 0xD2, 0x40, 0x1C, 0x00, 0xE0, 0x08, 0x46, 0xC7, 0xB2, 0x04, 0xE0,
+ 0x4F, 0xF0, 0x00, 0x0A, 0x56, 0x46, 0xD1, 0x46, 0x57, 0x46, 0x11, 0x98, 0x4F, 0xF0, 0x00, 0x0B,
+ 0x08, 0xB1, 0x98, 0xF8, 0x0E, 0xB1, 0x93, 0x4A, 0x11, 0x68, 0x01, 0xF1, 0x5A, 0x00, 0xC2, 0xE9,
+ 0x01, 0x10, 0x02, 0x46, 0x5F, 0xF0, 0x2C, 0x00, 0x00, 0x23, 0x01, 0xF8, 0x10, 0x00, 0x13, 0x54,
+ 0x40, 0x1E, 0xC0, 0xB2, 0xF8, 0xD2, 0x0B, 0xF1, 0x01, 0x00, 0x0D, 0x90, 0xC0, 0xB2, 0x06, 0x90,
+ 0x0F, 0x98, 0x09, 0xEB, 0x89, 0x01, 0x00, 0xEB, 0xC1, 0x05, 0x09, 0xF1, 0x01, 0x00, 0x09, 0x95,
+ 0x39, 0xE0, 0x00, 0xBF, 0x0A, 0xF1, 0x01, 0x00, 0x0A, 0x95, 0xC4, 0xB2, 0x28, 0x35, 0x2E, 0xE0,
+ 0x29, 0x19, 0x08, 0x68, 0xE0, 0xB1, 0x28, 0x5D, 0x38, 0xB3, 0x11, 0xF8, 0x01, 0x1C, 0x00, 0x20,
+ 0x01, 0xB1, 0x08, 0x46, 0x0A, 0x99, 0x09, 0x5D, 0x09, 0xB1, 0xFF, 0xF7, 0x57, 0xFF, 0x12, 0x99,
+ 0x61, 0xB1, 0x0A, 0x99, 0x21, 0x44, 0x0E, 0x91, 0x11, 0xF8, 0x01, 0x1C, 0x09, 0xB1, 0xFF, 0xF7,
+ 0x4D, 0xFF, 0x0E, 0x99, 0x49, 0x78, 0x09, 0xB1, 0xFF, 0xF7, 0x48, 0xFF, 0x18, 0xB1, 0x0B, 0xE0,
+ 0xE4, 0x1C, 0xE4, 0xB2, 0x09, 0xE0, 0x06, 0x98, 0x2D, 0x28, 0x03, 0xD2, 0x41, 0x1C, 0xC9, 0xB2,
+ 0x06, 0x91, 0x01, 0xE0, 0x40, 0x1E, 0xC0, 0xB2, 0x28, 0x55, 0x64, 0x1C, 0xE4, 0xB2, 0xB4, 0x42,
+ 0xCE, 0xD9, 0x07, 0x98, 0x40, 0x1C, 0xC0, 0xB2, 0x07, 0x90, 0xB8, 0x42, 0xC2, 0xD9, 0x06, 0x98,
+ 0x00, 0xF0, 0x6F, 0xF8, 0x06, 0x90, 0x0D, 0x98, 0x00, 0x22, 0xC0, 0xB2, 0x62, 0x4C, 0x09, 0xE0,
+ 0x08, 0xEB, 0x80, 0x03, 0x21, 0x68, 0x19, 0x60, 0x08, 0xEB, 0x40, 0x01, 0x40, 0x1C, 0xA1, 0xF8,
+ 0xB4, 0x20, 0xC0, 0xB2, 0x06, 0x99, 0x88, 0x42, 0xF2, 0xD3, 0x09, 0x98, 0x4D, 0x46, 0x28, 0x30,
+ 0x00, 0x90, 0x39, 0xE0, 0x54, 0x46, 0x30, 0xE0, 0x20, 0x44, 0x01, 0x21, 0x09, 0x58, 0xE9, 0xB3,
+ 0x41, 0x78, 0x41, 0xB3, 0x53, 0x4A, 0x52, 0x68, 0x12, 0xF8, 0x11, 0x10, 0x41, 0x70, 0x08, 0xEB,
+ 0x41, 0x00, 0x08, 0xEB, 0x81, 0x09, 0x30, 0xF8, 0xB4, 0x2F, 0x52, 0x1C, 0x02, 0x80, 0x99, 0xF8,
+ 0x00, 0x10, 0x20, 0x46, 0x04, 0xF0, 0xE6, 0xFB, 0x89, 0xF8, 0x00, 0x00, 0x99, 0xF8, 0x01, 0x10,
+ 0x20, 0x46, 0x04, 0xF0, 0xE3, 0xFB, 0x89, 0xF8, 0x01, 0x00, 0x99, 0xF8, 0x02, 0x10, 0x28, 0x46,
+ 0x04, 0xF0, 0xD8, 0xFB, 0x89, 0xF8, 0x02, 0x00, 0x99, 0xF8, 0x03, 0x10, 0x28, 0x46, 0x04, 0xF0,
+ 0xD5, 0xFB, 0x89, 0xF8, 0x03, 0x00, 0x64, 0x1C, 0xE4, 0xB2, 0x00, 0x98, 0xB4, 0x42, 0xCB, 0xD3,
+ 0x28, 0x30, 0x6D, 0x1C, 0xED, 0xB2, 0x00, 0x90, 0xBD, 0x42, 0xC3, 0xD3, 0x06, 0x98, 0x40, 0x1E,
+ 0xC2, 0xB2, 0x88, 0xF8, 0x0E, 0x21, 0x11, 0x98, 0x00, 0x28, 0x00, 0xE0, 0x0B, 0xE0, 0x0D, 0xD0,
+ 0xA2, 0xEB, 0x0B, 0x00, 0x01, 0x28, 0x09, 0xD1, 0x11, 0x99, 0x0F, 0x98, 0x13, 0xB0, 0x43, 0x46,
+ 0xBD, 0xE8, 0xF0, 0x4F, 0xDD, 0xE6, 0xE4, 0x1C, 0xE4, 0xB2, 0xDC, 0xE7, 0x13, 0xB0, 0xBD, 0xE8,
+ 0xF0, 0x8F, 0x2D, 0xE9, 0xF0, 0x5F, 0x07, 0x46, 0x00, 0x25, 0xDF, 0xF8, 0xA8, 0xB0, 0x2C, 0xE0,
+ 0xDB, 0xF8, 0x08, 0x10, 0x4E, 0x5D, 0xDB, 0xF8, 0x04, 0x10, 0x01, 0xEB, 0x45, 0x0A, 0x1A, 0xE0,
+ 0x0A, 0xEB, 0x06, 0x00, 0x80, 0x46, 0x10, 0xF8, 0x01, 0x1C, 0x10, 0xF8, 0x02, 0x9C, 0x08, 0x46,
+ 0x00, 0xF0, 0x35, 0xF8, 0x04, 0x46, 0x48, 0x46, 0x00, 0xF0, 0x31, 0xF8, 0x84, 0x42, 0x02, 0xD2,
+ 0x01, 0x46, 0x20, 0x46, 0x0C, 0x46, 0xDB, 0xF8, 0x04, 0x10, 0x76, 0x1E, 0xF6, 0xB2, 0x01, 0xF8,
+ 0x10, 0x40, 0x08, 0xF8, 0x02, 0x4C, 0x01, 0x2E, 0xE2, 0xD8, 0x04, 0xD1, 0x28, 0x46, 0x00, 0xF0,
+ 0x1E, 0xF8, 0x8A, 0xF8, 0x00, 0x00, 0x6D, 0x1C, 0xED, 0xB2, 0xBD, 0x42, 0xD0, 0xD3, 0x00, 0x21,
+ 0x08, 0x46, 0xDB, 0xF8, 0x04, 0x40, 0x0D, 0xE0, 0x04, 0xEB, 0x40, 0x02, 0x13, 0x78, 0x83, 0x42,
+ 0x03, 0xD1, 0x11, 0x70, 0x49, 0x1C, 0xC9, 0xB2, 0x02, 0xE0, 0x14, 0xF8, 0x13, 0x30, 0x13, 0x70,
+ 0x40, 0x1C, 0xC0, 0xB2, 0xB8, 0x42, 0xEF, 0xD3, 0x08, 0x46, 0xBD, 0xE8, 0xF0, 0x9F, 0x05, 0x4A,
+ 0xD2, 0xE9, 0x01, 0x12, 0x02, 0xE0, 0x00, 0xBF, 0x11, 0xF8, 0x10, 0x00, 0x13, 0x5C, 0x00, 0x2B,
+ 0xFA, 0xD1, 0x70, 0x47, 0x3C, 0x06, 0x10, 0x00, 0x52, 0x6C, 0x01, 0x00, 0xBA, 0x06, 0x10, 0x00,
+ 0xB9, 0x06, 0x10, 0x00, 0xF0, 0xB5, 0xB2, 0xF9, 0x00, 0x40, 0x06, 0x68, 0xB4, 0x42, 0x02, 0xDD,
+ 0xB3, 0xF9, 0x00, 0x10, 0x30, 0xE0, 0x02, 0xEB, 0x41, 0x04, 0x34, 0xF9, 0x02, 0x5C, 0xB5, 0x42,
+ 0x04, 0xDA, 0x03, 0xEB, 0x41, 0x01, 0x31, 0xF9, 0x02, 0x1C, 0x25, 0xE0, 0x00, 0x25, 0x2C, 0x46,
+ 0x4F, 0x1E, 0x0C, 0xE0, 0x32, 0xF9, 0x14, 0xC0, 0xB4, 0x45, 0x07, 0xDC, 0x02, 0xEB, 0x44, 0x0C,
+ 0xBC, 0xF9, 0x02, 0xC0, 0xB4, 0x45, 0x01, 0xDB, 0x25, 0x46, 0x0C, 0x46, 0x64, 0x1C, 0xBC, 0x42,
+ 0xF0, 0xDB, 0x32, 0xF9, 0x15, 0x40, 0x03, 0xEB, 0x45, 0x01, 0x33, 0xF9, 0x15, 0x30, 0xB1, 0xF9,
+ 0x02, 0x10, 0x02, 0xEB, 0x45, 0x02, 0x36, 0x1B, 0xC9, 0x1A, 0xB2, 0xF9, 0x02, 0x20, 0x71, 0x43,
+ 0x12, 0x1B, 0x91, 0xFB, 0xF2, 0xF1, 0x19, 0x44, 0x01, 0x60, 0xF0, 0xBD, 0x2D, 0xE9, 0xF0, 0x41,
+ 0x05, 0x46, 0x1E, 0x48, 0x00, 0x68, 0x00, 0xF2, 0x0F, 0x53, 0x11, 0xB3, 0x03, 0x29, 0x27, 0xD0,
+ 0x04, 0x29, 0x30, 0xD1, 0x90, 0xF8, 0x08, 0x15, 0x90, 0xF8, 0x09, 0x45, 0x90, 0xF8, 0x0E, 0x05,
+ 0x03, 0xEB, 0x41, 0x06, 0x03, 0xEB, 0x40, 0x0C, 0x50, 0x18, 0x20, 0x44, 0x0C, 0xEB, 0x41, 0x07,
+ 0x64, 0x28, 0x20, 0xDC, 0x01, 0x29, 0x03, 0xDD, 0x28, 0x1D, 0x62, 0x46, 0xFF, 0xF7, 0xA2, 0xFF,
+ 0x01, 0x2C, 0x18, 0xDD, 0x28, 0x46, 0x33, 0x46, 0x3A, 0x46, 0x21, 0x46, 0xBD, 0xE8, 0xF0, 0x41,
+ 0x98, 0xE7, 0x90, 0xF8, 0x08, 0x15, 0x90, 0xF8, 0x09, 0x45, 0x90, 0xF8, 0x0A, 0x25, 0x05, 0xE0,
+ 0x90, 0xF8, 0x08, 0x15, 0x90, 0xF8, 0x09, 0x45, 0x90, 0xF8, 0x0D, 0x25, 0x03, 0xEB, 0x41, 0x06,
+ 0x03, 0xEB, 0x42, 0x0C, 0xD8, 0xE7, 0xBD, 0xE8, 0xF0, 0x81, 0x00, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x30, 0xB5, 0x42, 0x78, 0x03, 0x78, 0xCD, 0x8C, 0xD2, 0x1A, 0xC3, 0x78, 0x80, 0x78, 0x52, 0x1C,
+ 0x1B, 0x1A, 0x5B, 0x1C, 0x05, 0xFB, 0x02, 0xF0, 0xD4, 0x18, 0x5D, 0x43, 0x40, 0x00, 0x6A, 0x00,
+ 0xB0, 0xFB, 0xF4, 0xF0, 0xB2, 0xFB, 0xF4, 0xF2, 0x90, 0x42, 0x02, 0xD9, 0xC8, 0x85, 0x0A, 0x86,
+ 0x30, 0xBD, 0xCA, 0x85, 0x08, 0x86, 0x30, 0xBD, 0x2D, 0xE9, 0xFF, 0x4F, 0x00, 0x27, 0x83, 0xB0,
+ 0x15, 0x46, 0x01, 0x97, 0x00, 0x97, 0x3E, 0x46, 0x3C, 0x46, 0xBB, 0x46, 0x91, 0x78, 0x09, 0xF0,
+ 0xF9, 0xFE, 0x82, 0x46, 0xAB, 0x78, 0x95, 0xF8, 0x03, 0x90, 0x28, 0xE0, 0x29, 0x78, 0x95, 0xF8,
+ 0x01, 0xC0, 0x8E, 0x46, 0x1D, 0xE0, 0x00, 0xBF, 0x1A, 0xF8, 0x01, 0x20, 0x04, 0x98, 0x82, 0x42,
+ 0x15, 0xD1, 0xA1, 0xEB, 0x0E, 0x02, 0x52, 0x1C, 0xDD, 0xF8, 0x00, 0x80, 0xA9, 0xEB, 0x03, 0x00,
+ 0x02, 0xFB, 0x02, 0x88, 0x40, 0x1C, 0xCD, 0xF8, 0x00, 0x80, 0xDD, 0xF8, 0x04, 0x80, 0x16, 0x44,
+ 0x02, 0xFB, 0x00, 0x82, 0x00, 0xFB, 0x00, 0xBB, 0x04, 0x44, 0x7F, 0x1C, 0x01, 0x92, 0x49, 0x1C,
+ 0xC9, 0xB2, 0x8C, 0x45, 0xE0, 0xD2, 0x5B, 0x1C, 0x0A, 0xF1, 0x28, 0x0A, 0xDB, 0xB2, 0x99, 0x45,
+ 0xD4, 0xD2, 0x4F, 0xF4, 0x7A, 0x70, 0x70, 0x43, 0x90, 0xFB, 0xF7, 0xF1, 0x4F, 0xF4, 0x7A, 0x70,
+ 0x60, 0x43, 0x90, 0xFB, 0xF7, 0xF3, 0x4F, 0xF4, 0x7A, 0x72, 0x01, 0xFB, 0x04, 0xF0, 0x71, 0x43,
+ 0x90, 0xFB, 0xF2, 0xF7, 0x91, 0xFB, 0xF2, 0xF6, 0x63, 0x43, 0x93, 0xFB, 0xF2, 0xF3, 0x01, 0x98,
+ 0x00, 0x99, 0xC0, 0x1B, 0x89, 0x1B, 0xAB, 0xEB, 0x03, 0x02, 0x06, 0x9C, 0x8B, 0x1A, 0x00, 0xD5,
+ 0x5B, 0x42, 0x00, 0x28, 0x00, 0xDA, 0x40, 0x42, 0x03, 0xEB, 0x40, 0x00, 0x11, 0x44, 0x0A, 0x18,
+ 0x08, 0x1A, 0x52, 0x10, 0x40, 0x10, 0x4F, 0xF4, 0x7A, 0x71, 0x48, 0x43, 0x90, 0xFB, 0xF2, 0xF0,
+ 0xC0, 0xF5, 0x7A, 0x70, 0xA0, 0x85, 0x06, 0x99, 0x07, 0xB0, 0x28, 0x46, 0xBD, 0xE8, 0xF0, 0x4F,
+ 0x6E, 0xE7, 0x00, 0x00, 0x2D, 0xE9, 0xFF, 0x5F, 0x00, 0x25, 0xDD, 0xF8, 0x38, 0x80, 0x8B, 0x46,
+ 0x99, 0x46, 0x14, 0x46, 0x2E, 0x46, 0x2F, 0x46, 0x41, 0x46, 0x00, 0x98, 0x09, 0xF0, 0x88, 0xFE,
+ 0x30, 0xF9, 0x19, 0x00, 0xA0, 0x42, 0x32, 0xDD, 0xA8, 0xF1, 0x01, 0x00, 0x44, 0xB2, 0x08, 0xF1,
+ 0x01, 0x08, 0x28, 0xE0, 0x00, 0x2C, 0x22, 0xDB, 0xA9, 0x48, 0x00, 0x78, 0x84, 0x42, 0x1E, 0xDA,
+ 0xE1, 0xB2, 0x00, 0x98, 0x09, 0xF0, 0x74, 0xFE, 0xA6, 0x49, 0x82, 0x46, 0xA9, 0xF1, 0x01, 0x00,
+ 0x0A, 0x78, 0x40, 0xB2, 0x09, 0xF1, 0x01, 0x01, 0x0E, 0xE0, 0x00, 0x28, 0x08, 0xDB, 0x90, 0x42,
+ 0x06, 0xDA, 0x3B, 0xF9, 0x15, 0x30, 0x3A, 0xF9, 0x10, 0xC0, 0x1F, 0x44, 0x0C, 0xFB, 0x03, 0x66,
+ 0x6D, 0x1C, 0x40, 0x1C, 0xED, 0xB2, 0x40, 0xB2, 0x88, 0x42, 0xEE, 0xDD, 0x01, 0xE0, 0xED, 0x1C,
+ 0xED, 0xB2, 0x64, 0x1C, 0x64, 0xB2, 0x44, 0x45, 0xD4, 0xDD, 0x0F, 0xB1, 0x01, 0xE0, 0x06, 0x46,
+ 0x01, 0x27, 0x96, 0xFB, 0xF7, 0xF0, 0x04, 0xB0, 0x00, 0xB2, 0xBD, 0xE8, 0xF0, 0x9F, 0x2D, 0xE9,
+ 0xFF, 0x4F, 0x83, 0xB0, 0x91, 0x46, 0x10, 0x99, 0x08, 0x78, 0x00, 0xB1, 0x40, 0x1E, 0xC4, 0xB2,
+ 0x88, 0x78, 0x00, 0xB1, 0x40, 0x1E, 0xDF, 0xF8, 0x2C, 0xA2, 0xC5, 0xB2, 0x4A, 0x78, 0x9A, 0xF8,
+ 0x00, 0x00, 0x43, 0x1E, 0x9A, 0x42, 0x01, 0xDA, 0x50, 0x1C, 0x00, 0xE0, 0x40, 0x1E, 0xDF, 0xF8,
+ 0x10, 0xB2, 0xC6, 0xB2, 0xC9, 0x78, 0x9B, 0xF8, 0x00, 0x00, 0x42, 0x1E, 0x91, 0x42, 0x01, 0xDA,
+ 0x48, 0x1C, 0x00, 0xE0, 0x40, 0x1E, 0xC7, 0xB2, 0xCC, 0xB9, 0xA8, 0x46, 0x12, 0xE0, 0x01, 0x46,
+ 0x03, 0x98, 0x09, 0xF0, 0x2B, 0xFE, 0xCD, 0xE9, 0x00, 0x80, 0x23, 0x46, 0x49, 0x46, 0x06, 0x9A,
+ 0x04, 0x98, 0xFF, 0xF7, 0x7F, 0xFF, 0x01, 0x99, 0x21, 0xF8, 0x14, 0x00, 0x08, 0xF1, 0x01, 0x00,
+ 0x00, 0xF0, 0xFF, 0x08, 0x40, 0x46, 0xB8, 0x45, 0xE9, 0xD9, 0x64, 0x1C, 0xE4, 0xB2, 0x71, 0x48,
+ 0x00, 0x78, 0x40, 0x1E, 0x86, 0x42, 0x19, 0xD1, 0xA8, 0x46, 0x12, 0xE0, 0x01, 0x46, 0x03, 0x98,
+ 0x09, 0xF0, 0x0C, 0xFE, 0x82, 0x46, 0x33, 0x46, 0xCD, 0xF8, 0x00, 0x80, 0x49, 0x46, 0x06, 0x9A,
+ 0x04, 0x98, 0xFF, 0xF7, 0x5F, 0xFF, 0x2A, 0xF8, 0x16, 0x00, 0x08, 0xF1, 0x01, 0x00, 0x00, 0xF0,
+ 0xFF, 0x08, 0x40, 0x46, 0xB8, 0x45, 0xE9, 0xD9, 0x76, 0x1E, 0xF6, 0xB2, 0xC5, 0xB9, 0x29, 0x46,
+ 0x03, 0x98, 0x09, 0xF0, 0xF3, 0xFD, 0x82, 0x46, 0xA0, 0x46, 0x0C, 0xE0, 0x03, 0x46, 0x00, 0x95,
+ 0x49, 0x46, 0x06, 0x9A, 0x04, 0x98, 0xFF, 0xF7, 0x45, 0xFF, 0x2A, 0xF8, 0x18, 0x00, 0x08, 0xF1,
+ 0x01, 0x00, 0x00, 0xF0, 0xFF, 0x08, 0x40, 0x46, 0xB0, 0x45, 0xEF, 0xD9, 0x6D, 0x1C, 0xED, 0xB2,
+ 0x9B, 0xF8, 0x00, 0x00, 0x40, 0x1E, 0x87, 0x42, 0x18, 0xD1, 0x39, 0x46, 0x03, 0x98, 0x09, 0xF0,
+ 0xD5, 0xFD, 0x82, 0x46, 0xA0, 0x46, 0x0C, 0xE0, 0x03, 0x46, 0x00, 0x97, 0x49, 0x46, 0x06, 0x9A,
+ 0x04, 0x98, 0xFF, 0xF7, 0x27, 0xFF, 0x2A, 0xF8, 0x18, 0x00, 0x08, 0xF1, 0x01, 0x00, 0x00, 0xF0,
+ 0xFF, 0x08, 0x40, 0x46, 0xB0, 0x45, 0xEF, 0xD9, 0x7F, 0x1E, 0xFF, 0xB2, 0xB9, 0xF9, 0x12, 0x00,
+ 0x5F, 0xEA, 0x00, 0x0A, 0x65, 0xD1, 0x4F, 0xF0, 0x01, 0x0A, 0x62, 0xE0, 0x29, 0x46, 0x03, 0x98,
+ 0x09, 0xF0, 0xB4, 0xFD, 0x01, 0x90, 0x68, 0x1E, 0xC1, 0xB2, 0x04, 0x98, 0x09, 0xF0, 0xA0, 0xFD,
+ 0x00, 0x90, 0x29, 0x46, 0x04, 0x98, 0x09, 0xF0, 0x9B, 0xFD, 0x21, 0x46, 0x4D, 0xE0, 0x00, 0xBF,
+ 0x30, 0xF9, 0x11, 0x20, 0x06, 0x9B, 0x9A, 0x42, 0x42, 0xDD, 0x00, 0x9A, 0xB9, 0xF9, 0x00, 0x80,
+ 0x02, 0xEB, 0x41, 0x02, 0x00, 0x23, 0x32, 0xF9, 0x02, 0xCD, 0x0C, 0xFB, 0x08, 0x33, 0xB2, 0xF9,
+ 0x02, 0xC0, 0xB9, 0xF9, 0x02, 0x80, 0x0C, 0xFB, 0x08, 0x33, 0xB2, 0xF9, 0x04, 0xC0, 0xB9, 0xF9,
+ 0x04, 0x80, 0x0C, 0xFB, 0x08, 0x3B, 0x2B, 0x4B, 0xB9, 0xF9, 0x06, 0x80, 0x93, 0xF8, 0x00, 0xE0,
+ 0x02, 0xEB, 0x4E, 0x02, 0xB2, 0xF9, 0x00, 0xC0, 0x0C, 0xFB, 0x08, 0xB3, 0xB2, 0xF9, 0x02, 0xC0,
+ 0xB9, 0xF9, 0x08, 0x80, 0x0C, 0xFB, 0x08, 0x33, 0xB2, 0xF9, 0x04, 0xC0, 0xB9, 0xF9, 0x0A, 0x80,
+ 0x02, 0xEB, 0x4E, 0x02, 0x0C, 0xFB, 0x08, 0x3B, 0xB2, 0xF9, 0x00, 0xC0, 0xB9, 0xF9, 0x0C, 0x80,
+ 0x0C, 0xFB, 0x08, 0xB3, 0xB2, 0xF9, 0x02, 0xC0, 0xB9, 0xF9, 0x0E, 0x80, 0x0C, 0xFB, 0x08, 0x33,
+ 0xB2, 0xF9, 0x04, 0xC0, 0xB9, 0xF9, 0x10, 0x20, 0x0C, 0xFB, 0x02, 0x32, 0x92, 0xFB, 0xFA, 0xF2,
+ 0x01, 0x9B, 0x23, 0xF8, 0x11, 0x20, 0x49, 0x1C, 0xC9, 0xB2, 0xB1, 0x42, 0xB0, 0xD9, 0x6D, 0x1C,
+ 0xED, 0xB2, 0xBD, 0x42, 0x9A, 0xD9, 0x07, 0xB0, 0xBD, 0xE8, 0xF0, 0x8F, 0x1C, 0xB5, 0x0E, 0x4C,
+ 0x24, 0x68, 0x01, 0x94, 0x01, 0xAC, 0x00, 0x94, 0xFF, 0xF7, 0xF1, 0xFE, 0x1C, 0xBD, 0x38, 0xB5,
+ 0x0D, 0x4D, 0x00, 0x90, 0x09, 0x48, 0x0C, 0x46, 0x09, 0x4A, 0x00, 0x68, 0xB0, 0xF9, 0xE2, 0x32,
+ 0x08, 0x48, 0x01, 0x68, 0x28, 0x68, 0xFF, 0xF7, 0xE2, 0xFE, 0x28, 0x68, 0xA0, 0x60, 0x38, 0xBD,
+ 0xB9, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0x64, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x3E, 0x6C, 0x01, 0x00, 0xDC, 0x06, 0x10, 0x00, 0x34, 0x06, 0x10, 0x00, 0x2D, 0xE9, 0xFF, 0x5F,
+ 0x89, 0x46, 0x14, 0x46, 0x91, 0x78, 0x0F, 0x9D, 0x09, 0xF0, 0x0C, 0xFD, 0x82, 0x46, 0xA1, 0x78,
+ 0xA8, 0x68, 0x09, 0xF0, 0x0D, 0xFD, 0x47, 0xF6, 0xFF, 0x76, 0xA7, 0x78, 0xF5, 0x43, 0xFE, 0x49,
+ 0x94, 0xF8, 0x03, 0x80, 0x91, 0xF8, 0x00, 0xB0, 0x18, 0xE0, 0x22, 0x78, 0x94, 0xF8, 0x01, 0xC0,
+ 0x0D, 0xE0, 0x00, 0xBF, 0x1A, 0xF8, 0x02, 0x10, 0x49, 0x45, 0x07, 0xD1, 0x30, 0xF9, 0x12, 0x30,
+ 0xB3, 0x42, 0x00, 0xDA, 0x1E, 0x46, 0xAB, 0x42, 0x00, 0xDD, 0x1D, 0x46, 0x52, 0x1C, 0x94, 0x45,
+ 0xF0, 0xD2, 0x0A, 0xF1, 0x28, 0x0A, 0x00, 0xEB, 0x4B, 0x00, 0x7F, 0x1C, 0xB8, 0x45, 0xE4, 0xD2,
+ 0x03, 0x98, 0x06, 0x80, 0x0E, 0x98, 0x05, 0x80, 0xBD, 0xE8, 0xFF, 0x9F, 0x2D, 0xE9, 0xFF, 0x4F,
+ 0xEA, 0x48, 0x85, 0xB0, 0x9A, 0x46, 0x04, 0x68, 0x03, 0xA9, 0xCD, 0xE9, 0x00, 0x1A, 0xDD, 0xE9,
+ 0x05, 0x01, 0x15, 0x46, 0x04, 0xF5, 0x88, 0x79, 0x02, 0xAB, 0xFF, 0xF7, 0xB7, 0xFF, 0xE4, 0x48,
+ 0xBD, 0xF9, 0x0C, 0x20, 0xDF, 0xF8, 0x8C, 0xB3, 0x00, 0x68, 0x5F, 0x46, 0x90, 0xF8, 0xE4, 0x12,
+ 0xBD, 0xF9, 0x08, 0x00, 0x12, 0x1A, 0x4A, 0x43, 0x64, 0x21, 0x92, 0xFB, 0xF1, 0xF1, 0x0E, 0x18,
+ 0x01, 0x22, 0x29, 0x46, 0xDB, 0xF8, 0x00, 0x00, 0xFF, 0xF7, 0xBC, 0xFA, 0xD7, 0xF8, 0x00, 0x80,
+ 0xDD, 0xE9, 0x05, 0x07, 0x31, 0xB2, 0x01, 0x91, 0xA9, 0x78, 0x09, 0xF0, 0xAB, 0xFC, 0x06, 0x46,
+ 0xA9, 0x78, 0x40, 0x46, 0x09, 0xF0, 0xB4, 0xFC, 0x00, 0x90, 0xA9, 0x78, 0xDA, 0xF8, 0x08, 0x00,
+ 0x09, 0xF0, 0xA6, 0xFC, 0x01, 0x46, 0xAA, 0x78, 0x01, 0x23, 0xDF, 0xF8, 0x2C, 0xE3, 0x1D, 0xE0,
+ 0x28, 0x78, 0x0E, 0xE0, 0x16, 0xF8, 0x00, 0xC0, 0xBC, 0x45, 0x09, 0xD1, 0x31, 0xF9, 0x10, 0x80,
+ 0xDD, 0xF8, 0x04, 0xC0, 0xE0, 0x45, 0x03, 0xDD, 0xDD, 0xF8, 0x00, 0xC0, 0x0C, 0xF8, 0x00, 0x30,
+ 0x40, 0x1C, 0x95, 0xF8, 0x01, 0xC0, 0x84, 0x45, 0xEC, 0xD2, 0x00, 0x98, 0x52, 0x1C, 0x28, 0x30,
+ 0x00, 0x90, 0x9E, 0xF8, 0x00, 0x00, 0x28, 0x36, 0x01, 0xEB, 0x40, 0x01, 0xE8, 0x78, 0x90, 0x42,
+ 0xDE, 0xD2, 0x00, 0x20, 0x84, 0xF8, 0x0E, 0x01, 0x28, 0x68, 0x20, 0x60, 0x01, 0x23, 0x00, 0x22,
+ 0x21, 0x46, 0x5E, 0x46, 0xDB, 0xF8, 0x00, 0x00, 0xFF, 0xF7, 0x76, 0xFB, 0x94, 0xF8, 0x0E, 0x01,
+ 0x01, 0x28, 0x26, 0xD9, 0x01, 0x25, 0x1D, 0xE0, 0x48, 0x22, 0x51, 0x46, 0x48, 0x46, 0x09, 0xF0,
+ 0x1E, 0xFB, 0x04, 0xEB, 0x45, 0x00, 0x04, 0xEB, 0x85, 0x07, 0xB0, 0xF8, 0xB4, 0x10, 0xA9, 0xF8,
+ 0x26, 0x10, 0x49, 0x46, 0x38, 0x46, 0xFF, 0xF7, 0x33, 0xFD, 0x3A, 0x46, 0xE9, 0xB2, 0x4B, 0x46,
+ 0x30, 0x68, 0xFE, 0xF7, 0xC4, 0xFB, 0x3A, 0x46, 0xE9, 0xB2, 0x4B, 0x46, 0x30, 0x68, 0xFE, 0xF7,
+ 0x3B, 0xFB, 0x6D, 0x1C, 0x94, 0xF8, 0x0E, 0x01, 0xA8, 0x42, 0xDD, 0xD2, 0x09, 0xB0, 0xBD, 0xE8,
+ 0xF0, 0x8F, 0xDD, 0xE9, 0x05, 0x01, 0x53, 0x46, 0x2A, 0x46, 0xFE, 0xF7, 0x2D, 0xFB, 0xF5, 0xE7,
+ 0x2D, 0xE9, 0xFE, 0x4F, 0x06, 0x46, 0x9A, 0x48, 0x1D, 0x46, 0x0B, 0x46, 0x00, 0x68, 0x9A, 0x49,
+ 0x0E, 0x9C, 0xB0, 0xF9, 0xE6, 0x92, 0x89, 0x79, 0x93, 0x46, 0x09, 0xB1, 0xB0, 0xF9, 0x62, 0x93,
+ 0x96, 0x49, 0x09, 0x78, 0x09, 0xB1, 0xB0, 0xF9, 0xF4, 0x92, 0x18, 0x46, 0x0C, 0x99, 0x09, 0xF0,
+ 0x21, 0xFC, 0x82, 0x46, 0x30, 0x46, 0x0C, 0x99, 0x09, 0xF0, 0x2A, 0xFC, 0x01, 0x90, 0xA0, 0x68,
+ 0x0C, 0x99, 0x09, 0xF0, 0x1D, 0xFC, 0x2C, 0x78, 0x6D, 0x78, 0x80, 0x46, 0x64, 0xB9, 0x9A, 0xF8,
+ 0x00, 0x10, 0x59, 0x45, 0x07, 0xD1, 0xB8, 0xF9, 0x00, 0x10, 0x0D, 0x98, 0x81, 0x42, 0x02, 0xDD,
+ 0x01, 0x99, 0x01, 0x20, 0x08, 0x70, 0x01, 0x24, 0x7F, 0x48, 0x00, 0x78, 0x40, 0x1E, 0x85, 0x42,
+ 0x55, 0xD1, 0x1A, 0xF8, 0x05, 0x10, 0x59, 0x45, 0x07, 0xD1, 0x38, 0xF9, 0x15, 0x10, 0x0D, 0x98,
+ 0x81, 0x42, 0x02, 0xDD, 0x01, 0x99, 0x01, 0x20, 0x48, 0x55, 0x6D, 0x1E, 0xED, 0xB2, 0x46, 0xE0,
+ 0x1A, 0xF8, 0x04, 0x10, 0x59, 0x45, 0x41, 0xD1, 0x38, 0xF9, 0x14, 0x10, 0x0D, 0x98, 0x02, 0x91,
+ 0x81, 0x42, 0x3B, 0xDD, 0x00, 0x20, 0x00, 0x90, 0x75, 0x49, 0x0C, 0x98, 0x22, 0x46, 0x01, 0xEB,
+ 0xC0, 0x00, 0x00, 0x21, 0xD0, 0xE9, 0x00, 0x67, 0x01, 0x20, 0x09, 0xF0, 0x1C, 0xFA, 0x06, 0x40,
+ 0x0F, 0x40, 0x3E, 0x43, 0x27, 0xD1, 0x02, 0x98, 0x08, 0xEB, 0x44, 0x02, 0x48, 0x44, 0x32, 0xF9,
+ 0x02, 0x1C, 0x00, 0xB2, 0x81, 0x42, 0x05, 0xDD, 0xB2, 0xF9, 0x02, 0x30, 0x83, 0x42, 0x01, 0xDD,
+ 0x01, 0x23, 0x00, 0x93, 0x01, 0x2C, 0x09, 0xD9, 0x32, 0xF9, 0x04, 0x3C, 0x83, 0x42, 0x05, 0xDD,
+ 0xB2, 0xF9, 0x02, 0x30, 0x83, 0x42, 0x01, 0xDD, 0x01, 0x23, 0x00, 0x93, 0x5A, 0x4B, 0x1B, 0x78,
+ 0x9B, 0x1E, 0x9C, 0x42, 0x05, 0xD2, 0x81, 0x42, 0x03, 0xDD, 0xB2, 0xF9, 0x04, 0x10, 0x81, 0x42,
+ 0x04, 0xDC, 0x00, 0x98, 0x10, 0xB9, 0x01, 0x99, 0x01, 0x20, 0x08, 0x55, 0x64, 0x1C, 0xAC, 0x42,
+ 0xB6, 0xD9, 0xBD, 0xE8, 0xFE, 0x8F, 0x2D, 0xE9, 0xFF, 0x4F, 0x54, 0x48, 0x9B, 0xB0, 0xDF, 0xF8,
+ 0x48, 0x91, 0x01, 0x78, 0x4E, 0x48, 0x28, 0x9E, 0x1D, 0x46, 0x00, 0x68, 0x39, 0xB1, 0x90, 0xF8,
+ 0xF3, 0x42, 0xB0, 0xF9, 0xF4, 0x12, 0x11, 0x91, 0xB0, 0xF9, 0xF6, 0x02, 0x11, 0xE0, 0x99, 0xF8,
+ 0x06, 0x10, 0x39, 0xB1, 0x90, 0xF8, 0x68, 0x43, 0xB0, 0xF9, 0x62, 0x13, 0x11, 0x91, 0xB0, 0xF9,
+ 0x64, 0x03, 0x06, 0xE0, 0x90, 0xF8, 0xE5, 0x42, 0xB0, 0xF9, 0xE6, 0x12, 0x11, 0x91, 0xB0, 0xF9,
+ 0xE8, 0x02, 0x44, 0x4F, 0x83, 0x46, 0xAC, 0xB1, 0x04, 0xA8, 0xCD, 0xE9, 0x00, 0x06, 0xDD, 0xE9,
+ 0x1C, 0x01, 0x02, 0xAB, 0x2A, 0x46, 0xFF, 0xF7, 0x61, 0xFE, 0xBD, 0xF9, 0x10, 0x00, 0xBD, 0xF9,
+ 0x08, 0x10, 0x40, 0x1A, 0x60, 0x43, 0x64, 0x21, 0x90, 0xFB, 0xF1, 0xF0, 0x39, 0x88, 0x08, 0x44,
+ 0x00, 0xB2, 0x01, 0xE0, 0xB7, 0xF9, 0x00, 0x00, 0xAC, 0x78, 0x95, 0xF8, 0x03, 0x80, 0x82, 0x46,
+ 0x4C, 0xB9, 0xCD, 0xE9, 0x00, 0x4A, 0x1B, 0xAF, 0x02, 0x96, 0x97, 0xE8, 0x07, 0x00, 0x2B, 0x46,
+ 0xFF, 0xF7, 0x1E, 0xFF, 0x01, 0x24, 0x30, 0x48, 0x00, 0x78, 0x40, 0x1E, 0x80, 0x45, 0x0D, 0xD1,
+ 0xCD, 0xF8, 0x00, 0x80, 0xCD, 0xE9, 0x01, 0xA6, 0x1B, 0xAF, 0x2B, 0x46, 0x97, 0xE8, 0x07, 0x00,
+ 0xFF, 0xF7, 0x0E, 0xFF, 0xA8, 0xF1, 0x01, 0x00, 0x00, 0xF0, 0xFF, 0x08, 0xA0, 0x1E, 0xC1, 0xB2,
+ 0xB0, 0x68, 0x09, 0xF0, 0x45, 0xFB, 0x09, 0x90, 0x60, 0x1E, 0xC1, 0xB2, 0xB0, 0x68, 0x09, 0xF0,
+ 0x3F, 0xFB, 0x0B, 0x90, 0x21, 0x46, 0xB0, 0x68, 0x09, 0xF0, 0x3A, 0xFB, 0x06, 0x90, 0x60, 0x1C,
+ 0xC1, 0xB2, 0xB0, 0x68, 0x09, 0xF0, 0x34, 0xFB, 0x05, 0x90, 0xA0, 0x1C, 0xC1, 0xB2, 0xB0, 0x68,
+ 0x09, 0xF0, 0x2E, 0xFB, 0x0C, 0x90, 0x21, 0x46, 0x1C, 0x98, 0x09, 0xF0, 0x23, 0xFB, 0x07, 0x90,
+ 0x21, 0x46, 0x1B, 0x98, 0x09, 0xF0, 0x2C, 0xFB, 0x02, 0x90, 0x00, 0x20, 0x14, 0x90, 0x01, 0x20,
+ 0x13, 0x90, 0x00, 0x20, 0x12, 0x90, 0x99, 0xF8, 0x04, 0x00, 0x58, 0xB3, 0x08, 0x48, 0x00, 0x68,
+ 0x90, 0xF8, 0x40, 0x14, 0x09, 0x07, 0x25, 0xD5, 0x6A, 0x78, 0x29, 0x78, 0x52, 0x1A, 0x90, 0xF8,
+ 0x4B, 0x14, 0x52, 0x1C, 0x8A, 0x42, 0x11, 0xE0, 0xBA, 0x06, 0x10, 0x00, 0x48, 0x06, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0x30, 0x06, 0x10, 0x00, 0xEC, 0x06, 0x10, 0x00, 0x28, 0x05, 0x10, 0x00,
+ 0x48, 0x3F, 0x10, 0x00, 0x6E, 0x06, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0x0A, 0xDC, 0xEA, 0x78,
+ 0xAB, 0x78, 0xD2, 0x1A, 0x52, 0x1C, 0x8A, 0x42, 0x04, 0xDC, 0xB0, 0xF8, 0x49, 0x04, 0xB1, 0x69,
+ 0x81, 0x42, 0x01, 0xDD, 0x00, 0x20, 0x13, 0x90, 0xA1, 0x46, 0xE0, 0xE0, 0x2C, 0x78, 0xBC, 0xE0,
+ 0x07, 0x98, 0x01, 0x5D, 0x1D, 0x98, 0x81, 0x42, 0x67, 0xD1, 0x00, 0x20, 0x10, 0x90, 0xFD, 0x49,
+ 0x16, 0x90, 0x15, 0x90, 0x01, 0xEB, 0xC9, 0x00, 0x22, 0x46, 0xD0, 0xE9, 0x00, 0x67, 0x01, 0x20,
+ 0x00, 0x21, 0x09, 0xF0, 0x10, 0xF9, 0x06, 0x40, 0x0F, 0x40, 0x3E, 0x43, 0x7D, 0xD1, 0x06, 0x98,
+ 0x0B, 0x9A, 0x30, 0xF8, 0x14, 0x10, 0x11, 0x98, 0x32, 0xF9, 0x14, 0x20, 0x08, 0x44, 0x00, 0xB2,
+ 0x59, 0x44, 0x09, 0xB2, 0x82, 0x42, 0x07, 0xDD, 0x05, 0x9B, 0x33, 0xF9, 0x14, 0x30, 0x83, 0x42,
+ 0x02, 0xDD, 0x01, 0x23, 0x10, 0x93, 0x15, 0x93, 0xF4, 0xB3, 0xEB, 0x4B, 0x1B, 0x78, 0x5E, 0x1E,
+ 0xB4, 0x42, 0x68, 0xD0, 0x06, 0x9E, 0x06, 0xEB, 0x44, 0x0E, 0x3E, 0xF9, 0x02, 0x6C, 0x86, 0x42,
+ 0x06, 0xDD, 0xBE, 0xF9, 0x02, 0x70, 0x87, 0x42, 0x02, 0xDD, 0x01, 0x27, 0x10, 0x97, 0x16, 0x97,
+ 0x0B, 0x9F, 0x07, 0xEB, 0x44, 0x07, 0x37, 0xF9, 0x02, 0xCC, 0x8C, 0x45, 0x0B, 0xDD, 0xDD, 0xF8,
+ 0x14, 0xC0, 0x0C, 0xEB, 0x44, 0x0C, 0xBC, 0xF9, 0x02, 0xC0, 0x8C, 0x45, 0x03, 0xDD, 0x4F, 0xF0,
+ 0x01, 0x0C, 0xCD, 0xF8, 0x40, 0xC0, 0xB7, 0xF9, 0x02, 0x70, 0x8F, 0x42, 0x06, 0xDD, 0x05, 0x9F,
+ 0x07, 0xEB, 0x44, 0x07, 0x37, 0xF9, 0x02, 0x7C, 0x8F, 0x42, 0x47, 0xDC, 0x10, 0x99, 0xA9, 0xBB,
+ 0xB9, 0xF1, 0x01, 0x0F, 0x0F, 0xD9, 0x09, 0x99, 0x31, 0xF9, 0x14, 0x10, 0x81, 0x42, 0x0A, 0xDD,
+ 0x05, 0x99, 0x31, 0xF9, 0x14, 0x10, 0x01, 0xE0, 0x2D, 0xE0, 0x4D, 0xE0, 0x81, 0x42, 0x02, 0xDD,
+ 0x01, 0x21, 0x10, 0x91, 0x15, 0x91, 0xC9, 0x4F, 0x3F, 0x78, 0xBF, 0x1E, 0xB9, 0x45, 0x09, 0xD2,
+ 0x82, 0x42, 0x07, 0xDD, 0x0C, 0x99, 0x31, 0xF9, 0x14, 0x10, 0x81, 0x42, 0x02, 0xDD, 0x01, 0x21,
+ 0x10, 0x91, 0x15, 0x91, 0x01, 0x2C, 0x0A, 0xD9, 0x3E, 0xF9, 0x04, 0x1C, 0x81, 0x42, 0x06, 0xDD,
+ 0xBE, 0xF9, 0x02, 0x10, 0x81, 0x42, 0x02, 0xDD, 0x01, 0x21, 0x10, 0x91, 0x16, 0x91, 0x9B, 0x1E,
+ 0x9C, 0x42, 0x08, 0xD2, 0x86, 0x42, 0x06, 0xDD, 0x01, 0xE0, 0x06, 0xE0, 0x0E, 0xE0, 0xBE, 0xF9,
+ 0x04, 0x10, 0x81, 0x42, 0x0C, 0xDC, 0x10, 0x98, 0x40, 0xB9, 0x06, 0x98, 0x30, 0xF9, 0x14, 0x10,
+ 0x51, 0x45, 0x03, 0xDD, 0x02, 0x99, 0x01, 0x20, 0x08, 0x55, 0x15, 0xE0, 0x16, 0x98, 0x98, 0xB1,
+ 0x15, 0x98, 0x88, 0xB1, 0x13, 0x98, 0x78, 0xB1, 0x02, 0x99, 0x01, 0x20, 0x08, 0x55, 0x12, 0x98,
+ 0x40, 0x1C, 0x80, 0xB2, 0x12, 0x90, 0xAA, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x4C, 0x14, 0x12, 0x98,
+ 0x81, 0x42, 0x01, 0xD2, 0x01, 0x20, 0x14, 0x90, 0x64, 0x1C, 0x68, 0x78, 0xA0, 0x42, 0xBF, 0xF4,
+ 0x3F, 0xAF, 0xA1, 0x48, 0x09, 0x99, 0x09, 0xF1, 0x01, 0x09, 0x00, 0x78, 0x01, 0xEB, 0x40, 0x01,
+ 0x09, 0x91, 0x0B, 0x99, 0x01, 0xEB, 0x40, 0x01, 0x0B, 0x91, 0x06, 0x99, 0x01, 0xEB, 0x40, 0x01,
+ 0x06, 0x91, 0x05, 0x99, 0x01, 0xEB, 0x40, 0x01, 0x05, 0x91, 0x0C, 0x99, 0x01, 0xEB, 0x40, 0x00,
+ 0x0C, 0x90, 0x07, 0x98, 0x28, 0x30, 0x07, 0x90, 0x02, 0x98, 0x28, 0x30, 0x02, 0x90, 0xC1, 0x45,
+ 0x7F, 0xF6, 0x1C, 0xAF, 0x14, 0x98, 0x1F, 0xB0, 0xB9, 0xE5, 0x2D, 0xE9, 0xF0, 0x47, 0x04, 0x46,
+ 0x91, 0x42, 0x5A, 0xD0, 0x07, 0xD9, 0x01, 0xEB, 0x02, 0x00, 0xA0, 0xEB, 0x02, 0x01, 0xCA, 0xB2,
+ 0xA0, 0xEB, 0x02, 0x00, 0xC1, 0xB2, 0x01, 0xEB, 0xC1, 0x00, 0x04, 0xEB, 0xC0, 0x01, 0x02, 0xEB,
+ 0xC2, 0x00, 0x04, 0xEB, 0xC0, 0x00, 0xB1, 0xF9, 0x32, 0x50, 0xB0, 0xF9, 0x32, 0x30, 0x81, 0x46,
+ 0x9D, 0x42, 0x00, 0xDA, 0x4B, 0x86, 0xCB, 0x8C, 0xC5, 0x8C, 0x2B, 0x44, 0xCB, 0x84, 0x0B, 0x8D,
+ 0x05, 0x8D, 0x2B, 0x44, 0x0B, 0x85, 0x4B, 0x8D, 0x45, 0x8D, 0x2B, 0x44, 0x4B, 0x85, 0x8B, 0x69,
+ 0x85, 0x69, 0x2B, 0x44, 0x8B, 0x61, 0xC7, 0x69, 0xCE, 0x69, 0xCB, 0x68, 0xD9, 0xF8, 0x0C, 0x00,
+ 0x4F, 0xEA, 0x23, 0x4C, 0x4F, 0xEA, 0x20, 0x48, 0x0C, 0xFB, 0x06, 0xFC, 0xF5, 0x19, 0x07, 0xFB,
+ 0x08, 0xCC, 0x9C, 0xFB, 0xF5, 0xFC, 0x6C, 0xF3, 0x1F, 0x43, 0x0F, 0xFA, 0x83, 0xFC, 0x0C, 0xFB,
+ 0x06, 0xFC, 0x00, 0xB2, 0x07, 0xFB, 0x00, 0xC0, 0x90, 0xFB, 0xF5, 0xF0, 0x60, 0xF3, 0x0F, 0x03,
+ 0xCD, 0x61, 0xCB, 0x60, 0x94, 0xF8, 0x38, 0x04, 0x41, 0x1E, 0x91, 0x42, 0x08, 0xD0, 0x00, 0xEB,
+ 0xC0, 0x00, 0x04, 0xEB, 0xC0, 0x01, 0x48, 0x46, 0x48, 0x22, 0x48, 0x39, 0x09, 0xF0, 0x5F, 0xF8,
+ 0x94, 0xF8, 0x38, 0x04, 0x40, 0x1E, 0x84, 0xF8, 0x38, 0x04, 0xBD, 0xE8, 0xF0, 0x87, 0x2D, 0xE9,
+ 0xFE, 0x4F, 0x07, 0x46, 0x5E, 0x48, 0x8A, 0x46, 0x00, 0x25, 0x01, 0x68, 0xB1, 0xF8, 0xEA, 0x02,
+ 0x00, 0xFB, 0x00, 0xF9, 0x5B, 0x48, 0x80, 0x79, 0x18, 0xB1, 0xB1, 0xF8, 0x66, 0x03, 0x00, 0xFB,
+ 0x00, 0xF9, 0x54, 0x46, 0x4F, 0xF0, 0x01, 0x0B, 0x9D, 0xE0, 0x00, 0xBF, 0x0B, 0xFA, 0x04, 0xF0,
+ 0x26, 0x46, 0x00, 0x90, 0x66, 0xE0, 0x00, 0xBF, 0x00, 0x99, 0x0B, 0xFA, 0x06, 0xF0, 0x08, 0x42,
+ 0x5F, 0xD0, 0x06, 0xEB, 0xC6, 0x00, 0x07, 0xEB, 0xC0, 0x00, 0xC2, 0x68, 0x20, 0x46, 0x4D, 0xE0,
+ 0xB0, 0x42, 0x4A, 0xD0, 0x00, 0xEB, 0xC0, 0x01, 0x07, 0xEB, 0xC1, 0x01, 0xC9, 0x68, 0x4F, 0xEA,
+ 0x21, 0x4C, 0xAC, 0xEB, 0x22, 0x4C, 0x89, 0x1A, 0x0F, 0xFA, 0x8C, 0xFC, 0x09, 0xB2, 0x6C, 0xF3,
+ 0x1F, 0x45, 0x61, 0xF3, 0x0F, 0x05, 0x8E, 0x46, 0x41, 0x49, 0xCD, 0xF8, 0x04, 0xC0, 0x09, 0x68,
+ 0x91, 0xF8, 0xE0, 0xC2, 0xCD, 0xF8, 0x08, 0xC0, 0x5F, 0xEA, 0x8C, 0x6C, 0x0B, 0xD5, 0x91, 0xF8,
+ 0xED, 0x82, 0x0E, 0xFB, 0x08, 0xFC, 0x4F, 0xF0, 0x64, 0x08, 0x9C, 0xFB, 0xF8, 0xF8, 0x08, 0xEB,
+ 0x0E, 0x0C, 0x6C, 0xF3, 0x0F, 0x05, 0xDD, 0xF8, 0x08, 0xC0, 0x5F, 0xEA, 0xCC, 0x6C, 0x0C, 0xD5,
+ 0x91, 0xF8, 0xED, 0x12, 0xDD, 0xF8, 0x04, 0xC0, 0x0C, 0xFB, 0x01, 0xFC, 0x64, 0x21, 0x9C, 0xFB,
+ 0xF1, 0xFC, 0x01, 0x99, 0x61, 0x44, 0x61, 0xF3, 0x1F, 0x45, 0x4F, 0xEA, 0x25, 0x4C, 0x0C, 0xFB,
+ 0x0C, 0xFC, 0x29, 0xB2, 0x01, 0xFB, 0x01, 0xC1, 0x49, 0x45, 0x06, 0xD2, 0xDD, 0xF8, 0x00, 0xC0,
+ 0x0B, 0xFA, 0x00, 0xF1, 0x41, 0xEA, 0x0C, 0x01, 0x00, 0x91, 0x40, 0x1C, 0x83, 0x42, 0xAF, 0xD8,
+ 0x04, 0x21, 0x68, 0x46, 0x07, 0xF0, 0xD6, 0xFC, 0x01, 0x28, 0x02, 0xD1, 0x00, 0x98, 0x40, 0x00,
+ 0x00, 0x90, 0x76, 0x1C, 0x97, 0xF8, 0x38, 0x34, 0xB3, 0x42, 0x95, 0xD8, 0x04, 0x21, 0x68, 0x46,
+ 0x07, 0xF0, 0xC8, 0xFC, 0x01, 0x28, 0x0B, 0xD0, 0x04, 0x21, 0x68, 0x46, 0x07, 0xF0, 0xC2, 0xFC,
+ 0x97, 0xF8, 0x38, 0x64, 0xA6, 0xEB, 0x0A, 0x01, 0x88, 0x42, 0x05, 0xD1, 0x87, 0xF8, 0x38, 0xA4,
+ 0x57, 0xE5, 0x64, 0x1C, 0xE4, 0xB2, 0x01, 0xE0, 0x00, 0x99, 0x58, 0x46, 0x00, 0xFA, 0x04, 0xF2,
+ 0x0A, 0x42, 0xF6, 0xD0, 0xD8, 0x46, 0x09, 0xE0, 0x00, 0x99, 0x08, 0xFA, 0x06, 0xF0, 0x08, 0x42,
+ 0x04, 0xD0, 0xF2, 0xB2, 0x21, 0x46, 0x38, 0x46, 0xFF, 0xF7, 0xEF, 0xFE, 0x76, 0x1E, 0xA6, 0x42,
+ 0xF2, 0xD8, 0x64, 0x1C, 0xE4, 0xB2, 0x97, 0xF8, 0x38, 0x04, 0xA0, 0x42, 0x3F, 0xF6, 0x5E, 0xAF,
+ 0x37, 0xE5, 0x00, 0x00, 0x48, 0x3F, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0xEC, 0x06, 0x10, 0x00, 0x2D, 0xE9, 0xF0, 0x4F, 0xFE, 0x4C, 0x01, 0x2B,
+ 0x24, 0x68, 0x94, 0xF8, 0xF9, 0x72, 0x94, 0xF8, 0xF8, 0x62, 0x05, 0xD0, 0x0B, 0x78, 0x4C, 0x78,
+ 0xFA, 0x4D, 0x9C, 0x46, 0x69, 0x0C, 0x0D, 0xE0, 0x8B, 0x78, 0xCC, 0x78, 0xF8, 0xE7, 0x00, 0xBF,
+ 0x30, 0xF9, 0x1C, 0x80, 0xA8, 0x45, 0x00, 0xDD, 0x45, 0x46, 0x88, 0x45, 0x00, 0xDA, 0x41, 0x46,
+ 0x0C, 0xF1, 0x01, 0x0C, 0xA4, 0x45, 0xF3, 0xD9, 0xB5, 0x42, 0x00, 0xDA, 0x35, 0x46, 0xB1, 0x42,
+ 0x00, 0xDA, 0x31, 0x46, 0x69, 0x1A, 0x79, 0x43, 0x64, 0x25, 0x91, 0xFB, 0xF5, 0xF1, 0x31, 0x44,
+ 0x0E, 0xB2, 0x30, 0xF9, 0x13, 0x50, 0x01, 0x21, 0xB5, 0x42, 0x05, 0xDD, 0x15, 0x68, 0x01, 0xFA,
+ 0x03, 0xFC, 0x45, 0xEA, 0x0C, 0x05, 0x15, 0x60, 0x30, 0xF9, 0x14, 0x50, 0xB5, 0x42, 0x05, 0xDD,
+ 0x15, 0x68, 0x01, 0xFA, 0x04, 0xFC, 0x45, 0xEA, 0x0C, 0x05, 0x15, 0x60, 0xE5, 0x1A, 0x05, 0xF0,
+ 0xFF, 0x09, 0xB9, 0xF1, 0x01, 0x0F, 0x37, 0xD9, 0x5D, 0x1C, 0xED, 0xB2, 0x30, 0xF9, 0x15, 0x80,
+ 0x08, 0xEB, 0x07, 0x0C, 0x0F, 0xFA, 0x8C, 0xFC, 0xB0, 0x45, 0x11, 0xDD, 0x00, 0xEB, 0x45, 0x0A,
+ 0x3A, 0xF9, 0x02, 0x8C, 0xE0, 0x45, 0x03, 0xDD, 0xBA, 0xF9, 0x02, 0x80, 0xE0, 0x45, 0x07, 0xDC,
+ 0xD2, 0xF8, 0x00, 0xC0, 0x01, 0xFA, 0x05, 0xF5, 0x4C, 0xEA, 0x05, 0x0C, 0xC2, 0xF8, 0x00, 0xC0,
+ 0x65, 0x1E, 0xED, 0xB2, 0x30, 0xF9, 0x15, 0x80, 0x08, 0xEB, 0x07, 0x0C, 0x0F, 0xFA, 0x8C, 0xFC,
+ 0xB0, 0x45, 0x11, 0xDD, 0x00, 0xEB, 0x45, 0x0A, 0x3A, 0xF9, 0x02, 0x8C, 0xE0, 0x45, 0x03, 0xDD,
+ 0xBA, 0xF9, 0x02, 0x80, 0xE0, 0x45, 0x07, 0xDC, 0xD2, 0xF8, 0x00, 0xC0, 0x01, 0xFA, 0x05, 0xF5,
+ 0x4C, 0xEA, 0x05, 0x0C, 0xC2, 0xF8, 0x00, 0xC0, 0xB9, 0xF1, 0x02, 0x0F, 0x2E, 0xD9, 0x9D, 0x1C,
+ 0xA4, 0xF1, 0x02, 0x09, 0x28, 0xE0, 0x00, 0xBF, 0x30, 0xF9, 0x15, 0xC0, 0xB4, 0x45, 0x22, 0xDD,
+ 0x00, 0xEB, 0x45, 0x0B, 0xBC, 0x44, 0x3B, 0xF9, 0x02, 0xEC, 0x0F, 0xFA, 0x8C, 0xFA, 0xD6, 0x45,
+ 0x03, 0xDD, 0xBB, 0xF9, 0x02, 0x80, 0xD0, 0x45, 0x15, 0xDC, 0x3B, 0xF9, 0x04, 0x8C, 0xD0, 0x45,
+ 0x03, 0xDD, 0xBB, 0xF9, 0x02, 0x80, 0xD0, 0x45, 0x0D, 0xDC, 0xD6, 0x45, 0x03, 0xDD, 0xBB, 0xF9,
+ 0x04, 0x80, 0xD0, 0x45, 0x07, 0xDC, 0xD2, 0xF8, 0x00, 0xC0, 0x01, 0xFA, 0x05, 0xF8, 0x4C, 0xEA,
+ 0x08, 0x0C, 0xC2, 0xF8, 0x00, 0xC0, 0x6D, 0x1C, 0x4D, 0x45, 0xD5, 0xD9, 0x00, 0x25, 0x28, 0x46,
+ 0x0C, 0xE0, 0x00, 0xBF, 0x16, 0x68, 0x01, 0xFA, 0x03, 0xF7, 0x3E, 0x42, 0x04, 0xD0, 0x25, 0xB9,
+ 0x01, 0x25, 0x40, 0x1C, 0xC0, 0xB2, 0x00, 0xE0, 0x00, 0x25, 0x5B, 0x1C, 0xA3, 0x42, 0xF1, 0xD9,
+ 0xBD, 0xE8, 0xF0, 0x8F, 0x2D, 0xE9, 0xFF, 0x4F, 0x9D, 0x48, 0x8F, 0xB0, 0xDF, 0xF8, 0x74, 0x92,
+ 0x04, 0x68, 0x16, 0x46, 0x04, 0xF5, 0x88, 0x70, 0x02, 0x90, 0x4F, 0xF0, 0xFF, 0x30, 0x0C, 0x90,
+ 0x00, 0x20, 0x0A, 0x90, 0x0B, 0x90, 0x98, 0x46, 0x01, 0x22, 0x31, 0x46, 0x4D, 0x46, 0xD9, 0xF8,
+ 0x00, 0x00, 0xFE, 0xF7, 0xFF, 0xFD, 0xCD, 0xF8, 0x00, 0x80, 0xDD, 0xE9, 0x0F, 0x12, 0x33, 0x46,
+ 0x28, 0x68, 0xFF, 0xF7, 0x48, 0xFC, 0x05, 0x00, 0x08, 0xD0, 0x8F, 0x49, 0x01, 0x20, 0x08, 0x70,
+ 0xB8, 0xF8, 0x24, 0x00, 0x40, 0xF0, 0x80, 0x01, 0xA8, 0xF8, 0x24, 0x10, 0x8B, 0x48, 0x00, 0x78,
+ 0x50, 0xB1, 0x4D, 0xB9, 0x98, 0xF8, 0x24, 0x00, 0x10, 0xF0, 0x06, 0x0F, 0x04, 0xD1, 0x31, 0x46,
+ 0x40, 0x46, 0xFE, 0xF7, 0x3D, 0xFE, 0x05, 0x46, 0x85, 0x48, 0x00, 0x78, 0x00, 0xB1, 0x01, 0x25,
+ 0x7D, 0x48, 0x00, 0x68, 0x90, 0xF8, 0xE1, 0x02, 0x40, 0x07, 0x2F, 0xD5, 0x01, 0x23, 0x0A, 0xAA,
+ 0x31, 0x46, 0x80, 0x48, 0xFF, 0xF7, 0xF0, 0xFE, 0x07, 0x46, 0x00, 0x23, 0x0B, 0xAA, 0x31, 0x46,
+ 0x7D, 0x48, 0xFF, 0xF7, 0xE9, 0xFE, 0x78, 0x43, 0x01, 0x28, 0x1F, 0xD1, 0xB7, 0x78, 0x1A, 0xE0,
+ 0xF9, 0xB2, 0x0F, 0x98, 0x08, 0xF0, 0xC4, 0xFF, 0x83, 0x46, 0xF9, 0xB2, 0xD9, 0xF8, 0x00, 0x00,
+ 0x08, 0xF0, 0xBE, 0xFF, 0x02, 0x46, 0x30, 0x78, 0x01, 0x23, 0x08, 0xE0, 0x1B, 0xF8, 0x00, 0xC0,
+ 0x10, 0x99, 0x8C, 0x45, 0x02, 0xD1, 0x11, 0x5C, 0x01, 0xB9, 0x13, 0x54, 0x40, 0x1C, 0x71, 0x78,
+ 0x81, 0x42, 0xF3, 0xDA, 0x7F, 0x1C, 0xF0, 0x78, 0xB8, 0x42, 0xE1, 0xDA, 0x00, 0x20, 0x84, 0xF8,
+ 0x0E, 0x01, 0x30, 0x68, 0x20, 0x60, 0x63, 0x48, 0x2B, 0x46, 0x00, 0x22, 0x21, 0x46, 0x00, 0x68,
+ 0xFE, 0xF7, 0x9A, 0xFE, 0x94, 0xF8, 0x0E, 0x01, 0x00, 0x25, 0x01, 0x28, 0x70, 0xD9, 0xFC, 0xF7,
+ 0xB4, 0xFB, 0x83, 0x46, 0xFC, 0xF7, 0xAF, 0xFB, 0x82, 0x46, 0x32, 0x68, 0xDD, 0xE9, 0x0F, 0x01,
+ 0x07, 0x92, 0x47, 0x46, 0x5D, 0x4B, 0x07, 0xAA, 0xFE, 0xF7, 0x3B, 0xFD, 0x60, 0xB1, 0x07, 0xA9,
+ 0x5A, 0x48, 0xFE, 0xF7, 0xCF, 0xFC, 0x3A, 0x46, 0x07, 0xA9, 0x58, 0x48, 0xFD, 0xF7, 0x1F, 0xFB,
+ 0xF8, 0x68, 0x78, 0x61, 0xF8, 0x69, 0x38, 0x62, 0x51, 0x48, 0x00, 0x78, 0xE8, 0xB3, 0x4A, 0x48,
+ 0x00, 0x68, 0x90, 0xF8, 0xE1, 0x02, 0x80, 0x07, 0x46, 0xD5, 0x94, 0xF8, 0x0E, 0x01, 0x01, 0x28,
+ 0x42, 0xD9, 0x4F, 0xF0, 0x00, 0x09, 0x01, 0x27, 0x10, 0xE0, 0x00, 0xBF, 0x45, 0x48, 0x04, 0xEB,
+ 0x87, 0x03, 0xFA, 0xB2, 0xD8, 0xF8, 0x08, 0x10, 0x00, 0x68, 0xFE, 0xF7, 0x89, 0xFD, 0x48, 0x49,
+ 0x48, 0x45, 0x41, 0xF8, 0x27, 0x00, 0x00, 0xDD, 0x81, 0x46, 0x7F, 0x1C, 0x94, 0xF8, 0x0E, 0x21,
+ 0xBA, 0x42, 0xEB, 0xD2, 0xB9, 0xF1, 0x00, 0x0F, 0x1F, 0xDD, 0x37, 0x49, 0x01, 0x20, 0x40, 0x4B,
+ 0x86, 0x46, 0x09, 0x68, 0x17, 0xE0, 0x00, 0xBF, 0x53, 0xF8, 0x20, 0x70, 0x4F, 0xF0, 0x64, 0x0C,
+ 0x07, 0xFB, 0x0C, 0xF7, 0x97, 0xFB, 0xF9, 0xF7, 0x91, 0xF8, 0xF2, 0xC2, 0xBC, 0x45, 0x09, 0xDD,
+ 0x0E, 0xFA, 0x00, 0xF7, 0xDD, 0xF8, 0x30, 0xC0, 0x00, 0xE0, 0x0D, 0xE0, 0x2C, 0xEA, 0x07, 0x0C,
+ 0xCD, 0xF8, 0x30, 0xC0, 0x40, 0x1C, 0x82, 0x42, 0xE6, 0xD2, 0x94, 0xF8, 0x0E, 0x01, 0x01, 0x28,
+ 0x02, 0xD9, 0x00, 0x20, 0x84, 0xF8, 0x0E, 0x01, 0x01, 0x27, 0xDF, 0xF8, 0x8C, 0x90, 0x35, 0xE0,
+ 0x7D, 0xE0, 0x48, 0x22, 0x41, 0x46, 0x02, 0x98, 0x08, 0xF0, 0xD1, 0xFD, 0xD9, 0xF8, 0x00, 0x10,
+ 0x04, 0xEB, 0x47, 0x00, 0xB0, 0xF8, 0xB4, 0x20, 0x91, 0xF8, 0xEC, 0x12, 0x8A, 0x42, 0x24, 0xD9,
+ 0x01, 0x21, 0x0C, 0x9A, 0xB9, 0x40, 0x11, 0x42, 0x1F, 0xD0, 0x02, 0x99, 0x89, 0x8C, 0x41, 0xF0,
+ 0x10, 0x02, 0x02, 0x99, 0x8A, 0x84, 0xB0, 0xF8, 0xB4, 0x10, 0x02, 0x98, 0xC1, 0x84, 0x04, 0xEB,
+ 0x87, 0x00, 0x08, 0x90, 0x02, 0x99, 0xFE, 0xF7, 0xD3, 0xFF, 0x12, 0x48, 0xF9, 0xB2, 0x02, 0x9B,
+ 0x08, 0x9A, 0x00, 0x68, 0xFD, 0xF7, 0x63, 0xFE, 0x0E, 0x48, 0xF9, 0xB2, 0x02, 0x9B, 0x08, 0x9A,
+ 0x00, 0x68, 0xFD, 0xF7, 0xD9, 0xFD, 0x6D, 0x1C, 0xED, 0xB2, 0x7F, 0x1C, 0x94, 0xF8, 0x0E, 0x01,
+ 0xB8, 0x42, 0xC6, 0xD2, 0xD9, 0xF8, 0x00, 0x00, 0x4C, 0x46, 0x90, 0xF8, 0xE0, 0x02, 0x00, 0x07,
+ 0x21, 0xD5, 0x01, 0x2D, 0x1F, 0xD9, 0x15, 0xE0, 0x20, 0x07, 0x10, 0x00, 0x00, 0x80, 0xFF, 0xFF,
+ 0x48, 0x06, 0x10, 0x00, 0x30, 0x06, 0x10, 0x00, 0x10, 0x06, 0x10, 0x00, 0x80, 0x06, 0x10, 0x00,
+ 0x28, 0x05, 0x10, 0x00, 0x36, 0x0B, 0x01, 0x20, 0x60, 0x0B, 0x01, 0x20, 0xA0, 0x3E, 0x10, 0x00,
+ 0xA8, 0x40, 0x10, 0x00, 0x59, 0x46, 0x50, 0x46, 0xFF, 0xF7, 0x21, 0xFD, 0xFC, 0xF7, 0xE5, 0xFA,
+ 0xA0, 0xEB, 0x0B, 0x00, 0xC5, 0xB2, 0x22, 0x68, 0x92, 0xF8, 0xE0, 0x02, 0x00, 0x06, 0x12, 0xD5,
+ 0x28, 0x46, 0x0E, 0xE0, 0x0B, 0xEB, 0x00, 0x01, 0xC9, 0xB2, 0x01, 0xEB, 0xC1, 0x03, 0x0A, 0xEB,
+ 0xC3, 0x01, 0x92, 0xF8, 0x07, 0x33, 0xC9, 0x8C, 0x99, 0x42, 0x02, 0xD9, 0x8A, 0xF8, 0x38, 0xB4,
+ 0x05, 0xE0, 0x40, 0x1E, 0xEE, 0xD2, 0x01, 0x2D, 0x09, 0xD0, 0x00, 0x2D, 0x05, 0xD1, 0xDD, 0xE9,
+ 0x0F, 0x01, 0x43, 0x46, 0x32, 0x46, 0xFD, 0xF7, 0x87, 0xFD, 0x13, 0xB0, 0x80, 0xE6, 0xE0, 0x48,
+ 0x00, 0x78, 0x00, 0x28, 0xF9, 0xD1, 0x9A, 0xF8, 0x38, 0x04, 0x40, 0x1E, 0x8A, 0xF8, 0x38, 0x04,
+ 0xED, 0xE7, 0x2D, 0xE9, 0xFF, 0x4F, 0xDB, 0x48, 0x81, 0xB0, 0x00, 0x68, 0x90, 0xF8, 0xE1, 0x02,
+ 0xC0, 0x07, 0x67, 0xD0, 0xA8, 0x21, 0x04, 0x98, 0x08, 0xF0, 0x95, 0xFD, 0x01, 0x25, 0x44, 0xE0,
+ 0x03, 0x98, 0x00, 0xEB, 0x85, 0x00, 0x82, 0x46, 0x84, 0x78, 0x90, 0xF8, 0x03, 0x90, 0xD2, 0x48,
+ 0x00, 0x78, 0x40, 0x1E, 0x84, 0x42, 0x01, 0xDA, 0x64, 0x1C, 0xE4, 0xB2, 0xB9, 0xF1, 0x00, 0x0F,
+ 0x03, 0xD0, 0xA9, 0xF1, 0x01, 0x00, 0x00, 0xF0, 0xFF, 0x09, 0x21, 0x46, 0x01, 0x98, 0x08, 0xF0,
+ 0x61, 0xFE, 0x83, 0x46, 0xA0, 0x46, 0x24, 0xE0, 0x03, 0x98, 0x04, 0x99, 0x10, 0xF8, 0x25, 0x40,
+ 0x01, 0xEB, 0xC8, 0x00, 0x00, 0x90, 0x12, 0xE0, 0x1B, 0xF8, 0x04, 0x00, 0xA8, 0x42, 0x0C, 0xD1,
+ 0x00, 0x98, 0x22, 0x46, 0x00, 0x21, 0xD0, 0xE9, 0x00, 0x67, 0x01, 0x20, 0x08, 0xF0, 0x8B, 0xFC,
+ 0x06, 0x43, 0x00, 0x98, 0x0F, 0x43, 0xC0, 0xE9, 0x00, 0x67, 0x64, 0x1C, 0xE4, 0xB2, 0x9A, 0xF8,
+ 0x01, 0x00, 0xA0, 0x42, 0xE8, 0xD2, 0x08, 0xF1, 0x01, 0x00, 0x0B, 0xF1, 0x28, 0x0B, 0x00, 0xF0,
+ 0xFF, 0x08, 0xC8, 0x45, 0xD8, 0xD9, 0x6D, 0x1C, 0xED, 0xB2, 0x02, 0x98, 0x85, 0x42, 0xB7, 0xD9,
+ 0x00, 0x21, 0xB1, 0x4E, 0x13, 0xE0, 0x04, 0x98, 0x00, 0xEB, 0xC1, 0x05, 0x01, 0xF1, 0x01, 0x01,
+ 0xD5, 0xE9, 0x00, 0x32, 0xDC, 0x18, 0x42, 0xEB, 0x02, 0x00, 0x52, 0x08, 0x4F, 0xEA, 0x33, 0x03,
+ 0x04, 0xEA, 0x03, 0x04, 0x00, 0xEA, 0x02, 0x00, 0xC5, 0xE9, 0x00, 0x40, 0xC9, 0xB2, 0x30, 0x78,
+ 0x81, 0x42, 0xE8, 0xD3, 0x05, 0xB0, 0x03, 0xE6, 0xF0, 0xB4, 0x14, 0x68, 0xC3, 0xF8, 0x3A, 0x40,
+ 0xA0, 0x4C, 0x25, 0x68, 0x95, 0xF8, 0xE0, 0x42, 0xA6, 0x07, 0x13, 0xD4, 0x93, 0xF8, 0x40, 0x60,
+ 0x01, 0x2E, 0x0F, 0xD0, 0x66, 0x07, 0x13, 0xD5, 0x64, 0x06, 0x09, 0xD5, 0x9B, 0x4C, 0x64, 0x78,
+ 0x5C, 0xB1, 0xB5, 0xF8, 0xF0, 0x42, 0x9A, 0x4D, 0x64, 0x43, 0x2D, 0x68, 0xA5, 0x42, 0x07, 0xDC,
+ 0xF0, 0xBC, 0xE7, 0xE5, 0xF0, 0xBC, 0xFF, 0xF7, 0x19, 0xB9, 0xB5, 0xF8, 0xEE, 0x42, 0xF2, 0xE7,
+ 0xF0, 0xBC, 0xFD, 0xF7, 0xE1, 0xBC, 0x2D, 0xE9, 0xFF, 0x4F, 0x94, 0x78, 0xD7, 0x78, 0x15, 0x78,
+ 0x56, 0x78, 0x81, 0xB0, 0x98, 0x46, 0x82, 0x46, 0x04, 0xB9, 0x01, 0x24, 0x8A, 0x48, 0x00, 0x78,
+ 0x40, 0x1E, 0x87, 0x42, 0x01, 0xD1, 0x7F, 0x1E, 0xFF, 0xB2, 0x05, 0xB9, 0x01, 0x25, 0xDF, 0xF8,
+ 0x24, 0xB2, 0x9B, 0xF8, 0x00, 0x00, 0x40, 0x1E, 0x86, 0x42, 0x01, 0xD1, 0x76, 0x1E, 0xF6, 0xB2,
+ 0x60, 0x1E, 0xC1, 0xB2, 0xD8, 0xF8, 0x08, 0x00, 0x08, 0xF0, 0xD2, 0xFD, 0x00, 0x90, 0x21, 0x46,
+ 0xD8, 0xF8, 0x08, 0x00, 0x08, 0xF0, 0xCC, 0xFD, 0x81, 0x46, 0x60, 0x1C, 0xC1, 0xB2, 0xD8, 0xF8,
+ 0x08, 0x00, 0x08, 0xF0, 0xC5, 0xFD, 0x80, 0x46, 0x21, 0x46, 0x50, 0x46, 0x08, 0xF0, 0xBA, 0xFD,
+ 0x35, 0xE0, 0x74, 0x4A, 0x29, 0x46, 0xD2, 0xF8, 0x00, 0xA0, 0x24, 0xE0, 0x43, 0x5C, 0x02, 0x9A,
+ 0x93, 0x42, 0x1E, 0xD1, 0x39, 0xF8, 0x11, 0x30, 0x9A, 0xF8, 0x27, 0x24, 0x1A, 0x44, 0x00, 0x9B,
+ 0x12, 0xB2, 0x03, 0xEB, 0x41, 0x03, 0x33, 0xF9, 0x02, 0xCC, 0x94, 0x45, 0x05, 0xDD, 0x08, 0xEB,
+ 0x41, 0x0C, 0xBC, 0xF9, 0x02, 0xC0, 0x94, 0x45, 0x09, 0xDC, 0xB3, 0xF9, 0x02, 0x30, 0x93, 0x42,
+ 0x07, 0xDD, 0x08, 0xEB, 0x41, 0x03, 0x33, 0xF9, 0x02, 0x3C, 0x93, 0x42, 0x01, 0xDD, 0x01, 0x20,
+ 0x78, 0xE7, 0x49, 0x1C, 0xC9, 0xB2, 0xB1, 0x42, 0xD8, 0xD9, 0xCD, 0xF8, 0x00, 0x90, 0x9B, 0xF8,
+ 0x00, 0x20, 0x64, 0x1C, 0xC1, 0x46, 0x08, 0xEB, 0x42, 0x08, 0xE4, 0xB2, 0x28, 0x30, 0xBC, 0x42,
+ 0xC7, 0xD9, 0x00, 0x20, 0x66, 0xE7, 0x2D, 0xE9, 0xF8, 0x4F, 0x4F, 0xF0, 0x00, 0x0A, 0x89, 0x46,
+ 0x04, 0x46, 0x01, 0x26, 0x57, 0x46, 0xD0, 0x46, 0x85, 0x78, 0x2A, 0xE0, 0x29, 0x46, 0xD9, 0xF8,
+ 0x08, 0x00, 0x08, 0xF0, 0x75, 0xFD, 0x02, 0x46, 0x00, 0x20, 0x21, 0x78, 0x63, 0x78, 0x04, 0xE0,
+ 0x32, 0xF9, 0x11, 0xC0, 0x49, 0x1C, 0x60, 0x44, 0xC9, 0xB2, 0x8B, 0x42, 0xF8, 0xD2, 0xA1, 0x78,
+ 0x4A, 0x1C, 0xAA, 0x42, 0x10, 0xD2, 0x47, 0x4A, 0x12, 0x68, 0xB2, 0xF8, 0x28, 0x34, 0x03, 0xEB,
+ 0x08, 0x02, 0xBA, 0x42, 0x01, 0xDA, 0x82, 0x42, 0x33, 0xDB, 0x89, 0x1C, 0xA9, 0x42, 0x03, 0xD2,
+ 0x52, 0x45, 0x01, 0xDA, 0x82, 0x42, 0x2C, 0xDB, 0xBA, 0x46, 0x47, 0x46, 0x6D, 0x1C, 0x80, 0x46,
+ 0xED, 0xB2, 0xE0, 0x78, 0xA8, 0x42, 0xD1, 0xD2, 0x00, 0x20, 0x00, 0x90, 0x82, 0x46, 0x83, 0x46,
+ 0x25, 0x78, 0x2E, 0xE0, 0x4F, 0xF0, 0x00, 0x08, 0xA7, 0x78, 0x09, 0xE0, 0x39, 0x46, 0xD9, 0xF8,
+ 0x08, 0x00, 0x08, 0xF0, 0x3D, 0xFD, 0x30, 0xF9, 0x15, 0x10, 0x7F, 0x1C, 0x88, 0x44, 0xFF, 0xB2,
+ 0xE0, 0x78, 0xB8, 0x42, 0xF2, 0xD2, 0x20, 0x78, 0x41, 0x1C, 0xA9, 0x42, 0x13, 0xD2, 0x2D, 0x49,
+ 0x09, 0x68, 0xB1, 0xF8, 0x28, 0x24, 0x02, 0xEB, 0x0B, 0x01, 0x51, 0x45, 0x03, 0xDA, 0x41, 0x45,
+ 0x01, 0xDA, 0x00, 0x26, 0x10, 0xE0, 0x80, 0x1C, 0xA8, 0x42, 0x04, 0xD2, 0x00, 0x98, 0x81, 0x42,
+ 0x01, 0xDA, 0x41, 0x45, 0xF5, 0xDB, 0xCD, 0xF8, 0x00, 0xA0, 0xDA, 0x46, 0x6D, 0x1C, 0xC3, 0x46,
+ 0xED, 0xB2, 0x60, 0x78, 0xA8, 0x42, 0xCD, 0xD2, 0x30, 0x46, 0xBD, 0xE8, 0xF8, 0x8F, 0x2D, 0xE9,
+ 0xF0, 0x41, 0x07, 0x46, 0x98, 0x8C, 0x14, 0x46, 0x00, 0x26, 0x1D, 0x46, 0xC2, 0x05, 0x02, 0xD5,
+ 0x01, 0x20, 0xBD, 0xE8, 0xF0, 0x81, 0x00, 0x06, 0x27, 0xD4, 0x16, 0x48, 0xEA, 0x8C, 0x00, 0x68,
+ 0x90, 0xF8, 0x23, 0x34, 0x9A, 0x42, 0x20, 0xD3, 0xB0, 0xF8, 0x24, 0x34, 0xAA, 0x69, 0x9A, 0x42,
+ 0x1B, 0xDC, 0x62, 0x78, 0x23, 0x78, 0x90, 0xF8, 0x26, 0x04, 0xD2, 0x1A, 0x52, 0x1C, 0x82, 0x42,
+ 0x13, 0xDC, 0xE2, 0x78, 0xA3, 0x78, 0xD2, 0x1A, 0x52, 0x1C, 0x82, 0x42, 0x0D, 0xDC, 0x2B, 0x46,
+ 0x22, 0x46, 0x38, 0x46, 0xFF, 0xF7, 0xEF, 0xFE, 0x01, 0x28, 0x06, 0xD1, 0x29, 0x46, 0x20, 0x46,
+ 0xFF, 0xF7, 0x59, 0xFF, 0x01, 0x28, 0x00, 0xD1, 0x01, 0x26, 0x30, 0x46, 0xD1, 0xE7, 0x00, 0x00,
+ 0x28, 0x05, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0xB9, 0x06, 0x10, 0x00, 0x4C, 0x07, 0x10, 0x00,
+ 0xF4, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0x70, 0x47, 0xF0, 0xB5, 0x68, 0x4B, 0x02, 0x46,
+ 0x0C, 0x46, 0x1F, 0x78, 0x67, 0x4B, 0x1E, 0x68, 0x67, 0x4B, 0x1D, 0x78, 0x19, 0xE0, 0x2B, 0x46,
+ 0x11, 0xE0, 0x00, 0xBF, 0x34, 0xF8, 0x13, 0xC0, 0x32, 0xF8, 0x13, 0xE0, 0xAC, 0xEB, 0x0E, 0x0C,
+ 0xB6, 0xF9, 0x46, 0xE0, 0x0F, 0xFA, 0x8C, 0xFC, 0x1E, 0xEB, 0x0C, 0x0F, 0x22, 0xD4, 0xB6, 0xF8,
+ 0x48, 0xE0, 0xE6, 0x45, 0x1E, 0xDB, 0x5B, 0x1E, 0xEC, 0xD1, 0x02, 0xEB, 0x45, 0x02, 0x04, 0xEB,
+ 0x45, 0x04, 0x7F, 0x1E, 0xE3, 0xD2, 0x96, 0xF8, 0x44, 0x20, 0x58, 0x4B, 0x02, 0xF0, 0x0F, 0x02,
+ 0xC2, 0xF1, 0x10, 0x04, 0xB3, 0xF9, 0x00, 0x30, 0x0A, 0xE0, 0x00, 0xBF, 0x31, 0xF9, 0x13, 0x50,
+ 0x30, 0xF9, 0x13, 0x60, 0x55, 0x43, 0x06, 0xFB, 0x04, 0x55, 0x2D, 0x11, 0x21, 0xF8, 0x13, 0x50,
+ 0x5B, 0x1E, 0xF3, 0xD2, 0xF0, 0xBD, 0xF0, 0xB5, 0x14, 0x46, 0x05, 0x9F, 0x0B, 0xE0, 0x00, 0xBF,
+ 0x31, 0xF9, 0x14, 0x50, 0x30, 0xF9, 0x14, 0x60, 0x05, 0xEB, 0x03, 0x0C, 0xB4, 0x45, 0x19, 0xDB,
+ 0xED, 0x1B, 0xB5, 0x42, 0x16, 0xDC, 0x64, 0x1E, 0xF2, 0xD1, 0x42, 0x4B, 0x1B, 0x68, 0x93, 0xF8,
+ 0x45, 0x30, 0x1B, 0x09, 0xC3, 0xF1, 0x10, 0x04, 0x0A, 0xE0, 0x00, 0xBF, 0x31, 0xF9, 0x12, 0x50,
+ 0x30, 0xF9, 0x12, 0x60, 0x5D, 0x43, 0x06, 0xFB, 0x04, 0x55, 0x2D, 0x11, 0x21, 0xF8, 0x12, 0x50,
+ 0x52, 0x1E, 0xF3, 0xD2, 0xF0, 0xBD, 0x38, 0xB5, 0x0D, 0x46, 0x04, 0x46, 0x00, 0xF0, 0x6D, 0xFF,
+ 0x00, 0x28, 0x25, 0xD1, 0xE8, 0x07, 0x08, 0xD0, 0x35, 0x48, 0x36, 0x49, 0x00, 0x78, 0x09, 0x68,
+ 0x01, 0x28, 0x20, 0x68, 0x1D, 0xD0, 0x00, 0xF0, 0x1F, 0xF8, 0xE8, 0x06, 0x18, 0xD5, 0x2D, 0x4D,
+ 0x28, 0x68, 0xB0, 0xF9, 0x50, 0x10, 0x00, 0x91, 0x29, 0x49, 0xB0, 0xF9, 0x4E, 0x30, 0x20, 0x69,
+ 0x0A, 0x78, 0x2D, 0x49, 0xFF, 0xF7, 0xB7, 0xFF, 0x28, 0x68, 0xB0, 0xF9, 0x54, 0x10, 0x00, 0x91,
+ 0x25, 0x49, 0xB0, 0xF9, 0x52, 0x30, 0x60, 0x69, 0x0A, 0x78, 0x28, 0x49, 0xFF, 0xF7, 0xAB, 0xFF,
+ 0x38, 0xBD, 0xFF, 0xF7, 0x6A, 0xFF, 0xE0, 0xE7, 0x2D, 0xE9, 0xF0, 0x43, 0x81, 0x46, 0x24, 0x48,
+ 0x03, 0x78, 0x1C, 0x48, 0x05, 0x2B, 0x00, 0x68, 0x90, 0xF8, 0x44, 0x20, 0x0D, 0xD0, 0x30, 0xF9,
+ 0x46, 0x5F, 0x02, 0xF0, 0x0F, 0x03, 0xB0, 0xF9, 0x02, 0x60, 0x00, 0xBF, 0x17, 0x48, 0xC3, 0xF1,
+ 0x10, 0x04, 0xB0, 0xF9, 0x00, 0x00, 0x02, 0x46, 0x11, 0xE0, 0x30, 0xF9, 0x4A, 0x5F, 0x13, 0x09,
+ 0xB0, 0xF9, 0x02, 0x60, 0xF2, 0xE7, 0x00, 0xBF, 0x31, 0xF9, 0x12, 0x70, 0x39, 0xF9, 0x12, 0xC0,
+ 0x07, 0xEB, 0x05, 0x08, 0xE0, 0x45, 0x11, 0xDB, 0xBF, 0x1B, 0x67, 0x45, 0x0E, 0xDC, 0x52, 0x1E,
+ 0xF2, 0xD2, 0x09, 0xE0, 0x31, 0xF9, 0x10, 0x20, 0x39, 0xF9, 0x10, 0x50, 0x5A, 0x43, 0x05, 0xFB,
+ 0x04, 0x22, 0x12, 0x11, 0x21, 0xF8, 0x10, 0x20, 0x40, 0x1E, 0xF3, 0xD2, 0xBD, 0xE8, 0xF0, 0x83,
+ 0xB9, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0xCA, 0x06, 0x10, 0x00,
+ 0xBF, 0x06, 0x10, 0x00, 0xE8, 0x06, 0x10, 0x00, 0x6A, 0x4A, 0x01, 0x20, 0x94, 0x4A, 0x01, 0x20,
+ 0xB8, 0x06, 0x10, 0x00, 0x2D, 0xE9, 0xFC, 0x47, 0x81, 0x46, 0xB9, 0x48, 0xB9, 0x4E, 0x88, 0x46,
+ 0x01, 0x78, 0x30, 0x68, 0x01, 0x25, 0x11, 0xB9, 0xB7, 0x49, 0x09, 0x78, 0x11, 0xB1, 0xB0, 0xF9,
+ 0xA1, 0x42, 0x0D, 0xE0, 0xB5, 0x49, 0xB0, 0xF9, 0x9B, 0x42, 0x49, 0x78, 0x01, 0x29, 0x02, 0xD0,
+ 0x02, 0x29, 0x03, 0xD0, 0x04, 0xE0, 0xB0, 0xF9, 0x9D, 0x42, 0x01, 0xE0, 0xB0, 0xF9, 0x9F, 0x42,
+ 0x02, 0x21, 0x01, 0xA8, 0x00, 0xF0, 0xBE, 0xFA, 0xAD, 0x4F, 0x42, 0x46, 0x49, 0x46, 0xB7, 0xF9,
+ 0x00, 0x30, 0x01, 0x98, 0x03, 0xF0, 0xE1, 0xFD, 0x31, 0x68, 0x91, 0xF8, 0xAA, 0x02, 0x02, 0x07,
+ 0x05, 0xD4, 0xC0, 0x06, 0x03, 0xD4, 0x91, 0xF8, 0xAB, 0x02, 0xC0, 0x07, 0x08, 0xD0, 0xB7, 0xF9,
+ 0x00, 0x00, 0x01, 0x99, 0x00, 0x90, 0x80, 0x23, 0xA2, 0x4A, 0x08, 0x46, 0x03, 0xF0, 0xDA, 0xFD,
+ 0xB7, 0xF9, 0x00, 0x10, 0x01, 0x9A, 0x0C, 0xE0, 0x32, 0xF9, 0x11, 0x00, 0x00, 0x28, 0x01, 0xDB,
+ 0x03, 0x46, 0x00, 0xE0, 0x43, 0x42, 0xA3, 0x42, 0x03, 0xDD, 0x9B, 0x49, 0x00, 0x25, 0x08, 0x80,
+ 0x01, 0xE0, 0x49, 0x1E, 0xF0, 0xD2, 0x02, 0x21, 0x01, 0xA8, 0x00, 0xF0, 0xEE, 0xFA, 0x28, 0x46,
+ 0xBD, 0xE8, 0xFC, 0x87, 0x8F, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x90, 0x02, 0x40, 0x07, 0x07, 0xD5,
+ 0x8E, 0x48, 0x92, 0x49, 0xC0, 0x78, 0x09, 0x78, 0x88, 0x42, 0x01, 0xD0, 0x01, 0x20, 0x70, 0x47,
+ 0x00, 0x20, 0x70, 0x47, 0x87, 0x48, 0x10, 0xB5, 0x00, 0x68, 0x90, 0xF8, 0x90, 0x12, 0x89, 0x07,
+ 0x27, 0xD5, 0x8B, 0x49, 0x0A, 0x78, 0x52, 0x1C, 0xD2, 0xB2, 0x0A, 0x70, 0x90, 0xF8, 0x92, 0x32,
+ 0x93, 0x42, 0x1E, 0xD2, 0x00, 0x22, 0x0A, 0x70, 0x90, 0xF8, 0x50, 0x12, 0x03, 0x24, 0xC9, 0x07,
+ 0x03, 0xD1, 0x90, 0xF8, 0x80, 0x02, 0xC0, 0x07, 0x00, 0xD0, 0x0B, 0x24, 0xF1, 0xF7, 0xC6, 0xFF,
+ 0x80, 0x48, 0x01, 0x68, 0x21, 0x43, 0x01, 0x60, 0xF1, 0xF7, 0xBE, 0xFF, 0x10, 0x21, 0x05, 0x20,
+ 0x02, 0xF0, 0x40, 0xFA, 0xBD, 0xE8, 0x10, 0x40, 0x4F, 0xF4, 0x80, 0x71, 0x00, 0x20, 0x06, 0xF0,
+ 0x32, 0xB9, 0x10, 0xBD, 0x6E, 0x49, 0x70, 0x4A, 0x70, 0x48, 0x09, 0x78, 0x12, 0x78, 0x43, 0x78,
+ 0x11, 0x43, 0x0B, 0x43, 0x72, 0x49, 0x4F, 0xF0, 0x00, 0x02, 0x08, 0xD1, 0x4B, 0x78, 0x5B, 0x1C,
+ 0xDB, 0xB2, 0x4B, 0x70, 0x05, 0x2B, 0x03, 0xD9, 0x01, 0x23, 0x43, 0x70, 0x83, 0x70, 0x4A, 0x70,
+ 0x70, 0x47, 0x2D, 0xE9, 0xF3, 0x4F, 0x6C, 0x48, 0x00, 0x27, 0xDF, 0xF8, 0x94, 0xA1, 0x90, 0xF8,
+ 0x00, 0x80, 0x60, 0x48, 0x81, 0xB0, 0x8B, 0x46, 0x00, 0x68, 0x3C, 0x46, 0xB9, 0x46, 0x90, 0xF8,
+ 0x90, 0x02, 0xC0, 0x07, 0x1F, 0xD0, 0x00, 0x26, 0x01, 0x9D, 0x16, 0xE0, 0x63, 0x4A, 0x28, 0x46,
+ 0x59, 0x46, 0x12, 0x78, 0x01, 0x2A, 0x13, 0xD0, 0xFF, 0xF7, 0x34, 0xFF, 0x10, 0xF0, 0x01, 0x0F,
+ 0x05, 0xD0, 0x01, 0x20, 0xB0, 0x40, 0x38, 0x43, 0x64, 0x1C, 0xC7, 0xB2, 0xE4, 0xB2, 0xBA, 0xF9,
+ 0x00, 0x00, 0x76, 0x1C, 0x05, 0xEB, 0x40, 0x05, 0x36, 0xB2, 0x46, 0x45, 0xE6, 0xDB, 0x04, 0xE0,
+ 0x00, 0xF0, 0x21, 0xF8, 0xEA, 0xE7, 0x44, 0x46, 0xFF, 0x27, 0x55, 0x48, 0x07, 0x70, 0x4C, 0xB1,
+ 0x00, 0x94, 0xBA, 0xF8, 0x00, 0x00, 0x3B, 0x46, 0x82, 0xB2, 0x41, 0x46, 0x01, 0x98, 0x00, 0xF0,
+ 0x4A, 0xF8, 0x07, 0xE0, 0x48, 0x48, 0xB0, 0xF9, 0x00, 0x10, 0x03, 0x20, 0x02, 0xF0, 0xDA, 0xF9,
+ 0x4F, 0xF0, 0x01, 0x09, 0x48, 0x46, 0xBD, 0xE8, 0xFE, 0x8F, 0x70, 0x47, 0x44, 0x48, 0x00, 0x21,
+ 0x01, 0x70, 0x41, 0x70, 0x70, 0x47, 0x70, 0xB5, 0x39, 0x4A, 0x13, 0x78, 0x39, 0x4A, 0x12, 0x68,
+ 0x13, 0xB9, 0x39, 0x4B, 0x1B, 0x78, 0x13, 0xB1, 0xB2, 0xF9, 0x99, 0x32, 0x0D, 0xE0, 0x37, 0x4C,
+ 0xB2, 0xF9, 0x93, 0x32, 0x64, 0x78, 0x01, 0x2C, 0x02, 0xD0, 0x02, 0x2C, 0x03, 0xD0, 0x04, 0xE0,
+ 0xB2, 0xF9, 0x95, 0x32, 0x01, 0xE0, 0xB2, 0xF9, 0x97, 0x32, 0x31, 0x4A, 0x39, 0x4C, 0xB2, 0xF9,
+ 0x00, 0x20, 0x25, 0x78, 0x13, 0xE0, 0x00, 0xBF, 0x30, 0xF9, 0x12, 0x40, 0x31, 0xF9, 0x12, 0x60,
+ 0xA4, 0x1B, 0x00, 0xD5, 0x64, 0x42, 0x24, 0xB2, 0x9C, 0x42, 0x08, 0xDD, 0x92, 0xFB, 0xF5, 0xF6,
+ 0x05, 0xFB, 0x16, 0x26, 0x1E, 0xB1, 0x28, 0x48, 0x04, 0x80, 0x00, 0x20, 0x70, 0xBD, 0x52, 0x1E,
+ 0xEA, 0xD2, 0x01, 0x20, 0x70, 0xBD, 0x2D, 0xE9, 0xF0, 0x03, 0x06, 0x9F, 0x00, 0x2F, 0x19, 0xD0,
+ 0x01, 0x25, 0x01, 0x2F, 0x02, 0xD0, 0x14, 0x46, 0xA9, 0x46, 0x2E, 0xE0, 0x00, 0x24, 0x0F, 0xE0,
+ 0x05, 0xFA, 0x04, 0xF6, 0x1E, 0x42, 0x0A, 0xD0, 0x00, 0x2C, 0x0B, 0xD0, 0x53, 0x00, 0x62, 0x43,
+ 0xBD, 0xE8, 0xF0, 0x03, 0x00, 0xEB, 0x42, 0x01, 0x1A, 0x46, 0x08, 0xF0, 0x03, 0xB9, 0x64, 0x1C,
+ 0x8C, 0x42, 0xED, 0xDB, 0xBD, 0xE8, 0xF0, 0x03, 0x70, 0x47, 0x00, 0xBF, 0x00, 0x26, 0x00, 0xEB,
+ 0x44, 0x0C, 0x35, 0x46, 0x0B, 0xE0, 0x00, 0xBF, 0x09, 0xFA, 0x05, 0xF8, 0x18, 0xEA, 0x03, 0x0F,
+ 0x02, 0xD0, 0xBC, 0xF9, 0x00, 0x80, 0x46, 0x44, 0x0C, 0xEB, 0x42, 0x0C, 0x6D, 0x1C, 0x8D, 0x42,
+ 0xF2, 0xDB, 0x96, 0xFB, 0xF7, 0xF5, 0x20, 0xF8, 0x14, 0x50, 0x64, 0x1E, 0xE6, 0xD2, 0xE1, 0xE7,
+ 0x1C, 0x05, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0x28, 0x05, 0x10, 0x00, 0x4C, 0x07, 0x10, 0x00,
+ 0xCA, 0x06, 0x10, 0x00, 0x78, 0x12, 0x01, 0x20, 0x56, 0x06, 0x10, 0x00, 0xBB, 0x06, 0x10, 0x00,
+ 0x4C, 0x06, 0x10, 0x00, 0x28, 0x07, 0x10, 0x00, 0xBC, 0x06, 0x10, 0x00, 0xBF, 0x06, 0x10, 0x00,
+ 0x51, 0x06, 0x10, 0x00, 0xBA, 0x06, 0x10, 0x00, 0x10, 0xB5, 0xF6, 0xF7, 0x51, 0xFD, 0xF7, 0xF7,
+ 0xE2, 0xF9, 0xBD, 0xE8, 0x10, 0x40, 0xF5, 0xF7, 0x51, 0xBF, 0x2D, 0xE9, 0xF0, 0x47, 0x81, 0x46,
+ 0x1F, 0x46, 0x90, 0x46, 0x0E, 0x46, 0x03, 0x20, 0x08, 0x9D, 0x06, 0xF0, 0xC4, 0xF8, 0x6E, 0x48,
+ 0x01, 0x68, 0x48, 0x46, 0xFF, 0xF7, 0x05, 0xFF, 0x04, 0x46, 0x03, 0x20, 0x06, 0xF0, 0xD6, 0xF8,
+ 0x64, 0xB1, 0xFF, 0xF7, 0xE7, 0xFE, 0xFF, 0xF7, 0xB5, 0xFE, 0x31, 0x46, 0x48, 0x46, 0xFF, 0xF7,
+ 0x44, 0xFF, 0x66, 0x48, 0x04, 0x70, 0x20, 0x46, 0xBD, 0xE8, 0xF0, 0x87, 0xFF, 0xF7, 0x3E, 0xFF,
+ 0x16, 0xB1, 0x30, 0x46, 0x00, 0xF0, 0xCC, 0xFB, 0x5D, 0xB1, 0x61, 0x48, 0x01, 0x78, 0x28, 0x46,
+ 0xF6, 0xF7, 0x45, 0xFD, 0x5F, 0x48, 0x80, 0x78, 0x18, 0xB1, 0x5F, 0x49, 0x00, 0x20, 0x08, 0x70,
+ 0x48, 0x70, 0x08, 0xF0, 0xC3, 0xF9, 0x05, 0x46, 0xFF, 0xF7, 0x84, 0xFE, 0x50, 0xEA, 0x05, 0x04,
+ 0xDB, 0xD1, 0x3A, 0x46, 0x41, 0x46, 0x48, 0x46, 0x00, 0xF0, 0x3C, 0xF8, 0x04, 0x46, 0xD4, 0xE7,
+ 0x10, 0xB5, 0x04, 0x46, 0x00, 0xF0, 0x7F, 0xF8, 0x00, 0x21, 0x20, 0x46, 0xFF, 0xF7, 0x15, 0xFF,
+ 0x00, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0xF8, 0xF7, 0x34, 0xFA, 0xFD, 0xF7, 0xA5, 0xFE, 0xFB, 0xF7,
+ 0xDF, 0xFC, 0xFF, 0xF7, 0xA1, 0xFF, 0xFF, 0xF7, 0x09, 0xFF, 0xFC, 0xF7, 0x7E, 0xFB, 0x47, 0x49,
+ 0x00, 0x20, 0x08, 0x70, 0x48, 0x70, 0x88, 0x80, 0xC8, 0x80, 0x88, 0x70, 0x88, 0x60, 0xC8, 0x60,
+ 0x08, 0x61, 0x14, 0x31, 0x08, 0x70, 0x88, 0x70, 0x44, 0x48, 0x00, 0x68, 0x10, 0xF8, 0x30, 0x2F,
+ 0x4A, 0x70, 0x40, 0x78, 0xC8, 0x70, 0x10, 0xBD, 0x10, 0xB5, 0xF8, 0xF7, 0x23, 0xFA, 0xFD, 0xF7,
+ 0x96, 0xFE, 0x00, 0xF0, 0xF0, 0xFB, 0xF7, 0xF7, 0x73, 0xF9, 0x3A, 0x48, 0x00, 0x21, 0x41, 0x70,
+ 0x81, 0x70, 0x10, 0xBD, 0xF8, 0xB5, 0x07, 0x46, 0x00, 0x24, 0x15, 0x46, 0x0E, 0x46, 0x8D, 0xF8,
+ 0x00, 0x40, 0x02, 0x20, 0x06, 0xF0, 0x47, 0xF8, 0x2A, 0x46, 0x31, 0x46, 0x38, 0x46, 0xF8, 0xF7,
+ 0x6A, 0xF8, 0x33, 0x49, 0x01, 0x28, 0xCA, 0x78, 0x60, 0xF3, 0x03, 0x02, 0xCA, 0x70, 0x28, 0xD0,
+ 0x02, 0x28, 0x2B, 0xD0, 0x04, 0x28, 0x00, 0xD0, 0x01, 0x24, 0x6B, 0x46, 0x2A, 0x46, 0x31, 0x46,
+ 0x38, 0x46, 0xFC, 0xF7, 0x63, 0xFA, 0x40, 0xEA, 0x04, 0x05, 0x02, 0x20, 0x06, 0xF0, 0x46, 0xF8,
+ 0x9D, 0xF8, 0x00, 0x00, 0x98, 0xB1, 0xF1, 0xF7, 0x09, 0xFE, 0x26, 0x48, 0x26, 0x4A, 0x01, 0x68,
+ 0x12, 0x68, 0x11, 0x43, 0x01, 0x60, 0xF1, 0xF7, 0xFF, 0xFD, 0x24, 0x48, 0x01, 0x78, 0x05, 0x20,
+ 0x02, 0xF0, 0x80, 0xF8, 0x4F, 0xF4, 0x80, 0x71, 0x00, 0x20, 0x05, 0xF0, 0x74, 0xFF, 0x28, 0x46,
+ 0xF8, 0xBD, 0xFD, 0xF7, 0x2B, 0xFE, 0xFB, 0xF7, 0x94, 0xFC, 0xD6, 0xE7, 0xFD, 0xF7, 0x86, 0xFB,
+ 0xFB, 0xF7, 0xBC, 0xFC, 0xD1, 0xE7, 0x70, 0xB5, 0x05, 0x46, 0x00, 0x24, 0x02, 0x20, 0x06, 0xF0,
+ 0x02, 0xF8, 0x28, 0x46, 0xF8, 0xF7, 0x4B, 0xF9, 0x11, 0x49, 0x01, 0x28, 0xCA, 0x78, 0x60, 0xF3,
+ 0x03, 0x02, 0xCA, 0x70, 0x02, 0xD0, 0x02, 0x28, 0x05, 0xD0, 0x08, 0xE0, 0xFD, 0xF7, 0x1B, 0xFE,
+ 0xFB, 0xF7, 0xEA, 0xFC, 0x03, 0xE0, 0xFD, 0xF7, 0xFA, 0xFC, 0xFB, 0xF7, 0xF6, 0xFC, 0x02, 0x20,
+ 0x06, 0xF0, 0x04, 0xF8, 0x20, 0x46, 0x70, 0xBD, 0xE8, 0x06, 0x10, 0x00, 0x50, 0x06, 0x10, 0x00,
+ 0x50, 0x07, 0x10, 0x00, 0x4C, 0x07, 0x10, 0x00, 0xEC, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0xE6, 0x43, 0x01, 0x20, 0x28, 0x07, 0x10, 0x00, 0x08, 0x06, 0x10, 0x00, 0x04, 0x06, 0x10, 0x00,
+ 0x06, 0x29, 0x04, 0xD2, 0x8B, 0x4A, 0x51, 0x5C, 0x8B, 0x4A, 0x53, 0x5C, 0x2B, 0xB1, 0x00, 0x21,
+ 0x01, 0x60, 0x01, 0x46, 0x48, 0x20, 0x01, 0xF0, 0x02, 0xBD, 0x86, 0x4B, 0x08, 0x33, 0x53, 0xF8,
+ 0x21, 0x30, 0x03, 0x60, 0x85, 0x4B, 0x43, 0xF8, 0x21, 0x00, 0x01, 0x20, 0x50, 0x54, 0x70, 0x47,
+ 0x0A, 0x46, 0x30, 0xB4, 0x00, 0x21, 0x06, 0x2A, 0x11, 0xD2, 0x7E, 0x4B, 0x9A, 0x5C, 0x72, 0xB1,
+ 0x7C, 0x4C, 0x03, 0x68, 0x08, 0x34, 0x54, 0xF8, 0x22, 0x40, 0xA3, 0x42, 0x07, 0xD1, 0x7A, 0x4B,
+ 0x99, 0x54, 0x7A, 0x4B, 0x43, 0xF8, 0x22, 0x10, 0x01, 0x60, 0x30, 0xBC, 0x70, 0x47, 0x01, 0x60,
+ 0x01, 0x46, 0x30, 0xBC, 0x49, 0x20, 0x01, 0xF0, 0xDA, 0xBC, 0x2D, 0xE9, 0xF0, 0x41, 0x01, 0x24,
+ 0x71, 0x4D, 0x72, 0x4F, 0x00, 0x26, 0x28, 0x5D, 0x38, 0xB1, 0x2E, 0x55, 0x57, 0xF8, 0x24, 0x10,
+ 0x4A, 0x20, 0x01, 0xF0, 0xCC, 0xFC, 0x47, 0xF8, 0x24, 0x60, 0x64, 0x1C, 0x03, 0x2C, 0xF2, 0xDB,
+ 0xBD, 0xE8, 0xF0, 0x81, 0x2D, 0xE9, 0xF0, 0x41, 0x68, 0x4D, 0x67, 0x4B, 0x00, 0x27, 0x0A, 0x46,
+ 0x0C, 0x35, 0x07, 0x60, 0x59, 0x68, 0x05, 0xF1, 0x14, 0x06, 0x03, 0xF1, 0x08, 0x04, 0x01, 0x2A,
+ 0x38, 0xD0, 0x02, 0x2A, 0x1C, 0xD0, 0x03, 0x2A, 0x3F, 0xD1, 0x00, 0x22, 0x04, 0xEB, 0x02, 0x0C,
+ 0xDC, 0xF8, 0x00, 0xE0, 0xBE, 0xF1, 0x00, 0x0F, 0x07, 0xD0, 0x52, 0x1C, 0x02, 0x2A, 0xF5, 0xDB,
+ 0x04, 0x29, 0x30, 0xD3, 0x01, 0x46, 0x4C, 0x20, 0x31, 0xE0, 0x55, 0xF8, 0x22, 0x40, 0x04, 0x60,
+ 0x46, 0xF8, 0x22, 0x00, 0x4F, 0xF0, 0x03, 0x30, 0x09, 0x1F, 0xCC, 0xF8, 0x00, 0x00, 0x32, 0xE0,
+ 0x5F, 0xF0, 0x00, 0x02, 0x04, 0xEB, 0x02, 0x0C, 0xBC, 0xF8, 0x00, 0xE0, 0xBE, 0xF1, 0x00, 0x0F,
+ 0x05, 0xD0, 0x52, 0x1C, 0x04, 0x2A, 0xF5, 0xDB, 0x02, 0x29, 0xE3, 0xD2, 0x13, 0xE0, 0x55, 0xF8,
+ 0x22, 0x40, 0x04, 0x60, 0x46, 0xF8, 0x22, 0x00, 0x40, 0xF2, 0x02, 0x20, 0xAC, 0xF8, 0x00, 0x00,
+ 0x89, 0x1E, 0x18, 0xE0, 0x5F, 0xF0, 0x04, 0x02, 0x14, 0xF8, 0x02, 0xC0, 0xBC, 0xF1, 0x00, 0x0F,
+ 0x09, 0xD0, 0x52, 0x1E, 0xF8, 0xD2, 0x00, 0x2F, 0xAA, 0xD1, 0x01, 0x46, 0x4B, 0x20, 0xBD, 0xE8,
+ 0xF0, 0x41, 0x01, 0xF0, 0x6C, 0xBC, 0x55, 0xF8, 0x22, 0x50, 0x05, 0x60, 0x46, 0xF8, 0x22, 0x00,
+ 0x01, 0x20, 0xA0, 0x54, 0x49, 0x1E, 0x59, 0x60, 0x9A, 0xE7, 0xF0, 0xB4, 0x37, 0x4E, 0x36, 0x4C,
+ 0x00, 0x25, 0x04, 0x22, 0x0C, 0x36, 0xD0, 0xF8, 0x00, 0x30, 0x08, 0x34, 0x56, 0xF8, 0x22, 0x70,
+ 0x9F, 0x42, 0x12, 0xD1, 0xA7, 0x5C, 0x8F, 0x42, 0x0F, 0xD1, 0x53, 0x1C, 0x11, 0xD0, 0x2F, 0x4D,
+ 0x00, 0x23, 0x20, 0x35, 0x03, 0x60, 0x45, 0xF8, 0x22, 0x30, 0xA3, 0x54, 0x01, 0x29, 0x11, 0xD0,
+ 0x02, 0x29, 0x11, 0xD0, 0x03, 0x29, 0x04, 0xD1, 0x08, 0xE0, 0x52, 0x1E, 0xE6, 0xD2, 0x00, 0x2D,
+ 0x08, 0xD1, 0xF0, 0xBC, 0x01, 0x46, 0x4D, 0x20, 0x01, 0xF0, 0x39, 0xBC, 0xA0, 0x18, 0x43, 0x70,
+ 0x83, 0x70, 0xC3, 0x70, 0xF0, 0xBC, 0x70, 0x47, 0xA0, 0x18, 0x43, 0x70, 0xFA, 0xE7, 0x70, 0xB5,
+ 0x1D, 0x4E, 0x1E, 0x4D, 0x05, 0x24, 0x08, 0x36, 0x20, 0x35, 0x0B, 0xE0, 0x30, 0x5D, 0x48, 0xB1,
+ 0x55, 0xF8, 0x24, 0x10, 0x4E, 0x20, 0x01, 0xF0, 0x22, 0xFC, 0x31, 0x5D, 0x55, 0xF8, 0x24, 0x00,
+ 0xFF, 0xF7, 0xBB, 0xFF, 0x64, 0x1E, 0xF1, 0xD2, 0x70, 0xBD, 0x70, 0xB5, 0x01, 0x20, 0x13, 0x4B,
+ 0x11, 0x4A, 0x00, 0x21, 0x43, 0xF8, 0x20, 0x10, 0x11, 0x54, 0x40, 0x1C, 0x03, 0x28, 0xF9, 0xDB,
+ 0x01, 0x20, 0x10, 0x70, 0x0C, 0x4A, 0x05, 0x20, 0x0C, 0x4B, 0x50, 0x60, 0x0C, 0x33, 0x0A, 0x4C,
+ 0x04, 0x20, 0x0B, 0x4A, 0x08, 0x34, 0x03, 0xF1, 0x14, 0x05, 0x73, 0x26, 0x46, 0x43, 0x02, 0xEB,
+ 0xC6, 0x06, 0x43, 0xF8, 0x20, 0x60, 0x21, 0x54, 0x45, 0xF8, 0x20, 0x10, 0x40, 0x1E, 0xF4, 0xD2,
+ 0x70, 0xBD, 0x00, 0x00, 0x58, 0x6C, 0x01, 0x00, 0x8C, 0x06, 0x10, 0x00, 0x88, 0x47, 0x10, 0x00,
+ 0x84, 0x5A, 0x10, 0x00, 0xF0, 0xB5, 0x06, 0x46, 0xF8, 0x48, 0x00, 0x24, 0xA7, 0xB0, 0x00, 0x78,
+ 0x0D, 0x46, 0x01, 0x28, 0x00, 0xD1, 0x01, 0x24, 0x98, 0x21, 0x68, 0x46, 0x07, 0xF0, 0x33, 0xFF,
+ 0xF3, 0x4A, 0x00, 0x21, 0x68, 0x46, 0x13, 0x78, 0x11, 0xE0, 0x00, 0xBF, 0x01, 0xFB, 0x03, 0xF7,
+ 0x22, 0x46, 0x09, 0xE0, 0x07, 0xEB, 0x02, 0x0E, 0x50, 0xF8, 0x22, 0xC0, 0x36, 0xF9, 0x1E, 0xE0,
+ 0xF4, 0x44, 0x40, 0xF8, 0x22, 0xC0, 0x52, 0x1C, 0x9A, 0x42, 0xF3, 0xDB, 0x49, 0x1C, 0xA9, 0x42,
+ 0xEC, 0xDB, 0x21, 0x46, 0x07, 0xE0, 0x00, 0xBF, 0x50, 0xF8, 0x21, 0x20, 0x92, 0xFB, 0xF5, 0xF2,
+ 0x40, 0xF8, 0x21, 0x20, 0x49, 0x1C, 0x99, 0x42, 0xF6, 0xDB, 0x00, 0x21, 0x17, 0xE0, 0x00, 0x27,
+ 0x3A, 0x46, 0x0C, 0xE0, 0x02, 0xFB, 0x03, 0x4C, 0x50, 0xF8, 0x24, 0xE0, 0x36, 0xF9, 0x1C, 0xC0,
+ 0xBC, 0xEB, 0x0E, 0x0C, 0x01, 0xD5, 0xCC, 0xF1, 0x00, 0x0C, 0x67, 0x44, 0x52, 0x1C, 0xAA, 0x42,
+ 0xF0, 0xDB, 0x97, 0xFB, 0xF5, 0xF2, 0x8A, 0x42, 0x00, 0xDD, 0x11, 0x46, 0x64, 0x1C, 0x9C, 0x42,
+ 0xE5, 0xDB, 0xB1, 0xF5, 0x80, 0x3F, 0x03, 0xDB, 0x4F, 0xF6, 0xFF, 0x70, 0x27, 0xB0, 0xF0, 0xBD,
+ 0x88, 0xB2, 0xFB, 0xE7, 0xF0, 0xB5, 0xCF, 0x49, 0xCF, 0x4B, 0x09, 0x68, 0x91, 0xF8, 0xAA, 0x22,
+ 0xD2, 0x07, 0x34, 0xD0, 0xCD, 0x4A, 0x57, 0x78, 0x4F, 0xB1, 0x91, 0xF8, 0xB5, 0x42, 0xB1, 0xF8,
+ 0xB2, 0x52, 0x04, 0xF0, 0x0F, 0x02, 0x26, 0x09, 0xB1, 0xF8, 0xB0, 0x42, 0x08, 0xE0, 0x91, 0xF8,
+ 0xB4, 0x42, 0xB1, 0xF8, 0xAE, 0x52, 0x04, 0xF0, 0x0F, 0x02, 0x26, 0x09, 0xB1, 0xF8, 0xAC, 0x42,
+ 0xDF, 0xF8, 0x0C, 0xC3, 0x9C, 0xF8, 0x00, 0xC0, 0xBC, 0xF1, 0x03, 0x0F, 0x06, 0xD1, 0x91, 0xF8,
+ 0xB6, 0x12, 0x0F, 0xB1, 0x0A, 0x09, 0x01, 0xE0, 0x01, 0xF0, 0x0F, 0x02, 0xA0, 0x42, 0x01, 0xDC,
+ 0x1A, 0x60, 0xF0, 0xBD, 0xA8, 0x42, 0x01, 0xDB, 0x1E, 0x60, 0xF0, 0xBD, 0x00, 0x1B, 0xB1, 0x1A,
+ 0x48, 0x43, 0x29, 0x1B, 0x90, 0xFB, 0xF1, 0xF0, 0x10, 0x44, 0x18, 0x60, 0xF0, 0xBD, 0x00, 0x20,
+ 0xFB, 0xE7, 0xB0, 0x48, 0xB1, 0x49, 0x00, 0x68, 0x0A, 0x78, 0x00, 0xF5, 0xF9, 0x70, 0x80, 0x5C,
+ 0xC8, 0x70, 0x70, 0x47, 0x2D, 0xE9, 0xF0, 0x0F, 0x83, 0x46, 0xAA, 0x48, 0xDF, 0xF8, 0xAC, 0x92,
+ 0xDF, 0xF8, 0xB0, 0xA2, 0x01, 0x68, 0x99, 0xF8, 0x00, 0x70, 0x00, 0x20, 0x91, 0xF8, 0x0A, 0x62,
+ 0x91, 0xF8, 0x09, 0x12, 0x3A, 0xF8, 0x17, 0x40, 0x6F, 0xF0, 0x00, 0x42, 0x4F, 0xEA, 0x11, 0x18,
+ 0x0B, 0x07, 0x1C, 0xD0, 0x01, 0xF0, 0x0F, 0x05, 0x9E, 0x4B, 0x00, 0x21, 0x8C, 0x46, 0x1B, 0x68,
+ 0xB4, 0x42, 0x93, 0xF8, 0x08, 0x32, 0x05, 0xDB, 0xA6, 0x1B, 0x06, 0xFB, 0x08, 0xF6, 0x96, 0xFB,
+ 0xF5, 0xF5, 0x2B, 0x44, 0x97, 0x4D, 0x99, 0xF8, 0x01, 0x60, 0x2D, 0x68, 0x01, 0x2E, 0x95, 0xF8,
+ 0xF1, 0x51, 0x06, 0xD0, 0x02, 0x2E, 0x07, 0xD0, 0x05, 0xF0, 0x03, 0x06, 0x13, 0xE0, 0x01, 0x25,
+ 0xE2, 0xE7, 0xC5, 0xF3, 0x81, 0x06, 0x0E, 0xE0, 0xC5, 0xF3, 0x01, 0x16, 0x0B, 0xE0, 0x00, 0xBF,
+ 0x3A, 0xF8, 0x1C, 0x50, 0xAA, 0x42, 0x01, 0xDD, 0x2A, 0x46, 0x60, 0x46, 0xA9, 0x42, 0x00, 0xDA,
+ 0x29, 0x46, 0x0C, 0xF1, 0x01, 0x0C, 0xB4, 0x45, 0xF2, 0xDD, 0xAB, 0xF8, 0x00, 0x10, 0xA1, 0x1A,
+ 0x99, 0x42, 0x01, 0xDC, 0xB7, 0x42, 0x04, 0xDD, 0x89, 0xF8, 0x00, 0x00, 0xBD, 0xE8, 0xF0, 0x0F,
+ 0x9F, 0xE7, 0xBD, 0xE8, 0xF0, 0x0F, 0x70, 0x47, 0x83, 0x49, 0x10, 0xB5, 0x0B, 0x78, 0x7D, 0x49,
+ 0x00, 0x22, 0x01, 0x2B, 0x09, 0x68, 0x02, 0xD1, 0x80, 0x4B, 0x1B, 0x78, 0x4B, 0xB1, 0xB1, 0xF8,
+ 0xFC, 0x31, 0x83, 0x42, 0x18, 0xD2, 0x91, 0xF8, 0xF0, 0x31, 0x1B, 0x07, 0x14, 0xD5, 0x03, 0x22,
+ 0x25, 0xE0, 0xB1, 0xF8, 0x02, 0x32, 0x83, 0x42, 0x03, 0xD2, 0x91, 0xF8, 0xF0, 0x31, 0x1B, 0x07,
+ 0xF5, 0xD4, 0xB1, 0xF8, 0x00, 0x32, 0x83, 0x42, 0x03, 0xD2, 0x91, 0xF8, 0xF0, 0x31, 0x5B, 0x07,
+ 0x0A, 0xD4, 0xB1, 0xF8, 0xFE, 0x31, 0x0B, 0xE0, 0xB1, 0xF8, 0xFA, 0x31, 0x83, 0x42, 0x05, 0xD2,
+ 0x91, 0xF8, 0xF0, 0x31, 0x5B, 0x07, 0x01, 0xD5, 0x02, 0x22, 0x08, 0xE0, 0xB1, 0xF8, 0xF8, 0x31,
+ 0x83, 0x42, 0x04, 0xD2, 0x91, 0xF8, 0xF0, 0x01, 0x80, 0x07, 0x00, 0xD5, 0x01, 0x22, 0x65, 0x48,
+ 0x62, 0x4C, 0x14, 0x38, 0x03, 0x78, 0x9A, 0x42, 0x11, 0xD1, 0x61, 0x78, 0x99, 0x42, 0x1A, 0xD0,
+ 0x63, 0x4A, 0x12, 0x78, 0x13, 0x43, 0x07, 0xD0, 0x81, 0x68, 0x49, 0x1E, 0x81, 0x60, 0x00, 0x29,
+ 0x11, 0xDC, 0x00, 0x78, 0x60, 0x70, 0x0E, 0xE0, 0x01, 0x70, 0x81, 0x68, 0xF7, 0xE7, 0x8A, 0xB1,
+ 0x02, 0x2A, 0x01, 0xD1, 0x01, 0x2B, 0x0D, 0xD0, 0x91, 0xF8, 0xF6, 0x11, 0x81, 0x60, 0x02, 0x70,
+ 0x00, 0x29, 0x00, 0xDC, 0x62, 0x70, 0x57, 0x48, 0x62, 0x78, 0x41, 0x7B, 0x62, 0xF3, 0x03, 0x01,
+ 0x41, 0x73, 0x10, 0xBD, 0x91, 0xF8, 0xF7, 0x11, 0xF0, 0xE7, 0x4A, 0x4A, 0x12, 0x68, 0x92, 0xF8,
+ 0xF0, 0x21, 0xD2, 0x06, 0x07, 0xD5, 0x42, 0x1A, 0x00, 0x2A, 0x04, 0xDC, 0xC1, 0xEB, 0xC1, 0x11,
+ 0x08, 0x44, 0xC0, 0xF3, 0xCF, 0x10, 0x70, 0x47, 0x44, 0x49, 0x00, 0x20, 0x41, 0x4A, 0x48, 0x70,
+ 0x44, 0x49, 0x14, 0x39, 0x48, 0x60, 0x88, 0x60, 0x08, 0x70, 0x12, 0x68, 0x92, 0xF8, 0xAA, 0x32,
+ 0xDB, 0x07, 0x03, 0xD0, 0x92, 0xF8, 0xB5, 0x22, 0x12, 0x09, 0x00, 0xE0, 0x00, 0x22, 0x3A, 0x4B,
+ 0x1A, 0x60, 0xC8, 0x60, 0x08, 0x61, 0x48, 0x61, 0x88, 0x61, 0x70, 0x47, 0x2D, 0xE9, 0xF8, 0x4F,
+ 0x00, 0x21, 0xAD, 0xF8, 0x00, 0x10, 0x32, 0x49, 0xDF, 0xF8, 0xD8, 0x90, 0xDF, 0xF8, 0xD4, 0x80,
+ 0x0E, 0x78, 0x4F, 0xF0, 0x10, 0x0B, 0x37, 0x01, 0x05, 0x46, 0x00, 0x24, 0xA9, 0xF1, 0x08, 0x09,
+ 0xA8, 0xF1, 0x14, 0x0A, 0x10, 0xE0, 0x32, 0x46, 0x59, 0x46, 0x28, 0x46, 0xFF, 0xF7, 0x5A, 0xFE,
+ 0x29, 0xF8, 0x14, 0x00, 0x43, 0x46, 0x38, 0xF8, 0x14, 0x10, 0xFF, 0xF7, 0xB6, 0xFF, 0x23, 0xF8,
+ 0x14, 0x00, 0x05, 0xEB, 0x47, 0x05, 0x64, 0x1C, 0x9A, 0xF8, 0x01, 0x00, 0x84, 0x42, 0xEA, 0xDD,
+ 0x68, 0x46, 0xFF, 0xF7, 0xE7, 0xFE, 0x1F, 0x48, 0x20, 0x4D, 0x00, 0x68, 0x90, 0xF8, 0xF0, 0x01,
+ 0x40, 0x06, 0x02, 0xD5, 0xBD, 0xF8, 0x00, 0x40, 0x02, 0xE0, 0x29, 0x78, 0x38, 0xF8, 0x11, 0x40,
+ 0x21, 0x48, 0x04, 0x80, 0x20, 0x46, 0xFF, 0xF7, 0x2F, 0xFF, 0x20, 0x46, 0xFF, 0xF7, 0x8A, 0xFE,
+ 0x29, 0x78, 0x39, 0xF8, 0x11, 0x00, 0x1B, 0x49, 0xA1, 0xF8, 0x1B, 0x00, 0xBD, 0xE8, 0xF8, 0x8F,
+ 0x2D, 0xE9, 0xFC, 0x5F, 0x0E, 0x49, 0x4F, 0xF0, 0x00, 0x09, 0x0D, 0x78, 0x0B, 0x49, 0x09, 0x78,
+ 0x01, 0x29, 0x00, 0xD1, 0x89, 0x46, 0x0F, 0x49, 0x10, 0x26, 0x14, 0x39, 0x04, 0x46, 0x4A, 0x68,
+ 0x12, 0x48, 0x92, 0xFB, 0xF6, 0xF3, 0x06, 0xFB, 0x13, 0x22, 0x6A, 0x43, 0x4F, 0x78, 0x4F, 0xEA,
+ 0x05, 0x1A, 0x00, 0xEB, 0x42, 0x08, 0x83, 0x46, 0x7F, 0x1C, 0x3D, 0xE0, 0xBF, 0x06, 0x10, 0x00,
+ 0xBA, 0x06, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0x78, 0x06, 0x10, 0x00, 0x4C, 0x07, 0x10, 0x00,
+ 0xB8, 0x06, 0x10, 0x00, 0xB0, 0x06, 0x10, 0x00, 0xC2, 0x06, 0x10, 0x00, 0xC3, 0x06, 0x10, 0x00,
+ 0xC1, 0x06, 0x10, 0x00, 0xE6, 0x43, 0x01, 0x20, 0x54, 0x06, 0x10, 0x00, 0x8C, 0x1C, 0x01, 0x20,
+ 0x2A, 0x48, 0x00, 0x68, 0x90, 0xF8, 0xF0, 0x11, 0x89, 0x06, 0x14, 0xD5, 0x28, 0x49, 0x49, 0x68,
+ 0x00, 0x29, 0x0E, 0xDD, 0xA5, 0xEB, 0x09, 0x02, 0xB0, 0xF9, 0x06, 0x12, 0xCD, 0xE9, 0x00, 0x12,
+ 0xB0, 0xF9, 0x04, 0x32, 0x04, 0xEB, 0x49, 0x00, 0x0B, 0xEB, 0x49, 0x02, 0x01, 0x46, 0x02, 0xF0,
+ 0xB3, 0xFF, 0x0B, 0xEB, 0x4A, 0x0B, 0x6A, 0x00, 0x21, 0x46, 0x40, 0x46, 0x07, 0xF0, 0x72, 0xFC,
+ 0x04, 0xEB, 0x45, 0x04, 0x08, 0xEB, 0x4A, 0x08, 0x7F, 0x1E, 0xD9, 0xD2, 0x18, 0x48, 0x4F, 0xF0,
+ 0xFF, 0x31, 0x01, 0xEB, 0x46, 0x01, 0x42, 0x68, 0x52, 0x1C, 0x42, 0x60, 0x91, 0x42, 0x00, 0xDA,
+ 0x46, 0x60, 0x40, 0x68, 0xB0, 0x42, 0x03, 0xDB, 0xBD, 0xE8, 0xFC, 0x5F, 0x11, 0x48, 0x45, 0xE7,
+ 0xBD, 0xE8, 0xFC, 0x9F, 0x28, 0xE7, 0x10, 0xB5, 0x0C, 0x4C, 0x20, 0x68, 0x90, 0xF8, 0xF1, 0x01,
+ 0xC0, 0xF3, 0x01, 0x12, 0xC0, 0xF3, 0x81, 0x01, 0x00, 0xF0, 0x03, 0x00, 0x02, 0xF0, 0x73, 0xF9,
+ 0x07, 0x49, 0x48, 0x70, 0x20, 0x68, 0x08, 0x49, 0x90, 0xF8, 0xEC, 0x00, 0x00, 0xF0, 0x03, 0x00,
+ 0x08, 0x70, 0xFF, 0xF7, 0x36, 0xFE, 0xBD, 0xE8, 0x10, 0x40, 0x0D, 0xE7, 0x20, 0x07, 0x10, 0x00,
+ 0x9C, 0x06, 0x10, 0x00, 0x8C, 0x1C, 0x01, 0x20, 0x4C, 0x07, 0x10, 0x00, 0x10, 0xB5, 0xD1, 0x48,
+ 0x00, 0x24, 0x84, 0x61, 0x44, 0x63, 0x04, 0x87, 0x80, 0xF8, 0x3A, 0x40, 0x04, 0x62, 0xC4, 0x61,
+ 0xFF, 0xF7, 0xDA, 0xFB, 0xFF, 0xF7, 0x59, 0xFD, 0xCB, 0x48, 0x04, 0x80, 0xBD, 0xE8, 0x10, 0x40,
+ 0x00, 0xF0, 0xED, 0xB8, 0xC9, 0x48, 0x10, 0xB5, 0x90, 0xF8, 0x2C, 0x10, 0xC6, 0x48, 0xC5, 0x4A,
+ 0x81, 0x70, 0x53, 0x7A, 0x01, 0x79, 0x63, 0xF3, 0x00, 0x01, 0xD3, 0x7A, 0x63, 0xF3, 0xC3, 0x01,
+ 0x93, 0x7A, 0x63, 0xF3, 0x82, 0x01, 0x13, 0x7B, 0x63, 0xF3, 0x41, 0x01, 0xC0, 0x4B, 0xDB, 0x78,
+ 0x03, 0x72, 0x93, 0x7B, 0xC3, 0x71, 0x53, 0x7B, 0xD4, 0x7B, 0x63, 0xF3, 0x04, 0x11, 0x83, 0x7B,
+ 0x12, 0x7C, 0x64, 0xF3, 0x03, 0x03, 0x62, 0xF3, 0x07, 0x13, 0xBA, 0x4A, 0x83, 0x73, 0x12, 0x78,
+ 0x80, 0xF8, 0x28, 0x20, 0xB8, 0x4A, 0x12, 0x88, 0xA0, 0xF8, 0x29, 0x20, 0xB7, 0x4A, 0x12, 0x78,
+ 0x62, 0xF3, 0x45, 0x11, 0xB6, 0x4A, 0x12, 0x78, 0x62, 0xF3, 0xC7, 0x11, 0x01, 0x71, 0xB5, 0x49,
+ 0x09, 0x88, 0xA0, 0xF8, 0x2B, 0x10, 0xB4, 0x49, 0x09, 0x88, 0xA0, 0xF8, 0x2D, 0x10, 0xB3, 0x49,
+ 0x0A, 0x78, 0xC1, 0x7B, 0x62, 0xF3, 0x03, 0x01, 0xC1, 0x73, 0xB1, 0x49, 0x0A, 0x78, 0x01, 0x7C,
+ 0x62, 0xF3, 0x00, 0x01, 0xAF, 0x4A, 0x12, 0x78, 0x62, 0xF3, 0x41, 0x01, 0xAE, 0x4A, 0x12, 0x78,
+ 0x62, 0xF3, 0xC3, 0x01, 0x21, 0xF0, 0x04, 0x01, 0x01, 0x74, 0xAC, 0x49, 0x0A, 0x88, 0x02, 0x87,
+ 0x4A, 0x88, 0x42, 0x87, 0x8A, 0x88, 0x82, 0x87, 0xC9, 0x88, 0xC1, 0x87, 0x10, 0xBD, 0x70, 0xB5,
+ 0x0D, 0x46, 0x80, 0xEA, 0x05, 0x06, 0xF0, 0x06, 0x01, 0xD5, 0xF6, 0xF7, 0xDC, 0xFC, 0x95, 0x4C,
+ 0xC5, 0xF3, 0x00, 0x10, 0x34, 0x34, 0x20, 0x70, 0xB0, 0x06, 0x01, 0xD5, 0xF5, 0xF7, 0x46, 0xFA,
+ 0xC5, 0xF3, 0x40, 0x10, 0x60, 0x70, 0x15, 0xF0, 0x48, 0x00, 0x00, 0xD0, 0x01, 0x20, 0xA0, 0x71,
+ 0x9B, 0x48, 0x00, 0x68, 0x90, 0xF8, 0x40, 0x14, 0x01, 0xF0, 0x01, 0x01, 0x21, 0x71, 0x90, 0xF8,
+ 0x20, 0x04, 0x00, 0xF0, 0x01, 0x00, 0x60, 0x71, 0x70, 0xBD, 0x70, 0xB5, 0x85, 0x4D, 0x87, 0x4C,
+ 0x29, 0x6A, 0x20, 0x6A, 0x81, 0xEA, 0x00, 0x06, 0x19, 0xB9, 0x10, 0xB1, 0xFF, 0xF7, 0x5E, 0xFF,
+ 0x09, 0xE0, 0xF0, 0x07, 0x03, 0xD0, 0xFF, 0xF7, 0x1D, 0xFB, 0xFF, 0xF7, 0x33, 0xFF, 0xF0, 0x06,
+ 0x01, 0xD5, 0xFF, 0xF7, 0xC1, 0xFA, 0xE1, 0x69, 0xA8, 0x69, 0xFF, 0xF7, 0xC0, 0xFF, 0xE0, 0x69,
+ 0xA8, 0x61, 0x20, 0x6A, 0x28, 0x62, 0x60, 0x6A, 0xE8, 0x61, 0xBD, 0xE8, 0x70, 0x40, 0x00, 0xF0,
+ 0x66, 0xB8, 0x70, 0xB5, 0x83, 0x4D, 0x2C, 0x68, 0x00, 0x2C, 0x0D, 0xD0, 0x71, 0x49, 0xA0, 0xB2,
+ 0x0A, 0x7A, 0xC9, 0x79, 0x01, 0xF0, 0x1F, 0xFA, 0x28, 0x68, 0xA0, 0x43, 0x28, 0x60, 0x03, 0xD1,
+ 0xBD, 0xE8, 0x70, 0x40, 0x01, 0xF0, 0x7C, 0xBA, 0x70, 0xBD, 0x6A, 0x48, 0x81, 0x7A, 0x01, 0x29,
+ 0x08, 0xD0, 0xC1, 0x7A, 0x01, 0x29, 0x05, 0xD0, 0x01, 0x7B, 0x01, 0x29, 0x02, 0xD0, 0x80, 0x7B,
+ 0x00, 0x28, 0x00, 0xD0, 0x01, 0x20, 0x70, 0x47, 0x08, 0xB5, 0xFF, 0xF7, 0x1F, 0xFF, 0x6B, 0x46,
+ 0x00, 0x22, 0x0B, 0x21, 0x02, 0x20, 0x05, 0xF0, 0x07, 0xFB, 0x00, 0x98, 0xC0, 0x07, 0x05, 0xD0,
+ 0x01, 0x21, 0x02, 0x20, 0x05, 0xF0, 0xE3, 0xFA, 0x00, 0xF0, 0x56, 0xF8, 0x00, 0x98, 0x80, 0x07,
+ 0x05, 0xD5, 0x02, 0x21, 0x08, 0x46, 0x05, 0xF0, 0xDA, 0xFA, 0x00, 0xF0, 0x7E, 0xF8, 0x00, 0x98,
+ 0x00, 0x07, 0xE4, 0xD5, 0x08, 0x21, 0x02, 0x20, 0x05, 0xF0, 0xD1, 0xFA, 0xDF, 0xE7, 0x60, 0x48,
+ 0x10, 0xB5, 0x02, 0x68, 0x4F, 0x48, 0x92, 0xF8, 0x31, 0x10, 0x41, 0x70, 0x92, 0xF8, 0x30, 0x20,
+ 0x82, 0x70, 0x51, 0x43, 0x41, 0x82, 0x4E, 0x49, 0xC9, 0x78, 0xC1, 0x70, 0x01, 0x21, 0x01, 0x71,
+ 0x41, 0x71, 0x81, 0x71, 0x01, 0x70, 0x00, 0x21, 0x81, 0x82, 0xC1, 0x71, 0x01, 0x72, 0xC1, 0x73,
+ 0x01, 0x74, 0xFF, 0xF7, 0xA7, 0xFA, 0xBD, 0xE8, 0x10, 0x40, 0xFE, 0xF7, 0xD5, 0xBF, 0x43, 0x49,
+ 0x40, 0x48, 0x91, 0xF8, 0x30, 0x20, 0x42, 0x70, 0x91, 0xF8, 0x31, 0x30, 0x83, 0x70, 0x5A, 0x43,
+ 0x42, 0x82, 0x91, 0xF8, 0x33, 0x20, 0xC2, 0x70, 0x11, 0xF8, 0x2C, 0x2F, 0x02, 0x70, 0x4A, 0x78,
+ 0x02, 0x71, 0x8A, 0x78, 0x42, 0x71, 0xCA, 0x78, 0x82, 0x71, 0x4A, 0x7A, 0xC2, 0x71, 0x89, 0x7A,
+ 0x01, 0x72, 0x45, 0x49, 0x41, 0x62, 0x01, 0xF2, 0x3C, 0x61, 0x81, 0x62, 0xA1, 0xF6, 0x78, 0x41,
+ 0xC1, 0x62, 0x42, 0x49, 0x01, 0x63, 0x70, 0x47, 0x38, 0xB5, 0x00, 0x24, 0x01, 0xF0, 0x10, 0xF8,
+ 0x01, 0x20, 0x05, 0xF0, 0xF8, 0xFA, 0xFF, 0xF7, 0x48, 0xFF, 0x2A, 0x4D, 0x95, 0xF8, 0x20, 0x00,
+ 0xC0, 0x07, 0x08, 0xD0, 0x29, 0x48, 0xC1, 0x68, 0x00, 0x91, 0xD0, 0xE9, 0x04, 0x23, 0x03, 0xC8,
+ 0xFF, 0xF7, 0x1B, 0xFA, 0x04, 0x46, 0xE8, 0x69, 0x50, 0xB1, 0x4C, 0xB9, 0x03, 0x20, 0x05, 0xF0,
+ 0xE2, 0xFA, 0x22, 0x48, 0xE9, 0x69, 0xFE, 0xF7, 0xF6, 0xFF, 0x03, 0x20, 0x05, 0xF0, 0xF6, 0xFA,
+ 0xFF, 0xF7, 0xA8, 0xFE, 0xFF, 0xF7, 0x4D, 0xFF, 0xFF, 0xF7, 0xE1, 0xFB, 0xFF, 0xF7, 0x35, 0xFB,
+ 0xBD, 0xE8, 0x38, 0x40, 0x01, 0x20, 0x05, 0xF0, 0xE9, 0xBA, 0x70, 0xB5, 0x00, 0x24, 0x00, 0xF0,
+ 0xDF, 0xFF, 0x01, 0x20, 0x05, 0xF0, 0xC7, 0xFA, 0xFF, 0xF7, 0x17, 0xFF, 0x11, 0x4D, 0x95, 0xF8,
+ 0x20, 0x00, 0xC0, 0x07, 0x04, 0xD0, 0x11, 0x48, 0x00, 0x68, 0xFF, 0xF7, 0x31, 0xFA, 0x04, 0x46,
+ 0xE8, 0x69, 0x50, 0xB1, 0x4C, 0xB9, 0x03, 0x20, 0x05, 0xF0, 0xB5, 0xFA, 0x0B, 0x48, 0xE9, 0x69,
+ 0xFE, 0xF7, 0xC9, 0xFF, 0x03, 0x20, 0x05, 0xF0, 0xC9, 0xFA, 0xFF, 0xF7, 0x7B, 0xFE, 0xFF, 0xF7,
+ 0x20, 0xFF, 0xFF, 0xF7, 0xB4, 0xFB, 0xFF, 0xF7, 0x08, 0xFB, 0xBD, 0xE8, 0x70, 0x40, 0x01, 0x20,
+ 0x05, 0xF0, 0xBC, 0xBA, 0xB8, 0x06, 0x10, 0x00, 0xE6, 0x43, 0x01, 0x20, 0x7C, 0x6C, 0x10, 0x00,
+ 0x4C, 0x07, 0x10, 0x00, 0x51, 0x06, 0x10, 0x00, 0x56, 0x06, 0x10, 0x00, 0x80, 0x06, 0x10, 0x00,
+ 0x52, 0x07, 0x10, 0x00, 0x54, 0x07, 0x10, 0x00, 0x56, 0x07, 0x10, 0x00, 0x0E, 0x05, 0x10, 0x00,
+ 0x0C, 0x05, 0x10, 0x00, 0x1C, 0x05, 0x10, 0x00, 0x21, 0x05, 0x10, 0x00, 0xB0, 0x06, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0x40, 0x07, 0x10, 0x00, 0xC8, 0x35, 0x01, 0x20, 0x2E, 0x44, 0x01, 0x20,
+ 0x0D, 0x4D, 0x00, 0x20, 0x0D, 0x4C, 0x28, 0x60, 0x28, 0x68, 0x40, 0x1C, 0x28, 0x60, 0xF1, 0xF7,
+ 0x55, 0xF8, 0x07, 0xF0, 0x20, 0xF9, 0x01, 0x28, 0x20, 0x68, 0x09, 0xD0, 0x40, 0xF0, 0x04, 0x00,
+ 0x20, 0x60, 0x05, 0xF0, 0x6C, 0xFF, 0x00, 0xBF, 0xF1, 0xF7, 0x46, 0xF8, 0x30, 0xBF, 0xEB, 0xE7,
+ 0x20, 0xF0, 0x04, 0x00, 0x20, 0x60, 0xF7, 0xE7, 0xFC, 0x06, 0x10, 0x00, 0x10, 0xED, 0x00, 0xE0,
+ 0x2D, 0xE9, 0xF0, 0x47, 0x1C, 0x4D, 0xDF, 0xF8, 0x74, 0x90, 0xA8, 0x46, 0x20, 0x21, 0x1A, 0x48,
+ 0x07, 0xF0, 0xBF, 0xFA, 0x18, 0x48, 0x00, 0xF0, 0xE9, 0xFB, 0x47, 0x1E, 0xD9, 0xF8, 0x00, 0x00,
+ 0x98, 0xF8, 0x00, 0x60, 0x08, 0xF1, 0x01, 0x04, 0x90, 0xF8, 0xA0, 0x00, 0xC0, 0x07, 0x05, 0xD0,
+ 0x3A, 0x46, 0x21, 0x46, 0x30, 0x46, 0x00, 0xF0, 0x23, 0xF8, 0x28, 0xB9, 0x3A, 0x46, 0x21, 0x46,
+ 0x30, 0x46, 0x00, 0xF0, 0xEB, 0xF8, 0x18, 0xB1, 0x01, 0x20, 0x29, 0x68, 0x01, 0xF0, 0x9A, 0xFA,
+ 0x00, 0xF0, 0xE6, 0xFB, 0x00, 0x28, 0xD9, 0xD1, 0xBD, 0xE8, 0xF0, 0x87, 0x38, 0xB5, 0x0C, 0x46,
+ 0x05, 0x46, 0x05, 0xF0, 0x88, 0xF9, 0x69, 0x46, 0x28, 0x46, 0x05, 0xF0, 0xBE, 0xF9, 0x00, 0x99,
+ 0x21, 0x42, 0xF8, 0xD1, 0x38, 0xBD, 0x00, 0x00, 0xBC, 0x47, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x70, 0xB5, 0x0D, 0x46, 0xA0, 0xF1, 0x10, 0x01, 0x03, 0x29, 0x01, 0xD8, 0x5C, 0x48, 0x05, 0xE0,
+ 0xA0, 0xF1, 0x30, 0x01, 0x0A, 0x29, 0x06, 0xD8, 0x59, 0x48, 0x20, 0x30, 0x50, 0xF8, 0x31, 0x30,
+ 0x00, 0xEB, 0xC1, 0x00, 0x0D, 0xE0, 0x62, 0x28, 0x02, 0xD0, 0x4F, 0xF0, 0x00, 0x04, 0x10, 0xE0,
+ 0x53, 0x49, 0xA0, 0xF1, 0x62, 0x00, 0xA1, 0xF1, 0x08, 0x01, 0x51, 0xF8, 0x30, 0x30, 0x01, 0xEB,
+ 0xC0, 0x00, 0x4F, 0xF0, 0x01, 0x04, 0x40, 0x68, 0x90, 0x42, 0x02, 0xD1, 0x11, 0x46, 0x28, 0x46,
+ 0x98, 0x47, 0x20, 0x46, 0x70, 0xBD, 0x4B, 0x49, 0x01, 0x20, 0x08, 0x60, 0x04, 0x21, 0xFF, 0xF7,
+ 0xBD, 0xBF, 0x48, 0x49, 0x00, 0x20, 0x08, 0x60, 0x04, 0x21, 0x01, 0x20, 0xFF, 0xF7, 0xB6, 0xBF,
+ 0x05, 0xF0, 0xA1, 0xBE, 0x10, 0xB5, 0xF0, 0xF7, 0xC1, 0xFF, 0x43, 0x49, 0x0E, 0x20, 0x08, 0x60,
+ 0xF0, 0xF7, 0xBA, 0xFF, 0x02, 0x21, 0x05, 0x20, 0x01, 0xF0, 0x3C, 0xFA, 0xBD, 0xE8, 0x10, 0x40,
+ 0x4F, 0xF4, 0x80, 0x71, 0x00, 0x20, 0xFF, 0xF7, 0xA1, 0xBF, 0x3C, 0x49, 0x02, 0x78, 0x0A, 0x70,
+ 0x42, 0x78, 0x4A, 0x70, 0x01, 0x78, 0x4B, 0x07, 0x39, 0x49, 0x0A, 0x68, 0x02, 0xD5, 0x42, 0xF0,
+ 0x04, 0x02, 0x01, 0xE0, 0x22, 0xF0, 0x04, 0x02, 0x0A, 0x60, 0x02, 0x78, 0x12, 0x07, 0x0A, 0x68,
+ 0x02, 0xD5, 0x42, 0xF0, 0x02, 0x02, 0x01, 0xE0, 0x22, 0xF0, 0x02, 0x02, 0x0A, 0x60, 0x02, 0x78,
+ 0xD2, 0x06, 0x0A, 0x68, 0x02, 0xD5, 0x42, 0xF0, 0x01, 0x02, 0x01, 0xE0, 0x22, 0xF0, 0x01, 0x02,
+ 0x0A, 0x60, 0x00, 0x78, 0x40, 0x06, 0x02, 0xD5, 0x42, 0xF0, 0x10, 0x00, 0x01, 0xE0, 0x22, 0xF0,
+ 0x10, 0x00, 0x08, 0x60, 0x08, 0x21, 0x01, 0x20, 0xFF, 0xF7, 0x70, 0xBF, 0x00, 0x78, 0x28, 0xB1,
+ 0x01, 0x28, 0x05, 0xD1, 0x10, 0x21, 0x01, 0x20, 0xFF, 0xF7, 0x68, 0xBF, 0x04, 0x21, 0xFA, 0xE7,
+ 0x70, 0x47, 0x02, 0x78, 0x1E, 0x48, 0x01, 0x68, 0x22, 0xB1, 0x01, 0x2A, 0x05, 0xD0, 0x02, 0x2A,
+ 0x08, 0xD1, 0x0B, 0xE0, 0x21, 0xF0, 0x08, 0x01, 0x01, 0xE0, 0x41, 0xF0, 0x08, 0x01, 0x21, 0xF0,
+ 0x40, 0x01, 0x01, 0x60, 0x08, 0x21, 0x01, 0x20, 0xFF, 0xF7, 0x50, 0xBF, 0x21, 0xF0, 0x08, 0x01,
+ 0x41, 0xF0, 0x40, 0x01, 0xF5, 0xE7, 0x70, 0x47, 0x70, 0x47, 0x70, 0x47, 0x70, 0x47, 0x70, 0x47,
+ 0x70, 0x47, 0x00, 0x21, 0x00, 0x78, 0x0A, 0x46, 0x83, 0x07, 0x01, 0xD5, 0x20, 0x21, 0x00, 0x22,
+ 0xC0, 0x07, 0x01, 0xD0, 0x41, 0xF0, 0x02, 0x01, 0x0A, 0x48, 0xC0, 0xE9, 0x00, 0x12, 0x0A, 0x4A,
+ 0x00, 0x20, 0xC2, 0xE9, 0x00, 0x00, 0x70, 0x47, 0x70, 0x47, 0x00, 0xF0, 0xBF, 0xBC, 0x00, 0x00,
+ 0x74, 0x6C, 0x01, 0x00, 0x44, 0x07, 0x10, 0x00, 0x28, 0x07, 0x10, 0x00, 0xDC, 0x48, 0x10, 0x00,
+ 0x48, 0x07, 0x10, 0x00, 0x48, 0x05, 0x10, 0x00, 0x50, 0x05, 0x10, 0x00, 0x0B, 0x46, 0x10, 0xB5,
+ 0x00, 0x21, 0xA0, 0x38, 0x09, 0x28, 0x06, 0xD8, 0xFE, 0x49, 0x51, 0xF8, 0x20, 0x40, 0x11, 0x46,
+ 0x18, 0x46, 0xA0, 0x47, 0x01, 0x46, 0x08, 0x46, 0x10, 0xBD, 0x00, 0x20, 0x70, 0x47, 0x70, 0xB5,
+ 0x00, 0x23, 0x00, 0x29, 0x11, 0xDD, 0x02, 0x78, 0x49, 0x1E, 0x03, 0x2A, 0x0D, 0xD2, 0xF5, 0x4D,
+ 0x40, 0x1C, 0x28, 0x35, 0x55, 0xF8, 0x32, 0x40, 0x05, 0xEB, 0xC2, 0x02, 0x52, 0x68, 0x00, 0x2A,
+ 0x03, 0xDB, 0x91, 0x42, 0x01, 0xD1, 0xA0, 0x47, 0x03, 0x46, 0x18, 0x46, 0x70, 0xBD, 0x10, 0xB5,
+ 0x00, 0x78, 0x00, 0x21, 0xC0, 0x07, 0x00, 0xD0, 0x01, 0x21, 0xEB, 0x48, 0x02, 0x68, 0x8A, 0x42,
+ 0x04, 0xD0, 0x01, 0x60, 0x04, 0x21, 0x01, 0x20, 0xFF, 0xF7, 0xE8, 0xFE, 0x01, 0x20, 0x10, 0xBD,
+ 0x10, 0xB5, 0x10, 0x21, 0x01, 0x20, 0xFF, 0xF7, 0xE1, 0xFE, 0xE3, 0x49, 0x00, 0x20, 0x08, 0x60,
+ 0x01, 0x20, 0x10, 0xBD, 0x70, 0xB5, 0x00, 0x23, 0x00, 0x29, 0x0F, 0xDD, 0x02, 0x78, 0x49, 0x1E,
+ 0x05, 0x2A, 0x0B, 0xD2, 0xDB, 0x4D, 0x40, 0x1C, 0x40, 0x35, 0x55, 0xF8, 0x32, 0x40, 0x05, 0xEB,
+ 0xC2, 0x02, 0x52, 0x68, 0x91, 0x42, 0x01, 0xD1, 0xA0, 0x47, 0x03, 0x46, 0x18, 0x46, 0x70, 0xBD,
+ 0x10, 0xB5, 0x00, 0x78, 0xC2, 0x07, 0xD5, 0x48, 0x01, 0x68, 0x02, 0xD0, 0x41, 0xF0, 0x02, 0x01,
+ 0x01, 0xE0, 0x21, 0xF0, 0x02, 0x01, 0x01, 0x60, 0x08, 0x21, 0x01, 0x20, 0xFF, 0xF7, 0xB6, 0xFE,
+ 0x01, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0x78, 0xC2, 0x07, 0xCC, 0x48, 0x01, 0x68, 0x02, 0xD0,
+ 0x41, 0xF0, 0x04, 0x01, 0x01, 0xE0, 0x21, 0xF0, 0x04, 0x01, 0x01, 0x60, 0x08, 0x21, 0x01, 0x20,
+ 0xFF, 0xF7, 0xA4, 0xFE, 0x01, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x01, 0x78, 0xCB, 0x07, 0xC3, 0x49,
+ 0x0A, 0x68, 0x02, 0xD0, 0x42, 0xF0, 0x08, 0x02, 0x01, 0xE0, 0x22, 0xF0, 0x08, 0x02, 0x0A, 0x60,
+ 0x00, 0x78, 0x80, 0x07, 0x02, 0xD5, 0x42, 0xF0, 0x40, 0x00, 0x01, 0xE0, 0x22, 0xF0, 0x40, 0x00,
+ 0x08, 0x60, 0x08, 0x21, 0x01, 0x20, 0xFF, 0xF7, 0x89, 0xFE, 0x01, 0x20, 0x10, 0xBD, 0x10, 0xB5,
+ 0x00, 0x21, 0x03, 0x78, 0x0A, 0x46, 0x9C, 0x07, 0x01, 0xD5, 0x20, 0x21, 0x00, 0x22, 0x5C, 0x07,
+ 0x01, 0xD5, 0x41, 0xF0, 0x02, 0x01, 0x1C, 0x07, 0x01, 0xD5, 0x41, 0xF0, 0x04, 0x01, 0xDC, 0x06,
+ 0x01, 0xD5, 0x41, 0xF0, 0x08, 0x01, 0x9B, 0x06, 0x01, 0xD5, 0x41, 0xF0, 0x10, 0x01, 0x43, 0x78,
+ 0x5C, 0x07, 0x01, 0xD5, 0x41, 0xF0, 0x40, 0x01, 0x1C, 0x07, 0x01, 0xD5, 0x41, 0xF0, 0x80, 0x01,
+ 0xDC, 0x06, 0x01, 0xD5, 0x41, 0xF4, 0x80, 0x71, 0x9C, 0x06, 0x01, 0xD5, 0x41, 0xF4, 0x00, 0x71,
+ 0x5C, 0x06, 0x01, 0xD5, 0x41, 0xF4, 0x80, 0x61, 0x1B, 0x06, 0x01, 0xD5, 0x41, 0xF4, 0x00, 0x61,
+ 0x80, 0x78, 0xC3, 0x07, 0x01, 0xD0, 0x41, 0xF4, 0x80, 0x51, 0x83, 0x07, 0x01, 0xD5, 0x41, 0xF4,
+ 0x00, 0x51, 0x43, 0x07, 0x01, 0xD5, 0x41, 0xF4, 0x80, 0x41, 0x03, 0x07, 0x01, 0xD5, 0x41, 0xF4,
+ 0x00, 0x41, 0xC3, 0x06, 0x01, 0xD5, 0x41, 0xF4, 0x80, 0x31, 0x83, 0x06, 0x01, 0xD5, 0x41, 0xF4,
+ 0x00, 0x31, 0x43, 0x06, 0x01, 0xD5, 0x41, 0xF4, 0x80, 0x21, 0x00, 0x06, 0x01, 0xD5, 0x41, 0xF4,
+ 0x00, 0x21, 0x8F, 0x48, 0xC0, 0xE9, 0x00, 0x12, 0x01, 0x20, 0x10, 0xBD, 0x00, 0x20, 0x70, 0x47,
+ 0x70, 0xB5, 0x00, 0x23, 0x00, 0x29, 0x0F, 0xDD, 0x02, 0x78, 0x49, 0x1E, 0x0B, 0x2A, 0x0B, 0xD2,
+ 0x84, 0x4D, 0x40, 0x1C, 0x68, 0x35, 0x55, 0xF8, 0x32, 0x40, 0x05, 0xEB, 0xC2, 0x02, 0x52, 0x68,
+ 0x91, 0x42, 0x01, 0xD1, 0xA0, 0x47, 0x03, 0x46, 0x18, 0x46, 0x70, 0xBD, 0x01, 0x46, 0x00, 0x20,
+ 0x09, 0x78, 0x05, 0x29, 0x06, 0xD2, 0x7B, 0x48, 0xC0, 0x30, 0x50, 0xF8, 0x21, 0x20, 0x00, 0x21,
+ 0x08, 0x46, 0x10, 0x47, 0x70, 0x47, 0x10, 0xB5, 0x05, 0xF0, 0xF5, 0xFC, 0x01, 0x20, 0x10, 0xBD,
+ 0x10, 0xB5, 0x00, 0xF0, 0x93, 0xFB, 0x01, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x4F, 0xF4, 0x00, 0x61,
+ 0x00, 0x20, 0xFF, 0xF7, 0xFB, 0xFD, 0x01, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x4F, 0xF4, 0x80, 0x51,
+ 0x00, 0x20, 0xFF, 0xF7, 0xF3, 0xFD, 0x01, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x4F, 0xF4, 0x00, 0x01,
+ 0x00, 0x20, 0xFF, 0xF7, 0xEB, 0xFD, 0x01, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0x78, 0xC0, 0x07,
+ 0x02, 0xD0, 0x00, 0xF0, 0x02, 0xFB, 0x01, 0xE0, 0x00, 0xF0, 0xF6, 0xFA, 0x01, 0x20, 0x10, 0xBD,
+ 0x10, 0xB5, 0x00, 0x88, 0x00, 0x24, 0xC1, 0x07, 0x00, 0xD0, 0x02, 0x24, 0x81, 0x07, 0x01, 0xD5,
+ 0x44, 0xF0, 0x04, 0x04, 0x40, 0x07, 0x01, 0xD5, 0x44, 0xF0, 0x08, 0x04, 0x94, 0xB1, 0xF0, 0xF7,
+ 0xDD, 0xFD, 0x5C, 0x48, 0x01, 0x68, 0x21, 0x43, 0x01, 0x60, 0xF0, 0xF7, 0xD5, 0xFD, 0x02, 0x21,
+ 0x05, 0x20, 0x01, 0xF0, 0x57, 0xF8, 0x4F, 0xF4, 0x80, 0x71, 0x00, 0x20, 0xFF, 0xF7, 0xBE, 0xFD,
+ 0x01, 0x20, 0x10, 0xBD, 0x00, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0x88, 0x00, 0x24, 0xC1, 0x07,
+ 0x00, 0xD0, 0x01, 0x24, 0x81, 0x07, 0x01, 0xD5, 0x44, 0xF0, 0x02, 0x04, 0x41, 0x07, 0x01, 0xD5,
+ 0x44, 0xF0, 0x04, 0x04, 0x00, 0x07, 0x01, 0xD5, 0x44, 0xF0, 0x08, 0x04, 0x6C, 0xB1, 0xF0, 0xF7,
+ 0xB5, 0xFD, 0x49, 0x48, 0x01, 0x68, 0x21, 0x43, 0x01, 0x60, 0xF0, 0xF7, 0xAD, 0xFD, 0x02, 0x21,
+ 0x00, 0x20, 0xFF, 0xF7, 0x9B, 0xFD, 0x01, 0x20, 0x10, 0xBD, 0x00, 0x20, 0x10, 0xBD, 0x10, 0xB5,
+ 0x00, 0x88, 0x00, 0x24, 0xC1, 0x07, 0x00, 0xD0, 0x01, 0x24, 0x81, 0x07, 0x01, 0xD5, 0x44, 0xF0,
+ 0x02, 0x04, 0x41, 0x07, 0x01, 0xD5, 0x44, 0xF0, 0x04, 0x04, 0x00, 0x07, 0x01, 0xD5, 0x44, 0xF0,
+ 0x08, 0x04, 0x74, 0xB1, 0xF0, 0xF7, 0x92, 0xFD, 0x38, 0x48, 0x01, 0x68, 0x21, 0x43, 0x01, 0x60,
+ 0xF0, 0xF7, 0x8A, 0xFD, 0x4F, 0xF4, 0x00, 0x11, 0x00, 0x20, 0xFF, 0xF7, 0x77, 0xFD, 0x01, 0x20,
+ 0x10, 0xBD, 0x00, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0x88, 0x00, 0x24, 0xC1, 0x07, 0x00, 0xD0,
+ 0x01, 0x24, 0x81, 0x07, 0x01, 0xD5, 0x44, 0xF0, 0x02, 0x04, 0x41, 0x07, 0x01, 0xD5, 0x44, 0xF0,
+ 0x04, 0x04, 0x01, 0x07, 0x01, 0xD5, 0x44, 0xF0, 0x08, 0x04, 0xC1, 0x06, 0x01, 0xD5, 0x44, 0xF0,
+ 0x10, 0x04, 0x81, 0x06, 0x01, 0xD5, 0x44, 0xF0, 0x20, 0x04, 0x41, 0x06, 0x01, 0xD5, 0x44, 0xF0,
+ 0x40, 0x04, 0x01, 0x06, 0x01, 0xD5, 0x44, 0xF0, 0x80, 0x04, 0xC1, 0x05, 0x01, 0xD5, 0x44, 0xF4,
+ 0x80, 0x74, 0x81, 0x05, 0x01, 0xD5, 0x44, 0xF4, 0x00, 0x74, 0x40, 0x05, 0x01, 0xD5, 0x44, 0xF4,
+ 0x80, 0x64, 0x6C, 0xB1, 0xF0, 0xF7, 0x52, 0xFD, 0x19, 0x48, 0x01, 0x68, 0x21, 0x43, 0x01, 0x60,
+ 0xF0, 0xF7, 0x4A, 0xFD, 0x80, 0x21, 0x00, 0x20, 0xFF, 0xF7, 0x38, 0xFD, 0x01, 0x20, 0x10, 0xBD,
+ 0x00, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0x88, 0x00, 0x24, 0xC1, 0x07, 0x00, 0xD0, 0x01, 0x24,
+ 0x81, 0x07, 0x01, 0xD5, 0x44, 0xF0, 0x02, 0x04, 0x40, 0x07, 0x01, 0xD5, 0x44, 0xF0, 0x04, 0x04,
+ 0x0C, 0xB3, 0xF0, 0xF7, 0x33, 0xFD, 0x0B, 0x48, 0x01, 0x68, 0x21, 0x43, 0x01, 0x60, 0xF0, 0xF7,
+ 0x2B, 0xFD, 0x11, 0xE0, 0xEC, 0x6C, 0x01, 0x00, 0x44, 0x07, 0x10, 0x00, 0x48, 0x07, 0x10, 0x00,
+ 0x48, 0x05, 0x10, 0x00, 0x28, 0x07, 0x10, 0x00, 0x2C, 0x07, 0x10, 0x00, 0x30, 0x07, 0x10, 0x00,
+ 0x38, 0x07, 0x10, 0x00, 0x3C, 0x07, 0x10, 0x00, 0x4F, 0xF4, 0x80, 0x01, 0x00, 0x20, 0xFF, 0xF7,
+ 0x05, 0xFD, 0x01, 0x20, 0x10, 0xBD, 0x00, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0x78, 0x00, 0x24,
+ 0xC1, 0x07, 0x00, 0xD0, 0x01, 0x24, 0x81, 0x07, 0x01, 0xD5, 0x44, 0xF0, 0x02, 0x04, 0x40, 0x07,
+ 0x01, 0xD5, 0x44, 0xF0, 0x04, 0x04, 0x74, 0xB1, 0xF0, 0xF7, 0x00, 0xFD, 0x2B, 0x48, 0x01, 0x68,
+ 0x21, 0x43, 0x01, 0x60, 0xF0, 0xF7, 0xF8, 0xFC, 0x4F, 0xF4, 0x80, 0x61, 0x00, 0x20, 0xFF, 0xF7,
+ 0xE5, 0xFC, 0x01, 0x20, 0x10, 0xBD, 0x00, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0x78, 0x00, 0x24,
+ 0x51, 0x28, 0x3D, 0xD0, 0x18, 0xDC, 0x30, 0x28, 0x36, 0xD0, 0x0C, 0xDC, 0x11, 0x28, 0x2F, 0xD0,
+ 0x04, 0xDC, 0x01, 0x28, 0x29, 0xD0, 0x10, 0x28, 0x30, 0xD1, 0x29, 0xE0, 0x12, 0x28, 0x27, 0xD0,
+ 0x13, 0x28, 0x2B, 0xD1, 0x24, 0xE0, 0x31, 0x28, 0x26, 0xD0, 0x32, 0x28, 0x24, 0xD0, 0x33, 0x28,
+ 0x22, 0xD0, 0x50, 0x28, 0x22, 0xD1, 0x23, 0xE0, 0x73, 0x28, 0x24, 0xD0, 0x0C, 0xDC, 0x70, 0x28,
+ 0x21, 0xD0, 0x04, 0xDC, 0x52, 0x28, 0x1B, 0xD0, 0x53, 0x28, 0x17, 0xD1, 0x18, 0xE0, 0x71, 0x28,
+ 0x19, 0xD0, 0x72, 0x28, 0x12, 0xD1, 0x16, 0xE0, 0x74, 0x28, 0x14, 0xD0, 0x75, 0x28, 0x12, 0xD0,
+ 0x76, 0x28, 0x10, 0xD0, 0x77, 0x28, 0x09, 0xD1, 0x0D, 0xE0, 0x00, 0xF0, 0x90, 0xFC, 0x01, 0xE0,
+ 0x00, 0xF0, 0x25, 0xFD, 0x01, 0x24, 0x01, 0xE0, 0x00, 0xF0, 0x41, 0xFD, 0x20, 0x46, 0x10, 0xBD,
+ 0x00, 0xF0, 0x2B, 0xFD, 0xF6, 0xE7, 0x00, 0xF0, 0x0B, 0xFE, 0xF3, 0xE7, 0x34, 0x07, 0x10, 0x00,
+ 0x1B, 0x49, 0x08, 0xB5, 0x1B, 0x4C, 0x00, 0x20, 0x08, 0x60, 0x20, 0x60, 0x00, 0xF0, 0x77, 0xFC,
+ 0x00, 0xF0, 0x02, 0xFF, 0x6B, 0x46, 0x00, 0x22, 0x19, 0x21, 0x03, 0x20, 0x04, 0xF0, 0x5C, 0xFE,
+ 0x00, 0x98, 0xC0, 0x07, 0x05, 0xD0, 0x01, 0x21, 0x03, 0x20, 0x04, 0xF0, 0x38, 0xFE, 0xFF, 0xF7,
+ 0x4F, 0xFC, 0x00, 0x98, 0x00, 0x07, 0x0B, 0xD5, 0x08, 0x21, 0x03, 0x20, 0x04, 0xF0, 0x2F, 0xFE,
+ 0x20, 0x68, 0x08, 0x21, 0x40, 0xF0, 0x08, 0x00, 0x20, 0x60, 0x01, 0x20, 0xFF, 0xF7, 0x6E, 0xFC,
+ 0x00, 0x98, 0xC0, 0x06, 0xDE, 0xD5, 0x10, 0x21, 0x03, 0x20, 0x04, 0xF0, 0x20, 0xFE, 0x20, 0x68,
+ 0x08, 0x21, 0x20, 0xF0, 0x08, 0x00, 0x20, 0x60, 0x01, 0x20, 0xFF, 0xF7, 0x5F, 0xFC, 0xD1, 0xE7,
+ 0x44, 0x07, 0x10, 0x00, 0x48, 0x07, 0x10, 0x00, 0x1F, 0x48, 0x00, 0x21, 0x01, 0x70, 0x41, 0x70,
+ 0x81, 0x70, 0x70, 0x47, 0x10, 0xB5, 0x1C, 0x4C, 0x03, 0x46, 0xA0, 0x78, 0x10, 0x28, 0x14, 0xD2,
+ 0x0F, 0x29, 0x00, 0xD9, 0x0F, 0x21, 0x19, 0x48, 0x22, 0x78, 0x00, 0xEB, 0x02, 0x10, 0x0A, 0x46,
+ 0x00, 0xF8, 0x01, 0x1B, 0x19, 0x46, 0x06, 0xF0, 0x5D, 0xFE, 0x20, 0x78, 0x40, 0x1C, 0x00, 0xF0,
+ 0x0F, 0x00, 0x20, 0x70, 0xA0, 0x78, 0x40, 0x1C, 0xA0, 0x70, 0x10, 0xBD, 0x70, 0xB5, 0x0E, 0x4C,
+ 0x00, 0x25, 0xA1, 0x78, 0x91, 0xB1, 0x0D, 0x49, 0x62, 0x78, 0x01, 0xEB, 0x02, 0x11, 0x11, 0xF8,
+ 0x01, 0x5B, 0x2A, 0x46, 0x06, 0xF0, 0x46, 0xFE, 0x60, 0x78, 0x40, 0x1C, 0x00, 0xF0, 0x0F, 0x00,
+ 0x60, 0x70, 0x72, 0xB6, 0xA0, 0x78, 0x40, 0x1E, 0xA0, 0x70, 0x62, 0xB6, 0x28, 0x46, 0x70, 0xBD,
+ 0x01, 0x48, 0x80, 0x78, 0x70, 0x47, 0x00, 0x00, 0x00, 0x07, 0x10, 0x00, 0xDC, 0x47, 0x10, 0x00,
+ 0x70, 0xB5, 0x14, 0x46, 0x00, 0x20, 0x06, 0xF0, 0x51, 0xFF, 0x00, 0xF0, 0x01, 0x05, 0x01, 0x20,
+ 0x06, 0xF0, 0x4C, 0xFF, 0x00, 0xF0, 0x01, 0x00, 0x45, 0xEA, 0x00, 0x10, 0x20, 0x70, 0x01, 0x20,
+ 0x70, 0xBD, 0x80, 0x20, 0x10, 0x70, 0x01, 0x20, 0x70, 0x47, 0x10, 0xB5, 0x14, 0x46, 0x05, 0xF0,
+ 0xCE, 0xFA, 0x53, 0x21, 0x21, 0x70, 0x54, 0x21, 0x61, 0x70, 0x01, 0x0A, 0xA1, 0x70, 0xE0, 0x70,
+ 0x05, 0xF0, 0xC9, 0xFA, 0x20, 0x71, 0x05, 0x20, 0x10, 0xBD, 0x38, 0xB5, 0x14, 0x46, 0x74, 0x4A,
+ 0x13, 0x68, 0x59, 0x8E, 0x98, 0x8E, 0x93, 0xF8, 0x36, 0x30, 0xDB, 0x07, 0x0A, 0xD0, 0x03, 0x0A,
+ 0x23, 0x70, 0x60, 0x70, 0x0D, 0x0A, 0xA5, 0x70, 0xE1, 0x70, 0x23, 0x71, 0x60, 0x71, 0xA5, 0x71,
+ 0xE1, 0x71, 0x09, 0xE0, 0x0B, 0x0A, 0x23, 0x70, 0x61, 0x70, 0x05, 0x0A, 0xA5, 0x70, 0xE0, 0x70,
+ 0x23, 0x71, 0x61, 0x71, 0xA5, 0x71, 0xE0, 0x71, 0x10, 0x68, 0x90, 0xF8, 0x31, 0x10, 0x21, 0x72,
+ 0x10, 0xF8, 0x30, 0x1F, 0x61, 0x72, 0x90, 0xF8, 0x31, 0x04, 0x8D, 0xF8, 0x00, 0x00, 0x01, 0x21,
+ 0x68, 0x46, 0x05, 0xF0, 0x1F, 0xFB, 0xA0, 0x72, 0x0B, 0x20, 0x38, 0xBD, 0x00, 0x20, 0x10, 0x70,
+ 0x10, 0x20, 0x50, 0x70, 0x5A, 0x48, 0x01, 0x68, 0x49, 0x8A, 0x09, 0x0A, 0x91, 0x70, 0x01, 0x68,
+ 0x89, 0x7C, 0xD1, 0x70, 0x00, 0x68, 0x01, 0x7E, 0x11, 0x71, 0x40, 0x7E, 0x50, 0x71, 0x06, 0x20,
+ 0x70, 0x47, 0x08, 0xB5, 0x02, 0x23, 0x00, 0x93, 0x52, 0x4B, 0x00, 0xF0, 0x8D, 0xF8, 0x02, 0x20,
+ 0x08, 0xBD, 0x08, 0xB5, 0x01, 0x23, 0x00, 0x93, 0x4E, 0x4B, 0x9B, 0x1C, 0x00, 0xF0, 0x84, 0xF8,
+ 0x01, 0x20, 0x08, 0xBD, 0x08, 0xB5, 0x01, 0x23, 0x00, 0x93, 0x4A, 0x4B, 0xDB, 0x1C, 0x00, 0xF0,
+ 0x7B, 0xF8, 0x01, 0x20, 0x08, 0xBD, 0x08, 0xB5, 0x01, 0x23, 0x00, 0x93, 0x45, 0x4B, 0x1B, 0x1D,
+ 0x00, 0xF0, 0x72, 0xF8, 0x01, 0x20, 0x08, 0xBD, 0x08, 0xB5, 0x01, 0x23, 0x00, 0x93, 0x41, 0x4B,
+ 0x5B, 0x1D, 0x00, 0xF0, 0x69, 0xF8, 0x01, 0x20, 0x08, 0xBD, 0x08, 0xB5, 0x01, 0x23, 0x00, 0x93,
+ 0x3C, 0x4B, 0x9B, 0x1D, 0x00, 0xF0, 0x60, 0xF8, 0x01, 0x20, 0x08, 0xBD, 0x08, 0xB5, 0x02, 0x23,
+ 0x00, 0x93, 0x38, 0x4B, 0xDB, 0x1D, 0x00, 0xF0, 0x57, 0xF8, 0x02, 0x20, 0x08, 0xBD, 0x08, 0xB5,
+ 0x02, 0x23, 0x00, 0x93, 0x33, 0x4B, 0x09, 0x33, 0x00, 0xF0, 0x4E, 0xF8, 0x02, 0x20, 0x08, 0xBD,
+ 0x08, 0xB5, 0x01, 0x23, 0x00, 0x93, 0x2F, 0x4B, 0x0B, 0x33, 0x00, 0xF0, 0x45, 0xF8, 0x01, 0x20,
+ 0x08, 0xBD, 0x08, 0xB5, 0x01, 0x23, 0x00, 0x93, 0x2A, 0x4B, 0x0C, 0x33, 0x00, 0xF0, 0x3C, 0xF8,
+ 0x01, 0x20, 0x08, 0xBD, 0x08, 0xB5, 0x01, 0x23, 0x00, 0x93, 0x26, 0x4B, 0x0D, 0x33, 0x00, 0xF0,
+ 0x33, 0xF8, 0x01, 0x20, 0x08, 0xBD, 0x2D, 0xE9, 0xF0, 0x41, 0x0F, 0x46, 0x04, 0x46, 0x00, 0x78,
+ 0x49, 0x1E, 0xCE, 0xB2, 0x00, 0x25, 0x63, 0x1C, 0xA0, 0xF1, 0x20, 0x01, 0x04, 0x29, 0x09, 0xD8,
+ 0x1D, 0x49, 0x01, 0xEB, 0x80, 0x00, 0x31, 0x46, 0x50, 0xF8, 0x80, 0x4C, 0x18, 0x46, 0xA0, 0x47,
+ 0x05, 0x46, 0x16, 0xE0, 0xA0, 0xF1, 0x30, 0x01, 0x0B, 0x29, 0x12, 0xD8, 0x16, 0x49, 0x14, 0x31,
+ 0x01, 0xEB, 0x80, 0x00, 0x31, 0x46, 0x50, 0xF8, 0xC0, 0x5C, 0x18, 0x46, 0xA8, 0x47, 0x05, 0x46,
+ 0x3E, 0xB1, 0x39, 0x46, 0x20, 0x46, 0xFF, 0xF7, 0xCD, 0xFE, 0x01, 0x21, 0x03, 0x20, 0x04, 0xF0,
+ 0xAA, 0xFC, 0x28, 0x46, 0xBD, 0xE8, 0xF0, 0x81, 0x70, 0xB5, 0x1D, 0x46, 0x04, 0x9C, 0x16, 0x46,
+ 0x39, 0xB1, 0xA1, 0x42, 0x00, 0xD9, 0x21, 0x46, 0x0A, 0x46, 0x01, 0x46, 0x18, 0x46, 0x06, 0xF0,
+ 0x29, 0xFD, 0x22, 0x46, 0x29, 0x46, 0x30, 0x46, 0xBD, 0xE8, 0x70, 0x40, 0x06, 0xF0, 0x22, 0xBD,
+ 0x20, 0x07, 0x10, 0x00, 0xDC, 0x48, 0x10, 0x00, 0xC0, 0x6D, 0x01, 0x00, 0x10, 0xB5, 0x4F, 0xF4,
+ 0x86, 0x71, 0x4F, 0x48, 0x06, 0xF0, 0xB7, 0xFD, 0x4D, 0x49, 0x01, 0x20, 0xC1, 0xF8, 0x04, 0x01,
+ 0x4C, 0x48, 0x00, 0x21, 0x41, 0x60, 0x01, 0x70, 0x06, 0xF0, 0x3E, 0xFE, 0x00, 0x21, 0xBD, 0xE8,
+ 0x10, 0x40, 0x01, 0x20, 0x05, 0xF0, 0x5B, 0xBD, 0x10, 0xB5, 0x72, 0xB6, 0x00, 0x20, 0x45, 0x49,
+ 0x08, 0x70, 0x06, 0xF0, 0x31, 0xFE, 0x62, 0xB6, 0x10, 0xBD, 0x10, 0xB5, 0x72, 0xB6, 0x01, 0x20,
+ 0x40, 0x49, 0x08, 0x70, 0x3E, 0x48, 0xD0, 0xF8, 0x08, 0x01, 0x08, 0xB1, 0x06, 0xF0, 0xC8, 0xFD,
+ 0x62, 0xB6, 0x10, 0xBD, 0x10, 0xB5, 0x72, 0xB6, 0x39, 0x49, 0x3A, 0x4B, 0x0A, 0x46, 0xD1, 0xF8,
+ 0x08, 0x41, 0xD2, 0xF8, 0x00, 0x21, 0x52, 0x1C, 0x20, 0x2C, 0x07, 0xD2, 0x64, 0x1C, 0x22, 0xF0,
+ 0x20, 0x02, 0xC1, 0xF8, 0x08, 0x41, 0xC1, 0xF8, 0x00, 0x21, 0x0F, 0xE0, 0xD1, 0xF8, 0x04, 0x41,
+ 0x22, 0xF0, 0x20, 0x02, 0x64, 0x1C, 0xC1, 0xF8, 0x00, 0x21, 0x24, 0xF0, 0x20, 0x02, 0xC1, 0xF8,
+ 0x04, 0x21, 0x5A, 0x68, 0x52, 0x1C, 0x5A, 0x60, 0xD1, 0xF8, 0x00, 0x21, 0x04, 0x68, 0x01, 0xEB,
+ 0xC2, 0x02, 0x14, 0x60, 0x40, 0x68, 0x50, 0x60, 0x18, 0x78, 0x01, 0x28, 0x05, 0xD1, 0xD1, 0xF8,
+ 0x08, 0x01, 0x01, 0x28, 0x01, 0xD1, 0x06, 0xF0, 0x93, 0xFD, 0x62, 0xB6, 0x10, 0xBD, 0x20, 0x4A,
+ 0xD2, 0xF8, 0x08, 0x01, 0x00, 0x28, 0x0E, 0xD0, 0xD2, 0xF8, 0x04, 0x01, 0x02, 0xEB, 0xC0, 0x00,
+ 0xC3, 0x79, 0x23, 0xF0, 0x3F, 0x03, 0xC3, 0x71, 0x92, 0xF8, 0x08, 0x21, 0x52, 0x1E, 0x02, 0xF0,
+ 0x3F, 0x02, 0x13, 0x43, 0xC3, 0x71, 0x70, 0x47, 0x10, 0xB5, 0x72, 0xB6, 0x14, 0x48, 0xD0, 0xF8,
+ 0x08, 0x11, 0x59, 0xB1, 0x49, 0x1E, 0xC0, 0xF8, 0x08, 0x11, 0xD0, 0xF8, 0x04, 0x21, 0x02, 0xF1,
+ 0x01, 0x02, 0x22, 0xF0, 0x20, 0x02, 0xC0, 0xF8, 0x04, 0x21, 0x05, 0xD1, 0x0D, 0x48, 0x00, 0x78,
+ 0x01, 0x28, 0x01, 0xD1, 0x06, 0xF0, 0xC0, 0xFD, 0x62, 0xB6, 0x10, 0xBD, 0x10, 0xB5, 0x72, 0xB6,
+ 0x00, 0x20, 0x07, 0x49, 0x01, 0x22, 0xC1, 0xF8, 0x08, 0x01, 0xC1, 0xE9, 0x40, 0x02, 0x05, 0x49,
+ 0x48, 0x60, 0x08, 0x78, 0x01, 0x28, 0x01, 0xD1, 0x06, 0xF0, 0xAE, 0xFD, 0x62, 0xB6, 0x10, 0xBD,
+ 0x40, 0x42, 0x01, 0x20, 0x04, 0x07, 0x10, 0x00, 0x18, 0x49, 0x00, 0x22, 0x0A, 0x80, 0x4A, 0x80,
+ 0x4F, 0xF4, 0x70, 0x42, 0x8A, 0x80, 0x16, 0x4A, 0xCA, 0x80, 0x16, 0x4A, 0x0A, 0x81, 0x16, 0x4A,
+ 0xC0, 0xE9, 0x00, 0x21, 0x05, 0x21, 0x01, 0x72, 0x70, 0x47, 0x1C, 0xB5, 0x12, 0x49, 0xAD, 0xF8,
+ 0x04, 0x00, 0x02, 0x20, 0x8D, 0xF8, 0x06, 0x00, 0x04, 0x20, 0x09, 0x1F, 0x8D, 0xF8, 0x07, 0x00,
+ 0x00, 0x91, 0x68, 0x46, 0x05, 0xF0, 0x59, 0xFE, 0x1C, 0xBD, 0x30, 0xB4, 0x03, 0x46, 0x0A, 0x4A,
+ 0x00, 0x20, 0x14, 0x78, 0x9C, 0x42, 0x02, 0xD1, 0x30, 0xBC, 0x05, 0xF0, 0x84, 0xBE, 0x40, 0x1C,
+ 0xC0, 0xB2, 0x52, 0x1D, 0x05, 0x28, 0xF4, 0xD3, 0x30, 0xBC, 0x70, 0x47, 0xEA, 0x48, 0x10, 0x00,
+ 0x4C, 0x62, 0x01, 0x20, 0xD0, 0x5E, 0x01, 0x20, 0x04, 0x6E, 0x01, 0x00, 0x10, 0xB5, 0x88, 0xB3,
+ 0xA3, 0x49, 0xA4, 0x4A, 0x09, 0x78, 0x37, 0x29, 0x71, 0xD0, 0x11, 0xDC, 0xA1, 0xF1, 0x20, 0x01,
+ 0x17, 0x29, 0x2C, 0xD2, 0xDF, 0xE8, 0x01, 0xF0, 0x6B, 0x6B, 0x6B, 0x6B, 0x6B, 0x2B, 0x2B, 0x2B,
+ 0x2B, 0x2B, 0x2B, 0x2B, 0x2B, 0x2B, 0x2B, 0x2B, 0x6B, 0x6B, 0x6B, 0x6B, 0x6B, 0x6B, 0x6B, 0x00,
+ 0x97, 0x4B, 0x70, 0x29, 0x5B, 0x78, 0x26, 0xD0, 0x0C, 0xDC, 0x3B, 0x29, 0x57, 0xD0, 0x04, 0xDC,
+ 0x38, 0x29, 0x54, 0xD0, 0x3A, 0x29, 0x12, 0xD1, 0x51, 0xE0, 0x60, 0x29, 0x19, 0xD0, 0x61, 0x29,
+ 0x0D, 0xD1, 0x16, 0xE0, 0x73, 0x29, 0x28, 0xD0, 0x05, 0xDC, 0x71, 0x29, 0x15, 0xD0, 0x72, 0x29,
+ 0x05, 0xD1, 0x14, 0xE0, 0x0B, 0xE0, 0x85, 0x29, 0x0B, 0xD0, 0x86, 0x29, 0x09, 0xD0, 0xC1, 0xB2,
+ 0x87, 0x48, 0xFF, 0xF7, 0x8F, 0xFD, 0x01, 0x21, 0x03, 0x20, 0x04, 0xF0, 0x6C, 0xFB, 0x00, 0x20,
+ 0x10, 0xBD, 0x02, 0x20, 0x10, 0xBD, 0x93, 0x70, 0xF9, 0xE7, 0xD3, 0x70, 0xF7, 0xE7, 0x90, 0x78,
+ 0x01, 0x28, 0x08, 0xD0, 0x00, 0x28, 0xF2, 0xD1, 0x7F, 0x48, 0x81, 0xB2, 0x5F, 0xF0, 0x72, 0x00,
+ 0xFF, 0xF7, 0x93, 0xFF, 0xEB, 0xE7, 0x7D, 0x48, 0xF7, 0xE7, 0xD0, 0x78, 0x01, 0x28, 0x05, 0xD0,
+ 0x00, 0x28, 0xE4, 0xD1, 0x7A, 0x48, 0x81, 0xB2, 0x73, 0x20, 0xF1, 0xE7, 0x79, 0x4C, 0x7A, 0x49,
+ 0x20, 0x68, 0x90, 0xF8, 0x31, 0x00, 0x42, 0x00, 0x71, 0x48, 0x10, 0x30, 0x06, 0xF0, 0xD2, 0xFB,
+ 0x20, 0x68, 0x10, 0xF8, 0x30, 0x1F, 0x4A, 0x00, 0x6D, 0x49, 0x40, 0x78, 0x10, 0x31, 0x01, 0xEB,
+ 0x40, 0x00, 0x72, 0x49, 0x06, 0xF0, 0xC6, 0xFB, 0x69, 0x48, 0x10, 0x30, 0xE3, 0xE7, 0xC1, 0xB2,
+ 0x52, 0x68, 0x67, 0x48, 0xFF, 0xF7, 0x57, 0xFE, 0x6D, 0x49, 0x08, 0x70, 0x01, 0x20, 0x10, 0xBD,
+ 0x2D, 0xE9, 0xF0, 0x5F, 0xDF, 0xF8, 0x88, 0xB1, 0x62, 0x4E, 0x62, 0x4D, 0x89, 0x46, 0x07, 0x46,
+ 0x4F, 0xF0, 0x01, 0x0A, 0xAB, 0xF1, 0x10, 0x0B, 0x08, 0x36, 0x28, 0x78, 0xA8, 0x46, 0x04, 0x46,
+ 0x07, 0x28, 0x76, 0xD2, 0xDF, 0xE8, 0x00, 0xF0, 0x04, 0x21, 0x25, 0x40, 0x5C, 0x78, 0x8A, 0x00,
+ 0x01, 0x2F, 0x02, 0xD0, 0x02, 0x2F, 0x0C, 0xD0, 0x6B, 0xE0, 0x48, 0x46, 0xFF, 0xF7, 0x5E, 0xFF,
+ 0x02, 0x28, 0x02, 0xD0, 0x01, 0x28, 0x02, 0xD0, 0x63, 0xE0, 0x03, 0x24, 0x61, 0xE0, 0x02, 0x24,
+ 0x5F, 0xE0, 0x68, 0x78, 0x01, 0x28, 0x04, 0xD0, 0x4D, 0x48, 0x10, 0x21, 0x10, 0x38, 0x06, 0xF0,
+ 0x08, 0xFC, 0x54, 0x46, 0x86, 0xF8, 0x00, 0xA0, 0x53, 0xE0, 0x02, 0x2F, 0x4F, 0xD0, 0x00, 0x24,
+ 0x39, 0xE0, 0x02, 0x2F, 0x1A, 0xD1, 0x4E, 0x4C, 0x45, 0x48, 0xD8, 0xF8, 0x04, 0x10, 0x22, 0x78,
+ 0x10, 0x38, 0x06, 0xF0, 0x77, 0xFB, 0x03, 0x20, 0x30, 0x70, 0x20, 0x78, 0xB0, 0x80, 0x98, 0xF8,
+ 0x00, 0x00, 0x04, 0x24, 0x88, 0xF8, 0x01, 0x00, 0x88, 0xF8, 0x00, 0x40, 0x27, 0xE0, 0x00, 0x24,
+ 0x88, 0xF8, 0x01, 0x00, 0x2C, 0x70, 0xB8, 0xE7, 0x02, 0x2F, 0x02, 0xD0, 0x01, 0x2F, 0xF6, 0xD0,
+ 0x2F, 0xE0, 0xFF, 0xF7, 0xA4, 0xFE, 0x08, 0x21, 0x28, 0xB1, 0x02, 0x22, 0x32, 0x70, 0x70, 0x80,
+ 0xB1, 0x80, 0x05, 0x24, 0x25, 0xE0, 0x68, 0x78, 0x06, 0x28, 0x04, 0xD0, 0x00, 0x22, 0xCB, 0xF8,
+ 0x00, 0x20, 0xCB, 0xF8, 0x04, 0x20, 0x03, 0x20, 0x30, 0x70, 0xB1, 0x80, 0x06, 0x24, 0x18, 0xE0,
+ 0x02, 0x2F, 0x08, 0xD0, 0x02, 0x24, 0xC0, 0xB2, 0x88, 0xF8, 0x01, 0x00, 0x2C, 0x70, 0x29, 0x48,
+ 0x08, 0x30, 0xBD, 0xE8, 0xF0, 0x9F, 0x68, 0x78, 0x04, 0x28, 0x08, 0xD0, 0x0B, 0xEB, 0x09, 0x00,
+ 0x49, 0x46, 0x10, 0xF8, 0x01, 0x2C, 0x22, 0x48, 0x10, 0x38, 0x06, 0xF0, 0xAA, 0xFB, 0x86, 0xF8,
+ 0x00, 0xA0, 0xA8, 0x46, 0x28, 0x78, 0xE7, 0xE7, 0x02, 0x2F, 0x05, 0xD0, 0xB9, 0xF1, 0x08, 0x0F,
+ 0x93, 0xD1, 0xFF, 0xF7, 0x81, 0xFE, 0x90, 0xE7, 0xFF, 0xF7, 0x7E, 0xFE, 0x98, 0xF8, 0x00, 0x00,
+ 0x03, 0x24, 0x88, 0xF8, 0x01, 0x00, 0x88, 0xF8, 0x00, 0x40, 0x6E, 0xE7, 0x02, 0x2F, 0x4F, 0xF0,
+ 0x03, 0x04, 0xAD, 0xD0, 0xCF, 0xE7, 0x13, 0x49, 0x00, 0x22, 0x0A, 0x70, 0x4A, 0x70, 0x18, 0x4A,
+ 0x52, 0x1C, 0x4A, 0x60, 0x17, 0x49, 0x01, 0x60, 0x0D, 0x49, 0x10, 0x39, 0x81, 0x80, 0x10, 0x31,
+ 0xC1, 0x80, 0x10, 0x21, 0x01, 0x81, 0x0E, 0x21, 0x13, 0x48, 0x06, 0xF0, 0x82, 0xBB, 0x10, 0xB5,
+ 0x04, 0x46, 0x06, 0xF0, 0x31, 0xFC, 0x06, 0x48, 0x21, 0x68, 0x10, 0x38, 0x01, 0x60, 0x61, 0x68,
+ 0x41, 0x60, 0x0E, 0x48, 0xFF, 0xF7, 0x91, 0xFE, 0xBD, 0xE8, 0x10, 0x40, 0x06, 0xF0, 0xC8, 0xBB,
+ 0x5C, 0x43, 0x01, 0x20, 0x0C, 0x07, 0x10, 0x00, 0x4C, 0x62, 0x01, 0x20, 0xC8, 0x35, 0x01, 0x20,
+ 0xD0, 0x5E, 0x01, 0x20, 0x20, 0x07, 0x10, 0x00, 0x36, 0x0B, 0x01, 0x20, 0x60, 0x0B, 0x01, 0x20,
+ 0xF4, 0x48, 0x10, 0x00, 0x51, 0x00, 0x01, 0x00, 0xDC, 0x48, 0x10, 0x00, 0x4C, 0x43, 0x00, 0x00,
+ 0x70, 0xB5, 0x0F, 0x4D, 0x48, 0x21, 0x28, 0x46, 0x2C, 0x88, 0x06, 0xF0, 0x52, 0xFB, 0x64, 0x1C,
+ 0x2C, 0x80, 0x70, 0xBD, 0x0A, 0x48, 0x00, 0x1F, 0x00, 0x21, 0x01, 0x60, 0x70, 0x47, 0x08, 0x49,
+ 0x09, 0x1F, 0x0A, 0x88, 0x02, 0x42, 0x04, 0xD0, 0xC8, 0x78, 0x18, 0xB1, 0xC8, 0x78, 0x40, 0x1E,
+ 0xC8, 0x70, 0x70, 0x47, 0xA5, 0x20, 0x88, 0x70, 0x88, 0x78, 0xA5, 0x28, 0xFC, 0xD0, 0x70, 0x47,
+ 0xE6, 0x43, 0x01, 0x20, 0x37, 0x48, 0x00, 0x21, 0x01, 0x60, 0x41, 0x60, 0x70, 0x47, 0xC0, 0xB2,
+ 0x01, 0x22, 0x00, 0xF0, 0x29, 0xFB, 0xFE, 0xE7, 0x34, 0x4C, 0x00, 0x25, 0x32, 0x4F, 0xC4, 0xE9,
+ 0x02, 0x25, 0xC4, 0xE9, 0x00, 0x71, 0x06, 0x46, 0x00, 0x68, 0x20, 0x61, 0x70, 0x68, 0x60, 0x61,
+ 0xB0, 0x68, 0xA0, 0x61, 0xF0, 0x68, 0xE0, 0x61, 0x30, 0x69, 0x20, 0x62, 0x70, 0x69, 0x60, 0x62,
+ 0xB0, 0x69, 0xA0, 0x62, 0xF0, 0x69, 0xE0, 0x62, 0x29, 0x48, 0x90, 0x46, 0x89, 0x46, 0x00, 0x68,
+ 0x20, 0x63, 0x27, 0x48, 0x10, 0x38, 0x00, 0x68, 0x60, 0x63, 0x25, 0x48, 0x0C, 0x38, 0x00, 0x68,
+ 0xA0, 0x63, 0x23, 0x48, 0x08, 0x38, 0x00, 0x68, 0xE0, 0x63, 0x21, 0x48, 0x00, 0x1D, 0x00, 0x68,
+ 0x20, 0x64, 0x1F, 0x48, 0x34, 0x38, 0x00, 0x68, 0x60, 0x64, 0x1D, 0x48, 0x14, 0x38, 0x00, 0x68,
+ 0xC4, 0xE9, 0x12, 0x05, 0x04, 0xF0, 0x37, 0xFB, 0x20, 0x65, 0xC4, 0xE9, 0x15, 0x65, 0xE5, 0x65,
+ 0x25, 0x66, 0x65, 0x66, 0xA5, 0x66, 0xE5, 0x66, 0x25, 0x67, 0x65, 0x67, 0xC4, 0xE9, 0x1E, 0x57,
+ 0x41, 0x46, 0x48, 0x46, 0xFF, 0xF7, 0xB3, 0xFF, 0x10, 0xB5, 0x04, 0x46, 0xC0, 0xB2, 0x00, 0x22,
+ 0x00, 0xF0, 0xDA, 0xFA, 0x0F, 0x49, 0x08, 0x68, 0x08, 0x28, 0x03, 0xDA, 0x09, 0x4A, 0x14, 0x54,
+ 0x40, 0x1C, 0x08, 0x60, 0x10, 0xBD, 0x01, 0x28, 0x06, 0xD0, 0x02, 0x28, 0x08, 0xD1, 0x00, 0x22,
+ 0x11, 0x46, 0x13, 0x20, 0x00, 0xF0, 0xC8, 0xBA, 0x00, 0x22, 0x11, 0x46, 0x12, 0x20, 0xF9, 0xE7,
+ 0x70, 0x47, 0x00, 0x00, 0x70, 0xEF, 0x01, 0x20, 0xAF, 0x05, 0x50, 0xFA, 0x80, 0xEF, 0x01, 0x20,
+ 0x38, 0xED, 0x00, 0xE0, 0x1C, 0x07, 0x10, 0x00, 0xF4, 0x48, 0xF5, 0x4A, 0x01, 0x68, 0x91, 0x42,
+ 0x04, 0xD1, 0xF4, 0x49, 0x02, 0x79, 0x4A, 0x70, 0x00, 0x21, 0x01, 0x60, 0x70, 0x47, 0xF0, 0xB5,
+ 0x85, 0xB0, 0x40, 0xF2, 0x14, 0x71, 0xED, 0x48, 0x06, 0xF0, 0xCD, 0xFA, 0x68, 0x46, 0x05, 0xF0,
+ 0x11, 0xFA, 0xEA, 0x4C, 0x01, 0x26, 0x00, 0x25, 0x66, 0x70, 0xA5, 0x80, 0xA5, 0x71, 0x02, 0x20,
+ 0xE0, 0x71, 0x04, 0xF0, 0xD8, 0xFE, 0x20, 0x81, 0x04, 0xF0, 0xD1, 0xFE, 0x07, 0x02, 0x04, 0xF0,
+ 0xCE, 0xFE, 0x47, 0xEA, 0x10, 0x20, 0x60, 0x81, 0x10, 0x20, 0x20, 0x82, 0x40, 0xF2, 0x32, 0x20,
+ 0x60, 0x82, 0xE1, 0x48, 0x00, 0x68, 0x41, 0x8A, 0xA1, 0x82, 0x01, 0x8A, 0xE1, 0x82, 0xC1, 0x8A,
+ 0x21, 0x83, 0x81, 0x8A, 0x61, 0x83, 0xDD, 0x49, 0xB1, 0xF8, 0xFA, 0x2E, 0xA4, 0xF8, 0x44, 0x20,
+ 0xB1, 0xF8, 0xF8, 0x1E, 0xA4, 0xF8, 0x46, 0x10, 0x90, 0xF8, 0x3F, 0x10, 0x21, 0x77, 0xD8, 0x49,
+ 0x09, 0x68, 0x09, 0x7E, 0x61, 0x77, 0x90, 0xF8, 0xA0, 0x10, 0x01, 0xF0, 0x01, 0x01, 0xA1, 0x77,
+ 0x01, 0x99, 0x21, 0x62, 0x00, 0x99, 0x61, 0x62, 0x02, 0x99, 0xA1, 0x62, 0xDD, 0xE9, 0x03, 0x12,
+ 0x41, 0xEA, 0x02, 0x21, 0xE1, 0x62, 0x81, 0x69, 0x21, 0x63, 0xC1, 0x69, 0xC4, 0xE9, 0x0D, 0x15,
+ 0xE5, 0x63, 0xA4, 0xF8, 0x40, 0x50, 0x84, 0xF8, 0x42, 0x50, 0x84, 0xF8, 0x43, 0x60, 0x41, 0x8E,
+ 0xA4, 0xF8, 0x50, 0x10, 0x81, 0x8E, 0xA4, 0xF8, 0x52, 0x10, 0x90, 0xF8, 0x31, 0x10, 0x84, 0xF8,
+ 0x54, 0x10, 0x90, 0xF8, 0x30, 0x00, 0x84, 0xF8, 0x55, 0x00, 0xC2, 0x48, 0xA4, 0xF8, 0x80, 0x00,
+ 0xC1, 0x48, 0xA4, 0xF8, 0x88, 0x00, 0xC1, 0x48, 0xA4, 0xF8, 0x8A, 0x00, 0xC0, 0x48, 0xA4, 0xF8,
+ 0x8C, 0x00, 0xC0, 0x48, 0xA4, 0xF8, 0x8E, 0x00, 0xBF, 0x48, 0xA4, 0xF8, 0x90, 0x00, 0xBF, 0x48,
+ 0xA4, 0xF8, 0x92, 0x00, 0xBE, 0x48, 0xA4, 0xF8, 0x94, 0x00, 0xBE, 0x48, 0xA4, 0xF8, 0x96, 0x00,
+ 0xBD, 0x48, 0xA4, 0xF8, 0x98, 0x00, 0xBD, 0x48, 0xA4, 0xF8, 0x9A, 0x00, 0xBC, 0x48, 0xA4, 0xF8,
+ 0x9C, 0x00, 0xAC, 0x49, 0xBB, 0x48, 0xA4, 0xF8, 0x9E, 0x00, 0x88, 0x88, 0x40, 0x1C, 0x88, 0x80,
+ 0xA5, 0x22, 0x22, 0x70, 0x66, 0x70, 0x60, 0x80, 0x0E, 0x70, 0x05, 0xB0, 0xF0, 0xBD, 0x70, 0xB5,
+ 0xA4, 0x4C, 0x05, 0x46, 0x00, 0x23, 0xA0, 0x88, 0xA0, 0x49, 0x40, 0x1C, 0x82, 0xB2, 0xA2, 0x80,
+ 0x28, 0x46, 0x01, 0xF0, 0x2A, 0xFB, 0x25, 0x70, 0x70, 0xBD, 0x70, 0xB5, 0x05, 0x46, 0x40, 0xF2,
+ 0x14, 0x71, 0x9A, 0x48, 0x06, 0xF0, 0x27, 0xFA, 0x9A, 0x4C, 0x98, 0x49, 0xA0, 0x88, 0x40, 0x1C,
+ 0x82, 0xB2, 0xA2, 0x80, 0x28, 0x46, 0x01, 0xF0, 0x45, 0xFB, 0x25, 0x70, 0x70, 0xBD, 0x70, 0xB5,
+ 0x94, 0x4C, 0x05, 0x46, 0x20, 0x78, 0xA8, 0x42, 0x04, 0xD0, 0x40, 0xF2, 0x14, 0x71, 0x8F, 0x48,
+ 0x06, 0xF0, 0x11, 0xFA, 0xA0, 0x48, 0xA5, 0x70, 0x0F, 0x21, 0x01, 0x60, 0x8B, 0x48, 0x8B, 0x4A,
+ 0x10, 0x30, 0xA0, 0x60, 0x48, 0x20, 0x10, 0x71, 0x8B, 0x48, 0x02, 0xF1, 0x58, 0x05, 0xE5, 0x60,
+ 0x00, 0x68, 0x90, 0xF8, 0x31, 0x10, 0x90, 0xF8, 0x30, 0x30, 0x51, 0x71, 0x01, 0xFB, 0x03, 0xF0,
+ 0x05, 0xEB, 0x40, 0x06, 0x93, 0x71, 0x26, 0x61, 0x08, 0x44, 0xD1, 0x71, 0x05, 0xEB, 0x40, 0x00,
+ 0x60, 0x61, 0x13, 0x72, 0x70, 0xBD, 0x2D, 0xE9, 0xF0, 0x41, 0x05, 0x46, 0x7D, 0x4C, 0x17, 0x46,
+ 0x88, 0x46, 0xC0, 0x07, 0x04, 0xD0, 0x48, 0x22, 0x7E, 0x49, 0xA0, 0x68, 0x06, 0xF0, 0x42, 0xF9,
+ 0x79, 0x4E, 0xA8, 0x07, 0x12, 0xD5, 0x31, 0x68, 0x91, 0xF8, 0x31, 0x00, 0x91, 0xF8, 0x30, 0x10,
+ 0x48, 0x43, 0xA1, 0x78, 0x30, 0x29, 0x2D, 0xD0, 0x31, 0x29, 0x2D, 0xD0, 0x32, 0x29, 0x31, 0xD0,
+ 0x77, 0x49, 0xE3, 0x68, 0x42, 0x00, 0x18, 0x46, 0x06, 0xF0, 0x2C, 0xF9, 0x68, 0x07, 0x0E, 0xD5,
+ 0x30, 0x68, 0x90, 0xF8, 0x31, 0x20, 0xA0, 0x78, 0x30, 0x28, 0x25, 0xD0, 0x31, 0x28, 0x25, 0xD0,
+ 0x32, 0x28, 0x28, 0xD0, 0x72, 0x49, 0x52, 0x00, 0x20, 0x69, 0x06, 0xF0, 0x1B, 0xF9, 0x28, 0x07,
+ 0x2C, 0xD5, 0x30, 0x68, 0x90, 0xF8, 0x30, 0x20, 0xA0, 0x78, 0x30, 0x28, 0x1D, 0xD0, 0x31, 0x28,
+ 0x1D, 0xD0, 0x32, 0x28, 0x20, 0xD0, 0x6E, 0x49, 0x60, 0x69, 0xBD, 0xE8, 0xF0, 0x41, 0x52, 0x00,
+ 0x06, 0xF0, 0x08, 0xB9, 0x60, 0x49, 0xD4, 0xE7, 0x60, 0x49, 0xB8, 0xF1, 0x00, 0x0F, 0xD0, 0xD0,
+ 0x6A, 0x49, 0xCE, 0xE7, 0x5F, 0x49, 0xCC, 0xE7, 0x5F, 0x49, 0xDC, 0xE7, 0x5F, 0x49, 0x00, 0x2F,
+ 0xD9, 0xD0, 0x67, 0x49, 0xD7, 0xE7, 0x5F, 0x49, 0xD5, 0xE7, 0x5F, 0x49, 0xE4, 0xE7, 0x5F, 0x49,
+ 0x00, 0x2F, 0xE1, 0xD0, 0x63, 0x49, 0xDF, 0xE7, 0x5E, 0x49, 0xDD, 0xE7, 0xBD, 0xE8, 0xF0, 0x81,
+ 0x4A, 0x4A, 0xA5, 0x20, 0x10, 0x70, 0x4B, 0x48, 0x81, 0x78, 0x51, 0x70, 0x01, 0x70, 0x81, 0x88,
+ 0x49, 0x1C, 0x81, 0x80, 0x51, 0x80, 0x70, 0x47, 0x30, 0xB5, 0x85, 0xB0, 0x0C, 0x46, 0x00, 0x21,
+ 0x8D, 0xF8, 0x0C, 0x10, 0x4F, 0xF4, 0x83, 0x25, 0x04, 0xAB, 0x03, 0xAA, 0x69, 0x46, 0x00, 0xF0,
+ 0xD0, 0xF8, 0x9D, 0xF8, 0x10, 0x00, 0x10, 0xB1, 0x11, 0x28, 0x15, 0xD0, 0x19, 0xE0, 0x9D, 0xF8,
+ 0x0C, 0x00, 0xB0, 0xB1, 0x00, 0x98, 0x05, 0xEB, 0x80, 0x01, 0x04, 0xF1, 0x10, 0x00, 0x0A, 0x78,
+ 0x93, 0x07, 0x03, 0xD0, 0x02, 0xF0, 0xFC, 0x02, 0x08, 0x32, 0x00, 0xE0, 0x12, 0x1D, 0xD2, 0xB2,
+ 0x06, 0xF0, 0xB8, 0xF8, 0x05, 0xB0, 0x30, 0xBD, 0x32, 0x20, 0x00, 0x22, 0x20, 0x71, 0x11, 0x46,
+ 0x03, 0xE0, 0x30, 0x20, 0x00, 0x22, 0x20, 0x71, 0x11, 0x46, 0x00, 0xF0, 0x25, 0xF9, 0xF1, 0xE7,
+ 0x70, 0xB5, 0x05, 0x46, 0x4F, 0xF4, 0xC8, 0x71, 0x28, 0x48, 0x06, 0xF0, 0x44, 0xF9, 0x29, 0x4C,
+ 0x26, 0x49, 0xA0, 0x88, 0x40, 0x1C, 0xA0, 0x80, 0x28, 0x46, 0xFF, 0xF7, 0xBD, 0xFF, 0x23, 0x48,
+ 0x25, 0x70, 0xA5, 0x21, 0x01, 0x70, 0x45, 0x70, 0xA1, 0x88, 0x41, 0x80, 0x70, 0xBD, 0x2D, 0xE9,
+ 0xF8, 0x4F, 0x0C, 0x46, 0x88, 0x46, 0x0E, 0x1D, 0x00, 0x23, 0x9B, 0x22, 0x04, 0x21, 0x20, 0x46,
+ 0x00, 0xF0, 0x60, 0xFE, 0x80, 0xBB, 0x21, 0x78, 0x00, 0x23, 0x9B, 0x22, 0x30, 0x46, 0x00, 0xF0,
+ 0x59, 0xFE, 0x61, 0x78, 0x88, 0x42, 0x76, 0xD1, 0x20, 0x78, 0x81, 0x07, 0x0A, 0xD0, 0x02, 0x22,
+ 0x02, 0xEB, 0x90, 0x05, 0x00, 0x21, 0x01, 0xE0, 0x31, 0x54, 0x40, 0x1C, 0xB0, 0xEB, 0x85, 0x0F,
+ 0xFA, 0xD3, 0x02, 0xE0, 0x01, 0x21, 0x01, 0xEB, 0x90, 0x05, 0x00, 0x24, 0xDF, 0xF8, 0x2C, 0xB0,
+ 0x4F, 0xF4, 0x83, 0x26, 0xA1, 0x46, 0x4F, 0xF0, 0x33, 0x0A, 0x00, 0xBF, 0x56, 0xF8, 0x24, 0x00,
+ 0x06, 0xEB, 0x84, 0x07, 0x40, 0x1C, 0x37, 0xD0, 0x00, 0x23, 0x9B, 0x22, 0x04, 0x21, 0x38, 0x46,
+ 0x00, 0xF0, 0x30, 0xFE, 0x08, 0xB1, 0x45, 0xE0, 0x4D, 0xE0, 0x2F, 0xE0, 0x00, 0x00, 0x01, 0x20,
+ 0x4B, 0x43, 0x55, 0x46, 0x58, 0x07, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0xFC, 0x4C, 0x01, 0x20,
+ 0x24, 0x07, 0x10, 0x00, 0xE6, 0x43, 0x01, 0x20, 0x4C, 0x62, 0x01, 0x20, 0x8C, 0x2F, 0x01, 0x20,
+ 0xC8, 0x35, 0x01, 0x20, 0x2E, 0x44, 0x01, 0x20, 0x68, 0x5F, 0x01, 0x20, 0xC0, 0x0A, 0x01, 0x20,
+ 0x36, 0x0B, 0x01, 0x20, 0x6A, 0x4A, 0x01, 0x20, 0x92, 0x5F, 0x01, 0x20, 0xEA, 0x0A, 0x01, 0x20,
+ 0x60, 0x0B, 0x01, 0x20, 0x94, 0x4A, 0x01, 0x20, 0x40, 0x07, 0x10, 0x00, 0x3C, 0x0C, 0x01, 0x20,
+ 0x84, 0x09, 0x01, 0x20, 0xAE, 0x09, 0x01, 0x20, 0x00, 0x94, 0x0C, 0xE0, 0x38, 0x78, 0x81, 0x07,
+ 0x04, 0xEB, 0x90, 0x04, 0x01, 0xD0, 0xA4, 0x1C, 0x00, 0xE0, 0x64, 0x1C, 0x60, 0x2C, 0xB5, 0xD3,
+ 0xB9, 0xF1, 0x00, 0x0F, 0x06, 0xD0, 0x29, 0x46, 0x40, 0x46, 0x00, 0x9A, 0x04, 0xF0, 0xEB, 0xFD,
+ 0x00, 0x28, 0x0C, 0xD0, 0x00, 0x22, 0x8B, 0xF8, 0x04, 0xA0, 0x11, 0x46, 0x33, 0x20, 0xBD, 0xE8,
+ 0xF8, 0x4F, 0x00, 0xF0, 0x81, 0xB8, 0x00, 0x22, 0x11, 0x46, 0x31, 0x20, 0xF7, 0xE7, 0xBD, 0xE8,
+ 0xF8, 0x8F, 0x2D, 0xE9, 0xFF, 0x5F, 0x00, 0x26, 0x0E, 0x60, 0x1D, 0x46, 0x88, 0x46, 0x4F, 0xF4,
+ 0x83, 0x27, 0xB1, 0x46, 0x34, 0x46, 0xB3, 0x46, 0x83, 0xF8, 0x00, 0x60, 0x57, 0xF8, 0x24, 0x00,
+ 0x07, 0xEB, 0x84, 0x0A, 0x40, 0x1C, 0x23, 0xD0, 0x00, 0x23, 0x9B, 0x22, 0x04, 0x21, 0x50, 0x46,
+ 0x00, 0xF0, 0xC0, 0xFD, 0x18, 0xB1, 0x12, 0x20, 0x28, 0x70, 0x01, 0x26, 0x1C, 0xE0, 0x9A, 0xF8,
+ 0x02, 0x20, 0x00, 0x99, 0x9A, 0xF8, 0x00, 0x00, 0x8A, 0x42, 0x08, 0xD1, 0x02, 0x99, 0x02, 0x9A,
+ 0x4F, 0xF0, 0x01, 0x09, 0x09, 0x78, 0x49, 0x1C, 0x11, 0x70, 0xC8, 0xF8, 0x00, 0x40, 0x81, 0x07,
+ 0x04, 0xEB, 0x90, 0x04, 0x01, 0xD0, 0xA4, 0x1C, 0x00, 0xE0, 0x64, 0x1C, 0x60, 0x2C, 0xD5, 0xD3,
+ 0x16, 0xB9, 0xB9, 0xF1, 0x01, 0x0F, 0x0B, 0xD0, 0x56, 0xEA, 0x09, 0x06, 0x06, 0xD1, 0x11, 0x20,
+ 0x28, 0x70, 0x02, 0x99, 0x81, 0xF8, 0x00, 0xB0, 0xC8, 0xF8, 0x00, 0xB0, 0xBD, 0xE8, 0xFF, 0x9F,
+ 0xD8, 0xF8, 0x00, 0x00, 0x00, 0x23, 0x07, 0xEB, 0x80, 0x04, 0x20, 0x1D, 0x21, 0x78, 0x9B, 0x22,
+ 0x00, 0xF0, 0x88, 0xFD, 0x61, 0x78, 0x88, 0x42, 0x02, 0xD1, 0x85, 0xF8, 0x00, 0xB0, 0xED, 0xE7,
+ 0x12, 0x20, 0x28, 0x70, 0xEA, 0xE7, 0x00, 0x00, 0x19, 0x48, 0x00, 0x68, 0x90, 0xF8, 0xA0, 0x00,
+ 0xC0, 0x07, 0x01, 0xD0, 0x00, 0xF0, 0x2E, 0xB8, 0x00, 0xF0, 0xDE, 0xB8, 0x38, 0xB5, 0x04, 0x46,
+ 0x69, 0x46, 0x18, 0x30, 0x00, 0xF0, 0xF9, 0xFD, 0x11, 0x48, 0x00, 0x99, 0x00, 0x68, 0x90, 0xF8,
+ 0xA0, 0x00, 0xC0, 0x07, 0x20, 0x46, 0x02, 0xD0, 0x00, 0xF0, 0x39, 0xF8, 0x38, 0xBD, 0x00, 0xF0,
+ 0xD5, 0xF8, 0x38, 0xBD, 0x00, 0xF0, 0x25, 0xB9, 0x09, 0x4B, 0x1B, 0x68, 0x93, 0xF8, 0xA0, 0x30,
+ 0xDB, 0x07, 0x01, 0xD1, 0x00, 0xF0, 0x3B, 0xB9, 0x70, 0x47, 0x05, 0x49, 0x09, 0x68, 0x91, 0xF8,
+ 0xA0, 0x10, 0xC9, 0x07, 0x01, 0xD0, 0x00, 0xF0, 0x99, 0xB8, 0x00, 0xF0, 0x42, 0xB9, 0x00, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0x1C, 0xB5, 0x56, 0x49, 0xD1, 0xE9, 0x00, 0x01, 0xCD, 0xE9, 0x00, 0x01,
+ 0x68, 0x46, 0xFF, 0xF7, 0x77, 0xFA, 0x1C, 0xBD, 0x88, 0x42, 0x0E, 0xD9, 0x90, 0x42, 0x01, 0xD3,
+ 0xFF, 0x20, 0x70, 0x47, 0x91, 0x42, 0x08, 0xD0, 0x40, 0x1A, 0xC0, 0xEB, 0x00, 0x20, 0x51, 0x1A,
+ 0xB0, 0xFB, 0xF1, 0xF0, 0x10, 0xF0, 0xFF, 0x00, 0xF3, 0xD1, 0x01, 0x20, 0x70, 0x47, 0x7C, 0xB5,
+ 0x05, 0x46, 0x00, 0x20, 0x00, 0x90, 0x01, 0x90, 0x95, 0xF8, 0x4B, 0x21, 0x9D, 0xF8, 0x00, 0x00,
+ 0x62, 0xF3, 0x85, 0x00, 0x8D, 0xF8, 0x00, 0x00, 0x95, 0xF8, 0x3E, 0x21, 0x03, 0x2A, 0x3E, 0xD0,
+ 0x08, 0x2A, 0x40, 0xD0, 0x20, 0xF0, 0xC0, 0x00, 0x10, 0xF1, 0x80, 0x00, 0x8D, 0xF8, 0x00, 0x00,
+ 0x0A, 0x14, 0x10, 0x09, 0x8D, 0xF8, 0x01, 0x00, 0x9D, 0xF8, 0x03, 0x00, 0x39, 0x4E, 0x62, 0xF3,
+ 0x07, 0x10, 0x0A, 0x09, 0x8D, 0xF8, 0x02, 0x20, 0x61, 0xF3, 0x03, 0x00, 0x8D, 0xF8, 0x03, 0x00,
+ 0x30, 0x68, 0xB0, 0xF8, 0xA3, 0x20, 0xB0, 0xF8, 0xA1, 0x10, 0x35, 0xF8, 0x4A, 0x0F, 0xFF, 0xF7,
+ 0xBB, 0xFF, 0x8D, 0xF8, 0x04, 0x00, 0x30, 0x68, 0xB0, 0xF8, 0xA7, 0x20, 0xB0, 0xF8, 0xA5, 0x10,
+ 0x68, 0x88, 0xFF, 0xF7, 0xB1, 0xFF, 0x8D, 0xF8, 0x05, 0x00, 0x29, 0x7A, 0x9D, 0xF8, 0x06, 0x00,
+ 0x61, 0xF3, 0x05, 0x00, 0x8D, 0xF8, 0x06, 0x00, 0x95, 0xF8, 0xF6, 0x10, 0x09, 0x29, 0x1E, 0xD2,
+ 0xDF, 0xE8, 0x01, 0xF0, 0x0C, 0x0E, 0x10, 0x12, 0x14, 0x16, 0x18, 0x1A, 0x1C, 0x00, 0x20, 0xF0,
+ 0xC0, 0x00, 0x40, 0x30, 0xC2, 0xE7, 0x40, 0xF0, 0xC0, 0x00, 0xBF, 0xE7, 0x00, 0x24, 0x0E, 0xE0,
+ 0x01, 0x24, 0x0C, 0xE0, 0x02, 0x24, 0x0A, 0xE0, 0x03, 0x24, 0x08, 0xE0, 0x04, 0x24, 0x06, 0xE0,
+ 0x05, 0x24, 0x04, 0xE0, 0x06, 0x24, 0x02, 0xE0, 0x07, 0x24, 0x00, 0xE0, 0x08, 0x24, 0xA1, 0x08,
+ 0x61, 0xF3, 0x87, 0x10, 0x8D, 0xF8, 0x06, 0x00, 0x9D, 0xF8, 0x07, 0x00, 0x64, 0xF3, 0x87, 0x10,
+ 0x8D, 0xF8, 0x07, 0x00, 0x68, 0x46, 0xFF, 0xF7, 0xED, 0xF9, 0x7C, 0xBD, 0x1C, 0xB5, 0x0C, 0x4A,
+ 0xD2, 0xE9, 0x06, 0x12, 0xCD, 0xE9, 0x00, 0x12, 0x9D, 0xF8, 0x00, 0x10, 0x21, 0xF0, 0x3C, 0x01,
+ 0x8D, 0xF8, 0x00, 0x10, 0x80, 0x78, 0x01, 0x28, 0x03, 0xD0, 0x05, 0x28, 0x07, 0xD1, 0x01, 0x20,
+ 0x00, 0xE0, 0x00, 0x20, 0x8D, 0xF8, 0x01, 0x00, 0x68, 0x46, 0xFF, 0xF7, 0xD3, 0xF9, 0x1C, 0xBD,
+ 0x20, 0x6E, 0x01, 0x00, 0x20, 0x07, 0x10, 0x00, 0x1C, 0xB5, 0x51, 0x49, 0xD1, 0xE9, 0x00, 0x01,
+ 0xCD, 0xE9, 0x00, 0x01, 0x68, 0x46, 0xFF, 0xF7, 0xC5, 0xF9, 0x1C, 0xBD, 0x1C, 0xB5, 0x4C, 0x4B,
+ 0xD3, 0xE9, 0x02, 0x23, 0xCD, 0xE9, 0x00, 0x23, 0x90, 0xF8, 0x3E, 0x21, 0x03, 0x2A, 0x19, 0xD0,
+ 0x08, 0x2A, 0x9D, 0xF8, 0x00, 0x20, 0x22, 0xF0, 0xF0, 0x02, 0x19, 0xD0, 0x12, 0xF1, 0x20, 0x02,
+ 0x8D, 0xF8, 0x00, 0x20, 0x90, 0xF8, 0x3F, 0x21, 0x02, 0xF0, 0x0F, 0x02, 0x01, 0x2A, 0x11, 0xD0,
+ 0x02, 0x2A, 0x15, 0xD0, 0x04, 0x2A, 0x9D, 0xF8, 0x01, 0x20, 0x22, 0xF0, 0x0F, 0x02, 0x2E, 0xD0,
+ 0x52, 0x1C, 0x12, 0xE0, 0x9D, 0xF8, 0x00, 0x20, 0x22, 0xF0, 0xF0, 0x02, 0x10, 0x32, 0xE7, 0xE7,
+ 0x30, 0x32, 0xE5, 0xE7, 0x9D, 0xF8, 0x01, 0x20, 0x22, 0xF0, 0x0F, 0x02, 0x92, 0x1C, 0x04, 0xE0,
+ 0x9D, 0xF8, 0x01, 0x20, 0x22, 0xF0, 0x0F, 0x02, 0xD2, 0x1C, 0x8D, 0xF8, 0x01, 0x20, 0x90, 0xF8,
+ 0x4B, 0x01, 0xD2, 0xB2, 0x60, 0xF3, 0x07, 0x12, 0x8D, 0xF8, 0x01, 0x20, 0xBD, 0xF8, 0x02, 0x00,
+ 0x0A, 0x0C, 0x62, 0xF3, 0x0B, 0x00, 0xAD, 0xF8, 0x02, 0x00, 0xBD, 0xF8, 0x03, 0x00, 0x61, 0xF3,
+ 0x0F, 0x10, 0xAD, 0xF8, 0x03, 0x00, 0x68, 0x46, 0xFF, 0xF7, 0x74, 0xF9, 0x1C, 0xBD, 0x12, 0x1D,
+ 0xE3, 0xE7, 0x1C, 0xB5, 0x23, 0x4A, 0x12, 0x68, 0x92, 0xF8, 0xA0, 0x20, 0x93, 0x07, 0x15, 0xD5,
+ 0x08, 0x28, 0x13, 0xD2, 0xDF, 0xE8, 0x00, 0xF0, 0x12, 0x04, 0x06, 0x06, 0x06, 0x06, 0x06, 0x06,
+ 0x12, 0x06, 0x0B, 0xD5, 0x1A, 0x4B, 0xD3, 0xE9, 0x06, 0x23, 0xCD, 0xE9, 0x00, 0x23, 0x8D, 0xF8,
+ 0x01, 0x00, 0xCD, 0xF8, 0x02, 0x10, 0x68, 0x46, 0xFF, 0xF7, 0x54, 0xF9, 0x1C, 0xBD, 0x1C, 0xB5,
+ 0x13, 0x4C, 0xD4, 0xE9, 0x08, 0x34, 0xCD, 0xE9, 0x00, 0x34, 0x8D, 0xF8, 0x01, 0x00, 0xCD, 0xF8,
+ 0x02, 0x10, 0x68, 0x46, 0x12, 0xB1, 0xFF, 0xF7, 0x32, 0xFB, 0x1C, 0xBD, 0xFF, 0xF7, 0x42, 0xF9,
+ 0x1C, 0xBD, 0x1C, 0xB5, 0x41, 0x78, 0x00, 0x29, 0x10, 0xD0, 0x05, 0x29, 0x0E, 0xD2, 0x08, 0x4A,
+ 0xD2, 0xE9, 0x0C, 0x12, 0xCD, 0xE9, 0x00, 0x12, 0xD0, 0xF8, 0x02, 0x10, 0xCD, 0xF8, 0x02, 0x10,
+ 0xC0, 0x88, 0xAD, 0xF8, 0x06, 0x00, 0x68, 0x46, 0xFF, 0xF7, 0x2C, 0xF9, 0x1C, 0xBD, 0x00, 0x00,
+ 0x40, 0x6E, 0x01, 0x00, 0x20, 0x07, 0x10, 0x00, 0x2D, 0xE9, 0xFF, 0x41, 0x03, 0xF0, 0xCE, 0xFC,
+ 0x23, 0x4D, 0x00, 0x24, 0xC4, 0xEB, 0xC4, 0x01, 0x05, 0xEB, 0x81, 0x00, 0x00, 0xF1, 0x0C, 0x03,
+ 0x02, 0x7E, 0xC8, 0xCB, 0xCD, 0xE9, 0x02, 0x72, 0xCD, 0xE9, 0x00, 0x36, 0xD0, 0xE9, 0x01, 0x23,
+ 0x55, 0xF8, 0x21, 0x00, 0x00, 0x21, 0x03, 0xF0, 0x63, 0xFE, 0x18, 0xB1, 0x21, 0x46, 0x43, 0x20,
+ 0xFF, 0xF7, 0x45, 0xFB, 0x64, 0x1C, 0x05, 0x2C, 0xE4, 0xDB, 0x15, 0x4D, 0x00, 0x24, 0x8C, 0x35,
+ 0x05, 0xEB, 0xC4, 0x00, 0x00, 0x21, 0x42, 0x68, 0x55, 0xF8, 0x34, 0x00, 0x03, 0xF0, 0xD7, 0xFD,
+ 0x18, 0xB1, 0x21, 0x46, 0x44, 0x20, 0xFF, 0xF7, 0x32, 0xFB, 0x64, 0x1C, 0x06, 0x2C, 0xEF, 0xDB,
+ 0x0B, 0x4D, 0x00, 0x24, 0x15, 0xF1, 0xBC, 0x05, 0x00, 0x21, 0x55, 0xF8, 0x24, 0x00, 0x03, 0xF0,
+ 0x2C, 0xFD, 0x18, 0xB1, 0x21, 0x46, 0x45, 0x20, 0xFF, 0xF7, 0x21, 0xFB, 0x64, 0x1C, 0x04, 0x2C,
+ 0xF2, 0xDB, 0xBD, 0xE8, 0xFF, 0x81, 0x01, 0x46, 0x46, 0x20, 0xFF, 0xF7, 0x18, 0xBB, 0x00, 0x00,
+ 0x78, 0x6E, 0x01, 0x00, 0x2D, 0xE9, 0xF0, 0x41, 0x64, 0x27, 0x07, 0x26, 0x00, 0x25, 0xFB, 0x4C,
+ 0xFB, 0x48, 0x04, 0xF0, 0x4A, 0xFD, 0xFA, 0x48, 0x03, 0x68, 0xD9, 0x1C, 0x40, 0xD0, 0x42, 0x68,
+ 0xD1, 0x1C, 0x3D, 0xD0, 0x81, 0x68, 0x11, 0xF1, 0x03, 0x0F, 0x39, 0xD0, 0x00, 0x69, 0x10, 0xF1,
+ 0x03, 0x0F, 0x35, 0xD0, 0xB3, 0xF1, 0xFF, 0x3F, 0x35, 0xD0, 0x13, 0xF1, 0x02, 0x0F, 0x32, 0xD0,
+ 0xB2, 0xF1, 0xFF, 0x3F, 0x2F, 0xD0, 0x12, 0xF1, 0x02, 0x0F, 0x2C, 0xD0, 0xB1, 0xF1, 0xFF, 0x3F,
+ 0x29, 0xD0, 0x11, 0xF1, 0x02, 0x0F, 0x26, 0xD0, 0xB0, 0xF1, 0xFF, 0x3F, 0x23, 0xD0, 0x10, 0xF1,
+ 0x02, 0x0F, 0x20, 0xD0, 0xDF, 0xB2, 0xD6, 0xB2, 0xCD, 0xB2, 0x04, 0x46, 0x2A, 0x46, 0x31, 0x46,
+ 0x38, 0x46, 0x04, 0xF0, 0x6D, 0xFA, 0x20, 0x46, 0x04, 0xF0, 0x77, 0xFA, 0xE1, 0x48, 0x04, 0xF0,
+ 0xEA, 0xFC, 0x03, 0x28, 0x14, 0xD0, 0x01, 0x28, 0x15, 0xD0, 0x02, 0x28, 0x13, 0xD0, 0xDD, 0x48,
+ 0x01, 0x68, 0x01, 0x29, 0x15, 0xD1, 0xBD, 0xE8, 0xF0, 0x41, 0x00, 0x1D, 0x03, 0xF0, 0x78, 0xBE,
+ 0x00, 0x21, 0x24, 0x20, 0x01, 0xE0, 0x00, 0x21, 0x25, 0x20, 0xFF, 0xF7, 0x0D, 0xFB, 0xDD, 0xE7,
+ 0x00, 0x21, 0x29, 0x20, 0x01, 0xE0, 0x00, 0x21, 0x2A, 0x20, 0xBD, 0xE8, 0xF0, 0x41, 0xFF, 0xF7,
+ 0x03, 0xBB, 0xBD, 0xE8, 0xF0, 0x81, 0x2D, 0xE9, 0xFC, 0x41, 0xCF, 0x4F, 0x01, 0x25, 0x01, 0xAA,
+ 0x69, 0x46, 0x38, 0x46, 0x04, 0xF0, 0xB0, 0xFA, 0xCC, 0x4C, 0x4F, 0xF4, 0x8C, 0x56, 0x01, 0x28,
+ 0x03, 0xD1, 0x00, 0x98, 0xB0, 0xF5, 0x8C, 0x6F, 0x04, 0xD0, 0x00, 0x21, 0x22, 0x20, 0xFF, 0xF7,
+ 0xEB, 0xFA, 0x30, 0xE0, 0x32, 0x46, 0x39, 0x46, 0x20, 0x68, 0x05, 0xF0, 0x4B, 0xFD, 0x20, 0x68,
+ 0x04, 0xF0, 0xAD, 0xFA, 0x01, 0x28, 0x04, 0xD0, 0x00, 0x21, 0x23, 0x20, 0xFF, 0xF7, 0xDC, 0xFA,
+ 0x00, 0x25, 0x21, 0x68, 0x26, 0x22, 0x0F, 0x46, 0xBD, 0x48, 0x01, 0xF5, 0x86, 0x51, 0x05, 0xF0,
+ 0x39, 0xFD, 0x41, 0xF2, 0xE6, 0x00, 0x39, 0x18, 0xB9, 0x48, 0x36, 0x22, 0x28, 0x30, 0x05, 0xF0,
+ 0x31, 0xFD, 0x41, 0xF2, 0x1C, 0x10, 0x39, 0x18, 0xB5, 0x48, 0x26, 0x22, 0x48, 0x30, 0x05, 0xF0,
+ 0x29, 0xFD, 0x41, 0xF2, 0x42, 0x10, 0x39, 0x18, 0xB1, 0x48, 0x36, 0x22, 0x70, 0x30, 0x05, 0xF0,
+ 0x21, 0xFD, 0x00, 0x2D, 0x03, 0xD1, 0x31, 0x46, 0x20, 0x68, 0x05, 0xF0, 0x9A, 0xFD, 0xBD, 0xE8,
+ 0xFC, 0x81, 0xF8, 0xB5, 0x0D, 0x46, 0x06, 0x46, 0x00, 0x24, 0x6A, 0x46, 0x04, 0xF0, 0x64, 0xFA,
+ 0x01, 0x28, 0x01, 0xD0, 0x01, 0x24, 0x0B, 0xE0, 0x28, 0x68, 0x31, 0x46, 0x82, 0x00, 0xA5, 0x48,
+ 0x05, 0xF0, 0x4D, 0xFD, 0xA3, 0x48, 0x04, 0xF0, 0x6A, 0xFA, 0x01, 0x28, 0x00, 0xD0, 0x02, 0x24,
+ 0x20, 0x46, 0xF8, 0xBD, 0x2D, 0xE9, 0xF8, 0x43, 0x9E, 0x4D, 0x9F, 0x4F, 0x10, 0x35, 0x40, 0xF6,
+ 0xFC, 0x61, 0x9E, 0x48, 0x05, 0xF0, 0x97, 0xFD, 0x69, 0x46, 0x38, 0x46, 0xFF, 0xF7, 0xD9, 0xFF,
+ 0x01, 0x28, 0x05, 0xD0, 0x02, 0x28, 0x06, 0xD0, 0xE8, 0x78, 0x01, 0x28, 0x0A, 0xD1, 0x05, 0xE0,
+ 0x00, 0x21, 0xA0, 0x20, 0x38, 0xE0, 0x00, 0x21, 0xA1, 0x20, 0x35, 0xE0, 0xA8, 0x78, 0x08, 0xB9,
+ 0x28, 0x88, 0x10, 0xB1, 0x00, 0x21, 0xA2, 0x20, 0x2E, 0xE0, 0xDF, 0xF8, 0x44, 0x82, 0xE9, 0x88,
+ 0xD8, 0xF8, 0x00, 0x00, 0x82, 0x8A, 0x91, 0x42, 0x02, 0xD0, 0x00, 0x21, 0xA3, 0x20, 0x05, 0xE0,
+ 0xA9, 0x88, 0xC0, 0x8A, 0x81, 0x42, 0x03, 0xD0, 0x00, 0x21, 0xA4, 0x20, 0xFF, 0xF7, 0x64, 0xFA,
+ 0x86, 0x48, 0xE9, 0x88, 0x00, 0x24, 0xA0, 0xF8, 0xF8, 0x1E, 0xA9, 0x88, 0xA0, 0xF8, 0xFA, 0x1E,
+ 0x00, 0x98, 0x69, 0x46, 0x07, 0xEB, 0x80, 0x07, 0x38, 0x46, 0xFF, 0xF7, 0xA2, 0xFF, 0x78, 0xB1,
+ 0x01, 0x28, 0x03, 0xD0, 0x00, 0x21, 0xA6, 0x20, 0xFF, 0xF7, 0x4E, 0xFA, 0xE0, 0x07, 0x24, 0xD0,
+ 0xA0, 0x07, 0x24, 0xD4, 0x11, 0x21, 0xA7, 0x20, 0xFF, 0xF7, 0x46, 0xFA, 0xBD, 0xE8, 0xF8, 0x83,
+ 0x6E, 0x78, 0x29, 0x46, 0x30, 0x46, 0x00, 0xF0, 0x1F, 0xFE, 0x50, 0xB1, 0x10, 0x2E, 0x0B, 0xD0,
+ 0x11, 0x2E, 0x0C, 0xD0, 0x12, 0x2E, 0x0D, 0xD0, 0x13, 0x2E, 0xD9, 0xD1, 0x44, 0xF0, 0x10, 0x04,
+ 0xD6, 0xE7, 0x31, 0x46, 0xA8, 0x20, 0xDF, 0xE7, 0x44, 0xF0, 0x01, 0x04, 0xD0, 0xE7, 0x44, 0xF0,
+ 0x02, 0x04, 0xCD, 0xE7, 0x44, 0xF0, 0x08, 0x04, 0xCA, 0xE7, 0x10, 0x21, 0xDB, 0xE7, 0x20, 0x07,
+ 0x0B, 0xD4, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x50, 0x12, 0xC9, 0x07, 0x03, 0xD1, 0x90, 0xF8,
+ 0x80, 0x02, 0xC0, 0x07, 0x01, 0xD0, 0x12, 0x21, 0xCD, 0xE7, 0xE0, 0x06, 0xCE, 0xD4, 0x13, 0x21,
+ 0xC9, 0xE7, 0x38, 0xB5, 0x02, 0xF0, 0xEC, 0xFE, 0x03, 0xF0, 0x47, 0xFD, 0x00, 0x23, 0x01, 0x22,
+ 0x19, 0x46, 0x05, 0x20, 0x05, 0xF0, 0x12, 0xFB, 0x00, 0x23, 0x1A, 0x46, 0x01, 0x21, 0x06, 0x20,
+ 0x05, 0xF0, 0x0C, 0xFB, 0x03, 0x21, 0x01, 0x20, 0x00, 0xF0, 0x20, 0xF9, 0x04, 0x21, 0x07, 0x20,
+ 0x00, 0xF0, 0x1C, 0xF9, 0x04, 0x21, 0x08, 0x20, 0x00, 0xF0, 0x18, 0xF9, 0x50, 0x4C, 0x20, 0x68,
+ 0x90, 0xF8, 0xB0, 0x10, 0xC9, 0x07, 0x3B, 0xD0, 0x00, 0x25, 0x00, 0x95, 0x90, 0xF8, 0xC2, 0x00,
+ 0x2B, 0x46, 0xC0, 0xF3, 0x02, 0x10, 0x01, 0x22, 0x29, 0x46, 0x04, 0xF0, 0xA6, 0xFC, 0x20, 0x68,
+ 0x0A, 0x21, 0x90, 0xF8, 0xC2, 0x00, 0xC0, 0xF3, 0x02, 0x10, 0x04, 0xF0, 0xC6, 0xFC, 0x00, 0x95,
+ 0x20, 0x68, 0x00, 0x23, 0x01, 0x22, 0x90, 0xF8, 0xC2, 0x00, 0x19, 0x46, 0x00, 0xF0, 0x07, 0x00,
+ 0x04, 0xF0, 0x93, 0xFC, 0x20, 0x68, 0x01, 0x21, 0x90, 0xF8, 0xC2, 0x00, 0x00, 0xF0, 0x07, 0x00,
+ 0x04, 0xF0, 0xB3, 0xFC, 0x0D, 0x20, 0x00, 0xF0, 0xC3, 0xF8, 0x0D, 0x20, 0x00, 0xF0, 0xCB, 0xF8,
+ 0x07, 0x21, 0x0D, 0x20, 0x00, 0xF0, 0xD2, 0xF8, 0x0C, 0x20, 0x00, 0xF0, 0xB9, 0xF8, 0x0C, 0x20,
+ 0x00, 0xF0, 0xC1, 0xF8, 0xBD, 0xE8, 0x38, 0x40, 0x07, 0x21, 0x0C, 0x20, 0x00, 0xF0, 0xC6, 0xB8,
+ 0x38, 0xBD, 0x30, 0xB5, 0x8D, 0xB0, 0xEF, 0xF7, 0x09, 0xFA, 0x00, 0x24, 0x60, 0xB2, 0x00, 0xF0,
+ 0xA7, 0xF8, 0x64, 0x1C, 0x1F, 0x2C, 0xF9, 0xDB, 0xEF, 0xF7, 0xFE, 0xF9, 0x04, 0xF0, 0x96, 0xF8,
+ 0xFF, 0xF7, 0xCA, 0xF9, 0x04, 0xF0, 0x44, 0xFC, 0x07, 0x21, 0x0A, 0x20, 0x00, 0xF0, 0xBE, 0xF8,
+ 0x07, 0x21, 0x0B, 0x20, 0x00, 0xF0, 0xBA, 0xF8, 0xFE, 0xF7, 0xE0, 0xFE, 0xFE, 0xF7, 0x84, 0xFD,
+ 0x04, 0xA8, 0xFF, 0xF7, 0xE0, 0xF8, 0x0A, 0xA9, 0x07, 0xA8, 0xFE, 0xF7, 0x7D, 0xFF, 0x04, 0xA8,
+ 0x00, 0x90, 0x07, 0xA8, 0x02, 0x90, 0x0A, 0xA8, 0x03, 0x90, 0x68, 0x46, 0x04, 0xF0, 0x28, 0xFE,
+ 0x02, 0x21, 0x00, 0x20, 0x00, 0xF0, 0xA2, 0xF8, 0x05, 0xF0, 0x1C, 0xFD, 0x00, 0x78, 0xC0, 0x07,
+ 0x01, 0xD1, 0xFE, 0xF7, 0xE2, 0xFE, 0xFF, 0xF7, 0x25, 0xF9, 0xFF, 0xF7, 0x13, 0xFE, 0x04, 0xF0,
+ 0xAF, 0xF8, 0x10, 0x4D, 0x01, 0xAA, 0x69, 0x46, 0x28, 0x46, 0x04, 0xF0, 0x25, 0xF9, 0x0C, 0x4C,
+ 0x01, 0x28, 0x19, 0xD0, 0x00, 0x21, 0x20, 0x20, 0x27, 0xE0, 0x00, 0x00, 0x00, 0x90, 0xD0, 0x03,
+ 0x34, 0x7F, 0x10, 0x00, 0x08, 0x84, 0x10, 0x00, 0xF0, 0x9F, 0x01, 0x00, 0x24, 0x07, 0x10, 0x00,
+ 0xF8, 0x5B, 0x01, 0x20, 0x84, 0x5A, 0x10, 0x00, 0xF0, 0xBF, 0x01, 0x00, 0xFC, 0x4C, 0x01, 0x20,
+ 0x20, 0x07, 0x10, 0x00, 0xF0, 0xEF, 0x01, 0x00, 0x00, 0x98, 0xB0, 0xF5, 0x26, 0x7F, 0xE1, 0xD1,
+ 0x4F, 0xF4, 0x26, 0x62, 0x29, 0x46, 0x20, 0x68, 0x05, 0xF0, 0xAC, 0xFB, 0x20, 0x68, 0x04, 0xF0,
+ 0x0E, 0xF9, 0x01, 0x28, 0x03, 0xD0, 0x00, 0x21, 0x21, 0x20, 0xFF, 0xF7, 0xF0, 0xF8, 0xFF, 0xF7,
+ 0x3A, 0xFE, 0x04, 0xF0, 0xA7, 0xFB, 0x05, 0x00, 0x2D, 0xD0, 0x01, 0x21, 0x01, 0x2D, 0x02, 0xD0,
+ 0x2E, 0x20, 0xFF, 0xF7, 0xE4, 0xF8, 0x20, 0x68, 0x90, 0xF8, 0x3D, 0x10, 0xA9, 0x42, 0x02, 0xD0,
+ 0x2F, 0x20, 0xFF, 0xF7, 0xDC, 0xF8, 0xFF, 0xF7, 0x8D, 0xFE, 0x00, 0xF0, 0x5B, 0xF8, 0xFF, 0xF7,
+ 0x10, 0xFF, 0x20, 0x68, 0x90, 0xF8, 0x60, 0x03, 0xC1, 0x07, 0x0F, 0xD0, 0x81, 0x07, 0x0D, 0xD5,
+ 0x00, 0x21, 0x01, 0x23, 0x40, 0x09, 0x1A, 0x46, 0x00, 0x91, 0x04, 0xF0, 0xD6, 0xFB, 0x20, 0x68,
+ 0x00, 0x21, 0x90, 0xF8, 0x60, 0x03, 0x40, 0x09, 0x04, 0xF0, 0xF7, 0xFB, 0x0D, 0xB0, 0xBD, 0xE8,
+ 0x30, 0x40, 0xFF, 0xF7, 0x9F, 0xB8, 0x00, 0x21, 0x2D, 0x20, 0xFF, 0xF7, 0x05, 0xF9, 0xDA, 0xE7,
+ 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21, 0x91, 0x40, 0x40, 0x09, 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20,
+ 0xC0, 0xF8, 0x80, 0x11, 0x70, 0x47, 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21, 0x91, 0x40, 0x40, 0x09,
+ 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20, 0xC0, 0xF8, 0x80, 0x12, 0x70, 0x47, 0x49, 0x07, 0x09, 0x0E,
+ 0x00, 0x28, 0x06, 0xDA, 0x00, 0xF0, 0x0F, 0x00, 0x00, 0xF1, 0xE0, 0x20, 0x80, 0xF8, 0x14, 0x1D,
+ 0x70, 0x47, 0x00, 0xF1, 0xE0, 0x20, 0x80, 0xF8, 0x00, 0x14, 0x70, 0x47, 0x70, 0xB5, 0x0D, 0x46,
+ 0x04, 0x46, 0xFF, 0xF7, 0xE0, 0xFF, 0x29, 0x46, 0x20, 0x46, 0xFF, 0xF7, 0xE7, 0xFF, 0x04, 0xF0,
+ 0x1F, 0x01, 0x01, 0x20, 0x88, 0x40, 0x61, 0x09, 0x89, 0x00, 0x01, 0xF1, 0xE0, 0x21, 0xC1, 0xF8,
+ 0x00, 0x01, 0x70, 0xBD, 0x10, 0xB5, 0x04, 0xF0, 0x2B, 0xFE, 0x00, 0x23, 0x01, 0x22, 0x19, 0x46,
+ 0x18, 0x46, 0x05, 0xF0, 0xCB, 0xF9, 0x00, 0x23, 0x01, 0x22, 0x19, 0x46, 0x10, 0x46, 0x05, 0xF0,
+ 0xC5, 0xF9, 0x00, 0x23, 0x1A, 0x46, 0x01, 0x21, 0x02, 0x20, 0x05, 0xF0, 0xBF, 0xF9, 0x00, 0x23,
+ 0x1A, 0x46, 0x01, 0x21, 0x03, 0x20, 0x05, 0xF0, 0xB9, 0xF9, 0x00, 0x23, 0x1A, 0x46, 0x01, 0x21,
+ 0x04, 0x20, 0x05, 0xF0, 0xB3, 0xF9, 0x17, 0x48, 0x00, 0x68, 0x01, 0x7C, 0xA5, 0x29, 0x08, 0xD0,
+ 0x4F, 0xF4, 0xFA, 0x40, 0x05, 0xF0, 0x94, 0xF9, 0x00, 0x21, 0x28, 0x20, 0xFF, 0xF7, 0x9C, 0xF8,
+ 0x02, 0xE0, 0x40, 0x69, 0x05, 0xF0, 0x8C, 0xF9, 0x04, 0x21, 0x02, 0x20, 0xFF, 0xF7, 0xB6, 0xFF,
+ 0x04, 0x21, 0x03, 0x20, 0xFF, 0xF7, 0xB2, 0xFF, 0x04, 0x21, 0x08, 0x46, 0xFF, 0xF7, 0xAE, 0xFF,
+ 0x04, 0x21, 0x05, 0x20, 0xFF, 0xF7, 0xAA, 0xFF, 0x04, 0x21, 0x06, 0x20, 0xFF, 0xF7, 0xA6, 0xFF,
+ 0x01, 0x21, 0x09, 0x20, 0xFF, 0xF7, 0xA2, 0xFF, 0x04, 0x21, 0xBD, 0xE8, 0x10, 0x40, 0x0F, 0x20,
+ 0x9C, 0xE7, 0x00, 0x00, 0x24, 0x07, 0x10, 0x00, 0x88, 0x42, 0x00, 0xD3, 0x08, 0x46, 0x90, 0x42,
+ 0x00, 0xD3, 0x10, 0x46, 0x70, 0x47, 0x88, 0x42, 0x00, 0xD8, 0x08, 0x46, 0x90, 0x42, 0x00, 0xD8,
+ 0x10, 0x46, 0x70, 0x47, 0x88, 0x42, 0x00, 0xD3, 0x08, 0x46, 0x70, 0x47, 0x88, 0x42, 0x00, 0xD8,
+ 0x08, 0x46, 0x70, 0x47, 0x02, 0x46, 0x00, 0x20, 0x4F, 0xF0, 0x80, 0x41, 0x00, 0xE0, 0x89, 0x08,
+ 0x91, 0x42, 0xFC, 0xD8, 0x07, 0xE0, 0x43, 0x18, 0x93, 0x42, 0x02, 0xD8, 0xD2, 0x1A, 0x00, 0xEB,
+ 0x41, 0x00, 0x40, 0x08, 0x89, 0x08, 0x00, 0x29, 0xF5, 0xD1, 0x82, 0x42, 0x00, 0xD9, 0x40, 0x1C,
+ 0x70, 0x47, 0x10, 0xB5, 0x5A, 0x43, 0x02, 0x9C, 0x06, 0xE0, 0x00, 0xBF, 0x50, 0xF8, 0x22, 0x30,
+ 0x93, 0xFB, 0xF4, 0xF3, 0x41, 0xF8, 0x22, 0x30, 0x52, 0x1E, 0xF7, 0xD2, 0x10, 0xBD, 0x10, 0xB5,
+ 0x5A, 0x43, 0x02, 0x9C, 0x05, 0xE0, 0x00, 0xBF, 0x50, 0xF8, 0x22, 0x30, 0x63, 0x43, 0x41, 0xF8,
+ 0x22, 0x30, 0x52, 0x1E, 0xF8, 0xD2, 0x10, 0xBD, 0x30, 0xB5, 0x03, 0x9C, 0x63, 0x43, 0x06, 0xE0,
+ 0x50, 0xF8, 0x23, 0x40, 0x51, 0xF8, 0x23, 0x50, 0x2C, 0x44, 0x42, 0xF8, 0x23, 0x40, 0x5B, 0x1E,
+ 0xF6, 0xD2, 0x30, 0xBD, 0x30, 0xB5, 0x03, 0x9C, 0x63, 0x43, 0x06, 0xE0, 0x50, 0xF8, 0x23, 0x40,
+ 0x51, 0xF8, 0x23, 0x50, 0x64, 0x1B, 0x42, 0xF8, 0x23, 0x40, 0x5B, 0x1E, 0xF6, 0xD2, 0x30, 0xBD,
+ 0x2D, 0xE9, 0xF0, 0x4F, 0x83, 0x46, 0xDD, 0xE9, 0x09, 0x06, 0xDD, 0xF8, 0x34, 0xE0, 0x8A, 0x46,
+ 0x82, 0x42, 0x26, 0xD1, 0x00, 0x20, 0x22, 0xE0, 0x00, 0xFB, 0x02, 0xF1, 0x0B, 0xEB, 0x81, 0x09,
+ 0x0B, 0x9D, 0x00, 0xFB, 0x0E, 0xF1, 0x00, 0x24, 0x05, 0xEB, 0x81, 0x05, 0x14, 0xE0, 0x00, 0x21,
+ 0x0F, 0xE0, 0x00, 0xBF, 0x01, 0xFB, 0x06, 0xF8, 0x03, 0xEB, 0x88, 0x08, 0x59, 0xF8, 0x21, 0xC0,
+ 0x55, 0xF8, 0x24, 0x70, 0x58, 0xF8, 0x24, 0x80, 0x49, 0x1C, 0x0C, 0xFB, 0x08, 0x77, 0x45, 0xF8,
+ 0x24, 0x70, 0x91, 0x42, 0xEE, 0xDB, 0x64, 0x1C, 0xB4, 0x42, 0xE8, 0xDB, 0x40, 0x1C, 0x50, 0x45,
+ 0xDA, 0xDB, 0xBD, 0xE8, 0xF0, 0x8F, 0x10, 0xB5, 0x00, 0x23, 0x1C, 0x46, 0x42, 0xB1, 0x02, 0x78,
+ 0x52, 0x1C, 0xD2, 0xB2, 0x02, 0x70, 0x8A, 0x42, 0x03, 0xD9, 0x04, 0x70, 0x01, 0x23, 0x00, 0xE0,
+ 0x04, 0x70, 0x18, 0x46, 0x10, 0xBD, 0x30, 0xB5, 0x00, 0x24, 0x01, 0x23, 0x05, 0xE0, 0x0C, 0x44,
+ 0xE4, 0x1C, 0x04, 0xF0, 0x03, 0x05, 0x64, 0x1B, 0x5B, 0x1C, 0x93, 0x42, 0xF7, 0xDD, 0x00, 0xEB,
+ 0x44, 0x00, 0x30, 0xBD, 0x70, 0xB5, 0x05, 0x46, 0x18, 0x46, 0x00, 0x24, 0x0E, 0xE0, 0x2B, 0x5D,
+ 0x58, 0x40, 0x08, 0x23, 0x06, 0x06, 0x02, 0xD5, 0x82, 0xEA, 0x40, 0x00, 0x00, 0xE0, 0x40, 0x00,
+ 0x5B, 0x1E, 0xC0, 0xB2, 0x13, 0xF0, 0xFF, 0x03, 0xF4, 0xD1, 0x64, 0x1C, 0x8C, 0x42, 0xEE, 0xD3,
+ 0x70, 0xBD, 0x00, 0x00, 0x88, 0x42, 0x01, 0xDD, 0x08, 0x46, 0x70, 0x47, 0x00, 0x28, 0xFC, 0xDA,
+ 0x00, 0x20, 0x70, 0x47, 0x10, 0xB5, 0x04, 0xE0, 0x00, 0xEB, 0x82, 0x03, 0x53, 0xF8, 0x04, 0x4C,
+ 0x1C, 0x60, 0x52, 0x1E, 0xF8, 0xD1, 0x09, 0x68, 0x01, 0x60, 0x10, 0xBD, 0x10, 0xB5, 0x03, 0xE0,
+ 0x00, 0xEB, 0x82, 0x03, 0x0C, 0x68, 0x1C, 0x60, 0x52, 0x1E, 0xF9, 0xD2, 0x10, 0xBD, 0x30, 0xB5,
+ 0x06, 0xE0, 0x00, 0xBF, 0x00, 0xEB, 0xC2, 0x03, 0x73, 0xE9, 0x02, 0x45, 0xC3, 0xE9, 0x02, 0x45,
+ 0x52, 0x1E, 0xF7, 0xD1, 0xD1, 0xE9, 0x00, 0x21, 0xC0, 0xE9, 0x00, 0x21, 0x30, 0xBD, 0x10, 0xB5,
+ 0x53, 0x88, 0x4C, 0x88, 0x1B, 0x1B, 0x43, 0x80, 0x12, 0x88, 0x09, 0x88, 0x51, 0x1A, 0x01, 0x80,
+ 0x10, 0xBD, 0x08, 0xB5, 0x0A, 0x46, 0x01, 0x46, 0x68, 0x46, 0xFF, 0xF7, 0xF0, 0xFF, 0xBD, 0xF9,
+ 0x00, 0x10, 0xBD, 0xF9, 0x02, 0x00, 0x49, 0x43, 0x00, 0xFB, 0x00, 0x10, 0x08, 0xBD, 0x10, 0xB5,
+ 0x88, 0x42, 0x02, 0x9C, 0x01, 0xDC, 0x20, 0x46, 0x10, 0xBD, 0x90, 0x42, 0x01, 0xDB, 0x18, 0x46,
+ 0x10, 0xBD, 0x8A, 0x42, 0x00, 0xD1, 0x52, 0x1C, 0x40, 0x1A, 0xE3, 0x1A, 0x58, 0x43, 0x51, 0x1A,
+ 0x90, 0xFB, 0xF1, 0xF0, 0x20, 0x1A, 0x10, 0xBD, 0x10, 0xB5, 0x00, 0x2A, 0x03, 0xD0, 0x43, 0x1A,
+ 0x93, 0x42, 0x01, 0xDD, 0x88, 0x18, 0x10, 0xBD, 0x54, 0x42, 0xA3, 0x42, 0xFB, 0xDA, 0x88, 0x1A,
+ 0x10, 0xBD, 0x91, 0x42, 0x01, 0xD1, 0x00, 0xB2, 0x70, 0x47, 0x50, 0x43, 0xB0, 0xFB, 0xF1, 0xF0,
+ 0x00, 0xB2, 0x90, 0x42, 0xF7, 0xD9, 0x10, 0x46, 0xF5, 0xE7, 0x70, 0xB5, 0x0B, 0x00, 0x02, 0x46,
+ 0x3D, 0xD0, 0x1F, 0x4C, 0x20, 0x68, 0x46, 0x8E, 0x85, 0x8E, 0xB2, 0xF9, 0x02, 0x00, 0x31, 0xB2,
+ 0xFF, 0xF7, 0x80, 0xFF, 0x58, 0x80, 0xB2, 0xF9, 0x00, 0x00, 0x29, 0xB2, 0xFF, 0xF7, 0x7A, 0xFF,
+ 0x18, 0x80, 0x21, 0x68, 0x91, 0xF8, 0x36, 0x10, 0x49, 0x07, 0x02, 0xD5, 0x59, 0x88, 0x71, 0x1A,
+ 0x59, 0x80, 0x21, 0x68, 0x91, 0xF8, 0x36, 0x10, 0x89, 0x07, 0x01, 0xD5, 0x28, 0x1A, 0x18, 0x80,
+ 0x20, 0x68, 0x90, 0xF8, 0x36, 0x10, 0x09, 0x07, 0x10, 0xD5, 0xB0, 0xF8, 0x37, 0x20, 0xB3, 0xF9,
+ 0x02, 0x00, 0x31, 0x46, 0xFF, 0xF7, 0xC5, 0xFF, 0x58, 0x80, 0x20, 0x68, 0x29, 0x46, 0xB0, 0xF8,
+ 0x39, 0x20, 0xB3, 0xF9, 0x00, 0x00, 0xFF, 0xF7, 0xBC, 0xFF, 0x18, 0x80, 0x20, 0x68, 0x90, 0xF8,
+ 0x36, 0x00, 0xC0, 0x07, 0x03, 0xD0, 0x58, 0x88, 0x19, 0x88, 0x59, 0x80, 0x18, 0x80, 0x70, 0xBD,
+ 0x20, 0x07, 0x10, 0x00, 0x73, 0xB5, 0x1D, 0x24, 0xDD, 0xE9, 0x00, 0x56, 0x22, 0x46, 0x01, 0x20,
+ 0x00, 0x21, 0x05, 0xF0, 0x18, 0xF9, 0x28, 0x40, 0x31, 0x40, 0x08, 0x43, 0x01, 0xD1, 0x64, 0x1E,
+ 0xF4, 0xD5, 0x08, 0x21, 0x68, 0x46, 0x03, 0xF0, 0x85, 0xFE, 0xFF, 0x49, 0xC0, 0xB2, 0x09, 0x68,
+ 0x91, 0xF8, 0xE7, 0x10, 0xC1, 0xF3, 0x41, 0x11, 0x02, 0x29, 0x0E, 0xD3, 0x10, 0xF0, 0x03, 0x00,
+ 0x05, 0xD0, 0x01, 0x28, 0x05, 0xD0, 0x02, 0x28, 0x05, 0xD0, 0x1F, 0x20, 0x0A, 0xE0, 0x0F, 0x20,
+ 0x08, 0xE0, 0x7F, 0x20, 0x06, 0xE0, 0x3F, 0x20, 0x04, 0xE0, 0xC0, 0x07, 0x01, 0xD0, 0x07, 0x20,
+ 0x00, 0xE0, 0x03, 0x20, 0x00, 0x21, 0x62, 0x1C, 0x05, 0xF0, 0xED, 0xF8, 0xDD, 0xE9, 0x00, 0x23,
+ 0x10, 0x43, 0x19, 0x43, 0xCD, 0xE9, 0x00, 0x01, 0x7C, 0xBD, 0x10, 0xB5, 0xEB, 0x4C, 0x50, 0x28,
+ 0x14, 0xD0, 0x08, 0xDC, 0x10, 0x28, 0x11, 0xD0, 0x11, 0x28, 0x25, 0xD0, 0x12, 0x28, 0x23, 0xD0,
+ 0x13, 0x28, 0x06, 0xD1, 0x20, 0xE0, 0x51, 0x28, 0x1E, 0xD0, 0x52, 0x28, 0x1C, 0xD0, 0x53, 0x28,
+ 0x1A, 0xD0, 0x00, 0x20, 0xC4, 0xE9, 0x02, 0x00, 0x01, 0x46, 0x1B, 0xE0, 0x02, 0xF0, 0x5D, 0xFC,
+ 0xC4, 0xE9, 0x02, 0x01, 0x02, 0xF0, 0x5B, 0xFC, 0xC4, 0xE9, 0x04, 0x01, 0xDA, 0x48, 0x00, 0x68,
+ 0x90, 0xF8, 0xE7, 0x00, 0x10, 0xF0, 0x60, 0x0F, 0x0E, 0xD0, 0xD4, 0xE9, 0x02, 0x01, 0xFF, 0xF7,
+ 0x99, 0xFF, 0xC4, 0xE9, 0x02, 0x01, 0x07, 0xE0, 0x02, 0xF0, 0x47, 0xFC, 0xC4, 0xE9, 0x02, 0x01,
+ 0x02, 0xF0, 0x45, 0xFC, 0xC4, 0xE9, 0x04, 0x01, 0xD0, 0x48, 0x08, 0x21, 0x08, 0x30, 0x03, 0xF0,
+ 0x21, 0xFE, 0x20, 0x60, 0xCD, 0x48, 0x08, 0x21, 0x10, 0x30, 0x03, 0xF0, 0x1B, 0xFE, 0x60, 0x60,
+ 0x10, 0xBD, 0x2D, 0xE9, 0xFF, 0x5F, 0x08, 0x21, 0xDD, 0xE9, 0x0F, 0xBA, 0x68, 0x46, 0x03, 0xF0,
+ 0x11, 0xFE, 0x81, 0x46, 0x08, 0x21, 0x02, 0xA8, 0x03, 0xF0, 0x0C, 0xFE, 0x07, 0x46, 0x00, 0x24,
+ 0x20, 0x46, 0x3A, 0xE0, 0x42, 0x46, 0x01, 0x20, 0x00, 0x21, 0x05, 0xF0, 0x8C, 0xF8, 0xDD, 0xE9,
+ 0x00, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x2D, 0xD0, 0x00, 0x25, 0x2E, 0x46, 0x27, 0xE0,
+ 0x32, 0x46, 0x01, 0x20, 0x00, 0x21, 0x05, 0xF0, 0x7E, 0xF8, 0xDD, 0xE9, 0x02, 0x23, 0x10, 0x40,
+ 0x19, 0x40, 0x08, 0x43, 0x1B, 0xD0, 0x11, 0x98, 0x60, 0xB1, 0x04, 0xFB, 0x07, 0xF0, 0x0B, 0xEB,
+ 0x05, 0x01, 0x41, 0x56, 0x32, 0x46, 0x01, 0xEB, 0x0A, 0x03, 0x21, 0x46, 0x0E, 0x98, 0x03, 0xF0,
+ 0x5C, 0xFA, 0x0B, 0xE0, 0x32, 0x46, 0x21, 0x46, 0x0E, 0x98, 0x03, 0xF0, 0x4B, 0xFA, 0x04, 0xFB,
+ 0x07, 0xF1, 0x0B, 0xEB, 0x05, 0x02, 0xA0, 0xEB, 0x0A, 0x00, 0x88, 0x54, 0x6D, 0x1C, 0x76, 0x1C,
+ 0xBD, 0x42, 0xD5, 0xDB, 0x64, 0x1C, 0x08, 0xF1, 0x01, 0x00, 0x80, 0x46, 0x4C, 0x45, 0xC1, 0xDB,
+ 0xBD, 0xE8, 0xFF, 0x9F, 0x2D, 0xE9, 0xFF, 0x5F, 0x84, 0xB0, 0x08, 0x21, 0xDD, 0xE9, 0x12, 0xAB,
+ 0x04, 0xA8, 0x03, 0xF0, 0xBF, 0xFD, 0x81, 0x46, 0x08, 0x21, 0x06, 0xA8, 0x03, 0xF0, 0xBA, 0xFD,
+ 0x80, 0x46, 0x00, 0x25, 0x0A, 0xF1, 0x48, 0x00, 0x02, 0x90, 0x20, 0x30, 0x01, 0x90, 0x48, 0x38,
+ 0x2C, 0x46, 0x2F, 0x46, 0x2E, 0x46, 0x00, 0x90, 0x34, 0xE0, 0x3A, 0x46, 0x01, 0x20, 0x00, 0x21,
+ 0x05, 0xF0, 0x31, 0xF8, 0xDD, 0xE9, 0x04, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x27, 0xD0,
+ 0x16, 0x98, 0x90, 0xB1, 0x1B, 0xF8, 0x05, 0x10, 0x14, 0x98, 0x08, 0x44, 0xC2, 0xB2, 0x39, 0x46,
+ 0x02, 0x98, 0x03, 0xF0, 0x7E, 0xFA, 0x05, 0xEB, 0x09, 0x01, 0x41, 0x44, 0x1B, 0xF9, 0x01, 0x20,
+ 0x39, 0x46, 0x50, 0x46, 0x03, 0xF0, 0x4D, 0xFA, 0x10, 0xE0, 0x39, 0x46, 0x02, 0x98, 0x03, 0xF0,
+ 0x6E, 0xFA, 0x14, 0x99, 0x40, 0x1A, 0x0B, 0xF8, 0x05, 0x00, 0x39, 0x46, 0x50, 0x46, 0x03, 0xF0,
+ 0x39, 0xFA, 0x05, 0xEB, 0x09, 0x02, 0x42, 0x44, 0x0B, 0xF8, 0x02, 0x00, 0x6D, 0x1C, 0xED, 0xB2,
+ 0x7F, 0x1C, 0xFF, 0xB2, 0x4D, 0x45, 0xC8, 0xDB, 0x38, 0xE0, 0x32, 0x46, 0x01, 0x20, 0x00, 0x21,
+ 0x04, 0xF0, 0xF9, 0xFF, 0xDD, 0xE9, 0x06, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x2B, 0xD0,
+ 0x16, 0x98, 0xA0, 0xB1, 0x04, 0xEB, 0x09, 0x01, 0x15, 0x98, 0x1B, 0xF8, 0x01, 0x10, 0x08, 0x44,
+ 0xC2, 0xB2, 0x31, 0x46, 0x01, 0x98, 0x03, 0xF0, 0x44, 0xFA, 0x04, 0xEB, 0x49, 0x01, 0x41, 0x44,
+ 0x1B, 0xF9, 0x01, 0x20, 0x31, 0x46, 0x00, 0x98, 0x03, 0xF0, 0x13, 0xFA, 0x12, 0xE0, 0x31, 0x46,
+ 0x01, 0x98, 0x03, 0xF0, 0x34, 0xFA, 0x15, 0x99, 0x04, 0xEB, 0x09, 0x02, 0x40, 0x1A, 0x0B, 0xF8,
+ 0x02, 0x00, 0x31, 0x46, 0x00, 0x98, 0x03, 0xF0, 0xFD, 0xF9, 0x04, 0xEB, 0x49, 0x02, 0x42, 0x44,
+ 0x0B, 0xF8, 0x02, 0x00, 0x64, 0x1C, 0xE4, 0xB2, 0x76, 0x1C, 0xF6, 0xB2, 0x44, 0x45, 0xC4, 0xDB,
+ 0x08, 0xB0, 0xBD, 0xE8, 0xF0, 0x9F, 0x2D, 0xE9, 0xFF, 0x5F, 0x08, 0x21, 0xDD, 0xE9, 0x0E, 0xBA,
+ 0x68, 0x46, 0x03, 0xF0, 0x2F, 0xFD, 0x81, 0x46, 0x08, 0x21, 0x02, 0xA8, 0x03, 0xF0, 0x2A, 0xFD,
+ 0x07, 0x46, 0x00, 0x24, 0x20, 0x46, 0x29, 0xE0, 0x42, 0x46, 0x01, 0x20, 0x00, 0x21, 0x04, 0xF0,
+ 0xAA, 0xFF, 0xDD, 0xE9, 0x00, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x1C, 0xD0, 0x00, 0x25,
+ 0x2E, 0x46, 0x16, 0xE0, 0x32, 0x46, 0x01, 0x20, 0x00, 0x21, 0x04, 0xF0, 0x9C, 0xFF, 0xDD, 0xE9,
+ 0x02, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x0A, 0xD0, 0x32, 0x46, 0x21, 0x46, 0x58, 0x46,
+ 0x03, 0xF0, 0x78, 0xF9, 0x04, 0xFB, 0x07, 0x52, 0x80, 0x00, 0x2A, 0xF8, 0x12, 0x00, 0x6D, 0x1C,
+ 0x76, 0x1C, 0xBD, 0x42, 0xE6, 0xDB, 0x64, 0x1C, 0x08, 0xF1, 0x01, 0x00, 0x80, 0x46, 0x4C, 0x45,
+ 0xD2, 0xDB, 0x2D, 0xE7, 0x2D, 0xE9, 0xFF, 0x5F, 0x08, 0x21, 0xDD, 0xE9, 0x0E, 0x9B, 0x68, 0x46,
+ 0x03, 0xF0, 0xF0, 0xFC, 0x82, 0x46, 0x08, 0x21, 0x02, 0xA8, 0x03, 0xF0, 0xEB, 0xFC, 0x00, 0x25,
+ 0x80, 0x46, 0x2C, 0x46, 0x2F, 0x46, 0x2E, 0x46, 0x3D, 0xE0, 0x3A, 0x46, 0x01, 0x20, 0x00, 0x21,
+ 0x04, 0xF0, 0x69, 0xFF, 0xDD, 0xE9, 0x00, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43, 0x30, 0xD0,
+ 0x39, 0x46, 0x09, 0xF1, 0x48, 0x00, 0x03, 0xF0, 0xBA, 0xF9, 0x27, 0x49, 0x99, 0xF8, 0x92, 0xC0,
+ 0x0A, 0x68, 0x92, 0xF8, 0x43, 0x31, 0x92, 0xF8, 0x42, 0x21, 0x03, 0xF0, 0x0F, 0x01, 0x49, 0x1C,
+ 0x0C, 0xFB, 0x01, 0xFC, 0x01, 0x21, 0x01, 0xEB, 0x13, 0x13, 0x00, 0xFB, 0x03, 0xC3, 0x99, 0xF8,
+ 0x90, 0x00, 0x01, 0xEB, 0x12, 0x11, 0x00, 0xFB, 0x01, 0x30, 0x10, 0x99, 0x19, 0xB9, 0x4F, 0xF6,
+ 0xFF, 0x71, 0x01, 0xEA, 0x80, 0x00, 0x2B, 0xF8, 0x15, 0x00, 0x39, 0x46, 0x48, 0x46, 0x03, 0xF0,
+ 0x69, 0xF9, 0x05, 0xEB, 0x0A, 0x02, 0x80, 0x00, 0x42, 0x44, 0x6D, 0x1C, 0x2B, 0xF8, 0x12, 0x00,
+ 0xED, 0xB2, 0x7F, 0x1C, 0xFF, 0xB2, 0x55, 0x45, 0xBF, 0xDB, 0x44, 0xE0, 0x32, 0x46, 0x01, 0x20,
+ 0x00, 0x21, 0x04, 0xF0, 0x28, 0xFF, 0xDD, 0xE9, 0x02, 0x23, 0x10, 0x40, 0x19, 0x40, 0x08, 0x43,
+ 0x37, 0xD0, 0x31, 0x46, 0x09, 0xF1, 0x68, 0x00, 0x03, 0xF0, 0x79, 0xF9, 0x06, 0x49, 0x99, 0xF8,
+ 0x93, 0x50, 0x0B, 0x68, 0x93, 0xF8, 0x48, 0x11, 0x01, 0xF0, 0x0F, 0x02, 0x52, 0x1C, 0x55, 0x43,
+ 0x01, 0x22, 0x02, 0xEB, 0x11, 0x11, 0x03, 0xE0, 0x20, 0x07, 0x10, 0x00, 0x70, 0x07, 0x10, 0x00,
+ 0x00, 0xFB, 0x01, 0x51, 0x93, 0xF8, 0x47, 0x31, 0x99, 0xF8, 0x91, 0x00, 0x02, 0xEB, 0x13, 0x12,
+ 0x00, 0xFB, 0x02, 0x10, 0x10, 0x99, 0x19, 0xB9, 0x4F, 0xF6, 0xFF, 0x71, 0x01, 0xEA, 0x80, 0x00,
+ 0x04, 0xEB, 0x0A, 0x02, 0x31, 0x46, 0x2B, 0xF8, 0x12, 0x00, 0x09, 0xF1, 0x20, 0x00, 0x03, 0xF0,
+ 0x21, 0xF9, 0x04, 0xEB, 0x4A, 0x02, 0x80, 0x00, 0x42, 0x44, 0x64, 0x1C, 0x2B, 0xF8, 0x12, 0x00,
+ 0xE4, 0xB2, 0x76, 0x1C, 0xF6, 0xB2, 0x44, 0x45, 0xB8, 0xDB, 0x91, 0xE6, 0x2D, 0xE9, 0xF0, 0x43,
+ 0x89, 0x46, 0x00, 0x27, 0x85, 0xB0, 0x90, 0x46, 0x04, 0x46, 0x3E, 0x46, 0xFC, 0x21, 0x04, 0xF0,
+ 0x90, 0xFF, 0xFE, 0x4D, 0xB8, 0xF1, 0x01, 0x0F, 0x28, 0x78, 0x20, 0x71, 0x28, 0x79, 0x60, 0x71,
+ 0xFB, 0x48, 0x90, 0xF8, 0x82, 0x1B, 0xA1, 0x71, 0x90, 0xF8, 0x83, 0x1B, 0xE1, 0x71, 0x90, 0xF8,
+ 0x84, 0x1B, 0xA1, 0x72, 0x90, 0xF8, 0x85, 0x1B, 0xE1, 0x72, 0x90, 0xF8, 0x80, 0x1B, 0x21, 0x73,
+ 0x90, 0xF8, 0x81, 0x0B, 0x60, 0x73, 0x08, 0xD1, 0xF2, 0x48, 0x00, 0x68, 0x81, 0x7F, 0xA5, 0x29,
+ 0x03, 0xD1, 0x90, 0xF9, 0x1F, 0x70, 0x90, 0xF9, 0x20, 0x60, 0x04, 0xF1, 0x10, 0x00, 0xCD, 0xE9,
+ 0x01, 0x07, 0xED, 0x48, 0x00, 0x90, 0x00, 0x21, 0xCD, 0xE9, 0x03, 0x61, 0x05, 0xF1, 0x08, 0x00,
+ 0x0F, 0xC8, 0xFF, 0xF7, 0x57, 0xFE, 0xA5, 0x20, 0x20, 0x70, 0x12, 0x20, 0x60, 0x70, 0xA4, 0xF8,
+ 0x02, 0x90, 0xD5, 0xE9, 0x00, 0x01, 0x08, 0x44, 0x13, 0x30, 0x80, 0x08, 0x05, 0xB0, 0xBD, 0xE8,
+ 0xF0, 0x83, 0x2D, 0xE9, 0xF0, 0x43, 0x89, 0x46, 0x00, 0x27, 0x85, 0xB0, 0x90, 0x46, 0x04, 0x46,
+ 0x3E, 0x46, 0xFC, 0x21, 0x04, 0xF0, 0x45, 0xFF, 0xD8, 0x4D, 0xB8, 0xF1, 0x01, 0x0F, 0x28, 0x78,
+ 0x20, 0x71, 0x28, 0x79, 0x60, 0x71, 0xD6, 0x48, 0x90, 0xF8, 0x1A, 0x1C, 0xA1, 0x71, 0x90, 0xF8,
+ 0x1B, 0x1C, 0xE1, 0x71, 0x90, 0xF8, 0x1C, 0x1C, 0xA1, 0x72, 0x90, 0xF8, 0x1D, 0x1C, 0xE1, 0x72,
+ 0x90, 0xF8, 0x18, 0x1C, 0x21, 0x73, 0x90, 0xF8, 0x19, 0x0C, 0x60, 0x73, 0x09, 0xD1, 0xCD, 0x48,
+ 0x00, 0x68, 0x90, 0xF8, 0x21, 0x10, 0xA5, 0x29, 0x03, 0xD1, 0x10, 0xF9, 0x22, 0x7F, 0x90, 0xF9,
+ 0x01, 0x60, 0x04, 0xF1, 0x10, 0x00, 0xCD, 0xE9, 0x01, 0x07, 0xC7, 0x48, 0x00, 0x21, 0x98, 0x30,
+ 0x00, 0x90, 0xCD, 0xE9, 0x03, 0x61, 0x05, 0xF1, 0x08, 0x00, 0x0F, 0xC8, 0xFF, 0xF7, 0x0A, 0xFE,
+ 0xA5, 0x20, 0x20, 0x70, 0x13, 0x20, 0x60, 0x70, 0xA4, 0xF8, 0x02, 0x90, 0xD5, 0xE9, 0x00, 0x01,
+ 0x08, 0x44, 0x13, 0x30, 0x80, 0x08, 0xB1, 0xE7, 0x70, 0xB5, 0x0E, 0x46, 0x04, 0x46, 0x00, 0x25,
+ 0xFF, 0xF7, 0x63, 0xFD, 0x10, 0x2C, 0x06, 0xD0, 0x11, 0x2C, 0x08, 0xD0, 0x12, 0x2C, 0x0C, 0xD0,
+ 0x13, 0x2C, 0x08, 0xD1, 0x0D, 0xE0, 0x30, 0x46, 0x00, 0xF0, 0x7D, 0xF8, 0x02, 0xE0, 0x30, 0x46,
+ 0x00, 0xF0, 0x9C, 0xF8, 0x05, 0x46, 0x28, 0x46, 0x70, 0xBD, 0x30, 0x46, 0x00, 0xF0, 0xB8, 0xF8,
+ 0xF8, 0xE7, 0x30, 0x46, 0x00, 0xF0, 0xEE, 0xF8, 0xF4, 0xE7, 0x2D, 0xE9, 0xF0, 0x41, 0x1E, 0x46,
+ 0x17, 0x46, 0x88, 0x46, 0x05, 0x46, 0x00, 0x24, 0xFF, 0xF7, 0x3F, 0xFD, 0x10, 0x2D, 0x06, 0xD0,
+ 0x11, 0x2D, 0x0A, 0xD0, 0x12, 0x2D, 0x11, 0xD0, 0x13, 0x2D, 0x0C, 0xD1, 0x14, 0xE0, 0x32, 0x46,
+ 0x39, 0x46, 0x40, 0x46, 0x00, 0xF0, 0x11, 0xF9, 0x04, 0xE0, 0x32, 0x46, 0x39, 0x46, 0x40, 0x46,
+ 0x00, 0xF0, 0x4A, 0xF9, 0x04, 0x46, 0x20, 0x46, 0xBD, 0xE8, 0xF0, 0x81, 0x32, 0x46, 0x39, 0x46,
+ 0x40, 0x46, 0xFF, 0xF7, 0x23, 0xFF, 0xF5, 0xE7, 0x32, 0x46, 0x39, 0x46, 0x40, 0x46, 0xFF, 0xF7,
+ 0x68, 0xFF, 0xEF, 0xE7, 0xFE, 0xB5, 0x17, 0x46, 0x06, 0x46, 0x0D, 0x46, 0xFF, 0xF7, 0x15, 0xFD,
+ 0x8E, 0x48, 0x05, 0xF1, 0x10, 0x04, 0x50, 0x2E, 0x01, 0x78, 0x29, 0x71, 0x01, 0x79, 0x69, 0x71,
+ 0x07, 0xD0, 0x51, 0x2E, 0x07, 0xD0, 0x8D, 0x49, 0x52, 0x2E, 0x0F, 0xD0, 0x53, 0x2E, 0x25, 0xD1,
+ 0x13, 0xE0, 0x87, 0x49, 0x00, 0xE0, 0x8A, 0x49, 0xCD, 0xE9, 0x00, 0x14, 0x84, 0x68, 0x0C, 0x30,
+ 0x90, 0xE8, 0x0E, 0x00, 0x20, 0x46, 0xFF, 0xF7, 0x1E, 0xFE, 0x17, 0xE0, 0x09, 0x68, 0x82, 0x4A,
+ 0x91, 0xF8, 0x42, 0x11, 0x01, 0xF0, 0x01, 0x01, 0x06, 0xE0, 0x09, 0x68, 0x7E, 0x4A, 0x91, 0xF8,
+ 0x92, 0x19, 0x98, 0x32, 0x01, 0xF0, 0x01, 0x01, 0x00, 0x92, 0xCD, 0xE9, 0x01, 0x41, 0x84, 0x68,
+ 0x0C, 0x30, 0x90, 0xE8, 0x0E, 0x00, 0x20, 0x46, 0xFF, 0xF7, 0x44, 0xFE, 0xA5, 0x20, 0x28, 0x70,
+ 0x6E, 0x70, 0x6F, 0x80, 0xFE, 0xBD, 0x1F, 0xB5, 0x70, 0x49, 0x02, 0x79, 0x0B, 0x68, 0x9A, 0x42,
+ 0x03, 0xD1, 0x42, 0x79, 0x4B, 0x68, 0x9A, 0x42, 0x02, 0xD0, 0x00, 0x20, 0x04, 0xB0, 0x10, 0xBD,
+ 0x6C, 0x4B, 0x00, 0x22, 0x1B, 0x68, 0x9C, 0x7E, 0xA5, 0x2C, 0x01, 0xD1, 0x93, 0xF9, 0x1B, 0x20,
+ 0x67, 0x4C, 0x0D, 0xF1, 0x04, 0x0C, 0x01, 0x23, 0x10, 0x30, 0x00, 0x94, 0x8C, 0xE8, 0x0D, 0x00,
+ 0x08, 0x31, 0x0F, 0xC9, 0xFF, 0xF7, 0xFD, 0xFC, 0x01, 0x20, 0xE7, 0xE7, 0x1F, 0xB5, 0x5F, 0x49,
+ 0x02, 0x79, 0x0B, 0x68, 0x9A, 0x42, 0x03, 0xD1, 0x42, 0x79, 0x4B, 0x68, 0x9A, 0x42, 0x01, 0xD0,
+ 0x00, 0x20, 0xDB, 0xE7, 0x5B, 0x4B, 0x00, 0x22, 0x1B, 0x68, 0x1C, 0x7F, 0xA5, 0x2C, 0x01, 0xD1,
+ 0x93, 0xF9, 0x1D, 0x20, 0x5A, 0x4B, 0x0D, 0xF1, 0x04, 0x0C, 0x01, 0x24, 0x10, 0x30, 0x00, 0x93,
+ 0x8C, 0xE8, 0x15, 0x00, 0x08, 0x31, 0x0F, 0xC9, 0xFF, 0xF7, 0xDB, 0xFC, 0x01, 0x20, 0xC5, 0xE7,
+ 0x30, 0xB5, 0x4E, 0x49, 0x02, 0x79, 0x85, 0xB0, 0x0B, 0x68, 0x9A, 0x42, 0x03, 0xD1, 0x42, 0x79,
+ 0x4B, 0x68, 0x9A, 0x42, 0x02, 0xD0, 0x00, 0x20, 0x05, 0xB0, 0x30, 0xBD, 0x48, 0x4A, 0x85, 0x79,
+ 0x00, 0x24, 0x82, 0xF8, 0x82, 0x5B, 0xC5, 0x79, 0x82, 0xF8, 0x83, 0x5B, 0x85, 0x7A, 0x82, 0xF8,
+ 0x84, 0x5B, 0xC5, 0x7A, 0x82, 0xF8, 0x85, 0x5B, 0x05, 0x7B, 0x82, 0xF8, 0x80, 0x5B, 0x45, 0x7B,
+ 0x82, 0xF8, 0x81, 0x5B, 0x3F, 0x4A, 0x23, 0x46, 0x12, 0x68, 0x95, 0x7F, 0xA5, 0x2D, 0x03, 0xD1,
+ 0x92, 0xF9, 0x1F, 0x40, 0x92, 0xF9, 0x20, 0x30, 0x10, 0x30, 0xCD, 0xE9, 0x01, 0x04, 0x3A, 0x48,
+ 0x01, 0x22, 0x00, 0x90, 0xCD, 0xE9, 0x03, 0x32, 0x08, 0x31, 0x0F, 0xC9, 0xFF, 0xF7, 0xF2, 0xFC,
+ 0x01, 0x20, 0xD1, 0xE7, 0x30, 0xB5, 0x31, 0x49, 0x02, 0x79, 0x85, 0xB0, 0x0B, 0x68, 0x9A, 0x42,
+ 0x03, 0xD1, 0x42, 0x79, 0x4B, 0x68, 0x9A, 0x42, 0x01, 0xD0, 0x00, 0x20, 0xC4, 0xE7, 0x2C, 0x4A,
+ 0x85, 0x79, 0x00, 0x24, 0x82, 0xF8, 0x1A, 0x5C, 0xC5, 0x79, 0x82, 0xF8, 0x1B, 0x5C, 0x85, 0x7A,
+ 0x82, 0xF8, 0x1C, 0x5C, 0xC5, 0x7A, 0x82, 0xF8, 0x1D, 0x5C, 0x05, 0x7B, 0x82, 0xF8, 0x18, 0x5C,
+ 0x45, 0x7B, 0x82, 0xF8, 0x19, 0x5C, 0x23, 0x4A, 0x23, 0x46, 0x12, 0x68, 0x92, 0xF8, 0x21, 0x50,
+ 0xA5, 0x2D, 0x03, 0xD1, 0x12, 0xF9, 0x22, 0x4F, 0x92, 0xF9, 0x01, 0x30, 0x10, 0x30, 0xCD, 0xE9,
+ 0x01, 0x04, 0x1D, 0x48, 0x01, 0x22, 0x98, 0x30, 0x00, 0x90, 0xCD, 0xE9, 0x03, 0x32, 0x08, 0x31,
+ 0x0F, 0xC9, 0xFF, 0xF7, 0xB7, 0xFC, 0x01, 0x20, 0x96, 0xE7, 0x2D, 0xE9, 0xFF, 0x41, 0x0F, 0x46,
+ 0x15, 0x46, 0x04, 0x46, 0x00, 0x26, 0x40, 0xF2, 0x7A, 0x31, 0x04, 0xF0, 0xB2, 0xFD, 0x0F, 0x48,
+ 0x01, 0x2D, 0x01, 0x78, 0x21, 0x71, 0x01, 0x79, 0x61, 0x71, 0x06, 0xD1, 0x0D, 0x49, 0x09, 0x68,
+ 0x8A, 0x7E, 0xA5, 0x2A, 0x01, 0xD1, 0x91, 0xF9, 0x1B, 0x60, 0x04, 0xF1, 0x10, 0x01, 0x08, 0x4B,
+ 0xCD, 0xE9, 0x00, 0x31, 0x00, 0x22, 0xCD, 0xE9, 0x02, 0x62, 0x00, 0xF1, 0x08, 0x01, 0x05, 0x46,
+ 0x0F, 0xC9, 0xFF, 0xF7, 0x3E, 0xFC, 0xA5, 0x20, 0x20, 0x70, 0x0B, 0xE0, 0x70, 0x07, 0x10, 0x00,
+ 0xFC, 0x4C, 0x01, 0x20, 0x24, 0x07, 0x10, 0x00, 0xEC, 0x57, 0x01, 0x20, 0x20, 0x07, 0x10, 0x00,
+ 0x24, 0x52, 0x01, 0x20, 0x10, 0x20, 0x60, 0x70, 0x67, 0x80, 0xD5, 0xE9, 0x00, 0x01, 0x48, 0x43,
+ 0x13, 0x30, 0x80, 0x08, 0x04, 0xB0, 0xB7, 0xE6, 0x2D, 0xE9, 0xFF, 0x41, 0x0F, 0x46, 0x15, 0x46,
+ 0x04, 0x46, 0x00, 0x26, 0x40, 0xF2, 0x7A, 0x31, 0x04, 0xF0, 0x73, 0xFD, 0x13, 0x48, 0x01, 0x2D,
+ 0x01, 0x78, 0x21, 0x71, 0x01, 0x79, 0x61, 0x71, 0x06, 0xD1, 0x11, 0x49, 0x09, 0x68, 0x0A, 0x7F,
+ 0xA5, 0x2A, 0x01, 0xD1, 0x91, 0xF9, 0x1D, 0x60, 0x04, 0xF1, 0x10, 0x01, 0x0D, 0x4A, 0xCD, 0xE9,
+ 0x00, 0x21, 0x00, 0x23, 0xCD, 0xE9, 0x02, 0x63, 0x00, 0xF1, 0x08, 0x01, 0x05, 0x46, 0x0F, 0xC9,
+ 0xFF, 0xF7, 0xFF, 0xFB, 0xA5, 0x20, 0x20, 0x70, 0x11, 0x20, 0x60, 0x70, 0x67, 0x80, 0xD5, 0xE9,
+ 0x00, 0x01, 0x48, 0x43, 0x13, 0x30, 0x80, 0x08, 0xCC, 0xE7, 0x00, 0x00, 0x70, 0x07, 0x10, 0x00,
+ 0x24, 0x07, 0x10, 0x00, 0x24, 0x52, 0x01, 0x20, 0x2D, 0xE9, 0xF0, 0x4F, 0x47, 0xF6, 0xFF, 0x77,
+ 0xDD, 0xE9, 0x09, 0xB5, 0x81, 0x46, 0xFE, 0x43, 0x01, 0x20, 0x9A, 0x46, 0x84, 0x46, 0x2C, 0x46,
+ 0x15, 0xE0, 0x00, 0xBF, 0x32, 0xF8, 0x14, 0x30, 0x31, 0xF8, 0x14, 0x80, 0xA3, 0xEB, 0x08, 0x03,
+ 0x1B, 0xB2, 0xBB, 0x42, 0x00, 0xDA, 0x1F, 0x46, 0xB3, 0x42, 0x00, 0xDD, 0x1E, 0x46, 0x00, 0x2B,
+ 0x02, 0xDD, 0x4F, 0xF0, 0x00, 0x0C, 0x02, 0xE0, 0x01, 0xDA, 0x4F, 0xF0, 0x00, 0x00, 0x64, 0x1E,
+ 0xE8, 0xD2, 0x20, 0xB1, 0xBC, 0xF1, 0x00, 0x0F, 0x01, 0xD0, 0x00, 0x23, 0x09, 0xE0, 0x50, 0xEA,
+ 0x0C, 0x03, 0x06, 0xD0, 0x20, 0xB9, 0xBC, 0xF1, 0x00, 0x0F, 0x01, 0xD0, 0x33, 0x46, 0x00, 0xE0,
+ 0x3B, 0x46, 0x00, 0x27, 0x3C, 0x46, 0x28, 0x46, 0xCB, 0xF1, 0x00, 0x0C, 0x0E, 0xE0, 0x00, 0xBF,
+ 0x32, 0xF8, 0x10, 0x60, 0x31, 0xF8, 0x10, 0x80, 0xA6, 0xEB, 0x08, 0x06, 0xF6, 0x1A, 0x36, 0xB2,
+ 0x56, 0x45, 0x03, 0xDA, 0x66, 0x45, 0x01, 0xDD, 0x37, 0x44, 0x64, 0x1C, 0x40, 0x1E, 0xEF, 0xD2,
+ 0x04, 0xB9, 0x01, 0x24, 0x97, 0xFB, 0xF4, 0xF0, 0x18, 0x44, 0x04, 0xE0, 0x31, 0xF8, 0x15, 0x20,
+ 0x02, 0x44, 0x29, 0xF8, 0x15, 0x20, 0x6D, 0x1E, 0xF8, 0xD2, 0xBD, 0xE8, 0xF0, 0x8F, 0xF0, 0xB5,
+ 0x00, 0x23, 0x06, 0x46, 0x1D, 0x46, 0x18, 0x46, 0x0E, 0xE0, 0x00, 0xBF, 0x36, 0xF9, 0x11, 0x40,
+ 0x00, 0x2C, 0x01, 0xDB, 0x27, 0x46, 0x00, 0xE0, 0x67, 0x42, 0x97, 0x42, 0x04, 0xDD, 0x00, 0x2C,
+ 0x01, 0xDD, 0x25, 0x44, 0x00, 0xE0, 0x23, 0x44, 0x49, 0x1E, 0xEF, 0xD2, 0x00, 0x2B, 0x00, 0xDA,
+ 0x5B, 0x42, 0xAB, 0x42, 0x00, 0xDD, 0x01, 0x20, 0xF0, 0xBD, 0x30, 0xB5, 0x07, 0xE0, 0x00, 0xBF,
+ 0x32, 0xF8, 0x13, 0x40, 0x31, 0xF8, 0x13, 0x50, 0x64, 0x1B, 0x20, 0xF8, 0x13, 0x40, 0x5B, 0x1E,
+ 0xF6, 0xD2, 0x30, 0xBD, 0x70, 0xB5, 0x04, 0x9C, 0x09, 0xE0, 0x00, 0xBF, 0x31, 0xF9, 0x14, 0x50,
+ 0x32, 0xF9, 0x14, 0x60, 0x75, 0x43, 0x95, 0xFB, 0xF3, 0xF5, 0x20, 0xF8, 0x14, 0x50, 0x64, 0x1E,
+ 0xF4, 0xD2, 0x70, 0xBD, 0xF0, 0xB4, 0x04, 0x9C, 0x1C, 0xB9, 0x13, 0xB1, 0xC3, 0xF1, 0x10, 0x04,
+ 0x0E, 0xE0, 0xF0, 0xBC, 0x52, 0x00, 0x04, 0xF0, 0x25, 0xBC, 0x00, 0xBF, 0x30, 0xF9, 0x12, 0x50,
+ 0x31, 0xF9, 0x12, 0x60, 0x5D, 0x43, 0x06, 0xFB, 0x04, 0x55, 0x2D, 0x11, 0x20, 0xF8, 0x12, 0x50,
+ 0x52, 0x1E, 0xF3, 0xD2, 0xF0, 0xBC, 0x70, 0x47, 0xF0, 0xB4, 0x04, 0x9C, 0x1C, 0xB9, 0x13, 0xB1,
+ 0xC3, 0xF1, 0x80, 0x04, 0x0E, 0xE0, 0xF0, 0xBC, 0x52, 0x00, 0x04, 0xF0, 0x0B, 0xBC, 0x00, 0xBF,
+ 0x30, 0xF9, 0x12, 0x50, 0x31, 0xF9, 0x12, 0x60, 0x5D, 0x43, 0x06, 0xFB, 0x04, 0x55, 0xED, 0x11,
+ 0x20, 0xF8, 0x12, 0x50, 0x52, 0x1E, 0xF3, 0xD2, 0xE4, 0xE7, 0xF0, 0xB5, 0x00, 0x2B, 0x19, 0xD0,
+ 0x14, 0x46, 0x15, 0xE0, 0x00, 0x27, 0x3E, 0x46, 0xE5, 0x1A, 0x04, 0xEB, 0x03, 0x0C, 0x08, 0xE0,
+ 0x00, 0x2D, 0x05, 0xDB, 0x95, 0x42, 0x03, 0xDA, 0x31, 0xF9, 0x15, 0xE0, 0x76, 0x1C, 0x77, 0x44,
+ 0x6D, 0x1C, 0xAC, 0x45, 0xF4, 0xDA, 0x1E, 0xB1, 0x97, 0xFB, 0xF6, 0xF5, 0x20, 0xF8, 0x14, 0x50,
+ 0x64, 0x1E, 0xE7, 0xD2, 0xF0, 0xBD, 0x70, 0xB5, 0x0C, 0x46, 0x15, 0x46, 0xA0, 0x42, 0x04, 0xD0,
+ 0x01, 0x46, 0x6A, 0x00, 0x20, 0x46, 0x04, 0xF0, 0xD5, 0xFB, 0x00, 0x20, 0x17, 0xE0, 0x01, 0x46,
+ 0x42, 0x1C, 0x07, 0xE0, 0x34, 0xF9, 0x12, 0x60, 0x34, 0xF9, 0x11, 0x30, 0x9E, 0x42, 0x00, 0xDA,
+ 0x11, 0x46, 0x52, 0x1C, 0xAA, 0x42, 0xF5, 0xD3, 0x88, 0x42, 0x07, 0xD0, 0x34, 0xF8, 0x10, 0x20,
+ 0x34, 0xF8, 0x11, 0x30, 0x24, 0xF8, 0x10, 0x30, 0x24, 0xF8, 0x11, 0x20, 0x40, 0x1C, 0xB0, 0xEB,
+ 0x55, 0x0F, 0xE4, 0xD9, 0xE9, 0x07, 0x25, 0xF0, 0x01, 0x00, 0x01, 0xD0, 0x20, 0x5E, 0x70, 0xBD,
+ 0x21, 0x5E, 0x20, 0x44, 0x30, 0xF9, 0x02, 0x0C, 0x08, 0x44, 0x40, 0x10, 0x70, 0xBD, 0x03, 0x46,
+ 0x00, 0x20, 0x04, 0xE0, 0x33, 0xF9, 0x11, 0x20, 0x82, 0x42, 0x00, 0xDD, 0x10, 0x46, 0x49, 0x1E,
+ 0xF8, 0xD2, 0x70, 0x47, 0x03, 0x46, 0x47, 0xF6, 0xFF, 0x70, 0x04, 0xE0, 0x33, 0xF9, 0x11, 0x20,
+ 0x82, 0x42, 0x00, 0xDA, 0x10, 0x46, 0x49, 0x1E, 0xF8, 0xD2, 0x00, 0x28, 0x00, 0xDA, 0x00, 0x20,
+ 0x70, 0x47, 0x02, 0x46, 0xB0, 0xF9, 0x00, 0x00, 0x05, 0xE0, 0x00, 0xBF, 0x32, 0xF9, 0x11, 0x30,
+ 0x83, 0x42, 0x00, 0xDD, 0x18, 0x46, 0x49, 0x1E, 0xF8, 0xD2, 0x70, 0x47, 0x70, 0xB5, 0x05, 0x00,
+ 0x03, 0xD0, 0xBD, 0xE8, 0x70, 0x40, 0x01, 0xF0, 0xBE, 0xBE, 0xFE, 0x4C, 0x21, 0x46, 0x60, 0x78,
+ 0x09, 0x79, 0x30, 0xB1, 0x0A, 0x46, 0xFC, 0x49, 0x01, 0xF1, 0x98, 0x00, 0x01, 0xF0, 0x00, 0xFF,
+ 0x0B, 0xE0, 0x20, 0x78, 0x01, 0xF0, 0xCA, 0xFF, 0x21, 0x79, 0x4A, 0x00, 0xF6, 0x49, 0x01, 0xEB,
+ 0x40, 0x01, 0xF5, 0x48, 0x98, 0x30, 0x04, 0xF0, 0x65, 0xFB, 0xA0, 0x78, 0xF3, 0x4E, 0x28, 0xB1,
+ 0xF1, 0x48, 0x21, 0x79, 0x98, 0x30, 0x01, 0xF0, 0xB7, 0xFF, 0x01, 0xE0, 0xEE, 0x48, 0x98, 0x30,
+ 0xB0, 0x62, 0xE0, 0x78, 0x86, 0xF8, 0x31, 0x00, 0x28, 0x46, 0xD2, 0xE7, 0x2D, 0xE9, 0xF0, 0x5F,
+ 0xDF, 0xF8, 0xAC, 0xA3, 0x83, 0x46, 0x01, 0x27, 0xDA, 0xF8, 0x00, 0x00, 0xB8, 0x46, 0x00, 0xF2,
+ 0x5E, 0x64, 0x90, 0xF8, 0xA6, 0x19, 0x90, 0xF8, 0xA2, 0x59, 0x01, 0xF0, 0x1F, 0x06, 0x90, 0xF8,
+ 0xA3, 0x19, 0x01, 0xF0, 0x0F, 0x09, 0x90, 0xF8, 0x82, 0x19, 0x89, 0x06, 0x09, 0xD5, 0x90, 0xF8,
+ 0x58, 0x16, 0x90, 0xF8, 0x59, 0x06, 0x0A, 0x22, 0x04, 0xEB, 0x40, 0x00, 0x01, 0xF0, 0xFC, 0xFE,
+ 0x07, 0x46, 0xDA, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x82, 0x19, 0x49, 0x06, 0x09, 0xD5, 0x90, 0xF8,
+ 0x68, 0x18, 0x90, 0xF8, 0x69, 0x08, 0x0A, 0x22, 0x04, 0xEB, 0x40, 0x00, 0x01, 0xF0, 0xEC, 0xFE,
+ 0x80, 0x46, 0x4F, 0xF4, 0x80, 0x70, 0x85, 0x42, 0x00, 0xD9, 0x05, 0x46, 0x1F, 0x2E, 0x00, 0xD9,
+ 0x1F, 0x26, 0x76, 0x1C, 0x75, 0x43, 0x09, 0xF1, 0x01, 0x00, 0x45, 0x43, 0xC5, 0xEB, 0x45, 0x20,
+ 0x78, 0x43, 0xDA, 0xF8, 0x00, 0x20, 0x00, 0xFB, 0x08, 0xF0, 0xC0, 0x13, 0x92, 0xF8, 0x50, 0x1A,
+ 0xB0, 0xFA, 0x80, 0xF0, 0xC0, 0xF1, 0x20, 0x00, 0x89, 0x06, 0x11, 0xD5, 0x00, 0x21, 0xBB, 0xF1,
+ 0x01, 0x0F, 0x06, 0xD1, 0x92, 0xF8, 0x56, 0x1A, 0x0A, 0x07, 0x01, 0xF0, 0x07, 0x01, 0x00, 0xD5,
+ 0x49, 0x42, 0x40, 0x18, 0x01, 0xD5, 0x00, 0x20, 0x02, 0xE0, 0x1F, 0x28, 0x00, 0xDD, 0x1F, 0x20,
+ 0xC0, 0xB2, 0xBD, 0xE8, 0xF0, 0x9F, 0xF8, 0xB5, 0x04, 0x46, 0x02, 0xF0, 0xD7, 0xFB, 0xB6, 0x48,
+ 0x02, 0xF0, 0xD9, 0xFB, 0xA4, 0xF8, 0x76, 0x00, 0x01, 0xF0, 0x98, 0xFE, 0xB1, 0x4A, 0x0B, 0x46,
+ 0x12, 0x78, 0x00, 0x92, 0x02, 0x46, 0x20, 0x46, 0x02, 0xF0, 0x97, 0xFE, 0x01, 0xF0, 0x93, 0xFE,
+ 0x02, 0x46, 0x0B, 0x46, 0x20, 0x46, 0x02, 0xF0, 0xC6, 0xFE, 0x20, 0x46, 0x01, 0xF0, 0x3A, 0xFF,
+ 0xAB, 0x4D, 0xB4, 0xF8, 0x56, 0x10, 0x28, 0x68, 0xB0, 0xF8, 0xC0, 0x20, 0x62, 0xF3, 0x09, 0x01,
+ 0xA4, 0xF8, 0x56, 0x10, 0x90, 0xF8, 0xBF, 0x00, 0x94, 0xF8, 0x55, 0x10, 0x60, 0xF3, 0x03, 0x01,
+ 0x84, 0xF8, 0x55, 0x10, 0x05, 0x21, 0x20, 0x46, 0x02, 0xF0, 0xD6, 0xFB, 0x28, 0x68, 0x90, 0xF8,
+ 0xA2, 0x09, 0xA0, 0x70, 0x28, 0x68, 0x21, 0x8E, 0xB0, 0xF8, 0xA7, 0x29, 0x62, 0xF3, 0x08, 0x01,
+ 0x21, 0x86, 0x90, 0xF8, 0xA3, 0x09, 0x00, 0xF0, 0x0F, 0x01, 0x20, 0x46, 0x02, 0xF0, 0x3B, 0xFC,
+ 0x28, 0x68, 0x34, 0xF8, 0x1A, 0x1F, 0xB0, 0xF8, 0x8E, 0x29, 0x62, 0xF3, 0x0B, 0x01, 0x24, 0xF8,
+ 0x14, 0x19, 0xB0, 0xF8, 0x8C, 0x29, 0x61, 0x89, 0x62, 0xF3, 0x0B, 0x01, 0x61, 0x81, 0xB0, 0xF8,
+ 0x8A, 0x19, 0xE0, 0x88, 0x61, 0xF3, 0x0B, 0x00, 0xE0, 0x80, 0x14, 0xF8, 0x05, 0x0C, 0x20, 0xF0,
+ 0x0C, 0x00, 0x04, 0xF8, 0x05, 0x0C, 0x29, 0x68, 0x20, 0x7B, 0x91, 0xF8, 0xA4, 0x29, 0x62, 0xF3,
+ 0x01, 0x00, 0x20, 0x73, 0x91, 0xF8, 0xA4, 0x29, 0x92, 0x08, 0x62, 0xF3, 0x87, 0x10, 0x20, 0x73,
+ 0x91, 0xF8, 0x82, 0x09, 0x62, 0x78, 0x60, 0xF3, 0x00, 0x02, 0x62, 0x70, 0x91, 0xF8, 0x85, 0x09,
+ 0xE0, 0x70, 0x80, 0x48, 0x90, 0xF8, 0x18, 0x1C, 0x21, 0x76, 0x90, 0xF8, 0x19, 0x1C, 0xA1, 0x77,
+ 0x90, 0xF8, 0x1A, 0x1C, 0x61, 0x76, 0x90, 0xF8, 0x1B, 0x1C, 0xE1, 0x77, 0x90, 0xF8, 0x1C, 0x1C,
+ 0xA1, 0x76, 0x90, 0xF8, 0x1D, 0x0C, 0x84, 0xF8, 0x20, 0x00, 0x77, 0x48, 0x02, 0xF0, 0x53, 0xFB,
+ 0xA4, 0xF8, 0x7A, 0x00, 0x74, 0x48, 0x20, 0x30, 0x02, 0xF0, 0x4D, 0xFB, 0xA4, 0xF8, 0x7E, 0x00,
+ 0x28, 0x68, 0x90, 0xF8, 0x92, 0x19, 0x0A, 0x09, 0x14, 0xF8, 0x16, 0x1F, 0x62, 0xF3, 0x07, 0x11,
+ 0x04, 0xF8, 0x06, 0x1B, 0x90, 0xF8, 0x97, 0x29, 0x13, 0x09, 0x22, 0x78, 0x63, 0xF3, 0x07, 0x12,
+ 0x22, 0x70, 0x90, 0xF8, 0x93, 0x69, 0x14, 0xF8, 0x05, 0x3C, 0x66, 0xF3, 0x03, 0x03, 0x04, 0xF8,
+ 0x05, 0x3C, 0x90, 0xF8, 0x98, 0x79, 0x66, 0x78, 0x67, 0xF3, 0x03, 0x06, 0x66, 0x70, 0x90, 0xF8,
+ 0x93, 0x79, 0x3F, 0x09, 0x67, 0xF3, 0x07, 0x13, 0x04, 0xF8, 0x05, 0x3C, 0x90, 0xF8, 0x98, 0x39,
+ 0x1B, 0x09, 0x63, 0xF3, 0x07, 0x16, 0x66, 0x70, 0x90, 0xF8, 0x92, 0x39, 0x63, 0xF3, 0x00, 0x01,
+ 0x04, 0xF8, 0x06, 0x1C, 0x90, 0xF8, 0x97, 0x19, 0x61, 0xF3, 0x00, 0x02, 0x04, 0xF8, 0x22, 0x29,
+ 0x90, 0xF8, 0x87, 0x09, 0x01, 0x09, 0x20, 0x46, 0x02, 0xF0, 0xB9, 0xFB, 0x52, 0x48, 0x48, 0x38,
+ 0x02, 0xF0, 0x09, 0xFB, 0xA4, 0xF8, 0x7E, 0x00, 0x4F, 0x48, 0x27, 0x34, 0x28, 0x38, 0x02, 0xF0,
+ 0x02, 0xFB, 0xA4, 0xF8, 0x5B, 0x00, 0x29, 0x68, 0x91, 0xF8, 0x84, 0x09, 0xC2, 0x08, 0x20, 0x78,
+ 0x62, 0xF3, 0x83, 0x00, 0x20, 0x70, 0x91, 0xF8, 0x84, 0x29, 0x52, 0x08, 0x62, 0xF3, 0x00, 0x00,
+ 0x20, 0x70, 0x91, 0xF8, 0x84, 0x29, 0x92, 0x08, 0x62, 0xF3, 0x41, 0x00, 0x04, 0xF8, 0x27, 0x09,
+ 0x91, 0xF8, 0x83, 0x09, 0xC0, 0xF3, 0x81, 0x01, 0x20, 0x46, 0x02, 0xF0, 0x71, 0xFB, 0x28, 0x68,
+ 0x90, 0xF8, 0x83, 0x09, 0x01, 0x09, 0x14, 0xF8, 0x28, 0x0F, 0x61, 0xF3, 0x04, 0x10, 0x20, 0x70,
+ 0x3A, 0x48, 0x02, 0xF0, 0xD8, 0xFA, 0xA4, 0xF8, 0x60, 0x00, 0x38, 0x48, 0x20, 0x30, 0x02, 0xF0,
+ 0xD2, 0xFA, 0xA4, 0xF8, 0x62, 0x00, 0x28, 0x68, 0x90, 0xF8, 0x82, 0x19, 0xCA, 0x08, 0xE1, 0x7A,
+ 0x62, 0xF3, 0x45, 0x11, 0xE1, 0x72, 0x90, 0xF8, 0x83, 0x09, 0x41, 0x08, 0x20, 0x78, 0x61, 0xF3,
+ 0x45, 0x10, 0x04, 0xF8, 0x28, 0x09, 0x20, 0x46, 0x01, 0xF0, 0x18, 0xFE, 0x28, 0x68, 0x90, 0xF8,
+ 0x86, 0x09, 0x81, 0x09, 0x20, 0x46, 0x02, 0xF0, 0x30, 0xFB, 0x14, 0xF8, 0x32, 0x0F, 0x20, 0xF0,
+ 0x0F, 0x00, 0x0A, 0x30, 0x20, 0xF0, 0xF0, 0x00, 0xA0, 0x30, 0x04, 0xF8, 0x1C, 0x09, 0x29, 0x68,
+ 0x91, 0xF8, 0x82, 0x09, 0x82, 0x09, 0x60, 0x7F, 0x62, 0xF3, 0x00, 0x00, 0x60, 0x77, 0x91, 0xF8,
+ 0x82, 0x29, 0x52, 0x09, 0x62, 0xF3, 0x41, 0x00, 0x60, 0x77, 0x91, 0xF8, 0x88, 0x29, 0x62, 0xF3,
+ 0x82, 0x00, 0x60, 0x77, 0x91, 0xF8, 0xA5, 0x09, 0xE2, 0x7D, 0x60, 0xF3, 0x04, 0x02, 0xE2, 0x75,
+ 0x91, 0xF8, 0xA6, 0x09, 0xA1, 0x7D, 0x60, 0xF3, 0x04, 0x01, 0x0F, 0x48, 0xA1, 0x75, 0x08, 0x38,
+ 0x94, 0xF8, 0x49, 0x10, 0x40, 0x78, 0x60, 0xF3, 0x04, 0x01, 0x0F, 0x48, 0x84, 0xF8, 0x49, 0x10,
+ 0xE8, 0x30, 0x02, 0xF0, 0x80, 0xFA, 0xA4, 0xF8, 0x64, 0x00, 0x28, 0x68, 0x21, 0x8B, 0xB0, 0xF8,
+ 0x90, 0x29, 0x62, 0xF3, 0x0B, 0x01, 0x21, 0x83, 0x90, 0xF8, 0x86, 0x19, 0xCA, 0x08, 0x94, 0xF8,
+ 0x22, 0x10, 0x0D, 0xE0, 0x88, 0x07, 0x10, 0x00, 0x00, 0x5D, 0x01, 0x20, 0xD4, 0x83, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0xFC, 0x4C, 0x01, 0x20, 0xCC, 0x58, 0x01, 0x20, 0xB0, 0x5C, 0x01, 0x20,
+ 0x62, 0xF3, 0x02, 0x01, 0x84, 0xF8, 0x22, 0x10, 0x10, 0xF8, 0xE3, 0x2F, 0x21, 0x7D, 0x62, 0xF3,
+ 0x03, 0x01, 0x21, 0x75, 0x10, 0xF8, 0xE3, 0x1B, 0x0A, 0x09, 0x61, 0x7D, 0x62, 0xF3, 0x03, 0x01,
+ 0x61, 0x75, 0x90, 0xF8, 0xC1, 0x17, 0xE2, 0x7C, 0x61, 0xF3, 0x02, 0x02, 0xE2, 0x74, 0x90, 0xF8,
+ 0xBC, 0x17, 0xCA, 0x09, 0xA1, 0x7F, 0x62, 0xF3, 0x00, 0x01, 0xA1, 0x77, 0x94, 0xF8, 0x49, 0x10,
+ 0x41, 0xF0, 0x20, 0x01, 0x84, 0xF8, 0x49, 0x10, 0x90, 0xF8, 0xBE, 0x27, 0xD2, 0x09, 0x62, 0xF3,
+ 0xC7, 0x11, 0x84, 0xF8, 0x49, 0x10, 0xB0, 0xF8, 0xD6, 0x27, 0xB4, 0xF8, 0x4A, 0x10, 0x62, 0xF3,
+ 0x08, 0x01, 0xA4, 0xF8, 0x4A, 0x10, 0x00, 0x21, 0xE1, 0x64, 0x90, 0xF8, 0xC1, 0x17, 0xCA, 0x08,
+ 0x61, 0x7F, 0x62, 0xF3, 0xC3, 0x01, 0x61, 0x77, 0x90, 0xF8, 0xBC, 0x27, 0x12, 0x09, 0x62, 0xF3,
+ 0x04, 0x11, 0x61, 0x77, 0x90, 0xF8, 0xDF, 0x17, 0x4A, 0x09, 0x14, 0xF8, 0x0E, 0x1C, 0x62, 0xF3,
+ 0x06, 0x11, 0x04, 0xF8, 0x0E, 0x1C, 0x90, 0xF8, 0xBD, 0x17, 0x4A, 0x09, 0xA1, 0x78, 0x62, 0xF3,
+ 0xC3, 0x01, 0xA1, 0x70, 0x90, 0xF8, 0xBC, 0x27, 0x52, 0x08, 0x62, 0xF3, 0x04, 0x11, 0xA1, 0x70,
+ 0x90, 0xF8, 0xBE, 0x07, 0x80, 0x09, 0x60, 0xF3, 0x86, 0x11, 0xA1, 0x70, 0xF8, 0xBD, 0x2D, 0xE9,
+ 0xF0, 0x41, 0x04, 0x46, 0x54, 0x4D, 0x01, 0x20, 0x01, 0x29, 0xA8, 0x70, 0x08, 0xD0, 0x00, 0x26,
+ 0x52, 0x4F, 0x02, 0x29, 0x08, 0xD0, 0x03, 0x29, 0x14, 0xD0, 0x04, 0x29, 0x2C, 0xD1, 0x3D, 0xE0,
+ 0x20, 0x46, 0x00, 0xF0, 0x94, 0xF8, 0x27, 0xE0, 0x4D, 0x48, 0xFF, 0xF7, 0x0C, 0xFE, 0x20, 0x46,
+ 0x00, 0xF0, 0x8D, 0xF8, 0xB8, 0x78, 0x94, 0xF8, 0x5F, 0x10, 0x60, 0xF3, 0x04, 0x01, 0x84, 0xF8,
+ 0x5F, 0x10, 0x14, 0xE0, 0x46, 0x48, 0xFF, 0xF7, 0xFE, 0xFD, 0x20, 0x46, 0x00, 0xF0, 0x7F, 0xF8,
+ 0xF8, 0x78, 0x94, 0xF8, 0x5F, 0x10, 0x60, 0xF3, 0x04, 0x01, 0x84, 0xF8, 0x5F, 0x10, 0x94, 0xF8,
+ 0x27, 0x00, 0x20, 0xF0, 0x02, 0x00, 0x40, 0xF0, 0x01, 0x00, 0x84, 0xF8, 0x27, 0x00, 0x00, 0x21,
+ 0x20, 0x46, 0x02, 0xF0, 0x68, 0xFA, 0xAE, 0x70, 0x20, 0x46, 0x01, 0xF0, 0x83, 0xFE, 0x94, 0xF8,
+ 0x27, 0x00, 0x40, 0xF3, 0x00, 0x00, 0x40, 0x1C, 0xE8, 0x70, 0x36, 0x48, 0x00, 0x68, 0x90, 0xF8,
+ 0x83, 0x09, 0xC0, 0xF3, 0x80, 0x10, 0x68, 0x70, 0xBD, 0xE8, 0xF0, 0x81, 0x20, 0x46, 0x00, 0xF0,
+ 0x56, 0xF8, 0x20, 0x46, 0x01, 0xF0, 0x67, 0xFD, 0xE5, 0xE7, 0x7C, 0xB5, 0x01, 0xF0, 0x6E, 0xFC,
+ 0xCD, 0xE9, 0x00, 0x01, 0x08, 0x21, 0x68, 0x46, 0x02, 0xF0, 0xF4, 0xFD, 0x26, 0x49, 0x27, 0x4D,
+ 0x08, 0x71, 0x00, 0x20, 0x28, 0x60, 0x28, 0x71, 0x01, 0x24, 0xE0, 0xB2, 0xFF, 0xF7, 0x56, 0xFD,
+ 0x28, 0x55, 0x64, 0x1C, 0x05, 0x2C, 0xF8, 0xDB, 0x21, 0x48, 0xFF, 0xF7, 0xB4, 0xFD, 0x7C, 0xBD,
+ 0x30, 0xB5, 0x01, 0x7A, 0xA9, 0xB0, 0x79, 0xB3, 0x68, 0x46, 0xFF, 0xF7, 0x88, 0xFF, 0x9D, 0xF8,
+ 0x12, 0x00, 0x00, 0xF0, 0x03, 0x00, 0x01, 0xF0, 0xB1, 0xFC, 0x1A, 0x4C, 0x20, 0x68, 0x00, 0xF2,
+ 0x5E, 0x63, 0x00, 0xF6, 0x6E, 0x05, 0x90, 0xF8, 0x58, 0x16, 0x90, 0xF8, 0x59, 0x06, 0x03, 0xEB,
+ 0x40, 0x00, 0x01, 0xF0, 0x74, 0xFC, 0x20, 0x68, 0x90, 0xF8, 0x68, 0x18, 0x90, 0xF8, 0x69, 0x08,
+ 0x05, 0xEB, 0x40, 0x00, 0x01, 0xF0, 0x82, 0xFC, 0x0B, 0x48, 0x41, 0x78, 0x01, 0xB1, 0x02, 0x21,
+ 0x01, 0x70, 0xC9, 0xB2, 0x68, 0x46, 0x02, 0xF0, 0x80, 0xFC, 0x0B, 0x49, 0x68, 0x46, 0x02, 0xF0,
+ 0xB1, 0xFC, 0x00, 0x20, 0x29, 0xB0, 0x30, 0xBD, 0xFF, 0xE7, 0x01, 0x20, 0xFA, 0xE7, 0xA4, 0x22,
+ 0x03, 0x49, 0x04, 0xF0, 0x67, 0xB8, 0x00, 0x00, 0x88, 0x07, 0x10, 0x00, 0xF8, 0x5C, 0x01, 0x20,
+ 0x48, 0x7F, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0xBD, 0x20, 0x01, 0x00, 0x10, 0xB5, 0x04, 0x00,
+ 0x03, 0xD0, 0xBD, 0xE8, 0x10, 0x40, 0x01, 0xF0, 0x96, 0xBB, 0x26, 0x48, 0x26, 0x49, 0x00, 0x78,
+ 0x42, 0x00, 0x01, 0xEB, 0x40, 0x01, 0x24, 0x48, 0x98, 0x30, 0x04, 0xF0, 0x4B, 0xF8, 0x22, 0x48,
+ 0x22, 0x49, 0x98, 0x30, 0x48, 0x62, 0x20, 0x46, 0xEB, 0xE7, 0x1C, 0xB5, 0x01, 0xF0, 0xF6, 0xFB,
+ 0xCD, 0xE9, 0x00, 0x01, 0x08, 0x21, 0x68, 0x46, 0x02, 0xF0, 0x7C, 0xFD, 0x19, 0x49, 0x1C, 0x4C,
+ 0x08, 0x70, 0x20, 0x46, 0x00, 0xF0, 0x39, 0xFB, 0x17, 0x48, 0x02, 0xF0, 0x24, 0xF9, 0xA4, 0xF8,
+ 0x76, 0x00, 0x18, 0x48, 0x01, 0x68, 0xA0, 0x7C, 0x91, 0xF8, 0x50, 0x21, 0x62, 0xF3, 0x01, 0x00,
+ 0xA0, 0x74, 0x91, 0xF8, 0x50, 0x11, 0x89, 0x08, 0x61, 0xF3, 0x87, 0x10, 0xA0, 0x74, 0x1C, 0xBD,
+ 0x10, 0xB5, 0x0F, 0x4C, 0xC0, 0x79, 0x98, 0xB1, 0x20, 0x46, 0x01, 0xF0, 0xCB, 0xFD, 0xA0, 0x7C,
+ 0x00, 0xF0, 0x03, 0x00, 0x01, 0xF0, 0x32, 0xFC, 0x00, 0xF0, 0x7B, 0xFB, 0x04, 0x21, 0x20, 0x46,
+ 0x02, 0xF0, 0x1B, 0xFC, 0x08, 0x49, 0x20, 0x46, 0x02, 0xF0, 0x4C, 0xFC, 0x00, 0x20, 0x10, 0xBD,
+ 0x01, 0x20, 0x10, 0xBD, 0x8D, 0x07, 0x10, 0x00, 0xE4, 0x5D, 0x01, 0x20, 0xD4, 0x83, 0x10, 0x00,
+ 0xEC, 0x7F, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00, 0x0D, 0x27, 0x01, 0x00, 0xFE, 0x49, 0xC8, 0x68,
+ 0x40, 0x1C, 0xC8, 0x60, 0x02, 0x28, 0x03, 0xD2, 0xFC, 0x49, 0x51, 0xF8, 0x20, 0x00, 0x00, 0x47,
+ 0x00, 0x20, 0x01, 0xF0, 0x30, 0xBB, 0x00, 0x28, 0x01, 0xD1, 0x02, 0xF0, 0x1C, 0xBC, 0x70, 0x47,
+ 0xF5, 0x49, 0x10, 0xB5, 0x0A, 0x79, 0x01, 0x20, 0x52, 0x1C, 0xD2, 0xB2, 0x0A, 0x71, 0x0B, 0x7A,
+ 0x9A, 0x42, 0x0A, 0xD8, 0x48, 0x7A, 0x48, 0xB1, 0x02, 0xF0, 0x5E, 0xFB, 0x03, 0x20, 0xED, 0xF7,
+ 0xCF, 0xFD, 0xEF, 0x48, 0x02, 0xF0, 0x4C, 0xFB, 0x00, 0x20, 0x10, 0xBD, 0x02, 0xF0, 0x03, 0xFC,
+ 0xFA, 0xE7, 0x70, 0xB5, 0x18, 0xB1, 0xBD, 0xE8, 0x70, 0x40, 0x01, 0xF0, 0x0C, 0xBB, 0xE6, 0x4C,
+ 0x20, 0x7A, 0x08, 0xB1, 0xE7, 0x4D, 0x01, 0xE0, 0xE6, 0x4D, 0x76, 0x3D, 0xA0, 0x78, 0xE6, 0x49,
+ 0x82, 0x00, 0x01, 0xF5, 0xAE, 0x70, 0x04, 0xF0, 0x02, 0xF8, 0x60, 0x79, 0x28, 0xB1, 0xA2, 0x78,
+ 0xE1, 0x49, 0x28, 0x46, 0x01, 0xF0, 0x44, 0xFB, 0x0C, 0xE0, 0xA1, 0x78, 0x60, 0x78, 0x01, 0xF0,
+ 0x0D, 0xFC, 0xA1, 0x78, 0xC0, 0xB2, 0x4A, 0x00, 0xDB, 0x49, 0x01, 0xEB, 0x40, 0x01, 0x28, 0x46,
+ 0x03, 0xF0, 0xA8, 0xFF, 0xD7, 0x48, 0x23, 0x79, 0xA2, 0x78, 0x29, 0x46, 0x76, 0x38, 0x01, 0xF0,
+ 0x40, 0xFB, 0xFF, 0xF7, 0xB5, 0xFF, 0x00, 0x28, 0x15, 0xD0, 0xA0, 0x79, 0xD3, 0x4D, 0x01, 0x28,
+ 0x0B, 0xD0, 0xD0, 0x48, 0x76, 0x38, 0xA8, 0x61, 0xE0, 0x79, 0x85, 0xF8, 0x30, 0x00, 0x20, 0x79,
+ 0x85, 0xF8, 0x20, 0x00, 0xBD, 0xE8, 0x70, 0x40, 0x90, 0xE7, 0xCA, 0x48, 0xA1, 0x78, 0x76, 0x38,
+ 0x01, 0xF0, 0xE2, 0xFB, 0xEF, 0xE7, 0x70, 0xBD, 0x70, 0xB5, 0x18, 0xB1, 0xBD, 0xE8, 0x70, 0x40,
+ 0x01, 0xF0, 0xC1, 0xBA, 0xC0, 0x4C, 0x20, 0x7A, 0x08, 0xB1, 0xC2, 0x4D, 0x01, 0xE0, 0xC1, 0x4D,
+ 0x4C, 0x3D, 0xE0, 0x78, 0xC0, 0x49, 0x82, 0x00, 0x01, 0xF5, 0xD8, 0x70, 0x03, 0xF0, 0xB7, 0xFF,
+ 0x60, 0x79, 0x28, 0xB1, 0xE2, 0x78, 0xBC, 0x49, 0x28, 0x46, 0x01, 0xF0, 0xF9, 0xFA, 0x0C, 0xE0,
+ 0xE1, 0x78, 0x60, 0x78, 0x01, 0xF0, 0xC8, 0xFB, 0xE1, 0x78, 0xC0, 0xB2, 0x4A, 0x00, 0xB6, 0x49,
+ 0x01, 0xEB, 0x40, 0x01, 0x28, 0x46, 0x03, 0xF0, 0x5D, 0xFF, 0xB2, 0x48, 0x23, 0x79, 0xE2, 0x78,
+ 0x29, 0x46, 0x4C, 0x38, 0x01, 0xF0, 0xF5, 0xFA, 0xFF, 0xF7, 0x6A, 0xFF, 0x00, 0x28, 0x15, 0xD0,
+ 0xA0, 0x79, 0xAE, 0x4D, 0x01, 0x28, 0x0B, 0xD0, 0xAA, 0x48, 0x4C, 0x38, 0xE8, 0x61, 0xE0, 0x79,
+ 0x85, 0xF8, 0x30, 0x00, 0x20, 0x79, 0x85, 0xF8, 0x21, 0x00, 0xBD, 0xE8, 0x70, 0x40, 0x45, 0xE7,
+ 0xA4, 0x48, 0xE1, 0x78, 0x4C, 0x38, 0x01, 0xF0, 0x98, 0xFB, 0xEF, 0xE7, 0x70, 0xBD, 0x2D, 0xE9,
+ 0xF1, 0x4F, 0xDF, 0xF8, 0x8C, 0xB2, 0x01, 0x27, 0xB8, 0x46, 0xDB, 0xF8, 0x00, 0x00, 0x00, 0xF2,
+ 0x5E, 0x64, 0x90, 0xF8, 0x55, 0x11, 0x90, 0xF8, 0x4E, 0x51, 0x01, 0xF0, 0x03, 0x0A, 0x90, 0xF8,
+ 0x52, 0x11, 0x01, 0xF0, 0x1F, 0x06, 0x90, 0xF8, 0x4F, 0x11, 0x01, 0xF0, 0x0F, 0x09, 0x90, 0xF8,
+ 0x32, 0x11, 0x89, 0x06, 0x09, 0xD5, 0x90, 0xF8, 0x56, 0x16, 0x90, 0xF8, 0x57, 0x06, 0x0A, 0x22,
+ 0x04, 0xEB, 0x40, 0x00, 0x01, 0xF0, 0xE0, 0xFA, 0x07, 0x46, 0xDB, 0xF8, 0x00, 0x00, 0x90, 0xF8,
+ 0x32, 0x11, 0x49, 0x06, 0x09, 0xD5, 0x90, 0xF8, 0x66, 0x18, 0x90, 0xF8, 0x67, 0x08, 0x0A, 0x22,
+ 0x04, 0xEB, 0x40, 0x00, 0x01, 0xF0, 0xD0, 0xFA, 0x80, 0x46, 0x4F, 0xF4, 0x80, 0x70, 0x85, 0x42,
+ 0x00, 0xD9, 0x05, 0x46, 0x1F, 0x2E, 0x00, 0xD9, 0x1F, 0x26, 0x76, 0x1C, 0x75, 0x43, 0x09, 0xF1,
+ 0x01, 0x00, 0x45, 0x43, 0xC5, 0xEB, 0x45, 0x20, 0x78, 0x43, 0x00, 0xFB, 0x08, 0xF0, 0x0A, 0xF1,
+ 0x01, 0x01, 0x48, 0x43, 0xDB, 0xF8, 0x00, 0x20, 0xC0, 0x13, 0xB0, 0xFA, 0x80, 0xF0, 0x92, 0xF8,
+ 0x50, 0x1A, 0xC0, 0xF1, 0x20, 0x00, 0xC9, 0x06, 0x11, 0xD5, 0x00, 0x9B, 0x00, 0x21, 0x01, 0x2B,
+ 0x06, 0xD1, 0x92, 0xF8, 0x55, 0x1A, 0x0A, 0x07, 0x01, 0xF0, 0x07, 0x01, 0x00, 0xD5, 0x49, 0x42,
+ 0x40, 0x18, 0x01, 0xD5, 0x00, 0x20, 0x02, 0xE0, 0x1F, 0x28, 0x00, 0xDD, 0x1F, 0x20, 0xC0, 0xB2,
+ 0xBD, 0xE8, 0xF8, 0x8F, 0xF8, 0xB5, 0x04, 0x46, 0x01, 0xF0, 0xB8, 0xFF, 0x6A, 0x48, 0x01, 0xF0,
+ 0xBA, 0xFF, 0xA4, 0xF8, 0x76, 0x00, 0x01, 0xF0, 0x79, 0xFA, 0x63, 0x4D, 0x0B, 0x46, 0x6A, 0x78,
+ 0x00, 0x92, 0x02, 0x46, 0x20, 0x46, 0x02, 0xF0, 0x78, 0xFA, 0x01, 0xF0, 0x74, 0xFA, 0x02, 0x46,
+ 0x0B, 0x46, 0x20, 0x46, 0x02, 0xF0, 0xA7, 0xFA, 0x20, 0x46, 0x01, 0xF0, 0x1B, 0xFB, 0x60, 0x4E,
+ 0xB4, 0xF8, 0x56, 0x10, 0x30, 0x68, 0xB0, 0xF8, 0xC0, 0x20, 0x62, 0xF3, 0x09, 0x01, 0xA4, 0xF8,
+ 0x56, 0x10, 0x90, 0xF8, 0xBF, 0x00, 0x94, 0xF8, 0x55, 0x10, 0x60, 0xF3, 0x03, 0x01, 0x84, 0xF8,
+ 0x55, 0x10, 0x05, 0x21, 0x20, 0x46, 0x01, 0xF0, 0xB7, 0xFF, 0x30, 0x68, 0x90, 0xF8, 0x4E, 0x01,
+ 0xA0, 0x70, 0x30, 0x68, 0x21, 0x8E, 0xB0, 0xF8, 0x53, 0x21, 0x62, 0xF3, 0x08, 0x01, 0x21, 0x86,
+ 0x90, 0xF8, 0x4F, 0x01, 0x00, 0xF0, 0x0F, 0x01, 0x20, 0x46, 0x02, 0xF0, 0x1C, 0xF8, 0x30, 0x68,
+ 0x34, 0xF8, 0x1A, 0x1F, 0xB0, 0xF8, 0x3E, 0x21, 0x62, 0xF3, 0x0B, 0x01, 0x24, 0xF8, 0x12, 0x19,
+ 0xB0, 0xF8, 0x3C, 0x21, 0x21, 0x89, 0x62, 0xF3, 0x0B, 0x01, 0x21, 0x81, 0xB0, 0xF8, 0x3A, 0x21,
+ 0xA1, 0x88, 0x62, 0xF3, 0x0B, 0x01, 0xA1, 0x80, 0x90, 0xF8, 0x50, 0x21, 0xA1, 0x7A, 0x62, 0xF3,
+ 0x01, 0x01, 0xA1, 0x72, 0x90, 0xF8, 0x50, 0x21, 0x92, 0x08, 0x62, 0xF3, 0x87, 0x11, 0xA1, 0x72,
+ 0x90, 0xF8, 0x32, 0x11, 0x14, 0xF8, 0x01, 0x2C, 0x61, 0xF3, 0x00, 0x02, 0x04, 0xF8, 0x01, 0x2C,
+ 0x90, 0xF8, 0x35, 0x01, 0x60, 0x70, 0x37, 0x48, 0x90, 0xF8, 0x80, 0x1B, 0xA1, 0x75, 0x90, 0xF8,
+ 0x81, 0x1B, 0x21, 0x77, 0x90, 0xF8, 0x82, 0x1B, 0xE1, 0x75, 0x90, 0xF8, 0x83, 0x1B, 0x61, 0x77,
+ 0x90, 0xF8, 0x84, 0x1B, 0x21, 0x76, 0x90, 0xF8, 0x85, 0x0B, 0xA0, 0x77, 0x2E, 0x48, 0x01, 0xF0,
+ 0x3A, 0xFF, 0xA4, 0xF8, 0x78, 0x00, 0x2C, 0x48, 0x20, 0x30, 0x01, 0xF0, 0x34, 0xFF, 0xA4, 0xF8,
+ 0x7C, 0x00, 0x30, 0x68, 0x14, 0xF8, 0x14, 0x2F, 0x90, 0xF8, 0x42, 0x11, 0x09, 0x09, 0x61, 0xF3,
+ 0x07, 0x12, 0x04, 0xF8, 0x06, 0x2B, 0x90, 0xF8, 0x47, 0x11, 0x0B, 0x09, 0x21, 0x78, 0x63, 0xF3,
+ 0x07, 0x11, 0x21, 0x70, 0x90, 0xF8, 0x43, 0x31, 0x14, 0xF8, 0x05, 0x7C, 0x63, 0xF3, 0x03, 0x07,
+ 0x04, 0xF8, 0x05, 0x7C, 0x90, 0xF8, 0x48, 0xC1, 0x63, 0x78, 0x6C, 0xF3, 0x03, 0x03, 0x63, 0x70,
+ 0x90, 0xF8, 0x43, 0xC1, 0x4F, 0xEA, 0x1C, 0x1C, 0x6C, 0xF3, 0x07, 0x17, 0x04, 0xF8, 0x05, 0x7C,
+ 0x90, 0xF8, 0x48, 0x71, 0x3F, 0x09, 0x67, 0xF3, 0x07, 0x13, 0x63, 0x70, 0x90, 0xF8, 0x42, 0x31,
+ 0x63, 0xF3, 0x00, 0x02, 0x04, 0xF8, 0x06, 0x2C, 0x90, 0xF8, 0x47, 0x21, 0x62, 0xF3, 0x00, 0x01,
+ 0x04, 0xF8, 0x22, 0x19, 0x90, 0xF8, 0x37, 0x01, 0x01, 0x09, 0x20, 0x46, 0x01, 0xF0, 0x9F, 0xFF,
+ 0x09, 0x48, 0x48, 0x38, 0x12, 0xE0, 0x00, 0x00, 0x90, 0x07, 0x10, 0x00, 0x44, 0x6F, 0x01, 0x00,
+ 0xE7, 0x27, 0x01, 0x00, 0xDE, 0x5F, 0x01, 0x20, 0xD0, 0x5E, 0x01, 0x20, 0xD4, 0x83, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0xFC, 0x4C, 0x01, 0x20, 0x34, 0x58, 0x01, 0x20, 0x01, 0xF0, 0xDB, 0xFE,
+ 0xA4, 0xF8, 0x7E, 0x00, 0xF7, 0x48, 0x27, 0x34, 0x01, 0xF0, 0xD5, 0xFE, 0xA4, 0xF8, 0x5B, 0x00,
+ 0x31, 0x68, 0x91, 0xF8, 0x34, 0x01, 0xC2, 0x08, 0x20, 0x78, 0x62, 0xF3, 0x83, 0x00, 0x20, 0x70,
+ 0x91, 0xF8, 0x34, 0x21, 0x52, 0x08, 0x62, 0xF3, 0x00, 0x00, 0x20, 0x70, 0x91, 0xF8, 0x34, 0x21,
+ 0x92, 0x08, 0x62, 0xF3, 0x41, 0x00, 0x04, 0xF8, 0x27, 0x09, 0x91, 0xF8, 0x33, 0x01, 0xC0, 0xF3,
+ 0x81, 0x01, 0x20, 0x46, 0x01, 0xF0, 0x44, 0xFF, 0x30, 0x68, 0x90, 0xF8, 0x33, 0x01, 0x01, 0x09,
+ 0x14, 0xF8, 0x28, 0x0F, 0x61, 0xF3, 0x04, 0x10, 0x20, 0x70, 0xE3, 0x48, 0x01, 0xF0, 0xAB, 0xFE,
+ 0xA4, 0xF8, 0x60, 0x00, 0xE0, 0x48, 0x20, 0x30, 0x01, 0xF0, 0xA5, 0xFE, 0xA4, 0xF8, 0x62, 0x00,
+ 0x30, 0x68, 0x90, 0xF8, 0x32, 0x11, 0xCA, 0x08, 0xE1, 0x7A, 0x62, 0xF3, 0x45, 0x11, 0xE1, 0x72,
+ 0x90, 0xF8, 0x33, 0x01, 0x41, 0x08, 0x20, 0x78, 0x61, 0xF3, 0x45, 0x10, 0x04, 0xF8, 0x28, 0x09,
+ 0x20, 0x46, 0x01, 0xF0, 0xD9, 0xF9, 0x30, 0x68, 0x90, 0xF8, 0x36, 0x01, 0x81, 0x09, 0x20, 0x46,
+ 0x01, 0xF0, 0x03, 0xFF, 0x14, 0xF8, 0x32, 0x0F, 0x20, 0xF0, 0x0F, 0x00, 0x0A, 0x30, 0x20, 0xF0,
+ 0xF0, 0x00, 0xA0, 0x30, 0x04, 0xF8, 0x1C, 0x09, 0x31, 0x68, 0x91, 0xF8, 0x32, 0x01, 0x82, 0x09,
+ 0x60, 0x7F, 0x62, 0xF3, 0x00, 0x00, 0x60, 0x77, 0x91, 0xF8, 0x32, 0x21, 0x52, 0x09, 0x62, 0xF3,
+ 0x41, 0x00, 0x60, 0x77, 0x91, 0xF8, 0x38, 0x21, 0x62, 0xF3, 0x82, 0x00, 0x60, 0x77, 0x91, 0xF8,
+ 0x51, 0x01, 0xE2, 0x7D, 0x60, 0xF3, 0x04, 0x02, 0xE2, 0x75, 0x91, 0xF8, 0x52, 0x01, 0xA1, 0x7D,
+ 0x60, 0xF3, 0x04, 0x01, 0xBD, 0x48, 0xA1, 0x75, 0x94, 0xF8, 0x49, 0x10, 0x40, 0x78, 0x60, 0xF3,
+ 0x04, 0x01, 0x84, 0xF8, 0x49, 0x10, 0xBA, 0x48, 0x01, 0xF0, 0x55, 0xFE, 0xA4, 0xF8, 0x64, 0x00,
+ 0x30, 0x68, 0x21, 0x8B, 0xB0, 0xF8, 0x40, 0x21, 0x62, 0xF3, 0x0B, 0x01, 0x21, 0x83, 0x90, 0xF8,
+ 0x36, 0x11, 0xCA, 0x08, 0x94, 0xF8, 0x22, 0x10, 0x62, 0xF3, 0x02, 0x01, 0x84, 0xF8, 0x22, 0x10,
+ 0x10, 0xF8, 0xE3, 0x1F, 0x22, 0x7D, 0x61, 0xF3, 0x03, 0x02, 0x22, 0x75, 0x10, 0xF8, 0x4F, 0x1B,
+ 0x0A, 0x09, 0x61, 0x7D, 0x62, 0xF3, 0x03, 0x01, 0x61, 0x75, 0x42, 0x79, 0xE1, 0x7C, 0x62, 0xF3,
+ 0x02, 0x01, 0xE1, 0x74, 0x01, 0x78, 0xCA, 0x09, 0xA1, 0x7F, 0x62, 0xF3, 0x00, 0x01, 0xA1, 0x77,
+ 0x94, 0xF8, 0x49, 0x10, 0x41, 0xF0, 0x20, 0x01, 0x84, 0xF8, 0x49, 0x10, 0x82, 0x78, 0xD2, 0x09,
+ 0x62, 0xF3, 0xC7, 0x11, 0x84, 0xF8, 0x49, 0x10, 0x42, 0x8B, 0xB4, 0xF8, 0x4A, 0x10, 0x62, 0xF3,
+ 0x08, 0x01, 0xA4, 0xF8, 0x4A, 0x10, 0x00, 0x21, 0xE1, 0x64, 0x41, 0x79, 0xCA, 0x08, 0x61, 0x7F,
+ 0x62, 0xF3, 0xC3, 0x01, 0x61, 0x77, 0x02, 0x78, 0x12, 0x09, 0x62, 0xF3, 0x04, 0x11, 0x61, 0x77,
+ 0xC1, 0x7F, 0x4A, 0x09, 0x14, 0xF8, 0x0E, 0x1C, 0x62, 0xF3, 0x06, 0x11, 0x04, 0xF8, 0x0E, 0x1C,
+ 0x41, 0x78, 0x4A, 0x09, 0xA1, 0x78, 0x62, 0xF3, 0xC3, 0x01, 0xA1, 0x70, 0x02, 0x78, 0x52, 0x08,
+ 0x62, 0xF3, 0x04, 0x11, 0xA1, 0x70, 0x82, 0x78, 0x92, 0x09, 0x62, 0xF3, 0x86, 0x11, 0xA1, 0x70,
+ 0x40, 0x78, 0xC0, 0xF3, 0x80, 0x10, 0x68, 0x71, 0xF8, 0xBD, 0xA4, 0x22, 0x85, 0x49, 0x03, 0xF0,
+ 0xF9, 0xBC, 0x2D, 0xE9, 0xF0, 0x41, 0x84, 0x4D, 0x00, 0x26, 0x01, 0x27, 0x2E, 0x71, 0xAF, 0x71,
+ 0x2E, 0x72, 0x0C, 0x46, 0x6E, 0x72, 0x01, 0x28, 0x07, 0xD0, 0x7C, 0x4F, 0x02, 0x28, 0x2C, 0xD0,
+ 0x03, 0x28, 0x14, 0xD0, 0x04, 0x28, 0x3A, 0xD1, 0x44, 0xE0, 0x08, 0x46, 0xFF, 0xF7, 0xE5, 0xFF,
+ 0x7A, 0x48, 0x01, 0x68, 0x91, 0xF8, 0x38, 0x01, 0x42, 0x06, 0x30, 0xD5, 0x91, 0xF8, 0x55, 0x11,
+ 0x00, 0x06, 0x01, 0xF0, 0x03, 0x01, 0x29, 0x72, 0x29, 0xD5, 0x6F, 0x72, 0x27, 0xE0, 0x71, 0x48,
+ 0xFF, 0xF7, 0x00, 0xFE, 0x20, 0x46, 0xFF, 0xF7, 0xD0, 0xFF, 0xF8, 0x78, 0x94, 0xF8, 0x5F, 0x10,
+ 0x60, 0xF3, 0x04, 0x01, 0x84, 0xF8, 0x5F, 0x10, 0x94, 0xF8, 0x27, 0x00, 0x20, 0xF0, 0x02, 0x00,
+ 0x40, 0xF0, 0x01, 0x00, 0x84, 0xF8, 0x27, 0x00, 0x0C, 0xE0, 0x66, 0x48, 0xFF, 0xF7, 0xEA, 0xFD,
+ 0x20, 0x46, 0xFF, 0xF7, 0xBA, 0xFF, 0xB8, 0x78, 0x94, 0xF8, 0x5F, 0x10, 0x60, 0xF3, 0x04, 0x01,
+ 0x84, 0xF8, 0x5F, 0x10, 0x00, 0x21, 0x20, 0x46, 0x01, 0xF0, 0x3D, 0xFE, 0xAE, 0x71, 0x20, 0x46,
+ 0x01, 0xF0, 0x58, 0xFA, 0x94, 0xF8, 0x27, 0x00, 0x40, 0xF3, 0x00, 0x00, 0x40, 0x1C, 0xE8, 0x71,
+ 0xBD, 0xE8, 0xF0, 0x81, 0x08, 0x46, 0xFF, 0xF7, 0xA0, 0xFF, 0x20, 0x46, 0x01, 0xF0, 0x43, 0xF9,
+ 0xEC, 0xE7, 0x70, 0xB5, 0x55, 0x4D, 0x28, 0x68, 0x00, 0xF2, 0x5E, 0x62, 0x00, 0xF6, 0x6E, 0x04,
+ 0x90, 0xF8, 0x56, 0x16, 0x90, 0xF8, 0x57, 0x06, 0x02, 0xEB, 0x40, 0x00, 0x01, 0xF0, 0x77, 0xF8,
+ 0x28, 0x68, 0x90, 0xF8, 0x66, 0x18, 0x90, 0xF8, 0x67, 0x08, 0x04, 0xEB, 0x40, 0x00, 0xBD, 0xE8,
+ 0x70, 0x40, 0x01, 0xF0, 0x83, 0xB8, 0x70, 0xB5, 0x47, 0x4D, 0x46, 0x4C, 0x68, 0x79, 0xA4, 0x34,
+ 0x00, 0xB1, 0x02, 0x20, 0x68, 0x70, 0x21, 0x46, 0x28, 0x78, 0xFF, 0xF7, 0x7A, 0xFF, 0x43, 0x48,
+ 0x01, 0x68, 0xA0, 0x7C, 0x91, 0xF8, 0x50, 0x21, 0x62, 0xF3, 0x01, 0x00, 0xA0, 0x74, 0x91, 0xF8,
+ 0x50, 0x11, 0x89, 0x08, 0x61, 0xF3, 0x87, 0x10, 0xA0, 0x74, 0x69, 0x78, 0x20, 0x46, 0x02, 0xF0,
+ 0x6C, 0xF8, 0xA0, 0x7C, 0x00, 0xF0, 0x03, 0x00, 0x01, 0xF0, 0x78, 0xF8, 0xFF, 0xF7, 0xC1, 0xFF,
+ 0x20, 0x46, 0xBD, 0xE8, 0x70, 0x40, 0x36, 0x49, 0x02, 0xF0, 0x94, 0xB8, 0x70, 0xB5, 0x32, 0x4D,
+ 0x30, 0x4C, 0x68, 0x79, 0xA4, 0x34, 0x08, 0xB1, 0x03, 0x20, 0x00, 0xE0, 0x01, 0x20, 0x68, 0x70,
+ 0x21, 0x46, 0x28, 0x78, 0xFF, 0xF7, 0x4D, 0xFF, 0x2C, 0x48, 0x01, 0x68, 0x91, 0xF8, 0x50, 0x01,
+ 0x02, 0x09, 0xA0, 0x7C, 0x62, 0xF3, 0x01, 0x00, 0xA0, 0x74, 0x91, 0xF8, 0x50, 0x11, 0x89, 0x09,
+ 0x61, 0xF3, 0x87, 0x10, 0xA0, 0x74, 0x69, 0x78, 0x20, 0x46, 0x02, 0xF0, 0x3E, 0xF8, 0xA0, 0x7C,
+ 0x00, 0xF0, 0x03, 0x00, 0x01, 0xF0, 0x4A, 0xF8, 0xFF, 0xF7, 0x93, 0xFF, 0x20, 0x46, 0xBD, 0xE8,
+ 0x70, 0x40, 0x20, 0x49, 0x02, 0xF0, 0x66, 0xB8, 0x7F, 0xB5, 0x00, 0xF0, 0xD7, 0xFF, 0xCD, 0xE9,
+ 0x00, 0x01, 0x00, 0xF0, 0xD8, 0xFF, 0xCD, 0xE9, 0x02, 0x01, 0x08, 0x21, 0x68, 0x46, 0x02, 0xF0,
+ 0x59, 0xF9, 0x15, 0x4C, 0x08, 0x21, 0xA0, 0x70, 0x02, 0xA8, 0x02, 0xF0, 0x53, 0xF9, 0xE0, 0x70,
+ 0x0E, 0x4D, 0x01, 0x24, 0xE0, 0xB2, 0xFF, 0xF7, 0xD2, 0xFC, 0x28, 0x55, 0x64, 0x1C, 0x05, 0x2C,
+ 0xF8, 0xDB, 0x0C, 0x48, 0xFF, 0xF7, 0x36, 0xFD, 0x7F, 0xBD, 0x10, 0xB5, 0x80, 0x79, 0xD8, 0xB1,
+ 0x09, 0x49, 0x08, 0x70, 0x4F, 0xF0, 0xFF, 0x30, 0xC8, 0x60, 0xFF, 0xF7, 0xFF, 0xFB, 0x00, 0x20,
+ 0x10, 0xBD, 0x00, 0x00, 0x0C, 0x58, 0x01, 0x20, 0xB0, 0x5C, 0x01, 0x20, 0xC8, 0x5E, 0x01, 0x20,
+ 0xB4, 0x59, 0x01, 0x20, 0x90, 0x80, 0x10, 0x00, 0x90, 0x07, 0x10, 0x00, 0x20, 0x07, 0x10, 0x00,
+ 0x23, 0x28, 0x01, 0x00, 0xB9, 0x28, 0x01, 0x00, 0x01, 0x20, 0xE9, 0xE7, 0x10, 0xB9, 0x77, 0x4A,
+ 0x75, 0x49, 0x11, 0x61, 0x00, 0xF0, 0x1F, 0xBF, 0x70, 0xB5, 0x75, 0x4D, 0x04, 0x46, 0x28, 0x68,
+ 0x90, 0xF8, 0xF1, 0x01, 0xC0, 0xF3, 0x01, 0x12, 0xC0, 0xF3, 0x81, 0x01, 0x00, 0xF0, 0x03, 0x00,
+ 0xFE, 0xF7, 0xF9, 0xF8, 0x40, 0x1C, 0xC6, 0xB2, 0x20, 0x46, 0x00, 0xF0, 0xAA, 0xFC, 0x6A, 0x48,
+ 0x01, 0xF0, 0xB9, 0xFC, 0xA4, 0xF8, 0x76, 0x00, 0x00, 0x22, 0x31, 0x46, 0x20, 0x46, 0x01, 0xF0,
+ 0xB7, 0xFE, 0xA0, 0x79, 0x40, 0xF0, 0x02, 0x00, 0xA0, 0x71, 0x94, 0xF8, 0x6F, 0x00, 0x66, 0xF3,
+ 0x02, 0x00, 0x84, 0xF8, 0x6F, 0x00, 0x94, 0xF8, 0x6E, 0x00, 0x20, 0xF0, 0x0F, 0x00, 0x84, 0xF8,
+ 0x6E, 0x00, 0x29, 0x68, 0xA0, 0x7C, 0x91, 0xF8, 0x08, 0x21, 0x62, 0xF3, 0x01, 0x00, 0xA0, 0x74,
+ 0x91, 0xF8, 0x08, 0x21, 0x92, 0x08, 0x62, 0xF3, 0x87, 0x10, 0xA0, 0x74, 0xA0, 0x89, 0x6F, 0xF3,
+ 0x0B, 0x00, 0xA0, 0x81, 0x91, 0xF8, 0x07, 0x01, 0xA0, 0x70, 0x94, 0xF8, 0x5E, 0x00, 0x00, 0x21,
+ 0x20, 0xF0, 0x01, 0x00, 0x84, 0xF8, 0x5E, 0x00, 0x20, 0x46, 0x01, 0xF0, 0x24, 0xFD, 0x28, 0x68,
+ 0x90, 0xF8, 0xF2, 0x11, 0x84, 0xF8, 0x66, 0x10, 0x90, 0xF8, 0xF3, 0x11, 0x84, 0xF8, 0x68, 0x10,
+ 0x90, 0xF8, 0xF4, 0x11, 0x84, 0xF8, 0x6A, 0x10, 0x90, 0xF8, 0xF5, 0x01, 0x84, 0xF8, 0x6C, 0x00,
+ 0x45, 0x48, 0x94, 0xF8, 0x5F, 0x10, 0x00, 0x1F, 0x40, 0x78, 0x60, 0xF3, 0x04, 0x01, 0x84, 0xF8,
+ 0x5F, 0x10, 0x20, 0x46, 0xBD, 0xE8, 0x70, 0x40, 0x00, 0xF0, 0xA2, 0xBF, 0x10, 0xB5, 0x3E, 0x4C,
+ 0x00, 0x20, 0x24, 0x1F, 0x20, 0x80, 0x01, 0x20, 0x00, 0xF0, 0x3B, 0xF8, 0x60, 0x70, 0x3D, 0x48,
+ 0xFF, 0xF7, 0x8A, 0xFF, 0x3B, 0x48, 0x3C, 0x49, 0xB0, 0xF8, 0x4E, 0x00, 0x08, 0x80, 0x10, 0xBD,
+ 0x2D, 0xE9, 0xF0, 0x41, 0x37, 0x4E, 0x40, 0x79, 0x48, 0xB3, 0x37, 0x49, 0x30, 0x46, 0x09, 0x88,
+ 0xA6, 0xF8, 0x4E, 0x10, 0x01, 0xF0, 0x06, 0xF9, 0xB0, 0x7C, 0x00, 0xF0, 0x03, 0x00, 0x00, 0xF0,
+ 0x6D, 0xFF, 0x2F, 0x4F, 0x3C, 0x68, 0x04, 0xF2, 0x5E, 0x60, 0x94, 0xF8, 0x51, 0x26, 0x04, 0xF6,
+ 0x6E, 0x05, 0x94, 0xF8, 0x50, 0x16, 0x00, 0xEB, 0x42, 0x00, 0x00, 0xF0, 0x30, 0xFF, 0x38, 0x68,
+ 0x90, 0xF8, 0x60, 0x18, 0x90, 0xF8, 0x61, 0x08, 0x05, 0xEB, 0x40, 0x00, 0x00, 0xF0, 0x3E, 0xFF,
+ 0x26, 0x49, 0x30, 0x46, 0x01, 0xF0, 0x6C, 0xFE, 0x00, 0x20, 0xBD, 0xE8, 0xF0, 0x81, 0x01, 0x20,
+ 0xFB, 0xE7, 0x30, 0xB5, 0x1E, 0x49, 0x4F, 0xF4, 0x80, 0x75, 0x0A, 0x68, 0x92, 0xF8, 0x07, 0x11,
+ 0x92, 0xF8, 0xF5, 0x30, 0x92, 0xF8, 0xF2, 0x40, 0x03, 0xF0, 0x1F, 0x03, 0x04, 0xF0, 0x0F, 0x04,
+ 0xA9, 0x42, 0x00, 0xD9, 0x29, 0x46, 0x1F, 0x2B, 0x00, 0xD9, 0x1F, 0x23, 0x5B, 0x1C, 0x59, 0x43,
+ 0x49, 0x00, 0x64, 0x1C, 0x61, 0x43, 0xC1, 0xEB, 0x41, 0x21, 0xC9, 0x13, 0x92, 0xF8, 0x50, 0x3A,
+ 0xB1, 0xFA, 0x81, 0xF1, 0xC1, 0xF1, 0x20, 0x01, 0x9B, 0x07, 0x13, 0xD5, 0x00, 0x23, 0x01, 0x28,
+ 0x09, 0xD1, 0x92, 0xF8, 0x54, 0x0A, 0x02, 0x07, 0x03, 0xD5, 0x00, 0xF0, 0x07, 0x00, 0x43, 0x42,
+ 0x01, 0xE0, 0x00, 0xF0, 0x07, 0x03, 0xC9, 0x18, 0x01, 0xD5, 0x00, 0x21, 0x02, 0xE0, 0x1F, 0x29,
+ 0x00, 0xDD, 0x1F, 0x21, 0xC8, 0xB2, 0x30, 0xBD, 0x1C, 0x61, 0x01, 0x20, 0xD4, 0x83, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0xD8, 0x81, 0x10, 0x00, 0xA0, 0x07, 0x10, 0x00, 0xFD, 0x2F, 0x01, 0x00,
+ 0x2D, 0xE9, 0xF0, 0x5F, 0xDF, 0xF8, 0xA4, 0x85, 0x83, 0x46, 0x4F, 0xF0, 0x01, 0x09, 0xD8, 0xF8,
+ 0x00, 0x00, 0x00, 0x27, 0xAB, 0xF1, 0x02, 0x01, 0x00, 0xF2, 0x5E, 0x64, 0xCA, 0x46, 0x06, 0x29,
+ 0x50, 0xD2, 0xDF, 0xE8, 0x01, 0xF0, 0x4F, 0x09, 0x2C, 0x4F, 0x2C, 0x03, 0x10, 0xF8, 0xD5, 0x6F,
+ 0x00, 0x79, 0x00, 0xF0, 0x1F, 0x05, 0x6D, 0xE0, 0x90, 0xF8, 0xF5, 0x10, 0x90, 0xF8, 0x01, 0x61,
+ 0x01, 0xF0, 0x1F, 0x05, 0x90, 0xF8, 0xF2, 0x10, 0x01, 0xF0, 0x0F, 0x07, 0x90, 0xF8, 0xE6, 0x10,
+ 0x89, 0x06, 0x09, 0xD5, 0x90, 0xF8, 0x52, 0x16, 0x90, 0xF8, 0x53, 0x06, 0x0A, 0x22, 0x04, 0xEB,
+ 0x40, 0x00, 0x00, 0xF0, 0x81, 0xFE, 0x81, 0x46, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0xE6, 0x10,
+ 0x49, 0x06, 0x4F, 0xD5, 0x90, 0xF8, 0x62, 0x18, 0x90, 0xF8, 0x63, 0x08, 0x44, 0xE0, 0x90, 0xF8,
+ 0xFC, 0x10, 0x90, 0xF8, 0xF8, 0x60, 0x01, 0xF0, 0x1F, 0x05, 0x90, 0xF8, 0xF9, 0x10, 0x01, 0xF0,
+ 0x0F, 0x07, 0x90, 0xF8, 0xE6, 0x10, 0x89, 0x06, 0x09, 0xD5, 0x90, 0xF8, 0x54, 0x16, 0x90, 0xF8,
+ 0x55, 0x06, 0x0A, 0x22, 0x04, 0xEB, 0x40, 0x00, 0x00, 0xF0, 0x5E, 0xFE, 0x81, 0x46, 0xD8, 0xF8,
+ 0x00, 0x00, 0x90, 0xF8, 0xE6, 0x10, 0x49, 0x06, 0x2C, 0xD5, 0x90, 0xF8, 0x64, 0x18, 0x90, 0xF8,
+ 0x65, 0x08, 0x21, 0xE0, 0x90, 0xF8, 0xF5, 0x10, 0x90, 0xF8, 0xF1, 0x60, 0x01, 0xF0, 0x1F, 0x05,
+ 0x90, 0xF8, 0xF2, 0x10, 0x01, 0xF0, 0x0F, 0x07, 0x90, 0xF8, 0xE6, 0x10, 0x89, 0x06, 0x09, 0xD5,
+ 0x90, 0xF8, 0x50, 0x16, 0x90, 0xF8, 0x51, 0x06, 0x0A, 0x22, 0x04, 0xEB, 0x40, 0x00, 0x00, 0xF0,
+ 0x3B, 0xFE, 0x81, 0x46, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0xE6, 0x10, 0x49, 0x06, 0x09, 0xD5,
+ 0x90, 0xF8, 0x60, 0x18, 0x90, 0xF8, 0x61, 0x08, 0x0A, 0x22, 0x04, 0xEB, 0x40, 0x00, 0x00, 0xF0,
+ 0x2B, 0xFE, 0x82, 0x46, 0x4F, 0xF4, 0x80, 0x70, 0x86, 0x42, 0x00, 0xD9, 0x06, 0x46, 0x1F, 0x2D,
+ 0x00, 0xD9, 0x1F, 0x25, 0x6D, 0x1C, 0x6E, 0x43, 0x70, 0x00, 0x7F, 0x1C, 0x78, 0x43, 0xC0, 0xEB,
+ 0x40, 0x20, 0x00, 0xFB, 0x09, 0xF0, 0x00, 0xFB, 0x0A, 0xF0, 0xC0, 0x13, 0xD8, 0xF8, 0x00, 0x20,
+ 0xB0, 0xFA, 0x80, 0xF0, 0xC0, 0xF1, 0x20, 0x01, 0x92, 0xF8, 0x50, 0x0A, 0xC0, 0x07, 0x23, 0xD0,
+ 0x00, 0x20, 0xBB, 0xF1, 0x05, 0x0F, 0x11, 0xD2, 0xDF, 0xE8, 0x0B, 0xF0, 0x10, 0x03, 0x06, 0x09,
+ 0x14, 0x00, 0x92, 0xF8, 0x51, 0x0A, 0x04, 0xE0, 0x92, 0xF8, 0x51, 0x0A, 0x0C, 0xE0, 0x92, 0xF8,
+ 0x52, 0x0A, 0x02, 0x07, 0x00, 0xF0, 0x07, 0x00, 0x00, 0xD5, 0x40, 0x42, 0x09, 0x18, 0x08, 0xD5,
+ 0x00, 0x21, 0x09, 0xE0, 0x92, 0xF8, 0x52, 0x0A, 0x02, 0x06, 0xC0, 0xF3, 0x02, 0x10, 0xF4, 0xD4,
+ 0xF4, 0xE7, 0x1F, 0x29, 0x00, 0xDD, 0x1F, 0x21, 0xC8, 0xB2, 0xBD, 0xE8, 0xF0, 0x9F, 0x70, 0xB5,
+ 0x06, 0x00, 0x03, 0xD0, 0xBD, 0xE8, 0x70, 0x40, 0x00, 0xF0, 0x55, 0xBD, 0x00, 0xF0, 0x18, 0xF8,
+ 0x01, 0x28, 0x14, 0xD1, 0xFE, 0x4D, 0xFF, 0x4C, 0x28, 0x78, 0x01, 0x28, 0x09, 0xD0, 0xFE, 0x48,
+ 0xA0, 0x60, 0x68, 0x78, 0xE0, 0x73, 0xE8, 0x78, 0xA0, 0x73, 0xE8, 0x88, 0xA0, 0x81, 0x30, 0x46,
+ 0xE8, 0xE7, 0x6A, 0x79, 0x29, 0x79, 0xF8, 0x48, 0x00, 0xF0, 0x55, 0xFE, 0xF0, 0xE7, 0x70, 0xBD,
+ 0xF3, 0x4B, 0x10, 0xB5, 0xD9, 0x78, 0xF5, 0x4C, 0x49, 0x1C, 0xCA, 0xB2, 0xDA, 0x70, 0x99, 0x78,
+ 0x01, 0x20, 0x8A, 0x42, 0x11, 0xD8, 0x19, 0x79, 0x58, 0x79, 0x41, 0x43, 0xEE, 0x48, 0xFD, 0xF7,
+ 0xB2, 0xFF, 0x01, 0xF0, 0xD8, 0xFA, 0xA4, 0xF8, 0x76, 0x00, 0x20, 0x46, 0x00, 0xF0, 0x92, 0xFF,
+ 0xEB, 0x49, 0x20, 0x46, 0x01, 0xF0, 0x14, 0xFD, 0x00, 0x20, 0x10, 0xBD, 0x38, 0xB5, 0x04, 0x46,
+ 0x01, 0xF0, 0xC4, 0xFA, 0xE4, 0x48, 0x01, 0xF0, 0xC6, 0xFA, 0xA4, 0xF8, 0x76, 0x00, 0x00, 0xF0,
+ 0x85, 0xFD, 0xDE, 0x4D, 0x0B, 0x46, 0x2A, 0x68, 0x92, 0xF8, 0xE7, 0x20, 0xC2, 0xF3, 0x41, 0x12,
+ 0x00, 0x92, 0x02, 0x46, 0x20, 0x46, 0x01, 0xF0, 0x9A, 0xFC, 0x00, 0xF0, 0x7C, 0xFD, 0x02, 0x46,
+ 0x0B, 0x46, 0x20, 0x46, 0x01, 0xF0, 0xBF, 0xFC, 0xA0, 0x79, 0x20, 0xF0, 0x02, 0x00, 0xA0, 0x71,
+ 0x94, 0xF8, 0x6F, 0x00, 0x20, 0xF0, 0x07, 0x00, 0x84, 0xF8, 0x6F, 0x00, 0x94, 0xF8, 0x6E, 0x00,
+ 0x20, 0xF0, 0x0F, 0x00, 0x84, 0xF8, 0x6E, 0x00, 0x28, 0x68, 0x90, 0xF8, 0xE7, 0x00, 0xC0, 0xF3,
+ 0x41, 0x11, 0x20, 0x46, 0x01, 0xF0, 0xC1, 0xFC, 0x28, 0x68, 0x90, 0xF8, 0xE7, 0x00, 0x10, 0xF0,
+ 0x60, 0x0F, 0x94, 0xF8, 0x5E, 0x00, 0x7D, 0xD0, 0x40, 0xF0, 0x01, 0x00, 0x84, 0xF8, 0x5E, 0x00,
+ 0x20, 0x46, 0x00, 0xF0, 0xFF, 0xFD, 0x05, 0x21, 0x20, 0x46, 0x01, 0xF0, 0xAD, 0xFA, 0x28, 0x68,
+ 0x90, 0xF8, 0xF1, 0x00, 0xA0, 0x70, 0x28, 0x68, 0x21, 0x8E, 0xB0, 0xF8, 0xF6, 0x20, 0x62, 0xF3,
+ 0x08, 0x01, 0x21, 0x86, 0x90, 0xF8, 0xF2, 0x00, 0x00, 0xF0, 0x0F, 0x01, 0x20, 0x46, 0x01, 0xF0,
+ 0x12, 0xFB, 0x28, 0x68, 0x62, 0x7D, 0x90, 0xF8, 0xEA, 0x10, 0x61, 0xF3, 0x03, 0x02, 0x62, 0x75,
+ 0xB0, 0xF8, 0xED, 0x20, 0x21, 0x8A, 0x62, 0xF3, 0x0B, 0x01, 0x21, 0x82, 0x90, 0xF8, 0xF3, 0x10,
+ 0xA2, 0x7D, 0x61, 0xF3, 0x03, 0x02, 0xA2, 0x75, 0x10, 0xF8, 0xF2, 0x1F, 0x0A, 0x09, 0xA1, 0x7C,
+ 0x62, 0xF3, 0x01, 0x01, 0xA1, 0x74, 0x10, 0xF8, 0x0C, 0x29, 0x92, 0x09, 0x62, 0xF3, 0x87, 0x11,
+ 0xA1, 0x74, 0x01, 0x78, 0xE2, 0x79, 0x61, 0xF3, 0x00, 0x02, 0xE2, 0x71, 0xC1, 0x78, 0x61, 0x72,
+ 0x80, 0x79, 0x01, 0x09, 0x20, 0x46, 0x01, 0xF0, 0xF2, 0xFA, 0xA6, 0x48, 0x01, 0xF0, 0x43, 0xFA,
+ 0xA4, 0xF8, 0x7C, 0x00, 0x29, 0x68, 0x11, 0xF8, 0xE8, 0x0F, 0xC2, 0x08, 0x94, 0xF8, 0x27, 0x00,
+ 0x62, 0xF3, 0x83, 0x00, 0x84, 0xF8, 0x27, 0x00, 0x11, 0xF8, 0x01, 0x29, 0x52, 0x08, 0x62, 0xF3,
+ 0x00, 0x00, 0x84, 0xF8, 0x27, 0x00, 0x4A, 0x78, 0x92, 0x08, 0x62, 0xF3, 0x41, 0x00, 0x84, 0xF8,
+ 0x27, 0x00, 0x08, 0x78, 0xC0, 0xF3, 0x81, 0x01, 0x20, 0x46, 0x01, 0xF0, 0xB1, 0xFA, 0x28, 0x68,
+ 0x90, 0xF8, 0xE7, 0x00, 0x01, 0x09, 0x14, 0xF8, 0x28, 0x0F, 0x61, 0xF3, 0x04, 0x10, 0x20, 0x70,
+ 0x91, 0x48, 0x01, 0xF0, 0x18, 0xFA, 0xA4, 0xF8, 0x5E, 0x00, 0x28, 0x68, 0x10, 0xF8, 0xE6, 0x1F,
+ 0xCA, 0x08, 0x00, 0xE0, 0x8F, 0xE0, 0xE1, 0x7A, 0x62, 0xF3, 0x45, 0x11, 0xE1, 0x72, 0x40, 0x78,
+ 0x41, 0x08, 0x20, 0x78, 0x61, 0xF3, 0x45, 0x10, 0x04, 0xF8, 0x28, 0x09, 0x20, 0x46, 0x00, 0xF0,
+ 0x3F, 0xFD, 0x28, 0x68, 0x90, 0xF8, 0xEC, 0x00, 0xC0, 0xF3, 0x81, 0x01, 0x20, 0x46, 0x01, 0xF0,
+ 0x74, 0xFA, 0x14, 0xF8, 0x32, 0x0F, 0x20, 0xF0, 0x0F, 0x00, 0x0A, 0x30, 0x20, 0xF0, 0xF0, 0x00,
+ 0xA0, 0x30, 0x04, 0xF8, 0x0A, 0x09, 0x29, 0x68, 0x11, 0xF8, 0xE6, 0x0F, 0x82, 0x09, 0xE0, 0x7A,
+ 0x62, 0xF3, 0x00, 0x00, 0xE0, 0x72, 0x0A, 0x78, 0x52, 0x09, 0x62, 0xF3, 0x41, 0x00, 0xE0, 0x72,
+ 0x4A, 0x79, 0xD2, 0x09, 0x62, 0xF3, 0x82, 0x00, 0xE0, 0x72, 0x88, 0x7B, 0x62, 0x79, 0x60, 0xF3,
+ 0x04, 0x02, 0x62, 0x71, 0xC8, 0x7B, 0x21, 0x79, 0x60, 0xF3, 0x04, 0x01, 0x6F, 0x48, 0x21, 0x71,
+ 0x94, 0xF8, 0x37, 0x10, 0x40, 0x78, 0x60, 0xF3, 0x04, 0x01, 0x84, 0xF8, 0x37, 0x10, 0x6C, 0x48,
+ 0x01, 0xF0, 0xC9, 0xF9, 0xA4, 0xF8, 0x52, 0x00, 0x28, 0x68, 0xE1, 0x88, 0xB0, 0xF8, 0xEF, 0x20,
+ 0x62, 0xF3, 0x0B, 0x01, 0xE1, 0x80, 0x90, 0xF8, 0xEB, 0x10, 0xCA, 0x08, 0x21, 0x7C, 0x62, 0xF3,
+ 0x02, 0x01, 0x21, 0x74, 0x10, 0xF8, 0xE2, 0x1F, 0xA2, 0x78, 0x61, 0xF3, 0x03, 0x02, 0xA2, 0x70,
+ 0x01, 0x78, 0x0A, 0x09, 0xE1, 0x78, 0x62, 0xF3, 0x03, 0x01, 0xE1, 0x70, 0x01, 0x7A, 0x0A, 0x09,
+ 0x61, 0x78, 0x62, 0xF3, 0x02, 0x01, 0x61, 0x70, 0x81, 0x79, 0x8A, 0x09, 0x21, 0x78, 0x62, 0xF3,
+ 0x00, 0x01, 0x21, 0x70, 0x01, 0x79, 0xCA, 0x09, 0x21, 0x7B, 0x62, 0xF3, 0x00, 0x01, 0x21, 0x73,
+ 0x94, 0xF8, 0x37, 0x10, 0x41, 0xF0, 0x20, 0x01, 0x84, 0xF8, 0x37, 0x10, 0x21, 0x8F, 0x6F, 0xF3,
+ 0x08, 0x01, 0x01, 0xF5, 0x80, 0x71, 0x21, 0x87, 0x00, 0x21, 0xC4, 0xF8, 0x3A, 0x10, 0x41, 0x7A,
+ 0x8A, 0x09, 0xE1, 0x7A, 0x62, 0xF3, 0xC3, 0x01, 0xE1, 0x72, 0x00, 0x79, 0x00, 0x09, 0x60, 0xF3,
+ 0x04, 0x11, 0xE1, 0x72, 0x38, 0xBD, 0x20, 0xF0, 0x01, 0x00, 0xEF, 0xE6, 0x38, 0xB5, 0x04, 0x46,
+ 0x01, 0xF0, 0x74, 0xF9, 0x3C, 0x48, 0x01, 0xF0, 0x76, 0xF9, 0xA4, 0xF8, 0x76, 0x00, 0x00, 0xF0,
+ 0x35, 0xFC, 0x36, 0x4D, 0x0B, 0x46, 0x2A, 0x68, 0x92, 0xF8, 0xC9, 0x20, 0xC2, 0xF3, 0x41, 0x12,
+ 0x00, 0x92, 0x02, 0x46, 0x20, 0x46, 0x01, 0xF0, 0x4A, 0xFB, 0x00, 0xF0, 0x2C, 0xFC, 0x02, 0x46,
+ 0x0B, 0x46, 0x20, 0x46, 0x01, 0xF0, 0x6F, 0xFB, 0xA0, 0x79, 0x20, 0xF0, 0x02, 0x00, 0xA0, 0x71,
+ 0x94, 0xF8, 0x6F, 0x00, 0x20, 0xF0, 0x07, 0x00, 0x84, 0xF8, 0x6F, 0x00, 0x94, 0xF8, 0x6E, 0x00,
+ 0x20, 0xF0, 0x0F, 0x00, 0x84, 0xF8, 0x6E, 0x00, 0x28, 0x68, 0x90, 0xF8, 0xC9, 0x00, 0xC0, 0xF3,
+ 0x41, 0x11, 0x20, 0x46, 0x01, 0xF0, 0x71, 0xFB, 0x28, 0x68, 0x90, 0xF8, 0xC9, 0x00, 0x10, 0xF0,
+ 0x60, 0x0F, 0x94, 0xF8, 0x5E, 0x00, 0x7D, 0xD0, 0x40, 0xF0, 0x01, 0x00, 0x84, 0xF8, 0x5E, 0x00,
+ 0x20, 0x46, 0x00, 0xF0, 0xAF, 0xFC, 0x28, 0x68, 0x90, 0xF8, 0xD5, 0x00, 0xA0, 0x70, 0x28, 0x68,
+ 0x21, 0x8E, 0xB0, 0xF8, 0xDA, 0x20, 0x62, 0xF3, 0x08, 0x01, 0x21, 0x86, 0x90, 0xF8, 0xD6, 0x00,
+ 0x00, 0xF0, 0x0F, 0x01, 0x20, 0x46, 0x01, 0xF0, 0xC6, 0xF9, 0x28, 0x68, 0x62, 0x7D, 0x90, 0xF8,
+ 0xCC, 0x10, 0x61, 0xF3, 0x03, 0x02, 0x62, 0x75, 0xB0, 0xF8, 0xCF, 0x20, 0x21, 0x8A, 0x62, 0xF3,
+ 0x0B, 0x01, 0x21, 0x82, 0x90, 0xF8, 0xD7, 0x10, 0xA2, 0x7D, 0x61, 0xF3, 0x03, 0x02, 0xA2, 0x75,
+ 0xB0, 0xF8, 0xD1, 0x20, 0xA1, 0x89, 0x62, 0xF3, 0x0B, 0x01, 0xA1, 0x81, 0x10, 0xF8, 0xD6, 0x1F,
+ 0x0A, 0x09, 0xA1, 0x7C, 0x62, 0xF3, 0x01, 0x01, 0xA1, 0x74, 0x13, 0xE0, 0x20, 0x07, 0x10, 0x00,
+ 0xA2, 0x07, 0x10, 0x00, 0xD4, 0x83, 0x10, 0x00, 0x4C, 0x62, 0x01, 0x20, 0x20, 0x83, 0x10, 0x00,
+ 0x8F, 0x33, 0x01, 0x00, 0xFC, 0x4C, 0x01, 0x20, 0x88, 0x5C, 0x01, 0x20, 0x4E, 0x7B, 0x01, 0x20,
+ 0xB4, 0x59, 0x01, 0x20, 0x10, 0xF8, 0x0D, 0x29, 0x92, 0x09, 0x62, 0xF3, 0x87, 0x11, 0xA1, 0x74,
+ 0x10, 0xF8, 0x01, 0x1C, 0xE2, 0x79, 0x61, 0xF3, 0x00, 0x02, 0xE2, 0x71, 0x81, 0x78, 0x61, 0x72,
+ 0x40, 0x79, 0x01, 0x09, 0x20, 0x46, 0x01, 0xF0, 0x8A, 0xF9, 0xFB, 0x48, 0x01, 0xF0, 0xDB, 0xF8,
+ 0xA4, 0xF8, 0x7C, 0x00, 0x28, 0x68, 0x10, 0xF8, 0xCA, 0x1F, 0xCA, 0x08, 0x94, 0xF8, 0x27, 0x10,
+ 0x62, 0xF3, 0x83, 0x01, 0x84, 0xF8, 0x27, 0x10, 0x10, 0xF8, 0x01, 0x29, 0x52, 0x08, 0x62, 0xF3,
+ 0x00, 0x01, 0x84, 0xF8, 0x27, 0x10, 0x42, 0x78, 0x92, 0x08, 0x62, 0xF3, 0x41, 0x01, 0x84, 0xF8,
+ 0x27, 0x10, 0x00, 0xE0, 0xAA, 0xE0, 0x00, 0x78, 0xC0, 0xF3, 0x81, 0x01, 0x20, 0x46, 0x01, 0xF0,
+ 0x47, 0xF9, 0x28, 0x68, 0x90, 0xF8, 0xC9, 0x00, 0x01, 0x09, 0x14, 0xF8, 0x28, 0x0F, 0x61, 0xF3,
+ 0x04, 0x10, 0x20, 0x70, 0xE5, 0x48, 0x01, 0xF0, 0xAE, 0xF8, 0xA4, 0xF8, 0x5E, 0x00, 0x28, 0x68,
+ 0x10, 0xF8, 0xC8, 0x1F, 0xCA, 0x08, 0xE1, 0x7A, 0x62, 0xF3, 0x45, 0x11, 0xE1, 0x72, 0x40, 0x78,
+ 0x41, 0x08, 0x20, 0x78, 0x61, 0xF3, 0x45, 0x10, 0x04, 0xF8, 0x28, 0x09, 0x20, 0x46, 0x00, 0xF0,
+ 0xD7, 0xFB, 0x28, 0x68, 0x90, 0xF8, 0xCE, 0x00, 0xC0, 0xF3, 0x81, 0x01, 0x20, 0x46, 0x01, 0xF0,
+ 0x0C, 0xF9, 0x14, 0xF8, 0x32, 0x0F, 0x20, 0xF0, 0x0F, 0x00, 0x0A, 0x30, 0x20, 0xF0, 0xF0, 0x00,
+ 0xA0, 0x30, 0x04, 0xF8, 0x0A, 0x09, 0x29, 0x68, 0x11, 0xF8, 0xC8, 0x0F, 0x82, 0x09, 0xE0, 0x7A,
+ 0x62, 0xF3, 0x00, 0x00, 0xE0, 0x72, 0x0A, 0x78, 0x52, 0x09, 0x62, 0xF3, 0x41, 0x00, 0xE0, 0x72,
+ 0x4A, 0x79, 0xD2, 0x09, 0x62, 0xF3, 0x82, 0x00, 0xE0, 0x72, 0x08, 0x7C, 0x62, 0x79, 0x60, 0xF3,
+ 0x04, 0x02, 0x62, 0x71, 0x48, 0x7C, 0x21, 0x79, 0x60, 0xF3, 0x04, 0x01, 0xC4, 0x48, 0x21, 0x71,
+ 0x94, 0xF8, 0x37, 0x10, 0xC0, 0x79, 0x60, 0xF3, 0x04, 0x01, 0x84, 0xF8, 0x37, 0x10, 0xC1, 0x48,
+ 0x01, 0xF0, 0x61, 0xF8, 0xA4, 0xF8, 0x52, 0x00, 0x28, 0x68, 0xE1, 0x88, 0xB0, 0xF8, 0xD3, 0x20,
+ 0x62, 0xF3, 0x0B, 0x01, 0xE1, 0x80, 0x10, 0xF8, 0xCD, 0x1F, 0xCA, 0x08, 0x21, 0x7C, 0x62, 0xF3,
+ 0x02, 0x01, 0x21, 0x74, 0x10, 0xF8, 0x15, 0x1F, 0xA2, 0x78, 0x61, 0xF3, 0x03, 0x02, 0xA2, 0x70,
+ 0x10, 0xF8, 0x1A, 0x19, 0x0A, 0x09, 0xE1, 0x78, 0x62, 0xF3, 0x03, 0x01, 0xE1, 0x70, 0x01, 0x79,
+ 0x0A, 0x09, 0x61, 0x78, 0x62, 0xF3, 0x02, 0x01, 0x61, 0x70, 0x81, 0x78, 0x8A, 0x09, 0x21, 0x78,
+ 0x62, 0xF3, 0x00, 0x01, 0x21, 0x70, 0x01, 0x78, 0xCA, 0x09, 0x21, 0x7B, 0x62, 0xF3, 0x00, 0x01,
+ 0x21, 0x73, 0x01, 0x79, 0xCA, 0x09, 0x94, 0xF8, 0x37, 0x10, 0x62, 0xF3, 0x45, 0x11, 0x84, 0xF8,
+ 0x37, 0x10, 0x21, 0x8F, 0x6F, 0xF3, 0x08, 0x01, 0x01, 0xF5, 0x80, 0x71, 0x21, 0x87, 0x00, 0x21,
+ 0xC4, 0xF8, 0x3A, 0x10, 0x41, 0x79, 0x8A, 0x09, 0xE1, 0x7A, 0x62, 0xF3, 0xC3, 0x01, 0xE1, 0x72,
+ 0x00, 0x78, 0x00, 0x09, 0x60, 0xF3, 0x04, 0x11, 0xE1, 0x72, 0x38, 0xBD, 0x20, 0xF0, 0x01, 0x00,
+ 0xD4, 0xE6, 0x10, 0xB5, 0x04, 0x46, 0xA4, 0x22, 0x97, 0x49, 0x02, 0xF0, 0x1B, 0xFF, 0x97, 0x48,
+ 0xC1, 0x78, 0xA0, 0x89, 0x61, 0xF3, 0x0B, 0x00, 0xA0, 0x81, 0x10, 0xBD, 0x38, 0xB5, 0x04, 0x46,
+ 0xFF, 0xF7, 0xEF, 0xFF, 0x00, 0xF0, 0xC2, 0xFA, 0x00, 0x22, 0x00, 0x92, 0x02, 0x46, 0x0B, 0x46,
+ 0x20, 0x46, 0x01, 0xF0, 0xDC, 0xF9, 0x8E, 0x48, 0x00, 0xF0, 0xF5, 0xFF, 0x8D, 0x4D, 0xA4, 0xF8,
+ 0x7C, 0x00, 0x28, 0x68, 0x90, 0xF8, 0xF8, 0x00, 0xA0, 0x70, 0x28, 0x68, 0x90, 0xF8, 0xF9, 0x00,
+ 0x00, 0xF0, 0x0F, 0x01, 0x20, 0x46, 0x01, 0xF0, 0x86, 0xF8, 0x28, 0x68, 0x10, 0xF8, 0xF9, 0x1F,
+ 0x0A, 0x09, 0xA1, 0x7C, 0x62, 0xF3, 0x01, 0x01, 0xA1, 0x74, 0x02, 0x78, 0x92, 0x09, 0x62, 0xF3,
+ 0x87, 0x11, 0xA1, 0x74, 0x81, 0x78, 0x94, 0xF8, 0x2D, 0x20, 0x61, 0xF3, 0x04, 0x02, 0x84, 0xF8,
+ 0x2D, 0x20, 0xC1, 0x78, 0x94, 0xF8, 0x2C, 0x20, 0x61, 0xF3, 0x04, 0x02, 0x84, 0xF8, 0x2C, 0x20,
+ 0x40, 0x78, 0xA1, 0x7D, 0x60, 0xF3, 0x03, 0x01, 0xA1, 0x75, 0x00, 0x21, 0x20, 0x46, 0x01, 0xF0,
+ 0xEC, 0xF9, 0x14, 0xF8, 0x5E, 0x0F, 0x20, 0xF0, 0x01, 0x00, 0x04, 0xF8, 0x01, 0x0B, 0x6C, 0x48,
+ 0x01, 0x79, 0x20, 0x78, 0x61, 0xF3, 0x04, 0x00, 0x04, 0xF8, 0x2F, 0x09, 0x28, 0x68, 0xB0, 0xF8,
+ 0xFD, 0x10, 0x20, 0x88, 0x61, 0xF3, 0x08, 0x00, 0x20, 0x80, 0x38, 0xBD, 0x70, 0xB5, 0x69, 0x4E,
+ 0x03, 0x28, 0x34, 0x68, 0x04, 0xF2, 0x5E, 0x62, 0x04, 0xF6, 0x6E, 0x05, 0x14, 0xD0, 0x04, 0x28,
+ 0x20, 0xD0, 0x94, 0xF8, 0x51, 0x06, 0x94, 0xF8, 0x50, 0x16, 0x02, 0xEB, 0x40, 0x00, 0x00, 0xF0,
+ 0x96, 0xFA, 0x30, 0x68, 0x90, 0xF8, 0x60, 0x18, 0x90, 0xF8, 0x61, 0x08, 0x05, 0xEB, 0x40, 0x00,
+ 0xBD, 0xE8, 0x70, 0x40, 0x00, 0xF0, 0xA2, 0xBA, 0x94, 0xF8, 0x53, 0x06, 0x94, 0xF8, 0x52, 0x16,
+ 0x02, 0xEB, 0x40, 0x00, 0x00, 0xF0, 0x83, 0xFA, 0x30, 0x68, 0x90, 0xF8, 0x62, 0x18, 0x90, 0xF8,
+ 0x63, 0x08, 0xEB, 0xE7, 0x94, 0xF8, 0x55, 0x06, 0x94, 0xF8, 0x54, 0x16, 0x02, 0xEB, 0x40, 0x00,
+ 0x00, 0xF0, 0x75, 0xFA, 0x30, 0x68, 0x90, 0xF8, 0x64, 0x18, 0x90, 0xF8, 0x65, 0x08, 0xDD, 0xE7,
+ 0x2D, 0xE9, 0xF8, 0x43, 0x4C, 0x4D, 0x4F, 0xF0, 0x01, 0x09, 0x00, 0x26, 0x85, 0xF8, 0x00, 0x90,
+ 0xAE, 0x70, 0x43, 0x4F, 0xDF, 0xF8, 0x1C, 0x81, 0x04, 0x46, 0xEE, 0x70, 0x09, 0x29, 0x72, 0xD2,
+ 0xDF, 0xE8, 0x01, 0xF0, 0x71, 0x05, 0x09, 0x18, 0x44, 0x2F, 0x48, 0x5E, 0x62, 0x00, 0x20, 0x46,
+ 0xFF, 0xF7, 0x47, 0xFF, 0x67, 0xE0, 0x20, 0x46, 0xFF, 0xF7, 0x43, 0xFF, 0xB8, 0x78, 0x94, 0xF8,
+ 0x5F, 0x10, 0x60, 0xF3, 0x04, 0x01, 0x84, 0xF8, 0x5F, 0x10, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8,
+ 0xFF, 0x00, 0x12, 0xE0, 0x20, 0x46, 0xFF, 0xF7, 0x34, 0xFF, 0xD8, 0xF8, 0x00, 0x10, 0x91, 0xF8,
+ 0x01, 0x11, 0xA1, 0x70, 0xF9, 0x78, 0x94, 0xF8, 0x5F, 0x20, 0x61, 0xF3, 0x04, 0x02, 0x84, 0xF8,
+ 0x5F, 0x20, 0xD8, 0xF8, 0x00, 0x00, 0x90, 0xF8, 0x00, 0x01, 0x00, 0xF0, 0x03, 0x00, 0xA8, 0x70,
+ 0x41, 0xE0, 0x20, 0x46, 0xFF, 0xF7, 0x1D, 0xFF, 0x00, 0x21, 0x20, 0x46, 0x00, 0xF0, 0xCB, 0xFF,
+ 0x94, 0xF8, 0x5E, 0x00, 0x20, 0xF0, 0x01, 0x00, 0x84, 0xF8, 0x5E, 0x00, 0x78, 0x79, 0x94, 0xF8,
+ 0x5F, 0x10, 0x60, 0xF3, 0x04, 0x01, 0x84, 0xF8, 0x5F, 0x10, 0x17, 0xE0, 0x20, 0x46, 0xFF, 0xF7,
+ 0x15, 0xFF, 0x28, 0xE0, 0x20, 0x46, 0xFF, 0xF7, 0x11, 0xFF, 0x00, 0x21, 0x20, 0x46, 0x00, 0xF0,
+ 0xB2, 0xFF, 0xB8, 0x79, 0x94, 0xF8, 0x5F, 0x10, 0x60, 0xF3, 0x04, 0x01, 0x84, 0xF8, 0x5F, 0x10,
+ 0x94, 0xF8, 0x5E, 0x00, 0x20, 0xF0, 0x01, 0x00, 0x84, 0xF8, 0x5E, 0x00, 0x2E, 0x70, 0x12, 0xE0,
+ 0x20, 0x46, 0xFF, 0xF7, 0x83, 0xFD, 0xF9, 0xE7, 0x20, 0x46, 0xFF, 0xF7, 0xEA, 0xFE, 0x01, 0x22,
+ 0x00, 0x23, 0x20, 0x46, 0x00, 0x96, 0x01, 0xF0, 0xDA, 0xF8, 0x20, 0x46, 0x00, 0xF0, 0xAB, 0xFA,
+ 0x20, 0x46, 0x00, 0xF0, 0x2D, 0xFA, 0x20, 0x46, 0x00, 0xF0, 0xAC, 0xFB, 0x94, 0xF8, 0x27, 0x00,
+ 0xC0, 0x07, 0x13, 0xD0, 0x6E, 0x70, 0x13, 0xE0, 0xFC, 0x4C, 0x01, 0x20, 0x88, 0x5C, 0x01, 0x20,
+ 0x4E, 0x7B, 0x01, 0x20, 0xB4, 0x59, 0x01, 0x20, 0x7C, 0x82, 0x10, 0x00, 0x4C, 0x07, 0x10, 0x00,
+ 0x24, 0x52, 0x01, 0x20, 0x20, 0x07, 0x10, 0x00, 0xA2, 0x07, 0x10, 0x00, 0x85, 0xF8, 0x01, 0x90,
+ 0xA0, 0x89, 0xC0, 0xF3, 0x0B, 0x00, 0xE8, 0x80, 0xBD, 0xE8, 0xF8, 0x83, 0x7F, 0xB5, 0x00, 0xF0,
+ 0x8D, 0xF9, 0xCD, 0xE9, 0x00, 0x01, 0x00, 0xF0, 0x8E, 0xF9, 0xCD, 0xE9, 0x02, 0x01, 0x08, 0x21,
+ 0x68, 0x46, 0x01, 0xF0, 0x0F, 0xFB, 0x1E, 0x4C, 0x08, 0x21, 0x20, 0x71, 0x02, 0xA8, 0x01, 0xF0,
+ 0x09, 0xFB, 0x60, 0x71, 0x1B, 0x48, 0x00, 0x21, 0x1A, 0x4D, 0x40, 0xF8, 0x02, 0x1F, 0x01, 0x24,
+ 0x41, 0x60, 0x01, 0x72, 0xAD, 0x1C, 0xE0, 0xB2, 0xFF, 0xF7, 0xD2, 0xFA, 0x28, 0x55, 0x64, 0x1C,
+ 0x09, 0x2C, 0xF8, 0xDB, 0x14, 0x48, 0xFF, 0xF7, 0xD9, 0xFB, 0x7F, 0xBD, 0x70, 0xB5, 0x12, 0x4D,
+ 0x04, 0x79, 0xA4, 0x35, 0x8C, 0xB1, 0x21, 0x46, 0x28, 0x46, 0xFF, 0xF7, 0x29, 0xFF, 0xA8, 0x7C,
+ 0x00, 0xF0, 0x03, 0x00, 0x00, 0xF0, 0xC2, 0xF9, 0x20, 0x46, 0xFF, 0xF7, 0xE7, 0xFE, 0x0B, 0x49,
+ 0x28, 0x46, 0x01, 0xF0, 0xD5, 0xF8, 0x00, 0x20, 0x70, 0xBD, 0x01, 0x20, 0x70, 0xBD, 0x40, 0xF2,
+ 0xFF, 0x10, 0xC1, 0xEB, 0x41, 0x21, 0x91, 0xFB, 0xF0, 0xF0, 0x00, 0x20, 0x70, 0x47, 0x00, 0x00,
+ 0xA2, 0x07, 0x10, 0x00, 0x4C, 0x7B, 0x01, 0x20, 0x7C, 0x82, 0x10, 0x00, 0x8F, 0x33, 0x01, 0x00,
+ 0x10, 0xB5, 0x00, 0xF0, 0x7D, 0xFA, 0xFF, 0xF7, 0xA9, 0xFF, 0xFF, 0xF7, 0x0F, 0xFA, 0xFF, 0xF7,
+ 0x5B, 0xF9, 0xFE, 0xF7, 0xC2, 0xFC, 0xFE, 0xF7, 0x38, 0xFD, 0x00, 0xF0, 0x8D, 0xFA, 0x7F, 0x49,
+ 0x00, 0x20, 0x48, 0x60, 0x70, 0x21, 0x7E, 0x48, 0x02, 0xF0, 0xF3, 0xFD, 0x7D, 0x48, 0x04, 0x68,
+ 0x94, 0xF8, 0xE7, 0x00, 0xC0, 0x06, 0x05, 0xD5, 0x28, 0x22, 0x04, 0xF5, 0x0F, 0x61, 0x78, 0x48,
+ 0x02, 0xF0, 0x68, 0xFD, 0x94, 0xF8, 0x33, 0x01, 0xC0, 0x06, 0x0F, 0xD5, 0x74, 0x48, 0x20, 0x22,
+ 0x04, 0xF6, 0x16, 0x11, 0x28, 0x30, 0x02, 0xF0, 0x5D, 0xFD, 0x04, 0xF6, 0x3F, 0x11, 0x70, 0x48,
+ 0x28, 0x22, 0xBD, 0xE8, 0x10, 0x40, 0x48, 0x30, 0x02, 0xF0, 0x54, 0xBD, 0x10, 0xBD, 0x38, 0xB1,
+ 0x01, 0x28, 0x07, 0xD0, 0x02, 0x28, 0x07, 0xD0, 0x17, 0x20, 0x00, 0x21, 0xFC, 0xF7, 0x97, 0xBA,
+ 0x14, 0x20, 0xFA, 0xE7, 0x15, 0x20, 0xF8, 0xE7, 0x16, 0x20, 0xF6, 0xE7, 0x00, 0x21, 0x01, 0x60,
+ 0x41, 0x60, 0x81, 0x60, 0xC1, 0x60, 0x70, 0x47, 0x70, 0xB5, 0x60, 0x4C, 0x62, 0x4D, 0x20, 0x68,
+ 0x09, 0xE0, 0x00, 0xBF, 0x55, 0xF8, 0x20, 0x10, 0x60, 0x48, 0x88, 0x47, 0x00, 0x28, 0x11, 0xD0,
+ 0x20, 0x68, 0x40, 0x1C, 0x20, 0x60, 0x07, 0x28, 0xF4, 0xD3, 0x60, 0x68, 0x5B, 0x49, 0x40, 0x1C,
+ 0x10, 0x31, 0x00, 0x22, 0x60, 0x60, 0x0A, 0x70, 0x48, 0x60, 0x58, 0x48, 0x01, 0x68, 0x10, 0x30,
+ 0x88, 0x47, 0x01, 0x20, 0x70, 0xBD, 0x70, 0xB5, 0x04, 0x46, 0x00, 0x79, 0x00, 0x25, 0x40, 0xB9,
+ 0xE0, 0x79, 0x30, 0xB9, 0xA0, 0x79, 0x20, 0xB9, 0x20, 0x7A, 0x10, 0xB9, 0x60, 0x79, 0x00, 0x28,
+ 0x17, 0xD0, 0x4E, 0x48, 0x34, 0x21, 0x10, 0x30, 0x02, 0xF0, 0xAD, 0xFD, 0x1E, 0xCC, 0x4B, 0x48,
+ 0x80, 0xE8, 0x1E, 0x00, 0x45, 0x48, 0x05, 0x60, 0x46, 0x48, 0x00, 0x68, 0x90, 0xF8, 0xB0, 0x00,
+ 0xC0, 0x07, 0x02, 0xD0, 0x00, 0xF0, 0x18, 0xFA, 0x02, 0xE0, 0x44, 0x48, 0xFF, 0xF7, 0xBC, 0xFF,
+ 0x01, 0x20, 0x70, 0xBD, 0x10, 0xB5, 0x01, 0x20, 0x00, 0xF0, 0x62, 0xF8, 0x07, 0x20, 0x00, 0xF0,
+ 0x5F, 0xF8, 0x08, 0x20, 0x00, 0xF0, 0x5C, 0xF8, 0x05, 0x20, 0x00, 0xF0, 0x59, 0xF8, 0x0D, 0x20,
+ 0x00, 0xF0, 0x56, 0xF8, 0x0C, 0x20, 0x00, 0xF0, 0x53, 0xF8, 0x01, 0x20, 0x00, 0xF0, 0x5B, 0xF8,
+ 0x07, 0x20, 0x00, 0xF0, 0x58, 0xF8, 0x08, 0x20, 0x00, 0xF0, 0x55, 0xF8, 0x05, 0x20, 0x00, 0xF0,
+ 0x52, 0xF8, 0x0D, 0x20, 0x00, 0xF0, 0x4F, 0xF8, 0x0C, 0x20, 0x00, 0xF0, 0x4C, 0xF8, 0x02, 0xF0,
+ 0x86, 0xF9, 0x00, 0xF0, 0xB3, 0xFD, 0x00, 0xF0, 0xDF, 0xF9, 0x01, 0x20, 0x00, 0xF0, 0x2D, 0xF8,
+ 0x07, 0x20, 0x00, 0xF0, 0x2A, 0xF8, 0x08, 0x20, 0x00, 0xF0, 0x27, 0xF8, 0xBD, 0xE8, 0x10, 0x40,
+ 0x05, 0x20, 0x00, 0xF0, 0x22, 0xB8, 0x70, 0xB5, 0x20, 0x4C, 0x05, 0x00, 0x0E, 0xD0, 0xFF, 0xF7,
+ 0xC1, 0xFF, 0x60, 0x68, 0x21, 0x49, 0x40, 0x1C, 0x10, 0x31, 0x60, 0x60, 0x0D, 0x70, 0x48, 0x60,
+ 0x1E, 0x48, 0x01, 0x68, 0xBD, 0xE8, 0x70, 0x40, 0x10, 0x30, 0x08, 0x47, 0x20, 0x68, 0x40, 0x1C,
+ 0x20, 0x60, 0xBD, 0xE8, 0x70, 0x40, 0x19, 0x48, 0x66, 0xE7, 0x00, 0xF0, 0x57, 0xB8, 0x00, 0xF0,
+ 0x5A, 0xB8, 0x00, 0xF0, 0x5D, 0xB8, 0x00, 0xF0, 0x61, 0xB8, 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21,
+ 0x91, 0x40, 0x40, 0x09, 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20, 0xC0, 0xF8, 0x00, 0x11, 0x70, 0x47,
+ 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21, 0x91, 0x40, 0x40, 0x09, 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20,
+ 0xC0, 0xF8, 0x80, 0x11, 0x70, 0x47, 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21, 0x91, 0x40, 0x40, 0x09,
+ 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20, 0xC0, 0xF8, 0x80, 0x12, 0x70, 0x47, 0xAC, 0x07, 0x10, 0x00,
+ 0x88, 0x5C, 0x01, 0x20, 0x20, 0x07, 0x10, 0x00, 0x4C, 0x6F, 0x01, 0x00, 0xC4, 0x83, 0x10, 0x00,
+ 0x70, 0xB5, 0x00, 0x23, 0x14, 0x46, 0x09, 0xE0, 0x31, 0xF9, 0x13, 0x50, 0x31, 0xF9, 0x14, 0x60,
+ 0x35, 0x44, 0x6D, 0x10, 0x20, 0xF8, 0x13, 0x50, 0x5B, 0x1C, 0x64, 0x1C, 0x93, 0x42, 0xF3, 0xDB,
+ 0x70, 0xBD, 0x30, 0xB4, 0x00, 0x24, 0x4B, 0xB9, 0x0C, 0xE0, 0x00, 0xBF, 0x30, 0xF8, 0x14, 0x30,
+ 0x31, 0xF8, 0x14, 0x50, 0x2B, 0x44, 0x20, 0xF8, 0x14, 0x30, 0x64, 0x1C, 0x94, 0x42, 0xF5, 0xDB,
+ 0x30, 0xBC, 0x70, 0x47, 0x30, 0xBC, 0x52, 0x00, 0x02, 0xF0, 0x4C, 0xBC, 0x99, 0x48, 0x00, 0x68,
+ 0xC1, 0x6A, 0x80, 0x6A, 0x70, 0x47, 0x97, 0x48, 0x00, 0x68, 0x41, 0x6A, 0x00, 0x6A, 0x70, 0x47,
+ 0x94, 0x48, 0x00, 0x68, 0x00, 0xF6, 0x38, 0x20, 0x03, 0xC8, 0x70, 0x47, 0x91, 0x48, 0x00, 0x68,
+ 0x00, 0xF5, 0x23, 0x60, 0x03, 0xC8, 0x70, 0x47, 0xF0, 0xB5, 0x00, 0x24, 0x25, 0x46, 0x8E, 0x4E,
+ 0x4F, 0xF4, 0x00, 0x67, 0x14, 0xE0, 0x00, 0xBF, 0x30, 0xF8, 0x15, 0x30, 0xC3, 0xF3, 0x0B, 0x03,
+ 0x5F, 0xEA, 0xD3, 0x2C, 0x00, 0xD0, 0x33, 0x43, 0x00, 0x2B, 0x00, 0xDA, 0x5B, 0x42, 0x9C, 0x42,
+ 0x00, 0xDA, 0x9C, 0xB2, 0xBC, 0x42, 0x02, 0xD3, 0x40, 0xF2, 0xFF, 0x74, 0x02, 0xE0, 0x6D, 0x1C,
+ 0x8D, 0x42, 0xE9, 0xDB, 0xD4, 0x40, 0x01, 0x20, 0xA0, 0x40, 0x80, 0xB2, 0xF0, 0xBD, 0x30, 0xB5,
+ 0x7F, 0x4B, 0x7E, 0x4C, 0x5A, 0x68, 0x82, 0x42, 0x02, 0xD1, 0xDA, 0x68, 0x8A, 0x42, 0x0A, 0xD0,
+ 0x00, 0x22, 0x06, 0xE0, 0x30, 0xF8, 0x12, 0x50, 0xC5, 0xF3, 0x0B, 0x05, 0x24, 0xF8, 0x12, 0x50,
+ 0x52, 0x1C, 0x8A, 0x42, 0xF6, 0xDB, 0xD9, 0x60, 0x58, 0x60, 0x30, 0xBD, 0x30, 0xB5, 0x74, 0x4B,
+ 0x74, 0x4C, 0x9A, 0x68, 0x82, 0x42, 0x02, 0xD1, 0x1A, 0x69, 0x8A, 0x42, 0x0D, 0xD0, 0x00, 0x22,
+ 0x07, 0xE0, 0x00, 0xBF, 0x30, 0xF8, 0x12, 0x50, 0xC5, 0xF3, 0x0B, 0x05, 0x24, 0xF8, 0x12, 0x50,
+ 0x52, 0x1C, 0x8A, 0x42, 0xF6, 0xDB, 0x19, 0x61, 0x98, 0x60, 0x30, 0xBD, 0x70, 0xB5, 0x04, 0x46,
+ 0x69, 0x48, 0x01, 0x68, 0x02, 0x29, 0x0A, 0xD1, 0x65, 0x4D, 0x29, 0x78, 0x8C, 0x42, 0x06, 0xD0,
+ 0x00, 0xEB, 0x84, 0x00, 0xD0, 0xF8, 0x9C, 0x00, 0x00, 0xF0, 0xD7, 0xFC, 0x2C, 0x70, 0x70, 0xBD,
+ 0x10, 0xB5, 0x04, 0x46, 0x90, 0xF8, 0x28, 0x00, 0x80, 0x06, 0x04, 0xD5, 0x5F, 0x48, 0x00, 0xF0,
+ 0xBA, 0xFC, 0xA4, 0xF8, 0x8C, 0x00, 0x10, 0xBD, 0x10, 0xB5, 0x04, 0x46, 0x90, 0xF8, 0x28, 0x00,
+ 0x80, 0x06, 0x0A, 0xD5, 0x59, 0x48, 0x28, 0x30, 0x00, 0xF0, 0xAD, 0xFC, 0x24, 0xF8, 0x8E, 0x0F,
+ 0x56, 0x48, 0x48, 0x30, 0x00, 0xF0, 0xA7, 0xFC, 0x60, 0x80, 0x10, 0xBD, 0x10, 0xB5, 0x04, 0x46,
+ 0x90, 0xF8, 0x28, 0x00, 0x80, 0x06, 0x05, 0xD5, 0x50, 0x48, 0x70, 0x30, 0x00, 0xF0, 0x9B, 0xFC,
+ 0xA4, 0xF8, 0x8E, 0x00, 0x10, 0xBD, 0x70, 0x47, 0x70, 0x47, 0x70, 0x47, 0x02, 0x00, 0x4F, 0xF0,
+ 0x00, 0x00, 0x00, 0xD1, 0x08, 0x46, 0x70, 0x47, 0x02, 0x46, 0x00, 0x20, 0x01, 0x2A, 0x00, 0xD1,
+ 0x08, 0x46, 0x70, 0x47, 0x3F, 0x49, 0x0A, 0x68, 0x92, 0xF8, 0xB0, 0x20, 0x93, 0x09, 0x10, 0xF8,
+ 0x3E, 0x2B, 0x63, 0xF3, 0x82, 0x02, 0x00, 0xF8, 0x3E, 0x2C, 0x0B, 0x68, 0x93, 0xF8, 0xB0, 0x30,
+ 0x5B, 0x09, 0x63, 0xF3, 0x45, 0x12, 0x00, 0xF8, 0x3E, 0x2C, 0x09, 0x68, 0xB1, 0xF8, 0xB7, 0x20,
+ 0x02, 0x82, 0x91, 0xF8, 0xB6, 0x20, 0x83, 0x7F, 0x62, 0xF3, 0x07, 0x13, 0x83, 0x77, 0x91, 0xF8,
+ 0xB3, 0x20, 0x13, 0x09, 0x02, 0x7D, 0x63, 0xF3, 0x06, 0x12, 0x02, 0x75, 0x31, 0xF8, 0xB1, 0x3F,
+ 0x02, 0x8B, 0x63, 0xF3, 0x09, 0x02, 0x02, 0x83, 0x11, 0xF8, 0x01, 0x2C, 0x93, 0x08, 0x82, 0x7D,
+ 0x63, 0xF3, 0x86, 0x12, 0x82, 0x75, 0x8A, 0x78, 0xC3, 0x7D, 0x62, 0xF3, 0x03, 0x03, 0xC3, 0x75,
+ 0xB1, 0xF8, 0x03, 0x30, 0xB0, 0xF8, 0x1B, 0x20, 0x63, 0xF3, 0x08, 0x02, 0xA0, 0xF8, 0x1B, 0x20,
+ 0x09, 0x89, 0x41, 0x82, 0x70, 0x47, 0x10, 0xF8, 0x37, 0x1F, 0x41, 0xF0, 0x10, 0x01, 0x00, 0xF8,
+ 0x1F, 0x19, 0x30, 0xF8, 0x0C, 0x2C, 0xC1, 0x7E, 0x6F, 0xF3, 0x0B, 0x02, 0x10, 0x32, 0x20, 0xF8,
+ 0x0C, 0x2C, 0x30, 0xF8, 0x08, 0x2C, 0x21, 0xF0, 0x23, 0x01, 0x6F, 0xF3, 0x0B, 0x02, 0x40, 0x32,
+ 0x20, 0xF8, 0x08, 0x2C, 0x08, 0x22, 0x00, 0xF8, 0x16, 0x2C, 0x90, 0xF8, 0x47, 0x20, 0x41, 0xF0,
+ 0x0C, 0x01, 0x22, 0xF0, 0x1F, 0x02, 0x80, 0xF8, 0x47, 0x20, 0x02, 0x8B, 0x6F, 0xF3, 0x08, 0x02,
+ 0x02, 0x83, 0xC1, 0x76, 0xC1, 0x8A, 0x6F, 0xF3, 0x0B, 0x01, 0x3F, 0x31, 0xC1, 0x82, 0x41, 0x7D,
+ 0x21, 0xF0, 0x1F, 0x01, 0x41, 0x75, 0x01, 0x7D, 0x21, 0xF0, 0x1F, 0x01, 0x01, 0x75, 0x90, 0xF8,
+ 0x46, 0x10, 0x21, 0xF0, 0x01, 0x01, 0x80, 0xF8, 0x46, 0x10, 0x01, 0x7C, 0x41, 0xF0, 0x20, 0x01,
+ 0x01, 0x74, 0x70, 0x47, 0x20, 0x07, 0x10, 0x00, 0x00, 0xF0, 0xFF, 0xFF, 0xB4, 0x59, 0x01, 0x20,
+ 0xB4, 0x07, 0x10, 0x00, 0xB4, 0x5B, 0x01, 0x20, 0x08, 0x84, 0x10, 0x00, 0xF8, 0x5B, 0x01, 0x20,
+ 0x70, 0x47, 0x10, 0xB5, 0x00, 0xF0, 0x93, 0xFE, 0x08, 0xB1, 0x01, 0x20, 0x10, 0xBD, 0x08, 0x48,
+ 0x00, 0xF0, 0x76, 0xFE, 0x00, 0x20, 0x10, 0xBD, 0x06, 0x48, 0x10, 0xB5, 0x00, 0x68, 0x90, 0xF8,
+ 0x70, 0x00, 0x40, 0x07, 0x01, 0xD5, 0x00, 0xF0, 0x77, 0xFE, 0x01, 0x20, 0x10, 0xBD, 0x00, 0x00,
+ 0x47, 0x3E, 0x01, 0x00, 0x20, 0x07, 0x10, 0x00, 0x7B, 0x48, 0x00, 0x21, 0x01, 0x70, 0x41, 0x70,
+ 0x81, 0x70, 0xC1, 0x70, 0x01, 0x71, 0x70, 0x47, 0x70, 0xB5, 0x0D, 0x20, 0x00, 0xF0, 0xD5, 0xF8,
+ 0x0D, 0x20, 0x00, 0xF0, 0xDD, 0xF8, 0x0C, 0x20, 0x00, 0xF0, 0xCF, 0xF8, 0x0C, 0x20, 0x00, 0xF0,
+ 0xD7, 0xF8, 0x71, 0x4C, 0x71, 0x48, 0xA5, 0x78, 0xC0, 0x7A, 0xA0, 0x70, 0xEC, 0xF7, 0xC6, 0xF8,
+ 0xA2, 0x78, 0x01, 0x21, 0x00, 0x20, 0xA2, 0xB1, 0x23, 0x78, 0xAB, 0xB1, 0x95, 0x42, 0x20, 0xD1,
+ 0x01, 0x2A, 0x1E, 0xD0, 0x02, 0x2A, 0xE2, 0x78, 0x1E, 0xD0, 0x02, 0xB1, 0xE0, 0x70, 0x22, 0x79,
+ 0x52, 0x1C, 0xD2, 0xB2, 0x22, 0x71, 0x04, 0x2A, 0x1E, 0xD3, 0x20, 0x71, 0xE1, 0x70, 0x04, 0x20,
+ 0x1B, 0xE0, 0x21, 0x70, 0xE0, 0x70, 0x02, 0xE0, 0x02, 0x22, 0x22, 0x70, 0xE1, 0x70, 0x20, 0x71,
+ 0xEC, 0xF7, 0xA2, 0xF8, 0xE0, 0x78, 0x60, 0x70, 0xBD, 0xE8, 0x70, 0x40, 0x5B, 0x48, 0xFF, 0xF7,
+ 0x6B, 0xBD, 0xE1, 0x70, 0x20, 0x71, 0xEA, 0xE7, 0x0A, 0xB1, 0xE0, 0x70, 0x00, 0xE0, 0xE1, 0x70,
+ 0x20, 0x71, 0xE0, 0x78, 0x00, 0x28, 0xE2, 0xD1, 0x05, 0x20, 0x20, 0x70, 0xBD, 0xE8, 0x70, 0x40,
+ 0xEC, 0xF7, 0x8A, 0xB8, 0x70, 0xB5, 0x50, 0x4C, 0x20, 0x78, 0x04, 0x28, 0x06, 0xD0, 0x20, 0x78,
+ 0x00, 0x25, 0x05, 0x28, 0x07, 0xD0, 0x25, 0x70, 0x65, 0x70, 0x70, 0xBD, 0x02, 0x20, 0x20, 0x70,
+ 0x01, 0x20, 0x60, 0x70, 0x0C, 0xE0, 0x4A, 0x48, 0xA1, 0x78, 0x00, 0x68, 0x02, 0x29, 0x0C, 0xD0,
+ 0x90, 0xF8, 0xBE, 0x00, 0x47, 0x49, 0x01, 0xF0, 0x0F, 0xFF, 0x03, 0x20, 0x20, 0x70, 0x65, 0x70,
+ 0xBD, 0xE8, 0x70, 0x40, 0x41, 0x48, 0xFF, 0xF7, 0x37, 0xBD, 0x90, 0xF8, 0xBD, 0x00, 0xF1, 0xE7,
+ 0x10, 0xB5, 0x0D, 0x20, 0x00, 0xF0, 0x61, 0xF8, 0x0C, 0x20, 0x00, 0xF0, 0x5E, 0xF8, 0x3A, 0x4C,
+ 0x3B, 0x49, 0xA0, 0x78, 0x03, 0x28, 0x04, 0xD0, 0x02, 0x28, 0x06, 0xD0, 0x01, 0x28, 0x08, 0xD0,
+ 0x0D, 0xE0, 0x08, 0x68, 0x90, 0xF8, 0xBE, 0x00, 0x06, 0xE0, 0x08, 0x68, 0x90, 0xF8, 0xBC, 0x00,
+ 0x02, 0xE0, 0x08, 0x68, 0x90, 0xF8, 0xBB, 0x00, 0x32, 0x49, 0x01, 0xF0, 0xE5, 0xFE, 0x03, 0x20,
+ 0x20, 0x70, 0x10, 0xBD, 0x2D, 0xE9, 0xF0, 0x41, 0x2D, 0x4C, 0x07, 0x46, 0x2A, 0x4D, 0x20, 0x68,
+ 0x4F, 0xF0, 0x00, 0x06, 0x90, 0xF8, 0xB0, 0x10, 0xC9, 0x07, 0x0E, 0xD0, 0x69, 0x78, 0x01, 0x29,
+ 0x0E, 0xD0, 0xA7, 0xF8, 0x4E, 0x60, 0x90, 0xF8, 0xB0, 0x00, 0xC0, 0xF3, 0xC1, 0x00, 0x02, 0x28,
+ 0x1E, 0xD0, 0x00, 0x21, 0x38, 0x46, 0x00, 0xF0, 0x4F, 0xFB, 0x6E, 0x70, 0xBD, 0xE8, 0xF0, 0x81,
+ 0x0D, 0x20, 0x00, 0xF0, 0x2D, 0xF8, 0x0C, 0x20, 0x00, 0xF0, 0x2A, 0xF8, 0x20, 0x68, 0x90, 0xF8,
+ 0xB0, 0x00, 0xC0, 0xF3, 0xC1, 0x00, 0x02, 0x28, 0x08, 0xD0, 0x0D, 0x20, 0x00, 0xF0, 0x0A, 0xF8,
+ 0x20, 0x68, 0x90, 0xF8, 0xB0, 0x00, 0xC0, 0xF3, 0xC1, 0x01, 0xE3, 0xE7, 0x0C, 0x20, 0xF5, 0xE7,
+ 0x02, 0x21, 0xDF, 0xE7, 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21, 0x91, 0x40, 0x40, 0x09, 0x80, 0x00,
+ 0x00, 0xF1, 0xE0, 0x20, 0xC0, 0xF8, 0x00, 0x11, 0x70, 0x47, 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21,
+ 0x91, 0x40, 0x40, 0x09, 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20, 0xC0, 0xF8, 0x80, 0x11, 0x70, 0x47,
+ 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21, 0x91, 0x40, 0x40, 0x09, 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20,
+ 0xC0, 0xF8, 0x80, 0x12, 0x70, 0x47, 0x00, 0x00, 0xC8, 0x07, 0x10, 0x00, 0xC4, 0x83, 0x10, 0x00,
+ 0x20, 0x07, 0x10, 0x00, 0x95, 0x42, 0x01, 0x00, 0x46, 0x48, 0x01, 0x68, 0x21, 0xF0, 0x04, 0x01,
+ 0x01, 0x60, 0x00, 0xF0, 0xFB, 0xBE, 0x10, 0xB5, 0xFF, 0xF7, 0xF6, 0xFF, 0x10, 0xBD, 0x10, 0xB5,
+ 0xFF, 0xF7, 0xF2, 0xFF, 0x10, 0xBD, 0x10, 0xB5, 0xFF, 0xF7, 0xEE, 0xFF, 0x10, 0xBD, 0x10, 0xB5,
+ 0xFF, 0xF7, 0xEA, 0xFF, 0xBD, 0xE8, 0x10, 0x40, 0x01, 0xF0, 0x57, 0xBC, 0x10, 0xB5, 0xFF, 0xF7,
+ 0xE3, 0xFF, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0xF0, 0xDC, 0xBA, 0x10, 0xB5, 0xFF, 0xF7, 0xDC, 0xFF,
+ 0xBD, 0xE8, 0x10, 0x40, 0x01, 0xF0, 0x5F, 0xBD, 0x10, 0xB5, 0xFF, 0xF7, 0xD5, 0xFF, 0xBD, 0xE8,
+ 0x10, 0x40, 0x01, 0xF0, 0xC3, 0xBD, 0x10, 0xB5, 0xFF, 0xF7, 0xCE, 0xFF, 0xBD, 0xE8, 0x10, 0x40,
+ 0x01, 0xF0, 0x27, 0xBE, 0x10, 0xB5, 0xFF, 0xF7, 0xC7, 0xFF, 0xBD, 0xE8, 0x10, 0x40, 0x01, 0xF0,
+ 0x8D, 0xBE, 0x10, 0xB5, 0xFF, 0xF7, 0xC0, 0xFF, 0xBD, 0xE8, 0x10, 0x40, 0x01, 0xF0, 0x05, 0xBF,
+ 0x10, 0xB5, 0xFF, 0xF7, 0xB9, 0xFF, 0xBD, 0xE8, 0x10, 0x40, 0x01, 0xF0, 0x6B, 0xBF, 0x10, 0xB5,
+ 0xFF, 0xF7, 0xB2, 0xFF, 0xBD, 0xE8, 0x10, 0x40, 0x01, 0xF0, 0xD1, 0xBF, 0x10, 0xB5, 0xFF, 0xF7,
+ 0xAB, 0xFF, 0xBD, 0xE8, 0x10, 0x40, 0x02, 0xF0, 0x95, 0xBA, 0x10, 0xB5, 0xFF, 0xF7, 0xA4, 0xFF,
+ 0xBD, 0xE8, 0x10, 0x40, 0x02, 0xF0, 0x6A, 0xBA, 0x10, 0xB5, 0xFF, 0xF7, 0x9D, 0xFF, 0xBD, 0xE8,
+ 0x10, 0x40, 0xFF, 0xF7, 0x15, 0xBF, 0x10, 0xB5, 0xFF, 0xF7, 0x96, 0xFF, 0xBD, 0xE8, 0x10, 0x40,
+ 0xFF, 0xF7, 0x0E, 0xBF, 0x10, 0xB5, 0xFF, 0xF7, 0x8F, 0xFF, 0x10, 0xBD, 0x10, 0xB5, 0xFF, 0xF7,
+ 0x8B, 0xFF, 0xBD, 0xE8, 0x10, 0x40, 0x01, 0xF0, 0xEE, 0xBF, 0x10, 0xB5, 0xFF, 0xF7, 0x84, 0xFF,
+ 0x10, 0xBD, 0x10, 0xB5, 0xFF, 0xF7, 0x80, 0xFF, 0x10, 0xBD, 0x10, 0xB5, 0xFF, 0xF7, 0x7C, 0xFF,
+ 0x10, 0xBD, 0x10, 0xB5, 0xFF, 0xF7, 0x78, 0xFF, 0x10, 0xBD, 0x10, 0xB5, 0xFF, 0xF7, 0x74, 0xFF,
+ 0x10, 0xBD, 0x70, 0x47, 0x10, 0xED, 0x00, 0xE0, 0x0E, 0xB5, 0x00, 0x20, 0x00, 0x90, 0x01, 0x90,
+ 0x02, 0x90, 0x09, 0x48, 0x00, 0xF0, 0x13, 0xFE, 0x68, 0x46, 0x00, 0xF0, 0x14, 0xFE, 0xFC, 0xF7,
+ 0x38, 0xFD, 0xFC, 0xF7, 0x39, 0xFB, 0x00, 0xF0, 0x31, 0xF8, 0x00, 0x21, 0x47, 0x20, 0xFB, 0xF7,
+ 0x96, 0xFE, 0x01, 0x20, 0x0E, 0xBD, 0x00, 0x00, 0x10, 0x00, 0x32, 0x02, 0x1A, 0x49, 0x10, 0xB5,
+ 0x01, 0x20, 0x08, 0x70, 0x19, 0x49, 0x00, 0x20, 0x08, 0x60, 0x19, 0x49, 0x08, 0x60, 0x19, 0x49,
+ 0x08, 0x60, 0x19, 0x49, 0x08, 0x60, 0x00, 0xF0, 0x7F, 0xF9, 0x00, 0xF0, 0x19, 0xF9, 0x00, 0xF0,
+ 0x7D, 0xF8, 0x00, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x12, 0x48, 0x02, 0xF0, 0x31, 0xF8, 0x10, 0x49,
+ 0x08, 0x60, 0x0E, 0x49, 0x09, 0x68, 0x88, 0x42, 0x07, 0xD0, 0x0B, 0x48, 0x00, 0x78, 0x02, 0x28,
+ 0x03, 0xD1, 0xBD, 0xE8, 0x10, 0x40, 0xEB, 0xF7, 0xC9, 0xBE, 0x10, 0xBD, 0x10, 0xB5, 0xFF, 0xF7,
+ 0xEA, 0xFF, 0x05, 0x49, 0x02, 0x20, 0x08, 0x70, 0xEB, 0xF7, 0xA4, 0xFE, 0x4F, 0xF0, 0xFF, 0x30,
+ 0x10, 0xBD, 0x01, 0x20, 0x70, 0x47, 0x00, 0x00, 0x18, 0x08, 0x10, 0x00, 0x1C, 0x08, 0x10, 0x00,
+ 0x20, 0x08, 0x10, 0x00, 0x24, 0x08, 0x10, 0x00, 0x28, 0x08, 0x10, 0x00, 0x70, 0xB5, 0x0D, 0x46,
+ 0x04, 0x46, 0xC1, 0x69, 0x80, 0x68, 0xCC, 0x22, 0x02, 0xF0, 0x73, 0xF9, 0xE1, 0x69, 0xA0, 0x68,
+ 0x00, 0x22, 0x08, 0x44, 0x4F, 0xF0, 0x80, 0x71, 0x40, 0xF8, 0x04, 0x1C, 0x21, 0x6A, 0x40, 0xE9,
+ 0x03, 0x21, 0x40, 0xF8, 0x10, 0x2C, 0x40, 0xF8, 0x14, 0x2C, 0x40, 0xF8, 0x18, 0x2C, 0x40, 0xF8,
+ 0x1C, 0x2C, 0x4F, 0xF0, 0x0B, 0x31, 0x40, 0xE9, 0x09, 0x15, 0x4F, 0xF0, 0x0A, 0x31, 0x40, 0xF8,
+ 0x28, 0x1C, 0x4F, 0xF0, 0x09, 0x31, 0x40, 0xF8, 0x2C, 0x1C, 0x4F, 0xF0, 0x08, 0x31, 0x40, 0xF8,
+ 0x30, 0x1C, 0x4F, 0xF0, 0x07, 0x31, 0x40, 0xF8, 0x34, 0x1C, 0x4F, 0xF0, 0x06, 0x31, 0x40, 0xF8,
+ 0x38, 0x1C, 0x4F, 0xF0, 0x05, 0x31, 0x40, 0xF8, 0x3C, 0x1C, 0x4F, 0xF0, 0x04, 0x31, 0x40, 0xF8,
+ 0x40, 0x1D, 0x70, 0xBD, 0x81, 0x69, 0x4F, 0xF0, 0xCC, 0x33, 0x0A, 0x68, 0x9A, 0x42, 0x08, 0xD1,
+ 0x4A, 0x68, 0x9A, 0x42, 0x05, 0xD1, 0x8A, 0x68, 0x9A, 0x42, 0x02, 0xD1, 0xC9, 0x68, 0x99, 0x42,
+ 0x02, 0xD0, 0x40, 0x68, 0xFC, 0xF7, 0xE7, 0xBA, 0x70, 0x47, 0x00, 0x00, 0x10, 0xB5, 0xA0, 0x21,
+ 0x49, 0x48, 0x02, 0xF0, 0x50, 0xF9, 0x00, 0x20, 0x10, 0xBD, 0x70, 0xB5, 0x06, 0x46, 0x46, 0x48,
+ 0x00, 0xEB, 0x06, 0x14, 0xEB, 0xF7, 0x8A, 0xFE, 0x05, 0x46, 0x20, 0x78, 0x28, 0xB1, 0x28, 0x46,
+ 0xEB, 0xF7, 0x8C, 0xFE, 0x6F, 0xF0, 0x01, 0x00, 0x70, 0xBD, 0x03, 0x20, 0x20, 0x70, 0x00, 0x20,
+ 0x66, 0x60, 0xE0, 0x60, 0x04, 0xF1, 0x08, 0x00, 0x01, 0xF0, 0x74, 0xFF, 0x28, 0x46, 0xEB, 0xF7,
+ 0x7D, 0xFE, 0x00, 0x20, 0x70, 0xBD, 0x2D, 0xE9, 0xF0, 0x41, 0x0F, 0x46, 0x36, 0x49, 0x00, 0x25,
+ 0x01, 0xEB, 0x00, 0x14, 0x2E, 0x46, 0xEB, 0xF7, 0x69, 0xFE, 0x80, 0x46, 0xE0, 0x68, 0x38, 0x43,
+ 0xE0, 0x60, 0x04, 0xF1, 0x08, 0x00, 0x07, 0x46, 0x01, 0xF0, 0x82, 0xFF, 0x0C, 0xE0, 0xE1, 0x68,
+ 0xC2, 0x6A, 0x11, 0x42, 0x03, 0xD0, 0x00, 0xF0, 0x0F, 0xF9, 0x01, 0x26, 0x00, 0xE0, 0x6D, 0x1C,
+ 0x29, 0x46, 0x38, 0x46, 0x01, 0xF0, 0x6A, 0xFF, 0x00, 0x28, 0xF0, 0xD1, 0x0E, 0xB1, 0xFF, 0xF7,
+ 0x3A, 0xFF, 0x40, 0x46, 0xEB, 0xF7, 0x52, 0xFE, 0x00, 0x20, 0xBD, 0xE8, 0xF0, 0x81, 0x70, 0xB5,
+ 0x0D, 0x46, 0x21, 0x49, 0x01, 0xEB, 0x00, 0x14, 0xEB, 0xF7, 0x40, 0xFE, 0xE1, 0x68, 0xA9, 0x43,
+ 0xE1, 0x60, 0xEB, 0xF7, 0x43, 0xFE, 0x00, 0x20, 0x70, 0xBD, 0x70, 0xB5, 0x0C, 0x46, 0x1A, 0x49,
+ 0x01, 0xEB, 0x00, 0x15, 0x00, 0x20, 0x20, 0x60, 0xEB, 0xF7, 0x30, 0xFE, 0xE9, 0x68, 0x21, 0x60,
+ 0xEB, 0xF7, 0x34, 0xFE, 0x00, 0x20, 0x70, 0xBD, 0x2D, 0xE9, 0xF0, 0x41, 0x0D, 0x46, 0x12, 0x49,
+ 0x1E, 0x46, 0x90, 0x46, 0x01, 0xEB, 0x00, 0x14, 0xEB, 0xF7, 0x20, 0xFE, 0x07, 0x46, 0x0F, 0x48,
+ 0x00, 0x68, 0xC5, 0x62, 0xE1, 0x68, 0x29, 0x42, 0x03, 0xD1, 0x04, 0xF1, 0x08, 0x01, 0x00, 0xF0,
+ 0xDD, 0xF8, 0x38, 0x46, 0xEB, 0xF7, 0x1A, 0xFE, 0xEB, 0xF7, 0x10, 0xFE, 0xE1, 0x68, 0x29, 0x40,
+ 0x31, 0x60, 0xB8, 0xF1, 0x01, 0x0F, 0x02, 0xD1, 0xE1, 0x68, 0xA9, 0x43, 0xE1, 0x60, 0xEB, 0xF7,
+ 0x0D, 0xFE, 0x00, 0x20, 0xB9, 0xE7, 0x00, 0x00, 0xB4, 0x84, 0x10, 0x00, 0x1C, 0x08, 0x10, 0x00,
+ 0x10, 0xB5, 0xC8, 0x21, 0x2E, 0x48, 0x02, 0xF0, 0xB6, 0xF8, 0x00, 0x20, 0x10, 0xBD, 0x2D, 0xE9,
+ 0xF0, 0x41, 0x06, 0x46, 0x2A, 0x49, 0x00, 0xEB, 0x86, 0x00, 0x17, 0x46, 0x01, 0xEB, 0x80, 0x04,
+ 0xEB, 0xF7, 0xEC, 0xFD, 0x05, 0x46, 0x20, 0x78, 0x30, 0xB1, 0x28, 0x46, 0xEB, 0xF7, 0xEE, 0xFD,
+ 0x6F, 0xF0, 0x01, 0x00, 0xBD, 0xE8, 0xF0, 0x81, 0x02, 0x20, 0x20, 0x70, 0x00, 0x20, 0x66, 0x60,
+ 0xC4, 0xE9, 0x02, 0x70, 0x04, 0xF1, 0x10, 0x00, 0x01, 0xF0, 0xD4, 0xFE, 0x28, 0x46, 0xEB, 0xF7,
+ 0xDD, 0xFD, 0x00, 0x20, 0xEE, 0xE7, 0x70, 0xB5, 0x19, 0x49, 0x00, 0xEB, 0x80, 0x00, 0x01, 0xEB,
+ 0x80, 0x04, 0xEB, 0xF7, 0xCB, 0xFD, 0x05, 0x46, 0xD4, 0xE9, 0x02, 0x01, 0x81, 0x42, 0x05, 0xDB,
+ 0x14, 0x48, 0x04, 0xF1, 0x10, 0x01, 0x00, 0x68, 0x00, 0xF0, 0x88, 0xF8, 0xE0, 0x68, 0x40, 0x1C,
+ 0xE0, 0x60, 0x28, 0x46, 0xEB, 0xF7, 0xC2, 0xFD, 0x00, 0x20, 0x70, 0xBD, 0x70, 0xB5, 0x0C, 0x49,
+ 0x00, 0xEB, 0x80, 0x00, 0x01, 0xEB, 0x80, 0x04, 0xEB, 0xF7, 0xB0, 0xFD, 0x05, 0x46, 0xE0, 0x68,
+ 0x08, 0xB1, 0x40, 0x1E, 0xE0, 0x60, 0x04, 0xF1, 0x10, 0x00, 0x01, 0xF0, 0xC9, 0xFE, 0x10, 0xB1,
+ 0x20, 0x69, 0x00, 0xF0, 0x11, 0xF8, 0x28, 0x46, 0xEB, 0xF7, 0xA8, 0xFD, 0x00, 0x20, 0x70, 0xBD,
+ 0x54, 0x85, 0x10, 0x00, 0x1C, 0x08, 0x10, 0x00, 0x10, 0xB5, 0x4F, 0xF4, 0xF0, 0x71, 0x3C, 0x48,
+ 0x02, 0xF0, 0x51, 0xF8, 0x00, 0x20, 0x10, 0xBD, 0x10, 0xB5, 0x04, 0x46, 0x90, 0xF8, 0x24, 0x00,
+ 0x02, 0x28, 0x0B, 0xD0, 0x20, 0x46, 0x01, 0xF0, 0xAD, 0xFE, 0x02, 0x20, 0x84, 0xF8, 0x24, 0x00,
+ 0x34, 0x49, 0x20, 0x46, 0x01, 0xF0, 0x82, 0xFE, 0xFF, 0xF7, 0x6D, 0xFE, 0x00, 0x20, 0x10, 0xBD,
+ 0x2D, 0xE9, 0xF0, 0x5F, 0x06, 0x46, 0x0A, 0xA8, 0x06, 0xEB, 0x46, 0x01, 0x90, 0xE8, 0x00, 0x0E,
+ 0x2B, 0x48, 0x1D, 0x46, 0x00, 0xEB, 0x01, 0x14, 0x90, 0x46, 0x20, 0x78, 0x18, 0xB1, 0x6F, 0xF0,
+ 0x01, 0x00, 0xBD, 0xE8, 0xF0, 0x9F, 0xEB, 0xF7, 0x69, 0xFD, 0x07, 0x46, 0x01, 0x20, 0x20, 0x70,
+ 0xC4, 0xF8, 0x14, 0x80, 0xC4, 0xE9, 0x01, 0x65, 0x04, 0xF1, 0x18, 0x01, 0x00, 0x20, 0x81, 0xE8,
+ 0x20, 0x06, 0x84, 0xF8, 0x24, 0x00, 0xE0, 0x60, 0x20, 0x61, 0xC4, 0xE9, 0x0A, 0x80, 0x59, 0x46,
+ 0x20, 0x46, 0xFF, 0xF7, 0x6B, 0xFE, 0xA0, 0x60, 0x20, 0x46, 0xFF, 0xF7, 0xBD, 0xFF, 0x38, 0x46,
+ 0xEB, 0xF7, 0x54, 0xFD, 0x00, 0x20, 0xDC, 0xE7, 0x10, 0xB5, 0x04, 0x46, 0x90, 0xF8, 0x24, 0x00,
+ 0x02, 0x28, 0x09, 0xD0, 0x20, 0x46, 0x01, 0xF0, 0x65, 0xFE, 0x02, 0x20, 0x84, 0xF8, 0x24, 0x00,
+ 0x10, 0x49, 0x20, 0x46, 0x01, 0xF0, 0x3A, 0xFE, 0x00, 0x20, 0x10, 0xBD, 0x70, 0xB5, 0x04, 0x46,
+ 0x90, 0xF8, 0x24, 0x00, 0x0D, 0x46, 0x02, 0x28, 0x0B, 0xD1, 0x20, 0x46, 0x01, 0xF0, 0x52, 0xFE,
+ 0x03, 0x20, 0x84, 0xF8, 0x24, 0x00, 0x29, 0x46, 0x20, 0x46, 0x01, 0xF0, 0x27, 0xFE, 0xFF, 0xF7,
+ 0x12, 0xFE, 0x00, 0x20, 0x70, 0xBD, 0x04, 0x48, 0x00, 0x68, 0x40, 0x68, 0x70, 0x47, 0x00, 0x00,
+ 0x1C, 0x86, 0x10, 0x00, 0x24, 0x08, 0x10, 0x00, 0x1C, 0x08, 0x10, 0x00, 0x6D, 0x48, 0x00, 0x21,
+ 0x01, 0x60, 0x01, 0x22, 0x02, 0x60, 0x01, 0x60, 0x70, 0x47, 0x10, 0xB5, 0xFF, 0xF7, 0xF6, 0xFF,
+ 0x69, 0x48, 0x00, 0x21, 0x01, 0x70, 0x41, 0x60, 0x00, 0xF0, 0xF5, 0xF9, 0xBD, 0xE8, 0x10, 0x40,
+ 0x00, 0xF0, 0xD7, 0xBA, 0x01, 0xF0, 0xBF, 0xBC, 0x01, 0xF0, 0x2A, 0xBD, 0x10, 0xB5, 0x01, 0xF0,
+ 0x27, 0xFD, 0x01, 0xF0, 0xB8, 0xFC, 0xBD, 0xE8, 0x10, 0x40, 0xE6, 0xE7, 0xA4, 0x22, 0x5F, 0x49,
+ 0x01, 0xF0, 0x10, 0xBF, 0x70, 0x47, 0x10, 0xB5, 0x04, 0x46, 0x80, 0x07, 0x02, 0xD0, 0x02, 0x20,
+ 0xFF, 0xF7, 0xB5, 0xF9, 0x4F, 0xF4, 0x80, 0x30, 0x20, 0x44, 0xC0, 0xF3, 0x4F, 0x00, 0x10, 0xBD,
+ 0x01, 0x46, 0x98, 0x22, 0x56, 0x48, 0x01, 0xF0, 0xFD, 0xBE, 0x55, 0x4A, 0x00, 0x21, 0x42, 0xF8,
+ 0x21, 0x00, 0x49, 0x1C, 0x26, 0x29, 0xFA, 0xDB, 0x70, 0x47, 0x01, 0xF0, 0xDE, 0xBC, 0x01, 0xF0,
+ 0x5C, 0xBC, 0x4D, 0x48, 0x10, 0xB5, 0x40, 0x68, 0x10, 0xB1, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0x47,
+ 0x00, 0x20, 0xFF, 0xF7, 0x94, 0xF9, 0x10, 0xBD, 0x06, 0x29, 0x37, 0xD2, 0xDF, 0xE8, 0x01, 0xF0,
+ 0x11, 0x24, 0x37, 0x24, 0x36, 0x03, 0x01, 0x78, 0x21, 0xF0, 0x12, 0x01, 0x00, 0xF8, 0x4D, 0x1B,
+ 0x01, 0x78, 0x21, 0xF0, 0x03, 0x01, 0x00, 0xF8, 0x07, 0x1B, 0x01, 0x78, 0x21, 0xF0, 0x80, 0x01,
+ 0x0C, 0xE0, 0x01, 0x78, 0x21, 0xF0, 0x02, 0x01, 0x41, 0xF0, 0x10, 0x01, 0x00, 0xF8, 0x4D, 0x1B,
+ 0x01, 0x78, 0x21, 0xF0, 0x03, 0x01, 0x00, 0xF8, 0x07, 0x1B, 0x12, 0xE0, 0x00, 0xF8, 0x44, 0x1B,
+ 0x01, 0x78, 0x21, 0xF0, 0x01, 0x01, 0x10, 0xE0, 0x02, 0x78, 0x01, 0x29, 0x42, 0xF0, 0x12, 0x02,
+ 0x02, 0x70, 0x00, 0xD0, 0x03, 0x21, 0x10, 0xF8, 0x4D, 0x2F, 0x61, 0xF3, 0x01, 0x02, 0x00, 0xF8,
+ 0x07, 0x2B, 0x01, 0x78, 0x41, 0xF0, 0x80, 0x01, 0xE8, 0xE7, 0x01, 0x70, 0x70, 0x47, 0x01, 0x78,
+ 0x21, 0xF0, 0x02, 0x01, 0x41, 0xF0, 0x10, 0x01, 0x00, 0xF8, 0x4D, 0x1B, 0x01, 0x78, 0x21, 0xF0,
+ 0x03, 0x01, 0x89, 0x1C, 0x00, 0xF8, 0x07, 0x1B, 0x01, 0x78, 0x41, 0xF0, 0x80, 0x01, 0x00, 0xF8,
+ 0x44, 0x1B, 0x01, 0x78, 0x41, 0xF0, 0x01, 0x01, 0xE7, 0xE7, 0x90, 0xF8, 0x29, 0x20, 0x02, 0x29,
+ 0x04, 0xD0, 0x03, 0x29, 0x22, 0xF0, 0x40, 0x01, 0x02, 0xD1, 0x04, 0xE0, 0x42, 0xF0, 0x40, 0x01,
+ 0x21, 0xF0, 0x80, 0x01, 0x01, 0xE0, 0x41, 0xF0, 0x80, 0x01, 0x80, 0xF8, 0x29, 0x10, 0x70, 0x47,
+ 0x90, 0xF8, 0x29, 0x20, 0x01, 0x29, 0x04, 0xD0, 0x02, 0x29, 0x07, 0xD0, 0x22, 0xF0, 0x10, 0x01,
+ 0x01, 0xE0, 0x42, 0xF0, 0x10, 0x01, 0x21, 0xF0, 0x20, 0x01, 0x01, 0xE0, 0x42, 0xF0, 0x30, 0x01,
+ 0x80, 0xF8, 0x29, 0x10, 0x70, 0x47, 0xC2, 0x78, 0x00, 0x29, 0x61, 0xF3, 0x03, 0x02, 0xC2, 0x70,
+ 0x04, 0xD0, 0x10, 0xF8, 0x5E, 0x1F, 0x41, 0xF0, 0x04, 0x01, 0x01, 0x70, 0x70, 0x47, 0xC2, 0x7A,
+ 0x01, 0x29, 0x61, 0xF3, 0x03, 0x02, 0xC2, 0x72, 0x02, 0xD9, 0x42, 0xF0, 0x10, 0x01, 0xC1, 0x72,
+ 0x70, 0x47, 0x00, 0x00, 0x90, 0x04, 0x00, 0x22, 0xD0, 0x07, 0x10, 0x00, 0x68, 0x6F, 0x01, 0x00,
+ 0xA4, 0x00, 0x03, 0x20, 0x01, 0xEB, 0x81, 0x01, 0x02, 0xEB, 0xC1, 0x01, 0x40, 0x5C, 0x01, 0x06,
+ 0x02, 0xD5, 0x00, 0xF0, 0x7F, 0x00, 0x40, 0x42, 0x70, 0x47, 0x00, 0x2B, 0x04, 0xDA, 0x5B, 0x42,
+ 0xDB, 0xB2, 0x43, 0xF0, 0x80, 0x03, 0x01, 0xE0, 0x03, 0xF0, 0x7F, 0x03, 0x01, 0xEB, 0x81, 0x01,
+ 0x02, 0xEB, 0xC1, 0x01, 0x43, 0x54, 0x70, 0x47, 0x2D, 0xE9, 0xF0, 0x5F, 0x00, 0x25, 0xDD, 0xE9,
+ 0x0A, 0x78, 0x91, 0x46, 0x9A, 0x46, 0x83, 0x46, 0x2E, 0x46, 0x2A, 0x46, 0x01, 0x20, 0x00, 0x21,
+ 0x01, 0xF0, 0xF1, 0xFD, 0x00, 0xEA, 0x09, 0x00, 0x01, 0xEA, 0x0A, 0x01, 0x08, 0x43, 0x14, 0xD0,
+ 0x00, 0x24, 0x22, 0x46, 0x01, 0x20, 0x00, 0x21, 0x01, 0xF0, 0xE5, 0xFD, 0x38, 0x40, 0x01, 0xEA,
+ 0x08, 0x01, 0x08, 0x43, 0x05, 0xD0, 0x00, 0x23, 0x22, 0x46, 0x31, 0x46, 0x58, 0x46, 0xFF, 0xF7,
+ 0xCC, 0xFF, 0x64, 0x1C, 0x28, 0x2C, 0xEC, 0xDB, 0x76, 0x1C, 0x6D, 0x1C, 0x38, 0x2D, 0xDC, 0xDB,
+ 0xBD, 0xE8, 0xF0, 0x9F, 0x40, 0x5C, 0x01, 0x06, 0x02, 0xD5, 0x00, 0xF0, 0x7F, 0x00, 0x40, 0x42,
+ 0x70, 0x47, 0x00, 0x2A, 0x04, 0xDA, 0x52, 0x42, 0xD2, 0xB2, 0x42, 0xF0, 0x80, 0x02, 0x01, 0xE0,
+ 0x02, 0xF0, 0x7F, 0x02, 0x42, 0x54, 0x70, 0x47, 0xF0, 0xB5, 0x00, 0x24, 0x15, 0x46, 0x0E, 0x46,
+ 0x84, 0x46, 0x23, 0x46, 0x01, 0x27, 0x0F, 0xE0, 0x06, 0xEB, 0xE3, 0x00, 0x03, 0xF0, 0x07, 0x01,
+ 0x00, 0x78, 0x07, 0xFA, 0x01, 0xF2, 0x02, 0x42, 0x05, 0xD0, 0x00, 0x22, 0x19, 0x46, 0x60, 0x46,
+ 0xFF, 0xF7, 0xDF, 0xFF, 0x64, 0x1C, 0x5B, 0x1C, 0xAC, 0x42, 0xED, 0xDB, 0xF0, 0xBD, 0x40, 0x5C,
+ 0x70, 0x47, 0x42, 0x54, 0x70, 0x47, 0xF0, 0xB5, 0x00, 0x24, 0x23, 0x46, 0x01, 0x26, 0x0D, 0xE0,
+ 0x01, 0xEB, 0xE3, 0x05, 0x03, 0xF0, 0x07, 0x07, 0x2D, 0x78, 0x06, 0xFA, 0x07, 0xFC, 0x1C, 0xEA,
+ 0x05, 0x0F, 0x02, 0xD0, 0x00, 0x25, 0xC5, 0x54, 0x64, 0x1C, 0x5B, 0x1C, 0x94, 0x42, 0xEF, 0xDB,
+ 0xF0, 0xBD, 0x40, 0x5C, 0x70, 0x47, 0x42, 0x54, 0x70, 0x47, 0x40, 0x5C, 0x70, 0x47, 0x42, 0x54,
+ 0x70, 0x47, 0x00, 0x00, 0x2D, 0xE9, 0xF0, 0x47, 0x33, 0x4C, 0x54, 0xF8, 0x3C, 0x8F, 0x67, 0x68,
+ 0xE6, 0x68, 0x25, 0x7C, 0xDF, 0xF8, 0xC4, 0x90, 0x01, 0x21, 0xC9, 0xF8, 0x00, 0x10, 0x4F, 0xF4,
+ 0x20, 0x70, 0xEB, 0xF7, 0x9D, 0xFB, 0x00, 0x21, 0xC9, 0xF8, 0x00, 0x10, 0xC4, 0xF8, 0x00, 0x80,
+ 0x67, 0x60, 0xE6, 0x60, 0x25, 0x74, 0xBD, 0xE8, 0xF0, 0x87, 0x27, 0x48, 0x00, 0x21, 0xC1, 0x63,
+ 0x01, 0x64, 0x81, 0x64, 0x80, 0xF8, 0x4C, 0x10, 0x70, 0x47, 0x70, 0xB5, 0x00, 0x24, 0x0D, 0x46,
+ 0x02, 0x46, 0x01, 0x20, 0x21, 0x46, 0x01, 0xF0, 0x56, 0xFD, 0x35, 0xB3, 0x1E, 0x4A, 0x48, 0x32,
+ 0x10, 0x60, 0x11, 0x71, 0xFF, 0xF7, 0xCE, 0xFF, 0x00, 0x20, 0x00, 0xF0, 0x23, 0xF8, 0x00, 0xB1,
+ 0x01, 0x24, 0xFF, 0xF7, 0xC7, 0xFF, 0x01, 0x20, 0x00, 0xF0, 0x1C, 0xF8, 0x08, 0xB1, 0x44, 0xF0,
+ 0x02, 0x04, 0xFF, 0xF7, 0xBF, 0xFF, 0x02, 0x20, 0x00, 0xF0, 0x14, 0xF8, 0x08, 0xB1, 0x44, 0xF0,
+ 0x04, 0x04, 0xFF, 0xF7, 0xB7, 0xFF, 0x03, 0x20, 0x00, 0xF0, 0x0C, 0xF8, 0x08, 0xB1, 0x44, 0xF0,
+ 0x08, 0x04, 0xFF, 0xF7, 0xAF, 0xFF, 0x20, 0x46, 0x70, 0xBD, 0x0B, 0x4A, 0x3C, 0x32, 0xC2, 0xE9,
+ 0x00, 0x01, 0xD7, 0xE7, 0x40, 0xF0, 0x20, 0x01, 0x07, 0x48, 0x10, 0xB5, 0x80, 0xF8, 0xA3, 0x10,
+ 0xE5, 0x21, 0x81, 0x71, 0x4F, 0xF4, 0x20, 0x70, 0xEB, 0xF7, 0x4A, 0xFB, 0x04, 0x48, 0xD0, 0xF8,
+ 0xC0, 0x02, 0xC0, 0xB2, 0x10, 0xBD, 0x00, 0x00, 0x00, 0x00, 0x03, 0x20, 0x68, 0x14, 0x60, 0x22,
+ 0x00, 0x10, 0x60, 0x22, 0x4E, 0x48, 0x01, 0x68, 0x00, 0x29, 0x01, 0xD0, 0x01, 0x20, 0x08, 0x47,
+ 0x70, 0x47, 0x10, 0xB5, 0xFF, 0xF7, 0x20, 0xFE, 0x4A, 0x49, 0x02, 0x20, 0x08, 0x70, 0x4A, 0x49,
+ 0x00, 0x20, 0x08, 0x60, 0x46, 0x48, 0x01, 0x68, 0x00, 0x29, 0x03, 0xD0, 0xBD, 0xE8, 0x10, 0x40,
+ 0x00, 0x20, 0x08, 0x47, 0x10, 0xBD, 0x42, 0x49, 0x00, 0x20, 0x08, 0x60, 0x70, 0x47, 0x7F, 0xB5,
+ 0x04, 0x46, 0x03, 0x99, 0x08, 0x9D, 0xC1, 0xF3, 0x14, 0x01, 0x03, 0x91, 0x08, 0x21, 0x02, 0xA8,
+ 0x00, 0xF0, 0x60, 0xFA, 0x01, 0x2D, 0x02, 0xD0, 0x02, 0x2D, 0x04, 0xD0, 0x06, 0xE0, 0xC0, 0x1C,
+ 0x20, 0xF0, 0x01, 0x00, 0x02, 0xE0, 0xC0, 0x1D, 0x20, 0xF0, 0x03, 0x00, 0xDD, 0xE9, 0x02, 0x12,
+ 0x44, 0xF8, 0x3C, 0x1F, 0x62, 0x60, 0x21, 0x7A, 0x60, 0xF3, 0x06, 0x01, 0x21, 0x72, 0x7F, 0xBD,
+ 0x01, 0x23, 0x40, 0xF8, 0x3C, 0x3F, 0x00, 0x23, 0x43, 0x60, 0x51, 0x43, 0x02, 0x7A, 0x61, 0xF3,
+ 0x06, 0x02, 0x02, 0x72, 0x70, 0x47, 0x1F, 0xB5, 0x04, 0x46, 0x03, 0x99, 0x02, 0xA8, 0x01, 0xF0,
+ 0x3F, 0x01, 0x03, 0x91, 0x08, 0x21, 0x00, 0xF0, 0x35, 0xFA, 0xDD, 0xE9, 0x02, 0x12, 0xA1, 0x64,
+ 0x84, 0xF8, 0x4C, 0x20, 0x14, 0xF8, 0x70, 0x1F, 0x60, 0xF3, 0x05, 0x01, 0x04, 0xF8, 0x1E, 0x19,
+ 0xE1, 0x7F, 0x60, 0xF3, 0x05, 0x01, 0xE1, 0x77, 0x1F, 0xBD, 0x08, 0xB5, 0x1F, 0xA2, 0x12, 0x68,
+ 0x00, 0x92, 0x1D, 0xF8, 0x01, 0x20, 0x10, 0xF8, 0x5D, 0x1F, 0x62, 0xF3, 0x01, 0x01, 0x01, 0x70,
+ 0x08, 0xBD, 0x18, 0x49, 0x03, 0x20, 0x08, 0x70, 0x17, 0x49, 0x19, 0x48, 0x08, 0x60, 0x19, 0x49,
+ 0x01, 0x20, 0xC1, 0xF8, 0xC0, 0x00, 0x18, 0x49, 0x4F, 0xF4, 0xFA, 0x70, 0xFF, 0xF7, 0xD5, 0xBD,
+ 0x10, 0xB5, 0x0A, 0x46, 0x81, 0x79, 0x21, 0xF0, 0x0D, 0x01, 0x41, 0xF0, 0xE0, 0x01, 0x81, 0x71,
+ 0x0B, 0x49, 0x0A, 0x60, 0x0B, 0x4A, 0x03, 0x21, 0x11, 0x70, 0x0B, 0x4A, 0x0C, 0x49, 0x11, 0x60,
+ 0x01, 0x46, 0xA4, 0x22, 0x0D, 0x48, 0x01, 0xF0, 0xB5, 0xFC, 0x0A, 0x49, 0x01, 0x20, 0xC1, 0xF8,
+ 0xC0, 0x00, 0xBD, 0xE8, 0x10, 0x40, 0x08, 0x49, 0x4F, 0xF4, 0xFA, 0x70, 0xFF, 0xF7, 0xB5, 0xBD,
+ 0xD8, 0x07, 0x10, 0x00, 0xD0, 0x07, 0x10, 0x00, 0xD4, 0x07, 0x10, 0x00, 0x00, 0x01, 0x02, 0x00,
+ 0x43, 0x4D, 0x01, 0x00, 0x00, 0x00, 0x60, 0x22, 0x35, 0x4D, 0x01, 0x00, 0x00, 0x00, 0x03, 0x20,
+ 0x16, 0x48, 0x01, 0x78, 0x01, 0x29, 0x08, 0xD1, 0x02, 0x21, 0x01, 0x70, 0x14, 0x48, 0x01, 0x68,
+ 0x19, 0xB1, 0x00, 0x22, 0x02, 0x60, 0x10, 0x46, 0x08, 0x47, 0x01, 0x20, 0xFE, 0xF7, 0x37, 0xBF,
+ 0x0E, 0x4A, 0x01, 0x21, 0x11, 0x70, 0x0E, 0x49, 0x08, 0x60, 0x0E, 0x49, 0xE8, 0x20, 0x88, 0x71,
+ 0x0D, 0x49, 0x0C, 0x20, 0xFF, 0xF7, 0x8B, 0xBD, 0x0A, 0x49, 0x08, 0x20, 0x88, 0x71, 0x0B, 0x49,
+ 0x00, 0x20, 0x08, 0x62, 0x05, 0x49, 0x08, 0x70, 0x05, 0x49, 0x08, 0x60, 0x70, 0x47, 0x03, 0x48,
+ 0x00, 0x78, 0x00, 0x28, 0x00, 0xD0, 0x01, 0x20, 0x70, 0x47, 0x00, 0x00, 0xD0, 0x07, 0x10, 0x00,
+ 0xDC, 0x07, 0x10, 0x00, 0x00, 0x00, 0x03, 0x20, 0x91, 0x4E, 0x01, 0x00, 0x00, 0x00, 0x60, 0x22,
+ 0x63, 0x48, 0x01, 0x68, 0x00, 0x29, 0x01, 0xD0, 0x01, 0x20, 0x08, 0x47, 0x70, 0x47, 0x10, 0xB5,
+ 0xFF, 0xF7, 0x3A, 0xFD, 0x5F, 0x49, 0x02, 0x20, 0x08, 0x70, 0x5F, 0x49, 0x00, 0x20, 0x08, 0x60,
+ 0x5B, 0x48, 0x01, 0x68, 0x00, 0x29, 0x03, 0xD0, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0x20, 0x08, 0x47,
+ 0x10, 0xBD, 0x57, 0x49, 0x00, 0x20, 0x08, 0x60, 0x70, 0x47, 0x7F, 0xB5, 0x04, 0x46, 0x03, 0x99,
+ 0x08, 0x9D, 0xC1, 0xF3, 0x14, 0x01, 0x03, 0x91, 0x08, 0x21, 0x02, 0xA8, 0x00, 0xF0, 0x7A, 0xF9,
+ 0xDD, 0xE9, 0x02, 0x12, 0xE1, 0x63, 0x22, 0x64, 0x94, 0xF8, 0x44, 0x20, 0xC1, 0xB2, 0x61, 0xF3,
+ 0x06, 0x02, 0x84, 0xF8, 0x44, 0x20, 0x0A, 0x2D, 0x0F, 0xD0, 0x0B, 0x2D, 0x0D, 0xD0, 0x14, 0xF8,
+ 0x72, 0x0F, 0x61, 0xF3, 0x05, 0x00, 0x20, 0x70, 0x60, 0x78, 0x61, 0xF3, 0x05, 0x00, 0x60, 0x70,
+ 0xA0, 0x78, 0x20, 0xF0, 0x08, 0x00, 0xA0, 0x70, 0x7F, 0xBD, 0x14, 0xF8, 0x72, 0x1F, 0x40, 0x1C,
+ 0x60, 0xF3, 0x05, 0x01, 0x21, 0x70, 0x61, 0x78, 0x60, 0xF3, 0x05, 0x01, 0x61, 0x70, 0xA0, 0x78,
+ 0x40, 0xF0, 0x08, 0x00, 0xEF, 0xE7, 0x1F, 0xB5, 0x04, 0x46, 0x03, 0x99, 0x02, 0xA8, 0x01, 0xF0,
+ 0x3F, 0x01, 0x03, 0x91, 0x08, 0x21, 0x00, 0xF0, 0x45, 0xF9, 0xDD, 0xE9, 0x02, 0x12, 0xA1, 0x64,
+ 0x84, 0xF8, 0x4C, 0x20, 0x14, 0xF8, 0x70, 0x1F, 0x60, 0xF3, 0x05, 0x01, 0x04, 0xF8, 0x1E, 0x19,
+ 0xE1, 0x7F, 0x60, 0xF3, 0x05, 0x01, 0xE1, 0x77, 0x1F, 0xBD, 0x30, 0xB5, 0x2F, 0xA2, 0x8C, 0xB0,
+ 0x3C, 0xCA, 0x8D, 0xE8, 0x3C, 0x00, 0x31, 0xA2, 0x0D, 0xF1, 0x10, 0x0C, 0x3C, 0xCA, 0x8C, 0xE8,
+ 0x3C, 0x00, 0x32, 0xA2, 0x0D, 0xF1, 0x20, 0x0C, 0x3C, 0xCA, 0x8C, 0xE8, 0x3C, 0x00, 0x04, 0xAA,
+ 0x53, 0x5C, 0x42, 0x7E, 0x63, 0xF3, 0x05, 0x02, 0x42, 0x76, 0x1D, 0xF8, 0x01, 0x30, 0x02, 0x7E,
+ 0x63, 0xF3, 0x42, 0x02, 0x02, 0x76, 0x08, 0xAA, 0x52, 0x5C, 0x81, 0x79, 0x62, 0xF3, 0x83, 0x01,
+ 0x81, 0x71, 0x0C, 0xB0, 0x30, 0xBD, 0x1B, 0x49, 0x04, 0x20, 0x08, 0x70, 0x1A, 0x49, 0x27, 0x48,
+ 0x08, 0x60, 0x27, 0x49, 0x01, 0x20, 0xC1, 0xF8, 0xC0, 0x00, 0x26, 0x49, 0x4F, 0xF4, 0xFA, 0x70,
+ 0xFF, 0xF7, 0xCB, 0xBC, 0x10, 0xB5, 0x0A, 0x46, 0xB0, 0xF8, 0x56, 0x10, 0x41, 0xF0, 0x01, 0x01,
+ 0xA0, 0xF8, 0x56, 0x10, 0x81, 0x79, 0x21, 0xF0, 0x01, 0x01, 0x41, 0xF0, 0xE0, 0x01, 0x81, 0x71,
+ 0x0B, 0x49, 0x0A, 0x60, 0x0B, 0x4A, 0x04, 0x21, 0x11, 0x70, 0x0B, 0x4A, 0x17, 0x49, 0x11, 0x60,
+ 0x01, 0x46, 0xA4, 0x22, 0x18, 0x48, 0x01, 0xF0, 0xA5, 0xFB, 0x15, 0x49, 0x01, 0x20, 0xC1, 0xF8,
+ 0xC0, 0x00, 0xBD, 0xE8, 0x10, 0x40, 0x13, 0x49, 0x4F, 0xF4, 0xFA, 0x70, 0xFF, 0xF7, 0xA5, 0xBC,
+ 0xE0, 0x07, 0x10, 0x00, 0xD0, 0x07, 0x10, 0x00, 0xD4, 0x07, 0x10, 0x00, 0x00, 0x00, 0x02, 0x02,
+ 0x01, 0x01, 0x03, 0x03, 0x03, 0x03, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x02, 0x02, 0x02, 0x02,
+ 0x02, 0x02, 0x02, 0x02, 0x01, 0x01, 0x02, 0x02, 0x01, 0x01, 0x00, 0x00, 0x02, 0x03, 0x02, 0x03,
+ 0x02, 0x03, 0x02, 0x03, 0x02, 0x03, 0x02, 0x02, 0x02, 0x03, 0x00, 0x00, 0x0F, 0x4F, 0x01, 0x00,
+ 0x00, 0x00, 0x60, 0x22, 0x01, 0x4F, 0x01, 0x00, 0x00, 0x00, 0x03, 0x20, 0x44, 0x49, 0x01, 0x20,
+ 0x08, 0x60, 0x17, 0x22, 0x41, 0x07, 0x01, 0xF8, 0x27, 0x2F, 0x0B, 0x22, 0x81, 0xF8, 0x41, 0x20,
+ 0x3F, 0x49, 0x18, 0x39, 0x08, 0x60, 0x4F, 0xF0, 0x08, 0x51, 0xC1, 0xF8, 0xC4, 0x03, 0x3C, 0x4B,
+ 0x00, 0x22, 0xB0, 0x33, 0x1A, 0x60, 0x3B, 0x4A, 0x10, 0x60, 0x12, 0x1D, 0x10, 0x60, 0x38, 0x4A,
+ 0x94, 0x3A, 0x10, 0x60, 0xC1, 0xF8, 0x74, 0x03, 0x70, 0x47, 0x35, 0x49, 0x01, 0x20, 0x94, 0x39,
+ 0x08, 0x60, 0x70, 0x47, 0x32, 0x49, 0x00, 0x20, 0x94, 0x39, 0x08, 0x60, 0x70, 0x47, 0x4F, 0xF0,
+ 0x00, 0x50, 0x00, 0x88, 0x70, 0x47, 0x4F, 0xF0, 0x00, 0x50, 0x80, 0x78, 0x70, 0x47, 0x4F, 0xF0,
+ 0x00, 0x51, 0x48, 0x60, 0x70, 0x47, 0x2C, 0x49, 0x02, 0x68, 0x0A, 0x60, 0x42, 0x68, 0x4A, 0x60,
+ 0x80, 0x68, 0x88, 0x60, 0x70, 0x47, 0x28, 0x49, 0x0A, 0x68, 0x02, 0x60, 0x4A, 0x68, 0x42, 0x60,
+ 0x89, 0x68, 0x81, 0x60, 0x70, 0x47, 0x22, 0x49, 0x01, 0x20, 0x30, 0x39, 0x08, 0x60, 0x70, 0x47,
+ 0x4F, 0xF0, 0x00, 0x50, 0x10, 0xF8, 0x22, 0x1F, 0x21, 0xF0, 0x03, 0x01, 0x01, 0x70, 0x70, 0x47,
+ 0x4F, 0xF0, 0x00, 0x53, 0x03, 0xF8, 0x1F, 0x0F, 0x59, 0x70, 0x98, 0x78, 0x62, 0xF3, 0x03, 0x00,
+ 0x98, 0x70, 0x70, 0x47, 0x19, 0x48, 0x00, 0x68, 0x70, 0x47, 0x18, 0x49, 0x08, 0x60, 0x70, 0x47,
+ 0x10, 0xB5, 0x4F, 0xF0, 0x00, 0x54, 0xE2, 0x7E, 0x20, 0x23, 0x22, 0xF0, 0xAF, 0x02, 0x03, 0xEA,
+ 0x40, 0x13, 0x13, 0x43, 0x04, 0x22, 0x02, 0xEA, 0x80, 0x00, 0x03, 0x43, 0x01, 0xF0, 0x03, 0x00,
+ 0x03, 0x43, 0xE3, 0x76, 0x10, 0xBD, 0x4F, 0xF0, 0x00, 0x50, 0x90, 0xF8, 0x22, 0x00, 0x10, 0xF0,
+ 0x03, 0x00, 0x05, 0xD0, 0x01, 0x28, 0x05, 0xD0, 0x02, 0x28, 0x05, 0xD0, 0x08, 0x48, 0x70, 0x47,
+ 0x08, 0x48, 0x70, 0x47, 0x08, 0x48, 0x70, 0x47, 0x08, 0x48, 0x70, 0x47, 0x70, 0x47, 0x70, 0x47,
+ 0xCC, 0x04, 0x00, 0x22, 0x80, 0x1E, 0x00, 0x22, 0x08, 0x00, 0x00, 0x20, 0xE4, 0x07, 0x10, 0x00,
+ 0x00, 0x12, 0x7A, 0x00, 0x00, 0x90, 0xD0, 0x03, 0x00, 0x48, 0xE8, 0x01, 0x00, 0x24, 0xF4, 0x00,
+ 0xF0, 0xB5, 0x07, 0x46, 0x00, 0x20, 0x03, 0x46, 0x2F, 0x4E, 0x10, 0xE0, 0x57, 0xF8, 0x23, 0x50,
+ 0x03, 0x22, 0x4F, 0xEA, 0xC2, 0x0C, 0x45, 0xFA, 0x0C, 0xF4, 0x84, 0xEA, 0x20, 0x64, 0xE4, 0xB2,
+ 0x52, 0x1E, 0x56, 0xF8, 0x24, 0x40, 0x84, 0xEA, 0x00, 0x20, 0xF2, 0xD5, 0x5B, 0x1C, 0x8B, 0x42,
+ 0xEC, 0xDB, 0xF0, 0xBD, 0x30, 0xB5, 0x04, 0x46, 0x23, 0x4B, 0x00, 0x20, 0x10, 0x3B, 0x08, 0xE0,
+ 0x62, 0x5C, 0x02, 0xF0, 0x0F, 0x05, 0x03, 0xEB, 0x12, 0x12, 0x5D, 0x5D, 0x12, 0x78, 0x28, 0x44,
+ 0x10, 0x44, 0x49, 0x1E, 0xF4, 0xD2, 0x30, 0xBD, 0x10, 0xB5, 0x84, 0x68, 0x03, 0x68, 0xE4, 0x43,
+ 0xA3, 0x42, 0x0A, 0xD1, 0xC4, 0x68, 0x43, 0x68, 0xE4, 0x43, 0xA3, 0x42, 0x05, 0xD1, 0x5B, 0x1C,
+ 0x0B, 0x60, 0x00, 0x68, 0x10, 0x60, 0x01, 0x20, 0x10, 0xBD, 0x00, 0x20, 0x10, 0xBD, 0x3C, 0xB5,
+ 0x04, 0x46, 0x00, 0x25, 0x01, 0xAA, 0x69, 0x46, 0xFF, 0xF7, 0xE6, 0xFF, 0x01, 0x28, 0x09, 0xD1,
+ 0x00, 0x99, 0x04, 0xF1, 0x10, 0x00, 0x09, 0x1F, 0xFF, 0xF7, 0xB2, 0xFF, 0x01, 0x99, 0x88, 0x42,
+ 0x00, 0xD1, 0x01, 0x25, 0x28, 0x46, 0x3C, 0xBD, 0x30, 0xB5, 0x0D, 0x46, 0x04, 0x46, 0x09, 0x1F,
+ 0x10, 0x30, 0xFF, 0xF7, 0xA5, 0xFF, 0x6D, 0x1E, 0x65, 0x60, 0xE9, 0x43, 0xE1, 0x60, 0x20, 0x60,
+ 0xC0, 0x43, 0xA0, 0x60, 0x30, 0xBD, 0x00, 0x00, 0x1C, 0x70, 0x01, 0x00, 0x01, 0x46, 0x80, 0x00,
+ 0xB1, 0xF5, 0x80, 0x3F, 0x02, 0xD2, 0x10, 0x38, 0xC0, 0xF3, 0x10, 0x00, 0x70, 0x47, 0xC0, 0xF3,
+ 0x87, 0x20, 0x70, 0x47, 0xB0, 0xF5, 0x80, 0x2F, 0x02, 0xD2, 0x10, 0x30, 0xC0, 0xF3, 0x10, 0x00,
+ 0xC0, 0xF3, 0x8F, 0x00, 0x70, 0x47, 0x30, 0xB5, 0xFE, 0x4C, 0x01, 0x25, 0x25, 0x60, 0xE5, 0x06,
+ 0x03, 0xF0, 0x1F, 0x03, 0x85, 0xF8, 0x6B, 0x30, 0x00, 0x23, 0x04, 0xE0, 0x51, 0xF8, 0x23, 0x50,
+ 0x40, 0xF8, 0x23, 0x50, 0x5B, 0x1C, 0x93, 0x42, 0xF8, 0xDB, 0x00, 0x20, 0x20, 0x60, 0x30, 0xBD,
+ 0xF0, 0xB5, 0xF4, 0x4F, 0x01, 0x23, 0x3B, 0x60, 0x5C, 0x07, 0x84, 0xF8, 0x6B, 0x30, 0x9B, 0x1E,
+ 0x03, 0x60, 0x00, 0x23, 0x12, 0xE0, 0x51, 0xF8, 0x23, 0x40, 0x40, 0xF8, 0x23, 0x40, 0x00, 0x24,
+ 0x50, 0xF8, 0x23, 0x50, 0x51, 0xF8, 0x23, 0x60, 0xB5, 0x42, 0x06, 0xD0, 0x6D, 0x1C, 0x04, 0xD1,
+ 0x64, 0x1C, 0x40, 0xF8, 0x23, 0x60, 0x03, 0x2C, 0xF2, 0xD3, 0x5B, 0x1C, 0x93, 0x42, 0xEA, 0xDB,
+ 0x00, 0x20, 0x38, 0x60, 0xF0, 0xBD, 0xF0, 0xB5, 0x13, 0x46, 0x0A, 0x46, 0x05, 0x46, 0x00, 0x24,
+ 0x98, 0x18, 0x4F, 0xF4, 0x83, 0x26, 0x60, 0x28, 0x01, 0xD9, 0x01, 0x24, 0x11, 0xE0, 0x00, 0x21,
+ 0x07, 0xE0, 0x5F, 0x18, 0x56, 0xF8, 0x27, 0x00, 0x40, 0x1C, 0x01, 0xD0, 0x11, 0x46, 0x02, 0x24,
+ 0x49, 0x1C, 0x91, 0x42, 0xF5, 0xD3, 0x24, 0xB9, 0x06, 0xEB, 0x83, 0x00, 0x29, 0x46, 0xFF, 0xF7,
+ 0xBF, 0xFF, 0x20, 0x46, 0xF0, 0xBD, 0x10, 0xB5, 0x03, 0x46, 0x08, 0x46, 0xFF, 0xF7, 0x8E, 0xFF,
+ 0x01, 0x46, 0xBD, 0xE8, 0x10, 0x40, 0x92, 0x00, 0x18, 0x46, 0x01, 0xF0, 0x40, 0xBA, 0x2D, 0xE9,
+ 0xF0, 0x41, 0x4F, 0xF0, 0x00, 0x58, 0x0C, 0x46, 0xD8, 0xF8, 0x14, 0x70, 0x01, 0x25, 0xC0, 0xF3,
+ 0x8F, 0x06, 0x6F, 0xF4, 0x80, 0x41, 0xC8, 0xF8, 0x14, 0x10, 0x0E, 0x20, 0x00, 0xF0, 0x13, 0xFA,
+ 0x20, 0x46, 0x00, 0xF0, 0x1B, 0xFA, 0x02, 0x00, 0xC3, 0x48, 0x4F, 0xF0, 0x01, 0x01, 0x18, 0xD0,
+ 0x01, 0x60, 0xA4, 0xF5, 0x80, 0x34, 0x40, 0x46, 0x20, 0xF8, 0x72, 0x6F, 0x44, 0x80, 0x60, 0x22,
+ 0x00, 0xF8, 0x01, 0x2C, 0xBD, 0x4A, 0x11, 0x60, 0xBB, 0x4A, 0xC4, 0x32, 0x11, 0x60, 0x30, 0xBF,
+ 0x81, 0x79, 0x72, 0x38, 0x89, 0x07, 0x00, 0xD0, 0x00, 0x25, 0x47, 0x61, 0x28, 0x46, 0xBD, 0xE8,
+ 0xF0, 0x81, 0x00, 0x22, 0x02, 0x60, 0xE6, 0xE7, 0x2D, 0xE9, 0xF0, 0x5F, 0x15, 0x46, 0xDF, 0xF8,
+ 0xC4, 0xB2, 0x00, 0x26, 0x01, 0x27, 0x0A, 0x46, 0x04, 0x46, 0x4F, 0xF0, 0x00, 0x5C, 0xB1, 0xF5,
+ 0x88, 0x2F, 0x15, 0xD3, 0xA3, 0x0A, 0xFF, 0xF7, 0x41, 0xFF, 0x59, 0x46, 0xCB, 0xF8, 0x00, 0x70,
+ 0x03, 0xF0, 0x1F, 0x03, 0x8C, 0xF8, 0x6B, 0x30, 0x00, 0x23, 0x04, 0xE0, 0x52, 0xF8, 0x23, 0x40,
+ 0x40, 0xF8, 0x23, 0x40, 0x5B, 0x1C, 0xAB, 0x42, 0xF8, 0xDB, 0x0E, 0x60, 0xBD, 0xE8, 0xF0, 0x9F,
+ 0xDC, 0xF8, 0x14, 0x90, 0xC1, 0xF3, 0x8F, 0x08, 0xE2, 0x46, 0x6F, 0xF4, 0x80, 0x41, 0xCC, 0xF8,
+ 0x14, 0x10, 0x0E, 0x20, 0x00, 0xF0, 0xBF, 0xF9, 0xCB, 0xF8, 0x00, 0x70, 0x20, 0x46, 0x00, 0xF0,
+ 0xC5, 0xF9, 0x01, 0x00, 0x98, 0x48, 0x19, 0xD0, 0x07, 0x60, 0xA4, 0xF5, 0x80, 0x34, 0xAA, 0xF8,
+ 0x72, 0x80, 0xAA, 0xF8, 0x74, 0x40, 0x6D, 0x1E, 0xAA, 0xF8, 0x76, 0x50, 0x40, 0x21, 0x8A, 0xF8,
+ 0x71, 0x10, 0x92, 0x49, 0x0E, 0x60, 0x90, 0x49, 0xC4, 0x31, 0x0F, 0x60, 0x0A, 0xF1, 0x52, 0x00,
+ 0x30, 0xBF, 0xCB, 0xF8, 0x00, 0x60, 0x40, 0xF8, 0x3E, 0x9C, 0xCF, 0xE7, 0x06, 0x60, 0xE6, 0xE7,
+ 0x2D, 0xE9, 0xF0, 0x41, 0x4F, 0xF0, 0x00, 0x54, 0x0F, 0x46, 0x54, 0xF8, 0x14, 0x6F, 0x85, 0x08,
+ 0x6F, 0xF4, 0x80, 0x40, 0x20, 0x60, 0x0E, 0x20, 0x00, 0xF0, 0x8D, 0xF9, 0x00, 0x20, 0xA0, 0x66,
+ 0xA4, 0xF8, 0x5E, 0x50, 0x7F, 0x1E, 0xA4, 0xF8, 0x62, 0x70, 0x84, 0xF8, 0x5D, 0x00, 0x7F, 0x49,
+ 0x08, 0x60, 0x7D, 0x49, 0x01, 0x20, 0xC4, 0x31, 0x08, 0x60, 0x30, 0xBF, 0xE0, 0x6E, 0x26, 0x60,
+ 0x85, 0xE7, 0x70, 0xB5, 0x06, 0x46, 0x0C, 0x46, 0x01, 0xF1, 0x10, 0x00, 0x11, 0x1F, 0x15, 0x46,
+ 0x89, 0xB2, 0xFF, 0xF7, 0xD5, 0xFF, 0x20, 0x60, 0x68, 0x1E, 0x60, 0x60, 0x20, 0x68, 0x2A, 0x46,
+ 0xC0, 0x43, 0xA0, 0x60, 0x60, 0x68, 0x21, 0x46, 0xC0, 0x43, 0xE0, 0x60, 0x30, 0x46, 0xBD, 0xE8,
+ 0x70, 0x40, 0x71, 0xE7, 0xF0, 0xB5, 0x07, 0x46, 0x00, 0x20, 0x04, 0x46, 0x6C, 0x4E, 0x15, 0xE0,
+ 0x57, 0xF8, 0x24, 0x50, 0x03, 0x23, 0x4F, 0xEA, 0xC3, 0x0C, 0x45, 0xFA, 0x0C, 0xF2, 0x80, 0xEA,
+ 0x02, 0x60, 0x08, 0x22, 0x00, 0x28, 0x02, 0xDA, 0x86, 0xEA, 0x40, 0x00, 0x00, 0xE0, 0x40, 0x00,
+ 0x52, 0x1E, 0x00, 0x2A, 0xF6, 0xDC, 0x5B, 0x1E, 0xED, 0xD5, 0x64, 0x1C, 0x8C, 0x42, 0xE7, 0xDB,
+ 0xF0, 0xBD, 0x30, 0xB5, 0x50, 0xF8, 0x08, 0x4B, 0x50, 0xF8, 0x08, 0x5B, 0x30, 0xF8, 0x0C, 0x1C,
+ 0x49, 0x1C, 0x89, 0xB2, 0xFF, 0xF7, 0xD6, 0xFF, 0x84, 0x42, 0x04, 0xD1, 0xE0, 0x43, 0x85, 0x42,
+ 0x01, 0xD1, 0x01, 0x20, 0x30, 0xBD, 0x00, 0x20, 0x30, 0xBD, 0x2D, 0xE9, 0xF0, 0x41, 0x01, 0x26,
+ 0x75, 0x07, 0x04, 0x46, 0x6F, 0x69, 0x6F, 0xF4, 0x80, 0x40, 0x68, 0x61, 0x0E, 0x20, 0x00, 0xF0,
+ 0x22, 0xF9, 0x4F, 0xF0, 0x00, 0x08, 0xC5, 0xF8, 0x7C, 0x80, 0x20, 0x46, 0x00, 0xF0, 0x26, 0xF9,
+ 0x02, 0x00, 0x49, 0x49, 0x4F, 0xF0, 0x01, 0x00, 0x15, 0xD0, 0x08, 0x60, 0xA4, 0xF5, 0x80, 0x34,
+ 0xA5, 0xF8, 0x74, 0x40, 0x20, 0x21, 0x85, 0xF8, 0x71, 0x10, 0x44, 0x49, 0x08, 0x60, 0x42, 0x49,
+ 0xC4, 0x31, 0x08, 0x60, 0x30, 0xBF, 0x95, 0xF8, 0x78, 0x00, 0x80, 0x07, 0x00, 0xD0, 0x00, 0x26,
+ 0x6F, 0x61, 0x30, 0x46, 0x0B, 0xE7, 0xC1, 0xF8, 0x00, 0x80, 0xE9, 0xE7, 0x00, 0xB5, 0x0B, 0x46,
+ 0xFF, 0xF7, 0x5C, 0xFE, 0x82, 0x68, 0x01, 0x68, 0xD2, 0x43, 0x91, 0x42, 0x08, 0xD1, 0x42, 0x68,
+ 0xC0, 0x68, 0xC0, 0x43, 0x82, 0x42, 0x03, 0xD1, 0x52, 0x1C, 0x1A, 0x80, 0x01, 0x20, 0x00, 0xBD,
+ 0x00, 0x20, 0x00, 0xBD, 0xF8, 0xB5, 0x04, 0x46, 0x10, 0x30, 0x0D, 0x46, 0x06, 0x46, 0xFF, 0xF7,
+ 0xBE, 0xFE, 0x01, 0x28, 0x01, 0xD0, 0x00, 0x20, 0xF8, 0xBD, 0x69, 0x46, 0x28, 0x46, 0xFF, 0xF7,
+ 0xDD, 0xFF, 0xBD, 0xF8, 0x00, 0x00, 0x00, 0x1F, 0x81, 0xB2, 0x30, 0x46, 0xFF, 0xF7, 0x38, 0xFF,
+ 0x20, 0x60, 0xBD, 0xF8, 0x00, 0x10, 0xC0, 0x43, 0x49, 0x1E, 0x61, 0x60, 0xA0, 0x60, 0xC8, 0x43,
+ 0xE0, 0x60, 0x01, 0x20, 0xF8, 0xBD, 0xF0, 0xB5, 0xB1, 0xB0, 0x07, 0x46, 0x6D, 0x46, 0x21, 0x4E,
+ 0x01, 0x24, 0xC0, 0x21, 0x28, 0x46, 0x01, 0xF0, 0x36, 0xF9, 0x00, 0x20, 0x56, 0xF8, 0x20, 0x10,
+ 0x45, 0xF8, 0x20, 0x10, 0x40, 0x1C, 0x30, 0x28, 0xF8, 0xD3, 0x1B, 0x49, 0x00, 0x98, 0x88, 0x42,
+ 0x10, 0xD1, 0x02, 0x98, 0x81, 0xB2, 0x05, 0xA8, 0x04, 0x46, 0xFF, 0xF7, 0x4B, 0xFF, 0x01, 0x99,
+ 0x81, 0x42, 0x06, 0xD1, 0xAC, 0x22, 0x21, 0x46, 0x38, 0x46, 0x01, 0xF0, 0xC0, 0xF8, 0x00, 0x24,
+ 0x00, 0xE0, 0x03, 0x24, 0x31, 0xB0, 0x20, 0x46, 0xF0, 0xBD, 0x70, 0xB5, 0x94, 0xB0, 0x04, 0x46,
+ 0x6D, 0x46, 0x0E, 0x4E, 0x50, 0x21, 0x28, 0x46, 0x01, 0xF0, 0x0D, 0xF9, 0x00, 0x20, 0x56, 0xF8,
+ 0x20, 0x10, 0x45, 0xF8, 0x20, 0x10, 0x40, 0x1C, 0x14, 0x28, 0xF8, 0xD3, 0x08, 0x48, 0x6F, 0xF0,
+ 0x02, 0x05, 0x0F, 0xE0, 0xC4, 0x1B, 0x00, 0x22, 0x78, 0x0D, 0x00, 0x22, 0x1C, 0x0F, 0x00, 0x22,
+ 0x1B, 0x00, 0x00, 0x80, 0x48, 0x1D, 0x04, 0x00, 0xB2, 0x57, 0x0E, 0x70, 0xAC, 0x1E, 0x04, 0x00,
+ 0x5C, 0x00, 0x0D, 0x60, 0x00, 0x99, 0x81, 0x42, 0x2E, 0xD1, 0xA8, 0x10, 0x20, 0x60, 0x60, 0x60,
+ 0xA0, 0x60, 0xE0, 0x60, 0x20, 0x61, 0x02, 0x98, 0x81, 0xB2, 0x05, 0xA8, 0xFF, 0xF7, 0x0A, 0xFF,
+ 0x01, 0x99, 0x81, 0x42, 0x20, 0xD1, 0xBD, 0xF8, 0x38, 0x10, 0x45, 0xF2, 0xAA, 0x50, 0x81, 0x42,
+ 0x0C, 0xD1, 0x9D, 0xF8, 0x1D, 0x10, 0x21, 0x60, 0x9D, 0xF8, 0x1C, 0x10, 0x01, 0xF0, 0x0F, 0x01,
+ 0x61, 0x60, 0x9D, 0xF8, 0x1E, 0x10, 0x01, 0xF0, 0x0F, 0x01, 0xA1, 0x60, 0xBD, 0xF8, 0x3C, 0x10,
+ 0x81, 0x42, 0x01, 0xD1, 0x08, 0x99, 0xE1, 0x60, 0xBD, 0xF8, 0x40, 0x10, 0x81, 0x42, 0x01, 0xD1,
+ 0x09, 0x98, 0x20, 0x61, 0x14, 0xB0, 0x70, 0xBD, 0x25, 0x60, 0x65, 0x60, 0xA5, 0x60, 0xE5, 0x60,
+ 0x25, 0x61, 0xF7, 0xE7, 0x70, 0xB5, 0x86, 0xB0, 0x04, 0x46, 0x6D, 0x46, 0x27, 0x4E, 0x14, 0x21,
+ 0x28, 0x46, 0x01, 0xF0, 0xB0, 0xF8, 0x00, 0x21, 0x56, 0xF8, 0x21, 0x00, 0x45, 0xF8, 0x21, 0x00,
+ 0x49, 0x1C, 0x05, 0x29, 0xF8, 0xD3, 0x00, 0x98, 0x20, 0x60, 0x01, 0x98, 0x60, 0x60, 0x02, 0x98,
+ 0xC0, 0xB2, 0xA0, 0x60, 0x03, 0x98, 0xC0, 0xB2, 0xE0, 0x60, 0x04, 0x98, 0xC0, 0xB2, 0x20, 0x61,
+ 0x06, 0xB0, 0x70, 0xBD, 0x1C, 0xB5, 0x00, 0x20, 0x18, 0x4A, 0x00, 0x90, 0x84, 0x3A, 0x6B, 0x46,
+ 0x01, 0x46, 0x01, 0x90, 0x52, 0xF8, 0x21, 0x40, 0x43, 0xF8, 0x21, 0x40, 0x49, 0x1C, 0x02, 0x29,
+ 0xF8, 0xD3, 0xBD, 0xF8, 0x00, 0x10, 0xA1, 0xF5, 0xAA, 0x42, 0xAA, 0x3A, 0x0A, 0xD1, 0xBD, 0xF8,
+ 0x04, 0x20, 0x11, 0x42, 0x06, 0xD1, 0x9D, 0xF8, 0x02, 0x10, 0x9D, 0xF8, 0x06, 0x20, 0x11, 0x42,
+ 0x00, 0xD1, 0x08, 0x46, 0x1C, 0xBD, 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21, 0x91, 0x40, 0x40, 0x09,
+ 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20, 0xC0, 0xF8, 0x80, 0x12, 0x70, 0x47, 0xB0, 0xF5, 0x80, 0x3F,
+ 0x01, 0xD2, 0x00, 0x20, 0x70, 0x47, 0x01, 0x20, 0x70, 0x47, 0x00, 0x00, 0x60, 0x1F, 0x04, 0x00,
+ 0x4F, 0xF0, 0x00, 0x50, 0x00, 0x21, 0x80, 0xF8, 0x3D, 0x10, 0x80, 0xF8, 0x3E, 0x10, 0x80, 0xF8,
+ 0x3F, 0x10, 0x3F, 0x22, 0x80, 0xF8, 0x32, 0x20, 0x00, 0xF8, 0x30, 0x1F, 0x02, 0x71, 0x41, 0x71,
+ 0x42, 0x70, 0x01, 0x72, 0x41, 0x72, 0x81, 0x72, 0xC1, 0x72, 0xC1, 0x71, 0x70, 0x47, 0x99, 0x4A,
+ 0x10, 0x60, 0x98, 0x48, 0x0C, 0x30, 0x01, 0x60, 0x70, 0x47, 0x30, 0xB5, 0x01, 0x24, 0x84, 0x40,
+ 0xE0, 0xB2, 0x4F, 0xF0, 0x00, 0x54, 0x00, 0x29, 0x03, 0x9D, 0x94, 0xF8, 0x32, 0x10, 0x01, 0xD0,
+ 0x01, 0x43, 0x00, 0xE0, 0x81, 0x43, 0x84, 0xF8, 0x32, 0x10, 0x94, 0xF8, 0x30, 0x10, 0x0A, 0xB1,
+ 0x01, 0x43, 0x00, 0xE0, 0x81, 0x43, 0x84, 0xF8, 0x30, 0x10, 0x94, 0xF8, 0x34, 0x10, 0x0B, 0xB1,
+ 0x01, 0x43, 0x00, 0xE0, 0x81, 0x43, 0x84, 0xF8, 0x34, 0x10, 0x14, 0xF8, 0x35, 0x1F, 0x0D, 0xB1,
+ 0x01, 0x43, 0x00, 0xE0, 0x81, 0x43, 0x21, 0x70, 0x30, 0xBD, 0x4F, 0xF0, 0x00, 0x52, 0x06, 0x28,
+ 0x17, 0xD2, 0xDF, 0xE8, 0x00, 0xF0, 0x03, 0x06, 0x09, 0x0F, 0x17, 0x1A, 0x12, 0xF8, 0x3D, 0x0F,
+ 0x04, 0xE0, 0x12, 0xF8, 0x3D, 0x0F, 0x07, 0xE0, 0x12, 0xF8, 0x3E, 0x0F, 0x20, 0xF0, 0x0F, 0x00,
+ 0x08, 0x43, 0x05, 0xE0, 0x12, 0xF8, 0x3E, 0x0F, 0x20, 0xF0, 0xF0, 0x00, 0x40, 0xEA, 0x01, 0x10,
+ 0x10, 0x70, 0x70, 0x47, 0x12, 0xF8, 0x3F, 0x0F, 0xF0, 0xE7, 0x12, 0xF8, 0x3F, 0x0F, 0xF3, 0xE7,
+ 0x01, 0x23, 0x83, 0x40, 0xD8, 0xB2, 0x4F, 0xF0, 0x00, 0x53, 0x00, 0x29, 0x93, 0xF8, 0x38, 0x10,
+ 0x01, 0xD0, 0x01, 0x43, 0x00, 0xE0, 0x81, 0x43, 0x83, 0xF8, 0x38, 0x10, 0x13, 0xF8, 0x39, 0x1F,
+ 0x0A, 0xB1, 0x01, 0x43, 0x00, 0xE0, 0x81, 0x43, 0x19, 0x70, 0x70, 0x47, 0x01, 0x23, 0x83, 0x40,
+ 0xD8, 0xB2, 0x4F, 0xF0, 0x00, 0x53, 0x00, 0x29, 0x93, 0xF8, 0x3A, 0x10, 0x01, 0xD0, 0x01, 0x43,
+ 0x00, 0xE0, 0x81, 0x43, 0x83, 0xF8, 0x3A, 0x10, 0x13, 0xF8, 0x3A, 0x1F, 0x0A, 0xB1, 0x01, 0x43,
+ 0x00, 0xE0, 0x81, 0x43, 0x19, 0x70, 0x70, 0x47, 0x2D, 0xE9, 0xF0, 0x41, 0x05, 0x46, 0x01, 0x20,
+ 0xA8, 0x40, 0xC4, 0xB2, 0x1E, 0x46, 0x90, 0x46, 0x0F, 0x46, 0x0A, 0x20, 0x00, 0xF0, 0x93, 0xF8,
+ 0x0B, 0x20, 0x00, 0xF0, 0x90, 0xF8, 0x0A, 0x20, 0x00, 0xF0, 0x98, 0xF8, 0x0B, 0x20, 0x00, 0xF0,
+ 0x95, 0xF8, 0x4F, 0xF0, 0x00, 0x50, 0x8F, 0xB1, 0x90, 0xF8, 0x2E, 0x10, 0xA1, 0x43, 0x80, 0xF8,
+ 0x2E, 0x10, 0x90, 0xF8, 0x38, 0x10, 0x21, 0x43, 0x80, 0xF8, 0x38, 0x10, 0x90, 0xF8, 0x3A, 0x10,
+ 0x21, 0x43, 0x80, 0xF8, 0x3A, 0x10, 0x48, 0x49, 0x41, 0xF8, 0x25, 0x60, 0xB8, 0xF1, 0x00, 0x0F,
+ 0x0D, 0xD0, 0x10, 0xF8, 0x2F, 0x1F, 0xA1, 0x43, 0x01, 0x70, 0x81, 0x7A, 0x21, 0x43, 0x81, 0x72,
+ 0x01, 0x7B, 0x21, 0x43, 0x01, 0x73, 0x40, 0x48, 0x20, 0x30, 0x40, 0xF8, 0x25, 0x60, 0x0A, 0x20,
+ 0x00, 0xF0, 0x56, 0xF8, 0xBD, 0xE8, 0xF0, 0x41, 0x0B, 0x20, 0x00, 0xF0, 0x51, 0xB8, 0x2D, 0xE9,
+ 0xF0, 0x41, 0x05, 0x46, 0x01, 0x20, 0xA8, 0x40, 0xC4, 0xB2, 0x17, 0x46, 0x0E, 0x46, 0x0A, 0x20,
+ 0x00, 0xF0, 0x51, 0xF8, 0x0B, 0x20, 0x00, 0xF0, 0x4E, 0xF8, 0x0A, 0x20, 0x00, 0xF0, 0x56, 0xF8,
+ 0x0B, 0x20, 0x00, 0xF0, 0x53, 0xF8, 0x31, 0x00, 0x4F, 0xF0, 0x00, 0x02, 0x4F, 0xF0, 0x00, 0x56,
+ 0x11, 0xD0, 0x96, 0xF8, 0x38, 0x00, 0xA0, 0x43, 0x86, 0xF8, 0x38, 0x00, 0x96, 0xF8, 0x2E, 0x00,
+ 0xA0, 0x43, 0x86, 0xF8, 0x2E, 0x00, 0x96, 0xF8, 0x3A, 0x00, 0xA0, 0x43, 0x86, 0xF8, 0x3A, 0x00,
+ 0x25, 0x48, 0x40, 0xF8, 0x25, 0x20, 0x97, 0xB1, 0x96, 0xF8, 0x39, 0x00, 0xA0, 0x43, 0x86, 0xF8,
+ 0x39, 0x00, 0x96, 0xF8, 0x2F, 0x00, 0xA0, 0x43, 0x86, 0xF8, 0x2F, 0x00, 0x96, 0xF8, 0x3B, 0x00,
+ 0xA0, 0x43, 0x86, 0xF8, 0x3B, 0x00, 0x1C, 0x48, 0x20, 0x30, 0x40, 0xF8, 0x25, 0x20, 0x96, 0xF8,
+ 0x38, 0x00, 0x10, 0xB1, 0x0A, 0x20, 0x00, 0xF0, 0x0B, 0xF8, 0x96, 0xF8, 0x39, 0x00, 0x00, 0x28,
+ 0x04, 0xD0, 0xBD, 0xE8, 0xF0, 0x41, 0x0B, 0x20, 0x00, 0xF0, 0x02, 0xB8, 0xBD, 0xE8, 0xF0, 0x81,
+ 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21, 0x91, 0x40, 0x40, 0x09, 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20,
+ 0xC0, 0xF8, 0x00, 0x11, 0x70, 0x47, 0x00, 0xF0, 0x1F, 0x02, 0x01, 0x21, 0x91, 0x40, 0x40, 0x09,
+ 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20, 0xC0, 0xF8, 0x80, 0x11, 0x70, 0x47, 0x00, 0xF0, 0x1F, 0x02,
+ 0x01, 0x21, 0x91, 0x40, 0x40, 0x09, 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20, 0xC0, 0xF8, 0x80, 0x12,
+ 0x70, 0x47, 0x00, 0x00, 0x38, 0x05, 0x00, 0x22, 0xFC, 0x87, 0x10, 0x00, 0xF0, 0xB5, 0x63, 0x4D,
+ 0xD0, 0xE9, 0x00, 0x23, 0x01, 0x21, 0xEC, 0x1D, 0xFF, 0x27, 0x4E, 0x07, 0x00, 0x7A, 0x86, 0xF8,
+ 0xE5, 0x70, 0x86, 0xF8, 0xE4, 0x70, 0x00, 0x27, 0xC6, 0xF8, 0xC0, 0x70, 0xC6, 0xF8, 0xE8, 0x70,
+ 0x86, 0xF8, 0xE1, 0x70, 0x86, 0xF8, 0xE3, 0x70, 0x8E, 0x46, 0x58, 0xE0, 0x17, 0x78, 0x05, 0xF8,
+ 0x01, 0x7B, 0x1F, 0x88, 0x24, 0xF8, 0x02, 0x7B, 0xD7, 0x78, 0x01, 0x2F, 0x02, 0xD0, 0x02, 0x2F,
+ 0x09, 0xD0, 0x10, 0xE0, 0x96, 0xF8, 0xE4, 0x70, 0x0E, 0xFA, 0x01, 0xFC, 0x27, 0xEA, 0x0C, 0x07,
+ 0x86, 0xF8, 0xE4, 0x70, 0x07, 0xE0, 0x96, 0xF8, 0xE5, 0x70, 0x0E, 0xFA, 0x01, 0xFC, 0x27, 0xEA,
+ 0x0C, 0x07, 0x86, 0xF8, 0xE5, 0x70, 0x4F, 0x1E, 0x07, 0xEB, 0x47, 0x0C, 0x27, 0xF0, 0x01, 0x07,
+ 0xBC, 0x44, 0x57, 0x78, 0x07, 0xF0, 0x07, 0x07, 0x07, 0xFA, 0x0C, 0xF7, 0xD6, 0xF8, 0xC0, 0xC0,
+ 0x47, 0xEA, 0x0C, 0x07, 0xC6, 0xF8, 0xC0, 0x70, 0x97, 0x78, 0x4F, 0xEA, 0x81, 0x0C, 0x07, 0xF0,
+ 0x0F, 0x07, 0xAC, 0xF1, 0x04, 0x0C, 0x07, 0xFA, 0x0C, 0xF7, 0xD6, 0xF8, 0xE8, 0xC0, 0x47, 0xEA,
+ 0x0C, 0x07, 0xC6, 0xF8, 0xE8, 0x70, 0x17, 0x79, 0xFF, 0x07, 0x07, 0xD0, 0x96, 0xF8, 0xE1, 0x70,
+ 0x0E, 0xFA, 0x01, 0xFC, 0x47, 0xEA, 0x0C, 0x07, 0x86, 0xF8, 0xE1, 0x70, 0x17, 0x79, 0xFF, 0x06,
+ 0x07, 0xD5, 0x96, 0xF8, 0xE3, 0x70, 0x0E, 0xFA, 0x01, 0xFC, 0x47, 0xEA, 0x0C, 0x07, 0x86, 0xF8,
+ 0xE3, 0x70, 0x49, 0x1C, 0x08, 0x29, 0x04, 0xD2, 0x40, 0x1E, 0x52, 0x1D, 0x9B, 0x1C, 0x00, 0x28,
+ 0xA4, 0xD1, 0xD6, 0xF8, 0xE8, 0x00, 0x70, 0x64, 0xF0, 0xBD, 0xF0, 0xB5, 0x27, 0x4D, 0x28, 0x4F,
+ 0x01, 0x26, 0xEC, 0x1D, 0xC2, 0x79, 0x00, 0x21, 0x03, 0x68, 0x79, 0x60, 0x27, 0x07, 0x4F, 0xF0,
+ 0xFF, 0x0E, 0x87, 0xF8, 0xE5, 0xE0, 0x87, 0xF8, 0xE4, 0x60, 0xC7, 0xF8, 0xC0, 0x10, 0x87, 0xF8,
+ 0xE1, 0x10, 0x87, 0xF8, 0xE3, 0x10, 0x18, 0xE0, 0x93, 0xF8, 0x00, 0xC0, 0x05, 0xF8, 0x01, 0xCB,
+ 0xB0, 0xF8, 0x04, 0xC0, 0x24, 0xF8, 0x02, 0xCB, 0x90, 0xF8, 0x06, 0xC0, 0x4F, 0xEA, 0x86, 0x0E,
+ 0x0C, 0xF0, 0x0F, 0x0C, 0xAE, 0xF1, 0x04, 0x0E, 0x0C, 0xFA, 0x0E, 0xFC, 0x76, 0x1C, 0x4C, 0xEA,
+ 0x01, 0x01, 0x08, 0x2E, 0x03, 0xD2, 0x52, 0x1E, 0x5B, 0x1C, 0x00, 0x2A, 0xE4, 0xD1, 0xC7, 0xF8,
+ 0xE8, 0x10, 0x79, 0x64, 0xF0, 0xBD, 0x0D, 0x4A, 0xD2, 0x1D, 0x22, 0xF8, 0x10, 0x10, 0x70, 0x47,
+ 0x10, 0xB5, 0x04, 0x46, 0x0B, 0x48, 0x00, 0x68, 0x38, 0xB1, 0xA0, 0x68, 0xFF, 0xF7, 0x46, 0xFF,
+ 0x20, 0x68, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0xF0, 0x0F, 0xB8, 0xE0, 0x68, 0xFF, 0xF7, 0x3E, 0xFF,
+ 0x60, 0x68, 0xBD, 0xE8, 0x10, 0x40, 0x00, 0xF0, 0xCB, 0xBC, 0x00, 0x00, 0xAB, 0x00, 0x00, 0x20,
+ 0x00, 0x20, 0x00, 0x22, 0xBC, 0x05, 0x00, 0x22, 0x10, 0xB5, 0x4F, 0x4B, 0x01, 0x68, 0x00, 0x24,
+ 0x99, 0x60, 0x81, 0x88, 0x59, 0x80, 0x02, 0x89, 0x9A, 0x80, 0x1C, 0x70, 0x4F, 0xF0, 0x00, 0x53,
+ 0xA3, 0xF8, 0x1A, 0x21, 0xA3, 0xF8, 0x1E, 0x11, 0xC0, 0x88, 0xA3, 0xF8, 0x20, 0x01, 0x47, 0x48,
+ 0x01, 0x22, 0xC0, 0xF8, 0xD8, 0x21, 0xC0, 0xF8, 0xDC, 0x21, 0xC0, 0xF8, 0xC4, 0x21, 0xC0, 0xF8,
+ 0xCC, 0x21, 0xC0, 0xF8, 0xD4, 0x21, 0xC0, 0xF8, 0x80, 0x21, 0x72, 0xB6, 0xD0, 0xF8, 0x08, 0x12,
+ 0x00, 0x29, 0xFB, 0xD1, 0x42, 0x60, 0x62, 0xB6, 0x10, 0xBD, 0x2D, 0xE9, 0xF0, 0x41, 0x3B, 0x4D,
+ 0xD5, 0xF8, 0x5C, 0x01, 0x38, 0x4F, 0x2E, 0x04, 0x01, 0x24, 0xBA, 0x68, 0x50, 0xB3, 0xB6, 0xF8,
+ 0x1C, 0x81, 0x5F, 0xEA, 0x08, 0x00, 0x0F, 0xD0, 0xD5, 0xF8, 0x54, 0x01, 0x10, 0xB9, 0xD5, 0xF8,
+ 0x00, 0x01, 0x48, 0xB1, 0x41, 0x46, 0x01, 0x20, 0x90, 0x47, 0xC5, 0xF8, 0xD8, 0x42, 0xD5, 0xF8,
+ 0x54, 0x01, 0x08, 0xB1, 0xC5, 0xF8, 0xD4, 0x42, 0xBA, 0x68, 0x41, 0x46, 0x02, 0x20, 0x90, 0x47,
+ 0x01, 0x78, 0x02, 0x29, 0x02, 0xD0, 0x03, 0x29, 0x02, 0xD0, 0x08, 0xE0, 0x41, 0x88, 0x00, 0xE0,
+ 0x79, 0x88, 0xA6, 0xF8, 0x1E, 0x11, 0x80, 0x88, 0xA6, 0xF8, 0x1A, 0x01, 0x3C, 0x70, 0xC5, 0xF8,
+ 0xDC, 0x42, 0x38, 0xE0, 0xD5, 0xF8, 0x44, 0x01, 0x80, 0xB1, 0xB6, 0xF8, 0x1C, 0x11, 0x03, 0x20,
+ 0x90, 0x47, 0x38, 0x78, 0x38, 0xB1, 0x78, 0x88, 0xA6, 0xF8, 0x1E, 0x01, 0xB8, 0x88, 0xA6, 0xF8,
+ 0x1A, 0x01, 0x00, 0x20, 0x38, 0x70, 0xC5, 0xF8, 0xC4, 0x42, 0x24, 0xE0, 0xD5, 0xF8, 0x58, 0x01,
+ 0x30, 0xB1, 0xB6, 0xF8, 0x1C, 0x11, 0x01, 0x20, 0x90, 0x47, 0xC5, 0xF8, 0xD8, 0x42, 0x1A, 0xE0,
+ 0xD5, 0xF8, 0x4C, 0x01, 0x28, 0xB9, 0xD5, 0xF8, 0x54, 0x01, 0x10, 0xB9, 0xD5, 0xF8, 0x00, 0x01,
+ 0x88, 0xB1, 0xB6, 0xF8, 0x1C, 0x11, 0x19, 0xB1, 0x01, 0x20, 0x90, 0x47, 0xC5, 0xF8, 0xD8, 0x42,
+ 0xD5, 0xF8, 0x4C, 0x01, 0x10, 0xB1, 0xC5, 0xF8, 0xCC, 0x42, 0x04, 0xE0, 0xD5, 0xF8, 0x54, 0x01,
+ 0x08, 0xB1, 0xC5, 0xF8, 0xD4, 0x42, 0xD5, 0xF8, 0x00, 0x01, 0x00, 0x28, 0x01, 0xD0, 0xC5, 0xF8,
+ 0x80, 0x42, 0xBD, 0xE8, 0xF0, 0x81, 0x00, 0x00, 0xE8, 0x07, 0x10, 0x00, 0x00, 0x20, 0x00, 0x22,
+ 0xF6, 0x49, 0x00, 0x20, 0x08, 0x60, 0x09, 0x1D, 0x08, 0x60, 0x09, 0x1D, 0x08, 0x60, 0x09, 0x1D,
+ 0x08, 0x60, 0x09, 0x1D, 0x08, 0x60, 0x09, 0x1D, 0x08, 0x60, 0x09, 0x1D, 0x08, 0x60, 0x09, 0x1D,
+ 0x08, 0x60, 0xEF, 0x49, 0x01, 0x22, 0x0A, 0x60, 0xED, 0x49, 0x14, 0x31, 0x08, 0x60, 0x51, 0x07,
+ 0x01, 0xF8, 0x50, 0x0F, 0xFF, 0x23, 0x4B, 0x70, 0x01, 0xF8, 0x05, 0x0C, 0x10, 0x23, 0x4B, 0x80,
+ 0x8A, 0x80, 0xE8, 0x49, 0x08, 0x60, 0x48, 0x60, 0x88, 0x60, 0xC8, 0x60, 0x08, 0x61, 0x48, 0x61,
+ 0x88, 0x61, 0xC8, 0x61, 0x08, 0x62, 0x70, 0x47, 0x70, 0x47, 0xE2, 0x4A, 0x11, 0x60, 0xDF, 0x49,
+ 0x00, 0x22, 0x0A, 0x60, 0x00, 0xEB, 0x80, 0x00, 0x4F, 0xF6, 0xFF, 0x72, 0x02, 0xEA, 0x40, 0x00,
+ 0x4F, 0xF0, 0x00, 0x52, 0xA2, 0xF8, 0x58, 0x00, 0x01, 0x20, 0x08, 0x60, 0x70, 0x47, 0xD9, 0x4B,
+ 0x10, 0xB5, 0xD6, 0x4A, 0x19, 0x60, 0x00, 0x21, 0x11, 0x60, 0x4F, 0xF0, 0x00, 0x51, 0x91, 0xF8,
+ 0x51, 0x40, 0xE4, 0x07, 0x07, 0xD0, 0x1B, 0x6A, 0x58, 0x43, 0x80, 0x09, 0xA1, 0xF8, 0x58, 0x00,
+ 0x01, 0x20, 0x10, 0x60, 0x10, 0xBD, 0x00, 0xEB, 0xC0, 0x03, 0x03, 0xEB, 0x00, 0x10, 0x4F, 0xF6,
+ 0xFF, 0x73, 0x03, 0xEA, 0x80, 0x00, 0xF1, 0xE7, 0x4F, 0xF0, 0x00, 0x50, 0xB0, 0xF8, 0x58, 0x00,
+ 0x70, 0x47, 0x4F, 0xF0, 0x00, 0x50, 0x90, 0xF8, 0x22, 0x00, 0xC0, 0xF3, 0x81, 0x00, 0x28, 0xB1,
+ 0x01, 0x28, 0x05, 0xD0, 0x02, 0x28, 0x05, 0xD0, 0xC3, 0x48, 0x70, 0x47, 0xC3, 0x48, 0x70, 0x47,
+ 0xC3, 0x48, 0x70, 0x47, 0xC3, 0x48, 0x70, 0x47, 0x70, 0xB5, 0x4F, 0xF0, 0x00, 0x54, 0x94, 0xF8,
+ 0x51, 0x00, 0xBA, 0x49, 0xC2, 0x07, 0x4F, 0xF0, 0x00, 0x00, 0x01, 0xD0, 0x08, 0x60, 0x70, 0xBD,
+ 0x65, 0x69, 0x72, 0xB6, 0x62, 0x69, 0x42, 0xF0, 0x04, 0x02, 0x62, 0x61, 0x08, 0x60, 0xB4, 0xF8,
+ 0x58, 0x10, 0x09, 0xB1, 0xA4, 0xF8, 0x58, 0x00, 0xFF, 0xF7, 0x75, 0xF9, 0x01, 0x46, 0xFF, 0xF7,
+ 0xD0, 0xFF, 0xB1, 0xFB, 0xF0, 0xF0, 0xEA, 0xF7, 0x63, 0xFA, 0x65, 0x61, 0x02, 0x20, 0x00, 0xF0,
+ 0x35, 0xFB, 0x62, 0xB6, 0x70, 0xBD, 0xAB, 0x48, 0x10, 0xB5, 0x00, 0x21, 0x04, 0x68, 0x01, 0x60,
+ 0xFF, 0xF7, 0xD2, 0xFF, 0x1C, 0xB1, 0xA4, 0x46, 0xBD, 0xE8, 0x10, 0x40, 0x60, 0x47, 0x00, 0x20,
+ 0xFF, 0xF7, 0x82, 0xFF, 0x10, 0xBD, 0xA3, 0x4A, 0x51, 0x60, 0xA0, 0x49, 0x00, 0x22, 0x09, 0x1D,
+ 0x0A, 0x60, 0x00, 0xEB, 0x80, 0x00, 0x4F, 0xF6, 0xFF, 0x72, 0x02, 0xEA, 0x40, 0x00, 0xCA, 0x06,
+ 0xA2, 0xF8, 0x5A, 0x00, 0x01, 0x20, 0x08, 0x60, 0x70, 0x47, 0x9A, 0x4B, 0x10, 0xB5, 0x59, 0x60,
+ 0x96, 0x4A, 0x00, 0x21, 0x12, 0x1D, 0x11, 0x60, 0xD1, 0x06, 0x91, 0xF8, 0x51, 0x40, 0xA4, 0x07,
+ 0x08, 0xD4, 0x00, 0xEB, 0xC0, 0x03, 0x03, 0xEB, 0x00, 0x10, 0x4F, 0xF6, 0xFF, 0x73, 0x03, 0xEA,
+ 0x80, 0x00, 0x02, 0xE0, 0x1B, 0x6A, 0x58, 0x43, 0x80, 0x09, 0xA1, 0xF8, 0x5A, 0x00, 0x01, 0x20,
+ 0x10, 0x60, 0x10, 0xBD, 0x4F, 0xF0, 0x00, 0x50, 0xB0, 0xF8, 0x5A, 0x00, 0x70, 0x47, 0x70, 0xB5,
+ 0x4F, 0xF0, 0x00, 0x54, 0x94, 0xF8, 0x51, 0x00, 0x8B, 0x49, 0x82, 0x07, 0x4F, 0xF0, 0x00, 0x00,
+ 0x1A, 0xD4, 0x65, 0x69, 0x72, 0xB6, 0x62, 0x69, 0x42, 0xF0, 0x08, 0x02, 0x62, 0x61, 0x08, 0x60,
+ 0xB4, 0xF8, 0x5A, 0x10, 0x09, 0xB1, 0xA4, 0xF8, 0x5A, 0x00, 0xFF, 0xF7, 0x0C, 0xF9, 0x01, 0x46,
+ 0xFF, 0xF7, 0x67, 0xFF, 0xB1, 0xFB, 0xF0, 0xF0, 0xEA, 0xF7, 0xFA, 0xF9, 0x65, 0x61, 0x03, 0x20,
+ 0x00, 0xF0, 0xCC, 0xFA, 0x62, 0xB6, 0x70, 0xBD, 0x08, 0x60, 0x70, 0xBD, 0x75, 0x48, 0x10, 0xB5,
+ 0x00, 0x21, 0x44, 0x68, 0x41, 0x60, 0xFF, 0xF7, 0xD2, 0xFF, 0x1C, 0xB1, 0xA4, 0x46, 0xBD, 0xE8,
+ 0x10, 0x40, 0x60, 0x47, 0x01, 0x20, 0xFF, 0xF7, 0x17, 0xFF, 0x10, 0xBD, 0x6D, 0x4A, 0x91, 0x60,
+ 0x71, 0x49, 0x00, 0x22, 0x09, 0x1D, 0x0A, 0x60, 0x00, 0xEB, 0x80, 0x00, 0x4F, 0xF6, 0xFF, 0x72,
+ 0x02, 0xEA, 0x40, 0x00, 0x8A, 0x06, 0xA2, 0xF8, 0x5C, 0x00, 0x01, 0x20, 0x08, 0x60, 0x70, 0x47,
+ 0x64, 0x4B, 0x10, 0xB5, 0x99, 0x60, 0x68, 0x4A, 0x00, 0x21, 0x12, 0x1D, 0x11, 0x60, 0x91, 0x06,
+ 0x91, 0xF8, 0x51, 0x40, 0x64, 0x07, 0x08, 0xD4, 0x00, 0xEB, 0xC0, 0x03, 0x03, 0xEB, 0x00, 0x10,
+ 0x4F, 0xF6, 0xFF, 0x73, 0x03, 0xEA, 0x80, 0x00, 0x02, 0xE0, 0x1B, 0x6A, 0x58, 0x43, 0x80, 0x09,
+ 0xA1, 0xF8, 0x5C, 0x00, 0x01, 0x20, 0x10, 0x60, 0x10, 0xBD, 0x4F, 0xF0, 0x00, 0x50, 0xB0, 0xF8,
+ 0x5C, 0x00, 0x70, 0x47, 0x70, 0xB5, 0x4F, 0xF0, 0x00, 0x54, 0x94, 0xF8, 0x51, 0x00, 0x57, 0x49,
+ 0x42, 0x07, 0x4F, 0xF0, 0x00, 0x00, 0x1A, 0xD4, 0x65, 0x69, 0x72, 0xB6, 0x62, 0x69, 0x42, 0xF0,
+ 0x10, 0x02, 0x62, 0x61, 0x08, 0x60, 0xB4, 0xF8, 0x5C, 0x10, 0x09, 0xB1, 0xA4, 0xF8, 0x5C, 0x00,
+ 0xFF, 0xF7, 0xA1, 0xF8, 0x01, 0x46, 0xFF, 0xF7, 0xFC, 0xFE, 0xB1, 0xFB, 0xF0, 0xF0, 0xEA, 0xF7,
+ 0x8F, 0xF9, 0x65, 0x61, 0x04, 0x20, 0x00, 0xF0, 0x61, 0xFA, 0x62, 0xB6, 0x70, 0xBD, 0x08, 0x60,
+ 0x70, 0xBD, 0x40, 0x48, 0x10, 0xB5, 0x00, 0x21, 0x84, 0x68, 0x81, 0x60, 0xFF, 0xF7, 0xD2, 0xFF,
+ 0x1C, 0xB1, 0xA4, 0x46, 0xBD, 0xE8, 0x10, 0x40, 0x60, 0x47, 0x02, 0x20, 0xFF, 0xF7, 0xAC, 0xFE,
+ 0x10, 0xBD, 0x38, 0x4A, 0xD1, 0x60, 0x3D, 0x49, 0x00, 0x22, 0x09, 0x1D, 0x0A, 0x60, 0x00, 0xEB,
+ 0x80, 0x00, 0x4F, 0xF6, 0xFF, 0x72, 0x02, 0xEA, 0x40, 0x00, 0x4F, 0xF0, 0x00, 0x52, 0xA2, 0xF8,
+ 0x5E, 0x00, 0x01, 0x20, 0x08, 0x60, 0x70, 0x47, 0x2E, 0x4B, 0x10, 0xB5, 0xD9, 0x60, 0x33, 0x4A,
+ 0x00, 0x21, 0x12, 0x1D, 0x11, 0x60, 0x4F, 0xF0, 0x00, 0x51, 0x91, 0xF8, 0x51, 0x40, 0x24, 0x07,
+ 0x08, 0xD4, 0x00, 0xEB, 0xC0, 0x03, 0x03, 0xEB, 0x00, 0x10, 0x4F, 0xF6, 0xFF, 0x73, 0x03, 0xEA,
+ 0x80, 0x00, 0x02, 0xE0, 0x1B, 0x6A, 0x58, 0x43, 0x80, 0x09, 0xA1, 0xF8, 0x5E, 0x00, 0x01, 0x20,
+ 0x10, 0x60, 0x10, 0xBD, 0x4F, 0xF0, 0x00, 0x50, 0xB0, 0xF8, 0x5E, 0x00, 0x70, 0x47, 0x70, 0xB5,
+ 0x4F, 0xF0, 0x00, 0x54, 0x94, 0xF8, 0x51, 0x00, 0x21, 0x49, 0x02, 0x07, 0x4F, 0xF0, 0x00, 0x00,
+ 0x1A, 0xD4, 0x65, 0x69, 0x72, 0xB6, 0x62, 0x69, 0x42, 0xF0, 0x20, 0x02, 0x62, 0x61, 0x08, 0x60,
+ 0xB4, 0xF8, 0x5E, 0x10, 0x09, 0xB1, 0xA4, 0xF8, 0x5E, 0x00, 0xFF, 0xF7, 0x34, 0xF8, 0x01, 0x46,
+ 0xFF, 0xF7, 0x8F, 0xFE, 0xB1, 0xFB, 0xF0, 0xF0, 0xEA, 0xF7, 0x22, 0xF9, 0x65, 0x61, 0x05, 0x20,
+ 0x00, 0xF0, 0xF4, 0xF9, 0x62, 0xB6, 0x70, 0xBD, 0x08, 0x60, 0x70, 0xBD, 0x09, 0x48, 0x10, 0xB5,
+ 0x00, 0x21, 0xC4, 0x68, 0xC1, 0x60, 0xFF, 0xF7, 0xD2, 0xFF, 0x1C, 0xB1, 0xA4, 0x46, 0xBD, 0xE8,
+ 0x10, 0x40, 0x60, 0x47, 0x03, 0x20, 0xFF, 0xF7, 0x3F, 0xFE, 0x10, 0xBD, 0x40, 0x09, 0x00, 0x22,
+ 0xC0, 0x04, 0x00, 0x22, 0xF4, 0x07, 0x10, 0x00, 0x00, 0x09, 0x3D, 0x00, 0x00, 0x48, 0xE8, 0x01,
+ 0x00, 0x24, 0xF4, 0x00, 0x00, 0x12, 0x7A, 0x00, 0x44, 0x09, 0x00, 0x22, 0x48, 0x09, 0x00, 0x22,
+ 0x4C, 0x09, 0x00, 0x22, 0xEB, 0x4A, 0x11, 0x61, 0xEB, 0x49, 0x00, 0x22, 0x0A, 0x60, 0x00, 0xEB,
+ 0x80, 0x00, 0x4F, 0xF6, 0xFF, 0x72, 0x02, 0xEA, 0x40, 0x00, 0x4F, 0xF0, 0x00, 0x52, 0xA2, 0xF8,
+ 0x60, 0x00, 0x01, 0x20, 0x08, 0x60, 0x70, 0x47, 0xE2, 0x4B, 0x10, 0xB5, 0xE2, 0x4A, 0x19, 0x61,
+ 0x00, 0x21, 0x11, 0x60, 0x4F, 0xF0, 0x00, 0x51, 0x91, 0xF8, 0x51, 0x40, 0xE4, 0x06, 0x08, 0xD4,
+ 0x00, 0xEB, 0xC0, 0x03, 0x03, 0xEB, 0x00, 0x10, 0x4F, 0xF6, 0xFF, 0x73, 0x03, 0xEA, 0x80, 0x00,
+ 0x02, 0xE0, 0x1B, 0x6A, 0x58, 0x43, 0x80, 0x09, 0xA1, 0xF8, 0x60, 0x00, 0x01, 0x20, 0x10, 0x60,
+ 0x10, 0xBD, 0x4F, 0xF0, 0x00, 0x50, 0xB0, 0xF8, 0x60, 0x00, 0x70, 0x47, 0x70, 0xB5, 0x4F, 0xF0,
+ 0x00, 0x54, 0x94, 0xF8, 0x51, 0x00, 0xD0, 0x49, 0xC2, 0x06, 0x4F, 0xF0, 0x00, 0x00, 0x1A, 0xD4,
+ 0x65, 0x69, 0x72, 0xB6, 0x62, 0x69, 0x42, 0xF0, 0x40, 0x02, 0x62, 0x61, 0x08, 0x60, 0xB4, 0xF8,
+ 0x60, 0x10, 0x09, 0xB1, 0xA4, 0xF8, 0x60, 0x00, 0xFE, 0xF7, 0xB5, 0xFF, 0x01, 0x46, 0xFF, 0xF7,
+ 0x10, 0xFE, 0xB1, 0xFB, 0xF0, 0xF0, 0xEA, 0xF7, 0xA3, 0xF8, 0x65, 0x61, 0x06, 0x20, 0x00, 0xF0,
+ 0x75, 0xF9, 0x62, 0xB6, 0x70, 0xBD, 0x08, 0x60, 0x70, 0xBD, 0xBE, 0x48, 0x10, 0xB5, 0x00, 0x21,
+ 0x04, 0x69, 0x01, 0x61, 0xFF, 0xF7, 0xD2, 0xFF, 0x1C, 0xB1, 0xA4, 0x46, 0xBD, 0xE8, 0x10, 0x40,
+ 0x60, 0x47, 0x04, 0x20, 0xFF, 0xF7, 0xC0, 0xFD, 0x10, 0xBD, 0xB6, 0x4A, 0x51, 0x61, 0xB6, 0x49,
+ 0x00, 0x22, 0x09, 0x1D, 0x0A, 0x60, 0x00, 0xEB, 0x80, 0x00, 0x4F, 0xF6, 0xFF, 0x72, 0x02, 0xEA,
+ 0x40, 0x00, 0x4F, 0xF0, 0x00, 0x52, 0xA2, 0xF8, 0x62, 0x00, 0x01, 0x20, 0x08, 0x60, 0x70, 0x47,
+ 0xAC, 0x4B, 0x10, 0xB5, 0x59, 0x61, 0xAC, 0x4A, 0x00, 0x21, 0x12, 0x1D, 0x11, 0x60, 0x4F, 0xF0,
+ 0x00, 0x51, 0x91, 0xF8, 0x51, 0x40, 0xA4, 0x06, 0x08, 0xD4, 0x00, 0xEB, 0xC0, 0x03, 0x03, 0xEB,
+ 0x00, 0x10, 0x4F, 0xF6, 0xFF, 0x73, 0x03, 0xEA, 0x80, 0x00, 0x02, 0xE0, 0x1B, 0x6A, 0x58, 0x43,
+ 0x80, 0x09, 0xA1, 0xF8, 0x62, 0x00, 0x01, 0x20, 0x10, 0x60, 0x10, 0xBD, 0x4F, 0xF0, 0x00, 0x50,
+ 0xB0, 0xF8, 0x62, 0x00, 0x70, 0x47, 0x70, 0xB5, 0x4F, 0xF0, 0x00, 0x54, 0x94, 0xF8, 0x51, 0x00,
+ 0x9A, 0x49, 0x82, 0x06, 0x4F, 0xF0, 0x00, 0x00, 0x1A, 0xD4, 0x65, 0x69, 0x72, 0xB6, 0x62, 0x69,
+ 0x42, 0xF0, 0x80, 0x02, 0x62, 0x61, 0x08, 0x60, 0xB4, 0xF8, 0x62, 0x10, 0x09, 0xB1, 0xA4, 0xF8,
+ 0x62, 0x00, 0xFE, 0xF7, 0x48, 0xFF, 0x01, 0x46, 0xFF, 0xF7, 0xA3, 0xFD, 0xB1, 0xFB, 0xF0, 0xF0,
+ 0xEA, 0xF7, 0x36, 0xF8, 0x65, 0x61, 0x07, 0x20, 0x00, 0xF0, 0x08, 0xF9, 0x62, 0xB6, 0x70, 0xBD,
+ 0x08, 0x60, 0x70, 0xBD, 0x87, 0x48, 0x10, 0xB5, 0x00, 0x21, 0x44, 0x69, 0x41, 0x61, 0xFF, 0xF7,
+ 0xD2, 0xFF, 0x1C, 0xB1, 0xA4, 0x46, 0xBD, 0xE8, 0x10, 0x40, 0x60, 0x47, 0x05, 0x20, 0xFF, 0xF7,
+ 0x53, 0xFD, 0x10, 0xBD, 0x7F, 0x4A, 0x91, 0x61, 0x80, 0x49, 0x00, 0x22, 0x09, 0x1D, 0x0A, 0x60,
+ 0x00, 0xEB, 0x80, 0x00, 0x4F, 0xF6, 0xFF, 0x72, 0x02, 0xEA, 0x40, 0x00, 0x4F, 0xF0, 0x00, 0x52,
+ 0xA2, 0xF8, 0x64, 0x00, 0x01, 0x20, 0x08, 0x60, 0x70, 0x47, 0x76, 0x4B, 0x10, 0xB5, 0x99, 0x61,
+ 0x76, 0x4A, 0x00, 0x21, 0x12, 0x1D, 0x11, 0x60, 0x4F, 0xF0, 0x00, 0x51, 0x91, 0xF8, 0x51, 0x40,
+ 0x64, 0x06, 0x08, 0xD4, 0x00, 0xEB, 0xC0, 0x03, 0x03, 0xEB, 0x00, 0x10, 0x4F, 0xF6, 0xFF, 0x73,
+ 0x03, 0xEA, 0x80, 0x00, 0x02, 0xE0, 0x1B, 0x6A, 0x58, 0x43, 0x80, 0x09, 0xA1, 0xF8, 0x64, 0x00,
+ 0x01, 0x20, 0x10, 0x60, 0x10, 0xBD, 0x4F, 0xF0, 0x00, 0x50, 0xB0, 0xF8, 0x64, 0x00, 0x70, 0x47,
+ 0x70, 0xB5, 0x4F, 0xF0, 0x00, 0x54, 0x94, 0xF8, 0x51, 0x00, 0x65, 0x49, 0x42, 0x06, 0x4F, 0xF0,
+ 0x00, 0x00, 0x1A, 0xD4, 0x65, 0x69, 0x72, 0xB6, 0x62, 0x69, 0x42, 0xF4, 0x80, 0x72, 0x62, 0x61,
+ 0x08, 0x60, 0xB4, 0xF8, 0x64, 0x10, 0x09, 0xB1, 0xA4, 0xF8, 0x64, 0x00, 0xFE, 0xF7, 0xDB, 0xFE,
+ 0x01, 0x46, 0xFF, 0xF7, 0x36, 0xFD, 0xB1, 0xFB, 0xF0, 0xF0, 0xE9, 0xF7, 0xC9, 0xFF, 0x65, 0x61,
+ 0x08, 0x20, 0x00, 0xF0, 0x9B, 0xF8, 0x62, 0xB6, 0x70, 0xBD, 0x08, 0x60, 0x70, 0xBD, 0x51, 0x48,
+ 0x10, 0xB5, 0x00, 0x21, 0x84, 0x69, 0x81, 0x61, 0xFF, 0xF7, 0xD2, 0xFF, 0x1C, 0xB1, 0xA4, 0x46,
+ 0xBD, 0xE8, 0x10, 0x40, 0x60, 0x47, 0x06, 0x20, 0xFF, 0xF7, 0xE6, 0xFC, 0x10, 0xBD, 0x10, 0xB5,
+ 0xFE, 0xF7, 0xA0, 0xFE, 0x4F, 0xF0, 0x00, 0x51, 0x91, 0xF8, 0x22, 0x20, 0x42, 0xF2, 0x10, 0x73,
+ 0xB0, 0xFB, 0xF3, 0xF0, 0xC2, 0xF3, 0x81, 0x02, 0x52, 0x1C, 0xD0, 0x40, 0xB1, 0xF8, 0x54, 0x20,
+ 0xC9, 0x6C, 0x51, 0x43, 0x89, 0x09, 0xB0, 0xFB, 0xF1, 0xF0, 0x10, 0xBD, 0x40, 0x49, 0x00, 0x22,
+ 0x09, 0x1D, 0x0A, 0x60, 0x3B, 0x4A, 0x12, 0x6A, 0x50, 0x43, 0x4F, 0xF0, 0x00, 0x52, 0x80, 0x09,
+ 0xA2, 0xF8, 0x66, 0x00, 0x01, 0x20, 0x08, 0x60, 0x70, 0x47, 0x39, 0x49, 0x00, 0x20, 0x09, 0x1D,
+ 0x08, 0x60, 0x70, 0x47, 0x33, 0x49, 0xC8, 0x61, 0x35, 0x49, 0x01, 0x20, 0xA0, 0x31, 0x08, 0x60,
+ 0x09, 0x1D, 0x08, 0x60, 0x70, 0x47, 0x2F, 0x4A, 0x10, 0xB5, 0x00, 0x20, 0xD1, 0x69, 0xD0, 0x61,
+ 0x51, 0xB1, 0x2F, 0x4A, 0xA0, 0x32, 0x10, 0x60, 0x4F, 0xF0, 0x00, 0x50, 0xC0, 0x6C, 0xBD, 0xE8,
+ 0x10, 0x40, 0x20, 0xF0, 0x7F, 0x40, 0x08, 0x47, 0xFF, 0x20, 0xFF, 0xF7, 0x9D, 0xFC, 0x10, 0xBD,
+ 0x10, 0xB5, 0x04, 0x46, 0xFE, 0xF7, 0x56, 0xFE, 0x4F, 0xF0, 0x00, 0x51, 0x91, 0xF8, 0x22, 0x10,
+ 0x42, 0xF2, 0x10, 0x72, 0xB0, 0xFB, 0xF2, 0xF0, 0xC1, 0xF3, 0x81, 0x01, 0x49, 0x1C, 0xC8, 0x40,
+ 0x00, 0x03, 0xB0, 0xFB, 0xF4, 0xF0, 0x1B, 0x49, 0x08, 0x62, 0x10, 0xBD, 0x30, 0xB5, 0x4F, 0xF0,
+ 0x00, 0x55, 0x01, 0x24, 0x84, 0x40, 0x95, 0xF8, 0x51, 0x00, 0x09, 0xB1, 0x20, 0x43, 0x00, 0xE0,
+ 0xA0, 0x43, 0x85, 0xF8, 0x51, 0x00, 0x95, 0xF8, 0x4B, 0x00, 0x0A, 0xB1, 0x20, 0x43, 0x00, 0xE0,
+ 0xA0, 0x43, 0x85, 0xF8, 0x4B, 0x00, 0x15, 0xF8, 0x50, 0x0F, 0x0B, 0xB1, 0x20, 0x43, 0x00, 0xE0,
+ 0xA0, 0x43, 0x28, 0x70, 0x30, 0xBD, 0x4F, 0xF0, 0x00, 0x50, 0x10, 0xF8, 0x4A, 0x1F, 0xC0, 0x79,
+ 0x81, 0x43, 0x01, 0xD0, 0x01, 0x20, 0x70, 0x47, 0x00, 0x20, 0x70, 0x47, 0x00, 0xF0, 0x1F, 0x02,
+ 0x01, 0x21, 0x91, 0x40, 0x40, 0x09, 0x80, 0x00, 0x00, 0xF1, 0xE0, 0x20, 0xC0, 0xF8, 0x80, 0x12,
+ 0x70, 0x47, 0x00, 0x00, 0xF4, 0x07, 0x10, 0x00, 0x50, 0x09, 0x00, 0x22, 0x54, 0x09, 0x00, 0x22,
+ 0x58, 0x09, 0x00, 0x22, 0x00, 0x21, 0x01, 0x60, 0x08, 0x46, 0x70, 0x47, 0x30, 0xB5, 0x0A, 0x68,
+ 0x12, 0xB1, 0x0B, 0x46, 0x84, 0x6A, 0x08, 0xE0, 0x00, 0x22, 0x08, 0x60, 0xC0, 0xE9, 0x03, 0x21,
+ 0x0A, 0xE0, 0x02, 0xF1, 0x0C, 0x03, 0xD2, 0x68, 0x12, 0xB1, 0x95, 0x6A, 0xA5, 0x42, 0xF8, 0xDC,
+ 0x1A, 0x68, 0xC0, 0xE9, 0x03, 0x21, 0x18, 0x60, 0x00, 0x20, 0x30, 0xBD, 0x00, 0x68, 0x00, 0x22,
+ 0x01, 0xE0, 0xC0, 0x68, 0x52, 0x1C, 0x8A, 0x42, 0x01, 0xD0, 0x00, 0x28, 0xF9, 0xD1, 0x70, 0x47,
+ 0x00, 0x68, 0x70, 0x47, 0x02, 0x69, 0x0A, 0xB1, 0x11, 0x68, 0x31, 0xB9, 0x6F, 0xF0, 0x04, 0x00,
+ 0x70, 0x47, 0x01, 0xF1, 0x0C, 0x02, 0xC9, 0x68, 0x39, 0xB1, 0x81, 0x42, 0xF9, 0xD1, 0xC9, 0x68,
+ 0x11, 0x60, 0x00, 0x21, 0x01, 0x61, 0x08, 0x46, 0x70, 0x47, 0x6F, 0xF0, 0x03, 0x00, 0x70, 0x47,
+ 0xC1, 0x7D, 0x01, 0x29, 0x05, 0xD1, 0x49, 0x07, 0x11, 0xF8, 0x2D, 0x2F, 0x42, 0xF0, 0x02, 0x02,
+ 0x0A, 0x70, 0x14, 0x49, 0x02, 0x68, 0x4A, 0x60, 0x42, 0x68, 0x8A, 0x60, 0x80, 0x7D, 0x08, 0x70,
+ 0x70, 0x47, 0x30, 0xB4, 0x4F, 0xF0, 0x00, 0x50, 0x10, 0xF8, 0xE2, 0x1F, 0xC2, 0x89, 0x0D, 0x48,
+ 0xCC, 0x07, 0x43, 0x68, 0x03, 0xD0, 0x02, 0x24, 0x44, 0x70, 0x20, 0x46, 0x04, 0xE0, 0x01, 0x24,
+ 0x44, 0x70, 0x52, 0x1C, 0x92, 0xB2, 0x20, 0x46, 0x30, 0xBC, 0x18, 0x47, 0x4F, 0xF0, 0x00, 0x50,
+ 0x90, 0xF8, 0xE2, 0x10, 0x03, 0x48, 0x02, 0x78, 0x91, 0x42, 0x01, 0xD1, 0x80, 0x68, 0x00, 0x47,
+ 0x70, 0x47, 0x00, 0x00, 0x2C, 0x08, 0x10, 0x00, 0x53, 0xEA, 0x02, 0x0C, 0x00, 0xF0, 0x69, 0x80,
+ 0x2D, 0xE9, 0xF0, 0x4B, 0x4F, 0xF0, 0x00, 0x06, 0x00, 0x2B, 0x1F, 0xBF, 0xB3, 0xFA, 0x83, 0xF5,
+ 0x03, 0xFA, 0x05, 0xF4, 0x24, 0xFA, 0x05, 0xF6, 0x5E, 0x40, 0x12, 0xBF, 0x16, 0x43, 0xB2, 0xFA,
+ 0x82, 0xF5, 0x02, 0xFA, 0x05, 0xF4, 0xC5, 0xF1, 0x20, 0x05, 0x1E, 0xBF, 0x22, 0xFA, 0x05, 0xFC,
+ 0x44, 0xEA, 0x0C, 0x04, 0x20, 0x35, 0x56, 0xEA, 0x04, 0x4C, 0x4F, 0xEA, 0x14, 0x44, 0x18, 0xBF,
+ 0x64, 0x1C, 0x4F, 0xF0, 0x00, 0x08, 0x4F, 0xF0, 0x00, 0x09, 0x90, 0x42, 0x71, 0xEB, 0x03, 0x0C,
+ 0x39, 0xD3, 0x00, 0x29, 0x19, 0xBF, 0xB1, 0xFA, 0x81, 0xF7, 0x01, 0xFA, 0x07, 0xF6, 0xB0, 0xFA,
+ 0x80, 0xF7, 0x00, 0xFA, 0x07, 0xF6, 0xC7, 0xF1, 0x20, 0x07, 0x1E, 0xBF, 0x20, 0xFA, 0x07, 0xFC,
+ 0x46, 0xEA, 0x0C, 0x06, 0x20, 0x37, 0xB6, 0xFB, 0xF4, 0xFC, 0xA7, 0xEB, 0x05, 0x07, 0x10, 0x3F,
+ 0x07, 0xF0, 0x1F, 0x0B, 0xCB, 0xF1, 0x20, 0x06, 0x0C, 0xFA, 0x0B, 0xFB, 0x2C, 0xFA, 0x06, 0xF6,
+ 0x44, 0xBF, 0xB3, 0x46, 0x00, 0x26, 0x20, 0x2F, 0xA4, 0xBF, 0x5E, 0x46, 0x4F, 0xF0, 0x00, 0x0B,
+ 0x5B, 0xEA, 0x06, 0x0C, 0x08, 0xBF, 0x4F, 0xF0, 0x01, 0x0B, 0x19, 0xEB, 0x0B, 0x09, 0xAB, 0xFB,
+ 0x02, 0x7C, 0x48, 0xEB, 0x06, 0x08, 0xC0, 0x1B, 0x06, 0xFB, 0x02, 0xCC, 0x0B, 0xFB, 0x03, 0xCC,
+ 0x71, 0xEB, 0x0C, 0x01, 0xC1, 0xE7, 0x0B, 0x46, 0x02, 0x46, 0x41, 0x46, 0x48, 0x46, 0xBD, 0xE8,
+ 0xF0, 0x8B, 0x13, 0xB5, 0x4F, 0xF0, 0x00, 0x00, 0x4F, 0xF0, 0x00, 0x01, 0xAF, 0xF3, 0x00, 0x80,
+ 0xBD, 0xE8, 0x1C, 0x40, 0x70, 0x47, 0xB2, 0xF1, 0x20, 0x03, 0x0A, 0xD5, 0xC2, 0xF1, 0x20, 0x03,
+ 0x01, 0xFA, 0x02, 0xF1, 0x20, 0xFA, 0x03, 0xF3, 0x00, 0xFA, 0x02, 0xF0, 0x41, 0xEA, 0x03, 0x01,
+ 0x70, 0x47, 0x00, 0xFA, 0x03, 0xF1, 0x4F, 0xF0, 0x00, 0x00, 0x70, 0x47, 0x10, 0xB5, 0x4C, 0x10,
+ 0x84, 0xEA, 0x53, 0x04, 0x04, 0xD5, 0x40, 0x42, 0xC1, 0xF1, 0x00, 0x01, 0x38, 0xBF, 0x49, 0x1E,
+ 0x1B, 0x42, 0x04, 0xD5, 0x52, 0x42, 0xC3, 0xF1, 0x00, 0x03, 0x38, 0xBF, 0x5B, 0x1E, 0xFF, 0xF7,
+ 0x63, 0xFF, 0x14, 0xF0, 0x80, 0x4F, 0x04, 0xD0, 0x40, 0x42, 0xC1, 0xF1, 0x00, 0x01, 0x38, 0xBF,
+ 0x49, 0x1E, 0x14, 0xF0, 0x00, 0x4F, 0x04, 0xD0, 0x52, 0x42, 0xC3, 0xF1, 0x00, 0x03, 0x38, 0xBF,
+ 0x5B, 0x1E, 0x10, 0xBD, 0x03, 0x2A, 0x40, 0xF2, 0x30, 0x80, 0x10, 0xF0, 0x03, 0x0C, 0x00, 0xF0,
+ 0x15, 0x80, 0x11, 0xF8, 0x01, 0x3B, 0xBC, 0xF1, 0x02, 0x0F, 0x62, 0x44, 0x98, 0xBF, 0x11, 0xF8,
+ 0x01, 0xCB, 0x00, 0xF8, 0x01, 0x3B, 0x38, 0xBF, 0x11, 0xF8, 0x01, 0x3B, 0xA2, 0xF1, 0x04, 0x02,
+ 0x98, 0xBF, 0x00, 0xF8, 0x01, 0xCB, 0x38, 0xBF, 0x00, 0xF8, 0x01, 0x3B, 0x11, 0xF0, 0x03, 0x03,
+ 0x00, 0xF0, 0x25, 0x80, 0x08, 0x3A, 0xC0, 0xF0, 0x08, 0x80, 0x51, 0xF8, 0x04, 0x3B, 0x08, 0x3A,
+ 0x51, 0xF8, 0x04, 0xCB, 0xA0, 0xE8, 0x08, 0x10, 0xF5, 0xE7, 0x12, 0x1D, 0x5C, 0xBF, 0x51, 0xF8,
+ 0x04, 0x3B, 0x40, 0xF8, 0x04, 0x3B, 0xAF, 0xF3, 0x00, 0x80, 0xD2, 0x07, 0x24, 0xBF, 0x11, 0xF8,
+ 0x01, 0x3B, 0x11, 0xF8, 0x01, 0xCB, 0x48, 0xBF, 0x11, 0xF8, 0x01, 0x2B, 0x24, 0xBF, 0x00, 0xF8,
+ 0x01, 0x3B, 0x00, 0xF8, 0x01, 0xCB, 0x48, 0xBF, 0x00, 0xF8, 0x01, 0x2B, 0x70, 0x47, 0x10, 0xB5,
+ 0x20, 0x3A, 0xC0, 0xF0, 0x0B, 0x80, 0xB1, 0xE8, 0x18, 0x50, 0x20, 0x3A, 0xA0, 0xE8, 0x18, 0x50,
+ 0xB1, 0xE8, 0x18, 0x50, 0xA0, 0xE8, 0x18, 0x50, 0xBF, 0xF4, 0xF5, 0xAF, 0x5F, 0xEA, 0x02, 0x7C,
+ 0x24, 0xBF, 0xB1, 0xE8, 0x18, 0x50, 0xA0, 0xE8, 0x18, 0x50, 0x44, 0xBF, 0x18, 0xC9, 0x18, 0xC0,
+ 0xBD, 0xE8, 0x10, 0x40, 0x5F, 0xEA, 0x82, 0x7C, 0x24, 0xBF, 0x51, 0xF8, 0x04, 0x3B, 0x40, 0xF8,
+ 0x04, 0x3B, 0x08, 0xBF, 0x70, 0x47, 0xD2, 0x07, 0x28, 0xBF, 0x31, 0xF8, 0x02, 0x3B, 0x48, 0xBF,
+ 0x11, 0xF8, 0x01, 0x2B, 0x28, 0xBF, 0x20, 0xF8, 0x02, 0x3B, 0x48, 0xBF, 0x00, 0xF8, 0x01, 0x2B,
+ 0x70, 0x47, 0x02, 0xF0, 0xFF, 0x03, 0x43, 0xEA, 0x03, 0x22, 0x42, 0xEA, 0x02, 0x42, 0x00, 0xF0,
+ 0x02, 0xB8, 0x4F, 0xF0, 0x00, 0x02, 0x04, 0x29, 0xC0, 0xF0, 0x12, 0x80, 0x10, 0xF0, 0x03, 0x0C,
+ 0x00, 0xF0, 0x1B, 0x80, 0xCC, 0xF1, 0x04, 0x0C, 0xBC, 0xF1, 0x02, 0x0F, 0x18, 0xBF, 0x00, 0xF8,
+ 0x01, 0x2B, 0xA8, 0xBF, 0x20, 0xF8, 0x02, 0x2B, 0xA1, 0xEB, 0x0C, 0x01, 0x00, 0xF0, 0x0D, 0xB8,
+ 0x5F, 0xEA, 0xC1, 0x7C, 0x24, 0xBF, 0x00, 0xF8, 0x01, 0x2B, 0x00, 0xF8, 0x01, 0x2B, 0x48, 0xBF,
+ 0x00, 0xF8, 0x01, 0x2B, 0x70, 0x47, 0x4F, 0xF0, 0x00, 0x02, 0x00, 0xB5, 0x13, 0x46, 0x94, 0x46,
+ 0x96, 0x46, 0x20, 0x39, 0x22, 0xBF, 0xA0, 0xE8, 0x0C, 0x50, 0xA0, 0xE8, 0x0C, 0x50, 0xB1, 0xF1,
+ 0x20, 0x01, 0xBF, 0xF4, 0xF7, 0xAF, 0x09, 0x07, 0x28, 0xBF, 0xA0, 0xE8, 0x0C, 0x50, 0x48, 0xBF,
+ 0x0C, 0xC0, 0x5D, 0xF8, 0x04, 0xEB, 0x89, 0x00, 0x28, 0xBF, 0x40, 0xF8, 0x04, 0x2B, 0x08, 0xBF,
+ 0x70, 0x47, 0x48, 0xBF, 0x20, 0xF8, 0x02, 0x2B, 0x11, 0xF0, 0x80, 0x4F, 0x18, 0xBF, 0x00, 0xF8,
+ 0x01, 0x2B, 0x70, 0x47, 0x04, 0x46, 0xAF, 0xF3, 0x00, 0x80, 0x20, 0x46, 0xE9, 0xF7, 0x36, 0xFC,
+ 0x01, 0x49, 0x18, 0x20, 0xAB, 0xBE, 0xFE, 0xE7, 0x26, 0x00, 0x02, 0x00, 0x70, 0x47, 0x00, 0x00,
+ 0x01, 0x49, 0x00, 0x20, 0x08, 0x60, 0x70, 0x47, 0x3C, 0x05, 0x00, 0x22, 0x2D, 0xE9, 0xF0, 0x41,
+ 0x0F, 0x4F, 0x00, 0x24, 0x4F, 0xF0, 0x01, 0x08, 0x4F, 0xF0, 0x00, 0x56, 0x08, 0xFA, 0x04, 0xF0,
+ 0xC5, 0xB2, 0x96, 0xF8, 0x2F, 0x00, 0x28, 0x42, 0x0C, 0xD0, 0x96, 0xF8, 0x39, 0x00, 0x28, 0x42,
+ 0x08, 0xD0, 0x57, 0xF8, 0x24, 0x00, 0x00, 0xB1, 0x80, 0x47, 0x96, 0xF8, 0x2F, 0x00, 0xA8, 0x43,
+ 0x86, 0xF8, 0x2F, 0x00, 0x64, 0x1C, 0x06, 0x2C, 0xE8, 0xDB, 0xBD, 0xE8, 0xF0, 0x81, 0x00, 0x00,
+ 0x1C, 0x88, 0x10, 0x00, 0x2D, 0xE9, 0xF0, 0x41, 0x0F, 0x4F, 0x00, 0x24, 0x4F, 0xF0, 0x01, 0x08,
+ 0x4F, 0xF0, 0x00, 0x56, 0x08, 0xFA, 0x04, 0xF0, 0xC5, 0xB2, 0x96, 0xF8, 0x2E, 0x00, 0x28, 0x42,
+ 0x0C, 0xD0, 0x96, 0xF8, 0x38, 0x00, 0x28, 0x42, 0x08, 0xD0, 0x57, 0xF8, 0x24, 0x00, 0x00, 0xB1,
+ 0x80, 0x47, 0x96, 0xF8, 0x2E, 0x00, 0xA8, 0x43, 0x86, 0xF8, 0x2E, 0x00, 0x64, 0x1C, 0x06, 0x2C,
+ 0xE8, 0xDB, 0xBD, 0xE8, 0xF0, 0x81, 0x00, 0x00, 0xFC, 0x87, 0x10, 0x00, 0x01, 0x21, 0x81, 0x40,
+ 0xC8, 0xB2, 0x4F, 0xF0, 0x00, 0x51, 0x91, 0xF8, 0x33, 0x10, 0x01, 0x42, 0x01, 0xD0, 0x01, 0x20,
+ 0x70, 0x47, 0x00, 0x20, 0x70, 0x47, 0x00, 0x00, 0x01, 0x49, 0x01, 0x20, 0x08, 0x60, 0x70, 0x47,
+ 0x3C, 0x05, 0x00, 0x22, 0x00, 0x48, 0x70, 0x47, 0x59, 0x07, 0x10, 0x00, 0x07, 0x48, 0x41, 0x78,
+ 0x49, 0xB1, 0x80, 0x78, 0x38, 0xB1, 0x06, 0x48, 0x80, 0x79, 0x01, 0x28, 0x03, 0xD1, 0x05, 0x48,
+ 0x00, 0x78, 0x01, 0x28, 0x00, 0xD0, 0x00, 0x20, 0x70, 0x47, 0x00, 0x00, 0x4C, 0x07, 0x10, 0x00,
+ 0xEC, 0x06, 0x10, 0x00, 0x04, 0x05, 0x10, 0x00, 0x03, 0x48, 0x41, 0x78, 0x11, 0xB9, 0x80, 0x78,
+ 0x00, 0x28, 0x00, 0xD0, 0x01, 0x20, 0x70, 0x47, 0x4C, 0x07, 0x10, 0x00, 0x01, 0x48, 0x00, 0x78,
+ 0x70, 0x47, 0x00, 0x00, 0x50, 0x06, 0x10, 0x00, 0xC0, 0x6E, 0x70, 0x47, 0x90, 0xF8, 0x73, 0x00,
+ 0x70, 0x47, 0x90, 0xF8, 0x72, 0x00, 0x70, 0x47, 0x05, 0x48, 0x41, 0x78, 0x82, 0x78, 0x11, 0x44,
+ 0x02, 0x78, 0xC0, 0x78, 0x10, 0x44, 0x08, 0x18, 0x00, 0xD0, 0x01, 0x20, 0x70, 0x47, 0x00, 0x00,
+ 0x0C, 0x06, 0x10, 0x00, 0x70, 0xB5, 0xD0, 0xE9, 0x00, 0x21, 0x4C, 0x08, 0x4F, 0xEA, 0x32, 0x03,
+ 0x96, 0x18, 0x41, 0xEB, 0x01, 0x05, 0x33, 0x43, 0x2C, 0x43, 0x13, 0x43, 0x0C, 0x43, 0xC0, 0xE9,
+ 0x00, 0x34, 0x70, 0xBD, 0x01, 0xEB, 0x81, 0x01, 0x00, 0xEB, 0xC1, 0x00, 0x29, 0x30, 0x70, 0x47,
+ 0x02, 0x4A, 0x12, 0x78, 0x51, 0x43, 0x00, 0xEB, 0x41, 0x00, 0x70, 0x47, 0xBA, 0x06, 0x10, 0x00,
+ 0x01, 0xEB, 0x81, 0x01, 0x00, 0xEB, 0xC1, 0x00, 0x29, 0x30, 0x70, 0x47, 0x02, 0x4A, 0x12, 0x78,
+ 0x51, 0x43, 0x00, 0xEB, 0x41, 0x00, 0x70, 0x47, 0xBA, 0x06, 0x10, 0x00, 0x10, 0x00, 0x00, 0x00,
+ 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xFF, 0x00, 0xBE, 0x94, 0xFE, 0xFF, 0x27, 0xC4,
+ 0xFE, 0xFF, 0x59, 0xF2, 0xFE, 0xFF, 0x6B, 0x1F, 0xFF, 0xFF, 0x81, 0x4B, 0xFF, 0xFF, 0xC5, 0x76,
+ 0xFF, 0xFF, 0x67, 0xA1, 0xFF, 0xFF, 0x97, 0xCB, 0xFF, 0xFF, 0x88, 0xF5, 0xFF, 0xFF, 0x6C, 0x1F,
+ 0x00, 0x00, 0x76, 0x49, 0x00, 0x00, 0xD9, 0x73, 0x00, 0x00, 0xC6, 0x9E, 0x00, 0x00, 0x6D, 0xCA,
+ 0x00, 0x00, 0xFC, 0xF6, 0x00, 0x00, 0x9A, 0x24, 0x01, 0x00, 0x66, 0x53, 0x01, 0x00, 0xA3, 0x8D,
+ 0x00, 0x00, 0x71, 0x74, 0x00, 0x00, 0x02, 0x60, 0x00, 0x00, 0xA2, 0x4F, 0x00, 0x00, 0xC8, 0x42,
+ 0x00, 0x00, 0x0C, 0x39, 0x00, 0x00, 0x21, 0x32, 0x00, 0x00, 0xD2, 0x2D, 0x00, 0x00, 0x00, 0x2C,
+ 0x00, 0x00, 0x9B, 0x2C, 0x00, 0x00, 0xA9, 0x2F, 0x00, 0x00, 0x40, 0x35, 0x00, 0x00, 0x8B, 0x3D,
+ 0x00, 0x00, 0xCB, 0x48, 0x00, 0x00, 0x59, 0x57, 0x00, 0x00, 0xAD, 0x69, 0x00, 0x00, 0x65, 0x80,
+ 0x00, 0x00, 0x30, 0x11, 0x00, 0x00, 0x4C, 0x1C, 0x00, 0x00, 0x2D, 0x29, 0x00, 0x00, 0x86, 0x37,
+ 0x00, 0x00, 0x24, 0x47, 0x00, 0x00, 0xE8, 0x57, 0x00, 0x00, 0xC4, 0x69, 0x00, 0x00, 0xB5, 0x7C,
+ 0x00, 0x00, 0xC4, 0x90, 0x00, 0x00, 0x03, 0xA6, 0x00, 0x00, 0x8F, 0xBC, 0x00, 0x00, 0x8C, 0xD4,
+ 0x00, 0x00, 0x29, 0xEE, 0x00, 0x00, 0x9C, 0x09, 0x01, 0x00, 0x2B, 0x27, 0x01, 0x00, 0x24, 0x47,
+ 0x01, 0x00, 0xE6, 0x69, 0x01, 0x00, 0x98, 0x08, 0x00, 0x00, 0x26, 0x0E, 0x00, 0x00, 0x97, 0x14,
+ 0x00, 0x00, 0xC3, 0x1B, 0x00, 0x00, 0x92, 0x23, 0x00, 0x00, 0xF4, 0x2B, 0x00, 0x00, 0xE2, 0x34,
+ 0x00, 0x00, 0x5A, 0x3E, 0x00, 0x00, 0x62, 0x48, 0x00, 0x00, 0x02, 0x53, 0x00, 0x00, 0x48, 0x5E,
+ 0x00, 0x00, 0x46, 0x6A, 0x00, 0x00, 0x14, 0x77, 0x00, 0x00, 0xCE, 0x84, 0x00, 0x00, 0x95, 0x93,
+ 0x00, 0x00, 0x92, 0xA3, 0x00, 0x00, 0xF3, 0xB4, 0x00, 0x00, 0x01, 0x00, 0x02, 0x00, 0x01, 0x00,
+ 0x02, 0x00, 0x04, 0x00, 0x02, 0x00, 0x01, 0x00, 0x02, 0x00, 0x01, 0x00, 0x10, 0x00, 0xFF, 0xFF,
+ 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0x14, 0x00, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF,
+ 0x0C, 0x00, 0xFF, 0x00, 0xFF, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x02, 0x02, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xE4, 0x40, 0x10, 0x00, 0x30, 0x46, 0x10, 0x00, 0x2B, 0xF5, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x07, 0xF4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x13, 0xF4, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x21, 0xF4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x25, 0xF4, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x4B, 0xF4, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0xAD, 0xF4, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0xC3, 0xF4, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xF7, 0xF4, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0xF9, 0xF4, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xFB, 0xF4, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0xFD, 0xF4, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0xFF, 0xF4, 0x00, 0x00,
+ 0x02, 0x00, 0x00, 0x00, 0x01, 0xF5, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x03, 0xF5, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0x29, 0xF5, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x6F, 0xF5, 0x00, 0x00,
+ 0x6B, 0xF5, 0x00, 0x00, 0xD5, 0xF5, 0x00, 0x00, 0x6B, 0xF5, 0x00, 0x00, 0x31, 0xF7, 0x00, 0x00,
+ 0x6B, 0xF5, 0x00, 0x00, 0x6B, 0xF5, 0x00, 0x00, 0x6B, 0xF5, 0x00, 0x00, 0x6B, 0xF5, 0x00, 0x00,
+ 0x6B, 0xF5, 0x00, 0x00, 0x9F, 0xF5, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xC1, 0xF5, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x6B, 0xF5, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0xF6, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0x25, 0xF6, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x49, 0xF6, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0x7F, 0xF6, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x2D, 0xF7, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0x5D, 0xF7, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0xBB, 0xF7, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0xD1, 0xF7, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x19, 0xF8, 0x00, 0x00,
+ 0x02, 0x00, 0x00, 0x00, 0xA7, 0xF8, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x8B, 0xF9, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0xCB, 0xF9, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x5F, 0xF8, 0x00, 0x00,
+ 0x02, 0x00, 0x00, 0x00, 0x25, 0xF9, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x6B, 0xF5, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x6B, 0xF5, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x77, 0xF7, 0x00, 0x00,
+ 0x81, 0xF7, 0x00, 0x00, 0x8B, 0xF7, 0x00, 0x00, 0x9B, 0xF7, 0x00, 0x00, 0xAB, 0xF7, 0x00, 0x00,
+ 0x61, 0xFB, 0x00, 0x00, 0x83, 0xFB, 0x00, 0x00, 0x8B, 0xFB, 0x00, 0x00, 0xAB, 0xFB, 0x00, 0x00,
+ 0x0D, 0xFC, 0x00, 0x00, 0x33, 0xFC, 0x00, 0x00, 0x43, 0xFC, 0x00, 0x00, 0x55, 0xFC, 0x00, 0x00,
+ 0x67, 0xFC, 0x00, 0x00, 0x79, 0xFC, 0x00, 0x00, 0x8B, 0xFC, 0x00, 0x00, 0x9D, 0xFC, 0x00, 0x00,
+ 0xAF, 0xFC, 0x00, 0x00, 0xC1, 0xFC, 0x00, 0x00, 0xD3, 0xFC, 0x00, 0x00, 0xE5, 0xFC, 0x00, 0x00,
+ 0x85, 0x86, 0x60, 0x61, 0xD0, 0x02, 0x0F, 0x01, 0x00, 0xA6, 0x02, 0x0F, 0x03, 0x00, 0xA8, 0x02,
+ 0x0B, 0x03, 0x00, 0x72, 0x00, 0x0B, 0x01, 0x00, 0x73, 0x00, 0x08, 0x01, 0x00, 0x00, 0x00, 0x00,
+ 0x49, 0x00, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x4D, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x41, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x53, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x43, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0xF3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xE3, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x53, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0xB4, 0x6C, 0x10, 0x00, 0x00, 0x04, 0x00, 0x00, 0x41, 0x35, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x02, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0xB4, 0x70, 0x10, 0x00,
+ 0x00, 0x04, 0x00, 0x00, 0x59, 0x44, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00,
+ 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0xB4, 0x74, 0x10, 0x00, 0x00, 0x08, 0x00, 0x00,
+ 0x19, 0xF1, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00,
+ 0x02, 0x00, 0x00, 0x00, 0xB4, 0x7C, 0x10, 0x00, 0x00, 0x02, 0x00, 0x00, 0x61, 0xFA, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00,
+ 0xB4, 0x7E, 0x10, 0x00, 0x80, 0x00, 0x00, 0x00, 0xF1, 0xF2, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00,
+ 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00,
+ 0x03, 0x00, 0x00, 0x00, 0x1D, 0x2F, 0x01, 0x00, 0xC7, 0x2E, 0x01, 0x00, 0xB3, 0x41, 0x01, 0x00,
+ 0x81, 0x26, 0x01, 0x00, 0x01, 0x31, 0x01, 0x00, 0x5D, 0x3C, 0x01, 0x00, 0x81, 0x27, 0x01, 0x00,
+ 0xBB, 0x2F, 0x01, 0x00, 0xC9, 0x41, 0x01, 0x00, 0x00, 0x04, 0x40, 0x00, 0x10, 0x21, 0x00, 0x00,
+ 0x00, 0x1F, 0x01, 0x00, 0x06, 0x00, 0x00, 0x00, 0x30, 0x00, 0x01, 0x00, 0x00, 0x04, 0x06, 0x00,
+ 0x08, 0x02, 0x50, 0x00, 0x60, 0x02, 0x0C, 0x0C, 0x00, 0x00, 0x60, 0x02, 0x0C, 0x30, 0x0C, 0x1A,
+ 0x31, 0x10, 0x0F, 0x0F, 0x00, 0x00, 0x2F, 0x00, 0x01, 0x00, 0xAA, 0x2C, 0x01, 0x00, 0x00, 0x00,
+ 0x03, 0x00, 0x00, 0x00, 0xFF, 0xFF, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x18, 0x00, 0x00, 0x00,
+ 0xFF, 0xFF, 0xFF, 0xFF, 0x3F, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x60, 0x02, 0x01, 0x80, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x0C, 0x00,
+ 0x0C, 0x00, 0x0C, 0x00, 0x0C, 0x00, 0x07, 0x01, 0x26, 0x26, 0x12, 0x12, 0x11, 0x00, 0x00, 0x00,
+ 0x00, 0x38, 0x00, 0x3C, 0x20, 0x3D, 0xD0, 0x41, 0xEC, 0x41, 0x08, 0x42, 0x1C, 0x42, 0x30, 0x42,
+ 0x44, 0x42, 0x60, 0x42, 0x74, 0x42, 0x88, 0x42, 0xA4, 0x42, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x20, 0x00, 0x01, 0x01, 0x02,
+ 0x01, 0x02, 0x02, 0x03, 0x01, 0x02, 0x02, 0x03, 0x02, 0x03, 0x03, 0x04, 0x00, 0x00, 0x00, 0x00,
+ 0x1B, 0x00, 0x00, 0x80, 0x2D, 0x00, 0x00, 0x80, 0x36, 0x00, 0x00, 0x00, 0x41, 0x00, 0x00, 0x80,
+ 0x5A, 0x00, 0x00, 0x00, 0x6C, 0x00, 0x00, 0x00, 0x77, 0x00, 0x00, 0x80, 0x99, 0x00, 0x00, 0x80,
+ 0x82, 0x00, 0x00, 0x00, 0xB4, 0x00, 0x00, 0x00, 0xAF, 0x00, 0x00, 0x80, 0xD8, 0x00, 0x00, 0x00,
+ 0xC3, 0x00, 0x00, 0x80, 0xF5, 0x00, 0x00, 0x80, 0xEE, 0x00, 0x00, 0x00, 0x29, 0x01, 0x00, 0x80,
+ 0x32, 0x01, 0x00, 0x00, 0x04, 0x01, 0x00, 0x00, 0x1F, 0x01, 0x00, 0x80, 0x68, 0x01, 0x00, 0x00,
+ 0x73, 0x01, 0x00, 0x80, 0x45, 0x01, 0x00, 0x80, 0x5E, 0x01, 0x00, 0x00, 0xB0, 0x01, 0x00, 0x00,
+ 0xAB, 0x01, 0x00, 0x80, 0x9D, 0x01, 0x00, 0x80, 0x86, 0x01, 0x00, 0x00, 0xF1, 0x01, 0x00, 0x80,
+ 0xEA, 0x01, 0x00, 0x00, 0xDC, 0x01, 0x00, 0x00, 0xC7, 0x01, 0x00, 0x80, 0x49, 0x02, 0x00, 0x80,
+ 0x52, 0x02, 0x00, 0x00, 0x64, 0x02, 0x00, 0x00, 0x7F, 0x02, 0x00, 0x80, 0x08, 0x02, 0x00, 0x00,
+ 0x13, 0x02, 0x00, 0x80, 0x25, 0x02, 0x00, 0x80, 0x3E, 0x02, 0x00, 0x00, 0xD0, 0x02, 0x00, 0x00,
+ 0xCB, 0x02, 0x00, 0x80, 0xFD, 0x02, 0x00, 0x80, 0xE6, 0x02, 0x00, 0x00, 0x91, 0x02, 0x00, 0x80,
+ 0x8A, 0x02, 0x00, 0x00, 0xBC, 0x02, 0x00, 0x00, 0xA7, 0x02, 0x00, 0x80, 0x60, 0x03, 0x00, 0x00,
+ 0x7B, 0x03, 0x00, 0x80, 0x4D, 0x03, 0x00, 0x80, 0x56, 0x03, 0x00, 0x00, 0x21, 0x03, 0x00, 0x80,
+ 0x3A, 0x03, 0x00, 0x00, 0x0C, 0x03, 0x00, 0x00, 0x17, 0x03, 0x00, 0x80, 0xF9, 0x03, 0x00, 0x80,
+ 0xE2, 0x03, 0x00, 0x00, 0xD4, 0x03, 0x00, 0x00, 0xCF, 0x03, 0x00, 0x80, 0xB8, 0x03, 0x00, 0x00,
+ 0xA3, 0x03, 0x00, 0x80, 0x95, 0x03, 0x00, 0x80, 0x8E, 0x03, 0x00, 0x00, 0x89, 0x04, 0x00, 0x80,
+ 0x92, 0x04, 0x00, 0x00, 0xA4, 0x04, 0x00, 0x00, 0xBF, 0x04, 0x00, 0x80, 0xC8, 0x04, 0x00, 0x00,
+ 0xD3, 0x04, 0x00, 0x80, 0xE5, 0x04, 0x00, 0x80, 0xFE, 0x04, 0x00, 0x00, 0x10, 0x04, 0x00, 0x00,
+ 0x0B, 0x04, 0x00, 0x80, 0x3D, 0x04, 0x00, 0x80, 0x26, 0x04, 0x00, 0x00, 0x51, 0x04, 0x00, 0x80,
+ 0x4A, 0x04, 0x00, 0x00, 0x7C, 0x04, 0x00, 0x00, 0x67, 0x04, 0x00, 0x80, 0xA0, 0x05, 0x00, 0x00,
+ 0xBB, 0x05, 0x00, 0x80, 0x8D, 0x05, 0x00, 0x80, 0x96, 0x05, 0x00, 0x00, 0xE1, 0x05, 0x00, 0x80,
+ 0xFA, 0x05, 0x00, 0x00, 0xCC, 0x05, 0x00, 0x00, 0xD7, 0x05, 0x00, 0x80, 0x39, 0x05, 0x00, 0x80,
+ 0x22, 0x05, 0x00, 0x00, 0x14, 0x05, 0x00, 0x00, 0x0F, 0x05, 0x00, 0x80, 0x78, 0x05, 0x00, 0x00,
+ 0x63, 0x05, 0x00, 0x80, 0x55, 0x05, 0x00, 0x80, 0x4E, 0x05, 0x00, 0x00, 0xC0, 0x06, 0x00, 0x00,
+ 0xDB, 0x06, 0x00, 0x80, 0xED, 0x06, 0x00, 0x80, 0xF6, 0x06, 0x00, 0x00, 0x81, 0x06, 0x00, 0x80,
+ 0x9A, 0x06, 0x00, 0x00, 0xAC, 0x06, 0x00, 0x00, 0xB7, 0x06, 0x00, 0x80, 0x59, 0x06, 0x00, 0x80,
+ 0x42, 0x06, 0x00, 0x00, 0x74, 0x06, 0x00, 0x00, 0x6F, 0x06, 0x00, 0x80, 0x18, 0x06, 0x00, 0x00,
+ 0x03, 0x06, 0x00, 0x80, 0x35, 0x06, 0x00, 0x80, 0x2E, 0x06, 0x00, 0x00, 0xE9, 0x07, 0x00, 0x80,
+ 0xF2, 0x07, 0x00, 0x00, 0xC4, 0x07, 0x00, 0x00, 0xDF, 0x07, 0x00, 0x80, 0xA8, 0x07, 0x00, 0x00,
+ 0xB3, 0x07, 0x00, 0x80, 0x85, 0x07, 0x00, 0x80, 0x9E, 0x07, 0x00, 0x00, 0x70, 0x07, 0x00, 0x00,
+ 0x6B, 0x07, 0x00, 0x80, 0x5D, 0x07, 0x00, 0x80, 0x46, 0x07, 0x00, 0x00, 0x31, 0x07, 0x00, 0x80,
+ 0x2A, 0x07, 0x00, 0x00, 0x1C, 0x07, 0x00, 0x00, 0x07, 0x07, 0x00, 0x80, 0x09, 0x09, 0x00, 0x80,
+ 0x12, 0x09, 0x00, 0x00, 0x24, 0x09, 0x00, 0x00, 0x3F, 0x09, 0x00, 0x80, 0x48, 0x09, 0x00, 0x00,
+ 0x53, 0x09, 0x00, 0x80, 0x65, 0x09, 0x00, 0x80, 0x7E, 0x09, 0x00, 0x00, 0x90, 0x09, 0x00, 0x00,
+ 0x8B, 0x09, 0x00, 0x80, 0xBD, 0x09, 0x00, 0x80, 0xA6, 0x09, 0x00, 0x00, 0xD1, 0x09, 0x00, 0x80,
+ 0xCA, 0x09, 0x00, 0x00, 0xFC, 0x09, 0x00, 0x00, 0xE7, 0x09, 0x00, 0x80, 0x20, 0x08, 0x00, 0x00,
+ 0x3B, 0x08, 0x00, 0x80, 0x0D, 0x08, 0x00, 0x80, 0x16, 0x08, 0x00, 0x00, 0x61, 0x08, 0x00, 0x80,
+ 0x7A, 0x08, 0x00, 0x00, 0x4C, 0x08, 0x00, 0x00, 0x57, 0x08, 0x00, 0x80, 0xB9, 0x08, 0x00, 0x80,
+ 0xA2, 0x08, 0x00, 0x00, 0x94, 0x08, 0x00, 0x00, 0x8F, 0x08, 0x00, 0x80, 0xF8, 0x08, 0x00, 0x00,
+ 0xE3, 0x08, 0x00, 0x80, 0xD5, 0x08, 0x00, 0x80, 0xCE, 0x08, 0x00, 0x00, 0x40, 0x0B, 0x00, 0x00,
+ 0x5B, 0x0B, 0x00, 0x80, 0x6D, 0x0B, 0x00, 0x80, 0x76, 0x0B, 0x00, 0x00, 0x01, 0x0B, 0x00, 0x80,
+ 0x1A, 0x0B, 0x00, 0x00, 0x2C, 0x0B, 0x00, 0x00, 0x37, 0x0B, 0x00, 0x80, 0xD9, 0x0B, 0x00, 0x80,
+ 0xC2, 0x0B, 0x00, 0x00, 0xF4, 0x0B, 0x00, 0x00, 0xEF, 0x0B, 0x00, 0x80, 0x98, 0x0B, 0x00, 0x00,
+ 0x83, 0x0B, 0x00, 0x80, 0xB5, 0x0B, 0x00, 0x80, 0xAE, 0x0B, 0x00, 0x00, 0x69, 0x0A, 0x00, 0x80,
+ 0x72, 0x0A, 0x00, 0x00, 0x44, 0x0A, 0x00, 0x00, 0x5F, 0x0A, 0x00, 0x80, 0x28, 0x0A, 0x00, 0x00,
+ 0x33, 0x0A, 0x00, 0x80, 0x05, 0x0A, 0x00, 0x80, 0x1E, 0x0A, 0x00, 0x00, 0xF0, 0x0A, 0x00, 0x00,
+ 0xEB, 0x0A, 0x00, 0x80, 0xDD, 0x0A, 0x00, 0x80, 0xC6, 0x0A, 0x00, 0x00, 0xB1, 0x0A, 0x00, 0x80,
+ 0xAA, 0x0A, 0x00, 0x00, 0x9C, 0x0A, 0x00, 0x00, 0x87, 0x0A, 0x00, 0x80, 0x80, 0x0D, 0x00, 0x00,
+ 0x9B, 0x0D, 0x00, 0x80, 0xAD, 0x0D, 0x00, 0x80, 0xB6, 0x0D, 0x00, 0x00, 0xC1, 0x0D, 0x00, 0x80,
+ 0xDA, 0x0D, 0x00, 0x00, 0xEC, 0x0D, 0x00, 0x00, 0xF7, 0x0D, 0x00, 0x80, 0x19, 0x0D, 0x00, 0x80,
+ 0x02, 0x0D, 0x00, 0x00, 0x34, 0x0D, 0x00, 0x00, 0x2F, 0x0D, 0x00, 0x80, 0x58, 0x0D, 0x00, 0x00,
+ 0x43, 0x0D, 0x00, 0x80, 0x75, 0x0D, 0x00, 0x80, 0x6E, 0x0D, 0x00, 0x00, 0xA9, 0x0C, 0x00, 0x80,
+ 0xB2, 0x0C, 0x00, 0x00, 0x84, 0x0C, 0x00, 0x00, 0x9F, 0x0C, 0x00, 0x80, 0xE8, 0x0C, 0x00, 0x00,
+ 0xF3, 0x0C, 0x00, 0x80, 0xC5, 0x0C, 0x00, 0x80, 0xDE, 0x0C, 0x00, 0x00, 0x30, 0x0C, 0x00, 0x00,
+ 0x2B, 0x0C, 0x00, 0x80, 0x1D, 0x0C, 0x00, 0x80, 0x06, 0x0C, 0x00, 0x00, 0x71, 0x0C, 0x00, 0x80,
+ 0x6A, 0x0C, 0x00, 0x00, 0x5C, 0x0C, 0x00, 0x00, 0x47, 0x0C, 0x00, 0x80, 0xC9, 0x0F, 0x00, 0x80,
+ 0xD2, 0x0F, 0x00, 0x00, 0xE4, 0x0F, 0x00, 0x00, 0xFF, 0x0F, 0x00, 0x80, 0x88, 0x0F, 0x00, 0x00,
+ 0x93, 0x0F, 0x00, 0x80, 0xA5, 0x0F, 0x00, 0x80, 0xBE, 0x0F, 0x00, 0x00, 0x50, 0x0F, 0x00, 0x00,
+ 0x4B, 0x0F, 0x00, 0x80, 0x7D, 0x0F, 0x00, 0x80, 0x66, 0x0F, 0x00, 0x00, 0x11, 0x0F, 0x00, 0x80,
+ 0x0A, 0x0F, 0x00, 0x00, 0x3C, 0x0F, 0x00, 0x00, 0x27, 0x0F, 0x00, 0x80, 0xE0, 0x0E, 0x00, 0x00,
+ 0xFB, 0x0E, 0x00, 0x80, 0xCD, 0x0E, 0x00, 0x80, 0xD6, 0x0E, 0x00, 0x00, 0xA1, 0x0E, 0x00, 0x80,
+ 0xBA, 0x0E, 0x00, 0x00, 0x8C, 0x0E, 0x00, 0x00, 0x97, 0x0E, 0x00, 0x80, 0x79, 0x0E, 0x00, 0x80,
+ 0x62, 0x0E, 0x00, 0x00, 0x54, 0x0E, 0x00, 0x00, 0x4F, 0x0E, 0x00, 0x80, 0x38, 0x0E, 0x00, 0x00,
+ 0x23, 0x0E, 0x00, 0x80, 0x15, 0x0E, 0x00, 0x80, 0x0E, 0x0E, 0x00, 0x00, 0x4C, 0x74, 0x01, 0x00,
+ 0x00, 0x04, 0x10, 0x00, 0x38, 0x04, 0x00, 0x00, 0x1C, 0x01, 0x00, 0x00, 0x84, 0x74, 0x01, 0x00,
+ 0x00, 0xFC, 0x10, 0x00, 0x40, 0x00, 0x00, 0x00, 0x78, 0x01, 0x00, 0x00, 0x84, 0x74, 0x01, 0x00,
+ 0x38, 0x08, 0x10, 0x00, 0x04, 0x80, 0x00, 0x00, 0x94, 0x01, 0x00, 0x00, 0x01, 0xFF, 0x01, 0x75,
+ 0x14, 0x90, 0x5F, 0x01, 0x32, 0x01, 0x32, 0x02, 0x29, 0x08, 0x41, 0xA9, 0x08, 0x09, 0x1A, 0x0C,
+ 0xB2, 0x64, 0x29, 0x0C, 0x69, 0x48, 0x29, 0x04, 0x0A, 0x11, 0x04, 0x10, 0x01, 0xFF, 0x01, 0x42,
+ 0x07, 0x8D, 0xF0, 0x01, 0x20, 0x04, 0x49, 0x10, 0xAA, 0xFF, 0x0C, 0x59, 0x01, 0x01, 0x1D, 0x04,
+ 0x50, 0x90, 0xD0, 0x03, 0x30, 0xB5, 0x72, 0xB6, 0x01, 0x23, 0x0C, 0x4D, 0x2B, 0x60, 0x0C, 0x4A,
+ 0x00, 0x24, 0x81, 0xB1, 0x13, 0x60, 0x4F, 0xF0, 0x00, 0x51, 0x00, 0xF0, 0x1F, 0x00, 0x81, 0xF8,
+ 0x6A, 0x00, 0x07, 0x48, 0x1C, 0x38, 0x03, 0x60, 0x01, 0x68, 0x00, 0x29, 0xFC, 0xD1, 0x14, 0x60,
+ 0x2C, 0x60, 0x62, 0xB6, 0x30, 0xBD, 0x14, 0x60, 0xED, 0xE7, 0x00, 0x00, 0xC0, 0x1B, 0x00, 0x22,
+ 0x78, 0x0D, 0x00, 0x22, 0xF6, 0x63, 0xE8, 0xA2, 0x97, 0x02, 0x00, 0x00, 0x09, 0x9C, 0x17, 0x5D,
+ 0x68, 0xFD, 0xFF, 0xFF, 0x10, 0x00, 0x10, 0x00, 0x00, 0x00, 0x01, 0x00, 0x10, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0xFF, 0xFF, 0xFF, 0x3F, 0x00, 0x00, 0x00, 0x00, 0xFF, 0x7F, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x1E, 0x0F, 0x6F, 0x08, 0x37, 0x04, 0x0F, 0x6F, 0x08, 0x37, 0x04, 0xB8,
+ 0x01, 0x01, 0x00, 0x01, 0x01, 0x20, 0x20, 0x50, 0xAA, 0x99, 0x00, 0x05, 0x00, 0x05, 0xC8, 0x00,
+ 0xC8, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x00, 0x01, 0x70, 0x00, 0x70, 0x00, 0x08, 0xB0,
+ 0x00, 0xB0, 0x00, 0x00, 0x03, 0xAA, 0x38, 0xC8, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x05, 0x08, 0x0F, 0x0F, 0x1F, 0x0F, 0x00, 0x02, 0x0F, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x0B, 0x01, 0xA0, 0x00, 0x64, 0x00, 0x64, 0x00, 0xD0, 0x0F, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x82, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, 0xFF, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x16, 0x08, 0x00, 0x02, 0x08, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00, 0xA0,
+ 0x50, 0x4C, 0x26, 0x01, 0x08, 0x00, 0x13, 0x00, 0x00, 0x00, 0x00, 0x00, 0x81, 0x06, 0x52, 0x4F,
+ 0x04, 0xD8, 0x00, 0x50, 0x00, 0x00, 0x00, 0x2F, 0x00, 0x08, 0xB0, 0x06, 0x03, 0x03, 0x00, 0x00,
+ 0xBB, 0xBB, 0xC8, 0x00, 0xC8, 0x00, 0x88, 0x88, 0x00, 0x00, 0xE9, 0xA6, 0x1A, 0x1F, 0x04, 0xD8,
+ 0x18, 0x32, 0x00, 0x20, 0x00, 0x24, 0x60, 0x06, 0x00, 0x00, 0x00, 0x00, 0x24, 0x60, 0x06, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x01, 0x24, 0x00, 0x00, 0x00, 0x00, 0x00, 0x24, 0x06, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x89, 0x16, 0x50, 0x4F, 0x04, 0x18, 0x00, 0x30, 0x00, 0x06,
+ 0x00, 0x2F, 0x00, 0x40, 0x10, 0x06, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xA8, 0x66, 0x9E, 0x1F, 0x98, 0x18, 0x01, 0x01, 0x30, 0x00,
+ 0x30, 0x00, 0x40, 0x00, 0x10, 0x00, 0x21, 0x03, 0x00, 0x00, 0x1F, 0x21, 0x01, 0x00, 0x00, 0x1F,
+ 0x00, 0x01, 0x28, 0x00, 0x99, 0x24, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x11, 0x00, 0x2C, 0x01, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xB8, 0x0B, 0x00, 0x00, 0x00, 0x00, 0x0E, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x0E, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x02, 0x00, 0x01, 0x01, 0x64, 0x00, 0x64, 0x00, 0xD0, 0x00, 0x80, 0x0C, 0xF0, 0x0A,
+ 0xD0, 0x00, 0x00, 0x15, 0x00, 0x12, 0x50, 0x01, 0x50, 0x01, 0x00, 0x15, 0x00, 0x12, 0x50, 0x01,
+ 0x80, 0x0C, 0xF0, 0x0A, 0xD0, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x33, 0x0E, 0x02, 0x0B, 0x14, 0x0F, 0x02, 0x64, 0x90, 0x01, 0x00, 0x00,
+ 0x00, 0x00, 0x30, 0x00, 0x00, 0x00, 0x00, 0x00, 0xFF, 0x0F, 0xFF, 0x0F, 0x14, 0x14, 0x20, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x0B, 0x00, 0x80, 0x00, 0x00, 0x00, 0xA0, 0xA0, 0x02, 0x40, 0x90, 0x01,
+ 0x90, 0x01, 0x80, 0x00, 0x80, 0x00, 0xF4, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x59, 0x00, 0x64, 0x00, 0x80, 0x01, 0x64, 0x00, 0x80, 0x01, 0x96, 0x00,
+ 0x64, 0x00, 0x20, 0x00, 0x02, 0x00, 0x96, 0x00, 0x64, 0x00, 0x64, 0x00, 0xC8, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x64, 0x00, 0x05, 0x01, 0xF4, 0x01, 0x2C, 0x01, 0x02, 0xFF, 0x7F, 0xFF, 0x7F, 0x02,
+ 0x32, 0x64, 0x00, 0x00, 0x02, 0x02, 0xA2, 0x20, 0x00, 0x29, 0x64, 0x00, 0x64, 0x00, 0xC8, 0x00,
+ 0x58, 0x02, 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x03, 0x00, 0x64, 0x00, 0x32, 0x00, 0x20, 0x00, 0x14, 0x0A, 0x84, 0x03,
+ 0x64, 0x00, 0x00, 0x04, 0x02, 0x00, 0x20, 0x00, 0x05, 0x00, 0x04, 0x00, 0x04, 0x00, 0x00, 0x00,
+ 0x05, 0x00, 0x05, 0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x81, 0x0E,
+ 0x10, 0x00, 0x40, 0x00, 0x18, 0x00, 0x00, 0x01, 0x62, 0x84, 0x40, 0x01, 0x20, 0x00, 0x10, 0x00,
+ 0x64, 0x00, 0x64, 0x00, 0xC8, 0x00, 0xC8, 0x00, 0x02, 0x00, 0x01, 0xA0, 0x00, 0xA0, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01,
+ 0x00, 0x01, 0x00, 0x00, 0x0C, 0x04, 0x20, 0x00, 0x10, 0x10, 0x20, 0x00, 0x20, 0x00, 0x96, 0x00,
+ 0x00, 0x0A, 0x00, 0x0F, 0x00, 0x0F, 0x00, 0x00, 0x40, 0x00, 0x50, 0x00, 0x00, 0x40, 0x40, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x2C, 0x01, 0x64, 0x00, 0x2C, 0x01, 0x00, 0x28, 0x02, 0x01, 0x00, 0x00,
+ 0x00, 0x00, 0x56, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x5A, 0x00, 0x14, 0x00, 0x01, 0x03, 0x01, 0x0A, 0x00, 0x10,
+ 0x11, 0x11, 0x00, 0x00, 0x00, 0x02, 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x50, 0x00, 0x70, 0x00, 0x32, 0x00, 0x40, 0x00, 0x00, 0x00,
+ 0xC8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x7D, 0x03, 0x64, 0x00, 0x00, 0x02, 0x10, 0x01, 0x00, 0x02, 0x64, 0x00,
+ 0x06, 0x64, 0x00, 0x64, 0x00, 0x0E, 0x00, 0x05, 0x00, 0x10, 0x50, 0x00, 0x32, 0x00, 0x10, 0x21,
+ 0x0F, 0x02, 0x10, 0x1F, 0x00, 0x0A, 0x00, 0x14, 0x81, 0x11, 0x1A, 0x00, 0x16, 0x00, 0x00, 0x10,
+ 0x0A, 0x00, 0x40, 0x00, 0x00, 0x20, 0x00, 0x0A, 0xE8, 0x10, 0x00, 0x40, 0x00, 0x40, 0x94, 0x94,
+ 0x28, 0x88, 0x62, 0x10, 0x00, 0x64, 0x00, 0x10, 0x00, 0x30, 0x00, 0x0A, 0x00, 0x18, 0x00, 0x05,
+ 0x20, 0x05, 0x20, 0x18, 0x20, 0xD8, 0x05, 0x32, 0xF8, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x04, 0x03, 0xC8, 0x00, 0xC8, 0x00, 0x0A, 0x0A, 0x64, 0x00, 0x64, 0x00,
+ 0x01, 0x03, 0x64, 0x00, 0x64, 0x00, 0x01, 0x03, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x96, 0x00, 0x0A, 0x06, 0x00, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x32, 0x1A,
+ 0xFA, 0x00, 0x20, 0x05, 0x20, 0x0A, 0x05, 0x00, 0x03, 0x10, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x64, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0A,
+ 0x00, 0x14, 0x05, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x2E, 0x03, 0x50, 0x00, 0x50, 0x00, 0x2C, 0x01, 0x32, 0xFF, 0x7F, 0x0B,
+ 0x01, 0x32, 0x00, 0x20, 0x03, 0x00, 0x0A, 0x20, 0x14, 0x05, 0xF8, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x02, 0x03, 0x04, 0x00, 0x01, 0x02, 0x03, 0x50, 0x01,
+ 0x50, 0x01, 0x50, 0x01, 0x50, 0x01, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x96, 0x7D, 0x6E, 0x67, 0x65, 0x64, 0x64, 0x64, 0x64, 0x64, 0x64, 0x64,
+ 0x64, 0x65, 0x67, 0x6E, 0x7D, 0x96, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x0F, 0x13, 0x22, 0x22,
+ 0x22, 0x00, 0x00, 0x0F, 0x00, 0x28, 0x00, 0x3C, 0x00, 0x50, 0x00, 0xA0, 0x00, 0x40, 0x01, 0xE0,
+ 0x01, 0xD0, 0x02, 0xBF, 0x03, 0x5F, 0x04, 0xFF, 0x04, 0x4F, 0x05, 0x63, 0x05, 0x77, 0x05, 0x90,
+ 0x05, 0x0F, 0x00, 0x28, 0x00, 0x3C, 0x00, 0x50, 0x00, 0xA0, 0x00, 0x40, 0x01, 0xE0, 0x01, 0xD0,
+ 0x02, 0xC0, 0x03, 0x00, 0x05, 0x3F, 0x06, 0x2F, 0x07, 0x1F, 0x08, 0xBF, 0x08, 0x5F, 0x09, 0xAF,
+ 0x09, 0xC3, 0x09, 0xD7, 0x09, 0xF5, 0x09, 0x00, 0x00, 0x0A, 0x00, 0x19, 0x00, 0x28, 0x00, 0x6C,
+ 0x00, 0x27, 0x01, 0xD1, 0x01, 0xD0, 0x02, 0xC6, 0x03, 0x72, 0x04, 0x21, 0x05, 0x69, 0x05, 0x86,
+ 0x05, 0x95, 0x05, 0x9F, 0x05, 0x00, 0x00, 0x0F, 0x00, 0x1C, 0x00, 0x28, 0x00, 0x78, 0x00, 0x1C,
+ 0x01, 0xC6, 0x01, 0xBE, 0x02, 0xAE, 0x03, 0x00, 0x05, 0x4F, 0x06, 0x44, 0x07, 0x39, 0x08, 0xDE,
+ 0x08, 0x8A, 0x09, 0xD8, 0x09, 0xE3, 0x09, 0xF1, 0x09, 0xFF, 0x09, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x40, 0x00, 0x40, 0x00, 0x00, 0x00, 0x64, 0x80, 0x10, 0xE5, 0x00, 0x00,
+ 0x64, 0x80, 0xA4, 0x00, 0xA7, 0x00, 0xB2, 0x00, 0xC3, 0x00, 0xDB, 0x00, 0xFA, 0x00, 0x1F, 0x01,
+ 0x4A, 0x01, 0x7A, 0x01, 0xB0, 0x01, 0xEC, 0x01, 0x2B, 0x02, 0x6F, 0x02, 0xB6, 0x02, 0x01, 0x03,
+ 0x4D, 0x03, 0x9C, 0x03, 0xEC, 0x03, 0x3D, 0x04, 0x8E, 0x04, 0xDF, 0x04, 0x2E, 0x05, 0x7C, 0x05,
+ 0xC8, 0x05, 0x11, 0x06, 0x56, 0x06, 0x98, 0x06, 0xD5, 0x06, 0x0E, 0x07, 0x42, 0x07, 0x6F, 0x07,
+ 0x97, 0x07, 0xB9, 0x07, 0xD5, 0x07, 0xE9, 0x07, 0xF7, 0x07, 0xFE, 0x07, 0xFE, 0x07, 0xF7, 0x07,
+ 0xE9, 0x07, 0xD5, 0x07, 0xB9, 0x07, 0x97, 0x07, 0x6F, 0x07, 0x42, 0x07, 0x0E, 0x07, 0xD5, 0x06,
+ 0x98, 0x06, 0x56, 0x06, 0x11, 0x06, 0xC8, 0x05, 0x7C, 0x05, 0x2E, 0x05, 0xDF, 0x04, 0x8E, 0x04,
+ 0x3D, 0x04, 0xEC, 0x03, 0x9C, 0x03, 0x4D, 0x03, 0x01, 0x03, 0xB6, 0x02, 0x6F, 0x02, 0x2B, 0x02,
+ 0xEC, 0x01, 0xB0, 0x01, 0x7A, 0x01, 0x4A, 0x01, 0x1F, 0x01, 0xFA, 0x00, 0xDB, 0x00, 0xC3, 0x00,
+ 0xB2, 0x00, 0xA7, 0x00, 0xA4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x6A, 0x00, 0x6C, 0x00, 0x72, 0x00, 0x7C, 0x00, 0x89, 0x00, 0x9A, 0x00, 0xAF, 0x00,
+ 0xC7, 0x00, 0xE2, 0x00, 0x01, 0x01, 0x22, 0x01, 0x46, 0x01, 0x6D, 0x01, 0x95, 0x01, 0xC0, 0x01,
+ 0xED, 0x01, 0x1B, 0x02, 0x4A, 0x02, 0x7A, 0x02, 0xAA, 0x02, 0xDB, 0x02, 0x0B, 0x03, 0x3B, 0x03,
+ 0x6B, 0x03, 0x99, 0x03, 0xC7, 0x03, 0xF2, 0x03, 0x1C, 0x04, 0x44, 0x04, 0x69, 0x04, 0x8C, 0x04,
+ 0xAC, 0x04, 0xC9, 0x04, 0xE3, 0x04, 0xF9, 0x04, 0x0C, 0x05, 0x1B, 0x05, 0x26, 0x05, 0x2E, 0x05,
+ 0x32, 0x05, 0x32, 0x05, 0x2E, 0x05, 0x26, 0x05, 0x1B, 0x05, 0x0C, 0x05, 0xF9, 0x04, 0xE3, 0x04,
+ 0xC9, 0x04, 0xAC, 0x04, 0x8C, 0x04, 0x69, 0x04, 0x44, 0x04, 0x1C, 0x04, 0xF2, 0x03, 0xC7, 0x03,
+ 0x99, 0x03, 0x6B, 0x03, 0x3B, 0x03, 0x0B, 0x03, 0xDB, 0x02, 0xAA, 0x02, 0x7A, 0x02, 0x4A, 0x02,
+ 0x1B, 0x02, 0xED, 0x01, 0xC0, 0x01, 0x95, 0x01, 0x6D, 0x01, 0x46, 0x01, 0x22, 0x01, 0x01, 0x01,
+ 0xE2, 0x00, 0xC7, 0x00, 0xAF, 0x00, 0x9A, 0x00, 0x89, 0x00, 0x7C, 0x00, 0x72, 0x00, 0x6C, 0x00,
+ 0x6A, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x06, 0x00, 0x06,
+ 0x00, 0x06, 0x00, 0x06, 0x00, 0x06, 0x00, 0x06, 0x00, 0x06, 0x00, 0x06, 0x00, 0x06, 0x00, 0x06,
+ 0x00, 0x06, 0x00, 0x06, 0x00, 0x06, 0x00, 0x06, 0x00, 0x06, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x04, 0x20, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x02, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0xCD, 0x00, 0xCD, 0x00, 0xCD, 0x00, 0xCD, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x03, 0x05, 0x07, 0x08, 0x0C, 0x0D, 0x0F, 0x0F, 0x0F, 0x0E, 0x0E,
+ 0x0E, 0x0D, 0x0D, 0x0D, 0x0A, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xA8, 0x64, 0x9A, 0x1F, 0x18, 0x18, 0x01, 0x01, 0x30, 0x00,
+ 0x30, 0x00, 0x40, 0x00, 0x10, 0x00, 0x21, 0x07, 0x00, 0x00, 0x1F, 0x20, 0x03, 0x00, 0x00, 0x1F,
+ 0x00, 0x01, 0x00, 0x00, 0x00, 0x00, 0x06, 0x00, 0x99, 0x24, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x50, 0x46, 0x50, 0x10, 0x10, 0xB0, 0x20, 0xFF, 0xD0, 0x00, 0x06, 0x14,
+ 0x14, 0x2A, 0x3A, 0x40, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x14, 0x6E, 0x6E,
+ 0x47, 0x1A, 0x38, 0x00, 0x00, 0x00, 0x01, 0x0F, 0x1E, 0x00, 0x20, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xA9, 0x24, 0x94, 0x1F, 0x18, 0x18, 0x01, 0x00, 0x25, 0x00,
+ 0x25, 0x00, 0x50, 0x00, 0x24, 0x00, 0x70, 0x01, 0x00, 0x00, 0x1F, 0x80, 0x01, 0x00, 0x00, 0x1F,
+ 0x00, 0x01, 0x10, 0x00, 0x55, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x88, 0x88, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x77, 0x00, 0x00, 0x00, 0x00, 0x09, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
+ 0x00, 0x00, 0x00, 0x00, 0x4E, 0xD3, 0x25, 0xD2,
+};
+
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsCompensation.c b/drivers/input/touchscreen/stm/fts_lib/ftsCompensation.c
new file mode 100644
index 0000000..0fe5f0c
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsCompensation.c
@@ -0,0 +1,953 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS functions for getting Initialization Data *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+/*!
+ * \file ftsCompensation.c
+ * \brief Contains all the function to work with Initialization Data
+ */
+
+#include "ftsCompensation.h"
+#include "ftsCore.h"
+#include "ftsError.h"
+#include "ftsFrame.h"
+#include "ftsHardware.h"
+#include "ftsIO.h"
+#include "ftsSoftware.h"
+#include "ftsTime.h"
+#include "ftsTool.h"
+
+
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <stdarg.h>
+#include <linux/serio.h>
+#include <linux/time.h>
+#include <linux/delay.h>
+#include <linux/ctype.h>
+
+
+/**
+ * Request to the FW to load the specified Initialization Data
+ * @param type type of Initialization data to load @link load_opt Load Host
+ * Data Option @endlink
+ * @return OK if success or an error code which specify the type of error
+ */
+int requestCompensationData(u8 type)
+{
+ int ret = ERROR_OP_NOT_ALLOW;
+ int retry = 0;
+
+ pr_info("%s: Requesting compensation data...\n", __func__);
+ while (retry < RETRY_COMP_DATA_READ) {
+ ret = writeSysCmd(SYS_CMD_LOAD_DATA, &type, 1);
+ /* send request to load in memory the Compensation Data */
+ if (ret < OK) {
+ pr_err("%s: failed at %d attemp!\n",
+ __func__, retry + 1);
+ retry += 1;
+ } else {
+ pr_info("%s: Request Compensation data FINISHED!\n",
+ __func__);
+ return OK;
+ }
+ }
+
+ pr_err("%s: Requesting compensation data... ERROR %08X\n",
+ __func__, ret | ERROR_REQU_COMP_DATA);
+ return ret | ERROR_REQU_COMP_DATA;
+}
+
+
+/**
+ * Read Initialization Data Header and check that the type loaded match
+ * with the one previously requested
+ * @param type type of Initialization data requested @link load_opt Load Host
+ * Data Option @endlink
+ * @param header pointer to DataHeader variable which will contain the header
+ * @param address pointer to a variable which will contain the updated address
+ * to the next data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readCompensationDataHeader(u8 type, DataHeader *header, u64 *address)
+{
+ u64 offset = ADDR_FRAMEBUFFER;
+ u8 data[COMP_DATA_HEADER];
+ int ret;
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, offset, data,
+ COMP_DATA_HEADER, DUMMY_FRAMEBUFFER);
+ if (ret < OK) { /* i2c function have already a retry mechanism */
+ pr_err("%s: error while reading data header ERROR %08X\n",
+ __func__, ret);
+ return ret;
+ }
+
+ pr_info("Read Data Header done!\n");
+
+ if (data[0] != HEADER_SIGNATURE) {
+ pr_err("%s: The Header Signature was wrong! %02X != %02X ERROR %08X\n",
+ __func__, data[0], HEADER_SIGNATURE,
+ ERROR_WRONG_DATA_SIGN);
+ return ERROR_WRONG_DATA_SIGN;
+ }
+
+
+ if (data[1] != type) {
+ pr_err("%s: Wrong type found! %02X!=%02X ERROR %08X\n",
+ __func__, data[1], type, ERROR_DIFF_DATA_TYPE);
+ return ERROR_DIFF_DATA_TYPE;
+ }
+
+ pr_info("Type = %02X of Compensation data OK!\n", type);
+
+ header->type = type;
+
+ *address = offset + COMP_DATA_HEADER;
+
+ return OK;
+}
+
+
+/**
+ * Read MS Global Initialization data from the buffer such as Cx1
+ * @param address pointer to a variable which contain the address from where
+ * to read the data and will contain the updated address to the next data
+ * @param global pointer to MutualSenseData variable which will contain the MS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readMutualSenseGlobalData(u64 *address, MutualSenseData *global)
+{
+ u8 data[COMP_DATA_GLOBAL];
+ int ret;
+
+ pr_info("Address for Global data= %llx\n", *address);
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, *address, data,
+ COMP_DATA_GLOBAL, DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: error while reading info data ERROR %08X\n",
+ __func__, ret);
+ return ret;
+ }
+ pr_info("Global data Read !\n");
+
+ global->header.force_node = data[0];
+ global->header.sense_node = data[1];
+ global->cx1 = data[2];
+ /* all other bytes are reserved atm */
+
+ pr_info("force_len = %d sense_len = %d CX1 = %d\n",
+ global->header.force_node, global->header.sense_node,
+ global->cx1);
+
+ *address += COMP_DATA_GLOBAL;
+ return OK;
+}
+
+
+/**
+ * Read MS Initialization data for each node from the buffer
+ * @param address a variable which contain the address from where to read the
+ * data
+ * @param node pointer to MutualSenseData variable which will contain the MS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readMutualSenseNodeData(u64 address, MutualSenseData *node)
+{
+ int ret;
+ int size = node->header.force_node * node->header.sense_node;
+
+ pr_info("Address for Node data = %llx\n", address);
+
+ node->node_data = (i8 *)kmalloc(size * (sizeof(i8)), GFP_KERNEL);
+
+ if (node->node_data == NULL) {
+ pr_err("%s: can not allocate node_data... ERROR %08X",
+ __func__, ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ pr_info("Node Data to read %d bytes\n", size);
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, address,
+ node->node_data, size, DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: error while reading node data ERROR %08X\n",
+ __func__, ret);
+ kfree(node->node_data);
+ return ret;
+ }
+ node->node_data_size = size;
+
+ pr_info("Read node data OK!\n");
+
+ return size;
+}
+
+/**
+ * Perform all the steps to read the necessary info for MS Initialization data
+ * from the buffer and store it in a MutualSenseData variable
+ * @param type type of MS Initialization data to read @link load_opt Load Host
+ * Data Option @endlink
+ * @param data pointer to MutualSenseData variable which will contain the MS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readMutualSenseCompensationData(u8 type, MutualSenseData *data)
+{
+ int ret;
+ u64 address;
+
+ data->node_data = NULL;
+
+ if (!(type == LOAD_CX_MS_TOUCH || type == LOAD_CX_MS_LOW_POWER ||
+ type == LOAD_CX_MS_KEY || type == LOAD_CX_MS_FORCE)) {
+ pr_err("%s: Choose a MS type of compensation data ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ ret = requestCompensationData(type);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_REQU_COMP_DATA);
+ return ret | ERROR_REQU_COMP_DATA;
+ }
+
+ ret = readCompensationDataHeader(type, &(data->header), &address);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_HEADER);
+ return ret | ERROR_COMP_DATA_HEADER;
+ }
+
+ ret = readMutualSenseGlobalData(&address, data);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_GLOBAL);
+ return ret | ERROR_COMP_DATA_GLOBAL;
+ }
+
+ ret = readMutualSenseNodeData(address, data);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_NODE);
+ return ret | ERROR_COMP_DATA_NODE;
+ }
+
+ return OK;
+}
+
+/**
+ * Read SS Global Initialization data from the buffer such as Ix1/Cx1 for force
+ * and sense
+ * @param address pointer to a variable which contain the address from where
+ * to read the data and will contain the updated address to the next data
+ * @param global pointer to MutualSenseData variable which will contain the SS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readSelfSenseGlobalData(u64 *address, SelfSenseData *global)
+{
+ int ret;
+ u8 data[COMP_DATA_GLOBAL];
+
+ pr_info("Address for Global data= %llx\n", *address);
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, *address, data,
+ COMP_DATA_GLOBAL, DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: error while reading the data... ERROR %08X\n",
+ __func__, ret);
+ return ret;
+ }
+
+ pr_info("Global data Read !\n");
+
+
+ global->header.force_node = data[0];
+ global->header.sense_node = data[1];
+ global->f_ix1 = data[2];
+ global->s_ix1 = data[3];
+ global->f_cx1 = (i8)data[4];
+ global->s_cx1 = (i8)data[5];
+ global->f_max_n = data[6];
+ global->s_max_n = data[7];
+
+ pr_info("force_len = %d sense_len = %d f_ix1 = %d s_ix1 = %d f_cx1 = %d s_cx1 = %d\n",
+ global->header.force_node, global->header.sense_node,
+ global->f_ix1, global->s_ix1, global->f_cx1, global->s_cx1);
+ pr_info("max_n = %d s_max_n = %d\n", global->f_max_n,
+ global->s_max_n);
+
+
+ *address += COMP_DATA_GLOBAL;
+
+ return OK;
+}
+
+/**
+ * Read SS Initialization data for each node of force and sense channels from
+ * the buffer
+ * @param address a variable which contain the address from where to read the
+ * data
+ * @param node pointer to SelfSenseData variable which will contain the SS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readSelfSenseNodeData(u64 address, SelfSenseData *node)
+{
+ int size = node->header.force_node * 2 + node->header.sense_node * 2;
+ u8 data[size];
+ int ret;
+
+ node->ix2_fm = (u8 *)kmalloc(node->header.force_node * (sizeof(u8)),
+ GFP_KERNEL);
+ if (node->ix2_fm == NULL) {
+ pr_err("%s: can not allocate memory for ix2_fm... ERROR %08X",
+ __func__, ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ node->cx2_fm = (i8 *)kmalloc(node->header.force_node * (sizeof(i8)),
+ GFP_KERNEL);
+ if (node->cx2_fm == NULL) {
+ pr_err("%s: can not allocate memory for cx2_fm ... ERROR %08X",
+ __func__, ERROR_ALLOC);
+ kfree(node->ix2_fm);
+ return ERROR_ALLOC;
+ }
+ node->ix2_sn = (u8 *)kmalloc(node->header.sense_node * (sizeof(u8)),
+ GFP_KERNEL);
+ if (node->ix2_sn == NULL) {
+ pr_err("%s: can not allocate memory for ix2_sn ERROR %08X",
+ __func__, ERROR_ALLOC);
+ kfree(node->ix2_fm);
+ kfree(node->cx2_fm);
+ return ERROR_ALLOC;
+ }
+ node->cx2_sn = (i8 *)kmalloc(node->header.sense_node * (sizeof(i8)),
+ GFP_KERNEL);
+ if (node->cx2_sn == NULL) {
+ pr_err("%s: can not allocate memory for cx2_sn ERROR %08X",
+ __func__, ERROR_ALLOC);
+ kfree(node->ix2_fm);
+ kfree(node->cx2_fm);
+ kfree(node->ix2_sn);
+ return ERROR_ALLOC;
+ }
+
+
+ pr_info("Address for Node data = %llx\n", address);
+
+ pr_info("Node Data to read %d bytes\n", size);
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, address, data,
+ size, DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: error while reading data... ERROR %08X\n",
+ __func__, ret);
+ kfree(node->ix2_fm);
+ kfree(node->cx2_fm);
+ kfree(node->ix2_sn);
+ kfree(node->cx2_sn);
+ return ret;
+ }
+
+ pr_info("Read node data ok!\n");
+
+ memcpy(node->ix2_fm, data, node->header.force_node);
+ memcpy(node->ix2_sn, &data[node->header.force_node],
+ node->header.sense_node);
+ memcpy(node->cx2_fm, &data[node->header.force_node +
+ node->header.sense_node],
+ node->header.force_node);
+ memcpy(node->cx2_sn, &data[node->header.force_node * 2 +
+ node->header.sense_node],
+ node->header.sense_node);
+
+ return OK;
+}
+
+/**
+ * Perform all the steps to read the necessary info for SS Initialization data
+ * from the buffer and store it in a SelfSenseData variable
+ * @param type type of SS Initialization data to read @link load_opt Load Host
+ * Data Option @endlink
+ * @param data pointer to SelfSenseData variable which will contain the SS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readSelfSenseCompensationData(u8 type, SelfSenseData *data)
+{
+ int ret;
+ u64 address;
+
+ data->ix2_fm = NULL;
+ data->cx2_fm = NULL;
+ data->ix2_sn = NULL;
+ data->cx2_sn = NULL;
+
+ if (!(type == LOAD_CX_SS_TOUCH || type == LOAD_CX_SS_TOUCH_IDLE ||
+ type == LOAD_CX_SS_KEY || type == LOAD_CX_SS_FORCE)) {
+ pr_err("%s: Choose a SS type of compensation data ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ ret = requestCompensationData(type);
+ if (ret < 0) {
+ pr_err("%s: error while requesting data... ERROR %08X\n",
+ __func__, ERROR_REQU_COMP_DATA);
+ return ret | ERROR_REQU_COMP_DATA;
+ }
+
+ ret = readCompensationDataHeader(type, &(data->header), &address);
+ if (ret < 0) {
+ pr_err("%s: error while reading data header... ERROR %08X\n",
+ __func__, ERROR_COMP_DATA_HEADER);
+ return ret | ERROR_COMP_DATA_HEADER;
+ }
+
+ ret = readSelfSenseGlobalData(&address, data);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_GLOBAL);
+ return ret | ERROR_COMP_DATA_GLOBAL;
+ }
+
+ ret = readSelfSenseNodeData(address, data);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_NODE);
+ return ret | ERROR_COMP_DATA_NODE;
+ }
+
+ return OK;
+}
+
+/**
+ * Read TOT MS Global Initialization data from the buffer such as number of
+ * force and sense channels
+ * @param address pointer to a variable which contain the address from where
+ * to read the data and will contain the updated address to the next data
+ * @param global pointer to a variable which will contain the TOT MS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readTotMutualSenseGlobalData(u64 *address, TotMutualSenseData *global)
+{
+ int ret;
+ u8 data[COMP_DATA_GLOBAL];
+
+ pr_info("Address for Global data= %llx\n", *address);
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, *address, data,
+ COMP_DATA_GLOBAL, DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: error while reading info data ERROR %08X\n",
+ __func__, ret);
+ return ret;
+ }
+ pr_info("Global data Read !\n");
+
+ global->header.force_node = data[0];
+ global->header.sense_node = data[1];
+ /* all other bytes are reserved atm */
+
+ pr_info("force_len = %d sense_len = %d\n",
+ global->header.force_node, global->header.sense_node);
+
+ *address += COMP_DATA_GLOBAL;
+ return OK;
+}
+
+
+/**
+ * Read TOT MS Initialization data for each node from the buffer
+ * @param address a variable which contain the address from where to read the
+ * data
+ * @param node pointer to MutualSenseData variable which will contain the TOT
+ * MS initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readTotMutualSenseNodeData(u64 address, TotMutualSenseData *node)
+{
+ int ret, i;
+ int size = node->header.force_node * node->header.sense_node;
+ int toRead = size * sizeof(u16);
+ u8 data[toRead];
+
+ pr_info("Address for Node data = %llx\n", address);
+
+ node->node_data = (short *)kmalloc(size * (sizeof(short)), GFP_KERNEL);
+
+ if (node->node_data == NULL) {
+ pr_err("%s: can not allocate node_data... ERROR %08X",
+ __func__, ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ pr_info("Node Data to read %d bytes\n", size);
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, address, data,
+ toRead, DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: error while reading node data ERROR %08X\n",
+ __func__, ret);
+ kfree(node->node_data);
+ return ret;
+ }
+ node->node_data_size = size;
+
+ for (i = 0; i < size; i++)
+ node->node_data[i] = ((short)data[i * 2 + 1]) << 8 |
+ data[i * 2];
+
+ pr_info("Read node data OK!\n");
+
+ return size;
+}
+
+/**
+ * Perform all the steps to read the necessary info for TOT MS Initialization
+ * data from the buffer and store it in a TotMutualSenseData variable
+ * @param type type of TOT MS Initialization data to read @link load_opt Load
+ * Host Data Option @endlink
+ * @param data pointer to a variable which will contain the TOT MS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readTotMutualSenseCompensationData(u8 type, TotMutualSenseData *data)
+{
+ int ret;
+ u64 address;
+
+ data->node_data = NULL;
+
+ if (!(type == LOAD_PANEL_CX_TOT_MS_TOUCH || type ==
+ LOAD_PANEL_CX_TOT_MS_LOW_POWER ||
+ type == LOAD_PANEL_CX_TOT_MS_KEY ||
+ type == LOAD_PANEL_CX_TOT_MS_FORCE)) {
+ pr_err("%s: Choose a TOT MS type of compensation data ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ ret = requestCompensationData(type);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_REQU_COMP_DATA);
+ return ret | ERROR_REQU_COMP_DATA;
+ }
+
+ ret = readCompensationDataHeader(type, &(data->header), &address);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_HEADER);
+ return ret | ERROR_COMP_DATA_HEADER;
+ }
+
+ ret = readTotMutualSenseGlobalData(&address, data);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_GLOBAL);
+ return ret | ERROR_COMP_DATA_GLOBAL;
+ }
+
+ ret = readTotMutualSenseNodeData(address, data);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_NODE);
+ return ret | ERROR_COMP_DATA_NODE;
+ }
+
+ return OK;
+}
+
+/**
+ * Read TOT SS Global Initialization data from the buffer such as number of
+ * force and sense channels
+ * @param address pointer to a variable which contain the address from where
+ * to read the data and will contain the updated address to the next data
+ * @param global pointer to a variable which will contain the TOT SS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readTotSelfSenseGlobalData(u64 *address, TotSelfSenseData *global)
+{
+ int ret;
+ u8 data[COMP_DATA_GLOBAL];
+
+ pr_info("Address for Global data= %llx\n", *address);
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, *address, data,
+ COMP_DATA_GLOBAL, DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: error while reading the data... ERROR %08X\n",
+ __func__, ret);
+ return ret;
+ }
+
+ pr_info("Global data Read !\n");
+
+
+ global->header.force_node = data[0];
+ global->header.sense_node = data[1];
+
+
+ pr_info("force_len = %d sense_len = %d\n",
+ global->header.force_node, global->header.sense_node);
+
+
+ *address += COMP_DATA_GLOBAL;
+
+ return OK;
+}
+
+/**
+ * Read TOT SS Global Initialization data from the buffer such as number of
+ * force and sense channels
+ * @param address pointer to a variable which contain the address from where
+ * to read the data and will contain the updated address to the next data
+ * @param node pointer to a variable which will contain the TOT SS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readTotSelfSenseNodeData(u64 address, TotSelfSenseData *node)
+{
+ int size = node->header.force_node * 2 + node->header.sense_node * 2;
+ int toRead = size * 2; /* *2 2 bytes each node */
+ u8 data[toRead];
+ int ret, i, j = 0;
+
+ node->ix_fm = (u16 *)kmalloc(node->header.force_node * (sizeof(u16)),
+ GFP_KERNEL);
+ if (node->ix_fm == NULL) {
+ pr_err("%s: can not allocate memory for ix2_fm... ERROR %08X",
+ __func__, ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ node->cx_fm = (short *)kmalloc(node->header.force_node *
+ (sizeof(short)), GFP_KERNEL);
+ if (node->cx_fm == NULL) {
+ pr_err("%s: can not allocate memory for cx2_fm ... ERROR %08X",
+ __func__, ERROR_ALLOC);
+ kfree(node->ix_fm);
+ return ERROR_ALLOC;
+ }
+ node->ix_sn = (u16 *)kmalloc(node->header.sense_node * (sizeof(u16)),
+ GFP_KERNEL);
+ if (node->ix_sn == NULL) {
+ pr_err("%s: can not allocate memory for ix2_sn ERROR %08X",
+ __func__, ERROR_ALLOC);
+ kfree(node->ix_fm);
+ kfree(node->cx_fm);
+ return ERROR_ALLOC;
+ }
+ node->cx_sn = (short *)kmalloc(node->header.sense_node *
+ (sizeof(short)), GFP_KERNEL);
+ if (node->cx_sn == NULL) {
+ pr_err("%s: can not allocate memory for cx2_sn ERROR %08X",
+ __func__, ERROR_ALLOC);
+ kfree(node->ix_fm);
+ kfree(node->cx_fm);
+ kfree(node->ix_sn);
+ return ERROR_ALLOC;
+ }
+
+
+ pr_info("Address for Node data = %llx\n", address);
+
+ pr_info("Node Data to read %d bytes\n", size);
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, address, data,
+ toRead, DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: error while reading data... ERROR %08X\n",
+ __func__, ret);
+ kfree(node->ix_fm);
+ kfree(node->cx_fm);
+ kfree(node->ix_sn);
+ kfree(node->cx_sn);
+ return ret;
+ }
+
+ pr_info("Read node data ok!\n");
+
+ j = 0;
+ for (i = 0; i < node->header.force_node; i++) {
+ node->ix_fm[i] = ((u16)data[j + 1]) << 8 | data[j];
+ j += 2;
+ }
+
+ for (i = 0; i < node->header.sense_node; i++) {
+ node->ix_sn[i] = ((u16)data[j + 1]) << 8 | data[j];
+ j += 2;
+ }
+
+ for (i = 0; i < node->header.force_node; i++) {
+ node->cx_fm[i] = ((short)data[j + 1]) << 8 | data[j];
+ j += 2;
+ }
+
+ for (i = 0; i < node->header.sense_node; i++) {
+ node->cx_sn[i] = ((short)data[j + 1]) << 8 | data[j];
+ j += 2;
+ }
+
+ if (j != toRead)
+ pr_err("%s: parsed a wrong number of bytes %d!=%d\n",
+ __func__, j, toRead);
+
+ return OK;
+}
+
+/**
+ * Perform all the steps to read the necessary info for TOT SS Initialization
+ * data from the buffer and store it in a TotSelfSenseData variable
+ * @param type type of TOT MS Initialization data to read @link load_opt Load
+ * Host Data Option @endlink
+ * @param data pointer to a variable which will contain the TOT MS
+ * initialization data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readTotSelfSenseCompensationData(u8 type, TotSelfSenseData *data)
+{
+ int ret;
+ u64 address;
+
+ data->ix_fm = NULL;
+ data->cx_fm = NULL;
+ data->ix_sn = NULL;
+ data->cx_sn = NULL;
+
+ if (!(type == LOAD_PANEL_CX_TOT_SS_TOUCH || type ==
+ LOAD_PANEL_CX_TOT_SS_TOUCH_IDLE || type ==
+ LOAD_PANEL_CX_TOT_SS_KEY ||
+ type == LOAD_PANEL_CX_TOT_SS_FORCE)) {
+ pr_err("%s: Choose a TOT SS type of compensation data ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ ret = requestCompensationData(type);
+ if (ret < 0) {
+ pr_err("%s: error while requesting data... ERROR %08X\n",
+ __func__, ERROR_REQU_COMP_DATA);
+ return ret | ERROR_REQU_COMP_DATA;
+ }
+
+ ret = readCompensationDataHeader(type, &(data->header), &address);
+ if (ret < 0) {
+ pr_err("%s: error while reading data header... ERROR %08X\n",
+ __func__, ERROR_COMP_DATA_HEADER);
+ return ret | ERROR_COMP_DATA_HEADER;
+ }
+
+ ret = readTotSelfSenseGlobalData(&address, data);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_GLOBAL);
+ return ret | ERROR_COMP_DATA_GLOBAL;
+ }
+
+ ret = readTotSelfSenseNodeData(address, data);
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_NODE);
+ return ret | ERROR_COMP_DATA_NODE;
+ }
+
+ return OK;
+}
+
+
+/**
+ * Read Initialization Data Header for the Coefficients and check that the type
+ * loaded match with the one previously requested
+ * @param type type of Coefficients data requested @link load_opt Load Host
+ * Data Option @endlink
+ * @param msHeader pointer to DataHeader variable for the MS Coefficients
+ * @param ssHeader pointer to DataHeader variable for the SS Coefficients
+ * @param address pointer to a variable which will contain the updated address
+ * to the next data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readSensitivityCoeffHeader(u8 type, DataHeader *msHeader,
+ DataHeader *ssHeader, u64 *address)
+{
+ u64 offset = ADDR_FRAMEBUFFER;
+ u8 data[SYNCFRAME_DATA_HEADER];
+ int ret;
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, offset, data,
+ SYNCFRAME_DATA_HEADER, DUMMY_FRAMEBUFFER);
+ if (ret < OK) { /* i2c function have already a retry mechanism */
+ pr_err("%s: error while reading data header ERROR %08X\n",
+ __func__, ret);
+ return ret;
+ }
+
+ pr_info("Read Data Header done!\n");
+
+ if (data[0] != HEADER_SIGNATURE) {
+ pr_err("%s: The Header Signature was wrong! %02X != %02X ERROR %08X\n",
+ __func__, data[0], HEADER_SIGNATURE,
+ ERROR_WRONG_DATA_SIGN);
+ return ERROR_WRONG_DATA_SIGN;
+ }
+
+
+ if (data[1] != type) {
+ pr_err("%s: Wrong type found! %02X!=%02X ERROR %08X\n",
+ __func__, data[1], type, ERROR_DIFF_DATA_TYPE);
+ return ERROR_DIFF_DATA_TYPE;
+ }
+
+ pr_info("Type = %02X of Compensation data OK!\n", type);
+
+ msHeader->type = type;
+ ssHeader->type = type;
+
+ msHeader->force_node = data[5];
+ msHeader->sense_node = data[6];
+ pr_info("MS Force Len = %d Sense Len = %d\n",
+ msHeader->force_node, msHeader->sense_node);
+
+ ssHeader->force_node = data[7];
+ ssHeader->sense_node = data[8];
+ pr_info("SS Force Len = %d Sense Len = %d\n",
+ ssHeader->force_node, ssHeader->sense_node);
+
+ *address = offset + SYNCFRAME_DATA_HEADER;
+
+ return OK;
+}
+
+
+/**
+ * Read MS and SS Sensitivity Coefficients for from the IC
+ * @param address a variable which contain the address from where to read the
+ * data
+ * @param msCoeff pointer to MutualSenseCoeff variable which will contain the
+ * MS Coefficient data
+ * @param ssCoeff pointer to SelfSenseCoeff variable which will contain the SS
+ * Coefficient data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readSensitivityCoeffNodeData(u64 address, MutualSenseCoeff *msCoeff,
+ SelfSenseCoeff *ssCoeff)
+{
+ int size = msCoeff->header.force_node * msCoeff->header.sense_node +
+ (ssCoeff->header.force_node + ssCoeff->header.sense_node);
+ u8 data[size];
+ int ret;
+
+ msCoeff->node_data_size = msCoeff->header.force_node *
+ msCoeff->header.sense_node;
+
+ msCoeff->ms_coeff = (u8 *)kmalloc(msCoeff->node_data_size *
+ (sizeof(u8)), GFP_KERNEL);
+
+ ssCoeff->ss_force_coeff = (u8 *)kmalloc(ssCoeff->header.force_node *
+ (sizeof(u8)), GFP_KERNEL);
+
+ ssCoeff->ss_sense_coeff = (u8 *)kmalloc(ssCoeff->header.sense_node *
+ (sizeof(u8)), GFP_KERNEL);
+ if (msCoeff->ms_coeff == NULL ||
+ ssCoeff->ss_force_coeff == NULL ||
+ ssCoeff->ss_sense_coeff == NULL) {
+
+ pr_err("%s: can not allocate memory for coeff ERROR %08X",
+ __func__, ERROR_ALLOC);
+
+ kfree(msCoeff->ms_coeff);
+ msCoeff->ms_coeff = NULL;
+
+ kfree(ssCoeff->ss_force_coeff);
+ ssCoeff->ss_force_coeff = NULL;
+
+ kfree(ssCoeff->ss_sense_coeff);
+ ssCoeff->ss_sense_coeff = NULL;
+
+ return ERROR_ALLOC;
+ }
+
+ pr_info("Address for Node data = %llx\n", address);
+
+ pr_info("Node Data to read %d bytes\n", size);
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, address, data,
+ size, DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: error while reading data... ERROR %08X\n",
+ __func__, ret);
+ kfree(msCoeff->ms_coeff);
+ msCoeff->ms_coeff = NULL;
+ kfree(ssCoeff->ss_force_coeff);
+ ssCoeff->ss_force_coeff = NULL;
+ kfree(ssCoeff->ss_sense_coeff);
+ ssCoeff->ss_sense_coeff = NULL;
+ return ret;
+ }
+
+ pr_info("Read node data ok!\n");
+
+ memcpy(msCoeff->ms_coeff, data, msCoeff->node_data_size);
+ memcpy(ssCoeff->ss_force_coeff, &data[msCoeff->node_data_size],
+ ssCoeff->header.force_node);
+ memcpy(ssCoeff->ss_sense_coeff, &data[msCoeff->node_data_size +
+ ssCoeff->header.force_node],
+ ssCoeff->header.sense_node);
+
+ return OK;
+}
+
+
+/**
+ * Perform all the steps to read Sensitivity Coefficients and store into the
+ * corresponding variables
+ * @param msCoeff pointer to a variable which will contain the MS Sensitivity
+ * Coefficients
+ * @param ssCoeff pointer to a variable which will contain the SS Sensitivity
+ * Coefficients
+ * @return OK if success or an error code which specify the type of error
+ */
+int readSensitivityCoefficientsData(MutualSenseCoeff *msCoeff,
+ SelfSenseCoeff *ssCoeff)
+{
+ int ret;
+ u64 address;
+
+ msCoeff->ms_coeff = NULL;
+ ssCoeff->ss_force_coeff = NULL;
+ ssCoeff->ss_sense_coeff = NULL;
+
+
+ ret = requestCompensationData(LOAD_SENS_CAL_COEFF);
+ if (ret < OK) {
+ pr_err("%s: error while requesting data... ERROR %08X\n",
+ __func__, ERROR_REQU_COMP_DATA);
+ return ret | ERROR_REQU_COMP_DATA;
+ }
+
+ ret = readSensitivityCoeffHeader(LOAD_SENS_CAL_COEFF,
+ &(msCoeff->header), &(ssCoeff->header),
+ &address);
+ if (ret < OK) {
+ pr_err("%s: error while reading data header... ERROR %08X\n",
+ __func__, ERROR_COMP_DATA_HEADER);
+ return ret | ERROR_COMP_DATA_HEADER;
+ }
+
+ ret = readSensitivityCoeffNodeData(address, msCoeff, ssCoeff);
+ if (ret < OK) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_COMP_DATA_NODE);
+ return ret | ERROR_COMP_DATA_NODE;
+ }
+
+ return OK;
+}
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsCompensation.h b/drivers/input/touchscreen/stm/fts_lib/ftsCompensation.h
new file mode 100644
index 0000000..34f4c97
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsCompensation.h
@@ -0,0 +1,165 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS functions for getting Initialization Data **
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsCompensation.h
+ * \brief Contains all the definitions and structs to work with Initialization
+ * Data
+ */
+
+#ifndef FTS_COMPENSATION_H
+#define FTS_COMPENSATION_H
+
+#include "ftsCore.h"
+#include "ftsSoftware.h"
+
+
+
+#define RETRY_COMP_DATA_READ 2 /* /< max number of attempts to read
+ * initialization data */
+
+
+/* Bytes dimension of Compensation Data Format */
+
+#define COMP_DATA_HEADER DATA_HEADER /* /< size in bytes of
+ * initialization data header */
+#define COMP_DATA_GLOBAL (16 - COMP_DATA_HEADER) /* /< size in bytes
+ * of initialization
+ * data general info */
+
+
+#define HEADER_SIGNATURE 0xA5 /* /< signature used as starting byte of
+ * data loaded in memory */
+
+
+
+/**
+ * Struct which contains the general info about Frames and Initialization Data
+ */
+typedef struct {
+ int force_node; /* /< Number of Force Channels in the
+ * frame/Initialization data */
+ int sense_node; /* /< Number of Sense Channels in the
+ * frame/Initialization data */
+ int type; /* /< Type of frame/Initialization data */
+} DataHeader;
+
+/**
+ * Struct which contains the MS Initialization data
+ */
+typedef struct {
+ DataHeader header; /* /< Header */
+ i8 cx1; /* /< Cx1 value (can be negative)) */
+ i8 *node_data; /* /< Pointer to an array of bytes which contains the
+ * CX2 data (can be negative) */
+ int node_data_size; /* /< size of the data */
+} MutualSenseData;
+
+
+/**
+ * Struct which contains the SS Initialization data
+ */
+typedef struct {
+ DataHeader header; /* /< Header */
+ u8 f_ix1; /* /< IX1 Force */
+ u8 s_ix1; /* /< IX1 Sense */
+ i8 f_cx1; /* /< CX1 Force (can be negative) */
+ i8 s_cx1; /* /< CX1 Sense (can be negative) */
+ u8 f_max_n; /* /< Force MaxN */
+ u8 s_max_n; /* /< Sense MaxN */
+
+ u8 *ix2_fm; /* /< pointer to an array of bytes which contains Force
+ * Ix2 data node */
+ u8 *ix2_sn; /* /< pointer to an array of bytes which contains Sense
+ * Ix2 data node */
+ i8 *cx2_fm; /* /< pointer to an array of bytes which contains Force
+ * Cx2 data node
+ * (can be negative) */
+ i8 *cx2_sn; /* /< pointer to an array of bytes which contains Sense
+ * Cx2 data node
+ * (can be negative)) */
+} SelfSenseData;
+
+/**
+ * Struct which contains the TOT MS Initialization data
+ */
+typedef struct {
+ DataHeader header; /* /< Header */
+ short *node_data; /* /< pointer to an array of ushort which
+ * contains TOT MS Initialization data */
+ int node_data_size; /* /< size of data */
+} TotMutualSenseData;
+
+/**
+ * Struct which contains the TOT SS Initialization data
+ */
+typedef struct {
+ DataHeader header; /* /< Header */
+
+ u16 *ix_fm; /* /< pointer to an array of ushort which contains TOT
+ * SS IX Force data */
+ u16 *ix_sn; /* /< pointer to an array of ushort which contains TOT
+ * SS IX Sense data */
+ short *cx_fm; /* /< pointer to an array of ushort which contains TOT
+ * SS CX Force data
+ * (can be negative) */
+ short *cx_sn; /* /< pointer to an array of ushort which contains TOT
+ * SS CX Sense data
+ * (can be negative) */
+} TotSelfSenseData;
+
+/**
+ * Struct which contains the Mutual Sense Sensitivity Calibration Coefficients
+ */
+typedef struct {
+ DataHeader header; /* /< Header */
+ u8 *ms_coeff; /* /< Pointer to an array of bytes which contains the MS
+ * Sens coeff */
+ int node_data_size; /* /< size of coefficients */
+} MutualSenseCoeff;
+
+/**
+ * Struct which contains the Self Sense Sensitivity Calibration Coefficients
+ */
+typedef struct {
+ DataHeader header; /* /< Header */
+ u8 *ss_force_coeff; /* /< Pointer to an array of bytes which
+ * contains the SS Sens Force coeff */
+ u8 *ss_sense_coeff; /* /< Pointer to an array of bytes which
+ * contains the SS Sens Sense coeff */
+} SelfSenseCoeff;
+
+int requestCompensationData(u8 type);
+int readCompensationDataHeader(u8 type, DataHeader *header, u64 *address);
+int readMutualSenseGlobalData(u64 *address, MutualSenseData *global);
+int readMutualSenseNodeData(u64 address, MutualSenseData *node);
+int readMutualSenseCompensationData(u8 type, MutualSenseData *data);
+int readSelfSenseGlobalData(u64 *address, SelfSenseData *global);
+int readSelfSenseNodeData(u64 address, SelfSenseData *node);
+int readSelfSenseCompensationData(u8 type, SelfSenseData *data);
+int readToTMutualSenseGlobalData(u64 *address, TotMutualSenseData *global);
+int readToTMutualSenseNodeData(u64 address, TotMutualSenseData *node);
+int readTotMutualSenseCompensationData(u8 type, TotMutualSenseData *data);
+int readTotSelfSenseGlobalData(u64 *address, TotSelfSenseData *global);
+int readTotSelfSenseNodeData(u64 address, TotSelfSenseData *node);
+int readTotSelfSenseCompensationData(u8 type, TotSelfSenseData *data);
+int readSensitivityCoeffHeader(u8 type, DataHeader *msHeader,
+ DataHeader *ssHeader, u64 *address);
+int readSensitivityCoeffNodeData(u64 address, MutualSenseCoeff *msCoeff,
+ SelfSenseCoeff *ssCoeff);
+int readSensitivityCoefficientsData(MutualSenseCoeff *msData,
+ SelfSenseCoeff *ssData);
+
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsCore.c b/drivers/input/touchscreen/stm/fts_lib/ftsCore.c
new file mode 100644
index 0000000..4dbdc4e
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsCore.c
@@ -0,0 +1,1181 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS Core functions *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsCore.c
+ * \brief Contains the implementation of the Core functions
+ */
+
+#include <linux/spinlock.h>
+#include <linux/delay.h>
+#include <linux/slab.h>
+#include <linux/interrupt.h>
+#include <linux/gpio.h>
+#include "ftsCompensation.h"
+#include "ftsCore.h"
+#include "ftsError.h"
+#include "ftsIO.h"
+#include "ftsTest.h"
+#include "ftsTime.h"
+#include "ftsTool.h"
+#include "ftsFrame.h"
+
+/** @addtogroup system_info
+ * @{
+ */
+SysInfo systemInfo; /* Global System Info variable, accessible in all the
+ * driver */
+/** @}*/
+
+static int reset_gpio = GPIO_NOT_DEFINED;/* /< gpio number of the rest pin,
+ * the value is GPIO_NOT_DEFINED
+ * if the reset pin is not connected */
+static int system_reseted_up; /* /< flag checked during resume to understand
+ * if there was a system reset
+ * and restore the proper state */
+static int system_reseted_down; /* /< flag checked during suspend to understand
+ * if there was a system reset
+ * and restore the proper state */
+static int disable_irq_count = 1; /* /< count the number of call to
+ * disable_irq,
+ * start with 1 because at the boot IRQ
+ * are already disabled */
+spinlock_t fts_int; /* /< spinlock to control the access to the
+ * disable_irq_counter */
+
+
+/**
+ * Initialize core variables of the library.
+ * Must be called during the probe before any other lib function
+ * @param info pointer to fts_ts_info which contains info about the device and
+ * its hw setup
+ * @return OK if success or an error code which specify the type of error
+ */
+int initCore(struct fts_ts_info *info)
+{
+ int ret = OK;
+
+ pr_info("%s: Initialization of the Core...\n", __func__);
+ ret |= openChannel(info->client);
+ ret |= resetErrorList();
+ ret |= initTestToDo();
+ setResetGpio(info->board->reset_gpio);
+ if (ret < OK)
+ pr_err("%s: Initialization Core ERROR %08X!\n",
+ __func__, ret);
+ else
+ pr_info("%s: Initialization Finished!\n",
+ __func__);
+ return ret;
+}
+
+/**
+ * Set the reset_gpio variable with the actual gpio number of the board link to
+ * the reset pin
+ * @param gpio gpio number link to the reset pin of the IC
+ */
+void setResetGpio(int gpio)
+{
+ reset_gpio = gpio;
+ pr_info("setResetGpio: reset_gpio = %d\n", reset_gpio);
+}
+
+/**
+ * Perform a system reset of the IC.
+ * If the reset pin is associated to a gpio, the function execute an hw reset
+ * (toggling of reset pin) otherwise send an hw command to the IC
+ * @return OK if success or an error code which specify the type of error
+ */
+int fts_system_reset(void)
+{
+ u8 readData[FIFO_EVENT_SIZE];
+ int event_to_search;
+ int res = -1;
+ int i;
+ u8 data[1] = { SYSTEM_RESET_VALUE };
+
+ event_to_search = (int)EVT_ID_CONTROLLER_READY;
+
+ pr_info("System resetting...\n");
+ for (i = 0; i < RETRY_SYSTEM_RESET && res < 0; i++) {
+ resetErrorList();
+ fts_disableInterrupt();
+ /* disable interrupt before resetting to be able to get boot
+ * events */
+
+ if (reset_gpio == GPIO_NOT_DEFINED)
+ res = fts_writeU8UX(FTS_CMD_HW_REG_W, ADDR_SIZE_HW_REG,
+ ADDR_SYSTEM_RESET, data, ARRAY_SIZE(
+ data));
+ else {
+ gpio_set_value(reset_gpio, 0);
+ mdelay(10);
+ gpio_set_value(reset_gpio, 1);
+ res = OK;
+ }
+ if (res < OK)
+ pr_err("fts_system_reset: ERROR %08X\n", ERROR_BUS_W);
+ else {
+ res = pollForEvent(&event_to_search, 1, readData,
+ GENERAL_TIMEOUT);
+ if (res < OK)
+ pr_err("fts_system_reset: ERROR %08X\n", res);
+ }
+ }
+ if (res < OK) {
+ pr_err("fts_system_reset...failed after 3 attempts: ERROR %08X\n",
+ (res | ERROR_SYSTEM_RESET_FAIL));
+ return res | ERROR_SYSTEM_RESET_FAIL;
+ } else {
+ pr_debug("System reset DONE!\n");
+ system_reseted_down = 1;
+ system_reseted_up = 1;
+ return OK;
+ }
+}
+
+/**
+ * Return the value of system_resetted_down.
+ * @return the flag value: 0 if not set, 1 if set
+ */
+int isSystemResettedDown(void)
+{
+ return system_reseted_down;
+}
+
+/**
+ * Return the value of system_resetted_up.
+ * @return the flag value: 0 if not set, 1 if set
+ */
+int isSystemResettedUp(void)
+{
+ return system_reseted_up;
+}
+
+
+/**
+ * Set the value of system_reseted_down flag
+ * @param val value to write in the flag
+ */
+void setSystemResetedDown(int val)
+{
+ system_reseted_down = val;
+}
+
+/**
+ * Set the value of system_reseted_up flag
+ * @param val value to write in the flag
+ */
+void setSystemResetedUp(int val)
+{
+ system_reseted_up = val;
+}
+
+
+/** @addtogroup events_group
+ * @{
+ */
+
+/**
+ * Poll the FIFO looking for a specified event within a timeout. Support a
+ * retry mechanism.
+ * @param event_to_search pointer to an array of int where each element
+ * correspond to a byte of the event to find.
+ * If the element of the array has value -1, the byte of the event,
+ * in the same position of the element is ignored.
+ * @param event_bytes size of event_to_search
+ * @param readData pointer to an array of byte which will contain the event
+ * found
+ * @param time_to_wait time to wait before going in timeout
+ * @return OK if success or an error code which specify the type of error
+ */
+int pollForEvent(int *event_to_search, int event_bytes, u8 *readData, int
+ time_to_wait)
+{
+ int i, find, retry, count_err;
+ int time_to_count;
+ int err_handling = OK;
+ StopWatch clock;
+
+ u8 cmd[1] = { FIFO_CMD_READONE };
+ char temp[128] = { 0 };
+
+ find = 0;
+ retry = 0;
+ count_err = 0;
+ time_to_count = time_to_wait / TIMEOUT_RESOLUTION;
+
+ startStopWatch(&clock);
+ while (find != 1 && retry < time_to_count &&
+ fts_writeReadU8UX(cmd[0], 0, 0, readData, FIFO_EVENT_SIZE,
+ DUMMY_FIFO)
+ >= OK) {
+ /* Log of errors */
+ if (readData[0] == EVT_ID_ERROR) {
+ pr_debug("%s\n",
+ printHex("ERROR EVENT = ",
+ readData,
+ FIFO_EVENT_SIZE,
+ temp,
+ sizeof(temp)));
+ memset(temp, 0, 128);
+ count_err++;
+ err_handling = errorHandler(readData, FIFO_EVENT_SIZE);
+ if ((err_handling & 0xF0FF0000) ==
+ ERROR_HANDLER_STOP_PROC) {
+ pr_err("pollForEvent: forced to be stopped! ERROR %08X\n",
+ err_handling);
+ return err_handling;
+ }
+ } else {
+ if (readData[0] != EVT_ID_NOEVENT) {
+ pr_debug("%s\n",
+ printHex("READ EVENT = ", readData,
+ FIFO_EVENT_SIZE,
+ temp,
+ sizeof(temp)));
+ memset(temp, 0, 128);
+ }
+ if (readData[0] == EVT_ID_CONTROLLER_READY &&
+ event_to_search[0] != EVT_ID_CONTROLLER_READY) {
+ pr_err("pollForEvent: Unmanned Controller Ready Event! Setting reset flags...\n");
+ setSystemResetedUp(1);
+ setSystemResetedDown(1);
+ }
+ }
+
+ find = 1;
+
+ for (i = 0; i < event_bytes; i++) {
+ if (event_to_search[i] != -1 && (int)readData[i] !=
+ event_to_search[i]) {
+ find = 0;
+ break;
+ }
+ }
+
+ retry++;
+ mdelay(TIMEOUT_RESOLUTION);
+ }
+ stopStopWatch(&clock);
+ if ((retry >= time_to_count) && find != 1) {
+ pr_err("pollForEvent: ERROR %08X\n", ERROR_TIMEOUT);
+ return ERROR_TIMEOUT;
+ } else if (find == 1) {
+ pr_debug("%s\n",
+ printHex("FOUND EVENT = ",
+ readData,
+ FIFO_EVENT_SIZE,
+ temp,
+ sizeof(temp)));
+ memset(temp, 0, 128);
+ pr_debug("Event found in %d ms (%d iterations)! Number of errors found = %d\n",
+ elapsedMillisecond(&clock), retry, count_err);
+ return count_err;
+ } else {
+ pr_err("pollForEvent: ERROR %08X\n", ERROR_BUS_R);
+ return ERROR_BUS_R;
+ }
+}
+
+/** @}*/
+
+/**
+ * Check that the FW sent the echo even after a command was sent
+ * @param cmd pointer to an array of byte which contain the command previously
+ * sent
+ * @param size size of cmd
+ * @return OK if success or an error code which specify the type of error
+ */
+int checkEcho(u8 *cmd, int size)
+{
+ int ret, i;
+ int event_to_search[FIFO_EVENT_SIZE];
+ u8 readData[FIFO_EVENT_SIZE];
+
+
+ if (size < 1) {
+ pr_err("checkEcho: Error Size = %d not valid!\n", size);
+ return ERROR_OP_NOT_ALLOW;
+ } else {
+ if ((size + 3) > FIFO_EVENT_SIZE)
+ size = FIFO_EVENT_SIZE - 3;
+ /* Echo event 0x43 0x01 xx xx xx xx xx fifo_status
+ * therefore command with more than 5 bytes will be trunked */
+
+ event_to_search[0] = EVT_ID_STATUS_UPDATE;
+ event_to_search[1] = EVT_TYPE_STATUS_ECHO;
+ for (i = 2; i < size + 2; i++)
+ event_to_search[i] = cmd[i - 2];
+ ret = pollForEvent(event_to_search, size + 2, readData,
+ TIEMOUT_ECHO);
+ if (ret < OK) {
+ pr_err("checkEcho: Echo Event not found! ERROR %08X\n",
+ ret);
+ return ret | ERROR_CHECK_ECHO_FAIL;
+ } else if (ret > OK) {
+ pr_err("checkEcho: Echo Event found but with some error events before! num_error = %d\n",
+ ret);
+ return ERROR_CHECK_ECHO_FAIL;
+ }
+
+ pr_info("ECHO OK!\n");
+ return ret;
+ }
+}
+
+
+/** @addtogroup scan_mode
+ * @{
+ */
+/**
+ * Set a scan mode in the IC
+ * @param mode scan mode to set; possible values @link scan_opt Scan Mode
+ * Option @endlink
+ * @param settings option for the selected scan mode
+ * (for example @link active_bitmask Active Mode Bitmask @endlink)
+ * @return OK if success or an error code which specify the type of error
+ */
+int setScanMode(u8 mode, u8 settings)
+{
+ u8 cmd[3] = { FTS_CMD_SCAN_MODE, mode, settings };
+ int ret, size = 3;
+
+ pr_debug("%s: Setting scan mode: mode = %02X settings = %02X !\n",
+ __func__, mode, settings);
+ if (mode == SCAN_MODE_LOW_POWER)
+ size = 2;
+ ret = fts_write(cmd, size);
+ /* use write instead of writeFw because can be called while the
+ * interrupt are enabled */
+ if (ret < OK) {
+ pr_err("%s: write failed...ERROR %08X !\n",
+ __func__, ret);
+ return ret | ERROR_SET_SCAN_MODE_FAIL;
+ }
+ pr_debug("%s: Setting scan mode OK!\n", __func__);
+ return OK;
+}
+/** @}*/
+
+
+/** @addtogroup feat_sel
+ * @{
+ */
+/**
+ * Set a feature and its option in the IC
+ * @param feat feature to set; possible values @link feat_opt Feature Selection
+ * Option @endlink
+ * @param settings pointer to an array of byte which store the options for
+ * the selected feature (for example the gesture mask to activate
+ * @link gesture_opt Gesture IDs @endlink)
+ * @param size in bytes of settings
+ * @return OK if success or an error code which specify the type of error
+ */
+int setFeatures(u8 feat, u8 *settings, int size)
+{
+ u8 cmd[2 + size];
+ int i = 0;
+ int ret;
+ char buff[(2 + 1) * size + 1];
+ int buff_len = sizeof(buff);
+ int index = 0;
+
+ pr_info("%s: Setting feature: feat = %02X !\n", __func__, feat);
+ cmd[0] = FTS_CMD_FEATURE;
+ cmd[1] = feat;
+ for (i = 0; i < size; i++) {
+ cmd[2 + i] = settings[i];
+ index += scnprintf(buff + index, buff_len - index,
+ "%02X ", settings[i]);
+ }
+ pr_info("%s: Settings = %s\n", __func__, buff);
+ ret = fts_write(cmd, 2 + size);
+ /* use write instead of writeFw because can be called while the
+ * interrupts are enabled */
+ if (ret < OK) {
+ pr_err("%s: write failed...ERROR %08X !\n", __func__, ret);
+ return ret | ERROR_SET_FEATURE_FAIL;
+ }
+ pr_info("%s: Setting feature OK!\n", __func__);
+ return OK;
+}
+/** @}*/
+
+/** @addtogroup sys_cmd
+ * @{
+ */
+/**
+ * Write a system command to the IC
+ * @param sys_cmd System Command to execute; possible values
+ * @link sys_opt System Command Option @endlink
+ * @param sett settings option for the selected system command
+ * (@link sys_special_opt Special Command Option @endlink, @link ito_opt
+ * ITO Test Option @endlink, @link load_opt Load Host Data Option @endlink)
+ * @param size in bytes of settings
+ * @return OK if success or an error code which specify the type of error
+ */
+int writeSysCmd(u8 sys_cmd, u8 *sett, int size)
+{
+ u8 cmd[2 + size];
+ int ret;
+ char buff[(2 + 1) * size + 1];
+ int buff_len = sizeof(buff);
+ int index = 0;
+
+ cmd[0] = FTS_CMD_SYSTEM;
+ cmd[1] = sys_cmd;
+
+ for (ret = 0; ret < size; ret++) {
+ cmd[2 + ret] = sett[ret];
+ index += scnprintf(buff + index, buff_len - index,
+ "%02X ", sett[ret]);
+ }
+ pr_info("%s: Command = %02X %02X %s\n", __func__, cmd[0],
+ cmd[1], buff);
+ pr_info("%s: Writing Sys command...\n", __func__);
+ if (sys_cmd != SYS_CMD_LOAD_DATA)
+ ret = fts_writeFwCmd(cmd, 2 + size);
+ else {
+ if (size >= 1)
+ ret = requestSyncFrame(sett[0]);
+ else {
+ pr_err("%s: No setting argument! ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+ }
+ if (ret < OK)
+ pr_err("%s: ERROR %08X\n", __func__, ret);
+ else
+ pr_info("%s: FINISHED!\n", __func__);
+
+ return ret;
+}
+/** @}*/
+
+/** @addtogroup system_info
+ * @{
+ */
+/**
+ * Initialize the System Info Struct with default values according to the error
+ * found during the reading
+ * @param i2cError 1 if there was an I2C error while reading the System Info
+ * data from memory, other value if another error occurred
+ * @return OK if success or an error code which specify the type of error
+ */
+int defaultSysInfo(int i2cError)
+{
+ int i;
+
+ pr_info("Setting default System Info...\n");
+
+ if (i2cError == 1) {
+ systemInfo.u16_fwVer = 0xFFFF;
+ systemInfo.u16_cfgProjectId = 0xFFFF;
+ for (i = 0; i < RELEASE_INFO_SIZE; i++)
+ systemInfo.u8_releaseInfo[i] = 0xFF;
+ systemInfo.u16_cxVer = 0xFFFF;
+ } else {
+ systemInfo.u16_fwVer = 0x0000;
+ systemInfo.u16_cfgProjectId = 0x0000;
+ for (i = 0; i < RELEASE_INFO_SIZE; i++)
+ systemInfo.u8_releaseInfo[i] = 0x00;
+ systemInfo.u16_cxVer = 0x0000;
+ }
+
+ systemInfo.u8_scrRxLen = 0;
+ systemInfo.u8_scrTxLen = 0;
+
+ pr_info("default System Info DONE!\n");
+ return OK;
+}
+
+/**
+ * Read the System Info data from memory. System Info is loaded automatically
+ * after every system reset.
+ * @param request if 1, will be asked to the FW to reload the data, otherwise
+ * attempt to read it directly from memory
+ * @return OK if success or an error code which specify the type of error
+ */
+int readSysInfo(int request)
+{
+ int ret, i, index = 0;
+ u8 sett = LOAD_SYS_INFO;
+ u8 data[SYS_INFO_SIZE] = { 0 };
+ char temp[256] = { 0 };
+
+ if (request == 1) {
+ pr_info("%s: Requesting System Info...\n", __func__);
+
+ ret = writeSysCmd(SYS_CMD_LOAD_DATA, &sett, 1);
+ if (ret < OK) {
+ pr_err("%s: error while writing the sys cmd ERROR %08X\n",
+ __func__, ret);
+ goto FAIL;
+ }
+ }
+
+ pr_info("%s: Reading System Info...\n", __func__);
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16,
+ ADDR_FRAMEBUFFER, data, SYS_INFO_SIZE,
+ DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: error while reading the system data ERROR %08X\n",
+ __func__, ret);
+ goto FAIL;
+ }
+
+ pr_info("%s: Parsing System Info...\n", __func__);
+
+ if (data[0] != HEADER_SIGNATURE) {
+ pr_err("%s: The Header Signature is wrong! sign: %02X != %02X ERROR %08X\n",
+ __func__, data[0], HEADER_SIGNATURE,
+ ERROR_WRONG_DATA_SIGN);
+ ret = ERROR_WRONG_DATA_SIGN;
+ goto FAIL;
+ }
+
+
+ if (data[1] != LOAD_SYS_INFO) {
+ pr_err("%s: The Data ID is wrong! ids: %02X != %02X ERROR %08X\n",
+ __func__, data[3], LOAD_SYS_INFO,
+ ERROR_DIFF_DATA_TYPE);
+ ret = ERROR_DIFF_DATA_TYPE;
+ goto FAIL;
+ }
+
+ index += 4;
+ u8ToU16(&data[index], &systemInfo.u16_apiVer_rev);
+ index += 2;
+ systemInfo.u8_apiVer_minor = data[index++];
+ systemInfo.u8_apiVer_major = data[index++];
+ u8ToU16(&data[index], &systemInfo.u16_chip0Ver);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_chip0Id);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_chip1Ver);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_chip1Id);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_fwVer);
+ index += 2;
+ pr_info("FW VER = %04X\n", systemInfo.u16_fwVer);
+ u8ToU16(&data[index], &systemInfo.u16_svnRev);
+ index += 2;
+ pr_info("SVN REV = %04X\n", systemInfo.u16_svnRev);
+ u8ToU16(&data[index], &systemInfo.u16_cfgVer);
+ index += 2;
+ pr_info("CONFIG VER = %04X\n", systemInfo.u16_cfgVer);
+ u8ToU16(&data[index], &systemInfo.u16_cfgProjectId);
+ index += 2;
+ pr_info("CONFIG PROJECT ID = %04X\n", systemInfo.u16_cfgProjectId);
+ u8ToU16(&data[index], &systemInfo.u16_cxVer);
+ index += 2;
+ pr_info("CX VER = %04X\n", systemInfo.u16_cxVer);
+ u8ToU16(&data[index], &systemInfo.u16_cxProjectId);
+ index += 2;
+ pr_info("CX PROJECT ID = %04X\n", systemInfo.u16_cxProjectId);
+ systemInfo.u8_cfgAfeVer = data[index++];
+ systemInfo.u8_cxAfeVer = data[index++];
+ systemInfo.u8_panelCfgAfeVer = data[index++];
+ pr_info("AFE VER: CFG = %02X - CX = %02X - PANEL = %02X\n",
+ systemInfo.u8_cfgAfeVer, systemInfo.u8_cxAfeVer,
+ systemInfo.u8_panelCfgAfeVer);
+ systemInfo.u8_protocol = data[index++];
+ pr_info("Protocol = %02X\n", systemInfo.u8_protocol);
+ /* index+= 1; */
+ /* skip reserved area */
+
+ /* pr_err("Die Info = "); */
+ for (i = 0; i < DIE_INFO_SIZE; i++)
+ systemInfo.u8_dieInfo[i] = data[index++];
+
+ /* pr_err("\n"); */
+ pr_info("%s\n",
+ printHex("Die Info = ",
+ systemInfo.u8_dieInfo,
+ DIE_INFO_SIZE, temp, sizeof(temp)));
+ memset(temp, 0, 256);
+
+
+ /* pr_err("Release Info = "); */
+ for (i = 0; i < RELEASE_INFO_SIZE; i++)
+ systemInfo.u8_releaseInfo[i] = data[index++];
+
+ /* pr_err("\n"); */
+
+ pr_info("%s\n",
+ printHex("Release Info = ",
+ systemInfo.u8_releaseInfo,
+ RELEASE_INFO_SIZE,
+ temp,
+ sizeof(temp)));
+ memset(temp, 0, 256);
+
+ u8ToU32(&data[index], &systemInfo.u32_fwCrc);
+ index += 4;
+ u8ToU32(&data[index], &systemInfo.u32_cfgCrc);
+ index += 4;
+
+ index += 16; /* skip reserved area */
+
+ u8ToU16(&data[index], &systemInfo.u16_scrResX);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_scrResY);
+ index += 2;
+ pr_info("Screen Resolution = %d x %d\n",
+ systemInfo.u16_scrResX, systemInfo.u16_scrResY);
+ systemInfo.u8_scrTxLen = data[index++];
+ pr_info("TX Len = %d\n", systemInfo.u8_scrTxLen);
+ systemInfo.u8_scrRxLen = data[index++];
+ pr_info("RX Len = %d\n", systemInfo.u8_scrRxLen);
+ systemInfo.u8_keyLen = data[index++];
+ pr_info("Key Len = %d\n", systemInfo.u8_keyLen);
+ systemInfo.u8_forceLen = data[index++];
+ pr_info("Force Len = %d\n", systemInfo.u8_forceLen);
+
+ index += 40; /* skip reserved area */
+
+ u8ToU16(&data[index], &systemInfo.u16_dbgInfoAddr);
+ index += 2;
+
+ index += 6; /* skip reserved area */
+
+ u8ToU16(&data[index], &systemInfo.u16_msTchRawAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_msTchFilterAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_msTchStrenAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_msTchBaselineAddr);
+ index += 2;
+
+ u8ToU16(&data[index], &systemInfo.u16_ssTchTxRawAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssTchTxFilterAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssTchTxStrenAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssTchTxBaselineAddr);
+ index += 2;
+
+ u8ToU16(&data[index], &systemInfo.u16_ssTchRxRawAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssTchRxFilterAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssTchRxStrenAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssTchRxBaselineAddr);
+ index += 2;
+
+ u8ToU16(&data[index], &systemInfo.u16_keyRawAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_keyFilterAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_keyStrenAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_keyBaselineAddr);
+ index += 2;
+
+ u8ToU16(&data[index], &systemInfo.u16_frcRawAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_frcFilterAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_frcStrenAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_frcBaselineAddr);
+ index += 2;
+
+ u8ToU16(&data[index], &systemInfo.u16_ssHvrTxRawAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssHvrTxFilterAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssHvrTxStrenAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssHvrTxBaselineAddr);
+ index += 2;
+
+ u8ToU16(&data[index], &systemInfo.u16_ssHvrRxRawAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssHvrRxFilterAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssHvrRxStrenAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssHvrRxBaselineAddr);
+ index += 2;
+
+ u8ToU16(&data[index], &systemInfo.u16_ssPrxTxRawAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssPrxTxFilterAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssPrxTxStrenAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssPrxTxBaselineAddr);
+ index += 2;
+
+ u8ToU16(&data[index], &systemInfo.u16_ssPrxRxRawAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssPrxRxFilterAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssPrxRxStrenAddr);
+ index += 2;
+ u8ToU16(&data[index], &systemInfo.u16_ssPrxRxBaselineAddr);
+ index += 2;
+
+ pr_info("Parsed %d bytes!\n", index);
+
+
+ if (index != SYS_INFO_SIZE) {
+ pr_err("%s: index = %d different from %d ERROR %08X\n",
+ __func__, index, SYS_INFO_SIZE,
+ ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ pr_info("System Info Read DONE!\n");
+ return OK;
+
+FAIL:
+ defaultSysInfo(isI2cError(ret));
+ return ret;
+}
+/** @}*/
+
+
+/**
+ * Read data from the Config Memory
+ * @param offset Starting address in the Config Memory of data to read
+ * @param outBuf pointer of a byte array which contain the bytes to read
+ * @param len number of bytes to read
+ * @return OK if success or an error code which specify the type of error
+ */
+int readConfig(u16 offset, u8 *outBuf, int len)
+{
+ int ret;
+ u64 final_address = offset + ADDR_CONFIG_OFFSET;
+
+ pr_info("%s: Starting to read config memory at %llx ...\n",
+ __func__, final_address);
+ ret = fts_writeReadU8UX(FTS_CMD_CONFIG_R, BITS_16, final_address,
+ outBuf, len, DUMMY_CONFIG);
+ if (ret < OK) {
+ pr_err("%s: Impossible to read Config Memory... ERROR %08X!\n",
+ __func__, ret);
+ return ret;
+ }
+
+ pr_info("%s: Read config memory FINISHED!\n", __func__);
+ return OK;
+}
+
+/**
+ * Write data into the Config Memory
+ * @param offset Starting address in the Config Memory where write the data
+ * @param data pointer of a byte array which contain the data to write
+ * @param len number of bytes to write
+ * @return OK if success or an error code which specify the type of error
+ */
+int writeConfig(u16 offset, u8 *data, int len)
+{
+ int ret;
+ u64 final_address = offset + ADDR_CONFIG_OFFSET;
+
+ pr_info("%s: Starting to write config memory at %llx ...\n",
+ __func__, final_address);
+ ret = fts_writeU8UX(FTS_CMD_CONFIG_W, BITS_16, final_address, data,
+ len);
+ if (ret < OK) {
+ pr_err("%s: Impossible to write Config Memory... ERROR %08X!\n",
+ __func__, ret);
+ return ret;
+ }
+
+ pr_info("%s: Write config memory FINISHED!\n", __func__);
+ return OK;
+}
+
+/**
+ * Disable the interrupt so the ISR of the driver can not be called
+ * @return OK if success or an error code which specify the type of error
+ */
+int fts_disableInterrupt(void)
+{
+ unsigned long flag;
+
+ if (getClient() != NULL) {
+ spin_lock_irqsave(&fts_int, flag);
+ pr_debug("Number of disable = %d\n", disable_irq_count);
+ if (disable_irq_count == 0) {
+ pr_debug("Executing Disable...\n");
+ disable_irq_nosync(getClient()->irq);
+ disable_irq_count++;
+ }
+ /* disable_irq is re-entrant so it is required to keep track
+ * of the number of calls of this when reenabling */
+ spin_unlock_irqrestore(&fts_int, flag);
+ pr_debug("Interrupt Disabled!\n");
+ return OK;
+ } else {
+ pr_err("%s: Impossible get client irq... ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+}
+
+
+/**
+ * Disable the interrupt async so the ISR of the driver can not be called
+ * @return OK if success or an error code which specify the type of error
+ */
+int fts_disableInterruptNoSync(void)
+{
+ if (getClient() != NULL) {
+ spin_lock_irq(&fts_int);
+ pr_debug("Number of disable = %d\n", disable_irq_count);
+ if (disable_irq_count == 0) {
+ pr_debug("Executing Disable...\n");
+ disable_irq_nosync(getClient()->irq);
+ disable_irq_count++;
+ }
+ /* disable_irq is re-entrant so it is required to keep track
+ * of the number of calls of this when reenabling */
+
+ spin_unlock(&fts_int);
+ pr_debug("Interrupt No Sync Disabled!\n");
+ return OK;
+ } else {
+ pr_err("%s: Impossible get client irq... ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+}
+
+/**
+ * Reset the disable_irq count
+ * @return OK
+ */
+int fts_resetDisableIrqCount(void)
+{
+ disable_irq_count = 0;
+ return OK;
+}
+
+/**
+ * Enable the interrupt so the ISR of the driver can be called
+ * @return OK if success or an error code which specify the type of error
+ */
+int fts_enableInterrupt(void)
+{
+ unsigned long flag;
+
+ if (getClient() != NULL) {
+ spin_lock_irqsave(&fts_int, flag);
+ pr_debug("Number of re-enable = %d\n", disable_irq_count);
+ while (disable_irq_count > 0) {
+ /* loop N times according on the pending number of
+ * disable_irq to truly re-enable the int */
+ pr_debug("Executing Enable...\n");
+ enable_irq(getClient()->irq);
+ disable_irq_count--;
+ }
+
+ spin_unlock_irqrestore(&fts_int, flag);
+ pr_debug("Interrupt Enabled!\n");
+ return OK;
+ } else {
+ pr_err("%s: Impossible get client irq... ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+}
+
+/**
+ * Check if there is a crc error in the IC which prevent the fw to run.
+ * @return OK if no CRC error, or a number >OK according the CRC error found
+ */
+int fts_crc_check(void)
+{
+ u8 val;
+ u8 crc_status;
+ int res;
+ u8 error_to_search[6] = { EVT_TYPE_ERROR_CRC_CFG_HEAD,
+ EVT_TYPE_ERROR_CRC_CFG,
+ EVT_TYPE_ERROR_CRC_CX,
+ EVT_TYPE_ERROR_CRC_CX_HEAD,
+ EVT_TYPE_ERROR_CRC_CX_SUB,
+ EVT_TYPE_ERROR_CRC_CX_SUB_HEAD };
+
+
+ res = fts_writeReadU8UX(FTS_CMD_HW_REG_R, ADDR_SIZE_HW_REG, ADDR_CRC,
+ &val, 1, DUMMY_HW_REG);
+ /* read 2 bytes because the first one is a dummy byte! */
+ if (res < OK) {
+ pr_err("%s Cannot read crc status ERROR %08X\n", __func__, res);
+ return res;
+ }
+
+ crc_status = val & CRC_MASK;
+ if (crc_status != OK) { /* CRC error if crc_status!=0 */
+ pr_err("%s CRC ERROR = %02X\n", __func__, crc_status);
+ return CRC_CODE;
+ }
+
+ pr_info("%s: Verifying if Config CRC Error...\n", __func__);
+ res = fts_system_reset();
+ if (res >= OK) {
+ res = pollForErrorType(error_to_search, 2);
+ if (res < OK) {
+ pr_info("%s: No Config CRC Error Found!\n", __func__);
+ pr_info("%s: Verifying if Cx CRC Error...\n", __func__);
+ res = pollForErrorType(&error_to_search[2], 4);
+ if (res < OK) {
+ pr_info("%s: No Cx CRC Error Found!\n",
+ __func__);
+ return OK;
+ } else {
+ pr_err("%s: Cx CRC Error found! CRC ERROR = %02X\n",
+ __func__, res);
+ return CRC_CX;
+ }
+ } else {
+ pr_err("%s: Config CRC Error found! CRC ERROR = %02X\n",
+ __func__, res);
+ return CRC_CONFIG;
+ }
+ } else {
+ pr_err("%s: Error while executing system reset! ERROR %08X\n",
+ __func__, res);
+ return res;
+ }
+
+ return OK;
+}
+
+/**
+ * Request a host data and use the sync method to understand when the FW load
+ * it
+ * @param type the type ID of host data to load (@link load_opt Load Host Data
+ * Option @endlink)
+ * @return OK if success or an error code which specify the type of error
+ */
+int requestSyncFrame(u8 type)
+{
+ u8 request[3] = { FTS_CMD_SYSTEM, SYS_CMD_LOAD_DATA, type };
+ u8 readData[DATA_HEADER] = { 0 };
+ int ret, retry = 0, retry2 = 0, time_to_count;
+ int count, new_count;
+
+ pr_info("%s: Starting to get a sync frame...\n", __func__);
+
+ while (retry2 < RETRY_MAX_REQU_DATA) {
+ pr_info("%s: Reading count...\n", __func__);
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16,
+ ADDR_FRAMEBUFFER, readData, DATA_HEADER,
+ DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: Error while reading count! ERROR %08X\n",
+ __func__, ret | ERROR_REQU_DATA);
+ ret |= ERROR_REQU_DATA;
+ retry2++;
+ continue;
+ }
+
+ if (readData[0] != HEADER_SIGNATURE)
+ pr_err("%s: Invalid Signature while reading count! ERROR %08X\n",
+ __func__, ret | ERROR_REQU_DATA);
+
+ count = (readData[3] << 8) | readData[2];
+ new_count = count;
+ pr_info("%s: Base count = %d\n", __func__, count);
+
+ pr_info("%s: Requesting frame %02X attempt = %d\n",
+ __func__, type, retry2 + 1);
+ ret = fts_write(request, ARRAY_SIZE(request));
+ if (ret >= OK) {
+ pr_info("%s: Polling for new count...\n", __func__);
+ time_to_count = TIMEOUT_REQU_DATA / TIMEOUT_RESOLUTION;
+ while (count == new_count && retry < time_to_count) {
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R,
+ BITS_16,
+ ADDR_FRAMEBUFFER,
+ readData,
+ DATA_HEADER,
+ DUMMY_FRAMEBUFFER);
+ if ((ret >= OK) && (readData[0] ==
+ HEADER_SIGNATURE) &&
+ (readData[1] == type))
+ new_count = ((readData[3] << 8) |
+ readData[2]);
+ else
+ pr_err("%s: invalid Signature or can not read count... ERROR %08X\n",
+ __func__, ret);
+ retry++;
+ mdelay(TIMEOUT_RESOLUTION);
+ }
+
+ if (count == new_count) {
+ pr_err("%s: New count not received! ERROR %08X\n",
+ __func__,
+ ERROR_TIMEOUT | ERROR_REQU_DATA);
+ ret = ERROR_TIMEOUT | ERROR_REQU_DATA;
+ } else {
+ pr_info("%s: New count found! count = %d! Frame ready!\n",
+ __func__, new_count);
+ return OK;
+ }
+ }
+ retry2++;
+ }
+ pr_err("%s: Request Data failed! ERROR %08X\n", __func__, ret);
+ return ret;
+}
+
+/**
+ * Set the Active Scanning Frequency to a defined value
+ * @param freq scanning frequency to set in Hz
+ * @return OK if success or an error code which specify the type of error
+ * @warning The scan frequency can be set only for the MS scan!
+ */
+int setActiveScanFrequency(u32 freq)
+{
+ int res;
+ u8 temp[2] = { 0 };
+ u16 t_cycle;
+
+ pr_info("%s: Setting the scanning frequency to %uHz...\n",
+ __func__, freq);
+
+ /* read MRN register */
+ res = readConfig(ADDR_CONFIG_MRN, temp, 1);
+ if (res < OK) {
+ pr_err("%s: error while reading mrn count! ERROR %08X\n",
+ __func__, res);
+ return res;
+ }
+
+ /* setting r count to 0 (= 1 R cycle used) and write it back */
+ temp[0] &= (~0x03);
+ res = writeConfig(ADDR_CONFIG_MRN, temp, 1);
+ if (res < OK) {
+ pr_err("%s: error while writing mrn count! ERROR %08X\n",
+ __func__, res);
+ return res;
+ }
+
+ /* set first R cycle slot according the specified frequency */
+ /* read T cycle */
+ res = readConfig(ADDR_CONFIG_T_CYCLE, temp, 2);
+ if (res < OK) {
+ pr_err("%s: error while reading T cycle! ERROR %08X\n",
+ __func__, res);
+ return res;
+ }
+ t_cycle = ((u16)(temp[1] << 8)) | temp[0];
+
+
+ /* compute the value of R cycle according the formula
+ * scan_freq = 30Mhz/(2*(T_cycle+R_cycle)) */
+ temp[0] = (30000000) / (freq * 2) - t_cycle;
+ /* write R cycle in Config Area */
+ pr_info("%s: T cycle = %d (0x%04X) => R0 cycle = %d (0x%02X)\n",
+ __func__, t_cycle, t_cycle, temp[0], temp[0]);
+ res = writeConfig(ADDR_CONFIG_R0_CYCLE, temp, 1);
+ if (res < OK) {
+ pr_err("%s: error while writing R0 cycle! ERROR %08X\n",
+ __func__, res);
+ return res;
+ }
+
+ pr_info("%s: Saving Config into the flash ...\n", __func__);
+ /* save config */
+ temp[0] = SAVE_FW_CONF;
+ res = writeSysCmd(SYS_CMD_SAVE_FLASH, temp, 1);
+ if (res < OK) {
+ pr_err("%s: error while saving config into the flash! ERROR %08X\n",
+ __func__, res);
+ return res;
+ }
+
+ /* system reset */
+ res = fts_system_reset();
+ if (res < OK) {
+ pr_err("%s: error at system reset! ERROR %08X\n",
+ __func__, res);
+ return res;
+ }
+
+ pr_info("%s: Setting the scanning frequency FINISHED!\n", __func__);
+ return OK;
+}
+
+/**
+ * Write Host Data Memory
+ * @param type type of data to write
+ * @param data pointer to the data which are written
+ * @param msForceLen number of force (Tx) channels used with Mutual
+ * @param msSenseLen number of sense (Rx) channels used with Mutual
+ * @param ssForceLen number of force (Tx) channel used with Self
+ * @param ssSenseLen number of sense (Rx) channel used with Self
+ * @param save if =1 will save the host data written into the flash
+ * @return OK if success or an error code which specify the type of error
+ */
+int writeHostDataMemory(u8 type, u8 *data, u8 msForceLen, u8 msSenseLen,
+ u8 ssForceLen, u8 ssSenseLen, int save)
+{
+ int res;
+ int size = (msForceLen * msSenseLen) + (ssForceLen + ssSenseLen);
+ u8 sett = SPECIAL_WRITE_HOST_MEM_TO_FLASH;
+ u8 temp[size + SYNCFRAME_DATA_HEADER];
+
+ memset(temp, 0, size + SYNCFRAME_DATA_HEADER);
+ pr_info("%s: Starting to write Host Data Memory\n", __func__);
+
+ temp[0] = 0x5A;
+ temp[1] = type;
+ temp[5] = msForceLen;
+ temp[6] = msSenseLen;
+ temp[7] = ssForceLen;
+ temp[8] = ssSenseLen;
+
+ memcpy(&temp[16], data, size);
+
+ pr_info("%s: Write Host Data Memory in buffer...\n", __func__);
+ res = fts_writeU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16,
+ ADDR_FRAMEBUFFER, temp, size +
+ SYNCFRAME_DATA_HEADER);
+
+ if (res < OK) {
+ pr_err("%s: error while writing the buffer! ERROR %08X\n",
+ __func__, res);
+ return res;
+ }
+
+ /* save host data memory into the flash */
+ if (save == 1) {
+ pr_info("%s: Trigger writing into the flash...\n", __func__);
+ res = writeSysCmd(SYS_CMD_SPECIAL, &sett, 1);
+ if (res < OK) {
+ pr_err("%s: error while writing into the flash! ERROR %08X\n",
+ __func__, res);
+ return res;
+ }
+ }
+
+
+ pr_info("%s: write Host Data Memory FINISHED!\n", __func__);
+ return OK;
+}
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsCore.h b/drivers/input/touchscreen/stm/fts_lib/ftsCore.h
new file mode 100644
index 0000000..ea7608c
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsCore.h
@@ -0,0 +1,207 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS Core definitions **
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsCore.h
+ * \brief Contains all the definitions and structs of Core functionalities
+ */
+
+#ifndef FTS_CORE_H
+#define FTS_CORE_H
+
+#include "ftsHardware.h"
+#include "ftsSoftware.h"
+#include "../fts.h"
+
+/* HW DATA */
+#define GPIO_NOT_DEFINED -1 /* /< value assumed by reset_gpio when
+ * the reset pin of the IC is not
+ * connected */
+
+
+#define ADDR_SIZE_HW_REG BITS_32 /* /< value of AddrSize for Hw register
+ * in FTI @see AddrSize */
+
+#define DATA_HEADER 4 /* /< size in byte of the header loaded
+ * with the data in the frambuffer */
+
+/**
+ * Type of CRC errors
+ */
+typedef enum {
+ CRC_CODE = 1, /* /< CRC in the code section */
+ CRC_CONFIG = 2, /* /< CRC in the config section */
+ CRC_CX = 3, /* /< CRC in the cx section */
+ CRC_PANEL = 4 /* /< CRC in the panel section */
+} CRC_Error;
+
+/* CHIP INFO */
+/** @defgroup system_info System Info
+ * System Info Data collect the most important information about hw and fw
+ * @{
+ */
+/* Size in bytes of System Info data */
+#define SYS_INFO_SIZE 208 /* Num bytes of die info */
+#define DIE_INFO_SIZE 16 /* Num bytes of external release
+ * in config */
+#define EXTERNAL_RELEASE_INFO_SIZE 8 /* Num bytes of release info in
+ * sys info
+ * (first bytes are external
+ * release) */
+#define RELEASE_INFO_SIZE (EXTERNAL_RELEASE_INFO_SIZE)
+/** @}*/
+
+/* RETRY MECHANISM */
+#define RETRY_MAX_REQU_DATA 2 /* /< Max number of attempts
+ * performed
+ * when requesting data */
+#define RETRY_SYSTEM_RESET 3 /* /< Max number of attempts
+ * performed
+ * to reset the IC */
+
+/** @addtogroup system_info
+ * @{
+ */
+
+/**
+ * Struct which contains fundamental information about the chip and its
+ * configuration
+ */
+typedef struct {
+ u16 u16_apiVer_rev; /* /< API revision version */
+ u8 u8_apiVer_minor; /* /< API minor version */
+ u8 u8_apiVer_major; /* /< API major version */
+ u16 u16_chip0Ver; /* /< Dev0 version */
+ u16 u16_chip0Id; /* /< Dev0 ID */
+ u16 u16_chip1Ver; /* /< Dev1 version */
+ u16 u16_chip1Id; /* /< Dev1 ID */
+ u16 u16_fwVer; /* /< Fw version */
+ u16 u16_svnRev; /* /< SVN Revision */
+ u16 u16_cfgVer; /* /< Config Version */
+ u16 u16_cfgProjectId; /* /< Config Project ID */
+ u16 u16_cxVer; /* /< Cx Version */
+ u16 u16_cxProjectId; /* /< Cx Project ID */
+ u8 u8_cfgAfeVer; /* /< AFE version in Config */
+ u8 u8_cxAfeVer; /* /< AFE version in CX */
+ u8 u8_panelCfgAfeVer; /* /< AFE version in PanelMem */
+ u8 u8_protocol; /* /< Touch Report Protocol */
+ u8 u8_dieInfo[DIE_INFO_SIZE]; /* /< Die information */
+ u8 u8_releaseInfo[RELEASE_INFO_SIZE]; /* /< Release information */
+ u32 u32_fwCrc; /* /< Crc of FW */
+ u32 u32_cfgCrc; /* /< Crc of config */
+
+ u16 u16_scrResX;/* /< X resolution on main screen */
+ u16 u16_scrResY;/* /< Y resolution on main screen */
+ u8 u8_scrTxLen; /* /< Tx length */
+ u8 u8_scrRxLen; /* /< Rx length */
+ u8 u8_keyLen; /* /< Key Len */
+ u8 u8_forceLen; /* /< Force Len */
+
+ u16 u16_dbgInfoAddr; /* /< Offset of debug Info structure */
+
+ u16 u16_msTchRawAddr; /* /< Offset of MS touch raw frame */
+ u16 u16_msTchFilterAddr;/* /< Offset of MS touch filter frame */
+ u16 u16_msTchStrenAddr; /* /< Offset of MS touch strength frame */
+ u16 u16_msTchBaselineAddr; /* /< Offset of MS touch baseline frame
+ * */
+
+ u16 u16_ssTchTxRawAddr; /* /< Offset of SS touch force raw frame */
+ u16 u16_ssTchTxFilterAddr; /* /< Offset of SS touch force filter
+ * frame */
+ u16 u16_ssTchTxStrenAddr;/* /< Offset of SS touch force strength frame
+ * */
+ u16 u16_ssTchTxBaselineAddr; /* /< Offset of SS touch force baseline
+ * frame */
+
+ u16 u16_ssTchRxRawAddr; /* /< Offset of SS touch sense raw frame */
+ u16 u16_ssTchRxFilterAddr; /* /< Offset of SS touch sense filter
+ * frame */
+ u16 u16_ssTchRxStrenAddr;/* /< Offset of SS touch sense strength frame
+ * */
+ u16 u16_ssTchRxBaselineAddr; /* /< Offset of SS touch sense baseline
+ * frame */
+
+ u16 u16_keyRawAddr; /* /< Offset of key raw frame */
+ u16 u16_keyFilterAddr; /* /< Offset of key filter frame */
+ u16 u16_keyStrenAddr; /* /< Offset of key strength frame */
+ u16 u16_keyBaselineAddr; /* /< Offset of key baseline frame */
+
+ u16 u16_frcRawAddr; /* /< Offset of force touch raw frame */
+ u16 u16_frcFilterAddr; /* /< Offset of force touch filter frame */
+ u16 u16_frcStrenAddr; /* /< Offset of force touch strength frame */
+ u16 u16_frcBaselineAddr;/* /< Offset of force touch baseline frame */
+
+ u16 u16_ssHvrTxRawAddr; /* /< Offset of SS hover Force raw frame */
+ u16 u16_ssHvrTxFilterAddr; /* /< Offset of SS hover Force filter
+ * frame */
+ u16 u16_ssHvrTxStrenAddr;/* /< Offset of SS hover Force strength frame
+ * */
+ u16 u16_ssHvrTxBaselineAddr; /* /< Offset of SS hover Force baseline
+ * frame */
+
+ u16 u16_ssHvrRxRawAddr; /* /< Offset of SS hover Sense raw frame */
+ u16 u16_ssHvrRxFilterAddr; /* /< Offset of SS hover Sense filter
+ * frame */
+ u16 u16_ssHvrRxStrenAddr; /* /< Offset of SS hover Sense strength
+ * frame */
+ u16 u16_ssHvrRxBaselineAddr; /* /< Offset of SS hover Sense baseline
+ * frame */
+
+ u16 u16_ssPrxTxRawAddr; /* /< Offset of SS proximity force raw frame */
+ u16 u16_ssPrxTxFilterAddr; /* /< Offset of SS proximity force
+ * filter frame */
+ u16 u16_ssPrxTxStrenAddr;/* /< Offset of SS proximity force strength
+ * frame */
+ u16 u16_ssPrxTxBaselineAddr; /* /< Offset of SS proximity force
+ * baseline frame */
+
+ u16 u16_ssPrxRxRawAddr; /* /< Offset of SS proximity sense raw frame */
+ u16 u16_ssPrxRxFilterAddr; /* /< Offset of SS proximity sense
+ * filter frame */
+ u16 u16_ssPrxRxStrenAddr;/* /< Offset of SS proximity sense strength
+ * frame */
+ u16 u16_ssPrxRxBaselineAddr; /* /< Offset of SS proximity sense
+ * baseline frame */
+} SysInfo;
+
+/** @}*/
+
+int initCore(struct fts_ts_info *info);
+void setResetGpio(int gpio);
+int fts_system_reset(void);
+int isSystemResettedUp(void);
+int isSystemResettedDown(void);
+void setSystemResetedUp(int val);
+void setSystemResetedDown(int val);
+int pollForEvent(int *event_to_search, int event_bytes, u8 *readData, int
+ time_to_wait);
+int checkEcho(u8 *cmd, int size);
+int setScanMode(u8 mode, u8 settings);
+int setFeatures(u8 feat, u8 *settings, int size);
+int defaultSysInfo(int i2cError);
+int writeSysCmd(u8 sys_cmd, u8 *sett, int size);
+int readSysInfo(int request);
+int readConfig(u16 offset, u8 *outBuf, int len);
+int writeConfig(u16 offset, u8 *data, int len);
+int fts_disableInterrupt(void);
+int fts_disableInterruptNoSync(void);
+int fts_resetDisableIrqCount(void);
+int fts_enableInterrupt(void);
+int fts_crc_check(void);
+int requestSyncFrame(u8 type);
+int setActiveScanFrequency(u32 freq);
+int writeHostDataMemory(u8 type, u8 *data, u8 msForceLen, u8 msSenseLen,
+ u8 ssForceLen, u8 ssSenseLen, int save);
+#endif /* FTS_CORE_H */
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsError.c b/drivers/input/touchscreen/stm/fts_lib/ftsError.c
new file mode 100644
index 0000000..2ae4337
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsError.c
@@ -0,0 +1,349 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS error/info kernel log reporting *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsError.c
+ * \brief Contains all the function which handle with Error conditions
+ */
+
+#include <linux/device.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/delay.h>
+
+
+#include "../fts.h"
+#include "ftsCore.h"
+#include "ftsError.h"
+#include "ftsIO.h"
+#include "ftsTool.h"
+#include "ftsCompensation.h"
+
+
+static ErrorList errors;/* /< private variable which implement the Error List */
+
+/**
+ * Check if an error code is related to an I2C failure
+ * @param error error code to check
+ * @return 1 if the first level error code is I2C related otherwise 0
+ */
+int isI2cError(int error)
+{
+ if (((error & 0x000000FF) >= (ERROR_BUS_R & 0x000000FF)) &&
+ ((error & 0x000000FF) <= (ERROR_BUS_O & 0x000000FF)))
+ return 1;
+ else
+ return 0;
+}
+
+
+/**
+ * Dump in the kernel log some debug info in case of FW hang
+ * @param outBuf (optional)pointer to bytes array where to copy the debug info,
+ * if NULL the data will just printed on the kernel log
+ * @param size dimension in bytes of outBuf,
+ * if > ERROR_DUMP_ROW_SIZE*ERROR_DUMP_COL_SIZE, only the first
+ * ERROR_DUMP_ROW_SIZE*ERROR_DUMP_COL_SIZE bytes will be copied
+ * @return OK if success or an error code which specify the type of error
+ */
+int dumpErrorInfo(u8 *outBuf, int size)
+{
+ int ret, i;
+ u8 data[ERROR_DUMP_ROW_SIZE * ERROR_DUMP_COL_SIZE] = { 0 };
+ u32 sign = 0;
+
+ pr_err("%s: Starting dump of error info...\n", __func__);
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, ADDR_ERROR_DUMP,
+ data, ERROR_DUMP_ROW_SIZE * ERROR_DUMP_COL_SIZE,
+ DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("%s: reading data ERROR %08X\n", __func__,
+ ret);
+ return ret;
+ } else {
+ int buff_len, index = 0;
+ char *buff;
+
+ buff_len = (2 + 1) * ERROR_DUMP_COL_SIZE + 1;
+ buff = kzalloc(buff_len, GFP_KERNEL);
+ if (buff == NULL) {
+ pr_err("%s: fail to allocate buffer\n", __func__);
+ return -ENOMEM;
+ }
+
+ if (outBuf != NULL) {
+ sign = size > ERROR_DUMP_ROW_SIZE *
+ ERROR_DUMP_COL_SIZE ? ERROR_DUMP_ROW_SIZE *
+ ERROR_DUMP_COL_SIZE : size;
+ memcpy(outBuf, data, sign);
+ pr_err("%s: error info copied in the buffer!\n",
+ __func__);
+ }
+ pr_err("%s: Error Info =\n", __func__);
+ u8ToU32(data, &sign);
+ if (sign != ERROR_DUMP_SIGNATURE)
+ pr_err("%s: Wrong Error Signature! Data may be invalid!\n",
+ __func__);
+ else
+ pr_err("%s: Error Signature OK! Data are valid!\n",
+ __func__);
+
+ for (i = 0; i < ERROR_DUMP_ROW_SIZE * ERROR_DUMP_COL_SIZE;
+ i++) {
+ index += scnprintf(buff + index, buff_len - index,
+ "%02X ", data[i]);
+ if (i % ERROR_DUMP_COL_SIZE ==
+ (ERROR_DUMP_COL_SIZE - 1)) {
+ pr_err("%s: %d) %s\n", __func__,
+ i / ERROR_DUMP_COL_SIZE,
+ buff);
+ index = 0;
+ }
+ }
+
+ kfree(buff);
+ pr_err("%s: dump of error info FINISHED!\n", __func__);
+ return OK;
+ }
+}
+
+
+/**
+ * Implement recovery strategies to be used when an error event is found
+ * while polling the FIFO
+ * @param event error event found during the polling
+ * @param size size of event
+ * @return OK if the error event doesn't require any action or the recovery
+ * strategy doesn't have any impact in the possible procedure that trigger the
+ * error,
+ * otherwise return an error code which specify the kind of error.
+ * If ERROR_HANDLER_STOP_PROC the calling function must stop!
+ */
+int errorHandler(u8 *event, int size)
+{
+ int res = OK;
+ struct fts_ts_info *info = NULL;
+
+ if (getDev() != NULL)
+ info = dev_get_drvdata(getDev());
+
+ if (info != NULL && event != NULL && size > 1 &&
+ event[0] == EVT_ID_ERROR) {
+ pr_debug("errorHandler: Starting handling...\n");
+ addErrorIntoList(event, size);
+ switch (event[1]) { /* TODO: write an error log for
+ * undefined command subtype 0xBA */
+ case EVT_TYPE_ERROR_ESD: /* esd */
+ res = fts_chip_powercycle(info);
+ if (res < OK)
+ pr_err("errorHandler: Error performing powercycle ERROR %08X\n",
+ res);
+
+ res = fts_system_reset();
+ if (res < OK)
+ pr_err("errorHandler: Cannot reset the device ERROR %08X\n",
+ res);
+ res = (ERROR_HANDLER_STOP_PROC | res);
+ break;
+
+ case EVT_TYPE_ERROR_WATCHDOG: /* watchdog */
+ dumpErrorInfo(NULL, 0);
+ res = fts_system_reset();
+ if (res < OK)
+ pr_err("errorHandler: Cannot reset the device ERROR %08X\n",
+ res);
+ res = (ERROR_HANDLER_STOP_PROC | res);
+ break;
+
+ case EVT_TYPE_ERROR_ITO_FORCETOGND:
+ pr_err("errorHandler: Force Short to GND!\n");
+ break;
+ case EVT_TYPE_ERROR_ITO_SENSETOGND:
+ pr_err("errorHandler: Sense short to GND!\n");
+ break;
+ case EVT_TYPE_ERROR_ITO_FORCETOVDD:
+ pr_err("errorHandler: Force short to VDD!\n");
+ break;
+ case EVT_TYPE_ERROR_ITO_SENSETOVDD:
+ pr_err("errorHandler: Sense short to VDD!\n");
+ break;
+ case EVT_TYPE_ERROR_ITO_FORCE_P2P:
+ pr_err("errorHandler: Force Pin to Pin Short!\n");
+ break;
+ case EVT_TYPE_ERROR_ITO_SENSE_P2P:
+ pr_err("errorHandler: Sense Pin to Pin Short!\n");
+ break;
+ case EVT_TYPE_ERROR_ITO_FORCEOPEN:
+ pr_err("errorHandler: Force Open !\n");
+ break;
+ case EVT_TYPE_ERROR_ITO_SENSEOPEN:
+ pr_err("errorHandler: Sense Open !\n");
+ break;
+ case EVT_TYPE_ERROR_ITO_KEYOPEN:
+ pr_err("errorHandler: Key Open !\n");
+ break;
+
+ case EVT_TYPE_ERROR_FLASH_FAILED:
+ pr_err("errorHandler: Previous flash failed!\n");
+ info->reflash_fw = 1;
+ break;
+
+ default:
+ pr_debug("errorHandler: No Action taken!\n");
+ break;
+ }
+ pr_debug("errorHandler: handling Finished! res = %08X\n",
+ res);
+ return res;
+ } else {
+ pr_err("errorHandler: event Null or not correct size! ERROR %08X\n",
+ ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+}
+
+
+/**
+ * Add an error event into the Error List
+ * @param event error event to add
+ * @param size size of event
+ * @return OK
+ */
+int addErrorIntoList(u8 *event, int size)
+{
+ int i = 0;
+
+ pr_debug("Adding error in to ErrorList...\n");
+
+ memcpy(&errors.list[errors.last_index * FIFO_EVENT_SIZE], event, size);
+ i = FIFO_EVENT_SIZE - size;
+ if (i > 0) {
+ pr_info("Filling last %d bytes of the event with zero...\n", i);
+ memset(&errors.list[errors.last_index * FIFO_EVENT_SIZE + size],
+ 0, i);
+ }
+ pr_debug("Adding error in to ErrorList... FINISHED!\n");
+
+ errors.count += 1;
+ if (errors.count > FIFO_DEPTH)
+ pr_err("ErrorList is going in overflow... the first %d event(s) were override!\n",
+ errors.count - FIFO_DEPTH);
+ errors.last_index = (errors.last_index + 1) % FIFO_DEPTH;
+
+ return OK;
+}
+
+/**
+ * Reset the Error List setting the count and last_index to 0.
+ * @return OK
+ */
+int resetErrorList(void)
+{
+ errors.count = 0;
+ errors.last_index = 0;
+ memset(errors.list, 0, FIFO_DEPTH * FIFO_EVENT_SIZE);
+ /* if count is not considered is better reset also the list in order to
+ * avoid to read data previously copied into the list */
+ return OK;
+}
+
+/**
+ * Get the number of error events copied into the Error List
+ * @return the number of error events into the Error List
+ */
+int getErrorListCount(void)
+{
+ if (errors.count > FIFO_DEPTH)
+ return FIFO_DEPTH;
+ else
+ return errors.count;
+}
+
+/* in case of success return the index of the event found */
+/**
+ * Scroll the Error List looking for the event specified
+ * @param event_to_search event_to_search pointer to an array of int where
+ * each element correspond to a byte of the event to find. If the element
+ * of the array has value -1, the byte of the event, in the same position
+ * of the element is ignored.
+ * @param event_bytes size of event_to_search
+ * @return a value >=0 if the event is found which represent the index of
+ * the Error List where the event is located otherwise an error code
+ */
+int pollErrorList(int *event_to_search, int event_bytes)
+{
+ int i = 0, j = 0, find = 0;
+ int count = getErrorListCount();
+
+ pr_debug("Starting to poll ErrorList...\n");
+ while (find != 1 && i < count) {
+ find = 1;
+ for (j = 0; j < event_bytes; j++) {
+ if ((event_to_search[i] != -1) &&
+ ((int)errors.list[i * FIFO_EVENT_SIZE + j] !=
+ event_to_search[i])) {
+ find = 0;
+ break;
+ }
+ }
+ i++;
+ }
+ if (find == 1) {
+ pr_debug("Error Found into ErrorList!\n");
+ return i - 1; /* there is i++ at the end of the while */
+ } else {
+ pr_err("Error Not Found into ErrorList! ERROR %08X\n",
+ ERROR_TIMEOUT);
+ return ERROR_TIMEOUT;
+ }
+}
+
+
+
+/**
+ * Poll the Error List looking for any error types passed in the arguments.
+ * Return at the first match!
+ * @param list pointer to a list of error types to look for
+ * @param size size of list
+ * @return error type found if success or ERROR_TIMEOUT
+ */
+int pollForErrorType(u8 *list, int size)
+{
+ int i = 0, j = 0, find = 0;
+ int count = getErrorListCount();
+
+ pr_info("%s: Starting to poll ErrorList... count = %d\n",
+ __func__, count);
+ while (find != 1 && i < count) {
+ for (j = 0; j < size; j++) {
+ if (list[j] == errors.list[i * FIFO_EVENT_SIZE + 1]) {
+ find = 1;
+ break;
+ }
+ }
+ i++;
+ }
+ if (find == 1) {
+ pr_info("%s: Error Type %02X into ErrorList!\n",
+ __func__, list[j]);
+ return list[j];
+ } else {
+ pr_err("%s: Error Type Not Found into ErrorList! ERROR %08X\n",
+ __func__, ERROR_TIMEOUT);
+ return ERROR_TIMEOUT;
+ }
+}
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsError.h b/drivers/input/touchscreen/stm/fts_lib/ftsError.h
new file mode 100644
index 0000000..f12d9ee
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsError.h
@@ -0,0 +1,231 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS error/info kernel log reporting *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsError.h
+ * \brief Contains all the definitions and structs which refer to Error
+ * conditions
+ */
+
+#ifndef FTS_ERROR_H
+#define FTS_ERROR_H
+
+#include "ftsHardware.h"
+#include "ftsSoftware.h"
+
+
+/** @defgroup error_codes Error Codes
+ * Error codes that can be reported by the driver functions.
+ * An error code is made up by 4 bytes, each byte indicate a logic error
+ * level.\n
+ * From the LSB to the MSB, the logic level increase going from a low level
+ * error (I2C,TIMEOUT) to an high level error (flashing procedure fail,
+ * production test fail etc)
+ * @{
+ */
+
+/* FIRST LEVEL ERROR CODE */
+/** @defgroup first_level First Level Error Code
+ * @ingroup error_codes
+ * Errors related to low level operation which are not under control of driver,
+ * such as: communication protocol (I2C/SPI), timeout, file operations ...
+ * @{
+ */
+#define OK (0x00000000) /* /< No ERROR */
+#define ERROR_ALLOC (0x80000001) /* /< allocation of memory
+ * failed */
+#define ERROR_BUS_R (0x80000002) /* /< i2c/spi read failed */
+#define ERROR_BUS_W (0x80000003) /* /< i2c/spi write failed */
+#define ERROR_BUS_WR (0x80000004) /* /< i2c/spi write/read
+ * failed */
+#define ERROR_BUS_O (0x80000005) /* /< error during
+ * opening an i2c device */
+#define ERROR_OP_NOT_ALLOW (0x80000006) /* /< operation not allowed */
+#define ERROR_TIMEOUT (0x80000007) /* /< timeout expired!
+ * exceed the max number
+ * of retries or the max
+ * waiting time */
+#define ERROR_FILE_NOT_FOUND (0x80000008) /* /< the file that i
+ * want to open is not found */
+#define ERROR_FILE_PARSE (0x80000009) /* /< error during parsing
+ * the file */
+#define ERROR_FILE_READ (0x8000000A) /* /< error during
+ * reading the file */
+#define ERROR_LABEL_NOT_FOUND (0x8000000B) /* /< label not found */
+#define ERROR_FW_NO_UPDATE (0x8000000C) /* /< fw in the chip
+ * newer than the one in
+ * the memmh */
+#define ERROR_FLASH_UNKNOWN (0x8000000D) /* /< flash status busy
+ * or unknown */
+/** @}*/
+
+/* SECOND LEVEL ERROR CODE */
+/** @defgroup second_level Second Level Error Code
+ * @ingroup error_codes
+ * Errors related to simple logic operations in the IC which require one
+ * command or which are part of a more complex procedure
+ * @{
+ */
+#define ERROR_DISABLE_INTER (0x80000200) /* /< unable to
+ * disable the
+ * interrupt */
+#define ERROR_ENABLE_INTER (0x80000300) /* /< unable to activate
+ * the interrupt */
+#define ERROR_READ_CONFIG (0x80000400) /* /< failed to read
+ * config memory */
+#define ERROR_GET_OFFSET (0x80000500) /* /< unable to
+ * read an offset from
+ * memory */
+#define ERROR_GET_FRAME_DATA (0x80000600) /* /< unable to
+ * retrieve the data of
+ * a required frame */
+#define ERROR_DIFF_DATA_TYPE (0x80000700) /* /< FW answers
+ * with an event that
+ * has a different
+ * address respect the
+ * request done */
+#define ERROR_WRONG_DATA_SIGN (0x80000800) /* /< the signature of
+ * the host data is not
+ * HEADER_SIGNATURE */
+#define ERROR_SET_SCAN_MODE_FAIL (0x80000900) /* /< setting
+ * the scanning mode
+ * failed
+ * (sense on/off
+ * etc...) */
+#define ERROR_SET_FEATURE_FAIL (0x80000A00) /* /< setting a specific
+ * feature failed */
+#define ERROR_SYSTEM_RESET_FAIL (0x80000B00) /* /< the command
+ * SYSTEM RESET
+ * failed */
+#define ERROR_FLASH_NOT_READY (0x80000C00) /* /< flash status not
+ * ready within a
+ * timeout */
+#define ERROR_FW_VER_READ (0x80000D00) /* /< unable to retrieve
+ * fw_vers or the
+ * config_id */
+#define ERROR_GESTURE_ENABLE_FAIL (0x80000E00) /* /< unable to
+ * enable/disable
+ * the gesture */
+#define ERROR_GESTURE_START_ADD (0x80000F00) /* /< unable to start to
+ * add custom gesture */
+#define ERROR_GESTURE_FINISH_ADD (0x80001000) /* /< unable to finish
+ * to add custom gesture
+ */
+#define ERROR_GESTURE_DATA_ADD (0x80001100) /* /< unable to add
+ * custom gesture data
+ * */
+#define ERROR_GESTURE_REMOVE (0x80001200) /* /< unable to remove
+ * custom gesture data
+ * */
+#define ERROR_FEATURE_ENABLE_DISABLE (0x80001300) /* /< unable to
+ * enable/disable a
+ * feature mode in
+ * the IC */
+#define ERROR_NOISE_PARAMETERS (0x80001400) /* /< unable to set/read
+ * noise parameter in
+ * the IC */
+#define ERROR_CH_LEN (0x80001500) /* /< unable to retrieve
+ * the force and/or
+ * sense length */
+/** @}*/
+
+/* THIRD LEVEL ERROR CODE */
+/** @defgroup third_level Third Level Error Code
+ * @ingroup error_codes
+ * Errors related to logic operations in the IC which require more
+ * commands/steps or which are part of a more complex procedure
+ * @{
+ */
+#define ERROR_REQU_COMP_DATA (0x80010000) /* /< compensation data
+ * request failed */
+#define ERROR_REQU_DATA (0x80020000) /* /< data request
+ * failed */
+#define ERROR_COMP_DATA_HEADER (0x80030000) /* /< unable to retrieve
+ * the compensation data
+ * header */
+#define ERROR_COMP_DATA_GLOBAL (0x80040000) /* /< unable to retrieve
+ * the global
+ * compensation data */
+#define ERROR_COMP_DATA_NODE (0x80050000) /* /< unable to retrieve
+ * the compensation data
+ * for each node */
+#define ERROR_TEST_CHECK_FAIL (0x80060000) /* /< check of
+ * production limits or
+ * of fw answers failed */
+#define ERROR_MEMH_READ (0x80070000) /* /< memh reading
+ * failed */
+#define ERROR_FLASH_BURN_FAILED (0x80080000) /* /< flash burn failed */
+#define ERROR_MS_TUNING (0x80090000) /* /< ms tuning failed */
+#define ERROR_SS_TUNING (0x800A0000) /* /< ss tuning failed */
+#define ERROR_LP_TIMER_TUNING (0x800B0000) /* /< lp timer
+ * calibration failed */
+#define ERROR_SAVE_CX_TUNING (0x800C0000) /* /< save cx data to
+ * flash failed */
+#define ERROR_HANDLER_STOP_PROC (0x800D0000) /* /< stop the poll of the FIFO
+ * if particular errors are
+ * found */
+#define ERROR_CHECK_ECHO_FAIL (0x800E0000) /* /< unable to retrieve
+ * echo event */
+#define ERROR_GET_FRAME (0x800F0000) /* /< unable to get frame */
+/** @}*/
+
+/* FOURTH LEVEL ERROR CODE */
+/** @defgroup fourth_level Fourth Level Error Code
+ * @ingroup error_codes
+ * Errors related to the highest logic operations in the IC which have an
+ * important impact on the driver flow or which require several commands and
+ * steps to be executed
+ * @{
+ */
+#define ERROR_PROD_TEST_DATA (0x81000000) /* /< production data
+ * test failed */
+#define ERROR_FLASH_PROCEDURE (0x82000000) /* /< fw update
+ * procedure failed */
+#define ERROR_PROD_TEST_ITO (0x83000000) /* /< production
+ * ito test failed */
+#define ERROR_PROD_TEST_INITIALIZATION (0x84000000) /* /< production
+ * initialization
+ * test failed */
+#define ERROR_GET_INIT_STATUS (0x85000000) /* /< mismatch of the MS or
+ * SS tuning_version */
+/** @}*/
+
+/** @}*/ /* end of error_commands section */
+
+/**
+ * Struct which store an ordered list of the errors events encountered during
+ *the polling of a FIFO.
+ * The max number of error events that can be stored is equal to FIFO_DEPTH
+ */
+typedef struct {
+ u8 list[FIFO_DEPTH * FIFO_EVENT_SIZE]; /* /< byte array which contains
+ * the series of error events
+ * encountered from the last
+ * reset of the list. */
+ int count; /* /< number of error events stored in the list */
+ int last_index; /* /< index of the list where will be stored the next
+ * error event. Subtract -1 to have the index of the
+ * last error event! */
+} ErrorList;
+
+int isI2cError(int error);
+int dumpErrorInfo(u8 *outBuf, int size);
+int errorHandler(u8 *event, int size);
+int addErrorIntoList(u8 *event, int size);
+int getErrorListCount(void);
+int resetErrorList(void);
+int pollErrorList(int *event_to_search, int event_bytes);
+int pollForErrorType(u8 *list, int size);
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsFlash.c b/drivers/input/touchscreen/stm/fts_lib/ftsFlash.c
new file mode 100644
index 0000000..7245889
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsFlash.c
@@ -0,0 +1,991 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS API for Flashing the IC *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+
+/*!
+ * \file ftsFlash.c
+ * \brief Contains all the functions to handle the FW update process
+ */
+
+#include "ftsCore.h"
+#include "ftsCompensation.h"
+#include "ftsError.h"
+#include "ftsFlash.h"
+#include "ftsFrame.h"
+#include "ftsIO.h"
+#include "ftsSoftware.h"
+#include "ftsTest.h"
+#include "ftsTime.h"
+#include "ftsTool.h"
+#include "../fts.h" /* needed for including the define FW_H_FILE */
+
+
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <stdarg.h>
+#include <linux/serio.h>
+#include <linux/time.h>
+#include <linux/delay.h>
+#include <linux/ctype.h>
+#include <linux/fs.h>
+#include <linux/uaccess.h>
+#include <linux/firmware.h>
+
+
+#ifdef FW_H_FILE
+#include "../fts_fw.h"
+#endif
+
+
+extern SysInfo systemInfo; /* /< forward declaration of the global variable
+ * of containing System Info Data */
+
+
+/**
+ * Retrieve the actual FW data from the system (bin file or header file)
+ * @param pathToFile name of FW file to load or "NULL" if the FW data should be
+ * loaded by a .h file
+ * @param data pointer to the pointer which will contains the FW data
+ * @param size pointer to a variable which will contain the size of the loaded
+ * data
+ * @return OK if success or an error code which specify the type of error
+ */
+int getFWdata(const char *pathToFile, u8 **data, int *size)
+{
+ const struct firmware *fw = NULL;
+ struct device *dev = getDev();
+ struct fts_ts_info *info = NULL;
+ int res, from = 0;
+ char *path = (char *)pathToFile;
+
+ if (dev != NULL)
+ info = dev_get_drvdata(dev);
+
+ pr_info("getFWdata starting ...\n");
+ if (strncmp(pathToFile, "NULL", 4) == 0) {
+ from = 1;
+ if (info != NULL && info->board->fw_name)
+ path = (char *)info->board->fw_name;
+ else
+ path = PATH_FILE_FW;
+ }
+ /* keep the switch case because if the argument passed is null but
+ * the option from .h is not set we still try to load from bin */
+ switch (from) {
+#ifdef FW_H_FILE
+ case 1:
+ pr_info("Read FW from .h file!\n");
+ *size = FW_SIZE_NAME;
+ *data = (u8 *)kmalloc((*size) * sizeof(u8), GFP_KERNEL);
+ if (*data == NULL) {
+ pr_err("getFWdata: Impossible to allocate memory! ERROR %08X\n",
+ ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+ memcpy(*data, (u8 *)FW_ARRAY_NAME, (*size));
+
+ break;
+#endif
+ default:
+ pr_info("Read FW from BIN file %s !\n", path);
+
+ if (dev != NULL) {
+ res = request_firmware(&fw, path, dev);
+ if (res == 0) {
+ *size = fw->size;
+ *data = (u8 *)kmalloc((*size) * sizeof(u8),
+ GFP_KERNEL);
+ if (*data == NULL) {
+ pr_err("getFWdata: Impossible to allocate memory! ERROR %08X\n",
+ ERROR_ALLOC);
+ release_firmware(fw);
+ return ERROR_ALLOC;
+ }
+ memcpy(*data, (u8 *)fw->data, (*size));
+ release_firmware(fw);
+ } else {
+ pr_err("getFWdata: No File found! ERROR %08X\n",
+ ERROR_FILE_NOT_FOUND);
+ return ERROR_FILE_NOT_FOUND;
+ }
+ } else {
+ pr_err("getFWdata: No device found! ERROR %08X\n",
+ ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+ }
+
+ pr_info("getFWdata Finished!\n");
+ return OK;
+}
+
+
+/**
+ * Perform all the steps to read the FW that should be burnt in the IC from
+ * the system and parse it in order to fill a Firmware struct with the relevant
+ * info
+ * @param path name of FW file to load or "NULL" if the FW data should be
+ * loaded by a .h file
+ * @param fw pointer to a Firmware variable which will contains the FW data and
+ * info
+ * @param keep_cx if 1, the CX area will be loaded otherwise will be skipped
+ * @return OK if success or an error code which specify the type of error
+ */
+int readFwFile(const char *path, Firmware *fw, int keep_cx)
+{
+ int res;
+ int orig_size;
+ u8 *orig_data = NULL;
+
+
+ res = getFWdata(path, &orig_data, &orig_size);
+ if (res < OK) {
+ pr_err("readFwFile: impossible retrieve FW... ERROR %08X\n",
+ ERROR_MEMH_READ);
+ return res | ERROR_MEMH_READ;
+ }
+ res = parseBinFile(orig_data, orig_size, fw, keep_cx);
+ if (res < OK) {
+ pr_err("readFwFile: impossible parse ERROR %08X\n",
+ ERROR_MEMH_READ);
+ return res | ERROR_MEMH_READ;
+ }
+
+ return OK;
+}
+
+/**
+ * Perform all the steps necessary to burn the FW into the IC
+ * @param path name of FW file to load or "NULL" if the FW data should be
+ * loaded by a .h file
+ * @param force if 1, the flashing procedure will be forced and executed
+ * regardless the additional info, otherwise the FW in the file will be burnt
+ * only if it is newer than the one running in the IC
+ * @param keep_cx if 1, the CX area will be loaded and burnt otherwise will be
+ * skipped and the area will be untouched
+ * @return OK if success or an error code which specify the type of error
+ */
+int flashProcedure(const char *path, int force, int keep_cx)
+{
+ Firmware fw;
+ int res;
+
+ fw.data = NULL;
+ pr_info("Reading Fw file...\n");
+ res = readFwFile(path, &fw, keep_cx);
+ if (res < OK) {
+ pr_err("flashProcedure: ERROR %08X\n",
+ (res | ERROR_FLASH_PROCEDURE));
+ kfree(fw.data);
+ return res | ERROR_FLASH_PROCEDURE;
+ }
+ pr_info("Fw file read COMPLETED!\n");
+
+ pr_info("Starting flashing procedure...\n");
+ res = flash_burn(fw, force, keep_cx);
+ if (res < OK && res != (ERROR_FW_NO_UPDATE | ERROR_FLASH_BURN_FAILED)) {
+ pr_err("flashProcedure: ERROR %08X\n",
+ ERROR_FLASH_PROCEDURE);
+ kfree(fw.data);
+ return res | ERROR_FLASH_PROCEDURE;
+ }
+ pr_info("flashing procedure Finished!\n");
+ kfree(fw.data);
+
+ return res;
+}
+
+/**
+ * Poll the Flash Status Registers after the execution of a command to check
+ * if the Flash becomes ready within a timeout
+ * @param type register to check according to the previous command sent
+ * @return OK if success or an error code which specify the type of error
+ */
+int wait_for_flash_ready(u8 type)
+{
+ u8 cmd[5] = { FTS_CMD_HW_REG_R, 0x20, 0x00, 0x00, type };
+
+ u8 readData[2] = { 0 };
+ int i, res = -1;
+
+ pr_info("Waiting for flash ready ...\n");
+ for (i = 0; i < FLASH_RETRY_COUNT && res != 0; i++) {
+ res = fts_writeRead(cmd, ARRAY_SIZE(cmd), readData, 2);
+ if (res < OK)
+ pr_err("wait_for_flash_ready: ERROR %08X\n",
+ ERROR_BUS_W);
+ else {
+#ifdef I2C_INTERFACE /* in case of spi there is a dummy byte */
+ res = readData[0] & 0x80;
+#else
+ res = readData[1] & 0x80;
+#endif
+
+ pr_info("flash status = %d\n", res);
+ }
+ mdelay(FLASH_WAIT_BEFORE_RETRY);
+ }
+
+ if (i == FLASH_RETRY_COUNT && res != 0) {
+ pr_err("Wait for flash TIMEOUT! ERROR %08X\n",
+ ERROR_TIMEOUT);
+ return ERROR_TIMEOUT;
+ }
+
+ pr_info("Flash READY!\n");
+ return OK;
+}
+
+
+/**
+ * Put the M3 in hold
+ * @return OK if success or an error code which specify the type of error
+ */
+int hold_m3(void)
+{
+ int ret;
+ u8 cmd[1] = { 0x01 };
+
+ pr_info("Command m3 hold...\n");
+ ret = fts_writeU8UX(FTS_CMD_HW_REG_W, ADDR_SIZE_HW_REG,
+ ADDR_SYSTEM_RESET, cmd, 1);
+ if (ret < OK) {
+ pr_err("hold_m3: ERROR %08X\n", ret);
+ return ret;
+ }
+ pr_info("Hold M3 DONE!\n");
+
+#if !defined(I2C_INTERFACE) && defined(SPI4_WIRE)
+ /* configure manually SPI4 because when no fw is running the chip use
+ * SPI3 by default */
+ pr_info("Setting SPI4 mode...\n");
+ cmd[0] = 0x10;
+ ret = fts_writeU8UX(FTS_CMD_HW_REG_W, ADDR_SIZE_HW_REG,
+ ADDR_GPIO_DIRECTION, cmd, 1);
+ if (ret < OK) {
+ pr_err("hold_m3: can not set gpio dir ERROR %08X\n", ret);
+ return ret;
+ }
+
+ cmd[0] = 0x02;
+ ret = fts_writeU8UX(FTS_CMD_HW_REG_W, ADDR_SIZE_HW_REG,
+ ADDR_GPIO_PULLUP, cmd, 1);
+ if (ret < OK) {
+ pr_err("hold_m3: can not set gpio pull-up ERROR %08X\n", ret);
+ return ret;
+ }
+
+ cmd[0] = 0x07;
+ ret = fts_writeU8UX(FTS_CMD_HW_REG_W, ADDR_SIZE_HW_REG,
+ ADDR_GPIO_CONFIG_REG2, cmd, 1);
+ if (ret < OK) {
+ pr_err("hold_m3: can not set gpio config ERROR %08X\n", ret);
+ return ret;
+ }
+
+ cmd[0] = 0x30;
+ ret = fts_writeU8UX(FTS_CMD_HW_REG_W, ADDR_SIZE_HW_REG,
+ ADDR_GPIO_CONFIG_REG0, cmd, 1);
+ if (ret < OK) {
+ pr_err("hold_m3: can not set gpio config ERROR %08X\n", ret);
+ return ret;
+ }
+
+ cmd[0] = SPI4_MASK;
+ ret = fts_writeU8UX(FTS_CMD_HW_REG_W, ADDR_SIZE_HW_REG, ADDR_ICR, cmd,
+ 1);
+ if (ret < OK) {
+ pr_err("hold_m3: can not set spi4 mode ERROR %08X\n", ret);
+ return ret;
+ }
+ mdelay(1); /* wait for the GPIO to stabilize */
+#endif
+
+ return OK;
+}
+
+
+
+
+/**
+ * Parse the raw data read from a FW file in order to fill properly the fields
+ * of a Firmware variable
+ * @param fw_data raw FW data loaded from system
+ * @param fw_size size of fw_data
+ * @param fwData pointer to a Firmware variable which will contain the
+ * processed data
+ * @param keep_cx if 1, the CX area will be loaded and burnt otherwise will be
+ * skipped and the area will be untouched
+ * @return OK if success or an error code which specify the type of error
+ */
+int parseBinFile(u8 *fw_data, int fw_size, Firmware *fwData, int keep_cx)
+{
+ int dimension, index = 0;
+ u32 temp;
+ int res, i;
+ char buff[(2 + 1) * EXTERNAL_RELEASE_INFO_SIZE + 1];
+ int buff_len = sizeof(buff);
+ int buff_index = 0;
+
+ /* the file should contain at least the header plus the content_crc */
+ if (fw_size < FW_HEADER_SIZE + FW_BYTES_ALIGN || fw_data == NULL) {
+ pr_err("parseBinFile: Read only %d instead of %d... ERROR %08X\n",
+ fw_size, FW_HEADER_SIZE + FW_BYTES_ALIGN,
+ ERROR_FILE_PARSE);
+ res = ERROR_FILE_PARSE;
+ goto END;
+ } else {
+ /* start parsing of bytes */
+ u8ToU32(&fw_data[index], &temp);
+ if (temp != FW_HEADER_SIGNATURE) {
+ pr_err("parseBinFile: Wrong Signature %08X ... ERROR %08X\n",
+ temp, ERROR_FILE_PARSE);
+ res = ERROR_FILE_PARSE;
+ goto END;
+ }
+ pr_info("parseBinFile: Fw Signature OK!\n");
+ index += FW_BYTES_ALIGN;
+ u8ToU32(&fw_data[index], &temp);
+ if (temp != FW_FTB_VER) {
+ pr_err("parseBinFile: Wrong ftb_version %08X ... ERROR %08X\n",
+ temp, ERROR_FILE_PARSE);
+ res = ERROR_FILE_PARSE;
+ goto END;
+ }
+ pr_info("parseBinFile: ftb_version OK!\n");
+ index += FW_BYTES_ALIGN;
+ if (fw_data[index] != DCHIP_ID_0 || fw_data[index + 1] !=
+ DCHIP_ID_1) {
+ pr_err("parseBinFile: Wrong target %02X != %02X %02X != %02X ... ERROR %08X\n",
+ fw_data[index], DCHIP_ID_0,
+ fw_data[index + 1],
+ DCHIP_ID_1, ERROR_FILE_PARSE);
+ res = ERROR_FILE_PARSE;
+ goto END;
+ }
+ index += FW_BYTES_ALIGN;
+ u8ToU32(&fw_data[index], &temp);
+ pr_info("parseBinFile: FILE SVN REV = %08X\n", temp);
+
+ index += FW_BYTES_ALIGN;
+ u8ToU32(&fw_data[index], &temp);
+ fwData->fw_ver = temp;
+ pr_info("parseBinFile: FILE Fw Version = %04X\n",
+ fwData->fw_ver);
+
+ index += FW_BYTES_ALIGN;
+ u8ToU32(&fw_data[index], &temp);
+ pr_info("parseBinFile: FILE Config Project ID = %08X\n", temp);
+
+ index += FW_BYTES_ALIGN;
+ u8ToU32(&fw_data[index], &temp);
+ fwData->config_ver = temp;
+ pr_info("parseBinFile: FILE Config Version = %08X\n",
+ fwData->config_ver);
+
+ index += FW_BYTES_ALIGN * 2; /* skip reserved data */
+
+ index += FW_BYTES_ALIGN;
+ for (i = 0; i < EXTERNAL_RELEASE_INFO_SIZE; i++) {
+ fwData->externalRelease[i] = fw_data[index++];
+ buff_index += scnprintf(buff + buff_index,
+ buff_len - buff_index,
+ "%02X ",
+ fwData->externalRelease[i]);
+ }
+ pr_info("parseBinFile: File External Release = %s\n", buff);
+
+ /* index+=FW_BYTES_ALIGN; */
+ u8ToU32(&fw_data[index], &temp);
+ fwData->sec0_size = temp;
+ pr_info("parseBinFile: sec0_size = %08X (%d bytes)\n",
+ fwData->sec0_size, fwData->sec0_size);
+
+ index += FW_BYTES_ALIGN;
+ u8ToU32(&fw_data[index], &temp);
+ fwData->sec1_size = temp;
+ pr_info("parseBinFile: sec1_size = %08X (%d bytes)\n",
+ fwData->sec1_size, fwData->sec1_size);
+
+ index += FW_BYTES_ALIGN;
+ u8ToU32(&fw_data[index], &temp);
+ fwData->sec2_size = temp;
+ pr_info("parseBinFile: sec2_size = %08X (%d bytes)\n",
+ fwData->sec2_size, fwData->sec2_size);
+
+ index += FW_BYTES_ALIGN;
+ u8ToU32(&fw_data[index], &temp);
+ fwData->sec3_size = temp;
+ pr_info("parseBinFile: sec3_size = %08X (%d bytes)\n",
+ fwData->sec3_size, fwData->sec3_size);
+
+ index += FW_BYTES_ALIGN;/* skip header crc */
+
+ /* if (!keep_cx) */
+ /* { */
+ dimension = fwData->sec0_size + fwData->sec1_size +
+ fwData->sec2_size + fwData->sec3_size;
+ temp = fw_size;
+ /*} else
+ * {
+ * dimension = fwData->sec0_size + fwData->sec1_size;
+ * temp = fw_size - fwData->sec2_size - fwData->sec3_size;
+ * fwData->sec2_size = 0;
+ * fwData->sec3_size = 0;
+ * }*/
+
+ if (dimension + FW_HEADER_SIZE + FW_BYTES_ALIGN != temp) {
+ pr_err("parseBinFile: Read only %d instead of %d... ERROR %08X\n",
+ fw_size, dimension + FW_HEADER_SIZE +
+ FW_BYTES_ALIGN, ERROR_FILE_PARSE);
+ res = ERROR_FILE_PARSE;
+ goto END;
+ }
+
+ fwData->data = (u8 *)kmalloc(dimension * sizeof(u8),
+ GFP_KERNEL);
+ if (fwData->data == NULL) {
+ pr_err("parseBinFile: ERROR %08X\n", ERROR_ALLOC);
+ res = ERROR_ALLOC;
+ goto END;
+ }
+
+ index += FW_BYTES_ALIGN;
+ memcpy(fwData->data, &fw_data[index], dimension);
+ if (fwData->sec2_size != 0)
+ u8ToU16(&fwData->data[fwData->sec0_size +
+ fwData->sec1_size +
+ FW_CX_VERSION], &fwData->cx_ver);
+
+ else {
+ pr_err("parseBinFile: Initialize cx_ver to default value!\n");
+ fwData->cx_ver = systemInfo.u16_cxVer;
+ }
+
+ pr_info("parseBinFile: CX Version = %04X\n", fwData->cx_ver);
+
+ fwData->data_size = dimension;
+
+ pr_info("READ FW DONE %d bytes!\n", fwData->data_size);
+ res = OK;
+ goto END;
+ }
+
+END:
+ kfree(fw_data);
+ return res;
+}
+
+/**
+ * Unlock the flash to be programmed
+ * @return OK if success or an error code which specify the type of error
+ */
+int flash_unlock(void)
+{
+ u8 cmd[6] = { FTS_CMD_HW_REG_W, 0x20, 0x00, 0x00,
+ FLASH_UNLOCK_CODE0, FLASH_UNLOCK_CODE1 };
+
+ pr_info("Command unlock ...\n");
+ if (fts_write(cmd, ARRAY_SIZE(cmd)) < OK) {
+ pr_err("flash_unlock: ERROR %08X\n", ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ pr_info("Unlock flash DONE!\n");
+
+ return OK;
+}
+
+/**
+ * Unlock the flash to be erased
+ * @return OK if success or an error code which specify the type of error
+ */
+int flash_erase_unlock(void)
+{
+ u8 cmd[6] = { FTS_CMD_HW_REG_W, 0x20, 0x00,
+ 0x00,
+ FLASH_ERASE_UNLOCK_CODE0, FLASH_ERASE_UNLOCK_CODE1 };
+
+ pr_info("Try to erase unlock flash...\n");
+
+ pr_info("Command erase unlock ...\n");
+ if (fts_write(cmd, ARRAY_SIZE(cmd)) < 0) {
+ pr_err("flash_erase_unlock: ERROR %08X\n", ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ pr_info("Erase Unlock flash DONE!\n");
+
+ return OK;
+}
+
+/**
+ * Erase the full flash
+ * @return OK if success or an error code which specify the type of error
+ */
+int flash_full_erase(void)
+{
+ int status;
+
+ u8 cmd1[6] = { FTS_CMD_HW_REG_W, 0x20, 0x00, 0x00,
+ FLASH_ERASE_CODE0 + 1, 0x00 };
+ u8 cmd[6] = { FTS_CMD_HW_REG_W, 0x20, 0x00, 0x00,
+ FLASH_ERASE_CODE0, FLASH_ERASE_CODE1 };
+
+ if (fts_write(cmd1, ARRAY_SIZE(cmd1)) < OK) {
+ pr_err("flash_erase_page_by_page: ERROR %08X\n", ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ pr_info("Command full erase sent ...\n");
+ if (fts_write(cmd, ARRAY_SIZE(cmd)) < OK) {
+ pr_err("flash_full_erase: ERROR %08X\n", ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ status = wait_for_flash_ready(FLASH_ERASE_CODE0);
+
+ if (status != OK) {
+ pr_err("flash_full_erase: ERROR %08X\n", ERROR_FLASH_NOT_READY);
+ return status | ERROR_FLASH_NOT_READY;
+ /* Flash not ready within the chosen time, better exit! */
+ }
+
+ pr_info("Full Erase flash DONE!\n");
+
+ return OK;
+}
+
+/**
+ * Erase the flash page by page, giving the possibility to skip the CX area and
+ * maintain therefore its value
+ * @param keep_cx if SKIP_PANEL_INIT the Panel Init pages will be skipped,
+ * if > SKIP_PANEL_CX_INIT Cx and Panel Init pages otherwise all the pages will
+ * be deleted
+ * @return OK if success or an error code which specify the type of error
+ */
+int flash_erase_page_by_page(ErasePage keep_cx)
+{
+ u8 status, i = 0;
+ u8 cmd1[6] = { FTS_CMD_HW_REG_W, 0x20, 0x00, 0x00,
+ FLASH_ERASE_CODE0 + 1, 0x00 };
+ u8 cmd[6] = { FTS_CMD_HW_REG_W, 0x20, 0x00, 0x00, FLASH_ERASE_CODE0,
+ 0xA0 };
+ u8 cmd2[9] = { FTS_CMD_HW_REG_W, 0x20, 0x00,
+ 0x01, 0x28,
+ 0xFF,
+ 0xFF, 0xFF, 0xFF };
+ u8 mask[4] = { 0 };
+ char buff[(2 + 1) * 4 + 1];
+ int buff_len = sizeof(buff);
+ int index = 0;
+
+ for (i = FLASH_CX_PAGE_START; i <= FLASH_CX_PAGE_END && keep_cx >=
+ SKIP_PANEL_CX_INIT; i++) {
+ pr_info("Skipping erase CX page %d!\n", i);
+ fromIDtoMask(i, mask, 4);
+ }
+
+
+ for (i = FLASH_PANEL_PAGE_START; i <= FLASH_PANEL_PAGE_END && keep_cx >=
+ SKIP_PANEL_INIT; i++) {
+ pr_info("Skipping erase Panel Init page %d!\n", i);
+ fromIDtoMask(i, mask, 4);
+ }
+
+ for (i = 0; i < 4; i++) {
+ cmd2[5 + i] = cmd2[5 + i] & (~mask[i]);
+ index += scnprintf(buff + index, buff_len - index,
+ "%02X ", cmd2[5 + i]);
+ }
+ pr_info("Setting the page mask = %s\n", buff);
+
+ pr_info("Writing page mask...\n");
+ if (fts_write(cmd2, ARRAY_SIZE(cmd2)) < OK) {
+ pr_err("flash_erase_page_by_page: Page mask ERROR %08X\n",
+ ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ if (fts_write(cmd1, ARRAY_SIZE(cmd1)) < OK) {
+ pr_err("flash_erase_page_by_page: Disable info ERROR %08X\n",
+ ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ pr_info("Command erase pages sent ...\n");
+ if (fts_write(cmd, ARRAY_SIZE(cmd)) < OK) {
+ pr_err("flash_erase_page_by_page: Erase ERROR %08X\n",
+ ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ status = wait_for_flash_ready(FLASH_ERASE_CODE0);
+
+ if (status != OK) {
+ pr_err("flash_erase_page_by_page: ERROR %08X\n",
+ ERROR_FLASH_NOT_READY);
+ return status | ERROR_FLASH_NOT_READY;
+ /* Flash not ready within the chosen time, better exit! */
+ }
+
+ pr_info("Erase flash page by page DONE!\n");
+
+ return OK;
+}
+
+/**
+ * Start the DMA procedure which actually transfer and burn the data loaded
+ * from memory into the Flash
+ * @return OK if success or an error code which specify the type of error
+ */
+int start_flash_dma(void)
+{
+ int status;
+ u8 cmd[6] = { FLASH_CMD_WRITE_REGISTER, 0x20, 0x00, 0x00,
+ FLASH_DMA_CODE0, FLASH_DMA_CODE1 };
+
+ /* write the command to erase the flash */
+
+ pr_info("Command flash DMA ...\n");
+ if (fts_write(cmd, ARRAY_SIZE(cmd)) < OK) {
+ pr_err("start_flash_dma: ERROR %08X\n", ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ status = wait_for_flash_ready(FLASH_DMA_CODE0);
+
+ if (status != OK) {
+ pr_err("start_flash_dma: ERROR %08X\n", ERROR_FLASH_NOT_READY);
+ return status | ERROR_FLASH_NOT_READY;
+ /* Flash not ready within the chosen time, better exit! */
+ }
+
+ pr_info("flash DMA DONE!\n");
+
+ return OK;
+}
+
+/**
+ * Copy the FW data that should be burnt in the Flash into the memory and then
+ * the DMA will take care about burning it into the Flash
+ * @param address address in memory where to copy the data, possible values
+ * are FLASH_ADDR_CODE, FLASH_ADDR_CONFIG, FLASH_ADDR_CX
+ * @param data pointer to an array of byte which contain the data that should
+ * be copied into the memory
+ * @param size size of data
+ * @return OK if success or an error code which specify the type of error
+ */
+int fillFlash(u32 address, u8 *data, int size)
+{
+ int remaining = size, index = 0;
+ int toWrite = 0;
+ int byteBlock = 0;
+ int wheel = 0;
+ u32 addr = 0;
+ int res;
+ int delta;
+ u8 *buff = NULL;
+
+ buff = kmalloc(max(DMA_CHUNK + 5, 12), GFP_KERNEL);
+ if (buff == NULL) {
+ pr_err("fillFlash: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ while (remaining > 0) {
+ byteBlock = 0;
+ addr = 0x00100000;
+
+ while (byteBlock < FLASH_CHUNK && remaining > 0) {
+ index = 0;
+ if (remaining >= DMA_CHUNK) {
+ if ((byteBlock + DMA_CHUNK) <= FLASH_CHUNK) {
+ /* pr_err("fillFlash: 1\n"); */
+ toWrite = DMA_CHUNK;
+ remaining -= DMA_CHUNK;
+ byteBlock += DMA_CHUNK;
+ } else {
+ /* pr_err("fillFlash: 2\n); */
+ delta = FLASH_CHUNK - byteBlock;
+ toWrite = delta;
+ remaining -= delta;
+ byteBlock += delta;
+ }
+ } else {
+ if ((byteBlock + remaining) <= FLASH_CHUNK) {
+ /* pr_err("fillFlash: 3\n"); */
+ toWrite = remaining;
+ byteBlock += remaining;
+ remaining = 0;
+ } else {
+ /* pr_err("fillFlash: 4\n"); */
+ delta = FLASH_CHUNK - byteBlock;
+ toWrite = delta;
+ remaining -= delta;
+ byteBlock += delta;
+ }
+ }
+
+ buff[index++] = FTS_CMD_HW_REG_W;
+ buff[index++] = (u8)((addr & 0xFF000000) >> 24);
+ buff[index++] = (u8)((addr & 0x00FF0000) >> 16);
+ buff[index++] = (u8)((addr & 0x0000FF00) >> 8);
+ buff[index++] = (u8)(addr & 0x000000FF);
+
+ memcpy(&buff[index], data, toWrite);
+ /* pr_err("Command = %02X , address = %02X %02X
+ * , bytes = %d, data = %02X %02X, %02X %02X\n",
+ * buff[0], buff[1], buff[2], toWrite, buff[3],
+ * buff[4], buff[3 + toWrite-2],
+ * buff[3 + toWrite-1]); */
+ if (fts_write_heap(buff, index + toWrite) < OK) {
+ pr_err("fillFlash: ERROR %08X\n", ERROR_BUS_W);
+ kfree(buff);
+ return ERROR_BUS_W;
+ }
+
+ /* mdelay(5); */
+ addr += toWrite;
+ data += toWrite;
+ }
+
+
+ /* configuring the DMA */
+ byteBlock = byteBlock / 4 - 1;
+ index = 0;
+
+ buff[index++] = FLASH_CMD_WRITE_REGISTER;
+ buff[index++] = 0x20;
+ buff[index++] = 0x00;
+ buff[index++] = 0x00;
+ buff[index++] = FLASH_DMA_CONFIG;
+ buff[index++] = 0x00;
+ buff[index++] = 0x00;
+
+ addr = address + ((wheel * FLASH_CHUNK) / 4);
+ buff[index++] = (u8)((addr & 0x000000FF));
+ buff[index++] = (u8)((addr & 0x0000FF00) >> 8);
+ buff[index++] = (u8)(byteBlock & 0x000000FF);
+ buff[index++] = (u8)((byteBlock & 0x0000FF00) >> 8);
+ buff[index++] = 0x00;
+
+ pr_info("DMA Command = %02X , address = %02X %02X, words = %02X %02X\n",
+ buff[0], buff[8], buff[7], buff[10], buff[9]);
+
+ if (fts_write_heap(buff, index) < OK) {
+ pr_err("Error during filling Flash! ERROR %08X\n",
+ ERROR_BUS_W);
+ kfree(buff);
+ return ERROR_BUS_W;
+ }
+
+ res = start_flash_dma();
+ if (res < OK) {
+ pr_err("Error during flashing DMA! ERROR %08X\n", res);
+ kfree(buff);
+ return res;
+ }
+ wheel++;
+ }
+ kfree(buff);
+ return OK;
+}
+
+
+/*
+ * Execute the procedure to burn a FW on FTM5/FTI IC
+ *
+ * @param fw - structure which contain the FW to be burnt
+ * @param force_burn - if >0, the flashing procedure will be forced and
+ * executed
+ * regardless the additional info, otherwise the FW in the file will be
+ * burned only if it is different from the one running in the IC
+ * @param keep_cx - if 1, the function preserves the CX/Panel Init area.
+ * Otherwise, it will be cleared.
+ *
+ * @return OK if success or an error code which specifies the type of error
+ * encountered
+ */
+int flash_burn(Firmware fw, int force_burn, int keep_cx)
+{
+ int res;
+
+ if (!force_burn) {
+ /* Compare firmware, config, and CX versions */
+ if (fw.fw_ver != (uint32_t)systemInfo.u16_fwVer ||
+ fw.config_ver != (uint32_t)systemInfo.u16_cfgVer ||
+ fw.cx_ver != (uint32_t)systemInfo.u16_cxVer)
+ goto start;
+
+ for (res = EXTERNAL_RELEASE_INFO_SIZE - 1; res >= 0; res--) {
+ if (fw.externalRelease[res] !=
+ systemInfo.u8_releaseInfo[res])
+ goto start;
+ }
+
+ pr_info("flash_burn: Firmware in the chip matches the firmware to flash! NO UPDATE ERROR %08X\n",
+ ERROR_FW_NO_UPDATE);
+ return ERROR_FW_NO_UPDATE | ERROR_FLASH_BURN_FAILED;
+ } else if (force_burn == CRC_CX && fw.sec2_size == 0) {
+ /* burn procedure to update the CX memory, if not present just
+ * skip it!
+ */
+ for (res = EXTERNAL_RELEASE_INFO_SIZE - 1; res >= 0; res--) {
+ if (fw.externalRelease[res] !=
+ systemInfo.u8_releaseInfo[res]) {
+ /* Avoid loading the CX because it is missing
+ * in the bin file, it just need to update
+ * to last fw+cfg because a new release */
+ force_burn = 0;
+ goto start;
+ }
+ }
+ pr_info("flash_burn: CRC in CX but fw does not contain CX data! NO UPDATE ERROR %08X\n",
+ ERROR_FW_NO_UPDATE);
+ return ERROR_FW_NO_UPDATE | ERROR_FLASH_BURN_FAILED;
+ }
+
+ /* Programming procedure start */
+start:
+ pr_info("Programming Procedure for flashing started:\n");
+
+ pr_info(" 1) SYSTEM RESET:\n");
+ res = fts_system_reset();
+ if (res < 0) {
+ pr_err(" system reset FAILED!\n");
+ /* If there is no firmware, there is no controller ready event
+ * and there will be a timeout, we can keep going. But if
+ * there is an I2C error, we must exit.
+ */
+ if (res != (ERROR_SYSTEM_RESET_FAIL | ERROR_TIMEOUT))
+ return res | ERROR_FLASH_BURN_FAILED;
+ } else
+ pr_info(" system reset COMPLETED!\n");
+
+ msleep(100); /* required by HW for safe flash procedure */
+
+ pr_info(" 2) HOLD M3 :\n");
+
+ res = hold_m3();
+ if (res < OK) {
+ pr_err(" hold_m3 FAILED!\n");
+ return res | ERROR_FLASH_BURN_FAILED;
+ }
+ pr_info(" hold_m3 COMPLETED!\n");
+
+ pr_info(" 3) FLASH UNLOCK:\n");
+ res = flash_unlock();
+ if (res < OK) {
+ pr_err(" flash unlock FAILED! ERROR %08X\n",
+ ERROR_FLASH_BURN_FAILED);
+ return res | ERROR_FLASH_BURN_FAILED;
+ }
+ pr_info(" flash unlock COMPLETED!\n");
+
+ pr_info(" 4) FLASH ERASE UNLOCK:\n");
+ res = flash_erase_unlock();
+ if (res < 0) {
+ pr_err(" flash unlock FAILED! ERROR %08X\n",
+ ERROR_FLASH_BURN_FAILED);
+ return res | ERROR_FLASH_BURN_FAILED;
+ }
+ pr_info(" flash unlock COMPLETED!\n");
+
+ pr_info(" 5) FLASH ERASE:\n");
+ if (keep_cx > 0) {
+ if (fw.sec2_size != 0)
+ res = flash_erase_page_by_page(SKIP_PANEL_INIT);
+ else
+ res = flash_erase_page_by_page(SKIP_PANEL_CX_INIT);
+ } else {
+ res = flash_erase_page_by_page(SKIP_PANEL_INIT);
+ if (fw.sec2_size == 0)
+ pr_err("WARNING!!! Erasing CX memory but no CX in fw file! touch will not work right after fw update!\n");
+ }
+
+ if (res < OK) {
+ pr_err(" flash erase FAILED! ERROR %08X\n",
+ ERROR_FLASH_BURN_FAILED);
+ return res | ERROR_FLASH_BURN_FAILED;
+ }
+ pr_info(" flash erase COMPLETED!\n");
+
+ pr_info(" 6) LOAD PROGRAM:\n");
+ res = fillFlash(FLASH_ADDR_CODE, &fw.data[0], fw.sec0_size);
+ if (res < OK) {
+ pr_err(" load program ERROR %08X\n",
+ ERROR_FLASH_BURN_FAILED);
+ return res | ERROR_FLASH_BURN_FAILED;
+ }
+ pr_info(" load program DONE!\n");
+
+ pr_info(" 7) LOAD CONFIG:\n");
+ res = fillFlash(FLASH_ADDR_CONFIG, &(fw.data[fw.sec0_size]),
+ fw.sec1_size);
+ if (res < OK) {
+ pr_err(" load config ERROR %08X\n",
+ ERROR_FLASH_BURN_FAILED);
+ return res | ERROR_FLASH_BURN_FAILED;
+ }
+ pr_info(" load config DONE!\n");
+
+ if (fw.sec2_size != 0 && (force_burn == CRC_CX || keep_cx <= 0)) {
+ pr_info(" 7.1) LOAD CX:\n");
+ res = fillFlash(FLASH_ADDR_CX,
+ &(fw.data[fw.sec0_size + fw.sec1_size]),
+ fw.sec2_size);
+ if (res < OK) {
+ pr_err(" load cx ERROR %08X\n",
+ ERROR_FLASH_BURN_FAILED);
+ return res | ERROR_FLASH_BURN_FAILED;
+ }
+ pr_info(" load cx DONE!\n");
+ }
+
+ pr_info(" Flash burn COMPLETED!\n");
+
+ pr_info(" 8) SYSTEM RESET:\n");
+ res = fts_system_reset();
+ if (res < 0) {
+ pr_err(" system reset FAILED! ERROR %08X\n",
+ ERROR_FLASH_BURN_FAILED);
+ return res | ERROR_FLASH_BURN_FAILED;
+ }
+ pr_info(" system reset COMPLETED!\n");
+
+ pr_info(" 9) FINAL CHECK:\n");
+ res = readSysInfo(0);
+ if (res < 0) {
+ pr_err("flash_burn: Unable to retrieve Chip INFO! ERROR %08X\n",
+ ERROR_FLASH_BURN_FAILED);
+ return res | ERROR_FLASH_BURN_FAILED;
+ }
+
+ for (res = 0; res < EXTERNAL_RELEASE_INFO_SIZE; res++) {
+ if (fw.externalRelease[res] != systemInfo.u8_releaseInfo[res]) {
+ /* External release is printed during readSysInfo */
+ pr_info(" Firmware in the chip different from the one that was burn!\n");
+ return ERROR_FLASH_BURN_FAILED;
+ }
+ }
+
+ pr_info(" Final check OK!\n");
+
+ return OK;
+}
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsFlash.h b/drivers/input/touchscreen/stm/fts_lib/ftsFlash.h
new file mode 100644
index 0000000..708c8ba
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsFlash.h
@@ -0,0 +1,115 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS API for Flashing the IC *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsFlash.h
+ * \brief Contains all the definitions and structs to handle the FW update
+ *process
+ */
+
+#ifndef FTS_FLASH_H
+#define FTS_FLASH_H
+
+#include "ftsSoftware.h"
+
+/* Flash possible status */
+#define FLASH_READY 0 /* /< value to indicate that the flash
+ * is ready */
+#define FLASH_BUSY 1 /* /< value to indicate that the flash
+ * is busy */
+#define FLASH_UNKNOWN -1 /* /< value to indicate an unknown
+ * status of the flash */
+
+#define FLASH_STATUS_BYTES 1 /* /< number of bytes to check for read
+ * the flash status */
+
+
+
+/* Flash timing parameters */
+#define FLASH_RETRY_COUNT 200 /* /< number of attempts to read the
+ * flash status */
+#define FLASH_WAIT_BEFORE_RETRY 50 /* /< time to wait in ms between status
+ * readings */
+
+
+#ifdef FW_H_FILE
+#define PATH_FILE_FW "NULL"
+#else
+#define PATH_FILE_FW "ftm5_fw.ftb" /* new FW bin file name */
+#endif
+
+#define FLASH_CHUNK (64 * 1024) /* /< Max number of bytes that
+ * the DMA can burn on the flash
+ * in one shot in FTI */
+#define DMA_CHUNK 32 /* /< Max number of bytes that can be
+ * written in I2C to the DMA */
+
+/**
+ * Define which kind of erase page by page should be performed
+ */
+typedef enum {
+ ERASE_ALL = 0, /* /< erase all the pages */
+ SKIP_PANEL_INIT = 1, /* /< skip erase Panel Init Pages */
+ SKIP_PANEL_CX_INIT = 2 /* /< skip erase Panel Init and CX Pages */
+} ErasePage;
+
+/** @addtogroup fw_file
+ * @{
+ */
+
+/**
+ * Struct which contains information and data of the FW that should be burnt
+ *into the IC
+ */
+typedef struct {
+ u8 *data; /* /< pointer to an array of bytes which represent the
+ * FW data */
+ u16 fw_ver; /* /< FW version of the FW file */
+ u16 config_ver; /* Config version of the FW file */
+ u16 cx_ver; /* /< Cx version of the FW file */
+ u8 externalRelease[EXTERNAL_RELEASE_INFO_SIZE]; /* /< External Release
+ * Info of the FW file
+ * */
+ int data_size; /* /< dimension of data (the actual data to be burnt) */
+ u32 sec0_size; /* /< dimension of section 0 (FW) in .ftb file */
+ u32 sec1_size; /* /< dimension of section 1 (Config) in .ftb file */
+ u32 sec2_size; /* /< dimension of section 2 (Cx) in .ftb file */
+ u32 sec3_size; /* /< dimension of section 3 (TBD) in .ftb file */
+} Firmware;
+
+/** @}*/
+
+/** @addtogroup flash_command
+ * @{
+ */
+
+int wait_for_flash_ready(u8 type);
+int hold_m3(void);
+int flash_erase_unlock(void);
+int flash_full_erase(void);
+int flash_erase_page_by_page(ErasePage keep_cx);
+int start_flash_dma(void);
+int fillFlash(u32 address, u8 *data, int size);
+
+int flash_unlock(void);
+int getFWdata(const char *pathToFile, u8 **data, int *size);
+int parseBinFile(u8 *fw_data, int fw_size, Firmware *fw, int keep_cx);
+int readFwFile(const char *path, Firmware *fw, int keep_cx);
+int flash_burn(Firmware fw, int force_burn, int keep_cx);
+int flashProcedure(const char *path, int force, int keep_cx);
+
+#endif
+
+/** @}*/
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsFrame.c b/drivers/input/touchscreen/stm/fts_lib/ftsFrame.c
new file mode 100644
index 0000000..a18a273
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsFrame.c
@@ -0,0 +1,620 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS functions for getting frames *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsFrame.c
+ * \brief Contains all the functions to work with frames
+ */
+
+#include "ftsCompensation.h"
+#include "ftsCore.h"
+#include "ftsError.h"
+#include "ftsFrame.h"
+#include "ftsHardware.h"
+#include "ftsIO.h"
+#include "ftsSoftware.h"
+#include "ftsTool.h"
+#include "ftsTime.h"
+
+
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <stdarg.h>
+#include <linux/serio.h>
+#include <linux/time.h>
+#include <linux/delay.h>
+#include <linux/ctype.h>
+
+
+
+
+
+extern SysInfo systemInfo; /* /< forward declaration of the global variable
+ * of containing System Info Data */
+
+
+/**
+ * Read the channels lengths from the config memory
+ * @return OK if success or an error code which specify the type of error
+ */
+int getChannelsLength(void)
+{
+ int ret;
+ u8 data[2];
+
+ if (data == NULL) {
+ pr_err("getChannelsLength: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ ret = readConfig(ADDR_CONFIG_SENSE_LEN, data, 2);
+ if (ret < OK) {
+ pr_err("getChannelsLength: ERROR %08X\n", ret);
+
+ return ret;
+ }
+
+ systemInfo.u8_scrRxLen = (int)data[0];
+ systemInfo.u8_scrTxLen = (int)data[1];
+
+ pr_info("Force_len = %d Sense_Len = %d\n",
+ systemInfo.u8_scrTxLen, systemInfo.u8_scrRxLen);
+
+ return OK;
+}
+
+
+
+/**
+ * Read and pack the frame data related to the nodes
+ * @param address address in memory when the frame data node start
+ * @param size amount of data to read
+ * @param frame pointer to an array of bytes which will contain the frame node
+ * data
+ * @return OK if success or an error code which specify the type of error
+ */
+int getFrameData(u16 address, int size, short *frame)
+{
+ int i, j, ret;
+ u8 *data = (u8 *)kmalloc(size * sizeof(u8), GFP_KERNEL);
+
+ if (data == NULL) {
+ pr_err("getFrameData: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, address, data,
+ size, DUMMY_FRAMEBUFFER);
+ if (ret < OK) {
+ pr_err("getFrameData: ERROR %08X\n", ERROR_BUS_R);
+ kfree(data);
+ return ERROR_BUS_R;
+ }
+ j = 0;
+ for (i = 0; i < size; i += 2) {
+ frame[j] = (short)((data[i + 1] << 8) + data[i]);
+ j++;
+ }
+ kfree(data);
+ return OK;
+}
+
+
+/**
+ * Return the number of Sense Channels (Rx)
+ * @return number of Rx channels
+ */
+int getSenseLen(void)
+{
+ if (systemInfo.u8_scrRxLen == 0)
+ getChannelsLength();
+ return systemInfo.u8_scrRxLen;
+}
+
+/**
+ * Return the number of Force Channels (Tx)
+ * @return number of Tx channels
+ */
+int getForceLen(void)
+{
+ if (systemInfo.u8_scrTxLen == 0)
+ getChannelsLength();
+ return systemInfo.u8_scrTxLen;
+}
+
+
+/******************** New API **************************/
+
+/**
+ * Read a MS Frame from frame buffer memory
+ * @param type type of MS frame to read
+ * @param frame pointer to MutualSenseFrame variable which will contain the
+ * data
+ * @return > 0 if success specifying the number of node into the frame or
+ * an error code which specify the type of error
+ */
+int getMSFrame3(MSFrameType type, MutualSenseFrame *frame)
+{
+ u16 offset;
+ int ret, force_len, sense_len;
+
+ force_len = getForceLen();
+ sense_len = getSenseLen();
+
+ frame->node_data = NULL;
+
+ pr_info("%s: Starting to get frame %02X\n", __func__,
+ type);
+ switch (type) {
+ case MS_RAW:
+ offset = systemInfo.u16_msTchRawAddr;
+ goto LOAD_NORM;
+ case MS_FILTER:
+ offset = systemInfo.u16_msTchFilterAddr;
+
+ goto LOAD_NORM;
+ case MS_STRENGTH:
+ offset = systemInfo.u16_msTchStrenAddr;
+ goto LOAD_NORM;
+ case MS_BASELINE:
+ offset = systemInfo.u16_msTchBaselineAddr;
+LOAD_NORM:
+ if (force_len == 0 || sense_len == 0) {
+ pr_err("%s: number of channels not initialized ERROR %08X\n",
+ __func__, ERROR_CH_LEN);
+ return ERROR_CH_LEN | ERROR_GET_FRAME;
+ }
+
+ break;
+
+ case MS_KEY_RAW:
+ offset = systemInfo.u16_keyRawAddr;
+ goto LOAD_KEY;
+ case MS_KEY_FILTER:
+ offset = systemInfo.u16_keyFilterAddr;
+ goto LOAD_KEY;
+ case MS_KEY_STRENGTH:
+ offset = systemInfo.u16_keyStrenAddr;
+ goto LOAD_KEY;
+ case MS_KEY_BASELINE:
+ offset = systemInfo.u16_keyBaselineAddr;
+LOAD_KEY:
+ if (systemInfo.u8_keyLen == 0) {
+ pr_err("%s: number of channels not initialized ERROR %08X\n",
+ __func__, ERROR_CH_LEN);
+ return ERROR_CH_LEN | ERROR_GET_FRAME;
+ }
+ force_len = 1;
+ sense_len = systemInfo.u8_keyLen;
+ break;
+
+ case FRC_RAW:
+ offset = systemInfo.u16_frcRawAddr;
+ goto LOAD_FRC;
+ case FRC_FILTER:
+ offset = systemInfo.u16_frcFilterAddr;
+ goto LOAD_FRC;
+ case FRC_STRENGTH:
+ offset = systemInfo.u16_frcStrenAddr;
+ goto LOAD_FRC;
+ case FRC_BASELINE:
+ offset = systemInfo.u16_frcBaselineAddr;
+LOAD_FRC:
+ if (force_len == 0) {
+ pr_err("%s: number of channels not initialized ERROR %08X\n",
+ __func__, ERROR_CH_LEN);
+ return ERROR_CH_LEN | ERROR_GET_FRAME;
+ }
+ sense_len = 1;
+ break;
+ default:
+ pr_err("%s: Invalid type ERROR %08X\n", __func__,
+ ERROR_OP_NOT_ALLOW | ERROR_GET_FRAME);
+ return ERROR_OP_NOT_ALLOW | ERROR_GET_FRAME;
+ }
+
+ frame->node_data_size = ((force_len) * sense_len);
+ frame->header.force_node = force_len;
+ frame->header.sense_node = sense_len;
+ frame->header.type = type;
+
+ pr_info("%s: Force_len = %d Sense_len = %d Offset = %04X\n",
+ __func__, force_len, sense_len, offset);
+
+ frame->node_data = (short *)kmalloc(frame->node_data_size *
+ sizeof(short), GFP_KERNEL);
+ if (frame->node_data == NULL) {
+ pr_err("%s: ERROR %08X\n", __func__,
+ ERROR_ALLOC | ERROR_GET_FRAME);
+ return ERROR_ALLOC | ERROR_GET_FRAME;
+ }
+
+ ret = getFrameData(offset, frame->node_data_size * BYTES_PER_NODE,
+ (frame->node_data));
+ if (ret < OK) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_GET_FRAME_DATA);
+ kfree(frame->node_data);
+ frame->node_data = NULL;
+ return ret | ERROR_GET_FRAME_DATA | ERROR_GET_FRAME;
+ }
+ /* if you want to access one node i,j,
+ * compute the offset like: offset = i*columns + j = > frame[i, j] */
+
+ pr_info("Frame acquired!\n");
+ return frame->node_data_size;
+ /* return the number of data put inside frame */
+}
+
+/**
+ * Read a SS Frame from frame buffer
+ * @param type type of SS frame to read
+ * @param frame pointer to SelfSenseFrame variable which will contain the data
+ * @return > 0 if success specifying the number of node into frame or an
+ * error code which specify the type of error
+ */
+int getSSFrame3(SSFrameType type, SelfSenseFrame *frame)
+{
+ u16 offset_force, offset_sense;
+ int ret;
+
+ frame->force_data = NULL;
+ frame->sense_data = NULL;
+
+ frame->header.force_node = getForceLen(); /* use getForce/SenseLen
+ * because introduce
+ * a recover mechanism
+ * in case of len =0 */
+ frame->header.sense_node = getSenseLen();
+
+ if (frame->header.force_node == 0 || frame->header.sense_node == 0) {
+ pr_err("%s: number of channels not initialized ERROR %08X\n",
+ __func__, ERROR_CH_LEN);
+ return ERROR_CH_LEN | ERROR_GET_FRAME;
+ }
+
+
+ pr_info("%s: Starting to get frame %02X\n", __func__, type);
+ switch (type) {
+ case SS_RAW:
+ offset_force = systemInfo.u16_ssTchTxRawAddr;
+ offset_sense = systemInfo.u16_ssTchRxRawAddr;
+ break;
+ case SS_FILTER:
+ offset_force = systemInfo.u16_ssTchTxFilterAddr;
+ offset_sense = systemInfo.u16_ssTchRxFilterAddr;
+ break;
+ case SS_STRENGTH:
+ offset_force = systemInfo.u16_ssTchTxStrenAddr;
+ offset_sense = systemInfo.u16_ssTchRxStrenAddr;
+ break;
+ case SS_BASELINE:
+ offset_force = systemInfo.u16_ssTchTxBaselineAddr;
+ offset_sense = systemInfo.u16_ssTchRxBaselineAddr;
+ break;
+
+ case SS_HVR_RAW:
+ offset_force = systemInfo.u16_ssHvrTxRawAddr;
+ offset_sense = systemInfo.u16_ssHvrRxRawAddr;
+ break;
+ case SS_HVR_FILTER:
+ offset_force = systemInfo.u16_ssHvrTxFilterAddr;
+ offset_sense = systemInfo.u16_ssHvrRxFilterAddr;
+ break;
+ case SS_HVR_STRENGTH:
+ offset_force = systemInfo.u16_ssHvrTxStrenAddr;
+ offset_sense = systemInfo.u16_ssHvrRxStrenAddr;
+ break;
+ case SS_HVR_BASELINE:
+ offset_force = systemInfo.u16_ssHvrTxBaselineAddr;
+ offset_sense = systemInfo.u16_ssHvrRxBaselineAddr;
+ break;
+
+ case SS_PRX_RAW:
+ offset_force = systemInfo.u16_ssPrxTxRawAddr;
+ offset_sense = systemInfo.u16_ssPrxRxRawAddr;
+ break;
+ case SS_PRX_FILTER:
+ offset_force = systemInfo.u16_ssPrxTxFilterAddr;
+ offset_sense = systemInfo.u16_ssPrxRxFilterAddr;
+ break;
+ case SS_PRX_STRENGTH:
+ offset_force = systemInfo.u16_ssPrxTxStrenAddr;
+ offset_sense = systemInfo.u16_ssPrxRxStrenAddr;
+ break;
+ case SS_PRX_BASELINE:
+ offset_force = systemInfo.u16_ssPrxTxBaselineAddr;
+ offset_sense = systemInfo.u16_ssPrxRxBaselineAddr;
+ break;
+
+ default:
+ pr_err("%s: Invalid type ERROR %08X\n", __func__,
+ ERROR_OP_NOT_ALLOW | ERROR_GET_FRAME);
+ return ERROR_OP_NOT_ALLOW | ERROR_GET_FRAME;
+ }
+
+ frame->header.type = type;
+
+ pr_info("%s: Force_len = %d Sense_len = %d Offset_force = %04X Offset_sense = %04X\n",
+ __func__, frame->header.force_node,
+ frame->header.sense_node,
+ offset_force, offset_sense);
+
+ frame->force_data = (short *)kmalloc(frame->header.force_node *
+ sizeof(short), GFP_KERNEL);
+ if (frame->force_data == NULL) {
+ pr_err("%s: can not allocate force_data ERROR %08X\n",
+ __func__, ERROR_ALLOC | ERROR_GET_FRAME);
+ return ERROR_ALLOC | ERROR_GET_FRAME;
+ }
+
+ frame->sense_data = (short *)kmalloc(frame->header.sense_node *
+ sizeof(short), GFP_KERNEL);
+ if (frame->sense_data == NULL) {
+ kfree(frame->force_data);
+ frame->force_data = NULL;
+ pr_err("%s: can not allocate sense_data ERROR %08X\n",
+ __func__, ERROR_ALLOC | ERROR_GET_FRAME);
+ return ERROR_ALLOC | ERROR_GET_FRAME;
+ }
+
+ ret = getFrameData(offset_force, frame->header.force_node *
+ BYTES_PER_NODE, (frame->force_data));
+ if (ret < OK) {
+ pr_err("%s: error while reading force data ERROR %08X\n",
+ __func__, ERROR_GET_FRAME_DATA);
+ kfree(frame->force_data);
+ frame->force_data = NULL;
+ kfree(frame->sense_data);
+ frame->sense_data = NULL;
+ return ret | ERROR_GET_FRAME_DATA | ERROR_GET_FRAME;
+ }
+
+ ret = getFrameData(offset_sense, frame->header.sense_node *
+ BYTES_PER_NODE, (frame->sense_data));
+ if (ret < OK) {
+ pr_err("%s: error while reading sense data ERROR %08X\n",
+ __func__, ERROR_GET_FRAME_DATA);
+ kfree(frame->force_data);
+ frame->force_data = NULL;
+ kfree(frame->sense_data);
+ frame->sense_data = NULL;
+ return ret | ERROR_GET_FRAME_DATA | ERROR_GET_FRAME;
+ }
+ /* if you want to access one node i,j,
+ * the offset like: offset = i*columns + j = > frame[i, j] */
+
+ pr_info("Frame acquired!\n");
+ return frame->header.force_node + frame->header.sense_node;
+ /* return the number of data put inside frame */
+}
+
+
+/**
+ * Read Initialization Data Header and check that the type loaded match with
+ * the one previously requested
+ * @param type type of Initialization data requested @link load_opt Load Host
+ * Data Option @endlink
+ * @param msHeader pointer to DataHeader variable which will contain the header
+ * info for the MS frame
+ * @param ssHeader pointer to DataHeader variable which will contain the header
+ * info for the SS frame
+ * @param address pointer to a variable which will contain the updated address
+ * to the next data
+ * @return OK if success or an error code which specify the type of error
+ */
+int readSyncDataHeader(u8 type, DataHeader *msHeader, DataHeader *ssHeader,
+ u64 *address)
+{
+ u64 offset = ADDR_FRAMEBUFFER;
+ u8 data[SYNCFRAME_DATA_HEADER];
+ int ret;
+
+ ret = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, offset, data,
+ SYNCFRAME_DATA_HEADER, DUMMY_FRAMEBUFFER);
+ if (ret < OK) { /* i2c function have already a retry mechanism */
+ pr_err("%s: error while reading data header ERROR %08X\n",
+ __func__, ret);
+ return ret;
+ }
+
+ pr_info("Read Data Header done!\n");
+
+ if (data[0] != HEADER_SIGNATURE) {
+ pr_err("%s: The Header Signature was wrong! %02X != %02X ERROR %08X\n",
+ __func__, data[0], HEADER_SIGNATURE,
+ ERROR_WRONG_DATA_SIGN);
+ return ERROR_WRONG_DATA_SIGN;
+ }
+
+
+ if (data[1] != type) {
+ pr_err("%s: Wrong type found! %02X!=%02X ERROR %08X\n",
+ __func__, data[1], type, ERROR_DIFF_DATA_TYPE);
+ return ERROR_DIFF_DATA_TYPE;
+ }
+
+ pr_info("Type = %02X of SyncFrame data OK!\n", type);
+
+ msHeader->force_node = data[5];
+ msHeader->sense_node = data[6];
+ pr_info("MS Frame force_node = %d, sense_node = %d\n",
+ msHeader->force_node, msHeader->sense_node);
+
+ ssHeader->force_node = data[7];
+ ssHeader->sense_node = data[8];
+ pr_info("SS Frame force_node = %d, sense_node = %d\n",
+ ssHeader->force_node, ssHeader->sense_node);
+
+ *address = offset + SYNCFRAME_DATA_HEADER + data[4];
+
+ return OK;
+}
+
+/**
+ * Read a Sync Frame from frame buffer which contain MS and SS data collected
+ * for the same scan
+ * @param type type of Sync frame to read, possible values:
+ * LOAD_SYNC_FRAME_RAW, LOAD_SYNC_FRAME_FILTER, LOAD_SYNC_FRAME_BASELINE,
+ * LOAD_SYNC_FRAME_STRENGTH
+ * @param msFrame pointer to MutualSenseFrame variable which will contain the
+ * MS data
+ * @param ssFrame pointer to SelfSenseFrame variable which will contain the SS
+ * data
+ * @return >0 if success specifying the total number of nodes copied into
+ * msFrame and ssFrame or an error code which specify the type of error
+ */
+int getSyncFrame(u8 type, MutualSenseFrame *msFrame, SelfSenseFrame *ssFrame)
+{
+ int res;
+ u64 address;
+
+ msFrame->node_data = NULL;
+ ssFrame->force_data = NULL;
+ ssFrame->sense_data = NULL;
+
+ pr_info("%s: Starting to get Sync Frame %02X...\n", __func__, type);
+ switch (type) {
+ case LOAD_SYNC_FRAME_RAW:
+ msFrame->header.type = MS_RAW;
+ ssFrame->header.type = SS_RAW;
+ break;
+
+ case LOAD_SYNC_FRAME_FILTER:
+ msFrame->header.type = MS_FILTER;
+ ssFrame->header.type = SS_FILTER;
+ break;
+
+ case LOAD_SYNC_FRAME_BASELINE:
+ msFrame->header.type = MS_BASELINE;
+ ssFrame->header.type = SS_BASELINE;
+ break;
+
+ case LOAD_SYNC_FRAME_STRENGTH:
+ msFrame->header.type = MS_STRENGTH;
+ ssFrame->header.type = SS_STRENGTH;
+ break;
+
+ default:
+ return ERROR_OP_NOT_ALLOW | ERROR_GET_FRAME;
+ }
+
+ pr_info("%s: Requesting Sync Frame %02X...\n", __func__, type);
+ res = requestSyncFrame(type);
+ if (res < OK) {
+ pr_err("%s: error while requesting Sync Frame ERROR %08X\n",
+ __func__, res | ERROR_GET_FRAME_DATA);
+ return res | ERROR_GET_FRAME_DATA;
+ }
+
+ res = readSyncDataHeader(type, &(msFrame->header), &(ssFrame->header),
+ &address);
+ if (res < OK) {
+ pr_err("%s: error while reading Sync Frame header... ERROR %08X\n",
+ __func__, res | ERROR_GET_FRAME_DATA);
+ return res | ERROR_GET_FRAME_DATA;
+ }
+
+ msFrame->node_data_size = msFrame->header.force_node *
+ msFrame->header.sense_node;
+
+ msFrame->node_data = (short *)kmalloc(msFrame->node_data_size *
+ sizeof(short), GFP_KERNEL);
+ if (msFrame->node_data == NULL) {
+ pr_err("%s: impossible allocate memory for MS frame... ERROR %08X\n",
+ __func__, ERROR_ALLOC | ERROR_GET_FRAME);
+ return ERROR_ALLOC | ERROR_GET_FRAME;
+ }
+
+ pr_info("%s: Getting MS frame at %llx...\n", __func__, address);
+ res = getFrameData(address, (msFrame->node_data_size) * BYTES_PER_NODE,
+ (msFrame->node_data));
+ if (res < OK) {
+ pr_err("%s: error while getting MS data...ERROR %08X\n",
+ __func__, res);
+ res |= ERROR_GET_FRAME_DATA | ERROR_GET_FRAME;
+ goto ERROR;
+ }
+
+ /* move the offset */
+ address += (msFrame->node_data_size) * BYTES_PER_NODE;
+
+ ssFrame->force_data = (short *)kmalloc(ssFrame->header.force_node *
+ sizeof(short), GFP_KERNEL);
+ if (ssFrame->force_data == NULL) {
+ pr_err("%s: impossible allocate memory for SS force frame...ERROR %08X\n",
+ __func__, ERROR_ALLOC | ERROR_GET_FRAME);
+ res = ERROR_ALLOC | ERROR_GET_FRAME;
+ goto ERROR;
+ }
+
+ pr_info("%s: Getting SS force frame at %llx...\n", __func__, address);
+ res = getFrameData(address, (ssFrame->header.force_node) *
+ BYTES_PER_NODE, (ssFrame->force_data));
+ if (res < OK) {
+ pr_err("%s: error while getting SS force data...ERROR %08X\n",
+ __func__, res);
+ res |= ERROR_GET_FRAME_DATA | ERROR_GET_FRAME;
+ goto ERROR;
+ }
+
+ /* move the offset */
+ address += (ssFrame->header.force_node) * BYTES_PER_NODE;
+
+ ssFrame->sense_data = (short *)kmalloc(ssFrame->header.sense_node *
+ sizeof(short), GFP_KERNEL);
+ if (ssFrame->sense_data == NULL) {
+ pr_err("%s: impossible allocate memory for SS sense frame...ERROR %08X\n",
+ __func__, ERROR_ALLOC | ERROR_GET_FRAME);
+ res = ERROR_ALLOC | ERROR_GET_FRAME;
+ goto ERROR;
+ }
+
+ pr_info("%s: Getting SS sense frame at %llx...\n", __func__, address);
+ res = getFrameData(address, (ssFrame->header.sense_node) *
+ BYTES_PER_NODE, (ssFrame->sense_data));
+ if (res < OK) {
+ pr_err("%s: error while getting SS sense data...ERROR %08X\n",
+ __func__, res);
+ res |= ERROR_GET_FRAME_DATA | ERROR_GET_FRAME;
+ goto ERROR;
+ }
+
+ERROR:
+ if (res < OK) {
+ if (msFrame->node_data != NULL) {
+ kfree(msFrame->node_data);
+ msFrame->node_data = NULL;
+ }
+
+ if (ssFrame->force_data != NULL) {
+ kfree(ssFrame->force_data);
+ ssFrame->force_data = NULL;
+ }
+
+ if (ssFrame->sense_data != NULL) {
+ kfree(ssFrame->sense_data);
+ ssFrame->sense_data = NULL;
+ }
+ pr_err("Getting Sync Frame FAILED! ERROR %08X!\n", res);
+ } else {
+ pr_info("Getting Sync Frame FINISHED!\n");
+ res = msFrame->node_data_size + ssFrame->header.force_node +
+ ssFrame->header.sense_node;
+ }
+ return res;
+}
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsFrame.h b/drivers/input/touchscreen/stm/fts_lib/ftsFrame.h
new file mode 100644
index 0000000..bdd5125
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsFrame.h
@@ -0,0 +1,113 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS functions for getting frames *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsFrame.h
+ * \brief Contains all the definitions and structs to work with frames
+ */
+
+
+#ifndef FTS_FRAME_H
+#define FTS_FRAME_H
+
+#include "ftsSoftware.h"
+#include "ftsCore.h"
+
+
+
+/* Number of data bytes for each node */
+#define BYTES_PER_NODE 2 /* /< number of data bytes for each node
+ * */
+
+
+#define RETRY_FRAME_DATA_READ 2 /* /< max number of attempts to read a
+ * frame */
+#define SYNCFRAME_DATA_HEADER (DATA_HEADER + 12) /* /< number of bytes of
+ * Sync Frame Header */
+
+/**
+ * Possible types of MS frames
+ */
+typedef enum {
+ MS_RAW = 0, /* /< Mutual Sense Raw Frame */
+ MS_FILTER = 1, /* /< Mutual Sense Filtered Frame */
+ MS_STRENGTH = 2, /* /< Mutual Sense Strength Frame (Baseline-Raw)
+ * */
+ MS_BASELINE = 3, /* /< Mutual Sense Baseline Frame */
+ MS_KEY_RAW = 4, /* /< Mutual Sense Key Raw Frame */
+ MS_KEY_FILTER = 5, /* /< Mutual Sense Key Filter Frame */
+ MS_KEY_STRENGTH = 6, /* /< Mutual Sense Key Strength Frame
+ * (Baseline-Raw) */
+ MS_KEY_BASELINE = 7, /* /< Mutual Sense Key Baseline Frame */
+ FRC_RAW = 8, /* /< Force Raw Frame */
+ FRC_FILTER = 9, /* /< Force Filtered Frame */
+ FRC_STRENGTH = 10, /* /< Force Strength Frame (Baseline-Raw) */
+ FRC_BASELINE = 11 /* /< Force Baseline Frame */
+} MSFrameType;
+
+
+/**
+ * Possible types of SS frames
+ */
+typedef enum {
+ SS_RAW = 0, /* /< Self Sense Raw Frame */
+ SS_FILTER = 1, /* /< Self Sense Filtered Frame */
+ SS_STRENGTH = 2, /* /< Self Sense Strength Frame (Baseline-Raw)
+ * */
+ SS_BASELINE = 3, /* /< Self Sense Baseline Frame */
+ SS_HVR_RAW = 4, /* /< Self Sense Hover Raw Frame */
+ SS_HVR_FILTER = 5, /* /< Self Sense Hover Filter Frame */
+ SS_HVR_STRENGTH = 6, /* /< Self Sense Hover Strength Frame
+ * (Baseline-Raw) */
+ SS_HVR_BASELINE = 7, /* /< Self Sense Hover Baseline Frame */
+ SS_PRX_RAW = 8, /* /< Self Sense Proximity Raw Frame */
+ SS_PRX_FILTER = 9, /* /< Self Sense Proximity Filtered Frame */
+ SS_PRX_STRENGTH = 10, /* /< Self Sense Proximity Strength Frame
+ * (Baseline-Raw) */
+ SS_PRX_BASELINE = 11 /* /< Self Sense Proximity Baseline Frame */
+} SSFrameType;
+
+
+/**
+ * Struct which contains the data of a MS Frame
+ */
+typedef struct {
+ DataHeader header; /* /< Header which contain basic info of the
+ * frame */
+ short *node_data; /* /< Data of the frame */
+ int node_data_size; /* /< Dimension of the data of the frame */
+} MutualSenseFrame;
+
+/**
+ * Struct which contains the data of a SS Frame
+ */
+typedef struct {
+ DataHeader header; /* /< Header which contain basic info of the
+ * frame */
+ short *force_data; /* /< Force Channels Data */
+ short *sense_data; /* /< Sense Channels Data */
+} SelfSenseFrame;
+
+
+int getChannelsLength(void);
+int getFrameData(u16 address, int size, short *frame);
+int getSenseLen(void);
+int getForceLen(void);
+int getMSFrame3(MSFrameType type, MutualSenseFrame *frame);
+int getSSFrame3(SSFrameType type, SelfSenseFrame *frame);
+int readSyncDataHeader(u8 type, DataHeader *msHeader, DataHeader *ssHeader,
+ u64 *address);
+int getSyncFrame(u8 type, MutualSenseFrame *msFrame, SelfSenseFrame *ssFrame);
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsGesture.c b/drivers/input/touchscreen/stm/fts_lib/ftsGesture.c
new file mode 100644
index 0000000..904c2ef8
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsGesture.c
@@ -0,0 +1,376 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS Gesture Utilities **
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsGesture.c
+ * \brief Contains all the functions and variable to handle the Gesture
+ * Detection features
+ */
+
+#include "ftsSoftware.h"
+#include "ftsCore.h"
+#include "ftsError.h"
+#include "ftsGesture.h"
+#include "ftsIO.h"
+#include "ftsTime.h"
+#include "ftsTool.h"
+
+/* /< store the gesture bitmask which the host want to enable.
+ * If bit set 1 the corresponding gesture will be detected in Gesture Mode */
+static u8 gesture_mask[GESTURE_MASK_SIZE] = { 0 };
+/* /< store the x coordinates of the points draw by the user
+ * when a gesture is detected */
+u16 gesture_coordinates_x[GESTURE_MAX_COORDS_PAIRS_REPORT] = { 0 };
+/* /< store the y coordinates of the points draw by the user
+ * when a gesture is detected */
+u16 gesture_coordinates_y[GESTURE_MAX_COORDS_PAIRS_REPORT] = { 0 };
+/* /< number of coordinates pairs (points) reported with the detected gesture */
+int gesture_coords_reported = ERROR_OP_NOT_ALLOW;
+static u8 refreshGestureMask; /* /< flag which indicate if there is
+ * the need to set the gesture mask in the FW */
+struct mutex gestureMask_mutex; /* /< mutex used to control access on gesture
+ * shared variables */
+
+/**
+ * Update the gesture mask stored in the driver and have to be used in gesture
+ * mode
+ * @param mask pointer to a byte array which store the gesture mask update
+ * that want to be performed.
+ * @param size dimension in byte of mask. This size can be <=
+ * GESTURE_MASK_SIZE.
+ * If size < GESTURE_MASK_SIZE the bytes of mask are considering continuos
+ * and starting from the less significant byte.
+ * @param en 0 = enable the gestures set in mask, 1 = disable the gestures set
+ * in mask
+ * @return OK if success or an error code which specify the type of error
+ */
+int updateGestureMask(u8 *mask, int size, int en)
+{
+ u8 temp;
+ int i;
+
+ if (mask != NULL) {
+ if (size <= GESTURE_MASK_SIZE) {
+ if (en == FEAT_ENABLE) {
+ mutex_lock(&gestureMask_mutex);
+ pr_info("updateGestureMask: setting gesture mask to enable...\n");
+ if (mask != NULL)
+ for (i = 0; i < size; i++)
+ gesture_mask[i] =
+ gesture_mask[i] |
+ mask[i];
+ /* back up of the gesture enabled */
+ refreshGestureMask = 1;
+ pr_info("updateGestureMask: gesture mask to enable SET!\n");
+ mutex_unlock(&gestureMask_mutex);
+ return OK;
+ } else if (en == FEAT_DISABLE) {
+ mutex_lock(&gestureMask_mutex);
+ pr_info("updateGestureMask: setting gesture mask to disable...\n");
+ for (i = 0; i < size; i++) {
+ temp = gesture_mask[i] ^ mask[i];
+ /* enabled XOR disabled */
+ gesture_mask[i] = temp &
+ gesture_mask[i];
+ /* temp AND enabled
+ * disable the gestures that were
+ * enabled */
+ }
+ pr_info("updateGestureMask: gesture mask to disable SET!\n");
+ refreshGestureMask = 1;
+ mutex_unlock(&gestureMask_mutex);
+ return OK;
+ } else {
+ pr_err("updateGestureMask: Enable parameter Invalid! %d != %d or %d ERROR %08X",
+ en,
+ FEAT_DISABLE, FEAT_ENABLE,
+ ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+ } else {
+ pr_err("updateGestureMask: Size not valid! %d > %d ERROR %08X\n",
+ size, GESTURE_MASK_SIZE, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+ } else {
+ pr_err("updateGestureMask: Mask NULL! ERROR %08X\n",
+ ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+}
+
+/**
+ * Enable in the FW the gesture mask to be used in gesture mode
+ * @param mask pointer to a byte array which store the gesture mask update
+ * that want to be sent to the FW, if NULL, will be used gesture_mask
+ * set previously without any changes.
+ * @param size dimension in byte of mask. This size can be <=
+ * GESTURE_MASK_SIZE.
+ * If size < GESTURE_MASK_SIZE the bytes of mask are considering continuos and
+ * starting from the less significant byte.
+ * @return OK if success or an error code which specify the type of error
+ */
+int enableGesture(u8 *mask, int size)
+{
+ int i, res;
+
+ pr_info("Trying to enable gesture...\n");
+
+ if (size <= GESTURE_MASK_SIZE) {
+ mutex_lock(&gestureMask_mutex);
+ if (mask != NULL)
+ for (i = 0; i < size; i++)
+ gesture_mask[i] = gesture_mask[i] | mask[i];
+ /* back up of the gesture enabled */
+
+ res = setFeatures(FEAT_SEL_GESTURE, gesture_mask,
+ GESTURE_MASK_SIZE);
+ if (res < OK) {
+ pr_err("enableGesture: ERROR %08X\n", res);
+ goto END;
+ }
+
+ pr_info("enableGesture DONE!\n");
+ res = OK;
+
+END:
+ mutex_unlock(&gestureMask_mutex);
+ return res;
+ } else {
+ pr_err("enableGesture: Size not valid! %d > %d ERROR %08X\n",
+ size, GESTURE_MASK_SIZE, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+}
+
+/**
+ * Disable in the FW the gesture mask to be used in gesture mode
+ * @param mask pointer to a byte array which store the gesture mask update that
+ * want to be sent to the FW, if NULL, all the gestures will be disabled.
+ * @param size dimension in byte of mask. This size can be <=
+ * GESTURE_MASK_SIZE.
+ * If size < GESTURE_MASK_SIZE the bytes of mask are considering continuos and
+ * starting from the less significant byte.
+ * @return OK if success or an error code which specify the type of error
+ */
+int disableGesture(u8 *mask, int size)
+{
+ u8 temp;
+ int i, res;
+ u8 *pointer;
+
+
+ pr_info("Trying to disable gesture...\n");
+
+
+ if (size <= GESTURE_MASK_SIZE) {
+ mutex_lock(&gestureMask_mutex);
+ if (mask != NULL) {
+ for (i = 0; i < size; i++) {
+ temp = gesture_mask[i] ^ mask[i];
+ /* enabled mask XOR disabled mask */
+ gesture_mask[i] = temp & gesture_mask[i];
+ /* temp AND enabled
+ * disable the gestures that are specified and
+ * previously enabled */
+ }
+
+ pointer = gesture_mask;
+ } else {
+ i = 0; /* if NULL is passed disable all the possible
+ * gestures */
+ pointer = (u8 *)&i;
+ }
+
+ res = setFeatures(FEAT_SEL_GESTURE, pointer, GESTURE_MASK_SIZE);
+ if (res < OK) {
+ pr_err("disableGesture: ERROR %08X\n", res);
+ goto END;
+ }
+
+ pr_info("disableGesture DONE!\n");
+
+ res = OK;
+
+END:
+ mutex_unlock(&gestureMask_mutex);
+ return res;
+ } else {
+ pr_err("disableGesture: Size not valid! %d > %d ERROR %08X\n",
+ size, GESTURE_MASK_SIZE, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+}
+
+/**
+ * Perform all the steps required to put the chip in gesture mode
+ * @param reload if set to 1, before entering in gesture mode it will re-enable
+ * in the FW the last defined gesture mask
+ * @return OK if success or an error code which specify the type of error
+ */
+int enterGestureMode(int reload)
+{
+ int res, ret;
+
+ res = fts_disableInterrupt();
+ if (res < OK) {
+ pr_err("enterGestureMode: ERROR %08X\n",
+ res | ERROR_DISABLE_INTER);
+ return res | ERROR_DISABLE_INTER;
+ }
+
+ if (reload == 1 || refreshGestureMask == 1) {
+ res = enableGesture(NULL, 0);
+ if (res < OK) {
+ pr_err("enterGestureMode: enableGesture ERROR %08X\n",
+ res);
+ goto END;
+ }
+
+ refreshGestureMask = 0;
+ }
+
+ res = setScanMode(SCAN_MODE_LOW_POWER, 0);
+ if (res < OK) {
+ pr_err("enterGestureMode: enter gesture mode ERROR %08X\n",
+ res);
+ goto END;
+ }
+
+ res = OK;
+END:
+ ret = fts_enableInterrupt();
+ if (ret < OK) {
+ pr_err("enterGestureMode: fts_enableInterrupt ERROR %08X\n",
+ res | ERROR_ENABLE_INTER);
+ res |= ret | ERROR_ENABLE_INTER;
+ }
+
+
+ return res;
+}
+
+/**
+ * Check if one or more Gesture IDs are currently enabled in gesture_mask
+ * @return FEAT_ENABLE if one or more gesture ids are enabled, FEAT_DISABLE if
+ * all the gesture ids are currently disabled
+ */
+int isAnyGestureActive(void)
+{
+ int res = 0;
+
+ while (res < (GESTURE_MASK_SIZE - 1) && gesture_mask[res] == 0)
+ /* -1 because in any case the last gesture mask byte will
+ * be evaluated with the following if */
+ res++;
+
+ if (gesture_mask[res] != 0) {
+ pr_info("%s: Active Gestures Found! gesture_mask[%d] = %02X !\n",
+ __func__, res, gesture_mask[res]);
+ return FEAT_ENABLE;
+ } else {
+ pr_info("%s: All Gestures Disabled!\n", __func__);
+ return FEAT_DISABLE;
+ }
+}
+
+
+/**
+ * Read from the frame buffer the gesture coordinates pairs of the points draw
+ * by an user when a gesture is detected
+ * @param event pointer to a byte array which contains the gesture event
+ * reported
+ * by the fw when a gesture is detected
+ * @return OK if success or an error code which specify the type of error
+ */
+int readGestureCoords(u8 *event)
+{
+ int i = 0;
+ u64 address = 0;
+ int res;
+
+ u8 val[GESTURE_MAX_COORDS_PAIRS_REPORT * 4];
+
+ /* the max coordinates to read are GESTURE_COORDS_REPORT_MAX*4
+ * (because each coordinate is a short(*2) and we have x and y) */
+
+
+ if (event[0] == EVT_ID_USER_REPORT &&
+ event[1] == EVT_TYPE_USER_GESTURE) {
+ address = (event[4] << 8) | event[3]; /* Offset in framebuff */
+ gesture_coords_reported = event[5]; /* number of pairs
+ * coords reported */
+ if (gesture_coords_reported > GESTURE_MAX_COORDS_PAIRS_REPORT) {
+ pr_err("%s: FW reported more than %d points for the gestures! Decreasing to %d\n",
+ __func__, gesture_coords_reported,
+ GESTURE_MAX_COORDS_PAIRS_REPORT);
+ gesture_coords_reported =
+ GESTURE_MAX_COORDS_PAIRS_REPORT;
+ }
+
+ pr_info("%s: Offset: %llx , coords pairs = %d\n",
+ __func__, address, gesture_coords_reported);
+
+ res = fts_writeReadU8UX(FTS_CMD_FRAMEBUFFER_R, BITS_16, address,
+ val, (gesture_coords_reported * 2 * 2),
+ DUMMY_FRAMEBUFFER);
+ /* *2 because each coord is made by 2 bytes,
+ * *2 because there are x and y */
+ if (res < OK) {
+ pr_err("%s: Cannot read the coordinates! ERROR %08X\n",
+ __func__, res);
+ gesture_coords_reported = ERROR_OP_NOT_ALLOW;
+ return res;
+ }
+
+ /* all the points of the gesture are stored in val */
+ for (i = 0; i < gesture_coords_reported; i++) {
+ gesture_coordinates_x[i] =
+ (((u16)val[i * 2 + 1]) & 0x0F) << 8 |
+ (((u16)val[i * 2]) & 0xFF);
+ gesture_coordinates_y[i] =
+ (((u16)val[gesture_coords_reported *
+ 2 + i * 2 + 1]) & 0x0F) << 8 |
+ (((u16)val[gesture_coords_reported *
+ 2 + i * 2]) & 0xFF);
+ }
+
+
+ pr_info("%s: Reading Gesture Coordinates DONE!\n", __func__);
+ return OK;
+ } else {
+ pr_err("%s: The event passsed as argument is invalid! ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+}
+
+/**
+ * Return the coordinates of the points stored during the last detected gesture
+ * @param x output parameter which will store the address of the array
+ * containing the x coordinates
+ * @param y output parameter which will store the address of the array
+ * containing the y coordinates
+ * @return the number of points (x,y) stored and therefore the size of the x
+ * and y array returned.
+ */
+int getGestureCoords(u16 **x, u16 **y)
+{
+ *x = gesture_coordinates_x;
+ *y = gesture_coordinates_y;
+ pr_info("%s: Number of gesture coordinates pairs returned = %d\n",
+ __func__, gesture_coords_reported);
+ return gesture_coords_reported;
+}
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsGesture.h b/drivers/input/touchscreen/stm/fts_lib/ftsGesture.h
new file mode 100644
index 0000000..cb59737f
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsGesture.h
@@ -0,0 +1,47 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS Gesture Utilities *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsGesture.h
+ * \brief Contains all the macro and prototypes to handle the Gesture Detection
+ * features
+ */
+
+
+#ifndef FTS_GESTURE_H_
+#define FTS_GESTURE_H_
+
+
+
+
+#include "ftsHardware.h"
+
+#define GESTURE_MASK_SIZE 4 /* /< number of bytes of the
+ * gesture mask */
+
+#define GESTURE_MAX_COORDS_PAIRS_REPORT 100 /* /< max number of gestures
+ * coordinates pairs reported */
+
+
+
+int updateGestureMask(u8 *mask, int size, int en);
+int disableGesture(u8 *mask, int size);
+int enableGesture(u8 *mask, int size);
+int enterGestureMode(int reload);
+int isAnyGestureActive(void);
+int readGestureCoords(u8 *event);
+int getGestureCoords(u16 **x, u16 **y);
+
+#endif /* ! _GESTURE_H_ */
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsHardware.h b/drivers/input/touchscreen/stm/fts_lib/ftsHardware.h
new file mode 100644
index 0000000..845da2b
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsHardware.h
@@ -0,0 +1,282 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * HW related data *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsHardware.h
+ * \brief Contains all the definitions and information related to the IC
+ * from an hardware point of view
+ */
+
+#ifndef FTS_HARDWARE_H
+#define FTS_HARDWARE_H
+
+/* DIGITAL CHIP INFO */
+#define DCHIP_ID_0 0x36 /* /< LSB chip ID for FTM5 */
+#define DCHIP_ID_1 0x39 /* /< MSB chip ID for FTM5 */
+
+#define DCHIP_FW_VER_BYTE 2 /* /< number of bytes of the fw
+ * versions */
+
+/* CHUNKS */
+#define READ_CHUNK 1024 /* /< chunk dimension of
+ * a single i2c read,
+ * max allowed value is 2kB */
+#define WRITE_CHUNK 1024 /* /< chunk dimension of
+ * a single i2c write,
+ * max allowed value is 2kB */
+#define MEMORY_CHUNK 1024 /* /< chunk dimenasion of
+ * a single i2c write on mem,
+ * max allowed value is 2kB */
+
+/* PROTOCOL INFO */
+#define I2C_INTERFACE /* /< comment if the chip use SPI bus */
+#ifdef I2C_INTERFACE
+#define I2C_SAD 0x49 /* /< slave address of the IC */
+#else
+#define SPI4_WIRE /* /< comment if the master is SPI3
+ * wires
+ * (MOSI and MISO share same line) */
+#define SPI_DELAY_CS 10 /* /< time in usec to wait
+ * before rising the CS */
+#define SPI_CLOCK_FREQ 7000000 /* /< clock frequency in Hz of
+ * the SPI bus */
+#endif
+
+#define IER_ENABLE 0x41 /* /< value to write in IER_ADDR
+ * to enable the interrupts */
+#define IER_DISABLE 0x00 /* /< value to write in IER_ADDR
+ * to disable the interrupts
+ **/
+
+/* FLASH COMMAND */
+/** @defgroup flash_command Flash Commands
+ * All the commands that works with the flash of the IC
+ * @{
+ */
+#define FLASH_CMD_UNLOCK 0xF7
+
+#define FLASH_CMD_READ_REGISTER 0xFA
+#define FLASH_CMD_WRITE_REGISTER 0xFA
+
+/* FLASH UNLOCK PARAMETER */
+#define FLASH_UNLOCK_CODE0 0x25
+#define FLASH_UNLOCK_CODE1 0x20
+
+/* FLASH ERASE and DMA PARAMETER */
+#define FLASH_ERASE_START 0x80
+#define FLASH_ERASE_CODE1 0xC0
+#define FLASH_DMA_CODE1 0xC0
+#define FLASH_ERASE_UNLOCK_CODE0 0xDE
+#define FLASH_ERASE_UNLOCK_CODE1 0x03
+#define FLASH_ERASE_CODE0 0x6A
+#define FLASH_DMA_CODE0 0x71
+#define FLASH_DMA_CONFIG 0x72
+#define FLASH_NUM_PAGE 32 /* /< number of pages in main
+ * flash */
+#define FLASH_CX_PAGE_START 28 /* /< starting page which
+ * contain Cx data */
+#define FLASH_CX_PAGE_END 30 /* /< last page which contain Cx
+ * data */
+#define FLASH_PANEL_PAGE_START 26 /* /< starting page which
+ * contain Panel Init data */
+#define FLASH_PANEL_PAGE_END 27 /* /< last page which contain
+ * Panel Init data */
+/** @} */
+
+/* FLASH ADDRESS */
+#define FLASH_ADDR_CODE 0x00000000 /* /< address of code in
+ * FTI */
+#define FLASH_ADDR_CONFIG 0x00007C00 /* /< address of config
+ * in FTI */
+#define FLASH_ADDR_CX 0x00007000 /* /< address of Init
+ * data in FTI */
+
+/* SIZES FW, CODE, CONFIG, MEMH */
+/** @defgroup fw_file FW file info
+ * All the info related to the fw file
+ * @{
+ */
+#define FW_HEADER_SIZE 64 /* /< dimension of the header in
+ * .fts file */
+#define FW_HEADER_SIGNATURE 0xAA55AA55 /* /< header signature
+ * */
+#define FW_FTB_VER 0x00000001 /* /< .ftb version */
+#define FW_BYTES_ALIGN 4 /* /< allignment of the info */
+#define FW_BIN_VER_OFFSET 16 /* /< offset of the fw version
+ * in the .ftb file */
+#define FW_BIN_CONFIG_ID_OFFSET 20 /* /< offset of the config id in
+ * the .ftb file */
+#define FW_CX_VERSION (16 + 4)/* CX version offset in sec2 of
+ * FW */
+/** @} */
+
+/* FIFO */
+#define FIFO_EVENT_SIZE 8 /* /< number of bytes of one
+ * event */
+#define FIFO_DEPTH 32 /* /< max number of events that
+ * the FIFO can collect before
+ * going in overflow in FTM5 */
+
+#ifdef I2C_INTERFACE
+#define FIFO_CMD_READALL 0x86 /* /< command to read all
+ * the events in the FIFO */
+#else
+#define FIFO_CMD_READALL 0x87 /* /< command to read all
+ * the events in the FIFO */
+#endif
+#define FIFO_CMD_READONE FIFO_CMD_READALL/* /< commad to read
+ * one event from FIFO
+ **/
+
+
+/* OP CODES FOR MEMORY (based on protocol) */
+#ifdef I2C_INTERFACE
+#define FTS_CMD_HW_REG_R 0xFA /* /< command to read an hw register if
+ * FTI */
+#define FTS_CMD_HW_REG_W 0xFA /* /< command to write an hw register if
+ * FTI */
+#define FTS_CMD_FRAMEBUFFER_R 0xA6 /* /< command to read the framebuffer if
+ * FTI */
+#define FTS_CMD_CONFIG_R 0xA8 /* /< command to read the config memory
+ * if FTI */
+#define FTS_CMD_CONFIG_W 0xA8 /* /< command to write the config memory
+ * if FTI */
+#else
+#define FTS_CMD_HW_REG_R 0xFB /* /< command to read an hw register if
+ * FTI */
+#define FTS_CMD_HW_REG_W 0xFA /* /< command to write an hw register if
+ * FTI */
+#define FTS_CMD_FRAMEBUFFER_R 0xA7 /* /< command to read the framebuffer if
+ * FTI */
+#define FTS_CMD_CONFIG_R 0xA9 /* /< command to read the config memory
+ * if FTI */
+#define FTS_CMD_CONFIG_W 0xA8 /* /< command to write the config memory
+ * if FTI */
+#endif
+
+/* DUMMY BYTES DATA */
+#ifndef I2C_INTERFACE
+#define DUMMY_HW_REG 1 /* /< 1 if the first byte read from HW
+ * register is dummy */
+#define DUMMY_FRAMEBUFFER 1 /* /< 1 if the first byte read from
+ * Frame buffer is dummy */
+#define DUMMY_CONFIG 1 /* /< 1 if the first byte read from
+ * Config Memory is dummy */
+#define DUMMY_FIFO 1 /* /< 1 if the first byte read from FIFO
+ * is dummy */
+#else
+#define DUMMY_HW_REG 0 /* /< 1 if the first byte read from HW
+ * register is dummy */
+#define DUMMY_FRAMEBUFFER 0 /* /< 1 if the first byte read from
+ * Frame buffer is dummy */
+#define DUMMY_CONFIG 0 /* /< 1 if the first byte read from
+ * Config Memory is dummy */
+#define DUMMY_FIFO 0 /* /< 1 if the first byte read from FIFO
+ * is dummy */
+#endif
+
+/** @defgroup hw_adr HW Address
+ * @ingroup address
+ * Important addresses of hardware registers (and sometimes their important
+ * values)
+ * @{
+ */
+
+/* IMPORTANT HW ADDRESSES (u64) */
+#define ADDR_FRAMEBUFFER ((u64)0x0000000000000000) /* /< frame
+ * buffer
+ * address in
+ * memory */
+#define ADDR_ERROR_DUMP ((u64)0x000000000000EF80) /* /< start
+ * address dump
+ * error log */
+
+/* SYSTEM RESET INFO */
+#define ADDR_SYSTEM_RESET ((u64)0x0000000020000024) /* /< address of
+ * System
+ * control
+ * register in
+ * FTI */
+/* value to write in SYSTEM_RESET_ADDRESS to perform a system reset in FTM5 */
+#define SYSTEM_RESET_VALUE 0x80
+
+
+/* REMAP REGISTER */
+#define ADDR_BOOT_OPTION ((u64)0x0000000020000025) /* /< address of
+ * Boot option
+ * register */
+
+/* INTERRUPT INFO */
+#define ADDR_IER ((u64)0x0000000020000029) /* /< address of
+ * the Interrupt
+ * enable
+ * register in
+ * FTMI */
+
+/* Chip ID/Fw Version */
+#define ADDR_DCHIP_ID ((u64)0x0000000020000000) /* /< chip id
+ * address for
+ * FTI */
+#define ADDR_DCHIP_FW_VER ((u64)0x0000000020000004) /* /< fw version
+ * address for
+ * FTI */
+
+/* INTERFACE REGISTER */
+#define ADDR_ICR ((u64)0x000000002000002D) /* /< address of
+ * Device
+ * control
+ * register
+ * to set the
+ * communication
+ * protocol
+ * (SPI/I2C) */
+
+#define SPI4_MASK 0x02 /* /< bit to set spi4 */
+
+/* CRC ADDR */
+#define ADDR_CRC ((u64)0x0000000020000078) /* /< address of
+ * the CRC
+ * control
+ * register in
+ * FTI */
+#define CRC_MASK 0x03 /* /< bitmask which reveal if
+ * there is a CRC error in the flash */
+
+#define ADDR_CONFIG_OFFSET ((u64)0x0000000000000000) /* /< config
+ * address in
+ * memory if FTI
+ * */
+
+#define ADDR_GPIO_INPUT ((u64)0x0000000020000030) /* /< address of
+ * GPIO input
+ * register */
+#define ADDR_GPIO_DIRECTION ((u64)0x0000000020000032) /* /< address of
+ * GPIO
+ * direction
+ * register */
+#define ADDR_GPIO_PULLUP ((u64)0x0000000020000034) /* /< address of
+ * GPIO pullup
+ * register */
+#define ADDR_GPIO_CONFIG_REG0 ((u64)0x000000002000003D) /* /< address of
+ * GPIO config
+ * register */
+#define ADDR_GPIO_CONFIG_REG2 ((u64)0x000000002000003F) /* /< address of
+ * GPIO config
+ * register */
+
+/**@}*/
+
+
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsIO.c b/drivers/input/touchscreen/stm/fts_lib/ftsIO.c
new file mode 100644
index 0000000..9443fdd
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsIO.c
@@ -0,0 +1,924 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * I2C/SPI Communication *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsIO.c
+ * \brief Contains all the functions which handle with the I2C/SPI
+ *communication
+ */
+
+
+#include "ftsSoftware.h"
+
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <stdarg.h>
+#include <linux/delay.h>
+#include <linux/ctype.h>
+#include <linux/of_gpio.h>
+
+#ifdef I2C_INTERFACE
+#include <linux/i2c.h>
+#include <linux/i2c-dev.h>
+static u16 I2CSAD; /* /< slave address of the IC in the i2c bus */
+#else
+#include <linux/spi/spidev.h>
+#endif
+
+static void *client; /* /< bus client retrived by the OS and
+ * used to execute the bus transfers */
+
+
+
+#include "ftsCore.h"
+#include "ftsError.h"
+#include "ftsHardware.h"
+#include "ftsIO.h"
+
+
+/**
+ * Initialize the static client variable of the fts_lib library in order
+ * to allow any i2c/spi transaction in the driver (Must be called in the probe)
+ * @param clt pointer to i2c_client or spi_device struct which identify the bus
+ * slave device
+ * @return OK
+ */
+int openChannel(void *clt)
+{
+ client = clt;
+#ifdef I2C_INTERFACE
+ I2CSAD = ((struct i2c_client *)clt)->addr;
+ pr_info("openChannel: SAD: %02X\n", I2CSAD);
+#else
+ pr_info("%s: spi_master: flags = %04X !\n", __func__,
+ ((struct spi_device *)client)->master->flags);
+ pr_info("%s: spi_device: max_speed = %d chip select = %02X bits_per_words = %d mode = %04X !\n",
+ __func__, ((struct spi_device *)client)->max_speed_hz,
+ ((struct spi_device *)client)->chip_select,
+ ((struct spi_device *)client)->bits_per_word,
+ ((struct spi_device *)client)->mode);
+ pr_info("openChannel: completed!\n");
+#endif
+ return OK;
+}
+
+#ifdef I2C_INTERFACE
+/**
+ * Change the I2C slave address which will be used during the transaction
+ * (For Debug Only)
+ * @param sad new slave address id
+ * @return OK
+ */
+int changeSAD(u8 sad)
+{
+ I2CSAD = sad;
+ return OK;
+}
+#endif
+
+
+/**
+ * Retrieve the pointer to the device struct of the IC
+ * @return a the device struct pointer if client was previously set
+ * or NULL in all the other cases
+ */
+struct device *getDev(void)
+{
+ if (client != NULL)
+ return &(getClient()->dev);
+ else
+ return NULL;
+}
+
+
+#ifdef I2C_INTERFACE
+/**
+ * Retrieve the pointer of the i2c_client struct representing the IC as i2c
+ * slave
+ * @return client if it was previously set or NULL in all the other cases
+ */
+struct i2c_client *getClient()
+{
+ if (client != NULL)
+ return (struct i2c_client *)client;
+ else
+ return NULL;
+}
+#else
+/**
+ * Retrieve the pointer of the spi_device struct representing the IC as spi
+ * slave
+ * @return client if it was previously set or NULL in all the other cases
+ */
+struct spi_device *getClient()
+{
+ if (client != NULL)
+ return (struct spi_device *)client;
+ else
+ return NULL;
+}
+#endif
+
+struct fts_ts_info *getDrvInfo(void)
+{
+ struct device *dev = getDev();
+ struct fts_ts_info *info = NULL;
+
+ if (dev != NULL)
+ info = dev_get_drvdata(dev);
+ return info;
+}
+
+/****************** New I2C API *********************/
+
+/**
+ * Perform a direct bus read
+ * @param outBuf pointer of a byte array which should contain the byte read
+ * from the IC
+ * @param byteToRead number of bytes to read
+ * @return OK if success or an error code which specify the type of error
+ */
+static int fts_read_internal(u8 *outBuf, int byteToRead, bool dma_safe)
+{
+ int ret = -1;
+ int retry = 0;
+ struct fts_ts_info *info = getDrvInfo();
+#ifdef I2C_INTERFACE
+ struct i2c_msg I2CMsg[1];
+#else
+ struct spi_message msg;
+ struct spi_transfer transfer[1] = { { 0 } };
+#endif
+
+ if (dma_safe == false && byteToRead > sizeof(info->io_read_buf)) {
+ pr_err("%s: preallocated buffers are too small!\n", __func__);
+ return ERROR_ALLOC;
+ }
+
+#ifdef I2C_INTERFACE
+ I2CMsg[0].addr = (__u16)I2CSAD;
+ I2CMsg[0].flags = (__u16)I2C_M_RD;
+ I2CMsg[0].len = (__u16)byteToRead;
+ if (dma_safe == false)
+ I2CMsg[0].buf = (__u8 *)info->io_read_buf;
+ else
+ I2CMsg[0].buf = (__u8 *)outBuf;
+#else
+ spi_message_init(&msg);
+
+ transfer[0].len = byteToRead;
+ transfer[0].delay_usecs = SPI_DELAY_CS;
+ transfer[0].tx_buf = NULL;
+ if (dma_safe == false)
+ transfer[0].rx_buf = info->io_read_buf;
+ else
+ transfer[0].rx_buf = outBuf;
+ spi_message_add_tail(&transfer[0], &msg);
+#endif
+
+ if (client == NULL)
+ return ERROR_BUS_O;
+ while (retry < I2C_RETRY && ret < OK) {
+#ifdef I2C_INTERFACE
+ ret = i2c_transfer(getClient()->adapter, I2CMsg, 1);
+#else
+ ret = spi_sync(getClient(), &msg);
+#endif
+
+ retry++;
+ if (ret < OK)
+ mdelay(I2C_WAIT_BEFORE_RETRY);
+ /* pr_err("fts_writeCmd: attempt %d\n", retry); */
+ }
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_BUS_R);
+ return ERROR_BUS_R;
+ }
+
+ if (dma_safe == false)
+ memcpy(outBuf, info->io_read_buf, byteToRead);
+
+ return OK;
+}
+
+
+/**
+ * Perform a bus write followed by a bus read without a stop condition
+ * @param cmd byte array containing the command to write
+ * @param cmdLength size of cmd
+ * @param outBuf pointer of a byte array which should contain the bytes read
+ * from the IC
+ * @param byteToRead number of bytes to read
+ * @return OK if success or an error code which specify the type of error
+ */
+static int fts_writeRead_internal(u8 *cmd, int cmdLength, u8 *outBuf,
+ int byteToRead, bool dma_safe)
+{
+ int ret = -1;
+ int retry = 0;
+ struct fts_ts_info *info = getDrvInfo();
+#ifdef I2C_INTERFACE
+ struct i2c_msg I2CMsg[2];
+#else
+ struct spi_message msg;
+ struct spi_transfer transfer[2] = { { 0 }, { 0 } };
+#endif
+
+ if (dma_safe == false && (cmdLength > sizeof(info->io_write_buf) ||
+ byteToRead > sizeof(info->io_read_buf))) {
+ pr_err("%s: preallocated buffers are too small!\n", __func__);
+ return ERROR_ALLOC;
+ }
+
+#ifdef I2C_INTERFACE
+ if (dma_safe == false) {
+ memcpy(info->io_write_buf, cmd, cmdLength);
+ cmd = info->io_write_buf;
+ }
+
+ /* write msg */
+ I2CMsg[0].addr = (__u16)I2CSAD;
+ I2CMsg[0].flags = (__u16)0;
+ I2CMsg[0].len = (__u16)cmdLength;
+ I2CMsg[0].buf = (__u8 *)cmd;
+
+ /* read msg */
+ I2CMsg[1].addr = (__u16)I2CSAD;
+ I2CMsg[1].flags = I2C_M_RD;
+ I2CMsg[1].len = byteToRead;
+ if (dma_safe == false)
+ I2CMsg[1].buf = (__u8 *)info->io_read_buf;
+ else
+ I2CMsg[1].buf = (__u8 *)outBuf;
+#else
+ if (dma_safe == false) {
+ memcpy(info->io_write_buf, cmd, cmdLength);
+ cmd = info->io_write_buf;
+ }
+
+ spi_message_init(&msg);
+
+ transfer[0].len = cmdLength;
+ transfer[0].tx_buf = cmd;
+ transfer[0].rx_buf = NULL;
+ spi_message_add_tail(&transfer[0], &msg);
+
+ transfer[1].len = byteToRead;
+ transfer[1].delay_usecs = SPI_DELAY_CS;
+ transfer[1].tx_buf = NULL;
+ if (dma_safe == false)
+ transfer[1].rx_buf = info->io_read_buf;
+ else
+ transfer[1].rx_buf = outBuf;
+ spi_message_add_tail(&transfer[1], &msg);
+
+#endif
+
+ if (client == NULL)
+ return ERROR_BUS_O;
+
+ while (retry < I2C_RETRY && ret < OK) {
+#ifdef I2C_INTERFACE
+ ret = i2c_transfer(getClient()->adapter, I2CMsg, 2);
+#else
+ ret = spi_sync(getClient(), &msg);
+#endif
+
+ retry++;
+ if (ret < OK)
+ mdelay(I2C_WAIT_BEFORE_RETRY);
+ }
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_BUS_WR);
+ return ERROR_BUS_WR;
+ }
+
+ if (dma_safe == false)
+ memcpy(outBuf, info->io_read_buf, byteToRead);
+
+ return OK;
+}
+
+
+/**
+ * Perform a bus write
+ * @param cmd byte array containing the command to write
+ * @param cmdLength size of cmd
+ * @return OK if success or an error code which specify the type of error
+ */
+static int fts_write_internal(u8 *cmd, int cmdLength, bool dma_safe)
+{
+ int ret = -1;
+ int retry = 0;
+ struct fts_ts_info *info = getDrvInfo();
+#ifdef I2C_INTERFACE
+ struct i2c_msg I2CMsg[1];
+#else
+ struct spi_message msg;
+ struct spi_transfer transfer[1] = { { 0 } };
+#endif
+
+ if (dma_safe == false && cmdLength > sizeof(info->io_write_buf)) {
+ pr_err("%s: preallocated buffers are too small!\n", __func__);
+ return ERROR_ALLOC;
+ }
+
+#ifdef I2C_INTERFACE
+ if (dma_safe == false) {
+ memcpy(info->io_write_buf, cmd, cmdLength);
+ cmd = info->io_write_buf;
+ }
+
+ I2CMsg[0].addr = (__u16)I2CSAD;
+ I2CMsg[0].flags = (__u16)0;
+ I2CMsg[0].len = (__u16)cmdLength;
+ I2CMsg[0].buf = (__u8 *)cmd;
+#else
+ if (dma_safe == false) {
+ memcpy(info->io_write_buf, cmd, cmdLength);
+ cmd = info->io_write_buf;
+ }
+
+ spi_message_init(&msg);
+
+ transfer[0].len = cmdLength;
+ transfer[0].delay_usecs = SPI_DELAY_CS;
+ transfer[0].tx_buf = cmd;
+ transfer[0].rx_buf = NULL;
+ spi_message_add_tail(&transfer[0], &msg);
+#endif
+
+
+ if (client == NULL)
+ return ERROR_BUS_O;
+ while (retry < I2C_RETRY && ret < OK) {
+#ifdef I2C_INTERFACE
+ ret = i2c_transfer(getClient()->adapter, I2CMsg, 1);
+#else
+ ret = spi_sync(getClient(), &msg);
+#endif
+
+ retry++;
+ if (ret < OK)
+ mdelay(I2C_WAIT_BEFORE_RETRY);
+ /* pr_err("fts_writeCmd: attempt %d\n", retry); */
+ }
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+ return OK;
+}
+
+/**
+ * Write a FW command to the IC and check automatically the echo event
+ * @param cmd byte array containing the command to send
+ * @param cmdLength size of cmd
+ * @return OK if success, or an error code which specify the type of error
+ */
+static int fts_writeFwCmd_internal(u8 *cmd, int cmdLength, bool dma_safe)
+{
+ int ret = -1;
+ int ret2 = -1;
+ int retry = 0;
+ struct fts_ts_info *info = getDrvInfo();
+#ifdef I2C_INTERFACE
+ struct i2c_msg I2CMsg[1];
+#else
+ struct spi_message msg;
+ struct spi_transfer transfer[1] = { { 0 } };
+#endif
+
+ if (dma_safe == false && cmdLength > sizeof(info->io_write_buf)) {
+ pr_err("%s: preallocated buffers are too small!\n", __func__);
+ return ERROR_ALLOC;
+ }
+
+#ifdef I2C_INTERFACE
+ if (dma_safe == false) {
+ memcpy(info->io_write_buf, cmd, cmdLength);
+ cmd = info->io_write_buf;
+ }
+
+ I2CMsg[0].addr = (__u16)I2CSAD;
+ I2CMsg[0].flags = (__u16)0;
+ I2CMsg[0].len = (__u16)cmdLength;
+ I2CMsg[0].buf = (__u8 *)cmd;
+#else
+ if (dma_safe == false) {
+ memcpy(info->io_write_buf, cmd, cmdLength);
+ cmd = info->io_write_buf;
+ }
+
+ spi_message_init(&msg);
+
+ transfer[0].len = cmdLength;
+ transfer[0].delay_usecs = SPI_DELAY_CS;
+ transfer[0].tx_buf = cmd;
+ transfer[0].rx_buf = NULL;
+ spi_message_add_tail(&transfer[0], &msg);
+#endif
+
+ if (client == NULL)
+ return ERROR_BUS_O;
+ resetErrorList();
+ while (retry < I2C_RETRY && (ret < OK || ret2 < OK)) {
+#ifdef I2C_INTERFACE
+ ret = i2c_transfer(getClient()->adapter, I2CMsg, 1);
+#else
+ ret = spi_sync(getClient(), &msg);
+#endif
+ retry++;
+ if (ret >= 0)
+ ret2 = checkEcho(cmd, cmdLength);
+ if (ret < OK || ret2 < OK)
+ mdelay(I2C_WAIT_BEFORE_RETRY);
+ /* pr_err("fts_writeCmd: attempt %d\n", retry); */
+ }
+ if (ret < 0) {
+ pr_err("fts_writeFwCmd: ERROR %08X\n", ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+ if (ret2 < OK) {
+ pr_err("fts_writeFwCmd: check echo ERROR %08X\n", ret2);
+ return ret2;
+ }
+ return OK;
+}
+
+
+/**
+ * Perform two bus write and one bus read without any stop condition
+ * In case of FTI this function is not supported and the same sequence
+ * can be achieved calling fts_write followed by an fts_writeRead.
+ * @param writeCmd1 byte array containing the first command to write
+ * @param writeCmdLength size of writeCmd1
+ * @param readCmd1 byte array containing the second command to write
+ * @param readCmdLength size of readCmd1
+ * @param outBuf pointer of a byte array which should contain the bytes read
+ * from the IC
+ * @param byteToRead number of bytes to read
+ * @return OK if success or an error code which specify the type of error
+ */
+static int fts_writeThenWriteRead_internal(u8 *writeCmd1, int writeCmdLength,
+ u8 *readCmd1, int readCmdLength,
+ u8 *outBuf, int byteToRead,
+ bool dma_safe)
+{
+ int ret = -1;
+ int retry = 0;
+ struct fts_ts_info *info = getDrvInfo();
+#ifdef I2C_INTERFACE
+ struct i2c_msg I2CMsg[3];
+#else
+ struct spi_message msg;
+ struct spi_transfer transfer[3] = { { 0 }, { 0 }, { 0 } };
+#endif
+
+ if (dma_safe == false && (writeCmdLength > sizeof(info->io_write_buf) ||
+ readCmdLength > sizeof(info->io_extra_write_buf) ||
+ byteToRead > sizeof(info->io_read_buf))) {
+ pr_err("%s: preallocated buffers are too small!\n", __func__);
+ return ERROR_ALLOC;
+ }
+
+#ifdef I2C_INTERFACE
+ if (dma_safe == false) {
+ memcpy(info->io_write_buf, writeCmd1, writeCmdLength);
+ writeCmd1 = info->io_write_buf;
+ memcpy(info->io_extra_write_buf, readCmd1, readCmdLength);
+ readCmd1 = info->io_extra_write_buf;
+ }
+
+ /* write msg */
+ I2CMsg[0].addr = (__u16)I2CSAD;
+ I2CMsg[0].flags = (__u16)0;
+ I2CMsg[0].len = (__u16)writeCmdLength;
+ I2CMsg[0].buf = (__u8 *)writeCmd1;
+
+ /* write msg */
+ I2CMsg[1].addr = (__u16)I2CSAD;
+ I2CMsg[1].flags = (__u16)0;
+ I2CMsg[1].len = (__u16)readCmdLength;
+ I2CMsg[1].buf = (__u8 *)readCmd1;
+
+ /* read msg */
+ I2CMsg[2].addr = (__u16)I2CSAD;
+ I2CMsg[2].flags = I2C_M_RD;
+ I2CMsg[2].len = byteToRead;
+ if (dma_safe == false)
+ I2CMsg[2].buf = (__u8 *)info->io_read_buf;
+ else
+ I2CMsg[2].buf = (__u8 *)outBuf;
+#else
+ if (dma_safe == false) {
+ memcpy(info->io_write_buf, writeCmd1, writeCmdLength);
+ writeCmd1 = info->io_write_buf;
+ memcpy(info->io_extra_write_buf, readCmd1, readCmdLength);
+ readCmd1 = info->io_extra_write_buf;
+ }
+
+ spi_message_init(&msg);
+
+ transfer[0].len = writeCmdLength;
+ transfer[0].tx_buf = writeCmd1;
+ transfer[0].rx_buf = NULL;
+ spi_message_add_tail(&transfer[0], &msg);
+
+ transfer[1].len = readCmdLength;
+ transfer[1].tx_buf = readCmd1;
+ transfer[1].rx_buf = NULL;
+ spi_message_add_tail(&transfer[1], &msg);
+
+ transfer[2].len = byteToRead;
+ transfer[2].delay_usecs = SPI_DELAY_CS;
+ transfer[2].tx_buf = NULL;
+ if (dma_safe == false)
+ transfer[2].rx_buf = info->io_read_buf;
+ else
+ transfer[2].rx_buf = outBuf;
+ spi_message_add_tail(&transfer[2], &msg);
+#endif
+
+ if (client == NULL)
+ return ERROR_BUS_O;
+ while (retry < I2C_RETRY && ret < OK) {
+#ifdef I2C_INTERFACE
+ ret = i2c_transfer(getClient()->adapter, I2CMsg, 3);
+#else
+ ret = spi_sync(getClient(), &msg);
+#endif
+ retry++;
+ if (ret < OK)
+ mdelay(I2C_WAIT_BEFORE_RETRY);
+ }
+
+ if (ret < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_BUS_WR);
+ return ERROR_BUS_WR;
+ }
+
+ if (dma_safe == false)
+ memcpy(outBuf, info->io_read_buf, byteToRead);
+
+ return OK;
+}
+
+/* Wrapper API for i2c read and write */
+int fts_read(u8 *outBuf, int byteToRead)
+{
+ return fts_read_internal(outBuf, byteToRead, false);
+}
+
+int fts_read_heap(u8 *outBuf, int byteToRead)
+{
+ return fts_read_internal(outBuf, byteToRead, true);
+}
+
+int fts_writeRead(u8 *cmd, int cmdLength, u8 *outBuf, int byteToRead)
+{
+ return fts_writeRead_internal(cmd, cmdLength, outBuf, byteToRead,
+ false);
+}
+
+int fts_writeRead_heap(u8 *cmd, int cmdLength, u8 *outBuf, int byteToRead)
+{
+ return fts_writeRead_internal(cmd, cmdLength, outBuf, byteToRead, true);
+}
+
+int fts_write(u8 *cmd, int cmdLength)
+{
+ return fts_write_internal(cmd, cmdLength, false);
+}
+
+int fts_write_heap(u8 *cmd, int cmdLength)
+{
+ return fts_write_internal(cmd, cmdLength, true);
+}
+
+int fts_writeFwCmd(u8 *cmd, int cmdLength)
+{
+ return fts_writeFwCmd_internal(cmd, cmdLength, false);
+}
+
+int fts_writeFwCmd_heap(u8 *cmd, int cmdLength)
+{
+ return fts_writeFwCmd_internal(cmd, cmdLength, true);
+}
+
+int fts_writeThenWriteRead(u8 *writeCmd1, int writeCmdLength,
+ u8 *readCmd1, int readCmdLength,
+ u8 *outBuf, int byteToRead)
+{
+ return fts_writeThenWriteRead_internal(writeCmd1, writeCmdLength,
+ readCmd1, readCmdLength,
+ outBuf, byteToRead, false);
+}
+
+int fts_writeThenWriteRead_heap(u8 *writeCmd1, int writeCmdLength,
+ u8 *readCmd1, int readCmdLength,
+ u8 *outBuf, int byteToRead)
+{
+ return fts_writeThenWriteRead_internal(writeCmd1, writeCmdLength,
+ readCmd1, readCmdLength,
+ outBuf, byteToRead, true);
+}
+
+/**
+ * Perform a chunked write with one byte op code and 1 to 8 bytes address
+ * @param cmd byte containing the op code to write
+ * @param addrSize address size in byte
+ * @param address the starting address
+ * @param data pointer of a byte array which contain the bytes to write
+ * @param dataSize size of data
+ * @return OK if success or an error code which specify the type of error
+ */
+/* this function works only if the address is max 8 bytes */
+int fts_writeU8UX(u8 cmd, AddrSize addrSize, u64 address, u8 *data,
+ int dataSize)
+{
+ u8 *finalCmd;
+ int remaining = dataSize;
+ int toWrite = 0, i = 0;
+ struct fts_ts_info *info = getDrvInfo();
+
+ finalCmd = info->io_write_buf;
+
+ if (addrSize <= sizeof(u64)) {
+ while (remaining > 0) {
+ if (remaining >= WRITE_CHUNK) {
+ toWrite = WRITE_CHUNK;
+ remaining -= WRITE_CHUNK;
+ } else {
+ toWrite = remaining;
+ remaining = 0;
+ }
+
+ finalCmd[0] = cmd;
+ pr_debug("%s: addrSize = %d\n", __func__, addrSize);
+ for (i = 0; i < addrSize; i++) {
+ finalCmd[i + 1] = (u8)((address >> ((addrSize -
+ 1 - i) *
+ 8)) & 0xFF);
+ pr_debug("%s: cmd[%d] = %02X\n",
+ __func__, i + 1, finalCmd[i + 1]);
+ }
+
+ memcpy(&finalCmd[addrSize + 1], data, toWrite);
+
+ if (fts_write_heap(finalCmd, 1 + addrSize + toWrite)
+ < OK) {
+ pr_err(" %s: ERROR %08X\n",
+ __func__, ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ address += toWrite;
+
+ data += toWrite;
+ }
+ } else
+ pr_err("%s: address size bigger than max allowed %lu... ERROR %08X\n",
+ __func__, sizeof(u64), ERROR_OP_NOT_ALLOW);
+
+ return OK;
+}
+
+/**
+ * Perform a chunked write read with one byte op code and 1 to 8 bytes address
+ * and dummy byte support.
+ * @param cmd byte containing the op code to write
+ * @param addrSize address size in byte
+ * @param address the starting address
+ * @param outBuf pointer of a byte array which contain the bytes to read
+ * @param byteToRead number of bytes to read
+ * @param hasDummyByte if the first byte of each reading is dummy (must be
+ * skipped)
+ * set to 1, otherwise if it is valid set to 0 (or any other value)
+ * @return OK if success or an error code which specify the type of error
+ */
+int fts_writeReadU8UX(u8 cmd, AddrSize addrSize, u64 address, u8 *outBuf,
+ int byteToRead, int hasDummyByte)
+{
+ u8 *finalCmd;
+ u8 *buff;
+ int remaining = byteToRead;
+ int toRead = 0, i = 0;
+ struct fts_ts_info *info = getDrvInfo();
+
+ finalCmd = info->io_write_buf;
+ buff = info->io_read_buf;
+
+ while (remaining > 0) {
+ if (remaining >= READ_CHUNK) {
+ toRead = READ_CHUNK;
+ remaining -= READ_CHUNK;
+ } else {
+ toRead = remaining;
+ remaining = 0;
+ }
+
+ finalCmd[0] = cmd;
+ for (i = 0; i < addrSize; i++)
+ finalCmd[i + 1] = (u8)((address >> ((addrSize - 1 - i) *
+ 8)) & 0xFF);
+
+ if (hasDummyByte == 1) {
+ if (fts_writeRead_heap(finalCmd, 1 + addrSize, buff,
+ toRead + 1) < OK) {
+ pr_err("%s: read error... ERROR %08X\n",
+ __func__, ERROR_BUS_WR);
+ return ERROR_BUS_WR;
+ }
+ memcpy(outBuf, buff + 1, toRead);
+ } else {
+ if (fts_writeRead_heap(finalCmd, 1 + addrSize, buff,
+ toRead) < OK) {
+ pr_err("%s: read error... ERROR %08X\n",
+ __func__, ERROR_BUS_WR);
+ return ERROR_BUS_WR;
+ }
+ memcpy(outBuf, buff, toRead);
+ }
+
+ address += toRead;
+
+ outBuf += toRead;
+ }
+
+ return OK;
+}
+
+/**
+ * Perform a chunked write followed by a second write with one byte op code
+ * for each write and 1 to 8 bytes address (the sum of the 2 address size of
+ * the two writes can not exceed 8 bytes)
+ * @param cmd1 byte containing the op code of first write
+ * @param addrSize1 address size in byte of first write
+ * @param cmd2 byte containing the op code of second write
+ * @param addrSize2 address size in byte of second write
+ * @param address the starting address
+ * @param data pointer of a byte array which contain the bytes to write
+ * @param dataSize size of data
+ * @return OK if success or an error code which specify the type of error
+ */
+/* this function works only if the sum of two addresses in the two commands is
+ * max 8 bytes */
+int fts_writeU8UXthenWriteU8UX(u8 cmd1, AddrSize addrSize1, u8 cmd2,
+ AddrSize addrSize2, u64 address, u8 *data,
+ int dataSize)
+{
+ u8 *finalCmd1;
+ u8 *finalCmd2;
+ int remaining = dataSize;
+ int toWrite = 0, i = 0;
+ struct fts_ts_info *info = getDrvInfo();
+
+ finalCmd1 = info->io_write_buf;
+ finalCmd2 = info->io_extra_write_buf;
+
+ while (remaining > 0) {
+ if (remaining >= WRITE_CHUNK) {
+ toWrite = WRITE_CHUNK;
+ remaining -= WRITE_CHUNK;
+ } else {
+ toWrite = remaining;
+ remaining = 0;
+ }
+
+ finalCmd1[0] = cmd1;
+ for (i = 0; i < addrSize1; i++)
+ finalCmd1[i + 1] = (u8)((address >> ((addrSize1 +
+ addrSize2 - 1 -
+ i) * 8)) & 0xFF);
+
+ finalCmd2[0] = cmd2;
+ for (i = addrSize1; i < addrSize1 + addrSize2; i++)
+ finalCmd2[i - addrSize1 + 1] = (u8)((address >>
+ ((addrSize1 +
+ addrSize2 - 1 -
+ i) * 8)) & 0xFF);
+
+ memcpy(&finalCmd2[addrSize2 + 1], data, toWrite);
+
+ if (fts_write_heap(finalCmd1, 1 + addrSize1) < OK) {
+ pr_err("%s: first write error... ERROR %08X\n",
+ __func__, ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ if (fts_write_heap(finalCmd2, 1 + addrSize2 + toWrite) < OK) {
+ pr_err("%s: second write error... ERROR %08X\n",
+ __func__, ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ address += toWrite;
+
+ data += toWrite;
+ }
+
+ return OK;
+}
+
+/**
+ * Perform a chunked write followed by a write read with one byte op code
+ * and 1 to 8 bytes address for each write and dummy byte support.
+ * @param cmd1 byte containing the op code of first write
+ * @param addrSize1 address size in byte of first write
+ * @param cmd2 byte containing the op code of second write read
+ * @param addrSize2 address size in byte of second write read
+ * @param address the starting address
+ * @param outBuf pointer of a byte array which contain the bytes to read
+ * @param byteToRead number of bytes to read
+ * @param hasDummyByte if the first byte of each reading is dummy (must be
+ * skipped) set to 1,
+ * otherwise if it is valid set to 0 (or any other value)
+ * @return OK if success or an error code which specify the type of error
+ */
+/* this function works only if the sum of two addresses in the two commands is
+ * max 8 bytes */
+int fts_writeU8UXthenWriteReadU8UX(u8 cmd1, AddrSize addrSize1, u8 cmd2,
+ AddrSize addrSize2, u64 address, u8 *outBuf,
+ int byteToRead, int hasDummyByte)
+{
+ u8 *finalCmd1;
+ u8 *finalCmd2;
+ u8 *buff;
+ int remaining = byteToRead;
+ int toRead = 0, i = 0;
+ struct fts_ts_info *info = getDrvInfo();
+
+ finalCmd1 = info->io_write_buf;
+ finalCmd2 = info->io_extra_write_buf;
+ buff = info->io_read_buf;
+
+ while (remaining > 0) {
+ if (remaining >= READ_CHUNK) {
+ toRead = READ_CHUNK;
+ remaining -= READ_CHUNK;
+ } else {
+ toRead = remaining;
+ remaining = 0;
+ }
+
+
+ finalCmd1[0] = cmd1;
+ for (i = 0; i < addrSize1; i++)
+ finalCmd1[i + 1] = (u8)((address >> ((addrSize1 +
+ addrSize2 - 1 -
+ i) * 8)) & 0xFF);
+
+ finalCmd2[0] = cmd2;
+ for (i = addrSize1; i < addrSize1 + addrSize2; i++)
+ finalCmd2[i - addrSize1 + 1] = (u8)((address >>
+ ((addrSize1 +
+ addrSize2 - 1 -
+ i) * 8)) & 0xFF);
+
+ if (fts_write_heap(finalCmd1, 1 + addrSize1) < OK) {
+ pr_err("%s: first write error... ERROR %08X\n",
+ __func__, ERROR_BUS_W);
+ return ERROR_BUS_W;
+ }
+
+ if (hasDummyByte == 1) {
+ if (fts_writeRead_heap(finalCmd2, 1 + addrSize2, buff,
+ toRead + 1) < OK) {
+ pr_err("%s: read error... ERROR %08X\n",
+ __func__, ERROR_BUS_WR);
+ return ERROR_BUS_WR;
+ }
+ memcpy(outBuf, buff + 1, toRead);
+ } else {
+ if (fts_writeRead_heap(finalCmd2, 1 + addrSize2, buff,
+ toRead) < OK) {
+ pr_err("%s: read error... ERROR %08X\n",
+ __func__, ERROR_BUS_WR);
+ return ERROR_BUS_WR;
+ }
+ memcpy(outBuf, buff, toRead);
+ }
+
+ address += toRead;
+
+ outBuf += toRead;
+ }
+
+ return OK;
+}
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsIO.h b/drivers/input/touchscreen/stm/fts_lib/ftsIO.h
new file mode 100644
index 0000000..f470991
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsIO.h
@@ -0,0 +1,78 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * I2C/SPI Communication *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+/*!
+ * \file ftsIO.h
+ * \brief Contains all the definitions and prototypes used and implemented in
+ * ftsIO.c
+ */
+
+#ifndef FTS_IO_H
+#define FTS_IO_H
+
+#include "ftsSoftware.h"
+
+#define I2C_RETRY 3 /* /< number of retry in case of i2c
+ * failure */
+#define I2C_WAIT_BEFORE_RETRY 2 /* /< wait in ms before retry an i2c
+ * transaction */
+
+#ifdef I2C_INTERFACE
+#include <linux/i2c.h>
+#include <linux/i2c-dev.h>
+struct i2c_client *getClient(void);
+#else
+#include <linux/spi/spi.h>
+struct spi_device *getClient(void);
+#endif
+
+
+
+int openChannel(void *clt);
+struct device *getDev(void);
+
+
+
+/*************** NEW I2C API ****************/
+int changeSAD(u8 sad);
+int fts_read(u8 *outBuf, int byteToRead);
+int fts_read_heap(u8 *outBuf, int byteToRead);
+int fts_writeRead(u8 *cmd, int cmdLength, u8 *outBuf, int byteToRead);
+int fts_writeRead_heap(u8 *cmd, int cmdLength, u8 *outBuf, int byteToRead);
+int fts_write(u8 *cmd, int cmdLength);
+int fts_write_heap(u8 *cmd, int cmdLength);
+int fts_writeFwCmd(u8 *cmd, int cmdLength);
+int fts_writeFwCmd_heap(u8 *cmd, int cmdLength);
+int fts_writeThenWriteRead(u8 *writeCmd1, int writeCmdLength,
+ u8 *readCmd1, int readCmdLength,
+ u8 *outBuf, int byteToRead);
+int fts_writeThenWriteRead_heap(u8 *writeCmd1, int writeCmdLength,
+ u8 *readCmd1, int readCmdLength,
+ u8 *outBuf, int byteToRead);
+
+/* chunked version of fts_write */
+int fts_writeU8UX(u8 cmd, AddrSize addrSize, u64 address, u8 *data,
+ int dataSize);
+/* chunked version of fts_writeRead */
+int fts_writeReadU8UX(u8 cmd, AddrSize addrSize, u64 address, u8 *outBuf,
+ int byteToRead, int hasDummyByte);
+/* chunked, write followed by another write */
+int fts_writeU8UXthenWriteU8UX(u8 cmd1, AddrSize addrSize1, u8 cmd2,
+ AddrSize addrSize2, u64 address, u8 *data,
+ int dataSize);
+/* chunked, write followed by a writeRead */
+int fts_writeU8UXthenWriteReadU8UX(u8 cmd1, AddrSize addrSize1, u8 cmd2,
+ AddrSize addrSize2, u64 address, u8 *outBuf,
+ int count, int hasDummyByte);
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsSoftware.h b/drivers/input/touchscreen/stm/fts_lib/ftsSoftware.h
new file mode 100644
index 0000000..d00fdc8
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsSoftware.h
@@ -0,0 +1,510 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FW related data *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsSoftware.h
+ * \brief Contains all the definitions and information related to the IC
+ * from a fw/driver point of view
+ */
+
+
+#ifndef FTS_SOFTWARE_H
+#define FTS_SOFTWARE_H
+#include <linux/types.h>
+#include "ftsHardware.h"
+
+/* signed type */
+typedef signed char i8; /* /< basic type that represent one signed byte (or 8
+ * bits) */
+
+/**
+ * Enumerator which contains all the possible address length expressed in
+ *bytes.
+ */
+typedef enum {
+ NO_ADDR = 0,
+ BITS_8 = 1,
+ BITS_16 = 2,
+ BITS_24 = 3,
+ BITS_32 = 4,
+ BITS_40 = 5,
+ BITS_48 = 6,
+ BITS_56 = 7,
+ BITS_64 = 8
+} AddrSize;
+
+
+/******************** NEW API *********************/
+
+/* HOST COMMAND */
+/** @defgroup host_command Fw Host op codes
+ * Valid op codes for fw commands
+ * @{
+ */
+
+/** @defgroup scan_mode Scan Mode
+ * @ingroup host_command
+ * Set the scanning mode required according to the parameters
+ * @{
+ */
+#define FTS_CMD_SCAN_MODE 0xA0 /* /< OP Code to set scan mode */
+/** @} */
+
+/** @defgroup feat_sel Feature Select
+ * @ingroup host_command
+ * Set the system defined features to enable/disable according the parameters
+ * @{
+ */
+#define FTS_CMD_FEATURE 0xA2 /* /< OP code to set features */
+/** @} */
+
+/** @defgroup sys_cmd System Command
+ * @ingroup host_command
+ * Execute a system command to perform core tasks
+ * @{
+ */
+#define FTS_CMD_SYSTEM 0xA4 /* /< OP code to write s system command
+ * */
+/** @} */
+
+/** @} */ /* end host_command group */
+
+/* SCAN MODE OPTION (0xA0) */
+/* Scan mode selection */
+/** @defgroup scan_opt Scan Mode Option
+ * @ingroup scan_mode
+ * Valid scanning modes and their options
+ * @{
+ */
+#define SCAN_MODE_ACTIVE 0x00 /* /< Select the Active scanning mode */
+#define SCAN_MODE_LOW_POWER 0x01 /* /< Select the low power scanning mode
+ * */
+#define SCAN_MODE_JIG_1 0x02 /* /< Select the Jig test 1 */
+#define SCAN_MODE_LOCKED 0x03 /* /< Select the Scan mode which will be
+ * locked */
+/** @}*/
+
+/* Active mode option (bitmask) */
+/** @defgroup active_bitmask Active Mode Bitmask
+ * @ingroup scan_opt
+ * Bitmask to use to enables the specific scanning with the SCAN_MODE_ACTIVE
+ * option
+ * @{
+ */
+#define ACTIVE_MULTI_TOUCH 0x01 /* /< Bit 0 MS/SS scan */
+#define ACTIVE_KEY 0x02 /* /< Bit 1 Key scan */
+#define ACTIVE_HOVER 0x04 /* /< Bit 2 Hover scan */
+#define ACTIVE_PROXIMITY 0x08 /* /< Bit 3 Proximity scan */
+#define ACTIVE_FORCE 0x10 /* /< Bit 4 Force scan */
+/** @}*/
+
+/* Locked mode option (locked mode) */
+/** @defgroup locked_opt Locked Mode Option
+ * @ingroup scan_opt
+ * Options to enable and lock specific scanning with the SCAN_MODE_LOCKED
+ * option
+ * @{
+ */
+#define LOCKED_ACTIVE 0x00 /* /< Active Scan Mode */
+#define LOCKED_HOVER 0x01 /* /< Hover Scan Mode */
+#define LOCKED_IDLE 0x02 /* /< Idle Scan Mode */
+#define LOCKED_ONLY_SELF 0x03 /* /< Only Self Sense scan mode */
+#define LOCKED_ONLY_MUTUAL_0 0x04 /* /< Only Mutual Sense scan mode at
+ * lowest freq */
+#define LOCKED_ONLY_MUTUAL_1 0x05 /* /< Only Mutual Sense scan mode */
+#define LOCKED_ONLY_MUTUAL_2 0x06 /* /< Only Mutual Sense scan mode */
+#define LOCKED_ONLY_MUTUAL_3 0x07 /* /< Only Mutual Sense scan mode at
+ * highest freq */
+#define LOCKED_LP_DETECT 0x10 /* /< Low Power SS */
+#define LOCKED_LP_ACTIVE 0x11 /* /< Low Power MS */
+/** @}*/
+
+/* FEATURE SELECT OPTION (0xA2) */
+/* Feature Selection */
+/** @defgroup feat_opt Feature Selection Option
+ * @ingroup feat_sel
+ * System defined features that can be enable/disable
+ * @{
+ */
+#define FEAT_SEL_GLOVE 0x00 /* /< Glove Mode */
+#define FEAT_SEL_COVER 0x01 /* /< Cover Mode */
+#define FEAT_SEL_CHARGER 0x02 /* /< Charger Mode */
+#define FEAT_SEL_GESTURE 0x03 /* /< Gesture Mode */
+#define FEAT_SEL_GRIP 0x04 /* /< Grip Detection */
+#define FEAT_SEL_STYLUS 0x07 /* /< Stylus Mode
+ * (this is a driver define, not
+ * available in FW) */
+/** @}*/
+
+/* Feature Settings */
+#define FEAT_ENABLE 1 /* /< General value to enable a feature
+ * */
+#define FEAT_DISABLE 0 /* /< General value to disable a feature
+ * */
+
+
+/* Charger */
+/** @defgroup charger_opt Charger Mode Option
+ * @ingroup feat_sel
+ * Option for Charger Mode, it is a bitmask where the each bit indicate a
+ * different kind of chager
+ * @{
+ */
+#define CHARGER_CABLE 0x01 /* /< normal usb charger */
+#define CHARGER_WIRLESS 0x02 /* /< wireless charger */
+/** @}*/
+
+/* Gestures */
+/** @defgroup gesture_opt Gesture Mode Option
+ * @ingroup feat_sel
+ * Gesture IDs of the predefined gesture recognized by the fw.
+ * The ID represent also the position of the corresponding bit in the gesture
+ * mask
+ * @{
+ */
+#define GEST_ID_UP_1F 0x01 /* /< Bottom to Top line */
+#define GEST_ID_DOWN_1F 0x02 /* /< Top to bottom line */
+#define GEST_ID_LEFT_1F 0x03 /* /< Right to left line */
+#define GEST_ID_RIGHT_1F 0x04 /* /< Left to right line */
+#define GEST_ID_DBLTAP 0x05 /* /< Double Tap */
+#define GEST_ID_O 0x06 /* /< 'O' */
+#define GEST_ID_C 0x07 /* /< 'C' */
+#define GEST_ID_M 0x08 /* /< 'M' */
+#define GEST_ID_W 0x09 /* /< 'W' */
+#define GEST_ID_E 0x0A /* /< 'e' */
+#define GEST_ID_L 0x0B /* /< 'L' */
+#define GEST_ID_F 0x0C /* /< 'F' */
+#define GEST_ID_V 0x0D /* /< 'V' */
+#define GEST_ID_AT 0x0E /* /< '@' */
+#define GEST_ID_S 0x0F /* /< 'S' */
+#define GEST_ID_Z 0x10 /* /< 'Z' */
+#define GEST_ID_LEFTBRACE 0x11 /* /< '<' */
+#define GEST_ID_RIGHTBRACE 0x12 /* /< '>' */
+#define GEST_ID_CARET 0x13 /* /< '^' */
+/** @}*/
+
+/* WRYTE SYSTEM COMMAND (0xA4) */
+/* System command */
+/** @defgroup sys_opt System Command Option
+ * @ingroup sys_cmd
+ * Valid System Command Parameters
+ * @{
+ */
+#define SYS_CMD_SPECIAL 0x00 /* /< Special Commands */
+#define SYS_CMD_INT 0x01 /* /< FW Interrupt Control */
+#define SYS_CMD_FORCE_CAL 0x02 /* /< Force Calibration */
+#define SYS_CMD_CX_TUNING 0x03 /* /< CX initialization */
+#define SYS_CMD_ITO 0x04 /* /< ITO test */
+#define SYS_CMD_SAVE_FLASH 0x05 /* /< Saving to flash */
+#define SYS_CMD_LOAD_DATA 0x06 /* /< Load Host data memory */
+#define SYS_CMD_SPECIAL_TUNING 0x08 /* /< Perform some special tuning */
+/** @} */
+
+/* System command settings */
+/* Special commands */
+/** @defgroup sys_special_opt Special Command Option
+ * @ingroup sys_cmd
+ * Valid special command
+ * @{
+ */
+#define SPECIAL_SYS_RESET 0x00 /* /< System Reset triggered by
+ * the FW */
+#define SPECIAL_FIFO_FLUSH 0x01 /* /< Flush of the FIFO */
+#define SPECIAL_PANEL_INIT 0x02 /* /< Panel Initialization */
+#define SPECIAL_FULL_PANEL_INIT 0x03 /* /< Full panel initialization
+ * */
+#define SPECIAL_WRITE_HOST_MEM_TO_FLASH 0x04 /* /< Write */
+/** @} */
+
+
+/* Force Cal and Cx auto tuning */
+/** @defgroup forcecal_cx_opt Force Cal and Tuning Option
+ * @ingroup sys_cmd
+ * Valid bitmask for triggering forcecal or performing manual autotune
+ * @{
+ */
+#define CAL_MS_TOUCH 0x01 /* /< Mutual Sense Touch */
+#define CAL_MS_LOW_POWER 0x02 /* /< Mutual Sense Touch in low power
+ * mode */
+#define CAL_SS_TOUCH 0x04 /* /< Self Sense Touch */
+#define CAL_SS_IDLE 0x08 /* /< Self Sense Touch in idle mode */
+#define CAL_MS_KEY 0x10 /* /< Mutual Sense Key */
+#define CAL_SS_KEY 0x20 /* /< Self Sense Key */
+#define CAL_MS_FORCE 0x40 /* /< Mutual Sense Force */
+#define CAL_SS_FORCE 0x80 /* /< Self Sense Force */
+/** @} */
+
+/* ITO checks (position of the bit in the mask) */
+/** @defgroup ito_opt ITO Test Option
+ * @ingroup sys_cmd
+ * Valid option for the ITO test
+ * @{
+ */
+#define ITO_FORCE_OPEN 0x00 /* /< Check if some force channels is
+ * open */
+#define ITO_SENSE_OPEN 0x01 /* /< Check if some sense channels is
+ * open */
+#define ITO_FORCE_GROUND 0x02 /* /< Check if some force channels is
+ * short to ground */
+#define ITO_SENSE_GROUND 0x03 /* /< Check if some sense channels is
+ * short to ground */
+#define ITO_FORCE_VDD 0x04 /* /< Check if some force channels is
+ * short to VDD */
+#define ITO_SENSE_VDD 0x05 /* /< Check if some sense channels is
+ * short to VDD */
+#define ITO_FORCE_FORCE 0x06 /* /< Check force to force channels */
+#define ITO_FORCE_SENSE 0x07 /* /< Check force to sense channels */
+#define ITO_SENSE_SENSE 0x08 /* /< Check sense to sense channels */
+#define ITO_KEY_FORCE_OPEN 0x09 /* /< Check if some force channels used
+ * for the key is open */
+#define ITO_KEY_SENSE_OPEN 0x0A /* /< Check if some sense channels used
+ * for the key is open */
+/** @}*/
+
+/* Save flash */
+/** @defgroup save_opt Save to Flash Option
+ * @ingroup sys_cmd
+ * Valid option for saving data to the Flash
+ * @{
+ */
+#define SAVE_FW_CONF 0x01 /* /< Save the confing to the flash */
+#define SAVE_CX 0x02 /* /< Save the CX to the flash */
+#define SAVE_PANEL_CONF 0x04 /* /< Save the Panel configuration to the flash
+ * */
+/** @}*/
+
+/* Load Data */
+/** @defgroup load_opt Load Host Data Option
+ * @ingroup sys_cmd
+ * Valid option to ask to the FW to load host data into the memory
+ * @{
+ */
+#define LOAD_SYS_INFO 0x01 /* /< Load System Info */
+#define LOAD_CX_MS_TOUCH 0x10 /* /< Load MS Init Data for
+ * Active Mode */
+#define LOAD_CX_MS_LOW_POWER 0x11 /* /< Load MS Init Data for Low
+ * Power Mode */
+#define LOAD_CX_SS_TOUCH 0x12 /* /< Load SS Init Data for
+ * Active Mode */
+#define LOAD_CX_SS_TOUCH_IDLE 0x13 /* /< Load SS Init Data for Low
+ * Power Mode */
+#define LOAD_CX_MS_KEY 0x14 /* /< Load MS Init Data for Key
+ * */
+#define LOAD_CX_SS_KEY 0x15 /* /< Load SS Init Data for Key
+ * */
+#define LOAD_CX_MS_FORCE 0x16 /* /< Load MS Init Data for
+ * Force */
+#define LOAD_CX_SS_FORCE 0x17 /* /< Load SS Init Data for
+ * Force */
+#define LOAD_SYNC_FRAME_RAW 0x30 /* /< Load a Synchronized Raw
+ * Frame */
+#define LOAD_SYNC_FRAME_FILTER 0x31 /* /< Load a Synchronized Filter
+ * Frame */
+#define LOAD_SYNC_FRAME_STRENGTH 0x33 /* /< Load a Synchronized
+ * Strength Frame */
+#define LOAD_SYNC_FRAME_BASELINE 0x32 /* /< Load a Synchronized
+ * Baseline Frame */
+#define LOAD_PANEL_CX_TOT_MS_TOUCH 0x50 /* /< Load TOT MS Init Data for
+ * Active Mode */
+#define LOAD_PANEL_CX_TOT_MS_LOW_POWER 0x51 /* /< Load TOT MS Init Data for
+ * Low Power Mode */
+#define LOAD_PANEL_CX_TOT_SS_TOUCH 0x52 /* /< Load TOT SS Init Data for
+ * Active Mode */
+#define LOAD_PANEL_CX_TOT_SS_TOUCH_IDLE 0x53 /* /< Load TOT SS Init Data for
+ * Low Power Mode */
+#define LOAD_PANEL_CX_TOT_MS_KEY 0x54 /* /< Load TOT MS Init Data for
+ * Key */
+#define LOAD_PANEL_CX_TOT_SS_KEY 0x55 /* /< Load TOT SS Init Data for
+ * Key */
+#define LOAD_PANEL_CX_TOT_MS_FORCE 0x56 /* /< Load TOT MS Init Data for
+ * Force */
+#define LOAD_PANEL_CX_TOT_SS_FORCE 0x57 /* /< Load TOT SS Init Data for
+ * Force */
+#define LOAD_SENS_CAL_COEFF 0xC0 /* /< Load Sesitivity
+ * Calibration Coefficients */
+/** @}*/
+
+/* Special Tuning */
+/** @defgroup spcl_tun_opt Special Tuning Option
+ * @ingroup sys_cmd
+ * Valid special tuning operations which the fw can perform (bitmask)
+ * @{
+ */
+#define SPECIAL_TUNING_LP_TIMER 0x01 /* /< Perform LP Timer calibration */
+#define SPECIAL_TUNING_IOFF 0x02 /* /< Perform Ioff calibration */
+
+/** @}*/
+
+/* EVENT ID */
+/** @defgroup events_group FW Event IDs and Types
+ * Event IDs and Types pushed by the FW into the FIFO
+ * @{
+ */
+#define EVT_ID_NOEVENT 0x00 /* /< No Events */
+#define EVT_ID_CONTROLLER_READY 0x03 /* /< Controller ready, issued after a
+ * system reset. */
+#define EVT_ID_ENTER_POINT 0x13 /* /< Touch enter in the sensing area */
+#define EVT_ID_MOTION_POINT 0x23 /* /< Touch motion (a specific touch
+ * changed position) */
+#define EVT_ID_LEAVE_POINT 0x33 /* /< Touch leave the sensing area */
+#define EVT_ID_STATUS_UPDATE 0x43 /* /< FW report a system condition
+ * change */
+#define EVT_ID_USER_REPORT 0x53 /* /< User related events triggered
+ * (keys, gestures, proximity etc) */
+#define EVT_ID_DEBUG 0xE3 /* /< Debug Info */
+#define EVT_ID_ERROR 0xF3 /* /< Error Event */
+
+/* /< Max number of unique event IDs supported */
+#define NUM_EVT_ID (((EVT_ID_ERROR & 0xF0) >> 4) + 1)
+
+/** @}*/
+
+/* STATUS TYPE */
+/** @defgroup status_type Status Event Types
+ * @ingroup events_group
+ * Types of EVT_ID_STATUS_UPDATE events
+ * @{
+ */
+#define EVT_TYPE_STATUS_ECHO 0x01 /* /< Echo event,
+ * contain the first 5 bytes of
+ * the FW command sent */
+#define EVT_TYPE_STATUS_FRAME_DROP 0x03 /* /< Some frame was skipped
+ * during the elaboration */
+#define EVT_TYPE_STATUS_FORCE_CAL 0x05 /* /< Force Calibration has
+ * triggered */
+#define EVT_TYPE_STATUS_WATER 0x06 /* /< Water Mode */
+#define EVT_TYPE_STATUS_SS_RAW_SAT 0x07 /* /< Self Sense data saturated
+ * */
+/** @} */
+
+/* USER TYPE */
+/** @defgroup user_type User Event Types
+ * @ingroup events_group
+ * Types of EVT_ID_USER_REPORT events generated by thw FW
+ * @{
+ */
+#define EVT_TYPE_USER_KEY 0x00 /* /< Keys pressed/relesed event report
+ * */
+#define EVT_TYPE_USER_PROXIMITY 0x01 /* /< Proximity detection event report
+ * */
+#define EVT_TYPE_USER_GESTURE 0x02 /* /< Gesture detection event report */
+/** @}*/
+
+/* ERROR TYPE */
+/** @defgroup error_type Error Event Types
+ * @ingroup events_group
+ * Types of EVT_ID_ERROR events reported by the FW
+ * @{
+ */
+#define EVT_TYPE_ERROR_WATCHDOG 0x06 /* /< Watchdog timer expired */
+
+#define EVT_TYPE_ERROR_CRC_CFG_HEAD 0x20 /* /< CRC error in the Config
+ * Area Header */
+#define EVT_TYPE_ERROR_CRC_CFG 0x21 /* /< CRC error in the Config
+ * Area */
+#define EVT_TYPE_ERROR_CRC_PANEL_HEAD 0x22 /* /< CRC error in the Panel
+ * Area Header */
+#define EVT_TYPE_ERROR_CRC_PANEL 0x23 /* /< CRC error in the Panel
+ * Area */
+
+#define EVT_TYPE_ERROR_ITO_FORCETOGND 0x60 /* /< Force channel/s short to
+ * ground */
+#define EVT_TYPE_ERROR_ITO_SENSETOGND 0x61 /* /< Sense channel/s short to
+ * ground */
+#define EVT_TYPE_ERROR_ITO_FORCETOVDD 0x62 /* /< Force channel/s short to
+ * VDD */
+#define EVT_TYPE_ERROR_ITO_SENSETOVDD 0x63 /* /< Sense channel/s short to
+ * VDD */
+#define EVT_TYPE_ERROR_ITO_FORCE_P2P 0x64 /* /< Pin to Pin short Force
+ * channel/s */
+#define EVT_TYPE_ERROR_ITO_SENSE_P2P 0x65 /* /< Pin to Pin short Sense
+ * channel/s */
+#define EVT_TYPE_ERROR_ITO_FORCEOPEN 0x66 /* /< Force Panel open */
+#define EVT_TYPE_ERROR_ITO_SENSEOPEN 0x67 /* /< Sense Panel open */
+#define EVT_TYPE_ERROR_ITO_KEYOPEN 0x68 /* /< Key open */
+
+#define EVT_TYPE_ERROR_CRC_CX_HEAD 0xA0 /* /< CRC error in the CX Area
+ * Header */
+#define EVT_TYPE_ERROR_CRC_CX 0xA1 /* /< CRC error in the CX Area
+ * */
+#define EVT_TYPE_ERROR_FLASH_FAILED 0xA4 /* Flash error, cause unknown */
+#define EVT_TYPE_ERROR_CRC_CX_SUB_HEAD 0xA5 /* /< CRC error in the CX
+ * Subsection Area Header */
+#define EVT_TYPE_ERROR_CRC_CX_SUB 0xA6 /* /< CRC error in the CX
+ * Subsection Area */
+
+#define EVT_TYPE_ERROR_ESD 0xF0 /* /< ESD error */
+/** @}*/
+
+
+/** @defgroup address Chip Address
+ * Collection of HW and SW Addresses useful to collect different kind of data
+ * @{
+ */
+
+/** @defgroup config_adr SW Address
+ * @ingroup address
+ * Important addresses of data stored into Config memory (and sometimes their
+ * dimensions)
+ * @{
+ */
+#define ADDR_CONFIG_ID 0x0010 /* /< Starting Address of the config ID
+ * */
+#define CONFIG_ID_BYTE 2 /* /< Number of bytes of config ID */
+#define ADDR_CONFIG_SENSE_LEN 0x0030 /* /< Address where is stored the number
+ * of sense channels */
+#define ADDR_CONFIG_FORCE_LEN 0x0031 /* /< Address where is stored the number
+ * of force channels */
+#define ADDR_CONFIG_AUTOCAL 0x0040 /* /< Address where is stored the Auto
+ * Calibration register */
+#define ADDR_CONFIG_T_CYCLE 0x00ED /* /< Address where is stored the MS T
+ * cycle */
+#define ADDR_CONFIG_MNM 0x01F0 /* /< Address where is stored the MNM
+ * register */
+#define ADDR_CONFIG_MRN 0x01F1 /* /< Address where is stored the MRN
+ * register */
+#define ADDR_CONFIG_R0_CYCLE 0x01F2 /* /< Address where is stored the first
+ * R cycle */
+/** @}*/
+
+/** @}*/
+
+/* ERROR INFO */
+#define ERROR_DUMP_ROW_SIZE 32 /* /< number of rows of the error memory
+ * */
+#define ERROR_DUMP_COL_SIZE 4 /* /< number of bytes for each row of
+ * the error memory */
+#define ERROR_DUMP_SIGNATURE 0xFA5005AF /* /< first row signature of a
+ * proper dump */
+
+/* Touch Types */
+#define TOUCH_TYPE_INVALID 0x00 /* /< Invalid touch type */
+#define TOUCH_TYPE_FINGER 0x01 /* /< Finger touch */
+#define TOUCH_TYPE_GLOVE 0x02 /* /< Glove touch */
+#define TOUCH_TYPE_STYLUS 0x03 /* /< Stylus touch */
+#define TOUCH_TYPE_PALM 0x04 /* /< Palm touch */
+#define TOUCH_TYPE_HOVER 0x05 /* /< Hovering touch */
+
+/* Keys code */
+#define FTS_KEY_0 0x01 /* /< Key 0 bit */
+#define FTS_KEY_1 0x02 /* /< Key 1 bit */
+#define FTS_KEY_2 0x04 /* /< Key 2 bit */
+#define FTS_KEY_3 0x08 /* /< Key 3 bit */
+#define FTS_KEY_4 0x10 /* /< Key 4 bit */
+#define FTS_KEY_5 0x20 /* /< Key 5 bit */
+#define FTS_KEY_6 0x40 /* /< Key 6 bit */
+#define FTS_KEY_7 0x80 /* /< Key 7 bit */
+
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsTest.c b/drivers/input/touchscreen/stm/fts_lib/ftsTest.c
new file mode 100644
index 0000000..f0dedb8
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsTest.c
@@ -0,0 +1,5268 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS API for MP test *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsTest.c
+ * \brief Contains all the functions related to the Mass Production Test
+ */
+
+#include "ftsCompensation.h"
+#include "ftsCore.h"
+#include "ftsError.h"
+#include "ftsFrame.h"
+#include "ftsHardware.h"
+#include "ftsIO.h"
+#include "ftsSoftware.h"
+#include "ftsTest.h"
+#include "ftsTime.h"
+#include "ftsTool.h"
+#include "../fts.h"
+
+
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <stdarg.h>
+#include <linux/serio.h>
+#include <linux/time.h>
+#include <linux/delay.h>
+#include <linux/ctype.h>
+#include <linux/firmware.h>
+
+
+#ifdef LIMITS_H_FILE
+#include "../fts_limits.h"
+#endif
+
+
+TestToDo tests; /* /< global variable that specify the tests to perform during
+ * the Mass Production Test */
+static LimitFile limit_file; /* /< variable which contains the limit file
+ * during test */
+
+/**
+ * Initialize the testToDo variable with the default tests to perform during
+ * the Mass Production Test
+ * @return OK
+ */
+int initTestToDo(void)
+{
+ /*** Initialize Limit File ***/
+ limit_file.size = 0;
+ limit_file.data = NULL;
+ strlcpy(limit_file.name, " ", MAX_LIMIT_FILE_NAME);
+
+
+ tests.MutualRawAdjITO = 1;
+
+ tests.MutualRaw = 1;
+ tests.MutualRawGap = 0;
+ tests.MutualRawAdj = 0;
+
+ tests.MutualRawLP = 1;
+ tests.MutualRawGapLP = 0;
+ tests.MutualRawAdjLP = 0;
+
+ tests.MutualCx1 = 0;
+ tests.MutualCx2 = 0;
+ tests.MutualCx2Adj = 0;
+ tests.MutualCxTotal = 0;
+ tests.MutualCxTotalAdj = 0;
+
+#ifdef PHONE_KEY
+ tests.MutualKeyRaw = 1;
+#else
+ tests.MutualKeyRaw = 0;
+#endif
+ tests.MutualKeyCx1 = 0;
+ tests.MutualKeyCx2 = 0;
+#ifdef PHONE_KEY
+ tests.MutualKeyCxTotal = 0;
+#else
+ tests.MutualKeyCxTotal = 0;
+#endif
+
+ tests.SelfForceRaw = 1;
+ tests.SelfForceRawGap = 0;
+
+ tests.SelfForceRawLP = 1;
+ tests.SelfForceRawGapLP = 0;
+
+ tests.SelfForceIx1 = 0;
+ tests.SelfForceIx2 = 0;
+ tests.SelfForceIx2Adj = 0;
+ tests.SelfForceIxTotal = 0;
+ tests.SelfForceIxTotalAdj = 0;
+ tests.SelfForceCx1 = 0;
+ tests.SelfForceCx2 = 0;
+ tests.SelfForceCx2Adj = 0;
+ tests.SelfForceCxTotal = 0;
+ tests.SelfForceCxTotalAdj = 0;
+
+ tests.SelfSenseRaw = 1;
+ tests.SelfSenseRawGap = 0;
+
+ tests.SelfSenseRawLP = 1;
+ tests.SelfSenseRawGapLP = 0;
+
+ tests.SelfSenseIx1 = 0;
+ tests.SelfSenseIx2 = 0;
+ tests.SelfSenseIx2Adj = 0;
+ tests.SelfSenseIxTotal = 0;
+ tests.SelfSenseIxTotalAdj = 0;
+ tests.SelfSenseCx1 = 0;
+ tests.SelfSenseCx2 = 0;
+ tests.SelfSenseCx2Adj = 0;
+ tests.SelfSenseCxTotal = 0;
+ tests.SelfSenseCxTotalAdj = 0;
+
+ return OK;
+}
+
+/**
+ * Compute the Horizontal adjacent matrix doing the abs of the difference
+ * between the column i with the i-1 one. \n
+ * Both the original data matrix and the adj matrix are disposed as 1 dimension
+ * array one row after the other \n
+ * The resulting matrix has one column less than the starting original one \n
+ * @param data pointer to the array of signed bytes containing the original
+ * data
+ * @param row number of rows of the original data
+ * @param column number of columns of the original data
+ * @param result pointer of a pointer to an array of unsigned bytes which will
+ * contain the adj matrix
+ * @return OK if success or an error code which specify the type of error
+ */
+int computeAdjHoriz(i8 *data, int row, int column, u8 **result)
+{
+ int i, j;
+ int size = row * (column - 1);
+
+ if (column < 2) {
+ pr_err("computeAdjHoriz: ERROR %08X\n", ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ *result = (u8 *)kmalloc(size * sizeof(u8), GFP_KERNEL);
+ if (*result == NULL) {
+ pr_err("computeAdjHoriz: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ for (i = 0; i < row; i++)
+ for (j = 1; j < column; j++)
+ *(*result + (i * (column - 1) + (j - 1))) =
+ abs(data[i * column + j] - data[i * column +
+ (j - 1)]);
+
+ return OK;
+}
+
+/**
+ * Compute the Horizontal adjacent matrix of short values doing the abs of
+ * the difference between the column i with the i-1 one.
+ * Both the original data matrix and the adj matrix are disposed as 1 dimension
+ * array one row after the other \n
+ * The resulting matrix has one column less than the starting original one \n
+ * @param data pointer to the array of signed bytes containing the original
+ * data
+ * @param row number of rows of the original data
+ * @param column number of columns of the original data
+ * @param result pointer of a pointer to an array of unsigned bytes which
+ * will contain the adj matrix
+ * @return OK if success or an error code which specify the type of error
+ */
+int computeAdjHorizTotal(short *data, int row, int column, u16 **result)
+{
+ int i, j;
+ int size = row * (column - 1);
+
+ if (column < 2) {
+ pr_err("computeAdjHorizTotal: ERROR %08X\n",
+ ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ *result = (u16 *)kmalloc(size * sizeof(u16), GFP_KERNEL);
+ if (*result == NULL) {
+ pr_err("computeAdjHorizTotal: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ for (i = 0; i < row; i++)
+ for (j = 1; j < column; j++)
+ *(*result + (i * (column - 1) + (j - 1))) =
+ abs(data[i * column + j] - data[i * column +
+ (j - 1)]);
+
+ return OK;
+}
+
+/**
+ * Compute the Vertical adjacent matrix doing the abs of the difference between
+ * the row i with the i-1 one.
+ * Both the original data matrix and the adj matrix are disposed as 1 dimension
+ * array one row after the other. \n
+ * The resulting matrix has one column less than the starting original one \n
+ * @param data pointer to the array of signed bytes containing the original
+ * data
+ * @param row number of rows of the original data
+ * @param column number of columns of the original data
+ * @param result pointer of a pointer to an array of unsigned bytes which will
+ * contain the adj matrix
+ * @return OK if success or an error code which specify the type of error
+ */
+int computeAdjVert(i8 *data, int row, int column, u8 **result)
+{
+ int i, j;
+ int size = (row - 1) * (column);
+
+ if (row < 2) {
+ pr_err("computeAdjVert: ERROR %08X\n", ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ *result = (u8 *)kmalloc(size * sizeof(u8), GFP_KERNEL);
+ if (*result == NULL) {
+ pr_err("computeAdjVert: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ for (i = 1; i < row; i++)
+ for (j = 0; j < column; j++)
+ *(*result + ((i - 1) * column + j)) =
+ abs(data[i * column + j] - data[(i - 1) *
+ column + j]);
+
+ return OK;
+}
+
+/**
+ * Compute the Vertical adjacent matrix of short values doing the abs of
+ * the difference between the row i with the i-1 one.
+ * Both the original data matrix and the adj matrix are disposed as 1 dimension
+ * array one row after the other. \n
+ * The resulting matrix has one column less than the starting original one \n
+ * @param data pointer to the array of signed bytes containing the original
+ * data
+ * @param row number of rows of the original data
+ * @param column number of columns of the original data
+ * @param result pointer of a pointer to an array of unsigned bytes which will
+ * contain the adj matrix
+ * @return OK if success or an error code which specify the type of error
+ */
+int computeAdjVertTotal(short *data, int row, int column, u16 **result)
+{
+ int i, j;
+ int size = (row - 1) * (column);
+
+ if (row < 2) {
+ pr_err("computeAdjVertTotal: ERROR %08X\n", ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ *result = (u16 *)kmalloc(size * sizeof(u16), GFP_KERNEL);
+ if (*result == NULL) {
+ pr_err("computeAdjVertTotal: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ for (i = 1; i < row; i++)
+ for (j = 0; j < column; j++)
+ *(*result + ((i - 1) * column + j)) =
+ abs(data[i * column + j] - data[(i - 1) *
+ column + j]);
+
+ return OK;
+}
+
+/**
+ * Compute the Horizontal adjacent matrix doing the abs of the difference
+ * between the column i with the i-1 one. \n
+ * Both the original data matrix and the adj matrix are disposed as 1 dimension
+ * array one row after the other \n
+ * The resulting matrix has one column less than the starting original one \n
+ * @param data pointer to the array of unsigned bytes containing the original
+ * data
+ * @param row number of rows of the original data
+ * @param column number of columns of the original data
+ * @param result pointer of a pointer to an array of unsigned bytes which will
+ * contain the adj matrix
+ * @return OK if success or an error code which specify the type of error
+ */
+int computeAdjHorizFromU(u8 *data, int row, int column, u8 **result)
+{
+ int i, j;
+ int size = row * (column - 1);
+
+ if (column < 2) {
+ pr_err("computeAdjHoriz: ERROR %08X\n", ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ *result = (u8 *)kmalloc(size * sizeof(u8), GFP_KERNEL);
+ if (*result == NULL) {
+ pr_err("computeAdjHoriz: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ for (i = 0; i < row; i++)
+ for (j = 1; j < column; j++)
+ *(*result + (i * (column - 1) + (j - 1))) =
+ abs(data[i * column + j] - data[i * column +
+ (j - 1)]);
+
+ return OK;
+}
+
+/**
+ * Compute the Horizontal adjacent matrix of u16 values doing the abs of
+ * the difference between the column i with the i-1 one.
+ * Both the original data matrix and the adj matrix are disposed as 1 dimension
+ * array one row after the other \n
+ * The resulting matrix has one column less than the starting original one \n
+ * @param data pointer to the array of unsigned bytes containing the original
+ * data
+ * @param row number of rows of the original data
+ * @param column number of columns of the original data
+ * @param result pointer of a pointer to an array of unsigned bytes which will
+ * contain the adj matrix
+ * @return OK if success or an error code which specify the type of error
+ */
+int computeAdjHorizTotalFromU(u16 *data, int row, int column, u16 **result)
+{
+ int i, j;
+ int size = row * (column - 1);
+
+ if (column < 2) {
+ pr_err("computeAdjHorizTotal: ERROR %08X\n",
+ ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ *result = (u16 *)kmalloc(size * sizeof(u16), GFP_KERNEL);
+ if (*result == NULL) {
+ pr_err("computeAdjHorizTotal: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ for (i = 0; i < row; i++)
+ for (j = 1; j < column; j++)
+ *(*result + (i * (column - 1) + (j - 1))) =
+ abs(data[i * column + j] - data[i * column +
+ (j - 1)]);
+
+ return OK;
+}
+
+/**
+ * Compute the Vertical adjacent matrix doing the abs of the difference between
+ * the row i with the i-1 one.
+ * Both the original data matrix and the adj matrix are disposed as 1 dimension
+ * array one row after the other. \n
+ * The resulting matrix has one column less than the starting original one \n
+ * @param data pointer to the array of unsigned bytes containing the original
+ * data
+ * @param row number of rows of the original data
+ * @param column number of columns of the original data
+ * @param result pointer of a pointer to an array of unsigned bytes which will
+ * contain the adj matrix
+ * @return OK if success or an error code which specify the type of error
+ */
+int computeAdjVertFromU(u8 *data, int row, int column, u8 **result)
+{
+ int i, j;
+ int size = (row - 1) * (column);
+
+ if (row < 2) {
+ pr_err("computeAdjVert: ERROR %08X\n", ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ *result = (u8 *)kmalloc(size * sizeof(u8), GFP_KERNEL);
+ if (*result == NULL) {
+ pr_err("computeAdjVert: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ for (i = 1; i < row; i++)
+ for (j = 0; j < column; j++)
+ *(*result + ((i - 1) * column + j)) =
+ abs(data[i * column + j] - data[(i - 1) *
+ column + j]);
+
+ return OK;
+}
+
+/**
+ * Compute the Vertical adjacent matrix of u16 values doing the abs of
+ * the difference between the row i with the i-1 one.
+ * Both the original data matrix and the adj matrix are disposed as 1 dimension
+ * array one row after the other. \n
+ * The resulting matrix has one column less than the starting original one \n
+ * @param data pointer to the array of unsigned bytes containing the original
+ * data
+ * @param row number of rows of the original data
+ * @param column number of columns of the original data
+ * @param result pointer of a pointer to an array of unsigned bytes which will
+ * contain the adj matrix
+ * @return OK if success or an error code which specify the type of error
+ */
+int computeAdjVertTotalFromU(u16 *data, int row, int column, u16 **result)
+{
+ int i, j;
+ int size = (row - 1) * (column);
+
+ if (row < 2) {
+ pr_err("computeAdjVertTotal: ERROR %08X\n", ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ *result = (u16 *)kmalloc(size * sizeof(u16), GFP_KERNEL);
+ if (*result == NULL) {
+ pr_err("computeAdjVertTotal: ERROR %08X\n", ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+
+ for (i = 1; i < row; i++)
+ for (j = 0; j < column; j++)
+ *(*result + ((i - 1) * column + j)) =
+ abs(data[i * column + j] - data[(i - 1) *
+ column + j]);
+
+ return OK;
+}
+
+
+
+/**
+ * Check that each value of a matrix of short doesn't exceed a min and a Max
+ * value
+ * (these values are included in the interval). \n
+ * The matrix is stored as 1 dimension array one row after the other. \n
+ * @param data pointer to the array of short containing the data to check
+ * @param row number of rows of data
+ * @param column number of columns of data
+ * @param min minimum value allowed
+ * @param max Maximum value allowed
+ * @return the number of elements that overcome the specified interval (0 = OK)
+ */
+int checkLimitsMinMax(short *data, int row, int column, int min, int max)
+{
+ int i, j;
+ int count = 0;
+
+ for (i = 0; i < row; i++) {
+ for (j = 0; j < column; j++) {
+ if (data[i * column + j] < min || data[i * column + j] >
+ max) {
+ pr_err("checkLimitsMinMax: Node[%d,%d] = %d exceed limit [%d, %d]\n",
+ i, j, data[i * column + j], min, max);
+ count++;
+ }
+ }
+ }
+
+ return count; /* if count is 0 = OK, test completed successfully */
+}
+
+/**
+ * Check that the difference between the max and min of a matrix of short
+ * is less or equal to a threshold.\n
+ * The matrix is stored as 1 dimension array one row after the other.
+ * @param data pointer to the array of short containing the data to check
+ * @param row number of rows of data
+ * @param column number of columns of data
+ * @param threshold threshold value allowed
+ * @return OK if the difference is <= to threshold otherwise
+ * ERROR_TEST_CHECK_FAIL
+ */
+int checkLimitsGap(short *data, int row, int column, int threshold)
+{
+ int i, j;
+ int min_node;
+ int max_node;
+
+ if (row == 0 || column == 0) {
+ pr_err("checkLimitsGap: invalid number of rows = %d or columns = %d ERROR %08X\n",
+ row, column, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ min_node = data[0];
+ max_node = data[0];
+
+ for (i = 0; i < row; i++) {
+ for (j = 0; j < column; j++) {
+ if (data[i * column + j] < min_node)
+ min_node = data[i * column + j];
+ else if (data[i * column + j] > max_node)
+ max_node = data[i * column + j];
+ }
+ }
+
+ if (max_node - min_node > threshold) {
+ pr_err("checkLimitsGap: GAP = %d exceed limit %d\n",
+ max_node - min_node, threshold);
+ return ERROR_TEST_CHECK_FAIL;
+ } else
+ return OK;
+}
+
+/**
+ * Check that each value of a matrix of i8 doesn't exceed a specific min and
+ * Max value set for each node (these values are included in the interval). \n
+ * The matrixes of data, min and max values are stored as 1 dimension arrays
+ * one row after the other.
+ * @param data pointer to the array of short containing the data to check
+ * @param row number of rows of data
+ * @param column number of columns of data
+ * @param min pointer to a matrix which specify the minimum value allowed for
+ * each node
+ * @param max pointer to a matrix which specify the Maximum value allowed for
+ * each node
+ * @return the number of elements that overcome the specified interval (0 = OK)
+ */
+int checkLimitsMap(i8 *data, int row, int column, int *min, int *max)
+{
+ int i, j;
+ int count = 0;
+
+ for (i = 0; i < row; i++) {
+ for (j = 0; j < column; j++) {
+ if (data[i * column + j] < min[i * column + j] ||
+ data[i * column + j] > max[i * column + j]) {
+ pr_err("checkLimitsMap: Node[%d,%d] = %d exceed limit [%d, %d]\n",
+ i, j, data[i * column + j],
+ min[i * column + j],
+ max[i * column + j]);
+ count++;
+ }
+ }
+ }
+
+ return count; /* if count is 0 = OK, test completed successfully */
+}
+
+/**
+ * Check that each value of a matrix of short doesn't exceed a specific min and
+ * Max value set for each node (these values are included in the interval).
+ * The matrixes of data, min and max values are stored as 1 dimension arrays
+ * one row after the other.
+ * @param data pointer to the array of short containing the data to check
+ * @param row number of rows of data
+ * @param column number of columns of data
+ * @param min pointer to a matrix which specify the minimum value allowed for
+ * each node
+ * @param max pointer to a matrix which specify the Maximum value allowed for
+ * each node
+ * @return the number of elements that overcome the specified interval (0 = OK)
+ */
+int checkLimitsMapTotal(short *data, int row, int column, int *min, int *max)
+{
+ int i, j;
+ int count = 0;
+
+ for (i = 0; i < row; i++) {
+ for (j = 0; j < column; j++) {
+ if (data[i * column + j] < min[i * column + j] ||
+ data[i * column + j] > max[i * column + j]) {
+ pr_err("checkLimitsMapTotal: Node[%d,%d] = %d exceed limit [%d, %d]\n",
+ i, j, data[i * column + j],
+ min[i * column + j],
+ max[i * column + j]);
+ count++;
+ }
+ }
+ }
+
+ return count; /* if count is 0 = OK, test completed successfully */
+}
+
+/**
+ * Check that each value of a matrix of u8 doesn't exceed a specific min and
+ * Max value set for each node (these values are included in the interval). \n
+ * The matrixes of data, min and max values are stored as 1 dimension arrays
+ * one row after the other.
+ * @param data pointer to the array of short containing the data to check
+ * @param row number of rows of data
+ * @param column number of columns of data
+ * @param min pointer to a matrix which specify the minimum value allowed for
+ * each node
+ * @param max pointer to a matrix which specify the Maximum value allowed for
+ * each node
+ * @return the number of elements that overcome the specified interval (0 = OK)
+ */
+int checkLimitsMapFromU(u8 *data, int row, int column, int *min, int *max)
+{
+ int i, j;
+ int count = 0;
+
+ for (i = 0; i < row; i++) {
+ for (j = 0; j < column; j++) {
+ if (data[i * column + j] < min[i * column + j] ||
+ data[i * column + j] > max[i * column + j]) {
+ pr_err("checkLimitsMap: Node[%d,%d] = %d exceed limit [%d, %d]\n",
+ i, j, data[i * column + j],
+ min[i * column + j],
+ max[i * column + j]);
+ count++;
+ }
+ }
+ }
+
+ return count; /* if count is 0 = OK, test completed successfully */
+}
+
+/**
+ * Check that each value of a matrix of u16 doesn't exceed a specific min and
+ * Max value set for each node (these values are included in the interval).
+ * The matrixes of data, min and max values are stored as 1 dimension arrays
+ * one row after the other.
+ * @param data pointer to the array of short containing the data to check
+ * @param row number of rows of data
+ * @param column number of columns of data
+ * @param min pointer to a matrix which specify the minimum value allowed for
+ * each node
+ * @param max pointer to a matrix which specify the Maximum value allowed for
+ * each node
+ * @return the number of elements that overcome the specified interval (0 = OK)
+ */
+int checkLimitsMapTotalFromU(u16 *data, int row, int column, int *min, int *max)
+{
+ int i, j;
+ int count = 0;
+
+ for (i = 0; i < row; i++) {
+ for (j = 0; j < column; j++) {
+ if (data[i * column + j] < min[i * column + j] ||
+ data[i * column + j] > max[i * column + j]) {
+ pr_err("checkLimitsMapTotal: Node[%d,%d] = %d exceed limit [%d, %d]\n",
+ i, j, data[i * column + j],
+ min[i * column + j],
+ max[i * column + j]);
+ count++;
+ }
+ }
+ }
+
+ return count; /* if count is 0 = OK, test completed successfully */
+}
+
+/**
+ * Check that each value of a matrix of u8 doesn't exceed a specific Max value
+ * set for each node (max value is included in the interval).
+ * The matrixes of data and max values are stored as 1 dimension arrays one row
+ * after the other.
+ * @param data pointer to the array of short containing the data to check
+ * @param row number of rows of data
+ * @param column number of columns of data
+ * @param max pointer to a matrix which specify the Maximum value allowed for
+ * each node
+ * @return the number of elements that overcome the specified interval (0 = OK)
+ */
+int checkLimitsMapAdj(u8 *data, int row, int column, int *max)
+{
+ int i, j;
+ int count = 0;
+
+ for (i = 0; i < row; i++) {
+ for (j = 0; j < column; j++) {
+ if (data[i * column + j] > max[i * column + j]) {
+ pr_err("checkLimitsMapAdj: Node[%d,%d] = %d exceed limit > %d\n",
+ i, j,
+ data[i * column + j],
+ max[i * column + j]);
+ count++;
+ }
+ }
+ }
+
+ return count; /* if count is 0 = OK, test completed successfully */
+}
+
+/**
+ * Check that each value of a matrix of u16 doesn't exceed a specific Max value
+ * set for each node (max value is included in the interval).
+ * The matrixes of data and max values are stored as 1 dimension arrays one row
+ * after the other.
+ * @param data pointer to the array of short containing the data to check
+ * @param row number of rows of data
+ * @param column number of columns of data
+ * @param max pointer to a matrix which specify the Maximum value allowed for
+ * each node
+ * @return the number of elements that overcome the specified interval (0 = OK)
+ */
+int checkLimitsMapAdjTotal(u16 *data, int row, int column, int *max)
+{
+ int i, j;
+ int count = 0;
+
+ for (i = 0; i < row; i++) {
+ for (j = 0; j < column; j++) {
+ if (data[i * column + j] > max[i * column + j]) {
+ pr_err("checkLimitsMapAdjTotal: Node[%d,%d] = %d exceed limit > %d\n",
+ i, j,
+ data[i * column + j],
+ max[i * column + j]);
+ count++;
+ }
+ }
+ }
+
+ return count; /* if count is 0 = OK, test completed successfully */
+}
+
+/**
+ * Perform an ITO test setting all the possible options
+ * (see @link ito_opt ITO Options @endlink) and checking MS Raw ADJ if enabled
+ * @param path_limits name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param todo pointer to a TestToDo variable which select the test to do
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_ito(char *path_limits, TestToDo *todo)
+{
+ int res = OK;
+ u8 sett[2] = { 0x00, 0x00 };
+ MutualSenseFrame msRawFrame;
+ int *thresholds = NULL;
+ u16 *adj = NULL;
+ int trows, tcolumns;
+
+ pr_info("ITO Production test is starting...\n");
+
+ res = fts_system_reset();
+ if (res < 0) {
+ pr_err("%s: ERROR %08X\n", __func__, ERROR_PROD_TEST_ITO);
+ return res | ERROR_PROD_TEST_ITO;
+ }
+
+ sett[0] = SPECIAL_TUNING_IOFF;
+ pr_info("Trimming Ioff...\n");
+ res = writeSysCmd(SYS_CMD_SPECIAL_TUNING, sett, 2);
+ if (res < OK) {
+ pr_err("production_test_ito: Trimm Ioff ERROR %08X\n",
+ (res | ERROR_PROD_TEST_ITO));
+ return res | ERROR_PROD_TEST_ITO;
+ }
+
+ sett[0] = 0xFF;
+ sett[1] = 0xFF;
+ pr_info("ITO Check command sent...\n");
+ res = writeSysCmd(SYS_CMD_ITO, sett, 2);
+ if (res < OK) {
+ pr_err("production_test_ito: ERROR %08X\n",
+ (res | ERROR_PROD_TEST_ITO));
+ return res | ERROR_PROD_TEST_ITO;
+ }
+
+ pr_info("ITO Command = OK!\n");
+
+ pr_info("MS RAW ITO ADJ TEST:\n");
+ if (todo->MutualRawAdjITO == 1) {
+ pr_info("Collecting MS Raw data...\n");
+ res |= getMSFrame3(MS_RAW, &msRawFrame);
+ if (res < OK) {
+ pr_err("%s: getMSFrame failed... ERROR %08X\n",
+ __func__, ERROR_PROD_TEST_ITO);
+ goto ERROR;
+ }
+
+ print_frame_short("MS Raw ITO frame =",
+ array1dTo2d_short(
+ msRawFrame.node_data,
+ msRawFrame.node_data_size,
+ msRawFrame.header.sense_node),
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node);
+
+ pr_info("MS RAW ITO ADJ HORIZONTAL TEST:\n");
+ res = computeAdjHorizTotal(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ &adj);
+ if (res < OK) {
+ pr_err("%s: computeAdjHoriz failed... ERROR %08X\n",
+ __func__, ERROR_PROD_TEST_ITO);
+ goto ERROR;
+ }
+
+ res = parseProductionTestLimits(path_limits, &limit_file,
+ MS_RAW_ITO_ADJH, &thresholds,
+ &trows, &tcolumns);
+ if (res < OK || (trows != msRawFrame.header.force_node ||
+ tcolumns != msRawFrame.header.sense_node -
+ 1)) {
+ pr_err("%s: parseProductionTestLimits MS_RAW_ITO_ADJH failed... ERROR %08X\n",
+ __func__, ERROR_PROD_TEST_DATA);
+ goto ERROR;
+ }
+
+
+ res = checkLimitsMapAdjTotal(adj, msRawFrame.header.force_node,
+ msRawFrame.header.sense_node - 1,
+ thresholds);
+ if (res != OK) {
+ pr_err("production_test_data: checkLimitsAdj MS RAW ITO ADJH failed... ERROR COUNT = %d\n",
+ res);
+ pr_err("MS RAW ITO ADJ HORIZONTAL TEST:.................FAIL\n\n");
+ res = ERROR_PROD_TEST_ITO;
+ goto ERROR;
+ } else
+ pr_info("MS RAW ITO ADJ HORIZONTAL TEST:.................OK\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+
+ kfree(adj);
+ adj = NULL;
+
+ pr_info("MS RAW ITO ADJ VERTICAL TEST:\n");
+ res = computeAdjVertTotal(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ &adj);
+ if (res < OK) {
+ pr_err("%s: computeAdjVert failed... ERROR %08X\n",
+ __func__, ERROR_PROD_TEST_ITO);
+ goto ERROR;
+ }
+
+ res = parseProductionTestLimits(path_limits, &limit_file,
+ MS_RAW_ITO_ADJV, &thresholds,
+ &trows, &tcolumns);
+ if (res < OK || (trows != msRawFrame.header.force_node - 1 ||
+ tcolumns != msRawFrame.header.sense_node)) {
+ pr_err("%s: parseProductionTestLimits MS_RAW_ITO_ADJV failed... ERROR %08X\n",
+ __func__, ERROR_PROD_TEST_ITO);
+ goto ERROR;
+ }
+
+
+ res = checkLimitsMapAdjTotal(adj, msRawFrame.header.force_node -
+ 1, msRawFrame.header.sense_node,
+ thresholds);
+ if (res != OK) {
+ pr_err("%s: checkLimitsAdj MS RAW ITO ADJV failed... ERROR COUNT = %d\n",
+ __func__, res);
+ pr_err("MS RAW ITO ADJ VERTICAL TEST:.................FAIL\n\n");
+ res = ERROR_PROD_TEST_ITO;
+ goto ERROR;
+ } else
+ pr_info("MS RAW ITO ADJ VERTICAL TEST:.................OK\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+
+ kfree(adj);
+ adj = NULL;
+ } else
+ pr_info("MS RAW ITO ADJ TEST:.................SKIPPED\n");
+
+ERROR:
+ if (thresholds != NULL)
+ kfree(thresholds);
+ if (adj != NULL)
+ kfree(adj);
+ if (msRawFrame.node_data != NULL)
+ kfree(msRawFrame.node_data);
+ freeLimitsFile(&limit_file);
+ res |= fts_system_reset();
+ if (res < OK) {
+ pr_err("production_test_ito: ERROR %08X\n",
+ ERROR_PROD_TEST_ITO);
+ res = (res | ERROR_PROD_TEST_ITO);
+ }
+ return res;
+}
+
+/**
+ * Perform the Initialization of the IC
+ * @param type type of initialization to do
+ * (see @link sys_special_opt Initialization Options (Full or Panel) @endlink)
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_initialization(u8 type)
+{
+ int res;
+
+ pr_info("INITIALIZATION Production test is starting...\n");
+ if (type != SPECIAL_PANEL_INIT && type != SPECIAL_FULL_PANEL_INIT) {
+ pr_err("production_test_initialization: Type incompatible! Type = %02X ERROR %08X\n",
+ type, ERROR_OP_NOT_ALLOW |
+ ERROR_PROD_TEST_INITIALIZATION);
+ return ERROR_OP_NOT_ALLOW | ERROR_PROD_TEST_INITIALIZATION;
+ }
+
+ res = fts_system_reset();
+ if (res < 0) {
+ pr_err("production_test_initialization: ERROR %08X\n",
+ ERROR_PROD_TEST_INITIALIZATION);
+ return res | ERROR_PROD_TEST_INITIALIZATION;
+ }
+
+ pr_info("INITIALIZATION command sent... %02X\n", type);
+ res = writeSysCmd(SYS_CMD_SPECIAL, &type, 1);
+ if (res < OK) {
+ pr_err("production_test_initialization: ERROR %08X\n",
+ (res | ERROR_PROD_TEST_INITIALIZATION));
+ return res | ERROR_PROD_TEST_INITIALIZATION;
+ }
+
+
+ pr_info("Refresh Sys Info...\n");
+ res |= readSysInfo(1); /* need to update the chipInfo in order
+ * to refresh several versions */
+
+ if (res < 0) {
+ pr_err("production_test_initialization: read sys info ERROR %08X\n",
+ ERROR_PROD_TEST_INITIALIZATION);
+ res = (res | ERROR_PROD_TEST_INITIALIZATION);
+ }
+
+ return res;
+}
+
+
+// @param signature value of the MP flag to save if the Mass Production Test succeed
+/**
+ * Perform a FULL (ITO + INIT + DATA CHECK) Mass Production Test of the IC
+ * @param pathThresholds name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param stop_on_fail if 1, the test flow stops at the first data check
+ * failure
+ * otherwise it keeps going performing all the selected test
+ * @param saveInit if >0 (possible values: NO_INIT, SPECIAL_PANEL_INIT or
+ * SPECIAL_FULL_PANEL_INIT),
+ * the Initialization of the IC is executed otherwise it is skipped
+ * @param todo pointer to a TestToDo variable which select the test to do
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_main(char *pathThresholds, int stop_on_fail, int saveInit,
+ TestToDo *todo)
+{
+ int res, ret;
+
+ pr_info("MAIN Production test is starting...\n");
+
+ pr_info("ITO TEST:\n");
+ res = production_test_ito(pathThresholds, todo);
+ if (res < 0) {
+ pr_err("Error during ITO TEST! ERROR %08X\n", res);
+ goto END;/* in case of ITO TEST failure is no sense keep going
+ * */
+ } else
+ pr_info("ITO TEST OK!\n");
+
+ pr_info("INITIALIZATION TEST :\n");
+ if (saveInit != NO_INIT) {
+ res = production_test_initialization((u8)saveInit);
+ if (res < 0) {
+ pr_err("Error during INITIALIZATION TEST! ERROR %08X\n",
+ res);
+ if (stop_on_fail)
+ goto END;
+ } else
+ pr_info("INITIALIZATION TEST OK!\n");
+ } else
+ pr_info("INITIALIZATION TEST :................. SKIPPED\n");
+
+ if (saveInit == 1) {
+ pr_info("Cleaning up...\n");
+ ret = fts_system_reset();
+ if (ret < 0) {
+ pr_err("production_test_main: system reset ERROR %08X\n",
+ ret);
+ res |= ret;
+ if (stop_on_fail)
+ goto END;
+ }
+ }
+
+ pr_info("PRODUCTION DATA TEST:\n");
+ ret = production_test_data(pathThresholds, stop_on_fail, todo);
+ if (ret < 0)
+ pr_err("Error during PRODUCTION DATA TEST! ERROR %08X\n", ret);
+ else
+ pr_info("PRODUCTION DATA TEST OK!\n");
+
+ res |= ret;
+ /* the OR is important because if the data test is OK but
+ * the init test fail, the main production test result should = FAIL */
+
+END:
+ if (res < 0) {
+ pr_err("MAIN Production test finished.................FAILED\n");
+ return res;
+ } else {
+ pr_info("MAIN Production test finished.................OK\n");
+ return OK;
+ }
+}
+
+/**
+ * Perform all the test selected in a TestTodo variable related to MS raw data
+ * (touch, keys etc..)
+ * @param path_limits name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param stop_on_fail if 1, the test flow stops at the first data check
+ * failure
+ * otherwise it keeps going performing all the selected test
+ * @param todo pointer to a TestToDo variable which select the test to do
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_ms_raw(char *path_limits, int stop_on_fail, TestToDo *todo)
+{
+ int ret, count_fail = 0;
+ MutualSenseFrame msRawFrame;
+
+
+ int *thresholds = NULL;
+ int trows, tcolumns;
+
+ u16 *adj = NULL;
+
+ /************** Mutual Sense Test *************/
+ pr_info("MS RAW DATA TEST is starting...\n");
+ if (todo->MutualRaw == 1 || todo->MutualRawGap == 1 ||
+ todo->MutualRawAdj == 1) {
+ ret = setScanMode(SCAN_MODE_LOCKED, LOCKED_ACTIVE);
+ mdelay(WAIT_FOR_FRESH_FRAMES);
+ ret |= setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ mdelay(WAIT_AFTER_SENSEOFF);
+ ret |= getMSFrame3(MS_RAW, &msRawFrame);
+ if (ret < OK) {
+ pr_err("production_test_data: getMSFrame failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ return ret | ERROR_PROD_TEST_DATA;
+ }
+
+ print_frame_short("MS Raw frame =",
+ array1dTo2d_short(
+ msRawFrame.node_data,
+ msRawFrame.node_data_size,
+ msRawFrame.header.sense_node),
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node);
+
+ pr_info("MS RAW MIN MAX TEST:\n");
+ if (todo->MutualRaw == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_RAW_MIN_MAX,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < OK || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_RAW_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+
+ ret = checkLimitsMinMax(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ thresholds[0],
+ thresholds[1]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax MS RAW failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS RAW MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail == 1)
+ goto ERROR;
+ } else
+ pr_info("MS RAW MIN MAX TEST:.................OK\n");
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("MS RAW MIN MAX TEST:.................SKIPPED\n");
+
+ pr_info("MS RAW GAP TEST:\n");
+ if (todo->MutualRawGap == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file, MS_RAW_GAP,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_RAW_GAP failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsGap(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ thresholds[0]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsGap MS RAW failed... ERROR = %08X\n",
+ ret);
+ count_fail += 1;
+ if (stop_on_fail == 1)
+ goto ERROR;
+ } else
+ pr_info("MS RAW GAP TEST:.................OK\n\n");
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("MS RAW GAP TEST:.................SKIPPED\n");
+
+ pr_info("MS RAW ADJ TEST:\n");
+ if (todo->MutualRawAdj == 1) {
+ pr_info("MS RAW ADJ HORIZONTAL TEST:\n");
+ ret = computeAdjHorizTotal(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ &adj);
+ if (ret < OK) {
+ pr_err("production_test_data: computeAdjHoriz failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_RAW_ADJH,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < OK || (trows !=
+ msRawFrame.header.force_node ||
+ tcolumns !=
+ msRawFrame.header.sense_node - 1)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_RAW_ADJH failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+
+ ret = checkLimitsMapAdjTotal(adj,
+ msRawFrame.header.
+ force_node,
+ msRawFrame.header.
+ sense_node - 1,
+ thresholds);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsAdj MS RAW ADJH failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS RAW ADJ HORIZONTAL TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail == 1)
+ goto ERROR;
+ } else
+ pr_info("MS RAW ADJ HORIZONTAL TEST:.................OK\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+
+ kfree(adj);
+ adj = NULL;
+
+ pr_info("MS RAW ADJ VERTICAL TEST:\n");
+ ret = computeAdjVertTotal(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ &adj);
+ if (ret < OK) {
+ pr_err("production_test_data: computeAdjVert failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_RAW_ADJV,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < OK || (trows != msRawFrame.header.force_node -
+ 1 || tcolumns !=
+ msRawFrame.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_RAW_ADJV failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+
+ ret = checkLimitsMapAdjTotal(adj,
+ msRawFrame.header.
+ force_node - 1,
+ msRawFrame.header.
+ sense_node, thresholds);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsAdj MS RAW ADJV failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS RAW ADJ VERTICAL TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail == 1)
+ goto ERROR;
+ } else
+ pr_info("MS RAW ADJ VERTICAL TEST:.................OK\n");
+ kfree(thresholds);
+ thresholds = NULL;
+
+ kfree(adj);
+ adj = NULL;
+ } else
+ pr_info("MS RAW ADJ TEST:.................SKIPPED\n");
+ } else
+ pr_info("MS RAW FRAME TEST:.................SKIPPED\n");
+
+ pr_info("MS KEY RAW TEST:\n");
+ if (todo->MutualKeyRaw == 1) {
+ ret = production_test_ms_key_raw(path_limits);
+ if (ret < 0) {
+ pr_err("production_test_data: production_test_ms_key_raw failed... ERROR = %08X\n",
+ ret);
+ count_fail += 1;
+ if (count_fail == 1) {
+ pr_err("MS RAW DATA TEST:.................FAIL fails_count = %d\n\n",
+ count_fail);
+ goto ERROR_LIMITS;
+ }
+ }
+ } else
+ pr_info("MS KEY RAW TEST:.................SKIPPED\n");
+
+ ret = production_test_ms_raw_lp(path_limits, stop_on_fail, todo);
+ if (ret < 0) {
+ pr_err("production_test_data: production_test_ms_raw_lp failed... ERROR = %08X\n",
+ ret);
+ count_fail += 1;
+ if (count_fail == 1) {
+ pr_err("MS RAW DATA TEST:.................FAIL fails_count = %d\n\n",
+ count_fail);
+ goto ERROR_LIMITS;
+ }
+ }
+
+ERROR:
+
+ if (count_fail == 0) {
+ if (msRawFrame.node_data != NULL) {
+ kfree(msRawFrame.node_data);
+ msRawFrame.node_data = NULL;
+ }
+ pr_info("MS RAW DATA TEST finished!.................OK\n");
+ return OK;
+ } else {
+ if (msRawFrame.node_data != NULL)
+ kfree(msRawFrame.node_data);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ if (adj != NULL)
+ kfree(adj);
+ pr_err("MS RAW DATA TEST:.................FAIL fails_count = %d\n\n",
+ count_fail);
+ return ERROR_PROD_TEST_DATA | ERROR_TEST_CHECK_FAIL;
+ }
+
+
+ERROR_LIMITS:
+ if (msRawFrame.node_data != NULL)
+ kfree(msRawFrame.node_data);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ return ret;
+}
+
+
+/**
+ * Perform all the test selected in a TestTodo variable related to MS low power
+ * raw data
+ * @param path_limits name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param stop_on_fail if 1, the test flow stops at the first data check
+ * failure
+ * otherwise it keeps going performing all the selected test
+ * @param todo pointer to a TestToDo variable which select the test to do
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_ms_raw_lp(char *path_limits, int stop_on_fail,
+ TestToDo *todo)
+{
+ int ret, count_fail = 0;
+ MutualSenseFrame msRawFrame;
+
+
+ int *thresholds = NULL;
+ int trows, tcolumns;
+
+ u16 *adj = NULL;
+
+ /************** Mutual Sense Test **************/
+ pr_info("MS RAW LP DATA TEST:\n");
+ if (todo->MutualRawLP == 1 || todo->MutualRawGapLP == 1 ||
+ todo->MutualRawAdjLP == 1) {
+ ret = setScanMode(SCAN_MODE_LOCKED, LOCKED_LP_ACTIVE);
+ mdelay(WAIT_FOR_FRESH_FRAMES);
+ ret |= setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ mdelay(WAIT_AFTER_SENSEOFF);
+ ret |= getMSFrame3(MS_RAW, &msRawFrame);
+ if (ret < 0) {
+ pr_err("production_test_data: getMSFrame failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ return ret | ERROR_PROD_TEST_DATA;
+ }
+
+ print_frame_short("MS Raw LP frame =",
+ array1dTo2d_short(
+ msRawFrame.node_data,
+ msRawFrame.node_data_size,
+ msRawFrame.header.sense_node),
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node);
+
+ pr_info("MS RAW LP MIN MAX TEST:\n");
+ if (todo->MutualRawLP == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_RAW_LP_MIN_MAX,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_RAW_LP_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+
+ ret = checkLimitsMinMax(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ thresholds[0],
+ thresholds[1]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax MS RAW LP failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS RAW LP MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail == 1)
+ goto ERROR;
+ } else
+ pr_info("MS RAW LP MIN MAX TEST:.................OK\n");
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("MS RAW LP MIN MAX TEST:.................SKIPPED\n");
+
+ pr_info("MS RAW LP GAP TEST:\n");
+ if (todo->MutualRawGapLP == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_RAW_LP_GAP,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_RAW_LP_GAP failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsGap(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ thresholds[0]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsGap MS RAW LP failed... ERROR = %08X\n",
+ ret);
+ count_fail += 1;
+ if (stop_on_fail == 1)
+ goto ERROR;
+ } else
+ pr_info("MS RAW LP GAP TEST:.................OK\n\n");
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("MS RAW LP GAP TEST:.................SKIPPED\n");
+
+ pr_info("MS RAW LP ADJ TEST:\n");
+ if (todo->MutualRawAdjLP == 1) {
+ pr_info("MS RAW LP ADJ HORIZONTAL TEST:\n");
+ ret = computeAdjHorizTotal(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ &adj);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjHoriz failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_RAW_LP_ADJH,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != msRawFrame.header.force_node ||
+ tcolumns !=
+ msRawFrame.header.sense_node - 1)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_RAW_LP_ADJH failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+
+ ret = checkLimitsMapAdjTotal(adj,
+ msRawFrame.header.
+ force_node,
+ msRawFrame.header.
+ sense_node - 1,
+ thresholds);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsAdj MS RAW LP ADJH failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS RAW LP ADJ HORIZONTAL TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail == 1)
+ goto ERROR;
+ } else
+ pr_info("MS RAW LP ADJ HORIZONTAL TEST:.................OK\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+
+ kfree(adj);
+ adj = NULL;
+
+ pr_info("MS RAW LP ADJ VERTICAL TEST:\n");
+ ret = computeAdjVertTotal(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ &adj);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjVert failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_RAW_LP_ADJV,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != msRawFrame.header.force_node -
+ 1 || tcolumns !=
+ msRawFrame.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_RAW_ADJV failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+
+ ret = checkLimitsMapAdjTotal(adj,
+ msRawFrame.header.
+ force_node - 1,
+ msRawFrame.header.
+ sense_node, thresholds);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsAdj MS RAW ADJV failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS RAW LP ADJ VERTICAL TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail == 1)
+ goto ERROR;
+ } else
+ pr_info("MS RAW LP ADJ VERTICAL TEST:.................OK\n");
+ kfree(thresholds);
+ thresholds = NULL;
+
+ kfree(adj);
+ adj = NULL;
+ } else
+ pr_info("MS RAW LP ADJ TEST:.................SKIPPED\n");
+ } else
+ pr_info("MS RAW LP FRAME TEST:.................SKIPPED\n");
+
+ERROR:
+ if (count_fail == 0) {
+ if (msRawFrame.node_data != NULL) {
+ kfree(msRawFrame.node_data);
+ msRawFrame.node_data = NULL;
+ }
+ pr_info("MS RAW DATA TEST finished!.................OK\n");
+ return OK;
+ } else {
+ if (msRawFrame.node_data != NULL)
+ kfree(msRawFrame.node_data);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ if (adj != NULL)
+ kfree(adj);
+ pr_err("MS RAW LP DATA TEST:.................FAIL fails_count = %d\n\n",
+ count_fail);
+ return ERROR_PROD_TEST_DATA | ERROR_TEST_CHECK_FAIL;
+ }
+
+
+ERROR_LIMITS:
+ if (msRawFrame.node_data != NULL)
+ kfree(msRawFrame.node_data);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ return ret;
+}
+
+/**
+ * Perform MS raw test for keys data
+ * @param path_limits name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_ms_key_raw(char *path_limits)
+{
+ int ret;
+ MutualSenseFrame msRawFrame;
+
+ int *thresholds = NULL;
+ int trows, tcolumns;
+
+ /************** Mutual Sense Test **************/
+ pr_info("MS KEY RAW DATA TEST is starting...\n");
+ ret = setScanMode(SCAN_MODE_ACTIVE, 0xFF);
+ mdelay(WAIT_FOR_FRESH_FRAMES);
+ ret |= setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ mdelay(WAIT_AFTER_SENSEOFF);
+ ret |= getMSFrame3(MS_KEY_RAW, &msRawFrame);
+ if (ret < 0) {
+ pr_err("production_test_data: getMSKeyFrame failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ return ret | ERROR_PROD_TEST_DATA;
+ }
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_KEY_RAW_MIN_MAX, &thresholds, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_KEY_RAW_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMinMax(msRawFrame.node_data,
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node,
+ thresholds[0], thresholds[1]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax MS KEY RAW failed... ERROR COUNT = %d\n",
+ ret);
+ goto ERROR;
+ } else
+ pr_info("MS KEY RAW TEST:.................OK\n\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+
+ kfree(msRawFrame.node_data);
+ msRawFrame.node_data = NULL;
+ return OK;
+
+ERROR:
+ print_frame_short("MS Key Raw frame =", array1dTo2d_short(
+ msRawFrame.node_data,
+ msRawFrame.node_data_size,
+ msRawFrame.header.sense_node),
+ msRawFrame.header.force_node,
+ msRawFrame.header.sense_node);
+ if (msRawFrame.node_data != NULL)
+ kfree(msRawFrame.node_data);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ pr_err("MS KEY RAW TEST:.................FAIL\n\n");
+ return ERROR_PROD_TEST_DATA | ERROR_TEST_CHECK_FAIL;
+
+ERROR_LIMITS:
+ if (msRawFrame.node_data != NULL)
+ kfree(msRawFrame.node_data);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ return ret;
+}
+
+/**
+ * Perform all the tests selected in a TestTodo variable related to MS Init
+ * data (touch, keys etc..)
+ * @param path_limits name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param stop_on_fail if 1, the test flow stops at the first data check
+ * failure
+ * otherwise it keeps going performing all the selected test
+ * @param todo pointer to a TestToDo variable which select the test to do
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_ms_cx(char *path_limits, int stop_on_fail, TestToDo *todo)
+{
+ int ret;
+ int count_fail = 0;
+
+ int *thresholds = NULL;
+ int *thresholds_min = NULL;
+ int *thresholds_max = NULL;
+ int trows, tcolumns;
+
+ MutualSenseData msCompData;
+ TotMutualSenseData totCompData;
+
+ u8 *adjhor = NULL;
+
+ u8 *adjvert = NULL;
+
+ u16 container;
+ /* u16 *total_cx = NULL; */
+ u16 *total_adjhor = NULL;
+ u16 *total_adjvert = NULL;
+
+
+ /* MS CX TEST */
+ pr_info("MS CX Testes are starting...\n");
+
+ ret = readMutualSenseCompensationData(LOAD_CX_MS_TOUCH, &msCompData);
+ /* read MS compensation data */
+ if (ret < 0) {
+ pr_err("production_test_data: readMutualSenseCompensationData failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ return ret | ERROR_PROD_TEST_DATA;
+ }
+
+ ret = readTotMutualSenseCompensationData(LOAD_PANEL_CX_TOT_MS_TOUCH,
+ &totCompData);
+ /* read TOT MS compensation data */
+ if (ret < 0) {
+ pr_err("production_test_data: readTotMutualSenseCompensationData failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ kfree(msCompData.node_data);
+ msCompData.node_data = NULL;
+ return ret | ERROR_PROD_TEST_DATA;
+ }
+
+ pr_info("MS CX1 TEST:\n");
+ if (todo->MutualCx1 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_CX1_MIN_MAX, &thresholds,
+ &trows, &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_CX1_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ container = (u16)msCompData.cx1;
+ ret = checkLimitsMinMax(&container, 1, 1, thresholds[0],
+ thresholds[1]);
+ /* check the limits */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax MS CX1 failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS CX1 TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("MS CX1 TEST:.................OK\n\n");
+ } else
+ pr_info("MS CX1 TEST:.................SKIPPED\n\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+
+ pr_info("MS CX2 MIN MAX TEST:\n");
+ if (todo->MutualCx2 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_CX2_MAP_MIN, &thresholds_min,
+ &trows, &tcolumns);
+ /* load min thresholds */
+ if (ret < 0 || (trows != msCompData.header.force_node ||
+ tcolumns != msCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_CX2_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_CX2_MAP_MAX, &thresholds_max,
+ &trows, &tcolumns);
+ /* load max thresholds */
+ if (ret < 0 || (trows != msCompData.header.force_node ||
+ tcolumns != msCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_CX2_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMap(msCompData.node_data,
+ msCompData.header.force_node,
+ msCompData.header.sense_node,
+ thresholds_min, thresholds_max);
+ /* check the limits */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap MS CX2 MIN MAX failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS CX2 MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("MS CX2 MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("MS CX2 MIN MAX TEST:.................SKIPPED\n\n");
+
+ pr_info("MS CX2 ADJ TEST:\n");
+ if (todo->MutualCx2Adj == 1) {
+ /* MS CX2 ADJ HORIZ */
+ pr_info("MS CX2 ADJ HORIZ TEST:\n");
+
+ ret = computeAdjHoriz(msCompData.node_data,
+ msCompData.header.force_node,
+ msCompData.header.sense_node,
+ &adjhor);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjHoriz failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("MS CX2 ADJ HORIZ computed!\n");
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_CX2_ADJH_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != msCompData.header.force_node ||
+ tcolumns != msCompData.header.sense_node - 1)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_CX2_ADJH_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdj(adjhor, msCompData.header.force_node,
+ msCompData.header.sense_node - 1,
+ thresholds_max);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMapAdj CX2 ADJH failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS CX2 ADJ HORIZ TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("MS CX2 ADJ HORIZ TEST:.................OK\n\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(adjhor);
+ adjhor = NULL;
+
+ /* MS CX2 ADJ VERT */
+ pr_info("MS CX2 ADJ VERT TEST:\n");
+
+ ret = computeAdjVert(msCompData.node_data,
+ msCompData.header.force_node,
+ msCompData.header.sense_node,
+ &adjvert);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjVert failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("MS CX2 ADJ VERT computed!\n");
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_CX2_ADJV_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != msCompData.header.force_node - 1 ||
+ tcolumns != msCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_CX2_ADJV_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdj(adjvert, msCompData.header.force_node -
+ 1, msCompData.header.sense_node - 1,
+ thresholds_max);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMapAdj CX2 ADJV failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS CX2 ADJ HORIZ TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("MS CX2 ADJ VERT TEST:.................OK\n\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(adjvert);
+ adjvert = NULL;
+ } else
+ pr_info("MS CX2 ADJ TEST:.................SKIPPED\n\n");
+
+ /* START OF TOTAL CHECK */
+ pr_info("MS TOTAL CX TEST:\n");
+
+ if (todo->MutualCxTotal == 1 || todo->MutualCxTotalAdj == 1) {
+ pr_info("MS TOTAL CX MIN MAX TEST:\n");
+ if (todo->MutualCxTotal == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_TOTAL_CX_MAP_MIN,
+ &thresholds_min,
+ &trows, &tcolumns);
+ /* load min thresholds */
+ if (ret < 0 || (trows !=
+ totCompData.header.force_node ||
+ tcolumns !=
+ totCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_TOTAL_CX_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_TOTAL_CX_MAP_MAX,
+ &thresholds_max,
+ &trows, &tcolumns);
+ /* load max thresholds */
+ if (ret < 0 || (trows !=
+ totCompData.header.force_node ||
+ tcolumns !=
+ totCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_TOTAL_CX_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapTotal(totCompData.node_data,
+ totCompData.header.force_node,
+ totCompData.header.sense_node,
+ thresholds_min,
+ thresholds_max);
+ /* check the limits */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap MS TOTAL CX TEST failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS TOTAL CX MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("MS TOTAL CX MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("MS TOTAL CX MIN MAX TEST:.................SKIPPED\n\n");
+
+
+ pr_info("MS TOTAL CX ADJ TEST:\n");
+ if (todo->MutualCxTotalAdj == 1) {
+ /* MS TOTAL CX ADJ HORIZ */
+ pr_info("MS TOTAL CX ADJ HORIZ TEST:\n");
+
+ ret = computeAdjHorizTotal(totCompData.node_data,
+ totCompData.header.force_node,
+ totCompData.header.sense_node,
+ &total_adjhor);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjHoriz failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("MS TOTAL CX ADJ HORIZ computed!\n");
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_TOTAL_CX_ADJH_MAP_MAX,
+ &thresholds_max,
+ &trows, &tcolumns);
+ if (ret < 0 || (trows !=
+ totCompData.header.force_node ||
+ tcolumns !=
+ totCompData.header.sense_node - 1)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_TOTAL_CX_ADJH_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdjTotal(total_adjhor,
+ totCompData.header.
+ force_node,
+ totCompData.header.
+ sense_node - 1,
+ thresholds_max);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMapAdj MS TOTAL CX ADJH failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS TOTAL CX ADJ HORIZ TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("MS TOTAL CX ADJ HORIZ TEST:.................OK\n\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(total_adjhor);
+ total_adjhor = NULL;
+
+ /* MS TOTAL CX ADJ VERT */
+ pr_info("MS TOTAL CX ADJ VERT TEST:\n");
+
+ ret = computeAdjVertTotal(totCompData.node_data,
+ totCompData.header.force_node,
+ totCompData.header.sense_node,
+ &total_adjvert);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjVert failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("MS TOTAL CX ADJ VERT computed!\n");
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ MS_TOTAL_CX_ADJV_MAP_MAX,
+ &thresholds_max,
+ &trows, &tcolumns);
+ if (ret < 0 || (trows != totCompData.header.force_node -
+ 1 || tcolumns !=
+ totCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_TOTAL_CX_ADJV_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdjTotal(total_adjvert,
+ totCompData.header.
+ force_node - 1,
+ totCompData.header.
+ sense_node - 1,
+ thresholds_max);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMapAdj MS TOTAL CX ADJV failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS TOTAL CX ADJ HORIZ TEST:.................FAIL\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("MS TOTAL CX ADJ VERT TEST:.................OK\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(total_adjvert);
+ total_adjvert = NULL;
+ } else
+ pr_info("MS TOTAL CX ADJ TEST:.................SKIPPED\n");
+
+ kfree(totCompData.node_data);
+ totCompData.node_data = NULL;
+ } else
+ pr_info("MS TOTAL CX TEST:.................SKIPPED\n");
+
+
+
+ if ((todo->MutualKeyCx1 | todo->MutualKeyCx2 |
+ todo->MutualKeyCxTotal) == 1) {
+ ret = production_test_ms_key_cx(path_limits, stop_on_fail,
+ todo);
+ if (ret < 0) {
+ count_fail += 1;
+ pr_err("production_test_data: production_test_ms_key_cx failed... ERROR = %08X\n",
+ ret);
+ pr_err("MS CX testes finished!.................FAILED fails_count = %d\n\n",
+ count_fail);
+ return ret;
+ }
+ } else
+ pr_info("MS KEY CX TEST:.................SKIPPED\n");
+
+ERROR:
+
+ if (count_fail == 0) {
+ pr_info("MS CX testes finished!.................OK\n");
+ kfree(msCompData.node_data);
+ msCompData.node_data = NULL;
+ return OK;
+ } else {
+ print_frame_i8("MS Init Data (Cx2) =", array1dTo2d_i8(
+ msCompData.node_data,
+ msCompData.node_data_size,
+ msCompData.header.sense_node),
+ msCompData.header.force_node,
+ msCompData.header.sense_node);
+ print_frame_short(" TOT MS Init Data (Cx) =", array1dTo2d_short(
+ totCompData.node_data,
+ totCompData.node_data_size,
+ totCompData.header.sense_node),
+ totCompData.header.force_node,
+ totCompData.header.sense_node);
+ pr_err("MS CX testes finished!.................FAILED fails_count = %d\n\n",
+ count_fail);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ if (thresholds_min != NULL)
+ kfree(thresholds_min);
+ if (thresholds_max != NULL)
+ kfree(thresholds_max);
+ if (adjhor != NULL)
+ kfree(adjhor);
+ if (adjvert != NULL)
+ kfree(adjvert);
+ if (totCompData.node_data != NULL)
+ kfree(totCompData.node_data);
+ if (total_adjhor != NULL)
+ kfree(total_adjhor);
+ if (total_adjvert != NULL)
+ kfree(total_adjvert);
+ if (msCompData.node_data != NULL)
+ kfree(msCompData.node_data);
+ return ERROR_TEST_CHECK_FAIL | ERROR_PROD_TEST_DATA;
+ }
+
+ERROR_LIMITS:
+ if (thresholds != NULL)
+ kfree(thresholds);
+ if (thresholds_min != NULL)
+ kfree(thresholds_min);
+ if (thresholds_max != NULL)
+ kfree(thresholds_max);
+ if (adjhor != NULL)
+ kfree(adjhor);
+ if (adjvert != NULL)
+ kfree(adjvert);
+ if (totCompData.node_data != NULL)
+ kfree(totCompData.node_data);
+ if (total_adjhor != NULL)
+ kfree(total_adjhor);
+ if (total_adjvert != NULL)
+ kfree(total_adjvert);
+ if (msCompData.node_data != NULL)
+ kfree(msCompData.node_data);
+ return ret;
+}
+
+/**
+ * Perform all the tests selected in a TestTodo variable related to MS Init
+ * data of the keys
+ * @param path_limits name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param stop_on_fail if 1, the test flow stops at the first data check
+ * failure
+ * otherwise it keeps going performing all the selected test
+ * @param todo pointer to a TestToDo variable which select the test to do
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_ms_key_cx(char *path_limits, int stop_on_fail,
+ TestToDo *todo)
+{
+ int ret;
+ int count_fail = 0;
+ int num_keys = 0;
+
+ int *thresholds = NULL;
+ int *thresholds_min = NULL;
+ int *thresholds_max = NULL;
+ int trows, tcolumns;
+
+ MutualSenseData msCompData;
+ TotMutualSenseData totCompData;
+
+
+ short container;
+
+
+ /* MS CX TEST */
+ pr_info("MS KEY CX Testes are starting...\n");
+
+ ret = readMutualSenseCompensationData(LOAD_CX_MS_KEY, &msCompData);
+ /* read MS compensation data */
+ if (ret < 0) {
+ pr_err("production_test_data: readMutualSenseCompensationData failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ return ret | ERROR_PROD_TEST_DATA;
+ }
+
+ if (msCompData.header.force_node > msCompData.header.sense_node)
+ /* the meaningful data are only in the first row,
+ * the other rows are only a copy of the first one */
+ num_keys = msCompData.header.force_node;
+ else
+ num_keys = msCompData.header.sense_node;
+
+ pr_info("MS KEY CX1 TEST:\n");
+ if (todo->MutualKeyCx1 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_KEY_CX1_MIN_MAX, &thresholds,
+ &trows, &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_KEY_CX1_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ container = (short)msCompData.cx1;
+ ret = checkLimitsMinMax(&container, 1, 1, thresholds[0],
+ thresholds[1]);
+ /* check the limits */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax MS CX1 failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS KEY CX1 TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("MS KEY CX1 TEST:.................OK\n\n");
+ } else
+ pr_info("MS KEY CX1 TEST:.................SKIPPED\n\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+
+ pr_info("MS KEY CX2 TEST:\n");
+ if (todo->MutualKeyCx2 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_KEY_CX2_MAP_MIN,
+ &thresholds_min, &trows,
+ &tcolumns);
+ /* load min thresholds */
+ if (ret < 0 || (trows != msCompData.header.force_node ||
+ tcolumns != msCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_KEY_CX2_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_KEY_CX2_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load max thresholds */
+ if (ret < 0 || (trows != msCompData.header.force_node ||
+ tcolumns != msCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_KEY_CX2_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMap(msCompData.node_data,
+ msCompData.header.force_node,
+ msCompData.header.sense_node,
+ thresholds_min, thresholds_max);
+ /* check the limits */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap MS KEY CX2 failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS KEY CX2 TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("MS KEY CX2 TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("MS CX2 TEST:.................SKIPPED\n\n");
+
+ /* START OF TOTAL CHECK */
+ pr_info("MS KEY TOTAL CX TEST:\n");
+
+ if (todo->MutualKeyCxTotal == 1) {
+ ret = readTotMutualSenseCompensationData(
+ LOAD_PANEL_CX_TOT_MS_KEY, &totCompData);
+ if (ret < 0) {
+ pr_err("production_test_data: computeTotalCx failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_KEY_TOTAL_CX_MAP_MIN,
+ &thresholds_min, &trows,
+ &tcolumns);
+ /* load min thresholds */
+ if (ret < 0 || (trows != totCompData.header.force_node ||
+ tcolumns != totCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_KEY_TOTAL_CX_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ MS_KEY_TOTAL_CX_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load max thresholds */
+ if (ret < 0 || (trows != totCompData.header.force_node ||
+ tcolumns != totCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits MS_KEY_TOTAL_CX_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapTotal(totCompData.node_data,
+ totCompData.header.force_node,
+ totCompData.header.sense_node,
+ thresholds_min, thresholds_max);
+ /* check the limits */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap MS TOTAL KEY CX TEST failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("MS KEY TOTAL CX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("MS KEY TOTAL CX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+
+ kfree(totCompData.node_data);
+ totCompData.node_data = NULL;
+ } else
+ pr_info("MS KEY TOTAL CX TEST:.................SKIPPED\n");
+
+
+ERROR:
+ if (count_fail == 0) {
+ pr_info("MS KEY CX testes finished!.................OK\n");
+ kfree(msCompData.node_data);
+ msCompData.node_data = NULL;
+ return OK;
+ } else {
+ print_frame_i8("MS Key Init Data (Cx2) =", array1dTo2d_i8(
+ msCompData.node_data,
+ msCompData.node_data_size,
+ msCompData.header.sense_node),
+ msCompData.header.force_node,
+ msCompData.header.sense_node);
+ pr_err("MS Key CX testes finished!.................FAILED fails_count = %d\n\n",
+ count_fail);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ if (thresholds_min != NULL)
+ kfree(thresholds_min);
+ if (thresholds_max != NULL)
+ kfree(thresholds_max);
+ if (msCompData.node_data != NULL)
+ kfree(msCompData.node_data);
+ if (totCompData.node_data != NULL)
+ kfree(totCompData.node_data);
+ return ERROR_TEST_CHECK_FAIL | ERROR_PROD_TEST_DATA;
+ }
+
+ERROR_LIMITS:
+ if (thresholds != NULL)
+ kfree(thresholds);
+ if (thresholds_min != NULL)
+ kfree(thresholds_min);
+ if (thresholds_max != NULL)
+ kfree(thresholds_max);
+ if (msCompData.node_data != NULL)
+ kfree(msCompData.node_data);
+ if (totCompData.node_data != NULL)
+ kfree(totCompData.node_data);
+ return ret;
+}
+
+/**
+ * Perform all the test selected in a TestTodo variable related to SS raw data
+ *(touch, keys etc..)
+ * @param path_limits name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param stop_on_fail if 1, the test flow stops at the first data check
+ * failure
+ * otherwise it keeps going performing all the selected test
+ * @param todo pointer to a TestToDo variable which select the test to do
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_ss_raw(char *path_limits, int stop_on_fail, TestToDo *todo)
+{
+ int ret;
+ int count_fail = 0;
+ int rows, columns;
+
+ SelfSenseFrame ssRawFrame;
+
+ int *thresholds = NULL;
+ int trows, tcolumns;
+
+ /* SS TEST */
+ pr_info("SS RAW Testes are starting...\n");
+
+ /************** Self Sense Test **************/
+
+ pr_info("Getting SS Frame...\n");
+ ret = setScanMode(SCAN_MODE_LOCKED, LOCKED_ACTIVE);
+ mdelay(WAIT_FOR_FRESH_FRAMES);
+ ret |= setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ mdelay(WAIT_AFTER_SENSEOFF);
+ ret |= getSSFrame3(SS_RAW, &ssRawFrame);
+ if (ret < 0) {
+ pr_err("production_test_data: getSSFrame failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ return ret | ERROR_PROD_TEST_DATA;
+ }
+
+ print_frame_short("SS Raw force frame =",
+ array1dTo2d_short(
+ ssRawFrame.force_data,
+ ssRawFrame.header.force_node,
+ 1),
+ ssRawFrame.header.force_node, 1);
+ print_frame_short("SS Raw sense frame =",
+ array1dTo2d_short(
+ ssRawFrame.sense_data,
+ ssRawFrame.header.sense_node,
+ ssRawFrame.header.sense_node),
+ 1, ssRawFrame.header.sense_node);
+
+ /* SS RAW (PROXIMITY) FORCE TEST */
+ pr_info("SS RAW FORCE TEST:\n");
+
+
+
+ if (todo->SelfForceRaw == 1 || todo->SelfForceRawGap == 1) {
+ columns = 1; /* there are no data for the sense channels
+ * because is a force frame */
+ rows = ssRawFrame.header.force_node;
+
+ pr_info("SS RAW FORCE MIN MAX TEST:\n");
+ if (todo->SelfForceRaw == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_RAW_FORCE_MIN_MAX,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_RAW_FORCE_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMinMax(ssRawFrame.force_data, rows,
+ columns, thresholds[0],
+ thresholds[1]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax SS RAW FORCE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS RAW (PROXIMITY) FORCE MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail) {
+ ret = ERROR_PROD_TEST_DATA |
+ ERROR_TEST_CHECK_FAIL;
+ goto ERROR_LIMITS;
+ }
+ } else
+ pr_info("SS RAW FORCE MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("SS RAW FORCE MIN MAX TEST:.................SKIPPED\n\n");
+
+ pr_info("SS RAW FORCE GAP TEST:\n");
+ if (todo->SelfForceRawGap == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_RAW_FORCE_GAP,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_RAW_FORCE_GAP failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsGap(ssRawFrame.force_data, rows,
+ columns, thresholds[0]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsGap SS RAW FORCE GAP failed... ERROR = %08X\n",
+ ret);
+ pr_err("SS RAW FORCE GAP TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail) {
+ ret = ERROR_PROD_TEST_DATA |
+ ERROR_TEST_CHECK_FAIL;
+ goto ERROR_LIMITS;
+ }
+ } else
+ pr_info("SS RAW FORCE GAP TEST:.................OK\n\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("SS RAW FORCE GAP TEST:.................SKIPPED\n\n");
+
+ kfree(ssRawFrame.force_data);
+ ssRawFrame.force_data = NULL;
+ } else
+ pr_info("SS RAW FORCE TEST:.................SKIPPED\n\n");
+
+ /* SS RAW (PROXIMITY) SENSE TEST */
+ pr_info("SS RAW SENSE TEST:\n");
+
+ if (todo->SelfSenseRaw == 1 || todo->SelfSenseRawGap == 1) {
+ columns = ssRawFrame.header.sense_node;
+ rows = 1;/* there are no data for the force channels
+ * because is a sense frame */
+
+ pr_info("SS RAW SENSE MIN MAX TEST:\n");
+ if (todo->SelfSenseRaw == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_RAW_SENSE_MIN_MAX,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_RAW_SENSE_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMinMax(ssRawFrame.sense_data, rows,
+ columns, thresholds[0],
+ thresholds[1]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax SS RAW SENSE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS RAW SENSE MIN MAX TEST:.................FAIL\n");
+ count_fail += 1;
+ if (stop_on_fail) {
+ ret = ERROR_PROD_TEST_DATA |
+ ERROR_TEST_CHECK_FAIL;
+ goto ERROR_LIMITS;
+ }
+ } else
+ pr_info("SS RAW SENSE MIN MAX TEST:.................OK\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("SS RAW SENSE MIN MAX TEST:.................SKIPPED\n");
+
+ pr_info("SS RAW SENSE GAP TEST:\n");
+ if (todo->SelfSenseRawGap == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_RAW_SENSE_GAP,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_RAW_SENSE_GAP failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsGap(ssRawFrame.sense_data, rows,
+ columns, thresholds[0]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsGap SS RAW SENSE GAP failed... ERROR = %08X\n",
+ ret);
+ pr_err("SS RAW SENSE GAP TEST:.................FAIL\n");
+ count_fail += 1;
+ if (stop_on_fail) {
+ ret = ERROR_PROD_TEST_DATA |
+ ERROR_TEST_CHECK_FAIL;
+ goto ERROR_LIMITS;
+ }
+ } else
+ pr_info("SS RAW SENSE GAP TEST:.................OK\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("SS RAW SENSE GAP TEST:.................SKIPPED\n");
+
+ kfree(ssRawFrame.sense_data);
+ ssRawFrame.sense_data = NULL;
+ }
+
+ ret = production_test_ss_raw_lp(path_limits, stop_on_fail, todo);
+ if (ret < OK) {
+ pr_err("production_test_data: production_test_ss_raw_lp failed... ERROR = %08X\n",
+ ret);
+ count_fail += 1;
+ }
+
+ if (count_fail == 0) {
+ pr_info("SS RAW testes finished!.................OK\n\n");
+ return OK;
+ } else {
+ pr_err("SS RAW testes finished!.................FAILED fails_count = %d\n\n",
+ count_fail);
+ return ERROR_TEST_CHECK_FAIL | ERROR_PROD_TEST_DATA;
+ }
+
+ERROR_LIMITS:
+ if (ssRawFrame.force_data != NULL)
+ kfree(ssRawFrame.force_data);
+ if (ssRawFrame.sense_data != NULL)
+ kfree(ssRawFrame.sense_data);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ return ret;
+}
+
+
+/**
+ * Perform all the test selected in a TestTodo variable related to SS raw data
+ * low power
+ * @param path_limits name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param stop_on_fail if 1, the test flow stops at the first data check
+ * failure
+ * otherwise it keeps going performing all the selected test
+ * @param todo pointer to a TestToDo variable which select the test to do
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_ss_raw_lp(char *path_limits, int stop_on_fail,
+ TestToDo *todo)
+{
+ int ret;
+ int count_fail = 0;
+ int rows, columns;
+
+ SelfSenseFrame ssRawFrame;
+
+ int *thresholds = NULL;
+ int trows, tcolumns;
+
+ /* SS TEST */
+ pr_info("SS RAW LP Testes are starting...\n");
+
+ /************** Self Sense Test **************/
+
+ pr_info("Getting SS LP Frame...\n");
+ ret = setScanMode(SCAN_MODE_LOCKED, LOCKED_LP_DETECT);
+ mdelay(WAIT_FOR_FRESH_FRAMES);
+ ret |= setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ mdelay(WAIT_AFTER_SENSEOFF);
+ ret |= getSSFrame3(SS_RAW, &ssRawFrame);
+ if (ret < 0) {
+ pr_err("production_test_data: getSSFrame failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ return ret | ERROR_PROD_TEST_DATA;
+ }
+
+ print_frame_short("SS Raw LP force frame =",
+ array1dTo2d_short(
+ ssRawFrame.force_data,
+ ssRawFrame.header.force_node, 1),
+ ssRawFrame.header.force_node, 1);
+ print_frame_short("SS Raw LP sense frame =",
+ array1dTo2d_short(
+ ssRawFrame.sense_data,
+ ssRawFrame.header.sense_node,
+ ssRawFrame.header.sense_node),
+ 1, ssRawFrame.header.sense_node);
+
+ /* SS RAW (PROXIMITY) FORCE TEST */
+ pr_info("SS RAW LP FORCE TEST:\n");
+
+ if (todo->SelfForceRawLP == 1 || todo->SelfForceRawGapLP == 1) {
+ columns = 1; /* there are no data for the sense channels
+ * because is a force frame */
+ rows = ssRawFrame.header.force_node;
+
+ pr_info("SS RAW LP FORCE MIN MAX TEST:\n");
+ if (todo->SelfForceRawLP == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_RAW_LP_FORCE_MIN_MAX,
+ &thresholds,
+ &trows, &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_RAW_FORCE_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMinMax(ssRawFrame.force_data, rows,
+ columns, thresholds[0],
+ thresholds[1]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax SS RAW FORCE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS RAW LP FORCE MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail) {
+ ret = ERROR_PROD_TEST_DATA |
+ ERROR_TEST_CHECK_FAIL;
+ goto ERROR_LIMITS;
+ }
+ } else
+ pr_info("SS RAW LP FORCE MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("SS RAW LP FORCE MIN MAX TEST:.................SKIPPED\n\n");
+
+ pr_info("SS RAW LP FORCE GAP TEST:\n");
+ if (todo->SelfForceRawGapLP == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_RAW_LP_FORCE_GAP,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < OK || (trows != 1 || tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_RAW_FORCE_GAP failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsGap(ssRawFrame.force_data, rows,
+ columns, thresholds[0]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsGap SS RAW FORCE GAP failed... ERROR = %08X\n",
+ ret);
+ pr_err("SS RAW LP FORCE GAP TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail) {
+ ret = ERROR_PROD_TEST_DATA |
+ ERROR_TEST_CHECK_FAIL;
+ goto ERROR_LIMITS;
+ }
+ } else
+ pr_info("SS RAW LP FORCE GAP TEST:.................OK\n\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("SS RAW LP FORCE GAP TEST:.................SKIPPED\n\n");
+
+ kfree(ssRawFrame.force_data);
+ ssRawFrame.force_data = NULL;
+ } else
+ pr_info("SS RAW LP FORCE TEST:.................SKIPPED\n\n");
+
+ /* SS RAW (PROXIMITY) SENSE TEST */
+ pr_info("SS RAW LP SENSE TEST:\n");
+
+ if (todo->SelfSenseRawLP == 1 || todo->SelfSenseRawGapLP == 1) {
+ columns = ssRawFrame.header.sense_node;
+ rows = 1;/* there are no data for the force channels
+ * because is a sense frame */
+
+ pr_info("SS RAW LP SENSE MIN MAX TEST:\n");
+ if (todo->SelfSenseRawLP == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_RAW_LP_SENSE_MIN_MAX,
+ &thresholds,
+ &trows, &tcolumns);
+ if (ret < OK || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_RAW_SENSE_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMinMax(ssRawFrame.sense_data, rows,
+ columns, thresholds[0],
+ thresholds[1]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax SS RAW SENSE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS RAW LP SENSE MIN MAX TEST:.................FAIL\n");
+ count_fail += 1;
+ if (stop_on_fail) {
+ ret = ERROR_PROD_TEST_DATA |
+ ERROR_TEST_CHECK_FAIL;
+ goto ERROR_LIMITS;
+ }
+ } else
+ pr_info("SS RAW SENSE MIN MAX TEST:.................OK\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("SS RAW LP SENSE MIN MAX TEST:.................SKIPPED\n");
+
+ pr_info("SS RAW LP SENSE GAP TEST:\n");
+ if (todo->SelfSenseRawGapLP == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_RAW_LP_SENSE_GAP,
+ &thresholds, &trows,
+ &tcolumns);
+ if (ret < OK || (trows != 1 || tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_RAW_SENSE_GAP failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsGap(ssRawFrame.sense_data, rows,
+ columns, thresholds[0]);
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsGap SS RAW SENSE GAP failed... ERROR = %08X\n",
+ ret);
+ pr_err("SS RAW LP SENSE GAP TEST:.................FAIL\n");
+ count_fail += 1;
+ if (stop_on_fail) {
+ ret = ERROR_PROD_TEST_DATA |
+ ERROR_TEST_CHECK_FAIL;
+ goto ERROR_LIMITS;
+ }
+ } else
+ pr_info("SS RAW LP SENSE GAP TEST:.................OK\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("SS RAW LP SENSE GAP TEST:.................SKIPPED\n");
+
+ kfree(ssRawFrame.sense_data);
+ ssRawFrame.sense_data = NULL;
+ }
+
+ if (count_fail == 0) {
+ pr_info("SS RAW LP testes finished!.................OK\n\n");
+ return OK;
+ } else {
+ pr_err("SS RAW LP testes finished!.................FAILED fails_count = %d\n\n",
+ count_fail);
+ return ERROR_TEST_CHECK_FAIL | ERROR_PROD_TEST_DATA;
+ }
+
+ERROR_LIMITS:
+ if (ssRawFrame.force_data != NULL)
+ kfree(ssRawFrame.force_data);
+ if (ssRawFrame.sense_data != NULL)
+ kfree(ssRawFrame.sense_data);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ return ret;
+}
+
+/**
+ * Perform all the tests selected in a TestTodo variable related to SS Init
+ * data (touch, keys etc..)
+ * @param path_limits name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param stop_on_fail if 1, the test flow stops at the first data check
+ * failure
+ * otherwise it keeps going performing all the selected test
+ * @param todo pointer to a TestToDo variable which select the test to do
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_ss_ix_cx(char *path_limits, int stop_on_fail,
+ TestToDo *todo)
+{
+ int ret;
+ int count_fail = 0;
+
+ int *thresholds = NULL;
+ int trows, tcolumns;
+ int *thresholds_min = NULL;
+ int *thresholds_max = NULL;
+
+ SelfSenseData ssCompData;
+ TotSelfSenseData totCompData;
+
+ u8 *adjhor = NULL;
+ u8 *adjvert = NULL;
+
+ short container;
+
+ u16 *total_adjhor = NULL;
+ u16 *total_adjvert = NULL;
+
+ pr_info("SS IX CX testes are starting...\n");
+ ret = readSelfSenseCompensationData(LOAD_CX_SS_TOUCH, &ssCompData);
+ /* read the SS compensation data */
+ if (ret < 0) {
+ pr_err("production_test_data: readSelfSenseCompensationData failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ return ret | ERROR_PROD_TEST_DATA;
+ }
+
+ ret = readTotSelfSenseCompensationData(LOAD_PANEL_CX_TOT_SS_TOUCH,
+ &totCompData);
+ /* read the TOT SS compensation data */
+ if (ret < 0) {
+ pr_err("production_test_data: readTotSelfSenseCompensationData failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ kfree(ssCompData.ix2_fm);
+ kfree(ssCompData.ix2_sn);
+ kfree(ssCompData.cx2_fm);
+ kfree(ssCompData.cx2_sn);
+ return ret | ERROR_PROD_TEST_DATA;
+ }
+
+ /************* SS FORCE IX **************/
+ /* SS IX1 FORCE TEST */
+ pr_info("SS IX1 FORCE TEST:\n");
+ if (todo->SelfForceIx1 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_IX1_FORCE_MIN_MAX,
+ &thresholds, &trows, &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_IX1_FORCE_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ container = (short)ssCompData.f_ix1;
+ ret = checkLimitsMinMax(&container, 1, 1, thresholds[0],
+ thresholds[1]);
+ /* check the limits */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax SS IX1 FORCE TEST failed... ERROR COUNT = %d\n",
+ ret);
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS IX1 FORCE TEST:.................OK\n\n");
+ } else
+ pr_info("SS IX1 FORCE TEST:.................SKIPPED\n\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ /* SS IX2 FORCE TEST */
+ pr_info("SS IX2 FORCE MIN MAX TEST:\n");
+ if (todo->SelfForceIx2 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_IX2_FORCE_MAP_MIN,
+ &thresholds_min, &trows,
+ &tcolumns);
+ /* load the min thresholds */
+ if (ret < 0 || (trows != ssCompData.header.force_node ||
+ tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_IX2_FORCE_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_IX2_FORCE_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != ssCompData.header.force_node ||
+ tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_IX2_FORCE_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapFromU(ssCompData.ix2_fm,
+ ssCompData.header.force_node, 1,
+ thresholds_min,
+ thresholds_max); /* check the
+ * values with
+ * thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS IX2 FORCE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS IX2 FORCE MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS IX2 FORCE MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("SS IX2 FORCE MIN MAX TEST:.................SKIPPED\n\n");
+
+ pr_info("SS IX2 FORCE ADJ TEST:\n");
+ if (todo->SelfForceIx2Adj == 1) {
+ /* SS IX2 FORCE ADJV TEST */
+ pr_info("SS IX2 FORCE ADJVERT TEST:\n");
+ ret = computeAdjVertFromU(ssCompData.ix2_fm,
+ ssCompData.header.force_node, 1,
+ &adjvert);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjVert SS IX2 FORCE ADJV failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("SS IX2 FORCE ADJV computed!\n");
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_IX2_FORCE_ADJV_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns); /* load the max
+ * thresholds */
+ if (ret < 0 || (trows != ssCompData.header.force_node - 1 ||
+ tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_IX2_FORCE_ADJV_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdj(adjvert, ssCompData.header.force_node -
+ 1, 1, thresholds_max); /* check the
+ * values with
+ * thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS IX2 FORCE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS IX2 FORCE ADJV TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS IX2 FORCE ADJV TEST:.................OK\n\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(adjvert);
+ adjvert = NULL;
+ } else
+ pr_info("SS IX2 FORCE ADJ TEST:.................SKIPPED\n\n");
+
+ /* SS TOTAL FORCE IX */
+ pr_info("SS TOTAL IX FORCE TEST:\n");
+ if (todo->SelfForceIxTotal == 1 || todo->SelfForceIxTotalAdj == 1) {
+ pr_info("SS TOTAL IX FORCE MIN MAX TEST:\n");
+ if (todo->SelfForceIxTotal == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_IX_FORCE_MAP_MIN,
+ &thresholds_min,
+ &trows, &tcolumns);
+ /* load the min thresholds */
+ if (ret < 0 || (trows !=
+ totCompData.header.force_node ||
+ tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_IX_FORCE_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_IX_FORCE_MAP_MAX,
+ &thresholds_max,
+ &trows, &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows !=
+ totCompData.header.force_node ||
+ tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_IX_FORCE_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapTotalFromU(totCompData.ix_fm,
+ totCompData.header.
+ force_node, 1,
+ thresholds_min,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS TOTAL IX FORCE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS TOTAL IX FORCE MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS TOTAL IX FORCE MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("SS TOTAL IX FORCE MIN MAX TEST:.................SKIPPED\n");
+
+ pr_info("SS TOTAL IX FORCE ADJ TEST:\n");
+ if (todo->SelfForceIxTotalAdj == 1) {
+ /* SS TOTAL IX FORCE ADJV TEST */
+ pr_info("SS TOTAL IX FORCE ADJVERT TEST:\n");
+ ret = computeAdjVertTotalFromU(totCompData.ix_fm,
+ totCompData.header.
+ force_node, 1,
+ &total_adjvert);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjVert SS TOTAL IX FORCE ADJV failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("SS TOTAL IX FORCE ADJV computed!\n");
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_IX_FORCE_ADJV_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != totCompData.header.force_node -
+ 1 || tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_IX_FORCE_ADJV_MAP_MAX... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdjTotal(total_adjvert,
+ totCompData.header.
+ force_node - 1, 1,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS TOTAL IX FORCE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS TOTAL IX FORCE ADJV TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS TOTAL IX FORCE ADJV TEST:.................OK\n\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(total_adjvert);
+ total_adjvert = NULL;
+ } else
+ pr_info("SS TOTAL IX FORCE ADJ TEST:.................SKIPPED\n");
+ } else
+ pr_info("SS TOTAL IX FORCE TEST:.................SKIPPED\n\n");
+
+
+ /************** SS SENSE IX **************/
+ /* SS IX1 SENSE TEST */
+ pr_info("SS IX1 SENSE TEST:\n");
+ if (todo->SelfSenseIx1 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_IX1_SENSE_MIN_MAX,
+ &thresholds, &trows, &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_IX1_SENSE_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ container = (short)ssCompData.s_ix1;
+ ret = checkLimitsMinMax(&container, 1, 1, thresholds[0],
+ thresholds[1]);
+ /* check the limits */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax SS IX1 SENSE TEST failed... ERROR COUNT = %d\n",
+ ret);
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS IX1 SENSE TEST:.................OK\n\n");
+ } else
+ pr_info("SS IX1 SENSE TEST:.................SKIPPED\n\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ /* SS IX2 SENSE TEST */
+ pr_info("SS IX2 SENSE MIN MAX TEST:\n");
+ if (todo->SelfSenseIx2 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_IX2_SENSE_MAP_MIN,
+ &thresholds_min, &trows,
+ &tcolumns);
+ /* load the min thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ ssCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_IX2_SENSE_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_IX2_SENSE_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ ssCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_IX2_SENSE_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapFromU(ssCompData.ix2_sn, 1,
+ ssCompData.header.sense_node,
+ thresholds_min, thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS IX2 SENSE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS IX2 SENSE MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS IX2 SENSE MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("SS IX2 SENSE MIN MAX TEST:.................SKIPPED\n\n");
+
+ pr_info("SS IX2 SENSE ADJ TEST:\n");
+ if (todo->SelfSenseIx2Adj == 1) {
+ /* SS IX2 SENSE ADJH TEST */
+ pr_info("SS IX2 SENSE ADJHORIZ TEST:\n");
+ ret = computeAdjHorizFromU(ssCompData.ix2_sn, 1,
+ ssCompData.header.sense_node,
+ &adjhor);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjHoriz SS IX2 SENSE ADJH failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("SS IX2 SENSE ADJ HORIZ computed!\n");
+
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_IX2_SENSE_ADJH_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ ssCompData.header.sense_node - 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_IX2_SENSE_ADJH_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdj(adjhor, 1,
+ ssCompData.header.sense_node - 1,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMapAdj SS IX2 SENSE ADJH failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS IX2 SENSE ADJH TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS IX2 SENSE ADJH TEST:.................OK\n\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(adjhor);
+ adjhor = NULL;
+ } else
+ pr_info("SS IX2 SENSE ADJ TEST:.................SKIPPED\n");
+
+ /* SS TOTAL IX SENSE */
+ pr_info("SS TOTAL IX SENSE TEST:\n");
+ if (todo->SelfSenseIxTotal == 1 || todo->SelfSenseIxTotalAdj == 1) {
+ pr_info("SS TOTAL IX SENSE MIN MAX TEST:\n");
+ if (todo->SelfSenseIxTotal == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_IX_SENSE_MAP_MIN,
+ &thresholds_min,
+ &trows, &tcolumns);
+ /* load the min thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ totCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_IX_SENSE_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_IX_SENSE_MAP_MAX,
+ &thresholds_max,
+ &trows, &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ totCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_IX_SENSE_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapTotalFromU(totCompData.ix_sn, 1,
+ totCompData.header.
+ sense_node,
+ thresholds_min,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS TOTAL IX SENSE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS TOTAL IX SENSE MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS TOTAL IX SENSE MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("SS TOTAL IX SENSE MIN MAX TEST:.................SKIPPED\n");
+
+
+ pr_info("SS TOTAL IX SENSE ADJ TEST:\n");
+ if (todo->SelfSenseIxTotalAdj == 1) {
+ /* SS TOTAL IX SENSE ADJH TEST */
+ pr_info("SS TOTAL IX SENSE ADJHORIZ TEST:\n");
+ ret = computeAdjHorizTotalFromU(totCompData.ix_sn, 1,
+ totCompData.header.
+ sense_node,
+ &total_adjhor);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjHoriz SS TOTAL IX SENSE ADJH failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("SS TOTAL IX SENSE ADJ HORIZ computed!\n");
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_IX_SENSE_ADJH_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ totCompData.header.sense_node - 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_IX_SENSE_ADJH_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdjTotal(total_adjhor, 1,
+ totCompData.header.
+ sense_node - 1,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMapAdj SS TOTAL IX SENSE ADJH failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS TOTAL IX SENSE ADJH TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS TOTAL IX SENSE ADJH TEST:.................OK\n\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(total_adjhor);
+ total_adjhor = NULL;
+ } else
+ pr_info("SS TOTAL IX SENSE ADJ TEST:.................SKIPPED\n");
+ } else
+ pr_info("SS TOTAL IX SENSE TEST:.................SKIPPED\n");
+
+ /************* SS SENSE CX **************/
+ /* SS CX1 FORCE TEST */
+ pr_info("SS CX1 FORCE TEST:\n");
+ if (todo->SelfForceCx1 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_CX1_FORCE_MIN_MAX,
+ &thresholds, &trows, &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_CX1_FORCE_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ container = (short)ssCompData.f_cx1;
+ ret = checkLimitsMinMax(&container, 1, 1, thresholds[0],
+ thresholds[1]);
+ /* check the limits */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax SS CX1 FORCE TEST failed... ERROR COUNT = %d\n",
+ ret);
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS CX1 FORCE TEST:.................OK\n\n");
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("SS CX1 FORCE TEST:.................SKIPPED\n\n");
+
+ /* SS CX2 FORCE TEST */
+ pr_info("SS CX2 FORCE MIN MAX TEST:\n");
+ if (todo->SelfForceCx2 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_CX2_FORCE_MAP_MIN,
+ &thresholds_min, &trows,
+ &tcolumns);
+ /* load the min thresholds */
+ if (ret < 0 || (trows != ssCompData.header.force_node ||
+ tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_CX2_FORCE_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_CX2_FORCE_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != ssCompData.header.force_node ||
+ tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_CX2_FORCE_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMap(ssCompData.cx2_fm,
+ ssCompData.header.force_node, 1,
+ thresholds_min,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS CX2 FORCE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS CX2 FORCE MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS CX2 FORCE MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("SS CX2 FORCE MIN MAX TEST:.................SKIPPED\n");
+
+ pr_info("SS CX2 FORCE ADJ TEST:\n");
+ if (todo->SelfForceCx2Adj == 1) {
+ /* SS CX2 FORCE ADJV TEST */
+ pr_info("SS CX2 FORCE ADJVERT TEST:\n");
+ ret = computeAdjVert(ssCompData.cx2_fm,
+ ssCompData.header.force_node, 1, &adjvert);
+ /* compute the ADJV for CX2 FORCE */
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjVert SS CX2 FORCE ADJV failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("SS CX2 FORCE ADJV computed!\n");
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_CX2_FORCE_ADJV_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != ssCompData.header.force_node - 1 ||
+ tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_CX2_FORCE_ADJV_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdj(adjvert, ssCompData.header.force_node -
+ 1, 1, thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS IX2 FORCE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS CX2 FORCE ADJV TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS CX2 FORCE ADJV TEST:.................OK\n\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(adjvert);
+ adjvert = NULL;
+ } else
+ pr_info("SS CX2 FORCE ADJ TEST:.................SKIPPED\n\n");
+
+ /* SS TOTAL CX FORCE */
+ pr_info("SS TOTAL CX FORCE TEST:\n");
+ if (todo->SelfForceCxTotal == 1 || todo->SelfForceCxTotalAdj == 1) {
+ pr_info("SS TOTAL CX FORCE MIN MAX TEST:\n");
+ if (todo->SelfForceCxTotal == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_CX_FORCE_MAP_MIN,
+ &thresholds_min,
+ &trows, &tcolumns);
+ /* load the min thresholds */
+ if (ret < 0 || (trows !=
+ totCompData.header.force_node ||
+ tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_CX_FORCE_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_CX_FORCE_MAP_MAX,
+ &thresholds_max,
+ &trows, &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows !=
+ totCompData.header.force_node ||
+ tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_CX_FORCE_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapTotal(totCompData.cx_fm,
+ totCompData.header.force_node,
+ 1, thresholds_min,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS TOTAL FORCE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS TOTAL FORCE MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS TOTAL FORCE MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("SS TOTAL CX FORCE MIN MAX TEST:.................SKIPPED\n");
+
+ /* SS TOTAL CX FORCE ADJV TEST */
+ pr_info("SS TOTAL CX FORCE ADJ TEST:\n");
+ if (todo->SelfForceCxTotalAdj == 1) {
+ pr_info("SS TOTAL CX FORCE ADJVERT TEST:\n");
+ ret = computeAdjVertTotal(totCompData.cx_fm,
+ totCompData.header.force_node,
+ 1, &total_adjvert);
+ /* compute the ADJV for CX2 FORCE */
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjVert SS TOTAL CX FORCE ADJV failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("SS TOTAL CX FORCE ADJV computed!\n");
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_CX_FORCE_ADJV_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != totCompData.header.force_node -
+ 1 || tcolumns != 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_CX_FORCE_ADJV_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdjTotal(total_adjvert,
+ totCompData.header.
+ force_node - 1, 1,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS TOTAL CX FORCE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS TOTAL CX FORCE ADJV TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS TOTAL CX FORCE ADJV TEST:.................OK\n\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(total_adjvert);
+ total_adjvert = NULL;
+ } else
+ pr_info("SS TOTAL CX FORCE ADJ TEST:.................SKIPPED\n");
+ } else
+ pr_info("SS TOTAL CX FORCE TEST:.................SKIPPED\n\n");
+
+
+
+ /************* SS SENSE CX *************/
+ /* SS CX1 SENSE TEST */
+ pr_info("SS CX1 SENSE TEST:\n");
+ if (todo->SelfSenseCx1 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_CX1_SENSE_MIN_MAX,
+ &thresholds, &trows, &tcolumns);
+ if (ret < 0 || (trows != 1 || tcolumns != 2)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_CX1_SENSE_MIN_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ container = (short)ssCompData.s_cx1;
+ ret = checkLimitsMinMax(&container, 1, 1, thresholds[0],
+ thresholds[1]);
+ /* check the limits */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMinMax SS CX1 SENSE TEST failed... ERROR COUNT = %d\n",
+ ret);
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS CX1 SENSE TEST:.................OK\n\n");
+
+ kfree(thresholds);
+ thresholds = NULL;
+ } else
+ pr_info("SS CX1 SENSE TEST:.................SKIPPED\n\n");
+
+
+ /* SS CX2 SENSE TEST */
+ pr_info("SS CX2 SENSE MIN MAX TEST:\n");
+ if (todo->SelfSenseCx2 == 1) {
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_CX2_SENSE_MAP_MIN,
+ &thresholds_min, &trows,
+ &tcolumns);
+ /* load the min thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ ssCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_CX2_SENSE_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_CX2_SENSE_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ ssCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_CX2_SENSE_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMap(ssCompData.cx2_sn, 1,
+ ssCompData.header.sense_node,
+ thresholds_min, thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS CX2 SENSE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS CX2 SENSE MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS CX2 SENSE MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("SS CX2 SENSE MIN MAX TEST:.................SKIPPED\n");
+
+ pr_info("SS CX2 SENSE ADJ TEST:\n");
+ if (todo->SelfSenseCx2Adj == 1) {
+ /* SS CX2 SENSE ADJH TEST */
+ pr_info("SS CX2 SENSE ADJHORIZ TEST:\n");
+ ret = computeAdjHoriz(ssCompData.cx2_sn, 1,
+ ssCompData.header.sense_node, &adjhor);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjHoriz SS CX2 SENSE ADJH failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("SS CX2 SENSE ADJH computed!\n");
+
+
+ ret = parseProductionTestLimits(path_limits, &limit_file,
+ SS_CX2_SENSE_ADJH_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ ssCompData.header.sense_node - 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_IX2_SENSE_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdj(adjhor, 1,
+ ssCompData.header.sense_node - 1,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMapAdj SS CX2 SENSE ADJH failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS CX2 SENSE ADJH TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS CX2 SENSE ADJH TEST:.................OK\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(adjhor);
+ adjhor = NULL;
+ } else
+ pr_info("SS CX2 SENSE ADJ TEST:.................SKIPPED\n\n");
+
+ /* SS TOTAL CX SENSE */
+ pr_info("SS TOTAL CX SENSE TEST:\n");
+ if (todo->SelfSenseCxTotal == 1 || todo->SelfSenseCxTotalAdj == 1) {
+ pr_info("SS TOTAL CX SENSE MIN MAX TEST:\n");
+ if (todo->SelfSenseCxTotal == 1) {
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_CX_SENSE_MAP_MIN,
+ &thresholds_min,
+ &trows, &tcolumns);
+ /* load the min thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ totCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_CX_SENSE_MAP_MIN failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_CX_SENSE_MAP_MAX,
+ &thresholds_max,
+ &trows, &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ totCompData.header.sense_node)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_CX_SENSE_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapTotal(totCompData.cx_sn, 1,
+ totCompData.header.sense_node,
+ thresholds_min,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMap SS TOTAL CX SENSE failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS TOTAL CX SENSE MIN MAX TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS TOTAL CX SENSE MIN MAX TEST:.................OK\n\n");
+
+ kfree(thresholds_min);
+ thresholds_min = NULL;
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ } else
+ pr_info("SS TOTAL CX SENSE MIN MAX TEST:.................SKIPPED\n");
+
+
+ /* SS TOTAL IX SENSE ADJH TEST */
+ pr_info("SS TOTAL CX SENSE ADJ TEST:\n");
+ if (todo->SelfSenseCxTotalAdj == 1) {
+ pr_info("SS TOTAL CX SENSE ADJHORIZ TEST:\n");
+ ret = computeAdjHorizTotal(totCompData.cx_sn, 1,
+ totCompData.header.sense_node,
+ &total_adjhor);
+ if (ret < 0) {
+ pr_err("production_test_data: computeAdjHoriz SS TOTAL CX SENSE ADJH failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+ pr_info("SS TOTAL CX SENSE ADJ HORIZ computed!\n");
+
+
+ ret = parseProductionTestLimits(path_limits,
+ &limit_file,
+ SS_TOTAL_CX_SENSE_ADJH_MAP_MAX,
+ &thresholds_max, &trows,
+ &tcolumns);
+ /* load the max thresholds */
+ if (ret < 0 || (trows != 1 || tcolumns !=
+ totCompData.header.sense_node - 1)) {
+ pr_err("production_test_data: parseProductionTestLimits SS_TOTAL_CX_SENSE_ADJH_MAP_MAX failed... ERROR %08X\n",
+ ERROR_PROD_TEST_DATA);
+ ret |= ERROR_PROD_TEST_DATA;
+ goto ERROR_LIMITS;
+ }
+
+ ret = checkLimitsMapAdjTotal(total_adjhor, 1,
+ totCompData.header.
+ sense_node - 1,
+ thresholds_max);
+ /* check the values with thresholds */
+ if (ret != OK) {
+ pr_err("production_test_data: checkLimitsMapAdj SS TOTAL CX SENSE ADJH failed... ERROR COUNT = %d\n",
+ ret);
+ pr_err("SS TOTAL CX SENSE ADJH TEST:.................FAIL\n\n");
+ count_fail += 1;
+ if (stop_on_fail)
+ goto ERROR;
+ } else
+ pr_info("SS TOTAL CX SENSE ADJH TEST:.................OK\n\n");
+
+ kfree(thresholds_max);
+ thresholds_max = NULL;
+ kfree(total_adjhor);
+ total_adjhor = NULL;
+ } else
+ pr_info("SS TOTAL CX SENSE ADJ TEST:.................SKIPPED\n");
+ } else
+ pr_info("SS TOTAL CX SENSE TEST:.................SKIPPED\n");
+
+
+
+ERROR:
+ if (count_fail == 0) {
+ kfree(ssCompData.ix2_fm);
+ ssCompData.ix2_fm = NULL;
+ kfree(ssCompData.ix2_sn);
+ ssCompData.ix2_sn = NULL;
+ kfree(ssCompData.cx2_fm);
+ ssCompData.cx2_fm = NULL;
+ kfree(ssCompData.cx2_sn);
+ ssCompData.cx2_sn = NULL;
+ kfree(totCompData.ix_fm);
+ totCompData.ix_fm = NULL;
+ kfree(totCompData.ix_sn);
+ totCompData.ix_sn = NULL;
+ kfree(totCompData.cx_fm);
+ totCompData.cx_fm = NULL;
+ kfree(totCompData.cx_sn);
+ totCompData.cx_sn = NULL;
+ pr_info("SS IX CX testes finished!.................OK\n\n");
+ return OK;
+ } else {
+ /* print all kind of data in just one row for readability reason */
+ print_frame_u8("SS Init Data Ix2_fm = ", array1dTo2d_u8(
+ ssCompData.ix2_fm,
+ ssCompData.header.force_node, 1),
+ ssCompData.header.force_node, 1);
+ print_frame_i8("SS Init Data Cx2_fm = ", array1dTo2d_i8(
+ ssCompData.cx2_fm,
+ ssCompData.header.force_node, 1),
+ ssCompData.header.force_node, 1);
+ print_frame_u8("SS Init Data Ix2_sn = ", array1dTo2d_u8(
+ ssCompData.ix2_sn,
+ ssCompData.header.sense_node,
+ ssCompData.header.sense_node), 1,
+ ssCompData.header.sense_node);
+ print_frame_i8("SS Init Data Cx2_sn = ", array1dTo2d_i8(
+ ssCompData.cx2_sn,
+ ssCompData.header.sense_node,
+ ssCompData.header.sense_node), 1,
+ ssCompData.header.sense_node);
+ print_frame_u16("TOT SS Init Data Ix_fm = ", array1dTo2d_u16(
+ totCompData.ix_fm,
+ totCompData.header.force_node, 1),
+ totCompData.header.force_node, 1);
+ print_frame_short("TOT SS Init Data Cx_fm = ",
+ array1dTo2d_short(totCompData.cx_fm,
+ totCompData.header.
+ force_node, 1),
+ totCompData.header.force_node, 1);
+ print_frame_u16("TOT SS Init Data Ix_sn = ", array1dTo2d_u16(
+ totCompData.ix_sn,
+ totCompData.header.sense_node,
+ totCompData.header.sense_node), 1,
+ totCompData.header.sense_node);
+ print_frame_short("TOT SS Init Data Cx_sn = ",
+ array1dTo2d_short(totCompData.cx_sn,
+ totCompData.header.
+ sense_node,
+ totCompData.header.
+ sense_node),
+ 1, totCompData.header.sense_node);
+ pr_err("SS IX CX testes finished!.................FAILED fails_count = %d\n\n",
+ count_fail);
+ if (thresholds != NULL)
+ kfree(thresholds);
+ if (thresholds_min != NULL)
+ kfree(thresholds_min);
+ if (thresholds_max != NULL)
+ kfree(thresholds_max);
+ if (adjhor != NULL)
+ kfree(adjhor);
+ if (adjvert != NULL)
+ kfree(adjvert);
+ if (total_adjhor != NULL)
+ kfree(total_adjhor);
+ if (total_adjvert != NULL)
+ kfree(total_adjvert);
+ if (ssCompData.ix2_fm != NULL)
+ kfree(ssCompData.ix2_fm);
+ if (ssCompData.ix2_sn != NULL)
+ kfree(ssCompData.ix2_sn);
+ if (ssCompData.cx2_fm != NULL)
+ kfree(ssCompData.cx2_fm);
+ if (ssCompData.cx2_sn != NULL)
+ kfree(ssCompData.cx2_sn);
+ if (totCompData.ix_fm != NULL)
+ kfree(totCompData.ix_fm);
+ if (totCompData.ix_sn != NULL)
+ kfree(totCompData.ix_sn);
+ if (totCompData.cx_fm != NULL)
+ kfree(totCompData.cx_fm);
+ if (totCompData.cx_sn != NULL)
+ kfree(totCompData.cx_sn);
+ return ERROR_TEST_CHECK_FAIL | ERROR_PROD_TEST_DATA;
+ }
+
+ERROR_LIMITS:
+ if (thresholds != NULL)
+ kfree(thresholds);
+ if (thresholds_min != NULL)
+ kfree(thresholds_min);
+ if (thresholds_max != NULL)
+ kfree(thresholds_max);
+ if (adjhor != NULL)
+ kfree(adjhor);
+ if (adjvert != NULL)
+ kfree(adjvert);
+ if (total_adjhor != NULL)
+ kfree(total_adjhor);
+ if (total_adjvert != NULL)
+ kfree(total_adjvert);
+ if (ssCompData.ix2_fm != NULL)
+ kfree(ssCompData.ix2_fm);
+ if (ssCompData.ix2_sn != NULL)
+ kfree(ssCompData.ix2_sn);
+ if (ssCompData.cx2_fm != NULL)
+ kfree(ssCompData.cx2_fm);
+ if (ssCompData.cx2_sn != NULL)
+ kfree(ssCompData.cx2_sn);
+ if (totCompData.ix_fm != NULL)
+ kfree(totCompData.ix_fm);
+ if (totCompData.ix_sn != NULL)
+ kfree(totCompData.ix_sn);
+ if (totCompData.cx_fm != NULL)
+ kfree(totCompData.cx_fm);
+ if (totCompData.cx_sn != NULL)
+ kfree(totCompData.cx_sn);
+ return ret;
+}
+
+/**
+ * Perform a complete Data Test check of the IC
+ * @param path_limits name of Production Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param stop_on_fail if 1, the test flow stops at the first data check
+ * failure
+ * otherwise it keeps going performing all the selected test
+ * @param todo pointer to a TestToDo variable which select the test to do
+ * @return OK if success or an error code which specify the type of error
+ */
+int production_test_data(char *path_limits, int stop_on_fail, TestToDo *todo)
+{
+ int res = OK, ret;
+
+ if (todo == NULL) {
+ pr_err("production_test_data: No TestToDo specified!! ERROR = %08X\n",
+ (ERROR_OP_NOT_ALLOW | ERROR_PROD_TEST_DATA));
+ return ERROR_OP_NOT_ALLOW | ERROR_PROD_TEST_DATA;
+ }
+
+
+ pr_info("DATA Production test is starting...\n");
+
+
+ ret = production_test_ms_raw(path_limits, stop_on_fail, todo);
+ res |= ret;
+ if (ret < 0) {
+ pr_err("production_test_data: production_test_ms_raw failed... ERROR = %08X\n",
+ ret);
+ if (stop_on_fail == 1)
+ goto END;
+ }
+
+
+
+ ret = production_test_ms_cx(path_limits, stop_on_fail, todo);
+ res |= ret;
+ if (ret < 0) {
+ pr_err("production_test_data: production_test_ms_cx failed... ERROR = %08X\n",
+ ret);
+ if (stop_on_fail == 1)
+ goto END;
+ }
+
+
+ ret = production_test_ss_raw(path_limits, stop_on_fail, todo);
+ res |= ret;
+ if (ret < 0) {
+ pr_err("production_test_data: production_test_ss_raw failed... ERROR = %08X\n",
+ ret);
+ if (stop_on_fail == 1)
+ goto END;
+ }
+
+ ret = production_test_ss_ix_cx(path_limits, stop_on_fail, todo);
+ res |= ret;
+ if (ret < 0) {
+ pr_err("production_test_data: production_test_ss_ix_cx failed... ERROR = %08X\n",
+ ret);
+ if (stop_on_fail == 1)
+ goto END;
+ }
+
+END:
+ freeLimitsFile(&limit_file); /* /< release the limit file loaded
+ * during the test */
+ if (res < OK)
+ pr_err("DATA Production test failed!\n");
+ else
+ pr_info("DATA Production test finished!\n");
+ return res;
+}
+
+
+/*************** TP Sensitivity calibration API ********************/
+
+/**
+ * Perform the Pre Calibration MS Test when the stimpad is down
+ * @param[out] frame pointer to the frame which will contain
+ * the average frame resulting from the test
+ * @param target reference value for the frame, each node should be
+ * around +-percentage% this value
+ * @param percentage percentage of the target value which define
+ * the valid interval for the frame, if <0 the test will be skipped
+ * @return OK if success or an error code which specify the type of error
+ */
+int tp_sensitivity_test_pre_cal_ms(MutualSenseFrame *finalFrame, short target,
+ int percentage)
+{
+ int ret = OK;
+ int count = 0, i = 0, j = 0;
+ short min, max;
+ MutualSenseFrame frame;
+
+ finalFrame->node_data = NULL;
+
+
+ pr_info("%s: Start TP sensitivity MS Pre Cal...\n", __func__);
+ pr_info("%s: IMPORTANT!!! Stimpad should be on the display of the device!\n",
+ __func__);
+ ret = getMSFrame3(MS_STRENGTH, &frame);
+ if (ret < OK) {
+ pr_err("%s: can not read MS Frame... ERROR %08X\n",
+ __func__, ret);
+ goto ERROR;
+ }
+
+ finalFrame->header = frame.header;
+ finalFrame->node_data_size = frame.node_data_size;
+
+ finalFrame->node_data = (short *)kzalloc(frame.node_data_size *
+ sizeof(short), GFP_KERNEL);
+ if (finalFrame->node_data == NULL) {
+ pr_err("%s: can not allocate node_data ERROR %08X\n",
+ __func__, ERROR_ALLOC | ERROR_GET_FRAME);
+ ret = ERROR_ALLOC | ERROR_GET_FRAME;
+ goto ERROR;
+ }
+
+ /* collecting frames */
+ do {
+ for (i = 0; i < finalFrame->node_data_size; i++) {
+ finalFrame->node_data[i] += (frame.node_data[i] * 10) /
+ SENS_TEST_NUM_FRAMES;
+ }
+
+ if (frame.node_data != NULL) {
+ kfree(frame.node_data);
+ frame.node_data = NULL;
+ }
+
+ count++;
+
+ /* exclude one more reading at the end*/
+ if (count < SENS_TEST_NUM_FRAMES)
+ ret = getMSFrame3(MS_STRENGTH, &frame);
+ } while ((count < SENS_TEST_NUM_FRAMES) && (ret >= OK));
+
+ if (ret < OK) {
+ pr_err("%s: Error while capturing the frame %d! ERROR %08X\n",
+ __func__, count, ret);
+ goto ERROR;
+ }
+
+ ret = OK;
+ /* check against +-percentage% target */
+ pr_info("%s: Computing average frame...\n", __func__);
+
+ min = target - (target * percentage / 100);
+ max = target + (target * percentage / 100);
+
+ for (i = 0; i < finalFrame->header.force_node; i++) {
+ for (j = 0; j < finalFrame->header.sense_node; j++) {
+ finalFrame->node_data[i *
+ finalFrame->header.sense_node +
+ j] /= 10;
+ /*if percentage is <0 skip this test, just collect data */
+ if ((percentage > 0) &&
+ ((finalFrame->node_data[i * finalFrame->header.
+ sense_node
+ + j] >
+ max) ||
+ (finalFrame->node_data[i *
+ finalFrame->header.
+ sense_node
+ + j] <
+ min))) {
+ pr_err("%s: MS Force Node[%d, %d] = %d exceed limit [%d, %d]\n",
+ __func__, i, j,
+ finalFrame->node_data[i *
+ finalFrame
+ ->header.
+ sense_node + j],
+ min, max);
+ ret = ERROR_TEST_CHECK_FAIL;
+ }
+ }
+ }
+
+
+ /* print average frame in the log */
+ print_frame_short("MS FS Mean =",
+ array1dTo2d_short(
+ finalFrame->node_data,
+ finalFrame->node_data_size,
+ finalFrame->header.sense_node),
+ finalFrame->header.force_node,
+ finalFrame->header.sense_node);
+
+ if (ret != OK)
+ pr_err("%s: TP sensitivity MS Pre Cal test FAILED... ERROR %08X\n",
+ __func__, ret);
+ else
+ pr_info("%s: TP sensitivity MS Pre Cal FINISHED!\n",
+ __func__);
+
+ return ret;
+
+
+ERROR:
+ if (frame.node_data != NULL) {
+ kfree(frame.node_data);
+ frame.node_data = NULL;
+ }
+
+
+ if (finalFrame->node_data != NULL) {
+ kfree(finalFrame->node_data);
+ finalFrame->node_data = NULL;
+ }
+
+ return ret;
+}
+
+
+
+/**
+ * Perform the Pre Calibration SS Test when the stimpad is down
+ * @param[out] frame pointer to the frame which will contain the average frame
+ * resulting from the test
+ * @param target reference value for the frame, each node should be around
+ * +-percentage% this value
+ * @param percentage percentage of the target value which define the valid
+ * interval for the frame
+ * @return OK if success or an error code which specify the type of error
+ */
+int tp_sensitivity_test_pre_cal_ss(SelfSenseFrame *finalFrame, short target,
+ int percentage)
+{
+ int ret = OK;
+ int count = 0, i = 0;
+ short min, max;
+ SelfSenseFrame frame;
+ int *temp_force = NULL;
+ int *temp_sense = NULL;
+
+ finalFrame->force_data = NULL;
+ finalFrame->sense_data = NULL;
+
+ pr_info("%s: Start TP sensitivity SS Pre Cal...\n", __func__);
+ pr_info("%s: IMPORTANT!!! Stimpad should be on the display of the device!\n",
+ __func__);
+ ret = getSSFrame3(SS_STRENGTH, &frame);
+ if (ret < OK) {
+ pr_err("%s: can not read SS Frame... ERROR %08X\n",
+ __func__, ret);
+ goto ERROR;
+ }
+
+ finalFrame->header = frame.header;
+
+ finalFrame->force_data = (short *)kzalloc(frame.header.force_node *
+ sizeof(short), GFP_KERNEL);
+ temp_force = (int *)kzalloc(frame.header.force_node *
+ sizeof(int), GFP_KERNEL);
+ finalFrame->sense_data = (short *)kzalloc(frame.header.sense_node *
+ sizeof(short), GFP_KERNEL);
+ temp_sense = (int *)kzalloc(frame.header.sense_node *
+ sizeof(int), GFP_KERNEL);
+ if (finalFrame->force_data == NULL ||
+ temp_force == NULL ||
+ finalFrame->sense_data == NULL ||
+ temp_sense == NULL) {
+
+ pr_err("%s: can not allocate memory ERROR %08X\n",
+ __func__, ERROR_ALLOC | ERROR_GET_FRAME);
+ ret = ERROR_ALLOC | ERROR_GET_FRAME;
+ goto ERROR;
+ }
+
+ /* collecting frames */
+ do {
+ for (i = 0; i < finalFrame->header.force_node; i++)
+ temp_force[i] += frame.force_data[i];
+
+ for (i = 0; i < finalFrame->header.sense_node; i++)
+ temp_sense[i] += frame.sense_data[i];
+
+ count++;
+
+ if (frame.force_data != NULL) {
+ kfree(frame.force_data);
+ frame.force_data = NULL;
+ }
+ if (frame.sense_data != NULL) {
+ kfree(frame.sense_data);
+ frame.sense_data = NULL;
+ }
+
+ /* exclude one more reading at the end*/
+ if (count < SENS_TEST_NUM_FRAMES)
+ ret = getSSFrame3(SS_STRENGTH, &frame);
+ } while ((count < SENS_TEST_NUM_FRAMES) && (ret >= OK));
+
+ if (ret < OK) {
+ pr_err("%s: Error while capturing the frame %d! ERROR %08X\n",
+ __func__, count, ret);
+ goto ERROR;
+ }
+
+ ret = OK;
+
+ /* compute the average and check against +-percentage% target */
+ min = target - (target * percentage / 100);
+ max = target + (target * percentage / 100);
+
+ for (i = 0; i < finalFrame->header.force_node; i++) {
+ finalFrame->force_data[i] = temp_force[i] /
+ SENS_TEST_NUM_FRAMES;
+ if ((percentage > 0) && ((finalFrame->force_data[i] > max) ||
+ (finalFrame->force_data[i] < min))) {
+ pr_err("%s: SS Force Node[%d] = %d exceed limit [%d, %d]\n",
+ __func__, i, finalFrame->force_data[i],
+ min, max);
+ ret = ERROR_TEST_CHECK_FAIL;
+ }
+ }
+
+ for (i = 0; i < finalFrame->header.sense_node; i++) {
+ finalFrame->sense_data[i] = temp_sense[i] /
+ SENS_TEST_NUM_FRAMES;
+ if ((finalFrame->sense_data[i] > max) ||
+ (finalFrame->sense_data[i] < min)) {
+ pr_err("%s: SS Sense Node[%d] = %d exceed limit [%d, %d]\n",
+ __func__, i, finalFrame->sense_data[i],
+ min, max);
+ ret = ERROR_TEST_CHECK_FAIL;
+ }
+ }
+
+ /* print average frame in the log */
+ print_frame_short("SS FS force Mean =",
+ array1dTo2d_short(
+ finalFrame->force_data,
+ finalFrame->header.force_node,
+ 1),
+ finalFrame->header.force_node, 1);
+ print_frame_short("SS FS sense Mean =",
+ array1dTo2d_short(
+ finalFrame->sense_data,
+ finalFrame->header.sense_node,
+ finalFrame->header.sense_node),
+ 1, finalFrame->header.sense_node);
+
+
+ kfree(temp_force);
+ temp_force = NULL;
+
+ kfree(temp_sense);
+ temp_sense = NULL;
+
+ if (ret < OK)
+ pr_err("%s: TP sensitivity SS Pre Cal test FAILED... ERROR %08X\n",
+ __func__, ret);
+ else {
+ pr_info("%s: TP sensitivity SS Pre Cal FINISHED!\n",
+ __func__);
+ ret = OK;
+ }
+
+ return ret;
+
+
+ERROR:
+
+ kfree(temp_force);
+ temp_force = NULL;
+
+ kfree(temp_sense);
+ temp_sense = NULL;
+
+ kfree(frame.force_data);
+ frame.force_data = NULL;
+
+ kfree(frame.sense_data);
+ frame.sense_data = NULL;
+
+ kfree(finalFrame->force_data);
+ finalFrame->force_data = NULL;
+
+ kfree(finalFrame->sense_data);
+ finalFrame->sense_data = NULL;
+
+ return ret;
+}
+
+/**
+ * Compute Digital gains for calibration
+ * @param frame pointer to the frame used as reference to compute the gains
+ * @param target reference target value for computing the gains
+ * @param saveGain if 1, will save the gain table into the chip otherwise will
+ * not save it
+ * @return OK if success or an error code which specify the type of error
+ */
+int tp_sensitivity_compute_gains(MutualSenseFrame *frame, short target,
+ int saveGain)
+{
+ int ret = OK;
+ int i = 0;
+ u8 gains[frame->node_data_size];
+
+ if ((frame->node_data == NULL) || (frame->node_data_size == 0)) {
+ pr_err("%s: Invalid frame data passed as argument! ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ memset(gains, 0, frame->node_data_size);
+
+ pr_info("%s: Start to compute Digital Gains...\n", __func__);
+ for (i = 0; i < frame->node_data_size; i++)
+ gains[i] = ((target * 100) / frame->node_data[i]) > 255 ?
+ (u8)(255) : (u8)(((target * 100) /
+ frame->node_data[i]));
+ /* clamp the max value to 255 because gain is only one byte */
+
+
+ /* print average frame in the log */
+ print_frame_u8("MS Digital Gain =",
+ array1dTo2d_u8(
+ gains,
+ frame->node_data_size,
+ frame->header.sense_node),
+ frame->header.force_node,
+ frame->header.sense_node);
+
+
+ /* if(saveGain==1){ */
+ /* write gains into the IC */
+ ret = writeHostDataMemory(LOAD_SENS_CAL_COEFF, gains,
+ frame->header.force_node,
+ frame->header.sense_node, 0, 0, saveGain);
+ if (ret != OK)
+ pr_err("%s: impossible to write digital gains! ERROR %08X\n",
+ __func__, ret);
+ /* } */
+
+ if (ret < OK)
+ pr_err("%s: compute Digital Gains FAILED! ERROR %08X\n",
+ __func__, ret);
+ else {
+ pr_info("%s: compute Digital Gains FINISHED!\n", __func__);
+ ret = OK;
+ }
+
+ return ret;
+}
+
+/**
+ * Perform the Post Calibration MS Test when the stimpad is down
+ * @param[out] finalFrame pointer to the frame which will contain
+ * the average frame resulting from the test
+ * @param[out] deltas pointer to the frame which will contain
+ * the FS Uniform frame (worst_neighborhood/mean)
+ * @param target reference value for the frame, each node should be
+ * around +-percentage% this value
+ * @param percentage percentage of the target value which define
+ * the valid interval for the frame, if <0 the test will be skipped
+ * @param[out] mean_normal pointer to the variable which will contain the mean
+ * of the normal area
+ * @param[out] mean_edge pointer to the variable which will contain the mean of
+ * the edge area
+ * @return OK if success or an error code which specify the type of error
+ */
+int tp_sensitivity_test_post_cal_ms(MutualSenseFrame *finalFrame,
+ MutualSenseFrame *deltas, short target,
+ int percentage, int *mean_normal,
+ int *mean_edge)
+{
+ short currentNode;
+ int final_force_num;
+ int final_sense_num;
+ short *final_node;
+ int delta_sense_num;
+ short *delta_node;
+ short *delta;
+ short adjNode;
+ int ret = OK;
+ int i = 0, j = 0, min, max;
+
+
+ if ((finalFrame == NULL) || (deltas == NULL) || (mean_normal == NULL) ||
+ (mean_edge == NULL)) {
+ pr_err("%s: Invalid arguments Passed! ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ *mean_normal = 0;
+ *mean_edge = 0;
+
+ finalFrame->node_data = NULL;
+ deltas->node_data = NULL;
+
+ pr_info("%s: Start TP sensitivity MS Post Cal...\n", __func__);
+ pr_info("%s: IMPORTANT!!! Stimpad should be on the display of the device!\n",
+ __func__);
+
+ /* collect frames skipping the tests + print on the log */
+ ret = tp_sensitivity_test_pre_cal_ms(finalFrame, target, -1);
+ if (ret < OK) {
+ pr_err("%s: can not collect MS Frame... ERROR %08X\n",
+ __func__, ret);
+ goto ERROR;
+ }
+
+
+ deltas->header = finalFrame->header;
+ deltas->node_data_size = finalFrame->node_data_size;
+
+ deltas->node_data = (short *)kzalloc(deltas->node_data_size *
+ sizeof(short), GFP_KERNEL);
+ if (deltas->node_data == NULL) {
+ pr_err("%s: can not allocate deltas node_data ERROR %08X\n",
+ __func__, ERROR_ALLOC | ERROR_GET_FRAME);
+ ret = ERROR_ALLOC | ERROR_GET_FRAME;
+ goto ERROR;
+ }
+
+ /* compute the average of the whole panel and check against
+ * +-percentage% target */
+ pr_info("%s: Computing average of whole panel and delta for each node...\n",
+ __func__);
+
+ final_force_num = finalFrame->header.force_node;
+ final_sense_num = finalFrame->header.sense_node;
+ final_node = finalFrame->node_data;
+ delta_sense_num = deltas->header.sense_node;
+ delta_node = deltas->node_data;
+
+
+ for (i = 0; i < final_force_num; i++) {
+ for (j = 0; j < final_sense_num; j++) {
+ currentNode = finalFrame->node_data[i *
+ finalFrame->header.
+ sense_node + j];
+ delta = &delta_node[i * delta_sense_num + j];
+
+ if ((i == 0) ||
+ (i == (final_force_num - 1)) ||
+ (j == 0) ||
+ (j == (final_sense_num - 1))) {
+ /* edge nodes */
+ *mean_edge += currentNode;
+ if ((i == 0) ||
+ (i == final_force_num - 1)) {
+ /* need to check adj node up or down for
+ * nodes in the corners */
+ if ((i == 0) &&
+ ((j == 0) ||
+ (j == final_sense_num - 1))) {
+ adjNode = currentNode -
+ final_node[(i + 1) *
+ final_sense_num + j];
+ if (abs(adjNode) > *delta)
+ *delta = abs(adjNode);
+ }
+
+ if ((i == (final_force_num - 1)) &&
+ ((j == 0) ||
+ (j == final_sense_num - 1))) {
+ adjNode = currentNode -
+ final_node[(i - 1) *
+ final_sense_num + j];
+ if (abs(adjNode) > *delta)
+ *delta = abs(adjNode);
+ }
+
+ /* scan the row */
+ if ((j - 1) >= 0) {
+ adjNode = currentNode -
+ final_node[i *
+ final_sense_num +
+ (j - 1)];
+ if (abs(adjNode) > *delta)
+ *delta = abs(adjNode);
+ }
+
+ if ((j + 1) < final_sense_num) {
+ adjNode = currentNode -
+ final_node[i *
+ final_sense_num +
+ (j + 1)];
+ if (abs(adjNode) > *delta)
+ *delta = abs(adjNode);
+ }
+ }
+
+ if ((j == 0) ||
+ (j == final_sense_num - 1)) {
+ /* scan the column */
+ if ((i - 1) >= 0) {
+ adjNode = currentNode -
+ final_node[(i - 1) *
+ final_sense_num + j];
+ if (abs(adjNode) > *delta)
+ *delta = abs(adjNode);
+ }
+
+ if ((i + 1) < final_force_num) {
+ adjNode = currentNode -
+ final_node[(i + 1) *
+ final_sense_num + j];
+ if (abs(adjNode) > *delta)
+ *delta = abs(adjNode);
+ }
+ }
+ } else {
+ /*normal nodes */
+ *mean_normal += currentNode;
+
+ /* picking up the worst difference between
+ * one pixel and its neighbors */
+ if ((i - 1) >= 1) {
+ adjNode = currentNode -
+ final_node[(i - 1) *
+ final_sense_num + j];
+ if (abs(adjNode) > *delta)
+ *delta = abs(adjNode);
+ }
+
+ if ((i + 1) < (final_force_num - 1)) {
+ adjNode = currentNode -
+ final_node[(i + 1) *
+ final_sense_num + j];
+ if (abs(adjNode) > *delta)
+ *delta = abs(adjNode);
+ }
+ if ((j - 1) >= 1) {
+ adjNode = currentNode -
+ final_node[i *
+ final_sense_num + (j - 1)];
+ if (abs(adjNode) > *delta)
+ *delta = abs(adjNode);
+ }
+
+ if ((j + 1) < (final_sense_num - 1)) {
+ adjNode = currentNode -
+ final_node[i *
+ final_sense_num + (j + 1)];
+ if (abs(adjNode) > *delta)
+ *delta = abs(adjNode);
+ }
+ }
+ }
+ }
+
+ *mean_normal /= (finalFrame->header.force_node - 2) *
+ (finalFrame->header.sense_node - 2);
+ *mean_edge /= (finalFrame->header.force_node * 2) +
+ (finalFrame->header.sense_node - 2) * 2;
+
+ pr_info("%s: Normal Frame average = %d\n", __func__, *mean_normal);
+ pr_info("%s: Edge Frame average = %d\n", __func__, *mean_edge);
+ /* compute the average and check against +-% target */
+ min = target - (target * percentage / 100);
+ max = target + (target * percentage / 100);
+
+ if ((percentage > 0) && ((*mean_normal < min) || (*mean_normal >
+ max))) {
+ pr_err("%s: Normal Frame average = %d exceed limit [%d, %d]\n",
+ __func__, *mean_normal, min, max);
+ ret = ERROR_TEST_CHECK_FAIL;
+ }
+
+ if ((percentage > 0) && ((*mean_edge < min) || (*mean_edge > max))) {
+ pr_err("%s: Edge Frame average = %d exceed limit [%d, %d]\n",
+ __func__, *mean_edge, min, max);
+ ret = ERROR_TEST_CHECK_FAIL;
+ }
+
+ for (i = 0; i < deltas->header.force_node; i++) {
+ for (j = 0; j < deltas->header.sense_node; j++) {
+ if ((i == 0) || (i == deltas->header.force_node) ||
+ (j == 0) || (j == deltas->header.sense_node))
+ deltas->node_data[i *
+ deltas->header.sense_node +
+ j] =
+ deltas->node_data[i *
+ deltas->header.
+ sense_node + j] *
+ 100 /
+ (*mean_edge);
+ else
+ deltas->node_data[i *
+ deltas->header.sense_node +
+ j] =
+ deltas->node_data[i *
+ deltas->header.
+ sense_node + j] *
+ 100 /
+ (*mean_normal);
+
+ if ((percentage > 0) && (deltas->node_data[i *
+ deltas->
+ header.
+ sense_node +
+ j] >
+ percentage)) {
+ pr_err("%s: Delta Node[%d, %d] = %d exceed limit [%d]\n",
+ __func__, i, j,
+ deltas->node_data[i *
+ deltas
+ ->header.sense_node +
+ j], percentage);
+ ret = ERROR_TEST_CHECK_FAIL;
+ }
+ }
+ }
+
+
+ /* print average frame in the log */
+ print_frame_short("FS Uniform (%) =",
+ array1dTo2d_short(
+ deltas->node_data,
+ deltas->node_data_size,
+ deltas->header.sense_node),
+ deltas->header.force_node,
+ deltas->header.sense_node);
+
+
+ if (ret < OK)
+ pr_err("%s: TP sensitivity MS Post Cal test FAILED... ERROR %08X\n",
+ __func__, ret);
+ else {
+ pr_info("%s: TP sensitivity MS Post Cal FINISHED!\n",
+ __func__);
+ ret = OK;
+ }
+
+ return ret;
+
+
+ERROR:
+ if (deltas->node_data != NULL) {
+ kfree(deltas->node_data);
+ deltas->node_data = NULL;
+ }
+
+
+ if (finalFrame->node_data != NULL) {
+ kfree(finalFrame->node_data);
+ finalFrame->node_data = NULL;
+ }
+
+ return ret;
+}
+
+
+/**
+ * Compute Digital gains for calibration
+ * @param enter if =1 turn on TP Sensitivity mode, otherwise will turn it off
+ * @param saveGain if 1, will save the gain table into the chip otherwise will
+ * not save it
+ * @return OK if success or an error code which specify the type of error
+ */
+int tp_sensitivity_mode(u8 enter, int saveGain)
+{
+ int res, ret = OK;
+ u8 cmd[4] = { 0xC0, 0x00, 0x00, 0x00 };
+ u8 sett = SPECIAL_WRITE_HOST_MEM_TO_FLASH;
+
+ pr_info("%s: Start TP Sensitivity Mode... enter = %02X\n",
+ __func__, enter);
+ if (enter == 1) {
+ /* enter TP Sensitivity mode*/
+ ret = fts_disableInterrupt();
+ pr_info("%s: Entering TP Sensitivity Mode disabling algos...\n",
+ __func__);
+ cmd[3] = 0x01;
+ res = fts_writeFwCmd(cmd, 4);
+ if (res < OK)
+ pr_err("%s: Error while turning on TP Sens Mode! ERROR %08X\n",
+ __func__, res);
+ } else {
+ /* exit TP Sensitivity mode*/
+ pr_info("%s: Exiting TP Sensitivity Mode enabling algos...\n",
+ __func__);
+ res = fts_writeFwCmd(cmd, 4);
+ if (res < OK)
+ pr_err("%s: Error while turning off TP Sens Mode! ERROR %08X\n",
+ __func__, res);
+
+ if (saveGain == 1) {
+ pr_info("%s: Trigger writing gains into the flash...\n",
+ __func__);
+ ret = writeSysCmd(SYS_CMD_SPECIAL, &sett, 1);
+ if (ret < OK)
+ pr_err("%s: error while writing gains into the flash! ERROR %08X\n",
+ __func__, res);
+ }
+
+ res |= senseOn();
+ res |= fts_enableInterrupt();
+ }
+
+ res |= ret;
+
+ if (res < OK)
+ pr_err("%s: TP Sensitivity Mode... ERROR %08X!\n",
+ __func__, res);
+ else
+ pr_info("%s: TP Sensitivity Mode FINISHED!\n", __func__);
+
+ return res;
+}
+
+
+/**
+ * Compute Digital gains for calibration
+ * @param scan select the scan mode which should be enabled
+ * @param enableGains =1 apply gains when computing the strength otherwise
+ * the gains will be ignored
+ * @return OK if success or an error code which specify the type of error
+ */
+int tp_sensitivity_set_scan_mode(u8 scan, int enableGains)
+{
+ int res, ret = OK;
+ u8 cmd[4] = { 0xC0, 0x00, 0x01, 0x00 };
+
+
+ pr_info("%s: Set TP Sensitivity Scan Mode... scan = %02X, enableGains = %d\n",
+ __func__, scan, enableGains);
+
+
+ if (enableGains == 1) {
+ /* Consider Sensitivity Gains when computing Strength */
+ cmd[3] = 0x01;
+ ret = fts_writeFwCmd(cmd, 4);
+ if (ret < OK)
+ pr_err("%s: Error while enabling Gains in TP Sens Mode! ERROR %08X\n",
+ __func__, ret);
+ } else {
+ /* Exclude Sensitivity Gains when computing Strength */
+ ret = fts_writeFwCmd(cmd, 4);
+ if (ret < OK)
+ pr_err("%s: Error while disabling Gain in TP Sens Mode! ERROR %08X\n",
+ __func__, ret);
+ }
+
+ res = setScanMode(SCAN_MODE_LOCKED, scan);
+ if (res < OK)
+ pr_err("Error while setting the scan frequency... ERROR %08X\n",
+ res);
+
+ res |= ret;
+
+ if (res < OK)
+ pr_err("%s: Set TP Sensitivity Scan Mode... ERROR %08X!\n",
+ __func__, res);
+ else
+ pr_info("%s: Set TP Sensitivity Scan FINISHED!\n", __func__);
+
+ return res;
+}
+
+
+
+
+/**
+ * Compute the standard deviation for each node form a series of frames
+ * @param numFrames number of frames to collect to compute the standard
+ * deviation
+ * @param[out] std pointer to the frame which will contain the standard
+ * deviation for each node
+ * @return OK if success or an error code which specify the type of error
+ */
+int tp_sensitivity_test_std_ms(int numFrames, MutualSenseFrame *std)
+{
+ int ret = OK;
+ int i = 0, count = 0;
+ MutualSenseFrame frame;
+ int *mean = NULL;/* store the mean value for each node */
+ unsigned long *stdTemp = NULL;
+
+
+ if (std == NULL) {
+ pr_err("%s: Invalid arguments Passed! ERROR %08X\n",
+ __func__, ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+
+ std->node_data = NULL;
+
+ pr_info("%s: Start TP sensitivity STD... collecting %d frames!\n",
+ __func__, numFrames);
+
+ /* collect frames skipping the tests + print on the log */
+ ret = getMSFrame3(MS_STRENGTH, &frame);
+ if (ret < OK) {
+ pr_err("%s: can not read MS Frame... ERROR %08X\n",
+ __func__, ret);
+ goto ERROR;
+ }
+
+ std->header = frame.header;
+ std->node_data_size = frame.node_data_size;
+
+ std->node_data = (short *)kzalloc(std->node_data_size * sizeof(short),
+ GFP_KERNEL);
+ mean = (int *)kzalloc(std->node_data_size * sizeof(int), GFP_KERNEL);
+ stdTemp = (unsigned long *)kzalloc(std->node_data_size *
+ sizeof(unsigned long),
+ GFP_KERNEL);
+ if (std->node_data == NULL ||
+ mean == NULL ||
+ stdTemp == NULL) {
+ pr_err("%s: can not allocate memory ERROR %08X\n",
+ __func__, ERROR_ALLOC | ERROR_GET_FRAME);
+ ret = ERROR_ALLOC | ERROR_GET_FRAME;
+ goto ERROR;
+ }
+
+ /* collecting frames */
+ do {
+ for (i = 0; i < frame.node_data_size; i++) {
+ mean[i] += frame.node_data[i];
+ stdTemp[i] += frame.node_data[i] * frame.node_data[i];
+ }
+ count++;
+
+ if (frame.node_data != NULL) {
+ kfree(frame.node_data);
+ frame.node_data = NULL;
+ }
+
+ /* exclude one more reading at the end*/
+ if (count < numFrames)
+ ret = getMSFrame3(MS_STRENGTH, &frame);
+ } while ((count < numFrames) && (ret >= OK));
+
+ if (ret < OK) {
+ pr_err("%s: error while collecting the frames! ERROR%08X\n",
+ __func__, ret);
+ goto ERROR;
+ }
+
+ /* compute the average for each node */
+ pr_info("%s: Computing std for each node...\n", __func__);
+
+ for (i = 0; i < std->node_data_size; i++) {
+ mean[i] /= numFrames;
+ stdTemp[i] = stdTemp[i] / numFrames - (mean[i] * mean[i]);
+ std->node_data[i] = (short)int_sqrt(stdTemp[i]);
+ }
+
+ kfree(stdTemp);
+ stdTemp = NULL;
+ kfree(mean);
+ mean = NULL;
+
+ /* print average frame in the log */
+ print_frame_short("STD =",
+ array1dTo2d_short(
+ std->node_data,
+ std->node_data_size,
+ std->header.sense_node),
+ std->header.force_node,
+ std->header.sense_node);
+
+ if (ret < OK)
+ pr_err("%s: TP sensitivity STD test FAILED... ERROR %08X\n",
+ __func__, ret);
+ else {
+ pr_info("%s: TP sensitivity STD FINISHED!\n",
+ __func__);
+ ret = OK;
+ }
+
+ return ret;
+
+ERROR:
+
+ kfree(frame.node_data);
+ frame.node_data = NULL;
+
+ kfree(std->node_data);
+ std->node_data = NULL;
+
+ kfree(stdTemp);
+ stdTemp = NULL;
+
+ kfree(mean);
+ mean = NULL;
+
+ return ret;
+}
+
+
+/**
+ * Retrieve the actual Test Limit data from the system (bin file or header
+ * file)
+ * @param path name of Production Test Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param file pointer to the LimitFile struct which will contains the limits
+ * data
+ * @return OK if success or an error code which specify the type of error
+ */
+int getLimitsFile(char *path, LimitFile *file)
+{
+ const struct firmware *fw = NULL;
+ struct device *dev = NULL;
+ int fd = -1;
+
+ pr_info("Get Limits File starting... %s\n", path);
+
+ if (file->data != NULL) {
+ /* to avoid memory leak on consecutive call of
+ * the function with the same pointer */
+ pr_err("Pointer to Limits Data already contains something... freeing its content!\n");
+ kfree(file->data);
+ file->data = NULL;
+ file->size = 0;
+ }
+
+ strlcpy(file->name, path, MAX_LIMIT_FILE_NAME);
+ if (strncmp(path, "NULL", 4) == 0) {
+#ifdef LIMITS_H_FILE
+ pr_info("Loading Limits File from .h!\n");
+ file->size = LIMITS_SIZE_NAME;
+ file->data = (char *)kmalloc((file->size) * sizeof(char),
+ GFP_KERNEL);
+ if (file->data != NULL) {
+ memcpy(file->data, (char *)(LIMITS_ARRAY_NAME),
+ file->size);
+ return OK;
+ } else {
+ pr_err("Error while allocating data... ERROR %08X\n",
+ path, ERROR_ALLOC);
+ return ERROR_ALLOC;
+ }
+#else
+ pr_err("limit file path NULL... ERROR %08X\n",
+ ERROR_FILE_NOT_FOUND);
+ return ERROR_FILE_NOT_FOUND;
+#endif
+ } else {
+ dev = getDev();
+ if (dev != NULL) {
+ pr_info("Loading Limits File from .csv!\n");
+ fd = request_firmware(&fw, path, dev);
+ if (fd == 0) {
+ pr_info("Start to copy %s...\n", path);
+ file->size = fw->size;
+ file->data = (char *)kmalloc((file->size) *
+ sizeof(char),
+ GFP_KERNEL);
+ if (file->data != NULL) {
+ memcpy(file->data, (char *)fw->data,
+ file->size);
+ pr_info("Limit file Size = %d\n",
+ file->size);
+ release_firmware(fw);
+ return OK;
+ } else {
+ pr_err("Error while allocating data... ERROR %08X\n",
+ ERROR_ALLOC);
+ release_firmware(fw);
+ return ERROR_ALLOC;
+ }
+ } else {
+ pr_err("Request the file %s failed... ERROR %08X\n",
+ path, ERROR_FILE_NOT_FOUND);
+ return ERROR_FILE_NOT_FOUND;
+ }
+ } else {
+ pr_err("Error while getting the device ERROR %08X\n",
+ ERROR_FILE_READ);
+ return ERROR_FILE_READ;
+ }
+ }
+}
+
+/**
+ * Reset and release the memory which store a Production Limit File previously
+ * loaded
+ * @param file pointer to the LimitFile struct to free
+ * @return OK if success or an error code which specify the type of error
+ */
+
+int freeLimitsFile(LimitFile *file)
+{
+ pr_info("Freeing Limit File ...\n");
+ if (file != NULL) {
+ if (file->data != NULL) {
+ kfree(file->data);
+ file->data = NULL;
+ } else
+ pr_err("Limit File was already freed!\n");
+ file->size = 0;
+ strlcpy(file->name, " ", MAX_LIMIT_FILE_NAME);
+ return OK;
+ } else {
+ pr_err("Passed a NULL argument! ERROR %08X\n",
+ ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+}
+
+/**
+ * Reset and release the memory which store the current Limit File
+ * previously loaded
+ * @return OK if success or an error code which specify the type of error
+ */
+int freeCurrentLimitsFile(void)
+{
+ return freeLimitsFile(&limit_file);
+}
+
+/**
+ * Parse the raw data read from a Production test limit file in order
+ * to find the specified information
+ * If no limits file data are passed, the function loads and stores the limit
+ * file from the system
+ * @param path name of Production Test Limit file to load or
+ * "NULL" if the limits data should be loaded by a .h file
+ * @param file pointer to LimitFile struct that should be parsed or
+ * NULL if the limit file in the system should be loaded and then parsed
+ * @param label string which identify a particular set of data in the file that
+ * want to be loaded
+ * @param data pointer to the pointer which will contains the specified limits
+ * data
+ * as 1 dimension matrix with data arranged row after row
+ * @param row pointer to a int variable which will contain the number of row of
+ * data
+ * @param column pointer to a int variable which will contain the number of
+ * column of data
+ * @return OK if success or an error code which specify the type of error
+ */
+int parseProductionTestLimits(char *path, LimitFile *file, char *label,
+ int **data, int *row, int *column)
+{
+ int find = 0;
+ char *token = NULL;
+ int i = 0;
+ int j = 0;
+ int z = 0;
+
+ char *line2 = NULL;
+ char line[800];
+ char *buf = NULL;
+ int n, size, pointer = 0, ret = OK;
+ char *data_file = NULL;
+
+ if (file == NULL || strcmp(path, file->name) != 0 || file->size == 0) {
+ pr_info("No limit File data passed... try to get them from the system!\n");
+ ret = getLimitsFile(LIMITS_FILE, &limit_file);
+ if (ret < OK) {
+ pr_err("parseProductionTestLimits: ERROR %08X\n",
+ ERROR_FILE_NOT_FOUND);
+ return ERROR_FILE_NOT_FOUND;
+ }
+ size = limit_file.size;
+ data_file = limit_file.data;
+ } else {
+ pr_info("Limit File data passed as arguments!\n");
+ size = file->size;
+ data_file = file->data;
+ }
+
+
+
+ pr_info("The size of the limits file is %d bytes...\n", size);
+
+
+
+ while (find == 0) {
+ /* start to look for the wanted label */
+ if (readLine(&data_file[pointer], line, size - pointer, &n) <
+ 0) {
+ find = -1;
+ break;
+ }
+ pointer += n;
+ if (line[0] == '*') {
+ /* each header row start with * ex. *label,n_row,n_colum */
+ line2 = kstrdup(line, GFP_KERNEL);
+ if (line2 == NULL) {
+ pr_err("parseProductionTestLimits: kstrdup ERROR %08X\n",
+ ERROR_ALLOC);
+ ret = ERROR_ALLOC;
+ goto END;
+ }
+ buf = line2;
+ line2 += 1;
+ token = strsep(&line2, ",");
+ if (strcmp(token, label) == 0) {
+ /* if the row is the wanted one, r
+ * retrieve rows and columns info */
+ find = 1;
+ token = strsep(&line2, ",");
+ if (token != NULL) {
+ sscanf(token, "%d", row);
+ pr_info("Row = %d\n", *row);
+ } else {
+ pr_err("parseProductionTestLimits 1: ERROR %08X\n",
+ ERROR_FILE_PARSE);
+ ret = ERROR_FILE_PARSE;
+ goto END;
+ }
+ token = strsep(&line2, ",");
+ if (token != NULL) {
+ sscanf(token, "%d", column);
+ pr_info("Column = %d\n", *column);
+ } else {
+ pr_err("parseProductionTestLimits 2: ERROR %08X\n",
+ ERROR_FILE_PARSE);
+ ret = ERROR_FILE_PARSE;
+ goto END;
+ }
+
+ kfree(buf);
+ buf = NULL;
+ *data = (int *)kmalloc(((*row) * (*column)) *
+ sizeof(int), GFP_KERNEL);
+ /* allocate the memory for containing the data */
+ j = 0;
+ if (*data == NULL) {
+ pr_err("parseProductionTestLimits: ERROR %08X\n",
+ ERROR_ALLOC);
+ ret = ERROR_ALLOC;
+ goto END;
+ }
+
+
+ /* start to read the data */
+ for (i = 0; i < *row; i++) {
+ if (readLine(&data_file[pointer], line,
+ size - pointer, &n) < 0) {
+ pr_err("parseProductionTestLimits : ERROR %08X\n",
+ ERROR_FILE_READ);
+ ret = ERROR_FILE_READ;
+ goto END;
+ }
+ pointer += n;
+ line2 = kstrdup(line, GFP_KERNEL);
+ if (line2 == NULL) {
+ pr_err("parseProductionTestLimits: kstrdup ERROR %08X\n",
+ ERROR_ALLOC);
+ ret = ERROR_ALLOC;
+ goto END;
+ }
+ buf = line2;
+ token = strsep(&line2, ",");
+ for (z = 0; (z < *column) && (token !=
+ NULL);
+ z++) {
+ sscanf(token, "%d", ((*data) +
+ j));
+ j++;
+ token = strsep(&line2, ",");
+ }
+ kfree(buf);
+ buf = NULL;
+ }
+ if (j == ((*row) * (*column))) {
+ /* check that all the data are read */
+ pr_info("READ DONE!\n");
+ ret = OK;
+ goto END;
+ }
+ pr_err("parseProductionTestLimits 3: ERROR %08X\n",
+ ERROR_FILE_PARSE);
+ ret = ERROR_FILE_PARSE;
+ goto END;
+ }
+ kfree(buf);
+ buf = NULL;
+ }
+ }
+ pr_err("parseProductionTestLimits: ERROR %08X\n",
+ ERROR_LABEL_NOT_FOUND);
+ ret = ERROR_LABEL_NOT_FOUND;
+END:
+ if (buf != NULL)
+ kfree(buf);
+ return ret;
+}
+
+
+/**
+ * Read one line of a text file passed as array of byte and terminate it with
+ * a termination character '\0'
+ * @param data text file as array of bytes
+ * @param line pointer to an array of char that will contain the line read
+ * @param size size of data
+ * @param n pointer to a int variable which will contain the number of
+ * characters of the line
+ * @return OK if success or an error code which specify the type of error
+ */
+int readLine(char *data, char *line, int size, int *n)
+{
+ int i = 0;
+
+ if (size < 1)
+ return ERROR_OP_NOT_ALLOW;
+
+ while (data[i] != '\n' && i < size) {
+ line[i] = data[i];
+ i++;
+ }
+ *n = i + 1;
+ line[i] = '\0';
+
+ return OK;
+}
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsTest.h b/drivers/input/touchscreen/stm/fts_lib/ftsTest.h
new file mode 100644
index 0000000..7554709
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsTest.h
@@ -0,0 +1,343 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS API for MP test **
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsTest.h
+ * \brief Contains all the definitions and structs related to the Mass
+ *Production Test
+ */
+
+#ifndef FTS_TEST_H
+#define FTS_TEST_H
+
+#include "ftsSoftware.h"
+#include "ftsFrame.h"
+
+#ifndef LIMITS_H_FILE
+/* /< Name of the Production Test Limit File */
+#define LIMITS_FILE "stm_fts_production_limits.csv"
+#else
+#define LIMITS_FILE "NULL"
+#endif
+
+#define WAIT_FOR_FRESH_FRAMES 200 /* /< Time in ms to wait after
+ * start to sensing before
+ * reading a frame */
+#define WAIT_AFTER_SENSEOFF 50 /* /< Time in ms to wait after
+ * stop sensing and before
+ * reading a frame from
+ * memory */
+
+#define NO_INIT 0 /* /< No Initialization required
+ * during the MP */
+
+#define RETRY_INIT_BOOT 3 /* /< number of retry of the
+ * init process at boot */
+
+#define SENS_TEST_NUM_FRAMES 100 /* /< number of frames to read */
+#define SENS_TEST_PERC_TARGET_PRECAL 20 /* /< +-% of target value within
+ * all the node of the frames
+ * should be contained */
+#define SENS_TEST_PERC_TARGET_POSTCAL 4 /* /< +-% of target value within
+ * should be contained the
+ * avarege of the nodes and the
+ * deltas for each node */
+
+/** @defgroup mp_test Mass Production Test
+ * Mass production test API.
+ * Mass Production Test (MP) should be executed at least one time in the life
+ * of every device \n
+ * It used to verify that tit is not present any hardware damage and
+ * initialize some value of the chip in order to guarantee the working
+ * performance \n
+ * The MP test is made up by 3 steps:
+ * - ITO test = production_test_ito() \n
+ * - Initialization = production_test_initialization() \n
+ * - Data Test = production_test_data(),
+ * it is possible to select which items test thanks to the TestToDo struct\n
+ * To execute the Data Test it is mandatory load some thresholds that
+ * are stored in the Limit File.
+ * @{
+ */
+
+/** @defgroup limit_file Limit File
+ * @ingroup mp_test
+ * Production Test Limit File is a csv which contains thresholds of the data to
+ * test.
+ * This file can be loaded from the file system or stored as a header file
+ * according to the LIMITS_H_FILE define \n
+ * For each selectable test item there can be one or more associated labels
+ * which store the corresponding thresholds \n
+ * @{
+ */
+/* LABELS PRODUCTION TEST LIMITS FILE */
+/** @defgroup test_labels Test Items Labels
+ * @ingroup limit_file
+ * Labels present in the Limit File and associated to the test items of
+ * TestToDo
+ * @{
+ */
+#define MS_RAW_MIN_MAX "MS_RAW_DATA_MIN_MAX"
+#define MS_RAW_GAP "MS_RAW_DATA_GAP"
+#define MS_RAW_ADJH "MS_RAW_DATA_ADJ_HORIZONTAL"
+#define MS_RAW_ADJV "MS_RAW_DATA_ADJ_VERTICAL"
+#define MS_RAW_ITO_ADJH "MS_RAW_ITO_DATA_ADJ_HORIZONTAL"
+#define MS_RAW_ITO_ADJV "MS_RAW_ITO_DATA_ADJ_VERTICAL"
+#define MS_RAW_LP_MIN_MAX "MS_RAW_LOWPOWER_DATA_MIN_MAX"
+#define MS_RAW_LP_GAP "MS_RAW_LOWPOWER_DATA_GAP"
+#define MS_RAW_LP_ADJH "MS_RAW_LOWPOWER_DATA_ADJ_HORIZONTAL"
+#define MS_RAW_LP_ADJV "MS_RAW_LOWPOWER_DATA_ADJ_VERTICAL"
+#define MS_CX1_MIN_MAX "MS_TOUCH_ACTIVE_CX1_MIN_MAX"
+#define MS_CX2_MAP_MIN "MS_TOUCH_ACTIVE_CX2_MIN"
+#define MS_CX2_MAP_MAX "MS_TOUCH_ACTIVE_CX2_MAX"
+#define MS_CX2_ADJH_MAP_MAX "MS_TOUCH_ACTIVE_CX2_ADJ_HORIZONTAL"
+#define MS_CX2_ADJV_MAP_MAX "MS_TOUCH_ACTIVE_CX2_ADJ_VERTICAL"
+#define MS_TOTAL_CX_MAP_MIN "MS_TOUCH_ACTIVE_TOTAL_CX_MIN"
+#define MS_TOTAL_CX_MAP_MAX "MS_TOUCH_ACTIVE_TOTAL_CX_MAX"
+#define MS_TOTAL_CX_ADJH_MAP_MAX \
+ "MS_TOUCH_ACTIVE_TOTAL_CX_ADJ_HORIZONTAL"
+#define MS_TOTAL_CX_ADJV_MAP_MAX "MS_TOUCH_ACTIVE_TOTAL_CX_ADJ_VERTICAL"
+#define SS_RAW_FORCE_MIN_MAX "SS_RAW_DATA_FORCE_MIN_MAX"
+#define SS_RAW_SENSE_MIN_MAX "SS_RAW_DATA_SENSE_MIN_MAX"
+#define SS_RAW_FORCE_GAP "SS_RAW_DATA_FORCE_GAP"
+#define SS_RAW_SENSE_GAP "SS_RAW_DATA_SENSE_GAP"
+#define SS_RAW_LP_FORCE_MIN_MAX "SS_RAW_LOWPOWER_DATA_FORCE_MIN_MAX"
+#define SS_RAW_LP_SENSE_MIN_MAX "SS_RAW_LOWPOWER_DATA_SENSE_MIN_MAX"
+#define SS_RAW_LP_FORCE_GAP "SS_RAW_LOWPOWER_DATA_FORCE_GAP"
+#define SS_RAW_LP_SENSE_GAP "SS_RAW_LOWPOWER_DATA_SENSE_GAP"
+#define SS_IX1_FORCE_MIN_MAX "SS_TOUCH_ACTIVE_IX1_FORCE_MIN_MAX"
+#define SS_IX1_SENSE_MIN_MAX "SS_TOUCH_ACTIVE_IX1_SENSE_MIN_MAX"
+#define SS_CX1_FORCE_MIN_MAX "SS_TOUCH_ACTIVE_CX1_FORCE_MIN_MAX"
+#define SS_CX1_SENSE_MIN_MAX "SS_TOUCH_ACTIVE_CX1_SENSE_MIN_MAX"
+#define SS_IX2_FORCE_MAP_MIN "SS_TOUCH_ACTIVE_IX2_FORCE_MIN"
+#define SS_IX2_FORCE_MAP_MAX "SS_TOUCH_ACTIVE_IX2_FORCE_MAX"
+#define SS_IX2_SENSE_MAP_MIN "SS_TOUCH_ACTIVE_IX2_SENSE_MIN"
+#define SS_IX2_SENSE_MAP_MAX "SS_TOUCH_ACTIVE_IX2_SENSE_MAX"
+#define SS_IX2_FORCE_ADJV_MAP_MAX "SS_TOUCH_ACTIVE_IX2_ADJ_VERTICAL"
+#define SS_IX2_SENSE_ADJH_MAP_MAX "SS_TOUCH_ACTIVE_IX2_ADJ_HORIZONTAL"
+#define SS_CX2_FORCE_MAP_MIN "SS_TOUCH_ACTIVE_CX2_FORCE_MIN"
+#define SS_CX2_FORCE_MAP_MAX "SS_TOUCH_ACTIVE_CX2_FORCE_MAX"
+#define SS_CX2_SENSE_MAP_MIN "SS_TOUCH_ACTIVE_CX2_SENSE_MIN"
+#define SS_CX2_SENSE_MAP_MAX "SS_TOUCH_ACTIVE_CX2_SENSE_MAX"
+#define SS_CX2_FORCE_ADJV_MAP_MAX "SS_TOUCH_ACTIVE_CX2_ADJ_VERTICAL"
+#define SS_CX2_SENSE_ADJH_MAP_MAX "SS_TOUCH_ACTIVE_CX2_ADJ_HORIZONTAL"
+
+/* TOTAL SS */
+#define SS_TOTAL_IX_FORCE_MAP_MIN "SS_TOUCH_ACTIVE_TOTAL_IX_FORCE_MIN"
+#define SS_TOTAL_IX_FORCE_MAP_MAX "SS_TOUCH_ACTIVE_TOTAL_IX_FORCE_MAX"
+#define SS_TOTAL_IX_SENSE_MAP_MIN "SS_TOUCH_ACTIVE_TOTAL_IX_SENSE_MIN"
+#define SS_TOTAL_IX_SENSE_MAP_MAX "SS_TOUCH_ACTIVE_TOTAL_IX_SENSE_MAX"
+#define SS_TOTAL_IX_FORCE_ADJV_MAP_MAX "SS_TOUCH_ACTIVE_TOTAL_IX_ADJ_VERTICAL"
+#define SS_TOTAL_IX_SENSE_ADJH_MAP_MAX \
+ "SS_TOUCH_ACTIVE_TOTAL_IX_ADJ_HORIZONTAL"
+#define SS_TOTAL_CX_FORCE_MAP_MIN "SS_TOUCH_ACTIVE_TOTAL_CX_FORCE_MIN"
+#define SS_TOTAL_CX_FORCE_MAP_MAX "SS_TOUCH_ACTIVE_TOTAL_CX_FORCE_MAX"
+#define SS_TOTAL_CX_SENSE_MAP_MIN "SS_TOUCH_ACTIVE_TOTAL_CX_SENSE_MIN"
+#define SS_TOTAL_CX_SENSE_MAP_MAX "SS_TOUCH_ACTIVE_TOTAL_CX_SENSE_MAX"
+#define SS_TOTAL_CX_FORCE_ADJV_MAP_MAX "SS_TOUCH_ACTIVE_TOTAL_CX_ADJ_VERTICAL"
+#define SS_TOTAL_CX_SENSE_ADJH_MAP_MAX \
+ "SS_TOUCH_ACTIVE_TOTAL_CX_ADJ_HORIZONTAL"
+
+/* KEYS */
+#define MS_KEY_RAW_MIN_MAX "MS_KEY_RAW_DATA_MIN_MAX"
+#define MS_KEY_CX1_MIN_MAX "MS_KEY_CX1_MIN_MAX"
+#define MS_KEY_CX2_MAP_MIN "MS_KEY_CX2_MIN"
+#define MS_KEY_CX2_MAP_MAX "MS_KEY_CX2_MAX"
+#define MS_KEY_TOTAL_CX_MAP_MIN "MS_KEY_TOTAL_CX_MIN"
+#define MS_KEY_TOTAL_CX_MAP_MAX "MS_KEY_TOTAL_CX_MAX"
+
+/* CONSTANT TOTAL IX */
+#define SS_IX1_FORCE_W "IX1_FORCE_W"
+#define SS_IX2_FORCE_W "IX2_FORCE_W"
+#define SS_IX1_SENSE_W "IX1_SENSE_W"
+#define SS_IX2_SENSE_W "IX2_SENSE_W"
+/** @}*/
+
+
+/**
+ * Struct used to specify which test perform during the Mass Production Test.
+ * For each test item selected in this structure, there should be one or
+ * more labels associated in the Limit file from where load the thresholds
+ */
+typedef struct {
+ int MutualRaw; /* /< MS Raw min/Max test */
+ int MutualRawGap; /* /< MS Raw Gap(max-min) test */
+ int MutualRawAdj; /* /< MS Raw Adjacent test */
+ int MutualRawLP; /* /< MS Low Power Raw min/Max test */
+ int MutualRawGapLP; /* /< MS Low Power Raw Gap(max-min) test */
+ int MutualRawAdjLP; /* /< MS Low Power Raw Adjacent test */
+ int MutualRawAdjITO; /* /< MS Raw Adjacent test during ITO test */
+
+ int MutualCx1; /* /< MS Cx1 min/Max test */
+ int MutualCx2; /* /< MS Cx2 min/Max (for each node) test */
+ int MutualCx2Adj; /* /< MS Vertical and Horizontal Adj Cx2 min/Max
+ * (for each node) test */
+ int MutualCxTotal; /* /< MS Total Cx min/Max (for each node) test
+ * */
+ int MutualCxTotalAdj; /* /< MS Total vertical and Horizontal Adj Cx2
+ * min/Max
+ * (for each node) test */
+
+ int MutualKeyRaw; /* /< MS Raw Key min/Max test */
+ int MutualKeyCx1; /* /< MS Cx1 Key min/Max test */
+ int MutualKeyCx2; /* /< MS Cx2 Key min/Max (for each node) test */
+ int MutualKeyCxTotal; /* /< MS Total Cx Key min/Max (for each node)
+ * test */
+
+ int SelfForceRaw; /* /< SS Force Raw min/Max test */
+ int SelfForceRawGap; /* /< SS Force Raw Gap(max-min) test */
+ int SelfForceRawLP; /* /< SS Low Power Force Raw min/Max test */
+ int SelfForceRawGapLP; /* /< SS Low Power Force Raw Gap(max-min) test */
+
+ int SelfForceIx1; /* /< SS Force Ix1 min/Max test */
+ int SelfForceIx2; /* /< SS Force Ix2 min/Max (for each node) test
+ * */
+ int SelfForceIx2Adj; /* /< SS Vertical Adj Force Ix2 min/Max
+ * (for each node) test */
+ int SelfForceIxTotal; /* /< SS Total Force Ix min/Max (for each node)
+ * test */
+ int SelfForceIxTotalAdj; /* /< SS Total Vertical Adj Force Ix
+ * min/Max
+ * (for each node) test */
+ int SelfForceCx1; /* /< SS Force Cx1 min/Max test */
+ int SelfForceCx2; /* /< SS Force Cx2 min/Max (for each node) test */
+ int SelfForceCx2Adj; /* /< SS Vertical Adj Force Cx2 min/Max (for
+ * each node) test */
+ int SelfForceCxTotal; /* /< SS Total Force Cx min/Max (for each node)
+ * test */
+ int SelfForceCxTotalAdj; /* /< SS Total Vertical Adj Force Cx
+ * min/Max
+ * (for each node) test */
+
+ int SelfSenseRaw; /* /< SS Sense Raw min/Max test */
+ int SelfSenseRawGap; /* /< SS Sense Raw Gap(max-min) test */
+ int SelfSenseRawLP; /* /< SS Low Power Sense Raw min/Max test */
+ int SelfSenseRawGapLP; /* /< SS Low Power Sense Raw Gap(max-min) test */
+
+ int SelfSenseIx1; /* /< SS Sense Ix1 min/Max test */
+ int SelfSenseIx2; /* /< SS Sense Ix2 min/Max (for each node) test */
+ int SelfSenseIx2Adj; /* /< SS Horizontal Adj Sense Ix2 min/Max
+ * (for each node) test */
+ int SelfSenseIxTotal; /* /< SS Total Horizontal Sense Ix min/Max
+ * (for each node) test */
+ int SelfSenseIxTotalAdj; /* /< SS Total Horizontal Adj Sense Ix
+ * min/Max
+ * (for each node) test */
+ int SelfSenseCx1; /* /< SS Sense Cx1 min/Max test */
+ int SelfSenseCx2; /* /< SS Sense Cx2 min/Max (for each node) test */
+ int SelfSenseCx2Adj; /* /< SS Horizontal Adj Sense Cx2 min/Max
+ * (for each node) test */
+ int SelfSenseCxTotal; /* /< SS Total Sense Cx min/Max (for each node)
+ * test */
+ int SelfSenseCxTotalAdj; /* /< SS Total Horizontal Adj Sense Cx
+ * min/Max
+ * (for each node) test */
+} TestToDo;
+
+
+#define MAX_LIMIT_FILE_NAME 100 /* /< max number of chars of the limit file name
+ * */
+
+/**
+ * Struct which store the data coming from a Production Limit File
+ */
+typedef struct {
+ char *data; /* /< pointer to an array of char which contains
+ * the content of the Production Limit File */
+ int size; /* /< size of data */
+ char name[MAX_LIMIT_FILE_NAME]; /* /< identifier of the source from
+ * where the limits data were loaded
+ * (if loaded from a file it will be
+ * the file name, while if loaded
+ * from .h will be "NULL") */
+} LimitFile;
+
+
+
+int initTestToDo(void);
+/**@}*/
+
+/**@}*/
+
+
+int computeAdjHoriz(i8 *data, int row, int column, u8 **result);
+int computeAdjHorizTotal(short *data, int row, int column, u16 **result);
+int computeAdjVert(i8 *data, int row, int column, u8 **result);
+int computeAdjVertTotal(short *data, int row, int column, u16 **result);
+int computeAdjHorizFromU(u8 *data, int row, int column, u8 **result);
+int computeAdjHorizTotalFromU(u16 *data, int row, int column, u16 **result);
+int computeAdjVertFromU(u8 *data, int row, int column, u8 **result);
+int computeAdjVertTotalFromU(u16 *data, int row, int column, u16 **result);
+int checkLimitsMinMax(short *data, int row, int column, int min, int max);
+int checkLimitsMap(i8 *data, int row, int column, int *min, int *max);
+int checkLimitsMapTotal(short *data, int row, int column, int *min, int *max);
+int checkLimitsMapFromU(u8 *data, int row, int column, int *min, int *max);
+int checkLimitsMapTotalFromU(u16 *data, int row, int column, int *min,
+ int *max);
+int checkLimitsMapAdj(u8 *data, int row, int column, int *max);
+int checkLimitsMapAdjTotal(u16 *data, int row, int column, int *max);
+
+/** @defgroup mp_api MP API
+ * @ingroup mp_test
+ * Functions to execute the MP test.
+ * The parameters of these functions allow to customize their behavior
+ * in order to satisfy different scenarios
+ * @{
+ */
+int production_test_ito(char *path_limits, TestToDo *todo);
+int production_test_initialization(u8 type);
+int production_test_main(char *pathThresholds, int stop_on_fail, int saveInit,
+ TestToDo *todo);
+int production_test_ms_raw(char *path_limits, int stop_on_fail, TestToDo *todo);
+int production_test_ms_raw_lp(char *path_limits, int stop_on_fail,
+ TestToDo *todo);
+int production_test_ms_cx(char *path_limits, int stop_on_fail, TestToDo *todo);
+int production_test_ss_raw(char *path_limits, int stop_on_fail, TestToDo *todo);
+int production_test_ss_raw_lp(char *path_limits, int stop_on_fail,
+ TestToDo *todo);
+int production_test_ss_ix_cx(char *path_limits, int stop_on_fail,
+ TestToDo *todo);
+int production_test_data(char *path_limits, int stop_on_fail, TestToDo *todo);
+int production_test_ms_key_cx(char *path_limits, int stop_on_fail,
+ TestToDo *todo);
+int production_test_ms_key_raw(char *path_limits);
+/** @}*/
+
+/**
+ * @addtogroup limit_file
+ * @{
+ */
+int parseProductionTestLimits(char *path, LimitFile *file, char *label,
+ int **data, int *row, int *column);
+int readLine(char *data, char *line, int size, int *n);
+int getLimitsFile(char *path, LimitFile *file);
+int freeLimitsFile(LimitFile *file);
+int freeCurrentLimitsFile(void);
+/**@}*/
+
+int tp_sensitivity_test_pre_cal_ms(MutualSenseFrame *finalFrame, short target,
+ int percentage);
+int tp_sensitivity_test_pre_cal_ss(SelfSenseFrame *finalFrame, short target,
+ int percentage);
+int tp_sensitivity_compute_gains(MutualSenseFrame *frame, short target,
+ int saveGain);
+int tp_sensitivity_test_post_cal_ms(MutualSenseFrame *finalFrame,
+ MutualSenseFrame *deltas, short target,
+ int percentage, int *mean_normal,
+ int *mean_edge);
+int tp_sensitivity_set_scan_mode(u8 scan, int enableGains);
+int tp_sensitivity_mode(u8 enter, int saveGain);
+int tp_sensitivity_test_std_ms(int numFrames, MutualSenseFrame *std);
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsTime.c b/drivers/input/touchscreen/stm/fts_lib/ftsTime.c
new file mode 100644
index 0000000..9ab18fd6
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsTime.c
@@ -0,0 +1,84 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS Utility for mesuring/handling the time *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsTime.c
+ * \brief Contains all functions to handle and measure the time in the driver
+ */
+
+#include "ftsTime.h"
+
+
+#include <linux/errno.h>
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <stdarg.h>
+#include <linux/time.h>
+#include <linux/delay.h>
+#include <linux/ctype.h>
+
+
+/**
+ * Take the starting time and save it in a StopWatch variable
+ * @param w pointer of a StopWatch struct
+ */
+void startStopWatch(StopWatch *w)
+{
+ w->start = current_kernel_time();
+}
+
+/**
+ * Take the stop time and save it in a StopWatch variable
+ * @param w pointer of a StopWatch struct
+ */
+void stopStopWatch(StopWatch *w)
+{
+ w->end = current_kernel_time();
+}
+
+/**
+ * Compute the amount of time spent from when the startStopWatch and then
+ * the stopStopWatch were called on the StopWatch variable
+ * @param w pointer of a StopWatch struct
+ * @return amount of time in ms (the return value is meaningless
+ * if the startStopWatch and stopStopWatch were not called before)
+ */
+int elapsedMillisecond(StopWatch *w)
+{
+ int result;
+
+ result = ((w->end.tv_sec - w->start.tv_sec) * 1000) +
+ (w->end.tv_nsec - w->start.tv_nsec) / 1000000;
+ return result;
+}
+
+/**
+ * Compute the amount of time spent from when the startStopWatch and
+ * then the stopStopWatch were called on the StopWatch variable
+ * @param w pointer of a StopWatch struct
+ * @return amount of time in ns (the return value is meaningless
+ * if the startStopWatch and stopStopWatch were not called before)
+ */
+int elapsedNanosecond(StopWatch *w)
+{
+ int result;
+
+ result = ((w->end.tv_sec - w->start.tv_sec) * 1000000000) +
+ (w->end.tv_nsec - w->start.tv_nsec);
+ return result;
+}
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsTime.h b/drivers/input/touchscreen/stm/fts_lib/ftsTime.h
new file mode 100644
index 0000000..73ba5d5
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsTime.h
@@ -0,0 +1,70 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS Utility for measuring/handling the time *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsTime.h
+ * \brief Contains all the definitions and structs to handle and measure the
+ * time in the driver
+ */
+
+#ifndef FTS_TIME_H
+#define FTS_TIME_H
+
+
+#include <linux/time.h>
+
+/* TIMEOUT */
+/** @defgroup timeouts Timeouts
+ * Definitions of all the Timeout used in several operations
+ * @{
+ */
+#define TIMEOUT_RESOLUTION 2
+/* /< timeout resolution in ms (all timeout should be multiples of this unit) */
+#define GENERAL_TIMEOUT (50 * TIMEOUT_RESOLUTION)
+/* /< general timeout in ms */
+#define RELEASE_INFO_TIMEOUT (15 * TIMEOUT_RESOLUTION)
+/* /< timeout to request release info in ms */
+
+
+#define TIMEOUT_REQU_COMP_DATA (100 * TIMEOUT_RESOLUTION)
+/* /< timeout to request compensation data in ms */
+#define TIMEOUT_REQU_DATA (200 * TIMEOUT_RESOLUTION)
+/* /< timeout to request data in ms */
+#define TIMEOUT_ITO_TEST_RESULT (100 * TIMEOUT_RESOLUTION)
+/* /< timeout to perform ito test in ms */
+#define TIMEOUT_INITIALIZATION_TEST_RESULT (5000 * TIMEOUT_RESOLUTION)
+/* /< timeout to perform initialization test in ms */
+#define TIEMOUT_ECHO \
+ TIMEOUT_INITIALIZATION_TEST_RESULT
+/* /< timeout of the echo command, should be the max of
+ * all the possible commands (used in worst case) */
+/** @}*/
+
+
+/**
+ * Struct used to measure the time elapsed between a starting and ending point.
+ */
+typedef struct {
+ struct timespec start; /* /< store the starting time */
+ struct timespec end; /* /< store the finishing time */
+} StopWatch;
+
+
+void startStopWatch(StopWatch *w);
+void stopStopWatch(StopWatch *w);
+int elapsedMillisecond(StopWatch *w);
+int elapsedNanosecond(StopWatch *w);
+
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsTool.c b/drivers/input/touchscreen/stm/fts_lib/ftsTool.c
new file mode 100644
index 0000000..5d34dfb
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsTool.c
@@ -0,0 +1,730 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS Utility Functions *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsTool.c
+ * \brief Contains all the functions to support common operation inside the
+ * driver
+ */
+
+#include "ftsCompensation.h"
+#include "ftsCore.h"
+#include "ftsError.h"
+#include "ftsHardware.h"
+#include "ftsIO.h"
+#include "ftsSoftware.h"
+#include "ftsTime.h"
+#include "ftsTool.h"
+#include "../fts.h" /* needed for the tag define */
+
+
+#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <stdarg.h>
+#include <linux/serio.h>
+#include <linux/init.h>
+#include <linux/delay.h>
+#include <linux/ctype.h>
+
+
+/**
+ * Print an array of byte in a HEX string and attach at the beginning a label.
+ * The function allocate memory that should be free outside the function itself
+ * @param label string to attach at the beginning
+ * @param buff pointer to the byte array that should be printed as HEX string
+ * @param count size of buff
+ * @param result pointer to the array of characters that compose the HEX final
+ * string
+ * @param size size of result
+ * @return pointer to the array of characters that compose the HEX string,
+ * (same address of result)
+ * @warning result MUST be allocated outside the function and should be
+ * big enough to contain the data converted as HEX!
+ */
+char *printHex(char *label, u8 *buff, int count, u8 *result, int size)
+{
+ int i, offset = 0;
+
+ offset = scnprintf(result + offset, size - offset, "%s", label);
+ for (i = 0; i < count; i++) {
+ offset +=
+ scnprintf(result + offset,
+ size - offset, "%02X ", buff[i]);
+ /* this append automatically a null terminator char */
+ }
+ return result;
+}
+
+/**
+ * Clear the FIFO from any event
+ * @return OK if success or an error code which specify the type of error
+ */
+int flushFIFO(void)
+{
+ int ret;
+ u8 sett = SPECIAL_FIFO_FLUSH;
+
+ ret = writeSysCmd(SYS_CMD_SPECIAL, &sett, 1); /* flush the FIFO */
+ if (ret < OK) {
+ pr_err("flushFIFO: ERROR %08X\n", ret);
+ return ret;
+ }
+
+ pr_info("FIFO flushed!\n");
+ return OK;
+}
+
+
+
+/**
+ * Convert an array of bytes to an array of u16 taking two bytes at time,
+ * src has LSB first.
+ * @param src pointer to the source byte array
+ * @param src_length size of src
+ * @param dst pointer to the destination array.
+ * @return the final size of dst (half of the source) or ERROR_OP_NOT_ALLOW
+ * if the size of src is not multiple of 2.
+ */
+int u8ToU16n(u8 *src, int src_length, u16 *dst)
+{
+ int i, j;
+
+ if (src_length % 2 != 0)
+ return ERROR_OP_NOT_ALLOW;
+ else {
+ j = 0;
+ dst = (u16 *)kmalloc((src_length / 2) * sizeof(u16),
+ GFP_KERNEL);
+ for (i = 0; i < src_length; i += 2) {
+ dst[j] = ((src[i + 1] & 0x00FF) << 8) +
+ (src[i] & 0x00FF);
+ j++;
+ }
+ }
+
+ return src_length / 2;
+}
+
+/**
+ * Convert an array of 2 bytes to a u16, src has LSB first (little endian).
+ * @param src pointer to the source byte array
+ * @param dst pointer to the destination u16.
+ * @return OK
+ */
+int u8ToU16(u8 *src, u16 *dst)
+{
+ *dst = (u16)(((src[1] & 0x00FF) << 8) + (src[0] & 0x00FF));
+ return OK;
+}
+
+/**
+ * Convert an array of 2 bytes to a u16, src has MSB first (big endian).
+ * @param src pointer to the source byte array
+ * @param dst pointer to the destination u16.
+ * @return OK
+ */
+int u8ToU16_be(u8 *src, u16 *dst)
+{
+ *dst = (u16)(((src[0] & 0x00FF) << 8) + (src[1] & 0x00FF));
+ return OK;
+}
+
+/**
+ * Convert an array of u16 to an array of u8, dst has MSB first (big endian).
+ * @param src pointer to the source array of u16
+ * @param src_length size of src
+ * @param dst pointer to the destination array of u8. This array should be free
+ * when no need anymore
+ * @return size of dst (src size multiply by 2)
+ */
+int u16ToU8n_be(u16 *src, int src_length, u8 *dst)
+{
+ int i, j;
+
+ dst = (u8 *)kmalloc((2 * src_length) * sizeof(u8), GFP_KERNEL);
+ j = 0;
+ for (i = 0; i < src_length; i++) {
+ dst[j] = (u8)(src[i] & 0xFF00) >> 8;
+ dst[j + 1] = (u8)(src[i] & 0x00FF);
+ j += 2;
+ }
+
+ return src_length * 2;
+}
+
+/**
+ * Convert a u16 to an array of 2 u8, dst has MSB first (big endian).
+ * @param src u16 to convert
+ * @param dst pointer to the destination array of 2 u8.
+ * @return OK
+ */
+int u16ToU8_be(u16 src, u8 *dst)
+{
+ dst[0] = (u8)((src & 0xFF00) >> 8);
+ dst[1] = (u8)(src & 0x00FF);
+ return OK;
+}
+
+
+/**
+ * Convert a u16 to an array of 2 u8, dst has LSB first (little endian).
+ * @param src u16 to convert
+ * @param dst pointer to the destination array of 2 u8.
+ * @return OK
+ */
+int u16ToU8(u16 src, u8 *dst)
+{
+ dst[1] = (u8)((src & 0xFF00) >> 8);
+ dst[0] = (u8)(src & 0x00FF);
+ return OK;
+}
+
+/**
+ * Convert an array of bytes to a u32, src has LSB first (little endian).
+ * @param src array of bytes to convert
+ * @param dst pointer to the destination u32 variable.
+ * @return OK
+ */
+int u8ToU32(u8 *src, u32 *dst)
+{
+ *dst = (u32)(((src[3] & 0xFF) << 24) + ((src[2] & 0xFF) << 16) +
+ ((src[1] & 0xFF) << 8) + (src[0] & 0xFF));
+ return OK;
+}
+
+/**
+ * Convert an array of bytes to a u32, src has MSB first (big endian).
+ * @param src array of bytes to convert
+ * @param dst pointer to the destination u32 variable.
+ * @return OK
+ */
+int u8ToU32_be(u8 *src, u32 *dst)
+{
+ *dst = (u32)(((src[0] & 0xFF) << 24) + ((src[1] & 0xFF) << 16) +
+ ((src[2] & 0xFF) << 8) + (src[3] & 0xFF));
+ return OK;
+}
+
+
+/**
+ * Convert a u32 to an array of 4 bytes, dst has LSB first (little endian).
+ * @param src u32 value to convert
+ * @param dst pointer to the destination array of 4 bytes.
+ * @return OK
+ */
+int u32ToU8(u32 src, u8 *dst)
+{
+ dst[3] = (u8)((src & 0xFF000000) >> 24);
+ dst[2] = (u8)((src & 0x00FF0000) >> 16);
+ dst[1] = (u8)((src & 0x0000FF00) >> 8);
+ dst[0] = (u8)(src & 0x000000FF);
+ return OK;
+}
+
+/**
+ * Convert a u32 to an array of 4 bytes, dst has MSB first (big endian).
+ * @param src u32 value to convert
+ * @param dst pointer to the destination array of 4 bytes.
+ * @return OK
+ */
+int u32ToU8_be(u32 src, u8 *dst)
+{
+ dst[0] = (u8)((src & 0xFF000000) >> 24);
+ dst[1] = (u8)((src & 0x00FF0000) >> 16);
+ dst[2] = (u8)((src & 0x0000FF00) >> 8);
+ dst[3] = (u8)(src & 0x000000FF);
+ return OK;
+}
+
+/**
+ * Execute a function passed as argment and retry it defined number of times if
+ * not successful
+ * @param code pointer to a function which return an int and doesn't have any
+ * parameters
+ * @param wait_before_retry interval of time in ms to wait between one trial
+ * and another one
+ * @param retry_count max number of retry to attemp
+ * @return last return value obtained from the last execution of the code
+ *function
+ */
+int attempt_function(int (*code)(void), unsigned long wait_before_retry, int
+ retry_count)
+{
+ int result;
+ int count = 0;
+
+ do {
+ result = code();
+ count++;
+ mdelay(wait_before_retry);
+ } while (count < retry_count && result < 0);
+
+
+ if (count == retry_count)
+ return result | ERROR_TIMEOUT;
+ else
+ return result;
+}
+
+/**
+ * Enable all the possible sensing mode supported by the FW
+ * @return OK if success or an error code which specify the type of error
+ */
+int senseOn(void)
+{
+ int ret;
+
+ ret = setScanMode(SCAN_MODE_ACTIVE, 0xFF); /* enable all */
+ if (ret < OK) {
+ pr_err("senseOn: ERROR %08X\n", ret);
+ return ret;
+ }
+
+ pr_info("senseOn: SENSE ON\n");
+ return OK;
+}
+
+/**
+ * Disable all the sensing mode
+ * @return OK if success or an error code which specify the type of error
+ */
+int senseOff(void)
+{
+ int ret;
+
+ ret = setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ if (ret < OK) {
+ pr_err("senseOff: ERROR %08X\n", ret);
+ return ret;
+ }
+
+ pr_info("senseOff: SENSE OFF\n");
+ return OK;
+}
+
+
+
+/**
+ * Clean up the IC status executing a system reset and giving
+ * the possibility to re-enabling the sensing
+ * @param enableTouch if 1, re-enable the sensing and the interrupt of the IC
+ * @return OK if success or an error code which specify the type of error
+ */
+int cleanUp(int enableTouch)
+{
+ int res;
+
+ pr_info("cleanUp: system reset...\n");
+ res = fts_system_reset();
+ if (res < OK)
+ return res;
+ if (enableTouch) {
+ pr_info("cleanUp: enabling touches...\n");
+ res = senseOn(); /* already enable everything */
+ if (res < OK)
+ return res;
+ pr_info("cleanUp: enabling interrupts...\n");
+ res = fts_enableInterrupt();
+ if (res < OK)
+ return res;
+ }
+ return OK;
+}
+
+/**
+ * Transform an array of short in a matrix of short with a defined number of
+ * columns and the resulting number of rows
+ * @param data array of bytes to convert
+ * @param size size of data
+ * @param columns number of columns that the resulting matrix should have.
+ * @return a reference to a matrix of short where for each row there are
+ * columns elements
+ */
+short **array1dTo2d_short(short *data, int size, int columns)
+{
+ int i;
+ short **matrix = (short **)kmalloc(((int)(size / columns)) *
+ sizeof(short *), GFP_KERNEL);
+
+ if (matrix != NULL) {
+ for (i = 0; i < (int)(size / columns); i++)
+ matrix[i] = (short *)kmalloc(columns * sizeof(short),
+ GFP_KERNEL);
+
+ for (i = 0; i < size; i++)
+ matrix[i / columns][i % columns] = data[i];
+ }
+
+ return matrix;
+}
+
+/**
+ * Transform an array of u16 in a matrix of u16 with a defined number of
+ * columns and the resulting number of rows
+ * @param data array of bytes to convert
+ * @param size size of data
+ * @param columns number of columns that the resulting matrix should have.
+ * @return a reference to a matrix of u16 where for each row there are columns
+ * elements
+ */
+u16 **array1dTo2d_u16(u16 *data, int size, int columns)
+{
+ int i;
+ u16 **matrix = (u16 **)kmalloc(((int)(size / columns)) * sizeof(u16 *),
+ GFP_KERNEL);
+
+ if (matrix != NULL) {
+ for (i = 0; i < (int)(size / columns); i++)
+ matrix[i] = (u16 *)kmalloc(columns * sizeof(u16),
+ GFP_KERNEL);
+
+ for (i = 0; i < size; i++)
+ matrix[i / columns][i % columns] = data[i];
+ }
+
+ return matrix;
+}
+
+/**
+ * Transform an array of u8 in a matrix of u8 with a defined number of
+ * columns and the resulting number of rows
+ * @param data array of bytes to convert
+ * @param size size of data
+ * @param columns number of columns that the resulting matrix should have.
+ * @return a reference to a matrix of short where for each row there are
+ * columns elements
+ */
+u8 **array1dTo2d_u8(u8 *data, int size, int columns)
+{
+ int i;
+ u8 **matrix = (u8 **)kmalloc(((int)(size / columns)) * sizeof(u8 *),
+ GFP_KERNEL);
+
+ if (matrix != NULL) {
+ for (i = 0; i < (int)(size / columns); i++)
+ matrix[i] = (u8 *)kmalloc(columns * sizeof(u8),
+ GFP_KERNEL);
+
+ for (i = 0; i < size; i++)
+ matrix[i / columns][i % columns] = data[i];
+ }
+
+ return matrix;
+}
+
+/**
+ * Transform an array of i8 in a matrix of i8 with a defined number of
+ * columns and the resulting number of rows
+ * @param data array of bytes to convert
+ * @param size size of data
+ * @param columns number of columns that the resulting matrix should have.
+ * @return a reference to a matrix of short where for each row there are
+ * columns elements
+ */
+i8 **array1dTo2d_i8(i8 *data, int size, int columns)
+{
+ int i;
+ i8 **matrix = (i8 **)kmalloc(((int)(size / columns)) * sizeof(i8 *),
+ GFP_KERNEL);
+
+ if (matrix != NULL) {
+ for (i = 0; i < (int)(size / columns); i++)
+ matrix[i] = (i8 *)kmalloc(columns * sizeof(i8),
+ GFP_KERNEL);
+
+ for (i = 0; i < size; i++)
+ matrix[i / columns][i % columns] = data[i];
+ }
+
+ return matrix;
+}
+
+/**
+ * Print in the kernel log a label followed by a matrix of short row x columns
+ * and free its memory
+ * @param label pointer to the string to print before the actual matrix
+ * @param matrix reference to the matrix of short which contain the actual data
+ * @param row number of rows on which the matrix should be print
+ * @param column number of columns for each row
+ */
+void print_frame_short(char *label, short **matrix, int row, int column)
+{
+ int i, j;
+ int buff_len, index;
+ char *buff;
+
+ buff_len = (6 + 1) * column + 1; /* -32768 str len: 6 */
+ buff = kzalloc(buff_len, GFP_KERNEL);
+ if (buff == NULL) {
+ pr_err("%s: fail to allocate buffer\n", __func__);
+ return;
+ }
+
+ pr_info("%s\n", label);
+ for (i = 0; i < row; i++) {
+ index = 0;
+ for (j = 0; j < column; j++)
+ index += scnprintf(buff + index, buff_len - index,
+ "%d ", matrix[i][j]);
+ pr_info("%s\n", buff);
+ kfree(matrix[i]);
+ }
+ kfree(matrix);
+ kfree(buff);
+}
+
+/**
+ * Print in the kernel log a label followed by a matrix of u16 row x columns
+ * and free its memory
+ * @param label pointer to the string to print before the actual matrix
+ * @param matrix reference to the matrix of u16 which contain the actual data
+ * @param row number of rows on which the matrix should be print
+ * @param column number of columns for each row
+ */
+void print_frame_u16(char *label, u16 **matrix, int row, int column)
+{
+ int i, j;
+ int buff_len, index;
+ char *buff;
+
+ buff_len = (5 + 1) * column + 1; /* 65535 str len: 5 */
+ buff = kzalloc(buff_len, GFP_KERNEL);
+ if (buff == NULL) {
+ pr_err("%s: fail to allocate buffer\n", __func__);
+ return;
+ }
+
+ pr_info("%s\n", label);
+ for (i = 0; i < row; i++) {
+ index = 0;
+ for (j = 0; j < column; j++)
+ index += scnprintf(buff + index, buff_len - index,
+ "%d ", matrix[i][j]);
+ pr_info("%s\n", buff);
+ kfree(matrix[i]);
+ }
+ kfree(matrix);
+ kfree(buff);
+}
+
+/**
+ * Print in the kernel log a label followed by a matrix of u8 row x columns and
+ * free its memory
+ * @param label pointer to the string to print before the actual matrix
+ * @param matrix reference to the matrix of u8 which contain the actual data
+ * @param row number of rows on which the matrix should be print
+ * @param column number of columns for each row
+ */
+void print_frame_u8(char *label, u8 **matrix, int row, int column)
+{
+ int i, j;
+ int buff_len, index;
+ char *buff;
+
+ buff_len = (3 + 1) * column + 1; /* 255 str len: 3 */
+ buff = kzalloc(buff_len, GFP_KERNEL);
+ if (buff == NULL) {
+ pr_err("%s: fail to allocate buffer\n", __func__);
+ return;
+ }
+
+ pr_info("%s\n", label);
+ for (i = 0; i < row; i++) {
+ index = 0;
+ for (j = 0; j < column; j++)
+ index += scnprintf(buff + index, buff_len - index,
+ "%d ", matrix[i][j]);
+ pr_info("%s\n", buff);
+ kfree(matrix[i]);
+ }
+ kfree(matrix);
+ kfree(buff);
+}
+
+/**
+ * Print in the kernel log a label followed by a matrix of i8 row x columns and
+ * free its memory
+ * @param label pointer to the string to print before the actual matrix
+ * @param matrix reference to the matrix of u8 which contain the actual data
+ * @param row number of rows on which the matrix should be print
+ * @param column number of columns for each row
+ */
+void print_frame_i8(char *label, i8 **matrix, int row, int column)
+{
+ int i, j;
+ int buff_len, index;
+ char *buff;
+
+ buff_len = (4 + 1) * column + 1; /* -128 str len: 4 */
+ buff = kzalloc(buff_len, GFP_KERNEL);
+ if (buff == NULL) {
+ pr_err("%s: fail to allocate buffer\n", __func__);
+ return;
+ }
+
+ pr_info("%s\n", label);
+ for (i = 0; i < row; i++) {
+ index = 0;
+ for (j = 0; j < column; j++)
+ index += scnprintf(buff + index, buff_len - index,
+ "%d ", matrix[i][j]);
+ pr_info("%s\n", buff);
+ kfree(matrix[i]);
+ }
+ kfree(matrix);
+ kfree(buff);
+}
+
+/**
+ * Print in the kernel log a label followed by a matrix of u32 row x columns
+ * and free its memory
+ * @param label pointer to the string to print before the actual matrix
+ * @param matrix reference to the matrix of u32 which contain the actual data
+ * @param row number of rows on which the matrix should be print
+ * @param column number of columns for each row
+ */
+void print_frame_u32(char *label, u32 **matrix, int row, int column)
+{
+ int i, j;
+ int buff_len, index;
+ char *buff;
+
+ buff_len = (10 + 1) * column + 1; /* 4294967295 str len: 10 */
+ buff = kzalloc(buff_len, GFP_KERNEL);
+ if (buff == NULL) {
+ pr_err("%s: fail to allocate buffer\n", __func__);
+ return;
+ }
+
+ pr_info("%s\n", label);
+ for (i = 0; i < row; i++) {
+ index = 0;
+ for (j = 0; j < column; j++)
+ index += scnprintf(buff + index, buff_len - index,
+ "%d ", matrix[i][j]);
+ pr_info("%s\n", buff);
+ kfree(matrix[i]);
+ }
+ kfree(matrix);
+ kfree(buff);
+}
+
+/**
+ * Print in the kernel log a label followed by a matrix of int row x columns
+ * and free its memory
+ * @param label pointer to the string to print before the actual matrix
+ * @param matrix reference to the matrix of int which contain the actual data
+ * @param row number of rows on which the matrix should be print
+ * @param column number of columns for each row
+ */
+void print_frame_int(char *label, int **matrix, int row, int column)
+{
+ int i, j;
+ int buff_len, index;
+ char *buff;
+
+ buff_len = (11 + 1) * column + 1; /* -2147483648 str len: 11 */
+ buff = kzalloc(buff_len, GFP_KERNEL);
+ if (buff == NULL) {
+ pr_err("%s: fail to allocate buffer\n", __func__);
+ return;
+ }
+
+ pr_info("%s\n", label);
+ for (i = 0; i < row; i++) {
+ index = 0;
+ for (j = 0; j < column; j++)
+ index += scnprintf(buff + index, buff_len - index,
+ "%d ", matrix[i][j]);
+ pr_info("%s\n", buff);
+ kfree(matrix[i]);
+ }
+ kfree(matrix);
+ kfree(buff);
+}
+
+
+/**
+ * Convert an array of bytes to an u64, src has MSB first (big endian).
+ * @param src array of bytes
+ * @param dest pointer to the destination u64.
+ * @param size size of src (can be <= 8)
+ * @return OK if success or ERROR_OP_NOT_ALLOW if size exceed 8
+ */
+int u8ToU64_be(u8 *src, u64 *dest, int size)
+{
+ int i = 0;
+
+ /* u64 temp =0; */
+ if (size > sizeof(u64))
+ return ERROR_OP_NOT_ALLOW;
+
+ *dest = 0;
+ for (i = 0; i < size; i++)
+ *dest |= (u64)(src[i]) << ((size - 1 - i) * 8);
+
+ return OK;
+}
+
+/**
+ * Convert an u64 to an array of bytes, dest has MSB first (big endian).
+ * @param src value of u64
+ * @param dest pointer to the destination array of bytes.
+ * @param size size of src (can be <= 8)
+ * @return OK if success or ERROR_OP_NOT_ALLOW if size exceed 8
+ */
+int u64ToU8_be(u64 src, u8 *dest, int size)
+{
+ int i = 0;
+
+ if (size > sizeof(u64))
+ return ERROR_OP_NOT_ALLOW;
+ else
+ for (i = 0; i < size; i++)
+ dest[i] = (u8)((src >> ((size - 1 - i) * 8)) & 0xFF);
+
+ return OK;
+}
+
+
+
+/*********** NEW API *************/
+
+/**
+ * Convert a value of an id in a bitmask with a 1 in the position of the value
+ * of the id
+ * @param id Value of the ID to convert
+ * @param mask pointer to the bitmask that will be updated with the value of id
+ * @param size dimension in bytes of mask
+ * @return OK if success or ERROR_OP_NOT_ALLOW if size of mask is not enough to
+ * contain ID
+ */
+int fromIDtoMask(u8 id, u8 *mask, int size)
+{
+ if (((int)((id) / 8)) < size) {
+ pr_info("%s: ID = %d Index = %d Position = %d !\n",
+ __func__, id, ((int)((id) / 8)), (id % 8));
+ mask[((int)((id) / 8))] |= 0x01 << (id % 8);
+ return OK;
+ } else {
+ pr_err("%s: Bitmask too small! Impossible contain ID = %d %d>=%d! ERROR %08X\n",
+ __func__, id, ((int)((id) / 8)), size,
+ ERROR_OP_NOT_ALLOW);
+ return ERROR_OP_NOT_ALLOW;
+ }
+}
diff --git a/drivers/input/touchscreen/stm/fts_lib/ftsTool.h b/drivers/input/touchscreen/stm/fts_lib/ftsTool.h
new file mode 100644
index 0000000..284ef6c
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_lib/ftsTool.h
@@ -0,0 +1,58 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com **
+ **************************************************************************
+ * *
+ * FTS Utility Functions *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file ftsTool.h
+ * \brief Contains all the definitions to support common operations inside the
+ * driver
+ */
+
+#ifndef FTS_TOOL_H
+#define FTS_TOOL_H
+
+char *printHex(char *label, u8 *buff, int count, u8 *result, int size);
+int u8ToU16(u8 *src, u16 *dst);
+int u8ToU16_be(u8 *src, u16 *dst);
+int u8ToU16n(u8 *src, int src_length, u16 *dst);
+int u16ToU8(u16 src, u8 *dst);
+int u16ToU8_be(u16 src, u8 *dst);
+int u16ToU8n_be(u16 *src, int src_length, u8 *dst);
+int u8ToU32(u8 *src, u32 *dst);
+int u8ToU32_be(u8 *src, u32 *dst);
+int u32ToU8(u32 src, u8 *dst);
+int u32ToU8_be(u32 src, u8 *dst);
+int u8ToU64_be(u8 *src, u64 *dest, int size);
+int u64ToU8_be(u64 src, u8 *dest, int size);
+int attempt_function(int (*code)(void), unsigned long wait_before_retry, int
+ retry_count);
+int senseOn(void);
+int senseOff(void);
+void print_frame_short(char *label, short **matrix, int row, int column);
+short **array1dTo2d_short(short *data, int size, int columns);
+void print_frame_u16(char *label, u16 **matrix, int row, int column);
+u16 **array1dTo2d_u16(u16 *data, int size, int columns);
+u8 **array1dTo2d_u8(u8 *data, int size, int columns);
+i8 **array1dTo2d_i8(i8 *data, int size, int columns);
+void print_frame_u8(char *label, u8 **matrix, int row, int column);
+void print_frame_i8(char *label, i8 **matrix, int row, int column);
+void print_frame_u32(char *label, u32 **matrix, int row, int column);
+void print_frame_int(char *label, int **matrix, int row, int column);
+int cleanUp(int enableTouch);
+int flushFIFO(void);
+
+/* New API */
+int fromIDtoMask(u8 id, u8 *mask, int size);
+
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_limits.h b/drivers/input/touchscreen/stm/fts_limits.h
new file mode 100644
index 0000000..5ecfa54
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_limits.h
@@ -0,0 +1,1064 @@
+/*!
+* \file fts_limits.h
+* \brief Contains the size and the byte array of the production test limit file which contains thresholds used to test data
+*/
+
+#ifndef FTS_LIMITS_H
+#define FTS_LIMITS_H
+//This is an auto generated header file
+//--->Remember to change the name of the two variables!<---
+const uint32_t myArray2_size = 16725; ///< size of the array containing the production limit file
+
+/**
+ * Array containing the production limit file
+ */
+const uint8_t myArray2[] = {
+ 0x2A, 0x53, 0x54, 0x4F, 0x50, 0x5F, 0x4F, 0x4E, 0x5F, 0x46, 0x41, 0x49, 0x4C, 0x2C, 0x31, 0x2C,
+ 0x31, 0x0A, 0x31, 0x0A, 0x2A, 0x53, 0x54, 0x4F, 0x50, 0x5F, 0x4F, 0x4E, 0x5F, 0x45, 0x52, 0x52,
+ 0x4F, 0x52, 0x2C, 0x31, 0x2C, 0x31, 0x0A, 0x31, 0x0A, 0x2A, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F,
+ 0x46, 0x57, 0x5F, 0x55, 0x50, 0x44, 0x41, 0x54, 0x45, 0x2C, 0x31, 0x2C, 0x31, 0x0A, 0x30, 0x0A,
+ 0x2A, 0x53, 0x53, 0x5F, 0x49, 0x58, 0x31, 0x5F, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F, 0x57, 0x2C,
+ 0x31, 0x2C, 0x31, 0x0A, 0x32, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x49, 0x58, 0x32, 0x5F, 0x46, 0x4F,
+ 0x52, 0x43, 0x45, 0x5F, 0x57, 0x2C, 0x31, 0x2C, 0x31, 0x0A, 0x31, 0x0A, 0x2A, 0x53, 0x53, 0x5F,
+ 0x49, 0x58, 0x31, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x57, 0x2C, 0x31, 0x2C, 0x31, 0x0A,
+ 0x32, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x49, 0x58, 0x32, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F,
+ 0x57, 0x2C, 0x31, 0x2C, 0x31, 0x0A, 0x31, 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x52, 0x41, 0x57, 0x5F,
+ 0x44, 0x41, 0x54, 0x41, 0x5F, 0x4D, 0x49, 0x4E, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32,
+ 0x0A, 0x31, 0x30, 0x30, 0x30, 0x2C, 0x31, 0x35, 0x30, 0x30, 0x30, 0x0A, 0x2A, 0x4D, 0x53, 0x5F,
+ 0x52, 0x41, 0x57, 0x5F, 0x44, 0x41, 0x54, 0x41, 0x5F, 0x47, 0x41, 0x50, 0x2C, 0x31, 0x2C, 0x31,
+ 0x0A, 0x33, 0x30, 0x30, 0x30, 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F,
+ 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x43, 0x58, 0x31, 0x5F, 0x4D, 0x49, 0x4E, 0x5F, 0x4D,
+ 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32, 0x0A, 0x30, 0x2C, 0x36, 0x33, 0x0A, 0x2A, 0x4D, 0x53, 0x5F,
+ 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x43, 0x58, 0x32,
+ 0x5F, 0x4D, 0x49, 0x4E, 0x2C, 0x32, 0x32, 0x2C, 0x32, 0x37, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x2A,
+ 0x4D, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F,
+ 0x43, 0x58, 0x32, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x32, 0x32, 0x2C, 0x32, 0x37, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43,
+ 0x54, 0x49, 0x56, 0x45, 0x5F, 0x43, 0x58, 0x32, 0x5F, 0x41, 0x44, 0x4A, 0x5F, 0x48, 0x4F, 0x52,
+ 0x49, 0x5A, 0x4F, 0x4E, 0x54, 0x41, 0x4C, 0x2C, 0x32, 0x32, 0x2C, 0x32, 0x36, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49,
+ 0x56, 0x45, 0x5F, 0x43, 0x58, 0x32, 0x5F, 0x41, 0x44, 0x4A, 0x5F, 0x56, 0x45, 0x52, 0x54, 0x49,
+ 0x43, 0x41, 0x4C, 0x2C, 0x32, 0x31, 0x2C, 0x32, 0x37, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x2A,
+ 0x4D, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F,
+ 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x43, 0x58, 0x5F, 0x4D, 0x49, 0x4E, 0x2C, 0x32, 0x32, 0x2C,
+ 0x32, 0x37, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48,
+ 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x43, 0x58,
+ 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x32, 0x32, 0x2C, 0x32, 0x37, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56,
+ 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x43, 0x58, 0x5F, 0x41, 0x44, 0x4A, 0x5F, 0x48,
+ 0x4F, 0x52, 0x49, 0x5A, 0x4F, 0x4E, 0x54, 0x41, 0x4C, 0x2C, 0x32, 0x32, 0x2C, 0x32, 0x36, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43,
+ 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x43, 0x58, 0x5F, 0x41, 0x44,
+ 0x4A, 0x5F, 0x56, 0x45, 0x52, 0x54, 0x49, 0x43, 0x41, 0x4C, 0x2C, 0x32, 0x31, 0x2C, 0x32, 0x37,
+ 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36,
+ 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34,
+ 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C,
+ 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x4B, 0x45, 0x59, 0x5F, 0x52, 0x41,
+ 0x57, 0x5F, 0x44, 0x41, 0x54, 0x41, 0x5F, 0x4D, 0x49, 0x4E, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31,
+ 0x2C, 0x32, 0x0A, 0x30, 0x2C, 0x38, 0x30, 0x30, 0x30, 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x4B, 0x45,
+ 0x59, 0x5F, 0x43, 0x58, 0x31, 0x5F, 0x4D, 0x49, 0x4E, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C,
+ 0x32, 0x0A, 0x30, 0x2C, 0x36, 0x34, 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x4B, 0x45, 0x59, 0x5F, 0x43,
+ 0x58, 0x32, 0x5F, 0x4D, 0x49, 0x4E, 0x2C, 0x31, 0x2C, 0x33, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x0A, 0x2A, 0x4D, 0x53, 0x5F, 0x4B, 0x45, 0x59, 0x5F, 0x43, 0x58, 0x32, 0x5F, 0x4D, 0x41, 0x58,
+ 0x2C, 0x31, 0x2C, 0x33, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x2A, 0x4D,
+ 0x53, 0x5F, 0x4B, 0x45, 0x59, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x43, 0x58, 0x5F, 0x4D,
+ 0x49, 0x4E, 0x2C, 0x31, 0x2C, 0x33, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x2A, 0x4D, 0x53,
+ 0x5F, 0x4B, 0x45, 0x59, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x43, 0x58, 0x5F, 0x4D, 0x41,
+ 0x58, 0x2C, 0x31, 0x2C, 0x33, 0x0A, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x2C, 0x36, 0x34, 0x0A, 0x2A,
+ 0x53, 0x53, 0x5F, 0x52, 0x41, 0x57, 0x5F, 0x44, 0x41, 0x54, 0x41, 0x5F, 0x46, 0x4F, 0x52, 0x43,
+ 0x45, 0x5F, 0x4D, 0x49, 0x4E, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32, 0x0A, 0x31, 0x30,
+ 0x30, 0x30, 0x2C, 0x31, 0x35, 0x30, 0x30, 0x30, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x52, 0x41, 0x57,
+ 0x5F, 0x44, 0x41, 0x54, 0x41, 0x5F, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F, 0x47, 0x41, 0x50, 0x2C,
+ 0x31, 0x2C, 0x31, 0x0A, 0x33, 0x30, 0x30, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43,
+ 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x49, 0x58, 0x31, 0x5F, 0x46, 0x4F, 0x52,
+ 0x43, 0x45, 0x5F, 0x4D, 0x49, 0x4E, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32, 0x0A, 0x30,
+ 0x2C, 0x36, 0x33, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43,
+ 0x54, 0x49, 0x56, 0x45, 0x5F, 0x49, 0x58, 0x32, 0x5F, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F, 0x4D,
+ 0x49, 0x4E, 0x2C, 0x32, 0x32, 0x2C, 0x31, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A,
+ 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A,
+ 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A,
+ 0x30, 0x0A, 0x30, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43,
+ 0x54, 0x49, 0x56, 0x45, 0x5F, 0x49, 0x58, 0x32, 0x5F, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F, 0x4D,
+ 0x41, 0x58, 0x2C, 0x32, 0x32, 0x2C, 0x31, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45,
+ 0x5F, 0x49, 0x58, 0x32, 0x5F, 0x41, 0x44, 0x4A, 0x5F, 0x56, 0x45, 0x52, 0x54, 0x49, 0x43, 0x41,
+ 0x4C, 0x2C, 0x32, 0x31, 0x2C, 0x31, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32,
+ 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32,
+ 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32,
+ 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32,
+ 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32,
+ 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54,
+ 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41,
+ 0x4C, 0x5F, 0x49, 0x58, 0x5F, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F, 0x4D, 0x49, 0x4E, 0x2C, 0x32,
+ 0x32, 0x2C, 0x31, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A,
+ 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A,
+ 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A,
+ 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45,
+ 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x49, 0x58, 0x5F, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F,
+ 0x4D, 0x41, 0x58, 0x2C, 0x32, 0x32, 0x2C, 0x31, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56,
+ 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x49, 0x58, 0x5F, 0x41, 0x44, 0x4A, 0x5F, 0x56,
+ 0x45, 0x52, 0x54, 0x49, 0x43, 0x41, 0x4C, 0x2C, 0x32, 0x31, 0x2C, 0x31, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56,
+ 0x45, 0x5F, 0x43, 0x58, 0x31, 0x5F, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F, 0x4D, 0x49, 0x4E, 0x5F,
+ 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32, 0x0A, 0x30, 0x2C, 0x36, 0x33, 0x0A, 0x2A, 0x53, 0x53,
+ 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x43, 0x58,
+ 0x32, 0x5F, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F, 0x4D, 0x49, 0x4E, 0x2C, 0x32, 0x32, 0x2C, 0x31,
+ 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30,
+ 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30,
+ 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x2A, 0x53, 0x53,
+ 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x43, 0x58,
+ 0x32, 0x5F, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x32, 0x32, 0x2C, 0x31,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35,
+ 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55,
+ 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x43, 0x58, 0x32, 0x5F, 0x41, 0x44,
+ 0x4A, 0x5F, 0x56, 0x45, 0x52, 0x54, 0x49, 0x43, 0x41, 0x4C, 0x2C, 0x32, 0x31, 0x2C, 0x31, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A,
+ 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43,
+ 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x43, 0x58, 0x5F, 0x46, 0x4F,
+ 0x52, 0x43, 0x45, 0x5F, 0x4D, 0x49, 0x4E, 0x2C, 0x32, 0x32, 0x2C, 0x31, 0x0A, 0x30, 0x0A, 0x30,
+ 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30,
+ 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30,
+ 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x30, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55,
+ 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F,
+ 0x43, 0x58, 0x5F, 0x46, 0x4F, 0x52, 0x43, 0x45, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x32, 0x32, 0x2C,
+ 0x31, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35,
+ 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35,
+ 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35,
+ 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35,
+ 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35,
+ 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F,
+ 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C,
+ 0x5F, 0x43, 0x58, 0x5F, 0x41, 0x44, 0x4A, 0x5F, 0x56, 0x45, 0x52, 0x54, 0x49, 0x43, 0x41, 0x4C,
+ 0x2C, 0x32, 0x31, 0x2C, 0x31, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35,
+ 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35,
+ 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35,
+ 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35,
+ 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35,
+ 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x52, 0x41,
+ 0x57, 0x5F, 0x44, 0x41, 0x54, 0x41, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x4D, 0x49, 0x4E,
+ 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32, 0x0A, 0x31, 0x30, 0x30, 0x30, 0x2C, 0x31, 0x35,
+ 0x30, 0x30, 0x30, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x52, 0x41, 0x57, 0x5F, 0x44, 0x41, 0x54, 0x41,
+ 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x47, 0x41, 0x50, 0x2C, 0x31, 0x2C, 0x31, 0x0A, 0x33,
+ 0x30, 0x30, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54,
+ 0x49, 0x56, 0x45, 0x5F, 0x49, 0x58, 0x31, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x4D, 0x49,
+ 0x4E, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32, 0x0A, 0x30, 0x2C, 0x36, 0x33, 0x0A, 0x2A,
+ 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F,
+ 0x49, 0x58, 0x32, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x4D, 0x49, 0x4E, 0x2C, 0x31, 0x2C,
+ 0x32, 0x37, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55,
+ 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x49, 0x58, 0x32, 0x5F, 0x53, 0x45,
+ 0x4E, 0x53, 0x45, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32, 0x37, 0x0A, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55,
+ 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x49, 0x58, 0x32, 0x5F, 0x41, 0x44,
+ 0x4A, 0x5F, 0x48, 0x4F, 0x52, 0x49, 0x5A, 0x4F, 0x4E, 0x54, 0x41, 0x4C, 0x2C, 0x31, 0x2C, 0x32,
+ 0x36, 0x0A, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F,
+ 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C,
+ 0x5F, 0x49, 0x58, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x4D, 0x49, 0x4E, 0x2C, 0x31, 0x2C,
+ 0x32, 0x37, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30,
+ 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55,
+ 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F,
+ 0x49, 0x58, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32,
+ 0x37, 0x0A, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53,
+ 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54,
+ 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x49, 0x58, 0x5F, 0x41, 0x44, 0x4A, 0x5F, 0x48, 0x4F, 0x52, 0x49,
+ 0x5A, 0x4F, 0x4E, 0x54, 0x41, 0x4C, 0x2C, 0x31, 0x2C, 0x32, 0x36, 0x0A, 0x32, 0x35, 0x35, 0x2C,
+ 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C,
+ 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C,
+ 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C,
+ 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C,
+ 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C,
+ 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C,
+ 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43,
+ 0x54, 0x49, 0x56, 0x45, 0x5F, 0x43, 0x58, 0x31, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x4D,
+ 0x49, 0x4E, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32, 0x0A, 0x30, 0x2C, 0x36, 0x33, 0x0A,
+ 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45,
+ 0x5F, 0x43, 0x58, 0x32, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x4D, 0x49, 0x4E, 0x2C, 0x31,
+ 0x2C, 0x32, 0x37, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C,
+ 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C,
+ 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C,
+ 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F,
+ 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x43, 0x58, 0x32, 0x5F, 0x53,
+ 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C, 0x32, 0x37, 0x0A, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35,
+ 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F,
+ 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x43, 0x58, 0x32, 0x5F, 0x41,
+ 0x44, 0x4A, 0x5F, 0x48, 0x4F, 0x52, 0x49, 0x5A, 0x4F, 0x4E, 0x54, 0x41, 0x4C, 0x2C, 0x31, 0x2C,
+ 0x32, 0x36, 0x0A, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54,
+ 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41,
+ 0x4C, 0x5F, 0x43, 0x58, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x4D, 0x49, 0x4E, 0x2C, 0x31,
+ 0x2C, 0x32, 0x37, 0x0A, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C,
+ 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C,
+ 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C,
+ 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x2C, 0x30, 0x0A, 0x2A, 0x53, 0x53, 0x5F, 0x54, 0x4F,
+ 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F, 0x54, 0x4F, 0x54, 0x41, 0x4C,
+ 0x5F, 0x43, 0x58, 0x5F, 0x53, 0x45, 0x4E, 0x53, 0x45, 0x5F, 0x4D, 0x41, 0x58, 0x2C, 0x31, 0x2C,
+ 0x32, 0x37, 0x0A, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32,
+ 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x0A, 0x2A,
+ 0x53, 0x53, 0x5F, 0x54, 0x4F, 0x55, 0x43, 0x48, 0x5F, 0x41, 0x43, 0x54, 0x49, 0x56, 0x45, 0x5F,
+ 0x54, 0x4F, 0x54, 0x41, 0x4C, 0x5F, 0x43, 0x58, 0x5F, 0x41, 0x44, 0x4A, 0x5F, 0x48, 0x4F, 0x52,
+ 0x49, 0x5A, 0x4F, 0x4E, 0x54, 0x41, 0x4C, 0x2C, 0x31, 0x2C, 0x32, 0x36, 0x0A, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35, 0x2C, 0x32, 0x35, 0x35,
+ 0x2C, 0x32, 0x35, 0x35, 0x0A,
+};
+
+#endif
diff --git a/drivers/input/touchscreen/stm/fts_proc.c b/drivers/input/touchscreen/stm/fts_proc.c
new file mode 100644
index 0000000..2b5f84d
--- /dev/null
+++ b/drivers/input/touchscreen/stm/fts_proc.c
@@ -0,0 +1,3344 @@
+/*
+ *
+ **************************************************************************
+ ** STMicroelectronics **
+ **************************************************************************
+ ** marco.cali@st.com *
+ **************************************************************************
+ * *
+ * Utilities published in /proc/fts *
+ * *
+ **************************************************************************
+ **************************************************************************
+ *
+ */
+
+/*!
+ * \file fts_proc.c
+ * \brief contains the function and variables needed to publish a file node in
+ * the file system which allow to communicate with the IC from userspace
+ */
+
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/proc_fs.h>
+#include <linux/fs.h>
+#include <linux/seq_file.h>
+#include <linux/delay.h>
+#include <linux/uaccess.h>
+#include "fts.h"
+#include "fts_lib/ftsCompensation.h"
+#include "fts_lib/ftsCore.h"
+#include "fts_lib/ftsIO.h"
+#include "fts_lib/ftsError.h"
+#include "fts_lib/ftsFrame.h"
+#include "fts_lib/ftsFlash.h"
+#include "fts_lib/ftsTest.h"
+#include "fts_lib/ftsTime.h"
+#include "fts_lib/ftsTool.h"
+
+
+#define DRIVER_TEST_FILE_NODE "driver_test" /* /< name of file node
+ * published */
+#define CHUNK_PROC 1024 /* /< Max chunk of data printed on
+ * the sequential file in each
+ * iteration */
+#define DIAGNOSTIC_NUM_FRAME 10 /* /< number of frames reading
+ * iterations during the diagnostic
+ * test */
+
+
+
+/** @defgroup proc_file_code Proc File Node
+ * @ingroup file_nodes
+ * The /proc/fts/driver_test file node provide expose the most important API
+ * implemented into the driver to execute any possible operation into the IC \n
+ * Thanks to a series of Operation Codes, each of them, with a different set of
+ * parameter, it is possible to select a function to execute\n
+ * The result of the function is usually returned into the shell as an ASCII
+ * hex string where each byte is encoded in two chars.\n
+ * @{
+ */
+
+/* Bus operations */
+#define CMD_READ 0x00 /* /< I2C/SPI read: need
+ * to pass: byteToRead1
+ * byteToRead0
+ * (optional) dummyByte
+ * */
+#define CMD_WRITE 0x01 /* /< I2C/SPI write:
+ * need to pass: cmd[0]
+ * cmd[1] …
+ * cmd[cmdLength-1] */
+#define CMD_WRITEREAD 0x02 /* /< I2C/SPI writeRead:
+ * need to pass: cmd[0]
+ * cmd[1] …
+ * cmd[cmdLength-1]
+ * byteToRead1
+ * byteToRead0 dummyByte
+ * */
+#define CMD_WRITETHENWRITEREAD 0x03 /* /< I2C/SPI write then
+ * writeRead: need to
+ * pass: cmdSize1
+ * cmdSize2 cmd1[0]
+ * cmd1[1] …
+ * cmd1[cmdSize1-1]
+ * cmd2[0] cmd2[1] …
+ * cmd2[cmdSize2-1]
+ * byteToRead1
+ * byteToRead0 */
+#define CMD_WRITEU8UX 0x04
+ /* /< I2C/SPI
+ * writeU8UX:
+ * need to pass: cmd
+ * addrSize addr[0] …
+ * addr[addrSize-1]
+ * data[0] data[1] … */
+#define CMD_WRITEREADU8UX 0x05 /* /< I2C/SPI
+ * writeReadU8UX: need
+ * to pass: cmd addrSize
+ * addr[0] …
+ * addr[addrSize-1]
+ * byteToRead1
+ * byteToRead0
+ * hasDummyByte */
+#define CMD_WRITEU8UXTHENWRITEU8UX 0x06
+ /* /< I2C/SPI writeU8UX
+ * then writeU8UX: need
+ * to pass: cmd1
+ * addrSize1 cmd2
+ * addrSize2 addr[0] …
+ * addr[addrSize1+
+ * addrSize2-1]
+ * data[0] data[1] … */
+#define CMD_WRITEU8UXTHENWRITEREADU8UX 0x07 /* /< I2C/SPI writeU8UX
+ * then writeReadU8UX:
+ * need to pass: cmd1
+ * addrSize1 cmd2
+ * addrSize2 addr[0] …
+ * addr[addrSize1+
+ * addrSize2-1]
+ * byteToRead1
+ * byteToRead0
+ * hasDummybyte */
+#define CMD_GETLIMITSFILE 0x08 /* /< Get the Production
+ * Limits File and print
+ * its content into the
+ * shell: need to pass:
+ * path(optional)
+ * otherwise select the
+ * approach chosen at
+ * compile time */
+#define CMD_GETFWFILE 0x09 /* /< Get the FW file
+ * and print its content
+ * into the shell: need
+ * to pass: path
+ * (optional) otherwise
+ * select the approach
+ * chosen at compile
+ * time */
+#define CMD_VERSION 0x0A /* /< Get the driver
+ * version and other
+ * driver setting info
+ * */
+#define CMD_READCONFIG 0x0B /* /< Read The config
+ * memory, need to pass:
+ * addr[0] addr[1]
+ * byteToRead0
+ * byteToRead1 */
+
+
+/* GUI utils byte ver */
+#define CMD_READ_BYTE 0xF0 /* /< Byte output
+ * version of I2C/SPI
+ * read @see CMD_READ */
+#define CMD_WRITE_BYTE 0xF1 /* /< Byte output
+ * version of I2C/SPI
+ * write @see CMD_WRITE
+ * */
+#define CMD_WRITEREAD_BYTE 0xF2 /* /< Byte output
+ * version of I2C/SPI
+ * writeRead @see
+ * CMD_WRITEREAD */
+#define CMD_WRITETHENWRITEREAD_BYTE 0xF3
+ /* /< Byte output
+ * version of I2C/SPI
+ * write then writeRead
+ * @see
+ * CMD_WRITETHENWRITEREAD
+ * */
+#define CMD_WRITEU8UX_BYTE 0xF4 /* /< Byte output
+ * version of I2C/SPI
+ * writeU8UX @see
+ * CMD_WRITEU8UX */
+#define CMD_WRITEREADU8UX_BYTE 0xF5 /* /< Byte output
+ * version of I2C/SPI
+ * writeReadU8UX @see
+ * CMD_WRITEREADU8UX */
+#define CMD_WRITEU8UXTHENWRITEU8UX_BYTE 0xF6
+ /* /< Byte output
+ * version of I2C/SPI
+ * writeU8UX then
+ * writeU8UX @see
+ * CMD_WRITEU8UXTHENWRITEU8UX
+ * */
+#define CMD_WRITEU8UXTHENWRITEREADU8UX_BYTE 0xF7
+ /* /< Byte output
+ * version of I2C/SPI
+ * writeU8UX then
+ * writeReadU8UX @see
+ * CMD_WRITEU8UXTHENWRITEREADU8UX
+ * */
+#define CMD_GETLIMITSFILE_BYTE 0xF8 /* /< Byte output
+ * version of Production
+ * Limits File @see
+ * CMD_GETLIMITSFILE */
+#define CMD_GETFWFILE_BYTE 0xF9 /* /< Byte output
+ * version of FW file
+ * need to pass: @see
+ * CMD_GETFWFILE */
+#define CMD_VERSION_BYTE 0xFA /* /< Byte output
+ * version of driver
+ * version and setting
+ * @see CMD_VERSION */
+#define CMD_CHANGE_OUTPUT_MODE 0xFF /* /< Select the output
+ * mode of the
+ * scriptless protocol,
+ * need to pass:
+ * bin_output = 1 data
+ * returned as binary,
+ * bin_output =0 data
+ * returned as hex
+ * string */
+
+/* Core/Tools */
+#define CMD_POLLFOREVENT 0x11 /* /< Poll the FIFO for
+ * an event: need to
+ * pass: eventLength
+ * event[0] event[1] …
+ * event[eventLength-1]
+ * timeToWait1
+ * timeToWait0 */
+#define CMD_SYSTEMRESET 0x12 /* /< System Reset */
+#define CMD_CLEANUP 0x13 /* /< Perform a system
+ * reset and optionally
+ * re-enable the
+ * scanning, need to
+ * pass: enableTouch */
+#define CMD_POWERCYCLE 0x14 /* /< Execute a power
+ * cycle toggling the
+ * regulators */
+#define CMD_READSYSINFO 0x15 /* /< Read the System
+ * Info information from
+ * the framebuffer, need
+ * to pass: doRequest */
+#define CMD_FWWRITE 0x16
+ /* /< Write a FW
+ * command: need to
+ * pass: cmd[0] cmd[1]
+ * … cmd[cmdLength-1] */
+#define CMD_INTERRUPT 0x17 /* /< Allow to enable or
+ * disable the
+ * interrupts, need to
+ * pass: enable (if 1
+ * will enable the
+ * interrupt) */
+
+/* Frame */
+#define CMD_GETFORCELEN 0x20 /* /< Get the number of
+ * Force channels */
+#define CMD_GETSENSELEN 0x21 /* /< Get the number of
+ * Sense channels */
+#define CMD_GETMSFRAME 0x23 /* /< Get a MS frame:
+ * need to pass:
+ * MSFrameType */
+#define CMD_GETSSFRAME 0x24 /* /< Get a SS frame:
+ * need to pass:
+ * SSFrameType */
+
+/* Compensation */
+#define CMD_REQCOMPDATA 0x30 /* /< Request Init data:
+ * need to pass: type */
+#define CMD_READCOMPDATAHEAD 0x31 /* /< Read Init data
+ * header: need to pass:
+ * type */
+#define CMD_READMSCOMPDATA 0x32 /* /< Read MS Init data:
+ * need to pass: type */
+#define CMD_READSSCOMPDATA 0x33 /* /< Read SS Init data:
+ * need to pass: type */
+#define CMD_READTOTMSCOMPDATA 0x35 /* /< Read Tot MS Init
+ * data: need to pass:
+ * type */
+#define CMD_READTOTSSCOMPDATA 0x36 /* /< Read Tot SS Init
+ * data: need to pass:
+ * type */
+#define CMD_READSENSCOEFF 0x37 /* /< Read MS and SS
+ * Sensitivity
+ * Calibration
+ * Coefficients */
+
+/* FW Update */
+#define CMD_GETFWVER 0x40 /* /< Get the FW version
+ * of the IC */
+#define CMD_FLASHUNLOCK 0x42 /* /< Unlock the flash
+ * */
+#define CMD_READFWFILE 0x43 /* /< Try to read the FW
+ * file, need to pass:
+ * keep_cx */
+#define CMD_FLASHPROCEDURE 0x44 /* /< Perform a full
+ * flashing procedure:
+ * need to pass: force
+ * keep_cx */
+#define CMD_FLASHERASEUNLOCK 0x45 /* /< Unlock the erase
+ * of the flash */
+#define CMD_FLASHERASEPAGE 0x46
+ /* /< Erase page by page
+ * the flash, need to
+ * pass: keep_cx, if
+ * keep_cx>SKIP_PANEL_INIT
+ * Panel Init Page will
+ * be skipped, if
+ * >SKIP_PANEL_CX_INIT
+ * Cx and Panel Init
+ * Pages will be skipped
+ * otherwise if
+ * =ERASE_ALL all the
+ * pages will be deleted
+ * */
+
+/* MP test */
+#define CMD_ITOTEST 0x50 /* /< Perform an ITO
+ * test */
+#define CMD_INITTEST 0x51 /* /< Perform an
+ * Initialization test:
+ * need to pass: type */
+#define CMD_MSRAWTEST 0x52 /* /< Perform MS raw
+ * test: need to pass
+ * stop_on_fail */
+#define CMD_MSINITDATATEST 0x53 /* /< Perform MS Init
+ * data test: need to
+ * pass stop_on_fail */
+#define CMD_SSRAWTEST 0x54 /* /< Perform SS raw
+ * test: need to pass
+ * stop_on_fail */
+#define CMD_SSINITDATATEST 0x55 /* /< Perform SS Init
+ * data test: need to
+ * pass stop_on_fail */
+#define CMD_MAINTEST 0x56 /* /< Perform a full
+ * Mass production test:
+ * need to pass
+ * stop_on_fail saveInit
+ * */
+#define CMD_FREELIMIT 0x57 /* /< Free (if any)
+ * limit file which was
+ * loaded during any
+ * test procedure */
+
+/* Diagnostic */
+#define CMD_DIAGNOSTIC 0x60 /* /< Perform a series
+ * of commands and
+ * collect severals data
+ * to detect any
+ * malfunction */
+
+#define CMD_CHANGE_SAD 0x70 /* /< Allow to change
+ * the SAD address (for
+ * debugging) */
+
+/* Debug functionalities requested by Google for B1 Project */
+#define CMD_TRIGGER_FORCECAL 0x80 /* /< Trigger manually
+ * forcecal for MS and
+ * SS */
+#define CMD_BASELINE_ADAPTATION 0x81 /* /< Enable/Disable
+ * Baseline adaptation,
+ * need to pass: enable
+ * */
+#define CMD_FREQ_HOP 0x82 /* /< Enable/Disable
+ * Frequency hopping,
+ * need to pass: enable
+ * */
+#define CMD_SET_OPERATING_FREQ 0x83 /* /< Set a defined
+ * scanning frequency in
+ * Hz passed as 4 bytes
+ * in big endian, need
+ * to pass: freq3 freq2
+ * freq1 freq0 */
+#define CMD_READ_SYNC_FRAME 0x84
+ /* /< Read Sync Frame
+ * which contain MS and
+ * SS data, need to
+ * pass: frameType (this
+ * parameter can be
+ * LOAD_SYNC_FRAME_STRENGTH
+ * or LOAD_SYNC_FRAME_BASELINE)
+ * */
+
+
+#define CMD_TP_SENS_MODE 0x90 /* /< Enter/Exit in the
+ * TP Sensitivity
+ * Calibration mode,
+ * need to pass: enter
+ * (optional)saveGain */
+#define CMD_TP_SENS_SET_SCAN_MODE 0x91 /* /< Set scan mode type
+ * which should be used
+ * for the test before
+ * the stimpad is down,
+ * need to pass: type */
+#define CMD_TP_SENS_PRECAL_SS 0x92 /* /< Perform Pre
+ * Calibration for SS
+ * steps when stimpad is
+ * down */
+#define CMD_TP_SENS_PRECAL_MS 0x93 /* /< Perform Pre
+ * Calibration for MS
+ * steps when stimpad is
+ * down */
+#define CMD_TP_SENS_POSTCAL_MS 0x94 /* /< Perform Post
+ * Calibration for MS
+ * steps when stimpad is
+ * down */
+#define CMD_TP_SENS_STD 0x95 /* /< Compute the
+ * Standard deviation of
+ * a certain number of
+ * frames, need to pass:
+ * numFrames */
+
+#define CMD_FORCE_TOUCH_ACTIVE 0xA0 /* /< Prevent the driver
+ * from transitioning
+ * the ownership of the
+ * bus to SLPI
+ */
+
+static u8 bin_output; /* /< Select the output type of the scriptless
+ * protocol (binary = 1 or hex string = 0) */
+/** @}*/
+
+/** @defgroup scriptless Scriptless Protocol
+ * @ingroup proc_file_code
+ * Scriptless Protocol allows ST Software (such as FingerTip Studio etc) to
+ * communicate with the IC from an user space.
+ * This mode gives access to common bus operations (write, read etc) and
+ * support additional functionalities. \n
+ * The protocol is based on exchange of binary messages included between a
+ * start and an end byte
+ * @{
+ */
+
+#define MESSAGE_START_BYTE 0x7B /* /< start byte of each message
+ * transferred in Scriptless Mode */
+#define MESSAGE_END_BYTE 0x7D /* /< end byte of each message
+ * transferred in Scriptless Mode */
+#define MESSAGE_MIN_HEADER_SIZE 8 /* /< minimun number of bytes of the
+ * structure of a messages exchanged
+ * with host (include start/end byte,
+ * counter, actions, msg_size) */
+
+/**
+ * Possible actions that can be requested by an host
+ */
+typedef enum {
+ ACTION_WRITE = (u16) 0x0001, /* /< Bus Write
+ * */
+ ACTION_READ = (u16) 0x0002, /* /< Bus Read
+ * */
+ ACTION_WRITE_READ = (u16) 0x0003, /* /< Bus Write
+ * followed by a
+ * Read */
+ ACTION_GET_VERSION = (u16) 0x0004, /* /< Get
+ * Version of
+ * the protocol
+ * (equal to the
+ * first 2 bye
+ * of driver
+ * version) */
+ ACTION_WRITEU8UX = (u16) 0x0011, /* /< Bus Write
+ * with support
+ * to different
+ * address size
+ * */
+ ACTION_WRITEREADU8UX = (u16) 0x0012, /* /< Bus
+ * writeRead
+ * with support
+ * to different
+ * address size
+ * */
+ ACTION_WRITETHENWRITEREAD = (u16) 0x0013, /* /< Bus write
+ * followed by a
+ * writeRead */
+ ACTION_WRITEU8XTHENWRITEREADU8UX = (u16) 0x0014, /* /< Bus write
+ * followed by a
+ * writeRead
+ * with support
+ * to different
+ * address size
+ * */
+ ACTION_WRITEU8UXTHENWRITEU8UX = (u16) 0x0015, /* /< Bus write
+ * followed by a
+ * write with
+ * support to
+ * different
+ * address size
+ * */
+ ACTION_GET_FW = (u16) 0x1000, /* /< Get Fw
+ * file content
+ * used by the
+ * driver */
+ ACTION_GET_LIMIT = (u16) 0x1001 /* /< Get Limit
+ * File content
+ * used by the
+ * driver */
+} Actions;
+
+/**
+ * Struct used to contain info of the message received by the host in
+ * Scriptless mode
+ */
+typedef struct {
+ u16 msg_size; /* /< total size of the message in bytes */
+ u16 counter; /* /< counter ID to identify a message */
+ Actions action; /* /< type of operation requested by the host @see
+ * Actions */
+ u8 dummy; /* /< (optional)in case of any kind of read operations,
+ * specify if the first byte is dummy */
+} Message;
+
+/** @}*/
+
+
+
+extern TestToDo tests;
+extern SysInfo systemInfo;
+
+static int limit; /* /< store the amount of data to print into the
+ * shell */
+static int chunk; /* /< store the chuk of data that should be printed in
+ * this iteration */
+static int printed; /* /< store the amount of data already printed in the
+ * shell */
+static struct proc_dir_entry *fts_dir; /* /< reference to the directory
+ * fts under /proc */
+static u8 *driver_test_buff; /* /< pointer to an array of bytes used
+ * to store the result of the function
+ * executed */
+char buf_chunk[CHUNK_PROC] = { 0 }; /* /< buffer used to store the message
+ * info received */
+static Message mess = { 0 }; /* /< store the information of the Scriptless
+ * message received */
+
+
+/************************ SEQUENTIAL FILE UTILITIES **************************/
+/**
+ * This function is called at the beginning of the stream to a sequential file
+ * or every time into the sequential were already written PAGE_SIZE bytes and
+ * the stream need to restart
+ * @param s pointer to the sequential file on which print the data
+ * @param pos pointer to the offset where write the data
+ * @return NULL if there is no data to print or the pointer to the beginning of
+ * the data that need to be printed
+ */
+static void *fts_seq_start(struct seq_file *s, loff_t *pos)
+{
+ pr_info("%s: Entering start(), pos = %lld limit = %d printed = %d\n",
+ __func__, *pos, limit, printed);
+
+ if (driver_test_buff == NULL && *pos == 0) {
+ int size = 13 * sizeof(u8);
+
+ pr_info("%s: No data to print!\n", __func__);
+ driver_test_buff = (u8 *)kmalloc(size, GFP_KERNEL);
+ limit = scnprintf(driver_test_buff,
+ size,
+ "{ %08X }\n", ERROR_OP_NOT_ALLOW);
+ /* pr_err("%s: len = %d driver_test_buff = %s\n",
+ * __func__, limit, driver_test_buff); */
+ } else {
+ if (*pos != 0)
+ *pos += chunk - 1;
+
+ if (*pos >= limit)
+ /* pr_err("%s: Apparently, we're done.\n", __func__); */
+ return NULL;
+ }
+
+ chunk = CHUNK_PROC;
+ if (limit - *pos < CHUNK_PROC)
+ chunk = limit - *pos;
+ /* pr_err("%s: In start(),
+ * updated pos = %Ld limit = %d printed = %d chunk = %d\n",
+ * __func__, *pos, limit, printed, chunk); */
+ memset(buf_chunk, 0, CHUNK_PROC);
+ memcpy(buf_chunk, &driver_test_buff[(int)*pos], chunk);
+
+ return buf_chunk;
+}
+
+/**
+ * This function actually print a chunk amount of data in the sequential file
+ * @param s pointer to the sequential file where to print the data
+ * @param v pointer to the data to print
+ * @return 0
+ */
+static int fts_seq_show(struct seq_file *s, void *v)
+{
+ /* pr_err("%s: In show()\n", __func__); */
+ seq_write(s, (u8 *)v, chunk);
+ printed += chunk;
+ return 0;
+}
+
+/**
+ * This function update the pointer and the counters to the next data to be
+ * printed
+ * @param s pointer to the sequential file where to print the data
+ * @param v pointer to the data to print
+ * @param pos pointer to the offset where write the next data
+ * @return NULL if there is no data to print or the pointer to the beginning of
+ * the next data that need to be printed
+ */
+static void *fts_seq_next(struct seq_file *s, void *v, loff_t *pos)
+{
+ /* int* val_ptr; */
+ /* pr_err("%s: In next(), v = %X, pos = %Ld.\n", __func__,
+ * v, *pos); */
+ (*pos) += chunk;/* increase my position counter */
+ chunk = CHUNK_PROC;
+
+ /* pr_err("%s: In next(),
+ * updated pos = %Ld limit = %d printed = %d\n",
+ * __func__, *pos, limit,printed); */
+ if (*pos >= limit) /* are we done? */
+ return NULL;
+ else if (limit - *pos < CHUNK_PROC)
+ chunk = limit - *pos;
+
+
+ memset(buf_chunk, 0, CHUNK_PROC);
+ memcpy(buf_chunk, &driver_test_buff[(int)*pos], chunk);
+ return buf_chunk;
+}
+
+
+/**
+ * This function is called when there are no more data to print the stream
+ *need to be terminated or when PAGE_SIZE data were already written into the
+ *sequential file
+ * @param s pointer to the sequential file where to print the data
+ * @param v pointer returned by fts_seq_next
+ */
+static void fts_seq_stop(struct seq_file *s, void *v)
+{
+ /* pr_err("%s: Entering stop().\n", __func__); */
+
+ if (v) {
+ /* pr_err("%s: v is %X.\n", __func__, v); */
+ } else {
+ /* pr_err("%s: v is null.\n", __func__); */
+ limit = 0;
+ chunk = 0;
+ printed = 0;
+ if (driver_test_buff != NULL) {
+ /* pr_err("%s: Freeing and clearing driver_test_buff.\n",
+ * __func__); */
+ kfree(driver_test_buff);
+ driver_test_buff = NULL;
+ } else {
+ /* pr_err("%s: driver_test_buff is already null.\n",
+ * __func__); */
+ }
+ }
+}
+
+/**
+ * Struct where define and specify the functions which implements the flow for
+ * writing on a sequential file
+ */
+static const struct seq_operations fts_seq_ops = {
+ .start = fts_seq_start,
+ .next = fts_seq_next,
+ .stop = fts_seq_stop,
+ .show = fts_seq_show
+};
+
+/**
+ * This function open a sequential file
+ * @param inode Inode in the file system that was called and triggered this
+ * function
+ * @param file file associated to the file node
+ * @return error code, 0 if success
+ */
+static int fts_open(struct inode *inode, struct file *file)
+{
+ return seq_open(file, &fts_seq_ops);
+};
+
+
+/*****************************************************************************/
+
+/**************************** DRIVER TEST ************************************/
+
+/** @addtogroup proc_file_code
+ * @{
+ */
+
+/**
+ * Receive the OP code and the inputs from shell when the file node is called,
+ * parse it and then execute the corresponding function
+ * echo cmd+parameters > /proc/fts/driver_test to execute the select command
+ * cat /proc/fts/driver_test to obtain the result into the
+ * shell \n
+ * the string returned in the shell is made up as follow: \n
+ * { = start byte \n
+ * the answer content and format strictly depend on the cmd executed. In
+ * general can be: an HEX string or a byte array (e.g in case of 0xF- commands)
+ * \n
+ * } = end byte \n
+ */
+static ssize_t fts_driver_test_write(struct file *file, const char __user *buf,
+ size_t count, loff_t *pos)
+{
+ int numberParam = 0;
+ struct fts_ts_info *info = dev_get_drvdata(getDev());
+ char *p = NULL;
+ char *pbuf = NULL;
+ char path[100] = { 0 };
+ int res = -1, j, index = 0;
+ int size = 6;
+ int temp, byte_call = 0;
+ u16 byteToRead = 0;
+ u32 fileSize = 0;
+ u8 *readData = NULL;
+ u8 *cmd = NULL; /* worst case needs count bytes */
+ u32 *funcToTest = NULL;
+ u64 addr = 0;
+ MutualSenseFrame frameMS;
+ MutualSenseFrame deltas;
+ SelfSenseFrame frameSS;
+
+ DataHeader dataHead;
+ MutualSenseData compData;
+ SelfSenseData comData;
+ TotMutualSenseData totCompData;
+ TotSelfSenseData totComData;
+ MutualSenseCoeff msCoeff;
+ SelfSenseCoeff ssCoeff;
+ int meanNorm = 0, meanEdge = 0;
+
+ u64 address;
+
+ Firmware fw;
+ LimitFile lim;
+
+ mess.dummy = 0;
+ mess.action = 0;
+ mess.msg_size = 0;
+
+ if (fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, true) < 0) {
+ res = ERROR_BUS_WR;
+ pr_err("%s: bus is not accessible.\n", __func__);
+ if (driver_test_buff)
+ limit = scnprintf(driver_test_buff, size, "{ %08X }\n",
+ res);
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+ goto ERROR;
+ }
+
+ pbuf = kmalloc(count * sizeof(*pbuf), GFP_KERNEL);
+ if (!pbuf) {
+ res = ERROR_ALLOC;
+ goto ERROR;
+ }
+
+ funcToTest = kmalloc(((count + 1) / 3) * sizeof(*funcToTest),
+ GFP_KERNEL);
+ if (!funcToTest) {
+ res = ERROR_ALLOC;
+ goto ERROR;
+ }
+
+ /*for(temp = 0; temp<count; temp++){
+ * pr_err("p[%d] = %02X\n", temp, p[temp]);
+ * }*/
+ if (access_ok(VERIFY_READ, buf, count) < OK ||
+ copy_from_user(pbuf, buf, count) != 0) {
+ res = ERROR_ALLOC;
+ goto END;
+ }
+
+ cmd = (u8 *)kmalloc_array(count, sizeof(u8), GFP_KERNEL);
+ if (cmd == NULL) {
+ res = ERROR_ALLOC;
+ pr_err("%s: Impossible allocate memory... ERROR %08X!\n",
+ __func__, res);
+ goto ERROR;
+ }
+
+ p = pbuf;
+ if (count > MESSAGE_MIN_HEADER_SIZE - 1 && p[0] == MESSAGE_START_BYTE) {
+ pr_info("Enter in Byte Mode!\n");
+ byte_call = 1;
+ mess.msg_size = (p[1] << 8) | p[2];
+ mess.counter = (p[3] << 8) | p[4];
+ mess.action = (p[5] << 8) | p[6];
+ pr_info("Message received: size = %d, counter_id = %d, action = %04X\n",
+ mess.msg_size, mess.counter, mess.action);
+ size = MESSAGE_MIN_HEADER_SIZE + 2; /* +2 error code */
+ if (count < mess.msg_size || p[count - 2] != MESSAGE_END_BYTE) {
+ pr_err("number of byte received or end byte wrong! msg_size = %d != %zu, last_byte = %02X != %02X ... ERROR %08X\n",
+ mess.msg_size, count, p[count - 1],
+ MESSAGE_END_BYTE, ERROR_OP_NOT_ALLOW);
+ res = ERROR_OP_NOT_ALLOW;
+ goto END;
+ } else {
+ numberParam = mess.msg_size - MESSAGE_MIN_HEADER_SIZE +
+ 1; /* +1 because put the internal
+ * op code */
+ size = MESSAGE_MIN_HEADER_SIZE + 2; /* +2 send also
+ * the first 2
+ * lsb of the
+ * error code */
+ switch (mess.action) {
+ case ACTION_READ:
+ /* numberParam =
+ * mess.msg_size-MESSAGE_MIN_HEADER_SIZE+1; */
+ cmd[0] = funcToTest[0] = CMD_READ_BYTE;
+ break;
+
+ case ACTION_WRITE:
+ cmd[0] = funcToTest[0] = CMD_WRITE_BYTE;
+ break;
+
+ case ACTION_WRITE_READ:
+ cmd[0] = funcToTest[0] = CMD_WRITEREAD_BYTE;
+ break;
+
+ case ACTION_GET_VERSION:
+ cmd[0] = funcToTest[0] = CMD_VERSION_BYTE;
+ break;
+
+ case ACTION_WRITETHENWRITEREAD:
+ cmd[0] = funcToTest[0] =
+ CMD_WRITETHENWRITEREAD_BYTE;
+ break;
+
+ case ACTION_WRITEU8UX:
+ cmd[0] = funcToTest[0] = CMD_WRITEU8UX_BYTE;
+ break;
+
+ case ACTION_WRITEREADU8UX:
+ cmd[0] = funcToTest[0] = CMD_WRITEREADU8UX_BYTE;
+ break;
+
+ case ACTION_WRITEU8UXTHENWRITEU8UX:
+ cmd[0] = funcToTest[0] =
+ CMD_WRITEU8UXTHENWRITEU8UX_BYTE;
+ break;
+
+ case ACTION_WRITEU8XTHENWRITEREADU8UX:
+ cmd[0] = funcToTest[0] =
+ CMD_WRITEU8UXTHENWRITEREADU8UX_BYTE;
+ break;
+
+ case ACTION_GET_FW:
+ cmd[0] = funcToTest[0] = CMD_GETFWFILE_BYTE;
+ break;
+
+ case ACTION_GET_LIMIT:
+ cmd[0] = funcToTest[0] = CMD_GETLIMITSFILE_BYTE;
+ break;
+
+ default:
+ pr_err("Invalid Action = %d ... ERROR %08X\n",
+ mess.action, ERROR_OP_NOT_ALLOW);
+ res = ERROR_OP_NOT_ALLOW;
+ goto END;
+ }
+
+ if (numberParam - 1 != 0)
+ memcpy(&cmd[1], &p[7], numberParam - 1);
+ /* -1 because i need to exclude the cmd[0] */
+ }
+ } else {
+ if (((count + 1) / 3) >= 1) {
+ sscanf(p, "%02X ", &funcToTest[0]);
+ p += 3;
+ cmd[0] = (u8)funcToTest[0];
+ numberParam = 1;
+ } else {
+ res = ERROR_OP_NOT_ALLOW;
+ goto END;
+ }
+
+ pr_info("functionToTest[0] = %02X cmd[0]= %02X\n",
+ funcToTest[0], cmd[0]);
+ switch (funcToTest[0]) {
+ case CMD_GETFWFILE:
+ case CMD_GETLIMITSFILE:
+ if (count - 2 - 1 > 1) {
+ numberParam = 2;/** the first byte is an hex
+ * string coded in three byte
+ * (2 chars for hex and the
+ * space)
+ * and -1 for the space at the
+ * end */
+ sscanf(p, "%100s", path);
+ }
+ break;
+
+ default:
+ for (; numberParam < (count + 1) / 3; numberParam++) {
+ sscanf(p, "%02X ", &funcToTest[numberParam]);
+ p += 3;
+ cmd[numberParam] = (u8)funcToTest[numberParam];
+ pr_info("functionToTest[%d] = %02X cmd[%d]= %02X\n",
+ numberParam, funcToTest[numberParam],
+ numberParam, cmd[numberParam]);
+ }
+ }
+ }
+
+
+ fw.data = NULL;
+ lim.data = NULL;
+
+ pr_info("Number of Parameters = %d\n", numberParam);
+
+ /* elaborate input */
+ if (numberParam >= 1) {
+ switch (funcToTest[0]) {
+ case CMD_VERSION_BYTE:
+ pr_info("%s: Get Version Byte\n", __func__);
+ byteToRead = 2;
+ mess.dummy = 0;
+ readData = (u8 *)kmalloc(byteToRead * sizeof(u8),
+ GFP_KERNEL);
+ size += byteToRead;
+ if (readData != NULL) {
+ readData[0] = (u8)(FTS_TS_DRV_VER >> 24);
+ readData[1] = (u8)(FTS_TS_DRV_VER >> 16);
+ pr_info("%s: Version = %02X%02X\n",
+ __func__, readData[0], readData[1]);
+ res = OK;
+ } else {
+ res = ERROR_ALLOC;
+ pr_err("%s: Impossible allocate memory... ERROR %08X\n",
+ __func__, res);
+ }
+ break;
+
+
+ case CMD_VERSION:
+ byteToRead = 2 * sizeof(u32);
+ mess.dummy = 0;
+ readData = (u8 *)kmalloc(byteToRead * sizeof(u8),
+ GFP_KERNEL);
+ u32ToU8_be(FTS_TS_DRV_VER, readData);
+ fileSize = 0;
+ /* first two bytes bitmask of features enabled in the
+ * IC, second two bytes bitmask of features enabled in
+ * the driver */
+
+#ifdef FW_H_FILE
+ fileSize |= 0x00010000;
+#endif
+
+#ifdef LIMITS_H_FILE
+ fileSize |= 0x00020000;
+#endif
+
+#ifdef USE_ONE_FILE_NODE
+ fileSize |= 0x00040000;
+#endif
+
+#ifdef FW_UPDATE_ON_PROBE
+ fileSize |= 0x00080000;
+#endif
+
+#ifdef PRE_SAVED_METHOD
+ fileSize |= 0x00100000;
+#endif
+
+#ifdef USE_GESTURE_MASK
+ fileSize |= 0x00100000;
+#endif
+
+#ifdef I2C_INTERFACE
+ fileSize |= 0x00200000;
+#endif
+
+#ifdef PHONE_KEY /* it is a feature enabled in the config of the chip */
+ fileSize |= 0x00000100;
+#endif
+
+#ifdef GESTURE_MODE
+ fromIDtoMask(FEAT_SEL_GESTURE, (u8 *)&fileSize, 4);
+#endif
+
+
+#ifdef GRIP_MODE
+ fromIDtoMask(FEAT_SEL_GRIP, (u8 *)&fileSize, 4);
+#endif
+
+#ifdef CHARGER_MODE
+ fromIDtoMask(FEAT_SEL_CHARGER, (u8 *)&fileSize, 4);
+#endif
+
+#ifdef GLOVE_MODE
+ fromIDtoMask(FEAT_SEL_GLOVE, (u8 *)&fileSize, 4);
+#endif
+
+
+#ifdef COVER_MODE
+ fromIDtoMask(FEAT_SEL_COVER, (u8 *)&fileSize, 4);
+#endif
+
+#ifdef STYLUS_MODE
+ fromIDtoMask(FEAT_SEL_STYLUS, (u8 *)&fileSize, 4);
+#endif
+
+ u32ToU8_be(fileSize, &readData[4]);
+ res = OK;
+ size += (byteToRead * sizeof(u8));
+ break;
+
+ case CMD_WRITEREAD:
+ case CMD_WRITEREAD_BYTE:
+ if (numberParam >= 5) { /* need to pass: cmd[0] cmd[1]
+ * … cmd[cmdLength-1]
+ * byteToRead1 byteToRead0
+ * dummyByte */
+ temp = numberParam - 4;
+ if (cmd[numberParam - 1] == 0)
+ mess.dummy = 0;
+ else
+ mess.dummy = 1;
+
+ u8ToU16_be(&cmd[numberParam - 3], &byteToRead);
+ pr_info("bytesToRead = %d\n",
+ byteToRead + mess.dummy);
+
+ readData = (u8 *)kmalloc((byteToRead +
+ mess.dummy) *
+ sizeof(u8),
+ GFP_KERNEL);
+ res = fts_writeRead_heap(&cmd[1], temp,
+ readData, byteToRead + mess.dummy);
+ size += (byteToRead * sizeof(u8));
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_WRITE:
+ case CMD_WRITE_BYTE:
+ if (numberParam >= 2) { /* need to pass: cmd[0] cmd[1]
+ * … cmd[cmdLength-1] */
+ temp = numberParam - 1;
+
+ res = fts_write_heap(&cmd[1], temp);
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_READ:
+ case CMD_READ_BYTE:
+ if (numberParam >= 3) { /* need to pass: byteToRead1
+ * byteToRead0 (optional)
+ * dummyByte */
+ if (numberParam == 3 ||
+ (numberParam == 4 &&
+ cmd[numberParam - 1] == 0))
+ mess.dummy = 0;
+ else
+ mess.dummy = 1;
+ u8ToU16_be(&cmd[1], &byteToRead);
+ readData = (u8 *)kmalloc((byteToRead +
+ mess.dummy) *
+ sizeof(u8),
+ GFP_KERNEL);
+ res = fts_read_heap(readData, byteToRead +
+ mess.dummy);
+ size += (byteToRead * sizeof(u8));
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_WRITETHENWRITEREAD:
+ case CMD_WRITETHENWRITEREAD_BYTE:
+ /* need to pass: cmdSize1 cmdSize2 cmd1[0] cmd1[1] …
+ * cmd1[cmdSize1-1] cmd2[0] cmd2[1] … cmd2[cmdSize2-1]
+ * byteToRead1 byteToRead0 */
+ if (numberParam >= 6) {
+ u8ToU16_be(&cmd[numberParam - 2], &byteToRead);
+ readData = (u8 *)kmalloc(byteToRead *
+ sizeof(u8),
+ GFP_KERNEL);
+ res = fts_writeThenWriteRead_heap(
+ &cmd[3], cmd[1],
+ &cmd[3 + (int)cmd[1]], cmd[2],
+ readData, byteToRead);
+ size += (byteToRead * sizeof(u8));
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_WRITEU8UX:
+ case CMD_WRITEU8UX_BYTE:
+ /* need to pass:
+ * cmd addrSize addr[0] … addr[addrSize-1]
+ * data[0] data[1] … */
+ if (numberParam >= 4) {
+ if (cmd[2] <= sizeof(u64)) {
+ u8ToU64_be(&cmd[3], &addr, cmd[2]);
+ pr_info("addr = %llx\n", addr);
+ res = fts_writeU8UX(cmd[1], cmd[2],
+ addr,
+ &cmd[3 + cmd[2]],
+ (numberParam - cmd[2] - 3));
+ } else {
+ pr_err("Wrong address size!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+
+ case CMD_WRITEREADU8UX:
+ case CMD_WRITEREADU8UX_BYTE:
+ /* need to pass:
+ * cmd addrSize addr[0] … addr[addrSize-1]
+ * byteToRead1 byteToRead0 hasDummyByte */
+ if (numberParam >= 6) {
+ if (cmd[2] <= sizeof(u64)) {
+ u8ToU64_be(&cmd[3], &addr, cmd[2]);
+ u8ToU16_be(&cmd[numberParam - 3],
+ &byteToRead);
+ readData = (u8 *)kmalloc(byteToRead *
+ sizeof(u8),
+ GFP_KERNEL);
+ pr_info("addr = %llx byteToRead = %d\n",
+ addr, byteToRead);
+ res = fts_writeReadU8UX(cmd[1], cmd[2],
+ addr, readData,
+ byteToRead,
+ cmd[numberParam - 1]);
+ size += (byteToRead * sizeof(u8));
+ } else {
+ pr_err("Wrong address size!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_WRITEU8UXTHENWRITEU8UX:
+ case CMD_WRITEU8UXTHENWRITEU8UX_BYTE:
+ /* need to pass:
+ * cmd1 addrSize1 cmd2 addrSize2 addr[0] …
+ * addr[addrSize1+addrSize2-1] data[0] data[1] … */
+ if (numberParam >= 6) {
+ if ((cmd[2] + cmd[4]) <= sizeof(u64)) {
+ u8ToU64_be(&cmd[5], &addr, cmd[2] +
+ cmd[4]);
+
+ pr_info("addr = %llx\n", addr);
+ res = fts_writeU8UXthenWriteU8UX(cmd[1],
+ cmd[2], cmd[3],
+ cmd[4], addr,
+ &cmd[5 + cmd[2] + cmd[4]],
+ (numberParam - cmd[2]
+ - cmd[4] - 5));
+ } else {
+ pr_err("Wrong address size!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_WRITEU8UXTHENWRITEREADU8UX:
+ case CMD_WRITEU8UXTHENWRITEREADU8UX_BYTE:
+ /* need to pass:
+ * cmd1 addrSize1 cmd2 addrSize2 addr[0] …
+ * addr[addrSize1+addrSize2-1] byteToRead1 byteToRead0
+ * hasDummybyte */
+ if (numberParam >= 8) {
+ if ((cmd[2] + cmd[4]) <= sizeof(u64)) {
+ u8ToU64_be(&cmd[5], &addr, cmd[2] +
+ cmd[4]);
+ pr_info("%s: cmd[5] = %02X, addr = %llx\n",
+ __func__, cmd[5], addr);
+ u8ToU16_be(&cmd[numberParam - 3],
+ &byteToRead);
+ readData = (u8 *)kmalloc(byteToRead *
+ sizeof(u8),
+ GFP_KERNEL);
+ res = fts_writeU8UXthenWriteReadU8UX(
+ cmd[1], cmd[2], cmd[3], cmd[4],
+ addr,
+ readData, byteToRead,
+ cmd[numberParam - 1]);
+ size += (byteToRead * sizeof(u8));
+ } else {
+ pr_err("Wrong total address size!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+ break;
+
+ case CMD_CHANGE_OUTPUT_MODE:
+ /* need to pass: bin_output */
+ if (numberParam >= 2) {
+ bin_output = cmd[1];
+ pr_info("Setting Scriptless output mode: %d\n",
+ bin_output);
+ res = OK;
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_FWWRITE:
+ if (numberParam >= 3) { /* need to pass: cmd[0] cmd[1]
+ * … cmd[cmdLength-1] */
+ if (numberParam >= 2) {
+ temp = numberParam - 1;
+ res = fts_writeFwCmd_heap(&cmd[1],
+ temp);
+ } else {
+ pr_err("Wrong parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_INTERRUPT:
+ /* need to pass: enable */
+ if (numberParam >= 2) {
+ if (cmd[1] == 1)
+ res = fts_enableInterrupt();
+ else
+ res = fts_disableInterrupt();
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+
+ case CMD_READCONFIG:
+ if (numberParam == 5) { /* need to pass: addr[0]
+ * addr[1] byteToRead1
+ * byteToRead0 */
+ byteToRead = ((funcToTest[3] << 8) |
+ funcToTest[4]);
+ readData = (u8 *)kmalloc(byteToRead *
+ sizeof(u8),
+ GFP_KERNEL);
+ res = readConfig((u16)((((u8)funcToTest[1] &
+ 0x00FF) << 8) +
+ ((u8)funcToTest[2] &
+ 0x00FF)),
+ readData, byteToRead);
+ size += (byteToRead * sizeof(u8));
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_POLLFOREVENT:
+ if (numberParam >= 5) { /* need to pass: eventLength
+ * event[0] event[1] …
+ * event[eventLength-1]
+ * timeTowait1 timeTowait0 */
+ temp = (int)funcToTest[1];
+ if (numberParam == 5 + (temp - 1) &&
+ temp != 0) {
+ readData = (u8 *)kmalloc(
+ FIFO_EVENT_SIZE * sizeof(u8),
+ GFP_KERNEL);
+ res = pollForEvent(
+ (int *)&funcToTest[2], temp,
+ readData,
+ ((funcToTest[temp + 2] &
+ 0x00FF) << 8) +
+ (funcToTest[temp + 3] &
+ 0x00FF));
+ if (res >= OK)
+ res = OK; /* pollForEvent
+ * return the
+ * number of
+ * error found
+ * */
+ size += (FIFO_EVENT_SIZE * sizeof(u8));
+ byteToRead = FIFO_EVENT_SIZE;
+ } else {
+ pr_err("Wrong parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_SYSTEMRESET:
+ res = fts_system_reset();
+
+ break;
+
+ case CMD_READSYSINFO:
+ if (numberParam == 2) /* need to pass: doRequest */
+ res = readSysInfo(funcToTest[1]);
+ else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+ break;
+
+ case CMD_CLEANUP:/* TOUCH ENABLE/DISABLE */
+ if (numberParam == 2) /* need to pass: enableTouch */
+ res = cleanUp(funcToTest[1]);
+ else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+ break;
+
+ case CMD_GETFORCELEN: /* read number Tx channels */
+ temp = getForceLen();
+ if (temp < OK)
+ res = temp;
+ else {
+ size += (1 * sizeof(u8));
+ res = OK;
+ }
+ break;
+
+ case CMD_GETSENSELEN: /* read number Rx channels */
+ temp = getSenseLen();
+ if (temp < OK)
+ res = temp;
+ else {
+ size += (1 * sizeof(u8));
+ res = OK;
+ }
+ break;
+
+
+ case CMD_GETMSFRAME:
+ if (numberParam == 2) {
+ pr_info("Get 1 MS Frame\n");
+ setScanMode(SCAN_MODE_ACTIVE, 0xFF);
+ mdelay(WAIT_FOR_FRESH_FRAMES);
+ setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ mdelay(WAIT_AFTER_SENSEOFF);
+ /* flushFIFO(); //delete the events related to
+ * some touch (allow to call this function while
+ * touching the screen without having a flooding
+ * of the FIFO) */
+ res = getMSFrame3((MSFrameType)cmd[1],
+ &frameMS);
+ if (res < 0)
+ pr_err("Error while taking the MS frame... ERROR %08X\n",
+ res);
+
+ else {
+ pr_info("The frame size is %d words\n",
+ res);
+ size += (res * sizeof(short) + 2);
+ /* +2 to add force and sense channels
+ * set res to OK because if getMSFrame
+ * is successful
+ * res = number of words read
+ */
+ res = OK;
+ print_frame_short("MS frame =",
+ array1dTo2d_short(
+ frameMS.node_data,
+ frameMS.node_data_size,
+ frameMS.header.sense_node),
+ frameMS.header.force_node,
+ frameMS.header.sense_node);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+
+ /*read self raw*/
+ case CMD_GETSSFRAME:
+ if (numberParam == 2) {
+ pr_info("Get 1 SS Frame\n");
+ /* flushFIFO(); //delete the events related to
+ * some touch (allow to call this function while
+ * touching the screen without having a flooding
+ * of the FIFO) */
+ setScanMode(SCAN_MODE_ACTIVE, 0xFF);
+ mdelay(WAIT_FOR_FRESH_FRAMES);
+ setScanMode(SCAN_MODE_ACTIVE, 0x00);
+ mdelay(WAIT_AFTER_SENSEOFF);
+ res = getSSFrame3((SSFrameType)cmd[1],
+ &frameSS);
+
+ if (res < OK)
+ pr_err("Error while taking the SS frame... ERROR %08X\n",
+ res);
+
+ else {
+ pr_info("The frame size is %d words\n",
+ res);
+ size += (res * sizeof(short) + 2);
+ /* +2 to add force and sense channels
+ * set res to OK because if getMSFrame
+ * is successful
+ * res = number of words read
+ */
+ res = OK;
+ print_frame_short("SS force frame =",
+ array1dTo2d_short(
+ frameSS.force_data,
+ frameSS.header.force_node,
+ 1),
+ frameSS.header.force_node, 1);
+ print_frame_short("SS sense frame =",
+ array1dTo2d_short(
+ frameSS.sense_data,
+ frameSS.header.sense_node,
+ frameSS.header.sense_node),
+ 1,
+ frameSS.header.sense_node);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_REQCOMPDATA: /* request comp data */
+ if (numberParam == 2) {
+ pr_info("Requesting Compensation Data\n");
+ res = requestCompensationData(cmd[1]);
+
+ if (res < OK)
+ pr_err("Error requesting compensation data ERROR %08X\n",
+ res);
+ else
+ pr_info("Requesting Compensation Data Finished!\n");
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_READCOMPDATAHEAD: /* read comp data header */
+ if (numberParam == 2) {
+ pr_info("Requesting Compensation Data\n");
+ res = requestCompensationData(cmd[1]);
+ if (res < OK)
+ pr_err("Error requesting compensation data ERROR %08X\n",
+ res);
+ else {
+ pr_info("Requesting Compensation Data Finished!\n");
+ res = readCompensationDataHeader(
+ (u8)funcToTest[1], &dataHead,
+ &address);
+ if (res < OK)
+ pr_err("Read Compensation Data Header ERROR %08X\n",
+ res);
+ else {
+ pr_info("Read Compensation Data Header OK!\n");
+ size += (1 * sizeof(u8));
+ }
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+
+ case CMD_READMSCOMPDATA:/* read mutual comp data */
+ if (numberParam == 2) {
+ pr_info("Get MS Compensation Data\n");
+ res = readMutualSenseCompensationData(cmd[1],
+ &compData);
+
+ if (res < OK)
+ pr_err("Error reading MS compensation data ERROR %08X\n",
+ res);
+ else {
+ pr_info("MS Compensation Data Reading Finished!\n");
+ size = ((compData.node_data_size + 10) *
+ sizeof(i8));
+ print_frame_i8("MS Data (Cx2) =",
+ array1dTo2d_i8(
+ compData.node_data,
+ compData.node_data_size,
+ compData.header.sense_node),
+ compData.header.force_node,
+ compData.header.sense_node);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_READSSCOMPDATA:
+ if (numberParam == 2) { /* read self comp data */
+ pr_info("Get SS Compensation Data...\n");
+ res = readSelfSenseCompensationData(cmd[1],
+ &comData);
+ if (res < OK)
+ pr_err("Error reading SS compensation data ERROR %08X\n",
+ res);
+ else {
+ pr_info("SS Compensation Data Reading Finished!\n");
+ size = ((comData.header.force_node +
+ comData.header.sense_node) *
+ 2 + 13) *
+ sizeof(i8);
+ print_frame_i8("SS Data Ix2_fm = ",
+ array1dTo2d_i8(
+ comData.ix2_fm,
+ comData.header.force_node,
+ comData.header.force_node),
+ 1,
+ comData.header.force_node);
+ print_frame_i8("SS Data Cx2_fm = ",
+ array1dTo2d_i8(
+ comData.cx2_fm,
+ comData.header.force_node,
+ comData.header.force_node),
+ 1,
+ comData.header.force_node);
+ print_frame_i8("SS Data Ix2_sn = ",
+ array1dTo2d_i8(
+ comData.ix2_sn,
+ comData.header.sense_node,
+ comData.header.sense_node),
+ 1,
+ comData.header.sense_node);
+ print_frame_i8("SS Data Cx2_sn = ",
+ array1dTo2d_i8(
+ comData.cx2_sn,
+ comData.header.sense_node,
+ comData.header.sense_node),
+ 1,
+ comData.header.sense_node);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_READTOTMSCOMPDATA: /* read mutual comp data */
+ if (numberParam == 2) {
+ pr_info("Get TOT MS Compensation Data\n");
+ res = readTotMutualSenseCompensationData(cmd[1],
+ &totCompData);
+
+ if (res < OK)
+ pr_err("Error reading TOT MS compensation data ERROR %08X\n",
+ res);
+ else {
+ pr_info("TOT MS Compensation Data Reading Finished!\n");
+ size = (totCompData.node_data_size *
+ sizeof(short) + 9);
+ print_frame_short("MS Data (TOT Cx) =",
+ array1dTo2d_short(
+ totCompData.node_data,
+ totCompData.node_data_size,
+ totCompData.header.sense_node),
+ totCompData.header.force_node,
+ totCompData.header.sense_node);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_READTOTSSCOMPDATA:
+ if (numberParam == 2) { /* read self comp data */
+ pr_info("Get TOT SS Compensation Data...\n");
+ res = readTotSelfSenseCompensationData(cmd[1],
+ &totComData);
+ if (res < OK)
+ pr_err("Error reading TOT SS compensation data ERROR %08X\n",
+ res);
+ else {
+ pr_info("TOT SS Compensation Data Reading Finished!\n");
+ size = ((totComData.header.force_node +
+ totComData.header.sense_node) *
+ 2 *
+ sizeof(short) + 9);
+ print_frame_u16("SS Data TOT Ix_fm = ",
+ array1dTo2d_u16(
+ totComData.ix_fm,
+ totComData.header.force_node,
+ totComData.header.force_node),
+ 1,
+ totComData.header.force_node);
+ print_frame_short(
+ "SS Data TOT Cx_fm = ",
+ array1dTo2d_short(
+ totComData.cx_fm,
+ totComData.header.force_node,
+ totComData.header.force_node),
+ 1,
+ totComData.header.force_node);
+ print_frame_u16("SS Data TOT Ix_sn = ",
+ array1dTo2d_u16(
+ totComData.ix_sn,
+ totComData.header.sense_node,
+ totComData.header.sense_node),
+ 1,
+ totComData.header.sense_node);
+ print_frame_short(
+ "SS Data TOT Cx_sn = ",
+ array1dTo2d_short(
+ totComData.cx_sn,
+ totComData.header.sense_node,
+ totComData.header.sense_node),
+ 1,
+ totComData.header.sense_node);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_READSENSCOEFF:
+ /* read MS and SS Sensitivity Coefficients */
+ pr_info("Get Sensitivity Calibration Coefficients...\n");
+ res = readSensitivityCoefficientsData(&msCoeff,
+ &ssCoeff);
+ if (res < OK)
+ pr_err("Error reading Sensitivity Calibration Coefficients ERROR %08X\n",
+ res);
+ else {
+ pr_info("Sensitivity Calibration Coefficients Reading Finished!\n");
+ size += (((msCoeff.node_data_size) +
+ ssCoeff.header.force_node +
+ ssCoeff.header.sense_node) *
+ sizeof(u8) + 4);
+ print_frame_u8("MS Sensitivity Coeff = ",
+ array1dTo2d_u8(msCoeff.ms_coeff,
+ msCoeff.
+ node_data_size,
+ msCoeff.header.
+ sense_node),
+ msCoeff.header.force_node,
+ msCoeff.header.sense_node);
+ print_frame_u8("SS Sensitivity Coeff force = ",
+ array1dTo2d_u8(
+ ssCoeff.ss_force_coeff,
+ ssCoeff.header.
+ force_node, 1),
+ ssCoeff.header.force_node, 1);
+ print_frame_u8("SS Sensitivity Coeff sense = ",
+ array1dTo2d_u8(
+ ssCoeff.ss_sense_coeff,
+ ssCoeff.header.
+ sense_node,
+ ssCoeff.header.
+ sense_node), 1,
+ ssCoeff.header.sense_node);
+ }
+ break;
+
+ case CMD_GETFWVER:
+ size += (EXTERNAL_RELEASE_INFO_SIZE)*sizeof(u8);
+ break;
+
+ case CMD_FLASHUNLOCK:
+ res = flash_unlock();
+ if (res < OK)
+ pr_err("Impossible Unlock Flash ERROR %08X\n",
+ res);
+ else
+ pr_info("Flash Unlock OK!\n");
+ break;
+
+ case CMD_READFWFILE:
+ if (numberParam == 2) { /* read fw file */
+ pr_info("Reading FW File...\n");
+ res = readFwFile(info->board->fw_name, &fw,
+ funcToTest[1]);
+ if (res < OK)
+ pr_err("Error reading FW File ERROR %08X\n",
+ res);
+ else
+ pr_info("Read FW File Finished!\n");
+ kfree(fw.data);
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_FLASHPROCEDURE:
+ if (numberParam == 3) { /* flashing procedure */
+ pr_info("Starting Flashing Procedure...\n");
+ res = flashProcedure(info->board->fw_name,
+ cmd[1],
+ cmd[2]);
+ if (res < OK)
+ pr_err("Error during flash procedure ERROR %08X\n",
+ res);
+ else
+ pr_info("Flash Procedure Finished!\n");
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_FLASHERASEUNLOCK:
+ res = flash_erase_unlock();
+ if (res < OK)
+ pr_err("Error during flash erase unlock... ERROR %08X\n",
+ res);
+ else
+ pr_info("Flash Erase Unlock Finished!\n");
+ break;
+
+ case CMD_FLASHERASEPAGE:
+ if (numberParam == 2) { /* need to pass: keep_cx */
+ pr_info("Starting Flashing Page Erase...\n");
+ res = flash_erase_page_by_page(cmd[1]);
+ if (res < OK)
+ pr_err("Error during flash page erase... ERROR %08X\n",
+ res);
+ else
+ pr_info("Flash Page Erase Finished!\n");
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ /*ITO TEST*/
+ case CMD_ITOTEST:
+ res = production_test_ito(LIMITS_FILE, &tests);
+ break;
+
+ /*Initialization*/
+ case CMD_INITTEST:
+ if (numberParam == 2)
+ res = production_test_initialization(cmd[1]);
+
+ else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+
+ case CMD_MSRAWTEST: /* MS Raw DATA TEST */
+ if (numberParam == 2) /* need to specify if stopOnFail
+ * */
+ res = production_test_ms_raw(LIMITS_FILE,
+ cmd[1], &tests);
+ else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_MSINITDATATEST:/* MS CX DATA TEST */
+ if (numberParam == 2) /* need stopOnFail */
+ res = production_test_ms_cx(LIMITS_FILE, cmd[1],
+ &tests);
+ else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_SSRAWTEST: /* SS RAW DATA TEST */
+ if (numberParam == 2) /* need stopOnFail */
+ res = production_test_ss_raw(LIMITS_FILE,
+ cmd[1], &tests);
+ else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_SSINITDATATEST:/* SS IX CX DATA TEST */
+ if (numberParam == 2) /* need stopOnFail */
+ res = production_test_ss_ix_cx(LIMITS_FILE,
+ cmd[1], &tests);
+ else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ /*PRODUCTION TEST*/
+ case CMD_MAINTEST:
+ if (numberParam == 3) /* need to specify if stopOnFail
+ * and saveInit */
+ res = production_test_main(LIMITS_FILE, cmd[1],
+ cmd[2], &tests);
+ else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_FREELIMIT:
+ res = freeCurrentLimitsFile();
+ break;
+
+ case CMD_POWERCYCLE:
+ res = fts_chip_powercycle(info);
+ break;
+
+ case CMD_GETLIMITSFILE:
+ /* need to pass: path(optional) return error code +
+ * number of byte read otherwise GUI could not now how
+ * many byte read */
+ if (numberParam >= 1) {
+ lim.data = NULL;
+ lim.size = 0;
+ if (numberParam == 1)
+ res = getLimitsFile(LIMITS_FILE, &lim);
+ else
+ res = getLimitsFile(path, &lim);
+ readData = lim.data;
+ fileSize = lim.size;
+ size += (fileSize * sizeof(u8));
+ if (byte_call == 1)
+ size += sizeof(u32); /* transmit as
+ * first 4 bytes
+ * the size */
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_GETLIMITSFILE_BYTE:
+ /* need to pass: byteToRead1 byteToRead0 */
+ if (numberParam >= 3) {
+ lim.data = NULL;
+ lim.size = 0;
+
+ u8ToU16_be(&cmd[1], &byteToRead);
+ addr = ((u64)byteToRead) * 4; /* number of
+ * words */
+
+ res = getLimitsFile(LIMITS_FILE, &lim);
+
+ readData = lim.data;
+ fileSize = lim.size;
+
+ if (fileSize > addr) {
+ pr_err("Limits dimension expected by Host is less than actual size: expected = %d, real = %d\n",
+ byteToRead, fileSize);
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+ size += (addr * sizeof(u8));
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_GETFWFILE:
+ /* need to pass: from (optional) otherwise select the
+ * approach chosen at compile time */
+ if (numberParam >= 1) {
+ if (numberParam == 1)
+ res = getFWdata(info->board->fw_name,
+ &readData, &fileSize);
+ else
+ res = getFWdata(path, &readData,
+ &fileSize);
+
+ size += (fileSize * sizeof(u8));
+ if (byte_call == 1)
+ size += sizeof(u32); /* transmit as
+ * first 4 bytes
+ * the size */
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_GETFWFILE_BYTE:
+ /* need to pass: byteToRead1 byteToRead0 */
+ if (numberParam == 3) {
+ u8ToU16_be(&cmd[1], &byteToRead);
+ addr = ((u64)byteToRead) * 4; /* number of
+ * words */
+ res = getFWdata(info->board->fw_name, &readData,
+ &fileSize);
+ if (fileSize > addr) {
+ pr_err("FW dimension expected by Host is less than actual size: expected = %d, real = %d\n",
+ byteToRead, fileSize);
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+ size += (addr * sizeof(u8)); /* return always
+ * the amount
+ * requested by
+ * host, if real
+ * size is
+ * smaller, the
+ * data are
+ * padded to
+ * zero */
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ /* finish all the diagnostic command with a goto ERROR in order
+ * to skip the modification on driver_test_buff
+ * remember to set properly the limit and printed variables in
+ * order to make the seq_file logic to work */
+ case CMD_DIAGNOSTIC:
+ index = 0;
+ size = 0;
+ fileSize = 256 * 1024 * sizeof(char);
+ driver_test_buff = (u8 *)kzalloc(fileSize, GFP_KERNEL);
+ readData = (u8 *)kmalloc((ERROR_DUMP_ROW_SIZE *
+ ERROR_DUMP_COL_SIZE) *
+ sizeof(u8), GFP_KERNEL);
+ if (driver_test_buff == NULL || readData == NULL) {
+ res = ERROR_ALLOC;
+ pr_err("Impossible allocate memory for buffers! ERROR %08X!\n",
+ res);
+ goto END;
+ }
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "DIAGNOSTIC TEST:\n1) I2C Test: ");
+ index += j;
+
+ res = fts_writeReadU8UX(FTS_CMD_HW_REG_R,
+ ADDR_SIZE_HW_REG, ADDR_DCHIP_ID,
+ (u8 *)&temp, 2,
+ DUMMY_HW_REG);
+ if (res < OK) {
+ pr_err("Error during I2C test: ERROR %08X!\n",
+ res);
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index, "ERROR %08X\n",
+ res);
+ index += j;
+ res = ERROR_OP_NOT_ALLOW;
+ goto END_DIAGNOSTIC;
+ }
+
+ temp &= 0xFFFF;
+ pr_info("Chip ID = %04X!\n", temp);
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "DATA = %04X, expected = %02X%02X\n",
+ temp, DCHIP_ID_1,
+ DCHIP_ID_0);
+ index += j;
+ if (temp != ((DCHIP_ID_1 << 8) | DCHIP_ID_0)) {
+ pr_err("Wrong CHIP ID, Diagnostic failed!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ goto END_DIAGNOSTIC;
+ }
+
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "Present Driver Mode: %08X\n",
+ info->mode);
+ index += j;
+
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "2) FW running: Sensing On...");
+ index += j;
+ pr_info("Sensing On!\n");
+ readData[0] = FTS_CMD_SCAN_MODE;
+ readData[1] = SCAN_MODE_ACTIVE;
+ readData[2] = 0x1;
+ fts_write_heap(readData, 3);
+ res = checkEcho(readData, 3);
+ if (res < OK) {
+ pr_err("No Echo received.. ERROR %08X !\n",
+ res);
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "No echo found... ERROR %08X!\n",
+ res);
+ index += j;
+ goto END_DIAGNOSTIC;
+ } else {
+ pr_info("Echo FOUND... OK!\n");
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "Echo FOUND... OK!\n");
+ index += j;
+ }
+
+ pr_info("Reading Frames...!\n");
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "3) Read Frames:\n");
+ index += j;
+ for (temp = 0; temp < DIAGNOSTIC_NUM_FRAME; temp++) {
+ pr_info("Iteration n. %d...\n", temp + 1);
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "Iteration n. %d...\n",
+ temp + 1);
+ index += j;
+ for (addr = 0; addr < 3; addr++) {
+ switch (addr) {
+ case 0:
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "MS RAW FRAME =");
+ index += j;
+ res |= getMSFrame3(MS_RAW,
+ &frameMS);
+ break;
+ case 2:
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "MS STRENGTH FRAME =");
+ index += j;
+ res |= getMSFrame3(MS_STRENGTH,
+ &frameMS);
+ break;
+ case 1:
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "MS BASELINE FRAME =");
+ index += j;
+ res |= getMSFrame3(MS_BASELINE,
+ &frameMS);
+ break;
+ }
+ if (res < OK) {
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "No data! ERROR %08X\n",
+ res);
+ index += j;
+ } else {
+ for (address = 0; address <
+ frameMS.node_data_size;
+ address++) {
+ if (address %
+ frameMS.header.
+ sense_node == 0) {
+ j = scnprintf(
+ &driver_test_buff
+ [index],
+ fileSize -
+ index,
+ "\n");
+ index += j;
+ }
+ j = scnprintf(
+ &driver_test_buff
+ [index],
+ fileSize - index,
+ "%5d, ",
+ frameMS.
+ node_data[address]);
+ index += j;
+ }
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index, "\n");
+ index += j;
+ }
+ if (frameMS.node_data != NULL)
+ kfree(frameMS.node_data);
+ }
+ for (addr = 0; addr < 3; addr++) {
+ switch (addr) {
+ case 0:
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "SS RAW FRAME =\n");
+ index += j;
+ res |= getSSFrame3(SS_RAW,
+ &frameSS);
+ break;
+ case 2:
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "SS STRENGTH FRAME =\n");
+ index += j;
+ res |= getSSFrame3(SS_STRENGTH,
+ &frameSS);
+ break;
+ case 1:
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "SS BASELINE FRAME =\n");
+ index += j;
+ res |= getSSFrame3(SS_BASELINE,
+ &frameSS);
+ break;
+ }
+ if (res < OK) {
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "No data! ERROR %08X\n",
+ res);
+ index += j;
+ } else {
+ int num;
+ short *data;
+
+ num = frameSS.header.force_node;
+ data = frameSS.force_data;
+ for (address = 0;
+ address < num;
+ address++) {
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "%d\n",
+ data[address]);
+ index += j;
+ }
+
+ num = frameSS.header.sense_node;
+ data = frameSS.sense_data;
+ for (address = 0;
+ address < num;
+ address++) {
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "%d, ",
+ data[address]);
+ index += j;
+ }
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index, "\n");
+ index += j;
+ }
+ if (frameSS.force_data != NULL)
+ kfree(frameSS.force_data);
+ if (frameSS.sense_data != NULL)
+ kfree(frameSS.sense_data);
+ }
+ }
+
+
+ pr_info("Reading error info...\n");
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "4) FW INFO DUMP: ");
+ index += j;
+ temp = dumpErrorInfo(readData, ERROR_DUMP_ROW_SIZE *
+ ERROR_DUMP_COL_SIZE);
+ /* OR to detect if there are failures also in the
+ * previous reading of frames and write the correct
+ * result */
+ if (temp < OK) {
+ pr_err("Error during dump: ERROR %08X!\n",
+ res);
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index, "ERROR %08X\n",
+ temp);
+ index += j;
+ } else {
+ pr_info("DUMP OK!\n");
+ for (temp = 0; temp < ERROR_DUMP_ROW_SIZE *
+ ERROR_DUMP_COL_SIZE; temp++) {
+ if (temp % ERROR_DUMP_COL_SIZE == 0) {
+ j = scnprintf(
+ &driver_test_buff[index],
+ fileSize - index,
+ "\n%2d - ",
+ temp / ERROR_DUMP_COL_SIZE);
+ index += j;
+ }
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index, "%02X ",
+ readData[temp]);
+ index += j;
+ }
+ }
+ res |= temp;
+
+END_DIAGNOSTIC:
+ if (res < OK) {
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "\nRESULT = FAIL\n");
+ index += j;
+ } else {
+ j = scnprintf(&driver_test_buff[index],
+ fileSize - index,
+ "\nRESULT = FINISHED\n");
+ index += j;
+ }
+ /* the sting is already terminated with the null char by
+ * scnprintf */
+ limit = index;
+ printed = 0;
+ goto ERROR;
+ break;
+
+ case CMD_CHANGE_SAD:
+ res = changeSAD(cmd[1]);
+ break;
+
+ case CMD_TRIGGER_FORCECAL:
+ cmd[0] = CAL_MS_TOUCH | CAL_SS_TOUCH;
+ cmd[1] = 0x00;
+ fts_disableInterrupt();
+ res = writeSysCmd(SYS_CMD_FORCE_CAL, cmd, 2);
+ res |= fts_enableInterrupt();
+ if (res < OK)
+ pr_err("can not trigger Force Cal! ERROR %08X\n",
+ res);
+ else
+ pr_info("MS and SS force cal triggered!\n");
+ break;
+
+ case CMD_BASELINE_ADAPTATION:
+ /* need to pass: enable */
+ if (numberParam == 2) {
+ if (cmd[1] == 0x01)
+ pr_info("Enabling Baseline adaptation...\n");
+ else {
+ pr_info("Disabling Baseline adaptation...\n");
+ cmd[1] = 0x00; /* set to zero to
+ * disable baseline
+ * adaptation */
+ }
+
+ res = writeConfig(ADDR_CONFIG_AUTOCAL, &cmd[1],
+ 1);
+ if (res < OK)
+ pr_err("Baseline adaptation operation FAILED! ERROR %08X\n",
+ res);
+ else
+ pr_info("Baseline adaptation operation OK!\n");
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_FREQ_HOP:
+ /* need to pass: enable */
+ if (numberParam == 2) {
+ pr_info("Reading MNM register...\n");
+ res = readConfig(ADDR_CONFIG_MNM, &cmd[2], 1);
+ if (res < OK) {
+ pr_err("Reading MNM register... ERROR %08X!\n",
+ res);
+ break;
+ }
+
+ if (cmd[1] == 0x01) {
+ pr_info("Enabling Frequency Hopping... %02X => %02X\n",
+ cmd[2], cmd[2] | 0x01);
+ cmd[2] |= 0x01; /* set bit 0 to enable
+ * Frequency Hopping */
+ } else {
+ pr_info("Disabling Frequency Hopping... %02X => %02X\n",
+ cmd[2], cmd[2] & (~0x01));
+ cmd[2] &= (~0x01); /* reset bit 0
+ * to disable
+ * Frequency
+ * Hopping */
+ }
+
+ res = writeConfig(ADDR_CONFIG_MNM, &cmd[2], 1);
+ if (res < OK)
+ pr_err("Frequency Hopping operation FAILED! ERROR %08X\n",
+ res);
+ else
+ pr_info("Frequency Hopping operation OK!\n");
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_READ_SYNC_FRAME:
+ /* need to pass: frameType (this parameter can be
+ * LOAD_SYNC_FRAME_STRENGTH or LOAD_SYNC_FRAME_BASELINE)
+ * */
+ if (numberParam == 2) {
+ pr_info("Reading Sync Frame...\n");
+ res = getSyncFrame(cmd[1], &frameMS, &frameSS);
+ if (res < OK)
+ pr_err("Error while taking the Sync Frame frame... ERROR %08X\n",
+ res);
+
+ else {
+ pr_info("The total frames size is %d words\n",
+ res);
+ size += (res * sizeof(short) + 4);
+ /* +4 to add force and sense channels
+ * for MS and SS
+ * set res to OK because if getSyncFrame
+ * is successful
+ * res = number of words read
+ */
+ res = OK;
+
+ print_frame_short("MS frame =",
+ array1dTo2d_short(
+ frameMS.node_data,
+ frameMS.node_data_size,
+ frameMS.header.sense_node),
+ frameMS.header.force_node,
+ frameMS.header.sense_node);
+ print_frame_short("SS force frame =",
+ array1dTo2d_short(
+ frameSS.force_data,
+ frameSS.header.force_node,
+ 1),
+ frameSS.header.force_node, 1);
+ print_frame_short("SS sense frame =",
+ array1dTo2d_short(
+ frameSS.sense_data,
+ frameSS.header.sense_node,
+ frameSS.header.sense_node),
+ 1,
+ frameSS.header.sense_node);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_SET_OPERATING_FREQ:
+ /* need to pass: freq3 freq2 freq1 freq0 */
+ if (numberParam == 5) {
+ res = fts_disableInterrupt();
+ if (res >= OK) {
+ pr_info("Setting Scan Freq...\n");
+ u8ToU32_be(&cmd[1], &fileSize);
+ /* fileSize is used just as container
+ * variable, sorry for the name! */
+
+ res = setActiveScanFrequency(fileSize);
+ if (res < OK)
+ pr_err("Error while setting the scan frequency... ERROR %08X\n",
+ res);
+ else {
+ /* setActiveScan Frequency leave
+ * the chip in reset state but
+ * with the new scan freq set */
+ /* need to enable the scan mode
+ * and re-enable the interrupts
+ * */
+ res |= setScanMode(
+ SCAN_MODE_LOCKED,
+ LOCKED_ACTIVE);
+ /* this is a choice to force
+ * the IC to use the freq set */
+ res |= fts_enableInterrupt();
+ pr_info("Setting Scan Freq... res = %08X\n",
+ res);
+ }
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+ break;
+
+ case CMD_TP_SENS_MODE:
+ /* need to pass: enter (optional)saveGain */
+ if (numberParam >= 2) {
+ if (numberParam == 2)
+ cmd[2] = 0; /* by default never save
+ * the gain (used only
+ * when exit) */
+
+ res = tp_sensitivity_mode(cmd[1], cmd[2]);
+ if (res < OK)
+ pr_err("Error while setting TP Sens mode... ERROR %08X\n",
+ res);
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+
+ case CMD_TP_SENS_SET_SCAN_MODE:
+ /* need to pass: scan_type, enableGains */
+ if (numberParam == 3) {
+ res = tp_sensitivity_set_scan_mode(cmd[1],
+ cmd[2]);
+ /* this force the IC to lock in a scan mode */
+ if (res < OK)
+ pr_err("Error while setting TP Sens scan mode... ERROR %08X\n",
+ res);
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+ break;
+
+ case CMD_TP_SENS_PRECAL_SS:
+ /* need to pass: target1 target0 percentage(optional) */
+ if (numberParam >= 3) {
+ if (numberParam > 3)
+ temp = cmd[3];
+ else
+ temp = SENS_TEST_PERC_TARGET_PRECAL;
+
+ pr_info("Setting target = %d and percentage = %d\n",
+ (cmd[1] << 8 | cmd[2]), temp);
+
+ res = tp_sensitivity_test_pre_cal_ss(&frameSS,
+ (cmd[1] << 8 |
+ cmd[2]),
+ temp);
+ if (res < OK)
+ pr_err("Error while setting the scan frequency... ERROR %08X\n",
+ res);
+
+ if ((frameSS.force_data != NULL) &&
+ (frameSS.sense_data != NULL)) {
+ size += ((frameSS.header.force_node +
+ frameSS.header.sense_node) *
+ sizeof(short) + 2);
+ /*make error code positive to print the
+ * frame*/
+ res &= (~0x80000000);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_TP_SENS_PRECAL_MS:
+ /* need to pass: target1 target0 calibrate
+ * percentage(optional) */
+ if (numberParam >= 4) {
+ if (numberParam > 4)
+ temp = cmd[4];
+ else
+ temp = SENS_TEST_PERC_TARGET_PRECAL;
+
+ pr_info("Setting target = %d and percentage = %d\n",
+ (cmd[1] << 8 | cmd[2]), temp);
+
+ res = tp_sensitivity_test_pre_cal_ms(&frameMS,
+ (cmd[1] << 8 |
+ cmd[2]),
+ temp);
+ if (res < OK)
+ pr_err("Error during TP Sensitivity Precal ... ERROR %08X\n",
+ res);
+
+ if (cmd[3] != 0) {
+ pr_info("Computing gains with target = %d and saveGain = %d\n",
+ (cmd[1] << 8 | cmd[2]), cmd[3]);
+ temp = tp_sensitivity_compute_gains(
+ &frameMS, (cmd[1] << 8 |
+ cmd[2]),
+ cmd[3]);
+ if (temp < OK)
+ pr_err("Error during TP Sensitivity Calibration... ERROR %08X\n",
+ temp);
+ res |= temp;
+ }
+
+ if (frameMS.node_data != NULL) {
+ size += (frameMS.node_data_size *
+ sizeof(short) + 2);
+ /*make error code positive to print the
+ * frame*/
+ res &= (~0x80000000);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_TP_SENS_POSTCAL_MS:
+ /* need to pass: target1 target0 executeTest
+ * percentage(optional) */
+ if (numberParam >= 4) {
+ if (cmd[3] != 0) {
+ if (numberParam > 4)
+ temp = cmd[4];
+ else
+ temp =
+ SENS_TEST_PERC_TARGET_POSTCAL;
+ } else
+ temp = -1;
+
+ pr_info("Setting target = %d and percentage = %d\n",
+ (cmd[1] << 8 | cmd[2]), temp);
+
+ res = tp_sensitivity_test_post_cal_ms(&frameMS,
+ &deltas,
+ (cmd[1] << 8 |
+ cmd[2]),
+ temp,
+ &meanNorm,
+ &meanEdge);
+ if (res < OK)
+ pr_err("Error during TP Sensitivity Post Cal ... ERROR %08X\n",
+ res);
+
+ /* processing for a proper printing on the shell
+ * */
+ if ((frameMS.node_data != NULL) &&
+ (deltas.node_data != NULL)) {
+ size += ((frameMS.node_data_size +
+ deltas.node_data_size) *
+ sizeof(short) +
+ 2 + 8);/* +2 force and
+ * sense len, +8
+ * mean_normal/edge
+ * */
+ /*make error code positive to print the
+ * frame*/
+ res &= (~0x80000000);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+
+ case CMD_TP_SENS_STD:
+ /* need to pass: numFrames */
+ if (numberParam >= 2) {
+ res = tp_sensitivity_test_std_ms(cmd[1],
+ &frameMS);
+ if (res < OK)
+ pr_err("Error during TP Sensitivity STD... ERROR %08X\n",
+ res);
+
+ /* processing for a proper printing on the shell
+ * */
+ if (frameMS.node_data != NULL) {
+ size += ((frameMS.node_data_size) *
+ sizeof(short) + 2);
+ /* +2 force and sense len */
+ /*make error code positive to print the
+ * frame*/
+ res &= (~0x80000000);
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ case CMD_FORCE_TOUCH_ACTIVE:
+ /* Single parameter indicates force touch state */
+ if (numberParam == 2) {
+ if (cmd[1] > 1) {
+ pr_err("Parameter should be 1 or 0\n");
+ res = ERROR_OP_NOT_ALLOW;
+ } else {
+ fts_set_bus_ref(info,
+ FTS_BUS_REF_FORCE_ACTIVE,
+ cmd[1]);
+ res = OK;
+ }
+ } else {
+ pr_err("Wrong number of parameters!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+ break;
+
+ default:
+ pr_err("COMMAND ID NOT VALID!!!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ break;
+ }
+
+ /* res2 = fts_enableInterrupt();
+ * the interrupt was disabled on purpose in this node because it
+ * can be used for testing procedure and between one step and
+ * another the interrupt wan to be kept disabled
+ * if (res2 < 0) {
+ * pr_err("stm_driver_test_show: ERROR %08X\n",
+ * (res2 | ERROR_ENABLE_INTER));
+ * }*/
+ } else {
+ pr_err("NO COMMAND SPECIFIED!!! do: 'echo [cmd_code] [args] > stm_fts_cmd' before looking for result!\n");
+ res = ERROR_OP_NOT_ALLOW;
+ }
+
+END: /* here start the reporting phase, assembling the data to send in the
+ * file node */
+ if (driver_test_buff != NULL) {
+ pr_info("Consecutive echo on the file node, free the buffer with the previous result\n");
+ kfree(driver_test_buff);
+ }
+
+ if (byte_call == 0) {
+ size *= 2;
+ size += 2; /* add \n and \0 (terminator char) */
+ } else {
+ if (bin_output != 1) {
+ size *= 2; /* need to code each byte as HEX string */
+ size -= 1; /* start byte is just one, the extra
+ * byte of end byte taken by \n */
+ } else
+ size += 1; /* add \n */
+ }
+
+ pr_info("Size = %d\n", size);
+ driver_test_buff = (u8 *)kzalloc(size, GFP_KERNEL);
+ pr_info("Finish to allocate memory!\n");
+ if (driver_test_buff == NULL) {
+ pr_err("Unable to allocate driver_test_buff! ERROR %08X\n",
+ ERROR_ALLOC);
+ goto ERROR;
+ }
+
+ if (byte_call == 0) {
+ index = 0;
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "{ ");
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%08X", res);
+ if (res >= OK) {
+ /*all the other cases are already fine printing only the
+ * res.*/
+ switch (funcToTest[0]) {
+ case CMD_VERSION:
+ case CMD_READ:
+ case CMD_WRITEREAD:
+ case CMD_WRITETHENWRITEREAD:
+ case CMD_WRITEREADU8UX:
+ case CMD_WRITEU8UXTHENWRITEREADU8UX:
+ case CMD_READCONFIG:
+ case CMD_POLLFOREVENT:
+ /* pr_err("Data = "); */
+ if (mess.dummy == 1)
+ j = 1;
+ else
+ j = 0;
+ for (; j < byteToRead + mess.dummy; j++) {
+ /* pr_err("%02X ", readData[j]); */
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X", readData[j]);
+ /* this approach is much more faster */
+ }
+ /* pr_err("\n"); */
+ break;
+ case CMD_GETFWFILE:
+ case CMD_GETLIMITSFILE:
+ pr_info("Start To parse!\n");
+ for (j = 0; j < fileSize; j++) {
+ /* pr_err("%02X ", readData[j]); */
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X", readData[j]);
+ }
+ pr_info("Finish to parse!\n");
+ break;
+ case CMD_GETFORCELEN:
+ case CMD_GETSENSELEN:
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)temp);
+ break;
+ case CMD_GETMSFRAME:
+ case CMD_TP_SENS_PRECAL_MS:
+ case CMD_TP_SENS_POSTCAL_MS:
+ case CMD_TP_SENS_STD:
+
+ if (res != OK)
+ driver_test_buff[2] = '8';
+ /* convert back error code to negative */
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)frameMS.header.force_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)frameMS.header.sense_node);
+
+ for (j = 0; j < frameMS.node_data_size; j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (frameMS.node_data[j] & 0xFF00) >> 8,
+ frameMS.node_data[j] & 0xFF);
+ }
+
+ kfree(frameMS.node_data);
+
+ if (funcToTest[0] == CMD_TP_SENS_POSTCAL_MS) {
+ /* print also mean and deltas */
+ for (j = 0; j < deltas.node_data_size;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (deltas.node_data[j] &
+ 0xFF00) >> 8,
+ deltas.node_data[j] &
+ 0xFF);
+ }
+ kfree(deltas.node_data);
+
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%08X", meanNorm);
+
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%08X", meanEdge);
+ }
+ break;
+ case CMD_GETSSFRAME:
+ case CMD_TP_SENS_PRECAL_SS:
+ if (res != OK)
+ driver_test_buff[2] = '8';
+ /* convert back error code to negative */
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)frameSS.header.force_node);
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)frameSS.header.sense_node);
+ /* Copying self raw data Force */
+ for (j = 0; j < frameSS.header.force_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (frameSS.force_data[j] & 0xFF00) >> 8,
+ frameSS.force_data[j] & 0xFF);
+ }
+
+ /* Copying self raw data Sense */
+ for (j = 0; j < frameSS.header.sense_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (frameSS.sense_data[j] & 0xFF00) >> 8,
+ frameSS.sense_data[j] & 0xFF);
+ }
+
+ kfree(frameSS.force_data);
+ kfree(frameSS.sense_data);
+ break;
+
+ case CMD_READMSCOMPDATA:
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)compData.header.type);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)compData.header.force_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)compData.header.sense_node);
+
+ /* Cpying CX1 value */
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ compData.cx1 & 0xFF);
+
+ /* Copying CX2 values */
+ for (j = 0; j < compData.node_data_size; j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ compData.node_data[j] & 0xFF);
+ }
+
+ kfree(compData.node_data);
+ break;
+
+ case CMD_READSSCOMPDATA:
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)comData.header.type);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ comData.header.force_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ comData.header.sense_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ comData.f_ix1 & 0xFF);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ comData.s_ix1 & 0xFF);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ comData.f_cx1 & 0xFF);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ comData.s_cx1 & 0xFF);
+
+ /* Copying IX2 Force */
+ for (j = 0; j < comData.header.force_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ comData.ix2_fm[j] & 0xFF);
+ }
+
+ /* Copying IX2 Sense */
+ for (j = 0; j < comData.header.sense_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ comData.ix2_sn[j] & 0xFF);
+ }
+
+ /* Copying CX2 Force */
+ for (j = 0; j < comData.header.force_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ comData.cx2_fm[j] & 0xFF);
+ }
+
+ /* Copying CX2 Sense */
+ for (j = 0; j < comData.header.sense_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ comData.cx2_sn[j] & 0xFF);
+ }
+
+ kfree(comData.ix2_fm);
+ kfree(comData.ix2_sn);
+ kfree(comData.cx2_fm);
+ kfree(comData.cx2_sn);
+ break;
+
+
+
+ case CMD_READTOTMSCOMPDATA:
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)totCompData.header.type);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)totCompData.header.force_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)totCompData.header.sense_node);
+
+ /* Copying TOT CX values */
+ for (j = 0; j < totCompData.node_data_size;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (totCompData.node_data[j] &
+ 0xFF00) >> 8,
+ totCompData.node_data[j] &
+ 0xFF);
+ }
+
+ kfree(totCompData.node_data);
+ break;
+
+ case CMD_READTOTSSCOMPDATA:
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)totComData.header.type);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ totComData.header.force_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ totComData.header.sense_node);
+
+ /* Copying TOT IX Force */
+ for (j = 0; j < totComData.header.force_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (totComData.ix_fm[j] &
+ 0xFF00) >> 8,
+ totComData.ix_fm[j] &
+ 0xFF);
+ }
+
+ /* Copying TOT IX Sense */
+ for (j = 0; j < totComData.header.sense_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (totComData.ix_sn[j] &
+ 0xFF00) >> 8,
+ totComData.ix_sn[j] &
+ 0xFF);
+ }
+
+ /* Copying TOT CX Force */
+ for (j = 0; j < totComData.header.force_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (totComData.cx_fm[j] &
+ 0xFF00) >> 8,
+ totComData.cx_fm[j] &
+ 0xFF);
+ }
+
+ /* Copying CX2 Sense */
+ for (j = 0; j < totComData.header.sense_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (totComData.cx_sn[j] &
+ 0xFF00) >> 8,
+ totComData.cx_sn[j] &
+ 0xFF);
+ }
+
+ kfree(totComData.ix_fm);
+ kfree(totComData.ix_sn);
+ kfree(totComData.cx_fm);
+ kfree(totComData.cx_sn);
+ break;
+
+ case CMD_READSENSCOEFF:
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)msCoeff.header.force_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)msCoeff.header.sense_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)ssCoeff.header.force_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)ssCoeff.header.sense_node);
+
+ /* Copying MS Coefficients */
+ for (j = 0; j < msCoeff.node_data_size; j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ msCoeff.ms_coeff[j] & 0xFF);
+ }
+
+ /* Copying SS force Coefficients */
+ for (j = 0; j < ssCoeff.header.force_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ ssCoeff.ss_force_coeff[j] &
+ 0xFF);
+ }
+
+ /* Copying SS sense Coefficients */
+ for (j = 0; j < ssCoeff.header.sense_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ ssCoeff.ss_sense_coeff[j] &
+ 0xFF);
+ }
+
+ kfree(msCoeff.ms_coeff);
+ kfree(ssCoeff.ss_force_coeff);
+ kfree(ssCoeff.ss_sense_coeff);
+ break;
+
+ case CMD_GETFWVER:
+ for (j = 0; j < EXTERNAL_RELEASE_INFO_SIZE;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ systemInfo.u8_releaseInfo[j]);
+ }
+ break;
+
+ case CMD_READCOMPDATAHEAD:
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ dataHead.type);
+ break;
+
+ case CMD_READ_SYNC_FRAME:
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)frameMS.header.force_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)frameMS.header.sense_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)frameSS.header.force_node);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%02X",
+ (u8)frameSS.header.sense_node);
+
+ /* Copying mutual data */
+ for (j = 0; j < frameMS.node_data_size; j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (frameMS.node_data[j] &
+ 0xFF00) >> 8,
+ frameMS.node_data[j] &
+ 0xFF);
+ }
+
+ /* Copying self data Force */
+ for (j = 0; j < frameSS.header.force_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (frameSS.force_data[j] &
+ 0xFF00) >> 8,
+ frameSS.force_data[j] &
+ 0xFF);
+ }
+
+
+ /* Copying self data Sense */
+ for (j = 0; j < frameSS.header.sense_node;
+ j++) {
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (frameSS.sense_data[j] &
+ 0xFF00) >> 8,
+ frameSS.sense_data[j] &
+ 0xFF);
+ }
+
+ kfree(frameMS.node_data);
+ kfree(frameSS.force_data);
+ kfree(frameSS.sense_data);
+ break;
+
+
+ default:
+ break;
+ }
+ }
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, " }\n");
+ limit = size - 1;/* avoid to print \0 in the shell */
+ printed = 0;
+ } else {
+ /* start byte */
+ driver_test_buff[index++] = MESSAGE_START_BYTE;
+ if (bin_output == 1) {
+ /* msg_size */
+ driver_test_buff[index++] = (size & 0xFF00) >> 8;
+ driver_test_buff[index++] = (size & 0x00FF);
+ /* counter id */
+ driver_test_buff[index++] =
+ (mess.counter & 0xFF00) >> 8;
+ driver_test_buff[index++] = (mess.counter & 0x00FF);
+ /* action */
+ driver_test_buff[index++] = (mess.action & 0xFF00) >> 8;
+ driver_test_buff[index++] = (mess.action & 0x00FF);
+ /* error */
+ driver_test_buff[index++] = (res & 0xFF00) >> 8;
+ driver_test_buff[index++] = (res & 0x00FF);
+ } else {
+ if (funcToTest[0] == CMD_GETLIMITSFILE_BYTE ||
+ funcToTest[0] == CMD_GETFWFILE_BYTE)
+ index += scnprintf(&driver_test_buff[index],
+ size - index,
+ "%02X%02X",
+ (((fileSize + 3) / 4) & 0xFF00) >> 8,
+ ((fileSize + 3) / 4) & 0x00FF);
+ else
+ index += scnprintf(&driver_test_buff[index],
+ size - index,
+ "%02X%02X", (size & 0xFF00) >> 8,
+ size & 0xFF);
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%04X",
+ (u16)mess.counter);
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%04X",
+ (u16)mess.action);
+ index += scnprintf(&driver_test_buff[index],
+ size - index,
+ "%02X%02X", (res & 0xFF00) >> 8,
+ res & 0xFF);
+ }
+
+ switch (funcToTest[0]) {
+ case CMD_VERSION_BYTE:
+ case CMD_READ_BYTE:
+ case CMD_WRITEREAD_BYTE:
+ case CMD_WRITETHENWRITEREAD_BYTE:
+ case CMD_WRITEREADU8UX_BYTE:
+ case CMD_WRITEU8UXTHENWRITEREADU8UX_BYTE:
+ if (bin_output == 1) {
+ if (mess.dummy == 1)
+ memcpy(&driver_test_buff[index],
+ &readData[1], byteToRead);
+ else
+ memcpy(&driver_test_buff[index],
+ readData, byteToRead);
+ index += byteToRead;
+ } else {
+ j = mess.dummy;
+ for (; j < byteToRead + mess.dummy; j++)
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ (u8)readData[j]);
+ }
+ break;
+
+ case CMD_GETLIMITSFILE_BYTE:
+ case CMD_GETFWFILE_BYTE:
+ if (bin_output == 1) {
+ /* override the msg_size with dimension in words
+ * */
+ driver_test_buff[1] = (
+ ((fileSize + 3) / 4) & 0xFF00) >> 8;
+ driver_test_buff[2] = (
+ ((fileSize + 3) / 4) & 0x00FF);
+
+ if (readData != NULL)
+ memcpy(&driver_test_buff[index],
+ readData, fileSize);
+ else
+ pr_err("readData = NULL... returning junk data!");
+ index += addr; /* in this case the byte to read
+ * are stored in addr because it
+ * is a u64 end byte need to be
+ * inserted at the end of the
+ * padded memory */
+ } else {
+ /* snprintf(&driver_test_buff[1], 3, "%02X",
+ * (((fileSize + 3) / 4)&0xFF00) >> 8); */
+ /* snprintf(&driver_test_buff[3], 3, "%02X",
+ * ((fileSize + 3) / 4)&0x00FF); */
+ for (j = 0; j < fileSize; j++)
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X",
+ (u8)readData[j]);
+ for (; j < addr; j++)
+ index += scnprintf(
+ &driver_test_buff[index],
+ size - index,
+ "%02X", 0); /* pad memory
+ * with 0x00 */
+ }
+ break;
+ default:
+ break;
+ }
+
+ index += scnprintf(&driver_test_buff[index],
+ size - index, "%c\n", MESSAGE_END_BYTE);
+ /*for(j=0; j<size; j++){
+ * pr_err("%c", driver_test_buff[j]);
+ * }*/
+ limit = size;
+ printed = 0;
+ }
+ERROR:
+ numberParam = 0;/* need to reset the number of parameters in order to
+ * wait the next command, comment if you want to repeat
+ * the last command sent just doing a cat */
+
+ /* pr_err(0,"numberParameters = %d\n", numberParam); */
+
+ kfree(readData);
+ kfree(cmd);
+ kfree(funcToTest);
+ kfree(pbuf);
+
+ fts_set_bus_ref(info, FTS_BUS_REF_SYSFS, false);
+
+ return count;
+}
+
+/** @}*/
+
+/**
+ * file_operations struct which define the functions for the canonical
+ * operation on a device file node (open. read, write etc.)
+ */
+static struct file_operations fts_driver_test_ops = {
+ .open = fts_open,
+ .read = seq_read,
+ .write = fts_driver_test_write,
+ .llseek = seq_lseek,
+ .release = seq_release
+};
+
+/*****************************************************************************/
+
+/**
+ * This function is called in the probe to initialize and create the directory
+ * proc/fts and the driver test file node DRIVER_TEST_FILE_NODE into the /proc
+ * file system
+ * @return OK if success or an error code which specify the type of error
+ */
+int fts_proc_init(void)
+{
+ struct proc_dir_entry *entry;
+
+ int retval = 0;
+
+
+ fts_dir = proc_mkdir_data("fts", 0777, NULL, NULL);
+ if (fts_dir == NULL) { /* directory creation failed */
+ retval = -ENOMEM;
+ goto out;
+ }
+
+ entry = proc_create(DRIVER_TEST_FILE_NODE, 0777, fts_dir,
+ &fts_driver_test_ops);
+
+ if (entry)
+ pr_info("%s: proc entry CREATED!\n", __func__);
+ else {
+ pr_err("%s: error creating proc entry!\n", __func__);
+ retval = -ENOMEM;
+ goto badfile;
+ }
+ return OK;
+badfile:
+ remove_proc_entry("fts", NULL);
+out:
+ return retval;
+}
+
+/**
+ * Delete and Clean from the file system, all the references to the driver test
+ * file node
+ * @return OK
+ */
+int fts_proc_remove(void)
+{
+ remove_proc_entry(DRIVER_TEST_FILE_NODE, fts_dir);
+ remove_proc_entry("fts", NULL);
+ return OK;
+}
diff --git a/drivers/input/touchscreen/touch_bus_negotiator.c b/drivers/input/touchscreen/touch_bus_negotiator.c
new file mode 100644
index 0000000..9471220
--- /dev/null
+++ b/drivers/input/touchscreen/touch_bus_negotiator.c
@@ -0,0 +1,276 @@
+/* drivers/input/touchscreen/touch_bus_negotiator.c
+ *
+ * Touch Bus Negotiator for Google Pixel devices.
+ *
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/printk.h>
+#include <linux/slab.h>
+#include "touch_bus_negotiator.h"
+
+static void tbn_on_event(struct work_struct *work)
+{
+ int err = 0;
+ struct tbn_context *tbn =
+ container_of(work, struct tbn_context, on_event);
+
+ do {
+ dev_dbg(tbn->dev, "received message from server\n");
+ } while ((err = qmi_recv_msg(tbn->qmi_handle)) == 0);
+
+ if (err != -ENOMSG)
+ dev_err(tbn->dev, "qmi_recv_msg() failed with %d\n", err);
+}
+
+static void tbn_process_event(
+ struct qmi_handle *handle,
+ enum qmi_event_type event,
+ void *context)
+{
+ struct tbn_context *tbn = (struct tbn_context *)context;
+
+ switch (event) {
+ case QMI_RECV_MSG:
+ dev_dbg(tbn->dev, "received event from QMI server\n");
+ queue_work(tbn->event_processor, &tbn->on_event);
+ break;
+ default:
+ dev_err(tbn->dev, "unhandled QMI event: %d\n", (int)event);
+ }
+}
+
+static int tbn_on_server_event(
+ struct notifier_block *nb,
+ unsigned long event,
+ void *data)
+{
+ struct tbn_context *tbn =
+ container_of(nb, struct tbn_context, server_event_notifier);
+
+ switch (event) {
+ case QMI_SERVER_ARRIVE:
+ schedule_work(&tbn->on_server_arrive);
+ break;
+ case QMI_SERVER_EXIT:
+ schedule_work(&tbn->on_server_exit);
+ break;
+ default:
+ dev_err(tbn->dev, "unhandled server event: %d\n", (int)event);
+ }
+
+ return 0;
+}
+
+int tbn_request_bus(struct tbn_context *tbn)
+{
+ int err = 0;
+ struct msg_desc req_desc = { 0 }, rsp_desc = { 0 };
+ struct tbn_kernel_request_bus_v01 req;
+ struct tbn_ssc_release_bus_v01 rsp;
+
+ dev_info(tbn->dev, "kernel requesting bus access from SLPI\n");
+
+ if (!tbn || !tbn->qmi_handle)
+ return -EINVAL;
+
+ req_desc.max_msg_len = TBN_KERNEL_REQUEST_BUS_V01_MAX_MSG_LEN;
+ req_desc.msg_id = QMI_TBN_KERNEL_REQUEST_BUS_V01;
+ req_desc.ei_array = tbn_kernel_request_bus_v01_ei;
+
+ rsp_desc.max_msg_len = TBN_SSC_RELEASE_BUS_V01_MAX_MSG_LEN;
+ rsp_desc.msg_id = QMI_TBN_SSC_RELEASE_BUS_V01;
+ rsp_desc.ei_array = tbn_ssc_release_bus_v01_ei;
+
+ mutex_lock(&tbn->service_lock);
+ err = qmi_send_req_wait(
+ tbn->qmi_handle,
+ &req_desc, &req, sizeof(req),
+ &rsp_desc, &rsp, sizeof(rsp),
+ TBN_REQUEST_BUS_TIMEOUT_MS);
+ mutex_unlock(&tbn->service_lock);
+ if (err) {
+ dev_err(tbn->dev, "qmi_send_req_wait() failed with: %d\n", err);
+ return err;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(tbn_request_bus);
+
+int tbn_release_bus(struct tbn_context *tbn)
+{
+ int err = 0;
+ struct msg_desc req_desc = { 0 }, rsp_desc = { 0 };
+ struct tbn_kernel_release_bus_v01 req;
+ struct tbn_ssc_acquire_bus_v01 rsp;
+
+ dev_info(tbn->dev, "kernel releasing bus access from SLPI\n");
+
+ if (!tbn || !tbn->qmi_handle)
+ return -EINVAL;
+
+ req_desc.max_msg_len = TBN_KERNEL_RELEASE_BUS_V01_MAX_MSG_LEN;
+ req_desc.msg_id = QMI_TBN_KERNEL_RELEASE_BUS_V01;
+ req_desc.ei_array = tbn_kernel_release_bus_v01_ei;
+
+ rsp_desc.max_msg_len = TBN_SSC_ACQUIRE_BUS_V01_MAX_MSG_LEN;
+ rsp_desc.msg_id = QMI_TBN_SSC_ACQUIRE_BUS_V01;
+ rsp_desc.ei_array = tbn_ssc_acquire_bus_v01_ei;
+
+ mutex_lock(&tbn->service_lock);
+ err = qmi_send_req_wait(
+ tbn->qmi_handle,
+ &req_desc, &req, sizeof(req),
+ &rsp_desc, &rsp, sizeof(rsp),
+ TBN_RELEASE_BUS_TIMEOUT_MS);
+ mutex_unlock(&tbn->service_lock);
+ if (err) {
+ dev_err(tbn->dev, "qmi_send_req_wait() failed with: %d\n", err);
+ return err;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(tbn_release_bus);
+
+static void tbn_connect_to_remote_server(struct tbn_context *tbn)
+{
+ int err = 0;
+
+ dev_dbg(tbn->dev, "connecting to remote tbn server on SLPI\n");
+
+ tbn->qmi_handle = qmi_handle_create(tbn_process_event, tbn);
+ if (!tbn->qmi_handle) {
+ dev_err(tbn->dev, "failed to create qmi handle\n");
+ return;
+ }
+
+ err = qmi_connect_to_service(
+ tbn->qmi_handle,
+ TBN_SERVICE_ID_V01,
+ TBN_SERVICE_VERS_V01,
+ 0 /* service instance */);
+ if (err) {
+ dev_err(tbn->dev, "failed to connect to qmi service, "
+ "err = %d\n", err);
+ qmi_handle_destroy(tbn->qmi_handle);
+ tbn->qmi_handle = NULL;
+ }
+
+ dev_dbg(tbn->dev, "connected to remote tbn server on SLPI\n");
+}
+
+static void tbn_on_server_arrive(struct work_struct *work)
+{
+ struct tbn_context *tbn =
+ container_of(work, struct tbn_context, on_server_arrive);
+
+ dev_info(tbn->dev, "remote tbn server online, connecting\n");
+
+ mutex_lock(&tbn->service_lock);
+ tbn_connect_to_remote_server(tbn);
+ mutex_unlock(&tbn->service_lock);
+}
+
+static void tbn_on_server_exit(struct work_struct *work)
+{
+ struct tbn_context *tbn =
+ container_of(work, struct tbn_context, on_server_exit);
+
+ dev_info(tbn->dev, "remote tbn server offline, disconnecting\n");
+
+ mutex_lock(&tbn->service_lock);
+ qmi_handle_destroy(tbn->qmi_handle);
+ tbn->qmi_handle = NULL;
+ mutex_unlock(&tbn->service_lock);
+}
+
+struct tbn_context *tbn_init(struct device *dev)
+{
+ int err = 0;
+ struct tbn_context *tbn = NULL;
+
+ tbn = kzalloc(sizeof(struct tbn_context), GFP_KERNEL);
+ if (!tbn) {
+ dev_err(dev, "failed to allocate tbn context\n");
+ goto fail_allocate_tbn_context;
+ }
+
+ tbn->dev = dev;
+ tbn->server_event_notifier.notifier_call = tbn_on_server_event;
+ mutex_init(&tbn->service_lock);
+ INIT_WORK(&tbn->on_server_arrive, tbn_on_server_arrive);
+ INIT_WORK(&tbn->on_server_exit, tbn_on_server_exit);
+ INIT_WORK(&tbn->on_event, tbn_on_event);
+
+ tbn->event_processor =
+ create_singlethread_workqueue("tbn_event_processor");
+ if (!tbn->event_processor) {
+ dev_err(tbn->dev, "failed to create event processing thread\n");
+ goto fail_create_workqueue;
+ }
+
+ err = qmi_svc_event_notifier_register(
+ TBN_SERVICE_ID_V01,
+ TBN_SERVICE_VERS_V01,
+ 0 /* service instance */,
+ &tbn->server_event_notifier);
+ if (err) {
+ dev_err(tbn->dev, "failed to register server event notifier\n");
+ goto fail_register_server_event_notifier;
+ }
+
+ dev_info(tbn->dev, "bus negotiator initialized: %p\n", tbn);
+
+ return tbn;
+
+fail_register_server_event_notifier:
+ destroy_workqueue(tbn->event_processor);
+fail_create_workqueue:
+ kfree(tbn);
+fail_allocate_tbn_context:
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(tbn_init);
+
+void tbn_cleanup(struct tbn_context *tbn)
+{
+ dev_info(tbn->dev, "destructing bus negotiator: %p\n", tbn);
+
+ if (!tbn)
+ return;
+
+ qmi_svc_event_notifier_unregister(
+ TBN_SERVICE_ID_V01,
+ TBN_SERVICE_VERS_V01,
+ 0 /* service instance */,
+ &tbn->server_event_notifier);
+
+ mutex_lock(&tbn->service_lock);
+
+ if (tbn->qmi_handle)
+ qmi_handle_destroy(tbn->qmi_handle);
+
+ mutex_unlock(&tbn->service_lock);
+
+ if (tbn->event_processor)
+ destroy_workqueue(tbn->event_processor);
+
+ kfree(tbn);
+}
+EXPORT_SYMBOL_GPL(tbn_cleanup);
+
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("QMI based Touch Bus Negotiator");
diff --git a/drivers/input/touchscreen/touch_bus_negotiator.h b/drivers/input/touchscreen/touch_bus_negotiator.h
new file mode 100644
index 0000000..83e61a6
--- /dev/null
+++ b/drivers/input/touchscreen/touch_bus_negotiator.h
@@ -0,0 +1,47 @@
+/* drivers/input/touchscreen/touch_bus_negotiator.h
+ *
+ * Touch Bus Negotiator for Google Pixel devices.
+ *
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#ifndef TOUCHSCREEN_BUS_NEGOTIATOR_H
+#define TOUCHSCREEN_BUS_NEGOTIATOR_H
+
+#include <linux/workqueue.h>
+#include <linux/notifier.h>
+#include <linux/mutex.h>
+#include <linux/device.h>
+#include "qmi_tbn_v01.h"
+
+#define TBN_REQUEST_BUS_TIMEOUT_MS 500
+#define TBN_RELEASE_BUS_TIMEOUT_MS 500
+
+struct tbn_context {
+ struct device *dev;
+ struct qmi_handle *qmi_handle;
+ struct notifier_block server_event_notifier;
+ struct workqueue_struct *event_processor;
+ struct work_struct on_server_arrive;
+ struct work_struct on_server_exit;
+ struct work_struct on_event;
+ struct mutex service_lock;
+};
+
+struct tbn_context *tbn_init(struct device *dev);
+void tbn_cleanup(struct tbn_context *);
+int tbn_request_bus(struct tbn_context *);
+int tbn_release_bus(struct tbn_context *);
+
+
+#endif /* TOUCHSCREEN_BUS_NEGOTIATOR_H */
diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c
index 0c910a8..bba1b9f 100644
--- a/drivers/iommu/amd_iommu.c
+++ b/drivers/iommu/amd_iommu.c
@@ -288,7 +288,13 @@ static u16 get_alias(struct device *dev)
/* The callers make sure that get_device_id() does not fail here */
devid = get_device_id(dev);
+
+ /* For ACPI HID devices, we simply return the devid as such */
+ if (!dev_is_pci(dev))
+ return devid;
+
ivrs_alias = amd_iommu_alias_table[devid];
+
pci_for_each_dma_alias(pdev, __last_alias, &pci_alias);
if (ivrs_alias == pci_alias)
@@ -2452,9 +2458,9 @@ static void __unmap_single(struct dma_ops_domain *dma_dom,
}
if (amd_iommu_unmap_flush) {
- dma_ops_free_iova(dma_dom, dma_addr, pages);
domain_flush_tlb(&dma_dom->domain);
domain_flush_complete(&dma_dom->domain);
+ dma_ops_free_iova(dma_dom, dma_addr, pages);
} else {
queue_add(dma_dom, dma_addr, pages);
}
diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c
index 7f294f7..ff4be11 100644
--- a/drivers/iommu/arm-smmu-v3.c
+++ b/drivers/iommu/arm-smmu-v3.c
@@ -1233,6 +1233,7 @@ static irqreturn_t arm_smmu_priq_thread(int irq, void *dev)
/* Sync our overflow flag, as we believe we're up to speed */
q->cons = Q_OVF(q, q->prod) | Q_WRP(q, q->cons) | Q_IDX(q, q->cons);
+ writel(q->cons, q->cons_reg);
return IRQ_HANDLED;
}
diff --git a/drivers/iommu/dmar.c b/drivers/iommu/dmar.c
index 8c53748..63110fb 100644
--- a/drivers/iommu/dmar.c
+++ b/drivers/iommu/dmar.c
@@ -1328,8 +1328,8 @@ void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
qi_submit_sync(&desc, iommu);
}
-void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 qdep,
- u64 addr, unsigned mask)
+void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid,
+ u16 qdep, u64 addr, unsigned mask)
{
struct qi_desc desc;
@@ -1344,7 +1344,7 @@ void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 qdep,
qdep = 0;
desc.low = QI_DEV_IOTLB_SID(sid) | QI_DEV_IOTLB_QDEP(qdep) |
- QI_DIOTLB_TYPE;
+ QI_DIOTLB_TYPE | QI_DEV_IOTLB_PFSID(pfsid);
qi_submit_sync(&desc, iommu);
}
diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index 88bbc8c..86e3496 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -421,6 +421,7 @@ struct device_domain_info {
struct list_head global; /* link to global list */
u8 bus; /* PCI bus number */
u8 devfn; /* PCI devfn number */
+ u16 pfsid; /* SRIOV physical function source ID */
u8 pasid_supported:3;
u8 pasid_enabled:1;
u8 pri_supported:1;
@@ -1511,6 +1512,20 @@ static void iommu_enable_dev_iotlb(struct device_domain_info *info)
return;
pdev = to_pci_dev(info->dev);
+ /* For IOMMU that supports device IOTLB throttling (DIT), we assign
+ * PFSID to the invalidation desc of a VF such that IOMMU HW can gauge
+ * queue depth at PF level. If DIT is not set, PFSID will be treated as
+ * reserved, which should be set to 0.
+ */
+ if (!ecap_dit(info->iommu->ecap))
+ info->pfsid = 0;
+ else {
+ struct pci_dev *pf_pdev;
+
+ /* pdev will be returned if device is not a vf */
+ pf_pdev = pci_physfn(pdev);
+ info->pfsid = PCI_DEVID(pf_pdev->bus->number, pf_pdev->devfn);
+ }
#ifdef CONFIG_INTEL_IOMMU_SVM
/* The PCIe spec, in its wisdom, declares that the behaviour of
@@ -1576,7 +1591,8 @@ static void iommu_flush_dev_iotlb(struct dmar_domain *domain,
sid = info->bus << 8 | info->devfn;
qdep = info->ats_qdep;
- qi_flush_dev_iotlb(info->iommu, sid, qdep, addr, mask);
+ qi_flush_dev_iotlb(info->iommu, sid, info->pfsid,
+ qdep, addr, mask);
}
spin_unlock_irqrestore(&device_domain_lock, flags);
}
@@ -1612,8 +1628,7 @@ static void iommu_flush_iotlb_psi(struct intel_iommu *iommu,
* flush. However, device IOTLB doesn't need to be flushed in this case.
*/
if (!cap_caching_mode(iommu->cap) || !map)
- iommu_flush_dev_iotlb(get_iommu_domain(iommu, did),
- addr, mask);
+ iommu_flush_dev_iotlb(domain, addr, mask);
}
static void iommu_disable_protect_mem_regions(struct intel_iommu *iommu)
@@ -2069,7 +2084,7 @@ static int domain_context_mapping_one(struct dmar_domain *domain,
* than default. Unnecessary for PT mode.
*/
if (translation != CONTEXT_TT_PASS_THROUGH) {
- for (agaw = domain->agaw; agaw != iommu->agaw; agaw--) {
+ for (agaw = domain->agaw; agaw > iommu->agaw; agaw--) {
ret = -ENOMEM;
pgd = phys_to_virt(dma_pte_addr(pgd));
if (!dma_pte_present(pgd))
@@ -2083,7 +2098,7 @@ static int domain_context_mapping_one(struct dmar_domain *domain,
translation = CONTEXT_TT_MULTI_LEVEL;
context_set_address_root(context, virt_to_phys(pgd));
- context_set_address_width(context, iommu->agaw);
+ context_set_address_width(context, agaw);
} else {
/*
* In pass through mode, AW must be programmed to
@@ -3039,7 +3054,7 @@ static int copy_context_table(struct intel_iommu *iommu,
}
if (old_ce)
- iounmap(old_ce);
+ memunmap(old_ce);
ret = 0;
if (devfn < 0x80)
diff --git a/drivers/iommu/intel-svm.c b/drivers/iommu/intel-svm.c
index f846f01..7dc2f8d 100644
--- a/drivers/iommu/intel-svm.c
+++ b/drivers/iommu/intel-svm.c
@@ -558,7 +558,7 @@ static irqreturn_t prq_event_thread(int irq, void *d)
pr_err("%s: Page request without PASID: %08llx %08llx\n",
iommu->name, ((unsigned long long *)req)[0],
((unsigned long long *)req)[1]);
- goto bad_req;
+ goto no_pasid;
}
if (!svm || svm->pasid != req->pasid) {
diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c
index ace331d..3d2e9ca 100644
--- a/drivers/iommu/ipmmu-vmsa.c
+++ b/drivers/iommu/ipmmu-vmsa.c
@@ -44,7 +44,7 @@ struct ipmmu_vmsa_domain {
struct io_pgtable_ops *iop;
unsigned int context_id;
- spinlock_t lock; /* Protects mappings */
+ struct mutex mutex; /* Protects mappings */
};
struct ipmmu_vmsa_archdata {
@@ -372,6 +372,9 @@ static int ipmmu_domain_init_context(struct ipmmu_vmsa_domain *domain)
static void ipmmu_domain_destroy_context(struct ipmmu_vmsa_domain *domain)
{
+ if (!domain->mmu)
+ return;
+
/*
* Disable the context. Flush the TLB as required when modifying the
* context registers.
@@ -464,7 +467,7 @@ static struct iommu_domain *ipmmu_domain_alloc(unsigned type)
if (!domain)
return NULL;
- spin_lock_init(&domain->lock);
+ mutex_init(&domain->mutex);
return &domain->io_domain;
}
@@ -488,7 +491,6 @@ static int ipmmu_attach_device(struct iommu_domain *io_domain,
struct ipmmu_vmsa_archdata *archdata = dev->archdata.iommu;
struct ipmmu_vmsa_device *mmu = archdata->mmu;
struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain);
- unsigned long flags;
unsigned int i;
int ret = 0;
@@ -497,7 +499,7 @@ static int ipmmu_attach_device(struct iommu_domain *io_domain,
return -ENXIO;
}
- spin_lock_irqsave(&domain->lock, flags);
+ mutex_lock(&domain->mutex);
if (!domain->mmu) {
/* The domain hasn't been used yet, initialize it. */
@@ -513,7 +515,7 @@ static int ipmmu_attach_device(struct iommu_domain *io_domain,
ret = -EINVAL;
}
- spin_unlock_irqrestore(&domain->lock, flags);
+ mutex_unlock(&domain->mutex);
if (ret < 0)
return ret;
diff --git a/drivers/irqchip/irq-bcm7038-l1.c b/drivers/irqchip/irq-bcm7038-l1.c
index c2662a1..6e24fac 100644
--- a/drivers/irqchip/irq-bcm7038-l1.c
+++ b/drivers/irqchip/irq-bcm7038-l1.c
@@ -215,6 +215,7 @@ static int bcm7038_l1_set_affinity(struct irq_data *d,
return 0;
}
+#ifdef CONFIG_SMP
static void bcm7038_l1_cpu_offline(struct irq_data *d)
{
struct cpumask *mask = irq_data_get_affinity_mask(d);
@@ -239,6 +240,7 @@ static void bcm7038_l1_cpu_offline(struct irq_data *d)
}
irq_set_affinity_locked(d, &new_affinity, false);
}
+#endif
static int __init bcm7038_l1_init_one(struct device_node *dn,
unsigned int idx,
@@ -291,7 +293,9 @@ static struct irq_chip bcm7038_l1_irq_chip = {
.irq_mask = bcm7038_l1_mask,
.irq_unmask = bcm7038_l1_unmask,
.irq_set_affinity = bcm7038_l1_set_affinity,
+#ifdef CONFIG_SMP
.irq_cpu_offline = bcm7038_l1_cpu_offline,
+#endif
};
static int bcm7038_l1_map(struct irq_domain *d, unsigned int virq,
diff --git a/drivers/irqchip/irq-gic-v3-its-pci-msi.c b/drivers/irqchip/irq-gic-v3-its-pci-msi.c
index aee1c60..cc58b1b 100644
--- a/drivers/irqchip/irq-gic-v3-its-pci-msi.c
+++ b/drivers/irqchip/irq-gic-v3-its-pci-msi.c
@@ -133,6 +133,8 @@ static int __init its_pci_of_msi_init(void)
for (np = of_find_matching_node(NULL, its_device_id); np;
np = of_find_matching_node(np, its_device_id)) {
+ if (!of_device_is_available(np))
+ continue;
if (!of_property_read_bool(np, "msi-controller"))
continue;
diff --git a/drivers/irqchip/irq-gic-v3-its-platform-msi.c b/drivers/irqchip/irq-gic-v3-its-platform-msi.c
index 470b4aa..e4768fc 100644
--- a/drivers/irqchip/irq-gic-v3-its-platform-msi.c
+++ b/drivers/irqchip/irq-gic-v3-its-platform-msi.c
@@ -80,6 +80,8 @@ static int __init its_pmsi_init(void)
for (np = of_find_matching_node(NULL, its_device_id); np;
np = of_find_matching_node(np, its_device_id)) {
+ if (!of_device_is_available(np))
+ continue;
if (!of_property_read_bool(np, "msi-controller"))
continue;
diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index ac15e5d..558c758 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1807,6 +1807,8 @@ static int __init its_of_probe(struct device_node *node)
for (np = of_find_matching_node(node, its_device_id); np;
np = of_find_matching_node(np, its_device_id)) {
+ if (!of_device_is_available(np))
+ continue;
if (!of_property_read_bool(np, "msi-controller")) {
pr_warn("%s: no msi-controller property, ITS ignored\n",
np->full_name);
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 6993899..08fcdbf 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -213,11 +213,10 @@ static void gic_redist_wait_for_rwp(void)
}
#ifdef CONFIG_ARM64
-static DEFINE_STATIC_KEY_FALSE(is_cavium_thunderx);
static u64 __maybe_unused gic_read_iar(void)
{
- if (static_branch_unlikely(&is_cavium_thunderx))
+ if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_23154))
return gic_read_iar_cavium_thunderx();
else
return gic_read_iar_common();
@@ -1024,7 +1023,7 @@ static void gic_send_sgi(u64 cluster_id, u16 tlist, unsigned int irq)
MPIDR_TO_SGI_AFFINITY(cluster_id, 1) |
tlist << ICC_SGI1R_TARGET_LIST_SHIFT);
- pr_debug("CPU%d: ICC_SGI1R_EL1 %llx\n", smp_processor_id(), val);
+ pr_devel("CPU%d: ICC_SGI1R_EL1 %llx\n", smp_processor_id(), val);
gic_write_sgi1r(val);
}
@@ -1334,14 +1333,6 @@ static const struct irq_domain_ops partition_domain_ops = {
.select = gic_irq_domain_select,
};
-static void gicv3_enable_quirks(void)
-{
-#ifdef CONFIG_ARM64
- if (cpus_have_cap(ARM64_WORKAROUND_CAVIUM_23154))
- static_branch_enable(&is_cavium_thunderx);
-#endif
-}
-
static int __init gic_init_bases(void __iomem *dist_base,
struct redist_region *rdist_regs,
u32 nr_redist_regions,
@@ -1364,8 +1355,6 @@ static int __init gic_init_bases(void __iomem *dist_base,
gic_data.nr_redist_regions = nr_redist_regions;
gic_data.redist_stride = redist_stride;
- gicv3_enable_quirks();
-
/*
* Find out how many interrupts are supported.
* The GIC only supports up to 1020 interrupt sources (SGI+PPI+SPI)
diff --git a/drivers/irqchip/irq-gic.c b/drivers/irqchip/irq-gic.c
index 8f3aa54..679c33e 100644
--- a/drivers/irqchip/irq-gic.c
+++ b/drivers/irqchip/irq-gic.c
@@ -41,6 +41,7 @@
#include <linux/irqchip.h>
#include <linux/irqchip/chained_irq.h>
#include <linux/irqchip/arm-gic.h>
+#include <linux/wakeup_reason.h>
#include <linux/msm_rtb.h>
#include <asm/cputype.h>
@@ -391,12 +392,13 @@ static void __exception_irq_entry gic_handle_irq(struct pt_regs *regs)
} while (1);
}
-static void gic_handle_cascade_irq(struct irq_desc *desc)
+static bool gic_handle_cascade_irq(struct irq_desc *desc)
{
struct gic_chip_data *chip_data = irq_desc_get_handler_data(desc);
struct irq_chip *chip = irq_desc_get_chip(desc);
unsigned int cascade_irq, gic_irq;
unsigned long status;
+ int handled = false;
chained_irq_enter(chip, desc);
@@ -410,10 +412,11 @@ static void gic_handle_cascade_irq(struct irq_desc *desc)
if (unlikely(gic_irq < 32 || gic_irq > 1020))
handle_bad_irq(desc);
else
- generic_handle_irq(cascade_irq);
+ handled = generic_handle_irq(cascade_irq);
out:
chained_irq_exit(chip, desc);
+ return handled;
}
static struct irq_chip gic_chip = {
diff --git a/drivers/irqchip/irq-partition-percpu.c b/drivers/irqchip/irq-partition-percpu.c
index ccd72c2..4b19e09 100644
--- a/drivers/irqchip/irq-partition-percpu.c
+++ b/drivers/irqchip/irq-partition-percpu.c
@@ -121,12 +121,14 @@ static struct irq_chip partition_irq_chip = {
.irq_print_chip = partition_irq_print_chip,
};
-static void partition_handle_irq(struct irq_desc *desc)
+static bool partition_handle_irq(struct irq_desc *desc)
{
struct partition_desc *part = irq_desc_get_handler_data(desc);
struct irq_chip *chip = irq_desc_get_chip(desc);
int cpu = smp_processor_id();
int hwirq;
+ bool handled = false;
+ int res = 0;
chained_irq_enter(chip, desc);
@@ -136,14 +138,16 @@ static void partition_handle_irq(struct irq_desc *desc)
}
if (unlikely(hwirq == part->nr_parts)) {
- handle_bad_irq(desc);
+ handled = handle_bad_irq(desc);
} else {
unsigned int irq;
irq = irq_find_mapping(part->domain, hwirq);
- generic_handle_irq(irq);
+ res = generic_handle_irq(irq);
}
chained_irq_exit(chip, desc);
+
+ return handled || res == 1;
}
static int partition_domain_alloc(struct irq_domain *domain, unsigned int virq,
diff --git a/drivers/irqchip/msm_show_resume_irq.c b/drivers/irqchip/msm_show_resume_irq.c
index 5211496..cfadb7c 100644
--- a/drivers/irqchip/msm_show_resume_irq.c
+++ b/drivers/irqchip/msm_show_resume_irq.c
@@ -15,7 +15,7 @@
#include <linux/kernel.h>
#include <linux/init.h>
-int msm_show_resume_irq_mask;
+int msm_show_resume_irq_mask = 1;
module_param_named(
debug_mask, msm_show_resume_irq_mask, int, S_IRUGO | S_IWUSR | S_IWGRP
diff --git a/drivers/isdn/capi/kcapi.c b/drivers/isdn/capi/kcapi.c
index dd7e38a..d15347d 100644
--- a/drivers/isdn/capi/kcapi.c
+++ b/drivers/isdn/capi/kcapi.c
@@ -851,7 +851,7 @@ u16 capi20_get_manufacturer(u32 contr, u8 *buf)
u16 ret;
if (contr == 0) {
- strlcpy(buf, capi_manufakturer, CAPI_MANUFACTURER_LEN);
+ strncpy(buf, capi_manufakturer, CAPI_MANUFACTURER_LEN);
return CAPI_NOERROR;
}
@@ -859,7 +859,7 @@ u16 capi20_get_manufacturer(u32 contr, u8 *buf)
ctr = get_capi_ctr_by_nr(contr);
if (ctr && ctr->state == CAPI_CTR_RUNNING) {
- strlcpy(buf, ctr->manu, CAPI_MANUFACTURER_LEN);
+ strncpy(buf, ctr->manu, CAPI_MANUFACTURER_LEN);
ret = CAPI_NOERROR;
} else
ret = CAPI_REGNOTINSTALLED;
diff --git a/drivers/isdn/hardware/eicon/diva.c b/drivers/isdn/hardware/eicon/diva.c
index d91dd58..37aaea8 100644
--- a/drivers/isdn/hardware/eicon/diva.c
+++ b/drivers/isdn/hardware/eicon/diva.c
@@ -387,10 +387,10 @@ void divasa_xdi_driver_unload(void)
** Receive and process command from user mode utility
*/
void *diva_xdi_open_adapter(void *os_handle, const void __user *src,
- int length,
+ int length, void *mptr,
divas_xdi_copy_from_user_fn_t cp_fn)
{
- diva_xdi_um_cfg_cmd_t msg;
+ diva_xdi_um_cfg_cmd_t *msg = (diva_xdi_um_cfg_cmd_t *)mptr;
diva_os_xdi_adapter_t *a = NULL;
diva_os_spin_lock_magic_t old_irql;
struct list_head *tmp;
@@ -400,21 +400,21 @@ void *diva_xdi_open_adapter(void *os_handle, const void __user *src,
length, sizeof(diva_xdi_um_cfg_cmd_t)))
return NULL;
}
- if ((*cp_fn) (os_handle, &msg, src, sizeof(msg)) <= 0) {
+ if ((*cp_fn) (os_handle, msg, src, sizeof(*msg)) <= 0) {
DBG_ERR(("A: A(?) open, write error"))
return NULL;
}
diva_os_enter_spin_lock(&adapter_lock, &old_irql, "open_adapter");
list_for_each(tmp, &adapter_queue) {
a = list_entry(tmp, diva_os_xdi_adapter_t, link);
- if (a->controller == (int)msg.adapter)
+ if (a->controller == (int)msg->adapter)
break;
a = NULL;
}
diva_os_leave_spin_lock(&adapter_lock, &old_irql, "open_adapter");
if (!a) {
- DBG_ERR(("A: A(%d) open, adapter not found", msg.adapter))
+ DBG_ERR(("A: A(%d) open, adapter not found", msg->adapter))
}
return (a);
@@ -436,8 +436,10 @@ void diva_xdi_close_adapter(void *adapter, void *os_handle)
int
diva_xdi_write(void *adapter, void *os_handle, const void __user *src,
- int length, divas_xdi_copy_from_user_fn_t cp_fn)
+ int length, void *mptr,
+ divas_xdi_copy_from_user_fn_t cp_fn)
{
+ diva_xdi_um_cfg_cmd_t *msg = (diva_xdi_um_cfg_cmd_t *)mptr;
diva_os_xdi_adapter_t *a = (diva_os_xdi_adapter_t *) adapter;
void *data;
@@ -458,7 +460,13 @@ diva_xdi_write(void *adapter, void *os_handle, const void __user *src,
return (-2);
}
- length = (*cp_fn) (os_handle, data, src, length);
+ if (msg) {
+ *(diva_xdi_um_cfg_cmd_t *)data = *msg;
+ length = (*cp_fn) (os_handle, (char *)data + sizeof(*msg),
+ src + sizeof(*msg), length - sizeof(*msg));
+ } else {
+ length = (*cp_fn) (os_handle, data, src, length);
+ }
if (length > 0) {
if ((*(a->interface.cmd_proc))
(a, (diva_xdi_um_cfg_cmd_t *) data, length)) {
diff --git a/drivers/isdn/hardware/eicon/diva.h b/drivers/isdn/hardware/eicon/diva.h
index e979085..a0a607c 100644
--- a/drivers/isdn/hardware/eicon/diva.h
+++ b/drivers/isdn/hardware/eicon/diva.h
@@ -19,10 +19,11 @@ int diva_xdi_read(void *adapter, void *os_handle, void __user *dst,
int max_length, divas_xdi_copy_to_user_fn_t cp_fn);
int diva_xdi_write(void *adapter, void *os_handle, const void __user *src,
- int length, divas_xdi_copy_from_user_fn_t cp_fn);
+ int length, void *msg,
+ divas_xdi_copy_from_user_fn_t cp_fn);
void *diva_xdi_open_adapter(void *os_handle, const void __user *src,
- int length,
+ int length, void *msg,
divas_xdi_copy_from_user_fn_t cp_fn);
void diva_xdi_close_adapter(void *adapter, void *os_handle);
diff --git a/drivers/isdn/hardware/eicon/divasmain.c b/drivers/isdn/hardware/eicon/divasmain.c
index 32f3451..1e8b991 100644
--- a/drivers/isdn/hardware/eicon/divasmain.c
+++ b/drivers/isdn/hardware/eicon/divasmain.c
@@ -591,19 +591,22 @@ static int divas_release(struct inode *inode, struct file *file)
static ssize_t divas_write(struct file *file, const char __user *buf,
size_t count, loff_t *ppos)
{
+ diva_xdi_um_cfg_cmd_t msg;
int ret = -EINVAL;
if (!file->private_data) {
file->private_data = diva_xdi_open_adapter(file, buf,
- count,
+ count, &msg,
xdi_copy_from_user);
- }
- if (!file->private_data) {
- return (-ENODEV);
+ if (!file->private_data)
+ return (-ENODEV);
+ ret = diva_xdi_write(file->private_data, file,
+ buf, count, &msg, xdi_copy_from_user);
+ } else {
+ ret = diva_xdi_write(file->private_data, file,
+ buf, count, NULL, xdi_copy_from_user);
}
- ret = diva_xdi_write(file->private_data, file,
- buf, count, xdi_copy_from_user);
switch (ret) {
case -1: /* Message should be removed from rx mailbox first */
ret = -EBUSY;
@@ -622,11 +625,12 @@ static ssize_t divas_write(struct file *file, const char __user *buf,
static ssize_t divas_read(struct file *file, char __user *buf,
size_t count, loff_t *ppos)
{
+ diva_xdi_um_cfg_cmd_t msg;
int ret = -EINVAL;
if (!file->private_data) {
file->private_data = diva_xdi_open_adapter(file, buf,
- count,
+ count, &msg,
xdi_copy_from_user);
}
if (!file->private_data) {
diff --git a/drivers/isdn/i4l/isdn_common.c b/drivers/isdn/i4l/isdn_common.c
index e4c43a1..8088c34 100644
--- a/drivers/isdn/i4l/isdn_common.c
+++ b/drivers/isdn/i4l/isdn_common.c
@@ -1655,13 +1655,7 @@ isdn_ioctl(struct file *file, uint cmd, ulong arg)
} else
return -EINVAL;
case IIOCDBGVAR:
- if (arg) {
- if (copy_to_user(argp, &dev, sizeof(ulong)))
- return -EFAULT;
- return 0;
- } else
- return -EINVAL;
- break;
+ return -EINVAL;
default:
if ((cmd & IIOCDRVCTL) == IIOCDRVCTL)
cmd = ((cmd >> _IOC_NRSHIFT) & _IOC_NRMASK) & ISDN_DRVIOCTL_MASK;
diff --git a/drivers/leds/leds-qpnp-wled.c b/drivers/leds/leds-qpnp-wled.c
index 26f677b..949245b 100644
--- a/drivers/leds/leds-qpnp-wled.c
+++ b/drivers/leds/leds-qpnp-wled.c
@@ -35,6 +35,7 @@
/* ctrl registers */
#define QPNP_WLED_FAULT_STATUS(b) (b + 0x08)
#define QPNP_WLED_INT_RT_STS(b) (b + 0x10)
+#define QPNP_WLED_EXT_PIN_CTL(b) (b + 0x45)
#define QPNP_WLED_EN_REG(b) (b + 0x46)
#define QPNP_WLED_FDBK_OP_REG(b) (b + 0x48)
#define QPNP_WLED_VREF_REG(b) (b + 0x49)
@@ -92,6 +93,8 @@
#define QPNP_WLED_LCD_AUTO_PFM_THRESH_MASK GENMASK(3, 0)
#define QPNP_WLED_EN_PSM_BIT BIT(7)
+#define QPNP_WLED_EXT_PIN_CTL_BIT BIT(7)
+
#define QPNP_WLED_ILIM_MASK GENMASK(2, 0)
#define QPNP_WLED_ILIM_OVERWRITE BIT(7)
#define PMI8994_WLED_ILIM_MIN_MA 105
@@ -1608,7 +1611,7 @@ static irqreturn_t qpnp_wled_sc_irq_handler(int irq, void *_wled)
{
struct qpnp_wled *wled = _wled;
int rc;
- u8 val;
+ u8 val, ext_ctl;
rc = qpnp_wled_read_reg(wled,
QPNP_WLED_FAULT_STATUS(wled->ctrl_base), &val);
@@ -1617,12 +1620,32 @@ static irqreturn_t qpnp_wled_sc_irq_handler(int irq, void *_wled)
return IRQ_HANDLED;
}
- pr_err("WLED short circuit detected %d times fault_status=%x\n",
- ++wled->sc_cnt, val);
+ rc = qpnp_wled_read_reg(wled,
+ QPNP_WLED_EXT_PIN_CTL(wled->ctrl_base), &ext_ctl);
+
+ if (rc < 0) {
+ pr_err("Can't read WLED_EXT_PIN_CTL\n");
+ return IRQ_HANDLED;
+ }
+
+ pr_err("WLED short circuit: n=%d, ext_pin_ctl=%x, fault=%x\n",
+ ++wled->sc_cnt, ext_ctl, val);
mutex_lock(&wled->lock);
- qpnp_wled_module_en(wled, wled->ctrl_base, false);
- msleep(QPNP_WLED_SC_DLY_MS);
- qpnp_wled_module_en(wled, wled->ctrl_base, true);
+ if ((ext_ctl & QPNP_WLED_EXT_PIN_CTL_BIT) == 0x00) {
+ /* Disable, then re-enable WLED regulator. */
+ qpnp_wled_module_en(wled, wled->ctrl_base, false);
+ msleep(QPNP_WLED_SC_DLY_MS);
+ qpnp_wled_module_en(wled, wled->ctrl_base, true);
+ } else {
+ /* Disable, then restore external pin control. */
+ qpnp_wled_write_reg(wled,
+ QPNP_WLED_EXT_PIN_CTL(wled->ctrl_base),
+ ext_ctl & ~QPNP_WLED_EXT_PIN_CTL_BIT);
+ msleep(QPNP_WLED_SC_DLY_MS);
+ qpnp_wled_write_reg(wled,
+ QPNP_WLED_EXT_PIN_CTL(wled->ctrl_base), ext_ctl);
+ }
+
mutex_unlock(&wled->lock);
return IRQ_HANDLED;
@@ -2266,12 +2289,16 @@ static int qpnp_wled_config(struct qpnp_wled *wled)
return rc;
if (wled->en_ext_pfet_sc_pro) {
- reg = QPNP_WLED_EXT_FET_DTEST2;
- rc = qpnp_wled_sec_write_reg(wled,
+ if (!(wled->pmic_rev_id->pmic_subtype == PMI8998_SUBTYPE
+ && wled->pmic_rev_id->rev4 ==
+ PMI8998_V2P0_REV4)) {
+ reg = QPNP_WLED_EXT_FET_DTEST2;
+ rc = qpnp_wled_sec_write_reg(wled,
QPNP_WLED_TEST1_REG(wled->ctrl_base),
reg);
- if (rc)
- return rc;
+ if (rc)
+ return rc;
+ }
}
} else {
rc = qpnp_wled_read_reg(wled,
diff --git a/drivers/leds/trigger/ledtrig-heartbeat.c b/drivers/leds/trigger/ledtrig-heartbeat.c
index 410c39c..4600f95 100644
--- a/drivers/leds/trigger/ledtrig-heartbeat.c
+++ b/drivers/leds/trigger/ledtrig-heartbeat.c
@@ -17,6 +17,7 @@
#include <linux/slab.h>
#include <linux/timer.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/leds.h>
#include <linux/reboot.h>
#include "../leds.h"
diff --git a/drivers/leds/trigger/ledtrig-transient.c b/drivers/leds/trigger/ledtrig-transient.c
index 7e6011b..d6ab851 100644
--- a/drivers/leds/trigger/ledtrig-transient.c
+++ b/drivers/leds/trigger/ledtrig-transient.c
@@ -24,15 +24,18 @@
#include <linux/device.h>
#include <linux/slab.h>
#include <linux/timer.h>
+#include <linux/hrtimer.h>
#include <linux/leds.h>
#include "../leds.h"
struct transient_trig_data {
+ struct led_classdev *led_cdev;
int activate;
int state;
int restore_state;
unsigned long duration;
struct timer_list timer;
+ struct hrtimer hrtimer;
};
static void transient_timer_function(unsigned long data)
@@ -44,6 +47,54 @@ static void transient_timer_function(unsigned long data)
led_set_brightness_nosleep(led_cdev, transient_data->restore_state);
}
+static enum hrtimer_restart transient_hrtimer_function(struct hrtimer *timer)
+{
+ struct transient_trig_data *transient_data =
+ container_of(timer, struct transient_trig_data, hrtimer);
+
+ transient_timer_function((unsigned long)transient_data->led_cdev);
+
+ return HRTIMER_NORESTART;
+}
+
+static inline void transient_timer_setup(struct led_classdev *led_cdev)
+{
+ struct transient_trig_data *tdata = led_cdev->trigger_data;
+
+ if (led_cdev->flags & LED_BRIGHTNESS_FAST) {
+ tdata->led_cdev = led_cdev;
+ hrtimer_init(&tdata->hrtimer, CLOCK_MONOTONIC,
+ HRTIMER_MODE_REL);
+ tdata->hrtimer.function = transient_hrtimer_function;
+ } else {
+ setup_timer(&tdata->timer, transient_timer_function,
+ (unsigned long)led_cdev);
+ }
+}
+
+static inline void transient_timer_start(struct led_classdev *led_cdev)
+{
+ struct transient_trig_data *tdata = led_cdev->trigger_data;
+
+ if (led_cdev->flags & LED_BRIGHTNESS_FAST) {
+ hrtimer_start(&tdata->hrtimer, ms_to_ktime(tdata->duration),
+ HRTIMER_MODE_REL);
+ } else {
+ mod_timer(&tdata->timer,
+ jiffies + msecs_to_jiffies(tdata->duration));
+ }
+}
+
+static inline void transient_timer_cancel(struct led_classdev *led_cdev)
+{
+ struct transient_trig_data *tdata = led_cdev->trigger_data;
+
+ if (led_cdev->flags & LED_BRIGHTNESS_FAST)
+ hrtimer_cancel(&tdata->hrtimer);
+ else
+ del_timer_sync(&tdata->timer);
+}
+
static ssize_t transient_activate_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
@@ -70,7 +121,7 @@ static ssize_t transient_activate_store(struct device *dev,
/* cancel the running timer */
if (state == 0 && transient_data->activate == 1) {
- del_timer(&transient_data->timer);
+ transient_timer_cancel(led_cdev);
transient_data->activate = state;
led_set_brightness_nosleep(led_cdev,
transient_data->restore_state);
@@ -84,8 +135,7 @@ static ssize_t transient_activate_store(struct device *dev,
led_set_brightness_nosleep(led_cdev, transient_data->state);
transient_data->restore_state =
(transient_data->state == LED_FULL) ? LED_OFF : LED_FULL;
- mod_timer(&transient_data->timer,
- jiffies + msecs_to_jiffies(transient_data->duration));
+ transient_timer_start(led_cdev);
}
/* state == 0 && transient_data->activate == 0
@@ -182,8 +232,7 @@ static void transient_trig_activate(struct led_classdev *led_cdev)
if (rc)
goto err_out_state;
- setup_timer(&tdata->timer, transient_timer_function,
- (unsigned long) led_cdev);
+ transient_timer_setup(led_cdev);
led_cdev->activated = true;
return;
@@ -203,7 +252,7 @@ static void transient_trig_deactivate(struct led_classdev *led_cdev)
struct transient_trig_data *transient_data = led_cdev->trigger_data;
if (led_cdev->activated) {
- del_timer_sync(&transient_data->timer);
+ transient_timer_cancel(led_cdev);
led_set_brightness_nosleep(led_cdev,
transient_data->restore_state);
device_remove_file(led_cdev->dev, &dev_attr_activate);
diff --git a/drivers/macintosh/rack-meter.c b/drivers/macintosh/rack-meter.c
index 7755271..c5aba26 100644
--- a/drivers/macintosh/rack-meter.c
+++ b/drivers/macintosh/rack-meter.c
@@ -52,8 +52,8 @@ struct rackmeter_dma {
struct rackmeter_cpu {
struct delayed_work sniffer;
struct rackmeter *rm;
- cputime64_t prev_wall;
- cputime64_t prev_idle;
+ u64 prev_wall;
+ u64 prev_idle;
int zero;
} ____cacheline_aligned;
@@ -81,7 +81,7 @@ static int rackmeter_ignore_nice;
/* This is copied from cpufreq_ondemand, maybe we should put it in
* a common header somewhere
*/
-static inline cputime64_t get_cpu_idle_time(unsigned int cpu)
+static inline u64 get_cpu_idle_time(unsigned int cpu)
{
u64 retval;
@@ -154,8 +154,8 @@ static void rackmeter_do_pause(struct rackmeter *rm, int pause)
DBDMA_DO_STOP(rm->dma_regs);
return;
}
- memset(rdma->buf1, 0, ARRAY_SIZE(rdma->buf1));
- memset(rdma->buf2, 0, ARRAY_SIZE(rdma->buf2));
+ memset(rdma->buf1, 0, sizeof(rdma->buf1));
+ memset(rdma->buf2, 0, sizeof(rdma->buf2));
rm->dma_buf_v->mark = 0;
@@ -217,23 +217,23 @@ static void rackmeter_do_timer(struct work_struct *work)
container_of(work, struct rackmeter_cpu, sniffer.work);
struct rackmeter *rm = rcpu->rm;
unsigned int cpu = smp_processor_id();
- cputime64_t cur_jiffies, total_idle_ticks;
- unsigned int total_ticks, idle_ticks;
+ u64 cur_nsecs, total_idle_nsecs;
+ u64 total_nsecs, idle_nsecs;
int i, offset, load, cumm, pause;
- cur_jiffies = jiffies64_to_cputime64(get_jiffies_64());
- total_ticks = (unsigned int) (cur_jiffies - rcpu->prev_wall);
- rcpu->prev_wall = cur_jiffies;
+ cur_nsecs = jiffies64_to_nsecs(get_jiffies_64());
+ total_nsecs = cur_nsecs - rcpu->prev_wall;
+ rcpu->prev_wall = cur_nsecs;
- total_idle_ticks = get_cpu_idle_time(cpu);
- idle_ticks = (unsigned int) (total_idle_ticks - rcpu->prev_idle);
- idle_ticks = min(idle_ticks, total_ticks);
- rcpu->prev_idle = total_idle_ticks;
+ total_idle_nsecs = get_cpu_idle_time(cpu);
+ idle_nsecs = total_idle_nsecs - rcpu->prev_idle;
+ idle_nsecs = min(idle_nsecs, total_nsecs);
+ rcpu->prev_idle = total_idle_nsecs;
/* We do a very dumb calculation to update the LEDs for now,
* we'll do better once we have actual PWM implemented
*/
- load = (9 * (total_ticks - idle_ticks)) / total_ticks;
+ load = div64_u64(9 * (total_nsecs - idle_nsecs), total_nsecs);
offset = cpu << 3;
cumm = 0;
@@ -278,7 +278,7 @@ static void rackmeter_init_cpu_sniffer(struct rackmeter *rm)
continue;
rcpu = &rm->cpu[cpu];
rcpu->prev_idle = get_cpu_idle_time(cpu);
- rcpu->prev_wall = jiffies64_to_cputime64(get_jiffies_64());
+ rcpu->prev_wall = jiffies64_to_nsecs(get_jiffies_64());
schedule_delayed_work_on(cpu, &rm->cpu[cpu].sniffer,
msecs_to_jiffies(CPU_SAMPLING_RATE));
}
diff --git a/drivers/macintosh/via-pmu.c b/drivers/macintosh/via-pmu.c
index 91081dc..32c6967 100644
--- a/drivers/macintosh/via-pmu.c
+++ b/drivers/macintosh/via-pmu.c
@@ -531,8 +531,9 @@ init_pmu(void)
int timeout;
struct adb_request req;
- out_8(&via[B], via[B] | TREQ); /* negate TREQ */
- out_8(&via[DIRB], (via[DIRB] | TREQ) & ~TACK); /* TACK in, TREQ out */
+ /* Negate TREQ. Set TACK to input and TREQ to output. */
+ out_8(&via[B], in_8(&via[B]) | TREQ);
+ out_8(&via[DIRB], (in_8(&via[DIRB]) | TREQ) & ~TACK);
pmu_request(&req, NULL, 2, PMU_SET_INTR_MASK, pmu_intr_mask);
timeout = 100000;
@@ -1454,8 +1455,8 @@ pmu_sr_intr(void)
struct adb_request *req;
int bite = 0;
- if (via[B] & TREQ) {
- printk(KERN_ERR "PMU: spurious SR intr (%x)\n", via[B]);
+ if (in_8(&via[B]) & TREQ) {
+ printk(KERN_ERR "PMU: spurious SR intr (%x)\n", in_8(&via[B]));
out_8(&via[IFR], SR_INT);
return NULL;
}
diff --git a/drivers/mailbox/mailbox-xgene-slimpro.c b/drivers/mailbox/mailbox-xgene-slimpro.c
index dd2afbc..26d2f89 100644
--- a/drivers/mailbox/mailbox-xgene-slimpro.c
+++ b/drivers/mailbox/mailbox-xgene-slimpro.c
@@ -195,9 +195,9 @@ static int slimpro_mbox_probe(struct platform_device *pdev)
platform_set_drvdata(pdev, ctx);
regs = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- mb_base = devm_ioremap(&pdev->dev, regs->start, resource_size(regs));
- if (!mb_base)
- return -ENOMEM;
+ mb_base = devm_ioremap_resource(&pdev->dev, regs);
+ if (IS_ERR(mb_base))
+ return PTR_ERR(mb_base);
/* Setup mailbox links */
for (i = 0; i < MBOX_CNT; i++) {
diff --git a/drivers/md/Kconfig b/drivers/md/Kconfig
index 89fc93b..6ed482c 100644
--- a/drivers/md/Kconfig
+++ b/drivers/md/Kconfig
@@ -277,6 +277,24 @@
If unsure, say N.
+config DM_DEFAULT_KEY
+ tristate "Default-key crypt target support"
+ depends on BLK_DEV_DM
+ depends on PFK
+ ---help---
+ This (currently Android-specific) device-mapper target allows you to
+ create a device that assigns a default encryption key to bios that
+ don't already have one. This can sit between inline cryptographic
+ acceleration hardware and filesystems that use it. This ensures that
+ where the filesystem doesn't explicitly specify a key, such as for
+ filesystem metadata, a default key will be used instead, leaving no
+ sectors unencrypted.
+
+ To compile this code as a module, choose M here: the module will be
+ called dm-default-key.
+
+ If unsure, say N.
+
config DM_REQ_CRYPT
tristate "Req Crypt target support"
depends on BLK_DEV_DM
@@ -476,21 +494,6 @@
If unsure, say N.
-config DM_VERITY_HASH_PREFETCH_MIN_SIZE_128
- bool "Prefetch size 128"
-
-config DM_VERITY_HASH_PREFETCH_MIN_SIZE
- int "Verity hash prefetch minimum size"
- depends on DM_VERITY
- range 1 4096
- default 128 if DM_VERITY_HASH_PREFETCH_MIN_SIZE_128
- default 1
- ---help---
- This sets minimum number of hash blocks to prefetch for dm-verity.
- For devices like eMMC, having larger prefetch size like 128 can improve
- performance with increased memory consumption for keeping more hashes
- in RAM.
-
config DM_VERITY_FEC
bool "Verity forward error correction support"
depends on DM_VERITY
@@ -543,7 +546,6 @@
depends on ASYMMETRIC_KEY_TYPE
depends on ASYMMETRIC_PUBLIC_KEY_SUBTYPE
depends on MD_LINEAR=y
- select DM_VERITY_HASH_PREFETCH_MIN_SIZE_128
---help---
This device-mapper target is virtually a VERITY target. This
target is setup by reading the metadata contents piggybacked
diff --git a/drivers/md/Makefile b/drivers/md/Makefile
index f14e2fc..c3bf33b 100644
--- a/drivers/md/Makefile
+++ b/drivers/md/Makefile
@@ -40,6 +40,7 @@
obj-$(CONFIG_DM_BUFIO) += dm-bufio.o
obj-$(CONFIG_DM_BIO_PRISON) += dm-bio-prison.o
obj-$(CONFIG_DM_CRYPT) += dm-crypt.o
+obj-$(CONFIG_DM_DEFAULT_KEY) += dm-default-key.o
obj-$(CONFIG_DM_DELAY) += dm-delay.o
obj-$(CONFIG_DM_FLAKEY) += dm-flakey.o
obj-$(CONFIG_DM_MULTIPATH) += dm-multipath.o dm-round-robin.o
diff --git a/drivers/md/bcache/alloc.c b/drivers/md/bcache/alloc.c
index d23337e..dd344ee 100644
--- a/drivers/md/bcache/alloc.c
+++ b/drivers/md/bcache/alloc.c
@@ -284,8 +284,10 @@ do { \
break; \
\
mutex_unlock(&(ca)->set->bucket_lock); \
- if (kthread_should_stop()) \
+ if (kthread_should_stop()) { \
+ set_current_state(TASK_RUNNING); \
return 0; \
+ } \
\
schedule(); \
mutex_lock(&(ca)->set->bucket_lock); \
diff --git a/drivers/md/bcache/bcache.h b/drivers/md/bcache/bcache.h
index 02619ca..7fe7df5 100644
--- a/drivers/md/bcache/bcache.h
+++ b/drivers/md/bcache/bcache.h
@@ -904,7 +904,7 @@ void bcache_write_super(struct cache_set *);
int bch_flash_dev_create(struct cache_set *c, uint64_t size);
-int bch_cached_dev_attach(struct cached_dev *, struct cache_set *);
+int bch_cached_dev_attach(struct cached_dev *, struct cache_set *, uint8_t *);
void bch_cached_dev_detach(struct cached_dev *);
void bch_cached_dev_run(struct cached_dev *);
void bcache_device_stop(struct bcache_device *);
diff --git a/drivers/md/bcache/btree.c b/drivers/md/bcache/btree.c
index cac297f..4e34afb 100644
--- a/drivers/md/bcache/btree.c
+++ b/drivers/md/bcache/btree.c
@@ -1864,14 +1864,17 @@ void bch_initial_gc_finish(struct cache_set *c)
*/
for_each_cache(ca, c, i) {
for_each_bucket(b, ca) {
- if (fifo_full(&ca->free[RESERVE_PRIO]))
+ if (fifo_full(&ca->free[RESERVE_PRIO]) &&
+ fifo_full(&ca->free[RESERVE_BTREE]))
break;
if (bch_can_invalidate_bucket(ca, b) &&
!GC_MARK(b)) {
__bch_invalidate_one_bucket(ca, b);
- fifo_push(&ca->free[RESERVE_PRIO],
- b - ca->buckets);
+ if (!fifo_push(&ca->free[RESERVE_PRIO],
+ b - ca->buckets))
+ fifo_push(&ca->free[RESERVE_BTREE],
+ b - ca->buckets);
}
}
}
@@ -2364,7 +2367,7 @@ static int refill_keybuf_fn(struct btree_op *op, struct btree *b,
struct keybuf *buf = refill->buf;
int ret = MAP_CONTINUE;
- if (bkey_cmp(k, refill->end) >= 0) {
+ if (bkey_cmp(k, refill->end) > 0) {
ret = MAP_DONE;
goto out;
}
diff --git a/drivers/md/bcache/request.c b/drivers/md/bcache/request.c
index 723302c..faa9bfe 100644
--- a/drivers/md/bcache/request.c
+++ b/drivers/md/bcache/request.c
@@ -633,11 +633,11 @@ static void do_bio_hook(struct search *s, struct bio *orig_bio)
static void search_free(struct closure *cl)
{
struct search *s = container_of(cl, struct search, cl);
- bio_complete(s);
if (s->iop.bio)
bio_put(s->iop.bio);
+ bio_complete(s);
closure_debug_destroy(cl);
mempool_free(s, s->d->c->search);
}
diff --git a/drivers/md/bcache/super.c b/drivers/md/bcache/super.c
index e4c2d5d..11c9953 100644
--- a/drivers/md/bcache/super.c
+++ b/drivers/md/bcache/super.c
@@ -938,7 +938,8 @@ void bch_cached_dev_detach(struct cached_dev *dc)
cached_dev_put(dc);
}
-int bch_cached_dev_attach(struct cached_dev *dc, struct cache_set *c)
+int bch_cached_dev_attach(struct cached_dev *dc, struct cache_set *c,
+ uint8_t *set_uuid)
{
uint32_t rtime = cpu_to_le32(get_seconds());
struct uuid_entry *u;
@@ -947,7 +948,8 @@ int bch_cached_dev_attach(struct cached_dev *dc, struct cache_set *c)
bdevname(dc->bdev, buf);
- if (memcmp(dc->sb.set_uuid, c->sb.set_uuid, 16))
+ if ((set_uuid && memcmp(set_uuid, c->sb.set_uuid, 16)) ||
+ (!set_uuid && memcmp(dc->sb.set_uuid, c->sb.set_uuid, 16)))
return -ENOENT;
if (dc->disk.c) {
@@ -1191,7 +1193,7 @@ static void register_bdev(struct cache_sb *sb, struct page *sb_page,
list_add(&dc->list, &uncached_devices);
list_for_each_entry(c, &bch_cache_sets, list)
- bch_cached_dev_attach(dc, c);
+ bch_cached_dev_attach(dc, c, NULL);
if (BDEV_STATE(&dc->sb) == BDEV_STATE_NONE ||
BDEV_STATE(&dc->sb) == BDEV_STATE_STALE)
@@ -1714,7 +1716,7 @@ static void run_cache_set(struct cache_set *c)
bcache_write_super(c);
list_for_each_entry_safe(dc, t, &uncached_devices, list)
- bch_cached_dev_attach(dc, c);
+ bch_cached_dev_attach(dc, c, NULL);
flash_devs_run(c);
@@ -1831,6 +1833,7 @@ void bch_cache_release(struct kobject *kobj)
static int cache_alloc(struct cache *ca)
{
size_t free;
+ size_t btree_buckets;
struct bucket *b;
__module_get(THIS_MODULE);
@@ -1840,9 +1843,19 @@ static int cache_alloc(struct cache *ca)
ca->journal.bio.bi_max_vecs = 8;
ca->journal.bio.bi_io_vec = ca->journal.bio.bi_inline_vecs;
+ /*
+ * when ca->sb.njournal_buckets is not zero, journal exists,
+ * and in bch_journal_replay(), tree node may split,
+ * so bucket of RESERVE_BTREE type is needed,
+ * the worst situation is all journal buckets are valid journal,
+ * and all the keys need to replay,
+ * so the number of RESERVE_BTREE type buckets should be as much
+ * as journal buckets
+ */
+ btree_buckets = ca->sb.njournal_buckets ?: 8;
free = roundup_pow_of_two(ca->sb.nbuckets) >> 10;
- if (!init_fifo(&ca->free[RESERVE_BTREE], 8, GFP_KERNEL) ||
+ if (!init_fifo(&ca->free[RESERVE_BTREE], btree_buckets, GFP_KERNEL) ||
!init_fifo_exact(&ca->free[RESERVE_PRIO], prio_buckets(ca), GFP_KERNEL) ||
!init_fifo(&ca->free[RESERVE_MOVINGGC], free, GFP_KERNEL) ||
!init_fifo(&ca->free[RESERVE_NONE], free, GFP_KERNEL) ||
diff --git a/drivers/md/bcache/sysfs.c b/drivers/md/bcache/sysfs.c
index 4fbb553..5a5c1f1 100644
--- a/drivers/md/bcache/sysfs.c
+++ b/drivers/md/bcache/sysfs.c
@@ -191,7 +191,7 @@ STORE(__cached_dev)
{
struct cached_dev *dc = container_of(kobj, struct cached_dev,
disk.kobj);
- ssize_t v = size;
+ ssize_t v;
struct cache_set *c;
struct kobj_uevent_env *env;
@@ -263,17 +263,20 @@ STORE(__cached_dev)
}
if (attr == &sysfs_attach) {
- if (bch_parse_uuid(buf, dc->sb.set_uuid) < 16)
+ uint8_t set_uuid[16];
+
+ if (bch_parse_uuid(buf, set_uuid) < 16)
return -EINVAL;
+ v = -ENOENT;
list_for_each_entry(c, &bch_cache_sets, list) {
- v = bch_cached_dev_attach(dc, c);
+ v = bch_cached_dev_attach(dc, c, set_uuid);
if (!v)
return size;
}
pr_err("Can't attach %s: cache set not found", buf);
- size = v;
+ return v;
}
if (attr == &sysfs_detach && dc->disk.c)
diff --git a/drivers/md/bcache/writeback.c b/drivers/md/bcache/writeback.c
index 4ce2b19..cdf388d 100644
--- a/drivers/md/bcache/writeback.c
+++ b/drivers/md/bcache/writeback.c
@@ -420,18 +420,27 @@ static int bch_writeback_thread(void *arg)
while (!kthread_should_stop()) {
down_write(&dc->writeback_lock);
- if (!atomic_read(&dc->has_dirty) ||
- (!test_bit(BCACHE_DEV_DETACHING, &dc->disk.flags) &&
- !dc->writeback_running)) {
+ set_current_state(TASK_INTERRUPTIBLE);
+ /*
+ * If the bache device is detaching, skip here and continue
+ * to perform writeback. Otherwise, if no dirty data on cache,
+ * or there is dirty data on cache but writeback is disabled,
+ * the writeback thread should sleep here and wait for others
+ * to wake up it.
+ */
+ if (!test_bit(BCACHE_DEV_DETACHING, &dc->disk.flags) &&
+ (!atomic_read(&dc->has_dirty) || !dc->writeback_running)) {
up_write(&dc->writeback_lock);
- set_current_state(TASK_INTERRUPTIBLE);
- if (kthread_should_stop())
+ if (kthread_should_stop()) {
+ set_current_state(TASK_RUNNING);
return 0;
+ }
schedule();
continue;
}
+ set_current_state(TASK_RUNNING);
searched_full_index = refill_dirty(dc);
@@ -441,6 +450,16 @@ static int bch_writeback_thread(void *arg)
cached_dev_put(dc);
SET_BDEV_STATE(&dc->sb, BDEV_STATE_CLEAN);
bch_write_bdev_super(dc, NULL);
+ /*
+ * If bcache device is detaching via sysfs interface,
+ * writeback thread should stop after there is no dirty
+ * data on cache. BCACHE_DEV_DETACHING flag is set in
+ * bch_cached_dev_detach().
+ */
+ if (test_bit(BCACHE_DEV_DETACHING, &dc->disk.flags)) {
+ up_write(&dc->writeback_lock);
+ break;
+ }
}
up_write(&dc->writeback_lock);
diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c
index 3ec647e..c837def 100644
--- a/drivers/md/dm-bufio.c
+++ b/drivers/md/dm-bufio.c
@@ -373,9 +373,6 @@ static void __cache_size_refresh(void)
static void *alloc_buffer_data(struct dm_bufio_client *c, gfp_t gfp_mask,
enum data_mode *data_mode)
{
- unsigned noio_flag;
- void *ptr;
-
if (c->block_size <= DM_BUFIO_BLOCK_SIZE_SLAB_LIMIT) {
*data_mode = DATA_MODE_SLAB;
return kmem_cache_alloc(DM_BUFIO_CACHE(c), gfp_mask);
@@ -399,16 +396,16 @@ static void *alloc_buffer_data(struct dm_bufio_client *c, gfp_t gfp_mask,
* all allocations done by this process (including pagetables) are done
* as if GFP_NOIO was specified.
*/
+ if (gfp_mask & __GFP_NORETRY) {
+ unsigned noio_flag = memalloc_noio_save();
+ void *ptr = __vmalloc(c->block_size, gfp_mask | __GFP_HIGHMEM,
+ PAGE_KERNEL);
- if (gfp_mask & __GFP_NORETRY)
- noio_flag = memalloc_noio_save();
-
- ptr = __vmalloc(c->block_size, gfp_mask | __GFP_HIGHMEM, PAGE_KERNEL);
-
- if (gfp_mask & __GFP_NORETRY)
memalloc_noio_restore(noio_flag);
+ return ptr;
+ }
- return ptr;
+ return __vmalloc(c->block_size, gfp_mask | __GFP_HIGHMEM, PAGE_KERNEL);
}
/*
@@ -822,12 +819,14 @@ enum new_flag {
static struct dm_buffer *__alloc_buffer_wait_no_callback(struct dm_bufio_client *c, enum new_flag nf)
{
struct dm_buffer *b;
+ bool tried_noio_alloc = false;
/*
* dm-bufio is resistant to allocation failures (it just keeps
* one buffer reserved in cases all the allocations fail).
* So set flags to not try too hard:
- * GFP_NOIO: don't recurse into the I/O layer
+ * GFP_NOWAIT: don't wait; if we need to sleep we'll release our
+ * mutex and wait ourselves.
* __GFP_NORETRY: don't retry and rather return failure
* __GFP_NOMEMALLOC: don't use emergency reserves
* __GFP_NOWARN: don't print a warning in case of failure
@@ -837,7 +836,7 @@ static struct dm_buffer *__alloc_buffer_wait_no_callback(struct dm_bufio_client
*/
while (1) {
if (dm_bufio_cache_size_latch != 1) {
- b = alloc_buffer(c, GFP_NOIO | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN);
+ b = alloc_buffer(c, GFP_NOWAIT | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN);
if (b)
return b;
}
@@ -845,6 +844,15 @@ static struct dm_buffer *__alloc_buffer_wait_no_callback(struct dm_bufio_client
if (nf == NF_PREFETCH)
return NULL;
+ if (dm_bufio_cache_size_latch != 1 && !tried_noio_alloc) {
+ dm_bufio_unlock(c);
+ b = alloc_buffer(c, GFP_NOIO | __GFP_NORETRY | __GFP_NOMEMALLOC | __GFP_NOWARN);
+ dm_bufio_lock(c);
+ if (b)
+ return b;
+ tried_noio_alloc = true;
+ }
+
if (!list_empty(&c->reserved_buffers)) {
b = list_entry(c->reserved_buffers.next,
struct dm_buffer, lru_list);
@@ -1590,19 +1598,11 @@ dm_bufio_shrink_scan(struct shrinker *shrink, struct shrink_control *sc)
static unsigned long
dm_bufio_shrink_count(struct shrinker *shrink, struct shrink_control *sc)
{
- struct dm_bufio_client *c;
- unsigned long count;
- unsigned long retain_target;
+ struct dm_bufio_client *c = container_of(shrink, struct dm_bufio_client, shrinker);
+ unsigned long count = READ_ONCE(c->n_buffers[LIST_CLEAN]) +
+ READ_ONCE(c->n_buffers[LIST_DIRTY]);
+ unsigned long retain_target = get_retain_buffers(c);
- c = container_of(shrink, struct dm_bufio_client, shrinker);
- if (sc->gfp_mask & __GFP_FS)
- dm_bufio_lock(c);
- else if (!dm_bufio_trylock(c))
- return 0;
-
- count = c->n_buffers[LIST_CLEAN] + c->n_buffers[LIST_DIRTY];
- retain_target = get_retain_buffers(c);
- dm_bufio_unlock(c);
return (count < retain_target) ? 0 : (count - retain_target);
}
diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c
index 6937ca4..62eb4b7 100644
--- a/drivers/md/dm-cache-metadata.c
+++ b/drivers/md/dm-cache-metadata.c
@@ -344,7 +344,7 @@ static int __write_initial_superblock(struct dm_cache_metadata *cmd)
disk_super->version = cpu_to_le32(MAX_CACHE_VERSION);
memset(disk_super->policy_name, 0, sizeof(disk_super->policy_name));
memset(disk_super->policy_version, 0, sizeof(disk_super->policy_version));
- disk_super->policy_hint_size = 0;
+ disk_super->policy_hint_size = cpu_to_le32(0);
__copy_sm_root(cmd, disk_super);
@@ -659,6 +659,7 @@ static int __commit_transaction(struct dm_cache_metadata *cmd,
disk_super->policy_version[0] = cpu_to_le32(cmd->policy_version[0]);
disk_super->policy_version[1] = cpu_to_le32(cmd->policy_version[1]);
disk_super->policy_version[2] = cpu_to_le32(cmd->policy_version[2]);
+ disk_super->policy_hint_size = cpu_to_le32(cmd->policy_hint_size);
disk_super->read_hits = cpu_to_le32(cmd->stats.read_hits);
disk_super->read_misses = cpu_to_le32(cmd->stats.read_misses);
@@ -1261,8 +1262,8 @@ static int __load_mappings(struct dm_cache_metadata *cmd,
if (hints_valid) {
r = dm_array_cursor_next(&cmd->hint_cursor);
if (r) {
- DMERR("dm_array_cursor_next for hint failed");
- goto out;
+ dm_array_cursor_end(&cmd->hint_cursor);
+ hints_valid = false;
}
}
}
diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index bed056c..f3993a4 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -3390,8 +3390,13 @@ static dm_cblock_t get_cache_dev_size(struct cache *cache)
static bool can_resize(struct cache *cache, dm_cblock_t new_size)
{
- if (from_cblock(new_size) > from_cblock(cache->cache_size))
- return true;
+ if (from_cblock(new_size) > from_cblock(cache->cache_size)) {
+ if (cache->sized) {
+ DMERR("%s: unable to extend cache due to missing cache table reload",
+ cache_device_name(cache));
+ return false;
+ }
+ }
/*
* We can't drop a dirty block when shrinking the cache.
diff --git a/drivers/md/dm-default-key.c b/drivers/md/dm-default-key.c
new file mode 100644
index 0000000..7feb21b
--- /dev/null
+++ b/drivers/md/dm-default-key.c
@@ -0,0 +1,225 @@
+/*
+ * Copyright (C) 2017 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/device-mapper.h>
+#include <linux/module.h>
+#include <linux/pfk.h>
+
+#define DM_MSG_PREFIX "default-key"
+
+struct default_key_c {
+ struct dm_dev *dev;
+ sector_t start;
+ struct blk_encryption_key key;
+};
+
+static void default_key_dtr(struct dm_target *ti)
+{
+ struct default_key_c *dkc = ti->private;
+
+ if (dkc->dev)
+ dm_put_device(ti, dkc->dev);
+ kzfree(dkc);
+}
+
+/*
+ * Construct a default-key mapping: <mode> <key> <dev_path> <start>
+ */
+static int default_key_ctr(struct dm_target *ti, unsigned int argc, char **argv)
+{
+ struct default_key_c *dkc;
+ size_t key_size;
+ unsigned long long tmp;
+ char dummy;
+ int err;
+
+ if (argc != 4) {
+ ti->error = "Invalid argument count";
+ return -EINVAL;
+ }
+
+ dkc = kzalloc(sizeof(*dkc), GFP_KERNEL);
+ if (!dkc) {
+ ti->error = "Out of memory";
+ return -ENOMEM;
+ }
+ ti->private = dkc;
+
+ if (strcmp(argv[0], "AES-256-XTS") != 0) {
+ ti->error = "Unsupported encryption mode";
+ err = -EINVAL;
+ goto bad;
+ }
+
+ key_size = strlen(argv[1]);
+ if (key_size != 2 * BLK_ENCRYPTION_KEY_SIZE_AES_256_XTS) {
+ ti->error = "Unsupported key size";
+ err = -EINVAL;
+ goto bad;
+ }
+ key_size /= 2;
+
+ if (hex2bin(dkc->key.raw, argv[1], key_size) != 0) {
+ ti->error = "Malformed key string";
+ err = -EINVAL;
+ goto bad;
+ }
+
+ err = dm_get_device(ti, argv[2], dm_table_get_mode(ti->table),
+ &dkc->dev);
+ if (err) {
+ ti->error = "Device lookup failed";
+ goto bad;
+ }
+
+ if (sscanf(argv[3], "%llu%c", &tmp, &dummy) != 1) {
+ ti->error = "Invalid start sector";
+ err = -EINVAL;
+ goto bad;
+ }
+ dkc->start = tmp;
+
+ if (!blk_queue_inlinecrypt(bdev_get_queue(dkc->dev->bdev))) {
+ ti->error = "Device does not support inline encryption";
+ err = -EINVAL;
+ goto bad;
+ }
+
+ /* Pass flush requests through to the underlying device. */
+ ti->num_flush_bios = 1;
+
+ /*
+ * We pass discard requests through to the underlying device, although
+ * the discarded blocks will be zeroed, which leaks information about
+ * unused blocks. It's also impossible for dm-default-key to know not
+ * to decrypt discarded blocks, so they will not be read back as zeroes
+ * and we must set discard_zeroes_data_unsupported.
+ */
+ ti->num_discard_bios = 1;
+ ti->discard_zeroes_data_unsupported = true;
+
+ /*
+ * It's unclear whether WRITE_SAME would work with inline encryption; it
+ * would depend on whether the hardware duplicates the data before or
+ * after encryption. But since the internal storage in "muskie" devices
+ * (MSM8998-based) doesn't claim to support WRITE_SAME anyway, we don't
+ * currently have a way to test it. Leave it disabled it for now.
+ */
+ /*ti->num_write_same_bios = 1;*/
+
+ return 0;
+
+bad:
+ default_key_dtr(ti);
+ return err;
+}
+
+static int default_key_map(struct dm_target *ti, struct bio *bio)
+{
+ const struct default_key_c *dkc = ti->private;
+
+ bio->bi_bdev = dkc->dev->bdev;
+ if (bio_sectors(bio)) {
+ bio->bi_iter.bi_sector = dkc->start +
+ dm_target_offset(ti, bio->bi_iter.bi_sector);
+ }
+
+ if (!bio->bi_crypt_key && !bio->bi_crypt_skip)
+ bio->bi_crypt_key = &dkc->key;
+
+ return DM_MAPIO_REMAPPED;
+}
+
+static void default_key_status(struct dm_target *ti, status_type_t type,
+ unsigned int status_flags, char *result,
+ unsigned int maxlen)
+{
+ const struct default_key_c *dkc = ti->private;
+ unsigned int sz = 0;
+
+ switch (type) {
+ case STATUSTYPE_INFO:
+ result[0] = '\0';
+ break;
+
+ case STATUSTYPE_TABLE:
+
+ /* encryption mode */
+ DMEMIT("AES-256-XTS");
+
+ /* reserved for key; dm-crypt shows it, but we don't for now */
+ DMEMIT(" -");
+
+ /* name of underlying device, and the start sector in it */
+ DMEMIT(" %s %llu", dkc->dev->name,
+ (unsigned long long)dkc->start);
+ break;
+ }
+}
+
+static int default_key_prepare_ioctl(struct dm_target *ti,
+ struct block_device **bdev, fmode_t *mode)
+{
+ struct default_key_c *dkc = ti->private;
+ struct dm_dev *dev = dkc->dev;
+
+ *bdev = dev->bdev;
+
+ /*
+ * Only pass ioctls through if the device sizes match exactly.
+ */
+ if (dkc->start ||
+ ti->len != i_size_read(dev->bdev->bd_inode) >> SECTOR_SHIFT)
+ return 1;
+ return 0;
+}
+
+static int default_key_iterate_devices(struct dm_target *ti,
+ iterate_devices_callout_fn fn,
+ void *data)
+{
+ struct default_key_c *dkc = ti->private;
+
+ return fn(ti, dkc->dev, dkc->start, ti->len, data);
+}
+
+static struct target_type default_key_target = {
+ .name = "default-key",
+ .version = {1, 0, 0},
+ .module = THIS_MODULE,
+ .ctr = default_key_ctr,
+ .dtr = default_key_dtr,
+ .map = default_key_map,
+ .status = default_key_status,
+ .prepare_ioctl = default_key_prepare_ioctl,
+ .iterate_devices = default_key_iterate_devices,
+};
+
+static int __init dm_default_key_init(void)
+{
+ return dm_register_target(&default_key_target);
+}
+
+static void __exit dm_default_key_exit(void)
+{
+ dm_unregister_target(&default_key_target);
+}
+
+module_init(dm_default_key_init);
+module_exit(dm_default_key_exit);
+
+MODULE_AUTHOR("Paul Lawrence <paullawrence@google.com>");
+MODULE_AUTHOR("Paul Crowley <paulcrowley@google.com>");
+MODULE_AUTHOR("Eric Biggers <ebiggers@google.com>");
+MODULE_DESCRIPTION(DM_NAME " target for encrypting filesystem metadata");
+MODULE_LICENSE("GPL");
diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c
index f688bfe..446d76e 100644
--- a/drivers/md/dm-ioctl.c
+++ b/drivers/md/dm-ioctl.c
@@ -1692,8 +1692,7 @@ static void free_params(struct dm_ioctl *param, size_t param_size, int param_fla
}
static int copy_params(struct dm_ioctl __user *user, struct dm_ioctl *param_kernel,
- int ioctl_flags,
- struct dm_ioctl **param, int *param_flags)
+ int ioctl_flags, struct dm_ioctl **param, int *param_flags)
{
struct dm_ioctl *dmi;
int secure_data;
@@ -1738,18 +1737,13 @@ static int copy_params(struct dm_ioctl __user *user, struct dm_ioctl *param_kern
*param_flags |= DM_PARAMS_MALLOC;
- if (copy_from_user(dmi, user, param_kernel->data_size))
- goto bad;
+ /* Copy from param_kernel (which was already copied from user) */
+ memcpy(dmi, param_kernel, minimum_data_size);
+ if (copy_from_user(&dmi->data, (char __user *)user + minimum_data_size,
+ param_kernel->data_size - minimum_data_size))
+ goto bad;
data_copied:
- /*
- * Abort if something changed the ioctl data while it was being copied.
- */
- if (dmi->data_size != param_kernel->data_size) {
- DMERR("rejecting ioctl: data size modified while processing parameters");
- goto bad;
- }
-
/* Wipe the user buffer so we do not return it to userspace */
if (secure_data && clear_user(user, param_kernel->data_size))
goto bad;
diff --git a/drivers/md/dm-kcopyd.c b/drivers/md/dm-kcopyd.c
index 9e9d04cb..56fcccc 100644
--- a/drivers/md/dm-kcopyd.c
+++ b/drivers/md/dm-kcopyd.c
@@ -454,6 +454,8 @@ static int run_complete_job(struct kcopyd_job *job)
if (atomic_dec_and_test(&kc->nr_jobs))
wake_up(&kc->destroyq);
+ cond_resched();
+
return 0;
}
diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c
index ee75e35..3f389b2 100644
--- a/drivers/md/dm-raid.c
+++ b/drivers/md/dm-raid.c
@@ -2880,6 +2880,11 @@ static int raid_ctr(struct dm_target *ti, unsigned int argc, char **argv)
set_bit(RT_FLAG_UPDATE_SBS, &rs->runtime_flags);
rs_set_new(rs);
} else if (rs_is_recovering(rs)) {
+ /* Rebuild particular devices */
+ if (test_bit(__CTR_FLAG_REBUILD, &rs->ctr_flags)) {
+ set_bit(RT_FLAG_UPDATE_SBS, &rs->runtime_flags);
+ rs_setup_recovery(rs, MaxSector);
+ }
/* A recovering raid set may be resized */
; /* skip setup rs */
} else if (rs_is_reshaping(rs)) {
diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
index b75ccef..0294631 100644
--- a/drivers/md/dm-table.c
+++ b/drivers/md/dm-table.c
@@ -511,14 +511,14 @@ static int adjoin(struct dm_table *table, struct dm_target *ti)
* On the other hand, dm-switch needs to process bulk data using messages and
* excessive use of GFP_NOIO could cause trouble.
*/
-static char **realloc_argv(unsigned *array_size, char **old_argv)
+static char **realloc_argv(unsigned *size, char **old_argv)
{
char **argv;
unsigned new_size;
gfp_t gfp;
- if (*array_size) {
- new_size = *array_size * 2;
+ if (*size) {
+ new_size = *size * 2;
gfp = GFP_KERNEL;
} else {
new_size = 8;
@@ -526,8 +526,8 @@ static char **realloc_argv(unsigned *array_size, char **old_argv)
}
argv = kmalloc(new_size * sizeof(*argv), gfp);
if (argv) {
- memcpy(argv, old_argv, *array_size * sizeof(*argv));
- *array_size = new_size;
+ memcpy(argv, old_argv, *size * sizeof(*argv));
+ *size = new_size;
}
kfree(old_argv);
@@ -1495,6 +1495,16 @@ static int queue_supports_sg_merge(struct dm_target *ti, struct dm_dev *dev,
return q && !test_bit(QUEUE_FLAG_NO_SG_MERGE, &q->queue_flags);
}
+static int queue_supports_inline_encryption(struct dm_target *ti,
+ struct dm_dev *dev,
+ sector_t start, sector_t len,
+ void *data)
+{
+ struct request_queue *q = bdev_get_queue(dev->bdev);
+
+ return q && blk_queue_inlinecrypt(q);
+}
+
static bool dm_table_all_devices_attribute(struct dm_table *t,
iterate_devices_callout_fn func)
{
@@ -1615,6 +1625,11 @@ void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,
else
queue_flag_set_unlocked(QUEUE_FLAG_NO_SG_MERGE, q);
+ if (dm_table_all_devices_attribute(t, queue_supports_inline_encryption))
+ queue_flag_set_unlocked(QUEUE_FLAG_INLINECRYPT, q);
+ else
+ queue_flag_clear_unlocked(QUEUE_FLAG_INLINECRYPT, q);
+
dm_table_verify_integrity(t);
/*
@@ -1638,6 +1653,9 @@ void dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,
smp_mb();
if (dm_table_request_based(t))
queue_flag_set_unlocked(QUEUE_FLAG_STACKABLE, q);
+
+ /* io_pages is used for readahead */
+ q->backing_dev_info->io_pages = limits->max_sectors >> (PAGE_SHIFT - 9);
}
unsigned int dm_table_get_num_targets(struct dm_table *t)
diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c
index e976f4f..149fbac 100644
--- a/drivers/md/dm-thin-metadata.c
+++ b/drivers/md/dm-thin-metadata.c
@@ -190,6 +190,12 @@ struct dm_pool_metadata {
sector_t data_block_size;
/*
+ * We reserve a section of the metadata for commit overhead.
+ * All reported space does *not* include this.
+ */
+ dm_block_t metadata_reserve;
+
+ /*
* Set if a transaction has to be aborted but the attempt to roll back
* to the previous (good) transaction failed. The only pool metadata
* operation possible in this state is the closing of the device.
@@ -827,6 +833,20 @@ static int __commit_transaction(struct dm_pool_metadata *pmd)
return dm_tm_commit(pmd->tm, sblock);
}
+static void __set_metadata_reserve(struct dm_pool_metadata *pmd)
+{
+ int r;
+ dm_block_t total;
+ dm_block_t max_blocks = 4096; /* 16M */
+
+ r = dm_sm_get_nr_blocks(pmd->metadata_sm, &total);
+ if (r) {
+ DMERR("could not get size of metadata device");
+ pmd->metadata_reserve = max_blocks;
+ } else
+ pmd->metadata_reserve = min(max_blocks, div_u64(total, 10));
+}
+
struct dm_pool_metadata *dm_pool_metadata_open(struct block_device *bdev,
sector_t data_block_size,
bool format_device)
@@ -860,6 +880,8 @@ struct dm_pool_metadata *dm_pool_metadata_open(struct block_device *bdev,
return ERR_PTR(r);
}
+ __set_metadata_reserve(pmd);
+
return pmd;
}
@@ -1831,6 +1853,13 @@ int dm_pool_get_free_metadata_block_count(struct dm_pool_metadata *pmd,
down_read(&pmd->root_lock);
if (!pmd->fail_io)
r = dm_sm_get_nr_free(pmd->metadata_sm, result);
+
+ if (!r) {
+ if (*result < pmd->metadata_reserve)
+ *result = 0;
+ else
+ *result -= pmd->metadata_reserve;
+ }
up_read(&pmd->root_lock);
return r;
@@ -1943,8 +1972,11 @@ int dm_pool_resize_metadata_dev(struct dm_pool_metadata *pmd, dm_block_t new_cou
int r = -EINVAL;
down_write(&pmd->root_lock);
- if (!pmd->fail_io)
+ if (!pmd->fail_io) {
r = __resize_space_map(pmd->metadata_sm, new_count);
+ if (!r)
+ __set_metadata_reserve(pmd);
+ }
up_write(&pmd->root_lock);
return r;
diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c
index eb419a5..e697283 100644
--- a/drivers/md/dm-thin.c
+++ b/drivers/md/dm-thin.c
@@ -200,7 +200,13 @@ struct dm_thin_new_mapping;
enum pool_mode {
PM_WRITE, /* metadata may be changed */
PM_OUT_OF_DATA_SPACE, /* metadata may be changed, though data may not be allocated */
+
+ /*
+ * Like READ_ONLY, except may switch back to WRITE on metadata resize. Reported as READ_ONLY.
+ */
+ PM_OUT_OF_METADATA_SPACE,
PM_READ_ONLY, /* metadata may not be changed */
+
PM_FAIL, /* all I/O fails */
};
@@ -1384,7 +1390,37 @@ static void schedule_external_copy(struct thin_c *tc, dm_block_t virt_block,
static void set_pool_mode(struct pool *pool, enum pool_mode new_mode);
-static void check_for_space(struct pool *pool)
+static void requeue_bios(struct pool *pool);
+
+static bool is_read_only_pool_mode(enum pool_mode mode)
+{
+ return (mode == PM_OUT_OF_METADATA_SPACE || mode == PM_READ_ONLY);
+}
+
+static bool is_read_only(struct pool *pool)
+{
+ return is_read_only_pool_mode(get_pool_mode(pool));
+}
+
+static void check_for_metadata_space(struct pool *pool)
+{
+ int r;
+ const char *ooms_reason = NULL;
+ dm_block_t nr_free;
+
+ r = dm_pool_get_free_metadata_block_count(pool->pmd, &nr_free);
+ if (r)
+ ooms_reason = "Could not get free metadata blocks";
+ else if (!nr_free)
+ ooms_reason = "No free metadata blocks";
+
+ if (ooms_reason && !is_read_only(pool)) {
+ DMERR("%s", ooms_reason);
+ set_pool_mode(pool, PM_OUT_OF_METADATA_SPACE);
+ }
+}
+
+static void check_for_data_space(struct pool *pool)
{
int r;
dm_block_t nr_free;
@@ -1396,8 +1432,10 @@ static void check_for_space(struct pool *pool)
if (r)
return;
- if (nr_free)
+ if (nr_free) {
set_pool_mode(pool, PM_WRITE);
+ requeue_bios(pool);
+ }
}
/*
@@ -1408,14 +1446,16 @@ static int commit(struct pool *pool)
{
int r;
- if (get_pool_mode(pool) >= PM_READ_ONLY)
+ if (get_pool_mode(pool) >= PM_OUT_OF_METADATA_SPACE)
return -EINVAL;
r = dm_pool_commit_metadata(pool->pmd);
if (r)
metadata_operation_failed(pool, "dm_pool_commit_metadata", r);
- else
- check_for_space(pool);
+ else {
+ check_for_metadata_space(pool);
+ check_for_data_space(pool);
+ }
return r;
}
@@ -1474,10 +1514,26 @@ static int alloc_data_block(struct thin_c *tc, dm_block_t *result)
r = dm_pool_alloc_data_block(pool->pmd, result);
if (r) {
- metadata_operation_failed(pool, "dm_pool_alloc_data_block", r);
+ if (r == -ENOSPC)
+ set_pool_mode(pool, PM_OUT_OF_DATA_SPACE);
+ else
+ metadata_operation_failed(pool, "dm_pool_alloc_data_block", r);
return r;
}
+ r = dm_pool_get_free_metadata_block_count(pool->pmd, &free_blocks);
+ if (r) {
+ metadata_operation_failed(pool, "dm_pool_get_free_metadata_block_count", r);
+ return r;
+ }
+
+ if (!free_blocks) {
+ /* Let's commit before we use up the metadata reserve. */
+ r = commit(pool);
+ if (r)
+ return r;
+ }
+
return 0;
}
@@ -1509,6 +1565,7 @@ static int should_error_unserviceable_bio(struct pool *pool)
case PM_OUT_OF_DATA_SPACE:
return pool->pf.error_if_no_space ? -ENOSPC : 0;
+ case PM_OUT_OF_METADATA_SPACE:
case PM_READ_ONLY:
case PM_FAIL:
return -EIO;
@@ -2472,8 +2529,9 @@ static void set_pool_mode(struct pool *pool, enum pool_mode new_mode)
error_retry_list(pool);
break;
+ case PM_OUT_OF_METADATA_SPACE:
case PM_READ_ONLY:
- if (old_mode != new_mode)
+ if (!is_read_only_pool_mode(old_mode))
notify_of_pool_mode_change(pool, "read-only");
dm_pool_metadata_read_only(pool->pmd);
pool->process_bio = process_bio_read_only;
@@ -2511,6 +2569,8 @@ static void set_pool_mode(struct pool *pool, enum pool_mode new_mode)
case PM_WRITE:
if (old_mode != new_mode)
notify_of_pool_mode_change(pool, "write");
+ if (old_mode == PM_OUT_OF_DATA_SPACE)
+ cancel_delayed_work_sync(&pool->no_space_timeout);
pool->out_of_data_space = false;
pool->pf.error_if_no_space = pt->requested_pf.error_if_no_space;
dm_pool_metadata_read_write(pool->pmd);
@@ -3409,6 +3469,10 @@ static int maybe_resize_metadata_dev(struct dm_target *ti, bool *need_commit)
DMINFO("%s: growing the metadata device from %llu to %llu blocks",
dm_device_name(pool->pool_md),
sb_metadata_dev_size, metadata_dev_size);
+
+ if (get_pool_mode(pool) == PM_OUT_OF_METADATA_SPACE)
+ set_pool_mode(pool, PM_WRITE);
+
r = dm_pool_resize_metadata_dev(pool->pmd, metadata_dev_size);
if (r) {
metadata_operation_failed(pool, "dm_pool_resize_metadata_dev", r);
@@ -3712,7 +3776,7 @@ static int pool_message(struct dm_target *ti, unsigned argc, char **argv)
struct pool_c *pt = ti->private;
struct pool *pool = pt->pool;
- if (get_pool_mode(pool) >= PM_READ_ONLY) {
+ if (get_pool_mode(pool) >= PM_OUT_OF_METADATA_SPACE) {
DMERR("%s: unable to service pool target messages in READ_ONLY or FAIL mode",
dm_device_name(pool->pool_md));
return -EOPNOTSUPP;
@@ -3786,6 +3850,7 @@ static void pool_status(struct dm_target *ti, status_type_t type,
dm_block_t nr_blocks_data;
dm_block_t nr_blocks_metadata;
dm_block_t held_root;
+ enum pool_mode mode;
char buf[BDEVNAME_SIZE];
char buf2[BDEVNAME_SIZE];
struct pool_c *pt = ti->private;
@@ -3856,9 +3921,10 @@ static void pool_status(struct dm_target *ti, status_type_t type,
else
DMEMIT("- ");
- if (pool->pf.mode == PM_OUT_OF_DATA_SPACE)
+ mode = get_pool_mode(pool);
+ if (mode == PM_OUT_OF_DATA_SPACE)
DMEMIT("out_of_data_space ");
- else if (pool->pf.mode == PM_READ_ONLY)
+ else if (is_read_only_pool_mode(mode))
DMEMIT("ro ");
else
DMEMIT("rw ");
diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c
index d96aa84..ac23441 100644
--- a/drivers/md/dm-verity-target.c
+++ b/drivers/md/dm-verity-target.c
@@ -525,7 +525,6 @@ static void verity_prefetch_io(struct work_struct *work)
container_of(work, struct dm_verity_prefetch_work, work);
struct dm_verity *v = pw->v;
int i;
- sector_t prefetch_size;
for (i = v->levels - 2; i >= 0; i--) {
sector_t hash_block_start;
@@ -548,14 +547,8 @@ static void verity_prefetch_io(struct work_struct *work)
hash_block_end = v->hash_blocks - 1;
}
no_prefetch_cluster:
- // for emmc, it is more efficient to send bigger read
- prefetch_size = max((sector_t)CONFIG_DM_VERITY_HASH_PREFETCH_MIN_SIZE,
- hash_block_end - hash_block_start + 1);
- if ((hash_block_start + prefetch_size) >= (v->hash_start + v->hash_blocks)) {
- prefetch_size = hash_block_end - hash_block_start + 1;
- }
dm_bufio_prefetch(v->bufio, hash_block_start,
- prefetch_size);
+ hash_block_end - hash_block_start + 1);
}
kfree(pw);
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index fcc2b57..e870b09 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -302,15 +302,6 @@ static void recover_bitmaps(struct md_thread *thread)
while (cinfo->recovery_map) {
slot = fls64((u64)cinfo->recovery_map) - 1;
- /* Clear suspend_area associated with the bitmap */
- spin_lock_irq(&cinfo->suspend_lock);
- list_for_each_entry_safe(s, tmp, &cinfo->suspend_list, list)
- if (slot == s->slot) {
- list_del(&s->list);
- kfree(s);
- }
- spin_unlock_irq(&cinfo->suspend_lock);
-
snprintf(str, 64, "bitmap%04d", slot);
bm_lockres = lockres_init(mddev, str, NULL, 1);
if (!bm_lockres) {
@@ -329,6 +320,16 @@ static void recover_bitmaps(struct md_thread *thread)
pr_err("md-cluster: Could not copy data from bitmap %d\n", slot);
goto clear_bit;
}
+
+ /* Clear suspend_area associated with the bitmap */
+ spin_lock_irq(&cinfo->suspend_lock);
+ list_for_each_entry_safe(s, tmp, &cinfo->suspend_list, list)
+ if (slot == s->slot) {
+ list_del(&s->list);
+ kfree(s);
+ }
+ spin_unlock_irq(&cinfo->suspend_lock);
+
if (hi > 0) {
if (lo < mddev->recovery_cp)
mddev->recovery_cp = lo;
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 93059dd..bf2d2df 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -2694,7 +2694,8 @@ state_store(struct md_rdev *rdev, const char *buf, size_t len)
err = 0;
}
} else if (cmd_match(buf, "re-add")) {
- if (test_bit(Faulty, &rdev->flags) && (rdev->raid_disk == -1)) {
+ if (test_bit(Faulty, &rdev->flags) && (rdev->raid_disk == -1) &&
+ rdev->saved_raid_disk >= 0) {
/* clear_bit is performed _after_ all the devices
* have their local Faulty bit cleared. If any writes
* happen in the meantime in the local node, they
@@ -6191,6 +6192,9 @@ static int hot_remove_disk(struct mddev *mddev, dev_t dev)
char b[BDEVNAME_SIZE];
struct md_rdev *rdev;
+ if (!mddev->pers)
+ return -ENODEV;
+
rdev = find_rdev(mddev, dev);
if (!rdev)
return -ENXIO;
@@ -8200,6 +8204,19 @@ void md_do_sync(struct md_thread *thread)
set_mask_bits(&mddev->flags, 0,
BIT(MD_CHANGE_PENDING) | BIT(MD_CHANGE_DEVS));
+ if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) &&
+ !test_bit(MD_RECOVERY_INTR, &mddev->recovery) &&
+ mddev->delta_disks > 0 &&
+ mddev->pers->finish_reshape &&
+ mddev->pers->size &&
+ mddev->queue) {
+ mddev_lock_nointr(mddev);
+ md_set_array_sectors(mddev, mddev->pers->size(mddev, 0, 0));
+ mddev_unlock(mddev);
+ set_capacity(mddev->gendisk, mddev->array_sectors);
+ revalidate_disk(mddev->gendisk);
+ }
+
spin_lock(&mddev->lock);
if (!test_bit(MD_RECOVERY_INTR, &mddev->recovery)) {
/* We completed so min/max setting can be forgotten if used. */
@@ -8259,6 +8276,7 @@ static int remove_and_add_spares(struct mddev *mddev,
if (mddev->pers->hot_remove_disk(
mddev, rdev) == 0) {
sysfs_unlink_rdev(mddev, rdev);
+ rdev->saved_raid_disk = rdev->raid_disk;
rdev->raid_disk = -1;
removed++;
}
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 208fbf7..b2bed72 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1589,6 +1589,7 @@ static int raid1_add_disk(struct mddev *mddev, struct md_rdev *rdev)
*/
if (rdev->saved_raid_disk >= 0 &&
rdev->saved_raid_disk >= first &&
+ rdev->saved_raid_disk < conf->raid_disks &&
conf->mirrors[rdev->saved_raid_disk].rdev == NULL)
first = last = rdev->saved_raid_disk;
@@ -1673,6 +1674,17 @@ static int raid1_remove_disk(struct mddev *mddev, struct md_rdev *rdev)
struct md_rdev *repl =
conf->mirrors[conf->raid_disks + number].rdev;
freeze_array(conf, 0);
+ if (atomic_read(&repl->nr_pending)) {
+ /* It means that some queued IO of retry_list
+ * hold repl. Thus, we cannot set replacement
+ * as NULL, avoiding rdev NULL pointer
+ * dereference in sync_request_write and
+ * handle_write_finished.
+ */
+ err = -EBUSY;
+ unfreeze_array(conf);
+ goto abort;
+ }
clear_bit(Replacement, &repl->flags);
p->rdev = repl;
conf->mirrors[conf->raid_disks + number].rdev = NULL;
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 2b04c720..f1ad80b 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1734,6 +1734,7 @@ static int raid10_add_disk(struct mddev *mddev, struct md_rdev *rdev)
first = last = rdev->raid_disk;
if (rdev->saved_raid_disk >= first &&
+ rdev->saved_raid_disk < conf->geo.raid_disks &&
conf->mirrors[rdev->saved_raid_disk].rdev == NULL)
mirror = rdev->saved_raid_disk;
else
@@ -2636,7 +2637,8 @@ static void handle_write_completed(struct r10conf *conf, struct r10bio *r10_bio)
for (m = 0; m < conf->copies; m++) {
int dev = r10_bio->devs[m].devnum;
rdev = conf->mirrors[dev].rdev;
- if (r10_bio->devs[m].bio == NULL)
+ if (r10_bio->devs[m].bio == NULL ||
+ r10_bio->devs[m].bio->bi_end_io == NULL)
continue;
if (!r10_bio->devs[m].bio->bi_error) {
rdev_clear_badblocks(
@@ -2651,7 +2653,8 @@ static void handle_write_completed(struct r10conf *conf, struct r10bio *r10_bio)
md_error(conf->mddev, rdev);
}
rdev = conf->mirrors[dev].replacement;
- if (r10_bio->devs[m].repl_bio == NULL)
+ if (r10_bio->devs[m].repl_bio == NULL ||
+ r10_bio->devs[m].repl_bio->bi_end_io == NULL)
continue;
if (!r10_bio->devs[m].repl_bio->bi_error) {
@@ -3732,6 +3735,13 @@ static int raid10_run(struct mddev *mddev)
disk->rdev->saved_raid_disk < 0)
conf->fullsync = 1;
}
+
+ if (disk->replacement &&
+ !test_bit(In_sync, &disk->replacement->flags) &&
+ disk->replacement->saved_raid_disk < 0) {
+ conf->fullsync = 1;
+ }
+
disk->recovery_disabled = mddev->recovery_disabled - 1;
}
@@ -4372,11 +4382,12 @@ static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr,
allow_barrier(conf);
}
+ raise_barrier(conf, 0);
read_more:
/* Now schedule reads for blocks from sector_nr to last */
r10_bio = mempool_alloc(conf->r10buf_pool, GFP_NOIO);
r10_bio->state = 0;
- raise_barrier(conf, sectors_done != 0);
+ raise_barrier(conf, 1);
atomic_set(&r10_bio->remaining, 0);
r10_bio->mddev = mddev;
r10_bio->sector = sector_nr;
@@ -4483,6 +4494,8 @@ static sector_t reshape_request(struct mddev *mddev, sector_t sector_nr,
if (sector_nr <= last)
goto read_more;
+ lower_barrier(conf);
+
/* Now that we have done the whole section we can
* update reshape_progress
*/
@@ -4682,17 +4695,11 @@ static void raid10_finish_reshape(struct mddev *mddev)
return;
if (mddev->delta_disks > 0) {
- sector_t size = raid10_size(mddev, 0, 0);
- md_set_array_sectors(mddev, size);
if (mddev->recovery_cp > mddev->resync_max_sectors) {
mddev->recovery_cp = mddev->resync_max_sectors;
set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
}
- mddev->resync_max_sectors = size;
- if (mddev->queue) {
- set_capacity(mddev->gendisk, mddev->array_sectors);
- revalidate_disk(mddev->gendisk);
- }
+ mddev->resync_max_sectors = mddev->array_sectors;
} else {
int d;
rcu_read_lock();
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index a7549a4..d15e29d 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -2049,15 +2049,16 @@ static int grow_one_stripe(struct r5conf *conf, gfp_t gfp)
static int grow_stripes(struct r5conf *conf, int num)
{
struct kmem_cache *sc;
+ size_t namelen = sizeof(conf->cache_name[0]);
int devs = max(conf->raid_disks, conf->previous_raid_disks);
if (conf->mddev->gendisk)
- sprintf(conf->cache_name[0],
+ snprintf(conf->cache_name[0], namelen,
"raid%d-%s", conf->level, mdname(conf->mddev));
else
- sprintf(conf->cache_name[0],
+ snprintf(conf->cache_name[0], namelen,
"raid%d-%p", conf->level, conf->mddev);
- sprintf(conf->cache_name[1], "%s-alt", conf->cache_name[0]);
+ snprintf(conf->cache_name[1], namelen, "%.27s-alt", conf->cache_name[0]);
conf->active_name = 0;
sc = kmem_cache_create(conf->cache_name[conf->active_name],
@@ -4206,6 +4207,12 @@ static void analyse_stripe(struct stripe_head *sh, struct stripe_head_state *s)
s->failed++;
if (rdev && !test_bit(Faulty, &rdev->flags))
do_recovery = 1;
+ else if (!rdev) {
+ rdev = rcu_dereference(
+ conf->disks[i].replacement);
+ if (rdev && !test_bit(Faulty, &rdev->flags))
+ do_recovery = 1;
+ }
}
}
if (test_bit(STRIPE_SYNCING, &sh->state)) {
@@ -7614,13 +7621,7 @@ static void raid5_finish_reshape(struct mddev *mddev)
if (!test_bit(MD_RECOVERY_INTR, &mddev->recovery)) {
- if (mddev->delta_disks > 0) {
- md_set_array_sectors(mddev, raid5_size(mddev, 0, 0));
- if (mddev->queue) {
- set_capacity(mddev->gendisk, mddev->array_sectors);
- revalidate_disk(mddev->gendisk);
- }
- } else {
+ if (mddev->delta_disks <= 0) {
int d;
spin_lock_irq(&conf->device_lock);
mddev->degraded = calc_degraded(conf);
diff --git a/drivers/media/common/siano/smsendian.c b/drivers/media/common/siano/smsendian.c
index bfe831c..b95a631 100644
--- a/drivers/media/common/siano/smsendian.c
+++ b/drivers/media/common/siano/smsendian.c
@@ -35,7 +35,7 @@ void smsendian_handle_tx_message(void *buffer)
switch (msg->x_msg_header.msg_type) {
case MSG_SMS_DATA_DOWNLOAD_REQ:
{
- msg->msg_data[0] = le32_to_cpu(msg->msg_data[0]);
+ msg->msg_data[0] = le32_to_cpu((__force __le32)(msg->msg_data[0]));
break;
}
@@ -44,7 +44,7 @@ void smsendian_handle_tx_message(void *buffer)
sizeof(struct sms_msg_hdr))/4;
for (i = 0; i < msg_words; i++)
- msg->msg_data[i] = le32_to_cpu(msg->msg_data[i]);
+ msg->msg_data[i] = le32_to_cpu((__force __le32)msg->msg_data[i]);
break;
}
@@ -64,7 +64,7 @@ void smsendian_handle_rx_message(void *buffer)
{
struct sms_version_res *ver =
(struct sms_version_res *) msg;
- ver->chip_model = le16_to_cpu(ver->chip_model);
+ ver->chip_model = le16_to_cpu((__force __le16)ver->chip_model);
break;
}
@@ -81,7 +81,7 @@ void smsendian_handle_rx_message(void *buffer)
sizeof(struct sms_msg_hdr))/4;
for (i = 0; i < msg_words; i++)
- msg->msg_data[i] = le32_to_cpu(msg->msg_data[i]);
+ msg->msg_data[i] = le32_to_cpu((__force __le32)msg->msg_data[i]);
break;
}
@@ -95,9 +95,9 @@ void smsendian_handle_message_header(void *msg)
#ifdef __BIG_ENDIAN
struct sms_msg_hdr *phdr = (struct sms_msg_hdr *)msg;
- phdr->msg_type = le16_to_cpu(phdr->msg_type);
- phdr->msg_length = le16_to_cpu(phdr->msg_length);
- phdr->msg_flags = le16_to_cpu(phdr->msg_flags);
+ phdr->msg_type = le16_to_cpu((__force __le16)phdr->msg_type);
+ phdr->msg_length = le16_to_cpu((__force __le16)phdr->msg_length);
+ phdr->msg_flags = le16_to_cpu((__force __le16)phdr->msg_flags);
#endif /* __BIG_ENDIAN */
}
EXPORT_SYMBOL_GPL(smsendian_handle_message_header);
diff --git a/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c b/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
index 1684810..d2f72f3 100644
--- a/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
+++ b/drivers/media/common/v4l2-tpg/v4l2-tpg-core.c
@@ -1618,7 +1618,7 @@ typedef struct { u16 __; u8 _; } __packed x24;
unsigned s; \
\
for (s = 0; s < len; s++) { \
- u8 chr = font8x16[text[s] * 16 + line]; \
+ u8 chr = font8x16[(u8)text[s] * 16 + line]; \
\
if (hdiv == 2 && tpg->hflip) { \
pos[3] = (chr & (0x01 << 6) ? fg : bg); \
@@ -1650,7 +1650,7 @@ typedef struct { u16 __; u8 _; } __packed x24;
pos[7] = (chr & (0x01 << 0) ? fg : bg); \
} \
\
- pos += (tpg->hflip ? -8 : 8) / hdiv; \
+ pos += (tpg->hflip ? -8 : 8) / (int)hdiv; \
} \
} \
} while (0)
diff --git a/drivers/media/dvb-core/dvb_frontend.c b/drivers/media/dvb-core/dvb_frontend.c
index 01511e5..2f054db 100644
--- a/drivers/media/dvb-core/dvb_frontend.c
+++ b/drivers/media/dvb-core/dvb_frontend.c
@@ -251,8 +251,20 @@ static void dvb_frontend_add_event(struct dvb_frontend *fe,
wake_up_interruptible (&events->wait_queue);
}
+static int dvb_frontend_test_event(struct dvb_frontend_private *fepriv,
+ struct dvb_fe_events *events)
+{
+ int ret;
+
+ up(&fepriv->sem);
+ ret = events->eventw != events->eventr;
+ down(&fepriv->sem);
+
+ return ret;
+}
+
static int dvb_frontend_get_event(struct dvb_frontend *fe,
- struct dvb_frontend_event *event, int flags)
+ struct dvb_frontend_event *event, int flags)
{
struct dvb_frontend_private *fepriv = fe->frontend_priv;
struct dvb_fe_events *events = &fepriv->events;
@@ -270,13 +282,8 @@ static int dvb_frontend_get_event(struct dvb_frontend *fe,
if (flags & O_NONBLOCK)
return -EWOULDBLOCK;
- up(&fepriv->sem);
-
- ret = wait_event_interruptible (events->wait_queue,
- events->eventw != events->eventr);
-
- if (down_interruptible (&fepriv->sem))
- return -ERESTARTSYS;
+ ret = wait_event_interruptible(events->wait_queue,
+ dvb_frontend_test_event(fepriv, events));
if (ret < 0)
return ret;
diff --git a/drivers/media/dvb-frontends/helene.c b/drivers/media/dvb-frontends/helene.c
index e06bcd4..800f386 100644
--- a/drivers/media/dvb-frontends/helene.c
+++ b/drivers/media/dvb-frontends/helene.c
@@ -898,7 +898,10 @@ static int helene_x_pon(struct helene_priv *priv)
helene_write_regs(priv, 0x99, cdata, sizeof(cdata));
/* 0x81 - 0x94 */
- data[0] = 0x18; /* xtal 24 MHz */
+ if (priv->xtal == SONY_HELENE_XTAL_16000)
+ data[0] = 0x10; /* xtal 16 MHz */
+ else
+ data[0] = 0x18; /* xtal 24 MHz */
data[1] = (uint8_t)(0x80 | (0x04 & 0x1F)); /* 4 x 25 = 100uA */
data[2] = (uint8_t)(0x80 | (0x26 & 0x7F)); /* 38 x 0.25 = 9.5pF */
data[3] = 0x80; /* REFOUT signal output 500mVpp */
diff --git a/drivers/media/i2c/cx25840/cx25840-core.c b/drivers/media/i2c/cx25840/cx25840-core.c
index d558ed3..cc56660 100644
--- a/drivers/media/i2c/cx25840/cx25840-core.c
+++ b/drivers/media/i2c/cx25840/cx25840-core.c
@@ -467,8 +467,13 @@ static void cx23885_initialize(struct i2c_client *client)
{
DEFINE_WAIT(wait);
struct cx25840_state *state = to_state(i2c_get_clientdata(client));
+ u32 clk_freq = 0;
struct workqueue_struct *q;
+ /* cx23885 sets hostdata to clk_freq pointer */
+ if (v4l2_get_subdev_hostdata(&state->sd))
+ clk_freq = *((u32 *)v4l2_get_subdev_hostdata(&state->sd));
+
/*
* Come out of digital power down
* The CX23888, at least, needs this, otherwise registers aside from
@@ -504,8 +509,13 @@ static void cx23885_initialize(struct i2c_client *client)
* 50.0 MHz * (0xb + 0xe8ba26/0x2000000)/4 = 5 * 28.636363 MHz
* 572.73 MHz before post divide
*/
- /* HVR1850 or 50MHz xtal */
- cx25840_write(client, 0x2, 0x71);
+ if (clk_freq == 25000000) {
+ /* 888/ImpactVCBe or 25Mhz xtal */
+ ; /* nothing to do */
+ } else {
+ /* HVR1850 or 50MHz xtal */
+ cx25840_write(client, 0x2, 0x71);
+ }
cx25840_write4(client, 0x11c, 0x01d1744c);
cx25840_write4(client, 0x118, 0x00000416);
cx25840_write4(client, 0x404, 0x0010253e);
@@ -548,9 +558,15 @@ static void cx23885_initialize(struct i2c_client *client)
/* HVR1850 */
switch (state->id) {
case CX23888_AV:
- /* 888/HVR1250 specific */
- cx25840_write4(client, 0x10c, 0x13333333);
- cx25840_write4(client, 0x108, 0x00000515);
+ if (clk_freq == 25000000) {
+ /* 888/ImpactVCBe or 25MHz xtal */
+ cx25840_write4(client, 0x10c, 0x01b6db7b);
+ cx25840_write4(client, 0x108, 0x00000512);
+ } else {
+ /* 888/HVR1250 or 50MHz xtal */
+ cx25840_write4(client, 0x10c, 0x13333333);
+ cx25840_write4(client, 0x108, 0x00000515);
+ }
break;
default:
cx25840_write4(client, 0x10c, 0x002be2c9);
@@ -580,7 +596,7 @@ static void cx23885_initialize(struct i2c_client *client)
* 368.64 MHz before post divide
* 122.88 MHz / 0xa = 12.288 MHz
*/
- /* HVR1850 or 50MHz xtal */
+ /* HVR1850 or 50MHz xtal or 25MHz xtal */
cx25840_write4(client, 0x114, 0x017dbf48);
cx25840_write4(client, 0x110, 0x000a030e);
break;
diff --git a/drivers/media/i2c/smiapp/smiapp-core.c b/drivers/media/i2c/smiapp/smiapp-core.c
index 44f8c7e..8ffa13f 100644
--- a/drivers/media/i2c/smiapp/smiapp-core.c
+++ b/drivers/media/i2c/smiapp/smiapp-core.c
@@ -991,7 +991,7 @@ static int smiapp_read_nvm(struct smiapp_sensor *sensor,
if (rval)
goto out;
- for (i = 0; i < 1000; i++) {
+ for (i = 1000; i > 0; i--) {
rval = smiapp_read(
sensor,
SMIAPP_REG_U8_DATA_TRANSFER_IF_1_STATUS, &s);
@@ -1002,11 +1002,10 @@ static int smiapp_read_nvm(struct smiapp_sensor *sensor,
if (s & SMIAPP_DATA_TRANSFER_IF_1_STATUS_RD_READY)
break;
- if (--i == 0) {
- rval = -ETIMEDOUT;
- goto out;
- }
-
+ }
+ if (!i) {
+ rval = -ETIMEDOUT;
+ goto out;
}
for (i = 0; i < SMIAPP_NVM_PAGE_SIZE; i++) {
diff --git a/drivers/media/i2c/soc_camera/ov772x.c b/drivers/media/i2c/soc_camera/ov772x.c
index 7e68762..fa1cb24 100644
--- a/drivers/media/i2c/soc_camera/ov772x.c
+++ b/drivers/media/i2c/soc_camera/ov772x.c
@@ -834,7 +834,7 @@ static int ov772x_set_params(struct ov772x_priv *priv,
* set COM8
*/
if (priv->band_filter) {
- ret = ov772x_mask_set(client, COM8, BNDF_ON_OFF, 1);
+ ret = ov772x_mask_set(client, COM8, BNDF_ON_OFF, BNDF_ON_OFF);
if (!ret)
ret = ov772x_mask_set(client, BDBASE,
0xff, 256 - priv->band_filter);
diff --git a/drivers/media/i2c/tvp5150.c b/drivers/media/i2c/tvp5150.c
index 59aa4da..e83f811 100644
--- a/drivers/media/i2c/tvp5150.c
+++ b/drivers/media/i2c/tvp5150.c
@@ -897,9 +897,6 @@ static int tvp5150_set_selection(struct v4l2_subdev *sd,
/* tvp5150 has some special limits */
rect.left = clamp(rect.left, 0, TVP5150_MAX_CROP_LEFT);
- rect.width = clamp_t(unsigned int, rect.width,
- TVP5150_H_MAX - TVP5150_MAX_CROP_LEFT - rect.left,
- TVP5150_H_MAX - rect.left);
rect.top = clamp(rect.top, 0, TVP5150_MAX_CROP_TOP);
/* Calculate height based on current standard */
@@ -913,9 +910,16 @@ static int tvp5150_set_selection(struct v4l2_subdev *sd,
else
hmax = TVP5150_V_MAX_OTHERS;
- rect.height = clamp_t(unsigned int, rect.height,
+ /*
+ * alignments:
+ * - width = 2 due to UYVY colorspace
+ * - height, image = no special alignment
+ */
+ v4l_bound_align_image(&rect.width,
+ TVP5150_H_MAX - TVP5150_MAX_CROP_LEFT - rect.left,
+ TVP5150_H_MAX - rect.left, 1, &rect.height,
hmax - TVP5150_MAX_CROP_TOP - rect.top,
- hmax - rect.top);
+ hmax - rect.top, 0, 0);
tvp5150_write(sd, TVP5150_VERT_BLANKING_START, rect.top);
tvp5150_write(sd, TVP5150_VERT_BLANKING_STOP,
@@ -1527,7 +1531,7 @@ static int tvp5150_probe(struct i2c_client *c,
27000000, 1, 27000000);
v4l2_ctrl_new_std_menu_items(&core->hdl, &tvp5150_ctrl_ops,
V4L2_CID_TEST_PATTERN,
- ARRAY_SIZE(tvp5150_test_patterns),
+ ARRAY_SIZE(tvp5150_test_patterns) - 1,
0, 0, tvp5150_test_patterns);
sd->ctrl_handler = &core->hdl;
if (core->hdl.error) {
diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
index cc8de56..6f46c59 100644
--- a/drivers/media/media-device.c
+++ b/drivers/media/media-device.c
@@ -60,7 +60,7 @@ static int media_device_close(struct file *filp)
static long media_device_get_info(struct media_device *dev, void *arg)
{
- struct media_device_info *info = (struct media_device_info *)arg;
+ struct media_device_info *info = arg;
memset(info, 0, sizeof(*info));
@@ -100,7 +100,7 @@ static struct media_entity *find_entity(struct media_device *mdev, u32 id)
static long media_device_enum_entities(struct media_device *mdev, void *arg)
{
- struct media_entity_desc *entd = (struct media_entity_desc *)arg;
+ struct media_entity_desc *entd = arg;
struct media_entity *ent;
ent = find_entity(mdev, entd->id);
@@ -153,7 +153,7 @@ static void media_device_kpad_to_upad(const struct media_pad *kpad,
static long media_device_enum_links(struct media_device *mdev, void *arg)
{
- struct media_links_enum *links = (struct media_links_enum *)arg;
+ struct media_links_enum *links = arg;
struct media_entity *entity;
entity = find_entity(mdev, links->entity);
@@ -201,7 +201,7 @@ static long media_device_enum_links(struct media_device *mdev, void *arg)
static long media_device_setup_link(struct media_device *mdev, void *arg)
{
- struct media_link_desc *linkd = (struct media_link_desc *)arg;
+ struct media_link_desc *linkd = arg;
struct media_link *link = NULL;
struct media_entity *source;
struct media_entity *sink;
@@ -229,7 +229,7 @@ static long media_device_setup_link(struct media_device *mdev, void *arg)
static long media_device_get_topology(struct media_device *mdev, void *arg)
{
- struct media_v2_topology *topo = (struct media_v2_topology *)arg;
+ struct media_v2_topology *topo = arg;
struct media_entity *entity;
struct media_interface *intf;
struct media_pad *pad;
diff --git a/drivers/media/pci/cx23885/altera-ci.c b/drivers/media/pci/cx23885/altera-ci.c
index aaf4e46..a0c1ff9 100644
--- a/drivers/media/pci/cx23885/altera-ci.c
+++ b/drivers/media/pci/cx23885/altera-ci.c
@@ -660,6 +660,10 @@ static int altera_hw_filt_init(struct altera_ci_config *config, int hw_filt_nr)
}
temp_int = append_internal(inter);
+ if (!temp_int) {
+ ret = -ENOMEM;
+ goto err;
+ }
inter->filts_used = 1;
inter->dev = config->dev;
inter->fpga_rw = config->fpga_rw;
@@ -694,6 +698,7 @@ static int altera_hw_filt_init(struct altera_ci_config *config, int hw_filt_nr)
__func__, ret);
kfree(pid_filt);
+ kfree(inter);
return ret;
}
@@ -728,6 +733,10 @@ int altera_ci_init(struct altera_ci_config *config, int ci_nr)
}
temp_int = append_internal(inter);
+ if (!temp_int) {
+ ret = -ENOMEM;
+ goto err;
+ }
inter->cis_used = 1;
inter->dev = config->dev;
inter->fpga_rw = config->fpga_rw;
@@ -796,6 +805,7 @@ int altera_ci_init(struct altera_ci_config *config, int ci_nr)
ci_dbg_print("%s: Cannot initialize CI: Error %d.\n", __func__, ret);
kfree(state);
+ kfree(inter);
return ret;
}
diff --git a/drivers/media/pci/cx23885/cx23885-cards.c b/drivers/media/pci/cx23885/cx23885-cards.c
index 99ba8d6..427ece1 100644
--- a/drivers/media/pci/cx23885/cx23885-cards.c
+++ b/drivers/media/pci/cx23885/cx23885-cards.c
@@ -2282,6 +2282,10 @@ void cx23885_card_setup(struct cx23885_dev *dev)
&dev->i2c_bus[2].i2c_adap,
"cx25840", 0x88 >> 1, NULL);
if (dev->sd_cx25840) {
+ /* set host data for clk_freq configuration */
+ v4l2_set_subdev_hostdata(dev->sd_cx25840,
+ &dev->clk_freq);
+
dev->sd_cx25840->grp_id = CX23885_HW_AV_CORE;
v4l2_subdev_call(dev->sd_cx25840, core, load_fw);
}
diff --git a/drivers/media/pci/cx23885/cx23885-core.c b/drivers/media/pci/cx23885/cx23885-core.c
index c86b109..dcbb3a2 100644
--- a/drivers/media/pci/cx23885/cx23885-core.c
+++ b/drivers/media/pci/cx23885/cx23885-core.c
@@ -872,6 +872,16 @@ static int cx23885_dev_setup(struct cx23885_dev *dev)
if (cx23885_boards[dev->board].clk_freq > 0)
dev->clk_freq = cx23885_boards[dev->board].clk_freq;
+ if (dev->board == CX23885_BOARD_HAUPPAUGE_IMPACTVCBE &&
+ dev->pci->subsystem_device == 0x7137) {
+ /* Hauppauge ImpactVCBe device ID 0x7137 is populated
+ * with an 888, and a 25Mhz crystal, instead of the
+ * usual third overtone 50Mhz. The default clock rate must
+ * be overridden so the cx25840 is properly configured
+ */
+ dev->clk_freq = 25000000;
+ }
+
dev->pci_bus = dev->pci->bus->number;
dev->pci_slot = PCI_SLOT(dev->pci->devfn);
cx23885_irq_add(dev, 0x001f00);
diff --git a/drivers/media/pci/cx25821/cx25821-core.c b/drivers/media/pci/cx25821/cx25821-core.c
index 9a5f912..0d4cacb 100644
--- a/drivers/media/pci/cx25821/cx25821-core.c
+++ b/drivers/media/pci/cx25821/cx25821-core.c
@@ -871,6 +871,10 @@ static int cx25821_dev_setup(struct cx25821_dev *dev)
dev->nr = ++cx25821_devcount;
sprintf(dev->name, "cx25821[%d]", dev->nr);
+ if (dev->nr >= ARRAY_SIZE(card)) {
+ CX25821_INFO("dev->nr >= %zd", ARRAY_SIZE(card));
+ return -ENODEV;
+ }
if (dev->pci->device != 0x8210) {
pr_info("%s(): Exiting. Incorrect Hardware device = 0x%02x\n",
__func__, dev->pci->device);
@@ -886,9 +890,6 @@ static int cx25821_dev_setup(struct cx25821_dev *dev)
dev->channels[i].sram_channels = &cx25821_sram_channels[i];
}
- if (dev->nr > 1)
- CX25821_INFO("dev->nr > 1!");
-
/* board config */
dev->board = 1; /* card[dev->nr]; */
dev->_max_num_decoders = MAX_DECODERS;
diff --git a/drivers/media/pci/saa7164/saa7164-fw.c b/drivers/media/pci/saa7164/saa7164-fw.c
index 269e078..93d5319 100644
--- a/drivers/media/pci/saa7164/saa7164-fw.c
+++ b/drivers/media/pci/saa7164/saa7164-fw.c
@@ -430,7 +430,8 @@ int saa7164_downloadfirmware(struct saa7164_dev *dev)
__func__, fw->size);
if (fw->size != fwlength) {
- printk(KERN_ERR "xc5000: firmware incorrect size\n");
+ printk(KERN_ERR "saa7164: firmware incorrect size %zu != %u\n",
+ fw->size, fwlength);
ret = -ENOMEM;
goto out;
}
diff --git a/drivers/media/pci/tw686x/tw686x-video.c b/drivers/media/pci/tw686x/tw686x-video.c
index c3fafa9..3a06c00 100644
--- a/drivers/media/pci/tw686x/tw686x-video.c
+++ b/drivers/media/pci/tw686x/tw686x-video.c
@@ -1190,6 +1190,14 @@ int tw686x_video_init(struct tw686x_dev *dev)
return err;
}
+ /* Initialize vc->dev and vc->ch for the error path */
+ for (ch = 0; ch < max_channels(dev); ch++) {
+ struct tw686x_video_channel *vc = &dev->video_channels[ch];
+
+ vc->dev = dev;
+ vc->ch = ch;
+ }
+
for (ch = 0; ch < max_channels(dev); ch++) {
struct tw686x_video_channel *vc = &dev->video_channels[ch];
struct video_device *vdev;
@@ -1198,9 +1206,6 @@ int tw686x_video_init(struct tw686x_dev *dev)
spin_lock_init(&vc->qlock);
INIT_LIST_HEAD(&vc->vidq_queued);
- vc->dev = dev;
- vc->ch = ch;
-
/* default settings */
err = tw686x_set_standard(vc, V4L2_STD_NTSC);
if (err)
@@ -1228,7 +1233,8 @@ int tw686x_video_init(struct tw686x_dev *dev)
vc->vidq.timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_MONOTONIC;
vc->vidq.min_buffers_needed = 2;
vc->vidq.lock = &vc->vb_mutex;
- vc->vidq.gfp_flags = GFP_DMA32;
+ vc->vidq.gfp_flags = dev->dma_mode != TW686X_DMA_MODE_MEMCPY ?
+ GFP_DMA32 : 0;
vc->vidq.dev = &dev->pci_dev->dev;
err = vb2_queue_init(&vc->vidq);
diff --git a/drivers/media/platform/exynos4-is/fimc-isp-video.c b/drivers/media/platform/exynos4-is/fimc-isp-video.c
index 400ce0c..e00fa03 100644
--- a/drivers/media/platform/exynos4-is/fimc-isp-video.c
+++ b/drivers/media/platform/exynos4-is/fimc-isp-video.c
@@ -384,12 +384,17 @@ static void __isp_video_try_fmt(struct fimc_isp *isp,
struct v4l2_pix_format_mplane *pixm,
const struct fimc_fmt **fmt)
{
- *fmt = fimc_isp_find_format(&pixm->pixelformat, NULL, 2);
+ const struct fimc_fmt *__fmt;
+
+ __fmt = fimc_isp_find_format(&pixm->pixelformat, NULL, 2);
+
+ if (fmt)
+ *fmt = __fmt;
pixm->colorspace = V4L2_COLORSPACE_SRGB;
pixm->field = V4L2_FIELD_NONE;
- pixm->num_planes = (*fmt)->memplanes;
- pixm->pixelformat = (*fmt)->fourcc;
+ pixm->num_planes = __fmt->memplanes;
+ pixm->pixelformat = __fmt->fourcc;
/*
* TODO: double check with the docmentation these width/height
* constraints are correct.
diff --git a/drivers/media/platform/fsl-viu.c b/drivers/media/platform/fsl-viu.c
index ae8c6b3..7f0ed5a 100644
--- a/drivers/media/platform/fsl-viu.c
+++ b/drivers/media/platform/fsl-viu.c
@@ -1417,7 +1417,7 @@ static int viu_of_probe(struct platform_device *op)
sizeof(struct viu_reg), DRV_NAME)) {
dev_err(&op->dev, "Error while requesting mem region\n");
ret = -EBUSY;
- goto err;
+ goto err_irq;
}
/* remap registers */
@@ -1425,7 +1425,7 @@ static int viu_of_probe(struct platform_device *op)
if (!viu_regs) {
dev_err(&op->dev, "Can't map register set\n");
ret = -ENOMEM;
- goto err;
+ goto err_irq;
}
/* Prepare our private structure */
@@ -1433,7 +1433,7 @@ static int viu_of_probe(struct platform_device *op)
if (!viu_dev) {
dev_err(&op->dev, "Can't allocate private structure\n");
ret = -ENOMEM;
- goto err;
+ goto err_irq;
}
viu_dev->vr = viu_regs;
@@ -1449,16 +1449,21 @@ static int viu_of_probe(struct platform_device *op)
ret = v4l2_device_register(viu_dev->dev, &viu_dev->v4l2_dev);
if (ret < 0) {
dev_err(&op->dev, "v4l2_device_register() failed: %d\n", ret);
- goto err;
+ goto err_irq;
}
ad = i2c_get_adapter(0);
+ if (!ad) {
+ ret = -EFAULT;
+ dev_err(&op->dev, "couldn't get i2c adapter\n");
+ goto err_v4l2;
+ }
v4l2_ctrl_handler_init(&viu_dev->hdl, 5);
if (viu_dev->hdl.error) {
ret = viu_dev->hdl.error;
dev_err(&op->dev, "couldn't register control\n");
- goto err_vdev;
+ goto err_i2c;
}
/* This control handler will inherit the control(s) from the
sub-device(s). */
@@ -1476,7 +1481,7 @@ static int viu_of_probe(struct platform_device *op)
vdev = video_device_alloc();
if (vdev == NULL) {
ret = -ENOMEM;
- goto err_vdev;
+ goto err_hdl;
}
*vdev = viu_template;
@@ -1497,7 +1502,7 @@ static int viu_of_probe(struct platform_device *op)
ret = video_register_device(viu_dev->vdev, VFL_TYPE_GRABBER, -1);
if (ret < 0) {
video_device_release(viu_dev->vdev);
- goto err_vdev;
+ goto err_unlock;
}
/* enable VIU clock */
@@ -1505,12 +1510,12 @@ static int viu_of_probe(struct platform_device *op)
if (IS_ERR(clk)) {
dev_err(&op->dev, "failed to lookup the clock!\n");
ret = PTR_ERR(clk);
- goto err_clk;
+ goto err_vdev;
}
ret = clk_prepare_enable(clk);
if (ret) {
dev_err(&op->dev, "failed to enable the clock!\n");
- goto err_clk;
+ goto err_vdev;
}
viu_dev->clk = clk;
@@ -1521,7 +1526,7 @@ static int viu_of_probe(struct platform_device *op)
if (request_irq(viu_dev->irq, viu_intr, 0, "viu", (void *)viu_dev)) {
dev_err(&op->dev, "Request VIU IRQ failed.\n");
ret = -ENODEV;
- goto err_irq;
+ goto err_clk;
}
mutex_unlock(&viu_dev->lock);
@@ -1529,16 +1534,19 @@ static int viu_of_probe(struct platform_device *op)
dev_info(&op->dev, "Freescale VIU Video Capture Board\n");
return ret;
-err_irq:
- clk_disable_unprepare(viu_dev->clk);
err_clk:
- video_unregister_device(viu_dev->vdev);
+ clk_disable_unprepare(viu_dev->clk);
err_vdev:
- v4l2_ctrl_handler_free(&viu_dev->hdl);
+ video_unregister_device(viu_dev->vdev);
+err_unlock:
mutex_unlock(&viu_dev->lock);
+err_hdl:
+ v4l2_ctrl_handler_free(&viu_dev->hdl);
+err_i2c:
i2c_put_adapter(ad);
+err_v4l2:
v4l2_device_unregister(&viu_dev->v4l2_dev);
-err:
+err_irq:
irq_dispose_mapping(viu_irq);
return ret;
}
diff --git a/drivers/media/platform/msm/Kconfig b/drivers/media/platform/msm/Kconfig
index 9e4ea8d..164b85c 100644
--- a/drivers/media/platform/msm/Kconfig
+++ b/drivers/media/platform/msm/Kconfig
@@ -53,4 +53,4 @@
source "drivers/media/platform/msm/sde/Kconfig"
source "drivers/media/platform/msm/dvb/Kconfig"
source "drivers/media/platform/msm/broadcast/Kconfig"
-
+source "drivers/media/platform/msm/camera/Kconfig"
diff --git a/drivers/media/platform/msm/camera/Kconfig b/drivers/media/platform/msm/camera/Kconfig
new file mode 100644
index 0000000..b36346c
--- /dev/null
+++ b/drivers/media/platform/msm/camera/Kconfig
@@ -0,0 +1,8 @@
+#
+# CAMERA
+#
+
+menuconfig CAMERA_FW_UPDATE
+ tristate "Qualcomm Technologies, Inc. Camera Firmware Update"
+ depends on I2C
+
diff --git a/drivers/media/platform/msm/camera/cam_cdm/Makefile b/drivers/media/platform/msm/camera/cam_cdm/Makefile
index ad4ec04..4131cc8 100644
--- a/drivers/media/platform/msm/camera/cam_cdm/Makefile
+++ b/drivers/media/platform/msm/camera/cam_cdm/Makefile
@@ -7,3 +7,7 @@
obj-$(CONFIG_SPECTRA_CAMERA) += cam_cdm_soc.o cam_cdm_util.o cam_cdm_intf.o \
cam_cdm_core_common.o cam_cdm_virtual_core.o \
cam_cdm_hw_core.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-pointer-bool-conversion \
diff --git a/drivers/media/platform/msm/camera/cam_core/cam_context.c b/drivers/media/platform/msm/camera/cam_core/cam_context.c
index 670349d..98fff48 100644
--- a/drivers/media/platform/msm/camera/cam_core/cam_context.c
+++ b/drivers/media/platform/msm/camera/cam_core/cam_context.c
@@ -40,7 +40,6 @@ static int cam_context_handle_hw_event(void *context, uint32_t evt_id,
int cam_context_shutdown(struct cam_context *ctx)
{
int rc = 0;
- int32_t ctx_hdl = ctx->dev_hdl;
if (ctx->state_machine[ctx->state].ioctl_ops.stop_dev) {
rc = ctx->state_machine[ctx->state].ioctl_ops.stop_dev(
@@ -55,8 +54,6 @@ int cam_context_shutdown(struct cam_context *ctx)
CAM_ERR(CAM_CORE, "Error while dev release %d", rc);
}
- if (!rc)
- rc = cam_destroy_device_hdl(ctx_hdl);
return rc;
}
@@ -355,7 +352,7 @@ int cam_context_handle_start_dev(struct cam_context *ctx,
{
int rc = 0;
- if (!ctx || !ctx->state_machine) {
+ if (!ctx->state_machine) {
CAM_ERR(CAM_CORE, "Context is not ready");
return -EINVAL;
}
@@ -384,7 +381,7 @@ int cam_context_handle_stop_dev(struct cam_context *ctx,
{
int rc = 0;
- if (!ctx || !ctx->state_machine) {
+ if (!ctx->state_machine) {
CAM_ERR(CAM_CORE, "Context is not ready");
return -EINVAL;
}
diff --git a/drivers/media/platform/msm/camera/cam_core/cam_context.h b/drivers/media/platform/msm/camera/cam_core/cam_context.h
index ffceea2..bdce6ec 100644
--- a/drivers/media/platform/msm/camera/cam_core/cam_context.h
+++ b/drivers/media/platform/msm/camera/cam_core/cam_context.h
@@ -15,17 +15,17 @@
#include <linux/mutex.h>
#include <linux/spinlock.h>
+#include "../cam_req_mgr/cam_req_mgr_interface.h"
#include <linux/kref.h>
-#include "cam_req_mgr_interface.h"
#include "cam_hw_mgr_intf.h"
/* Forward declarations */
struct cam_context;
/* max request number */
-#define CAM_CTX_REQ_MAX 20
-#define CAM_CTX_CFG_MAX 20
-#define CAM_CTX_RES_MAX 20
+#define CAM_CTX_REQ_MAX 48
+#define CAM_CTX_CFG_MAX 48
+#define CAM_CTX_RES_MAX 48
/**
* enum cam_ctx_state - context top level states
@@ -70,7 +70,7 @@ struct cam_ctx_request {
uint32_t num_in_map_entries;
struct cam_hw_fence_map_entry out_map_entries[CAM_CTX_CFG_MAX];
uint32_t num_out_map_entries;
- atomic_t num_in_acked;
+ uint32_t num_in_acked;
uint32_t num_out_acked;
int flushed;
struct cam_context *ctx;
diff --git a/drivers/media/platform/msm/camera/cam_core/cam_context_utils.c b/drivers/media/platform/msm/camera/cam_core/cam_context_utils.c
index 839155f..85e9058 100644
--- a/drivers/media/platform/msm/camera/cam_core/cam_context_utils.c
+++ b/drivers/media/platform/msm/camera/cam_core/cam_context_utils.c
@@ -189,7 +189,8 @@ static void cam_context_sync_callback(int32_t sync_obj, int status, void *data)
return;
}
- if (atomic_inc_return(&req->num_in_acked) == req->num_in_map_entries) {
+ req->num_in_acked++;
+ if (req->num_in_acked == req->num_in_map_entries) {
apply.request_id = req->request_id;
/*
* take mutex to ensure that another thread does
@@ -341,7 +342,6 @@ int32_t cam_context_prepare_dev_to_hw(struct cam_context *ctx,
req->num_hw_update_entries = cfg.num_hw_update_entries;
req->num_out_map_entries = cfg.num_out_map_entries;
req->num_in_map_entries = cfg.num_in_map_entries;
- atomic_set(&req->num_in_acked, 0);
req->request_id = packet->header.request_id;
req->status = 1;
req->req_priv = cfg.priv;
@@ -481,7 +481,6 @@ int32_t cam_context_acquire_dev_to_hw(struct cam_context *ctx,
release.ctxt_to_hw_map = ctx->ctxt_to_hw_map;
ctx->hw_mgr_intf->hw_release(ctx->hw_mgr_intf->hw_mgr_priv, &release);
ctx->ctxt_to_hw_map = NULL;
- ctx->dev_hdl = -1;
end:
return rc;
}
@@ -493,7 +492,6 @@ int32_t cam_context_flush_ctx_to_hw(struct cam_context *ctx)
struct cam_ctx_request *req;
uint32_t i;
int rc = 0;
- bool free_req;
CAM_DBG(CAM_CTXT, "[%s] E: NRT flush ctx", ctx->dev_name);
@@ -530,21 +528,6 @@ int32_t cam_context_flush_ctx_to_hw(struct cam_context *ctx)
flush_args.flush_req_pending[flush_args.num_req_pending++] =
req->req_priv;
-
- free_req = false;
- for (i = 0; i < req->num_in_map_entries; i++) {
- rc = cam_sync_deregister_callback(
- cam_context_sync_callback,
- (void *)req,
- req->in_map_entries[i].sync_id);
- if (!rc) {
- cam_context_putref(ctx);
- if (atomic_inc_return(&req->num_in_acked) ==
- req->num_in_map_entries)
- free_req = true;
- }
- }
-
for (i = 0; i < req->num_out_map_entries; i++) {
if (req->out_map_entries[i].sync_id != -1) {
rc = cam_sync_signal(
@@ -552,25 +535,15 @@ int32_t cam_context_flush_ctx_to_hw(struct cam_context *ctx)
CAM_SYNC_STATE_SIGNALED_ERROR);
if (rc == -EALREADY) {
CAM_ERR(CAM_CTXT,
- "Req: %llu already signalled, sync_id:%d",
- req->request_id,
- req->out_map_entries[i].sync_id);
+ "Req: %llu already signalled, sync_id:%d",
+ req->request_id,
+ req->out_map_entries[i].
+ sync_id);
break;
}
}
}
- /*
- * If we have deregistered the last sync callback, req will
- * not be put on the free list. So put it on the free list here
- */
- if (free_req) {
- req->ctx = NULL;
- spin_lock(&ctx->lock);
- list_add_tail(&req->list, &ctx->free_req_list);
- spin_unlock(&ctx->lock);
- }
-
if (cam_debug_ctx_req_list & ctx->dev_id)
CAM_INFO(CAM_CTXT,
"[%s][%d] : Deleting req[%llu] from temp_list",
@@ -656,9 +629,7 @@ int32_t cam_context_flush_req_to_hw(struct cam_context *ctx,
struct cam_ctx_request *req = NULL;
struct cam_hw_flush_args flush_args;
uint32_t i;
- int32_t sync_id = 0;
int rc = 0;
- bool free_req = false;
CAM_DBG(CAM_CTXT, "[%s] E: NRT flush req", ctx->dev_name);
@@ -711,51 +682,32 @@ int32_t cam_context_flush_req_to_hw(struct cam_context *ctx,
}
if (req) {
- if (flush_args.num_req_pending) {
- for (i = 0; i < req->num_in_map_entries; i++) {
- rc = cam_sync_deregister_callback(
- cam_context_sync_callback,
- (void *)req,
- req->in_map_entries[i].sync_id);
- if (rc)
- continue;
-
- cam_context_putref(ctx);
- if (atomic_inc_return(&req->num_in_acked) ==
- req->num_in_map_entries)
- free_req = true;
- }
- }
-
if (flush_args.num_req_pending || flush_args.num_req_active) {
- for (i = 0; i < req->num_out_map_entries; i++) {
- sync_id =
- req->out_map_entries[i].sync_id;
- if (sync_id != -1) {
- rc = cam_sync_signal(sync_id,
+ for (i = 0; i < req->num_out_map_entries; i++)
+ if (req->out_map_entries[i].sync_id != -1) {
+ rc = cam_sync_signal(
+ req->out_map_entries[i].sync_id,
CAM_SYNC_STATE_SIGNALED_ERROR);
if (rc == -EALREADY) {
CAM_ERR(CAM_CTXT,
- "Req: %llu already signalled, sync_id:%d",
- req->request_id, sync_id);
+ "Req: %llu already signalled, sync_id:%d",
+ req->request_id,
+ req->out_map_entries[i].
+ sync_id);
break;
}
}
- }
- if (flush_args.num_req_active || free_req) {
- req->ctx = NULL;
+ if (flush_args.num_req_active) {
spin_lock(&ctx->lock);
list_add_tail(&req->list, &ctx->free_req_list);
spin_unlock(&ctx->lock);
+ req->ctx = NULL;
if (cam_debug_ctx_req_list & ctx->dev_id)
CAM_INFO(CAM_CTXT,
- "[%s][%d] : Moving req[%llu] from %s to free_list",
+ "[%s][%d] : Moving req[%llu] from active_list to free_list",
ctx->dev_name, ctx->ctx_id,
- req->request_id,
- flush_args.num_req_active ?
- "active_list" :
- "pending_list");
+ req->request_id);
}
}
}
diff --git a/drivers/media/platform/msm/camera/cam_core/cam_hw_mgr_intf.h b/drivers/media/platform/msm/camera/cam_core/cam_hw_mgr_intf.h
index f7990b6..3b8f056 100644
--- a/drivers/media/platform/msm/camera/cam_core/cam_hw_mgr_intf.h
+++ b/drivers/media/platform/msm/camera/cam_core/cam_hw_mgr_intf.h
@@ -200,9 +200,9 @@ struct cam_hw_config_args {
struct cam_hw_flush_args {
void *ctxt_to_hw_map;
uint32_t num_req_pending;
- void *flush_req_pending[20];
+ void *flush_req_pending[48];
uint32_t num_req_active;
- void *flush_req_active[20];
+ void *flush_req_active[48];
enum flush_type_t flush_type;
};
diff --git a/drivers/media/platform/msm/camera/cam_core/cam_node.c b/drivers/media/platform/msm/camera/cam_core/cam_node.c
index 3f24c6d..4e9034e 100644
--- a/drivers/media/platform/msm/camera/cam_core/cam_node.c
+++ b/drivers/media/platform/msm/camera/cam_core/cam_node.c
@@ -18,34 +18,6 @@
#include "cam_trace.h"
#include "cam_debug_util.h"
-static void cam_node_print_ctx_state(
- struct cam_node *node)
-{
- int i;
- struct cam_context *ctx;
-
- CAM_INFO(CAM_CORE, "[%s] state=%d, ctx_size %d",
- node->name, node->state, node->ctx_size);
-
- mutex_lock(&node->list_mutex);
- for (i = 0; i < node->ctx_size; i++) {
- ctx = &node->ctx_list[i];
-
- spin_lock(&ctx->lock);
- CAM_INFO(CAM_CORE,
- "[%s][%d] : state=%d, refcount=%d, active_req_list=%d, pending_req_list=%d, wait_req_list=%d, free_req_list=%d",
- ctx->dev_name ? ctx->dev_name : "null",
- i, ctx->state,
- atomic_read(&(ctx->refcount.refcount)),
- list_empty(&ctx->active_req_list),
- list_empty(&ctx->pending_req_list),
- list_empty(&ctx->wait_req_list),
- list_empty(&ctx->free_req_list));
- spin_unlock(&ctx->lock);
- }
- mutex_unlock(&node->list_mutex);
-}
-
static struct cam_context *cam_node_get_ctxt_from_free_list(
struct cam_node *node)
{
@@ -103,10 +75,6 @@ static int __cam_node_handle_acquire_dev(struct cam_node *node,
ctx = cam_node_get_ctxt_from_free_list(node);
if (!ctx) {
- CAM_ERR(CAM_CORE, "No free ctx in free list node %s",
- node->name);
- cam_node_print_ctx_state(node);
-
rc = -ENOMEM;
goto err;
}
@@ -118,9 +86,6 @@ static int __cam_node_handle_acquire_dev(struct cam_node *node,
goto free_ctx;
}
- CAM_DBG(CAM_CORE, "[%s] Acquire ctx_id %d",
- node->name, ctx->ctx_id);
-
return 0;
free_ctx:
cam_context_putref(ctx);
@@ -295,10 +260,6 @@ static int __cam_node_handle_release_dev(struct cam_node *node,
CAM_ERR(CAM_CORE, "destroy device handle is failed node %s",
node->name);
- CAM_DBG(CAM_CORE, "[%s] Release ctx_id=%d, refcount=%d",
- node->name, ctx->ctx_id,
- atomic_read(&(ctx->refcount.refcount)));
-
cam_context_putref(ctx);
return rc;
}
@@ -413,16 +374,14 @@ int cam_node_deinit(struct cam_node *node)
int cam_node_shutdown(struct cam_node *node)
{
int i = 0;
- int rc = 0;
if (!node)
return -EINVAL;
for (i = 0; i < node->ctx_size; i++) {
- if (node->ctx_list[i].dev_hdl > 0) {
- rc = cam_context_shutdown(&(node->ctx_list[i]));
- if (rc)
- continue;
+ if (node->ctx_list[i].dev_hdl >= 0) {
+ cam_context_shutdown(&(node->ctx_list[i]));
+ cam_destroy_device_hdl(node->ctx_list[i].dev_hdl);
cam_context_putref(&(node->ctx_list[i]));
}
}
diff --git a/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_hw.c b/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_hw.c
index 90603de..5e4ff0d 100644
--- a/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_hw.c
+++ b/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_hw.c
@@ -22,6 +22,11 @@
#include "cam_cpas_hw_intf.h"
#include "cam_cpas_soc.h"
+#define CAM_CPAS_AXI_MIN_MNOC_AB_BW (2048 * 1024)
+#define CAM_CPAS_AXI_MIN_MNOC_IB_BW (2048 * 1024)
+#define CAM_CPAS_AXI_MIN_CAMNOC_AB_BW (2048 * 1024)
+#define CAM_CPAS_AXI_MIN_CAMNOC_IB_BW (3000000000L)
+
static uint cam_min_camnoc_ib_bw;
module_param(cam_min_camnoc_ib_bw, uint, 0644);
@@ -77,8 +82,8 @@ static int cam_cpas_util_vote_bus_client_level(
if (level == bus_client->curr_vote_level)
return 0;
- CAM_DBG(CAM_CPAS, "Bus client=[%d][%s] index[%d]",
- bus_client->client_id, bus_client->name, level);
+ CAM_DBG(CAM_CPAS, "Bus client[%d] index[%d]", bus_client->client_id,
+ level);
msm_bus_scale_client_update_request(bus_client->client_id, level);
bus_client->curr_vote_level = level;
@@ -147,8 +152,8 @@ static int cam_cpas_util_vote_bus_client_bw(
path->vectors[0].ab = ab;
path->vectors[0].ib = ib;
- CAM_DBG(CAM_CPAS, "Bus client=[%d][%s] :ab[%llu] ib[%llu], index[%d]",
- bus_client->client_id, bus_client->name, ab, ib, idx);
+ CAM_DBG(CAM_CPAS, "Bus client[%d] :ab[%llu] ib[%llu], index[%d]",
+ bus_client->client_id, ab, ib, idx);
msm_bus_scale_client_update_request(bus_client->client_id, idx);
return 0;
@@ -203,12 +208,10 @@ static int cam_cpas_util_register_bus_client(
bus_client->num_paths = pdata->usecase[0].num_paths;
bus_client->curr_vote_level = 0;
bus_client->valid = true;
- bus_client->name = pdata->name;
mutex_init(&bus_client->lock);
- CAM_DBG(CAM_CPAS, "Bus Client=[%d][%s] : src=%d, dst=%d",
- bus_client->client_id, bus_client->name,
- bus_client->src, bus_client->dst);
+ CAM_DBG(CAM_CPAS, "Bus Client : src=%d, dst=%d, bus_client=%d",
+ bus_client->src, bus_client->dst, bus_client->client_id);
return 0;
fail_unregister_client:
@@ -460,7 +463,6 @@ static int cam_cpas_hw_reg_write(struct cam_hw_info *cpas_hw,
{
struct cam_hw_soc_info *soc_info = &cpas_hw->soc_info;
struct cam_cpas *cpas_core = (struct cam_cpas *) cpas_hw->core_info;
- struct cam_cpas_client *cpas_client = NULL;
int reg_base_index = cpas_core->regbase_index[reg_base];
uint32_t client_indx = CAM_CPAS_GET_CLIENT_IDX(client_handle);
int rc = 0;
@@ -476,12 +478,9 @@ static int cam_cpas_hw_reg_write(struct cam_hw_info *cpas_hw,
return -EINVAL;
mutex_lock(&cpas_core->client_mutex[client_indx]);
- cpas_client = cpas_core->cpas_client[client_indx];
if (!CAM_CPAS_CLIENT_STARTED(cpas_core, client_indx)) {
- CAM_ERR(CAM_CPAS, "client=[%d][%s][%d] has not started",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index);
+ CAM_ERR(CAM_CPAS, "client has not started%d", client_indx);
rc = -EPERM;
goto unlock_client;
}
@@ -504,7 +503,6 @@ static int cam_cpas_hw_reg_read(struct cam_hw_info *cpas_hw,
{
struct cam_hw_soc_info *soc_info = &cpas_hw->soc_info;
struct cam_cpas *cpas_core = (struct cam_cpas *) cpas_hw->core_info;
- struct cam_cpas_client *cpas_client = NULL;
int reg_base_index = cpas_core->regbase_index[reg_base];
uint32_t reg_value;
uint32_t client_indx = CAM_CPAS_GET_CLIENT_IDX(client_handle);
@@ -524,12 +522,9 @@ static int cam_cpas_hw_reg_read(struct cam_hw_info *cpas_hw,
return -EINVAL;
mutex_lock(&cpas_core->client_mutex[client_indx]);
- cpas_client = cpas_core->cpas_client[client_indx];
if (!CAM_CPAS_CLIENT_STARTED(cpas_core, client_indx)) {
- CAM_ERR(CAM_CPAS, "client=[%d][%s][%d] has not started",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index);
+ CAM_ERR(CAM_CPAS, "client has not started%d", client_indx);
rc = -EPERM;
goto unlock_client;
}
@@ -585,9 +580,8 @@ static int cam_cpas_util_set_camnoc_axi_clk_rate(
soc_private->camnoc_axi_clk_bw_margin) / 100;
if ((required_camnoc_bw > 0) &&
- (required_camnoc_bw <
- soc_private->camnoc_axi_min_ib_bw))
- required_camnoc_bw = soc_private->camnoc_axi_min_ib_bw;
+ (required_camnoc_bw < CAM_CPAS_AXI_MIN_CAMNOC_IB_BW))
+ required_camnoc_bw = CAM_CPAS_AXI_MIN_CAMNOC_IB_BW;
clk_rate = required_camnoc_bw / soc_private->camnoc_bus_width;
@@ -701,7 +695,6 @@ static int cam_cpas_hw_update_axi_vote(struct cam_hw_info *cpas_hw,
{
struct cam_axi_vote axi_vote;
struct cam_cpas *cpas_core = (struct cam_cpas *) cpas_hw->core_info;
- struct cam_cpas_client *cpas_client = NULL;
uint32_t client_indx = CAM_CPAS_GET_CLIENT_IDX(client_handle);
int rc = 0;
@@ -726,20 +719,16 @@ static int cam_cpas_hw_update_axi_vote(struct cam_hw_info *cpas_hw,
mutex_lock(&cpas_hw->hw_mutex);
mutex_lock(&cpas_core->client_mutex[client_indx]);
- cpas_client = cpas_core->cpas_client[client_indx];
if (!CAM_CPAS_CLIENT_STARTED(cpas_core, client_indx)) {
- CAM_ERR(CAM_CPAS, "client=[%d][%s][%d] has not started",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index);
+ CAM_ERR(CAM_CPAS, "client has not started %d", client_indx);
rc = -EPERM;
goto unlock_client;
}
CAM_DBG(CAM_CPAS,
- "Client=[%d][%s][%d] Requested compressed[%llu], uncompressed[%llu]",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index, axi_vote.compressed_bw,
+ "Client[%d] Requested compressed[%llu], uncompressed[%llu]",
+ client_indx, axi_vote.compressed_bw,
axi_vote.uncompressed_bw);
rc = cam_cpas_util_apply_client_axi_vote(cpas_hw,
@@ -820,8 +809,7 @@ static int cam_cpas_util_apply_client_ahb_vote(struct cam_hw_info *cpas_hw,
mutex_lock(&ahb_bus_client->lock);
cpas_client->ahb_level = required_level;
- CAM_DBG(CAM_CPAS, "Client=[%d][%s] required level[%d], curr_level[%d]",
- ahb_bus_client->client_id, ahb_bus_client->name,
+ CAM_DBG(CAM_CPAS, "Clients required level[%d], curr_level[%d]",
required_level, ahb_bus_client->curr_vote_level);
if (required_level == ahb_bus_client->curr_vote_level)
@@ -865,7 +853,6 @@ static int cam_cpas_hw_update_ahb_vote(struct cam_hw_info *cpas_hw,
{
struct cam_ahb_vote ahb_vote;
struct cam_cpas *cpas_core = (struct cam_cpas *) cpas_hw->core_info;
- struct cam_cpas_client *cpas_client = NULL;
uint32_t client_indx = CAM_CPAS_GET_CLIENT_IDX(client_handle);
int rc = 0;
@@ -888,21 +875,17 @@ static int cam_cpas_hw_update_ahb_vote(struct cam_hw_info *cpas_hw,
mutex_lock(&cpas_hw->hw_mutex);
mutex_lock(&cpas_core->client_mutex[client_indx]);
- cpas_client = cpas_core->cpas_client[client_indx];
if (!CAM_CPAS_CLIENT_STARTED(cpas_core, client_indx)) {
- CAM_ERR(CAM_CPAS, "client=[%d][%s][%d] has not started",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index);
+ CAM_ERR(CAM_CPAS, "client has not started %d", client_indx);
rc = -EPERM;
goto unlock_client;
}
CAM_DBG(CAM_CPAS,
- "client=[%d][%s][%d] : type[%d], level[%d], freq[%ld], applied[%d]",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index, ahb_vote.type,
- ahb_vote.vote.level, ahb_vote.vote.freq,
+ "client[%d] : type[%d], level[%d], freq[%ld], applied[%d]",
+ client_indx, ahb_vote.type, ahb_vote.vote.level,
+ ahb_vote.vote.freq,
cpas_core->cpas_client[client_indx]->ahb_level);
rc = cam_cpas_util_apply_client_ahb_vote(cpas_hw,
@@ -965,37 +948,32 @@ static int cam_cpas_hw_start(void *hw_priv, void *start_args,
mutex_lock(&cpas_hw->hw_mutex);
mutex_lock(&cpas_core->client_mutex[client_indx]);
- cpas_client = cpas_core->cpas_client[client_indx];
if (!CAM_CPAS_CLIENT_REGISTERED(cpas_core, client_indx)) {
- CAM_ERR(CAM_CPAS, "client=[%d] is not registered",
- client_indx);
+ CAM_ERR(CAM_CPAS, "client is not registered %d", client_indx);
rc = -EPERM;
goto done;
}
if (CAM_CPAS_CLIENT_STARTED(cpas_core, client_indx)) {
- CAM_ERR(CAM_CPAS, "client=[%d][%s][%d] is in start state",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index);
+ CAM_ERR(CAM_CPAS, "Client %d is in start state", client_indx);
rc = -EPERM;
goto done;
}
- CAM_DBG(CAM_CPAS,
- "AHB :client=[%d][%s][%d] type[%d], level[%d], applied[%d]",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index,
- ahb_vote->type, ahb_vote->vote.level, cpas_client->ahb_level);
+ cpas_client = cpas_core->cpas_client[client_indx];
+
+ CAM_DBG(CAM_CPAS, "AHB :client[%d] type[%d], level[%d], applied[%d]",
+ client_indx, ahb_vote->type, ahb_vote->vote.level,
+ cpas_client->ahb_level);
rc = cam_cpas_util_apply_client_ahb_vote(cpas_hw, cpas_client,
ahb_vote, &applied_level);
if (rc)
goto done;
CAM_DBG(CAM_CPAS,
- "AXI client=[%d][%s][%d] compressed_bw[%llu], uncompressed_bw[%llu]",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index, axi_vote->compressed_bw,
+ "AXI client[%d] compressed_bw[%llu], uncompressed_bw[%llu]",
+ client_indx, axi_vote->compressed_bw,
axi_vote->uncompressed_bw);
rc = cam_cpas_util_apply_client_axi_vote(cpas_hw,
cpas_client, axi_vote);
@@ -1032,9 +1010,9 @@ static int cam_cpas_hw_start(void *hw_priv, void *start_args,
cpas_client->started = true;
cpas_core->streamon_clients++;
- CAM_DBG(CAM_CPAS, "client=[%d][%s][%d] streamon_clients=%d",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index, cpas_core->streamon_clients);
+ CAM_DBG(CAM_CPAS, "client=%s, streamon_clients=%d",
+ soc_private->client_name[client_indx],
+ cpas_core->streamon_clients);
done:
mutex_unlock(&cpas_core->client_mutex[client_indx]);
mutex_unlock(&cpas_hw->hw_mutex);
@@ -1084,20 +1062,18 @@ static int cam_cpas_hw_stop(void *hw_priv, void *stop_args,
mutex_lock(&cpas_hw->hw_mutex);
mutex_lock(&cpas_core->client_mutex[client_indx]);
- cpas_client = cpas_core->cpas_client[client_indx];
- CAM_DBG(CAM_CPAS, "Client=[%d][%s][%d] streamon_clients=%d",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index, cpas_core->streamon_clients);
+ CAM_DBG(CAM_CPAS, "client=%s, streamon_clients=%d",
+ soc_private->client_name[client_indx],
+ cpas_core->streamon_clients);
if (!CAM_CPAS_CLIENT_STARTED(cpas_core, client_indx)) {
- CAM_ERR(CAM_CPAS, "Client=[%d][%s][%d] is not started",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index);
+ CAM_ERR(CAM_CPAS, "Client %d is not started", client_indx);
rc = -EPERM;
goto done;
}
+ cpas_client = cpas_core->cpas_client[client_indx];
cpas_client->started = false;
cpas_core->streamon_clients--;
@@ -1231,9 +1207,8 @@ static int cam_cpas_hw_register_client(struct cam_hw_info *cpas_hw,
cpas_client, client_indx);
if (rc) {
CAM_ERR(CAM_CPAS,
- "axi_port_insert failed Client=[%d][%s][%d], rc=%d",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index, rc);
+ "axi_port_insert failed client_indx=%d, rc=%d",
+ client_indx, rc);
kfree(cpas_client);
mutex_unlock(&cpas_hw->hw_mutex);
return -EINVAL;
@@ -1248,9 +1223,8 @@ static int cam_cpas_hw_register_client(struct cam_hw_info *cpas_hw,
mutex_unlock(&cpas_hw->hw_mutex);
- CAM_DBG(CAM_CPAS, "client=[%d][%s][%d], registered_clients=%d",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index, cpas_core->registered_clients);
+ CAM_DBG(CAM_CPAS, "client_indx=%d, registered_clients=%d",
+ client_indx, cpas_core->registered_clients);
return 0;
}
@@ -1259,7 +1233,6 @@ static int cam_cpas_hw_unregister_client(struct cam_hw_info *cpas_hw,
uint32_t client_handle)
{
struct cam_cpas *cpas_core = (struct cam_cpas *) cpas_hw->core_info;
- struct cam_cpas_client *cpas_client = NULL;
uint32_t client_indx = CAM_CPAS_GET_CLIENT_IDX(client_handle);
int rc = 0;
@@ -1268,20 +1241,15 @@ static int cam_cpas_hw_unregister_client(struct cam_hw_info *cpas_hw,
mutex_lock(&cpas_hw->hw_mutex);
mutex_lock(&cpas_core->client_mutex[client_indx]);
- cpas_client = cpas_core->cpas_client[client_indx];
if (!CAM_CPAS_CLIENT_REGISTERED(cpas_core, client_indx)) {
- CAM_ERR(CAM_CPAS, "Client=[%d][%s][%d] not registered",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index);
+ CAM_ERR(CAM_CPAS, "client not registered %d", client_indx);
rc = -EPERM;
goto done;
}
if (CAM_CPAS_CLIENT_STARTED(cpas_core, client_indx)) {
- CAM_ERR(CAM_CPAS, "Client=[%d][%s][%d] is not stopped",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index);
+ CAM_ERR(CAM_CPAS, "Client %d is not stopped", client_indx);
rc = -EPERM;
goto done;
}
@@ -1289,9 +1257,8 @@ static int cam_cpas_hw_unregister_client(struct cam_hw_info *cpas_hw,
cam_cpas_util_remove_client_from_axi_port(
cpas_core->cpas_client[client_indx]);
- CAM_DBG(CAM_CPAS, "client=[%d][%s][%d], registered_clients=%d",
- client_indx, cpas_client->data.identifier,
- cpas_client->data.cell_index, cpas_core->registered_clients);
+ CAM_DBG(CAM_CPAS, "client_indx=%d, registered_clients=%d",
+ client_indx, cpas_core->registered_clients);
kfree(cpas_core->cpas_client[client_indx]);
cpas_core->cpas_client[client_indx] = NULL;
diff --git a/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_hw.h b/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_hw.h
index d51b152..2e660b1 100644
--- a/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_hw.h
+++ b/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_hw.h
@@ -20,11 +20,6 @@
#define CAM_CPAS_MAX_CLIENTS 30
#define CAM_CPAS_INFLIGHT_WORKS 5
-#define CAM_CPAS_AXI_MIN_MNOC_AB_BW (2048 * 1024)
-#define CAM_CPAS_AXI_MIN_MNOC_IB_BW (2048 * 1024)
-#define CAM_CPAS_AXI_MIN_CAMNOC_AB_BW (2048 * 1024)
-#define CAM_CPAS_AXI_MIN_CAMNOC_IB_BW (3000000000L)
-
#define CAM_CPAS_GET_CLIENT_IDX(handle) (handle)
#define CAM_CPAS_GET_CLIENT_HANDLE(indx) (indx)
@@ -123,7 +118,6 @@ struct cam_cpas_client {
* @dyn_vote: Whether dynamic voting enabled
* @lock: Mutex lock used while voting on this client
* @valid: Whether bus client is valid
- * @name: Name of the bus client
*
*/
struct cam_cpas_bus_client {
@@ -137,7 +131,6 @@ struct cam_cpas_bus_client {
bool dyn_vote;
struct mutex lock;
bool valid;
- const char *name;
};
/**
diff --git a/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_soc.c b/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_soc.c
index b73b32a..8f9ec14 100644
--- a/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_soc.c
+++ b/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_soc.c
@@ -56,27 +56,6 @@ int cam_cpas_get_custom_dt_info(struct platform_device *pdev,
CAM_DBG(CAM_CPAS, "CPAS HW VERSION %x", soc_private->hw_version);
- soc_private->camnoc_axi_min_ib_bw = 0;
- rc = of_property_read_u64(of_node,
- "camnoc-axi-min-ib-bw",
- &soc_private->camnoc_axi_min_ib_bw);
- if (rc == -EOVERFLOW) {
- soc_private->camnoc_axi_min_ib_bw = 0;
- rc = of_property_read_u32(of_node,
- "camnoc-axi-min-ib-bw",
- (u32 *)&soc_private->camnoc_axi_min_ib_bw);
- }
-
- if (rc) {
- CAM_DBG(CAM_CPAS,
- "failed to read camnoc-axi-min-ib-bw rc:%d", rc);
- soc_private->camnoc_axi_min_ib_bw =
- CAM_CPAS_AXI_MIN_CAMNOC_IB_BW;
- }
-
- CAM_DBG(CAM_CPAS, "camnoc-axi-min-ib-bw = %llu",
- soc_private->camnoc_axi_min_ib_bw);
-
soc_private->client_id_based = of_property_read_bool(of_node,
"client-id-based");
diff --git a/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_soc.h b/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_soc.h
index f6ae8a8..91e8d0c0 100644
--- a/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_soc.h
+++ b/drivers/media/platform/msm/camera/cam_cpas/cam_cpas_soc.h
@@ -48,7 +48,6 @@ struct cam_cpas_vdd_ahb_mapping {
* @camnoc_bus_width : CAMNOC Bus width
* @camnoc_axi_clk_bw_margin : BW Margin in percentage to add while calculating
* camnoc axi clock
- * @camnoc_axi_min_ib_bw: Min camnoc BW which varies based on target
*
*/
struct cam_cpas_private_soc {
@@ -67,7 +66,6 @@ struct cam_cpas_private_soc {
bool control_camnoc_axi_clk;
uint32_t camnoc_bus_width;
uint32_t camnoc_axi_clk_bw_margin;
- uint64_t camnoc_axi_min_ib_bw;
};
int cam_cpas_soc_init_resources(struct cam_hw_soc_info *soc_info,
diff --git a/drivers/media/platform/msm/camera/cam_cpas/cpas_top/cam_cpastop_hw.c b/drivers/media/platform/msm/camera/cam_cpas/cpas_top/cam_cpastop_hw.c
index 158bbb9..9b703c0 100644
--- a/drivers/media/platform/msm/camera/cam_cpas/cpas_top/cam_cpastop_hw.c
+++ b/drivers/media/platform/msm/camera/cam_cpas/cpas_top/cam_cpastop_hw.c
@@ -374,8 +374,7 @@ static void cam_cpastop_work(struct work_struct *work)
if ((payload->irq_status & camnoc_info->irq_err[i].sbm_port) &&
(camnoc_info->irq_err[i].enable)) {
irq_type = camnoc_info->irq_err[i].irq_type;
- CAM_ERR_RATE_LIMIT(CAM_CPAS,
- "Error occurred, type=%d", irq_type);
+ CAM_ERR(CAM_CPAS, "Error occurred, type=%d", irq_type);
memset(&irq_data, 0x0, sizeof(irq_data));
irq_data.irq_type = (enum cam_camnoc_irq_type)irq_type;
@@ -493,9 +492,8 @@ static int cam_cpastop_poweron(struct cam_hw_info *cpas_hw)
}
}
- if ((soc_private && ((soc_private->soc_id == SDM710_SOC_ID) ||
- ((soc_private->soc_id == SDM670_SOC_ID) &&
- (soc_private->hw_rev == SDM670_V1_1))))) {
+ if ((soc_private && soc_private->soc_id == SDM670_SOC_ID) &&
+ (soc_private->hw_rev == SDM670_V1_1)) {
struct cam_cpas_reg *reg_info;
int tcsr_index;
@@ -553,9 +551,8 @@ static int cam_cpastop_poweroff(struct cam_hw_info *cpas_hw)
}
}
- if ((soc_private && ((soc_private->soc_id == SDM710_SOC_ID)
- || ((soc_private->soc_id == SDM670_SOC_ID) &&
- (soc_private->hw_rev == SDM670_V1_1))))) {
+ if ((soc_private && soc_private->soc_id == SDM670_SOC_ID) &&
+ (soc_private->hw_rev == SDM670_V1_1)) {
struct cam_cpas_reg *reg_info;
int tcsr_index;
diff --git a/drivers/media/platform/msm/camera/cam_fd/cam_fd_dev.c b/drivers/media/platform/msm/camera/cam_fd/cam_fd_dev.c
index 3f01244..6963f33 100644
--- a/drivers/media/platform/msm/camera/cam_fd/cam_fd_dev.c
+++ b/drivers/media/platform/msm/camera/cam_fd/cam_fd_dev.c
@@ -43,12 +43,12 @@ struct cam_fd_dev {
bool probe_done;
};
-static struct cam_fd_dev g_fd_dev;
+static struct cam_fd_dev *g_fd_dev;
static int cam_fd_dev_open(struct v4l2_subdev *sd,
struct v4l2_subdev_fh *fh)
{
- struct cam_fd_dev *fd_dev = &g_fd_dev;
+ struct cam_fd_dev *fd_dev = g_fd_dev;
if (!fd_dev->probe_done) {
CAM_ERR(CAM_FD, "FD Dev not initialized, fd_dev=%pK", fd_dev);
@@ -66,7 +66,7 @@ static int cam_fd_dev_open(struct v4l2_subdev *sd,
static int cam_fd_dev_close(struct v4l2_subdev *sd,
struct v4l2_subdev_fh *fh)
{
- struct cam_fd_dev *fd_dev = &g_fd_dev;
+ struct cam_fd_dev *fd_dev = g_fd_dev;
struct cam_node *node = v4l2_get_subdevdata(sd);
if (!fd_dev->probe_done) {
@@ -101,16 +101,16 @@ static int cam_fd_dev_probe(struct platform_device *pdev)
struct cam_hw_mgr_intf hw_mgr_intf;
struct cam_node *node;
- g_fd_dev.sd.internal_ops = &cam_fd_subdev_internal_ops;
+ g_fd_dev->sd.internal_ops = &cam_fd_subdev_internal_ops;
/* Initialze the v4l2 subdevice first. (create cam_node) */
- rc = cam_subdev_probe(&g_fd_dev.sd, pdev, CAM_FD_DEV_NAME,
+ rc = cam_subdev_probe(&g_fd_dev->sd, pdev, CAM_FD_DEV_NAME,
CAM_FD_DEVICE_TYPE);
if (rc) {
CAM_ERR(CAM_FD, "FD cam_subdev_probe failed, rc=%d", rc);
return rc;
}
- node = (struct cam_node *) g_fd_dev.sd.token;
+ node = (struct cam_node *) g_fd_dev->sd.token;
rc = cam_fd_hw_mgr_init(pdev->dev.of_node, &hw_mgr_intf);
if (rc) {
@@ -120,8 +120,8 @@ static int cam_fd_dev_probe(struct platform_device *pdev)
}
for (i = 0; i < CAM_CTX_MAX; i++) {
- rc = cam_fd_context_init(&g_fd_dev.fd_ctx[i],
- &g_fd_dev.base_ctx[i], &node->hw_mgr_intf, i);
+ rc = cam_fd_context_init(&g_fd_dev->fd_ctx[i],
+ &g_fd_dev->base_ctx[i], &node->hw_mgr_intf, i);
if (rc) {
CAM_ERR(CAM_FD, "FD context init failed i=%d, rc=%d",
i, rc);
@@ -129,15 +129,15 @@ static int cam_fd_dev_probe(struct platform_device *pdev)
}
}
- rc = cam_node_init(node, &hw_mgr_intf, g_fd_dev.base_ctx, CAM_CTX_MAX,
+ rc = cam_node_init(node, &hw_mgr_intf, g_fd_dev->base_ctx, CAM_CTX_MAX,
CAM_FD_DEV_NAME);
if (rc) {
CAM_ERR(CAM_FD, "FD node init failed, rc=%d", rc);
goto deinit_ctx;
}
- mutex_init(&g_fd_dev.lock);
- g_fd_dev.probe_done = true;
+ mutex_init(&g_fd_dev->lock);
+ g_fd_dev->probe_done = true;
CAM_DBG(CAM_FD, "Camera FD probe complete");
@@ -145,11 +145,11 @@ static int cam_fd_dev_probe(struct platform_device *pdev)
deinit_ctx:
for (--i; i >= 0; i--) {
- if (cam_fd_context_deinit(&g_fd_dev.fd_ctx[i]))
+ if (cam_fd_context_deinit(&g_fd_dev->fd_ctx[i]))
CAM_ERR(CAM_FD, "FD context %d deinit failed", i);
}
unregister_subdev:
- if (cam_subdev_remove(&g_fd_dev.sd))
+ if (cam_subdev_remove(&g_fd_dev->sd))
CAM_ERR(CAM_FD, "Failed in subdev remove");
return rc;
@@ -160,7 +160,7 @@ static int cam_fd_dev_remove(struct platform_device *pdev)
int i, rc;
for (i = 0; i < CAM_CTX_MAX; i++) {
- rc = cam_fd_context_deinit(&g_fd_dev.fd_ctx[i]);
+ rc = cam_fd_context_deinit(&g_fd_dev->fd_ctx[i]);
if (rc)
CAM_ERR(CAM_FD, "FD context %d deinit failed, rc=%d",
i, rc);
@@ -170,12 +170,12 @@ static int cam_fd_dev_remove(struct platform_device *pdev)
if (rc)
CAM_ERR(CAM_FD, "Failed in hw mgr deinit, rc=%d", rc);
- rc = cam_subdev_remove(&g_fd_dev.sd);
+ rc = cam_subdev_remove(&g_fd_dev->sd);
if (rc)
CAM_ERR(CAM_FD, "Unregister failed, rc=%d", rc);
- mutex_destroy(&g_fd_dev.lock);
- g_fd_dev.probe_done = false;
+ mutex_destroy(&g_fd_dev->lock);
+ g_fd_dev->probe_done = false;
return rc;
}
@@ -187,7 +187,7 @@ static const struct of_device_id cam_fd_dt_match[] = {
{}
};
-static struct platform_driver cam_fd_driver = {
+struct platform_driver cam_fd_driver = {
.probe = cam_fd_dev_probe,
.remove = cam_fd_dev_remove,
.driver = {
@@ -199,12 +199,17 @@ static struct platform_driver cam_fd_driver = {
static int __init cam_fd_dev_init_module(void)
{
+ g_fd_dev = kzalloc(sizeof(struct cam_fd_dev), GFP_KERNEL);
+ if (!g_fd_dev)
+ return -ENOMEM;
return platform_driver_register(&cam_fd_driver);
}
static void __exit cam_fd_dev_exit_module(void)
{
platform_driver_unregister(&cam_fd_driver);
+ kfree(g_fd_dev);
+ g_fd_dev = NULL;
}
module_init(cam_fd_dev_init_module);
diff --git a/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_intf.h b/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_intf.h
index 2c364e01..19bcb7d 100644
--- a/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_intf.h
+++ b/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_intf.h
@@ -35,6 +35,7 @@ struct hfi_mem {
* @cmd_q: command queue hfi memory for host to firmware communication
* @msg_q: message queue hfi memory for firmware to host communication
* @dbg_q: debug queue hfi memory for firmware debug information
+ * @sfr_buf: buffer for subsystem failure reason[SFR]
* @sec_heap: secondary heap hfi memory for firmware
* @qdss: qdss mapped memory for fw
* @icp_base: icp base address
@@ -44,6 +45,7 @@ struct hfi_mem_info {
struct hfi_mem cmd_q;
struct hfi_mem msg_q;
struct hfi_mem dbg_q;
+ struct hfi_mem sfr_buf;
struct hfi_mem sec_heap;
struct hfi_mem shmem;
struct hfi_mem qdss;
@@ -121,12 +123,6 @@ void cam_hfi_deinit(void __iomem *icp_base);
int hfi_set_debug_level(u64 a5_dbg_type, uint32_t lvl);
/**
- * hfi_set_fw_dump_level() - set firmware dump level
- * @lvl: level of firmware dump level
- */
-int hfi_set_fw_dump_level(uint32_t lvl);
-
-/**
* hfi_enable_ipe_bps_pc() - Enable interframe pc
* Host sends a command to firmware to enable interframe
* power collapse for IPE and BPS hardware.
diff --git a/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_reg.h b/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_reg.h
index 2153cea..f652cfa 100644
--- a/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_reg.h
+++ b/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_reg.h
@@ -43,6 +43,7 @@
#define HFI_REG_UNCACHED_HEAP_SIZE 0x60
#define HFI_REG_QDSS_IOVA 0x6C
#define HFI_REG_QDSS_IOVA_SIZE 0x70
+#define HFI_REG_SFR_PTR 0x68
/* end of ICP CSR registers */
/* flags for ICP CSR registers */
@@ -72,6 +73,7 @@
#define ICP_CMD_Q_SIZE_IN_BYTES 4096
#define ICP_MSG_Q_SIZE_IN_BYTES 4096
#define ICP_DBG_Q_SIZE_IN_BYTES 102400
+#define ICP_MSG_SFR_SIZE_IN_BYTES 4096
#define ICP_SHARED_MEM_IN_BYTES (1024 * 1024)
#define ICP_UNCACHED_HEAP_SIZE_IN_BYTES (2 * 1024 * 1024)
@@ -128,10 +130,14 @@ enum reg_settings {
/**
* @INTR_DISABLE: Disable interrupt
* @INTR_ENABLE: Enable interrupt
+ * @INTR_ENABLE_WD0: Enable WD0
+ * @INTR_ENABLE_WD1: Enable WD1
*/
enum intr_status {
INTR_DISABLE,
- INTR_ENABLE
+ INTR_ENABLE,
+ INTR_ENABLE_WD0,
+ INTR_ENABLE_WD1 = 0x4
};
/**
@@ -286,6 +292,16 @@ struct hfi_q_hdr {
};
/**
+ * struct sfr_buf
+ * @size: Number of characters
+ * @msg : Subsystem failure reason
+ */
+struct sfr_buf {
+ uint32_t size;
+ char msg[ICP_MSG_SFR_SIZE_IN_BYTES];
+};
+
+/**
* struct hfi_q_tbl
* @q_tbl_hdr: Queue table header
* @q_hdr: Queue header info, it holds info of cmd, msg and debug queues
diff --git a/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_session_defs.h b/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_session_defs.h
index 69e6ccf..7b2cb8b 100644
--- a/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_session_defs.h
+++ b/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_session_defs.h
@@ -261,8 +261,8 @@ enum bps_io_images {
};
struct frame_buffer {
- uint32_t buf_ptr[MAX_NUM_OF_IMAGE_PLANES];
- uint32_t meta_buf_ptr[MAX_NUM_OF_IMAGE_PLANES];
+ uint32_t buffer_ptr[MAX_NUM_OF_IMAGE_PLANES];
+ uint32_t meta_buffer_ptr[MAX_NUM_OF_IMAGE_PLANES];
} __packed;
struct bps_frame_process_data {
diff --git a/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_sys_defs.h b/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_sys_defs.h
index 311886f..91190b6 100644
--- a/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_sys_defs.h
+++ b/drivers/media/platform/msm/camera/cam_icp/fw_inc/hfi_sys_defs.h
@@ -160,7 +160,6 @@
#define HFI_PROP_SYS_IMAGE_VER (HFI_PROPERTY_ICP_COMMON_START + 0x3)
#define HFI_PROP_SYS_SUPPORTED (HFI_PROPERTY_ICP_COMMON_START + 0x4)
#define HFI_PROP_SYS_IPEBPS_PC (HFI_PROPERTY_ICP_COMMON_START + 0x5)
-#define HFI_PROP_SYS_FW_DUMP_CFG (HFI_PROPERTY_ICP_COMMON_START + 0x8)
/* Capabilities reported at sys init */
#define HFI_CAPS_PLACEHOLDER_1 (HFI_COMMON_BASE + 0x1)
@@ -181,18 +180,6 @@
/* Disable ARM9 watchdog. */
#define HFI_DEBUG_CFG_ARM9WD 0x10000000
-
-/*
- * HFI_FW_DUMP levels
- * HFI_FW_DUMP_xx
- */
-#define HFI_FW_DUMP_DISABLED 0x00000000
-#define HFI_FW_DUMP_ON_FAILURE 0x00000001
-#define HFI_FW_DUMP_ALWAYS 0x00000002
-
-/* Number of available dump levels. */
-#define NUM_HFI_DUMP_LVL 0x00000003
-
/* Debug Msg Communication types:
* Section describes different modes (HFI_DEBUG_MODE_X)
* available to communicate the debug messages
diff --git a/drivers/media/platform/msm/camera/cam_icp/hfi.c b/drivers/media/platform/msm/camera/cam_icp/hfi.c
index a0752f5..a53f73f 100644
--- a/drivers/media/platform/msm/camera/cam_icp/hfi.c
+++ b/drivers/media/platform/msm/camera/cam_icp/hfi.c
@@ -324,42 +324,6 @@ int hfi_set_debug_level(u64 a5_dbg_type, uint32_t lvl)
return 0;
}
-int hfi_set_fw_dump_level(uint32_t lvl)
-{
- uint8_t *prop = NULL;
- struct hfi_cmd_prop *fw_dump_level_switch_prop = NULL;
- uint32_t size = 0;
-
- CAM_DBG(CAM_HFI, "fw dump ENTER");
-
- size = sizeof(struct hfi_cmd_prop) + sizeof(lvl);
- prop = kzalloc(size, GFP_KERNEL);
- if (!prop)
- return -ENOMEM;
-
- fw_dump_level_switch_prop = (struct hfi_cmd_prop *)prop;
- fw_dump_level_switch_prop->size = size;
- fw_dump_level_switch_prop->pkt_type = HFI_CMD_SYS_SET_PROPERTY;
- fw_dump_level_switch_prop->num_prop = 1;
- fw_dump_level_switch_prop->prop_data[0] = HFI_PROP_SYS_FW_DUMP_CFG;
- fw_dump_level_switch_prop->prop_data[1] = lvl;
-
- CAM_DBG(CAM_HFI, "prop->size = %d\n"
- "prop->pkt_type = %d\n"
- "prop->num_prop = %d\n"
- "prop->prop_data[0] = %d\n"
- "prop->prop_data[1] = %d\n",
- fw_dump_level_switch_prop->size,
- fw_dump_level_switch_prop->pkt_type,
- fw_dump_level_switch_prop->num_prop,
- fw_dump_level_switch_prop->prop_data[0],
- fw_dump_level_switch_prop->prop_data[1]);
-
- hfi_write_cmd(prop);
- kfree(prop);
- return 0;
-}
-
void hfi_send_system_cmd(uint32_t type, uint64_t data, uint32_t size)
{
switch (type) {
@@ -556,7 +520,7 @@ int cam_hfi_resume(struct hfi_mem_info *hfi_mem,
return -EINVAL;
}
- cam_io_w_mb((uint32_t)INTR_ENABLE,
+ cam_io_w_mb((uint32_t)(INTR_ENABLE|INTR_ENABLE_WD0),
icp_base + HFI_REG_A5_CSR_A2HOSTINTEN);
fw_version = cam_io_r(icp_base + HFI_REG_FW_VERSION);
@@ -566,6 +530,8 @@ int cam_hfi_resume(struct hfi_mem_info *hfi_mem,
CAM_DBG(CAM_HFI, "wfi status = %x", (int)data);
cam_io_w_mb((uint32_t)hfi_mem->qtbl.iova, icp_base + HFI_REG_QTBL_PTR);
+ cam_io_w_mb((uint32_t)hfi_mem->sfr_buf.iova,
+ icp_base + HFI_REG_SFR_PTR);
cam_io_w_mb((uint32_t)hfi_mem->shmem.iova,
icp_base + HFI_REG_SHARED_MEM_PTR);
cam_io_w_mb((uint32_t)hfi_mem->shmem.len,
@@ -591,6 +557,7 @@ int cam_hfi_init(uint8_t event_driven_mode, struct hfi_mem_info *hfi_mem,
struct hfi_q_hdr *cmd_q_hdr, *msg_q_hdr, *dbg_q_hdr;
uint32_t hw_version, soc_version, fw_version, status = 0;
uint32_t retry_cnt = 0;
+ struct sfr_buf *sfr_buffer;
mutex_lock(&hfi_cmd_q_mutex);
mutex_lock(&hfi_msg_q_mutex);
@@ -605,7 +572,8 @@ int cam_hfi_init(uint8_t event_driven_mode, struct hfi_mem_info *hfi_mem,
if (g_hfi->hfi_state != HFI_DEINIT) {
CAM_ERR(CAM_HFI, "hfi_init: invalid state");
- return -EINVAL;
+ rc = -EINVAL;
+ goto alloc_fail;
}
memcpy(&g_hfi->map, hfi_mem, sizeof(g_hfi->map));
@@ -672,6 +640,9 @@ int cam_hfi_init(uint8_t event_driven_mode, struct hfi_mem_info *hfi_mem,
dbg_q_hdr->qhdr_read_idx = RESET;
dbg_q_hdr->qhdr_write_idx = RESET;
+ sfr_buffer = (struct sfr_buf *)hfi_mem->sfr_buf.kva;
+ sfr_buffer->size = ICP_MSG_SFR_SIZE_IN_BYTES;
+
switch (event_driven_mode) {
case INTR_MODE:
cmd_q_hdr->qhdr_type = Q_CMD;
@@ -744,7 +715,10 @@ int cam_hfi_init(uint8_t event_driven_mode, struct hfi_mem_info *hfi_mem,
break;
}
- cam_io_w_mb((uint32_t)hfi_mem->qtbl.iova, icp_base + HFI_REG_QTBL_PTR);
+ cam_io_w_mb((uint32_t)hfi_mem->qtbl.iova,
+ icp_base + HFI_REG_QTBL_PTR);
+ cam_io_w_mb((uint32_t)hfi_mem->sfr_buf.iova,
+ icp_base + HFI_REG_SFR_PTR);
cam_io_w_mb((uint32_t)hfi_mem->shmem.iova,
icp_base + HFI_REG_SHARED_MEM_PTR);
cam_io_w_mb((uint32_t)hfi_mem->shmem.len,
@@ -801,7 +775,7 @@ int cam_hfi_init(uint8_t event_driven_mode, struct hfi_mem_info *hfi_mem,
g_hfi->hfi_state = HFI_READY;
g_hfi->cmd_q_state = true;
g_hfi->msg_q_state = true;
- cam_io_w_mb((uint32_t)INTR_ENABLE,
+ cam_io_w_mb((uint32_t)(INTR_ENABLE|INTR_ENABLE_WD0),
icp_base + HFI_REG_A5_CSR_A2HOSTINTEN);
mutex_unlock(&hfi_cmd_q_mutex);
diff --git a/drivers/media/platform/msm/camera/cam_icp/icp_hw/a5_hw/a5_core.h b/drivers/media/platform/msm/camera/cam_icp/icp_hw/a5_hw/a5_core.h
index 4aa6b4b..f4bc813 100644
--- a/drivers/media/platform/msm/camera/cam_icp/icp_hw/a5_hw/a5_core.h
+++ b/drivers/media/platform/msm/camera/cam_icp/icp_hw/a5_hw/a5_core.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -25,8 +25,8 @@
#define A5_CSR_BASE 2
#define A5_HOST_INT 0x1
-#define A5_WDT_0 0x10
-#define A5_WDT_1 0x100
+#define A5_WDT_0 0x2
+#define A5_WDT_1 0x4
#define ELF_GUARD_PAGE (2 * 1024 * 1024)
diff --git a/drivers/media/platform/msm/camera/cam_icp/icp_hw/bps_hw/bps_core.c b/drivers/media/platform/msm/camera/cam_icp/icp_hw/bps_hw/bps_core.c
index 5bd7f1c..d01637436 100644
--- a/drivers/media/platform/msm/camera/cam_icp/icp_hw/bps_hw/bps_core.c
+++ b/drivers/media/platform/msm/camera/cam_icp/icp_hw/bps_hw/bps_core.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -18,6 +18,7 @@
#include <linux/firmware.h>
#include <linux/delay.h>
#include <linux/timer.h>
+#include <linux/iopoll.h>
#include "cam_io_util.h"
#include "cam_hw.h"
#include "cam_hw_intf.h"
@@ -30,6 +31,9 @@
#include "cam_icp_hw_mgr_intf.h"
#include "cam_cpas_api.h"
#include "cam_debug_util.h"
+#include "hfi_reg.h"
+
+#define HFI_MAX_POLL_TRY 5
static int cam_bps_cpas_vote(struct cam_bps_device_core_info *core_info,
struct cam_icp_cpas_vote *cpas_vote)
@@ -210,6 +214,77 @@ static int cam_bps_handle_resume(struct cam_hw_info *bps_dev)
return rc;
}
+static int cam_bps_cmd_reset(struct cam_hw_soc_info *soc_info,
+ struct cam_bps_device_core_info *core_info)
+{
+ uint32_t retry_cnt = 0;
+ uint32_t status = 0;
+ int pwr_ctrl, pwr_status, rc = 0;
+ bool reset_bps_cdm_fail = false;
+ bool reset_bps_top_fail = false;
+
+ CAM_DBG(CAM_ICP, "CAM_ICP_BPS_CMD_RESET");
+ /* Reset BPS CDM core*/
+ cam_io_w_mb((uint32_t)0xF,
+ soc_info->reg_map[0].mem_base + BPS_CDM_RST_CMD);
+ while (retry_cnt < HFI_MAX_POLL_TRY) {
+ readw_poll_timeout((soc_info->reg_map[0].mem_base +
+ BPS_CDM_IRQ_STATUS),
+ status, ((status & BPS_RST_DONE_IRQ_STATUS_BIT) == 0x1),
+ 100, 10000);
+
+ CAM_DBG(CAM_ICP, "bps_cdm_irq_status = %u", status);
+
+ if ((status & BPS_RST_DONE_IRQ_STATUS_BIT) == 0x1)
+ break;
+ retry_cnt++;
+ }
+ status = cam_io_r_mb(soc_info->reg_map[0].mem_base +
+ BPS_CDM_IRQ_STATUS);
+ if ((status & BPS_RST_DONE_IRQ_STATUS_BIT) != 0x1) {
+ CAM_ERR(CAM_ICP, "BPS CDM rst failed status 0x%x", status);
+ reset_bps_cdm_fail = true;
+ }
+
+ /* Reset BPS core*/
+ status = 0;
+ cam_io_w_mb((uint32_t)0x3,
+ soc_info->reg_map[0].mem_base + BPS_TOP_RST_CMD);
+ while (retry_cnt < HFI_MAX_POLL_TRY) {
+ readw_poll_timeout((soc_info->reg_map[0].mem_base +
+ BPS_TOP_IRQ_STATUS),
+ status, ((status & BPS_RST_DONE_IRQ_STATUS_BIT) == 0x1),
+ 100, 10000);
+
+ CAM_DBG(CAM_ICP, "bps_top_irq_status = %u", status);
+
+ if ((status & BPS_RST_DONE_IRQ_STATUS_BIT) == 0x1)
+ break;
+ retry_cnt++;
+ }
+ status = cam_io_r_mb(soc_info->reg_map[0].mem_base +
+ BPS_TOP_IRQ_STATUS);
+ if ((status & BPS_RST_DONE_IRQ_STATUS_BIT) != 0x1) {
+ CAM_ERR(CAM_ICP, "BPS top rst failed status 0x%x", status);
+ reset_bps_top_fail = true;
+ }
+
+ cam_bps_get_gdsc_control(soc_info);
+ cam_cpas_reg_read(core_info->cpas_handle,
+ CAM_CPAS_REG_CPASTOP, core_info->bps_hw_info->pwr_ctrl,
+ true, &pwr_ctrl);
+ cam_cpas_reg_read(core_info->cpas_handle,
+ CAM_CPAS_REG_CPASTOP, core_info->bps_hw_info->pwr_status,
+ true, &pwr_status);
+ CAM_DBG(CAM_ICP, "(After) pwr_ctrl = %x pwr_status = %x",
+ pwr_ctrl, pwr_status);
+
+ if (reset_bps_cdm_fail || reset_bps_top_fail)
+ rc = -EAGAIN;
+
+ return rc;
+}
+
int cam_bps_process_cmd(void *device_priv, uint32_t cmd_type,
void *cmd_args, uint32_t arg_size)
{
@@ -311,7 +386,12 @@ int cam_bps_process_cmd(void *device_priv, uint32_t cmd_type,
cam_bps_toggle_clk(soc_info, false);
core_info->clk_enable = false;
break;
+ case CAM_ICP_BPS_CMD_RESET:
+ rc = cam_bps_cmd_reset(soc_info, core_info);
+ break;
default:
+ CAM_ERR(CAM_ICP, "Invalid Cmd Type:%u", cmd_type);
+ rc = -EINVAL;
break;
}
return rc;
diff --git a/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/cam_icp_hw_mgr.c b/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/cam_icp_hw_mgr.c
index 16e97ea..f00c32c 100644
--- a/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/cam_icp_hw_mgr.c
+++ b/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/cam_icp_hw_mgr.c
@@ -57,8 +57,6 @@
#define ICP_DEV_TYPE_TO_CLK_TYPE(dev_type) \
((dev_type == CAM_ICP_RES_TYPE_BPS) ? ICP_CLK_HW_BPS : ICP_CLK_HW_IPE)
-#define ICP_DEVICE_IDLE_TIMEOUT 400
-
static struct cam_icp_hw_mgr icp_hw_mgr;
static int cam_icp_send_ubwc_cfg(struct cam_icp_hw_mgr *hw_mgr)
@@ -247,38 +245,6 @@ static int cam_icp_ctx_clk_info_init(struct cam_icp_hw_ctx_data *ctx_data)
return 0;
}
-static bool cam_icp_frame_pending(struct cam_icp_hw_ctx_data *ctx_data)
-{
- return !bitmap_empty(ctx_data->hfi_frame_process.bitmap,
- CAM_FRAME_CMD_MAX);
-}
-
-static int cam_icp_ctx_timer_reset(struct cam_icp_hw_ctx_data *ctx_data)
-{
- if (ctx_data && ctx_data->watch_dog) {
- ctx_data->watch_dog_reset_counter++;
- CAM_DBG(CAM_ICP, "reset timer : ctx_id = %d, counter=%d",
- ctx_data->ctx_id, ctx_data->watch_dog_reset_counter);
- crm_timer_reset(ctx_data->watch_dog);
- }
-
- return 0;
-}
-
-static void cam_icp_device_timer_reset(struct cam_icp_hw_mgr *hw_mgr,
- int device_index)
-{
- if ((device_index >= ICP_CLK_HW_MAX) || (!hw_mgr))
- return;
-
- if (hw_mgr->clk_info[device_index].watch_dog) {
- CAM_DBG(CAM_ICP, "reset timer : device_index = %d",
- device_index);
- crm_timer_reset(hw_mgr->clk_info[device_index].watch_dog);
- hw_mgr->clk_info[device_index].watch_dog_reset_counter++;
- }
-}
-
static int32_t cam_icp_deinit_idle_clk(void *priv, void *data)
{
struct cam_icp_hw_mgr *hw_mgr = (struct cam_icp_hw_mgr *)priv;
@@ -293,8 +259,6 @@ static int32_t cam_icp_deinit_idle_clk(void *priv, void *data)
struct cam_hw_intf *bps_dev_intf = NULL;
struct cam_hw_intf *dev_intf = NULL;
struct cam_a5_clk_update_cmd clk_upd_cmd;
- int rc = 0;
- bool busy = false;
ipe0_dev_intf = hw_mgr->ipe0_dev_intf;
ipe1_dev_intf = hw_mgr->ipe1_dev_intf;
@@ -304,35 +268,19 @@ static int32_t cam_icp_deinit_idle_clk(void *priv, void *data)
clk_info->curr_clk = 0;
clk_info->over_clked = 0;
- mutex_lock(&hw_mgr->hw_mgr_mutex);
-
for (i = 0; i < CAM_ICP_CTX_MAX; i++) {
ctx_data = &hw_mgr->ctx_data[i];
mutex_lock(&ctx_data->ctx_mutex);
if ((ctx_data->state == CAM_ICP_CTX_STATE_ACQUIRED) &&
- (ICP_DEV_TYPE_TO_CLK_TYPE(
- ctx_data->icp_dev_acquire_info->dev_type)
- == clk_info->hw_type)) {
- busy = cam_icp_frame_pending(ctx_data);
- if (busy) {
- mutex_unlock(&ctx_data->ctx_mutex);
- break;
- }
+ (ICP_DEV_TYPE_TO_CLK_TYPE(ctx_data->
+ icp_dev_acquire_info->dev_type) == clk_info->hw_type))
cam_icp_ctx_clk_info_init(ctx_data);
- }
mutex_unlock(&ctx_data->ctx_mutex);
}
- if (busy) {
- cam_icp_device_timer_reset(hw_mgr, clk_info->hw_type);
- rc = -EBUSY;
- goto done;
- }
-
if ((!ipe0_dev_intf) || (!bps_dev_intf)) {
CAM_ERR(CAM_ICP, "dev intfs are wrong, failed to update clk");
- rc = -EINVAL;
- goto done;
+ return -EINVAL;
}
if (clk_info->hw_type == ICP_CLK_HW_BPS) {
@@ -343,7 +291,7 @@ static int32_t cam_icp_deinit_idle_clk(void *priv, void *data)
id = CAM_ICP_IPE_CMD_DISABLE_CLK;
} else {
CAM_ERR(CAM_ICP, "Error");
- goto done;
+ return 0;
}
CAM_DBG(CAM_ICP, "Disable %d", clk_info->hw_type);
@@ -360,9 +308,7 @@ static int32_t cam_icp_deinit_idle_clk(void *priv, void *data)
&clk_upd_cmd,
sizeof(struct cam_a5_clk_update_cmd));
-done:
- mutex_unlock(&hw_mgr->hw_mgr_mutex);
- return rc;
+ return 0;
}
static int32_t cam_icp_ctx_timer(void *priv, void *data)
@@ -393,12 +339,6 @@ static int32_t cam_icp_ctx_timer(void *priv, void *data)
return 0;
}
- if (cam_icp_frame_pending(ctx_data)) {
- cam_icp_ctx_timer_reset(ctx_data);
- mutex_unlock(&ctx_data->ctx_mutex);
- return -EBUSY;
- }
-
CAM_DBG(CAM_ICP,
"E :ctx_id = %d ubw = %lld cbw = %lld curr_fc = %u bc = %u",
ctx_data->ctx_id,
@@ -440,8 +380,8 @@ static int32_t cam_icp_ctx_timer(void *priv, void *data)
ctx_data->clk_info.base_clk = 0;
clk_update.ahb_vote.type = CAM_VOTE_DYNAMIC;
- clk_update.ahb_vote.vote.freq = 0;
- clk_update.ahb_vote_valid = false;
+ clk_update.ahb_vote.vote.freq = clk_info->curr_clk;
+ clk_update.ahb_vote_valid = true;
clk_update.axi_vote.compressed_bw = clk_info->compressed_bw;
clk_update.axi_vote.uncompressed_bw = clk_info->uncompressed_bw;
clk_update.axi_vote_valid = true;
@@ -551,7 +491,7 @@ static int cam_icp_device_timer_start(struct cam_icp_hw_mgr *hw_mgr)
for (i = 0; i < ICP_CLK_HW_MAX; i++) {
if (!hw_mgr->clk_info[i].watch_dog) {
rc = crm_timer_init(&hw_mgr->clk_info[i].watch_dog,
- ICP_DEVICE_IDLE_TIMEOUT, &hw_mgr->clk_info[i],
+ 3000, &hw_mgr->clk_info[i],
&cam_icp_device_timer_cb);
if (rc)
@@ -593,6 +533,30 @@ static void cam_icp_device_timer_stop(struct cam_icp_hw_mgr *hw_mgr)
}
}
+static int cam_icp_ctx_timer_reset(struct cam_icp_hw_ctx_data *ctx_data)
+{
+ if (ctx_data && ctx_data->watch_dog) {
+ ctx_data->watch_dog_reset_counter++;
+ CAM_DBG(CAM_ICP, "reset timer : ctx_id = %d, counter=%d",
+ ctx_data->ctx_id, ctx_data->watch_dog_reset_counter);
+ crm_timer_reset(ctx_data->watch_dog);
+ }
+
+ return 0;
+}
+
+static void cam_icp_device_timer_reset(struct cam_icp_hw_mgr *hw_mgr,
+ int device_index)
+{
+ if ((device_index >= ICP_CLK_HW_MAX) || (!hw_mgr))
+ return;
+
+ if (hw_mgr->clk_info[device_index].watch_dog) {
+ crm_timer_reset(hw_mgr->clk_info[device_index].watch_dog);
+ hw_mgr->clk_info[device_index].watch_dog_reset_counter++;
+ }
+}
+
static uint32_t cam_icp_mgr_calc_base_clk(uint32_t frame_cycles,
uint64_t budget)
{
@@ -1042,8 +1006,8 @@ static int cam_icp_update_cpas_vote(struct cam_icp_hw_mgr *hw_mgr,
}
clk_update.ahb_vote.type = CAM_VOTE_DYNAMIC;
- clk_update.ahb_vote.vote.freq = 0;
- clk_update.ahb_vote_valid = false;
+ clk_update.ahb_vote.vote.freq = clk_info->curr_clk;
+ clk_update.ahb_vote_valid = true;
clk_update.axi_vote.compressed_bw = clk_info->compressed_bw;
clk_update.axi_vote.uncompressed_bw = clk_info->uncompressed_bw;
clk_update.axi_vote_valid = true;
@@ -1179,7 +1143,7 @@ static int cam_icp_mgr_ipe_bps_power_collapse(struct cam_icp_hw_mgr *hw_mgr,
if (hw_mgr->bps_ctxt_cnt)
goto end;
- if (icp_hw_mgr.ipe_bps_pc_flag) {
+ if (icp_hw_mgr.ipe_bps_pc_flag && !hw_mgr->recovery) {
rc = bps_dev_intf->hw_ops.process_cmd(
bps_dev_intf->hw_priv,
CAM_ICP_BPS_CMD_POWER_COLLAPSE,
@@ -1201,11 +1165,10 @@ static int cam_icp_mgr_ipe_bps_power_collapse(struct cam_icp_hw_mgr *hw_mgr,
if (hw_mgr->ipe_ctxt_cnt)
goto end;
- if (icp_hw_mgr.ipe_bps_pc_flag) {
+ if (icp_hw_mgr.ipe_bps_pc_flag && !hw_mgr->recovery) {
rc = ipe0_dev_intf->hw_ops.process_cmd(
ipe0_dev_intf->hw_priv,
CAM_ICP_IPE_CMD_POWER_COLLAPSE, NULL, 0);
-
}
if (hw_mgr->ipe_clk_state)
@@ -1213,7 +1176,7 @@ static int cam_icp_mgr_ipe_bps_power_collapse(struct cam_icp_hw_mgr *hw_mgr,
ipe0_dev_intf->hw_priv, NULL, 0);
if (ipe1_dev_intf) {
- if (icp_hw_mgr.ipe_bps_pc_flag) {
+ if (icp_hw_mgr.ipe_bps_pc_flag && !hw_mgr->recovery) {
rc = ipe1_dev_intf->hw_ops.process_cmd(
ipe1_dev_intf->hw_priv,
CAM_ICP_IPE_CMD_POWER_COLLAPSE,
@@ -1226,7 +1189,7 @@ static int cam_icp_mgr_ipe_bps_power_collapse(struct cam_icp_hw_mgr *hw_mgr,
}
hw_mgr->ipe_clk_state = false;
- if (icp_hw_mgr.ipe_bps_pc_flag) {
+ if (icp_hw_mgr.ipe_bps_pc_flag && !hw_mgr->recovery) {
hw_mgr->core_info = hw_mgr->core_info &
(~(ICP_PWR_CLP_IPE0 | ICP_PWR_CLP_IPE1));
}
@@ -1285,22 +1248,6 @@ static int cam_icp_get_a5_dbg_type(void *data, u64 *val)
DEFINE_SIMPLE_ATTRIBUTE(cam_icp_debug_type_fs, cam_icp_get_a5_dbg_type,
cam_icp_set_a5_dbg_type, "%08llu");
-static int cam_icp_set_a5_fw_dump_lvl(void *data, u64 val)
-{
- if (val < NUM_HFI_DUMP_LVL)
- icp_hw_mgr.a5_fw_dump_lvl = val;
- return 0;
-}
-
-static int cam_icp_get_a5_fw_dump_lvl(void *data, u64 *val)
-{
- *val = icp_hw_mgr.a5_fw_dump_lvl;
- return 0;
-}
-
-DEFINE_SIMPLE_ATTRIBUTE(cam_icp_debug_fw_dump, cam_icp_get_a5_fw_dump_lvl,
- cam_icp_set_a5_fw_dump_lvl, "%08llu");
-
static int cam_icp_hw_mgr_create_debugfs_entry(void)
{
int rc = 0;
@@ -1317,7 +1264,6 @@ static int cam_icp_hw_mgr_create_debugfs_entry(void)
rc = -ENOMEM;
goto err;
}
- icp_hw_mgr.icp_pc_flag = false;
if (!debugfs_create_bool("ipe_bps_pc",
0644,
@@ -1328,8 +1274,6 @@ static int cam_icp_hw_mgr_create_debugfs_entry(void)
goto err;
}
- icp_hw_mgr.ipe_bps_pc_flag = false;
-
if (!debugfs_create_file("icp_debug_clk",
0644,
icp_hw_mgr.dentry, NULL,
@@ -1351,7 +1295,7 @@ static int cam_icp_hw_mgr_create_debugfs_entry(void)
0644,
icp_hw_mgr.dentry,
NULL, &cam_icp_debug_type_fs)) {
- CAM_ERR(CAM_ICP, "failed to create a5_debug_type");
+ CAM_ERR(CAM_ICP, "failed to create a5_debug_type\n");
rc = -ENOMEM;
goto err;
}
@@ -1360,16 +1304,7 @@ static int cam_icp_hw_mgr_create_debugfs_entry(void)
0644,
icp_hw_mgr.dentry,
NULL, &cam_icp_debug_fs)) {
- CAM_ERR(CAM_ICP, "failed to create a5_dbg_lvl");
- rc = -ENOMEM;
- goto err;
- }
-
- if (!debugfs_create_file("a5_fw_dump_lvl",
- 0644,
- icp_hw_mgr.dentry,
- NULL, &cam_icp_debug_fw_dump)) {
- CAM_ERR(CAM_ICP, "failed to create a5_fw_dump_lvl");
+ CAM_ERR(CAM_ICP, "failed to create a5_dbg_lvl\n");
rc = -ENOMEM;
goto err;
}
@@ -1459,6 +1394,10 @@ static int cam_icp_mgr_handle_frame_process(uint32_t *msg_ptr, int flag)
CAM_DBG(CAM_ICP, "ctx : %pK, request_id :%lld",
(void *)ctx_data->context_priv, request_id);
+ clk_type = ICP_DEV_TYPE_TO_CLK_TYPE(ctx_data->icp_dev_acquire_info->
+ dev_type);
+ cam_icp_device_timer_reset(&icp_hw_mgr, clk_type);
+
mutex_lock(&ctx_data->ctx_mutex);
cam_icp_ctx_timer_reset(ctx_data);
if (ctx_data->state != CAM_ICP_CTX_STATE_ACQUIRED) {
@@ -1468,10 +1407,6 @@ static int cam_icp_mgr_handle_frame_process(uint32_t *msg_ptr, int flag)
return 0;
}
- clk_type = ICP_DEV_TYPE_TO_CLK_TYPE(
- ctx_data->icp_dev_acquire_info->dev_type);
- cam_icp_device_timer_reset(&icp_hw_mgr, clk_type);
-
hfi_frame_process = &ctx_data->hfi_frame_process;
for (i = 0; i < CAM_FRAME_CMD_MAX; i++)
if (hfi_frame_process->request_id[i] == request_id)
@@ -1718,11 +1653,121 @@ static int cam_icp_mgr_process_direct_ack_msg(uint32_t *msg_ptr)
return rc;
}
+static int cam_icp_ipebps_reset(struct cam_icp_hw_mgr *hw_mgr)
+{
+ int rc = 0;
+ struct cam_hw_intf *ipe0_dev_intf;
+ struct cam_hw_intf *ipe1_dev_intf;
+ struct cam_hw_intf *bps_dev_intf;
+
+ ipe0_dev_intf = hw_mgr->ipe0_dev_intf;
+ ipe1_dev_intf = hw_mgr->ipe1_dev_intf;
+ bps_dev_intf = hw_mgr->bps_dev_intf;
+
+ rc = bps_dev_intf->hw_ops.process_cmd(
+ bps_dev_intf->hw_priv,
+ CAM_ICP_BPS_CMD_RESET,
+ NULL, 0);
+ if (rc)
+ CAM_ERR(CAM_ICP, "bps reset failed");
+
+ rc = ipe0_dev_intf->hw_ops.process_cmd(
+ ipe0_dev_intf->hw_priv,
+ CAM_ICP_IPE_CMD_RESET,
+ NULL, 0);
+ if (rc)
+ CAM_ERR(CAM_ICP, "ipe0 reset failed");
+
+ if (ipe1_dev_intf) {
+ rc = ipe1_dev_intf->hw_ops.process_cmd(
+ ipe1_dev_intf->hw_priv,
+ CAM_ICP_IPE_CMD_RESET,
+ NULL, 0);
+ if (rc)
+ CAM_ERR(CAM_ICP, "ipe1 reset failed");
+ }
+
+ return 0;
+}
+
+static int cam_icp_mgr_trigger_recovery(struct cam_icp_hw_mgr *hw_mgr)
+{
+ int rc = 0;
+ int i = 0;
+ struct sfr_buf *sfr_buffer = NULL;
+
+ CAM_DBG(CAM_ICP, "Enter");
+
+ mutex_lock(&hw_mgr->hw_mgr_mutex);
+ if (hw_mgr->recovery) {
+ CAM_ERR(CAM_ICP, "Recovery is set");
+ mutex_unlock(&hw_mgr->hw_mgr_mutex);
+ return rc;
+ }
+
+ sfr_buffer = (struct sfr_buf *)icp_hw_mgr.hfi_mem.sfr_buf.kva;
+ CAM_WARN(CAM_ICP, "SFR:%s", sfr_buffer->msg);
+
+ cam_icp_ipebps_reset(hw_mgr);
+
+ hw_mgr->recovery = true;
+
+ if (hw_mgr->clk_info[ICP_CLK_HW_BPS].watch_dog) {
+ hw_mgr->clk_info[ICP_CLK_HW_BPS].watch_dog_reset_counter = 0;
+ crm_timer_exit(&hw_mgr->clk_info[ICP_CLK_HW_BPS].watch_dog);
+ hw_mgr->clk_info[ICP_CLK_HW_BPS].watch_dog = NULL;
+ }
+ if (hw_mgr->clk_info[ICP_CLK_HW_IPE].watch_dog) {
+ hw_mgr->clk_info[ICP_CLK_HW_IPE].watch_dog_reset_counter = 0;
+ crm_timer_exit(&hw_mgr->clk_info[ICP_CLK_HW_IPE].watch_dog);
+ hw_mgr->clk_info[ICP_CLK_HW_IPE].watch_dog = NULL;
+ }
+
+ for (i = 0; i < CAM_ICP_CTX_MAX; i++) {
+ mutex_lock(&hw_mgr->ctx_data[i].ctx_mutex);
+ if (hw_mgr->ctx_data[i].state != CAM_ICP_CTX_STATE_RELEASE)
+ cam_icp_ctx_timer_stop(&hw_mgr->ctx_data[i]);
+ mutex_unlock(&hw_mgr->ctx_data[i].ctx_mutex);
+ }
+
+ mutex_unlock(&hw_mgr->hw_mgr_mutex);
+
+ CAM_DBG(CAM_ICP, "Done");
+ return rc;
+}
+static int cam_icp_mgr_process_fatal_error(
+ struct cam_icp_hw_mgr *hw_mgr, uint32_t *msg_ptr)
+{
+ struct hfi_msg_event_notify *event_notify;
+ int rc = 0;
+
+ CAM_DBG(CAM_ICP, "Enter");
+
+ event_notify = (struct hfi_msg_event_notify *)msg_ptr;
+ if (!event_notify) {
+ CAM_ERR(CAM_ICP, "Empty event message");
+ return -EINVAL;
+ }
+
+ CAM_DBG(CAM_ICP, "evt_id: %u evt_data1: %u evt_data2: %u",
+ event_notify->event_id,
+ event_notify->event_data1,
+ event_notify->event_data2);
+
+ if (event_notify->event_id == HFI_EVENT_SYS_ERROR) {
+ CAM_INFO(CAM_ICP, "received HFI_EVENT_SYS_ERROR");
+ rc = cam_icp_mgr_trigger_recovery(hw_mgr);
+ }
+
+ return rc;
+}
+
static void cam_icp_mgr_process_dbg_buf(void)
{
uint32_t *msg_ptr = NULL, *pkt_ptr = NULL;
struct hfi_msg_debug *dbg_msg;
uint32_t read_len, size_processed = 0;
+ uint64_t timestamp = 0;
char *dbg_buf;
int rc = 0;
@@ -1736,7 +1781,9 @@ static void cam_icp_mgr_process_dbg_buf(void)
if (pkt_ptr[ICP_PACKET_TYPE] == HFI_MSG_SYS_DEBUG) {
dbg_msg = (struct hfi_msg_debug *)pkt_ptr;
dbg_buf = (char *)&dbg_msg->msg_data;
- trace_cam_icp_fw_dbg(dbg_buf);
+ timestamp = ((((uint64_t)(dbg_msg->timestamp_hi) << 32)
+ | dbg_msg->timestamp_lo) >> 16);
+ trace_cam_icp_fw_dbg(dbg_buf, timestamp/2);
}
size_processed += (pkt_ptr[ICP_PACKET_SIZE] >>
BYTE_WORD_SHIFT);
@@ -1802,6 +1849,10 @@ static int cam_icp_process_msg_pkt_type(
CAM_DBG(CAM_ICP, "received EVENT_NOTIFY");
size_processed = (
(struct hfi_msg_event_notify *)msg_ptr)->size;
+ rc = cam_icp_mgr_process_fatal_error(hw_mgr, msg_ptr);
+ if (rc)
+ CAM_ERR(CAM_ICP, "failed in processing evt notify");
+
break;
default:
@@ -1862,6 +1913,13 @@ static int32_t cam_icp_mgr_process_msg(void *priv, void *data)
HFI_DEBUG_MODE_QUEUE)
cam_icp_mgr_process_dbg_buf();
+ if ((task_data->irq_status & A5_WDT_0) ||
+ (task_data->irq_status & A5_WDT_1)) {
+ CAM_ERR_RATE_LIMIT(CAM_ICP, "watch dog interrupt from A5");
+
+ rc = cam_icp_mgr_trigger_recovery(hw_mgr);
+ }
+
return rc;
}
@@ -1945,6 +2003,31 @@ static int cam_icp_alloc_secheap_mem(struct cam_mem_mgr_memory_desc *secheap)
return rc;
}
+static int cam_icp_alloc_sfr_mem(struct cam_mem_mgr_memory_desc *sfr)
+{
+ int rc;
+ struct cam_mem_mgr_request_desc alloc;
+ struct cam_mem_mgr_memory_desc out;
+
+ memset(&alloc, 0, sizeof(alloc));
+ memset(&out, 0, sizeof(out));
+ alloc.size = SZ_8K;
+ alloc.align = 0;
+ alloc.flags = CAM_MEM_FLAG_HW_READ_WRITE |
+ CAM_MEM_FLAG_HW_SHARED_ACCESS;
+
+ alloc.smmu_hdl = icp_hw_mgr.iommu_hdl;
+ rc = cam_mem_mgr_request_mem(&alloc, &out);
+ if (rc)
+ return rc;
+
+ *sfr = out;
+ CAM_DBG(CAM_ICP, "kva: %llX, iova: %x, hdl: %x, len: %lld",
+ out.kva, out.iova, out.mem_handle, out.len);
+
+ return rc;
+}
+
static int cam_icp_alloc_shared_mem(struct cam_mem_mgr_memory_desc *qtbl)
{
int rc;
@@ -2060,6 +2143,12 @@ static int cam_icp_allocate_hfi_mem(void)
goto dbg_q_alloc_failed;
}
+ rc = cam_icp_alloc_sfr_mem(&icp_hw_mgr.hfi_mem.sfr_buf);
+ if (rc) {
+ CAM_ERR(CAM_ICP, "Unable to allocate sfr buffer");
+ goto sfr_buf_alloc_failed;
+ }
+
rc = cam_icp_alloc_secheap_mem(&icp_hw_mgr.hfi_mem.sec_heap);
if (rc) {
CAM_ERR(CAM_ICP, "Unable to allocate sec heap memory");
@@ -2068,6 +2157,8 @@ static int cam_icp_allocate_hfi_mem(void)
return rc;
sec_heap_alloc_failed:
+ cam_mem_mgr_release_mem(&icp_hw_mgr.hfi_mem.sfr_buf);
+sfr_buf_alloc_failed:
cam_mem_mgr_release_mem(&icp_hw_mgr.hfi_mem.dbg_q);
dbg_q_alloc_failed:
cam_mem_mgr_release_mem(&icp_hw_mgr.hfi_mem.msg_q);
@@ -2212,7 +2303,7 @@ static int cam_icp_mgr_icp_power_collapse(struct cam_icp_hw_mgr *hw_mgr)
}
a5_dev = (struct cam_hw_info *)a5_dev_intf->hw_priv;
- if (!hw_mgr->icp_pc_flag) {
+ if (!hw_mgr->icp_pc_flag || hw_mgr->recovery) {
cam_hfi_disable_cpu(
a5_dev->soc_info.reg_map[A5_SIERRA_BASE].mem_base);
rc = cam_icp_mgr_hw_close_k(hw_mgr, NULL);
@@ -2243,27 +2334,34 @@ static int cam_icp_mgr_hfi_resume(struct cam_icp_hw_mgr *hw_mgr)
hfi_mem.qtbl.kva = icp_hw_mgr.hfi_mem.qtbl.kva;
hfi_mem.qtbl.iova = icp_hw_mgr.hfi_mem.qtbl.iova;
hfi_mem.qtbl.len = icp_hw_mgr.hfi_mem.qtbl.len;
- CAM_DBG(CAM_ICP, "kva = %llX IOVA = %X length = %lld\n",
+ CAM_DBG(CAM_ICP, "qtbl kva = %llX IOVA = %X length = %lld\n",
hfi_mem.qtbl.kva, hfi_mem.qtbl.iova, hfi_mem.qtbl.len);
hfi_mem.cmd_q.kva = icp_hw_mgr.hfi_mem.cmd_q.kva;
hfi_mem.cmd_q.iova = icp_hw_mgr.hfi_mem.cmd_q.iova;
hfi_mem.cmd_q.len = icp_hw_mgr.hfi_mem.cmd_q.len;
- CAM_DBG(CAM_ICP, "kva = %llX IOVA = %X length = %lld\n",
+ CAM_DBG(CAM_ICP, "cmd_q kva = %llX IOVA = %X length = %lld\n",
hfi_mem.cmd_q.kva, hfi_mem.cmd_q.iova, hfi_mem.cmd_q.len);
hfi_mem.msg_q.kva = icp_hw_mgr.hfi_mem.msg_q.kva;
hfi_mem.msg_q.iova = icp_hw_mgr.hfi_mem.msg_q.iova;
hfi_mem.msg_q.len = icp_hw_mgr.hfi_mem.msg_q.len;
- CAM_DBG(CAM_ICP, "kva = %llX IOVA = %X length = %lld\n",
+ CAM_DBG(CAM_ICP, "msg_q kva = %llX IOVA = %X length = %lld\n",
hfi_mem.msg_q.kva, hfi_mem.msg_q.iova, hfi_mem.msg_q.len);
hfi_mem.dbg_q.kva = icp_hw_mgr.hfi_mem.dbg_q.kva;
hfi_mem.dbg_q.iova = icp_hw_mgr.hfi_mem.dbg_q.iova;
hfi_mem.dbg_q.len = icp_hw_mgr.hfi_mem.dbg_q.len;
- CAM_DBG(CAM_ICP, "kva = %llX IOVA = %X length = %lld\n",
+ CAM_DBG(CAM_ICP, "dbg_q kva = %llX IOVA = %X length = %lld\n",
hfi_mem.dbg_q.kva, hfi_mem.dbg_q.iova, hfi_mem.dbg_q.len);
+ hfi_mem.sfr_buf.kva = icp_hw_mgr.hfi_mem.sfr_buf.kva;
+ hfi_mem.sfr_buf.iova = icp_hw_mgr.hfi_mem.sfr_buf.iova;
+ hfi_mem.sfr_buf.len = icp_hw_mgr.hfi_mem.sfr_buf.len;
+ CAM_DBG(CAM_ICP, "sfr kva = %llX IOVA = %X length = %lld\n",
+ hfi_mem.sfr_buf.kva, hfi_mem.sfr_buf.iova,
+ hfi_mem.sfr_buf.len);
+
hfi_mem.sec_heap.kva = icp_hw_mgr.hfi_mem.sec_heap.kva;
hfi_mem.sec_heap.iova = icp_hw_mgr.hfi_mem.sec_heap.iova;
hfi_mem.sec_heap.len = icp_hw_mgr.hfi_mem.sec_heap.len;
@@ -2283,22 +2381,15 @@ static int cam_icp_mgr_abort_handle(
{
int rc = 0;
unsigned long rem_jiffies;
- size_t packet_size;
int timeout = 100;
- struct hfi_cmd_ipebps_async *abort_cmd;
+ struct hfi_cmd_work_data *task_data;
+ struct hfi_cmd_ipebps_async *abort_cmd = ctx_data->abort_cmd;
+ struct crm_workq_task *task;
- packet_size =
- sizeof(struct hfi_cmd_ipebps_async) +
- sizeof(struct hfi_cmd_abort) -
- sizeof(((struct hfi_cmd_ipebps_async *)0)->payload.direct);
- abort_cmd = kzalloc(packet_size, GFP_KERNEL);
- CAM_DBG(CAM_ICP, "abort pkt size = %d", (int) packet_size);
- if (!abort_cmd) {
- rc = -ENOMEM;
- return rc;
- }
+ task = cam_req_mgr_workq_get_task(icp_hw_mgr.cmd_work);
+ if (!task)
+ return -ENOMEM;
- abort_cmd->size = packet_size;
abort_cmd->pkt_type = HFI_CMD_IPEBPS_ASYNC_COMMAND_DIRECT;
if (ctx_data->icp_dev_acquire_info->dev_type == CAM_ICP_RES_TYPE_BPS)
abort_cmd->opcode = HFI_IPEBPS_CMD_OPCODE_BPS_ABORT;
@@ -2311,9 +2402,14 @@ static int cam_icp_mgr_abort_handle(
abort_cmd->user_data1 = (uint64_t)ctx_data;
abort_cmd->user_data2 = (uint64_t)0x0;
- rc = hfi_write_cmd(abort_cmd);
+ task_data = (struct hfi_cmd_work_data *)task->payload;
+ task_data->data = (void *)abort_cmd;
+ task_data->request_id = 0;
+ task_data->type = ICP_WORKQ_TASK_CMD_TYPE;
+ task->process_cb = cam_icp_mgr_process_cmd;
+ rc = cam_req_mgr_workq_enqueue_task(task, &icp_hw_mgr,
+ CRM_TASK_PRIORITY_0);
if (rc) {
- kfree(abort_cmd);
return rc;
}
CAM_DBG(CAM_ICP, "fw_handle = %x ctx_data = %pK",
@@ -2325,7 +2421,6 @@ static int cam_icp_mgr_abort_handle(
CAM_ERR(CAM_ICP, "FW timeout/err in abort handle command");
}
- kfree(abort_cmd);
return rc;
}
@@ -2335,20 +2430,14 @@ static int cam_icp_mgr_destroy_handle(
int rc = 0;
int timeout = 100;
unsigned long rem_jiffies;
- size_t packet_size;
- struct hfi_cmd_ipebps_async *destroy_cmd;
+ struct hfi_cmd_work_data *task_data;
+ struct hfi_cmd_ipebps_async *destroy_cmd = ctx_data->destroy_cmd;
+ struct crm_workq_task *task;
- packet_size =
- sizeof(struct hfi_cmd_ipebps_async) +
- sizeof(struct hfi_cmd_abort_destroy) -
- sizeof(((struct hfi_cmd_ipebps_async *)0)->payload.direct);
- destroy_cmd = kzalloc(packet_size, GFP_KERNEL);
- if (!destroy_cmd) {
- rc = -ENOMEM;
- return rc;
- }
+ task = cam_req_mgr_workq_get_task(icp_hw_mgr.cmd_work);
+ if (!task)
+ return -ENOMEM;
- destroy_cmd->size = packet_size;
destroy_cmd->pkt_type = HFI_CMD_IPEBPS_ASYNC_COMMAND_DIRECT;
if (ctx_data->icp_dev_acquire_info->dev_type == CAM_ICP_RES_TYPE_BPS)
destroy_cmd->opcode = HFI_IPEBPS_CMD_OPCODE_BPS_DESTROY;
@@ -2363,9 +2452,14 @@ static int cam_icp_mgr_destroy_handle(
memcpy(destroy_cmd->payload.direct, &ctx_data->temp_payload,
sizeof(uint64_t));
- rc = hfi_write_cmd(destroy_cmd);
+ task_data = (struct hfi_cmd_work_data *)task->payload;
+ task_data->data = (void *)destroy_cmd;
+ task_data->request_id = 0;
+ task_data->type = ICP_WORKQ_TASK_CMD_TYPE;
+ task->process_cb = cam_icp_mgr_process_cmd;
+ rc = cam_req_mgr_workq_enqueue_task(task, &icp_hw_mgr,
+ CRM_TASK_PRIORITY_0);
if (rc) {
- kfree(destroy_cmd);
return rc;
}
CAM_DBG(CAM_ICP, "fw_handle = %x ctx_data = %pK",
@@ -2380,7 +2474,6 @@ static int cam_icp_mgr_destroy_handle(
HFI_DEBUG_MODE_QUEUE)
cam_icp_mgr_process_dbg_buf();
}
- kfree(destroy_cmd);
return rc;
}
@@ -2405,7 +2498,8 @@ static int cam_icp_mgr_release_ctx(struct cam_icp_hw_mgr *hw_mgr, int ctx_id)
cam_icp_mgr_ipe_bps_power_collapse(hw_mgr,
&hw_mgr->ctx_data[ctx_id], 0);
hw_mgr->ctx_data[ctx_id].state = CAM_ICP_CTX_STATE_RELEASE;
- CAM_DBG(CAM_ICP, "E: ctx_id = %d", ctx_id);
+ CAM_DBG(CAM_ICP, "E: ctx_id = %d recovery = %d",
+ ctx_id, hw_mgr->recovery);
cam_icp_mgr_abort_handle(&hw_mgr->ctx_data[ctx_id]);
cam_icp_mgr_destroy_handle(&hw_mgr->ctx_data[ctx_id]);
cam_icp_mgr_cleanup_ctx(&hw_mgr->ctx_data[ctx_id]);
@@ -2638,6 +2732,10 @@ static int cam_icp_mgr_hfi_init(struct cam_icp_hw_mgr *hw_mgr)
hfi_mem.dbg_q.iova = icp_hw_mgr.hfi_mem.dbg_q.iova;
hfi_mem.dbg_q.len = icp_hw_mgr.hfi_mem.dbg_q.len;
+ hfi_mem.sfr_buf.kva = icp_hw_mgr.hfi_mem.sfr_buf.kva;
+ hfi_mem.sfr_buf.iova = icp_hw_mgr.hfi_mem.sfr_buf.iova;
+ hfi_mem.sfr_buf.len = icp_hw_mgr.hfi_mem.sfr_buf.len;
+
hfi_mem.sec_heap.kva = icp_hw_mgr.hfi_mem.sec_heap.kva;
hfi_mem.sec_heap.iova = icp_hw_mgr.hfi_mem.sec_heap.iova;
hfi_mem.sec_heap.len = icp_hw_mgr.hfi_mem.sec_heap.len;
@@ -2797,6 +2895,7 @@ static int cam_icp_mgr_hw_open(void *hw_mgr_priv, void *download_fw_args)
hw_mgr->ctxt_cnt = 0;
hw_mgr->fw_download = true;
+ hw_mgr->recovery = false;
CAM_INFO(CAM_ICP, "FW download done successfully");
@@ -3000,8 +3099,6 @@ static int cam_icp_mgr_process_cmd_desc(struct cam_icp_hw_mgr *hw_mgr,
uint64_t cpu_addr = 0;
struct ipe_frame_process_data *frame_process_data = NULL;
struct bps_frame_process_data *bps_frame_process_data = NULL;
- struct frame_set *ipe_set = NULL;
- struct frame_buffer *bps_bufs = NULL;
cmd_desc = (struct cam_cmd_buf_desc *)
((uint32_t *) &packet->payload + packet->cmd_buf_offset/4);
@@ -3050,11 +3147,14 @@ static int cam_icp_mgr_process_cmd_desc(struct cam_icp_hw_mgr *hw_mgr,
frame_process_data->cdm_buffer_addr = 0;
frame_process_data->cdm_prog_base = 0;
for (i = 0; i < frame_process_data->frames_in_batch; i++) {
- ipe_set = &frame_process_data->framesets[i];
for (j = 0; j < IPE_IO_IMAGES_MAX; j++) {
for (k = 0; k < MAX_NUM_OF_IMAGE_PLANES; k++) {
- ipe_set->buffers[j].buf_ptr[k] = 0;
- ipe_set->buffers[j].meta_buf_ptr[k] = 0;
+ frame_process_data->
+ framesets[i].buffers[j].
+ buffer_ptr[k] = 0;
+ frame_process_data->
+ framesets[i].buffers[j].
+ meta_buffer_ptr[k] = 0;
}
}
}
@@ -3071,10 +3171,11 @@ static int cam_icp_mgr_process_cmd_desc(struct cam_icp_hw_mgr *hw_mgr,
bps_frame_process_data->strip_lib_out_addr = 0;
bps_frame_process_data->cdm_prog_addr = 0;
for (i = 0; i < BPS_IO_IMAGES_MAX; i++) {
- bps_bufs = &bps_frame_process_data->buffers[i];
for (j = 0; j < MAX_NUM_OF_IMAGE_PLANES; j++) {
- bps_bufs->buf_ptr[j] = 0;
- bps_bufs->meta_buf_ptr[j] = 0;
+ bps_frame_process_data->
+ buffers[i].buffer_ptr[j] = 0;
+ bps_frame_process_data->
+ buffers[i].meta_buffer_ptr[j] = 0;
}
}
}
@@ -3480,6 +3581,8 @@ static int cam_icp_mgr_hw_flush(void *hw_priv, void *hw_flush_args)
{
struct cam_hw_flush_args *flush_args = hw_flush_args;
struct cam_icp_hw_ctx_data *ctx_data;
+ struct cam_icp_hw_mgr *hw_mgr = hw_priv;
+ bool need_abort = false;
if ((!hw_priv) || (!hw_flush_args)) {
CAM_ERR(CAM_ICP, "Input params are Null:");
@@ -3501,8 +3604,12 @@ static int cam_icp_mgr_hw_flush(void *hw_priv, void *hw_flush_args)
switch (flush_args->flush_type) {
case CAM_FLUSH_TYPE_ALL:
- if (flush_args->num_req_active)
+ mutex_lock(&hw_mgr->hw_mgr_mutex);
+ need_abort = !hw_mgr->recovery && flush_args->num_req_active;
+ mutex_unlock(&hw_mgr->hw_mgr_mutex);
+ if (need_abort) {
cam_icp_mgr_abort_handle(ctx_data);
+ }
mutex_lock(&ctx_data->ctx_mutex);
cam_icp_mgr_flush_all(ctx_data, flush_args);
mutex_unlock(&ctx_data->ctx_mutex);
@@ -3540,7 +3647,7 @@ static int cam_icp_mgr_release_hw(void *hw_mgr_priv, void *release_hw_args)
return -EINVAL;
}
- CAM_DBG(CAM_ICP, "Enter");
+ CAM_DBG(CAM_ICP, "Enter recovery set %d", hw_mgr->recovery);
ctx_data = release_hw->ctxt_to_hw_map;
if (!ctx_data) {
CAM_ERR(CAM_ICP, "NULL ctx data");
@@ -3561,9 +3668,15 @@ static int cam_icp_mgr_release_hw(void *hw_mgr_priv, void *release_hw_args)
}
mutex_unlock(&hw_mgr->ctx_data[ctx_id].ctx_mutex);
- if (release_hw->active_req) {
- cam_icp_mgr_abort_handle(ctx_data);
- cam_icp_mgr_send_abort_status(ctx_data);
+ mutex_lock(&hw_mgr->hw_mgr_mutex);
+ if (!hw_mgr->recovery) {
+ mutex_unlock(&hw_mgr->hw_mgr_mutex);
+ if (release_hw->active_req) {
+ cam_icp_mgr_abort_handle(ctx_data);
+ cam_icp_mgr_send_abort_status(ctx_data);
+ }
+ } else {
+ mutex_unlock(&hw_mgr->hw_mgr_mutex);
}
mutex_lock(&hw_mgr->hw_mgr_mutex);
@@ -3577,9 +3690,13 @@ static int cam_icp_mgr_release_hw(void *hw_mgr_priv, void *release_hw_args)
}
mutex_unlock(&hw_mgr->hw_mgr_mutex);
- if (!hw_mgr->bps_ctxt_cnt || !hw_mgr->ipe_ctxt_cnt)
+ if ((!hw_mgr->bps_ctxt_cnt || !hw_mgr->ipe_ctxt_cnt))
cam_icp_device_timer_stop(hw_mgr);
+ kfree(ctx_data->abort_cmd);
+ ctx_data->abort_cmd = NULL;
+ kfree(ctx_data->destroy_cmd);
+ ctx_data->destroy_cmd = NULL;
CAM_DBG(CAM_ICP, "Exit");
return rc;
}
@@ -3808,6 +3925,7 @@ static int cam_icp_mgr_acquire_hw(void *hw_mgr_priv, void *acquire_hw_args)
uint32_t ctx_id = 0;
uint64_t io_buf_addr;
size_t io_buf_size;
+ size_t packet_size;
struct cam_icp_hw_mgr *hw_mgr = hw_mgr_priv;
struct cam_icp_hw_ctx_data *ctx_data = NULL;
struct cam_hw_acquire_args *args = acquire_hw_args;
@@ -3836,6 +3954,30 @@ static int cam_icp_mgr_acquire_hw(void *hw_mgr_priv, void *acquire_hw_args)
ctx_data = &hw_mgr->ctx_data[ctx_id];
ctx_data->ctx_id = ctx_id;
+ packet_size =
+ sizeof(struct hfi_cmd_ipebps_async) +
+ sizeof(struct hfi_cmd_abort) -
+ sizeof(((struct hfi_cmd_ipebps_async *)0)->payload.direct);
+ ctx_data->abort_cmd = kzalloc(packet_size, GFP_KERNEL);
+ CAM_DBG(CAM_ICP, "abort pkt size = %d", (int) packet_size);
+ if (!ctx_data->abort_cmd) {
+ return -ENOMEM;
+ }
+ ctx_data->abort_cmd->size = packet_size;
+
+ packet_size =
+ sizeof(struct hfi_cmd_ipebps_async) +
+ sizeof(struct hfi_cmd_abort_destroy) -
+ sizeof(((struct hfi_cmd_ipebps_async *)0)->payload.direct);
+ ctx_data->destroy_cmd = kzalloc(packet_size, GFP_KERNEL);
+ CAM_DBG(CAM_ICP, "destroy pkt size = %d", (int) packet_size);
+ if (!ctx_data->destroy_cmd) {
+ kfree(ctx_data->abort_cmd);
+ ctx_data->abort_cmd = NULL;
+ return -ENOMEM;
+ }
+ ctx_data->destroy_cmd->size = packet_size;
+
mutex_lock(&ctx_data->ctx_mutex);
rc = cam_icp_get_acquire_info(hw_mgr, args, ctx_data);
if (rc)
@@ -3869,8 +4011,6 @@ static int cam_icp_mgr_acquire_hw(void *hw_mgr_priv, void *acquire_hw_args)
hfi_set_debug_level(icp_hw_mgr.a5_debug_type,
icp_hw_mgr.a5_dbg_lvl);
- hfi_set_fw_dump_level(icp_hw_mgr.a5_fw_dump_lvl);
-
rc = cam_icp_send_ubwc_cfg(hw_mgr);
if (rc)
goto ubwc_cfg_failed;
@@ -4034,8 +4174,8 @@ static int cam_icp_mgr_alloc_devs(struct device_node *of_node)
CAM_ERR(CAM_ICP, "read num bps devices failed");
goto num_bps_failed;
}
- icp_hw_mgr.devices[CAM_ICP_DEV_BPS] = kcalloc(num_dev,
- sizeof(struct cam_hw_intf *), GFP_KERNEL);
+ icp_hw_mgr.devices[CAM_ICP_DEV_BPS] = kzalloc(
+ sizeof(struct cam_hw_intf *) * num_dev, GFP_KERNEL);
if (!icp_hw_mgr.devices[CAM_ICP_DEV_BPS]) {
rc = -ENOMEM;
goto num_bps_failed;
@@ -4176,6 +4316,9 @@ static int cam_icp_mgr_create_wq(void)
if (rc)
goto debugfs_create_failed;
+ icp_hw_mgr.icp_pc_flag = true;
+ icp_hw_mgr.ipe_bps_pc_flag = true;
+
for (i = 0; i < ICP_WORKQ_NUM_TASK; i++)
icp_hw_mgr.msg_work->task.pool[i].payload =
&icp_hw_mgr.msg_work_data[i];
diff --git a/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/cam_icp_hw_mgr.h b/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/cam_icp_hw_mgr.h
index 8746ee2..a614b80 100644
--- a/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/cam_icp_hw_mgr.h
+++ b/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/cam_icp_hw_mgr.h
@@ -30,7 +30,7 @@
#define CAM_ICP_ROLE_PARENT 1
#define CAM_ICP_ROLE_CHILD 2
-#define CAM_FRAME_CMD_MAX 20
+#define CAM_FRAME_CMD_MAX 48
#define CAM_MAX_OUT_RES 6
#define CAM_MAX_IN_RES 8
@@ -79,6 +79,7 @@
* @sec_heap: Memory info of secondary heap
* @fw_buf: Memory info of firmware
* @qdss_buf: Memory info of qdss
+ * @sfr_buf: Memory info for sfr buffer
*/
struct icp_hfi_mem_info {
struct cam_mem_mgr_memory_desc qtbl;
@@ -88,6 +89,7 @@ struct icp_hfi_mem_info {
struct cam_mem_mgr_memory_desc sec_heap;
struct cam_mem_mgr_memory_desc fw_buf;
struct cam_mem_mgr_memory_desc qdss_buf;
+ struct cam_mem_mgr_memory_desc sfr_buf;
struct cam_smmu_region_info shmem;
};
@@ -208,6 +210,8 @@ struct cam_icp_hw_ctx_data {
struct cam_ctx_clk_info clk_info;
struct cam_req_mgr_timer *watch_dog;
uint32_t watch_dog_reset_counter;
+ struct hfi_cmd_ipebps_async *abort_cmd;
+ struct hfi_cmd_ipebps_async *destroy_cmd;
};
/**
@@ -279,7 +283,6 @@ struct cam_icp_clk_info {
* @a5_jtag_debug: entry to enable A5 JTAG debugging
* @a5_debug_type : entry to enable FW debug message/qdss
* @a5_dbg_lvl : debug level set to FW.
- * @a5_fw_dump_lvl : level set for dumping the FW data
* @ipe0_enable: Flag for IPE0
* @ipe1_enable: Flag for IPE1
* @bps_enable: Flag for BPS
@@ -290,6 +293,9 @@ struct cam_icp_clk_info {
* @bps_dev_intf: Device interface for BPS
* @ipe_clk_state: IPE clock state flag
* @bps_clk_state: BPS clock state flag
+ * @recovery: Flag to validate if in previous session FW
+ * reported a fatal error or wdt. If set FW is
+ * re-downloaded for new camera session.
*/
struct cam_icp_hw_mgr {
struct mutex hw_mgr_mutex;
@@ -326,7 +332,6 @@ struct cam_icp_hw_mgr {
bool a5_jtag_debug;
u64 a5_debug_type;
u64 a5_dbg_lvl;
- u64 a5_fw_dump_lvl;
bool ipe0_enable;
bool ipe1_enable;
bool bps_enable;
@@ -337,6 +342,7 @@ struct cam_icp_hw_mgr {
struct cam_hw_intf *bps_dev_intf;
bool ipe_clk_state;
bool bps_clk_state;
+ bool recovery;
};
static int cam_icp_mgr_hw_close(void *hw_priv, void *hw_close_args);
diff --git a/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/include/cam_bps_hw_intf.h b/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/include/cam_bps_hw_intf.h
index 4f07172..0f76a05 100644
--- a/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/include/cam_bps_hw_intf.h
+++ b/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/include/cam_bps_hw_intf.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -18,6 +18,15 @@
#include "cam_hw_mgr_intf.h"
#include "cam_icp_hw_intf.h"
+/* BPS register */
+#define BPS_TOP_RST_CMD 0x1008
+#define BPS_CDM_RST_CMD 0x10
+#define BPS_CDM_IRQ_STATUS 0x44
+#define BPS_TOP_IRQ_STATUS 0x100C
+
+/* BPS CDM/TOP status register */
+#define BPS_RST_DONE_IRQ_STATUS_BIT 0x1
+
enum cam_icp_bps_cmd_type {
CAM_ICP_BPS_CMD_FW_DOWNLOAD,
CAM_ICP_BPS_CMD_POWER_COLLAPSE,
@@ -28,6 +37,7 @@ enum cam_icp_bps_cmd_type {
CAM_ICP_BPS_CMD_CPAS_STOP,
CAM_ICP_BPS_CMD_UPDATE_CLK,
CAM_ICP_BPS_CMD_DISABLE_CLK,
+ CAM_ICP_BPS_CMD_RESET,
CAM_ICP_BPS_CMD_MAX,
};
diff --git a/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/include/cam_ipe_hw_intf.h b/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/include/cam_ipe_hw_intf.h
index 0943bef..d1e3b9a 100644
--- a/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/include/cam_ipe_hw_intf.h
+++ b/drivers/media/platform/msm/camera/cam_icp/icp_hw/icp_hw_mgr/include/cam_ipe_hw_intf.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -18,6 +18,15 @@
#include "cam_hw_mgr_intf.h"
#include "cam_icp_hw_intf.h"
+/* IPE registers */
+#define IPE_TOP_RST_CMD 0x1008
+#define IPE_CDM_RST_CMD 0x10
+#define IPE_CDM_IRQ_STATUS 0x44
+#define IPE_TOP_IRQ_STATUS 0x100C
+
+/* IPE CDM/TOP status register */
+#define IPE_RST_DONE_IRQ_STATUS_BIT 0x1
+
enum cam_icp_ipe_cmd_type {
CAM_ICP_IPE_CMD_FW_DOWNLOAD,
CAM_ICP_IPE_CMD_POWER_COLLAPSE,
@@ -28,6 +37,7 @@ enum cam_icp_ipe_cmd_type {
CAM_ICP_IPE_CMD_CPAS_STOP,
CAM_ICP_IPE_CMD_UPDATE_CLK,
CAM_ICP_IPE_CMD_DISABLE_CLK,
+ CAM_ICP_IPE_CMD_RESET,
CAM_ICP_IPE_CMD_MAX,
};
diff --git a/drivers/media/platform/msm/camera/cam_icp/icp_hw/ipe_hw/ipe_core.c b/drivers/media/platform/msm/camera/cam_icp/icp_hw/ipe_hw/ipe_core.c
index 87478af..620a4bd 100644
--- a/drivers/media/platform/msm/camera/cam_icp/icp_hw/ipe_hw/ipe_core.c
+++ b/drivers/media/platform/msm/camera/cam_icp/icp_hw/ipe_hw/ipe_core.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -18,6 +18,7 @@
#include <linux/firmware.h>
#include <linux/delay.h>
#include <linux/timer.h>
+#include <linux/iopoll.h>
#include "cam_io_util.h"
#include "cam_hw.h"
#include "cam_hw_intf.h"
@@ -29,6 +30,9 @@
#include "cam_icp_hw_mgr_intf.h"
#include "cam_cpas_api.h"
#include "cam_debug_util.h"
+#include "hfi_reg.h"
+
+#define HFI_MAX_POLL_TRY 5
static int cam_ipe_caps_vote(struct cam_ipe_device_core_info *core_info,
struct cam_icp_cpas_vote *cpas_vote)
@@ -206,6 +210,77 @@ static int cam_ipe_handle_resume(struct cam_hw_info *ipe_dev)
return rc;
}
+static int cam_ipe_cmd_reset(struct cam_hw_soc_info *soc_info,
+ struct cam_ipe_device_core_info *core_info)
+{
+ int pwr_ctrl, pwr_status, rc = 0;
+ uint32_t status = 0, retry_cnt = 0;
+ bool reset_ipe_cdm_fail = false;
+ bool reset_ipe_top_fail = false;
+
+ CAM_DBG(CAM_ICP, "CAM_ICP_IPE_CMD_RESET");
+ /* IPE CDM core reset*/
+ cam_io_w_mb((uint32_t)0xF,
+ soc_info->reg_map[0].mem_base + IPE_CDM_RST_CMD);
+ while (retry_cnt < HFI_MAX_POLL_TRY) {
+ readw_poll_timeout((soc_info->reg_map[0].mem_base +
+ IPE_CDM_IRQ_STATUS),
+ status, ((status & IPE_RST_DONE_IRQ_STATUS_BIT) == 0x1),
+ 100, 10000);
+
+ CAM_DBG(CAM_HFI, "ipe_cdm_irq_status = %u", status);
+
+ if ((status & IPE_RST_DONE_IRQ_STATUS_BIT) == 0x1)
+ break;
+ retry_cnt++;
+ }
+ status = cam_io_r_mb(soc_info->reg_map[0].mem_base +
+ IPE_CDM_IRQ_STATUS);
+ if ((status & IPE_RST_DONE_IRQ_STATUS_BIT) != 0x1) {
+ CAM_ERR(CAM_ICP, "IPE CDM rst failed status 0x%x", status);
+ reset_ipe_cdm_fail = true;
+ }
+
+ /* IPE reset*/
+ status = 0;
+ cam_io_w_mb((uint32_t)0x3,
+ soc_info->reg_map[0].mem_base + IPE_TOP_RST_CMD);
+ while (retry_cnt < HFI_MAX_POLL_TRY) {
+ readw_poll_timeout((soc_info->reg_map[0].mem_base +
+ IPE_TOP_IRQ_STATUS),
+ status, ((status & IPE_RST_DONE_IRQ_STATUS_BIT) == 0x1),
+ 100, 10000);
+
+ CAM_DBG(CAM_HFI, "ipe_top_irq_status = %u", status);
+
+
+ if ((status & IPE_RST_DONE_IRQ_STATUS_BIT) == 0x1)
+ break;
+ retry_cnt++;
+ }
+ status = cam_io_r_mb(soc_info->reg_map[0].mem_base +
+ IPE_TOP_IRQ_STATUS);
+ if ((status & IPE_RST_DONE_IRQ_STATUS_BIT) != 0x1) {
+ CAM_ERR(CAM_ICP, "IPE top rst failed status 0x%x", status);
+ reset_ipe_top_fail = true;
+ }
+
+ cam_ipe_get_gdsc_control(soc_info);
+ cam_cpas_reg_read(core_info->cpas_handle,
+ CAM_CPAS_REG_CPASTOP, core_info->ipe_hw_info->pwr_ctrl,
+ true, &pwr_ctrl);
+ cam_cpas_reg_read(core_info->cpas_handle,
+ CAM_CPAS_REG_CPASTOP, core_info->ipe_hw_info->pwr_status,
+ true, &pwr_status);
+ CAM_DBG(CAM_ICP, "(After)pwr_ctrl = %x pwr_status = %x",
+ pwr_ctrl, pwr_status);
+
+ if (reset_ipe_cdm_fail || reset_ipe_top_fail)
+ rc = -EAGAIN;
+
+ return rc;
+}
+
int cam_ipe_process_cmd(void *device_priv, uint32_t cmd_type,
void *cmd_args, uint32_t arg_size)
{
@@ -302,7 +377,12 @@ int cam_ipe_process_cmd(void *device_priv, uint32_t cmd_type,
cam_ipe_toggle_clk(soc_info, false);
core_info->clk_enable = false;
break;
+ case CAM_ICP_IPE_CMD_RESET:
+ rc = cam_ipe_cmd_reset(soc_info, core_info);
+ break;
default:
+ CAM_ERR(CAM_ICP, "Invalid Cmd Type:%u", cmd_type);
+ rc = -EINVAL;
break;
}
return rc;
diff --git a/drivers/media/platform/msm/camera/cam_isp/cam_isp_context.c b/drivers/media/platform/msm/camera/cam_isp/cam_isp_context.c
index 50cdc7d..b8329f4 100644
--- a/drivers/media/platform/msm/camera/cam_isp/cam_isp_context.c
+++ b/drivers/media/platform/msm/camera/cam_isp/cam_isp_context.c
@@ -26,97 +26,6 @@
static const char isp_dev_name[] = "isp";
-#define INC_STATE_MONITOR_HEAD(head) \
- (atomic64_add_return(1, head) % \
- CAM_ISP_CTX_STATE_MONITOR_MAX_ENTRIES)
-
-static void __cam_isp_ctx_update_state_monitor_array(
- struct cam_isp_context *ctx_isp,
- enum cam_isp_state_change_trigger trigger_type,
- uint32_t req_id)
-{
- int iterator = 0;
-
- iterator = INC_STATE_MONITOR_HEAD(&ctx_isp->state_monitor_head);
- ctx_isp->cam_isp_ctx_state_monitor[iterator].curr_state =
- ctx_isp->substate_activated;
- ctx_isp->cam_isp_ctx_state_monitor[iterator].trigger =
- trigger_type;
- ctx_isp->cam_isp_ctx_state_monitor[iterator].req_id =
- req_id;
- ctx_isp->cam_isp_ctx_state_monitor[iterator].evt_time_stamp =
- jiffies_to_msecs(jiffies);
-}
-
-static const char *__cam_isp_ctx_substate_val_to_type(
- uint32_t type)
-{
- switch (type) {
- case CAM_ISP_CTX_ACTIVATED_SOF:
- return "SOF";
- case CAM_ISP_CTX_ACTIVATED_APPLIED:
- return "APPLIED";
- case CAM_ISP_CTX_ACTIVATED_EPOCH:
- return "EPOCH";
- case CAM_ISP_CTX_ACTIVATED_BUBBLE:
- return "BUBBLE";
- case CAM_ISP_CTX_ACTIVATED_BUBBLE_APPLIED:
- return "BUBBLE_APPLIED";
- case CAM_ISP_CTX_ACTIVATED_HALT:
- return "HALT";
- default:
- return "CAM_ISP_CTX_INVALID_STATE";
- }
-}
-
-static const char *__cam_isp_hw_evt_val_to_type(
- uint32_t evt_id)
-{
- switch (evt_id) {
- case CAM_ISP_STATE_CHANGE_TRIGGER_ERROR:
- return "ERROR";
- case CAM_ISP_STATE_CHANGE_TRIGGER_SOF:
- return "SOF";
- case CAM_ISP_STATE_CHANGE_TRIGGER_REG_UPDATE:
- return "REG_UPDATE";
- case CAM_ISP_STATE_CHANGE_TRIGGER_EPOCH:
- return "EPOCH";
- case CAM_ISP_STATE_CHANGE_TRIGGER_EOF:
- return "EOF";
- case CAM_ISP_STATE_CHANGE_TRIGGER_DONE:
- return "DONE";
- default:
- return "CAM_ISP_EVENT_INVALID";
- }
-}
-
-static void __cam_isp_ctx_dump_state_monitor_array(
- struct cam_isp_context *ctx_isp)
-{
- int i = 0;
- uint64_t state_head = 0;
- uint64_t index;
-
- state_head = atomic64_read(&ctx_isp->state_monitor_head);
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "Dumping state information for preceding requests");
-
- for (i = CAM_ISP_CTX_STATE_MONITOR_MAX_ENTRIES - 1; i >= 0;
- i--) {
- index = (((state_head - i) +
- CAM_ISP_CTX_STATE_MONITOR_MAX_ENTRIES) %
- CAM_ISP_CTX_STATE_MONITOR_MAX_ENTRIES);
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "time[0x%llx] req_id[%u] state[%s] evt_type[%s]",
- ctx_isp->cam_isp_ctx_state_monitor[index].evt_time_stamp,
- ctx_isp->cam_isp_ctx_state_monitor[index].req_id,
- __cam_isp_ctx_substate_val_to_type(
- ctx_isp->cam_isp_ctx_state_monitor[index].curr_state),
- __cam_isp_hw_evt_val_to_type(
- ctx_isp->cam_isp_ctx_state_monitor[index].trigger));
- }
-}
-
static int __cam_isp_ctx_enqueue_request_in_order(
struct cam_context *ctx, struct cam_ctx_request *req)
{
@@ -225,46 +134,46 @@ static int __cam_isp_ctx_enqueue_init_request(
return rc;
}
-static const char *__cam_isp_resource_handle_id_to_type(
- uint32_t resource_handle)
+static const char *__cam_isp_resource_handle_id_to_type
+ (uint32_t resource_handle)
{
switch (resource_handle) {
case CAM_ISP_IFE_OUT_RES_FULL:
- return "FULL";
+ return "CAM_ISP_IFE_OUT_RES_FULL";
case CAM_ISP_IFE_OUT_RES_DS4:
- return "DS4";
+ return "CAM_ISP_IFE_OUT_RES_DS4";
case CAM_ISP_IFE_OUT_RES_DS16:
- return "DS16";
+ return "CAM_ISP_IFE_OUT_RES_DS16";
case CAM_ISP_IFE_OUT_RES_RAW_DUMP:
- return "RAW_DUMP";
+ return "CAM_ISP_IFE_OUT_RES_RAW_DUMP";
case CAM_ISP_IFE_OUT_RES_FD:
- return "FD";
+ return "CAM_ISP_IFE_OUT_RES_FD";
case CAM_ISP_IFE_OUT_RES_PDAF:
- return "PDAF";
+ return "CAM_ISP_IFE_OUT_RES_PDAF";
case CAM_ISP_IFE_OUT_RES_RDI_0:
- return "RDI_0";
+ return "CAM_ISP_IFE_OUT_RES_RDI_0";
case CAM_ISP_IFE_OUT_RES_RDI_1:
- return "RDI_1";
+ return "CAM_ISP_IFE_OUT_RES_RDI_1";
case CAM_ISP_IFE_OUT_RES_RDI_2:
- return "RDI_2";
+ return "CAM_ISP_IFE_OUT_RES_RDI_2";
case CAM_ISP_IFE_OUT_RES_RDI_3:
- return "RDI_3";
+ return "CAM_ISP_IFE_OUT_RES_RDI_3";
case CAM_ISP_IFE_OUT_RES_STATS_HDR_BE:
- return "STATS_HDR_BE";
+ return "CAM_ISP_IFE_OUT_RES_STATS_HDR_BE";
case CAM_ISP_IFE_OUT_RES_STATS_HDR_BHIST:
- return "STATS_HDR_BHIST";
+ return "CAM_ISP_IFE_OUT_RES_STATS_HDR_BHIST";
case CAM_ISP_IFE_OUT_RES_STATS_TL_BG:
- return "STATS_TL_BG";
+ return "CAM_ISP_IFE_OUT_RES_STATS_TL_BG";
case CAM_ISP_IFE_OUT_RES_STATS_BF:
- return "STATS_BF";
+ return "CAM_ISP_IFE_OUT_RES_STATS_BF";
case CAM_ISP_IFE_OUT_RES_STATS_AWB_BG:
- return "STATS_AWB_BG";
+ return "CAM_ISP_IFE_OUT_RES_STATS_AWB_BG";
case CAM_ISP_IFE_OUT_RES_STATS_BHIST:
- return "STATS_BHIST";
+ return "CAM_ISP_IFE_OUT_RES_STATS_BHIST";
case CAM_ISP_IFE_OUT_RES_STATS_RS:
- return "STATS_RS";
+ return "CAM_ISP_IFE_OUT_RES_STATS_RS";
case CAM_ISP_IFE_OUT_RES_STATS_CS:
- return "STATS_CS";
+ return "CAM_ISP_IFE_OUT_RES_STATS_CS";
default:
return "CAM_ISP_Invalid_Resource_Type";
}
@@ -441,9 +350,6 @@ static int __cam_isp_ctx_handle_buf_done_in_activated_state(
}
end:
- __cam_isp_ctx_update_state_monitor_array(ctx_isp,
- CAM_ISP_STATE_CHANGE_TRIGGER_DONE,
- ctx_isp->base->req_list->request_id);
return rc;
}
@@ -635,11 +541,6 @@ static int __cam_isp_ctx_sof_in_activated_state(
{
int rc = 0;
struct cam_isp_hw_sof_event_data *sof_event_data = evt_data;
- struct cam_ctx_request *req;
- struct cam_context *ctx = ctx_isp->base;
-
- req = list_last_entry(&ctx->pending_req_list,
- struct cam_ctx_request, list);
if (!evt_data) {
CAM_ERR(CAM_ISP, "in valid sof event data");
@@ -648,9 +549,7 @@ static int __cam_isp_ctx_sof_in_activated_state(
ctx_isp->frame_id++;
ctx_isp->sof_timestamp_val = sof_event_data->timestamp;
- ctx_isp->boot_timestamp = sof_event_data->boot_time;
- __cam_isp_ctx_update_state_monitor_array(ctx_isp,
- CAM_ISP_STATE_CHANGE_TRIGGER_SOF, req->request_id);
+ ctx_isp->boot_timestamp = sof_event_data->boot_timestamp;
CAM_DBG(CAM_ISP, "frame id: %lld time stamp:0x%llx",
ctx_isp->frame_id, ctx_isp->sof_timestamp_val);
@@ -661,11 +560,11 @@ static int __cam_isp_ctx_reg_upd_in_sof(struct cam_isp_context *ctx_isp,
void *evt_data)
{
int rc = 0;
- struct cam_ctx_request *req = NULL;
+ struct cam_ctx_request *req;
struct cam_isp_ctx_req *req_isp;
struct cam_context *ctx = ctx_isp->base;
- if (ctx->state != CAM_CTX_ACTIVATED && ctx_isp->frame_id > 1) {
+ if (ctx->state != CAM_CTX_ACTIVATED) {
CAM_DBG(CAM_ISP, "invalid RUP");
goto end;
}
@@ -693,11 +592,6 @@ static int __cam_isp_ctx_reg_upd_in_sof(struct cam_isp_context *ctx_isp,
CAM_ISP_CTX_ACTIVATED_EPOCH;
}
}
- if (req != NULL) {
- __cam_isp_ctx_update_state_monitor_array(ctx_isp,
- CAM_ISP_STATE_CHANGE_TRIGGER_REG_UPDATE,
- req->request_id);
- }
end:
return rc;
}
@@ -765,15 +659,6 @@ static int __cam_isp_ctx_epoch_in_applied(struct cam_isp_context *ctx_isp,
CAM_DBG(CAM_ISP, "next substate %d",
ctx_isp->substate_activated);
end:
- if (request_id == 0) {
- req = list_last_entry(&ctx->active_req_list,
- struct cam_ctx_request, list);
- __cam_isp_ctx_update_state_monitor_array(ctx_isp,
- CAM_ISP_STATE_CHANGE_TRIGGER_EPOCH, req->request_id);
- } else {
- __cam_isp_ctx_update_state_monitor_array(ctx_isp,
- CAM_ISP_STATE_CHANGE_TRIGGER_EPOCH, request_id);
- }
return 0;
}
@@ -796,7 +681,6 @@ static int __cam_isp_ctx_sof_in_epoch(struct cam_isp_context *ctx_isp,
int rc = 0;
struct cam_context *ctx = ctx_isp->base;
struct cam_isp_hw_sof_event_data *sof_event_data = evt_data;
- struct cam_ctx_request *req;
if (!evt_data) {
CAM_ERR(CAM_ISP, "in valid sof event data");
@@ -805,17 +689,13 @@ static int __cam_isp_ctx_sof_in_epoch(struct cam_isp_context *ctx_isp,
ctx_isp->frame_id++;
ctx_isp->sof_timestamp_val = sof_event_data->timestamp;
- ctx_isp->boot_timestamp = sof_event_data->boot_time;
+ ctx_isp->boot_timestamp = sof_event_data->boot_timestamp;
if (list_empty(&ctx->active_req_list))
ctx_isp->substate_activated = CAM_ISP_CTX_ACTIVATED_SOF;
else
CAM_DBG(CAM_ISP, "Still need to wait for the buf done");
- req = list_last_entry(&ctx->active_req_list,
- struct cam_ctx_request, list);
- __cam_isp_ctx_update_state_monitor_array(ctx_isp,
- CAM_ISP_STATE_CHANGE_TRIGGER_SOF, ctx->req_list->request_id);
CAM_DBG(CAM_ISP, "next substate %d",
ctx_isp->substate_activated);
@@ -914,10 +794,6 @@ static int __cam_isp_ctx_epoch_in_bubble_applied(
ctx_isp->substate_activated = CAM_ISP_CTX_ACTIVATED_BUBBLE;
CAM_DBG(CAM_ISP, "next substate %d", ctx_isp->substate_activated);
end:
- req = list_last_entry(&ctx->active_req_list, struct cam_ctx_request,
- list);
- __cam_isp_ctx_update_state_monitor_array(ctx_isp,
- CAM_ISP_STATE_CHANGE_TRIGGER_EPOCH, req->request_id);
return 0;
}
@@ -929,9 +805,6 @@ static int __cam_isp_ctx_buf_done_in_bubble_applied(
(struct cam_isp_hw_done_event_data *) evt_data;
rc = __cam_isp_ctx_handle_buf_done_in_activated_state(ctx_isp, done, 1);
- __cam_isp_ctx_update_state_monitor_array(ctx_isp,
- CAM_ISP_STATE_CHANGE_TRIGGER_DONE,
- ctx_isp->base->req_list->request_id);
return rc;
}
@@ -946,7 +819,6 @@ static int __cam_isp_ctx_handle_error(struct cam_isp_context *ctx_isp,
struct cam_isp_ctx_req *req_isp = NULL;
struct cam_req_mgr_error_notify notify;
uint64_t error_request_id;
- struct cam_hw_fence_map_entry *fence_map_out = NULL;
struct cam_context *ctx = ctx_isp->base;
struct cam_isp_hw_error_event_data *error_event_data =
@@ -975,18 +847,18 @@ static int __cam_isp_ctx_handle_error(struct cam_isp_context *ctx_isp,
if (!req_isp->bubble_report) {
for (i = 0; i < req_isp->num_fence_map_out;
i++) {
- fence_map_out =
- &req_isp->fence_map_out[i];
CAM_ERR(CAM_ISP, "req %llu, Sync fd %x",
- req->request_id,
- req_isp->fence_map_out[i].sync_id);
+ req->request_id,
+ req_isp->fence_map_out[i].
+ sync_id);
if (req_isp->fence_map_out[i].sync_id
!= -1) {
rc = cam_sync_signal(
- fence_map_out->sync_id,
+ req_isp->fence_map_out[i].
+ sync_id,
CAM_SYNC_STATE_SIGNALED_ERROR);
- fence_map_out->sync_id =
- -1;
+ req_isp->fence_map_out[i].
+ sync_id = -1;
}
}
list_del_init(&req->list);
@@ -1054,14 +926,6 @@ static int __cam_isp_ctx_handle_error(struct cam_isp_context *ctx_isp,
rc = -EFAULT;
}
-
- list_del_init(&req->list);
- list_add(&req->list, &ctx->pending_req_list);
- /* might need to check if active list is empty */
- if (req != NULL) {
- __cam_isp_ctx_update_state_monitor_array(ctx_isp,
- CAM_ISP_STATE_CHANGE_TRIGGER_ERROR, req->request_id);
- }
CAM_DBG(CAM_ISP, "Exit");
return rc;
}
@@ -1078,7 +942,7 @@ static int __cam_isp_ctx_sof_in_flush(
}
ctx_isp->frame_id++;
ctx_isp->sof_timestamp_val = sof_event_data->timestamp;
- ctx_isp->boot_timestamp = sof_event_data->boot_time;
+ ctx_isp->boot_timestamp = sof_event_data->boot_timestamp;
CAM_DBG(CAM_ISP, "frame id: %lld time stamp:0x%llx",
ctx_isp->frame_id, ctx_isp->sof_timestamp_val);
@@ -1183,7 +1047,7 @@ static int __cam_isp_ctx_apply_req_in_activated_state(
struct cam_ctx_request *active_req;
struct cam_isp_ctx_req *req_isp;
struct cam_isp_ctx_req *active_req_isp;
- struct cam_isp_context *ctx_isp = NULL;
+ struct cam_isp_context *ctx_isp;
struct cam_hw_config_args cfg;
if (list_empty(&ctx->pending_req_list)) {
@@ -1258,11 +1122,6 @@ static int __cam_isp_ctx_apply_req_in_activated_state(
spin_unlock_bh(&ctx->lock);
}
end:
- if (ctx_isp != NULL) {
- __cam_isp_ctx_update_state_monitor_array(ctx_isp,
- CAM_ISP_STATE_CHANGE_TRIGGER_SOF,
- ctx->req_list->request_id);
- }
return rc;
}
@@ -1325,7 +1184,9 @@ static int __cam_isp_ctx_flush_req(struct cam_context *ctx,
struct list_head flush_list;
INIT_LIST_HEAD(&flush_list);
+ spin_lock_bh(&ctx->lock);
if (list_empty(req_list)) {
+ spin_unlock_bh(&ctx->lock);
CAM_DBG(CAM_ISP, "request list is empty");
return 0;
}
@@ -1344,6 +1205,7 @@ static int __cam_isp_ctx_flush_req(struct cam_context *ctx,
list_del_init(&req->list);
list_add_tail(&req->list, &flush_list);
}
+ spin_unlock_bh(&ctx->lock);
list_for_each_entry_safe(req, req_temp, &flush_list, list) {
req_isp = (struct cam_isp_ctx_req *) req->req_priv;
@@ -1361,7 +1223,9 @@ static int __cam_isp_ctx_flush_req(struct cam_context *ctx,
req_isp->fence_map_out[i].sync_id = -1;
}
}
+ spin_lock_bh(&ctx->lock);
list_add_tail(&req->list, &ctx->free_req_list);
+ spin_unlock_bh(&ctx->lock);
}
if (flush_req->type == CAM_REQ_MGR_FLUSH_TYPE_CANCEL_REQ &&
@@ -1382,9 +1246,7 @@ static int __cam_isp_ctx_flush_req_in_top_state(
int rc = 0;
CAM_DBG(CAM_ISP, "try to flush pending list");
- spin_lock_bh(&ctx->lock);
rc = __cam_isp_ctx_flush_req(ctx, &ctx->pending_req_list, flush_req);
- spin_unlock_bh(&ctx->lock);
CAM_DBG(CAM_ISP, "Flush request in top state %d",
ctx->state);
return rc;
@@ -1401,7 +1263,6 @@ static int __cam_isp_ctx_flush_req_in_activated(
CAM_DBG(CAM_ISP, "Flush request in state %d", ctx->state);
rc = __cam_isp_ctx_flush_req(ctx, &ctx->pending_req_list, flush_req);
-
/* only if request is found in pending queue, move to flush state*/
if (!rc) {
spin_lock_bh(&ctx->lock);
@@ -1419,10 +1280,10 @@ static int __cam_isp_ctx_flush_req_in_ready(
int rc = 0;
CAM_DBG(CAM_ISP, "try to flush pending list");
- spin_lock_bh(&ctx->lock);
rc = __cam_isp_ctx_flush_req(ctx, &ctx->pending_req_list, flush_req);
/* if nothing is in pending req list, change state to acquire*/
+ spin_lock_bh(&ctx->lock);
if (list_empty(&ctx->pending_req_list))
ctx->state = CAM_CTX_ACQUIRED;
spin_unlock_bh(&ctx->lock);
@@ -1508,7 +1369,7 @@ static int __cam_isp_ctx_rdi_only_sof_in_top_state(
ctx_isp->frame_id++;
ctx_isp->sof_timestamp_val = sof_event_data->timestamp;
- ctx_isp->boot_timestamp = sof_event_data->boot_time;
+ ctx_isp->boot_timestamp = sof_event_data->boot_timestamp;
CAM_DBG(CAM_ISP, "frame id: %lld time stamp:0x%llx",
ctx_isp->frame_id, ctx_isp->sof_timestamp_val);
@@ -1563,7 +1424,7 @@ static int __cam_isp_ctx_rdi_only_sof_in_applied_state(
ctx_isp->frame_id++;
ctx_isp->sof_timestamp_val = sof_event_data->timestamp;
- ctx_isp->boot_timestamp = sof_event_data->boot_time;
+ ctx_isp->boot_timestamp = sof_event_data->boot_timestamp;
CAM_DBG(CAM_ISP, "frame id: %lld time stamp:0x%llx",
ctx_isp->frame_id, ctx_isp->sof_timestamp_val);
@@ -1595,7 +1456,7 @@ static int __cam_isp_ctx_rdi_only_sof_in_bubble_applied(
ctx_isp->frame_id++;
ctx_isp->sof_timestamp_val = sof_event_data->timestamp;
- ctx_isp->boot_timestamp = sof_event_data->boot_time;
+ ctx_isp->boot_timestamp = sof_event_data->boot_timestamp;
CAM_DBG(CAM_ISP, "frame id: %lld time stamp:0x%llx",
ctx_isp->frame_id, ctx_isp->sof_timestamp_val);
@@ -1681,7 +1542,7 @@ static int __cam_isp_ctx_rdi_only_sof_in_bubble_state(
ctx_isp->frame_id++;
ctx_isp->sof_timestamp_val = sof_event_data->timestamp;
- ctx_isp->boot_timestamp = sof_event_data->boot_time;
+ ctx_isp->boot_timestamp = sof_event_data->boot_timestamp;
CAM_DBG(CAM_ISP, "frame id: %lld time stamp:0x%llx",
ctx_isp->frame_id, ctx_isp->sof_timestamp_val);
/*
@@ -1744,7 +1605,6 @@ static int __cam_isp_ctx_rdi_only_reg_upd_in_bubble_applied_state(
struct cam_req_mgr_trigger_notify notify;
uint64_t request_id = 0;
- ctx_isp->substate_activated = CAM_ISP_CTX_ACTIVATED_EPOCH;
/* notify reqmgr with sof signal*/
if (ctx->ctx_crm_intf && ctx->ctx_crm_intf->notify_trigger) {
if (list_empty(&ctx->pending_req_list)) {
@@ -1756,19 +1616,13 @@ static int __cam_isp_ctx_rdi_only_reg_upd_in_bubble_applied_state(
list_del_init(&req->list);
req_isp = (struct cam_isp_ctx_req *) req->req_priv;
- request_id =
- (req_isp->hw_update_data.packet_opcode_type ==
- CAM_ISP_PACKET_INIT_DEV) ?
- 0 : req->request_id;
-
+ request_id = req->request_id;
if (req_isp->num_fence_map_out != 0) {
list_add_tail(&req->list, &ctx->active_req_list);
ctx_isp->active_req_cnt++;
CAM_DBG(CAM_ISP,
"move request %lld to active list(cnt = %d)",
req->request_id, ctx_isp->active_req_cnt);
- /* if packet has buffers, set correct request id */
- request_id = req->request_id;
} else {
/* no io config, so the request is completed. */
list_add_tail(&req->list, &ctx->free_req_list);
@@ -1788,11 +1642,10 @@ static int __cam_isp_ctx_rdi_only_reg_upd_in_bubble_applied_state(
} else {
CAM_ERR(CAM_ISP, "Can not notify SOF to CRM");
}
- if (request_id)
- ctx_isp->reported_req_id = request_id;
-
__cam_isp_ctx_send_sof_timestamp(ctx_isp, request_id,
CAM_REQ_MGR_SOF_EVENT_SUCCESS);
+
+ ctx_isp->substate_activated = CAM_ISP_CTX_ACTIVATED_EPOCH;
CAM_DBG(CAM_ISP, "next substate %d", ctx_isp->substate_activated);
return 0;
@@ -1998,9 +1851,8 @@ static int __cam_isp_ctx_release_dev_in_top_state(struct cam_context *ctx,
flush_req.dev_hdl = ctx->dev_hdl;
CAM_DBG(CAM_ISP, "try to flush pending list");
- spin_lock_bh(&ctx->lock);
rc = __cam_isp_ctx_flush_req(ctx, &ctx->pending_req_list, &flush_req);
- spin_unlock_bh(&ctx->lock);
+
ctx->state = CAM_CTX_AVAILABLE;
trace_cam_context_state("ISP", ctx);
@@ -2094,7 +1946,7 @@ static int __cam_isp_ctx_config_dev_in_top_state(
req->request_id = packet->header.request_id;
req->status = 1;
- CAM_DBG(CAM_ISP, "Packet request id %lld packet opcode:%d",
+ CAM_DBG(CAM_ISP, "Packet request id 0x%llx packet opcode:%d",
packet->header.request_id,
req_isp->hw_update_data.packet_opcode_type);
@@ -2237,7 +2089,6 @@ static int __cam_isp_ctx_acquire_dev_in_available(struct cam_context *ctx,
cam_isp_ctx_activated_state_machine;
}
- ctx_isp->rdi_only_context = hw_cmd_args.u.is_rdi_only_context;
ctx_isp->hw_ctx = param.ctxt_to_hw_map;
req_hdl_param.session_hdl = cmd->session_handle;
@@ -2384,10 +2235,7 @@ static int __cam_isp_ctx_start_dev_in_ready(struct cam_context *ctx,
ctx_isp->frame_id = 0;
ctx_isp->active_req_cnt = 0;
ctx_isp->reported_req_id = 0;
- ctx_isp->substate_activated = ctx_isp->rdi_only_context ?
- CAM_ISP_CTX_ACTIVATED_APPLIED :
- (req_isp->num_fence_map_out) ? CAM_ISP_CTX_ACTIVATED_EPOCH :
- CAM_ISP_CTX_ACTIVATED_SOF;
+ ctx_isp->substate_activated = CAM_ISP_CTX_ACTIVATED_SOF;
/*
* Only place to change state before calling the hw due to
@@ -2405,11 +2253,6 @@ static int __cam_isp_ctx_start_dev_in_ready(struct cam_context *ctx,
goto end;
}
CAM_DBG(CAM_ISP, "start device success");
-
- if (req_isp->num_fence_map_out) {
- list_del_init(&req->list);
- list_add_tail(&req->list, &ctx->active_req_list);
- }
end:
return rc;
}
@@ -2593,16 +2436,16 @@ static int __cam_isp_ctx_apply_req(struct cam_context *ctx,
struct cam_req_mgr_apply_request *apply)
{
int rc = 0;
- struct cam_ctx_ops *ctx_ops = NULL;
struct cam_isp_context *ctx_isp =
(struct cam_isp_context *) ctx->ctx_priv;
trace_cam_apply_req("ISP", apply->request_id);
CAM_DBG(CAM_ISP, "Enter: apply req in Substate %d request _id:%lld",
ctx_isp->substate_activated, apply->request_id);
- ctx_ops = &ctx_isp->substate_machine[ctx_isp->substate_activated];
- if (ctx_ops->crm_ops.apply_req) {
- rc = ctx_ops->crm_ops.apply_req(ctx, apply);
+ if (ctx_isp->substate_machine[ctx_isp->substate_activated].
+ crm_ops.apply_req) {
+ rc = ctx_isp->substate_machine[ctx_isp->substate_activated].
+ crm_ops.apply_req(ctx, apply);
} else {
CAM_ERR_RATE_LIMIT(CAM_ISP,
"No handle function in activated substate %d",
@@ -2623,7 +2466,6 @@ static int __cam_isp_ctx_handle_irq_in_activated(void *context,
uint32_t evt_id, void *evt_data)
{
int rc = 0;
- struct cam_isp_ctx_irq_ops *irq_ops = NULL;
struct cam_context *ctx = (struct cam_context *)context;
struct cam_isp_context *ctx_isp =
(struct cam_isp_context *)ctx->ctx_priv;
@@ -2635,13 +2477,13 @@ static int __cam_isp_ctx_handle_irq_in_activated(void *context,
CAM_DBG(CAM_ISP, "Enter: State %d, Substate %d, evt id %d",
ctx->state, ctx_isp->substate_activated, evt_id);
- irq_ops = &ctx_isp->substate_machine_irq[ctx_isp->substate_activated];
- if (irq_ops->irq_ops[evt_id]) {
- rc = irq_ops->irq_ops[evt_id](ctx_isp, evt_data);
+ if (ctx_isp->substate_machine_irq[ctx_isp->substate_activated].
+ irq_ops[evt_id]) {
+ rc = ctx_isp->substate_machine_irq[ctx_isp->substate_activated].
+ irq_ops[evt_id](ctx_isp, evt_data);
} else {
CAM_DBG(CAM_ISP, "No handle function for substate %d",
ctx_isp->substate_activated);
- __cam_isp_ctx_dump_state_monitor_array(ctx_isp);
}
CAM_DBG(CAM_ISP, "Exit: State %d Substate %d",
ctx->state, ctx_isp->substate_activated);
@@ -2755,12 +2597,6 @@ int cam_isp_context_init(struct cam_isp_context *ctx,
ctx_base->state_machine = cam_isp_ctx_top_state_machine;
ctx_base->ctx_priv = ctx;
- /* initializing current state for error logging */
- for (i = 0; i < CAM_ISP_CTX_STATE_MONITOR_MAX_ENTRIES; i++) {
- ctx->cam_isp_ctx_state_monitor[i].curr_state =
- CAM_ISP_CTX_ACTIVATED_MAX;
- }
- atomic64_set(&ctx->state_monitor_head, -1);
err:
return rc;
}
diff --git a/drivers/media/platform/msm/camera/cam_isp/cam_isp_context.h b/drivers/media/platform/msm/camera/cam_isp/cam_isp_context.h
index a939f2d..110c344 100644
--- a/drivers/media/platform/msm/camera/cam_isp/cam_isp_context.h
+++ b/drivers/media/platform/msm/camera/cam_isp/cam_isp_context.h
@@ -28,16 +28,11 @@
#define CAM_ISP_CTX_RES_MAX 20
/*
- * Maximum configuration entry size - This is based on the
+ * Maxiimum configuration entry size - This is based on the
* worst case DUAL IFE use case plus some margin.
*/
#define CAM_ISP_CTX_CFG_MAX 22
-/*
- * Maximum entries in state monitoring array for error logging
- */
-#define CAM_ISP_CTX_STATE_MONITOR_MAX_ENTRIES 20
-
/* forward declaration */
struct cam_isp_context;
@@ -61,19 +56,6 @@ enum cam_isp_ctx_activated_substate {
CAM_ISP_CTX_ACTIVATED_MAX,
};
-/**
- * enum cam_isp_state_change_trigger - Different types of ISP events
- *
- */
-enum cam_isp_state_change_trigger {
- CAM_ISP_STATE_CHANGE_TRIGGER_ERROR,
- CAM_ISP_STATE_CHANGE_TRIGGER_SOF,
- CAM_ISP_STATE_CHANGE_TRIGGER_REG_UPDATE,
- CAM_ISP_STATE_CHANGE_TRIGGER_EPOCH,
- CAM_ISP_STATE_CHANGE_TRIGGER_EOF,
- CAM_ISP_STATE_CHANGE_TRIGGER_DONE,
- CAM_ISP_STATE_CHANGE_TRIGGER_MAX
-};
/**
* struct cam_isp_ctx_irq_ops - Function table for handling IRQ callbacks
@@ -119,47 +101,24 @@ struct cam_isp_ctx_req {
};
/**
- * struct cam_isp_context_state_monitor - ISP context state
- * monitoring for
- * debug purposes
+ * struct cam_isp_context - ISP context object
*
- *@curr_state: Current sub state that received req
- *@req_type: Event type of incoming req
- *@req_id: Request id
- *@evt_time_stamp Current time stamp
- *
- */
-struct cam_isp_context_state_monitor {
- enum cam_isp_ctx_activated_substate curr_state;
- enum cam_isp_state_change_trigger trigger;
- uint32_t req_id;
- int64_t frame_id;
- uint64_t evt_time_stamp;
-};
-
-/**
- * struct cam_isp_context - ISP context object
- *
- * @base: Common context object pointer
- * @frame_id: Frame id tracking for the isp context
- * @substate_actiavted: Current substate for the activated state.
- * @substate_machine: ISP substate machine for external interface
- * @substate_machine_irq: ISP substate machine for irq handling
- * @req_base: Common request object storage
- * @req_isp: ISP private request object storage
- * @hw_ctx: HW object returned by the acquire device command
- * @sof_timestamp_val: Captured time stamp value at sof hw event
+ * @base: Common context object pointer
+ * @frame_id: Frame id tracking for the isp context
+ * @substate_actiavted: Current substate for the activated state.
+ * @substate_machine: ISP substate machine for external interface
+ * @substate_machine_irq: ISP substate machine for irq handling
+ * @req_base: Common request object storage
+ * @req_isp: ISP private request object storage
+ * @hw_ctx: HW object returned by the acquire device command
+ * @sof_timestamp_val: Captured time stamp value at sof hw event
* @boot_timestamp: Boot time stamp for a given req_id
- * @active_req_cnt: Counter for the active request
- * @reported_req_id: Last reported request id
- * @subscribe_event: The irq event mask that CRM subscribes to, IFE
- * will invoke CRM cb at those event.
- * @last_applied_req_id: Last applied request id
- * @frame_skip_count: Number of frame to skip before change state
- * @state_monitor_head: Write index to the state monitoring array
- * @cam_isp_ctx_state_monitor: State monitoring array
- * @rdi_only_context: Get context type information.
- * true, if context is rdi only context
+ * @active_req_cnt: Counter for the active request
+ * @reported_req_id: Last reported request id
+ * @subscribe_event: The irq event mask that CRM subscribes to, IFE will
+ * invoke CRM cb at those event.
+ * @last_applied_req_id: Last applied request id
+ * @frame_skip_count: Number of frame to skip before change state
*
*/
struct cam_isp_context {
@@ -181,10 +140,6 @@ struct cam_isp_context {
uint32_t subscribe_event;
int64_t last_applied_req_id;
uint32_t frame_skip_count;
- atomic64_t state_monitor_head;
- struct cam_isp_context_state_monitor cam_isp_ctx_state_monitor[
- CAM_ISP_CTX_STATE_MONITOR_MAX_ENTRIES];
- bool rdi_only_context;
};
/**
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/Makefile b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/Makefile
index 1f7dc76..3cc4205 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/Makefile
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/Makefile
@@ -13,3 +13,6 @@
obj-$(CONFIG_SPECTRA_CAMERA) += hw_utils/ isp_hw/
obj-$(CONFIG_SPECTRA_CAMERA) += cam_isp_hw_mgr.o cam_ife_hw_mgr.o
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-enum-conversion \
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/cam_ife_hw_mgr.c b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/cam_ife_hw_mgr.c
index c1aa501..6394533 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/cam_ife_hw_mgr.c
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/cam_ife_hw_mgr.c
@@ -210,8 +210,7 @@ static int cam_ife_hw_mgr_start_hw_res(
int rc = -1;
struct cam_hw_intf *hw_intf;
- /* Start slave (which is right split) first */
- for (i = CAM_ISP_HW_SPLIT_MAX - 1; i >= 0; i--) {
+ for (i = 0; i < CAM_ISP_HW_SPLIT_MAX; i++) {
if (!isp_hw_res->hw_res[i])
continue;
hw_intf = isp_hw_res->hw_res[i]->hw_intf;
@@ -960,7 +959,6 @@ static int cam_ife_hw_mgr_acquire_res_ife_csid_ipp(
{
int rc = -1;
int i;
- int master_idx = -1;
struct cam_ife_hw_mgr *ife_hw_mgr;
struct cam_ife_hw_mgr_res *csid_res;
@@ -1012,27 +1010,18 @@ static int cam_ife_hw_mgr_acquire_res_ife_csid_ipp(
if (!cid_res->hw_res[i])
continue;
- hw_intf = ife_hw_mgr->csid_devices[
- cid_res->hw_res[i]->hw_intf->hw_idx];
-
csid_acquire.node_res = NULL;
if (csid_res->is_dual_vfe) {
- if (i == CAM_ISP_HW_SPLIT_LEFT) {
- master_idx = hw_intf->hw_idx;
+ if (i == CAM_ISP_HW_SPLIT_LEFT)
csid_acquire.sync_mode =
CAM_ISP_HW_SYNC_MASTER;
- } else {
- if (master_idx == -1) {
- CAM_ERR(CAM_ISP,
- "No Master found");
- goto err;
- }
+ else
csid_acquire.sync_mode =
CAM_ISP_HW_SYNC_SLAVE;
- csid_acquire.master_idx = master_idx;
- }
}
+ hw_intf = ife_hw_mgr->csid_devices[
+ cid_res->hw_res[i]->hw_intf->hw_idx];
rc = hw_intf->hw_ops.reserve(hw_intf->hw_priv,
&csid_acquire, sizeof(csid_acquire));
if (rc) {
@@ -1367,7 +1356,6 @@ static int cam_ife_mgr_acquire_hw(void *hw_mgr_priv,
uint32_t num_rdi_port_per_in = 0;
uint32_t total_pix_port = 0;
uint32_t total_rdi_port = 0;
- uint32_t in_port_length = 0;
CAM_DBG(CAM_ISP, "Enter...");
@@ -1428,27 +1416,9 @@ static int cam_ife_mgr_acquire_hw(void *hw_mgr_priv,
isp_resource[i].res_hdl,
isp_resource[i].length);
- in_port_length = sizeof(struct cam_isp_in_port_info);
-
- if (in_port_length > isp_resource[i].length) {
- CAM_ERR(CAM_ISP, "buffer size is not enough");
- rc = -EINVAL;
- goto free_res;
- }
-
in_port = memdup_user((void __user *)isp_resource[i].res_hdl,
isp_resource[i].length);
- if (!IS_ERR(in_port)) {
- in_port_length = sizeof(struct cam_isp_in_port_info) +
- (in_port->num_out_res - 1) *
- sizeof(struct cam_isp_out_port_info);
- if (in_port_length > isp_resource[i].length) {
- CAM_ERR(CAM_ISP, "buffer size is not enough");
- rc = -EINVAL;
- kfree(in_port);
- goto free_res;
- }
-
+ if (in_port > 0) {
rc = cam_ife_mgr_acquire_hw_for_ctx(ife_ctx, in_port,
&num_pix_port_per_in, &num_rdi_port_per_in);
total_pix_port += num_pix_port_per_in;
@@ -1852,6 +1822,7 @@ static int cam_ife_mgr_stop_hw(void *hw_mgr_priv, void *stop_hw_args)
CAM_DBG(CAM_ISP, "Going to stop IFE Out");
/* IFE out resources */
+ /* This will stop the IFE so that there will be no BH call. */
for (i = 0; i < CAM_IFE_HW_OUT_RES_MAX; i++)
cam_ife_hw_mgr_stop_hw_res(&ctx->res_list_ife_out[i]);
/* get master base index first */
@@ -1861,9 +1832,9 @@ static int cam_ife_mgr_stop_hw(void *hw_mgr_priv, void *stop_hw_args)
break;
}
}
-
+ /* After this, there will be no BH call so that we can call
+ cam_tasklet_stop here */
cam_tasklet_stop(ctx->common.tasklet_info);
-
/*
* If Context does not have PIX resources and has only RDI resource
* then take the first base index.
@@ -3462,8 +3433,9 @@ static int cam_ife_hw_mgr_process_camif_sof(
evt_payload);
if (atomic_read(&ife_hwr_mgr_ctx->overflow_pending))
break;
- if (!sof_status)
+ if (!sof_status) {
rc = 0;
+ }
}
break;
@@ -3538,6 +3510,7 @@ static int cam_ife_hw_mgr_handle_sof(
cam_hw_event_cb_func ife_hw_irq_sof_cb;
struct cam_isp_hw_sof_event_data sof_done_event_data;
uint32_t sof_status = 0;
+ uint64_t boot_timestamp;
bool sof_sent = false;
CAM_DBG(CAM_ISP, "Enter");
@@ -3574,7 +3547,10 @@ static int cam_ife_hw_mgr_handle_sof(
cam_ife_mgr_cmd_get_sof_timestamp(
ife_hw_mgr_ctx,
&sof_done_event_data.timestamp,
- &sof_done_event_data.boot_time);
+ &boot_timestamp);
+
+ sof_done_event_data.boot_timestamp =
+ boot_timestamp;
ife_hw_irq_sof_cb(
ife_hw_mgr_ctx->common.cb_priv,
@@ -3596,7 +3572,7 @@ static int cam_ife_hw_mgr_handle_sof(
cam_ife_mgr_cmd_get_sof_timestamp(
ife_hw_mgr_ctx,
&sof_done_event_data.timestamp,
- &sof_done_event_data.boot_time);
+ &sof_done_event_data.boot_timestamp);
ife_hw_irq_sof_cb(
ife_hw_mgr_ctx->common.cb_priv,
@@ -4070,8 +4046,6 @@ int cam_ife_hw_mgr_init(struct cam_hw_mgr_intf *hw_mgr_intf)
int rc = -EFAULT;
int i, j;
struct cam_iommu_handle cdm_handles;
- struct cam_ife_hw_mgr_ctx *ctx_pool;
- struct cam_ife_hw_mgr_res *res_list_ife_out;
CAM_DBG(CAM_ISP, "Enter");
@@ -4176,10 +4150,9 @@ int cam_ife_hw_mgr_init(struct cam_hw_mgr_intf *hw_mgr_intf)
INIT_LIST_HEAD(&g_ife_hw_mgr.ctx_pool[i].res_list_ife_cid);
INIT_LIST_HEAD(&g_ife_hw_mgr.ctx_pool[i].res_list_ife_csid);
INIT_LIST_HEAD(&g_ife_hw_mgr.ctx_pool[i].res_list_ife_src);
- ctx_pool = &g_ife_hw_mgr.ctx_pool[i];
for (j = 0; j < CAM_IFE_HW_OUT_RES_MAX; j++) {
- res_list_ife_out = &ctx_pool->res_list_ife_out[j];
- INIT_LIST_HEAD(&res_list_ife_out->list);
+ INIT_LIST_HEAD(&g_ife_hw_mgr.ctx_pool[i].
+ res_list_ife_out[j].list);
}
/* init context pool */
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/cam_isp_packet_parser.c b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/cam_isp_packet_parser.c
index e869e2b..3606af9a 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/cam_isp_packet_parser.c
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/cam_isp_packet_parser.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -414,8 +414,6 @@ int cam_isp_add_io_buffers(
struct cam_ife_hw_mgr_res *hw_mgr_res;
struct cam_isp_hw_get_cmd_update update_buf;
struct cam_isp_hw_get_wm_update wm_update;
- struct cam_hw_fence_map_entry *out_map_entries;
- struct cam_hw_fence_map_entry *in_map_entries;
uint32_t kmd_buf_remain_size;
uint32_t i, j, num_out_buf, num_in_buf;
uint32_t res_id_out, res_id_in, plane_id;
@@ -460,15 +458,14 @@ int cam_isp_add_io_buffers(
CAM_DBG(CAM_ISP,
"configure output io with fill fence %d",
fill_fence);
- out_map_entries =
- &prepare->out_map_entries[num_out_buf];
if (fill_fence) {
if (num_out_buf <
prepare->max_out_map_entries) {
- out_map_entries->resource_handle =
- io_cfg[i].resource_type;
- out_map_entries->sync_id =
- io_cfg[i].fence;
+ prepare->out_map_entries[num_out_buf].
+ resource_handle =
+ io_cfg[i].resource_type;
+ prepare->out_map_entries[num_out_buf].
+ sync_id = io_cfg[i].fence;
num_out_buf++;
} else {
CAM_ERR(CAM_ISP, "ln_out:%d max_ln:%d",
@@ -489,14 +486,14 @@ int cam_isp_add_io_buffers(
CAM_DBG(CAM_ISP,
"configure input io with fill fence %d",
fill_fence);
- in_map_entries =
- &prepare->in_map_entries[num_in_buf];
if (fill_fence) {
if (num_in_buf < prepare->max_in_map_entries) {
- in_map_entries->resource_handle =
- io_cfg[i].resource_type;
- in_map_entries->sync_id =
- io_cfg[i].fence;
+ prepare->in_map_entries[num_in_buf].
+ resource_handle =
+ io_cfg[i].resource_type;
+ prepare->in_map_entries[num_in_buf].
+ sync_id =
+ io_cfg[i].fence;
num_in_buf++;
} else {
CAM_ERR(CAM_ISP, "ln_in:%d imax_ln:%d",
@@ -579,15 +576,8 @@ int cam_isp_add_io_buffers(
io_addr[plane_id] +=
io_cfg[i].offsets[plane_id];
CAM_DBG(CAM_ISP,
- "get io_addr for plane %d: 0x%llx, mem_hdl=0x%x",
- plane_id, io_addr[plane_id],
- io_cfg[i].mem_handle[plane_id]);
-
- CAM_DBG(CAM_ISP,
- "mmu_hdl=0x%x, size=%d, end=0x%x",
- mmu_hdl, (int)size,
- io_addr[plane_id]+size);
-
+ "get io_addr for plane %d: 0x%llx",
+ plane_id, io_addr[plane_id]);
}
if (!plane_id) {
CAM_ERR(CAM_ISP, "No valid planes for res%d",
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/cam_tasklet_util.c b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/cam_tasklet_util.c
index 3a89732..7b4c825 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/cam_tasklet_util.c
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/cam_tasklet_util.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -70,25 +70,27 @@ struct cam_tasklet_info {
void *ctx_priv;
};
-/**
- * cam_tasklet_get_cmd()
- *
- * @brief: Get free cmd from tasklet
- *
- * @tasklet: Tasklet Info structure to get cmd from
- * @tasklet_cmd: Return tasklet_cmd pointer if successful
- *
- * @return: 0: Success
- * Negative: Failure
- */
-static int cam_tasklet_get_cmd(
- struct cam_tasklet_info *tasklet,
- struct cam_tasklet_queue_cmd **tasklet_cmd)
+struct cam_irq_bh_api tasklet_bh_api = {
+ .bottom_half_enqueue_func = cam_tasklet_enqueue_cmd,
+ .get_bh_payload_func = cam_tasklet_get_cmd,
+ .put_bh_payload_func = cam_tasklet_put_cmd,
+};
+
+int cam_tasklet_get_cmd(
+ void *bottom_half,
+ void **bh_cmd)
{
int rc = 0;
unsigned long flags;
+ struct cam_tasklet_info *tasklet = bottom_half;
+ struct cam_tasklet_queue_cmd *tasklet_cmd = NULL;
- *tasklet_cmd = NULL;
+ *bh_cmd = NULL;
+
+ if (tasklet == NULL) {
+ CAM_ERR_RATE_LIMIT(CAM_ISP, "tasklet is NULL");
+ return -EINVAL;
+ }
if (!atomic_read(&tasklet->tasklet_active)) {
CAM_ERR_RATE_LIMIT(CAM_ISP, "Tasklet is not active!\n");
@@ -102,9 +104,10 @@ static int cam_tasklet_get_cmd(
rc = -ENODEV;
goto spin_unlock;
} else {
- *tasklet_cmd = list_first_entry(&tasklet->free_cmd_list,
+ tasklet_cmd = list_first_entry(&tasklet->free_cmd_list,
struct cam_tasklet_queue_cmd, list);
- list_del_init(&(*tasklet_cmd)->list);
+ list_del_init(&(tasklet_cmd)->list);
+ *bh_cmd = tasklet_cmd;
}
spin_unlock:
@@ -112,25 +115,28 @@ static int cam_tasklet_get_cmd(
return rc;
}
-/**
- * cam_tasklet_put_cmd()
- *
- * @brief: Put back cmd to free list
- *
- * @tasklet: Tasklet Info structure to put cmd into
- * @tasklet_cmd: tasklet_cmd pointer that needs to be put back
- *
- * @return: Void
- */
-static void cam_tasklet_put_cmd(
- struct cam_tasklet_info *tasklet,
- struct cam_tasklet_queue_cmd **tasklet_cmd)
+void cam_tasklet_put_cmd(
+ void *bottom_half,
+ void **bh_cmd)
{
unsigned long flags;
+ struct cam_tasklet_info *tasklet = bottom_half;
+ struct cam_tasklet_queue_cmd *tasklet_cmd = *bh_cmd;
+
+ if (tasklet == NULL) {
+ CAM_ERR_RATE_LIMIT(CAM_ISP, "tasklet is NULL");
+ return;
+ }
+
+ if (tasklet_cmd == NULL) {
+ CAM_ERR_RATE_LIMIT(CAM_ISP, "Invalid tasklet_cmd");
+ return;
+ }
spin_lock_irqsave(&tasklet->tasklet_lock, flags);
- list_add_tail(&(*tasklet_cmd)->list,
- &tasklet->free_cmd_list);
+ list_del_init(&tasklet_cmd->list);
+ list_add_tail(&tasklet_cmd->list, &tasklet->free_cmd_list);
+ *bh_cmd = NULL;
spin_unlock_irqrestore(&tasklet->tasklet_lock, flags);
}
@@ -157,12 +163,6 @@ static int cam_tasklet_dequeue_cmd(
*tasklet_cmd = NULL;
- if (!atomic_read(&tasklet->tasklet_active)) {
- CAM_ERR(CAM_ISP, "Tasklet is not active!");
- rc = -EPIPE;
- return rc;
- }
-
CAM_DBG(CAM_ISP, "Dequeue before lock.");
spin_lock_irqsave(&tasklet->tasklet_lock, flags);
if (list_empty(&tasklet->used_cmd_list)) {
@@ -181,24 +181,31 @@ static int cam_tasklet_dequeue_cmd(
return rc;
}
-int cam_tasklet_enqueue_cmd(
+void cam_tasklet_enqueue_cmd(
void *bottom_half,
+ void *bh_cmd,
void *handler_priv,
void *evt_payload_priv,
CAM_IRQ_HANDLER_BOTTOM_HALF bottom_half_handler)
{
- struct cam_tasklet_info *tasklet = bottom_half;
- struct cam_tasklet_queue_cmd *tasklet_cmd = NULL;
unsigned long flags;
- int rc;
+ struct cam_tasklet_queue_cmd *tasklet_cmd = bh_cmd;
+ struct cam_tasklet_info *tasklet = bottom_half;
if (!bottom_half) {
- CAM_ERR(CAM_ISP, "NULL bottom half");
- return -EINVAL;
+ CAM_ERR_RATE_LIMIT(CAM_ISP, "NULL bottom half");
+ return;
}
- rc = cam_tasklet_get_cmd(tasklet, &tasklet_cmd);
+ if (!bh_cmd) {
+ CAM_ERR(CAM_ISP, "NULL bh cmd");
+ return;
+ }
+ if (!atomic_read(&tasklet->tasklet_active)) {
+ CAM_ERR_RATE_LIMIT(CAM_ISP, "Tasklet is not active\n");
+ return;
+ }
if (tasklet_cmd) {
CAM_DBG(CAM_ISP, "Enqueue tasklet cmd");
tasklet_cmd->bottom_half_handler = bottom_half_handler;
@@ -211,8 +218,6 @@ int cam_tasklet_enqueue_cmd(
} else {
CAM_ERR(CAM_ISP, "tasklet cmd is NULL!");
}
-
- return rc;
}
int cam_tasklet_init(
@@ -244,7 +249,7 @@ int cam_tasklet_init(
}
tasklet_init(&tasklet->tasklet, cam_tasklet_action,
(unsigned long)tasklet);
- cam_tasklet_stop(tasklet);
+ tasklet_disable(&tasklet->tasklet);
*tasklet_info = tasklet;
@@ -255,32 +260,30 @@ void cam_tasklet_deinit(void **tasklet_info)
{
struct cam_tasklet_info *tasklet = *tasklet_info;
- atomic_set(&tasklet->tasklet_active, 0);
- tasklet_kill(&tasklet->tasklet);
+ if (atomic_read(&tasklet->tasklet_active)) {
+ atomic_set(&tasklet->tasklet_active, 0);
+ tasklet_kill(&tasklet->tasklet);
+ tasklet_disable(&tasklet->tasklet);
+ }
kfree(tasklet);
*tasklet_info = NULL;
}
-static void cam_tasklet_flush(void *tasklet_info)
+static inline void cam_tasklet_flush(struct cam_tasklet_info *tasklet_info)
{
- unsigned long data;
- struct cam_tasklet_info *tasklet = tasklet_info;
-
- data = (unsigned long)tasklet;
- cam_tasklet_action(data);
+ cam_tasklet_action((unsigned long)tasklet_info);
}
int cam_tasklet_start(void *tasklet_info)
{
struct cam_tasklet_info *tasklet = tasklet_info;
- int i = 0;
+ int i;
if (atomic_read(&tasklet->tasklet_active)) {
CAM_ERR(CAM_ISP, "Tasklet already active. idx = %d",
tasklet->index);
return -EBUSY;
}
- atomic_set(&tasklet->tasklet_active, 1);
/* clean up the command queue first */
for (i = 0; i < CAM_TASKLETQ_SIZE; i++) {
@@ -289,6 +292,8 @@ int cam_tasklet_start(void *tasklet_info)
&tasklet->free_cmd_list);
}
+ atomic_set(&tasklet->tasklet_active, 1);
+
tasklet_enable(&tasklet->tasklet);
return 0;
@@ -298,9 +303,10 @@ void cam_tasklet_stop(void *tasklet_info)
{
struct cam_tasklet_info *tasklet = tasklet_info;
- cam_tasklet_flush(tasklet);
atomic_set(&tasklet->tasklet_active, 0);
+ tasklet_kill(&tasklet->tasklet);
tasklet_disable(&tasklet->tasklet);
+ cam_tasklet_flush(tasklet);
}
/*
@@ -323,7 +329,7 @@ static void cam_tasklet_action(unsigned long data)
while (!cam_tasklet_dequeue_cmd(tasklet_info, &tasklet_cmd)) {
tasklet_cmd->bottom_half_handler(tasklet_info->ctx_priv,
tasklet_cmd->payload);
- cam_tasklet_put_cmd(tasklet_info, &tasklet_cmd);
+ cam_tasklet_put_cmd(tasklet_info, (void **)(&tasklet_cmd));
}
}
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/include/cam_isp_packet_parser.h b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/include/cam_isp_packet_parser.h
index e3f2ce2..c73c8e5 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/include/cam_isp_packet_parser.h
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/include/cam_isp_packet_parser.h
@@ -16,7 +16,7 @@
#include <linux/types.h>
#include <uapi/media/cam_isp.h>
#include "cam_isp_hw_mgr_intf.h"
-#include "cam_ife_hw_mgr.h"
+#include "../../cam_ife_hw_mgr.h"
#include "cam_hw_intf.h"
#include "cam_packet_util.h"
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/include/cam_tasklet_util.h b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/include/cam_tasklet_util.h
index 0e4bf12..8bd93d8 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/include/cam_tasklet_util.h
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/include/cam_tasklet_util.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -76,6 +76,7 @@ void cam_tasklet_stop(void *tasklet);
* @brief: Enqueue the tasklet_cmd to used list
*
* @bottom_half: Tasklet info to enqueue onto
+ * @bh_cmd: Tasklet cmd used to enqueue task
* @handler_priv: Private Handler data that will be passed to the
* handler function
* @evt_payload_priv: Event payload that will be passed to the handler
@@ -83,13 +84,40 @@ void cam_tasklet_stop(void *tasklet);
* @bottom_half_handler: Callback function that will be called by tasklet
* for handling event
*
- * @return: 0: Success
- * Negative: Failure
+ * @return: Void
*/
-int cam_tasklet_enqueue_cmd(
+void cam_tasklet_enqueue_cmd(
void *bottom_half,
+ void *bh_cmd,
void *handler_priv,
void *evt_payload_priv,
CAM_IRQ_HANDLER_BOTTOM_HALF bottom_half_handler);
+/**
+ * cam_tasklet_get_cmd()
+ *
+ * @brief: Get free cmd from tasklet
+ *
+ * @bottom_half: Tasklet Info structure to get cmd from
+ * @bh_cmd: Return tasklet_cmd pointer if successful
+ *
+ * @return: 0: Success
+ * Negative: Failure
+ */
+int cam_tasklet_get_cmd(void *bottom_half, void **bh_cmd);
+
+/**
+ * cam_tasklet_put_cmd()
+ *
+ * @brief: Put back cmd to free list
+ *
+ * @bottom_half: Tasklet Info structure to put cmd into
+ * @bh_cmd: tasklet_cmd pointer that needs to be put back
+ *
+ * @return: Void
+ */
+void cam_tasklet_put_cmd(void *bottom_half, void **bh_cmd);
+
+extern struct cam_irq_bh_api tasklet_bh_api;
+
#endif /* _CAM_TASKLET_UTIL_H_ */
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/irq_controller/cam_irq_controller.c b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/irq_controller/cam_irq_controller.c
index feb79cc..72a68ad 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/irq_controller/cam_irq_controller.c
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/irq_controller/cam_irq_controller.c
@@ -13,6 +13,8 @@
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/list.h>
+#include <linux/ratelimit.h>
+
#include "cam_io_util.h"
#include "cam_irq_controller.h"
#include "cam_debug_util.h"
@@ -42,7 +44,7 @@ struct cam_irq_evt_handler {
CAM_IRQ_HANDLER_TOP_HALF top_half_handler;
CAM_IRQ_HANDLER_BOTTOM_HALF bottom_half_handler;
void *bottom_half;
- CAM_IRQ_BOTTOM_HALF_ENQUEUE_FUNC bottom_half_enqueue_func;
+ struct cam_irq_bh_api irq_bh_api;
struct list_head list_node;
struct list_head th_list_node;
int index;
@@ -232,7 +234,7 @@ int cam_irq_controller_subscribe_irq(void *irq_controller,
CAM_IRQ_HANDLER_TOP_HALF top_half_handler,
CAM_IRQ_HANDLER_BOTTOM_HALF bottom_half_handler,
void *bottom_half,
- CAM_IRQ_BOTTOM_HALF_ENQUEUE_FUNC bottom_half_enqueue_func)
+ struct cam_irq_bh_api *irq_bh_api)
{
struct cam_irq_controller *controller = irq_controller;
struct cam_irq_evt_handler *evt_handler = NULL;
@@ -255,12 +257,24 @@ int cam_irq_controller_subscribe_irq(void *irq_controller,
}
if (bottom_half_handler &&
- (!bottom_half || !bottom_half_enqueue_func)) {
+ (!bottom_half || !irq_bh_api)) {
CAM_ERR(CAM_ISP,
"Invalid params: bh_handler=%pK bh=%pK bh_enq_f=%pK",
bottom_half_handler,
bottom_half,
- bottom_half_enqueue_func);
+ irq_bh_api);
+ return -EINVAL;
+ }
+
+ if (irq_bh_api &&
+ (!irq_bh_api->bottom_half_enqueue_func ||
+ !irq_bh_api->get_bh_payload_func ||
+ !irq_bh_api->put_bh_payload_func)) {
+ CAM_ERR(CAM_ISP,
+ "Invalid: enqueue_func=%pK get_bh=%pK put_bh=%pK",
+ irq_bh_api->bottom_half_enqueue_func,
+ irq_bh_api->get_bh_payload_func,
+ irq_bh_api->put_bh_payload_func);
return -EINVAL;
}
@@ -295,9 +309,11 @@ int cam_irq_controller_subscribe_irq(void *irq_controller,
evt_handler->top_half_handler = top_half_handler;
evt_handler->bottom_half_handler = bottom_half_handler;
evt_handler->bottom_half = bottom_half;
- evt_handler->bottom_half_enqueue_func = bottom_half_enqueue_func;
evt_handler->index = controller->hdl_idx++;
+ if (irq_bh_api)
+ evt_handler->irq_bh_api = *irq_bh_api;
+
/* Avoid rollover to negative values */
if (controller->hdl_idx > 0x3FFFFFFF)
controller->hdl_idx = 1;
@@ -336,11 +352,9 @@ int cam_irq_controller_subscribe_irq(void *irq_controller,
int cam_irq_controller_enable_irq(void *irq_controller, uint32_t handle)
{
- struct cam_irq_controller *controller = irq_controller;
- struct cam_irq_evt_handler *evt_handler = NULL;
- struct cam_irq_evt_handler *evt_handler_temp;
- struct cam_irq_register_obj *irq_register = NULL;
- enum cam_irq_priority_level priority;
+ struct cam_irq_controller *controller = irq_controller;
+ struct cam_irq_evt_handler *evt_handler = NULL;
+ struct cam_irq_evt_handler *evt_handler_temp;
unsigned long flags = 0;
unsigned int i;
uint32_t irq_mask;
@@ -371,14 +385,14 @@ int cam_irq_controller_enable_irq(void *irq_controller, uint32_t handle)
return rc;
}
- priority = evt_handler->priority;
for (i = 0; i < controller->num_registers; i++) {
- irq_register = &controller->irq_register_arr[i];
- irq_register->top_half_enable_mask[priority] |=
+ controller->irq_register_arr[i].
+ top_half_enable_mask[evt_handler->priority] |=
evt_handler->evt_bit_mask_arr[i];
irq_mask = cam_io_r_mb(controller->mem_base +
- irq_register->mask_reg_offset);
+ controller->irq_register_arr[i].
+ mask_reg_offset);
irq_mask |= evt_handler->evt_bit_mask_arr[i];
cam_io_w_mb(irq_mask, controller->mem_base +
@@ -392,11 +406,9 @@ int cam_irq_controller_enable_irq(void *irq_controller, uint32_t handle)
int cam_irq_controller_disable_irq(void *irq_controller, uint32_t handle)
{
- struct cam_irq_controller *controller = irq_controller;
- struct cam_irq_evt_handler *evt_handler = NULL;
- struct cam_irq_evt_handler *evt_handler_temp;
- struct cam_irq_register_obj *irq_register;
- enum cam_irq_priority_level priority;
+ struct cam_irq_controller *controller = irq_controller;
+ struct cam_irq_evt_handler *evt_handler = NULL;
+ struct cam_irq_evt_handler *evt_handler_temp;
unsigned long flags = 0;
unsigned int i;
uint32_t irq_mask;
@@ -427,27 +439,31 @@ int cam_irq_controller_disable_irq(void *irq_controller, uint32_t handle)
return rc;
}
- priority = evt_handler->priority;
for (i = 0; i < controller->num_registers; i++) {
- irq_register = &controller->irq_register_arr[i];
- irq_register->top_half_enable_mask[priority] &=
+ controller->irq_register_arr[i].
+ top_half_enable_mask[evt_handler->priority] &=
~(evt_handler->evt_bit_mask_arr[i]);
irq_mask = cam_io_r_mb(controller->mem_base +
- irq_register->mask_reg_offset);
+ controller->irq_register_arr[i].
+ mask_reg_offset);
CAM_DBG(CAM_ISP, "irq_mask 0x%x before disable 0x%x",
- irq_register->mask_reg_offset, irq_mask);
+ controller->irq_register_arr[i].mask_reg_offset,
+ irq_mask);
irq_mask &= ~(evt_handler->evt_bit_mask_arr[i]);
cam_io_w_mb(irq_mask, controller->mem_base +
- irq_register->mask_reg_offset);
+ controller->irq_register_arr[i].
+ mask_reg_offset);
CAM_DBG(CAM_ISP, "irq_mask 0x%x after disable 0x%x",
- irq_register->mask_reg_offset, irq_mask);
+ controller->irq_register_arr[i].mask_reg_offset,
+ irq_mask);
/* Clear the IRQ bits of this handler */
cam_io_w_mb(evt_handler->evt_bit_mask_arr[i],
controller->mem_base +
- irq_register->clear_reg_offset);
+ controller->irq_register_arr[i].
+ clear_reg_offset);
if (controller->global_clear_offset)
cam_io_w_mb(
@@ -464,11 +480,9 @@ int cam_irq_controller_disable_irq(void *irq_controller, uint32_t handle)
int cam_irq_controller_unsubscribe_irq(void *irq_controller,
uint32_t handle)
{
- struct cam_irq_controller *controller = irq_controller;
- struct cam_irq_evt_handler *evt_handler = NULL;
- struct cam_irq_evt_handler *evt_handler_temp;
- struct cam_irq_register_obj *irq_register;
- enum cam_irq_priority_level priority;
+ struct cam_irq_controller *controller = irq_controller;
+ struct cam_irq_evt_handler *evt_handler = NULL;
+ struct cam_irq_evt_handler *evt_handler_temp;
uint32_t i;
uint32_t found = 0;
uint32_t irq_mask;
@@ -492,24 +506,26 @@ int cam_irq_controller_unsubscribe_irq(void *irq_controller,
}
}
- priority = evt_handler->priority;
if (found) {
for (i = 0; i < controller->num_registers; i++) {
- irq_register = &controller->irq_register_arr[i];
- irq_register->top_half_enable_mask[priority] &=
+ controller->irq_register_arr[i].
+ top_half_enable_mask[evt_handler->priority] &=
~(evt_handler->evt_bit_mask_arr[i]);
irq_mask = cam_io_r_mb(controller->mem_base +
- irq_register->mask_reg_offset);
+ controller->irq_register_arr[i].
+ mask_reg_offset);
irq_mask &= ~(evt_handler->evt_bit_mask_arr[i]);
cam_io_w_mb(irq_mask, controller->mem_base +
- irq_register->mask_reg_offset);
+ controller->irq_register_arr[i].
+ mask_reg_offset);
/* Clear the IRQ bits of this handler */
cam_io_w_mb(evt_handler->evt_bit_mask_arr[i],
controller->mem_base +
- irq_register->clear_reg_offset);
+ controller->irq_register_arr[i].
+ clear_reg_offset);
if (controller->global_clear_offset)
cam_io_w_mb(
controller->global_clear_bitmask,
@@ -564,6 +580,8 @@ static void cam_irq_controller_th_processing(
bool is_irq_match;
int rc = -EINVAL;
int i;
+ void *bh_cmd = NULL;
+ struct cam_irq_bh_api *irq_bh_api = NULL;
CAM_DBG(CAM_ISP, "Enter");
@@ -588,6 +606,19 @@ static void cam_irq_controller_th_processing(
evt_handler->evt_bit_mask_arr[i];
}
+ irq_bh_api = &evt_handler->irq_bh_api;
+ bh_cmd = NULL;
+
+ if (evt_handler->bottom_half_handler) {
+ rc = irq_bh_api->get_bh_payload_func(
+ evt_handler->bottom_half, &bh_cmd);
+ if (rc || !bh_cmd) {
+ CAM_ERR_RATE_LIMIT(CAM_ISP,
+ "Can't get bh payload");
+ continue;
+ }
+ }
+
/*
* irq_status_arr[0] is dummy argument passed. the entire
* status array is passed in th_payload.
@@ -597,16 +628,21 @@ static void cam_irq_controller_th_processing(
controller->irq_status_arr[0],
(void *)th_payload);
- if (!rc && evt_handler->bottom_half_handler) {
+ if (rc && bh_cmd) {
+ irq_bh_api->put_bh_payload_func(
+ evt_handler->bottom_half, &bh_cmd);
+ continue;
+ }
+
+ if (evt_handler->bottom_half_handler) {
CAM_DBG(CAM_ISP, "Enqueuing bottom half for %s",
controller->name);
- if (evt_handler->bottom_half_enqueue_func) {
- evt_handler->bottom_half_enqueue_func(
- evt_handler->bottom_half,
- evt_handler->handler_priv,
- th_payload->evt_payload_priv,
- evt_handler->bottom_half_handler);
- }
+ irq_bh_api->bottom_half_enqueue_func(
+ evt_handler->bottom_half,
+ bh_cmd,
+ evt_handler->handler_priv,
+ th_payload->evt_payload_priv,
+ evt_handler->bottom_half_handler);
}
}
@@ -642,8 +678,7 @@ irqreturn_t cam_irq_controller_clear_and_mask(int irq_num, void *priv)
irqreturn_t cam_irq_controller_handle_irq(int irq_num, void *priv)
{
- struct cam_irq_controller *controller = priv;
- struct cam_irq_register_obj *irq_register;
+ struct cam_irq_controller *controller = priv;
bool need_th_processing[CAM_IRQ_PRIORITY_MAX] = {false};
int i;
int j;
@@ -655,7 +690,6 @@ irqreturn_t cam_irq_controller_handle_irq(int irq_num, void *priv)
controller, controller->name, &controller->lock);
spin_lock(&controller->lock);
for (i = 0; i < controller->num_registers; i++) {
- irq_register = &controller->irq_register_arr[i];
controller->irq_status_arr[i] = cam_io_r_mb(
controller->mem_base +
controller->irq_register_arr[i].status_reg_offset);
@@ -666,7 +700,8 @@ irqreturn_t cam_irq_controller_handle_irq(int irq_num, void *priv)
controller->irq_register_arr[i].status_reg_offset,
controller->irq_status_arr[i]);
for (j = 0; j < CAM_IRQ_PRIORITY_MAX; j++) {
- if (irq_register->top_half_enable_mask[j] &
+ if (controller->irq_register_arr[i].
+ top_half_enable_mask[j] &
controller->irq_status_arr[i])
need_th_processing[j] = true;
CAM_DBG(CAM_ISP,
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/irq_controller/cam_irq_controller.h b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/irq_controller/cam_irq_controller.h
index e3071ac..c3c1e7c 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/irq_controller/cam_irq_controller.h
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/hw_utils/irq_controller/cam_irq_controller.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -111,10 +111,22 @@ typedef int (*CAM_IRQ_HANDLER_TOP_HALF)(uint32_t evt_id,
typedef int (*CAM_IRQ_HANDLER_BOTTOM_HALF)(void *handler_priv,
void *evt_payload_priv);
-typedef int (*CAM_IRQ_BOTTOM_HALF_ENQUEUE_FUNC)(void *bottom_half,
- void *handler_priv, void *evt_payload_priv,
+typedef void (*CAM_IRQ_BOTTOM_HALF_ENQUEUE_FUNC)(void *bottom_half,
+ void *bh_cmd, void *handler_priv, void *evt_payload_priv,
CAM_IRQ_HANDLER_BOTTOM_HALF);
+typedef int (*CAM_IRQ_GET_TASKLET_PAYLOAD_FUNC)(void *bottom_half,
+ void **bh_cmd);
+
+typedef void (*CAM_IRQ_PUT_TASKLET_PAYLOAD_FUNC)(void *bottom_half,
+ void **bh_cmd);
+
+struct cam_irq_bh_api {
+ CAM_IRQ_BOTTOM_HALF_ENQUEUE_FUNC bottom_half_enqueue_func;
+ CAM_IRQ_GET_TASKLET_PAYLOAD_FUNC get_bh_payload_func;
+ CAM_IRQ_PUT_TASKLET_PAYLOAD_FUNC put_bh_payload_func;
+};
+
/*
* cam_irq_controller_init()
*
@@ -165,7 +177,7 @@ int cam_irq_controller_subscribe_irq(void *irq_controller,
CAM_IRQ_HANDLER_TOP_HALF top_half_handler,
CAM_IRQ_HANDLER_BOTTOM_HALF bottom_half_handler,
void *bottom_half,
- CAM_IRQ_BOTTOM_HALF_ENQUEUE_FUNC bottom_half_enqueue_func);
+ struct cam_irq_bh_api *irq_bh_api);
/*
* cam_irq_controller_unsubscribe_irq()
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/include/cam_isp_hw_mgr_intf.h b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/include/cam_isp_hw_mgr_intf.h
index 5410858..371bd934 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/include/cam_isp_hw_mgr_intf.h
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/include/cam_isp_hw_mgr_intf.h
@@ -108,13 +108,13 @@ struct cam_isp_prepare_hw_update_data {
/**
* struct cam_isp_hw_sof_event_data - Event payload for CAM_HW_EVENT_SOF
*
- * @timestamp: Time stamp for the sof event
- * @boot_time: Boot time stamp for the sof event
+ * @timestamp: Time stamp for the sof event
+ * @boot_timestamp: Boot time stamp for the sof event
*
*/
struct cam_isp_hw_sof_event_data {
uint64_t timestamp;
- uint64_t boot_time;
+ uint64_t boot_timestamp;
};
/**
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/ife_csid_hw/cam_ife_csid_core.c b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/ife_csid_hw/cam_ife_csid_core.c
index 3edae4a..e7ce550 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/ife_csid_hw/cam_ife_csid_core.c
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/ife_csid_hw/cam_ife_csid_core.c
@@ -356,8 +356,6 @@ static int cam_ife_csid_global_reset(struct cam_ife_csid_hw *csid_hw)
struct cam_ife_csid_reg_offset *csid_reg;
int rc = 0;
uint32_t val = 0, i;
- uint32_t status;
-
soc_info = &csid_hw->hw_info->soc_info;
csid_reg = csid_hw->csid_info->csid_reg;
@@ -372,6 +370,8 @@ static int cam_ife_csid_global_reset(struct cam_ife_csid_hw *csid_hw)
CAM_DBG(CAM_ISP, "CSID:%d Csid reset",
csid_hw->hw_intf->hw_idx);
+ init_completion(&csid_hw->csid_top_complete);
+
/* Mask all interrupts */
cam_io_w_mb(0, soc_info->reg_map[0].mem_base +
csid_reg->csi2_reg->csid_csi2_rx_irq_mask_addr);
@@ -417,19 +417,21 @@ static int cam_ife_csid_global_reset(struct cam_ife_csid_hw *csid_hw)
cam_io_w_mb(0x2, soc_info->reg_map[0].mem_base +
csid_reg->rdi_reg[i]->csid_rdi_cfg0_addr);
- /* perform the top CSID HW and SW registers reset */
- cam_io_w_mb(csid_reg->cmn_reg->csid_rst_stb_sw_all,
+ /* perform the top CSID HW reset */
+ cam_io_w_mb(csid_reg->cmn_reg->csid_rst_stb,
soc_info->reg_map[0].mem_base +
csid_reg->cmn_reg->csid_rst_strobes_addr);
- rc = readl_poll_timeout(soc_info->reg_map[0].mem_base +
- csid_reg->cmn_reg->csid_top_irq_status_addr,
- status, (status & 0x1) == 0x1,
- CAM_IFE_CSID_TIMEOUT_SLEEP_US, CAM_IFE_CSID_TIMEOUT_ALL_US);
- if (rc < 0) {
- CAM_ERR(CAM_ISP, "CSID:%d csid_reset fail rc = %d",
- csid_hw->hw_intf->hw_idx, rc);
- rc = -ETIMEDOUT;
+ CAM_DBG(CAM_ISP, " Waiting for reset complete from irq handler");
+ rc = wait_for_completion_timeout(&csid_hw->csid_top_complete,
+ msecs_to_jiffies(IFE_CSID_TIMEOUT));
+ if (rc <= 0) {
+ CAM_ERR(CAM_ISP, "CSID:%d reset completion in fail rc = %d",
+ csid_hw->hw_intf->hw_idx, rc);
+ if (rc == 0)
+ rc = -ETIMEDOUT;
+ } else {
+ rc = 0;
}
val = cam_io_r_mb(soc_info->reg_map[0].mem_base +
@@ -851,37 +853,11 @@ static int cam_ife_csid_path_reserve(struct cam_ife_csid_hw *csid_hw,
path_data->cid = reserve->cid;
path_data->in_format = reserve->in_port->format;
path_data->out_format = reserve->out_port->format;
+ path_data->master_idx = reserve->master_idx;
path_data->sync_mode = reserve->sync_mode;
path_data->height = reserve->in_port->height;
path_data->start_line = reserve->in_port->line_start;
path_data->end_line = reserve->in_port->line_stop;
-
- /* Enable RDI crop for single ife use case only */
- switch (reserve->res_id) {
- case CAM_IFE_PIX_PATH_RES_RDI_0:
- case CAM_IFE_PIX_PATH_RES_RDI_1:
- case CAM_IFE_PIX_PATH_RES_RDI_2:
- case CAM_IFE_PIX_PATH_RES_RDI_3:
- if (reserve->in_port->usage_type)
- path_data->crop_enable = false;
- else
- path_data->crop_enable = true;
-
- break;
- case CAM_IFE_PIX_PATH_RES_IPP:
- path_data->crop_enable = true;
- break;
- default:
- rc = -EINVAL;
- goto end;
- }
-
- CAM_DBG(CAM_ISP,
- "Res id: %d height:%d line_start %d line_stop %d crop_en %d",
- reserve->res_id, reserve->in_port->height,
- reserve->in_port->line_start, reserve->in_port->line_stop,
- path_data->crop_enable);
-
if (reserve->in_port->res_type == CAM_ISP_IFE_IN_RES_TPG) {
path_data->dt = CAM_IFE_CSID_TPG_DT_VAL;
path_data->vc = CAM_IFE_CSID_TPG_VC_VAL;
@@ -891,36 +867,31 @@ static int cam_ife_csid_path_reserve(struct cam_ife_csid_hw *csid_hw,
}
if (reserve->sync_mode == CAM_ISP_HW_SYNC_MASTER) {
+ path_data->crop_enable = 1;
path_data->start_pixel = reserve->in_port->left_start;
path_data->end_pixel = reserve->in_port->left_stop;
path_data->width = reserve->in_port->left_width;
- CAM_DBG(CAM_ISP, "CSID:%d master:startpixel 0x%x endpixel:0x%x",
+ CAM_DBG(CAM_ISP, "CSID:%dmaster:startpixel 0x%x endpixel:0x%x",
csid_hw->hw_intf->hw_idx, path_data->start_pixel,
path_data->end_pixel);
- CAM_DBG(CAM_ISP, "CSID:%d master:line start:0x%x line end:0x%x",
+ CAM_DBG(CAM_ISP, "CSID:%dmaster:line start:0x%x line end:0x%x",
csid_hw->hw_intf->hw_idx, path_data->start_line,
path_data->end_line);
} else if (reserve->sync_mode == CAM_ISP_HW_SYNC_SLAVE) {
- path_data->master_idx = reserve->master_idx;
- CAM_DBG(CAM_ISP, "CSID:%d master_idx=%d",
- csid_hw->hw_intf->hw_idx, path_data->master_idx);
+ path_data->crop_enable = 1;
path_data->start_pixel = reserve->in_port->right_start;
path_data->end_pixel = reserve->in_port->right_stop;
path_data->width = reserve->in_port->right_width;
CAM_DBG(CAM_ISP, "CSID:%d slave:start:0x%x end:0x%x width 0x%x",
csid_hw->hw_intf->hw_idx, path_data->start_pixel,
path_data->end_pixel, path_data->width);
- CAM_DBG(CAM_ISP, "CSID:%d slave:line start:0x%x line end:0x%x",
+ CAM_DBG(CAM_ISP, "CSID:%dmaster:line start:0x%x line end:0x%x",
csid_hw->hw_intf->hw_idx, path_data->start_line,
path_data->end_line);
} else {
+ path_data->crop_enable = 0;
path_data->width = reserve->in_port->left_width;
path_data->start_pixel = reserve->in_port->left_start;
- path_data->end_pixel = reserve->in_port->left_stop;
- CAM_DBG(CAM_ISP, "Res id: %d left width %d start: %d stop:%d",
- reserve->res_id, reserve->in_port->left_width,
- reserve->in_port->left_start,
- reserve->in_port->left_stop);
}
CAM_DBG(CAM_ISP, "Res %d width %d height %d", reserve->res_id,
@@ -936,7 +907,7 @@ static int cam_ife_csid_enable_hw(struct cam_ife_csid_hw *csid_hw)
int rc = 0;
struct cam_ife_csid_reg_offset *csid_reg;
struct cam_hw_soc_info *soc_info;
- uint32_t i, val;
+ uint32_t i, status, val;
csid_reg = csid_hw->csid_info->csid_reg;
soc_info = &csid_hw->hw_info->soc_info;
@@ -965,10 +936,42 @@ static int cam_ife_csid_enable_hw(struct cam_ife_csid_hw *csid_hw)
goto err;
}
+
+ CAM_DBG(CAM_ISP, "CSID:%d enable top irq interrupt",
+ csid_hw->hw_intf->hw_idx);
+
csid_hw->hw_info->hw_state = CAM_HW_STATE_POWER_UP;
- /* Reset CSID top */
+ /* Enable the top IRQ interrupt */
+ cam_io_w_mb(1, soc_info->reg_map[0].mem_base +
+ csid_reg->cmn_reg->csid_top_irq_mask_addr);
+
rc = cam_ife_csid_global_reset(csid_hw);
if (rc) {
+ CAM_ERR(CAM_ISP, "CSID:%d csid_reset fail rc = %d",
+ csid_hw->hw_intf->hw_idx, rc);
+ rc = -ETIMEDOUT;
+ goto disable_soc;
+ }
+
+ /*
+ * Reset the SW registers
+ * SW register reset also reset the mask irq, so poll the irq status
+ * to check the reset complete.
+ */
+ CAM_DBG(CAM_ISP, "CSID:%d Reset Software registers",
+ csid_hw->hw_intf->hw_idx);
+
+ cam_io_w_mb(csid_reg->cmn_reg->csid_rst_stb_sw_all,
+ soc_info->reg_map[0].mem_base +
+ csid_reg->cmn_reg->csid_rst_strobes_addr);
+
+ rc = readl_poll_timeout(soc_info->reg_map[0].mem_base +
+ csid_reg->cmn_reg->csid_top_irq_status_addr,
+ status, (status & 0x1) == 0x1,
+ CAM_IFE_CSID_TIMEOUT_SLEEP_US, CAM_IFE_CSID_TIMEOUT_ALL_US);
+ if (rc < 0) {
+ CAM_ERR(CAM_ISP, "software register reset timeout.....");
+ rc = -ETIMEDOUT;
goto disable_soc;
}
@@ -993,6 +996,10 @@ static int cam_ife_csid_enable_hw(struct cam_ife_csid_hw *csid_hw)
cam_io_w_mb(1, soc_info->reg_map[0].mem_base +
csid_reg->cmn_reg->csid_irq_cmd_addr);
+ /* Enable the top IRQ interrupt */
+ cam_io_w_mb(1, soc_info->reg_map[0].mem_base +
+ csid_reg->cmn_reg->csid_top_irq_mask_addr);
+
val = cam_io_r_mb(soc_info->reg_map[0].mem_base +
csid_reg->cmn_reg->csid_hw_version_addr);
CAM_DBG(CAM_ISP, "CSID:%d CSID HW version: 0x%x",
@@ -1058,7 +1065,6 @@ static int cam_ife_csid_tpg_start(struct cam_ife_csid_hw *csid_hw,
int rc = 0;
uint32_t val = 0;
struct cam_hw_soc_info *soc_info;
- struct cam_ife_csid_reg_offset *csid_reg = NULL;
csid_hw->tpg_start_cnt++;
if (csid_hw->tpg_start_cnt == 1) {
@@ -1103,10 +1109,10 @@ static int cam_ife_csid_tpg_start(struct cam_ife_csid_hw *csid_hw,
}
/* Enable the IFE force clock on for dual isp case */
- csid_reg = csid_hw->csid_info->csid_reg;
if (csid_hw->tpg_cfg.usage_type) {
rc = cam_ife_csid_enable_ife_force_clock_on(soc_info,
- csid_reg->tpg_reg->tpg_cpas_ife_reg_offset);
+ csid_hw->csid_info->csid_reg->tpg_reg->
+ tpg_cpas_ife_reg_offset);
if (rc)
return rc;
}
@@ -1116,9 +1122,10 @@ static int cam_ife_csid_tpg_start(struct cam_ife_csid_hw *csid_hw,
val |= (0x80 << 8);
val |= (((csid_hw->csi2_rx_cfg.lane_num - 1) & 0x3) << 4);
val |= 7;
-
cam_io_w_mb(val, soc_info->reg_map[0].mem_base +
- csid_reg->tpg_reg->csid_tpg_ctrl_addr);
+ csid_hw->csid_info->csid_reg->tpg_reg->
+ csid_tpg_ctrl_addr);
+
val = cam_io_r_mb(soc_info->reg_map[0].mem_base + 0x600);
CAM_DBG(CAM_ISP, "reg 0x%x = 0x%x", 0x600, val);
}
@@ -1131,7 +1138,6 @@ static int cam_ife_csid_tpg_stop(struct cam_ife_csid_hw *csid_hw,
{
int rc = 0;
struct cam_hw_soc_info *soc_info;
- struct cam_ife_csid_reg_offset *csid_reg = NULL;
if (csid_hw->tpg_start_cnt)
csid_hw->tpg_start_cnt--;
@@ -1140,7 +1146,6 @@ static int cam_ife_csid_tpg_stop(struct cam_ife_csid_hw *csid_hw,
return 0;
soc_info = &csid_hw->hw_info->soc_info;
- csid_reg = csid_hw->csid_info->csid_reg;
/* disable the TPG */
if (!csid_hw->tpg_start_cnt) {
@@ -1150,7 +1155,8 @@ static int cam_ife_csid_tpg_stop(struct cam_ife_csid_hw *csid_hw,
/* Disable the IFE force clock on for dual isp case */
if (csid_hw->tpg_cfg.usage_type)
rc = cam_ife_csid_disable_ife_force_clock_on(soc_info,
- csid_reg->tpg_reg->tpg_cpas_ife_reg_offset);
+ csid_hw->csid_info->csid_reg->tpg_reg->
+ tpg_cpas_ife_reg_offset);
/*stop the TPG */
cam_io_w_mb(0, soc_info->reg_map[0].mem_base +
@@ -1435,6 +1441,19 @@ static int cam_ife_csid_init_config_ipp_path(
cam_io_w_mb(1, soc_info->reg_map[0].mem_base +
csid_reg->ipp_reg->csid_ipp_line_drop_period_addr);
+ /*Set master or slave IPP */
+ if (path_data->sync_mode == CAM_ISP_HW_SYNC_MASTER)
+ /*Set halt mode as master */
+ val = CSID_HALT_MODE_MASTER << 2;
+ else if (path_data->sync_mode == CAM_ISP_HW_SYNC_SLAVE)
+ /*Set halt mode as slave and set master idx */
+ val = path_data->master_idx << 4 | CSID_HALT_MODE_SLAVE << 2;
+ else
+ /* Default is internal halt mode */
+ val = 0;
+
+ cam_io_w_mb(val, soc_info->reg_map[0].mem_base +
+ csid_reg->ipp_reg->csid_ipp_ctrl_addr);
/* Enable the IPP path */
val = cam_io_r_mb(soc_info->reg_map[0].mem_base +
@@ -1536,31 +1555,19 @@ static int cam_ife_csid_enable_ipp_path(
CAM_DBG(CAM_ISP, "Enable IPP path");
-
- /* Set master or slave IPP */
- if (path_data->sync_mode == CAM_ISP_HW_SYNC_MASTER)
- /*Set halt mode as master */
- val = CSID_HALT_MODE_MASTER << 2;
- else if (path_data->sync_mode == CAM_ISP_HW_SYNC_SLAVE)
- /*Set halt mode as slave and set master idx */
- val = path_data->master_idx << 4 | CSID_HALT_MODE_SLAVE << 2;
- else
- /* Default is internal halt mode */
- val = 0;
-
- /*
- * Resume at frame boundary if Master or No Sync.
- * Slave will get resume command from Master.
- */
- if (path_data->sync_mode == CAM_ISP_HW_SYNC_MASTER ||
- path_data->sync_mode == CAM_ISP_HW_SYNC_NONE)
+ /* Resume at frame boundary */
+ if (path_data->sync_mode == CAM_ISP_HW_SYNC_MASTER) {
+ val = cam_io_r_mb(soc_info->reg_map[0].mem_base +
+ csid_reg->ipp_reg->csid_ipp_ctrl_addr);
val |= CAM_CSID_RESUME_AT_FRAME_BOUNDARY;
-
- cam_io_w_mb(val, soc_info->reg_map[0].mem_base +
- csid_reg->ipp_reg->csid_ipp_ctrl_addr);
-
- CAM_DBG(CAM_ISP, "CSID:%d IPP Ctrl val: 0x%x",
- csid_hw->hw_intf->hw_idx, val);
+ cam_io_w_mb(val, soc_info->reg_map[0].mem_base +
+ csid_reg->ipp_reg->csid_ipp_ctrl_addr);
+ } else if (path_data->sync_mode == CAM_ISP_HW_SYNC_NONE) {
+ cam_io_w_mb(CAM_CSID_RESUME_AT_FRAME_BOUNDARY,
+ soc_info->reg_map[0].mem_base +
+ csid_reg->ipp_reg->csid_ipp_ctrl_addr);
+ }
+ /* for slave mode, not need to resume for slave device */
/* Enable the required ipp interrupts */
val = CSID_PATH_INFO_RST_DONE | CSID_PATH_ERROR_FIFO_OVERFLOW;
@@ -1572,7 +1579,6 @@ static int cam_ife_csid_enable_ipp_path(
cam_io_w_mb(val, soc_info->reg_map[0].mem_base +
csid_reg->ipp_reg->csid_ipp_irq_mask_addr);
- CAM_DBG(CAM_ISP, "Enable IPP IRQ mask 0x%x", val);
res->res_state = CAM_ISP_RESOURCE_STATE_STREAMING;
@@ -1663,10 +1669,6 @@ static int cam_ife_csid_init_config_rdi_path(
if (rc)
return rc;
- /* if path decode format is payload only then RDI crop is not applied */
- if (path_format == 0xF)
- path_data->crop_enable = 0;
-
/*
* RDI path config and enable the time stamp capture
* Enable the measurement blocks
@@ -1903,7 +1905,6 @@ static int cam_ife_csid_get_time_stamp(
struct cam_isp_resource_node *res;
struct cam_ife_csid_reg_offset *csid_reg;
struct cam_hw_soc_info *soc_info;
- struct cam_ife_csid_rdi_reg_offset *rdi_reg;
struct timespec64 ts;
uint32_t time_32, id;
@@ -1937,14 +1938,15 @@ static int cam_ife_csid_get_time_stamp(
time_stamp->time_stamp_val |= time_32;
} else {
id = res->res_id;
- rdi_reg = csid_reg->rdi_reg[id];
time_32 = cam_io_r_mb(soc_info->reg_map[0].mem_base +
- rdi_reg->csid_rdi_timestamp_curr1_sof_addr);
+ csid_reg->rdi_reg[id]->
+ csid_rdi_timestamp_curr1_sof_addr);
time_stamp->time_stamp_val = time_32;
time_stamp->time_stamp_val = time_stamp->time_stamp_val << 32;
time_32 = cam_io_r_mb(soc_info->reg_map[0].mem_base +
- rdi_reg->csid_rdi_timestamp_curr0_sof_addr);
+ csid_reg->rdi_reg[id]->
+ csid_rdi_timestamp_curr0_sof_addr);
time_stamp->time_stamp_val |= time_32;
}
@@ -2171,38 +2173,24 @@ static int cam_ife_csid_reset_retain_sw_reg(
struct cam_ife_csid_hw *csid_hw)
{
int rc = 0;
- uint32_t status;
struct cam_ife_csid_reg_offset *csid_reg =
csid_hw->csid_info->csid_reg;
- struct cam_hw_soc_info *soc_info;
-
- soc_info = &csid_hw->hw_info->soc_info;
- /* clear the top interrupt first */
- cam_io_w_mb(1, soc_info->reg_map[0].mem_base +
- csid_reg->cmn_reg->csid_top_irq_clear_addr);
- cam_io_w_mb(1, soc_info->reg_map[0].mem_base +
- csid_reg->cmn_reg->csid_irq_cmd_addr);
cam_io_w_mb(csid_reg->cmn_reg->csid_rst_stb,
- soc_info->reg_map[0].mem_base +
+ csid_hw->hw_info->soc_info.reg_map[0].mem_base +
csid_reg->cmn_reg->csid_rst_strobes_addr);
- rc = readl_poll_timeout(soc_info->reg_map[0].mem_base +
- csid_reg->cmn_reg->csid_top_irq_status_addr,
- status, (status & 0x1) == 0x1,
- CAM_IFE_CSID_TIMEOUT_SLEEP_US, CAM_IFE_CSID_TIMEOUT_ALL_US);
- if (rc < 0) {
- CAM_ERR(CAM_ISP, "CSID:%d csid_reset fail rc = %d",
- csid_hw->hw_intf->hw_idx, rc);
- rc = -ETIMEDOUT;
- } else {
- CAM_DBG(CAM_ISP, "CSID:%d hw reset completed %d",
+
+ CAM_DBG(CAM_ISP, " Waiting for SW reset complete from irq handler");
+ rc = wait_for_completion_timeout(&csid_hw->csid_top_complete,
+ msecs_to_jiffies(IFE_CSID_TIMEOUT));
+ if (rc <= 0) {
+ CAM_ERR(CAM_ISP, "CSID:%d reset completion in fail rc = %d",
csid_hw->hw_intf->hw_idx, rc);
+ if (rc == 0)
+ rc = -ETIMEDOUT;
+ } else {
rc = 0;
}
- cam_io_w_mb(1, soc_info->reg_map[0].mem_base +
- csid_reg->cmn_reg->csid_top_irq_clear_addr);
- cam_io_w_mb(1, soc_info->reg_map[0].mem_base +
- csid_reg->cmn_reg->csid_irq_cmd_addr);
return rc;
}
@@ -2278,6 +2266,7 @@ static int cam_ife_csid_init_hw(void *hw_priv,
rc = cam_ife_csid_reset_retain_sw_reg(csid_hw);
if (rc < 0) {
CAM_ERR(CAM_ISP, "CSID: Failed in SW reset");
+ goto end;
}
if (rc)
@@ -2507,7 +2496,6 @@ irqreturn_t cam_ife_csid_irq(int irq_num, void *data)
struct cam_ife_csid_hw *csid_hw;
struct cam_hw_soc_info *soc_info;
struct cam_ife_csid_reg_offset *csid_reg;
- struct cam_ife_csid_csi2_rx_reg_offset *csi2_reg;
uint32_t i, irq_status_top, irq_status_rx, irq_status_ipp = 0;
uint32_t irq_status_rdi[4] = {0, 0, 0, 0};
uint32_t val;
@@ -2523,7 +2511,6 @@ irqreturn_t cam_ife_csid_irq(int irq_num, void *data)
csid_reg = csid_hw->csid_info->csid_reg;
soc_info = &csid_hw->hw_info->soc_info;
- csi2_reg = csid_reg->csi2_reg;
/* read */
irq_status_top = cam_io_r_mb(soc_info->reg_map[0].mem_base +
@@ -2542,6 +2529,8 @@ irqreturn_t cam_ife_csid_irq(int irq_num, void *data)
csid_reg->rdi_reg[i]->csid_rdi_irq_status_addr);
/* clear */
+ cam_io_w_mb(irq_status_top, soc_info->reg_map[0].mem_base +
+ csid_reg->cmn_reg->csid_top_irq_clear_addr);
cam_io_w_mb(irq_status_rx, soc_info->reg_map[0].mem_base +
csid_reg->csi2_reg->csid_csi2_rx_irq_clear_addr);
if (csid_reg->cmn_reg->no_pix)
@@ -2562,6 +2551,13 @@ irqreturn_t cam_ife_csid_irq(int irq_num, void *data)
CAM_DBG(CAM_ISP, "irq_status_rdi1= 0x%x", irq_status_rdi[1]);
CAM_DBG(CAM_ISP, "irq_status_rdi2= 0x%x", irq_status_rdi[2]);
+ if (irq_status_top) {
+ CAM_DBG(CAM_ISP, "CSID global reset complete......Exit");
+ complete(&csid_hw->csid_top_complete);
+ return IRQ_HANDLED;
+ }
+
+
if (irq_status_rx & BIT(csid_reg->csi2_reg->csi2_rst_done_shift_val)) {
CAM_DBG(CAM_ISP, "csi rx reset complete");
complete(&csid_hw->csid_csi2_complete);
@@ -2622,46 +2618,38 @@ irqreturn_t cam_ife_csid_irq(int irq_num, void *data)
if (csid_hw->csid_debug & CSID_DEBUG_ENABLE_EOT_IRQ) {
if (irq_status_rx & CSID_CSI2_RX_INFO_PHY_DL0_EOT_CAPTURED) {
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID:%d PHY_DL0_EOT_CAPTURED",
+ CAM_ERR(CAM_ISP, "CSID:%d PHY_DL0_EOT_CAPTURED",
csid_hw->hw_intf->hw_idx);
}
if (irq_status_rx & CSID_CSI2_RX_INFO_PHY_DL1_EOT_CAPTURED) {
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID:%d PHY_DL1_EOT_CAPTURED",
+ CAM_ERR(CAM_ISP, "CSID:%d PHY_DL1_EOT_CAPTURED",
csid_hw->hw_intf->hw_idx);
}
if (irq_status_rx & CSID_CSI2_RX_INFO_PHY_DL2_EOT_CAPTURED) {
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID:%d PHY_DL2_EOT_CAPTURED",
+ CAM_ERR(CAM_ISP, "CSID:%d PHY_DL2_EOT_CAPTURED",
csid_hw->hw_intf->hw_idx);
}
if (irq_status_rx & CSID_CSI2_RX_INFO_PHY_DL3_EOT_CAPTURED) {
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID:%d PHY_DL3_EOT_CAPTURED",
+ CAM_ERR(CAM_ISP, "CSID:%d PHY_DL3_EOT_CAPTURED",
csid_hw->hw_intf->hw_idx);
}
}
if (csid_hw->csid_debug & CSID_DEBUG_ENABLE_SOT_IRQ) {
if (irq_status_rx & CSID_CSI2_RX_INFO_PHY_DL0_SOT_CAPTURED) {
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID:%d PHY_DL0_SOT_CAPTURED",
+ CAM_ERR(CAM_ISP, "CSID:%d PHY_DL0_SOT_CAPTURED",
csid_hw->hw_intf->hw_idx);
}
if (irq_status_rx & CSID_CSI2_RX_INFO_PHY_DL1_SOT_CAPTURED) {
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID:%d PHY_DL1_SOT_CAPTURED",
+ CAM_ERR(CAM_ISP, "CSID:%d PHY_DL1_SOT_CAPTURED",
csid_hw->hw_intf->hw_idx);
}
if (irq_status_rx & CSID_CSI2_RX_INFO_PHY_DL2_SOT_CAPTURED) {
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID:%d PHY_DL2_SOT_CAPTURED",
+ CAM_ERR(CAM_ISP, "CSID:%d PHY_DL2_SOT_CAPTURED",
csid_hw->hw_intf->hw_idx);
}
if (irq_status_rx & CSID_CSI2_RX_INFO_PHY_DL3_SOT_CAPTURED) {
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID:%d PHY_DL3_SOT_CAPTURED",
+ CAM_ERR(CAM_ISP, "CSID:%d PHY_DL3_SOT_CAPTURED",
csid_hw->hw_intf->hw_idx);
}
}
@@ -2671,16 +2659,19 @@ irqreturn_t cam_ife_csid_irq(int irq_num, void *data)
CAM_ERR(CAM_ISP, "CSID:%d LONG_PKT_CAPTURED",
csid_hw->hw_intf->hw_idx);
val = cam_io_r_mb(soc_info->reg_map[0].mem_base +
- csi2_reg->csid_csi2_rx_captured_long_pkt_0_addr);
+ csid_reg->csi2_reg->
+ csid_csi2_rx_captured_long_pkt_0_addr);
CAM_ERR(CAM_ISP, "CSID:%d long packet VC :%d DT:%d WC:%d",
csid_hw->hw_intf->hw_idx,
(val >> 22), ((val >> 16) & 0x3F), (val & 0xFFFF));
val = cam_io_r_mb(soc_info->reg_map[0].mem_base +
- csi2_reg->csid_csi2_rx_captured_long_pkt_1_addr);
+ csid_reg->csi2_reg->
+ csid_csi2_rx_captured_long_pkt_1_addr);
CAM_ERR(CAM_ISP, "CSID:%d long packet ECC :%d",
csid_hw->hw_intf->hw_idx, val);
val = cam_io_r_mb(soc_info->reg_map[0].mem_base +
- csi2_reg->csid_csi2_rx_captured_long_pkt_ftr_addr);
+ csid_reg->csi2_reg->
+ csid_csi2_rx_captured_long_pkt_ftr_addr);
CAM_ERR(CAM_ISP, "CSID:%d long pkt cal CRC:%d expected CRC:%d",
csid_hw->hw_intf->hw_idx, (val >> 16), (val & 0xFFFF));
}
@@ -2689,12 +2680,14 @@ irqreturn_t cam_ife_csid_irq(int irq_num, void *data)
CAM_ERR(CAM_ISP, "CSID:%d SHORT_PKT_CAPTURED",
csid_hw->hw_intf->hw_idx);
val = cam_io_r_mb(soc_info->reg_map[0].mem_base +
- csi2_reg->csid_csi2_rx_captured_short_pkt_0_addr);
+ csid_reg->csi2_reg->
+ csid_csi2_rx_captured_short_pkt_0_addr);
CAM_ERR(CAM_ISP, "CSID:%d short pkt VC :%d DT:%d LC:%d",
csid_hw->hw_intf->hw_idx,
(val >> 22), ((val >> 16) & 0x1F), (val & 0xFFFF));
val = cam_io_r_mb(soc_info->reg_map[0].mem_base +
- csi2_reg->csid_csi2_rx_captured_short_pkt_1_addr);
+ csid_reg->csi2_reg->
+ csid_csi2_rx_captured_short_pkt_1_addr);
CAM_ERR(CAM_ISP, "CSID:%d short packet ECC :%d",
csid_hw->hw_intf->hw_idx, val);
}
@@ -2704,7 +2697,8 @@ irqreturn_t cam_ife_csid_irq(int irq_num, void *data)
CAM_ERR(CAM_ISP, "CSID:%d CPHY_PKT_HDR_CAPTURED",
csid_hw->hw_intf->hw_idx);
val = cam_io_r_mb(soc_info->reg_map[0].mem_base +
- csi2_reg->csid_csi2_rx_captured_cphy_pkt_hdr_addr);
+ csid_reg->csi2_reg->
+ csid_csi2_rx_captured_cphy_pkt_hdr_addr);
CAM_ERR(CAM_ISP, "CSID:%d cphy packet VC :%d DT:%d WC:%d",
csid_hw->hw_intf->hw_idx,
(val >> 22), ((val >> 16) & 0x1F), (val & 0xFFFF));
@@ -2721,17 +2715,16 @@ irqreturn_t cam_ife_csid_irq(int irq_num, void *data)
if ((irq_status_ipp & CSID_PATH_INFO_INPUT_SOF) &&
(csid_hw->csid_debug & CSID_DEBUG_ENABLE_SOF_IRQ))
- CAM_ERR_RATE_LIMIT(CAM_ISP, "CSID:%d IPP SOF received",
+ CAM_ERR(CAM_ISP, "CSID:%d IPP SOF received",
csid_hw->hw_intf->hw_idx);
if ((irq_status_ipp & CSID_PATH_INFO_INPUT_EOF) &&
(csid_hw->csid_debug & CSID_DEBUG_ENABLE_EOF_IRQ))
- CAM_ERR_RATE_LIMIT(CAM_ISP, "CSID:%d IPP EOF received",
+ CAM_ERR(CAM_ISP, "CSID:%d IPP EOF received",
csid_hw->hw_intf->hw_idx);
if (irq_status_ipp & CSID_PATH_ERROR_FIFO_OVERFLOW) {
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID:%d IPP fifo over flow",
+ CAM_ERR(CAM_ISP, "CSID:%d IPP fifo over flow",
csid_hw->hw_intf->hw_idx);
/*Stop IPP path immediately */
cam_io_w_mb(CAM_CSID_HALT_IMMEDIATELY,
@@ -2749,17 +2742,14 @@ irqreturn_t cam_ife_csid_irq(int irq_num, void *data)
if ((irq_status_rdi[i] & CSID_PATH_INFO_INPUT_SOF) &&
(csid_hw->csid_debug & CSID_DEBUG_ENABLE_SOF_IRQ))
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID RDI:%d SOF received", i);
+ CAM_ERR(CAM_ISP, "CSID RDI:%d SOF received", i);
if ((irq_status_rdi[i] & CSID_PATH_INFO_INPUT_EOF) &&
(csid_hw->csid_debug & CSID_DEBUG_ENABLE_EOF_IRQ))
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID RDI:%d EOF received", i);
+ CAM_ERR(CAM_ISP, "CSID RDI:%d EOF received", i);
if (irq_status_rdi[i] & CSID_PATH_ERROR_FIFO_OVERFLOW) {
- CAM_ERR_RATE_LIMIT(CAM_ISP,
- "CSID:%d RDI fifo over flow",
+ CAM_ERR(CAM_ISP, "CSID:%d RDI fifo over flow",
csid_hw->hw_intf->hw_idx);
/*Stop RDI path immediately */
cam_io_w_mb(CAM_CSID_HALT_IMMEDIATELY,
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/cam_vfe_core.c b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/cam_vfe_core.c
index fd38a96..9fe3656 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/cam_vfe_core.c
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/cam_vfe_core.c
@@ -560,7 +560,7 @@ int cam_vfe_start(void *hw_priv, void *start_args, uint32_t arg_size)
cam_vfe_irq_top_half,
cam_ife_mgr_do_tasklet,
isp_res->tasklet_info,
- cam_tasklet_enqueue_cmd);
+ &tasklet_bh_api);
if (isp_res->irq_handle < 1)
rc = -ENOMEM;
} else if (isp_res->rdi_only_ctx) {
@@ -573,7 +573,7 @@ int cam_vfe_start(void *hw_priv, void *start_args, uint32_t arg_size)
cam_vfe_irq_top_half,
cam_ife_mgr_do_tasklet,
isp_res->tasklet_info,
- cam_tasklet_enqueue_cmd);
+ &tasklet_bh_api);
if (isp_res->irq_handle < 1)
rc = -ENOMEM;
}
@@ -606,7 +606,7 @@ int cam_vfe_start(void *hw_priv, void *start_args, uint32_t arg_size)
cam_vfe_irq_err_top_half,
cam_ife_mgr_do_tasklet,
core_info->tasklet_info,
- cam_tasklet_enqueue_cmd);
+ &tasklet_bh_api);
if (core_info->irq_err_handle < 1) {
CAM_ERR(CAM_ISP, "Error handle subscribe failure");
rc = -ENOMEM;
@@ -639,6 +639,7 @@ int cam_vfe_stop(void *hw_priv, void *stop_args, uint32_t arg_size)
if (isp_res->res_type == CAM_ISP_RESOURCE_VFE_IN) {
cam_irq_controller_unsubscribe_irq(
core_info->vfe_irq_controller, isp_res->irq_handle);
+ isp_res->irq_handle = 0;
rc = core_info->vfe_top->hw_ops.stop(
core_info->vfe_top->top_priv, isp_res,
sizeof(struct cam_isp_resource_node));
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe170/cam_vfe170.h b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe170/cam_vfe170.h
index 984adf7..a4ba2e1 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe170/cam_vfe170.h
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe170/cam_vfe170.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe_bus/cam_vfe_bus_ver2.c b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe_bus/cam_vfe_bus_ver2.c
index 3c37b83..f518bfc 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe_bus/cam_vfe_bus_ver2.c
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe_bus/cam_vfe_bus_ver2.c
@@ -106,7 +106,6 @@ struct cam_vfe_bus_ver2_common_data {
struct cam_vfe_bus_irq_evt_payload evt_payload[
CAM_VFE_BUS_VER2_PAYLOAD_MAX];
struct list_head free_payload_list;
- spinlock_t spin_lock;
struct mutex bus_mutex;
uint32_t secure_mode;
uint32_t num_sec_out;
@@ -215,23 +214,16 @@ static int cam_vfe_bus_get_evt_payload(
struct cam_vfe_bus_ver2_common_data *common_data,
struct cam_vfe_bus_irq_evt_payload **evt_payload)
{
- int rc;
-
- spin_lock(&common_data->spin_lock);
if (list_empty(&common_data->free_payload_list)) {
*evt_payload = NULL;
CAM_ERR_RATE_LIMIT(CAM_ISP, "No free payload");
- rc = -ENODEV;
- goto done;
+ return -ENODEV;
}
*evt_payload = list_first_entry(&common_data->free_payload_list,
struct cam_vfe_bus_irq_evt_payload, list);
list_del_init(&(*evt_payload)->list);
- rc = 0;
-done:
- spin_unlock(&common_data->spin_lock);
- return rc;
+ return 0;
}
static enum cam_vfe_bus_comp_grp_id
@@ -262,7 +254,6 @@ static int cam_vfe_bus_put_evt_payload(void *core_info,
struct cam_vfe_bus_ver2_common_data *common_data = NULL;
uint32_t *ife_irq_regs = NULL;
uint32_t status_reg0, status_reg1, status_reg2;
- unsigned long flags;
if (!core_info) {
CAM_ERR(CAM_ISP, "Invalid param core_info NULL");
@@ -285,12 +276,8 @@ static int cam_vfe_bus_put_evt_payload(void *core_info,
}
common_data = core_info;
-
- spin_lock_irqsave(&common_data->spin_lock, flags);
list_add_tail(&(*evt_payload)->list,
&common_data->free_payload_list);
- spin_unlock_irqrestore(&common_data->spin_lock, flags);
-
*evt_payload = NULL;
CAM_DBG(CAM_ISP, "Done");
@@ -855,9 +842,18 @@ static int cam_vfe_bus_acquire_wm(
if (rsrc_data->index < 3) {
/* Write master 0-2 refers to RDI 0/ RDI 1/RDI 2 */
switch (rsrc_data->format) {
+ case CAM_FORMAT_MIPI_RAW_10:
+ rsrc_data->width =
+ ALIGNUP((rsrc_data->width * 10 + 7) / 8, 16);
+ rsrc_data->stride = rsrc_data->width;
+ rsrc_data->height = rsrc_data->height;
+ rsrc_data->pack_fmt = 0x0;
+ rsrc_data->en_cfg = 0x1;
+ CAM_DBG(CAM_ISP, "RAW_10: w=%d, h=%d", rsrc_data->width,
+ rsrc_data->height);
+ break;
case CAM_FORMAT_MIPI_RAW_6:
case CAM_FORMAT_MIPI_RAW_8:
- case CAM_FORMAT_MIPI_RAW_10:
case CAM_FORMAT_MIPI_RAW_12:
case CAM_FORMAT_MIPI_RAW_14:
case CAM_FORMAT_MIPI_RAW_16:
@@ -876,16 +872,34 @@ static int cam_vfe_bus_acquire_wm(
rsrc_data->stride = rsrc_data->width;
break;
case CAM_FORMAT_PLAIN16_10:
- case CAM_FORMAT_PLAIN16_12:
- case CAM_FORMAT_PLAIN16_14:
- case CAM_FORMAT_PLAIN16_16:
- case CAM_FORMAT_PLAIN32_20:
rsrc_data->width = CAM_VFE_RDI_BUS_DEFAULT_WIDTH;
rsrc_data->height = 0;
rsrc_data->stride = CAM_VFE_RDI_BUS_DEFAULT_STRIDE;
rsrc_data->pack_fmt = 0x0;
rsrc_data->en_cfg = 0x3;
break;
+ case CAM_FORMAT_PLAIN16_12:
+ rsrc_data->en_cfg = 0x1;
+ rsrc_data->pack_fmt = 0x3;
+ rsrc_data->width = rsrc_data->width * 2;
+ rsrc_data->stride = rsrc_data->width;
+ break;
+ case CAM_FORMAT_PLAIN16_14:
+ rsrc_data->en_cfg = 0x1;
+ rsrc_data->pack_fmt = 0x4;
+ rsrc_data->width = rsrc_data->width * 2;
+ rsrc_data->stride = rsrc_data->width;
+ break;
+ case CAM_FORMAT_PLAIN16_16:
+ rsrc_data->en_cfg = 0x1;
+ rsrc_data->pack_fmt = 0x5;
+ rsrc_data->width = rsrc_data->width * 2;
+ rsrc_data->stride = rsrc_data->width;
+ break;
+ case CAM_FORMAT_PLAIN32_20:
+ rsrc_data->en_cfg = 0x1;
+ rsrc_data->pack_fmt = 0x9;
+ break;
case CAM_FORMAT_PLAIN64:
rsrc_data->en_cfg = 0x1;
rsrc_data->pack_fmt = 0xA;
@@ -1084,7 +1098,7 @@ static int cam_vfe_bus_start_wm(struct cam_isp_resource_node *wm_res)
bus_irq_reg_mask, wm_res,
wm_res->top_half_handler,
cam_ife_mgr_do_tasklet_buf_done,
- wm_res->tasklet_info, cam_tasklet_enqueue_cmd);
+ wm_res->tasklet_info, &tasklet_bh_api);
if (wm_res->irq_handle < 0) {
CAM_ERR(CAM_ISP, "Subscribe IRQ failed for WM %d",
rsrc_data->index);
@@ -1565,7 +1579,7 @@ static int cam_vfe_bus_start_comp_grp(struct cam_isp_resource_node *comp_grp)
bus_irq_reg_mask, comp_grp,
comp_grp->top_half_handler,
cam_ife_mgr_do_tasklet_buf_done,
- comp_grp->tasklet_info, cam_tasklet_enqueue_cmd);
+ comp_grp->tasklet_info, &tasklet_bh_api);
if (comp_grp->irq_handle < 0) {
CAM_ERR(CAM_ISP, "Subscribe IRQ failed for comp_grp %d",
rsrc_data->comp_grp_type);
@@ -2251,11 +2265,13 @@ static int cam_vfe_bus_ver2_handle_irq(uint32_t evt_id,
struct cam_irq_th_payload *th_payload)
{
struct cam_vfe_bus_ver2_priv *bus_priv;
+ int rc = 0;
bus_priv = th_payload->handler_priv;
CAM_DBG(CAM_ISP, "Enter");
- return cam_irq_controller_handle_irq(evt_id,
+ rc = cam_irq_controller_handle_irq(evt_id,
bus_priv->common_data.bus_irq_controller);
+ return (rc == IRQ_HANDLED) ? 0 : -EINVAL;
}
static int cam_vfe_bus_error_irq_top_half(uint32_t evt_id,
@@ -2286,7 +2302,6 @@ static int cam_vfe_bus_update_wm(void *priv, void *cmd_args,
struct cam_buf_io_cfg *io_cfg;
struct cam_vfe_bus_ver2_vfe_out_data *vfe_out_data = NULL;
struct cam_vfe_bus_ver2_wm_resource_data *wm_data = NULL;
- struct cam_vfe_bus_ver2_reg_offset_ubwc_client *ubwc_client = NULL;
uint32_t *reg_val_pair;
uint32_t i, j, size = 0;
uint32_t frame_inc = 0, ubwc_bw_limit = 0, camera_hw_version, val;
@@ -2322,7 +2337,7 @@ static int cam_vfe_bus_update_wm(void *priv, void *cmd_args,
}
wm_data = vfe_out_data->wm_res[i]->res_priv;
- ubwc_client = wm_data->hw_regs->ubwc_regs;
+
/* update width register */
CAM_VFE_ADD_REG_VAL_PAIR(reg_val_pair, j,
wm_data->hw_regs->buffer_width_cfg,
@@ -2332,7 +2347,6 @@ static int cam_vfe_bus_update_wm(void *priv, void *cmd_args,
/* For initial configuration program all bus registers */
val = io_cfg->planes[i].plane_stride;
- CAM_DBG(CAM_ISP, "before stride %d", val);
val = ALIGNUP(val, 16);
if (val != io_cfg->planes[i].plane_stride &&
val != wm_data->stride)
@@ -2341,14 +2355,15 @@ static int cam_vfe_bus_update_wm(void *priv, void *cmd_args,
io_cfg->planes[i].plane_stride,
val);
- if ((wm_data->stride != val ||
- !wm_data->init_cfg_done) && (wm_data->index >= 3)) {
- CAM_VFE_ADD_REG_VAL_PAIR(reg_val_pair, j,
- wm_data->hw_regs->stride,
- io_cfg->planes[i].plane_stride);
- wm_data->stride = val;
- CAM_DBG(CAM_ISP, "WM %d image stride 0x%x",
- wm_data->index, reg_val_pair[j-1]);
+ if (wm_data->stride != val || !wm_data->init_cfg_done) {
+ if ((wm_data->index >= 3) || !(wm_data->en_cfg & 0x2)) {
+ CAM_VFE_ADD_REG_VAL_PAIR(reg_val_pair, j,
+ wm_data->hw_regs->stride,
+ io_cfg->planes[i].plane_stride);
+ wm_data->stride = val;
+ CAM_DBG(CAM_ISP, "WM %d image stride 0x%x",
+ wm_data->index, reg_val_pair[j-1]);
+ }
}
if (wm_data->en_ubwc) {
@@ -2394,8 +2409,9 @@ static int cam_vfe_bus_update_wm(void *priv, void *cmd_args,
* update offset value.
*/
CAM_VFE_ADD_REG_VAL_PAIR(reg_val_pair,
- j, ubwc_client->h_init,
- wm_data->offset);
+ j,
+ wm_data->hw_regs->ubwc_regs->
+ h_init, wm_data->offset);
wm_data->h_init = wm_data->offset;
}
} else if (wm_data->h_init !=
@@ -2423,7 +2439,8 @@ static int cam_vfe_bus_update_wm(void *priv, void *cmd_args,
io_cfg->planes[i].meta_stride ||
!wm_data->init_cfg_done) {
CAM_VFE_ADD_REG_VAL_PAIR(reg_val_pair, j,
- ubwc_client->meta_stride,
+ wm_data->hw_regs->ubwc_regs->
+ meta_stride,
io_cfg->planes[i].meta_stride);
wm_data->ubwc_meta_stride =
io_cfg->planes[i].meta_stride;
@@ -2447,7 +2464,8 @@ static int cam_vfe_bus_update_wm(void *priv, void *cmd_args,
io_cfg->planes[i].meta_offset ||
!wm_data->init_cfg_done) {
CAM_VFE_ADD_REG_VAL_PAIR(reg_val_pair, j,
- ubwc_client->meta_offset,
+ wm_data->hw_regs->ubwc_regs->
+ meta_offset,
io_cfg->planes[i].meta_offset);
wm_data->ubwc_meta_offset =
io_cfg->planes[i].meta_offset;
@@ -2842,7 +2860,7 @@ static int cam_vfe_bus_process_cmd(
case CAM_ISP_HW_CMD_STOP_BUS_ERR_IRQ:
bus_priv = (struct cam_vfe_bus_ver2_priv *) priv;
if (bus_priv->error_irq_handle) {
- CAM_DBG(CAM_ISP, "Mask off bus error irq handler");
+ CAM_INFO(CAM_ISP, "Mask off bus error irq handler");
rc = cam_irq_controller_unsubscribe_irq(
bus_priv->common_data.bus_irq_controller,
bus_priv->error_irq_handle);
@@ -2956,7 +2974,6 @@ int cam_vfe_bus_ver2_init(
}
}
- spin_lock_init(&bus_priv->common_data.spin_lock);
INIT_LIST_HEAD(&bus_priv->common_data.free_payload_list);
for (i = 0; i < CAM_VFE_BUS_VER2_PAYLOAD_MAX; i++) {
INIT_LIST_HEAD(&bus_priv->common_data.evt_payload[i].list);
diff --git a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe_top/cam_vfe_camif_ver2.c b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe_top/cam_vfe_camif_ver2.c
index 90c8006..734cbdb 100644
--- a/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe_top/cam_vfe_camif_ver2.c
+++ b/drivers/media/platform/msm/camera/cam_isp/isp_hw_mgr/isp_hw/vfe_hw/vfe_top/cam_vfe_camif_ver2.c
@@ -138,9 +138,6 @@ int cam_vfe_camif_ver2_acquire_resource(
camif_data->first_line = acquire_data->vfe_in.in_port->line_start;
camif_data->last_line = acquire_data->vfe_in.in_port->line_stop;
- CAM_DBG(CAM_ISP, "hw id:%d pix_pattern:%d dsp_mode=%d",
- camif_res->hw_intf->hw_idx,
- camif_data->pix_pattern, camif_data->dsp_mode);
return rc;
}
@@ -252,8 +249,6 @@ static int cam_vfe_camif_resource_start(
/* Reg Update */
cam_io_w_mb(rsrc_data->reg_data->reg_update_cmd_data,
rsrc_data->mem_base + rsrc_data->camif_reg->reg_update_cmd);
- CAM_DBG(CAM_ISP, "hw id:%d RUP val:%d", camif_res->hw_intf->hw_idx,
- rsrc_data->reg_data->reg_update_cmd_data);
CAM_DBG(CAM_ISP, "Start Camif IFE %d Done", camif_res->hw_intf->hw_idx);
return 0;
diff --git a/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/cam_jpeg_hw_mgr.c b/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/cam_jpeg_hw_mgr.c
index 2fec5ba..5ad96dd 100644
--- a/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/cam_jpeg_hw_mgr.c
+++ b/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/cam_jpeg_hw_mgr.c
@@ -145,7 +145,8 @@ static int cam_jpeg_mgr_process_irq(void *priv, void *data)
}
rc = cam_mem_get_cpu_buf(
- p_cfg_req->hw_cfg_args.hw_update_entries[CAM_JPEG_PARAM].handle,
+ p_cfg_req->hw_cfg_args.
+ hw_update_entries[CAM_JPEG_PARAM].handle,
(uint64_t *)&kaddr, &cmd_buf_len);
if (rc) {
CAM_ERR(CAM_JPEG, "unable to get info for cmd buf: %x %d",
@@ -156,20 +157,20 @@ static int cam_jpeg_mgr_process_irq(void *priv, void *data)
cmd_buf_kaddr = (uint32_t *)kaddr;
cmd_buf_kaddr =
- (cmd_buf_kaddr +
- (p_cfg_req->hw_cfg_args.hw_update_entries[CAM_JPEG_PARAM].offset
- / sizeof(uint32_t)));
+ (cmd_buf_kaddr + (p_cfg_req->hw_cfg_args.
+ hw_update_entries[CAM_JPEG_PARAM].offset / sizeof(uint32_t)));
p_params = (struct cam_jpeg_config_inout_param_info *)cmd_buf_kaddr;
p_params->output_size = task_data->result_size;
CAM_DBG(CAM_JPEG, "Encoded Size %d", task_data->result_size);
- buf_data.num_handles =
- p_cfg_req->hw_cfg_args.num_out_map_entries;
+ buf_data.num_handles = p_cfg_req->
+ hw_cfg_args.num_out_map_entries;
for (i = 0; i < buf_data.num_handles; i++) {
buf_data.resource_handle[i] =
- p_cfg_req->hw_cfg_args.out_map_entries[i].resource_handle;
+ p_cfg_req->hw_cfg_args.
+ out_map_entries[i].resource_handle;
}
buf_data.request_id =
(uint64_t)p_cfg_req->hw_cfg_args.priv;
@@ -266,8 +267,8 @@ static int cam_jpeg_insert_cdm_change_base(
uint32_t *ch_base_iova_addr;
size_t ch_base_len;
- rc = cam_mem_get_cpu_buf(
- config_args->hw_update_entries[CAM_JPEG_CHBASE].handle,
+ rc = cam_mem_get_cpu_buf(config_args->
+ hw_update_entries[CAM_JPEG_CHBASE].handle,
&iova_addr, &ch_base_len);
if (rc) {
CAM_ERR(CAM_JPEG,
@@ -284,10 +285,10 @@ static int cam_jpeg_insert_cdm_change_base(
dev_type = ctx_data->jpeg_dev_acquire_info.dev_type;
mem_cam_base = hw_mgr->cdm_reg_map[dev_type][0]->mem_cam_base;
- size =
- hw_mgr->cdm_info[dev_type][0].cdm_ops->cdm_required_size_changebase();
- hw_mgr->cdm_info[dev_type][0].cdm_ops->cdm_write_changebase(
- ch_base_iova_addr, mem_cam_base);
+ size = hw_mgr->cdm_info[dev_type][0].cdm_ops->
+ cdm_required_size_changebase();
+ hw_mgr->cdm_info[dev_type][0].cdm_ops->
+ cdm_write_changebase(ch_base_iova_addr, mem_cam_base);
cdm_cmd = ctx_data->cdm_cmd;
cdm_cmd->cmd[cdm_cmd->cmd_arrary_count].bl_addr.mem_handle =
@@ -321,7 +322,6 @@ static int cam_jpeg_mgr_process_cmd(void *priv, void *data)
struct cam_jpeg_set_irq_cb irq_cb;
struct cam_jpeg_hw_cfg_req *p_cfg_req = NULL;
struct cam_hw_done_event_data buf_data;
- struct cam_hw_config_args *hw_cfg_args = NULL;
if (!hw_mgr || !task_data) {
CAM_ERR(CAM_JPEG, "Invalid arguments %pK %pK",
@@ -444,8 +444,8 @@ static int cam_jpeg_mgr_process_cmd(void *priv, void *data)
for (i = CAM_JPEG_CFG; i < (config_args->num_hw_update_entries - 1);
i++) {
cmd = (config_args->hw_update_entries + i);
- cdm_cmd->cmd[cdm_cmd->cmd_arrary_count].bl_addr.mem_handle
- = cmd->handle;
+ cdm_cmd->cmd[cdm_cmd->cmd_arrary_count].
+ bl_addr.mem_handle = cmd->handle;
cdm_cmd->cmd[cdm_cmd->cmd_arrary_count].offset =
cmd->offset;
cdm_cmd->cmd[cdm_cmd->cmd_arrary_count].len =
@@ -456,7 +456,8 @@ static int cam_jpeg_mgr_process_cmd(void *priv, void *data)
}
rc = cam_cdm_submit_bls(
- hw_mgr->cdm_info[dev_type][0].cdm_handle, cdm_cmd);
+ hw_mgr->cdm_info[dev_type][0].cdm_handle,
+ cdm_cmd);
if (rc) {
CAM_ERR(CAM_JPEG, "Failed to apply the configs %d", rc);
goto end_callcb;
@@ -482,14 +483,15 @@ static int cam_jpeg_mgr_process_cmd(void *priv, void *data)
end_callcb:
mutex_unlock(&hw_mgr->hw_mgr_mutex);
if (p_cfg_req) {
- hw_cfg_args = &p_cfg_req->hw_cfg_args;
- buf_data.num_handles =
- hw_cfg_args->num_out_map_entries;
+ buf_data.num_handles = p_cfg_req->
+ hw_cfg_args.num_out_map_entries;
for (i = 0; i < buf_data.num_handles; i++) {
buf_data.resource_handle[i] =
- hw_cfg_args->out_map_entries[i].resource_handle;
+ p_cfg_req->hw_cfg_args.
+ out_map_entries[i].resource_handle;
}
- buf_data.request_id = (uint64_t)p_cfg_req->hw_cfg_args.priv;
+ buf_data.request_id =
+ (uint64_t)p_cfg_req->hw_cfg_args.priv;
ctx_data->ctxt_event_cb(ctx_data->context_priv, 0, &buf_data);
}
@@ -751,8 +753,8 @@ static int cam_jpeg_mgr_flush(void *hw_mgr_priv,
p_cfg_req = hw_mgr->dev_hw_cfg_args[dev_type][0];
if (hw_mgr->device_in_use[dev_type][0] == true &&
p_cfg_req != NULL) {
- if ((struct cam_jpeg_hw_ctx_data *)
- p_cfg_req->hw_cfg_args.ctxt_to_hw_map == ctx_data) {
+ if ((struct cam_jpeg_hw_ctx_data *)p_cfg_req->
+ hw_cfg_args.ctxt_to_hw_map == ctx_data) {
/* stop reset Unregister CB and deinit */
irq_cb.jpeg_hw_mgr_cb = cam_jpeg_hw_mgr_cb;
irq_cb.data = NULL;
@@ -809,8 +811,8 @@ static int cam_jpeg_mgr_flush(void *hw_mgr_priv,
list_for_each_entry_safe(cfg_req, req_temp,
&hw_mgr->hw_config_req_list, list) {
- if ((cfg_req) && ((struct cam_jpeg_hw_ctx_data *)
- cfg_req->hw_cfg_args.ctxt_to_hw_map != ctx_data))
+ if ((cfg_req) && ((struct cam_jpeg_hw_ctx_data *)cfg_req->
+ hw_cfg_args.ctxt_to_hw_map != ctx_data))
continue;
list_del_init(&cfg_req->list);
@@ -963,6 +965,7 @@ static int cam_jpeg_mgr_release_hw(void *hw_mgr_priv, void *release_hw_args)
if (hw_mgr->cdm_info[dev_type][0].ref_cnt == 0) {
mutex_unlock(&hw_mgr->hw_mgr_mutex);
CAM_ERR(CAM_JPEG, "Error Unbalanced deinit");
+ kfree(ctx_data->cdm_cmd);
return -EFAULT;
}
@@ -977,20 +980,16 @@ static int cam_jpeg_mgr_release_hw(void *hw_mgr_priv, void *release_hw_args)
cam_cdm_release(hw_mgr->cdm_info[dev_type][0].cdm_handle);
}
- mutex_unlock(&hw_mgr->hw_mgr_mutex);
-
rc = cam_jpeg_mgr_release_ctx(hw_mgr, ctx_data);
if (rc) {
mutex_unlock(&hw_mgr->hw_mgr_mutex);
CAM_ERR(CAM_JPEG, "JPEG release ctx failed");
kfree(ctx_data->cdm_cmd);
- ctx_data->cdm_cmd = NULL;
-
return -EINVAL;
}
+ mutex_unlock(&hw_mgr->hw_mgr_mutex);
kfree(ctx_data->cdm_cmd);
- ctx_data->cdm_cmd = NULL;
CAM_DBG(CAM_JPEG, "handle %llu", ctx_data);
return rc;
@@ -1091,8 +1090,8 @@ static int cam_jpeg_mgr_acquire_hw(void *hw_mgr_priv, void *acquire_hw_args)
hw_mgr->cdm_info[dev_type][0].ref_cnt++;
}
- size =
- hw_mgr->cdm_info[dev_type][0].cdm_ops->cdm_required_size_changebase();
+ size = hw_mgr->cdm_info[dev_type][0].
+ cdm_ops->cdm_required_size_changebase();
if (hw_mgr->cdm_info[dev_type][0].ref_cnt == 1)
if (cam_cdm_stream_on(
@@ -1379,8 +1378,8 @@ static int cam_jpeg_init_devices(struct device_node *of_node,
int cam_jpeg_hw_mgr_init(struct device_node *of_node, uint64_t *hw_mgr_hdl)
{
int i, rc;
- uint32_t num_dev;
- uint32_t num_dma_dev;
+ uint32_t num_dev = 0;
+ uint32_t num_dma_dev = 0;
struct cam_hw_mgr_intf *hw_mgr_intf;
struct cam_iommu_handle cdm_handles;
diff --git a/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_dma_hw/jpeg_dma_core.c b/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_dma_hw/jpeg_dma_core.c
index bc6dce9..2d343dd 100644
--- a/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_dma_hw/jpeg_dma_core.c
+++ b/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_dma_hw/jpeg_dma_core.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -47,8 +47,9 @@ int cam_jpeg_dma_init_hw(void *device_priv,
}
soc_info = &jpeg_dma_dev->soc_info;
- core_info = (struct cam_jpeg_dma_device_core_info *)
- jpeg_dma_dev->core_info;
+ core_info =
+ (struct cam_jpeg_dma_device_core_info *)jpeg_dma_dev->
+ core_info;
if (!soc_info || !core_info) {
CAM_ERR(CAM_JPEG, "soc_info = %pK core_info = %pK",
@@ -158,8 +159,9 @@ int cam_jpeg_dma_process_cmd(void *device_priv, uint32_t cmd_type,
return -EINVAL;
}
- core_info = (struct cam_jpeg_dma_device_core_info *)
- jpeg_dma_dev->core_info;
+ core_info =
+ (struct cam_jpeg_dma_device_core_info *)jpeg_dma_dev->
+ core_info;
switch (cmd_type) {
case CAM_JPEG_CMD_SET_IRQ_CB:
diff --git a/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_dma_hw/jpeg_dma_dev.c b/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_dma_hw/jpeg_dma_dev.c
index ef10406..05ae7cc 100644
--- a/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_dma_hw/jpeg_dma_dev.c
+++ b/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_dma_hw/jpeg_dma_dev.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -153,8 +153,8 @@ static int cam_jpeg_dma_probe(struct platform_device *pdev)
rc = -ENOMEM;
goto error_alloc_core;
}
- core_info = (struct cam_jpeg_dma_device_core_info *)
- jpeg_dma_dev->core_info;
+ core_info = (struct cam_jpeg_dma_device_core_info *)jpeg_dma_dev->
+ core_info;
match_dev = of_match_device(pdev->dev.driver->of_match_table,
&pdev->dev);
diff --git a/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_enc_hw/jpeg_enc_core.c b/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_enc_hw/jpeg_enc_core.c
index 9fa691b..87c489c 100644
--- a/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_enc_hw/jpeg_enc_core.c
+++ b/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_enc_hw/jpeg_enc_core.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -58,8 +58,9 @@ int cam_jpeg_enc_init_hw(void *device_priv,
}
soc_info = &jpeg_enc_dev->soc_info;
- core_info = (struct cam_jpeg_enc_device_core_info *)
- jpeg_enc_dev->core_info;
+ core_info =
+ (struct cam_jpeg_enc_device_core_info *)jpeg_enc_dev->
+ core_info;
if (!soc_info || !core_info) {
CAM_ERR(CAM_JPEG, "soc_info = %pK core_info = %pK",
@@ -167,8 +168,9 @@ irqreturn_t cam_jpeg_enc_irq(int irq_num, void *data)
return IRQ_HANDLED;
}
soc_info = &jpeg_enc_dev->soc_info;
- core_info = (struct cam_jpeg_enc_device_core_info *)
- jpeg_enc_dev->core_info;
+ core_info =
+ (struct cam_jpeg_enc_device_core_info *)jpeg_enc_dev->
+ core_info;
hw_info = core_info->jpeg_enc_hw_info;
mem_base = soc_info->reg_map[0].mem_base;
@@ -186,7 +188,8 @@ irqreturn_t cam_jpeg_enc_irq(int irq_num, void *data)
spin_lock(&jpeg_enc_dev->hw_lock);
if (core_info->core_state == CAM_JPEG_ENC_CORE_READY) {
encoded_size = cam_io_r_mb(mem_base +
- core_info->jpeg_enc_hw_info->reg_offset.encode_size);
+ core_info->jpeg_enc_hw_info->reg_offset.
+ encode_size);
if (core_info->irq_cb.jpeg_hw_mgr_cb) {
core_info->irq_cb.jpeg_hw_mgr_cb(irq_status,
encoded_size,
@@ -260,8 +263,9 @@ int cam_jpeg_enc_reset_hw(void *data,
}
/* maskdisable.clrirq.maskenable.resetcmd */
soc_info = &jpeg_enc_dev->soc_info;
- core_info = (struct cam_jpeg_enc_device_core_info *)
- jpeg_enc_dev->core_info;
+ core_info =
+ (struct cam_jpeg_enc_device_core_info *)jpeg_enc_dev->
+ core_info;
hw_info = core_info->jpeg_enc_hw_info;
mem_base = soc_info->reg_map[0].mem_base;
@@ -344,12 +348,13 @@ int cam_jpeg_enc_stop_hw(void *data,
return -EINVAL;
}
soc_info = &jpeg_enc_dev->soc_info;
- core_info = (struct cam_jpeg_enc_device_core_info *)
- jpeg_enc_dev->core_info;
+ core_info =
+ (struct cam_jpeg_enc_device_core_info *)jpeg_enc_dev->
+ core_info;
hw_info = core_info->jpeg_enc_hw_info;
mem_base = soc_info->reg_map[0].mem_base;
- mutex_unlock(&core_info->core_mutex);
+ mutex_lock(&core_info->core_mutex);
spin_lock(&jpeg_enc_dev->hw_lock);
if (core_info->core_state == CAM_JPEG_ENC_CORE_ABORTING) {
CAM_ERR(CAM_JPEG, "alrady stopping");
@@ -393,8 +398,9 @@ int cam_jpeg_enc_process_cmd(void *device_priv, uint32_t cmd_type,
return -EINVAL;
}
- core_info = (struct cam_jpeg_enc_device_core_info *)
- jpeg_enc_dev->core_info;
+ core_info =
+ (struct cam_jpeg_enc_device_core_info *)jpeg_enc_dev->
+ core_info;
switch (cmd_type) {
case CAM_JPEG_CMD_SET_IRQ_CB:
diff --git a/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_enc_hw/jpeg_enc_dev.c b/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_enc_hw/jpeg_enc_dev.c
index d1eb526..2180448 100644
--- a/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_enc_hw/jpeg_enc_dev.c
+++ b/drivers/media/platform/msm/camera/cam_jpeg/jpeg_hw/jpeg_enc_hw/jpeg_enc_dev.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -152,8 +152,8 @@ static int cam_jpeg_enc_probe(struct platform_device *pdev)
rc = -ENOMEM;
goto error_alloc_core;
}
- core_info = (struct cam_jpeg_enc_device_core_info *)
- jpeg_enc_dev->core_info;
+ core_info = (struct cam_jpeg_enc_device_core_info *)jpeg_enc_dev->
+ core_info;
match_dev = of_match_device(pdev->dev.driver->of_match_table,
&pdev->dev);
diff --git a/drivers/media/platform/msm/camera/cam_lrme/lrme_hw_mgr/cam_lrme_hw_mgr.c b/drivers/media/platform/msm/camera/cam_lrme/lrme_hw_mgr/cam_lrme_hw_mgr.c
index 898997a..1aec782 100644
--- a/drivers/media/platform/msm/camera/cam_lrme/lrme_hw_mgr/cam_lrme_hw_mgr.c
+++ b/drivers/media/platform/msm/camera/cam_lrme/lrme_hw_mgr/cam_lrme_hw_mgr.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -391,14 +391,13 @@ static int cam_lrme_mgr_util_submit_req(void *priv, void *data)
work_data = (struct cam_lrme_mgr_work_data *)data;
hw_device = work_data->hw_device;
- rc = cam_lrme_mgr_util_get_frame_req(
- &hw_device->frame_pending_list_high, &frame_req,
- &hw_device->high_req_lock);
+ rc = cam_lrme_mgr_util_get_frame_req(&hw_device->
+ frame_pending_list_high, &frame_req, &hw_device->high_req_lock);
if (!frame_req) {
- rc = cam_lrme_mgr_util_get_frame_req(
- &hw_device->frame_pending_list_normal, &frame_req,
- &hw_device->normal_req_lock);
+ rc = cam_lrme_mgr_util_get_frame_req(&hw_device->
+ frame_pending_list_normal, &frame_req,
+ &hw_device->normal_req_lock);
if (frame_req)
req_prio = 1;
}
@@ -479,7 +478,7 @@ static int cam_lrme_mgr_util_release(struct cam_lrme_hw_mgr *hw_mgr,
uint32_t device_index)
{
int rc = 0;
- struct cam_lrme_device *hw_device;
+ struct cam_lrme_device *hw_device = NULL;
rc = cam_lrme_mgr_util_get_device(hw_mgr, device_index, &hw_device);
if (rc) {
@@ -653,7 +652,7 @@ static int cam_lrme_mgr_hw_flush(void *hw_mgr_priv, void *hw_flush_args)
{ int rc = 0, i;
struct cam_lrme_hw_mgr *hw_mgr = hw_mgr_priv;
struct cam_hw_flush_args *args;
- struct cam_lrme_device *hw_device;
+ struct cam_lrme_device *hw_device = NULL;
struct cam_lrme_frame_request *frame_req = NULL, *req_to_flush = NULL;
struct cam_lrme_frame_request **req_list = NULL;
uint32_t device_index;
@@ -735,7 +734,7 @@ static int cam_lrme_mgr_hw_start(void *hw_mgr_priv, void *hw_start_args)
struct cam_lrme_hw_mgr *hw_mgr = hw_mgr_priv;
struct cam_hw_start_args *args =
(struct cam_hw_start_args *)hw_start_args;
- struct cam_lrme_device *hw_device;
+ struct cam_lrme_device *hw_device = NULL;
uint32_t device_index;
if (!hw_mgr || !args) {
@@ -774,7 +773,7 @@ static int cam_lrme_mgr_hw_stop(void *hw_mgr_priv, void *stop_args)
struct cam_lrme_hw_mgr *hw_mgr = hw_mgr_priv;
struct cam_hw_stop_args *args =
(struct cam_hw_stop_args *)stop_args;
- struct cam_lrme_device *hw_device;
+ struct cam_lrme_device *hw_device = NULL;
uint32_t device_index;
if (!hw_mgr_priv || !stop_args) {
@@ -816,7 +815,7 @@ static int cam_lrme_mgr_hw_prepare_update(void *hw_mgr_priv,
struct cam_lrme_hw_mgr *hw_mgr = hw_mgr_priv;
struct cam_hw_prepare_update_args *args =
(struct cam_hw_prepare_update_args *)hw_prepare_update_args;
- struct cam_lrme_device *hw_device;
+ struct cam_lrme_device *hw_device = NULL;
struct cam_kmd_buf_info kmd_buf;
struct cam_lrme_hw_cmd_config_args config_args;
struct cam_lrme_frame_request *frame_req = NULL;
diff --git a/drivers/media/platform/msm/camera/cam_lrme/lrme_hw_mgr/lrme_hw/cam_lrme_hw_core.c b/drivers/media/platform/msm/camera/cam_lrme/lrme_hw_mgr/lrme_hw/cam_lrme_hw_core.c
index 6f98354..b6ac8fa 100644
--- a/drivers/media/platform/msm/camera/cam_lrme/lrme_hw_mgr/lrme_hw/cam_lrme_hw_core.c
+++ b/drivers/media/platform/msm/camera/cam_lrme/lrme_hw_mgr/lrme_hw/cam_lrme_hw_core.c
@@ -449,8 +449,8 @@ static int cam_lrme_hw_util_flush_ctx(struct cam_hw_info *lrme_hw,
cb_args.cb_type = CAM_LRME_CB_PUT_FRAME;
cb_args.frame_req = req_submit;
if (lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb)
- lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb(
- lrme_core->hw_mgr_cb.data, &cb_args);
+ lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb(lrme_core->
+ hw_mgr_cb.data, &cb_args);
} else if (req_submit) {
submit_args.frame_req = req_submit;
submit_args.hw_update_entries = req_submit->hw_update_entries;
@@ -468,8 +468,8 @@ static int cam_lrme_hw_util_flush_ctx(struct cam_hw_info *lrme_hw,
cb_args.cb_type = CAM_LRME_CB_PUT_FRAME;
cb_args.frame_req = req_proc;
if (lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb)
- lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb(
- lrme_core->hw_mgr_cb.data, &cb_args);
+ lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb(lrme_core->
+ hw_mgr_cb.data, &cb_args);
} else if (req_proc) {
submit_args.frame_req = req_proc;
submit_args.hw_update_entries = req_proc->hw_update_entries;
@@ -511,8 +511,8 @@ static int cam_lrme_hw_util_flush_req(struct cam_hw_info *lrme_hw,
cb_args.cb_type = CAM_LRME_CB_PUT_FRAME;
cb_args.frame_req = req_submit;
if (lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb)
- lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb(
- lrme_core->hw_mgr_cb.data, &cb_args);
+ lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb(lrme_core->
+ hw_mgr_cb.data, &cb_args);
} else if (req_submit) {
submit_args.frame_req = req_submit;
submit_args.hw_update_entries = req_submit->hw_update_entries;
@@ -530,8 +530,8 @@ static int cam_lrme_hw_util_flush_req(struct cam_hw_info *lrme_hw,
cb_args.cb_type = CAM_LRME_CB_PUT_FRAME;
cb_args.frame_req = req_proc;
if (lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb)
- lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb(
- lrme_core->hw_mgr_cb.data, &cb_args);
+ lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb(lrme_core->
+ hw_mgr_cb.data, &cb_args);
} else if (req_proc) {
submit_args.frame_req = req_proc;
submit_args.hw_update_entries = req_proc->hw_update_entries;
@@ -745,8 +745,8 @@ int cam_lrme_hw_process_irq(void *priv, void *data)
}
if (lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb) {
- lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb(
- lrme_core->hw_mgr_cb.data, &cb_args);
+ lrme_core->hw_mgr_cb.cam_lrme_hw_mgr_cb(lrme_core->
+ hw_mgr_cb.data, &cb_args);
} else {
CAM_ERR(CAM_LRME, "No hw mgr cb");
rc = -EINVAL;
@@ -874,7 +874,8 @@ int cam_lrme_hw_stop(void *hw_priv, void *hw_stop_args, uint32_t arg_size)
lrme_core->state = CAM_LRME_CORE_STATE_INIT;
} else {
CAM_ERR(CAM_LRME, "HW in wrong state %d", lrme_core->state);
- return -EINVAL;
+ rc = -EINVAL;
+ goto unlock;
}
unlock:
@@ -926,7 +927,8 @@ int cam_lrme_hw_submit_req(void *hw_priv, void *hw_submit_args,
if (lrme_core->req_submit != NULL) {
CAM_ERR(CAM_LRME, "req_submit is not NULL");
- return -EBUSY;
+ rc = -EBUSY;
+ goto error;
}
rc = cam_lrme_hw_util_submit_req(lrme_core, frame_req);
diff --git a/drivers/media/platform/msm/camera/cam_req_mgr/cam_mem_mgr.c b/drivers/media/platform/msm/camera/cam_req_mgr/cam_mem_mgr.c
index 93e4249..643b0afc 100644
--- a/drivers/media/platform/msm/camera/cam_req_mgr/cam_mem_mgr.c
+++ b/drivers/media/platform/msm/camera/cam_req_mgr/cam_mem_mgr.c
@@ -646,10 +646,6 @@ int cam_mem_mgr_map(struct cam_mem_mgr_map_cmd *cmd)
CAM_SMMU_REGION_IO);
if (rc)
goto map_fail;
- } else {
- rc = ion_handle_get_size(tbl.client, ion_hdl, &len);
- if (rc)
- return rc;
}
idx = cam_mem_get_slot();
diff --git a/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_core.c b/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_core.c
index ff18fa74..0c162d0b 100644
--- a/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_core.c
+++ b/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_core.c
@@ -183,7 +183,6 @@ static int __cam_req_mgr_traverse(struct cam_req_mgr_traverse *traverse_data)
int32_t curr_idx = traverse_data->idx;
struct cam_req_mgr_req_tbl *tbl;
struct cam_req_mgr_apply *apply_data;
- struct cam_req_mgr_tbl_slot *slot = NULL;
if (!traverse_data->tbl || !traverse_data->apply_data) {
CAM_ERR(CAM_CRM, "NULL pointer %pK %pK",
@@ -194,18 +193,17 @@ static int __cam_req_mgr_traverse(struct cam_req_mgr_traverse *traverse_data)
tbl = traverse_data->tbl;
apply_data = traverse_data->apply_data;
- slot = &tbl->slot[curr_idx];
CAM_DBG(CAM_CRM,
"Enter pd %d idx %d state %d skip %d status %d skip_idx %d",
tbl->pd, curr_idx, tbl->slot[curr_idx].state,
tbl->skip_traverse, traverse_data->in_q->slot[curr_idx].status,
traverse_data->in_q->slot[curr_idx].skip_idx);
- if ((slot->inject_delay > 0) &&
+ if ((tbl->inject_delay > 0) &&
(traverse_data->self_link == true)) {
CAM_DBG(CAM_CRM, "Injecting Delay of one frame");
apply_data[tbl->pd].req_id = -1;
- slot->inject_delay--;
+ tbl->inject_delay--;
/* This pd table is not ready to proceed with asked idx */
SET_FAILURE_BIT(traverse_data->result, tbl->pd);
return -EAGAIN;
@@ -232,7 +230,7 @@ static int __cam_req_mgr_traverse(struct cam_req_mgr_traverse *traverse_data)
traverse_data->in_q, curr_idx);
apply_data[tbl->pd].idx = curr_idx;
- CAM_DBG(CAM_CRM, "req_id: %lld with pd of %d",
+ CAM_DBG(CAM_CRM, "req_id: %d with pd of %d",
apply_data[tbl->pd].req_id,
apply_data[tbl->pd].pd);
/*
@@ -250,11 +248,6 @@ static int __cam_req_mgr_traverse(struct cam_req_mgr_traverse *traverse_data)
}
} else {
/* This pd table is not ready to proceed with asked idx */
- CAM_INFO(CAM_CRM,
- "Skip Frame: req: %lld not ready pd: %d open_req count: %d",
- CRM_GET_REQ_ID(traverse_data->in_q, curr_idx),
- tbl->pd,
- traverse_data->open_req_cnt);
SET_FAILURE_BIT(traverse_data->result, tbl->pd);
return -EAGAIN;
}
@@ -457,9 +450,6 @@ static int __cam_req_mgr_send_req(struct cam_req_mgr_core_link *link,
rc = dev->ops->apply_req(&apply_req);
if (rc < 0)
break;
-
- if (pd == link->max_delay)
- link->open_req_cnt--;
}
}
}
@@ -470,7 +460,6 @@ static int __cam_req_mgr_send_req(struct cam_req_mgr_core_link *link,
for (; i >= 0; i--) {
dev = &link->l_dev[i];
evt_data.evt_type = CAM_REQ_MGR_LINK_EVT_ERR;
- evt_data.dev_hdl = dev->dev_hdl;
evt_data.link_hdl = link->link_hdl;
evt_data.req_id = apply_req.request_id;
evt_data.u.error = CRM_KMD_ERR_BUBBLE;
@@ -520,7 +509,6 @@ static int __cam_req_mgr_check_link_is_ready(struct cam_req_mgr_core_link *link,
traverse_data.result = 0;
traverse_data.validate_only = validate_only;
traverse_data.self_link = self_link;
- traverse_data.open_req_cnt = link->open_req_cnt;
/*
* Traverse through all pd tables, if result is success,
* apply the settings
@@ -671,7 +659,7 @@ static int __cam_req_mgr_validate_sof_cnt(
link->link_hdl, link->sof_counter,
sync_link->sof_counter, sync_diff, link->sync_self_ref);
- if (sync_diff > SYNC_LINK_SOF_CNT_MAX_LMT) {
+ if (sync_diff != link->sync_self_ref) {
link->sync_link->frame_skip_flag = true;
CAM_WARN(CAM_CRM,
"Detected anomaly, skip link_hdl %x self_counter=%lld other_counter=%lld sync_self_ref=%lld",
@@ -715,16 +703,6 @@ static int __cam_req_mgr_process_sync_req(
link->link_hdl, req_id, link->sync_self_ref, link->sof_counter,
link->frame_skip_flag, link->sync_link->sync_self_ref);
- if (sync_link->sync_link_sof_skip) {
- CAM_DBG(CAM_CRM,
- "No req applied on corresponding SOF on sync link: %x",
- sync_link->link_hdl);
- sync_link->sync_link_sof_skip = false;
- /*It is to manage compensate inject delay for each pd*/
- __cam_req_mgr_check_link_is_ready(link, slot->idx, true, true);
- return -EINVAL;
- }
-
if (link->sof_counter == -1) {
__cam_req_mgr_sof_cnt_initialize(link);
} else if ((link->frame_skip_flag) &&
@@ -742,7 +720,6 @@ static int __cam_req_mgr_process_sync_req(
CAM_DBG(CAM_CRM,
"Req: %lld [My link]not available link: %x, rc=%d",
req_id, link->link_hdl, rc);
- link->sync_link_sof_skip = true;
goto failure;
}
@@ -791,7 +768,6 @@ static int __cam_req_mgr_process_sync_req(
"Req: %lld [Other link] not ready to apply on link: %x",
req_id, sync_link->link_hdl);
rc = -EPERM;
- link->sync_link_sof_skip = true;
goto failure;
}
@@ -908,9 +884,6 @@ static int __cam_req_mgr_process_req(struct cam_req_mgr_core_link *link,
}
spin_unlock_bh(&link->link_state_spin_lock);
- if (link->sync_link_sof_skip)
- link->sync_link_sof_skip = false;
-
if (link->trigger_mask == link->subscribe_event) {
slot->status = CRM_SLOT_STATUS_REQ_APPLIED;
link->trigger_mask = 0;
@@ -1527,7 +1500,6 @@ int cam_req_mgr_process_sched_req(void *priv, void *data)
slot->sync_mode = sched_req->sync_mode;
slot->skip_idx = 0;
slot->recover = sched_req->bubble_enable;
- link->open_req_cnt++;
__cam_req_mgr_inc_idx(&in_q->wr_idx, 1, in_q->num_slots);
mutex_unlock(&link->req.lock);
@@ -1597,13 +1569,10 @@ int cam_req_mgr_process_add_req(void *priv, void *data)
goto end;
}
- slot = &tbl->slot[idx];
- if (add_req->skip_before_applying > slot->inject_delay) {
- slot->inject_delay = add_req->skip_before_applying;
- CAM_DBG(CAM_CRM, "Req_id %llu injecting delay %u",
- add_req->req_id, add_req->skip_before_applying);
- }
+ if (add_req->skip_before_applying > tbl->inject_delay)
+ tbl->inject_delay = add_req->skip_before_applying;
+ slot = &tbl->slot[idx];
if (slot->state != CRM_REQ_STATE_PENDING &&
slot->state != CRM_REQ_STATE_EMPTY) {
CAM_WARN(CAM_CRM, "Unexpected state %d for slot %d map %x",
@@ -1702,8 +1671,8 @@ int cam_req_mgr_process_error(void *priv, void *data)
evt_data.u.error = err_info->error;
if (device->ops &&
device->ops->process_evt)
- rc = device->ops->process_evt(
- &evt_data);
+ rc = device->ops->
+ process_evt(&evt_data);
}
}
/* Bring processing pointer to bubbled req id */
@@ -2547,13 +2516,11 @@ int cam_req_mgr_sync_config(
link1->sof_counter = -1;
link1->sync_self_ref = -1;
link1->frame_skip_flag = false;
- link1->sync_link_sof_skip = false;
link1->sync_link = link2;
link2->sof_counter = -1;
link2->sync_self_ref = -1;
link2->frame_skip_flag = false;
- link2->sync_link_sof_skip = false;
link2->sync_link = link1;
cam_session->sync_mode = sync_info->sync_mode;
diff --git a/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_core.h b/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_core.h
index 73ffb81..bea5a80 100644
--- a/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_core.h
+++ b/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_core.h
@@ -20,7 +20,7 @@
#define CAM_REQ_MGR_MAX_LINKED_DEV 16
#define MAX_REQ_SLOTS 48
-#define CAM_REQ_MGR_WATCHDOG_TIMEOUT 5000
+#define CAM_REQ_MGR_WATCHDOG_TIMEOUT 10000
#define CAM_REQ_MGR_SCHED_REQ_TIMEOUT 1000
#define CAM_REQ_MGR_SIMULATE_SCHED_REQ 30
@@ -32,8 +32,6 @@
#define MAX_SYNC_COUNT 65535
-#define SYNC_LINK_SOF_CNT_MAX_LMT 1
-
/**
* enum crm_workq_task_type
* @codes: to identify which type of task is present
@@ -134,7 +132,6 @@ enum cam_req_mgr_link_state {
* @validate_only : Whether to validate only and/or update settings
* @self_link : To indicate whether the check is for the given link or the
* other sync link
- * @open_req_cnt : Count of open requests yet to be serviced in the kernel.
*/
struct cam_req_mgr_traverse {
int32_t idx;
@@ -144,7 +141,6 @@ struct cam_req_mgr_traverse {
struct cam_req_mgr_req_queue *in_q;
bool validate_only;
bool self_link;
- int32_t open_req_cnt;
};
/**
@@ -166,13 +162,11 @@ struct cam_req_mgr_apply {
* @idx : slot index
* @req_ready_map : mask tracking which all devices have request ready
* @state : state machine for life cycle of a slot
- * @inject_delay : insert extra bubbling for flash type of use cases
*/
struct cam_req_mgr_tbl_slot {
int32_t idx;
uint32_t req_ready_map;
enum crm_req_state state;
- uint32_t inject_delay;
};
/**
@@ -187,6 +181,7 @@ struct cam_req_mgr_tbl_slot {
* @pd_delta : differnce between this table's pipeline delay and next
* @num_slots : number of request slots present in the table
* @slot : array of slots tracking requests availability at devices
+ * @inject_delay : insert extra bubbling for flash type of use cases
*/
struct cam_req_mgr_req_tbl {
int32_t id;
@@ -198,6 +193,7 @@ struct cam_req_mgr_req_tbl {
int32_t pd_delta;
int32_t num_slots;
struct cam_req_mgr_tbl_slot slot[MAX_REQ_SLOTS];
+ uint32_t inject_delay;
};
/**
@@ -300,11 +296,6 @@ struct cam_req_mgr_connected_device {
* @sync_self_ref : reference sync count against which the difference
* between sync_counts for a given link is checked
* @frame_skip_flag : flag that determines if a frame needs to be skipped
- * @sync_link_sof_skip : flag determines if a pkt is not available for a given
- * frame in a particular link skip corresponding
- * frame in sync link as well.
- * @open_req_cnt : Counter to keep track of open requests that are yet
- * to be serviced in the kernel.
*
*/
struct cam_req_mgr_core_link {
@@ -327,8 +318,6 @@ struct cam_req_mgr_core_link {
int64_t sof_counter;
int64_t sync_self_ref;
bool frame_skip_flag;
- bool sync_link_sof_skip;
- int32_t open_req_cnt;
};
/**
diff --git a/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_dev.c b/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_dev.c
index 9a93feb..cade169 100644
--- a/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_dev.c
+++ b/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_dev.c
@@ -583,6 +583,8 @@ EXPORT_SYMBOL(cam_unregister_subdev);
static int cam_req_mgr_remove(struct platform_device *pdev)
{
+ kmem_cache_destroy(g_cam_req_mgr_timer_cachep);
+ g_cam_req_mgr_timer_cachep = NULL;
cam_req_mgr_core_device_deinit();
cam_mem_mgr_deinit();
cam_req_mgr_util_deinit();
@@ -639,9 +641,8 @@ static int cam_req_mgr_probe(struct platform_device *pdev)
if (g_cam_req_mgr_timer_cachep == NULL) {
g_cam_req_mgr_timer_cachep = kmem_cache_create("crm_timer",
- sizeof(struct cam_req_mgr_timer), 64,
- SLAB_CONSISTENCY_CHECKS | SLAB_RED_ZONE |
- SLAB_POISON | SLAB_STORE_USER, NULL);
+ sizeof(struct cam_req_mgr_timer), 0,
+ SLAB_HWCACHE_ALIGN, NULL);
if (!g_cam_req_mgr_timer_cachep)
CAM_ERR(CAM_CRM,
"Failed to create kmem_cache for crm_timer");
diff --git a/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_timer.c b/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_timer.c
index 124b336..47ce72656 100644
--- a/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_timer.c
+++ b/drivers/media/platform/msm/camera/cam_req_mgr/cam_req_mgr_timer.c
@@ -92,9 +92,10 @@ void crm_timer_exit(struct cam_req_mgr_timer **crm_timer)
CAM_DBG(CAM_CRM, "destroy timer %pK @ %pK", *crm_timer, crm_timer);
if (*crm_timer) {
del_timer_sync(&(*crm_timer)->sys_timer);
- if (g_cam_req_mgr_timer_cachep)
+ if (g_cam_req_mgr_timer_cachep) {
kmem_cache_free(g_cam_req_mgr_timer_cachep, *crm_timer);
- *crm_timer = NULL;
+ *crm_timer = NULL;
+ }
}
}
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/Makefile b/drivers/media/platform/msm/camera/cam_sensor_module/Makefile
index 65c2327..2954349 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/Makefile
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/Makefile
@@ -8,3 +8,4 @@
obj-$(CONFIG_SPECTRA_CAMERA) += cam_flash/
obj-$(CONFIG_SPECTRA_CAMERA) += cam_eeprom/
obj-$(CONFIG_SPECTRA_CAMERA) += cam_ois/
+obj-$(CONFIG_CAMERA_FW_UPDATE) += cam_fw_update/
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_core.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_core.c
index 72f87060..9853f81 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_core.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_core.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -17,6 +17,12 @@
#include "cam_trace.h"
#include "cam_res_mgr_api.h"
+
+/* Implemented in fw_update.c */
+extern int checkRearVCMLTC(struct camera_io_master *io_info);
+extern void clearRearVCMInitDownload(struct camera_io_master *io_info);
+
+
int32_t cam_actuator_construct_default_power_setting(
struct cam_sensor_power_ctrl_t *power_info)
{
@@ -33,7 +39,7 @@ int32_t cam_actuator_construct_default_power_setting(
power_info->power_setting[0].seq_type = SENSOR_VAF;
power_info->power_setting[0].seq_val = CAM_VAF;
power_info->power_setting[0].config_val = 1;
- power_info->power_setting[0].delay = 2;
+ power_info->power_setting[0].delay = 12;
power_info->power_down_setting_size = 1;
power_info->power_down_setting =
@@ -53,8 +59,6 @@ int32_t cam_actuator_construct_default_power_setting(
free_power_settings:
kfree(power_info->power_setting);
- power_info->power_setting = NULL;
- power_info->power_setting_size = 0;
return rc;
}
@@ -148,16 +152,19 @@ static int32_t cam_actuator_power_down(struct cam_actuator_ctrl_t *a_ctrl)
}
camera_io_release(&a_ctrl->io_master_info);
+ clearRearVCMInitDownload(&(a_ctrl->io_master_info));
return rc;
}
static int32_t cam_actuator_i2c_modes_util(
- struct camera_io_master *io_master_info,
+ struct cam_actuator_ctrl_t *a_ctrl,
struct i2c_settings_list *i2c_list)
{
int32_t rc = 0;
uint32_t i, size;
+ struct camera_io_master *io_master_info =
+ &(a_ctrl->io_master_info);
if (i2c_list->op_code == CAM_SENSOR_I2C_WRITE_RANDOM) {
rc = camera_io_dev_write(io_master_info,
@@ -195,18 +202,50 @@ static int32_t cam_actuator_i2c_modes_util(
for (i = 0; i < size; i++) {
rc = camera_io_dev_poll(
io_master_info,
- i2c_list->i2c_settings.reg_setting[i].reg_addr,
- i2c_list->i2c_settings.reg_setting[i].reg_data,
- i2c_list->i2c_settings.reg_setting[i].data_mask,
+ i2c_list->i2c_settings.
+ reg_setting[i].reg_addr,
+ i2c_list->i2c_settings.
+ reg_setting[i].reg_data,
+ i2c_list->i2c_settings.
+ reg_setting[i].data_mask,
i2c_list->i2c_settings.addr_type,
- i2c_list->i2c_settings.data_type,
- i2c_list->i2c_settings.reg_setting[i].delay);
+ i2c_list->i2c_settings.data_type,
+ i2c_list->i2c_settings.
+ reg_setting[i].delay);
if (rc < 0) {
CAM_ERR(CAM_ACTUATOR,
"i2c poll apply setting Fail: %d", rc);
return rc;
}
}
+ } else if (i2c_list->op_code == CAM_SENSOR_I2C_READ) {
+ size = i2c_list->i2c_settings.size;
+ for (i = 0; i < size; i++) {
+ rc = camera_io_dev_read(
+ io_master_info,
+ i2c_list->i2c_settings.
+ reg_setting[i].reg_addr,
+ &(i2c_list->i2c_settings.
+ reg_setting[i].reg_data),
+ i2c_list->i2c_settings.addr_type,
+ i2c_list->i2c_settings.data_type);
+ if (rc < 0) {
+ CAM_ERR(CAM_ACTUATOR,
+ "i2c read Fail: %d", rc);
+ return rc;
+ }
+ CAM_INFO(CAM_ACTUATOR,
+ "i2c read addr: 0x%x, data: 0x%x",
+ i2c_list->i2c_settings.reg_setting[i].reg_addr,
+ i2c_list->i2c_settings.reg_setting[i].reg_data);
+ }
+
+ /* Store the read out reg_data */
+ if (a_ctrl->cmd_buf_regread != NULL && i > 0) {
+ *(a_ctrl->cmd_buf_regread) =
+ i2c_list->i2c_settings.
+ reg_setting[i - 1].reg_data;
+ }
}
return rc;
@@ -264,13 +303,14 @@ int32_t cam_actuator_apply_settings(struct cam_actuator_ctrl_t *a_ctrl,
list_for_each_entry(i2c_list,
&(i2c_set->list_head), list) {
rc = cam_actuator_i2c_modes_util(
- &(a_ctrl->io_master_info),
+ a_ctrl,
i2c_list);
if (rc < 0) {
CAM_ERR(CAM_ACTUATOR,
- "Failed to apply settings: %d",
- rc);
- return rc;
+ "Failed to apply settings: %d",rc);
+ } else {
+ CAM_DBG(CAM_ACTUATOR,
+ "Success:reqid: %d", i2c_set->request_id);
}
}
@@ -360,6 +400,28 @@ int32_t cam_actuator_establish_link(
return 0;
}
+static void cam_actuator_update_req_mgr(
+ struct cam_actuator_ctrl_t *a_ctrl,
+ struct cam_packet *csl_packet)
+{
+ struct cam_req_mgr_add_request add_req;
+
+ add_req.link_hdl = a_ctrl->bridge_intf.link_hdl;
+ add_req.req_id = csl_packet->header.request_id;
+ add_req.dev_hdl = a_ctrl->bridge_intf.device_hdl;
+ add_req.skip_before_applying = 0;
+
+ if (a_ctrl->bridge_intf.crm_cb &&
+ a_ctrl->bridge_intf.crm_cb->add_req) {
+ a_ctrl->bridge_intf.crm_cb->add_req(&add_req);
+ CAM_DBG(CAM_ACTUATOR, "Req Id: %lld added to Bridge",
+ add_req.req_id);
+ } else {
+ CAM_ERR(CAM_ACTUATOR, "Can't add Rxed Req Id: %lld",
+ csl_packet->header.request_id);
+ }
+}
+
int32_t cam_actuator_publish_dev_info(struct cam_req_mgr_device_info *info)
{
if (!info) {
@@ -392,15 +454,20 @@ int32_t cam_actuator_i2c_pkt_parse(struct cam_actuator_ctrl_t *a_ctrl,
struct i2c_data_settings *i2c_data = NULL;
struct i2c_settings_array *i2c_reg_settings = NULL;
struct cam_cmd_buf_desc *cmd_desc = NULL;
- struct cam_req_mgr_add_request add_req;
struct cam_actuator_soc_private *soc_private = NULL;
struct cam_sensor_power_ctrl_t *power_info = NULL;
+ struct i2c_settings_list *i2c_list = NULL, *i2c_next = NULL;
+ struct cam_sensor_i2c_reg_array *reg_sets;
+ uint32_t size;
if (!a_ctrl || !arg) {
CAM_ERR(CAM_ACTUATOR, "Invalid Args");
return -EINVAL;
}
+ /* To reset the pointer of reg_data */
+ a_ctrl->cmd_buf_regread = NULL;
+
soc_private =
(struct cam_actuator_soc_private *)a_ctrl->soc_info.soc_private;
@@ -490,7 +557,6 @@ int32_t cam_actuator_i2c_pkt_parse(struct cam_actuator_ctrl_t *a_ctrl,
i2c_reg_settings->request_id = 0;
i2c_reg_settings->is_settings_valid = 1;
rc = cam_sensor_i2c_command_parser(
- &a_ctrl->io_master_info,
i2c_reg_settings,
&cmd_desc[i], 1);
if (rc < 0) {
@@ -513,6 +579,31 @@ int32_t cam_actuator_i2c_pkt_parse(struct cam_actuator_ctrl_t *a_ctrl,
a_ctrl->cam_act_state = CAM_ACTUATOR_CONFIG;
}
+ rc = checkRearVCMLTC(&(a_ctrl->io_master_info));
+ if (rc == 0) {
+ /* Remove the first 2 items in initial settings */
+ /* It is already done in func checkRearVCMLTC() */
+ list_for_each_entry_safe(i2c_list, i2c_next,
+ &(a_ctrl->i2c_data.init_settings.list_head),
+ list) {
+ size = i2c_list->i2c_settings.size;
+ for (i = 0; i < size; i++) {
+ reg_sets = &i2c_list->i2c_settings
+ .reg_setting[i];
+ if ((((reg_sets->reg_addr == 0xE0) &&
+ (reg_sets->reg_data == 0x01)) ||
+ ((reg_sets->reg_addr == 0xB3) &&
+ (reg_sets->reg_data == 0x00)))
+ && size == 1) {
+ kfree(i2c_list->i2c_settings
+ .reg_setting);
+ list_del(&(i2c_list->list));
+ kfree(i2c_list);
+ }
+ }
+ }
+ }
+
rc = cam_actuator_apply_settings(a_ctrl,
&a_ctrl->i2c_data.init_settings);
if (rc < 0) {
@@ -536,6 +627,7 @@ int32_t cam_actuator_i2c_pkt_parse(struct cam_actuator_ctrl_t *a_ctrl,
a_ctrl->cam_act_state);
return rc;
}
+
a_ctrl->setting_apply_state = ACT_APPLY_SETTINGS_NOW;
i2c_data = &(a_ctrl->i2c_data);
@@ -547,15 +639,14 @@ int32_t cam_actuator_i2c_pkt_parse(struct cam_actuator_ctrl_t *a_ctrl,
offset = (uint32_t *)&csl_packet->payload;
offset += csl_packet->cmd_buf_offset / sizeof(uint32_t);
cmd_desc = (struct cam_cmd_buf_desc *)(offset);
- rc = cam_sensor_i2c_command_parser(
- &a_ctrl->io_master_info,
- i2c_reg_settings,
+ rc = cam_sensor_i2c_command_parser(i2c_reg_settings,
cmd_desc, 1);
if (rc < 0) {
CAM_ERR(CAM_ACTUATOR,
"Auto move lens parsing failed: %d", rc);
return rc;
}
+ cam_actuator_update_req_mgr(a_ctrl, csl_packet);
break;
case CAM_ACTUATOR_PACKET_MANUAL_MOVE_LENS:
if (a_ctrl->cam_act_state < CAM_ACTUATOR_CONFIG) {
@@ -565,39 +656,82 @@ int32_t cam_actuator_i2c_pkt_parse(struct cam_actuator_ctrl_t *a_ctrl,
a_ctrl->cam_act_state);
return rc;
}
+
+ a_ctrl->setting_apply_state = ACT_APPLY_SETTINGS_LATER;
i2c_data = &(a_ctrl->i2c_data);
i2c_reg_settings = &i2c_data->per_frame[
csl_packet->header.request_id % MAX_PER_FRAME_ARRAY];
+ i2c_reg_settings->request_id =
+ csl_packet->header.request_id;
+ i2c_reg_settings->is_settings_valid = 1;
+ offset = (uint32_t *)&csl_packet->payload;
+ offset += csl_packet->cmd_buf_offset / sizeof(uint32_t);
+ cmd_desc = (struct cam_cmd_buf_desc *)(offset);
+ rc = cam_sensor_i2c_command_parser(i2c_reg_settings,
+ cmd_desc, 1);
+ if (rc < 0) {
+ CAM_ERR(CAM_ACTUATOR,
+ "Manual move lens parsing failed: %d", rc);
+ return rc;
+ }
+ cam_actuator_update_req_mgr(a_ctrl, csl_packet);
+ break;
+
+ case CAM_ACTUATOR_PACKET_REG_READ:
+ if (a_ctrl->cam_act_state < CAM_ACTUATOR_CONFIG) {
+ rc = -EINVAL;
+ CAM_WARN(CAM_ACTUATOR,
+ "Not in right state to read actuator reg: %d",
+ a_ctrl->cam_act_state);
+ return rc;
+ }
+ a_ctrl->setting_apply_state = ACT_APPLY_SETTINGS_NOW;
+
+ i2c_data = &(a_ctrl->i2c_data);
+ i2c_reg_settings = &i2c_data->init_settings;
+
i2c_data->init_settings.request_id =
csl_packet->header.request_id;
i2c_reg_settings->is_settings_valid = 1;
offset = (uint32_t *)&csl_packet->payload;
offset += csl_packet->cmd_buf_offset / sizeof(uint32_t);
cmd_desc = (struct cam_cmd_buf_desc *)(offset);
- rc = cam_sensor_i2c_command_parser(
- &a_ctrl->io_master_info,
- i2c_reg_settings,
+
+ // To keep the pointer of reg_data
+ rc = cam_mem_get_cpu_buf(cmd_desc[0].mem_handle,
+ &generic_ptr, &len_of_buff);
+ if (rc < 0) {
+ CAM_ERR(CAM_ACTUATOR, "Failed to get cpu buf");
+ return rc;
+ }
+ cmd_buf = (uint32_t *)generic_ptr;
+ if (!cmd_buf) {
+ CAM_ERR(CAM_ACTUATOR, "invalid cmd buf");
+ return -EINVAL;
+ }
+ cmd_buf += cmd_desc->offset / sizeof(uint32_t);
+ a_ctrl->cmd_buf_regread = cmd_buf + 2;
+
+ rc = cam_sensor_i2c_command_parser(i2c_reg_settings,
cmd_desc, 1);
if (rc < 0) {
CAM_ERR(CAM_ACTUATOR,
- "Manual move lens parsing failed: %d", rc);
+ "read actuator reg parsing failed: %d", rc);
return rc;
}
break;
- }
- if ((csl_packet->header.op_code & 0xFFFFFF) !=
- CAM_ACTUATOR_PACKET_OPCODE_INIT) {
- add_req.link_hdl = a_ctrl->bridge_intf.link_hdl;
- add_req.req_id = csl_packet->header.request_id;
- add_req.dev_hdl = a_ctrl->bridge_intf.device_hdl;
- add_req.skip_before_applying = 0;
- if (a_ctrl->bridge_intf.crm_cb &&
- a_ctrl->bridge_intf.crm_cb->add_req)
- a_ctrl->bridge_intf.crm_cb->add_req(&add_req);
- CAM_DBG(CAM_ACTUATOR, "Req Id: %lld added to Bridge",
- add_req.req_id);
+ case CAM_PKT_NOP_OPCODE:
+ if (a_ctrl->cam_act_state < CAM_ACTUATOR_CONFIG) {
+ rc = -EINVAL;
+ CAM_WARN(CAM_ACTUATOR,
+ "Rxed NOP packets without linking state: %d",
+ a_ctrl->cam_act_state);
+ return rc;
+ }
+ cam_actuator_update_req_mgr(a_ctrl, csl_packet);
+ break;
}
return rc;
@@ -606,10 +740,6 @@ int32_t cam_actuator_i2c_pkt_parse(struct cam_actuator_ctrl_t *a_ctrl,
void cam_actuator_shutdown(struct cam_actuator_ctrl_t *a_ctrl)
{
int rc;
- struct cam_actuator_soc_private *soc_private =
- (struct cam_actuator_soc_private *)a_ctrl->soc_info.soc_private;
- struct cam_sensor_power_ctrl_t *power_info =
- &soc_private->power_info;
if (a_ctrl->cam_act_state == CAM_ACTUATOR_INIT)
return;
@@ -628,14 +758,6 @@ void cam_actuator_shutdown(struct cam_actuator_ctrl_t *a_ctrl)
a_ctrl->bridge_intf.link_hdl = -1;
a_ctrl->bridge_intf.session_hdl = -1;
}
-
- kfree(power_info->power_setting);
- kfree(power_info->power_down_setting);
- power_info->power_setting = NULL;
- power_info->power_down_setting = NULL;
- power_info->power_setting_size = 0;
- power_info->power_down_setting_size = 0;
-
a_ctrl->cam_act_state = CAM_ACTUATOR_INIT;
}
@@ -644,19 +766,12 @@ int32_t cam_actuator_driver_cmd(struct cam_actuator_ctrl_t *a_ctrl,
{
int rc = 0;
struct cam_control *cmd = (struct cam_control *)arg;
- struct cam_actuator_soc_private *soc_private = NULL;
- struct cam_sensor_power_ctrl_t *power_info = NULL;
if (!a_ctrl || !cmd) {
- CAM_ERR(CAM_ACTUATOR, "Invalid Args");
+ CAM_ERR(CAM_ACTUATOR, " Invalid Args");
return -EINVAL;
}
- soc_private =
- (struct cam_actuator_soc_private *)a_ctrl->soc_info.soc_private;
-
- power_info = &soc_private->power_info;
-
if (cmd->handle_type != CAM_HANDLE_USER_POINTER) {
CAM_ERR(CAM_ACTUATOR, "Invalid handle type: %d",
cmd->handle_type);
@@ -739,12 +854,6 @@ int32_t cam_actuator_driver_cmd(struct cam_actuator_ctrl_t *a_ctrl,
a_ctrl->bridge_intf.link_hdl = -1;
a_ctrl->bridge_intf.session_hdl = -1;
a_ctrl->cam_act_state = CAM_ACTUATOR_INIT;
- kfree(power_info->power_setting);
- kfree(power_info->power_down_setting);
- power_info->power_setting = NULL;
- power_info->power_down_setting = NULL;
- power_info->power_down_setting_size = 0;
- power_info->power_setting_size = 0;
}
break;
case CAM_QUERY_CAP: {
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_dev.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_dev.c
index 96fdfeb..40170ba 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_dev.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_dev.c
@@ -294,6 +294,42 @@ static const struct of_device_id cam_actuator_driver_dt_match[] = {
{}
};
+static bool act_first_probed;
+static char act_ltc_str[80] = {0};
+
+bool check_act_ltc_disable(void)
+{
+ if (strncmp(act_ltc_str, "0", 1) == 0) {
+ CAM_INFO(CAM_ACTUATOR, "[LTC] LTC is disable");
+ return true;
+ } else {
+ CAM_INFO(CAM_ACTUATOR, "[LTC] LTC is enable");
+ return false;
+ }
+}
+
+static ssize_t act_ltc_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ CAM_INFO(CAM_ACTUATOR, "[LTC] act_ltc_show: %s",
+ act_ltc_str);
+ return scnprintf(buf, PAGE_SIZE, "%s", act_ltc_str);
+}
+
+static ssize_t act_ltc_store(struct device *kobj,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ memset(act_ltc_str, 0, sizeof(act_ltc_str));
+ scnprintf(act_ltc_str,
+ sizeof(act_ltc_str), "%s", buf);
+ CAM_INFO(CAM_ACTUATOR, "[LTC] act_ltc_store: %s",
+ act_ltc_str);
+ return count;
+}
+
+static DEVICE_ATTR(act_ltc, 0644, act_ltc_show, act_ltc_store);
+
static int32_t cam_actuator_driver_platform_probe(
struct platform_device *pdev)
{
@@ -372,6 +408,19 @@ static int32_t cam_actuator_driver_platform_probe(
v4l2_set_subdevdata(&a_ctrl->v4l2_dev_str.sd, a_ctrl);
a_ctrl->cam_act_state = CAM_ACTUATOR_INIT;
+ if (act_first_probed == false) {
+ act_first_probed = true;
+ memset(act_ltc_str, 0, sizeof(act_ltc_str));
+ CAM_INFO(CAM_SENSOR,"[LTC] pdev->name=%s",
+ pdev->name);
+ rc = sysfs_create_file(pdev->dev.kobj.parent,
+ &dev_attr_act_ltc.attr);
+ if (rc) {
+ CAM_ERR(CAM_SENSOR,
+ "[LTC] sysfs create fail");
+ }
+ }
+
return rc;
free_mem:
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_dev.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_dev.h
index c4333a0..eaf2308 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_dev.h
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_actuator/cam_actuator_dev.h
@@ -124,6 +124,7 @@ struct cam_actuator_ctrl_t {
struct cam_actuator_query_cap act_info;
struct intf_params bridge_intf;
char device_name[20];
+ uint32_t *cmd_buf_regread; // The pointer of reg_data to store
};
#endif /* _CAM_ACTUATOR_DEV_H_ */
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_core.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_core.c
index b47f4f3..01a717d 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_core.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_core.c
@@ -94,8 +94,8 @@ static int32_t cam_cci_validate_queue(struct cci_device *cci_dev,
CAM_DBG(CAM_CCI, "CCI_I2C_M0_Q0_CUR_WORD_CNT_ADDR %d len %d max %d",
read_val, len,
cci_dev->cci_i2c_queue_info[master][queue].max_queue_size);
- if ((read_val + len + 1) >
- cci_dev->cci_i2c_queue_info[master][queue].max_queue_size) {
+ if ((read_val + len + 1) > cci_dev->
+ cci_i2c_queue_info[master][queue].max_queue_size) {
uint32_t reg_val = 0;
uint32_t report_val = CCI_I2C_REPORT_CMD | (1 << 8);
@@ -113,18 +113,16 @@ static int32_t cam_cci_validate_queue(struct cci_device *cci_dev,
CAM_DBG(CAM_CCI, "CCI_QUEUE_START_ADDR");
spin_lock_irqsave(
&cci_dev->cci_master_info[master].lock_q[queue], flags);
- atomic_set(
- &cci_dev->cci_master_info[master].done_pending[queue],
- 1);
+ atomic_set(&cci_dev->cci_master_info[master].
+ done_pending[queue], 1);
cam_io_w_mb(reg_val, base +
CCI_QUEUE_START_ADDR);
CAM_DBG(CAM_CCI, "wait_for_completion_timeout");
atomic_set(&cci_dev->cci_master_info[master].q_free[queue], 1);
spin_unlock_irqrestore(
&cci_dev->cci_master_info[master].lock_q[queue], flags);
- rc = wait_for_completion_timeout(
- &cci_dev->cci_master_info[master].report_q[queue],
- CCI_TIMEOUT);
+ rc = wait_for_completion_timeout(&cci_dev->
+ cci_master_info[master].report_q[queue], CCI_TIMEOUT);
if (rc <= 0) {
CAM_ERR(CAM_CCI, "Wait_for_completion_timeout: rc: %d",
rc);
@@ -205,6 +203,8 @@ static void cam_cci_dump_registers(struct cci_device *cci_dev,
CAM_INFO(CAM_CCI, "****CCI MASTER %d Registers ****",
master);
for (i = 0; i < DEBUG_MASTER_REG_COUNT; i++) {
+ if (i == 6)
+ continue;
reg_offset = DEBUG_MASTER_REG_START + master*0x100 + i * 4;
read_val = cam_io_r_mb(base + reg_offset);
CAM_INFO(CAM_CCI, "offset = 0x%X value = 0x%X",
@@ -244,8 +244,8 @@ static uint32_t cam_cci_wait(struct cci_device *cci_dev,
return -EINVAL;
}
- rc = wait_for_completion_timeout(
- &cci_dev->cci_master_info[master].report_q[queue], CCI_TIMEOUT);
+ rc = wait_for_completion_timeout(&cci_dev->
+ cci_master_info[master].report_q[queue], CCI_TIMEOUT);
CAM_DBG(CAM_CCI, "wait DONE_for_completion_timeout");
if (rc <= 0) {
@@ -338,9 +338,8 @@ static int32_t cam_cci_transfer_end(struct cci_device *cci_dev,
return rc;
}
} else {
- atomic_set(
- &cci_dev->cci_master_info[master].done_pending[queue],
- 1);
+ atomic_set(&cci_dev->cci_master_info[master].
+ done_pending[queue], 1);
spin_unlock_irqrestore(
&cci_dev->cci_master_info[master].lock_q[queue], flags);
rc = cam_cci_wait(cci_dev, master, queue);
@@ -377,8 +376,9 @@ static int32_t cam_cci_get_queue_free_size(struct cci_device *cci_dev,
CCI_I2C_M0_Q0_CUR_WORD_CNT_ADDR + reg_offset);
CAM_DBG(CAM_CCI, "CCI_I2C_M0_Q0_CUR_WORD_CNT_ADDR %d max %d", read_val,
cci_dev->cci_i2c_queue_info[master][queue].max_queue_size);
- return ((cci_dev->cci_i2c_queue_info[master][queue].max_queue_size) -
- read_val);
+ return (cci_dev->
+ cci_i2c_queue_info[master][queue].max_queue_size) -
+ read_val;
}
static void cam_cci_process_half_q(struct cci_device *cci_dev,
@@ -414,9 +414,8 @@ static int32_t cam_cci_process_full_q(struct cci_device *cci_dev,
spin_lock_irqsave(&cci_dev->cci_master_info[master].lock_q[queue],
flags);
if (atomic_read(&cci_dev->cci_master_info[master].q_free[queue]) == 1) {
- atomic_set(
- &cci_dev->cci_master_info[master].done_pending[queue],
- 1);
+ atomic_set(&cci_dev->cci_master_info[master].
+ done_pending[queue], 1);
spin_unlock_irqrestore(
&cci_dev->cci_master_info[master].lock_q[queue], flags);
rc = cam_cci_wait(cci_dev, master, queue);
@@ -474,10 +473,6 @@ static int32_t cam_cci_calc_cmd_len(struct cci_device *cci_dev,
if (cmd->reg_addr + 1 ==
(cmd+1)->reg_addr) {
len += data_len;
- if (len > cci_dev->payload_size) {
- len = len - data_len;
- break;
- }
(*pack)++;
} else {
break;
@@ -683,8 +678,8 @@ static int32_t cam_cci_data_queue(struct cci_device *cci_dev,
spin_unlock_irqrestore(&cci_dev->cci_master_info[master].lock_q[queue],
flags);
- max_queue_size =
- cci_dev->cci_i2c_queue_info[master][queue].max_queue_size;
+ max_queue_size = cci_dev->cci_i2c_queue_info[master][queue].
+ max_queue_size;
if (c_ctrl->cmd == MSM_CCI_I2C_WRITE_SEQ)
queue_size = max_queue_size;
@@ -866,180 +861,6 @@ static int32_t cam_cci_data_queue(struct cci_device *cci_dev,
return rc;
}
-static int32_t cam_cci_burst_read(struct v4l2_subdev *sd,
- struct cam_cci_ctrl *c_ctrl)
-{
- int32_t rc = 0;
- uint32_t val = 0, i = 0;
- unsigned long rem_jiffies;
- int32_t read_words = 0, exp_words = 0;
- int32_t index = 0, first_byte = 0, total_read_words = 0;
- enum cci_i2c_master_t master;
- enum cci_i2c_queue_t queue = QUEUE_1;
- struct cci_device *cci_dev = NULL;
- struct cam_cci_read_cfg *read_cfg = NULL;
- struct cam_hw_soc_info *soc_info = NULL;
- void __iomem *base = NULL;
-
- cci_dev = v4l2_get_subdevdata(sd);
- master = c_ctrl->cci_info->cci_i2c_master;
- read_cfg = &c_ctrl->cfg.cci_i2c_read_cfg;
-
- if (c_ctrl->cci_info->cci_i2c_master >= MASTER_MAX
- || c_ctrl->cci_info->cci_i2c_master < 0) {
- CAM_ERR(CAM_CCI, "Invalid I2C master addr");
- return -EINVAL;
- }
-
- soc_info = &cci_dev->soc_info;
- base = soc_info->reg_map[0].mem_base;
- mutex_lock(&cci_dev->cci_master_info[master].mutex_q[queue]);
-
- /*
- * Todo: If there is a change in frequency of operation
- * Wait for previos transaction to complete
- */
-
- /* Set the I2C Frequency */
- rc = cam_cci_set_clk_param(cci_dev, c_ctrl);
- if (rc < 0) {
- CAM_ERR(CAM_CCI, "cam_cci_set_clk_param failed rc = %d", rc);
- goto rel_mutex;
- }
-
- /*
- * Call validate queue to make sure queue is empty before starting.
- * If this call fails, don't proceed with i2c_read call. This is to
- * avoid overflow / underflow of queue
- */
- rc = cam_cci_validate_queue(cci_dev,
- cci_dev->cci_i2c_queue_info[master][queue].max_queue_size - 1,
- master, queue);
- if (rc < 0) {
- CAM_ERR(CAM_CCI, "Initial validataion failed rc %d", rc);
- goto rel_mutex;
- }
-
- if (c_ctrl->cci_info->retries > CCI_I2C_READ_MAX_RETRIES) {
- CAM_ERR(CAM_CCI, "More than max retries");
- goto rel_mutex;
- }
-
- if (read_cfg->data == NULL) {
- CAM_ERR(CAM_CCI, "Data ptr is NULL");
- goto rel_mutex;
- }
-
- if (read_cfg->addr_type >= CAMERA_SENSOR_I2C_TYPE_MAX) {
- CAM_ERR(CAM_CCI, "failed : Invalid addr type: %u",
- read_cfg->addr_type);
- rc = -EINVAL;
- goto rel_mutex;
- }
-
- CAM_DBG(CAM_CCI, "set param sid 0x%x retries %d id_map %d",
- c_ctrl->cci_info->sid, c_ctrl->cci_info->retries,
- c_ctrl->cci_info->id_map);
- val = CCI_I2C_SET_PARAM_CMD | c_ctrl->cci_info->sid << 4 |
- c_ctrl->cci_info->retries << 16 |
- c_ctrl->cci_info->id_map << 18;
- rc = cam_cci_write_i2c_queue(cci_dev, val, master, queue);
- if (rc < 0) {
- CAM_DBG(CAM_CCI, "failed rc: %d", rc);
- goto rel_mutex;
- }
-
- val = CCI_I2C_LOCK_CMD;
- rc = cam_cci_write_i2c_queue(cci_dev, val, master, queue);
- if (rc < 0) {
- CAM_DBG(CAM_CCI, "failed rc: %d", rc);
- goto rel_mutex;
- }
-
- val = CCI_I2C_WRITE_DISABLE_P_CMD | (read_cfg->addr_type << 4);
- for (i = 0; i < read_cfg->addr_type; i++) {
- val |= ((read_cfg->addr >> (i << 3)) & 0xFF) <<
- ((read_cfg->addr_type - i) << 3);
- }
-
- rc = cam_cci_write_i2c_queue(cci_dev, val, master, queue);
- if (rc < 0) {
- CAM_DBG(CAM_CCI, "failed rc: %d", rc);
- goto rel_mutex;
- }
-
- val = CCI_I2C_READ_CMD | (read_cfg->num_byte << 4);
- rc = cam_cci_write_i2c_queue(cci_dev, val, master, queue);
- if (rc < 0) {
- CAM_DBG(CAM_CCI, "failed rc: %d", rc);
- goto rel_mutex;
- }
-
- val = CCI_I2C_UNLOCK_CMD;
- rc = cam_cci_write_i2c_queue(cci_dev, val, master, queue);
- if (rc < 0) {
- CAM_DBG(CAM_CCI, "failed rc: %d", rc);
- goto rel_mutex;
- }
-
- val = cam_io_r_mb(base + CCI_I2C_M0_Q0_CUR_WORD_CNT_ADDR
- + master * 0x200 + queue * 0x100);
- CAM_DBG(CAM_CCI, "cur word cnt 0x%x", val);
- cam_io_w_mb(val, base + CCI_I2C_M0_Q0_EXEC_WORD_CNT_ADDR
- + master * 0x200 + queue * 0x100);
-
- val = 1 << ((master * 2) + queue);
- cam_io_w_mb(val, base + CCI_QUEUE_START_ADDR);
- exp_words = ((read_cfg->num_byte / 4) + 1);
-
- while (exp_words != total_read_words) {
- rem_jiffies = wait_for_completion_timeout(
- &cci_dev->cci_master_info[master].reset_complete,
- CCI_TIMEOUT);
- if (!rem_jiffies) {
- rc = -ETIMEDOUT;
- val = cam_io_r_mb(base +
- CCI_I2C_M0_READ_BUF_LEVEL_ADDR +
- master * 0x100);
- CAM_ERR(CAM_CCI,
- "wait_for_completion_timeout rc = %d FIFO buf_lvl:0x%x",
- rc, val);
-#ifdef DUMP_CCI_REGISTERS
- cam_cci_dump_registers(cci_dev, master, queue);
-#endif
- cam_cci_flush_queue(cci_dev, master);
- goto rel_mutex;
- }
-
- read_words = cam_io_r_mb(base +
- CCI_I2C_M0_READ_BUF_LEVEL_ADDR + master * 0x100);
- total_read_words += read_words;
- do {
- val = cam_io_r_mb(base +
- CCI_I2C_M0_READ_DATA_ADDR + master * 0x100);
- for (i = 0; (i < 4) &&
- (index < read_cfg->num_byte); i++) {
- CAM_DBG(CAM_CCI, "i:%d index:%d", i, index);
- if (!first_byte) {
- CAM_DBG(CAM_CCI, "sid 0x%x",
- val & 0xFF);
- first_byte++;
- } else {
- read_cfg->data[index] =
- (val >> (i * 8)) & 0xFF;
- CAM_DBG(CAM_CCI, "data[%d] 0x%x", index,
- read_cfg->data[index]);
- index++;
- }
- }
- } while (--read_words > 0);
- }
-
-rel_mutex:
- mutex_unlock(&cci_dev->cci_master_info[master].mutex_q[queue]);
- return rc;
-}
-
static int32_t cam_cci_read(struct v4l2_subdev *sd,
struct cam_cci_ctrl *c_ctrl)
{
@@ -1168,19 +989,15 @@ static int32_t cam_cci_read(struct v4l2_subdev *sd,
cam_io_w_mb(val, base + CCI_QUEUE_START_ADDR);
CAM_DBG(CAM_CCI, "wait_for_completion_timeout");
- rc = wait_for_completion_timeout(
- &cci_dev->cci_master_info[master].reset_complete, CCI_TIMEOUT);
+ rc = wait_for_completion_timeout(&cci_dev->
+ cci_master_info[master].reset_complete, CCI_TIMEOUT);
if (rc <= 0) {
#ifdef DUMP_CCI_REGISTERS
cam_cci_dump_registers(cci_dev, master, queue);
#endif
if (rc == 0)
rc = -ETIMEDOUT;
- val = cam_io_r_mb(base +
- CCI_I2C_M0_READ_BUF_LEVEL_ADDR + master * 0x100);
- CAM_ERR(CAM_CCI,
- "wait_for_completion_timeout rc = %d FIFO buf_lvl: 0x%x",
- rc, val);
+ CAM_ERR(CAM_CCI, "wait_for_completion_timeout rc = %d", rc);
cam_cci_flush_queue(cci_dev, master);
goto rel_mutex;
} else {
@@ -1400,26 +1217,19 @@ static int32_t cam_cci_read_bytes(struct v4l2_subdev *sd,
read_bytes = read_cfg->num_byte;
do {
- if (read_bytes > CCI_I2C_MAX_BYTE_COUNT)
- read_cfg->num_byte = CCI_I2C_MAX_BYTE_COUNT;
+ if (read_bytes > CCI_READ_MAX)
+ read_cfg->num_byte = CCI_READ_MAX;
else
read_cfg->num_byte = read_bytes;
-
- if (read_cfg->num_byte > CCI_READ_MAX)
- rc = cam_cci_burst_read(sd, c_ctrl);
- else
- rc = cam_cci_read(sd, c_ctrl);
-
- if (!rc) {
- CAM_ERR(CAM_CCI, "failed to read rc:%d", rc);
+ rc = cam_cci_read(sd, c_ctrl);
+ if (rc < 0) {
+ CAM_ERR(CAM_CCI, "failed rc %d", rc);
goto ERROR;
}
-
- if (read_bytes > CCI_I2C_MAX_BYTE_COUNT) {
- read_cfg->addr += (CCI_I2C_MAX_BYTE_COUNT /
- read_cfg->data_type);
- read_cfg->data += CCI_I2C_MAX_BYTE_COUNT;
- read_bytes -= CCI_I2C_MAX_BYTE_COUNT;
+ if (read_bytes > CCI_READ_MAX) {
+ read_cfg->addr += CCI_READ_MAX;
+ read_cfg->data += CCI_READ_MAX;
+ read_bytes -= CCI_READ_MAX;
} else {
read_bytes = 0;
}
@@ -1533,8 +1343,10 @@ int32_t cam_cci_core_cfg(struct v4l2_subdev *sd,
struct cam_cci_ctrl *cci_ctrl)
{
int32_t rc = 0;
+ struct cci_device *cci_dev = v4l2_get_subdevdata(sd);
CAM_DBG(CAM_CCI, "cmd %d", cci_ctrl->cmd);
+ mutex_lock(&cci_dev->mutex);
switch (cci_ctrl->cmd) {
case MSM_CCI_INIT:
rc = cam_cci_init(sd, cci_ctrl);
@@ -1562,6 +1374,7 @@ int32_t cam_cci_core_cfg(struct v4l2_subdev *sd,
default:
rc = -ENOIOCTLCMD;
}
+ mutex_unlock(&cci_dev->mutex);
cci_ctrl->status = rc;
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_dev.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_dev.c
index c8ca85d..64b1e72 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_dev.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_dev.c
@@ -58,54 +58,36 @@ static long cam_cci_subdev_compat_ioctl(struct v4l2_subdev *sd,
irqreturn_t cam_cci_irq(int irq_num, void *data)
{
- uint32_t irq_status0 = 0;
- uint32_t irq_status1 = 0;
+ uint32_t irq;
struct cci_device *cci_dev = data;
struct cam_hw_soc_info *soc_info =
&cci_dev->soc_info;
void __iomem *base = soc_info->reg_map[0].mem_base;
unsigned long flags;
- bool burst_read_assert = false;
- irq_status0 = cam_io_r_mb(base + CCI_IRQ_STATUS_0_ADDR);
- irq_status1 = cam_io_r_mb(base + CCI_IRQ_STATUS_1_ADDR);
- cam_io_w_mb(irq_status0, base + CCI_IRQ_CLEAR_0_ADDR);
- cam_io_w_mb(irq_status1, base + CCI_IRQ_CLEAR_1_ADDR);
+ irq = cam_io_r_mb(base + CCI_IRQ_STATUS_0_ADDR);
+ cam_io_w_mb(irq, base + CCI_IRQ_CLEAR_0_ADDR);
cam_io_w_mb(0x1, base + CCI_IRQ_GLOBAL_CLEAR_CMD_ADDR);
- CAM_DBG(CAM_CCI, "irq0:%x irq1:%x", irq_status0, irq_status1);
- if (irq_status0 & CCI_IRQ_STATUS_0_RST_DONE_ACK_BMSK) {
+ if (irq & CCI_IRQ_STATUS_0_RST_DONE_ACK_BMSK) {
if (cci_dev->cci_master_info[MASTER_0].reset_pending == TRUE) {
cci_dev->cci_master_info[MASTER_0].reset_pending =
FALSE;
- complete(
- &cci_dev->cci_master_info[MASTER_0].reset_complete);
+ complete(&cci_dev->cci_master_info[MASTER_0].
+ reset_complete);
}
if (cci_dev->cci_master_info[MASTER_1].reset_pending == TRUE) {
cci_dev->cci_master_info[MASTER_1].reset_pending =
FALSE;
- complete(
- &cci_dev->cci_master_info[MASTER_1].reset_complete);
+ complete(&cci_dev->cci_master_info[MASTER_1].
+ reset_complete);
}
}
-
- if ((irq_status0 & CCI_IRQ_STATUS_0_I2C_M0_RD_DONE_BMSK) &&
- (irq_status1 & CCI_IRQ_STATUS_1_I2C_M0_RD_THRESHOLD)) {
- cci_dev->cci_master_info[MASTER_0].status = 0;
- complete(&cci_dev->cci_master_info[MASTER_0].reset_complete);
- burst_read_assert = true;
- }
- if ((irq_status0 & CCI_IRQ_STATUS_0_I2C_M0_RD_DONE_BMSK) &&
- (!burst_read_assert)) {
+ if (irq & CCI_IRQ_STATUS_0_I2C_M0_RD_DONE_BMSK) {
cci_dev->cci_master_info[MASTER_0].status = 0;
complete(&cci_dev->cci_master_info[MASTER_0].reset_complete);
}
- if ((irq_status1 & CCI_IRQ_STATUS_1_I2C_M0_RD_THRESHOLD) &&
- (!burst_read_assert)) {
- cci_dev->cci_master_info[MASTER_0].status = 0;
- complete(&cci_dev->cci_master_info[MASTER_0].reset_complete);
- }
- if (irq_status0 & CCI_IRQ_STATUS_0_I2C_M0_Q0_REPORT_BMSK) {
+ if (irq & CCI_IRQ_STATUS_0_I2C_M0_Q0_REPORT_BMSK) {
struct cam_cci_master_info *cci_master_info;
cci_master_info = &cci_dev->cci_master_info[MASTER_0];
@@ -122,7 +104,7 @@ irqreturn_t cam_cci_irq(int irq_num, void *data)
&cci_dev->cci_master_info[MASTER_0].lock_q[QUEUE_0],
flags);
}
- if (irq_status0 & CCI_IRQ_STATUS_0_I2C_M0_Q1_REPORT_BMSK) {
+ if (irq & CCI_IRQ_STATUS_0_I2C_M0_Q1_REPORT_BMSK) {
struct cam_cci_master_info *cci_master_info;
cci_master_info = &cci_dev->cci_master_info[MASTER_0];
@@ -139,23 +121,11 @@ irqreturn_t cam_cci_irq(int irq_num, void *data)
&cci_dev->cci_master_info[MASTER_0].lock_q[QUEUE_1],
flags);
}
- if ((irq_status0 & CCI_IRQ_STATUS_0_I2C_M1_RD_DONE_BMSK) &&
- (irq_status1 & CCI_IRQ_STATUS_1_I2C_M1_RD_THRESHOLD)) {
- cci_dev->cci_master_info[MASTER_1].status = 0;
- complete(&cci_dev->cci_master_info[MASTER_1].reset_complete);
- burst_read_assert = true;
- }
- if ((irq_status0 & CCI_IRQ_STATUS_0_I2C_M1_RD_DONE_BMSK) &&
- (!burst_read_assert)) {
+ if (irq & CCI_IRQ_STATUS_0_I2C_M1_RD_DONE_BMSK) {
cci_dev->cci_master_info[MASTER_1].status = 0;
complete(&cci_dev->cci_master_info[MASTER_1].reset_complete);
}
- if ((irq_status1 & CCI_IRQ_STATUS_1_I2C_M1_RD_THRESHOLD) &&
- (!burst_read_assert)) {
- cci_dev->cci_master_info[MASTER_1].status = 0;
- complete(&cci_dev->cci_master_info[MASTER_1].reset_complete);
- }
- if (irq_status0 & CCI_IRQ_STATUS_0_I2C_M1_Q0_REPORT_BMSK) {
+ if (irq & CCI_IRQ_STATUS_0_I2C_M1_Q0_REPORT_BMSK) {
struct cam_cci_master_info *cci_master_info;
cci_master_info = &cci_dev->cci_master_info[MASTER_1];
@@ -172,7 +142,7 @@ irqreturn_t cam_cci_irq(int irq_num, void *data)
&cci_dev->cci_master_info[MASTER_1].lock_q[QUEUE_0],
flags);
}
- if (irq_status0 & CCI_IRQ_STATUS_0_I2C_M1_Q1_REPORT_BMSK) {
+ if (irq & CCI_IRQ_STATUS_0_I2C_M1_Q1_REPORT_BMSK) {
struct cam_cci_master_info *cci_master_info;
cci_master_info = &cci_dev->cci_master_info[MASTER_1];
@@ -189,27 +159,27 @@ irqreturn_t cam_cci_irq(int irq_num, void *data)
&cci_dev->cci_master_info[MASTER_1].lock_q[QUEUE_1],
flags);
}
- if (irq_status0 & CCI_IRQ_STATUS_0_I2C_M0_Q0Q1_HALT_ACK_BMSK) {
+ if (irq & CCI_IRQ_STATUS_0_I2C_M0_Q0Q1_HALT_ACK_BMSK) {
cci_dev->cci_master_info[MASTER_0].reset_pending = TRUE;
cam_io_w_mb(CCI_M0_RESET_RMSK,
base + CCI_RESET_CMD_ADDR);
}
- if (irq_status0 & CCI_IRQ_STATUS_0_I2C_M1_Q0Q1_HALT_ACK_BMSK) {
+ if (irq & CCI_IRQ_STATUS_0_I2C_M1_Q0Q1_HALT_ACK_BMSK) {
cci_dev->cci_master_info[MASTER_1].reset_pending = TRUE;
cam_io_w_mb(CCI_M1_RESET_RMSK,
base + CCI_RESET_CMD_ADDR);
}
- if (irq_status0 & CCI_IRQ_STATUS_0_I2C_M0_ERROR_BMSK) {
+ if (irq & CCI_IRQ_STATUS_0_I2C_M0_ERROR_BMSK) {
cci_dev->cci_master_info[MASTER_0].status = -EINVAL;
cam_io_w_mb(CCI_M0_HALT_REQ_RMSK,
base + CCI_HALT_REQ_ADDR);
- CAM_DBG(CAM_CCI, "MASTER_0 error 0x%x", irq_status0);
+ CAM_DBG(CAM_CCI, "MASTER_0 error 0x%x", irq);
}
- if (irq_status0 & CCI_IRQ_STATUS_0_I2C_M1_ERROR_BMSK) {
+ if (irq & CCI_IRQ_STATUS_0_I2C_M1_ERROR_BMSK) {
cci_dev->cci_master_info[MASTER_1].status = -EINVAL;
cam_io_w_mb(CCI_M1_HALT_REQ_RMSK,
base + CCI_HALT_REQ_ADDR);
- CAM_DBG(CAM_CCI, "MASTER_1 error 0x%x", irq_status0);
+ CAM_DBG(CAM_CCI, "MASTER_1 error 0x%x", irq);
}
return IRQ_HANDLED;
}
@@ -283,6 +253,7 @@ static int cam_cci_platform_probe(struct platform_device *pdev)
soc_info = &new_cci_dev->soc_info;
+ mutex_init(&new_cci_dev->mutex);
new_cci_dev->v4l2_dev_str.pdev = pdev;
soc_info->pdev = pdev;
@@ -343,6 +314,7 @@ static int cam_cci_platform_probe(struct platform_device *pdev)
return rc;
cci_no_resource:
+ mutex_destroy(&new_cci_dev->mutex);
kfree(new_cci_dev);
return rc;
}
@@ -355,6 +327,7 @@ static int cam_cci_device_remove(struct platform_device *pdev)
cam_cpas_unregister_client(cci_dev->cpas_handle);
cam_cci_soc_remove(pdev, cci_dev);
+ mutex_destroy(&cci_dev->mutex);
devm_kfree(&pdev->dev, cci_dev);
return 0;
}
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_dev.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_dev.h
index d48ffd1..e75a9a9 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_dev.h
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_dev.h
@@ -45,7 +45,7 @@
#define CYCLES_PER_MICRO_SEC_DEFAULT 4915
#define CCI_MAX_DELAY 1000000
-#define CCI_TIMEOUT msecs_to_jiffies(1500)
+#define CCI_TIMEOUT msecs_to_jiffies(50)
#define NUM_MASTERS 2
#define NUM_QUEUES 2
@@ -65,14 +65,19 @@
#define MAX_LRME_V4l2_EVENTS 30
/* Max bytes that can be read per CCI read transaction */
-#define CCI_READ_MAX 256
+#define CCI_READ_MAX 12
#define CCI_I2C_READ_MAX_RETRIES 3
#define CCI_I2C_MAX_READ 8192
#define CCI_I2C_MAX_WRITE 8192
-#define CCI_I2C_MAX_BYTE_COUNT 65535
#define CAMX_CCI_DEV_NAME "cam-cci-driver"
+/* Max bytes that can be read per CCI read transaction */
+#define CCI_READ_MAX 12
+#define CCI_I2C_READ_MAX_RETRIES 3
+#define CCI_I2C_MAX_READ 8192
+#define CCI_I2C_MAX_WRITE 8192
+
#define PRIORITY_QUEUE (QUEUE_0)
#define SYNC_QUEUE (QUEUE_1)
@@ -120,7 +125,6 @@ struct cam_cci_read_cfg {
uint16_t addr_type;
uint8_t *data;
uint16_t num_byte;
- uint16_t data_type;
};
struct cam_cci_i2c_queue_info {
@@ -216,6 +220,7 @@ struct cci_device {
uint8_t payload_size;
char device_name[20];
uint32_t cpas_handle;
+ struct mutex mutex;
};
enum cam_cci_i2c_cmd_type {
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_hwreg.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_hwreg.h
index 31c8e26..c18593e 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_hwreg.h
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_hwreg.h
@@ -43,36 +43,27 @@
#define CCI_I2C_M0_Q0_LOAD_DATA_ADDR 0x00000310
#define CCI_IRQ_MASK_0_ADDR 0x00000c04
#define CCI_IRQ_MASK_0_RMSK 0x7fff7ff7
-#define CCI_IRQ_MASK_1_ADDR 0x00000c10
-#define CCI_IRQ_MASK_1_RMSK 0x00110000
#define CCI_IRQ_CLEAR_0_ADDR 0x00000c08
-#define CCI_IRQ_CLEAR_1_ADDR 0x00000c14
#define CCI_IRQ_STATUS_0_ADDR 0x00000c0c
-#define CCI_IRQ_STATUS_1_ADDR 0x00000c18
#define CCI_IRQ_STATUS_0_I2C_M1_Q0Q1_HALT_ACK_BMSK 0x4000000
#define CCI_IRQ_STATUS_0_I2C_M0_Q0Q1_HALT_ACK_BMSK 0x2000000
#define CCI_IRQ_STATUS_0_RST_DONE_ACK_BMSK 0x1000000
#define CCI_IRQ_STATUS_0_I2C_M1_Q1_REPORT_BMSK 0x100000
#define CCI_IRQ_STATUS_0_I2C_M1_Q0_REPORT_BMSK 0x10000
#define CCI_IRQ_STATUS_0_I2C_M1_RD_DONE_BMSK 0x1000
-#define CCI_IRQ_STATUS_1_I2C_M1_RD_THRESHOLD 0x100000
#define CCI_IRQ_STATUS_0_I2C_M0_Q1_REPORT_BMSK 0x100
#define CCI_IRQ_STATUS_0_I2C_M0_Q0_REPORT_BMSK 0x10
#define CCI_IRQ_STATUS_0_I2C_M0_ERROR_BMSK 0x18000EE6
#define CCI_IRQ_STATUS_0_I2C_M1_ERROR_BMSK 0x60EE6000
#define CCI_IRQ_STATUS_0_I2C_M0_RD_DONE_BMSK 0x1
-#define CCI_IRQ_STATUS_1_I2C_M0_RD_THRESHOLD 0x10000
-#define CCI_I2C_M0_RD_THRESHOLD_ADDR 0x00000120
-#define CCI_I2C_M1_RD_THRESHOLD_ADDR 0x00000220
-#define CCI_I2C_RD_THRESHOLD_VALUE 0x38
#define CCI_IRQ_GLOBAL_CLEAR_CMD_ADDR 0x00000c00
#define DEBUG_TOP_REG_START 0x0
#define DEBUG_TOP_REG_COUNT 14
#define DEBUG_MASTER_REG_START 0x100
-#define DEBUG_MASTER_REG_COUNT 9
+#define DEBUG_MASTER_REG_COUNT 8
#define DEBUG_MASTER_QUEUE_REG_START 0x300
-#define DEBUG_MASTER_QUEUE_REG_COUNT 7
+#define DEBUG_MASTER_QUEUE_REG_COUNT 6
#define DEBUG_INTR_REG_START 0xC00
#define DEBUG_INTR_REG_COUNT 7
#endif /* _CAM_CCI_HWREG_ */
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_soc.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_soc.c
index e0b27ca..14737f9 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_soc.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_cci/cam_cci_soc.c
@@ -53,11 +53,11 @@ int cam_cci_init(struct v4l2_subdev *sd,
mutex_lock(&cci_dev->cci_master_info[master].mutex);
flush_workqueue(cci_dev->write_wq[master]);
/* Re-initialize the completion */
- reinit_completion(
- &cci_dev->cci_master_info[master].reset_complete);
+ reinit_completion(&cci_dev->
+ cci_master_info[master].reset_complete);
for (i = 0; i < NUM_QUEUES; i++)
- reinit_completion(
- &cci_dev->cci_master_info[master].report_q[i]);
+ reinit_completion(&cci_dev->
+ cci_master_info[master].report_q[i]);
/* Set reset pending flag to TRUE */
cci_dev->cci_master_info[master].reset_pending = TRUE;
/* Set proper mask to RESET CMD address */
@@ -69,7 +69,8 @@ int cam_cci_init(struct v4l2_subdev *sd,
base + CCI_RESET_CMD_ADDR);
/* wait for reset done irq */
rc = wait_for_completion_timeout(
- &cci_dev->cci_master_info[master].reset_complete,
+ &cci_dev->cci_master_info[master].
+ reset_complete,
CCI_TIMEOUT);
if (rc <= 0)
CAM_ERR(CAM_CCI, "wait failed %d", rc);
@@ -93,8 +94,8 @@ int cam_cci_init(struct v4l2_subdev *sd,
/* Re-initialize the completion */
reinit_completion(&cci_dev->cci_master_info[master].reset_complete);
for (i = 0; i < NUM_QUEUES; i++)
- reinit_completion(
- &cci_dev->cci_master_info[master].report_q[i]);
+ reinit_completion(&cci_dev->cci_master_info[master].
+ report_q[i]);
/* Enable Regulators and IRQ*/
rc = cam_soc_util_enable_platform_resource(soc_info, true,
@@ -115,15 +116,19 @@ int cam_cci_init(struct v4l2_subdev *sd,
for (i = 0; i < NUM_MASTERS; i++) {
for (j = 0; j < NUM_QUEUES; j++) {
if (j == QUEUE_0)
- cci_dev->cci_i2c_queue_info[i][j].max_queue_size
- = CCI_I2C_QUEUE_0_SIZE;
+ cci_dev->cci_i2c_queue_info[i][j].
+ max_queue_size =
+ CCI_I2C_QUEUE_0_SIZE;
else
- cci_dev->cci_i2c_queue_info[i][j].max_queue_size
- = CCI_I2C_QUEUE_1_SIZE;
+ cci_dev->cci_i2c_queue_info[i][j].
+ max_queue_size =
+ CCI_I2C_QUEUE_1_SIZE;
- CAM_DBG(CAM_CCI, "CCI Master[%d] :: Q0 : %d Q1 : %d", i,
- cci_dev->cci_i2c_queue_info[i][j].max_queue_size,
- cci_dev->cci_i2c_queue_info[i][j].max_queue_size);
+ CAM_DBG(CAM_CCI, "CCI Master[%d] :: Q0 : %d Q1 : %d", i
+ , cci_dev->cci_i2c_queue_info[i][j].
+ max_queue_size,
+ cci_dev->cci_i2c_queue_info[i][j].
+ max_queue_size);
}
}
@@ -146,10 +151,6 @@ int cam_cci_init(struct v4l2_subdev *sd,
base + CCI_IRQ_MASK_0_ADDR);
cam_io_w_mb(CCI_IRQ_MASK_0_RMSK,
base + CCI_IRQ_CLEAR_0_ADDR);
- cam_io_w_mb(CCI_IRQ_MASK_1_RMSK,
- base + CCI_IRQ_MASK_1_ADDR);
- cam_io_w_mb(CCI_IRQ_MASK_1_RMSK,
- base + CCI_IRQ_CLEAR_1_ADDR);
cam_io_w_mb(0x1, base + CCI_IRQ_GLOBAL_CLEAR_CMD_ADDR);
for (i = 0; i < MASTER_MAX; i++) {
@@ -161,13 +162,6 @@ int cam_cci_init(struct v4l2_subdev *sd,
flush_workqueue(cci_dev->write_wq[i]);
}
}
-
- /* Set RD FIFO threshold for M0 & M1 */
- cam_io_w_mb(CCI_I2C_RD_THRESHOLD_VALUE,
- base + CCI_I2C_M0_RD_THRESHOLD_ADDR);
- cam_io_w_mb(CCI_I2C_RD_THRESHOLD_VALUE,
- base + CCI_I2C_M1_RD_THRESHOLD_ADDR);
-
cci_dev->cci_state = CCI_STATE_ENABLED;
return 0;
@@ -197,13 +191,13 @@ static void cam_cci_init_cci_params(struct cci_device *new_cci_dev)
for (i = 0; i < NUM_MASTERS; i++) {
new_cci_dev->cci_master_info[i].status = 0;
mutex_init(&new_cci_dev->cci_master_info[i].mutex);
- init_completion(
- &new_cci_dev->cci_master_info[i].reset_complete);
+ init_completion(&new_cci_dev->
+ cci_master_info[i].reset_complete);
for (j = 0; j < NUM_QUEUES; j++) {
mutex_init(&new_cci_dev->cci_master_info[i].mutex_q[j]);
- init_completion(
- &new_cci_dev->cci_master_info[i].report_q[j]);
+ init_completion(&new_cci_dev->
+ cci_master_info[i].report_q[j]);
spin_lock_init(
&new_cci_dev->cci_master_info[i].lock_q[j]);
}
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_csiphy/cam_csiphy_core.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_csiphy/cam_csiphy_core.c
index dbbac08..c6b998ca 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_csiphy/cam_csiphy_core.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_csiphy/cam_csiphy_core.c
@@ -64,14 +64,16 @@ void cam_csiphy_reset(struct csiphy_device *csiphy_dev)
for (i = 0; i < size; i++) {
cam_io_w_mb(
- csiphy_dev->ctrl_reg->csiphy_reset_reg[i].reg_data,
+ csiphy_dev->ctrl_reg->
+ csiphy_reset_reg[i].reg_data,
base +
- csiphy_dev->ctrl_reg->csiphy_reset_reg[i].reg_addr);
+ csiphy_dev->ctrl_reg->
+ csiphy_reset_reg[i].reg_addr);
usleep_range(csiphy_dev->ctrl_reg->
csiphy_reset_reg[i].delay * 1000,
- csiphy_dev->ctrl_reg->csiphy_reset_reg[i].delay * 1000
- + 10);
+ csiphy_dev->ctrl_reg->
+ csiphy_reset_reg[i].delay * 1000 + 10);
}
}
@@ -148,10 +150,11 @@ void cam_csiphy_cphy_irq_config(struct csiphy_device *csiphy_dev)
csiphy_dev->soc_info.reg_map[0].mem_base;
for (i = 0; i < csiphy_dev->num_irq_registers; i++)
- cam_io_w_mb(
- csiphy_dev->ctrl_reg->csiphy_irq_reg[i].reg_data,
+ cam_io_w_mb(csiphy_dev->ctrl_reg->
+ csiphy_irq_reg[i].reg_data,
csiphybase +
- csiphy_dev->ctrl_reg->csiphy_irq_reg[i].reg_addr);
+ csiphy_dev->ctrl_reg->
+ csiphy_irq_reg[i].reg_addr);
}
void cam_csiphy_cphy_irq_disable(struct csiphy_device *csiphy_dev)
@@ -161,8 +164,10 @@ void cam_csiphy_cphy_irq_disable(struct csiphy_device *csiphy_dev)
csiphy_dev->soc_info.reg_map[0].mem_base;
for (i = 0; i < csiphy_dev->num_irq_registers; i++)
- cam_io_w_mb(0x0, csiphybase +
- csiphy_dev->ctrl_reg->csiphy_irq_reg[i].reg_addr);
+ cam_io_w_mb(0x0,
+ csiphybase +
+ csiphy_dev->ctrl_reg->
+ csiphy_irq_reg[i].reg_addr);
}
irqreturn_t cam_csiphy_irq(int irq_num, void *data)
@@ -172,7 +177,6 @@ irqreturn_t cam_csiphy_irq(int irq_num, void *data)
struct csiphy_device *csiphy_dev =
(struct csiphy_device *)data;
struct cam_hw_soc_info *soc_info = NULL;
- struct csiphy_reg_parms_t *csiphy_reg = NULL;
void __iomem *base = NULL;
if (!csiphy_dev) {
@@ -182,24 +186,30 @@ irqreturn_t cam_csiphy_irq(int irq_num, void *data)
soc_info = &csiphy_dev->soc_info;
base = csiphy_dev->soc_info.reg_map[0].mem_base;
- csiphy_reg = &csiphy_dev->ctrl_reg->csiphy_reg;
for (i = 0; i < csiphy_dev->num_irq_registers; i++) {
- irq = cam_io_r(base +
- csiphy_reg->mipi_csiphy_interrupt_status0_addr +
- (0x4 * i));
- cam_io_w_mb(irq, base +
- csiphy_reg->mipi_csiphy_interrupt_clear0_addr +
- (0x4 * i));
+ irq = cam_io_r(
+ base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_interrupt_status0_addr + 0x4*i);
+ cam_io_w_mb(irq,
+ base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_interrupt_clear0_addr + 0x4*i);
CAM_ERR_RATE_LIMIT(CAM_CSIPHY,
"CSIPHY%d_IRQ_STATUS_ADDR%d = 0x%x",
soc_info->index, i, irq);
- cam_io_w_mb(0x0, base +
- csiphy_reg->mipi_csiphy_interrupt_clear0_addr +
- (0x4 * i));
+ cam_io_w_mb(0x0,
+ base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_interrupt_clear0_addr + 0x4*i);
}
- cam_io_w_mb(0x1, base + csiphy_reg->mipi_csiphy_glbl_irq_cmd_addr);
- cam_io_w_mb(0x0, base + csiphy_reg->mipi_csiphy_glbl_irq_cmd_addr);
+ cam_io_w_mb(0x1, base +
+ csiphy_dev->ctrl_reg->
+ csiphy_reg.mipi_csiphy_glbl_irq_cmd_addr);
+ cam_io_w_mb(0x0, base +
+ csiphy_dev->ctrl_reg->
+ csiphy_reg.mipi_csiphy_glbl_irq_cmd_addr);
return IRQ_HANDLED;
}
@@ -212,7 +222,6 @@ int32_t cam_csiphy_config_dev(struct csiphy_device *csiphy_dev)
uint8_t lane_cnt, lane_pos = 0;
uint16_t settle_cnt = 0;
void __iomem *csiphybase;
- struct csiphy_reg_t *csiphy_common_reg = NULL;
struct csiphy_reg_t (*reg_array)[MAX_SETTINGS_PER_LANE];
lane_cnt = csiphy_dev->csiphy_info.lane_cnt;
@@ -231,8 +240,8 @@ int32_t cam_csiphy_config_dev(struct csiphy_device *csiphy_dev)
reg_array =
csiphy_dev->ctrl_reg->csiphy_2ph_reg;
csiphy_dev->num_irq_registers = 11;
- cfg_size =
- csiphy_dev->ctrl_reg->csiphy_reg.csiphy_2ph_config_array_size;
+ cfg_size = csiphy_dev->ctrl_reg->csiphy_reg.
+ csiphy_2ph_config_array_size;
lane_mask = csiphy_dev->csiphy_info.lane_mask & 0x1f;
for (i = 0; i < MAX_DPHY_DATA_LN; i++) {
@@ -253,8 +262,8 @@ int32_t cam_csiphy_config_dev(struct csiphy_device *csiphy_dev)
reg_array =
csiphy_dev->ctrl_reg->csiphy_3ph_reg;
csiphy_dev->num_irq_registers = 11;
- cfg_size =
- csiphy_dev->ctrl_reg->csiphy_reg.csiphy_3ph_config_array_size;
+ cfg_size = csiphy_dev->ctrl_reg->csiphy_reg.
+ csiphy_3ph_config_array_size;
lane_mask = csiphy_dev->csiphy_info.lane_mask & 0x7;
mask = lane_mask;
@@ -267,24 +276,32 @@ int32_t cam_csiphy_config_dev(struct csiphy_device *csiphy_dev)
}
size = csiphy_dev->ctrl_reg->csiphy_reg.csiphy_common_array_size;
+
for (i = 0; i < size; i++) {
- csiphy_common_reg = &csiphy_dev->ctrl_reg->csiphy_common_reg[i];
- switch (csiphy_common_reg->csiphy_param_type) {
- case CSIPHY_LANE_ENABLE:
+ switch (csiphy_dev->ctrl_reg->
+ csiphy_common_reg[i].csiphy_param_type) {
+ case CSIPHY_LANE_ENABLE:
cam_io_w_mb(lane_enable,
csiphybase +
- csiphy_common_reg->reg_addr);
- usleep_range(csiphy_common_reg->delay * 1000,
- csiphy_common_reg->delay * 1000 + 10);
+ csiphy_dev->ctrl_reg->
+ csiphy_common_reg[i].reg_addr);
+ usleep_range(csiphy_dev->ctrl_reg->
+ csiphy_common_reg[i].delay*1000,
+ csiphy_dev->ctrl_reg->
+ csiphy_common_reg[i].delay*1000 + 10);
break;
- case CSIPHY_DEFAULT_PARAMS:
- cam_io_w_mb(csiphy_common_reg->reg_data,
+ case CSIPHY_DEFAULT_PARAMS:
+ cam_io_w_mb(csiphy_dev->ctrl_reg->
+ csiphy_common_reg[i].reg_data,
csiphybase +
- csiphy_common_reg->reg_addr);
- usleep_range(csiphy_common_reg->delay * 1000,
- csiphy_common_reg->delay * 1000 + 10);
+ csiphy_dev->ctrl_reg->
+ csiphy_common_reg[i].reg_addr);
+ usleep_range(csiphy_dev->ctrl_reg->
+ csiphy_common_reg[i].delay*1000,
+ csiphy_dev->ctrl_reg->
+ csiphy_common_reg[i].delay*1000 + 10);
break;
- default:
+ default:
break;
}
}
@@ -300,8 +317,8 @@ int32_t cam_csiphy_config_dev(struct csiphy_device *csiphy_dev)
if (csiphy_dev->csiphy_info.combo_mode == 1 &&
(lane_pos >= 3))
settle_cnt =
- (csiphy_dev->csiphy_info.settle_time_combo_sensor /
- 200000000);
+ (csiphy_dev->csiphy_info.
+ settle_time_combo_sensor / 200000000);
for (i = 0; i < cfg_size; i++) {
switch (reg_array[lane_pos][i].csiphy_param_type) {
case CSIPHY_LANE_ENABLE:
@@ -429,7 +446,6 @@ int32_t cam_csiphy_core_cfg(void *phy_dev,
{
struct csiphy_device *csiphy_dev =
(struct csiphy_device *)phy_dev;
- struct intf_params *bridge_intf = NULL;
struct cam_control *cmd = (struct cam_control *)arg;
int32_t rc = 0;
@@ -438,12 +454,6 @@ int32_t cam_csiphy_core_cfg(void *phy_dev,
return -EINVAL;
}
- if (cmd->handle_type != CAM_HANDLE_USER_POINTER) {
- CAM_ERR(CAM_CSIPHY, "Invalid handle type: %d",
- cmd->handle_type);
- return -EINVAL;
- }
-
CAM_DBG(CAM_CSIPHY, "Opcode received: %d", cmd->op_code);
mutex_lock(&csiphy_dev->mutex);
switch (cmd->op_code) {
@@ -514,10 +524,11 @@ int32_t cam_csiphy_core_cfg(void *phy_dev,
csiphy_acq_dev.device_handle =
cam_create_device_hdl(&bridge_params);
- bridge_intf = &csiphy_dev->bridge_intf;
- bridge_intf->device_hdl[csiphy_acq_params.combo_mode]
- = csiphy_acq_dev.device_handle;
- bridge_intf->session_hdl[csiphy_acq_params.combo_mode] =
+ csiphy_dev->bridge_intf.
+ device_hdl[csiphy_acq_params.combo_mode] =
+ csiphy_acq_dev.device_handle;
+ csiphy_dev->bridge_intf.
+ session_hdl[csiphy_acq_params.combo_mode] =
csiphy_acq_dev.session_handle;
if (copy_to_user((void __user *)cmd->handle,
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_csiphy/cam_csiphy_dev.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_csiphy/cam_csiphy_dev.c
index e2f061f..18ba369 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_csiphy/cam_csiphy_dev.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_csiphy/cam_csiphy_dev.c
@@ -236,7 +236,7 @@ static struct platform_driver csiphy_driver = {
},
};
-static int32_t __init cam_csiphy_init_module(void)
+static int __init cam_csiphy_init_module(void)
{
return platform_driver_register(&csiphy_driver);
}
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_core.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_core.c
index 5376484..b7053f2 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_core.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_core.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -62,6 +62,7 @@ static int cam_eeprom_read_memory(struct cam_eeprom_ctrl_t *e_ctrl,
i2c_reg_settings.addr_type = emap[j].page.addr_type;
i2c_reg_settings.data_type = emap[j].page.data_type;
i2c_reg_settings.size = 1;
+ i2c_reg_settings.delay = 0;
i2c_reg_array.reg_addr = emap[j].page.addr;
i2c_reg_array.reg_data = emap[j].page.data;
i2c_reg_array.delay = emap[j].page.delay;
@@ -79,6 +80,7 @@ static int cam_eeprom_read_memory(struct cam_eeprom_ctrl_t *e_ctrl,
i2c_reg_settings.addr_type = emap[j].pageen.addr_type;
i2c_reg_settings.data_type = emap[j].pageen.data_type;
i2c_reg_settings.size = 1;
+ i2c_reg_settings.delay = 0;
i2c_reg_array.reg_addr = emap[j].pageen.addr;
i2c_reg_array.reg_data = emap[j].pageen.data;
i2c_reg_array.delay = emap[j].pageen.delay;
@@ -109,7 +111,6 @@ static int cam_eeprom_read_memory(struct cam_eeprom_ctrl_t *e_ctrl,
rc = camera_io_dev_read_seq(&e_ctrl->io_master_info,
emap[j].mem.addr, memptr,
emap[j].mem.addr_type,
- emap[j].mem.data_type,
emap[j].mem.valid_size);
if (rc) {
CAM_ERR(CAM_EEPROM, "read failed rc %d",
@@ -315,8 +316,8 @@ int32_t cam_eeprom_parse_read_memory_map(struct device_node *of_node,
power_down:
cam_eeprom_power_down(e_ctrl);
data_mem_free:
- kfree(e_ctrl->cal_data.mapdata);
- kfree(e_ctrl->cal_data.map);
+ vfree(e_ctrl->cal_data.mapdata);
+ vfree(e_ctrl->cal_data.map);
e_ctrl->cal_data.num_data = 0;
e_ctrl->cal_data.num_map = 0;
e_ctrl->cam_eeprom_state = CAM_EEPROM_ACQUIRE;
@@ -444,6 +445,7 @@ static int32_t cam_eeprom_parse_memory_map(
map[*num_map + cnt].page.data_type =
i2c_random_wr->header.data_type;
map[*num_map + cnt].page.valid_size = 1;
+ map[*num_map + cnt].page.delay = 0;
}
*num_map += (i2c_random_wr->header.count - 1);
@@ -460,6 +462,7 @@ static int32_t cam_eeprom_parse_memory_map(
map[*num_map].mem.data_type = i2c_cont_rd->header.data_type;
map[*num_map].mem.valid_size =
i2c_cont_rd->header.count;
+ map[*num_map].mem.delay = 0;
cmd_buf += cmd_length_in_bytes / sizeof(int32_t);
processed_size +=
cmd_length_in_bytes;
@@ -543,9 +546,9 @@ static int32_t cam_eeprom_init_pkt_parser(struct cam_eeprom_ctrl_t *e_ctrl,
(struct cam_eeprom_soc_private *)e_ctrl->soc_info.soc_private;
struct cam_sensor_power_ctrl_t *power_info = &soc_private->power_info;
- e_ctrl->cal_data.map = kcalloc((MSM_EEPROM_MEMORY_MAP_MAX_SIZE *
- MSM_EEPROM_MAX_MEM_MAP_CNT),
- (sizeof(struct cam_eeprom_memory_map_t)), GFP_KERNEL);
+ e_ctrl->cal_data.map = vzalloc((MSM_EEPROM_MEMORY_MAP_MAX_SIZE *
+ MSM_EEPROM_MAX_MEM_MAP_CNT) *
+ (sizeof(struct cam_eeprom_memory_map_t)));
if (!e_ctrl->cal_data.map) {
rc = -ENOMEM;
CAM_ERR(CAM_EEPROM, "failed");
@@ -708,6 +711,11 @@ static int32_t cam_eeprom_pkt_parse(struct cam_eeprom_ctrl_t *e_ctrl, void *arg)
ioctl_ctrl = (struct cam_control *)arg;
+ if (ioctl_ctrl->handle_type != CAM_HANDLE_USER_POINTER) {
+ CAM_ERR(CAM_EEPROM, "Invalid Handle Type");
+ return -EINVAL;
+ }
+
if (copy_from_user(&dev_config, (void __user *) ioctl_ctrl->handle,
sizeof(dev_config)))
return -EFAULT;
@@ -738,8 +746,8 @@ static int32_t cam_eeprom_pkt_parse(struct cam_eeprom_ctrl_t *e_ctrl, void *arg)
return rc;
}
rc = cam_eeprom_get_cal_data(e_ctrl, csl_packet);
- kfree(e_ctrl->cal_data.mapdata);
- kfree(e_ctrl->cal_data.map);
+ vfree(e_ctrl->cal_data.mapdata);
+ vfree(e_ctrl->cal_data.map);
e_ctrl->cal_data.num_data = 0;
e_ctrl->cal_data.num_map = 0;
CAM_DBG(CAM_EEPROM,
@@ -754,7 +762,7 @@ static int32_t cam_eeprom_pkt_parse(struct cam_eeprom_ctrl_t *e_ctrl, void *arg)
}
e_ctrl->cal_data.mapdata =
- kzalloc(e_ctrl->cal_data.num_data, GFP_KERNEL);
+ vzalloc(e_ctrl->cal_data.num_data);
if (!e_ctrl->cal_data.mapdata) {
rc = -ENOMEM;
CAM_ERR(CAM_EEPROM, "failed");
@@ -779,14 +787,8 @@ static int32_t cam_eeprom_pkt_parse(struct cam_eeprom_ctrl_t *e_ctrl, void *arg)
rc = cam_eeprom_get_cal_data(e_ctrl, csl_packet);
rc = cam_eeprom_power_down(e_ctrl);
e_ctrl->cam_eeprom_state = CAM_EEPROM_ACQUIRE;
- kfree(e_ctrl->cal_data.mapdata);
- kfree(e_ctrl->cal_data.map);
- kfree(power_info->power_setting);
- kfree(power_info->power_down_setting);
- power_info->power_setting = NULL;
- power_info->power_down_setting = NULL;
- power_info->power_setting_size = 0;
- power_info->power_down_setting_size = 0;
+ vfree(e_ctrl->cal_data.mapdata);
+ vfree(e_ctrl->cal_data.map);
e_ctrl->cal_data.num_data = 0;
e_ctrl->cal_data.num_map = 0;
break;
@@ -797,13 +799,11 @@ static int32_t cam_eeprom_pkt_parse(struct cam_eeprom_ctrl_t *e_ctrl, void *arg)
power_down:
cam_eeprom_power_down(e_ctrl);
memdata_free:
- kfree(e_ctrl->cal_data.mapdata);
+ vfree(e_ctrl->cal_data.mapdata);
error:
kfree(power_info->power_setting);
kfree(power_info->power_down_setting);
- power_info->power_setting = NULL;
- power_info->power_down_setting = NULL;
- kfree(e_ctrl->cal_data.map);
+ vfree(e_ctrl->cal_data.map);
e_ctrl->cal_data.num_data = 0;
e_ctrl->cal_data.num_map = 0;
e_ctrl->cam_eeprom_state = CAM_EEPROM_INIT;
@@ -838,10 +838,6 @@ void cam_eeprom_shutdown(struct cam_eeprom_ctrl_t *e_ctrl)
kfree(power_info->power_setting);
kfree(power_info->power_down_setting);
- power_info->power_setting = NULL;
- power_info->power_down_setting = NULL;
- power_info->power_setting_size = 0;
- power_info->power_down_setting_size = 0;
}
e_ctrl->cam_eeprom_state = CAM_EEPROM_INIT;
@@ -860,14 +856,8 @@ int32_t cam_eeprom_driver_cmd(struct cam_eeprom_ctrl_t *e_ctrl, void *arg)
struct cam_eeprom_query_cap_t eeprom_cap = {0};
struct cam_control *cmd = (struct cam_control *)arg;
- if (!e_ctrl || !cmd) {
- CAM_ERR(CAM_EEPROM, "Invalid Arguments");
- return -EINVAL;
- }
-
- if (cmd->handle_type != CAM_HANDLE_USER_POINTER) {
- CAM_ERR(CAM_EEPROM, "Invalid handle type: %d",
- cmd->handle_type);
+ if (!e_ctrl) {
+ CAM_ERR(CAM_EEPROM, "e_ctrl is NULL");
return -EINVAL;
}
@@ -884,7 +874,7 @@ int32_t cam_eeprom_driver_cmd(struct cam_eeprom_ctrl_t *e_ctrl, void *arg)
&eeprom_cap,
sizeof(struct cam_eeprom_query_cap_t))) {
CAM_ERR(CAM_EEPROM, "Failed Copy to User");
- return -EFAULT;
+ rc = -EFAULT;
goto release_mutex;
}
CAM_DBG(CAM_EEPROM, "eeprom_cap: ID: %d", eeprom_cap.slot_info);
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_dev.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_dev.c
index 519c815..68c5eea 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_dev.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_dev.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -69,9 +69,6 @@ int32_t cam_eeprom_update_i2c_info(struct cam_eeprom_ctrl_t *e_ctrl,
cci_client->retries = 3;
cci_client->id_map = 0;
cci_client->i2c_freq_mode = i2c_info->i2c_freq_mode;
- } else if (e_ctrl->io_master_info.master_type == I2C_MASTER) {
- e_ctrl->io_master_info.client->addr = i2c_info->slave_addr;
- CAM_DBG(CAM_EEPROM, "Slave addr: 0x%x", i2c_info->slave_addr);
}
return 0;
}
@@ -191,9 +188,6 @@ static int cam_eeprom_i2c_driver_probe(struct i2c_client *client,
e_ctrl->io_master_info.master_type = I2C_MASTER;
e_ctrl->io_master_info.client = client;
e_ctrl->eeprom_device_type = MSM_CAMERA_I2C_DEVICE;
- e_ctrl->cal_data.mapdata = NULL;
- e_ctrl->cal_data.map = NULL;
- e_ctrl->userspace_probe = false;
rc = cam_eeprom_parse_dt(e_ctrl);
if (rc) {
@@ -211,6 +205,10 @@ static int cam_eeprom_i2c_driver_probe(struct i2c_client *client,
if (rc)
goto free_soc;
+ e_ctrl->cal_data.mapdata = NULL;
+ e_ctrl->cal_data.map = NULL;
+ e_ctrl->userspace_probe = false;
+
if (soc_private->i2c_info.slave_addr != 0)
e_ctrl->io_master_info.client->addr =
soc_private->i2c_info.slave_addr;
@@ -261,10 +259,9 @@ static int cam_eeprom_i2c_driver_remove(struct i2c_client *client)
for (i = 0; i < soc_info->num_clk; i++)
devm_clk_put(soc_info->dev, soc_info->clk[i]);
- mutex_destroy(&(e_ctrl->eeprom_mutex));
- kfree(soc_private);
- kfree(e_ctrl->io_master_info.cci_client);
- v4l2_set_subdevdata(&e_ctrl->v4l2_dev_str.sd, NULL);
+ if (soc_private)
+ kfree(soc_private);
+
kfree(e_ctrl);
return 0;
@@ -395,8 +392,6 @@ static int cam_eeprom_spi_driver_remove(struct spi_device *sdev)
kfree(soc_private->power_info.gpio_num_info);
kfree(soc_private);
}
- mutex_destroy(&(e_ctrl->eeprom_mutex));
- v4l2_set_subdevdata(&e_ctrl->v4l2_dev_str.sd, NULL);
kfree(e_ctrl);
return 0;
@@ -492,11 +487,8 @@ static int cam_eeprom_platform_driver_remove(struct platform_device *pdev)
for (i = 0; i < soc_info->num_clk; i++)
devm_clk_put(soc_info->dev, soc_info->clk[i]);
- mutex_destroy(&(e_ctrl->eeprom_mutex));
kfree(soc_info->soc_private);
kfree(e_ctrl->io_master_info.cci_client);
- platform_set_drvdata(pdev, NULL);
- v4l2_set_subdevdata(&e_ctrl->v4l2_dev_str.sd, NULL);
kfree(e_ctrl);
return 0;
}
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_dev.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_dev.h
index 4a2190d..7a0a82e 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_dev.h
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_eeprom/cam_eeprom_dev.h
@@ -34,7 +34,7 @@
#define PROPERTY_MAXSIZE 32
#define MSM_EEPROM_MEMORY_MAP_MAX_SIZE 80
-#define MSM_EEPROM_MAX_MEM_MAP_CNT 8
+#define MSM_EEPROM_MAX_MEM_MAP_CNT 32
#define MSM_EEPROM_MEM_MAP_PROPERTIES_CNT 8
enum cam_eeprom_state {
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/Makefile b/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/Makefile
index c7889a5..7d7582e 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/Makefile
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/Makefile
@@ -8,3 +8,7 @@
ccflags-y += -Idrivers/media/platform/msm/camera/cam_smmu/
obj-$(CONFIG_SPECTRA_CAMERA) += cam_flash_dev.o cam_flash_core.o cam_flash_soc.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-enum-conversion \
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/cam_flash_core.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/cam_flash_core.c
index 9b74826..e7110b8 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/cam_flash_core.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/cam_flash_core.c
@@ -284,8 +284,8 @@ static int delete_req(struct cam_flash_ctrl *fctrl, uint64_t req_id)
for (i = 0; i < flash_data->cmn_attr.count; i++)
flash_data->led_current_ma[i] = 0;
} else {
- fctrl->flash_init_setting.cmn_attr.is_settings_valid
- = false;
+ fctrl->flash_init_setting.cmn_attr.
+ is_settings_valid = false;
}
} else {
for (i = 0; i < MAX_PER_FRAME_ARRAY; i++) {
@@ -513,7 +513,7 @@ int cam_flash_parser(struct cam_flash_ctrl *fctrl, void *arg)
uint64_t generic_ptr;
uint32_t *cmd_buf = NULL;
uint32_t *offset = NULL;
- uint32_t frm_offset = 0;
+ uint32_t frame_offset = 0;
size_t len_of_buffer;
struct cam_control *ioctl_ctrl = NULL;
struct cam_packet *csl_packet = NULL;
@@ -525,7 +525,6 @@ int cam_flash_parser(struct cam_flash_ctrl *fctrl, void *arg)
struct cam_flash_set_rer *flash_rer_info = NULL;
struct cam_flash_set_on_off *flash_operation_info = NULL;
struct cam_flash_query_curr *flash_query_info = NULL;
- struct cam_flash_frame_setting *flash_data = NULL;
if (!fctrl || !arg) {
CAM_ERR(CAM_FLASH, "fctrl/arg is NULL");
@@ -617,19 +616,25 @@ int cam_flash_parser(struct cam_flash_ctrl *fctrl, void *arg)
case CAM_FLASH_PACKET_OPCODE_SET_OPS: {
offset = (uint32_t *)((uint8_t *)&csl_packet->payload +
csl_packet->cmd_buf_offset);
- frm_offset = csl_packet->header.request_id %
+ frame_offset = csl_packet->header.request_id %
MAX_PER_FRAME_ARRAY;
- flash_data = &fctrl->per_frame[frm_offset];
-
- if (flash_data->cmn_attr.is_settings_valid == true) {
- flash_data->cmn_attr.request_id = 0;
- flash_data->cmn_attr.is_settings_valid = false;
- for (i = 0; i < flash_data->cmn_attr.count; i++)
- flash_data->led_current_ma[i] = 0;
+ if (fctrl->per_frame[frame_offset].cmn_attr.is_settings_valid
+ == true) {
+ fctrl->per_frame[frame_offset].cmn_attr.request_id = 0;
+ fctrl->per_frame[frame_offset].
+ cmn_attr.is_settings_valid = false;
+ for (i = 0;
+ i < fctrl->per_frame[frame_offset].cmn_attr.count;
+ i++) {
+ fctrl->per_frame[frame_offset].
+ led_current_ma[i] = 0;
+ }
}
- flash_data->cmn_attr.request_id = csl_packet->header.request_id;
- flash_data->cmn_attr.is_settings_valid = true;
+ fctrl->per_frame[frame_offset].cmn_attr.request_id =
+ csl_packet->header.request_id;
+ fctrl->per_frame[frame_offset].cmn_attr.is_settings_valid =
+ true;
cmd_desc = (struct cam_cmd_buf_desc *)(offset);
rc = cam_mem_get_cpu_buf(cmd_desc->mem_handle,
(uint64_t *)&generic_ptr, &len_of_buffer);
@@ -650,7 +655,8 @@ int cam_flash_parser(struct cam_flash_ctrl *fctrl, void *arg)
CAM_FLASH_STATE_ACQUIRE)) {
CAM_WARN(CAM_FLASH,
"Rxed Flash fire ops without linking");
- flash_data->cmn_attr.is_settings_valid = false;
+ fctrl->per_frame[frame_offset].
+ cmn_attr.is_settings_valid = false;
return 0;
}
@@ -662,12 +668,16 @@ int cam_flash_parser(struct cam_flash_ctrl *fctrl, void *arg)
return -EINVAL;
}
- flash_data->opcode = flash_operation_info->opcode;
- flash_data->cmn_attr.count =
+ fctrl->per_frame[frame_offset].opcode =
+ flash_operation_info->opcode;
+ fctrl->per_frame[frame_offset].cmn_attr.count =
flash_operation_info->count;
- for (i = 0; i < flash_operation_info->count; i++)
- flash_data->led_current_ma[i]
- = flash_operation_info->led_current_ma[i];
+ for (i = 0;
+ i < flash_operation_info->count; i++)
+ fctrl->per_frame[frame_offset].
+ led_current_ma[i]
+ = flash_operation_info->
+ led_current_ma[i];
}
break;
default:
@@ -772,10 +782,8 @@ int cam_flash_parser(struct cam_flash_ctrl *fctrl, void *arg)
(fctrl->flash_state == CAM_FLASH_STATE_ACQUIRE)) {
CAM_WARN(CAM_FLASH,
"Rxed NOP packets without linking");
- frm_offset = csl_packet->header.request_id %
- MAX_PER_FRAME_ARRAY;
- fctrl->per_frame[frm_offset].cmn_attr.is_settings_valid
- = false;
+ fctrl->per_frame[frame_offset].
+ cmn_attr.is_settings_valid = false;
return 0;
}
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/cam_flash_dev.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/cam_flash_dev.c
index f9411fc..6120e02 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/cam_flash_dev.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_flash/cam_flash_dev.c
@@ -29,12 +29,6 @@ static int32_t cam_flash_driver_cmd(struct cam_flash_ctrl *fctrl,
return -EINVAL;
}
- if (cmd->handle_type != CAM_HANDLE_USER_POINTER) {
- CAM_ERR(CAM_FLASH, "Invalid handle type: %d",
- cmd->handle_type);
- return -EINVAL;
- }
-
mutex_lock(&(fctrl->flash_mutex));
switch (cmd->op_code) {
case CAM_ACQUIRE_DEV: {
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_04_17_00.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_04_17_00.h
new file mode 100644
index 0000000..a6ab089
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_04_17_00.h
@@ -0,0 +1,27495 @@
+/**
+ * @brief LC898123F40 OIS PRODUCT
+ *
+ * @author (C) 2017 ON Semiconductor.
+ *
+ **/
+
+// Time Stamp : 2018/02/21 13:35:34
+
+#define FromCodeBlockSize_04_17_00 11
+
+// [04170008] [00000001] [00000000]
+
+//#define MODULE_VENDOR 0x04
+//#define END_USER 0x17
+//#define USE_ACT 0x00
+
+const unsigned char CcMagicCodeF40_04_17_00[] = {
+0x00,
+0x01,
+0x02,
+0x03,
+0x04,
+0x05,
+0x06,
+0x07,
+0x08,
+0x09,
+0x00,
+0x00,
+0x00,
+0x00,
+0x54,
+0x00,
+0x00,
+0x00,
+0x13,
+0x00,
+0x00,
+0x00,
+0x00,
+0x06,
+0x98,
+0x00,
+0x00,
+0x00,
+0x01,
+0xfa,
+0x00,
+0x00,
+0x00,
+0x13,
+0x54,
+0x00,
+0x00,
+0x80,
+0x00,
+0x90,
+0x00,
+0x00,
+0x00,
+0xbb,
+0x67,
+0x00,
+0x8a,
+0x41,
+0x45,
+0xee,
+0x00,
+0x00,
+0x00,
+0xba,
+0x82,
+0x00,
+0xa9,
+0xce,
+0x6c,
+0xe2,
+0x00,
+0x00,
+0x00,
+0xc0,
+0x36,
+0x00,
+0x99,
+0xbd,
+0xd9,
+0x4c,
+0x00,
+0x00,
+0x00,
+0xb8,
+0xca,
+0x00,
+0xf7,
+0x7a,
+0xc1,
+0x7f,
+0x00,
+0x00,
+0x00,
+0xd0,
+0x7c,
+0x00,
+0x12,
+0x61,
+0x71,
+0x6f,
+0x00,
+0x00,
+0x00,
+0xbd,
+0xbb,
+0x00,
+0xf8,
+0x40,
+0xa3,
+0xf9,
+0x00,
+0x00,
+0x00,
+0xc2,
+0x21,
+0x00,
+0xec,
+0x9c,
+0xaa,
+0xba,
+0x00,
+0x00,
+0x00,
+0xcb,
+0xd0,
+0x00,
+0x90,
+0x1c,
+0x81,
+0x9e,
+0x00,
+0x00,
+0x00,
+0xa8,
+0xd8,
+0x00,
+0x61,
+0xe9,
+0xc7,
+0x7a,
+0x00,
+0x00,
+0x00,
+0x60,
+0x8e,
+0x00,
+0xef,
+0x1b,
+0x2d,
+0x67,
+0x00,
+0x00,
+0x00,
+0xb2,
+0x24,
+0x00,
+0x40,
+0x74,
+0x8a,
+0xac,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+};
+
+const unsigned char CcFromCodeF40_04_17_00[] = {
+0x64,
+0x00,
+0x00,
+0x02,
+0x87,
+0x64,
+0x00,
+0x00,
+0xfe,
+0xe7,
+0x64,
+0x00,
+0x01,
+0x02,
+0x67,
+0x64,
+0x00,
+0x01,
+0x08,
+0x87,
+0x64,
+0x00,
+0x01,
+0x08,
+0xa7,
+0x46,
+0x0b,
+0x03,
+0x80,
+0x00,
+0x64,
+0x00,
+0x01,
+0x08,
+0xc7,
+0x64,
+0x00,
+0x01,
+0x08,
+0xe7,
+0x64,
+0x00,
+0x01,
+0x09,
+0x07,
+0x64,
+0x00,
+0x01,
+0x09,
+0x27,
+0x64,
+0x00,
+0x01,
+0x09,
+0x47,
+0x64,
+0x00,
+0x01,
+0x09,
+0x67,
+0x64,
+0x00,
+0x01,
+0x09,
+0x87,
+0x64,
+0x00,
+0x01,
+0x09,
+0xa7,
+0x64,
+0x00,
+0x01,
+0x09,
+0xc7,
+0x46,
+0x0b,
+0x03,
+0x80,
+0x00,
+0x00,
+0x04,
+0x17,
+0x00,
+0x08,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xba,
+0x11,
+0x2b,
+0xa1,
+0x13,
+0x68,
+0x00,
+0x03,
+0x00,
+0x38,
+0x68,
+0x34,
+0x00,
+0x09,
+0x20,
+0x5c,
+0x81,
+0x09,
+0x8e,
+0xb9,
+0x5c,
+0x03,
+0xc3,
+0x0c,
+0x02,
+0x5c,
+0x96,
+0x00,
+0x40,
+0x52,
+0xa0,
+0x02,
+0x06,
+0x80,
+0x01,
+0x9e,
+0x7c,
+0x28,
+0x00,
+0xd2,
+0x84,
+0x05,
+0x0a,
+0x05,
+0x60,
+0x68,
+0x0f,
+0xc4,
+0xdb,
+0x00,
+0x84,
+0x05,
+0x0a,
+0x03,
+0x60,
+0x68,
+0x02,
+0x44,
+0xdf,
+0xc0,
+0xab,
+0xfc,
+0x08,
+0x02,
+0x50,
+0x40,
+0x00,
+0x00,
+0x80,
+0x60,
+0x66,
+0x00,
+0x01,
+0x10,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x0f,
+0xe0,
+0x5c,
+0x00,
+0x40,
+0x80,
+0x20,
+0x68,
+0x38,
+0x1c,
+0x09,
+0x24,
+0x5c,
+0x04,
+0x00,
+0x40,
+0x50,
+0xa2,
+0x26,
+0x08,
+0x60,
+0x7a,
+0x40,
+0x00,
+0x00,
+0x40,
+0x50,
+0x66,
+0x00,
+0x00,
+0xf8,
+0x40,
+0x68,
+0x20,
+0x00,
+0x00,
+0x24,
+0x68,
+0x00,
+0x00,
+0x08,
+0x20,
+0x98,
+0x90,
+0x96,
+0x60,
+0x00,
+0x12,
+0xfc,
+0x85,
+0xc0,
+0x13,
+0x18,
+0x80,
+0x89,
+0x8e,
+0x80,
+0x6e,
+0x00,
+0x03,
+0xb4,
+0x70,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb7,
+0x20,
+0x66,
+0x00,
+0x01,
+0x6f,
+0xe0,
+0x66,
+0x00,
+0x01,
+0x70,
+0xc0,
+0x68,
+0x00,
+0x01,
+0x0f,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb7,
+0x20,
+0x68,
+0x00,
+0x00,
+0x00,
+0xb9,
+0xba,
+0x11,
+0x00,
+0x00,
+0x00,
+0x66,
+0x00,
+0x01,
+0x1b,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x19,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x1c,
+0xc0,
+0x66,
+0x00,
+0x01,
+0x60,
+0x40,
+0x66,
+0x00,
+0x00,
+0xfd,
+0xa0,
+0x68,
+0x20,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x00,
+0x45,
+0xc0,
+0x66,
+0x00,
+0x00,
+0x1c,
+0xe0,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x66,
+0x00,
+0x01,
+0x52,
+0x40,
+0x6e,
+0x40,
+0x00,
+0x08,
+0x38,
+0x68,
+0x00,
+0x20,
+0x00,
+0x08,
+0x54,
+0x48,
+0x03,
+0x00,
+0x19,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x02,
+0x59,
+0x00,
+0x03,
+0x01,
+0x03,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x51,
+0x52,
+0x06,
+0x93,
+0xc0,
+0x99,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x52,
+0x38,
+0x1d,
+0x26,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x02,
+0x40,
+0x80,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x00,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x40,
+0x38,
+0x1c,
+0x22,
+0x40,
+0x80,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x50,
+0x66,
+0x00,
+0x01,
+0x20,
+0x80,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x7a,
+0x38,
+0x01,
+0xc6,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x96,
+0x60,
+0x04,
+0x03,
+0x30,
+0x86,
+0x80,
+0x00,
+0x00,
+0x0f,
+0x98,
+0x80,
+0x7a,
+0x66,
+0x00,
+0x40,
+0x34,
+0x60,
+0x32,
+0x82,
+0x0b,
+0xc0,
+0x98,
+0x88,
+0x00,
+0x06,
+0xc7,
+0x00,
+0x80,
+0x20,
+0x25,
+0x50,
+0x20,
+0x30,
+0x12,
+0x15,
+0x20,
+0x29,
+0x08,
+0x05,
+0x06,
+0xc7,
+0x00,
+0x80,
+0x25,
+0x23,
+0x21,
+0x40,
+0x40,
+0x00,
+0x03,
+0xff,
+0x02,
+0x66,
+0x00,
+0x40,
+0x34,
+0x20,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x68,
+0x38,
+0x08,
+0x10,
+0x24,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x5c,
+0x96,
+0x12,
+0x20,
+0x24,
+0x5c,
+0x0e,
+0x90,
+0x80,
+0x64,
+0x5c,
+0x09,
+0xc2,
+0x20,
+0x24,
+0x88,
+0x0e,
+0x4a,
+0x20,
+0x24,
+0x88,
+0x16,
+0x4a,
+0x20,
+0x24,
+0x88,
+0x1e,
+0x4a,
+0x20,
+0x24,
+0xa2,
+0x02,
+0x48,
+0x82,
+0x64,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x5c,
+0x00,
+0xca,
+0xc0,
+0xa8,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x03,
+0x32,
+0xa1,
+0x8b,
+0xff,
+0xa1,
+0xa0,
+0x00,
+0x48,
+0x03,
+0x03,
+0x25,
+0x89,
+0x8b,
+0xc0,
+0xc9,
+0x88,
+0x2e,
+0x02,
+0x48,
+0x9a,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x51,
+0x68,
+0x00,
+0x08,
+0x27,
+0x21,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x60,
+0x42,
+0x02,
+0xf8,
+0x23,
+0x52,
+0x40,
+0x00,
+0x00,
+0x60,
+0x61,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x20,
+0xa2,
+0x58,
+0x10,
+0xbc,
+0x35,
+0x88,
+0x83,
+0x60,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x98,
+0x80,
+0x26,
+0xc0,
+0x00,
+0x14,
+0xa0,
+0x03,
+0x00,
+0x80,
+0x68,
+0x00,
+0x02,
+0x32,
+0x20,
+0xbc,
+0x0b,
+0x89,
+0x88,
+0x02,
+0x30,
+0x08,
+0x0b,
+0xc2,
+0x81,
+0x5c,
+0x00,
+0x43,
+0x05,
+0xda,
+0x68,
+0x38,
+0x08,
+0x03,
+0x20,
+0x6c,
+0x70,
+0x10,
+0x06,
+0x52,
+0x84,
+0x1d,
+0x0b,
+0xc2,
+0x07,
+0x5c,
+0xb1,
+0x00,
+0x80,
+0x20,
+0x6c,
+0x70,
+0x10,
+0x20,
+0x0a,
+0x5c,
+0x00,
+0x40,
+0x40,
+0x08,
+0x88,
+0x0a,
+0x48,
+0x40,
+0x81,
+0x86,
+0x08,
+0x28,
+0x81,
+0x20,
+0x88,
+0x1a,
+0x48,
+0x40,
+0x83,
+0x86,
+0x08,
+0x98,
+0x82,
+0x20,
+0x68,
+0x00,
+0x00,
+0x96,
+0x24,
+0x80,
+0x00,
+0xb6,
+0xc0,
+0x00,
+0x30,
+0x64,
+0xa6,
+0xc0,
+0x00,
+0x2f,
+0x04,
+0x86,
+0xc0,
+0x00,
+0x31,
+0xc5,
+0x16,
+0xc4,
+0x00,
+0x24,
+0x65,
+0x26,
+0xc0,
+0x00,
+0x12,
+0xc5,
+0x38,
+0x60,
+0xc9,
+0x6c,
+0x00,
+0x02,
+0x9a,
+0x4b,
+0x40,
+0x00,
+0x00,
+0x40,
+0x50,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x7a,
+0x66,
+0x00,
+0x00,
+0xce,
+0x40,
+0x68,
+0x00,
+0x00,
+0x4b,
+0x20,
+0x5c,
+0x87,
+0x00,
+0x82,
+0xa4,
+0x68,
+0x00,
+0x00,
+0x5f,
+0x21,
+0x80,
+0x00,
+0x05,
+0x14,
+0x22,
+0x00,
+0x80,
+0x05,
+0x14,
+0x22,
+0x86,
+0x02,
+0x46,
+0xc6,
+0x81,
+0x00,
+0x04,
+0x88,
+0x82,
+0xe1,
+0x88,
+0x3e,
+0x06,
+0x83,
+0x40,
+0x80,
+0x02,
+0x5b,
+0xa0,
+0xc8,
+0x86,
+0xc4,
+0x90,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x66,
+0x00,
+0x01,
+0x1d,
+0x40,
+0x66,
+0x00,
+0x02,
+0x3c,
+0x60,
+0x6c,
+0x00,
+0x01,
+0x94,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x68,
+0x00,
+0x00,
+0xc2,
+0x20,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x24,
+0x68,
+0x00,
+0x00,
+0xc6,
+0x21,
+0x68,
+0x20,
+0x00,
+0xe6,
+0x25,
+0x68,
+0x00,
+0x00,
+0xac,
+0x22,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x23,
+0x66,
+0x00,
+0x01,
+0x38,
+0xc0,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x68,
+0x00,
+0x01,
+0x50,
+0x20,
+0x5c,
+0x83,
+0x0a,
+0xc0,
+0x20,
+0x5c,
+0x88,
+0x10,
+0x02,
+0x80,
+0x5c,
+0x87,
+0x18,
+0x40,
+0x02,
+0x57,
+0x00,
+0x80,
+0x83,
+0x20,
+0x5c,
+0x80,
+0x89,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0x8f,
+0x24,
+0x5c,
+0x08,
+0x50,
+0x03,
+0x01,
+0x51,
+0x60,
+0x20,
+0x22,
+0x00,
+0x86,
+0x00,
+0x35,
+0x70,
+0x0c,
+0x00,
+0x18,
+0x35,
+0x70,
+0x2d,
+0x80,
+0x30,
+0x15,
+0xc8,
+0x19,
+0x98,
+0x0c,
+0x35,
+0x16,
+0x0f,
+0x84,
+0x00,
+0x35,
+0x70,
+0x2c,
+0x18,
+0x00,
+0x16,
+0x80,
+0x00,
+0x1d,
+0x52,
+0x45,
+0x16,
+0x07,
+0x18,
+0x00,
+0x05,
+0x16,
+0x03,
+0x96,
+0x0d,
+0x76,
+0x80,
+0x00,
+0x17,
+0x92,
+0x15,
+0xc9,
+0x61,
+0x18,
+0xe4,
+0x95,
+0x24,
+0x55,
+0x80,
+0xa0,
+0x08,
+0x48,
+0x01,
+0x57,
+0x00,
+0x49,
+0x84,
+0xf9,
+0x5c,
+0x09,
+0xc1,
+0x8e,
+0x43,
+0x52,
+0x04,
+0xd9,
+0x61,
+0xd7,
+0x96,
+0x25,
+0x46,
+0x80,
+0x00,
+0x18,
+0x42,
+0x19,
+0x80,
+0x41,
+0x51,
+0x60,
+0x68,
+0x0a,
+0x08,
+0x84,
+0x80,
+0x25,
+0x70,
+0x89,
+0x08,
+0x2a,
+0x09,
+0x80,
+0x82,
+0x51,
+0x60,
+0xb8,
+0x83,
+0xa5,
+0x96,
+0x0d,
+0x79,
+0x60,
+0x55,
+0x98,
+0x4f,
+0x98,
+0x40,
+0x02,
+0x51,
+0x60,
+0xa0,
+0x68,
+0x01,
+0x51,
+0x60,
+0x71,
+0x60,
+0xd6,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x09,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x01,
+0x51,
+0x60,
+0x61,
+0x60,
+0xd4,
+0x51,
+0x61,
+0x71,
+0x60,
+0xd6,
+0x96,
+0x0f,
+0x69,
+0x60,
+0xd4,
+0x00,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x26,
+0xa0,
+0x35,
+0x94,
+0x0c,
+0x16,
+0x0b,
+0x95,
+0x50,
+0x24,
+0xbf,
+0x19,
+0x95,
+0xc0,
+0xe9,
+0x16,
+0x0f,
+0x14,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xa1,
+0xc0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0xf0,
+0xe7,
+0x68,
+0x20,
+0x00,
+0x04,
+0x20,
+0x5c,
+0x0b,
+0xe2,
+0xbf,
+0xf0,
+0x94,
+0x03,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x09,
+0x9a,
+0x01,
+0x20,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0xa0,
+0x00,
+0x12,
+0xa0,
+0x64,
+0x3b,
+0x10,
+0x43,
+0x23,
+0x20,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x38,
+0x17,
+0x42,
+0x59,
+0x28,
+0xbc,
+0x09,
+0x95,
+0xc0,
+0x32,
+0x2c,
+0x02,
+0x06,
+0x82,
+0x00,
+0x01,
+0x92,
+0x12,
+0xa0,
+0x64,
+0x3b,
+0x10,
+0x43,
+0x23,
+0xa0,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x5c,
+0x0a,
+0xe3,
+0x00,
+0x76,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xa9,
+0x5c,
+0x00,
+0x68,
+0x80,
+0x76,
+0x68,
+0x20,
+0x00,
+0x1b,
+0x21,
+0x38,
+0x09,
+0xc2,
+0xa0,
+0x76,
+0x3b,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x04,
+0x76,
+0x89,
+0x8e,
+0x88,
+0x88,
+0x0a,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x04,
+0x08,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x00,
+0x47,
+0x68,
+0x5c,
+0x00,
+0xa9,
+0x8e,
+0x88,
+0x5c,
+0x82,
+0x08,
+0x80,
+0xa0,
+0x39,
+0x02,
+0x0a,
+0x04,
+0x20,
+0x80,
+0x08,
+0xb8,
+0x00,
+0x09,
+0x51,
+0x91,
+0x50,
+0x00,
+0x00,
+0x80,
+0x00,
+0x16,
+0x83,
+0x40,
+0x0a,
+0x02,
+0x18,
+0x00,
+0x08,
+0x84,
+0x84,
+0xb6,
+0x80,
+0x00,
+0x05,
+0x82,
+0x28,
+0x00,
+0x09,
+0x80,
+0x00,
+0xa8,
+0x00,
+0x0b,
+0x6c,
+0x40,
+0x00,
+0x78,
+0x50,
+0x6c,
+0x40,
+0x00,
+0xc6,
+0x51,
+0x85,
+0x0c,
+0xb6,
+0x80,
+0x00,
+0x04,
+0x42,
+0x38,
+0x00,
+0x00,
+0x84,
+0x80,
+0x15,
+0x48,
+0x44,
+0x80,
+0x00,
+0xb8,
+0x40,
+0x82,
+0x80,
+0x00,
+0x38,
+0x48,
+0x51,
+0xa0,
+0x6c,
+0x06,
+0xc4,
+0x00,
+0x15,
+0xc5,
+0x3b,
+0x10,
+0x03,
+0x94,
+0x03,
+0x96,
+0x80,
+0x00,
+0x09,
+0xe2,
+0x15,
+0x44,
+0x66,
+0x05,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x18,
+0xc5,
+0x25,
+0x90,
+0x10,
+0x05,
+0x8c,
+0xa8,
+0x50,
+0x49,
+0x42,
+0x04,
+0x48,
+0x48,
+0x50,
+0x84,
+0x8c,
+0xb8,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x00,
+0x27,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x18,
+0x08,
+0x40,
+0x00,
+0x00,
+0x80,
+0xa0,
+0x68,
+0x00,
+0x01,
+0x00,
+0x09,
+0x94,
+0x03,
+0xc2,
+0x89,
+0x65,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0xa1,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x5c,
+0x08,
+0xe0,
+0x80,
+0xe0,
+0x24,
+0x12,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x06,
+0x60,
+0x00,
+0x02,
+0x9c,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x80,
+0x59,
+0x40,
+0x3c,
+0x28,
+0x96,
+0x53,
+0x20,
+0x28,
+0xbc,
+0x0c,
+0x98,
+0x80,
+0x36,
+0x88,
+0x07,
+0x66,
+0x80,
+0x00,
+0x15,
+0x02,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x86,
+0x80,
+0x04,
+0x01,
+0xcc,
+0x88,
+0x80,
+0x36,
+0x64,
+0x00,
+0x00,
+0x42,
+0x4f,
+0xa8,
+0x01,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0xab,
+0xfd,
+0x04,
+0x20,
+0x6f,
+0x88,
+0x0f,
+0x65,
+0xc0,
+0x03,
+0x08,
+0x04,
+0x98,
+0x82,
+0x4a,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa0,
+0x88,
+0x20,
+0x85,
+0x50,
+0x32,
+0x08,
+0x12,
+0x05,
+0xd4,
+0x83,
+0x08,
+0x18,
+0x85,
+0x50,
+0x32,
+0x20,
+0x02,
+0x08,
+0x80,
+0x09,
+0x58,
+0x0b,
+0x80,
+0x81,
+0xc8,
+0xbf,
+0xf2,
+0xa8,
+0x81,
+0x60,
+0x00,
+0x00,
+0x08,
+0x80,
+0xb6,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xfd,
+0x08,
+0x80,
+0x48,
+0x88,
+0x0f,
+0x66,
+0x82,
+0x00,
+0x08,
+0x52,
+0x08,
+0x81,
+0x7a,
+0x88,
+0x1e,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x0a,
+0x55,
+0x03,
+0xb8,
+0x81,
+0xa0,
+0x5d,
+0x4e,
+0x38,
+0x80,
+0x0a,
+0x59,
+0x09,
+0xc2,
+0x00,
+0x20,
+0x55,
+0x03,
+0xa0,
+0x81,
+0x4b,
+0xbf,
+0xf3,
+0xa8,
+0x80,
+0x48,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x40,
+0x80,
+0x68,
+0x20,
+0x00,
+0x85,
+0x20,
+0x88,
+0x20,
+0x05,
+0x16,
+0x02,
+0x20,
+0x08,
+0x08,
+0x80,
+0xb6,
+0x46,
+0x0a,
+0x41,
+0x40,
+0x64,
+0x94,
+0x0e,
+0x0a,
+0x80,
+0x30,
+0x5c,
+0x00,
+0x22,
+0xbf,
+0x20,
+0x88,
+0x1c,
+0x8a,
+0x40,
+0xc0,
+0x88,
+0x16,
+0x08,
+0x82,
+0x76,
+0x68,
+0x00,
+0x40,
+0x1a,
+0x48,
+0x88,
+0x2c,
+0x86,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x88,
+0x55,
+0x03,
+0x20,
+0x81,
+0x20,
+0x5d,
+0x48,
+0x30,
+0x82,
+0x88,
+0x59,
+0x0f,
+0x82,
+0x00,
+0x20,
+0x43,
+0xfa,
+0x50,
+0x81,
+0x60,
+0x55,
+0x03,
+0x20,
+0x81,
+0xca,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x42,
+0x80,
+0x5c,
+0x81,
+0x02,
+0x40,
+0xc0,
+0x5c,
+0x00,
+0x0a,
+0x00,
+0x71,
+0xa0,
+0xc1,
+0x29,
+0x48,
+0x35,
+0x95,
+0x03,
+0x7a,
+0x41,
+0xa1,
+0x23,
+0x08,
+0xc5,
+0x50,
+0x26,
+0x18,
+0x22,
+0x95,
+0xd4,
+0x80,
+0x9c,
+0x08,
+0x25,
+0x90,
+0xe4,
+0x15,
+0x0b,
+0x65,
+0x70,
+0xb9,
+0x15,
+0x03,
+0x45,
+0x70,
+0xf0,
+0x1c,
+0x88,
+0x34,
+0x3f,
+0x9d,
+0x15,
+0x8f,
+0x29,
+0x58,
+0x70,
+0x5c,
+0x86,
+0x0a,
+0x42,
+0xa3,
+0x88,
+0xa0,
+0x85,
+0x16,
+0x12,
+0x20,
+0x87,
+0x05,
+0x1a,
+0x12,
+0x20,
+0x46,
+0x15,
+0xb0,
+0x81,
+0x20,
+0x82,
+0x25,
+0x15,
+0x88,
+0x08,
+0x16,
+0x22,
+0xe1,
+0x12,
+0x2e,
+0x11,
+0x12,
+0x2b,
+0x09,
+0x51,
+0x58,
+0x90,
+0x18,
+0x41,
+0x51,
+0x59,
+0x30,
+0x18,
+0x42,
+0x54,
+0x09,
+0x00,
+0x18,
+0x40,
+0x54,
+0x08,
+0x10,
+0x18,
+0x7a,
+0x51,
+0x58,
+0x00,
+0x18,
+0x4a,
+0x51,
+0x58,
+0x80,
+0x18,
+0x40,
+0x81,
+0xac,
+0x08,
+0x80,
+0x63,
+0x40,
+0x00,
+0x00,
+0x81,
+0xe3,
+0x68,
+0x20,
+0x00,
+0x90,
+0x23,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x66,
+0x00,
+0x00,
+0x37,
+0xe0,
+0x5c,
+0x81,
+0x00,
+0x8a,
+0x08,
+0x51,
+0xa1,
+0x20,
+0x81,
+0xa3,
+0x5b,
+0x08,
+0x12,
+0xc0,
+0xc1,
+0x51,
+0x58,
+0x80,
+0x81,
+0x20,
+0x2e,
+0x11,
+0x25,
+0x70,
+0x88,
+0xa0,
+0x03,
+0x05,
+0x15,
+0x84,
+0xa0,
+0x46,
+0x15,
+0x15,
+0x89,
+0x01,
+0x84,
+0x15,
+0x15,
+0x93,
+0x01,
+0x84,
+0x25,
+0x40,
+0x90,
+0x01,
+0x84,
+0x05,
+0x40,
+0x81,
+0x01,
+0x87,
+0xa5,
+0x15,
+0x80,
+0x01,
+0x84,
+0xa5,
+0x15,
+0x88,
+0x01,
+0x84,
+0x08,
+0x1a,
+0xc0,
+0x88,
+0x06,
+0x36,
+0x82,
+0x00,
+0x09,
+0x8a,
+0x36,
+0x82,
+0x00,
+0x09,
+0x2a,
+0x46,
+0x82,
+0x00,
+0x09,
+0x5a,
+0x56,
+0x60,
+0x00,
+0x03,
+0x7e,
+0x8a,
+0x08,
+0x22,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x0e,
+0x00,
+0x00,
+0x00,
+0xba,
+0x14,
+0x82,
+0xe1,
+0xa8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x59,
+0x06,
+0x42,
+0xbf,
+0xd0,
+0x49,
+0xa9,
+0xb8,
+0x80,
+0x76,
+0xbc,
+0x19,
+0x89,
+0xa0,
+0x00,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x10,
+0x02,
+0xa0,
+0x76,
+0x68,
+0x00,
+0x09,
+0xc4,
+0x09,
+0x50,
+0x41,
+0x83,
+0xa1,
+0x11,
+0x44,
+0x08,
+0x81,
+0x01,
+0x58,
+0x5b,
+0x40,
+0x11,
+0x02,
+0x59,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x42,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x19,
+0x02,
+0x12,
+0x29,
+0x85,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x1a,
+0x33,
+0x60,
+0x00,
+0x37,
+0x80,
+0x0b,
+0xc1,
+0x77,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x10,
+0x02,
+0xa0,
+0x64,
+0x68,
+0x07,
+0xff,
+0xff,
+0xc9,
+0x50,
+0x41,
+0x03,
+0xa1,
+0x11,
+0x44,
+0x48,
+0x81,
+0x01,
+0x58,
+0x5b,
+0x40,
+0x11,
+0x02,
+0x59,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x42,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x19,
+0x02,
+0x12,
+0x29,
+0x85,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x02,
+0x33,
+0x60,
+0x00,
+0x37,
+0x80,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5b,
+0x02,
+0x03,
+0x01,
+0x01,
+0x50,
+0x43,
+0x11,
+0x84,
+0x0a,
+0x46,
+0x08,
+0x89,
+0x83,
+0x88,
+0x44,
+0x41,
+0x09,
+0x84,
+0x8b,
+0x98,
+0x30,
+0xa4,
+0x47,
+0x44,
+0xab,
+0xfe,
+0x05,
+0x08,
+0x20,
+0x08,
+0x07,
+0x60,
+0x88,
+0x82,
+0x37,
+0x80,
+0x06,
+0x80,
+0x00,
+0x9c,
+0x40,
+0x80,
+0x84,
+0x80,
+0x6a,
+0x00,
+0x4e,
+0x20,
+0x02,
+0x5b,
+0x44,
+0x11,
+0x01,
+0x58,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x40,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x26,
+0xa0,
+0x04,
+0xe2,
+0x00,
+0x12,
+0x98,
+0x52,
+0x32,
+0x01,
+0x0b,
+0xc0,
+0x3b,
+0x88,
+0x03,
+0x63,
+0x60,
+0x00,
+0x37,
+0x80,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0xab,
+0xf8,
+0x09,
+0x40,
+0x36,
+0x88,
+0x04,
+0xa8,
+0x80,
+0xe5,
+0x88,
+0x16,
+0x48,
+0x81,
+0xe3,
+0x88,
+0x26,
+0x28,
+0x82,
+0xf6,
+0x66,
+0x00,
+0x00,
+0x31,
+0xa8,
+0x40,
+0x00,
+0x01,
+0x48,
+0x35,
+0x5c,
+0x00,
+0x38,
+0x83,
+0x48,
+0x6a,
+0x00,
+0x02,
+0x71,
+0x00,
+0x5b,
+0x40,
+0x00,
+0x83,
+0xcb,
+0x88,
+0x45,
+0x49,
+0x05,
+0x58,
+0x00,
+0x00,
+0x08,
+0x80,
+0x0a,
+0x51,
+0x85,
+0xf8,
+0x82,
+0x22,
+0x98,
+0x2e,
+0x89,
+0xd0,
+0x02,
+0x88,
+0x4e,
+0x86,
+0x60,
+0x00,
+0x03,
+0x1a,
+0x89,
+0x50,
+0x35,
+0x5b,
+0x48,
+0x03,
+0x01,
+0xfe,
+0x52,
+0xcd,
+0x01,
+0x82,
+0x0b,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xc9,
+0x55,
+0x01,
+0xd1,
+0x80,
+0x0a,
+0xbc,
+0x07,
+0x88,
+0x86,
+0x52,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0xcd,
+0x88,
+0x38,
+0x16,
+0x80,
+0x01,
+0xff,
+0xfc,
+0x2b,
+0xc0,
+0x87,
+0x68,
+0x00,
+0x20,
+0x00,
+0x09,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x4d,
+0x40,
+0x00,
+0x00,
+0x83,
+0x81,
+0x68,
+0x00,
+0x20,
+0x00,
+0x02,
+0x59,
+0x0a,
+0x40,
+0x81,
+0xa2,
+0xbc,
+0x03,
+0xb3,
+0x90,
+0x10,
+0x40,
+0x00,
+0x01,
+0x50,
+0x52,
+0x55,
+0x02,
+0x70,
+0x81,
+0xe2,
+0x00,
+0x00,
+0x08,
+0x84,
+0xa8,
+0x88,
+0x82,
+0x35,
+0x18,
+0x5b,
+0x08,
+0x4c,
+0xa9,
+0xd8,
+0x04,
+0x98,
+0x2a,
+0x88,
+0x60,
+0x08,
+0x9d,
+0x80,
+0x38,
+0x83,
+0x0a,
+0x88,
+0x6e,
+0x46,
+0x60,
+0x00,
+0x03,
+0x1e,
+0x88,
+0x58,
+0x09,
+0x88,
+0x38,
+0xb3,
+0x20,
+0xf8,
+0xbc,
+0x13,
+0x85,
+0xc0,
+0xff,
+0x99,
+0x20,
+0x19,
+0x05,
+0x12,
+0x40,
+0x00,
+0x01,
+0x07,
+0x58,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x40,
+0x08,
+0x84,
+0x16,
+0x55,
+0x00,
+0x09,
+0x07,
+0x10,
+0x6a,
+0x00,
+0x02,
+0x71,
+0x02,
+0x29,
+0x88,
+0x23,
+0x20,
+0x10,
+0xbc,
+0x03,
+0xb3,
+0x81,
+0xff,
+0x36,
+0x04,
+0x13,
+0x78,
+0x41,
+0x52,
+0xce,
+0x41,
+0x83,
+0x4b,
+0x36,
+0x9c,
+0x26,
+0x80,
+0x01,
+0xff,
+0xfc,
+0xbb,
+0xc0,
+0x68,
+0x98,
+0x08,
+0xa3,
+0x01,
+0xf0,
+0xbc,
+0x09,
+0x56,
+0xa0,
+0x01,
+0xff,
+0xfc,
+0x1b,
+0xc0,
+0x67,
+0x68,
+0x00,
+0x20,
+0x00,
+0x0b,
+0x30,
+0x1f,
+0x0b,
+0xc0,
+0x25,
+0x6a,
+0x00,
+0x20,
+0x00,
+0x01,
+0x5c,
+0x80,
+0x80,
+0x81,
+0x22,
+0x98,
+0x34,
+0x19,
+0x50,
+0x51,
+0x88,
+0x16,
+0x20,
+0x00,
+0x00,
+0x88,
+0x6a,
+0x38,
+0x83,
+0x08,
+0x66,
+0x00,
+0x00,
+0x35,
+0x88,
+0x55,
+0x00,
+0x08,
+0x58,
+0x09,
+0x5c,
+0x0f,
+0xf9,
+0x83,
+0x0a,
+0x36,
+0x98,
+0x12,
+0x59,
+0xc0,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xca,
+0xbc,
+0x07,
+0x89,
+0x80,
+0x4b,
+0x30,
+0x1b,
+0x8b,
+0xc0,
+0xcd,
+0x88,
+0x48,
+0xb6,
+0xa0,
+0x01,
+0xff,
+0xfc,
+0x0b,
+0xc0,
+0x87,
+0x68,
+0x00,
+0x20,
+0x00,
+0x0a,
+0x30,
+0x1b,
+0x8b,
+0xc0,
+0x4d,
+0x40,
+0x00,
+0x00,
+0x84,
+0x8b,
+0x6a,
+0x00,
+0x20,
+0x00,
+0x00,
+0x5c,
+0x80,
+0x80,
+0x80,
+0xa2,
+0x5d,
+0x8e,
+0x39,
+0x83,
+0x00,
+0x59,
+0x0d,
+0xc0,
+0x86,
+0x09,
+0x95,
+0x05,
+0x08,
+0x83,
+0x49,
+0x88,
+0x3c,
+0xb8,
+0x80,
+0xe2,
+0xbf,
+0x75,
+0xa8,
+0x80,
+0x0a,
+0x88,
+0x2b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x08,
+0x00,
+0x00,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0x79,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x08,
+0x68,
+0x00,
+0x01,
+0x84,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x48,
+0x68,
+0x00,
+0x01,
+0x8f,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x88,
+0x68,
+0x00,
+0x40,
+0x1d,
+0xc8,
+0x68,
+0x20,
+0x01,
+0xcb,
+0x20,
+0x66,
+0x00,
+0x00,
+0x26,
+0x08,
+0x5c,
+0x00,
+0xeb,
+0x80,
+0x00,
+0x68,
+0x00,
+0x40,
+0x1e,
+0x88,
+0x68,
+0x20,
+0x01,
+0xce,
+0x20,
+0x66,
+0x00,
+0x00,
+0x26,
+0x08,
+0x38,
+0x01,
+0xd8,
+0x80,
+0x36,
+0x68,
+0x00,
+0x40,
+0x1f,
+0x48,
+0x68,
+0x20,
+0x01,
+0xd1,
+0x20,
+0x64,
+0x00,
+0x00,
+0x26,
+0x0f,
+0x5c,
+0x00,
+0xea,
+0x80,
+0x10,
+0x39,
+0x02,
+0x08,
+0x00,
+0x7a,
+0x80,
+0x07,
+0xa8,
+0x08,
+0x7a,
+0x80,
+0x07,
+0xa8,
+0x08,
+0x7a,
+0x80,
+0x87,
+0xa4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xfe,
+0x08,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x66,
+0x80,
+0x04,
+0x01,
+0x00,
+0x88,
+0x81,
+0x7a,
+0x88,
+0x1c,
+0x86,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x00,
+0x55,
+0x02,
+0x00,
+0x80,
+0x20,
+0x5d,
+0x00,
+0x03,
+0x00,
+0xe2,
+0x58,
+0x04,
+0x02,
+0x00,
+0x20,
+0x88,
+0x18,
+0x14,
+0x3f,
+0x9d,
+0x08,
+0x06,
+0x05,
+0x50,
+0x26,
+0x08,
+0x15,
+0x00,
+0x00,
+0x00,
+0x88,
+0x0b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x68,
+0x68,
+0x34,
+0x04,
+0x14,
+0x21,
+0x6c,
+0x68,
+0x08,
+0x28,
+0x49,
+0x00,
+0x00,
+0x08,
+0x48,
+0x89,
+0x84,
+0x04,
+0x94,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xc0,
+0x4f,
+0x5c,
+0x09,
+0xab,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x08,
+0x25,
+0x96,
+0x0b,
+0xff,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x6c,
+0x68,
+0x08,
+0x28,
+0x49,
+0x00,
+0x00,
+0x08,
+0x40,
+0x09,
+0x84,
+0x8c,
+0x94,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xc0,
+0x4f,
+0x5c,
+0x09,
+0xab,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x08,
+0x25,
+0x96,
+0x0b,
+0xff,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x59,
+0x01,
+0x41,
+0x48,
+0x2f,
+0xbc,
+0x0d,
+0x95,
+0xc0,
+0x87,
+0x14,
+0x02,
+0xd3,
+0x20,
+0x20,
+0xbc,
+0x05,
+0x05,
+0x24,
+0xdc,
+0x3a,
+0x14,
+0x89,
+0x48,
+0x60,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0x4d,
+0x43,
+0xa1,
+0x48,
+0x94,
+0x06,
+0x00,
+0x00,
+0x00,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x40,
+0x52,
+0x0d,
+0xc3,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x00,
+0x00,
+0x00,
+0x52,
+0x0d,
+0x43,
+0xa1,
+0x48,
+0x94,
+0x06,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x05,
+0x20,
+0x40,
+0x7a,
+0x46,
+0x0a,
+0x40,
+0x40,
+0xfa,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x40,
+0x46,
+0x08,
+0x88,
+0x02,
+0x0a,
+0x08,
+0x80,
+0x04,
+0x60,
+0xa4,
+0x00,
+0x24,
+0xc8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x01,
+0x84,
+0x00,
+0x88,
+0x40,
+0x8a,
+0x30,
+0x93,
+0x0b,
+0xc0,
+0x5c,
+0xb0,
+0x00,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x40,
+0xfa,
+0x98,
+0xe8,
+0x8b,
+0xa1,
+0x40,
+0x84,
+0x08,
+0x03,
+0x28,
+0x00,
+0xbc,
+0x05,
+0x08,
+0x40,
+0x00,
+0x55,
+0x02,
+0x23,
+0xa1,
+0x48,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x80,
+0x43,
+0xc0,
+0x5f,
+0x5c,
+0x00,
+0x31,
+0x8e,
+0x80,
+0x55,
+0x02,
+0x01,
+0x40,
+0x46,
+0x3a,
+0x80,
+0x03,
+0x01,
+0x00,
+0xbf,
+0xfb,
+0x2b,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x1c,
+0x04,
+0x21,
+0x38,
+0x00,
+0xe8,
+0x48,
+0x48,
+0xa0,
+0x82,
+0x18,
+0x48,
+0x7a,
+0xa0,
+0x82,
+0x18,
+0x48,
+0x4a,
+0xa0,
+0xcc,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x80,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xb0,
+0x68,
+0x00,
+0x00,
+0x9e,
+0x20,
+0x68,
+0x00,
+0x01,
+0x50,
+0x24,
+0x5c,
+0x00,
+0x10,
+0x00,
+0x01,
+0x5c,
+0x08,
+0x2a,
+0x00,
+0xa1,
+0x55,
+0x00,
+0xb1,
+0x48,
+0x2f,
+0x52,
+0xcb,
+0xc0,
+0x22,
+0x08,
+0x55,
+0x00,
+0x98,
+0x82,
+0x48,
+0xa0,
+0xd4,
+0x18,
+0x40,
+0x09,
+0x88,
+0x36,
+0x44,
+0x20,
+0x44,
+0x88,
+0x3f,
+0x65,
+0x50,
+0x0a,
+0x08,
+0x2e,
+0x16,
+0x80,
+0x00,
+0x09,
+0x62,
+0x06,
+0xc0,
+0x00,
+0x29,
+0xa0,
+0x28,
+0x40,
+0x03,
+0x84,
+0x08,
+0x83,
+0x81,
+0x08,
+0x25,
+0x83,
+0x8b,
+0xc1,
+0x09,
+0x55,
+0x01,
+0x81,
+0x82,
+0x8b,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xae,
+0x38,
+0x12,
+0x82,
+0x58,
+0x30,
+0xbc,
+0x09,
+0x85,
+0x50,
+0x1c,
+0x18,
+0x2c,
+0xa6,
+0x80,
+0x00,
+0x09,
+0x82,
+0x06,
+0xc0,
+0x00,
+0x29,
+0xc0,
+0xb8,
+0x40,
+0x8a,
+0x40,
+0x00,
+0x00,
+0x40,
+0x00,
+0x57,
+0x02,
+0xc8,
+0x82,
+0x03,
+0x57,
+0x06,
+0x92,
+0x08,
+0x00,
+0x2e,
+0x16,
+0x31,
+0x08,
+0x4e,
+0x54,
+0x0e,
+0x81,
+0xa0,
+0x42,
+0x6c,
+0x40,
+0x01,
+0x3e,
+0x08,
+0x44,
+0x20,
+0x81,
+0x80,
+0x09,
+0x6e,
+0x40,
+0x01,
+0x9d,
+0x23,
+0x6c,
+0x40,
+0x03,
+0x24,
+0x0b,
+0x6c,
+0x40,
+0x01,
+0x6e,
+0x08,
+0x20,
+0x0c,
+0x94,
+0xc1,
+0x42,
+0x80,
+0x84,
+0x18,
+0x82,
+0x61,
+0x00,
+0x00,
+0x06,
+0xe4,
+0x00,
+0x19,
+0xf2,
+0x46,
+0xc4,
+0x00,
+0x25,
+0x60,
+0x92,
+0x01,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x0b,
+0x51,
+0x8b,
+0x48,
+0x48,
+0x40,
+0x30,
+0x07,
+0x86,
+0xe4,
+0x00,
+0x35,
+0x52,
+0x65,
+0x00,
+0xc8,
+0x3c,
+0x53,
+0xd8,
+0x60,
+0x40,
+0x22,
+0x96,
+0xc2,
+0xe1,
+0x3c,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x48,
+0x22,
+0x96,
+0x43,
+0x01,
+0x60,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x48,
+0xbc,
+0x45,
+0x46,
+0xc0,
+0x00,
+0x36,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x05,
+0x16,
+0xc0,
+0x00,
+0x37,
+0x60,
+0xa3,
+0x20,
+0x30,
+0x40,
+0x00,
+0x03,
+0xc3,
+0x90,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x0a,
+0x2e,
+0x9a,
+0x42,
+0xa0,
+0x64,
+0x6c,
+0x40,
+0x02,
+0x1c,
+0x0a,
+0x6c,
+0x00,
+0x03,
+0x68,
+0x48,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0xec,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x48,
+0x51,
+0x89,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x49,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x7a,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x51,
+0x8b,
+0x6b,
+0xc2,
+0xcf,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x49,
+0x68,
+0x00,
+0x00,
+0x70,
+0x25,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x24,
+0x82,
+0x87,
+0xa6,
+0x80,
+0x00,
+0x1b,
+0x12,
+0x68,
+0x20,
+0x7a,
+0xa2,
+0x80,
+0x38,
+0x28,
+0x7a,
+0xa2,
+0x00,
+0x78,
+0x58,
+0xfa,
+0x68,
+0x00,
+0x01,
+0xb8,
+0x22,
+0x82,
+0x07,
+0xa8,
+0x78,
+0xfa,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x7a,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x49,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x87,
+0x00,
+0x86,
+0xc4,
+0x00,
+0x3e,
+0x24,
+0x88,
+0x51,
+0xfa,
+0x87,
+0x1f,
+0xa8,
+0x68,
+0xfa,
+0x86,
+0x0f,
+0xa0,
+0x00,
+0x00,
+0x42,
+0x05,
+0x78,
+0x50,
+0x0a,
+0x85,
+0x8c,
+0xab,
+0xc0,
+0x8f,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0xa1,
+0x82,
+0x58,
+0x84,
+0x65,
+0x66,
+0x00,
+0x00,
+0xcd,
+0x80,
+0x68,
+0x00,
+0x00,
+0x89,
+0x20,
+0x88,
+0x2a,
+0x18,
+0x40,
+0x48,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x40,
+0x00,
+0x00,
+0x82,
+0x20,
+0x66,
+0x00,
+0x00,
+0xcd,
+0x80,
+0x68,
+0x00,
+0x00,
+0x92,
+0x20,
+0x68,
+0x00,
+0x00,
+0x89,
+0x21,
+0x5c,
+0x82,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x80,
+0x80,
+0x98,
+0x82,
+0x61,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x2c,
+0x5c,
+0x87,
+0x00,
+0x82,
+0x20,
+0x88,
+0x1e,
+0xc8,
+0x02,
+0x48,
+0x68,
+0x20,
+0x00,
+0xdf,
+0x2c,
+0x68,
+0x20,
+0x00,
+0xdc,
+0x2d,
+0x68,
+0x20,
+0x00,
+0xde,
+0x22,
+0x88,
+0x26,
+0x06,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x46,
+0x80,
+0x00,
+0x0a,
+0x82,
+0x16,
+0x82,
+0x00,
+0x0d,
+0x32,
+0x58,
+0x80,
+0x6c,
+0x88,
+0x0e,
+0xd8,
+0x81,
+0x62,
+0x66,
+0x00,
+0x00,
+0x99,
+0x48,
+0x40,
+0x00,
+0x01,
+0x82,
+0x09,
+0x5c,
+0x82,
+0x00,
+0x82,
+0x20,
+0x68,
+0x00,
+0x00,
+0x92,
+0x21,
+0x84,
+0x34,
+0x86,
+0x80,
+0x00,
+0x07,
+0xa2,
+0x08,
+0x08,
+0x09,
+0x40,
+0x00,
+0x00,
+0x82,
+0x61,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x5c,
+0x87,
+0x00,
+0x82,
+0x20,
+0x88,
+0x42,
+0x18,
+0x02,
+0x48,
+0x88,
+0x1e,
+0x16,
+0x82,
+0x00,
+0x0d,
+0xfa,
+0xc6,
+0x82,
+0x00,
+0x0d,
+0xd2,
+0xd6,
+0x82,
+0x00,
+0x0d,
+0xea,
+0x28,
+0x82,
+0x60,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x00,
+0x00,
+0xaa,
+0x21,
+0x68,
+0x20,
+0x00,
+0xd5,
+0x25,
+0x88,
+0x06,
+0xc8,
+0x80,
+0xed,
+0x88,
+0x16,
+0x26,
+0x60,
+0x00,
+0x09,
+0x94,
+0x89,
+0x82,
+0x09,
+0x5c,
+0x82,
+0x00,
+0x83,
+0x21,
+0x68,
+0x00,
+0x01,
+0x49,
+0x20,
+0x84,
+0x80,
+0x98,
+0x82,
+0x21,
+0x80,
+0x04,
+0x98,
+0x82,
+0x60,
+0x40,
+0x00,
+0x00,
+0x4b,
+0x48,
+0x68,
+0x00,
+0x01,
+0x3a,
+0x20,
+0x68,
+0x20,
+0x01,
+0x8f,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x59,
+0x03,
+0x40,
+0x82,
+0x21,
+0xbc,
+0x66,
+0x88,
+0x48,
+0x48,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc5,
+0xd1,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x2a,
+0x8e,
+0xc3,
+0x20,
+0xe0,
+0xbc,
+0x27,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xe0,
+0x86,
+0xc0,
+0x00,
+0x11,
+0x40,
+0xa2,
+0x81,
+0x30,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x4b,
+0x38,
+0x10,
+0x72,
+0x41,
+0xed,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x49,
+0x2e,
+0x13,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x06,
+0xd3,
+0x81,
+0x0d,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x24,
+0x16,
+0x56,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x90,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x2a,
+0x8e,
+0xd3,
+0x20,
+0xe8,
+0xbc,
+0x0f,
+0x93,
+0x90,
+0x20,
+0x68,
+0x00,
+0x00,
+0x70,
+0x21,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x09,
+0x80,
+0x87,
+0xaa,
+0x08,
+0x00,
+0x80,
+0x87,
+0xa8,
+0x40,
+0xfa,
+0x6c,
+0x40,
+0x01,
+0x5a,
+0x49,
+0x84,
+0x8f,
+0xab,
+0xc0,
+0x27,
+0x5c,
+0x81,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x38,
+0x18,
+0x62,
+0x89,
+0xa5,
+0x30,
+0x1a,
+0x8b,
+0xc2,
+0x70,
+0x6c,
+0x40,
+0x02,
+0x0e,
+0x0b,
+0x6c,
+0x00,
+0x01,
+0x26,
+0x00,
+0x28,
+0x1c,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x04,
+0xb3,
+0x81,
+0x25,
+0x24,
+0x16,
+0x46,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x82,
+0xe1,
+0xc4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x38,
+0x12,
+0xf2,
+0x41,
+0xe4,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x28,
+0x9a,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x11,
+0x16,
+0x80,
+0x00,
+0x07,
+0xa2,
+0x16,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x88,
+0x08,
+0x7a,
+0xa0,
+0x80,
+0x08,
+0x08,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0xa4,
+0x88,
+0x48,
+0xfa,
+0xbc,
+0x05,
+0x7b,
+0xc0,
+0x4f,
+0x38,
+0x12,
+0x5b,
+0xc0,
+0x2f,
+0x38,
+0x12,
+0x53,
+0x81,
+0x25,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x21,
+0x6c,
+0x40,
+0x03,
+0xd2,
+0x08,
+0x84,
+0xb8,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0xb5,
+0x24,
+0xbe,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x7d,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x21,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x09,
+0x84,
+0xb8,
+0xb3,
+0x01,
+0x38,
+0xbc,
+0x05,
+0x93,
+0x81,
+0x2e,
+0x52,
+0x4d,
+0x6b,
+0xc0,
+0x5f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x24,
+0x1a,
+0xd6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x93,
+0x81,
+0x35,
+0x6c,
+0x40,
+0x01,
+0xb8,
+0x08,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x89,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x08,
+0x52,
+0x4b,
+0x28,
+0x83,
+0xb6,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x6f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x52,
+0x0b,
+0x28,
+0x83,
+0xb6,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x38,
+0x13,
+0xd6,
+0xc4,
+0x00,
+0x1b,
+0xa0,
+0x83,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x38,
+0x1d,
+0x56,
+0xc0,
+0x00,
+0x36,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x0a,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x86,
+0xc0,
+0x00,
+0x37,
+0x60,
+0xa3,
+0x20,
+0x30,
+0xbc,
+0x04,
+0x15,
+0x24,
+0xb2,
+0x3c,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x82,
+0x41,
+0x64,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xa0,
+0x86,
+0xc4,
+0x00,
+0x25,
+0x60,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x00,
+0x02,
+0x32,
+0x21,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x14,
+0xa6,
+0x1a,
+0x80,
+0x50,
+0x68,
+0x00,
+0x00,
+0x8a,
+0x20,
+0x5c,
+0x85,
+0x02,
+0xbf,
+0xa0,
+0x80,
+0x20,
+0x98,
+0x81,
+0x60,
+0x68,
+0x20,
+0x01,
+0xb4,
+0x2c,
+0x88,
+0x1f,
+0x66,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x46,
+0x82,
+0x00,
+0x0c,
+0xe2,
+0x56,
+0x80,
+0x00,
+0x1a,
+0x22,
+0x08,
+0x80,
+0x6c,
+0x66,
+0x00,
+0x00,
+0xb0,
+0x40,
+0x68,
+0x20,
+0x00,
+0xdc,
+0x2c,
+0x6c,
+0x00,
+0x03,
+0x44,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x60,
+0x00,
+0x09,
+0x82,
+0x89,
+0x82,
+0x09,
+0x68,
+0x00,
+0x00,
+0x93,
+0x20,
+0x5c,
+0x85,
+0x00,
+0x81,
+0x24,
+0x80,
+0x20,
+0x98,
+0x60,
+0x48,
+0x88,
+0x26,
+0x06,
+0x82,
+0x00,
+0x1b,
+0x62,
+0xc6,
+0x82,
+0x00,
+0x0b,
+0x42,
+0x46,
+0x82,
+0x00,
+0x0c,
+0xf2,
+0x56,
+0x80,
+0x00,
+0x1a,
+0x42,
+0x08,
+0x80,
+0x6c,
+0x66,
+0x00,
+0x00,
+0xb0,
+0x40,
+0x68,
+0x20,
+0x00,
+0xdd,
+0x2c,
+0x6c,
+0x00,
+0x03,
+0x48,
+0x48,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcf,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x60,
+0x00,
+0x09,
+0x82,
+0x89,
+0x82,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x20,
+0x88,
+0x22,
+0x48,
+0x02,
+0x09,
+0x86,
+0x04,
+0x88,
+0x81,
+0x60,
+0x68,
+0x00,
+0x00,
+0x70,
+0x21,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb1,
+0xa0,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x24,
+0x88,
+0x22,
+0x0a,
+0x20,
+0x44,
+0x80,
+0x20,
+0x98,
+0x60,
+0x48,
+0x88,
+0x2e,
+0x48,
+0x82,
+0x60,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb1,
+0xa0,
+0x88,
+0x22,
+0x06,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x4a,
+0x00,
+0x40,
+0x86,
+0x60,
+0x36,
+0xc4,
+0x00,
+0x2a,
+0xc0,
+0x25,
+0x80,
+0x4c,
+0x04,
+0x04,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x08,
+0xbc,
+0x06,
+0xa5,
+0xc0,
+0x82,
+0x88,
+0x36,
+0x05,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x20,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x09,
+0x84,
+0x60,
+0x33,
+0x00,
+0x98,
+0xbc,
+0x05,
+0xa3,
+0x81,
+0x0c,
+0x52,
+0x49,
+0x6b,
+0xc0,
+0x6f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x52,
+0x09,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x88,
+0x12,
+0x16,
+0x82,
+0x00,
+0x0d,
+0xe2,
+0x56,
+0x60,
+0x00,
+0x0b,
+0x70,
+0x88,
+0x49,
+0x09,
+0x88,
+0x22,
+0x06,
+0x80,
+0x00,
+0x0a,
+0x02,
+0x48,
+0x41,
+0x09,
+0x40,
+0x00,
+0x00,
+0x60,
+0x48,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xde,
+0xa5,
+0x66,
+0x00,
+0x00,
+0xb7,
+0x00,
+0x68,
+0x00,
+0x00,
+0xa0,
+0x20,
+0x68,
+0x20,
+0x00,
+0xd7,
+0x24,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x02,
+0x5c,
+0x83,
+0x04,
+0x00,
+0xc0,
+0x5d,
+0x0e,
+0x2a,
+0x00,
+0x03,
+0x51,
+0x85,
+0x68,
+0x00,
+0x48,
+0x98,
+0x26,
+0x88,
+0x82,
+0x64,
+0x9e,
+0x00,
+0x48,
+0x60,
+0x09,
+0x68,
+0x20,
+0x00,
+0xcc,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcd,
+0x21,
+0x86,
+0x04,
+0x98,
+0x48,
+0x49,
+0x88,
+0x3e,
+0x08,
+0x84,
+0x63,
+0x88,
+0x4d,
+0x08,
+0x81,
+0x48,
+0x66,
+0x00,
+0x00,
+0xc5,
+0x28,
+0x40,
+0x00,
+0x02,
+0x20,
+0x00,
+0x5c,
+0x09,
+0x28,
+0x84,
+0x20,
+0x88,
+0x3a,
+0x49,
+0x46,
+0x2a,
+0x52,
+0xca,
+0x82,
+0x24,
+0x84,
+0xbc,
+0x09,
+0x85,
+0xc8,
+0x30,
+0x08,
+0x3e,
+0x46,
+0x82,
+0x00,
+0x15,
+0x92,
+0x40,
+0x00,
+0x00,
+0x42,
+0x04,
+0x78,
+0x22,
+0x03,
+0x40,
+0x00,
+0x00,
+0x60,
+0x0b,
+0x68,
+0x20,
+0x01,
+0x5d,
+0x24,
+0x00,
+0x00,
+0x08,
+0x22,
+0x03,
+0x40,
+0x00,
+0x00,
+0x60,
+0x0b,
+0x5c,
+0x09,
+0x78,
+0x84,
+0x4b,
+0x25,
+0x9d,
+0x0b,
+0xc0,
+0x88,
+0x88,
+0x55,
+0x30,
+0x00,
+0x00,
+0x88,
+0x40,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x04,
+0x9b,
+0xc4,
+0x9f,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x49,
+0x25,
+0x95,
+0x0b,
+0xc3,
+0x01,
+0x6c,
+0x00,
+0x01,
+0x40,
+0x0a,
+0x66,
+0x00,
+0x00,
+0xca,
+0xc8,
+0x5c,
+0x00,
+0x28,
+0x40,
+0x0b,
+0x6c,
+0x40,
+0x01,
+0xae,
+0x09,
+0x5d,
+0x0a,
+0x28,
+0x82,
+0x20,
+0x23,
+0x0a,
+0xa9,
+0x84,
+0xa8,
+0x9c,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2a,
+0x80,
+0xb8,
+0x40,
+0x0a,
+0x08,
+0xb2,
+0x09,
+0x80,
+0x82,
+0x2e,
+0x0a,
+0x73,
+0x20,
+0x38,
+0xbc,
+0x03,
+0xd8,
+0x85,
+0x09,
+0xbc,
+0x02,
+0xf0,
+0x8d,
+0x20,
+0x98,
+0xe8,
+0x28,
+0x84,
+0x09,
+0x2e,
+0x0a,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0x00,
+0x99,
+0x80,
+0x82,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0xa4,
+0x6c,
+0x40,
+0x02,
+0xaa,
+0x09,
+0x2e,
+0x12,
+0xd3,
+0x20,
+0x28,
+0xbc,
+0x20,
+0x56,
+0xc4,
+0x00,
+0x18,
+0x05,
+0x26,
+0xc4,
+0x00,
+0x15,
+0x05,
+0x2b,
+0xc1,
+0xb7,
+0x6c,
+0x40,
+0x01,
+0x50,
+0x52,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x7f,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x52,
+0x59,
+0x01,
+0x03,
+0x01,
+0x05,
+0xbc,
+0x09,
+0x96,
+0xc4,
+0x00,
+0x3d,
+0x00,
+0x82,
+0x59,
+0x60,
+0xbc,
+0x0e,
+0x12,
+0x49,
+0x65,
+0x6c,
+0x40,
+0x03,
+0xd0,
+0x49,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x97,
+0x6c,
+0x40,
+0x02,
+0xbc,
+0x02,
+0x24,
+0x16,
+0x56,
+0xc4,
+0x00,
+0x3d,
+0x04,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x05,
+0x26,
+0xc4,
+0x00,
+0x18,
+0x05,
+0x23,
+0x90,
+0x20,
+0x88,
+0x3a,
+0x08,
+0x84,
+0x89,
+0x5b,
+0x4a,
+0x12,
+0x00,
+0x04,
+0x80,
+0x00,
+0x95,
+0xb4,
+0xa1,
+0x98,
+0x08,
+0x29,
+0x80,
+0xc3,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x0b,
+0x30,
+0x8d,
+0x0b,
+0xc0,
+0x6c,
+0x5c,
+0x08,
+0xa0,
+0x82,
+0x60,
+0x52,
+0x49,
+0xe3,
+0xc0,
+0x5f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x24,
+0x13,
+0xc6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x83,
+0x81,
+0x1c,
+0x86,
+0x08,
+0x25,
+0xb4,
+0x41,
+0x08,
+0x10,
+0x35,
+0xb4,
+0x61,
+0x18,
+0x08,
+0xb9,
+0x80,
+0x82,
+0x30,
+0x9d,
+0x0b,
+0xc0,
+0xcc,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x02,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x52,
+0x48,
+0xa3,
+0xc0,
+0xcf,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x52,
+0x08,
+0xa3,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x66,
+0x00,
+0x00,
+0xb7,
+0xe0,
+0x88,
+0x22,
+0x08,
+0x83,
+0xa4,
+0x88,
+0x2a,
+0x5a,
+0x00,
+0x68,
+0x86,
+0xb4,
+0x88,
+0x81,
+0x60,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x40,
+0x00,
+0x00,
+0x60,
+0x89,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcf,
+0x25,
+0x66,
+0x00,
+0x00,
+0xb7,
+0xe0,
+0x5c,
+0x09,
+0x68,
+0x82,
+0x20,
+0x88,
+0x32,
+0x59,
+0x46,
+0xab,
+0x52,
+0xca,
+0xc0,
+0x81,
+0x24,
+0x86,
+0xb4,
+0x84,
+0x20,
+0x9c,
+0xa2,
+0x40,
+0x85,
+0xc0,
+0x89,
+0x14,
+0x02,
+0xd3,
+0x81,
+0x0c,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x98,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x03,
+0x84,
+0x08,
+0xb5,
+0x40,
+0xed,
+0x84,
+0x00,
+0xb6,
+0xc0,
+0x00,
+0x11,
+0x80,
+0x82,
+0x81,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x43,
+0x25,
+0x8a,
+0x8b,
+0xc0,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x7a,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x7a,
+0x38,
+0x13,
+0x42,
+0x59,
+0x28,
+0xbc,
+0x15,
+0x06,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x95,
+0xc0,
+0xb9,
+0x98,
+0xe8,
+0x26,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x85,
+0x24,
+0x76,
+0xac,
+0x02,
+0x06,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x06,
+0xc4,
+0x00,
+0x25,
+0x07,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0xa4,
+0x86,
+0xc4,
+0x00,
+0x18,
+0xa4,
+0x86,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x94,
+0x21,
+0xff,
+0x94,
+0x24,
+0x28,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x88,
+0x20,
+0x68,
+0x00,
+0x00,
+0x91,
+0x21,
+0x66,
+0x00,
+0x01,
+0x59,
+0xc0,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x32,
+0x06,
+0x8b,
+0xc1,
+0x61,
+0x68,
+0x20,
+0x01,
+0x00,
+0x21,
+0x68,
+0x20,
+0x00,
+0xad,
+0x20,
+0xa0,
+0xc2,
+0x28,
+0x81,
+0x62,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x40,
+0x00,
+0x00,
+0x81,
+0x22,
+0x68,
+0x20,
+0x00,
+0xc5,
+0x20,
+0x68,
+0x20,
+0x01,
+0x00,
+0x21,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x38,
+0x09,
+0x40,
+0x00,
+0x03,
+0xc2,
+0x0f,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x49,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x38,
+0x19,
+0xc2,
+0x89,
+0x2d,
+0x30,
+0x12,
+0x8b,
+0xc1,
+0x81,
+0x68,
+0x20,
+0x00,
+0xad,
+0x20,
+0x68,
+0x20,
+0x01,
+0x01,
+0x21,
+0x68,
+0x20,
+0x00,
+0xfe,
+0x22,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x68,
+0x20,
+0x00,
+0xfe,
+0x22,
+0x68,
+0x20,
+0x00,
+0xc5,
+0x20,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x48,
+0x40,
+0x00,
+0x02,
+0x10,
+0x61,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x09,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x40,
+0x55,
+0x3f,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x49,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x88,
+0x1b,
+0x6b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x60,
+0xab,
+0xfd,
+0x06,
+0x82,
+0x00,
+0x1d,
+0xa2,
+0xc6,
+0x82,
+0x00,
+0x1d,
+0xb2,
+0xd6,
+0x82,
+0x00,
+0x1d,
+0xc2,
+0x26,
+0x82,
+0x00,
+0x1f,
+0x32,
+0x38,
+0x82,
+0x76,
+0x68,
+0x00,
+0x01,
+0x44,
+0x20,
+0x68,
+0x20,
+0x01,
+0x8f,
+0x24,
+0x68,
+0x00,
+0x01,
+0x52,
+0x21,
+0x68,
+0x20,
+0x01,
+0xd8,
+0x25,
+0x88,
+0x06,
+0xc8,
+0x80,
+0xed,
+0x88,
+0x16,
+0x28,
+0x81,
+0xe3,
+0x66,
+0x00,
+0x00,
+0x99,
+0x48,
+0x6c,
+0x00,
+0x02,
+0x96,
+0x09,
+0x68,
+0x00,
+0x02,
+0x3b,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x68,
+0x00,
+0x01,
+0x44,
+0x24,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0x86,
+0x34,
+0x8a,
+0x80,
+0x30,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x09,
+0x58,
+0x0b,
+0x02,
+0xbf,
+0xe0,
+0xbc,
+0x7b,
+0x88,
+0x80,
+0x76,
+0x68,
+0x20,
+0x01,
+0xb1,
+0x24,
+0x68,
+0x00,
+0x01,
+0x9c,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x6c,
+0x00,
+0x01,
+0x16,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x64,
+0x48,
+0x68,
+0x20,
+0x01,
+0xb1,
+0x24,
+0x68,
+0x00,
+0x01,
+0xa0,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x6c,
+0x00,
+0x01,
+0x28,
+0x09,
+0x68,
+0x20,
+0x01,
+0x1b,
+0x20,
+0x68,
+0x20,
+0x01,
+0xf5,
+0x24,
+0x5c,
+0x81,
+0x02,
+0x05,
+0xc1,
+0x84,
+0x00,
+0x98,
+0x48,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x08,
+0x01,
+0x6c,
+0x00,
+0x01,
+0x1a,
+0x0a,
+0x30,
+0x02,
+0x8b,
+0xc4,
+0x3d,
+0x6c,
+0x00,
+0x03,
+0x72,
+0x48,
+0x5c,
+0x83,
+0x0a,
+0x20,
+0x01,
+0x80,
+0xa0,
+0x88,
+0x0a,
+0x8b,
+0x49,
+0xb8,
+0xe2,
+0x05,
+0x60,
+0x22,
+0xfd,
+0x42,
+0xa8,
+0x64,
+0x28,
+0x11,
+0x22,
+0x28,
+0x54,
+0x5b,
+0x40,
+0x00,
+0x48,
+0x48,
+0xa0,
+0x8c,
+0x18,
+0x0a,
+0x08,
+0x84,
+0x80,
+0x95,
+0x40,
+0xb1,
+0x18,
+0x00,
+0x05,
+0x17,
+0xeb,
+0xa0,
+0xc8,
+0x12,
+0xa8,
+0x7b,
+0x54,
+0x06,
+0x98,
+0x40,
+0x02,
+0x6c,
+0x40,
+0x02,
+0x36,
+0x7a,
+0x30,
+0x08,
+0x05,
+0x14,
+0x2f,
+0xbc,
+0x21,
+0xb8,
+0x08,
+0xcb,
+0x2e,
+0x16,
+0x43,
+0x69,
+0x00,
+0x98,
+0x00,
+0x83,
+0x00,
+0xa0,
+0xbc,
+0x1b,
+0x38,
+0x80,
+0xca,
+0x88,
+0x16,
+0x18,
+0x81,
+0xd1,
+0x66,
+0x00,
+0x02,
+0x4d,
+0x28,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x08,
+0x6e,
+0x00,
+0x03,
+0x4c,
+0x12,
+0x66,
+0x00,
+0x02,
+0x52,
+0xa8,
+0x55,
+0x00,
+0x0b,
+0x80,
+0x00,
+0x66,
+0x00,
+0x02,
+0x4f,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x09,
+0x57,
+0x4b,
+0x20,
+0x81,
+0x21,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x48,
+0x5c,
+0x81,
+0x02,
+0xc0,
+0x61,
+0x88,
+0x18,
+0x18,
+0x80,
+0x8a,
+0x68,
+0x20,
+0x01,
+0xf5,
+0x24,
+0x80,
+0xa5,
+0x18,
+0x08,
+0xd1,
+0x42,
+0x01,
+0x78,
+0x0a,
+0x4a,
+0x84,
+0x84,
+0xa3,
+0x90,
+0x61,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3e,
+0xa0,
+0x82,
+0xe8,
+0x64,
+0x82,
+0x24,
+0x80,
+0x00,
+0x00,
+0x86,
+0x00,
+0x85,
+0x78,
+0x32,
+0x06,
+0x18,
+0x95,
+0x74,
+0xd6,
+0x02,
+0x0c,
+0x88,
+0x22,
+0x48,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x23,
+0x60,
+0x85,
+0x50,
+0x32,
+0x06,
+0x00,
+0x92,
+0xf1,
+0xad,
+0x6c,
+0x40,
+0x02,
+0x36,
+0x48,
+0x40,
+0x00,
+0x00,
+0x60,
+0x49,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x1b,
+0x92,
+0x0b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x20,
+0xa2,
+0x28,
+0x46,
+0x40,
+0x00,
+0x13,
+0x26,
+0xfa,
+0x01,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x65,
+0x88,
+0x0f,
+0x66,
+0x60,
+0x00,
+0x13,
+0x34,
+0x8a,
+0x20,
+0x84,
+0x88,
+0x02,
+0x08,
+0x81,
+0x24,
+0x94,
+0x02,
+0x55,
+0x00,
+0xb0,
+0x08,
+0x0b,
+0x68,
+0x60,
+0x09,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x08,
+0x44,
+0x08,
+0x02,
+0x80,
+0x10,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0xab,
+0xfd,
+0x08,
+0x81,
+0xe4,
+0x88,
+0x04,
+0x98,
+0x80,
+0xe5,
+0x88,
+0x16,
+0x18,
+0x82,
+0x60,
+0x88,
+0x2f,
+0x66,
+0x82,
+0x00,
+0x0d,
+0x02,
+0x46,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x08,
+0x00,
+0x88,
+0x22,
+0x06,
+0xc4,
+0x00,
+0x1a,
+0xe0,
+0x95,
+0xd0,
+0xa3,
+0x04,
+0x00,
+0x95,
+0x70,
+0xb0,
+0x04,
+0x1c,
+0x85,
+0x18,
+0x5b,
+0x18,
+0x00,
+0x86,
+0x82,
+0x00,
+0x12,
+0xd2,
+0x15,
+0xb4,
+0x80,
+0x18,
+0x2a,
+0x89,
+0x80,
+0x0a,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x00,
+0x58,
+0x41,
+0x80,
+0x81,
+0xa4,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0xa2,
+0x08,
+0x36,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x28,
+0x84,
+0x21,
+0xa1,
+0x86,
+0x44,
+0x20,
+0x25,
+0x21,
+0x8a,
+0x29,
+0x48,
+0x3f,
+0x38,
+0x12,
+0x92,
+0x58,
+0x70,
+0xbc,
+0x77,
+0x08,
+0x83,
+0x26,
+0x88,
+0x0a,
+0x79,
+0x70,
+0x2c,
+0x59,
+0x03,
+0x00,
+0x81,
+0x25,
+0xbc,
+0x11,
+0x98,
+0x78,
+0x7a,
+0x00,
+0x00,
+0x08,
+0x68,
+0x88,
+0x30,
+0x16,
+0x0b,
+0xc0,
+0x7b,
+0x84,
+0x38,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x09,
+0x49,
+0x8e,
+0x88,
+0x86,
+0x8c,
+0x99,
+0x70,
+0x64,
+0xbc,
+0x05,
+0x73,
+0x01,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0x86,
+0x8c,
+0x99,
+0x70,
+0x64,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3d,
+0x20,
+0x88,
+0x59,
+0x89,
+0x30,
+0x12,
+0x8b,
+0xc2,
+0x48,
+0x39,
+0x02,
+0x06,
+0x82,
+0x00,
+0x1e,
+0xe2,
+0x75,
+0xc8,
+0x20,
+0x85,
+0xa8,
+0x95,
+0xc8,
+0x51,
+0x03,
+0x80,
+0xa5,
+0x70,
+0xd4,
+0x21,
+0x00,
+0x65,
+0x90,
+0x18,
+0x01,
+0x2d,
+0x00,
+0x00,
+0x00,
+0x83,
+0xb0,
+0x98,
+0x50,
+0x0a,
+0x57,
+0x0b,
+0xb0,
+0x78,
+0x09,
+0x85,
+0x04,
+0xab,
+0xc0,
+0x6c,
+0x85,
+0xa8,
+0xa3,
+0x01,
+0x70,
+0xbc,
+0x08,
+0x28,
+0x59,
+0xc8,
+0x86,
+0x14,
+0x9b,
+0xc0,
+0x57,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x34,
+0x85,
+0x9c,
+0x88,
+0x61,
+0x49,
+0x00,
+0x00,
+0x08,
+0x50,
+0x09,
+0x57,
+0x0a,
+0x80,
+0x32,
+0x09,
+0x57,
+0x0a,
+0x00,
+0x83,
+0xa4,
+0xbc,
+0x03,
+0xf8,
+0x70,
+0x40,
+0x88,
+0x3a,
+0x40,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x60,
+0x09,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0xb3,
+0x6c,
+0x40,
+0x02,
+0x30,
+0x08,
+0x28,
+0x12,
+0xd3,
+0x08,
+0xa8,
+0xbc,
+0x03,
+0x4b,
+0xc0,
+0x6f,
+0x5c,
+0x0b,
+0xe8,
+0x60,
+0x49,
+0xbc,
+0x03,
+0xf5,
+0xc0,
+0xbe,
+0x86,
+0x05,
+0x23,
+0x81,
+0x7d,
+0x25,
+0x97,
+0x8b,
+0xc2,
+0x01,
+0x38,
+0x16,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x0f,
+0x93,
+0x81,
+0x75,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x49,
+0x98,
+0xe8,
+0x98,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x38,
+0x17,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x13,
+0x08,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x03,
+0x94,
+0x87,
+0x5b,
+0xc0,
+0xe7,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x49,
+0x38,
+0x16,
+0x58,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x62,
+0x38,
+0x17,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x05,
+0x08,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x25,
+0x24,
+0x17,
+0x89,
+0x48,
+0x70,
+0xbc,
+0x42,
+0xf8,
+0x80,
+0xa2,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x6c,
+0x88,
+0x12,
+0x55,
+0x70,
+0x14,
+0x08,
+0x32,
+0x3b,
+0xc0,
+0x5f,
+0x40,
+0x00,
+0x00,
+0x68,
+0xc0,
+0x54,
+0x0a,
+0x00,
+0x83,
+0x23,
+0x86,
+0x8c,
+0x0b,
+0x00,
+0x0d,
+0x88,
+0x0a,
+0x29,
+0x58,
+0x65,
+0x00,
+0x00,
+0x08,
+0x50,
+0x09,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0x23,
+0x2a,
+0x06,
+0xd8,
+0x50,
+0x49,
+0x68,
+0x20,
+0x01,
+0xeb,
+0x26,
+0x5c,
+0x81,
+0x0b,
+0x01,
+0x7d,
+0x52,
+0xcb,
+0xc0,
+0x30,
+0x88,
+0x5c,
+0x82,
+0x00,
+0x70,
+0x00,
+0x82,
+0x0c,
+0x88,
+0x60,
+0x50,
+0x68,
+0x20,
+0x01,
+0x19,
+0x23,
+0x87,
+0x08,
+0xb8,
+0x83,
+0xa7,
+0x81,
+0xa0,
+0x04,
+0x20,
+0x8c,
+0x06,
+0x0c,
+0xb5,
+0xc0,
+0x02,
+0x07,
+0x85,
+0x02,
+0x41,
+0x64,
+0x5d,
+0x48,
+0x20,
+0x43,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x6c,
+0x5c,
+0x0b,
+0xa9,
+0x48,
+0x74,
+0x52,
+0x4b,
+0x03,
+0xc0,
+0x7f,
+0x5c,
+0x09,
+0x29,
+0x48,
+0x70,
+0x52,
+0x0b,
+0x03,
+0xc0,
+0x3f,
+0x5c,
+0x09,
+0x29,
+0x48,
+0x70,
+0x38,
+0x12,
+0x52,
+0x59,
+0x70,
+0xbc,
+0x08,
+0x88,
+0x58,
+0x52,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2a,
+0xe0,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x04,
+0x96,
+0xc4,
+0x00,
+0x18,
+0x04,
+0x99,
+0x88,
+0x09,
+0x68,
+0x00,
+0x00,
+0x84,
+0x08,
+0x58,
+0x09,
+0x40,
+0x68,
+0x89,
+0x84,
+0x1c,
+0x9b,
+0xc1,
+0x48,
+0x85,
+0x00,
+0x99,
+0x88,
+0x08,
+0x68,
+0x00,
+0x00,
+0x8d,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x98,
+0xea,
+0x16,
+0x60,
+0x00,
+0x0b,
+0x8a,
+0x08,
+0x82,
+0x20,
+0xbc,
+0x10,
+0xf8,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xb5,
+0x21,
+0x66,
+0x00,
+0x00,
+0xb8,
+0xa0,
+0x88,
+0x22,
+0x0b,
+0xc0,
+0x9f,
+0x40,
+0x00,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xae,
+0x21,
+0x66,
+0x00,
+0x00,
+0xb8,
+0xa0,
+0x88,
+0x22,
+0x00,
+0x00,
+0x00,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x56,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x51,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x38,
+0x12,
+0xc2,
+0x59,
+0x28,
+0xbc,
+0x05,
+0x08,
+0x84,
+0xa1,
+0x40,
+0x00,
+0x00,
+0x40,
+0x09,
+0x42,
+0x29,
+0xb8,
+0x48,
+0x49,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0xb0,
+0x84,
+0x18,
+0x88,
+0x40,
+0x0a,
+0x57,
+0x0d,
+0x00,
+0x84,
+0xa1,
+0x98,
+0x00,
+0xa3,
+0x69,
+0x80,
+0x98,
+0x00,
+0xa3,
+0x01,
+0x70,
+0xbc,
+0x03,
+0x58,
+0x48,
+0x48,
+0xbc,
+0x01,
+0x78,
+0x84,
+0xa1,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x09,
+0x84,
+0x80,
+0x85,
+0x40,
+0xb0,
+0x04,
+0x00,
+0xa9,
+0x80,
+0x00,
+0x30,
+0x03,
+0x0b,
+0xc0,
+0x2d,
+0x88,
+0x0c,
+0x88,
+0x40,
+0x50,
+0x2e,
+0x16,
+0x08,
+0x40,
+0x09,
+0x98,
+0x00,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x01,
+0x38,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x88,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x56,
+0x08,
+0x68,
+0x00,
+0x2b,
+0xf2,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0xe4,
+0x68,
+0x00,
+0x27,
+0x10,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0x84,
+0x68,
+0x00,
+0x18,
+0x6a,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0x24,
+0xbc,
+0x08,
+0xf2,
+0x28,
+0x6d,
+0xbc,
+0x06,
+0xf2,
+0x28,
+0xad,
+0xbc,
+0x04,
+0xf2,
+0x28,
+0xed,
+0xbc,
+0x02,
+0xf2,
+0x29,
+0x2d,
+0x22,
+0x96,
+0xd8,
+0x84,
+0xa1,
+0x88,
+0x22,
+0x08,
+0x48,
+0x08,
+0x84,
+0x00,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x03,
+0x38,
+0x80,
+0x88,
+0x2e,
+0x16,
+0x08,
+0x48,
+0x40,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x84,
+0x80,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x04,
+0xd8,
+0x80,
+0x88,
+0x28,
+0x16,
+0x08,
+0x48,
+0x40,
+0x00,
+0x00,
+0x08,
+0x82,
+0xb6,
+0x84,
+0x00,
+0x84,
+0x60,
+0xa4,
+0x08,
+0x00,
+0x95,
+0x70,
+0x94,
+0x28,
+0x03,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x09,
+0xe2,
+0x22,
+0xe4,
+0x86,
+0x00,
+0xa4,
+0x43,
+0x00,
+0x16,
+0x9a,
+0x55,
+0x00,
+0xa0,
+0x08,
+0x02,
+0x56,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xd5,
+0x2c,
+0x94,
+0x06,
+0x80,
+0x94,
+0x20,
+0x24,
+0x18,
+0x00,
+0x80,
+0x81,
+0x00,
+0xba,
+0x14,
+0x89,
+0x80,
+0x08,
+0x00,
+0x00,
+0x08,
+0x40,
+0x89,
+0x46,
+0x0a,
+0x40,
+0x68,
+0x88,
+0x08,
+0x48,
+0x09,
+0x80,
+0x08,
+0xab,
+0xfd,
+0x0a,
+0x21,
+0x64,
+0xa0,
+0x88,
+0x18,
+0x80,
+0x64,
+0xa2,
+0x06,
+0x48,
+0x80,
+0xe1,
+0xa0,
+0x84,
+0x18,
+0x81,
+0x64,
+0x88,
+0x1c,
+0x98,
+0x82,
+0x60,
+0x88,
+0x2f,
+0x66,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x08,
+0x00,
+0x88,
+0x12,
+0x08,
+0x82,
+0x21,
+0xa0,
+0x06,
+0x08,
+0x40,
+0x0a,
+0x88,
+0x18,
+0x94,
+0x43,
+0x00,
+0x04,
+0xa4,
+0x8a,
+0x06,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb5,
+0xe8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x09,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x59,
+0x03,
+0x40,
+0x80,
+0xa4,
+0xbc,
+0x15,
+0x8a,
+0x20,
+0x20,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x5c,
+0x09,
+0x20,
+0x80,
+0xc8,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xc8,
+0x88,
+0x02,
+0x46,
+0xc4,
+0x00,
+0x2a,
+0xc0,
+0x96,
+0x60,
+0x00,
+0x15,
+0x82,
+0x88,
+0x60,
+0x8a,
+0x55,
+0x01,
+0x28,
+0x80,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0xc9,
+0xbc,
+0x0f,
+0xf8,
+0x80,
+0x88,
+0xbc,
+0x0d,
+0xf8,
+0x80,
+0x88,
+0x86,
+0x08,
+0x93,
+0x69,
+0x40,
+0x98,
+0x00,
+0x96,
+0xc4,
+0x00,
+0x21,
+0x00,
+0xa3,
+0x09,
+0x70,
+0xbc,
+0x05,
+0x58,
+0x80,
+0x20,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x09,
+0x84,
+0x0c,
+0x90,
+0x00,
+0x00,
+0x88,
+0x2b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2c,
+0x38,
+0x12,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x09,
+0x08,
+0x63,
+0x88,
+0x6c,
+0x40,
+0x03,
+0xd2,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x2e,
+0x08,
+0x08,
+0x40,
+0x09,
+0x80,
+0x09,
+0xba,
+0x14,
+0x89,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x80,
+0xa2,
+0x16,
+0x48,
+0x20,
+0x0a,
+0x84,
+0x00,
+0x84,
+0x41,
+0x00,
+0x02,
+0x00,
+0xa8,
+0x40,
+0x88,
+0x44,
+0x14,
+0x00,
+0x60,
+0x08,
+0xba,
+0x14,
+0x84,
+0x42,
+0x40,
+0x04,
+0x04,
+0x99,
+0x80,
+0x08,
+0xa2,
+0x24,
+0x48,
+0x60,
+0x08,
+0x44,
+0x20,
+0x00,
+0x42,
+0xc9,
+0x98,
+0x00,
+0x98,
+0x60,
+0x88,
+0x44,
+0x20,
+0x01,
+0x68,
+0xa5,
+0xba,
+0x14,
+0x82,
+0x01,
+0x40,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x5c,
+0x83,
+0x08,
+0x43,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x0a,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x00,
+0x58,
+0x0c,
+0x02,
+0xbf,
+0xd0,
+0x80,
+0x08,
+0xba,
+0x08,
+0x42,
+0x84,
+0x00,
+0x19,
+0x88,
+0x42,
+0x88,
+0x0c,
+0x88,
+0x81,
+0x49,
+0x42,
+0x13,
+0x40,
+0x80,
+0x4a,
+0x88,
+0x1c,
+0xb3,
+0x20,
+0x10,
+0xbc,
+0x21,
+0x06,
+0x82,
+0x00,
+0x11,
+0xb2,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x5c,
+0x38,
+0x40,
+0x08,
+0x85,
+0x80,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x10,
+0xd8,
+0x4a,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x48,
+0x8a,
+0x84,
+0x80,
+0x32,
+0x80,
+0xf4,
+0x22,
+0xfe,
+0x75,
+0x54,
+0x3c,
+0x98,
+0x44,
+0xb1,
+0x51,
+0x73,
+0x51,
+0x42,
+0x60,
+0x10,
+0x53,
+0x81,
+0x2c,
+0x84,
+0x20,
+0x1f,
+0x85,
+0x04,
+0x95,
+0x50,
+0x1c,
+0x85,
+0x0c,
+0x93,
+0x90,
+0x20,
+0x00,
+0x00,
+0x08,
+0x48,
+0x88,
+0x57,
+0x4b,
+0x30,
+0x48,
+0x08,
+0x57,
+0x8b,
+0x20,
+0x48,
+0xca,
+0xbc,
+0x04,
+0xf8,
+0x48,
+0x48,
+0xbc,
+0x02,
+0xf3,
+0x90,
+0x20,
+0x39,
+0x02,
+0x03,
+0x81,
+0x25,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2c,
+0x25,
+0x96,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xa0,
+0xab,
+0xc0,
+0x59,
+0x55,
+0x01,
+0xa0,
+0x82,
+0x4a,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x24,
+0xe0,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x56,
+0x09,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x88,
+0x88,
+0x10,
+0xab,
+0x1f,
+0x47,
+0x30,
+0x1f,
+0x0b,
+0xc1,
+0x2c,
+0x98,
+0xe8,
+0x88,
+0x82,
+0x08,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xe7,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xae,
+0x38,
+0x13,
+0x72,
+0x59,
+0xf0,
+0xbc,
+0x09,
+0x16,
+0xc4,
+0x00,
+0x15,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x2b,
+0xc0,
+0xb3,
+0x01,
+0xf0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x23,
+0x6c,
+0x40,
+0x02,
+0x4e,
+0x08,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x26,
+0x59,
+0x03,
+0x80,
+0x81,
+0x8a,
+0x28,
+0x13,
+0x35,
+0x70,
+0x99,
+0x98,
+0x0c,
+0x8b,
+0xc0,
+0xf8,
+0x98,
+0x0c,
+0xb6,
+0xc4,
+0x00,
+0x25,
+0x00,
+0x33,
+0x20,
+0x18,
+0xbc,
+0x0a,
+0x13,
+0x01,
+0x88,
+0xbc,
+0x06,
+0x45,
+0x80,
+0xc4,
+0x08,
+0x18,
+0x8b,
+0xc0,
+0xfb,
+0x88,
+0x00,
+0x19,
+0x82,
+0xc8,
+0xbc,
+0x0c,
+0x7b,
+0xc0,
+0xbf,
+0x88,
+0x00,
+0x18,
+0x80,
+0x81,
+0x32,
+0x00,
+0x8b,
+0xc0,
+0x64,
+0x32,
+0x00,
+0x8b,
+0xc0,
+0x5b,
+0x55,
+0x01,
+0xa0,
+0x80,
+0x01,
+0x98,
+0x2c,
+0x8b,
+0xc0,
+0x17,
+0x88,
+0x00,
+0x13,
+0x00,
+0x40,
+0xbc,
+0x44,
+0x03,
+0x20,
+0x10,
+0xbc,
+0x42,
+0x06,
+0xc0,
+0x00,
+0x36,
+0x80,
+0xa3,
+0x20,
+0x30,
+0xbc,
+0x05,
+0x98,
+0x82,
+0x0a,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc3,
+0x90,
+0x30,
+0x17,
+0x0b,
+0xc3,
+0x70,
+0x68,
+0x20,
+0x01,
+0x0b,
+0x20,
+0x84,
+0x90,
+0xa8,
+0x00,
+0x00,
+0x30,
+0x03,
+0x0b,
+0xc0,
+0x22,
+0xbc,
+0x04,
+0xf8,
+0x4a,
+0xfa,
+0x84,
+0xa8,
+0xa2,
+0xa0,
+0x76,
+0x84,
+0xac,
+0xa0,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa6,
+0xc0,
+0x00,
+0x36,
+0x60,
+0x03,
+0x01,
+0x80,
+0xbc,
+0x04,
+0x56,
+0xc0,
+0x00,
+0x37,
+0x40,
+0x03,
+0x01,
+0x80,
+0xbc,
+0x03,
+0x4b,
+0xc2,
+0x1f,
+0x40,
+0x00,
+0x00,
+0x4a,
+0x08,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x5c,
+0x82,
+0x08,
+0x08,
+0x7a,
+0x80,
+0x07,
+0xaa,
+0x08,
+0x02,
+0x80,
+0x87,
+0xaa,
+0x00,
+0x04,
+0x80,
+0x07,
+0xa8,
+0x50,
+0xfa,
+0x86,
+0x0f,
+0xa8,
+0x48,
+0xfa,
+0x68,
+0x00,
+0x01,
+0xb9,
+0x23,
+0x84,
+0x0f,
+0xa6,
+0x80,
+0x00,
+0x1b,
+0x22,
+0x16,
+0x82,
+0x00,
+0x1f,
+0x12,
+0x08,
+0x08,
+0x80,
+0x85,
+0x80,
+0xa6,
+0xc4,
+0x00,
+0x20,
+0xa4,
+0x96,
+0xc0,
+0x00,
+0x14,
+0xc7,
+0xa8,
+0x48,
+0x7a,
+0x85,
+0x97,
+0xa8,
+0x40,
+0x50,
+0x84,
+0x0c,
+0xab,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x85,
+0x00,
+0x88,
+0x40,
+0x09,
+0x57,
+0x8b,
+0x22,
+0xbf,
+0xd0,
+0x85,
+0x04,
+0x88,
+0x80,
+0x63,
+0x88,
+0x0e,
+0x28,
+0x81,
+0x61,
+0x88,
+0x1e,
+0x08,
+0x82,
+0x76,
+0x00,
+0x00,
+0x08,
+0x40,
+0x09,
+0x36,
+0x14,
+0x52,
+0xe9,
+0x64,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x85,
+0x80,
+0x88,
+0x48,
+0x09,
+0x2f,
+0x16,
+0x48,
+0x58,
+0x48,
+0x00,
+0x00,
+0x08,
+0x48,
+0x09,
+0x36,
+0x14,
+0x52,
+0xe9,
+0x64,
+0x44,
+0x00,
+0x00,
+0x58,
+0x48,
+0x98,
+0x00,
+0x98,
+0x50,
+0x08,
+0x44,
+0x00,
+0x00,
+0x48,
+0x08,
+0x44,
+0x01,
+0x00,
+0x40,
+0x08,
+0x44,
+0x01,
+0x01,
+0xa0,
+0x00,
+0x44,
+0x40,
+0x01,
+0x80,
+0x88,
+0x08,
+0x18,
+0x06,
+0x60,
+0x00,
+0x13,
+0x52,
+0x89,
+0x81,
+0x09,
+0x88,
+0x12,
+0x08,
+0x81,
+0xa1,
+0x84,
+0x00,
+0x98,
+0x48,
+0x0a,
+0x44,
+0x48,
+0x00,
+0x80,
+0xa0,
+0x88,
+0x02,
+0x19,
+0x80,
+0x0b,
+0x84,
+0x00,
+0x94,
+0x46,
+0x80,
+0x04,
+0x80,
+0xa4,
+0x47,
+0x10,
+0x18,
+0x00,
+0xb4,
+0x46,
+0x00,
+0x18,
+0x28,
+0xb5,
+0x11,
+0xe0,
+0x18,
+0x08,
+0xa4,
+0x44,
+0x10,
+0x18,
+0x00,
+0x83,
+0x20,
+0x20,
+0x51,
+0x1e,
+0x83,
+0xc0,
+0x29,
+0x98,
+0x00,
+0xa9,
+0x82,
+0x48,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x11,
+0x98,
+0x2c,
+0xa3,
+0x69,
+0x40,
+0x5b,
+0x48,
+0x01,
+0x80,
+0x09,
+0x98,
+0x00,
+0x03,
+0x00,
+0x28,
+0xbc,
+0x06,
+0x43,
+0x69,
+0xc0,
+0x5b,
+0x4c,
+0x01,
+0x80,
+0x09,
+0x98,
+0x00,
+0xb3,
+0x01,
+0xe8,
+0xbc,
+0x04,
+0x58,
+0x40,
+0x48,
+0x42,
+0x01,
+0x7b,
+0x00,
+0x0c,
+0x84,
+0x84,
+0xa9,
+0x8e,
+0x88,
+0x00,
+0x00,
+0x08,
+0x82,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x46,
+0x08,
+0x8a,
+0xbf,
+0xe0,
+0x51,
+0x61,
+0xb0,
+0x80,
+0x49,
+0x51,
+0x61,
+0xe0,
+0x80,
+0xf6,
+0x1a,
+0x0a,
+0x29,
+0x83,
+0x09,
+0x98,
+0x38,
+0x82,
+0x81,
+0x2d,
+0x88,
+0x14,
+0x96,
+0x60,
+0x00,
+0x13,
+0x52,
+0x8b,
+0xa1,
+0x01,
+0x88,
+0x10,
+0x94,
+0x40,
+0x80,
+0x08,
+0x00,
+0x99,
+0x80,
+0x08,
+0x23,
+0x42,
+0x43,
+0x01,
+0x60,
+0xbc,
+0x02,
+0x5b,
+0xc0,
+0x2f,
+0x2e,
+0x16,
+0x49,
+0x8e,
+0x88,
+0x88,
+0x0b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x00,
+0x01,
+0x3a,
+0x22,
+0x55,
+0x03,
+0x20,
+0x00,
+0x7a,
+0x5d,
+0x08,
+0x20,
+0x08,
+0x7a,
+0x32,
+0x2a,
+0x0b,
+0xff,
+0xaa,
+0x81,
+0x07,
+0xab,
+0xa1,
+0x40,
+0x5c,
+0x81,
+0x00,
+0x40,
+0x08,
+0x82,
+0x00,
+0xa4,
+0x41,
+0x00,
+0x04,
+0x80,
+0xa8,
+0x20,
+0x08,
+0x44,
+0x44,
+0x00,
+0x60,
+0x08,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x0a,
+0x08,
+0x80,
+0x09,
+0x80,
+0x08,
+0x68,
+0x00,
+0x00,
+0x6c,
+0x20,
+0x68,
+0x20,
+0x00,
+0x24,
+0x24,
+0x5c,
+0x81,
+0x02,
+0xc0,
+0x41,
+0x68,
+0x20,
+0x00,
+0x4b,
+0x21,
+0x5c,
+0x08,
+0xad,
+0x00,
+0x81,
+0x44,
+0x10,
+0x00,
+0x08,
+0x88,
+0x84,
+0x00,
+0xb4,
+0x46,
+0x08,
+0x2b,
+0xf7,
+0x06,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xe6,
+0xe4,
+0x00,
+0x11,
+0x22,
+0x75,
+0x2c,
+0xb8,
+0x08,
+0x16,
+0x18,
+0x81,
+0xe4,
+0x88,
+0x27,
+0x68,
+0x82,
+0xd5,
+0x90,
+0x45,
+0x84,
+0x22,
+0x3c,
+0x88,
+0x55,
+0x49,
+0x03,
+0x59,
+0x68,
+0x20,
+0x00,
+0x85,
+0x24,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x0a,
+0x66,
+0x00,
+0x00,
+0xe4,
+0xc8,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x09,
+0x68,
+0x20,
+0x00,
+0x85,
+0x20,
+0x5c,
+0x82,
+0x40,
+0x88,
+0x48,
+0xa0,
+0x08,
+0xc6,
+0xc0,
+0x00,
+0x12,
+0xa0,
+0x96,
+0xc0,
+0x00,
+0x11,
+0x80,
+0xa6,
+0x60,
+0x00,
+0x0e,
+0x4c,
+0x89,
+0x62,
+0x07,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x5c,
+0x08,
+0xe8,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x90,
+0x2c,
+0x52,
+0xcb,
+0x80,
+0x85,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x88,
+0x06,
+0xcb,
+0xc1,
+0xa9,
+0x40,
+0x00,
+0x00,
+0x88,
+0x09,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x0a,
+0x44,
+0x10,
+0x00,
+0x85,
+0x89,
+0x68,
+0x20,
+0x00,
+0x98,
+0xa0,
+0x88,
+0x84,
+0x88,
+0x80,
+0x60,
+0x68,
+0x20,
+0x00,
+0x92,
+0xa4,
+0x68,
+0x20,
+0x00,
+0x95,
+0xa5,
+0x90,
+0x65,
+0x88,
+0x85,
+0xd4,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x40,
+0x00,
+0x03,
+0xc5,
+0x4f,
+0x44,
+0x10,
+0x00,
+0x88,
+0xc8,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x08,
+0x44,
+0x20,
+0x80,
+0x85,
+0x88,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x42,
+0x25,
+0x79,
+0x06,
+0x59,
+0x44,
+0x10,
+0x00,
+0x85,
+0xd5,
+0x68,
+0x00,
+0x00,
+0x6e,
+0x20,
+0x68,
+0x20,
+0x00,
+0x85,
+0x24,
+0x80,
+0x00,
+0x98,
+0x85,
+0xe0,
+0x66,
+0x00,
+0x00,
+0xe4,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x40,
+0x0a,
+0x5c,
+0x82,
+0x40,
+0x85,
+0xa0,
+0x68,
+0x20,
+0x00,
+0x85,
+0x21,
+0x88,
+0x84,
+0x8a,
+0x08,
+0x8c,
+0x84,
+0x00,
+0x96,
+0xc0,
+0x00,
+0x0d,
+0xc0,
+0xa6,
+0x60,
+0x00,
+0x0e,
+0x4c,
+0x89,
+0x62,
+0x07,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x5c,
+0x08,
+0xe8,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x90,
+0x2c,
+0x52,
+0xcb,
+0x80,
+0x85,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x88,
+0x06,
+0xcb,
+0xc1,
+0xa9,
+0x40,
+0x00,
+0x00,
+0x88,
+0x09,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x0a,
+0x44,
+0x10,
+0x00,
+0x85,
+0x89,
+0x68,
+0x20,
+0x00,
+0x98,
+0xa0,
+0x88,
+0x84,
+0x88,
+0x80,
+0x60,
+0x68,
+0x20,
+0x00,
+0x92,
+0xa4,
+0x68,
+0x20,
+0x00,
+0x95,
+0xa5,
+0x90,
+0x65,
+0x88,
+0x85,
+0xd4,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xcf,
+0x44,
+0x10,
+0x00,
+0x88,
+0xc8,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x08,
+0x44,
+0x20,
+0x80,
+0x85,
+0x88,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x44,
+0x10,
+0x01,
+0x06,
+0x59,
+0x40,
+0x00,
+0x00,
+0x85,
+0xd5,
+0x68,
+0x00,
+0x00,
+0x50,
+0x20,
+0x90,
+0x61,
+0x1a,
+0x04,
+0x21,
+0x88,
+0x59,
+0x58,
+0x85,
+0xe1,
+0x90,
+0x65,
+0x88,
+0x87,
+0x54,
+0x66,
+0x00,
+0x01,
+0x31,
+0x28,
+0x5c,
+0x00,
+0x71,
+0x80,
+0x49,
+0x90,
+0x61,
+0x06,
+0x80,
+0x00,
+0x06,
+0x42,
+0x08,
+0x87,
+0x14,
+0xa0,
+0x42,
+0x18,
+0x86,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x86,
+0x61,
+0x66,
+0x00,
+0x01,
+0x31,
+0x28,
+0x5c,
+0x00,
+0x71,
+0x80,
+0x09,
+0x5c,
+0x82,
+0x10,
+0x86,
+0x20,
+0x5c,
+0x8c,
+0x01,
+0x03,
+0x10,
+0x5c,
+0x8e,
+0x09,
+0x04,
+0x12,
+0x88,
+0x12,
+0x18,
+0x85,
+0xa5,
+0xa0,
+0x56,
+0x48,
+0x82,
+0x94,
+0x54,
+0x08,
+0x18,
+0x86,
+0x8a,
+0x88,
+0x51,
+0x65,
+0x40,
+0xc8,
+0x88,
+0x1a,
+0x28,
+0x48,
+0x08,
+0xa2,
+0xd6,
+0x08,
+0x21,
+0x0a,
+0x4d,
+0x86,
+0x80,
+0x50,
+0x08,
+0x80,
+0x10,
+0xa4,
+0xf8,
+0x48,
+0x02,
+0x04,
+0x38,
+0x00,
+0x41,
+0x82,
+0x2c,
+0x08,
+0x02,
+0xc2,
+0x88,
+0x16,
+0x46,
+0x82,
+0x00,
+0x07,
+0xe2,
+0xc8,
+0x81,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x1c,
+0x21,
+0x68,
+0x20,
+0x00,
+0x24,
+0x24,
+0x68,
+0x20,
+0x00,
+0x72,
+0x25,
+0x40,
+0x00,
+0x00,
+0x80,
+0x6c,
+0x68,
+0x00,
+0x01,
+0x27,
+0x22,
+0x66,
+0x00,
+0x00,
+0xe0,
+0x40,
+0x68,
+0x20,
+0x00,
+0x7f,
+0xac,
+0x88,
+0x84,
+0x86,
+0x80,
+0x00,
+0x03,
+0x02,
+0x18,
+0x81,
+0x20,
+0x68,
+0x20,
+0x00,
+0x4b,
+0x24,
+0x68,
+0x20,
+0x00,
+0x78,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x80,
+0x00,
+0x12,
+0xa2,
+0x26,
+0x60,
+0x00,
+0x0e,
+0x04,
+0x08,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x83,
+0x20,
+0x68,
+0x20,
+0x00,
+0x84,
+0x21,
+0xa4,
+0x22,
+0x36,
+0x60,
+0x00,
+0x0c,
+0x52,
+0x8a,
+0x42,
+0x02,
+0x5c,
+0x8d,
+0x00,
+0x81,
+0xa0,
+0x5c,
+0x08,
+0x71,
+0x8e,
+0x88,
+0xa0,
+0x26,
+0x09,
+0x42,
+0x0f,
+0x25,
+0x9b,
+0x8b,
+0xc0,
+0x39,
+0x55,
+0x01,
+0x28,
+0x81,
+0x21,
+0x88,
+0x80,
+0x98,
+0x40,
+0x0b,
+0x54,
+0x0f,
+0x42,
+0x0a,
+0x61,
+0x94,
+0xa0,
+0xd2,
+0x59,
+0xa8,
+0xbc,
+0x03,
+0x98,
+0x40,
+0xc0,
+0x00,
+0x00,
+0x08,
+0x88,
+0x88,
+0x00,
+0x00,
+0x08,
+0x48,
+0x0a,
+0x54,
+0x0d,
+0x00,
+0x82,
+0x36,
+0xba,
+0x14,
+0x88,
+0x48,
+0xc0,
+0x40,
+0x00,
+0x02,
+0x80,
+0x90,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xc0,
+0x5c,
+0x8f,
+0x08,
+0x81,
+0x60,
+0xa0,
+0x04,
+0x3a,
+0x08,
+0x00,
+0x81,
+0xa0,
+0x9a,
+0x1a,
+0x21,
+0x85,
+0x80,
+0x85,
+0x70,
+0x94,
+0x00,
+0xa8,
+0x95,
+0x70,
+0xa0,
+0x08,
+0x1e,
+0x1a,
+0x10,
+0x01,
+0x81,
+0x00,
+0x99,
+0x80,
+0x08,
+0x54,
+0x0b,
+0x02,
+0x28,
+0x26,
+0x98,
+0x00,
+0x98,
+0x49,
+0x49,
+0xa2,
+0x80,
+0x18,
+0x84,
+0x25,
+0x85,
+0x04,
+0x8a,
+0x30,
+0x23,
+0xa3,
+0x00,
+0x2a,
+0x18,
+0x26,
+0xa2,
+0x06,
+0x48,
+0x80,
+0x66,
+0x88,
+0x26,
+0x68,
+0x82,
+0xe4,
+0x88,
+0x36,
+0x08,
+0x83,
+0xf6,
+0x66,
+0x00,
+0x00,
+0xec,
+0x60,
+0x5c,
+0x8f,
+0x00,
+0x81,
+0xa0,
+0x5c,
+0x08,
+0x28,
+0x82,
+0xa4,
+0xa0,
+0x20,
+0x59,
+0x6a,
+0x0f,
+0x52,
+0xcb,
+0xc0,
+0x40,
+0x48,
+0xbc,
+0x05,
+0x95,
+0xc0,
+0x03,
+0x08,
+0x32,
+0x08,
+0x81,
+0x22,
+0x00,
+0x00,
+0x08,
+0x51,
+0x8a,
+0x86,
+0x80,
+0x95,
+0x40,
+0xb8,
+0x20,
+0x14,
+0x09,
+0x80,
+0x09,
+0xa2,
+0x2e,
+0x48,
+0x68,
+0xc9,
+0x88,
+0x16,
+0x48,
+0x81,
+0xe0,
+0x66,
+0x00,
+0x00,
+0xf3,
+0xa0,
+0x88,
+0x22,
+0x58,
+0x84,
+0x22,
+0x88,
+0x12,
+0x48,
+0x81,
+0xa0,
+0xa2,
+0x82,
+0x18,
+0x83,
+0xb6,
+0xa1,
+0x02,
+0x5a,
+0x80,
+0x40,
+0xa0,
+0x04,
+0x0a,
+0x20,
+0x64,
+0x64,
+0x00,
+0x00,
+0xf4,
+0x8f,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x22,
+0x86,
+0x00,
+0x84,
+0x42,
+0x00,
+0x06,
+0x08,
+0x95,
+0x00,
+0xe0,
+0x3a,
+0x14,
+0x80,
+0x89,
+0x80,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x6e,
+0x40,
+0x00,
+0x08,
+0x3c,
+0x38,
+0x15,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x30,
+0x05,
+0x15,
+0x96,
+0x16,
+0x1b,
+0x65,
+0x19,
+0x5b,
+0x08,
+0x02,
+0x04,
+0x41,
+0x00,
+0x18,
+0xe8,
+0xa9,
+0x80,
+0x08,
+0x23,
+0x07,
+0x79,
+0x82,
+0xe8,
+0x9c,
+0x00,
+0x19,
+0x48,
+0x37,
+0x30,
+0x1e,
+0x0b,
+0xc0,
+0x42,
+0x2a,
+0x07,
+0x63,
+0xb1,
+0x86,
+0x32,
+0x17,
+0x0b,
+0xff,
+0x62,
+0x32,
+0x0f,
+0x0b,
+0xc1,
+0xd0,
+0x51,
+0x83,
+0xab,
+0x01,
+0x00,
+0x50,
+0x41,
+0x11,
+0x82,
+0x68,
+0x55,
+0x00,
+0xb1,
+0xe0,
+0x04,
+0x46,
+0x08,
+0x89,
+0x60,
+0x35,
+0x50,
+0x41,
+0x49,
+0x83,
+0x89,
+0x55,
+0x00,
+0x79,
+0x83,
+0x48,
+0x44,
+0x41,
+0x09,
+0xe8,
+0x05,
+0x44,
+0x6d,
+0x49,
+0x68,
+0x36,
+0x21,
+0x01,
+0x00,
+0x84,
+0x82,
+0x37,
+0x80,
+0x02,
+0x28,
+0x80,
+0x6c,
+0x40,
+0x01,
+0x36,
+0x09,
+0x51,
+0x87,
+0xb1,
+0x83,
+0x08,
+0x36,
+0x14,
+0x72,
+0x81,
+0xa4,
+0x2f,
+0x16,
+0x52,
+0xe9,
+0xed,
+0x23,
+0x32,
+0xdb,
+0xa1,
+0x48,
+0x46,
+0x08,
+0x09,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x8a,
+0x68,
+0x00,
+0x00,
+0x1c,
+0x20,
+0x38,
+0x0a,
+0x42,
+0xa0,
+0x76,
+0x3a,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x8a,
+0x68,
+0x00,
+0x00,
+0x30,
+0x20,
+0x38,
+0x0a,
+0x42,
+0xa0,
+0x76,
+0x3a,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x44,
+0x20,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xf0,
+0x68,
+0x00,
+0x00,
+0x58,
+0x21,
+0x88,
+0x07,
+0x69,
+0x8e,
+0x88,
+0x40,
+0x00,
+0x02,
+0x00,
+0x02,
+0x55,
+0x03,
+0x20,
+0x08,
+0x7a,
+0x3a,
+0x10,
+0x43,
+0x22,
+0x20,
+0xbf,
+0xfb,
+0xa8,
+0x10,
+0x7a,
+0x40,
+0x00,
+0x02,
+0x01,
+0x80,
+0x66,
+0x00,
+0x01,
+0x30,
+0x68,
+0x5c,
+0x00,
+0x62,
+0x04,
+0x21,
+0x68,
+0x00,
+0x00,
+0x58,
+0x20,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x10,
+0x40,
+0x00,
+0x02,
+0x01,
+0x80,
+0x64,
+0x00,
+0x01,
+0x30,
+0x6f,
+0x5c,
+0x00,
+0x62,
+0x04,
+0x21,
+0x55,
+0x01,
+0x40,
+0x60,
+0x0b,
+0x5c,
+0x82,
+0x0a,
+0x20,
+0x26,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x02,
+0x30,
+0x27,
+0x87,
+0x00,
+0x84,
+0x44,
+0x54,
+0x07,
+0x80,
+0x84,
+0x42,
+0x54,
+0x23,
+0x82,
+0x49,
+0x80,
+0x82,
+0x8c,
+0x0c,
+0x0a,
+0x20,
+0x26,
+0x86,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x80,
+0x70,
+0x0b,
+0x55,
+0x00,
+0xa2,
+0x30,
+0x27,
+0x44,
+0x5d,
+0x60,
+0x78,
+0x0a,
+0x44,
+0x15,
+0x40,
+0x48,
+0x0b,
+0x55,
+0x01,
+0x09,
+0x80,
+0x8a,
+0x57,
+0x8f,
+0xb0,
+0x50,
+0x08,
+0x57,
+0x49,
+0x92,
+0x38,
+0x24,
+0x8c,
+0x0c,
+0x4a,
+0x20,
+0x22,
+0x86,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x82,
+0x10,
+0x21,
+0x85,
+0x00,
+0x84,
+0x44,
+0x5e,
+0x04,
+0x80,
+0x84,
+0x42,
+0x5e,
+0x20,
+0x82,
+0x69,
+0x80,
+0xc3,
+0x8c,
+0x0c,
+0x1a,
+0x30,
+0x22,
+0x87,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x01,
+0x84,
+0x88,
+0x85,
+0x00,
+0xb4,
+0x45,
+0xd4,
+0x21,
+0x02,
+0x15,
+0x50,
+0x0f,
+0x84,
+0x80,
+0xa4,
+0x47,
+0x54,
+0x20,
+0x82,
+0x45,
+0x50,
+0x1c,
+0x18,
+0x08,
+0x28,
+0xc0,
+0xc0,
+0xa2,
+0x02,
+0x28,
+0x60,
+0x0b,
+0x8c,
+0x02,
+0x64,
+0x45,
+0xd8,
+0x05,
+0x00,
+0xb4,
+0x45,
+0x88,
+0x21,
+0x02,
+0x19,
+0x80,
+0xc3,
+0x55,
+0x00,
+0xf0,
+0x48,
+0x0b,
+0xa0,
+0x82,
+0x44,
+0x45,
+0xda,
+0x0c,
+0x06,
+0x15,
+0x40,
+0x4d,
+0x22,
+0x02,
+0x28,
+0x60,
+0x0a,
+0x44,
+0x31,
+0x80,
+0x50,
+0x09,
+0x44,
+0x08,
+0x01,
+0x80,
+0x89,
+0x96,
+0x82,
+0x45,
+0x00,
+0x8d,
+0x21,
+0x02,
+0x08,
+0x40,
+0x08,
+0x44,
+0x21,
+0x82,
+0x28,
+0x09,
+0xa0,
+0x02,
+0x48,
+0x20,
+0x08,
+0x98,
+0x08,
+0xa4,
+0x44,
+0x10,
+0x20,
+0x80,
+0xd9,
+0x48,
+0x25,
+0x50,
+0x0a,
+0x01,
+0x68,
+0x24,
+0x50,
+0x08,
+0xd8,
+0x20,
+0x08,
+0x98,
+0x00,
+0x94,
+0x42,
+0x54,
+0x02,
+0x00,
+0x99,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0x40,
+0x80,
+0x20,
+0x85,
+0x80,
+0x99,
+0x80,
+0x88,
+0x57,
+0x8b,
+0x28,
+0x40,
+0x08,
+0x57,
+0x49,
+0x68,
+0x60,
+0x08,
+0x44,
+0x21,
+0x01,
+0x68,
+0xa5,
+0x50,
+0x0a,
+0x90,
+0x60,
+0x89,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x88,
+0x08,
+0x12,
+0x09,
+0x80,
+0x88,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x06,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x04,
+0x60,
+0xa4,
+0x18,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x5c,
+0x00,
+0x02,
+0xbf,
+0xf0,
+0x55,
+0x00,
+0x11,
+0x8e,
+0x81,
+0x5c,
+0x81,
+0x00,
+0x80,
+0x49,
+0x5c,
+0x82,
+0x0a,
+0x00,
+0x03,
+0x55,
+0x00,
+0x5b,
+0x80,
+0x00,
+0x55,
+0x02,
+0xe8,
+0x20,
+0x08,
+0x5d,
+0x0a,
+0x18,
+0x20,
+0x0b,
+0x59,
+0x04,
+0xc0,
+0xc0,
+0x8e,
+0xbf,
+0xf9,
+0xa1,
+0xb4,
+0xb3,
+0x5c,
+0x00,
+0x18,
+0x20,
+0x09,
+0x88,
+0x00,
+0x84,
+0x40,
+0xd4,
+0x16,
+0x82,
+0x55,
+0x40,
+0x08,
+0x0d,
+0x83,
+0x25,
+0x00,
+0xa0,
+0x0d,
+0x97,
+0xa5,
+0x50,
+0x10,
+0x18,
+0x00,
+0x25,
+0xc0,
+0x00,
+0x0d,
+0x86,
+0x08,
+0x80,
+0x52,
+0x40,
+0x00,
+0x02,
+0x18,
+0x86,
+0x55,
+0x02,
+0x68,
+0x20,
+0x08,
+0x5d,
+0x0a,
+0x08,
+0x20,
+0x0b,
+0x59,
+0x04,
+0x40,
+0xc0,
+0x8e,
+0xbf,
+0xf9,
+0xa1,
+0xb4,
+0xf3,
+0x86,
+0x00,
+0x88,
+0x80,
+0x09,
+0x44,
+0x25,
+0x61,
+0x68,
+0xa4,
+0x54,
+0x00,
+0x80,
+0xda,
+0x32,
+0x50,
+0x08,
+0x00,
+0xf1,
+0x7a,
+0x55,
+0x01,
+0x41,
+0x80,
+0x02,
+0x40,
+0x00,
+0x00,
+0x48,
+0x09,
+0x46,
+0x0a,
+0x40,
+0x50,
+0x08,
+0x57,
+0x8a,
+0xa8,
+0xda,
+0x60,
+0x57,
+0x49,
+0x62,
+0x80,
+0x10,
+0xab,
+0xfa,
+0x06,
+0x80,
+0x04,
+0x00,
+0x08,
+0x8b,
+0x00,
+0x10,
+0x88,
+0x07,
+0x6a,
+0x40,
+0xc0,
+0x88,
+0x0c,
+0x88,
+0x81,
+0x50,
+0x88,
+0x1e,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x00,
+0x55,
+0x02,
+0x10,
+0x81,
+0xa0,
+0x5d,
+0x04,
+0x10,
+0x80,
+0x80,
+0x59,
+0x10,
+0x80,
+0x81,
+0x52,
+0x55,
+0x02,
+0x22,
+0x00,
+0x20,
+0xbf,
+0xf3,
+0xa8,
+0x80,
+0xc8,
+0x5c,
+0x81,
+0x02,
+0x40,
+0xc0,
+0x80,
+0x00,
+0x25,
+0x94,
+0x08,
+0x00,
+0x00,
+0x88,
+0x00,
+0x0a,
+0x42,
+0x0b,
+0xe8,
+0x81,
+0x52,
+0x5c,
+0x00,
+0x00,
+0x80,
+0xe0,
+0xbc,
+0x0d,
+0xf8,
+0x81,
+0xca,
+0x98,
+0x2a,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x80,
+0x55,
+0x08,
+0x30,
+0x82,
+0x82,
+0x55,
+0x02,
+0xa0,
+0x82,
+0x00,
+0x55,
+0x02,
+0x00,
+0x81,
+0x02,
+0x40,
+0x00,
+0x00,
+0x81,
+0xca,
+0x58,
+0x04,
+0x00,
+0x82,
+0xc8,
+0xbf,
+0xf1,
+0xa8,
+0x82,
+0x50,
+0xbc,
+0x04,
+0xf5,
+0xc8,
+0x10,
+0x08,
+0x0a,
+0x03,
+0x90,
+0x20,
+0x88,
+0x0a,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x25,
+0x94,
+0x08,
+0x04,
+0x00,
+0x88,
+0x80,
+0xd2,
+0xbc,
+0x13,
+0xd5,
+0xc0,
+0x00,
+0x04,
+0x08,
+0xab,
+0xc0,
+0xcf,
+0x88,
+0x14,
+0xa9,
+0x82,
+0xa0,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa0,
+0x88,
+0x10,
+0x05,
+0x50,
+0x83,
+0x08,
+0x20,
+0x25,
+0x50,
+0x2a,
+0x08,
+0x18,
+0x05,
+0x50,
+0x20,
+0x08,
+0x08,
+0x28,
+0x81,
+0x4a,
+0x58,
+0x04,
+0x00,
+0x82,
+0x48,
+0xbf,
+0xf2,
+0xa8,
+0x81,
+0xd0,
+0x00,
+0x00,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x60,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x66,
+0x00,
+0x00,
+0xea,
+0x60,
+0x40,
+0x00,
+0x03,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x00,
+0xae,
+0x64,
+0x6e,
+0x00,
+0x00,
+0xd6,
+0x64,
+0x66,
+0x00,
+0x00,
+0xcc,
+0x80,
+0x66,
+0x00,
+0x01,
+0xb3,
+0x40,
+0x88,
+0x03,
+0x64,
+0x60,
+0xa4,
+0x30,
+0x30,
+0xc6,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x4a,
+0x80,
+0x10,
+0xab,
+0xfd,
+0x08,
+0x80,
+0xc9,
+0x5c,
+0x09,
+0xf8,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x59,
+0x02,
+0x5b,
+0x88,
+0x1d,
+0x70,
+0x00,
+0x00,
+0x6e,
+0x00,
+0x00,
+0xae,
+0x2d,
+0x25,
+0x9e,
+0x8b,
+0xc0,
+0x60,
+0x6c,
+0x68,
+0x08,
+0x10,
+0x03,
+0x6c,
+0x00,
+0x00,
+0xd8,
+0x53,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x67,
+0x6c,
+0x68,
+0x08,
+0x16,
+0x03,
+0x6c,
+0x00,
+0x00,
+0xd8,
+0x53,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x00,
+0xd6,
+0x2b,
+0x25,
+0x9d,
+0x8b,
+0xc0,
+0x50,
+0x6c,
+0x68,
+0x08,
+0x12,
+0x09,
+0x6c,
+0x00,
+0x00,
+0xda,
+0x49,
+0xbc,
+0x07,
+0x72,
+0x59,
+0xe8,
+0xbc,
+0x05,
+0x06,
+0xc6,
+0x80,
+0x81,
+0x60,
+0x96,
+0xc0,
+0x00,
+0x0d,
+0xa4,
+0x90,
+0x00,
+0x00,
+0x5c,
+0x02,
+0x29,
+0x02,
+0x13,
+0x68,
+0x00,
+0x00,
+0x00,
+0x75,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x75,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x49,
+0x00,
+0x00,
+0x08,
+0x81,
+0x97,
+0x88,
+0x00,
+0xb4,
+0x60,
+0xb4,
+0x08,
+0x13,
+0x58,
+0x80,
+0x89,
+0x40,
+0x00,
+0x02,
+0x80,
+0x30,
+0xab,
+0xef,
+0x08,
+0x8a,
+0xc8,
+0x5c,
+0x09,
+0x78,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x58,
+0x82,
+0x6f,
+0x88,
+0x2e,
+0xe8,
+0x83,
+0x6d,
+0x88,
+0x3e,
+0xc8,
+0x84,
+0x6b,
+0x88,
+0x4e,
+0xa8,
+0x85,
+0x69,
+0x88,
+0x5e,
+0x88,
+0x86,
+0x67,
+0x88,
+0x6e,
+0x68,
+0x87,
+0x65,
+0x88,
+0x7e,
+0x48,
+0x88,
+0x63,
+0x88,
+0x8e,
+0x28,
+0x89,
+0x61,
+0x88,
+0x9e,
+0x08,
+0x8a,
+0x4a,
+0x88,
+0x0c,
+0x99,
+0x0b,
+0x5b,
+0x88,
+0xc5,
+0x79,
+0x0d,
+0x59,
+0x88,
+0xcd,
+0x59,
+0x0e,
+0x5a,
+0x88,
+0xf5,
+0x69,
+0x10,
+0x58,
+0x88,
+0xfd,
+0x40,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9e,
+0x0b,
+0xc0,
+0xd8,
+0x88,
+0x1f,
+0x63,
+0x81,
+0x36,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x48,
+0xba,
+0x10,
+0x16,
+0x60,
+0x00,
+0x17,
+0x88,
+0x0b,
+0xc1,
+0x47,
+0x66,
+0x00,
+0x01,
+0x74,
+0x40,
+0xbc,
+0x11,
+0x73,
+0x81,
+0x26,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0xa8,
+0xba,
+0x10,
+0x16,
+0xc7,
+0x02,
+0x80,
+0xc0,
+0x83,
+0x81,
+0x26,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x40,
+0x66,
+0x00,
+0x01,
+0x78,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x27,
+0x66,
+0x00,
+0x01,
+0x74,
+0x40,
+0x90,
+0xe1,
+0x29,
+0x0d,
+0x11,
+0x90,
+0xb1,
+0x38,
+0x89,
+0xa0,
+0x88,
+0x92,
+0x18,
+0x88,
+0xa2,
+0x88,
+0x82,
+0x38,
+0x87,
+0xa4,
+0x88,
+0x72,
+0x58,
+0x86,
+0xa6,
+0x88,
+0x62,
+0x78,
+0x85,
+0xa8,
+0x88,
+0x52,
+0x98,
+0x84,
+0xaa,
+0x88,
+0x42,
+0xb8,
+0x80,
+0x0b,
+0x88,
+0x08,
+0x98,
+0x8a,
+0x88,
+0x88,
+0xa0,
+0xa8,
+0x83,
+0xac,
+0x88,
+0x32,
+0xd8,
+0x82,
+0xae,
+0x88,
+0x22,
+0xf8,
+0x81,
+0xb6,
+0x88,
+0x13,
+0x59,
+0x10,
+0x10,
+0x88,
+0xf1,
+0x68,
+0x8c,
+0x95,
+0x46,
+0x0b,
+0x40,
+0x8c,
+0x17,
+0x88,
+0xf9,
+0x4a,
+0x81,
+0x10,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0xab,
+0xef,
+0x08,
+0x8a,
+0xc8,
+0x5c,
+0x09,
+0x78,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x58,
+0x82,
+0x6f,
+0x88,
+0x2e,
+0xe8,
+0x83,
+0x6d,
+0x88,
+0x3e,
+0xc8,
+0x84,
+0x6b,
+0x88,
+0x4e,
+0xa8,
+0x85,
+0x69,
+0x88,
+0x5e,
+0x88,
+0x86,
+0x67,
+0x88,
+0x6e,
+0x68,
+0x87,
+0x65,
+0x88,
+0x7e,
+0x48,
+0x88,
+0x63,
+0x88,
+0x8e,
+0x28,
+0x89,
+0x61,
+0x88,
+0x9e,
+0x08,
+0x8a,
+0x4a,
+0x88,
+0x0c,
+0x99,
+0x0b,
+0x5b,
+0x88,
+0xc5,
+0x79,
+0x0d,
+0x59,
+0x88,
+0xcd,
+0x59,
+0x0e,
+0x5a,
+0x88,
+0xf5,
+0x69,
+0x10,
+0x58,
+0x88,
+0xfd,
+0x40,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9e,
+0x0b,
+0xc0,
+0xd8,
+0x88,
+0x1f,
+0x63,
+0x81,
+0x36,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x48,
+0xba,
+0x10,
+0x16,
+0x60,
+0x00,
+0x18,
+0x2c,
+0x0b,
+0xc1,
+0x47,
+0x66,
+0x00,
+0x01,
+0x7e,
+0x80,
+0xbc,
+0x11,
+0x73,
+0x81,
+0x26,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0xa8,
+0xba,
+0x10,
+0x16,
+0xc7,
+0x02,
+0x88,
+0xc0,
+0x83,
+0x81,
+0x26,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x40,
+0x66,
+0x00,
+0x01,
+0x82,
+0xc0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x27,
+0x66,
+0x00,
+0x01,
+0x7e,
+0x80,
+0x90,
+0xe1,
+0x29,
+0x0d,
+0x11,
+0x90,
+0xb1,
+0x38,
+0x89,
+0xa0,
+0x88,
+0x92,
+0x18,
+0x88,
+0xa2,
+0x88,
+0x82,
+0x38,
+0x87,
+0xa4,
+0x88,
+0x72,
+0x58,
+0x86,
+0xa6,
+0x88,
+0x62,
+0x78,
+0x85,
+0xa8,
+0x88,
+0x52,
+0x98,
+0x84,
+0xaa,
+0x88,
+0x42,
+0xb8,
+0x80,
+0x0b,
+0x88,
+0x08,
+0x98,
+0x8a,
+0x88,
+0x88,
+0xa0,
+0xa8,
+0x83,
+0xac,
+0x88,
+0x32,
+0xd8,
+0x82,
+0xae,
+0x88,
+0x22,
+0xf8,
+0x81,
+0xb6,
+0x88,
+0x13,
+0x59,
+0x10,
+0x10,
+0x88,
+0xf1,
+0x68,
+0x8c,
+0x95,
+0x46,
+0x0b,
+0x40,
+0x8c,
+0x17,
+0x88,
+0xf9,
+0x4a,
+0x81,
+0x10,
+0x60,
+0x02,
+0x80,
+0x00,
+0x10,
+0x98,
+0xea,
+0x03,
+0x90,
+0x20,
+0x80,
+0x07,
+0xa6,
+0x00,
+0x30,
+0x00,
+0x01,
+0x06,
+0x82,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x80,
+0x07,
+0xab,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x1c,
+0x05,
+0x20,
+0x5c,
+0xbf,
+0x03,
+0x00,
+0x0c,
+0x84,
+0x07,
+0xa9,
+0xc0,
+0x00,
+0x68,
+0x00,
+0x40,
+0x20,
+0x0a,
+0x84,
+0x04,
+0xaa,
+0x00,
+0x40,
+0x84,
+0x04,
+0x8a,
+0x04,
+0xc0,
+0x84,
+0x00,
+0xa5,
+0xd0,
+0xc3,
+0x30,
+0x7f,
+0x83,
+0x00,
+0x30,
+0xbc,
+0x28,
+0x06,
+0x83,
+0x81,
+0xc0,
+0x02,
+0x06,
+0x83,
+0x40,
+0x02,
+0x42,
+0x15,
+0xc8,
+0x10,
+0x84,
+0x00,
+0xaa,
+0x00,
+0xc2,
+0x84,
+0x84,
+0xa8,
+0x50,
+0x48,
+0x68,
+0x34,
+0x08,
+0x44,
+0x23,
+0x84,
+0x00,
+0xa8,
+0x48,
+0xca,
+0x85,
+0x04,
+0x8a,
+0x18,
+0x21,
+0x84,
+0x00,
+0xa8,
+0x58,
+0x4a,
+0x85,
+0x04,
+0x8a,
+0xc7,
+0xa0,
+0x84,
+0x00,
+0xa8,
+0x08,
+0xca,
+0x85,
+0x04,
+0x8a,
+0xc4,
+0x42,
+0x84,
+0x00,
+0xa8,
+0x08,
+0xca,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa8,
+0x08,
+0x4a,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa8,
+0x48,
+0x4a,
+0x85,
+0x04,
+0x8a,
+0x08,
+0x21,
+0x84,
+0x00,
+0x84,
+0x60,
+0xa4,
+0x00,
+0xb4,
+0x88,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x68,
+0x34,
+0x04,
+0x23,
+0x20,
+0x5c,
+0xbf,
+0x0b,
+0x04,
+0x04,
+0x68,
+0x38,
+0x1c,
+0x05,
+0x21,
+0x5c,
+0x82,
+0x00,
+0x40,
+0x48,
+0x5c,
+0x00,
+0x70,
+0x48,
+0x7a,
+0x5c,
+0xbd,
+0x09,
+0xc8,
+0x81,
+0x68,
+0x00,
+0x40,
+0x28,
+0x2e,
+0x5c,
+0x08,
+0x20,
+0x08,
+0x6e,
+0x84,
+0x84,
+0xaa,
+0x0c,
+0xa1,
+0x9c,
+0x08,
+0x28,
+0x48,
+0x00,
+0x85,
+0x05,
+0x00,
+0x00,
+0x00,
+0x84,
+0x00,
+0x02,
+0x41,
+0x00,
+0x84,
+0x05,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xff,
+0xc0,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x02,
+0x38,
+0x11,
+0x82,
+0x58,
+0x10,
+0xbc,
+0x32,
+0x06,
+0x83,
+0x81,
+0xc0,
+0x02,
+0x16,
+0x83,
+0x40,
+0x42,
+0x02,
+0x08,
+0x48,
+0x02,
+0x84,
+0x05,
+0x2a,
+0x00,
+0x60,
+0xb0,
+0x48,
+0x28,
+0x40,
+0x52,
+0xa0,
+0x88,
+0x18,
+0x40,
+0x02,
+0x24,
+0x11,
+0x28,
+0x40,
+0x52,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x02,
+0x25,
+0x91,
+0x0b,
+0xff,
+0xc0,
+0x84,
+0x00,
+0x22,
+0x58,
+0x10,
+0xbc,
+0x01,
+0x1b,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x40,
+0x28,
+0x6e,
+0x80,
+0x86,
+0xe8,
+0x48,
+0x4a,
+0xa0,
+0xcc,
+0x18,
+0x48,
+0x0a,
+0x6c,
+0x68,
+0x08,
+0x40,
+0x4a,
+0xb0,
+0x50,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x40,
+0x0a,
+0x24,
+0x13,
+0x68,
+0x40,
+0x4a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x0a,
+0x25,
+0x93,
+0x0b,
+0xff,
+0xc0,
+0x84,
+0x00,
+0x82,
+0x58,
+0x20,
+0xbc,
+0x01,
+0x1b,
+0xa1,
+0x40,
+0xba,
+0x14,
+0x88,
+0x40,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x14,
+0x40,
+0x68,
+0x34,
+0x04,
+0x04,
+0x20,
+0x5c,
+0x02,
+0xe2,
+0xff,
+0xc0,
+0x5c,
+0x00,
+0x70,
+0x40,
+0x48,
+0x9c,
+0x00,
+0x0b,
+0x04,
+0x80,
+0x68,
+0x34,
+0x00,
+0x1e,
+0x21,
+0x84,
+0x05,
+0x0b,
+0x1d,
+0xf2,
+0x84,
+0x85,
+0x2a,
+0x08,
+0x81,
+0xb1,
+0x81,
+0xc8,
+0x48,
+0x48,
+0xa0,
+0x22,
+0x06,
+0x81,
+0x50,
+0xc8,
+0x40,
+0x08,
+0x40,
+0x50,
+0x68,
+0x00,
+0x01,
+0x41,
+0x48,
+0x88,
+0x03,
+0x6a,
+0x06,
+0x00,
+0x46,
+0x0a,
+0x40,
+0x48,
+0xc8,
+0x84,
+0x04,
+0xaa,
+0x80,
+0x10,
+0x68,
+0x34,
+0x00,
+0x09,
+0x20,
+0x5c,
+0x08,
+0xb3,
+0x00,
+0x3c,
+0x5c,
+0x81,
+0x00,
+0x40,
+0x00,
+0x52,
+0x0c,
+0x03,
+0x00,
+0x26,
+0x84,
+0x05,
+0x0a,
+0x00,
+0xa0,
+0x84,
+0x04,
+0x8a,
+0x00,
+0x40,
+0x80,
+0x04,
+0x88,
+0x00,
+0x4a,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x4a,
+0x84,
+0x7f,
+0xa0,
+0x00,
+0x00,
+0x68,
+0x34,
+0x08,
+0x4a,
+0x20,
+0x38,
+0x01,
+0xc8,
+0x40,
+0x48,
+0x40,
+0x00,
+0x00,
+0x40,
+0xc8,
+0x64,
+0x00,
+0x01,
+0x1e,
+0xa7,
+0x68,
+0x00,
+0x01,
+0xbc,
+0x20,
+0x39,
+0x02,
+0x08,
+0x00,
+0x21,
+0x68,
+0x34,
+0x08,
+0x20,
+0x22,
+0x84,
+0x80,
+0x88,
+0x00,
+0x21,
+0x85,
+0x04,
+0x8a,
+0x10,
+0x22,
+0x84,
+0x80,
+0x88,
+0x00,
+0x21,
+0x84,
+0x02,
+0x08,
+0x10,
+0x48,
+0x00,
+0x00,
+0x08,
+0x48,
+0x08,
+0x81,
+0x04,
+0x80,
+0x00,
+0x00,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x08,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x68,
+0x34,
+0x08,
+0x57,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x04,
+0x7a,
+0x5c,
+0x81,
+0x03,
+0x04,
+0x24,
+0x5c,
+0x13,
+0x70,
+0x40,
+0x48,
+0xa0,
+0x02,
+0x08,
+0x00,
+0x7a,
+0x80,
+0x04,
+0x88,
+0x40,
+0x7a,
+0xa0,
+0x52,
+0x08,
+0x02,
+0x4a,
+0xb0,
+0x60,
+0x48,
+0x40,
+0x48,
+0xa0,
+0x48,
+0x08,
+0x00,
+0x7a,
+0xac,
+0x9a,
+0x08,
+0x02,
+0x7a,
+0x84,
+0x07,
+0xaa,
+0x01,
+0x00,
+0xac,
+0xb0,
+0x08,
+0x00,
+0x7a,
+0xac,
+0x40,
+0x04,
+0x60,
+0xa4,
+0x00,
+0x24,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x34,
+0x00,
+0x17,
+0x20,
+0x68,
+0x03,
+0xfb,
+0xff,
+0xc9,
+0x5c,
+0x08,
+0x20,
+0x40,
+0x7a,
+0x5c,
+0x0b,
+0x32,
+0x00,
+0x80,
+0x5c,
+0xbd,
+0x00,
+0x40,
+0x02,
+0x54,
+0x4a,
+0xaa,
+0x03,
+0x21,
+0x5c,
+0x00,
+0x40,
+0x40,
+0x49,
+0x5c,
+0x09,
+0xd2,
+0xbf,
+0xe0,
+0x84,
+0x80,
+0x95,
+0x20,
+0x96,
+0x88,
+0x0f,
+0x65,
+0xc0,
+0x02,
+0x84,
+0x84,
+0x96,
+0x83,
+0x80,
+0x80,
+0xf2,
+0x28,
+0x40,
+0x08,
+0x52,
+0x0d,
+0x20,
+0x80,
+0x49,
+0x5c,
+0x1d,
+0x68,
+0x40,
+0x48,
+0xa0,
+0x5a,
+0x08,
+0x40,
+0x7a,
+0x9d,
+0x00,
+0x08,
+0x50,
+0x50,
+0x84,
+0x05,
+0x28,
+0x81,
+0x60,
+0x66,
+0x00,
+0x01,
+0x2e,
+0x20,
+0x5c,
+0x08,
+0x30,
+0x80,
+0x09,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x55,
+0x03,
+0x6b,
+0x03,
+0x24,
+0x30,
+0x12,
+0x8b,
+0xff,
+0x6a,
+0x5c,
+0x1d,
+0x68,
+0x80,
+0x49,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x1a,
+0xeb,
+0x00,
+0x0e,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x1a,
+0xa9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x0d,
+0xe9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0x69,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0xa9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0xeb,
+0x00,
+0x26,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x07,
+0x2b,
+0x00,
+0xc6,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x07,
+0x6b,
+0x00,
+0x36,
+0x5c,
+0x02,
+0x68,
+0x81,
+0x20,
+0x5c,
+0x01,
+0xe0,
+0x80,
+0xb6,
+0x6c,
+0x70,
+0x10,
+0x1e,
+0x49,
+0x5c,
+0x03,
+0xe0,
+0x40,
+0x48,
+0x5c,
+0x00,
+0x72,
+0x05,
+0x80,
+0x84,
+0x07,
+0xaa,
+0x00,
+0x40,
+0x84,
+0x07,
+0xab,
+0x05,
+0xdd,
+0xa0,
+0x02,
+0x08,
+0x40,
+0x49,
+0xa0,
+0x16,
+0x08,
+0x40,
+0x48,
+0x46,
+0x0a,
+0x42,
+0x05,
+0x00,
+0x84,
+0x04,
+0xaa,
+0x80,
+0x20,
+0x5c,
+0x08,
+0x01,
+0x8e,
+0x8a,
+0x5c,
+0x00,
+0x51,
+0x82,
+0x88,
+0x68,
+0x00,
+0x03,
+0xe8,
+0x09,
+0x6c,
+0x70,
+0x10,
+0x0e,
+0x0b,
+0x25,
+0x83,
+0x8b,
+0xc0,
+0x41,
+0xba,
+0x14,
+0x86,
+0xc7,
+0x01,
+0x00,
+0xe5,
+0x20,
+0x00,
+0x00,
+0x2a,
+0x07,
+0x63,
+0x01,
+0x70,
+0xbf,
+0xf5,
+0x26,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x83,
+0x81,
+0xe6,
+0x52,
+0x0d,
+0x33,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x4a,
+0x5c,
+0x3f,
+0xe3,
+0x80,
+0x00,
+0x68,
+0x38,
+0x08,
+0x00,
+0x20,
+0x5c,
+0x09,
+0xe2,
+0xbf,
+0x90,
+0x52,
+0x09,
+0x62,
+0x00,
+0x41,
+0x5d,
+0x8c,
+0x12,
+0x08,
+0x20,
+0x5c,
+0x00,
+0x00,
+0x80,
+0x60,
+0xa0,
+0x16,
+0x0a,
+0x05,
+0x00,
+0xa0,
+0x4c,
+0x28,
+0x80,
+0xe2,
+0xa1,
+0x04,
+0x28,
+0x81,
+0x62,
+0xa1,
+0x02,
+0x28,
+0x81,
+0xe2,
+0xa1,
+0x16,
+0x2a,
+0x15,
+0x02,
+0x88,
+0x4c,
+0x88,
+0x83,
+0x62,
+0x88,
+0x3e,
+0x08,
+0x84,
+0x61,
+0x88,
+0x55,
+0x28,
+0x82,
+0x4a,
+0x88,
+0x2c,
+0x98,
+0x85,
+0xf6,
+0x40,
+0x00,
+0x03,
+0x04,
+0x04,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x48,
+0x5c,
+0x00,
+0xa0,
+0x48,
+0x7a,
+0x5c,
+0x00,
+0x68,
+0x48,
+0xc9,
+0x84,
+0x8c,
+0xa8,
+0x86,
+0x50,
+0x00,
+0x00,
+0x08,
+0x80,
+0x21,
+0x00,
+0x00,
+0x08,
+0x4d,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x40,
+0x49,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x5c,
+0x00,
+0xa0,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x68,
+0x81,
+0x21,
+0x84,
+0x07,
+0xa8,
+0x41,
+0x7a,
+0x00,
+0x00,
+0x08,
+0x84,
+0x8a,
+0x84,
+0x8c,
+0xa0,
+0x00,
+0x00,
+0x88,
+0x1a,
+0x08,
+0x83,
+0x21,
+0x84,
+0x5c,
+0x88,
+0x48,
+0x49,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x5c,
+0xbb,
+0x00,
+0x83,
+0x20,
+0x88,
+0x50,
+0x99,
+0xc0,
+0x00,
+0x84,
+0x00,
+0xa3,
+0xb1,
+0x80,
+0x30,
+0x14,
+0x0b,
+0xc0,
+0x89,
+0x5c,
+0x19,
+0x20,
+0x86,
+0x0a,
+0x88,
+0x5b,
+0x6b,
+0xa1,
+0x48,
+0x5c,
+0x00,
+0x22,
+0x80,
+0x70,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x55,
+0x03,
+0x80,
+0x84,
+0x21,
+0x58,
+0x08,
+0x00,
+0x82,
+0x89,
+0x88,
+0x20,
+0xa4,
+0x3e,
+0x4d,
+0x08,
+0x3a,
+0x0b,
+0x04,
+0x04,
+0x88,
+0x5b,
+0x6b,
+0xa1,
+0x48,
+0x5c,
+0x3f,
+0xe2,
+0x80,
+0x70,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x08,
+0x00,
+0x20,
+0x5c,
+0x09,
+0xf3,
+0x00,
+0x14,
+0x52,
+0x0d,
+0x68,
+0x40,
+0x7a,
+0x5c,
+0x00,
+0x72,
+0x00,
+0x40,
+0x84,
+0x07,
+0xaa,
+0x00,
+0x20,
+0x84,
+0x04,
+0x9a,
+0x01,
+0x60,
+0x84,
+0x04,
+0x8a,
+0x05,
+0x00,
+0x84,
+0x04,
+0xaa,
+0xbf,
+0xf0,
+0xa0,
+0x4a,
+0x08,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x66,
+0x60,
+0x00,
+0x12,
+0x6c,
+0x08,
+0x80,
+0xb6,
+0x88,
+0x02,
+0x0b,
+0xa1,
+0x48,
+0x84,
+0x00,
+0x8a,
+0x80,
+0x10,
+0x62,
+0x00,
+0x00,
+0x00,
+0x36,
+0x5c,
+0x81,
+0x09,
+0x82,
+0x24,
+0x5c,
+0x80,
+0x81,
+0x82,
+0x60,
+0xbb,
+0x00,
+0x08,
+0x00,
+0xcc,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x38,
+0x00,
+0xe2,
+0x11,
+0x34,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x6d,
+0x84,
+0x86,
+0x06,
+0x20,
+0x00,
+0x00,
+0x01,
+0x43,
+0x90,
+0x20,
+0x00,
+0x00,
+0x08,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x00,
+0x61,
+0x88,
+0x2c,
+0x50,
+0x8d,
+0x20,
+0x48,
+0x20,
+0x51,
+0x85,
+0x3a,
+0xc0,
+0x20,
+0x98,
+0x2e,
+0xe0,
+0x00,
+0x00,
+0x9c,
+0x40,
+0x08,
+0x40,
+0x49,
+0x62,
+0x00,
+0x00,
+0x00,
+0x24,
+0x84,
+0x86,
+0x09,
+0x8e,
+0x80,
+0x80,
+0x40,
+0x92,
+0x81,
+0x40,
+0xba,
+0x14,
+0x85,
+0x04,
+0xc0,
+0x04,
+0x86,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x06,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x04,
+0x60,
+0xa4,
+0x18,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x02,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x09,
+0x80,
+0x82,
+0x55,
+0x00,
+0xb8,
+0x20,
+0x08,
+0x8c,
+0x06,
+0x0a,
+0x20,
+0x01,
+0x8c,
+0x12,
+0xe4,
+0x42,
+0x00,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x04,
+0x88,
+0x94,
+0x46,
+0xc8,
+0x06,
+0x08,
+0x85,
+0xb0,
+0x82,
+0x98,
+0x04,
+0xa5,
+0x78,
+0x9a,
+0x18,
+0x48,
+0x05,
+0x74,
+0xb1,
+0x3a,
+0x14,
+0x88,
+0xc1,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x32,
+0x02,
+0x8b,
+0xc3,
+0x58,
+0x22,
+0x86,
+0xe5,
+0xb8,
+0xa2,
+0x30,
+0x0f,
+0x05,
+0x70,
+0x80,
+0x30,
+0x00,
+0xf5,
+0x08,
+0x1c,
+0x30,
+0x4e,
+0xa2,
+0xa6,
+0x63,
+0x4b,
+0xc1,
+0x23,
+0x3c,
+0x08,
+0x20,
+0x0c,
+0xd2,
+0x35,
+0xd4,
+0x68,
+0x20,
+0x01,
+0x51,
+0x20,
+0x28,
+0x16,
+0x25,
+0x14,
+0x2a,
+0x84,
+0x00,
+0x85,
+0x70,
+0xb0,
+0xb0,
+0x0f,
+0xa5,
+0x16,
+0xe6,
+0x04,
+0x08,
+0x92,
+0x80,
+0x20,
+0x68,
+0x1f,
+0xff,
+0xff,
+0xc8,
+0x36,
+0x80,
+0x35,
+0x44,
+0x84,
+0x98,
+0x0c,
+0x82,
+0x32,
+0x09,
+0x28,
+0x08,
+0x02,
+0x10,
+0x3f,
+0x20,
+0x90,
+0xc2,
+0x81,
+0x3f,
+0x08,
+0xb0,
+0x02,
+0x23,
+0xc0,
+0x98,
+0x00,
+0x80,
+0x83,
+0x00,
+0x22,
+0x3c,
+0x02,
+0xe0,
+0x28,
+0x98,
+0x00,
+0x80,
+0x8c,
+0x00,
+0x22,
+0x04,
+0x09,
+0x80,
+0x0b,
+0x08,
+0xb0,
+0x02,
+0x23,
+0xc0,
+0x98,
+0x00,
+0xa0,
+0x8b,
+0x00,
+0x22,
+0x3c,
+0x02,
+0xe0,
+0x28,
+0x98,
+0x00,
+0xa4,
+0x47,
+0x00,
+0x3a,
+0x14,
+0x82,
+0x20,
+0x40,
+0x98,
+0x00,
+0x8b,
+0xa1,
+0x48,
+0x98,
+0xe8,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x08,
+0x29,
+0x58,
+0x2c,
+0x52,
+0xcb,
+0x02,
+0xbf,
+0xc0,
+0x88,
+0x0e,
+0x14,
+0x20,
+0x7c,
+0x88,
+0x16,
+0x28,
+0x80,
+0x76,
+0x00,
+0x00,
+0x09,
+0x5a,
+0x2e,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0xa0,
+0x95,
+0xba,
+0xe3,
+0x20,
+0x30,
+0xbc,
+0x04,
+0x18,
+0x59,
+0x08,
+0x55,
+0x03,
+0x23,
+0xc0,
+0x4f,
+0x85,
+0x94,
+0x82,
+0x49,
+0x60,
+0x40,
+0x00,
+0x01,
+0x58,
+0x60,
+0x5c,
+0x82,
+0x0a,
+0x18,
+0xa1,
+0x88,
+0x1e,
+0x10,
+0x00,
+0x00,
+0x80,
+0x88,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x06,
+0x95,
+0xc8,
+0x10,
+0x08,
+0x26,
+0x13,
+0x81,
+0x05,
+0x95,
+0x8a,
+0xc2,
+0x59,
+0x60,
+0xbc,
+0x45,
+0x18,
+0x82,
+0xe5,
+0x88,
+0x36,
+0x30,
+0x00,
+0x00,
+0x8c,
+0x02,
+0xa8,
+0x20,
+0x09,
+0x44,
+0x08,
+0x00,
+0x55,
+0x22,
+0x82,
+0x00,
+0x84,
+0x44,
+0x40,
+0x02,
+0x00,
+0x88,
+0x50,
+0x09,
+0x44,
+0x25,
+0x01,
+0x82,
+0x40,
+0x98,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x00,
+0x00,
+0x08,
+0xc1,
+0x2a,
+0x82,
+0x00,
+0x94,
+0x40,
+0x80,
+0x06,
+0x00,
+0x84,
+0x44,
+0x48,
+0x06,
+0x08,
+0x95,
+0x50,
+0x0a,
+0x18,
+0x48,
+0x00,
+0x81,
+0xa4,
+0x98,
+0x08,
+0x28,
+0xc1,
+0x60,
+0x40,
+0x00,
+0x00,
+0x83,
+0xd2,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x04,
+0x5c,
+0x08,
+0x71,
+0x48,
+0xad,
+0x25,
+0x9a,
+0x8b,
+0xc1,
+0xa1,
+0x94,
+0x82,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x07,
+0x88,
+0x83,
+0x88,
+0x88,
+0x12,
+0x06,
+0x60,
+0x00,
+0x14,
+0xc4,
+0x0b,
+0xc1,
+0x7f,
+0x40,
+0x00,
+0x00,
+0x81,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x4a,
+0x28,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x00,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0x42,
+0x03,
+0x40,
+0x81,
+0x20,
+0x94,
+0xa6,
+0x46,
+0x60,
+0x00,
+0x14,
+0xc4,
+0x88,
+0x83,
+0x88,
+0xbc,
+0x08,
+0xf8,
+0x81,
+0xa0,
+0xbc,
+0x06,
+0xf8,
+0x81,
+0xa0,
+0xbc,
+0x04,
+0xf8,
+0x81,
+0xa0,
+0x88,
+0x1a,
+0x08,
+0x82,
+0xe5,
+0x88,
+0x36,
+0x30,
+0x00,
+0x00,
+0x88,
+0x22,
+0x58,
+0x41,
+0x08,
+0x59,
+0x01,
+0x02,
+0x2c,
+0xd8,
+0x42,
+0x04,
+0x48,
+0x82,
+0xa5,
+0x5c,
+0x81,
+0x00,
+0x81,
+0xe0,
+0x38,
+0x10,
+0x48,
+0x83,
+0x20,
+0x00,
+0x00,
+0x09,
+0x40,
+0xad,
+0x25,
+0x92,
+0x8b,
+0xc3,
+0xc1,
+0x00,
+0x00,
+0x08,
+0x81,
+0x20,
+0x88,
+0x0a,
+0x3a,
+0x02,
+0x80,
+0x8d,
+0x82,
+0xa8,
+0x28,
+0x09,
+0x44,
+0x08,
+0x00,
+0x40,
+0x20,
+0x82,
+0x80,
+0x84,
+0x44,
+0x40,
+0x04,
+0x00,
+0x98,
+0x28,
+0x08,
+0x44,
+0x25,
+0x01,
+0x82,
+0x40,
+0x98,
+0x08,
+0x25,
+0x50,
+0x0b,
+0x8d,
+0x86,
+0x06,
+0x80,
+0x00,
+0x0d,
+0x72,
+0x08,
+0xd9,
+0x2a,
+0x82,
+0x80,
+0x94,
+0x40,
+0x80,
+0x06,
+0x80,
+0x84,
+0x44,
+0x48,
+0x06,
+0x88,
+0x94,
+0x46,
+0xd2,
+0x18,
+0x48,
+0x09,
+0x80,
+0x82,
+0x8d,
+0x96,
+0x08,
+0x80,
+0xd2,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x04,
+0x5c,
+0x08,
+0x71,
+0x48,
+0xad,
+0x25,
+0x9a,
+0x8b,
+0xc1,
+0x41,
+0x94,
+0x82,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x05,
+0x88,
+0x80,
+0x88,
+0x88,
+0x12,
+0x06,
+0x60,
+0x00,
+0x14,
+0xee,
+0x0b,
+0xc0,
+0xc7,
+0x66,
+0x00,
+0x01,
+0x47,
+0xa8,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x00,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0x42,
+0x02,
+0x40,
+0x81,
+0x20,
+0x94,
+0xbe,
+0x46,
+0x60,
+0x00,
+0x14,
+0xee,
+0x88,
+0x80,
+0x88,
+0x38,
+0x11,
+0x48,
+0x81,
+0xa0,
+0x00,
+0x00,
+0x09,
+0x40,
+0x2d,
+0x25,
+0x92,
+0x8b,
+0xc2,
+0xc1,
+0x68,
+0x20,
+0x01,
+0x7c,
+0x23,
+0x5c,
+0x81,
+0x12,
+0xc0,
+0x80,
+0x68,
+0x20,
+0x01,
+0x82,
+0x24,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x20,
+0x52,
+0x49,
+0x40,
+0x19,
+0x0a,
+0x68,
+0x20,
+0x00,
+0xe6,
+0x25,
+0x5c,
+0x82,
+0x08,
+0x21,
+0x08,
+0x81,
+0x80,
+0x98,
+0x01,
+0x4a,
+0x82,
+0x94,
+0x88,
+0x00,
+0x49,
+0x00,
+0x00,
+0x08,
+0x20,
+0x0b,
+0x82,
+0x28,
+0x98,
+0x1a,
+0x88,
+0x6c,
+0x40,
+0x02,
+0xfc,
+0x0a,
+0x82,
+0x84,
+0xb8,
+0x2a,
+0xc9,
+0x80,
+0x2c,
+0x86,
+0xc4,
+0x00,
+0x1c,
+0x44,
+0xa0,
+0x00,
+0x00,
+0x88,
+0x32,
+0x28,
+0x60,
+0x0b,
+0x85,
+0x80,
+0x28,
+0x60,
+0x88,
+0x85,
+0x88,
+0x96,
+0xc4,
+0x00,
+0x30,
+0x80,
+0xa8,
+0x68,
+0x4b,
+0x84,
+0x05,
+0x28,
+0x68,
+0xc8,
+0x84,
+0x0c,
+0x99,
+0x50,
+0xe0,
+0x6c,
+0x40,
+0x01,
+0xd0,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x40,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x94,
+0x3a,
+0xc3,
+0x20,
+0x20,
+0xbc,
+0x21,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0xbc,
+0x03,
+0xf9,
+0x43,
+0x64,
+0xb0,
+0x00,
+0xd9,
+0x43,
+0x65,
+0xa0,
+0x02,
+0x19,
+0x48,
+0x2d,
+0x59,
+0x01,
+0x42,
+0x08,
+0x12,
+0x42,
+0x08,
+0xc1,
+0x50,
+0x2d,
+0xa1,
+0x40,
+0x95,
+0xc0,
+0x84,
+0x14,
+0x02,
+0xe2,
+0x58,
+0x30,
+0xbc,
+0x07,
+0x99,
+0x42,
+0xae,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x91,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x70,
+0xb0,
+0x00,
+0xcb,
+0xc0,
+0x57,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x30,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x11,
+0xb0,
+0x00,
+0xcb,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x50,
+0x00,
+0x00,
+0xba,
+0x14,
+0x8b,
+0x00,
+0x0c,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x94,
+0x22,
+0xc3,
+0x20,
+0x20,
+0xbc,
+0x1c,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0xbc,
+0x03,
+0xf9,
+0x41,
+0xe4,
+0xb0,
+0x00,
+0xd9,
+0x41,
+0xe5,
+0x5c,
+0x08,
+0x6a,
+0x00,
+0x11,
+0x94,
+0x02,
+0xe5,
+0x2c,
+0xb8,
+0x14,
+0x82,
+0xdb,
+0xc0,
+0x79,
+0x94,
+0x8a,
+0xe3,
+0x20,
+0x28,
+0xbc,
+0x09,
+0x13,
+0x20,
+0x30,
+0xbc,
+0x07,
+0x0b,
+0x00,
+0x0c,
+0xbc,
+0x05,
+0x73,
+0x20,
+0x28,
+0xbc,
+0x03,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x01,
+0x1b,
+0x00,
+0x0c,
+0xba,
+0x14,
+0x89,
+0x41,
+0x66,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x48,
+0xb0,
+0x00,
+0xc0,
+0x00,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x08,
+0x82,
+0x85,
+0x00,
+0xa2,
+0xa0,
+0x76,
+0x81,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x85,
+0x00,
+0x03,
+0x00,
+0x30,
+0xbc,
+0x1d,
+0x4a,
+0x00,
+0x41,
+0x84,
+0x80,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x01,
+0x38,
+0x41,
+0x48,
+0xa0,
+0x82,
+0x28,
+0x50,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x25,
+0x40,
+0x00,
+0x00,
+0x48,
+0xc8,
+0x5b,
+0x48,
+0x03,
+0x01,
+0x05,
+0x50,
+0x4b,
+0x08,
+0x10,
+0x0a,
+0x5c,
+0x82,
+0x00,
+0xd1,
+0x32,
+0x54,
+0x02,
+0x90,
+0x41,
+0x08,
+0x57,
+0x0d,
+0x20,
+0xd1,
+0x7a,
+0x81,
+0x04,
+0x89,
+0x80,
+0x08,
+0x50,
+0x4b,
+0x10,
+0xd1,
+0x30,
+0x54,
+0x04,
+0x03,
+0xa1,
+0x48,
+0x8d,
+0x17,
+0x80,
+0x00,
+0x00,
+0xba,
+0x14,
+0x88,
+0x4a,
+0xfa,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x08,
+0xc2,
+0x85,
+0x00,
+0xa2,
+0xa0,
+0x76,
+0x81,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x85,
+0x00,
+0x03,
+0x00,
+0x30,
+0xbc,
+0x1f,
+0x4a,
+0x01,
+0x81,
+0x84,
+0x80,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x01,
+0x38,
+0x46,
+0x48,
+0xa0,
+0x82,
+0x28,
+0x50,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x25,
+0x40,
+0x00,
+0x00,
+0x48,
+0xc8,
+0x5c,
+0x82,
+0x0b,
+0x01,
+0x05,
+0x5b,
+0x48,
+0x10,
+0x46,
+0x0a,
+0x81,
+0x00,
+0x02,
+0xe0,
+0x36,
+0x50,
+0x4b,
+0x00,
+0x10,
+0xca,
+0x98,
+0x08,
+0x88,
+0xd0,
+0x31,
+0x28,
+0x00,
+0x85,
+0x04,
+0xb0,
+0x0d,
+0x0d,
+0x80,
+0x00,
+0x00,
+0x8d,
+0x03,
+0x25,
+0x40,
+0x08,
+0x3a,
+0x14,
+0x88,
+0xd0,
+0x78,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x08,
+0x22,
+0x08,
+0xe0,
+0xa0,
+0x4d,
+0xa9,
+0x50,
+0x2e,
+0x25,
+0x93,
+0x0b,
+0xc0,
+0x69,
+0x84,
+0x07,
+0xa3,
+0x81,
+0x0c,
+0x52,
+0x09,
+0x83,
+0xa1,
+0x48,
+0x94,
+0x8e,
+0x00,
+0x00,
+0x00,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x20,
+0x01,
+0x2b,
+0x20,
+0x5c,
+0x81,
+0x03,
+0x00,
+0x0c,
+0x5c,
+0x06,
+0x68,
+0x02,
+0x0a,
+0x68,
+0x20,
+0x01,
+0x05,
+0x21,
+0x51,
+0x89,
+0x80,
+0x40,
+0x01,
+0x51,
+0x8a,
+0x00,
+0x08,
+0x50,
+0x68,
+0x20,
+0x01,
+0xf0,
+0x22,
+0x84,
+0x80,
+0x28,
+0x10,
+0x52,
+0x81,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x0f,
+0xc2,
+0x36,
+0x82,
+0x00,
+0x0d,
+0x72,
+0x18,
+0x10,
+0x7a,
+0x68,
+0x00,
+0x00,
+0xdc,
+0x24,
+0x85,
+0x07,
+0xaa,
+0x10,
+0x42,
+0x84,
+0x90,
+0xb8,
+0x58,
+0x02,
+0x85,
+0x88,
+0xa8,
+0x48,
+0x48,
+0xa2,
+0x06,
+0x35,
+0xc0,
+0x02,
+0x86,
+0x04,
+0x98,
+0x12,
+0x52,
+0x68,
+0x00,
+0x00,
+0xa6,
+0x21,
+0xa1,
+0xc2,
+0x0a,
+0x10,
+0x64,
+0xab,
+0xff,
+0x0e,
+0x21,
+0x04,
+0x84,
+0x07,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0xa7,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0xa7,
+0xa6,
+0xc0,
+0x00,
+0x37,
+0x64,
+0x86,
+0xc0,
+0x00,
+0x36,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x19,
+0x84,
+0xb6,
+0xc4,
+0x00,
+0x19,
+0xa4,
+0xb6,
+0xc4,
+0x00,
+0x31,
+0x44,
+0xb6,
+0xc4,
+0x00,
+0x31,
+0xa4,
+0xb9,
+0x58,
+0x65,
+0x6c,
+0x40,
+0x01,
+0x06,
+0x51,
+0x6c,
+0x40,
+0x01,
+0x08,
+0x51,
+0x84,
+0x87,
+0xa8,
+0x50,
+0x4a,
+0x86,
+0x04,
+0xa8,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x23,
+0x8b,
+0x68,
+0x41,
+0x4a,
+0x66,
+0x00,
+0x00,
+0x4c,
+0x28,
+0x40,
+0x00,
+0x03,
+0x19,
+0x04,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x68,
+0x00,
+0x04,
+0xe2,
+0x0a,
+0x88,
+0x03,
+0x68,
+0x41,
+0x4a,
+0x64,
+0x00,
+0x00,
+0x4c,
+0x2f,
+0x5c,
+0x02,
+0xa2,
+0x80,
+0x10,
+0x30,
+0x1a,
+0x8b,
+0xc1,
+0x58,
+0x6c,
+0x40,
+0x02,
+0xc0,
+0x08,
+0x84,
+0x00,
+0xb3,
+0x69,
+0xc0,
+0x98,
+0x00,
+0xb3,
+0x09,
+0x38,
+0xbc,
+0x0d,
+0x46,
+0xc4,
+0x00,
+0x2b,
+0xc0,
+0x83,
+0x01,
+0x30,
+0xbc,
+0x03,
+0x2b,
+0xc0,
+0x4f,
+0x6c,
+0x40,
+0x02,
+0xbe,
+0x08,
+0x6c,
+0x40,
+0x02,
+0xb0,
+0x08,
+0x28,
+0x13,
+0x04,
+0x20,
+0x17,
+0x98,
+0x00,
+0xa2,
+0xf1,
+0x75,
+0x98,
+0x28,
+0x9b,
+0xa1,
+0x48,
+0x98,
+0x24,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0xdd,
+0x22,
+0x6c,
+0x00,
+0x01,
+0xb8,
+0x08,
+0x85,
+0x00,
+0xa3,
+0x01,
+0x30,
+0x55,
+0x03,
+0xab,
+0xc5,
+0xf9,
+0x85,
+0x04,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xa7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x08,
+0x08,
+0x5b,
+0x48,
+0x00,
+0x40,
+0x08,
+0x5b,
+0x48,
+0x11,
+0x54,
+0x24,
+0x32,
+0x06,
+0x0b,
+0xc3,
+0x39,
+0x9a,
+0x00,
+0x03,
+0x01,
+0x30,
+0xbc,
+0x09,
+0x48,
+0x48,
+0x0a,
+0x36,
+0x98,
+0x09,
+0x80,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x44,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x1b,
+0xc7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0xbc,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x06,
+0x0a,
+0x2a,
+0x06,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x40,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xc4,
+0x85,
+0xc0,
+0xce,
+0xb0,
+0x13,
+0x76,
+0xe0,
+0x00,
+0x14,
+0x8a,
+0xe6,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x15,
+0xc8,
+0x10,
+0x18,
+0xe8,
+0x86,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x95,
+0x2c,
+0xf8,
+0x14,
+0xa4,
+0x4b,
+0xc0,
+0x99,
+0x5c,
+0x0b,
+0xa0,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2c,
+0x20,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0x04,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x0a,
+0x52,
+0x49,
+0xa3,
+0xa1,
+0x48,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x00,
+0x00,
+0x03,
+0x01,
+0x30,
+0xbc,
+0x05,
+0x38,
+0x48,
+0x0a,
+0x36,
+0x98,
+0x09,
+0x80,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x42,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x1b,
+0xc7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0xbc,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x04,
+0x0a,
+0x2a,
+0x06,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x0e,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xc4,
+0x86,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x16,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0xc0,
+0x07,
+0x30,
+0x17,
+0x55,
+0x20,
+0xb2,
+0x2c,
+0x02,
+0x09,
+0x4a,
+0x46,
+0x84,
+0x87,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x8b,
+0xa1,
+0x40,
+0x68,
+0x20,
+0x01,
+0x46,
+0x21,
+0x68,
+0x00,
+0x00,
+0xcd,
+0x22,
+0x39,
+0x02,
+0x08,
+0x08,
+0x08,
+0x81,
+0x07,
+0xa8,
+0x10,
+0x48,
+0x68,
+0x00,
+0x00,
+0xd2,
+0x20,
+0x84,
+0x80,
+0x88,
+0x00,
+0x7a,
+0x81,
+0x04,
+0x88,
+0x00,
+0x7a,
+0x68,
+0x00,
+0x00,
+0xca,
+0x23,
+0x80,
+0x04,
+0x86,
+0x80,
+0x00,
+0x58,
+0x8a,
+0xca,
+0x10,
+0x01,
+0x81,
+0x86,
+0xc8,
+0x10,
+0x7a,
+0xa0,
+0x00,
+0x48,
+0x58,
+0x7a,
+0x85,
+0x06,
+0x14,
+0x60,
+0xa4,
+0x00,
+0x07,
+0xa8,
+0x40,
+0x64,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0c,
+0x16,
+0xc0,
+0x00,
+0x1a,
+0x22,
+0x06,
+0x80,
+0x00,
+0x58,
+0xd2,
+0xc6,
+0xc0,
+0x00,
+0x1a,
+0xc2,
+0x16,
+0xc0,
+0x00,
+0x19,
+0x46,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0c,
+0x8a,
+0xbf,
+0xf0,
+0x6c,
+0x00,
+0x01,
+0xa2,
+0x20,
+0x68,
+0x00,
+0x05,
+0x88,
+0xac,
+0x6c,
+0x00,
+0x01,
+0xac,
+0x21,
+0x6c,
+0x00,
+0x01,
+0x94,
+0x6c,
+0x42,
+0x09,
+0xf8,
+0x40,
+0x7a,
+0x84,
+0x87,
+0xa8,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x66,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0xaa,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x65,
+0x6c,
+0x00,
+0x01,
+0xac,
+0x20,
+0x42,
+0x03,
+0xf8,
+0x80,
+0x36,
+0x40,
+0x00,
+0x00,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x05,
+0x96,
+0xa0,
+0x88,
+0x03,
+0x66,
+0xc0,
+0x00,
+0x19,
+0x46,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0d,
+0x06,
+0xc0,
+0x00,
+0x1a,
+0x22,
+0x06,
+0x80,
+0x00,
+0x58,
+0x8a,
+0xc6,
+0xc0,
+0x00,
+0x1a,
+0xc2,
+0x16,
+0xc0,
+0x00,
+0x19,
+0x46,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x64,
+0x00,
+0x01,
+0x66,
+0xe7,
+0x68,
+0x00,
+0x00,
+0xce,
+0x20,
+0x68,
+0x20,
+0x01,
+0x48,
+0x21,
+0xab,
+0xfe,
+0x08,
+0x40,
+0x08,
+0x84,
+0x80,
+0x94,
+0x40,
+0x80,
+0x08,
+0x07,
+0x66,
+0x60,
+0x00,
+0x16,
+0xae,
+0x89,
+0x80,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x80,
+0xc8,
+0x68,
+0x20,
+0x01,
+0x48,
+0x20,
+0x68,
+0x00,
+0x00,
+0xd3,
+0x21,
+0x80,
+0x00,
+0x88,
+0x48,
+0x09,
+0x44,
+0x20,
+0x00,
+0x81,
+0x60,
+0x66,
+0x00,
+0x01,
+0x6a,
+0xe8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x9c,
+0x09,
+0x84,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x19,
+0xa0,
+0xa5,
+0x40,
+0xd6,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x0c,
+0xe2,
+0x05,
+0x44,
+0x17,
+0x08,
+0x08,
+0x96,
+0xc0,
+0x00,
+0x1a,
+0x60,
+0xb6,
+0xc0,
+0x00,
+0x1a,
+0x40,
+0x25,
+0x40,
+0x5f,
+0x80,
+0x04,
+0xa5,
+0x44,
+0x1f,
+0xa0,
+0x00,
+0x28,
+0x00,
+0x0a,
+0x68,
+0x00,
+0x00,
+0xd3,
+0x21,
+0x44,
+0x30,
+0x00,
+0x40,
+0xa0,
+0x80,
+0x84,
+0xb8,
+0x50,
+0xc0,
+0x84,
+0x04,
+0x0a,
+0x08,
+0x00,
+0x80,
+0x80,
+0x94,
+0x40,
+0x80,
+0x04,
+0x8a,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x0c,
+0x08,
+0x48,
+0x40,
+0x40,
+0x00,
+0x02,
+0x80,
+0x20,
+0x44,
+0x29,
+0x02,
+0xc0,
+0x20,
+0x68,
+0x20,
+0x01,
+0x4a,
+0x20,
+0x51,
+0x48,
+0x9b,
+0x3f,
+0xc4,
+0x80,
+0x00,
+0x29,
+0x80,
+0xc9,
+0x44,
+0x0d,
+0xc0,
+0x00,
+0x02,
+0x98,
+0x0c,
+0x84,
+0x40,
+0xdc,
+0x00,
+0x00,
+0x29,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0xc0,
+0x00,
+0x02,
+0x98,
+0x0c,
+0x84,
+0x40,
+0xdc,
+0x00,
+0x00,
+0x29,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0x40,
+0x00,
+0x03,
+0x98,
+0x08,
+0x84,
+0x40,
+0x90,
+0x04,
+0x00,
+0x85,
+0x14,
+0x69,
+0x18,
+0x08,
+0xa2,
+0xe0,
+0x9a,
+0x55,
+0x00,
+0xd1,
+0x80,
+0x89,
+0x1a,
+0x0d,
+0x99,
+0xa0,
+0x42,
+0x08,
+0x63,
+0x02,
+0x20,
+0xdb,
+0x57,
+0x06,
+0x99,
+0x80,
+0xc9,
+0x98,
+0x0c,
+0xa0,
+0x86,
+0x30,
+0x22,
+0x0d,
+0xb5,
+0x70,
+0x69,
+0x98,
+0x0c,
+0x99,
+0x80,
+0xca,
+0x08,
+0x63,
+0x02,
+0x20,
+0xdb,
+0x57,
+0x06,
+0x91,
+0x80,
+0xc9,
+0x98,
+0x08,
+0xa0,
+0x86,
+0x20,
+0x22,
+0x0d,
+0x25,
+0x70,
+0x51,
+0x3a,
+0x14,
+0x82,
+0x20,
+0x92,
+0x40,
+0x00,
+0x01,
+0x80,
+0x88,
+0x85,
+0x00,
+0x88,
+0x40,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc1,
+0x50,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0xbd,
+0x84,
+0x80,
+0x82,
+0xe1,
+0x30,
+0x98,
+0x00,
+0x88,
+0x40,
+0x48,
+0x00,
+0x00,
+0x08,
+0x50,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0xb4,
+0xba,
+0x14,
+0x88,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x02,
+0x81,
+0x30,
+0x98,
+0x00,
+0x88,
+0x40,
+0x48,
+0x00,
+0x00,
+0x08,
+0x50,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x12,
+0x84,
+0x04,
+0xab,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x14,
+0x07,
+0x20,
+0x5c,
+0x00,
+0x63,
+0x07,
+0xf6,
+0x5c,
+0x1f,
+0x30,
+0x40,
+0x4a,
+0xa0,
+0x0c,
+0x0a,
+0x04,
+0x21,
+0x84,
+0x04,
+0x84,
+0x60,
+0xa4,
+0x04,
+0x84,
+0xa8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x14,
+0x04,
+0x20,
+0x38,
+0x10,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x52,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x17,
+0x21,
+0x5c,
+0x90,
+0x8a,
+0xc0,
+0x08,
+0x80,
+0xaa,
+0x09,
+0x48,
+0x2e,
+0x32,
+0x23,
+0x0b,
+0xc1,
+0x4c,
+0x68,
+0x00,
+0x00,
+0x06,
+0xa1,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x46,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0xba,
+0x14,
+0x86,
+0xe0,
+0x00,
+0x00,
+0xd6,
+0x60,
+0x00,
+0x00,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0x46,
+0x0a,
+0x41,
+0x48,
+0xc0,
+0x84,
+0x86,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x02,
+0x21,
+0x68,
+0x00,
+0x00,
+0x18,
+0x20,
+0x84,
+0x87,
+0xaa,
+0xbf,
+0xf0,
+0x84,
+0xf0,
+0x85,
+0x90,
+0x10,
+0x20,
+0x63,
+0x29,
+0x8e,
+0x8a,
+0x95,
+0x06,
+0x68,
+0x80,
+0x76,
+0xbc,
+0x05,
+0x98,
+0x40,
+0x21,
+0x5c,
+0x00,
+0x63,
+0xc2,
+0xff,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x44,
+0x39,
+0x00,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x00,
+0x00,
+0x94,
+0x84,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x91,
+0x43,
+0x01,
+0x2e,
+0x80,
+0x26,
+0x10,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x0b,
+0x52,
+0xcd,
+0xc1,
+0x40,
+0x2d,
+0x54,
+0x09,
+0x43,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x01,
+0x40,
+0x60,
+0x5c,
+0x00,
+0x63,
+0xc1,
+0x5f,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x7a,
+0xa0,
+0x05,
+0x98,
+0x40,
+0x22,
+0xba,
+0x0a,
+0x88,
+0x80,
+0xe1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x91,
+0x40,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x61,
+0x8e,
+0x8a,
+0x94,
+0x04,
+0x6a,
+0x06,
+0x21,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x84,
+0x06,
+0x10,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x05,
+0x20,
+0x38,
+0x12,
+0xc8,
+0x40,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xc0,
+0x68,
+0x5c,
+0x00,
+0x73,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x4a,
+0x64,
+0x00,
+0x01,
+0x7a,
+0x07,
+0x5c,
+0xbd,
+0x03,
+0x00,
+0x4c,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x4a,
+0x9c,
+0x00,
+0x08,
+0x40,
+0x48,
+0x84,
+0x07,
+0xa4,
+0x60,
+0xa4,
+0x18,
+0xe8,
+0x86,
+0xe0,
+0x00,
+0x00,
+0xd6,
+0x40,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x17,
+0x20,
+0x00,
+0x00,
+0x0a,
+0x06,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x22,
+0x30,
+0xbc,
+0x15,
+0xc5,
+0xc8,
+0x04,
+0x04,
+0x02,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x09,
+0x56,
+0x20,
+0x00,
+0x00,
+0x04,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x40,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x02,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x0f,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0x66,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0x60,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x02,
+0xe6,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x47,
+0x20,
+0x5c,
+0x00,
+0x63,
+0x07,
+0xf6,
+0x5c,
+0x1f,
+0x30,
+0x40,
+0x4a,
+0xa0,
+0x0c,
+0x0a,
+0x04,
+0x21,
+0x84,
+0x04,
+0x84,
+0x60,
+0xa4,
+0x04,
+0x84,
+0xa8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x14,
+0x44,
+0x20,
+0x38,
+0x10,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x52,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x42,
+0x21,
+0x68,
+0x00,
+0x01,
+0x23,
+0x20,
+0x84,
+0x87,
+0xaa,
+0xbf,
+0xf0,
+0x84,
+0xf0,
+0x85,
+0x90,
+0x10,
+0x20,
+0x63,
+0x29,
+0x8e,
+0x8a,
+0x95,
+0x06,
+0x68,
+0x80,
+0x76,
+0xbc,
+0x05,
+0x98,
+0x40,
+0x21,
+0x5c,
+0x00,
+0x63,
+0xc2,
+0xff,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x44,
+0x39,
+0x00,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x80,
+0x00,
+0x94,
+0x84,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x91,
+0x43,
+0x01,
+0x2e,
+0x80,
+0x26,
+0x10,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x0b,
+0x52,
+0xcd,
+0xc1,
+0x40,
+0x2d,
+0x54,
+0x09,
+0x43,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x01,
+0x40,
+0x60,
+0x5c,
+0x00,
+0x63,
+0xc1,
+0x5f,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x68,
+0x00,
+0x01,
+0x0f,
+0x20,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x7a,
+0xa0,
+0x05,
+0x98,
+0x40,
+0x22,
+0xba,
+0x0a,
+0x88,
+0x80,
+0xe1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x91,
+0x40,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x61,
+0x8e,
+0x8a,
+0x94,
+0x04,
+0x6a,
+0x06,
+0x21,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x84,
+0x06,
+0x10,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x45,
+0x20,
+0x38,
+0x12,
+0xc8,
+0x40,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xc0,
+0x68,
+0x5c,
+0x00,
+0x73,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x4a,
+0x64,
+0x00,
+0x01,
+0x84,
+0x47,
+0x5c,
+0xbd,
+0x03,
+0x00,
+0x4c,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x4a,
+0x9c,
+0x00,
+0x08,
+0x40,
+0x48,
+0x84,
+0x07,
+0xa4,
+0x60,
+0xa4,
+0x18,
+0xe8,
+0x86,
+0xe0,
+0x00,
+0x22,
+0x36,
+0x40,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0x22,
+0x20,
+0x00,
+0x00,
+0x0a,
+0x06,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x22,
+0x30,
+0xbc,
+0x15,
+0xc5,
+0xc8,
+0x04,
+0x04,
+0x02,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x09,
+0x56,
+0x20,
+0x00,
+0x00,
+0x04,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x40,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x84,
+0x48,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x0f,
+0x6e,
+0x00,
+0x02,
+0x23,
+0x66,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x88,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x02,
+0x23,
+0x60,
+0x6c,
+0x70,
+0x28,
+0x84,
+0x48,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x24,
+0x46,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x06,
+0x1e,
+0x20,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x33,
+0x26,
+0x00,
+0x00,
+0x00,
+0xab,
+0xfd,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0x78,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x83,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x8e,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x8e,
+0x20,
+0x68,
+0x00,
+0x01,
+0x83,
+0x24,
+0xa0,
+0x04,
+0x06,
+0xc4,
+0x00,
+0x3a,
+0x60,
+0x88,
+0x40,
+0x09,
+0x44,
+0x20,
+0x02,
+0x20,
+0x44,
+0x68,
+0x00,
+0x01,
+0x78,
+0x21,
+0x86,
+0x00,
+0x98,
+0x80,
+0xe4,
+0xa0,
+0x84,
+0x46,
+0xc4,
+0x00,
+0x39,
+0xe0,
+0x84,
+0x42,
+0x10,
+0x06,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x39,
+0x60,
+0x94,
+0x44,
+0x80,
+0x1a,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3a,
+0xc0,
+0x94,
+0x60,
+0x88,
+0x88,
+0x16,
+0x44,
+0x40,
+0x88,
+0x18,
+0x00,
+0x84,
+0xd0,
+0xd0,
+0x88,
+0x1e,
+0x02,
+0x2c,
+0x40,
+0x51,
+0x62,
+0x80,
+0x82,
+0x4c,
+0x51,
+0x62,
+0x41,
+0x83,
+0x09,
+0x84,
+0x04,
+0xc8,
+0x82,
+0xc9,
+0x68,
+0x20,
+0x01,
+0xf8,
+0x24,
+0x66,
+0x00,
+0x01,
+0xa0,
+0xc8,
+0x46,
+0x08,
+0x08,
+0x82,
+0x0a,
+0x88,
+0x20,
+0x98,
+0x82,
+0x48,
+0x68,
+0x20,
+0x01,
+0xfb,
+0x24,
+0x66,
+0x00,
+0x01,
+0xa0,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x82,
+0x8a,
+0x5c,
+0x82,
+0x00,
+0x81,
+0x20,
+0x88,
+0x20,
+0x98,
+0x80,
+0xa4,
+0x80,
+0x04,
+0x98,
+0x60,
+0x48,
+0x40,
+0x00,
+0x00,
+0x82,
+0x60,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x56,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x82,
+0x00,
+0x80,
+0xa0,
+0x88,
+0x22,
+0x48,
+0x81,
+0x21,
+0x80,
+0x00,
+0x98,
+0x61,
+0xc8,
+0x88,
+0x16,
+0x08,
+0x49,
+0x48,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x62,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x82,
+0x00,
+0x81,
+0xa0,
+0x88,
+0x12,
+0x48,
+0x80,
+0xa1,
+0x80,
+0x00,
+0x98,
+0x61,
+0xc8,
+0x88,
+0x0e,
+0x08,
+0x49,
+0x48,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x6e,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x81,
+0x00,
+0x82,
+0x21,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x88,
+0x1a,
+0x08,
+0x80,
+0xa2,
+0x80,
+0x80,
+0x9a,
+0x20,
+0x64,
+0x84,
+0x14,
+0x88,
+0x51,
+0xc8,
+0x88,
+0x1e,
+0x18,
+0x82,
+0x64,
+0x68,
+0x00,
+0x01,
+0x56,
+0x23,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x40,
+0x00,
+0x02,
+0x18,
+0x40,
+0x5c,
+0x83,
+0x08,
+0x81,
+0xa1,
+0x5c,
+0x88,
+0x00,
+0x81,
+0x22,
+0x5c,
+0x81,
+0x08,
+0x08,
+0xc8,
+0x80,
+0xa4,
+0x88,
+0x81,
+0x61,
+0x68,
+0x00,
+0x01,
+0x62,
+0x20,
+0x81,
+0x08,
+0x98,
+0x82,
+0x24,
+0x88,
+0x1e,
+0x26,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x00,
+0x40,
+0x5c,
+0x83,
+0x08,
+0x81,
+0xa1,
+0x5c,
+0x81,
+0x00,
+0x80,
+0xa2,
+0x5c,
+0x88,
+0x08,
+0x08,
+0xc8,
+0x80,
+0xac,
+0x88,
+0x80,
+0xe1,
+0x68,
+0x00,
+0x01,
+0x6e,
+0x20,
+0x81,
+0x00,
+0x98,
+0x82,
+0x24,
+0x88,
+0x1e,
+0x26,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x00,
+0x40,
+0x5c,
+0x83,
+0x00,
+0x81,
+0xa4,
+0x5c,
+0x88,
+0x08,
+0x81,
+0x20,
+0x82,
+0x04,
+0x88,
+0x22,
+0xc8,
+0x88,
+0x1e,
+0x46,
+0x60,
+0x00,
+0x1a,
+0x14,
+0x08,
+0x81,
+0x24,
+0x88,
+0x0a,
+0x08,
+0x63,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa1,
+0x40,
+0x88,
+0x0a,
+0x48,
+0x81,
+0xa0,
+0x40,
+0x00,
+0x00,
+0x63,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x06,
+0x4d,
+0x24,
+0x88,
+0x1a,
+0x08,
+0x80,
+0x36,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x64,
+0xba,
+0x14,
+0x88,
+0x43,
+0x48,
+0x40,
+0x00,
+0x02,
+0x80,
+0x30,
+0x68,
+0x00,
+0x01,
+0x89,
+0x24,
+0xab,
+0xfc,
+0x06,
+0xc0,
+0x00,
+0x11,
+0x40,
+0xb6,
+0xc0,
+0x00,
+0x12,
+0x60,
+0x06,
+0xc0,
+0x00,
+0x29,
+0x40,
+0x16,
+0xc0,
+0x00,
+0x32,
+0x80,
+0xa8,
+0x60,
+0x09,
+0x88,
+0x05,
+0x18,
+0x80,
+0xd0,
+0x88,
+0x14,
+0xb8,
+0x81,
+0xf6,
+0x68,
+0x00,
+0x01,
+0x7e,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa7,
+0x28,
+0x40,
+0x00,
+0x00,
+0x40,
+0x08,
+0x68,
+0x00,
+0x01,
+0x89,
+0x21,
+0x68,
+0x20,
+0x01,
+0xdd,
+0x20,
+0x68,
+0x00,
+0x01,
+0x7e,
+0x24,
+0x5c,
+0x83,
+0x00,
+0x40,
+0x88,
+0x80,
+0xa0,
+0xb5,
+0x70,
+0x9c,
+0x02,
+0x20,
+0x98,
+0x40,
+0x08,
+0x88,
+0x36,
+0x45,
+0x70,
+0x94,
+0x08,
+0x24,
+0x08,
+0x82,
+0xe1,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0x54,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x98,
+0x00,
+0x98,
+0x83,
+0x20,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x84,
+0x04,
+0x86,
+0x80,
+0x00,
+0x16,
+0x02,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x88,
+0x82,
+0x09,
+0x55,
+0x01,
+0x28,
+0x83,
+0x20,
+0x5c,
+0x83,
+0x00,
+0x82,
+0xa4,
+0x80,
+0x20,
+0x88,
+0x60,
+0x49,
+0x40,
+0x00,
+0x00,
+0x82,
+0x60,
+0x68,
+0x00,
+0x01,
+0x54,
+0x21,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc1,
+0x22,
+0x66,
+0x00,
+0x01,
+0xa1,
+0xa0,
+0x5c,
+0x83,
+0x00,
+0x82,
+0xa0,
+0x68,
+0x00,
+0x01,
+0x60,
+0x21,
+0x80,
+0x20,
+0x88,
+0x82,
+0xe0,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc3,
+0x22,
+0x66,
+0x00,
+0x01,
+0xa1,
+0xa0,
+0x88,
+0x10,
+0x83,
+0x69,
+0x00,
+0x98,
+0x00,
+0x96,
+0xc4,
+0x00,
+0x3a,
+0x80,
+0xb5,
+0x80,
+0xf4,
+0x18,
+0xe8,
+0x9b,
+0xc0,
+0x3a,
+0x55,
+0x01,
+0x70,
+0x80,
+0x88,
+0xb0,
+0x00,
+0xd3,
+0x69,
+0x00,
+0x98,
+0x00,
+0x03,
+0x01,
+0xc0,
+0xbc,
+0x03,
+0xa5,
+0x50,
+0x18,
+0x08,
+0x00,
+0x8b,
+0x00,
+0x08,
+0x36,
+0x90,
+0x19,
+0x80,
+0x48,
+0x30,
+0x1e,
+0x0b,
+0xc0,
+0x12,
+0xb0,
+0x00,
+0xe3,
+0x20,
+0x28,
+0xbc,
+0x05,
+0x98,
+0x82,
+0x20,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x21,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x60,
+0x6c,
+0x40,
+0x03,
+0xaa,
+0x08,
+0x6c,
+0x00,
+0x03,
+0x94,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x96,
+0x48,
+0xa0,
+0x0c,
+0x08,
+0x40,
+0x09,
+0x88,
+0x06,
+0x06,
+0x82,
+0x00,
+0x0a,
+0xa2,
+0x46,
+0x80,
+0x00,
+0x1c,
+0x62,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x08,
+0x82,
+0xa0,
+0x88,
+0x0c,
+0x8a,
+0x00,
+0xc0,
+0x84,
+0x00,
+0x98,
+0x81,
+0x60,
+0x68,
+0x20,
+0x00,
+0xc2,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc8,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x66,
+0x00,
+0x01,
+0xaa,
+0x88,
+0x55,
+0x01,
+0x28,
+0x80,
+0x88,
+0x6c,
+0x00,
+0x03,
+0x94,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x68,
+0x00,
+0x01,
+0x81,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x42,
+0x02,
+0xb8,
+0x40,
+0xc8,
+0x2a,
+0x7e,
+0x46,
+0xc0,
+0x00,
+0x39,
+0x44,
+0x86,
+0xc0,
+0x00,
+0x30,
+0x47,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x96,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x68,
+0x00,
+0x01,
+0x8c,
+0x20,
+0x00,
+0x00,
+0x04,
+0x20,
+0x37,
+0x84,
+0x00,
+0x88,
+0x40,
+0xc8,
+0x2a,
+0x7e,
+0x46,
+0xc0,
+0x00,
+0x39,
+0x64,
+0x86,
+0xc0,
+0x00,
+0x31,
+0xa7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x00,
+0x80,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x04,
+0x7a,
+0x6c,
+0x00,
+0x03,
+0x1a,
+0x7a,
+0x66,
+0x00,
+0x01,
+0xa6,
+0x68,
+0x84,
+0x20,
+0x98,
+0x81,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x46,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa6,
+0x68,
+0x40,
+0x00,
+0x00,
+0x42,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x4a,
+0x48,
+0x68,
+0x00,
+0x06,
+0x80,
+0x20,
+0x88,
+0x1b,
+0x6b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x40,
+0x68,
+0x00,
+0x06,
+0x1c,
+0xa0,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x33,
+0x26,
+0x00,
+0x00,
+0x00,
+0x84,
+0x00,
+0x88,
+0x40,
+0x8a,
+0x57,
+0x0d,
+0x03,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x00,
+0x00,
+0x00,
+0xa2,
+0x02,
+0x48,
+0x60,
+0x08,
+0x44,
+0x20,
+0x00,
+0x60,
+0x89,
+0xba,
+0x14,
+0x80,
+0x89,
+0x80,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0xa0,
+0x04,
+0x08,
+0x41,
+0x88,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x0a,
+0x2e,
+0x13,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x81,
+0x02,
+0x22,
+0x23,
+0x81,
+0x80,
+0x94,
+0x40,
+0x80,
+0x05,
+0x80,
+0x95,
+0x00,
+0xa0,
+0x20,
+0x8c,
+0x19,
+0x80,
+0x09,
+0x80,
+0x84,
+0x9a,
+0x1c,
+0x64,
+0x82,
+0x00,
+0x84,
+0x42,
+0x00,
+0x06,
+0x00,
+0x98,
+0x48,
+0x08,
+0x44,
+0x0c,
+0x02,
+0xbf,
+0xd0,
+0x98,
+0x00,
+0x9a,
+0x24,
+0xe4,
+0x80,
+0x84,
+0x98,
+0x80,
+0x61,
+0x88,
+0x0e,
+0x48,
+0x81,
+0x62,
+0x88,
+0x1e,
+0x08,
+0x82,
+0x76,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0xa0,
+0x80,
+0x08,
+0x80,
+0x20,
+0x88,
+0x0a,
+0x2a,
+0x00,
+0x40,
+0x88,
+0x06,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x85,
+0x50,
+0x12,
+0xa1,
+0x06,
+0x48,
+0x80,
+0x20,
+0x88,
+0x12,
+0x2a,
+0x04,
+0x60,
+0x84,
+0x00,
+0x95,
+0x70,
+0x94,
+0x05,
+0x08,
+0x95,
+0x70,
+0xa0,
+0x08,
+0x1a,
+0x09,
+0x80,
+0x08,
+0x59,
+0x01,
+0x00,
+0x44,
+0xc8,
+0xbc,
+0x0e,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x17,
+0x38,
+0x50,
+0x08,
+0x2e,
+0x12,
+0xd5,
+0xb4,
+0xa0,
+0x05,
+0x0c,
+0x99,
+0x80,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x22,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0xe5,
+0x36,
+0x10,
+0x58,
+0x50,
+0xc9,
+0xbc,
+0x0b,
+0x78,
+0x50,
+0x08,
+0x28,
+0x12,
+0xd5,
+0xb4,
+0xa0,
+0x05,
+0x0c,
+0x99,
+0x80,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x22,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x25,
+0x85,
+0x0c,
+0x80,
+0x00,
+0x00,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0xc0,
+0x08,
+0x44,
+0x20,
+0x03,
+0xa1,
+0x11,
+0x98,
+0x00,
+0x96,
+0x80,
+0x39,
+0x4b,
+0xb8,
+0x84,
+0x42,
+0x00,
+0x3a,
+0x14,
+0x82,
+0x2b,
+0x80,
+0x46,
+0x08,
+0x09,
+0x83,
+0x08,
+0x55,
+0x01,
+0x12,
+0xbf,
+0xf0,
+0x5b,
+0x0c,
+0x30,
+0x80,
+0x4a,
+0x44,
+0x58,
+0x81,
+0x84,
+0x48,
+0x44,
+0x21,
+0x81,
+0x80,
+0x41,
+0x5b,
+0x04,
+0x32,
+0xc0,
+0x20,
+0x44,
+0x40,
+0x81,
+0x84,
+0x48,
+0x5b,
+0x0a,
+0x29,
+0x84,
+0x8a,
+0x44,
+0x58,
+0x01,
+0x84,
+0x0b,
+0x44,
+0x39,
+0x00,
+0x80,
+0x09,
+0x44,
+0x39,
+0x81,
+0x80,
+0xca,
+0x5c,
+0x00,
+0xeb,
+0xa1,
+0x11,
+0x2a,
+0x02,
+0x74,
+0x46,
+0x88,
+0x1a,
+0x14,
+0x14,
+0x46,
+0x88,
+0x18,
+0x34,
+0x34,
+0xc0,
+0xc0,
+0x9a,
+0x00,
+0x09,
+0x83,
+0x8b,
+0x44,
+0x08,
+0x01,
+0x83,
+0x08,
+0x54,
+0x07,
+0xd1,
+0x83,
+0x41,
+0x54,
+0x08,
+0x41,
+0x83,
+0x0b,
+0x44,
+0x48,
+0x83,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x03,
+0xae,
+0x08,
+0x9a,
+0x00,
+0x24,
+0xc2,
+0x40,
+0x18,
+0x34,
+0x96,
+0x82,
+0x00,
+0x1d,
+0xd2,
+0x02,
+0x81,
+0x79,
+0x80,
+0x04,
+0x29,
+0x80,
+0x4b,
+0x44,
+0x60,
+0x00,
+0x00,
+0x40,
+0xba,
+0x14,
+0x88,
+0x40,
+0x40,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x38,
+0x13,
+0x22,
+0x58,
+0xb0,
+0xbc,
+0x05,
+0x0b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0xcc,
+0x21,
+0x68,
+0x20,
+0x01,
+0xe1,
+0x24,
+0x5c,
+0x81,
+0x08,
+0x48,
+0x02,
+0x5c,
+0x82,
+0x00,
+0x20,
+0x8a,
+0x68,
+0x00,
+0x01,
+0xd0,
+0x20,
+0x30,
+0x19,
+0x04,
+0x20,
+0xec,
+0x00,
+0x00,
+0x08,
+0x02,
+0x8a,
+0x68,
+0x00,
+0x01,
+0xcc,
+0x20,
+0x55,
+0x02,
+0x92,
+0xc0,
+0xa0,
+0x58,
+0x01,
+0x00,
+0x00,
+0x52,
+0xbc,
+0x03,
+0x56,
+0xc0,
+0x00,
+0x3a,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0x23,
+0x00,
+0xa0,
+0xbc,
+0x02,
+0x36,
+0xc0,
+0x00,
+0x3a,
+0x24,
+0x83,
+0x01,
+0xa8,
+0xbc,
+0x03,
+0x56,
+0xc0,
+0x00,
+0x3a,
+0x44,
+0x90,
+0x00,
+0x00,
+0x84,
+0x10,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x04,
+0x3b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xa6,
+0x49,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x80,
+0x00,
+0xb5,
+0x70,
+0xe1,
+0x06,
+0x00,
+0x35,
+0x84,
+0x68,
+0x00,
+0x28,
+0x94,
+0x20,
+0x1d,
+0x98,
+0xe8,
+0x85,
+0x70,
+0xb8,
+0x98,
+0x20,
+0x2b,
+0x00,
+0x0c,
+0x30,
+0x8c,
+0x8b,
+0xc0,
+0x23,
+0x40,
+0x00,
+0x03,
+0x00,
+0x0a,
+0x5b,
+0x4e,
+0x08,
+0x60,
+0x83,
+0x36,
+0x80,
+0x05,
+0xb4,
+0xc0,
+0x18,
+0x00,
+0xb5,
+0xb4,
+0xa0,
+0x98,
+0x04,
+0xa5,
+0x78,
+0xdf,
+0x98,
+0x00,
+0x95,
+0x84,
+0x7c,
+0x18,
+0x04,
+0xab,
+0xc3,
+0x0b,
+0x2f,
+0x1a,
+0xd3,
+0x08,
+0xe8,
+0xbc,
+0x2d,
+0x33,
+0x20,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0xbc,
+0x04,
+0x86,
+0xc4,
+0x00,
+0x3c,
+0xc0,
+0x83,
+0x20,
+0x10,
+0xbc,
+0x08,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x0f,
+0x56,
+0xc4,
+0x00,
+0x3c,
+0xa0,
+0xa2,
+0xe1,
+0xad,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x49,
+0xbc,
+0x09,
+0x73,
+0x01,
+0x28,
+0xbc,
+0x07,
+0x36,
+0xc4,
+0x00,
+0x3c,
+0x80,
+0xa2,
+0x81,
+0xad,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x49,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0x32,
+0x06,
+0x8b,
+0xc0,
+0x63,
+0x68,
+0x00,
+0x01,
+0xce,
+0x21,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x7a,
+0x84,
+0x8f,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x6a,
+0x38,
+0x00,
+0xd6,
+0x80,
+0x00,
+0x1c,
+0xe2,
+0x16,
+0xc0,
+0x00,
+0x39,
+0xc4,
+0x98,
+0x48,
+0xc8,
+0x68,
+0x20,
+0x00,
+0xfc,
+0x21,
+0x39,
+0x0a,
+0x18,
+0x48,
+0x08,
+0x80,
+0x24,
+0x88,
+0x40,
+0x48,
+0xa0,
+0x06,
+0x08,
+0x48,
+0x88,
+0x80,
+0x24,
+0x84,
+0x60,
+0xa4,
+0x00,
+0x2c,
+0x88,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0x68,
+0x00,
+0x01,
+0x54,
+0x20,
+0x68,
+0x00,
+0x01,
+0x60,
+0x21,
+0x68,
+0x00,
+0x01,
+0x6c,
+0x22,
+0x55,
+0x01,
+0x33,
+0x80,
+0x00,
+0x55,
+0x03,
+0xb0,
+0x00,
+0x7a,
+0x5d,
+0x0c,
+0x30,
+0x08,
+0x7a,
+0x32,
+0x2b,
+0x0b,
+0xff,
+0xaa,
+0x40,
+0x00,
+0x00,
+0x10,
+0x7a,
+0x68,
+0x00,
+0x01,
+0x85,
+0x22,
+0x68,
+0x00,
+0x01,
+0x90,
+0x21,
+0x68,
+0x00,
+0x01,
+0x7a,
+0x20,
+0x55,
+0x03,
+0x20,
+0x10,
+0x7a,
+0x5d,
+0x08,
+0x20,
+0x08,
+0x7a,
+0x32,
+0x26,
+0x0b,
+0xff,
+0xaa,
+0x80,
+0x07,
+0xa6,
+0x80,
+0x00,
+0x1c,
+0xc2,
+0x06,
+0x82,
+0x00,
+0x0f,
+0xc2,
+0x15,
+0xc8,
+0x20,
+0x80,
+0x07,
+0xa5,
+0xc0,
+0x16,
+0x04,
+0x0f,
+0xa8,
+0x00,
+0x7a,
+0xa0,
+0x08,
+0x08,
+0x48,
+0x0a,
+0x80,
+0x2c,
+0xa8,
+0x40,
+0x4a,
+0xa0,
+0x06,
+0x08,
+0x48,
+0x8a,
+0x80,
+0x2c,
+0xa6,
+0x80,
+0x00,
+0x1c,
+0xa2,
+0x16,
+0x80,
+0x00,
+0x61,
+0xca,
+0xc6,
+0xc4,
+0x00,
+0x3a,
+0xa0,
+0x08,
+0x02,
+0xca,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x6c,
+0x84,
+0x85,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x04,
+0x88,
+0x48,
+0xd0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xa0,
+0x05,
+0x89,
+0x8e,
+0x88,
+0x94,
+0x06,
+0x4a,
+0x00,
+0x08,
+0xa0,
+0x22,
+0x18,
+0x48,
+0x60,
+0xa0,
+0xe3,
+0x09,
+0x40,
+0x64,
+0xa0,
+0x01,
+0x0a,
+0x02,
+0x01,
+0x84,
+0x86,
+0x0a,
+0x08,
+0x80,
+0x84,
+0x07,
+0xaa,
+0x06,
+0xa0,
+0x94,
+0x06,
+0x4a,
+0x04,
+0x40,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x84,
+0x0e,
+0xc6,
+0x80,
+0x00,
+0x6e,
+0x2a,
+0xcb,
+0xa1,
+0x48,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x94,
+0x5a,
+0x83,
+0x20,
+0x00,
+0xbc,
+0x0a,
+0x59,
+0x46,
+0x28,
+0x22,
+0x90,
+0x42,
+0xab,
+0xe0,
+0x23,
+0x08,
+0x06,
+0x82,
+0x00,
+0x1f,
+0xe2,
+0x19,
+0x84,
+0x28,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x21,
+0xba,
+0x01,
+0x0b,
+0xa1,
+0x40,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0xa0,
+0x05,
+0x09,
+0x40,
+0x64,
+0xa0,
+0x01,
+0x0a,
+0x02,
+0x01,
+0x84,
+0x86,
+0x0a,
+0x0e,
+0x60,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x20,
+0x64,
+0x00,
+0x01,
+0x84,
+0x47,
+0x64,
+0x00,
+0x01,
+0x71,
+0x87,
+0x39,
+0x00,
+0x89,
+0x48,
+0x0c,
+0x51,
+0xb1,
+0x21,
+0x48,
+0x0e,
+0x51,
+0xa1,
+0xb1,
+0x48,
+0x08,
+0x51,
+0x90,
+0x11,
+0x48,
+0x28,
+0x29,
+0x1a,
+0x42,
+0x90,
+0xa4,
+0x54,
+0x81,
+0x23,
+0xa1,
+0x48,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x80,
+0x40,
+0x40,
+0x00,
+0x51,
+0xf0,
+0x23,
+0x07,
+0xfa,
+0x28,
+0x8a,
+0x15,
+0x1e,
+0x02,
+0x14,
+0x84,
+0x12,
+0x88,
+0xa1,
+0x51,
+0xd0,
+0x21,
+0x48,
+0x41,
+0x28,
+0x8a,
+0x15,
+0x44,
+0x40,
+0x14,
+0x84,
+0x1b,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x00,
+0x00,
+0x00,
+0x94,
+0x82,
+0xc5,
+0xd4,
+0x83,
+0x14,
+0x8a,
+0xc2,
+0x32,
+0x36,
+0x3a,
+0x90,
+0x43,
+0xa9,
+0x86,
+0x54,
+0x89,
+0xa3,
+0xa1,
+0x48,
+0x94,
+0x07,
+0x40,
+0x00,
+0x00,
+0x94,
+0x03,
+0x85,
+0x15,
+0x02,
+0x30,
+0x7f,
+0xa2,
+0x88,
+0xa1,
+0x54,
+0x44,
+0x01,
+0x48,
+0x61,
+0xba,
+0x14,
+0x89,
+0x48,
+0xe0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x80,
+0x4a,
+0x02,
+0xc0,
+0x5c,
+0x80,
+0x92,
+0xbf,
+0xf0,
+0x62,
+0x00,
+0x00,
+0x00,
+0xb4,
+0x5c,
+0x81,
+0x02,
+0x40,
+0x02,
+0x5c,
+0x00,
+0x00,
+0x40,
+0x24,
+0x60,
+0x00,
+0x00,
+0x00,
+0x35,
+0x8d,
+0x05,
+0x80,
+0x00,
+0x00,
+0x94,
+0x88,
+0xc9,
+0x52,
+0xc4,
+0x00,
+0x00,
+0x0a,
+0x10,
+0x0a,
+0x8d,
+0x03,
+0x2b,
+0xb1,
+0x82,
+0xba,
+0x14,
+0x88,
+0x40,
+0x64,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x5c,
+0x80,
+0x52,
+0xbf,
+0xf0,
+0xa0,
+0x2c,
+0x0a,
+0x40,
+0x02,
+0x62,
+0x00,
+0x00,
+0x00,
+0xa4,
+0x5c,
+0x81,
+0x08,
+0x40,
+0x24,
+0x5c,
+0x80,
+0x82,
+0x06,
+0x61,
+0xbb,
+0x00,
+0x06,
+0x00,
+0x00,
+0x00,
+0x03,
+0x58,
+0xd0,
+0xd8,
+0x00,
+0x00,
+0x09,
+0x53,
+0x0c,
+0x94,
+0x94,
+0x40,
+0x00,
+0x00,
+0xa1,
+0x00,
+0xab,
+0xa1,
+0x48,
+0x84,
+0x06,
+0x4a,
+0x80,
+0x10,
+0xa0,
+0x06,
+0x19,
+0x48,
+0x28,
+0x55,
+0x5e,
+0x02,
+0x08,
+0x0a,
+0x51,
+0x90,
+0x01,
+0x50,
+0x2a,
+0x54,
+0x04,
+0x22,
+0x14,
+0x13,
+0x95,
+0x82,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x04,
+0x20,
+0xcc,
+0x08,
+0x07,
+0x69,
+0x82,
+0x22,
+0x32,
+0x0c,
+0x0b,
+0xc1,
+0x38,
+0xa0,
+0x81,
+0x02,
+0xa7,
+0x80,
+0x22,
+0x88,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x1b,
+0x56,
+0x20,
+0x00,
+0x00,
+0x08,
+0x6a,
+0x00,
+0x01,
+0x00,
+0x00,
+0x09,
+0x82,
+0x20,
+0x88,
+0x16,
+0x18,
+0x80,
+0xc8,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x18,
+0x80,
+0x80,
+0x55,
+0x08,
+0x22,
+0x08,
+0x21,
+0xbc,
+0x0d,
+0x79,
+0x49,
+0x28,
+0x42,
+0x05,
+0x39,
+0x50,
+0x60,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x8a,
+0x10,
+0x00,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x20,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x94,
+0x5a,
+0xc5,
+0x90,
+0x30,
+0x20,
+0x2c,
+0x04,
+0x20,
+0x34,
+0x2b,
+0xff,
+0x08,
+0x80,
+0x76,
+0xa0,
+0x65,
+0x98,
+0x80,
+0x36,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x01,
+0x0a,
+0x06,
+0x61,
+0x88,
+0x0e,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x08,
+0x80,
+0xa0,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x10,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x04,
+0x60,
+0x94,
+0x5a,
+0xc3,
+0x20,
+0x60,
+0xbc,
+0x16,
+0x8a,
+0xbf,
+0xf0,
+0x55,
+0x3f,
+0x23,
+0x00,
+0xfe,
+0x5b,
+0x48,
+0x13,
+0x00,
+0x28,
+0x37,
+0x08,
+0x52,
+0xe1,
+0x76,
+0x62,
+0x00,
+0x00,
+0x00,
+0x26,
+0x36,
+0x80,
+0x05,
+0x04,
+0xc9,
+0x18,
+0xeb,
+0x52,
+0xf8,
+0x12,
+0x2b,
+0x96,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x02,
+0xb9,
+0x83,
+0x88,
+0x36,
+0x10,
+0x4a,
+0x00,
+0x69,
+0x64,
+0x00,
+0x01,
+0xbd,
+0xef,
+0xa8,
+0x01,
+0x08,
+0x80,
+0x76,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xf6,
+0x83,
+0x80,
+0x64,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x1e,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x10,
+0xa0,
+0x06,
+0x19,
+0x48,
+0x28,
+0x55,
+0x5e,
+0x12,
+0x08,
+0x0a,
+0x51,
+0x90,
+0x91,
+0x50,
+0x28,
+0x68,
+0x20,
+0x00,
+0x00,
+0x2c,
+0x54,
+0x00,
+0xa1,
+0x8b,
+0x00,
+0x54,
+0x01,
+0x22,
+0x14,
+0x12,
+0x95,
+0x02,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x04,
+0x20,
+0xcc,
+0x08,
+0x07,
+0x69,
+0x82,
+0x22,
+0x32,
+0x0c,
+0x0b,
+0xc1,
+0x38,
+0xa0,
+0x81,
+0x02,
+0xa7,
+0x80,
+0x22,
+0x88,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x1b,
+0x56,
+0x20,
+0x00,
+0x00,
+0x08,
+0x6a,
+0x00,
+0x01,
+0x00,
+0x00,
+0x09,
+0x82,
+0x20,
+0x88,
+0x16,
+0x18,
+0x80,
+0xc8,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x18,
+0x80,
+0x80,
+0x55,
+0x08,
+0x22,
+0x08,
+0x21,
+0xbc,
+0x0d,
+0x79,
+0x49,
+0x28,
+0x42,
+0x05,
+0x39,
+0x50,
+0x60,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x8a,
+0x10,
+0x00,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x20,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x94,
+0x5a,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x08,
+0x80,
+0x76,
+0x42,
+0x10,
+0xc2,
+0x00,
+0x01,
+0xa0,
+0x2a,
+0x03,
+0x21,
+0x80,
+0xbc,
+0x19,
+0x8a,
+0x08,
+0x71,
+0x2a,
+0x78,
+0x02,
+0x28,
+0xc4,
+0x32,
+0x02,
+0x0b,
+0xc2,
+0x45,
+0x62,
+0x00,
+0x00,
+0x00,
+0xf4,
+0xa0,
+0xa3,
+0x08,
+0x80,
+0xe0,
+0x88,
+0x16,
+0x10,
+0x00,
+0x00,
+0x88,
+0x0a,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x08,
+0x81,
+0x21,
+0x88,
+0x0a,
+0x0a,
+0x08,
+0x21,
+0x84,
+0x02,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x02,
+0x1b,
+0xc1,
+0x07,
+0xa0,
+0x63,
+0x18,
+0x80,
+0x36,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x02,
+0x0a,
+0x06,
+0x41,
+0x88,
+0x0e,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x08,
+0x80,
+0xa0,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x20,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x04,
+0x60,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x05,
+0x9a,
+0xbf,
+0xf0,
+0x94,
+0x82,
+0xc5,
+0x90,
+0x50,
+0x20,
+0xa4,
+0xa8,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x64,
+0x20,
+0x3c,
+0x20,
+0x06,
+0x18,
+0x50,
+0x20,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xb6,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x0b,
+0x68,
+0x80,
+0x20,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x10,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x19,
+0x46,
+0x2c,
+0xb0,
+0x70,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x02,
+0x16,
+0x40,
+0x00,
+0x1d,
+0x0c,
+0x7b,
+0xa1,
+0x40,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xae,
+0x59,
+0x01,
+0x82,
+0xbf,
+0xd0,
+0xbd,
+0x7b,
+0x88,
+0x80,
+0x76,
+0x32,
+0x07,
+0x0b,
+0xd6,
+0x20,
+0x32,
+0x0b,
+0x0b,
+0xd5,
+0x40,
+0x32,
+0x0f,
+0x0b,
+0xd3,
+0x40,
+0x32,
+0x13,
+0x0b,
+0xd2,
+0x60,
+0x32,
+0x17,
+0x0b,
+0xd0,
+0x60,
+0x32,
+0x1b,
+0x0b,
+0xcf,
+0x80,
+0x32,
+0x1f,
+0x0b,
+0xce,
+0x00,
+0x32,
+0x23,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x00,
+0xf9,
+0x8e,
+0x8b,
+0x32,
+0x03,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x82,
+0x8b,
+0xcb,
+0x68,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0xac,
+0x32,
+0x27,
+0x0b,
+0xcb,
+0x20,
+0x32,
+0x03,
+0x8b,
+0xc8,
+0x20,
+0x32,
+0x2b,
+0x0b,
+0xc8,
+0x00,
+0x32,
+0x03,
+0x8b,
+0xc4,
+0xf0,
+0x32,
+0x2f,
+0x0b,
+0xc4,
+0xd0,
+0x32,
+0x03,
+0x8b,
+0xc0,
+0x20,
+0x32,
+0x33,
+0x0b,
+0xd5,
+0xf1,
+0x59,
+0x05,
+0x02,
+0xc0,
+0x20,
+0xbc,
+0x2c,
+0x86,
+0x80,
+0x00,
+0x09,
+0x82,
+0x13,
+0x21,
+0xa0,
+0xbd,
+0x58,
+0x9a,
+0x08,
+0x00,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0x66,
+0x00,
+0x01,
+0xbc,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x98,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0xa0,
+0x02,
+0x08,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x01,
+0xbc,
+0xa8,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xa0,
+0x6c,
+0x00,
+0x01,
+0x30,
+0x0b,
+0x51,
+0xa1,
+0xf8,
+0x40,
+0x08,
+0x23,
+0x42,
+0x45,
+0x90,
+0x1c,
+0x04,
+0x04,
+0x86,
+0xc0,
+0x00,
+0x13,
+0x04,
+0xb4,
+0x20,
+0x34,
+0xa0,
+0x16,
+0x85,
+0xc0,
+0x97,
+0x14,
+0x02,
+0xf6,
+0xc0,
+0x00,
+0x13,
+0x20,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x04,
+0x02,
+0x41,
+0xb8,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xbd,
+0x34,
+0x72,
+0x49,
+0xb8,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xbd,
+0x30,
+0x7a,
+0x40,
+0x80,
+0x80,
+0x80,
+0xa5,
+0x16,
+0x1a,
+0x08,
+0x0e,
+0x19,
+0x40,
+0x74,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x0a,
+0x1a,
+0x40,
+0x80,
+0x84,
+0x80,
+0xa2,
+0x2c,
+0x34,
+0x40,
+0x00,
+0x01,
+0x40,
+0x74,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x70,
+0xbd,
+0x16,
+0x73,
+0x20,
+0xa0,
+0xbc,
+0x21,
+0x03,
+0x21,
+0xa0,
+0xbd,
+0x12,
+0x93,
+0x81,
+0xc6,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x0b,
+0x25,
+0x9b,
+0x8b,
+0xc0,
+0x69,
+0x38,
+0x11,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0xc8,
+0x02,
+0xc2,
+0xc0,
+0xbc,
+0x06,
+0x96,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x00,
+0x00,
+0x02,
+0x41,
+0xbe,
+0xa0,
+0x67,
+0x18,
+0x02,
+0x4a,
+0x68,
+0x00,
+0x08,
+0x14,
+0x22,
+0x84,
+0x06,
+0x26,
+0x82,
+0x00,
+0x22,
+0x02,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xcf,
+0x37,
+0x68,
+0x20,
+0x02,
+0x20,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xce,
+0x77,
+0x32,
+0x0a,
+0x0b,
+0xc2,
+0x00,
+0x32,
+0x1a,
+0x0b,
+0xce,
+0x39,
+0x38,
+0x1c,
+0x66,
+0xc0,
+0x00,
+0x03,
+0x60,
+0xb2,
+0x59,
+0xb8,
+0xbc,
+0x05,
+0x93,
+0x81,
+0x14,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x52,
+0xc8,
+0x02,
+0xc2,
+0xc0,
+0xbc,
+0x06,
+0x96,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x00,
+0x00,
+0x02,
+0x41,
+0xbe,
+0xa0,
+0x67,
+0x18,
+0x02,
+0x4a,
+0x68,
+0x00,
+0x08,
+0x01,
+0x22,
+0x84,
+0x06,
+0x26,
+0x82,
+0x00,
+0x21,
+0xf2,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xcc,
+0x57,
+0x68,
+0x20,
+0x02,
+0x1f,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xcb,
+0x97,
+0x32,
+0x0a,
+0x0b,
+0xcb,
+0x78,
+0x38,
+0x1c,
+0x76,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x82,
+0x59,
+0xe0,
+0xbc,
+0xb2,
+0x83,
+0x81,
+0x12,
+0x25,
+0x88,
+0x0b,
+0xc0,
+0x69,
+0x39,
+0x2c,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0x05,
+0x20,
+0x68,
+0x00,
+0x07,
+0xfb,
+0x21,
+0x52,
+0x0f,
+0x38,
+0x00,
+0x61,
+0x51,
+0xb1,
+0xb2,
+0xc2,
+0x70,
+0x80,
+0x24,
+0xb0,
+0x00,
+0x00,
+0x94,
+0x02,
+0xf5,
+0x19,
+0x1f,
+0x94,
+0x0a,
+0xc2,
+0x91,
+0x3f,
+0x29,
+0x1b,
+0xe6,
+0xc4,
+0x00,
+0x43,
+0xc4,
+0xab,
+0xc9,
+0x77,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0xa0,
+0x82,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x02,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x03,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc8,
+0x17,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc7,
+0x57,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0xa0,
+0x82,
+0x1a,
+0x00,
+0x30,
+0x88,
+0x0e,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x12,
+0x18,
+0x80,
+0xa0,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x05,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x04,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x90,
+0x40,
+0x00,
+0x03,
+0xc5,
+0x77,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc4,
+0xb7,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0xa0,
+0x82,
+0x1a,
+0x00,
+0x40,
+0x88,
+0x0e,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x12,
+0x18,
+0x80,
+0xa0,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x06,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x04,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x90,
+0x40,
+0x00,
+0x03,
+0xc2,
+0xd7,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc2,
+0x17,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0xa0,
+0x82,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x04,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x03,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xb7,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x0b,
+0x19,
+0x46,
+0x2c,
+0xb0,
+0x78,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x05,
+0x0b,
+0x07,
+0x8e,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xfb,
+0xc7,
+0x64,
+0x00,
+0x01,
+0xea,
+0xe7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xac,
+0xb0,
+0x78,
+0x62,
+0x89,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x60,
+0x38,
+0x08,
+0x63,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xf4,
+0xa7,
+0x64,
+0x00,
+0x01,
+0xeb,
+0xe7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x07,
+0x60,
+0xab,
+0xfe,
+0x09,
+0x40,
+0x2c,
+0x59,
+0x0f,
+0x00,
+0x80,
+0x76,
+0x42,
+0x36,
+0x42,
+0x04,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x23,
+0xa0,
+0xbc,
+0x17,
+0x03,
+0x23,
+0xe0,
+0xbc,
+0x7a,
+0x9a,
+0x40,
+0x40,
+0x68,
+0x00,
+0x00,
+0x08,
+0x21,
+0x40,
+0x00,
+0x01,
+0x88,
+0x09,
+0x66,
+0x00,
+0x01,
+0x2f,
+0xc8,
+0x5c,
+0x00,
+0x71,
+0x88,
+0x48,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x40,
+0x40,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc6,
+0x57,
+0x32,
+0x0b,
+0x0b,
+0xc3,
+0xe8,
+0x38,
+0x15,
+0x46,
+0x80,
+0x00,
+0x00,
+0x7a,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x8a,
+0x40,
+0x40,
+0x88,
+0x10,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x19,
+0x96,
+0x83,
+0x40,
+0x00,
+0xa2,
+0x06,
+0xc6,
+0x80,
+0x01,
+0x40,
+0x86,
+0x80,
+0x00,
+0x18,
+0x00,
+0xa5,
+0x48,
+0xd2,
+0x20,
+0x22,
+0x06,
+0xc6,
+0x80,
+0x01,
+0x44,
+0x85,
+0xc0,
+0xaa,
+0x2c,
+0x36,
+0x05,
+0xc0,
+0xb7,
+0x04,
+0x00,
+0x92,
+0x49,
+0x2d,
+0x80,
+0x24,
+0x90,
+0x00,
+0x00,
+0x84,
+0x00,
+0x92,
+0x49,
+0x2c,
+0x84,
+0x04,
+0x89,
+0x8e,
+0x48,
+0x52,
+0x4d,
+0x23,
+0xc4,
+0x3f,
+0x40,
+0x00,
+0x01,
+0x82,
+0x39,
+0x6c,
+0x68,
+0x00,
+0x14,
+0x08,
+0x68,
+0x3f,
+0xfe,
+0x7f,
+0xca,
+0x54,
+0x4d,
+0x22,
+0x02,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x14,
+0x48,
+0x5c,
+0x0a,
+0xa2,
+0xc3,
+0x60,
+0x84,
+0x00,
+0xa2,
+0x41,
+0x36,
+0x80,
+0x24,
+0xa0,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa2,
+0x41,
+0x34,
+0x40,
+0x00,
+0x00,
+0x40,
+0x48,
+0x66,
+0x00,
+0x01,
+0x7c,
+0xe0,
+0x66,
+0x00,
+0x01,
+0x7d,
+0xc0,
+0x5c,
+0x0b,
+0x71,
+0x8e,
+0x48,
+0x52,
+0x0d,
+0x23,
+0xc2,
+0x7f,
+0x40,
+0x00,
+0x01,
+0x82,
+0x39,
+0x6c,
+0x68,
+0x00,
+0x36,
+0x0a,
+0x25,
+0x93,
+0x0b,
+0xc0,
+0x38,
+0x5c,
+0x00,
+0x62,
+0x40,
+0x40,
+0x98,
+0xe8,
+0x88,
+0x81,
+0x48,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc1,
+0x37,
+0x32,
+0x0b,
+0x0b,
+0xc0,
+0x78,
+0xa4,
+0x04,
+0x06,
+0x80,
+0x00,
+0x00,
+0x7a,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xc0,
+0xbf,
+0x88,
+0x13,
+0x98,
+0x81,
+0x79,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x20,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0x70,
+0x68,
+0x20,
+0x02,
+0x21,
+0x2c,
+0x5c,
+0x07,
+0xe2,
+0x40,
+0x40,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0x22,
+0xc8,
+0x00,
+0x6c,
+0x68,
+0x20,
+0x02,
+0x23,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0x42,
+0xc8,
+0x00,
+0x6c,
+0x68,
+0x20,
+0x02,
+0x25,
+0x2c,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0x62,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x22,
+0x72,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x68,
+0x20,
+0x02,
+0x28,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0x92,
+0x18,
+0x00,
+0x61,
+0x68,
+0x20,
+0x02,
+0x2b,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0xc2,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0xa2,
+0xc6,
+0xe0,
+0x00,
+0x00,
+0xea,
+0xe5,
+0x52,
+0x18,
+0x04,
+0x06,
+0xc5,
+0x18,
+0x40,
+0x20,
+0x5e,
+0x05,
+0x80,
+0x98,
+0x18,
+0x42,
+0x89,
+0xc0,
+0x00,
+0x42,
+0x1c,
+0x60,
+0x40,
+0x20,
+0x98,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x32,
+0x06,
+0xe0,
+0x00,
+0x00,
+0xda,
+0xc3,
+0x20,
+0xa0,
+0xbc,
+0x22,
+0x03,
+0x21,
+0xa0,
+0xbc,
+0x2f,
+0x16,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x13,
+0x81,
+0xc4,
+0x84,
+0x80,
+0xa2,
+0x59,
+0x30,
+0xbc,
+0x05,
+0x9a,
+0x0e,
+0xc2,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x20,
+0x02,
+0x0e,
+0x23,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x9d,
+0x80,
+0x38,
+0x58,
+0x00,
+0x98,
+0xb0,
+0x23,
+0x00,
+0x80,
+0xbc,
+0x05,
+0x98,
+0x50,
+0x50,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0x09,
+0xa2,
+0x0e,
+0x71,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x48,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x09,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x00,
+0x42,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0xa8,
+0x09,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x09,
+0x00,
+0x00,
+0x00,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xac,
+0xb0,
+0x78,
+0x62,
+0x89,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x60,
+0x38,
+0x10,
+0x63,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xfd,
+0xc7,
+0x64,
+0x00,
+0x01,
+0xfc,
+0xc7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0xab,
+0xff,
+0x0a,
+0x06,
+0x81,
+0x88,
+0x06,
+0x18,
+0x80,
+0xf6,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x02,
+0x08,
+0x80,
+0xb6,
+0xa8,
+0x01,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x04,
+0x60,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x68,
+0x20,
+0x01,
+0x8e,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x00,
+0x42,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x6c,
+0x40,
+0x04,
+0x3c,
+0x00,
+0x23,
+0xe0,
+0x43,
+0x2a,
+0x20,
+0xbc,
+0x05,
+0x82,
+0x34,
+0x00,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x6c,
+0x00,
+0x00,
+0xde,
+0x50,
+0x6c,
+0x00,
+0x00,
+0xdc,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x08,
+0x40,
+0x02,
+0x24,
+0x81,
+0x08,
+0x40,
+0x50,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x3e,
+0x0a,
+0x32,
+0x83,
+0x06,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x05,
+0x1e,
+0x1a,
+0x3c,
+0x29,
+0x82,
+0x88,
+0x36,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x02,
+0x5b,
+0x44,
+0x0b,
+0x00,
+0xf8,
+0x37,
+0x04,
+0x52,
+0xe1,
+0x47,
+0x68,
+0x00,
+0x13,
+0x8d,
+0x09,
+0x62,
+0x00,
+0x00,
+0x00,
+0x27,
+0x50,
+0x4e,
+0x49,
+0x8e,
+0xb5,
+0x5b,
+0x4a,
+0x02,
+0xc0,
+0x20,
+0x2f,
+0x80,
+0x92,
+0x99,
+0x50,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x2b,
+0x98,
+0x34,
+0x93,
+0x61,
+0x45,
+0x46,
+0x08,
+0x8b,
+0x01,
+0x00,
+0x08,
+0x12,
+0x05,
+0x1a,
+0x1a,
+0x18,
+0x38,
+0x26,
+0x80,
+0x00,
+0x0c,
+0xd2,
+0x06,
+0xc0,
+0x00,
+0x19,
+0x60,
+0xa5,
+0x20,
+0x1b,
+0x00,
+0x04,
+0x28,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x19,
+0x64,
+0xa6,
+0x80,
+0x00,
+0x06,
+0xe2,
+0xc4,
+0x20,
+0x8f,
+0x80,
+0x04,
+0x88,
+0x40,
+0xec,
+0x68,
+0x00,
+0x00,
+0xcd,
+0x20,
+0x39,
+0x02,
+0x06,
+0xc4,
+0x00,
+0x44,
+0x00,
+0x85,
+0x94,
+0x10,
+0x00,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x0d,
+0xc7,
+0xa4,
+0x20,
+0x1c,
+0x80,
+0x07,
+0xa8,
+0x40,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xcb,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x04,
+0x40,
+0x0a,
+0x32,
+0x83,
+0x06,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x05,
+0x1e,
+0x1a,
+0x3c,
+0x29,
+0x82,
+0x88,
+0x36,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x02,
+0x5b,
+0x44,
+0x0b,
+0x00,
+0xf8,
+0x37,
+0x04,
+0x52,
+0xe1,
+0x47,
+0x68,
+0x00,
+0x13,
+0x8d,
+0x09,
+0x62,
+0x00,
+0x00,
+0x00,
+0x27,
+0x50,
+0x4e,
+0x49,
+0x8e,
+0xb5,
+0x5b,
+0x4a,
+0x02,
+0xc0,
+0x20,
+0x2f,
+0x80,
+0x92,
+0x99,
+0x50,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x2b,
+0x98,
+0x34,
+0x93,
+0x61,
+0x45,
+0x46,
+0x08,
+0x8b,
+0x01,
+0x00,
+0x08,
+0x12,
+0x05,
+0x1a,
+0x1a,
+0x18,
+0x38,
+0x26,
+0x80,
+0x00,
+0x0d,
+0x22,
+0x06,
+0xc0,
+0x00,
+0x19,
+0x60,
+0xa5,
+0x20,
+0x1b,
+0x00,
+0x04,
+0x28,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x19,
+0x64,
+0xa6,
+0x80,
+0x00,
+0x06,
+0xf2,
+0xc4,
+0x20,
+0x8f,
+0x80,
+0x04,
+0x88,
+0x40,
+0xec,
+0x68,
+0x00,
+0x00,
+0xd2,
+0x20,
+0x39,
+0x02,
+0x06,
+0xc4,
+0x00,
+0x43,
+0xe0,
+0x85,
+0x94,
+0x10,
+0x00,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x0d,
+0xe7,
+0xa4,
+0x20,
+0x1c,
+0x80,
+0x07,
+0xa8,
+0x40,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xcb,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x09,
+0x5d,
+0x0a,
+0x23,
+0x01,
+0xc6,
+0x52,
+0x0d,
+0x72,
+0xbf,
+0xe0,
+0x59,
+0x01,
+0x00,
+0x80,
+0xfa,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x4a,
+0x88,
+0x17,
+0x64,
+0x20,
+0x74,
+0x18,
+0xe8,
+0x95,
+0x50,
+0x16,
+0x08,
+0x04,
+0x89,
+0x8e,
+0x88,
+0x88,
+0x00,
+0x93,
+0x20,
+0x68,
+0xbc,
+0x07,
+0x8b,
+0x00,
+0x0d,
+0x88,
+0x00,
+0x95,
+0x90,
+0x54,
+0x30,
+0x00,
+0xdb,
+0xc0,
+0x29,
+0x98,
+0x24,
+0x89,
+0x8e,
+0x89,
+0x88,
+0x0c,
+0x93,
+0x20,
+0x68,
+0xbc,
+0x0c,
+0x98,
+0x81,
+0xc8,
+0x66,
+0x00,
+0x00,
+0xe8,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x46,
+0x20,
+0x5c,
+0x81,
+0x0a,
+0xc2,
+0x20,
+0x80,
+0x28,
+0x98,
+0x00,
+0x08,
+0x57,
+0x09,
+0x40,
+0x81,
+0x88,
+0x84,
+0x04,
+0x03,
+0x20,
+0x60,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xa1,
+0x66,
+0x00,
+0x00,
+0xe9,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x5a,
+0x20,
+0x5c,
+0x81,
+0x0a,
+0xc2,
+0x20,
+0x80,
+0x28,
+0x98,
+0x00,
+0x08,
+0x2e,
+0x12,
+0x88,
+0x40,
+0x40,
+0x98,
+0xe8,
+0x86,
+0x80,
+0x00,
+0x05,
+0x72,
+0x06,
+0x80,
+0x00,
+0x06,
+0xb2,
+0x16,
+0x60,
+0x00,
+0x04,
+0xa2,
+0x88,
+0x80,
+0x89,
+0xb0,
+0x00,
+0xc6,
+0x80,
+0x00,
+0x05,
+0x72,
+0x06,
+0x80,
+0x00,
+0x06,
+0xb2,
+0x16,
+0x60,
+0x00,
+0x04,
+0xa2,
+0x88,
+0x81,
+0x89,
+0x88,
+0x00,
+0x93,
+0x20,
+0x28,
+0xbc,
+0x11,
+0x16,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x03,
+0x81,
+0xc5,
+0x84,
+0x00,
+0x82,
+0x49,
+0x64,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x0a,
+0x52,
+0x4b,
+0xa8,
+0x40,
+0x48,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x49,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x42,
+0x03,
+0x7a,
+0x06,
+0xc0,
+0x40,
+0x00,
+0x00,
+0x40,
+0x6c,
+0x68,
+0x00,
+0x08,
+0x40,
+0x20,
+0x6c,
+0x00,
+0x00,
+0x0a,
+0x60,
+0x00,
+0x00,
+0x08,
+0x81,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x20,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0x56,
+0x20,
+0x68,
+0x20,
+0x00,
+0x82,
+0x22,
+0x84,
+0x00,
+0x88,
+0x50,
+0x0a,
+0x58,
+0x0d,
+0x02,
+0xbf,
+0xf0,
+0x68,
+0x20,
+0x00,
+0x81,
+0x21,
+0xbc,
+0x03,
+0x88,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x68,
+0x00,
+0x00,
+0x6a,
+0x20,
+0x6c,
+0x40,
+0x01,
+0x04,
+0x08,
+0x84,
+0x00,
+0xa3,
+0x01,
+0x30,
+0x68,
+0x20,
+0x00,
+0x82,
+0x22,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x48,
+0x68,
+0x20,
+0x00,
+0x81,
+0x21,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x6c,
+0x00,
+0x00,
+0xac,
+0x08,
+0x6c,
+0x40,
+0x01,
+0x04,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x41,
+0x6c,
+0x00,
+0x00,
+0xd4,
+0x08,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x68,
+0x00,
+0x08,
+0x40,
+0x20,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x1f,
+0x6c,
+0x00,
+0x00,
+0x0a,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x66,
+0xc4,
+0x00,
+0x44,
+0x20,
+0x05,
+0x24,
+0x82,
+0x04,
+0x04,
+0xaa,
+0x06,
+0xc0,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x48,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xf2,
+0x59,
+0x60,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x2d,
+0x52,
+0x4b,
+0xc3,
+0xc0,
+0x6f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x52,
+0x0b,
+0xc3,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x52,
+0x0d,
+0x22,
+0xbf,
+0xf0,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x20,
+0x25,
+0x96,
+0x06,
+0xc4,
+0x00,
+0x44,
+0x64,
+0x8b,
+0xc1,
+0x28,
+0x5c,
+0x08,
+0xa9,
+0x40,
+0x2f,
+0x52,
+0x4b,
+0xc3,
+0x01,
+0x2c,
+0x5c,
+0x81,
+0x01,
+0x40,
+0xad,
+0x25,
+0x92,
+0x8b,
+0xc3,
+0xb9,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x99,
+0x21,
+0x24,
+0x92,
+0x88,
+0x0a,
+0x7a,
+0x84,
+0x87,
+0xaa,
+0x09,
+0x89,
+0x94,
+0x86,
+0x0b,
+0xc3,
+0x17,
+0x25,
+0x97,
+0x8b,
+0xc1,
+0x60,
+0x38,
+0x10,
+0x82,
+0x58,
+0x20,
+0xbc,
+0x04,
+0x06,
+0xc0,
+0x00,
+0x0e,
+0xa7,
+0xa6,
+0xc0,
+0x00,
+0x0f,
+0xe7,
+0xa2,
+0x59,
+0x60,
+0xbc,
+0x08,
+0x16,
+0xc4,
+0x00,
+0x25,
+0x60,
+0x82,
+0x31,
+0x24,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x48,
+0x23,
+0x16,
+0x46,
+0xc0,
+0x00,
+0x14,
+0xc4,
+0x82,
+0x41,
+0x78,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x7a,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x43,
+0xe0,
+0x86,
+0xc4,
+0x00,
+0x44,
+0x00,
+0x92,
+0x91,
+0x64,
+0x32,
+0x82,
+0x0b,
+0xc0,
+0xc0,
+0x40,
+0x00,
+0x00,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x60,
+0x40,
+0x5c,
+0x0e,
+0x30,
+0x80,
+0x36,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x20,
+0x6c,
+0x40,
+0x04,
+0x3e,
+0x7a,
+0x6c,
+0x40,
+0x04,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x6e,
+0x21,
+0x00,
+0x00,
+0x08,
+0x48,
+0xfa,
+0x84,
+0x87,
+0xa3,
+0x81,
+0x1d,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x0b,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x68,
+0x5c,
+0x09,
+0xa1,
+0x40,
+0xad,
+0x52,
+0x49,
+0x43,
+0xc0,
+0xcf,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0x6c,
+0x40,
+0x02,
+0xc2,
+0x00,
+0x52,
+0x09,
+0x53,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x01,
+0x50,
+0x50,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x50,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe2,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9b,
+0xc8,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xca,
+0x80,
+0x10,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x00,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xa2,
+0x58,
+0x20,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x20,
+0x52,
+0x00,
+0x83,
+0xc0,
+0x5f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x24,
+0x81,
+0x06,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x03,
+0xa1,
+0x00,
+0x22,
+0x84,
+0x56,
+0x82,
+0x00,
+0x0d,
+0x72,
+0x02,
+0xa8,
+0xe8,
+0x51,
+0x84,
+0x00,
+0x40,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x21,
+0x52,
+0x4d,
+0x21,
+0x84,
+0x28,
+0xa0,
+0x02,
+0x08,
+0x48,
+0x00,
+0x52,
+0x4c,
+0x01,
+0xc0,
+0x00,
+0x84,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x44,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x19,
+0x84,
+0xa6,
+0xc4,
+0x00,
+0x19,
+0xa4,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0xa4,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0x44,
+0xa6,
+0x80,
+0x00,
+0x6e,
+0x6a,
+0xca,
+0x0e,
+0xc0,
+0x46,
+0x0a,
+0x40,
+0x48,
+0x50,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x08,
+0x5c,
+0x0e,
+0x32,
+0xc0,
+0x20,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x20,
+0x60,
+0x00,
+0x00,
+0x00,
+0x2c,
+0x52,
+0x0d,
+0x22,
+0xbf,
+0xf0,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x48,
+0x80,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x0e,
+0x82,
+0x06,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0xa5,
+0xc8,
+0x30,
+0x08,
+0x07,
+0x66,
+0x82,
+0x00,
+0x0e,
+0x22,
+0x18,
+0x00,
+0x4a,
+0x80,
+0x84,
+0xa8,
+0x40,
+0x4a,
+0x84,
+0x84,
+0xa6,
+0x80,
+0x00,
+0x0a,
+0xc2,
+0x06,
+0x60,
+0x00,
+0x04,
+0xe0,
+0x83,
+0x82,
+0xc4,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0x08,
+0x5c,
+0x08,
+0x23,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0xac,
+0x20,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x08,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x40,
+0x6c,
+0x40,
+0x01,
+0xf8,
+0x0a,
+0x5c,
+0x83,
+0x08,
+0x40,
+0xc8,
+0x80,
+0x04,
+0xa6,
+0x80,
+0x00,
+0x09,
+0x62,
+0x1a,
+0x01,
+0x20,
+0x80,
+0x04,
+0xa8,
+0x02,
+0xc8,
+0xa0,
+0x82,
+0x28,
+0x40,
+0x61,
+0x40,
+0x00,
+0x00,
+0x45,
+0x62,
+0x68,
+0x00,
+0x00,
+0xc2,
+0x20,
+0x68,
+0x00,
+0x00,
+0xc6,
+0x21,
+0x66,
+0x00,
+0x00,
+0x45,
+0x00,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0x28,
+0x5c,
+0x00,
+0x63,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x20,
+0x5c,
+0x82,
+0x00,
+0x80,
+0x36,
+0x68,
+0x00,
+0x08,
+0x97,
+0x2c,
+0x68,
+0x00,
+0x01,
+0x00,
+0x08,
+0xa0,
+0x0e,
+0x06,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0xc4,
+0x60,
+0xa4,
+0x00,
+0x24,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x68,
+0x00,
+0x01,
+0x32,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x84,
+0x10,
+0xa2,
+0x91,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc3,
+0x31,
+0x68,
+0x00,
+0x00,
+0xb2,
+0x20,
+0x68,
+0x00,
+0x00,
+0x9e,
+0x21,
+0x8c,
+0x03,
+0x25,
+0x15,
+0x49,
+0x20,
+0x14,
+0x2a,
+0x15,
+0x80,
+0x6c,
+0x40,
+0x02,
+0x12,
+0x08,
+0x84,
+0x00,
+0xa5,
+0x84,
+0x98,
+0x0d,
+0x03,
+0x05,
+0x15,
+0x40,
+0x04,
+0x84,
+0xeb,
+0xc0,
+0x8d,
+0x84,
+0x8c,
+0xc3,
+0x81,
+0xce,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x09,
+0x24,
+0x1a,
+0xe6,
+0xc4,
+0x00,
+0x44,
+0xa4,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x45,
+0x0a,
+0x30,
+0x93,
+0x0b,
+0xc0,
+0x65,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x08,
+0x38,
+0x1d,
+0x62,
+0x41,
+0xa4,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x48,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x66,
+0xc4,
+0x00,
+0x44,
+0xa0,
+0x95,
+0x24,
+0x96,
+0x04,
+0x04,
+0xa6,
+0xc4,
+0x00,
+0x44,
+0xa4,
+0x86,
+0x80,
+0x00,
+0x6e,
+0x6a,
+0xc4,
+0x60,
+0xa4,
+0x20,
+0x6c,
+0x08,
+0x40,
+0x6c,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x6c,
+0x40,
+0x04,
+0x52,
+0x08,
+0x68,
+0x00,
+0x47,
+0xff,
+0xca,
+0x68,
+0x38,
+0x1c,
+0x04,
+0x20,
+0x54,
+0x4d,
+0x23,
+0x00,
+0x08,
+0x5c,
+0x00,
+0x20,
+0x40,
+0x48,
+0x5c,
+0x09,
+0xf2,
+0x00,
+0x20,
+0x84,
+0x07,
+0xa8,
+0x40,
+0xd0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x38,
+0x10,
+0x00,
+0x25,
+0x98,
+0x0b,
+0xc0,
+0x50,
+0x6c,
+0x70,
+0x38,
+0x00,
+0x08,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x45,
+0x24,
+0x83,
+0x22,
+0xa0,
+0xbf,
+0xf5,
+0xa2,
+0xa0,
+0x64,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x07,
+0x60,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xac,
+0x5c,
+0x09,
+0xf1,
+0x40,
+0x2d,
+0x52,
+0x0d,
+0x03,
+0x00,
+0xec,
+0x30,
+0x12,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x8e,
+0x0b,
+0xc0,
+0x89,
+0xa0,
+0x44,
+0x86,
+0xc4,
+0x00,
+0x45,
+0x80,
+0x83,
+0x81,
+0x05,
+0x52,
+0x0b,
+0x23,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x48,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x21,
+0x39,
+0x7e,
+0x08,
+0x48,
+0x08,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x09,
+0x68,
+0x00,
+0x08,
+0xb9,
+0xac,
+0x6c,
+0x00,
+0x03,
+0x8a,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xa8,
+0x49,
+0xa0,
+0xc4,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x06,
+0xc8,
+0x48,
+0x4a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x00,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x20,
+0x5c,
+0x08,
+0x0b,
+0x00,
+0x0a,
+0x68,
+0x38,
+0x08,
+0x00,
+0x21,
+0x52,
+0xc2,
+0x00,
+0x40,
+0x52,
+0x5c,
+0xbf,
+0x03,
+0xc1,
+0x28,
+0x51,
+0xe0,
+0x21,
+0xc0,
+0x00,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xc0,
+0x54,
+0x41,
+0x03,
+0x04,
+0x01,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x51,
+0x51,
+0xd0,
+0x23,
+0x07,
+0xfb,
+0x54,
+0x47,
+0x0a,
+0x08,
+0x41,
+0x54,
+0x46,
+0x00,
+0x48,
+0x7a,
+0x42,
+0x07,
+0xf8,
+0x48,
+0xd1,
+0x40,
+0x00,
+0x00,
+0x48,
+0xd0,
+0x68,
+0x00,
+0x3f,
+0xc0,
+0x00,
+0x54,
+0x41,
+0x03,
+0x01,
+0x79,
+0x52,
+0x02,
+0x0b,
+0x07,
+0xf8,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x7a,
+0x51,
+0xd0,
+0x62,
+0x08,
+0x41,
+0x54,
+0x41,
+0x00,
+0x48,
+0x7a,
+0x84,
+0x8d,
+0x03,
+0x80,
+0x10,
+0x68,
+0x00,
+0x08,
+0xc7,
+0x2c,
+0x84,
+0x05,
+0x06,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0xc4,
+0x60,
+0xa4,
+0x20,
+0x50,
+0x08,
+0x40,
+0x52,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x08,
+0x5c,
+0x00,
+0x43,
+0x01,
+0x06,
+0x6c,
+0x70,
+0x10,
+0x0e,
+0x50,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x61,
+0x6c,
+0x70,
+0x10,
+0x02,
+0x08,
+0x51,
+0xb1,
+0x23,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x48,
+0x68,
+0x00,
+0x08,
+0xcc,
+0x20,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x8a,
+0x08,
+0x5c,
+0xbf,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x21,
+0x5c,
+0x0e,
+0x21,
+0xc0,
+0x00,
+0x5c,
+0x01,
+0xe8,
+0x48,
+0x0b,
+0x6c,
+0x00,
+0x03,
+0xa8,
+0x0a,
+0x52,
+0x49,
+0xe0,
+0x40,
+0x4a,
+0x5c,
+0x09,
+0xf2,
+0x04,
+0x40,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xaf,
+0x52,
+0x4d,
+0xc0,
+0x48,
+0x48,
+0xa0,
+0xec,
+0x18,
+0x40,
+0x49,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xa0,
+0x58,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x86,
+0xc8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x08,
+0x38,
+0x1c,
+0x66,
+0xc6,
+0x80,
+0x03,
+0xc0,
+0x02,
+0x41,
+0xa4,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x48,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0xa0,
+0x68,
+0x34,
+0x00,
+0x00,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x08,
+0xa0,
+0x3c,
+0x08,
+0x40,
+0x00,
+0x6c,
+0x00,
+0x03,
+0xb6,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xb8,
+0x50,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x45,
+0x40,
+0x82,
+0xa8,
+0xe0,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0xe1,
+0x68,
+0x34,
+0x00,
+0x00,
+0x20,
+0x6c,
+0x00,
+0x03,
+0xb6,
+0x00,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x50,
+0xa0,
+0x3c,
+0x06,
+0xc0,
+0x00,
+0x3b,
+0x80,
+0x28,
+0x40,
+0x52,
+0x42,
+0x0f,
+0x7b,
+0x0f,
+0xf0,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x50,
+0x6c,
+0x68,
+0x00,
+0x3c,
+0x7a,
+0x68,
+0x00,
+0x04,
+0xd9,
+0x00,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x02,
+0x54,
+0x40,
+0x83,
+0x01,
+0x0a,
+0x25,
+0x8a,
+0x06,
+0xc6,
+0x80,
+0x00,
+0x05,
+0x0b,
+0xc0,
+0x48,
+0xb0,
+0xff,
+0x2b,
+0xc0,
+0xef,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x52,
+0x5c,
+0x08,
+0x03,
+0x1f,
+0xf2,
+0x25,
+0x82,
+0x0b,
+0xc0,
+0x89,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x52,
+0x38,
+0x11,
+0x06,
+0xc6,
+0x80,
+0x03,
+0xa0,
+0x22,
+0x40,
+0x10,
+0x6c,
+0x68,
+0x00,
+0x3a,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x00,
+0x24,
+0x98,
+0x68,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xf2,
+0x59,
+0x60,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x3d,
+0x52,
+0x4b,
+0xc3,
+0xc0,
+0x6f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x52,
+0x0b,
+0xc3,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0xca,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc5,
+0xc0,
+0x5c,
+0x08,
+0x12,
+0xc0,
+0x21,
+0x25,
+0x88,
+0x06,
+0x80,
+0x00,
+0x1d,
+0xd2,
+0x0b,
+0xc0,
+0x88,
+0x5c,
+0x85,
+0x00,
+0x00,
+0xa1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x42,
+0x07,
+0x78,
+0x48,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xcc,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe1,
+0x22,
+0x00,
+0x00,
+0x08,
+0x10,
+0x23,
+0x00,
+0x00,
+0x08,
+0x58,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0x74,
+0x00,
+0x00,
+0x08,
+0x48,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0xf4,
+0x5c,
+0x08,
+0x52,
+0x00,
+0x01,
+0x25,
+0x88,
+0x0b,
+0xc0,
+0x68,
+0x80,
+0x0a,
+0x20,
+0x00,
+0x00,
+0x42,
+0x07,
+0x78,
+0x50,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xce,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe2,
+0x23,
+0x00,
+0x00,
+0x08,
+0x18,
+0x24,
+0x00,
+0x00,
+0x08,
+0x60,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x58,
+0x74,
+0x00,
+0x00,
+0x08,
+0x50,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x58,
+0xf4,
+0x38,
+0x11,
+0x22,
+0x58,
+0x80,
+0xbc,
+0x07,
+0x88,
+0x48,
+0xa1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x42,
+0x07,
+0x78,
+0x48,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xd0,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe3,
+0x22,
+0x00,
+0x00,
+0x08,
+0x10,
+0x23,
+0x00,
+0x00,
+0x08,
+0x58,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0x74,
+0x00,
+0x00,
+0x08,
+0x48,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0xf4,
+0x38,
+0x11,
+0xa2,
+0x58,
+0x80,
+0xbc,
+0x07,
+0x88,
+0x40,
+0xa0,
+0x00,
+0x00,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x3d,
+0x25,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0xe4,
+0x21,
+0x00,
+0x00,
+0x08,
+0x08,
+0x22,
+0x00,
+0x00,
+0x08,
+0x50,
+0x00,
+0x22,
+0xc0,
+0x49,
+0x48,
+0x74,
+0x00,
+0x00,
+0x08,
+0x40,
+0x00,
+0x22,
+0xc0,
+0x49,
+0x48,
+0xf4,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x37,
+0x08,
+0x63,
+0x70,
+0x44,
+0x2e,
+0x13,
+0x42,
+0xa0,
+0x64,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x55,
+0x38,
+0x20,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x0e,
+0x06,
+0x20,
+0x00,
+0x00,
+0x01,
+0x45,
+0x00,
+0x88,
+0x18,
+0xeb,
+0x50,
+0x00,
+0x00,
+0x2f,
+0x80,
+0x9b,
+0xc0,
+0x13,
+0x28,
+0x00,
+0x92,
+0x09,
+0x08,
+0x57,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x20,
+0x10,
+0x95,
+0x0c,
+0x84,
+0x19,
+0x20,
+0x3b,
+0xa1,
+0x48,
+0x5b,
+0xc2,
+0x01,
+0x8e,
+0x83,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x48,
+0x55,
+0x00,
+0x59,
+0x8e,
+0x80,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x0c,
+0xe2,
+0xbf,
+0xf0,
+0x50,
+0xc8,
+0x41,
+0x00,
+0x59,
+0x6a,
+0x00,
+0x03,
+0xff,
+0xc1,
+0x54,
+0x42,
+0x0b,
+0x00,
+0x0d,
+0x6a,
+0x00,
+0x03,
+0xff,
+0x80,
+0x30,
+0x80,
+0x8b,
+0xc0,
+0x79,
+0x2a,
+0x02,
+0xe6,
+0xe0,
+0x00,
+0x3d,
+0x81,
+0x19,
+0x00,
+0x13,
+0x28,
+0x85,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x01,
+0x19,
+0x8e,
+0x89,
+0x51,
+0x82,
+0x89,
+0x82,
+0x8b,
+0x6e,
+0x00,
+0x03,
+0xd4,
+0x13,
+0x30,
+0x8c,
+0x8b,
+0xc0,
+0x3c,
+0x6a,
+0x00,
+0x03,
+0xff,
+0xc1,
+0x98,
+0xe8,
+0xb2,
+0x19,
+0x13,
+0x28,
+0x85,
+0x93,
+0x08,
+0x08,
+0xbc,
+0x05,
+0x16,
+0xe0,
+0x00,
+0x3d,
+0x81,
+0x02,
+0x88,
+0x10,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x11,
+0x98,
+0xe8,
+0xa3,
+0x20,
+0x30,
+0x52,
+0x08,
+0x81,
+0x00,
+0x12,
+0xbc,
+0x08,
+0x92,
+0x41,
+0x12,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x41,
+0x32,
+0x03,
+0x8b,
+0xc0,
+0x31,
+0x2a,
+0x01,
+0x0b,
+0xc0,
+0x17,
+0x2a,
+0x01,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x5d,
+0x4a,
+0x33,
+0x01,
+0x03,
+0x68,
+0x00,
+0x01,
+0xff,
+0x4b,
+0x58,
+0x0f,
+0x83,
+0x5f,
+0xf8,
+0x50,
+0x47,
+0x19,
+0x84,
+0xc8,
+0x54,
+0x40,
+0x53,
+0x01,
+0xfe,
+0xbc,
+0x2f,
+0xa5,
+0x08,
+0xcd,
+0x18,
+0x38,
+0x33,
+0x01,
+0xe8,
+0xbc,
+0x0b,
+0x46,
+0x80,
+0x00,
+0x1f,
+0xf4,
+0xb3,
+0x01,
+0xe8,
+0xbc,
+0x0c,
+0x15,
+0x50,
+0x0f,
+0xb5,
+0x00,
+0x32,
+0x80,
+0xcb,
+0x37,
+0x8c,
+0x33,
+0x20,
+0x18,
+0xbc,
+0x06,
+0xb2,
+0xa0,
+0x3b,
+0x6e,
+0x00,
+0x03,
+0xdc,
+0x11,
+0xba,
+0x14,
+0x82,
+0x80,
+0x51,
+0x37,
+0x84,
+0x03,
+0x20,
+0x28,
+0xbc,
+0x19,
+0x33,
+0x61,
+0x45,
+0x32,
+0x02,
+0x8b,
+0xc1,
+0x20,
+0x38,
+0x20,
+0x73,
+0x01,
+0xe8,
+0xbc,
+0x05,
+0x23,
+0x28,
+0x08,
+0xbc,
+0x0d,
+0x93,
+0x88,
+0x09,
+0x38,
+0x00,
+0x1b,
+0xc0,
+0xa7,
+0x36,
+0x14,
+0x72,
+0x19,
+0x4b,
+0x28,
+0x9b,
+0xd2,
+0x11,
+0x49,
+0x32,
+0x80,
+0x8b,
+0xc0,
+0x29,
+0x38,
+0x80,
+0x93,
+0x80,
+0x01,
+0x54,
+0x82,
+0xcb,
+0x80,
+0x00,
+0x54,
+0x40,
+0x41,
+0x8e,
+0x89,
+0x40,
+0x00,
+0x01,
+0x83,
+0x03,
+0x5c,
+0x0a,
+0x73,
+0x50,
+0x00,
+0x25,
+0x19,
+0xe2,
+0x80,
+0x09,
+0x32,
+0x03,
+0x05,
+0xbc,
+0x20,
+0xbc,
+0x03,
+0x85,
+0x1d,
+0x44,
+0xb0,
+0x00,
+0xf9,
+0x8e,
+0x8b,
+0x55,
+0x43,
+0xf3,
+0x3f,
+0xff,
+0x29,
+0x9f,
+0x62,
+0x09,
+0x30,
+0x28,
+0x80,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x01,
+0x19,
+0x8e,
+0x89,
+0x50,
+0x49,
+0x43,
+0x01,
+0xa5,
+0x21,
+0x14,
+0x02,
+0x80,
+0x10,
+0x5b,
+0xc0,
+0x03,
+0xa1,
+0x48,
+0x28,
+0x04,
+0x13,
+0x78,
+0x40,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x2a,
+0x38,
+0x00,
+0xd9,
+0x8e,
+0x89,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x20,
+0x5b,
+0x08,
+0x23,
+0x80,
+0x00,
+0x5b,
+0x88,
+0x33,
+0x00,
+0xb7,
+0x54,
+0x0f,
+0x83,
+0x01,
+0x06,
+0x68,
+0x00,
+0x01,
+0x0c,
+0x8b,
+0x57,
+0x01,
+0xcb,
+0x01,
+0xff,
+0x50,
+0x4d,
+0x53,
+0x01,
+0xa5,
+0x20,
+0x98,
+0x92,
+0x11,
+0xd2,
+0x21,
+0x14,
+0x92,
+0x19,
+0xa3,
+0x28,
+0x05,
+0x22,
+0x10,
+0x18,
+0x5b,
+0xc4,
+0x13,
+0xa1,
+0x48,
+0x28,
+0x01,
+0x03,
+0x78,
+0x00,
+0xba,
+0x14,
+0x83,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x0d,
+0x23,
+0x01,
+0xfe,
+0x21,
+0x90,
+0x26,
+0xa0,
+0x00,
+0x1f,
+0xfc,
+0x12,
+0x88,
+0x51,
+0x50,
+0xcc,
+0x11,
+0x83,
+0x4a,
+0x68,
+0x00,
+0x01,
+0x07,
+0x89,
+0x30,
+0x17,
+0x06,
+0xe0,
+0x00,
+0x3e,
+0x41,
+0x1b,
+0xc0,
+0x8c,
+0x54,
+0x42,
+0x01,
+0x83,
+0x89,
+0xb1,
+0xff,
+0xf3,
+0x01,
+0xf0,
+0xbc,
+0x0b,
+0x3b,
+0xa1,
+0x48,
+0x98,
+0xe8,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0xff,
+0xc8,
+0x30,
+0x13,
+0x0b,
+0xc1,
+0x41,
+0x32,
+0x80,
+0x0b,
+0xc1,
+0x20,
+0x98,
+0xe8,
+0x9b,
+0xc1,
+0x07,
+0x68,
+0x00,
+0x01,
+0x0c,
+0xcb,
+0x2e,
+0x1b,
+0xe2,
+0x41,
+0x00,
+0x32,
+0x02,
+0x85,
+0x0c,
+0xc0,
+0x3c,
+0x02,
+0x89,
+0x83,
+0x08,
+0x36,
+0x10,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x02,
+0xa3,
+0x80,
+0x0e,
+0x98,
+0xe8,
+0xa2,
+0x99,
+0x76,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x60,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x49,
+0x6c,
+0x00,
+0x03,
+0xe0,
+0x08,
+0x6c,
+0x00,
+0x03,
+0xe2,
+0x08,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x0f,
+0xe3,
+0x01,
+0xa5,
+0x50,
+0xc8,
+0x42,
+0xbf,
+0xd0,
+0x50,
+0xc8,
+0x81,
+0x83,
+0x0a,
+0x50,
+0xca,
+0x59,
+0x83,
+0x0b,
+0x6a,
+0x00,
+0x01,
+0xff,
+0xc0,
+0x54,
+0x40,
+0xd9,
+0x00,
+0x5a,
+0x54,
+0xcf,
+0xb3,
+0x01,
+0x07,
+0x50,
+0xca,
+0x99,
+0x83,
+0xc9,
+0x54,
+0x40,
+0xc0,
+0x81,
+0x4a,
+0x50,
+0x4f,
+0x81,
+0x83,
+0x0a,
+0x21,
+0x10,
+0x06,
+0xe0,
+0x00,
+0x3f,
+0x01,
+0x35,
+0x40,
+0x61,
+0x90,
+0x25,
+0x86,
+0x80,
+0x00,
+0x1f,
+0xfc,
+0x86,
+0xe0,
+0x00,
+0x3e,
+0x81,
+0x03,
+0x01,
+0x28,
+0x28,
+0x81,
+0x25,
+0xbc,
+0x61,
+0xbc,
+0x76,
+0x82,
+0x88,
+0x08,
+0x30,
+0x13,
+0x0b,
+0xc1,
+0x41,
+0x32,
+0x81,
+0x0b,
+0xc0,
+0x91,
+0x20,
+0x9e,
+0x92,
+0x90,
+0x09,
+0x32,
+0x80,
+0x8b,
+0xc0,
+0xa1,
+0xba,
+0x14,
+0x86,
+0xe0,
+0x00,
+0x3e,
+0xc1,
+0x04,
+0x60,
+0x80,
+0xa8,
+0x03,
+0x09,
+0x00,
+0x12,
+0x64,
+0x00,
+0x02,
+0x44,
+0x8f,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0xba,
+0x14,
+0x8b,
+0xa1,
+0x01,
+0x55,
+0x00,
+0xc2,
+0x80,
+0x30,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0xb1,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x41,
+0xba,
+0x14,
+0x89,
+0x02,
+0x10,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0x5b,
+0x80,
+0x2b,
+0x00,
+0x09,
+0x2a,
+0x5a,
+0xd2,
+0x11,
+0x40,
+0x2e,
+0x14,
+0xd3,
+0x20,
+0x30,
+0xbc,
+0x0d,
+0x13,
+0x28,
+0x10,
+0xbc,
+0x05,
+0x1b,
+0xa1,
+0x48,
+0x40,
+0x00,
+0x01,
+0x02,
+0x10,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0x5b,
+0x84,
+0x23,
+0x00,
+0x09,
+0x2a,
+0x5a,
+0x42,
+0x11,
+0x12,
+0x57,
+0x08,
+0x73,
+0x80,
+0x00,
+0x5c,
+0x0d,
+0x0b,
+0xa1,
+0x11,
+0x52,
+0x02,
+0x99,
+0x82,
+0x42,
+0x52,
+0x02,
+0x09,
+0x82,
+0x80,
+0x23,
+0x2d,
+0xb5,
+0x0c,
+0xed,
+0x98,
+0x3c,
+0xa5,
+0x19,
+0x44,
+0x98,
+0x3c,
+0x95,
+0x40,
+0x08,
+0x18,
+0x2c,
+0x35,
+0x0c,
+0x64,
+0x98,
+0x34,
+0xb4,
+0x46,
+0x89,
+0x18,
+0x34,
+0x84,
+0x41,
+0x11,
+0x19,
+0x24,
+0x14,
+0x47,
+0x19,
+0x18,
+0x4c,
+0xa5,
+0xbc,
+0x41,
+0x18,
+0x40,
+0xb5,
+0x40,
+0x44,
+0x19,
+0x2c,
+0x14,
+0x40,
+0x99,
+0x32,
+0x00,
+0xd5,
+0xbc,
+0x00,
+0x10,
+0x05,
+0xb3,
+0x08,
+0x80,
+0xbc,
+0x03,
+0xa5,
+0x40,
+0xbe,
+0xb4,
+0x00,
+0xa3,
+0x80,
+0x02,
+0x50,
+0x8c,
+0x1b,
+0x00,
+0x0c,
+0x28,
+0x0c,
+0x92,
+0x11,
+0x92,
+0x21,
+0x98,
+0x03,
+0x78,
+0x41,
+0x28,
+0x01,
+0x25,
+0x84,
+0x64,
+0x10,
+0x01,
+0x33,
+0x78,
+0x82,
+0x54,
+0x04,
+0xd3,
+0xc0,
+0x3a,
+0x5b,
+0xc4,
+0x11,
+0x82,
+0x00,
+0x98,
+0xe8,
+0x82,
+0x09,
+0xa3,
+0x32,
+0x80,
+0x85,
+0x40,
+0x69,
+0x3c,
+0x02,
+0x93,
+0x78,
+0x81,
+0x98,
+0xe8,
+0x02,
+0x09,
+0x82,
+0x29,
+0x08,
+0x92,
+0x30,
+0x4a,
+0x32,
+0x01,
+0x0b,
+0xc0,
+0x4a,
+0x46,
+0x08,
+0x08,
+0x81,
+0x08,
+0x55,
+0x3f,
+0x69,
+0x92,
+0x81,
+0x64,
+0x00,
+0x02,
+0x47,
+0xef,
+0xa8,
+0x03,
+0x03,
+0x28,
+0x00,
+0xbc,
+0x10,
+0x16,
+0x80,
+0x00,
+0x1f,
+0xfc,
+0x93,
+0x01,
+0x70,
+0xbc,
+0x02,
+0x13,
+0x28,
+0x10,
+0xbc,
+0x0a,
+0x12,
+0x09,
+0xf1,
+0x29,
+0x08,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x0b,
+0x95,
+0x50,
+0x0c,
+0x3a,
+0x10,
+0x1b,
+0xa1,
+0x48,
+0x6e,
+0x00,
+0x03,
+0xec,
+0x10,
+0xa8,
+0x03,
+0x09,
+0x00,
+0x12,
+0x64,
+0x00,
+0x02,
+0x44,
+0x8f,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x66,
+0x66,
+0x66,
+0x66,
+0x00,
+0x3f,
+0xe6,
+0x66,
+0x66,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0xe0,
+0x00,
+0x00,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x07,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf0,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf0,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x01,
+0x13,
+0x92,
+0x73,
+0x00,
+0x7f,
+0xf3,
+0xa9,
+0x53,
+0x00,
+0x01,
+0x13,
+0x92,
+0x73,
+0x00,
+0xdc,
+0x56,
+0xfb,
+0x6f,
+0x00,
+0x7f,
+0xb8,
+0x17,
+0x23,
+0x00,
+0x23,
+0xf0,
+0xed,
+0x6d,
+0x00,
+0x90,
+0x26,
+0x15,
+0xe9,
+0x00,
+0x60,
+0xf0,
+0x19,
+0x01,
+0x00,
+0x71,
+0x16,
+0x2e,
+0xe9,
+0x00,
+0x81,
+0x47,
+0x58,
+0x21,
+0x00,
+0x7e,
+0xa4,
+0x32,
+0x5b,
+0x00,
+0x7f,
+0xeb,
+0x8a,
+0x7d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x65,
+0xac,
+0x8c,
+0x2f,
+0x00,
+0x5a,
+0x9d,
+0xf7,
+0xab,
+0x00,
+0x32,
+0xf5,
+0x2c,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x0f,
+0xd7,
+0x21,
+0x8f,
+0x00,
+0x60,
+0x51,
+0xbc,
+0xe3,
+0x00,
+0x0f,
+0xd7,
+0x21,
+0x8f,
+0x00,
+0x8a,
+0x4a,
+0x1d,
+0x99,
+0x00,
+0x75,
+0xb5,
+0xe2,
+0x67,
+0x00,
+0x3b,
+0x62,
+0x92,
+0x93,
+0x00,
+0xc8,
+0x64,
+0x0c,
+0xc5,
+0x00,
+0x3c,
+0x39,
+0x60,
+0xa7,
+0x00,
+0x94,
+0x28,
+0xff,
+0x89,
+0x00,
+0x6b,
+0xd7,
+0x00,
+0x77,
+0x00,
+0x36,
+0xc4,
+0x2d,
+0xd3,
+0x00,
+0xd1,
+0x5f,
+0x0d,
+0x4f,
+0x00,
+0x37,
+0xdc,
+0xc4,
+0xde,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x01,
+0x13,
+0x92,
+0x73,
+0x00,
+0x7f,
+0xf3,
+0xa9,
+0x53,
+0x00,
+0x01,
+0x13,
+0x92,
+0x73,
+0x00,
+0xdc,
+0x56,
+0xfb,
+0x6f,
+0x00,
+0x7f,
+0xb8,
+0x17,
+0x23,
+0x00,
+0x23,
+0xf0,
+0xed,
+0x6d,
+0x00,
+0x90,
+0x26,
+0x15,
+0xe9,
+0x00,
+0x60,
+0xf0,
+0x19,
+0x01,
+0x00,
+0x71,
+0x16,
+0x2e,
+0xe9,
+0x00,
+0x81,
+0x47,
+0x58,
+0x21,
+0x00,
+0x7e,
+0xa4,
+0x32,
+0x5b,
+0x00,
+0x7f,
+0xeb,
+0x8a,
+0x7d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x65,
+0xac,
+0x8c,
+0x2f,
+0x00,
+0x5a,
+0x9d,
+0xf7,
+0xab,
+0x00,
+0x32,
+0xf5,
+0x2c,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x0f,
+0xd7,
+0x21,
+0x8f,
+0x00,
+0x60,
+0x51,
+0xbc,
+0xe3,
+0x00,
+0x0f,
+0xd7,
+0x21,
+0x8f,
+0x00,
+0x8a,
+0x4a,
+0x1d,
+0x99,
+0x00,
+0x75,
+0xb5,
+0xe2,
+0x67,
+0x00,
+0x3b,
+0x62,
+0x92,
+0x93,
+0x00,
+0xc8,
+0x64,
+0x0c,
+0xc5,
+0x00,
+0x3c,
+0x39,
+0x60,
+0xa7,
+0x00,
+0x94,
+0x28,
+0xff,
+0x89,
+0x00,
+0x6b,
+0xd7,
+0x00,
+0x77,
+0x00,
+0x36,
+0xc4,
+0x2d,
+0xd3,
+0x00,
+0xd1,
+0x5f,
+0x0d,
+0x4f,
+0x00,
+0x37,
+0xdc,
+0xc4,
+0xde,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x03,
+0x03,
+0x01,
+0x00,
+0x00,
+0x01,
+0x00,
+0x01,
+0x01,
+0x00,
+0x01,
+0x01,
+0x03,
+0x03,
+0x00,
+0x00,
+0x04,
+0x9d,
+0x1d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x07,
+0xff,
+0xff,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0xb1,
+0xca,
+0x9e,
+0xb7,
+0x00,
+0x4c,
+0x9f,
+0xb1,
+0xad,
+0x00,
+0x54,
+0x54,
+0x77,
+0x9b,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x7f,
+0xf7,
+0xc6,
+0x15,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7e,
+0x67,
+0x33,
+0xd7,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0xb1,
+0xca,
+0x9e,
+0xb7,
+0x00,
+0x4c,
+0x9f,
+0xb1,
+0xad,
+0x00,
+0x54,
+0x54,
+0x77,
+0x9b,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x7f,
+0xf7,
+0xc6,
+0x15,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7e,
+0x67,
+0x33,
+0xd7,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x02,
+0x01,
+0x00,
+0x01,
+0x00,
+0x02,
+0x01,
+0x00,
+0x00,
+0x0a,
+0x47,
+0xe7,
+0x00,
+0x7f,
+0xeb,
+0x70,
+0x33,
+0x00,
+0x00,
+0x0a,
+0x47,
+0xe7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x64,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x20,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x44,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x15,
+0xd2,
+0xf3,
+0x00,
+0x0c,
+0xc9,
+0x9a,
+0x66,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x30,
+0x00,
+0x01,
+0x20,
+0x00,
+0x00,
+0x00,
+0x00,
+0x20,
+0x00,
+0x00,
+0x00,
+0x03,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xaf,
+0x00,
+0x00,
+0x00,
+0x00,
+0x70,
+0x00,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0xe3,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x03,
+0x00,
+0x00,
+0x3c,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x61,
+0xa8,
+0x00,
+0x00,
+0xd6,
+0x82,
+0x03,
+0x00,
+0x00,
+0x00,
+0x09,
+0xc4,
+0x00,
+0x00,
+0x00,
+0x4e,
+0x20,
+0x00,
+0x00,
+0x00,
+0x13,
+0x88,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0xc4,
+0x10,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0xc3,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0d,
+0xaf,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0xc6,
+0x00,
+0x00,
+0x00,
+0x02,
+0x8c,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x64,
+0x06,
+0x40,
+0x63,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x30,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x50,
+0x00,
+0x00,
+0x00,
+0x01,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xcc,
+0xcc,
+0xcc,
+0x00,
+0x19,
+0x99,
+0x99,
+0x99,
+0x00,
+0x33,
+0x33,
+0x33,
+0x33,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x66,
+0x66,
+0x66,
+0x66,
+0x00,
+0x5f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x46,
+0x66,
+0x66,
+0x65,
+0x00,
+0x2a,
+0xaa,
+0xaa,
+0xaa,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x73,
+0x33,
+0x33,
+0x32,
+0x00,
+0x7a,
+0xe1,
+0x47,
+0xad,
+0x00,
+0x43,
+0xd7,
+0x00,
+0x00,
+0x00,
+0x43,
+0xd7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0x00,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x64,
+0x87,
+0xed,
+0x4a,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x04,
+0xa0,
+0x00,
+0xff,
+0xfe,
+0x5f,
+0xe7,
+0x00,
+0x00,
+0x5b,
+0x05,
+0xb0,
+0x00,
+0xf5,
+0x55,
+0x55,
+0x56,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x40,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x5f,
+0x37,
+0x59,
+0xdf,
+0x00,
+0x5f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x40,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x7f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x18,
+0x00,
+0x00,
+0x00,
+0xae,
+0xb8,
+0x0f,
+0x00,
+0x7f,
+0xf4,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x00,
+0x41,
+0x00,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf8,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x49,
+0xe7,
+0x60,
+0x00,
+0x0c,
+0x28,
+0xf5,
+0xc0,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0xf1,
+0x91,
+0x96,
+0x00,
+0x7a,
+0x1c,
+0xdc,
+0xd3,
+0x00,
+0x02,
+0xf1,
+0x91,
+0x96,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xd6,
+0xe3,
+0xb3,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x00,
+0x66,
+0x84,
+0xe5,
+0x00,
+0x7f,
+0x32,
+0xf6,
+0x37,
+0x00,
+0x00,
+0x66,
+0x84,
+0xe5,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xd6,
+0xe3,
+0xb3,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x8c,
+0x5d,
+0xa4,
+0x55,
+0x00,
+0x67,
+0xf9,
+0x80,
+0xe3,
+0x00,
+0x74,
+0x57,
+0x25,
+0x39,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x68,
+0xc1,
+0x4b,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x06,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1d,
+0x53,
+0x00,
+0x00,
+0x13,
+0x9d,
+0x01,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1d,
+0xa1,
+0x2f,
+0x68,
+0x00,
+0x00,
+0x00,
+0x00,
+0xfa,
+0x00,
+0x00,
+0x00,
+0xc8,
+0xd7,
+0x00,
+0x00,
+0x7d,
+0x86,
+0x6e,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0xad,
+0x4f,
+0xc5,
+0xad,
+0x00,
+0x7c,
+0xd3,
+0x79,
+0x2f,
+0x00,
+0x55,
+0xdc,
+0xc1,
+0x23,
+0x00,
+0x00,
+0x00,
+0x0d,
+0xba,
+0x00,
+0x00,
+0x00,
+0x44,
+0xa2,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x0c,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x40,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x56,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x7a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xb2,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xe8,
+0x00,
+0x00,
+0x00,
+0x1d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x9c,
+0x00,
+0x00,
+0x00,
+0x20,
+0x9c,
+0x00,
+0x00,
+0x00,
+0x21,
+0x40,
+0x00,
+0x00,
+0x00,
+0x21,
+0x62,
+0x00,
+0x00,
+0x00,
+0x21,
+0xd4,
+0x00,
+0x00,
+0x00,
+0x22,
+0x08,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x22,
+0x98,
+0x00,
+0x00,
+0x00,
+0x23,
+0xa4,
+0x00,
+0x00,
+0x00,
+0x22,
+0xc0,
+0x00,
+0x00,
+0x00,
+0x22,
+0xc0,
+0x00,
+0x00,
+0x00,
+0x23,
+0x52,
+};
+
+#define VERNUM_04_17_00 0x04170008
+#define CAL_ID_04_17_00 0x00000001
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_04_17_01.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_04_17_01.h
new file mode 100644
index 0000000..1154f5a
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_04_17_01.h
@@ -0,0 +1,27495 @@
+/**
+ * @brief LC898123F40 OIS PRODUCT
+ *
+ * @author (C) 2017 ON Semiconductor.
+ *
+ **/
+
+// Time Stamp : 2018/02/21 13:34:21
+
+#define FromCodeBlockSize_04_17_01 11
+
+// [04170108] [00000002] [00000100]
+
+//#define MODULE_VENDOR 0x04
+//#define END_USER 0x17
+//#define USE_ACT 0x01
+
+const unsigned char CcMagicCodeF40_04_17_01[] = {
+0x00,
+0x01,
+0x02,
+0x03,
+0x04,
+0x05,
+0x06,
+0x07,
+0x08,
+0x09,
+0x00,
+0x00,
+0x00,
+0x00,
+0x54,
+0x00,
+0x00,
+0x00,
+0x13,
+0x00,
+0x00,
+0x00,
+0x00,
+0x06,
+0x98,
+0x00,
+0x00,
+0x00,
+0x01,
+0xfa,
+0x00,
+0x00,
+0x00,
+0x13,
+0x54,
+0x00,
+0x00,
+0x80,
+0x00,
+0x90,
+0x00,
+0x00,
+0x00,
+0xbb,
+0x67,
+0x00,
+0x8a,
+0x41,
+0x47,
+0xef,
+0x00,
+0x00,
+0x00,
+0xba,
+0x82,
+0x00,
+0xa9,
+0xce,
+0x6c,
+0xe2,
+0x00,
+0x00,
+0x00,
+0xc0,
+0x36,
+0x00,
+0x99,
+0xbd,
+0xd9,
+0x4c,
+0x00,
+0x00,
+0x00,
+0xb8,
+0xca,
+0x00,
+0xf7,
+0x7a,
+0xc1,
+0x7f,
+0x00,
+0x00,
+0x00,
+0xd0,
+0x7c,
+0x00,
+0x12,
+0x61,
+0x71,
+0x6f,
+0x00,
+0x00,
+0x00,
+0xbd,
+0xbb,
+0x00,
+0xf8,
+0x40,
+0xa3,
+0xf9,
+0x00,
+0x00,
+0x00,
+0xc2,
+0x21,
+0x00,
+0xec,
+0x9c,
+0xaa,
+0xba,
+0x00,
+0x00,
+0x00,
+0xcb,
+0xd0,
+0x00,
+0x90,
+0x1c,
+0x81,
+0x9e,
+0x00,
+0x00,
+0x00,
+0xa8,
+0xd8,
+0x00,
+0x61,
+0xe9,
+0xc7,
+0x7a,
+0x00,
+0x00,
+0x00,
+0x60,
+0x8f,
+0x00,
+0xd7,
+0x48,
+0xf4,
+0x43,
+0x00,
+0x00,
+0x00,
+0xb2,
+0x24,
+0x00,
+0x40,
+0x74,
+0x8a,
+0xac,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+};
+
+const unsigned char CcFromCodeF40_04_17_01[] = {
+0x64,
+0x00,
+0x00,
+0x02,
+0x87,
+0x64,
+0x00,
+0x00,
+0xfe,
+0xe7,
+0x64,
+0x00,
+0x01,
+0x02,
+0x67,
+0x64,
+0x00,
+0x01,
+0x08,
+0x87,
+0x64,
+0x00,
+0x01,
+0x08,
+0xa7,
+0x46,
+0x0b,
+0x03,
+0x80,
+0x00,
+0x64,
+0x00,
+0x01,
+0x08,
+0xc7,
+0x64,
+0x00,
+0x01,
+0x08,
+0xe7,
+0x64,
+0x00,
+0x01,
+0x09,
+0x07,
+0x64,
+0x00,
+0x01,
+0x09,
+0x27,
+0x64,
+0x00,
+0x01,
+0x09,
+0x47,
+0x64,
+0x00,
+0x01,
+0x09,
+0x67,
+0x64,
+0x00,
+0x01,
+0x09,
+0x87,
+0x64,
+0x00,
+0x01,
+0x09,
+0xa7,
+0x64,
+0x00,
+0x01,
+0x09,
+0xc7,
+0x46,
+0x0b,
+0x03,
+0x80,
+0x00,
+0x00,
+0x04,
+0x17,
+0x01,
+0x08,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xba,
+0x11,
+0x2b,
+0xa1,
+0x13,
+0x68,
+0x00,
+0x03,
+0x00,
+0x38,
+0x68,
+0x34,
+0x00,
+0x09,
+0x20,
+0x5c,
+0x81,
+0x09,
+0x8e,
+0xb9,
+0x5c,
+0x03,
+0xc3,
+0x0c,
+0x02,
+0x5c,
+0x96,
+0x00,
+0x40,
+0x52,
+0xa0,
+0x02,
+0x06,
+0x80,
+0x01,
+0x9e,
+0x7c,
+0x28,
+0x00,
+0xd2,
+0x84,
+0x05,
+0x0a,
+0x05,
+0x60,
+0x68,
+0x0f,
+0xc4,
+0xdb,
+0x00,
+0x84,
+0x05,
+0x0a,
+0x03,
+0x60,
+0x68,
+0x02,
+0x44,
+0xdf,
+0xc0,
+0xab,
+0xfc,
+0x08,
+0x02,
+0x50,
+0x40,
+0x00,
+0x00,
+0x80,
+0x60,
+0x66,
+0x00,
+0x01,
+0x10,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x0f,
+0xe0,
+0x5c,
+0x00,
+0x40,
+0x80,
+0x20,
+0x68,
+0x38,
+0x1c,
+0x09,
+0x24,
+0x5c,
+0x04,
+0x00,
+0x40,
+0x50,
+0xa2,
+0x26,
+0x08,
+0x60,
+0x7a,
+0x40,
+0x00,
+0x00,
+0x40,
+0x50,
+0x66,
+0x00,
+0x00,
+0xf8,
+0x40,
+0x68,
+0x20,
+0x00,
+0x00,
+0x24,
+0x68,
+0x00,
+0x00,
+0x08,
+0x20,
+0x98,
+0x90,
+0x96,
+0x60,
+0x00,
+0x12,
+0xfc,
+0x85,
+0xc0,
+0x13,
+0x18,
+0x80,
+0x89,
+0x8e,
+0x80,
+0x6e,
+0x00,
+0x03,
+0xb4,
+0x70,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb7,
+0x20,
+0x66,
+0x00,
+0x01,
+0x6f,
+0xe0,
+0x66,
+0x00,
+0x01,
+0x70,
+0xc0,
+0x68,
+0x00,
+0x01,
+0x0f,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb7,
+0x20,
+0x68,
+0x00,
+0x00,
+0x00,
+0xb9,
+0xba,
+0x11,
+0x00,
+0x00,
+0x00,
+0x66,
+0x00,
+0x01,
+0x1b,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x19,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x1c,
+0xc0,
+0x66,
+0x00,
+0x01,
+0x60,
+0x40,
+0x66,
+0x00,
+0x00,
+0xfd,
+0xa0,
+0x68,
+0x20,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x00,
+0x45,
+0xc0,
+0x66,
+0x00,
+0x00,
+0x1c,
+0xe0,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x66,
+0x00,
+0x01,
+0x52,
+0x40,
+0x6e,
+0x40,
+0x00,
+0x08,
+0x38,
+0x68,
+0x00,
+0x20,
+0x00,
+0x08,
+0x54,
+0x48,
+0x03,
+0x00,
+0x19,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x02,
+0x59,
+0x00,
+0x03,
+0x01,
+0x03,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x51,
+0x52,
+0x06,
+0x93,
+0xc0,
+0x99,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x52,
+0x38,
+0x1d,
+0x26,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x02,
+0x40,
+0x80,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x00,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x40,
+0x38,
+0x1c,
+0x22,
+0x40,
+0x80,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x50,
+0x66,
+0x00,
+0x01,
+0x20,
+0x80,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x7a,
+0x38,
+0x01,
+0xc6,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x96,
+0x60,
+0x04,
+0x03,
+0x30,
+0x86,
+0x80,
+0x00,
+0x00,
+0x0f,
+0x98,
+0x80,
+0x7a,
+0x66,
+0x00,
+0x40,
+0x34,
+0x60,
+0x32,
+0x82,
+0x0b,
+0xc0,
+0x98,
+0x88,
+0x00,
+0x06,
+0xc7,
+0x00,
+0x80,
+0x20,
+0x25,
+0x50,
+0x20,
+0x30,
+0x12,
+0x15,
+0x20,
+0x29,
+0x08,
+0x05,
+0x06,
+0xc7,
+0x00,
+0x80,
+0x25,
+0x23,
+0x21,
+0x40,
+0x40,
+0x00,
+0x03,
+0xff,
+0x02,
+0x66,
+0x00,
+0x40,
+0x34,
+0x20,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x68,
+0x38,
+0x08,
+0x10,
+0x24,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x5c,
+0x96,
+0x12,
+0x20,
+0x24,
+0x5c,
+0x0e,
+0x90,
+0x80,
+0x64,
+0x5c,
+0x09,
+0xc2,
+0x20,
+0x24,
+0x88,
+0x0e,
+0x4a,
+0x20,
+0x24,
+0x88,
+0x16,
+0x4a,
+0x20,
+0x24,
+0x88,
+0x1e,
+0x4a,
+0x20,
+0x24,
+0xa2,
+0x02,
+0x48,
+0x82,
+0x64,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x5c,
+0x00,
+0xca,
+0xc0,
+0xa8,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x03,
+0x32,
+0xa1,
+0x8b,
+0xff,
+0xa1,
+0xa0,
+0x00,
+0x48,
+0x03,
+0x03,
+0x25,
+0x89,
+0x8b,
+0xc0,
+0xc9,
+0x88,
+0x2e,
+0x02,
+0x48,
+0x9a,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x51,
+0x68,
+0x00,
+0x08,
+0x27,
+0x21,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x60,
+0x42,
+0x02,
+0xf8,
+0x23,
+0x52,
+0x40,
+0x00,
+0x00,
+0x60,
+0x61,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x20,
+0xa2,
+0x58,
+0x10,
+0xbc,
+0x35,
+0x88,
+0x83,
+0x60,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x98,
+0x80,
+0x26,
+0xc0,
+0x00,
+0x14,
+0xa0,
+0x03,
+0x00,
+0x80,
+0x68,
+0x00,
+0x02,
+0x32,
+0x20,
+0xbc,
+0x0b,
+0x89,
+0x88,
+0x02,
+0x30,
+0x08,
+0x0b,
+0xc2,
+0x81,
+0x5c,
+0x00,
+0x43,
+0x05,
+0xda,
+0x68,
+0x38,
+0x08,
+0x03,
+0x20,
+0x6c,
+0x70,
+0x10,
+0x06,
+0x52,
+0x84,
+0x1d,
+0x0b,
+0xc2,
+0x07,
+0x5c,
+0xb1,
+0x00,
+0x80,
+0x20,
+0x6c,
+0x70,
+0x10,
+0x20,
+0x0a,
+0x5c,
+0x00,
+0x40,
+0x40,
+0x08,
+0x88,
+0x0a,
+0x48,
+0x40,
+0x81,
+0x86,
+0x08,
+0x28,
+0x81,
+0x20,
+0x88,
+0x1a,
+0x48,
+0x40,
+0x83,
+0x86,
+0x08,
+0x98,
+0x82,
+0x20,
+0x68,
+0x00,
+0x00,
+0x96,
+0x24,
+0x80,
+0x00,
+0xb6,
+0xc0,
+0x00,
+0x30,
+0x64,
+0xa6,
+0xc0,
+0x00,
+0x2f,
+0x04,
+0x86,
+0xc0,
+0x00,
+0x31,
+0xc5,
+0x16,
+0xc4,
+0x00,
+0x24,
+0x65,
+0x26,
+0xc0,
+0x00,
+0x12,
+0xc5,
+0x38,
+0x60,
+0xc9,
+0x6c,
+0x00,
+0x02,
+0x9a,
+0x4b,
+0x40,
+0x00,
+0x00,
+0x40,
+0x50,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x7a,
+0x66,
+0x00,
+0x00,
+0xce,
+0x40,
+0x68,
+0x00,
+0x00,
+0x4b,
+0x20,
+0x5c,
+0x87,
+0x00,
+0x82,
+0xa4,
+0x68,
+0x00,
+0x00,
+0x5f,
+0x21,
+0x80,
+0x00,
+0x05,
+0x14,
+0x22,
+0x00,
+0x80,
+0x05,
+0x14,
+0x22,
+0x86,
+0x02,
+0x46,
+0xc6,
+0x81,
+0x00,
+0x04,
+0x88,
+0x82,
+0xe1,
+0x88,
+0x3e,
+0x06,
+0x83,
+0x40,
+0x80,
+0x02,
+0x5b,
+0xa0,
+0xc8,
+0x86,
+0xc4,
+0x90,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x66,
+0x00,
+0x01,
+0x1d,
+0x40,
+0x66,
+0x00,
+0x02,
+0x3c,
+0x60,
+0x6c,
+0x00,
+0x01,
+0x94,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x68,
+0x00,
+0x00,
+0xc2,
+0x20,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x24,
+0x68,
+0x00,
+0x00,
+0xc6,
+0x21,
+0x68,
+0x20,
+0x00,
+0xe6,
+0x25,
+0x68,
+0x00,
+0x00,
+0xac,
+0x22,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x23,
+0x66,
+0x00,
+0x01,
+0x38,
+0xc0,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x68,
+0x00,
+0x01,
+0x50,
+0x20,
+0x5c,
+0x83,
+0x0a,
+0xc0,
+0x20,
+0x5c,
+0x88,
+0x10,
+0x02,
+0x80,
+0x5c,
+0x87,
+0x18,
+0x40,
+0x02,
+0x57,
+0x00,
+0x80,
+0x83,
+0x20,
+0x5c,
+0x80,
+0x89,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0x8f,
+0x24,
+0x5c,
+0x08,
+0x50,
+0x03,
+0x01,
+0x51,
+0x60,
+0x20,
+0x22,
+0x00,
+0x86,
+0x00,
+0x35,
+0x70,
+0x0c,
+0x00,
+0x18,
+0x35,
+0x70,
+0x2d,
+0x80,
+0x30,
+0x15,
+0xc8,
+0x19,
+0x98,
+0x0c,
+0x35,
+0x16,
+0x0f,
+0x84,
+0x00,
+0x35,
+0x70,
+0x2c,
+0x18,
+0x00,
+0x16,
+0x80,
+0x00,
+0x1d,
+0x52,
+0x45,
+0x16,
+0x07,
+0x18,
+0x00,
+0x05,
+0x16,
+0x03,
+0x96,
+0x0d,
+0x76,
+0x80,
+0x00,
+0x17,
+0x92,
+0x15,
+0xc9,
+0x61,
+0x18,
+0xe4,
+0x95,
+0x24,
+0x55,
+0x80,
+0xa0,
+0x08,
+0x48,
+0x01,
+0x57,
+0x00,
+0x49,
+0x84,
+0xf9,
+0x5c,
+0x09,
+0xc1,
+0x8e,
+0x43,
+0x52,
+0x04,
+0xd9,
+0x61,
+0xd7,
+0x96,
+0x25,
+0x46,
+0x80,
+0x00,
+0x18,
+0x42,
+0x19,
+0x80,
+0x41,
+0x51,
+0x60,
+0x68,
+0x0a,
+0x08,
+0x84,
+0x80,
+0x25,
+0x70,
+0x89,
+0x08,
+0x2a,
+0x09,
+0x80,
+0x82,
+0x51,
+0x60,
+0xb8,
+0x83,
+0xa5,
+0x96,
+0x0d,
+0x79,
+0x60,
+0x55,
+0x98,
+0x4f,
+0x98,
+0x40,
+0x02,
+0x51,
+0x60,
+0xa0,
+0x68,
+0x01,
+0x51,
+0x60,
+0x71,
+0x60,
+0xd6,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x09,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x01,
+0x51,
+0x60,
+0x61,
+0x60,
+0xd4,
+0x51,
+0x61,
+0x71,
+0x60,
+0xd6,
+0x96,
+0x0f,
+0x69,
+0x60,
+0xd4,
+0x00,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x26,
+0xa0,
+0x35,
+0x94,
+0x0c,
+0x16,
+0x0b,
+0x95,
+0x50,
+0x24,
+0xbf,
+0x19,
+0x95,
+0xc0,
+0xe9,
+0x16,
+0x0f,
+0x14,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xa1,
+0xc0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0xf0,
+0xe7,
+0x68,
+0x20,
+0x00,
+0x04,
+0x20,
+0x5c,
+0x0b,
+0xe2,
+0xbf,
+0xf0,
+0x94,
+0x03,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x09,
+0x9a,
+0x01,
+0x20,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0xa0,
+0x00,
+0x12,
+0xa0,
+0x64,
+0x3b,
+0x10,
+0x43,
+0x23,
+0x20,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x38,
+0x17,
+0x42,
+0x59,
+0x28,
+0xbc,
+0x09,
+0x95,
+0xc0,
+0x32,
+0x2c,
+0x02,
+0x06,
+0x82,
+0x00,
+0x01,
+0x92,
+0x12,
+0xa0,
+0x64,
+0x3b,
+0x10,
+0x43,
+0x23,
+0xa0,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x5c,
+0x0a,
+0xe3,
+0x00,
+0x76,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xa9,
+0x5c,
+0x00,
+0x68,
+0x80,
+0x76,
+0x68,
+0x20,
+0x00,
+0x1b,
+0x21,
+0x38,
+0x09,
+0xc2,
+0xa0,
+0x76,
+0x3b,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x04,
+0x76,
+0x89,
+0x8e,
+0x88,
+0x88,
+0x0a,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x04,
+0x08,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x00,
+0x47,
+0x68,
+0x5c,
+0x00,
+0xa9,
+0x8e,
+0x88,
+0x5c,
+0x82,
+0x08,
+0x80,
+0xa0,
+0x39,
+0x02,
+0x0a,
+0x04,
+0x20,
+0x80,
+0x08,
+0xb8,
+0x00,
+0x09,
+0x51,
+0x91,
+0x50,
+0x00,
+0x00,
+0x80,
+0x00,
+0x16,
+0x83,
+0x40,
+0x0a,
+0x02,
+0x18,
+0x00,
+0x08,
+0x84,
+0x84,
+0xb6,
+0x80,
+0x00,
+0x05,
+0x82,
+0x28,
+0x00,
+0x09,
+0x80,
+0x00,
+0xa8,
+0x00,
+0x0b,
+0x6c,
+0x40,
+0x00,
+0x78,
+0x50,
+0x6c,
+0x40,
+0x00,
+0xc6,
+0x51,
+0x85,
+0x0c,
+0xb6,
+0x80,
+0x00,
+0x04,
+0x42,
+0x38,
+0x00,
+0x00,
+0x84,
+0x80,
+0x15,
+0x48,
+0x44,
+0x80,
+0x00,
+0xb8,
+0x40,
+0x82,
+0x80,
+0x00,
+0x38,
+0x48,
+0x51,
+0xa0,
+0x6c,
+0x06,
+0xc4,
+0x00,
+0x15,
+0xc5,
+0x3b,
+0x10,
+0x03,
+0x94,
+0x03,
+0x96,
+0x80,
+0x00,
+0x09,
+0xe2,
+0x15,
+0x44,
+0x66,
+0x05,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x18,
+0xc5,
+0x25,
+0x90,
+0x10,
+0x05,
+0x8c,
+0xa8,
+0x50,
+0x49,
+0x42,
+0x04,
+0x48,
+0x48,
+0x50,
+0x84,
+0x8c,
+0xb8,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x00,
+0x27,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x18,
+0x08,
+0x40,
+0x00,
+0x00,
+0x80,
+0xa0,
+0x68,
+0x00,
+0x01,
+0x00,
+0x09,
+0x94,
+0x03,
+0xc2,
+0x89,
+0x65,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0xa1,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x5c,
+0x08,
+0xe0,
+0x80,
+0xe0,
+0x24,
+0x12,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x06,
+0x60,
+0x00,
+0x02,
+0x9c,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x80,
+0x59,
+0x40,
+0x3c,
+0x28,
+0x96,
+0x53,
+0x20,
+0x28,
+0xbc,
+0x0c,
+0x98,
+0x80,
+0x36,
+0x88,
+0x07,
+0x66,
+0x80,
+0x00,
+0x15,
+0x02,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x86,
+0x80,
+0x04,
+0x01,
+0xcc,
+0x88,
+0x80,
+0x36,
+0x64,
+0x00,
+0x00,
+0x42,
+0x4f,
+0xa8,
+0x01,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0xab,
+0xfd,
+0x04,
+0x20,
+0x6f,
+0x88,
+0x0f,
+0x65,
+0xc0,
+0x03,
+0x08,
+0x04,
+0x98,
+0x82,
+0x4a,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa0,
+0x88,
+0x20,
+0x85,
+0x50,
+0x32,
+0x08,
+0x12,
+0x05,
+0xd4,
+0x83,
+0x08,
+0x18,
+0x85,
+0x50,
+0x32,
+0x20,
+0x02,
+0x08,
+0x80,
+0x09,
+0x58,
+0x0b,
+0x80,
+0x81,
+0xc8,
+0xbf,
+0xf2,
+0xa8,
+0x81,
+0x60,
+0x00,
+0x00,
+0x08,
+0x80,
+0xb6,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xfd,
+0x08,
+0x80,
+0x48,
+0x88,
+0x0f,
+0x66,
+0x82,
+0x00,
+0x08,
+0x52,
+0x08,
+0x81,
+0x7a,
+0x88,
+0x1e,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x0a,
+0x55,
+0x03,
+0xb8,
+0x81,
+0xa0,
+0x5d,
+0x4e,
+0x38,
+0x80,
+0x0a,
+0x59,
+0x09,
+0xc2,
+0x00,
+0x20,
+0x55,
+0x03,
+0xa0,
+0x81,
+0x4b,
+0xbf,
+0xf3,
+0xa8,
+0x80,
+0x48,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x40,
+0x80,
+0x68,
+0x20,
+0x00,
+0x85,
+0x20,
+0x88,
+0x20,
+0x05,
+0x16,
+0x02,
+0x20,
+0x08,
+0x08,
+0x80,
+0xb6,
+0x46,
+0x0a,
+0x41,
+0x40,
+0x64,
+0x94,
+0x0e,
+0x0a,
+0x80,
+0x30,
+0x5c,
+0x00,
+0x22,
+0xbf,
+0x20,
+0x88,
+0x1c,
+0x8a,
+0x40,
+0xc0,
+0x88,
+0x16,
+0x08,
+0x82,
+0x76,
+0x68,
+0x00,
+0x40,
+0x1a,
+0x48,
+0x88,
+0x2c,
+0x86,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x88,
+0x55,
+0x03,
+0x20,
+0x81,
+0x20,
+0x5d,
+0x48,
+0x30,
+0x82,
+0x88,
+0x59,
+0x0f,
+0x82,
+0x00,
+0x20,
+0x43,
+0xfa,
+0x50,
+0x81,
+0x60,
+0x55,
+0x03,
+0x20,
+0x81,
+0xca,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x42,
+0x80,
+0x5c,
+0x81,
+0x02,
+0x40,
+0xc0,
+0x5c,
+0x00,
+0x0a,
+0x00,
+0x71,
+0xa0,
+0xc1,
+0x29,
+0x48,
+0x35,
+0x95,
+0x03,
+0x7a,
+0x41,
+0xa1,
+0x23,
+0x08,
+0xc5,
+0x50,
+0x26,
+0x18,
+0x22,
+0x95,
+0xd4,
+0x80,
+0x9c,
+0x08,
+0x25,
+0x90,
+0xe4,
+0x15,
+0x0b,
+0x65,
+0x70,
+0xb9,
+0x15,
+0x03,
+0x45,
+0x70,
+0xf0,
+0x1c,
+0x88,
+0x34,
+0x3f,
+0x9d,
+0x15,
+0x8f,
+0x29,
+0x58,
+0x70,
+0x5c,
+0x86,
+0x0a,
+0x42,
+0xa3,
+0x88,
+0xa0,
+0x85,
+0x16,
+0x12,
+0x20,
+0x87,
+0x05,
+0x1a,
+0x12,
+0x20,
+0x46,
+0x15,
+0xb0,
+0x81,
+0x20,
+0x82,
+0x25,
+0x15,
+0x88,
+0x08,
+0x16,
+0x22,
+0xe1,
+0x12,
+0x2e,
+0x11,
+0x12,
+0x2b,
+0x09,
+0x51,
+0x58,
+0x90,
+0x18,
+0x41,
+0x51,
+0x59,
+0x30,
+0x18,
+0x42,
+0x54,
+0x09,
+0x00,
+0x18,
+0x40,
+0x54,
+0x08,
+0x10,
+0x18,
+0x7a,
+0x51,
+0x58,
+0x00,
+0x18,
+0x4a,
+0x51,
+0x58,
+0x80,
+0x18,
+0x40,
+0x81,
+0xac,
+0x08,
+0x80,
+0x63,
+0x40,
+0x00,
+0x00,
+0x81,
+0xe3,
+0x68,
+0x20,
+0x00,
+0x90,
+0x23,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x66,
+0x00,
+0x00,
+0x37,
+0xe0,
+0x5c,
+0x81,
+0x00,
+0x8a,
+0x08,
+0x51,
+0xa1,
+0x20,
+0x81,
+0xa3,
+0x5b,
+0x08,
+0x12,
+0xc0,
+0xc1,
+0x51,
+0x58,
+0x80,
+0x81,
+0x20,
+0x2e,
+0x11,
+0x25,
+0x70,
+0x88,
+0xa0,
+0x03,
+0x05,
+0x15,
+0x84,
+0xa0,
+0x46,
+0x15,
+0x15,
+0x89,
+0x01,
+0x84,
+0x15,
+0x15,
+0x93,
+0x01,
+0x84,
+0x25,
+0x40,
+0x90,
+0x01,
+0x84,
+0x05,
+0x40,
+0x81,
+0x01,
+0x87,
+0xa5,
+0x15,
+0x80,
+0x01,
+0x84,
+0xa5,
+0x15,
+0x88,
+0x01,
+0x84,
+0x08,
+0x1a,
+0xc0,
+0x88,
+0x06,
+0x36,
+0x82,
+0x00,
+0x09,
+0x8a,
+0x36,
+0x82,
+0x00,
+0x09,
+0x2a,
+0x46,
+0x82,
+0x00,
+0x09,
+0x5a,
+0x56,
+0x60,
+0x00,
+0x03,
+0x7e,
+0x8a,
+0x08,
+0x22,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x0e,
+0x00,
+0x00,
+0x00,
+0xba,
+0x14,
+0x82,
+0xe1,
+0xa8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x59,
+0x06,
+0x42,
+0xbf,
+0xd0,
+0x49,
+0xa9,
+0xb8,
+0x80,
+0x76,
+0xbc,
+0x19,
+0x89,
+0xa0,
+0x00,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x10,
+0x02,
+0xa0,
+0x76,
+0x68,
+0x00,
+0x09,
+0xc4,
+0x09,
+0x50,
+0x41,
+0x83,
+0xa1,
+0x11,
+0x44,
+0x08,
+0x81,
+0x01,
+0x58,
+0x5b,
+0x40,
+0x11,
+0x02,
+0x59,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x42,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x19,
+0x02,
+0x12,
+0x29,
+0x85,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x1a,
+0x33,
+0x60,
+0x00,
+0x37,
+0x80,
+0x0b,
+0xc1,
+0x77,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x10,
+0x02,
+0xa0,
+0x64,
+0x68,
+0x07,
+0xff,
+0xff,
+0xc9,
+0x50,
+0x41,
+0x03,
+0xa1,
+0x11,
+0x44,
+0x48,
+0x81,
+0x01,
+0x58,
+0x5b,
+0x40,
+0x11,
+0x02,
+0x59,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x42,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x19,
+0x02,
+0x12,
+0x29,
+0x85,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x02,
+0x33,
+0x60,
+0x00,
+0x37,
+0x80,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5b,
+0x02,
+0x03,
+0x01,
+0x01,
+0x50,
+0x43,
+0x11,
+0x84,
+0x0a,
+0x46,
+0x08,
+0x89,
+0x83,
+0x88,
+0x44,
+0x41,
+0x09,
+0x84,
+0x8b,
+0x98,
+0x30,
+0xa4,
+0x47,
+0x44,
+0xab,
+0xfe,
+0x05,
+0x08,
+0x20,
+0x08,
+0x07,
+0x60,
+0x88,
+0x82,
+0x37,
+0x80,
+0x06,
+0x80,
+0x00,
+0x9c,
+0x40,
+0x80,
+0x84,
+0x80,
+0x6a,
+0x00,
+0x4e,
+0x20,
+0x02,
+0x5b,
+0x44,
+0x11,
+0x01,
+0x58,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x40,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x26,
+0xa0,
+0x04,
+0xe2,
+0x00,
+0x12,
+0x98,
+0x52,
+0x32,
+0x01,
+0x0b,
+0xc0,
+0x3b,
+0x88,
+0x03,
+0x63,
+0x60,
+0x00,
+0x37,
+0x80,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0xab,
+0xf8,
+0x09,
+0x40,
+0x36,
+0x88,
+0x04,
+0xa8,
+0x80,
+0xe5,
+0x88,
+0x16,
+0x48,
+0x81,
+0xe3,
+0x88,
+0x26,
+0x28,
+0x82,
+0xf6,
+0x66,
+0x00,
+0x00,
+0x31,
+0xa8,
+0x40,
+0x00,
+0x01,
+0x48,
+0x35,
+0x5c,
+0x00,
+0x38,
+0x83,
+0x48,
+0x6a,
+0x00,
+0x02,
+0x71,
+0x00,
+0x5b,
+0x40,
+0x00,
+0x83,
+0xcb,
+0x88,
+0x45,
+0x49,
+0x05,
+0x58,
+0x00,
+0x00,
+0x08,
+0x80,
+0x0a,
+0x51,
+0x85,
+0xf8,
+0x82,
+0x22,
+0x98,
+0x2e,
+0x89,
+0xd0,
+0x02,
+0x88,
+0x4e,
+0x86,
+0x60,
+0x00,
+0x03,
+0x1a,
+0x89,
+0x50,
+0x35,
+0x5b,
+0x48,
+0x03,
+0x01,
+0xfe,
+0x52,
+0xcd,
+0x01,
+0x82,
+0x0b,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xc9,
+0x55,
+0x01,
+0xd1,
+0x80,
+0x0a,
+0xbc,
+0x07,
+0x88,
+0x86,
+0x52,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0xcd,
+0x88,
+0x38,
+0x16,
+0x80,
+0x01,
+0xff,
+0xfc,
+0x2b,
+0xc0,
+0x87,
+0x68,
+0x00,
+0x20,
+0x00,
+0x09,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x4d,
+0x40,
+0x00,
+0x00,
+0x83,
+0x81,
+0x68,
+0x00,
+0x20,
+0x00,
+0x02,
+0x59,
+0x0a,
+0x40,
+0x81,
+0xa2,
+0xbc,
+0x03,
+0xb3,
+0x90,
+0x10,
+0x40,
+0x00,
+0x01,
+0x50,
+0x52,
+0x55,
+0x02,
+0x70,
+0x81,
+0xe2,
+0x00,
+0x00,
+0x08,
+0x84,
+0xa8,
+0x88,
+0x82,
+0x35,
+0x18,
+0x5b,
+0x08,
+0x4c,
+0xa9,
+0xd8,
+0x04,
+0x98,
+0x2a,
+0x88,
+0x60,
+0x08,
+0x9d,
+0x80,
+0x38,
+0x83,
+0x0a,
+0x88,
+0x6e,
+0x46,
+0x60,
+0x00,
+0x03,
+0x1e,
+0x88,
+0x58,
+0x09,
+0x88,
+0x38,
+0xb3,
+0x20,
+0xf8,
+0xbc,
+0x13,
+0x85,
+0xc0,
+0xff,
+0x99,
+0x20,
+0x19,
+0x05,
+0x12,
+0x40,
+0x00,
+0x01,
+0x07,
+0x58,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x40,
+0x08,
+0x84,
+0x16,
+0x55,
+0x00,
+0x09,
+0x07,
+0x10,
+0x6a,
+0x00,
+0x02,
+0x71,
+0x02,
+0x29,
+0x88,
+0x23,
+0x20,
+0x10,
+0xbc,
+0x03,
+0xb3,
+0x81,
+0xff,
+0x36,
+0x04,
+0x13,
+0x78,
+0x41,
+0x52,
+0xce,
+0x41,
+0x83,
+0x4b,
+0x36,
+0x9c,
+0x26,
+0x80,
+0x01,
+0xff,
+0xfc,
+0xbb,
+0xc0,
+0x68,
+0x98,
+0x08,
+0xa3,
+0x01,
+0xf0,
+0xbc,
+0x09,
+0x56,
+0xa0,
+0x01,
+0xff,
+0xfc,
+0x1b,
+0xc0,
+0x67,
+0x68,
+0x00,
+0x20,
+0x00,
+0x0b,
+0x30,
+0x1f,
+0x0b,
+0xc0,
+0x25,
+0x6a,
+0x00,
+0x20,
+0x00,
+0x01,
+0x5c,
+0x80,
+0x80,
+0x81,
+0x22,
+0x98,
+0x34,
+0x19,
+0x50,
+0x51,
+0x88,
+0x16,
+0x20,
+0x00,
+0x00,
+0x88,
+0x6a,
+0x38,
+0x83,
+0x08,
+0x66,
+0x00,
+0x00,
+0x35,
+0x88,
+0x55,
+0x00,
+0x08,
+0x58,
+0x09,
+0x5c,
+0x0f,
+0xf9,
+0x83,
+0x0a,
+0x36,
+0x98,
+0x12,
+0x59,
+0xc0,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xca,
+0xbc,
+0x07,
+0x89,
+0x80,
+0x4b,
+0x30,
+0x1b,
+0x8b,
+0xc0,
+0xcd,
+0x88,
+0x48,
+0xb6,
+0xa0,
+0x01,
+0xff,
+0xfc,
+0x0b,
+0xc0,
+0x87,
+0x68,
+0x00,
+0x20,
+0x00,
+0x0a,
+0x30,
+0x1b,
+0x8b,
+0xc0,
+0x4d,
+0x40,
+0x00,
+0x00,
+0x84,
+0x8b,
+0x6a,
+0x00,
+0x20,
+0x00,
+0x00,
+0x5c,
+0x80,
+0x80,
+0x80,
+0xa2,
+0x5d,
+0x8e,
+0x39,
+0x83,
+0x00,
+0x59,
+0x0d,
+0xc0,
+0x86,
+0x09,
+0x95,
+0x05,
+0x08,
+0x83,
+0x49,
+0x88,
+0x3c,
+0xb8,
+0x80,
+0xe2,
+0xbf,
+0x75,
+0xa8,
+0x80,
+0x0a,
+0x88,
+0x2b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x08,
+0x00,
+0x00,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0x79,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x08,
+0x68,
+0x00,
+0x01,
+0x84,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x48,
+0x68,
+0x00,
+0x01,
+0x8f,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x88,
+0x68,
+0x00,
+0x40,
+0x1d,
+0xc8,
+0x68,
+0x20,
+0x01,
+0xcb,
+0x20,
+0x66,
+0x00,
+0x00,
+0x26,
+0x08,
+0x5c,
+0x00,
+0xeb,
+0x80,
+0x00,
+0x68,
+0x00,
+0x40,
+0x1e,
+0x88,
+0x68,
+0x20,
+0x01,
+0xce,
+0x20,
+0x66,
+0x00,
+0x00,
+0x26,
+0x08,
+0x38,
+0x01,
+0xd8,
+0x80,
+0x36,
+0x68,
+0x00,
+0x40,
+0x1f,
+0x48,
+0x68,
+0x20,
+0x01,
+0xd1,
+0x20,
+0x64,
+0x00,
+0x00,
+0x26,
+0x0f,
+0x5c,
+0x00,
+0xea,
+0x80,
+0x10,
+0x39,
+0x02,
+0x08,
+0x00,
+0x7a,
+0x80,
+0x07,
+0xa8,
+0x08,
+0x7a,
+0x80,
+0x07,
+0xa8,
+0x08,
+0x7a,
+0x80,
+0x87,
+0xa4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xfe,
+0x08,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x66,
+0x80,
+0x04,
+0x01,
+0x00,
+0x88,
+0x81,
+0x7a,
+0x88,
+0x1c,
+0x86,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x00,
+0x55,
+0x02,
+0x00,
+0x80,
+0x20,
+0x5d,
+0x00,
+0x03,
+0x00,
+0xe2,
+0x58,
+0x04,
+0x02,
+0x00,
+0x20,
+0x88,
+0x18,
+0x14,
+0x3f,
+0x9d,
+0x08,
+0x06,
+0x05,
+0x50,
+0x26,
+0x08,
+0x15,
+0x00,
+0x00,
+0x00,
+0x88,
+0x0b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x68,
+0x68,
+0x34,
+0x04,
+0x14,
+0x21,
+0x6c,
+0x68,
+0x08,
+0x28,
+0x49,
+0x00,
+0x00,
+0x08,
+0x48,
+0x89,
+0x84,
+0x04,
+0x94,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xc0,
+0x4f,
+0x5c,
+0x09,
+0xab,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x08,
+0x25,
+0x96,
+0x0b,
+0xff,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x6c,
+0x68,
+0x08,
+0x28,
+0x49,
+0x00,
+0x00,
+0x08,
+0x40,
+0x09,
+0x84,
+0x8c,
+0x94,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xc0,
+0x4f,
+0x5c,
+0x09,
+0xab,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x08,
+0x25,
+0x96,
+0x0b,
+0xff,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x59,
+0x01,
+0x41,
+0x48,
+0x2f,
+0xbc,
+0x0d,
+0x95,
+0xc0,
+0x87,
+0x14,
+0x02,
+0xd3,
+0x20,
+0x20,
+0xbc,
+0x05,
+0x05,
+0x24,
+0xdc,
+0x3a,
+0x14,
+0x89,
+0x48,
+0x60,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0x4d,
+0x43,
+0xa1,
+0x48,
+0x94,
+0x06,
+0x00,
+0x00,
+0x00,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x40,
+0x52,
+0x0d,
+0xc3,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x00,
+0x00,
+0x00,
+0x52,
+0x0d,
+0x43,
+0xa1,
+0x48,
+0x94,
+0x06,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x05,
+0x20,
+0x40,
+0x7a,
+0x46,
+0x0a,
+0x40,
+0x40,
+0xfa,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x40,
+0x46,
+0x08,
+0x88,
+0x02,
+0x0a,
+0x08,
+0x80,
+0x04,
+0x60,
+0xa4,
+0x00,
+0x24,
+0xc8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x01,
+0x84,
+0x00,
+0x88,
+0x40,
+0x8a,
+0x30,
+0x93,
+0x0b,
+0xc0,
+0x5c,
+0xb0,
+0x00,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x40,
+0xfa,
+0x98,
+0xe8,
+0x8b,
+0xa1,
+0x40,
+0x84,
+0x08,
+0x03,
+0x28,
+0x00,
+0xbc,
+0x05,
+0x08,
+0x40,
+0x00,
+0x55,
+0x02,
+0x23,
+0xa1,
+0x48,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x80,
+0x43,
+0xc0,
+0x5f,
+0x5c,
+0x00,
+0x31,
+0x8e,
+0x80,
+0x55,
+0x02,
+0x01,
+0x40,
+0x46,
+0x3a,
+0x80,
+0x03,
+0x01,
+0x00,
+0xbf,
+0xfb,
+0x2b,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x1c,
+0x04,
+0x21,
+0x38,
+0x00,
+0xe8,
+0x48,
+0x48,
+0xa0,
+0x82,
+0x18,
+0x48,
+0x7a,
+0xa0,
+0x82,
+0x18,
+0x48,
+0x4a,
+0xa0,
+0xcc,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x80,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xb0,
+0x68,
+0x00,
+0x00,
+0x9e,
+0x20,
+0x68,
+0x00,
+0x01,
+0x50,
+0x24,
+0x5c,
+0x00,
+0x10,
+0x00,
+0x01,
+0x5c,
+0x08,
+0x2a,
+0x00,
+0xa1,
+0x55,
+0x00,
+0xb1,
+0x48,
+0x2f,
+0x52,
+0xcb,
+0xc0,
+0x22,
+0x08,
+0x55,
+0x00,
+0x98,
+0x82,
+0x48,
+0xa0,
+0xd4,
+0x18,
+0x40,
+0x09,
+0x88,
+0x36,
+0x44,
+0x20,
+0x44,
+0x88,
+0x3f,
+0x65,
+0x50,
+0x0a,
+0x08,
+0x2e,
+0x16,
+0x80,
+0x00,
+0x09,
+0x62,
+0x06,
+0xc0,
+0x00,
+0x29,
+0xa0,
+0x28,
+0x40,
+0x03,
+0x84,
+0x08,
+0x83,
+0x81,
+0x08,
+0x25,
+0x83,
+0x8b,
+0xc1,
+0x09,
+0x55,
+0x01,
+0x81,
+0x82,
+0x8b,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xae,
+0x38,
+0x12,
+0x82,
+0x58,
+0x30,
+0xbc,
+0x09,
+0x85,
+0x50,
+0x1c,
+0x18,
+0x2c,
+0xa6,
+0x80,
+0x00,
+0x09,
+0x82,
+0x06,
+0xc0,
+0x00,
+0x29,
+0xc0,
+0xb8,
+0x40,
+0x8a,
+0x40,
+0x00,
+0x00,
+0x40,
+0x00,
+0x57,
+0x02,
+0xc8,
+0x82,
+0x03,
+0x57,
+0x06,
+0x92,
+0x08,
+0x00,
+0x2e,
+0x16,
+0x31,
+0x08,
+0x4e,
+0x54,
+0x0e,
+0x81,
+0xa0,
+0x42,
+0x6c,
+0x40,
+0x01,
+0x3e,
+0x08,
+0x44,
+0x20,
+0x81,
+0x80,
+0x09,
+0x6e,
+0x40,
+0x01,
+0x9d,
+0x23,
+0x6c,
+0x40,
+0x03,
+0x24,
+0x0b,
+0x6c,
+0x40,
+0x01,
+0x6e,
+0x08,
+0x20,
+0x0c,
+0x94,
+0xc1,
+0x42,
+0x80,
+0x84,
+0x18,
+0x82,
+0x61,
+0x00,
+0x00,
+0x06,
+0xe4,
+0x00,
+0x19,
+0xf2,
+0x46,
+0xc4,
+0x00,
+0x25,
+0x60,
+0x92,
+0x01,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x0b,
+0x51,
+0x8b,
+0x48,
+0x48,
+0x40,
+0x30,
+0x07,
+0x86,
+0xe4,
+0x00,
+0x35,
+0x52,
+0x65,
+0x00,
+0xc8,
+0x3c,
+0x53,
+0xd8,
+0x60,
+0x40,
+0x22,
+0x96,
+0xc2,
+0xe1,
+0x3c,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x48,
+0x22,
+0x96,
+0x43,
+0x01,
+0x60,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x48,
+0xbc,
+0x45,
+0x46,
+0xc0,
+0x00,
+0x36,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x05,
+0x16,
+0xc0,
+0x00,
+0x37,
+0x60,
+0xa3,
+0x20,
+0x30,
+0x40,
+0x00,
+0x03,
+0xc3,
+0x90,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x0a,
+0x2e,
+0x9a,
+0x42,
+0xa0,
+0x64,
+0x6c,
+0x40,
+0x02,
+0x1c,
+0x0a,
+0x6c,
+0x00,
+0x03,
+0x68,
+0x48,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0xec,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x48,
+0x51,
+0x89,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x49,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x7a,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x51,
+0x8b,
+0x6b,
+0xc2,
+0xcf,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x49,
+0x68,
+0x00,
+0x00,
+0x70,
+0x25,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x24,
+0x82,
+0x87,
+0xa6,
+0x80,
+0x00,
+0x1b,
+0x12,
+0x68,
+0x20,
+0x7a,
+0xa2,
+0x80,
+0x38,
+0x28,
+0x7a,
+0xa2,
+0x00,
+0x78,
+0x58,
+0xfa,
+0x68,
+0x00,
+0x01,
+0xb8,
+0x22,
+0x82,
+0x07,
+0xa8,
+0x78,
+0xfa,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x7a,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x49,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x87,
+0x00,
+0x86,
+0xc4,
+0x00,
+0x3e,
+0x24,
+0x88,
+0x51,
+0xfa,
+0x87,
+0x1f,
+0xa8,
+0x68,
+0xfa,
+0x86,
+0x0f,
+0xa0,
+0x00,
+0x00,
+0x42,
+0x05,
+0x78,
+0x50,
+0x0a,
+0x85,
+0x8c,
+0xab,
+0xc0,
+0x8f,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0xa1,
+0x82,
+0x58,
+0x84,
+0x65,
+0x66,
+0x00,
+0x00,
+0xcd,
+0x80,
+0x68,
+0x00,
+0x00,
+0x89,
+0x20,
+0x88,
+0x2a,
+0x18,
+0x40,
+0x48,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x40,
+0x00,
+0x00,
+0x82,
+0x20,
+0x66,
+0x00,
+0x00,
+0xcd,
+0x80,
+0x68,
+0x00,
+0x00,
+0x92,
+0x20,
+0x68,
+0x00,
+0x00,
+0x89,
+0x21,
+0x5c,
+0x82,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x80,
+0x80,
+0x98,
+0x82,
+0x61,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x2c,
+0x5c,
+0x87,
+0x00,
+0x82,
+0x20,
+0x88,
+0x1e,
+0xc8,
+0x02,
+0x48,
+0x68,
+0x20,
+0x00,
+0xdf,
+0x2c,
+0x68,
+0x20,
+0x00,
+0xdc,
+0x2d,
+0x68,
+0x20,
+0x00,
+0xde,
+0x22,
+0x88,
+0x26,
+0x06,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x46,
+0x80,
+0x00,
+0x0a,
+0x82,
+0x16,
+0x82,
+0x00,
+0x0d,
+0x32,
+0x58,
+0x80,
+0x6c,
+0x88,
+0x0e,
+0xd8,
+0x81,
+0x62,
+0x66,
+0x00,
+0x00,
+0x99,
+0x48,
+0x40,
+0x00,
+0x01,
+0x82,
+0x09,
+0x5c,
+0x82,
+0x00,
+0x82,
+0x20,
+0x68,
+0x00,
+0x00,
+0x92,
+0x21,
+0x84,
+0x34,
+0x86,
+0x80,
+0x00,
+0x07,
+0xa2,
+0x08,
+0x08,
+0x09,
+0x40,
+0x00,
+0x00,
+0x82,
+0x61,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x5c,
+0x87,
+0x00,
+0x82,
+0x20,
+0x88,
+0x42,
+0x18,
+0x02,
+0x48,
+0x88,
+0x1e,
+0x16,
+0x82,
+0x00,
+0x0d,
+0xfa,
+0xc6,
+0x82,
+0x00,
+0x0d,
+0xd2,
+0xd6,
+0x82,
+0x00,
+0x0d,
+0xea,
+0x28,
+0x82,
+0x60,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x00,
+0x00,
+0xaa,
+0x21,
+0x68,
+0x20,
+0x00,
+0xd5,
+0x25,
+0x88,
+0x06,
+0xc8,
+0x80,
+0xed,
+0x88,
+0x16,
+0x26,
+0x60,
+0x00,
+0x09,
+0x94,
+0x89,
+0x82,
+0x09,
+0x5c,
+0x82,
+0x00,
+0x83,
+0x21,
+0x68,
+0x00,
+0x01,
+0x49,
+0x20,
+0x84,
+0x80,
+0x98,
+0x82,
+0x21,
+0x80,
+0x04,
+0x98,
+0x82,
+0x60,
+0x40,
+0x00,
+0x00,
+0x4b,
+0x48,
+0x68,
+0x00,
+0x01,
+0x3a,
+0x20,
+0x68,
+0x20,
+0x01,
+0x8f,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x59,
+0x03,
+0x40,
+0x82,
+0x21,
+0xbc,
+0x66,
+0x88,
+0x48,
+0x48,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc5,
+0xd1,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x2a,
+0x8e,
+0xc3,
+0x20,
+0xe0,
+0xbc,
+0x27,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xe0,
+0x86,
+0xc0,
+0x00,
+0x11,
+0x40,
+0xa2,
+0x81,
+0x30,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x4b,
+0x38,
+0x10,
+0x72,
+0x41,
+0xed,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x49,
+0x2e,
+0x13,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x06,
+0xd3,
+0x81,
+0x0d,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x24,
+0x16,
+0x56,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x90,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x2a,
+0x8e,
+0xd3,
+0x20,
+0xe8,
+0xbc,
+0x0f,
+0x93,
+0x90,
+0x20,
+0x68,
+0x00,
+0x00,
+0x70,
+0x21,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x09,
+0x80,
+0x87,
+0xaa,
+0x08,
+0x00,
+0x80,
+0x87,
+0xa8,
+0x40,
+0xfa,
+0x6c,
+0x40,
+0x01,
+0x5a,
+0x49,
+0x84,
+0x8f,
+0xab,
+0xc0,
+0x27,
+0x5c,
+0x81,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x38,
+0x18,
+0x62,
+0x89,
+0xa5,
+0x30,
+0x1a,
+0x8b,
+0xc2,
+0x70,
+0x6c,
+0x40,
+0x02,
+0x0e,
+0x0b,
+0x6c,
+0x00,
+0x01,
+0x26,
+0x00,
+0x28,
+0x1c,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x04,
+0xb3,
+0x81,
+0x25,
+0x24,
+0x16,
+0x46,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x82,
+0xe1,
+0xc4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x38,
+0x12,
+0xf2,
+0x41,
+0xe4,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x28,
+0x9a,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x11,
+0x16,
+0x80,
+0x00,
+0x07,
+0xa2,
+0x16,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x88,
+0x08,
+0x7a,
+0xa0,
+0x80,
+0x08,
+0x08,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0xa4,
+0x88,
+0x48,
+0xfa,
+0xbc,
+0x05,
+0x7b,
+0xc0,
+0x4f,
+0x38,
+0x12,
+0x5b,
+0xc0,
+0x2f,
+0x38,
+0x12,
+0x53,
+0x81,
+0x25,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x21,
+0x6c,
+0x40,
+0x03,
+0xd2,
+0x08,
+0x84,
+0xb8,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0xb5,
+0x24,
+0xbe,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x7d,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x21,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x09,
+0x84,
+0xb8,
+0xb3,
+0x01,
+0x38,
+0xbc,
+0x05,
+0x93,
+0x81,
+0x2e,
+0x52,
+0x4d,
+0x6b,
+0xc0,
+0x5f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x24,
+0x1a,
+0xd6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x93,
+0x81,
+0x35,
+0x6c,
+0x40,
+0x01,
+0xb8,
+0x08,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x89,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x08,
+0x52,
+0x4b,
+0x28,
+0x83,
+0xb6,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x6f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x52,
+0x0b,
+0x28,
+0x83,
+0xb6,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x38,
+0x13,
+0xd6,
+0xc4,
+0x00,
+0x1b,
+0xa0,
+0x83,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x38,
+0x1d,
+0x56,
+0xc0,
+0x00,
+0x36,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x0a,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x86,
+0xc0,
+0x00,
+0x37,
+0x60,
+0xa3,
+0x20,
+0x30,
+0xbc,
+0x04,
+0x15,
+0x24,
+0xb2,
+0x3c,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x82,
+0x41,
+0x64,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xa0,
+0x86,
+0xc4,
+0x00,
+0x25,
+0x60,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x00,
+0x02,
+0x32,
+0x21,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x14,
+0xa6,
+0x1a,
+0x80,
+0x50,
+0x68,
+0x00,
+0x00,
+0x8a,
+0x20,
+0x5c,
+0x85,
+0x02,
+0xbf,
+0xa0,
+0x80,
+0x20,
+0x98,
+0x81,
+0x60,
+0x68,
+0x20,
+0x01,
+0xb4,
+0x2c,
+0x88,
+0x1f,
+0x66,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x46,
+0x82,
+0x00,
+0x0c,
+0xe2,
+0x56,
+0x80,
+0x00,
+0x1a,
+0x22,
+0x08,
+0x80,
+0x6c,
+0x66,
+0x00,
+0x00,
+0xb0,
+0x40,
+0x68,
+0x20,
+0x00,
+0xdc,
+0x2c,
+0x6c,
+0x00,
+0x03,
+0x44,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x60,
+0x00,
+0x09,
+0x82,
+0x89,
+0x82,
+0x09,
+0x68,
+0x00,
+0x00,
+0x93,
+0x20,
+0x5c,
+0x85,
+0x00,
+0x81,
+0x24,
+0x80,
+0x20,
+0x98,
+0x60,
+0x48,
+0x88,
+0x26,
+0x06,
+0x82,
+0x00,
+0x1b,
+0x62,
+0xc6,
+0x82,
+0x00,
+0x0b,
+0x42,
+0x46,
+0x82,
+0x00,
+0x0c,
+0xf2,
+0x56,
+0x80,
+0x00,
+0x1a,
+0x42,
+0x08,
+0x80,
+0x6c,
+0x66,
+0x00,
+0x00,
+0xb0,
+0x40,
+0x68,
+0x20,
+0x00,
+0xdd,
+0x2c,
+0x6c,
+0x00,
+0x03,
+0x48,
+0x48,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcf,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x60,
+0x00,
+0x09,
+0x82,
+0x89,
+0x82,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x20,
+0x88,
+0x22,
+0x48,
+0x02,
+0x09,
+0x86,
+0x04,
+0x88,
+0x81,
+0x60,
+0x68,
+0x00,
+0x00,
+0x70,
+0x21,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb1,
+0xa0,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x24,
+0x88,
+0x22,
+0x0a,
+0x20,
+0x44,
+0x80,
+0x20,
+0x98,
+0x60,
+0x48,
+0x88,
+0x2e,
+0x48,
+0x82,
+0x60,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb1,
+0xa0,
+0x88,
+0x22,
+0x06,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x4a,
+0x00,
+0x40,
+0x86,
+0x60,
+0x36,
+0xc4,
+0x00,
+0x2a,
+0xc0,
+0x25,
+0x80,
+0x4c,
+0x04,
+0x04,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x08,
+0xbc,
+0x06,
+0xa5,
+0xc0,
+0x82,
+0x88,
+0x36,
+0x05,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x20,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x09,
+0x84,
+0x60,
+0x33,
+0x00,
+0x98,
+0xbc,
+0x05,
+0xa3,
+0x81,
+0x0c,
+0x52,
+0x49,
+0x6b,
+0xc0,
+0x6f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x52,
+0x09,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x88,
+0x12,
+0x16,
+0x82,
+0x00,
+0x0d,
+0xe2,
+0x56,
+0x60,
+0x00,
+0x0b,
+0x70,
+0x88,
+0x49,
+0x09,
+0x88,
+0x22,
+0x06,
+0x80,
+0x00,
+0x0a,
+0x02,
+0x48,
+0x41,
+0x09,
+0x40,
+0x00,
+0x00,
+0x60,
+0x48,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xde,
+0xa5,
+0x66,
+0x00,
+0x00,
+0xb7,
+0x00,
+0x68,
+0x00,
+0x00,
+0xa0,
+0x20,
+0x68,
+0x20,
+0x00,
+0xd7,
+0x24,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x02,
+0x5c,
+0x83,
+0x04,
+0x00,
+0xc0,
+0x5d,
+0x0e,
+0x2a,
+0x00,
+0x03,
+0x51,
+0x85,
+0x68,
+0x00,
+0x48,
+0x98,
+0x26,
+0x88,
+0x82,
+0x64,
+0x9e,
+0x00,
+0x48,
+0x60,
+0x09,
+0x68,
+0x20,
+0x00,
+0xcc,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcd,
+0x21,
+0x86,
+0x04,
+0x98,
+0x48,
+0x49,
+0x88,
+0x3e,
+0x08,
+0x84,
+0x63,
+0x88,
+0x4d,
+0x08,
+0x81,
+0x48,
+0x66,
+0x00,
+0x00,
+0xc5,
+0x28,
+0x40,
+0x00,
+0x02,
+0x20,
+0x00,
+0x5c,
+0x09,
+0x28,
+0x84,
+0x20,
+0x88,
+0x3a,
+0x49,
+0x46,
+0x2a,
+0x52,
+0xca,
+0x82,
+0x24,
+0x84,
+0xbc,
+0x09,
+0x85,
+0xc8,
+0x30,
+0x08,
+0x3e,
+0x46,
+0x82,
+0x00,
+0x15,
+0x92,
+0x40,
+0x00,
+0x00,
+0x42,
+0x04,
+0x78,
+0x22,
+0x03,
+0x40,
+0x00,
+0x00,
+0x60,
+0x0b,
+0x68,
+0x20,
+0x01,
+0x5d,
+0x24,
+0x00,
+0x00,
+0x08,
+0x22,
+0x03,
+0x40,
+0x00,
+0x00,
+0x60,
+0x0b,
+0x5c,
+0x09,
+0x78,
+0x84,
+0x4b,
+0x25,
+0x9d,
+0x0b,
+0xc0,
+0x88,
+0x88,
+0x55,
+0x30,
+0x00,
+0x00,
+0x88,
+0x40,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x04,
+0x9b,
+0xc4,
+0x9f,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x49,
+0x25,
+0x95,
+0x0b,
+0xc3,
+0x01,
+0x6c,
+0x00,
+0x01,
+0x40,
+0x0a,
+0x66,
+0x00,
+0x00,
+0xca,
+0xc8,
+0x5c,
+0x00,
+0x28,
+0x40,
+0x0b,
+0x6c,
+0x40,
+0x01,
+0xae,
+0x09,
+0x5d,
+0x0a,
+0x28,
+0x82,
+0x20,
+0x23,
+0x0a,
+0xa9,
+0x84,
+0xa8,
+0x9c,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2a,
+0x80,
+0xb8,
+0x40,
+0x0a,
+0x08,
+0xb2,
+0x09,
+0x80,
+0x82,
+0x2e,
+0x0a,
+0x73,
+0x20,
+0x38,
+0xbc,
+0x03,
+0xd8,
+0x85,
+0x09,
+0xbc,
+0x02,
+0xf0,
+0x8d,
+0x20,
+0x98,
+0xe8,
+0x28,
+0x84,
+0x09,
+0x2e,
+0x0a,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0x00,
+0x99,
+0x80,
+0x82,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0xa4,
+0x6c,
+0x40,
+0x02,
+0xaa,
+0x09,
+0x2e,
+0x12,
+0xd3,
+0x20,
+0x28,
+0xbc,
+0x20,
+0x56,
+0xc4,
+0x00,
+0x18,
+0x05,
+0x26,
+0xc4,
+0x00,
+0x15,
+0x05,
+0x2b,
+0xc1,
+0xb7,
+0x6c,
+0x40,
+0x01,
+0x50,
+0x52,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x7f,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x52,
+0x59,
+0x01,
+0x03,
+0x01,
+0x05,
+0xbc,
+0x09,
+0x96,
+0xc4,
+0x00,
+0x3d,
+0x00,
+0x82,
+0x59,
+0x60,
+0xbc,
+0x0e,
+0x12,
+0x49,
+0x65,
+0x6c,
+0x40,
+0x03,
+0xd0,
+0x49,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x97,
+0x6c,
+0x40,
+0x02,
+0xbc,
+0x02,
+0x24,
+0x16,
+0x56,
+0xc4,
+0x00,
+0x3d,
+0x04,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x05,
+0x26,
+0xc4,
+0x00,
+0x18,
+0x05,
+0x23,
+0x90,
+0x20,
+0x88,
+0x3a,
+0x08,
+0x84,
+0x89,
+0x5b,
+0x4a,
+0x12,
+0x00,
+0x04,
+0x80,
+0x00,
+0x95,
+0xb4,
+0xa1,
+0x98,
+0x08,
+0x29,
+0x80,
+0xc3,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x0b,
+0x30,
+0x8d,
+0x0b,
+0xc0,
+0x6c,
+0x5c,
+0x08,
+0xa0,
+0x82,
+0x60,
+0x52,
+0x49,
+0xe3,
+0xc0,
+0x5f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x24,
+0x13,
+0xc6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x83,
+0x81,
+0x1c,
+0x86,
+0x08,
+0x25,
+0xb4,
+0x41,
+0x08,
+0x10,
+0x35,
+0xb4,
+0x61,
+0x18,
+0x08,
+0xb9,
+0x80,
+0x82,
+0x30,
+0x9d,
+0x0b,
+0xc0,
+0xcc,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x02,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x52,
+0x48,
+0xa3,
+0xc0,
+0xcf,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x52,
+0x08,
+0xa3,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x66,
+0x00,
+0x00,
+0xb7,
+0xe0,
+0x88,
+0x22,
+0x08,
+0x83,
+0xa4,
+0x88,
+0x2a,
+0x5a,
+0x00,
+0x68,
+0x86,
+0xb4,
+0x88,
+0x81,
+0x60,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x40,
+0x00,
+0x00,
+0x60,
+0x89,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcf,
+0x25,
+0x66,
+0x00,
+0x00,
+0xb7,
+0xe0,
+0x5c,
+0x09,
+0x68,
+0x82,
+0x20,
+0x88,
+0x32,
+0x59,
+0x46,
+0xab,
+0x52,
+0xca,
+0xc0,
+0x81,
+0x24,
+0x86,
+0xb4,
+0x84,
+0x20,
+0x9c,
+0xa2,
+0x40,
+0x85,
+0xc0,
+0x89,
+0x14,
+0x02,
+0xd3,
+0x81,
+0x0c,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x98,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x03,
+0x84,
+0x08,
+0xb5,
+0x40,
+0xed,
+0x84,
+0x00,
+0xb6,
+0xc0,
+0x00,
+0x11,
+0x80,
+0x82,
+0x81,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x43,
+0x25,
+0x8a,
+0x8b,
+0xc0,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x7a,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x7a,
+0x38,
+0x13,
+0x42,
+0x59,
+0x28,
+0xbc,
+0x15,
+0x06,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x95,
+0xc0,
+0xb9,
+0x98,
+0xe8,
+0x26,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x85,
+0x24,
+0x76,
+0xac,
+0x02,
+0x06,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x06,
+0xc4,
+0x00,
+0x25,
+0x07,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0xa4,
+0x86,
+0xc4,
+0x00,
+0x18,
+0xa4,
+0x86,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x94,
+0x21,
+0xff,
+0x94,
+0x24,
+0x28,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x88,
+0x20,
+0x68,
+0x00,
+0x00,
+0x91,
+0x21,
+0x66,
+0x00,
+0x01,
+0x59,
+0xc0,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x32,
+0x06,
+0x8b,
+0xc1,
+0x61,
+0x68,
+0x20,
+0x01,
+0x00,
+0x21,
+0x68,
+0x20,
+0x00,
+0xad,
+0x20,
+0xa0,
+0xc2,
+0x28,
+0x81,
+0x62,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x40,
+0x00,
+0x00,
+0x81,
+0x22,
+0x68,
+0x20,
+0x00,
+0xc5,
+0x20,
+0x68,
+0x20,
+0x01,
+0x00,
+0x21,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x38,
+0x09,
+0x40,
+0x00,
+0x03,
+0xc2,
+0x0f,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x49,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x38,
+0x19,
+0xc2,
+0x89,
+0x2d,
+0x30,
+0x12,
+0x8b,
+0xc1,
+0x81,
+0x68,
+0x20,
+0x00,
+0xad,
+0x20,
+0x68,
+0x20,
+0x01,
+0x01,
+0x21,
+0x68,
+0x20,
+0x00,
+0xfe,
+0x22,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x68,
+0x20,
+0x00,
+0xfe,
+0x22,
+0x68,
+0x20,
+0x00,
+0xc5,
+0x20,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x48,
+0x40,
+0x00,
+0x02,
+0x10,
+0x61,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x09,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x40,
+0x55,
+0x3f,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x49,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x88,
+0x1b,
+0x6b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x60,
+0xab,
+0xfd,
+0x06,
+0x82,
+0x00,
+0x1d,
+0xa2,
+0xc6,
+0x82,
+0x00,
+0x1d,
+0xb2,
+0xd6,
+0x82,
+0x00,
+0x1d,
+0xc2,
+0x26,
+0x82,
+0x00,
+0x1f,
+0x32,
+0x38,
+0x82,
+0x76,
+0x68,
+0x00,
+0x01,
+0x44,
+0x20,
+0x68,
+0x20,
+0x01,
+0x8f,
+0x24,
+0x68,
+0x00,
+0x01,
+0x52,
+0x21,
+0x68,
+0x20,
+0x01,
+0xd8,
+0x25,
+0x88,
+0x06,
+0xc8,
+0x80,
+0xed,
+0x88,
+0x16,
+0x28,
+0x81,
+0xe3,
+0x66,
+0x00,
+0x00,
+0x99,
+0x48,
+0x6c,
+0x00,
+0x02,
+0x96,
+0x09,
+0x68,
+0x00,
+0x02,
+0x3b,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x68,
+0x00,
+0x01,
+0x44,
+0x24,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0x86,
+0x34,
+0x8a,
+0x80,
+0x30,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x09,
+0x58,
+0x0b,
+0x02,
+0xbf,
+0xe0,
+0xbc,
+0x7b,
+0x88,
+0x80,
+0x76,
+0x68,
+0x20,
+0x01,
+0xb1,
+0x24,
+0x68,
+0x00,
+0x01,
+0x9c,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x6c,
+0x00,
+0x01,
+0x16,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x64,
+0x48,
+0x68,
+0x20,
+0x01,
+0xb1,
+0x24,
+0x68,
+0x00,
+0x01,
+0xa0,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x6c,
+0x00,
+0x01,
+0x28,
+0x09,
+0x68,
+0x20,
+0x01,
+0x1b,
+0x20,
+0x68,
+0x20,
+0x01,
+0xf5,
+0x24,
+0x5c,
+0x81,
+0x02,
+0x05,
+0xc1,
+0x84,
+0x00,
+0x98,
+0x48,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x08,
+0x01,
+0x6c,
+0x00,
+0x01,
+0x1a,
+0x0a,
+0x30,
+0x02,
+0x8b,
+0xc4,
+0x3d,
+0x6c,
+0x00,
+0x03,
+0x72,
+0x48,
+0x5c,
+0x83,
+0x0a,
+0x20,
+0x01,
+0x80,
+0xa0,
+0x88,
+0x0a,
+0x8b,
+0x49,
+0xb8,
+0xe2,
+0x05,
+0x60,
+0x22,
+0xfd,
+0x42,
+0xa8,
+0x64,
+0x28,
+0x11,
+0x22,
+0x28,
+0x54,
+0x5b,
+0x40,
+0x00,
+0x48,
+0x48,
+0xa0,
+0x8c,
+0x18,
+0x0a,
+0x08,
+0x84,
+0x80,
+0x95,
+0x40,
+0xb1,
+0x18,
+0x00,
+0x05,
+0x17,
+0xeb,
+0xa0,
+0xc8,
+0x12,
+0xa8,
+0x7b,
+0x54,
+0x06,
+0x98,
+0x40,
+0x02,
+0x6c,
+0x40,
+0x02,
+0x36,
+0x7a,
+0x30,
+0x08,
+0x05,
+0x14,
+0x2f,
+0xbc,
+0x21,
+0xb8,
+0x08,
+0xcb,
+0x2e,
+0x16,
+0x43,
+0x69,
+0x00,
+0x98,
+0x00,
+0x83,
+0x00,
+0xa0,
+0xbc,
+0x1b,
+0x38,
+0x80,
+0xca,
+0x88,
+0x16,
+0x18,
+0x81,
+0xd1,
+0x66,
+0x00,
+0x02,
+0x4d,
+0x28,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x08,
+0x6e,
+0x00,
+0x03,
+0x4c,
+0x12,
+0x66,
+0x00,
+0x02,
+0x52,
+0xa8,
+0x55,
+0x00,
+0x0b,
+0x80,
+0x00,
+0x66,
+0x00,
+0x02,
+0x4f,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x09,
+0x57,
+0x4b,
+0x20,
+0x81,
+0x21,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x48,
+0x5c,
+0x81,
+0x02,
+0xc0,
+0x61,
+0x88,
+0x18,
+0x18,
+0x80,
+0x8a,
+0x68,
+0x20,
+0x01,
+0xf5,
+0x24,
+0x80,
+0xa5,
+0x18,
+0x08,
+0xd1,
+0x42,
+0x01,
+0x78,
+0x0a,
+0x4a,
+0x84,
+0x84,
+0xa3,
+0x90,
+0x61,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3e,
+0xa0,
+0x82,
+0xe8,
+0x64,
+0x82,
+0x24,
+0x80,
+0x00,
+0x00,
+0x86,
+0x00,
+0x85,
+0x78,
+0x32,
+0x06,
+0x18,
+0x95,
+0x74,
+0xd6,
+0x02,
+0x0c,
+0x88,
+0x22,
+0x48,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x23,
+0x60,
+0x85,
+0x50,
+0x32,
+0x06,
+0x00,
+0x92,
+0xf1,
+0xad,
+0x6c,
+0x40,
+0x02,
+0x36,
+0x48,
+0x40,
+0x00,
+0x00,
+0x60,
+0x49,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x1b,
+0x92,
+0x0b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x20,
+0xa2,
+0x28,
+0x46,
+0x40,
+0x00,
+0x13,
+0x26,
+0xfa,
+0x01,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x65,
+0x88,
+0x0f,
+0x66,
+0x60,
+0x00,
+0x13,
+0x34,
+0x8a,
+0x20,
+0x84,
+0x88,
+0x02,
+0x08,
+0x81,
+0x24,
+0x94,
+0x02,
+0x55,
+0x00,
+0xb0,
+0x08,
+0x0b,
+0x68,
+0x60,
+0x09,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x08,
+0x44,
+0x08,
+0x02,
+0x80,
+0x10,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0xab,
+0xfd,
+0x08,
+0x81,
+0xe4,
+0x88,
+0x04,
+0x98,
+0x80,
+0xe5,
+0x88,
+0x16,
+0x18,
+0x82,
+0x60,
+0x88,
+0x2f,
+0x66,
+0x82,
+0x00,
+0x0d,
+0x02,
+0x46,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x08,
+0x00,
+0x88,
+0x22,
+0x06,
+0xc4,
+0x00,
+0x1a,
+0xe0,
+0x95,
+0xd0,
+0xa3,
+0x04,
+0x00,
+0x95,
+0x70,
+0xb0,
+0x04,
+0x1c,
+0x85,
+0x18,
+0x5b,
+0x18,
+0x00,
+0x86,
+0x82,
+0x00,
+0x12,
+0xd2,
+0x15,
+0xb4,
+0x80,
+0x18,
+0x2a,
+0x89,
+0x80,
+0x0a,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x00,
+0x58,
+0x41,
+0x80,
+0x81,
+0xa4,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0xa2,
+0x08,
+0x36,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x28,
+0x84,
+0x21,
+0xa1,
+0x86,
+0x44,
+0x20,
+0x25,
+0x21,
+0x8a,
+0x29,
+0x48,
+0x3f,
+0x38,
+0x12,
+0x92,
+0x58,
+0x70,
+0xbc,
+0x77,
+0x08,
+0x83,
+0x26,
+0x88,
+0x0a,
+0x79,
+0x70,
+0x2c,
+0x59,
+0x03,
+0x00,
+0x81,
+0x25,
+0xbc,
+0x11,
+0x98,
+0x78,
+0x7a,
+0x00,
+0x00,
+0x08,
+0x68,
+0x88,
+0x30,
+0x16,
+0x0b,
+0xc0,
+0x7b,
+0x84,
+0x38,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x09,
+0x49,
+0x8e,
+0x88,
+0x86,
+0x8c,
+0x99,
+0x70,
+0x64,
+0xbc,
+0x05,
+0x73,
+0x01,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0x86,
+0x8c,
+0x99,
+0x70,
+0x64,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3d,
+0x20,
+0x88,
+0x59,
+0x89,
+0x30,
+0x12,
+0x8b,
+0xc2,
+0x48,
+0x39,
+0x02,
+0x06,
+0x82,
+0x00,
+0x1e,
+0xe2,
+0x75,
+0xc8,
+0x20,
+0x85,
+0xa8,
+0x95,
+0xc8,
+0x51,
+0x03,
+0x80,
+0xa5,
+0x70,
+0xd4,
+0x21,
+0x00,
+0x65,
+0x90,
+0x18,
+0x01,
+0x2d,
+0x00,
+0x00,
+0x00,
+0x83,
+0xb0,
+0x98,
+0x50,
+0x0a,
+0x57,
+0x0b,
+0xb0,
+0x78,
+0x09,
+0x85,
+0x04,
+0xab,
+0xc0,
+0x6c,
+0x85,
+0xa8,
+0xa3,
+0x01,
+0x70,
+0xbc,
+0x08,
+0x28,
+0x59,
+0xc8,
+0x86,
+0x14,
+0x9b,
+0xc0,
+0x57,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x34,
+0x85,
+0x9c,
+0x88,
+0x61,
+0x49,
+0x00,
+0x00,
+0x08,
+0x50,
+0x09,
+0x57,
+0x0a,
+0x80,
+0x32,
+0x09,
+0x57,
+0x0a,
+0x00,
+0x83,
+0xa4,
+0xbc,
+0x03,
+0xf8,
+0x70,
+0x40,
+0x88,
+0x3a,
+0x40,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x60,
+0x09,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0xb3,
+0x6c,
+0x40,
+0x02,
+0x30,
+0x08,
+0x28,
+0x12,
+0xd3,
+0x08,
+0xa8,
+0xbc,
+0x03,
+0x4b,
+0xc0,
+0x6f,
+0x5c,
+0x0b,
+0xe8,
+0x60,
+0x49,
+0xbc,
+0x03,
+0xf5,
+0xc0,
+0xbe,
+0x86,
+0x05,
+0x23,
+0x81,
+0x7d,
+0x25,
+0x97,
+0x8b,
+0xc2,
+0x01,
+0x38,
+0x16,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x0f,
+0x93,
+0x81,
+0x75,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x49,
+0x98,
+0xe8,
+0x98,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x38,
+0x17,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x13,
+0x08,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x03,
+0x94,
+0x87,
+0x5b,
+0xc0,
+0xe7,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x49,
+0x38,
+0x16,
+0x58,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x62,
+0x38,
+0x17,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x05,
+0x08,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x25,
+0x24,
+0x17,
+0x89,
+0x48,
+0x70,
+0xbc,
+0x42,
+0xf8,
+0x80,
+0xa2,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x6c,
+0x88,
+0x12,
+0x55,
+0x70,
+0x14,
+0x08,
+0x32,
+0x3b,
+0xc0,
+0x5f,
+0x40,
+0x00,
+0x00,
+0x68,
+0xc0,
+0x54,
+0x0a,
+0x00,
+0x83,
+0x23,
+0x86,
+0x8c,
+0x0b,
+0x00,
+0x0d,
+0x88,
+0x0a,
+0x29,
+0x58,
+0x65,
+0x00,
+0x00,
+0x08,
+0x50,
+0x09,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0x23,
+0x2a,
+0x06,
+0xd8,
+0x50,
+0x49,
+0x68,
+0x20,
+0x01,
+0xeb,
+0x26,
+0x5c,
+0x81,
+0x0b,
+0x01,
+0x7d,
+0x52,
+0xcb,
+0xc0,
+0x30,
+0x88,
+0x5c,
+0x82,
+0x00,
+0x70,
+0x00,
+0x82,
+0x0c,
+0x88,
+0x60,
+0x50,
+0x68,
+0x20,
+0x01,
+0x19,
+0x23,
+0x87,
+0x08,
+0xb8,
+0x83,
+0xa7,
+0x81,
+0xa0,
+0x04,
+0x20,
+0x8c,
+0x06,
+0x0c,
+0xb5,
+0xc0,
+0x02,
+0x07,
+0x85,
+0x02,
+0x41,
+0x64,
+0x5d,
+0x48,
+0x20,
+0x43,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x6c,
+0x5c,
+0x0b,
+0xa9,
+0x48,
+0x74,
+0x52,
+0x4b,
+0x03,
+0xc0,
+0x7f,
+0x5c,
+0x09,
+0x29,
+0x48,
+0x70,
+0x52,
+0x0b,
+0x03,
+0xc0,
+0x3f,
+0x5c,
+0x09,
+0x29,
+0x48,
+0x70,
+0x38,
+0x12,
+0x52,
+0x59,
+0x70,
+0xbc,
+0x08,
+0x88,
+0x58,
+0x52,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2a,
+0xe0,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x04,
+0x96,
+0xc4,
+0x00,
+0x18,
+0x04,
+0x99,
+0x88,
+0x09,
+0x68,
+0x00,
+0x00,
+0x84,
+0x08,
+0x58,
+0x09,
+0x40,
+0x68,
+0x89,
+0x84,
+0x1c,
+0x9b,
+0xc1,
+0x48,
+0x85,
+0x00,
+0x99,
+0x88,
+0x08,
+0x68,
+0x00,
+0x00,
+0x8d,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x98,
+0xea,
+0x16,
+0x60,
+0x00,
+0x0b,
+0x8a,
+0x08,
+0x82,
+0x20,
+0xbc,
+0x10,
+0xf8,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xb5,
+0x21,
+0x66,
+0x00,
+0x00,
+0xb8,
+0xa0,
+0x88,
+0x22,
+0x0b,
+0xc0,
+0x9f,
+0x40,
+0x00,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xae,
+0x21,
+0x66,
+0x00,
+0x00,
+0xb8,
+0xa0,
+0x88,
+0x22,
+0x00,
+0x00,
+0x00,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x56,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x51,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x38,
+0x12,
+0xc2,
+0x59,
+0x28,
+0xbc,
+0x05,
+0x08,
+0x84,
+0xa1,
+0x40,
+0x00,
+0x00,
+0x40,
+0x09,
+0x42,
+0x29,
+0xb8,
+0x48,
+0x49,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0xb0,
+0x84,
+0x18,
+0x88,
+0x40,
+0x0a,
+0x57,
+0x0d,
+0x00,
+0x84,
+0xa1,
+0x98,
+0x00,
+0xa3,
+0x69,
+0x80,
+0x98,
+0x00,
+0xa3,
+0x01,
+0x70,
+0xbc,
+0x03,
+0x58,
+0x48,
+0x48,
+0xbc,
+0x01,
+0x78,
+0x84,
+0xa1,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x09,
+0x84,
+0x80,
+0x85,
+0x40,
+0xb0,
+0x04,
+0x00,
+0xa9,
+0x80,
+0x00,
+0x30,
+0x03,
+0x0b,
+0xc0,
+0x2d,
+0x88,
+0x0c,
+0x88,
+0x40,
+0x50,
+0x2e,
+0x16,
+0x08,
+0x40,
+0x09,
+0x98,
+0x00,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x01,
+0x38,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x88,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x56,
+0x08,
+0x68,
+0x00,
+0x2b,
+0xf2,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0xe4,
+0x68,
+0x00,
+0x27,
+0x10,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0x84,
+0x68,
+0x00,
+0x18,
+0x6a,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0x24,
+0xbc,
+0x08,
+0xf2,
+0x28,
+0x6d,
+0xbc,
+0x06,
+0xf2,
+0x28,
+0xad,
+0xbc,
+0x04,
+0xf2,
+0x28,
+0xed,
+0xbc,
+0x02,
+0xf2,
+0x29,
+0x2d,
+0x22,
+0x96,
+0xd8,
+0x84,
+0xa1,
+0x88,
+0x22,
+0x08,
+0x48,
+0x08,
+0x84,
+0x00,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x03,
+0x38,
+0x80,
+0x88,
+0x2e,
+0x16,
+0x08,
+0x48,
+0x40,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x84,
+0x80,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x04,
+0xd8,
+0x80,
+0x88,
+0x28,
+0x16,
+0x08,
+0x48,
+0x40,
+0x00,
+0x00,
+0x08,
+0x82,
+0xb6,
+0x84,
+0x00,
+0x84,
+0x60,
+0xa4,
+0x08,
+0x00,
+0x95,
+0x70,
+0x94,
+0x28,
+0x03,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x09,
+0xe2,
+0x22,
+0xe4,
+0x86,
+0x00,
+0xa4,
+0x43,
+0x00,
+0x16,
+0x9a,
+0x55,
+0x00,
+0xa0,
+0x08,
+0x02,
+0x56,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xd5,
+0x2c,
+0x94,
+0x06,
+0x80,
+0x94,
+0x20,
+0x24,
+0x18,
+0x00,
+0x80,
+0x81,
+0x00,
+0xba,
+0x14,
+0x89,
+0x80,
+0x08,
+0x00,
+0x00,
+0x08,
+0x40,
+0x89,
+0x46,
+0x0a,
+0x40,
+0x68,
+0x88,
+0x08,
+0x48,
+0x09,
+0x80,
+0x08,
+0xab,
+0xfd,
+0x0a,
+0x21,
+0x64,
+0xa0,
+0x88,
+0x18,
+0x80,
+0x64,
+0xa2,
+0x06,
+0x48,
+0x80,
+0xe1,
+0xa0,
+0x84,
+0x18,
+0x81,
+0x64,
+0x88,
+0x1c,
+0x98,
+0x82,
+0x60,
+0x88,
+0x2f,
+0x66,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x08,
+0x00,
+0x88,
+0x12,
+0x08,
+0x82,
+0x21,
+0xa0,
+0x06,
+0x08,
+0x40,
+0x0a,
+0x88,
+0x18,
+0x94,
+0x43,
+0x00,
+0x04,
+0xa4,
+0x8a,
+0x06,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb5,
+0xe8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x09,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x59,
+0x03,
+0x40,
+0x80,
+0xa4,
+0xbc,
+0x15,
+0x8a,
+0x20,
+0x20,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x5c,
+0x09,
+0x20,
+0x80,
+0xc8,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xc8,
+0x88,
+0x02,
+0x46,
+0xc4,
+0x00,
+0x2a,
+0xc0,
+0x96,
+0x60,
+0x00,
+0x15,
+0x82,
+0x88,
+0x60,
+0x8a,
+0x55,
+0x01,
+0x28,
+0x80,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0xc9,
+0xbc,
+0x0f,
+0xf8,
+0x80,
+0x88,
+0xbc,
+0x0d,
+0xf8,
+0x80,
+0x88,
+0x86,
+0x08,
+0x93,
+0x69,
+0x40,
+0x98,
+0x00,
+0x96,
+0xc4,
+0x00,
+0x21,
+0x00,
+0xa3,
+0x09,
+0x70,
+0xbc,
+0x05,
+0x58,
+0x80,
+0x20,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x09,
+0x84,
+0x0c,
+0x90,
+0x00,
+0x00,
+0x88,
+0x2b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2c,
+0x38,
+0x12,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x09,
+0x08,
+0x63,
+0x88,
+0x6c,
+0x40,
+0x03,
+0xd2,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x2e,
+0x08,
+0x08,
+0x40,
+0x09,
+0x80,
+0x09,
+0xba,
+0x14,
+0x89,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x80,
+0xa2,
+0x16,
+0x48,
+0x20,
+0x0a,
+0x84,
+0x00,
+0x84,
+0x41,
+0x00,
+0x02,
+0x00,
+0xa8,
+0x40,
+0x88,
+0x44,
+0x14,
+0x00,
+0x60,
+0x08,
+0xba,
+0x14,
+0x84,
+0x42,
+0x40,
+0x04,
+0x04,
+0x99,
+0x80,
+0x08,
+0xa2,
+0x24,
+0x48,
+0x60,
+0x08,
+0x44,
+0x20,
+0x00,
+0x42,
+0xc9,
+0x98,
+0x00,
+0x98,
+0x60,
+0x88,
+0x44,
+0x20,
+0x01,
+0x68,
+0xa5,
+0xba,
+0x14,
+0x82,
+0x01,
+0x40,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x5c,
+0x83,
+0x08,
+0x43,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x0a,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x00,
+0x58,
+0x0c,
+0x02,
+0xbf,
+0xd0,
+0x80,
+0x08,
+0xba,
+0x08,
+0x42,
+0x84,
+0x00,
+0x19,
+0x88,
+0x42,
+0x88,
+0x0c,
+0x88,
+0x81,
+0x49,
+0x42,
+0x13,
+0x40,
+0x80,
+0x4a,
+0x88,
+0x1c,
+0xb3,
+0x20,
+0x10,
+0xbc,
+0x21,
+0x06,
+0x82,
+0x00,
+0x11,
+0xb2,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x5c,
+0x38,
+0x40,
+0x08,
+0x85,
+0x80,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x10,
+0xd8,
+0x4a,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x48,
+0x8a,
+0x84,
+0x80,
+0x32,
+0x80,
+0xf4,
+0x22,
+0xfe,
+0x75,
+0x54,
+0x3c,
+0x98,
+0x44,
+0xb1,
+0x51,
+0x73,
+0x51,
+0x42,
+0x60,
+0x10,
+0x53,
+0x81,
+0x2c,
+0x84,
+0x20,
+0x1f,
+0x85,
+0x04,
+0x95,
+0x50,
+0x1c,
+0x85,
+0x0c,
+0x93,
+0x90,
+0x20,
+0x00,
+0x00,
+0x08,
+0x48,
+0x88,
+0x57,
+0x4b,
+0x30,
+0x48,
+0x08,
+0x57,
+0x8b,
+0x20,
+0x48,
+0xca,
+0xbc,
+0x04,
+0xf8,
+0x48,
+0x48,
+0xbc,
+0x02,
+0xf3,
+0x90,
+0x20,
+0x39,
+0x02,
+0x03,
+0x81,
+0x25,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2c,
+0x25,
+0x96,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xa0,
+0xab,
+0xc0,
+0x59,
+0x55,
+0x01,
+0xa0,
+0x82,
+0x4a,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x24,
+0xe0,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x56,
+0x09,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x88,
+0x88,
+0x10,
+0xab,
+0x1f,
+0x47,
+0x30,
+0x1f,
+0x0b,
+0xc1,
+0x2c,
+0x98,
+0xe8,
+0x88,
+0x82,
+0x08,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xe7,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xae,
+0x38,
+0x13,
+0x72,
+0x59,
+0xf0,
+0xbc,
+0x09,
+0x16,
+0xc4,
+0x00,
+0x15,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x2b,
+0xc0,
+0xb3,
+0x01,
+0xf0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x23,
+0x6c,
+0x40,
+0x02,
+0x4e,
+0x08,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x26,
+0x59,
+0x03,
+0x80,
+0x81,
+0x8a,
+0x28,
+0x13,
+0x35,
+0x70,
+0x99,
+0x98,
+0x0c,
+0x8b,
+0xc0,
+0xf8,
+0x98,
+0x0c,
+0xb6,
+0xc4,
+0x00,
+0x25,
+0x00,
+0x33,
+0x20,
+0x18,
+0xbc,
+0x0a,
+0x13,
+0x01,
+0x88,
+0xbc,
+0x06,
+0x45,
+0x80,
+0xc4,
+0x08,
+0x18,
+0x8b,
+0xc0,
+0xfb,
+0x88,
+0x00,
+0x19,
+0x82,
+0xc8,
+0xbc,
+0x0c,
+0x7b,
+0xc0,
+0xbf,
+0x88,
+0x00,
+0x18,
+0x80,
+0x81,
+0x32,
+0x00,
+0x8b,
+0xc0,
+0x64,
+0x32,
+0x00,
+0x8b,
+0xc0,
+0x5b,
+0x55,
+0x01,
+0xa0,
+0x80,
+0x01,
+0x98,
+0x2c,
+0x8b,
+0xc0,
+0x17,
+0x88,
+0x00,
+0x13,
+0x00,
+0x40,
+0xbc,
+0x44,
+0x03,
+0x20,
+0x10,
+0xbc,
+0x42,
+0x06,
+0xc0,
+0x00,
+0x36,
+0x80,
+0xa3,
+0x20,
+0x30,
+0xbc,
+0x05,
+0x98,
+0x82,
+0x0a,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc3,
+0x90,
+0x30,
+0x17,
+0x0b,
+0xc3,
+0x70,
+0x68,
+0x20,
+0x01,
+0x0b,
+0x20,
+0x84,
+0x90,
+0xa8,
+0x00,
+0x00,
+0x30,
+0x03,
+0x0b,
+0xc0,
+0x22,
+0xbc,
+0x04,
+0xf8,
+0x4a,
+0xfa,
+0x84,
+0xa8,
+0xa2,
+0xa0,
+0x76,
+0x84,
+0xac,
+0xa0,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa6,
+0xc0,
+0x00,
+0x36,
+0x60,
+0x03,
+0x01,
+0x80,
+0xbc,
+0x04,
+0x56,
+0xc0,
+0x00,
+0x37,
+0x40,
+0x03,
+0x01,
+0x80,
+0xbc,
+0x03,
+0x4b,
+0xc2,
+0x1f,
+0x40,
+0x00,
+0x00,
+0x4a,
+0x08,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x5c,
+0x82,
+0x08,
+0x08,
+0x7a,
+0x80,
+0x07,
+0xaa,
+0x08,
+0x02,
+0x80,
+0x87,
+0xaa,
+0x00,
+0x04,
+0x80,
+0x07,
+0xa8,
+0x50,
+0xfa,
+0x86,
+0x0f,
+0xa8,
+0x48,
+0xfa,
+0x68,
+0x00,
+0x01,
+0xb9,
+0x23,
+0x84,
+0x0f,
+0xa6,
+0x80,
+0x00,
+0x1b,
+0x22,
+0x16,
+0x82,
+0x00,
+0x1f,
+0x12,
+0x08,
+0x08,
+0x80,
+0x85,
+0x80,
+0xa6,
+0xc4,
+0x00,
+0x20,
+0xa4,
+0x96,
+0xc0,
+0x00,
+0x14,
+0xc7,
+0xa8,
+0x48,
+0x7a,
+0x85,
+0x97,
+0xa8,
+0x40,
+0x50,
+0x84,
+0x0c,
+0xab,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x85,
+0x00,
+0x88,
+0x40,
+0x09,
+0x57,
+0x8b,
+0x22,
+0xbf,
+0xd0,
+0x85,
+0x04,
+0x88,
+0x80,
+0x63,
+0x88,
+0x0e,
+0x28,
+0x81,
+0x61,
+0x88,
+0x1e,
+0x08,
+0x82,
+0x76,
+0x00,
+0x00,
+0x08,
+0x40,
+0x09,
+0x36,
+0x14,
+0x52,
+0xe9,
+0x64,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x85,
+0x80,
+0x88,
+0x48,
+0x09,
+0x2f,
+0x16,
+0x48,
+0x58,
+0x48,
+0x00,
+0x00,
+0x08,
+0x48,
+0x09,
+0x36,
+0x14,
+0x52,
+0xe9,
+0x64,
+0x44,
+0x00,
+0x00,
+0x58,
+0x48,
+0x98,
+0x00,
+0x98,
+0x50,
+0x08,
+0x44,
+0x00,
+0x00,
+0x48,
+0x08,
+0x44,
+0x01,
+0x00,
+0x40,
+0x08,
+0x44,
+0x01,
+0x01,
+0xa0,
+0x00,
+0x44,
+0x40,
+0x01,
+0x80,
+0x88,
+0x08,
+0x18,
+0x06,
+0x60,
+0x00,
+0x13,
+0x52,
+0x89,
+0x81,
+0x09,
+0x88,
+0x12,
+0x08,
+0x81,
+0xa1,
+0x84,
+0x00,
+0x98,
+0x48,
+0x0a,
+0x44,
+0x48,
+0x00,
+0x80,
+0xa0,
+0x88,
+0x02,
+0x19,
+0x80,
+0x0b,
+0x84,
+0x00,
+0x94,
+0x46,
+0x80,
+0x04,
+0x80,
+0xa4,
+0x47,
+0x10,
+0x18,
+0x00,
+0xb4,
+0x46,
+0x00,
+0x18,
+0x28,
+0xb5,
+0x11,
+0xe0,
+0x18,
+0x08,
+0xa4,
+0x44,
+0x10,
+0x18,
+0x00,
+0x83,
+0x20,
+0x20,
+0x51,
+0x1e,
+0x83,
+0xc0,
+0x29,
+0x98,
+0x00,
+0xa9,
+0x82,
+0x48,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x11,
+0x98,
+0x2c,
+0xa3,
+0x69,
+0x40,
+0x5b,
+0x48,
+0x01,
+0x80,
+0x09,
+0x98,
+0x00,
+0x03,
+0x00,
+0x28,
+0xbc,
+0x06,
+0x43,
+0x69,
+0xc0,
+0x5b,
+0x4c,
+0x01,
+0x80,
+0x09,
+0x98,
+0x00,
+0xb3,
+0x01,
+0xe8,
+0xbc,
+0x04,
+0x58,
+0x40,
+0x48,
+0x42,
+0x01,
+0x7b,
+0x00,
+0x0c,
+0x84,
+0x84,
+0xa9,
+0x8e,
+0x88,
+0x00,
+0x00,
+0x08,
+0x82,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x46,
+0x08,
+0x8a,
+0xbf,
+0xe0,
+0x51,
+0x61,
+0xb0,
+0x80,
+0x49,
+0x51,
+0x61,
+0xe0,
+0x80,
+0xf6,
+0x1a,
+0x0a,
+0x29,
+0x83,
+0x09,
+0x98,
+0x38,
+0x82,
+0x81,
+0x2d,
+0x88,
+0x14,
+0x96,
+0x60,
+0x00,
+0x13,
+0x52,
+0x8b,
+0xa1,
+0x01,
+0x88,
+0x10,
+0x94,
+0x40,
+0x80,
+0x08,
+0x00,
+0x99,
+0x80,
+0x08,
+0x23,
+0x42,
+0x43,
+0x01,
+0x60,
+0xbc,
+0x02,
+0x5b,
+0xc0,
+0x2f,
+0x2e,
+0x16,
+0x49,
+0x8e,
+0x88,
+0x88,
+0x0b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x00,
+0x01,
+0x3a,
+0x22,
+0x55,
+0x03,
+0x20,
+0x00,
+0x7a,
+0x5d,
+0x08,
+0x20,
+0x08,
+0x7a,
+0x32,
+0x2a,
+0x0b,
+0xff,
+0xaa,
+0x81,
+0x07,
+0xab,
+0xa1,
+0x40,
+0x5c,
+0x81,
+0x00,
+0x40,
+0x08,
+0x82,
+0x00,
+0xa4,
+0x41,
+0x00,
+0x04,
+0x80,
+0xa8,
+0x20,
+0x08,
+0x44,
+0x44,
+0x00,
+0x60,
+0x08,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x0a,
+0x08,
+0x80,
+0x09,
+0x80,
+0x08,
+0x68,
+0x00,
+0x00,
+0x6c,
+0x20,
+0x68,
+0x20,
+0x00,
+0x24,
+0x24,
+0x5c,
+0x81,
+0x02,
+0xc0,
+0x41,
+0x68,
+0x20,
+0x00,
+0x4b,
+0x21,
+0x5c,
+0x08,
+0xad,
+0x00,
+0x81,
+0x44,
+0x10,
+0x00,
+0x08,
+0x88,
+0x84,
+0x00,
+0xb4,
+0x46,
+0x08,
+0x2b,
+0xf7,
+0x06,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xe6,
+0xe4,
+0x00,
+0x11,
+0x22,
+0x75,
+0x2c,
+0xb8,
+0x08,
+0x16,
+0x18,
+0x81,
+0xe4,
+0x88,
+0x27,
+0x68,
+0x82,
+0xd5,
+0x90,
+0x45,
+0x84,
+0x22,
+0x3c,
+0x88,
+0x55,
+0x49,
+0x03,
+0x59,
+0x68,
+0x20,
+0x00,
+0x85,
+0x24,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x0a,
+0x66,
+0x00,
+0x00,
+0xe4,
+0xc8,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x09,
+0x68,
+0x20,
+0x00,
+0x85,
+0x20,
+0x5c,
+0x82,
+0x40,
+0x88,
+0x48,
+0xa0,
+0x08,
+0xc6,
+0xc0,
+0x00,
+0x12,
+0xa0,
+0x96,
+0xc0,
+0x00,
+0x11,
+0x80,
+0xa6,
+0x60,
+0x00,
+0x0e,
+0x4c,
+0x89,
+0x62,
+0x07,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x5c,
+0x08,
+0xe8,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x90,
+0x2c,
+0x52,
+0xcb,
+0x80,
+0x85,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x88,
+0x06,
+0xcb,
+0xc1,
+0xa9,
+0x40,
+0x00,
+0x00,
+0x88,
+0x09,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x0a,
+0x44,
+0x10,
+0x00,
+0x85,
+0x89,
+0x68,
+0x20,
+0x00,
+0x98,
+0xa0,
+0x88,
+0x84,
+0x88,
+0x80,
+0x60,
+0x68,
+0x20,
+0x00,
+0x92,
+0xa4,
+0x68,
+0x20,
+0x00,
+0x95,
+0xa5,
+0x90,
+0x65,
+0x88,
+0x85,
+0xd4,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x40,
+0x00,
+0x03,
+0xc5,
+0x4f,
+0x44,
+0x10,
+0x00,
+0x88,
+0xc8,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x08,
+0x44,
+0x20,
+0x80,
+0x85,
+0x88,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x42,
+0x25,
+0x79,
+0x06,
+0x59,
+0x44,
+0x10,
+0x00,
+0x85,
+0xd5,
+0x68,
+0x00,
+0x00,
+0x6e,
+0x20,
+0x68,
+0x20,
+0x00,
+0x85,
+0x24,
+0x80,
+0x00,
+0x98,
+0x85,
+0xe0,
+0x66,
+0x00,
+0x00,
+0xe4,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x40,
+0x0a,
+0x5c,
+0x82,
+0x40,
+0x85,
+0xa0,
+0x68,
+0x20,
+0x00,
+0x85,
+0x21,
+0x88,
+0x84,
+0x8a,
+0x08,
+0x8c,
+0x84,
+0x00,
+0x96,
+0xc0,
+0x00,
+0x0d,
+0xc0,
+0xa6,
+0x60,
+0x00,
+0x0e,
+0x4c,
+0x89,
+0x62,
+0x07,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x5c,
+0x08,
+0xe8,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x90,
+0x2c,
+0x52,
+0xcb,
+0x80,
+0x85,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x88,
+0x06,
+0xcb,
+0xc1,
+0xa9,
+0x40,
+0x00,
+0x00,
+0x88,
+0x09,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x0a,
+0x44,
+0x10,
+0x00,
+0x85,
+0x89,
+0x68,
+0x20,
+0x00,
+0x98,
+0xa0,
+0x88,
+0x84,
+0x88,
+0x80,
+0x60,
+0x68,
+0x20,
+0x00,
+0x92,
+0xa4,
+0x68,
+0x20,
+0x00,
+0x95,
+0xa5,
+0x90,
+0x65,
+0x88,
+0x85,
+0xd4,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xcf,
+0x44,
+0x10,
+0x00,
+0x88,
+0xc8,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x08,
+0x44,
+0x20,
+0x80,
+0x85,
+0x88,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x44,
+0x10,
+0x01,
+0x06,
+0x59,
+0x40,
+0x00,
+0x00,
+0x85,
+0xd5,
+0x68,
+0x00,
+0x00,
+0x50,
+0x20,
+0x90,
+0x61,
+0x1a,
+0x04,
+0x21,
+0x88,
+0x59,
+0x58,
+0x85,
+0xe1,
+0x90,
+0x65,
+0x88,
+0x87,
+0x54,
+0x66,
+0x00,
+0x01,
+0x31,
+0x28,
+0x5c,
+0x00,
+0x71,
+0x80,
+0x49,
+0x90,
+0x61,
+0x06,
+0x80,
+0x00,
+0x06,
+0x42,
+0x08,
+0x87,
+0x14,
+0xa0,
+0x42,
+0x18,
+0x86,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x86,
+0x61,
+0x66,
+0x00,
+0x01,
+0x31,
+0x28,
+0x5c,
+0x00,
+0x71,
+0x80,
+0x09,
+0x5c,
+0x82,
+0x10,
+0x86,
+0x20,
+0x5c,
+0x8c,
+0x01,
+0x03,
+0x10,
+0x5c,
+0x8e,
+0x09,
+0x04,
+0x12,
+0x88,
+0x12,
+0x18,
+0x85,
+0xa5,
+0xa0,
+0x56,
+0x48,
+0x82,
+0x94,
+0x54,
+0x08,
+0x18,
+0x86,
+0x8a,
+0x88,
+0x51,
+0x65,
+0x40,
+0xc8,
+0x88,
+0x1a,
+0x28,
+0x48,
+0x08,
+0xa2,
+0xd6,
+0x08,
+0x21,
+0x0a,
+0x4d,
+0x86,
+0x80,
+0x50,
+0x08,
+0x80,
+0x10,
+0xa4,
+0xf8,
+0x48,
+0x02,
+0x04,
+0x38,
+0x00,
+0x41,
+0x82,
+0x2c,
+0x08,
+0x02,
+0xc2,
+0x88,
+0x16,
+0x46,
+0x82,
+0x00,
+0x07,
+0xe2,
+0xc8,
+0x81,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x1c,
+0x21,
+0x68,
+0x20,
+0x00,
+0x24,
+0x24,
+0x68,
+0x20,
+0x00,
+0x72,
+0x25,
+0x40,
+0x00,
+0x00,
+0x80,
+0x6c,
+0x68,
+0x00,
+0x01,
+0x27,
+0x22,
+0x66,
+0x00,
+0x00,
+0xe0,
+0x40,
+0x68,
+0x20,
+0x00,
+0x7f,
+0xac,
+0x88,
+0x84,
+0x86,
+0x80,
+0x00,
+0x03,
+0x02,
+0x18,
+0x81,
+0x20,
+0x68,
+0x20,
+0x00,
+0x4b,
+0x24,
+0x68,
+0x20,
+0x00,
+0x78,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x80,
+0x00,
+0x12,
+0xa2,
+0x26,
+0x60,
+0x00,
+0x0e,
+0x04,
+0x08,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x83,
+0x20,
+0x68,
+0x20,
+0x00,
+0x84,
+0x21,
+0xa4,
+0x22,
+0x36,
+0x60,
+0x00,
+0x0c,
+0x52,
+0x8a,
+0x42,
+0x02,
+0x5c,
+0x8d,
+0x00,
+0x81,
+0xa0,
+0x5c,
+0x08,
+0x71,
+0x8e,
+0x88,
+0xa0,
+0x26,
+0x09,
+0x42,
+0x0f,
+0x25,
+0x9b,
+0x8b,
+0xc0,
+0x39,
+0x55,
+0x01,
+0x28,
+0x81,
+0x21,
+0x88,
+0x80,
+0x98,
+0x40,
+0x0b,
+0x54,
+0x0f,
+0x42,
+0x0a,
+0x61,
+0x94,
+0xa0,
+0xd2,
+0x59,
+0xa8,
+0xbc,
+0x03,
+0x98,
+0x40,
+0xc0,
+0x00,
+0x00,
+0x08,
+0x88,
+0x88,
+0x00,
+0x00,
+0x08,
+0x48,
+0x0a,
+0x54,
+0x0d,
+0x00,
+0x82,
+0x36,
+0xba,
+0x14,
+0x88,
+0x48,
+0xc0,
+0x40,
+0x00,
+0x02,
+0x80,
+0x90,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xc0,
+0x5c,
+0x8f,
+0x08,
+0x81,
+0x60,
+0xa0,
+0x04,
+0x3a,
+0x08,
+0x00,
+0x81,
+0xa0,
+0x9a,
+0x1a,
+0x21,
+0x85,
+0x80,
+0x85,
+0x70,
+0x94,
+0x00,
+0xa8,
+0x95,
+0x70,
+0xa0,
+0x08,
+0x1e,
+0x1a,
+0x10,
+0x01,
+0x81,
+0x00,
+0x99,
+0x80,
+0x08,
+0x54,
+0x0b,
+0x02,
+0x28,
+0x26,
+0x98,
+0x00,
+0x98,
+0x49,
+0x49,
+0xa2,
+0x80,
+0x18,
+0x84,
+0x25,
+0x85,
+0x04,
+0x8a,
+0x30,
+0x23,
+0xa3,
+0x00,
+0x2a,
+0x18,
+0x26,
+0xa2,
+0x06,
+0x48,
+0x80,
+0x66,
+0x88,
+0x26,
+0x68,
+0x82,
+0xe4,
+0x88,
+0x36,
+0x08,
+0x83,
+0xf6,
+0x66,
+0x00,
+0x00,
+0xec,
+0x60,
+0x5c,
+0x8f,
+0x00,
+0x81,
+0xa0,
+0x5c,
+0x08,
+0x28,
+0x82,
+0xa4,
+0xa0,
+0x20,
+0x59,
+0x6a,
+0x0f,
+0x52,
+0xcb,
+0xc0,
+0x40,
+0x48,
+0xbc,
+0x05,
+0x95,
+0xc0,
+0x03,
+0x08,
+0x32,
+0x08,
+0x81,
+0x22,
+0x00,
+0x00,
+0x08,
+0x51,
+0x8a,
+0x86,
+0x80,
+0x95,
+0x40,
+0xb8,
+0x20,
+0x14,
+0x09,
+0x80,
+0x09,
+0xa2,
+0x2e,
+0x48,
+0x68,
+0xc9,
+0x88,
+0x16,
+0x48,
+0x81,
+0xe0,
+0x66,
+0x00,
+0x00,
+0xf3,
+0xa0,
+0x88,
+0x22,
+0x58,
+0x84,
+0x22,
+0x88,
+0x12,
+0x48,
+0x81,
+0xa0,
+0xa2,
+0x82,
+0x18,
+0x83,
+0xb6,
+0xa1,
+0x02,
+0x5a,
+0x80,
+0x40,
+0xa0,
+0x04,
+0x0a,
+0x20,
+0x64,
+0x64,
+0x00,
+0x00,
+0xf4,
+0x8f,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x22,
+0x86,
+0x00,
+0x84,
+0x42,
+0x00,
+0x06,
+0x08,
+0x95,
+0x00,
+0xe0,
+0x3a,
+0x14,
+0x80,
+0x89,
+0x80,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x6e,
+0x40,
+0x00,
+0x08,
+0x3c,
+0x38,
+0x15,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x30,
+0x05,
+0x15,
+0x96,
+0x16,
+0x1b,
+0x65,
+0x19,
+0x5b,
+0x08,
+0x02,
+0x04,
+0x41,
+0x00,
+0x18,
+0xe8,
+0xa9,
+0x80,
+0x08,
+0x23,
+0x07,
+0x79,
+0x82,
+0xe8,
+0x9c,
+0x00,
+0x19,
+0x48,
+0x37,
+0x30,
+0x1e,
+0x0b,
+0xc0,
+0x42,
+0x2a,
+0x07,
+0x63,
+0xb1,
+0x86,
+0x32,
+0x17,
+0x0b,
+0xff,
+0x62,
+0x32,
+0x0f,
+0x0b,
+0xc1,
+0xd0,
+0x51,
+0x83,
+0xab,
+0x01,
+0x00,
+0x50,
+0x41,
+0x11,
+0x82,
+0x68,
+0x55,
+0x00,
+0xb1,
+0xe0,
+0x04,
+0x46,
+0x08,
+0x89,
+0x60,
+0x35,
+0x50,
+0x41,
+0x49,
+0x83,
+0x89,
+0x55,
+0x00,
+0x79,
+0x83,
+0x48,
+0x44,
+0x41,
+0x09,
+0xe8,
+0x05,
+0x44,
+0x6d,
+0x49,
+0x68,
+0x36,
+0x21,
+0x01,
+0x00,
+0x84,
+0x82,
+0x37,
+0x80,
+0x02,
+0x28,
+0x80,
+0x6c,
+0x40,
+0x01,
+0x36,
+0x09,
+0x51,
+0x87,
+0xb1,
+0x83,
+0x08,
+0x36,
+0x14,
+0x72,
+0x81,
+0xa4,
+0x2f,
+0x16,
+0x52,
+0xe9,
+0xed,
+0x23,
+0x32,
+0xdb,
+0xa1,
+0x48,
+0x46,
+0x08,
+0x09,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x8a,
+0x68,
+0x00,
+0x00,
+0x1c,
+0x20,
+0x38,
+0x0a,
+0x42,
+0xa0,
+0x76,
+0x3a,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x8a,
+0x68,
+0x00,
+0x00,
+0x30,
+0x20,
+0x38,
+0x0a,
+0x42,
+0xa0,
+0x76,
+0x3a,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x44,
+0x20,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xf0,
+0x68,
+0x00,
+0x00,
+0x58,
+0x21,
+0x88,
+0x07,
+0x69,
+0x8e,
+0x88,
+0x40,
+0x00,
+0x02,
+0x00,
+0x02,
+0x55,
+0x03,
+0x20,
+0x08,
+0x7a,
+0x3a,
+0x10,
+0x43,
+0x22,
+0x20,
+0xbf,
+0xfb,
+0xa8,
+0x10,
+0x7a,
+0x40,
+0x00,
+0x02,
+0x01,
+0x80,
+0x66,
+0x00,
+0x01,
+0x30,
+0x68,
+0x5c,
+0x00,
+0x62,
+0x04,
+0x21,
+0x68,
+0x00,
+0x00,
+0x58,
+0x20,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x10,
+0x40,
+0x00,
+0x02,
+0x01,
+0x80,
+0x64,
+0x00,
+0x01,
+0x30,
+0x6f,
+0x5c,
+0x00,
+0x62,
+0x04,
+0x21,
+0x55,
+0x01,
+0x40,
+0x60,
+0x0b,
+0x5c,
+0x82,
+0x0a,
+0x20,
+0x26,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x02,
+0x30,
+0x27,
+0x87,
+0x00,
+0x84,
+0x44,
+0x54,
+0x07,
+0x80,
+0x84,
+0x42,
+0x54,
+0x23,
+0x82,
+0x49,
+0x80,
+0x82,
+0x8c,
+0x0c,
+0x0a,
+0x20,
+0x26,
+0x86,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x80,
+0x70,
+0x0b,
+0x55,
+0x00,
+0xa2,
+0x30,
+0x27,
+0x44,
+0x5d,
+0x60,
+0x78,
+0x0a,
+0x44,
+0x15,
+0x40,
+0x48,
+0x0b,
+0x55,
+0x01,
+0x09,
+0x80,
+0x8a,
+0x57,
+0x8f,
+0xb0,
+0x50,
+0x08,
+0x57,
+0x49,
+0x92,
+0x38,
+0x24,
+0x8c,
+0x0c,
+0x4a,
+0x20,
+0x22,
+0x86,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x82,
+0x10,
+0x21,
+0x85,
+0x00,
+0x84,
+0x44,
+0x5e,
+0x04,
+0x80,
+0x84,
+0x42,
+0x5e,
+0x20,
+0x82,
+0x69,
+0x80,
+0xc3,
+0x8c,
+0x0c,
+0x1a,
+0x30,
+0x22,
+0x87,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x01,
+0x84,
+0x88,
+0x85,
+0x00,
+0xb4,
+0x45,
+0xd4,
+0x21,
+0x02,
+0x15,
+0x50,
+0x0f,
+0x84,
+0x80,
+0xa4,
+0x47,
+0x54,
+0x20,
+0x82,
+0x45,
+0x50,
+0x1c,
+0x18,
+0x08,
+0x28,
+0xc0,
+0xc0,
+0xa2,
+0x02,
+0x28,
+0x60,
+0x0b,
+0x8c,
+0x02,
+0x64,
+0x45,
+0xd8,
+0x05,
+0x00,
+0xb4,
+0x45,
+0x88,
+0x21,
+0x02,
+0x19,
+0x80,
+0xc3,
+0x55,
+0x00,
+0xf0,
+0x48,
+0x0b,
+0xa0,
+0x82,
+0x44,
+0x45,
+0xda,
+0x0c,
+0x06,
+0x15,
+0x40,
+0x4d,
+0x22,
+0x02,
+0x28,
+0x60,
+0x0a,
+0x44,
+0x31,
+0x80,
+0x50,
+0x09,
+0x44,
+0x08,
+0x01,
+0x80,
+0x89,
+0x96,
+0x82,
+0x45,
+0x00,
+0x8d,
+0x21,
+0x02,
+0x08,
+0x40,
+0x08,
+0x44,
+0x21,
+0x82,
+0x28,
+0x09,
+0xa0,
+0x02,
+0x48,
+0x20,
+0x08,
+0x98,
+0x08,
+0xa4,
+0x44,
+0x10,
+0x20,
+0x80,
+0xd9,
+0x48,
+0x25,
+0x50,
+0x0a,
+0x01,
+0x68,
+0x24,
+0x50,
+0x08,
+0xd8,
+0x20,
+0x08,
+0x98,
+0x00,
+0x94,
+0x42,
+0x54,
+0x02,
+0x00,
+0x99,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0x40,
+0x80,
+0x20,
+0x85,
+0x80,
+0x99,
+0x80,
+0x88,
+0x57,
+0x8b,
+0x28,
+0x40,
+0x08,
+0x57,
+0x49,
+0x68,
+0x60,
+0x08,
+0x44,
+0x21,
+0x01,
+0x68,
+0xa5,
+0x50,
+0x0a,
+0x90,
+0x60,
+0x89,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x88,
+0x08,
+0x12,
+0x09,
+0x80,
+0x88,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x06,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x04,
+0x60,
+0xa4,
+0x18,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x5c,
+0x00,
+0x02,
+0xbf,
+0xf0,
+0x55,
+0x00,
+0x11,
+0x8e,
+0x81,
+0x5c,
+0x81,
+0x00,
+0x80,
+0x49,
+0x5c,
+0x82,
+0x0a,
+0x00,
+0x03,
+0x55,
+0x00,
+0x5b,
+0x80,
+0x00,
+0x55,
+0x02,
+0xe8,
+0x20,
+0x08,
+0x5d,
+0x0a,
+0x18,
+0x20,
+0x0b,
+0x59,
+0x04,
+0xc0,
+0xc0,
+0x8e,
+0xbf,
+0xf9,
+0xa1,
+0xb4,
+0xb3,
+0x5c,
+0x00,
+0x18,
+0x20,
+0x09,
+0x88,
+0x00,
+0x84,
+0x40,
+0xd4,
+0x16,
+0x82,
+0x55,
+0x40,
+0x08,
+0x0d,
+0x83,
+0x25,
+0x00,
+0xa0,
+0x0d,
+0x97,
+0xa5,
+0x50,
+0x10,
+0x18,
+0x00,
+0x25,
+0xc0,
+0x00,
+0x0d,
+0x86,
+0x08,
+0x80,
+0x52,
+0x40,
+0x00,
+0x02,
+0x18,
+0x86,
+0x55,
+0x02,
+0x68,
+0x20,
+0x08,
+0x5d,
+0x0a,
+0x08,
+0x20,
+0x0b,
+0x59,
+0x04,
+0x40,
+0xc0,
+0x8e,
+0xbf,
+0xf9,
+0xa1,
+0xb4,
+0xf3,
+0x86,
+0x00,
+0x88,
+0x80,
+0x09,
+0x44,
+0x25,
+0x61,
+0x68,
+0xa4,
+0x54,
+0x00,
+0x80,
+0xda,
+0x32,
+0x50,
+0x08,
+0x00,
+0xf1,
+0x7a,
+0x55,
+0x01,
+0x41,
+0x80,
+0x02,
+0x40,
+0x00,
+0x00,
+0x48,
+0x09,
+0x46,
+0x0a,
+0x40,
+0x50,
+0x08,
+0x57,
+0x8a,
+0xa8,
+0xda,
+0x60,
+0x57,
+0x49,
+0x62,
+0x80,
+0x10,
+0xab,
+0xfa,
+0x06,
+0x80,
+0x04,
+0x00,
+0x08,
+0x8b,
+0x00,
+0x10,
+0x88,
+0x07,
+0x6a,
+0x40,
+0xc0,
+0x88,
+0x0c,
+0x88,
+0x81,
+0x50,
+0x88,
+0x1e,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x00,
+0x55,
+0x02,
+0x10,
+0x81,
+0xa0,
+0x5d,
+0x04,
+0x10,
+0x80,
+0x80,
+0x59,
+0x10,
+0x80,
+0x81,
+0x52,
+0x55,
+0x02,
+0x22,
+0x00,
+0x20,
+0xbf,
+0xf3,
+0xa8,
+0x80,
+0xc8,
+0x5c,
+0x81,
+0x02,
+0x40,
+0xc0,
+0x80,
+0x00,
+0x25,
+0x94,
+0x08,
+0x00,
+0x00,
+0x88,
+0x00,
+0x0a,
+0x42,
+0x0b,
+0xe8,
+0x81,
+0x52,
+0x5c,
+0x00,
+0x00,
+0x80,
+0xe0,
+0xbc,
+0x0d,
+0xf8,
+0x81,
+0xca,
+0x98,
+0x2a,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x80,
+0x55,
+0x08,
+0x30,
+0x82,
+0x82,
+0x55,
+0x02,
+0xa0,
+0x82,
+0x00,
+0x55,
+0x02,
+0x00,
+0x81,
+0x02,
+0x40,
+0x00,
+0x00,
+0x81,
+0xca,
+0x58,
+0x04,
+0x00,
+0x82,
+0xc8,
+0xbf,
+0xf1,
+0xa8,
+0x82,
+0x50,
+0xbc,
+0x04,
+0xf5,
+0xc8,
+0x10,
+0x08,
+0x0a,
+0x03,
+0x90,
+0x20,
+0x88,
+0x0a,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x25,
+0x94,
+0x08,
+0x04,
+0x00,
+0x88,
+0x80,
+0xd2,
+0xbc,
+0x13,
+0xd5,
+0xc0,
+0x00,
+0x04,
+0x08,
+0xab,
+0xc0,
+0xcf,
+0x88,
+0x14,
+0xa9,
+0x82,
+0xa0,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa0,
+0x88,
+0x10,
+0x05,
+0x50,
+0x83,
+0x08,
+0x20,
+0x25,
+0x50,
+0x2a,
+0x08,
+0x18,
+0x05,
+0x50,
+0x20,
+0x08,
+0x08,
+0x28,
+0x81,
+0x4a,
+0x58,
+0x04,
+0x00,
+0x82,
+0x48,
+0xbf,
+0xf2,
+0xa8,
+0x81,
+0xd0,
+0x00,
+0x00,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x60,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x66,
+0x00,
+0x00,
+0xea,
+0x60,
+0x40,
+0x00,
+0x03,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x00,
+0xae,
+0x64,
+0x6e,
+0x00,
+0x00,
+0xd6,
+0x64,
+0x66,
+0x00,
+0x00,
+0xcc,
+0x80,
+0x66,
+0x00,
+0x01,
+0xb3,
+0x40,
+0x88,
+0x03,
+0x64,
+0x60,
+0xa4,
+0x30,
+0x30,
+0xc6,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x4a,
+0x80,
+0x10,
+0xab,
+0xfd,
+0x08,
+0x80,
+0xc9,
+0x5c,
+0x09,
+0xf8,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x59,
+0x02,
+0x5b,
+0x88,
+0x1d,
+0x70,
+0x00,
+0x00,
+0x6e,
+0x00,
+0x00,
+0xae,
+0x2d,
+0x25,
+0x9e,
+0x8b,
+0xc0,
+0x60,
+0x6c,
+0x68,
+0x08,
+0x10,
+0x03,
+0x6c,
+0x00,
+0x00,
+0xd8,
+0x53,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x67,
+0x6c,
+0x68,
+0x08,
+0x16,
+0x03,
+0x6c,
+0x00,
+0x00,
+0xd8,
+0x53,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x00,
+0xd6,
+0x2b,
+0x25,
+0x9d,
+0x8b,
+0xc0,
+0x50,
+0x6c,
+0x68,
+0x08,
+0x12,
+0x09,
+0x6c,
+0x00,
+0x00,
+0xda,
+0x49,
+0xbc,
+0x07,
+0x72,
+0x59,
+0xe8,
+0xbc,
+0x05,
+0x06,
+0xc6,
+0x80,
+0x81,
+0x60,
+0x96,
+0xc0,
+0x00,
+0x0d,
+0xa4,
+0x90,
+0x00,
+0x00,
+0x5c,
+0x02,
+0x29,
+0x02,
+0x13,
+0x68,
+0x00,
+0x00,
+0x00,
+0x75,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x75,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x49,
+0x00,
+0x00,
+0x08,
+0x81,
+0x97,
+0x88,
+0x00,
+0xb4,
+0x60,
+0xb4,
+0x08,
+0x13,
+0x58,
+0x80,
+0x89,
+0x40,
+0x00,
+0x02,
+0x80,
+0x30,
+0xab,
+0xef,
+0x08,
+0x8a,
+0xc8,
+0x5c,
+0x09,
+0x78,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x58,
+0x82,
+0x6f,
+0x88,
+0x2e,
+0xe8,
+0x83,
+0x6d,
+0x88,
+0x3e,
+0xc8,
+0x84,
+0x6b,
+0x88,
+0x4e,
+0xa8,
+0x85,
+0x69,
+0x88,
+0x5e,
+0x88,
+0x86,
+0x67,
+0x88,
+0x6e,
+0x68,
+0x87,
+0x65,
+0x88,
+0x7e,
+0x48,
+0x88,
+0x63,
+0x88,
+0x8e,
+0x28,
+0x89,
+0x61,
+0x88,
+0x9e,
+0x08,
+0x8a,
+0x4a,
+0x88,
+0x0c,
+0x99,
+0x0b,
+0x5b,
+0x88,
+0xc5,
+0x79,
+0x0d,
+0x59,
+0x88,
+0xcd,
+0x59,
+0x0e,
+0x5a,
+0x88,
+0xf5,
+0x69,
+0x10,
+0x58,
+0x88,
+0xfd,
+0x40,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9e,
+0x0b,
+0xc0,
+0xd8,
+0x88,
+0x1f,
+0x63,
+0x81,
+0x36,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x48,
+0xba,
+0x10,
+0x16,
+0x60,
+0x00,
+0x17,
+0x88,
+0x0b,
+0xc1,
+0x47,
+0x66,
+0x00,
+0x01,
+0x74,
+0x40,
+0xbc,
+0x11,
+0x73,
+0x81,
+0x26,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0xa8,
+0xba,
+0x10,
+0x16,
+0xc7,
+0x02,
+0x80,
+0xc0,
+0x83,
+0x81,
+0x26,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x40,
+0x66,
+0x00,
+0x01,
+0x78,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x27,
+0x66,
+0x00,
+0x01,
+0x74,
+0x40,
+0x90,
+0xe1,
+0x29,
+0x0d,
+0x11,
+0x90,
+0xb1,
+0x38,
+0x89,
+0xa0,
+0x88,
+0x92,
+0x18,
+0x88,
+0xa2,
+0x88,
+0x82,
+0x38,
+0x87,
+0xa4,
+0x88,
+0x72,
+0x58,
+0x86,
+0xa6,
+0x88,
+0x62,
+0x78,
+0x85,
+0xa8,
+0x88,
+0x52,
+0x98,
+0x84,
+0xaa,
+0x88,
+0x42,
+0xb8,
+0x80,
+0x0b,
+0x88,
+0x08,
+0x98,
+0x8a,
+0x88,
+0x88,
+0xa0,
+0xa8,
+0x83,
+0xac,
+0x88,
+0x32,
+0xd8,
+0x82,
+0xae,
+0x88,
+0x22,
+0xf8,
+0x81,
+0xb6,
+0x88,
+0x13,
+0x59,
+0x10,
+0x10,
+0x88,
+0xf1,
+0x68,
+0x8c,
+0x95,
+0x46,
+0x0b,
+0x40,
+0x8c,
+0x17,
+0x88,
+0xf9,
+0x4a,
+0x81,
+0x10,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0xab,
+0xef,
+0x08,
+0x8a,
+0xc8,
+0x5c,
+0x09,
+0x78,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x58,
+0x82,
+0x6f,
+0x88,
+0x2e,
+0xe8,
+0x83,
+0x6d,
+0x88,
+0x3e,
+0xc8,
+0x84,
+0x6b,
+0x88,
+0x4e,
+0xa8,
+0x85,
+0x69,
+0x88,
+0x5e,
+0x88,
+0x86,
+0x67,
+0x88,
+0x6e,
+0x68,
+0x87,
+0x65,
+0x88,
+0x7e,
+0x48,
+0x88,
+0x63,
+0x88,
+0x8e,
+0x28,
+0x89,
+0x61,
+0x88,
+0x9e,
+0x08,
+0x8a,
+0x4a,
+0x88,
+0x0c,
+0x99,
+0x0b,
+0x5b,
+0x88,
+0xc5,
+0x79,
+0x0d,
+0x59,
+0x88,
+0xcd,
+0x59,
+0x0e,
+0x5a,
+0x88,
+0xf5,
+0x69,
+0x10,
+0x58,
+0x88,
+0xfd,
+0x40,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9e,
+0x0b,
+0xc0,
+0xd8,
+0x88,
+0x1f,
+0x63,
+0x81,
+0x36,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x48,
+0xba,
+0x10,
+0x16,
+0x60,
+0x00,
+0x18,
+0x2c,
+0x0b,
+0xc1,
+0x47,
+0x66,
+0x00,
+0x01,
+0x7e,
+0x80,
+0xbc,
+0x11,
+0x73,
+0x81,
+0x26,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0xa8,
+0xba,
+0x10,
+0x16,
+0xc7,
+0x02,
+0x88,
+0xc0,
+0x83,
+0x81,
+0x26,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x40,
+0x66,
+0x00,
+0x01,
+0x82,
+0xc0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x27,
+0x66,
+0x00,
+0x01,
+0x7e,
+0x80,
+0x90,
+0xe1,
+0x29,
+0x0d,
+0x11,
+0x90,
+0xb1,
+0x38,
+0x89,
+0xa0,
+0x88,
+0x92,
+0x18,
+0x88,
+0xa2,
+0x88,
+0x82,
+0x38,
+0x87,
+0xa4,
+0x88,
+0x72,
+0x58,
+0x86,
+0xa6,
+0x88,
+0x62,
+0x78,
+0x85,
+0xa8,
+0x88,
+0x52,
+0x98,
+0x84,
+0xaa,
+0x88,
+0x42,
+0xb8,
+0x80,
+0x0b,
+0x88,
+0x08,
+0x98,
+0x8a,
+0x88,
+0x88,
+0xa0,
+0xa8,
+0x83,
+0xac,
+0x88,
+0x32,
+0xd8,
+0x82,
+0xae,
+0x88,
+0x22,
+0xf8,
+0x81,
+0xb6,
+0x88,
+0x13,
+0x59,
+0x10,
+0x10,
+0x88,
+0xf1,
+0x68,
+0x8c,
+0x95,
+0x46,
+0x0b,
+0x40,
+0x8c,
+0x17,
+0x88,
+0xf9,
+0x4a,
+0x81,
+0x10,
+0x60,
+0x02,
+0x80,
+0x00,
+0x10,
+0x98,
+0xea,
+0x03,
+0x90,
+0x20,
+0x80,
+0x07,
+0xa6,
+0x00,
+0x30,
+0x00,
+0x01,
+0x06,
+0x82,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x80,
+0x07,
+0xab,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x1c,
+0x05,
+0x20,
+0x5c,
+0xbf,
+0x03,
+0x00,
+0x0c,
+0x84,
+0x07,
+0xa9,
+0xc0,
+0x00,
+0x68,
+0x00,
+0x40,
+0x20,
+0x0a,
+0x84,
+0x04,
+0xaa,
+0x00,
+0x40,
+0x84,
+0x04,
+0x8a,
+0x04,
+0xc0,
+0x84,
+0x00,
+0xa5,
+0xd0,
+0xc3,
+0x30,
+0x7f,
+0x83,
+0x00,
+0x30,
+0xbc,
+0x28,
+0x06,
+0x83,
+0x81,
+0xc0,
+0x02,
+0x06,
+0x83,
+0x40,
+0x02,
+0x42,
+0x15,
+0xc8,
+0x10,
+0x84,
+0x00,
+0xaa,
+0x00,
+0xc2,
+0x84,
+0x84,
+0xa8,
+0x50,
+0x48,
+0x68,
+0x34,
+0x08,
+0x44,
+0x23,
+0x84,
+0x00,
+0xa8,
+0x48,
+0xca,
+0x85,
+0x04,
+0x8a,
+0x18,
+0x21,
+0x84,
+0x00,
+0xa8,
+0x58,
+0x4a,
+0x85,
+0x04,
+0x8a,
+0xc7,
+0xa0,
+0x84,
+0x00,
+0xa8,
+0x08,
+0xca,
+0x85,
+0x04,
+0x8a,
+0xc4,
+0x42,
+0x84,
+0x00,
+0xa8,
+0x08,
+0xca,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa8,
+0x08,
+0x4a,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa8,
+0x48,
+0x4a,
+0x85,
+0x04,
+0x8a,
+0x08,
+0x21,
+0x84,
+0x00,
+0x84,
+0x60,
+0xa4,
+0x00,
+0xb4,
+0x88,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x68,
+0x34,
+0x04,
+0x23,
+0x20,
+0x5c,
+0xbf,
+0x0b,
+0x04,
+0x04,
+0x68,
+0x38,
+0x1c,
+0x05,
+0x21,
+0x5c,
+0x82,
+0x00,
+0x40,
+0x48,
+0x5c,
+0x00,
+0x70,
+0x48,
+0x7a,
+0x5c,
+0xbd,
+0x09,
+0xc8,
+0x81,
+0x68,
+0x00,
+0x40,
+0x28,
+0x2e,
+0x5c,
+0x08,
+0x20,
+0x08,
+0x6e,
+0x84,
+0x84,
+0xaa,
+0x0c,
+0xa1,
+0x9c,
+0x08,
+0x28,
+0x48,
+0x00,
+0x85,
+0x05,
+0x00,
+0x00,
+0x00,
+0x84,
+0x00,
+0x02,
+0x41,
+0x00,
+0x84,
+0x05,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xff,
+0xc0,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x02,
+0x38,
+0x11,
+0x82,
+0x58,
+0x10,
+0xbc,
+0x32,
+0x06,
+0x83,
+0x81,
+0xc0,
+0x02,
+0x16,
+0x83,
+0x40,
+0x42,
+0x02,
+0x08,
+0x48,
+0x02,
+0x84,
+0x05,
+0x2a,
+0x00,
+0x60,
+0xb0,
+0x48,
+0x28,
+0x40,
+0x52,
+0xa0,
+0x88,
+0x18,
+0x40,
+0x02,
+0x24,
+0x11,
+0x28,
+0x40,
+0x52,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x02,
+0x25,
+0x91,
+0x0b,
+0xff,
+0xc0,
+0x84,
+0x00,
+0x22,
+0x58,
+0x10,
+0xbc,
+0x01,
+0x1b,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x40,
+0x28,
+0x6e,
+0x80,
+0x86,
+0xe8,
+0x48,
+0x4a,
+0xa0,
+0xcc,
+0x18,
+0x48,
+0x0a,
+0x6c,
+0x68,
+0x08,
+0x40,
+0x4a,
+0xb0,
+0x50,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x40,
+0x0a,
+0x24,
+0x13,
+0x68,
+0x40,
+0x4a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x0a,
+0x25,
+0x93,
+0x0b,
+0xff,
+0xc0,
+0x84,
+0x00,
+0x82,
+0x58,
+0x20,
+0xbc,
+0x01,
+0x1b,
+0xa1,
+0x40,
+0xba,
+0x14,
+0x88,
+0x40,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x14,
+0x40,
+0x68,
+0x34,
+0x04,
+0x04,
+0x20,
+0x5c,
+0x02,
+0xe2,
+0xff,
+0xc0,
+0x5c,
+0x00,
+0x70,
+0x40,
+0x48,
+0x9c,
+0x00,
+0x0b,
+0x04,
+0x80,
+0x68,
+0x34,
+0x00,
+0x1e,
+0x21,
+0x84,
+0x05,
+0x0b,
+0x1d,
+0xf2,
+0x84,
+0x85,
+0x2a,
+0x08,
+0x81,
+0xb1,
+0x81,
+0xc8,
+0x48,
+0x48,
+0xa0,
+0x22,
+0x06,
+0x81,
+0x50,
+0xc8,
+0x40,
+0x08,
+0x40,
+0x50,
+0x68,
+0x00,
+0x01,
+0x41,
+0x48,
+0x88,
+0x03,
+0x6a,
+0x06,
+0x00,
+0x46,
+0x0a,
+0x40,
+0x48,
+0xc8,
+0x84,
+0x04,
+0xaa,
+0x80,
+0x10,
+0x68,
+0x34,
+0x00,
+0x09,
+0x20,
+0x5c,
+0x08,
+0xb3,
+0x00,
+0x3c,
+0x5c,
+0x81,
+0x00,
+0x40,
+0x00,
+0x52,
+0x0c,
+0x03,
+0x00,
+0x26,
+0x84,
+0x05,
+0x0a,
+0x00,
+0xa0,
+0x84,
+0x04,
+0x8a,
+0x00,
+0x40,
+0x80,
+0x04,
+0x88,
+0x00,
+0x4a,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x4a,
+0x84,
+0x7f,
+0xa0,
+0x00,
+0x00,
+0x68,
+0x34,
+0x08,
+0x4a,
+0x20,
+0x38,
+0x01,
+0xc8,
+0x40,
+0x48,
+0x40,
+0x00,
+0x00,
+0x40,
+0xc8,
+0x64,
+0x00,
+0x01,
+0x1e,
+0xa7,
+0x68,
+0x00,
+0x01,
+0xbc,
+0x20,
+0x39,
+0x02,
+0x08,
+0x00,
+0x21,
+0x68,
+0x34,
+0x08,
+0x20,
+0x22,
+0x84,
+0x80,
+0x88,
+0x00,
+0x21,
+0x85,
+0x04,
+0x8a,
+0x10,
+0x22,
+0x84,
+0x80,
+0x88,
+0x00,
+0x21,
+0x84,
+0x02,
+0x08,
+0x10,
+0x48,
+0x00,
+0x00,
+0x08,
+0x48,
+0x08,
+0x81,
+0x04,
+0x80,
+0x00,
+0x00,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x08,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x68,
+0x34,
+0x08,
+0x57,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x04,
+0x7a,
+0x5c,
+0x81,
+0x03,
+0x04,
+0x24,
+0x5c,
+0x13,
+0x70,
+0x40,
+0x48,
+0xa0,
+0x02,
+0x08,
+0x00,
+0x7a,
+0x80,
+0x04,
+0x88,
+0x40,
+0x7a,
+0xa0,
+0x52,
+0x08,
+0x02,
+0x4a,
+0xb0,
+0x60,
+0x48,
+0x40,
+0x48,
+0xa0,
+0x48,
+0x08,
+0x00,
+0x7a,
+0xac,
+0x9a,
+0x08,
+0x02,
+0x7a,
+0x84,
+0x07,
+0xaa,
+0x01,
+0x00,
+0xac,
+0xb0,
+0x08,
+0x00,
+0x7a,
+0xac,
+0x40,
+0x04,
+0x60,
+0xa4,
+0x00,
+0x24,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x34,
+0x00,
+0x17,
+0x20,
+0x68,
+0x03,
+0xfb,
+0xff,
+0xc9,
+0x5c,
+0x08,
+0x20,
+0x40,
+0x7a,
+0x5c,
+0x0b,
+0x32,
+0x00,
+0x80,
+0x5c,
+0xbd,
+0x00,
+0x40,
+0x02,
+0x54,
+0x4a,
+0xaa,
+0x03,
+0x21,
+0x5c,
+0x00,
+0x40,
+0x40,
+0x49,
+0x5c,
+0x09,
+0xd2,
+0xbf,
+0xe0,
+0x84,
+0x80,
+0x95,
+0x20,
+0x96,
+0x88,
+0x0f,
+0x65,
+0xc0,
+0x02,
+0x84,
+0x84,
+0x96,
+0x83,
+0x80,
+0x80,
+0xf2,
+0x28,
+0x40,
+0x08,
+0x52,
+0x0d,
+0x20,
+0x80,
+0x49,
+0x5c,
+0x1d,
+0x68,
+0x40,
+0x48,
+0xa0,
+0x5a,
+0x08,
+0x40,
+0x7a,
+0x9d,
+0x00,
+0x08,
+0x50,
+0x50,
+0x84,
+0x05,
+0x28,
+0x81,
+0x60,
+0x66,
+0x00,
+0x01,
+0x2e,
+0x20,
+0x5c,
+0x08,
+0x30,
+0x80,
+0x09,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x55,
+0x03,
+0x6b,
+0x03,
+0x24,
+0x30,
+0x12,
+0x8b,
+0xff,
+0x6a,
+0x5c,
+0x1d,
+0x68,
+0x80,
+0x49,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x1a,
+0xeb,
+0x00,
+0x0e,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x1a,
+0xa9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x0d,
+0xe9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0x69,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0xa9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0xeb,
+0x00,
+0x26,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x07,
+0x2b,
+0x00,
+0xc6,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x07,
+0x6b,
+0x00,
+0x36,
+0x5c,
+0x02,
+0x68,
+0x81,
+0x20,
+0x5c,
+0x01,
+0xe0,
+0x80,
+0xb6,
+0x6c,
+0x70,
+0x10,
+0x1e,
+0x49,
+0x5c,
+0x03,
+0xe0,
+0x40,
+0x48,
+0x5c,
+0x00,
+0x72,
+0x05,
+0x80,
+0x84,
+0x07,
+0xaa,
+0x00,
+0x40,
+0x84,
+0x07,
+0xab,
+0x05,
+0xdd,
+0xa0,
+0x02,
+0x08,
+0x40,
+0x49,
+0xa0,
+0x16,
+0x08,
+0x40,
+0x48,
+0x46,
+0x0a,
+0x42,
+0x05,
+0x00,
+0x84,
+0x04,
+0xaa,
+0x80,
+0x20,
+0x5c,
+0x08,
+0x01,
+0x8e,
+0x8a,
+0x5c,
+0x00,
+0x51,
+0x82,
+0x88,
+0x68,
+0x00,
+0x03,
+0xe8,
+0x09,
+0x6c,
+0x70,
+0x10,
+0x0e,
+0x0b,
+0x25,
+0x83,
+0x8b,
+0xc0,
+0x41,
+0xba,
+0x14,
+0x86,
+0xc7,
+0x01,
+0x00,
+0xe5,
+0x20,
+0x00,
+0x00,
+0x2a,
+0x07,
+0x63,
+0x01,
+0x70,
+0xbf,
+0xf5,
+0x26,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x83,
+0x81,
+0xe6,
+0x52,
+0x0d,
+0x33,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x4a,
+0x5c,
+0x3f,
+0xe3,
+0x80,
+0x00,
+0x68,
+0x38,
+0x08,
+0x00,
+0x20,
+0x5c,
+0x09,
+0xe2,
+0xbf,
+0x90,
+0x52,
+0x09,
+0x62,
+0x00,
+0x41,
+0x5d,
+0x8c,
+0x12,
+0x08,
+0x20,
+0x5c,
+0x00,
+0x00,
+0x80,
+0x60,
+0xa0,
+0x16,
+0x0a,
+0x05,
+0x00,
+0xa0,
+0x4c,
+0x28,
+0x80,
+0xe2,
+0xa1,
+0x04,
+0x28,
+0x81,
+0x62,
+0xa1,
+0x02,
+0x28,
+0x81,
+0xe2,
+0xa1,
+0x16,
+0x2a,
+0x15,
+0x02,
+0x88,
+0x4c,
+0x88,
+0x83,
+0x62,
+0x88,
+0x3e,
+0x08,
+0x84,
+0x61,
+0x88,
+0x55,
+0x28,
+0x82,
+0x4a,
+0x88,
+0x2c,
+0x98,
+0x85,
+0xf6,
+0x40,
+0x00,
+0x03,
+0x04,
+0x04,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x48,
+0x5c,
+0x00,
+0xa0,
+0x48,
+0x7a,
+0x5c,
+0x00,
+0x68,
+0x48,
+0xc9,
+0x84,
+0x8c,
+0xa8,
+0x86,
+0x50,
+0x00,
+0x00,
+0x08,
+0x80,
+0x21,
+0x00,
+0x00,
+0x08,
+0x4d,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x40,
+0x49,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x5c,
+0x00,
+0xa0,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x68,
+0x81,
+0x21,
+0x84,
+0x07,
+0xa8,
+0x41,
+0x7a,
+0x00,
+0x00,
+0x08,
+0x84,
+0x8a,
+0x84,
+0x8c,
+0xa0,
+0x00,
+0x00,
+0x88,
+0x1a,
+0x08,
+0x83,
+0x21,
+0x84,
+0x5c,
+0x88,
+0x48,
+0x49,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x5c,
+0xbb,
+0x00,
+0x83,
+0x20,
+0x88,
+0x50,
+0x99,
+0xc0,
+0x00,
+0x84,
+0x00,
+0xa3,
+0xb1,
+0x80,
+0x30,
+0x14,
+0x0b,
+0xc0,
+0x89,
+0x5c,
+0x19,
+0x20,
+0x86,
+0x0a,
+0x88,
+0x5b,
+0x6b,
+0xa1,
+0x48,
+0x5c,
+0x00,
+0x22,
+0x80,
+0x70,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x55,
+0x03,
+0x80,
+0x84,
+0x21,
+0x58,
+0x08,
+0x00,
+0x82,
+0x89,
+0x88,
+0x20,
+0xa4,
+0x3e,
+0x4d,
+0x08,
+0x3a,
+0x0b,
+0x04,
+0x04,
+0x88,
+0x5b,
+0x6b,
+0xa1,
+0x48,
+0x5c,
+0x3f,
+0xe2,
+0x80,
+0x70,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x08,
+0x00,
+0x20,
+0x5c,
+0x09,
+0xf3,
+0x00,
+0x14,
+0x52,
+0x0d,
+0x68,
+0x40,
+0x7a,
+0x5c,
+0x00,
+0x72,
+0x00,
+0x40,
+0x84,
+0x07,
+0xaa,
+0x00,
+0x20,
+0x84,
+0x04,
+0x9a,
+0x01,
+0x60,
+0x84,
+0x04,
+0x8a,
+0x05,
+0x00,
+0x84,
+0x04,
+0xaa,
+0xbf,
+0xf0,
+0xa0,
+0x4a,
+0x08,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x66,
+0x60,
+0x00,
+0x12,
+0x6c,
+0x08,
+0x80,
+0xb6,
+0x88,
+0x02,
+0x0b,
+0xa1,
+0x48,
+0x84,
+0x00,
+0x8a,
+0x80,
+0x10,
+0x62,
+0x00,
+0x00,
+0x00,
+0x36,
+0x5c,
+0x81,
+0x09,
+0x82,
+0x24,
+0x5c,
+0x80,
+0x81,
+0x82,
+0x60,
+0xbb,
+0x00,
+0x08,
+0x00,
+0xcc,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x38,
+0x00,
+0xe2,
+0x11,
+0x34,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x6d,
+0x84,
+0x86,
+0x06,
+0x20,
+0x00,
+0x00,
+0x01,
+0x43,
+0x90,
+0x20,
+0x00,
+0x00,
+0x08,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x00,
+0x61,
+0x88,
+0x2c,
+0x50,
+0x8d,
+0x20,
+0x48,
+0x20,
+0x51,
+0x85,
+0x3a,
+0xc0,
+0x20,
+0x98,
+0x2e,
+0xe0,
+0x00,
+0x00,
+0x9c,
+0x40,
+0x08,
+0x40,
+0x49,
+0x62,
+0x00,
+0x00,
+0x00,
+0x24,
+0x84,
+0x86,
+0x09,
+0x8e,
+0x80,
+0x80,
+0x40,
+0x92,
+0x81,
+0x40,
+0xba,
+0x14,
+0x85,
+0x04,
+0xc0,
+0x04,
+0x86,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x06,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x04,
+0x60,
+0xa4,
+0x18,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x02,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x09,
+0x80,
+0x82,
+0x55,
+0x00,
+0xb8,
+0x20,
+0x08,
+0x8c,
+0x06,
+0x0a,
+0x20,
+0x01,
+0x8c,
+0x12,
+0xe4,
+0x42,
+0x00,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x04,
+0x88,
+0x94,
+0x46,
+0xc8,
+0x06,
+0x08,
+0x85,
+0xb0,
+0x82,
+0x98,
+0x04,
+0xa5,
+0x78,
+0x9a,
+0x18,
+0x48,
+0x05,
+0x74,
+0xb1,
+0x3a,
+0x14,
+0x88,
+0xc1,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x32,
+0x02,
+0x8b,
+0xc3,
+0x58,
+0x22,
+0x86,
+0xe5,
+0xb8,
+0xa2,
+0x30,
+0x0f,
+0x05,
+0x70,
+0x80,
+0x30,
+0x00,
+0xf5,
+0x08,
+0x1c,
+0x30,
+0x4e,
+0xa2,
+0xa6,
+0x63,
+0x4b,
+0xc1,
+0x23,
+0x3c,
+0x08,
+0x20,
+0x0c,
+0xd2,
+0x35,
+0xd4,
+0x68,
+0x20,
+0x01,
+0x51,
+0x20,
+0x28,
+0x16,
+0x25,
+0x14,
+0x2a,
+0x84,
+0x00,
+0x85,
+0x70,
+0xb0,
+0xb0,
+0x0f,
+0xa5,
+0x16,
+0xe6,
+0x04,
+0x08,
+0x92,
+0x80,
+0x20,
+0x68,
+0x1f,
+0xff,
+0xff,
+0xc8,
+0x36,
+0x80,
+0x35,
+0x44,
+0x84,
+0x98,
+0x0c,
+0x82,
+0x32,
+0x09,
+0x28,
+0x08,
+0x02,
+0x10,
+0x3f,
+0x20,
+0x90,
+0xc2,
+0x81,
+0x3f,
+0x08,
+0xb0,
+0x02,
+0x23,
+0xc0,
+0x98,
+0x00,
+0x80,
+0x83,
+0x00,
+0x22,
+0x3c,
+0x02,
+0xe0,
+0x28,
+0x98,
+0x00,
+0x80,
+0x8c,
+0x00,
+0x22,
+0x04,
+0x09,
+0x80,
+0x0b,
+0x08,
+0xb0,
+0x02,
+0x23,
+0xc0,
+0x98,
+0x00,
+0xa0,
+0x8b,
+0x00,
+0x22,
+0x3c,
+0x02,
+0xe0,
+0x28,
+0x98,
+0x00,
+0xa4,
+0x47,
+0x00,
+0x3a,
+0x14,
+0x82,
+0x20,
+0x40,
+0x98,
+0x00,
+0x8b,
+0xa1,
+0x48,
+0x98,
+0xe8,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x08,
+0x29,
+0x58,
+0x2c,
+0x52,
+0xcb,
+0x02,
+0xbf,
+0xc0,
+0x88,
+0x0e,
+0x14,
+0x20,
+0x7c,
+0x88,
+0x16,
+0x28,
+0x80,
+0x76,
+0x00,
+0x00,
+0x09,
+0x5a,
+0x2e,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0xa0,
+0x95,
+0xba,
+0xe3,
+0x20,
+0x30,
+0xbc,
+0x04,
+0x18,
+0x59,
+0x08,
+0x55,
+0x03,
+0x23,
+0xc0,
+0x4f,
+0x85,
+0x94,
+0x82,
+0x49,
+0x60,
+0x40,
+0x00,
+0x01,
+0x58,
+0x60,
+0x5c,
+0x82,
+0x0a,
+0x18,
+0xa1,
+0x88,
+0x1e,
+0x10,
+0x00,
+0x00,
+0x80,
+0x88,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x06,
+0x95,
+0xc8,
+0x10,
+0x08,
+0x26,
+0x13,
+0x81,
+0x05,
+0x95,
+0x8a,
+0xc2,
+0x59,
+0x60,
+0xbc,
+0x45,
+0x18,
+0x82,
+0xe5,
+0x88,
+0x36,
+0x30,
+0x00,
+0x00,
+0x8c,
+0x02,
+0xa8,
+0x20,
+0x09,
+0x44,
+0x08,
+0x00,
+0x55,
+0x22,
+0x82,
+0x00,
+0x84,
+0x44,
+0x40,
+0x02,
+0x00,
+0x88,
+0x50,
+0x09,
+0x44,
+0x25,
+0x01,
+0x82,
+0x40,
+0x98,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x00,
+0x00,
+0x08,
+0xc1,
+0x2a,
+0x82,
+0x00,
+0x94,
+0x40,
+0x80,
+0x06,
+0x00,
+0x84,
+0x44,
+0x48,
+0x06,
+0x08,
+0x95,
+0x50,
+0x0a,
+0x18,
+0x48,
+0x00,
+0x81,
+0xa4,
+0x98,
+0x08,
+0x28,
+0xc1,
+0x60,
+0x40,
+0x00,
+0x00,
+0x83,
+0xd2,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x04,
+0x5c,
+0x08,
+0x71,
+0x48,
+0xad,
+0x25,
+0x9a,
+0x8b,
+0xc1,
+0xa1,
+0x94,
+0x82,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x07,
+0x88,
+0x83,
+0x88,
+0x88,
+0x12,
+0x06,
+0x60,
+0x00,
+0x14,
+0xc4,
+0x0b,
+0xc1,
+0x7f,
+0x40,
+0x00,
+0x00,
+0x81,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x4a,
+0x28,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x00,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0x42,
+0x03,
+0x40,
+0x81,
+0x20,
+0x94,
+0xa6,
+0x46,
+0x60,
+0x00,
+0x14,
+0xc4,
+0x88,
+0x83,
+0x88,
+0xbc,
+0x08,
+0xf8,
+0x81,
+0xa0,
+0xbc,
+0x06,
+0xf8,
+0x81,
+0xa0,
+0xbc,
+0x04,
+0xf8,
+0x81,
+0xa0,
+0x88,
+0x1a,
+0x08,
+0x82,
+0xe5,
+0x88,
+0x36,
+0x30,
+0x00,
+0x00,
+0x88,
+0x22,
+0x58,
+0x41,
+0x08,
+0x59,
+0x01,
+0x02,
+0x2c,
+0xd8,
+0x42,
+0x04,
+0x48,
+0x82,
+0xa5,
+0x5c,
+0x81,
+0x00,
+0x81,
+0xe0,
+0x38,
+0x10,
+0x48,
+0x83,
+0x20,
+0x00,
+0x00,
+0x09,
+0x40,
+0xad,
+0x25,
+0x92,
+0x8b,
+0xc3,
+0xc1,
+0x00,
+0x00,
+0x08,
+0x81,
+0x20,
+0x88,
+0x0a,
+0x3a,
+0x02,
+0x80,
+0x8d,
+0x82,
+0xa8,
+0x28,
+0x09,
+0x44,
+0x08,
+0x00,
+0x40,
+0x20,
+0x82,
+0x80,
+0x84,
+0x44,
+0x40,
+0x04,
+0x00,
+0x98,
+0x28,
+0x08,
+0x44,
+0x25,
+0x01,
+0x82,
+0x40,
+0x98,
+0x08,
+0x25,
+0x50,
+0x0b,
+0x8d,
+0x86,
+0x06,
+0x80,
+0x00,
+0x0d,
+0x72,
+0x08,
+0xd9,
+0x2a,
+0x82,
+0x80,
+0x94,
+0x40,
+0x80,
+0x06,
+0x80,
+0x84,
+0x44,
+0x48,
+0x06,
+0x88,
+0x94,
+0x46,
+0xd2,
+0x18,
+0x48,
+0x09,
+0x80,
+0x82,
+0x8d,
+0x96,
+0x08,
+0x80,
+0xd2,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x04,
+0x5c,
+0x08,
+0x71,
+0x48,
+0xad,
+0x25,
+0x9a,
+0x8b,
+0xc1,
+0x41,
+0x94,
+0x82,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x05,
+0x88,
+0x80,
+0x88,
+0x88,
+0x12,
+0x06,
+0x60,
+0x00,
+0x14,
+0xee,
+0x0b,
+0xc0,
+0xc7,
+0x66,
+0x00,
+0x01,
+0x47,
+0xa8,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x00,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0x42,
+0x02,
+0x40,
+0x81,
+0x20,
+0x94,
+0xbe,
+0x46,
+0x60,
+0x00,
+0x14,
+0xee,
+0x88,
+0x80,
+0x88,
+0x38,
+0x11,
+0x48,
+0x81,
+0xa0,
+0x00,
+0x00,
+0x09,
+0x40,
+0x2d,
+0x25,
+0x92,
+0x8b,
+0xc2,
+0xc1,
+0x68,
+0x20,
+0x01,
+0x7c,
+0x23,
+0x5c,
+0x81,
+0x12,
+0xc0,
+0x80,
+0x68,
+0x20,
+0x01,
+0x82,
+0x24,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x20,
+0x52,
+0x49,
+0x40,
+0x19,
+0x0a,
+0x68,
+0x20,
+0x00,
+0xe6,
+0x25,
+0x5c,
+0x82,
+0x08,
+0x21,
+0x08,
+0x81,
+0x80,
+0x98,
+0x01,
+0x4a,
+0x82,
+0x94,
+0x88,
+0x00,
+0x49,
+0x00,
+0x00,
+0x08,
+0x20,
+0x0b,
+0x82,
+0x28,
+0x98,
+0x1a,
+0x88,
+0x6c,
+0x40,
+0x02,
+0xfc,
+0x0a,
+0x82,
+0x84,
+0xb8,
+0x2a,
+0xc9,
+0x80,
+0x2c,
+0x86,
+0xc4,
+0x00,
+0x1c,
+0x44,
+0xa0,
+0x00,
+0x00,
+0x88,
+0x32,
+0x28,
+0x60,
+0x0b,
+0x85,
+0x80,
+0x28,
+0x60,
+0x88,
+0x85,
+0x88,
+0x96,
+0xc4,
+0x00,
+0x30,
+0x80,
+0xa8,
+0x68,
+0x4b,
+0x84,
+0x05,
+0x28,
+0x68,
+0xc8,
+0x84,
+0x0c,
+0x99,
+0x50,
+0xe0,
+0x6c,
+0x40,
+0x01,
+0xd0,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x40,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x94,
+0x3a,
+0xc3,
+0x20,
+0x20,
+0xbc,
+0x21,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0xbc,
+0x03,
+0xf9,
+0x43,
+0x64,
+0xb0,
+0x00,
+0xd9,
+0x43,
+0x65,
+0xa0,
+0x02,
+0x19,
+0x48,
+0x2d,
+0x59,
+0x01,
+0x42,
+0x08,
+0x12,
+0x42,
+0x08,
+0xc1,
+0x50,
+0x2d,
+0xa1,
+0x40,
+0x95,
+0xc0,
+0x84,
+0x14,
+0x02,
+0xe2,
+0x58,
+0x30,
+0xbc,
+0x07,
+0x99,
+0x42,
+0xae,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x91,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x70,
+0xb0,
+0x00,
+0xcb,
+0xc0,
+0x57,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x30,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x11,
+0xb0,
+0x00,
+0xcb,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x50,
+0x00,
+0x00,
+0xba,
+0x14,
+0x8b,
+0x00,
+0x0c,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x94,
+0x22,
+0xc3,
+0x20,
+0x20,
+0xbc,
+0x1c,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0xbc,
+0x03,
+0xf9,
+0x41,
+0xe4,
+0xb0,
+0x00,
+0xd9,
+0x41,
+0xe5,
+0x5c,
+0x08,
+0x6a,
+0x00,
+0x11,
+0x94,
+0x02,
+0xe5,
+0x2c,
+0xb8,
+0x14,
+0x82,
+0xdb,
+0xc0,
+0x79,
+0x94,
+0x8a,
+0xe3,
+0x20,
+0x28,
+0xbc,
+0x09,
+0x13,
+0x20,
+0x30,
+0xbc,
+0x07,
+0x0b,
+0x00,
+0x0c,
+0xbc,
+0x05,
+0x73,
+0x20,
+0x28,
+0xbc,
+0x03,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x01,
+0x1b,
+0x00,
+0x0c,
+0xba,
+0x14,
+0x89,
+0x41,
+0x66,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x48,
+0xb0,
+0x00,
+0xc0,
+0x00,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x08,
+0x82,
+0x85,
+0x00,
+0xa2,
+0xa0,
+0x76,
+0x81,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x85,
+0x00,
+0x03,
+0x00,
+0x30,
+0xbc,
+0x1d,
+0x4a,
+0x00,
+0x41,
+0x84,
+0x80,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x01,
+0x38,
+0x41,
+0x48,
+0xa0,
+0x82,
+0x28,
+0x50,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x25,
+0x40,
+0x00,
+0x00,
+0x48,
+0xc8,
+0x5b,
+0x48,
+0x03,
+0x01,
+0x05,
+0x50,
+0x4b,
+0x08,
+0x10,
+0x0a,
+0x5c,
+0x82,
+0x00,
+0xd1,
+0x32,
+0x54,
+0x02,
+0x90,
+0x41,
+0x08,
+0x57,
+0x0d,
+0x20,
+0xd1,
+0x7a,
+0x81,
+0x04,
+0x89,
+0x80,
+0x08,
+0x50,
+0x4b,
+0x10,
+0xd1,
+0x30,
+0x54,
+0x04,
+0x03,
+0xa1,
+0x48,
+0x8d,
+0x17,
+0x80,
+0x00,
+0x00,
+0xba,
+0x14,
+0x88,
+0x4a,
+0xfa,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x08,
+0xc2,
+0x85,
+0x00,
+0xa2,
+0xa0,
+0x76,
+0x81,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x85,
+0x00,
+0x03,
+0x00,
+0x30,
+0xbc,
+0x1f,
+0x4a,
+0x01,
+0x81,
+0x84,
+0x80,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x01,
+0x38,
+0x46,
+0x48,
+0xa0,
+0x82,
+0x28,
+0x50,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x25,
+0x40,
+0x00,
+0x00,
+0x48,
+0xc8,
+0x5c,
+0x82,
+0x0b,
+0x01,
+0x05,
+0x5b,
+0x48,
+0x10,
+0x46,
+0x0a,
+0x81,
+0x00,
+0x02,
+0xe0,
+0x36,
+0x50,
+0x4b,
+0x00,
+0x10,
+0xca,
+0x98,
+0x08,
+0x88,
+0xd0,
+0x31,
+0x28,
+0x00,
+0x85,
+0x04,
+0xb0,
+0x0d,
+0x0d,
+0x80,
+0x00,
+0x00,
+0x8d,
+0x03,
+0x25,
+0x40,
+0x08,
+0x3a,
+0x14,
+0x88,
+0xd0,
+0x78,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x08,
+0x22,
+0x08,
+0xe0,
+0xa0,
+0x4d,
+0xa9,
+0x50,
+0x2e,
+0x25,
+0x93,
+0x0b,
+0xc0,
+0x69,
+0x84,
+0x07,
+0xa3,
+0x81,
+0x0c,
+0x52,
+0x09,
+0x83,
+0xa1,
+0x48,
+0x94,
+0x8e,
+0x00,
+0x00,
+0x00,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x20,
+0x01,
+0x2b,
+0x20,
+0x5c,
+0x81,
+0x03,
+0x00,
+0x0c,
+0x5c,
+0x06,
+0x68,
+0x02,
+0x0a,
+0x68,
+0x20,
+0x01,
+0x05,
+0x21,
+0x51,
+0x89,
+0x80,
+0x40,
+0x01,
+0x51,
+0x8a,
+0x00,
+0x08,
+0x50,
+0x68,
+0x20,
+0x01,
+0xf0,
+0x22,
+0x84,
+0x80,
+0x28,
+0x10,
+0x52,
+0x81,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x0f,
+0xc2,
+0x36,
+0x82,
+0x00,
+0x0d,
+0x72,
+0x18,
+0x10,
+0x7a,
+0x68,
+0x00,
+0x00,
+0xdc,
+0x24,
+0x85,
+0x07,
+0xaa,
+0x10,
+0x42,
+0x84,
+0x90,
+0xb8,
+0x58,
+0x02,
+0x85,
+0x88,
+0xa8,
+0x48,
+0x48,
+0xa2,
+0x06,
+0x35,
+0xc0,
+0x02,
+0x86,
+0x04,
+0x98,
+0x12,
+0x52,
+0x68,
+0x00,
+0x00,
+0xa6,
+0x21,
+0xa1,
+0xc2,
+0x0a,
+0x10,
+0x64,
+0xab,
+0xff,
+0x0e,
+0x21,
+0x04,
+0x84,
+0x07,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0xa7,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0xa7,
+0xa6,
+0xc0,
+0x00,
+0x37,
+0x64,
+0x86,
+0xc0,
+0x00,
+0x36,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x19,
+0x84,
+0xb6,
+0xc4,
+0x00,
+0x19,
+0xa4,
+0xb6,
+0xc4,
+0x00,
+0x31,
+0x44,
+0xb6,
+0xc4,
+0x00,
+0x31,
+0xa4,
+0xb9,
+0x58,
+0x65,
+0x6c,
+0x40,
+0x01,
+0x06,
+0x51,
+0x6c,
+0x40,
+0x01,
+0x08,
+0x51,
+0x84,
+0x87,
+0xa8,
+0x50,
+0x4a,
+0x86,
+0x04,
+0xa8,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x23,
+0x8b,
+0x68,
+0x41,
+0x4a,
+0x66,
+0x00,
+0x00,
+0x4c,
+0x28,
+0x40,
+0x00,
+0x03,
+0x19,
+0x04,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x68,
+0x00,
+0x04,
+0xe2,
+0x0a,
+0x88,
+0x03,
+0x68,
+0x41,
+0x4a,
+0x64,
+0x00,
+0x00,
+0x4c,
+0x2f,
+0x5c,
+0x02,
+0xa2,
+0x80,
+0x10,
+0x30,
+0x1a,
+0x8b,
+0xc1,
+0x58,
+0x6c,
+0x40,
+0x02,
+0xc0,
+0x08,
+0x84,
+0x00,
+0xb3,
+0x69,
+0xc0,
+0x98,
+0x00,
+0xb3,
+0x09,
+0x38,
+0xbc,
+0x0d,
+0x46,
+0xc4,
+0x00,
+0x2b,
+0xc0,
+0x83,
+0x01,
+0x30,
+0xbc,
+0x03,
+0x2b,
+0xc0,
+0x4f,
+0x6c,
+0x40,
+0x02,
+0xbe,
+0x08,
+0x6c,
+0x40,
+0x02,
+0xb0,
+0x08,
+0x28,
+0x13,
+0x04,
+0x20,
+0x17,
+0x98,
+0x00,
+0xa2,
+0xf1,
+0x75,
+0x98,
+0x28,
+0x9b,
+0xa1,
+0x48,
+0x98,
+0x24,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0xdd,
+0x22,
+0x6c,
+0x00,
+0x01,
+0xb8,
+0x08,
+0x85,
+0x00,
+0xa3,
+0x01,
+0x30,
+0x55,
+0x03,
+0xab,
+0xc5,
+0xf9,
+0x85,
+0x04,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xa7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x08,
+0x08,
+0x5b,
+0x48,
+0x00,
+0x40,
+0x08,
+0x5b,
+0x48,
+0x11,
+0x54,
+0x24,
+0x32,
+0x06,
+0x0b,
+0xc3,
+0x39,
+0x9a,
+0x00,
+0x03,
+0x01,
+0x30,
+0xbc,
+0x09,
+0x48,
+0x48,
+0x0a,
+0x36,
+0x98,
+0x09,
+0x80,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x44,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x1b,
+0xc7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0xbc,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x06,
+0x0a,
+0x2a,
+0x06,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x40,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xc4,
+0x85,
+0xc0,
+0xce,
+0xb0,
+0x13,
+0x76,
+0xe0,
+0x00,
+0x14,
+0x8a,
+0xe6,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x15,
+0xc8,
+0x10,
+0x18,
+0xe8,
+0x86,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x95,
+0x2c,
+0xf8,
+0x14,
+0xa4,
+0x4b,
+0xc0,
+0x99,
+0x5c,
+0x0b,
+0xa0,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2c,
+0x20,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0x04,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x0a,
+0x52,
+0x49,
+0xa3,
+0xa1,
+0x48,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x00,
+0x00,
+0x03,
+0x01,
+0x30,
+0xbc,
+0x05,
+0x38,
+0x48,
+0x0a,
+0x36,
+0x98,
+0x09,
+0x80,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x42,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x1b,
+0xc7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0xbc,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x04,
+0x0a,
+0x2a,
+0x06,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x0e,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xc4,
+0x86,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x16,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0xc0,
+0x07,
+0x30,
+0x17,
+0x55,
+0x20,
+0xb2,
+0x2c,
+0x02,
+0x09,
+0x4a,
+0x46,
+0x84,
+0x87,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x8b,
+0xa1,
+0x40,
+0x68,
+0x20,
+0x01,
+0x46,
+0x21,
+0x68,
+0x00,
+0x00,
+0xcd,
+0x22,
+0x39,
+0x02,
+0x08,
+0x08,
+0x08,
+0x81,
+0x07,
+0xa8,
+0x10,
+0x48,
+0x68,
+0x00,
+0x00,
+0xd2,
+0x20,
+0x84,
+0x80,
+0x88,
+0x00,
+0x7a,
+0x81,
+0x04,
+0x88,
+0x00,
+0x7a,
+0x68,
+0x00,
+0x00,
+0xca,
+0x23,
+0x80,
+0x04,
+0x86,
+0x80,
+0x00,
+0x58,
+0x8a,
+0xca,
+0x10,
+0x01,
+0x81,
+0x86,
+0xc8,
+0x10,
+0x7a,
+0xa0,
+0x00,
+0x48,
+0x58,
+0x7a,
+0x85,
+0x06,
+0x14,
+0x60,
+0xa4,
+0x00,
+0x07,
+0xa8,
+0x40,
+0x64,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0c,
+0x16,
+0xc0,
+0x00,
+0x1a,
+0x22,
+0x06,
+0x80,
+0x00,
+0x58,
+0xd2,
+0xc6,
+0xc0,
+0x00,
+0x1a,
+0xc2,
+0x16,
+0xc0,
+0x00,
+0x19,
+0x46,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0c,
+0x8a,
+0xbf,
+0xf0,
+0x6c,
+0x00,
+0x01,
+0xa2,
+0x20,
+0x68,
+0x00,
+0x05,
+0x88,
+0xac,
+0x6c,
+0x00,
+0x01,
+0xac,
+0x21,
+0x6c,
+0x00,
+0x01,
+0x94,
+0x6c,
+0x42,
+0x09,
+0xf8,
+0x40,
+0x7a,
+0x84,
+0x87,
+0xa8,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x66,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0xaa,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x65,
+0x6c,
+0x00,
+0x01,
+0xac,
+0x20,
+0x42,
+0x03,
+0xf8,
+0x80,
+0x36,
+0x40,
+0x00,
+0x00,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x05,
+0x96,
+0xa0,
+0x88,
+0x03,
+0x66,
+0xc0,
+0x00,
+0x19,
+0x46,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0d,
+0x06,
+0xc0,
+0x00,
+0x1a,
+0x22,
+0x06,
+0x80,
+0x00,
+0x58,
+0x8a,
+0xc6,
+0xc0,
+0x00,
+0x1a,
+0xc2,
+0x16,
+0xc0,
+0x00,
+0x19,
+0x46,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x64,
+0x00,
+0x01,
+0x66,
+0xe7,
+0x68,
+0x00,
+0x00,
+0xce,
+0x20,
+0x68,
+0x20,
+0x01,
+0x48,
+0x21,
+0xab,
+0xfe,
+0x08,
+0x40,
+0x08,
+0x84,
+0x80,
+0x94,
+0x40,
+0x80,
+0x08,
+0x07,
+0x66,
+0x60,
+0x00,
+0x16,
+0xae,
+0x89,
+0x80,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x80,
+0xc8,
+0x68,
+0x20,
+0x01,
+0x48,
+0x20,
+0x68,
+0x00,
+0x00,
+0xd3,
+0x21,
+0x80,
+0x00,
+0x88,
+0x48,
+0x09,
+0x44,
+0x20,
+0x00,
+0x81,
+0x60,
+0x66,
+0x00,
+0x01,
+0x6a,
+0xe8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x9c,
+0x09,
+0x84,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x19,
+0xa0,
+0xa5,
+0x40,
+0xd6,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x0c,
+0xe2,
+0x05,
+0x44,
+0x17,
+0x08,
+0x08,
+0x96,
+0xc0,
+0x00,
+0x1a,
+0x60,
+0xb6,
+0xc0,
+0x00,
+0x1a,
+0x40,
+0x25,
+0x40,
+0x5f,
+0x80,
+0x04,
+0xa5,
+0x44,
+0x1f,
+0xa0,
+0x00,
+0x28,
+0x00,
+0x0a,
+0x68,
+0x00,
+0x00,
+0xd3,
+0x21,
+0x44,
+0x30,
+0x00,
+0x40,
+0xa0,
+0x80,
+0x84,
+0xb8,
+0x50,
+0xc0,
+0x84,
+0x04,
+0x0a,
+0x08,
+0x00,
+0x80,
+0x80,
+0x94,
+0x40,
+0x80,
+0x04,
+0x8a,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x0c,
+0x08,
+0x48,
+0x40,
+0x40,
+0x00,
+0x02,
+0x80,
+0x20,
+0x44,
+0x29,
+0x02,
+0xc0,
+0x20,
+0x68,
+0x20,
+0x01,
+0x4a,
+0x20,
+0x51,
+0x48,
+0x9b,
+0x3f,
+0xc4,
+0x80,
+0x00,
+0x29,
+0x80,
+0xc9,
+0x44,
+0x0d,
+0xc0,
+0x00,
+0x02,
+0x98,
+0x0c,
+0x84,
+0x40,
+0xdc,
+0x00,
+0x00,
+0x29,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0xc0,
+0x00,
+0x02,
+0x98,
+0x0c,
+0x84,
+0x40,
+0xdc,
+0x00,
+0x00,
+0x29,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0x40,
+0x00,
+0x03,
+0x98,
+0x08,
+0x84,
+0x40,
+0x90,
+0x04,
+0x00,
+0x85,
+0x14,
+0x69,
+0x18,
+0x08,
+0xa2,
+0xe0,
+0x9a,
+0x55,
+0x00,
+0xd1,
+0x80,
+0x89,
+0x1a,
+0x0d,
+0x99,
+0xa0,
+0x42,
+0x08,
+0x63,
+0x02,
+0x20,
+0xdb,
+0x57,
+0x06,
+0x99,
+0x80,
+0xc9,
+0x98,
+0x0c,
+0xa0,
+0x86,
+0x30,
+0x22,
+0x0d,
+0xb5,
+0x70,
+0x69,
+0x98,
+0x0c,
+0x99,
+0x80,
+0xca,
+0x08,
+0x63,
+0x02,
+0x20,
+0xdb,
+0x57,
+0x06,
+0x91,
+0x80,
+0xc9,
+0x98,
+0x08,
+0xa0,
+0x86,
+0x20,
+0x22,
+0x0d,
+0x25,
+0x70,
+0x51,
+0x3a,
+0x14,
+0x82,
+0x20,
+0x92,
+0x40,
+0x00,
+0x01,
+0x80,
+0x88,
+0x85,
+0x00,
+0x88,
+0x40,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc1,
+0x50,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0xbd,
+0x84,
+0x80,
+0x82,
+0xe1,
+0x30,
+0x98,
+0x00,
+0x88,
+0x40,
+0x48,
+0x00,
+0x00,
+0x08,
+0x50,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0xb4,
+0xba,
+0x14,
+0x88,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x02,
+0x81,
+0x30,
+0x98,
+0x00,
+0x88,
+0x40,
+0x48,
+0x00,
+0x00,
+0x08,
+0x50,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x12,
+0x84,
+0x04,
+0xab,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x14,
+0x07,
+0x20,
+0x5c,
+0x00,
+0x63,
+0x07,
+0xf6,
+0x5c,
+0x1f,
+0x30,
+0x40,
+0x4a,
+0xa0,
+0x0c,
+0x0a,
+0x04,
+0x21,
+0x84,
+0x04,
+0x84,
+0x60,
+0xa4,
+0x04,
+0x84,
+0xa8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x14,
+0x04,
+0x20,
+0x38,
+0x10,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x52,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x17,
+0x21,
+0x5c,
+0x90,
+0x8a,
+0xc0,
+0x08,
+0x80,
+0xaa,
+0x09,
+0x48,
+0x2e,
+0x32,
+0x23,
+0x0b,
+0xc1,
+0x4c,
+0x68,
+0x00,
+0x00,
+0x06,
+0xa1,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x46,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0xba,
+0x14,
+0x86,
+0xe0,
+0x00,
+0x00,
+0xd6,
+0x60,
+0x00,
+0x00,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0x46,
+0x0a,
+0x41,
+0x48,
+0xc0,
+0x84,
+0x86,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x02,
+0x21,
+0x68,
+0x00,
+0x00,
+0x18,
+0x20,
+0x84,
+0x87,
+0xaa,
+0xbf,
+0xf0,
+0x84,
+0xf0,
+0x85,
+0x90,
+0x10,
+0x20,
+0x63,
+0x29,
+0x8e,
+0x8a,
+0x95,
+0x06,
+0x68,
+0x80,
+0x76,
+0xbc,
+0x05,
+0x98,
+0x40,
+0x21,
+0x5c,
+0x00,
+0x63,
+0xc2,
+0xff,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x44,
+0x39,
+0x00,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x00,
+0x00,
+0x94,
+0x84,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x91,
+0x43,
+0x01,
+0x2e,
+0x80,
+0x26,
+0x10,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x0b,
+0x52,
+0xcd,
+0xc1,
+0x40,
+0x2d,
+0x54,
+0x09,
+0x43,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x01,
+0x40,
+0x60,
+0x5c,
+0x00,
+0x63,
+0xc1,
+0x5f,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x7a,
+0xa0,
+0x05,
+0x98,
+0x40,
+0x22,
+0xba,
+0x0a,
+0x88,
+0x80,
+0xe1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x91,
+0x40,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x61,
+0x8e,
+0x8a,
+0x94,
+0x04,
+0x6a,
+0x06,
+0x21,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x84,
+0x06,
+0x10,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x05,
+0x20,
+0x38,
+0x12,
+0xc8,
+0x40,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xc0,
+0x68,
+0x5c,
+0x00,
+0x73,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x4a,
+0x64,
+0x00,
+0x01,
+0x7a,
+0x07,
+0x5c,
+0xbd,
+0x03,
+0x00,
+0x4c,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x4a,
+0x9c,
+0x00,
+0x08,
+0x40,
+0x48,
+0x84,
+0x07,
+0xa4,
+0x60,
+0xa4,
+0x18,
+0xe8,
+0x86,
+0xe0,
+0x00,
+0x00,
+0xd6,
+0x40,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x17,
+0x20,
+0x00,
+0x00,
+0x0a,
+0x06,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x22,
+0x30,
+0xbc,
+0x15,
+0xc5,
+0xc8,
+0x04,
+0x04,
+0x02,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x09,
+0x56,
+0x20,
+0x00,
+0x00,
+0x04,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x40,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x02,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x0f,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0x66,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0x60,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x02,
+0xe6,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x47,
+0x20,
+0x5c,
+0x00,
+0x63,
+0x07,
+0xf6,
+0x5c,
+0x1f,
+0x30,
+0x40,
+0x4a,
+0xa0,
+0x0c,
+0x0a,
+0x04,
+0x21,
+0x84,
+0x04,
+0x84,
+0x60,
+0xa4,
+0x04,
+0x84,
+0xa8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x14,
+0x44,
+0x20,
+0x38,
+0x10,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x52,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x42,
+0x21,
+0x68,
+0x00,
+0x01,
+0x23,
+0x20,
+0x84,
+0x87,
+0xaa,
+0xbf,
+0xf0,
+0x84,
+0xf0,
+0x85,
+0x90,
+0x10,
+0x20,
+0x63,
+0x29,
+0x8e,
+0x8a,
+0x95,
+0x06,
+0x68,
+0x80,
+0x76,
+0xbc,
+0x05,
+0x98,
+0x40,
+0x21,
+0x5c,
+0x00,
+0x63,
+0xc2,
+0xff,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x44,
+0x39,
+0x00,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x80,
+0x00,
+0x94,
+0x84,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x91,
+0x43,
+0x01,
+0x2e,
+0x80,
+0x26,
+0x10,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x0b,
+0x52,
+0xcd,
+0xc1,
+0x40,
+0x2d,
+0x54,
+0x09,
+0x43,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x01,
+0x40,
+0x60,
+0x5c,
+0x00,
+0x63,
+0xc1,
+0x5f,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x68,
+0x00,
+0x01,
+0x0f,
+0x20,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x7a,
+0xa0,
+0x05,
+0x98,
+0x40,
+0x22,
+0xba,
+0x0a,
+0x88,
+0x80,
+0xe1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x91,
+0x40,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x61,
+0x8e,
+0x8a,
+0x94,
+0x04,
+0x6a,
+0x06,
+0x21,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x84,
+0x06,
+0x10,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x45,
+0x20,
+0x38,
+0x12,
+0xc8,
+0x40,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xc0,
+0x68,
+0x5c,
+0x00,
+0x73,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x4a,
+0x64,
+0x00,
+0x01,
+0x84,
+0x47,
+0x5c,
+0xbd,
+0x03,
+0x00,
+0x4c,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x4a,
+0x9c,
+0x00,
+0x08,
+0x40,
+0x48,
+0x84,
+0x07,
+0xa4,
+0x60,
+0xa4,
+0x18,
+0xe8,
+0x86,
+0xe0,
+0x00,
+0x22,
+0x36,
+0x40,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0x22,
+0x20,
+0x00,
+0x00,
+0x0a,
+0x06,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x22,
+0x30,
+0xbc,
+0x15,
+0xc5,
+0xc8,
+0x04,
+0x04,
+0x02,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x09,
+0x56,
+0x20,
+0x00,
+0x00,
+0x04,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x40,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x84,
+0x48,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x0f,
+0x6e,
+0x00,
+0x02,
+0x23,
+0x66,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x88,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x02,
+0x23,
+0x60,
+0x6c,
+0x70,
+0x28,
+0x84,
+0x48,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x24,
+0x46,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x06,
+0x1e,
+0x20,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x33,
+0x26,
+0x00,
+0x00,
+0x00,
+0xab,
+0xfd,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0x78,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x83,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x8e,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x8e,
+0x20,
+0x68,
+0x00,
+0x01,
+0x83,
+0x24,
+0xa0,
+0x04,
+0x06,
+0xc4,
+0x00,
+0x3a,
+0x60,
+0x88,
+0x40,
+0x09,
+0x44,
+0x20,
+0x02,
+0x20,
+0x44,
+0x68,
+0x00,
+0x01,
+0x78,
+0x21,
+0x86,
+0x00,
+0x98,
+0x80,
+0xe4,
+0xa0,
+0x84,
+0x46,
+0xc4,
+0x00,
+0x39,
+0xe0,
+0x84,
+0x42,
+0x10,
+0x06,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x39,
+0x60,
+0x94,
+0x44,
+0x80,
+0x1a,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3a,
+0xc0,
+0x94,
+0x60,
+0x88,
+0x88,
+0x16,
+0x44,
+0x40,
+0x88,
+0x18,
+0x00,
+0x84,
+0xd0,
+0xd0,
+0x88,
+0x1e,
+0x02,
+0x2c,
+0x40,
+0x51,
+0x62,
+0x80,
+0x82,
+0x4c,
+0x51,
+0x62,
+0x41,
+0x83,
+0x09,
+0x84,
+0x04,
+0xc8,
+0x82,
+0xc9,
+0x68,
+0x20,
+0x01,
+0xf8,
+0x24,
+0x66,
+0x00,
+0x01,
+0xa0,
+0xc8,
+0x46,
+0x08,
+0x08,
+0x82,
+0x0a,
+0x88,
+0x20,
+0x98,
+0x82,
+0x48,
+0x68,
+0x20,
+0x01,
+0xfb,
+0x24,
+0x66,
+0x00,
+0x01,
+0xa0,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x82,
+0x8a,
+0x5c,
+0x82,
+0x00,
+0x81,
+0x20,
+0x88,
+0x20,
+0x98,
+0x80,
+0xa4,
+0x80,
+0x04,
+0x98,
+0x60,
+0x48,
+0x40,
+0x00,
+0x00,
+0x82,
+0x60,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x56,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x82,
+0x00,
+0x80,
+0xa0,
+0x88,
+0x22,
+0x48,
+0x81,
+0x21,
+0x80,
+0x00,
+0x98,
+0x61,
+0xc8,
+0x88,
+0x16,
+0x08,
+0x49,
+0x48,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x62,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x82,
+0x00,
+0x81,
+0xa0,
+0x88,
+0x12,
+0x48,
+0x80,
+0xa1,
+0x80,
+0x00,
+0x98,
+0x61,
+0xc8,
+0x88,
+0x0e,
+0x08,
+0x49,
+0x48,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x6e,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x81,
+0x00,
+0x82,
+0x21,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x88,
+0x1a,
+0x08,
+0x80,
+0xa2,
+0x80,
+0x80,
+0x9a,
+0x20,
+0x64,
+0x84,
+0x14,
+0x88,
+0x51,
+0xc8,
+0x88,
+0x1e,
+0x18,
+0x82,
+0x64,
+0x68,
+0x00,
+0x01,
+0x56,
+0x23,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x40,
+0x00,
+0x02,
+0x18,
+0x40,
+0x5c,
+0x83,
+0x08,
+0x81,
+0xa1,
+0x5c,
+0x88,
+0x00,
+0x81,
+0x22,
+0x5c,
+0x81,
+0x08,
+0x08,
+0xc8,
+0x80,
+0xa4,
+0x88,
+0x81,
+0x61,
+0x68,
+0x00,
+0x01,
+0x62,
+0x20,
+0x81,
+0x08,
+0x98,
+0x82,
+0x24,
+0x88,
+0x1e,
+0x26,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x00,
+0x40,
+0x5c,
+0x83,
+0x08,
+0x81,
+0xa1,
+0x5c,
+0x81,
+0x00,
+0x80,
+0xa2,
+0x5c,
+0x88,
+0x08,
+0x08,
+0xc8,
+0x80,
+0xac,
+0x88,
+0x80,
+0xe1,
+0x68,
+0x00,
+0x01,
+0x6e,
+0x20,
+0x81,
+0x00,
+0x98,
+0x82,
+0x24,
+0x88,
+0x1e,
+0x26,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x00,
+0x40,
+0x5c,
+0x83,
+0x00,
+0x81,
+0xa4,
+0x5c,
+0x88,
+0x08,
+0x81,
+0x20,
+0x82,
+0x04,
+0x88,
+0x22,
+0xc8,
+0x88,
+0x1e,
+0x46,
+0x60,
+0x00,
+0x1a,
+0x14,
+0x08,
+0x81,
+0x24,
+0x88,
+0x0a,
+0x08,
+0x63,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa1,
+0x40,
+0x88,
+0x0a,
+0x48,
+0x81,
+0xa0,
+0x40,
+0x00,
+0x00,
+0x63,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x06,
+0x4d,
+0x24,
+0x88,
+0x1a,
+0x08,
+0x80,
+0x36,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x64,
+0xba,
+0x14,
+0x88,
+0x43,
+0x48,
+0x40,
+0x00,
+0x02,
+0x80,
+0x30,
+0x68,
+0x00,
+0x01,
+0x89,
+0x24,
+0xab,
+0xfc,
+0x06,
+0xc0,
+0x00,
+0x11,
+0x40,
+0xb6,
+0xc0,
+0x00,
+0x12,
+0x60,
+0x06,
+0xc0,
+0x00,
+0x29,
+0x40,
+0x16,
+0xc0,
+0x00,
+0x32,
+0x80,
+0xa8,
+0x60,
+0x09,
+0x88,
+0x05,
+0x18,
+0x80,
+0xd0,
+0x88,
+0x14,
+0xb8,
+0x81,
+0xf6,
+0x68,
+0x00,
+0x01,
+0x7e,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa7,
+0x28,
+0x40,
+0x00,
+0x00,
+0x40,
+0x08,
+0x68,
+0x00,
+0x01,
+0x89,
+0x21,
+0x68,
+0x20,
+0x01,
+0xdd,
+0x20,
+0x68,
+0x00,
+0x01,
+0x7e,
+0x24,
+0x5c,
+0x83,
+0x00,
+0x40,
+0x88,
+0x80,
+0xa0,
+0xb5,
+0x70,
+0x9c,
+0x02,
+0x20,
+0x98,
+0x40,
+0x08,
+0x88,
+0x36,
+0x45,
+0x70,
+0x94,
+0x08,
+0x24,
+0x08,
+0x82,
+0xe1,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0x54,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x98,
+0x00,
+0x98,
+0x83,
+0x20,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x84,
+0x04,
+0x86,
+0x80,
+0x00,
+0x16,
+0x02,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x88,
+0x82,
+0x09,
+0x55,
+0x01,
+0x28,
+0x83,
+0x20,
+0x5c,
+0x83,
+0x00,
+0x82,
+0xa4,
+0x80,
+0x20,
+0x88,
+0x60,
+0x49,
+0x40,
+0x00,
+0x00,
+0x82,
+0x60,
+0x68,
+0x00,
+0x01,
+0x54,
+0x21,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc1,
+0x22,
+0x66,
+0x00,
+0x01,
+0xa1,
+0xa0,
+0x5c,
+0x83,
+0x00,
+0x82,
+0xa0,
+0x68,
+0x00,
+0x01,
+0x60,
+0x21,
+0x80,
+0x20,
+0x88,
+0x82,
+0xe0,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc3,
+0x22,
+0x66,
+0x00,
+0x01,
+0xa1,
+0xa0,
+0x88,
+0x10,
+0x83,
+0x69,
+0x00,
+0x98,
+0x00,
+0x96,
+0xc4,
+0x00,
+0x3a,
+0x80,
+0xb5,
+0x80,
+0xf4,
+0x18,
+0xe8,
+0x9b,
+0xc0,
+0x3a,
+0x55,
+0x01,
+0x70,
+0x80,
+0x88,
+0xb0,
+0x00,
+0xd3,
+0x69,
+0x00,
+0x98,
+0x00,
+0x03,
+0x01,
+0xc0,
+0xbc,
+0x03,
+0xa5,
+0x50,
+0x18,
+0x08,
+0x00,
+0x8b,
+0x00,
+0x08,
+0x36,
+0x90,
+0x19,
+0x80,
+0x48,
+0x30,
+0x1e,
+0x0b,
+0xc0,
+0x12,
+0xb0,
+0x00,
+0xe3,
+0x20,
+0x28,
+0xbc,
+0x05,
+0x98,
+0x82,
+0x20,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x21,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x60,
+0x6c,
+0x40,
+0x03,
+0xaa,
+0x08,
+0x6c,
+0x00,
+0x03,
+0x94,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x96,
+0x48,
+0xa0,
+0x0c,
+0x08,
+0x40,
+0x09,
+0x88,
+0x06,
+0x06,
+0x82,
+0x00,
+0x0a,
+0xa2,
+0x46,
+0x80,
+0x00,
+0x1c,
+0x62,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x08,
+0x82,
+0xa0,
+0x88,
+0x0c,
+0x8a,
+0x00,
+0xc0,
+0x84,
+0x00,
+0x98,
+0x81,
+0x60,
+0x68,
+0x20,
+0x00,
+0xc2,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc8,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x66,
+0x00,
+0x01,
+0xaa,
+0x88,
+0x55,
+0x01,
+0x28,
+0x80,
+0x88,
+0x6c,
+0x00,
+0x03,
+0x94,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x68,
+0x00,
+0x01,
+0x81,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x42,
+0x02,
+0xb8,
+0x40,
+0xc8,
+0x2a,
+0x7e,
+0x46,
+0xc0,
+0x00,
+0x39,
+0x44,
+0x86,
+0xc0,
+0x00,
+0x30,
+0x47,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x96,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x68,
+0x00,
+0x01,
+0x8c,
+0x20,
+0x00,
+0x00,
+0x04,
+0x20,
+0x37,
+0x84,
+0x00,
+0x88,
+0x40,
+0xc8,
+0x2a,
+0x7e,
+0x46,
+0xc0,
+0x00,
+0x39,
+0x64,
+0x86,
+0xc0,
+0x00,
+0x31,
+0xa7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x00,
+0x80,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x04,
+0x7a,
+0x6c,
+0x00,
+0x03,
+0x1a,
+0x7a,
+0x66,
+0x00,
+0x01,
+0xa6,
+0x68,
+0x84,
+0x20,
+0x98,
+0x81,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x46,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa6,
+0x68,
+0x40,
+0x00,
+0x00,
+0x42,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x4a,
+0x48,
+0x68,
+0x00,
+0x06,
+0x80,
+0x20,
+0x88,
+0x1b,
+0x6b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x40,
+0x68,
+0x00,
+0x06,
+0x1c,
+0xa0,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x33,
+0x26,
+0x00,
+0x00,
+0x00,
+0x84,
+0x00,
+0x88,
+0x40,
+0x8a,
+0x57,
+0x0d,
+0x03,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x00,
+0x00,
+0x00,
+0xa2,
+0x02,
+0x48,
+0x60,
+0x08,
+0x44,
+0x20,
+0x00,
+0x60,
+0x89,
+0xba,
+0x14,
+0x80,
+0x89,
+0x80,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0xa0,
+0x04,
+0x08,
+0x41,
+0x88,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x0a,
+0x2e,
+0x13,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x81,
+0x02,
+0x22,
+0x23,
+0x81,
+0x80,
+0x94,
+0x40,
+0x80,
+0x05,
+0x80,
+0x95,
+0x00,
+0xa0,
+0x20,
+0x8c,
+0x19,
+0x80,
+0x09,
+0x80,
+0x84,
+0x9a,
+0x1c,
+0x64,
+0x82,
+0x00,
+0x84,
+0x42,
+0x00,
+0x06,
+0x00,
+0x98,
+0x48,
+0x08,
+0x44,
+0x0c,
+0x02,
+0xbf,
+0xd0,
+0x98,
+0x00,
+0x9a,
+0x24,
+0xe4,
+0x80,
+0x84,
+0x98,
+0x80,
+0x61,
+0x88,
+0x0e,
+0x48,
+0x81,
+0x62,
+0x88,
+0x1e,
+0x08,
+0x82,
+0x76,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0xa0,
+0x80,
+0x08,
+0x80,
+0x20,
+0x88,
+0x0a,
+0x2a,
+0x00,
+0x40,
+0x88,
+0x06,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x85,
+0x50,
+0x12,
+0xa1,
+0x06,
+0x48,
+0x80,
+0x20,
+0x88,
+0x12,
+0x2a,
+0x04,
+0x60,
+0x84,
+0x00,
+0x95,
+0x70,
+0x94,
+0x05,
+0x08,
+0x95,
+0x70,
+0xa0,
+0x08,
+0x1a,
+0x09,
+0x80,
+0x08,
+0x59,
+0x01,
+0x00,
+0x44,
+0xc8,
+0xbc,
+0x0e,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x17,
+0x38,
+0x50,
+0x08,
+0x2e,
+0x12,
+0xd5,
+0xb4,
+0xa0,
+0x05,
+0x0c,
+0x99,
+0x80,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x22,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0xe5,
+0x36,
+0x10,
+0x58,
+0x50,
+0xc9,
+0xbc,
+0x0b,
+0x78,
+0x50,
+0x08,
+0x28,
+0x12,
+0xd5,
+0xb4,
+0xa0,
+0x05,
+0x0c,
+0x99,
+0x80,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x22,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x25,
+0x85,
+0x0c,
+0x80,
+0x00,
+0x00,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0xc0,
+0x08,
+0x44,
+0x20,
+0x03,
+0xa1,
+0x11,
+0x98,
+0x00,
+0x96,
+0x80,
+0x39,
+0x4b,
+0xb8,
+0x84,
+0x42,
+0x00,
+0x3a,
+0x14,
+0x82,
+0x2b,
+0x80,
+0x46,
+0x08,
+0x09,
+0x83,
+0x08,
+0x55,
+0x01,
+0x12,
+0xbf,
+0xf0,
+0x5b,
+0x0c,
+0x30,
+0x80,
+0x4a,
+0x44,
+0x58,
+0x81,
+0x84,
+0x48,
+0x44,
+0x21,
+0x81,
+0x80,
+0x41,
+0x5b,
+0x04,
+0x32,
+0xc0,
+0x20,
+0x44,
+0x40,
+0x81,
+0x84,
+0x48,
+0x5b,
+0x0a,
+0x29,
+0x84,
+0x8a,
+0x44,
+0x58,
+0x01,
+0x84,
+0x0b,
+0x44,
+0x39,
+0x00,
+0x80,
+0x09,
+0x44,
+0x39,
+0x81,
+0x80,
+0xca,
+0x5c,
+0x00,
+0xeb,
+0xa1,
+0x11,
+0x2a,
+0x02,
+0x74,
+0x46,
+0x88,
+0x1a,
+0x14,
+0x14,
+0x46,
+0x88,
+0x18,
+0x34,
+0x34,
+0xc0,
+0xc0,
+0x9a,
+0x00,
+0x09,
+0x83,
+0x8b,
+0x44,
+0x08,
+0x01,
+0x83,
+0x08,
+0x54,
+0x07,
+0xd1,
+0x83,
+0x41,
+0x54,
+0x08,
+0x41,
+0x83,
+0x0b,
+0x44,
+0x48,
+0x83,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x03,
+0xae,
+0x08,
+0x9a,
+0x00,
+0x24,
+0xc2,
+0x40,
+0x18,
+0x34,
+0x96,
+0x82,
+0x00,
+0x1d,
+0xd2,
+0x02,
+0x81,
+0x79,
+0x80,
+0x04,
+0x29,
+0x80,
+0x4b,
+0x44,
+0x60,
+0x00,
+0x00,
+0x40,
+0xba,
+0x14,
+0x88,
+0x40,
+0x40,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x38,
+0x13,
+0x22,
+0x58,
+0xb0,
+0xbc,
+0x05,
+0x0b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0xcc,
+0x21,
+0x68,
+0x20,
+0x01,
+0xe1,
+0x24,
+0x5c,
+0x81,
+0x08,
+0x48,
+0x02,
+0x5c,
+0x82,
+0x00,
+0x20,
+0x8a,
+0x68,
+0x00,
+0x01,
+0xd0,
+0x20,
+0x30,
+0x19,
+0x04,
+0x20,
+0xec,
+0x00,
+0x00,
+0x08,
+0x02,
+0x8a,
+0x68,
+0x00,
+0x01,
+0xcc,
+0x20,
+0x55,
+0x02,
+0x92,
+0xc0,
+0xa0,
+0x58,
+0x01,
+0x00,
+0x00,
+0x52,
+0xbc,
+0x03,
+0x56,
+0xc0,
+0x00,
+0x3a,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0x23,
+0x00,
+0xa0,
+0xbc,
+0x02,
+0x36,
+0xc0,
+0x00,
+0x3a,
+0x24,
+0x83,
+0x01,
+0xa8,
+0xbc,
+0x03,
+0x56,
+0xc0,
+0x00,
+0x3a,
+0x44,
+0x90,
+0x00,
+0x00,
+0x84,
+0x10,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x04,
+0x3b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xa6,
+0x49,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x80,
+0x00,
+0xb5,
+0x70,
+0xe1,
+0x06,
+0x00,
+0x35,
+0x84,
+0x68,
+0x00,
+0x28,
+0x94,
+0x20,
+0x1d,
+0x98,
+0xe8,
+0x85,
+0x70,
+0xb8,
+0x98,
+0x20,
+0x2b,
+0x00,
+0x0c,
+0x30,
+0x8c,
+0x8b,
+0xc0,
+0x23,
+0x40,
+0x00,
+0x03,
+0x00,
+0x0a,
+0x5b,
+0x4e,
+0x08,
+0x60,
+0x83,
+0x36,
+0x80,
+0x05,
+0xb4,
+0xc0,
+0x18,
+0x00,
+0xb5,
+0xb4,
+0xa0,
+0x98,
+0x04,
+0xa5,
+0x78,
+0xdf,
+0x98,
+0x00,
+0x95,
+0x84,
+0x7c,
+0x18,
+0x04,
+0xab,
+0xc3,
+0x0b,
+0x2f,
+0x1a,
+0xd3,
+0x08,
+0xe8,
+0xbc,
+0x2d,
+0x33,
+0x20,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0xbc,
+0x04,
+0x86,
+0xc4,
+0x00,
+0x3c,
+0xc0,
+0x83,
+0x20,
+0x10,
+0xbc,
+0x08,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x0f,
+0x56,
+0xc4,
+0x00,
+0x3c,
+0xa0,
+0xa2,
+0xe1,
+0xad,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x49,
+0xbc,
+0x09,
+0x73,
+0x01,
+0x28,
+0xbc,
+0x07,
+0x36,
+0xc4,
+0x00,
+0x3c,
+0x80,
+0xa2,
+0x81,
+0xad,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x49,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0x32,
+0x06,
+0x8b,
+0xc0,
+0x63,
+0x68,
+0x00,
+0x01,
+0xce,
+0x21,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x7a,
+0x84,
+0x8f,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x6a,
+0x38,
+0x00,
+0xd6,
+0x80,
+0x00,
+0x1c,
+0xe2,
+0x16,
+0xc0,
+0x00,
+0x39,
+0xc4,
+0x98,
+0x48,
+0xc8,
+0x68,
+0x20,
+0x00,
+0xfc,
+0x21,
+0x39,
+0x0a,
+0x18,
+0x48,
+0x08,
+0x80,
+0x24,
+0x88,
+0x40,
+0x48,
+0xa0,
+0x06,
+0x08,
+0x48,
+0x88,
+0x80,
+0x24,
+0x84,
+0x60,
+0xa4,
+0x00,
+0x2c,
+0x88,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0x68,
+0x00,
+0x01,
+0x54,
+0x20,
+0x68,
+0x00,
+0x01,
+0x60,
+0x21,
+0x68,
+0x00,
+0x01,
+0x6c,
+0x22,
+0x55,
+0x01,
+0x33,
+0x80,
+0x00,
+0x55,
+0x03,
+0xb0,
+0x00,
+0x7a,
+0x5d,
+0x0c,
+0x30,
+0x08,
+0x7a,
+0x32,
+0x2b,
+0x0b,
+0xff,
+0xaa,
+0x40,
+0x00,
+0x00,
+0x10,
+0x7a,
+0x68,
+0x00,
+0x01,
+0x85,
+0x22,
+0x68,
+0x00,
+0x01,
+0x90,
+0x21,
+0x68,
+0x00,
+0x01,
+0x7a,
+0x20,
+0x55,
+0x03,
+0x20,
+0x10,
+0x7a,
+0x5d,
+0x08,
+0x20,
+0x08,
+0x7a,
+0x32,
+0x26,
+0x0b,
+0xff,
+0xaa,
+0x80,
+0x07,
+0xa6,
+0x80,
+0x00,
+0x1c,
+0xc2,
+0x06,
+0x82,
+0x00,
+0x0f,
+0xc2,
+0x15,
+0xc8,
+0x20,
+0x80,
+0x07,
+0xa5,
+0xc0,
+0x16,
+0x04,
+0x0f,
+0xa8,
+0x00,
+0x7a,
+0xa0,
+0x08,
+0x08,
+0x48,
+0x0a,
+0x80,
+0x2c,
+0xa8,
+0x40,
+0x4a,
+0xa0,
+0x06,
+0x08,
+0x48,
+0x8a,
+0x80,
+0x2c,
+0xa6,
+0x80,
+0x00,
+0x1c,
+0xa2,
+0x16,
+0x80,
+0x00,
+0x61,
+0xca,
+0xc6,
+0xc4,
+0x00,
+0x3a,
+0xa0,
+0x08,
+0x02,
+0xca,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x6c,
+0x84,
+0x85,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x04,
+0x88,
+0x48,
+0xd0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xa0,
+0x05,
+0x89,
+0x8e,
+0x88,
+0x94,
+0x06,
+0x4a,
+0x00,
+0x08,
+0xa0,
+0x22,
+0x18,
+0x48,
+0x60,
+0xa0,
+0xe3,
+0x09,
+0x40,
+0x64,
+0xa0,
+0x01,
+0x0a,
+0x02,
+0x01,
+0x84,
+0x86,
+0x0a,
+0x08,
+0x80,
+0x84,
+0x07,
+0xaa,
+0x06,
+0xa0,
+0x94,
+0x06,
+0x4a,
+0x04,
+0x40,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x84,
+0x0e,
+0xc6,
+0x80,
+0x00,
+0x6e,
+0x2a,
+0xcb,
+0xa1,
+0x48,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x94,
+0x5a,
+0x83,
+0x20,
+0x00,
+0xbc,
+0x0a,
+0x59,
+0x46,
+0x28,
+0x22,
+0x90,
+0x42,
+0xab,
+0xe0,
+0x23,
+0x08,
+0x06,
+0x82,
+0x00,
+0x1f,
+0xe2,
+0x19,
+0x84,
+0x28,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x21,
+0xba,
+0x01,
+0x0b,
+0xa1,
+0x40,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0xa0,
+0x05,
+0x09,
+0x40,
+0x64,
+0xa0,
+0x01,
+0x0a,
+0x02,
+0x01,
+0x84,
+0x86,
+0x0a,
+0x0e,
+0x60,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x20,
+0x64,
+0x00,
+0x01,
+0x84,
+0x47,
+0x64,
+0x00,
+0x01,
+0x71,
+0x87,
+0x39,
+0x00,
+0x89,
+0x48,
+0x0c,
+0x51,
+0xb1,
+0x21,
+0x48,
+0x0e,
+0x51,
+0xa1,
+0xb1,
+0x48,
+0x08,
+0x51,
+0x90,
+0x11,
+0x48,
+0x28,
+0x29,
+0x1a,
+0x42,
+0x90,
+0xa4,
+0x54,
+0x81,
+0x23,
+0xa1,
+0x48,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x80,
+0x40,
+0x40,
+0x00,
+0x51,
+0xf0,
+0x23,
+0x07,
+0xfa,
+0x28,
+0x8a,
+0x15,
+0x1e,
+0x02,
+0x14,
+0x84,
+0x12,
+0x88,
+0xa1,
+0x51,
+0xd0,
+0x21,
+0x48,
+0x41,
+0x28,
+0x8a,
+0x15,
+0x44,
+0x40,
+0x14,
+0x84,
+0x1b,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x00,
+0x00,
+0x00,
+0x94,
+0x82,
+0xc5,
+0xd4,
+0x83,
+0x14,
+0x8a,
+0xc2,
+0x32,
+0x36,
+0x3a,
+0x90,
+0x43,
+0xa9,
+0x86,
+0x54,
+0x89,
+0xa3,
+0xa1,
+0x48,
+0x94,
+0x07,
+0x40,
+0x00,
+0x00,
+0x94,
+0x03,
+0x85,
+0x15,
+0x02,
+0x30,
+0x7f,
+0xa2,
+0x88,
+0xa1,
+0x54,
+0x44,
+0x01,
+0x48,
+0x61,
+0xba,
+0x14,
+0x89,
+0x48,
+0xe0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x80,
+0x4a,
+0x02,
+0xc0,
+0x5c,
+0x80,
+0x92,
+0xbf,
+0xf0,
+0x62,
+0x00,
+0x00,
+0x00,
+0xb4,
+0x5c,
+0x81,
+0x02,
+0x40,
+0x02,
+0x5c,
+0x00,
+0x00,
+0x40,
+0x24,
+0x60,
+0x00,
+0x00,
+0x00,
+0x35,
+0x8d,
+0x05,
+0x80,
+0x00,
+0x00,
+0x94,
+0x88,
+0xc9,
+0x52,
+0xc4,
+0x00,
+0x00,
+0x0a,
+0x10,
+0x0a,
+0x8d,
+0x03,
+0x2b,
+0xb1,
+0x82,
+0xba,
+0x14,
+0x88,
+0x40,
+0x64,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x5c,
+0x80,
+0x52,
+0xbf,
+0xf0,
+0xa0,
+0x2c,
+0x0a,
+0x40,
+0x02,
+0x62,
+0x00,
+0x00,
+0x00,
+0xa4,
+0x5c,
+0x81,
+0x08,
+0x40,
+0x24,
+0x5c,
+0x80,
+0x82,
+0x06,
+0x61,
+0xbb,
+0x00,
+0x06,
+0x00,
+0x00,
+0x00,
+0x03,
+0x58,
+0xd0,
+0xd8,
+0x00,
+0x00,
+0x09,
+0x53,
+0x0c,
+0x94,
+0x94,
+0x40,
+0x00,
+0x00,
+0xa1,
+0x00,
+0xab,
+0xa1,
+0x48,
+0x84,
+0x06,
+0x4a,
+0x80,
+0x10,
+0xa0,
+0x06,
+0x19,
+0x48,
+0x28,
+0x55,
+0x5e,
+0x02,
+0x08,
+0x0a,
+0x51,
+0x90,
+0x01,
+0x50,
+0x2a,
+0x54,
+0x04,
+0x22,
+0x14,
+0x13,
+0x95,
+0x82,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x04,
+0x20,
+0xcc,
+0x08,
+0x07,
+0x69,
+0x82,
+0x22,
+0x32,
+0x0c,
+0x0b,
+0xc1,
+0x38,
+0xa0,
+0x81,
+0x02,
+0xa7,
+0x80,
+0x22,
+0x88,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x1b,
+0x56,
+0x20,
+0x00,
+0x00,
+0x08,
+0x6a,
+0x00,
+0x01,
+0x00,
+0x00,
+0x09,
+0x82,
+0x20,
+0x88,
+0x16,
+0x18,
+0x80,
+0xc8,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x18,
+0x80,
+0x80,
+0x55,
+0x08,
+0x22,
+0x08,
+0x21,
+0xbc,
+0x0d,
+0x79,
+0x49,
+0x28,
+0x42,
+0x05,
+0x39,
+0x50,
+0x60,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x8a,
+0x10,
+0x00,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x20,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x94,
+0x5a,
+0xc5,
+0x90,
+0x30,
+0x20,
+0x2c,
+0x04,
+0x20,
+0x34,
+0x2b,
+0xff,
+0x08,
+0x80,
+0x76,
+0xa0,
+0x65,
+0x98,
+0x80,
+0x36,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x01,
+0x0a,
+0x06,
+0x61,
+0x88,
+0x0e,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x08,
+0x80,
+0xa0,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x10,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x04,
+0x60,
+0x94,
+0x5a,
+0xc3,
+0x20,
+0x60,
+0xbc,
+0x16,
+0x8a,
+0xbf,
+0xf0,
+0x55,
+0x3f,
+0x23,
+0x00,
+0xfe,
+0x5b,
+0x48,
+0x13,
+0x00,
+0x28,
+0x37,
+0x08,
+0x52,
+0xe1,
+0x76,
+0x62,
+0x00,
+0x00,
+0x00,
+0x26,
+0x36,
+0x80,
+0x05,
+0x04,
+0xc9,
+0x18,
+0xeb,
+0x52,
+0xf8,
+0x12,
+0x2b,
+0x96,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x02,
+0xb9,
+0x83,
+0x88,
+0x36,
+0x10,
+0x4a,
+0x00,
+0x69,
+0x64,
+0x00,
+0x01,
+0xbd,
+0xef,
+0xa8,
+0x01,
+0x08,
+0x80,
+0x76,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xf6,
+0x83,
+0x80,
+0x64,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x1e,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x10,
+0xa0,
+0x06,
+0x19,
+0x48,
+0x28,
+0x55,
+0x5e,
+0x12,
+0x08,
+0x0a,
+0x51,
+0x90,
+0x91,
+0x50,
+0x28,
+0x68,
+0x20,
+0x00,
+0x00,
+0x2c,
+0x54,
+0x00,
+0xa1,
+0x8b,
+0x00,
+0x54,
+0x01,
+0x22,
+0x14,
+0x12,
+0x95,
+0x02,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x04,
+0x20,
+0xcc,
+0x08,
+0x07,
+0x69,
+0x82,
+0x22,
+0x32,
+0x0c,
+0x0b,
+0xc1,
+0x38,
+0xa0,
+0x81,
+0x02,
+0xa7,
+0x80,
+0x22,
+0x88,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x1b,
+0x56,
+0x20,
+0x00,
+0x00,
+0x08,
+0x6a,
+0x00,
+0x01,
+0x00,
+0x00,
+0x09,
+0x82,
+0x20,
+0x88,
+0x16,
+0x18,
+0x80,
+0xc8,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x18,
+0x80,
+0x80,
+0x55,
+0x08,
+0x22,
+0x08,
+0x21,
+0xbc,
+0x0d,
+0x79,
+0x49,
+0x28,
+0x42,
+0x05,
+0x39,
+0x50,
+0x60,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x8a,
+0x10,
+0x00,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x20,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x94,
+0x5a,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x08,
+0x80,
+0x76,
+0x42,
+0x10,
+0xc2,
+0x00,
+0x01,
+0xa0,
+0x2a,
+0x03,
+0x21,
+0x80,
+0xbc,
+0x19,
+0x8a,
+0x08,
+0x71,
+0x2a,
+0x78,
+0x02,
+0x28,
+0xc4,
+0x32,
+0x02,
+0x0b,
+0xc2,
+0x45,
+0x62,
+0x00,
+0x00,
+0x00,
+0xf4,
+0xa0,
+0xa3,
+0x08,
+0x80,
+0xe0,
+0x88,
+0x16,
+0x10,
+0x00,
+0x00,
+0x88,
+0x0a,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x08,
+0x81,
+0x21,
+0x88,
+0x0a,
+0x0a,
+0x08,
+0x21,
+0x84,
+0x02,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x02,
+0x1b,
+0xc1,
+0x07,
+0xa0,
+0x63,
+0x18,
+0x80,
+0x36,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x02,
+0x0a,
+0x06,
+0x41,
+0x88,
+0x0e,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x08,
+0x80,
+0xa0,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x20,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x04,
+0x60,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x05,
+0x9a,
+0xbf,
+0xf0,
+0x94,
+0x82,
+0xc5,
+0x90,
+0x50,
+0x20,
+0xa4,
+0xa8,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x64,
+0x20,
+0x3c,
+0x20,
+0x06,
+0x18,
+0x50,
+0x20,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xb6,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x0b,
+0x68,
+0x80,
+0x20,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x10,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x19,
+0x46,
+0x2c,
+0xb0,
+0x70,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x02,
+0x16,
+0x40,
+0x00,
+0x1d,
+0x0c,
+0x7b,
+0xa1,
+0x40,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xae,
+0x59,
+0x01,
+0x82,
+0xbf,
+0xd0,
+0xbd,
+0x7b,
+0x88,
+0x80,
+0x76,
+0x32,
+0x07,
+0x0b,
+0xd6,
+0x20,
+0x32,
+0x0b,
+0x0b,
+0xd5,
+0x40,
+0x32,
+0x0f,
+0x0b,
+0xd3,
+0x40,
+0x32,
+0x13,
+0x0b,
+0xd2,
+0x60,
+0x32,
+0x17,
+0x0b,
+0xd0,
+0x60,
+0x32,
+0x1b,
+0x0b,
+0xcf,
+0x80,
+0x32,
+0x1f,
+0x0b,
+0xce,
+0x00,
+0x32,
+0x23,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x00,
+0xf9,
+0x8e,
+0x8b,
+0x32,
+0x03,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x82,
+0x8b,
+0xcb,
+0x68,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0xac,
+0x32,
+0x27,
+0x0b,
+0xcb,
+0x20,
+0x32,
+0x03,
+0x8b,
+0xc8,
+0x20,
+0x32,
+0x2b,
+0x0b,
+0xc8,
+0x00,
+0x32,
+0x03,
+0x8b,
+0xc4,
+0xf0,
+0x32,
+0x2f,
+0x0b,
+0xc4,
+0xd0,
+0x32,
+0x03,
+0x8b,
+0xc0,
+0x20,
+0x32,
+0x33,
+0x0b,
+0xd5,
+0xf1,
+0x59,
+0x05,
+0x02,
+0xc0,
+0x20,
+0xbc,
+0x2c,
+0x86,
+0x80,
+0x00,
+0x09,
+0x82,
+0x13,
+0x21,
+0xa0,
+0xbd,
+0x58,
+0x9a,
+0x08,
+0x00,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0x66,
+0x00,
+0x01,
+0xbc,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x98,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0xa0,
+0x02,
+0x08,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x01,
+0xbc,
+0xa8,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xa0,
+0x6c,
+0x00,
+0x01,
+0x30,
+0x0b,
+0x51,
+0xa1,
+0xf8,
+0x40,
+0x08,
+0x23,
+0x42,
+0x45,
+0x90,
+0x1c,
+0x04,
+0x04,
+0x86,
+0xc0,
+0x00,
+0x13,
+0x04,
+0xb4,
+0x20,
+0x34,
+0xa0,
+0x16,
+0x85,
+0xc0,
+0x97,
+0x14,
+0x02,
+0xf6,
+0xc0,
+0x00,
+0x13,
+0x20,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x04,
+0x02,
+0x41,
+0xb8,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xbd,
+0x34,
+0x72,
+0x49,
+0xb8,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xbd,
+0x30,
+0x7a,
+0x40,
+0x80,
+0x80,
+0x80,
+0xa5,
+0x16,
+0x1a,
+0x08,
+0x0e,
+0x19,
+0x40,
+0x74,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x0a,
+0x1a,
+0x40,
+0x80,
+0x84,
+0x80,
+0xa2,
+0x2c,
+0x34,
+0x40,
+0x00,
+0x01,
+0x40,
+0x74,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x70,
+0xbd,
+0x16,
+0x73,
+0x20,
+0xa0,
+0xbc,
+0x21,
+0x03,
+0x21,
+0xa0,
+0xbd,
+0x12,
+0x93,
+0x81,
+0xc6,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x0b,
+0x25,
+0x9b,
+0x8b,
+0xc0,
+0x69,
+0x38,
+0x11,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0xc8,
+0x02,
+0xc2,
+0xc0,
+0xbc,
+0x06,
+0x96,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x00,
+0x00,
+0x02,
+0x41,
+0xbe,
+0xa0,
+0x67,
+0x18,
+0x02,
+0x4a,
+0x68,
+0x00,
+0x08,
+0x14,
+0x22,
+0x84,
+0x06,
+0x26,
+0x82,
+0x00,
+0x22,
+0x02,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xcf,
+0x37,
+0x68,
+0x20,
+0x02,
+0x20,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xce,
+0x77,
+0x32,
+0x0a,
+0x0b,
+0xc2,
+0x00,
+0x32,
+0x1a,
+0x0b,
+0xce,
+0x39,
+0x38,
+0x1c,
+0x66,
+0xc0,
+0x00,
+0x03,
+0x60,
+0xb2,
+0x59,
+0xb8,
+0xbc,
+0x05,
+0x93,
+0x81,
+0x14,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x52,
+0xc8,
+0x02,
+0xc2,
+0xc0,
+0xbc,
+0x06,
+0x96,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x00,
+0x00,
+0x02,
+0x41,
+0xbe,
+0xa0,
+0x67,
+0x18,
+0x02,
+0x4a,
+0x68,
+0x00,
+0x08,
+0x01,
+0x22,
+0x84,
+0x06,
+0x26,
+0x82,
+0x00,
+0x21,
+0xf2,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xcc,
+0x57,
+0x68,
+0x20,
+0x02,
+0x1f,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xcb,
+0x97,
+0x32,
+0x0a,
+0x0b,
+0xcb,
+0x78,
+0x38,
+0x1c,
+0x76,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x82,
+0x59,
+0xe0,
+0xbc,
+0xb2,
+0x83,
+0x81,
+0x12,
+0x25,
+0x88,
+0x0b,
+0xc0,
+0x69,
+0x39,
+0x2c,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0x05,
+0x20,
+0x68,
+0x00,
+0x07,
+0xfb,
+0x21,
+0x52,
+0x0f,
+0x38,
+0x00,
+0x61,
+0x51,
+0xb1,
+0xb2,
+0xc2,
+0x70,
+0x80,
+0x24,
+0xb0,
+0x00,
+0x00,
+0x94,
+0x02,
+0xf5,
+0x19,
+0x1f,
+0x94,
+0x0a,
+0xc2,
+0x91,
+0x3f,
+0x29,
+0x1b,
+0xe6,
+0xc4,
+0x00,
+0x43,
+0xc4,
+0xab,
+0xc9,
+0x77,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0xa0,
+0x82,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x02,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x03,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc8,
+0x17,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc7,
+0x57,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0xa0,
+0x82,
+0x1a,
+0x00,
+0x30,
+0x88,
+0x0e,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x12,
+0x18,
+0x80,
+0xa0,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x05,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x04,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x90,
+0x40,
+0x00,
+0x03,
+0xc5,
+0x77,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc4,
+0xb7,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0xa0,
+0x82,
+0x1a,
+0x00,
+0x40,
+0x88,
+0x0e,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x12,
+0x18,
+0x80,
+0xa0,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x06,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x04,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x90,
+0x40,
+0x00,
+0x03,
+0xc2,
+0xd7,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc2,
+0x17,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0xa0,
+0x82,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x04,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x03,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xb7,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x0b,
+0x19,
+0x46,
+0x2c,
+0xb0,
+0x78,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x05,
+0x0b,
+0x07,
+0x8e,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xfb,
+0xc7,
+0x64,
+0x00,
+0x01,
+0xea,
+0xe7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xac,
+0xb0,
+0x78,
+0x62,
+0x89,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x60,
+0x38,
+0x08,
+0x63,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xf4,
+0xa7,
+0x64,
+0x00,
+0x01,
+0xeb,
+0xe7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x07,
+0x60,
+0xab,
+0xfe,
+0x09,
+0x40,
+0x2c,
+0x59,
+0x0f,
+0x00,
+0x80,
+0x76,
+0x42,
+0x36,
+0x42,
+0x04,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x23,
+0xa0,
+0xbc,
+0x17,
+0x03,
+0x23,
+0xe0,
+0xbc,
+0x7a,
+0x9a,
+0x40,
+0x40,
+0x68,
+0x00,
+0x00,
+0x08,
+0x21,
+0x40,
+0x00,
+0x01,
+0x88,
+0x09,
+0x66,
+0x00,
+0x01,
+0x2f,
+0xc8,
+0x5c,
+0x00,
+0x71,
+0x88,
+0x48,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x40,
+0x40,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc6,
+0x57,
+0x32,
+0x0b,
+0x0b,
+0xc3,
+0xe8,
+0x38,
+0x15,
+0x46,
+0x80,
+0x00,
+0x00,
+0x7a,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x8a,
+0x40,
+0x40,
+0x88,
+0x10,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x19,
+0x96,
+0x83,
+0x40,
+0x00,
+0xa2,
+0x06,
+0xc6,
+0x80,
+0x01,
+0x40,
+0x86,
+0x80,
+0x00,
+0x18,
+0x00,
+0xa5,
+0x48,
+0xd2,
+0x20,
+0x22,
+0x06,
+0xc6,
+0x80,
+0x01,
+0x44,
+0x85,
+0xc0,
+0xaa,
+0x2c,
+0x36,
+0x05,
+0xc0,
+0xb7,
+0x04,
+0x00,
+0x92,
+0x49,
+0x2d,
+0x80,
+0x24,
+0x90,
+0x00,
+0x00,
+0x84,
+0x00,
+0x92,
+0x49,
+0x2c,
+0x84,
+0x04,
+0x89,
+0x8e,
+0x48,
+0x52,
+0x4d,
+0x23,
+0xc4,
+0x3f,
+0x40,
+0x00,
+0x01,
+0x82,
+0x39,
+0x6c,
+0x68,
+0x00,
+0x14,
+0x08,
+0x68,
+0x3f,
+0xfe,
+0x7f,
+0xca,
+0x54,
+0x4d,
+0x22,
+0x02,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x14,
+0x48,
+0x5c,
+0x0a,
+0xa2,
+0xc3,
+0x60,
+0x84,
+0x00,
+0xa2,
+0x41,
+0x36,
+0x80,
+0x24,
+0xa0,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa2,
+0x41,
+0x34,
+0x40,
+0x00,
+0x00,
+0x40,
+0x48,
+0x66,
+0x00,
+0x01,
+0x7c,
+0xe0,
+0x66,
+0x00,
+0x01,
+0x7d,
+0xc0,
+0x5c,
+0x0b,
+0x71,
+0x8e,
+0x48,
+0x52,
+0x0d,
+0x23,
+0xc2,
+0x7f,
+0x40,
+0x00,
+0x01,
+0x82,
+0x39,
+0x6c,
+0x68,
+0x00,
+0x36,
+0x0a,
+0x25,
+0x93,
+0x0b,
+0xc0,
+0x38,
+0x5c,
+0x00,
+0x62,
+0x40,
+0x40,
+0x98,
+0xe8,
+0x88,
+0x81,
+0x48,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc1,
+0x37,
+0x32,
+0x0b,
+0x0b,
+0xc0,
+0x78,
+0xa4,
+0x04,
+0x06,
+0x80,
+0x00,
+0x00,
+0x7a,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xc0,
+0xbf,
+0x88,
+0x13,
+0x98,
+0x81,
+0x79,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x20,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0x70,
+0x68,
+0x20,
+0x02,
+0x21,
+0x2c,
+0x5c,
+0x07,
+0xe2,
+0x40,
+0x40,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0x22,
+0xc8,
+0x00,
+0x6c,
+0x68,
+0x20,
+0x02,
+0x23,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0x42,
+0xc8,
+0x00,
+0x6c,
+0x68,
+0x20,
+0x02,
+0x25,
+0x2c,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0x62,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x22,
+0x72,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x68,
+0x20,
+0x02,
+0x28,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0x92,
+0x18,
+0x00,
+0x61,
+0x68,
+0x20,
+0x02,
+0x2b,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0xc2,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0xa2,
+0xc6,
+0xe0,
+0x00,
+0x00,
+0xea,
+0xe5,
+0x52,
+0x18,
+0x04,
+0x06,
+0xc5,
+0x18,
+0x40,
+0x20,
+0x5e,
+0x05,
+0x80,
+0x98,
+0x18,
+0x42,
+0x89,
+0xc0,
+0x00,
+0x42,
+0x1c,
+0x60,
+0x40,
+0x20,
+0x98,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x32,
+0x06,
+0xe0,
+0x00,
+0x00,
+0xda,
+0xc3,
+0x20,
+0xa0,
+0xbc,
+0x22,
+0x03,
+0x21,
+0xa0,
+0xbc,
+0x2f,
+0x16,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x13,
+0x81,
+0xc4,
+0x84,
+0x80,
+0xa2,
+0x59,
+0x30,
+0xbc,
+0x05,
+0x9a,
+0x0e,
+0xc2,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x20,
+0x02,
+0x0e,
+0x23,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x9d,
+0x80,
+0x38,
+0x58,
+0x00,
+0x98,
+0xb0,
+0x23,
+0x00,
+0x80,
+0xbc,
+0x05,
+0x98,
+0x50,
+0x50,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0x09,
+0xa2,
+0x0e,
+0x71,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x48,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x09,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x00,
+0x42,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0xa8,
+0x09,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x09,
+0x00,
+0x00,
+0x00,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xac,
+0xb0,
+0x78,
+0x62,
+0x89,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x60,
+0x38,
+0x10,
+0x63,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xfd,
+0xc7,
+0x64,
+0x00,
+0x01,
+0xfc,
+0xc7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0xab,
+0xff,
+0x0a,
+0x06,
+0x81,
+0x88,
+0x06,
+0x18,
+0x80,
+0xf6,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x02,
+0x08,
+0x80,
+0xb6,
+0xa8,
+0x01,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x04,
+0x60,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x68,
+0x20,
+0x01,
+0x8e,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x00,
+0x42,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x6c,
+0x40,
+0x04,
+0x3c,
+0x00,
+0x23,
+0xe0,
+0x43,
+0x2a,
+0x20,
+0xbc,
+0x05,
+0x82,
+0x34,
+0x00,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x6c,
+0x00,
+0x00,
+0xde,
+0x50,
+0x6c,
+0x00,
+0x00,
+0xdc,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x08,
+0x40,
+0x02,
+0x24,
+0x81,
+0x08,
+0x40,
+0x50,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x3e,
+0x0a,
+0x32,
+0x83,
+0x06,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x05,
+0x1e,
+0x1a,
+0x3c,
+0x29,
+0x82,
+0x88,
+0x36,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x02,
+0x5b,
+0x44,
+0x0b,
+0x00,
+0xf8,
+0x37,
+0x04,
+0x52,
+0xe1,
+0x47,
+0x68,
+0x00,
+0x13,
+0x8d,
+0x09,
+0x62,
+0x00,
+0x00,
+0x00,
+0x27,
+0x50,
+0x4e,
+0x49,
+0x8e,
+0xb5,
+0x5b,
+0x4a,
+0x02,
+0xc0,
+0x20,
+0x2f,
+0x80,
+0x92,
+0x99,
+0x50,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x2b,
+0x98,
+0x34,
+0x93,
+0x61,
+0x45,
+0x46,
+0x08,
+0x8b,
+0x01,
+0x00,
+0x08,
+0x12,
+0x05,
+0x1a,
+0x1a,
+0x18,
+0x38,
+0x26,
+0x80,
+0x00,
+0x0c,
+0xd2,
+0x06,
+0xc0,
+0x00,
+0x19,
+0x60,
+0xa5,
+0x20,
+0x1b,
+0x00,
+0x04,
+0x28,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x19,
+0x64,
+0xa6,
+0x80,
+0x00,
+0x06,
+0xe2,
+0xc4,
+0x20,
+0x8f,
+0x80,
+0x04,
+0x88,
+0x40,
+0xec,
+0x68,
+0x00,
+0x00,
+0xcd,
+0x20,
+0x39,
+0x02,
+0x06,
+0xc4,
+0x00,
+0x44,
+0x00,
+0x85,
+0x94,
+0x10,
+0x00,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x0d,
+0xc7,
+0xa4,
+0x20,
+0x1c,
+0x80,
+0x07,
+0xa8,
+0x40,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xcb,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x04,
+0x40,
+0x0a,
+0x32,
+0x83,
+0x06,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x05,
+0x1e,
+0x1a,
+0x3c,
+0x29,
+0x82,
+0x88,
+0x36,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x02,
+0x5b,
+0x44,
+0x0b,
+0x00,
+0xf8,
+0x37,
+0x04,
+0x52,
+0xe1,
+0x47,
+0x68,
+0x00,
+0x13,
+0x8d,
+0x09,
+0x62,
+0x00,
+0x00,
+0x00,
+0x27,
+0x50,
+0x4e,
+0x49,
+0x8e,
+0xb5,
+0x5b,
+0x4a,
+0x02,
+0xc0,
+0x20,
+0x2f,
+0x80,
+0x92,
+0x99,
+0x50,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x2b,
+0x98,
+0x34,
+0x93,
+0x61,
+0x45,
+0x46,
+0x08,
+0x8b,
+0x01,
+0x00,
+0x08,
+0x12,
+0x05,
+0x1a,
+0x1a,
+0x18,
+0x38,
+0x26,
+0x80,
+0x00,
+0x0d,
+0x22,
+0x06,
+0xc0,
+0x00,
+0x19,
+0x60,
+0xa5,
+0x20,
+0x1b,
+0x00,
+0x04,
+0x28,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x19,
+0x64,
+0xa6,
+0x80,
+0x00,
+0x06,
+0xf2,
+0xc4,
+0x20,
+0x8f,
+0x80,
+0x04,
+0x88,
+0x40,
+0xec,
+0x68,
+0x00,
+0x00,
+0xd2,
+0x20,
+0x39,
+0x02,
+0x06,
+0xc4,
+0x00,
+0x43,
+0xe0,
+0x85,
+0x94,
+0x10,
+0x00,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x0d,
+0xe7,
+0xa4,
+0x20,
+0x1c,
+0x80,
+0x07,
+0xa8,
+0x40,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xcb,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x09,
+0x5d,
+0x0a,
+0x23,
+0x01,
+0xc6,
+0x52,
+0x0d,
+0x72,
+0xbf,
+0xe0,
+0x59,
+0x01,
+0x00,
+0x80,
+0xfa,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x4a,
+0x88,
+0x17,
+0x64,
+0x20,
+0x74,
+0x18,
+0xe8,
+0x95,
+0x50,
+0x16,
+0x08,
+0x04,
+0x89,
+0x8e,
+0x88,
+0x88,
+0x00,
+0x93,
+0x20,
+0x68,
+0xbc,
+0x07,
+0x8b,
+0x00,
+0x0d,
+0x88,
+0x00,
+0x95,
+0x90,
+0x54,
+0x30,
+0x00,
+0xdb,
+0xc0,
+0x29,
+0x98,
+0x24,
+0x89,
+0x8e,
+0x89,
+0x88,
+0x0c,
+0x93,
+0x20,
+0x68,
+0xbc,
+0x0c,
+0x98,
+0x81,
+0xc8,
+0x66,
+0x00,
+0x00,
+0xe8,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x46,
+0x20,
+0x5c,
+0x81,
+0x0a,
+0xc2,
+0x20,
+0x80,
+0x28,
+0x98,
+0x00,
+0x08,
+0x57,
+0x09,
+0x40,
+0x81,
+0x88,
+0x84,
+0x04,
+0x03,
+0x20,
+0x60,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xa1,
+0x66,
+0x00,
+0x00,
+0xe9,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x5a,
+0x20,
+0x5c,
+0x81,
+0x0a,
+0xc2,
+0x20,
+0x80,
+0x28,
+0x98,
+0x00,
+0x08,
+0x2e,
+0x12,
+0x88,
+0x40,
+0x40,
+0x98,
+0xe8,
+0x86,
+0x80,
+0x00,
+0x05,
+0x72,
+0x06,
+0x80,
+0x00,
+0x06,
+0xb2,
+0x16,
+0x60,
+0x00,
+0x04,
+0xa2,
+0x88,
+0x80,
+0x89,
+0xb0,
+0x00,
+0xc6,
+0x80,
+0x00,
+0x05,
+0x72,
+0x06,
+0x80,
+0x00,
+0x06,
+0xb2,
+0x16,
+0x60,
+0x00,
+0x04,
+0xa2,
+0x88,
+0x81,
+0x89,
+0x88,
+0x00,
+0x93,
+0x20,
+0x28,
+0xbc,
+0x11,
+0x16,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x03,
+0x81,
+0xc5,
+0x84,
+0x00,
+0x82,
+0x49,
+0x64,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x0a,
+0x52,
+0x4b,
+0xa8,
+0x40,
+0x48,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x49,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x42,
+0x03,
+0x7a,
+0x06,
+0xc0,
+0x40,
+0x00,
+0x00,
+0x40,
+0x6c,
+0x68,
+0x00,
+0x08,
+0x40,
+0x20,
+0x6c,
+0x00,
+0x00,
+0x0a,
+0x60,
+0x00,
+0x00,
+0x08,
+0x81,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x20,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0x56,
+0x20,
+0x68,
+0x20,
+0x00,
+0x82,
+0x22,
+0x84,
+0x00,
+0x88,
+0x50,
+0x0a,
+0x58,
+0x0d,
+0x02,
+0xbf,
+0xf0,
+0x68,
+0x20,
+0x00,
+0x81,
+0x21,
+0xbc,
+0x03,
+0x88,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x68,
+0x00,
+0x00,
+0x6a,
+0x20,
+0x6c,
+0x40,
+0x01,
+0x04,
+0x08,
+0x84,
+0x00,
+0xa3,
+0x01,
+0x30,
+0x68,
+0x20,
+0x00,
+0x82,
+0x22,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x48,
+0x68,
+0x20,
+0x00,
+0x81,
+0x21,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x6c,
+0x00,
+0x00,
+0xac,
+0x08,
+0x6c,
+0x40,
+0x01,
+0x04,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x41,
+0x6c,
+0x00,
+0x00,
+0xd4,
+0x08,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x68,
+0x00,
+0x08,
+0x40,
+0x20,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x1f,
+0x6c,
+0x00,
+0x00,
+0x0a,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x66,
+0xc4,
+0x00,
+0x44,
+0x20,
+0x05,
+0x24,
+0x82,
+0x04,
+0x04,
+0xaa,
+0x06,
+0xc0,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x48,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xf2,
+0x59,
+0x60,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x2d,
+0x52,
+0x4b,
+0xc3,
+0xc0,
+0x6f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x52,
+0x0b,
+0xc3,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x52,
+0x0d,
+0x22,
+0xbf,
+0xf0,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x20,
+0x25,
+0x96,
+0x06,
+0xc4,
+0x00,
+0x44,
+0x64,
+0x8b,
+0xc1,
+0x28,
+0x5c,
+0x08,
+0xa9,
+0x40,
+0x2f,
+0x52,
+0x4b,
+0xc3,
+0x01,
+0x2c,
+0x5c,
+0x81,
+0x01,
+0x40,
+0xad,
+0x25,
+0x92,
+0x8b,
+0xc3,
+0xb9,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x99,
+0x21,
+0x24,
+0x92,
+0x88,
+0x0a,
+0x7a,
+0x84,
+0x87,
+0xaa,
+0x09,
+0x89,
+0x94,
+0x86,
+0x0b,
+0xc3,
+0x17,
+0x25,
+0x97,
+0x8b,
+0xc1,
+0x60,
+0x38,
+0x10,
+0x82,
+0x58,
+0x20,
+0xbc,
+0x04,
+0x06,
+0xc0,
+0x00,
+0x0e,
+0xa7,
+0xa6,
+0xc0,
+0x00,
+0x0f,
+0xe7,
+0xa2,
+0x59,
+0x60,
+0xbc,
+0x08,
+0x16,
+0xc4,
+0x00,
+0x25,
+0x60,
+0x82,
+0x31,
+0x24,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x48,
+0x23,
+0x16,
+0x46,
+0xc0,
+0x00,
+0x14,
+0xc4,
+0x82,
+0x41,
+0x78,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x7a,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x43,
+0xe0,
+0x86,
+0xc4,
+0x00,
+0x44,
+0x00,
+0x92,
+0x91,
+0x64,
+0x32,
+0x82,
+0x0b,
+0xc0,
+0xc0,
+0x40,
+0x00,
+0x00,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x60,
+0x40,
+0x5c,
+0x0e,
+0x30,
+0x80,
+0x36,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x20,
+0x6c,
+0x40,
+0x04,
+0x3e,
+0x7a,
+0x6c,
+0x40,
+0x04,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x6e,
+0x21,
+0x00,
+0x00,
+0x08,
+0x48,
+0xfa,
+0x84,
+0x87,
+0xa3,
+0x81,
+0x1d,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x0b,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x68,
+0x5c,
+0x09,
+0xa1,
+0x40,
+0xad,
+0x52,
+0x49,
+0x43,
+0xc0,
+0xcf,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0x6c,
+0x40,
+0x02,
+0xc2,
+0x00,
+0x52,
+0x09,
+0x53,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x01,
+0x50,
+0x50,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x50,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe2,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9b,
+0xc8,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xca,
+0x80,
+0x10,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x00,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xa2,
+0x58,
+0x20,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x20,
+0x52,
+0x00,
+0x83,
+0xc0,
+0x5f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x24,
+0x81,
+0x06,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x03,
+0xa1,
+0x00,
+0x22,
+0x84,
+0x56,
+0x82,
+0x00,
+0x0d,
+0x72,
+0x02,
+0xa8,
+0xe8,
+0x51,
+0x84,
+0x00,
+0x40,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x21,
+0x52,
+0x4d,
+0x21,
+0x84,
+0x28,
+0xa0,
+0x02,
+0x08,
+0x48,
+0x00,
+0x52,
+0x4c,
+0x01,
+0xc0,
+0x00,
+0x84,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x44,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x19,
+0x84,
+0xa6,
+0xc4,
+0x00,
+0x19,
+0xa4,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0xa4,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0x44,
+0xa6,
+0x80,
+0x00,
+0x6e,
+0x6a,
+0xca,
+0x0e,
+0xc0,
+0x46,
+0x0a,
+0x40,
+0x48,
+0x50,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x08,
+0x5c,
+0x0e,
+0x32,
+0xc0,
+0x20,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x20,
+0x60,
+0x00,
+0x00,
+0x00,
+0x2c,
+0x52,
+0x0d,
+0x22,
+0xbf,
+0xf0,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x48,
+0x80,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x0e,
+0x82,
+0x06,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0xa5,
+0xc8,
+0x30,
+0x08,
+0x07,
+0x66,
+0x82,
+0x00,
+0x0e,
+0x22,
+0x18,
+0x00,
+0x4a,
+0x80,
+0x84,
+0xa8,
+0x40,
+0x4a,
+0x84,
+0x84,
+0xa6,
+0x80,
+0x00,
+0x0a,
+0xc2,
+0x06,
+0x60,
+0x00,
+0x04,
+0xe0,
+0x83,
+0x82,
+0xc4,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0x08,
+0x5c,
+0x08,
+0x23,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0xac,
+0x20,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x08,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x40,
+0x6c,
+0x40,
+0x01,
+0xf8,
+0x0a,
+0x5c,
+0x83,
+0x08,
+0x40,
+0xc8,
+0x80,
+0x04,
+0xa6,
+0x80,
+0x00,
+0x09,
+0x62,
+0x1a,
+0x01,
+0x20,
+0x80,
+0x04,
+0xa8,
+0x02,
+0xc8,
+0xa0,
+0x82,
+0x28,
+0x40,
+0x61,
+0x40,
+0x00,
+0x00,
+0x45,
+0x62,
+0x68,
+0x00,
+0x00,
+0xc2,
+0x20,
+0x68,
+0x00,
+0x00,
+0xc6,
+0x21,
+0x66,
+0x00,
+0x00,
+0x45,
+0x00,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0x28,
+0x5c,
+0x00,
+0x63,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x20,
+0x5c,
+0x82,
+0x00,
+0x80,
+0x36,
+0x68,
+0x00,
+0x08,
+0x97,
+0x2c,
+0x68,
+0x00,
+0x01,
+0x00,
+0x08,
+0xa0,
+0x0e,
+0x06,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0xc4,
+0x60,
+0xa4,
+0x00,
+0x24,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x68,
+0x00,
+0x01,
+0x32,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x84,
+0x10,
+0xa2,
+0x91,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc3,
+0x31,
+0x68,
+0x00,
+0x00,
+0xb2,
+0x20,
+0x68,
+0x00,
+0x00,
+0x9e,
+0x21,
+0x8c,
+0x03,
+0x25,
+0x15,
+0x49,
+0x20,
+0x14,
+0x2a,
+0x15,
+0x80,
+0x6c,
+0x40,
+0x02,
+0x12,
+0x08,
+0x84,
+0x00,
+0xa5,
+0x84,
+0x98,
+0x0d,
+0x03,
+0x05,
+0x15,
+0x40,
+0x04,
+0x84,
+0xeb,
+0xc0,
+0x8d,
+0x84,
+0x8c,
+0xc3,
+0x81,
+0xce,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x09,
+0x24,
+0x1a,
+0xe6,
+0xc4,
+0x00,
+0x44,
+0xa4,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x45,
+0x0a,
+0x30,
+0x93,
+0x0b,
+0xc0,
+0x65,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x08,
+0x38,
+0x1d,
+0x62,
+0x41,
+0xa4,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x48,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x66,
+0xc4,
+0x00,
+0x44,
+0xa0,
+0x95,
+0x24,
+0x96,
+0x04,
+0x04,
+0xa6,
+0xc4,
+0x00,
+0x44,
+0xa4,
+0x86,
+0x80,
+0x00,
+0x6e,
+0x6a,
+0xc4,
+0x60,
+0xa4,
+0x20,
+0x6c,
+0x08,
+0x40,
+0x6c,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x6c,
+0x40,
+0x04,
+0x52,
+0x08,
+0x68,
+0x00,
+0x47,
+0xff,
+0xca,
+0x68,
+0x38,
+0x1c,
+0x04,
+0x20,
+0x54,
+0x4d,
+0x23,
+0x00,
+0x08,
+0x5c,
+0x00,
+0x20,
+0x40,
+0x48,
+0x5c,
+0x09,
+0xf2,
+0x00,
+0x20,
+0x84,
+0x07,
+0xa8,
+0x40,
+0xd0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x38,
+0x10,
+0x00,
+0x25,
+0x98,
+0x0b,
+0xc0,
+0x50,
+0x6c,
+0x70,
+0x38,
+0x00,
+0x08,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x45,
+0x24,
+0x83,
+0x22,
+0xa0,
+0xbf,
+0xf5,
+0xa2,
+0xa0,
+0x64,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x07,
+0x60,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xac,
+0x5c,
+0x09,
+0xf1,
+0x40,
+0x2d,
+0x52,
+0x0d,
+0x03,
+0x00,
+0xec,
+0x30,
+0x12,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x8e,
+0x0b,
+0xc0,
+0x89,
+0xa0,
+0x44,
+0x86,
+0xc4,
+0x00,
+0x45,
+0x80,
+0x83,
+0x81,
+0x05,
+0x52,
+0x0b,
+0x23,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x48,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x21,
+0x39,
+0x7e,
+0x08,
+0x48,
+0x08,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x09,
+0x68,
+0x00,
+0x08,
+0xb9,
+0xac,
+0x6c,
+0x00,
+0x03,
+0x8a,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xa8,
+0x49,
+0xa0,
+0xc4,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x06,
+0xc8,
+0x48,
+0x4a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x00,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x20,
+0x5c,
+0x08,
+0x0b,
+0x00,
+0x0a,
+0x68,
+0x38,
+0x08,
+0x00,
+0x21,
+0x52,
+0xc2,
+0x00,
+0x40,
+0x52,
+0x5c,
+0xbf,
+0x03,
+0xc1,
+0x28,
+0x51,
+0xe0,
+0x21,
+0xc0,
+0x00,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xc0,
+0x54,
+0x41,
+0x03,
+0x04,
+0x01,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x51,
+0x51,
+0xd0,
+0x23,
+0x07,
+0xfb,
+0x54,
+0x47,
+0x0a,
+0x08,
+0x41,
+0x54,
+0x46,
+0x00,
+0x48,
+0x7a,
+0x42,
+0x07,
+0xf8,
+0x48,
+0xd1,
+0x40,
+0x00,
+0x00,
+0x48,
+0xd0,
+0x68,
+0x00,
+0x3f,
+0xc0,
+0x00,
+0x54,
+0x41,
+0x03,
+0x01,
+0x79,
+0x52,
+0x02,
+0x0b,
+0x07,
+0xf8,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x7a,
+0x51,
+0xd0,
+0x62,
+0x08,
+0x41,
+0x54,
+0x41,
+0x00,
+0x48,
+0x7a,
+0x84,
+0x8d,
+0x03,
+0x80,
+0x10,
+0x68,
+0x00,
+0x08,
+0xc7,
+0x2c,
+0x84,
+0x05,
+0x06,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0xc4,
+0x60,
+0xa4,
+0x20,
+0x50,
+0x08,
+0x40,
+0x52,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x08,
+0x5c,
+0x00,
+0x43,
+0x01,
+0x06,
+0x6c,
+0x70,
+0x10,
+0x0e,
+0x50,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x61,
+0x6c,
+0x70,
+0x10,
+0x02,
+0x08,
+0x51,
+0xb1,
+0x23,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x48,
+0x68,
+0x00,
+0x08,
+0xcc,
+0x20,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x8a,
+0x08,
+0x5c,
+0xbf,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x21,
+0x5c,
+0x0e,
+0x21,
+0xc0,
+0x00,
+0x5c,
+0x01,
+0xe8,
+0x48,
+0x0b,
+0x6c,
+0x00,
+0x03,
+0xa8,
+0x0a,
+0x52,
+0x49,
+0xe0,
+0x40,
+0x4a,
+0x5c,
+0x09,
+0xf2,
+0x04,
+0x40,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xaf,
+0x52,
+0x4d,
+0xc0,
+0x48,
+0x48,
+0xa0,
+0xec,
+0x18,
+0x40,
+0x49,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xa0,
+0x58,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x86,
+0xc8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x08,
+0x38,
+0x1c,
+0x66,
+0xc6,
+0x80,
+0x03,
+0xc0,
+0x02,
+0x41,
+0xa4,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x48,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0xa0,
+0x68,
+0x34,
+0x00,
+0x00,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x08,
+0xa0,
+0x3c,
+0x08,
+0x40,
+0x00,
+0x6c,
+0x00,
+0x03,
+0xb6,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xb8,
+0x50,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x45,
+0x40,
+0x82,
+0xa8,
+0xe0,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0xe1,
+0x68,
+0x34,
+0x00,
+0x00,
+0x20,
+0x6c,
+0x00,
+0x03,
+0xb6,
+0x00,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x50,
+0xa0,
+0x3c,
+0x06,
+0xc0,
+0x00,
+0x3b,
+0x80,
+0x28,
+0x40,
+0x52,
+0x42,
+0x0f,
+0x7b,
+0x0f,
+0xf0,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x50,
+0x6c,
+0x68,
+0x00,
+0x3c,
+0x7a,
+0x68,
+0x00,
+0x04,
+0xd9,
+0x00,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x02,
+0x54,
+0x40,
+0x83,
+0x01,
+0x0a,
+0x25,
+0x8a,
+0x06,
+0xc6,
+0x80,
+0x00,
+0x05,
+0x0b,
+0xc0,
+0x48,
+0xb0,
+0xff,
+0x2b,
+0xc0,
+0xef,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x52,
+0x5c,
+0x08,
+0x03,
+0x1f,
+0xf2,
+0x25,
+0x82,
+0x0b,
+0xc0,
+0x89,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x52,
+0x38,
+0x11,
+0x06,
+0xc6,
+0x80,
+0x03,
+0xa0,
+0x22,
+0x40,
+0x10,
+0x6c,
+0x68,
+0x00,
+0x3a,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x00,
+0x24,
+0x98,
+0x68,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xf2,
+0x59,
+0x60,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x3d,
+0x52,
+0x4b,
+0xc3,
+0xc0,
+0x6f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x52,
+0x0b,
+0xc3,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0xca,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc5,
+0xc0,
+0x5c,
+0x08,
+0x12,
+0xc0,
+0x21,
+0x25,
+0x88,
+0x06,
+0x80,
+0x00,
+0x1d,
+0xd2,
+0x0b,
+0xc0,
+0x88,
+0x5c,
+0x85,
+0x00,
+0x00,
+0xa1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x42,
+0x07,
+0x78,
+0x48,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xcc,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe1,
+0x22,
+0x00,
+0x00,
+0x08,
+0x10,
+0x23,
+0x00,
+0x00,
+0x08,
+0x58,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0x74,
+0x00,
+0x00,
+0x08,
+0x48,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0xf4,
+0x5c,
+0x08,
+0x52,
+0x00,
+0x01,
+0x25,
+0x88,
+0x0b,
+0xc0,
+0x68,
+0x80,
+0x0a,
+0x20,
+0x00,
+0x00,
+0x42,
+0x07,
+0x78,
+0x50,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xce,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe2,
+0x23,
+0x00,
+0x00,
+0x08,
+0x18,
+0x24,
+0x00,
+0x00,
+0x08,
+0x60,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x58,
+0x74,
+0x00,
+0x00,
+0x08,
+0x50,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x58,
+0xf4,
+0x38,
+0x11,
+0x22,
+0x58,
+0x80,
+0xbc,
+0x07,
+0x88,
+0x48,
+0xa1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x42,
+0x07,
+0x78,
+0x48,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xd0,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe3,
+0x22,
+0x00,
+0x00,
+0x08,
+0x10,
+0x23,
+0x00,
+0x00,
+0x08,
+0x58,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0x74,
+0x00,
+0x00,
+0x08,
+0x48,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0xf4,
+0x38,
+0x11,
+0xa2,
+0x58,
+0x80,
+0xbc,
+0x07,
+0x88,
+0x40,
+0xa0,
+0x00,
+0x00,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x3d,
+0x25,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0xe4,
+0x21,
+0x00,
+0x00,
+0x08,
+0x08,
+0x22,
+0x00,
+0x00,
+0x08,
+0x50,
+0x00,
+0x22,
+0xc0,
+0x49,
+0x48,
+0x74,
+0x00,
+0x00,
+0x08,
+0x40,
+0x00,
+0x22,
+0xc0,
+0x49,
+0x48,
+0xf4,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x37,
+0x08,
+0x63,
+0x70,
+0x44,
+0x2e,
+0x13,
+0x42,
+0xa0,
+0x64,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x55,
+0x38,
+0x20,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x0e,
+0x06,
+0x20,
+0x00,
+0x00,
+0x01,
+0x45,
+0x00,
+0x88,
+0x18,
+0xeb,
+0x50,
+0x00,
+0x00,
+0x2f,
+0x80,
+0x9b,
+0xc0,
+0x13,
+0x28,
+0x00,
+0x92,
+0x09,
+0x08,
+0x57,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x20,
+0x10,
+0x95,
+0x0c,
+0x84,
+0x19,
+0x20,
+0x3b,
+0xa1,
+0x48,
+0x5b,
+0xc2,
+0x01,
+0x8e,
+0x83,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x48,
+0x55,
+0x00,
+0x59,
+0x8e,
+0x80,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x0c,
+0xe2,
+0xbf,
+0xf0,
+0x50,
+0xc8,
+0x41,
+0x00,
+0x59,
+0x6a,
+0x00,
+0x03,
+0xff,
+0xc1,
+0x54,
+0x42,
+0x0b,
+0x00,
+0x0d,
+0x6a,
+0x00,
+0x03,
+0xff,
+0x80,
+0x30,
+0x80,
+0x8b,
+0xc0,
+0x79,
+0x2a,
+0x02,
+0xe6,
+0xe0,
+0x00,
+0x3d,
+0x81,
+0x19,
+0x00,
+0x13,
+0x28,
+0x85,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x01,
+0x19,
+0x8e,
+0x89,
+0x51,
+0x82,
+0x89,
+0x82,
+0x8b,
+0x6e,
+0x00,
+0x03,
+0xd4,
+0x13,
+0x30,
+0x8c,
+0x8b,
+0xc0,
+0x3c,
+0x6a,
+0x00,
+0x03,
+0xff,
+0xc1,
+0x98,
+0xe8,
+0xb2,
+0x19,
+0x13,
+0x28,
+0x85,
+0x93,
+0x08,
+0x08,
+0xbc,
+0x05,
+0x16,
+0xe0,
+0x00,
+0x3d,
+0x81,
+0x02,
+0x88,
+0x10,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x11,
+0x98,
+0xe8,
+0xa3,
+0x20,
+0x30,
+0x52,
+0x08,
+0x81,
+0x00,
+0x12,
+0xbc,
+0x08,
+0x92,
+0x41,
+0x12,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x41,
+0x32,
+0x03,
+0x8b,
+0xc0,
+0x31,
+0x2a,
+0x01,
+0x0b,
+0xc0,
+0x17,
+0x2a,
+0x01,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x5d,
+0x4a,
+0x33,
+0x01,
+0x03,
+0x68,
+0x00,
+0x01,
+0xff,
+0x4b,
+0x58,
+0x0f,
+0x83,
+0x5f,
+0xf8,
+0x50,
+0x47,
+0x19,
+0x84,
+0xc8,
+0x54,
+0x40,
+0x53,
+0x01,
+0xfe,
+0xbc,
+0x2f,
+0xa5,
+0x08,
+0xcd,
+0x18,
+0x38,
+0x33,
+0x01,
+0xe8,
+0xbc,
+0x0b,
+0x46,
+0x80,
+0x00,
+0x1f,
+0xf4,
+0xb3,
+0x01,
+0xe8,
+0xbc,
+0x0c,
+0x15,
+0x50,
+0x0f,
+0xb5,
+0x00,
+0x32,
+0x80,
+0xcb,
+0x37,
+0x8c,
+0x33,
+0x20,
+0x18,
+0xbc,
+0x06,
+0xb2,
+0xa0,
+0x3b,
+0x6e,
+0x00,
+0x03,
+0xdc,
+0x11,
+0xba,
+0x14,
+0x82,
+0x80,
+0x51,
+0x37,
+0x84,
+0x03,
+0x20,
+0x28,
+0xbc,
+0x19,
+0x33,
+0x61,
+0x45,
+0x32,
+0x02,
+0x8b,
+0xc1,
+0x20,
+0x38,
+0x20,
+0x73,
+0x01,
+0xe8,
+0xbc,
+0x05,
+0x23,
+0x28,
+0x08,
+0xbc,
+0x0d,
+0x93,
+0x88,
+0x09,
+0x38,
+0x00,
+0x1b,
+0xc0,
+0xa7,
+0x36,
+0x14,
+0x72,
+0x19,
+0x4b,
+0x28,
+0x9b,
+0xd2,
+0x11,
+0x49,
+0x32,
+0x80,
+0x8b,
+0xc0,
+0x29,
+0x38,
+0x80,
+0x93,
+0x80,
+0x01,
+0x54,
+0x82,
+0xcb,
+0x80,
+0x00,
+0x54,
+0x40,
+0x41,
+0x8e,
+0x89,
+0x40,
+0x00,
+0x01,
+0x83,
+0x03,
+0x5c,
+0x0a,
+0x73,
+0x50,
+0x00,
+0x25,
+0x19,
+0xe2,
+0x80,
+0x09,
+0x32,
+0x03,
+0x05,
+0xbc,
+0x20,
+0xbc,
+0x03,
+0x85,
+0x1d,
+0x44,
+0xb0,
+0x00,
+0xf9,
+0x8e,
+0x8b,
+0x55,
+0x43,
+0xf3,
+0x3f,
+0xff,
+0x29,
+0x9f,
+0x62,
+0x09,
+0x30,
+0x28,
+0x80,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x01,
+0x19,
+0x8e,
+0x89,
+0x50,
+0x49,
+0x43,
+0x01,
+0xa5,
+0x21,
+0x14,
+0x02,
+0x80,
+0x10,
+0x5b,
+0xc0,
+0x03,
+0xa1,
+0x48,
+0x28,
+0x04,
+0x13,
+0x78,
+0x40,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x2a,
+0x38,
+0x00,
+0xd9,
+0x8e,
+0x89,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x20,
+0x5b,
+0x08,
+0x23,
+0x80,
+0x00,
+0x5b,
+0x88,
+0x33,
+0x00,
+0xb7,
+0x54,
+0x0f,
+0x83,
+0x01,
+0x06,
+0x68,
+0x00,
+0x01,
+0x0c,
+0x8b,
+0x57,
+0x01,
+0xcb,
+0x01,
+0xff,
+0x50,
+0x4d,
+0x53,
+0x01,
+0xa5,
+0x20,
+0x98,
+0x92,
+0x11,
+0xd2,
+0x21,
+0x14,
+0x92,
+0x19,
+0xa3,
+0x28,
+0x05,
+0x22,
+0x10,
+0x18,
+0x5b,
+0xc4,
+0x13,
+0xa1,
+0x48,
+0x28,
+0x01,
+0x03,
+0x78,
+0x00,
+0xba,
+0x14,
+0x83,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x0d,
+0x23,
+0x01,
+0xfe,
+0x21,
+0x90,
+0x26,
+0xa0,
+0x00,
+0x1f,
+0xfc,
+0x12,
+0x88,
+0x51,
+0x50,
+0xcc,
+0x11,
+0x83,
+0x4a,
+0x68,
+0x00,
+0x01,
+0x07,
+0x89,
+0x30,
+0x17,
+0x06,
+0xe0,
+0x00,
+0x3e,
+0x41,
+0x1b,
+0xc0,
+0x8c,
+0x54,
+0x42,
+0x01,
+0x83,
+0x89,
+0xb1,
+0xff,
+0xf3,
+0x01,
+0xf0,
+0xbc,
+0x0b,
+0x3b,
+0xa1,
+0x48,
+0x98,
+0xe8,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0xff,
+0xc8,
+0x30,
+0x13,
+0x0b,
+0xc1,
+0x41,
+0x32,
+0x80,
+0x0b,
+0xc1,
+0x20,
+0x98,
+0xe8,
+0x9b,
+0xc1,
+0x07,
+0x68,
+0x00,
+0x01,
+0x0c,
+0xcb,
+0x2e,
+0x1b,
+0xe2,
+0x41,
+0x00,
+0x32,
+0x02,
+0x85,
+0x0c,
+0xc0,
+0x3c,
+0x02,
+0x89,
+0x83,
+0x08,
+0x36,
+0x10,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x02,
+0xa3,
+0x80,
+0x0e,
+0x98,
+0xe8,
+0xa2,
+0x99,
+0x76,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x60,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x49,
+0x6c,
+0x00,
+0x03,
+0xe0,
+0x08,
+0x6c,
+0x00,
+0x03,
+0xe2,
+0x08,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x0f,
+0xe3,
+0x01,
+0xa5,
+0x50,
+0xc8,
+0x42,
+0xbf,
+0xd0,
+0x50,
+0xc8,
+0x81,
+0x83,
+0x0a,
+0x50,
+0xca,
+0x59,
+0x83,
+0x0b,
+0x6a,
+0x00,
+0x01,
+0xff,
+0xc0,
+0x54,
+0x40,
+0xd9,
+0x00,
+0x5a,
+0x54,
+0xcf,
+0xb3,
+0x01,
+0x07,
+0x50,
+0xca,
+0x99,
+0x83,
+0xc9,
+0x54,
+0x40,
+0xc0,
+0x81,
+0x4a,
+0x50,
+0x4f,
+0x81,
+0x83,
+0x0a,
+0x21,
+0x10,
+0x06,
+0xe0,
+0x00,
+0x3f,
+0x01,
+0x35,
+0x40,
+0x61,
+0x90,
+0x25,
+0x86,
+0x80,
+0x00,
+0x1f,
+0xfc,
+0x86,
+0xe0,
+0x00,
+0x3e,
+0x81,
+0x03,
+0x01,
+0x28,
+0x28,
+0x81,
+0x25,
+0xbc,
+0x61,
+0xbc,
+0x76,
+0x82,
+0x88,
+0x08,
+0x30,
+0x13,
+0x0b,
+0xc1,
+0x41,
+0x32,
+0x81,
+0x0b,
+0xc0,
+0x91,
+0x20,
+0x9e,
+0x92,
+0x90,
+0x09,
+0x32,
+0x80,
+0x8b,
+0xc0,
+0xa1,
+0xba,
+0x14,
+0x86,
+0xe0,
+0x00,
+0x3e,
+0xc1,
+0x04,
+0x60,
+0x80,
+0xa8,
+0x03,
+0x09,
+0x00,
+0x12,
+0x64,
+0x00,
+0x02,
+0x44,
+0x8f,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0xba,
+0x14,
+0x8b,
+0xa1,
+0x01,
+0x55,
+0x00,
+0xc2,
+0x80,
+0x30,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0xb1,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x41,
+0xba,
+0x14,
+0x89,
+0x02,
+0x10,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0x5b,
+0x80,
+0x2b,
+0x00,
+0x09,
+0x2a,
+0x5a,
+0xd2,
+0x11,
+0x40,
+0x2e,
+0x14,
+0xd3,
+0x20,
+0x30,
+0xbc,
+0x0d,
+0x13,
+0x28,
+0x10,
+0xbc,
+0x05,
+0x1b,
+0xa1,
+0x48,
+0x40,
+0x00,
+0x01,
+0x02,
+0x10,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0x5b,
+0x84,
+0x23,
+0x00,
+0x09,
+0x2a,
+0x5a,
+0x42,
+0x11,
+0x12,
+0x57,
+0x08,
+0x73,
+0x80,
+0x00,
+0x5c,
+0x0d,
+0x0b,
+0xa1,
+0x11,
+0x52,
+0x02,
+0x99,
+0x82,
+0x42,
+0x52,
+0x02,
+0x09,
+0x82,
+0x80,
+0x23,
+0x2d,
+0xb5,
+0x0c,
+0xed,
+0x98,
+0x3c,
+0xa5,
+0x19,
+0x44,
+0x98,
+0x3c,
+0x95,
+0x40,
+0x08,
+0x18,
+0x2c,
+0x35,
+0x0c,
+0x64,
+0x98,
+0x34,
+0xb4,
+0x46,
+0x89,
+0x18,
+0x34,
+0x84,
+0x41,
+0x11,
+0x19,
+0x24,
+0x14,
+0x47,
+0x19,
+0x18,
+0x4c,
+0xa5,
+0xbc,
+0x41,
+0x18,
+0x40,
+0xb5,
+0x40,
+0x44,
+0x19,
+0x2c,
+0x14,
+0x40,
+0x99,
+0x32,
+0x00,
+0xd5,
+0xbc,
+0x00,
+0x10,
+0x05,
+0xb3,
+0x08,
+0x80,
+0xbc,
+0x03,
+0xa5,
+0x40,
+0xbe,
+0xb4,
+0x00,
+0xa3,
+0x80,
+0x02,
+0x50,
+0x8c,
+0x1b,
+0x00,
+0x0c,
+0x28,
+0x0c,
+0x92,
+0x11,
+0x92,
+0x21,
+0x98,
+0x03,
+0x78,
+0x41,
+0x28,
+0x01,
+0x25,
+0x84,
+0x64,
+0x10,
+0x01,
+0x33,
+0x78,
+0x82,
+0x54,
+0x04,
+0xd3,
+0xc0,
+0x3a,
+0x5b,
+0xc4,
+0x11,
+0x82,
+0x00,
+0x98,
+0xe8,
+0x82,
+0x09,
+0xa3,
+0x32,
+0x80,
+0x85,
+0x40,
+0x69,
+0x3c,
+0x02,
+0x93,
+0x78,
+0x81,
+0x98,
+0xe8,
+0x02,
+0x09,
+0x82,
+0x29,
+0x08,
+0x92,
+0x30,
+0x4a,
+0x32,
+0x01,
+0x0b,
+0xc0,
+0x4a,
+0x46,
+0x08,
+0x08,
+0x81,
+0x08,
+0x55,
+0x3f,
+0x69,
+0x92,
+0x81,
+0x64,
+0x00,
+0x02,
+0x47,
+0xef,
+0xa8,
+0x03,
+0x03,
+0x28,
+0x00,
+0xbc,
+0x10,
+0x16,
+0x80,
+0x00,
+0x1f,
+0xfc,
+0x93,
+0x01,
+0x70,
+0xbc,
+0x02,
+0x13,
+0x28,
+0x10,
+0xbc,
+0x0a,
+0x12,
+0x09,
+0xf1,
+0x29,
+0x08,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x0b,
+0x95,
+0x50,
+0x0c,
+0x3a,
+0x10,
+0x1b,
+0xa1,
+0x48,
+0x6e,
+0x00,
+0x03,
+0xec,
+0x10,
+0xa8,
+0x03,
+0x09,
+0x00,
+0x12,
+0x64,
+0x00,
+0x02,
+0x44,
+0x8f,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x66,
+0x66,
+0x66,
+0x66,
+0x00,
+0x3f,
+0xe6,
+0x66,
+0x66,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0xe0,
+0x00,
+0x00,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x07,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf0,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf0,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0xcc,
+0x35,
+0x1d,
+0x00,
+0x7f,
+0xfa,
+0xdb,
+0xbd,
+0x00,
+0x00,
+0xcc,
+0x35,
+0x1d,
+0x00,
+0xe0,
+0xfe,
+0x9f,
+0x8f,
+0x00,
+0x7f,
+0xaf,
+0xe1,
+0xd7,
+0x00,
+0x1f,
+0x51,
+0x7e,
+0x9b,
+0x00,
+0x8e,
+0xf8,
+0x36,
+0xb7,
+0x00,
+0x63,
+0x21,
+0x4d,
+0x57,
+0x00,
+0x72,
+0x19,
+0x84,
+0x0d,
+0x00,
+0x81,
+0xcb,
+0x2e,
+0x5d,
+0x00,
+0x7e,
+0x01,
+0xcc,
+0x7d,
+0x00,
+0x7f,
+0xcc,
+0xfa,
+0xd9,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x5a,
+0x84,
+0x71,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x32,
+0xf5,
+0x2c,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x17,
+0x0d,
+0x9e,
+0x34,
+0x00,
+0x66,
+0xa9,
+0xa3,
+0xaf,
+0x00,
+0x17,
+0x0d,
+0x9e,
+0x34,
+0x00,
+0x86,
+0xfb,
+0x10,
+0x8b,
+0x00,
+0x79,
+0x04,
+0xef,
+0x75,
+0x00,
+0x3d,
+0x00,
+0xdd,
+0x9d,
+0x00,
+0xc4,
+0x8d,
+0xa7,
+0xe7,
+0x00,
+0x3e,
+0x71,
+0x7a,
+0x7b,
+0x00,
+0x9a,
+0x10,
+0x28,
+0x3b,
+0x00,
+0x65,
+0xef,
+0xd7,
+0xc5,
+0x00,
+0x32,
+0xc7,
+0x23,
+0xb1,
+0x00,
+0xd7,
+0x00,
+0xa4,
+0x41,
+0x00,
+0x36,
+0x38,
+0x38,
+0x0f,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0xcc,
+0x35,
+0x1d,
+0x00,
+0x7f,
+0xfa,
+0xdb,
+0xbd,
+0x00,
+0x00,
+0xcc,
+0x35,
+0x1d,
+0x00,
+0xe0,
+0xfe,
+0x9f,
+0x8f,
+0x00,
+0x7f,
+0xaf,
+0xe1,
+0xd7,
+0x00,
+0x1f,
+0x51,
+0x7e,
+0x9b,
+0x00,
+0x8e,
+0xf8,
+0x36,
+0xb7,
+0x00,
+0x63,
+0x21,
+0x4d,
+0x57,
+0x00,
+0x72,
+0x19,
+0x84,
+0x0d,
+0x00,
+0x81,
+0xcb,
+0x2e,
+0x5d,
+0x00,
+0x7e,
+0x01,
+0xcc,
+0x7d,
+0x00,
+0x7f,
+0xcc,
+0xfa,
+0xd9,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x5a,
+0x84,
+0x71,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x32,
+0xf5,
+0x2c,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x17,
+0x0d,
+0x9e,
+0x34,
+0x00,
+0x66,
+0xa9,
+0xa3,
+0xaf,
+0x00,
+0x17,
+0x0d,
+0x9e,
+0x34,
+0x00,
+0x86,
+0xfb,
+0x10,
+0x8b,
+0x00,
+0x79,
+0x04,
+0xef,
+0x75,
+0x00,
+0x3d,
+0x00,
+0xdd,
+0x9d,
+0x00,
+0xc4,
+0x8d,
+0xa7,
+0xe7,
+0x00,
+0x3e,
+0x71,
+0x7a,
+0x7b,
+0x00,
+0x9a,
+0x10,
+0x28,
+0x3b,
+0x00,
+0x65,
+0xef,
+0xd7,
+0xc5,
+0x00,
+0x32,
+0xc7,
+0x23,
+0xb1,
+0x00,
+0xd7,
+0x00,
+0xa4,
+0x41,
+0x00,
+0x36,
+0x38,
+0x38,
+0x0f,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x02,
+0x03,
+0x02,
+0x00,
+0x00,
+0x02,
+0x00,
+0x01,
+0x01,
+0x00,
+0x01,
+0x01,
+0x02,
+0x03,
+0x00,
+0x00,
+0x04,
+0x9d,
+0x1d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x07,
+0xff,
+0xff,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0xb1,
+0xca,
+0x9e,
+0xb7,
+0x00,
+0x4c,
+0x9f,
+0xb1,
+0xad,
+0x00,
+0x54,
+0x54,
+0x77,
+0x9b,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x7f,
+0xf7,
+0xc6,
+0x15,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7e,
+0x67,
+0x33,
+0xd7,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0xb1,
+0xca,
+0x9e,
+0xb7,
+0x00,
+0x4c,
+0x9f,
+0xb1,
+0xad,
+0x00,
+0x54,
+0x54,
+0x77,
+0x9b,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x7f,
+0xf7,
+0xc6,
+0x15,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7e,
+0x67,
+0x33,
+0xd7,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x02,
+0x01,
+0x00,
+0x01,
+0x00,
+0x02,
+0x01,
+0x00,
+0x00,
+0x0a,
+0x47,
+0xe7,
+0x00,
+0x7f,
+0xeb,
+0x70,
+0x33,
+0x00,
+0x00,
+0x0a,
+0x47,
+0xe7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x64,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x20,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x44,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x15,
+0xd2,
+0xf3,
+0x00,
+0x0c,
+0xc9,
+0x9a,
+0x66,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x30,
+0x00,
+0x01,
+0x20,
+0x00,
+0x00,
+0x00,
+0x00,
+0x20,
+0x00,
+0x00,
+0x00,
+0x03,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xaf,
+0x00,
+0x00,
+0x00,
+0x00,
+0x70,
+0x00,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0xe3,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x03,
+0x00,
+0x00,
+0x3c,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x61,
+0xa8,
+0x00,
+0x00,
+0xd6,
+0x82,
+0x03,
+0x00,
+0x00,
+0x00,
+0x09,
+0xc4,
+0x00,
+0x00,
+0x00,
+0x4e,
+0x20,
+0x00,
+0x00,
+0x00,
+0x13,
+0x88,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0xc4,
+0x10,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0xc3,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0d,
+0xaf,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0xc6,
+0x00,
+0x00,
+0x00,
+0x02,
+0x8c,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x64,
+0x06,
+0x40,
+0x63,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x30,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x50,
+0x00,
+0x00,
+0x00,
+0x01,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xcc,
+0xcc,
+0xcc,
+0x00,
+0x19,
+0x99,
+0x99,
+0x99,
+0x00,
+0x33,
+0x33,
+0x33,
+0x33,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x66,
+0x66,
+0x66,
+0x66,
+0x00,
+0x5f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x46,
+0x66,
+0x66,
+0x65,
+0x00,
+0x2a,
+0xaa,
+0xaa,
+0xaa,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x73,
+0x33,
+0x33,
+0x32,
+0x00,
+0x7a,
+0xe1,
+0x47,
+0xad,
+0x00,
+0x43,
+0xd7,
+0x00,
+0x00,
+0x00,
+0x43,
+0xd7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0x00,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x64,
+0x87,
+0xed,
+0x4a,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x04,
+0xa0,
+0x00,
+0xff,
+0xfe,
+0x5f,
+0xe7,
+0x00,
+0x00,
+0x5b,
+0x05,
+0xb0,
+0x00,
+0xf5,
+0x55,
+0x55,
+0x56,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x40,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x5f,
+0x37,
+0x59,
+0xdf,
+0x00,
+0x5f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x40,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x7f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x18,
+0x00,
+0x00,
+0x00,
+0xae,
+0xb8,
+0x0f,
+0x00,
+0x7f,
+0xf4,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x00,
+0x41,
+0x00,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf8,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x49,
+0xe7,
+0x60,
+0x00,
+0x0c,
+0x28,
+0xf5,
+0xc0,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0xf1,
+0x91,
+0x96,
+0x00,
+0x7a,
+0x1c,
+0xdc,
+0xd3,
+0x00,
+0x02,
+0xf1,
+0x91,
+0x96,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xd6,
+0xe3,
+0xb3,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x00,
+0x66,
+0x84,
+0xe5,
+0x00,
+0x7f,
+0x32,
+0xf6,
+0x37,
+0x00,
+0x00,
+0x66,
+0x84,
+0xe5,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xd6,
+0xe3,
+0xb3,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x8c,
+0x5d,
+0xa4,
+0x55,
+0x00,
+0x67,
+0xf9,
+0x80,
+0xe3,
+0x00,
+0x74,
+0x57,
+0x25,
+0x39,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x68,
+0xc1,
+0x4b,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x06,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1d,
+0x53,
+0x00,
+0x00,
+0x13,
+0x9d,
+0x01,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1d,
+0xa1,
+0x2f,
+0x68,
+0x00,
+0x00,
+0x00,
+0x00,
+0xfa,
+0x00,
+0x00,
+0x00,
+0xc8,
+0xd7,
+0x00,
+0x00,
+0x7d,
+0x86,
+0x6e,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0xad,
+0x4f,
+0xc5,
+0xad,
+0x00,
+0x7c,
+0xd3,
+0x79,
+0x2f,
+0x00,
+0x55,
+0xdc,
+0xc1,
+0x23,
+0x00,
+0x00,
+0x00,
+0x0d,
+0xba,
+0x00,
+0x00,
+0x00,
+0x44,
+0xa2,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x0c,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x40,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x56,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x7a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xb2,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xe8,
+0x00,
+0x00,
+0x00,
+0x1d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x9c,
+0x00,
+0x00,
+0x00,
+0x20,
+0x9c,
+0x00,
+0x00,
+0x00,
+0x21,
+0x40,
+0x00,
+0x00,
+0x00,
+0x21,
+0x62,
+0x00,
+0x00,
+0x00,
+0x21,
+0xd4,
+0x00,
+0x00,
+0x00,
+0x22,
+0x08,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x22,
+0x98,
+0x00,
+0x00,
+0x00,
+0x23,
+0xa4,
+0x00,
+0x00,
+0x00,
+0x22,
+0xc0,
+0x00,
+0x00,
+0x00,
+0x22,
+0xc0,
+0x00,
+0x00,
+0x00,
+0x23,
+0x52,
+};
+
+#define VERNUM_04_17_01 0x04170108
+#define CAL_ID_04_17_01 0x00000002
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_09_17_00.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_09_17_00.h
new file mode 100644
index 0000000..806e059
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_09_17_00.h
@@ -0,0 +1,27495 @@
+/**
+ * @brief LC898123F40 OIS PRODUCT
+ *
+ * @author (C) 2017 ON Semiconductor.
+ *
+ **/
+
+// Time Stamp : 2018/02/21 14:09:18
+
+#define FromCodeBlockSize_09_17_00 11
+
+// [09170008] [00000001] [00000000]
+
+//#define MODULE_VENDOR 0x09
+//#define END_USER 0x17
+//#define USE_ACT 0x00
+
+const unsigned char CcMagicCodeF40_09_17_00[] = {
+0x00,
+0x01,
+0x02,
+0x03,
+0x04,
+0x05,
+0x06,
+0x07,
+0x08,
+0x09,
+0x00,
+0x00,
+0x00,
+0x00,
+0x54,
+0x00,
+0x00,
+0x00,
+0x13,
+0x00,
+0x00,
+0x00,
+0x00,
+0x06,
+0x98,
+0x00,
+0x00,
+0x00,
+0x01,
+0xfa,
+0x00,
+0x00,
+0x00,
+0x13,
+0x54,
+0x00,
+0x00,
+0x80,
+0x00,
+0x90,
+0x00,
+0x00,
+0x00,
+0xbb,
+0x67,
+0x00,
+0x8f,
+0x41,
+0x45,
+0xee,
+0x00,
+0x00,
+0x00,
+0xba,
+0x82,
+0x00,
+0xa9,
+0xce,
+0x6c,
+0xe2,
+0x00,
+0x00,
+0x00,
+0xc0,
+0x36,
+0x00,
+0x99,
+0xbd,
+0xd9,
+0x4c,
+0x00,
+0x00,
+0x00,
+0xb8,
+0xca,
+0x00,
+0xf7,
+0x7a,
+0xc1,
+0x7f,
+0x00,
+0x00,
+0x00,
+0xd0,
+0x7c,
+0x00,
+0x12,
+0x61,
+0x71,
+0x6f,
+0x00,
+0x00,
+0x00,
+0xbd,
+0xbb,
+0x00,
+0xf8,
+0x40,
+0xa3,
+0xf9,
+0x00,
+0x00,
+0x00,
+0xc2,
+0x21,
+0x00,
+0xec,
+0x9c,
+0xaa,
+0xba,
+0x00,
+0x00,
+0x00,
+0xcb,
+0xd0,
+0x00,
+0x90,
+0x1c,
+0x81,
+0x9e,
+0x00,
+0x00,
+0x00,
+0xa8,
+0xd8,
+0x00,
+0x61,
+0xe9,
+0xc7,
+0x7a,
+0x00,
+0x00,
+0x00,
+0x60,
+0x8e,
+0x00,
+0xef,
+0x1b,
+0x2d,
+0x67,
+0x00,
+0x00,
+0x00,
+0xb2,
+0x24,
+0x00,
+0x40,
+0x74,
+0x8a,
+0xac,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+};
+
+const unsigned char CcFromCodeF40_09_17_00[] = {
+0x64,
+0x00,
+0x00,
+0x02,
+0x87,
+0x64,
+0x00,
+0x00,
+0xfe,
+0xe7,
+0x64,
+0x00,
+0x01,
+0x02,
+0x67,
+0x64,
+0x00,
+0x01,
+0x08,
+0x87,
+0x64,
+0x00,
+0x01,
+0x08,
+0xa7,
+0x46,
+0x0b,
+0x03,
+0x80,
+0x00,
+0x64,
+0x00,
+0x01,
+0x08,
+0xc7,
+0x64,
+0x00,
+0x01,
+0x08,
+0xe7,
+0x64,
+0x00,
+0x01,
+0x09,
+0x07,
+0x64,
+0x00,
+0x01,
+0x09,
+0x27,
+0x64,
+0x00,
+0x01,
+0x09,
+0x47,
+0x64,
+0x00,
+0x01,
+0x09,
+0x67,
+0x64,
+0x00,
+0x01,
+0x09,
+0x87,
+0x64,
+0x00,
+0x01,
+0x09,
+0xa7,
+0x64,
+0x00,
+0x01,
+0x09,
+0xc7,
+0x46,
+0x0b,
+0x03,
+0x80,
+0x00,
+0x00,
+0x09,
+0x17,
+0x00,
+0x08,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xba,
+0x11,
+0x2b,
+0xa1,
+0x13,
+0x68,
+0x00,
+0x03,
+0x00,
+0x38,
+0x68,
+0x34,
+0x00,
+0x09,
+0x20,
+0x5c,
+0x81,
+0x09,
+0x8e,
+0xb9,
+0x5c,
+0x03,
+0xc3,
+0x0c,
+0x02,
+0x5c,
+0x96,
+0x00,
+0x40,
+0x52,
+0xa0,
+0x02,
+0x06,
+0x80,
+0x01,
+0x9e,
+0x7c,
+0x28,
+0x00,
+0xd2,
+0x84,
+0x05,
+0x0a,
+0x05,
+0x60,
+0x68,
+0x0f,
+0xc4,
+0xdb,
+0x00,
+0x84,
+0x05,
+0x0a,
+0x03,
+0x60,
+0x68,
+0x02,
+0x44,
+0xdf,
+0xc0,
+0xab,
+0xfc,
+0x08,
+0x02,
+0x50,
+0x40,
+0x00,
+0x00,
+0x80,
+0x60,
+0x66,
+0x00,
+0x01,
+0x10,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x0f,
+0xe0,
+0x5c,
+0x00,
+0x40,
+0x80,
+0x20,
+0x68,
+0x38,
+0x1c,
+0x09,
+0x24,
+0x5c,
+0x04,
+0x00,
+0x40,
+0x50,
+0xa2,
+0x26,
+0x08,
+0x60,
+0x7a,
+0x40,
+0x00,
+0x00,
+0x40,
+0x50,
+0x66,
+0x00,
+0x00,
+0xf8,
+0x40,
+0x68,
+0x20,
+0x00,
+0x00,
+0x24,
+0x68,
+0x00,
+0x00,
+0x08,
+0x20,
+0x98,
+0x90,
+0x96,
+0x60,
+0x00,
+0x12,
+0xfc,
+0x85,
+0xc0,
+0x13,
+0x18,
+0x80,
+0x89,
+0x8e,
+0x80,
+0x6e,
+0x00,
+0x03,
+0xb4,
+0x70,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb7,
+0x20,
+0x66,
+0x00,
+0x01,
+0x6f,
+0xe0,
+0x66,
+0x00,
+0x01,
+0x70,
+0xc0,
+0x68,
+0x00,
+0x01,
+0x0f,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb7,
+0x20,
+0x68,
+0x00,
+0x00,
+0x00,
+0xb9,
+0xba,
+0x11,
+0x00,
+0x00,
+0x00,
+0x66,
+0x00,
+0x01,
+0x1b,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x19,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x1c,
+0xc0,
+0x66,
+0x00,
+0x01,
+0x60,
+0x40,
+0x66,
+0x00,
+0x00,
+0xfd,
+0xa0,
+0x68,
+0x20,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x00,
+0x45,
+0xc0,
+0x66,
+0x00,
+0x00,
+0x1c,
+0xe0,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x66,
+0x00,
+0x01,
+0x52,
+0x40,
+0x6e,
+0x40,
+0x00,
+0x08,
+0x38,
+0x68,
+0x00,
+0x20,
+0x00,
+0x08,
+0x54,
+0x48,
+0x03,
+0x00,
+0x19,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x02,
+0x59,
+0x00,
+0x03,
+0x01,
+0x03,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x51,
+0x52,
+0x06,
+0x93,
+0xc0,
+0x99,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x52,
+0x38,
+0x1d,
+0x26,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x02,
+0x40,
+0x80,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x00,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x40,
+0x38,
+0x1c,
+0x22,
+0x40,
+0x80,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x50,
+0x66,
+0x00,
+0x01,
+0x20,
+0x80,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x7a,
+0x38,
+0x01,
+0xc6,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x96,
+0x60,
+0x04,
+0x03,
+0x30,
+0x86,
+0x80,
+0x00,
+0x00,
+0x0f,
+0x98,
+0x80,
+0x7a,
+0x66,
+0x00,
+0x40,
+0x34,
+0x60,
+0x32,
+0x82,
+0x0b,
+0xc0,
+0x98,
+0x88,
+0x00,
+0x06,
+0xc7,
+0x00,
+0x80,
+0x20,
+0x25,
+0x50,
+0x20,
+0x30,
+0x12,
+0x15,
+0x20,
+0x29,
+0x08,
+0x05,
+0x06,
+0xc7,
+0x00,
+0x80,
+0x25,
+0x23,
+0x21,
+0x40,
+0x40,
+0x00,
+0x03,
+0xff,
+0x02,
+0x66,
+0x00,
+0x40,
+0x34,
+0x20,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x68,
+0x38,
+0x08,
+0x10,
+0x24,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x5c,
+0x96,
+0x12,
+0x20,
+0x24,
+0x5c,
+0x0e,
+0x90,
+0x80,
+0x64,
+0x5c,
+0x09,
+0xc2,
+0x20,
+0x24,
+0x88,
+0x0e,
+0x4a,
+0x20,
+0x24,
+0x88,
+0x16,
+0x4a,
+0x20,
+0x24,
+0x88,
+0x1e,
+0x4a,
+0x20,
+0x24,
+0xa2,
+0x02,
+0x48,
+0x82,
+0x64,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x5c,
+0x00,
+0xca,
+0xc0,
+0xa8,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x03,
+0x32,
+0xa1,
+0x8b,
+0xff,
+0xa1,
+0xa0,
+0x00,
+0x48,
+0x03,
+0x03,
+0x25,
+0x89,
+0x8b,
+0xc0,
+0xc9,
+0x88,
+0x2e,
+0x02,
+0x48,
+0x9a,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x51,
+0x68,
+0x00,
+0x08,
+0x27,
+0x21,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x60,
+0x42,
+0x02,
+0xf8,
+0x23,
+0x52,
+0x40,
+0x00,
+0x00,
+0x60,
+0x61,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x20,
+0xa2,
+0x58,
+0x10,
+0xbc,
+0x35,
+0x88,
+0x83,
+0x60,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x98,
+0x80,
+0x26,
+0xc0,
+0x00,
+0x14,
+0xa0,
+0x03,
+0x00,
+0x80,
+0x68,
+0x00,
+0x02,
+0x32,
+0x20,
+0xbc,
+0x0b,
+0x89,
+0x88,
+0x02,
+0x30,
+0x08,
+0x0b,
+0xc2,
+0x81,
+0x5c,
+0x00,
+0x43,
+0x05,
+0xda,
+0x68,
+0x38,
+0x08,
+0x03,
+0x20,
+0x6c,
+0x70,
+0x10,
+0x06,
+0x52,
+0x84,
+0x1d,
+0x0b,
+0xc2,
+0x07,
+0x5c,
+0xb1,
+0x00,
+0x80,
+0x20,
+0x6c,
+0x70,
+0x10,
+0x20,
+0x0a,
+0x5c,
+0x00,
+0x40,
+0x40,
+0x08,
+0x88,
+0x0a,
+0x48,
+0x40,
+0x81,
+0x86,
+0x08,
+0x28,
+0x81,
+0x20,
+0x88,
+0x1a,
+0x48,
+0x40,
+0x83,
+0x86,
+0x08,
+0x98,
+0x82,
+0x20,
+0x68,
+0x00,
+0x00,
+0x96,
+0x24,
+0x80,
+0x00,
+0xb6,
+0xc0,
+0x00,
+0x30,
+0x64,
+0xa6,
+0xc0,
+0x00,
+0x2f,
+0x04,
+0x86,
+0xc0,
+0x00,
+0x31,
+0xc5,
+0x16,
+0xc4,
+0x00,
+0x24,
+0x65,
+0x26,
+0xc0,
+0x00,
+0x12,
+0xc5,
+0x38,
+0x60,
+0xc9,
+0x6c,
+0x00,
+0x02,
+0x9a,
+0x4b,
+0x40,
+0x00,
+0x00,
+0x40,
+0x50,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x7a,
+0x66,
+0x00,
+0x00,
+0xce,
+0x40,
+0x68,
+0x00,
+0x00,
+0x4b,
+0x20,
+0x5c,
+0x87,
+0x00,
+0x82,
+0xa4,
+0x68,
+0x00,
+0x00,
+0x5f,
+0x21,
+0x80,
+0x00,
+0x05,
+0x14,
+0x22,
+0x00,
+0x80,
+0x05,
+0x14,
+0x22,
+0x86,
+0x02,
+0x46,
+0xc6,
+0x81,
+0x00,
+0x04,
+0x88,
+0x82,
+0xe1,
+0x88,
+0x3e,
+0x06,
+0x83,
+0x40,
+0x80,
+0x02,
+0x5b,
+0xa0,
+0xc8,
+0x86,
+0xc4,
+0x90,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x66,
+0x00,
+0x01,
+0x1d,
+0x40,
+0x66,
+0x00,
+0x02,
+0x3c,
+0x60,
+0x6c,
+0x00,
+0x01,
+0x94,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x68,
+0x00,
+0x00,
+0xc2,
+0x20,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x24,
+0x68,
+0x00,
+0x00,
+0xc6,
+0x21,
+0x68,
+0x20,
+0x00,
+0xe6,
+0x25,
+0x68,
+0x00,
+0x00,
+0xac,
+0x22,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x23,
+0x66,
+0x00,
+0x01,
+0x38,
+0xc0,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x68,
+0x00,
+0x01,
+0x50,
+0x20,
+0x5c,
+0x83,
+0x0a,
+0xc0,
+0x20,
+0x5c,
+0x88,
+0x10,
+0x02,
+0x80,
+0x5c,
+0x87,
+0x18,
+0x40,
+0x02,
+0x57,
+0x00,
+0x80,
+0x83,
+0x20,
+0x5c,
+0x80,
+0x89,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0x8f,
+0x24,
+0x5c,
+0x08,
+0x50,
+0x03,
+0x01,
+0x51,
+0x60,
+0x20,
+0x22,
+0x00,
+0x86,
+0x00,
+0x35,
+0x70,
+0x0c,
+0x00,
+0x18,
+0x35,
+0x70,
+0x2d,
+0x80,
+0x30,
+0x15,
+0xc8,
+0x19,
+0x98,
+0x0c,
+0x35,
+0x16,
+0x0f,
+0x84,
+0x00,
+0x35,
+0x70,
+0x2c,
+0x18,
+0x00,
+0x16,
+0x80,
+0x00,
+0x1d,
+0x52,
+0x45,
+0x16,
+0x07,
+0x18,
+0x00,
+0x05,
+0x16,
+0x03,
+0x96,
+0x0d,
+0x76,
+0x80,
+0x00,
+0x17,
+0x92,
+0x15,
+0xc9,
+0x61,
+0x18,
+0xe4,
+0x95,
+0x24,
+0x55,
+0x80,
+0xa0,
+0x08,
+0x48,
+0x01,
+0x57,
+0x00,
+0x49,
+0x84,
+0xf9,
+0x5c,
+0x09,
+0xc1,
+0x8e,
+0x43,
+0x52,
+0x04,
+0xd9,
+0x61,
+0xd7,
+0x96,
+0x25,
+0x46,
+0x80,
+0x00,
+0x18,
+0x42,
+0x19,
+0x80,
+0x41,
+0x51,
+0x60,
+0x68,
+0x0a,
+0x08,
+0x84,
+0x80,
+0x25,
+0x70,
+0x89,
+0x08,
+0x2a,
+0x09,
+0x80,
+0x82,
+0x51,
+0x60,
+0xb8,
+0x83,
+0xa5,
+0x96,
+0x0d,
+0x79,
+0x60,
+0x55,
+0x98,
+0x4f,
+0x98,
+0x40,
+0x02,
+0x51,
+0x60,
+0xa0,
+0x68,
+0x01,
+0x51,
+0x60,
+0x71,
+0x60,
+0xd6,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x09,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x01,
+0x51,
+0x60,
+0x61,
+0x60,
+0xd4,
+0x51,
+0x61,
+0x71,
+0x60,
+0xd6,
+0x96,
+0x0f,
+0x69,
+0x60,
+0xd4,
+0x00,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x26,
+0xa0,
+0x35,
+0x94,
+0x0c,
+0x16,
+0x0b,
+0x95,
+0x50,
+0x24,
+0xbf,
+0x19,
+0x95,
+0xc0,
+0xe9,
+0x16,
+0x0f,
+0x14,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xa1,
+0xc0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0xf0,
+0xe7,
+0x68,
+0x20,
+0x00,
+0x04,
+0x20,
+0x5c,
+0x0b,
+0xe2,
+0xbf,
+0xf0,
+0x94,
+0x03,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x09,
+0x9a,
+0x01,
+0x20,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0xa0,
+0x00,
+0x12,
+0xa0,
+0x64,
+0x3b,
+0x10,
+0x43,
+0x23,
+0x20,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x38,
+0x17,
+0x42,
+0x59,
+0x28,
+0xbc,
+0x09,
+0x95,
+0xc0,
+0x32,
+0x2c,
+0x02,
+0x06,
+0x82,
+0x00,
+0x01,
+0x92,
+0x12,
+0xa0,
+0x64,
+0x3b,
+0x10,
+0x43,
+0x23,
+0xa0,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x5c,
+0x0a,
+0xe3,
+0x00,
+0x76,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xa9,
+0x5c,
+0x00,
+0x68,
+0x80,
+0x76,
+0x68,
+0x20,
+0x00,
+0x1b,
+0x21,
+0x38,
+0x09,
+0xc2,
+0xa0,
+0x76,
+0x3b,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x04,
+0x76,
+0x89,
+0x8e,
+0x88,
+0x88,
+0x0a,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x04,
+0x08,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x00,
+0x47,
+0x68,
+0x5c,
+0x00,
+0xa9,
+0x8e,
+0x88,
+0x5c,
+0x82,
+0x08,
+0x80,
+0xa0,
+0x39,
+0x02,
+0x0a,
+0x04,
+0x20,
+0x80,
+0x08,
+0xb8,
+0x00,
+0x09,
+0x51,
+0x91,
+0x50,
+0x00,
+0x00,
+0x80,
+0x00,
+0x16,
+0x83,
+0x40,
+0x0a,
+0x02,
+0x18,
+0x00,
+0x08,
+0x84,
+0x84,
+0xb6,
+0x80,
+0x00,
+0x05,
+0x82,
+0x28,
+0x00,
+0x09,
+0x80,
+0x00,
+0xa8,
+0x00,
+0x0b,
+0x6c,
+0x40,
+0x00,
+0x78,
+0x50,
+0x6c,
+0x40,
+0x00,
+0xc6,
+0x51,
+0x85,
+0x0c,
+0xb6,
+0x80,
+0x00,
+0x04,
+0x42,
+0x38,
+0x00,
+0x00,
+0x84,
+0x80,
+0x15,
+0x48,
+0x44,
+0x80,
+0x00,
+0xb8,
+0x40,
+0x82,
+0x80,
+0x00,
+0x38,
+0x48,
+0x51,
+0xa0,
+0x6c,
+0x06,
+0xc4,
+0x00,
+0x15,
+0xc5,
+0x3b,
+0x10,
+0x03,
+0x94,
+0x03,
+0x96,
+0x80,
+0x00,
+0x09,
+0xe2,
+0x15,
+0x44,
+0x66,
+0x05,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x18,
+0xc5,
+0x25,
+0x90,
+0x10,
+0x05,
+0x8c,
+0xa8,
+0x50,
+0x49,
+0x42,
+0x04,
+0x48,
+0x48,
+0x50,
+0x84,
+0x8c,
+0xb8,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x00,
+0x27,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x18,
+0x08,
+0x40,
+0x00,
+0x00,
+0x80,
+0xa0,
+0x68,
+0x00,
+0x01,
+0x00,
+0x09,
+0x94,
+0x03,
+0xc2,
+0x89,
+0x65,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0xa1,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x5c,
+0x08,
+0xe0,
+0x80,
+0xe0,
+0x24,
+0x12,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x06,
+0x60,
+0x00,
+0x02,
+0x9c,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x80,
+0x59,
+0x40,
+0x3c,
+0x28,
+0x96,
+0x53,
+0x20,
+0x28,
+0xbc,
+0x0c,
+0x98,
+0x80,
+0x36,
+0x88,
+0x07,
+0x66,
+0x80,
+0x00,
+0x15,
+0x02,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x86,
+0x80,
+0x04,
+0x01,
+0xcc,
+0x88,
+0x80,
+0x36,
+0x64,
+0x00,
+0x00,
+0x42,
+0x4f,
+0xa8,
+0x01,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0xab,
+0xfd,
+0x04,
+0x20,
+0x6f,
+0x88,
+0x0f,
+0x65,
+0xc0,
+0x03,
+0x08,
+0x04,
+0x98,
+0x82,
+0x4a,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa0,
+0x88,
+0x20,
+0x85,
+0x50,
+0x32,
+0x08,
+0x12,
+0x05,
+0xd4,
+0x83,
+0x08,
+0x18,
+0x85,
+0x50,
+0x32,
+0x20,
+0x02,
+0x08,
+0x80,
+0x09,
+0x58,
+0x0b,
+0x80,
+0x81,
+0xc8,
+0xbf,
+0xf2,
+0xa8,
+0x81,
+0x60,
+0x00,
+0x00,
+0x08,
+0x80,
+0xb6,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xfd,
+0x08,
+0x80,
+0x48,
+0x88,
+0x0f,
+0x66,
+0x82,
+0x00,
+0x08,
+0x52,
+0x08,
+0x81,
+0x7a,
+0x88,
+0x1e,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x0a,
+0x55,
+0x03,
+0xb8,
+0x81,
+0xa0,
+0x5d,
+0x4e,
+0x38,
+0x80,
+0x0a,
+0x59,
+0x09,
+0xc2,
+0x00,
+0x20,
+0x55,
+0x03,
+0xa0,
+0x81,
+0x4b,
+0xbf,
+0xf3,
+0xa8,
+0x80,
+0x48,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x40,
+0x80,
+0x68,
+0x20,
+0x00,
+0x85,
+0x20,
+0x88,
+0x20,
+0x05,
+0x16,
+0x02,
+0x20,
+0x08,
+0x08,
+0x80,
+0xb6,
+0x46,
+0x0a,
+0x41,
+0x40,
+0x64,
+0x94,
+0x0e,
+0x0a,
+0x80,
+0x30,
+0x5c,
+0x00,
+0x22,
+0xbf,
+0x20,
+0x88,
+0x1c,
+0x8a,
+0x40,
+0xc0,
+0x88,
+0x16,
+0x08,
+0x82,
+0x76,
+0x68,
+0x00,
+0x40,
+0x1a,
+0x48,
+0x88,
+0x2c,
+0x86,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x88,
+0x55,
+0x03,
+0x20,
+0x81,
+0x20,
+0x5d,
+0x48,
+0x30,
+0x82,
+0x88,
+0x59,
+0x0f,
+0x82,
+0x00,
+0x20,
+0x43,
+0xfa,
+0x50,
+0x81,
+0x60,
+0x55,
+0x03,
+0x20,
+0x81,
+0xca,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x42,
+0x80,
+0x5c,
+0x81,
+0x02,
+0x40,
+0xc0,
+0x5c,
+0x00,
+0x0a,
+0x00,
+0x71,
+0xa0,
+0xc1,
+0x29,
+0x48,
+0x35,
+0x95,
+0x03,
+0x7a,
+0x41,
+0xa1,
+0x23,
+0x08,
+0xc5,
+0x50,
+0x26,
+0x18,
+0x22,
+0x95,
+0xd4,
+0x80,
+0x9c,
+0x08,
+0x25,
+0x90,
+0xe4,
+0x15,
+0x0b,
+0x65,
+0x70,
+0xb9,
+0x15,
+0x03,
+0x45,
+0x70,
+0xf0,
+0x1c,
+0x88,
+0x34,
+0x3f,
+0x9d,
+0x15,
+0x8f,
+0x29,
+0x58,
+0x70,
+0x5c,
+0x86,
+0x0a,
+0x42,
+0xa3,
+0x88,
+0xa0,
+0x85,
+0x16,
+0x12,
+0x20,
+0x87,
+0x05,
+0x1a,
+0x12,
+0x20,
+0x46,
+0x15,
+0xb0,
+0x81,
+0x20,
+0x82,
+0x25,
+0x15,
+0x88,
+0x08,
+0x16,
+0x22,
+0xe1,
+0x12,
+0x2e,
+0x11,
+0x12,
+0x2b,
+0x09,
+0x51,
+0x58,
+0x90,
+0x18,
+0x41,
+0x51,
+0x59,
+0x30,
+0x18,
+0x42,
+0x54,
+0x09,
+0x00,
+0x18,
+0x40,
+0x54,
+0x08,
+0x10,
+0x18,
+0x7a,
+0x51,
+0x58,
+0x00,
+0x18,
+0x4a,
+0x51,
+0x58,
+0x80,
+0x18,
+0x40,
+0x81,
+0xac,
+0x08,
+0x80,
+0x63,
+0x40,
+0x00,
+0x00,
+0x81,
+0xe3,
+0x68,
+0x20,
+0x00,
+0x90,
+0x23,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x66,
+0x00,
+0x00,
+0x37,
+0xe0,
+0x5c,
+0x81,
+0x00,
+0x8a,
+0x08,
+0x51,
+0xa1,
+0x20,
+0x81,
+0xa3,
+0x5b,
+0x08,
+0x12,
+0xc0,
+0xc1,
+0x51,
+0x58,
+0x80,
+0x81,
+0x20,
+0x2e,
+0x11,
+0x25,
+0x70,
+0x88,
+0xa0,
+0x03,
+0x05,
+0x15,
+0x84,
+0xa0,
+0x46,
+0x15,
+0x15,
+0x89,
+0x01,
+0x84,
+0x15,
+0x15,
+0x93,
+0x01,
+0x84,
+0x25,
+0x40,
+0x90,
+0x01,
+0x84,
+0x05,
+0x40,
+0x81,
+0x01,
+0x87,
+0xa5,
+0x15,
+0x80,
+0x01,
+0x84,
+0xa5,
+0x15,
+0x88,
+0x01,
+0x84,
+0x08,
+0x1a,
+0xc0,
+0x88,
+0x06,
+0x36,
+0x82,
+0x00,
+0x09,
+0x8a,
+0x36,
+0x82,
+0x00,
+0x09,
+0x2a,
+0x46,
+0x82,
+0x00,
+0x09,
+0x5a,
+0x56,
+0x60,
+0x00,
+0x03,
+0x7e,
+0x8a,
+0x08,
+0x22,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x0e,
+0x00,
+0x00,
+0x00,
+0xba,
+0x14,
+0x82,
+0xe1,
+0xa8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x59,
+0x06,
+0x42,
+0xbf,
+0xd0,
+0x49,
+0xa9,
+0xb8,
+0x80,
+0x76,
+0xbc,
+0x19,
+0x89,
+0xa0,
+0x00,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x10,
+0x02,
+0xa0,
+0x76,
+0x68,
+0x00,
+0x09,
+0xc4,
+0x09,
+0x50,
+0x41,
+0x83,
+0xa1,
+0x11,
+0x44,
+0x08,
+0x81,
+0x01,
+0x58,
+0x5b,
+0x40,
+0x11,
+0x02,
+0x59,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x42,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x19,
+0x02,
+0x12,
+0x29,
+0x85,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x1a,
+0x33,
+0x60,
+0x00,
+0x37,
+0x80,
+0x0b,
+0xc1,
+0x77,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x10,
+0x02,
+0xa0,
+0x64,
+0x68,
+0x07,
+0xff,
+0xff,
+0xc9,
+0x50,
+0x41,
+0x03,
+0xa1,
+0x11,
+0x44,
+0x48,
+0x81,
+0x01,
+0x58,
+0x5b,
+0x40,
+0x11,
+0x02,
+0x59,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x42,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x19,
+0x02,
+0x12,
+0x29,
+0x85,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x02,
+0x33,
+0x60,
+0x00,
+0x37,
+0x80,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5b,
+0x02,
+0x03,
+0x01,
+0x01,
+0x50,
+0x43,
+0x11,
+0x84,
+0x0a,
+0x46,
+0x08,
+0x89,
+0x83,
+0x88,
+0x44,
+0x41,
+0x09,
+0x84,
+0x8b,
+0x98,
+0x30,
+0xa4,
+0x47,
+0x44,
+0xab,
+0xfe,
+0x05,
+0x08,
+0x20,
+0x08,
+0x07,
+0x60,
+0x88,
+0x82,
+0x37,
+0x80,
+0x06,
+0x80,
+0x00,
+0x9c,
+0x40,
+0x80,
+0x84,
+0x80,
+0x6a,
+0x00,
+0x4e,
+0x20,
+0x02,
+0x5b,
+0x44,
+0x11,
+0x01,
+0x58,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x40,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x26,
+0xa0,
+0x04,
+0xe2,
+0x00,
+0x12,
+0x98,
+0x52,
+0x32,
+0x01,
+0x0b,
+0xc0,
+0x3b,
+0x88,
+0x03,
+0x63,
+0x60,
+0x00,
+0x37,
+0x80,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0xab,
+0xf8,
+0x09,
+0x40,
+0x36,
+0x88,
+0x04,
+0xa8,
+0x80,
+0xe5,
+0x88,
+0x16,
+0x48,
+0x81,
+0xe3,
+0x88,
+0x26,
+0x28,
+0x82,
+0xf6,
+0x66,
+0x00,
+0x00,
+0x31,
+0xa8,
+0x40,
+0x00,
+0x01,
+0x48,
+0x35,
+0x5c,
+0x00,
+0x38,
+0x83,
+0x48,
+0x6a,
+0x00,
+0x02,
+0x71,
+0x00,
+0x5b,
+0x40,
+0x00,
+0x83,
+0xcb,
+0x88,
+0x45,
+0x49,
+0x05,
+0x58,
+0x00,
+0x00,
+0x08,
+0x80,
+0x0a,
+0x51,
+0x85,
+0xf8,
+0x82,
+0x22,
+0x98,
+0x2e,
+0x89,
+0xd0,
+0x02,
+0x88,
+0x4e,
+0x86,
+0x60,
+0x00,
+0x03,
+0x1a,
+0x89,
+0x50,
+0x35,
+0x5b,
+0x48,
+0x03,
+0x01,
+0xfe,
+0x52,
+0xcd,
+0x01,
+0x82,
+0x0b,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xc9,
+0x55,
+0x01,
+0xd1,
+0x80,
+0x0a,
+0xbc,
+0x07,
+0x88,
+0x86,
+0x52,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0xcd,
+0x88,
+0x38,
+0x16,
+0x80,
+0x01,
+0xff,
+0xfc,
+0x2b,
+0xc0,
+0x87,
+0x68,
+0x00,
+0x20,
+0x00,
+0x09,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x4d,
+0x40,
+0x00,
+0x00,
+0x83,
+0x81,
+0x68,
+0x00,
+0x20,
+0x00,
+0x02,
+0x59,
+0x0a,
+0x40,
+0x81,
+0xa2,
+0xbc,
+0x03,
+0xb3,
+0x90,
+0x10,
+0x40,
+0x00,
+0x01,
+0x50,
+0x52,
+0x55,
+0x02,
+0x70,
+0x81,
+0xe2,
+0x00,
+0x00,
+0x08,
+0x84,
+0xa8,
+0x88,
+0x82,
+0x35,
+0x18,
+0x5b,
+0x08,
+0x4c,
+0xa9,
+0xd8,
+0x04,
+0x98,
+0x2a,
+0x88,
+0x60,
+0x08,
+0x9d,
+0x80,
+0x38,
+0x83,
+0x0a,
+0x88,
+0x6e,
+0x46,
+0x60,
+0x00,
+0x03,
+0x1e,
+0x88,
+0x58,
+0x09,
+0x88,
+0x38,
+0xb3,
+0x20,
+0xf8,
+0xbc,
+0x13,
+0x85,
+0xc0,
+0xff,
+0x99,
+0x20,
+0x19,
+0x05,
+0x12,
+0x40,
+0x00,
+0x01,
+0x07,
+0x58,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x40,
+0x08,
+0x84,
+0x16,
+0x55,
+0x00,
+0x09,
+0x07,
+0x10,
+0x6a,
+0x00,
+0x02,
+0x71,
+0x02,
+0x29,
+0x88,
+0x23,
+0x20,
+0x10,
+0xbc,
+0x03,
+0xb3,
+0x81,
+0xff,
+0x36,
+0x04,
+0x13,
+0x78,
+0x41,
+0x52,
+0xce,
+0x41,
+0x83,
+0x4b,
+0x36,
+0x9c,
+0x26,
+0x80,
+0x01,
+0xff,
+0xfc,
+0xbb,
+0xc0,
+0x68,
+0x98,
+0x08,
+0xa3,
+0x01,
+0xf0,
+0xbc,
+0x09,
+0x56,
+0xa0,
+0x01,
+0xff,
+0xfc,
+0x1b,
+0xc0,
+0x67,
+0x68,
+0x00,
+0x20,
+0x00,
+0x0b,
+0x30,
+0x1f,
+0x0b,
+0xc0,
+0x25,
+0x6a,
+0x00,
+0x20,
+0x00,
+0x01,
+0x5c,
+0x80,
+0x80,
+0x81,
+0x22,
+0x98,
+0x34,
+0x19,
+0x50,
+0x51,
+0x88,
+0x16,
+0x20,
+0x00,
+0x00,
+0x88,
+0x6a,
+0x38,
+0x83,
+0x08,
+0x66,
+0x00,
+0x00,
+0x35,
+0x88,
+0x55,
+0x00,
+0x08,
+0x58,
+0x09,
+0x5c,
+0x0f,
+0xf9,
+0x83,
+0x0a,
+0x36,
+0x98,
+0x12,
+0x59,
+0xc0,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xca,
+0xbc,
+0x07,
+0x89,
+0x80,
+0x4b,
+0x30,
+0x1b,
+0x8b,
+0xc0,
+0xcd,
+0x88,
+0x48,
+0xb6,
+0xa0,
+0x01,
+0xff,
+0xfc,
+0x0b,
+0xc0,
+0x87,
+0x68,
+0x00,
+0x20,
+0x00,
+0x0a,
+0x30,
+0x1b,
+0x8b,
+0xc0,
+0x4d,
+0x40,
+0x00,
+0x00,
+0x84,
+0x8b,
+0x6a,
+0x00,
+0x20,
+0x00,
+0x00,
+0x5c,
+0x80,
+0x80,
+0x80,
+0xa2,
+0x5d,
+0x8e,
+0x39,
+0x83,
+0x00,
+0x59,
+0x0d,
+0xc0,
+0x86,
+0x09,
+0x95,
+0x05,
+0x08,
+0x83,
+0x49,
+0x88,
+0x3c,
+0xb8,
+0x80,
+0xe2,
+0xbf,
+0x75,
+0xa8,
+0x80,
+0x0a,
+0x88,
+0x2b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x08,
+0x00,
+0x00,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0x79,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x08,
+0x68,
+0x00,
+0x01,
+0x84,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x48,
+0x68,
+0x00,
+0x01,
+0x8f,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x88,
+0x68,
+0x00,
+0x40,
+0x1d,
+0xc8,
+0x68,
+0x20,
+0x01,
+0xcb,
+0x20,
+0x66,
+0x00,
+0x00,
+0x26,
+0x08,
+0x5c,
+0x00,
+0xeb,
+0x80,
+0x00,
+0x68,
+0x00,
+0x40,
+0x1e,
+0x88,
+0x68,
+0x20,
+0x01,
+0xce,
+0x20,
+0x66,
+0x00,
+0x00,
+0x26,
+0x08,
+0x38,
+0x01,
+0xd8,
+0x80,
+0x36,
+0x68,
+0x00,
+0x40,
+0x1f,
+0x48,
+0x68,
+0x20,
+0x01,
+0xd1,
+0x20,
+0x64,
+0x00,
+0x00,
+0x26,
+0x0f,
+0x5c,
+0x00,
+0xea,
+0x80,
+0x10,
+0x39,
+0x02,
+0x08,
+0x00,
+0x7a,
+0x80,
+0x07,
+0xa8,
+0x08,
+0x7a,
+0x80,
+0x07,
+0xa8,
+0x08,
+0x7a,
+0x80,
+0x87,
+0xa4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xfe,
+0x08,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x66,
+0x80,
+0x04,
+0x01,
+0x00,
+0x88,
+0x81,
+0x7a,
+0x88,
+0x1c,
+0x86,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x00,
+0x55,
+0x02,
+0x00,
+0x80,
+0x20,
+0x5d,
+0x00,
+0x03,
+0x00,
+0xe2,
+0x58,
+0x04,
+0x02,
+0x00,
+0x20,
+0x88,
+0x18,
+0x14,
+0x3f,
+0x9d,
+0x08,
+0x06,
+0x05,
+0x50,
+0x26,
+0x08,
+0x15,
+0x00,
+0x00,
+0x00,
+0x88,
+0x0b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x68,
+0x68,
+0x34,
+0x04,
+0x14,
+0x21,
+0x6c,
+0x68,
+0x08,
+0x28,
+0x49,
+0x00,
+0x00,
+0x08,
+0x48,
+0x89,
+0x84,
+0x04,
+0x94,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xc0,
+0x4f,
+0x5c,
+0x09,
+0xab,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x08,
+0x25,
+0x96,
+0x0b,
+0xff,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x6c,
+0x68,
+0x08,
+0x28,
+0x49,
+0x00,
+0x00,
+0x08,
+0x40,
+0x09,
+0x84,
+0x8c,
+0x94,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xc0,
+0x4f,
+0x5c,
+0x09,
+0xab,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x08,
+0x25,
+0x96,
+0x0b,
+0xff,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x59,
+0x01,
+0x41,
+0x48,
+0x2f,
+0xbc,
+0x0d,
+0x95,
+0xc0,
+0x87,
+0x14,
+0x02,
+0xd3,
+0x20,
+0x20,
+0xbc,
+0x05,
+0x05,
+0x24,
+0xdc,
+0x3a,
+0x14,
+0x89,
+0x48,
+0x60,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0x4d,
+0x43,
+0xa1,
+0x48,
+0x94,
+0x06,
+0x00,
+0x00,
+0x00,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x40,
+0x52,
+0x0d,
+0xc3,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x00,
+0x00,
+0x00,
+0x52,
+0x0d,
+0x43,
+0xa1,
+0x48,
+0x94,
+0x06,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x05,
+0x20,
+0x40,
+0x7a,
+0x46,
+0x0a,
+0x40,
+0x40,
+0xfa,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x40,
+0x46,
+0x08,
+0x88,
+0x02,
+0x0a,
+0x08,
+0x80,
+0x04,
+0x60,
+0xa4,
+0x00,
+0x24,
+0xc8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x01,
+0x84,
+0x00,
+0x88,
+0x40,
+0x8a,
+0x30,
+0x93,
+0x0b,
+0xc0,
+0x5c,
+0xb0,
+0x00,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x40,
+0xfa,
+0x98,
+0xe8,
+0x8b,
+0xa1,
+0x40,
+0x84,
+0x08,
+0x03,
+0x28,
+0x00,
+0xbc,
+0x05,
+0x08,
+0x40,
+0x00,
+0x55,
+0x02,
+0x23,
+0xa1,
+0x48,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x80,
+0x43,
+0xc0,
+0x5f,
+0x5c,
+0x00,
+0x31,
+0x8e,
+0x80,
+0x55,
+0x02,
+0x01,
+0x40,
+0x46,
+0x3a,
+0x80,
+0x03,
+0x01,
+0x00,
+0xbf,
+0xfb,
+0x2b,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x1c,
+0x04,
+0x21,
+0x38,
+0x00,
+0xe8,
+0x48,
+0x48,
+0xa0,
+0x82,
+0x18,
+0x48,
+0x7a,
+0xa0,
+0x82,
+0x18,
+0x48,
+0x4a,
+0xa0,
+0xcc,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x80,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xb0,
+0x68,
+0x00,
+0x00,
+0x9e,
+0x20,
+0x68,
+0x00,
+0x01,
+0x50,
+0x24,
+0x5c,
+0x00,
+0x10,
+0x00,
+0x01,
+0x5c,
+0x08,
+0x2a,
+0x00,
+0xa1,
+0x55,
+0x00,
+0xb1,
+0x48,
+0x2f,
+0x52,
+0xcb,
+0xc0,
+0x22,
+0x08,
+0x55,
+0x00,
+0x98,
+0x82,
+0x48,
+0xa0,
+0xd4,
+0x18,
+0x40,
+0x09,
+0x88,
+0x36,
+0x44,
+0x20,
+0x44,
+0x88,
+0x3f,
+0x65,
+0x50,
+0x0a,
+0x08,
+0x2e,
+0x16,
+0x80,
+0x00,
+0x09,
+0x62,
+0x06,
+0xc0,
+0x00,
+0x29,
+0xa0,
+0x28,
+0x40,
+0x03,
+0x84,
+0x08,
+0x83,
+0x81,
+0x08,
+0x25,
+0x83,
+0x8b,
+0xc1,
+0x09,
+0x55,
+0x01,
+0x81,
+0x82,
+0x8b,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xae,
+0x38,
+0x12,
+0x82,
+0x58,
+0x30,
+0xbc,
+0x09,
+0x85,
+0x50,
+0x1c,
+0x18,
+0x2c,
+0xa6,
+0x80,
+0x00,
+0x09,
+0x82,
+0x06,
+0xc0,
+0x00,
+0x29,
+0xc0,
+0xb8,
+0x40,
+0x8a,
+0x40,
+0x00,
+0x00,
+0x40,
+0x00,
+0x57,
+0x02,
+0xc8,
+0x82,
+0x03,
+0x57,
+0x06,
+0x92,
+0x08,
+0x00,
+0x2e,
+0x16,
+0x31,
+0x08,
+0x4e,
+0x54,
+0x0e,
+0x81,
+0xa0,
+0x42,
+0x6c,
+0x40,
+0x01,
+0x3e,
+0x08,
+0x44,
+0x20,
+0x81,
+0x80,
+0x09,
+0x6e,
+0x40,
+0x01,
+0x9d,
+0x23,
+0x6c,
+0x40,
+0x03,
+0x24,
+0x0b,
+0x6c,
+0x40,
+0x01,
+0x6e,
+0x08,
+0x20,
+0x0c,
+0x94,
+0xc1,
+0x42,
+0x80,
+0x84,
+0x18,
+0x82,
+0x61,
+0x00,
+0x00,
+0x06,
+0xe4,
+0x00,
+0x19,
+0xf2,
+0x46,
+0xc4,
+0x00,
+0x25,
+0x60,
+0x92,
+0x01,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x0b,
+0x51,
+0x8b,
+0x48,
+0x48,
+0x40,
+0x30,
+0x07,
+0x86,
+0xe4,
+0x00,
+0x35,
+0x52,
+0x65,
+0x00,
+0xc8,
+0x3c,
+0x53,
+0xd8,
+0x60,
+0x40,
+0x22,
+0x96,
+0xc2,
+0xe1,
+0x3c,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x48,
+0x22,
+0x96,
+0x43,
+0x01,
+0x60,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x48,
+0xbc,
+0x45,
+0x46,
+0xc0,
+0x00,
+0x36,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x05,
+0x16,
+0xc0,
+0x00,
+0x37,
+0x60,
+0xa3,
+0x20,
+0x30,
+0x40,
+0x00,
+0x03,
+0xc3,
+0x90,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x0a,
+0x2e,
+0x9a,
+0x42,
+0xa0,
+0x64,
+0x6c,
+0x40,
+0x02,
+0x1c,
+0x0a,
+0x6c,
+0x00,
+0x03,
+0x68,
+0x48,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0xec,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x48,
+0x51,
+0x89,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x49,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x7a,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x51,
+0x8b,
+0x6b,
+0xc2,
+0xcf,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x49,
+0x68,
+0x00,
+0x00,
+0x70,
+0x25,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x24,
+0x82,
+0x87,
+0xa6,
+0x80,
+0x00,
+0x1b,
+0x12,
+0x68,
+0x20,
+0x7a,
+0xa2,
+0x80,
+0x38,
+0x28,
+0x7a,
+0xa2,
+0x00,
+0x78,
+0x58,
+0xfa,
+0x68,
+0x00,
+0x01,
+0xb8,
+0x22,
+0x82,
+0x07,
+0xa8,
+0x78,
+0xfa,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x7a,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x49,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x87,
+0x00,
+0x86,
+0xc4,
+0x00,
+0x3e,
+0x24,
+0x88,
+0x51,
+0xfa,
+0x87,
+0x1f,
+0xa8,
+0x68,
+0xfa,
+0x86,
+0x0f,
+0xa0,
+0x00,
+0x00,
+0x42,
+0x05,
+0x78,
+0x50,
+0x0a,
+0x85,
+0x8c,
+0xab,
+0xc0,
+0x8f,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0xa1,
+0x82,
+0x58,
+0x84,
+0x65,
+0x66,
+0x00,
+0x00,
+0xcd,
+0x80,
+0x68,
+0x00,
+0x00,
+0x89,
+0x20,
+0x88,
+0x2a,
+0x18,
+0x40,
+0x48,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x40,
+0x00,
+0x00,
+0x82,
+0x20,
+0x66,
+0x00,
+0x00,
+0xcd,
+0x80,
+0x68,
+0x00,
+0x00,
+0x92,
+0x20,
+0x68,
+0x00,
+0x00,
+0x89,
+0x21,
+0x5c,
+0x82,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x80,
+0x80,
+0x98,
+0x82,
+0x61,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x2c,
+0x5c,
+0x87,
+0x00,
+0x82,
+0x20,
+0x88,
+0x1e,
+0xc8,
+0x02,
+0x48,
+0x68,
+0x20,
+0x00,
+0xdf,
+0x2c,
+0x68,
+0x20,
+0x00,
+0xdc,
+0x2d,
+0x68,
+0x20,
+0x00,
+0xde,
+0x22,
+0x88,
+0x26,
+0x06,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x46,
+0x80,
+0x00,
+0x0a,
+0x82,
+0x16,
+0x82,
+0x00,
+0x0d,
+0x32,
+0x58,
+0x80,
+0x6c,
+0x88,
+0x0e,
+0xd8,
+0x81,
+0x62,
+0x66,
+0x00,
+0x00,
+0x99,
+0x48,
+0x40,
+0x00,
+0x01,
+0x82,
+0x09,
+0x5c,
+0x82,
+0x00,
+0x82,
+0x20,
+0x68,
+0x00,
+0x00,
+0x92,
+0x21,
+0x84,
+0x34,
+0x86,
+0x80,
+0x00,
+0x07,
+0xa2,
+0x08,
+0x08,
+0x09,
+0x40,
+0x00,
+0x00,
+0x82,
+0x61,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x5c,
+0x87,
+0x00,
+0x82,
+0x20,
+0x88,
+0x42,
+0x18,
+0x02,
+0x48,
+0x88,
+0x1e,
+0x16,
+0x82,
+0x00,
+0x0d,
+0xfa,
+0xc6,
+0x82,
+0x00,
+0x0d,
+0xd2,
+0xd6,
+0x82,
+0x00,
+0x0d,
+0xea,
+0x28,
+0x82,
+0x60,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x00,
+0x00,
+0xaa,
+0x21,
+0x68,
+0x20,
+0x00,
+0xd5,
+0x25,
+0x88,
+0x06,
+0xc8,
+0x80,
+0xed,
+0x88,
+0x16,
+0x26,
+0x60,
+0x00,
+0x09,
+0x94,
+0x89,
+0x82,
+0x09,
+0x5c,
+0x82,
+0x00,
+0x83,
+0x21,
+0x68,
+0x00,
+0x01,
+0x49,
+0x20,
+0x84,
+0x80,
+0x98,
+0x82,
+0x21,
+0x80,
+0x04,
+0x98,
+0x82,
+0x60,
+0x40,
+0x00,
+0x00,
+0x4b,
+0x48,
+0x68,
+0x00,
+0x01,
+0x3a,
+0x20,
+0x68,
+0x20,
+0x01,
+0x8f,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x59,
+0x03,
+0x40,
+0x82,
+0x21,
+0xbc,
+0x66,
+0x88,
+0x48,
+0x48,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc5,
+0xd1,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x2a,
+0x8e,
+0xc3,
+0x20,
+0xe0,
+0xbc,
+0x27,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xe0,
+0x86,
+0xc0,
+0x00,
+0x11,
+0x40,
+0xa2,
+0x81,
+0x30,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x4b,
+0x38,
+0x10,
+0x72,
+0x41,
+0xed,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x49,
+0x2e,
+0x13,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x06,
+0xd3,
+0x81,
+0x0d,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x24,
+0x16,
+0x56,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x90,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x2a,
+0x8e,
+0xd3,
+0x20,
+0xe8,
+0xbc,
+0x0f,
+0x93,
+0x90,
+0x20,
+0x68,
+0x00,
+0x00,
+0x70,
+0x21,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x09,
+0x80,
+0x87,
+0xaa,
+0x08,
+0x00,
+0x80,
+0x87,
+0xa8,
+0x40,
+0xfa,
+0x6c,
+0x40,
+0x01,
+0x5a,
+0x49,
+0x84,
+0x8f,
+0xab,
+0xc0,
+0x27,
+0x5c,
+0x81,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x38,
+0x18,
+0x62,
+0x89,
+0xa5,
+0x30,
+0x1a,
+0x8b,
+0xc2,
+0x70,
+0x6c,
+0x40,
+0x02,
+0x0e,
+0x0b,
+0x6c,
+0x00,
+0x01,
+0x26,
+0x00,
+0x28,
+0x1c,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x04,
+0xb3,
+0x81,
+0x25,
+0x24,
+0x16,
+0x46,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x82,
+0xe1,
+0xc4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x38,
+0x12,
+0xf2,
+0x41,
+0xe4,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x28,
+0x9a,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x11,
+0x16,
+0x80,
+0x00,
+0x07,
+0xa2,
+0x16,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x88,
+0x08,
+0x7a,
+0xa0,
+0x80,
+0x08,
+0x08,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0xa4,
+0x88,
+0x48,
+0xfa,
+0xbc,
+0x05,
+0x7b,
+0xc0,
+0x4f,
+0x38,
+0x12,
+0x5b,
+0xc0,
+0x2f,
+0x38,
+0x12,
+0x53,
+0x81,
+0x25,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x21,
+0x6c,
+0x40,
+0x03,
+0xd2,
+0x08,
+0x84,
+0xb8,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0xb5,
+0x24,
+0xbe,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x7d,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x21,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x09,
+0x84,
+0xb8,
+0xb3,
+0x01,
+0x38,
+0xbc,
+0x05,
+0x93,
+0x81,
+0x2e,
+0x52,
+0x4d,
+0x6b,
+0xc0,
+0x5f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x24,
+0x1a,
+0xd6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x93,
+0x81,
+0x35,
+0x6c,
+0x40,
+0x01,
+0xb8,
+0x08,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x89,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x08,
+0x52,
+0x4b,
+0x28,
+0x83,
+0xb6,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x6f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x52,
+0x0b,
+0x28,
+0x83,
+0xb6,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x38,
+0x13,
+0xd6,
+0xc4,
+0x00,
+0x1b,
+0xa0,
+0x83,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x38,
+0x1d,
+0x56,
+0xc0,
+0x00,
+0x36,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x0a,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x86,
+0xc0,
+0x00,
+0x37,
+0x60,
+0xa3,
+0x20,
+0x30,
+0xbc,
+0x04,
+0x15,
+0x24,
+0xb2,
+0x3c,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x82,
+0x41,
+0x64,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xa0,
+0x86,
+0xc4,
+0x00,
+0x25,
+0x60,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x00,
+0x02,
+0x32,
+0x21,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x14,
+0xa6,
+0x1a,
+0x80,
+0x50,
+0x68,
+0x00,
+0x00,
+0x8a,
+0x20,
+0x5c,
+0x85,
+0x02,
+0xbf,
+0xa0,
+0x80,
+0x20,
+0x98,
+0x81,
+0x60,
+0x68,
+0x20,
+0x01,
+0xb4,
+0x2c,
+0x88,
+0x1f,
+0x66,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x46,
+0x82,
+0x00,
+0x0c,
+0xe2,
+0x56,
+0x80,
+0x00,
+0x1a,
+0x22,
+0x08,
+0x80,
+0x6c,
+0x66,
+0x00,
+0x00,
+0xb0,
+0x40,
+0x68,
+0x20,
+0x00,
+0xdc,
+0x2c,
+0x6c,
+0x00,
+0x03,
+0x44,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x60,
+0x00,
+0x09,
+0x82,
+0x89,
+0x82,
+0x09,
+0x68,
+0x00,
+0x00,
+0x93,
+0x20,
+0x5c,
+0x85,
+0x00,
+0x81,
+0x24,
+0x80,
+0x20,
+0x98,
+0x60,
+0x48,
+0x88,
+0x26,
+0x06,
+0x82,
+0x00,
+0x1b,
+0x62,
+0xc6,
+0x82,
+0x00,
+0x0b,
+0x42,
+0x46,
+0x82,
+0x00,
+0x0c,
+0xf2,
+0x56,
+0x80,
+0x00,
+0x1a,
+0x42,
+0x08,
+0x80,
+0x6c,
+0x66,
+0x00,
+0x00,
+0xb0,
+0x40,
+0x68,
+0x20,
+0x00,
+0xdd,
+0x2c,
+0x6c,
+0x00,
+0x03,
+0x48,
+0x48,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcf,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x60,
+0x00,
+0x09,
+0x82,
+0x89,
+0x82,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x20,
+0x88,
+0x22,
+0x48,
+0x02,
+0x09,
+0x86,
+0x04,
+0x88,
+0x81,
+0x60,
+0x68,
+0x00,
+0x00,
+0x70,
+0x21,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb1,
+0xa0,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x24,
+0x88,
+0x22,
+0x0a,
+0x20,
+0x44,
+0x80,
+0x20,
+0x98,
+0x60,
+0x48,
+0x88,
+0x2e,
+0x48,
+0x82,
+0x60,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb1,
+0xa0,
+0x88,
+0x22,
+0x06,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x4a,
+0x00,
+0x40,
+0x86,
+0x60,
+0x36,
+0xc4,
+0x00,
+0x2a,
+0xc0,
+0x25,
+0x80,
+0x4c,
+0x04,
+0x04,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x08,
+0xbc,
+0x06,
+0xa5,
+0xc0,
+0x82,
+0x88,
+0x36,
+0x05,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x20,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x09,
+0x84,
+0x60,
+0x33,
+0x00,
+0x98,
+0xbc,
+0x05,
+0xa3,
+0x81,
+0x0c,
+0x52,
+0x49,
+0x6b,
+0xc0,
+0x6f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x52,
+0x09,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x88,
+0x12,
+0x16,
+0x82,
+0x00,
+0x0d,
+0xe2,
+0x56,
+0x60,
+0x00,
+0x0b,
+0x70,
+0x88,
+0x49,
+0x09,
+0x88,
+0x22,
+0x06,
+0x80,
+0x00,
+0x0a,
+0x02,
+0x48,
+0x41,
+0x09,
+0x40,
+0x00,
+0x00,
+0x60,
+0x48,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xde,
+0xa5,
+0x66,
+0x00,
+0x00,
+0xb7,
+0x00,
+0x68,
+0x00,
+0x00,
+0xa0,
+0x20,
+0x68,
+0x20,
+0x00,
+0xd7,
+0x24,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x02,
+0x5c,
+0x83,
+0x04,
+0x00,
+0xc0,
+0x5d,
+0x0e,
+0x2a,
+0x00,
+0x03,
+0x51,
+0x85,
+0x68,
+0x00,
+0x48,
+0x98,
+0x26,
+0x88,
+0x82,
+0x64,
+0x9e,
+0x00,
+0x48,
+0x60,
+0x09,
+0x68,
+0x20,
+0x00,
+0xcc,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcd,
+0x21,
+0x86,
+0x04,
+0x98,
+0x48,
+0x49,
+0x88,
+0x3e,
+0x08,
+0x84,
+0x63,
+0x88,
+0x4d,
+0x08,
+0x81,
+0x48,
+0x66,
+0x00,
+0x00,
+0xc5,
+0x28,
+0x40,
+0x00,
+0x02,
+0x20,
+0x00,
+0x5c,
+0x09,
+0x28,
+0x84,
+0x20,
+0x88,
+0x3a,
+0x49,
+0x46,
+0x2a,
+0x52,
+0xca,
+0x82,
+0x24,
+0x84,
+0xbc,
+0x09,
+0x85,
+0xc8,
+0x30,
+0x08,
+0x3e,
+0x46,
+0x82,
+0x00,
+0x15,
+0x92,
+0x40,
+0x00,
+0x00,
+0x42,
+0x04,
+0x78,
+0x22,
+0x03,
+0x40,
+0x00,
+0x00,
+0x60,
+0x0b,
+0x68,
+0x20,
+0x01,
+0x5d,
+0x24,
+0x00,
+0x00,
+0x08,
+0x22,
+0x03,
+0x40,
+0x00,
+0x00,
+0x60,
+0x0b,
+0x5c,
+0x09,
+0x78,
+0x84,
+0x4b,
+0x25,
+0x9d,
+0x0b,
+0xc0,
+0x88,
+0x88,
+0x55,
+0x30,
+0x00,
+0x00,
+0x88,
+0x40,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x04,
+0x9b,
+0xc4,
+0x9f,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x49,
+0x25,
+0x95,
+0x0b,
+0xc3,
+0x01,
+0x6c,
+0x00,
+0x01,
+0x40,
+0x0a,
+0x66,
+0x00,
+0x00,
+0xca,
+0xc8,
+0x5c,
+0x00,
+0x28,
+0x40,
+0x0b,
+0x6c,
+0x40,
+0x01,
+0xae,
+0x09,
+0x5d,
+0x0a,
+0x28,
+0x82,
+0x20,
+0x23,
+0x0a,
+0xa9,
+0x84,
+0xa8,
+0x9c,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2a,
+0x80,
+0xb8,
+0x40,
+0x0a,
+0x08,
+0xb2,
+0x09,
+0x80,
+0x82,
+0x2e,
+0x0a,
+0x73,
+0x20,
+0x38,
+0xbc,
+0x03,
+0xd8,
+0x85,
+0x09,
+0xbc,
+0x02,
+0xf0,
+0x8d,
+0x20,
+0x98,
+0xe8,
+0x28,
+0x84,
+0x09,
+0x2e,
+0x0a,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0x00,
+0x99,
+0x80,
+0x82,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0xa4,
+0x6c,
+0x40,
+0x02,
+0xaa,
+0x09,
+0x2e,
+0x12,
+0xd3,
+0x20,
+0x28,
+0xbc,
+0x20,
+0x56,
+0xc4,
+0x00,
+0x18,
+0x05,
+0x26,
+0xc4,
+0x00,
+0x15,
+0x05,
+0x2b,
+0xc1,
+0xb7,
+0x6c,
+0x40,
+0x01,
+0x50,
+0x52,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x7f,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x52,
+0x59,
+0x01,
+0x03,
+0x01,
+0x05,
+0xbc,
+0x09,
+0x96,
+0xc4,
+0x00,
+0x3d,
+0x00,
+0x82,
+0x59,
+0x60,
+0xbc,
+0x0e,
+0x12,
+0x49,
+0x65,
+0x6c,
+0x40,
+0x03,
+0xd0,
+0x49,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x97,
+0x6c,
+0x40,
+0x02,
+0xbc,
+0x02,
+0x24,
+0x16,
+0x56,
+0xc4,
+0x00,
+0x3d,
+0x04,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x05,
+0x26,
+0xc4,
+0x00,
+0x18,
+0x05,
+0x23,
+0x90,
+0x20,
+0x88,
+0x3a,
+0x08,
+0x84,
+0x89,
+0x5b,
+0x4a,
+0x12,
+0x00,
+0x04,
+0x80,
+0x00,
+0x95,
+0xb4,
+0xa1,
+0x98,
+0x08,
+0x29,
+0x80,
+0xc3,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x0b,
+0x30,
+0x8d,
+0x0b,
+0xc0,
+0x6c,
+0x5c,
+0x08,
+0xa0,
+0x82,
+0x60,
+0x52,
+0x49,
+0xe3,
+0xc0,
+0x5f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x24,
+0x13,
+0xc6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x83,
+0x81,
+0x1c,
+0x86,
+0x08,
+0x25,
+0xb4,
+0x41,
+0x08,
+0x10,
+0x35,
+0xb4,
+0x61,
+0x18,
+0x08,
+0xb9,
+0x80,
+0x82,
+0x30,
+0x9d,
+0x0b,
+0xc0,
+0xcc,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x02,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x52,
+0x48,
+0xa3,
+0xc0,
+0xcf,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x52,
+0x08,
+0xa3,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x66,
+0x00,
+0x00,
+0xb7,
+0xe0,
+0x88,
+0x22,
+0x08,
+0x83,
+0xa4,
+0x88,
+0x2a,
+0x5a,
+0x00,
+0x68,
+0x86,
+0xb4,
+0x88,
+0x81,
+0x60,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x40,
+0x00,
+0x00,
+0x60,
+0x89,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcf,
+0x25,
+0x66,
+0x00,
+0x00,
+0xb7,
+0xe0,
+0x5c,
+0x09,
+0x68,
+0x82,
+0x20,
+0x88,
+0x32,
+0x59,
+0x46,
+0xab,
+0x52,
+0xca,
+0xc0,
+0x81,
+0x24,
+0x86,
+0xb4,
+0x84,
+0x20,
+0x9c,
+0xa2,
+0x40,
+0x85,
+0xc0,
+0x89,
+0x14,
+0x02,
+0xd3,
+0x81,
+0x0c,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x98,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x03,
+0x84,
+0x08,
+0xb5,
+0x40,
+0xed,
+0x84,
+0x00,
+0xb6,
+0xc0,
+0x00,
+0x11,
+0x80,
+0x82,
+0x81,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x43,
+0x25,
+0x8a,
+0x8b,
+0xc0,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x7a,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x7a,
+0x38,
+0x13,
+0x42,
+0x59,
+0x28,
+0xbc,
+0x15,
+0x06,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x95,
+0xc0,
+0xb9,
+0x98,
+0xe8,
+0x26,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x85,
+0x24,
+0x76,
+0xac,
+0x02,
+0x06,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x06,
+0xc4,
+0x00,
+0x25,
+0x07,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0xa4,
+0x86,
+0xc4,
+0x00,
+0x18,
+0xa4,
+0x86,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x94,
+0x21,
+0xff,
+0x94,
+0x24,
+0x28,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x88,
+0x20,
+0x68,
+0x00,
+0x00,
+0x91,
+0x21,
+0x66,
+0x00,
+0x01,
+0x59,
+0xc0,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x32,
+0x06,
+0x8b,
+0xc1,
+0x61,
+0x68,
+0x20,
+0x01,
+0x00,
+0x21,
+0x68,
+0x20,
+0x00,
+0xad,
+0x20,
+0xa0,
+0xc2,
+0x28,
+0x81,
+0x62,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x40,
+0x00,
+0x00,
+0x81,
+0x22,
+0x68,
+0x20,
+0x00,
+0xc5,
+0x20,
+0x68,
+0x20,
+0x01,
+0x00,
+0x21,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x38,
+0x09,
+0x40,
+0x00,
+0x03,
+0xc2,
+0x0f,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x49,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x38,
+0x19,
+0xc2,
+0x89,
+0x2d,
+0x30,
+0x12,
+0x8b,
+0xc1,
+0x81,
+0x68,
+0x20,
+0x00,
+0xad,
+0x20,
+0x68,
+0x20,
+0x01,
+0x01,
+0x21,
+0x68,
+0x20,
+0x00,
+0xfe,
+0x22,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x68,
+0x20,
+0x00,
+0xfe,
+0x22,
+0x68,
+0x20,
+0x00,
+0xc5,
+0x20,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x48,
+0x40,
+0x00,
+0x02,
+0x10,
+0x61,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x09,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x40,
+0x55,
+0x3f,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x49,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x88,
+0x1b,
+0x6b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x60,
+0xab,
+0xfd,
+0x06,
+0x82,
+0x00,
+0x1d,
+0xa2,
+0xc6,
+0x82,
+0x00,
+0x1d,
+0xb2,
+0xd6,
+0x82,
+0x00,
+0x1d,
+0xc2,
+0x26,
+0x82,
+0x00,
+0x1f,
+0x32,
+0x38,
+0x82,
+0x76,
+0x68,
+0x00,
+0x01,
+0x44,
+0x20,
+0x68,
+0x20,
+0x01,
+0x8f,
+0x24,
+0x68,
+0x00,
+0x01,
+0x52,
+0x21,
+0x68,
+0x20,
+0x01,
+0xd8,
+0x25,
+0x88,
+0x06,
+0xc8,
+0x80,
+0xed,
+0x88,
+0x16,
+0x28,
+0x81,
+0xe3,
+0x66,
+0x00,
+0x00,
+0x99,
+0x48,
+0x6c,
+0x00,
+0x02,
+0x96,
+0x09,
+0x68,
+0x00,
+0x02,
+0x3b,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x68,
+0x00,
+0x01,
+0x44,
+0x24,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0x86,
+0x34,
+0x8a,
+0x80,
+0x30,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x09,
+0x58,
+0x0b,
+0x02,
+0xbf,
+0xe0,
+0xbc,
+0x7b,
+0x88,
+0x80,
+0x76,
+0x68,
+0x20,
+0x01,
+0xb1,
+0x24,
+0x68,
+0x00,
+0x01,
+0x9c,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x6c,
+0x00,
+0x01,
+0x16,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x64,
+0x48,
+0x68,
+0x20,
+0x01,
+0xb1,
+0x24,
+0x68,
+0x00,
+0x01,
+0xa0,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x6c,
+0x00,
+0x01,
+0x28,
+0x09,
+0x68,
+0x20,
+0x01,
+0x1b,
+0x20,
+0x68,
+0x20,
+0x01,
+0xf5,
+0x24,
+0x5c,
+0x81,
+0x02,
+0x05,
+0xc1,
+0x84,
+0x00,
+0x98,
+0x48,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x08,
+0x01,
+0x6c,
+0x00,
+0x01,
+0x1a,
+0x0a,
+0x30,
+0x02,
+0x8b,
+0xc4,
+0x3d,
+0x6c,
+0x00,
+0x03,
+0x72,
+0x48,
+0x5c,
+0x83,
+0x0a,
+0x20,
+0x01,
+0x80,
+0xa0,
+0x88,
+0x0a,
+0x8b,
+0x49,
+0xb8,
+0xe2,
+0x05,
+0x60,
+0x22,
+0xfd,
+0x42,
+0xa8,
+0x64,
+0x28,
+0x11,
+0x22,
+0x28,
+0x54,
+0x5b,
+0x40,
+0x00,
+0x48,
+0x48,
+0xa0,
+0x8c,
+0x18,
+0x0a,
+0x08,
+0x84,
+0x80,
+0x95,
+0x40,
+0xb1,
+0x18,
+0x00,
+0x05,
+0x17,
+0xeb,
+0xa0,
+0xc8,
+0x12,
+0xa8,
+0x7b,
+0x54,
+0x06,
+0x98,
+0x40,
+0x02,
+0x6c,
+0x40,
+0x02,
+0x36,
+0x7a,
+0x30,
+0x08,
+0x05,
+0x14,
+0x2f,
+0xbc,
+0x21,
+0xb8,
+0x08,
+0xcb,
+0x2e,
+0x16,
+0x43,
+0x69,
+0x00,
+0x98,
+0x00,
+0x83,
+0x00,
+0xa0,
+0xbc,
+0x1b,
+0x38,
+0x80,
+0xca,
+0x88,
+0x16,
+0x18,
+0x81,
+0xd1,
+0x66,
+0x00,
+0x02,
+0x4d,
+0x28,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x08,
+0x6e,
+0x00,
+0x03,
+0x4c,
+0x12,
+0x66,
+0x00,
+0x02,
+0x52,
+0xa8,
+0x55,
+0x00,
+0x0b,
+0x80,
+0x00,
+0x66,
+0x00,
+0x02,
+0x4f,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x09,
+0x57,
+0x4b,
+0x20,
+0x81,
+0x21,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x48,
+0x5c,
+0x81,
+0x02,
+0xc0,
+0x61,
+0x88,
+0x18,
+0x18,
+0x80,
+0x8a,
+0x68,
+0x20,
+0x01,
+0xf5,
+0x24,
+0x80,
+0xa5,
+0x18,
+0x08,
+0xd1,
+0x42,
+0x01,
+0x78,
+0x0a,
+0x4a,
+0x84,
+0x84,
+0xa3,
+0x90,
+0x61,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3e,
+0xa0,
+0x82,
+0xe8,
+0x64,
+0x82,
+0x24,
+0x80,
+0x00,
+0x00,
+0x86,
+0x00,
+0x85,
+0x78,
+0x32,
+0x06,
+0x18,
+0x95,
+0x74,
+0xd6,
+0x02,
+0x0c,
+0x88,
+0x22,
+0x48,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x23,
+0x60,
+0x85,
+0x50,
+0x32,
+0x06,
+0x00,
+0x92,
+0xf1,
+0xad,
+0x6c,
+0x40,
+0x02,
+0x36,
+0x48,
+0x40,
+0x00,
+0x00,
+0x60,
+0x49,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x1b,
+0x92,
+0x0b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x20,
+0xa2,
+0x28,
+0x46,
+0x40,
+0x00,
+0x13,
+0x26,
+0xfa,
+0x01,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x65,
+0x88,
+0x0f,
+0x66,
+0x60,
+0x00,
+0x13,
+0x34,
+0x8a,
+0x20,
+0x84,
+0x88,
+0x02,
+0x08,
+0x81,
+0x24,
+0x94,
+0x02,
+0x55,
+0x00,
+0xb0,
+0x08,
+0x0b,
+0x68,
+0x60,
+0x09,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x08,
+0x44,
+0x08,
+0x02,
+0x80,
+0x10,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0xab,
+0xfd,
+0x08,
+0x81,
+0xe4,
+0x88,
+0x04,
+0x98,
+0x80,
+0xe5,
+0x88,
+0x16,
+0x18,
+0x82,
+0x60,
+0x88,
+0x2f,
+0x66,
+0x82,
+0x00,
+0x0d,
+0x02,
+0x46,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x08,
+0x00,
+0x88,
+0x22,
+0x06,
+0xc4,
+0x00,
+0x1a,
+0xe0,
+0x95,
+0xd0,
+0xa3,
+0x04,
+0x00,
+0x95,
+0x70,
+0xb0,
+0x04,
+0x1c,
+0x85,
+0x18,
+0x5b,
+0x18,
+0x00,
+0x86,
+0x82,
+0x00,
+0x12,
+0xd2,
+0x15,
+0xb4,
+0x80,
+0x18,
+0x2a,
+0x89,
+0x80,
+0x0a,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x00,
+0x58,
+0x41,
+0x80,
+0x81,
+0xa4,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0xa2,
+0x08,
+0x36,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x28,
+0x84,
+0x21,
+0xa1,
+0x86,
+0x44,
+0x20,
+0x25,
+0x21,
+0x8a,
+0x29,
+0x48,
+0x3f,
+0x38,
+0x12,
+0x92,
+0x58,
+0x70,
+0xbc,
+0x77,
+0x08,
+0x83,
+0x26,
+0x88,
+0x0a,
+0x79,
+0x70,
+0x2c,
+0x59,
+0x03,
+0x00,
+0x81,
+0x25,
+0xbc,
+0x11,
+0x98,
+0x78,
+0x7a,
+0x00,
+0x00,
+0x08,
+0x68,
+0x88,
+0x30,
+0x16,
+0x0b,
+0xc0,
+0x7b,
+0x84,
+0x38,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x09,
+0x49,
+0x8e,
+0x88,
+0x86,
+0x8c,
+0x99,
+0x70,
+0x64,
+0xbc,
+0x05,
+0x73,
+0x01,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0x86,
+0x8c,
+0x99,
+0x70,
+0x64,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3d,
+0x20,
+0x88,
+0x59,
+0x89,
+0x30,
+0x12,
+0x8b,
+0xc2,
+0x48,
+0x39,
+0x02,
+0x06,
+0x82,
+0x00,
+0x1e,
+0xe2,
+0x75,
+0xc8,
+0x20,
+0x85,
+0xa8,
+0x95,
+0xc8,
+0x51,
+0x03,
+0x80,
+0xa5,
+0x70,
+0xd4,
+0x21,
+0x00,
+0x65,
+0x90,
+0x18,
+0x01,
+0x2d,
+0x00,
+0x00,
+0x00,
+0x83,
+0xb0,
+0x98,
+0x50,
+0x0a,
+0x57,
+0x0b,
+0xb0,
+0x78,
+0x09,
+0x85,
+0x04,
+0xab,
+0xc0,
+0x6c,
+0x85,
+0xa8,
+0xa3,
+0x01,
+0x70,
+0xbc,
+0x08,
+0x28,
+0x59,
+0xc8,
+0x86,
+0x14,
+0x9b,
+0xc0,
+0x57,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x34,
+0x85,
+0x9c,
+0x88,
+0x61,
+0x49,
+0x00,
+0x00,
+0x08,
+0x50,
+0x09,
+0x57,
+0x0a,
+0x80,
+0x32,
+0x09,
+0x57,
+0x0a,
+0x00,
+0x83,
+0xa4,
+0xbc,
+0x03,
+0xf8,
+0x70,
+0x40,
+0x88,
+0x3a,
+0x40,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x60,
+0x09,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0xb3,
+0x6c,
+0x40,
+0x02,
+0x30,
+0x08,
+0x28,
+0x12,
+0xd3,
+0x08,
+0xa8,
+0xbc,
+0x03,
+0x4b,
+0xc0,
+0x6f,
+0x5c,
+0x0b,
+0xe8,
+0x60,
+0x49,
+0xbc,
+0x03,
+0xf5,
+0xc0,
+0xbe,
+0x86,
+0x05,
+0x23,
+0x81,
+0x7d,
+0x25,
+0x97,
+0x8b,
+0xc2,
+0x01,
+0x38,
+0x16,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x0f,
+0x93,
+0x81,
+0x75,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x49,
+0x98,
+0xe8,
+0x98,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x38,
+0x17,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x13,
+0x08,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x03,
+0x94,
+0x87,
+0x5b,
+0xc0,
+0xe7,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x49,
+0x38,
+0x16,
+0x58,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x62,
+0x38,
+0x17,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x05,
+0x08,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x25,
+0x24,
+0x17,
+0x89,
+0x48,
+0x70,
+0xbc,
+0x42,
+0xf8,
+0x80,
+0xa2,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x6c,
+0x88,
+0x12,
+0x55,
+0x70,
+0x14,
+0x08,
+0x32,
+0x3b,
+0xc0,
+0x5f,
+0x40,
+0x00,
+0x00,
+0x68,
+0xc0,
+0x54,
+0x0a,
+0x00,
+0x83,
+0x23,
+0x86,
+0x8c,
+0x0b,
+0x00,
+0x0d,
+0x88,
+0x0a,
+0x29,
+0x58,
+0x65,
+0x00,
+0x00,
+0x08,
+0x50,
+0x09,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0x23,
+0x2a,
+0x06,
+0xd8,
+0x50,
+0x49,
+0x68,
+0x20,
+0x01,
+0xeb,
+0x26,
+0x5c,
+0x81,
+0x0b,
+0x01,
+0x7d,
+0x52,
+0xcb,
+0xc0,
+0x30,
+0x88,
+0x5c,
+0x82,
+0x00,
+0x70,
+0x00,
+0x82,
+0x0c,
+0x88,
+0x60,
+0x50,
+0x68,
+0x20,
+0x01,
+0x19,
+0x23,
+0x87,
+0x08,
+0xb8,
+0x83,
+0xa7,
+0x81,
+0xa0,
+0x04,
+0x20,
+0x8c,
+0x06,
+0x0c,
+0xb5,
+0xc0,
+0x02,
+0x07,
+0x85,
+0x02,
+0x41,
+0x64,
+0x5d,
+0x48,
+0x20,
+0x43,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x6c,
+0x5c,
+0x0b,
+0xa9,
+0x48,
+0x74,
+0x52,
+0x4b,
+0x03,
+0xc0,
+0x7f,
+0x5c,
+0x09,
+0x29,
+0x48,
+0x70,
+0x52,
+0x0b,
+0x03,
+0xc0,
+0x3f,
+0x5c,
+0x09,
+0x29,
+0x48,
+0x70,
+0x38,
+0x12,
+0x52,
+0x59,
+0x70,
+0xbc,
+0x08,
+0x88,
+0x58,
+0x52,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2a,
+0xe0,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x04,
+0x96,
+0xc4,
+0x00,
+0x18,
+0x04,
+0x99,
+0x88,
+0x09,
+0x68,
+0x00,
+0x00,
+0x84,
+0x08,
+0x58,
+0x09,
+0x40,
+0x68,
+0x89,
+0x84,
+0x1c,
+0x9b,
+0xc1,
+0x48,
+0x85,
+0x00,
+0x99,
+0x88,
+0x08,
+0x68,
+0x00,
+0x00,
+0x8d,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x98,
+0xea,
+0x16,
+0x60,
+0x00,
+0x0b,
+0x8a,
+0x08,
+0x82,
+0x20,
+0xbc,
+0x10,
+0xf8,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xb5,
+0x21,
+0x66,
+0x00,
+0x00,
+0xb8,
+0xa0,
+0x88,
+0x22,
+0x0b,
+0xc0,
+0x9f,
+0x40,
+0x00,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xae,
+0x21,
+0x66,
+0x00,
+0x00,
+0xb8,
+0xa0,
+0x88,
+0x22,
+0x00,
+0x00,
+0x00,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x56,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x51,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x38,
+0x12,
+0xc2,
+0x59,
+0x28,
+0xbc,
+0x05,
+0x08,
+0x84,
+0xa1,
+0x40,
+0x00,
+0x00,
+0x40,
+0x09,
+0x42,
+0x29,
+0xb8,
+0x48,
+0x49,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0xb0,
+0x84,
+0x18,
+0x88,
+0x40,
+0x0a,
+0x57,
+0x0d,
+0x00,
+0x84,
+0xa1,
+0x98,
+0x00,
+0xa3,
+0x69,
+0x80,
+0x98,
+0x00,
+0xa3,
+0x01,
+0x70,
+0xbc,
+0x03,
+0x58,
+0x48,
+0x48,
+0xbc,
+0x01,
+0x78,
+0x84,
+0xa1,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x09,
+0x84,
+0x80,
+0x85,
+0x40,
+0xb0,
+0x04,
+0x00,
+0xa9,
+0x80,
+0x00,
+0x30,
+0x03,
+0x0b,
+0xc0,
+0x2d,
+0x88,
+0x0c,
+0x88,
+0x40,
+0x50,
+0x2e,
+0x16,
+0x08,
+0x40,
+0x09,
+0x98,
+0x00,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x01,
+0x38,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x88,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x56,
+0x08,
+0x68,
+0x00,
+0x2b,
+0xf2,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0xe4,
+0x68,
+0x00,
+0x27,
+0x10,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0x84,
+0x68,
+0x00,
+0x18,
+0x6a,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0x24,
+0xbc,
+0x08,
+0xf2,
+0x28,
+0x6d,
+0xbc,
+0x06,
+0xf2,
+0x28,
+0xad,
+0xbc,
+0x04,
+0xf2,
+0x28,
+0xed,
+0xbc,
+0x02,
+0xf2,
+0x29,
+0x2d,
+0x22,
+0x96,
+0xd8,
+0x84,
+0xa1,
+0x88,
+0x22,
+0x08,
+0x48,
+0x08,
+0x84,
+0x00,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x03,
+0x38,
+0x80,
+0x88,
+0x2e,
+0x16,
+0x08,
+0x48,
+0x40,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x84,
+0x80,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x04,
+0xd8,
+0x80,
+0x88,
+0x28,
+0x16,
+0x08,
+0x48,
+0x40,
+0x00,
+0x00,
+0x08,
+0x82,
+0xb6,
+0x84,
+0x00,
+0x84,
+0x60,
+0xa4,
+0x08,
+0x00,
+0x95,
+0x70,
+0x94,
+0x28,
+0x03,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x09,
+0xe2,
+0x22,
+0xe4,
+0x86,
+0x00,
+0xa4,
+0x43,
+0x00,
+0x16,
+0x9a,
+0x55,
+0x00,
+0xa0,
+0x08,
+0x02,
+0x56,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xd5,
+0x2c,
+0x94,
+0x06,
+0x80,
+0x94,
+0x20,
+0x24,
+0x18,
+0x00,
+0x80,
+0x81,
+0x00,
+0xba,
+0x14,
+0x89,
+0x80,
+0x08,
+0x00,
+0x00,
+0x08,
+0x40,
+0x89,
+0x46,
+0x0a,
+0x40,
+0x68,
+0x88,
+0x08,
+0x48,
+0x09,
+0x80,
+0x08,
+0xab,
+0xfd,
+0x0a,
+0x21,
+0x64,
+0xa0,
+0x88,
+0x18,
+0x80,
+0x64,
+0xa2,
+0x06,
+0x48,
+0x80,
+0xe1,
+0xa0,
+0x84,
+0x18,
+0x81,
+0x64,
+0x88,
+0x1c,
+0x98,
+0x82,
+0x60,
+0x88,
+0x2f,
+0x66,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x08,
+0x00,
+0x88,
+0x12,
+0x08,
+0x82,
+0x21,
+0xa0,
+0x06,
+0x08,
+0x40,
+0x0a,
+0x88,
+0x18,
+0x94,
+0x43,
+0x00,
+0x04,
+0xa4,
+0x8a,
+0x06,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb5,
+0xe8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x09,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x59,
+0x03,
+0x40,
+0x80,
+0xa4,
+0xbc,
+0x15,
+0x8a,
+0x20,
+0x20,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x5c,
+0x09,
+0x20,
+0x80,
+0xc8,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xc8,
+0x88,
+0x02,
+0x46,
+0xc4,
+0x00,
+0x2a,
+0xc0,
+0x96,
+0x60,
+0x00,
+0x15,
+0x82,
+0x88,
+0x60,
+0x8a,
+0x55,
+0x01,
+0x28,
+0x80,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0xc9,
+0xbc,
+0x0f,
+0xf8,
+0x80,
+0x88,
+0xbc,
+0x0d,
+0xf8,
+0x80,
+0x88,
+0x86,
+0x08,
+0x93,
+0x69,
+0x40,
+0x98,
+0x00,
+0x96,
+0xc4,
+0x00,
+0x21,
+0x00,
+0xa3,
+0x09,
+0x70,
+0xbc,
+0x05,
+0x58,
+0x80,
+0x20,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x09,
+0x84,
+0x0c,
+0x90,
+0x00,
+0x00,
+0x88,
+0x2b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2c,
+0x38,
+0x12,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x09,
+0x08,
+0x63,
+0x88,
+0x6c,
+0x40,
+0x03,
+0xd2,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x2e,
+0x08,
+0x08,
+0x40,
+0x09,
+0x80,
+0x09,
+0xba,
+0x14,
+0x89,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x80,
+0xa2,
+0x16,
+0x48,
+0x20,
+0x0a,
+0x84,
+0x00,
+0x84,
+0x41,
+0x00,
+0x02,
+0x00,
+0xa8,
+0x40,
+0x88,
+0x44,
+0x14,
+0x00,
+0x60,
+0x08,
+0xba,
+0x14,
+0x84,
+0x42,
+0x40,
+0x04,
+0x04,
+0x99,
+0x80,
+0x08,
+0xa2,
+0x24,
+0x48,
+0x60,
+0x08,
+0x44,
+0x20,
+0x00,
+0x42,
+0xc9,
+0x98,
+0x00,
+0x98,
+0x60,
+0x88,
+0x44,
+0x20,
+0x01,
+0x68,
+0xa5,
+0xba,
+0x14,
+0x82,
+0x01,
+0x40,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x5c,
+0x83,
+0x08,
+0x43,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x0a,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x00,
+0x58,
+0x0c,
+0x02,
+0xbf,
+0xd0,
+0x80,
+0x08,
+0xba,
+0x08,
+0x42,
+0x84,
+0x00,
+0x19,
+0x88,
+0x42,
+0x88,
+0x0c,
+0x88,
+0x81,
+0x49,
+0x42,
+0x13,
+0x40,
+0x80,
+0x4a,
+0x88,
+0x1c,
+0xb3,
+0x20,
+0x10,
+0xbc,
+0x21,
+0x06,
+0x82,
+0x00,
+0x11,
+0xb2,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x5c,
+0x38,
+0x40,
+0x08,
+0x85,
+0x80,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x10,
+0xd8,
+0x4a,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x48,
+0x8a,
+0x84,
+0x80,
+0x32,
+0x80,
+0xf4,
+0x22,
+0xfe,
+0x75,
+0x54,
+0x3c,
+0x98,
+0x44,
+0xb1,
+0x51,
+0x73,
+0x51,
+0x42,
+0x60,
+0x10,
+0x53,
+0x81,
+0x2c,
+0x84,
+0x20,
+0x1f,
+0x85,
+0x04,
+0x95,
+0x50,
+0x1c,
+0x85,
+0x0c,
+0x93,
+0x90,
+0x20,
+0x00,
+0x00,
+0x08,
+0x48,
+0x88,
+0x57,
+0x4b,
+0x30,
+0x48,
+0x08,
+0x57,
+0x8b,
+0x20,
+0x48,
+0xca,
+0xbc,
+0x04,
+0xf8,
+0x48,
+0x48,
+0xbc,
+0x02,
+0xf3,
+0x90,
+0x20,
+0x39,
+0x02,
+0x03,
+0x81,
+0x25,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2c,
+0x25,
+0x96,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xa0,
+0xab,
+0xc0,
+0x59,
+0x55,
+0x01,
+0xa0,
+0x82,
+0x4a,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x24,
+0xe0,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x56,
+0x09,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x88,
+0x88,
+0x10,
+0xab,
+0x1f,
+0x47,
+0x30,
+0x1f,
+0x0b,
+0xc1,
+0x2c,
+0x98,
+0xe8,
+0x88,
+0x82,
+0x08,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xe7,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xae,
+0x38,
+0x13,
+0x72,
+0x59,
+0xf0,
+0xbc,
+0x09,
+0x16,
+0xc4,
+0x00,
+0x15,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x2b,
+0xc0,
+0xb3,
+0x01,
+0xf0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x23,
+0x6c,
+0x40,
+0x02,
+0x4e,
+0x08,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x26,
+0x59,
+0x03,
+0x80,
+0x81,
+0x8a,
+0x28,
+0x13,
+0x35,
+0x70,
+0x99,
+0x98,
+0x0c,
+0x8b,
+0xc0,
+0xf8,
+0x98,
+0x0c,
+0xb6,
+0xc4,
+0x00,
+0x25,
+0x00,
+0x33,
+0x20,
+0x18,
+0xbc,
+0x0a,
+0x13,
+0x01,
+0x88,
+0xbc,
+0x06,
+0x45,
+0x80,
+0xc4,
+0x08,
+0x18,
+0x8b,
+0xc0,
+0xfb,
+0x88,
+0x00,
+0x19,
+0x82,
+0xc8,
+0xbc,
+0x0c,
+0x7b,
+0xc0,
+0xbf,
+0x88,
+0x00,
+0x18,
+0x80,
+0x81,
+0x32,
+0x00,
+0x8b,
+0xc0,
+0x64,
+0x32,
+0x00,
+0x8b,
+0xc0,
+0x5b,
+0x55,
+0x01,
+0xa0,
+0x80,
+0x01,
+0x98,
+0x2c,
+0x8b,
+0xc0,
+0x17,
+0x88,
+0x00,
+0x13,
+0x00,
+0x40,
+0xbc,
+0x44,
+0x03,
+0x20,
+0x10,
+0xbc,
+0x42,
+0x06,
+0xc0,
+0x00,
+0x36,
+0x80,
+0xa3,
+0x20,
+0x30,
+0xbc,
+0x05,
+0x98,
+0x82,
+0x0a,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc3,
+0x90,
+0x30,
+0x17,
+0x0b,
+0xc3,
+0x70,
+0x68,
+0x20,
+0x01,
+0x0b,
+0x20,
+0x84,
+0x90,
+0xa8,
+0x00,
+0x00,
+0x30,
+0x03,
+0x0b,
+0xc0,
+0x22,
+0xbc,
+0x04,
+0xf8,
+0x4a,
+0xfa,
+0x84,
+0xa8,
+0xa2,
+0xa0,
+0x76,
+0x84,
+0xac,
+0xa0,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa6,
+0xc0,
+0x00,
+0x36,
+0x60,
+0x03,
+0x01,
+0x80,
+0xbc,
+0x04,
+0x56,
+0xc0,
+0x00,
+0x37,
+0x40,
+0x03,
+0x01,
+0x80,
+0xbc,
+0x03,
+0x4b,
+0xc2,
+0x1f,
+0x40,
+0x00,
+0x00,
+0x4a,
+0x08,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x5c,
+0x82,
+0x08,
+0x08,
+0x7a,
+0x80,
+0x07,
+0xaa,
+0x08,
+0x02,
+0x80,
+0x87,
+0xaa,
+0x00,
+0x04,
+0x80,
+0x07,
+0xa8,
+0x50,
+0xfa,
+0x86,
+0x0f,
+0xa8,
+0x48,
+0xfa,
+0x68,
+0x00,
+0x01,
+0xb9,
+0x23,
+0x84,
+0x0f,
+0xa6,
+0x80,
+0x00,
+0x1b,
+0x22,
+0x16,
+0x82,
+0x00,
+0x1f,
+0x12,
+0x08,
+0x08,
+0x80,
+0x85,
+0x80,
+0xa6,
+0xc4,
+0x00,
+0x20,
+0xa4,
+0x96,
+0xc0,
+0x00,
+0x14,
+0xc7,
+0xa8,
+0x48,
+0x7a,
+0x85,
+0x97,
+0xa8,
+0x40,
+0x50,
+0x84,
+0x0c,
+0xab,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x85,
+0x00,
+0x88,
+0x40,
+0x09,
+0x57,
+0x8b,
+0x22,
+0xbf,
+0xd0,
+0x85,
+0x04,
+0x88,
+0x80,
+0x63,
+0x88,
+0x0e,
+0x28,
+0x81,
+0x61,
+0x88,
+0x1e,
+0x08,
+0x82,
+0x76,
+0x00,
+0x00,
+0x08,
+0x40,
+0x09,
+0x36,
+0x14,
+0x52,
+0xe9,
+0x64,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x85,
+0x80,
+0x88,
+0x48,
+0x09,
+0x2f,
+0x16,
+0x48,
+0x58,
+0x48,
+0x00,
+0x00,
+0x08,
+0x48,
+0x09,
+0x36,
+0x14,
+0x52,
+0xe9,
+0x64,
+0x44,
+0x00,
+0x00,
+0x58,
+0x48,
+0x98,
+0x00,
+0x98,
+0x50,
+0x08,
+0x44,
+0x00,
+0x00,
+0x48,
+0x08,
+0x44,
+0x01,
+0x00,
+0x40,
+0x08,
+0x44,
+0x01,
+0x01,
+0xa0,
+0x00,
+0x44,
+0x40,
+0x01,
+0x80,
+0x88,
+0x08,
+0x18,
+0x06,
+0x60,
+0x00,
+0x13,
+0x52,
+0x89,
+0x81,
+0x09,
+0x88,
+0x12,
+0x08,
+0x81,
+0xa1,
+0x84,
+0x00,
+0x98,
+0x48,
+0x0a,
+0x44,
+0x48,
+0x00,
+0x80,
+0xa0,
+0x88,
+0x02,
+0x19,
+0x80,
+0x0b,
+0x84,
+0x00,
+0x94,
+0x46,
+0x80,
+0x04,
+0x80,
+0xa4,
+0x47,
+0x10,
+0x18,
+0x00,
+0xb4,
+0x46,
+0x00,
+0x18,
+0x28,
+0xb5,
+0x11,
+0xe0,
+0x18,
+0x08,
+0xa4,
+0x44,
+0x10,
+0x18,
+0x00,
+0x83,
+0x20,
+0x20,
+0x51,
+0x1e,
+0x83,
+0xc0,
+0x29,
+0x98,
+0x00,
+0xa9,
+0x82,
+0x48,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x11,
+0x98,
+0x2c,
+0xa3,
+0x69,
+0x40,
+0x5b,
+0x48,
+0x01,
+0x80,
+0x09,
+0x98,
+0x00,
+0x03,
+0x00,
+0x28,
+0xbc,
+0x06,
+0x43,
+0x69,
+0xc0,
+0x5b,
+0x4c,
+0x01,
+0x80,
+0x09,
+0x98,
+0x00,
+0xb3,
+0x01,
+0xe8,
+0xbc,
+0x04,
+0x58,
+0x40,
+0x48,
+0x42,
+0x01,
+0x7b,
+0x00,
+0x0c,
+0x84,
+0x84,
+0xa9,
+0x8e,
+0x88,
+0x00,
+0x00,
+0x08,
+0x82,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x46,
+0x08,
+0x8a,
+0xbf,
+0xe0,
+0x51,
+0x61,
+0xb0,
+0x80,
+0x49,
+0x51,
+0x61,
+0xe0,
+0x80,
+0xf6,
+0x1a,
+0x0a,
+0x29,
+0x83,
+0x09,
+0x98,
+0x38,
+0x82,
+0x81,
+0x2d,
+0x88,
+0x14,
+0x96,
+0x60,
+0x00,
+0x13,
+0x52,
+0x8b,
+0xa1,
+0x01,
+0x88,
+0x10,
+0x94,
+0x40,
+0x80,
+0x08,
+0x00,
+0x99,
+0x80,
+0x08,
+0x23,
+0x42,
+0x43,
+0x01,
+0x60,
+0xbc,
+0x02,
+0x5b,
+0xc0,
+0x2f,
+0x2e,
+0x16,
+0x49,
+0x8e,
+0x88,
+0x88,
+0x0b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x00,
+0x01,
+0x3a,
+0x22,
+0x55,
+0x03,
+0x20,
+0x00,
+0x7a,
+0x5d,
+0x08,
+0x20,
+0x08,
+0x7a,
+0x32,
+0x2a,
+0x0b,
+0xff,
+0xaa,
+0x81,
+0x07,
+0xab,
+0xa1,
+0x40,
+0x5c,
+0x81,
+0x00,
+0x40,
+0x08,
+0x82,
+0x00,
+0xa4,
+0x41,
+0x00,
+0x04,
+0x80,
+0xa8,
+0x20,
+0x08,
+0x44,
+0x44,
+0x00,
+0x60,
+0x08,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x0a,
+0x08,
+0x80,
+0x09,
+0x80,
+0x08,
+0x68,
+0x00,
+0x00,
+0x6c,
+0x20,
+0x68,
+0x20,
+0x00,
+0x24,
+0x24,
+0x5c,
+0x81,
+0x02,
+0xc0,
+0x41,
+0x68,
+0x20,
+0x00,
+0x4b,
+0x21,
+0x5c,
+0x08,
+0xad,
+0x00,
+0x81,
+0x44,
+0x10,
+0x00,
+0x08,
+0x88,
+0x84,
+0x00,
+0xb4,
+0x46,
+0x08,
+0x2b,
+0xf7,
+0x06,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xe6,
+0xe4,
+0x00,
+0x11,
+0x22,
+0x75,
+0x2c,
+0xb8,
+0x08,
+0x16,
+0x18,
+0x81,
+0xe4,
+0x88,
+0x27,
+0x68,
+0x82,
+0xd5,
+0x90,
+0x45,
+0x84,
+0x22,
+0x3c,
+0x88,
+0x55,
+0x49,
+0x03,
+0x59,
+0x68,
+0x20,
+0x00,
+0x85,
+0x24,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x0a,
+0x66,
+0x00,
+0x00,
+0xe4,
+0xc8,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x09,
+0x68,
+0x20,
+0x00,
+0x85,
+0x20,
+0x5c,
+0x82,
+0x40,
+0x88,
+0x48,
+0xa0,
+0x08,
+0xc6,
+0xc0,
+0x00,
+0x12,
+0xa0,
+0x96,
+0xc0,
+0x00,
+0x11,
+0x80,
+0xa6,
+0x60,
+0x00,
+0x0e,
+0x4c,
+0x89,
+0x62,
+0x07,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x5c,
+0x08,
+0xe8,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x90,
+0x2c,
+0x52,
+0xcb,
+0x80,
+0x85,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x88,
+0x06,
+0xcb,
+0xc1,
+0xa9,
+0x40,
+0x00,
+0x00,
+0x88,
+0x09,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x0a,
+0x44,
+0x10,
+0x00,
+0x85,
+0x89,
+0x68,
+0x20,
+0x00,
+0x98,
+0xa0,
+0x88,
+0x84,
+0x88,
+0x80,
+0x60,
+0x68,
+0x20,
+0x00,
+0x92,
+0xa4,
+0x68,
+0x20,
+0x00,
+0x95,
+0xa5,
+0x90,
+0x65,
+0x88,
+0x85,
+0xd4,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x40,
+0x00,
+0x03,
+0xc5,
+0x4f,
+0x44,
+0x10,
+0x00,
+0x88,
+0xc8,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x08,
+0x44,
+0x20,
+0x80,
+0x85,
+0x88,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x42,
+0x25,
+0x79,
+0x06,
+0x59,
+0x44,
+0x10,
+0x00,
+0x85,
+0xd5,
+0x68,
+0x00,
+0x00,
+0x6e,
+0x20,
+0x68,
+0x20,
+0x00,
+0x85,
+0x24,
+0x80,
+0x00,
+0x98,
+0x85,
+0xe0,
+0x66,
+0x00,
+0x00,
+0xe4,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x40,
+0x0a,
+0x5c,
+0x82,
+0x40,
+0x85,
+0xa0,
+0x68,
+0x20,
+0x00,
+0x85,
+0x21,
+0x88,
+0x84,
+0x8a,
+0x08,
+0x8c,
+0x84,
+0x00,
+0x96,
+0xc0,
+0x00,
+0x0d,
+0xc0,
+0xa6,
+0x60,
+0x00,
+0x0e,
+0x4c,
+0x89,
+0x62,
+0x07,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x5c,
+0x08,
+0xe8,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x90,
+0x2c,
+0x52,
+0xcb,
+0x80,
+0x85,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x88,
+0x06,
+0xcb,
+0xc1,
+0xa9,
+0x40,
+0x00,
+0x00,
+0x88,
+0x09,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x0a,
+0x44,
+0x10,
+0x00,
+0x85,
+0x89,
+0x68,
+0x20,
+0x00,
+0x98,
+0xa0,
+0x88,
+0x84,
+0x88,
+0x80,
+0x60,
+0x68,
+0x20,
+0x00,
+0x92,
+0xa4,
+0x68,
+0x20,
+0x00,
+0x95,
+0xa5,
+0x90,
+0x65,
+0x88,
+0x85,
+0xd4,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xcf,
+0x44,
+0x10,
+0x00,
+0x88,
+0xc8,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x08,
+0x44,
+0x20,
+0x80,
+0x85,
+0x88,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x44,
+0x10,
+0x01,
+0x06,
+0x59,
+0x40,
+0x00,
+0x00,
+0x85,
+0xd5,
+0x68,
+0x00,
+0x00,
+0x50,
+0x20,
+0x90,
+0x61,
+0x1a,
+0x04,
+0x21,
+0x88,
+0x59,
+0x58,
+0x85,
+0xe1,
+0x90,
+0x65,
+0x88,
+0x87,
+0x54,
+0x66,
+0x00,
+0x01,
+0x31,
+0x28,
+0x5c,
+0x00,
+0x71,
+0x80,
+0x49,
+0x90,
+0x61,
+0x06,
+0x80,
+0x00,
+0x06,
+0x42,
+0x08,
+0x87,
+0x14,
+0xa0,
+0x42,
+0x18,
+0x86,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x86,
+0x61,
+0x66,
+0x00,
+0x01,
+0x31,
+0x28,
+0x5c,
+0x00,
+0x71,
+0x80,
+0x09,
+0x5c,
+0x82,
+0x10,
+0x86,
+0x20,
+0x5c,
+0x8c,
+0x01,
+0x03,
+0x10,
+0x5c,
+0x8e,
+0x09,
+0x04,
+0x12,
+0x88,
+0x12,
+0x18,
+0x85,
+0xa5,
+0xa0,
+0x56,
+0x48,
+0x82,
+0x94,
+0x54,
+0x08,
+0x18,
+0x86,
+0x8a,
+0x88,
+0x51,
+0x65,
+0x40,
+0xc8,
+0x88,
+0x1a,
+0x28,
+0x48,
+0x08,
+0xa2,
+0xd6,
+0x08,
+0x21,
+0x0a,
+0x4d,
+0x86,
+0x80,
+0x50,
+0x08,
+0x80,
+0x10,
+0xa4,
+0xf8,
+0x48,
+0x02,
+0x04,
+0x38,
+0x00,
+0x41,
+0x82,
+0x2c,
+0x08,
+0x02,
+0xc2,
+0x88,
+0x16,
+0x46,
+0x82,
+0x00,
+0x07,
+0xe2,
+0xc8,
+0x81,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x1c,
+0x21,
+0x68,
+0x20,
+0x00,
+0x24,
+0x24,
+0x68,
+0x20,
+0x00,
+0x72,
+0x25,
+0x40,
+0x00,
+0x00,
+0x80,
+0x6c,
+0x68,
+0x00,
+0x01,
+0x27,
+0x22,
+0x66,
+0x00,
+0x00,
+0xe0,
+0x40,
+0x68,
+0x20,
+0x00,
+0x7f,
+0xac,
+0x88,
+0x84,
+0x86,
+0x80,
+0x00,
+0x03,
+0x02,
+0x18,
+0x81,
+0x20,
+0x68,
+0x20,
+0x00,
+0x4b,
+0x24,
+0x68,
+0x20,
+0x00,
+0x78,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x80,
+0x00,
+0x12,
+0xa2,
+0x26,
+0x60,
+0x00,
+0x0e,
+0x04,
+0x08,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x83,
+0x20,
+0x68,
+0x20,
+0x00,
+0x84,
+0x21,
+0xa4,
+0x22,
+0x36,
+0x60,
+0x00,
+0x0c,
+0x52,
+0x8a,
+0x42,
+0x02,
+0x5c,
+0x8d,
+0x00,
+0x81,
+0xa0,
+0x5c,
+0x08,
+0x71,
+0x8e,
+0x88,
+0xa0,
+0x26,
+0x09,
+0x42,
+0x0f,
+0x25,
+0x9b,
+0x8b,
+0xc0,
+0x39,
+0x55,
+0x01,
+0x28,
+0x81,
+0x21,
+0x88,
+0x80,
+0x98,
+0x40,
+0x0b,
+0x54,
+0x0f,
+0x42,
+0x0a,
+0x61,
+0x94,
+0xa0,
+0xd2,
+0x59,
+0xa8,
+0xbc,
+0x03,
+0x98,
+0x40,
+0xc0,
+0x00,
+0x00,
+0x08,
+0x88,
+0x88,
+0x00,
+0x00,
+0x08,
+0x48,
+0x0a,
+0x54,
+0x0d,
+0x00,
+0x82,
+0x36,
+0xba,
+0x14,
+0x88,
+0x48,
+0xc0,
+0x40,
+0x00,
+0x02,
+0x80,
+0x90,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xc0,
+0x5c,
+0x8f,
+0x08,
+0x81,
+0x60,
+0xa0,
+0x04,
+0x3a,
+0x08,
+0x00,
+0x81,
+0xa0,
+0x9a,
+0x1a,
+0x21,
+0x85,
+0x80,
+0x85,
+0x70,
+0x94,
+0x00,
+0xa8,
+0x95,
+0x70,
+0xa0,
+0x08,
+0x1e,
+0x1a,
+0x10,
+0x01,
+0x81,
+0x00,
+0x99,
+0x80,
+0x08,
+0x54,
+0x0b,
+0x02,
+0x28,
+0x26,
+0x98,
+0x00,
+0x98,
+0x49,
+0x49,
+0xa2,
+0x80,
+0x18,
+0x84,
+0x25,
+0x85,
+0x04,
+0x8a,
+0x30,
+0x23,
+0xa3,
+0x00,
+0x2a,
+0x18,
+0x26,
+0xa2,
+0x06,
+0x48,
+0x80,
+0x66,
+0x88,
+0x26,
+0x68,
+0x82,
+0xe4,
+0x88,
+0x36,
+0x08,
+0x83,
+0xf6,
+0x66,
+0x00,
+0x00,
+0xec,
+0x60,
+0x5c,
+0x8f,
+0x00,
+0x81,
+0xa0,
+0x5c,
+0x08,
+0x28,
+0x82,
+0xa4,
+0xa0,
+0x20,
+0x59,
+0x6a,
+0x0f,
+0x52,
+0xcb,
+0xc0,
+0x40,
+0x48,
+0xbc,
+0x05,
+0x95,
+0xc0,
+0x03,
+0x08,
+0x32,
+0x08,
+0x81,
+0x22,
+0x00,
+0x00,
+0x08,
+0x51,
+0x8a,
+0x86,
+0x80,
+0x95,
+0x40,
+0xb8,
+0x20,
+0x14,
+0x09,
+0x80,
+0x09,
+0xa2,
+0x2e,
+0x48,
+0x68,
+0xc9,
+0x88,
+0x16,
+0x48,
+0x81,
+0xe0,
+0x66,
+0x00,
+0x00,
+0xf3,
+0xa0,
+0x88,
+0x22,
+0x58,
+0x84,
+0x22,
+0x88,
+0x12,
+0x48,
+0x81,
+0xa0,
+0xa2,
+0x82,
+0x18,
+0x83,
+0xb6,
+0xa1,
+0x02,
+0x5a,
+0x80,
+0x40,
+0xa0,
+0x04,
+0x0a,
+0x20,
+0x64,
+0x64,
+0x00,
+0x00,
+0xf4,
+0x8f,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x22,
+0x86,
+0x00,
+0x84,
+0x42,
+0x00,
+0x06,
+0x08,
+0x95,
+0x00,
+0xe0,
+0x3a,
+0x14,
+0x80,
+0x89,
+0x80,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x6e,
+0x40,
+0x00,
+0x08,
+0x3c,
+0x38,
+0x15,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x30,
+0x05,
+0x15,
+0x96,
+0x16,
+0x1b,
+0x65,
+0x19,
+0x5b,
+0x08,
+0x02,
+0x04,
+0x41,
+0x00,
+0x18,
+0xe8,
+0xa9,
+0x80,
+0x08,
+0x23,
+0x07,
+0x79,
+0x82,
+0xe8,
+0x9c,
+0x00,
+0x19,
+0x48,
+0x37,
+0x30,
+0x1e,
+0x0b,
+0xc0,
+0x42,
+0x2a,
+0x07,
+0x63,
+0xb1,
+0x86,
+0x32,
+0x17,
+0x0b,
+0xff,
+0x62,
+0x32,
+0x0f,
+0x0b,
+0xc1,
+0xd0,
+0x51,
+0x83,
+0xab,
+0x01,
+0x00,
+0x50,
+0x41,
+0x11,
+0x82,
+0x68,
+0x55,
+0x00,
+0xb1,
+0xe0,
+0x04,
+0x46,
+0x08,
+0x89,
+0x60,
+0x35,
+0x50,
+0x41,
+0x49,
+0x83,
+0x89,
+0x55,
+0x00,
+0x79,
+0x83,
+0x48,
+0x44,
+0x41,
+0x09,
+0xe8,
+0x05,
+0x44,
+0x6d,
+0x49,
+0x68,
+0x36,
+0x21,
+0x01,
+0x00,
+0x84,
+0x82,
+0x37,
+0x80,
+0x02,
+0x28,
+0x80,
+0x6c,
+0x40,
+0x01,
+0x36,
+0x09,
+0x51,
+0x87,
+0xb1,
+0x83,
+0x08,
+0x36,
+0x14,
+0x72,
+0x81,
+0xa4,
+0x2f,
+0x16,
+0x52,
+0xe9,
+0xed,
+0x23,
+0x32,
+0xdb,
+0xa1,
+0x48,
+0x46,
+0x08,
+0x09,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x8a,
+0x68,
+0x00,
+0x00,
+0x1c,
+0x20,
+0x38,
+0x0a,
+0x42,
+0xa0,
+0x76,
+0x3a,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x8a,
+0x68,
+0x00,
+0x00,
+0x30,
+0x20,
+0x38,
+0x0a,
+0x42,
+0xa0,
+0x76,
+0x3a,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x44,
+0x20,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xf0,
+0x68,
+0x00,
+0x00,
+0x58,
+0x21,
+0x88,
+0x07,
+0x69,
+0x8e,
+0x88,
+0x40,
+0x00,
+0x02,
+0x00,
+0x02,
+0x55,
+0x03,
+0x20,
+0x08,
+0x7a,
+0x3a,
+0x10,
+0x43,
+0x22,
+0x20,
+0xbf,
+0xfb,
+0xa8,
+0x10,
+0x7a,
+0x40,
+0x00,
+0x02,
+0x01,
+0x80,
+0x66,
+0x00,
+0x01,
+0x30,
+0x68,
+0x5c,
+0x00,
+0x62,
+0x04,
+0x21,
+0x68,
+0x00,
+0x00,
+0x58,
+0x20,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x10,
+0x40,
+0x00,
+0x02,
+0x01,
+0x80,
+0x64,
+0x00,
+0x01,
+0x30,
+0x6f,
+0x5c,
+0x00,
+0x62,
+0x04,
+0x21,
+0x55,
+0x01,
+0x40,
+0x60,
+0x0b,
+0x5c,
+0x82,
+0x0a,
+0x20,
+0x26,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x02,
+0x30,
+0x27,
+0x87,
+0x00,
+0x84,
+0x44,
+0x54,
+0x07,
+0x80,
+0x84,
+0x42,
+0x54,
+0x23,
+0x82,
+0x49,
+0x80,
+0x82,
+0x8c,
+0x0c,
+0x0a,
+0x20,
+0x26,
+0x86,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x80,
+0x70,
+0x0b,
+0x55,
+0x00,
+0xa2,
+0x30,
+0x27,
+0x44,
+0x5d,
+0x60,
+0x78,
+0x0a,
+0x44,
+0x15,
+0x40,
+0x48,
+0x0b,
+0x55,
+0x01,
+0x09,
+0x80,
+0x8a,
+0x57,
+0x8f,
+0xb0,
+0x50,
+0x08,
+0x57,
+0x49,
+0x92,
+0x38,
+0x24,
+0x8c,
+0x0c,
+0x4a,
+0x20,
+0x22,
+0x86,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x82,
+0x10,
+0x21,
+0x85,
+0x00,
+0x84,
+0x44,
+0x5e,
+0x04,
+0x80,
+0x84,
+0x42,
+0x5e,
+0x20,
+0x82,
+0x69,
+0x80,
+0xc3,
+0x8c,
+0x0c,
+0x1a,
+0x30,
+0x22,
+0x87,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x01,
+0x84,
+0x88,
+0x85,
+0x00,
+0xb4,
+0x45,
+0xd4,
+0x21,
+0x02,
+0x15,
+0x50,
+0x0f,
+0x84,
+0x80,
+0xa4,
+0x47,
+0x54,
+0x20,
+0x82,
+0x45,
+0x50,
+0x1c,
+0x18,
+0x08,
+0x28,
+0xc0,
+0xc0,
+0xa2,
+0x02,
+0x28,
+0x60,
+0x0b,
+0x8c,
+0x02,
+0x64,
+0x45,
+0xd8,
+0x05,
+0x00,
+0xb4,
+0x45,
+0x88,
+0x21,
+0x02,
+0x19,
+0x80,
+0xc3,
+0x55,
+0x00,
+0xf0,
+0x48,
+0x0b,
+0xa0,
+0x82,
+0x44,
+0x45,
+0xda,
+0x0c,
+0x06,
+0x15,
+0x40,
+0x4d,
+0x22,
+0x02,
+0x28,
+0x60,
+0x0a,
+0x44,
+0x31,
+0x80,
+0x50,
+0x09,
+0x44,
+0x08,
+0x01,
+0x80,
+0x89,
+0x96,
+0x82,
+0x45,
+0x00,
+0x8d,
+0x21,
+0x02,
+0x08,
+0x40,
+0x08,
+0x44,
+0x21,
+0x82,
+0x28,
+0x09,
+0xa0,
+0x02,
+0x48,
+0x20,
+0x08,
+0x98,
+0x08,
+0xa4,
+0x44,
+0x10,
+0x20,
+0x80,
+0xd9,
+0x48,
+0x25,
+0x50,
+0x0a,
+0x01,
+0x68,
+0x24,
+0x50,
+0x08,
+0xd8,
+0x20,
+0x08,
+0x98,
+0x00,
+0x94,
+0x42,
+0x54,
+0x02,
+0x00,
+0x99,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0x40,
+0x80,
+0x20,
+0x85,
+0x80,
+0x99,
+0x80,
+0x88,
+0x57,
+0x8b,
+0x28,
+0x40,
+0x08,
+0x57,
+0x49,
+0x68,
+0x60,
+0x08,
+0x44,
+0x21,
+0x01,
+0x68,
+0xa5,
+0x50,
+0x0a,
+0x90,
+0x60,
+0x89,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x88,
+0x08,
+0x12,
+0x09,
+0x80,
+0x88,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x06,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x04,
+0x60,
+0xa4,
+0x18,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x5c,
+0x00,
+0x02,
+0xbf,
+0xf0,
+0x55,
+0x00,
+0x11,
+0x8e,
+0x81,
+0x5c,
+0x81,
+0x00,
+0x80,
+0x49,
+0x5c,
+0x82,
+0x0a,
+0x00,
+0x03,
+0x55,
+0x00,
+0x5b,
+0x80,
+0x00,
+0x55,
+0x02,
+0xe8,
+0x20,
+0x08,
+0x5d,
+0x0a,
+0x18,
+0x20,
+0x0b,
+0x59,
+0x04,
+0xc0,
+0xc0,
+0x8e,
+0xbf,
+0xf9,
+0xa1,
+0xb4,
+0xb3,
+0x5c,
+0x00,
+0x18,
+0x20,
+0x09,
+0x88,
+0x00,
+0x84,
+0x40,
+0xd4,
+0x16,
+0x82,
+0x55,
+0x40,
+0x08,
+0x0d,
+0x83,
+0x25,
+0x00,
+0xa0,
+0x0d,
+0x97,
+0xa5,
+0x50,
+0x10,
+0x18,
+0x00,
+0x25,
+0xc0,
+0x00,
+0x0d,
+0x86,
+0x08,
+0x80,
+0x52,
+0x40,
+0x00,
+0x02,
+0x18,
+0x86,
+0x55,
+0x02,
+0x68,
+0x20,
+0x08,
+0x5d,
+0x0a,
+0x08,
+0x20,
+0x0b,
+0x59,
+0x04,
+0x40,
+0xc0,
+0x8e,
+0xbf,
+0xf9,
+0xa1,
+0xb4,
+0xf3,
+0x86,
+0x00,
+0x88,
+0x80,
+0x09,
+0x44,
+0x25,
+0x61,
+0x68,
+0xa4,
+0x54,
+0x00,
+0x80,
+0xda,
+0x32,
+0x50,
+0x08,
+0x00,
+0xf1,
+0x7a,
+0x55,
+0x01,
+0x41,
+0x80,
+0x02,
+0x40,
+0x00,
+0x00,
+0x48,
+0x09,
+0x46,
+0x0a,
+0x40,
+0x50,
+0x08,
+0x57,
+0x8a,
+0xa8,
+0xda,
+0x60,
+0x57,
+0x49,
+0x62,
+0x80,
+0x10,
+0xab,
+0xfa,
+0x06,
+0x80,
+0x04,
+0x00,
+0x08,
+0x8b,
+0x00,
+0x10,
+0x88,
+0x07,
+0x6a,
+0x40,
+0xc0,
+0x88,
+0x0c,
+0x88,
+0x81,
+0x50,
+0x88,
+0x1e,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x00,
+0x55,
+0x02,
+0x10,
+0x81,
+0xa0,
+0x5d,
+0x04,
+0x10,
+0x80,
+0x80,
+0x59,
+0x10,
+0x80,
+0x81,
+0x52,
+0x55,
+0x02,
+0x22,
+0x00,
+0x20,
+0xbf,
+0xf3,
+0xa8,
+0x80,
+0xc8,
+0x5c,
+0x81,
+0x02,
+0x40,
+0xc0,
+0x80,
+0x00,
+0x25,
+0x94,
+0x08,
+0x00,
+0x00,
+0x88,
+0x00,
+0x0a,
+0x42,
+0x0b,
+0xe8,
+0x81,
+0x52,
+0x5c,
+0x00,
+0x00,
+0x80,
+0xe0,
+0xbc,
+0x0d,
+0xf8,
+0x81,
+0xca,
+0x98,
+0x2a,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x80,
+0x55,
+0x08,
+0x30,
+0x82,
+0x82,
+0x55,
+0x02,
+0xa0,
+0x82,
+0x00,
+0x55,
+0x02,
+0x00,
+0x81,
+0x02,
+0x40,
+0x00,
+0x00,
+0x81,
+0xca,
+0x58,
+0x04,
+0x00,
+0x82,
+0xc8,
+0xbf,
+0xf1,
+0xa8,
+0x82,
+0x50,
+0xbc,
+0x04,
+0xf5,
+0xc8,
+0x10,
+0x08,
+0x0a,
+0x03,
+0x90,
+0x20,
+0x88,
+0x0a,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x25,
+0x94,
+0x08,
+0x04,
+0x00,
+0x88,
+0x80,
+0xd2,
+0xbc,
+0x13,
+0xd5,
+0xc0,
+0x00,
+0x04,
+0x08,
+0xab,
+0xc0,
+0xcf,
+0x88,
+0x14,
+0xa9,
+0x82,
+0xa0,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa0,
+0x88,
+0x10,
+0x05,
+0x50,
+0x83,
+0x08,
+0x20,
+0x25,
+0x50,
+0x2a,
+0x08,
+0x18,
+0x05,
+0x50,
+0x20,
+0x08,
+0x08,
+0x28,
+0x81,
+0x4a,
+0x58,
+0x04,
+0x00,
+0x82,
+0x48,
+0xbf,
+0xf2,
+0xa8,
+0x81,
+0xd0,
+0x00,
+0x00,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x60,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x66,
+0x00,
+0x00,
+0xea,
+0x60,
+0x40,
+0x00,
+0x03,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x00,
+0xae,
+0x64,
+0x6e,
+0x00,
+0x00,
+0xd6,
+0x64,
+0x66,
+0x00,
+0x00,
+0xcc,
+0x80,
+0x66,
+0x00,
+0x01,
+0xb3,
+0x40,
+0x88,
+0x03,
+0x64,
+0x60,
+0xa4,
+0x30,
+0x30,
+0xc6,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x4a,
+0x80,
+0x10,
+0xab,
+0xfd,
+0x08,
+0x80,
+0xc9,
+0x5c,
+0x09,
+0xf8,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x59,
+0x02,
+0x5b,
+0x88,
+0x1d,
+0x70,
+0x00,
+0x00,
+0x6e,
+0x00,
+0x00,
+0xae,
+0x2d,
+0x25,
+0x9e,
+0x8b,
+0xc0,
+0x60,
+0x6c,
+0x68,
+0x08,
+0x10,
+0x03,
+0x6c,
+0x00,
+0x00,
+0xd8,
+0x53,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x67,
+0x6c,
+0x68,
+0x08,
+0x16,
+0x03,
+0x6c,
+0x00,
+0x00,
+0xd8,
+0x53,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x00,
+0xd6,
+0x2b,
+0x25,
+0x9d,
+0x8b,
+0xc0,
+0x50,
+0x6c,
+0x68,
+0x08,
+0x12,
+0x09,
+0x6c,
+0x00,
+0x00,
+0xda,
+0x49,
+0xbc,
+0x07,
+0x72,
+0x59,
+0xe8,
+0xbc,
+0x05,
+0x06,
+0xc6,
+0x80,
+0x81,
+0x60,
+0x96,
+0xc0,
+0x00,
+0x0d,
+0xa4,
+0x90,
+0x00,
+0x00,
+0x5c,
+0x02,
+0x29,
+0x02,
+0x13,
+0x68,
+0x00,
+0x00,
+0x00,
+0x75,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x75,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x49,
+0x00,
+0x00,
+0x08,
+0x81,
+0x97,
+0x88,
+0x00,
+0xb4,
+0x60,
+0xb4,
+0x08,
+0x13,
+0x58,
+0x80,
+0x89,
+0x40,
+0x00,
+0x02,
+0x80,
+0x30,
+0xab,
+0xef,
+0x08,
+0x8a,
+0xc8,
+0x5c,
+0x09,
+0x78,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x58,
+0x82,
+0x6f,
+0x88,
+0x2e,
+0xe8,
+0x83,
+0x6d,
+0x88,
+0x3e,
+0xc8,
+0x84,
+0x6b,
+0x88,
+0x4e,
+0xa8,
+0x85,
+0x69,
+0x88,
+0x5e,
+0x88,
+0x86,
+0x67,
+0x88,
+0x6e,
+0x68,
+0x87,
+0x65,
+0x88,
+0x7e,
+0x48,
+0x88,
+0x63,
+0x88,
+0x8e,
+0x28,
+0x89,
+0x61,
+0x88,
+0x9e,
+0x08,
+0x8a,
+0x4a,
+0x88,
+0x0c,
+0x99,
+0x0b,
+0x5b,
+0x88,
+0xc5,
+0x79,
+0x0d,
+0x59,
+0x88,
+0xcd,
+0x59,
+0x0e,
+0x5a,
+0x88,
+0xf5,
+0x69,
+0x10,
+0x58,
+0x88,
+0xfd,
+0x40,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9e,
+0x0b,
+0xc0,
+0xd8,
+0x88,
+0x1f,
+0x63,
+0x81,
+0x36,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x48,
+0xba,
+0x10,
+0x16,
+0x60,
+0x00,
+0x17,
+0x88,
+0x0b,
+0xc1,
+0x47,
+0x66,
+0x00,
+0x01,
+0x74,
+0x40,
+0xbc,
+0x11,
+0x73,
+0x81,
+0x26,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0xa8,
+0xba,
+0x10,
+0x16,
+0xc7,
+0x02,
+0x80,
+0xc0,
+0x83,
+0x81,
+0x26,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x40,
+0x66,
+0x00,
+0x01,
+0x78,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x27,
+0x66,
+0x00,
+0x01,
+0x74,
+0x40,
+0x90,
+0xe1,
+0x29,
+0x0d,
+0x11,
+0x90,
+0xb1,
+0x38,
+0x89,
+0xa0,
+0x88,
+0x92,
+0x18,
+0x88,
+0xa2,
+0x88,
+0x82,
+0x38,
+0x87,
+0xa4,
+0x88,
+0x72,
+0x58,
+0x86,
+0xa6,
+0x88,
+0x62,
+0x78,
+0x85,
+0xa8,
+0x88,
+0x52,
+0x98,
+0x84,
+0xaa,
+0x88,
+0x42,
+0xb8,
+0x80,
+0x0b,
+0x88,
+0x08,
+0x98,
+0x8a,
+0x88,
+0x88,
+0xa0,
+0xa8,
+0x83,
+0xac,
+0x88,
+0x32,
+0xd8,
+0x82,
+0xae,
+0x88,
+0x22,
+0xf8,
+0x81,
+0xb6,
+0x88,
+0x13,
+0x59,
+0x10,
+0x10,
+0x88,
+0xf1,
+0x68,
+0x8c,
+0x95,
+0x46,
+0x0b,
+0x40,
+0x8c,
+0x17,
+0x88,
+0xf9,
+0x4a,
+0x81,
+0x10,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0xab,
+0xef,
+0x08,
+0x8a,
+0xc8,
+0x5c,
+0x09,
+0x78,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x58,
+0x82,
+0x6f,
+0x88,
+0x2e,
+0xe8,
+0x83,
+0x6d,
+0x88,
+0x3e,
+0xc8,
+0x84,
+0x6b,
+0x88,
+0x4e,
+0xa8,
+0x85,
+0x69,
+0x88,
+0x5e,
+0x88,
+0x86,
+0x67,
+0x88,
+0x6e,
+0x68,
+0x87,
+0x65,
+0x88,
+0x7e,
+0x48,
+0x88,
+0x63,
+0x88,
+0x8e,
+0x28,
+0x89,
+0x61,
+0x88,
+0x9e,
+0x08,
+0x8a,
+0x4a,
+0x88,
+0x0c,
+0x99,
+0x0b,
+0x5b,
+0x88,
+0xc5,
+0x79,
+0x0d,
+0x59,
+0x88,
+0xcd,
+0x59,
+0x0e,
+0x5a,
+0x88,
+0xf5,
+0x69,
+0x10,
+0x58,
+0x88,
+0xfd,
+0x40,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9e,
+0x0b,
+0xc0,
+0xd8,
+0x88,
+0x1f,
+0x63,
+0x81,
+0x36,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x48,
+0xba,
+0x10,
+0x16,
+0x60,
+0x00,
+0x18,
+0x2c,
+0x0b,
+0xc1,
+0x47,
+0x66,
+0x00,
+0x01,
+0x7e,
+0x80,
+0xbc,
+0x11,
+0x73,
+0x81,
+0x26,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0xa8,
+0xba,
+0x10,
+0x16,
+0xc7,
+0x02,
+0x88,
+0xc0,
+0x83,
+0x81,
+0x26,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x40,
+0x66,
+0x00,
+0x01,
+0x82,
+0xc0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x27,
+0x66,
+0x00,
+0x01,
+0x7e,
+0x80,
+0x90,
+0xe1,
+0x29,
+0x0d,
+0x11,
+0x90,
+0xb1,
+0x38,
+0x89,
+0xa0,
+0x88,
+0x92,
+0x18,
+0x88,
+0xa2,
+0x88,
+0x82,
+0x38,
+0x87,
+0xa4,
+0x88,
+0x72,
+0x58,
+0x86,
+0xa6,
+0x88,
+0x62,
+0x78,
+0x85,
+0xa8,
+0x88,
+0x52,
+0x98,
+0x84,
+0xaa,
+0x88,
+0x42,
+0xb8,
+0x80,
+0x0b,
+0x88,
+0x08,
+0x98,
+0x8a,
+0x88,
+0x88,
+0xa0,
+0xa8,
+0x83,
+0xac,
+0x88,
+0x32,
+0xd8,
+0x82,
+0xae,
+0x88,
+0x22,
+0xf8,
+0x81,
+0xb6,
+0x88,
+0x13,
+0x59,
+0x10,
+0x10,
+0x88,
+0xf1,
+0x68,
+0x8c,
+0x95,
+0x46,
+0x0b,
+0x40,
+0x8c,
+0x17,
+0x88,
+0xf9,
+0x4a,
+0x81,
+0x10,
+0x60,
+0x02,
+0x80,
+0x00,
+0x10,
+0x98,
+0xea,
+0x03,
+0x90,
+0x20,
+0x80,
+0x07,
+0xa6,
+0x00,
+0x30,
+0x00,
+0x01,
+0x06,
+0x82,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x80,
+0x07,
+0xab,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x1c,
+0x05,
+0x20,
+0x5c,
+0xbf,
+0x03,
+0x00,
+0x0c,
+0x84,
+0x07,
+0xa9,
+0xc0,
+0x00,
+0x68,
+0x00,
+0x40,
+0x20,
+0x0a,
+0x84,
+0x04,
+0xaa,
+0x00,
+0x40,
+0x84,
+0x04,
+0x8a,
+0x04,
+0xc0,
+0x84,
+0x00,
+0xa5,
+0xd0,
+0xc3,
+0x30,
+0x7f,
+0x83,
+0x00,
+0x30,
+0xbc,
+0x28,
+0x06,
+0x83,
+0x81,
+0xc0,
+0x02,
+0x06,
+0x83,
+0x40,
+0x02,
+0x42,
+0x15,
+0xc8,
+0x10,
+0x84,
+0x00,
+0xaa,
+0x00,
+0xc2,
+0x84,
+0x84,
+0xa8,
+0x50,
+0x48,
+0x68,
+0x34,
+0x08,
+0x44,
+0x23,
+0x84,
+0x00,
+0xa8,
+0x48,
+0xca,
+0x85,
+0x04,
+0x8a,
+0x18,
+0x21,
+0x84,
+0x00,
+0xa8,
+0x58,
+0x4a,
+0x85,
+0x04,
+0x8a,
+0xc7,
+0xa0,
+0x84,
+0x00,
+0xa8,
+0x08,
+0xca,
+0x85,
+0x04,
+0x8a,
+0xc4,
+0x42,
+0x84,
+0x00,
+0xa8,
+0x08,
+0xca,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa8,
+0x08,
+0x4a,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa8,
+0x48,
+0x4a,
+0x85,
+0x04,
+0x8a,
+0x08,
+0x21,
+0x84,
+0x00,
+0x84,
+0x60,
+0xa4,
+0x00,
+0xb4,
+0x88,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x68,
+0x34,
+0x04,
+0x23,
+0x20,
+0x5c,
+0xbf,
+0x0b,
+0x04,
+0x04,
+0x68,
+0x38,
+0x1c,
+0x05,
+0x21,
+0x5c,
+0x82,
+0x00,
+0x40,
+0x48,
+0x5c,
+0x00,
+0x70,
+0x48,
+0x7a,
+0x5c,
+0xbd,
+0x09,
+0xc8,
+0x81,
+0x68,
+0x00,
+0x40,
+0x28,
+0x2e,
+0x5c,
+0x08,
+0x20,
+0x08,
+0x6e,
+0x84,
+0x84,
+0xaa,
+0x0c,
+0xa1,
+0x9c,
+0x08,
+0x28,
+0x48,
+0x00,
+0x85,
+0x05,
+0x00,
+0x00,
+0x00,
+0x84,
+0x00,
+0x02,
+0x41,
+0x00,
+0x84,
+0x05,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xff,
+0xc0,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x02,
+0x38,
+0x11,
+0x82,
+0x58,
+0x10,
+0xbc,
+0x32,
+0x06,
+0x83,
+0x81,
+0xc0,
+0x02,
+0x16,
+0x83,
+0x40,
+0x42,
+0x02,
+0x08,
+0x48,
+0x02,
+0x84,
+0x05,
+0x2a,
+0x00,
+0x60,
+0xb0,
+0x48,
+0x28,
+0x40,
+0x52,
+0xa0,
+0x88,
+0x18,
+0x40,
+0x02,
+0x24,
+0x11,
+0x28,
+0x40,
+0x52,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x02,
+0x25,
+0x91,
+0x0b,
+0xff,
+0xc0,
+0x84,
+0x00,
+0x22,
+0x58,
+0x10,
+0xbc,
+0x01,
+0x1b,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x40,
+0x28,
+0x6e,
+0x80,
+0x86,
+0xe8,
+0x48,
+0x4a,
+0xa0,
+0xcc,
+0x18,
+0x48,
+0x0a,
+0x6c,
+0x68,
+0x08,
+0x40,
+0x4a,
+0xb0,
+0x50,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x40,
+0x0a,
+0x24,
+0x13,
+0x68,
+0x40,
+0x4a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x0a,
+0x25,
+0x93,
+0x0b,
+0xff,
+0xc0,
+0x84,
+0x00,
+0x82,
+0x58,
+0x20,
+0xbc,
+0x01,
+0x1b,
+0xa1,
+0x40,
+0xba,
+0x14,
+0x88,
+0x40,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x14,
+0x40,
+0x68,
+0x34,
+0x04,
+0x04,
+0x20,
+0x5c,
+0x02,
+0xe2,
+0xff,
+0xc0,
+0x5c,
+0x00,
+0x70,
+0x40,
+0x48,
+0x9c,
+0x00,
+0x0b,
+0x04,
+0x80,
+0x68,
+0x34,
+0x00,
+0x1e,
+0x21,
+0x84,
+0x05,
+0x0b,
+0x1d,
+0xf2,
+0x84,
+0x85,
+0x2a,
+0x08,
+0x81,
+0xb1,
+0x81,
+0xc8,
+0x48,
+0x48,
+0xa0,
+0x22,
+0x06,
+0x81,
+0x50,
+0xc8,
+0x40,
+0x08,
+0x40,
+0x50,
+0x68,
+0x00,
+0x01,
+0x41,
+0x48,
+0x88,
+0x03,
+0x6a,
+0x06,
+0x00,
+0x46,
+0x0a,
+0x40,
+0x48,
+0xc8,
+0x84,
+0x04,
+0xaa,
+0x80,
+0x10,
+0x68,
+0x34,
+0x00,
+0x09,
+0x20,
+0x5c,
+0x08,
+0xb3,
+0x00,
+0x3c,
+0x5c,
+0x81,
+0x00,
+0x40,
+0x00,
+0x52,
+0x0c,
+0x03,
+0x00,
+0x26,
+0x84,
+0x05,
+0x0a,
+0x00,
+0xa0,
+0x84,
+0x04,
+0x8a,
+0x00,
+0x40,
+0x80,
+0x04,
+0x88,
+0x00,
+0x4a,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x4a,
+0x84,
+0x7f,
+0xa0,
+0x00,
+0x00,
+0x68,
+0x34,
+0x08,
+0x4a,
+0x20,
+0x38,
+0x01,
+0xc8,
+0x40,
+0x48,
+0x40,
+0x00,
+0x00,
+0x40,
+0xc8,
+0x64,
+0x00,
+0x01,
+0x1e,
+0xa7,
+0x68,
+0x00,
+0x01,
+0xbc,
+0x20,
+0x39,
+0x02,
+0x08,
+0x00,
+0x21,
+0x68,
+0x34,
+0x08,
+0x20,
+0x22,
+0x84,
+0x80,
+0x88,
+0x00,
+0x21,
+0x85,
+0x04,
+0x8a,
+0x10,
+0x22,
+0x84,
+0x80,
+0x88,
+0x00,
+0x21,
+0x84,
+0x02,
+0x08,
+0x10,
+0x48,
+0x00,
+0x00,
+0x08,
+0x48,
+0x08,
+0x81,
+0x04,
+0x80,
+0x00,
+0x00,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x08,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x68,
+0x34,
+0x08,
+0x57,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x04,
+0x7a,
+0x5c,
+0x81,
+0x03,
+0x04,
+0x24,
+0x5c,
+0x13,
+0x70,
+0x40,
+0x48,
+0xa0,
+0x02,
+0x08,
+0x00,
+0x7a,
+0x80,
+0x04,
+0x88,
+0x40,
+0x7a,
+0xa0,
+0x52,
+0x08,
+0x02,
+0x4a,
+0xb0,
+0x60,
+0x48,
+0x40,
+0x48,
+0xa0,
+0x48,
+0x08,
+0x00,
+0x7a,
+0xac,
+0x9a,
+0x08,
+0x02,
+0x7a,
+0x84,
+0x07,
+0xaa,
+0x01,
+0x00,
+0xac,
+0xb0,
+0x08,
+0x00,
+0x7a,
+0xac,
+0x40,
+0x04,
+0x60,
+0xa4,
+0x00,
+0x24,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x34,
+0x00,
+0x17,
+0x20,
+0x68,
+0x03,
+0xfb,
+0xff,
+0xc9,
+0x5c,
+0x08,
+0x20,
+0x40,
+0x7a,
+0x5c,
+0x0b,
+0x32,
+0x00,
+0x80,
+0x5c,
+0xbd,
+0x00,
+0x40,
+0x02,
+0x54,
+0x4a,
+0xaa,
+0x03,
+0x21,
+0x5c,
+0x00,
+0x40,
+0x40,
+0x49,
+0x5c,
+0x09,
+0xd2,
+0xbf,
+0xe0,
+0x84,
+0x80,
+0x95,
+0x20,
+0x96,
+0x88,
+0x0f,
+0x65,
+0xc0,
+0x02,
+0x84,
+0x84,
+0x96,
+0x83,
+0x80,
+0x80,
+0xf2,
+0x28,
+0x40,
+0x08,
+0x52,
+0x0d,
+0x20,
+0x80,
+0x49,
+0x5c,
+0x1d,
+0x68,
+0x40,
+0x48,
+0xa0,
+0x5a,
+0x08,
+0x40,
+0x7a,
+0x9d,
+0x00,
+0x08,
+0x50,
+0x50,
+0x84,
+0x05,
+0x28,
+0x81,
+0x60,
+0x66,
+0x00,
+0x01,
+0x2e,
+0x20,
+0x5c,
+0x08,
+0x30,
+0x80,
+0x09,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x55,
+0x03,
+0x6b,
+0x03,
+0x24,
+0x30,
+0x12,
+0x8b,
+0xff,
+0x6a,
+0x5c,
+0x1d,
+0x68,
+0x80,
+0x49,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x1a,
+0xeb,
+0x00,
+0x0e,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x1a,
+0xa9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x0d,
+0xe9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0x69,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0xa9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0xeb,
+0x00,
+0x26,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x07,
+0x2b,
+0x00,
+0xc6,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x07,
+0x6b,
+0x00,
+0x36,
+0x5c,
+0x02,
+0x68,
+0x81,
+0x20,
+0x5c,
+0x01,
+0xe0,
+0x80,
+0xb6,
+0x6c,
+0x70,
+0x10,
+0x1e,
+0x49,
+0x5c,
+0x03,
+0xe0,
+0x40,
+0x48,
+0x5c,
+0x00,
+0x72,
+0x05,
+0x80,
+0x84,
+0x07,
+0xaa,
+0x00,
+0x40,
+0x84,
+0x07,
+0xab,
+0x05,
+0xdd,
+0xa0,
+0x02,
+0x08,
+0x40,
+0x49,
+0xa0,
+0x16,
+0x08,
+0x40,
+0x48,
+0x46,
+0x0a,
+0x42,
+0x05,
+0x00,
+0x84,
+0x04,
+0xaa,
+0x80,
+0x20,
+0x5c,
+0x08,
+0x01,
+0x8e,
+0x8a,
+0x5c,
+0x00,
+0x51,
+0x82,
+0x88,
+0x68,
+0x00,
+0x03,
+0xe8,
+0x09,
+0x6c,
+0x70,
+0x10,
+0x0e,
+0x0b,
+0x25,
+0x83,
+0x8b,
+0xc0,
+0x41,
+0xba,
+0x14,
+0x86,
+0xc7,
+0x01,
+0x00,
+0xe5,
+0x20,
+0x00,
+0x00,
+0x2a,
+0x07,
+0x63,
+0x01,
+0x70,
+0xbf,
+0xf5,
+0x26,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x83,
+0x81,
+0xe6,
+0x52,
+0x0d,
+0x33,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x4a,
+0x5c,
+0x3f,
+0xe3,
+0x80,
+0x00,
+0x68,
+0x38,
+0x08,
+0x00,
+0x20,
+0x5c,
+0x09,
+0xe2,
+0xbf,
+0x90,
+0x52,
+0x09,
+0x62,
+0x00,
+0x41,
+0x5d,
+0x8c,
+0x12,
+0x08,
+0x20,
+0x5c,
+0x00,
+0x00,
+0x80,
+0x60,
+0xa0,
+0x16,
+0x0a,
+0x05,
+0x00,
+0xa0,
+0x4c,
+0x28,
+0x80,
+0xe2,
+0xa1,
+0x04,
+0x28,
+0x81,
+0x62,
+0xa1,
+0x02,
+0x28,
+0x81,
+0xe2,
+0xa1,
+0x16,
+0x2a,
+0x15,
+0x02,
+0x88,
+0x4c,
+0x88,
+0x83,
+0x62,
+0x88,
+0x3e,
+0x08,
+0x84,
+0x61,
+0x88,
+0x55,
+0x28,
+0x82,
+0x4a,
+0x88,
+0x2c,
+0x98,
+0x85,
+0xf6,
+0x40,
+0x00,
+0x03,
+0x04,
+0x04,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x48,
+0x5c,
+0x00,
+0xa0,
+0x48,
+0x7a,
+0x5c,
+0x00,
+0x68,
+0x48,
+0xc9,
+0x84,
+0x8c,
+0xa8,
+0x86,
+0x50,
+0x00,
+0x00,
+0x08,
+0x80,
+0x21,
+0x00,
+0x00,
+0x08,
+0x4d,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x40,
+0x49,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x5c,
+0x00,
+0xa0,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x68,
+0x81,
+0x21,
+0x84,
+0x07,
+0xa8,
+0x41,
+0x7a,
+0x00,
+0x00,
+0x08,
+0x84,
+0x8a,
+0x84,
+0x8c,
+0xa0,
+0x00,
+0x00,
+0x88,
+0x1a,
+0x08,
+0x83,
+0x21,
+0x84,
+0x5c,
+0x88,
+0x48,
+0x49,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x5c,
+0xbb,
+0x00,
+0x83,
+0x20,
+0x88,
+0x50,
+0x99,
+0xc0,
+0x00,
+0x84,
+0x00,
+0xa3,
+0xb1,
+0x80,
+0x30,
+0x14,
+0x0b,
+0xc0,
+0x89,
+0x5c,
+0x19,
+0x20,
+0x86,
+0x0a,
+0x88,
+0x5b,
+0x6b,
+0xa1,
+0x48,
+0x5c,
+0x00,
+0x22,
+0x80,
+0x70,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x55,
+0x03,
+0x80,
+0x84,
+0x21,
+0x58,
+0x08,
+0x00,
+0x82,
+0x89,
+0x88,
+0x20,
+0xa4,
+0x3e,
+0x4d,
+0x08,
+0x3a,
+0x0b,
+0x04,
+0x04,
+0x88,
+0x5b,
+0x6b,
+0xa1,
+0x48,
+0x5c,
+0x3f,
+0xe2,
+0x80,
+0x70,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x08,
+0x00,
+0x20,
+0x5c,
+0x09,
+0xf3,
+0x00,
+0x14,
+0x52,
+0x0d,
+0x68,
+0x40,
+0x7a,
+0x5c,
+0x00,
+0x72,
+0x00,
+0x40,
+0x84,
+0x07,
+0xaa,
+0x00,
+0x20,
+0x84,
+0x04,
+0x9a,
+0x01,
+0x60,
+0x84,
+0x04,
+0x8a,
+0x05,
+0x00,
+0x84,
+0x04,
+0xaa,
+0xbf,
+0xf0,
+0xa0,
+0x4a,
+0x08,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x66,
+0x60,
+0x00,
+0x12,
+0x6c,
+0x08,
+0x80,
+0xb6,
+0x88,
+0x02,
+0x0b,
+0xa1,
+0x48,
+0x84,
+0x00,
+0x8a,
+0x80,
+0x10,
+0x62,
+0x00,
+0x00,
+0x00,
+0x36,
+0x5c,
+0x81,
+0x09,
+0x82,
+0x24,
+0x5c,
+0x80,
+0x81,
+0x82,
+0x60,
+0xbb,
+0x00,
+0x08,
+0x00,
+0xcc,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x38,
+0x00,
+0xe2,
+0x11,
+0x34,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x6d,
+0x84,
+0x86,
+0x06,
+0x20,
+0x00,
+0x00,
+0x01,
+0x43,
+0x90,
+0x20,
+0x00,
+0x00,
+0x08,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x00,
+0x61,
+0x88,
+0x2c,
+0x50,
+0x8d,
+0x20,
+0x48,
+0x20,
+0x51,
+0x85,
+0x3a,
+0xc0,
+0x20,
+0x98,
+0x2e,
+0xe0,
+0x00,
+0x00,
+0x9c,
+0x40,
+0x08,
+0x40,
+0x49,
+0x62,
+0x00,
+0x00,
+0x00,
+0x24,
+0x84,
+0x86,
+0x09,
+0x8e,
+0x80,
+0x80,
+0x40,
+0x92,
+0x81,
+0x40,
+0xba,
+0x14,
+0x85,
+0x04,
+0xc0,
+0x04,
+0x86,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x06,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x04,
+0x60,
+0xa4,
+0x18,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x02,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x09,
+0x80,
+0x82,
+0x55,
+0x00,
+0xb8,
+0x20,
+0x08,
+0x8c,
+0x06,
+0x0a,
+0x20,
+0x01,
+0x8c,
+0x12,
+0xe4,
+0x42,
+0x00,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x04,
+0x88,
+0x94,
+0x46,
+0xc8,
+0x06,
+0x08,
+0x85,
+0xb0,
+0x82,
+0x98,
+0x04,
+0xa5,
+0x78,
+0x9a,
+0x18,
+0x48,
+0x05,
+0x74,
+0xb1,
+0x3a,
+0x14,
+0x88,
+0xc1,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x32,
+0x02,
+0x8b,
+0xc3,
+0x58,
+0x22,
+0x86,
+0xe5,
+0xb8,
+0xa2,
+0x30,
+0x0f,
+0x05,
+0x70,
+0x80,
+0x30,
+0x00,
+0xf5,
+0x08,
+0x1c,
+0x30,
+0x4e,
+0xa2,
+0xa6,
+0x63,
+0x4b,
+0xc1,
+0x23,
+0x3c,
+0x08,
+0x20,
+0x0c,
+0xd2,
+0x35,
+0xd4,
+0x68,
+0x20,
+0x01,
+0x51,
+0x20,
+0x28,
+0x16,
+0x25,
+0x14,
+0x2a,
+0x84,
+0x00,
+0x85,
+0x70,
+0xb0,
+0xb0,
+0x0f,
+0xa5,
+0x16,
+0xe6,
+0x04,
+0x08,
+0x92,
+0x80,
+0x20,
+0x68,
+0x1f,
+0xff,
+0xff,
+0xc8,
+0x36,
+0x80,
+0x35,
+0x44,
+0x84,
+0x98,
+0x0c,
+0x82,
+0x32,
+0x09,
+0x28,
+0x08,
+0x02,
+0x10,
+0x3f,
+0x20,
+0x90,
+0xc2,
+0x81,
+0x3f,
+0x08,
+0xb0,
+0x02,
+0x23,
+0xc0,
+0x98,
+0x00,
+0x80,
+0x83,
+0x00,
+0x22,
+0x3c,
+0x02,
+0xe0,
+0x28,
+0x98,
+0x00,
+0x80,
+0x8c,
+0x00,
+0x22,
+0x04,
+0x09,
+0x80,
+0x0b,
+0x08,
+0xb0,
+0x02,
+0x23,
+0xc0,
+0x98,
+0x00,
+0xa0,
+0x8b,
+0x00,
+0x22,
+0x3c,
+0x02,
+0xe0,
+0x28,
+0x98,
+0x00,
+0xa4,
+0x47,
+0x00,
+0x3a,
+0x14,
+0x82,
+0x20,
+0x40,
+0x98,
+0x00,
+0x8b,
+0xa1,
+0x48,
+0x98,
+0xe8,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x08,
+0x29,
+0x58,
+0x2c,
+0x52,
+0xcb,
+0x02,
+0xbf,
+0xc0,
+0x88,
+0x0e,
+0x14,
+0x20,
+0x7c,
+0x88,
+0x16,
+0x28,
+0x80,
+0x76,
+0x00,
+0x00,
+0x09,
+0x5a,
+0x2e,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0xa0,
+0x95,
+0xba,
+0xe3,
+0x20,
+0x30,
+0xbc,
+0x04,
+0x18,
+0x59,
+0x08,
+0x55,
+0x03,
+0x23,
+0xc0,
+0x4f,
+0x85,
+0x94,
+0x82,
+0x49,
+0x60,
+0x40,
+0x00,
+0x01,
+0x58,
+0x60,
+0x5c,
+0x82,
+0x0a,
+0x18,
+0xa1,
+0x88,
+0x1e,
+0x10,
+0x00,
+0x00,
+0x80,
+0x88,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x06,
+0x95,
+0xc8,
+0x10,
+0x08,
+0x26,
+0x13,
+0x81,
+0x05,
+0x95,
+0x8a,
+0xc2,
+0x59,
+0x60,
+0xbc,
+0x45,
+0x18,
+0x82,
+0xe5,
+0x88,
+0x36,
+0x30,
+0x00,
+0x00,
+0x8c,
+0x02,
+0xa8,
+0x20,
+0x09,
+0x44,
+0x08,
+0x00,
+0x55,
+0x22,
+0x82,
+0x00,
+0x84,
+0x44,
+0x40,
+0x02,
+0x00,
+0x88,
+0x50,
+0x09,
+0x44,
+0x25,
+0x01,
+0x82,
+0x40,
+0x98,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x00,
+0x00,
+0x08,
+0xc1,
+0x2a,
+0x82,
+0x00,
+0x94,
+0x40,
+0x80,
+0x06,
+0x00,
+0x84,
+0x44,
+0x48,
+0x06,
+0x08,
+0x95,
+0x50,
+0x0a,
+0x18,
+0x48,
+0x00,
+0x81,
+0xa4,
+0x98,
+0x08,
+0x28,
+0xc1,
+0x60,
+0x40,
+0x00,
+0x00,
+0x83,
+0xd2,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x04,
+0x5c,
+0x08,
+0x71,
+0x48,
+0xad,
+0x25,
+0x9a,
+0x8b,
+0xc1,
+0xa1,
+0x94,
+0x82,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x07,
+0x88,
+0x83,
+0x88,
+0x88,
+0x12,
+0x06,
+0x60,
+0x00,
+0x14,
+0xc4,
+0x0b,
+0xc1,
+0x7f,
+0x40,
+0x00,
+0x00,
+0x81,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x4a,
+0x28,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x00,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0x42,
+0x03,
+0x40,
+0x81,
+0x20,
+0x94,
+0xa6,
+0x46,
+0x60,
+0x00,
+0x14,
+0xc4,
+0x88,
+0x83,
+0x88,
+0xbc,
+0x08,
+0xf8,
+0x81,
+0xa0,
+0xbc,
+0x06,
+0xf8,
+0x81,
+0xa0,
+0xbc,
+0x04,
+0xf8,
+0x81,
+0xa0,
+0x88,
+0x1a,
+0x08,
+0x82,
+0xe5,
+0x88,
+0x36,
+0x30,
+0x00,
+0x00,
+0x88,
+0x22,
+0x58,
+0x41,
+0x08,
+0x59,
+0x01,
+0x02,
+0x2c,
+0xd8,
+0x42,
+0x04,
+0x48,
+0x82,
+0xa5,
+0x5c,
+0x81,
+0x00,
+0x81,
+0xe0,
+0x38,
+0x10,
+0x48,
+0x83,
+0x20,
+0x00,
+0x00,
+0x09,
+0x40,
+0xad,
+0x25,
+0x92,
+0x8b,
+0xc3,
+0xc1,
+0x00,
+0x00,
+0x08,
+0x81,
+0x20,
+0x88,
+0x0a,
+0x3a,
+0x02,
+0x80,
+0x8d,
+0x82,
+0xa8,
+0x28,
+0x09,
+0x44,
+0x08,
+0x00,
+0x40,
+0x20,
+0x82,
+0x80,
+0x84,
+0x44,
+0x40,
+0x04,
+0x00,
+0x98,
+0x28,
+0x08,
+0x44,
+0x25,
+0x01,
+0x82,
+0x40,
+0x98,
+0x08,
+0x25,
+0x50,
+0x0b,
+0x8d,
+0x86,
+0x06,
+0x80,
+0x00,
+0x0d,
+0x72,
+0x08,
+0xd9,
+0x2a,
+0x82,
+0x80,
+0x94,
+0x40,
+0x80,
+0x06,
+0x80,
+0x84,
+0x44,
+0x48,
+0x06,
+0x88,
+0x94,
+0x46,
+0xd2,
+0x18,
+0x48,
+0x09,
+0x80,
+0x82,
+0x8d,
+0x96,
+0x08,
+0x80,
+0xd2,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x04,
+0x5c,
+0x08,
+0x71,
+0x48,
+0xad,
+0x25,
+0x9a,
+0x8b,
+0xc1,
+0x41,
+0x94,
+0x82,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x05,
+0x88,
+0x80,
+0x88,
+0x88,
+0x12,
+0x06,
+0x60,
+0x00,
+0x14,
+0xee,
+0x0b,
+0xc0,
+0xc7,
+0x66,
+0x00,
+0x01,
+0x47,
+0xa8,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x00,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0x42,
+0x02,
+0x40,
+0x81,
+0x20,
+0x94,
+0xbe,
+0x46,
+0x60,
+0x00,
+0x14,
+0xee,
+0x88,
+0x80,
+0x88,
+0x38,
+0x11,
+0x48,
+0x81,
+0xa0,
+0x00,
+0x00,
+0x09,
+0x40,
+0x2d,
+0x25,
+0x92,
+0x8b,
+0xc2,
+0xc1,
+0x68,
+0x20,
+0x01,
+0x7c,
+0x23,
+0x5c,
+0x81,
+0x12,
+0xc0,
+0x80,
+0x68,
+0x20,
+0x01,
+0x82,
+0x24,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x20,
+0x52,
+0x49,
+0x40,
+0x19,
+0x0a,
+0x68,
+0x20,
+0x00,
+0xe6,
+0x25,
+0x5c,
+0x82,
+0x08,
+0x21,
+0x08,
+0x81,
+0x80,
+0x98,
+0x01,
+0x4a,
+0x82,
+0x94,
+0x88,
+0x00,
+0x49,
+0x00,
+0x00,
+0x08,
+0x20,
+0x0b,
+0x82,
+0x28,
+0x98,
+0x1a,
+0x88,
+0x6c,
+0x40,
+0x02,
+0xfc,
+0x0a,
+0x82,
+0x84,
+0xb8,
+0x2a,
+0xc9,
+0x80,
+0x2c,
+0x86,
+0xc4,
+0x00,
+0x1c,
+0x44,
+0xa0,
+0x00,
+0x00,
+0x88,
+0x32,
+0x28,
+0x60,
+0x0b,
+0x85,
+0x80,
+0x28,
+0x60,
+0x88,
+0x85,
+0x88,
+0x96,
+0xc4,
+0x00,
+0x30,
+0x80,
+0xa8,
+0x68,
+0x4b,
+0x84,
+0x05,
+0x28,
+0x68,
+0xc8,
+0x84,
+0x0c,
+0x99,
+0x50,
+0xe0,
+0x6c,
+0x40,
+0x01,
+0xd0,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x40,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x94,
+0x3a,
+0xc3,
+0x20,
+0x20,
+0xbc,
+0x21,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0xbc,
+0x03,
+0xf9,
+0x43,
+0x64,
+0xb0,
+0x00,
+0xd9,
+0x43,
+0x65,
+0xa0,
+0x02,
+0x19,
+0x48,
+0x2d,
+0x59,
+0x01,
+0x42,
+0x08,
+0x12,
+0x42,
+0x08,
+0xc1,
+0x50,
+0x2d,
+0xa1,
+0x40,
+0x95,
+0xc0,
+0x84,
+0x14,
+0x02,
+0xe2,
+0x58,
+0x30,
+0xbc,
+0x07,
+0x99,
+0x42,
+0xae,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x91,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x70,
+0xb0,
+0x00,
+0xcb,
+0xc0,
+0x57,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x30,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x11,
+0xb0,
+0x00,
+0xcb,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x50,
+0x00,
+0x00,
+0xba,
+0x14,
+0x8b,
+0x00,
+0x0c,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x94,
+0x22,
+0xc3,
+0x20,
+0x20,
+0xbc,
+0x1c,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0xbc,
+0x03,
+0xf9,
+0x41,
+0xe4,
+0xb0,
+0x00,
+0xd9,
+0x41,
+0xe5,
+0x5c,
+0x08,
+0x6a,
+0x00,
+0x11,
+0x94,
+0x02,
+0xe5,
+0x2c,
+0xb8,
+0x14,
+0x82,
+0xdb,
+0xc0,
+0x79,
+0x94,
+0x8a,
+0xe3,
+0x20,
+0x28,
+0xbc,
+0x09,
+0x13,
+0x20,
+0x30,
+0xbc,
+0x07,
+0x0b,
+0x00,
+0x0c,
+0xbc,
+0x05,
+0x73,
+0x20,
+0x28,
+0xbc,
+0x03,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x01,
+0x1b,
+0x00,
+0x0c,
+0xba,
+0x14,
+0x89,
+0x41,
+0x66,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x48,
+0xb0,
+0x00,
+0xc0,
+0x00,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x08,
+0x82,
+0x85,
+0x00,
+0xa2,
+0xa0,
+0x76,
+0x81,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x85,
+0x00,
+0x03,
+0x00,
+0x30,
+0xbc,
+0x1d,
+0x4a,
+0x00,
+0x41,
+0x84,
+0x80,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x01,
+0x38,
+0x41,
+0x48,
+0xa0,
+0x82,
+0x28,
+0x50,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x25,
+0x40,
+0x00,
+0x00,
+0x48,
+0xc8,
+0x5b,
+0x48,
+0x03,
+0x01,
+0x05,
+0x50,
+0x4b,
+0x08,
+0x10,
+0x0a,
+0x5c,
+0x82,
+0x00,
+0xd1,
+0x32,
+0x54,
+0x02,
+0x90,
+0x41,
+0x08,
+0x57,
+0x0d,
+0x20,
+0xd1,
+0x7a,
+0x81,
+0x04,
+0x89,
+0x80,
+0x08,
+0x50,
+0x4b,
+0x10,
+0xd1,
+0x30,
+0x54,
+0x04,
+0x03,
+0xa1,
+0x48,
+0x8d,
+0x17,
+0x80,
+0x00,
+0x00,
+0xba,
+0x14,
+0x88,
+0x4a,
+0xfa,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x08,
+0xc2,
+0x85,
+0x00,
+0xa2,
+0xa0,
+0x76,
+0x81,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x85,
+0x00,
+0x03,
+0x00,
+0x30,
+0xbc,
+0x1f,
+0x4a,
+0x01,
+0x81,
+0x84,
+0x80,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x01,
+0x38,
+0x46,
+0x48,
+0xa0,
+0x82,
+0x28,
+0x50,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x25,
+0x40,
+0x00,
+0x00,
+0x48,
+0xc8,
+0x5c,
+0x82,
+0x0b,
+0x01,
+0x05,
+0x5b,
+0x48,
+0x10,
+0x46,
+0x0a,
+0x81,
+0x00,
+0x02,
+0xe0,
+0x36,
+0x50,
+0x4b,
+0x00,
+0x10,
+0xca,
+0x98,
+0x08,
+0x88,
+0xd0,
+0x31,
+0x28,
+0x00,
+0x85,
+0x04,
+0xb0,
+0x0d,
+0x0d,
+0x80,
+0x00,
+0x00,
+0x8d,
+0x03,
+0x25,
+0x40,
+0x08,
+0x3a,
+0x14,
+0x88,
+0xd0,
+0x78,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x08,
+0x22,
+0x08,
+0xe0,
+0xa0,
+0x4d,
+0xa9,
+0x50,
+0x2e,
+0x25,
+0x93,
+0x0b,
+0xc0,
+0x69,
+0x84,
+0x07,
+0xa3,
+0x81,
+0x0c,
+0x52,
+0x09,
+0x83,
+0xa1,
+0x48,
+0x94,
+0x8e,
+0x00,
+0x00,
+0x00,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x20,
+0x01,
+0x2b,
+0x20,
+0x5c,
+0x81,
+0x03,
+0x00,
+0x0c,
+0x5c,
+0x06,
+0x68,
+0x02,
+0x0a,
+0x68,
+0x20,
+0x01,
+0x05,
+0x21,
+0x51,
+0x89,
+0x80,
+0x40,
+0x01,
+0x51,
+0x8a,
+0x00,
+0x08,
+0x50,
+0x68,
+0x20,
+0x01,
+0xf0,
+0x22,
+0x84,
+0x80,
+0x28,
+0x10,
+0x52,
+0x81,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x0f,
+0xc2,
+0x36,
+0x82,
+0x00,
+0x0d,
+0x72,
+0x18,
+0x10,
+0x7a,
+0x68,
+0x00,
+0x00,
+0xdc,
+0x24,
+0x85,
+0x07,
+0xaa,
+0x10,
+0x42,
+0x84,
+0x90,
+0xb8,
+0x58,
+0x02,
+0x85,
+0x88,
+0xa8,
+0x48,
+0x48,
+0xa2,
+0x06,
+0x35,
+0xc0,
+0x02,
+0x86,
+0x04,
+0x98,
+0x12,
+0x52,
+0x68,
+0x00,
+0x00,
+0xa6,
+0x21,
+0xa1,
+0xc2,
+0x0a,
+0x10,
+0x64,
+0xab,
+0xff,
+0x0e,
+0x21,
+0x04,
+0x84,
+0x07,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0xa7,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0xa7,
+0xa6,
+0xc0,
+0x00,
+0x37,
+0x64,
+0x86,
+0xc0,
+0x00,
+0x36,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x19,
+0x84,
+0xb6,
+0xc4,
+0x00,
+0x19,
+0xa4,
+0xb6,
+0xc4,
+0x00,
+0x31,
+0x44,
+0xb6,
+0xc4,
+0x00,
+0x31,
+0xa4,
+0xb9,
+0x58,
+0x65,
+0x6c,
+0x40,
+0x01,
+0x06,
+0x51,
+0x6c,
+0x40,
+0x01,
+0x08,
+0x51,
+0x84,
+0x87,
+0xa8,
+0x50,
+0x4a,
+0x86,
+0x04,
+0xa8,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x23,
+0x8b,
+0x68,
+0x41,
+0x4a,
+0x66,
+0x00,
+0x00,
+0x4c,
+0x28,
+0x40,
+0x00,
+0x03,
+0x19,
+0x04,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x68,
+0x00,
+0x04,
+0xe2,
+0x0a,
+0x88,
+0x03,
+0x68,
+0x41,
+0x4a,
+0x64,
+0x00,
+0x00,
+0x4c,
+0x2f,
+0x5c,
+0x02,
+0xa2,
+0x80,
+0x10,
+0x30,
+0x1a,
+0x8b,
+0xc1,
+0x58,
+0x6c,
+0x40,
+0x02,
+0xc0,
+0x08,
+0x84,
+0x00,
+0xb3,
+0x69,
+0xc0,
+0x98,
+0x00,
+0xb3,
+0x09,
+0x38,
+0xbc,
+0x0d,
+0x46,
+0xc4,
+0x00,
+0x2b,
+0xc0,
+0x83,
+0x01,
+0x30,
+0xbc,
+0x03,
+0x2b,
+0xc0,
+0x4f,
+0x6c,
+0x40,
+0x02,
+0xbe,
+0x08,
+0x6c,
+0x40,
+0x02,
+0xb0,
+0x08,
+0x28,
+0x13,
+0x04,
+0x20,
+0x17,
+0x98,
+0x00,
+0xa2,
+0xf1,
+0x75,
+0x98,
+0x28,
+0x9b,
+0xa1,
+0x48,
+0x98,
+0x24,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0xdd,
+0x22,
+0x6c,
+0x00,
+0x01,
+0xb8,
+0x08,
+0x85,
+0x00,
+0xa3,
+0x01,
+0x30,
+0x55,
+0x03,
+0xab,
+0xc5,
+0xf9,
+0x85,
+0x04,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xa7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x08,
+0x08,
+0x5b,
+0x48,
+0x00,
+0x40,
+0x08,
+0x5b,
+0x48,
+0x11,
+0x54,
+0x24,
+0x32,
+0x06,
+0x0b,
+0xc3,
+0x39,
+0x9a,
+0x00,
+0x03,
+0x01,
+0x30,
+0xbc,
+0x09,
+0x48,
+0x48,
+0x0a,
+0x36,
+0x98,
+0x09,
+0x80,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x44,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x1b,
+0xc7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0xbc,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x06,
+0x0a,
+0x2a,
+0x06,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x40,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xc4,
+0x85,
+0xc0,
+0xce,
+0xb0,
+0x13,
+0x76,
+0xe0,
+0x00,
+0x14,
+0x8a,
+0xe6,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x15,
+0xc8,
+0x10,
+0x18,
+0xe8,
+0x86,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x95,
+0x2c,
+0xf8,
+0x14,
+0xa4,
+0x4b,
+0xc0,
+0x99,
+0x5c,
+0x0b,
+0xa0,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2c,
+0x20,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0x04,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x0a,
+0x52,
+0x49,
+0xa3,
+0xa1,
+0x48,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x00,
+0x00,
+0x03,
+0x01,
+0x30,
+0xbc,
+0x05,
+0x38,
+0x48,
+0x0a,
+0x36,
+0x98,
+0x09,
+0x80,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x42,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x1b,
+0xc7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0xbc,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x04,
+0x0a,
+0x2a,
+0x06,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x0e,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xc4,
+0x86,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x16,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0xc0,
+0x07,
+0x30,
+0x17,
+0x55,
+0x20,
+0xb2,
+0x2c,
+0x02,
+0x09,
+0x4a,
+0x46,
+0x84,
+0x87,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x8b,
+0xa1,
+0x40,
+0x68,
+0x20,
+0x01,
+0x46,
+0x21,
+0x68,
+0x00,
+0x00,
+0xcd,
+0x22,
+0x39,
+0x02,
+0x08,
+0x08,
+0x08,
+0x81,
+0x07,
+0xa8,
+0x10,
+0x48,
+0x68,
+0x00,
+0x00,
+0xd2,
+0x20,
+0x84,
+0x80,
+0x88,
+0x00,
+0x7a,
+0x81,
+0x04,
+0x88,
+0x00,
+0x7a,
+0x68,
+0x00,
+0x00,
+0xca,
+0x23,
+0x80,
+0x04,
+0x86,
+0x80,
+0x00,
+0x58,
+0x8a,
+0xca,
+0x10,
+0x01,
+0x81,
+0x86,
+0xc8,
+0x10,
+0x7a,
+0xa0,
+0x00,
+0x48,
+0x58,
+0x7a,
+0x85,
+0x06,
+0x14,
+0x60,
+0xa4,
+0x00,
+0x07,
+0xa8,
+0x40,
+0x64,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0c,
+0x16,
+0xc0,
+0x00,
+0x1a,
+0x22,
+0x06,
+0x80,
+0x00,
+0x58,
+0xd2,
+0xc6,
+0xc0,
+0x00,
+0x1a,
+0xc2,
+0x16,
+0xc0,
+0x00,
+0x19,
+0x46,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0c,
+0x8a,
+0xbf,
+0xf0,
+0x6c,
+0x00,
+0x01,
+0xa2,
+0x20,
+0x68,
+0x00,
+0x05,
+0x88,
+0xac,
+0x6c,
+0x00,
+0x01,
+0xac,
+0x21,
+0x6c,
+0x00,
+0x01,
+0x94,
+0x6c,
+0x42,
+0x09,
+0xf8,
+0x40,
+0x7a,
+0x84,
+0x87,
+0xa8,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x66,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0xaa,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x65,
+0x6c,
+0x00,
+0x01,
+0xac,
+0x20,
+0x42,
+0x03,
+0xf8,
+0x80,
+0x36,
+0x40,
+0x00,
+0x00,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x05,
+0x96,
+0xa0,
+0x88,
+0x03,
+0x66,
+0xc0,
+0x00,
+0x19,
+0x46,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0d,
+0x06,
+0xc0,
+0x00,
+0x1a,
+0x22,
+0x06,
+0x80,
+0x00,
+0x58,
+0x8a,
+0xc6,
+0xc0,
+0x00,
+0x1a,
+0xc2,
+0x16,
+0xc0,
+0x00,
+0x19,
+0x46,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x64,
+0x00,
+0x01,
+0x66,
+0xe7,
+0x68,
+0x00,
+0x00,
+0xce,
+0x20,
+0x68,
+0x20,
+0x01,
+0x48,
+0x21,
+0xab,
+0xfe,
+0x08,
+0x40,
+0x08,
+0x84,
+0x80,
+0x94,
+0x40,
+0x80,
+0x08,
+0x07,
+0x66,
+0x60,
+0x00,
+0x16,
+0xae,
+0x89,
+0x80,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x80,
+0xc8,
+0x68,
+0x20,
+0x01,
+0x48,
+0x20,
+0x68,
+0x00,
+0x00,
+0xd3,
+0x21,
+0x80,
+0x00,
+0x88,
+0x48,
+0x09,
+0x44,
+0x20,
+0x00,
+0x81,
+0x60,
+0x66,
+0x00,
+0x01,
+0x6a,
+0xe8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x9c,
+0x09,
+0x84,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x19,
+0xa0,
+0xa5,
+0x40,
+0xd6,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x0c,
+0xe2,
+0x05,
+0x44,
+0x17,
+0x08,
+0x08,
+0x96,
+0xc0,
+0x00,
+0x1a,
+0x60,
+0xb6,
+0xc0,
+0x00,
+0x1a,
+0x40,
+0x25,
+0x40,
+0x5f,
+0x80,
+0x04,
+0xa5,
+0x44,
+0x1f,
+0xa0,
+0x00,
+0x28,
+0x00,
+0x0a,
+0x68,
+0x00,
+0x00,
+0xd3,
+0x21,
+0x44,
+0x30,
+0x00,
+0x40,
+0xa0,
+0x80,
+0x84,
+0xb8,
+0x50,
+0xc0,
+0x84,
+0x04,
+0x0a,
+0x08,
+0x00,
+0x80,
+0x80,
+0x94,
+0x40,
+0x80,
+0x04,
+0x8a,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x0c,
+0x08,
+0x48,
+0x40,
+0x40,
+0x00,
+0x02,
+0x80,
+0x20,
+0x44,
+0x29,
+0x02,
+0xc0,
+0x20,
+0x68,
+0x20,
+0x01,
+0x4a,
+0x20,
+0x51,
+0x48,
+0x9b,
+0x3f,
+0xc4,
+0x80,
+0x00,
+0x29,
+0x80,
+0xc9,
+0x44,
+0x0d,
+0xc0,
+0x00,
+0x02,
+0x98,
+0x0c,
+0x84,
+0x40,
+0xdc,
+0x00,
+0x00,
+0x29,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0xc0,
+0x00,
+0x02,
+0x98,
+0x0c,
+0x84,
+0x40,
+0xdc,
+0x00,
+0x00,
+0x29,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0x40,
+0x00,
+0x03,
+0x98,
+0x08,
+0x84,
+0x40,
+0x90,
+0x04,
+0x00,
+0x85,
+0x14,
+0x69,
+0x18,
+0x08,
+0xa2,
+0xe0,
+0x9a,
+0x55,
+0x00,
+0xd1,
+0x80,
+0x89,
+0x1a,
+0x0d,
+0x99,
+0xa0,
+0x42,
+0x08,
+0x63,
+0x02,
+0x20,
+0xdb,
+0x57,
+0x06,
+0x99,
+0x80,
+0xc9,
+0x98,
+0x0c,
+0xa0,
+0x86,
+0x30,
+0x22,
+0x0d,
+0xb5,
+0x70,
+0x69,
+0x98,
+0x0c,
+0x99,
+0x80,
+0xca,
+0x08,
+0x63,
+0x02,
+0x20,
+0xdb,
+0x57,
+0x06,
+0x91,
+0x80,
+0xc9,
+0x98,
+0x08,
+0xa0,
+0x86,
+0x20,
+0x22,
+0x0d,
+0x25,
+0x70,
+0x51,
+0x3a,
+0x14,
+0x82,
+0x20,
+0x92,
+0x40,
+0x00,
+0x01,
+0x80,
+0x88,
+0x85,
+0x00,
+0x88,
+0x40,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc1,
+0x50,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0xbd,
+0x84,
+0x80,
+0x82,
+0xe1,
+0x30,
+0x98,
+0x00,
+0x88,
+0x40,
+0x48,
+0x00,
+0x00,
+0x08,
+0x50,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0xb4,
+0xba,
+0x14,
+0x88,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x02,
+0x81,
+0x30,
+0x98,
+0x00,
+0x88,
+0x40,
+0x48,
+0x00,
+0x00,
+0x08,
+0x50,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x12,
+0x84,
+0x04,
+0xab,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x14,
+0x07,
+0x20,
+0x5c,
+0x00,
+0x63,
+0x07,
+0xf6,
+0x5c,
+0x1f,
+0x30,
+0x40,
+0x4a,
+0xa0,
+0x0c,
+0x0a,
+0x04,
+0x21,
+0x84,
+0x04,
+0x84,
+0x60,
+0xa4,
+0x04,
+0x84,
+0xa8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x14,
+0x04,
+0x20,
+0x38,
+0x10,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x52,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x17,
+0x21,
+0x5c,
+0x90,
+0x8a,
+0xc0,
+0x08,
+0x80,
+0xaa,
+0x09,
+0x48,
+0x2e,
+0x32,
+0x23,
+0x0b,
+0xc1,
+0x4c,
+0x68,
+0x00,
+0x00,
+0x06,
+0xa1,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x46,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0xba,
+0x14,
+0x86,
+0xe0,
+0x00,
+0x00,
+0xd6,
+0x60,
+0x00,
+0x00,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0x46,
+0x0a,
+0x41,
+0x48,
+0xc0,
+0x84,
+0x86,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x02,
+0x21,
+0x68,
+0x00,
+0x00,
+0x18,
+0x20,
+0x84,
+0x87,
+0xaa,
+0xbf,
+0xf0,
+0x84,
+0xf0,
+0x85,
+0x90,
+0x10,
+0x20,
+0x63,
+0x29,
+0x8e,
+0x8a,
+0x95,
+0x06,
+0x68,
+0x80,
+0x76,
+0xbc,
+0x05,
+0x98,
+0x40,
+0x21,
+0x5c,
+0x00,
+0x63,
+0xc2,
+0xff,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x44,
+0x39,
+0x00,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x00,
+0x00,
+0x94,
+0x84,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x91,
+0x43,
+0x01,
+0x2e,
+0x80,
+0x26,
+0x10,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x0b,
+0x52,
+0xcd,
+0xc1,
+0x40,
+0x2d,
+0x54,
+0x09,
+0x43,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x01,
+0x40,
+0x60,
+0x5c,
+0x00,
+0x63,
+0xc1,
+0x5f,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x7a,
+0xa0,
+0x05,
+0x98,
+0x40,
+0x22,
+0xba,
+0x0a,
+0x88,
+0x80,
+0xe1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x91,
+0x40,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x61,
+0x8e,
+0x8a,
+0x94,
+0x04,
+0x6a,
+0x06,
+0x21,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x84,
+0x06,
+0x10,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x05,
+0x20,
+0x38,
+0x12,
+0xc8,
+0x40,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xc0,
+0x68,
+0x5c,
+0x00,
+0x73,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x4a,
+0x64,
+0x00,
+0x01,
+0x7a,
+0x07,
+0x5c,
+0xbd,
+0x03,
+0x00,
+0x4c,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x4a,
+0x9c,
+0x00,
+0x08,
+0x40,
+0x48,
+0x84,
+0x07,
+0xa4,
+0x60,
+0xa4,
+0x18,
+0xe8,
+0x86,
+0xe0,
+0x00,
+0x00,
+0xd6,
+0x40,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x17,
+0x20,
+0x00,
+0x00,
+0x0a,
+0x06,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x22,
+0x30,
+0xbc,
+0x15,
+0xc5,
+0xc8,
+0x04,
+0x04,
+0x02,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x09,
+0x56,
+0x20,
+0x00,
+0x00,
+0x04,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x40,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x02,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x0f,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0x66,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0x60,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x02,
+0xe6,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x47,
+0x20,
+0x5c,
+0x00,
+0x63,
+0x07,
+0xf6,
+0x5c,
+0x1f,
+0x30,
+0x40,
+0x4a,
+0xa0,
+0x0c,
+0x0a,
+0x04,
+0x21,
+0x84,
+0x04,
+0x84,
+0x60,
+0xa4,
+0x04,
+0x84,
+0xa8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x14,
+0x44,
+0x20,
+0x38,
+0x10,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x52,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x42,
+0x21,
+0x68,
+0x00,
+0x01,
+0x23,
+0x20,
+0x84,
+0x87,
+0xaa,
+0xbf,
+0xf0,
+0x84,
+0xf0,
+0x85,
+0x90,
+0x10,
+0x20,
+0x63,
+0x29,
+0x8e,
+0x8a,
+0x95,
+0x06,
+0x68,
+0x80,
+0x76,
+0xbc,
+0x05,
+0x98,
+0x40,
+0x21,
+0x5c,
+0x00,
+0x63,
+0xc2,
+0xff,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x44,
+0x39,
+0x00,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x80,
+0x00,
+0x94,
+0x84,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x91,
+0x43,
+0x01,
+0x2e,
+0x80,
+0x26,
+0x10,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x0b,
+0x52,
+0xcd,
+0xc1,
+0x40,
+0x2d,
+0x54,
+0x09,
+0x43,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x01,
+0x40,
+0x60,
+0x5c,
+0x00,
+0x63,
+0xc1,
+0x5f,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x68,
+0x00,
+0x01,
+0x0f,
+0x20,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x7a,
+0xa0,
+0x05,
+0x98,
+0x40,
+0x22,
+0xba,
+0x0a,
+0x88,
+0x80,
+0xe1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x91,
+0x40,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x61,
+0x8e,
+0x8a,
+0x94,
+0x04,
+0x6a,
+0x06,
+0x21,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x84,
+0x06,
+0x10,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x45,
+0x20,
+0x38,
+0x12,
+0xc8,
+0x40,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xc0,
+0x68,
+0x5c,
+0x00,
+0x73,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x4a,
+0x64,
+0x00,
+0x01,
+0x84,
+0x47,
+0x5c,
+0xbd,
+0x03,
+0x00,
+0x4c,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x4a,
+0x9c,
+0x00,
+0x08,
+0x40,
+0x48,
+0x84,
+0x07,
+0xa4,
+0x60,
+0xa4,
+0x18,
+0xe8,
+0x86,
+0xe0,
+0x00,
+0x22,
+0x36,
+0x40,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0x22,
+0x20,
+0x00,
+0x00,
+0x0a,
+0x06,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x22,
+0x30,
+0xbc,
+0x15,
+0xc5,
+0xc8,
+0x04,
+0x04,
+0x02,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x09,
+0x56,
+0x20,
+0x00,
+0x00,
+0x04,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x40,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x84,
+0x48,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x0f,
+0x6e,
+0x00,
+0x02,
+0x23,
+0x66,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x88,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x02,
+0x23,
+0x60,
+0x6c,
+0x70,
+0x28,
+0x84,
+0x48,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x24,
+0x46,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x06,
+0x1e,
+0x20,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x33,
+0x26,
+0x00,
+0x00,
+0x00,
+0xab,
+0xfd,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0x78,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x83,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x8e,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x8e,
+0x20,
+0x68,
+0x00,
+0x01,
+0x83,
+0x24,
+0xa0,
+0x04,
+0x06,
+0xc4,
+0x00,
+0x3a,
+0x60,
+0x88,
+0x40,
+0x09,
+0x44,
+0x20,
+0x02,
+0x20,
+0x44,
+0x68,
+0x00,
+0x01,
+0x78,
+0x21,
+0x86,
+0x00,
+0x98,
+0x80,
+0xe4,
+0xa0,
+0x84,
+0x46,
+0xc4,
+0x00,
+0x39,
+0xe0,
+0x84,
+0x42,
+0x10,
+0x06,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x39,
+0x60,
+0x94,
+0x44,
+0x80,
+0x1a,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3a,
+0xc0,
+0x94,
+0x60,
+0x88,
+0x88,
+0x16,
+0x44,
+0x40,
+0x88,
+0x18,
+0x00,
+0x84,
+0xd0,
+0xd0,
+0x88,
+0x1e,
+0x02,
+0x2c,
+0x40,
+0x51,
+0x62,
+0x80,
+0x82,
+0x4c,
+0x51,
+0x62,
+0x41,
+0x83,
+0x09,
+0x84,
+0x04,
+0xc8,
+0x82,
+0xc9,
+0x68,
+0x20,
+0x01,
+0xf8,
+0x24,
+0x66,
+0x00,
+0x01,
+0xa0,
+0xc8,
+0x46,
+0x08,
+0x08,
+0x82,
+0x0a,
+0x88,
+0x20,
+0x98,
+0x82,
+0x48,
+0x68,
+0x20,
+0x01,
+0xfb,
+0x24,
+0x66,
+0x00,
+0x01,
+0xa0,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x82,
+0x8a,
+0x5c,
+0x82,
+0x00,
+0x81,
+0x20,
+0x88,
+0x20,
+0x98,
+0x80,
+0xa4,
+0x80,
+0x04,
+0x98,
+0x60,
+0x48,
+0x40,
+0x00,
+0x00,
+0x82,
+0x60,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x56,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x82,
+0x00,
+0x80,
+0xa0,
+0x88,
+0x22,
+0x48,
+0x81,
+0x21,
+0x80,
+0x00,
+0x98,
+0x61,
+0xc8,
+0x88,
+0x16,
+0x08,
+0x49,
+0x48,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x62,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x82,
+0x00,
+0x81,
+0xa0,
+0x88,
+0x12,
+0x48,
+0x80,
+0xa1,
+0x80,
+0x00,
+0x98,
+0x61,
+0xc8,
+0x88,
+0x0e,
+0x08,
+0x49,
+0x48,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x6e,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x81,
+0x00,
+0x82,
+0x21,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x88,
+0x1a,
+0x08,
+0x80,
+0xa2,
+0x80,
+0x80,
+0x9a,
+0x20,
+0x64,
+0x84,
+0x14,
+0x88,
+0x51,
+0xc8,
+0x88,
+0x1e,
+0x18,
+0x82,
+0x64,
+0x68,
+0x00,
+0x01,
+0x56,
+0x23,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x40,
+0x00,
+0x02,
+0x18,
+0x40,
+0x5c,
+0x83,
+0x08,
+0x81,
+0xa1,
+0x5c,
+0x88,
+0x00,
+0x81,
+0x22,
+0x5c,
+0x81,
+0x08,
+0x08,
+0xc8,
+0x80,
+0xa4,
+0x88,
+0x81,
+0x61,
+0x68,
+0x00,
+0x01,
+0x62,
+0x20,
+0x81,
+0x08,
+0x98,
+0x82,
+0x24,
+0x88,
+0x1e,
+0x26,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x00,
+0x40,
+0x5c,
+0x83,
+0x08,
+0x81,
+0xa1,
+0x5c,
+0x81,
+0x00,
+0x80,
+0xa2,
+0x5c,
+0x88,
+0x08,
+0x08,
+0xc8,
+0x80,
+0xac,
+0x88,
+0x80,
+0xe1,
+0x68,
+0x00,
+0x01,
+0x6e,
+0x20,
+0x81,
+0x00,
+0x98,
+0x82,
+0x24,
+0x88,
+0x1e,
+0x26,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x00,
+0x40,
+0x5c,
+0x83,
+0x00,
+0x81,
+0xa4,
+0x5c,
+0x88,
+0x08,
+0x81,
+0x20,
+0x82,
+0x04,
+0x88,
+0x22,
+0xc8,
+0x88,
+0x1e,
+0x46,
+0x60,
+0x00,
+0x1a,
+0x14,
+0x08,
+0x81,
+0x24,
+0x88,
+0x0a,
+0x08,
+0x63,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa1,
+0x40,
+0x88,
+0x0a,
+0x48,
+0x81,
+0xa0,
+0x40,
+0x00,
+0x00,
+0x63,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x06,
+0x4d,
+0x24,
+0x88,
+0x1a,
+0x08,
+0x80,
+0x36,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x64,
+0xba,
+0x14,
+0x88,
+0x43,
+0x48,
+0x40,
+0x00,
+0x02,
+0x80,
+0x30,
+0x68,
+0x00,
+0x01,
+0x89,
+0x24,
+0xab,
+0xfc,
+0x06,
+0xc0,
+0x00,
+0x11,
+0x40,
+0xb6,
+0xc0,
+0x00,
+0x12,
+0x60,
+0x06,
+0xc0,
+0x00,
+0x29,
+0x40,
+0x16,
+0xc0,
+0x00,
+0x32,
+0x80,
+0xa8,
+0x60,
+0x09,
+0x88,
+0x05,
+0x18,
+0x80,
+0xd0,
+0x88,
+0x14,
+0xb8,
+0x81,
+0xf6,
+0x68,
+0x00,
+0x01,
+0x7e,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa7,
+0x28,
+0x40,
+0x00,
+0x00,
+0x40,
+0x08,
+0x68,
+0x00,
+0x01,
+0x89,
+0x21,
+0x68,
+0x20,
+0x01,
+0xdd,
+0x20,
+0x68,
+0x00,
+0x01,
+0x7e,
+0x24,
+0x5c,
+0x83,
+0x00,
+0x40,
+0x88,
+0x80,
+0xa0,
+0xb5,
+0x70,
+0x9c,
+0x02,
+0x20,
+0x98,
+0x40,
+0x08,
+0x88,
+0x36,
+0x45,
+0x70,
+0x94,
+0x08,
+0x24,
+0x08,
+0x82,
+0xe1,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0x54,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x98,
+0x00,
+0x98,
+0x83,
+0x20,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x84,
+0x04,
+0x86,
+0x80,
+0x00,
+0x16,
+0x02,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x88,
+0x82,
+0x09,
+0x55,
+0x01,
+0x28,
+0x83,
+0x20,
+0x5c,
+0x83,
+0x00,
+0x82,
+0xa4,
+0x80,
+0x20,
+0x88,
+0x60,
+0x49,
+0x40,
+0x00,
+0x00,
+0x82,
+0x60,
+0x68,
+0x00,
+0x01,
+0x54,
+0x21,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc1,
+0x22,
+0x66,
+0x00,
+0x01,
+0xa1,
+0xa0,
+0x5c,
+0x83,
+0x00,
+0x82,
+0xa0,
+0x68,
+0x00,
+0x01,
+0x60,
+0x21,
+0x80,
+0x20,
+0x88,
+0x82,
+0xe0,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc3,
+0x22,
+0x66,
+0x00,
+0x01,
+0xa1,
+0xa0,
+0x88,
+0x10,
+0x83,
+0x69,
+0x00,
+0x98,
+0x00,
+0x96,
+0xc4,
+0x00,
+0x3a,
+0x80,
+0xb5,
+0x80,
+0xf4,
+0x18,
+0xe8,
+0x9b,
+0xc0,
+0x3a,
+0x55,
+0x01,
+0x70,
+0x80,
+0x88,
+0xb0,
+0x00,
+0xd3,
+0x69,
+0x00,
+0x98,
+0x00,
+0x03,
+0x01,
+0xc0,
+0xbc,
+0x03,
+0xa5,
+0x50,
+0x18,
+0x08,
+0x00,
+0x8b,
+0x00,
+0x08,
+0x36,
+0x90,
+0x19,
+0x80,
+0x48,
+0x30,
+0x1e,
+0x0b,
+0xc0,
+0x12,
+0xb0,
+0x00,
+0xe3,
+0x20,
+0x28,
+0xbc,
+0x05,
+0x98,
+0x82,
+0x20,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x21,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x60,
+0x6c,
+0x40,
+0x03,
+0xaa,
+0x08,
+0x6c,
+0x00,
+0x03,
+0x94,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x96,
+0x48,
+0xa0,
+0x0c,
+0x08,
+0x40,
+0x09,
+0x88,
+0x06,
+0x06,
+0x82,
+0x00,
+0x0a,
+0xa2,
+0x46,
+0x80,
+0x00,
+0x1c,
+0x62,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x08,
+0x82,
+0xa0,
+0x88,
+0x0c,
+0x8a,
+0x00,
+0xc0,
+0x84,
+0x00,
+0x98,
+0x81,
+0x60,
+0x68,
+0x20,
+0x00,
+0xc2,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc8,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x66,
+0x00,
+0x01,
+0xaa,
+0x88,
+0x55,
+0x01,
+0x28,
+0x80,
+0x88,
+0x6c,
+0x00,
+0x03,
+0x94,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x68,
+0x00,
+0x01,
+0x81,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x42,
+0x02,
+0xb8,
+0x40,
+0xc8,
+0x2a,
+0x7e,
+0x46,
+0xc0,
+0x00,
+0x39,
+0x44,
+0x86,
+0xc0,
+0x00,
+0x30,
+0x47,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x96,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x68,
+0x00,
+0x01,
+0x8c,
+0x20,
+0x00,
+0x00,
+0x04,
+0x20,
+0x37,
+0x84,
+0x00,
+0x88,
+0x40,
+0xc8,
+0x2a,
+0x7e,
+0x46,
+0xc0,
+0x00,
+0x39,
+0x64,
+0x86,
+0xc0,
+0x00,
+0x31,
+0xa7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x00,
+0x80,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x04,
+0x7a,
+0x6c,
+0x00,
+0x03,
+0x1a,
+0x7a,
+0x66,
+0x00,
+0x01,
+0xa6,
+0x68,
+0x84,
+0x20,
+0x98,
+0x81,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x46,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa6,
+0x68,
+0x40,
+0x00,
+0x00,
+0x42,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x4a,
+0x48,
+0x68,
+0x00,
+0x06,
+0x80,
+0x20,
+0x88,
+0x1b,
+0x6b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x40,
+0x68,
+0x00,
+0x06,
+0x1c,
+0xa0,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x33,
+0x26,
+0x00,
+0x00,
+0x00,
+0x84,
+0x00,
+0x88,
+0x40,
+0x8a,
+0x57,
+0x0d,
+0x03,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x00,
+0x00,
+0x00,
+0xa2,
+0x02,
+0x48,
+0x60,
+0x08,
+0x44,
+0x20,
+0x00,
+0x60,
+0x89,
+0xba,
+0x14,
+0x80,
+0x89,
+0x80,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0xa0,
+0x04,
+0x08,
+0x41,
+0x88,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x0a,
+0x2e,
+0x13,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x81,
+0x02,
+0x22,
+0x23,
+0x81,
+0x80,
+0x94,
+0x40,
+0x80,
+0x05,
+0x80,
+0x95,
+0x00,
+0xa0,
+0x20,
+0x8c,
+0x19,
+0x80,
+0x09,
+0x80,
+0x84,
+0x9a,
+0x1c,
+0x64,
+0x82,
+0x00,
+0x84,
+0x42,
+0x00,
+0x06,
+0x00,
+0x98,
+0x48,
+0x08,
+0x44,
+0x0c,
+0x02,
+0xbf,
+0xd0,
+0x98,
+0x00,
+0x9a,
+0x24,
+0xe4,
+0x80,
+0x84,
+0x98,
+0x80,
+0x61,
+0x88,
+0x0e,
+0x48,
+0x81,
+0x62,
+0x88,
+0x1e,
+0x08,
+0x82,
+0x76,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0xa0,
+0x80,
+0x08,
+0x80,
+0x20,
+0x88,
+0x0a,
+0x2a,
+0x00,
+0x40,
+0x88,
+0x06,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x85,
+0x50,
+0x12,
+0xa1,
+0x06,
+0x48,
+0x80,
+0x20,
+0x88,
+0x12,
+0x2a,
+0x04,
+0x60,
+0x84,
+0x00,
+0x95,
+0x70,
+0x94,
+0x05,
+0x08,
+0x95,
+0x70,
+0xa0,
+0x08,
+0x1a,
+0x09,
+0x80,
+0x08,
+0x59,
+0x01,
+0x00,
+0x44,
+0xc8,
+0xbc,
+0x0e,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x17,
+0x38,
+0x50,
+0x08,
+0x2e,
+0x12,
+0xd5,
+0xb4,
+0xa0,
+0x05,
+0x0c,
+0x99,
+0x80,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x22,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0xe5,
+0x36,
+0x10,
+0x58,
+0x50,
+0xc9,
+0xbc,
+0x0b,
+0x78,
+0x50,
+0x08,
+0x28,
+0x12,
+0xd5,
+0xb4,
+0xa0,
+0x05,
+0x0c,
+0x99,
+0x80,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x22,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x25,
+0x85,
+0x0c,
+0x80,
+0x00,
+0x00,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0xc0,
+0x08,
+0x44,
+0x20,
+0x03,
+0xa1,
+0x11,
+0x98,
+0x00,
+0x96,
+0x80,
+0x39,
+0x4b,
+0xb8,
+0x84,
+0x42,
+0x00,
+0x3a,
+0x14,
+0x82,
+0x2b,
+0x80,
+0x46,
+0x08,
+0x09,
+0x83,
+0x08,
+0x55,
+0x01,
+0x12,
+0xbf,
+0xf0,
+0x5b,
+0x0c,
+0x30,
+0x80,
+0x4a,
+0x44,
+0x58,
+0x81,
+0x84,
+0x48,
+0x44,
+0x21,
+0x81,
+0x80,
+0x41,
+0x5b,
+0x04,
+0x32,
+0xc0,
+0x20,
+0x44,
+0x40,
+0x81,
+0x84,
+0x48,
+0x5b,
+0x0a,
+0x29,
+0x84,
+0x8a,
+0x44,
+0x58,
+0x01,
+0x84,
+0x0b,
+0x44,
+0x39,
+0x00,
+0x80,
+0x09,
+0x44,
+0x39,
+0x81,
+0x80,
+0xca,
+0x5c,
+0x00,
+0xeb,
+0xa1,
+0x11,
+0x2a,
+0x02,
+0x74,
+0x46,
+0x88,
+0x1a,
+0x14,
+0x14,
+0x46,
+0x88,
+0x18,
+0x34,
+0x34,
+0xc0,
+0xc0,
+0x9a,
+0x00,
+0x09,
+0x83,
+0x8b,
+0x44,
+0x08,
+0x01,
+0x83,
+0x08,
+0x54,
+0x07,
+0xd1,
+0x83,
+0x41,
+0x54,
+0x08,
+0x41,
+0x83,
+0x0b,
+0x44,
+0x48,
+0x83,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x03,
+0xae,
+0x08,
+0x9a,
+0x00,
+0x24,
+0xc2,
+0x40,
+0x18,
+0x34,
+0x96,
+0x82,
+0x00,
+0x1d,
+0xd2,
+0x02,
+0x81,
+0x79,
+0x80,
+0x04,
+0x29,
+0x80,
+0x4b,
+0x44,
+0x60,
+0x00,
+0x00,
+0x40,
+0xba,
+0x14,
+0x88,
+0x40,
+0x40,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x38,
+0x13,
+0x22,
+0x58,
+0xb0,
+0xbc,
+0x05,
+0x0b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0xcc,
+0x21,
+0x68,
+0x20,
+0x01,
+0xe1,
+0x24,
+0x5c,
+0x81,
+0x08,
+0x48,
+0x02,
+0x5c,
+0x82,
+0x00,
+0x20,
+0x8a,
+0x68,
+0x00,
+0x01,
+0xd0,
+0x20,
+0x30,
+0x19,
+0x04,
+0x20,
+0xec,
+0x00,
+0x00,
+0x08,
+0x02,
+0x8a,
+0x68,
+0x00,
+0x01,
+0xcc,
+0x20,
+0x55,
+0x02,
+0x92,
+0xc0,
+0xa0,
+0x58,
+0x01,
+0x00,
+0x00,
+0x52,
+0xbc,
+0x03,
+0x56,
+0xc0,
+0x00,
+0x3a,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0x23,
+0x00,
+0xa0,
+0xbc,
+0x02,
+0x36,
+0xc0,
+0x00,
+0x3a,
+0x24,
+0x83,
+0x01,
+0xa8,
+0xbc,
+0x03,
+0x56,
+0xc0,
+0x00,
+0x3a,
+0x44,
+0x90,
+0x00,
+0x00,
+0x84,
+0x10,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x04,
+0x3b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xa6,
+0x49,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x80,
+0x00,
+0xb5,
+0x70,
+0xe1,
+0x06,
+0x00,
+0x35,
+0x84,
+0x68,
+0x00,
+0x28,
+0x94,
+0x20,
+0x1d,
+0x98,
+0xe8,
+0x85,
+0x70,
+0xb8,
+0x98,
+0x20,
+0x2b,
+0x00,
+0x0c,
+0x30,
+0x8c,
+0x8b,
+0xc0,
+0x23,
+0x40,
+0x00,
+0x03,
+0x00,
+0x0a,
+0x5b,
+0x4e,
+0x08,
+0x60,
+0x83,
+0x36,
+0x80,
+0x05,
+0xb4,
+0xc0,
+0x18,
+0x00,
+0xb5,
+0xb4,
+0xa0,
+0x98,
+0x04,
+0xa5,
+0x78,
+0xdf,
+0x98,
+0x00,
+0x95,
+0x84,
+0x7c,
+0x18,
+0x04,
+0xab,
+0xc3,
+0x0b,
+0x2f,
+0x1a,
+0xd3,
+0x08,
+0xe8,
+0xbc,
+0x2d,
+0x33,
+0x20,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0xbc,
+0x04,
+0x86,
+0xc4,
+0x00,
+0x3c,
+0xc0,
+0x83,
+0x20,
+0x10,
+0xbc,
+0x08,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x0f,
+0x56,
+0xc4,
+0x00,
+0x3c,
+0xa0,
+0xa2,
+0xe1,
+0xad,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x49,
+0xbc,
+0x09,
+0x73,
+0x01,
+0x28,
+0xbc,
+0x07,
+0x36,
+0xc4,
+0x00,
+0x3c,
+0x80,
+0xa2,
+0x81,
+0xad,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x49,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0x32,
+0x06,
+0x8b,
+0xc0,
+0x63,
+0x68,
+0x00,
+0x01,
+0xce,
+0x21,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x7a,
+0x84,
+0x8f,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x6a,
+0x38,
+0x00,
+0xd6,
+0x80,
+0x00,
+0x1c,
+0xe2,
+0x16,
+0xc0,
+0x00,
+0x39,
+0xc4,
+0x98,
+0x48,
+0xc8,
+0x68,
+0x20,
+0x00,
+0xfc,
+0x21,
+0x39,
+0x0a,
+0x18,
+0x48,
+0x08,
+0x80,
+0x24,
+0x88,
+0x40,
+0x48,
+0xa0,
+0x06,
+0x08,
+0x48,
+0x88,
+0x80,
+0x24,
+0x84,
+0x60,
+0xa4,
+0x00,
+0x2c,
+0x88,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0x68,
+0x00,
+0x01,
+0x54,
+0x20,
+0x68,
+0x00,
+0x01,
+0x60,
+0x21,
+0x68,
+0x00,
+0x01,
+0x6c,
+0x22,
+0x55,
+0x01,
+0x33,
+0x80,
+0x00,
+0x55,
+0x03,
+0xb0,
+0x00,
+0x7a,
+0x5d,
+0x0c,
+0x30,
+0x08,
+0x7a,
+0x32,
+0x2b,
+0x0b,
+0xff,
+0xaa,
+0x40,
+0x00,
+0x00,
+0x10,
+0x7a,
+0x68,
+0x00,
+0x01,
+0x85,
+0x22,
+0x68,
+0x00,
+0x01,
+0x90,
+0x21,
+0x68,
+0x00,
+0x01,
+0x7a,
+0x20,
+0x55,
+0x03,
+0x20,
+0x10,
+0x7a,
+0x5d,
+0x08,
+0x20,
+0x08,
+0x7a,
+0x32,
+0x26,
+0x0b,
+0xff,
+0xaa,
+0x80,
+0x07,
+0xa6,
+0x80,
+0x00,
+0x1c,
+0xc2,
+0x06,
+0x82,
+0x00,
+0x0f,
+0xc2,
+0x15,
+0xc8,
+0x20,
+0x80,
+0x07,
+0xa5,
+0xc0,
+0x16,
+0x04,
+0x0f,
+0xa8,
+0x00,
+0x7a,
+0xa0,
+0x08,
+0x08,
+0x48,
+0x0a,
+0x80,
+0x2c,
+0xa8,
+0x40,
+0x4a,
+0xa0,
+0x06,
+0x08,
+0x48,
+0x8a,
+0x80,
+0x2c,
+0xa6,
+0x80,
+0x00,
+0x1c,
+0xa2,
+0x16,
+0x80,
+0x00,
+0x61,
+0xca,
+0xc6,
+0xc4,
+0x00,
+0x3a,
+0xa0,
+0x08,
+0x02,
+0xca,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x6c,
+0x84,
+0x85,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x04,
+0x88,
+0x48,
+0xd0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xa0,
+0x05,
+0x89,
+0x8e,
+0x88,
+0x94,
+0x06,
+0x4a,
+0x00,
+0x08,
+0xa0,
+0x22,
+0x18,
+0x48,
+0x60,
+0xa0,
+0xe3,
+0x09,
+0x40,
+0x64,
+0xa0,
+0x01,
+0x0a,
+0x02,
+0x01,
+0x84,
+0x86,
+0x0a,
+0x08,
+0x80,
+0x84,
+0x07,
+0xaa,
+0x06,
+0xa0,
+0x94,
+0x06,
+0x4a,
+0x04,
+0x40,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x84,
+0x0e,
+0xc6,
+0x80,
+0x00,
+0x6e,
+0x2a,
+0xcb,
+0xa1,
+0x48,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x94,
+0x5a,
+0x83,
+0x20,
+0x00,
+0xbc,
+0x0a,
+0x59,
+0x46,
+0x28,
+0x22,
+0x90,
+0x42,
+0xab,
+0xe0,
+0x23,
+0x08,
+0x06,
+0x82,
+0x00,
+0x1f,
+0xe2,
+0x19,
+0x84,
+0x28,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x21,
+0xba,
+0x01,
+0x0b,
+0xa1,
+0x40,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0xa0,
+0x05,
+0x09,
+0x40,
+0x64,
+0xa0,
+0x01,
+0x0a,
+0x02,
+0x01,
+0x84,
+0x86,
+0x0a,
+0x0e,
+0x60,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x20,
+0x64,
+0x00,
+0x01,
+0x84,
+0x47,
+0x64,
+0x00,
+0x01,
+0x71,
+0x87,
+0x39,
+0x00,
+0x89,
+0x48,
+0x0c,
+0x51,
+0xb1,
+0x21,
+0x48,
+0x0e,
+0x51,
+0xa1,
+0xb1,
+0x48,
+0x08,
+0x51,
+0x90,
+0x11,
+0x48,
+0x28,
+0x29,
+0x1a,
+0x42,
+0x90,
+0xa4,
+0x54,
+0x81,
+0x23,
+0xa1,
+0x48,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x80,
+0x40,
+0x40,
+0x00,
+0x51,
+0xf0,
+0x23,
+0x07,
+0xfa,
+0x28,
+0x8a,
+0x15,
+0x1e,
+0x02,
+0x14,
+0x84,
+0x12,
+0x88,
+0xa1,
+0x51,
+0xd0,
+0x21,
+0x48,
+0x41,
+0x28,
+0x8a,
+0x15,
+0x44,
+0x40,
+0x14,
+0x84,
+0x1b,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x00,
+0x00,
+0x00,
+0x94,
+0x82,
+0xc5,
+0xd4,
+0x83,
+0x14,
+0x8a,
+0xc2,
+0x32,
+0x36,
+0x3a,
+0x90,
+0x43,
+0xa9,
+0x86,
+0x54,
+0x89,
+0xa3,
+0xa1,
+0x48,
+0x94,
+0x07,
+0x40,
+0x00,
+0x00,
+0x94,
+0x03,
+0x85,
+0x15,
+0x02,
+0x30,
+0x7f,
+0xa2,
+0x88,
+0xa1,
+0x54,
+0x44,
+0x01,
+0x48,
+0x61,
+0xba,
+0x14,
+0x89,
+0x48,
+0xe0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x80,
+0x4a,
+0x02,
+0xc0,
+0x5c,
+0x80,
+0x92,
+0xbf,
+0xf0,
+0x62,
+0x00,
+0x00,
+0x00,
+0xb4,
+0x5c,
+0x81,
+0x02,
+0x40,
+0x02,
+0x5c,
+0x00,
+0x00,
+0x40,
+0x24,
+0x60,
+0x00,
+0x00,
+0x00,
+0x35,
+0x8d,
+0x05,
+0x80,
+0x00,
+0x00,
+0x94,
+0x88,
+0xc9,
+0x52,
+0xc4,
+0x00,
+0x00,
+0x0a,
+0x10,
+0x0a,
+0x8d,
+0x03,
+0x2b,
+0xb1,
+0x82,
+0xba,
+0x14,
+0x88,
+0x40,
+0x64,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x5c,
+0x80,
+0x52,
+0xbf,
+0xf0,
+0xa0,
+0x2c,
+0x0a,
+0x40,
+0x02,
+0x62,
+0x00,
+0x00,
+0x00,
+0xa4,
+0x5c,
+0x81,
+0x08,
+0x40,
+0x24,
+0x5c,
+0x80,
+0x82,
+0x06,
+0x61,
+0xbb,
+0x00,
+0x06,
+0x00,
+0x00,
+0x00,
+0x03,
+0x58,
+0xd0,
+0xd8,
+0x00,
+0x00,
+0x09,
+0x53,
+0x0c,
+0x94,
+0x94,
+0x40,
+0x00,
+0x00,
+0xa1,
+0x00,
+0xab,
+0xa1,
+0x48,
+0x84,
+0x06,
+0x4a,
+0x80,
+0x10,
+0xa0,
+0x06,
+0x19,
+0x48,
+0x28,
+0x55,
+0x5e,
+0x02,
+0x08,
+0x0a,
+0x51,
+0x90,
+0x01,
+0x50,
+0x2a,
+0x54,
+0x04,
+0x22,
+0x14,
+0x13,
+0x95,
+0x82,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x04,
+0x20,
+0xcc,
+0x08,
+0x07,
+0x69,
+0x82,
+0x22,
+0x32,
+0x0c,
+0x0b,
+0xc1,
+0x38,
+0xa0,
+0x81,
+0x02,
+0xa7,
+0x80,
+0x22,
+0x88,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x1b,
+0x56,
+0x20,
+0x00,
+0x00,
+0x08,
+0x6a,
+0x00,
+0x01,
+0x00,
+0x00,
+0x09,
+0x82,
+0x20,
+0x88,
+0x16,
+0x18,
+0x80,
+0xc8,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x18,
+0x80,
+0x80,
+0x55,
+0x08,
+0x22,
+0x08,
+0x21,
+0xbc,
+0x0d,
+0x79,
+0x49,
+0x28,
+0x42,
+0x05,
+0x39,
+0x50,
+0x60,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x8a,
+0x10,
+0x00,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x20,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x94,
+0x5a,
+0xc5,
+0x90,
+0x30,
+0x20,
+0x2c,
+0x04,
+0x20,
+0x34,
+0x2b,
+0xff,
+0x08,
+0x80,
+0x76,
+0xa0,
+0x65,
+0x98,
+0x80,
+0x36,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x01,
+0x0a,
+0x06,
+0x61,
+0x88,
+0x0e,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x08,
+0x80,
+0xa0,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x10,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x04,
+0x60,
+0x94,
+0x5a,
+0xc3,
+0x20,
+0x60,
+0xbc,
+0x16,
+0x8a,
+0xbf,
+0xf0,
+0x55,
+0x3f,
+0x23,
+0x00,
+0xfe,
+0x5b,
+0x48,
+0x13,
+0x00,
+0x28,
+0x37,
+0x08,
+0x52,
+0xe1,
+0x76,
+0x62,
+0x00,
+0x00,
+0x00,
+0x26,
+0x36,
+0x80,
+0x05,
+0x04,
+0xc9,
+0x18,
+0xeb,
+0x52,
+0xf8,
+0x12,
+0x2b,
+0x96,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x02,
+0xb9,
+0x83,
+0x88,
+0x36,
+0x10,
+0x4a,
+0x00,
+0x69,
+0x64,
+0x00,
+0x01,
+0xbd,
+0xef,
+0xa8,
+0x01,
+0x08,
+0x80,
+0x76,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xf6,
+0x83,
+0x80,
+0x64,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x1e,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x10,
+0xa0,
+0x06,
+0x19,
+0x48,
+0x28,
+0x55,
+0x5e,
+0x12,
+0x08,
+0x0a,
+0x51,
+0x90,
+0x91,
+0x50,
+0x28,
+0x68,
+0x20,
+0x00,
+0x00,
+0x2c,
+0x54,
+0x00,
+0xa1,
+0x8b,
+0x00,
+0x54,
+0x01,
+0x22,
+0x14,
+0x12,
+0x95,
+0x02,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x04,
+0x20,
+0xcc,
+0x08,
+0x07,
+0x69,
+0x82,
+0x22,
+0x32,
+0x0c,
+0x0b,
+0xc1,
+0x38,
+0xa0,
+0x81,
+0x02,
+0xa7,
+0x80,
+0x22,
+0x88,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x1b,
+0x56,
+0x20,
+0x00,
+0x00,
+0x08,
+0x6a,
+0x00,
+0x01,
+0x00,
+0x00,
+0x09,
+0x82,
+0x20,
+0x88,
+0x16,
+0x18,
+0x80,
+0xc8,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x18,
+0x80,
+0x80,
+0x55,
+0x08,
+0x22,
+0x08,
+0x21,
+0xbc,
+0x0d,
+0x79,
+0x49,
+0x28,
+0x42,
+0x05,
+0x39,
+0x50,
+0x60,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x8a,
+0x10,
+0x00,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x20,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x94,
+0x5a,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x08,
+0x80,
+0x76,
+0x42,
+0x10,
+0xc2,
+0x00,
+0x01,
+0xa0,
+0x2a,
+0x03,
+0x21,
+0x80,
+0xbc,
+0x19,
+0x8a,
+0x08,
+0x71,
+0x2a,
+0x78,
+0x02,
+0x28,
+0xc4,
+0x32,
+0x02,
+0x0b,
+0xc2,
+0x45,
+0x62,
+0x00,
+0x00,
+0x00,
+0xf4,
+0xa0,
+0xa3,
+0x08,
+0x80,
+0xe0,
+0x88,
+0x16,
+0x10,
+0x00,
+0x00,
+0x88,
+0x0a,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x08,
+0x81,
+0x21,
+0x88,
+0x0a,
+0x0a,
+0x08,
+0x21,
+0x84,
+0x02,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x02,
+0x1b,
+0xc1,
+0x07,
+0xa0,
+0x63,
+0x18,
+0x80,
+0x36,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x02,
+0x0a,
+0x06,
+0x41,
+0x88,
+0x0e,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x08,
+0x80,
+0xa0,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x20,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x04,
+0x60,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x05,
+0x9a,
+0xbf,
+0xf0,
+0x94,
+0x82,
+0xc5,
+0x90,
+0x50,
+0x20,
+0xa4,
+0xa8,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x64,
+0x20,
+0x3c,
+0x20,
+0x06,
+0x18,
+0x50,
+0x20,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xb6,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x0b,
+0x68,
+0x80,
+0x20,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x10,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x19,
+0x46,
+0x2c,
+0xb0,
+0x70,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x02,
+0x16,
+0x40,
+0x00,
+0x1d,
+0x0c,
+0x7b,
+0xa1,
+0x40,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xae,
+0x59,
+0x01,
+0x82,
+0xbf,
+0xd0,
+0xbd,
+0x7b,
+0x88,
+0x80,
+0x76,
+0x32,
+0x07,
+0x0b,
+0xd6,
+0x20,
+0x32,
+0x0b,
+0x0b,
+0xd5,
+0x40,
+0x32,
+0x0f,
+0x0b,
+0xd3,
+0x40,
+0x32,
+0x13,
+0x0b,
+0xd2,
+0x60,
+0x32,
+0x17,
+0x0b,
+0xd0,
+0x60,
+0x32,
+0x1b,
+0x0b,
+0xcf,
+0x80,
+0x32,
+0x1f,
+0x0b,
+0xce,
+0x00,
+0x32,
+0x23,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x00,
+0xf9,
+0x8e,
+0x8b,
+0x32,
+0x03,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x82,
+0x8b,
+0xcb,
+0x68,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0xac,
+0x32,
+0x27,
+0x0b,
+0xcb,
+0x20,
+0x32,
+0x03,
+0x8b,
+0xc8,
+0x20,
+0x32,
+0x2b,
+0x0b,
+0xc8,
+0x00,
+0x32,
+0x03,
+0x8b,
+0xc4,
+0xf0,
+0x32,
+0x2f,
+0x0b,
+0xc4,
+0xd0,
+0x32,
+0x03,
+0x8b,
+0xc0,
+0x20,
+0x32,
+0x33,
+0x0b,
+0xd5,
+0xf1,
+0x59,
+0x05,
+0x02,
+0xc0,
+0x20,
+0xbc,
+0x2c,
+0x86,
+0x80,
+0x00,
+0x09,
+0x82,
+0x13,
+0x21,
+0xa0,
+0xbd,
+0x58,
+0x9a,
+0x08,
+0x00,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0x66,
+0x00,
+0x01,
+0xbc,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x98,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0xa0,
+0x02,
+0x08,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x01,
+0xbc,
+0xa8,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xa0,
+0x6c,
+0x00,
+0x01,
+0x30,
+0x0b,
+0x51,
+0xa1,
+0xf8,
+0x40,
+0x08,
+0x23,
+0x42,
+0x45,
+0x90,
+0x1c,
+0x04,
+0x04,
+0x86,
+0xc0,
+0x00,
+0x13,
+0x04,
+0xb4,
+0x20,
+0x34,
+0xa0,
+0x16,
+0x85,
+0xc0,
+0x97,
+0x14,
+0x02,
+0xf6,
+0xc0,
+0x00,
+0x13,
+0x20,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x04,
+0x02,
+0x41,
+0xb8,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xbd,
+0x34,
+0x72,
+0x49,
+0xb8,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xbd,
+0x30,
+0x7a,
+0x40,
+0x80,
+0x80,
+0x80,
+0xa5,
+0x16,
+0x1a,
+0x08,
+0x0e,
+0x19,
+0x40,
+0x74,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x0a,
+0x1a,
+0x40,
+0x80,
+0x84,
+0x80,
+0xa2,
+0x2c,
+0x34,
+0x40,
+0x00,
+0x01,
+0x40,
+0x74,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x70,
+0xbd,
+0x16,
+0x73,
+0x20,
+0xa0,
+0xbc,
+0x21,
+0x03,
+0x21,
+0xa0,
+0xbd,
+0x12,
+0x93,
+0x81,
+0xc6,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x0b,
+0x25,
+0x9b,
+0x8b,
+0xc0,
+0x69,
+0x38,
+0x11,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0xc8,
+0x02,
+0xc2,
+0xc0,
+0xbc,
+0x06,
+0x96,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x00,
+0x00,
+0x02,
+0x41,
+0xbe,
+0xa0,
+0x67,
+0x18,
+0x02,
+0x4a,
+0x68,
+0x00,
+0x08,
+0x14,
+0x22,
+0x84,
+0x06,
+0x26,
+0x82,
+0x00,
+0x22,
+0x02,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xcf,
+0x37,
+0x68,
+0x20,
+0x02,
+0x20,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xce,
+0x77,
+0x32,
+0x0a,
+0x0b,
+0xc2,
+0x00,
+0x32,
+0x1a,
+0x0b,
+0xce,
+0x39,
+0x38,
+0x1c,
+0x66,
+0xc0,
+0x00,
+0x03,
+0x60,
+0xb2,
+0x59,
+0xb8,
+0xbc,
+0x05,
+0x93,
+0x81,
+0x14,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x52,
+0xc8,
+0x02,
+0xc2,
+0xc0,
+0xbc,
+0x06,
+0x96,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x00,
+0x00,
+0x02,
+0x41,
+0xbe,
+0xa0,
+0x67,
+0x18,
+0x02,
+0x4a,
+0x68,
+0x00,
+0x08,
+0x01,
+0x22,
+0x84,
+0x06,
+0x26,
+0x82,
+0x00,
+0x21,
+0xf2,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xcc,
+0x57,
+0x68,
+0x20,
+0x02,
+0x1f,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xcb,
+0x97,
+0x32,
+0x0a,
+0x0b,
+0xcb,
+0x78,
+0x38,
+0x1c,
+0x76,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x82,
+0x59,
+0xe0,
+0xbc,
+0xb2,
+0x83,
+0x81,
+0x12,
+0x25,
+0x88,
+0x0b,
+0xc0,
+0x69,
+0x39,
+0x2c,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0x05,
+0x20,
+0x68,
+0x00,
+0x07,
+0xfb,
+0x21,
+0x52,
+0x0f,
+0x38,
+0x00,
+0x61,
+0x51,
+0xb1,
+0xb2,
+0xc2,
+0x70,
+0x80,
+0x24,
+0xb0,
+0x00,
+0x00,
+0x94,
+0x02,
+0xf5,
+0x19,
+0x1f,
+0x94,
+0x0a,
+0xc2,
+0x91,
+0x3f,
+0x29,
+0x1b,
+0xe6,
+0xc4,
+0x00,
+0x43,
+0xc4,
+0xab,
+0xc9,
+0x77,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0xa0,
+0x82,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x02,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x03,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc8,
+0x17,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc7,
+0x57,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0xa0,
+0x82,
+0x1a,
+0x00,
+0x30,
+0x88,
+0x0e,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x12,
+0x18,
+0x80,
+0xa0,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x05,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x04,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x90,
+0x40,
+0x00,
+0x03,
+0xc5,
+0x77,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc4,
+0xb7,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0xa0,
+0x82,
+0x1a,
+0x00,
+0x40,
+0x88,
+0x0e,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x12,
+0x18,
+0x80,
+0xa0,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x06,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x04,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x90,
+0x40,
+0x00,
+0x03,
+0xc2,
+0xd7,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc2,
+0x17,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0xa0,
+0x82,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x04,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x03,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xb7,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x0b,
+0x19,
+0x46,
+0x2c,
+0xb0,
+0x78,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x05,
+0x0b,
+0x07,
+0x8e,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xfb,
+0xc7,
+0x64,
+0x00,
+0x01,
+0xea,
+0xe7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xac,
+0xb0,
+0x78,
+0x62,
+0x89,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x60,
+0x38,
+0x08,
+0x63,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xf4,
+0xa7,
+0x64,
+0x00,
+0x01,
+0xeb,
+0xe7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x07,
+0x60,
+0xab,
+0xfe,
+0x09,
+0x40,
+0x2c,
+0x59,
+0x0f,
+0x00,
+0x80,
+0x76,
+0x42,
+0x36,
+0x42,
+0x04,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x23,
+0xa0,
+0xbc,
+0x17,
+0x03,
+0x23,
+0xe0,
+0xbc,
+0x7a,
+0x9a,
+0x40,
+0x40,
+0x68,
+0x00,
+0x00,
+0x08,
+0x21,
+0x40,
+0x00,
+0x01,
+0x88,
+0x09,
+0x66,
+0x00,
+0x01,
+0x2f,
+0xc8,
+0x5c,
+0x00,
+0x71,
+0x88,
+0x48,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x40,
+0x40,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc6,
+0x57,
+0x32,
+0x0b,
+0x0b,
+0xc3,
+0xe8,
+0x38,
+0x15,
+0x46,
+0x80,
+0x00,
+0x00,
+0x7a,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x8a,
+0x40,
+0x40,
+0x88,
+0x10,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x19,
+0x96,
+0x83,
+0x40,
+0x00,
+0xa2,
+0x06,
+0xc6,
+0x80,
+0x01,
+0x40,
+0x86,
+0x80,
+0x00,
+0x18,
+0x00,
+0xa5,
+0x48,
+0xd2,
+0x20,
+0x22,
+0x06,
+0xc6,
+0x80,
+0x01,
+0x44,
+0x85,
+0xc0,
+0xaa,
+0x2c,
+0x36,
+0x05,
+0xc0,
+0xb7,
+0x04,
+0x00,
+0x92,
+0x49,
+0x2d,
+0x80,
+0x24,
+0x90,
+0x00,
+0x00,
+0x84,
+0x00,
+0x92,
+0x49,
+0x2c,
+0x84,
+0x04,
+0x89,
+0x8e,
+0x48,
+0x52,
+0x4d,
+0x23,
+0xc4,
+0x3f,
+0x40,
+0x00,
+0x01,
+0x82,
+0x39,
+0x6c,
+0x68,
+0x00,
+0x14,
+0x08,
+0x68,
+0x3f,
+0xfe,
+0x7f,
+0xca,
+0x54,
+0x4d,
+0x22,
+0x02,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x14,
+0x48,
+0x5c,
+0x0a,
+0xa2,
+0xc3,
+0x60,
+0x84,
+0x00,
+0xa2,
+0x41,
+0x36,
+0x80,
+0x24,
+0xa0,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa2,
+0x41,
+0x34,
+0x40,
+0x00,
+0x00,
+0x40,
+0x48,
+0x66,
+0x00,
+0x01,
+0x7c,
+0xe0,
+0x66,
+0x00,
+0x01,
+0x7d,
+0xc0,
+0x5c,
+0x0b,
+0x71,
+0x8e,
+0x48,
+0x52,
+0x0d,
+0x23,
+0xc2,
+0x7f,
+0x40,
+0x00,
+0x01,
+0x82,
+0x39,
+0x6c,
+0x68,
+0x00,
+0x36,
+0x0a,
+0x25,
+0x93,
+0x0b,
+0xc0,
+0x38,
+0x5c,
+0x00,
+0x62,
+0x40,
+0x40,
+0x98,
+0xe8,
+0x88,
+0x81,
+0x48,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc1,
+0x37,
+0x32,
+0x0b,
+0x0b,
+0xc0,
+0x78,
+0xa4,
+0x04,
+0x06,
+0x80,
+0x00,
+0x00,
+0x7a,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xc0,
+0xbf,
+0x88,
+0x13,
+0x98,
+0x81,
+0x79,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x20,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0x70,
+0x68,
+0x20,
+0x02,
+0x21,
+0x2c,
+0x5c,
+0x07,
+0xe2,
+0x40,
+0x40,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0x22,
+0xc8,
+0x00,
+0x6c,
+0x68,
+0x20,
+0x02,
+0x23,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0x42,
+0xc8,
+0x00,
+0x6c,
+0x68,
+0x20,
+0x02,
+0x25,
+0x2c,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0x62,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x22,
+0x72,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x68,
+0x20,
+0x02,
+0x28,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0x92,
+0x18,
+0x00,
+0x61,
+0x68,
+0x20,
+0x02,
+0x2b,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0xc2,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0xa2,
+0xc6,
+0xe0,
+0x00,
+0x00,
+0xea,
+0xe5,
+0x52,
+0x18,
+0x04,
+0x06,
+0xc5,
+0x18,
+0x40,
+0x20,
+0x5e,
+0x05,
+0x80,
+0x98,
+0x18,
+0x42,
+0x89,
+0xc0,
+0x00,
+0x42,
+0x1c,
+0x60,
+0x40,
+0x20,
+0x98,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x32,
+0x06,
+0xe0,
+0x00,
+0x00,
+0xda,
+0xc3,
+0x20,
+0xa0,
+0xbc,
+0x22,
+0x03,
+0x21,
+0xa0,
+0xbc,
+0x2f,
+0x16,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x13,
+0x81,
+0xc4,
+0x84,
+0x80,
+0xa2,
+0x59,
+0x30,
+0xbc,
+0x05,
+0x9a,
+0x0e,
+0xc2,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x20,
+0x02,
+0x0e,
+0x23,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x9d,
+0x80,
+0x38,
+0x58,
+0x00,
+0x98,
+0xb0,
+0x23,
+0x00,
+0x80,
+0xbc,
+0x05,
+0x98,
+0x50,
+0x50,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0x09,
+0xa2,
+0x0e,
+0x71,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x48,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x09,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x00,
+0x42,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0xa8,
+0x09,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x09,
+0x00,
+0x00,
+0x00,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xac,
+0xb0,
+0x78,
+0x62,
+0x89,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x60,
+0x38,
+0x10,
+0x63,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xfd,
+0xc7,
+0x64,
+0x00,
+0x01,
+0xfc,
+0xc7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0xab,
+0xff,
+0x0a,
+0x06,
+0x81,
+0x88,
+0x06,
+0x18,
+0x80,
+0xf6,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x02,
+0x08,
+0x80,
+0xb6,
+0xa8,
+0x01,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x04,
+0x60,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x68,
+0x20,
+0x01,
+0x8e,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x00,
+0x42,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x6c,
+0x40,
+0x04,
+0x3c,
+0x00,
+0x23,
+0xe0,
+0x43,
+0x2a,
+0x20,
+0xbc,
+0x05,
+0x82,
+0x34,
+0x00,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x6c,
+0x00,
+0x00,
+0xde,
+0x50,
+0x6c,
+0x00,
+0x00,
+0xdc,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x08,
+0x40,
+0x02,
+0x24,
+0x81,
+0x08,
+0x40,
+0x50,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x3e,
+0x0a,
+0x32,
+0x83,
+0x06,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x05,
+0x1e,
+0x1a,
+0x3c,
+0x29,
+0x82,
+0x88,
+0x36,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x02,
+0x5b,
+0x44,
+0x0b,
+0x00,
+0xf8,
+0x37,
+0x04,
+0x52,
+0xe1,
+0x47,
+0x68,
+0x00,
+0x13,
+0x8d,
+0x09,
+0x62,
+0x00,
+0x00,
+0x00,
+0x27,
+0x50,
+0x4e,
+0x49,
+0x8e,
+0xb5,
+0x5b,
+0x4a,
+0x02,
+0xc0,
+0x20,
+0x2f,
+0x80,
+0x92,
+0x99,
+0x50,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x2b,
+0x98,
+0x34,
+0x93,
+0x61,
+0x45,
+0x46,
+0x08,
+0x8b,
+0x01,
+0x00,
+0x08,
+0x12,
+0x05,
+0x1a,
+0x1a,
+0x18,
+0x38,
+0x26,
+0x80,
+0x00,
+0x0c,
+0xd2,
+0x06,
+0xc0,
+0x00,
+0x19,
+0x60,
+0xa5,
+0x20,
+0x1b,
+0x00,
+0x04,
+0x28,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x19,
+0x64,
+0xa6,
+0x80,
+0x00,
+0x06,
+0xe2,
+0xc4,
+0x20,
+0x8f,
+0x80,
+0x04,
+0x88,
+0x40,
+0xec,
+0x68,
+0x00,
+0x00,
+0xcd,
+0x20,
+0x39,
+0x02,
+0x06,
+0xc4,
+0x00,
+0x44,
+0x00,
+0x85,
+0x94,
+0x10,
+0x00,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x0d,
+0xc7,
+0xa4,
+0x20,
+0x1c,
+0x80,
+0x07,
+0xa8,
+0x40,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xcb,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x04,
+0x40,
+0x0a,
+0x32,
+0x83,
+0x06,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x05,
+0x1e,
+0x1a,
+0x3c,
+0x29,
+0x82,
+0x88,
+0x36,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x02,
+0x5b,
+0x44,
+0x0b,
+0x00,
+0xf8,
+0x37,
+0x04,
+0x52,
+0xe1,
+0x47,
+0x68,
+0x00,
+0x13,
+0x8d,
+0x09,
+0x62,
+0x00,
+0x00,
+0x00,
+0x27,
+0x50,
+0x4e,
+0x49,
+0x8e,
+0xb5,
+0x5b,
+0x4a,
+0x02,
+0xc0,
+0x20,
+0x2f,
+0x80,
+0x92,
+0x99,
+0x50,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x2b,
+0x98,
+0x34,
+0x93,
+0x61,
+0x45,
+0x46,
+0x08,
+0x8b,
+0x01,
+0x00,
+0x08,
+0x12,
+0x05,
+0x1a,
+0x1a,
+0x18,
+0x38,
+0x26,
+0x80,
+0x00,
+0x0d,
+0x22,
+0x06,
+0xc0,
+0x00,
+0x19,
+0x60,
+0xa5,
+0x20,
+0x1b,
+0x00,
+0x04,
+0x28,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x19,
+0x64,
+0xa6,
+0x80,
+0x00,
+0x06,
+0xf2,
+0xc4,
+0x20,
+0x8f,
+0x80,
+0x04,
+0x88,
+0x40,
+0xec,
+0x68,
+0x00,
+0x00,
+0xd2,
+0x20,
+0x39,
+0x02,
+0x06,
+0xc4,
+0x00,
+0x43,
+0xe0,
+0x85,
+0x94,
+0x10,
+0x00,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x0d,
+0xe7,
+0xa4,
+0x20,
+0x1c,
+0x80,
+0x07,
+0xa8,
+0x40,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xcb,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x09,
+0x5d,
+0x0a,
+0x23,
+0x01,
+0xc6,
+0x52,
+0x0d,
+0x72,
+0xbf,
+0xe0,
+0x59,
+0x01,
+0x00,
+0x80,
+0xfa,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x4a,
+0x88,
+0x17,
+0x64,
+0x20,
+0x74,
+0x18,
+0xe8,
+0x95,
+0x50,
+0x16,
+0x08,
+0x04,
+0x89,
+0x8e,
+0x88,
+0x88,
+0x00,
+0x93,
+0x20,
+0x68,
+0xbc,
+0x07,
+0x8b,
+0x00,
+0x0d,
+0x88,
+0x00,
+0x95,
+0x90,
+0x54,
+0x30,
+0x00,
+0xdb,
+0xc0,
+0x29,
+0x98,
+0x24,
+0x89,
+0x8e,
+0x89,
+0x88,
+0x0c,
+0x93,
+0x20,
+0x68,
+0xbc,
+0x0c,
+0x98,
+0x81,
+0xc8,
+0x66,
+0x00,
+0x00,
+0xe8,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x46,
+0x20,
+0x5c,
+0x81,
+0x0a,
+0xc2,
+0x20,
+0x80,
+0x28,
+0x98,
+0x00,
+0x08,
+0x57,
+0x09,
+0x40,
+0x81,
+0x88,
+0x84,
+0x04,
+0x03,
+0x20,
+0x60,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xa1,
+0x66,
+0x00,
+0x00,
+0xe9,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x5a,
+0x20,
+0x5c,
+0x81,
+0x0a,
+0xc2,
+0x20,
+0x80,
+0x28,
+0x98,
+0x00,
+0x08,
+0x2e,
+0x12,
+0x88,
+0x40,
+0x40,
+0x98,
+0xe8,
+0x86,
+0x80,
+0x00,
+0x05,
+0x72,
+0x06,
+0x80,
+0x00,
+0x06,
+0xb2,
+0x16,
+0x60,
+0x00,
+0x04,
+0xa2,
+0x88,
+0x80,
+0x89,
+0xb0,
+0x00,
+0xc6,
+0x80,
+0x00,
+0x05,
+0x72,
+0x06,
+0x80,
+0x00,
+0x06,
+0xb2,
+0x16,
+0x60,
+0x00,
+0x04,
+0xa2,
+0x88,
+0x81,
+0x89,
+0x88,
+0x00,
+0x93,
+0x20,
+0x28,
+0xbc,
+0x11,
+0x16,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x03,
+0x81,
+0xc5,
+0x84,
+0x00,
+0x82,
+0x49,
+0x64,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x0a,
+0x52,
+0x4b,
+0xa8,
+0x40,
+0x48,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x49,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x42,
+0x03,
+0x7a,
+0x06,
+0xc0,
+0x40,
+0x00,
+0x00,
+0x40,
+0x6c,
+0x68,
+0x00,
+0x08,
+0x40,
+0x20,
+0x6c,
+0x00,
+0x00,
+0x0a,
+0x60,
+0x00,
+0x00,
+0x08,
+0x81,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x20,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0x56,
+0x20,
+0x68,
+0x20,
+0x00,
+0x82,
+0x22,
+0x84,
+0x00,
+0x88,
+0x50,
+0x0a,
+0x58,
+0x0d,
+0x02,
+0xbf,
+0xf0,
+0x68,
+0x20,
+0x00,
+0x81,
+0x21,
+0xbc,
+0x03,
+0x88,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x68,
+0x00,
+0x00,
+0x6a,
+0x20,
+0x6c,
+0x40,
+0x01,
+0x04,
+0x08,
+0x84,
+0x00,
+0xa3,
+0x01,
+0x30,
+0x68,
+0x20,
+0x00,
+0x82,
+0x22,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x48,
+0x68,
+0x20,
+0x00,
+0x81,
+0x21,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x6c,
+0x00,
+0x00,
+0xac,
+0x08,
+0x6c,
+0x40,
+0x01,
+0x04,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x41,
+0x6c,
+0x00,
+0x00,
+0xd4,
+0x08,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x68,
+0x00,
+0x08,
+0x40,
+0x20,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x1f,
+0x6c,
+0x00,
+0x00,
+0x0a,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x66,
+0xc4,
+0x00,
+0x44,
+0x20,
+0x05,
+0x24,
+0x82,
+0x04,
+0x04,
+0xaa,
+0x06,
+0xc0,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x48,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xf2,
+0x59,
+0x60,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x2d,
+0x52,
+0x4b,
+0xc3,
+0xc0,
+0x6f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x52,
+0x0b,
+0xc3,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x52,
+0x0d,
+0x22,
+0xbf,
+0xf0,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x20,
+0x25,
+0x96,
+0x06,
+0xc4,
+0x00,
+0x44,
+0x64,
+0x8b,
+0xc1,
+0x28,
+0x5c,
+0x08,
+0xa9,
+0x40,
+0x2f,
+0x52,
+0x4b,
+0xc3,
+0x01,
+0x2c,
+0x5c,
+0x81,
+0x01,
+0x40,
+0xad,
+0x25,
+0x92,
+0x8b,
+0xc3,
+0xb9,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x99,
+0x21,
+0x24,
+0x92,
+0x88,
+0x0a,
+0x7a,
+0x84,
+0x87,
+0xaa,
+0x09,
+0x89,
+0x94,
+0x86,
+0x0b,
+0xc3,
+0x17,
+0x25,
+0x97,
+0x8b,
+0xc1,
+0x60,
+0x38,
+0x10,
+0x82,
+0x58,
+0x20,
+0xbc,
+0x04,
+0x06,
+0xc0,
+0x00,
+0x0e,
+0xa7,
+0xa6,
+0xc0,
+0x00,
+0x0f,
+0xe7,
+0xa2,
+0x59,
+0x60,
+0xbc,
+0x08,
+0x16,
+0xc4,
+0x00,
+0x25,
+0x60,
+0x82,
+0x31,
+0x24,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x48,
+0x23,
+0x16,
+0x46,
+0xc0,
+0x00,
+0x14,
+0xc4,
+0x82,
+0x41,
+0x78,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x7a,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x43,
+0xe0,
+0x86,
+0xc4,
+0x00,
+0x44,
+0x00,
+0x92,
+0x91,
+0x64,
+0x32,
+0x82,
+0x0b,
+0xc0,
+0xc0,
+0x40,
+0x00,
+0x00,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x60,
+0x40,
+0x5c,
+0x0e,
+0x30,
+0x80,
+0x36,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x20,
+0x6c,
+0x40,
+0x04,
+0x3e,
+0x7a,
+0x6c,
+0x40,
+0x04,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x6e,
+0x21,
+0x00,
+0x00,
+0x08,
+0x48,
+0xfa,
+0x84,
+0x87,
+0xa3,
+0x81,
+0x1d,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x0b,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x68,
+0x5c,
+0x09,
+0xa1,
+0x40,
+0xad,
+0x52,
+0x49,
+0x43,
+0xc0,
+0xcf,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0x6c,
+0x40,
+0x02,
+0xc2,
+0x00,
+0x52,
+0x09,
+0x53,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x01,
+0x50,
+0x50,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x50,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe2,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9b,
+0xc8,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xca,
+0x80,
+0x10,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x00,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xa2,
+0x58,
+0x20,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x20,
+0x52,
+0x00,
+0x83,
+0xc0,
+0x5f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x24,
+0x81,
+0x06,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x03,
+0xa1,
+0x00,
+0x22,
+0x84,
+0x56,
+0x82,
+0x00,
+0x0d,
+0x72,
+0x02,
+0xa8,
+0xe8,
+0x51,
+0x84,
+0x00,
+0x40,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x21,
+0x52,
+0x4d,
+0x21,
+0x84,
+0x28,
+0xa0,
+0x02,
+0x08,
+0x48,
+0x00,
+0x52,
+0x4c,
+0x01,
+0xc0,
+0x00,
+0x84,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x44,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x19,
+0x84,
+0xa6,
+0xc4,
+0x00,
+0x19,
+0xa4,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0xa4,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0x44,
+0xa6,
+0x80,
+0x00,
+0x6e,
+0x6a,
+0xca,
+0x0e,
+0xc0,
+0x46,
+0x0a,
+0x40,
+0x48,
+0x50,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x08,
+0x5c,
+0x0e,
+0x32,
+0xc0,
+0x20,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x20,
+0x60,
+0x00,
+0x00,
+0x00,
+0x2c,
+0x52,
+0x0d,
+0x22,
+0xbf,
+0xf0,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x48,
+0x80,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x0e,
+0x82,
+0x06,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0xa5,
+0xc8,
+0x30,
+0x08,
+0x07,
+0x66,
+0x82,
+0x00,
+0x0e,
+0x22,
+0x18,
+0x00,
+0x4a,
+0x80,
+0x84,
+0xa8,
+0x40,
+0x4a,
+0x84,
+0x84,
+0xa6,
+0x80,
+0x00,
+0x0a,
+0xc2,
+0x06,
+0x60,
+0x00,
+0x04,
+0xe0,
+0x83,
+0x82,
+0xc4,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0x08,
+0x5c,
+0x08,
+0x23,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0xac,
+0x20,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x08,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x40,
+0x6c,
+0x40,
+0x01,
+0xf8,
+0x0a,
+0x5c,
+0x83,
+0x08,
+0x40,
+0xc8,
+0x80,
+0x04,
+0xa6,
+0x80,
+0x00,
+0x09,
+0x62,
+0x1a,
+0x01,
+0x20,
+0x80,
+0x04,
+0xa8,
+0x02,
+0xc8,
+0xa0,
+0x82,
+0x28,
+0x40,
+0x61,
+0x40,
+0x00,
+0x00,
+0x45,
+0x62,
+0x68,
+0x00,
+0x00,
+0xc2,
+0x20,
+0x68,
+0x00,
+0x00,
+0xc6,
+0x21,
+0x66,
+0x00,
+0x00,
+0x45,
+0x00,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0x28,
+0x5c,
+0x00,
+0x63,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x20,
+0x5c,
+0x82,
+0x00,
+0x80,
+0x36,
+0x68,
+0x00,
+0x08,
+0x97,
+0x2c,
+0x68,
+0x00,
+0x01,
+0x00,
+0x08,
+0xa0,
+0x0e,
+0x06,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0xc4,
+0x60,
+0xa4,
+0x00,
+0x24,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x68,
+0x00,
+0x01,
+0x32,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x84,
+0x10,
+0xa2,
+0x91,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc3,
+0x31,
+0x68,
+0x00,
+0x00,
+0xb2,
+0x20,
+0x68,
+0x00,
+0x00,
+0x9e,
+0x21,
+0x8c,
+0x03,
+0x25,
+0x15,
+0x49,
+0x20,
+0x14,
+0x2a,
+0x15,
+0x80,
+0x6c,
+0x40,
+0x02,
+0x12,
+0x08,
+0x84,
+0x00,
+0xa5,
+0x84,
+0x98,
+0x0d,
+0x03,
+0x05,
+0x15,
+0x40,
+0x04,
+0x84,
+0xeb,
+0xc0,
+0x8d,
+0x84,
+0x8c,
+0xc3,
+0x81,
+0xce,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x09,
+0x24,
+0x1a,
+0xe6,
+0xc4,
+0x00,
+0x44,
+0xa4,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x45,
+0x0a,
+0x30,
+0x93,
+0x0b,
+0xc0,
+0x65,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x08,
+0x38,
+0x1d,
+0x62,
+0x41,
+0xa4,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x48,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x66,
+0xc4,
+0x00,
+0x44,
+0xa0,
+0x95,
+0x24,
+0x96,
+0x04,
+0x04,
+0xa6,
+0xc4,
+0x00,
+0x44,
+0xa4,
+0x86,
+0x80,
+0x00,
+0x6e,
+0x6a,
+0xc4,
+0x60,
+0xa4,
+0x20,
+0x6c,
+0x08,
+0x40,
+0x6c,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x6c,
+0x40,
+0x04,
+0x52,
+0x08,
+0x68,
+0x00,
+0x47,
+0xff,
+0xca,
+0x68,
+0x38,
+0x1c,
+0x04,
+0x20,
+0x54,
+0x4d,
+0x23,
+0x00,
+0x08,
+0x5c,
+0x00,
+0x20,
+0x40,
+0x48,
+0x5c,
+0x09,
+0xf2,
+0x00,
+0x20,
+0x84,
+0x07,
+0xa8,
+0x40,
+0xd0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x38,
+0x10,
+0x00,
+0x25,
+0x98,
+0x0b,
+0xc0,
+0x50,
+0x6c,
+0x70,
+0x38,
+0x00,
+0x08,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x45,
+0x24,
+0x83,
+0x22,
+0xa0,
+0xbf,
+0xf5,
+0xa2,
+0xa0,
+0x64,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x07,
+0x60,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xac,
+0x5c,
+0x09,
+0xf1,
+0x40,
+0x2d,
+0x52,
+0x0d,
+0x03,
+0x00,
+0xec,
+0x30,
+0x12,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x8e,
+0x0b,
+0xc0,
+0x89,
+0xa0,
+0x44,
+0x86,
+0xc4,
+0x00,
+0x45,
+0x80,
+0x83,
+0x81,
+0x05,
+0x52,
+0x0b,
+0x23,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x48,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x21,
+0x39,
+0x7e,
+0x08,
+0x48,
+0x08,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x09,
+0x68,
+0x00,
+0x08,
+0xb9,
+0xac,
+0x6c,
+0x00,
+0x03,
+0x8a,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xa8,
+0x49,
+0xa0,
+0xc4,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x06,
+0xc8,
+0x48,
+0x4a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x00,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x20,
+0x5c,
+0x08,
+0x0b,
+0x00,
+0x0a,
+0x68,
+0x38,
+0x08,
+0x00,
+0x21,
+0x52,
+0xc2,
+0x00,
+0x40,
+0x52,
+0x5c,
+0xbf,
+0x03,
+0xc1,
+0x28,
+0x51,
+0xe0,
+0x21,
+0xc0,
+0x00,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xc0,
+0x54,
+0x41,
+0x03,
+0x04,
+0x01,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x51,
+0x51,
+0xd0,
+0x23,
+0x07,
+0xfb,
+0x54,
+0x47,
+0x0a,
+0x08,
+0x41,
+0x54,
+0x46,
+0x00,
+0x48,
+0x7a,
+0x42,
+0x07,
+0xf8,
+0x48,
+0xd1,
+0x40,
+0x00,
+0x00,
+0x48,
+0xd0,
+0x68,
+0x00,
+0x3f,
+0xc0,
+0x00,
+0x54,
+0x41,
+0x03,
+0x01,
+0x79,
+0x52,
+0x02,
+0x0b,
+0x07,
+0xf8,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x7a,
+0x51,
+0xd0,
+0x62,
+0x08,
+0x41,
+0x54,
+0x41,
+0x00,
+0x48,
+0x7a,
+0x84,
+0x8d,
+0x03,
+0x80,
+0x10,
+0x68,
+0x00,
+0x08,
+0xc7,
+0x2c,
+0x84,
+0x05,
+0x06,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0xc4,
+0x60,
+0xa4,
+0x20,
+0x50,
+0x08,
+0x40,
+0x52,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x08,
+0x5c,
+0x00,
+0x43,
+0x01,
+0x06,
+0x6c,
+0x70,
+0x10,
+0x0e,
+0x50,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x61,
+0x6c,
+0x70,
+0x10,
+0x02,
+0x08,
+0x51,
+0xb1,
+0x23,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x48,
+0x68,
+0x00,
+0x08,
+0xcc,
+0x20,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x8a,
+0x08,
+0x5c,
+0xbf,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x21,
+0x5c,
+0x0e,
+0x21,
+0xc0,
+0x00,
+0x5c,
+0x01,
+0xe8,
+0x48,
+0x0b,
+0x6c,
+0x00,
+0x03,
+0xa8,
+0x0a,
+0x52,
+0x49,
+0xe0,
+0x40,
+0x4a,
+0x5c,
+0x09,
+0xf2,
+0x04,
+0x40,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xaf,
+0x52,
+0x4d,
+0xc0,
+0x48,
+0x48,
+0xa0,
+0xec,
+0x18,
+0x40,
+0x49,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xa0,
+0x58,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x86,
+0xc8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x08,
+0x38,
+0x1c,
+0x66,
+0xc6,
+0x80,
+0x03,
+0xc0,
+0x02,
+0x41,
+0xa4,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x48,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0xa0,
+0x68,
+0x34,
+0x00,
+0x00,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x08,
+0xa0,
+0x3c,
+0x08,
+0x40,
+0x00,
+0x6c,
+0x00,
+0x03,
+0xb6,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xb8,
+0x50,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x45,
+0x40,
+0x82,
+0xa8,
+0xe0,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0xe1,
+0x68,
+0x34,
+0x00,
+0x00,
+0x20,
+0x6c,
+0x00,
+0x03,
+0xb6,
+0x00,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x50,
+0xa0,
+0x3c,
+0x06,
+0xc0,
+0x00,
+0x3b,
+0x80,
+0x28,
+0x40,
+0x52,
+0x42,
+0x0f,
+0x7b,
+0x0f,
+0xf0,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x50,
+0x6c,
+0x68,
+0x00,
+0x3c,
+0x7a,
+0x68,
+0x00,
+0x04,
+0xd9,
+0x00,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x02,
+0x54,
+0x40,
+0x83,
+0x01,
+0x0a,
+0x25,
+0x8a,
+0x06,
+0xc6,
+0x80,
+0x00,
+0x05,
+0x0b,
+0xc0,
+0x48,
+0xb0,
+0xff,
+0x2b,
+0xc0,
+0xef,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x52,
+0x5c,
+0x08,
+0x03,
+0x1f,
+0xf2,
+0x25,
+0x82,
+0x0b,
+0xc0,
+0x89,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x52,
+0x38,
+0x11,
+0x06,
+0xc6,
+0x80,
+0x03,
+0xa0,
+0x22,
+0x40,
+0x10,
+0x6c,
+0x68,
+0x00,
+0x3a,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x00,
+0x24,
+0x98,
+0x68,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xf2,
+0x59,
+0x60,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x3d,
+0x52,
+0x4b,
+0xc3,
+0xc0,
+0x6f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x52,
+0x0b,
+0xc3,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0xca,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc5,
+0xc0,
+0x5c,
+0x08,
+0x12,
+0xc0,
+0x21,
+0x25,
+0x88,
+0x06,
+0x80,
+0x00,
+0x1d,
+0xd2,
+0x0b,
+0xc0,
+0x88,
+0x5c,
+0x85,
+0x00,
+0x00,
+0xa1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x42,
+0x07,
+0x78,
+0x48,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xcc,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe1,
+0x22,
+0x00,
+0x00,
+0x08,
+0x10,
+0x23,
+0x00,
+0x00,
+0x08,
+0x58,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0x74,
+0x00,
+0x00,
+0x08,
+0x48,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0xf4,
+0x5c,
+0x08,
+0x52,
+0x00,
+0x01,
+0x25,
+0x88,
+0x0b,
+0xc0,
+0x68,
+0x80,
+0x0a,
+0x20,
+0x00,
+0x00,
+0x42,
+0x07,
+0x78,
+0x50,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xce,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe2,
+0x23,
+0x00,
+0x00,
+0x08,
+0x18,
+0x24,
+0x00,
+0x00,
+0x08,
+0x60,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x58,
+0x74,
+0x00,
+0x00,
+0x08,
+0x50,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x58,
+0xf4,
+0x38,
+0x11,
+0x22,
+0x58,
+0x80,
+0xbc,
+0x07,
+0x88,
+0x48,
+0xa1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x42,
+0x07,
+0x78,
+0x48,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xd0,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe3,
+0x22,
+0x00,
+0x00,
+0x08,
+0x10,
+0x23,
+0x00,
+0x00,
+0x08,
+0x58,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0x74,
+0x00,
+0x00,
+0x08,
+0x48,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0xf4,
+0x38,
+0x11,
+0xa2,
+0x58,
+0x80,
+0xbc,
+0x07,
+0x88,
+0x40,
+0xa0,
+0x00,
+0x00,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x3d,
+0x25,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0xe4,
+0x21,
+0x00,
+0x00,
+0x08,
+0x08,
+0x22,
+0x00,
+0x00,
+0x08,
+0x50,
+0x00,
+0x22,
+0xc0,
+0x49,
+0x48,
+0x74,
+0x00,
+0x00,
+0x08,
+0x40,
+0x00,
+0x22,
+0xc0,
+0x49,
+0x48,
+0xf4,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x37,
+0x08,
+0x63,
+0x70,
+0x44,
+0x2e,
+0x13,
+0x42,
+0xa0,
+0x64,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x55,
+0x38,
+0x20,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x0e,
+0x06,
+0x20,
+0x00,
+0x00,
+0x01,
+0x45,
+0x00,
+0x88,
+0x18,
+0xeb,
+0x50,
+0x00,
+0x00,
+0x2f,
+0x80,
+0x9b,
+0xc0,
+0x13,
+0x28,
+0x00,
+0x92,
+0x09,
+0x08,
+0x57,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x20,
+0x10,
+0x95,
+0x0c,
+0x84,
+0x19,
+0x20,
+0x3b,
+0xa1,
+0x48,
+0x5b,
+0xc2,
+0x01,
+0x8e,
+0x83,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x48,
+0x55,
+0x00,
+0x59,
+0x8e,
+0x80,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x0c,
+0xe2,
+0xbf,
+0xf0,
+0x50,
+0xc8,
+0x41,
+0x00,
+0x59,
+0x6a,
+0x00,
+0x03,
+0xff,
+0xc1,
+0x54,
+0x42,
+0x0b,
+0x00,
+0x0d,
+0x6a,
+0x00,
+0x03,
+0xff,
+0x80,
+0x30,
+0x80,
+0x8b,
+0xc0,
+0x79,
+0x2a,
+0x02,
+0xe6,
+0xe0,
+0x00,
+0x3d,
+0x81,
+0x19,
+0x00,
+0x13,
+0x28,
+0x85,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x01,
+0x19,
+0x8e,
+0x89,
+0x51,
+0x82,
+0x89,
+0x82,
+0x8b,
+0x6e,
+0x00,
+0x03,
+0xd4,
+0x13,
+0x30,
+0x8c,
+0x8b,
+0xc0,
+0x3c,
+0x6a,
+0x00,
+0x03,
+0xff,
+0xc1,
+0x98,
+0xe8,
+0xb2,
+0x19,
+0x13,
+0x28,
+0x85,
+0x93,
+0x08,
+0x08,
+0xbc,
+0x05,
+0x16,
+0xe0,
+0x00,
+0x3d,
+0x81,
+0x02,
+0x88,
+0x10,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x11,
+0x98,
+0xe8,
+0xa3,
+0x20,
+0x30,
+0x52,
+0x08,
+0x81,
+0x00,
+0x12,
+0xbc,
+0x08,
+0x92,
+0x41,
+0x12,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x41,
+0x32,
+0x03,
+0x8b,
+0xc0,
+0x31,
+0x2a,
+0x01,
+0x0b,
+0xc0,
+0x17,
+0x2a,
+0x01,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x5d,
+0x4a,
+0x33,
+0x01,
+0x03,
+0x68,
+0x00,
+0x01,
+0xff,
+0x4b,
+0x58,
+0x0f,
+0x83,
+0x5f,
+0xf8,
+0x50,
+0x47,
+0x19,
+0x84,
+0xc8,
+0x54,
+0x40,
+0x53,
+0x01,
+0xfe,
+0xbc,
+0x2f,
+0xa5,
+0x08,
+0xcd,
+0x18,
+0x38,
+0x33,
+0x01,
+0xe8,
+0xbc,
+0x0b,
+0x46,
+0x80,
+0x00,
+0x1f,
+0xf4,
+0xb3,
+0x01,
+0xe8,
+0xbc,
+0x0c,
+0x15,
+0x50,
+0x0f,
+0xb5,
+0x00,
+0x32,
+0x80,
+0xcb,
+0x37,
+0x8c,
+0x33,
+0x20,
+0x18,
+0xbc,
+0x06,
+0xb2,
+0xa0,
+0x3b,
+0x6e,
+0x00,
+0x03,
+0xdc,
+0x11,
+0xba,
+0x14,
+0x82,
+0x80,
+0x51,
+0x37,
+0x84,
+0x03,
+0x20,
+0x28,
+0xbc,
+0x19,
+0x33,
+0x61,
+0x45,
+0x32,
+0x02,
+0x8b,
+0xc1,
+0x20,
+0x38,
+0x20,
+0x73,
+0x01,
+0xe8,
+0xbc,
+0x05,
+0x23,
+0x28,
+0x08,
+0xbc,
+0x0d,
+0x93,
+0x88,
+0x09,
+0x38,
+0x00,
+0x1b,
+0xc0,
+0xa7,
+0x36,
+0x14,
+0x72,
+0x19,
+0x4b,
+0x28,
+0x9b,
+0xd2,
+0x11,
+0x49,
+0x32,
+0x80,
+0x8b,
+0xc0,
+0x29,
+0x38,
+0x80,
+0x93,
+0x80,
+0x01,
+0x54,
+0x82,
+0xcb,
+0x80,
+0x00,
+0x54,
+0x40,
+0x41,
+0x8e,
+0x89,
+0x40,
+0x00,
+0x01,
+0x83,
+0x03,
+0x5c,
+0x0a,
+0x73,
+0x50,
+0x00,
+0x25,
+0x19,
+0xe2,
+0x80,
+0x09,
+0x32,
+0x03,
+0x05,
+0xbc,
+0x20,
+0xbc,
+0x03,
+0x85,
+0x1d,
+0x44,
+0xb0,
+0x00,
+0xf9,
+0x8e,
+0x8b,
+0x55,
+0x43,
+0xf3,
+0x3f,
+0xff,
+0x29,
+0x9f,
+0x62,
+0x09,
+0x30,
+0x28,
+0x80,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x01,
+0x19,
+0x8e,
+0x89,
+0x50,
+0x49,
+0x43,
+0x01,
+0xa5,
+0x21,
+0x14,
+0x02,
+0x80,
+0x10,
+0x5b,
+0xc0,
+0x03,
+0xa1,
+0x48,
+0x28,
+0x04,
+0x13,
+0x78,
+0x40,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x2a,
+0x38,
+0x00,
+0xd9,
+0x8e,
+0x89,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x20,
+0x5b,
+0x08,
+0x23,
+0x80,
+0x00,
+0x5b,
+0x88,
+0x33,
+0x00,
+0xb7,
+0x54,
+0x0f,
+0x83,
+0x01,
+0x06,
+0x68,
+0x00,
+0x01,
+0x0c,
+0x8b,
+0x57,
+0x01,
+0xcb,
+0x01,
+0xff,
+0x50,
+0x4d,
+0x53,
+0x01,
+0xa5,
+0x20,
+0x98,
+0x92,
+0x11,
+0xd2,
+0x21,
+0x14,
+0x92,
+0x19,
+0xa3,
+0x28,
+0x05,
+0x22,
+0x10,
+0x18,
+0x5b,
+0xc4,
+0x13,
+0xa1,
+0x48,
+0x28,
+0x01,
+0x03,
+0x78,
+0x00,
+0xba,
+0x14,
+0x83,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x0d,
+0x23,
+0x01,
+0xfe,
+0x21,
+0x90,
+0x26,
+0xa0,
+0x00,
+0x1f,
+0xfc,
+0x12,
+0x88,
+0x51,
+0x50,
+0xcc,
+0x11,
+0x83,
+0x4a,
+0x68,
+0x00,
+0x01,
+0x07,
+0x89,
+0x30,
+0x17,
+0x06,
+0xe0,
+0x00,
+0x3e,
+0x41,
+0x1b,
+0xc0,
+0x8c,
+0x54,
+0x42,
+0x01,
+0x83,
+0x89,
+0xb1,
+0xff,
+0xf3,
+0x01,
+0xf0,
+0xbc,
+0x0b,
+0x3b,
+0xa1,
+0x48,
+0x98,
+0xe8,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0xff,
+0xc8,
+0x30,
+0x13,
+0x0b,
+0xc1,
+0x41,
+0x32,
+0x80,
+0x0b,
+0xc1,
+0x20,
+0x98,
+0xe8,
+0x9b,
+0xc1,
+0x07,
+0x68,
+0x00,
+0x01,
+0x0c,
+0xcb,
+0x2e,
+0x1b,
+0xe2,
+0x41,
+0x00,
+0x32,
+0x02,
+0x85,
+0x0c,
+0xc0,
+0x3c,
+0x02,
+0x89,
+0x83,
+0x08,
+0x36,
+0x10,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x02,
+0xa3,
+0x80,
+0x0e,
+0x98,
+0xe8,
+0xa2,
+0x99,
+0x76,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x60,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x49,
+0x6c,
+0x00,
+0x03,
+0xe0,
+0x08,
+0x6c,
+0x00,
+0x03,
+0xe2,
+0x08,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x0f,
+0xe3,
+0x01,
+0xa5,
+0x50,
+0xc8,
+0x42,
+0xbf,
+0xd0,
+0x50,
+0xc8,
+0x81,
+0x83,
+0x0a,
+0x50,
+0xca,
+0x59,
+0x83,
+0x0b,
+0x6a,
+0x00,
+0x01,
+0xff,
+0xc0,
+0x54,
+0x40,
+0xd9,
+0x00,
+0x5a,
+0x54,
+0xcf,
+0xb3,
+0x01,
+0x07,
+0x50,
+0xca,
+0x99,
+0x83,
+0xc9,
+0x54,
+0x40,
+0xc0,
+0x81,
+0x4a,
+0x50,
+0x4f,
+0x81,
+0x83,
+0x0a,
+0x21,
+0x10,
+0x06,
+0xe0,
+0x00,
+0x3f,
+0x01,
+0x35,
+0x40,
+0x61,
+0x90,
+0x25,
+0x86,
+0x80,
+0x00,
+0x1f,
+0xfc,
+0x86,
+0xe0,
+0x00,
+0x3e,
+0x81,
+0x03,
+0x01,
+0x28,
+0x28,
+0x81,
+0x25,
+0xbc,
+0x61,
+0xbc,
+0x76,
+0x82,
+0x88,
+0x08,
+0x30,
+0x13,
+0x0b,
+0xc1,
+0x41,
+0x32,
+0x81,
+0x0b,
+0xc0,
+0x91,
+0x20,
+0x9e,
+0x92,
+0x90,
+0x09,
+0x32,
+0x80,
+0x8b,
+0xc0,
+0xa1,
+0xba,
+0x14,
+0x86,
+0xe0,
+0x00,
+0x3e,
+0xc1,
+0x04,
+0x60,
+0x80,
+0xa8,
+0x03,
+0x09,
+0x00,
+0x12,
+0x64,
+0x00,
+0x02,
+0x44,
+0x8f,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0xba,
+0x14,
+0x8b,
+0xa1,
+0x01,
+0x55,
+0x00,
+0xc2,
+0x80,
+0x30,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0xb1,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x41,
+0xba,
+0x14,
+0x89,
+0x02,
+0x10,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0x5b,
+0x80,
+0x2b,
+0x00,
+0x09,
+0x2a,
+0x5a,
+0xd2,
+0x11,
+0x40,
+0x2e,
+0x14,
+0xd3,
+0x20,
+0x30,
+0xbc,
+0x0d,
+0x13,
+0x28,
+0x10,
+0xbc,
+0x05,
+0x1b,
+0xa1,
+0x48,
+0x40,
+0x00,
+0x01,
+0x02,
+0x10,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0x5b,
+0x84,
+0x23,
+0x00,
+0x09,
+0x2a,
+0x5a,
+0x42,
+0x11,
+0x12,
+0x57,
+0x08,
+0x73,
+0x80,
+0x00,
+0x5c,
+0x0d,
+0x0b,
+0xa1,
+0x11,
+0x52,
+0x02,
+0x99,
+0x82,
+0x42,
+0x52,
+0x02,
+0x09,
+0x82,
+0x80,
+0x23,
+0x2d,
+0xb5,
+0x0c,
+0xed,
+0x98,
+0x3c,
+0xa5,
+0x19,
+0x44,
+0x98,
+0x3c,
+0x95,
+0x40,
+0x08,
+0x18,
+0x2c,
+0x35,
+0x0c,
+0x64,
+0x98,
+0x34,
+0xb4,
+0x46,
+0x89,
+0x18,
+0x34,
+0x84,
+0x41,
+0x11,
+0x19,
+0x24,
+0x14,
+0x47,
+0x19,
+0x18,
+0x4c,
+0xa5,
+0xbc,
+0x41,
+0x18,
+0x40,
+0xb5,
+0x40,
+0x44,
+0x19,
+0x2c,
+0x14,
+0x40,
+0x99,
+0x32,
+0x00,
+0xd5,
+0xbc,
+0x00,
+0x10,
+0x05,
+0xb3,
+0x08,
+0x80,
+0xbc,
+0x03,
+0xa5,
+0x40,
+0xbe,
+0xb4,
+0x00,
+0xa3,
+0x80,
+0x02,
+0x50,
+0x8c,
+0x1b,
+0x00,
+0x0c,
+0x28,
+0x0c,
+0x92,
+0x11,
+0x92,
+0x21,
+0x98,
+0x03,
+0x78,
+0x41,
+0x28,
+0x01,
+0x25,
+0x84,
+0x64,
+0x10,
+0x01,
+0x33,
+0x78,
+0x82,
+0x54,
+0x04,
+0xd3,
+0xc0,
+0x3a,
+0x5b,
+0xc4,
+0x11,
+0x82,
+0x00,
+0x98,
+0xe8,
+0x82,
+0x09,
+0xa3,
+0x32,
+0x80,
+0x85,
+0x40,
+0x69,
+0x3c,
+0x02,
+0x93,
+0x78,
+0x81,
+0x98,
+0xe8,
+0x02,
+0x09,
+0x82,
+0x29,
+0x08,
+0x92,
+0x30,
+0x4a,
+0x32,
+0x01,
+0x0b,
+0xc0,
+0x4a,
+0x46,
+0x08,
+0x08,
+0x81,
+0x08,
+0x55,
+0x3f,
+0x69,
+0x92,
+0x81,
+0x64,
+0x00,
+0x02,
+0x47,
+0xef,
+0xa8,
+0x03,
+0x03,
+0x28,
+0x00,
+0xbc,
+0x10,
+0x16,
+0x80,
+0x00,
+0x1f,
+0xfc,
+0x93,
+0x01,
+0x70,
+0xbc,
+0x02,
+0x13,
+0x28,
+0x10,
+0xbc,
+0x0a,
+0x12,
+0x09,
+0xf1,
+0x29,
+0x08,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x0b,
+0x95,
+0x50,
+0x0c,
+0x3a,
+0x10,
+0x1b,
+0xa1,
+0x48,
+0x6e,
+0x00,
+0x03,
+0xec,
+0x10,
+0xa8,
+0x03,
+0x09,
+0x00,
+0x12,
+0x64,
+0x00,
+0x02,
+0x44,
+0x8f,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x66,
+0x66,
+0x66,
+0x66,
+0x00,
+0x3f,
+0xe6,
+0x66,
+0x66,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0xe0,
+0x00,
+0x00,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x07,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf0,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf0,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x01,
+0x13,
+0x92,
+0x73,
+0x00,
+0x7f,
+0xf3,
+0xa9,
+0x53,
+0x00,
+0x01,
+0x13,
+0x92,
+0x73,
+0x00,
+0xdc,
+0x56,
+0xfb,
+0x6f,
+0x00,
+0x7f,
+0xb8,
+0x17,
+0x23,
+0x00,
+0x23,
+0xf0,
+0xed,
+0x6d,
+0x00,
+0x90,
+0x26,
+0x15,
+0xe9,
+0x00,
+0x60,
+0xf0,
+0x19,
+0x01,
+0x00,
+0x71,
+0x16,
+0x2e,
+0xe9,
+0x00,
+0x81,
+0x47,
+0x58,
+0x21,
+0x00,
+0x7e,
+0xa4,
+0x32,
+0x5b,
+0x00,
+0x7f,
+0xeb,
+0x8a,
+0x7d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x65,
+0xac,
+0x8c,
+0x2f,
+0x00,
+0x5a,
+0x9d,
+0xf7,
+0xab,
+0x00,
+0x32,
+0xf5,
+0x2c,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x0f,
+0xd7,
+0x21,
+0x8f,
+0x00,
+0x60,
+0x51,
+0xbc,
+0xe3,
+0x00,
+0x0f,
+0xd7,
+0x21,
+0x8f,
+0x00,
+0x8a,
+0x4a,
+0x1d,
+0x99,
+0x00,
+0x75,
+0xb5,
+0xe2,
+0x67,
+0x00,
+0x3b,
+0x62,
+0x92,
+0x93,
+0x00,
+0xc8,
+0x64,
+0x0c,
+0xc5,
+0x00,
+0x3c,
+0x39,
+0x60,
+0xa7,
+0x00,
+0x94,
+0x28,
+0xff,
+0x89,
+0x00,
+0x6b,
+0xd7,
+0x00,
+0x77,
+0x00,
+0x36,
+0xc4,
+0x2d,
+0xd3,
+0x00,
+0xd1,
+0x5f,
+0x0d,
+0x4f,
+0x00,
+0x37,
+0xdc,
+0xc4,
+0xde,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x01,
+0x13,
+0x92,
+0x73,
+0x00,
+0x7f,
+0xf3,
+0xa9,
+0x53,
+0x00,
+0x01,
+0x13,
+0x92,
+0x73,
+0x00,
+0xdc,
+0x56,
+0xfb,
+0x6f,
+0x00,
+0x7f,
+0xb8,
+0x17,
+0x23,
+0x00,
+0x23,
+0xf0,
+0xed,
+0x6d,
+0x00,
+0x90,
+0x26,
+0x15,
+0xe9,
+0x00,
+0x60,
+0xf0,
+0x19,
+0x01,
+0x00,
+0x71,
+0x16,
+0x2e,
+0xe9,
+0x00,
+0x81,
+0x47,
+0x58,
+0x21,
+0x00,
+0x7e,
+0xa4,
+0x32,
+0x5b,
+0x00,
+0x7f,
+0xeb,
+0x8a,
+0x7d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x65,
+0xac,
+0x8c,
+0x2f,
+0x00,
+0x5a,
+0x9d,
+0xf7,
+0xab,
+0x00,
+0x32,
+0xf5,
+0x2c,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x0f,
+0xd7,
+0x21,
+0x8f,
+0x00,
+0x60,
+0x51,
+0xbc,
+0xe3,
+0x00,
+0x0f,
+0xd7,
+0x21,
+0x8f,
+0x00,
+0x8a,
+0x4a,
+0x1d,
+0x99,
+0x00,
+0x75,
+0xb5,
+0xe2,
+0x67,
+0x00,
+0x3b,
+0x62,
+0x92,
+0x93,
+0x00,
+0xc8,
+0x64,
+0x0c,
+0xc5,
+0x00,
+0x3c,
+0x39,
+0x60,
+0xa7,
+0x00,
+0x94,
+0x28,
+0xff,
+0x89,
+0x00,
+0x6b,
+0xd7,
+0x00,
+0x77,
+0x00,
+0x36,
+0xc4,
+0x2d,
+0xd3,
+0x00,
+0xd1,
+0x5f,
+0x0d,
+0x4f,
+0x00,
+0x37,
+0xdc,
+0xc4,
+0xde,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x03,
+0x03,
+0x01,
+0x00,
+0x00,
+0x01,
+0x00,
+0x01,
+0x01,
+0x00,
+0x01,
+0x01,
+0x03,
+0x03,
+0x00,
+0x00,
+0x04,
+0x9d,
+0x1d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x07,
+0xff,
+0xff,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0xb1,
+0xca,
+0x9e,
+0xb7,
+0x00,
+0x4c,
+0x9f,
+0xb1,
+0xad,
+0x00,
+0x54,
+0x54,
+0x77,
+0x9b,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x7f,
+0xf7,
+0xc6,
+0x15,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7e,
+0x67,
+0x33,
+0xd7,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0xb1,
+0xca,
+0x9e,
+0xb7,
+0x00,
+0x4c,
+0x9f,
+0xb1,
+0xad,
+0x00,
+0x54,
+0x54,
+0x77,
+0x9b,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x7f,
+0xf7,
+0xc6,
+0x15,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7e,
+0x67,
+0x33,
+0xd7,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x02,
+0x01,
+0x00,
+0x01,
+0x00,
+0x02,
+0x01,
+0x00,
+0x00,
+0x0a,
+0x47,
+0xe7,
+0x00,
+0x7f,
+0xeb,
+0x70,
+0x33,
+0x00,
+0x00,
+0x0a,
+0x47,
+0xe7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x64,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x20,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x44,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x15,
+0xd2,
+0xf3,
+0x00,
+0x0c,
+0xc9,
+0x9a,
+0x66,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x30,
+0x00,
+0x01,
+0x20,
+0x00,
+0x00,
+0x00,
+0x00,
+0x20,
+0x00,
+0x00,
+0x00,
+0x03,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xaf,
+0x00,
+0x00,
+0x00,
+0x00,
+0x70,
+0x00,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0xe3,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x03,
+0x00,
+0x00,
+0x3c,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x61,
+0xa8,
+0x00,
+0x00,
+0xd6,
+0x82,
+0x03,
+0x00,
+0x00,
+0x00,
+0x09,
+0xc4,
+0x00,
+0x00,
+0x00,
+0x4e,
+0x20,
+0x00,
+0x00,
+0x00,
+0x13,
+0x88,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0xc4,
+0x10,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0xc3,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0d,
+0xaf,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0xc6,
+0x00,
+0x00,
+0x00,
+0x02,
+0x8c,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x64,
+0x06,
+0x40,
+0x63,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x30,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x50,
+0x00,
+0x00,
+0x00,
+0x01,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xcc,
+0xcc,
+0xcc,
+0x00,
+0x19,
+0x99,
+0x99,
+0x99,
+0x00,
+0x33,
+0x33,
+0x33,
+0x33,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x66,
+0x66,
+0x66,
+0x66,
+0x00,
+0x5f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x46,
+0x66,
+0x66,
+0x65,
+0x00,
+0x2a,
+0xaa,
+0xaa,
+0xaa,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x73,
+0x33,
+0x33,
+0x32,
+0x00,
+0x7a,
+0xe1,
+0x47,
+0xad,
+0x00,
+0x43,
+0xd7,
+0x00,
+0x00,
+0x00,
+0x43,
+0xd7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0x00,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x64,
+0x87,
+0xed,
+0x4a,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x04,
+0xa0,
+0x00,
+0xff,
+0xfe,
+0x5f,
+0xe7,
+0x00,
+0x00,
+0x5b,
+0x05,
+0xb0,
+0x00,
+0xf5,
+0x55,
+0x55,
+0x56,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x40,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x5f,
+0x37,
+0x59,
+0xdf,
+0x00,
+0x5f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x40,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x7f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x18,
+0x00,
+0x00,
+0x00,
+0xae,
+0xb8,
+0x0f,
+0x00,
+0x7f,
+0xf4,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x00,
+0x41,
+0x00,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf8,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x49,
+0xe7,
+0x60,
+0x00,
+0x0c,
+0x28,
+0xf5,
+0xc0,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0xf1,
+0x91,
+0x96,
+0x00,
+0x7a,
+0x1c,
+0xdc,
+0xd3,
+0x00,
+0x02,
+0xf1,
+0x91,
+0x96,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xd6,
+0xe3,
+0xb3,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x00,
+0x66,
+0x84,
+0xe5,
+0x00,
+0x7f,
+0x32,
+0xf6,
+0x37,
+0x00,
+0x00,
+0x66,
+0x84,
+0xe5,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xd6,
+0xe3,
+0xb3,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x8c,
+0x5d,
+0xa4,
+0x55,
+0x00,
+0x67,
+0xf9,
+0x80,
+0xe3,
+0x00,
+0x74,
+0x57,
+0x25,
+0x39,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x68,
+0xc1,
+0x4b,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x06,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1d,
+0x53,
+0x00,
+0x00,
+0x13,
+0x9d,
+0x01,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1d,
+0xa1,
+0x2f,
+0x68,
+0x00,
+0x00,
+0x00,
+0x00,
+0xfa,
+0x00,
+0x00,
+0x00,
+0xc8,
+0xd7,
+0x00,
+0x00,
+0x7d,
+0x86,
+0x6e,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0xad,
+0x4f,
+0xc5,
+0xad,
+0x00,
+0x7c,
+0xd3,
+0x79,
+0x2f,
+0x00,
+0x55,
+0xdc,
+0xc1,
+0x23,
+0x00,
+0x00,
+0x00,
+0x0d,
+0xba,
+0x00,
+0x00,
+0x00,
+0x44,
+0xa2,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x0c,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x40,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x56,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x7a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xb2,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xe8,
+0x00,
+0x00,
+0x00,
+0x1d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x9c,
+0x00,
+0x00,
+0x00,
+0x20,
+0x9c,
+0x00,
+0x00,
+0x00,
+0x21,
+0x40,
+0x00,
+0x00,
+0x00,
+0x21,
+0x62,
+0x00,
+0x00,
+0x00,
+0x21,
+0xd4,
+0x00,
+0x00,
+0x00,
+0x22,
+0x08,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x22,
+0x98,
+0x00,
+0x00,
+0x00,
+0x23,
+0xa4,
+0x00,
+0x00,
+0x00,
+0x22,
+0xc0,
+0x00,
+0x00,
+0x00,
+0x22,
+0xc0,
+0x00,
+0x00,
+0x00,
+0x23,
+0x52,
+};
+
+#define VERNUM_09_17_00 0x09170008
+#define CAL_ID_09_17_00 0x00000001
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_09_17_01.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_09_17_01.h
new file mode 100644
index 0000000..ff6ac21
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/FromCode_09_17_01.h
@@ -0,0 +1,27495 @@
+/**
+ * @brief LC898123F40 OIS PRODUCT
+ *
+ * @author (C) 2017 ON Semiconductor.
+ *
+ **/
+
+// Time Stamp : 2018/02/21 14:04:27
+
+#define FromCodeBlockSize_09_17_01 11
+
+// [09170108] [00000002] [00000100]
+
+//#define MODULE_VENDOR 0x09
+//#define END_USER 0x17
+//#define USE_ACT 0x01
+
+const unsigned char CcMagicCodeF40_09_17_01[] = {
+0x00,
+0x01,
+0x02,
+0x03,
+0x04,
+0x05,
+0x06,
+0x07,
+0x08,
+0x09,
+0x00,
+0x00,
+0x00,
+0x00,
+0x54,
+0x00,
+0x00,
+0x00,
+0x13,
+0x00,
+0x00,
+0x00,
+0x00,
+0x06,
+0x98,
+0x00,
+0x00,
+0x00,
+0x01,
+0xfa,
+0x00,
+0x00,
+0x00,
+0x13,
+0x54,
+0x00,
+0x00,
+0x80,
+0x00,
+0x90,
+0x00,
+0x00,
+0x00,
+0xbb,
+0x67,
+0x00,
+0x8f,
+0x41,
+0x47,
+0xef,
+0x00,
+0x00,
+0x00,
+0xba,
+0x82,
+0x00,
+0xa9,
+0xce,
+0x6c,
+0xe2,
+0x00,
+0x00,
+0x00,
+0xc0,
+0x36,
+0x00,
+0x99,
+0xbd,
+0xd9,
+0x4c,
+0x00,
+0x00,
+0x00,
+0xb8,
+0xca,
+0x00,
+0xf7,
+0x7a,
+0xc1,
+0x7f,
+0x00,
+0x00,
+0x00,
+0xd0,
+0x7c,
+0x00,
+0x12,
+0x61,
+0x71,
+0x6f,
+0x00,
+0x00,
+0x00,
+0xbd,
+0xbb,
+0x00,
+0xf8,
+0x40,
+0xa3,
+0xf9,
+0x00,
+0x00,
+0x00,
+0xc2,
+0x21,
+0x00,
+0xec,
+0x9c,
+0xaa,
+0xba,
+0x00,
+0x00,
+0x00,
+0xcb,
+0xd0,
+0x00,
+0x90,
+0x1c,
+0x81,
+0x9e,
+0x00,
+0x00,
+0x00,
+0xa8,
+0xd8,
+0x00,
+0x61,
+0xe9,
+0xc7,
+0x7a,
+0x00,
+0x00,
+0x00,
+0x60,
+0x8f,
+0x00,
+0xd7,
+0x48,
+0xf4,
+0x43,
+0x00,
+0x00,
+0x00,
+0xb2,
+0x24,
+0x00,
+0x40,
+0x74,
+0x8a,
+0xac,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+0x00,
+0x00,
+0x01,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xfe,
+0x00,
+};
+
+const unsigned char CcFromCodeF40_09_17_01[] = {
+0x64,
+0x00,
+0x00,
+0x02,
+0x87,
+0x64,
+0x00,
+0x00,
+0xfe,
+0xe7,
+0x64,
+0x00,
+0x01,
+0x02,
+0x67,
+0x64,
+0x00,
+0x01,
+0x08,
+0x87,
+0x64,
+0x00,
+0x01,
+0x08,
+0xa7,
+0x46,
+0x0b,
+0x03,
+0x80,
+0x00,
+0x64,
+0x00,
+0x01,
+0x08,
+0xc7,
+0x64,
+0x00,
+0x01,
+0x08,
+0xe7,
+0x64,
+0x00,
+0x01,
+0x09,
+0x07,
+0x64,
+0x00,
+0x01,
+0x09,
+0x27,
+0x64,
+0x00,
+0x01,
+0x09,
+0x47,
+0x64,
+0x00,
+0x01,
+0x09,
+0x67,
+0x64,
+0x00,
+0x01,
+0x09,
+0x87,
+0x64,
+0x00,
+0x01,
+0x09,
+0xa7,
+0x64,
+0x00,
+0x01,
+0x09,
+0xc7,
+0x46,
+0x0b,
+0x03,
+0x80,
+0x00,
+0x00,
+0x09,
+0x17,
+0x01,
+0x08,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xba,
+0x11,
+0x2b,
+0xa1,
+0x13,
+0x68,
+0x00,
+0x03,
+0x00,
+0x38,
+0x68,
+0x34,
+0x00,
+0x09,
+0x20,
+0x5c,
+0x81,
+0x09,
+0x8e,
+0xb9,
+0x5c,
+0x03,
+0xc3,
+0x0c,
+0x02,
+0x5c,
+0x96,
+0x00,
+0x40,
+0x52,
+0xa0,
+0x02,
+0x06,
+0x80,
+0x01,
+0x9e,
+0x7c,
+0x28,
+0x00,
+0xd2,
+0x84,
+0x05,
+0x0a,
+0x05,
+0x60,
+0x68,
+0x0f,
+0xc4,
+0xdb,
+0x00,
+0x84,
+0x05,
+0x0a,
+0x03,
+0x60,
+0x68,
+0x02,
+0x44,
+0xdf,
+0xc0,
+0xab,
+0xfc,
+0x08,
+0x02,
+0x50,
+0x40,
+0x00,
+0x00,
+0x80,
+0x60,
+0x66,
+0x00,
+0x01,
+0x10,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x0f,
+0xe0,
+0x5c,
+0x00,
+0x40,
+0x80,
+0x20,
+0x68,
+0x38,
+0x1c,
+0x09,
+0x24,
+0x5c,
+0x04,
+0x00,
+0x40,
+0x50,
+0xa2,
+0x26,
+0x08,
+0x60,
+0x7a,
+0x40,
+0x00,
+0x00,
+0x40,
+0x50,
+0x66,
+0x00,
+0x00,
+0xf8,
+0x40,
+0x68,
+0x20,
+0x00,
+0x00,
+0x24,
+0x68,
+0x00,
+0x00,
+0x08,
+0x20,
+0x98,
+0x90,
+0x96,
+0x60,
+0x00,
+0x12,
+0xfc,
+0x85,
+0xc0,
+0x13,
+0x18,
+0x80,
+0x89,
+0x8e,
+0x80,
+0x6e,
+0x00,
+0x03,
+0xb4,
+0x70,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb7,
+0x20,
+0x66,
+0x00,
+0x01,
+0x6f,
+0xe0,
+0x66,
+0x00,
+0x01,
+0x70,
+0xc0,
+0x68,
+0x00,
+0x01,
+0x0f,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb7,
+0x20,
+0x68,
+0x00,
+0x00,
+0x00,
+0xb9,
+0xba,
+0x11,
+0x00,
+0x00,
+0x00,
+0x66,
+0x00,
+0x01,
+0x1b,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x19,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x1c,
+0xc0,
+0x66,
+0x00,
+0x01,
+0x60,
+0x40,
+0x66,
+0x00,
+0x00,
+0xfd,
+0xa0,
+0x68,
+0x20,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x00,
+0x45,
+0xc0,
+0x66,
+0x00,
+0x00,
+0x1c,
+0xe0,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x66,
+0x00,
+0x01,
+0x52,
+0x40,
+0x6e,
+0x40,
+0x00,
+0x08,
+0x38,
+0x68,
+0x00,
+0x20,
+0x00,
+0x08,
+0x54,
+0x48,
+0x03,
+0x00,
+0x19,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x02,
+0x59,
+0x00,
+0x03,
+0x01,
+0x03,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x51,
+0x52,
+0x06,
+0x93,
+0xc0,
+0x99,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x52,
+0x38,
+0x1d,
+0x26,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x02,
+0x40,
+0x80,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x00,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x40,
+0x38,
+0x1c,
+0x22,
+0x40,
+0x80,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x50,
+0x66,
+0x00,
+0x01,
+0x20,
+0x80,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x7a,
+0x38,
+0x01,
+0xc6,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x96,
+0x60,
+0x04,
+0x03,
+0x30,
+0x86,
+0x80,
+0x00,
+0x00,
+0x0f,
+0x98,
+0x80,
+0x7a,
+0x66,
+0x00,
+0x40,
+0x34,
+0x60,
+0x32,
+0x82,
+0x0b,
+0xc0,
+0x98,
+0x88,
+0x00,
+0x06,
+0xc7,
+0x00,
+0x80,
+0x20,
+0x25,
+0x50,
+0x20,
+0x30,
+0x12,
+0x15,
+0x20,
+0x29,
+0x08,
+0x05,
+0x06,
+0xc7,
+0x00,
+0x80,
+0x25,
+0x23,
+0x21,
+0x40,
+0x40,
+0x00,
+0x03,
+0xff,
+0x02,
+0x66,
+0x00,
+0x40,
+0x34,
+0x20,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x68,
+0x38,
+0x08,
+0x10,
+0x24,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x5c,
+0x96,
+0x12,
+0x20,
+0x24,
+0x5c,
+0x0e,
+0x90,
+0x80,
+0x64,
+0x5c,
+0x09,
+0xc2,
+0x20,
+0x24,
+0x88,
+0x0e,
+0x4a,
+0x20,
+0x24,
+0x88,
+0x16,
+0x4a,
+0x20,
+0x24,
+0x88,
+0x1e,
+0x4a,
+0x20,
+0x24,
+0xa2,
+0x02,
+0x48,
+0x82,
+0x64,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x5c,
+0x00,
+0xca,
+0xc0,
+0xa8,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x03,
+0x32,
+0xa1,
+0x8b,
+0xff,
+0xa1,
+0xa0,
+0x00,
+0x48,
+0x03,
+0x03,
+0x25,
+0x89,
+0x8b,
+0xc0,
+0xc9,
+0x88,
+0x2e,
+0x02,
+0x48,
+0x9a,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x51,
+0x68,
+0x00,
+0x08,
+0x27,
+0x21,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x60,
+0x42,
+0x02,
+0xf8,
+0x23,
+0x52,
+0x40,
+0x00,
+0x00,
+0x60,
+0x61,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x20,
+0xa2,
+0x58,
+0x10,
+0xbc,
+0x35,
+0x88,
+0x83,
+0x60,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x98,
+0x80,
+0x26,
+0xc0,
+0x00,
+0x14,
+0xa0,
+0x03,
+0x00,
+0x80,
+0x68,
+0x00,
+0x02,
+0x32,
+0x20,
+0xbc,
+0x0b,
+0x89,
+0x88,
+0x02,
+0x30,
+0x08,
+0x0b,
+0xc2,
+0x81,
+0x5c,
+0x00,
+0x43,
+0x05,
+0xda,
+0x68,
+0x38,
+0x08,
+0x03,
+0x20,
+0x6c,
+0x70,
+0x10,
+0x06,
+0x52,
+0x84,
+0x1d,
+0x0b,
+0xc2,
+0x07,
+0x5c,
+0xb1,
+0x00,
+0x80,
+0x20,
+0x6c,
+0x70,
+0x10,
+0x20,
+0x0a,
+0x5c,
+0x00,
+0x40,
+0x40,
+0x08,
+0x88,
+0x0a,
+0x48,
+0x40,
+0x81,
+0x86,
+0x08,
+0x28,
+0x81,
+0x20,
+0x88,
+0x1a,
+0x48,
+0x40,
+0x83,
+0x86,
+0x08,
+0x98,
+0x82,
+0x20,
+0x68,
+0x00,
+0x00,
+0x96,
+0x24,
+0x80,
+0x00,
+0xb6,
+0xc0,
+0x00,
+0x30,
+0x64,
+0xa6,
+0xc0,
+0x00,
+0x2f,
+0x04,
+0x86,
+0xc0,
+0x00,
+0x31,
+0xc5,
+0x16,
+0xc4,
+0x00,
+0x24,
+0x65,
+0x26,
+0xc0,
+0x00,
+0x12,
+0xc5,
+0x38,
+0x60,
+0xc9,
+0x6c,
+0x00,
+0x02,
+0x9a,
+0x4b,
+0x40,
+0x00,
+0x00,
+0x40,
+0x50,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x7a,
+0x66,
+0x00,
+0x00,
+0xce,
+0x40,
+0x68,
+0x00,
+0x00,
+0x4b,
+0x20,
+0x5c,
+0x87,
+0x00,
+0x82,
+0xa4,
+0x68,
+0x00,
+0x00,
+0x5f,
+0x21,
+0x80,
+0x00,
+0x05,
+0x14,
+0x22,
+0x00,
+0x80,
+0x05,
+0x14,
+0x22,
+0x86,
+0x02,
+0x46,
+0xc6,
+0x81,
+0x00,
+0x04,
+0x88,
+0x82,
+0xe1,
+0x88,
+0x3e,
+0x06,
+0x83,
+0x40,
+0x80,
+0x02,
+0x5b,
+0xa0,
+0xc8,
+0x86,
+0xc4,
+0x90,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x66,
+0x00,
+0x01,
+0x1d,
+0x40,
+0x66,
+0x00,
+0x02,
+0x3c,
+0x60,
+0x6c,
+0x00,
+0x01,
+0x94,
+0x20,
+0x40,
+0x00,
+0x03,
+0xa0,
+0x80,
+0x68,
+0x00,
+0x00,
+0xc2,
+0x20,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x24,
+0x68,
+0x00,
+0x00,
+0xc6,
+0x21,
+0x68,
+0x20,
+0x00,
+0xe6,
+0x25,
+0x68,
+0x00,
+0x00,
+0xac,
+0x22,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x23,
+0x66,
+0x00,
+0x01,
+0x38,
+0xc0,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x68,
+0x00,
+0x01,
+0x50,
+0x20,
+0x5c,
+0x83,
+0x0a,
+0xc0,
+0x20,
+0x5c,
+0x88,
+0x10,
+0x02,
+0x80,
+0x5c,
+0x87,
+0x18,
+0x40,
+0x02,
+0x57,
+0x00,
+0x80,
+0x83,
+0x20,
+0x5c,
+0x80,
+0x89,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0x8f,
+0x24,
+0x5c,
+0x08,
+0x50,
+0x03,
+0x01,
+0x51,
+0x60,
+0x20,
+0x22,
+0x00,
+0x86,
+0x00,
+0x35,
+0x70,
+0x0c,
+0x00,
+0x18,
+0x35,
+0x70,
+0x2d,
+0x80,
+0x30,
+0x15,
+0xc8,
+0x19,
+0x98,
+0x0c,
+0x35,
+0x16,
+0x0f,
+0x84,
+0x00,
+0x35,
+0x70,
+0x2c,
+0x18,
+0x00,
+0x16,
+0x80,
+0x00,
+0x1d,
+0x52,
+0x45,
+0x16,
+0x07,
+0x18,
+0x00,
+0x05,
+0x16,
+0x03,
+0x96,
+0x0d,
+0x76,
+0x80,
+0x00,
+0x17,
+0x92,
+0x15,
+0xc9,
+0x61,
+0x18,
+0xe4,
+0x95,
+0x24,
+0x55,
+0x80,
+0xa0,
+0x08,
+0x48,
+0x01,
+0x57,
+0x00,
+0x49,
+0x84,
+0xf9,
+0x5c,
+0x09,
+0xc1,
+0x8e,
+0x43,
+0x52,
+0x04,
+0xd9,
+0x61,
+0xd7,
+0x96,
+0x25,
+0x46,
+0x80,
+0x00,
+0x18,
+0x42,
+0x19,
+0x80,
+0x41,
+0x51,
+0x60,
+0x68,
+0x0a,
+0x08,
+0x84,
+0x80,
+0x25,
+0x70,
+0x89,
+0x08,
+0x2a,
+0x09,
+0x80,
+0x82,
+0x51,
+0x60,
+0xb8,
+0x83,
+0xa5,
+0x96,
+0x0d,
+0x79,
+0x60,
+0x55,
+0x98,
+0x4f,
+0x98,
+0x40,
+0x02,
+0x51,
+0x60,
+0xa0,
+0x68,
+0x01,
+0x51,
+0x60,
+0x71,
+0x60,
+0xd6,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x09,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x01,
+0x51,
+0x60,
+0x61,
+0x60,
+0xd4,
+0x51,
+0x61,
+0x71,
+0x60,
+0xd6,
+0x96,
+0x0f,
+0x69,
+0x60,
+0xd4,
+0x00,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x26,
+0xa0,
+0x35,
+0x94,
+0x0c,
+0x16,
+0x0b,
+0x95,
+0x50,
+0x24,
+0xbf,
+0x19,
+0x95,
+0xc0,
+0xe9,
+0x16,
+0x0f,
+0x14,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xa1,
+0xc0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0xf0,
+0xe7,
+0x68,
+0x20,
+0x00,
+0x04,
+0x20,
+0x5c,
+0x0b,
+0xe2,
+0xbf,
+0xf0,
+0x94,
+0x03,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x09,
+0x9a,
+0x01,
+0x20,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0xa0,
+0x00,
+0x12,
+0xa0,
+0x64,
+0x3b,
+0x10,
+0x43,
+0x23,
+0x20,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x38,
+0x17,
+0x42,
+0x59,
+0x28,
+0xbc,
+0x09,
+0x95,
+0xc0,
+0x32,
+0x2c,
+0x02,
+0x06,
+0x82,
+0x00,
+0x01,
+0x92,
+0x12,
+0xa0,
+0x64,
+0x3b,
+0x10,
+0x43,
+0x23,
+0xa0,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x5c,
+0x0a,
+0xe3,
+0x00,
+0x76,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xa9,
+0x5c,
+0x00,
+0x68,
+0x80,
+0x76,
+0x68,
+0x20,
+0x00,
+0x1b,
+0x21,
+0x38,
+0x09,
+0xc2,
+0xa0,
+0x76,
+0x3b,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x08,
+0x7a,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x04,
+0x76,
+0x89,
+0x8e,
+0x88,
+0x88,
+0x0a,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x04,
+0x08,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x00,
+0x47,
+0x68,
+0x5c,
+0x00,
+0xa9,
+0x8e,
+0x88,
+0x5c,
+0x82,
+0x08,
+0x80,
+0xa0,
+0x39,
+0x02,
+0x0a,
+0x04,
+0x20,
+0x80,
+0x08,
+0xb8,
+0x00,
+0x09,
+0x51,
+0x91,
+0x50,
+0x00,
+0x00,
+0x80,
+0x00,
+0x16,
+0x83,
+0x40,
+0x0a,
+0x02,
+0x18,
+0x00,
+0x08,
+0x84,
+0x84,
+0xb6,
+0x80,
+0x00,
+0x05,
+0x82,
+0x28,
+0x00,
+0x09,
+0x80,
+0x00,
+0xa8,
+0x00,
+0x0b,
+0x6c,
+0x40,
+0x00,
+0x78,
+0x50,
+0x6c,
+0x40,
+0x00,
+0xc6,
+0x51,
+0x85,
+0x0c,
+0xb6,
+0x80,
+0x00,
+0x04,
+0x42,
+0x38,
+0x00,
+0x00,
+0x84,
+0x80,
+0x15,
+0x48,
+0x44,
+0x80,
+0x00,
+0xb8,
+0x40,
+0x82,
+0x80,
+0x00,
+0x38,
+0x48,
+0x51,
+0xa0,
+0x6c,
+0x06,
+0xc4,
+0x00,
+0x15,
+0xc5,
+0x3b,
+0x10,
+0x03,
+0x94,
+0x03,
+0x96,
+0x80,
+0x00,
+0x09,
+0xe2,
+0x15,
+0x44,
+0x66,
+0x05,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x18,
+0xc5,
+0x25,
+0x90,
+0x10,
+0x05,
+0x8c,
+0xa8,
+0x50,
+0x49,
+0x42,
+0x04,
+0x48,
+0x48,
+0x50,
+0x84,
+0x8c,
+0xb8,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x00,
+0x27,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x18,
+0x08,
+0x40,
+0x00,
+0x00,
+0x80,
+0xa0,
+0x68,
+0x00,
+0x01,
+0x00,
+0x09,
+0x94,
+0x03,
+0xc2,
+0x89,
+0x65,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0xa1,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x5c,
+0x08,
+0xe0,
+0x80,
+0xe0,
+0x24,
+0x12,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x06,
+0x60,
+0x00,
+0x02,
+0x9c,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x80,
+0x59,
+0x40,
+0x3c,
+0x28,
+0x96,
+0x53,
+0x20,
+0x28,
+0xbc,
+0x0c,
+0x98,
+0x80,
+0x36,
+0x88,
+0x07,
+0x66,
+0x80,
+0x00,
+0x15,
+0x02,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x86,
+0x80,
+0x04,
+0x01,
+0xcc,
+0x88,
+0x80,
+0x36,
+0x64,
+0x00,
+0x00,
+0x42,
+0x4f,
+0xa8,
+0x01,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0xab,
+0xfd,
+0x04,
+0x20,
+0x6f,
+0x88,
+0x0f,
+0x65,
+0xc0,
+0x03,
+0x08,
+0x04,
+0x98,
+0x82,
+0x4a,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa0,
+0x88,
+0x20,
+0x85,
+0x50,
+0x32,
+0x08,
+0x12,
+0x05,
+0xd4,
+0x83,
+0x08,
+0x18,
+0x85,
+0x50,
+0x32,
+0x20,
+0x02,
+0x08,
+0x80,
+0x09,
+0x58,
+0x0b,
+0x80,
+0x81,
+0xc8,
+0xbf,
+0xf2,
+0xa8,
+0x81,
+0x60,
+0x00,
+0x00,
+0x08,
+0x80,
+0xb6,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xfd,
+0x08,
+0x80,
+0x48,
+0x88,
+0x0f,
+0x66,
+0x82,
+0x00,
+0x08,
+0x52,
+0x08,
+0x81,
+0x7a,
+0x88,
+0x1e,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x0a,
+0x55,
+0x03,
+0xb8,
+0x81,
+0xa0,
+0x5d,
+0x4e,
+0x38,
+0x80,
+0x0a,
+0x59,
+0x09,
+0xc2,
+0x00,
+0x20,
+0x55,
+0x03,
+0xa0,
+0x81,
+0x4b,
+0xbf,
+0xf3,
+0xa8,
+0x80,
+0x48,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x40,
+0x80,
+0x68,
+0x20,
+0x00,
+0x85,
+0x20,
+0x88,
+0x20,
+0x05,
+0x16,
+0x02,
+0x20,
+0x08,
+0x08,
+0x80,
+0xb6,
+0x46,
+0x0a,
+0x41,
+0x40,
+0x64,
+0x94,
+0x0e,
+0x0a,
+0x80,
+0x30,
+0x5c,
+0x00,
+0x22,
+0xbf,
+0x20,
+0x88,
+0x1c,
+0x8a,
+0x40,
+0xc0,
+0x88,
+0x16,
+0x08,
+0x82,
+0x76,
+0x68,
+0x00,
+0x40,
+0x1a,
+0x48,
+0x88,
+0x2c,
+0x86,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x88,
+0x55,
+0x03,
+0x20,
+0x81,
+0x20,
+0x5d,
+0x48,
+0x30,
+0x82,
+0x88,
+0x59,
+0x0f,
+0x82,
+0x00,
+0x20,
+0x43,
+0xfa,
+0x50,
+0x81,
+0x60,
+0x55,
+0x03,
+0x20,
+0x81,
+0xca,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x42,
+0x80,
+0x5c,
+0x81,
+0x02,
+0x40,
+0xc0,
+0x5c,
+0x00,
+0x0a,
+0x00,
+0x71,
+0xa0,
+0xc1,
+0x29,
+0x48,
+0x35,
+0x95,
+0x03,
+0x7a,
+0x41,
+0xa1,
+0x23,
+0x08,
+0xc5,
+0x50,
+0x26,
+0x18,
+0x22,
+0x95,
+0xd4,
+0x80,
+0x9c,
+0x08,
+0x25,
+0x90,
+0xe4,
+0x15,
+0x0b,
+0x65,
+0x70,
+0xb9,
+0x15,
+0x03,
+0x45,
+0x70,
+0xf0,
+0x1c,
+0x88,
+0x34,
+0x3f,
+0x9d,
+0x15,
+0x8f,
+0x29,
+0x58,
+0x70,
+0x5c,
+0x86,
+0x0a,
+0x42,
+0xa3,
+0x88,
+0xa0,
+0x85,
+0x16,
+0x12,
+0x20,
+0x87,
+0x05,
+0x1a,
+0x12,
+0x20,
+0x46,
+0x15,
+0xb0,
+0x81,
+0x20,
+0x82,
+0x25,
+0x15,
+0x88,
+0x08,
+0x16,
+0x22,
+0xe1,
+0x12,
+0x2e,
+0x11,
+0x12,
+0x2b,
+0x09,
+0x51,
+0x58,
+0x90,
+0x18,
+0x41,
+0x51,
+0x59,
+0x30,
+0x18,
+0x42,
+0x54,
+0x09,
+0x00,
+0x18,
+0x40,
+0x54,
+0x08,
+0x10,
+0x18,
+0x7a,
+0x51,
+0x58,
+0x00,
+0x18,
+0x4a,
+0x51,
+0x58,
+0x80,
+0x18,
+0x40,
+0x81,
+0xac,
+0x08,
+0x80,
+0x63,
+0x40,
+0x00,
+0x00,
+0x81,
+0xe3,
+0x68,
+0x20,
+0x00,
+0x90,
+0x23,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x66,
+0x00,
+0x00,
+0x37,
+0xe0,
+0x5c,
+0x81,
+0x00,
+0x8a,
+0x08,
+0x51,
+0xa1,
+0x20,
+0x81,
+0xa3,
+0x5b,
+0x08,
+0x12,
+0xc0,
+0xc1,
+0x51,
+0x58,
+0x80,
+0x81,
+0x20,
+0x2e,
+0x11,
+0x25,
+0x70,
+0x88,
+0xa0,
+0x03,
+0x05,
+0x15,
+0x84,
+0xa0,
+0x46,
+0x15,
+0x15,
+0x89,
+0x01,
+0x84,
+0x15,
+0x15,
+0x93,
+0x01,
+0x84,
+0x25,
+0x40,
+0x90,
+0x01,
+0x84,
+0x05,
+0x40,
+0x81,
+0x01,
+0x87,
+0xa5,
+0x15,
+0x80,
+0x01,
+0x84,
+0xa5,
+0x15,
+0x88,
+0x01,
+0x84,
+0x08,
+0x1a,
+0xc0,
+0x88,
+0x06,
+0x36,
+0x82,
+0x00,
+0x09,
+0x8a,
+0x36,
+0x82,
+0x00,
+0x09,
+0x2a,
+0x46,
+0x82,
+0x00,
+0x09,
+0x5a,
+0x56,
+0x60,
+0x00,
+0x03,
+0x7e,
+0x8a,
+0x08,
+0x22,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x0e,
+0x00,
+0x00,
+0x00,
+0xba,
+0x14,
+0x82,
+0xe1,
+0xa8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x59,
+0x06,
+0x42,
+0xbf,
+0xd0,
+0x49,
+0xa9,
+0xb8,
+0x80,
+0x76,
+0xbc,
+0x19,
+0x89,
+0xa0,
+0x00,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x10,
+0x02,
+0xa0,
+0x76,
+0x68,
+0x00,
+0x09,
+0xc4,
+0x09,
+0x50,
+0x41,
+0x83,
+0xa1,
+0x11,
+0x44,
+0x08,
+0x81,
+0x01,
+0x58,
+0x5b,
+0x40,
+0x11,
+0x02,
+0x59,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x42,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x19,
+0x02,
+0x12,
+0x29,
+0x85,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x1a,
+0x33,
+0x60,
+0x00,
+0x37,
+0x80,
+0x0b,
+0xc1,
+0x77,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x10,
+0x02,
+0xa0,
+0x64,
+0x68,
+0x07,
+0xff,
+0xff,
+0xc9,
+0x50,
+0x41,
+0x03,
+0xa1,
+0x11,
+0x44,
+0x48,
+0x81,
+0x01,
+0x58,
+0x5b,
+0x40,
+0x11,
+0x02,
+0x59,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x42,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x19,
+0x02,
+0x12,
+0x29,
+0x85,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x02,
+0x33,
+0x60,
+0x00,
+0x37,
+0x80,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5b,
+0x02,
+0x03,
+0x01,
+0x01,
+0x50,
+0x43,
+0x11,
+0x84,
+0x0a,
+0x46,
+0x08,
+0x89,
+0x83,
+0x88,
+0x44,
+0x41,
+0x09,
+0x84,
+0x8b,
+0x98,
+0x30,
+0xa4,
+0x47,
+0x44,
+0xab,
+0xfe,
+0x05,
+0x08,
+0x20,
+0x08,
+0x07,
+0x60,
+0x88,
+0x82,
+0x37,
+0x80,
+0x06,
+0x80,
+0x00,
+0x9c,
+0x40,
+0x80,
+0x84,
+0x80,
+0x6a,
+0x00,
+0x4e,
+0x20,
+0x02,
+0x5b,
+0x44,
+0x11,
+0x01,
+0x58,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x40,
+0x0b,
+0xa1,
+0x01,
+0x90,
+0x11,
+0x26,
+0xa0,
+0x04,
+0xe2,
+0x00,
+0x12,
+0x98,
+0x52,
+0x32,
+0x01,
+0x0b,
+0xc0,
+0x3b,
+0x88,
+0x03,
+0x63,
+0x60,
+0x00,
+0x37,
+0x80,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0xab,
+0xf8,
+0x09,
+0x40,
+0x36,
+0x88,
+0x04,
+0xa8,
+0x80,
+0xe5,
+0x88,
+0x16,
+0x48,
+0x81,
+0xe3,
+0x88,
+0x26,
+0x28,
+0x82,
+0xf6,
+0x66,
+0x00,
+0x00,
+0x31,
+0xa8,
+0x40,
+0x00,
+0x01,
+0x48,
+0x35,
+0x5c,
+0x00,
+0x38,
+0x83,
+0x48,
+0x6a,
+0x00,
+0x02,
+0x71,
+0x00,
+0x5b,
+0x40,
+0x00,
+0x83,
+0xcb,
+0x88,
+0x45,
+0x49,
+0x05,
+0x58,
+0x00,
+0x00,
+0x08,
+0x80,
+0x0a,
+0x51,
+0x85,
+0xf8,
+0x82,
+0x22,
+0x98,
+0x2e,
+0x89,
+0xd0,
+0x02,
+0x88,
+0x4e,
+0x86,
+0x60,
+0x00,
+0x03,
+0x1a,
+0x89,
+0x50,
+0x35,
+0x5b,
+0x48,
+0x03,
+0x01,
+0xfe,
+0x52,
+0xcd,
+0x01,
+0x82,
+0x0b,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xc9,
+0x55,
+0x01,
+0xd1,
+0x80,
+0x0a,
+0xbc,
+0x07,
+0x88,
+0x86,
+0x52,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0xcd,
+0x88,
+0x38,
+0x16,
+0x80,
+0x01,
+0xff,
+0xfc,
+0x2b,
+0xc0,
+0x87,
+0x68,
+0x00,
+0x20,
+0x00,
+0x09,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x4d,
+0x40,
+0x00,
+0x00,
+0x83,
+0x81,
+0x68,
+0x00,
+0x20,
+0x00,
+0x02,
+0x59,
+0x0a,
+0x40,
+0x81,
+0xa2,
+0xbc,
+0x03,
+0xb3,
+0x90,
+0x10,
+0x40,
+0x00,
+0x01,
+0x50,
+0x52,
+0x55,
+0x02,
+0x70,
+0x81,
+0xe2,
+0x00,
+0x00,
+0x08,
+0x84,
+0xa8,
+0x88,
+0x82,
+0x35,
+0x18,
+0x5b,
+0x08,
+0x4c,
+0xa9,
+0xd8,
+0x04,
+0x98,
+0x2a,
+0x88,
+0x60,
+0x08,
+0x9d,
+0x80,
+0x38,
+0x83,
+0x0a,
+0x88,
+0x6e,
+0x46,
+0x60,
+0x00,
+0x03,
+0x1e,
+0x88,
+0x58,
+0x09,
+0x88,
+0x38,
+0xb3,
+0x20,
+0xf8,
+0xbc,
+0x13,
+0x85,
+0xc0,
+0xff,
+0x99,
+0x20,
+0x19,
+0x05,
+0x12,
+0x40,
+0x00,
+0x01,
+0x07,
+0x58,
+0x66,
+0x00,
+0x02,
+0x42,
+0x88,
+0x5b,
+0x40,
+0x08,
+0x84,
+0x16,
+0x55,
+0x00,
+0x09,
+0x07,
+0x10,
+0x6a,
+0x00,
+0x02,
+0x71,
+0x02,
+0x29,
+0x88,
+0x23,
+0x20,
+0x10,
+0xbc,
+0x03,
+0xb3,
+0x81,
+0xff,
+0x36,
+0x04,
+0x13,
+0x78,
+0x41,
+0x52,
+0xce,
+0x41,
+0x83,
+0x4b,
+0x36,
+0x9c,
+0x26,
+0x80,
+0x01,
+0xff,
+0xfc,
+0xbb,
+0xc0,
+0x68,
+0x98,
+0x08,
+0xa3,
+0x01,
+0xf0,
+0xbc,
+0x09,
+0x56,
+0xa0,
+0x01,
+0xff,
+0xfc,
+0x1b,
+0xc0,
+0x67,
+0x68,
+0x00,
+0x20,
+0x00,
+0x0b,
+0x30,
+0x1f,
+0x0b,
+0xc0,
+0x25,
+0x6a,
+0x00,
+0x20,
+0x00,
+0x01,
+0x5c,
+0x80,
+0x80,
+0x81,
+0x22,
+0x98,
+0x34,
+0x19,
+0x50,
+0x51,
+0x88,
+0x16,
+0x20,
+0x00,
+0x00,
+0x88,
+0x6a,
+0x38,
+0x83,
+0x08,
+0x66,
+0x00,
+0x00,
+0x35,
+0x88,
+0x55,
+0x00,
+0x08,
+0x58,
+0x09,
+0x5c,
+0x0f,
+0xf9,
+0x83,
+0x0a,
+0x36,
+0x98,
+0x12,
+0x59,
+0xc0,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xca,
+0xbc,
+0x07,
+0x89,
+0x80,
+0x4b,
+0x30,
+0x1b,
+0x8b,
+0xc0,
+0xcd,
+0x88,
+0x48,
+0xb6,
+0xa0,
+0x01,
+0xff,
+0xfc,
+0x0b,
+0xc0,
+0x87,
+0x68,
+0x00,
+0x20,
+0x00,
+0x0a,
+0x30,
+0x1b,
+0x8b,
+0xc0,
+0x4d,
+0x40,
+0x00,
+0x00,
+0x84,
+0x8b,
+0x6a,
+0x00,
+0x20,
+0x00,
+0x00,
+0x5c,
+0x80,
+0x80,
+0x80,
+0xa2,
+0x5d,
+0x8e,
+0x39,
+0x83,
+0x00,
+0x59,
+0x0d,
+0xc0,
+0x86,
+0x09,
+0x95,
+0x05,
+0x08,
+0x83,
+0x49,
+0x88,
+0x3c,
+0xb8,
+0x80,
+0xe2,
+0xbf,
+0x75,
+0xa8,
+0x80,
+0x0a,
+0x88,
+0x2b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x08,
+0x00,
+0x00,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0x79,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x08,
+0x68,
+0x00,
+0x01,
+0x84,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x48,
+0x68,
+0x00,
+0x01,
+0x8f,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa8,
+0x68,
+0x00,
+0x40,
+0x1d,
+0x88,
+0x68,
+0x00,
+0x40,
+0x1d,
+0xc8,
+0x68,
+0x20,
+0x01,
+0xcb,
+0x20,
+0x66,
+0x00,
+0x00,
+0x26,
+0x08,
+0x5c,
+0x00,
+0xeb,
+0x80,
+0x00,
+0x68,
+0x00,
+0x40,
+0x1e,
+0x88,
+0x68,
+0x20,
+0x01,
+0xce,
+0x20,
+0x66,
+0x00,
+0x00,
+0x26,
+0x08,
+0x38,
+0x01,
+0xd8,
+0x80,
+0x36,
+0x68,
+0x00,
+0x40,
+0x1f,
+0x48,
+0x68,
+0x20,
+0x01,
+0xd1,
+0x20,
+0x64,
+0x00,
+0x00,
+0x26,
+0x0f,
+0x5c,
+0x00,
+0xea,
+0x80,
+0x10,
+0x39,
+0x02,
+0x08,
+0x00,
+0x7a,
+0x80,
+0x07,
+0xa8,
+0x08,
+0x7a,
+0x80,
+0x07,
+0xa8,
+0x08,
+0x7a,
+0x80,
+0x87,
+0xa4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xfe,
+0x08,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x66,
+0x80,
+0x04,
+0x01,
+0x00,
+0x88,
+0x81,
+0x7a,
+0x88,
+0x1c,
+0x86,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x00,
+0x55,
+0x02,
+0x00,
+0x80,
+0x20,
+0x5d,
+0x00,
+0x03,
+0x00,
+0xe2,
+0x58,
+0x04,
+0x02,
+0x00,
+0x20,
+0x88,
+0x18,
+0x14,
+0x3f,
+0x9d,
+0x08,
+0x06,
+0x05,
+0x50,
+0x26,
+0x08,
+0x15,
+0x00,
+0x00,
+0x00,
+0x88,
+0x0b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x68,
+0x68,
+0x34,
+0x04,
+0x14,
+0x21,
+0x6c,
+0x68,
+0x08,
+0x28,
+0x49,
+0x00,
+0x00,
+0x08,
+0x48,
+0x89,
+0x84,
+0x04,
+0x94,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xc0,
+0x4f,
+0x5c,
+0x09,
+0xab,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x08,
+0x25,
+0x96,
+0x0b,
+0xff,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x6c,
+0x68,
+0x08,
+0x28,
+0x49,
+0x00,
+0x00,
+0x08,
+0x40,
+0x09,
+0x84,
+0x8c,
+0x94,
+0x00,
+0x00,
+0x38,
+0x00,
+0x04,
+0x00,
+0x00,
+0x38,
+0x00,
+0x0b,
+0xc0,
+0x4f,
+0x5c,
+0x09,
+0xab,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x08,
+0x25,
+0x96,
+0x0b,
+0xff,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x59,
+0x01,
+0x41,
+0x48,
+0x2f,
+0xbc,
+0x0d,
+0x95,
+0xc0,
+0x87,
+0x14,
+0x02,
+0xd3,
+0x20,
+0x20,
+0xbc,
+0x05,
+0x05,
+0x24,
+0xdc,
+0x3a,
+0x14,
+0x89,
+0x48,
+0x60,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0x4d,
+0x43,
+0xa1,
+0x48,
+0x94,
+0x06,
+0x00,
+0x00,
+0x00,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x40,
+0x52,
+0x0d,
+0xc3,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x00,
+0x00,
+0x00,
+0x52,
+0x0d,
+0x43,
+0xa1,
+0x48,
+0x94,
+0x06,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x05,
+0x20,
+0x40,
+0x7a,
+0x46,
+0x0a,
+0x40,
+0x40,
+0xfa,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x40,
+0x46,
+0x08,
+0x88,
+0x02,
+0x0a,
+0x08,
+0x80,
+0x04,
+0x60,
+0xa4,
+0x00,
+0x24,
+0xc8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x01,
+0x84,
+0x00,
+0x88,
+0x40,
+0x8a,
+0x30,
+0x93,
+0x0b,
+0xc0,
+0x5c,
+0xb0,
+0x00,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x40,
+0xfa,
+0x98,
+0xe8,
+0x8b,
+0xa1,
+0x40,
+0x84,
+0x08,
+0x03,
+0x28,
+0x00,
+0xbc,
+0x05,
+0x08,
+0x40,
+0x00,
+0x55,
+0x02,
+0x23,
+0xa1,
+0x48,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x80,
+0x43,
+0xc0,
+0x5f,
+0x5c,
+0x00,
+0x31,
+0x8e,
+0x80,
+0x55,
+0x02,
+0x01,
+0x40,
+0x46,
+0x3a,
+0x80,
+0x03,
+0x01,
+0x00,
+0xbf,
+0xfb,
+0x2b,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x1c,
+0x04,
+0x21,
+0x38,
+0x00,
+0xe8,
+0x48,
+0x48,
+0xa0,
+0x82,
+0x18,
+0x48,
+0x7a,
+0xa0,
+0x82,
+0x18,
+0x48,
+0x4a,
+0xa0,
+0xcc,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x80,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xb0,
+0x68,
+0x00,
+0x00,
+0x9e,
+0x20,
+0x68,
+0x00,
+0x01,
+0x50,
+0x24,
+0x5c,
+0x00,
+0x10,
+0x00,
+0x01,
+0x5c,
+0x08,
+0x2a,
+0x00,
+0xa1,
+0x55,
+0x00,
+0xb1,
+0x48,
+0x2f,
+0x52,
+0xcb,
+0xc0,
+0x22,
+0x08,
+0x55,
+0x00,
+0x98,
+0x82,
+0x48,
+0xa0,
+0xd4,
+0x18,
+0x40,
+0x09,
+0x88,
+0x36,
+0x44,
+0x20,
+0x44,
+0x88,
+0x3f,
+0x65,
+0x50,
+0x0a,
+0x08,
+0x2e,
+0x16,
+0x80,
+0x00,
+0x09,
+0x62,
+0x06,
+0xc0,
+0x00,
+0x29,
+0xa0,
+0x28,
+0x40,
+0x03,
+0x84,
+0x08,
+0x83,
+0x81,
+0x08,
+0x25,
+0x83,
+0x8b,
+0xc1,
+0x09,
+0x55,
+0x01,
+0x81,
+0x82,
+0x8b,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xae,
+0x38,
+0x12,
+0x82,
+0x58,
+0x30,
+0xbc,
+0x09,
+0x85,
+0x50,
+0x1c,
+0x18,
+0x2c,
+0xa6,
+0x80,
+0x00,
+0x09,
+0x82,
+0x06,
+0xc0,
+0x00,
+0x29,
+0xc0,
+0xb8,
+0x40,
+0x8a,
+0x40,
+0x00,
+0x00,
+0x40,
+0x00,
+0x57,
+0x02,
+0xc8,
+0x82,
+0x03,
+0x57,
+0x06,
+0x92,
+0x08,
+0x00,
+0x2e,
+0x16,
+0x31,
+0x08,
+0x4e,
+0x54,
+0x0e,
+0x81,
+0xa0,
+0x42,
+0x6c,
+0x40,
+0x01,
+0x3e,
+0x08,
+0x44,
+0x20,
+0x81,
+0x80,
+0x09,
+0x6e,
+0x40,
+0x01,
+0x9d,
+0x23,
+0x6c,
+0x40,
+0x03,
+0x24,
+0x0b,
+0x6c,
+0x40,
+0x01,
+0x6e,
+0x08,
+0x20,
+0x0c,
+0x94,
+0xc1,
+0x42,
+0x80,
+0x84,
+0x18,
+0x82,
+0x61,
+0x00,
+0x00,
+0x06,
+0xe4,
+0x00,
+0x19,
+0xf2,
+0x46,
+0xc4,
+0x00,
+0x25,
+0x60,
+0x92,
+0x01,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x0b,
+0x51,
+0x8b,
+0x48,
+0x48,
+0x40,
+0x30,
+0x07,
+0x86,
+0xe4,
+0x00,
+0x35,
+0x52,
+0x65,
+0x00,
+0xc8,
+0x3c,
+0x53,
+0xd8,
+0x60,
+0x40,
+0x22,
+0x96,
+0xc2,
+0xe1,
+0x3c,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x48,
+0x22,
+0x96,
+0x43,
+0x01,
+0x60,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x48,
+0xbc,
+0x45,
+0x46,
+0xc0,
+0x00,
+0x36,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x05,
+0x16,
+0xc0,
+0x00,
+0x37,
+0x60,
+0xa3,
+0x20,
+0x30,
+0x40,
+0x00,
+0x03,
+0xc3,
+0x90,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x0a,
+0x2e,
+0x9a,
+0x42,
+0xa0,
+0x64,
+0x6c,
+0x40,
+0x02,
+0x1c,
+0x0a,
+0x6c,
+0x00,
+0x03,
+0x68,
+0x48,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0xec,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x48,
+0x51,
+0x89,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x49,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x7a,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x51,
+0x8b,
+0x6b,
+0xc2,
+0xcf,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x49,
+0x68,
+0x00,
+0x00,
+0x70,
+0x25,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x24,
+0x82,
+0x87,
+0xa6,
+0x80,
+0x00,
+0x1b,
+0x12,
+0x68,
+0x20,
+0x7a,
+0xa2,
+0x80,
+0x38,
+0x28,
+0x7a,
+0xa2,
+0x00,
+0x78,
+0x58,
+0xfa,
+0x68,
+0x00,
+0x01,
+0xb8,
+0x22,
+0x82,
+0x07,
+0xa8,
+0x78,
+0xfa,
+0x6c,
+0x00,
+0x01,
+0x4c,
+0x7a,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x49,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x87,
+0x00,
+0x86,
+0xc4,
+0x00,
+0x3e,
+0x24,
+0x88,
+0x51,
+0xfa,
+0x87,
+0x1f,
+0xa8,
+0x68,
+0xfa,
+0x86,
+0x0f,
+0xa0,
+0x00,
+0x00,
+0x42,
+0x05,
+0x78,
+0x50,
+0x0a,
+0x85,
+0x8c,
+0xab,
+0xc0,
+0x8f,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x23,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0xa1,
+0x82,
+0x58,
+0x84,
+0x65,
+0x66,
+0x00,
+0x00,
+0xcd,
+0x80,
+0x68,
+0x00,
+0x00,
+0x89,
+0x20,
+0x88,
+0x2a,
+0x18,
+0x40,
+0x48,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x40,
+0x00,
+0x00,
+0x82,
+0x20,
+0x66,
+0x00,
+0x00,
+0xcd,
+0x80,
+0x68,
+0x00,
+0x00,
+0x92,
+0x20,
+0x68,
+0x00,
+0x00,
+0x89,
+0x21,
+0x5c,
+0x82,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x80,
+0x80,
+0x98,
+0x82,
+0x61,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x68,
+0x20,
+0x01,
+0xf1,
+0x2c,
+0x5c,
+0x87,
+0x00,
+0x82,
+0x20,
+0x88,
+0x1e,
+0xc8,
+0x02,
+0x48,
+0x68,
+0x20,
+0x00,
+0xdf,
+0x2c,
+0x68,
+0x20,
+0x00,
+0xdc,
+0x2d,
+0x68,
+0x20,
+0x00,
+0xde,
+0x22,
+0x88,
+0x26,
+0x06,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x46,
+0x80,
+0x00,
+0x0a,
+0x82,
+0x16,
+0x82,
+0x00,
+0x0d,
+0x32,
+0x58,
+0x80,
+0x6c,
+0x88,
+0x0e,
+0xd8,
+0x81,
+0x62,
+0x66,
+0x00,
+0x00,
+0x99,
+0x48,
+0x40,
+0x00,
+0x01,
+0x82,
+0x09,
+0x5c,
+0x82,
+0x00,
+0x82,
+0x20,
+0x68,
+0x00,
+0x00,
+0x92,
+0x21,
+0x84,
+0x34,
+0x86,
+0x80,
+0x00,
+0x07,
+0xa2,
+0x08,
+0x08,
+0x09,
+0x40,
+0x00,
+0x00,
+0x82,
+0x61,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x5c,
+0x87,
+0x00,
+0x82,
+0x20,
+0x88,
+0x42,
+0x18,
+0x02,
+0x48,
+0x88,
+0x1e,
+0x16,
+0x82,
+0x00,
+0x0d,
+0xfa,
+0xc6,
+0x82,
+0x00,
+0x0d,
+0xd2,
+0xd6,
+0x82,
+0x00,
+0x0d,
+0xea,
+0x28,
+0x82,
+0x60,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x00,
+0x00,
+0xaa,
+0x21,
+0x68,
+0x20,
+0x00,
+0xd5,
+0x25,
+0x88,
+0x06,
+0xc8,
+0x80,
+0xed,
+0x88,
+0x16,
+0x26,
+0x60,
+0x00,
+0x09,
+0x94,
+0x89,
+0x82,
+0x09,
+0x5c,
+0x82,
+0x00,
+0x83,
+0x21,
+0x68,
+0x00,
+0x01,
+0x49,
+0x20,
+0x84,
+0x80,
+0x98,
+0x82,
+0x21,
+0x80,
+0x04,
+0x98,
+0x82,
+0x60,
+0x40,
+0x00,
+0x00,
+0x4b,
+0x48,
+0x68,
+0x00,
+0x01,
+0x3a,
+0x20,
+0x68,
+0x20,
+0x01,
+0x8f,
+0x24,
+0x66,
+0x00,
+0x00,
+0x97,
+0xe0,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x59,
+0x03,
+0x40,
+0x82,
+0x21,
+0xbc,
+0x66,
+0x88,
+0x48,
+0x48,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc5,
+0xd1,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x2a,
+0x8e,
+0xc3,
+0x20,
+0xe0,
+0xbc,
+0x27,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xe0,
+0x86,
+0xc0,
+0x00,
+0x11,
+0x40,
+0xa2,
+0x81,
+0x30,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x4b,
+0x38,
+0x10,
+0x72,
+0x41,
+0xed,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x49,
+0x2e,
+0x13,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x06,
+0xd3,
+0x81,
+0x0d,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x24,
+0x16,
+0x56,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x90,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x2a,
+0x8e,
+0xd3,
+0x20,
+0xe8,
+0xbc,
+0x0f,
+0x93,
+0x90,
+0x20,
+0x68,
+0x00,
+0x00,
+0x70,
+0x21,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x09,
+0x80,
+0x87,
+0xaa,
+0x08,
+0x00,
+0x80,
+0x87,
+0xa8,
+0x40,
+0xfa,
+0x6c,
+0x40,
+0x01,
+0x5a,
+0x49,
+0x84,
+0x8f,
+0xab,
+0xc0,
+0x27,
+0x5c,
+0x81,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x38,
+0x18,
+0x62,
+0x89,
+0xa5,
+0x30,
+0x1a,
+0x8b,
+0xc2,
+0x70,
+0x6c,
+0x40,
+0x02,
+0x0e,
+0x0b,
+0x6c,
+0x00,
+0x01,
+0x26,
+0x00,
+0x28,
+0x1c,
+0x13,
+0x20,
+0x08,
+0xbc,
+0x04,
+0xb3,
+0x81,
+0x25,
+0x24,
+0x16,
+0x46,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x82,
+0xe1,
+0xc4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x38,
+0x12,
+0xf2,
+0x41,
+0xe4,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x08,
+0x28,
+0x9a,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x11,
+0x16,
+0x80,
+0x00,
+0x07,
+0xa2,
+0x16,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x88,
+0x08,
+0x7a,
+0xa0,
+0x80,
+0x08,
+0x08,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0xa4,
+0x88,
+0x48,
+0xfa,
+0xbc,
+0x05,
+0x7b,
+0xc0,
+0x4f,
+0x38,
+0x12,
+0x5b,
+0xc0,
+0x2f,
+0x38,
+0x12,
+0x53,
+0x81,
+0x25,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x21,
+0x6c,
+0x40,
+0x03,
+0xd2,
+0x08,
+0x84,
+0xb8,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0xb5,
+0x24,
+0xbe,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x7d,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x21,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x09,
+0x84,
+0xb8,
+0xb3,
+0x01,
+0x38,
+0xbc,
+0x05,
+0x93,
+0x81,
+0x2e,
+0x52,
+0x4d,
+0x6b,
+0xc0,
+0x5f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x24,
+0x1a,
+0xd6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x93,
+0x81,
+0x35,
+0x6c,
+0x40,
+0x01,
+0xb8,
+0x08,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x89,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x08,
+0x52,
+0x4b,
+0x28,
+0x83,
+0xb6,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x6f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x52,
+0x0b,
+0x28,
+0x83,
+0xb6,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x38,
+0x13,
+0xd6,
+0xc4,
+0x00,
+0x1b,
+0xa0,
+0x83,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x38,
+0x1d,
+0x56,
+0xc0,
+0x00,
+0x36,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x0a,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x86,
+0xc0,
+0x00,
+0x37,
+0x60,
+0xa3,
+0x20,
+0x30,
+0xbc,
+0x04,
+0x15,
+0x24,
+0xb2,
+0x3c,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x82,
+0x41,
+0x64,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xa0,
+0x86,
+0xc4,
+0x00,
+0x25,
+0x60,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x96,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x00,
+0x02,
+0x32,
+0x21,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x14,
+0xa6,
+0x1a,
+0x80,
+0x50,
+0x68,
+0x00,
+0x00,
+0x8a,
+0x20,
+0x5c,
+0x85,
+0x02,
+0xbf,
+0xa0,
+0x80,
+0x20,
+0x98,
+0x81,
+0x60,
+0x68,
+0x20,
+0x01,
+0xb4,
+0x2c,
+0x88,
+0x1f,
+0x66,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x46,
+0x82,
+0x00,
+0x0c,
+0xe2,
+0x56,
+0x80,
+0x00,
+0x1a,
+0x22,
+0x08,
+0x80,
+0x6c,
+0x66,
+0x00,
+0x00,
+0xb0,
+0x40,
+0x68,
+0x20,
+0x00,
+0xdc,
+0x2c,
+0x6c,
+0x00,
+0x03,
+0x44,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x60,
+0x00,
+0x09,
+0x82,
+0x89,
+0x82,
+0x09,
+0x68,
+0x00,
+0x00,
+0x93,
+0x20,
+0x5c,
+0x85,
+0x00,
+0x81,
+0x24,
+0x80,
+0x20,
+0x98,
+0x60,
+0x48,
+0x88,
+0x26,
+0x06,
+0x82,
+0x00,
+0x1b,
+0x62,
+0xc6,
+0x82,
+0x00,
+0x0b,
+0x42,
+0x46,
+0x82,
+0x00,
+0x0c,
+0xf2,
+0x56,
+0x80,
+0x00,
+0x1a,
+0x42,
+0x08,
+0x80,
+0x6c,
+0x66,
+0x00,
+0x00,
+0xb0,
+0x40,
+0x68,
+0x20,
+0x00,
+0xdd,
+0x2c,
+0x6c,
+0x00,
+0x03,
+0x48,
+0x48,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcf,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x60,
+0x00,
+0x09,
+0x82,
+0x89,
+0x82,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x20,
+0x88,
+0x22,
+0x48,
+0x02,
+0x09,
+0x86,
+0x04,
+0x88,
+0x81,
+0x60,
+0x68,
+0x00,
+0x00,
+0x70,
+0x21,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb1,
+0xa0,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x24,
+0x88,
+0x22,
+0x0a,
+0x20,
+0x44,
+0x80,
+0x20,
+0x98,
+0x60,
+0x48,
+0x88,
+0x2e,
+0x48,
+0x82,
+0x60,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb1,
+0xa0,
+0x88,
+0x22,
+0x06,
+0x82,
+0x00,
+0x09,
+0xc2,
+0x4a,
+0x00,
+0x40,
+0x86,
+0x60,
+0x36,
+0xc4,
+0x00,
+0x2a,
+0xc0,
+0x25,
+0x80,
+0x4c,
+0x04,
+0x04,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x08,
+0xbc,
+0x06,
+0xa5,
+0xc0,
+0x82,
+0x88,
+0x36,
+0x05,
+0x24,
+0xb2,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x92,
+0x41,
+0x65,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x20,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x09,
+0x84,
+0x60,
+0x33,
+0x00,
+0x98,
+0xbc,
+0x05,
+0xa3,
+0x81,
+0x0c,
+0x52,
+0x49,
+0x6b,
+0xc0,
+0x6f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x52,
+0x09,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x49,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x88,
+0x12,
+0x16,
+0x82,
+0x00,
+0x0d,
+0xe2,
+0x56,
+0x60,
+0x00,
+0x0b,
+0x70,
+0x88,
+0x49,
+0x09,
+0x88,
+0x22,
+0x06,
+0x80,
+0x00,
+0x0a,
+0x02,
+0x48,
+0x41,
+0x09,
+0x40,
+0x00,
+0x00,
+0x60,
+0x48,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xde,
+0xa5,
+0x66,
+0x00,
+0x00,
+0xb7,
+0x00,
+0x68,
+0x00,
+0x00,
+0xa0,
+0x20,
+0x68,
+0x20,
+0x00,
+0xd7,
+0x24,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x02,
+0x5c,
+0x83,
+0x04,
+0x00,
+0xc0,
+0x5d,
+0x0e,
+0x2a,
+0x00,
+0x03,
+0x51,
+0x85,
+0x68,
+0x00,
+0x48,
+0x98,
+0x26,
+0x88,
+0x82,
+0x64,
+0x9e,
+0x00,
+0x48,
+0x60,
+0x09,
+0x68,
+0x20,
+0x00,
+0xcc,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcd,
+0x21,
+0x86,
+0x04,
+0x98,
+0x48,
+0x49,
+0x88,
+0x3e,
+0x08,
+0x84,
+0x63,
+0x88,
+0x4d,
+0x08,
+0x81,
+0x48,
+0x66,
+0x00,
+0x00,
+0xc5,
+0x28,
+0x40,
+0x00,
+0x02,
+0x20,
+0x00,
+0x5c,
+0x09,
+0x28,
+0x84,
+0x20,
+0x88,
+0x3a,
+0x49,
+0x46,
+0x2a,
+0x52,
+0xca,
+0x82,
+0x24,
+0x84,
+0xbc,
+0x09,
+0x85,
+0xc8,
+0x30,
+0x08,
+0x3e,
+0x46,
+0x82,
+0x00,
+0x15,
+0x92,
+0x40,
+0x00,
+0x00,
+0x42,
+0x04,
+0x78,
+0x22,
+0x03,
+0x40,
+0x00,
+0x00,
+0x60,
+0x0b,
+0x68,
+0x20,
+0x01,
+0x5d,
+0x24,
+0x00,
+0x00,
+0x08,
+0x22,
+0x03,
+0x40,
+0x00,
+0x00,
+0x60,
+0x0b,
+0x5c,
+0x09,
+0x78,
+0x84,
+0x4b,
+0x25,
+0x9d,
+0x0b,
+0xc0,
+0x88,
+0x88,
+0x55,
+0x30,
+0x00,
+0x00,
+0x88,
+0x40,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x04,
+0x9b,
+0xc4,
+0x9f,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x49,
+0x25,
+0x95,
+0x0b,
+0xc3,
+0x01,
+0x6c,
+0x00,
+0x01,
+0x40,
+0x0a,
+0x66,
+0x00,
+0x00,
+0xca,
+0xc8,
+0x5c,
+0x00,
+0x28,
+0x40,
+0x0b,
+0x6c,
+0x40,
+0x01,
+0xae,
+0x09,
+0x5d,
+0x0a,
+0x28,
+0x82,
+0x20,
+0x23,
+0x0a,
+0xa9,
+0x84,
+0xa8,
+0x9c,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2a,
+0x80,
+0xb8,
+0x40,
+0x0a,
+0x08,
+0xb2,
+0x09,
+0x80,
+0x82,
+0x2e,
+0x0a,
+0x73,
+0x20,
+0x38,
+0xbc,
+0x03,
+0xd8,
+0x85,
+0x09,
+0xbc,
+0x02,
+0xf0,
+0x8d,
+0x20,
+0x98,
+0xe8,
+0x28,
+0x84,
+0x09,
+0x2e,
+0x0a,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0x00,
+0x99,
+0x80,
+0x82,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0xa4,
+0x6c,
+0x40,
+0x02,
+0xaa,
+0x09,
+0x2e,
+0x12,
+0xd3,
+0x20,
+0x28,
+0xbc,
+0x20,
+0x56,
+0xc4,
+0x00,
+0x18,
+0x05,
+0x26,
+0xc4,
+0x00,
+0x15,
+0x05,
+0x2b,
+0xc1,
+0xb7,
+0x6c,
+0x40,
+0x01,
+0x50,
+0x52,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x7f,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x52,
+0x59,
+0x01,
+0x03,
+0x01,
+0x05,
+0xbc,
+0x09,
+0x96,
+0xc4,
+0x00,
+0x3d,
+0x00,
+0x82,
+0x59,
+0x60,
+0xbc,
+0x0e,
+0x12,
+0x49,
+0x65,
+0x6c,
+0x40,
+0x03,
+0xd0,
+0x49,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x97,
+0x6c,
+0x40,
+0x02,
+0xbc,
+0x02,
+0x24,
+0x16,
+0x56,
+0xc4,
+0x00,
+0x3d,
+0x04,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x05,
+0x26,
+0xc4,
+0x00,
+0x18,
+0x05,
+0x23,
+0x90,
+0x20,
+0x88,
+0x3a,
+0x08,
+0x84,
+0x89,
+0x5b,
+0x4a,
+0x12,
+0x00,
+0x04,
+0x80,
+0x00,
+0x95,
+0xb4,
+0xa1,
+0x98,
+0x08,
+0x29,
+0x80,
+0xc3,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x0b,
+0x30,
+0x8d,
+0x0b,
+0xc0,
+0x6c,
+0x5c,
+0x08,
+0xa0,
+0x82,
+0x60,
+0x52,
+0x49,
+0xe3,
+0xc0,
+0x5f,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x24,
+0x13,
+0xc6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x83,
+0x81,
+0x1c,
+0x86,
+0x08,
+0x25,
+0xb4,
+0x41,
+0x08,
+0x10,
+0x35,
+0xb4,
+0x61,
+0x18,
+0x08,
+0xb9,
+0x80,
+0x82,
+0x30,
+0x9d,
+0x0b,
+0xc0,
+0xcc,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x02,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x52,
+0x48,
+0xa3,
+0xc0,
+0xcf,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x52,
+0x08,
+0xa3,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x68,
+0x20,
+0x00,
+0x9c,
+0x24,
+0x68,
+0x20,
+0x00,
+0xce,
+0x25,
+0x66,
+0x00,
+0x00,
+0xb7,
+0xe0,
+0x88,
+0x22,
+0x08,
+0x83,
+0xa4,
+0x88,
+0x2a,
+0x5a,
+0x00,
+0x68,
+0x86,
+0xb4,
+0x88,
+0x81,
+0x60,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x20,
+0x40,
+0x00,
+0x00,
+0x60,
+0x89,
+0x68,
+0x20,
+0x00,
+0xb4,
+0x24,
+0x68,
+0x20,
+0x00,
+0xcf,
+0x25,
+0x66,
+0x00,
+0x00,
+0xb7,
+0xe0,
+0x5c,
+0x09,
+0x68,
+0x82,
+0x20,
+0x88,
+0x32,
+0x59,
+0x46,
+0xab,
+0x52,
+0xca,
+0xc0,
+0x81,
+0x24,
+0x86,
+0xb4,
+0x84,
+0x20,
+0x9c,
+0xa2,
+0x40,
+0x85,
+0xc0,
+0x89,
+0x14,
+0x02,
+0xd3,
+0x81,
+0x0c,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x98,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x03,
+0x84,
+0x08,
+0xb5,
+0x40,
+0xed,
+0x84,
+0x00,
+0xb6,
+0xc0,
+0x00,
+0x11,
+0x80,
+0x82,
+0x81,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x43,
+0x25,
+0x8a,
+0x8b,
+0xc0,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x7a,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x7a,
+0x38,
+0x13,
+0x42,
+0x59,
+0x28,
+0xbc,
+0x15,
+0x06,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x95,
+0xc0,
+0xb9,
+0x98,
+0xe8,
+0x26,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x85,
+0x24,
+0x76,
+0xac,
+0x02,
+0x06,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x06,
+0xc4,
+0x00,
+0x25,
+0x07,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0xa4,
+0x86,
+0xc4,
+0x00,
+0x18,
+0xa4,
+0x86,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x94,
+0x21,
+0xff,
+0x94,
+0x24,
+0x28,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x88,
+0x20,
+0x68,
+0x00,
+0x00,
+0x91,
+0x21,
+0x66,
+0x00,
+0x01,
+0x59,
+0xc0,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x32,
+0x06,
+0x8b,
+0xc1,
+0x61,
+0x68,
+0x20,
+0x01,
+0x00,
+0x21,
+0x68,
+0x20,
+0x00,
+0xad,
+0x20,
+0xa0,
+0xc2,
+0x28,
+0x81,
+0x62,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x40,
+0x00,
+0x00,
+0x81,
+0x22,
+0x68,
+0x20,
+0x00,
+0xc5,
+0x20,
+0x68,
+0x20,
+0x01,
+0x00,
+0x21,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x38,
+0x09,
+0x40,
+0x00,
+0x03,
+0xc2,
+0x0f,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x49,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x09,
+0x38,
+0x19,
+0xc2,
+0x89,
+0x2d,
+0x30,
+0x12,
+0x8b,
+0xc1,
+0x81,
+0x68,
+0x20,
+0x00,
+0xad,
+0x20,
+0x68,
+0x20,
+0x01,
+0x01,
+0x21,
+0x68,
+0x20,
+0x00,
+0xfe,
+0x22,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x68,
+0x20,
+0x00,
+0xfe,
+0x22,
+0x68,
+0x20,
+0x00,
+0xc5,
+0x20,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x48,
+0x40,
+0x00,
+0x02,
+0x10,
+0x61,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x09,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x40,
+0x55,
+0x3f,
+0x6b,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x50,
+0x49,
+0x68,
+0x00,
+0x01,
+0x3e,
+0x20,
+0x88,
+0x1b,
+0x6b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x60,
+0xab,
+0xfd,
+0x06,
+0x82,
+0x00,
+0x1d,
+0xa2,
+0xc6,
+0x82,
+0x00,
+0x1d,
+0xb2,
+0xd6,
+0x82,
+0x00,
+0x1d,
+0xc2,
+0x26,
+0x82,
+0x00,
+0x1f,
+0x32,
+0x38,
+0x82,
+0x76,
+0x68,
+0x00,
+0x01,
+0x44,
+0x20,
+0x68,
+0x20,
+0x01,
+0x8f,
+0x24,
+0x68,
+0x00,
+0x01,
+0x52,
+0x21,
+0x68,
+0x20,
+0x01,
+0xd8,
+0x25,
+0x88,
+0x06,
+0xc8,
+0x80,
+0xed,
+0x88,
+0x16,
+0x28,
+0x81,
+0xe3,
+0x66,
+0x00,
+0x00,
+0x99,
+0x48,
+0x6c,
+0x00,
+0x02,
+0x96,
+0x09,
+0x68,
+0x00,
+0x02,
+0x3b,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x68,
+0x00,
+0x01,
+0x44,
+0x24,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0x86,
+0x34,
+0x8a,
+0x80,
+0x30,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x09,
+0x58,
+0x0b,
+0x02,
+0xbf,
+0xe0,
+0xbc,
+0x7b,
+0x88,
+0x80,
+0x76,
+0x68,
+0x20,
+0x01,
+0xb1,
+0x24,
+0x68,
+0x00,
+0x01,
+0x9c,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x6c,
+0x00,
+0x01,
+0x16,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x64,
+0x48,
+0x68,
+0x20,
+0x01,
+0xb1,
+0x24,
+0x68,
+0x00,
+0x01,
+0xa0,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x6c,
+0x00,
+0x01,
+0x28,
+0x09,
+0x68,
+0x20,
+0x01,
+0x1b,
+0x20,
+0x68,
+0x20,
+0x01,
+0xf5,
+0x24,
+0x5c,
+0x81,
+0x02,
+0x05,
+0xc1,
+0x84,
+0x00,
+0x98,
+0x48,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x08,
+0x01,
+0x6c,
+0x00,
+0x01,
+0x1a,
+0x0a,
+0x30,
+0x02,
+0x8b,
+0xc4,
+0x3d,
+0x6c,
+0x00,
+0x03,
+0x72,
+0x48,
+0x5c,
+0x83,
+0x0a,
+0x20,
+0x01,
+0x80,
+0xa0,
+0x88,
+0x0a,
+0x8b,
+0x49,
+0xb8,
+0xe2,
+0x05,
+0x60,
+0x22,
+0xfd,
+0x42,
+0xa8,
+0x64,
+0x28,
+0x11,
+0x22,
+0x28,
+0x54,
+0x5b,
+0x40,
+0x00,
+0x48,
+0x48,
+0xa0,
+0x8c,
+0x18,
+0x0a,
+0x08,
+0x84,
+0x80,
+0x95,
+0x40,
+0xb1,
+0x18,
+0x00,
+0x05,
+0x17,
+0xeb,
+0xa0,
+0xc8,
+0x12,
+0xa8,
+0x7b,
+0x54,
+0x06,
+0x98,
+0x40,
+0x02,
+0x6c,
+0x40,
+0x02,
+0x36,
+0x7a,
+0x30,
+0x08,
+0x05,
+0x14,
+0x2f,
+0xbc,
+0x21,
+0xb8,
+0x08,
+0xcb,
+0x2e,
+0x16,
+0x43,
+0x69,
+0x00,
+0x98,
+0x00,
+0x83,
+0x00,
+0xa0,
+0xbc,
+0x1b,
+0x38,
+0x80,
+0xca,
+0x88,
+0x16,
+0x18,
+0x81,
+0xd1,
+0x66,
+0x00,
+0x02,
+0x4d,
+0x28,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x08,
+0x6e,
+0x00,
+0x03,
+0x4c,
+0x12,
+0x66,
+0x00,
+0x02,
+0x52,
+0xa8,
+0x55,
+0x00,
+0x0b,
+0x80,
+0x00,
+0x66,
+0x00,
+0x02,
+0x4f,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x09,
+0x57,
+0x4b,
+0x20,
+0x81,
+0x21,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x48,
+0x5c,
+0x81,
+0x02,
+0xc0,
+0x61,
+0x88,
+0x18,
+0x18,
+0x80,
+0x8a,
+0x68,
+0x20,
+0x01,
+0xf5,
+0x24,
+0x80,
+0xa5,
+0x18,
+0x08,
+0xd1,
+0x42,
+0x01,
+0x78,
+0x0a,
+0x4a,
+0x84,
+0x84,
+0xa3,
+0x90,
+0x61,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3e,
+0xa0,
+0x82,
+0xe8,
+0x64,
+0x82,
+0x24,
+0x80,
+0x00,
+0x00,
+0x86,
+0x00,
+0x85,
+0x78,
+0x32,
+0x06,
+0x18,
+0x95,
+0x74,
+0xd6,
+0x02,
+0x0c,
+0x88,
+0x22,
+0x48,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x23,
+0x60,
+0x85,
+0x50,
+0x32,
+0x06,
+0x00,
+0x92,
+0xf1,
+0xad,
+0x6c,
+0x40,
+0x02,
+0x36,
+0x48,
+0x40,
+0x00,
+0x00,
+0x60,
+0x49,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4d,
+0x60,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x1b,
+0x92,
+0x0b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x01,
+0x4a,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x20,
+0xa2,
+0x28,
+0x46,
+0x40,
+0x00,
+0x13,
+0x26,
+0xfa,
+0x01,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x65,
+0x88,
+0x0f,
+0x66,
+0x60,
+0x00,
+0x13,
+0x34,
+0x8a,
+0x20,
+0x84,
+0x88,
+0x02,
+0x08,
+0x81,
+0x24,
+0x94,
+0x02,
+0x55,
+0x00,
+0xb0,
+0x08,
+0x0b,
+0x68,
+0x60,
+0x09,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x08,
+0x44,
+0x08,
+0x02,
+0x80,
+0x10,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0xab,
+0xfd,
+0x08,
+0x81,
+0xe4,
+0x88,
+0x04,
+0x98,
+0x80,
+0xe5,
+0x88,
+0x16,
+0x18,
+0x82,
+0x60,
+0x88,
+0x2f,
+0x66,
+0x82,
+0x00,
+0x0d,
+0x02,
+0x46,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x08,
+0x00,
+0x88,
+0x22,
+0x06,
+0xc4,
+0x00,
+0x1a,
+0xe0,
+0x95,
+0xd0,
+0xa3,
+0x04,
+0x00,
+0x95,
+0x70,
+0xb0,
+0x04,
+0x1c,
+0x85,
+0x18,
+0x5b,
+0x18,
+0x00,
+0x86,
+0x82,
+0x00,
+0x12,
+0xd2,
+0x15,
+0xb4,
+0x80,
+0x18,
+0x2a,
+0x89,
+0x80,
+0x0a,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x00,
+0x58,
+0x41,
+0x80,
+0x81,
+0xa4,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0xa2,
+0x08,
+0x36,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0x28,
+0x84,
+0x21,
+0xa1,
+0x86,
+0x44,
+0x20,
+0x25,
+0x21,
+0x8a,
+0x29,
+0x48,
+0x3f,
+0x38,
+0x12,
+0x92,
+0x58,
+0x70,
+0xbc,
+0x77,
+0x08,
+0x83,
+0x26,
+0x88,
+0x0a,
+0x79,
+0x70,
+0x2c,
+0x59,
+0x03,
+0x00,
+0x81,
+0x25,
+0xbc,
+0x11,
+0x98,
+0x78,
+0x7a,
+0x00,
+0x00,
+0x08,
+0x68,
+0x88,
+0x30,
+0x16,
+0x0b,
+0xc0,
+0x7b,
+0x84,
+0x38,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x09,
+0x49,
+0x8e,
+0x88,
+0x86,
+0x8c,
+0x99,
+0x70,
+0x64,
+0xbc,
+0x05,
+0x73,
+0x01,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0x86,
+0x8c,
+0x99,
+0x70,
+0x64,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3d,
+0x20,
+0x88,
+0x59,
+0x89,
+0x30,
+0x12,
+0x8b,
+0xc2,
+0x48,
+0x39,
+0x02,
+0x06,
+0x82,
+0x00,
+0x1e,
+0xe2,
+0x75,
+0xc8,
+0x20,
+0x85,
+0xa8,
+0x95,
+0xc8,
+0x51,
+0x03,
+0x80,
+0xa5,
+0x70,
+0xd4,
+0x21,
+0x00,
+0x65,
+0x90,
+0x18,
+0x01,
+0x2d,
+0x00,
+0x00,
+0x00,
+0x83,
+0xb0,
+0x98,
+0x50,
+0x0a,
+0x57,
+0x0b,
+0xb0,
+0x78,
+0x09,
+0x85,
+0x04,
+0xab,
+0xc0,
+0x6c,
+0x85,
+0xa8,
+0xa3,
+0x01,
+0x70,
+0xbc,
+0x08,
+0x28,
+0x59,
+0xc8,
+0x86,
+0x14,
+0x9b,
+0xc0,
+0x57,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x34,
+0x85,
+0x9c,
+0x88,
+0x61,
+0x49,
+0x00,
+0x00,
+0x08,
+0x50,
+0x09,
+0x57,
+0x0a,
+0x80,
+0x32,
+0x09,
+0x57,
+0x0a,
+0x00,
+0x83,
+0xa4,
+0xbc,
+0x03,
+0xf8,
+0x70,
+0x40,
+0x88,
+0x3a,
+0x40,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x60,
+0x09,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0xb3,
+0x6c,
+0x40,
+0x02,
+0x30,
+0x08,
+0x28,
+0x12,
+0xd3,
+0x08,
+0xa8,
+0xbc,
+0x03,
+0x4b,
+0xc0,
+0x6f,
+0x5c,
+0x0b,
+0xe8,
+0x60,
+0x49,
+0xbc,
+0x03,
+0xf5,
+0xc0,
+0xbe,
+0x86,
+0x05,
+0x23,
+0x81,
+0x7d,
+0x25,
+0x97,
+0x8b,
+0xc2,
+0x01,
+0x38,
+0x16,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x0f,
+0x93,
+0x81,
+0x75,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x49,
+0x98,
+0xe8,
+0x98,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x38,
+0x17,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x13,
+0x08,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x03,
+0x94,
+0x87,
+0x5b,
+0xc0,
+0xe7,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x49,
+0x38,
+0x16,
+0x58,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x62,
+0x38,
+0x17,
+0x42,
+0x59,
+0x38,
+0xbc,
+0x05,
+0x08,
+0x43,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x25,
+0x24,
+0x17,
+0x89,
+0x48,
+0x70,
+0xbc,
+0x42,
+0xf8,
+0x80,
+0xa2,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x6c,
+0x88,
+0x12,
+0x55,
+0x70,
+0x14,
+0x08,
+0x32,
+0x3b,
+0xc0,
+0x5f,
+0x40,
+0x00,
+0x00,
+0x68,
+0xc0,
+0x54,
+0x0a,
+0x00,
+0x83,
+0x23,
+0x86,
+0x8c,
+0x0b,
+0x00,
+0x0d,
+0x88,
+0x0a,
+0x29,
+0x58,
+0x65,
+0x00,
+0x00,
+0x08,
+0x50,
+0x09,
+0x30,
+0x0a,
+0x8b,
+0xc0,
+0x23,
+0x2a,
+0x06,
+0xd8,
+0x50,
+0x49,
+0x68,
+0x20,
+0x01,
+0xeb,
+0x26,
+0x5c,
+0x81,
+0x0b,
+0x01,
+0x7d,
+0x52,
+0xcb,
+0xc0,
+0x30,
+0x88,
+0x5c,
+0x82,
+0x00,
+0x70,
+0x00,
+0x82,
+0x0c,
+0x88,
+0x60,
+0x50,
+0x68,
+0x20,
+0x01,
+0x19,
+0x23,
+0x87,
+0x08,
+0xb8,
+0x83,
+0xa7,
+0x81,
+0xa0,
+0x04,
+0x20,
+0x8c,
+0x06,
+0x0c,
+0xb5,
+0xc0,
+0x02,
+0x07,
+0x85,
+0x02,
+0x41,
+0x64,
+0x5d,
+0x48,
+0x20,
+0x43,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x6c,
+0x5c,
+0x0b,
+0xa9,
+0x48,
+0x74,
+0x52,
+0x4b,
+0x03,
+0xc0,
+0x7f,
+0x5c,
+0x09,
+0x29,
+0x48,
+0x70,
+0x52,
+0x0b,
+0x03,
+0xc0,
+0x3f,
+0x5c,
+0x09,
+0x29,
+0x48,
+0x70,
+0x38,
+0x12,
+0x52,
+0x59,
+0x70,
+0xbc,
+0x08,
+0x88,
+0x58,
+0x52,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2a,
+0xe0,
+0x96,
+0xc4,
+0x00,
+0x15,
+0x04,
+0x96,
+0xc4,
+0x00,
+0x18,
+0x04,
+0x99,
+0x88,
+0x09,
+0x68,
+0x00,
+0x00,
+0x84,
+0x08,
+0x58,
+0x09,
+0x40,
+0x68,
+0x89,
+0x84,
+0x1c,
+0x9b,
+0xc1,
+0x48,
+0x85,
+0x00,
+0x99,
+0x88,
+0x08,
+0x68,
+0x00,
+0x00,
+0x8d,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x98,
+0xea,
+0x16,
+0x60,
+0x00,
+0x0b,
+0x8a,
+0x08,
+0x82,
+0x20,
+0xbc,
+0x10,
+0xf8,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xb5,
+0x21,
+0x66,
+0x00,
+0x00,
+0xb8,
+0xa0,
+0x88,
+0x22,
+0x0b,
+0xc0,
+0x9f,
+0x40,
+0x00,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xae,
+0x21,
+0x66,
+0x00,
+0x00,
+0xb8,
+0xa0,
+0x88,
+0x22,
+0x00,
+0x00,
+0x00,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x56,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x51,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x38,
+0x12,
+0xc2,
+0x59,
+0x28,
+0xbc,
+0x05,
+0x08,
+0x84,
+0xa1,
+0x40,
+0x00,
+0x00,
+0x40,
+0x09,
+0x42,
+0x29,
+0xb8,
+0x48,
+0x49,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0xb0,
+0x84,
+0x18,
+0x88,
+0x40,
+0x0a,
+0x57,
+0x0d,
+0x00,
+0x84,
+0xa1,
+0x98,
+0x00,
+0xa3,
+0x69,
+0x80,
+0x98,
+0x00,
+0xa3,
+0x01,
+0x70,
+0xbc,
+0x03,
+0x58,
+0x48,
+0x48,
+0xbc,
+0x01,
+0x78,
+0x84,
+0xa1,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x09,
+0x84,
+0x80,
+0x85,
+0x40,
+0xb0,
+0x04,
+0x00,
+0xa9,
+0x80,
+0x00,
+0x30,
+0x03,
+0x0b,
+0xc0,
+0x2d,
+0x88,
+0x0c,
+0x88,
+0x40,
+0x50,
+0x2e,
+0x16,
+0x08,
+0x40,
+0x09,
+0x98,
+0x00,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x01,
+0x38,
+0x40,
+0x48,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x88,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x56,
+0x08,
+0x68,
+0x00,
+0x2b,
+0xf2,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0xe4,
+0x68,
+0x00,
+0x27,
+0x10,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0x84,
+0x68,
+0x00,
+0x18,
+0x6a,
+0x0a,
+0x30,
+0x9a,
+0x0b,
+0xc0,
+0x24,
+0xbc,
+0x08,
+0xf2,
+0x28,
+0x6d,
+0xbc,
+0x06,
+0xf2,
+0x28,
+0xad,
+0xbc,
+0x04,
+0xf2,
+0x28,
+0xed,
+0xbc,
+0x02,
+0xf2,
+0x29,
+0x2d,
+0x22,
+0x96,
+0xd8,
+0x84,
+0xa1,
+0x88,
+0x22,
+0x08,
+0x48,
+0x08,
+0x84,
+0x00,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x03,
+0x38,
+0x80,
+0x88,
+0x2e,
+0x16,
+0x08,
+0x48,
+0x40,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x84,
+0x80,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x04,
+0xd8,
+0x80,
+0x88,
+0x28,
+0x16,
+0x08,
+0x48,
+0x40,
+0x00,
+0x00,
+0x08,
+0x82,
+0xb6,
+0x84,
+0x00,
+0x84,
+0x60,
+0xa4,
+0x08,
+0x00,
+0x95,
+0x70,
+0x94,
+0x28,
+0x03,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x09,
+0xe2,
+0x22,
+0xe4,
+0x86,
+0x00,
+0xa4,
+0x43,
+0x00,
+0x16,
+0x9a,
+0x55,
+0x00,
+0xa0,
+0x08,
+0x02,
+0x56,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xd5,
+0x2c,
+0x94,
+0x06,
+0x80,
+0x94,
+0x20,
+0x24,
+0x18,
+0x00,
+0x80,
+0x81,
+0x00,
+0xba,
+0x14,
+0x89,
+0x80,
+0x08,
+0x00,
+0x00,
+0x08,
+0x40,
+0x89,
+0x46,
+0x0a,
+0x40,
+0x68,
+0x88,
+0x08,
+0x48,
+0x09,
+0x80,
+0x08,
+0xab,
+0xfd,
+0x0a,
+0x21,
+0x64,
+0xa0,
+0x88,
+0x18,
+0x80,
+0x64,
+0xa2,
+0x06,
+0x48,
+0x80,
+0xe1,
+0xa0,
+0x84,
+0x18,
+0x81,
+0x64,
+0x88,
+0x1c,
+0x98,
+0x82,
+0x60,
+0x88,
+0x2f,
+0x66,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x08,
+0x00,
+0x88,
+0x12,
+0x08,
+0x82,
+0x21,
+0xa0,
+0x06,
+0x08,
+0x40,
+0x0a,
+0x88,
+0x18,
+0x94,
+0x43,
+0x00,
+0x04,
+0xa4,
+0x8a,
+0x06,
+0x24,
+0x66,
+0x00,
+0x00,
+0xb5,
+0xe8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x09,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x25,
+0x59,
+0x03,
+0x40,
+0x80,
+0xa4,
+0xbc,
+0x15,
+0x8a,
+0x20,
+0x20,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2d,
+0x5c,
+0x09,
+0x20,
+0x80,
+0xc8,
+0x25,
+0x92,
+0x8b,
+0xc0,
+0xc8,
+0x88,
+0x02,
+0x46,
+0xc4,
+0x00,
+0x2a,
+0xc0,
+0x96,
+0x60,
+0x00,
+0x15,
+0x82,
+0x88,
+0x60,
+0x8a,
+0x55,
+0x01,
+0x28,
+0x80,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0xc9,
+0xbc,
+0x0f,
+0xf8,
+0x80,
+0x88,
+0xbc,
+0x0d,
+0xf8,
+0x80,
+0x88,
+0x86,
+0x08,
+0x93,
+0x69,
+0x40,
+0x98,
+0x00,
+0x96,
+0xc4,
+0x00,
+0x21,
+0x00,
+0xa3,
+0x09,
+0x70,
+0xbc,
+0x05,
+0x58,
+0x80,
+0x20,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x09,
+0x84,
+0x0c,
+0x90,
+0x00,
+0x00,
+0x88,
+0x2b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2c,
+0x38,
+0x12,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x09,
+0x08,
+0x63,
+0x88,
+0x6c,
+0x40,
+0x03,
+0xd2,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x40,
+0x6c,
+0x40,
+0x02,
+0x2e,
+0x08,
+0x08,
+0x40,
+0x09,
+0x80,
+0x09,
+0xba,
+0x14,
+0x89,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x80,
+0xa2,
+0x16,
+0x48,
+0x20,
+0x0a,
+0x84,
+0x00,
+0x84,
+0x41,
+0x00,
+0x02,
+0x00,
+0xa8,
+0x40,
+0x88,
+0x44,
+0x14,
+0x00,
+0x60,
+0x08,
+0xba,
+0x14,
+0x84,
+0x42,
+0x40,
+0x04,
+0x04,
+0x99,
+0x80,
+0x08,
+0xa2,
+0x24,
+0x48,
+0x60,
+0x08,
+0x44,
+0x20,
+0x00,
+0x42,
+0xc9,
+0x98,
+0x00,
+0x98,
+0x60,
+0x88,
+0x44,
+0x20,
+0x01,
+0x68,
+0xa5,
+0xba,
+0x14,
+0x82,
+0x01,
+0x40,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x5c,
+0x83,
+0x08,
+0x43,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x14,
+0x0a,
+0x6c,
+0x40,
+0x03,
+0xe0,
+0x00,
+0x58,
+0x0c,
+0x02,
+0xbf,
+0xd0,
+0x80,
+0x08,
+0xba,
+0x08,
+0x42,
+0x84,
+0x00,
+0x19,
+0x88,
+0x42,
+0x88,
+0x0c,
+0x88,
+0x81,
+0x49,
+0x42,
+0x13,
+0x40,
+0x80,
+0x4a,
+0x88,
+0x1c,
+0xb3,
+0x20,
+0x10,
+0xbc,
+0x21,
+0x06,
+0x82,
+0x00,
+0x11,
+0xb2,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x5c,
+0x38,
+0x40,
+0x08,
+0x85,
+0x80,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x10,
+0xd8,
+0x4a,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x48,
+0x8a,
+0x84,
+0x80,
+0x32,
+0x80,
+0xf4,
+0x22,
+0xfe,
+0x75,
+0x54,
+0x3c,
+0x98,
+0x44,
+0xb1,
+0x51,
+0x73,
+0x51,
+0x42,
+0x60,
+0x10,
+0x53,
+0x81,
+0x2c,
+0x84,
+0x20,
+0x1f,
+0x85,
+0x04,
+0x95,
+0x50,
+0x1c,
+0x85,
+0x0c,
+0x93,
+0x90,
+0x20,
+0x00,
+0x00,
+0x08,
+0x48,
+0x88,
+0x57,
+0x4b,
+0x30,
+0x48,
+0x08,
+0x57,
+0x8b,
+0x20,
+0x48,
+0xca,
+0xbc,
+0x04,
+0xf8,
+0x48,
+0x48,
+0xbc,
+0x02,
+0xf3,
+0x90,
+0x20,
+0x39,
+0x02,
+0x03,
+0x81,
+0x25,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2c,
+0x25,
+0x96,
+0x06,
+0xc4,
+0x00,
+0x20,
+0xa0,
+0xab,
+0xc0,
+0x59,
+0x55,
+0x01,
+0xa0,
+0x82,
+0x4a,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x24,
+0xe0,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x56,
+0x09,
+0x30,
+0x17,
+0x0b,
+0xc0,
+0x88,
+0x88,
+0x10,
+0xab,
+0x1f,
+0x47,
+0x30,
+0x1f,
+0x0b,
+0xc1,
+0x2c,
+0x98,
+0xe8,
+0x88,
+0x82,
+0x08,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xe7,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xae,
+0x38,
+0x13,
+0x72,
+0x59,
+0xf0,
+0xbc,
+0x09,
+0x16,
+0xc4,
+0x00,
+0x15,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x2b,
+0xc0,
+0xb3,
+0x01,
+0xf0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x23,
+0x6c,
+0x40,
+0x02,
+0x4e,
+0x08,
+0x6e,
+0x00,
+0x01,
+0xbe,
+0x26,
+0x59,
+0x03,
+0x80,
+0x81,
+0x8a,
+0x28,
+0x13,
+0x35,
+0x70,
+0x99,
+0x98,
+0x0c,
+0x8b,
+0xc0,
+0xf8,
+0x98,
+0x0c,
+0xb6,
+0xc4,
+0x00,
+0x25,
+0x00,
+0x33,
+0x20,
+0x18,
+0xbc,
+0x0a,
+0x13,
+0x01,
+0x88,
+0xbc,
+0x06,
+0x45,
+0x80,
+0xc4,
+0x08,
+0x18,
+0x8b,
+0xc0,
+0xfb,
+0x88,
+0x00,
+0x19,
+0x82,
+0xc8,
+0xbc,
+0x0c,
+0x7b,
+0xc0,
+0xbf,
+0x88,
+0x00,
+0x18,
+0x80,
+0x81,
+0x32,
+0x00,
+0x8b,
+0xc0,
+0x64,
+0x32,
+0x00,
+0x8b,
+0xc0,
+0x5b,
+0x55,
+0x01,
+0xa0,
+0x80,
+0x01,
+0x98,
+0x2c,
+0x8b,
+0xc0,
+0x17,
+0x88,
+0x00,
+0x13,
+0x00,
+0x40,
+0xbc,
+0x44,
+0x03,
+0x20,
+0x10,
+0xbc,
+0x42,
+0x06,
+0xc0,
+0x00,
+0x36,
+0x80,
+0xa3,
+0x20,
+0x30,
+0xbc,
+0x05,
+0x98,
+0x82,
+0x0a,
+0x6c,
+0x00,
+0x03,
+0x76,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc3,
+0x90,
+0x30,
+0x17,
+0x0b,
+0xc3,
+0x70,
+0x68,
+0x20,
+0x01,
+0x0b,
+0x20,
+0x84,
+0x90,
+0xa8,
+0x00,
+0x00,
+0x30,
+0x03,
+0x0b,
+0xc0,
+0x22,
+0xbc,
+0x04,
+0xf8,
+0x4a,
+0xfa,
+0x84,
+0xa8,
+0xa2,
+0xa0,
+0x76,
+0x84,
+0xac,
+0xa0,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa6,
+0xc0,
+0x00,
+0x36,
+0x60,
+0x03,
+0x01,
+0x80,
+0xbc,
+0x04,
+0x56,
+0xc0,
+0x00,
+0x37,
+0x40,
+0x03,
+0x01,
+0x80,
+0xbc,
+0x03,
+0x4b,
+0xc2,
+0x1f,
+0x40,
+0x00,
+0x00,
+0x4a,
+0x08,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x5c,
+0x82,
+0x08,
+0x08,
+0x7a,
+0x80,
+0x07,
+0xaa,
+0x08,
+0x02,
+0x80,
+0x87,
+0xaa,
+0x00,
+0x04,
+0x80,
+0x07,
+0xa8,
+0x50,
+0xfa,
+0x86,
+0x0f,
+0xa8,
+0x48,
+0xfa,
+0x68,
+0x00,
+0x01,
+0xb9,
+0x23,
+0x84,
+0x0f,
+0xa6,
+0x80,
+0x00,
+0x1b,
+0x22,
+0x16,
+0x82,
+0x00,
+0x1f,
+0x12,
+0x08,
+0x08,
+0x80,
+0x85,
+0x80,
+0xa6,
+0xc4,
+0x00,
+0x20,
+0xa4,
+0x96,
+0xc0,
+0x00,
+0x14,
+0xc7,
+0xa8,
+0x48,
+0x7a,
+0x85,
+0x97,
+0xa8,
+0x40,
+0x50,
+0x84,
+0x0c,
+0xab,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x85,
+0x00,
+0x88,
+0x40,
+0x09,
+0x57,
+0x8b,
+0x22,
+0xbf,
+0xd0,
+0x85,
+0x04,
+0x88,
+0x80,
+0x63,
+0x88,
+0x0e,
+0x28,
+0x81,
+0x61,
+0x88,
+0x1e,
+0x08,
+0x82,
+0x76,
+0x00,
+0x00,
+0x08,
+0x40,
+0x09,
+0x36,
+0x14,
+0x52,
+0xe9,
+0x64,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x85,
+0x80,
+0x88,
+0x48,
+0x09,
+0x2f,
+0x16,
+0x48,
+0x58,
+0x48,
+0x00,
+0x00,
+0x08,
+0x48,
+0x09,
+0x36,
+0x14,
+0x52,
+0xe9,
+0x64,
+0x44,
+0x00,
+0x00,
+0x58,
+0x48,
+0x98,
+0x00,
+0x98,
+0x50,
+0x08,
+0x44,
+0x00,
+0x00,
+0x48,
+0x08,
+0x44,
+0x01,
+0x00,
+0x40,
+0x08,
+0x44,
+0x01,
+0x01,
+0xa0,
+0x00,
+0x44,
+0x40,
+0x01,
+0x80,
+0x88,
+0x08,
+0x18,
+0x06,
+0x60,
+0x00,
+0x13,
+0x52,
+0x89,
+0x81,
+0x09,
+0x88,
+0x12,
+0x08,
+0x81,
+0xa1,
+0x84,
+0x00,
+0x98,
+0x48,
+0x0a,
+0x44,
+0x48,
+0x00,
+0x80,
+0xa0,
+0x88,
+0x02,
+0x19,
+0x80,
+0x0b,
+0x84,
+0x00,
+0x94,
+0x46,
+0x80,
+0x04,
+0x80,
+0xa4,
+0x47,
+0x10,
+0x18,
+0x00,
+0xb4,
+0x46,
+0x00,
+0x18,
+0x28,
+0xb5,
+0x11,
+0xe0,
+0x18,
+0x08,
+0xa4,
+0x44,
+0x10,
+0x18,
+0x00,
+0x83,
+0x20,
+0x20,
+0x51,
+0x1e,
+0x83,
+0xc0,
+0x29,
+0x98,
+0x00,
+0xa9,
+0x82,
+0x48,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x11,
+0x98,
+0x2c,
+0xa3,
+0x69,
+0x40,
+0x5b,
+0x48,
+0x01,
+0x80,
+0x09,
+0x98,
+0x00,
+0x03,
+0x00,
+0x28,
+0xbc,
+0x06,
+0x43,
+0x69,
+0xc0,
+0x5b,
+0x4c,
+0x01,
+0x80,
+0x09,
+0x98,
+0x00,
+0xb3,
+0x01,
+0xe8,
+0xbc,
+0x04,
+0x58,
+0x40,
+0x48,
+0x42,
+0x01,
+0x7b,
+0x00,
+0x0c,
+0x84,
+0x84,
+0xa9,
+0x8e,
+0x88,
+0x00,
+0x00,
+0x08,
+0x82,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x46,
+0x08,
+0x8a,
+0xbf,
+0xe0,
+0x51,
+0x61,
+0xb0,
+0x80,
+0x49,
+0x51,
+0x61,
+0xe0,
+0x80,
+0xf6,
+0x1a,
+0x0a,
+0x29,
+0x83,
+0x09,
+0x98,
+0x38,
+0x82,
+0x81,
+0x2d,
+0x88,
+0x14,
+0x96,
+0x60,
+0x00,
+0x13,
+0x52,
+0x8b,
+0xa1,
+0x01,
+0x88,
+0x10,
+0x94,
+0x40,
+0x80,
+0x08,
+0x00,
+0x99,
+0x80,
+0x08,
+0x23,
+0x42,
+0x43,
+0x01,
+0x60,
+0xbc,
+0x02,
+0x5b,
+0xc0,
+0x2f,
+0x2e,
+0x16,
+0x49,
+0x8e,
+0x88,
+0x88,
+0x0b,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x70,
+0x20,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0x68,
+0x00,
+0x00,
+0x7a,
+0x21,
+0x68,
+0x00,
+0x01,
+0x3a,
+0x22,
+0x55,
+0x03,
+0x20,
+0x00,
+0x7a,
+0x5d,
+0x08,
+0x20,
+0x08,
+0x7a,
+0x32,
+0x2a,
+0x0b,
+0xff,
+0xaa,
+0x81,
+0x07,
+0xab,
+0xa1,
+0x40,
+0x5c,
+0x81,
+0x00,
+0x40,
+0x08,
+0x82,
+0x00,
+0xa4,
+0x41,
+0x00,
+0x04,
+0x80,
+0xa8,
+0x20,
+0x08,
+0x44,
+0x44,
+0x00,
+0x60,
+0x08,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x0a,
+0x08,
+0x80,
+0x09,
+0x80,
+0x08,
+0x68,
+0x00,
+0x00,
+0x6c,
+0x20,
+0x68,
+0x20,
+0x00,
+0x24,
+0x24,
+0x5c,
+0x81,
+0x02,
+0xc0,
+0x41,
+0x68,
+0x20,
+0x00,
+0x4b,
+0x21,
+0x5c,
+0x08,
+0xad,
+0x00,
+0x81,
+0x44,
+0x10,
+0x00,
+0x08,
+0x88,
+0x84,
+0x00,
+0xb4,
+0x46,
+0x08,
+0x2b,
+0xf7,
+0x06,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xe6,
+0xe4,
+0x00,
+0x11,
+0x22,
+0x75,
+0x2c,
+0xb8,
+0x08,
+0x16,
+0x18,
+0x81,
+0xe4,
+0x88,
+0x27,
+0x68,
+0x82,
+0xd5,
+0x90,
+0x45,
+0x84,
+0x22,
+0x3c,
+0x88,
+0x55,
+0x49,
+0x03,
+0x59,
+0x68,
+0x20,
+0x00,
+0x85,
+0x24,
+0x6c,
+0x00,
+0x01,
+0x2a,
+0x0a,
+0x66,
+0x00,
+0x00,
+0xe4,
+0xc8,
+0x6c,
+0x00,
+0x01,
+0x18,
+0x09,
+0x68,
+0x20,
+0x00,
+0x85,
+0x20,
+0x5c,
+0x82,
+0x40,
+0x88,
+0x48,
+0xa0,
+0x08,
+0xc6,
+0xc0,
+0x00,
+0x12,
+0xa0,
+0x96,
+0xc0,
+0x00,
+0x11,
+0x80,
+0xa6,
+0x60,
+0x00,
+0x0e,
+0x4c,
+0x89,
+0x62,
+0x07,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x5c,
+0x08,
+0xe8,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x90,
+0x2c,
+0x52,
+0xcb,
+0x80,
+0x85,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x88,
+0x06,
+0xcb,
+0xc1,
+0xa9,
+0x40,
+0x00,
+0x00,
+0x88,
+0x09,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x0a,
+0x44,
+0x10,
+0x00,
+0x85,
+0x89,
+0x68,
+0x20,
+0x00,
+0x98,
+0xa0,
+0x88,
+0x84,
+0x88,
+0x80,
+0x60,
+0x68,
+0x20,
+0x00,
+0x92,
+0xa4,
+0x68,
+0x20,
+0x00,
+0x95,
+0xa5,
+0x90,
+0x65,
+0x88,
+0x85,
+0xd4,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x40,
+0x00,
+0x03,
+0xc5,
+0x4f,
+0x44,
+0x10,
+0x00,
+0x88,
+0xc8,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x08,
+0x44,
+0x20,
+0x80,
+0x85,
+0x88,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x42,
+0x25,
+0x79,
+0x06,
+0x59,
+0x44,
+0x10,
+0x00,
+0x85,
+0xd5,
+0x68,
+0x00,
+0x00,
+0x6e,
+0x20,
+0x68,
+0x20,
+0x00,
+0x85,
+0x24,
+0x80,
+0x00,
+0x98,
+0x85,
+0xe0,
+0x66,
+0x00,
+0x00,
+0xe4,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x40,
+0x0a,
+0x5c,
+0x82,
+0x40,
+0x85,
+0xa0,
+0x68,
+0x20,
+0x00,
+0x85,
+0x21,
+0x88,
+0x84,
+0x8a,
+0x08,
+0x8c,
+0x84,
+0x00,
+0x96,
+0xc0,
+0x00,
+0x0d,
+0xc0,
+0xa6,
+0x60,
+0x00,
+0x0e,
+0x4c,
+0x89,
+0x62,
+0x07,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x5c,
+0x08,
+0xe8,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x90,
+0x2c,
+0x52,
+0xcb,
+0x80,
+0x85,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x8a,
+0x24,
+0x68,
+0x20,
+0x00,
+0x8d,
+0x25,
+0x88,
+0x06,
+0xcb,
+0xc1,
+0xa9,
+0x40,
+0x00,
+0x00,
+0x88,
+0x09,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x0a,
+0x44,
+0x10,
+0x00,
+0x85,
+0x89,
+0x68,
+0x20,
+0x00,
+0x98,
+0xa0,
+0x88,
+0x84,
+0x88,
+0x80,
+0x60,
+0x68,
+0x20,
+0x00,
+0x92,
+0xa4,
+0x68,
+0x20,
+0x00,
+0x95,
+0xa5,
+0x90,
+0x65,
+0x88,
+0x85,
+0xd4,
+0x66,
+0x00,
+0x00,
+0xe5,
+0x40,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xcf,
+0x44,
+0x10,
+0x00,
+0x88,
+0xc8,
+0x6c,
+0x40,
+0x00,
+0x4a,
+0x08,
+0x44,
+0x20,
+0x80,
+0x85,
+0x88,
+0x6c,
+0x40,
+0x00,
+0x98,
+0x0a,
+0x44,
+0x10,
+0x01,
+0x06,
+0x59,
+0x40,
+0x00,
+0x00,
+0x85,
+0xd5,
+0x68,
+0x00,
+0x00,
+0x50,
+0x20,
+0x90,
+0x61,
+0x1a,
+0x04,
+0x21,
+0x88,
+0x59,
+0x58,
+0x85,
+0xe1,
+0x90,
+0x65,
+0x88,
+0x87,
+0x54,
+0x66,
+0x00,
+0x01,
+0x31,
+0x28,
+0x5c,
+0x00,
+0x71,
+0x80,
+0x49,
+0x90,
+0x61,
+0x06,
+0x80,
+0x00,
+0x06,
+0x42,
+0x08,
+0x87,
+0x14,
+0xa0,
+0x42,
+0x18,
+0x86,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x86,
+0x61,
+0x66,
+0x00,
+0x01,
+0x31,
+0x28,
+0x5c,
+0x00,
+0x71,
+0x80,
+0x09,
+0x5c,
+0x82,
+0x10,
+0x86,
+0x20,
+0x5c,
+0x8c,
+0x01,
+0x03,
+0x10,
+0x5c,
+0x8e,
+0x09,
+0x04,
+0x12,
+0x88,
+0x12,
+0x18,
+0x85,
+0xa5,
+0xa0,
+0x56,
+0x48,
+0x82,
+0x94,
+0x54,
+0x08,
+0x18,
+0x86,
+0x8a,
+0x88,
+0x51,
+0x65,
+0x40,
+0xc8,
+0x88,
+0x1a,
+0x28,
+0x48,
+0x08,
+0xa2,
+0xd6,
+0x08,
+0x21,
+0x0a,
+0x4d,
+0x86,
+0x80,
+0x50,
+0x08,
+0x80,
+0x10,
+0xa4,
+0xf8,
+0x48,
+0x02,
+0x04,
+0x38,
+0x00,
+0x41,
+0x82,
+0x2c,
+0x08,
+0x02,
+0xc2,
+0x88,
+0x16,
+0x46,
+0x82,
+0x00,
+0x07,
+0xe2,
+0xc8,
+0x81,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x1c,
+0x21,
+0x68,
+0x20,
+0x00,
+0x24,
+0x24,
+0x68,
+0x20,
+0x00,
+0x72,
+0x25,
+0x40,
+0x00,
+0x00,
+0x80,
+0x6c,
+0x68,
+0x00,
+0x01,
+0x27,
+0x22,
+0x66,
+0x00,
+0x00,
+0xe0,
+0x40,
+0x68,
+0x20,
+0x00,
+0x7f,
+0xac,
+0x88,
+0x84,
+0x86,
+0x80,
+0x00,
+0x03,
+0x02,
+0x18,
+0x81,
+0x20,
+0x68,
+0x20,
+0x00,
+0x4b,
+0x24,
+0x68,
+0x20,
+0x00,
+0x78,
+0x25,
+0x88,
+0x06,
+0xc6,
+0x80,
+0x00,
+0x12,
+0xa2,
+0x26,
+0x60,
+0x00,
+0x0e,
+0x04,
+0x08,
+0x88,
+0xc8,
+0x68,
+0x20,
+0x00,
+0x83,
+0x20,
+0x68,
+0x20,
+0x00,
+0x84,
+0x21,
+0xa4,
+0x22,
+0x36,
+0x60,
+0x00,
+0x0c,
+0x52,
+0x8a,
+0x42,
+0x02,
+0x5c,
+0x8d,
+0x00,
+0x81,
+0xa0,
+0x5c,
+0x08,
+0x71,
+0x8e,
+0x88,
+0xa0,
+0x26,
+0x09,
+0x42,
+0x0f,
+0x25,
+0x9b,
+0x8b,
+0xc0,
+0x39,
+0x55,
+0x01,
+0x28,
+0x81,
+0x21,
+0x88,
+0x80,
+0x98,
+0x40,
+0x0b,
+0x54,
+0x0f,
+0x42,
+0x0a,
+0x61,
+0x94,
+0xa0,
+0xd2,
+0x59,
+0xa8,
+0xbc,
+0x03,
+0x98,
+0x40,
+0xc0,
+0x00,
+0x00,
+0x08,
+0x88,
+0x88,
+0x00,
+0x00,
+0x08,
+0x48,
+0x0a,
+0x54,
+0x0d,
+0x00,
+0x82,
+0x36,
+0xba,
+0x14,
+0x88,
+0x48,
+0xc0,
+0x40,
+0x00,
+0x02,
+0x80,
+0x90,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xc0,
+0x5c,
+0x8f,
+0x08,
+0x81,
+0x60,
+0xa0,
+0x04,
+0x3a,
+0x08,
+0x00,
+0x81,
+0xa0,
+0x9a,
+0x1a,
+0x21,
+0x85,
+0x80,
+0x85,
+0x70,
+0x94,
+0x00,
+0xa8,
+0x95,
+0x70,
+0xa0,
+0x08,
+0x1e,
+0x1a,
+0x10,
+0x01,
+0x81,
+0x00,
+0x99,
+0x80,
+0x08,
+0x54,
+0x0b,
+0x02,
+0x28,
+0x26,
+0x98,
+0x00,
+0x98,
+0x49,
+0x49,
+0xa2,
+0x80,
+0x18,
+0x84,
+0x25,
+0x85,
+0x04,
+0x8a,
+0x30,
+0x23,
+0xa3,
+0x00,
+0x2a,
+0x18,
+0x26,
+0xa2,
+0x06,
+0x48,
+0x80,
+0x66,
+0x88,
+0x26,
+0x68,
+0x82,
+0xe4,
+0x88,
+0x36,
+0x08,
+0x83,
+0xf6,
+0x66,
+0x00,
+0x00,
+0xec,
+0x60,
+0x5c,
+0x8f,
+0x00,
+0x81,
+0xa0,
+0x5c,
+0x08,
+0x28,
+0x82,
+0xa4,
+0xa0,
+0x20,
+0x59,
+0x6a,
+0x0f,
+0x52,
+0xcb,
+0xc0,
+0x40,
+0x48,
+0xbc,
+0x05,
+0x95,
+0xc0,
+0x03,
+0x08,
+0x32,
+0x08,
+0x81,
+0x22,
+0x00,
+0x00,
+0x08,
+0x51,
+0x8a,
+0x86,
+0x80,
+0x95,
+0x40,
+0xb8,
+0x20,
+0x14,
+0x09,
+0x80,
+0x09,
+0xa2,
+0x2e,
+0x48,
+0x68,
+0xc9,
+0x88,
+0x16,
+0x48,
+0x81,
+0xe0,
+0x66,
+0x00,
+0x00,
+0xf3,
+0xa0,
+0x88,
+0x22,
+0x58,
+0x84,
+0x22,
+0x88,
+0x12,
+0x48,
+0x81,
+0xa0,
+0xa2,
+0x82,
+0x18,
+0x83,
+0xb6,
+0xa1,
+0x02,
+0x5a,
+0x80,
+0x40,
+0xa0,
+0x04,
+0x0a,
+0x20,
+0x64,
+0x64,
+0x00,
+0x00,
+0xf4,
+0x8f,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x22,
+0x86,
+0x00,
+0x84,
+0x42,
+0x00,
+0x06,
+0x08,
+0x95,
+0x00,
+0xe0,
+0x3a,
+0x14,
+0x80,
+0x89,
+0x80,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0x6e,
+0x40,
+0x00,
+0x08,
+0x3c,
+0x38,
+0x15,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x30,
+0x05,
+0x15,
+0x96,
+0x16,
+0x1b,
+0x65,
+0x19,
+0x5b,
+0x08,
+0x02,
+0x04,
+0x41,
+0x00,
+0x18,
+0xe8,
+0xa9,
+0x80,
+0x08,
+0x23,
+0x07,
+0x79,
+0x82,
+0xe8,
+0x9c,
+0x00,
+0x19,
+0x48,
+0x37,
+0x30,
+0x1e,
+0x0b,
+0xc0,
+0x42,
+0x2a,
+0x07,
+0x63,
+0xb1,
+0x86,
+0x32,
+0x17,
+0x0b,
+0xff,
+0x62,
+0x32,
+0x0f,
+0x0b,
+0xc1,
+0xd0,
+0x51,
+0x83,
+0xab,
+0x01,
+0x00,
+0x50,
+0x41,
+0x11,
+0x82,
+0x68,
+0x55,
+0x00,
+0xb1,
+0xe0,
+0x04,
+0x46,
+0x08,
+0x89,
+0x60,
+0x35,
+0x50,
+0x41,
+0x49,
+0x83,
+0x89,
+0x55,
+0x00,
+0x79,
+0x83,
+0x48,
+0x44,
+0x41,
+0x09,
+0xe8,
+0x05,
+0x44,
+0x6d,
+0x49,
+0x68,
+0x36,
+0x21,
+0x01,
+0x00,
+0x84,
+0x82,
+0x37,
+0x80,
+0x02,
+0x28,
+0x80,
+0x6c,
+0x40,
+0x01,
+0x36,
+0x09,
+0x51,
+0x87,
+0xb1,
+0x83,
+0x08,
+0x36,
+0x14,
+0x72,
+0x81,
+0xa4,
+0x2f,
+0x16,
+0x52,
+0xe9,
+0xed,
+0x23,
+0x32,
+0xdb,
+0xa1,
+0x48,
+0x46,
+0x08,
+0x09,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x8a,
+0x68,
+0x00,
+0x00,
+0x1c,
+0x20,
+0x38,
+0x0a,
+0x42,
+0xa0,
+0x76,
+0x3a,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x8a,
+0x68,
+0x00,
+0x00,
+0x30,
+0x20,
+0x38,
+0x0a,
+0x42,
+0xa0,
+0x76,
+0x3a,
+0x18,
+0x63,
+0x01,
+0x30,
+0xbf,
+0xfc,
+0xa8,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x44,
+0x20,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0xf0,
+0x68,
+0x00,
+0x00,
+0x58,
+0x21,
+0x88,
+0x07,
+0x69,
+0x8e,
+0x88,
+0x40,
+0x00,
+0x02,
+0x00,
+0x02,
+0x55,
+0x03,
+0x20,
+0x08,
+0x7a,
+0x3a,
+0x10,
+0x43,
+0x22,
+0x20,
+0xbf,
+0xfb,
+0xa8,
+0x10,
+0x7a,
+0x40,
+0x00,
+0x02,
+0x01,
+0x80,
+0x66,
+0x00,
+0x01,
+0x30,
+0x68,
+0x5c,
+0x00,
+0x62,
+0x04,
+0x21,
+0x68,
+0x00,
+0x00,
+0x58,
+0x20,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x10,
+0x40,
+0x00,
+0x02,
+0x01,
+0x80,
+0x64,
+0x00,
+0x01,
+0x30,
+0x6f,
+0x5c,
+0x00,
+0x62,
+0x04,
+0x21,
+0x55,
+0x01,
+0x40,
+0x60,
+0x0b,
+0x5c,
+0x82,
+0x0a,
+0x20,
+0x26,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x02,
+0x30,
+0x27,
+0x87,
+0x00,
+0x84,
+0x44,
+0x54,
+0x07,
+0x80,
+0x84,
+0x42,
+0x54,
+0x23,
+0x82,
+0x49,
+0x80,
+0x82,
+0x8c,
+0x0c,
+0x0a,
+0x20,
+0x26,
+0x86,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x80,
+0x70,
+0x0b,
+0x55,
+0x00,
+0xa2,
+0x30,
+0x27,
+0x44,
+0x5d,
+0x60,
+0x78,
+0x0a,
+0x44,
+0x15,
+0x40,
+0x48,
+0x0b,
+0x55,
+0x01,
+0x09,
+0x80,
+0x8a,
+0x57,
+0x8f,
+0xb0,
+0x50,
+0x08,
+0x57,
+0x49,
+0x92,
+0x38,
+0x24,
+0x8c,
+0x0c,
+0x4a,
+0x20,
+0x22,
+0x86,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x82,
+0x10,
+0x21,
+0x85,
+0x00,
+0x84,
+0x44,
+0x5e,
+0x04,
+0x80,
+0x84,
+0x42,
+0x5e,
+0x20,
+0x82,
+0x69,
+0x80,
+0xc3,
+0x8c,
+0x0c,
+0x1a,
+0x30,
+0x22,
+0x87,
+0x00,
+0xb8,
+0xc0,
+0x2a,
+0x44,
+0x19,
+0x01,
+0x84,
+0x88,
+0x85,
+0x00,
+0xb4,
+0x45,
+0xd4,
+0x21,
+0x02,
+0x15,
+0x50,
+0x0f,
+0x84,
+0x80,
+0xa4,
+0x47,
+0x54,
+0x20,
+0x82,
+0x45,
+0x50,
+0x1c,
+0x18,
+0x08,
+0x28,
+0xc0,
+0xc0,
+0xa2,
+0x02,
+0x28,
+0x60,
+0x0b,
+0x8c,
+0x02,
+0x64,
+0x45,
+0xd8,
+0x05,
+0x00,
+0xb4,
+0x45,
+0x88,
+0x21,
+0x02,
+0x19,
+0x80,
+0xc3,
+0x55,
+0x00,
+0xf0,
+0x48,
+0x0b,
+0xa0,
+0x82,
+0x44,
+0x45,
+0xda,
+0x0c,
+0x06,
+0x15,
+0x40,
+0x4d,
+0x22,
+0x02,
+0x28,
+0x60,
+0x0a,
+0x44,
+0x31,
+0x80,
+0x50,
+0x09,
+0x44,
+0x08,
+0x01,
+0x80,
+0x89,
+0x96,
+0x82,
+0x45,
+0x00,
+0x8d,
+0x21,
+0x02,
+0x08,
+0x40,
+0x08,
+0x44,
+0x21,
+0x82,
+0x28,
+0x09,
+0xa0,
+0x02,
+0x48,
+0x20,
+0x08,
+0x98,
+0x08,
+0xa4,
+0x44,
+0x10,
+0x20,
+0x80,
+0xd9,
+0x48,
+0x25,
+0x50,
+0x0a,
+0x01,
+0x68,
+0x24,
+0x50,
+0x08,
+0xd8,
+0x20,
+0x08,
+0x98,
+0x00,
+0x94,
+0x42,
+0x54,
+0x02,
+0x00,
+0x99,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0x40,
+0x80,
+0x20,
+0x85,
+0x80,
+0x99,
+0x80,
+0x88,
+0x57,
+0x8b,
+0x28,
+0x40,
+0x08,
+0x57,
+0x49,
+0x68,
+0x60,
+0x08,
+0x44,
+0x21,
+0x01,
+0x68,
+0xa5,
+0x50,
+0x0a,
+0x90,
+0x60,
+0x89,
+0x46,
+0x0a,
+0x41,
+0x80,
+0x88,
+0x08,
+0x12,
+0x09,
+0x80,
+0x88,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x06,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x04,
+0x60,
+0xa4,
+0x18,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x5c,
+0x00,
+0x02,
+0xbf,
+0xf0,
+0x55,
+0x00,
+0x11,
+0x8e,
+0x81,
+0x5c,
+0x81,
+0x00,
+0x80,
+0x49,
+0x5c,
+0x82,
+0x0a,
+0x00,
+0x03,
+0x55,
+0x00,
+0x5b,
+0x80,
+0x00,
+0x55,
+0x02,
+0xe8,
+0x20,
+0x08,
+0x5d,
+0x0a,
+0x18,
+0x20,
+0x0b,
+0x59,
+0x04,
+0xc0,
+0xc0,
+0x8e,
+0xbf,
+0xf9,
+0xa1,
+0xb4,
+0xb3,
+0x5c,
+0x00,
+0x18,
+0x20,
+0x09,
+0x88,
+0x00,
+0x84,
+0x40,
+0xd4,
+0x16,
+0x82,
+0x55,
+0x40,
+0x08,
+0x0d,
+0x83,
+0x25,
+0x00,
+0xa0,
+0x0d,
+0x97,
+0xa5,
+0x50,
+0x10,
+0x18,
+0x00,
+0x25,
+0xc0,
+0x00,
+0x0d,
+0x86,
+0x08,
+0x80,
+0x52,
+0x40,
+0x00,
+0x02,
+0x18,
+0x86,
+0x55,
+0x02,
+0x68,
+0x20,
+0x08,
+0x5d,
+0x0a,
+0x08,
+0x20,
+0x0b,
+0x59,
+0x04,
+0x40,
+0xc0,
+0x8e,
+0xbf,
+0xf9,
+0xa1,
+0xb4,
+0xf3,
+0x86,
+0x00,
+0x88,
+0x80,
+0x09,
+0x44,
+0x25,
+0x61,
+0x68,
+0xa4,
+0x54,
+0x00,
+0x80,
+0xda,
+0x32,
+0x50,
+0x08,
+0x00,
+0xf1,
+0x7a,
+0x55,
+0x01,
+0x41,
+0x80,
+0x02,
+0x40,
+0x00,
+0x00,
+0x48,
+0x09,
+0x46,
+0x0a,
+0x40,
+0x50,
+0x08,
+0x57,
+0x8a,
+0xa8,
+0xda,
+0x60,
+0x57,
+0x49,
+0x62,
+0x80,
+0x10,
+0xab,
+0xfa,
+0x06,
+0x80,
+0x04,
+0x00,
+0x08,
+0x8b,
+0x00,
+0x10,
+0x88,
+0x07,
+0x6a,
+0x40,
+0xc0,
+0x88,
+0x0c,
+0x88,
+0x81,
+0x50,
+0x88,
+0x1e,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x00,
+0x55,
+0x02,
+0x10,
+0x81,
+0xa0,
+0x5d,
+0x04,
+0x10,
+0x80,
+0x80,
+0x59,
+0x10,
+0x80,
+0x81,
+0x52,
+0x55,
+0x02,
+0x22,
+0x00,
+0x20,
+0xbf,
+0xf3,
+0xa8,
+0x80,
+0xc8,
+0x5c,
+0x81,
+0x02,
+0x40,
+0xc0,
+0x80,
+0x00,
+0x25,
+0x94,
+0x08,
+0x00,
+0x00,
+0x88,
+0x00,
+0x0a,
+0x42,
+0x0b,
+0xe8,
+0x81,
+0x52,
+0x5c,
+0x00,
+0x00,
+0x80,
+0xe0,
+0xbc,
+0x0d,
+0xf8,
+0x81,
+0xca,
+0x98,
+0x2a,
+0x06,
+0x60,
+0x00,
+0x04,
+0xea,
+0x08,
+0x81,
+0x80,
+0x55,
+0x08,
+0x30,
+0x82,
+0x82,
+0x55,
+0x02,
+0xa0,
+0x82,
+0x00,
+0x55,
+0x02,
+0x00,
+0x81,
+0x02,
+0x40,
+0x00,
+0x00,
+0x81,
+0xca,
+0x58,
+0x04,
+0x00,
+0x82,
+0xc8,
+0xbf,
+0xf1,
+0xa8,
+0x82,
+0x50,
+0xbc,
+0x04,
+0xf5,
+0xc8,
+0x10,
+0x08,
+0x0a,
+0x03,
+0x90,
+0x20,
+0x88,
+0x0a,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x25,
+0x94,
+0x08,
+0x04,
+0x00,
+0x88,
+0x80,
+0xd2,
+0xbc,
+0x13,
+0xd5,
+0xc0,
+0x00,
+0x04,
+0x08,
+0xab,
+0xc0,
+0xcf,
+0x88,
+0x14,
+0xa9,
+0x82,
+0xa0,
+0x66,
+0x00,
+0x00,
+0x4e,
+0xa0,
+0x88,
+0x10,
+0x05,
+0x50,
+0x83,
+0x08,
+0x20,
+0x25,
+0x50,
+0x2a,
+0x08,
+0x18,
+0x05,
+0x50,
+0x20,
+0x08,
+0x08,
+0x28,
+0x81,
+0x4a,
+0x58,
+0x04,
+0x00,
+0x82,
+0x48,
+0xbf,
+0xf2,
+0xa8,
+0x81,
+0xd0,
+0x00,
+0x00,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x60,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x66,
+0x00,
+0x00,
+0xea,
+0x60,
+0x40,
+0x00,
+0x03,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x00,
+0xae,
+0x64,
+0x6e,
+0x00,
+0x00,
+0xd6,
+0x64,
+0x66,
+0x00,
+0x00,
+0xcc,
+0x80,
+0x66,
+0x00,
+0x01,
+0xb3,
+0x40,
+0x88,
+0x03,
+0x64,
+0x60,
+0xa4,
+0x30,
+0x30,
+0xc6,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x4a,
+0x80,
+0x10,
+0xab,
+0xfd,
+0x08,
+0x80,
+0xc9,
+0x5c,
+0x09,
+0xf8,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x59,
+0x02,
+0x5b,
+0x88,
+0x1d,
+0x70,
+0x00,
+0x00,
+0x6e,
+0x00,
+0x00,
+0xae,
+0x2d,
+0x25,
+0x9e,
+0x8b,
+0xc0,
+0x60,
+0x6c,
+0x68,
+0x08,
+0x10,
+0x03,
+0x6c,
+0x00,
+0x00,
+0xd8,
+0x53,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x67,
+0x6c,
+0x68,
+0x08,
+0x16,
+0x03,
+0x6c,
+0x00,
+0x00,
+0xd8,
+0x53,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x00,
+0xd6,
+0x2b,
+0x25,
+0x9d,
+0x8b,
+0xc0,
+0x50,
+0x6c,
+0x68,
+0x08,
+0x12,
+0x09,
+0x6c,
+0x00,
+0x00,
+0xda,
+0x49,
+0xbc,
+0x07,
+0x72,
+0x59,
+0xe8,
+0xbc,
+0x05,
+0x06,
+0xc6,
+0x80,
+0x81,
+0x60,
+0x96,
+0xc0,
+0x00,
+0x0d,
+0xa4,
+0x90,
+0x00,
+0x00,
+0x5c,
+0x02,
+0x29,
+0x02,
+0x13,
+0x68,
+0x00,
+0x00,
+0x00,
+0x75,
+0x6c,
+0x68,
+0x08,
+0x06,
+0x75,
+0x6c,
+0x00,
+0x02,
+0x6a,
+0x49,
+0x00,
+0x00,
+0x08,
+0x81,
+0x97,
+0x88,
+0x00,
+0xb4,
+0x60,
+0xb4,
+0x08,
+0x13,
+0x58,
+0x80,
+0x89,
+0x40,
+0x00,
+0x02,
+0x80,
+0x30,
+0xab,
+0xef,
+0x08,
+0x8a,
+0xc8,
+0x5c,
+0x09,
+0x78,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x58,
+0x82,
+0x6f,
+0x88,
+0x2e,
+0xe8,
+0x83,
+0x6d,
+0x88,
+0x3e,
+0xc8,
+0x84,
+0x6b,
+0x88,
+0x4e,
+0xa8,
+0x85,
+0x69,
+0x88,
+0x5e,
+0x88,
+0x86,
+0x67,
+0x88,
+0x6e,
+0x68,
+0x87,
+0x65,
+0x88,
+0x7e,
+0x48,
+0x88,
+0x63,
+0x88,
+0x8e,
+0x28,
+0x89,
+0x61,
+0x88,
+0x9e,
+0x08,
+0x8a,
+0x4a,
+0x88,
+0x0c,
+0x99,
+0x0b,
+0x5b,
+0x88,
+0xc5,
+0x79,
+0x0d,
+0x59,
+0x88,
+0xcd,
+0x59,
+0x0e,
+0x5a,
+0x88,
+0xf5,
+0x69,
+0x10,
+0x58,
+0x88,
+0xfd,
+0x40,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9e,
+0x0b,
+0xc0,
+0xd8,
+0x88,
+0x1f,
+0x63,
+0x81,
+0x36,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x48,
+0xba,
+0x10,
+0x16,
+0x60,
+0x00,
+0x17,
+0x88,
+0x0b,
+0xc1,
+0x47,
+0x66,
+0x00,
+0x01,
+0x74,
+0x40,
+0xbc,
+0x11,
+0x73,
+0x81,
+0x26,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0xa8,
+0xba,
+0x10,
+0x16,
+0xc7,
+0x02,
+0x80,
+0xc0,
+0x83,
+0x81,
+0x26,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x40,
+0x66,
+0x00,
+0x01,
+0x78,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x27,
+0x66,
+0x00,
+0x01,
+0x74,
+0x40,
+0x90,
+0xe1,
+0x29,
+0x0d,
+0x11,
+0x90,
+0xb1,
+0x38,
+0x89,
+0xa0,
+0x88,
+0x92,
+0x18,
+0x88,
+0xa2,
+0x88,
+0x82,
+0x38,
+0x87,
+0xa4,
+0x88,
+0x72,
+0x58,
+0x86,
+0xa6,
+0x88,
+0x62,
+0x78,
+0x85,
+0xa8,
+0x88,
+0x52,
+0x98,
+0x84,
+0xaa,
+0x88,
+0x42,
+0xb8,
+0x80,
+0x0b,
+0x88,
+0x08,
+0x98,
+0x8a,
+0x88,
+0x88,
+0xa0,
+0xa8,
+0x83,
+0xac,
+0x88,
+0x32,
+0xd8,
+0x82,
+0xae,
+0x88,
+0x22,
+0xf8,
+0x81,
+0xb6,
+0x88,
+0x13,
+0x59,
+0x10,
+0x10,
+0x88,
+0xf1,
+0x68,
+0x8c,
+0x95,
+0x46,
+0x0b,
+0x40,
+0x8c,
+0x17,
+0x88,
+0xf9,
+0x4a,
+0x81,
+0x10,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x60,
+0xab,
+0xef,
+0x08,
+0x8a,
+0xc8,
+0x5c,
+0x09,
+0x78,
+0x80,
+0x4b,
+0x88,
+0x17,
+0x58,
+0x82,
+0x6f,
+0x88,
+0x2e,
+0xe8,
+0x83,
+0x6d,
+0x88,
+0x3e,
+0xc8,
+0x84,
+0x6b,
+0x88,
+0x4e,
+0xa8,
+0x85,
+0x69,
+0x88,
+0x5e,
+0x88,
+0x86,
+0x67,
+0x88,
+0x6e,
+0x68,
+0x87,
+0x65,
+0x88,
+0x7e,
+0x48,
+0x88,
+0x63,
+0x88,
+0x8e,
+0x28,
+0x89,
+0x61,
+0x88,
+0x9e,
+0x08,
+0x8a,
+0x4a,
+0x88,
+0x0c,
+0x99,
+0x0b,
+0x5b,
+0x88,
+0xc5,
+0x79,
+0x0d,
+0x59,
+0x88,
+0xcd,
+0x59,
+0x0e,
+0x5a,
+0x88,
+0xf5,
+0x69,
+0x10,
+0x58,
+0x88,
+0xfd,
+0x40,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9e,
+0x0b,
+0xc0,
+0xd8,
+0x88,
+0x1f,
+0x63,
+0x81,
+0x36,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x48,
+0xba,
+0x10,
+0x16,
+0x60,
+0x00,
+0x18,
+0x2c,
+0x0b,
+0xc1,
+0x47,
+0x66,
+0x00,
+0x01,
+0x7e,
+0x80,
+0xbc,
+0x11,
+0x73,
+0x81,
+0x26,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x08,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0xa8,
+0xba,
+0x10,
+0x16,
+0xc7,
+0x02,
+0x88,
+0xc0,
+0x83,
+0x81,
+0x26,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x40,
+0x66,
+0x00,
+0x01,
+0x82,
+0xc0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x27,
+0x66,
+0x00,
+0x01,
+0x7e,
+0x80,
+0x90,
+0xe1,
+0x29,
+0x0d,
+0x11,
+0x90,
+0xb1,
+0x38,
+0x89,
+0xa0,
+0x88,
+0x92,
+0x18,
+0x88,
+0xa2,
+0x88,
+0x82,
+0x38,
+0x87,
+0xa4,
+0x88,
+0x72,
+0x58,
+0x86,
+0xa6,
+0x88,
+0x62,
+0x78,
+0x85,
+0xa8,
+0x88,
+0x52,
+0x98,
+0x84,
+0xaa,
+0x88,
+0x42,
+0xb8,
+0x80,
+0x0b,
+0x88,
+0x08,
+0x98,
+0x8a,
+0x88,
+0x88,
+0xa0,
+0xa8,
+0x83,
+0xac,
+0x88,
+0x32,
+0xd8,
+0x82,
+0xae,
+0x88,
+0x22,
+0xf8,
+0x81,
+0xb6,
+0x88,
+0x13,
+0x59,
+0x10,
+0x10,
+0x88,
+0xf1,
+0x68,
+0x8c,
+0x95,
+0x46,
+0x0b,
+0x40,
+0x8c,
+0x17,
+0x88,
+0xf9,
+0x4a,
+0x81,
+0x10,
+0x60,
+0x02,
+0x80,
+0x00,
+0x10,
+0x98,
+0xea,
+0x03,
+0x90,
+0x20,
+0x80,
+0x07,
+0xa6,
+0x00,
+0x30,
+0x00,
+0x01,
+0x06,
+0x82,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x80,
+0x07,
+0xab,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x1c,
+0x05,
+0x20,
+0x5c,
+0xbf,
+0x03,
+0x00,
+0x0c,
+0x84,
+0x07,
+0xa9,
+0xc0,
+0x00,
+0x68,
+0x00,
+0x40,
+0x20,
+0x0a,
+0x84,
+0x04,
+0xaa,
+0x00,
+0x40,
+0x84,
+0x04,
+0x8a,
+0x04,
+0xc0,
+0x84,
+0x00,
+0xa5,
+0xd0,
+0xc3,
+0x30,
+0x7f,
+0x83,
+0x00,
+0x30,
+0xbc,
+0x28,
+0x06,
+0x83,
+0x81,
+0xc0,
+0x02,
+0x06,
+0x83,
+0x40,
+0x02,
+0x42,
+0x15,
+0xc8,
+0x10,
+0x84,
+0x00,
+0xaa,
+0x00,
+0xc2,
+0x84,
+0x84,
+0xa8,
+0x50,
+0x48,
+0x68,
+0x34,
+0x08,
+0x44,
+0x23,
+0x84,
+0x00,
+0xa8,
+0x48,
+0xca,
+0x85,
+0x04,
+0x8a,
+0x18,
+0x21,
+0x84,
+0x00,
+0xa8,
+0x58,
+0x4a,
+0x85,
+0x04,
+0x8a,
+0xc7,
+0xa0,
+0x84,
+0x00,
+0xa8,
+0x08,
+0xca,
+0x85,
+0x04,
+0x8a,
+0xc4,
+0x42,
+0x84,
+0x00,
+0xa8,
+0x08,
+0xca,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa8,
+0x08,
+0x4a,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa8,
+0x48,
+0x4a,
+0x85,
+0x04,
+0x8a,
+0x08,
+0x21,
+0x84,
+0x00,
+0x84,
+0x60,
+0xa4,
+0x00,
+0xb4,
+0x88,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x68,
+0x34,
+0x04,
+0x23,
+0x20,
+0x5c,
+0xbf,
+0x0b,
+0x04,
+0x04,
+0x68,
+0x38,
+0x1c,
+0x05,
+0x21,
+0x5c,
+0x82,
+0x00,
+0x40,
+0x48,
+0x5c,
+0x00,
+0x70,
+0x48,
+0x7a,
+0x5c,
+0xbd,
+0x09,
+0xc8,
+0x81,
+0x68,
+0x00,
+0x40,
+0x28,
+0x2e,
+0x5c,
+0x08,
+0x20,
+0x08,
+0x6e,
+0x84,
+0x84,
+0xaa,
+0x0c,
+0xa1,
+0x9c,
+0x08,
+0x28,
+0x48,
+0x00,
+0x85,
+0x05,
+0x00,
+0x00,
+0x00,
+0x84,
+0x00,
+0x02,
+0x41,
+0x00,
+0x84,
+0x05,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xff,
+0xc0,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x02,
+0x38,
+0x11,
+0x82,
+0x58,
+0x10,
+0xbc,
+0x32,
+0x06,
+0x83,
+0x81,
+0xc0,
+0x02,
+0x16,
+0x83,
+0x40,
+0x42,
+0x02,
+0x08,
+0x48,
+0x02,
+0x84,
+0x05,
+0x2a,
+0x00,
+0x60,
+0xb0,
+0x48,
+0x28,
+0x40,
+0x52,
+0xa0,
+0x88,
+0x18,
+0x40,
+0x02,
+0x24,
+0x11,
+0x28,
+0x40,
+0x52,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x02,
+0x25,
+0x91,
+0x0b,
+0xff,
+0xc0,
+0x84,
+0x00,
+0x22,
+0x58,
+0x10,
+0xbc,
+0x01,
+0x1b,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x40,
+0x28,
+0x6e,
+0x80,
+0x86,
+0xe8,
+0x48,
+0x4a,
+0xa0,
+0xcc,
+0x18,
+0x48,
+0x0a,
+0x6c,
+0x68,
+0x08,
+0x40,
+0x4a,
+0xb0,
+0x50,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x40,
+0x0a,
+0x24,
+0x13,
+0x68,
+0x40,
+0x4a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x68,
+0x08,
+0x46,
+0x0a,
+0x25,
+0x93,
+0x0b,
+0xff,
+0xc0,
+0x84,
+0x00,
+0x82,
+0x58,
+0x20,
+0xbc,
+0x01,
+0x1b,
+0xa1,
+0x40,
+0xba,
+0x14,
+0x88,
+0x40,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x14,
+0x40,
+0x68,
+0x34,
+0x04,
+0x04,
+0x20,
+0x5c,
+0x02,
+0xe2,
+0xff,
+0xc0,
+0x5c,
+0x00,
+0x70,
+0x40,
+0x48,
+0x9c,
+0x00,
+0x0b,
+0x04,
+0x80,
+0x68,
+0x34,
+0x00,
+0x1e,
+0x21,
+0x84,
+0x05,
+0x0b,
+0x1d,
+0xf2,
+0x84,
+0x85,
+0x2a,
+0x08,
+0x81,
+0xb1,
+0x81,
+0xc8,
+0x48,
+0x48,
+0xa0,
+0x22,
+0x06,
+0x81,
+0x50,
+0xc8,
+0x40,
+0x08,
+0x40,
+0x50,
+0x68,
+0x00,
+0x01,
+0x41,
+0x48,
+0x88,
+0x03,
+0x6a,
+0x06,
+0x00,
+0x46,
+0x0a,
+0x40,
+0x48,
+0xc8,
+0x84,
+0x04,
+0xaa,
+0x80,
+0x10,
+0x68,
+0x34,
+0x00,
+0x09,
+0x20,
+0x5c,
+0x08,
+0xb3,
+0x00,
+0x3c,
+0x5c,
+0x81,
+0x00,
+0x40,
+0x00,
+0x52,
+0x0c,
+0x03,
+0x00,
+0x26,
+0x84,
+0x05,
+0x0a,
+0x00,
+0xa0,
+0x84,
+0x04,
+0x8a,
+0x00,
+0x40,
+0x80,
+0x04,
+0x88,
+0x00,
+0x4a,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x4a,
+0x84,
+0x7f,
+0xa0,
+0x00,
+0x00,
+0x68,
+0x34,
+0x08,
+0x4a,
+0x20,
+0x38,
+0x01,
+0xc8,
+0x40,
+0x48,
+0x40,
+0x00,
+0x00,
+0x40,
+0xc8,
+0x64,
+0x00,
+0x01,
+0x1e,
+0xa7,
+0x68,
+0x00,
+0x01,
+0xbc,
+0x20,
+0x39,
+0x02,
+0x08,
+0x00,
+0x21,
+0x68,
+0x34,
+0x08,
+0x20,
+0x22,
+0x84,
+0x80,
+0x88,
+0x00,
+0x21,
+0x85,
+0x04,
+0x8a,
+0x10,
+0x22,
+0x84,
+0x80,
+0x88,
+0x00,
+0x21,
+0x84,
+0x02,
+0x08,
+0x10,
+0x48,
+0x00,
+0x00,
+0x08,
+0x48,
+0x08,
+0x81,
+0x04,
+0x80,
+0x00,
+0x00,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x08,
+0x85,
+0x04,
+0x80,
+0x00,
+0x00,
+0x68,
+0x34,
+0x08,
+0x57,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x04,
+0x7a,
+0x5c,
+0x81,
+0x03,
+0x04,
+0x24,
+0x5c,
+0x13,
+0x70,
+0x40,
+0x48,
+0xa0,
+0x02,
+0x08,
+0x00,
+0x7a,
+0x80,
+0x04,
+0x88,
+0x40,
+0x7a,
+0xa0,
+0x52,
+0x08,
+0x02,
+0x4a,
+0xb0,
+0x60,
+0x48,
+0x40,
+0x48,
+0xa0,
+0x48,
+0x08,
+0x00,
+0x7a,
+0xac,
+0x9a,
+0x08,
+0x02,
+0x7a,
+0x84,
+0x07,
+0xaa,
+0x01,
+0x00,
+0xac,
+0xb0,
+0x08,
+0x00,
+0x7a,
+0xac,
+0x40,
+0x04,
+0x60,
+0xa4,
+0x00,
+0x24,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x34,
+0x00,
+0x17,
+0x20,
+0x68,
+0x03,
+0xfb,
+0xff,
+0xc9,
+0x5c,
+0x08,
+0x20,
+0x40,
+0x7a,
+0x5c,
+0x0b,
+0x32,
+0x00,
+0x80,
+0x5c,
+0xbd,
+0x00,
+0x40,
+0x02,
+0x54,
+0x4a,
+0xaa,
+0x03,
+0x21,
+0x5c,
+0x00,
+0x40,
+0x40,
+0x49,
+0x5c,
+0x09,
+0xd2,
+0xbf,
+0xe0,
+0x84,
+0x80,
+0x95,
+0x20,
+0x96,
+0x88,
+0x0f,
+0x65,
+0xc0,
+0x02,
+0x84,
+0x84,
+0x96,
+0x83,
+0x80,
+0x80,
+0xf2,
+0x28,
+0x40,
+0x08,
+0x52,
+0x0d,
+0x20,
+0x80,
+0x49,
+0x5c,
+0x1d,
+0x68,
+0x40,
+0x48,
+0xa0,
+0x5a,
+0x08,
+0x40,
+0x7a,
+0x9d,
+0x00,
+0x08,
+0x50,
+0x50,
+0x84,
+0x05,
+0x28,
+0x81,
+0x60,
+0x66,
+0x00,
+0x01,
+0x2e,
+0x20,
+0x5c,
+0x08,
+0x30,
+0x80,
+0x09,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x55,
+0x03,
+0x6b,
+0x03,
+0x24,
+0x30,
+0x12,
+0x8b,
+0xff,
+0x6a,
+0x5c,
+0x1d,
+0x68,
+0x80,
+0x49,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x1a,
+0xeb,
+0x00,
+0x0e,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x1a,
+0xa9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x0d,
+0xe9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0x69,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0xa9,
+0x8e,
+0x8a,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x06,
+0xeb,
+0x00,
+0x26,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x07,
+0x2b,
+0x00,
+0xc6,
+0x66,
+0x00,
+0x01,
+0x28,
+0x68,
+0x5c,
+0x07,
+0x6b,
+0x00,
+0x36,
+0x5c,
+0x02,
+0x68,
+0x81,
+0x20,
+0x5c,
+0x01,
+0xe0,
+0x80,
+0xb6,
+0x6c,
+0x70,
+0x10,
+0x1e,
+0x49,
+0x5c,
+0x03,
+0xe0,
+0x40,
+0x48,
+0x5c,
+0x00,
+0x72,
+0x05,
+0x80,
+0x84,
+0x07,
+0xaa,
+0x00,
+0x40,
+0x84,
+0x07,
+0xab,
+0x05,
+0xdd,
+0xa0,
+0x02,
+0x08,
+0x40,
+0x49,
+0xa0,
+0x16,
+0x08,
+0x40,
+0x48,
+0x46,
+0x0a,
+0x42,
+0x05,
+0x00,
+0x84,
+0x04,
+0xaa,
+0x80,
+0x20,
+0x5c,
+0x08,
+0x01,
+0x8e,
+0x8a,
+0x5c,
+0x00,
+0x51,
+0x82,
+0x88,
+0x68,
+0x00,
+0x03,
+0xe8,
+0x09,
+0x6c,
+0x70,
+0x10,
+0x0e,
+0x0b,
+0x25,
+0x83,
+0x8b,
+0xc0,
+0x41,
+0xba,
+0x14,
+0x86,
+0xc7,
+0x01,
+0x00,
+0xe5,
+0x20,
+0x00,
+0x00,
+0x2a,
+0x07,
+0x63,
+0x01,
+0x70,
+0xbf,
+0xf5,
+0x26,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x83,
+0x81,
+0xe6,
+0x52,
+0x0d,
+0x33,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x4a,
+0x5c,
+0x3f,
+0xe3,
+0x80,
+0x00,
+0x68,
+0x38,
+0x08,
+0x00,
+0x20,
+0x5c,
+0x09,
+0xe2,
+0xbf,
+0x90,
+0x52,
+0x09,
+0x62,
+0x00,
+0x41,
+0x5d,
+0x8c,
+0x12,
+0x08,
+0x20,
+0x5c,
+0x00,
+0x00,
+0x80,
+0x60,
+0xa0,
+0x16,
+0x0a,
+0x05,
+0x00,
+0xa0,
+0x4c,
+0x28,
+0x80,
+0xe2,
+0xa1,
+0x04,
+0x28,
+0x81,
+0x62,
+0xa1,
+0x02,
+0x28,
+0x81,
+0xe2,
+0xa1,
+0x16,
+0x2a,
+0x15,
+0x02,
+0x88,
+0x4c,
+0x88,
+0x83,
+0x62,
+0x88,
+0x3e,
+0x08,
+0x84,
+0x61,
+0x88,
+0x55,
+0x28,
+0x82,
+0x4a,
+0x88,
+0x2c,
+0x98,
+0x85,
+0xf6,
+0x40,
+0x00,
+0x03,
+0x04,
+0x04,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x48,
+0x5c,
+0x00,
+0xa0,
+0x48,
+0x7a,
+0x5c,
+0x00,
+0x68,
+0x48,
+0xc9,
+0x84,
+0x8c,
+0xa8,
+0x86,
+0x50,
+0x00,
+0x00,
+0x08,
+0x80,
+0x21,
+0x00,
+0x00,
+0x08,
+0x4d,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x40,
+0x49,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x5c,
+0x00,
+0xa0,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x68,
+0x81,
+0x21,
+0x84,
+0x07,
+0xa8,
+0x41,
+0x7a,
+0x00,
+0x00,
+0x08,
+0x84,
+0x8a,
+0x84,
+0x8c,
+0xa0,
+0x00,
+0x00,
+0x88,
+0x1a,
+0x08,
+0x83,
+0x21,
+0x84,
+0x5c,
+0x88,
+0x48,
+0x49,
+0x66,
+0x00,
+0x01,
+0x26,
+0xc0,
+0x5c,
+0xbb,
+0x00,
+0x83,
+0x20,
+0x88,
+0x50,
+0x99,
+0xc0,
+0x00,
+0x84,
+0x00,
+0xa3,
+0xb1,
+0x80,
+0x30,
+0x14,
+0x0b,
+0xc0,
+0x89,
+0x5c,
+0x19,
+0x20,
+0x86,
+0x0a,
+0x88,
+0x5b,
+0x6b,
+0xa1,
+0x48,
+0x5c,
+0x00,
+0x22,
+0x80,
+0x70,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x55,
+0x03,
+0x80,
+0x84,
+0x21,
+0x58,
+0x08,
+0x00,
+0x82,
+0x89,
+0x88,
+0x20,
+0xa4,
+0x3e,
+0x4d,
+0x08,
+0x3a,
+0x0b,
+0x04,
+0x04,
+0x88,
+0x5b,
+0x6b,
+0xa1,
+0x48,
+0x5c,
+0x3f,
+0xe2,
+0x80,
+0x70,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x08,
+0x00,
+0x20,
+0x5c,
+0x09,
+0xf3,
+0x00,
+0x14,
+0x52,
+0x0d,
+0x68,
+0x40,
+0x7a,
+0x5c,
+0x00,
+0x72,
+0x00,
+0x40,
+0x84,
+0x07,
+0xaa,
+0x00,
+0x20,
+0x84,
+0x04,
+0x9a,
+0x01,
+0x60,
+0x84,
+0x04,
+0x8a,
+0x05,
+0x00,
+0x84,
+0x04,
+0xaa,
+0xbf,
+0xf0,
+0xa0,
+0x4a,
+0x08,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x66,
+0x60,
+0x00,
+0x12,
+0x6c,
+0x08,
+0x80,
+0xb6,
+0x88,
+0x02,
+0x0b,
+0xa1,
+0x48,
+0x84,
+0x00,
+0x8a,
+0x80,
+0x10,
+0x62,
+0x00,
+0x00,
+0x00,
+0x36,
+0x5c,
+0x81,
+0x09,
+0x82,
+0x24,
+0x5c,
+0x80,
+0x81,
+0x82,
+0x60,
+0xbb,
+0x00,
+0x08,
+0x00,
+0xcc,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x38,
+0x00,
+0xe2,
+0x11,
+0x34,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x6d,
+0x84,
+0x86,
+0x06,
+0x20,
+0x00,
+0x00,
+0x01,
+0x43,
+0x90,
+0x20,
+0x00,
+0x00,
+0x08,
+0x00,
+0x7a,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x00,
+0x61,
+0x88,
+0x2c,
+0x50,
+0x8d,
+0x20,
+0x48,
+0x20,
+0x51,
+0x85,
+0x3a,
+0xc0,
+0x20,
+0x98,
+0x2e,
+0xe0,
+0x00,
+0x00,
+0x9c,
+0x40,
+0x08,
+0x40,
+0x49,
+0x62,
+0x00,
+0x00,
+0x00,
+0x24,
+0x84,
+0x86,
+0x09,
+0x8e,
+0x80,
+0x80,
+0x40,
+0x92,
+0x81,
+0x40,
+0xba,
+0x14,
+0x85,
+0x04,
+0xc0,
+0x04,
+0x86,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x06,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x04,
+0x60,
+0xa4,
+0x18,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x00,
+0xc0,
+0x2a,
+0x82,
+0x00,
+0xb4,
+0x41,
+0x80,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x02,
+0x00,
+0x84,
+0x42,
+0x50,
+0x18,
+0x24,
+0x09,
+0x80,
+0x82,
+0x55,
+0x00,
+0xb8,
+0x20,
+0x08,
+0x8c,
+0x06,
+0x0a,
+0x20,
+0x01,
+0x8c,
+0x12,
+0xe4,
+0x42,
+0x00,
+0x02,
+0x00,
+0x84,
+0x44,
+0x40,
+0x04,
+0x88,
+0x94,
+0x46,
+0xc8,
+0x06,
+0x08,
+0x85,
+0xb0,
+0x82,
+0x98,
+0x04,
+0xa5,
+0x78,
+0x9a,
+0x18,
+0x48,
+0x05,
+0x74,
+0xb1,
+0x3a,
+0x14,
+0x88,
+0xc1,
+0x60,
+0x55,
+0x00,
+0xa3,
+0x80,
+0x00,
+0x32,
+0x02,
+0x8b,
+0xc3,
+0x58,
+0x22,
+0x86,
+0xe5,
+0xb8,
+0xa2,
+0x30,
+0x0f,
+0x05,
+0x70,
+0x80,
+0x30,
+0x00,
+0xf5,
+0x08,
+0x1c,
+0x30,
+0x4e,
+0xa2,
+0xa6,
+0x63,
+0x4b,
+0xc1,
+0x23,
+0x3c,
+0x08,
+0x20,
+0x0c,
+0xd2,
+0x35,
+0xd4,
+0x68,
+0x20,
+0x01,
+0x51,
+0x20,
+0x28,
+0x16,
+0x25,
+0x14,
+0x2a,
+0x84,
+0x00,
+0x85,
+0x70,
+0xb0,
+0xb0,
+0x0f,
+0xa5,
+0x16,
+0xe6,
+0x04,
+0x08,
+0x92,
+0x80,
+0x20,
+0x68,
+0x1f,
+0xff,
+0xff,
+0xc8,
+0x36,
+0x80,
+0x35,
+0x44,
+0x84,
+0x98,
+0x0c,
+0x82,
+0x32,
+0x09,
+0x28,
+0x08,
+0x02,
+0x10,
+0x3f,
+0x20,
+0x90,
+0xc2,
+0x81,
+0x3f,
+0x08,
+0xb0,
+0x02,
+0x23,
+0xc0,
+0x98,
+0x00,
+0x80,
+0x83,
+0x00,
+0x22,
+0x3c,
+0x02,
+0xe0,
+0x28,
+0x98,
+0x00,
+0x80,
+0x8c,
+0x00,
+0x22,
+0x04,
+0x09,
+0x80,
+0x0b,
+0x08,
+0xb0,
+0x02,
+0x23,
+0xc0,
+0x98,
+0x00,
+0xa0,
+0x8b,
+0x00,
+0x22,
+0x3c,
+0x02,
+0xe0,
+0x28,
+0x98,
+0x00,
+0xa4,
+0x47,
+0x00,
+0x3a,
+0x14,
+0x82,
+0x20,
+0x40,
+0x98,
+0x00,
+0x8b,
+0xa1,
+0x48,
+0x98,
+0xe8,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x08,
+0x29,
+0x58,
+0x2c,
+0x52,
+0xcb,
+0x02,
+0xbf,
+0xc0,
+0x88,
+0x0e,
+0x14,
+0x20,
+0x7c,
+0x88,
+0x16,
+0x28,
+0x80,
+0x76,
+0x00,
+0x00,
+0x09,
+0x5a,
+0x2e,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0xa0,
+0x95,
+0xba,
+0xe3,
+0x20,
+0x30,
+0xbc,
+0x04,
+0x18,
+0x59,
+0x08,
+0x55,
+0x03,
+0x23,
+0xc0,
+0x4f,
+0x85,
+0x94,
+0x82,
+0x49,
+0x60,
+0x40,
+0x00,
+0x01,
+0x58,
+0x60,
+0x5c,
+0x82,
+0x0a,
+0x18,
+0xa1,
+0x88,
+0x1e,
+0x10,
+0x00,
+0x00,
+0x80,
+0x88,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x06,
+0x95,
+0xc8,
+0x10,
+0x08,
+0x26,
+0x13,
+0x81,
+0x05,
+0x95,
+0x8a,
+0xc2,
+0x59,
+0x60,
+0xbc,
+0x45,
+0x18,
+0x82,
+0xe5,
+0x88,
+0x36,
+0x30,
+0x00,
+0x00,
+0x8c,
+0x02,
+0xa8,
+0x20,
+0x09,
+0x44,
+0x08,
+0x00,
+0x55,
+0x22,
+0x82,
+0x00,
+0x84,
+0x44,
+0x40,
+0x02,
+0x00,
+0x88,
+0x50,
+0x09,
+0x44,
+0x25,
+0x01,
+0x82,
+0x40,
+0x98,
+0x08,
+0x28,
+0xc0,
+0x60,
+0x00,
+0x00,
+0x08,
+0xc1,
+0x2a,
+0x82,
+0x00,
+0x94,
+0x40,
+0x80,
+0x06,
+0x00,
+0x84,
+0x44,
+0x48,
+0x06,
+0x08,
+0x95,
+0x50,
+0x0a,
+0x18,
+0x48,
+0x00,
+0x81,
+0xa4,
+0x98,
+0x08,
+0x28,
+0xc1,
+0x60,
+0x40,
+0x00,
+0x00,
+0x83,
+0xd2,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x04,
+0x5c,
+0x08,
+0x71,
+0x48,
+0xad,
+0x25,
+0x9a,
+0x8b,
+0xc1,
+0xa1,
+0x94,
+0x82,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x07,
+0x88,
+0x83,
+0x88,
+0x88,
+0x12,
+0x06,
+0x60,
+0x00,
+0x14,
+0xc4,
+0x0b,
+0xc1,
+0x7f,
+0x40,
+0x00,
+0x00,
+0x81,
+0xa0,
+0x66,
+0x00,
+0x01,
+0x4a,
+0x28,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x00,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0x42,
+0x03,
+0x40,
+0x81,
+0x20,
+0x94,
+0xa6,
+0x46,
+0x60,
+0x00,
+0x14,
+0xc4,
+0x88,
+0x83,
+0x88,
+0xbc,
+0x08,
+0xf8,
+0x81,
+0xa0,
+0xbc,
+0x06,
+0xf8,
+0x81,
+0xa0,
+0xbc,
+0x04,
+0xf8,
+0x81,
+0xa0,
+0x88,
+0x1a,
+0x08,
+0x82,
+0xe5,
+0x88,
+0x36,
+0x30,
+0x00,
+0x00,
+0x88,
+0x22,
+0x58,
+0x41,
+0x08,
+0x59,
+0x01,
+0x02,
+0x2c,
+0xd8,
+0x42,
+0x04,
+0x48,
+0x82,
+0xa5,
+0x5c,
+0x81,
+0x00,
+0x81,
+0xe0,
+0x38,
+0x10,
+0x48,
+0x83,
+0x20,
+0x00,
+0x00,
+0x09,
+0x40,
+0xad,
+0x25,
+0x92,
+0x8b,
+0xc3,
+0xc1,
+0x00,
+0x00,
+0x08,
+0x81,
+0x20,
+0x88,
+0x0a,
+0x3a,
+0x02,
+0x80,
+0x8d,
+0x82,
+0xa8,
+0x28,
+0x09,
+0x44,
+0x08,
+0x00,
+0x40,
+0x20,
+0x82,
+0x80,
+0x84,
+0x44,
+0x40,
+0x04,
+0x00,
+0x98,
+0x28,
+0x08,
+0x44,
+0x25,
+0x01,
+0x82,
+0x40,
+0x98,
+0x08,
+0x25,
+0x50,
+0x0b,
+0x8d,
+0x86,
+0x06,
+0x80,
+0x00,
+0x0d,
+0x72,
+0x08,
+0xd9,
+0x2a,
+0x82,
+0x80,
+0x94,
+0x40,
+0x80,
+0x06,
+0x80,
+0x84,
+0x44,
+0x48,
+0x06,
+0x88,
+0x94,
+0x46,
+0xd2,
+0x18,
+0x48,
+0x09,
+0x80,
+0x82,
+0x8d,
+0x96,
+0x08,
+0x80,
+0xd2,
+0x66,
+0x00,
+0x00,
+0x4c,
+0xc0,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x04,
+0x5c,
+0x08,
+0x71,
+0x48,
+0xad,
+0x25,
+0x9a,
+0x8b,
+0xc1,
+0x41,
+0x94,
+0x82,
+0xd2,
+0x59,
+0x28,
+0xbc,
+0x05,
+0x88,
+0x80,
+0x88,
+0x88,
+0x12,
+0x06,
+0x60,
+0x00,
+0x14,
+0xee,
+0x0b,
+0xc0,
+0xc7,
+0x66,
+0x00,
+0x01,
+0x47,
+0xa8,
+0x55,
+0x01,
+0x2a,
+0x08,
+0x00,
+0x59,
+0x01,
+0x00,
+0x83,
+0x21,
+0x42,
+0x02,
+0x40,
+0x81,
+0x20,
+0x94,
+0xbe,
+0x46,
+0x60,
+0x00,
+0x14,
+0xee,
+0x88,
+0x80,
+0x88,
+0x38,
+0x11,
+0x48,
+0x81,
+0xa0,
+0x00,
+0x00,
+0x09,
+0x40,
+0x2d,
+0x25,
+0x92,
+0x8b,
+0xc2,
+0xc1,
+0x68,
+0x20,
+0x01,
+0x7c,
+0x23,
+0x5c,
+0x81,
+0x12,
+0xc0,
+0x80,
+0x68,
+0x20,
+0x01,
+0x82,
+0x24,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x20,
+0x52,
+0x49,
+0x40,
+0x19,
+0x0a,
+0x68,
+0x20,
+0x00,
+0xe6,
+0x25,
+0x5c,
+0x82,
+0x08,
+0x21,
+0x08,
+0x81,
+0x80,
+0x98,
+0x01,
+0x4a,
+0x82,
+0x94,
+0x88,
+0x00,
+0x49,
+0x00,
+0x00,
+0x08,
+0x20,
+0x0b,
+0x82,
+0x28,
+0x98,
+0x1a,
+0x88,
+0x6c,
+0x40,
+0x02,
+0xfc,
+0x0a,
+0x82,
+0x84,
+0xb8,
+0x2a,
+0xc9,
+0x80,
+0x2c,
+0x86,
+0xc4,
+0x00,
+0x1c,
+0x44,
+0xa0,
+0x00,
+0x00,
+0x88,
+0x32,
+0x28,
+0x60,
+0x0b,
+0x85,
+0x80,
+0x28,
+0x60,
+0x88,
+0x85,
+0x88,
+0x96,
+0xc4,
+0x00,
+0x30,
+0x80,
+0xa8,
+0x68,
+0x4b,
+0x84,
+0x05,
+0x28,
+0x68,
+0xc8,
+0x84,
+0x0c,
+0x99,
+0x50,
+0xe0,
+0x6c,
+0x40,
+0x01,
+0xd0,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x40,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x94,
+0x3a,
+0xc3,
+0x20,
+0x20,
+0xbc,
+0x21,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0xbc,
+0x03,
+0xf9,
+0x43,
+0x64,
+0xb0,
+0x00,
+0xd9,
+0x43,
+0x65,
+0xa0,
+0x02,
+0x19,
+0x48,
+0x2d,
+0x59,
+0x01,
+0x42,
+0x08,
+0x12,
+0x42,
+0x08,
+0xc1,
+0x50,
+0x2d,
+0xa1,
+0x40,
+0x95,
+0xc0,
+0x84,
+0x14,
+0x02,
+0xe2,
+0x58,
+0x30,
+0xbc,
+0x07,
+0x99,
+0x42,
+0xae,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x91,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x70,
+0xb0,
+0x00,
+0xcb,
+0xc0,
+0x57,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x30,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x11,
+0xb0,
+0x00,
+0xcb,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x50,
+0x00,
+0x00,
+0xba,
+0x14,
+0x8b,
+0x00,
+0x0c,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x94,
+0x22,
+0xc3,
+0x20,
+0x20,
+0xbc,
+0x1c,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x03,
+0xa9,
+0x8e,
+0x88,
+0xbc,
+0x03,
+0xf9,
+0x41,
+0xe4,
+0xb0,
+0x00,
+0xd9,
+0x41,
+0xe5,
+0x5c,
+0x08,
+0x6a,
+0x00,
+0x11,
+0x94,
+0x02,
+0xe5,
+0x2c,
+0xb8,
+0x14,
+0x82,
+0xdb,
+0xc0,
+0x79,
+0x94,
+0x8a,
+0xe3,
+0x20,
+0x28,
+0xbc,
+0x09,
+0x13,
+0x20,
+0x30,
+0xbc,
+0x07,
+0x0b,
+0x00,
+0x0c,
+0xbc,
+0x05,
+0x73,
+0x20,
+0x28,
+0xbc,
+0x03,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x01,
+0x1b,
+0x00,
+0x0c,
+0xba,
+0x14,
+0x89,
+0x41,
+0x66,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x48,
+0xb0,
+0x00,
+0xc0,
+0x00,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x08,
+0x82,
+0x85,
+0x00,
+0xa2,
+0xa0,
+0x76,
+0x81,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x85,
+0x00,
+0x03,
+0x00,
+0x30,
+0xbc,
+0x1d,
+0x4a,
+0x00,
+0x41,
+0x84,
+0x80,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x01,
+0x38,
+0x41,
+0x48,
+0xa0,
+0x82,
+0x28,
+0x50,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x25,
+0x40,
+0x00,
+0x00,
+0x48,
+0xc8,
+0x5b,
+0x48,
+0x03,
+0x01,
+0x05,
+0x50,
+0x4b,
+0x08,
+0x10,
+0x0a,
+0x5c,
+0x82,
+0x00,
+0xd1,
+0x32,
+0x54,
+0x02,
+0x90,
+0x41,
+0x08,
+0x57,
+0x0d,
+0x20,
+0xd1,
+0x7a,
+0x81,
+0x04,
+0x89,
+0x80,
+0x08,
+0x50,
+0x4b,
+0x10,
+0xd1,
+0x30,
+0x54,
+0x04,
+0x03,
+0xa1,
+0x48,
+0x8d,
+0x17,
+0x80,
+0x00,
+0x00,
+0xba,
+0x14,
+0x88,
+0x4a,
+0xfa,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0x08,
+0xc2,
+0x85,
+0x00,
+0xa2,
+0xa0,
+0x76,
+0x81,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x85,
+0x00,
+0x03,
+0x00,
+0x30,
+0xbc,
+0x1f,
+0x4a,
+0x01,
+0x81,
+0x84,
+0x80,
+0xa3,
+0x01,
+0x30,
+0xbc,
+0x01,
+0x38,
+0x46,
+0x48,
+0xa0,
+0x82,
+0x28,
+0x50,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x25,
+0x40,
+0x00,
+0x00,
+0x48,
+0xc8,
+0x5c,
+0x82,
+0x0b,
+0x01,
+0x05,
+0x5b,
+0x48,
+0x10,
+0x46,
+0x0a,
+0x81,
+0x00,
+0x02,
+0xe0,
+0x36,
+0x50,
+0x4b,
+0x00,
+0x10,
+0xca,
+0x98,
+0x08,
+0x88,
+0xd0,
+0x31,
+0x28,
+0x00,
+0x85,
+0x04,
+0xb0,
+0x0d,
+0x0d,
+0x80,
+0x00,
+0x00,
+0x8d,
+0x03,
+0x25,
+0x40,
+0x08,
+0x3a,
+0x14,
+0x88,
+0xd0,
+0x78,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x08,
+0x22,
+0x08,
+0xe0,
+0xa0,
+0x4d,
+0xa9,
+0x50,
+0x2e,
+0x25,
+0x93,
+0x0b,
+0xc0,
+0x69,
+0x84,
+0x07,
+0xa3,
+0x81,
+0x0c,
+0x52,
+0x09,
+0x83,
+0xa1,
+0x48,
+0x94,
+0x8e,
+0x00,
+0x00,
+0x00,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x20,
+0x01,
+0x2b,
+0x20,
+0x5c,
+0x81,
+0x03,
+0x00,
+0x0c,
+0x5c,
+0x06,
+0x68,
+0x02,
+0x0a,
+0x68,
+0x20,
+0x01,
+0x05,
+0x21,
+0x51,
+0x89,
+0x80,
+0x40,
+0x01,
+0x51,
+0x8a,
+0x00,
+0x08,
+0x50,
+0x68,
+0x20,
+0x01,
+0xf0,
+0x22,
+0x84,
+0x80,
+0x28,
+0x10,
+0x52,
+0x81,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x0f,
+0xc2,
+0x36,
+0x82,
+0x00,
+0x0d,
+0x72,
+0x18,
+0x10,
+0x7a,
+0x68,
+0x00,
+0x00,
+0xdc,
+0x24,
+0x85,
+0x07,
+0xaa,
+0x10,
+0x42,
+0x84,
+0x90,
+0xb8,
+0x58,
+0x02,
+0x85,
+0x88,
+0xa8,
+0x48,
+0x48,
+0xa2,
+0x06,
+0x35,
+0xc0,
+0x02,
+0x86,
+0x04,
+0x98,
+0x12,
+0x52,
+0x68,
+0x00,
+0x00,
+0xa6,
+0x21,
+0xa1,
+0xc2,
+0x0a,
+0x10,
+0x64,
+0xab,
+0xff,
+0x0e,
+0x21,
+0x04,
+0x84,
+0x07,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0xa7,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0xa7,
+0xa6,
+0xc0,
+0x00,
+0x37,
+0x64,
+0x86,
+0xc0,
+0x00,
+0x36,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x19,
+0x84,
+0xb6,
+0xc4,
+0x00,
+0x19,
+0xa4,
+0xb6,
+0xc4,
+0x00,
+0x31,
+0x44,
+0xb6,
+0xc4,
+0x00,
+0x31,
+0xa4,
+0xb9,
+0x58,
+0x65,
+0x6c,
+0x40,
+0x01,
+0x06,
+0x51,
+0x6c,
+0x40,
+0x01,
+0x08,
+0x51,
+0x84,
+0x87,
+0xa8,
+0x50,
+0x4a,
+0x86,
+0x04,
+0xa8,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0xa8,
+0x20,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x23,
+0x8b,
+0x68,
+0x41,
+0x4a,
+0x66,
+0x00,
+0x00,
+0x4c,
+0x28,
+0x40,
+0x00,
+0x03,
+0x19,
+0x04,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4b,
+0xc0,
+0x68,
+0x00,
+0x01,
+0xab,
+0x20,
+0x68,
+0x00,
+0x04,
+0xe2,
+0x0a,
+0x88,
+0x03,
+0x68,
+0x41,
+0x4a,
+0x64,
+0x00,
+0x00,
+0x4c,
+0x2f,
+0x5c,
+0x02,
+0xa2,
+0x80,
+0x10,
+0x30,
+0x1a,
+0x8b,
+0xc1,
+0x58,
+0x6c,
+0x40,
+0x02,
+0xc0,
+0x08,
+0x84,
+0x00,
+0xb3,
+0x69,
+0xc0,
+0x98,
+0x00,
+0xb3,
+0x09,
+0x38,
+0xbc,
+0x0d,
+0x46,
+0xc4,
+0x00,
+0x2b,
+0xc0,
+0x83,
+0x01,
+0x30,
+0xbc,
+0x03,
+0x2b,
+0xc0,
+0x4f,
+0x6c,
+0x40,
+0x02,
+0xbe,
+0x08,
+0x6c,
+0x40,
+0x02,
+0xb0,
+0x08,
+0x28,
+0x13,
+0x04,
+0x20,
+0x17,
+0x98,
+0x00,
+0xa2,
+0xf1,
+0x75,
+0x98,
+0x28,
+0x9b,
+0xa1,
+0x48,
+0x98,
+0x24,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0xdd,
+0x22,
+0x6c,
+0x00,
+0x01,
+0xb8,
+0x08,
+0x85,
+0x00,
+0xa3,
+0x01,
+0x30,
+0x55,
+0x03,
+0xab,
+0xc5,
+0xf9,
+0x85,
+0x04,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xa7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x02,
+0x08,
+0x08,
+0x5b,
+0x48,
+0x00,
+0x40,
+0x08,
+0x5b,
+0x48,
+0x11,
+0x54,
+0x24,
+0x32,
+0x06,
+0x0b,
+0xc3,
+0x39,
+0x9a,
+0x00,
+0x03,
+0x01,
+0x30,
+0xbc,
+0x09,
+0x48,
+0x48,
+0x0a,
+0x36,
+0x98,
+0x09,
+0x80,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x44,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x1b,
+0xc7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0xbc,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x06,
+0x0a,
+0x2a,
+0x06,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x40,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xc4,
+0x85,
+0xc0,
+0xce,
+0xb0,
+0x13,
+0x76,
+0xe0,
+0x00,
+0x14,
+0x8a,
+0xe6,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x15,
+0xc8,
+0x10,
+0x18,
+0xe8,
+0x86,
+0xc0,
+0x00,
+0x14,
+0xe4,
+0x95,
+0x2c,
+0xf8,
+0x14,
+0xa4,
+0x4b,
+0xc0,
+0x99,
+0x5c,
+0x0b,
+0xa0,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x2c,
+0x20,
+0xa6,
+0xc4,
+0x00,
+0x15,
+0x04,
+0xa6,
+0xc4,
+0x00,
+0x18,
+0x04,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x0a,
+0x52,
+0x49,
+0xa3,
+0xa1,
+0x48,
+0x6c,
+0x40,
+0x03,
+0x1c,
+0x48,
+0x00,
+0x00,
+0x03,
+0x01,
+0x30,
+0xbc,
+0x05,
+0x38,
+0x48,
+0x0a,
+0x36,
+0x98,
+0x09,
+0x80,
+0x0a,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0x42,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x1b,
+0xc7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0xbc,
+0x08,
+0x6c,
+0x40,
+0x02,
+0x04,
+0x0a,
+0x2a,
+0x06,
+0x43,
+0x01,
+0xa0,
+0xbc,
+0x0e,
+0x96,
+0xc0,
+0x00,
+0x1b,
+0xc4,
+0x86,
+0x80,
+0x00,
+0x0d,
+0xf2,
+0x16,
+0xc4,
+0x00,
+0x31,
+0xc0,
+0x85,
+0xc0,
+0x07,
+0x30,
+0x17,
+0x55,
+0x20,
+0xb2,
+0x2c,
+0x02,
+0x09,
+0x4a,
+0x46,
+0x84,
+0x87,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0xc4,
+0x8b,
+0xa1,
+0x40,
+0x68,
+0x20,
+0x01,
+0x46,
+0x21,
+0x68,
+0x00,
+0x00,
+0xcd,
+0x22,
+0x39,
+0x02,
+0x08,
+0x08,
+0x08,
+0x81,
+0x07,
+0xa8,
+0x10,
+0x48,
+0x68,
+0x00,
+0x00,
+0xd2,
+0x20,
+0x84,
+0x80,
+0x88,
+0x00,
+0x7a,
+0x81,
+0x04,
+0x88,
+0x00,
+0x7a,
+0x68,
+0x00,
+0x00,
+0xca,
+0x23,
+0x80,
+0x04,
+0x86,
+0x80,
+0x00,
+0x58,
+0x8a,
+0xca,
+0x10,
+0x01,
+0x81,
+0x86,
+0xc8,
+0x10,
+0x7a,
+0xa0,
+0x00,
+0x48,
+0x58,
+0x7a,
+0x85,
+0x06,
+0x14,
+0x60,
+0xa4,
+0x00,
+0x07,
+0xa8,
+0x40,
+0x64,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0c,
+0x16,
+0xc0,
+0x00,
+0x1a,
+0x22,
+0x06,
+0x80,
+0x00,
+0x58,
+0xd2,
+0xc6,
+0xc0,
+0x00,
+0x1a,
+0xc2,
+0x16,
+0xc0,
+0x00,
+0x19,
+0x46,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0c,
+0x8a,
+0xbf,
+0xf0,
+0x6c,
+0x00,
+0x01,
+0xa2,
+0x20,
+0x68,
+0x00,
+0x05,
+0x88,
+0xac,
+0x6c,
+0x00,
+0x01,
+0xac,
+0x21,
+0x6c,
+0x00,
+0x01,
+0x94,
+0x6c,
+0x42,
+0x09,
+0xf8,
+0x40,
+0x7a,
+0x84,
+0x87,
+0xa8,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x66,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0xaa,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x65,
+0x6c,
+0x00,
+0x01,
+0xac,
+0x20,
+0x42,
+0x03,
+0xf8,
+0x80,
+0x36,
+0x40,
+0x00,
+0x00,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x05,
+0x96,
+0xa0,
+0x88,
+0x03,
+0x66,
+0xc0,
+0x00,
+0x19,
+0x46,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x08,
+0x38,
+0x10,
+0x62,
+0x59,
+0xa0,
+0xbc,
+0x0d,
+0x06,
+0xc0,
+0x00,
+0x1a,
+0x22,
+0x06,
+0x80,
+0x00,
+0x58,
+0x8a,
+0xc6,
+0xc0,
+0x00,
+0x1a,
+0xc2,
+0x16,
+0xc0,
+0x00,
+0x19,
+0x46,
+0xc4,
+0x60,
+0xa4,
+0x04,
+0x07,
+0xa8,
+0x48,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x64,
+0x00,
+0x01,
+0x66,
+0xe7,
+0x68,
+0x00,
+0x00,
+0xce,
+0x20,
+0x68,
+0x20,
+0x01,
+0x48,
+0x21,
+0xab,
+0xfe,
+0x08,
+0x40,
+0x08,
+0x84,
+0x80,
+0x94,
+0x40,
+0x80,
+0x08,
+0x07,
+0x66,
+0x60,
+0x00,
+0x16,
+0xae,
+0x89,
+0x80,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x80,
+0xc8,
+0x68,
+0x20,
+0x01,
+0x48,
+0x20,
+0x68,
+0x00,
+0x00,
+0xd3,
+0x21,
+0x80,
+0x00,
+0x88,
+0x48,
+0x09,
+0x44,
+0x20,
+0x00,
+0x81,
+0x60,
+0x66,
+0x00,
+0x01,
+0x6a,
+0xe8,
+0x40,
+0x00,
+0x01,
+0x80,
+0x09,
+0x5c,
+0x81,
+0x00,
+0x81,
+0x20,
+0x6c,
+0x00,
+0x01,
+0x9c,
+0x09,
+0x84,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x19,
+0xa0,
+0xa5,
+0x40,
+0xd6,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x0c,
+0xe2,
+0x05,
+0x44,
+0x17,
+0x08,
+0x08,
+0x96,
+0xc0,
+0x00,
+0x1a,
+0x60,
+0xb6,
+0xc0,
+0x00,
+0x1a,
+0x40,
+0x25,
+0x40,
+0x5f,
+0x80,
+0x04,
+0xa5,
+0x44,
+0x1f,
+0xa0,
+0x00,
+0x28,
+0x00,
+0x0a,
+0x68,
+0x00,
+0x00,
+0xd3,
+0x21,
+0x44,
+0x30,
+0x00,
+0x40,
+0xa0,
+0x80,
+0x84,
+0xb8,
+0x50,
+0xc0,
+0x84,
+0x04,
+0x0a,
+0x08,
+0x00,
+0x80,
+0x80,
+0x94,
+0x40,
+0x80,
+0x04,
+0x8a,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x0c,
+0x08,
+0x48,
+0x40,
+0x40,
+0x00,
+0x02,
+0x80,
+0x20,
+0x44,
+0x29,
+0x02,
+0xc0,
+0x20,
+0x68,
+0x20,
+0x01,
+0x4a,
+0x20,
+0x51,
+0x48,
+0x9b,
+0x3f,
+0xc4,
+0x80,
+0x00,
+0x29,
+0x80,
+0xc9,
+0x44,
+0x0d,
+0xc0,
+0x00,
+0x02,
+0x98,
+0x0c,
+0x84,
+0x40,
+0xdc,
+0x00,
+0x00,
+0x29,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0xc0,
+0x00,
+0x02,
+0x98,
+0x0c,
+0x84,
+0x40,
+0xdc,
+0x00,
+0x00,
+0x29,
+0x80,
+0xc8,
+0x44,
+0x0d,
+0x40,
+0x00,
+0x03,
+0x98,
+0x08,
+0x84,
+0x40,
+0x90,
+0x04,
+0x00,
+0x85,
+0x14,
+0x69,
+0x18,
+0x08,
+0xa2,
+0xe0,
+0x9a,
+0x55,
+0x00,
+0xd1,
+0x80,
+0x89,
+0x1a,
+0x0d,
+0x99,
+0xa0,
+0x42,
+0x08,
+0x63,
+0x02,
+0x20,
+0xdb,
+0x57,
+0x06,
+0x99,
+0x80,
+0xc9,
+0x98,
+0x0c,
+0xa0,
+0x86,
+0x30,
+0x22,
+0x0d,
+0xb5,
+0x70,
+0x69,
+0x98,
+0x0c,
+0x99,
+0x80,
+0xca,
+0x08,
+0x63,
+0x02,
+0x20,
+0xdb,
+0x57,
+0x06,
+0x91,
+0x80,
+0xc9,
+0x98,
+0x08,
+0xa0,
+0x86,
+0x20,
+0x22,
+0x0d,
+0x25,
+0x70,
+0x51,
+0x3a,
+0x14,
+0x82,
+0x20,
+0x92,
+0x40,
+0x00,
+0x01,
+0x80,
+0x88,
+0x85,
+0x00,
+0x88,
+0x40,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc1,
+0x50,
+0x30,
+0x13,
+0x0b,
+0xc0,
+0xbd,
+0x84,
+0x80,
+0x82,
+0xe1,
+0x30,
+0x98,
+0x00,
+0x88,
+0x40,
+0x48,
+0x00,
+0x00,
+0x08,
+0x50,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0xb4,
+0xba,
+0x14,
+0x88,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x02,
+0x81,
+0x30,
+0x98,
+0x00,
+0x88,
+0x40,
+0x48,
+0x00,
+0x00,
+0x08,
+0x50,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x12,
+0x84,
+0x04,
+0xab,
+0xa1,
+0x40,
+0x68,
+0x38,
+0x14,
+0x07,
+0x20,
+0x5c,
+0x00,
+0x63,
+0x07,
+0xf6,
+0x5c,
+0x1f,
+0x30,
+0x40,
+0x4a,
+0xa0,
+0x0c,
+0x0a,
+0x04,
+0x21,
+0x84,
+0x04,
+0x84,
+0x60,
+0xa4,
+0x04,
+0x84,
+0xa8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x14,
+0x04,
+0x20,
+0x38,
+0x10,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x52,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x17,
+0x21,
+0x5c,
+0x90,
+0x8a,
+0xc0,
+0x08,
+0x80,
+0xaa,
+0x09,
+0x48,
+0x2e,
+0x32,
+0x23,
+0x0b,
+0xc1,
+0x4c,
+0x68,
+0x00,
+0x00,
+0x06,
+0xa1,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x46,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0xba,
+0x14,
+0x86,
+0xe0,
+0x00,
+0x00,
+0xd6,
+0x60,
+0x00,
+0x00,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0x46,
+0x0a,
+0x41,
+0x48,
+0xc0,
+0x84,
+0x86,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x02,
+0x21,
+0x68,
+0x00,
+0x00,
+0x18,
+0x20,
+0x84,
+0x87,
+0xaa,
+0xbf,
+0xf0,
+0x84,
+0xf0,
+0x85,
+0x90,
+0x10,
+0x20,
+0x63,
+0x29,
+0x8e,
+0x8a,
+0x95,
+0x06,
+0x68,
+0x80,
+0x76,
+0xbc,
+0x05,
+0x98,
+0x40,
+0x21,
+0x5c,
+0x00,
+0x63,
+0xc2,
+0xff,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x44,
+0x39,
+0x00,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x00,
+0x00,
+0x94,
+0x84,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x91,
+0x43,
+0x01,
+0x2e,
+0x80,
+0x26,
+0x10,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x0b,
+0x52,
+0xcd,
+0xc1,
+0x40,
+0x2d,
+0x54,
+0x09,
+0x43,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x01,
+0x40,
+0x60,
+0x5c,
+0x00,
+0x63,
+0xc1,
+0x5f,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x7a,
+0xa0,
+0x05,
+0x98,
+0x40,
+0x22,
+0xba,
+0x0a,
+0x88,
+0x80,
+0xe1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x91,
+0x40,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x61,
+0x8e,
+0x8a,
+0x94,
+0x04,
+0x6a,
+0x06,
+0x21,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x48,
+0x84,
+0x06,
+0x10,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x05,
+0x20,
+0x38,
+0x12,
+0xc8,
+0x40,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xc0,
+0x68,
+0x5c,
+0x00,
+0x73,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x4a,
+0x64,
+0x00,
+0x01,
+0x7a,
+0x07,
+0x5c,
+0xbd,
+0x03,
+0x00,
+0x4c,
+0x6c,
+0x70,
+0x28,
+0x0a,
+0x4a,
+0x9c,
+0x00,
+0x08,
+0x40,
+0x48,
+0x84,
+0x07,
+0xa4,
+0x60,
+0xa4,
+0x18,
+0xe8,
+0x86,
+0xe0,
+0x00,
+0x00,
+0xd6,
+0x40,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x17,
+0x20,
+0x00,
+0x00,
+0x0a,
+0x06,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x22,
+0x30,
+0xbc,
+0x15,
+0xc5,
+0xc8,
+0x04,
+0x04,
+0x02,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x09,
+0x56,
+0x20,
+0x00,
+0x00,
+0x04,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x40,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x02,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x0f,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0x66,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x80,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0x60,
+0x6c,
+0x70,
+0x28,
+0x04,
+0x48,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x02,
+0xe6,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x47,
+0x20,
+0x5c,
+0x00,
+0x63,
+0x07,
+0xf6,
+0x5c,
+0x1f,
+0x30,
+0x40,
+0x4a,
+0xa0,
+0x0c,
+0x0a,
+0x04,
+0x21,
+0x84,
+0x04,
+0x84,
+0x60,
+0xa4,
+0x04,
+0x84,
+0xa8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x38,
+0x14,
+0x44,
+0x20,
+0x38,
+0x10,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x68,
+0x40,
+0x4a,
+0x00,
+0x00,
+0x08,
+0x41,
+0x0a,
+0x52,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x80,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x42,
+0x21,
+0x68,
+0x00,
+0x01,
+0x23,
+0x20,
+0x84,
+0x87,
+0xaa,
+0xbf,
+0xf0,
+0x84,
+0xf0,
+0x85,
+0x90,
+0x10,
+0x20,
+0x63,
+0x29,
+0x8e,
+0x8a,
+0x95,
+0x06,
+0x68,
+0x80,
+0x76,
+0xbc,
+0x05,
+0x98,
+0x40,
+0x21,
+0x5c,
+0x00,
+0x63,
+0xc2,
+0xff,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x85,
+0x62,
+0x00,
+0x00,
+0x00,
+0x44,
+0x39,
+0x00,
+0x80,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x80,
+0x00,
+0x94,
+0x84,
+0x00,
+0x00,
+0x00,
+0x5c,
+0x91,
+0x43,
+0x01,
+0x2e,
+0x80,
+0x26,
+0x10,
+0x00,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x0b,
+0x52,
+0xcd,
+0xc1,
+0x40,
+0x2d,
+0x54,
+0x09,
+0x43,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x01,
+0x40,
+0x60,
+0x5c,
+0x00,
+0x63,
+0xc1,
+0x5f,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x68,
+0x00,
+0x01,
+0x0f,
+0x20,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x7a,
+0xa0,
+0x05,
+0x98,
+0x40,
+0x22,
+0xba,
+0x0a,
+0x88,
+0x80,
+0xe1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x91,
+0x40,
+0x80,
+0xa0,
+0x5c,
+0x00,
+0x61,
+0x8e,
+0x8a,
+0x94,
+0x04,
+0x6a,
+0x06,
+0x21,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x48,
+0x84,
+0x06,
+0x10,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x14,
+0x45,
+0x20,
+0x38,
+0x12,
+0xc8,
+0x40,
+0x00,
+0x25,
+0x90,
+0x0b,
+0xc0,
+0x68,
+0x5c,
+0x00,
+0x73,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x4a,
+0x64,
+0x00,
+0x01,
+0x84,
+0x47,
+0x5c,
+0xbd,
+0x03,
+0x00,
+0x4c,
+0x6c,
+0x70,
+0x28,
+0x8a,
+0x4a,
+0x9c,
+0x00,
+0x08,
+0x40,
+0x48,
+0x84,
+0x07,
+0xa4,
+0x60,
+0xa4,
+0x18,
+0xe8,
+0x86,
+0xe0,
+0x00,
+0x22,
+0x36,
+0x40,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0x22,
+0x20,
+0x00,
+0x00,
+0x0a,
+0x06,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x22,
+0x30,
+0xbc,
+0x15,
+0xc5,
+0xc8,
+0x04,
+0x04,
+0x02,
+0x03,
+0x20,
+0x30,
+0xbc,
+0x09,
+0x56,
+0x20,
+0x00,
+0x00,
+0x04,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x40,
+0x0c,
+0x6c,
+0x70,
+0x28,
+0x82,
+0x48,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x02,
+0x61,
+0x8e,
+0x8a,
+0x6c,
+0x70,
+0x28,
+0x84,
+0x48,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x0f,
+0x6e,
+0x00,
+0x02,
+0x23,
+0x66,
+0x60,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x94,
+0x00,
+0xc6,
+0xc7,
+0x02,
+0x88,
+0x24,
+0x80,
+0x00,
+0x00,
+0x55,
+0x31,
+0x83,
+0x00,
+0x0c,
+0x6e,
+0x00,
+0x02,
+0x23,
+0x60,
+0x6c,
+0x70,
+0x28,
+0x84,
+0x48,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x24,
+0x46,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x06,
+0x1e,
+0x20,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x33,
+0x26,
+0x00,
+0x00,
+0x00,
+0xab,
+0xfd,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x01,
+0x78,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x83,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x8e,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa0,
+0x60,
+0x68,
+0x00,
+0x01,
+0x8e,
+0x20,
+0x68,
+0x00,
+0x01,
+0x83,
+0x24,
+0xa0,
+0x04,
+0x06,
+0xc4,
+0x00,
+0x3a,
+0x60,
+0x88,
+0x40,
+0x09,
+0x44,
+0x20,
+0x02,
+0x20,
+0x44,
+0x68,
+0x00,
+0x01,
+0x78,
+0x21,
+0x86,
+0x00,
+0x98,
+0x80,
+0xe4,
+0xa0,
+0x84,
+0x46,
+0xc4,
+0x00,
+0x39,
+0xe0,
+0x84,
+0x42,
+0x10,
+0x06,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x39,
+0x60,
+0x94,
+0x44,
+0x80,
+0x1a,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x3a,
+0xc0,
+0x94,
+0x60,
+0x88,
+0x88,
+0x16,
+0x44,
+0x40,
+0x88,
+0x18,
+0x00,
+0x84,
+0xd0,
+0xd0,
+0x88,
+0x1e,
+0x02,
+0x2c,
+0x40,
+0x51,
+0x62,
+0x80,
+0x82,
+0x4c,
+0x51,
+0x62,
+0x41,
+0x83,
+0x09,
+0x84,
+0x04,
+0xc8,
+0x82,
+0xc9,
+0x68,
+0x20,
+0x01,
+0xf8,
+0x24,
+0x66,
+0x00,
+0x01,
+0xa0,
+0xc8,
+0x46,
+0x08,
+0x08,
+0x82,
+0x0a,
+0x88,
+0x20,
+0x98,
+0x82,
+0x48,
+0x68,
+0x20,
+0x01,
+0xfb,
+0x24,
+0x66,
+0x00,
+0x01,
+0xa0,
+0xc8,
+0x40,
+0x00,
+0x00,
+0x82,
+0x8a,
+0x5c,
+0x82,
+0x00,
+0x81,
+0x20,
+0x88,
+0x20,
+0x98,
+0x80,
+0xa4,
+0x80,
+0x04,
+0x98,
+0x60,
+0x48,
+0x40,
+0x00,
+0x00,
+0x82,
+0x60,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x56,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x82,
+0x00,
+0x80,
+0xa0,
+0x88,
+0x22,
+0x48,
+0x81,
+0x21,
+0x80,
+0x00,
+0x98,
+0x61,
+0xc8,
+0x88,
+0x16,
+0x08,
+0x49,
+0x48,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x62,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x82,
+0x00,
+0x81,
+0xa0,
+0x88,
+0x12,
+0x48,
+0x80,
+0xa1,
+0x80,
+0x00,
+0x98,
+0x61,
+0xc8,
+0x88,
+0x0e,
+0x08,
+0x49,
+0x48,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x68,
+0x00,
+0x01,
+0x6e,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x5c,
+0x81,
+0x00,
+0x82,
+0x21,
+0x68,
+0x20,
+0x01,
+0xbb,
+0x24,
+0x88,
+0x1a,
+0x08,
+0x80,
+0xa2,
+0x80,
+0x80,
+0x9a,
+0x20,
+0x64,
+0x84,
+0x14,
+0x88,
+0x51,
+0xc8,
+0x88,
+0x1e,
+0x18,
+0x82,
+0x64,
+0x68,
+0x00,
+0x01,
+0x56,
+0x23,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x40,
+0x00,
+0x02,
+0x18,
+0x40,
+0x5c,
+0x83,
+0x08,
+0x81,
+0xa1,
+0x5c,
+0x88,
+0x00,
+0x81,
+0x22,
+0x5c,
+0x81,
+0x08,
+0x08,
+0xc8,
+0x80,
+0xa4,
+0x88,
+0x81,
+0x61,
+0x68,
+0x00,
+0x01,
+0x62,
+0x20,
+0x81,
+0x08,
+0x98,
+0x82,
+0x24,
+0x88,
+0x1e,
+0x26,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x00,
+0x40,
+0x5c,
+0x83,
+0x08,
+0x81,
+0xa1,
+0x5c,
+0x81,
+0x00,
+0x80,
+0xa2,
+0x5c,
+0x88,
+0x08,
+0x08,
+0xc8,
+0x80,
+0xac,
+0x88,
+0x80,
+0xe1,
+0x68,
+0x00,
+0x01,
+0x6e,
+0x20,
+0x81,
+0x00,
+0x98,
+0x82,
+0x24,
+0x88,
+0x1e,
+0x26,
+0x60,
+0x00,
+0x13,
+0x26,
+0x8a,
+0x00,
+0x40,
+0x5c,
+0x83,
+0x00,
+0x81,
+0xa4,
+0x5c,
+0x88,
+0x08,
+0x81,
+0x20,
+0x82,
+0x04,
+0x88,
+0x22,
+0xc8,
+0x88,
+0x1e,
+0x46,
+0x60,
+0x00,
+0x1a,
+0x14,
+0x08,
+0x81,
+0x24,
+0x88,
+0x0a,
+0x08,
+0x63,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa1,
+0x40,
+0x88,
+0x0a,
+0x48,
+0x81,
+0xa0,
+0x40,
+0x00,
+0x00,
+0x63,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x06,
+0x4d,
+0x24,
+0x88,
+0x1a,
+0x08,
+0x80,
+0x36,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x64,
+0xba,
+0x14,
+0x88,
+0x43,
+0x48,
+0x40,
+0x00,
+0x02,
+0x80,
+0x30,
+0x68,
+0x00,
+0x01,
+0x89,
+0x24,
+0xab,
+0xfc,
+0x06,
+0xc0,
+0x00,
+0x11,
+0x40,
+0xb6,
+0xc0,
+0x00,
+0x12,
+0x60,
+0x06,
+0xc0,
+0x00,
+0x29,
+0x40,
+0x16,
+0xc0,
+0x00,
+0x32,
+0x80,
+0xa8,
+0x60,
+0x09,
+0x88,
+0x05,
+0x18,
+0x80,
+0xd0,
+0x88,
+0x14,
+0xb8,
+0x81,
+0xf6,
+0x68,
+0x00,
+0x01,
+0x7e,
+0x20,
+0x66,
+0x00,
+0x01,
+0xa7,
+0x28,
+0x40,
+0x00,
+0x00,
+0x40,
+0x08,
+0x68,
+0x00,
+0x01,
+0x89,
+0x21,
+0x68,
+0x20,
+0x01,
+0xdd,
+0x20,
+0x68,
+0x00,
+0x01,
+0x7e,
+0x24,
+0x5c,
+0x83,
+0x00,
+0x40,
+0x88,
+0x80,
+0xa0,
+0xb5,
+0x70,
+0x9c,
+0x02,
+0x20,
+0x98,
+0x40,
+0x08,
+0x88,
+0x36,
+0x45,
+0x70,
+0x94,
+0x08,
+0x24,
+0x08,
+0x82,
+0xe1,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0x54,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0x98,
+0x00,
+0x98,
+0x83,
+0x20,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x84,
+0x04,
+0x86,
+0x80,
+0x00,
+0x16,
+0x02,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x88,
+0x82,
+0x09,
+0x55,
+0x01,
+0x28,
+0x83,
+0x20,
+0x5c,
+0x83,
+0x00,
+0x82,
+0xa4,
+0x80,
+0x20,
+0x88,
+0x60,
+0x49,
+0x40,
+0x00,
+0x00,
+0x82,
+0x60,
+0x68,
+0x00,
+0x01,
+0x54,
+0x21,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc1,
+0x22,
+0x66,
+0x00,
+0x01,
+0xa1,
+0xa0,
+0x5c,
+0x83,
+0x00,
+0x82,
+0xa0,
+0x68,
+0x00,
+0x01,
+0x60,
+0x21,
+0x80,
+0x20,
+0x88,
+0x82,
+0xe0,
+0x68,
+0x20,
+0x01,
+0xb8,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc3,
+0x22,
+0x66,
+0x00,
+0x01,
+0xa1,
+0xa0,
+0x88,
+0x10,
+0x83,
+0x69,
+0x00,
+0x98,
+0x00,
+0x96,
+0xc4,
+0x00,
+0x3a,
+0x80,
+0xb5,
+0x80,
+0xf4,
+0x18,
+0xe8,
+0x9b,
+0xc0,
+0x3a,
+0x55,
+0x01,
+0x70,
+0x80,
+0x88,
+0xb0,
+0x00,
+0xd3,
+0x69,
+0x00,
+0x98,
+0x00,
+0x03,
+0x01,
+0xc0,
+0xbc,
+0x03,
+0xa5,
+0x50,
+0x18,
+0x08,
+0x00,
+0x8b,
+0x00,
+0x08,
+0x36,
+0x90,
+0x19,
+0x80,
+0x48,
+0x30,
+0x1e,
+0x0b,
+0xc0,
+0x12,
+0xb0,
+0x00,
+0xe3,
+0x20,
+0x28,
+0xbc,
+0x05,
+0x98,
+0x82,
+0x20,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x21,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x60,
+0x6c,
+0x40,
+0x03,
+0xaa,
+0x08,
+0x6c,
+0x00,
+0x03,
+0x94,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x96,
+0x48,
+0xa0,
+0x0c,
+0x08,
+0x40,
+0x09,
+0x88,
+0x06,
+0x06,
+0x82,
+0x00,
+0x0a,
+0xa2,
+0x46,
+0x80,
+0x00,
+0x1c,
+0x62,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x08,
+0x82,
+0xa0,
+0x88,
+0x0c,
+0x8a,
+0x00,
+0xc0,
+0x84,
+0x00,
+0x98,
+0x81,
+0x60,
+0x68,
+0x20,
+0x00,
+0xc2,
+0x24,
+0x68,
+0x00,
+0x01,
+0xc8,
+0x20,
+0x66,
+0x00,
+0x01,
+0x32,
+0x60,
+0x66,
+0x00,
+0x01,
+0xaa,
+0x88,
+0x55,
+0x01,
+0x28,
+0x80,
+0x88,
+0x6c,
+0x00,
+0x03,
+0x94,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x68,
+0x00,
+0x01,
+0x81,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x42,
+0x02,
+0xb8,
+0x40,
+0xc8,
+0x2a,
+0x7e,
+0x46,
+0xc0,
+0x00,
+0x39,
+0x44,
+0x86,
+0xc0,
+0x00,
+0x30,
+0x47,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x96,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x64,
+0x68,
+0x00,
+0x01,
+0x8c,
+0x20,
+0x00,
+0x00,
+0x04,
+0x20,
+0x37,
+0x84,
+0x00,
+0x88,
+0x40,
+0xc8,
+0x2a,
+0x7e,
+0x46,
+0xc0,
+0x00,
+0x39,
+0x64,
+0x86,
+0xc0,
+0x00,
+0x31,
+0xa7,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x08,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x68,
+0x40,
+0x00,
+0x00,
+0x80,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x04,
+0x7a,
+0x6c,
+0x00,
+0x03,
+0x1a,
+0x7a,
+0x66,
+0x00,
+0x01,
+0xa6,
+0x68,
+0x84,
+0x20,
+0x98,
+0x81,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x46,
+0x48,
+0x66,
+0x00,
+0x01,
+0xa6,
+0x68,
+0x40,
+0x00,
+0x00,
+0x42,
+0x09,
+0x6c,
+0x00,
+0x03,
+0x4a,
+0x48,
+0x68,
+0x00,
+0x06,
+0x80,
+0x20,
+0x88,
+0x1b,
+0x6b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x60,
+0x40,
+0x00,
+0x02,
+0x80,
+0x40,
+0x68,
+0x00,
+0x06,
+0x1c,
+0xa0,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x33,
+0x26,
+0x00,
+0x00,
+0x00,
+0x84,
+0x00,
+0x88,
+0x40,
+0x8a,
+0x57,
+0x0d,
+0x03,
+0xa1,
+0x48,
+0x84,
+0x14,
+0x00,
+0x00,
+0x00,
+0xa2,
+0x02,
+0x48,
+0x60,
+0x08,
+0x44,
+0x20,
+0x00,
+0x60,
+0x89,
+0xba,
+0x14,
+0x80,
+0x89,
+0x80,
+0x40,
+0x00,
+0x01,
+0x80,
+0x08,
+0xa0,
+0x04,
+0x08,
+0x41,
+0x88,
+0x46,
+0x0a,
+0x40,
+0x40,
+0x0a,
+0x2e,
+0x13,
+0x09,
+0x80,
+0x08,
+0x5c,
+0x81,
+0x02,
+0x22,
+0x23,
+0x81,
+0x80,
+0x94,
+0x40,
+0x80,
+0x05,
+0x80,
+0x95,
+0x00,
+0xa0,
+0x20,
+0x8c,
+0x19,
+0x80,
+0x09,
+0x80,
+0x84,
+0x9a,
+0x1c,
+0x64,
+0x82,
+0x00,
+0x84,
+0x42,
+0x00,
+0x06,
+0x00,
+0x98,
+0x48,
+0x08,
+0x44,
+0x0c,
+0x02,
+0xbf,
+0xd0,
+0x98,
+0x00,
+0x9a,
+0x24,
+0xe4,
+0x80,
+0x84,
+0x98,
+0x80,
+0x61,
+0x88,
+0x0e,
+0x48,
+0x81,
+0x62,
+0x88,
+0x1e,
+0x08,
+0x82,
+0x76,
+0x66,
+0x00,
+0x01,
+0x32,
+0x68,
+0xa0,
+0x80,
+0x08,
+0x80,
+0x20,
+0x88,
+0x0a,
+0x2a,
+0x00,
+0x40,
+0x88,
+0x06,
+0x06,
+0x60,
+0x00,
+0x13,
+0x26,
+0x85,
+0x50,
+0x12,
+0xa1,
+0x06,
+0x48,
+0x80,
+0x20,
+0x88,
+0x12,
+0x2a,
+0x04,
+0x60,
+0x84,
+0x00,
+0x95,
+0x70,
+0x94,
+0x05,
+0x08,
+0x95,
+0x70,
+0xa0,
+0x08,
+0x1a,
+0x09,
+0x80,
+0x08,
+0x59,
+0x01,
+0x00,
+0x44,
+0xc8,
+0xbc,
+0x0e,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x17,
+0x38,
+0x50,
+0x08,
+0x2e,
+0x12,
+0xd5,
+0xb4,
+0xa0,
+0x05,
+0x0c,
+0x99,
+0x80,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x22,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0xe5,
+0x36,
+0x10,
+0x58,
+0x50,
+0xc9,
+0xbc,
+0x0b,
+0x78,
+0x50,
+0x08,
+0x28,
+0x12,
+0xd5,
+0xb4,
+0xa0,
+0x05,
+0x0c,
+0x99,
+0x80,
+0x09,
+0x6c,
+0x40,
+0x02,
+0x22,
+0x08,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x25,
+0x85,
+0x0c,
+0x80,
+0x00,
+0x00,
+0x88,
+0x23,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x03,
+0xc0,
+0x08,
+0x44,
+0x20,
+0x03,
+0xa1,
+0x11,
+0x98,
+0x00,
+0x96,
+0x80,
+0x39,
+0x4b,
+0xb8,
+0x84,
+0x42,
+0x00,
+0x3a,
+0x14,
+0x82,
+0x2b,
+0x80,
+0x46,
+0x08,
+0x09,
+0x83,
+0x08,
+0x55,
+0x01,
+0x12,
+0xbf,
+0xf0,
+0x5b,
+0x0c,
+0x30,
+0x80,
+0x4a,
+0x44,
+0x58,
+0x81,
+0x84,
+0x48,
+0x44,
+0x21,
+0x81,
+0x80,
+0x41,
+0x5b,
+0x04,
+0x32,
+0xc0,
+0x20,
+0x44,
+0x40,
+0x81,
+0x84,
+0x48,
+0x5b,
+0x0a,
+0x29,
+0x84,
+0x8a,
+0x44,
+0x58,
+0x01,
+0x84,
+0x0b,
+0x44,
+0x39,
+0x00,
+0x80,
+0x09,
+0x44,
+0x39,
+0x81,
+0x80,
+0xca,
+0x5c,
+0x00,
+0xeb,
+0xa1,
+0x11,
+0x2a,
+0x02,
+0x74,
+0x46,
+0x88,
+0x1a,
+0x14,
+0x14,
+0x46,
+0x88,
+0x18,
+0x34,
+0x34,
+0xc0,
+0xc0,
+0x9a,
+0x00,
+0x09,
+0x83,
+0x8b,
+0x44,
+0x08,
+0x01,
+0x83,
+0x08,
+0x54,
+0x07,
+0xd1,
+0x83,
+0x41,
+0x54,
+0x08,
+0x41,
+0x83,
+0x0b,
+0x44,
+0x48,
+0x83,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x03,
+0xae,
+0x08,
+0x9a,
+0x00,
+0x24,
+0xc2,
+0x40,
+0x18,
+0x34,
+0x96,
+0x82,
+0x00,
+0x1d,
+0xd2,
+0x02,
+0x81,
+0x79,
+0x80,
+0x04,
+0x29,
+0x80,
+0x4b,
+0x44,
+0x60,
+0x00,
+0x00,
+0x40,
+0xba,
+0x14,
+0x88,
+0x40,
+0x40,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x2e,
+0x38,
+0x13,
+0x22,
+0x58,
+0xb0,
+0xbc,
+0x05,
+0x0b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0xcc,
+0x21,
+0x68,
+0x20,
+0x01,
+0xe1,
+0x24,
+0x5c,
+0x81,
+0x08,
+0x48,
+0x02,
+0x5c,
+0x82,
+0x00,
+0x20,
+0x8a,
+0x68,
+0x00,
+0x01,
+0xd0,
+0x20,
+0x30,
+0x19,
+0x04,
+0x20,
+0xec,
+0x00,
+0x00,
+0x08,
+0x02,
+0x8a,
+0x68,
+0x00,
+0x01,
+0xcc,
+0x20,
+0x55,
+0x02,
+0x92,
+0xc0,
+0xa0,
+0x58,
+0x01,
+0x00,
+0x00,
+0x52,
+0xbc,
+0x03,
+0x56,
+0xc0,
+0x00,
+0x3a,
+0x04,
+0x80,
+0x00,
+0x00,
+0x84,
+0x00,
+0x23,
+0x00,
+0xa0,
+0xbc,
+0x02,
+0x36,
+0xc0,
+0x00,
+0x3a,
+0x24,
+0x83,
+0x01,
+0xa8,
+0xbc,
+0x03,
+0x56,
+0xc0,
+0x00,
+0x3a,
+0x44,
+0x90,
+0x00,
+0x00,
+0x84,
+0x10,
+0x83,
+0x01,
+0x28,
+0xbc,
+0x04,
+0x3b,
+0xa1,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xa6,
+0x49,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x80,
+0x00,
+0xb5,
+0x70,
+0xe1,
+0x06,
+0x00,
+0x35,
+0x84,
+0x68,
+0x00,
+0x28,
+0x94,
+0x20,
+0x1d,
+0x98,
+0xe8,
+0x85,
+0x70,
+0xb8,
+0x98,
+0x20,
+0x2b,
+0x00,
+0x0c,
+0x30,
+0x8c,
+0x8b,
+0xc0,
+0x23,
+0x40,
+0x00,
+0x03,
+0x00,
+0x0a,
+0x5b,
+0x4e,
+0x08,
+0x60,
+0x83,
+0x36,
+0x80,
+0x05,
+0xb4,
+0xc0,
+0x18,
+0x00,
+0xb5,
+0xb4,
+0xa0,
+0x98,
+0x04,
+0xa5,
+0x78,
+0xdf,
+0x98,
+0x00,
+0x95,
+0x84,
+0x7c,
+0x18,
+0x04,
+0xab,
+0xc3,
+0x0b,
+0x2f,
+0x1a,
+0xd3,
+0x08,
+0xe8,
+0xbc,
+0x2d,
+0x33,
+0x20,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0xbc,
+0x04,
+0x86,
+0xc4,
+0x00,
+0x3c,
+0xc0,
+0x83,
+0x20,
+0x10,
+0xbc,
+0x08,
+0x13,
+0x20,
+0x28,
+0xbc,
+0x0f,
+0x56,
+0xc4,
+0x00,
+0x3c,
+0xa0,
+0xa2,
+0xe1,
+0xad,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x49,
+0xbc,
+0x09,
+0x73,
+0x01,
+0x28,
+0xbc,
+0x07,
+0x36,
+0xc4,
+0x00,
+0x3c,
+0x80,
+0xa2,
+0x81,
+0xad,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x49,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0x32,
+0x06,
+0x8b,
+0xc0,
+0x63,
+0x68,
+0x00,
+0x01,
+0xce,
+0x21,
+0x6c,
+0x00,
+0x03,
+0x9c,
+0x7a,
+0x84,
+0x8f,
+0xa0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0x9e,
+0x09,
+0x30,
+0x12,
+0x8b,
+0xc0,
+0x6a,
+0x38,
+0x00,
+0xd6,
+0x80,
+0x00,
+0x1c,
+0xe2,
+0x16,
+0xc0,
+0x00,
+0x39,
+0xc4,
+0x98,
+0x48,
+0xc8,
+0x68,
+0x20,
+0x00,
+0xfc,
+0x21,
+0x39,
+0x0a,
+0x18,
+0x48,
+0x08,
+0x80,
+0x24,
+0x88,
+0x40,
+0x48,
+0xa0,
+0x06,
+0x08,
+0x48,
+0x88,
+0x80,
+0x24,
+0x84,
+0x60,
+0xa4,
+0x00,
+0x2c,
+0x88,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x01,
+0x8e,
+0x88,
+0x68,
+0x00,
+0x01,
+0x54,
+0x20,
+0x68,
+0x00,
+0x01,
+0x60,
+0x21,
+0x68,
+0x00,
+0x01,
+0x6c,
+0x22,
+0x55,
+0x01,
+0x33,
+0x80,
+0x00,
+0x55,
+0x03,
+0xb0,
+0x00,
+0x7a,
+0x5d,
+0x0c,
+0x30,
+0x08,
+0x7a,
+0x32,
+0x2b,
+0x0b,
+0xff,
+0xaa,
+0x40,
+0x00,
+0x00,
+0x10,
+0x7a,
+0x68,
+0x00,
+0x01,
+0x85,
+0x22,
+0x68,
+0x00,
+0x01,
+0x90,
+0x21,
+0x68,
+0x00,
+0x01,
+0x7a,
+0x20,
+0x55,
+0x03,
+0x20,
+0x10,
+0x7a,
+0x5d,
+0x08,
+0x20,
+0x08,
+0x7a,
+0x32,
+0x26,
+0x0b,
+0xff,
+0xaa,
+0x80,
+0x07,
+0xa6,
+0x80,
+0x00,
+0x1c,
+0xc2,
+0x06,
+0x82,
+0x00,
+0x0f,
+0xc2,
+0x15,
+0xc8,
+0x20,
+0x80,
+0x07,
+0xa5,
+0xc0,
+0x16,
+0x04,
+0x0f,
+0xa8,
+0x00,
+0x7a,
+0xa0,
+0x08,
+0x08,
+0x48,
+0x0a,
+0x80,
+0x2c,
+0xa8,
+0x40,
+0x4a,
+0xa0,
+0x06,
+0x08,
+0x48,
+0x8a,
+0x80,
+0x2c,
+0xa6,
+0x80,
+0x00,
+0x1c,
+0xa2,
+0x16,
+0x80,
+0x00,
+0x61,
+0xca,
+0xc6,
+0xc4,
+0x00,
+0x3a,
+0xa0,
+0x08,
+0x02,
+0xca,
+0x6c,
+0x00,
+0x03,
+0x32,
+0x6c,
+0x84,
+0x85,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x04,
+0x88,
+0x48,
+0xd0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xa0,
+0x05,
+0x89,
+0x8e,
+0x88,
+0x94,
+0x06,
+0x4a,
+0x00,
+0x08,
+0xa0,
+0x22,
+0x18,
+0x48,
+0x60,
+0xa0,
+0xe3,
+0x09,
+0x40,
+0x64,
+0xa0,
+0x01,
+0x0a,
+0x02,
+0x01,
+0x84,
+0x86,
+0x0a,
+0x08,
+0x80,
+0x84,
+0x07,
+0xaa,
+0x06,
+0xa0,
+0x94,
+0x06,
+0x4a,
+0x04,
+0x40,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x84,
+0x0e,
+0xc6,
+0x80,
+0x00,
+0x6e,
+0x2a,
+0xcb,
+0xa1,
+0x48,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x94,
+0x5a,
+0x83,
+0x20,
+0x00,
+0xbc,
+0x0a,
+0x59,
+0x46,
+0x28,
+0x22,
+0x90,
+0x42,
+0xab,
+0xe0,
+0x23,
+0x08,
+0x06,
+0x82,
+0x00,
+0x1f,
+0xe2,
+0x19,
+0x84,
+0x28,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x21,
+0xba,
+0x01,
+0x0b,
+0xa1,
+0x40,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0xa0,
+0x05,
+0x09,
+0x40,
+0x64,
+0xa0,
+0x01,
+0x0a,
+0x02,
+0x01,
+0x84,
+0x86,
+0x0a,
+0x0e,
+0x60,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x20,
+0x64,
+0x00,
+0x01,
+0x84,
+0x47,
+0x64,
+0x00,
+0x01,
+0x71,
+0x87,
+0x39,
+0x00,
+0x89,
+0x48,
+0x0c,
+0x51,
+0xb1,
+0x21,
+0x48,
+0x0e,
+0x51,
+0xa1,
+0xb1,
+0x48,
+0x08,
+0x51,
+0x90,
+0x11,
+0x48,
+0x28,
+0x29,
+0x1a,
+0x42,
+0x90,
+0xa4,
+0x54,
+0x81,
+0x23,
+0xa1,
+0x48,
+0x84,
+0x04,
+0x80,
+0x00,
+0x00,
+0x5c,
+0x80,
+0x40,
+0x40,
+0x00,
+0x51,
+0xf0,
+0x23,
+0x07,
+0xfa,
+0x28,
+0x8a,
+0x15,
+0x1e,
+0x02,
+0x14,
+0x84,
+0x12,
+0x88,
+0xa1,
+0x51,
+0xd0,
+0x21,
+0x48,
+0x41,
+0x28,
+0x8a,
+0x15,
+0x44,
+0x40,
+0x14,
+0x84,
+0x1b,
+0xa1,
+0x48,
+0x94,
+0x86,
+0x00,
+0x00,
+0x00,
+0x94,
+0x82,
+0xc5,
+0xd4,
+0x83,
+0x14,
+0x8a,
+0xc2,
+0x32,
+0x36,
+0x3a,
+0x90,
+0x43,
+0xa9,
+0x86,
+0x54,
+0x89,
+0xa3,
+0xa1,
+0x48,
+0x94,
+0x07,
+0x40,
+0x00,
+0x00,
+0x94,
+0x03,
+0x85,
+0x15,
+0x02,
+0x30,
+0x7f,
+0xa2,
+0x88,
+0xa1,
+0x54,
+0x44,
+0x01,
+0x48,
+0x61,
+0xba,
+0x14,
+0x89,
+0x48,
+0xe0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x80,
+0x4a,
+0x02,
+0xc0,
+0x5c,
+0x80,
+0x92,
+0xbf,
+0xf0,
+0x62,
+0x00,
+0x00,
+0x00,
+0xb4,
+0x5c,
+0x81,
+0x02,
+0x40,
+0x02,
+0x5c,
+0x00,
+0x00,
+0x40,
+0x24,
+0x60,
+0x00,
+0x00,
+0x00,
+0x35,
+0x8d,
+0x05,
+0x80,
+0x00,
+0x00,
+0x94,
+0x88,
+0xc9,
+0x52,
+0xc4,
+0x00,
+0x00,
+0x0a,
+0x10,
+0x0a,
+0x8d,
+0x03,
+0x2b,
+0xb1,
+0x82,
+0xba,
+0x14,
+0x88,
+0x40,
+0x64,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x5c,
+0x80,
+0x52,
+0xbf,
+0xf0,
+0xa0,
+0x2c,
+0x0a,
+0x40,
+0x02,
+0x62,
+0x00,
+0x00,
+0x00,
+0xa4,
+0x5c,
+0x81,
+0x08,
+0x40,
+0x24,
+0x5c,
+0x80,
+0x82,
+0x06,
+0x61,
+0xbb,
+0x00,
+0x06,
+0x00,
+0x00,
+0x00,
+0x03,
+0x58,
+0xd0,
+0xd8,
+0x00,
+0x00,
+0x09,
+0x53,
+0x0c,
+0x94,
+0x94,
+0x40,
+0x00,
+0x00,
+0xa1,
+0x00,
+0xab,
+0xa1,
+0x48,
+0x84,
+0x06,
+0x4a,
+0x80,
+0x10,
+0xa0,
+0x06,
+0x19,
+0x48,
+0x28,
+0x55,
+0x5e,
+0x02,
+0x08,
+0x0a,
+0x51,
+0x90,
+0x01,
+0x50,
+0x2a,
+0x54,
+0x04,
+0x22,
+0x14,
+0x13,
+0x95,
+0x82,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x04,
+0x20,
+0xcc,
+0x08,
+0x07,
+0x69,
+0x82,
+0x22,
+0x32,
+0x0c,
+0x0b,
+0xc1,
+0x38,
+0xa0,
+0x81,
+0x02,
+0xa7,
+0x80,
+0x22,
+0x88,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x1b,
+0x56,
+0x20,
+0x00,
+0x00,
+0x08,
+0x6a,
+0x00,
+0x01,
+0x00,
+0x00,
+0x09,
+0x82,
+0x20,
+0x88,
+0x16,
+0x18,
+0x80,
+0xc8,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x18,
+0x80,
+0x80,
+0x55,
+0x08,
+0x22,
+0x08,
+0x21,
+0xbc,
+0x0d,
+0x79,
+0x49,
+0x28,
+0x42,
+0x05,
+0x39,
+0x50,
+0x60,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x8a,
+0x10,
+0x00,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x20,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x94,
+0x5a,
+0xc5,
+0x90,
+0x30,
+0x20,
+0x2c,
+0x04,
+0x20,
+0x34,
+0x2b,
+0xff,
+0x08,
+0x80,
+0x76,
+0xa0,
+0x65,
+0x98,
+0x80,
+0x36,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x01,
+0x0a,
+0x06,
+0x61,
+0x88,
+0x0e,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x08,
+0x80,
+0xa0,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x10,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x04,
+0x60,
+0x94,
+0x5a,
+0xc3,
+0x20,
+0x60,
+0xbc,
+0x16,
+0x8a,
+0xbf,
+0xf0,
+0x55,
+0x3f,
+0x23,
+0x00,
+0xfe,
+0x5b,
+0x48,
+0x13,
+0x00,
+0x28,
+0x37,
+0x08,
+0x52,
+0xe1,
+0x76,
+0x62,
+0x00,
+0x00,
+0x00,
+0x26,
+0x36,
+0x80,
+0x05,
+0x04,
+0xc9,
+0x18,
+0xeb,
+0x52,
+0xf8,
+0x12,
+0x2b,
+0x96,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x02,
+0xb9,
+0x83,
+0x88,
+0x36,
+0x10,
+0x4a,
+0x00,
+0x69,
+0x64,
+0x00,
+0x01,
+0xbd,
+0xef,
+0xa8,
+0x01,
+0x08,
+0x80,
+0x76,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xf6,
+0x83,
+0x80,
+0x64,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x1e,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x10,
+0xa0,
+0x06,
+0x19,
+0x48,
+0x28,
+0x55,
+0x5e,
+0x12,
+0x08,
+0x0a,
+0x51,
+0x90,
+0x91,
+0x50,
+0x28,
+0x68,
+0x20,
+0x00,
+0x00,
+0x2c,
+0x54,
+0x00,
+0xa1,
+0x8b,
+0x00,
+0x54,
+0x01,
+0x22,
+0x14,
+0x12,
+0x95,
+0x02,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x04,
+0x20,
+0xcc,
+0x08,
+0x07,
+0x69,
+0x82,
+0x22,
+0x32,
+0x0c,
+0x0b,
+0xc1,
+0x38,
+0xa0,
+0x81,
+0x02,
+0xa7,
+0x80,
+0x22,
+0x88,
+0x63,
+0x20,
+0x30,
+0xbc,
+0x1b,
+0x56,
+0x20,
+0x00,
+0x00,
+0x08,
+0x6a,
+0x00,
+0x01,
+0x00,
+0x00,
+0x09,
+0x82,
+0x20,
+0x88,
+0x16,
+0x18,
+0x80,
+0xc8,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x18,
+0x80,
+0x80,
+0x55,
+0x08,
+0x22,
+0x08,
+0x21,
+0xbc,
+0x0d,
+0x79,
+0x49,
+0x28,
+0x42,
+0x05,
+0x39,
+0x50,
+0x60,
+0x88,
+0x0e,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x8a,
+0x10,
+0x00,
+0x88,
+0x03,
+0x68,
+0x80,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x20,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0x94,
+0x5a,
+0x85,
+0x90,
+0x40,
+0x2b,
+0xfe,
+0x08,
+0x80,
+0x76,
+0x42,
+0x10,
+0xc2,
+0x00,
+0x01,
+0xa0,
+0x2a,
+0x03,
+0x21,
+0x80,
+0xbc,
+0x19,
+0x8a,
+0x08,
+0x71,
+0x2a,
+0x78,
+0x02,
+0x28,
+0xc4,
+0x32,
+0x02,
+0x0b,
+0xc2,
+0x45,
+0x62,
+0x00,
+0x00,
+0x00,
+0xf4,
+0xa0,
+0xa3,
+0x08,
+0x80,
+0xe0,
+0x88,
+0x16,
+0x10,
+0x00,
+0x00,
+0x88,
+0x0a,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x08,
+0x81,
+0x21,
+0x88,
+0x0a,
+0x0a,
+0x08,
+0x21,
+0x84,
+0x02,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xba,
+0xc0,
+0x88,
+0x12,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x02,
+0x1b,
+0xc1,
+0x07,
+0xa0,
+0x63,
+0x18,
+0x80,
+0x36,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x02,
+0x0a,
+0x06,
+0x41,
+0x88,
+0x0e,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xba,
+0x08,
+0x80,
+0xa0,
+0x88,
+0x03,
+0x6a,
+0x80,
+0x20,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x04,
+0x60,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x02,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x05,
+0x9a,
+0xbf,
+0xf0,
+0x94,
+0x82,
+0xc5,
+0x90,
+0x50,
+0x20,
+0xa4,
+0xa8,
+0x80,
+0x60,
+0x88,
+0x0f,
+0x64,
+0x20,
+0x3c,
+0x20,
+0x06,
+0x18,
+0x50,
+0x20,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xb6,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x0b,
+0x68,
+0x80,
+0x20,
+0xb0,
+0x02,
+0x46,
+0x40,
+0x00,
+0x1b,
+0x9c,
+0xfa,
+0x80,
+0x10,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x06,
+0x19,
+0x46,
+0x2c,
+0xb0,
+0x70,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x02,
+0x16,
+0x40,
+0x00,
+0x1d,
+0x0c,
+0x7b,
+0xa1,
+0x40,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xae,
+0x59,
+0x01,
+0x82,
+0xbf,
+0xd0,
+0xbd,
+0x7b,
+0x88,
+0x80,
+0x76,
+0x32,
+0x07,
+0x0b,
+0xd6,
+0x20,
+0x32,
+0x0b,
+0x0b,
+0xd5,
+0x40,
+0x32,
+0x0f,
+0x0b,
+0xd3,
+0x40,
+0x32,
+0x13,
+0x0b,
+0xd2,
+0x60,
+0x32,
+0x17,
+0x0b,
+0xd0,
+0x60,
+0x32,
+0x1b,
+0x0b,
+0xcf,
+0x80,
+0x32,
+0x1f,
+0x0b,
+0xce,
+0x00,
+0x32,
+0x23,
+0x0b,
+0xc0,
+0x29,
+0x38,
+0x00,
+0xf9,
+0x8e,
+0x8b,
+0x32,
+0x03,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x82,
+0x8b,
+0xcb,
+0x68,
+0x6e,
+0x00,
+0x00,
+0x0d,
+0xac,
+0x32,
+0x27,
+0x0b,
+0xcb,
+0x20,
+0x32,
+0x03,
+0x8b,
+0xc8,
+0x20,
+0x32,
+0x2b,
+0x0b,
+0xc8,
+0x00,
+0x32,
+0x03,
+0x8b,
+0xc4,
+0xf0,
+0x32,
+0x2f,
+0x0b,
+0xc4,
+0xd0,
+0x32,
+0x03,
+0x8b,
+0xc0,
+0x20,
+0x32,
+0x33,
+0x0b,
+0xd5,
+0xf1,
+0x59,
+0x05,
+0x02,
+0xc0,
+0x20,
+0xbc,
+0x2c,
+0x86,
+0x80,
+0x00,
+0x09,
+0x82,
+0x13,
+0x21,
+0xa0,
+0xbd,
+0x58,
+0x9a,
+0x08,
+0x00,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0x66,
+0x00,
+0x01,
+0xbc,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x98,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0xa0,
+0x02,
+0x08,
+0x80,
+0xe0,
+0x66,
+0x00,
+0x01,
+0xbc,
+0xa8,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xa0,
+0x6c,
+0x00,
+0x01,
+0x30,
+0x0b,
+0x51,
+0xa1,
+0xf8,
+0x40,
+0x08,
+0x23,
+0x42,
+0x45,
+0x90,
+0x1c,
+0x04,
+0x04,
+0x86,
+0xc0,
+0x00,
+0x13,
+0x04,
+0xb4,
+0x20,
+0x34,
+0xa0,
+0x16,
+0x85,
+0xc0,
+0x97,
+0x14,
+0x02,
+0xf6,
+0xc0,
+0x00,
+0x13,
+0x20,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x04,
+0x02,
+0x41,
+0xb8,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xbd,
+0x34,
+0x72,
+0x49,
+0xb8,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xbd,
+0x30,
+0x7a,
+0x40,
+0x80,
+0x80,
+0x80,
+0xa5,
+0x16,
+0x1a,
+0x08,
+0x0e,
+0x19,
+0x40,
+0x74,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x0a,
+0x1a,
+0x40,
+0x80,
+0x84,
+0x80,
+0xa2,
+0x2c,
+0x34,
+0x40,
+0x00,
+0x01,
+0x40,
+0x74,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x68,
+0x00,
+0x00,
+0x07,
+0xa0,
+0xb0,
+0x02,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x70,
+0xbd,
+0x16,
+0x73,
+0x20,
+0xa0,
+0xbc,
+0x21,
+0x03,
+0x21,
+0xa0,
+0xbd,
+0x12,
+0x93,
+0x81,
+0xc6,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x0b,
+0x25,
+0x9b,
+0x8b,
+0xc0,
+0x69,
+0x38,
+0x11,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0xc8,
+0x02,
+0xc2,
+0xc0,
+0xbc,
+0x06,
+0x96,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x00,
+0x00,
+0x02,
+0x41,
+0xbe,
+0xa0,
+0x67,
+0x18,
+0x02,
+0x4a,
+0x68,
+0x00,
+0x08,
+0x14,
+0x22,
+0x84,
+0x06,
+0x26,
+0x82,
+0x00,
+0x22,
+0x02,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xcf,
+0x37,
+0x68,
+0x20,
+0x02,
+0x20,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xce,
+0x77,
+0x32,
+0x0a,
+0x0b,
+0xc2,
+0x00,
+0x32,
+0x1a,
+0x0b,
+0xce,
+0x39,
+0x38,
+0x1c,
+0x66,
+0xc0,
+0x00,
+0x03,
+0x60,
+0xb2,
+0x59,
+0xb8,
+0xbc,
+0x05,
+0x93,
+0x81,
+0x14,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x03,
+0x00,
+0x00,
+0x00,
+0x52,
+0xc8,
+0x02,
+0xc2,
+0xc0,
+0xbc,
+0x06,
+0x96,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x00,
+0x00,
+0x02,
+0x41,
+0xbe,
+0xa0,
+0x67,
+0x18,
+0x02,
+0x4a,
+0x68,
+0x00,
+0x08,
+0x01,
+0x22,
+0x84,
+0x06,
+0x26,
+0x82,
+0x00,
+0x21,
+0xf2,
+0x06,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xcc,
+0x57,
+0x68,
+0x20,
+0x02,
+0x1f,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xcb,
+0x97,
+0x32,
+0x0a,
+0x0b,
+0xcb,
+0x78,
+0x38,
+0x1c,
+0x76,
+0xc0,
+0x00,
+0x03,
+0x60,
+0x82,
+0x59,
+0xe0,
+0xbc,
+0xb2,
+0x83,
+0x81,
+0x12,
+0x25,
+0x88,
+0x0b,
+0xc0,
+0x69,
+0x39,
+0x2c,
+0x08,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0x05,
+0x20,
+0x68,
+0x00,
+0x07,
+0xfb,
+0x21,
+0x52,
+0x0f,
+0x38,
+0x00,
+0x61,
+0x51,
+0xb1,
+0xb2,
+0xc2,
+0x70,
+0x80,
+0x24,
+0xb0,
+0x00,
+0x00,
+0x94,
+0x02,
+0xf5,
+0x19,
+0x1f,
+0x94,
+0x0a,
+0xc2,
+0x91,
+0x3f,
+0x29,
+0x1b,
+0xe6,
+0xc4,
+0x00,
+0x43,
+0xc4,
+0xab,
+0xc9,
+0x77,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0xa0,
+0x82,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x02,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x03,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc8,
+0x17,
+0x68,
+0x00,
+0x01,
+0xd9,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc7,
+0x57,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0xa0,
+0x82,
+0x1a,
+0x00,
+0x30,
+0x88,
+0x0e,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x12,
+0x18,
+0x80,
+0xa0,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x05,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x04,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x90,
+0x40,
+0x00,
+0x03,
+0xc5,
+0x77,
+0x68,
+0x00,
+0x01,
+0xd6,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc4,
+0xb7,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0xa0,
+0x82,
+0x1a,
+0x00,
+0x40,
+0x88,
+0x0e,
+0x08,
+0x81,
+0x61,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x40,
+0x88,
+0x12,
+0x18,
+0x80,
+0xa0,
+0xa0,
+0x81,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x06,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x04,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x90,
+0x40,
+0x00,
+0x03,
+0xc2,
+0xd7,
+0x68,
+0x00,
+0x01,
+0xd5,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc2,
+0x17,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0xa0,
+0x82,
+0x18,
+0x80,
+0xe1,
+0x66,
+0x00,
+0x01,
+0xbd,
+0x48,
+0xa0,
+0x04,
+0x08,
+0x80,
+0xa0,
+0xb0,
+0x03,
+0x46,
+0x60,
+0x00,
+0x1b,
+0x9c,
+0x8a,
+0x04,
+0x80,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xb7,
+0x68,
+0x00,
+0x01,
+0xd8,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x98,
+0x80,
+0x86,
+0x80,
+0x00,
+0x00,
+0x40,
+0xa3,
+0x01,
+0xa0,
+0xbc,
+0x0b,
+0x19,
+0x46,
+0x2c,
+0xb0,
+0x78,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x05,
+0x0b,
+0x07,
+0x8e,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xfb,
+0xc7,
+0x64,
+0x00,
+0x01,
+0xea,
+0xe7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xac,
+0xb0,
+0x78,
+0x62,
+0x89,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x60,
+0x38,
+0x08,
+0x63,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xf4,
+0xa7,
+0x64,
+0x00,
+0x01,
+0xeb,
+0xe7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x07,
+0x60,
+0xab,
+0xfe,
+0x09,
+0x40,
+0x2c,
+0x59,
+0x0f,
+0x00,
+0x80,
+0x76,
+0x42,
+0x36,
+0x42,
+0x04,
+0x11,
+0x94,
+0x82,
+0xe3,
+0x23,
+0xa0,
+0xbc,
+0x17,
+0x03,
+0x23,
+0xe0,
+0xbc,
+0x7a,
+0x9a,
+0x40,
+0x40,
+0x68,
+0x00,
+0x00,
+0x08,
+0x21,
+0x40,
+0x00,
+0x01,
+0x88,
+0x09,
+0x66,
+0x00,
+0x01,
+0x2f,
+0xc8,
+0x5c,
+0x00,
+0x71,
+0x88,
+0x48,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa8,
+0x40,
+0x00,
+0x02,
+0x40,
+0x40,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc6,
+0x57,
+0x32,
+0x0b,
+0x0b,
+0xc3,
+0xe8,
+0x38,
+0x15,
+0x46,
+0x80,
+0x00,
+0x00,
+0x7a,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x8a,
+0x40,
+0x40,
+0x88,
+0x10,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x19,
+0x96,
+0x83,
+0x40,
+0x00,
+0xa2,
+0x06,
+0xc6,
+0x80,
+0x01,
+0x40,
+0x86,
+0x80,
+0x00,
+0x18,
+0x00,
+0xa5,
+0x48,
+0xd2,
+0x20,
+0x22,
+0x06,
+0xc6,
+0x80,
+0x01,
+0x44,
+0x85,
+0xc0,
+0xaa,
+0x2c,
+0x36,
+0x05,
+0xc0,
+0xb7,
+0x04,
+0x00,
+0x92,
+0x49,
+0x2d,
+0x80,
+0x24,
+0x90,
+0x00,
+0x00,
+0x84,
+0x00,
+0x92,
+0x49,
+0x2c,
+0x84,
+0x04,
+0x89,
+0x8e,
+0x48,
+0x52,
+0x4d,
+0x23,
+0xc4,
+0x3f,
+0x40,
+0x00,
+0x01,
+0x82,
+0x39,
+0x6c,
+0x68,
+0x00,
+0x14,
+0x08,
+0x68,
+0x3f,
+0xfe,
+0x7f,
+0xca,
+0x54,
+0x4d,
+0x22,
+0x02,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x14,
+0x48,
+0x5c,
+0x0a,
+0xa2,
+0xc3,
+0x60,
+0x84,
+0x00,
+0xa2,
+0x41,
+0x36,
+0x80,
+0x24,
+0xa0,
+0x00,
+0x00,
+0x84,
+0x00,
+0xa2,
+0x41,
+0x34,
+0x40,
+0x00,
+0x00,
+0x40,
+0x48,
+0x66,
+0x00,
+0x01,
+0x7c,
+0xe0,
+0x66,
+0x00,
+0x01,
+0x7d,
+0xc0,
+0x5c,
+0x0b,
+0x71,
+0x8e,
+0x48,
+0x52,
+0x0d,
+0x23,
+0xc2,
+0x7f,
+0x40,
+0x00,
+0x01,
+0x82,
+0x39,
+0x6c,
+0x68,
+0x00,
+0x36,
+0x0a,
+0x25,
+0x93,
+0x0b,
+0xc0,
+0x38,
+0x5c,
+0x00,
+0x62,
+0x40,
+0x40,
+0x98,
+0xe8,
+0x88,
+0x81,
+0x48,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x4b,
+0xc1,
+0x37,
+0x32,
+0x0b,
+0x0b,
+0xc0,
+0x78,
+0xa4,
+0x04,
+0x06,
+0x80,
+0x00,
+0x00,
+0x7a,
+0x16,
+0x60,
+0x00,
+0x1b,
+0xac,
+0x0b,
+0xc0,
+0xbf,
+0x88,
+0x13,
+0x98,
+0x81,
+0x79,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x04,
+0x20,
+0x66,
+0x00,
+0x01,
+0xb9,
+0xc8,
+0xb0,
+0x02,
+0x48,
+0x80,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x20,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x81,
+0x02,
+0xbf,
+0x70,
+0x68,
+0x20,
+0x02,
+0x21,
+0x2c,
+0x5c,
+0x07,
+0xe2,
+0x40,
+0x40,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0x22,
+0xc8,
+0x00,
+0x6c,
+0x68,
+0x20,
+0x02,
+0x23,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0x42,
+0xc8,
+0x00,
+0x6c,
+0x68,
+0x20,
+0x02,
+0x25,
+0x2c,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0x62,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x22,
+0x72,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x68,
+0x20,
+0x02,
+0x28,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0x92,
+0x18,
+0x00,
+0x61,
+0x68,
+0x20,
+0x02,
+0x2b,
+0x21,
+0x80,
+0x06,
+0x16,
+0x82,
+0x00,
+0x22,
+0xc2,
+0xc8,
+0x00,
+0x6c,
+0x80,
+0x06,
+0xc6,
+0x82,
+0x00,
+0x22,
+0xa2,
+0xc6,
+0xe0,
+0x00,
+0x00,
+0xea,
+0xe5,
+0x52,
+0x18,
+0x04,
+0x06,
+0xc5,
+0x18,
+0x40,
+0x20,
+0x5e,
+0x05,
+0x80,
+0x98,
+0x18,
+0x42,
+0x89,
+0xc0,
+0x00,
+0x42,
+0x1c,
+0x60,
+0x40,
+0x20,
+0x98,
+0x80,
+0x83,
+0x20,
+0x20,
+0xbc,
+0x32,
+0x06,
+0xe0,
+0x00,
+0x00,
+0xda,
+0xc3,
+0x20,
+0xa0,
+0xbc,
+0x22,
+0x03,
+0x21,
+0xa0,
+0xbc,
+0x2f,
+0x16,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x13,
+0x81,
+0xc4,
+0x84,
+0x80,
+0xa2,
+0x59,
+0x30,
+0xbc,
+0x05,
+0x9a,
+0x0e,
+0xc2,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x20,
+0x02,
+0x0e,
+0x23,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x9d,
+0x80,
+0x38,
+0x58,
+0x00,
+0x98,
+0xb0,
+0x23,
+0x00,
+0x80,
+0xbc,
+0x05,
+0x98,
+0x50,
+0x50,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x52,
+0x09,
+0xa2,
+0x0e,
+0x71,
+0x6c,
+0x00,
+0x00,
+0x36,
+0x48,
+0x64,
+0x00,
+0x01,
+0xba,
+0xcf,
+0xa8,
+0x09,
+0x08,
+0x80,
+0x76,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x00,
+0x42,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0xa8,
+0x09,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x09,
+0x00,
+0x00,
+0x00,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x90,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x00,
+0x0e,
+0xac,
+0xb0,
+0x78,
+0x62,
+0x89,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x60,
+0x38,
+0x10,
+0x63,
+0x01,
+0xa0,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x41,
+0x64,
+0x00,
+0x01,
+0xfd,
+0xc7,
+0x64,
+0x00,
+0x01,
+0xfc,
+0xc7,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0xab,
+0xff,
+0x0a,
+0x06,
+0x81,
+0x88,
+0x06,
+0x18,
+0x80,
+0xf6,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x02,
+0x08,
+0x80,
+0xb6,
+0xa8,
+0x01,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x04,
+0x60,
+0xab,
+0xff,
+0x08,
+0x80,
+0x76,
+0x68,
+0x20,
+0x01,
+0x8e,
+0x20,
+0x68,
+0x00,
+0x00,
+0x07,
+0x21,
+0x66,
+0x00,
+0x01,
+0xbb,
+0xa0,
+0x88,
+0x03,
+0x66,
+0x80,
+0x00,
+0x00,
+0x42,
+0x0b,
+0x00,
+0x24,
+0x64,
+0x00,
+0x01,
+0xb9,
+0xcf,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x6c,
+0x40,
+0x04,
+0x3c,
+0x00,
+0x23,
+0xe0,
+0x43,
+0x2a,
+0x20,
+0xbc,
+0x05,
+0x82,
+0x34,
+0x00,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x4f,
+0x6c,
+0x00,
+0x00,
+0xde,
+0x50,
+0x6c,
+0x00,
+0x00,
+0xdc,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x08,
+0x40,
+0x02,
+0x24,
+0x81,
+0x08,
+0x40,
+0x50,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x3e,
+0x0a,
+0x32,
+0x83,
+0x06,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x05,
+0x1e,
+0x1a,
+0x3c,
+0x29,
+0x82,
+0x88,
+0x36,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x02,
+0x5b,
+0x44,
+0x0b,
+0x00,
+0xf8,
+0x37,
+0x04,
+0x52,
+0xe1,
+0x47,
+0x68,
+0x00,
+0x13,
+0x8d,
+0x09,
+0x62,
+0x00,
+0x00,
+0x00,
+0x27,
+0x50,
+0x4e,
+0x49,
+0x8e,
+0xb5,
+0x5b,
+0x4a,
+0x02,
+0xc0,
+0x20,
+0x2f,
+0x80,
+0x92,
+0x99,
+0x50,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x2b,
+0x98,
+0x34,
+0x93,
+0x61,
+0x45,
+0x46,
+0x08,
+0x8b,
+0x01,
+0x00,
+0x08,
+0x12,
+0x05,
+0x1a,
+0x1a,
+0x18,
+0x38,
+0x26,
+0x80,
+0x00,
+0x0c,
+0xd2,
+0x06,
+0xc0,
+0x00,
+0x19,
+0x60,
+0xa5,
+0x20,
+0x1b,
+0x00,
+0x04,
+0x28,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x19,
+0x64,
+0xa6,
+0x80,
+0x00,
+0x06,
+0xe2,
+0xc4,
+0x20,
+0x8f,
+0x80,
+0x04,
+0x88,
+0x40,
+0xec,
+0x68,
+0x00,
+0x00,
+0xcd,
+0x20,
+0x39,
+0x02,
+0x06,
+0xc4,
+0x00,
+0x44,
+0x00,
+0x85,
+0x94,
+0x10,
+0x00,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x0d,
+0xc7,
+0xa4,
+0x20,
+0x1c,
+0x80,
+0x07,
+0xa8,
+0x40,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xcb,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x04,
+0x40,
+0x0a,
+0x32,
+0x83,
+0x06,
+0x80,
+0x03,
+0xff,
+0xfc,
+0x05,
+0x1e,
+0x1a,
+0x3c,
+0x29,
+0x82,
+0x88,
+0x36,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x02,
+0x5b,
+0x44,
+0x0b,
+0x00,
+0xf8,
+0x37,
+0x04,
+0x52,
+0xe1,
+0x47,
+0x68,
+0x00,
+0x13,
+0x8d,
+0x09,
+0x62,
+0x00,
+0x00,
+0x00,
+0x27,
+0x50,
+0x4e,
+0x49,
+0x8e,
+0xb5,
+0x5b,
+0x4a,
+0x02,
+0xc0,
+0x20,
+0x2f,
+0x80,
+0x92,
+0x99,
+0x50,
+0x32,
+0x00,
+0x0b,
+0xc0,
+0x2b,
+0x98,
+0x34,
+0x93,
+0x61,
+0x45,
+0x46,
+0x08,
+0x8b,
+0x01,
+0x00,
+0x08,
+0x12,
+0x05,
+0x1a,
+0x1a,
+0x18,
+0x38,
+0x26,
+0x80,
+0x00,
+0x0d,
+0x22,
+0x06,
+0xc0,
+0x00,
+0x19,
+0x60,
+0xa5,
+0x20,
+0x1b,
+0x00,
+0x04,
+0x28,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x19,
+0x64,
+0xa6,
+0x80,
+0x00,
+0x06,
+0xf2,
+0xc4,
+0x20,
+0x8f,
+0x80,
+0x04,
+0x88,
+0x40,
+0xec,
+0x68,
+0x00,
+0x00,
+0xd2,
+0x20,
+0x39,
+0x02,
+0x06,
+0xc4,
+0x00,
+0x43,
+0xe0,
+0x85,
+0x94,
+0x10,
+0x00,
+0x07,
+0xa8,
+0x00,
+0x7a,
+0x84,
+0x0f,
+0xa6,
+0xc0,
+0x00,
+0x0d,
+0xe7,
+0xa4,
+0x20,
+0x1c,
+0x80,
+0x07,
+0xa8,
+0x40,
+0xe0,
+0x6c,
+0x00,
+0x01,
+0x96,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xcb,
+0xa1,
+0x01,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x09,
+0x5d,
+0x0a,
+0x23,
+0x01,
+0xc6,
+0x52,
+0x0d,
+0x72,
+0xbf,
+0xe0,
+0x59,
+0x01,
+0x00,
+0x80,
+0xfa,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x4a,
+0x88,
+0x17,
+0x64,
+0x20,
+0x74,
+0x18,
+0xe8,
+0x95,
+0x50,
+0x16,
+0x08,
+0x04,
+0x89,
+0x8e,
+0x88,
+0x88,
+0x00,
+0x93,
+0x20,
+0x68,
+0xbc,
+0x07,
+0x8b,
+0x00,
+0x0d,
+0x88,
+0x00,
+0x95,
+0x90,
+0x54,
+0x30,
+0x00,
+0xdb,
+0xc0,
+0x29,
+0x98,
+0x24,
+0x89,
+0x8e,
+0x89,
+0x88,
+0x0c,
+0x93,
+0x20,
+0x68,
+0xbc,
+0x0c,
+0x98,
+0x81,
+0xc8,
+0x66,
+0x00,
+0x00,
+0xe8,
+0xe0,
+0x68,
+0x00,
+0x00,
+0x46,
+0x20,
+0x5c,
+0x81,
+0x0a,
+0xc2,
+0x20,
+0x80,
+0x28,
+0x98,
+0x00,
+0x08,
+0x57,
+0x09,
+0x40,
+0x81,
+0x88,
+0x84,
+0x04,
+0x03,
+0x20,
+0x60,
+0x40,
+0x00,
+0x03,
+0xc0,
+0xa1,
+0x66,
+0x00,
+0x00,
+0xe9,
+0xa0,
+0x68,
+0x00,
+0x00,
+0x5a,
+0x20,
+0x5c,
+0x81,
+0x0a,
+0xc2,
+0x20,
+0x80,
+0x28,
+0x98,
+0x00,
+0x08,
+0x2e,
+0x12,
+0x88,
+0x40,
+0x40,
+0x98,
+0xe8,
+0x86,
+0x80,
+0x00,
+0x05,
+0x72,
+0x06,
+0x80,
+0x00,
+0x06,
+0xb2,
+0x16,
+0x60,
+0x00,
+0x04,
+0xa2,
+0x88,
+0x80,
+0x89,
+0xb0,
+0x00,
+0xc6,
+0x80,
+0x00,
+0x05,
+0x72,
+0x06,
+0x80,
+0x00,
+0x06,
+0xb2,
+0x16,
+0x60,
+0x00,
+0x04,
+0xa2,
+0x88,
+0x81,
+0x89,
+0x88,
+0x00,
+0x93,
+0x20,
+0x28,
+0xbc,
+0x11,
+0x16,
+0x80,
+0x00,
+0x01,
+0xb2,
+0x03,
+0x81,
+0xc5,
+0x84,
+0x00,
+0x82,
+0x49,
+0x64,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x0a,
+0x52,
+0x4b,
+0xa8,
+0x40,
+0x48,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x49,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x42,
+0x03,
+0x7a,
+0x06,
+0xc0,
+0x40,
+0x00,
+0x00,
+0x40,
+0x6c,
+0x68,
+0x00,
+0x08,
+0x40,
+0x20,
+0x6c,
+0x00,
+0x00,
+0x0a,
+0x60,
+0x00,
+0x00,
+0x08,
+0x81,
+0x36,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x20,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0x56,
+0x20,
+0x68,
+0x20,
+0x00,
+0x82,
+0x22,
+0x84,
+0x00,
+0x88,
+0x50,
+0x0a,
+0x58,
+0x0d,
+0x02,
+0xbf,
+0xf0,
+0x68,
+0x20,
+0x00,
+0x81,
+0x21,
+0xbc,
+0x03,
+0x88,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x68,
+0x00,
+0x00,
+0x6a,
+0x20,
+0x6c,
+0x40,
+0x01,
+0x04,
+0x08,
+0x84,
+0x00,
+0xa3,
+0x01,
+0x30,
+0x68,
+0x20,
+0x00,
+0x82,
+0x22,
+0x40,
+0x00,
+0x03,
+0xc0,
+0x48,
+0x68,
+0x20,
+0x00,
+0x81,
+0x21,
+0x66,
+0x00,
+0x01,
+0x6e,
+0x40,
+0x6c,
+0x00,
+0x00,
+0xac,
+0x08,
+0x6c,
+0x40,
+0x01,
+0x04,
+0x0a,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x41,
+0x6c,
+0x00,
+0x00,
+0xd4,
+0x08,
+0x30,
+0x1a,
+0x0b,
+0xc0,
+0x60,
+0x68,
+0x00,
+0x08,
+0x40,
+0x20,
+0x40,
+0x00,
+0x03,
+0xc1,
+0x1f,
+0x6c,
+0x00,
+0x00,
+0x0a,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x66,
+0xc4,
+0x00,
+0x44,
+0x20,
+0x05,
+0x24,
+0x82,
+0x04,
+0x04,
+0xaa,
+0x06,
+0xc0,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x6c,
+0x40,
+0x04,
+0x42,
+0x48,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x88,
+0x03,
+0x6b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xf2,
+0x59,
+0x60,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x2d,
+0x52,
+0x4b,
+0xc3,
+0xc0,
+0x6f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x52,
+0x0b,
+0xc3,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x44,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x52,
+0x0d,
+0x22,
+0xbf,
+0xf0,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x20,
+0x25,
+0x96,
+0x06,
+0xc4,
+0x00,
+0x44,
+0x64,
+0x8b,
+0xc1,
+0x28,
+0x5c,
+0x08,
+0xa9,
+0x40,
+0x2f,
+0x52,
+0x4b,
+0xc3,
+0x01,
+0x2c,
+0x5c,
+0x81,
+0x01,
+0x40,
+0xad,
+0x25,
+0x92,
+0x8b,
+0xc3,
+0xb9,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x99,
+0x21,
+0x24,
+0x92,
+0x88,
+0x0a,
+0x7a,
+0x84,
+0x87,
+0xaa,
+0x09,
+0x89,
+0x94,
+0x86,
+0x0b,
+0xc3,
+0x17,
+0x25,
+0x97,
+0x8b,
+0xc1,
+0x60,
+0x38,
+0x10,
+0x82,
+0x58,
+0x20,
+0xbc,
+0x04,
+0x06,
+0xc0,
+0x00,
+0x0e,
+0xa7,
+0xa6,
+0xc0,
+0x00,
+0x0f,
+0xe7,
+0xa2,
+0x59,
+0x60,
+0xbc,
+0x08,
+0x16,
+0xc4,
+0x00,
+0x25,
+0x60,
+0x82,
+0x31,
+0x24,
+0x6c,
+0x40,
+0x02,
+0x0a,
+0x48,
+0x23,
+0x16,
+0x46,
+0xc0,
+0x00,
+0x14,
+0xc4,
+0x82,
+0x41,
+0x78,
+0x6c,
+0x00,
+0x01,
+0x4e,
+0x7a,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x43,
+0xe0,
+0x86,
+0xc4,
+0x00,
+0x44,
+0x00,
+0x92,
+0x91,
+0x64,
+0x32,
+0x82,
+0x0b,
+0xc0,
+0xc0,
+0x40,
+0x00,
+0x00,
+0x80,
+0x76,
+0x66,
+0x00,
+0x01,
+0x60,
+0x40,
+0x5c,
+0x0e,
+0x30,
+0x80,
+0x36,
+0x68,
+0x00,
+0x00,
+0xa4,
+0x20,
+0x6c,
+0x40,
+0x04,
+0x3e,
+0x7a,
+0x6c,
+0x40,
+0x04,
+0x40,
+0x7a,
+0x68,
+0x00,
+0x00,
+0x6e,
+0x21,
+0x00,
+0x00,
+0x08,
+0x48,
+0xfa,
+0x84,
+0x87,
+0xa3,
+0x81,
+0x1d,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x0b,
+0x25,
+0x97,
+0x8b,
+0xc0,
+0x68,
+0x5c,
+0x09,
+0xa1,
+0x40,
+0xad,
+0x52,
+0x49,
+0x43,
+0xc0,
+0xcf,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0x6c,
+0x40,
+0x02,
+0xc2,
+0x00,
+0x52,
+0x09,
+0x53,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x01,
+0x50,
+0x50,
+0x6c,
+0x40,
+0x01,
+0x80,
+0x50,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe2,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9b,
+0xc8,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x46,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xca,
+0x80,
+0x10,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x00,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xa2,
+0x58,
+0x20,
+0x6c,
+0x40,
+0x04,
+0x48,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x20,
+0x52,
+0x00,
+0x83,
+0xc0,
+0x5f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x24,
+0x81,
+0x06,
+0xe0,
+0x00,
+0x14,
+0x86,
+0x03,
+0xa1,
+0x00,
+0x22,
+0x84,
+0x56,
+0x82,
+0x00,
+0x0d,
+0x72,
+0x02,
+0xa8,
+0xe8,
+0x51,
+0x84,
+0x00,
+0x40,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x21,
+0x52,
+0x4d,
+0x21,
+0x84,
+0x28,
+0xa0,
+0x02,
+0x08,
+0x48,
+0x00,
+0x52,
+0x4c,
+0x01,
+0xc0,
+0x00,
+0x84,
+0x00,
+0xa6,
+0xc4,
+0x00,
+0x44,
+0x84,
+0x86,
+0xc4,
+0x00,
+0x19,
+0x84,
+0xa6,
+0xc4,
+0x00,
+0x19,
+0xa4,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0xa4,
+0xa6,
+0xc4,
+0x00,
+0x31,
+0x44,
+0xa6,
+0x80,
+0x00,
+0x6e,
+0x6a,
+0xca,
+0x0e,
+0xc0,
+0x46,
+0x0a,
+0x40,
+0x48,
+0x50,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x08,
+0x5c,
+0x0e,
+0x32,
+0xc0,
+0x20,
+0x68,
+0x20,
+0x00,
+0xe0,
+0x20,
+0x60,
+0x00,
+0x00,
+0x00,
+0x2c,
+0x52,
+0x0d,
+0x22,
+0xbf,
+0xf0,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x48,
+0x80,
+0x07,
+0xa6,
+0x82,
+0x00,
+0x0e,
+0x82,
+0x06,
+0xc4,
+0x00,
+0x1f,
+0xa0,
+0xa5,
+0xc8,
+0x30,
+0x08,
+0x07,
+0x66,
+0x82,
+0x00,
+0x0e,
+0x22,
+0x18,
+0x00,
+0x4a,
+0x80,
+0x84,
+0xa8,
+0x40,
+0x4a,
+0x84,
+0x84,
+0xa6,
+0x80,
+0x00,
+0x0a,
+0xc2,
+0x06,
+0x60,
+0x00,
+0x04,
+0xe0,
+0x83,
+0x82,
+0xc4,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4e,
+0x08,
+0x5c,
+0x08,
+0x23,
+0x80,
+0x00,
+0x68,
+0x00,
+0x00,
+0xac,
+0x20,
+0x6c,
+0x40,
+0x01,
+0xfa,
+0x08,
+0x5c,
+0x81,
+0x02,
+0x00,
+0x40,
+0x6c,
+0x40,
+0x01,
+0xf8,
+0x0a,
+0x5c,
+0x83,
+0x08,
+0x40,
+0xc8,
+0x80,
+0x04,
+0xa6,
+0x80,
+0x00,
+0x09,
+0x62,
+0x1a,
+0x01,
+0x20,
+0x80,
+0x04,
+0xa8,
+0x02,
+0xc8,
+0xa0,
+0x82,
+0x28,
+0x40,
+0x61,
+0x40,
+0x00,
+0x00,
+0x45,
+0x62,
+0x68,
+0x00,
+0x00,
+0xc2,
+0x20,
+0x68,
+0x00,
+0x00,
+0xc6,
+0x21,
+0x66,
+0x00,
+0x00,
+0x45,
+0x00,
+0x68,
+0x00,
+0x00,
+0xd7,
+0x20,
+0x66,
+0x00,
+0x00,
+0x4c,
+0x28,
+0x5c,
+0x00,
+0x63,
+0x80,
+0x00,
+0x68,
+0x00,
+0x01,
+0x2d,
+0x20,
+0x5c,
+0x82,
+0x00,
+0x80,
+0x36,
+0x68,
+0x00,
+0x08,
+0x97,
+0x2c,
+0x68,
+0x00,
+0x01,
+0x00,
+0x08,
+0xa0,
+0x0e,
+0x06,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0xc4,
+0x60,
+0xa4,
+0x00,
+0x24,
+0x88,
+0x40,
+0x48,
+0x40,
+0x00,
+0x02,
+0x80,
+0x10,
+0x68,
+0x00,
+0x01,
+0x32,
+0x20,
+0x00,
+0x00,
+0x08,
+0x40,
+0x08,
+0x84,
+0x10,
+0xa2,
+0x91,
+0xa4,
+0x32,
+0x02,
+0x0b,
+0xc3,
+0x31,
+0x68,
+0x00,
+0x00,
+0xb2,
+0x20,
+0x68,
+0x00,
+0x00,
+0x9e,
+0x21,
+0x8c,
+0x03,
+0x25,
+0x15,
+0x49,
+0x20,
+0x14,
+0x2a,
+0x15,
+0x80,
+0x6c,
+0x40,
+0x02,
+0x12,
+0x08,
+0x84,
+0x00,
+0xa5,
+0x84,
+0x98,
+0x0d,
+0x03,
+0x05,
+0x15,
+0x40,
+0x04,
+0x84,
+0xeb,
+0xc0,
+0x8d,
+0x84,
+0x8c,
+0xc3,
+0x81,
+0xce,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x09,
+0x24,
+0x1a,
+0xe6,
+0xc4,
+0x00,
+0x44,
+0xa4,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x45,
+0x0a,
+0x30,
+0x93,
+0x0b,
+0xc0,
+0x65,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x08,
+0x38,
+0x1d,
+0x62,
+0x41,
+0xa4,
+0x6c,
+0x40,
+0x04,
+0x4a,
+0x48,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x66,
+0xc4,
+0x00,
+0x44,
+0xa0,
+0x95,
+0x24,
+0x96,
+0x04,
+0x04,
+0xa6,
+0xc4,
+0x00,
+0x44,
+0xa4,
+0x86,
+0x80,
+0x00,
+0x6e,
+0x6a,
+0xc4,
+0x60,
+0xa4,
+0x20,
+0x6c,
+0x08,
+0x40,
+0x6c,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x40,
+0x6c,
+0x40,
+0x04,
+0x52,
+0x08,
+0x68,
+0x00,
+0x47,
+0xff,
+0xca,
+0x68,
+0x38,
+0x1c,
+0x04,
+0x20,
+0x54,
+0x4d,
+0x23,
+0x00,
+0x08,
+0x5c,
+0x00,
+0x20,
+0x40,
+0x48,
+0x5c,
+0x09,
+0xf2,
+0x00,
+0x20,
+0x84,
+0x07,
+0xa8,
+0x40,
+0xd0,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x70,
+0x38,
+0x10,
+0x00,
+0x25,
+0x98,
+0x0b,
+0xc0,
+0x50,
+0x6c,
+0x70,
+0x38,
+0x00,
+0x08,
+0xbc,
+0x05,
+0xf6,
+0xc4,
+0x00,
+0x45,
+0x24,
+0x83,
+0x22,
+0xa0,
+0xbf,
+0xf5,
+0xa2,
+0xa0,
+0x64,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x38,
+0x1c,
+0x48,
+0x40,
+0x0a,
+0x24,
+0x93,
+0x48,
+0x40,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x68,
+0x00,
+0x00,
+0x07,
+0x60,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xac,
+0x5c,
+0x09,
+0xf1,
+0x40,
+0x2d,
+0x52,
+0x0d,
+0x03,
+0x00,
+0xec,
+0x30,
+0x12,
+0x86,
+0xe0,
+0x00,
+0x14,
+0x8e,
+0x0b,
+0xc0,
+0x89,
+0xa0,
+0x44,
+0x86,
+0xc4,
+0x00,
+0x45,
+0x80,
+0x83,
+0x81,
+0x05,
+0x52,
+0x0b,
+0x23,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x48,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x21,
+0x39,
+0x7e,
+0x08,
+0x48,
+0x08,
+0x9c,
+0x80,
+0x18,
+0x48,
+0x09,
+0x68,
+0x00,
+0x08,
+0xb9,
+0xac,
+0x6c,
+0x00,
+0x03,
+0x8a,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xa8,
+0x49,
+0xa0,
+0xc4,
+0x14,
+0x60,
+0xa4,
+0x04,
+0x06,
+0xc8,
+0x48,
+0x4a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x00,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x20,
+0x5c,
+0x08,
+0x0b,
+0x00,
+0x0a,
+0x68,
+0x38,
+0x08,
+0x00,
+0x21,
+0x52,
+0xc2,
+0x00,
+0x40,
+0x52,
+0x5c,
+0xbf,
+0x03,
+0xc1,
+0x28,
+0x51,
+0xe0,
+0x21,
+0xc0,
+0x00,
+0x68,
+0x00,
+0x1f,
+0xff,
+0xc0,
+0x54,
+0x41,
+0x03,
+0x04,
+0x01,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x51,
+0x51,
+0xd0,
+0x23,
+0x07,
+0xfb,
+0x54,
+0x47,
+0x0a,
+0x08,
+0x41,
+0x54,
+0x46,
+0x00,
+0x48,
+0x7a,
+0x42,
+0x07,
+0xf8,
+0x48,
+0xd1,
+0x40,
+0x00,
+0x00,
+0x48,
+0xd0,
+0x68,
+0x00,
+0x3f,
+0xc0,
+0x00,
+0x54,
+0x41,
+0x03,
+0x01,
+0x79,
+0x52,
+0x02,
+0x0b,
+0x07,
+0xf8,
+0x6c,
+0x70,
+0x10,
+0x00,
+0x7a,
+0x51,
+0xd0,
+0x62,
+0x08,
+0x41,
+0x54,
+0x41,
+0x00,
+0x48,
+0x7a,
+0x84,
+0x8d,
+0x03,
+0x80,
+0x10,
+0x68,
+0x00,
+0x08,
+0xc7,
+0x2c,
+0x84,
+0x05,
+0x06,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0xc4,
+0x60,
+0xa4,
+0x20,
+0x50,
+0x08,
+0x40,
+0x52,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x08,
+0x5c,
+0x00,
+0x43,
+0x01,
+0x06,
+0x6c,
+0x70,
+0x10,
+0x0e,
+0x50,
+0x25,
+0x9a,
+0x0b,
+0xc0,
+0x61,
+0x6c,
+0x70,
+0x10,
+0x02,
+0x08,
+0x51,
+0xb1,
+0x23,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x58,
+0x48,
+0x68,
+0x00,
+0x08,
+0xcc,
+0x20,
+0xba,
+0x14,
+0x86,
+0xc0,
+0x00,
+0x00,
+0xa6,
+0x00,
+0x00,
+0x00,
+0x68,
+0x38,
+0x08,
+0x0f,
+0x20,
+0x6c,
+0x00,
+0x03,
+0x8a,
+0x08,
+0x5c,
+0xbf,
+0x00,
+0x40,
+0x48,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x21,
+0x5c,
+0x0e,
+0x21,
+0xc0,
+0x00,
+0x5c,
+0x01,
+0xe8,
+0x48,
+0x0b,
+0x6c,
+0x00,
+0x03,
+0xa8,
+0x0a,
+0x52,
+0x49,
+0xe0,
+0x40,
+0x4a,
+0x5c,
+0x09,
+0xf2,
+0x04,
+0x40,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xaf,
+0x52,
+0x4d,
+0xc0,
+0x48,
+0x48,
+0xa0,
+0xec,
+0x18,
+0x40,
+0x49,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x6e,
+0x00,
+0x01,
+0x48,
+0xe0,
+0xa0,
+0x58,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x86,
+0xc8,
+0x40,
+0x7a,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x08,
+0x38,
+0x1c,
+0x66,
+0xc6,
+0x80,
+0x03,
+0xc0,
+0x02,
+0x41,
+0xa4,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x48,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0xa0,
+0x68,
+0x34,
+0x00,
+0x00,
+0x20,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x08,
+0xa0,
+0x3c,
+0x08,
+0x40,
+0x00,
+0x6c,
+0x00,
+0x03,
+0xb6,
+0x48,
+0x6c,
+0x00,
+0x03,
+0xb8,
+0x50,
+0x00,
+0x00,
+0x06,
+0xc4,
+0x00,
+0x45,
+0x40,
+0x82,
+0xa8,
+0xe0,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0xe1,
+0x68,
+0x34,
+0x00,
+0x00,
+0x20,
+0x6c,
+0x00,
+0x03,
+0xb6,
+0x00,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x50,
+0xa0,
+0x3c,
+0x06,
+0xc0,
+0x00,
+0x3b,
+0x80,
+0x28,
+0x40,
+0x52,
+0x42,
+0x0f,
+0x7b,
+0x0f,
+0xf0,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x50,
+0x6c,
+0x68,
+0x00,
+0x3c,
+0x7a,
+0x68,
+0x00,
+0x04,
+0xd9,
+0x00,
+0x6c,
+0x68,
+0x00,
+0x00,
+0x02,
+0x54,
+0x40,
+0x83,
+0x01,
+0x0a,
+0x25,
+0x8a,
+0x06,
+0xc6,
+0x80,
+0x00,
+0x05,
+0x0b,
+0xc0,
+0x48,
+0xb0,
+0xff,
+0x2b,
+0xc0,
+0xef,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x52,
+0x5c,
+0x08,
+0x03,
+0x1f,
+0xf2,
+0x25,
+0x82,
+0x0b,
+0xc0,
+0x89,
+0x6c,
+0x68,
+0x08,
+0x02,
+0x52,
+0x38,
+0x11,
+0x06,
+0xc6,
+0x80,
+0x03,
+0xa0,
+0x22,
+0x40,
+0x10,
+0x6c,
+0x68,
+0x00,
+0x3a,
+0x50,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x00,
+0x24,
+0x98,
+0x68,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x54,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x08,
+0x5c,
+0x0e,
+0x33,
+0x01,
+0x05,
+0x24,
+0x1a,
+0x46,
+0xe0,
+0x00,
+0x14,
+0x82,
+0xf2,
+0x59,
+0x60,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x48,
+0xbc,
+0x05,
+0x83,
+0x81,
+0x3d,
+0x52,
+0x4b,
+0xc3,
+0xc0,
+0x6f,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x52,
+0x0b,
+0xc3,
+0x80,
+0x00,
+0x6e,
+0x00,
+0x01,
+0x48,
+0x60,
+0x68,
+0x00,
+0x00,
+0x1b,
+0x20,
+0x24,
+0x9a,
+0x48,
+0x40,
+0x09,
+0x24,
+0x9a,
+0xe8,
+0x40,
+0x4a,
+0x6c,
+0x40,
+0x04,
+0x56,
+0x48,
+0x68,
+0x00,
+0x06,
+0xe6,
+0xac,
+0x46,
+0x0a,
+0x42,
+0x06,
+0xc0,
+0x84,
+0x06,
+0xc0,
+0x00,
+0x00,
+0x6c,
+0x00,
+0x03,
+0xca,
+0x00,
+0x32,
+0x00,
+0x0b,
+0xc5,
+0xc0,
+0x5c,
+0x08,
+0x12,
+0xc0,
+0x21,
+0x25,
+0x88,
+0x06,
+0x80,
+0x00,
+0x1d,
+0xd2,
+0x0b,
+0xc0,
+0x88,
+0x5c,
+0x85,
+0x00,
+0x00,
+0xa1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x42,
+0x07,
+0x78,
+0x48,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xcc,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe1,
+0x22,
+0x00,
+0x00,
+0x08,
+0x10,
+0x23,
+0x00,
+0x00,
+0x08,
+0x58,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0x74,
+0x00,
+0x00,
+0x08,
+0x48,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0xf4,
+0x5c,
+0x08,
+0x52,
+0x00,
+0x01,
+0x25,
+0x88,
+0x0b,
+0xc0,
+0x68,
+0x80,
+0x0a,
+0x20,
+0x00,
+0x00,
+0x42,
+0x07,
+0x78,
+0x50,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xce,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe2,
+0x23,
+0x00,
+0x00,
+0x08,
+0x18,
+0x24,
+0x00,
+0x00,
+0x08,
+0x60,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x58,
+0x74,
+0x00,
+0x00,
+0x08,
+0x50,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x58,
+0xf4,
+0x38,
+0x11,
+0x22,
+0x58,
+0x80,
+0xbc,
+0x07,
+0x88,
+0x48,
+0xa1,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x42,
+0x07,
+0x78,
+0x48,
+0x02,
+0x6c,
+0x00,
+0x03,
+0xd0,
+0x52,
+0x68,
+0x00,
+0x01,
+0xe3,
+0x22,
+0x00,
+0x00,
+0x08,
+0x10,
+0x23,
+0x00,
+0x00,
+0x08,
+0x58,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0x74,
+0x00,
+0x00,
+0x08,
+0x48,
+0x02,
+0x22,
+0xc1,
+0x49,
+0x50,
+0xf4,
+0x38,
+0x11,
+0xa2,
+0x58,
+0x80,
+0xbc,
+0x07,
+0x88,
+0x40,
+0xa0,
+0x00,
+0x00,
+0x04,
+0x60,
+0xa4,
+0x04,
+0x00,
+0x06,
+0xc0,
+0x00,
+0x3d,
+0x25,
+0x00,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0xe4,
+0x21,
+0x00,
+0x00,
+0x08,
+0x08,
+0x22,
+0x00,
+0x00,
+0x08,
+0x50,
+0x00,
+0x22,
+0xc0,
+0x49,
+0x48,
+0x74,
+0x00,
+0x00,
+0x08,
+0x40,
+0x00,
+0x22,
+0xc0,
+0x49,
+0x48,
+0xf4,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x37,
+0x08,
+0x63,
+0x70,
+0x44,
+0x2e,
+0x13,
+0x42,
+0xa0,
+0x64,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0x55,
+0x38,
+0x20,
+0x63,
+0x01,
+0xa0,
+0xbc,
+0x0e,
+0x06,
+0x20,
+0x00,
+0x00,
+0x01,
+0x45,
+0x00,
+0x88,
+0x18,
+0xeb,
+0x50,
+0x00,
+0x00,
+0x2f,
+0x80,
+0x9b,
+0xc0,
+0x13,
+0x28,
+0x00,
+0x92,
+0x09,
+0x08,
+0x57,
+0x09,
+0xa3,
+0xa1,
+0x48,
+0x20,
+0x10,
+0x95,
+0x0c,
+0x84,
+0x19,
+0x20,
+0x3b,
+0xa1,
+0x48,
+0x5b,
+0xc2,
+0x01,
+0x8e,
+0x83,
+0x00,
+0x00,
+0x0b,
+0xa1,
+0x48,
+0x55,
+0x00,
+0x59,
+0x8e,
+0x80,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x0c,
+0xe2,
+0xbf,
+0xf0,
+0x50,
+0xc8,
+0x41,
+0x00,
+0x59,
+0x6a,
+0x00,
+0x03,
+0xff,
+0xc1,
+0x54,
+0x42,
+0x0b,
+0x00,
+0x0d,
+0x6a,
+0x00,
+0x03,
+0xff,
+0x80,
+0x30,
+0x80,
+0x8b,
+0xc0,
+0x79,
+0x2a,
+0x02,
+0xe6,
+0xe0,
+0x00,
+0x3d,
+0x81,
+0x19,
+0x00,
+0x13,
+0x28,
+0x85,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x01,
+0x19,
+0x8e,
+0x89,
+0x51,
+0x82,
+0x89,
+0x82,
+0x8b,
+0x6e,
+0x00,
+0x03,
+0xd4,
+0x13,
+0x30,
+0x8c,
+0x8b,
+0xc0,
+0x3c,
+0x6a,
+0x00,
+0x03,
+0xff,
+0xc1,
+0x98,
+0xe8,
+0xb2,
+0x19,
+0x13,
+0x28,
+0x85,
+0x93,
+0x08,
+0x08,
+0xbc,
+0x05,
+0x16,
+0xe0,
+0x00,
+0x3d,
+0x81,
+0x02,
+0x88,
+0x10,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x11,
+0x98,
+0xe8,
+0xa3,
+0x20,
+0x30,
+0x52,
+0x08,
+0x81,
+0x00,
+0x12,
+0xbc,
+0x08,
+0x92,
+0x41,
+0x12,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x41,
+0x32,
+0x03,
+0x8b,
+0xc0,
+0x31,
+0x2a,
+0x01,
+0x0b,
+0xc0,
+0x17,
+0x2a,
+0x01,
+0x0b,
+0xa1,
+0x48,
+0xa8,
+0x01,
+0x00,
+0x00,
+0x00,
+0x5d,
+0x4a,
+0x33,
+0x01,
+0x03,
+0x68,
+0x00,
+0x01,
+0xff,
+0x4b,
+0x58,
+0x0f,
+0x83,
+0x5f,
+0xf8,
+0x50,
+0x47,
+0x19,
+0x84,
+0xc8,
+0x54,
+0x40,
+0x53,
+0x01,
+0xfe,
+0xbc,
+0x2f,
+0xa5,
+0x08,
+0xcd,
+0x18,
+0x38,
+0x33,
+0x01,
+0xe8,
+0xbc,
+0x0b,
+0x46,
+0x80,
+0x00,
+0x1f,
+0xf4,
+0xb3,
+0x01,
+0xe8,
+0xbc,
+0x0c,
+0x15,
+0x50,
+0x0f,
+0xb5,
+0x00,
+0x32,
+0x80,
+0xcb,
+0x37,
+0x8c,
+0x33,
+0x20,
+0x18,
+0xbc,
+0x06,
+0xb2,
+0xa0,
+0x3b,
+0x6e,
+0x00,
+0x03,
+0xdc,
+0x11,
+0xba,
+0x14,
+0x82,
+0x80,
+0x51,
+0x37,
+0x84,
+0x03,
+0x20,
+0x28,
+0xbc,
+0x19,
+0x33,
+0x61,
+0x45,
+0x32,
+0x02,
+0x8b,
+0xc1,
+0x20,
+0x38,
+0x20,
+0x73,
+0x01,
+0xe8,
+0xbc,
+0x05,
+0x23,
+0x28,
+0x08,
+0xbc,
+0x0d,
+0x93,
+0x88,
+0x09,
+0x38,
+0x00,
+0x1b,
+0xc0,
+0xa7,
+0x36,
+0x14,
+0x72,
+0x19,
+0x4b,
+0x28,
+0x9b,
+0xd2,
+0x11,
+0x49,
+0x32,
+0x80,
+0x8b,
+0xc0,
+0x29,
+0x38,
+0x80,
+0x93,
+0x80,
+0x01,
+0x54,
+0x82,
+0xcb,
+0x80,
+0x00,
+0x54,
+0x40,
+0x41,
+0x8e,
+0x89,
+0x40,
+0x00,
+0x01,
+0x83,
+0x03,
+0x5c,
+0x0a,
+0x73,
+0x50,
+0x00,
+0x25,
+0x19,
+0xe2,
+0x80,
+0x09,
+0x32,
+0x03,
+0x05,
+0xbc,
+0x20,
+0xbc,
+0x03,
+0x85,
+0x1d,
+0x44,
+0xb0,
+0x00,
+0xf9,
+0x8e,
+0x8b,
+0x55,
+0x43,
+0xf3,
+0x3f,
+0xff,
+0x29,
+0x9f,
+0x62,
+0x09,
+0x30,
+0x28,
+0x80,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x01,
+0x19,
+0x8e,
+0x89,
+0x50,
+0x49,
+0x43,
+0x01,
+0xa5,
+0x21,
+0x14,
+0x02,
+0x80,
+0x10,
+0x5b,
+0xc0,
+0x03,
+0xa1,
+0x48,
+0x28,
+0x04,
+0x13,
+0x78,
+0x40,
+0x32,
+0x02,
+0x0b,
+0xc1,
+0xc0,
+0x32,
+0x02,
+0x0b,
+0xc0,
+0x2a,
+0x38,
+0x00,
+0xd9,
+0x8e,
+0x89,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x20,
+0x5b,
+0x08,
+0x23,
+0x80,
+0x00,
+0x5b,
+0x88,
+0x33,
+0x00,
+0xb7,
+0x54,
+0x0f,
+0x83,
+0x01,
+0x06,
+0x68,
+0x00,
+0x01,
+0x0c,
+0x8b,
+0x57,
+0x01,
+0xcb,
+0x01,
+0xff,
+0x50,
+0x4d,
+0x53,
+0x01,
+0xa5,
+0x20,
+0x98,
+0x92,
+0x11,
+0xd2,
+0x21,
+0x14,
+0x92,
+0x19,
+0xa3,
+0x28,
+0x05,
+0x22,
+0x10,
+0x18,
+0x5b,
+0xc4,
+0x13,
+0xa1,
+0x48,
+0x28,
+0x01,
+0x03,
+0x78,
+0x00,
+0xba,
+0x14,
+0x83,
+0x80,
+0x00,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0x5c,
+0x0d,
+0x23,
+0x01,
+0xfe,
+0x21,
+0x90,
+0x26,
+0xa0,
+0x00,
+0x1f,
+0xfc,
+0x12,
+0x88,
+0x51,
+0x50,
+0xcc,
+0x11,
+0x83,
+0x4a,
+0x68,
+0x00,
+0x01,
+0x07,
+0x89,
+0x30,
+0x17,
+0x06,
+0xe0,
+0x00,
+0x3e,
+0x41,
+0x1b,
+0xc0,
+0x8c,
+0x54,
+0x42,
+0x01,
+0x83,
+0x89,
+0xb1,
+0xff,
+0xf3,
+0x01,
+0xf0,
+0xbc,
+0x0b,
+0x3b,
+0xa1,
+0x48,
+0x98,
+0xe8,
+0x80,
+0x00,
+0x00,
+0x68,
+0x00,
+0x01,
+0xff,
+0xc8,
+0x30,
+0x13,
+0x0b,
+0xc1,
+0x41,
+0x32,
+0x80,
+0x0b,
+0xc1,
+0x20,
+0x98,
+0xe8,
+0x9b,
+0xc1,
+0x07,
+0x68,
+0x00,
+0x01,
+0x0c,
+0xcb,
+0x2e,
+0x1b,
+0xe2,
+0x41,
+0x00,
+0x32,
+0x02,
+0x85,
+0x0c,
+0xc0,
+0x3c,
+0x02,
+0x89,
+0x83,
+0x08,
+0x36,
+0x10,
+0x43,
+0x20,
+0x20,
+0xbc,
+0x02,
+0xa3,
+0x80,
+0x0e,
+0x98,
+0xe8,
+0xa2,
+0x99,
+0x76,
+0x32,
+0x03,
+0x0b,
+0xc0,
+0x60,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0x49,
+0x6c,
+0x00,
+0x03,
+0xe0,
+0x08,
+0x6c,
+0x00,
+0x03,
+0xe2,
+0x08,
+0x40,
+0x00,
+0x03,
+0xa1,
+0x40,
+0x5c,
+0x0f,
+0xe3,
+0x01,
+0xa5,
+0x50,
+0xc8,
+0x42,
+0xbf,
+0xd0,
+0x50,
+0xc8,
+0x81,
+0x83,
+0x0a,
+0x50,
+0xca,
+0x59,
+0x83,
+0x0b,
+0x6a,
+0x00,
+0x01,
+0xff,
+0xc0,
+0x54,
+0x40,
+0xd9,
+0x00,
+0x5a,
+0x54,
+0xcf,
+0xb3,
+0x01,
+0x07,
+0x50,
+0xca,
+0x99,
+0x83,
+0xc9,
+0x54,
+0x40,
+0xc0,
+0x81,
+0x4a,
+0x50,
+0x4f,
+0x81,
+0x83,
+0x0a,
+0x21,
+0x10,
+0x06,
+0xe0,
+0x00,
+0x3f,
+0x01,
+0x35,
+0x40,
+0x61,
+0x90,
+0x25,
+0x86,
+0x80,
+0x00,
+0x1f,
+0xfc,
+0x86,
+0xe0,
+0x00,
+0x3e,
+0x81,
+0x03,
+0x01,
+0x28,
+0x28,
+0x81,
+0x25,
+0xbc,
+0x61,
+0xbc,
+0x76,
+0x82,
+0x88,
+0x08,
+0x30,
+0x13,
+0x0b,
+0xc1,
+0x41,
+0x32,
+0x81,
+0x0b,
+0xc0,
+0x91,
+0x20,
+0x9e,
+0x92,
+0x90,
+0x09,
+0x32,
+0x80,
+0x8b,
+0xc0,
+0xa1,
+0xba,
+0x14,
+0x86,
+0xe0,
+0x00,
+0x3e,
+0xc1,
+0x04,
+0x60,
+0x80,
+0xa8,
+0x03,
+0x09,
+0x00,
+0x12,
+0x64,
+0x00,
+0x02,
+0x44,
+0x8f,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0xba,
+0x14,
+0x8b,
+0xa1,
+0x01,
+0x55,
+0x00,
+0xc2,
+0x80,
+0x30,
+0x32,
+0x02,
+0x8b,
+0xc0,
+0xb1,
+0x32,
+0x80,
+0x0b,
+0xc0,
+0x41,
+0xba,
+0x14,
+0x89,
+0x02,
+0x10,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0x5b,
+0x80,
+0x2b,
+0x00,
+0x09,
+0x2a,
+0x5a,
+0xd2,
+0x11,
+0x40,
+0x2e,
+0x14,
+0xd3,
+0x20,
+0x30,
+0xbc,
+0x0d,
+0x13,
+0x28,
+0x10,
+0xbc,
+0x05,
+0x1b,
+0xa1,
+0x48,
+0x40,
+0x00,
+0x01,
+0x02,
+0x10,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0x5b,
+0x84,
+0x23,
+0x00,
+0x09,
+0x2a,
+0x5a,
+0x42,
+0x11,
+0x12,
+0x57,
+0x08,
+0x73,
+0x80,
+0x00,
+0x5c,
+0x0d,
+0x0b,
+0xa1,
+0x11,
+0x52,
+0x02,
+0x99,
+0x82,
+0x42,
+0x52,
+0x02,
+0x09,
+0x82,
+0x80,
+0x23,
+0x2d,
+0xb5,
+0x0c,
+0xed,
+0x98,
+0x3c,
+0xa5,
+0x19,
+0x44,
+0x98,
+0x3c,
+0x95,
+0x40,
+0x08,
+0x18,
+0x2c,
+0x35,
+0x0c,
+0x64,
+0x98,
+0x34,
+0xb4,
+0x46,
+0x89,
+0x18,
+0x34,
+0x84,
+0x41,
+0x11,
+0x19,
+0x24,
+0x14,
+0x47,
+0x19,
+0x18,
+0x4c,
+0xa5,
+0xbc,
+0x41,
+0x18,
+0x40,
+0xb5,
+0x40,
+0x44,
+0x19,
+0x2c,
+0x14,
+0x40,
+0x99,
+0x32,
+0x00,
+0xd5,
+0xbc,
+0x00,
+0x10,
+0x05,
+0xb3,
+0x08,
+0x80,
+0xbc,
+0x03,
+0xa5,
+0x40,
+0xbe,
+0xb4,
+0x00,
+0xa3,
+0x80,
+0x02,
+0x50,
+0x8c,
+0x1b,
+0x00,
+0x0c,
+0x28,
+0x0c,
+0x92,
+0x11,
+0x92,
+0x21,
+0x98,
+0x03,
+0x78,
+0x41,
+0x28,
+0x01,
+0x25,
+0x84,
+0x64,
+0x10,
+0x01,
+0x33,
+0x78,
+0x82,
+0x54,
+0x04,
+0xd3,
+0xc0,
+0x3a,
+0x5b,
+0xc4,
+0x11,
+0x82,
+0x00,
+0x98,
+0xe8,
+0x82,
+0x09,
+0xa3,
+0x32,
+0x80,
+0x85,
+0x40,
+0x69,
+0x3c,
+0x02,
+0x93,
+0x78,
+0x81,
+0x98,
+0xe8,
+0x02,
+0x09,
+0x82,
+0x29,
+0x08,
+0x92,
+0x30,
+0x4a,
+0x32,
+0x01,
+0x0b,
+0xc0,
+0x4a,
+0x46,
+0x08,
+0x08,
+0x81,
+0x08,
+0x55,
+0x3f,
+0x69,
+0x92,
+0x81,
+0x64,
+0x00,
+0x02,
+0x47,
+0xef,
+0xa8,
+0x03,
+0x03,
+0x28,
+0x00,
+0xbc,
+0x10,
+0x16,
+0x80,
+0x00,
+0x1f,
+0xfc,
+0x93,
+0x01,
+0x70,
+0xbc,
+0x02,
+0x13,
+0x28,
+0x10,
+0xbc,
+0x0a,
+0x12,
+0x09,
+0xf1,
+0x29,
+0x08,
+0x93,
+0x28,
+0x08,
+0xbc,
+0x0b,
+0x95,
+0x50,
+0x0c,
+0x3a,
+0x10,
+0x1b,
+0xa1,
+0x48,
+0x6e,
+0x00,
+0x03,
+0xec,
+0x10,
+0xa8,
+0x03,
+0x09,
+0x00,
+0x12,
+0x64,
+0x00,
+0x02,
+0x44,
+0x8f,
+0x46,
+0x08,
+0x0a,
+0x80,
+0x30,
+0xba,
+0x14,
+0x8a,
+0x80,
+0x30,
+0x40,
+0x00,
+0x03,
+0x80,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x66,
+0x66,
+0x66,
+0x66,
+0x00,
+0x3f,
+0xe6,
+0x66,
+0x66,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0xe0,
+0x00,
+0x00,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x07,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf0,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf0,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0xcc,
+0x35,
+0x1d,
+0x00,
+0x7f,
+0xfa,
+0xdb,
+0xbd,
+0x00,
+0x00,
+0xcc,
+0x35,
+0x1d,
+0x00,
+0xe0,
+0xfe,
+0x9f,
+0x8f,
+0x00,
+0x7f,
+0xaf,
+0xe1,
+0xd7,
+0x00,
+0x1f,
+0x51,
+0x7e,
+0x9b,
+0x00,
+0x8e,
+0xf8,
+0x36,
+0xb7,
+0x00,
+0x63,
+0x21,
+0x4d,
+0x57,
+0x00,
+0x72,
+0x19,
+0x84,
+0x0d,
+0x00,
+0x81,
+0xcb,
+0x2e,
+0x5d,
+0x00,
+0x7e,
+0x01,
+0xcc,
+0x7d,
+0x00,
+0x7f,
+0xcc,
+0xfa,
+0xd9,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x5a,
+0x84,
+0x71,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x32,
+0xf5,
+0x2c,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x17,
+0x0d,
+0x9e,
+0x34,
+0x00,
+0x66,
+0xa9,
+0xa3,
+0xaf,
+0x00,
+0x17,
+0x0d,
+0x9e,
+0x34,
+0x00,
+0x86,
+0xfb,
+0x10,
+0x8b,
+0x00,
+0x79,
+0x04,
+0xef,
+0x75,
+0x00,
+0x3d,
+0x00,
+0xdd,
+0x9d,
+0x00,
+0xc4,
+0x8d,
+0xa7,
+0xe7,
+0x00,
+0x3e,
+0x71,
+0x7a,
+0x7b,
+0x00,
+0x9a,
+0x10,
+0x28,
+0x3b,
+0x00,
+0x65,
+0xef,
+0xd7,
+0xc5,
+0x00,
+0x32,
+0xc7,
+0x23,
+0xb1,
+0x00,
+0xd7,
+0x00,
+0xa4,
+0x41,
+0x00,
+0x36,
+0x38,
+0x38,
+0x0f,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0xcc,
+0x35,
+0x1d,
+0x00,
+0x7f,
+0xfa,
+0xdb,
+0xbd,
+0x00,
+0x00,
+0xcc,
+0x35,
+0x1d,
+0x00,
+0xe0,
+0xfe,
+0x9f,
+0x8f,
+0x00,
+0x7f,
+0xaf,
+0xe1,
+0xd7,
+0x00,
+0x1f,
+0x51,
+0x7e,
+0x9b,
+0x00,
+0x8e,
+0xf8,
+0x36,
+0xb7,
+0x00,
+0x63,
+0x21,
+0x4d,
+0x57,
+0x00,
+0x72,
+0x19,
+0x84,
+0x0d,
+0x00,
+0x81,
+0xcb,
+0x2e,
+0x5d,
+0x00,
+0x7e,
+0x01,
+0xcc,
+0x7d,
+0x00,
+0x7f,
+0xcc,
+0xfa,
+0xd9,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x5a,
+0x84,
+0x71,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x32,
+0xf5,
+0x2c,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x17,
+0x0d,
+0x9e,
+0x34,
+0x00,
+0x66,
+0xa9,
+0xa3,
+0xaf,
+0x00,
+0x17,
+0x0d,
+0x9e,
+0x34,
+0x00,
+0x86,
+0xfb,
+0x10,
+0x8b,
+0x00,
+0x79,
+0x04,
+0xef,
+0x75,
+0x00,
+0x3d,
+0x00,
+0xdd,
+0x9d,
+0x00,
+0xc4,
+0x8d,
+0xa7,
+0xe7,
+0x00,
+0x3e,
+0x71,
+0x7a,
+0x7b,
+0x00,
+0x9a,
+0x10,
+0x28,
+0x3b,
+0x00,
+0x65,
+0xef,
+0xd7,
+0xc5,
+0x00,
+0x32,
+0xc7,
+0x23,
+0xb1,
+0x00,
+0xd7,
+0x00,
+0xa4,
+0x41,
+0x00,
+0x36,
+0x38,
+0x38,
+0x0f,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x02,
+0x03,
+0x02,
+0x00,
+0x00,
+0x02,
+0x00,
+0x01,
+0x01,
+0x00,
+0x01,
+0x01,
+0x02,
+0x03,
+0x00,
+0x00,
+0x04,
+0x9d,
+0x1d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0xff,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x07,
+0xff,
+0xff,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0xb1,
+0xca,
+0x9e,
+0xb7,
+0x00,
+0x4c,
+0x9f,
+0xb1,
+0xad,
+0x00,
+0x54,
+0x54,
+0x77,
+0x9b,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x7f,
+0xf7,
+0xc6,
+0x15,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7e,
+0x67,
+0x33,
+0xd7,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0xb1,
+0xca,
+0x9e,
+0xb7,
+0x00,
+0x4c,
+0x9f,
+0xb1,
+0xad,
+0x00,
+0x54,
+0x54,
+0x77,
+0x9b,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x7f,
+0xf7,
+0xc6,
+0x15,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7e,
+0x67,
+0x33,
+0xd7,
+0x00,
+0x00,
+0xcc,
+0x66,
+0x13,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x02,
+0x01,
+0x00,
+0x01,
+0x00,
+0x02,
+0x01,
+0x00,
+0x00,
+0x0a,
+0x47,
+0xe7,
+0x00,
+0x7f,
+0xeb,
+0x70,
+0x33,
+0x00,
+0x00,
+0x0a,
+0x47,
+0xe7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xbf,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x64,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x20,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x44,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x15,
+0xd2,
+0xf3,
+0x00,
+0x0c,
+0xc9,
+0x9a,
+0x66,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x30,
+0x00,
+0x01,
+0x20,
+0x00,
+0x00,
+0x00,
+0x00,
+0x20,
+0x00,
+0x00,
+0x00,
+0x03,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xaf,
+0x00,
+0x00,
+0x00,
+0x00,
+0x70,
+0x00,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0xe3,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x03,
+0x00,
+0x00,
+0x3c,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x61,
+0xa8,
+0x00,
+0x00,
+0xd6,
+0x82,
+0x03,
+0x00,
+0x00,
+0x00,
+0x09,
+0xc4,
+0x00,
+0x00,
+0x00,
+0x4e,
+0x20,
+0x00,
+0x00,
+0x00,
+0x13,
+0x88,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0xc4,
+0x10,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0xc3,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0d,
+0xaf,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0xc6,
+0x00,
+0x00,
+0x00,
+0x02,
+0x8c,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x64,
+0x06,
+0x40,
+0x63,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x30,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x50,
+0x00,
+0x00,
+0x00,
+0x01,
+0x50,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xcc,
+0xcc,
+0xcc,
+0x00,
+0x19,
+0x99,
+0x99,
+0x99,
+0x00,
+0x33,
+0x33,
+0x33,
+0x33,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x66,
+0x66,
+0x66,
+0x66,
+0x00,
+0x5f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x46,
+0x66,
+0x66,
+0x65,
+0x00,
+0x2a,
+0xaa,
+0xaa,
+0xaa,
+0x00,
+0x80,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x73,
+0x33,
+0x33,
+0x32,
+0x00,
+0x7a,
+0xe1,
+0x47,
+0xad,
+0x00,
+0x43,
+0xd7,
+0x00,
+0x00,
+0x00,
+0x43,
+0xd7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0xff,
+0x00,
+0x60,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x64,
+0x87,
+0xed,
+0x4a,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x04,
+0xa0,
+0x00,
+0xff,
+0xfe,
+0x5f,
+0xe7,
+0x00,
+0x00,
+0x5b,
+0x05,
+0xb0,
+0x00,
+0xf5,
+0x55,
+0x55,
+0x56,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x40,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x5f,
+0x37,
+0x59,
+0xdf,
+0x00,
+0x5f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x40,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x7f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x18,
+0x00,
+0x00,
+0x00,
+0xae,
+0xb8,
+0x0f,
+0x00,
+0x7f,
+0xf4,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xfd,
+0x00,
+0x00,
+0x00,
+0x00,
+0x41,
+0x00,
+0x00,
+0x00,
+0x00,
+0x48,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xf8,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x08,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x0c,
+0xdc,
+0x61,
+0x39,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x49,
+0xe7,
+0x60,
+0x00,
+0x0c,
+0x28,
+0xf5,
+0xc0,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0xa0,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0xf1,
+0x91,
+0x96,
+0x00,
+0x7a,
+0x1c,
+0xdc,
+0xd3,
+0x00,
+0x02,
+0xf1,
+0x91,
+0x96,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x21,
+0xc7,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7d,
+0x11,
+0x96,
+0x79,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x57,
+0x6e,
+0xb8,
+0x35,
+0x00,
+0x14,
+0x48,
+0xa3,
+0xe5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x01,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xd6,
+0xe3,
+0xb3,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x00,
+0x66,
+0x84,
+0xe5,
+0x00,
+0x7f,
+0x32,
+0xf6,
+0x37,
+0x00,
+0x00,
+0x66,
+0x84,
+0xe5,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xd6,
+0xe3,
+0xb3,
+0x00,
+0x00,
+0x14,
+0x8e,
+0x26,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x80,
+0x00,
+0x00,
+0x01,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x8c,
+0x5d,
+0xa4,
+0x55,
+0x00,
+0x67,
+0xf9,
+0x80,
+0xe3,
+0x00,
+0x74,
+0x57,
+0x25,
+0x39,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x7f,
+0xd2,
+0xc7,
+0xff,
+0x00,
+0x00,
+0x16,
+0x9c,
+0x01,
+0x00,
+0x00,
+0x68,
+0xc1,
+0x4b,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x3f,
+0xff,
+0xff,
+0xf5,
+0x00,
+0x00,
+0x00,
+0x00,
+0x06,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x04,
+0x00,
+0x00,
+0x00,
+0x00,
+0x09,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1d,
+0x53,
+0x00,
+0x00,
+0x13,
+0x9d,
+0x01,
+0x00,
+0x00,
+0x0f,
+0xff,
+0xff,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1d,
+0xa1,
+0x2f,
+0x68,
+0x00,
+0x00,
+0x00,
+0x00,
+0xfa,
+0x00,
+0x00,
+0x00,
+0xc8,
+0xd7,
+0x00,
+0x00,
+0x7d,
+0x86,
+0x6e,
+0x00,
+0x00,
+0x00,
+0x00,
+0x02,
+0x00,
+0x00,
+0x00,
+0x00,
+0x05,
+0x00,
+0x00,
+0x00,
+0x00,
+0x0a,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0xaa,
+0xb1,
+0x85,
+0x99,
+0x00,
+0x55,
+0x56,
+0xb4,
+0x51,
+0x00,
+0xad,
+0x4f,
+0xc5,
+0xad,
+0x00,
+0x7c,
+0xd3,
+0x79,
+0x2f,
+0x00,
+0x55,
+0xdc,
+0xc1,
+0x23,
+0x00,
+0x00,
+0x00,
+0x0d,
+0xba,
+0x00,
+0x00,
+0x00,
+0x44,
+0xa2,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x7f,
+0xff,
+0xff,
+0xff,
+0x00,
+0x5a,
+0x82,
+0x79,
+0x93,
+0x00,
+0xa5,
+0x7d,
+0x86,
+0x6d,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x0c,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x40,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x56,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0x7a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x98,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xb2,
+0x00,
+0x00,
+0x00,
+0x1c,
+0xe8,
+0x00,
+0x00,
+0x00,
+0x1d,
+0x00,
+0x00,
+0x00,
+0x00,
+0x1e,
+0x9c,
+0x00,
+0x00,
+0x00,
+0x20,
+0x9c,
+0x00,
+0x00,
+0x00,
+0x21,
+0x40,
+0x00,
+0x00,
+0x00,
+0x21,
+0x62,
+0x00,
+0x00,
+0x00,
+0x21,
+0xd4,
+0x00,
+0x00,
+0x00,
+0x22,
+0x08,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x1b,
+0x9a,
+0x00,
+0x00,
+0x00,
+0x22,
+0x98,
+0x00,
+0x00,
+0x00,
+0x23,
+0xa4,
+0x00,
+0x00,
+0x00,
+0x22,
+0xc0,
+0x00,
+0x00,
+0x00,
+0x22,
+0xc0,
+0x00,
+0x00,
+0x00,
+0x23,
+0x52,
+};
+
+#define VERNUM_09_17_01 0x09170108
+#define CAL_ID_09_17_01 0x00000002
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/Makefile b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/Makefile
new file mode 100644
index 0000000..18850f2
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/Makefile
@@ -0,0 +1,10 @@
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_utils
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_cpas/include
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_sensor_module/cam_res_mgr
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_req_mgr
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_sensor_module/cam_cci
+ccflags-y += -Idrivers/media/platform/msm/camera/cam_smmu/
+
+obj-$(CONFIG_CAMERA_FW_UPDATE) += fw_update.o PhoneUpdate.o
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/PhoneUpdate.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/PhoneUpdate.c
new file mode 100644
index 0000000..120292f
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/PhoneUpdate.c
@@ -0,0 +1,884 @@
+/**
+ * @brief LC898123F40 Flash update
+ *
+ * @author Copyright (C) 2017, ON Semiconductor, all right reserved.
+ *
+ **/
+
+//**************************
+// Include Header File
+//**************************
+#include "PhoneUpdate.h"
+#include "PmemCode.h"
+
+//#include "FromCode.h"
+#if MODULE_VENDOR == 4
+#include "FromCode_04_05_06.h"
+#include "FromCode_04_05_08.h"
+#elif MODULE_VENDOR == 9 && MDL_VER == 2
+#include "FromCode_09_05_06.h"
+#include "FromCode_09_05_08.h"
+#elif MODULE_VENDOR == 9 && MDL_VER == 1
+#include "FromCode_09_05_05.h"
+#include "FromCode_09_05_07.h"
+#else
+#include "FromCode_04_17_00.h"
+#include "FromCode_04_17_01.h"
+#include "FromCode_09_17_00.h"
+#include "FromCode_09_17_01.h"
+#endif
+
+
+#define USER_RESERVE 3
+#define ERASE_BLOCKS (16 - USER_RESERVE)
+#define BURST_LENGTH ( 8*5 )
+#define DMB_COEFF_ADDRESS 0x21
+#define BLOCK_UNIT 0x200
+#define BLOCK_BYTE 2560
+#define SECTOR_SIZE 320
+#define HALF_SECTOR_ADD_UNIT 0x20
+#define FLASH_ACCESS_SIZE 32
+#define USER_AREA_START (BLOCK_UNIT * ERASE_BLOCKS)
+
+#define CNT100MS 1352
+#define CNT200MS 2703
+
+//****************************************************
+// CUSTOMER NECESSARY CREATING FUNCTION LIST
+//****************************************************
+/* for I2C communication */
+extern void RamWrite32A( UINT_16, UINT_32 );
+extern void RamRead32A( UINT_16, void * );
+/* for I2C Multi Translation : Burst Mode*/
+extern void CntWrt( void *, UINT_16) ;
+extern void CntRd3( UINT_32, void *, UINT_16 ) ;
+
+/* WPB control for LC898123F40*/
+extern void WPBCtrl( UINT_8 );
+/* for Wait timer [Need to adjust for your system] */
+extern void WitTim( UINT_16 );
+
+//**************************
+// extern Function LIST
+//**************************
+UINT_32 UlBufDat[ 64 ];
+
+//**************************
+// Table of download file
+//**************************
+
+const DOWNLOAD_TBL DTbl[] = {
+#if MODULE_VENDOR == 4
+ {0x0406, CcMagicCodeF40_04_05_06, sizeof(CcMagicCodeF40_04_05_06),
+ CcFromCodeF40_04_05_06, sizeof(CcFromCodeF40_04_05_06) },
+ {0x0408, CcMagicCodeF40_04_05_08, sizeof(CcMagicCodeF40_04_05_08),
+ CcFromCodeF40_04_05_08, sizeof(CcFromCodeF40_04_05_08) },
+#elif MODULE_VENDOR == 9 && MDL_VER == 2
+ {0x0906, CcMagicCodeF40_09_05_06, sizeof(CcMagicCodeF40_09_05_06),
+ CcFromCodeF40_09_05_06, sizeof(CcFromCodeF40_09_05_06) },
+ {0x0908, CcMagicCodeF40_09_05_08, sizeof(CcMagicCodeF40_09_05_08),
+ CcFromCodeF40_09_05_08, sizeof(CcFromCodeF40_09_05_08) },
+#elif MODULE_VENDOR == 9 && MDL_VER == 1
+ {0x0905, CcMagicCodeF40_09_05_05, sizeof(CcMagicCodeF40_09_05_05),
+ CcFromCodeF40_09_05_05, sizeof(CcFromCodeF40_09_05_05) },
+ {0x0907, CcMagicCodeF40_09_05_07, sizeof(CcMagicCodeF40_09_05_07),
+ CcFromCodeF40_09_05_07, sizeof(CcFromCodeF40_09_05_07) },
+#else
+ {0x0400, CcMagicCodeF40_04_17_00, sizeof(CcMagicCodeF40_04_17_00),
+ CcFromCodeF40_04_17_00, sizeof(CcFromCodeF40_04_17_00) },
+ {0x0401, CcMagicCodeF40_04_17_01, sizeof(CcMagicCodeF40_04_17_01),
+ CcFromCodeF40_04_17_01, sizeof(CcFromCodeF40_04_17_01) },
+ {0x0900, CcMagicCodeF40_09_17_00, sizeof(CcMagicCodeF40_09_17_00),
+ CcFromCodeF40_09_17_00, sizeof(CcFromCodeF40_09_17_00) },
+ {0x0901, CcMagicCodeF40_09_17_01, sizeof(CcMagicCodeF40_09_17_01),
+ CcFromCodeF40_09_17_01, sizeof(CcFromCodeF40_09_17_00) },
+#endif
+ {0xFFFF, (void *)0, 0, (void *)0, 0}
+};
+
+//**************************
+// Local Function Prototype
+//**************************
+
+
+//********************************************************************************
+// Function Name : F40_IOWrite32A
+//********************************************************************************
+void F40_IORead32A( UINT_32 IOadrs, UINT_32 *IOdata )
+{
+ RamWrite32A( F40_IO_ADR_ACCESS, IOadrs ) ;
+ RamRead32A ( F40_IO_DAT_ACCESS, IOdata ) ;
+}
+
+//********************************************************************************
+// Function Name : F40_IOWrite32A
+//********************************************************************************
+void F40_IOWrite32A( UINT_32 IOadrs, UINT_32 IOdata )
+{
+ RamWrite32A( F40_IO_ADR_ACCESS, IOadrs ) ;
+ RamWrite32A( F40_IO_DAT_ACCESS, IOdata ) ;
+}
+
+//********************************************************************************
+// Function Name : WPB level read
+//********************************************************************************
+UINT_8 F40_ReadWPB( void )
+{
+ UINT_32 UlReadVal, UlCnt=0;
+
+ do{
+ F40_IORead32A( FLASHROM_F40_WPB, &UlReadVal );
+ if( (UlReadVal & 0x00000004) != 0 )
+ return 1;
+ WitTim( 1 );
+ }while ( UlCnt++ < 10 );
+ return 0;
+}
+
+//********************************************************************************
+// Function Name : F40_UnlockCodeSet
+//********************************************************************************
+UINT_8 F40_UnlockCodeSet( void )
+{
+ UINT_32 UlReadVal ;
+
+ WPBCtrl( WPB_OFF ) ;
+ if ( F40_ReadWPB() != 1 )
+ return 5;
+
+ F40_IOWrite32A( FLASHROM_F40_UNLK_CODE1, 0xAAAAAAAA ) ;
+ F40_IOWrite32A( FLASHROM_F40_UNLK_CODE2, 0x55555555 ) ;
+ F40_IOWrite32A( FLASHROM_F40_RSTB_FLA, 0x00000001 ) ;
+ F40_IOWrite32A( FLASHROM_F40_CLK_FLAON, 0x00000010 ) ;
+ F40_IOWrite32A( FLASHROM_F40_UNLK_CODE3, 0x0000ACD5 ) ;
+ F40_IOWrite32A( FLASHROM_F40_WPB, 0x00000001 ) ;
+ RamRead32A( F40_IO_DAT_ACCESS, &UlReadVal ) ;
+
+ if ( (UlReadVal & 0x00000007) != 7 )
+ return 1;
+
+ return 0;
+}
+
+//********************************************************************************
+// Function Name : F40_UnlockCodeClear
+//********************************************************************************
+UINT_8 F40_UnlockCodeClear(void)
+{
+ UINT_32 UlReadVal ;
+
+ F40_IOWrite32A( FLASHROM_F40_WPB, 0x00000010 ) ;
+ RamRead32A( F40_IO_DAT_ACCESS, &UlReadVal ) ;
+
+ if( (UlReadVal & 0x00000080) != 0 )
+ return 3;
+
+ WPBCtrl( WPB_ON ) ;
+
+ return 0;
+}
+
+
+
+//********************************************************************************
+// Function Name : F40_FlashBlockErase
+//********************************************************************************
+UINT_8 F40_FlashBlockErase( UINT_32 SetAddress )
+{
+ UINT_32 UlReadVal, UlCnt ;
+ UINT_8 ans = 0 ;
+
+ if( SetAddress & 0x00010000 )
+ return 9;
+
+ ans = F40_UnlockCodeSet() ;
+ if( ans != 0 )
+ return ans;
+
+ F40_IOWrite32A( FLASHROM_F40_ADR, (SetAddress & 0xFFFFFE00) ) ;
+ F40_IOWrite32A( FLASHROM_F40_CMD, 0x00000006 ) ;
+
+ WitTim( 5 ) ;
+
+ UlCnt = 0 ;
+ do {
+ if( UlCnt++ > 100 ) {
+ ans = 2 ;
+ break ;
+ }
+
+ F40_IORead32A( FLASHROM_F40_INT, &UlReadVal ) ;
+ } while( ( UlReadVal & 0x00000080 ) != 0 ) ;
+
+ F40_UnlockCodeClear() ;
+
+ return ans;
+}
+
+
+//********************************************************************************
+// Function Name : F40_FlashBurstWrite
+//********************************************************************************
+UINT_8 F40_FlashBurstWrite( const UINT_8 *NcDataVal, UINT_32 NcDataLength, UINT_32 ScNvrMan )
+{
+ UINT_32 i, j, UlCnt ;
+ UINT_8 data[163] ;
+ UINT_32 UlReadVal ;
+ UINT_8 UcOddEvn ;
+ UINT_8 Remainder ;
+
+ data[0] = 0xF0 ;
+ data[1] = 0x08 ;
+ data[2] = BURST_LENGTH ;
+
+ for( i = 0 ; i < (NcDataLength / BURST_LENGTH) ; i++ ) {
+ UlCnt = 3 ;
+
+ UcOddEvn =i % 2 ;
+ data[1] = 0x08 + UcOddEvn ;
+
+ for( j = 0 ; j < BURST_LENGTH; j++ )
+ data[UlCnt++] = *NcDataVal++ ;
+
+ CntWrt( data, BURST_LENGTH + 3 ) ;
+ RamWrite32A( 0xF00A ,(UINT_32) ( ( BURST_LENGTH / 5 ) * i + ScNvrMan) ) ;
+ RamWrite32A( 0xF00B ,(UINT_32) (BURST_LENGTH / 5) ) ;
+
+ RamWrite32A( 0xF00C , 4 + 4 * UcOddEvn ) ;
+ }
+
+ Remainder = NcDataLength % BURST_LENGTH ;
+ if( Remainder != 0 ) {
+ data[2] = Remainder ;
+ UlCnt = 3 ;
+ UcOddEvn =i % 2 ;
+ data[1] = 0x08 + UcOddEvn ;
+
+ for( j = 0 ; j < Remainder; j++ )
+ data[UlCnt++] = *NcDataVal++ ;
+
+ CntWrt( data, BURST_LENGTH + 3 ) ;
+ RamWrite32A( 0xF00A ,(UINT_32) ( ( BURST_LENGTH / 5 ) * i + ScNvrMan) ) ;
+ RamWrite32A( 0xF00B ,(UINT_32) (Remainder /5) ) ;
+ RamWrite32A( 0xF00C , 4 + 4 * UcOddEvn ) ;
+ }
+
+ UlCnt = 0 ;
+ do {
+ if( UlCnt++ > 100 )
+ return 1;
+
+ RamRead32A( 0xF00C, &UlReadVal ) ;
+ } while ( UlReadVal != 0 ) ;
+
+ return 0;
+}
+
+
+//********************************************************************************
+// Function Name : F40_FlashSectorRead
+//********************************************************************************
+void F40_FlashSectorRead( UINT_32 UlAddress, UINT_8 *PucData )
+{
+ UINT_8 UcIndex, UcNum ;
+ UINT_8 UcReadDat[ 4 ] ;
+
+ F40_IOWrite32A( FLASHROM_F40_ADR, ( UlAddress & 0xFFFFFFC0 ) ) ;
+ F40_IOWrite32A( FLASHROM_F40_ACSCNT, 63 ) ;
+ UcNum = 64 ;
+
+ F40_IOWrite32A( FLASHROM_F40_CMD, 0x00000001 ) ;
+
+ for( UcIndex = 0 ; UcIndex < UcNum ; UcIndex++ ) {
+ RamWrite32A( F40_IO_ADR_ACCESS, FLASHROM_F40_RDATH ) ;
+ RamRead32A( F40_IO_DAT_ACCESS, UcReadDat ) ;
+ *PucData++ = UcReadDat[ 0 ] ;
+ RamWrite32A( F40_IO_ADR_ACCESS, FLASHROM_F40_RDATL ) ;
+ RamRead32A( F40_IO_DAT_ACCESS, UcReadDat ) ;
+ *PucData++ = UcReadDat[ 3 ] ;
+ *PucData++ = UcReadDat[ 2 ] ;
+ *PucData++ = UcReadDat[ 1 ] ;
+ *PucData++ = UcReadDat[ 0 ] ;
+ }
+}
+
+//*****************************************************************************
+// Function Name : F40_FlashSectorErase
+//*****************************************************************************
+UINT_8 F40_FlashSectorErase(UINT_32 SetAddress)
+{
+ UINT_32 UlReadVal, UlCnt;
+ UINT_8 ans = 0;
+
+ if ((SetAddress & 0x000100FF) > 0x0001007F)
+ return 9;
+
+ if (USER_AREA_START > (SetAddress & 0xFFFFFFC0))
+ return 9;
+
+ ans = F40_UnlockCodeSet();
+ if (ans != 0)
+ return ans;
+
+ F40_IOWrite32A(FLASHROM_F40_ADR, SetAddress & 0xFFFFFFC0);
+ F40_IOWrite32A(FLASHROM_F40_CMD, 4);
+
+ WitTim(5);
+ UlCnt = 0;
+ do {
+ if (UlCnt++ > 100) {
+ ans = 2;
+ break;
+ }
+
+ F40_IORead32A(FLASHROM_F40_INT, &UlReadVal);
+ } while ((UlReadVal & 0x00000080) != 0);
+
+ ans = F40_UnlockCodeClear();
+
+ return ans;
+}
+
+//*****************************************************************************
+// Function Name : F40_FlashSectorWrite
+//*****************************************************************************
+UINT_8 F40_FlashSectorWrite(UINT_32 UlAddress, UINT_8 *PucData)
+{
+ UINT_8 UcNum, UcStatus = 0, UcSize = 0;
+ UnDwdVal UnWriteDat;
+
+ if ((UlAddress & 0x000100FF) > 0x0001007F)
+ return 9;
+
+ if (USER_AREA_START > (UlAddress & 0xFFFFFFC0))
+ return 9;
+
+ UcStatus = F40_FlashSectorErase(UlAddress & 0xFFFFFFC0);
+ if (UcStatus != 0) {
+ return UcStatus;
+ }
+
+ UcStatus = F40_UnlockCodeSet();
+ if (UcStatus != 0) {
+ return UcStatus;
+ }
+
+ do {
+ F40_IOWrite32A(FLASHROM_F40_ACSCNT, (FLASH_ACCESS_SIZE - 1));
+ F40_IOWrite32A(FLASHROM_F40_ADR, UlAddress + UcSize);
+ F40_IOWrite32A(FLASHROM_F40_CMD, 2);
+
+ for (UcNum = 0; UcNum < FLASH_ACCESS_SIZE; UcNum++) {
+ UnWriteDat.StCdwVal.UcRamVa0 = *PucData++;
+ F40_IOWrite32A(FLASHROM_F40_WDATH, UnWriteDat.UlDwdVal);
+
+ UnWriteDat.StCdwVal.UcRamVa3 = *PucData++;
+ UnWriteDat.StCdwVal.UcRamVa2 = *PucData++;
+ UnWriteDat.StCdwVal.UcRamVa1 = *PucData++;
+ UnWriteDat.StCdwVal.UcRamVa0 = *PucData++;
+ F40_IOWrite32A(FLASHROM_F40_WDATL, UnWriteDat.UlDwdVal);
+ UcSize++;
+ }
+ } while (UcSize < 64);
+
+ UcStatus = F40_UnlockCodeClear();
+
+ return UcStatus;
+}
+
+//*****************************************************************************
+// Function Name : F40_FlashInt32Read
+//*****************************************************************************
+UINT_8 F40_FlashInt32Read( UINT_32 UlAddress, UINT_32 *PuiData )
+{
+ UINT_8 UcResult = 0 ;
+
+ F40_IOWrite32A( FLASHROM_F40_ADR, UlAddress ) ;
+ F40_IOWrite32A( FLASHROM_F40_ACSCNT, 0x00000000 ) ;
+ F40_IOWrite32A( FLASHROM_F40_CMD, 0x00000001 ) ;
+ F40_IORead32A( FLASHROM_F40_RDATL, PuiData ) ;
+
+ return UcResult ;
+}
+
+//********************************************************************************
+// Function Name : F40_CalcChecksum
+//********************************************************************************
+void F40_CalcChecksum( const UINT_8 *pData, UINT_32 len, UINT_32 *pSumH, UINT_32 *pSumL )
+{
+ UINT_64 sum = 0 ;
+ UINT_32 dat ;
+ UINT_16 i ;
+
+ for( i = 0; i < len / 5; i++ ) {
+ sum += (UINT_64)*pData++ << 32 ;
+
+ dat = (UINT_32)*pData++ << 24 ;
+ dat += (UINT_32)*pData++ << 16 ;
+ dat += (UINT_32)*pData++ << 8 ;
+ dat += (UINT_32)*pData++ ;
+ sum += (UINT_64)dat ;
+ }
+
+ *pSumH = (UINT_32)(sum >> 32) ;
+ *pSumL = (UINT_32)(sum & 0xFFFFFFFF) ;
+}
+
+//********************************************************************************
+// Function Name : F40_CalcBlockChksum
+//********************************************************************************
+void F40_CalcBlockChksum( UINT_8 num, UINT_32 *pSumH, UINT_32 *pSumL )
+{
+ UINT_8 SectorData[SECTOR_SIZE] ;
+ UINT_32 top ;
+ UINT_16 sec ;
+ UINT_64 sum = 0 ;
+ UINT_32 datH, datL ;
+
+ top = num * BLOCK_UNIT ;
+
+ for( sec = 0; sec < (BLOCK_BYTE / SECTOR_SIZE); sec++ ) {
+ F40_FlashSectorRead( top + sec * 64, SectorData ) ;
+
+ F40_CalcChecksum( SectorData, SECTOR_SIZE, &datH, &datL ) ;
+ sum += ((UINT_64)datH << 32) + datL ;
+ }
+
+ *pSumH = (UINT_32)(sum >> 32);
+ *pSumL = (UINT_32)(sum & 0xFFFFFFFF);
+}
+
+
+//********************************************************************************
+// Function Name : F40_FlashDownload
+//********************************************************************************
+UINT_8 F40_FlashDownload( UINT_8 chiperase, UINT_8 ModuleVendor, UINT_8 MdlVer )
+{
+ DOWNLOAD_TBL *ptr ;
+
+ ptr = ( DOWNLOAD_TBL * )DTbl ;
+ do {
+ if( ptr->Index == ( ((UINT_16)ModuleVendor<<8) + MdlVer) ) {
+ return F40_FlashUpdate( chiperase, ptr );
+ }
+ ptr++ ;
+ } while (ptr->Index != 0xFFFF ) ;
+
+ return 0xF0 ;
+}
+
+//********************************************************************************
+// Function Name : F40_FlashUpdate
+//********************************************************************************
+UINT_8 F40_FlashUpdate( UINT_8 flag, DOWNLOAD_TBL *ptr )
+{
+ INT_32 SiWrkVl0 ,SiWrkVl1 ;
+ INT_32 SiAdrVal ;
+ const UINT_8 *NcDatVal ;
+ UINT_32 UlReadVal, UlCnt ;
+ UINT_8 ans, i ;
+ UINT_16 UsChkBlocks ;
+ UINT_8 UserMagicCode[ ptr->SizeMagicCode ] ;
+
+//--------------------------------------------------------------------------------
+// 0.
+//--------------------------------------------------------------------------------
+ F40_IOWrite32A( SYSDSP_REMAP, 0x00001440 ) ;
+ WitTim( 25 ) ;
+ F40_IORead32A( SYSDSP_SOFTRES, (UINT_32 *)&SiWrkVl0 ) ;
+ SiWrkVl0 &= 0xFFFFEFFF ;
+ F40_IOWrite32A( SYSDSP_SOFTRES, SiWrkVl0 ) ;
+ RamWrite32A( 0xF006, 0x00000000 ) ;
+ F40_IOWrite32A( SYSDSP_DSPDIV, 0x00000001 ) ;
+ RamWrite32A( 0x0344, 0x00000014 ) ;
+ SiAdrVal =0x00100000;
+ for( UlCnt = 0 ;UlCnt < 25 ; UlCnt++ ) {
+ RamWrite32A( 0x0340, SiAdrVal ) ;
+ SiAdrVal += 0x00000008 ;
+ RamWrite32A( 0x0348, UlPmemCodeF40[ UlCnt*5 ] ) ;
+ RamWrite32A( 0x034C, UlPmemCodeF40[ UlCnt*5+1 ] ) ;
+ RamWrite32A( 0x0350, UlPmemCodeF40[ UlCnt*5+2 ] ) ;
+ RamWrite32A( 0x0354, UlPmemCodeF40[ UlCnt*5+3 ] ) ;
+ RamWrite32A( 0x0358, UlPmemCodeF40[ UlCnt*5+4 ] ) ;
+ RamWrite32A( 0x033c, 0x00000001 ) ;
+ }
+ for(UlCnt = 0 ;UlCnt < 9 ; UlCnt++ ){
+ CntWrt( (INT_8 *)&UpData_CommandFromTable[ UlCnt*6 ], 0x00000006 ) ;
+ }
+
+//--------------------------------------------------------------------------------
+// 1.
+//--------------------------------------------------------------------------------
+ if( flag ) {
+ ans = F40_UnlockCodeSet() ;
+ if ( ans != 0 )
+ return ans;
+
+ F40_IOWrite32A( FLASHROM_F40_ADR, 0x00000000 ) ;
+ F40_IOWrite32A( FLASHROM_F40_CMD, 0x00000005 ) ;
+ WitTim( 13 ) ;
+ UlCnt=0;
+ do {
+ if( UlCnt++ > 100 ) {
+ ans=0x10 ;
+ break ;
+ }
+ F40_IORead32A( FLASHROM_F40_INT, &UlReadVal ) ;
+ }while ( (UlReadVal & 0x00000080) != 0 ) ;
+
+ } else {
+ for( i = 0 ; i < ERASE_BLOCKS ; i++ ) {
+ ans = F40_FlashBlockErase( i * BLOCK_UNIT ) ;
+ if( ans != 0 ) {
+ return ans;
+ }
+ }
+ ans = F40_UnlockCodeSet() ;
+ if ( ans != 0 )
+ return ans;
+ }
+//--------------------------------------------------------------------------------
+// 2.
+//--------------------------------------------------------------------------------
+ F40_IOWrite32A( FLASHROM_F40_ADR, 0x00010000 ) ;
+ F40_IOWrite32A( FLASHROM_F40_CMD, 0x00000004 ) ;
+ WitTim( 5 ) ;
+ UlCnt=0;
+ do {
+ if( UlCnt++ > 100 ) {
+ ans = 0x10 ;
+ break ;
+ }
+ F40_IORead32A( FLASHROM_F40_INT, &UlReadVal ) ;
+ } while ( (UlReadVal & 0x00000080) != 0 ) ;
+
+//--------------------------------------------------------------------------------
+// 3.
+//--------------------------------------------------------------------------------
+ F40_FlashBurstWrite( ptr->FromCode, ptr->SizeFromCode, 0 ) ;
+
+ ans |= F40_UnlockCodeClear() ;
+ if ( ans != 0 )
+ return ans;
+
+//--------------------------------------------------------------------------------
+// 4.
+//--------------------------------------------------------------------------------
+ UsChkBlocks = ( ptr->SizeFromCode / 160 ) + 1 ;
+ RamWrite32A( 0xF00A, 0x00000000 ) ;
+ RamWrite32A( 0xF00B, UsChkBlocks ) ;
+ RamWrite32A( 0xF00C, 0x00000100 ) ;
+
+ NcDatVal = ptr->FromCode;
+ SiWrkVl0 = 0;
+ for( UlCnt = 0; UlCnt < ptr->SizeFromCode; UlCnt++ ) {
+ SiWrkVl0 += *NcDatVal++ ;
+ }
+ UsChkBlocks *= 160 ;
+ for( ; UlCnt < UsChkBlocks ; UlCnt++ ) {
+ SiWrkVl0 += 0xFF ;
+ }
+
+ UlCnt=0;
+ do {
+ if( UlCnt++ > 100 )
+ return 6;
+
+ RamRead32A( 0xF00C, &UlReadVal ) ;
+ } while( UlReadVal != 0 ) ;
+
+ RamRead32A( 0xF00D, &SiWrkVl1 ) ;
+
+ if( (int)SiWrkVl0 != (int)SiWrkVl1 )
+ return 0x20;
+
+//--------------------------------------------------------------------------------
+// X.
+//--------------------------------------------------------------------------------
+
+ if ( !flag ) {
+ UINT_32 sumH, sumL;
+ UINT_16 Idx;
+
+ // if you can use memcpy(), modify code.
+ for( UlCnt = 0; UlCnt < ptr->SizeMagicCode; UlCnt++ ) {
+ UserMagicCode[ UlCnt ] = ptr->MagicCode[ UlCnt ] ;
+ }
+
+ for( UlCnt = 0; UlCnt < USER_RESERVE; UlCnt++ ) {
+ F40_CalcBlockChksum( ERASE_BLOCKS + UlCnt, &sumH, &sumL ) ;
+ Idx = (ERASE_BLOCKS + UlCnt) * 2 * 5 + 1 + 40 ;
+ NcDatVal = (UINT_8 *)&sumH ;
+
+#ifdef _BIG_ENDIAN_
+ // for BIG ENDIAN SYSTEM
+ UserMagicCode[ Idx++ ] = *NcDatVal++ ;
+ UserMagicCode[ Idx++ ] = *NcDatVal++ ;
+ UserMagicCode[ Idx++ ] = *NcDatVal++ ;
+ UserMagicCode[ Idx++ ] = *NcDatVal++ ;
+ Idx++;
+ NcDatVal = (UINT_8 *)&sumL;
+ UserMagicCode[ Idx++ ] = *NcDatVal++ ;
+ UserMagicCode[ Idx++ ] = *NcDatVal++ ;
+ UserMagicCode[ Idx++ ] = *NcDatVal++ ;
+ UserMagicCode[ Idx++ ] = *NcDatVal++ ;
+#else
+ // for LITTLE ENDIAN SYSTEM
+ UserMagicCode[ Idx+3 ] = *NcDatVal++ ;
+ UserMagicCode[ Idx+2 ] = *NcDatVal++ ;
+ UserMagicCode[ Idx+1 ] = *NcDatVal++ ;
+ UserMagicCode[ Idx+0 ] = *NcDatVal++ ;
+ Idx+=5;
+ NcDatVal = (UINT_8 *)&sumL;
+ UserMagicCode[ Idx+3 ] = *NcDatVal++ ;
+ UserMagicCode[ Idx+2 ] = *NcDatVal++ ;
+ UserMagicCode[ Idx+1 ] = *NcDatVal++ ;
+ UserMagicCode[ Idx+0 ] = *NcDatVal++ ;
+#endif
+ }
+ NcDatVal = UserMagicCode ;
+
+ } else {
+ NcDatVal = ptr->MagicCode ;
+ }
+
+//--------------------------------------------------------------------------------
+// 5.
+//--------------------------------------------------------------------------------
+ ans = F40_UnlockCodeSet() ;
+ if ( ans != 0 )
+ return ans;
+
+ F40_FlashBurstWrite( NcDatVal, ptr->SizeMagicCode, 0x00010000 );
+ F40_UnlockCodeClear();
+
+//--------------------------------------------------------------------------------
+// 6.
+//--------------------------------------------------------------------------------
+ RamWrite32A( 0xF00A, 0x00010000 ) ;
+ RamWrite32A( 0xF00B, 0x00000002 ) ;
+ RamWrite32A( 0xF00C, 0x00000100 ) ;
+
+ SiWrkVl0 = 0;
+ for( UlCnt = 0; UlCnt < ptr->SizeMagicCode; UlCnt++ ) {
+ SiWrkVl0 += *NcDatVal++ ;
+ }
+ for( ; UlCnt < 320; UlCnt++ ) {
+ SiWrkVl0 += 0xFF ;
+ }
+
+ UlCnt=0 ;
+ do {
+ if( UlCnt++ > 100 )
+ return 6;
+
+ RamRead32A( 0xF00C, &UlReadVal ) ;
+ } while( UlReadVal != 0 ) ;
+ RamRead32A( 0xF00D, &SiWrkVl1 ) ;
+
+ if((int)SiWrkVl0 != (int)SiWrkVl1 )
+ return 0x30;
+
+ F40_IOWrite32A( SYSDSP_REMAP, 0x00001000 ) ;
+ return 0;
+}
+
+void F40_ReadCalData( UINT_32 *BufDat, UINT_32 *ChkSum )
+{
+ UINT_16 UsSize = 0, UsNum;
+
+ *ChkSum = 0;
+
+ do {
+ F40_IOWrite32A( FLASHROM_F40_ACSCNT, (FLASH_ACCESS_SIZE - 1));
+ F40_IOWrite32A( FLASHROM_F40_ADR, 0x00010040 + UsSize );
+ F40_IOWrite32A( FLASHROM_F40_CMD, 1 );
+
+ RamWrite32A( F40_IO_ADR_ACCESS, FLASHROM_F40_RDATL );
+
+ for( UsNum = 0; UsNum < FLASH_ACCESS_SIZE; UsNum++ ) {
+ RamRead32A( F40_IO_DAT_ACCESS, &(BufDat[ UsSize ]) );
+ *ChkSum += BufDat[ UsSize++ ];
+ }
+ } while( UsSize < 64 );
+}
+
+UINT_8 F40_GyroReCalib( stReCalib * pReCalib )
+{
+ UINT_8 UcSndDat;
+ UINT_32 UlRcvDat;
+ UINT_32 UlGofX, UlGofY;
+ UINT_32 UiChkSum;
+ UINT_32 UlStCnt = 0;
+
+ F40_ReadCalData( UlBufDat, &UiChkSum );
+
+ RamWrite32A( 0xF014, 0x00000000 );
+
+ do {
+ UcSndDat = F40_RdStatus(1);
+ } while( UcSndDat != 0 && ( UlStCnt++ < CNT100MS ));
+
+ RamRead32A( 0xF014, &UlRcvDat );
+ UcSndDat = (unsigned char)(UlRcvDat >> 24);
+
+ if( UlBufDat[ 49 ] == 0xFFFFFFFF )
+ pReCalib->SsFctryOffX = (UlBufDat[ 19 ] >> 16);
+ else
+ pReCalib->SsFctryOffX = (UlBufDat[ 49 ] >> 16);
+
+ if( UlBufDat[ 50 ] == 0xFFFFFFFF )
+ pReCalib->SsFctryOffY = (UlBufDat[ 20 ] >> 16);
+ else
+ pReCalib->SsFctryOffY = (UlBufDat[ 50 ] >> 16);
+
+ RamRead32A( 0x0278, &UlGofX );
+ RamRead32A( 0x027C, &UlGofY );
+
+ pReCalib->SsRecalOffX = (UlGofX >> 16);
+ pReCalib->SsRecalOffY = (UlGofY >> 16);
+ pReCalib->SsDiffX =
+ pReCalib->SsFctryOffX - pReCalib->SsRecalOffX;
+ pReCalib->SsDiffY =
+ pReCalib->SsFctryOffY - pReCalib->SsRecalOffY;
+
+ return UcSndDat;
+}
+
+UINT_8 F40_EraseCalData( void )
+{
+ UINT_32 UlReadVal, UlCnt;
+ UINT_8 ans;
+
+ ans = F40_UnlockCodeSet();
+ if( ans != 0 )
+ return ans;
+
+ F40_IOWrite32A( FLASHROM_F40_ADR, 0x00010040 );
+ F40_IOWrite32A( FLASHROM_F40_CMD, 4 );
+
+ WitTim( 5 );
+ UlCnt = 0;
+ do {
+ if( UlCnt++ > 100 ) {
+ ans = 2;
+ break;
+ }
+ F40_IORead32A( FLASHROM_F40_INT, &UlReadVal );
+ } while( ( UlReadVal & 0x00000080 ) != 0 );
+
+ ans = F40_UnlockCodeClear();
+
+ return ans;
+}
+
+UINT_8 F40_WrGyroOffsetData( void )
+{
+ UINT_32 UlFctryX, UlFctryY;
+ UINT_32 UlCurrX, UlCurrY;
+ UINT_32 UlGofX, UlGofY;
+ UINT_32 UiChkSum1, UiChkSum2;
+ UINT_32 UlSrvStat, UlOisStat;
+ UINT_8 ans;
+ UINT_32 UlStCnt = 0;
+ UINT_8 UcSndDat;
+
+ RamRead32A( 0xF010, &UlSrvStat );
+ RamRead32A( 0xF012, &UlOisStat );
+ RamWrite32A( 0xF010, 0x00000000 );
+
+ F40_ReadCalData( UlBufDat, &UiChkSum2 );
+
+ ans = F40_EraseCalData();
+ if( ans == 0 ) {
+ RamRead32A( 0x0278, &UlGofX );
+ RamRead32A( 0x027C, &UlGofY );
+
+ UlCurrX = UlBufDat[ 19 ];
+ UlCurrY = UlBufDat[ 20 ];
+ UlFctryX = UlBufDat[ 49 ];
+ UlFctryY = UlBufDat[ 50 ];
+
+ if( UlFctryX == 0xFFFFFFFF )
+ UlBufDat[ 49 ] = UlCurrX;
+
+ if( UlFctryY == 0xFFFFFFFF )
+ UlBufDat[ 50 ] = UlCurrY;
+
+ UlBufDat[ 19 ] = UlGofX;
+ UlBufDat[ 20 ] = UlGofY;
+
+ F40_WriteCalData( UlBufDat, &UiChkSum1 );
+
+ F40_ReadCalData( UlBufDat, &UiChkSum2 );
+ if( UiChkSum1 != UiChkSum2 )
+ ans = 0x10;
+ }
+
+ if( !UlSrvStat )
+ RamWrite32A( 0xF010, 0x00000000 );
+ else if( UlSrvStat == 3 )
+ RamWrite32A( 0xF010, 0x00000003 );
+ else
+ RamWrite32A( 0xF010, UlSrvStat );
+
+ do {
+ UcSndDat = F40_RdStatus(1);
+ } while( UcSndDat != 0 && ( UlStCnt++ < CNT200MS ));
+
+ if( UlOisStat != 0) {
+ RamWrite32A( 0xF012 , 0x00000001 );
+ UlStCnt = 0;
+ do {
+ UcSndDat = F40_RdStatus(1);
+ } while( UcSndDat != 0 && ( UlStCnt++ < CNT100MS ));
+ }
+
+ return ans;
+}
+
+UINT_8 F40_RdStatus( UINT_8 UcStBitChk )
+{
+ UINT_32 UlReadVal;
+
+ RamRead32A( 0xF100 , &UlReadVal );
+ if( UcStBitChk )
+ UlReadVal &= 0x01000000;
+
+ if( !UlReadVal )
+ return 0x00;
+ else
+ return 0x01;
+}
+
+UINT_8 F40_WriteCalData( UINT_32 *BufDat, UINT_32 *ChkSum )
+{
+ UINT_16 UsSize = 0, UsNum;
+ UINT_8 ans;
+ UINT_32 UlReadVal;
+
+ *ChkSum = 0;
+
+ ans = F40_UnlockCodeSet();
+ if( ans != 0 )
+ return ans;
+
+ F40_IOWrite32A(FLASHROM_F40_WDATH, 0x000000FF);
+
+ do {
+ F40_IOWrite32A(FLASHROM_F40_ACSCNT, (FLASH_ACCESS_SIZE - 1));
+ F40_IOWrite32A(FLASHROM_F40_ADR, 0x00010040 + UsSize);
+ F40_IOWrite32A(FLASHROM_F40_CMD, 2);
+ for( UsNum = 0; UsNum < FLASH_ACCESS_SIZE; UsNum++ ) {
+ F40_IOWrite32A(FLASHROM_F40_WDATL, BufDat[ UsSize ]);
+ do {
+ F40_IORead32A(FLASHROM_F40_INT, &UlReadVal);
+ } while( (UlReadVal & 0x00000020) != 0 );
+ *ChkSum += BufDat[ UsSize++ ];
+ }
+ } while( UsSize < 64 );
+
+ ans = F40_UnlockCodeClear();
+
+ return ans;
+}
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/PhoneUpdate.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/PhoneUpdate.h
new file mode 100644
index 0000000..c301b92
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/PhoneUpdate.h
@@ -0,0 +1,139 @@
+/**
+ * @brief LC898123F40 Global declaration & prototype declaration
+ *
+ * @author Copyright (C) 2017, ON Semiconductor, all right reserved.
+ *
+ **/
+
+#ifndef PHONEUPDATE_H_
+#define PHONEUPDATE_H_
+
+//==============================================================================
+//
+//==============================================================================
+#define MODULE_VENDOR 0
+#define MDL_VER 2
+
+#ifdef DEBUG
+ extern void dbg_printf(const char *, ...);
+ extern void dbg_Dump(const char *, int);
+ #define TRACE_INIT(x) dbgu_init(x)
+ #define TRACE_USB(fmt, ...) dbg_UsbData(fmt, ## __VA_ARGS__)
+ #define TRACE(fmt, ...) dbg_printf(fmt, ## __VA_ARGS__)
+ #define TRACE_DUMP(x,y) dbg_Dump(x,y)
+#else
+ #define TRACE_INIT(x)
+ #define TRACE(...)
+ #define TRACE_DUMP(x,y)
+ #define TRACE_USB(...)
+#endif
+
+#if 0
+typedef signed char INT_8;
+typedef short INT_16;
+typedef long INT_32;
+typedef long long INT_64;
+typedef unsigned char UINT_8;
+typedef unsigned short UINT_16;
+typedef unsigned long UINT_32;
+typedef unsigned long long UINT_64;
+#else
+typedef signed char INT_8;
+typedef signed short INT_16;
+typedef long INT_32;
+typedef long long INT_64;
+typedef unsigned char UINT_8;
+typedef unsigned short UINT_16;
+typedef unsigned int UINT_32;
+typedef unsigned long long UINT_64;
+#endif
+
+typedef struct {
+ UINT_16 Index;
+ const UINT_8 *MagicCode;
+ UINT_16 SizeMagicCode;
+ const UINT_8 *FromCode;
+ UINT_16 SizeFromCode;
+} DOWNLOAD_TBL;
+
+typedef struct STRECALIB {
+ INT_16 SsFctryOffX;
+ INT_16 SsFctryOffY;
+ INT_16 SsRecalOffX;
+ INT_16 SsRecalOffY;
+ INT_16 SsDiffX;
+ INT_16 SsDiffY;
+} stReCalib;
+
+typedef union DWDVAL{
+ UINT_32 UlDwdVal;
+ UINT_16 UsDwdVal[2];
+ struct {
+ UINT_16 UsLowVal;
+ UINT_16 UsHigVal;
+ } StDwdVal;
+ struct {
+ UINT_8 UcRamVa0;
+ UINT_8 UcRamVa1;
+ UINT_8 UcRamVa2;
+ UINT_8 UcRamVa3;
+ } StCdwVal;
+} UnDwdVal;
+
+#define WPB_OFF 0x01
+#define WPB_ON 0x00
+
+//==============================================================================
+//
+//==============================================================================
+#define F40_IO_ADR_ACCESS 0xC000
+#define F40_IO_DAT_ACCESS 0xD000
+#define SYSDSP_DSPDIV 0xD00014
+#define SYSDSP_SOFTRES 0xD0006C
+#define OSCRSEL 0xD00090
+#define OSCCURSEL 0xD00094
+#define SYSDSP_REMAP 0xD000AC
+#define SYSDSP_CVER 0xD00100
+#define FLASHROM_123F40 0xE07000
+#define FLASHROM_F40_RDATL (FLASHROM_123F40 + 0x00)
+#define FLASHROM_F40_RDATH (FLASHROM_123F40 + 0x04)
+#define FLASHROM_F40_WDATL (FLASHROM_123F40 + 0x08)
+#define FLASHROM_F40_WDATH (FLASHROM_123F40 + 0x0C)
+#define FLASHROM_F40_ADR (FLASHROM_123F40 + 0x10)
+#define FLASHROM_F40_ACSCNT (FLASHROM_123F40 + 0x14)
+#define FLASHROM_F40_CMD (FLASHROM_123F40 + 0x18)
+#define FLASHROM_F40_WPB (FLASHROM_123F40 + 0x1C)
+#define FLASHROM_F40_INT (FLASHROM_123F40 + 0x20)
+#define FLASHROM_F40_RSTB_FLA (FLASHROM_123F40 + 0x4CC)
+#define FLASHROM_F40_UNLK_CODE1 (FLASHROM_123F40 + 0x554)
+#define FLASHROM_F40_CLK_FLAON (FLASHROM_123F40 + 0x664)
+#define FLASHROM_F40_UNLK_CODE2 (FLASHROM_123F40 + 0xAA8)
+#define FLASHROM_F40_UNLK_CODE3 (FLASHROM_123F40 + 0xCCC)
+
+
+
+//==============================================================================
+// Prototype
+//==============================================================================
+extern void F40_Control( UINT_16 ) ;
+extern void F40_IORead32A( UINT_32, UINT_32 * ) ;
+extern void F40_IOWrite32A( UINT_32, UINT_32 ) ;
+extern UINT_8 F40_ReadWPB( void ) ;
+extern UINT_8 F40_UnlockCodeSet( void ) ;
+extern UINT_8 F40_UnlockCodeClear(void) ;
+extern UINT_8 F40_FlashDownload( UINT_8, UINT_8, UINT_8 ) ;
+extern UINT_8 F40_FlashUpdate( UINT_8, DOWNLOAD_TBL* ) ;
+extern UINT_8 F40_FlashBlockErase( UINT_32 ) ;
+extern UINT_8 F40_FlashBurstWrite( const UINT_8 *, UINT_32, UINT_32) ;
+extern void F40_FlashSectorRead( UINT_32, UINT_8 * ) ;
+extern UINT_8 F40_FlashSectorWrite( UINT_32, UINT_8 * );
+extern UINT_8 F40_FlashInt32Read( UINT_32, UINT_32 * ) ;
+extern void F40_CalcChecksum( const UINT_8 *, UINT_32, UINT_32 *, UINT_32 * ) ;
+extern void F40_CalcBlockChksum( UINT_8, UINT_32 *, UINT_32 * ) ;
+extern void F40_ReadCalData( UINT_32 *, UINT_32 * ) ;
+extern UINT_8 F40_GyroReCalib( stReCalib * ) ;
+extern UINT_8 F40_WrGyroOffsetData( void ) ;
+extern UINT_8 F40_RdStatus( UINT_8 ) ;
+extern UINT_8 F40_WriteCalData( UINT_32 *, UINT_32 * ) ;
+
+#endif /* #ifndef OIS_H_ */
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/PmemCode.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/PmemCode.h
new file mode 100644
index 0000000..f9f5a138
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/PmemCode.h
@@ -0,0 +1,47 @@
+/**
+ * @brief LC898123F40
+ *
+ * @author Copyright (C) 2017, ON Semiconductor, all right reserved.
+ *
+ **/
+//***** Pmem Code Header ******
+const UINT_32 UlPmemCodeF40[] = {
+ 0x68381c04, 0x215c0063, 0x002e8487, 0xaa086184, 0x84868040,
+ 0x02baca0c, 0x21460a40, 0x406c8484, 0xa0000068, 0x381c0421,
+ 0x6804002b, 0xac5c0060, 0x41805c01, 0xb0406c84, 0x850a0860,
+ 0x84048460, 0xa4204208, 0x404a4000, 0x03800068, 0x381c0421,
+ 0x6804002b, 0xac5c0060, 0x41805c01, 0x30406c84, 0x850a0860,
+ 0x84048460, 0xa4204208, 0x404a4000, 0x03800068, 0x381c0421,
+ 0x6804002b, 0xac5c0060, 0x41805c01, 0x30406c84, 0x850a0860,
+ 0x84048460, 0xa4204208, 0x404a4000, 0x0380005c, 0x81020061,
+ 0x5c089008, 0x00848015, 0x53e620c6, 0x28120168, 0x381c0423,
+ 0x25888420, 0x3c058508, 0x58c87600, 0x000ac076, 0x00100521,
+ 0xbc05f840, 0x61a00a07, 0x600000a2, 0x18486068, 0x04001720,
+ 0xba148850, 0x60400003, 0x80006838, 0x1c07215c, 0x0042bfe0,
+ 0x5cbf0048, 0x505c0080, 0x8060a008, 0x09c80184, 0x850880e0,
+ 0x40000081, 0x76680000, 0x78206601, 0x00084088, 0x02000000,
+ 0x842002a7, 0xc432820b, 0xff6c8424, 0x86c70380, 0xe7a68010,
+ 0x13eac880, 0xa088136a, 0x04608400, 0x08408228, 0x88084050,
+ 0x460a4204, 0x208406ca, 0x80205c09, 0x62bff0a4, 0x0018487a,
+ 0x40000380, 0x006c7038, 0x100a2593, 0x0bffc076, 0x00000ac0,
+ 0x84022683, 0x81c0323a, 0x100c8406, 0x40000095, 0x02c94864,
+ 0xa0819880, 0x08600000, 0x00945cbf, 0x0858485c, 0x8041d882,
+ 0x84023000, 0x00a180c8, 0x40640000, 0x09582c94, 0xa4400000,
+ 0xa0808460, 0xa4040088, 0x50484000, 0x0280106c, 0x70381008,
+ 0x3813e259, 0xa0bc0d0a, 0x00208408, 0x88400a28, 0x93484048,
+ 0x6c70380e, 0x7a680101, 0x3eac460a, 0x42042084, 0x06c00000,
+ 0x400003a1, 0x40000000, 0x00000000, 0x00000000, 0x00000000,
+};
+
+const UINT_8 UpData_CommandFromTable[] = {
+ 0x01, 0xbc, 0x00, 0x04, 0x05, 0x16, // bit 0, FromRead
+ 0x01, 0xc0, 0x00, 0x04, 0x05, 0x16, // bit 1, FromRead
+ 0x01, 0xc4, 0x00, 0x10, 0x00, 0x3e, // bit 2, FromWrite
+ 0x01, 0xc8, 0x00, 0x10, 0x00, 0x3e, // bit 3, FromWrite
+ 0x01, 0xcc, 0x00, 0x10, 0x00, 0x1e, // bit 4, FromSectorErase
+ 0x01, 0xd0, 0x00, 0x10, 0x00, 0x0e, // bit 5, FromBlockEraset
+ 0x01, 0xd4, 0x00, 0x10, 0x00, 0x2e, // bit 6, FromNvrErase
+ 0x01, 0xd8, 0x00, 0x10, 0x00, 0x00, // bit 7, FromAllErase
+ 0x01, 0xdc, 0x00, 0x04, 0x05, 0x3e, // bit 8, FromCheckSum
+};
+
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/VCM_firmware.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/VCM_firmware.h
new file mode 100644
index 0000000..679b9de
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/VCM_firmware.h
@@ -0,0 +1,128 @@
+#ifndef VCMFIRMWARE_H_
+#define VCMFIRMWARE_H_
+
+#include "cam_sensor_cmn_header.h"
+
+struct cam_sensor_i2c_reg_array VCM_LC898219_DVT_1_0[] = {
+ {0x08, 0xA0, 0, 0},
+ {0x0A, 0xA7, 0, 0},
+ {0x0B, 0x00, 0, 0},
+ {0x0C, 0x00, 0, 0},
+ {0x0D, 0x00, 0, 0},
+ {0x0E, 0x90, 0, 0},
+ {0x0F, 0x48, 0, 0},
+ {0x10, 0x3F, 0, 0},
+ {0x11, 0xFF, 0, 0},
+ {0x12, 0xA1, 0, 0},
+ {0x13, 0x01, 0, 0},
+ {0x1A, 0x20, 0, 0},
+ {0x1B, 0x30, 0, 0},
+ {0x1C, 0x00, 0, 0},
+ {0x1D, 0x00, 0, 0},
+ {0x28, 0xDD, 0, 0},
+ {0x29, 0xDC, 0, 0},
+ {0x2A, 0xBA, 0, 0},
+ {0x2B, 0x87, 0, 0},
+ {0x2C, 0x88, 0, 0},
+ {0x2D, 0x9B, 0, 0},
+ {0x2E, 0xEF, 0, 0},
+ {0x2F, 0xFF, 0, 0},
+ {0x40, 0x65, 0, 0},
+ {0x41, 0xB0, 0, 0},
+ {0x42, 0x41, 0, 0},
+ {0x43, 0x20, 0, 0},
+ {0x44, 0x77, 0, 0},
+ {0x45, 0xFF, 0, 0},
+ {0x46, 0x6A, 0, 0},
+ {0x47, 0xFF, 0, 0},
+ {0x48, 0x78, 0, 0},
+ {0x49, 0xF8, 0, 0},
+ {0x4A, 0x71, 0, 0},
+ {0x4B, 0x71, 0, 0},
+ {0x4C, 0xD3, 0, 0},
+ {0x4D, 0x5F, 0, 0},
+ {0x4E, 0x6C, 0, 0},
+ {0x4F, 0x00, 0, 0},
+ {0x50, 0x00, 0, 0},
+ {0x51, 0x0E, 0, 0},
+ {0x52, 0x00, 0, 0},
+ {0x53, 0x04, 0, 0},
+ {0x54, 0x13, 0, 0},
+ {0x55, 0x5F, 0, 0},
+ {0x56, 0x0B, 0, 0},
+ {0x57, 0xBF, 0, 0},
+ {0x58, 0x13, 0, 0},
+ {0x59, 0x0A, 0, 0},
+ {0x5A, 0x02, 0, 0},
+ {0x5B, 0x00, 0, 0},
+ {0x5C, 0x00, 0, 0},
+ {0x5D, 0x00, 0, 0},
+ {0x5E, 0x02, 0, 0},
+ {0x5F, 0xFF, 0, 0},
+ {0x60, 0x7F, 0, 0},
+ {0x61, 0xFF, 0, 0},
+ {0x62, 0x00, 0, 0},
+ {0x63, 0x00, 0, 0},
+ {0x64, 0x00, 0, 0},
+ {0x65, 0x00, 0, 0},
+ {0x66, 0x00, 0, 0},
+ {0x67, 0x00, 0, 0},
+ {0x68, 0x05, 0, 0},
+ {0x69, 0xF0, 0, 0},
+ {0x6A, 0x5F, 0, 0}
+};
+
+/* Temperature parameters version E */
+struct cam_sensor_i2c_reg_array VCM_LC898219_Temp_Params_verE[] = {
+ {0x14, 0x69, 0, 0},
+ {0x3F, 0x09, 0, 0},
+ {0x70, 0x00, 0, 0},
+ {0x71, 0x70, 0, 0},
+ {0x72, 0x0A, 0, 0},
+ {0x73, 0x00, 0, 0},
+ {0x74, 0x07, 0, 0},
+ {0x75, 0xFF, 0, 0},
+ {0x76, 0x9D, 0, 0},
+ {0x77, 0x4D, 0, 0},
+};
+
+/* Temperature parameters version H */
+struct cam_sensor_i2c_reg_array VCM_LC898219_Temp_Params_verH[] = {
+ {0x14, 0x69, 0, 0},
+ {0x3F, 0x09, 0, 0},
+ {0x70, 0x00, 0, 0},
+ {0x71, 0x70, 0, 0},
+ {0x72, 0x09, 0, 0},
+ {0x73, 0x00, 0, 0},
+ {0x74, 0x07, 0, 0},
+ {0x75, 0xFF, 0, 0},
+ {0x76, 0xA9, 0, 0},
+ {0x77, 0x48, 0, 0},
+};
+
+struct cam_sensor_i2c_reg_array VCM_AK7375_DVT_R16[] = {
+ {0x10, 0x34, 0, 0},
+ {0x11, 0x43, 0, 0},
+ {0x12, 0x5E, 0, 0},
+ {0x13, 0x84, 0, 0},
+ {0x14, 0x17, 0, 0},
+ {0x15, 0xFF, 0, 0},
+ {0x16, 0xFF, 0, 0},
+ {0x17, 0x6A, 0, 0},
+ {0x18, 0xDB, 0, 0},
+ {0x1A, 0x00, 0, 0},
+ {0x1B, 0x46, 0, 0},
+ {0x1C, 0xFF, 0, 0},
+ {0x1D, 0xFE, 0, 0},
+ {0x1E, 0xBE, 0, 0},
+ {0x1F, 0xC8, 0, 0},
+ {0x20, 0x0A, 0, 0},
+ {0x21, 0x14, 0, 0},
+ {0x22, 0x28, 0, 0},
+ {0x23, 0x00, 0, 0},
+ {0x24, 0x00, 0, 0},
+ {0x25, 0x43, 0, 0},
+ {0x26, 0x46, 0, 0},
+};
+
+#endif /* #ifndef VCMFIRMWARE_H_ */
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/fw_update.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/fw_update.c
new file mode 100644
index 0000000..ac39c72
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/fw_update.c
@@ -0,0 +1,1174 @@
+/* OIS calibration interface for LC898123 F40
+ *
+ */
+
+#include "fw_update.h"
+#include "VCM_firmware.h"
+
+#define OIS_CUR_FW_VERSION 0x08
+#define OIS_COMPONENT_I2C_ADDR_WRITE 0x7C
+#define VCM_COMPONENT_I2C_ADDR_WRITE 0xE4
+#define VCM_EEPROM_I2C_ADDR_WRITE 0xE6
+#define VCM_COMPONENT_I2C_ADDR_WRITE_FRONT_N 0x18
+#define VCM_EEPROM_I2C_ADDR_WRITE_FRONT_N 0xA0
+#define MK_SHARP 0x04170000
+#define MK_LGIT 0x09170000
+#define AF_EEPROM_DATA 0x80
+#define RETRY_MAX 3
+#define LUT_MAX 66
+#define LUT_MAX_FRONT 22
+#define SECTOR_SIZE 320
+
+#define VCM_AK7375_EEPROM_WRITE_MODE_ADDR 0xAE
+#define VCM_AK7375_EEPROM_WRITE_MODE_DATA 0x3B
+
+#define VCM_AK7375_EEPROM_STORE_ADDR 0x03
+#define VCM_AK7375_EEPROM_STORE_STATUS_ADDR 0x4B
+#define VCM_AK7375_EEPROM_STORE_MEM1_DATA 0x01
+#define VCM_AK7375_EEPROM_STORE_MEM2_DATA 0x02
+#define VCM_AK7375_EEPROM_STORE_MEM3_DATA 0x04
+
+#define VCM_AK7375_EEPROM_VERSION_ADDR 0x000A
+#define VCM_AK7375_EEPROM_VERSION_R13 0x0D
+
+static struct camera_io_master *g_io_master_info;
+bool g_first = true;
+static bool force_disable_LTC;
+static bool isRearVCMInitDownload;
+
+/* Implemented in cam_actuator_core.c */
+extern bool check_act_ltc_disable(void);
+
+void RamWrite32A( UINT_16 RamAddr, UINT_32 RamData )
+{
+ int rc = 0;
+ struct camera_io_master *io_master_info = g_io_master_info;
+ struct cam_sensor_i2c_reg_setting i2c_reg_settings;
+ struct cam_sensor_i2c_reg_array i2c_reg_array;
+
+ i2c_reg_settings.addr_type = CAMERA_SENSOR_I2C_TYPE_WORD;
+ i2c_reg_settings.data_type = CAMERA_SENSOR_I2C_TYPE_DWORD;
+ i2c_reg_settings.size = 1;
+ i2c_reg_settings.delay = 0;
+ i2c_reg_array.reg_addr = RamAddr;
+ i2c_reg_array.reg_data = RamData;
+ i2c_reg_array.delay = 0;
+ i2c_reg_settings.reg_setting = &i2c_reg_array;
+
+ rc = camera_io_dev_write(io_master_info, &i2c_reg_settings);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR, "[OISFW] %s : write failed", __func__);
+ }
+}
+
+void RamRead32A( UINT_16 RamAddr, UINT_32 *ReadData )
+{
+ int rc = 0;
+
+ rc = camera_io_dev_read(g_io_master_info, RamAddr,
+ ReadData, CAMERA_SENSOR_I2C_TYPE_WORD,
+ CAMERA_SENSOR_I2C_TYPE_DWORD);
+ if (rc < 0)
+ CAM_ERR(CAM_SENSOR, "[OISFW]:%s read i2c failed", __func__);
+}
+
+int RamWrite16A( struct camera_io_master *io_info,
+ UINT_32 RamAddr, UINT_32 RamData )
+{
+ int rc = 0;
+ struct cam_sensor_i2c_reg_setting i2c_reg_settings;
+ struct cam_sensor_i2c_reg_array i2c_reg_array;
+
+ i2c_reg_settings.addr_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ i2c_reg_settings.data_type = CAMERA_SENSOR_I2C_TYPE_WORD;
+ i2c_reg_settings.size = 1;
+ i2c_reg_settings.delay = 0;
+ i2c_reg_array.reg_addr = RamAddr;
+ i2c_reg_array.reg_data = RamData;
+ i2c_reg_array.delay = 0;
+ i2c_reg_settings.reg_setting = &i2c_reg_array;
+
+ rc = camera_io_dev_write(io_info, &i2c_reg_settings);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "%s: write 0x%x failed", __func__, RamAddr);
+ }
+
+ return rc;
+}
+
+int RamRead16A( struct camera_io_master *io_info,
+ UINT_32 RamAddr, UINT_32 *ReadData )
+{
+ int rc = 0;
+
+ rc = camera_io_dev_read(io_info, RamAddr,
+ ReadData, CAMERA_SENSOR_I2C_TYPE_BYTE,
+ CAMERA_SENSOR_I2C_TYPE_WORD);
+ if (rc < 0)
+ CAM_ERR(CAM_SENSOR,
+ "%s: read 0x%x failed", __func__, RamAddr);
+
+ return rc;
+}
+
+int RamWrite8A( struct camera_io_master *io_info,
+ UINT_32 RamAddr, UINT_32 RamData )
+{
+ int rc = 0;
+ struct cam_sensor_i2c_reg_setting i2c_reg_settings;
+ struct cam_sensor_i2c_reg_array i2c_reg_array;
+
+ i2c_reg_settings.addr_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ i2c_reg_settings.data_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ i2c_reg_settings.size = 1;
+ i2c_reg_settings.delay = 0;
+ i2c_reg_array.reg_addr = RamAddr;
+ i2c_reg_array.reg_data = RamData;
+ i2c_reg_array.delay = 0;
+ i2c_reg_settings.reg_setting = &i2c_reg_array;
+
+ rc = camera_io_dev_write(io_info, &i2c_reg_settings);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "%s: write 0x%x failed", __func__, RamAddr);
+ }
+
+ return rc;
+}
+
+int RamRead8A( struct camera_io_master *io_info,
+ UINT_32 RamAddr, UINT_32 *ReadData )
+{
+ int rc = 0;
+
+ rc = camera_io_dev_read(io_info, RamAddr,
+ ReadData, CAMERA_SENSOR_I2C_TYPE_BYTE,
+ CAMERA_SENSOR_I2C_TYPE_BYTE);
+ if (rc < 0)
+ CAM_ERR(CAM_SENSOR,
+ "%s: read 0x%x failed", __func__, RamAddr);
+
+ return rc;
+}
+
+void WitTim( UINT_16 UsWitTim )
+{
+ msleep(UsWitTim);
+}
+
+int CntWrt( UINT_8 *PcSetDat, UINT_16 UsDatNum)
+{
+ int rc = 0, cnt;
+ uint16_t total_bytes = UsDatNum-1;
+ uint8_t *ptr = NULL;
+ struct camera_io_master *io_master_info = g_io_master_info;
+ struct cam_sensor_i2c_reg_setting i2c_reg_setting;
+
+ i2c_reg_setting.addr_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ i2c_reg_setting.data_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ i2c_reg_setting.size = total_bytes;
+ i2c_reg_setting.delay = 0;
+ i2c_reg_setting.reg_setting = (struct cam_sensor_i2c_reg_array *)
+ kzalloc(sizeof(struct cam_sensor_i2c_reg_array) * total_bytes,
+ GFP_KERNEL);
+ if (!i2c_reg_setting.reg_setting) {
+ CAM_ERR(CAM_SENSOR, "[OISFW]:%s Failed in allocating i2c_array", __func__);
+ return -ENOMEM;
+ }
+
+ for (cnt = 0, ptr = &PcSetDat[1]; cnt < total_bytes;
+ cnt++, ptr++) {
+ i2c_reg_setting.reg_setting[cnt].reg_addr = PcSetDat[0];
+ i2c_reg_setting.reg_setting[cnt].reg_data = *ptr;
+ i2c_reg_setting.reg_setting[cnt].delay = 0;
+ i2c_reg_setting.reg_setting[cnt].data_mask = 0;
+ }
+
+ rc = camera_io_dev_write_continuous(io_master_info, &i2c_reg_setting, 1);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR, "[OISFW]:%s i2c write sequence error:%d",
+ __func__, rc);
+ }
+ kfree(i2c_reg_setting.reg_setting);
+ return rc;
+}
+
+int CntRd3( UINT_32 addr, void *PcSetDat, UINT_16 UsDatNum )
+{
+ int rc = 0;
+
+ rc = camera_io_dev_read_seq(g_io_master_info, addr, PcSetDat, CAMERA_SENSOR_I2C_TYPE_WORD, UsDatNum);
+ if (rc < 0)
+ CAM_ERR(CAM_SENSOR, "[OISFW]:%s read i2c failed", __func__);
+ return rc;
+}
+
+void WPBCtrl( UINT_8 UcCtrl )
+{
+ //do nothing because lc898123F40 uses UnlockCodeSet() to handle WPB by itself
+}
+
+int checkHighLevelCommand(int cnt)
+{
+ int i;
+ UINT_32 FWRead;
+
+
+ for (i =0; i< cnt ; i++) {
+ WitTim(10);
+ RamRead32A(0xF100, &FWRead);//Check high level command ready.
+ if (FWRead == 0x0 ) {
+ CAM_INFO(CAM_SENSOR, "[OISFW]:checkHighLevelCommand finish.");
+ return 0;
+ } else
+ CAM_INFO(CAM_SENSOR, "[OISFW]:Waiting...");
+ }
+ CAM_ERR(CAM_SENSOR, "[OISFW]:checkHighLevelCommand fail.");
+ return -EINVAL;
+}
+
+int doFWupdate(UINT_16 CAL_ID, UINT_32 MODULE_MAKER)
+{
+ int rc = 0;
+
+ if(MODULE_MAKER == MK_SHARP) {
+ if (CAL_ID == 0x0) {
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s not to update FW because of unknown CAL_ID.", __func__);
+ rc = -EINVAL;
+ } else if (CAL_ID == 0x1) {
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s F40_FlashDownload(0, 4, 0 )", __func__);
+ rc = F40_FlashDownload(0, 4, 0);
+ } else {
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s F40_FlashDownload(0, 4, 1 )", __func__);
+ rc = F40_FlashDownload(0, 4, 1);
+ }
+ } else if(MODULE_MAKER == MK_LGIT) {
+ if (CAL_ID == 0x0) {
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s not to update FW because of unknown CAL_ID.", __func__);
+ rc = -EINVAL;
+ } else if (CAL_ID == 0x1) {
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s F40_FlashDownload(0, 9, 0 )", __func__);
+ rc = F40_FlashDownload(0, 9, 0);
+ } else {
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s F40_FlashDownload(0, 9, 1 )", __func__);
+ rc = F40_FlashDownload(0, 9, 1);
+ }
+ } else {
+ CAM_ERR(CAM_SENSOR, "[OISFW]:%s unknown module maker.", __func__);
+ rc = -EINVAL;
+ }
+
+ if (rc == 0) {
+ /*Wait for FW update finish.*/
+ rc = checkHighLevelCommand(20);
+ } else {
+ CAM_ERR(CAM_SENSOR, "[OISFW]%s:OIS FW update failed rc = %d.",
+ __func__, rc);
+ rc = -EINVAL;
+ }
+
+ return rc;
+}
+
+bool checkOISFWversion(UINT_16 *cal_id, UINT_32 *module_maker)
+{
+ int rc;
+ UINT_16 RamAddr;
+ UINT_32 UlReadVal;
+ UINT_16 FW_version;
+ bool need_update = false;
+
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s", __func__);
+
+ RamAddr = 0x8000;
+ RamRead32A(RamAddr, &UlReadVal);
+ FW_version = UlReadVal & 0xFF;
+ *module_maker = UlReadVal & 0xFFFF0000;
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s module_version = 0x%02x",
+ __func__, UlReadVal);
+
+ RamAddr = 0x8004;
+ RamRead32A(RamAddr, &UlReadVal);
+ *cal_id = UlReadVal & 0xFF;
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s CAL_ID = 0x%04x, MODULE_MAKER = 0x%x",
+ __func__, *cal_id, *module_maker);
+
+ if (FW_version >= OIS_CUR_FW_VERSION) {
+ CAM_INFO(CAM_SENSOR, "[OISFW]%s: No need to update.", __func__);
+ return false;
+ } else {
+ rc = checkHighLevelCommand(100);
+ if (rc != 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[OISFW]:%s checkHighLevelCommand failed: %d",
+ __func__, rc);
+ need_update = false;
+ return -EINVAL;
+ }
+
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s checkHighLevelCommand = %d",
+ __func__, rc);
+
+ need_update = true;
+ }
+
+ return need_update;
+}
+
+int checkOISFWUpdate(struct cam_sensor_ctrl_t *s_ctrl)
+{
+ int rc = 0;
+ int i;
+ unsigned short cci_client_sid_backup;
+ UINT_32 FWRead;
+ UINT_16 FW_version;
+ UINT_16 cal_id;
+ UINT_32 module_maker;
+
+ if (g_first != true) {
+ CAM_INFO(CAM_SENSOR, "[OISFW]%s: No need.", __func__);
+ return 0;
+ }
+ g_first = false;
+
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s 1. sid = %d", __func__,
+ s_ctrl->io_master_info.cci_client->sid);
+
+ /* Bcakup the I2C slave address */
+ cci_client_sid_backup = s_ctrl->io_master_info.cci_client->sid;
+
+ /* Replace the I2C slave address with OIS component */
+ s_ctrl->io_master_info.cci_client->sid =
+ OIS_COMPONENT_I2C_ADDR_WRITE >> 1;
+
+ g_io_master_info = &(s_ctrl->io_master_info);
+ WitTim(100);
+
+ if( checkOISFWversion(&cal_id, &module_maker) == true ) {
+ rc = doFWupdate(cal_id, module_maker);
+
+ for (i = 0; i < 2 ; i++) {
+ RamRead32A(0x8000, &FWRead);
+ FW_version = FWRead & 0xFF;
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s 0x8000 = 0x%08x", __func__, FWRead);
+ if (FW_version != OIS_CUR_FW_VERSION) {
+ CAM_ERR(CAM_SENSOR,
+ "[OISFW]:FW version check failed after update. retry.");
+ rc = doFWupdate(cal_id, module_maker);
+ } else {
+ CAM_INFO(CAM_SENSOR,
+ "[OISFW]: FW vserion verify pass.");
+ break;
+ }
+ }
+ }
+
+ /* Restore the I2C slave address */
+ s_ctrl->io_master_info.cci_client->sid =
+ cci_client_sid_backup;
+ CAM_INFO(CAM_SENSOR, "[OISFW]:%s 2. sid = %d", __func__,
+ s_ctrl->io_master_info.cci_client->sid);
+
+ return rc;
+}
+
+int checkRearVCMFWUpdate(struct cam_sensor_ctrl_t *s_ctrl)
+{
+ int rc = 0;
+ int i = 0;
+ UINT_32 retry = 0;
+ UINT_32 size = 0;
+ UINT_32 UlReadVal;
+ UINT_32 VCM_rev = 0;
+ UINT_32 MTM_rev = 0;
+ UINT_32 regdata = 0;
+ bool needupdate = false;
+ unsigned short cci_client_sid_backup;
+ struct cam_sensor_i2c_reg_array *fwtable = NULL;
+
+ WitTim(300);
+
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s: original sid = %d", __func__,
+ s_ctrl->io_master_info.cci_client->sid);
+
+ cci_client_sid_backup = s_ctrl->io_master_info.cci_client->sid;
+
+ /* 1. Check actuator revision with slave address 0x7C */
+ s_ctrl->io_master_info.cci_client->sid =
+ OIS_COMPONENT_I2C_ADDR_WRITE >> 1;
+
+ RamWrite32A(0xF01B, 0x1A02);
+ RamRead32A(0xF01B, &UlReadVal);
+ VCM_rev = (UlReadVal & 0xFF000000) >> 24;
+
+ /* 2. Check MTM revision from AF EEPROM 0x6E */
+ s_ctrl->io_master_info.cci_client->sid =
+ VCM_EEPROM_I2C_ADDR_WRITE >> 1;
+
+ rc = RamRead8A(&(s_ctrl->io_master_info), 0x1F, &MTM_rev);
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]%s: 0x1A02 = 0x%08x, VCM rev = 0x%x, MTM rev = 0x%x",
+ __func__, UlReadVal, VCM_rev, MTM_rev);
+
+ /* 3. Whether to update or not */
+ if (VCM_rev == 0x5 && MTM_rev == 0xC4) {
+ fwtable = VCM_LC898219_DVT_1_0;
+ size = sizeof(VCM_LC898219_DVT_1_0) /
+ sizeof(struct cam_sensor_i2c_reg_array);
+ } else {
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]%s: Unsupported rev", __func__);
+ }
+
+ if (fwtable != NULL) {
+ s_ctrl->io_master_info.cci_client->sid =
+ VCM_EEPROM_I2C_ADDR_WRITE >> 1;
+ for (i = 0; i < LUT_MAX; i++) {
+ rc = RamRead8A(&(s_ctrl->io_master_info),
+ fwtable[i].reg_addr, ®data);
+ if (rc < 0 || regdata != fwtable[i].reg_data) {
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]%s: run FW update", __func__);
+ needupdate = true;
+ break;
+ }
+ }
+ }
+
+ if(needupdate == false) {
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]%s: By pass fw update", __func__);
+ goto dump_fw;
+ }
+
+ /* 4. Start upgrade process. Bake up origianl FW data first */
+ rc = BackupRearVCMData(&(s_ctrl->io_master_info));
+ if (rc < 0) {
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]backup data failed, skip FW update");
+ goto dump_fw;
+ }
+
+ /* 5. Replace slave address with 0xE4, do AF firmware update */
+ s_ctrl->io_master_info.cci_client->sid =
+ VCM_COMPONENT_I2C_ADDR_WRITE >> 1;
+ WitTim(8);
+
+ for (retry = 0; retry < RETRY_MAX; retry++) {
+ rc = RamRead8A(&(s_ctrl->io_master_info), 0xF0, ®data);
+ if (regdata == 0xA5) {
+ WitTim(1);
+ rc = camera_io_dev_poll(&(s_ctrl->io_master_info),
+ 0xE0, 0x00, 0, CAMERA_SENSOR_I2C_TYPE_BYTE,
+ CAMERA_SENSOR_I2C_TYPE_BYTE, 100);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]i2c poll 0xE0 failed");
+ continue;
+ }
+
+ rc = DownloadRearVCMFW(&(s_ctrl->io_master_info),
+ fwtable, size);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]Download firmware failed");
+ continue;
+ }
+
+ /* 6. Validate EEPROM data*/
+ rc = ValidateRearVCMFW(&(s_ctrl->io_master_info),
+ fwtable, size);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]Validate firmware failed");
+ continue;
+ }
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]%s: FW update success", __func__);
+ break;
+ } else {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]%s: NG module !", __func__);
+ break;
+ }
+ }
+
+dump_fw:
+ s_ctrl->io_master_info.cci_client->sid =
+ VCM_EEPROM_I2C_ADDR_WRITE >> 1;
+
+ for (i = 0x00; i < AF_EEPROM_DATA; i++) {
+ rc = RamRead8A(&(s_ctrl->io_master_info), i, ®data);
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s; addr:0x%x, data:0x%x, rc:%d",
+ __func__, i, regdata, rc);
+ }
+
+ /* 7. Restore the I2C slave address */
+ s_ctrl->io_master_info.cci_client->sid =
+ cci_client_sid_backup;
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s: restore sid = %d", __func__,
+ s_ctrl->io_master_info.cci_client->sid);
+
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s; rc = %d", __func__, rc);
+
+ return rc;
+}
+
+int checkRearVCMFWUpdate_temp(struct cam_sensor_ctrl_t *s_ctrl)
+{
+ int rc = 0;
+ UINT_32 retry = 0;
+ UINT_32 size = 0;
+ UINT_32 regdata = 0;
+ unsigned short cci_client_sid_backup;
+ uint32_t paramver = -1;
+ struct cam_sensor_i2c_reg_array *fwtable = NULL;
+ struct device_node *src_node = NULL;
+ struct device_node *of_node = s_ctrl->of_node;
+
+ force_disable_LTC = true;
+ src_node = of_parse_phandle(of_node, "actuator-src", 0);
+ if (!src_node) {
+ return 0;
+ } else {
+ rc = of_property_read_u32(src_node,
+ "param-index", ¶mver);
+ CAM_INFO(CAM_SENSOR, "[VCMFW] paramver=%d, rc=%d",
+ paramver, rc);
+ of_node_put(src_node);
+ src_node = NULL;
+ }
+
+ if (paramver == 5) {
+ force_disable_LTC = false;
+ fwtable = VCM_LC898219_Temp_Params_verE;
+ size = sizeof(VCM_LC898219_Temp_Params_verE) /
+ sizeof(struct cam_sensor_i2c_reg_array);
+ } else if (paramver == 8) {
+ force_disable_LTC = false;
+ fwtable = VCM_LC898219_Temp_Params_verH;
+ size = sizeof(VCM_LC898219_Temp_Params_verH) /
+ sizeof(struct cam_sensor_i2c_reg_array);
+ } else {
+ CAM_INFO(CAM_SENSOR, "[VCMFW] Not to update LTC param");
+ return 0;
+ }
+
+ cci_client_sid_backup = s_ctrl->io_master_info.cci_client->sid;
+
+ /* Check if need to update EEPROM data */
+ rc = ValidateRearVCMFW(&(s_ctrl->io_master_info),
+ fwtable, size);
+ if (rc == 0) {
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]No need to update firmware");
+ goto temp_restore_data;
+ }
+
+ /* 1. Replace slave address with 0xE4, do AF firmware update */
+ s_ctrl->io_master_info.cci_client->sid =
+ VCM_COMPONENT_I2C_ADDR_WRITE >> 1;
+ WitTim(8);
+
+ for (retry = 0; retry < RETRY_MAX; retry++) {
+ rc = RamRead8A(&(s_ctrl->io_master_info),
+ 0xF0, ®data);
+ if (regdata == 0xA5) {
+ WitTim(1);
+ rc = camera_io_dev_poll(
+ &(s_ctrl->io_master_info), 0xE0, 0x00, 0,
+ CAMERA_SENSOR_I2C_TYPE_BYTE,
+ CAMERA_SENSOR_I2C_TYPE_BYTE, 100);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]i2c poll 0xE0 failed");
+ continue;
+ }
+
+ /* Update EEPROM data*/
+ rc = DownloadRearVCMFW(&(s_ctrl->io_master_info),
+ fwtable, size);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]Download firmware failed");
+ continue;
+ }
+
+ /* Validate EEPROM data*/
+ rc = ValidateRearVCMFW(&(s_ctrl->io_master_info),
+ fwtable, size);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]Validate firmware failed");
+ continue;
+ }
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]%s: FW update success", __func__);
+ break;
+ } else {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]%s: NG module !\n", __func__);
+ break;
+ }
+ }
+
+temp_restore_data:
+ /* 2. Restore the I2C slave address */
+ s_ctrl->io_master_info.cci_client->sid =
+ cci_client_sid_backup;
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s: restore sid = %d\n",
+ __func__, s_ctrl->io_master_info.cci_client->sid);
+
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s; rc = %d\n", __func__, rc);
+
+ return rc;
+}
+
+int is_force_disable_LTC(void)
+{
+ return force_disable_LTC;
+}
+
+int checkRearVCMLTC(struct camera_io_master *io_info)
+{
+ int rc = -1;
+ UINT_32 retry = 0;
+ UINT_32 regdata = 0;
+ unsigned short cci_client_sid_backup;
+ UINT_32 EEPROM_3Fh = 0;
+ UINT_32 UlReadVal;
+ UINT_16 init_temp;
+ UINT_16 init_temp_A;
+ UINT_16 init_temp_B;
+
+ if (io_info->cci_client->sid != (VCM_COMPONENT_I2C_ADDR_WRITE >> 1)) {
+ return rc;
+ }
+
+ if(is_force_disable_LTC() == true) {
+ CAM_INFO(CAM_SENSOR, "[LTC] force disable LTC by project");
+ return rc;
+ }
+
+ if(check_act_ltc_disable() == true) {
+ CAM_INFO(CAM_SENSOR, "[LTC] dynamic disable LTC by user");
+ return rc;
+ }
+
+ CAM_INFO(CAM_SENSOR, "[LTC] isRearVCMInitDownload=%d",
+ isRearVCMInitDownload);
+ if (isRearVCMInitDownload == true)
+ return rc;
+
+ /* 0. Backup the I2C slave address */
+ cci_client_sid_backup = io_info->cci_client->sid;
+
+ /* Read EEPROM 3Fh first */
+ io_info->cci_client->sid = VCM_EEPROM_I2C_ADDR_WRITE >> 1;
+ RamRead8A(io_info, 0x3F, &EEPROM_3Fh);
+
+ /* 1. Replace slave address with 0xE4 */
+ io_info->cci_client->sid = VCM_COMPONENT_I2C_ADDR_WRITE >> 1;
+
+ /* Communication Check */
+ RamRead8A(io_info, 0xF0, ®data);
+ if (regdata == 0xA5) {
+ CAM_INFO(CAM_SENSOR,
+ "[LTC] check communication success");
+ } else {
+ CAM_ERR(CAM_SENSOR,
+ "[LTC] check communication error");
+ goto ltc_restore_data;
+ }
+ usleep_range(1000, 1010);
+
+ /* Enable Temperature Acquisition */
+ RamWrite8A(io_info, 0x8E, 0x15);
+ RamWrite8A(io_info, 0x8D, 0x20);
+
+ /* Standby Release */
+ RamWrite8A(io_info, 0x81, 0x80);
+ usleep_range(1000, 1010);
+
+ /* Read Initial Temperature Data */
+ RamRead16A(io_info, 0x0058, &UlReadVal);
+ init_temp = (UINT_16)UlReadVal;
+ CAM_INFO(CAM_SENSOR, "[LTC] init_temp = 0x%x",
+ init_temp);
+
+ /* Calculate Shift-Up Gain */
+ if ( ((EEPROM_3Fh & 0x38) >> 3) == 2)
+ init_temp_A = init_temp << 2;
+ else if ( ((EEPROM_3Fh & 0x38) >> 3) == 1)
+ init_temp_A = init_temp << 1;
+ else
+ init_temp_A = init_temp;
+
+ if ( (EEPROM_3Fh & 0x07) == 2)
+ init_temp_B = init_temp << 2;
+ else if ( (EEPROM_3Fh & 0x07) == 1)
+ init_temp_B = init_temp << 1;
+ else
+ init_temp_B = init_temp;
+
+ /* Initial Data Download */
+ RamWrite8A(io_info, 0xE0, 0x01);
+ msleep(8); /* wait 8 ms */
+
+ /* LSI wake up check */
+ for (retry = 0; retry < 10; retry++) {
+ RamRead8A(io_info, 0xB3, &UlReadVal);
+ if ( (UlReadVal & 0XE0) == 0 ) {
+ break;
+ } else {
+ if (retry >= 9) {
+ CAM_ERR(CAM_SENSOR,
+ "[LTC] LSI wake up check failed");
+ goto ltc_restore_data;
+ }
+ }
+ usleep_range(1000, 1010);
+ }
+
+ /* Write Init Temperature Data */
+ RamWrite16A(io_info, 0x30, init_temp_A);
+ RamWrite16A(io_info, 0x32, init_temp_A);
+ RamWrite16A(io_info, 0x76, init_temp_B);
+ RamWrite16A(io_info, 0x78, init_temp_B);
+
+ /* Enable Lens Temp Correction Function */
+ RamWrite8A(io_info, 0x8C, 0xE9);
+ CAM_INFO(CAM_SENSOR, "[LTC] Enable LTC Function");
+
+ isRearVCMInitDownload = true;
+ rc = 0;
+
+ltc_restore_data:
+ /* 2. Restore the I2C slave address */
+ io_info->cci_client->sid = cci_client_sid_backup;
+
+ return rc;
+}
+
+void clearRearVCMInitDownload(struct camera_io_master *io_info)
+{
+ if (io_info->cci_client->sid == (VCM_COMPONENT_I2C_ADDR_WRITE >> 1)) {
+ isRearVCMInitDownload = false;
+ CAM_INFO(CAM_SENSOR, "[LTC] clear isRearVCMInitDownload");
+ }
+}
+
+int DownloadRearVCMFW(struct camera_io_master *io_info,
+ struct cam_sensor_i2c_reg_array *fwtable, UINT_32 tbsize)
+{
+ int rc = 0;
+ int i = 0;
+ UINT_32 checkbusy_cnt = 0;
+ UINT_32 regdata = 0;
+ UINT_32 backup_data = 0;
+
+ /* Backup standby setting */
+ rc = RamRead8A(io_info, 0x81, &backup_data);
+ if (rc < 0) {
+ return rc;
+ }
+ regdata = backup_data & 0x7F;
+
+ rc = RamWrite8A(io_info, 0x81, regdata);
+ if (rc < 0) {
+ goto restore_data;
+ }
+ WitTim(100);
+
+ /* Enable EEPROM write mode */
+ rc = RamWrite8A(io_info, 0x98, 0xE2);
+ if (rc < 0) {
+ goto restore_data;
+ }
+ rc = RamWrite8A(io_info, 0x99, 0xAE);
+ if (rc < 0) {
+ goto restore_data;
+ }
+ WitTim(1);
+
+ /* EEPROM data writing */
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s: flash firmware, size %d",
+ __func__, tbsize);
+ for (i = 0; i < tbsize; i++) {
+ checkbusy_cnt = 0;
+ io_info->cci_client->sid = VCM_EEPROM_I2C_ADDR_WRITE >> 1;
+ rc = RamWrite8A(io_info,
+ fwtable[i].reg_addr, fwtable[i].reg_data);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]%s: write data failed, rc = %d",
+ __func__, rc);
+ goto restore_data;
+ }
+ io_info->cci_client->sid = VCM_COMPONENT_I2C_ADDR_WRITE >> 1;
+ do {
+ WitTim(5);
+ rc = RamRead8A(io_info, 0xE1, ®data);
+ checkbusy_cnt++;
+ } while (((regdata & 0x01) != 0) && checkbusy_cnt < RETRY_MAX);
+
+ if (checkbusy_cnt == RETRY_MAX) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]%s: write data warning!", __func__);
+ }
+ }
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s: write data complete.", __func__);
+
+restore_data:
+ /* Disable EEPROM write mode & restore standby setting */
+ io_info->cci_client->sid = VCM_COMPONENT_I2C_ADDR_WRITE >> 1;
+ rc = RamWrite8A(io_info, 0x98, 0x00);
+ rc = RamWrite8A(io_info, 0x99, 0x00);
+ rc = RamWrite8A(io_info, 0x81, backup_data);
+ return rc;
+}
+
+int ValidateRearVCMFW(struct camera_io_master *io_info,
+ struct cam_sensor_i2c_reg_array *fwtable, UINT_32 tbsize)
+{
+ int rc = 0;
+ int i = 0;
+
+ io_info->cci_client->sid = VCM_EEPROM_I2C_ADDR_WRITE >> 1;
+ for (i = 0; i < tbsize; i++) {
+ rc = camera_io_dev_poll(io_info,
+ fwtable[i].reg_addr, fwtable[i].reg_data, 0,
+ CAMERA_SENSOR_I2C_TYPE_BYTE,
+ CAMERA_SENSOR_I2C_TYPE_BYTE, 1);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]i2c poll 0x%x failed",
+ fwtable[i].reg_addr);
+ return -EINVAL;
+ }
+ }
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s: complete.", __func__);
+
+ return rc;
+}
+
+int BackupRearVCMData(struct camera_io_master *io_info)
+{
+ int rc;
+ UINT_8 UcIndex, UcBlockNum;
+ UINT_8 SectorData[SECTOR_SIZE];
+ UINT_8 *pData;
+ UINT_8 *NcDatVal;
+ UINT_32 i, regdata, UlSumH, UlSumL;
+ /* Define start address of section 14 (0x1DC0 ~ 0x1DFF) */
+ UINT_32 UlSecAddress = 0x00001DC0;
+
+ /* Read section 14 from sensor EEPROM */
+ io_info->cci_client->sid = OIS_COMPONENT_I2C_ADDR_WRITE >> 1;
+ F40_FlashSectorRead( UlSecAddress, SectorData );
+
+ io_info->cci_client->sid = VCM_EEPROM_I2C_ADDR_WRITE >> 1;
+ /* Overwrite 0x1DE0 ~ 0x1DFF with AF firmware data*/
+ pData = &SectorData[160];
+ for (i = 0x00; i < AF_EEPROM_DATA; i++) {
+ rc = RamRead8A(io_info, i, ®data);
+ if (rc < 0) {
+ return -EINVAL;
+ }
+
+ if (i % 4 == 0)
+ *pData++ = 0xFF;
+ *pData++ = (UINT_8)regdata;
+ }
+
+ /* Write the backup data to sensor EEPROM */
+ io_info->cci_client->sid = OIS_COMPONENT_I2C_ADDR_WRITE >> 1;
+ rc = F40_FlashSectorWrite(UlSecAddress, SectorData);
+ if (rc != 0) {
+ CAM_ERR(CAM_SENSOR, "[VCMFW]%s: backup failed", __func__);
+ /* Write default data for error handling */
+ for (i = 0; i < SECTOR_SIZE; i++) {
+ SectorData[i] = 0xFF;
+ }
+ F40_FlashSectorWrite(UlSecAddress, SectorData);
+ return -EFAULT;
+ }
+
+ /* Update checksum in NVR1 block 0x24~0x25 */
+ UcBlockNum = (UlSecAddress >> 9) & 0x0F;
+ F40_CalcBlockChksum(UcBlockNum, &UlSumH, &UlSumL);
+ F40_FlashSectorRead(0x00010000, SectorData);
+
+ UcIndex = 0x24 * 5 + 1;
+ NcDatVal = (UINT_8 *)&UlSumH;
+ SectorData[UcIndex+3] = *NcDatVal++;
+ SectorData[UcIndex+2] = *NcDatVal++;
+ SectorData[UcIndex+1] = *NcDatVal++;
+ SectorData[UcIndex+0] = *NcDatVal++;
+ UcIndex += 5;
+ NcDatVal = (UINT_8 *)&UlSumL;
+ SectorData[UcIndex+3] = *NcDatVal++;
+ SectorData[UcIndex+2] = *NcDatVal++;
+ SectorData[UcIndex+1] = *NcDatVal++;
+ SectorData[UcIndex+0] = *NcDatVal++;
+
+ rc = F40_FlashSectorWrite(0x00010000, SectorData);
+ if (rc != 0) {
+ CAM_ERR(CAM_SENSOR, "[VCMFW]%s: NVR NG", __func__);
+ return -EFAULT;
+ }
+
+ return rc;
+}
+
+int checkFrontVCMFWUpdate(struct cam_sensor_ctrl_t *s_ctrl)
+{
+ int rc;
+ int update_retry_cnt = 0;
+ UINT_16 cci_client_sid_backup;
+ struct camera_io_master *io_master_info = &(s_ctrl->io_master_info);
+ UINT_32 ReadData = 0;
+ int i;
+ struct cam_sensor_i2c_reg_array *fwtable;
+ UINT_32 size;
+
+ /* Bcakup the I2C slave address */
+ cci_client_sid_backup = s_ctrl->io_master_info.cci_client->sid;
+
+ /* Replace the I2C slave address with VCM eeprom */
+ s_ctrl->io_master_info.cci_client->sid =
+ VCM_EEPROM_I2C_ADDR_WRITE_FRONT_N >> 1;
+
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s: change sid to 0x%02x for EEPROM",
+ __func__, io_master_info->cci_client->sid);
+
+ /* Read Version */
+ rc = camera_io_dev_read(io_master_info,
+ VCM_AK7375_EEPROM_VERSION_ADDR,
+ &ReadData,
+ CAMERA_SENSOR_I2C_TYPE_WORD,
+ CAMERA_SENSOR_I2C_TYPE_BYTE);
+
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s: Version 0x%02x",
+ __func__, ReadData);
+
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]%s: Fail to read version",
+ __func__);
+ } else if (ReadData == VCM_AK7375_EEPROM_VERSION_R13) {
+ /* Replace the I2C slave address with VCM component */
+ s_ctrl->io_master_info.cci_client->sid =
+ VCM_COMPONENT_I2C_ADDR_WRITE_FRONT_N >> 1;
+
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s: change sid to 0x%02x",
+ __func__, io_master_info->cci_client->sid);
+
+ fwtable = VCM_AK7375_DVT_R16;
+ size = ARRAY_SIZE(VCM_AK7375_DVT_R16);
+
+ for (i = 0; i < LUT_MAX_FRONT; i++) {
+ rc = RamRead8A(io_master_info,
+ fwtable[i].reg_addr,
+ &ReadData);
+ if (rc < 0 || ReadData != fwtable[i].reg_data) {
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]%s: Need to run FW update to R16",
+ __func__);
+ update_retry_cnt = RETRY_MAX;
+ break;
+ }
+ }
+ }
+
+ if (!update_retry_cnt) {
+ s_ctrl->io_master_info.cci_client->sid = cci_client_sid_backup;
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]%s: No need to update, rollback sid to 0x%02x",
+ __func__, s_ctrl->io_master_info.cci_client->sid);
+ return rc;
+ }
+
+ /* FW Upgrade starting */
+ while (update_retry_cnt) {
+ rc = DownloadFrontVCMFW(io_master_info, fwtable, size);
+
+ if (rc != 0) {
+ update_retry_cnt--;
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]%s: write fail and retry %d",
+ __func__, update_retry_cnt);
+ continue;
+ }
+
+ rc = ValidateFrontVCMFW(io_master_info, fwtable, size);
+
+ if (rc == 0) {
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]%s: update successfully",
+ __func__);
+ break;
+ }
+
+ update_retry_cnt--;
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]%s: verify fail and retry %d",
+ __func__, update_retry_cnt);
+ }
+
+ /* Restore the I2C slave address */
+ s_ctrl->io_master_info.cci_client->sid = cci_client_sid_backup;
+
+ CAM_INFO(CAM_SENSOR, "[VCMFW]%s: rollback sid to 0x%02x rc %d",
+ __func__, s_ctrl->io_master_info.cci_client->sid, rc);
+
+ return rc;
+}
+
+/* Refer to the below document for function DownloadFrontVCMFW. */
+/* https://drive.google.com/file/d/1IsTbGh5EJ0VIC21l-2JDP5W9PcP6MOze/view */
+int DownloadFrontVCMFW(struct camera_io_master *io_info,
+ struct cam_sensor_i2c_reg_array *fwtable, UINT_32 tbsize)
+{
+ int rc;
+ struct cam_sensor_i2c_reg_setting i2c_reg_settings;
+ UINT_32 ReadData = 0;
+ int i;
+
+ i2c_reg_settings.reg_setting = fwtable;
+ i2c_reg_settings.size = tbsize;
+ i2c_reg_settings.addr_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ i2c_reg_settings.data_type = CAMERA_SENSOR_I2C_TYPE_BYTE;
+ i2c_reg_settings.delay = 0;
+
+ /* Change mode to write */
+ rc = RamWrite8A(io_info,
+ VCM_AK7375_EEPROM_WRITE_MODE_ADDR,
+ VCM_AK7375_EEPROM_WRITE_MODE_DATA);
+
+ if (rc != 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]%s: change to write mode failed",
+ __func__);
+ return rc;
+ }
+
+ /* Write all data */
+ rc = camera_io_dev_write(io_info, &i2c_reg_settings);
+
+ if (rc != 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]%s: update vcm firmware failed",
+ __func__);
+ return rc;
+ }
+
+ /* Store in EEPROM */
+ for (i = 0; i < RETRY_MAX; i++) {
+ rc = RamWrite8A(io_info,
+ VCM_AK7375_EEPROM_STORE_ADDR,
+ VCM_AK7375_EEPROM_STORE_MEM1_DATA);
+ msleep(72);
+
+ rc |= RamWrite8A(io_info,
+ VCM_AK7375_EEPROM_STORE_ADDR,
+ VCM_AK7375_EEPROM_STORE_MEM2_DATA);
+ msleep(216);
+
+ rc |= RamWrite8A(io_info,
+ VCM_AK7375_EEPROM_STORE_ADDR,
+ VCM_AK7375_EEPROM_STORE_MEM3_DATA);
+ msleep(108);
+
+ rc |= RamRead8A(io_info,
+ VCM_AK7375_EEPROM_STORE_STATUS_ADDR,
+ &ReadData);
+
+ /* Bit2 of regist status should be 0 if store successfully. */
+ rc |= (ReadData & (1 << 2));
+
+ if (rc == 0)
+ break;
+ }
+
+ return rc;
+}
+
+int ValidateFrontVCMFW(struct camera_io_master *io_info,
+ struct cam_sensor_i2c_reg_array *fwtable, UINT_32 tbsize)
+{
+ int rc = 0;
+ int i;
+
+ /* Poll all write data */
+ for (i = 0; i < tbsize; i++) {
+ rc = camera_io_dev_poll(io_info,
+ fwtable[i].reg_addr,
+ fwtable[i].reg_data,
+ 0,
+ CAMERA_SENSOR_I2C_TYPE_BYTE,
+ CAMERA_SENSOR_I2C_TYPE_BYTE,
+ 5);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "[VCMFW]%s: poll addr 0x%02x failed",
+ __func__, fwtable[i].reg_addr);
+ break;
+ } else {
+ CAM_INFO(CAM_SENSOR,
+ "[VCMFW]%s: poll addr 0x%02x 0x%02x[%d] ok",
+ __func__, fwtable[i].reg_addr,
+ fwtable[i].reg_data,
+ fwtable[i].reg_data);
+ }
+ }
+
+ return rc;
+}
+
+int GyroReCalib(struct camera_io_master *io_master_info,
+ stReCalib *cal_result)
+{
+ int rc;
+ stReCalib pReCalib;
+
+ g_io_master_info = io_master_info;
+ if (g_io_master_info == NULL)
+ return -EINVAL;
+
+ rc = F40_GyroReCalib(&pReCalib);
+ memcpy(cal_result, &pReCalib, sizeof(stReCalib));
+ if (rc != 0)
+ return rc;
+
+ CAM_INFO(CAM_SENSOR,
+ "[OISCali]%d, FctryOffX = %d(0x%x), FctryOffY = %d(0x%x)",
+ rc, cal_result->SsFctryOffX, cal_result->SsFctryOffX,
+ cal_result->SsFctryOffY, cal_result->SsFctryOffY);
+ CAM_INFO(CAM_SENSOR,
+ "[OISCali]%d, RecalOffX = %d(0x%x), RecalOffY = %d(0x%x)",
+ rc, cal_result->SsRecalOffX, cal_result->SsRecalOffX,
+ cal_result->SsRecalOffY, cal_result->SsRecalOffY);
+ CAM_INFO(CAM_SENSOR,
+ "[OISCali]%d, DiffX = %d(0x%x), DiffY = %d(0x%x)",
+ rc, cal_result->SsDiffX, cal_result->SsDiffX,
+ cal_result->SsDiffY, cal_result->SsDiffY);
+
+ if (abs(cal_result->SsRecalOffX) >= 0x600 ||
+ abs(cal_result->SsRecalOffY) >= 0x600 ||
+ abs(cal_result->SsDiffX) >= 0x1000 ||
+ abs(cal_result->SsDiffY) >= 0x1000) {
+ CAM_ERR(CAM_SENSOR,
+ "[OISCali] Check failed.");
+ return -EINVAL;
+ } else
+ return rc;
+}
+
+int WrGyroOffsetData(void)
+{
+ int rc;
+ rc = F40_WrGyroOffsetData();
+ return rc;
+}
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/fw_update.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/fw_update.h
new file mode 100644
index 0000000..8d90536
--- /dev/null
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_fw_update/fw_update.h
@@ -0,0 +1,39 @@
+/* OIS calibration interface for LC898123 F40
+ *
+ */
+#include "PhoneUpdate.h"
+#include "../cam_sensor/cam_sensor_dev.h"
+
+void RamWrite32A(UINT_16 RamAddr, UINT_32 RamData);
+void RamRead32A(UINT_16 RamAddr, UINT_32 *ReadData);
+int RamWrite8A(struct camera_io_master *io_info,
+ UINT_32 RamAddr, UINT_32 RamData);
+int RamRead8A(struct camera_io_master *io_info,
+ UINT_32 RamAddr, UINT_32 *ReadData);
+int CntWrt(UINT_8 *PcSetDat, UINT_16 CntWrt);
+int CntRd3(UINT_32 addr, void *PcSetDat, UINT_16 UsDatNum);
+void WitTim(UINT_16) ;
+void WPBCtrl(UINT_8 UcCtrl);
+bool checkOISFWversion(UINT_16 *cal_id, UINT_32 *module_maker);
+int checkOISFWUpdate(struct cam_sensor_ctrl_t *s_ctrl);
+int checkRearVCMFWUpdate(struct cam_sensor_ctrl_t *s_ctrl);
+int DownloadRearVCMFW(struct camera_io_master *io_info,
+ struct cam_sensor_i2c_reg_array *fwtable,
+ UINT_32 tbsize);
+int ValidateRearVCMFW(struct camera_io_master *io_info,
+ struct cam_sensor_i2c_reg_array *fwtable,
+ UINT_32 tbsize);
+int BackupRearVCMData(struct camera_io_master *io_info);
+int checkFrontVCMFWUpdate(struct cam_sensor_ctrl_t *s_ctrl);
+int DownloadFrontVCMFW(struct camera_io_master *io_info,
+ struct cam_sensor_i2c_reg_array *fwtable,
+ UINT_32 tbsize);
+int ValidateFrontVCMFW(struct camera_io_master *io_info,
+ struct cam_sensor_i2c_reg_array *fwtable,
+ UINT_32 tbsize);
+int GyroReCalib(struct camera_io_master *io_master_info,
+ stReCalib *cal_result);
+int WrGyroOffsetData(void);
+int checkRearVCMFWUpdate_temp(struct cam_sensor_ctrl_t *s_ctrl);
+int checkRearVCMLTC(struct camera_io_master *io_info);
+void clearRearVCMInitDownload(struct camera_io_master *io_info);
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_ois/cam_ois_core.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_ois/cam_ois_core.c
index 49fac7f..56da116 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_ois/cam_ois_core.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_ois/cam_ois_core.c
@@ -18,29 +18,61 @@
#include "cam_sensor_util.h"
#include "cam_debug_util.h"
#include "cam_res_mgr_api.h"
+#include "../cam_fw_update/fw_update.h"
+
+static struct cam_ois_timer_t ois_timer;
+DEFINE_MSM_MUTEX(ois_shift_mutex);
+
+int cam_ois_calibration(struct cam_ois_ctrl_t *o_ctrl,
+ stReCalib *cal_result)
+{
+ int rc;
+
+ rc = GyroReCalib(&o_ctrl->io_master_info, cal_result);
+ if (rc != 0)
+ CAM_ERR(CAM_OIS,
+ "[OISCali] ReCalib FAIL, rc = %d", rc);
+ else {
+ rc = WrGyroOffsetData();
+ if (rc != 0)
+ CAM_ERR(CAM_OIS,
+ "[OISCali] WrGyro FAIL, rc = %d", rc);
+ else
+ CAM_INFO(CAM_OIS,
+ "[OISCali] SUCCESS");
+ }
+
+ return rc;
+}
int32_t cam_ois_construct_default_power_setting(
struct cam_sensor_power_ctrl_t *power_info)
{
int rc = 0;
- power_info->power_setting_size = 1;
+ power_info->power_setting_size = 2;
power_info->power_setting =
(struct cam_sensor_power_setting *)
- kzalloc(sizeof(struct cam_sensor_power_setting),
+ kzalloc(sizeof(struct cam_sensor_power_setting) *
+ power_info->power_setting_size,
GFP_KERNEL);
if (!power_info->power_setting)
return -ENOMEM;
- power_info->power_setting[0].seq_type = SENSOR_VAF;
- power_info->power_setting[0].seq_val = CAM_VAF;
+ power_info->power_setting[0].seq_type = SENSOR_VIO;
+ power_info->power_setting[0].seq_val = CAM_VIO;
power_info->power_setting[0].config_val = 1;
- power_info->power_setting[0].delay = 2;
+ power_info->power_setting[0].delay = 0;
+ power_info->power_setting[1].seq_type = SENSOR_VAF;
+ power_info->power_setting[1].seq_val = CAM_VAF;
+ power_info->power_setting[1].config_val = 1;
+ power_info->power_setting[1].delay = 2;
power_info->power_down_setting_size = 1;
power_info->power_down_setting =
(struct cam_sensor_power_setting *)
- kzalloc(sizeof(struct cam_sensor_power_setting),
+ kzalloc(sizeof(struct cam_sensor_power_setting) *
+ power_info->power_down_setting_size,
GFP_KERNEL);
if (!power_info->power_down_setting) {
rc = -ENOMEM;
@@ -55,8 +87,6 @@ int32_t cam_ois_construct_default_power_setting(
free_power_settings:
kfree(power_info->power_setting);
- power_info->power_setting = NULL;
- power_info->power_setting_size = 0;
return rc;
}
@@ -235,13 +265,17 @@ static int cam_ois_apply_settings(struct cam_ois_ctrl_t *o_ctrl,
size = i2c_list->i2c_settings.size;
for (i = 0; i < size; i++) {
rc = camera_io_dev_poll(
- &(o_ctrl->io_master_info),
- i2c_list->i2c_settings.reg_setting[i].reg_addr,
- i2c_list->i2c_settings.reg_setting[i].reg_data,
- i2c_list->i2c_settings.reg_setting[i].data_mask,
- i2c_list->i2c_settings.addr_type,
- i2c_list->i2c_settings.data_type,
- i2c_list->i2c_settings.reg_setting[i].delay);
+ &(o_ctrl->io_master_info),
+ i2c_list->i2c_settings.
+ reg_setting[i].reg_addr,
+ i2c_list->i2c_settings.
+ reg_setting[i].reg_data,
+ i2c_list->i2c_settings.
+ reg_setting[i].data_mask,
+ i2c_list->i2c_settings.addr_type,
+ i2c_list->i2c_settings.data_type,
+ i2c_list->i2c_settings.
+ reg_setting[i].delay);
if (rc < 0) {
CAM_ERR(CAM_OIS,
"i2c poll apply setting Fail");
@@ -274,7 +308,7 @@ static int cam_ois_slaveInfo_pkt_parser(struct cam_ois_ctrl_t *o_ctrl,
o_ctrl->ois_fw_flag = ois_info->ois_fw_flag;
o_ctrl->is_ois_calib = ois_info->is_ois_calib;
memcpy(o_ctrl->ois_name, ois_info->ois_name, 32);
- o_ctrl->io_master_info.cci_client->retries = 3;
+ o_ctrl->io_master_info.cci_client->retries = 1;
o_ctrl->io_master_info.cci_client->id_map = 0;
memcpy(&(o_ctrl->opcode), &(ois_info->opcode),
sizeof(struct cam_ois_opcode));
@@ -413,6 +447,279 @@ static int cam_ois_fw_download(struct cam_ois_ctrl_t *o_ctrl)
}
/**
+ * cam_ois_shift_data_enqueue - enqueue shift data to ring buffer
+ * @time_readout: ctrl structure
+ * @shift_x: shift in x
+ * @shift_y: shift in y
+ * @buffer: rint buffer
+ *
+ * Returns success or failure
+ */
+static int cam_ois_shift_data_enqueue(int64_t time_readout, int16_t shift_x,
+ int16_t shift_y, struct cam_ois_shift_buffer *buffer)
+{
+ int rc = 0;
+
+ mutex_lock(&ois_shift_mutex);
+ if (buffer->write_pos >= CAM_OIS_SHIFT_DATA_BUFFER_SIZE ||
+ buffer->write_pos < 0) {
+ CAM_ERR(CAM_OIS,
+ "invalid OIS shift buffer index: %d",
+ buffer->write_pos);
+ rc = -EFAULT;
+ } else {
+ struct cam_ois_shift *pb = &buffer->buffer[buffer->write_pos];
+
+ pb->time_readout = time_readout;
+ pb->ois_shift_x = shift_x;
+ pb->ois_shift_y = shift_y;
+ buffer->write_pos++;
+ if (buffer->write_pos == CAM_OIS_SHIFT_DATA_BUFFER_SIZE) {
+ buffer->write_pos = 0;
+ buffer->is_full = true;
+ }
+ rc = 0;
+ }
+ mutex_unlock(&ois_shift_mutex);
+ return rc;
+}
+
+/**
+ * cam_ois_read_work - worker function of read timer
+ * @work: work
+ *
+ * Returns success or failure
+ */
+static void cam_ois_read_work(struct work_struct *work)
+{
+ uint8_t buf[8] = { 0 };
+ int32_t rc = 0;
+ int16_t shift_x, shift_y;
+ struct timespec ts;
+ int64_t time_readout;
+ struct cam_ois_timer_t *ois_timer_in;
+
+ ois_timer_in = container_of(work, struct cam_ois_timer_t, g_work);
+ get_monotonic_boottime(&ts);
+ rc = camera_io_dev_read_seq(&ois_timer_in->o_ctrl->io_master_info,
+ 0xE001, &buf[0], CAMERA_SENSOR_I2C_TYPE_WORD, 6);
+
+ if (rc != 0) {
+ ois_timer.i2c_fail_count++;
+ CAM_ERR(CAM_OIS,
+ "read seq fail. cnt = %d", ++ois_timer.i2c_fail_count);
+ if (ois_timer.i2c_fail_count >= MAX_FAIL_CNT) {
+ CAM_ERR(CAM_OIS, "Too many i2c failed. Stop timer.");
+ ois_timer.ois_timer_state = CAM_OIS_TIME_ERROR;
+ }
+ } else {
+ ois_timer.i2c_fail_count = 0;
+ time_readout = (int64_t)ts.tv_sec * 1000000000LL + ts.tv_nsec;
+ shift_x = (int16_t)(((uint16_t)buf[0] << 8) + (uint16_t)buf[1]);
+ shift_y = (int16_t)(((uint16_t)buf[2] << 8) + (uint16_t)buf[3]);
+
+ rc = cam_ois_shift_data_enqueue(time_readout, shift_x,
+ shift_y, &ois_timer_in->o_ctrl->buf);
+ if (rc != 0)
+ CAM_ERR(CAM_OIS, "OIS shift data enqueue failed");
+ }
+}
+
+/**
+ * cam_ois_shift_timer - ois shift reader timer expiracy callback func
+ * @timer: pointer to the hrtimer struct
+ *
+ * Returns hrtimer_restart state
+ */
+static enum hrtimer_restart cam_ois_shift_timer(struct hrtimer *timer)
+{
+ ktime_t currtime, interval;
+ struct cam_ois_timer_t *ois_timer_in;
+
+ ois_timer_in = container_of(timer, struct cam_ois_timer_t, hr_timer);
+ if (ois_timer.ois_timer_state != CAM_OIS_TIME_ERROR) {
+ queue_work(ois_timer_in->ois_wq, &ois_timer_in->g_work);
+ currtime = ktime_get();
+ interval = ktime_set(0, READ_OUT_TIME);
+ hrtimer_forward(timer, currtime, interval);
+
+ return HRTIMER_RESTART;
+ }
+
+ CAM_ERR(CAM_OIS, "HRTIMER_NORESTART");
+ return HRTIMER_NORESTART;
+}
+
+/**
+ * cam_ois_stop_offset_reader_thread - stop shift reader
+ *
+ * Returns success or failure
+ */
+static int cam_ois_stop_shift_reader(void)
+{
+ if (ois_timer.ois_timer_state == CAM_OIS_TIME_ACTIVE ||
+ ois_timer.ois_timer_state == CAM_OIS_TIME_ERROR) {
+ hrtimer_cancel(&ois_timer.hr_timer);
+ destroy_workqueue(ois_timer.ois_wq);
+ ois_timer.ois_timer_state = CAM_OIS_TIME_INACTIVE;
+ CAM_INFO(CAM_OIS, "Successfully stopped OIS shift reader.");
+ } else {
+ CAM_ERR(CAM_OIS,
+ "Invalid ois timer state:%d",
+ ois_timer.ois_timer_state);
+ }
+ return 0;
+}
+
+/**
+ * cam_ois_start_shift_reader_thread - start shift reader
+ * @o_ctrl: ctrl structure
+ *
+ * Returns success or failure
+ */
+static int cam_ois_start_shift_reader(struct cam_ois_ctrl_t *o_ctrl)
+{
+ ktime_t ktime;
+ int rc = 0;
+
+ if (ois_timer.ois_timer_state == CAM_OIS_TIME_ERROR) {
+ CAM_ERR(CAM_OIS, "OIS Timer Error.");
+ cam_ois_stop_shift_reader();
+ return -EFAULT;
+ }
+ ois_timer.i2c_fail_count = 0;
+ if (ois_timer.ois_timer_state != CAM_OIS_TIME_ACTIVE) {
+ o_ctrl->io_master_info.cci_client->i2c_freq_mode =
+ I2C_FAST_PLUS_MODE;
+ ois_timer.o_ctrl = o_ctrl;
+
+ // set worker function and work queue
+ INIT_WORK(&ois_timer.g_work, cam_ois_read_work);
+ ois_timer.ois_wq = alloc_workqueue("ois_wq", WQ_HIGHPRI, 1);
+ if (!ois_timer.ois_wq) {
+ CAM_ERR(CAM_OIS, "ois_wq create failed.");
+ return -EFAULT;
+ }
+
+ // set timer
+ ktime = ktime_set(0, READ_OUT_TIME);
+ hrtimer_init(&ois_timer.hr_timer, CLOCK_MONOTONIC,
+ HRTIMER_MODE_REL);
+ ois_timer.hr_timer.function = &cam_ois_shift_timer;
+ hrtimer_start(&ois_timer.hr_timer, ktime,
+ HRTIMER_MODE_REL);
+ ois_timer.ois_timer_state = CAM_OIS_TIME_ACTIVE;
+ } else {
+ CAM_ERR(CAM_OIS,
+ "invalid timer state = %d", ois_timer.ois_timer_state);
+ return -EFAULT;
+ }
+
+ mutex_lock(&ois_shift_mutex);
+ o_ctrl->buf.write_pos = 0;
+ o_ctrl->buf.is_full = false;
+ mutex_unlock(&ois_shift_mutex);
+
+ CAM_INFO(CAM_OIS, "Successfully started OIS shift reader.");
+ return rc;
+}
+
+/**
+ * cam_ois_get_shift - write ois shift data to user space
+ * @o_ctrl: ctrl structure
+ * @query_size_handle: handle of query_size in user space
+ * @shift_data_handle: handle of shift_data in user space
+ *
+ * Returns success or failure
+ */
+static int cam_ois_get_shift(struct cam_ois_ctrl_t *o_ctrl,
+ uint64_t query_size_handle, uint64_t shift_data_handle)
+{
+ int rc = 0;
+ struct cam_ois_shift buf[CAM_OIS_SHIFT_DATA_BUFFER_SIZE];
+ uint8_t query_size = 0;
+ int32_t read_pos = 0;
+ int32_t write_pos = 0;
+
+ // copy from ring buffer to local continuous buffer
+ mutex_lock(&ois_shift_mutex);
+ write_pos = o_ctrl->buf.write_pos;
+ if (o_ctrl->buf.is_full) {
+ read_pos = write_pos;
+ do {
+ buf[query_size++] = o_ctrl->buf.buffer[read_pos++];
+ if (read_pos == CAM_OIS_SHIFT_DATA_BUFFER_SIZE)
+ read_pos = 0;
+ } while (read_pos != write_pos);
+ } else {
+ read_pos = 0;
+ while (read_pos != write_pos)
+ buf[query_size++] = o_ctrl->buf.buffer[read_pos++];
+ }
+ // reset ring buffer
+ o_ctrl->buf.write_pos = 0;
+ o_ctrl->buf.is_full = false;
+ mutex_unlock(&ois_shift_mutex);
+
+ // copy to user space
+ if (copy_to_user((void __user *) query_size_handle, &query_size,
+ sizeof(uint8_t))) {
+ CAM_ERR(CAM_OIS, "ois_get_shift: copy to user failed!");
+ return -EFAULT;
+ }
+ if (copy_to_user((void __user *) shift_data_handle, buf,
+ sizeof(struct cam_ois_shift) * query_size)) {
+ CAM_ERR(CAM_OIS, "ois_get_shift: copy to user failed!");
+ return -EFAULT;
+ }
+
+ return rc;
+}
+
+/**
+ * cam_ois_read_reg - read register data and copy to user space
+ * @o_ctrl: ctrl structure
+ * @cmd_get_ois: handle of shift_data in user space
+ *
+ * Returns success or failure
+ */
+static int cam_ois_read_reg(struct cam_ois_ctrl_t *o_ctrl,
+ struct cam_cmd_get_ois_data *cmd_get_ois)
+{
+ uint8_t buf[8] = { 0 };
+ int32_t rc = 0;
+ uint32_t addr = cmd_get_ois->reg_addr;
+ int32_t num_bytes = cmd_get_ois->reg_data;
+
+ if (addr <= 0 || addr > 0xFFFF) {
+ CAM_ERR(CAM_OIS,
+ "Invalid addr while read OIS data: %x", addr);
+ return -EINVAL;
+ }
+
+ if (num_bytes <= 0 || num_bytes > 8) {
+ CAM_ERR(CAM_OIS,
+ "Invalid read size while read OIS data: %d", num_bytes);
+ return -EINVAL;
+ }
+
+ rc = camera_io_dev_read_seq(&o_ctrl->io_master_info,
+ addr, buf, CAMERA_SENSOR_I2C_TYPE_WORD, num_bytes);
+
+ if (rc) {
+ CAM_ERR(CAM_OIS, "camera_io_dev_read_seq failed!");
+ return rc;
+ }
+
+ if (copy_to_user((void __user *) cmd_get_ois->query_data_handle,
+ buf, sizeof(uint8_t) * num_bytes)) {
+ CAM_ERR(CAM_OIS, "ois_read_reg: copy to user failed!");
+ }
+
+ return rc;
+}
+
+/**
* cam_ois_pkt_parse - Parse csl packet
* @o_ctrl: ctrl structure
* @arg: Camera control command argument
@@ -438,6 +745,9 @@ static int cam_ois_pkt_parse(struct cam_ois_ctrl_t *o_ctrl, void *arg)
struct cam_ois_soc_private *soc_private =
(struct cam_ois_soc_private *)o_ctrl->soc_info.soc_private;
struct cam_sensor_power_ctrl_t *power_info = &soc_private->power_info;
+ struct cam_cmd_get_ois_data *cmd_get_ois = NULL;
+ int32_t *cal_rc;
+ stReCalib *cal_result;
ioctl_ctrl = (struct cam_control *)arg;
if (copy_from_user(&dev_config, (void __user *) ioctl_ctrl->handle,
@@ -519,7 +829,6 @@ static int cam_ois_pkt_parse(struct cam_ois_ctrl_t *o_ctrl, void *arg)
i2c_reg_settings->is_settings_valid = 1;
i2c_reg_settings->request_id = 0;
rc = cam_sensor_i2c_command_parser(
- &o_ctrl->io_master_info,
i2c_reg_settings,
&cmd_desc[i], 1);
if (rc < 0) {
@@ -528,15 +837,14 @@ static int cam_ois_pkt_parse(struct cam_ois_ctrl_t *o_ctrl, void *arg)
return rc;
}
} else if ((o_ctrl->is_ois_calib != 0) &&
- (o_ctrl->i2c_calib_data.is_settings_valid ==
- 0)) {
+ (o_ctrl->i2c_calib_data.
+ is_settings_valid == 0)) {
CAM_DBG(CAM_OIS,
"Received calib settings");
i2c_reg_settings = &(o_ctrl->i2c_calib_data);
i2c_reg_settings->is_settings_valid = 1;
i2c_reg_settings->request_id = 0;
rc = cam_sensor_i2c_command_parser(
- &o_ctrl->io_master_info,
i2c_reg_settings,
&cmd_desc[i], 1);
if (rc < 0) {
@@ -608,8 +916,7 @@ static int cam_ois_pkt_parse(struct cam_ois_ctrl_t *o_ctrl, void *arg)
i2c_reg_settings = &(o_ctrl->i2c_mode_data);
i2c_reg_settings->is_settings_valid = 1;
i2c_reg_settings->request_id = 0;
- rc = cam_sensor_i2c_command_parser(&o_ctrl->io_master_info,
- i2c_reg_settings,
+ rc = cam_sensor_i2c_command_parser(i2c_reg_settings,
cmd_desc, 1);
if (rc < 0) {
CAM_ERR(CAM_OIS, "OIS pkt parsing failed: %d", rc);
@@ -627,6 +934,74 @@ static int cam_ois_pkt_parse(struct cam_ois_ctrl_t *o_ctrl, void *arg)
CAM_ERR(CAM_OIS,
"Fail deleting Mode data: rc: %d", rc);
break;
+ case CAM_OIS_PACKET_OPCODE_SHIFT_READER_START:
+ rc = cam_ois_start_shift_reader(o_ctrl);
+ break;
+ case CAM_OIS_PACKET_OPCODE_SHIFT_READER_STOP:
+ rc = cam_ois_stop_shift_reader();
+ break;
+ case CAM_OIS_PACKET_OPCODE_SHIFT_GET:
+ case CAM_OIS_PACKET_OPCODE_READ:
+ if (csl_packet->num_cmd_buf != 1) {
+ CAM_ERR(CAM_OIS,
+ "More than one cmd buf found in shift_get");
+ return -EINVAL;
+ }
+
+ offset = (uint32_t *)&csl_packet->payload;
+ offset += (csl_packet->cmd_buf_offset / sizeof(uint32_t));
+ cmd_desc = (struct cam_cmd_buf_desc *)(offset);
+ total_cmd_buf_in_bytes = cmd_desc->length;
+ if (!total_cmd_buf_in_bytes) {
+ CAM_ERR(CAM_OIS,
+ "Empty cmd buf found in shift_get");
+ return -EINVAL;
+ }
+
+ rc = cam_mem_get_cpu_buf(cmd_desc->mem_handle,
+ (uint64_t *)&generic_ptr, &len_of_buff);
+ if (rc < 0) {
+ CAM_ERR(CAM_OIS, "Failed to get cpu buf");
+ return rc;
+ }
+ cmd_buf = (uint32_t *)generic_ptr;
+ if (!cmd_buf) {
+ CAM_ERR(CAM_OIS, "invalid cmd buf");
+ return -EINVAL;
+ }
+ cmd_buf += cmd_desc->offset / sizeof(uint32_t);
+ cmd_get_ois = (struct cam_cmd_get_ois_data *)cmd_buf;
+
+ if ((csl_packet->header.op_code & 0xFFFFFF) ==
+ CAM_OIS_PACKET_OPCODE_SHIFT_GET) {
+ rc = cam_ois_get_shift(o_ctrl,
+ cmd_get_ois->query_size_handle,
+ cmd_get_ois->query_data_handle);
+ } else if ((csl_packet->header.op_code & 0xFFFFFF) ==
+ CAM_OIS_PACKET_OPCODE_READ) {
+ rc = cam_ois_read_reg(o_ctrl, cmd_get_ois);
+ }
+ break;
+ case CAM_OIS_PACKET_OPCODE_CALIBRATION:
+ offset = (uint32_t *)&csl_packet->payload;
+ offset += (csl_packet->cmd_buf_offset / sizeof(uint32_t));
+ cmd_desc = (struct cam_cmd_buf_desc *)(offset);
+ rc = cam_mem_get_cpu_buf(cmd_desc[0].mem_handle,
+ (uint64_t *)&generic_ptr, &len_of_buff);
+ if (rc < 0) {
+ CAM_ERR(CAM_OIS, "Failed to get cpu buf");
+ return rc;
+ }
+ cmd_buf = (uint32_t *)generic_ptr;
+ if (!cmd_buf) {
+ CAM_ERR(CAM_OIS, "invalid cmd buf");
+ return -EINVAL;
+ }
+ cmd_buf += cmd_desc->offset / sizeof(uint32_t);
+ cal_rc = (int32_t *)cmd_buf;
+ cal_result = (stReCalib *)(cmd_buf + 1);
+ *cal_rc = cam_ois_calibration(o_ctrl, cal_result);
+ break;
default:
break;
}
@@ -638,10 +1013,7 @@ static int cam_ois_pkt_parse(struct cam_ois_ctrl_t *o_ctrl, void *arg)
void cam_ois_shutdown(struct cam_ois_ctrl_t *o_ctrl)
{
- int rc = 0;
- struct cam_ois_soc_private *soc_private =
- (struct cam_ois_soc_private *)o_ctrl->soc_info.soc_private;
- struct cam_sensor_power_ctrl_t *power_info = &soc_private->power_info;
+ int rc;
if (o_ctrl->cam_ois_state == CAM_OIS_INIT)
return;
@@ -661,13 +1033,6 @@ void cam_ois_shutdown(struct cam_ois_ctrl_t *o_ctrl)
o_ctrl->bridge_intf.session_hdl = -1;
}
- kfree(power_info->power_setting);
- kfree(power_info->power_down_setting);
- power_info->power_setting = NULL;
- power_info->power_down_setting = NULL;
- power_info->power_down_setting_size = 0;
- power_info->power_setting_size = 0;
-
o_ctrl->cam_ois_state = CAM_OIS_INIT;
}
@@ -680,27 +1045,15 @@ void cam_ois_shutdown(struct cam_ois_ctrl_t *o_ctrl)
*/
int cam_ois_driver_cmd(struct cam_ois_ctrl_t *o_ctrl, void *arg)
{
- int rc = 0;
- struct cam_ois_query_cap_t ois_cap = {0};
- struct cam_control *cmd = (struct cam_control *)arg;
- struct cam_ois_soc_private *soc_private = NULL;
- struct cam_sensor_power_ctrl_t *power_info = NULL;
+ int rc = 0;
+ struct cam_ois_query_cap_t ois_cap = {0};
+ struct cam_control *cmd = (struct cam_control *)arg;
- if (!o_ctrl || !cmd) {
- CAM_ERR(CAM_OIS, "Invalid arguments");
+ if (!o_ctrl) {
+ CAM_ERR(CAM_OIS, "e_ctrl is NULL");
return -EINVAL;
}
- if (cmd->handle_type != CAM_HANDLE_USER_POINTER) {
- CAM_ERR(CAM_OIS, "Invalid handle type: %d",
- cmd->handle_type);
- return -EINVAL;
- }
-
- soc_private =
- (struct cam_ois_soc_private *)o_ctrl->soc_info.soc_private;
- power_info = &soc_private->power_info;
-
mutex_lock(&(o_ctrl->ois_mutex));
switch (cmd->op_code) {
case CAM_QUERY_CAP:
@@ -771,13 +1124,6 @@ int cam_ois_driver_cmd(struct cam_ois_ctrl_t *o_ctrl, void *arg)
o_ctrl->bridge_intf.link_hdl = -1;
o_ctrl->bridge_intf.session_hdl = -1;
o_ctrl->cam_ois_state = CAM_OIS_INIT;
-
- kfree(power_info->power_setting);
- kfree(power_info->power_down_setting);
- power_info->power_setting = NULL;
- power_info->power_down_setting = NULL;
- power_info->power_down_setting_size = 0;
- power_info->power_setting_size = 0;
break;
case CAM_STOP_DEV:
if (o_ctrl->cam_ois_state != CAM_OIS_START) {
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_ois/cam_ois_dev.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_ois/cam_ois_dev.h
index 3b7195e..5b93f35 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_ois/cam_ois_dev.h
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_ois/cam_ois_dev.h
@@ -30,6 +30,11 @@
#define DEFINE_MSM_MUTEX(mutexname) \
static struct mutex mutexname = __MUTEX_INITIALIZER(mutexname)
+#define READ_OUT_TIME 5000000 /*5ms*/
+#define CAM_OIS_SHIFT_DATA_BUFFER_SIZE 15
+#define MAX_FAIL_CNT 3
+
+struct cam_ois_ctrl_t;
enum cam_ois_state {
CAM_OIS_INIT,
@@ -38,6 +43,13 @@ enum cam_ois_state {
CAM_OIS_START,
};
+enum cam_ois_timer_state_t {
+ CAM_OIS_TIME_INIT,
+ CAM_OIS_TIME_ACTIVE,
+ CAM_OIS_TIME_INACTIVE,
+ CAM_OIS_TIME_ERROR,
+};
+
/**
* struct cam_ois_registered_driver_t - registered driver info
* @platform_driver : flag indicates if platform driver is registered
@@ -89,6 +101,36 @@ struct cam_ois_intf_params {
};
/**
+ * struct cam_ois_shift_buffer - OIS shift data ring buffer
+ * @buffer : array of shift data readout
+ * @write_pos : next position to write
+ * @is_full : flag of full buffer
+ */
+struct cam_ois_shift_buffer {
+ struct cam_ois_shift buffer[CAM_OIS_SHIFT_DATA_BUFFER_SIZE];
+ int32_t write_pos;
+ bool is_full;
+};
+
+/**
+ * struct cam_ois_timer_t - OIS shift reader timer
+ * @hr_timer : timer
+ * @ois_wq : worker queue of actual worker procedure
+ * @g_work : actual worker
+ * @ois_timer_state : state of timer
+ * @o_ctrl : ois control structure
+ * @i2c_fail_count : number of i2c fails
+ */
+struct cam_ois_timer_t {
+ struct hrtimer hr_timer;
+ struct workqueue_struct *ois_wq;
+ struct work_struct g_work;
+ enum cam_ois_timer_state_t ois_timer_state;
+ struct cam_ois_ctrl_t *o_ctrl;
+ int i2c_fail_count;
+};
+
+/**
* struct cam_ois_ctrl_t - OIS ctrl private data
* @pdev : platform device
* @ois_mutex : ois mutex
@@ -107,7 +149,7 @@ struct cam_ois_intf_params {
* @is_ois_calib : flag for Calibration data
* @opcode : ois opcode
* @device_name : Device name
- *
+ * @buf : ois shift data buffer
*/
struct cam_ois_ctrl_t {
struct platform_device *pdev;
@@ -127,6 +169,7 @@ struct cam_ois_ctrl_t {
uint8_t ois_fw_flag;
uint8_t is_ois_calib;
struct cam_ois_opcode opcode;
+ struct cam_ois_shift_buffer buf;
};
#endif /*_CAM_OIS_DEV_H_ */
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/Makefile b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/Makefile
index 5a9441f..2f54be6 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/Makefile
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/Makefile
@@ -7,4 +7,7 @@
ccflags-y += -Idrivers/media/platform/msm/camera/cam_smmu/
ccflags-y += -Idrivers/media/platform/msm/camera/cam_utils
+# For Easel
+ccflags-y += -Idrivers/misc/mnh
+
obj-$(CONFIG_SPECTRA_CAMERA) += cam_sensor_dev.o cam_sensor_core.o cam_sensor_soc.o
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_core.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_core.c
index 356f66c..049ad15 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_core.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_core.c
@@ -17,6 +17,15 @@
#include "cam_soc_util.h"
#include "cam_trace.h"
+#ifdef CONFIG_MNH_SM_HOST
+#include "mnh-sm.h"
+
+int cam_sensor_config_easel(struct cam_sensor_ctrl_t *s_ctrl);
+#endif
+
+#ifdef CONFIG_CAMERA_FW_UPDATE
+#include "../cam_fw_update/fw_update.h"
+#endif
static void cam_sensor_update_req_mgr(
struct cam_sensor_ctrl_t *s_ctrl,
@@ -190,8 +199,9 @@ static int32_t cam_sensor_i2c_pkt_parse(struct cam_sensor_ctrl_t *s_ctrl,
}
i2c_reg_settings =
- &i2c_data->per_frame[csl_packet->header.request_id %
- MAX_PER_FRAME_ARRAY];
+ &i2c_data->
+ per_frame[csl_packet->header.request_id %
+ MAX_PER_FRAME_ARRAY];
CAM_DBG(CAM_SENSOR, "Received Packet: %lld req: %lld",
csl_packet->header.request_id % MAX_PER_FRAME_ARRAY,
csl_packet->header.request_id);
@@ -230,8 +240,7 @@ static int32_t cam_sensor_i2c_pkt_parse(struct cam_sensor_ctrl_t *s_ctrl,
offset += csl_packet->cmd_buf_offset / 4;
cmd_desc = (struct cam_cmd_buf_desc *)(offset);
- rc = cam_sensor_i2c_command_parser(&s_ctrl->io_master_info,
- i2c_reg_settings, cmd_desc, 1);
+ rc = cam_sensor_i2c_command_parser(i2c_reg_settings, cmd_desc, 1);
if (rc < 0) {
CAM_ERR(CAM_SENSOR, "Fail parsing I2C Pkt: %d", rc);
return rc;
@@ -254,6 +263,14 @@ static int32_t cam_sensor_i2c_modes_util(
int32_t rc = 0;
uint32_t i, size;
+ uint16_t default_sid = 0;
+ if ((io_master_info->master_type == CCI_MASTER) &&
+ (0 != i2c_list->i2c_settings.slave_addr)) {
+ default_sid = io_master_info->cci_client->sid;
+ io_master_info->cci_client->sid =
+ i2c_list->i2c_settings.slave_addr >> 1;
+ }
+
if (i2c_list->op_code == CAM_SENSOR_I2C_WRITE_RANDOM) {
rc = camera_io_dev_write(io_master_info,
&(i2c_list->i2c_settings));
@@ -290,18 +307,48 @@ static int32_t cam_sensor_i2c_modes_util(
for (i = 0; i < size; i++) {
rc = camera_io_dev_poll(
io_master_info,
- i2c_list->i2c_settings.reg_setting[i].reg_addr,
- i2c_list->i2c_settings.reg_setting[i].reg_data,
- i2c_list->i2c_settings.reg_setting[i].data_mask,
+ i2c_list->i2c_settings.
+ reg_setting[i].reg_addr,
+ i2c_list->i2c_settings.
+ reg_setting[i].reg_data,
+ i2c_list->i2c_settings.
+ reg_setting[i].data_mask,
i2c_list->i2c_settings.addr_type,
- i2c_list->i2c_settings.data_type,
- i2c_list->i2c_settings.reg_setting[i].delay);
+ i2c_list->i2c_settings.data_type,
+ i2c_list->i2c_settings.
+ reg_setting[i].delay);
if (rc < 0) {
CAM_ERR(CAM_SENSOR,
"i2c poll apply setting Fail: %d", rc);
return rc;
}
}
+ } else if (i2c_list->op_code == CAM_SENSOR_I2C_READ) {
+ size = i2c_list->i2c_settings.size;
+ for (i = 0; i < size; i++) {
+ rc = camera_io_dev_read(
+ io_master_info,
+ i2c_list->i2c_settings.
+ reg_setting[i].reg_addr,
+ &(i2c_list->i2c_settings.
+ reg_setting[i].reg_data),
+ i2c_list->i2c_settings.addr_type,
+ i2c_list->i2c_settings.data_type);
+ if (rc < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "i2c read Fail: %d", rc);
+ return rc;
+ }
+ CAM_INFO(CAM_SENSOR,
+ "i2c read addr: 0x%x, data: 0x%x",
+ i2c_list->i2c_settings.reg_setting[i].reg_addr,
+ i2c_list->i2c_settings.reg_setting[i].reg_data);
+ }
+ }
+
+ if ((io_master_info->master_type == CCI_MASTER) &&
+ (0 != i2c_list->i2c_settings.slave_addr)) {
+ io_master_info->cci_client->sid = default_sid;
}
return rc;
@@ -322,7 +369,7 @@ int32_t cam_sensor_update_i2c_info(struct cam_cmd_i2c_info *i2c_info,
}
cci_client->cci_i2c_master = s_ctrl->cci_i2c_master;
cci_client->sid = i2c_info->slave_addr >> 1;
- cci_client->retries = 3;
+ cci_client->retries = 1;
cci_client->id_map = 0;
cci_client->i2c_freq_mode = i2c_info->i2c_freq_mode;
CAM_DBG(CAM_SENSOR, " Master: %d sid: %d freq_mode: %d",
@@ -346,6 +393,7 @@ int32_t cam_sensor_update_slave_info(struct cam_cmd_probe *probe_info,
probe_info->expected_data;
s_ctrl->sensordata->slave_info.sensor_id_mask =
probe_info->data_mask;
+ s_ctrl->fw_update_flag = probe_info->fw_update_flag;
s_ctrl->sensor_probe_addr_type = probe_info->addr_type;
s_ctrl->sensor_probe_data_type = probe_info->data_type;
@@ -500,8 +548,8 @@ void cam_sensor_shutdown(struct cam_sensor_ctrl_t *s_ctrl)
&s_ctrl->sensordata->power_info;
int rc = 0;
- if ((s_ctrl->sensor_state == CAM_SENSOR_INIT) &&
- (s_ctrl->is_probe_succeed == 0))
+ s_ctrl->is_probe_succeed = 0;
+ if (s_ctrl->sensor_state == CAM_SENSOR_INIT)
return;
cam_sensor_release_resource(s_ctrl);
@@ -515,15 +563,12 @@ void cam_sensor_shutdown(struct cam_sensor_ctrl_t *s_ctrl)
s_ctrl->bridge_intf.device_hdl = -1;
s_ctrl->bridge_intf.link_hdl = -1;
s_ctrl->bridge_intf.session_hdl = -1;
+
kfree(power_info->power_setting);
kfree(power_info->power_down_setting);
- power_info->power_setting = NULL;
- power_info->power_down_setting = NULL;
- power_info->power_setting_size = 0;
- power_info->power_down_setting_size = 0;
+
s_ctrl->streamon_count = 0;
s_ctrl->streamoff_count = 0;
- s_ctrl->is_probe_succeed = 0;
s_ctrl->sensor_state = CAM_SENSOR_INIT;
}
@@ -562,21 +607,16 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
{
int rc = 0;
struct cam_control *cmd = (struct cam_control *)arg;
+ struct cam_sensor_power_setting *pu = NULL;
+ struct cam_sensor_power_setting *pd = NULL;
struct cam_sensor_power_ctrl_t *power_info =
&s_ctrl->sensordata->power_info;
+
if (!s_ctrl || !arg) {
CAM_ERR(CAM_SENSOR, "s_ctrl is NULL");
return -EINVAL;
}
- if (cmd->op_code != CAM_SENSOR_PROBE_CMD) {
- if (cmd->handle_type != CAM_HANDLE_USER_POINTER) {
- CAM_ERR(CAM_SENSOR, "Invalid handle type: %d",
- cmd->handle_type);
- return -EINVAL;
- }
- }
-
mutex_lock(&(s_ctrl->cam_sensor_mutex));
switch (cmd->op_code) {
case CAM_SENSOR_PROBE_CMD: {
@@ -585,18 +625,37 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
"Already Sensor Probed in the slot");
break;
}
+ /* Allocate memory for power up setting */
+ pu = kzalloc(sizeof(struct cam_sensor_power_setting) *
+ MAX_POWER_CONFIG, GFP_KERNEL);
+ if (!pu) {
+ rc = -ENOMEM;
+ goto release_mutex;
+ }
+
+ pd = kzalloc(sizeof(struct cam_sensor_power_setting) *
+ MAX_POWER_CONFIG, GFP_KERNEL);
+ if (!pd) {
+ kfree(pu);
+ rc = -ENOMEM;
+ goto release_mutex;
+ }
+
+ power_info->power_setting = pu;
+ power_info->power_down_setting = pd;
if (cmd->handle_type ==
CAM_HANDLE_MEM_HANDLE) {
rc = cam_handle_mem_ptr(cmd->handle, s_ctrl);
if (rc < 0) {
CAM_ERR(CAM_SENSOR, "Get Buffer Handle Failed");
+ kfree(pu);
+ kfree(pd);
goto release_mutex;
}
} else {
CAM_ERR(CAM_SENSOR, "Invalid Command Type: %d",
cmd->handle_type);
- return -EINVAL;
}
/* Parse and fill vreg params for powerup settings */
@@ -608,7 +667,9 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
CAM_ERR(CAM_SENSOR,
"Fail in filling vreg params for PUP rc %d",
rc);
- goto free_power_settings;
+ kfree(pu);
+ kfree(pd);
+ goto release_mutex;
}
/* Parse and fill vreg params for powerdown settings*/
@@ -620,14 +681,18 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
CAM_ERR(CAM_SENSOR,
"Fail in filling vreg params for PDOWN rc %d",
rc);
- goto free_power_settings;
+ kfree(pu);
+ kfree(pd);
+ goto release_mutex;
}
/* Power up and probe sensor */
rc = cam_sensor_power_up(s_ctrl);
if (rc < 0) {
CAM_ERR(CAM_SENSOR, "power up failed");
- goto free_power_settings;
+ kfree(pu);
+ kfree(pd);
+ goto release_mutex;
}
/* Match sensor ID */
@@ -635,7 +700,9 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
if (rc < 0) {
cam_sensor_power_down(s_ctrl);
msleep(20);
- goto free_power_settings;
+ kfree(pu);
+ kfree(pd);
+ goto release_mutex;
}
CAM_INFO(CAM_SENSOR,
@@ -644,10 +711,29 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
s_ctrl->sensordata->slave_info.sensor_slave_addr,
s_ctrl->sensordata->slave_info.sensor_id);
+#ifdef CONFIG_CAMERA_FW_UPDATE
+ if (s_ctrl->fw_update_flag & (1 << 0)) {
+ CAM_INFO(CAM_SENSOR, "[OISFW]Check OIS FW update");
+ rc = checkOISFWUpdate(s_ctrl);
+ }
+
+ if (s_ctrl->fw_update_flag & (1 << 1)) {
+ CAM_INFO(CAM_SENSOR, "[VCMFW]Check rear actuator FW");
+ rc = checkRearVCMFWUpdate(s_ctrl);
+ CAM_INFO(CAM_SENSOR, "[VCMFW]Check temperature params");
+ rc = checkRearVCMFWUpdate_temp(s_ctrl);
+ }
+
+ if (s_ctrl->fw_update_flag & (1 << 2))
+ rc = checkFrontVCMFWUpdate(s_ctrl);
+#endif
+
rc = cam_sensor_power_down(s_ctrl);
if (rc < 0) {
CAM_ERR(CAM_SENSOR, "fail in Sensor Power Down");
- goto free_power_settings;
+ kfree(pu);
+ kfree(pd);
+ goto release_mutex;
}
/*
* Set probe succeeded flag to 1 so that no other camera shall
@@ -661,15 +747,6 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
struct cam_sensor_acquire_dev sensor_acq_dev;
struct cam_create_dev_hdl bridge_params;
- if ((s_ctrl->is_probe_succeed == 0) ||
- (s_ctrl->sensor_state != CAM_SENSOR_INIT)) {
- CAM_WARN(CAM_SENSOR,
- "Not in right state to aquire %d",
- s_ctrl->sensor_state);
- rc = -EINVAL;
- goto release_mutex;
- }
-
if (s_ctrl->bridge_intf.device_hdl != -1) {
CAM_ERR(CAM_SENSOR, "Device is already acquired");
rc = -EINVAL;
@@ -710,9 +787,8 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
s_ctrl->sensor_state = CAM_SENSOR_ACQUIRE;
CAM_INFO(CAM_SENSOR,
- "CAM_ACQUIRE_DEV Success, sensor_id:0x%x,sensor_slave_addr:0x%x",
- s_ctrl->sensordata->slave_info.sensor_id,
- s_ctrl->sensordata->slave_info.sensor_slave_addr);
+ "CAM_ACQUIRE_DEV Success, sensor_id:0x%x",
+ s_ctrl->sensordata->slave_info.sensor_id);
}
break;
case CAM_RELEASE_DEV: {
@@ -751,9 +827,8 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
s_ctrl->sensor_state = CAM_SENSOR_INIT;
CAM_INFO(CAM_SENSOR,
- "CAM_RELEASE_DEV Success, sensor_id:0x%x,sensor_slave_addr:0x%x",
- s_ctrl->sensordata->slave_info.sensor_id,
- s_ctrl->sensordata->slave_info.sensor_slave_addr);
+ "CAM_RELEASE_DEV Success, sensor_id:0x%x",
+ s_ctrl->sensordata->slave_info.sensor_id);
s_ctrl->streamon_count = 0;
s_ctrl->streamoff_count = 0;
}
@@ -782,6 +857,10 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
if (s_ctrl->i2c_data.streamon_settings.is_settings_valid &&
(s_ctrl->i2c_data.streamon_settings.request_id == 0)) {
+#ifdef CONFIG_MNH_SM_HOST
+ if (s_ctrl->uses_easel)
+ cam_sensor_config_easel(s_ctrl);
+#endif
rc = cam_sensor_apply_settings(s_ctrl, 0,
CAM_SENSOR_PACKET_OPCODE_SENSOR_STREAMON);
if (rc < 0) {
@@ -792,9 +871,8 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
}
s_ctrl->sensor_state = CAM_SENSOR_START;
CAM_INFO(CAM_SENSOR,
- "CAM_START_DEV Success, sensor_id:0x%x,sensor_slave_addr:0x%x",
- s_ctrl->sensordata->slave_info.sensor_id,
- s_ctrl->sensordata->slave_info.sensor_slave_addr);
+ "CAM_START_DEV Success, sensor_id:0x%x",
+ s_ctrl->sensordata->slave_info.sensor_id);
}
break;
case CAM_STOP_DEV: {
@@ -819,9 +897,8 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
cam_sensor_release_resource(s_ctrl);
s_ctrl->sensor_state = CAM_SENSOR_ACQUIRE;
CAM_INFO(CAM_SENSOR,
- "CAM_STOP_DEV Success, sensor_id:0x%x,sensor_slave_addr:0x%x",
- s_ctrl->sensordata->slave_info.sensor_id,
- s_ctrl->sensordata->slave_info.sensor_slave_addr);
+ "CAM_STOP_DEV Success, sensor_id:0x%x",
+ s_ctrl->sensordata->slave_info.sensor_id);
}
break;
case CAM_CONFIG_DEV: {
@@ -878,16 +955,6 @@ int32_t cam_sensor_driver_cmd(struct cam_sensor_ctrl_t *s_ctrl,
release_mutex:
mutex_unlock(&(s_ctrl->cam_sensor_mutex));
return rc;
-
-free_power_settings:
- kfree(power_info->power_setting);
- kfree(power_info->power_down_setting);
- power_info->power_setting = NULL;
- power_info->power_down_setting = NULL;
- power_info->power_down_setting_size = 0;
- power_info->power_setting_size = 0;
- mutex_unlock(&(s_ctrl->cam_sensor_mutex));
- return rc;
}
int cam_sensor_publish_dev_info(struct cam_req_mgr_device_info *info)
@@ -943,6 +1010,65 @@ int cam_sensor_power(struct v4l2_subdev *sd, int on)
return 0;
}
+#ifdef CONFIG_MNH_SM_HOST
+int cam_sensor_config_easel(struct cam_sensor_ctrl_t *s_ctrl)
+{
+ int rc;
+ /*
+ * Camera ID 0 - FNCAM
+ * P1: RX1 -> TX0
+ */
+ struct mnh_mipi_config mipi_config0 = {
+ .txdev = MIPI_TX0,
+ .rxdev = MIPI_RX1,
+ .rx_rate = 696,
+ .tx_rate = 696,
+ .mode = MIPI_MODE_BYPASS,
+ .vc_en_mask = MNH_MIPI_VC_ALL_EN_MASK,
+ };
+ /*
+ * Camera ID 1 - RCAM
+ * P1: RX2 -> TX1
+ */
+ struct mnh_mipi_config mipi_config1 = {
+ .txdev = MIPI_TX1,
+ .rxdev = MIPI_RX2,
+ .rx_rate = 1368,
+ .tx_rate = 1368,
+ .mode = MIPI_MODE_BYPASS,
+ .vc_en_mask = MNH_MIPI_VC_ALL_EN_MASK,
+ };
+ /*
+ * Camera ID 2 - FWCAM
+ * P1: RX0 -> TX1
+ */
+ struct mnh_mipi_config mipi_config2 = {
+ .txdev = MIPI_TX1,
+ .rxdev = MIPI_RX0,
+ .rx_rate = 696,
+ .tx_rate = 696,
+ .mode = MIPI_MODE_BYPASS,
+ .vc_en_mask = MNH_MIPI_VC_ALL_EN_MASK,
+ };
+
+ if (!s_ctrl) {
+ CAM_ERR(CAM_SENSOR, "failed: %pK", s_ctrl);
+ return -EINVAL;
+ }
+
+ if (s_ctrl->id == 0)
+ rc = mnh_sm_mipi_config(mipi_config0);
+ else if (s_ctrl->id == 1)
+ rc = mnh_sm_mipi_config(mipi_config1);
+ else
+ rc = mnh_sm_mipi_config(mipi_config2);
+ if (rc)
+ CAM_ERR(CAM_SENSOR, "configure easel failed:%d", rc);
+
+ return rc;
+}
+#endif
+
int cam_sensor_power_up(struct cam_sensor_ctrl_t *s_ctrl)
{
int rc;
@@ -964,6 +1090,14 @@ int cam_sensor_power_up(struct cam_sensor_ctrl_t *s_ctrl)
return -EINVAL;
}
+#ifdef CONFIG_MNH_SM_HOST
+ if (s_ctrl->uses_easel) {
+ rc = mnh_sm_set_state(MNH_STATE_ACTIVE);
+ if (rc && (rc != -EHOSTUNREACH))
+ CAM_WARN(CAM_SENSOR, "power up easel failed:%d", rc);
+ }
+#endif
+
rc = cam_sensor_core_power_up(power_info, soc_info);
if (rc < 0) {
CAM_ERR(CAM_SENSOR, "power up the core is failed:%d", rc);
@@ -995,6 +1129,15 @@ int cam_sensor_power_down(struct cam_sensor_ctrl_t *s_ctrl)
CAM_ERR(CAM_SENSOR, "failed: power_info %pK", power_info);
return -EINVAL;
}
+
+#ifdef CONFIG_MNH_SM_HOST
+ if (s_ctrl->uses_easel) {
+ rc = mnh_sm_set_state(MNH_STATE_OFF);
+ if (rc)
+ CAM_WARN(CAM_SENSOR, "power down easel failed:%d", rc);
+ }
+#endif
+
rc = msm_camera_power_down(power_info, soc_info);
if (rc < 0) {
CAM_ERR(CAM_SENSOR, "power down the core is failed:%d", rc);
@@ -1102,9 +1245,9 @@ int cam_sensor_apply_settings(struct cam_sensor_ctrl_t *s_ctrl,
for (i = 0; i < MAX_PER_FRAME_ARRAY; i++) {
if ((del_req_id >
- s_ctrl->i2c_data.per_frame[i].request_id) && (
- s_ctrl->i2c_data.per_frame[i].is_settings_valid
- == 1)) {
+ s_ctrl->i2c_data.per_frame[i].request_id) &&
+ (s_ctrl->i2c_data.per_frame[i].
+ is_settings_valid == 1)) {
s_ctrl->i2c_data.per_frame[i].request_id = 0;
rc = delete_request(
&(s_ctrl->i2c_data.per_frame[i]));
@@ -1157,11 +1300,6 @@ int32_t cam_sensor_flush_request(struct cam_req_mgr_flush_request *flush_req)
return -EINVAL;
}
- if (s_ctrl->i2c_data.per_frame == NULL) {
- CAM_ERR(CAM_SENSOR, "i2c frame data is NULL");
- return -EINVAL;
- }
-
for (i = 0; i < MAX_PER_FRAME_ARRAY; i++) {
i2c_set = &(s_ctrl->i2c_data.per_frame[i]);
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_dev.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_dev.h
index cc6070c..fd28869 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_dev.h
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_dev.h
@@ -90,6 +90,7 @@ struct intf_params {
* @sensor_info: Sensor query cap structure
* @bridge_intf: Bridge interface structure
* @device_name: Sensor device structure
+ * @fw_update_flag: Update OIS firmware
* @streamon_count: Count to hold the number of times stream on called
* @streamoff_count: Count to hold the number of times stream off called
*/
@@ -111,6 +112,10 @@ struct cam_sensor_ctrl_t {
struct cam_sensor_query_cap sensor_info;
struct intf_params bridge_intf;
char device_name[20];
+ uint8_t fw_update_flag;
+#ifdef CONFIG_MNH_SM_HOST
+ bool uses_easel;
+#endif
uint32_t streamon_count;
uint32_t streamoff_count;
};
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_soc.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_soc.c
index 1c3ead0..5cb9720 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_soc.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor/cam_sensor_soc.c
@@ -172,6 +172,13 @@ static int32_t cam_sensor_driver_get_dt_data(struct cam_sensor_ctrl_t *s_ctrl)
sensordata->pos_yaw = 360;
}
+#ifdef CONFIG_MNH_SM_HOST
+ if (of_property_read_bool(of_node, "uses-easel"))
+ s_ctrl->uses_easel = true;
+ else
+ s_ctrl->uses_easel = false;
+#endif
+
return rc;
FREE_SENSOR_DATA:
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_cci_i2c.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_cci_i2c.c
index 1ec7d9a..b40b512 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_cci_i2c.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_cci_i2c.c
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -33,7 +33,6 @@ int32_t cam_cci_i2c_read(struct cam_sensor_cci_client *cci_client,
cci_ctrl.cci_info = cci_client;
cci_ctrl.cfg.cci_i2c_read_cfg.addr = addr;
cci_ctrl.cfg.cci_i2c_read_cfg.addr_type = addr_type;
- cci_ctrl.cfg.cci_i2c_read_cfg.data_type = data_type;
cci_ctrl.cfg.cci_i2c_read_cfg.data = buf;
cci_ctrl.cfg.cci_i2c_read_cfg.num_byte = data_type;
rc = v4l2_subdev_call(cci_client->cci_subdev,
@@ -60,7 +59,6 @@ int32_t cam_cci_i2c_read(struct cam_sensor_cci_client *cci_client,
int32_t cam_camera_cci_i2c_read_seq(struct cam_sensor_cci_client *cci_client,
uint32_t addr, uint8_t *data,
enum camera_sensor_i2c_type addr_type,
- enum camera_sensor_i2c_type data_type,
uint32_t num_byte)
{
int32_t rc = -EFAULT;
@@ -69,7 +67,6 @@ int32_t cam_camera_cci_i2c_read_seq(struct cam_sensor_cci_client *cci_client,
struct cam_cci_ctrl cci_ctrl;
if ((addr_type >= CAMERA_SENSOR_I2C_TYPE_MAX)
- || (data_type >= CAMERA_SENSOR_I2C_TYPE_MAX)
|| (num_byte > I2C_REG_DATA_MAX)) {
CAM_ERR(CAM_SENSOR, "addr_type %d num_byte %d", addr_type,
num_byte);
@@ -84,7 +81,6 @@ int32_t cam_camera_cci_i2c_read_seq(struct cam_sensor_cci_client *cci_client,
cci_ctrl.cci_info = cci_client;
cci_ctrl.cfg.cci_i2c_read_cfg.addr = addr;
cci_ctrl.cfg.cci_i2c_read_cfg.addr_type = addr_type;
- cci_ctrl.cfg.cci_i2c_read_cfg.data_type = data_type;
cci_ctrl.cfg.cci_i2c_read_cfg.data = buf;
cci_ctrl.cfg.cci_i2c_read_cfg.num_byte = num_byte;
cci_ctrl.status = -EFAULT;
@@ -167,7 +163,7 @@ int32_t cam_cci_i2c_write_continuous_table(
}
static int32_t cam_cci_i2c_compare(struct cam_sensor_cci_client *client,
- uint32_t addr, uint16_t data, uint16_t data_mask,
+ uint32_t addr, uint32_t data, uint32_t data_mask,
enum camera_sensor_i2c_type data_type,
enum camera_sensor_i2c_type addr_type)
{
@@ -179,14 +175,24 @@ static int32_t cam_cci_i2c_compare(struct cam_sensor_cci_client *client,
if (rc < 0)
return rc;
- reg_data = reg_data & 0xFFFF;
+ if (data_type == CAMERA_SENSOR_I2C_TYPE_BYTE)
+ reg_data = reg_data & 0xFF;
+ else if (data_type == CAMERA_SENSOR_I2C_TYPE_WORD)
+ reg_data = reg_data & 0xFFFF;
+ else if (data_type == CAMERA_SENSOR_I2C_TYPE_3B)
+ reg_data = reg_data & 0xFFFFFF;
+ else if (data_type == CAMERA_SENSOR_I2C_TYPE_DWORD)
+ reg_data = reg_data & 0xFFFFFFFF;
+ else
+ reg_data = reg_data & 0xFFFF;
+
if (data == (reg_data & ~data_mask))
return I2C_COMPARE_MATCH;
return I2C_COMPARE_MISMATCH;
}
int32_t cam_cci_i2c_poll(struct cam_sensor_cci_client *client,
- uint32_t addr, uint16_t data, uint16_t data_mask,
+ uint32_t addr, uint32_t data, uint32_t data_mask,
enum camera_sensor_i2c_type data_type,
enum camera_sensor_i2c_type addr_type,
uint32_t delay_ms)
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_i2c.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_i2c.h
index e79fffb..9c3c2e3 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_i2c.h
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_i2c.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -21,7 +21,7 @@
#include "cam_sensor_io.h"
#define I2C_POLL_TIME_MS 5
-#define MAX_POLL_DELAY_MS 100
+#define MAX_POLL_DELAY_MS 500
#define I2C_COMPARE_MATCH 0
#define I2C_COMPARE_MISMATCH 1
@@ -46,7 +46,6 @@ int32_t cam_cci_i2c_read(struct cam_sensor_cci_client *client,
* @addr: I2c address
* @data: I2C data
* @addr_type: I2c address type
- * @data_type: I2c data type
* @num_byte: number of bytes
*
* This API handles CCI sequential read
@@ -54,7 +53,6 @@ int32_t cam_cci_i2c_read(struct cam_sensor_cci_client *client,
int32_t cam_camera_cci_i2c_read_seq(struct cam_sensor_cci_client *client,
uint32_t addr, uint8_t *data,
enum camera_sensor_i2c_type addr_type,
- enum camera_sensor_i2c_type data_type,
uint32_t num_byte);
/**
@@ -100,7 +98,7 @@ int32_t cam_sensor_cci_i2c_util(struct cam_sensor_cci_client *cci_client,
* This API implements CCI based I2C poll
*/
int32_t cam_cci_i2c_poll(struct cam_sensor_cci_client *client,
- uint32_t addr, uint16_t data, uint16_t data_mask,
+ uint32_t addr, uint32_t data, uint32_t data_mask,
enum camera_sensor_i2c_type data_type,
enum camera_sensor_i2c_type addr_type,
uint32_t delay_ms);
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_io.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_io.c
index ed490fd5..9c44c05 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_io.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_io.c
@@ -14,12 +14,12 @@
#include "cam_sensor_i2c.h"
int32_t camera_io_dev_poll(struct camera_io_master *io_master_info,
- uint32_t addr, uint16_t data, uint32_t data_mask,
+ uint32_t addr, uint32_t data, uint32_t data_mask,
enum camera_sensor_i2c_type addr_type,
enum camera_sensor_i2c_type data_type,
uint32_t delay_ms)
{
- int16_t mask = data_mask & 0xFF;
+ uint32_t mask = data_mask & 0xFFFFFFFF;
if (!io_master_info) {
CAM_ERR(CAM_SENSOR, "Invalid Args");
@@ -69,12 +69,11 @@ int32_t camera_io_dev_read(struct camera_io_master *io_master_info,
int32_t camera_io_dev_read_seq(struct camera_io_master *io_master_info,
uint32_t addr, uint8_t *data,
- enum camera_sensor_i2c_type addr_type,
- enum camera_sensor_i2c_type data_type, int32_t num_bytes)
+ enum camera_sensor_i2c_type addr_type, int32_t num_bytes)
{
if (io_master_info->master_type == CCI_MASTER) {
return cam_camera_cci_i2c_read_seq(io_master_info->cci_client,
- addr, data, addr_type, data_type, num_bytes);
+ addr, data, addr_type, num_bytes);
} else if (io_master_info->master_type == I2C_MASTER) {
return cam_qup_i2c_read_seq(io_master_info->client,
addr, data, addr_type, num_bytes);
@@ -152,8 +151,32 @@ int32_t camera_io_dev_write_continuous(struct camera_io_master *io_master_info,
}
}
+static int read_rainbow_sensor_id(struct camera_io_master *io_master_info)
+{
+ int rc = 0;
+ uint16_t orig_slave_addr = io_master_info->cci_client->sid;
+ uint32_t data = 0;
+ uint32_t addr = 0x00;
+
+ io_master_info->cci_client->sid = 0x7f;
+
+ rc = camera_io_dev_read(io_master_info, addr,
+ &data, CAMERA_SENSOR_I2C_TYPE_BYTE,
+ CAMERA_SENSOR_I2C_TYPE_BYTE);
+
+ io_master_info->cci_client->sid = orig_slave_addr;
+
+ if (rc < 0)
+ pr_info("read_rainbow_sensor_id failed(non-fatal): %d\n", rc);
+ else
+ pr_info("read_rainbow_sensor_id success, id: 0x%x\n", data);
+
+ return rc;
+}
+
int32_t camera_io_init(struct camera_io_master *io_master_info)
{
+ int rc = 0;
if (!io_master_info) {
CAM_ERR(CAM_SENSOR, "Invalid Args");
return -EINVAL;
@@ -162,16 +185,20 @@ int32_t camera_io_init(struct camera_io_master *io_master_info)
if (io_master_info->master_type == CCI_MASTER) {
io_master_info->cci_client->cci_subdev =
cam_cci_get_subdev();
- return cam_sensor_cci_i2c_util(io_master_info->cci_client,
+ rc = cam_sensor_cci_i2c_util(io_master_info->cci_client,
MSM_CCI_INIT);
} else if ((io_master_info->master_type == I2C_MASTER) ||
(io_master_info->master_type == SPI_MASTER)) {
- return 0;
+ rc = 0;
} else {
CAM_ERR(CAM_SENSOR, "Invalid Comm. Master:%d",
io_master_info->master_type);
- return -EINVAL;
+ rc = -EINVAL;
}
+
+ read_rainbow_sensor_id(io_master_info);
+
+ return rc;
}
int32_t camera_io_release(struct camera_io_master *io_master_info)
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_io.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_io.h
index f1143c8..9207b73 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_io.h
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_io/cam_sensor_io.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -52,7 +52,6 @@ int32_t camera_io_dev_read(struct camera_io_master *io_master_info,
* @io_master_info: I2C/SPI master information
* @addr: I2C address
* @data: I2C data
- * @addr_type: I2C addr type
* @data_type: I2C data type
* @num_bytes: number of bytes
*
@@ -61,7 +60,6 @@ int32_t camera_io_dev_read(struct camera_io_master *io_master_info,
int32_t camera_io_dev_read_seq(struct camera_io_master *io_master_info,
uint32_t addr, uint8_t *data,
enum camera_sensor_i2c_type addr_type,
- enum camera_sensor_i2c_type data_type,
int32_t num_bytes);
/**
@@ -111,7 +109,7 @@ int32_t camera_io_dev_write_continuous(struct camera_io_master *io_master_info,
* This API abstracts poll functionality based on master type
*/
int32_t camera_io_dev_poll(struct camera_io_master *io_master_info,
- uint32_t addr, uint16_t data, uint32_t data_mask,
+ uint32_t addr, uint32_t data, uint32_t data_mask,
enum camera_sensor_i2c_type addr_type,
enum camera_sensor_i2c_type data_type,
uint32_t delay_ms);
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/Makefile b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/Makefile
index 9f0843d5..98ee3ae 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/Makefile
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/Makefile
@@ -4,6 +4,5 @@
ccflags-y += -Idrivers/media/platform/msm/camera/cam_sensor_module/cam_cci
ccflags-y += -Idrivers/media/platform/msm/camera/cam_sensor_module/cam_res_mgr
ccflags-y += -Idrivers/media/platform/msm/camera/cam_smmu/
-ccflags-y += -Idrivers/media/platform/msm/camera/cam_cpas/include
obj-$(CONFIG_SPECTRA_CAMERA) += cam_sensor_util.o
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_cmn_header.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_cmn_header.h
index fddff38..358abb8 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_cmn_header.h
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_cmn_header.h
@@ -165,7 +165,8 @@ enum cam_sensor_packet_opcodes {
enum cam_actuator_packet_opcodes {
CAM_ACTUATOR_PACKET_OPCODE_INIT,
CAM_ACTUATOR_PACKET_AUTO_MOVE_LENS,
- CAM_ACTUATOR_PACKET_MANUAL_MOVE_LENS
+ CAM_ACTUATOR_PACKET_MANUAL_MOVE_LENS,
+ CAM_ACTUATOR_PACKET_REG_READ
};
enum cam_eeprom_packet_opcodes {
@@ -174,7 +175,12 @@ enum cam_eeprom_packet_opcodes {
enum cam_ois_packet_opcodes {
CAM_OIS_PACKET_OPCODE_INIT,
- CAM_OIS_PACKET_OPCODE_OIS_CONTROL
+ CAM_OIS_PACKET_OPCODE_OIS_CONTROL,
+ CAM_OIS_PACKET_OPCODE_SHIFT_READER_START,
+ CAM_OIS_PACKET_OPCODE_SHIFT_READER_STOP,
+ CAM_OIS_PACKET_OPCODE_SHIFT_GET,
+ CAM_OIS_PACKET_OPCODE_READ,
+ CAM_OIS_PACKET_OPCODE_CALIBRATION
};
enum msm_bus_perf_setting {
@@ -269,6 +275,7 @@ struct cam_sensor_i2c_reg_setting {
enum camera_sensor_i2c_type addr_type;
enum camera_sensor_i2c_type data_type;
unsigned short delay;
+ unsigned short slave_addr;
};
struct i2c_settings_list {
@@ -319,9 +326,6 @@ enum msm_sensor_camera_id_t {
CAMERA_1,
CAMERA_2,
CAMERA_3,
- CAMERA_4,
- CAMERA_5,
- CAMERA_6,
MAX_CAMERAS,
};
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_util.c b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_util.c
index a1cc69c..c7bd291 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_util.c
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_util.c
@@ -93,10 +93,12 @@ int32_t cam_sensor_handle_delay(
list_entry(list_ptr, struct i2c_settings_list, list);
if (generic_op_code ==
CAMERA_SENSOR_WAIT_OP_HW_UCND)
- i2c_list->i2c_settings.reg_setting[offset - 1].delay =
+ i2c_list->i2c_settings.
+ reg_setting[offset - 1].delay =
cmd_uncond_wait->delay;
else
- i2c_list->i2c_settings.delay = cmd_uncond_wait->delay;
+ i2c_list->i2c_settings.delay =
+ cmd_uncond_wait->delay;
(*cmd_buf) +=
sizeof(
struct cam_cmd_unconditional_wait) / sizeof(uint32_t);
@@ -176,14 +178,20 @@ int32_t cam_sensor_handle_random_write(
cam_cmd_i2c_random_wr->header.addr_type;
i2c_list->i2c_settings.data_type =
cam_cmd_i2c_random_wr->header.data_type;
+ i2c_list->i2c_settings.slave_addr =
+ cam_cmd_i2c_random_wr->header.slave_addr;
for (cnt = 0; cnt < (cam_cmd_i2c_random_wr->header.count);
cnt++) {
i2c_list->i2c_settings.reg_setting[cnt].reg_addr =
- cam_cmd_i2c_random_wr->random_wr_payload[cnt].reg_addr;
- i2c_list->i2c_settings.reg_setting[cnt].reg_data =
- cam_cmd_i2c_random_wr->random_wr_payload[cnt].reg_data;
- i2c_list->i2c_settings.reg_setting[cnt].data_mask = 0;
+ cam_cmd_i2c_random_wr->
+ random_wr_payload[cnt].reg_addr;
+ i2c_list->i2c_settings.
+ reg_setting[cnt].reg_data =
+ cam_cmd_i2c_random_wr->
+ random_wr_payload[cnt].reg_data;
+ i2c_list->i2c_settings.
+ reg_setting[cnt].data_mask = 0;
}
*offset = cnt;
*list = &(i2c_list->list);
@@ -231,10 +239,14 @@ static int32_t cam_sensor_handle_continuous_write(
for (cnt = 0; cnt < (cam_cmd_i2c_continuous_wr->header.count);
cnt++) {
i2c_list->i2c_settings.reg_setting[cnt].reg_addr =
- cam_cmd_i2c_continuous_wr->reg_addr;
- i2c_list->i2c_settings.reg_setting[cnt].reg_data =
- cam_cmd_i2c_continuous_wr->data_read[cnt].reg_data;
- i2c_list->i2c_settings.reg_setting[cnt].data_mask = 0;
+ cam_cmd_i2c_continuous_wr->
+ reg_addr;
+ i2c_list->i2c_settings.
+ reg_setting[cnt].reg_data =
+ cam_cmd_i2c_continuous_wr->
+ data_read[cnt].reg_data;
+ i2c_list->i2c_settings.
+ reg_setting[cnt].data_mask = 0;
}
*offset = cnt;
*list = &(i2c_list->list);
@@ -242,37 +254,37 @@ static int32_t cam_sensor_handle_continuous_write(
return rc;
}
-static int cam_sensor_handle_slave_info(
- struct camera_io_master *io_master,
- uint32_t *cmd_buf)
+int32_t cam_sensor_handle_read(
+ struct cam_cmd_i2c_continuous_rd *cam_cmd,
+ struct i2c_settings_array *i2c_reg_settings,
+ uint32_t *byte_cnt, int32_t *offset,
+ struct list_head **list)
{
- int rc = 0;
- struct cam_cmd_i2c_info *i2c_info = (struct cam_cmd_i2c_info *)cmd_buf;
+ struct i2c_settings_list *i2c_list;
+ int32_t rc = 0;
- if (io_master == NULL || cmd_buf == NULL) {
- CAM_ERR(CAM_SENSOR, "Invalid args");
- return -EINVAL;
+ i2c_list = cam_sensor_get_i2c_ptr(i2c_reg_settings,
+ cam_cmd->header.count);
+ if (i2c_list == NULL ||
+ i2c_list->i2c_settings.reg_setting == NULL) {
+ CAM_ERR(CAM_SENSOR, "Failed in allocating i2c_list");
+ return -ENOMEM;
}
- switch (io_master->master_type) {
- case CCI_MASTER:
- io_master->cci_client->sid = (i2c_info->slave_addr >> 1);
- io_master->cci_client->i2c_freq_mode = i2c_info->i2c_freq_mode;
- break;
+ *byte_cnt = sizeof(struct cam_cmd_i2c_continuous_rd);
+ i2c_list->op_code = CAM_SENSOR_I2C_READ;
+ i2c_list->i2c_settings.addr_type =
+ cam_cmd->header.addr_type;
+ i2c_list->i2c_settings.data_type =
+ cam_cmd->header.data_type;
+ i2c_list->i2c_settings.size =
+ cam_cmd->header.count;
- case I2C_MASTER:
- io_master->client->addr = i2c_info->slave_addr;
- break;
+ i2c_list->i2c_settings.reg_setting[0].reg_addr =
+ cam_cmd->reg_addr;
- case SPI_MASTER:
- break;
-
- default:
- CAM_ERR(CAM_SENSOR, "Invalid master type: %d",
- io_master->master_type);
- rc = -EINVAL;
- break;
- }
+ (*offset) += 1;
+ *list = &(i2c_list->list);
return rc;
}
@@ -288,16 +300,12 @@ static int cam_sensor_handle_slave_info(
* WAIT + n x RND_WR with num_cmd_buf = 1. Do not exepect RD/WR
* with different cmd_type and op_code in one command buffer.
*/
-int cam_sensor_i2c_command_parser(
- struct camera_io_master *io_master,
- struct i2c_settings_array *i2c_reg_settings,
- struct cam_cmd_buf_desc *cmd_desc,
- int32_t num_cmd_buffers)
+int cam_sensor_i2c_command_parser(struct i2c_settings_array *i2c_reg_settings,
+ struct cam_cmd_buf_desc *cmd_desc, int32_t num_cmd_buffers)
{
int16_t rc = 0, i = 0;
size_t len_of_buff = 0;
uint64_t generic_ptr;
- uint16_t cmd_length_in_bytes = 0;
for (i = 0; i < num_cmd_buffers; i++) {
uint32_t *cmd_buf = NULL;
@@ -311,6 +319,7 @@ int cam_sensor_i2c_command_parser(
* It is not expected the same settings to
* be spread across multiple cmd buffers
*/
+
CAM_DBG(CAM_SENSOR, "Total cmd Buf in Bytes: %d",
cmd_desc[i].length);
@@ -411,22 +420,19 @@ int cam_sensor_i2c_command_parser(
}
break;
}
- case CAMERA_SENSOR_CMD_TYPE_I2C_INFO: {
- rc = cam_sensor_handle_slave_info(
- io_master, cmd_buf);
- if (rc) {
+ case CAMERA_SENSOR_CMD_TYPE_I2C_CONT_RD:
+ rc = cam_sensor_handle_read(
+ (struct cam_cmd_i2c_continuous_rd *)
+ cmd_buf,
+ i2c_reg_settings,
+ &byte_cnt, &j, &list);
+ if (rc < 0) {
CAM_ERR(CAM_SENSOR,
- "Handle slave info failed with rc: %d",
+ "read hdl failed: %d",
rc);
return rc;
}
- cmd_length_in_bytes =
- sizeof(struct cam_cmd_i2c_info);
- cmd_buf +=
- cmd_length_in_bytes / sizeof(uint32_t);
- byte_cnt += cmd_length_in_bytes;
- break;
- }
+ break;
default:
CAM_ERR(CAM_SENSOR, "Invalid Command Type:%d",
cmm_hdr->cmd_type);
@@ -686,7 +692,6 @@ int32_t cam_sensor_update_power_settings(void *cmd_buf,
{
int32_t rc = 0, tot_size = 0, last_cmd_type = 0;
int32_t i = 0, pwr_up = 0, pwr_down = 0;
- struct cam_sensor_power_setting *pwr_settings;
void *ptr = cmd_buf, *scr;
struct cam_cmd_power *pwr_cmd = (struct cam_cmd_power *)cmd_buf;
struct common_header *cmm_hdr = (struct common_header *)cmd_buf;
@@ -698,6 +703,7 @@ int32_t cam_sensor_update_power_settings(void *cmd_buf,
}
power_info->power_setting_size = 0;
+ kfree(power_info->power_setting);
power_info->power_setting =
(struct cam_sensor_power_setting *)
kzalloc(sizeof(struct cam_sensor_power_setting) *
@@ -706,15 +712,14 @@ int32_t cam_sensor_update_power_settings(void *cmd_buf,
return -ENOMEM;
power_info->power_down_setting_size = 0;
+ kfree(power_info->power_down_setting);
power_info->power_down_setting =
(struct cam_sensor_power_setting *)
kzalloc(sizeof(struct cam_sensor_power_setting) *
MAX_POWER_CONFIG, GFP_KERNEL);
if (!power_info->power_down_setting) {
- kfree(power_info->power_setting);
- power_info->power_setting = NULL;
- power_info->power_setting_size = 0;
- return -ENOMEM;
+ rc = -ENOMEM;
+ goto free_power_settings;
}
while (tot_size < cmd_length) {
@@ -724,24 +729,21 @@ int32_t cam_sensor_update_power_settings(void *cmd_buf,
(struct cam_cmd_power *)ptr;
power_info->power_setting_size += pwr_cmd->count;
- if (power_info->power_setting_size > MAX_POWER_CONFIG) {
- CAM_ERR(CAM_SENSOR,
- "Invalid: power up setting size %d",
- power_info->power_setting_size);
- rc = -EINVAL;
- goto free_power_settings;
- }
scr = ptr + sizeof(struct cam_cmd_power);
tot_size = tot_size + sizeof(struct cam_cmd_power);
if (pwr_cmd->count == 0)
- CAM_WARN(CAM_SENSOR, "pwr_up_size is zero");
+ CAM_WARN(CAM_SENSOR, "Un expected Command");
for (i = 0; i < pwr_cmd->count; i++, pwr_up++) {
- power_info->power_setting[pwr_up].seq_type =
- pwr_cmd->power_settings[i].power_seq_type;
- power_info->power_setting[pwr_up].config_val =
- pwr_cmd->power_settings[i].config_val_low;
+ power_info->
+ power_setting[pwr_up].seq_type =
+ pwr_cmd->power_settings[i].
+ power_seq_type;
+ power_info->
+ power_setting[pwr_up].config_val =
+ pwr_cmd->power_settings[i].
+ config_val_low;
power_info->power_setting[pwr_up].delay = 0;
if (i) {
scr = scr +
@@ -755,12 +757,16 @@ int32_t cam_sensor_update_power_settings(void *cmd_buf,
CAM_ERR(CAM_SENSOR,
"Error: Cmd Buffer is wrong");
rc = -EINVAL;
- goto free_power_settings;
+ goto free_power_down_settings;
}
CAM_DBG(CAM_SENSOR,
- "Seq Type[%d]: %d Config_val: %ld", pwr_up,
- power_info->power_setting[pwr_up].seq_type,
- power_info->power_setting[pwr_up].config_val);
+ "Seq Type[%d]: %d Config_val: %ld",
+ pwr_up,
+ power_info->
+ power_setting[pwr_up].seq_type,
+ power_info->
+ power_setting[pwr_up].
+ config_val);
}
last_cmd_type = CAMERA_SENSOR_CMD_TYPE_PWR_UP;
ptr = (void *) scr;
@@ -768,38 +774,32 @@ int32_t cam_sensor_update_power_settings(void *cmd_buf,
} else if (cmm_hdr->cmd_type == CAMERA_SENSOR_CMD_TYPE_WAIT) {
struct cam_cmd_unconditional_wait *wait_cmd =
(struct cam_cmd_unconditional_wait *)ptr;
- if ((wait_cmd->op_code ==
- CAMERA_SENSOR_WAIT_OP_SW_UCND) &&
- (last_cmd_type ==
- CAMERA_SENSOR_CMD_TYPE_PWR_UP)) {
- if (pwr_up > 0) {
- pwr_settings =
- &power_info->power_setting[pwr_up - 1];
- pwr_settings->delay +=
- wait_cmd->delay;
- } else {
- CAM_ERR(CAM_SENSOR,
- "Delay is expected only after valid power up setting");
+ if (wait_cmd->op_code ==
+ CAMERA_SENSOR_WAIT_OP_SW_UCND) {
+ if (last_cmd_type ==
+ CAMERA_SENSOR_CMD_TYPE_PWR_UP) {
+ if (pwr_up > 0)
+ power_info->
+ power_setting
+ [pwr_up - 1].delay +=
+ wait_cmd->delay;
+ else
+ CAM_ERR(CAM_SENSOR,
+ "Delay is expected only after valid power up setting");
+ } else if (last_cmd_type ==
+ CAMERA_SENSOR_CMD_TYPE_PWR_DOWN) {
+ if (pwr_down > 0)
+ power_info->
+ power_down_setting
+ [pwr_down - 1].delay +=
+ wait_cmd->delay;
+ else
+ CAM_ERR(CAM_SENSOR,
+ "Delay is expected only after valid power up setting");
}
- } else if ((wait_cmd->op_code ==
- CAMERA_SENSOR_WAIT_OP_SW_UCND) &&
- (last_cmd_type ==
- CAMERA_SENSOR_CMD_TYPE_PWR_DOWN)) {
- if (pwr_down > 0) {
- pwr_settings =
- &power_info->power_down_setting[
- pwr_down - 1];
- pwr_settings->delay +=
- wait_cmd->delay;
- } else {
- CAM_ERR(CAM_SENSOR,
- "Delay is expected only after valid power up setting");
- }
- } else {
+ } else
CAM_DBG(CAM_SENSOR, "Invalid op code: %d",
wait_cmd->op_code);
- }
-
tot_size = tot_size +
sizeof(struct cam_cmd_unconditional_wait);
if (tot_size > cmd_length) {
@@ -823,27 +823,23 @@ int32_t cam_sensor_update_power_settings(void *cmd_buf,
scr = ptr + sizeof(struct cam_cmd_power);
tot_size = tot_size + sizeof(struct cam_cmd_power);
power_info->power_down_setting_size += pwr_cmd->count;
- if (power_info->power_down_setting_size >
- MAX_POWER_CONFIG) {
- CAM_ERR(CAM_SENSOR,
- "Invalid: power down setting size %d",
- power_info->power_down_setting_size);
- rc = -EINVAL;
- goto free_power_settings;
- }
if (pwr_cmd->count == 0)
- CAM_ERR(CAM_SENSOR, "pwr_down size is zero");
+ CAM_ERR(CAM_SENSOR, "Invalid Command");
for (i = 0; i < pwr_cmd->count; i++, pwr_down++) {
- pwr_settings =
- &power_info->power_down_setting[pwr_down];
- pwr_settings->seq_type =
- pwr_cmd->power_settings[i].power_seq_type;
- pwr_settings->config_val =
- pwr_cmd->power_settings[i].config_val_low;
- power_info->power_down_setting[pwr_down].delay
- = 0;
+ power_info->
+ power_down_setting[pwr_down].
+ seq_type =
+ pwr_cmd->power_settings[i].
+ power_seq_type;
+ power_info->
+ power_down_setting[pwr_down].
+ config_val =
+ pwr_cmd->power_settings[i].
+ config_val_low;
+ power_info->
+ power_down_setting[pwr_down].delay = 0;
if (i) {
scr = scr +
sizeof(
@@ -857,12 +853,17 @@ int32_t cam_sensor_update_power_settings(void *cmd_buf,
CAM_ERR(CAM_SENSOR,
"Command Buffer is wrong");
rc = -EINVAL;
- goto free_power_settings;
+ goto free_power_down_settings;
}
CAM_DBG(CAM_SENSOR,
"Seq Type[%d]: %d Config_val: %ld",
- pwr_down, pwr_settings->seq_type,
- pwr_settings->config_val);
+ pwr_down,
+ power_info->
+ power_down_setting[pwr_down].
+ seq_type,
+ power_info->
+ power_down_setting[pwr_down].
+ config_val);
}
last_cmd_type = CAMERA_SENSOR_CMD_TYPE_PWR_DOWN;
ptr = (void *) scr;
@@ -871,19 +872,16 @@ int32_t cam_sensor_update_power_settings(void *cmd_buf,
CAM_ERR(CAM_SENSOR,
"Error: Un expected Header Type: %d",
cmm_hdr->cmd_type);
- rc = -EINVAL;
- goto free_power_settings;
}
}
return rc;
-free_power_settings:
+free_power_down_settings:
kfree(power_info->power_down_setting);
- kfree(power_info->power_setting);
power_info->power_down_setting = NULL;
+free_power_settings:
+ kfree(power_info->power_setting);
power_info->power_setting = NULL;
- power_info->power_down_setting_size = 0;
- power_info->power_setting_size = 0;
return rc;
}
@@ -912,6 +910,9 @@ int cam_get_dt_power_setting_data(struct device_node *of_node,
ps = kcalloc(count, sizeof(*ps), GFP_KERNEL);
if (!ps)
return -ENOMEM;
+
+ kfree(power_info->power_setting);
+
power_info->power_setting = ps;
for (i = 0; i < count; i++) {
@@ -969,6 +970,7 @@ int cam_get_dt_power_setting_data(struct device_node *of_node,
}
kfree(array);
+ kfree(power_info->power_down_setting);
power_info->power_down_setting =
kzalloc(sizeof(*ps) * count, GFP_KERNEL);
@@ -1263,62 +1265,10 @@ int msm_cam_sensor_handle_reg_gpio(int seq_type,
return 0;
}
-static int cam_config_mclk_reg(struct cam_sensor_power_ctrl_t *ctrl,
- struct cam_hw_soc_info *soc_info, int32_t index)
-{
- int32_t num_vreg = 0, j = 0, rc = 0, idx = 0;
- struct cam_sensor_power_setting *ps = NULL;
- struct cam_sensor_power_setting *pd = NULL;
-
- num_vreg = soc_info->num_rgltr;
-
- pd = &ctrl->power_down_setting[index];
-
- for (j = 0; j < num_vreg; j++) {
- if (!strcmp(soc_info->rgltr_name[j], "cam_clk")) {
- ps = NULL;
- for (idx = 0; idx < ctrl->power_setting_size; idx++) {
- if (ctrl->power_setting[idx].seq_type ==
- pd->seq_type) {
- ps = &ctrl->power_setting[idx];
- break;
- }
- }
-
- if (ps != NULL) {
- CAM_DBG(CAM_SENSOR, "Disable MCLK Regulator");
- rc = cam_soc_util_regulator_disable(
- soc_info->rgltr[j],
- soc_info->rgltr_name[j],
- soc_info->rgltr_min_volt[j],
- soc_info->rgltr_max_volt[j],
- soc_info->rgltr_op_mode[j],
- soc_info->rgltr_delay[j]);
-
- if (rc) {
- CAM_ERR(CAM_SENSOR,
- "MCLK REG DISALBE FAILED: %d",
- rc);
- return rc;
- }
-
- ps->data[0] =
- soc_info->rgltr[j];
-
- regulator_put(
- soc_info->rgltr[j]);
- soc_info->rgltr[j] = NULL;
- }
- }
- }
-
- return rc;
-}
-
int cam_sensor_core_power_up(struct cam_sensor_power_ctrl_t *ctrl,
struct cam_hw_soc_info *soc_info)
{
- int rc = 0, index = 0, no_gpio = 0, ret = 0, num_vreg, j = 0, i = 0;
+ int rc = 0, index = 0, no_gpio = 0, ret = 0, num_vreg, j = 0;
int32_t vreg_idx = -1;
struct cam_sensor_power_setting *power_setting = NULL;
struct msm_camera_gpio_num_info *gpio_num_info = NULL;
@@ -1377,13 +1327,6 @@ int cam_sensor_core_power_up(struct cam_sensor_power_ctrl_t *ctrl,
for (index = 0; index < ctrl->power_setting_size; index++) {
CAM_DBG(CAM_SENSOR, "index: %d", index);
power_setting = &ctrl->power_setting[index];
- if (!power_setting) {
- CAM_ERR(CAM_SENSOR,
- "Invalid power up settings for index %d",
- index);
- return -EINVAL;
- }
-
CAM_DBG(CAM_SENSOR, "seq_type %d", power_setting->seq_type);
switch (power_setting->seq_type) {
@@ -1415,7 +1358,6 @@ int cam_sensor_core_power_up(struct cam_sensor_power_ctrl_t *ctrl,
soc_info->rgltr_name[j],
rc);
soc_info->rgltr[j] = NULL;
- goto power_up_failed;
}
rc = cam_soc_util_regulator_enable(
@@ -1425,11 +1367,7 @@ int cam_sensor_core_power_up(struct cam_sensor_power_ctrl_t *ctrl,
soc_info->rgltr_max_volt[j],
soc_info->rgltr_op_mode[j],
soc_info->rgltr_delay[j]);
- if (rc) {
- CAM_ERR(CAM_SENSOR,
- "Reg enable failed");
- goto power_up_failed;
- }
+
power_setting->data[0] =
soc_info->rgltr[j];
}
@@ -1510,7 +1448,6 @@ int cam_sensor_core_power_up(struct cam_sensor_power_ctrl_t *ctrl,
rc);
soc_info->rgltr[vreg_idx] = NULL;
- goto power_up_failed;
}
rc = cam_soc_util_regulator_enable(
@@ -1520,12 +1457,7 @@ int cam_sensor_core_power_up(struct cam_sensor_power_ctrl_t *ctrl,
soc_info->rgltr_max_volt[vreg_idx],
soc_info->rgltr_op_mode[vreg_idx],
soc_info->rgltr_delay[vreg_idx]);
- if (rc) {
- CAM_ERR(CAM_SENSOR,
- "Reg Enable failed for %s",
- soc_info->rgltr_name[vreg_idx]);
- goto power_up_failed;
- }
+
power_setting->data[0] =
soc_info->rgltr[vreg_idx];
}
@@ -1568,18 +1500,6 @@ int cam_sensor_core_power_up(struct cam_sensor_power_ctrl_t *ctrl,
CAM_DBG(CAM_SENSOR, "type %d",
power_setting->seq_type);
switch (power_setting->seq_type) {
- case SENSOR_MCLK:
- for (i = soc_info->num_clk - 1; i >= 0; i--) {
- cam_soc_util_clk_disable(soc_info->clk[i],
- soc_info->clk_name[i]);
- }
- ret = cam_config_mclk_reg(ctrl, soc_info, index);
- if (ret < 0) {
- CAM_ERR(CAM_SENSOR,
- "config clk reg failed rc: %d", ret);
- continue;
- }
- break;
case SENSOR_RESET:
case SENSOR_STANDBY:
case SENSOR_CUSTOM_GPIO1:
@@ -1612,21 +1532,11 @@ int cam_sensor_core_power_up(struct cam_sensor_power_ctrl_t *ctrl,
soc_info->rgltr_op_mode[vreg_idx],
soc_info->rgltr_delay[vreg_idx]);
- if (rc) {
- CAM_ERR(CAM_SENSOR,
- "Fail to disalbe reg: %s",
- soc_info->rgltr_name[vreg_idx]);
- soc_info->rgltr[vreg_idx] = NULL;
- msm_cam_sensor_handle_reg_gpio(
- power_setting->seq_type,
- gpio_num_info,
- GPIOF_OUT_INIT_LOW);
- continue;
- }
power_setting->data[0] =
soc_info->rgltr[vreg_idx];
- regulator_put(soc_info->rgltr[vreg_idx]);
+ regulator_put(
+ soc_info->rgltr[vreg_idx]);
soc_info->rgltr[vreg_idx] = NULL;
}
else
@@ -1652,8 +1562,8 @@ int cam_sensor_core_power_up(struct cam_sensor_power_ctrl_t *ctrl,
if (ctrl->cam_pinctrl_status) {
ret = pinctrl_select_state(
- ctrl->pinctrl_info.pinctrl,
- ctrl->pinctrl_info.gpio_state_suspend);
+ ctrl->pinctrl_info.pinctrl,
+ ctrl->pinctrl_info.gpio_state_suspend);
if (ret)
CAM_ERR(CAM_SENSOR, "cannot set pin to suspend state");
cam_res_mgr_shared_pinctrl_select_state(false);
@@ -1692,12 +1602,60 @@ msm_camera_get_power_settings(struct cam_sensor_power_ctrl_t *ctrl,
return ps;
}
+static int cam_config_mclk_reg(struct cam_sensor_power_ctrl_t *ctrl,
+ struct cam_hw_soc_info *soc_info, int32_t index)
+{
+ int32_t num_vreg = 0, j = 0, rc = 0, idx = 0;
+ struct cam_sensor_power_setting *ps = NULL;
+ struct cam_sensor_power_setting *pd = NULL;
+
+ num_vreg = soc_info->num_rgltr;
+
+ pd = &ctrl->power_down_setting[index];
+
+ for (j = 0; j < num_vreg; j++) {
+ if (!strcmp(soc_info->rgltr_name[j], "cam_clk")) {
+
+ ps = NULL;
+ for (idx = 0; idx <
+ ctrl->power_setting_size; idx++) {
+ if (ctrl->power_setting[idx].
+ seq_type == pd->seq_type) {
+ ps = &ctrl->power_setting[idx];
+ break;
+ }
+ }
+
+ if (ps != NULL) {
+ CAM_DBG(CAM_SENSOR, "Disable Regulator");
+
+ rc = cam_soc_util_regulator_disable(
+ soc_info->rgltr[j],
+ soc_info->rgltr_name[j],
+ soc_info->rgltr_min_volt[j],
+ soc_info->rgltr_max_volt[j],
+ soc_info->rgltr_op_mode[j],
+ soc_info->rgltr_delay[j]);
+
+ ps->data[0] =
+ soc_info->rgltr[j];
+
+ regulator_put(
+ soc_info->rgltr[j]);
+ soc_info->rgltr[j] = NULL;
+ }
+ }
+ }
+
+ return rc;
+}
+
int msm_camera_power_down(struct cam_sensor_power_ctrl_t *ctrl,
struct cam_hw_soc_info *soc_info)
{
int index = 0, ret = 0, num_vreg = 0, i;
struct cam_sensor_power_setting *pd = NULL;
- struct cam_sensor_power_setting *ps = NULL;
+ struct cam_sensor_power_setting *ps;
struct msm_camera_gpio_num_info *gpio_num_info = NULL;
CAM_DBG(CAM_SENSOR, "Enter");
@@ -1714,37 +1672,25 @@ int msm_camera_power_down(struct cam_sensor_power_ctrl_t *ctrl,
return -EINVAL;
}
- if (ctrl->power_down_setting_size > MAX_POWER_CONFIG) {
- CAM_ERR(CAM_SENSOR, "Invalid: power setting size %d",
- ctrl->power_setting_size);
- return -EINVAL;
- }
-
for (index = 0; index < ctrl->power_down_setting_size; index++) {
- CAM_DBG(CAM_SENSOR, "power_down_index %d", index);
+ CAM_DBG(CAM_SENSOR, "index %d", index);
pd = &ctrl->power_down_setting[index];
- if (!pd) {
- CAM_ERR(CAM_SENSOR,
- "Invalid power down settings for index %d",
- index);
- return -EINVAL;
- }
-
ps = NULL;
- CAM_DBG(CAM_SENSOR, "seq_type %d", pd->seq_type);
+ CAM_DBG(CAM_SENSOR, "type %d", pd->seq_type);
switch (pd->seq_type) {
case SENSOR_MCLK:
+ ret = cam_config_mclk_reg(ctrl, soc_info, index);
+ if (ret < 0) {
+ CAM_ERR(CAM_SENSOR,
+ "config clk reg failed rc: %d", ret);
+ return ret;
+ }
+ //cam_soc_util_clk_disable_default(soc_info);
for (i = soc_info->num_clk - 1; i >= 0; i--) {
cam_soc_util_clk_disable(soc_info->clk[i],
soc_info->clk_name[i]);
}
- ret = cam_config_mclk_reg(ctrl, soc_info, index);
- if (ret < 0) {
- CAM_ERR(CAM_SENSOR,
- "config clk reg failed rc: %d", ret);
- continue;
- }
break;
case SENSOR_RESET:
case SENSOR_STANDBY:
@@ -1784,19 +1730,7 @@ int msm_camera_power_down(struct cam_sensor_power_ctrl_t *ctrl,
soc_info->rgltr_max_volt[ps->seq_val],
soc_info->rgltr_op_mode[ps->seq_val],
soc_info->rgltr_delay[ps->seq_val]);
- if (ret) {
- CAM_ERR(CAM_SENSOR,
- "Reg: %s disable failed",
- soc_info->rgltr_name[
- ps->seq_val]);
- soc_info->rgltr[ps->seq_val] =
- NULL;
- msm_cam_sensor_handle_reg_gpio(
- pd->seq_type,
- gpio_num_info,
- GPIOF_OUT_INIT_LOW);
- continue;
- }
+
ps->data[0] =
soc_info->rgltr[ps->seq_val];
diff --git a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_util.h b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_util.h
index c9ccc5c..5d45514 100644
--- a/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_util.h
+++ b/drivers/media/platform/msm/camera/cam_sensor_module/cam_sensor_utils/cam_sensor_util.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -18,13 +18,12 @@
#include <linux/gpio.h>
#include <linux/of_gpio.h>
#include <linux/of.h>
-#include <cam_sensor_cmn_header.h>
-#include <cam_req_mgr_util.h>
+#include "cam_sensor_cmn_header.h"
+#include "cam_req_mgr_util.h"
#include <cam_req_mgr_interface.h>
#include <cam_mem_mgr.h>
#include "cam_soc_util.h"
#include "cam_debug_util.h"
-#include "cam_sensor_io.h"
#define INVALID_VREG 100
@@ -35,8 +34,7 @@ int cam_get_dt_power_setting_data(struct device_node *of_node,
int msm_camera_pinctrl_init
(struct msm_pinctrl_info *sensor_pctrl, struct device *dev);
-int cam_sensor_i2c_command_parser(struct camera_io_master *io_master,
- struct i2c_settings_array *i2c_reg_settings,
+int cam_sensor_i2c_command_parser(struct i2c_settings_array *i2c_reg_settings,
struct cam_cmd_buf_desc *cmd_desc, int32_t num_cmd_buffers);
int32_t delete_request(struct i2c_settings_array *i2c_array);
diff --git a/drivers/media/platform/msm/camera/cam_smmu/cam_smmu_api.c b/drivers/media/platform/msm/camera/cam_smmu/cam_smmu_api.c
index ecfc566..c33ec10 100644
--- a/drivers/media/platform/msm/camera/cam_smmu/cam_smmu_api.c
+++ b/drivers/media/platform/msm/camera/cam_smmu/cam_smmu_api.c
@@ -602,7 +602,7 @@ static int cam_smmu_create_add_handle_in_table(char *name,
if (!strcmp(iommu_cb_set.cb_info[i].name, name)) {
mutex_lock(&iommu_cb_set.cb_info[i].lock);
if (iommu_cb_set.cb_info[i].handle != HANDLE_INIT) {
- CAM_ERR(CAM_SMMU,
+ CAM_WARN(CAM_SMMU,
"Error: %s already got handle 0x%x",
name,
iommu_cb_set.cb_info[i].handle);
@@ -2924,7 +2924,7 @@ int cam_smmu_destroy_handle(int handle)
cam_smmu_clean_kernel_buffer_list(idx);
}
- if (&iommu_cb_set.cb_info[idx].is_secure) {
+ if (iommu_cb_set.cb_info[idx].is_secure) {
if (iommu_cb_set.cb_info[idx].secure_count == 0) {
mutex_unlock(&iommu_cb_set.cb_info[idx].lock);
return -EPERM;
diff --git a/drivers/media/platform/msm/camera/cam_sync/cam_sync.c b/drivers/media/platform/msm/camera/cam_sync/cam_sync.c
index 517b7df..12fe85f 100644
--- a/drivers/media/platform/msm/camera/cam_sync/cam_sync.c
+++ b/drivers/media/platform/msm/camera/cam_sync/cam_sync.c
@@ -36,7 +36,8 @@ int cam_sync_create(int32_t *sync_obj, const char *name)
} while (bit);
spin_lock_bh(&sync_dev->row_spinlocks[idx]);
- rc = cam_sync_init_object(sync_dev->sync_table, idx, name);
+ rc = cam_sync_init_row(sync_dev->sync_table, idx, name,
+ CAM_SYNC_TYPE_INDV);
if (rc) {
CAM_ERR(CAM_SYNC, "Error: Unable to init row at idx = %ld",
idx);
@@ -123,7 +124,6 @@ int cam_sync_deregister_callback(sync_callback cb_func,
{
struct sync_table_row *row = NULL;
struct sync_callback_info *sync_cb, *temp;
- bool found = false;
if (sync_obj >= CAM_SYNC_MAX_OBJS || sync_obj <= 0)
return -EINVAL;
@@ -146,12 +146,11 @@ int cam_sync_deregister_callback(sync_callback cb_func,
sync_cb->cb_data == userdata) {
list_del_init(&sync_cb->list);
kfree(sync_cb);
- found = true;
}
}
spin_unlock_bh(&sync_dev->row_spinlocks[sync_obj]);
- return found ? 0 : -ENOENT;
+ return 0;
}
int cam_sync_signal(int32_t sync_obj, uint32_t status)
@@ -165,6 +164,7 @@ int cam_sync_signal(int32_t sync_obj, uint32_t status)
struct list_head sync_list;
struct cam_signalable_info *list_info = NULL;
struct cam_signalable_info *temp_list_info = NULL;
+ struct list_head parents_list;
/* Objects to be signaled will be added into this list */
INIT_LIST_HEAD(&sync_list);
@@ -216,20 +216,36 @@ int cam_sync_signal(int32_t sync_obj, uint32_t status)
return rc;
}
+ /* copy parent list to local and release child lock */
+ INIT_LIST_HEAD(&parents_list);
+ list_splice_init(&row->parents_list, &parents_list);
+ spin_unlock_bh(&sync_dev->row_spinlocks[sync_obj]);
+
+ if (list_empty(&parents_list))
+ goto dispatch_cb;
+
/*
* Now iterate over all parents of this object and if they too need to
* be signaled add them to the list
*/
list_for_each_entry(parent_info,
- &row->parents_list,
+ &parents_list,
list) {
parent_row = sync_dev->sync_table + parent_info->sync_id;
spin_lock_bh(&sync_dev->row_spinlocks[parent_info->sync_id]);
parent_row->remaining--;
- parent_row->state = cam_sync_util_get_state(
- parent_row->state,
+ rc = cam_sync_util_update_parent_state(
+ parent_row,
status);
+ if (rc) {
+ CAM_ERR(CAM_SYNC, "Invalid parent state %d",
+ parent_row->state);
+ spin_unlock_bh(
+ &sync_dev->row_spinlocks[parent_info->sync_id]);
+ kfree(parent_info);
+ continue;
+ }
if (!parent_row->remaining) {
rc = cam_sync_util_add_to_signalable_list
@@ -240,15 +256,13 @@ int cam_sync_signal(int32_t sync_obj, uint32_t status)
spin_unlock_bh(
&sync_dev->row_spinlocks[
parent_info->sync_id]);
- spin_unlock_bh(
- &sync_dev->row_spinlocks[sync_obj]);
- return rc;
+ continue;
}
}
spin_unlock_bh(&sync_dev->row_spinlocks[parent_info->sync_id]);
}
- spin_unlock_bh(&sync_dev->row_spinlocks[sync_obj]);
+dispatch_cb:
/*
* Now dispatch the various sync objects collected so far, in our
@@ -333,10 +347,8 @@ int cam_sync_merge(int32_t *sync_obj, uint32_t num_objs, int32_t *merged_obj)
return -EINVAL;
}
- rc = cam_sync_util_validate_merge(sync_obj,
- num_objs);
- if (rc < 0) {
- CAM_ERR(CAM_SYNC, "Validation failed, Merge not allowed");
+ if (num_objs <= 1) {
+ CAM_ERR(CAM_SYNC, "Single object merge is not allowed");
return -EINVAL;
}
@@ -348,7 +360,6 @@ int cam_sync_merge(int32_t *sync_obj, uint32_t num_objs, int32_t *merged_obj)
} while (bit);
spin_lock_bh(&sync_dev->row_spinlocks[idx]);
-
rc = cam_sync_init_group_object(sync_dev->sync_table,
idx, sync_obj,
num_objs);
diff --git a/drivers/media/platform/msm/camera/cam_sync/cam_sync_util.c b/drivers/media/platform/msm/camera/cam_sync/cam_sync_util.c
index ed69829..29f3807 100644
--- a/drivers/media/platform/msm/camera/cam_sync/cam_sync_util.c
+++ b/drivers/media/platform/msm/camera/cam_sync/cam_sync_util.c
@@ -31,20 +31,21 @@ int cam_sync_util_find_and_set_empty_row(struct sync_device *sync_dev,
return rc;
}
-int cam_sync_init_object(struct sync_table_row *table,
- uint32_t idx,
- const char *name)
+int cam_sync_init_row(struct sync_table_row *table,
+ uint32_t idx, const char *name, uint32_t type)
{
struct sync_table_row *row = table + idx;
if (!table || idx <= 0 || idx >= CAM_SYNC_MAX_OBJS)
return -EINVAL;
+ memset(row, 0, sizeof(*row));
+
if (name)
strlcpy(row->name, name, SYNC_DEBUG_NAME_LEN);
INIT_LIST_HEAD(&row->parents_list);
INIT_LIST_HEAD(&row->children_list);
- row->type = CAM_SYNC_TYPE_INDV;
+ row->type = type;
row->sync_id = idx;
row->state = CAM_SYNC_STATE_ACTIVE;
row->remaining = 0;
@@ -57,143 +58,95 @@ int cam_sync_init_object(struct sync_table_row *table,
return 0;
}
-uint32_t cam_sync_util_get_group_object_state(struct sync_table_row *table,
- uint32_t *sync_objs,
- uint32_t num_objs)
-{
- int i;
- struct sync_table_row *child_row = NULL;
- int success_count = 0;
- int active_count = 0;
-
- if (!table || !sync_objs)
- return CAM_SYNC_STATE_SIGNALED_ERROR;
-
- /*
- * We need to arrive at the state of the merged object based on
- * counts of error, active and success states of all children objects
- */
- for (i = 0; i < num_objs; i++) {
- child_row = table + sync_objs[i];
- switch (child_row->state) {
- case CAM_SYNC_STATE_SIGNALED_ERROR:
- return CAM_SYNC_STATE_SIGNALED_ERROR;
- case CAM_SYNC_STATE_SIGNALED_SUCCESS:
- success_count++;
- break;
- case CAM_SYNC_STATE_ACTIVE:
- active_count++;
- break;
- default:
- CAM_ERR(CAM_SYNC,
- "Invalid state of child object during merge");
- return CAM_SYNC_STATE_SIGNALED_ERROR;
- }
- }
-
- if (active_count)
- return CAM_SYNC_STATE_ACTIVE;
-
- if (success_count == num_objs)
- return CAM_SYNC_STATE_SIGNALED_SUCCESS;
-
- return CAM_SYNC_STATE_SIGNALED_ERROR;
-}
-
-static int cam_sync_util_get_group_object_remaining_count(
- struct sync_table_row *table,
- uint32_t *sync_objs,
- uint32_t num_objs)
-{
- int i;
- struct sync_table_row *child_row = NULL;
- int remaining_count = 0;
-
- if (!table || !sync_objs)
- return -EINVAL;
-
- for (i = 0; i < num_objs; i++) {
- child_row = table + sync_objs[i];
- if (child_row->state == CAM_SYNC_STATE_ACTIVE)
- remaining_count++;
- }
-
- return remaining_count;
-}
-
int cam_sync_init_group_object(struct sync_table_row *table,
uint32_t idx,
uint32_t *sync_objs,
uint32_t num_objs)
{
- int i;
- int remaining;
+ int i, rc = 0;
struct sync_child_info *child_info;
struct sync_parent_info *parent_info;
struct sync_table_row *row = table + idx;
struct sync_table_row *child_row = NULL;
- INIT_LIST_HEAD(&row->parents_list);
-
- INIT_LIST_HEAD(&row->children_list);
+ cam_sync_init_row(table, idx, "merged_fence", CAM_SYNC_TYPE_GROUP);
/*
- * While traversing parents and children, we allocate in a loop and in
- * case allocation fails, we call the clean up function which frees up
- * all memory allocation thus far
+ * While traversing for children, parent's row list is updated with
+ * child info and each child's row is updated with parent info.
+ * If any child state is ERROR or SUCCESS, it will not be added to list.
*/
for (i = 0; i < num_objs; i++) {
- child_info = kzalloc(sizeof(*child_info), GFP_ATOMIC);
-
- if (!child_info) {
- cam_sync_util_cleanup_children_list(row,
- SYNC_LIST_CLEAN_ALL, 0);
- return -ENOMEM;
- }
-
- child_info->sync_id = sync_objs[i];
- list_add_tail(&child_info->list, &row->children_list);
- }
-
- for (i = 0; i < num_objs; i++) {
- /* This gets us the row corresponding to the sync object */
child_row = table + sync_objs[i];
spin_lock_bh(&sync_dev->row_spinlocks[sync_objs[i]]);
+
+ /* validate child */
+ if ((child_row->type == CAM_SYNC_TYPE_GROUP) ||
+ (child_row->state == CAM_SYNC_STATE_INVALID)) {
+ spin_unlock_bh(&sync_dev->row_spinlocks[sync_objs[i]]);
+ CAM_ERR(CAM_SYNC,
+ "Invalid child fence:%i state:%u type:%u",
+ child_row->sync_id, child_row->state,
+ child_row->type);
+ rc = -EINVAL;
+ goto clean_children_info;
+ }
+
+ /* check for child's state */
+ if (child_row->state == CAM_SYNC_STATE_SIGNALED_ERROR) {
+ row->state = CAM_SYNC_STATE_SIGNALED_ERROR;
+ spin_unlock_bh(&sync_dev->row_spinlocks[sync_objs[i]]);
+ continue;
+ }
+ if (child_row->state != CAM_SYNC_STATE_ACTIVE) {
+ spin_unlock_bh(&sync_dev->row_spinlocks[sync_objs[i]]);
+ continue;
+ }
+
+ row->remaining++;
+
+ /* Add child info */
+ child_info = kzalloc(sizeof(*child_info), GFP_ATOMIC);
+ if (!child_info) {
+ spin_unlock_bh(&sync_dev->row_spinlocks[sync_objs[i]]);
+ rc = -ENOMEM;
+ goto clean_children_info;
+ }
+ child_info->sync_id = sync_objs[i];
+ list_add_tail(&child_info->list, &row->children_list);
+
+ /* Add parent info */
parent_info = kzalloc(sizeof(*parent_info), GFP_ATOMIC);
if (!parent_info) {
- cam_sync_util_cleanup_parents_list(child_row,
- SYNC_LIST_CLEAN_ALL, 0);
- cam_sync_util_cleanup_children_list(row,
- SYNC_LIST_CLEAN_ALL, 0);
spin_unlock_bh(&sync_dev->row_spinlocks[sync_objs[i]]);
- return -ENOMEM;
+ rc = -ENOMEM;
+ goto clean_children_info;
}
parent_info->sync_id = idx;
list_add_tail(&parent_info->list, &child_row->parents_list);
spin_unlock_bh(&sync_dev->row_spinlocks[sync_objs[i]]);
}
- row->type = CAM_SYNC_TYPE_GROUP;
- row->sync_id = idx;
- row->state = cam_sync_util_get_group_object_state(table,
- sync_objs, num_objs);
- remaining = cam_sync_util_get_group_object_remaining_count(table,
- sync_objs, num_objs);
- if (remaining < 0) {
- CAM_ERR(CAM_SYNC, "Failed getting remaining count");
- return -ENODEV;
+ if (!row->remaining) {
+ if (row->state != CAM_SYNC_STATE_SIGNALED_ERROR)
+ row->state = CAM_SYNC_STATE_SIGNALED_SUCCESS;
+ complete_all(&row->signaled);
}
- row->remaining = remaining;
-
- init_completion(&row->signaled);
- INIT_LIST_HEAD(&row->callback_list);
- INIT_LIST_HEAD(&row->user_payload_list);
-
- if (row->state != CAM_SYNC_STATE_ACTIVE)
- complete_all(&row->signaled);
-
return 0;
+
+clean_children_info:
+ row->state = CAM_SYNC_STATE_INVALID;
+ for (i = i-1; i >= 0; i--) {
+ spin_lock_bh(&sync_dev->row_spinlocks[sync_objs[i]]);
+ child_row = table + sync_objs[i];
+ cam_sync_util_cleanup_parents_list(child_row,
+ SYNC_LIST_CLEAN_ONE, idx);
+ spin_unlock_bh(&sync_dev->row_spinlocks[sync_objs[i]]);
+ }
+
+ cam_sync_util_cleanup_children_list(row, SYNC_LIST_CLEAN_ALL, 0);
+ return rc;
}
int cam_sync_deinit_object(struct sync_table_row *table, uint32_t idx)
@@ -356,32 +309,6 @@ void cam_sync_util_send_v4l2_event(uint32_t id,
sync_obj);
}
-int cam_sync_util_validate_merge(uint32_t *sync_obj, uint32_t num_objs)
-{
- int i;
- struct sync_table_row *row = NULL;
-
- if (num_objs <= 1) {
- CAM_ERR(CAM_SYNC, "Single object merge is not allowed");
- return -EINVAL;
- }
-
- for (i = 0; i < num_objs; i++) {
- row = sync_dev->sync_table + sync_obj[i];
- spin_lock_bh(&sync_dev->row_spinlocks[sync_obj[i]]);
- if (row->type == CAM_SYNC_TYPE_GROUP ||
- row->state == CAM_SYNC_STATE_INVALID) {
- CAM_ERR(CAM_SYNC,
- "Group obj %d can't be merged or obj UNINIT",
- sync_obj[i]);
- spin_unlock_bh(&sync_dev->row_spinlocks[sync_obj[i]]);
- return -EINVAL;
- }
- spin_unlock_bh(&sync_dev->row_spinlocks[sync_obj[i]]);
- }
- return 0;
-}
-
int cam_sync_util_add_to_signalable_list(int32_t sync_obj,
uint32_t status,
struct list_head *sync_list)
@@ -402,34 +329,27 @@ int cam_sync_util_add_to_signalable_list(int32_t sync_obj,
return 0;
}
-int cam_sync_util_get_state(int current_state,
+int cam_sync_util_update_parent_state(struct sync_table_row *parent_row,
int new_state)
{
- int result = CAM_SYNC_STATE_SIGNALED_ERROR;
+ int rc = 0;
- if (new_state != CAM_SYNC_STATE_SIGNALED_SUCCESS &&
- new_state != CAM_SYNC_STATE_SIGNALED_ERROR)
- return CAM_SYNC_STATE_SIGNALED_ERROR;
-
- switch (current_state) {
- case CAM_SYNC_STATE_INVALID:
- result = CAM_SYNC_STATE_SIGNALED_ERROR;
- break;
-
+ switch (parent_row->state) {
case CAM_SYNC_STATE_ACTIVE:
case CAM_SYNC_STATE_SIGNALED_SUCCESS:
- if (new_state == CAM_SYNC_STATE_SIGNALED_ERROR)
- result = CAM_SYNC_STATE_SIGNALED_ERROR;
- else if (new_state == CAM_SYNC_STATE_SIGNALED_SUCCESS)
- result = CAM_SYNC_STATE_SIGNALED_SUCCESS;
+ parent_row->state = new_state;
break;
case CAM_SYNC_STATE_SIGNALED_ERROR:
- result = CAM_SYNC_STATE_SIGNALED_ERROR;
+ break;
+
+ case CAM_SYNC_STATE_INVALID:
+ default:
+ rc = -EINVAL;
break;
}
- return result;
+ return rc;
}
void cam_sync_util_cleanup_children_list(struct sync_table_row *row,
diff --git a/drivers/media/platform/msm/camera/cam_sync/cam_sync_util.h b/drivers/media/platform/msm/camera/cam_sync/cam_sync_util.h
index ae7d542..d386166 100644
--- a/drivers/media/platform/msm/camera/cam_sync/cam_sync_util.h
+++ b/drivers/media/platform/msm/camera/cam_sync/cam_sync_util.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -14,7 +14,7 @@
#define __CAM_SYNC_UTIL_H__
-#include <cam_sync_api.h>
+#include "cam_sync_api.h"
#include "cam_sync_private.h"
#include "cam_debug_util.h"
@@ -41,12 +41,11 @@ int cam_sync_util_find_and_set_empty_row(struct sync_device *sync_dev,
* @param idx : Index of row to initialize
* @param name : Optional string representation of the sync object. Should be
* 63 characters or less
- *
+ * @param type : type of row to be initialized
* @return Status of operation. Negative in case of error. Zero otherwise.
*/
-int cam_sync_init_object(struct sync_table_row *table,
- uint32_t idx,
- const char *name);
+int cam_sync_init_row(struct sync_table_row *table,
+ uint32_t idx, const char *name, uint32_t type);
/**
* @brief: Function to uninitialize a row in the sync table
@@ -104,16 +103,6 @@ void cam_sync_util_send_v4l2_event(uint32_t id,
int len);
/**
- * @brief: Function to validate sync merge arguments
- *
- * @param sync_obj : Array of sync objects to merge
- * @param num_objs : Number of sync objects in the array
- *
- * @return Status of operation. Negative in case of error. Zero otherwise.
- */
-int cam_sync_util_validate_merge(uint32_t *sync_obj, uint32_t num_objs);
-
-/**
* @brief: Function which adds sync object information to the signalable list
*
* @param sync_obj : Sync object to add
@@ -135,7 +124,7 @@ int cam_sync_util_add_to_signalable_list(int32_t sync_obj,
*
* @return Next state of the sync object
*/
-int cam_sync_util_get_state(int current_state,
+int cam_sync_util_update_parent_state(struct sync_table_row *parent_row,
int new_state);
/**
diff --git a/drivers/media/platform/msm/camera/cam_utils/cam_soc_util.c b/drivers/media/platform/msm/camera/cam_utils/cam_soc_util.c
index d0a13ab..a071670 100644
--- a/drivers/media/platform/msm/camera/cam_utils/cam_soc_util.c
+++ b/drivers/media/platform/msm/camera/cam_utils/cam_soc_util.c
@@ -306,7 +306,7 @@ int cam_soc_util_get_option_clk_by_name(struct cam_hw_soc_info *soc_info,
*clk = cam_soc_util_option_clk_get(of_node, index);
if (IS_ERR(*clk)) {
- CAM_ERR(CAM_UTIL, "No clk named %s found. Dev %s", clk_name,
+ CAM_WARN(CAM_UTIL, "No clk named %s found. Dev %s", clk_name,
soc_info->dev_name);
*clk_index = -1;
return -EFAULT;
diff --git a/drivers/media/platform/msm/camera/cam_utils/cam_soc_util.h b/drivers/media/platform/msm/camera/cam_utils/cam_soc_util.h
index 18fad8d..1f2d46dd 100644
--- a/drivers/media/platform/msm/camera/cam_utils/cam_soc_util.h
+++ b/drivers/media/platform/msm/camera/cam_utils/cam_soc_util.h
@@ -42,7 +42,6 @@
/* soc id */
#define SDM670_SOC_ID 336
-#define SDM710_SOC_ID 360
/* Minor Version */
#define SDM670_V1_1 0x1
diff --git a/drivers/media/platform/msm/camera/cam_utils/cam_trace.h b/drivers/media/platform/msm/camera/cam_utils/cam_trace.h
index 90ec566..3bd1d84 100644
--- a/drivers/media/platform/msm/camera/cam_utils/cam_trace.h
+++ b/drivers/media/platform/msm/camera/cam_utils/cam_trace.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2017, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2017-2018, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -17,15 +17,15 @@
#undef TRACE_SYSTEM
#define TRACE_SYSTEM camera
#undef TRACE_INCLUDE_PATH
-#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_PATH ../../drivers/media/platform/msm/camera/cam_utils
#undef TRACE_INCLUDE_FILE
#define TRACE_INCLUDE_FILE cam_trace
#include <linux/tracepoint.h>
#include <media/cam_req_mgr.h>
-#include "cam_req_mgr_core.h"
-#include "cam_req_mgr_interface.h"
-#include "cam_context.h"
+#include "../cam_core/cam_context.h"
+#include "../cam_req_mgr/cam_req_mgr_core.h"
+#include "../cam_req_mgr/cam_req_mgr_interface.h"
TRACE_EVENT(cam_context_state,
TP_PROTO(const char *name, struct cam_context *ctx),
@@ -72,17 +72,19 @@ TRACE_EVENT(cam_isp_activated_irq,
);
TRACE_EVENT(cam_icp_fw_dbg,
- TP_PROTO(char *dbg_message),
- TP_ARGS(dbg_message),
+ TP_PROTO(char *dbg_message, uint64_t timestamp),
+ TP_ARGS(dbg_message, timestamp),
TP_STRUCT__entry(
__string(dbg_message, dbg_message)
+ __field(uint64_t, timestamp)
),
TP_fast_assign(
__assign_str(dbg_message, dbg_message);
+ __entry->timestamp = timestamp;
),
TP_printk(
- "%s: ",
- __get_str(dbg_message)
+ "%llu %s: ",
+ __entry->timestamp, __get_str(dbg_message)
)
);
diff --git a/drivers/media/platform/msm/camera_v2/common/cam_smmu_api.c b/drivers/media/platform/msm/camera_v2/common/cam_smmu_api.c
index 1d74309..08c8575 100644
--- a/drivers/media/platform/msm/camera_v2/common/cam_smmu_api.c
+++ b/drivers/media/platform/msm/camera_v2/common/cam_smmu_api.c
@@ -270,6 +270,53 @@ static void cam_smmu_print_table(void)
}
}
+
+static int cam_smmu_query_vaddr_in_range(int handle,
+ unsigned long fault_addr, unsigned long *start_addr,
+ unsigned long *end_addr, int *fd)
+{
+ int idx, rc = -EINVAL;
+ struct cam_dma_buff_info *mapping;
+ unsigned long sa, ea;
+
+ if (!start_addr || !end_addr || !fd) {
+ pr_err("Invalid params\n");
+ return -EINVAL;
+ }
+
+ idx = GET_SMMU_TABLE_IDX(handle);
+ if (handle == HANDLE_INIT || idx < 0 || idx >= iommu_cb_set.cb_num) {
+ pr_err("Error: handle or index invalid. idx = %d hdl = %x\n",
+ idx, handle);
+ return -EINVAL;
+ }
+
+ mutex_lock(&iommu_cb_set.cb_info[idx].lock);
+ if (iommu_cb_set.cb_info[idx].handle != handle) {
+ pr_err("Error: hdl is not valid, table_hdl = %x, hdl = %x\n",
+ iommu_cb_set.cb_info[idx].handle, handle);
+ mutex_unlock(&iommu_cb_set.cb_info[idx].lock);
+ return -EINVAL;
+ }
+
+ list_for_each_entry(mapping,
+ &iommu_cb_set.cb_info[idx].smmu_buf_list, list) {
+ sa = (unsigned long)mapping->paddr;
+ ea = (unsigned long)mapping->paddr + mapping->len;
+
+ if (sa <= fault_addr && fault_addr < ea) {
+ *start_addr = sa;
+ *end_addr = ea;
+ *fd = mapping->ion_fd;
+ rc = 0;
+ break;
+ }
+ }
+ mutex_unlock(&iommu_cb_set.cb_info[idx].lock);
+ return rc;
+}
+EXPORT_SYMBOL(cam_smmu_query_vaddr_in_range);
+
static void cam_smmu_check_vaddr_in_range(int idx, void *vaddr)
{
struct cam_dma_buff_info *mapping;
diff --git a/drivers/media/platform/msm/camera_v2/fd/msm_fd_dev.c b/drivers/media/platform/msm/camera_v2/fd/msm_fd_dev.c
index 6f2cfcf..eb5bdee 100644
--- a/drivers/media/platform/msm/camera_v2/fd/msm_fd_dev.c
+++ b/drivers/media/platform/msm/camera_v2/fd/msm_fd_dev.c
@@ -444,7 +444,7 @@ static int msm_fd_open(struct file *file)
}
ctx->mem_pool.fd_device = ctx->fd_device;
- ctx->stats = vzalloc(sizeof(*ctx->stats) * MSM_FD_MAX_RESULT_BUFS);
+ ctx->stats = vmalloc(sizeof(*ctx->stats) * MSM_FD_MAX_RESULT_BUFS);
if (!ctx->stats) {
dev_err(device->dev, "No memory for face statistics\n");
ret = -ENOMEM;
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_buf_mgr.c b/drivers/media/platform/msm/camera_v2/isp/msm_buf_mgr.c
index 691b492..fda45cb 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_buf_mgr.c
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_buf_mgr.c
@@ -26,7 +26,6 @@
#include <media/v4l2-ioctl.h>
#include <media/v4l2-device.h>
#include <media/videobuf2-core.h>
-#include <media/msmb_generic_buf_mgr.h>
#include "msm.h"
#include "msm_buf_mgr.h"
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_isp.h b/drivers/media/platform/msm/camera_v2/isp/msm_isp.h
index f560e83..56cce54 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_isp.h
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_isp.h
@@ -153,11 +153,9 @@ struct msm_vfe_irq_ops {
struct msm_isp_timestamp *ts);
void (*process_axi_irq)(struct vfe_device *vfe_dev,
uint32_t irq_status0, uint32_t irq_status1,
- uint32_t pingpong_status,
struct msm_isp_timestamp *ts);
void (*process_stats_irq)(struct vfe_device *vfe_dev,
uint32_t irq_status0, uint32_t irq_status1,
- uint32_t pingpong_status,
struct msm_isp_timestamp *ts);
void (*config_irq)(struct vfe_device *vfe_dev,
uint32_t irq_status0, uint32_t irq_status1,
@@ -596,7 +594,6 @@ struct msm_vfe_tasklet_queue_cmd {
struct list_head list;
uint32_t vfeInterruptStatus0;
uint32_t vfeInterruptStatus1;
- uint32_t vfe_pingpong_status;
struct msm_isp_timestamp ts;
uint8_t cmd_used;
struct vfe_device *vfe_dev;
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_isp40.c b/drivers/media/platform/msm/camera_v2/isp/msm_isp40.c
index 19b1aac..269d5633 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_isp40.c
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_isp40.c
@@ -35,13 +35,12 @@
#define VFE40_STATS_BURST_LEN_8916_VERSION 2
#define VFE40_FETCH_BURST_LEN 3
#define VFE40_UB_SIZE 1536 /* 1536 * 128 bits = 24KB */
-#define VFE40_STATS_SIZE 392
#define VFE40_UB_SIZE_8952 2048 /* 2048 * 128 bits = 32KB */
#define VFE40_UB_SIZE_8916 3072 /* 3072 * 128 bits = 48KB */
#define VFE40_EQUAL_SLICE_UB 190 /* (UB_SIZE - STATS SIZE)/6 */
#define VFE40_EQUAL_SLICE_UB_8916 236
#define VFE40_TOTAL_WM_UB 1144 /* UB_SIZE - STATS SIZE */
-#define VFE40_TOTAL_WM_UB_8916 2680
+#define VFE40_TOTAL_WM_UB_8916 1656
#define VFE40_WM_BASE(idx) (0x6C + 0x24 * idx)
#define VFE40_RDI_BASE(idx) (0x2E8 + 0x4 * idx)
#define VFE40_XBAR_BASE(idx) (0x58 + 0x4 * (idx / 2))
@@ -105,11 +104,7 @@ static uint32_t msm_vfe40_ub_reg_offset(struct vfe_device *vfe_dev, int idx)
static uint32_t msm_vfe40_get_ub_size(struct vfe_device *vfe_dev)
{
- if (vfe_dev->vfe_hw_version == VFE40_8916_VERSION ||
- vfe_dev->vfe_hw_version == VFE40_8939_VERSION ||
- vfe_dev->vfe_hw_version == VFE40_8937_VERSION ||
- vfe_dev->vfe_hw_version == VFE40_8953_VERSION ||
- vfe_dev->vfe_hw_version == VFE40_8917_VERSION) {
+ if (vfe_dev->vfe_hw_version == VFE40_8916_VERSION) {
vfe_dev->ub_info->wm_ub = VFE40_TOTAL_WM_UB_8916;
return VFE40_TOTAL_WM_UB_8916;
}
@@ -585,11 +580,6 @@ static void msm_vfe40_read_and_clear_irq_status(struct vfe_device *vfe_dev,
*irq_status0 &= vfe_dev->irq0_mask;
*irq_status1 &= vfe_dev->irq1_mask;
- if (*irq_status0 &&
- (*irq_status0 == msm_camera_io_r(vfe_dev->vfe_base + 0x38))) {
- msm_camera_io_w(*irq_status0, vfe_dev->vfe_base + 0x30);
- msm_camera_io_w_mb(1, vfe_dev->vfe_base + 0x24);
- }
if (*irq_status1 & (1 << 0)) {
vfe_dev->error_info.camif_status =
@@ -1058,18 +1048,15 @@ static int msm_vfe40_start_fetch_engine(struct vfe_device *vfe_dev,
fe_cfg->stream_id);
vfe_dev->fetch_engine_info.bufq_handle = bufq_handle;
- mutex_lock(&vfe_dev->buf_mgr->lock);
rc = vfe_dev->buf_mgr->ops->get_buf_by_index(
vfe_dev->buf_mgr, bufq_handle, fe_cfg->buf_idx, &buf);
if (rc < 0 || !buf) {
pr_err("%s: No fetch buffer rc= %d buf= %pK\n",
__func__, rc, buf);
- mutex_unlock(&vfe_dev->buf_mgr->lock);
return -EINVAL;
}
mapped_info = buf->mapped_info[0];
buf->state = MSM_ISP_BUFFER_STATE_DISPATCHED;
- mutex_unlock(&vfe_dev->buf_mgr->lock);
} else {
rc = vfe_dev->buf_mgr->ops->map_buf(vfe_dev->buf_mgr,
&mapped_info, fe_cfg->fd);
@@ -1122,15 +1109,14 @@ static int msm_vfe40_start_fetch_engine_multi_pass(struct vfe_device *vfe_dev,
mutex_lock(&vfe_dev->buf_mgr->lock);
rc = vfe_dev->buf_mgr->ops->get_buf_by_index(
vfe_dev->buf_mgr, bufq_handle, fe_cfg->buf_idx, &buf);
+ mutex_unlock(&vfe_dev->buf_mgr->lock);
if (rc < 0 || !buf) {
pr_err("%s: No fetch buffer rc= %d buf= %pK\n",
__func__, rc, buf);
- mutex_unlock(&vfe_dev->buf_mgr->lock);
return -EINVAL;
}
mapped_info = buf->mapped_info[0];
buf->state = MSM_ISP_BUFFER_STATE_DISPATCHED;
- mutex_unlock(&vfe_dev->buf_mgr->lock);
} else {
rc = vfe_dev->buf_mgr->ops->map_buf(vfe_dev->buf_mgr,
&mapped_info, fe_cfg->fd);
@@ -1228,10 +1214,6 @@ static void msm_vfe40_cfg_fetch_engine(struct vfe_device *vfe_dev,
case V4L2_PIX_FMT_P16GBRG10:
case V4L2_PIX_FMT_P16GRBG10:
case V4L2_PIX_FMT_P16RGGB10:
- case V4L2_PIX_FMT_P16BGGR12:
- case V4L2_PIX_FMT_P16GBRG12:
- case V4L2_PIX_FMT_P16GRBG12:
- case V4L2_PIX_FMT_P16RGGB12:
main_unpack_pattern = 0xB210;
break;
default:
@@ -2218,7 +2200,7 @@ static struct msm_vfe_axi_hardware_info msm_vfe40_axi_hw_info = {
.num_comp_mask = 3,
.num_rdi = 3,
.num_rdi_master = 3,
- .min_wm_ub = 96,
+ .min_wm_ub = 64,
.scratch_buf_range = SZ_32M + SZ_4M,
};
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_isp44.c b/drivers/media/platform/msm/camera_v2/isp/msm_isp44.c
index 8bd50d5..a9ff454 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_isp44.c
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_isp44.c
@@ -895,14 +895,13 @@ static int msm_vfe44_fetch_engine_start(struct vfe_device *vfe_dev,
mutex_lock(&vfe_dev->buf_mgr->lock);
rc = vfe_dev->buf_mgr->ops->get_buf_by_index(
vfe_dev->buf_mgr, bufq_handle, fe_cfg->buf_idx, &buf);
+ mutex_unlock(&vfe_dev->buf_mgr->lock);
if (rc < 0) {
pr_err("%s: No fetch buffer\n", __func__);
- mutex_unlock(&vfe_dev->buf_mgr->lock);
return -EINVAL;
}
mapped_info = buf->mapped_info[0];
buf->state = MSM_ISP_BUFFER_STATE_DISPATCHED;
- mutex_unlock(&vfe_dev->buf_mgr->lock);
} else {
rc = vfe_dev->buf_mgr->ops->map_buf(vfe_dev->buf_mgr,
&mapped_info, fe_cfg->fd);
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_isp46.c b/drivers/media/platform/msm/camera_v2/isp/msm_isp46.c
index e0712f7..0239fe7 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_isp46.c
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_isp46.c
@@ -833,15 +833,14 @@ static int msm_vfe46_start_fetch_engine(struct vfe_device *vfe_dev,
mutex_lock(&vfe_dev->buf_mgr->lock);
rc = vfe_dev->buf_mgr->ops->get_buf_by_index(
vfe_dev->buf_mgr, bufq_handle, fe_cfg->buf_idx, &buf);
+ mutex_unlock(&vfe_dev->buf_mgr->lock);
if (rc < 0 || !buf) {
pr_err("%s: No fetch buffer rc= %d buf= %pK\n",
__func__, rc, buf);
- mutex_unlock(&vfe_dev->buf_mgr->lock);
return -EINVAL;
}
mapped_info = buf->mapped_info[0];
buf->state = MSM_ISP_BUFFER_STATE_DISPATCHED;
- mutex_unlock(&vfe_dev->buf_mgr->lock);
} else {
rc = vfe_dev->buf_mgr->ops->map_buf(vfe_dev->buf_mgr,
&mapped_info, fe_cfg->fd);
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_isp47.c b/drivers/media/platform/msm/camera_v2/isp/msm_isp47.c
index b867eb3..410689f 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_isp47.c
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_isp47.c
@@ -1102,18 +1102,15 @@ int msm_vfe47_start_fetch_engine(struct vfe_device *vfe_dev,
fe_cfg->stream_id);
vfe_dev->fetch_engine_info.bufq_handle = bufq_handle;
- mutex_lock(&vfe_dev->buf_mgr->lock);
rc = vfe_dev->buf_mgr->ops->get_buf_by_index(
vfe_dev->buf_mgr, bufq_handle, fe_cfg->buf_idx, &buf);
if (rc < 0 || !buf) {
pr_err("%s: No fetch buffer rc= %d buf= %pK\n",
__func__, rc, buf);
- mutex_unlock(&vfe_dev->buf_mgr->lock);
return -EINVAL;
}
mapped_info = buf->mapped_info[0];
buf->state = MSM_ISP_BUFFER_STATE_DISPATCHED;
- mutex_unlock(&vfe_dev->buf_mgr->lock);
} else {
rc = vfe_dev->buf_mgr->ops->map_buf(vfe_dev->buf_mgr,
&mapped_info, fe_cfg->fd);
@@ -1166,15 +1163,14 @@ int msm_vfe47_start_fetch_engine_multi_pass(struct vfe_device *vfe_dev,
mutex_lock(&vfe_dev->buf_mgr->lock);
rc = vfe_dev->buf_mgr->ops->get_buf_by_index(
vfe_dev->buf_mgr, bufq_handle, fe_cfg->buf_idx, &buf);
+ mutex_unlock(&vfe_dev->buf_mgr->lock);
if (rc < 0 || !buf) {
pr_err("%s: No fetch buffer rc= %d buf= %pK\n",
__func__, rc, buf);
- mutex_unlock(&vfe_dev->buf_mgr->lock);
return -EINVAL;
}
mapped_info = buf->mapped_info[0];
buf->state = MSM_ISP_BUFFER_STATE_DISPATCHED;
- mutex_unlock(&vfe_dev->buf_mgr->lock);
} else {
rc = vfe_dev->buf_mgr->ops->map_buf(vfe_dev->buf_mgr,
&mapped_info, fe_cfg->fd);
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_isp_axi_util.c b/drivers/media/platform/msm/camera_v2/isp/msm_isp_axi_util.c
index 0b84327..6295d27 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_isp_axi_util.c
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_isp_axi_util.c
@@ -235,10 +235,6 @@ static int msm_isp_validate_axi_request(struct vfe_device *vfe_dev,
case V4L2_PIX_FMT_P16GBRG10:
case V4L2_PIX_FMT_P16GRBG10:
case V4L2_PIX_FMT_P16RGGB10:
- case V4L2_PIX_FMT_P16BGGR12:
- case V4L2_PIX_FMT_P16GBRG12:
- case V4L2_PIX_FMT_P16GRBG12:
- case V4L2_PIX_FMT_P16RGGB12:
case V4L2_PIX_FMT_JPEG:
case V4L2_PIX_FMT_META:
case V4L2_PIX_FMT_META10:
@@ -380,10 +376,6 @@ static uint32_t msm_isp_axi_get_plane_size(
case V4L2_PIX_FMT_P16GBRG10:
case V4L2_PIX_FMT_P16GRBG10:
case V4L2_PIX_FMT_P16RGGB10:
- case V4L2_PIX_FMT_P16BGGR12:
- case V4L2_PIX_FMT_P16GBRG12:
- case V4L2_PIX_FMT_P16GRBG12:
- case V4L2_PIX_FMT_P16RGGB12:
size = plane_cfg[plane_idx].output_height *
plane_cfg[plane_idx].output_width;
break;
@@ -3154,7 +3146,7 @@ static int msm_isp_start_axi_stream(struct vfe_device *vfe_dev_ioctl,
return -EINVAL;
msm_isp_get_timestamp(×tamp, vfe_dev_ioctl);
- mutex_lock(&vfe_dev_ioctl->buf_mgr->lock);
+
for (i = 0; i < stream_cfg_cmd->num_streams; i++) {
if (stream_cfg_cmd->stream_handle[i] == 0)
continue;
@@ -3162,7 +3154,6 @@ static int msm_isp_start_axi_stream(struct vfe_device *vfe_dev_ioctl,
HANDLE_TO_IDX(stream_cfg_cmd->stream_handle[i]));
if (!stream_info) {
pr_err("%s: stream_info is NULL", __func__);
- mutex_unlock(&vfe_dev_ioctl->buf_mgr->lock);
return -EINVAL;
}
if (SRC_TO_INTF(stream_info->stream_src) < VFE_SRC_MAX)
@@ -3172,7 +3163,6 @@ static int msm_isp_start_axi_stream(struct vfe_device *vfe_dev_ioctl,
else {
ISP_DBG("%s: invalid src info index\n", __func__);
rc = -EINVAL;
- mutex_unlock(&vfe_dev_ioctl->buf_mgr->lock);
goto error;
}
spin_lock_irqsave(&stream_info->lock, flags);
@@ -3184,7 +3174,6 @@ static int msm_isp_start_axi_stream(struct vfe_device *vfe_dev_ioctl,
}
if (rc) {
spin_unlock_irqrestore(&stream_info->lock, flags);
- mutex_unlock(&vfe_dev_ioctl->buf_mgr->lock);
goto error;
}
@@ -3207,7 +3196,6 @@ static int msm_isp_start_axi_stream(struct vfe_device *vfe_dev_ioctl,
HANDLE_TO_IDX(
stream_cfg_cmd->stream_handle[i]));
spin_unlock_irqrestore(&stream_info->lock, flags);
- mutex_unlock(&vfe_dev_ioctl->buf_mgr->lock);
goto error;
}
for (k = 0; k < stream_info->num_isp; k++) {
@@ -3266,7 +3254,6 @@ static int msm_isp_start_axi_stream(struct vfe_device *vfe_dev_ioctl,
spin_unlock_irqrestore(&stream_info->lock, flags);
streams[num_streams++] = stream_info;
}
- mutex_unlock(&vfe_dev_ioctl->buf_mgr->lock);
for (i = 0; i < MAX_VFE; i++) {
vfe_dev = update_vfes[i];
@@ -4045,12 +4032,10 @@ int msm_isp_update_axi_stream(struct vfe_device *vfe_dev, void *arg)
pr_err("%s: stream_info is null", __func__);
return -EINVAL;
}
- mutex_lock(&vfe_dev->buf_mgr->lock);
rc = msm_isp_request_frame(vfe_dev, stream_info,
update_info->user_stream_id,
update_info->frame_id,
MSM_ISP_INVALID_BUF_INDEX);
- mutex_unlock(&vfe_dev->buf_mgr->lock);
if (rc)
pr_err("%s failed to request frame!\n",
__func__);
@@ -4102,12 +4087,10 @@ int msm_isp_update_axi_stream(struct vfe_device *vfe_dev, void *arg)
pr_err("%s: stream_info is null", __func__);
return -EINVAL;
}
- mutex_lock(&vfe_dev->buf_mgr->lock);
rc = msm_isp_request_frame(vfe_dev, stream_info,
req_frm->user_stream_id,
req_frm->frame_id,
req_frm->buf_index);
- mutex_unlock(&vfe_dev->buf_mgr->lock);
if (rc)
pr_err("%s failed to request frame!\n",
__func__);
@@ -4310,11 +4293,11 @@ void msm_isp_process_axi_irq_stream(struct vfe_device *vfe_dev,
void msm_isp_process_axi_irq(struct vfe_device *vfe_dev,
uint32_t irq_status0, uint32_t irq_status1,
- uint32_t pingpong_status, struct msm_isp_timestamp *ts)
+ struct msm_isp_timestamp *ts)
{
int i, rc = 0;
uint32_t comp_mask = 0, wm_mask = 0;
- uint32_t stream_idx;
+ uint32_t pingpong_status, stream_idx;
struct msm_vfe_axi_stream *stream_info;
struct msm_vfe_axi_composite_info *comp_info;
struct msm_vfe_axi_shared_data *axi_data = &vfe_dev->axi_data;
@@ -4328,6 +4311,8 @@ void msm_isp_process_axi_irq(struct vfe_device *vfe_dev,
return;
ISP_DBG("%s: status: 0x%x\n", __func__, irq_status0);
+ pingpong_status =
+ vfe_dev->hw_info->vfe_ops.axi_ops.get_pingpong_status(vfe_dev);
for (i = 0; i < axi_data->hw_info->num_comp_mask; i++) {
rc = 0;
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_isp_axi_util.h b/drivers/media/platform/msm/camera_v2/isp/msm_isp_axi_util.h
index d27834f..5dcd967 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_isp_axi_util.h
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_isp_axi_util.h
@@ -56,7 +56,7 @@ void msm_isp_notify(struct vfe_device *vfe_dev, uint32_t event_type,
void msm_isp_process_axi_irq(struct vfe_device *vfe_dev,
uint32_t irq_status0, uint32_t irq_status1,
- uint32_t pingpong_status, struct msm_isp_timestamp *ts);
+ struct msm_isp_timestamp *ts);
void msm_isp_axi_disable_all_wm(struct vfe_device *vfe_dev);
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_isp_stats_util.c b/drivers/media/platform/msm/camera_v2/isp/msm_isp_stats_util.c
index 0634c0f..98be3dd 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_isp_stats_util.c
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_isp_stats_util.c
@@ -257,12 +257,13 @@ static int32_t msm_isp_stats_buf_divert(struct vfe_device *vfe_dev,
static int32_t msm_isp_stats_configure(struct vfe_device *vfe_dev,
uint32_t stats_irq_mask, struct msm_isp_timestamp *ts,
- uint32_t pingpong_status, bool is_composite)
+ bool is_composite)
{
int i, rc = 0;
struct msm_isp_event_data buf_event;
struct msm_isp_stats_event *stats_event = &buf_event.u.stats;
struct msm_vfe_stats_stream *stream_info = NULL;
+ uint32_t pingpong_status;
uint32_t comp_stats_type_mask = 0;
int result = 0;
@@ -271,6 +272,8 @@ static int32_t msm_isp_stats_configure(struct vfe_device *vfe_dev,
buf_event.mono_timestamp = ts->buf_time;
buf_event.frame_id = vfe_dev->axi_data.src_info[VFE_PIX_0].frame_id;
+ pingpong_status = vfe_dev->hw_info->
+ vfe_ops.stats_ops.get_pingpong_status(vfe_dev);
for (i = 0; i < vfe_dev->hw_info->stats_hw_info->num_stats_type; i++) {
if (!(stats_irq_mask & (1 << i)))
@@ -307,7 +310,7 @@ static int32_t msm_isp_stats_configure(struct vfe_device *vfe_dev,
void msm_isp_process_stats_irq(struct vfe_device *vfe_dev,
uint32_t irq_status0, uint32_t irq_status1,
- uint32_t pingpong_status, struct msm_isp_timestamp *ts)
+ struct msm_isp_timestamp *ts)
{
int j, rc;
uint32_t atomic_stats_mask = 0;
@@ -335,7 +338,7 @@ void msm_isp_process_stats_irq(struct vfe_device *vfe_dev,
/* Process non-composite irq */
if (stats_irq_mask) {
rc = msm_isp_stats_configure(vfe_dev, stats_irq_mask, ts,
- pingpong_status, comp_flag);
+ comp_flag);
}
/* Process composite irq */
@@ -348,7 +351,7 @@ void msm_isp_process_stats_irq(struct vfe_device *vfe_dev,
&vfe_dev->stats_data.stats_comp_mask[j]);
rc = msm_isp_stats_configure(vfe_dev, atomic_stats_mask,
- ts, pingpong_status, !comp_flag);
+ ts, !comp_flag);
}
}
}
@@ -1105,7 +1108,6 @@ static int msm_isp_start_stats_stream(struct vfe_device *vfe_dev_ioctl,
struct vfe_device *vfe_dev;
msm_isp_get_timestamp(×tamp, vfe_dev_ioctl);
- mutex_lock(&vfe_dev_ioctl->buf_mgr->lock);
num_stats_comp_mask =
vfe_dev_ioctl->hw_info->stats_hw_info->num_stats_comp_mask;
@@ -1126,7 +1128,6 @@ static int msm_isp_start_stats_stream(struct vfe_device *vfe_dev_ioctl,
}
if (rc) {
spin_unlock_irqrestore(&stream_info->lock, flags);
- mutex_unlock(&vfe_dev_ioctl->buf_mgr->lock);
goto error;
}
rc = msm_isp_init_stats_ping_pong_reg(
@@ -1134,7 +1135,6 @@ static int msm_isp_start_stats_stream(struct vfe_device *vfe_dev_ioctl,
if (rc < 0) {
spin_unlock_irqrestore(&stream_info->lock, flags);
pr_err("%s: No buffer for stream%d\n", __func__, idx);
- mutex_unlock(&vfe_dev_ioctl->buf_mgr->lock);
return rc;
}
init_completion(&stream_info->active_comp);
@@ -1169,7 +1169,6 @@ static int msm_isp_start_stats_stream(struct vfe_device *vfe_dev_ioctl,
stats_data->num_active_stream);
streams[num_stream++] = stream_info;
}
- mutex_unlock(&vfe_dev_ioctl->buf_mgr->lock);
for (k = 0; k < MAX_VFE; k++) {
if (!update_vfes[k] || num_active_streams[k])
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_isp_stats_util.h b/drivers/media/platform/msm/camera_v2/isp/msm_isp_stats_util.h
index 52df59f..9d5ae8c 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_isp_stats_util.h
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_isp_stats_util.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2013-2016, 2018 The Linux Foundation. All rights reserved.
+/* Copyright (c) 2013-2016, 2018, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -17,7 +17,7 @@
void msm_isp_process_stats_irq(struct vfe_device *vfe_dev,
uint32_t irq_status0, uint32_t irq_status1,
- uint32_t pingpong_status, struct msm_isp_timestamp *ts);
+ struct msm_isp_timestamp *ts);
int msm_isp_stats_create_stream(struct vfe_device *vfe_dev,
struct msm_vfe_stats_stream_request_cmd *stream_req_cmd,
struct msm_vfe_stats_stream *stream_info);
diff --git a/drivers/media/platform/msm/camera_v2/isp/msm_isp_util.c b/drivers/media/platform/msm/camera_v2/isp/msm_isp_util.c
index b589afa..290bdc0 100644
--- a/drivers/media/platform/msm/camera_v2/isp/msm_isp_util.c
+++ b/drivers/media/platform/msm/camera_v2/isp/msm_isp_util.c
@@ -402,10 +402,8 @@ static int msm_isp_start_fetch_engine_multi_pass(struct vfe_device *vfe_dev,
vfe_idx = msm_isp_get_vfe_idx_for_stream(vfe_dev, stream_info);
msm_isp_reset_framedrop(vfe_dev, stream_info);
- mutex_lock(&vfe_dev->buf_mgr->lock);
rc = msm_isp_cfg_offline_ping_pong_address(vfe_dev, stream_info,
VFE_PING_FLAG, fe_cfg->output_buf_idx);
- mutex_unlock(&vfe_dev->buf_mgr->lock);
if (rc < 0) {
pr_err("%s: Fetch engine config failed\n", __func__);
return -EINVAL;
@@ -926,7 +924,6 @@ static long msm_isp_ioctl_unlocked(struct v4l2_subdev *sd,
case VIDIOC_MSM_ISP_AXI_RESTART:
mutex_lock(&vfe_dev->core_mutex);
MSM_ISP_DUAL_VFE_MUTEX_LOCK(vfe_dev);
- mutex_lock(&vfe_dev->buf_mgr->lock);
if (atomic_read(&vfe_dev->error_info.overflow_state)
!= HALT_ENFORCED) {
rc = msm_isp_stats_restart(vfe_dev);
@@ -937,7 +934,6 @@ static long msm_isp_ioctl_unlocked(struct v4l2_subdev *sd,
pr_err_ratelimited("%s: no AXI restart, halt enforced.\n",
__func__);
}
- mutex_unlock(&vfe_dev->buf_mgr->lock);
MSM_ISP_DUAL_VFE_MUTEX_UNLOCK(vfe_dev);
mutex_unlock(&vfe_dev->core_mutex);
break;
@@ -1627,10 +1623,6 @@ int msm_isp_cal_word_per_line(uint32_t output_format,
case V4L2_PIX_FMT_P16GBRG10:
case V4L2_PIX_FMT_P16GRBG10:
case V4L2_PIX_FMT_P16RGGB10:
- case V4L2_PIX_FMT_P16BGGR12:
- case V4L2_PIX_FMT_P16GBRG12:
- case V4L2_PIX_FMT_P16GRBG12:
- case V4L2_PIX_FMT_P16RGGB12:
val = CAL_WORD(pixel_per_line, 1, 4);
break;
case V4L2_PIX_FMT_NV24:
@@ -1694,10 +1686,6 @@ enum msm_isp_pack_fmt msm_isp_get_pack_format(uint32_t output_format)
case V4L2_PIX_FMT_P16GBRG10:
case V4L2_PIX_FMT_P16GRBG10:
case V4L2_PIX_FMT_P16RGGB10:
- case V4L2_PIX_FMT_P16BGGR12:
- case V4L2_PIX_FMT_P16GBRG12:
- case V4L2_PIX_FMT_P16GRBG12:
- case V4L2_PIX_FMT_P16RGGB12:
return PLAIN16;
default:
msm_isp_print_fourcc_error(__func__, output_format);
@@ -1782,10 +1770,6 @@ int msm_isp_get_bit_per_pixel(uint32_t output_format)
case V4L2_PIX_FMT_QGRBG12:
case V4L2_PIX_FMT_QRGGB12:
case V4L2_PIX_FMT_Y12:
- case V4L2_PIX_FMT_P16BGGR12:
- case V4L2_PIX_FMT_P16GBRG12:
- case V4L2_PIX_FMT_P16GRBG12:
- case V4L2_PIX_FMT_P16RGGB12:
return 12;
case V4L2_PIX_FMT_SBGGR14:
case V4L2_PIX_FMT_SGBRG14:
@@ -2089,8 +2073,7 @@ void msm_isp_prepare_tasklet_debug_info(struct vfe_device *vfe_dev,
}
static void msm_isp_enqueue_tasklet_cmd(struct vfe_device *vfe_dev,
- uint32_t irq_status0, uint32_t irq_status1,
- uint32_t ping_pong_status)
+ uint32_t irq_status0, uint32_t irq_status1)
{
unsigned long flags;
struct msm_vfe_tasklet_queue_cmd *queue_cmd = NULL;
@@ -2112,8 +2095,8 @@ static void msm_isp_enqueue_tasklet_cmd(struct vfe_device *vfe_dev,
atomic_add(1, &vfe_dev->irq_cnt);
queue_cmd->vfeInterruptStatus0 = irq_status0;
queue_cmd->vfeInterruptStatus1 = irq_status1;
- queue_cmd->vfe_pingpong_status = ping_pong_status;
msm_isp_get_timestamp(&queue_cmd->ts, vfe_dev);
+
queue_cmd->cmd_used = 1;
queue_cmd->vfe_dev = vfe_dev;
@@ -2127,7 +2110,7 @@ static void msm_isp_enqueue_tasklet_cmd(struct vfe_device *vfe_dev,
irqreturn_t msm_isp_process_irq(int irq_num, void *data)
{
struct vfe_device *vfe_dev = (struct vfe_device *) data;
- uint32_t irq_status0, irq_status1, ping_pong_status;
+ uint32_t irq_status0, irq_status1;
uint32_t error_mask0, error_mask1;
vfe_dev->hw_info->vfe_ops.irq_ops.
@@ -2138,8 +2121,6 @@ irqreturn_t msm_isp_process_irq(int irq_num, void *data)
__func__, vfe_dev->pdev->id);
return IRQ_HANDLED;
}
- ping_pong_status = vfe_dev->hw_info->vfe_ops.axi_ops.
- get_pingpong_status(vfe_dev);
if (vfe_dev->hw_info->vfe_ops.irq_ops.preprocess_camif_irq) {
vfe_dev->hw_info->vfe_ops.irq_ops.preprocess_camif_irq(
vfe_dev, irq_status0);
@@ -2167,8 +2148,7 @@ irqreturn_t msm_isp_process_irq(int irq_num, void *data)
return IRQ_HANDLED;
}
msm_isp_prepare_irq_debug_info(vfe_dev, irq_status0, irq_status1);
- msm_isp_enqueue_tasklet_cmd(vfe_dev, irq_status0, irq_status1,
- ping_pong_status);
+ msm_isp_enqueue_tasklet_cmd(vfe_dev, irq_status0, irq_status1);
return IRQ_HANDLED;
}
@@ -2181,7 +2161,7 @@ void msm_isp_do_tasklet(unsigned long data)
struct msm_vfe_irq_ops *irq_ops;
struct msm_vfe_tasklet_queue_cmd *queue_cmd;
struct msm_isp_timestamp ts;
- uint32_t irq_status0, irq_status1, pingpong_status;
+ uint32_t irq_status0, irq_status1;
while (1) {
spin_lock_irqsave(&tasklet->tasklet_lock, flags);
@@ -2197,7 +2177,6 @@ void msm_isp_do_tasklet(unsigned long data)
queue_cmd->vfe_dev = NULL;
irq_status0 = queue_cmd->vfeInterruptStatus0;
irq_status1 = queue_cmd->vfeInterruptStatus1;
- pingpong_status = queue_cmd->vfe_pingpong_status;
ts = queue_cmd->ts;
spin_unlock_irqrestore(&tasklet->tasklet_lock, flags);
if (vfe_dev->vfe_open_cnt == 0) {
@@ -2222,11 +2201,9 @@ void msm_isp_do_tasklet(unsigned long data)
}
msm_isp_process_error_info(vfe_dev);
irq_ops->process_stats_irq(vfe_dev,
- irq_status0, irq_status1,
- pingpong_status, &ts);
+ irq_status0, irq_status1, &ts);
irq_ops->process_axi_irq(vfe_dev,
- irq_status0, irq_status1,
- pingpong_status, &ts);
+ irq_status0, irq_status1, &ts);
irq_ops->process_camif_irq(vfe_dev,
irq_status0, irq_status1, &ts);
irq_ops->process_reg_update(vfe_dev,
diff --git a/drivers/media/platform/msm/camera_v2/jpeg_dma/msm_jpeg_dma_dev.c b/drivers/media/platform/msm/camera_v2/jpeg_dma/msm_jpeg_dma_dev.c
index dc316b1..55e2646 100644
--- a/drivers/media/platform/msm/camera_v2/jpeg_dma/msm_jpeg_dma_dev.c
+++ b/drivers/media/platform/msm/camera_v2/jpeg_dma/msm_jpeg_dma_dev.c
@@ -816,13 +816,9 @@ static int msm_jpegdma_s_fmt_vid_out(struct file *file,
static int msm_jpegdma_reqbufs(struct file *file,
void *fh, struct v4l2_requestbuffers *req)
{
- int ret = 0;
struct jpegdma_ctx *ctx = msm_jpegdma_ctx_from_fh(fh);
- mutex_lock(&ctx->lock);
- ret = v4l2_m2m_reqbufs(file, ctx->m2m_ctx, req);
- mutex_unlock(&ctx->lock);
- return ret;
+ return v4l2_m2m_reqbufs(file, ctx->m2m_ctx, req);
}
/*
@@ -929,11 +925,11 @@ static int msm_jpegdma_streamoff(struct file *file,
{
struct jpegdma_ctx *ctx = msm_jpegdma_ctx_from_fh(fh);
int ret;
- mutex_lock(&ctx->lock);
+
ret = v4l2_m2m_streamoff(file, ctx->m2m_ctx, buf_type);
if (ret < 0)
dev_err(ctx->jdma_device->dev, "Stream off fails\n");
- mutex_unlock(&ctx->lock);
+
return ret;
}
diff --git a/drivers/media/platform/msm/camera_v2/jpeg_dma/msm_jpeg_dma_hw.c b/drivers/media/platform/msm/camera_v2/jpeg_dma/msm_jpeg_dma_hw.c
index 627d72c..702be49 100644
--- a/drivers/media/platform/msm/camera_v2/jpeg_dma/msm_jpeg_dma_hw.c
+++ b/drivers/media/platform/msm/camera_v2/jpeg_dma/msm_jpeg_dma_hw.c
@@ -79,10 +79,8 @@ static const struct msm_jpegdma_block msm_jpegdma_block_sel[] = {
*/
static inline long long jpegdma_do_div(long long num, long long den)
{
- uint64_t n = (uint64_t) num;
-
- do_div(n, (uint32_t)den);
- return n;
+ do_div(num, den);
+ return num;
}
/*
@@ -921,7 +919,7 @@ static int msm_jpegdma_hw_add_plane_offset(struct msm_jpegdma_plane *plane,
static int msm_jpegdma_hw_calc_config(struct msm_jpegdma_size_config *size_cfg,
struct msm_jpegdma_plane *plane)
{
- u64 scale_hor, scale_ver, phase = 0;
+ u64 scale_hor, scale_ver, phase;
u64 in_width, in_height;
u64 out_width, out_height;
struct msm_jpegdma_config *config;
diff --git a/drivers/media/platform/msm/camera_v2/msm_buf_mgr/msm_generic_buf_mgr.c b/drivers/media/platform/msm/camera_v2/msm_buf_mgr/msm_generic_buf_mgr.c
index c86528e..e350096 100644
--- a/drivers/media/platform/msm/camera_v2/msm_buf_mgr/msm_generic_buf_mgr.c
+++ b/drivers/media/platform/msm/camera_v2/msm_buf_mgr/msm_generic_buf_mgr.c
@@ -167,33 +167,6 @@ static int32_t msm_buf_mngr_buf_done(struct msm_buf_mngr_device *buf_mngr_dev,
return ret;
}
-static int32_t msm_buf_mngr_buf_error(struct msm_buf_mngr_device *buf_mngr_dev,
- struct msm_buf_mngr_info *buf_info)
-{
- unsigned long flags;
- struct msm_get_bufs *bufs, *save;
- int32_t ret = -EINVAL;
-
- spin_lock_irqsave(&buf_mngr_dev->buf_q_spinlock, flags);
- list_for_each_entry_safe(bufs, save, &buf_mngr_dev->buf_qhead, entry) {
- if ((bufs->session_id == buf_info->session_id) &&
- (bufs->stream_id == buf_info->stream_id) &&
- (bufs->index == buf_info->index)) {
- ret = buf_mngr_dev->vb2_ops.buf_error
- (bufs->vb2_v4l2_buf,
- buf_info->session_id,
- buf_info->stream_id,
- buf_info->frame_id,
- &buf_info->timestamp,
- buf_info->reserved);
- list_del_init(&bufs->entry);
- kfree(bufs);
- break;
- }
- }
- spin_unlock_irqrestore(&buf_mngr_dev->buf_q_spinlock, flags);
- return ret;
-}
static int32_t msm_buf_mngr_put_buf(struct msm_buf_mngr_device *buf_mngr_dev,
struct msm_buf_mngr_info *buf_info)
@@ -505,9 +478,6 @@ static int msm_cam_buf_mgr_ops(unsigned int cmd, void *argp)
case VIDIOC_MSM_BUF_MNGR_BUF_DONE:
rc = msm_buf_mngr_buf_done(msm_buf_mngr_dev, argp);
break;
- case VIDIOC_MSM_BUF_MNGR_BUF_ERROR:
- rc = msm_buf_mngr_buf_error(msm_buf_mngr_dev, argp);
- break;
case VIDIOC_MSM_BUF_MNGR_PUT_BUF:
rc = msm_buf_mngr_put_buf(msm_buf_mngr_dev, argp);
break;
@@ -575,13 +545,15 @@ static long msm_buf_mngr_subdev_ioctl(struct v4l2_subdev *sd,
k_ioctl = *ptr;
switch (k_ioctl.id) {
case MSM_CAMERA_BUF_MNGR_IOCTL_ID_GET_BUF_BY_IDX: {
- struct msm_buf_mngr_info buf_info, *tmp = NULL;
if (k_ioctl.size != sizeof(struct msm_buf_mngr_info))
return -EINVAL;
if (!k_ioctl.ioctl_ptr)
return -EINVAL;
- if (!is_compat_task()) {
+#ifndef CONFIG_COMPAT
+ {
+ struct msm_buf_mngr_info buf_info, *tmp = NULL;
+
MSM_CAM_GET_IOCTL_ARG_PTR(&tmp,
&k_ioctl.ioctl_ptr, sizeof(tmp));
if (copy_from_user(&buf_info, tmp,
@@ -590,7 +562,7 @@ static long msm_buf_mngr_subdev_ioctl(struct v4l2_subdev *sd,
}
k_ioctl.ioctl_ptr = (uintptr_t)&buf_info;
}
-
+#endif
argp = &k_ioctl;
rc = msm_cam_buf_mgr_ops(cmd, argp);
}
@@ -604,7 +576,6 @@ static long msm_buf_mngr_subdev_ioctl(struct v4l2_subdev *sd,
case VIDIOC_MSM_BUF_MNGR_GET_BUF:
case VIDIOC_MSM_BUF_MNGR_BUF_DONE:
case VIDIOC_MSM_BUF_MNGR_PUT_BUF:
- case VIDIOC_MSM_BUF_MNGR_BUF_ERROR:
rc = msm_cam_buf_mgr_ops(cmd, argp);
break;
case VIDIOC_MSM_BUF_MNGR_INIT:
@@ -753,9 +724,6 @@ static long msm_bmgr_subdev_fops_compat_ioctl(struct file *file,
case VIDIOC_MSM_BUF_MNGR_BUF_DONE32:
cmd = VIDIOC_MSM_BUF_MNGR_BUF_DONE;
break;
- case VIDIOC_MSM_BUF_MNGR_BUF_ERROR32:
- cmd = VIDIOC_MSM_BUF_MNGR_BUF_ERROR;
- break;
case VIDIOC_MSM_BUF_MNGR_PUT_BUF32:
cmd = VIDIOC_MSM_BUF_MNGR_PUT_BUF;
break;
@@ -774,7 +742,6 @@ static long msm_bmgr_subdev_fops_compat_ioctl(struct file *file,
switch (cmd) {
case VIDIOC_MSM_BUF_MNGR_GET_BUF:
case VIDIOC_MSM_BUF_MNGR_BUF_DONE:
- case VIDIOC_MSM_BUF_MNGR_BUF_ERROR:
case VIDIOC_MSM_BUF_MNGR_FLUSH:
case VIDIOC_MSM_BUF_MNGR_PUT_BUF: {
struct msm_buf_mngr_info32_t buf_info32;
diff --git a/drivers/media/platform/msm/camera_v2/msm_sd.h b/drivers/media/platform/msm/camera_v2/msm_sd.h
index 64cb785..f45c0d7 100644
--- a/drivers/media/platform/msm/camera_v2/msm_sd.h
+++ b/drivers/media/platform/msm/camera_v2/msm_sd.h
@@ -81,9 +81,6 @@ struct msm_sd_req_vb2_q {
unsigned int stream_id, uint32_t sequence, struct timeval *ts,
uint32_t reserved);
int (*flush_buf)(int session_id, unsigned int stream_id);
- int (*buf_error)(struct vb2_v4l2_buffer *vb2_v4l2_buf, int session_id,
- unsigned int stream_id, uint32_t sequence, struct timeval *ts,
- uint32_t reserved);
};
#define MSM_SD_NOTIFY_GET_SD 0x00000001
diff --git a/drivers/media/platform/msm/camera_v2/msm_vb2/msm_vb2.c b/drivers/media/platform/msm/camera_v2/msm_vb2/msm_vb2.c
index 6e528a6..5c13de5 100644
--- a/drivers/media/platform/msm/camera_v2/msm_vb2/msm_vb2.c
+++ b/drivers/media/platform/msm/camera_v2/msm_vb2/msm_vb2.c
@@ -443,73 +443,9 @@ static int msm_vb2_buf_done(struct vb2_v4l2_buffer *vb, int session_id,
vb2_v4l2_buf->sequence = sequence;
vb2_v4l2_buf->timecode.type = buf_type;
vb2_v4l2_buf->vb2_buf.timestamp =
- ((u64)ts->tv_sec * 1000000 +
- ts->tv_usec) * 1000;
- vb2_buffer_done(&vb2_v4l2_buf->vb2_buf,
- VB2_BUF_STATE_DONE);
- msm_vb2->in_freeq = 0;
- rc = 0;
- } else
- rc = -EINVAL;
- } else {
- pr_err(" VB buffer is NULL for ses_id=%d, str_id=%d\n",
- session_id, stream_id);
- rc = -EINVAL;
- }
- spin_unlock_irqrestore(&stream->stream_lock, flags);
- read_unlock_irqrestore(&session->stream_rwlock, rl_flags);
- return rc;
-}
-
-static int msm_vb2_buf_error(struct vb2_v4l2_buffer *vb, int session_id,
- unsigned int stream_id, uint32_t sequence,
- struct timeval *ts, uint32_t buf_type)
-{
- unsigned long flags, rl_flags;
- struct msm_vb2_buffer *msm_vb2;
- struct msm_stream *stream;
- struct msm_session *session;
- struct vb2_v4l2_buffer *vb2_v4l2_buf = NULL;
- int rc = 0;
-
- session = msm_get_session(session_id);
- if (IS_ERR_OR_NULL(session))
- return -EINVAL;
-
- read_lock_irqsave(&session->stream_rwlock, rl_flags);
-
- stream = msm_get_stream(session, stream_id);
- if (IS_ERR_OR_NULL(stream)) {
- read_unlock_irqrestore(&session->stream_rwlock, rl_flags);
- return -EINVAL;
- }
-
- spin_lock_irqsave(&stream->stream_lock, flags);
- if (vb) {
- list_for_each_entry(msm_vb2, &(stream->queued_list), list) {
- vb2_v4l2_buf = &(msm_vb2->vb2_v4l2_buf);
- if (vb2_v4l2_buf == vb)
- break;
- }
- if (vb2_v4l2_buf != vb) {
- pr_err("VB buffer is INVALID ses_id=%d, str_id=%d, vb=%pK\n",
- session_id, stream_id, vb);
- spin_unlock_irqrestore(&stream->stream_lock, flags);
- read_unlock_irqrestore(&session->stream_rwlock,
- rl_flags);
- return -EINVAL;
- }
- msm_vb2 =
- container_of(vb2_v4l2_buf, struct msm_vb2_buffer,
- vb2_v4l2_buf);
- /* put buf before buf done */
- if (msm_vb2->in_freeq) {
- vb2_v4l2_buf->sequence = sequence;
- vb2_v4l2_buf->timecode.type = buf_type;
- vb2_v4l2_buf->vb2_buf.timestamp =
(ts->tv_sec * 1000000 + ts->tv_usec) * 1000;
vb2_buffer_done(&vb2_v4l2_buf->vb2_buf,
- VB2_BUF_STATE_ERROR);
+ VB2_BUF_STATE_DONE);
msm_vb2->in_freeq = 0;
rc = 0;
} else
@@ -622,7 +558,6 @@ int msm_vb2_request_cb(struct msm_sd_req_vb2_q *req)
req->put_buf = msm_vb2_put_buf;
req->buf_done = msm_vb2_buf_done;
req->flush_buf = msm_vb2_flush_buf;
- req->buf_error = msm_vb2_buf_error;
return 0;
}
diff --git a/drivers/media/platform/msm/camera_v2/pproc/cpp/msm_cpp.c b/drivers/media/platform/msm/camera_v2/pproc/cpp/msm_cpp.c
index 85ca275..70bb3f2 100644
--- a/drivers/media/platform/msm/camera_v2/pproc/cpp/msm_cpp.c
+++ b/drivers/media/platform/msm/camera_v2/pproc/cpp/msm_cpp.c
@@ -949,14 +949,9 @@ static irqreturn_t msm_cpp_irq(int irq_num, void *data)
if (irq_status & 0x8) {
tx_level = msm_camera_io_r(cpp_dev->base +
MSM_CPP_MICRO_FIFO_TX_STAT) >> 2;
- if (tx_level < MSM_CPP_TX_FIFO_LEVEL) {
- for (i = 0; i < tx_level; i++) {
- tx_fifo[i] = msm_camera_io_r(cpp_dev->base +
- MSM_CPP_MICRO_FIFO_TX_DATA);
- }
- } else {
- pr_err("Fatal invalid tx level %d", tx_level);
- goto err;
+ for (i = 0; i < tx_level; i++) {
+ tx_fifo[i] = msm_camera_io_r(cpp_dev->base +
+ MSM_CPP_MICRO_FIFO_TX_DATA);
}
spin_lock_irqsave(&cpp_dev->tasklet_lock, flags);
queue_cmd = &cpp_dev->tasklet_queue_cmd[cpp_dev->taskletq_idx];
@@ -1011,7 +1006,6 @@ static irqreturn_t msm_cpp_irq(int irq_num, void *data)
pr_debug("DEBUG_R1: 0x%x\n",
msm_camera_io_r(cpp_dev->base + 0x8C));
}
-err:
msm_camera_io_w(irq_status, cpp_dev->base + MSM_CPP_MICRO_IRQGEN_CLR);
return IRQ_HANDLED;
}
@@ -1623,7 +1617,6 @@ static int msm_cpp_buffer_ops(struct cpp_device *cpp_dev,
case VIDIOC_MSM_BUF_MNGR_PUT_BUF:
case VIDIOC_MSM_BUF_MNGR_BUF_DONE:
case VIDIOC_MSM_BUF_MNGR_GET_BUF:
- case VIDIOC_MSM_BUF_MNGR_BUF_ERROR:
default: {
struct msm_buf_mngr_info *buff_mgr_info =
(struct msm_buf_mngr_info *)arg;
@@ -3611,7 +3604,7 @@ static long msm_cpp_subdev_ioctl(struct v4l2_subdev *sd,
break;
}
buff_mgr_info.frame_id = frame_info.frame_id;
- rc = msm_cpp_buffer_ops(cpp_dev, VIDIOC_MSM_BUF_MNGR_BUF_ERROR,
+ rc = msm_cpp_buffer_ops(cpp_dev, VIDIOC_MSM_BUF_MNGR_BUF_DONE,
0x0, &buff_mgr_info);
if (rc < 0) {
pr_err("error in buf done\n");
diff --git a/drivers/media/platform/msm/camera_v2/sensor/csid/msm_csid.c b/drivers/media/platform/msm/camera_v2/sensor/csid/msm_csid.c
index dfcb73aa..ba32526 100644
--- a/drivers/media/platform/msm/camera_v2/sensor/csid/msm_csid.c
+++ b/drivers/media/platform/msm/camera_v2/sensor/csid/msm_csid.c
@@ -229,9 +229,7 @@ static void msm_csid_set_sof_freeze_debug_reg(
static int msm_csid_reset(struct csid_device *csid_dev)
{
int32_t rc = 0;
- uint32_t irq = 0, irq_bitshift;
- irq_bitshift = csid_dev->ctrl_reg->csid_reg.csid_rst_done_irq_bitshift;
msm_camera_io_w(csid_dev->ctrl_reg->csid_reg.csid_rst_stb_all,
csid_dev->base +
csid_dev->ctrl_reg->csid_reg.csid_rst_cmd_addr);
@@ -240,23 +238,8 @@ static int msm_csid_reset(struct csid_device *csid_dev)
if (rc < 0) {
pr_err("wait_for_completion in msm_csid_reset fail rc = %d\n",
rc);
- } else if (rc == 0) {
- irq = msm_camera_io_r(csid_dev->base +
- csid_dev->ctrl_reg->csid_reg.csid_irq_status_addr);
- pr_err_ratelimited("%s CSID%d_IRQ_STATUS_ADDR = 0x%x\n",
- __func__, csid_dev->pdev->id, irq);
- if (irq & (0x1 << irq_bitshift)) {
- rc = 1;
- CDBG("%s succeeded", __func__);
- } else {
- rc = 0;
- pr_err("%s reset csid_irq_status failed = 0x%x\n",
- __func__, irq);
- }
if (rc == 0)
rc = -ETIMEDOUT;
- } else {
- CDBG("%s succeeded", __func__);
}
return rc;
}
diff --git a/drivers/media/platform/msm/camera_v2/sensor/csiphy/include/msm_csiphy_10_0_0_hwreg.h b/drivers/media/platform/msm/camera_v2/sensor/csiphy/include/msm_csiphy_10_0_0_hwreg.h
deleted file mode 100644
index 87c759c..0000000
--- a/drivers/media/platform/msm/camera_v2/sensor/csiphy/include/msm_csiphy_10_0_0_hwreg.h
+++ /dev/null
@@ -1,143 +0,0 @@
-/* Copyright (c) 2018, The Linux Foundation. All rights reserved.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 and
- * only version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
-
-#ifndef MSM_CSIPHY_10_0_0_HWREG_H
-#define MSM_CSIPHY_10_0_0_HWREG_H
-
-#include <sensor/csiphy/msm_csiphy.h>
-
-#define LANE_MASK_AGGR_MODE 0x1F
-#define LANE_MASK_PHY_A 0x13
-#define LANE_MASK_PHY_B 0x2C
-#define mask_phy_enable_A 0x3
-#define mask_phy_enable_B 0xC
-#define mask_base_dir_A 0x1
-#define mask_base_dir_B 0x2
-#define mask_force_mode_A 0x3
-#define mask_force_mode_B 0xC
-#define mask_enable_clk_A 0x1
-#define mask_enable_clk_B 0x2
-#define mask_ctrl_1_A 0x5
-#define mask_ctrl_1_B 0xA
-#define mask_hs_freq_range 0x7F
-#define mask_osc_freq_2 0xFF
-#define mask_osc_freq_3 0xF00
-
-static struct csiphy_reg_parms_t csiphy_v10_0_0 = {
- //ToDo: Fill these addresses from SWI
- .mipi_csiphy_interrupt_status0_addr = 0x5B4,
- .mipi_csiphy_interrupt_clear0_addr = 0x59C,
- .mipi_csiphy_glbl_irq_cmd_addr = 0x588,
- .mipi_csiphy_interrupt_clk_status0_addr = 0x5D8,
- .mipi_csiphy_interrupt_clk_clear0_addr = 0x5D0,
-};
-
-static struct csiphy_reg_snps_parms_t csiphy_v10_0_0_snps = {
- /*MIPI CSI PHY registers*/
- {0x560, 0x9}, /* mipi_csiphy_sys_ctrl */
- {0x5Ac, 0x9}, /* mipi_csiphy_sys_ctrl_1 */
- {0x564, 0xF}, /* mipi_csiphy_ctrl_1 */
- {0x568, 0x49}, /* mipi_csiphy_ctrl_2 */
- {0x56C, 0x49}, /* mipi_csiphy_ctrl_3 */
- {0x580, 0x7F}, /* mipi_csiphy_fifo_ctrl */
- {0x570, 0x0}, /* mipi_csiphy_enable */
- {0x578, 0x0}, /* mipi_csiphy_basedir */
- {0x57C, 0x0}, /* mipi_csiphy_force_mode */
- {0x574, 0x0}, /* mipi_csiphy_enable_clk */
- {0x58C, 0xFF}, /* mipi_csiphy_irq_mask_ctrl_lane_0 */
- {0x5C8, 0xFF}, /* mipi_csiphy_irq_mask_ctrl_lane_clk_0 */
- {0x20, 0x0}, /* mipi_csiphy_rx_sys_7_00 */
- {0x28, 0x43}, /* mipi_csiphy_rx_sys_9_00 */
- {0x380, 0x0}, /* mipi_csiphy_rx_startup_ovr_0_00 */
- {0x384, 0x0}, /* mipi_csiphy_rx_startup_ovr_1_00 */
- {0x388, 0xCC}, /* mipi_csiphy_rx_startup_ovr_2_00 */
- {0x38C, 0x1}, /* mipi_csiphy_rx_startup_ovr_3_00 */
- {0x390, 0x1}, /* mipi_csiphy_rx_startup_ovr_4_00 */
- {0x394, 0x1}, /* mipi_csiphy_rx_startup_ovr_5_00 */
- {0x324, 0x0}, /* mipi_csiphy_rx_startup_obs_2_00 */
- {0x6B0, 0x0}, /* mipi_csiphy_rx_cb_2_00 */
- {0x4CC, 0x1}, /* mipi_csiphy_rx_dual_phy_0_00 */
- {0xC0, 0x0}, /* mipi_csiphy_rx_clk_lane_3_00 */
- {0xC4, 0xA}, /* mipi_csiphy_rx_clk_lane_4_00 */
- {0xC8, 0x0}, /* mipi_csiphy_rx_clk_lane_6_00 */
- {0x12c, 0x0}, /* mipi_csiphy_rx_lane_0_7_00 */
- {0x220, 0x0}, /* mipi_csiphy_rx_lane_1_7_00 */
- {0xCC, 0x0}, /* mipi_csiphy_rx_clk_lane_7_00 */
-};
-
-static struct snps_freq_value snps_v100_freq_values[] = {
- {80, 0x0, 460 }, /* 80 - 97.125*/
- {90, 0x10, 460 }, /* 80 - 107.625*/
- {100, 0x20, 460 }, /* 83.125 - 118.125*/
- {110, 0x30, 460 }, /* 92.625 - 128.625*/
- {120, 0x1, 460 }, /* 102.125 - 139.125*/
- {130, 0x11, 460 }, /* 111.625 - 149.625*/
- {140, 0x21, 460 }, /* 121.125 - 160.125*/
- {150, 0x31, 460 }, /* 130.625 - 170.625*/
- {160, 0x2, 460 }, /* 140.125 - 181.125*/
- {170, 0x12, 460 }, /* 149.625 - 191.625*/
- {180, 0x22, 460 }, /* 159.125 - 202.125*/
- {190, 0x32, 460 }, /* 168.625 - 212.625*/
- {205, 0x3, 460 }, /* 182.875 - 228.375*/
- {220, 0x13, 460 }, /* 197.125 - 244.125*/
- {235, 0x23, 460 }, /* 211.375 - 259.875*/
- {250, 0x33, 460 }, /* 225.625 - 275.625*/
- {275, 0x4, 460 }, /* 249.375 - 301.875*/
- {300, 0x14, 460 }, /* 273.125 - 328.125*/
- {325, 0x25, 460 }, /* 296.875 - 354.375*/
- {350, 0x35, 460 }, /* 320.625 - 380.625*/
- {400, 0x5, 460 }, /* 368.125 - 433.125*/
- {450, 0x16, 460 }, /* 415.625 - 485.625*/
- {500, 0x26, 460 }, /* 463.125 - 538.125*/
- {550, 0x37, 460 }, /* 510.625 - 590.625*/
- {600, 0x7, 460 }, /* 558.125 - 643.125*/
- {650, 0x18, 460 }, /* 605.625 - 695.625*/
- {700, 0x28, 460 }, /* 653.125 - 748.125*/
- {750, 0x39, 460 }, /* 700.625 - 800.625*/
- {800, 0x9, 460 }, /* 748.125 - 853.125*/
- {850, 0x19, 460 }, /* 795.625 - 905.625*/
- {900, 0x29, 460 }, /* 843.125 - 958.125*/
- {950, 0x3a, 460 }, /* 890.625 - 1010.625*/
- {1000, 0xa, 460 }, /* 938.125 - 1063.125*/
- {1050, 0x1a, 460 }, /* 985.625 - 1115.625*/
- {1100, 0x2a, 460 }, /*1033.125 - 1168.125*/
- {1150, 0x3b, 460 }, /*1080.625 - 1220.625*/
- {1200, 0xb, 460 }, /*1128.125 - 1273.125*/
- {1250, 0x1b, 460 }, /*1175.625 - 1325.625*/
- {1300, 0x2b, 460 }, /*1223.125 - 1378.125*/
- {1350, 0x3c, 460 }, /*1270.625 - 1430.625*/
- {1400, 0xc, 460 }, /*1318.125 - 1483.125*/
- {1450, 0x1c, 460}, /*1365.625 - 1535.625*/
- {1500, 0x2c, 460 }, /*1413.125 - 1588.125*/
- {1550, 0x3d, 285 }, /*1460.625 - 1640.625*/
- {1600, 0xd, 295 }, /*1508.125 - 1693.125*/
- {1650, 0x1d, 304 }, /*1555.625 - 1745.625*/
- {1700, 0x2e, 313 }, /*1603.125 - 1798.125*/
- {1750, 0x3e, 322 }, /*1650.625 - 1850.625*/
- {1800, 0xe, 331 }, /*1698.125 - 1903.125*/
- {1850, 0x1e, 341 }, /*1745.625 - 1955.625*/
- {1900, 0x2f, 350 }, /*1793.125 - 2008.125*/
- {1950, 0x3f, 359 }, /*1840.625 - 2060.625*/
- {2000, 0xf, 368 }, /*1888.125 - 2113.125*/
- {2050, 0x40, 377 }, /*1935.625 - 2165.625*/
- {2100, 0x41, 387 }, /*1983.125 - 2218.125*/
- {2150, 0x42, 396 }, /*2030.625 - 2270.625*/
- {2200, 0x43, 405 }, /*2078.125 - 2323.125*/
- {2250, 0x44, 414 }, /*2125.625 - 2375.625*/
- {2300, 0x45, 423 }, /*2173.125 - 2428.125*/
- {2350, 0x46, 432 }, /*2220.625 - 2480.625*/
- {2400, 0x47, 442 }, /* 2268.125 - 2500*/
- {2450, 0x48, 451 }, /* 2315.625 - 2500*/
- {2500, 0x49, 460 }, /* 2363.125 - 2500*/
-};
-
-#endif
diff --git a/drivers/media/platform/msm/camera_v2/sensor/csiphy/msm_csiphy.c b/drivers/media/platform/msm/camera_v2/sensor/csiphy/msm_csiphy.c
index fceedb8..edf022e 100644
--- a/drivers/media/platform/msm/camera_v2/sensor/csiphy/msm_csiphy.c
+++ b/drivers/media/platform/msm/camera_v2/sensor/csiphy/msm_csiphy.c
@@ -16,7 +16,6 @@
#include <linux/of.h>
#include <linux/module.h>
#include <linux/irqreturn.h>
-#include <asm/div64.h>
#include "msm_csiphy.h"
#include "msm_sd.h"
#include "include/msm_csiphy_2_0_hwreg.h"
@@ -28,8 +27,6 @@
#include "include/msm_csiphy_3_5_hwreg.h"
#include "include/msm_csiphy_5_0_hwreg.h"
#include "include/msm_csiphy_5_0_1_hwreg.h"
-#include "include/msm_csiphy_10_0_0_hwreg.h"
-
#include "cam_hw_ops.h"
#define DBG_CSIPHY 0
@@ -46,7 +43,6 @@
#define CSIPHY_VERSION_V35 0x35
#define CSIPHY_VERSION_V50 0x500
#define CSIPHY_VERSION_V501 0x501
-#define CSIPHY_VERSION_V1000 0x1000
#define MSM_CSIPHY_DRV_NAME "msm_csiphy"
#define CLK_LANE_OFFSET 1
#define NUM_LANES_OFFSET 4
@@ -56,10 +52,6 @@
#define MAX_DPHY_DATA_LN 4
#define CLOCK_OFFSET 0x700
#define CSIPHY_SOF_DEBUG_COUNT 2
-#define MBPS 1000000
-#define SNPS_INTERPHY_OFFSET 0x800
-#define SET_THE_BIT(x) (0x1 << x)
-#define SNPS_MAX_DATA_RATE_PER_LANE 2500000000ULL
#undef CDBG
#define CDBG(fmt, args...) pr_debug(fmt, ##args)
@@ -82,42 +74,6 @@ static void msm_csiphy_write_settings(
}
}
-static void snps_irq_config(
- struct csiphy_device *csiphy_dev, bool enable)
-{
- uint16_t offset = 0x4;
- uint16_t data;
- void __iomem *csiphybase;
-
- csiphybase = csiphy_dev->base;
-
- if (enable)
- data = csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_irq_mask_ctrl_lane_0.data;
- else
- data = 0;
-
- msm_camera_io_w(data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_irq_mask_ctrl_lane_0.addr);
- msm_camera_io_w(data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_irq_mask_ctrl_lane_0.addr + offset);
- msm_camera_io_w(data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_irq_mask_ctrl_lane_0.addr + 2 * offset);
- msm_camera_io_w(data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_irq_mask_ctrl_lane_0.addr + 3 * offset);
-
- msm_camera_io_w(data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_irq_mask_ctrl_lane_clk_0.addr);
- msm_camera_io_w(data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_irq_mask_ctrl_lane_clk_0.addr + offset);
-}
-
static void msm_csiphy_cphy_irq_config(
struct csiphy_device *csiphy_dev,
struct msm_camera_csiphy_params *csiphy_params)
@@ -125,479 +81,50 @@ static void msm_csiphy_cphy_irq_config(
void __iomem *csiphybase;
csiphybase = csiphy_dev->base;
- if (csiphy_dev->is_snps_phy) {
- snps_irq_config(csiphy_dev, true);
- } else {
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl11.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl11.addr);
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl12.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl12.addr);
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl13.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl13.addr);
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl14.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl14.addr);
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl15.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl15.addr);
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl16.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl16.addr);
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl17.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl17.addr);
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl18.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl18.addr);
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl19.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl19.addr);
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl20.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl20.addr);
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl21.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl21.addr);
- }
-}
-
-static int msm_csiphy_snps_2_lane_config(
- struct csiphy_device *csiphy_dev,
- struct msm_camera_csiphy_params *csiphy_params,
- enum snps_csiphy_mode mode, int num_lanes)
-
-{
- uint64_t local_data_rate = 0;
- uint32_t offset;
- uint32_t value, i, diff, diff_i;
- void __iomem *csiphybase;
-
- csiphybase = csiphy_dev->base;
-
- if (csiphy_params->data_rate >
- SNPS_MAX_DATA_RATE_PER_LANE * num_lanes) {
- pr_err("unsupported data rate\n");
- return -EINVAL;
- }
-
- local_data_rate = csiphy_params->data_rate;
-
- if (mode == TWO_LANE_PHY_A)
- offset = 0x0;
- else if (mode == TWO_LANE_PHY_B)
- offset = SNPS_INTERPHY_OFFSET;
- else
- return -EINVAL;
-
- do_div(local_data_rate, num_lanes * MBPS);
- diff = abs(snps_v100_freq_values[0].default_bit_rate -
- local_data_rate);
- /* ToDo: Can be optimized to a O(1) search */
- for (i = 1; i < sizeof(snps_v100_freq_values)/
- sizeof(snps_v100_freq_values[0]); i++) {
- diff_i = abs(snps_v100_freq_values[i].default_bit_rate -
- local_data_rate);
- if (diff_i > diff) {
- i--;
- break;
- }
- diff = diff_i;
- }
-
- if (mode == TWO_LANE_PHY_A) {
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_sys_ctrl.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_sys_ctrl.addr + offset);
-
- msm_camera_io_w((snps_v100_freq_values[i].hs_freq &
- mask_hs_freq_range),
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_ctrl_3.addr + offset);
- } else {
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_sys_ctrl_1.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_sys_ctrl_1.addr);
-
- msm_camera_io_w((snps_v100_freq_values[i].hs_freq &
- mask_hs_freq_range),
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_ctrl_2.addr);
- }
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_sys_7_00.addr + offset);
- value |= SET_THE_BIT(5);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_sys_7_00.addr + offset);
-
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_sys_9_00.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_sys_9_00.addr + offset);
-
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_ovr_4_00.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_ovr_4_00.addr + offset);
-
- msm_camera_io_w((snps_v100_freq_values[i].osc_freq &
- mask_osc_freq_2),
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_ovr_2_00.addr + offset);
-
- msm_camera_io_w((snps_v100_freq_values[i].osc_freq &
- mask_osc_freq_3) >> 8,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_ovr_3_00.addr + offset);
-
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_ovr_5_00.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_ovr_5_00.addr + offset);
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_cb_2_00.addr + offset);
- value |= SET_THE_BIT(6);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_cb_2_00.addr + offset);
-
- return 0;
-}
-
-static int msm_csiphy_snps_lane_config(
- struct csiphy_device *csiphy_dev,
- struct msm_camera_csiphy_params *csiphy_params)
-{
- int ret;
- uint16_t lane_mask = 0;
- void __iomem *csiphybase;
- enum snps_csiphy_mode mode = INVALID_MODE;
- uint32_t value, num_tries, num_lanes, offset;
- uint32_t clk_mux_reg = 0;
-
- csiphybase = csiphy_dev->base;
- if (csiphy_dev->clk_mux_base != NULL)
- clk_mux_reg = msm_camera_io_r(csiphy_dev->clk_mux_base);
- else {
- pr_err("%s: invalid clk_mux_base\n", __func__);
- return -EINVAL;
- }
-
- /* lane mask usage
- * BIT LANE
- * 0(LSB) PHY A data 0
- * 1 PHY A data 1
- * 2 PHY B data 0
- * 3 PHY B data 1
- * 4 PHY A clk (clk 0)
- * 5 PHY B clk (clk 1)
- */
-
- lane_mask = csiphy_params->lane_mask & 0x3f;
- CDBG("%s:%d lane_maks: %d, cur_snps_state = %d\n",
- __func__, __LINE__, lane_mask, csiphy_dev->snps_state);
-
- if (lane_mask == LANE_MASK_AGGR_MODE) { /* Aggregate mdoe */
- /* 4 lane config */
- mode = AGGREGATE_MODE;
- num_lanes = 4;
- if (csiphy_dev->snps_state != NOT_CONFIGURED) {
- pr_err("%s: invalid request\n", __func__);
- return -EINVAL;
- }
- csiphy_dev->snps_state = CONFIGURED_AGGREGATE_MODE;
- clk_mux_reg &= ~0xff;
- clk_mux_reg |= csiphy_params->csid_core << 4;
- clk_mux_reg |= (uint32_t)csiphy_params->csid_core;
- } else if (lane_mask == LANE_MASK_PHY_A) { /* PHY A */
- /* 2 lane config */
- num_lanes = 2;
- if (csiphy_dev->snps_state == NOT_CONFIGURED) {
- mode = TWO_LANE_PHY_A;
- csiphy_dev->snps_state = CONFIGURED_TWO_LANE_PHY_A;
- } else if (csiphy_dev->snps_state ==
- CONFIGURED_TWO_LANE_PHY_B) {
- /* 2 lane + 2 lane config */
- mode = TWO_LANE_PHY_A;
- csiphy_dev->snps_state = CONFIGURED_COMBO_MODE;
- } else {
- pr_err("%s: invalid request\n", __func__);
- return -EINVAL;
- }
- clk_mux_reg &= ~0xf;
- clk_mux_reg |= (uint32_t)csiphy_params->csid_core;
- } else if (lane_mask == LANE_MASK_PHY_B) { /* PHY B */
- /* 2 lane config */
- num_lanes = 2;
- if (csiphy_dev->snps_state == NOT_CONFIGURED) {
- mode = TWO_LANE_PHY_B;
- csiphy_dev->snps_state = CONFIGURED_TWO_LANE_PHY_B;
- } else if (csiphy_dev->snps_state ==
- CONFIGURED_TWO_LANE_PHY_A) {
- /* 2 lane + 2 lane config */
- mode = TWO_LANE_PHY_B;
- csiphy_dev->snps_state = CONFIGURED_COMBO_MODE;
- } else {
- pr_err("%s: invalid request\n", __func__);
- return -EINVAL;
- }
- clk_mux_reg &= ~0xf0;
- clk_mux_reg |= csiphy_params->csid_core << 4;
- } else { /* None of available configurations */
- pr_err("%s: invalid configuration requested\n", __func__);
- return -EINVAL;
- }
-
- msm_camera_io_w(clk_mux_reg, csiphy_dev->clk_mux_base);
- /* ensure write is done */
- mb();
-
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_A) {
- ret = msm_csiphy_snps_2_lane_config(csiphy_dev,
- csiphy_params, TWO_LANE_PHY_A, num_lanes);
- if (ret < 0) {
- pr_err("%s:%d: Error in setting lane configuration\n",
- __func__, __LINE__);
- return ret;
- }
- }
-
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_B) {
- ret = msm_csiphy_snps_2_lane_config(csiphy_dev,
- csiphy_params, TWO_LANE_PHY_B, num_lanes);
- if (ret < 0) {
- pr_err("%s:%d: Error in setting lane configuration\n",
- __func__, __LINE__);
- return ret;
- }
- }
-
- snps_irq_config(csiphy_dev, csiphy_params);
-
- value = 0x0;
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_A)
- value |= mask_force_mode_A;
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_B)
- value |= mask_force_mode_B;
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_force_mode.addr);
-
- if (mode == AGGREGATE_MODE) {
- /* Programming PHY A as master and PHY B as slave */
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_dual_phy_0_00.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_dual_phy_0_00.addr);
-
- msm_camera_io_w(!(csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_dual_phy_0_00.data),
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_dual_phy_0_00.addr +
- SNPS_INTERPHY_OFFSET);
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_lane_0_7_00.addr);
- value |= SET_THE_BIT(5);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_lane_0_7_00.addr);
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_lane_0_7_00.addr +
- SNPS_INTERPHY_OFFSET);
- value |= SET_THE_BIT(5);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_lane_0_7_00.addr +
- SNPS_INTERPHY_OFFSET);
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_lane_1_7_00.addr);
- value |= SET_THE_BIT(5);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_lane_1_7_00.addr);
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_lane_1_7_00.addr +
- SNPS_INTERPHY_OFFSET);
- value |= SET_THE_BIT(5);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_lane_1_7_00.addr +
- SNPS_INTERPHY_OFFSET);
-
- value = csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_7_00.data;
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_7_00.addr);
- value |= SET_THE_BIT(3);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_7_00.addr +
- SNPS_INTERPHY_OFFSET);
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_ovr_0_00.addr +
- SNPS_INTERPHY_OFFSET);
- value |= SET_THE_BIT(0);
- value |= SET_THE_BIT(1);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_ovr_0_00.addr +
- SNPS_INTERPHY_OFFSET);
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_ovr_1_00.addr +
- SNPS_INTERPHY_OFFSET);
- value &= ~(SET_THE_BIT(0));
- value |= SET_THE_BIT(1);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_ovr_1_00.addr +
- SNPS_INTERPHY_OFFSET);
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_6_00.addr);
- value |= SET_THE_BIT(2);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_6_00.addr);
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_6_00.addr +
- SNPS_INTERPHY_OFFSET);
- value |= SET_THE_BIT(3);
- value |= SET_THE_BIT(7);
- value &= ~(SET_THE_BIT(2));
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_6_00.addr +
- SNPS_INTERPHY_OFFSET);
-
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_3_00.addr +
- SNPS_INTERPHY_OFFSET);
- value |= SET_THE_BIT(7);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_3_00.addr +
- SNPS_INTERPHY_OFFSET);
-
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_4_00.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_clk_lane_4_00.addr +
- SNPS_INTERPHY_OFFSET);
-
- value = csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_fifo_ctrl.data;
- value &= ~(SET_THE_BIT(0));
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_fifo_ctrl.addr);
- value |= SET_THE_BIT(0);
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_fifo_ctrl.addr);
- }
-
- value = 0x0;
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_A)
- value |= mask_phy_enable_A;
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_B)
- value |= mask_phy_enable_B;
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_enable.addr);
-
- value = 0x0;
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_A)
- value |= mask_base_dir_A;
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_B)
- value |= mask_base_dir_B;
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_basedir.addr);
-
- value = 0x0;
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_A)
- value |= mask_enable_clk_A;
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_B)
- value |= mask_enable_clk_B;
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_enable_clk.addr);
-
- value = 0x0;
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_A)
- value |= mask_ctrl_1_A;
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_B)
- value |= mask_ctrl_1_B;
- msm_camera_io_w(value,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_ctrl_1.addr);
-
- if (mode == AGGREGATE_MODE || mode == TWO_LANE_PHY_A)
- offset = 0x0;
- else
- offset = SNPS_INTERPHY_OFFSET;
-
- value = 0x0;
- num_tries = 0;
-
- do {
- num_tries++;
- value = msm_camera_io_r(csiphybase +
- csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_rx_startup_obs_2_00.addr + offset);
- if ((value | SET_THE_BIT(4)) == value)
- break;
- usleep_range(100, 150);
- } while (num_tries < 6);
-
- if ((value | SET_THE_BIT(4)) != value) {
- pr_err("%s: SNPS phy config failed\n", __func__);
- return -EINVAL;
- }
-
- msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_force_mode.data,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_snps_reg.
- mipi_csiphy_force_mode.addr);
-
- return 0;
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl11.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl11.addr);
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl12.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl12.addr);
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl13.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl13.addr);
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl14.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl14.addr);
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl15.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl15.addr);
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl16.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl16.addr);
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl17.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl17.addr);
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl18.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl18.addr);
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl19.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl19.addr);
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl20.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl20.addr);
+ msm_camera_io_w(csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl21.data,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl21.addr);
}
static int msm_csiphy_3phase_lane_config(
@@ -1255,26 +782,14 @@ static int msm_csiphy_lane_config(struct csiphy_device *csiphy_dev,
ratio = csiphy_dev->csiphy_max_clk/clk_rate;
csiphy_params->settle_cnt = csiphy_params->settle_cnt/ratio;
}
- CDBG("%s csiphy_params, mask = 0x%x cnt = %d, data rate = %llu\n",
+ CDBG("%s csiphy_params, mask = 0x%x cnt = %d\n",
__func__,
csiphy_params->lane_mask,
- csiphy_params->lane_cnt, csiphy_params->data_rate);
+ csiphy_params->lane_cnt);
CDBG("%s csiphy_params, settle cnt = 0x%x csid %d\n",
__func__, csiphy_params->settle_cnt,
csiphy_params->csid_core);
- if (csiphy_dev->is_snps_phy) {
- rc = msm_csiphy_snps_lane_config(csiphy_dev,
- csiphy_params);
- if (rc < 0) {
- pr_err("%s:%d: Error in setting lane configuration\n",
- __func__, __LINE__);
- }
- csiphy_dev->num_irq_registers = 4;
- csiphy_dev->num_clk_irq_registers = 2;
- return rc;
- }
-
if (csiphy_dev->hw_version >= CSIPHY_VERSION_V30 &&
csiphy_dev->clk_mux_base != NULL &&
csiphy_dev->hw_version < CSIPHY_VERSION_V50) {
@@ -1428,43 +943,39 @@ static void msm_csiphy_disable_irq(
void __iomem *csiphybase;
csiphybase = csiphy_dev->base;
- if (csiphy_dev->is_snps_phy) {
- snps_irq_config(csiphy_dev, false);
- } else {
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl11.addr);
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl12.addr);
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl13.addr);
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl14.addr);
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl15.addr);
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl16.addr);
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl17.addr);
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl18.addr);
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl19.addr);
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl20.addr);
- msm_camera_io_w(0,
- csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
- mipi_csiphy_3ph_cmn_ctrl21.addr);
- }
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl11.addr);
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl12.addr);
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl13.addr);
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl14.addr);
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl15.addr);
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl16.addr);
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl17.addr);
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl18.addr);
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl19.addr);
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl20.addr);
+ msm_camera_io_w(0,
+ csiphybase + csiphy_dev->ctrl_reg->csiphy_3ph_reg.
+ mipi_csiphy_3ph_cmn_ctrl21.addr);
}
static irqreturn_t msm_csiphy_irq(int irq_num, void *data)
@@ -1499,33 +1010,12 @@ static irqreturn_t msm_csiphy_irq(int irq_num, void *data)
csiphy_dev->ctrl_reg->csiphy_reg.
mipi_csiphy_interrupt_clear0_addr + 0x4*i);
}
-
- if (csiphy_dev->is_snps_phy) {
- for (i = 0; i < csiphy_dev->num_clk_irq_registers; i++) {
- irq = msm_camera_io_r(
- csiphy_dev->base +
- csiphy_dev->ctrl_reg->csiphy_reg.
- mipi_csiphy_interrupt_clk_status0_addr + 0x4*i);
- msm_camera_io_w(irq,
- csiphy_dev->base +
- csiphy_dev->ctrl_reg->csiphy_reg.
- mipi_csiphy_interrupt_clk_clear0_addr + 0x4*i);
- pr_err_ratelimited(
- "%s CSIPHY%d_IRQ_CLK_STATUS_ADDR%d = 0x%x\n",
- __func__, csiphy_dev->pdev->id, i, irq);
- msm_camera_io_w(0x0,
- csiphy_dev->base +
- csiphy_dev->ctrl_reg->csiphy_reg.
- mipi_csiphy_interrupt_clk_clear0_addr + 0x4*i);
- }
- }
msm_camera_io_w(0x1, csiphy_dev->base +
csiphy_dev->ctrl_reg->
csiphy_reg.mipi_csiphy_glbl_irq_cmd_addr);
msm_camera_io_w(0x0, csiphy_dev->base +
csiphy_dev->ctrl_reg->
csiphy_reg.mipi_csiphy_glbl_irq_cmd_addr);
-
return IRQ_HANDLED;
}
@@ -1549,23 +1039,6 @@ static void msm_csiphy_3ph_reset(struct csiphy_device *csiphy_dev)
mipi_csiphy_3ph_cmn_ctrl0.addr);
}
-static void msm_csiphy_snps_reset(struct csiphy_device *csiphy_dev)
-{
- //Need to toggle this register to enable IRQ
- msm_camera_io_w(0x1, csiphy_dev->base +
- csiphy_dev->ctrl_reg->
- csiphy_reg.mipi_csiphy_glbl_irq_cmd_addr);
- usleep_range(5000, 8000);
- msm_camera_io_w(0x0, csiphy_dev->base +
- csiphy_dev->ctrl_reg->
- csiphy_reg.mipi_csiphy_glbl_irq_cmd_addr);
-}
-
-static void msm_csiphy_snps_release(struct csiphy_device *csiphy_dev)
-{
- CDBG("Releasing SNPS phy\n");
-}
-
#if DBG_CSIPHY
static int msm_csiphy_init(struct csiphy_device *csiphy_dev)
{
@@ -1639,9 +1112,7 @@ static int msm_csiphy_init(struct csiphy_device *csiphy_dev)
if (rc < 0)
pr_err("%s: irq enable failed\n", __func__);
- if (csiphy_dev->is_snps_phy)
- msm_csiphy_snps_reset(csiphy_dev);
- else if (csiphy_dev->csiphy_3phase == CSI_3PHASE_HW)
+ if (csiphy_dev->csiphy_3phase == CSI_3PHASE_HW)
msm_csiphy_3ph_reset(csiphy_dev);
else
msm_csiphy_reset(csiphy_dev);
@@ -1659,7 +1130,6 @@ static int msm_csiphy_init(struct csiphy_device *csiphy_dev)
CDBG("%s:%d called csiphy_dev->hw_version 0x%x\n", __func__, __LINE__,
csiphy_dev->hw_version);
csiphy_dev->csiphy_state = CSIPHY_POWER_UP;
- csiphy_dev->snps_state = NOT_CONFIGURED;
return 0;
csiphy_enable_clk_fail:
@@ -1744,9 +1214,7 @@ static int msm_csiphy_init(struct csiphy_device *csiphy_dev)
}
CDBG("%s:%d clk enable success\n", __func__, __LINE__);
- if (csiphy_dev->is_snps_phy)
- msm_csiphy_snps_reset(csiphy_dev);
- else if (csiphy_dev->csiphy_3phase == CSI_3PHASE_HW)
+ if (csiphy_dev->csiphy_3phase == CSI_3PHASE_HW)
msm_csiphy_3ph_reset(csiphy_dev);
else
msm_csiphy_reset(csiphy_dev);
@@ -1765,7 +1233,6 @@ static int msm_csiphy_init(struct csiphy_device *csiphy_dev)
CDBG("%s:%d called csiphy_dev->hw_version 0x%x\n", __func__, __LINE__,
csiphy_dev->hw_version);
csiphy_dev->csiphy_state = CSIPHY_POWER_UP;
- csiphy_dev->snps_state = NOT_CONFIGURED;
return 0;
csiphy_enable_clk_fail:
@@ -1855,41 +1322,28 @@ static int msm_csiphy_release(struct csiphy_device *csiphy_dev, void *arg)
csiphy_dev->lane_mask[csiphy_dev->pdev->id] &=
~(csi_lane_mask);
- if (csiphy_dev->is_snps_phy) {
- msm_csiphy_snps_release(csiphy_dev);
- } else {
- i = 0;
- while (csi_lane_mask) {
- if (csi_lane_mask & 0x1) {
- msm_camera_io_w(0x0,
- csiphy_dev->base +
- csiphy_dev->ctrl_reg->
- csiphy_reg.
- mipi_csiphy_lnn_cfg2_addr +
- 0x40*i);
- msm_camera_io_w(0x0,
- csiphy_dev->base +
- csiphy_dev->ctrl_reg->
- csiphy_reg.
- mipi_csiphy_lnn_misc1_addr +
- 0x40*i);
- msm_camera_io_w(0x0,
- csiphy_dev->base +
- csiphy_dev->ctrl_reg->
- csiphy_reg.
- mipi_csiphy_lnn_test_imp +
- 0x40*i);
- }
- csi_lane_mask >>= 1;
- i++;
+ i = 0;
+ while (csi_lane_mask) {
+ if (csi_lane_mask & 0x1) {
+ msm_camera_io_w(0x0, csiphy_dev->base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_lnn_cfg2_addr + 0x40*i);
+ msm_camera_io_w(0x0, csiphy_dev->base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_lnn_misc1_addr + 0x40*i);
+ msm_camera_io_w(0x0, csiphy_dev->base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_lnn_test_imp + 0x40*i);
}
- msm_camera_io_w(0x0, csiphy_dev->base +
- csiphy_dev->ctrl_reg->csiphy_reg.
- mipi_csiphy_lnck_cfg2_addr);
- msm_camera_io_w(0x0, csiphy_dev->base +
- csiphy_dev->ctrl_reg->csiphy_reg.
- mipi_csiphy_glbl_pwr_cfg_addr);
+ csi_lane_mask >>= 1;
+ i++;
}
+ msm_camera_io_w(0x0, csiphy_dev->base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_lnck_cfg2_addr);
+ msm_camera_io_w(0x0, csiphy_dev->base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_glbl_pwr_cfg_addr);
}
rc = msm_camera_enable_irq(csiphy_dev->irq, false);
@@ -1914,7 +1368,6 @@ static int msm_csiphy_release(struct csiphy_device *csiphy_dev, void *arg)
NULL, 0, &csiphy_dev->csiphy_reg_ptr[0], 0);
csiphy_dev->csiphy_state = CSIPHY_POWER_DOWN;
- csiphy_dev->snps_state = NOT_CONFIGURED;
if (cam_config_ahb_clk(NULL, 0, CAM_AHB_CLIENT_CSIPHY,
CAM_AHB_SUSPEND_VOTE) < 0)
@@ -1989,41 +1442,28 @@ static int msm_csiphy_release(struct csiphy_device *csiphy_dev, void *arg)
csiphy_dev->lane_mask[csiphy_dev->pdev->id] &=
~(csi_lane_mask);
- if (csiphy_dev->is_snps_phy) {
- msm_csiphy_snps_release(csiphy_dev);
- } else {
- i = 0;
- while (csi_lane_mask) {
- if (csi_lane_mask & 0x1) {
- msm_camera_io_w(0x0,
- csiphy_dev->base +
- csiphy_dev->ctrl_reg->
- csiphy_reg.
- mipi_csiphy_lnn_cfg2_addr +
- 0x40*i);
- msm_camera_io_w(0x0,
- csiphy_dev->base +
- csiphy_dev->ctrl_reg->
- csiphy_reg.
- mipi_csiphy_lnn_misc1_addr +
- 0x40*i);
- msm_camera_io_w(0x0,
- csiphy_dev->base +
- csiphy_dev->ctrl_reg->
- csiphy_reg.
- mipi_csiphy_lnn_test_imp +
- 0x40*i);
- }
- csi_lane_mask >>= 1;
- i++;
+ i = 0;
+ while (csi_lane_mask) {
+ if (csi_lane_mask & 0x1) {
+ msm_camera_io_w(0x0, csiphy_dev->base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_lnn_cfg2_addr + 0x40*i);
+ msm_camera_io_w(0x0, csiphy_dev->base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_lnn_misc1_addr + 0x40*i);
+ msm_camera_io_w(0x0, csiphy_dev->base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_lnn_test_imp + 0x40*i);
}
- msm_camera_io_w(0x0, csiphy_dev->base +
- csiphy_dev->ctrl_reg->csiphy_reg.
- mipi_csiphy_lnck_cfg2_addr);
- msm_camera_io_w(0x0, csiphy_dev->base +
- csiphy_dev->ctrl_reg->csiphy_reg.
- mipi_csiphy_glbl_pwr_cfg_addr);
+ csi_lane_mask >>= 1;
+ i++;
}
+ msm_camera_io_w(0x0, csiphy_dev->base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_lnck_cfg2_addr);
+ msm_camera_io_w(0x0, csiphy_dev->base +
+ csiphy_dev->ctrl_reg->csiphy_reg.
+ mipi_csiphy_glbl_pwr_cfg_addr);
}
if (csiphy_dev->csiphy_sof_debug == SOF_DEBUG_ENABLE)
rc = msm_camera_enable_irq(csiphy_dev->irq, false);
@@ -2046,7 +1486,6 @@ static int msm_csiphy_release(struct csiphy_device *csiphy_dev, void *arg)
NULL, 0, &csiphy_dev->csiphy_reg_ptr[0], 0);
csiphy_dev->csiphy_state = CSIPHY_POWER_DOWN;
- csiphy_dev->snps_state = NOT_CONFIGURED;
if (cam_config_ahb_clk(NULL, 0, CAM_AHB_CLIENT_CSIPHY,
CAM_AHB_SUSPEND_VOTE) < 0)
@@ -2364,12 +1803,6 @@ static int csiphy_probe(struct platform_device *pdev)
new_csiphy_dev->csiphy_3phase = CSI_3PHASE_HW;
new_csiphy_dev->ctrl_reg->csiphy_combo_mode_settings =
csiphy_combo_mode_v5_0_1;
- } else if (of_device_is_compatible(new_csiphy_dev->pdev->dev.of_node,
- "qcom,csiphy-v10.00")) {
- new_csiphy_dev->ctrl_reg->csiphy_snps_reg = csiphy_v10_0_0_snps;
- new_csiphy_dev->ctrl_reg->csiphy_reg = csiphy_v10_0_0;
- new_csiphy_dev->hw_dts_version = CSIPHY_VERSION_V1000;
- new_csiphy_dev->is_snps_phy = 1;
} else {
pr_err("%s:%d, invalid hw version : 0x%x\n", __func__, __LINE__,
new_csiphy_dev->hw_dts_version);
diff --git a/drivers/media/platform/msm/camera_v2/sensor/csiphy/msm_csiphy.h b/drivers/media/platform/msm/camera_v2/sensor/csiphy/msm_csiphy.h
index 79baf3c..bc8f2d9 100644
--- a/drivers/media/platform/msm/camera_v2/sensor/csiphy/msm_csiphy.h
+++ b/drivers/media/platform/msm/camera_v2/sensor/csiphy/msm_csiphy.h
@@ -64,41 +64,6 @@ struct csiphy_reg_parms_t {
uint32_t mipi_csiphy_t_wakeup_cfg0_addr;
uint32_t csiphy_version;
uint32_t combo_clk_mask;
- uint32_t mipi_csiphy_interrupt_clk_status0_addr;
- uint32_t mipi_csiphy_interrupt_clk_clear0_addr;
-};
-
-struct csiphy_reg_snps_parms_t {
- /*MIPI CSI PHY registers*/
- struct csiphy_reg_t mipi_csiphy_sys_ctrl;
- struct csiphy_reg_t mipi_csiphy_sys_ctrl_1;
- struct csiphy_reg_t mipi_csiphy_ctrl_1;
- struct csiphy_reg_t mipi_csiphy_ctrl_2;
- struct csiphy_reg_t mipi_csiphy_ctrl_3;
- struct csiphy_reg_t mipi_csiphy_fifo_ctrl;
- struct csiphy_reg_t mipi_csiphy_enable;
- struct csiphy_reg_t mipi_csiphy_basedir;
- struct csiphy_reg_t mipi_csiphy_force_mode;
- struct csiphy_reg_t mipi_csiphy_enable_clk;
- struct csiphy_reg_t mipi_csiphy_irq_mask_ctrl_lane_0;
- struct csiphy_reg_t mipi_csiphy_irq_mask_ctrl_lane_clk_0;
- struct csiphy_reg_t mipi_csiphy_rx_sys_7_00;
- struct csiphy_reg_t mipi_csiphy_rx_sys_9_00;
- struct csiphy_reg_t mipi_csiphy_rx_startup_ovr_0_00;
- struct csiphy_reg_t mipi_csiphy_rx_startup_ovr_1_00;
- struct csiphy_reg_t mipi_csiphy_rx_startup_ovr_2_00;
- struct csiphy_reg_t mipi_csiphy_rx_startup_ovr_3_00;
- struct csiphy_reg_t mipi_csiphy_rx_startup_ovr_4_00;
- struct csiphy_reg_t mipi_csiphy_rx_startup_ovr_5_00;
- struct csiphy_reg_t mipi_csiphy_rx_startup_obs_2_00;
- struct csiphy_reg_t mipi_csiphy_rx_cb_2_00;
- struct csiphy_reg_t mipi_csiphy_rx_dual_phy_0_00;
- struct csiphy_reg_t mipi_csiphy_rx_clk_lane_3_00;
- struct csiphy_reg_t mipi_csiphy_rx_clk_lane_4_00;
- struct csiphy_reg_t mipi_csiphy_rx_clk_lane_6_00;
- struct csiphy_reg_t mipi_csiphy_rx_lane_0_7_00;
- struct csiphy_reg_t mipi_csiphy_rx_lane_1_7_00;
- struct csiphy_reg_t mipi_csiphy_rx_clk_lane_7_00;
};
struct csiphy_reg_3ph_parms_t {
@@ -185,7 +150,6 @@ struct csiphy_ctrl_t {
struct csiphy_reg_parms_t csiphy_reg;
struct csiphy_reg_3ph_parms_t csiphy_3ph_reg;
struct csiphy_settings_t csiphy_combo_mode_settings;
- struct csiphy_reg_snps_parms_t csiphy_snps_reg;
};
enum msm_csiphy_state_t {
@@ -193,27 +157,6 @@ enum msm_csiphy_state_t {
CSIPHY_POWER_DOWN,
};
-enum snps_csiphy_mode {
- AGGREGATE_MODE,
- TWO_LANE_PHY_A,
- TWO_LANE_PHY_B,
- INVALID_MODE,
-};
-
-enum snps_csiphy_state {
- NOT_CONFIGURED,
- CONFIGURED_AGGREGATE_MODE,
- CONFIGURED_TWO_LANE_PHY_A,
- CONFIGURED_TWO_LANE_PHY_B,
- CONFIGURED_COMBO_MODE,
-};
-
-struct snps_freq_value {
- uint32_t default_bit_rate;
- uint8_t hs_freq;
- uint16_t osc_freq;
-};
-
struct csiphy_device {
struct platform_device *pdev;
struct msm_sd_subdev msm_sd;
@@ -247,9 +190,6 @@ struct csiphy_device {
struct camera_vreg_t *csiphy_vreg;
struct regulator *csiphy_reg_ptr[MAX_REGULATOR];
int32_t regulator_count;
- uint8_t is_snps_phy;
- enum snps_csiphy_state snps_state;
- uint8_t num_clk_irq_registers;
};
#define VIDIOC_MSM_CSIPHY_RELEASE \
diff --git a/drivers/media/platform/msm/camera_v2/sensor/io/msm_camera_dt_util.c b/drivers/media/platform/msm/camera_v2/sensor/io/msm_camera_dt_util.c
index 9e99ba2..d044a50 100644
--- a/drivers/media/platform/msm/camera_v2/sensor/io/msm_camera_dt_util.c
+++ b/drivers/media/platform/msm/camera_v2/sensor/io/msm_camera_dt_util.c
@@ -67,8 +67,6 @@ int msm_camera_fill_vreg_params(struct camera_vreg_t *cam_vreg,
break;
}
}
- if (j == num_vreg)
- power_setting[i].seq_val = INVALID_VREG;
break;
case CAM_VIO:
@@ -88,8 +86,6 @@ int msm_camera_fill_vreg_params(struct camera_vreg_t *cam_vreg,
break;
}
}
- if (j == num_vreg)
- power_setting[i].seq_val = INVALID_VREG;
break;
case CAM_VANA:
@@ -109,8 +105,6 @@ int msm_camera_fill_vreg_params(struct camera_vreg_t *cam_vreg,
break;
}
}
- if (j == num_vreg)
- power_setting[i].seq_val = INVALID_VREG;
break;
case CAM_VAF:
@@ -130,8 +124,6 @@ int msm_camera_fill_vreg_params(struct camera_vreg_t *cam_vreg,
break;
}
}
- if (j == num_vreg)
- power_setting[i].seq_val = INVALID_VREG;
break;
case CAM_V_CUSTOM1:
@@ -152,9 +144,7 @@ int msm_camera_fill_vreg_params(struct camera_vreg_t *cam_vreg,
break;
}
}
- if (j == num_vreg)
- power_setting[i].seq_val = INVALID_VREG;
- break;
+
case CAM_V_CUSTOM2:
for (j = 0; j < num_vreg; j++) {
if (!strcmp(cam_vreg[j].reg_name,
@@ -173,8 +163,6 @@ int msm_camera_fill_vreg_params(struct camera_vreg_t *cam_vreg,
break;
}
}
- if (j == num_vreg)
- power_setting[i].seq_val = INVALID_VREG;
break;
default:
@@ -1501,9 +1489,6 @@ int msm_camera_power_up(struct msm_camera_power_ctrl_t *ctrl,
(int) power_setting->config_val);
break;
case SENSOR_VREG:
- if (power_setting->seq_val == INVALID_VREG)
- break;
-
if (power_setting->seq_val >= CAM_VREG_MAX) {
pr_err("%s vreg index %d >= max %d\n", __func__,
power_setting->seq_val,
diff --git a/drivers/media/platform/msm/sde/rotator/sde_rotator_core.c b/drivers/media/platform/msm/sde/rotator/sde_rotator_core.c
index ce50dcd..e248813 100644
--- a/drivers/media/platform/msm/sde/rotator/sde_rotator_core.c
+++ b/drivers/media/platform/msm/sde/rotator/sde_rotator_core.c
@@ -303,7 +303,7 @@ static int sde_rotator_footswitch_ctrl(struct sde_rot_mgr *mgr, bool on)
int ret = 0;
if (mgr->regulator_enable == on) {
- SDEROT_ERR("Regulators already in selected mode on=%d\n", on);
+ SDEROT_DBG("Regulators already in selected mode on=%d\n", on);
return 0;
}
@@ -337,6 +337,7 @@ static int sde_rotator_footswitch_ctrl(struct sde_rot_mgr *mgr, bool on)
}
mgr->regulator_enable = on;
+
return 0;
}
@@ -3213,6 +3214,7 @@ static void sde_rotator_suspend_cancel_rot_work(struct sde_rot_mgr *mgr)
int sde_rotator_runtime_suspend(struct device *dev)
{
struct sde_rot_mgr *mgr;
+ int rc = 0;
mgr = sde_rot_mgr_from_device(dev);
@@ -3227,10 +3229,12 @@ int sde_rotator_runtime_suspend(struct device *dev)
return -EBUSY;
}
- sde_rotator_footswitch_ctrl(mgr, false);
- ATRACE_END("runtime_active");
- SDEROT_DBG("exit runtime_active\n");
- return 0;
+ SDEROT_DBG("PM runtime suspend\n");
+ ATRACE_END("pm_runtime_active");
+ if (mgr->regulator_enable != false)
+ rc = sde_rotator_footswitch_ctrl(mgr, false);
+
+ return rc;
}
/*
@@ -3240,6 +3244,7 @@ int sde_rotator_runtime_suspend(struct device *dev)
int sde_rotator_runtime_resume(struct device *dev)
{
struct sde_rot_mgr *mgr;
+ int rc = 0;
mgr = sde_rot_mgr_from_device(dev);
@@ -3248,9 +3253,12 @@ int sde_rotator_runtime_resume(struct device *dev)
return -ENODEV;
}
- SDEROT_DBG("begin runtime_active\n");
- ATRACE_BEGIN("runtime_active");
- return sde_rotator_footswitch_ctrl(mgr, true);
+ SDEROT_DBG("PM runtime resume\n");
+ ATRACE_BEGIN("pm_runtime_active");
+ if (mgr->regulator_enable != true)
+ rc = sde_rotator_footswitch_ctrl(mgr, true);
+
+ return rc;
}
/*
@@ -3269,7 +3277,7 @@ int sde_rotator_runtime_idle(struct device *dev)
}
/* add check for any busy status, if any */
- SDEROT_DBG("idling ...\n");
+ SDEROT_DBG("PM runtime idle\n");
return 0;
}
@@ -3292,14 +3300,15 @@ int sde_rotator_pm_suspend(struct device *dev)
return -ENODEV;
}
-
+ SDEROT_DBG("PM system suspend\n");
sde_rot_mgr_lock(mgr);
atomic_inc(&mgr->device_suspended);
sde_rotator_suspend_cancel_rot_work(mgr);
mgr->minimum_bw_vote = 0;
sde_rotator_update_perf(mgr);
- ATRACE_END("pm_active");
- SDEROT_DBG("end pm active %d\n", atomic_read(&mgr->device_suspended));
+ SDEROT_DBG("end pm system active %d\n",
+ atomic_read(&mgr->device_suspended));
+ ATRACE_END("pm_system_active");
sde_rot_mgr_unlock(mgr);
return 0;
}
@@ -3320,18 +3329,19 @@ int sde_rotator_pm_resume(struct device *dev)
}
/*
- * It is possible that the runtime status of the device may
- * have been active when the system was suspended. Reset the runtime
- * status to suspended state after a complete system resume.
+ * When exiting system sleep resume, set the state to active
+ * to allow runtime resume to be executed next.
*/
pm_runtime_disable(dev);
pm_runtime_set_suspended(dev);
pm_runtime_set_active(dev);
pm_runtime_enable(dev);
+ SDEROT_DBG("PM system resume\n");
sde_rot_mgr_lock(mgr);
- SDEROT_DBG("begin pm active %d\n", atomic_read(&mgr->device_suspended));
- ATRACE_BEGIN("pm_active");
+ SDEROT_DBG("begin pm system active %d\n",
+ atomic_read(&mgr->device_suspended));
+ ATRACE_BEGIN("pm_system_active");
atomic_dec(&mgr->device_suspended);
sde_rotator_update_perf(mgr);
sde_rot_mgr_unlock(mgr);
@@ -3351,6 +3361,8 @@ int sde_rotator_suspend(struct platform_device *dev, pm_message_t state)
return -ENODEV;
}
+ SDEROT_DBG("PM only suspend\n");
+
sde_rot_mgr_lock(mgr);
atomic_inc(&mgr->device_suspended);
sde_rotator_suspend_cancel_rot_work(mgr);
@@ -3370,6 +3382,8 @@ int sde_rotator_resume(struct platform_device *dev)
return -ENODEV;
}
+ SDEROT_DBG("PM only resume\n");
+
sde_rot_mgr_lock(mgr);
atomic_dec(&mgr->device_suspended);
sde_rotator_update_perf(mgr);
diff --git a/drivers/media/platform/msm/vidc/Makefile b/drivers/media/platform/msm/vidc/Makefile
index e33eaa8..0564e97 100644
--- a/drivers/media/platform/msm/vidc/Makefile
+++ b/drivers/media/platform/msm/vidc/Makefile
@@ -21,3 +21,7 @@
obj-$(CONFIG_MSM_VIDC_V4L2) += governors/
obj-$(CONFIG_MSM_VIDC_VMEM) += vmem/
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-pointer-bool-conversion \
diff --git a/drivers/media/platform/msm/vidc/hfi_response_handler.c b/drivers/media/platform/msm/vidc/hfi_response_handler.c
index 0cdcbd6..20d8530 100644
--- a/drivers/media/platform/msm/vidc/hfi_response_handler.c
+++ b/drivers/media/platform/msm/vidc/hfi_response_handler.c
@@ -1840,9 +1840,6 @@ static int hfi_process_ignore(u32 device_id,
int hfi_process_msg_packet(u32 device_id, struct vidc_hal_msg_pkt_hdr *msg_hdr,
struct msm_vidc_cb_info *info)
{
- typedef int (*pkt_func_def)(u32, void *, struct msm_vidc_cb_info *info);
- pkt_func_def pkt_func = NULL;
-
if (!info || !msg_hdr || msg_hdr->size < VIDC_IFACEQ_MIN_PKT_SIZE) {
dprintk(VIDC_ERR, "%s: bad packet/packet size\n",
__func__);
@@ -1852,61 +1849,74 @@ int hfi_process_msg_packet(u32 device_id, struct vidc_hal_msg_pkt_hdr *msg_hdr,
dprintk(VIDC_DBG, "Parse response %#x\n", msg_hdr->packet);
switch (msg_hdr->packet) {
case HFI_MSG_EVENT_NOTIFY:
- pkt_func = (pkt_func_def)hfi_process_event_notify;
- break;
+ return hfi_process_event_notify(device_id,
+ (struct hfi_msg_event_notify_packet *)msg_hdr,
+ info);
case HFI_MSG_SYS_INIT_DONE:
- pkt_func = (pkt_func_def)hfi_process_sys_init_done;
- break;
+ return hfi_process_sys_init_done(device_id,
+ (struct hfi_msg_sys_init_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SYS_SESSION_INIT_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_init_done;
- break;
+ return hfi_process_session_init_done(device_id,
+ (struct hfi_msg_sys_session_init_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SYS_PROPERTY_INFO:
- pkt_func = (pkt_func_def)hfi_process_sys_property_info;
- break;
+ return hfi_process_sys_property_info(device_id,
+ (struct hfi_msg_sys_property_info_packet *)msg_hdr,
+ info);
case HFI_MSG_SYS_SESSION_END_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_end_done;
- break;
+ return hfi_process_session_end_done(device_id,
+ (struct hfi_msg_sys_session_end_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SESSION_LOAD_RESOURCES_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_load_res_done;
- break;
+ return hfi_process_session_load_res_done(device_id,
+ (struct hfi_msg_session_load_resources_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SESSION_START_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_start_done;
- break;
+ return hfi_process_session_start_done(device_id,
+ (struct hfi_msg_session_start_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SESSION_STOP_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_stop_done;
- break;
+ return hfi_process_session_stop_done(device_id,
+ (struct hfi_msg_session_stop_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SESSION_EMPTY_BUFFER_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_etb_done;
- break;
+ return hfi_process_session_etb_done(device_id,
+ (struct hfi_msg_session_empty_buffer_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SESSION_FILL_BUFFER_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_ftb_done;
- break;
+ return hfi_process_session_ftb_done(device_id, msg_hdr, info);
case HFI_MSG_SESSION_FLUSH_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_flush_done;
- break;
+ return hfi_process_session_flush_done(device_id,
+ (struct hfi_msg_session_flush_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SESSION_PROPERTY_INFO:
- pkt_func = (pkt_func_def)hfi_process_session_prop_info;
- break;
+ return hfi_process_session_prop_info(device_id,
+ (struct hfi_msg_session_property_info_packet *)msg_hdr,
+ info);
case HFI_MSG_SESSION_RELEASE_RESOURCES_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_rel_res_done;
- break;
+ return hfi_process_session_rel_res_done(device_id,
+ (struct hfi_msg_session_release_resources_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SYS_RELEASE_RESOURCE:
- pkt_func = (pkt_func_def)hfi_process_sys_rel_resource_done;
- break;
+ return hfi_process_sys_rel_resource_done(device_id,
+ (struct hfi_msg_sys_release_resource_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SESSION_RELEASE_BUFFERS_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_rel_buf_done;
- break;
+ return hfi_process_session_rel_buf_done(device_id,
+ (struct hfi_msg_session_release_buffers_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SYS_SESSION_ABORT_DONE:
- pkt_func = (pkt_func_def)hfi_process_session_abort_done;
- break;
+ return hfi_process_session_abort_done(device_id,
+ (struct hfi_msg_sys_session_abort_done_packet *)msg_hdr,
+ info);
case HFI_MSG_SESSION_SYNC_DONE:
- pkt_func = (pkt_func_def)hfi_process_ignore;
- break;
+ return hfi_process_ignore(device_id, msg_hdr, info);
default:
dprintk(VIDC_DBG, "Unable to parse message: %#x\n",
msg_hdr->packet);
break;
}
- return pkt_func ? pkt_func(device_id, msg_hdr, info) : -ENOTSUPP;
+ return -ENOTSUPP;
}
diff --git a/drivers/media/platform/msm/vidc/msm_vidc.c b/drivers/media/platform/msm/vidc/msm_vidc.c
index 651d0a2..815d137 100644
--- a/drivers/media/platform/msm/vidc/msm_vidc.c
+++ b/drivers/media/platform/msm/vidc/msm_vidc.c
@@ -1860,6 +1860,8 @@ static void msm_vidc_cleanup_instance(struct msm_vidc_inst *inst)
dprintk(VIDC_ERR,
"Failed to release mark_data buffers\n");
+ msm_comm_free_buffer_tags(inst);
+
msm_comm_release_eos_buffers(inst);
if (msm_comm_release_output_buffers(inst, true))
diff --git a/drivers/media/platform/msm/vidc/msm_vidc_common.c b/drivers/media/platform/msm/vidc/msm_vidc_common.c
index 9dfb56f..bc42db9 100644
--- a/drivers/media/platform/msm/vidc/msm_vidc_common.c
+++ b/drivers/media/platform/msm/vidc/msm_vidc_common.c
@@ -1747,6 +1747,7 @@ static void handle_event_change(enum hal_command_response cmd, void *data)
dprintk(VIDC_ERR,
"Failed : No buffer requirements : %x\n",
HAL_BUFFER_OUTPUT);
+ mutex_unlock(&inst->lock);
return;
}
@@ -1758,6 +1759,7 @@ static void handle_event_change(enum hal_command_response cmd, void *data)
dprintk(VIDC_ERR,
"Failed : No buffer requirements : %x\n",
HAL_BUFFER_OUTPUT2);
+ mutex_unlock(&inst->lock);
return;
}
@@ -1770,6 +1772,7 @@ static void handle_event_change(enum hal_command_response cmd, void *data)
dprintk(VIDC_ERR,
"Failed : No buffer requirements : %x\n",
HAL_BUFFER_OUTPUT);
+ mutex_unlock(&inst->lock);
return;
}
bufreq->buffer_count_min = event_notify->capture_buf_count;
diff --git a/drivers/media/platform/msm/vidc/msm_vidc_debug.h b/drivers/media/platform/msm/vidc/msm_vidc_debug.h
index dbebe5d..50bef1c 100644
--- a/drivers/media/platform/msm/vidc/msm_vidc_debug.h
+++ b/drivers/media/platform/msm/vidc/msm_vidc_debug.h
@@ -96,6 +96,12 @@ extern bool msm_vidc_syscache_disable;
__str; \
})
+#if defined(CONFIG_TRACING) && defined(DEBUG)
+#define msm_trace_printk(...) trace_printk(__VA_ARGS__)
+#else
+#define msm_trace_printk(...)
+#endif
+
#define dprintk(__level, __fmt, arg...) \
do { \
if (msm_vidc_debug & __level) { \
@@ -104,7 +110,7 @@ extern bool msm_vidc_syscache_disable;
VIDC_MSG_PRIO2STRING(__level), \
## arg); \
} else if (msm_vidc_debug_out == VIDC_OUT_FTRACE) { \
- trace_printk(KERN_DEBUG VIDC_DBG_TAG __fmt, \
+ msm_trace_printk(KERN_DEBUG VIDC_DBG_TAG __fmt, \
VIDC_MSG_PRIO2STRING(__level), \
## arg); \
} \
diff --git a/drivers/media/platform/msm/vidc/venus_hfi.h b/drivers/media/platform/msm/vidc/venus_hfi.h
index 4c4cb06..4da908a 100644
--- a/drivers/media/platform/msm/vidc/venus_hfi.h
+++ b/drivers/media/platform/msm/vidc/venus_hfi.h
@@ -235,7 +235,7 @@ struct venus_hfi_device {
struct msm_vidc_gov_data bus_vote;
bool power_enabled;
struct mutex lock;
- msm_vidc_callback callback;
+ hfi_cmd_response_callback callback;
struct vidc_mem_addr iface_q_table;
struct vidc_mem_addr qdss;
struct vidc_mem_addr sfr;
diff --git a/drivers/media/platform/omap3isp/isp.c b/drivers/media/platform/omap3isp/isp.c
index 0321d84..a21b12c5 100644
--- a/drivers/media/platform/omap3isp/isp.c
+++ b/drivers/media/platform/omap3isp/isp.c
@@ -304,7 +304,7 @@ static struct clk *isp_xclk_src_get(struct of_phandle_args *clkspec, void *data)
static int isp_xclk_init(struct isp_device *isp)
{
struct device_node *np = isp->dev->of_node;
- struct clk_init_data init;
+ struct clk_init_data init = { 0 };
unsigned int i;
for (i = 0; i < ARRAY_SIZE(isp->xclks); ++i)
@@ -1591,6 +1591,8 @@ static void isp_pm_complete(struct device *dev)
static void isp_unregister_entities(struct isp_device *isp)
{
+ media_device_unregister(&isp->media_dev);
+
omap3isp_csi2_unregister_entities(&isp->isp_csi2a);
omap3isp_ccp2_unregister_entities(&isp->isp_ccp2);
omap3isp_ccdc_unregister_entities(&isp->isp_ccdc);
@@ -1601,7 +1603,6 @@ static void isp_unregister_entities(struct isp_device *isp)
omap3isp_stat_unregister_entities(&isp->isp_hist);
v4l2_device_unregister(&isp->v4l2_dev);
- media_device_unregister(&isp->media_dev);
media_device_cleanup(&isp->media_dev);
}
@@ -1941,6 +1942,7 @@ static int isp_initialize_modules(struct isp_device *isp)
static void isp_detach_iommu(struct isp_device *isp)
{
+ arm_iommu_detach_device(isp->dev);
arm_iommu_release_mapping(isp->mapping);
isp->mapping = NULL;
iommu_group_remove_device(isp->dev);
@@ -1974,8 +1976,7 @@ static int isp_attach_iommu(struct isp_device *isp)
mapping = arm_iommu_create_mapping(&platform_bus_type, SZ_1G, SZ_2G);
if (IS_ERR(mapping)) {
dev_err(isp->dev, "failed to create ARM IOMMU mapping\n");
- ret = PTR_ERR(mapping);
- goto error;
+ return PTR_ERR(mapping);
}
isp->mapping = mapping;
@@ -1990,7 +1991,8 @@ static int isp_attach_iommu(struct isp_device *isp)
return 0;
error:
- isp_detach_iommu(isp);
+ arm_iommu_release_mapping(isp->mapping);
+ isp->mapping = NULL;
return ret;
}
diff --git a/drivers/media/platform/rcar_jpu.c b/drivers/media/platform/rcar_jpu.c
index d1746ec..db1110a 100644
--- a/drivers/media/platform/rcar_jpu.c
+++ b/drivers/media/platform/rcar_jpu.c
@@ -1280,7 +1280,7 @@ static int jpu_open(struct file *file)
/* ...issue software reset */
ret = jpu_reset(jpu);
if (ret)
- goto device_prepare_rollback;
+ goto jpu_reset_rollback;
}
jpu->ref_count++;
@@ -1288,6 +1288,8 @@ static int jpu_open(struct file *file)
mutex_unlock(&jpu->mutex);
return 0;
+jpu_reset_rollback:
+ clk_disable_unprepare(jpu->clk);
device_prepare_rollback:
mutex_unlock(&jpu->mutex);
v4l_prepare_rollback:
diff --git a/drivers/media/platform/s3c-camif/camif-capture.c b/drivers/media/platform/s3c-camif/camif-capture.c
index 0413a86..d9710b5 100644
--- a/drivers/media/platform/s3c-camif/camif-capture.c
+++ b/drivers/media/platform/s3c-camif/camif-capture.c
@@ -117,6 +117,8 @@ static int sensor_set_power(struct camif_dev *camif, int on)
if (camif->sensor.power_count == !on)
err = v4l2_subdev_call(sensor->sd, core, s_power, on);
+ if (err == -ENOIOCTLCMD)
+ err = 0;
if (!err)
sensor->power_count += on ? 1 : -1;
@@ -1256,16 +1258,17 @@ static void __camif_subdev_try_format(struct camif_dev *camif,
{
const struct s3c_camif_variant *variant = camif->variant;
const struct vp_pix_limits *pix_lim;
- int i = ARRAY_SIZE(camif_mbus_formats);
+ unsigned int i;
/* FIXME: constraints against codec or preview path ? */
pix_lim = &variant->vp_pix_limits[VP_CODEC];
- while (i-- >= 0)
+ for (i = 0; i < ARRAY_SIZE(camif_mbus_formats); i++)
if (camif_mbus_formats[i] == mf->code)
break;
- mf->code = camif_mbus_formats[i];
+ if (i == ARRAY_SIZE(camif_mbus_formats))
+ mf->code = camif_mbus_formats[0];
if (pad == CAMIF_SD_PAD_SINK) {
v4l_bound_align_image(&mf->width, 8, CAMIF_MAX_PIX_WIDTH,
diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc.c b/drivers/media/platform/s5p-mfc/s5p_mfc.c
index 7c24da51..8051c13 100644
--- a/drivers/media/platform/s5p-mfc/s5p_mfc.c
+++ b/drivers/media/platform/s5p-mfc/s5p_mfc.c
@@ -249,24 +249,24 @@ static void s5p_mfc_handle_frame_all_extracted(struct s5p_mfc_ctx *ctx)
static void s5p_mfc_handle_frame_copy_time(struct s5p_mfc_ctx *ctx)
{
struct s5p_mfc_dev *dev = ctx->dev;
- struct s5p_mfc_buf *dst_buf, *src_buf;
- size_t dec_y_addr;
+ struct s5p_mfc_buf *dst_buf, *src_buf;
+ u32 dec_y_addr;
unsigned int frame_type;
/* Make sure we actually have a new frame before continuing. */
frame_type = s5p_mfc_hw_call(dev->mfc_ops, get_dec_frame_type, dev);
if (frame_type == S5P_FIMV_DECODE_FRAME_SKIPPED)
return;
- dec_y_addr = s5p_mfc_hw_call(dev->mfc_ops, get_dec_y_adr, dev);
+ dec_y_addr = (u32)s5p_mfc_hw_call(dev->mfc_ops, get_dec_y_adr, dev);
/* Copy timestamp / timecode from decoded src to dst and set
appropriate flags. */
src_buf = list_entry(ctx->src_queue.next, struct s5p_mfc_buf, list);
list_for_each_entry(dst_buf, &ctx->dst_queue, list) {
- if (vb2_dma_contig_plane_dma_addr(&dst_buf->b->vb2_buf, 0)
- == dec_y_addr) {
- dst_buf->b->timecode =
- src_buf->b->timecode;
+ u32 addr = (u32)vb2_dma_contig_plane_dma_addr(&dst_buf->b->vb2_buf, 0);
+
+ if (addr == dec_y_addr) {
+ dst_buf->b->timecode = src_buf->b->timecode;
dst_buf->b->vb2_buf.timestamp =
src_buf->b->vb2_buf.timestamp;
dst_buf->b->flags &=
@@ -302,10 +302,10 @@ static void s5p_mfc_handle_frame_new(struct s5p_mfc_ctx *ctx, unsigned int err)
{
struct s5p_mfc_dev *dev = ctx->dev;
struct s5p_mfc_buf *dst_buf;
- size_t dspl_y_addr;
+ u32 dspl_y_addr;
unsigned int frame_type;
- dspl_y_addr = s5p_mfc_hw_call(dev->mfc_ops, get_dspl_y_adr, dev);
+ dspl_y_addr = (u32)s5p_mfc_hw_call(dev->mfc_ops, get_dspl_y_adr, dev);
if (IS_MFCV6_PLUS(dev))
frame_type = s5p_mfc_hw_call(dev->mfc_ops,
get_disp_frame_type, ctx);
@@ -324,9 +324,10 @@ static void s5p_mfc_handle_frame_new(struct s5p_mfc_ctx *ctx, unsigned int err)
/* The MFC returns address of the buffer, now we have to
* check which videobuf does it correspond to */
list_for_each_entry(dst_buf, &ctx->dst_queue, list) {
+ u32 addr = (u32)vb2_dma_contig_plane_dma_addr(&dst_buf->b->vb2_buf, 0);
+
/* Check if this is the buffer we're looking for */
- if (vb2_dma_contig_plane_dma_addr(&dst_buf->b->vb2_buf, 0)
- == dspl_y_addr) {
+ if (addr == dspl_y_addr) {
list_del(&dst_buf->list);
ctx->dst_queue_cnt--;
dst_buf->b->sequence = ctx->sequence;
diff --git a/drivers/media/platform/vivid/vivid-ctrls.c b/drivers/media/platform/vivid/vivid-ctrls.c
index aceb38d..b1c3725 100644
--- a/drivers/media/platform/vivid/vivid-ctrls.c
+++ b/drivers/media/platform/vivid/vivid-ctrls.c
@@ -1167,6 +1167,7 @@ static int vivid_radio_rx_s_ctrl(struct v4l2_ctrl *ctrl)
v4l2_ctrl_activate(dev->radio_rx_rds_ta, dev->radio_rx_rds_controls);
v4l2_ctrl_activate(dev->radio_rx_rds_tp, dev->radio_rx_rds_controls);
v4l2_ctrl_activate(dev->radio_rx_rds_ms, dev->radio_rx_rds_controls);
+ dev->radio_rx_dev.device_caps = dev->radio_rx_caps;
break;
case V4L2_CID_RDS_RECEPTION:
dev->radio_rx_rds_enabled = ctrl->val;
@@ -1241,6 +1242,7 @@ static int vivid_radio_tx_s_ctrl(struct v4l2_ctrl *ctrl)
dev->radio_tx_caps &= ~V4L2_CAP_READWRITE;
if (!dev->radio_tx_rds_controls)
dev->radio_tx_caps |= V4L2_CAP_READWRITE;
+ dev->radio_tx_dev.device_caps = dev->radio_tx_caps;
break;
case V4L2_CID_RDS_TX_PTY:
if (dev->radio_rx_rds_controls)
diff --git a/drivers/media/platform/vivid/vivid-vid-cap.c b/drivers/media/platform/vivid/vivid-vid-cap.c
index d5c84ec..25d4fd4 100644
--- a/drivers/media/platform/vivid/vivid-vid-cap.c
+++ b/drivers/media/platform/vivid/vivid-vid-cap.c
@@ -452,6 +452,8 @@ void vivid_update_format_cap(struct vivid_dev *dev, bool keep_controls)
tpg_s_rgb_range(&dev->tpg, v4l2_ctrl_g_ctrl(dev->rgb_range_cap));
break;
}
+ vfree(dev->bitmap_cap);
+ dev->bitmap_cap = NULL;
vivid_update_quality(dev);
tpg_reset_source(&dev->tpg, dev->src_rect.width, dev->src_rect.height, dev->field_cap);
dev->crop_cap = dev->src_rect;
diff --git a/drivers/media/radio/si470x/radio-si470x-i2c.c b/drivers/media/radio/si470x/radio-si470x-i2c.c
index ee0470a..f218886 100644
--- a/drivers/media/radio/si470x/radio-si470x-i2c.c
+++ b/drivers/media/radio/si470x/radio-si470x-i2c.c
@@ -96,7 +96,7 @@ MODULE_PARM_DESC(max_rds_errors, "RDS maximum block errors: *1*");
*/
int si470x_get_register(struct si470x_device *radio, int regnr)
{
- u16 buf[READ_REG_NUM];
+ __be16 buf[READ_REG_NUM];
struct i2c_msg msgs[1] = {
{
.addr = radio->client->addr,
@@ -121,7 +121,7 @@ int si470x_get_register(struct si470x_device *radio, int regnr)
int si470x_set_register(struct si470x_device *radio, int regnr)
{
int i;
- u16 buf[WRITE_REG_NUM];
+ __be16 buf[WRITE_REG_NUM];
struct i2c_msg msgs[1] = {
{
.addr = radio->client->addr,
@@ -151,7 +151,7 @@ int si470x_set_register(struct si470x_device *radio, int regnr)
static int si470x_get_all_registers(struct si470x_device *radio)
{
int i;
- u16 buf[READ_REG_NUM];
+ __be16 buf[READ_REG_NUM];
struct i2c_msg msgs[1] = {
{
.addr = radio->client->addr,
diff --git a/drivers/media/usb/cx231xx/cx231xx-cards.c b/drivers/media/usb/cx231xx/cx231xx-cards.c
index 921cf1e..69156aff 100644
--- a/drivers/media/usb/cx231xx/cx231xx-cards.c
+++ b/drivers/media/usb/cx231xx/cx231xx-cards.c
@@ -864,6 +864,9 @@ struct usb_device_id cx231xx_id_table[] = {
.driver_info = CX231XX_BOARD_CNXT_RDE_250},
{USB_DEVICE(0x0572, 0x58A0),
.driver_info = CX231XX_BOARD_CNXT_RDU_250},
+ /* AverMedia DVD EZMaker 7 */
+ {USB_DEVICE(0x07ca, 0xc039),
+ .driver_info = CX231XX_BOARD_CNXT_VIDEO_GRABBER},
{USB_DEVICE(0x2040, 0xb110),
.driver_info = CX231XX_BOARD_HAUPPAUGE_USB2_FM_PAL},
{USB_DEVICE(0x2040, 0xb111),
diff --git a/drivers/media/usb/dvb-usb-v2/af9035.c b/drivers/media/usb/dvb-usb-v2/af9035.c
index 8961dd7..64be30d 100644
--- a/drivers/media/usb/dvb-usb-v2/af9035.c
+++ b/drivers/media/usb/dvb-usb-v2/af9035.c
@@ -406,8 +406,10 @@ static int af9035_i2c_master_xfer(struct i2c_adapter *adap,
msg[0].addr == (state->af9033_i2c_addr[1] >> 1))
reg |= 0x100000;
- ret = af9035_wr_regs(d, reg, &msg[0].buf[3],
- msg[0].len - 3);
+ ret = (msg[0].len >= 3) ? af9035_wr_regs(d, reg,
+ &msg[0].buf[3],
+ msg[0].len - 3)
+ : -EOPNOTSUPP;
} else {
/* I2C write */
u8 buf[MAX_XFER_SIZE];
diff --git a/drivers/media/usb/em28xx/em28xx-cards.c b/drivers/media/usb/em28xx/em28xx-cards.c
index e397f54..e9403aa 100644
--- a/drivers/media/usb/em28xx/em28xx-cards.c
+++ b/drivers/media/usb/em28xx/em28xx-cards.c
@@ -2093,13 +2093,13 @@ struct em28xx_board em28xx_boards[] = {
.input = { {
.type = EM28XX_VMUX_COMPOSITE,
.vmux = TVP5150_COMPOSITE1,
- .amux = EM28XX_AUDIO_SRC_LINE,
+ .amux = EM28XX_AMUX_LINE_IN,
.gpio = terratec_av350_unmute_gpio,
}, {
.type = EM28XX_VMUX_SVIDEO,
.vmux = TVP5150_SVIDEO,
- .amux = EM28XX_AUDIO_SRC_LINE,
+ .amux = EM28XX_AMUX_LINE_IN,
.gpio = terratec_av350_unmute_gpio,
} },
},
diff --git a/drivers/media/usb/em28xx/em28xx-dvb.c b/drivers/media/usb/em28xx/em28xx-dvb.c
index 8cedef0..b0aea48 100644
--- a/drivers/media/usb/em28xx/em28xx-dvb.c
+++ b/drivers/media/usb/em28xx/em28xx-dvb.c
@@ -2016,6 +2016,8 @@ static int em28xx_dvb_fini(struct em28xx *dev)
}
}
+ em28xx_unregister_dvb(dvb);
+
/* remove I2C SEC */
client = dvb->i2c_client_sec;
if (client) {
@@ -2037,7 +2039,6 @@ static int em28xx_dvb_fini(struct em28xx *dev)
i2c_unregister_device(client);
}
- em28xx_unregister_dvb(dvb);
kfree(dvb);
dev->dvb = NULL;
kref_put(&dev->ref, em28xx_free_device);
diff --git a/drivers/media/usb/em28xx/em28xx-video.c b/drivers/media/usb/em28xx/em28xx-video.c
index 1f7fa05..1ed7ba3 100644
--- a/drivers/media/usb/em28xx/em28xx-video.c
+++ b/drivers/media/usb/em28xx/em28xx-video.c
@@ -1290,6 +1290,8 @@ static void em28xx_ctrl_notify(struct v4l2_ctrl *ctrl, void *priv)
{
struct em28xx *dev = priv;
+ dev->v4l2->field_count = 0;
+
/*
* In the case of non-AC97 volume controls, we still need
* to do some setups at em28xx, in order to mute/unmute
@@ -1435,9 +1437,9 @@ static int vidioc_try_fmt_vid_cap(struct file *file, void *priv,
fmt = format_by_fourcc(f->fmt.pix.pixelformat);
if (!fmt) {
- em28xx_videodbg("Fourcc format (%08x) invalid.\n",
- f->fmt.pix.pixelformat);
- return -EINVAL;
+ fmt = &format[0];
+ em28xx_videodbg("Fourcc format (%08x) invalid. Using default (%08x).\n",
+ f->fmt.pix.pixelformat, fmt->fourcc);
}
if (dev->board.is_em2800) {
diff --git a/drivers/media/usb/em28xx/em28xx.h b/drivers/media/usb/em28xx/em28xx.h
index d148463..6bf48a7 100644
--- a/drivers/media/usb/em28xx/em28xx.h
+++ b/drivers/media/usb/em28xx/em28xx.h
@@ -189,7 +189,7 @@
USB 2.0 spec says bulk packet size is always 512 bytes
*/
#define EM28XX_BULK_PACKET_MULTIPLIER 384
-#define EM28XX_DVB_BULK_PACKET_MULTIPLIER 384
+#define EM28XX_DVB_BULK_PACKET_MULTIPLIER 94
#define EM28XX_INTERLACED_DEFAULT 1
diff --git a/drivers/media/usb/stkwebcam/stk-sensor.c b/drivers/media/usb/stkwebcam/stk-sensor.c
index e546b01..2dcc8d0 100644
--- a/drivers/media/usb/stkwebcam/stk-sensor.c
+++ b/drivers/media/usb/stkwebcam/stk-sensor.c
@@ -228,7 +228,7 @@
static int stk_sensor_outb(struct stk_camera *dev, u8 reg, u8 val)
{
int i = 0;
- int tmpval = 0;
+ u8 tmpval = 0;
if (stk_camera_write_reg(dev, STK_IIC_TX_INDEX, reg))
return 1;
@@ -253,7 +253,7 @@ static int stk_sensor_outb(struct stk_camera *dev, u8 reg, u8 val)
static int stk_sensor_inb(struct stk_camera *dev, u8 reg, u8 *val)
{
int i = 0;
- int tmpval = 0;
+ u8 tmpval = 0;
if (stk_camera_write_reg(dev, STK_IIC_RX_INDEX, reg))
return 1;
@@ -274,7 +274,7 @@ static int stk_sensor_inb(struct stk_camera *dev, u8 reg, u8 *val)
if (stk_camera_read_reg(dev, STK_IIC_RX_VALUE, &tmpval))
return 1;
- *val = (u8) tmpval;
+ *val = tmpval;
return 0;
}
diff --git a/drivers/media/usb/stkwebcam/stk-webcam.c b/drivers/media/usb/stkwebcam/stk-webcam.c
index 22a9aae..1c48f2f 100644
--- a/drivers/media/usb/stkwebcam/stk-webcam.c
+++ b/drivers/media/usb/stkwebcam/stk-webcam.c
@@ -144,7 +144,7 @@ int stk_camera_write_reg(struct stk_camera *dev, u16 index, u8 value)
return 0;
}
-int stk_camera_read_reg(struct stk_camera *dev, u16 index, int *value)
+int stk_camera_read_reg(struct stk_camera *dev, u16 index, u8 *value)
{
struct usb_device *udev = dev->udev;
unsigned char *buf;
@@ -163,7 +163,7 @@ int stk_camera_read_reg(struct stk_camera *dev, u16 index, int *value)
sizeof(u8),
500);
if (ret >= 0)
- memcpy(value, buf, sizeof(u8));
+ *value = *buf;
kfree(buf);
return ret;
@@ -171,9 +171,10 @@ int stk_camera_read_reg(struct stk_camera *dev, u16 index, int *value)
static int stk_start_stream(struct stk_camera *dev)
{
- int value;
+ u8 value;
int i, ret;
- int value_116, value_117;
+ u8 value_116, value_117;
+
if (!is_present(dev))
return -ENODEV;
@@ -213,7 +214,7 @@ static int stk_start_stream(struct stk_camera *dev)
static int stk_stop_stream(struct stk_camera *dev)
{
- int value;
+ u8 value;
int i;
if (is_present(dev)) {
stk_camera_read_reg(dev, 0x0100, &value);
diff --git a/drivers/media/usb/stkwebcam/stk-webcam.h b/drivers/media/usb/stkwebcam/stk-webcam.h
index 9bbfa3d..92bb48e 100644
--- a/drivers/media/usb/stkwebcam/stk-webcam.h
+++ b/drivers/media/usb/stkwebcam/stk-webcam.h
@@ -129,7 +129,7 @@ struct stk_camera {
#define vdev_to_camera(d) container_of(d, struct stk_camera, vdev)
int stk_camera_write_reg(struct stk_camera *, u16, u8);
-int stk_camera_read_reg(struct stk_camera *, u16, int *);
+int stk_camera_read_reg(struct stk_camera *, u16, u8 *);
int stk_sensor_init(struct stk_camera *);
int stk_sensor_configure(struct stk_camera *);
diff --git a/drivers/media/usb/tm6000/tm6000-dvb.c b/drivers/media/usb/tm6000/tm6000-dvb.c
index 0426b21..ee88ae8 100644
--- a/drivers/media/usb/tm6000/tm6000-dvb.c
+++ b/drivers/media/usb/tm6000/tm6000-dvb.c
@@ -273,6 +273,11 @@ static int register_dvb(struct tm6000_core *dev)
ret = dvb_register_adapter(&dvb->adapter, "Trident TVMaster 6000 DVB-T",
THIS_MODULE, &dev->udev->dev, adapter_nr);
+ if (ret < 0) {
+ pr_err("tm6000: couldn't register the adapter!\n");
+ goto err;
+ }
+
dvb->adapter.priv = dev;
if (dvb->frontend) {
diff --git a/drivers/media/usb/uvc/uvc_video.c b/drivers/media/usb/uvc/uvc_video.c
index b5589d5..48503f30 100644
--- a/drivers/media/usb/uvc/uvc_video.c
+++ b/drivers/media/usb/uvc/uvc_video.c
@@ -163,14 +163,27 @@ static void uvc_fixup_video_ctrl(struct uvc_streaming *stream,
}
}
+static size_t uvc_video_ctrl_size(struct uvc_streaming *stream)
+{
+ /*
+ * Return the size of the video probe and commit controls, which depends
+ * on the protocol version.
+ */
+ if (stream->dev->uvc_version < 0x0110)
+ return 26;
+ else if (stream->dev->uvc_version < 0x0150)
+ return 34;
+ else
+ return 48;
+}
+
static int uvc_get_video_ctrl(struct uvc_streaming *stream,
struct uvc_streaming_control *ctrl, int probe, __u8 query)
{
+ __u16 size = uvc_video_ctrl_size(stream);
__u8 *data;
- __u16 size;
int ret;
- size = stream->dev->uvc_version >= 0x0110 ? 34 : 26;
if ((stream->dev->quirks & UVC_QUIRK_PROBE_DEF) &&
query == UVC_GET_DEF)
return -EIO;
@@ -225,7 +238,7 @@ static int uvc_get_video_ctrl(struct uvc_streaming *stream,
ctrl->dwMaxVideoFrameSize = get_unaligned_le32(&data[18]);
ctrl->dwMaxPayloadTransferSize = get_unaligned_le32(&data[22]);
- if (size == 34) {
+ if (size >= 34) {
ctrl->dwClockFrequency = get_unaligned_le32(&data[26]);
ctrl->bmFramingInfo = data[30];
ctrl->bPreferedVersion = data[31];
@@ -254,11 +267,10 @@ static int uvc_get_video_ctrl(struct uvc_streaming *stream,
static int uvc_set_video_ctrl(struct uvc_streaming *stream,
struct uvc_streaming_control *ctrl, int probe)
{
+ __u16 size = uvc_video_ctrl_size(stream);
__u8 *data;
- __u16 size;
int ret;
- size = stream->dev->uvc_version >= 0x0110 ? 34 : 26;
data = kzalloc(size, GFP_KERNEL);
if (data == NULL)
return -ENOMEM;
@@ -275,7 +287,7 @@ static int uvc_set_video_ctrl(struct uvc_streaming *stream,
put_unaligned_le32(ctrl->dwMaxVideoFrameSize, &data[18]);
put_unaligned_le32(ctrl->dwMaxPayloadTransferSize, &data[22]);
- if (size == 34) {
+ if (size >= 34) {
put_unaligned_le32(ctrl->dwClockFrequency, &data[26]);
data[30] = ctrl->bmFramingInfo;
data[31] = ctrl->bPreferedVersion;
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index d4e93f1..cb9596f 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -395,9 +395,7 @@ static int get_v4l2_plane32(struct v4l2_plane __user *up,
copy_in_user(&up->data_offset, &up32->data_offset,
sizeof(up->data_offset)) ||
copy_in_user(up->reserved, up32->reserved,
- sizeof(up->reserved)) ||
- copy_in_user(&up->length, &up32->length,
- sizeof(up->length)))
+ sizeof(up->reserved)))
return -EFAULT;
switch (memory) {
@@ -870,7 +868,7 @@ static int put_v4l2_ext_controls32(struct file *file,
get_user(kcontrols, &kp->controls))
return -EFAULT;
- if (!count)
+ if (!count || count > (U32_MAX/sizeof(*ucontrols)))
return 0;
if (get_user(p, &up->controls))
return -EFAULT;
diff --git a/drivers/media/v4l2-core/v4l2-event.c b/drivers/media/v4l2-core/v4l2-event.c
index 8d3171c..1fda287 100644
--- a/drivers/media/v4l2-core/v4l2-event.c
+++ b/drivers/media/v4l2-core/v4l2-event.c
@@ -119,14 +119,6 @@ static void __v4l2_event_queue_fh(struct v4l2_fh *fh, const struct v4l2_event *e
if (sev == NULL)
return;
- /*
- * If the event has been added to the fh->subscribed list, but its
- * add op has not completed yet elems will be 0, treat this as
- * not being subscribed.
- */
- if (!sev->elems)
- return;
-
/* Increase event sequence number on fh. */
fh->sequence++;
@@ -205,6 +197,22 @@ int v4l2_event_pending(struct v4l2_fh *fh)
}
EXPORT_SYMBOL_GPL(v4l2_event_pending);
+static void __v4l2_event_unsubscribe(struct v4l2_subscribed_event *sev)
+{
+ struct v4l2_fh *fh = sev->fh;
+ unsigned int i;
+
+ lockdep_assert_held(&fh->subscribe_lock);
+ assert_spin_locked(&fh->vdev->fh_lock);
+
+ /* Remove any pending events for this subscription */
+ for (i = 0; i < sev->in_use; i++) {
+ list_del(&sev->events[sev_pos(sev, i)].list);
+ fh->navailable--;
+ }
+ list_del(&sev->list);
+}
+
int v4l2_event_subscribe(struct v4l2_fh *fh,
const struct v4l2_event_subscription *sub, unsigned elems,
const struct v4l2_subscribed_event_ops *ops)
@@ -212,6 +220,7 @@ int v4l2_event_subscribe(struct v4l2_fh *fh,
struct v4l2_subscribed_event *sev, *found_ev;
unsigned long flags;
unsigned i;
+ int ret = 0;
if (sub->type == V4L2_EVENT_ALL)
return -EINVAL;
@@ -229,6 +238,9 @@ int v4l2_event_subscribe(struct v4l2_fh *fh,
sev->flags = sub->flags;
sev->fh = fh;
sev->ops = ops;
+ sev->elems = elems;
+
+ mutex_lock(&fh->subscribe_lock);
spin_lock_irqsave(&fh->vdev->fh_lock, flags);
found_ev = v4l2_event_subscribed(fh, sub->type, sub->id);
@@ -237,23 +249,21 @@ int v4l2_event_subscribe(struct v4l2_fh *fh,
spin_unlock_irqrestore(&fh->vdev->fh_lock, flags);
if (found_ev) {
+ /* Already listening */
kfree(sev);
- return 0; /* Already listening */
- }
-
- if (sev->ops && sev->ops->add) {
- int ret = sev->ops->add(sev, elems);
+ } else if (sev->ops && sev->ops->add) {
+ ret = sev->ops->add(sev, elems);
if (ret) {
- sev->ops = NULL;
- v4l2_event_unsubscribe(fh, sub);
- return ret;
+ spin_lock_irqsave(&fh->vdev->fh_lock, flags);
+ __v4l2_event_unsubscribe(sev);
+ spin_unlock_irqrestore(&fh->vdev->fh_lock, flags);
+ kfree(sev);
}
}
- /* Mark as ready for use */
- sev->elems = elems;
+ mutex_unlock(&fh->subscribe_lock);
- return 0;
+ return ret;
}
EXPORT_SYMBOL_GPL(v4l2_event_subscribe);
@@ -285,30 +295,27 @@ int v4l2_event_unsubscribe(struct v4l2_fh *fh,
{
struct v4l2_subscribed_event *sev;
unsigned long flags;
- int i;
if (sub->type == V4L2_EVENT_ALL) {
v4l2_event_unsubscribe_all(fh);
return 0;
}
+ mutex_lock(&fh->subscribe_lock);
+
spin_lock_irqsave(&fh->vdev->fh_lock, flags);
sev = v4l2_event_subscribed(fh, sub->type, sub->id);
- if (sev != NULL) {
- /* Remove any pending events for this subscription */
- for (i = 0; i < sev->in_use; i++) {
- list_del(&sev->events[sev_pos(sev, i)].list);
- fh->navailable--;
- }
- list_del(&sev->list);
- }
+ if (sev != NULL)
+ __v4l2_event_unsubscribe(sev);
spin_unlock_irqrestore(&fh->vdev->fh_lock, flags);
if (sev && sev->ops && sev->ops->del)
sev->ops->del(sev);
+ mutex_unlock(&fh->subscribe_lock);
+
kfree(sev);
return 0;
diff --git a/drivers/media/v4l2-core/v4l2-fh.c b/drivers/media/v4l2-core/v4l2-fh.c
index c183f09..0c5e690 100644
--- a/drivers/media/v4l2-core/v4l2-fh.c
+++ b/drivers/media/v4l2-core/v4l2-fh.c
@@ -50,6 +50,7 @@ void v4l2_fh_init(struct v4l2_fh *fh, struct video_device *vdev)
INIT_LIST_HEAD(&fh->available);
INIT_LIST_HEAD(&fh->subscribed);
fh->sequence = -1;
+ mutex_init(&fh->subscribe_lock);
}
EXPORT_SYMBOL_GPL(v4l2_fh_init);
@@ -95,6 +96,7 @@ void v4l2_fh_exit(struct v4l2_fh *fh)
return;
v4l_disable_media_source(fh->vdev);
v4l2_event_unsubscribe_all(fh);
+ mutex_destroy(&fh->subscribe_lock);
fh->vdev = NULL;
}
EXPORT_SYMBOL_GPL(v4l2_fh_exit);
diff --git a/drivers/media/v4l2-core/videobuf2-core.c b/drivers/media/v4l2-core/videobuf2-core.c
index 4299ce0..4df4a1f 100644
--- a/drivers/media/v4l2-core/videobuf2-core.c
+++ b/drivers/media/v4l2-core/videobuf2-core.c
@@ -1372,6 +1372,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
struct vb2_buffer *vb;
int ret;
+ if (q->error) {
+ dprintk(1, "fatal error occurred on queue\n");
+ return -EIO;
+ }
+
vb = q->bufs[index];
switch (vb->state) {
diff --git a/drivers/media/v4l2-core/videobuf2-vmalloc.c b/drivers/media/v4l2-core/videobuf2-vmalloc.c
index ab3227b..760cbf2 100644
--- a/drivers/media/v4l2-core/videobuf2-vmalloc.c
+++ b/drivers/media/v4l2-core/videobuf2-vmalloc.c
@@ -104,7 +104,7 @@ static void *vb2_vmalloc_get_userptr(struct device *dev, unsigned long vaddr,
if (nums[i-1] + 1 != nums[i])
goto fail_map;
buf->vaddr = (__force void *)
- ioremap_nocache(nums[0] << PAGE_SHIFT, size);
+ ioremap_nocache(__pfn_to_phys(nums[0]), size + offset);
} else {
buf->vaddr = vm_map_ram(frame_vector_pages(vec), n_pages, -1,
PAGE_KERNEL);
diff --git a/drivers/memory/tegra/mc.c b/drivers/memory/tegra/mc.c
index a4803ac..1d49a8d 100644
--- a/drivers/memory/tegra/mc.c
+++ b/drivers/memory/tegra/mc.c
@@ -20,14 +20,6 @@
#include "mc.h"
#define MC_INTSTATUS 0x000
-#define MC_INT_DECERR_MTS (1 << 16)
-#define MC_INT_SECERR_SEC (1 << 13)
-#define MC_INT_DECERR_VPR (1 << 12)
-#define MC_INT_INVALID_APB_ASID_UPDATE (1 << 11)
-#define MC_INT_INVALID_SMMU_PAGE (1 << 10)
-#define MC_INT_ARBITRATION_EMEM (1 << 9)
-#define MC_INT_SECURITY_VIOLATION (1 << 8)
-#define MC_INT_DECERR_EMEM (1 << 6)
#define MC_INTMASK 0x004
@@ -248,12 +240,13 @@ static const char *const error_names[8] = {
static irqreturn_t tegra_mc_irq(int irq, void *data)
{
struct tegra_mc *mc = data;
- unsigned long status, mask;
+ unsigned long status;
unsigned int bit;
/* mask all interrupts to avoid flooding */
- status = mc_readl(mc, MC_INTSTATUS);
- mask = mc_readl(mc, MC_INTMASK);
+ status = mc_readl(mc, MC_INTSTATUS) & mc->soc->intmask;
+ if (!status)
+ return IRQ_NONE;
for_each_set_bit(bit, &status, 32) {
const char *error = status_names[bit] ?: "unknown";
@@ -346,7 +339,6 @@ static int tegra_mc_probe(struct platform_device *pdev)
const struct of_device_id *match;
struct resource *res;
struct tegra_mc *mc;
- u32 value;
int err;
match = of_match_node(tegra_mc_of_match, pdev->dev.of_node);
@@ -414,11 +406,7 @@ static int tegra_mc_probe(struct platform_device *pdev)
WARN(!mc->soc->client_id_mask, "Missing client ID mask for this SoC\n");
- value = MC_INT_DECERR_MTS | MC_INT_SECERR_SEC | MC_INT_DECERR_VPR |
- MC_INT_INVALID_APB_ASID_UPDATE | MC_INT_INVALID_SMMU_PAGE |
- MC_INT_SECURITY_VIOLATION | MC_INT_DECERR_EMEM;
-
- mc_writel(mc, value, MC_INTMASK);
+ mc_writel(mc, mc->soc->intmask, MC_INTMASK);
return 0;
}
diff --git a/drivers/memory/tegra/mc.h b/drivers/memory/tegra/mc.h
index ddb1667..24e020b 100644
--- a/drivers/memory/tegra/mc.h
+++ b/drivers/memory/tegra/mc.h
@@ -14,6 +14,15 @@
#include <soc/tegra/mc.h>
+#define MC_INT_DECERR_MTS (1 << 16)
+#define MC_INT_SECERR_SEC (1 << 13)
+#define MC_INT_DECERR_VPR (1 << 12)
+#define MC_INT_INVALID_APB_ASID_UPDATE (1 << 11)
+#define MC_INT_INVALID_SMMU_PAGE (1 << 10)
+#define MC_INT_ARBITRATION_EMEM (1 << 9)
+#define MC_INT_SECURITY_VIOLATION (1 << 8)
+#define MC_INT_DECERR_EMEM (1 << 6)
+
static inline u32 mc_readl(struct tegra_mc *mc, unsigned long offset)
{
return readl(mc->regs + offset);
diff --git a/drivers/memory/tegra/tegra114.c b/drivers/memory/tegra/tegra114.c
index ba8fff3..6d2a5a8 100644
--- a/drivers/memory/tegra/tegra114.c
+++ b/drivers/memory/tegra/tegra114.c
@@ -930,4 +930,6 @@ const struct tegra_mc_soc tegra114_mc_soc = {
.atom_size = 32,
.client_id_mask = 0x7f,
.smmu = &tegra114_smmu_soc,
+ .intmask = MC_INT_INVALID_SMMU_PAGE | MC_INT_SECURITY_VIOLATION |
+ MC_INT_DECERR_EMEM,
};
diff --git a/drivers/memory/tegra/tegra124.c b/drivers/memory/tegra/tegra124.c
index 5a58e44..9f68a56 100644
--- a/drivers/memory/tegra/tegra124.c
+++ b/drivers/memory/tegra/tegra124.c
@@ -1020,6 +1020,9 @@ const struct tegra_mc_soc tegra124_mc_soc = {
.smmu = &tegra124_smmu_soc,
.emem_regs = tegra124_mc_emem_regs,
.num_emem_regs = ARRAY_SIZE(tegra124_mc_emem_regs),
+ .intmask = MC_INT_DECERR_MTS | MC_INT_SECERR_SEC | MC_INT_DECERR_VPR |
+ MC_INT_INVALID_APB_ASID_UPDATE | MC_INT_INVALID_SMMU_PAGE |
+ MC_INT_SECURITY_VIOLATION | MC_INT_DECERR_EMEM,
};
#endif /* CONFIG_ARCH_TEGRA_124_SOC */
@@ -1042,5 +1045,8 @@ const struct tegra_mc_soc tegra132_mc_soc = {
.atom_size = 32,
.client_id_mask = 0x7f,
.smmu = &tegra132_smmu_soc,
+ .intmask = MC_INT_DECERR_MTS | MC_INT_SECERR_SEC | MC_INT_DECERR_VPR |
+ MC_INT_INVALID_APB_ASID_UPDATE | MC_INT_INVALID_SMMU_PAGE |
+ MC_INT_SECURITY_VIOLATION | MC_INT_DECERR_EMEM,
};
#endif /* CONFIG_ARCH_TEGRA_132_SOC */
diff --git a/drivers/memory/tegra/tegra210.c b/drivers/memory/tegra/tegra210.c
index 5e144ab..47c78a6 100644
--- a/drivers/memory/tegra/tegra210.c
+++ b/drivers/memory/tegra/tegra210.c
@@ -1077,4 +1077,7 @@ const struct tegra_mc_soc tegra210_mc_soc = {
.atom_size = 64,
.client_id_mask = 0xff,
.smmu = &tegra210_smmu_soc,
+ .intmask = MC_INT_DECERR_MTS | MC_INT_SECERR_SEC | MC_INT_DECERR_VPR |
+ MC_INT_INVALID_APB_ASID_UPDATE | MC_INT_INVALID_SMMU_PAGE |
+ MC_INT_SECURITY_VIOLATION | MC_INT_DECERR_EMEM,
};
diff --git a/drivers/memory/tegra/tegra30.c b/drivers/memory/tegra/tegra30.c
index b447378..d068942 100644
--- a/drivers/memory/tegra/tegra30.c
+++ b/drivers/memory/tegra/tegra30.c
@@ -952,4 +952,6 @@ const struct tegra_mc_soc tegra30_mc_soc = {
.atom_size = 16,
.client_id_mask = 0x7f,
.smmu = &tegra30_smmu_soc,
+ .intmask = MC_INT_INVALID_SMMU_PAGE | MC_INT_SECURITY_VIOLATION |
+ MC_INT_DECERR_EMEM,
};
diff --git a/drivers/message/fusion/mptctl.c b/drivers/message/fusion/mptctl.c
index 02b5f69..14cf6df 100644
--- a/drivers/message/fusion/mptctl.c
+++ b/drivers/message/fusion/mptctl.c
@@ -2698,6 +2698,8 @@ mptctl_hp_targetinfo(unsigned long arg)
__FILE__, __LINE__, iocnum);
return -ENODEV;
}
+ if (karg.hdr.id >= MPT_MAX_FC_DEVICES)
+ return -EINVAL;
dctlprintk(ioc, printk(MYIOC_s_DEBUG_FMT "mptctl_hp_targetinfo called.\n",
ioc->name));
diff --git a/drivers/message/fusion/mptsas.c b/drivers/message/fusion/mptsas.c
index 7ee1667..00dff9b 100644
--- a/drivers/message/fusion/mptsas.c
+++ b/drivers/message/fusion/mptsas.c
@@ -1994,6 +1994,7 @@ static struct scsi_host_template mptsas_driver_template = {
.cmd_per_lun = 7,
.use_clustering = ENABLE_CLUSTERING,
.shost_attrs = mptscsih_host_attrs,
+ .no_write_same = 1,
};
static int mptsas_get_linkerrors(struct sas_phy *phy)
diff --git a/drivers/mfd/88pm860x-i2c.c b/drivers/mfd/88pm860x-i2c.c
index 84e3131..7b9052e 100644
--- a/drivers/mfd/88pm860x-i2c.c
+++ b/drivers/mfd/88pm860x-i2c.c
@@ -146,14 +146,14 @@ int pm860x_page_reg_write(struct i2c_client *i2c, int reg,
unsigned char zero;
int ret;
- i2c_lock_adapter(i2c->adapter);
+ i2c_lock_bus(i2c->adapter, I2C_LOCK_SEGMENT);
read_device(i2c, 0xFA, 0, &zero);
read_device(i2c, 0xFB, 0, &zero);
read_device(i2c, 0xFF, 0, &zero);
ret = write_device(i2c, reg, 1, &data);
read_device(i2c, 0xFE, 0, &zero);
read_device(i2c, 0xFC, 0, &zero);
- i2c_unlock_adapter(i2c->adapter);
+ i2c_unlock_bus(i2c->adapter, I2C_LOCK_SEGMENT);
return ret;
}
EXPORT_SYMBOL(pm860x_page_reg_write);
@@ -164,14 +164,14 @@ int pm860x_page_bulk_read(struct i2c_client *i2c, int reg,
unsigned char zero = 0;
int ret;
- i2c_lock_adapter(i2c->adapter);
+ i2c_lock_bus(i2c->adapter, I2C_LOCK_SEGMENT);
read_device(i2c, 0xfa, 0, &zero);
read_device(i2c, 0xfb, 0, &zero);
read_device(i2c, 0xff, 0, &zero);
ret = read_device(i2c, reg, count, buf);
read_device(i2c, 0xFE, 0, &zero);
read_device(i2c, 0xFC, 0, &zero);
- i2c_unlock_adapter(i2c->adapter);
+ i2c_unlock_bus(i2c->adapter, I2C_LOCK_SEGMENT);
return ret;
}
EXPORT_SYMBOL(pm860x_page_bulk_read);
diff --git a/drivers/mfd/cros_ec.c b/drivers/mfd/cros_ec.c
index abd8342..3e18d25 100644
--- a/drivers/mfd/cros_ec.c
+++ b/drivers/mfd/cros_ec.c
@@ -86,7 +86,11 @@ int cros_ec_register(struct cros_ec_device *ec_dev)
mutex_init(&ec_dev->lock);
- cros_ec_query_all(ec_dev);
+ err = cros_ec_query_all(ec_dev);
+ if (err) {
+ dev_err(dev, "Cannot identify the EC: error %d\n", err);
+ return err;
+ }
if (ec_dev->irq) {
err = request_threaded_irq(ec_dev->irq, NULL, ec_irq_thread,
diff --git a/drivers/mfd/hi655x-pmic.c b/drivers/mfd/hi655x-pmic.c
index 0fc6299..11347a3 100644
--- a/drivers/mfd/hi655x-pmic.c
+++ b/drivers/mfd/hi655x-pmic.c
@@ -49,7 +49,7 @@ static struct regmap_config hi655x_regmap_config = {
.reg_bits = 32,
.reg_stride = HI655X_STRIDE,
.val_bits = 8,
- .max_register = HI655X_BUS_ADDR(0xFFF),
+ .max_register = HI655X_BUS_ADDR(0x400) - HI655X_STRIDE,
};
static struct resource pwrkey_resources[] = {
diff --git a/drivers/mfd/intel-lpss.c b/drivers/mfd/intel-lpss.c
index 70c646b..19ac8bc 100644
--- a/drivers/mfd/intel-lpss.c
+++ b/drivers/mfd/intel-lpss.c
@@ -275,11 +275,11 @@ static void intel_lpss_init_dev(const struct intel_lpss *lpss)
intel_lpss_deassert_reset(lpss);
+ intel_lpss_set_remap_addr(lpss);
+
if (!intel_lpss_has_idma(lpss))
return;
- intel_lpss_set_remap_addr(lpss);
-
/* Make sure that SPI multiblock DMA transfers are re-enabled */
if (lpss->type == LPSS_DEV_SPI)
writel(value, lpss->priv + LPSS_PRIV_SSP_REG);
diff --git a/drivers/mfd/omap-usb-host.c b/drivers/mfd/omap-usb-host.c
index 7aab376..3785c63 100644
--- a/drivers/mfd/omap-usb-host.c
+++ b/drivers/mfd/omap-usb-host.c
@@ -548,8 +548,8 @@ static int usbhs_omap_get_dt_pdata(struct device *dev,
}
static const struct of_device_id usbhs_child_match_table[] = {
- { .compatible = "ti,omap-ehci", },
- { .compatible = "ti,omap-ohci", },
+ { .compatible = "ti,ehci-omap", },
+ { .compatible = "ti,ohci-omap3", },
{ }
};
@@ -875,6 +875,7 @@ static struct platform_driver usbhs_omap_driver = {
.pm = &usbhsomap_dev_pm_ops,
.of_match_table = usbhs_omap_dt_ids,
},
+ .probe = usbhs_omap_probe,
.remove = usbhs_omap_remove,
};
@@ -884,9 +885,9 @@ MODULE_ALIAS("platform:" USBHS_DRIVER_NAME);
MODULE_LICENSE("GPL v2");
MODULE_DESCRIPTION("usb host common core driver for omap EHCI and OHCI");
-static int __init omap_usbhs_drvinit(void)
+static int omap_usbhs_drvinit(void)
{
- return platform_driver_probe(&usbhs_omap_driver, usbhs_omap_probe);
+ return platform_driver_register(&usbhs_omap_driver);
}
/*
@@ -898,7 +899,7 @@ static int __init omap_usbhs_drvinit(void)
*/
fs_initcall_sync(omap_usbhs_drvinit);
-static void __exit omap_usbhs_drvexit(void)
+static void omap_usbhs_drvexit(void)
{
platform_driver_unregister(&usbhs_omap_driver);
}
diff --git a/drivers/mfd/qcom-i2c-pmic.c b/drivers/mfd/qcom-i2c-pmic.c
index 28c3343..6dbc272 100644
--- a/drivers/mfd/qcom-i2c-pmic.c
+++ b/drivers/mfd/qcom-i2c-pmic.c
@@ -438,7 +438,7 @@ static irqreturn_t i2c_pmic_irq_handler(int irq, void *dev_id)
static int i2c_pmic_parse_dt(struct i2c_pmic *chip)
{
struct device_node *node = chip->dev->of_node;
- int rc, i;
+ int rc = 0, i;
u32 temp;
if (!node) {
diff --git a/drivers/mfd/sm501.c b/drivers/mfd/sm501.c
index 4053435..3270b8d 100644
--- a/drivers/mfd/sm501.c
+++ b/drivers/mfd/sm501.c
@@ -714,6 +714,7 @@ sm501_create_subdev(struct sm501_devdata *sm, char *name,
smdev->pdev.name = name;
smdev->pdev.id = sm->pdev_id;
smdev->pdev.dev.parent = sm->dev;
+ smdev->pdev.dev.coherent_dma_mask = 0xffffffff;
if (res_count) {
smdev->pdev.resource = (struct resource *)(smdev+1);
diff --git a/drivers/mfd/ti_am335x_tscadc.c b/drivers/mfd/ti_am335x_tscadc.c
index c8f027b..798f0a8 100644
--- a/drivers/mfd/ti_am335x_tscadc.c
+++ b/drivers/mfd/ti_am335x_tscadc.c
@@ -209,14 +209,13 @@ static int ti_tscadc_probe(struct platform_device *pdev)
* The TSC_ADC_SS controller design assumes the OCP clock is
* at least 6x faster than the ADC clock.
*/
- clk = clk_get(&pdev->dev, "adc_tsc_fck");
+ clk = devm_clk_get(&pdev->dev, "adc_tsc_fck");
if (IS_ERR(clk)) {
dev_err(&pdev->dev, "failed to get TSC fck\n");
err = PTR_ERR(clk);
goto err_disable_clk;
}
clock_rate = clk_get_rate(clk);
- clk_put(clk);
tscadc->clk_div = clock_rate / ADC_CLK;
/* TSCADC_CLKDIV needs to be configured to the value minus 1 */
diff --git a/drivers/misc/Kconfig b/drivers/misc/Kconfig
index e3f4c39..5b6b46b 100644
--- a/drivers/misc/Kconfig
+++ b/drivers/misc/Kconfig
@@ -813,6 +813,53 @@
and the driver provides an API to check if this interrupt
is available on the current PMIC chip.
+config NFC_ST21NFC
+ tristate "ST Microelectronics ST21NFC NFC Controller Driver"
+ depends on I2C
+ ---help---
+ ST Microelectronics ST21NFC Near Field Communication controller support
+
+config CITADEL
+ tristate "Google Citadel"
+ depends on SPI
+ ---help---
+ Say Y here if you have a Citadel security microcontroller from Google Inc.
+ It will be accessible using a TPM-style transport interface from within
+ Linux.
+
+ To compile this driver as a module, choose M here: the
+ module will be called citadel-spi.
+
+config GOOGLE_EASEL
+ tristate "Google Easel/Paintbox Coprocessor Support"
+ help
+ Say 'y' if platform has the Paintbox IPU on an Easel
+ coprocessor, aka MonetteHill. This config is used to enable the
+ communications driver support, but can also be used for future
+ driver work.
+
+config GOOGLE_EASEL_AP
+ bool "Local system is Application Processor / Android"
+ depends on GOOGLE_EASEL
+ help
+ Say 'y' if this kernel is for the AP/Android side of the link. Say
+ 'n' if this kernel is for the Easel/MonetteHill side of the link.
+
+config CS40L2X_HAPTIC
+ tristate "Cirrus Logic CS40L2X haptics support"
+ depends on I2C
+ select REGMAP_I2C
+ help
+ Say Y to enable support for the CS CS40L2X haptics driver.
+
+ To compile this driver as a module, choose M here.
+
+config ACCESS_RAMOOPS
+ tristate "Driver to allow access for decrypting encrypted ramoops"
+ depends on PSTORE_CONSOLE
+ help
+ Say 'y' if ramoops is encrypted.
+
source "drivers/misc/c2port/Kconfig"
source "drivers/misc/eeprom/Kconfig"
source "drivers/misc/cb710/Kconfig"
@@ -825,4 +872,6 @@
source "drivers/misc/genwqe/Kconfig"
source "drivers/misc/echo/Kconfig"
source "drivers/misc/cxl/Kconfig"
+source "drivers/misc/mnh/Kconfig"
+source "drivers/misc/st33/Kconfig"
endmenu
diff --git a/drivers/misc/Makefile b/drivers/misc/Makefile
index f1c9467..0e48309 100644
--- a/drivers/misc/Makefile
+++ b/drivers/misc/Makefile
@@ -33,6 +33,7 @@
obj-$(CONFIG_TI_DAC7512) += ti_dac7512.o
obj-$(CONFIG_C2PORT) += c2port/
obj-$(CONFIG_HMC6352) += hmc6352.o
+obj-$(CONFIG_CS40L2X_HAPTIC) += cs40l2x.o cs40l2x-tables.o
obj-y += eeprom/
obj-y += cb710/
obj-$(CONFIG_SPEAR13XX_PCIE_GADGET) += spear13xx_pcie_gadget.o
@@ -62,6 +63,10 @@
obj-$(CONFIG_UID_SYS_STATS) += uid_sys_stats.o
obj-$(CONFIG_MEMORY_STATE_TIME) += memory_state_time.o
+obj-$(CONFIG_NFC_ST21NFC) += st21nfc.o
+
+obj-$(CONFIG_CITADEL) += citadel-spi.o
+
lkdtm-$(CONFIG_LKDTM) += lkdtm_core.o
lkdtm-$(CONFIG_LKDTM) += lkdtm_bugs.o
lkdtm-$(CONFIG_LKDTM) += lkdtm_heap.o
@@ -69,6 +74,12 @@
lkdtm-$(CONFIG_LKDTM) += lkdtm_rodata_objcopy.o
lkdtm-$(CONFIG_LKDTM) += lkdtm_usercopy.o
+obj-y += mnh/
+obj-$(CONFIG_GOOGLE_EASEL) += google-easel.o
+google-easel-objs += google-easel-comm.o google-easel-comm-dma.o google-easel-comm-hw.o
+
+obj-y += st33/
+obj-$(CONFIG_ACCESS_RAMOOPS) += access_ramoops.o
CFLAGS_lkdtm_rodata.o += $(DISABLE_LTO)
KCOV_INSTRUMENT_lkdtm_rodata.o := n
@@ -79,3 +90,8 @@
targets += lkdtm_rodata.o lkdtm_rodata_objcopy.o
$(obj)/lkdtm_rodata_objcopy.o: $(obj)/lkdtm_rodata.o FORCE
$(call if_changed,objcopy)
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-header-guard \
+ -Wno-ignored-attributes \
diff --git a/drivers/misc/access_ramoops.c b/drivers/misc/access_ramoops.c
new file mode 100644
index 0000000..0f8e66e
--- /dev/null
+++ b/drivers/misc/access_ramoops.c
@@ -0,0 +1,250 @@
+/*
+ * Persistent memory accessor
+ *
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/io.h>
+#include <linux/fs.h>
+#include <linux/platform_device.h>
+#include <linux/miscdevice.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+
+#define DEVICE_NAME "access_ramoops"
+#define MAX_OPEN 4
+
+struct access_ramoops_info {
+ char *name;
+ phys_addr_t phys;
+ void *addr;
+ size_t size;
+ struct mutex lock;
+ int is_mapped;
+ int is_open;
+
+ const char *label;
+ struct miscdevice miscdev;
+};
+
+static ssize_t label_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct miscdevice *miscdev = dev_get_drvdata(dev);
+ struct access_ramoops_info *info =
+ container_of(miscdev, struct access_ramoops_info, miscdev);
+
+ return snprintf(buf, PAGE_SIZE, "%s\n", info->label);
+}
+
+static ssize_t size_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct miscdevice *miscdev = dev_get_drvdata(dev);
+ struct access_ramoops_info *info =
+ container_of(miscdev, struct access_ramoops_info, miscdev);
+
+ return snprintf(buf, PAGE_SIZE, "%zd\n", info->size);
+}
+
+static DEVICE_ATTR_RO(label);
+static DEVICE_ATTR_RO(size);
+
+static struct attribute *access_ramoops_attrs[] = {
+ &dev_attr_label.attr,
+ &dev_attr_size.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(access_ramoops);
+
+static ssize_t access_ramoops_read(struct file *filp, char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ struct access_ramoops_info *info = filp->private_data;
+
+ return simple_read_from_buffer(buf, count, ppos,
+ info->addr, info->size);
+}
+
+static ssize_t access_ramoops_write(struct file *filp, const char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ struct access_ramoops_info *info = filp->private_data;
+
+ return simple_write_to_buffer(info->addr, info->size,
+ ppos, buf, count);
+}
+
+static int access_ramoops_open(struct inode *inode, struct file *filp)
+{
+ int res = 0;
+ struct access_ramoops_info *info =
+ container_of(filp->private_data, struct access_ramoops_info,
+ miscdev);
+
+ filp->private_data = info;
+ mutex_lock(&info->lock);
+
+ if (info->is_open == MAX_OPEN) {
+ res = -EBUSY;
+ goto out;
+ }
+
+ if (!info->is_open) {
+ if (!request_mem_region(info->phys, info->size,
+ "access_ramoops")) {
+ dev_err(info->miscdev.this_device,
+ "request mem region (%zx@%pa) failed\n",
+ info->size, &info->phys);
+ res = -ENOMEM;
+ goto out;
+ }
+ info->addr = ioremap(info->phys, info->size);
+ if (IS_ERR(info->addr)) {
+ dev_err(info->miscdev.this_device,
+ "unable to map region (%zx@%pa)\n",
+ info->size, &info->phys);
+ info->addr = NULL;
+ res = -ENOMEM;
+ goto out;
+ }
+ info->is_mapped = 1;
+ }
+ info->is_open++;
+out:
+ mutex_unlock(&info->lock);
+ return res;
+}
+
+static int access_ramoops_release(struct inode *inode, struct file *filp)
+{
+ struct access_ramoops_info *info = filp->private_data;
+
+ mutex_lock(&info->lock);
+ info->is_open--;
+ if (!info->is_open) {
+ info->is_mapped = 0;
+ iounmap(info->addr);
+ info->addr = NULL;
+ release_mem_region(info->phys, info->size);
+ }
+ filp->private_data = NULL;
+ mutex_unlock(&info->lock);
+ return 0;
+}
+
+static struct file_operations access_ramoops_fops = {
+ .owner = THIS_MODULE,
+ .open = access_ramoops_open,
+ .read = access_ramoops_read,
+ .write = access_ramoops_write,
+ .release = access_ramoops_release,
+};
+
+static int __init access_ramoops_probe(struct platform_device *pdev)
+{
+ int ret = 0;
+ struct access_ramoops_info *info;
+ struct device_node *of_node = pdev->dev.of_node;
+ struct device_node *mem_region;
+ struct resource res;
+
+ info = devm_kzalloc(&pdev->dev, sizeof(*info), GFP_KERNEL);
+ if (!info)
+ return -ENOMEM;
+ platform_set_drvdata(pdev, info);
+
+ mem_region = of_parse_phandle(of_node, "memory-region", 0);
+ if (!mem_region) {
+ dev_err(&pdev->dev, "no memory-region phandle\n");
+ return -ENODEV;
+ }
+
+ ret = of_property_read_string(of_node, "label", &info->label);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to get region label: %d\n", ret);
+ return -EINVAL;
+ }
+ info->name = devm_kasprintf(&pdev->dev, GFP_KERNEL, "access-%s",
+ info->label);
+ if (!info->name) {
+ dev_err(&pdev->dev, "failed to alloc name\n");
+ return -ENOMEM;
+ }
+
+ ret = of_address_to_resource(mem_region, 0, &res);
+ of_node_put(mem_region);
+ if (ret) {
+ dev_err(&pdev->dev,
+ "failed to get memory-region resource: %d\n", ret);
+ return -ENOMEM;
+ }
+
+ info->phys = res.start;
+ info->size = resource_size(&res);
+ mutex_init(&info->lock);
+
+ info->miscdev.minor = MISC_DYNAMIC_MINOR;
+ info->miscdev.name = info->name;
+ info->miscdev.fops = &access_ramoops_fops;
+ info->miscdev.groups = access_ramoops_groups;
+ info->miscdev.parent = &pdev->dev;
+ ret = misc_register(&info->miscdev);
+ if (ret) {
+ dev_err(info->miscdev.this_device,
+ "failed to register misc device %d\n", ret);
+ return ret;
+ }
+
+ dev_info(info->miscdev.this_device,
+ "registered '%s' %d:%d, (%zx@%pa)\n", info->label,
+ MISC_MAJOR, info->miscdev.minor, info->size, &info->phys);
+
+ return ret;
+}
+
+static int __exit access_ramoops_remove(struct platform_device *pdev)
+{
+ struct access_ramoops_info *info = platform_get_drvdata(pdev);
+
+ dev_info(info->miscdev.this_device, "removing");
+ misc_deregister(&info->miscdev);
+
+ if (!info->is_mapped)
+ return 0;
+
+ info->is_mapped = 0;
+ iounmap(info->addr);
+ release_mem_region(info->phys, info->size);
+ return 0;
+}
+
+static const struct of_device_id dt_match[] = {
+ { .compatible = "access_ramoops" },
+ {}
+};
+
+static struct platform_driver access_ramoops_driver = {
+ .driver = {
+ .name = "access_ramoops",
+ .of_match_table = dt_match,
+ },
+ .remove = __exit_p(access_ramoops_remove),
+};
+
+module_platform_driver_probe(access_ramoops_driver, access_ramoops_probe);
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Patrick Tjin <pattjin@google.com>");
+MODULE_DESCRIPTION("Persistent memory access driver");
diff --git a/drivers/misc/atmel-ssc.c b/drivers/misc/atmel-ssc.c
index 0516ecd..3a6e4ec 100644
--- a/drivers/misc/atmel-ssc.c
+++ b/drivers/misc/atmel-ssc.c
@@ -130,7 +130,7 @@ static const struct of_device_id atmel_ssc_dt_ids[] = {
MODULE_DEVICE_TABLE(of, atmel_ssc_dt_ids);
#endif
-static inline const struct atmel_ssc_platform_data * __init
+static inline const struct atmel_ssc_platform_data *
atmel_ssc_get_driver_data(struct platform_device *pdev)
{
if (pdev->dev.of_node) {
diff --git a/drivers/misc/citadel-spi.c b/drivers/misc/citadel-spi.c
new file mode 100644
index 0000000..b1d16cf
--- /dev/null
+++ b/drivers/misc/citadel-spi.c
@@ -0,0 +1,539 @@
+/*
+ * Citadel transport driver
+ *
+ * Copyright (C) 2017 Fernando Lugo <flugo@google.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/interrupt.h>
+#include <linux/ioctl.h>
+#include <linux/fs.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/err.h>
+#include <linux/slab.h>
+#include <linux/of.h>
+#include <linux/of_device.h>
+#include <linux/of_gpio.h>
+#include <linux/poll.h>
+#include <linux/uaccess.h>
+#include <linux/cdev.h>
+#include <linux/spi/spi.h>
+#include <linux/citadel.h>
+#include <linux/wait.h>
+
+#define CITADEL_TPM_READ 0x80000000
+
+#define MAX_DATA_SIZE 2044
+#define CITADEL_MAX_DEVICES 4
+#define CITADEL_TPM_TIMEOUT_MS 10
+
+struct citadel_data {
+ dev_t devt;
+ struct cdev cdev;
+ struct spi_device *spi;
+ int ctdl_ap_irq;
+ wait_queue_head_t waitq;
+ int ctdl_rst;
+ atomic_t users;
+ void *tx_buf;
+ void *rx_buf;
+};
+
+static struct class *citadel_class;
+static dev_t citadel_devt;
+
+static int citadel_wait_cmd_done(struct citadel_data *citadel)
+{
+ struct spi_device *spi = citadel->spi;
+ struct spi_message m;
+ int ret;
+ unsigned long to = jiffies + 1 + /* at least one jiffy */
+ msecs_to_jiffies(CITADEL_TPM_TIMEOUT_MS);
+ struct spi_transfer spi_xfer = {
+ .rx_buf = citadel->rx_buf,
+ .len = 1,
+ .cs_change = 1,
+ };
+ uint8_t *val = citadel->rx_buf;
+
+ /*
+ * We have sent the initial four-byte command to Citadel on MOSI, and
+ * now we're waiting for bit0 of the MISO byte to be set, indicating
+ * that we can continue with the rest of the transaction. If that bit
+ * is not immediately set, we'll keep sending one more don't-care byte
+ * on MOSI just to read MISO until it is. If Citadel is awake and
+ * functioning correctly its hardware implementation should always
+ * return 0x00 while it's thinking and return 0x01 when ready to
+ * continue. However, there are a couple of ways this can fail. First,
+ * if Citadel is in deep sleep, it should send 0x5A on MISO until it
+ * wakes up. This may take ~40ms or so but it's not unexpected, so if
+ * we time out we just give up and try again. Second, if Citadel
+ * unexpectedly reboots, it will probably return 0xFF or 0xDF, which
+ * will exit the loop but we shouldn't continue the transaction because
+ * Citadel won't know what's going on.
+ */
+ do {
+ if (time_after(jiffies, to)) {
+ dev_warn(&spi->dev, "Citadel SPI timed out\n");
+ return -EBUSY;
+ }
+ spi_message_init(&m);
+ spi_message_add_tail(&spi_xfer, &m);
+ ret = spi_sync_locked(spi, &m);
+ if (ret)
+ return ret;
+ } while (!*val);
+
+ /* Return EAGAIN if unexpected bytes were received. */
+ return *val & 0x01 ? 0 : -EAGAIN;
+}
+
+static int citadel_tpm_datagram(struct citadel_data *citadel,
+ struct citadel_ioc_tpm_datagram *dg)
+{
+ int is_read;
+ int citadel_is_awake;
+ int ret;
+ int ignore_result = 0;
+ struct spi_device *spi = citadel->spi;
+ struct spi_message m;
+ struct spi_transfer spi_xfer = {
+ .tx_buf = citadel->tx_buf,
+ .rx_buf = citadel->rx_buf,
+ .len = 4,
+ .cs_change = 1,
+ };
+ uint32_t *command = citadel->tx_buf;
+ uint32_t *response = citadel->rx_buf;
+
+ /* Read == from SPI, to userland. */
+ is_read = dg->command & CITADEL_TPM_READ;
+
+ /* Lock the SPI bus until we're completely done */
+ spi_bus_lock(spi->master);
+
+ /* The command must be big-endian */
+ *command = cpu_to_be32(dg->command);
+
+ /* Prepare to send the command */
+ spi_message_init(&m);
+ spi_message_add_tail(&spi_xfer, &m);
+
+ /* Send the command out */
+ ret = spi_sync_locked(spi, &m);
+ if (ret)
+ goto exit;
+
+ /* Verify that citadel is idle. If it isn't, deassert CS and return
+ * -EAGAIN. 0xdf is what citadel sends when the SPI FIFO is empty,
+ * and it is in the TPM wait mode. Once a command is sent to Citadel,
+ * the last bit shows whether or not Citadel is ready to send the
+ * response. This typically happens by a 0x01 byte, but may be
+ * preceded by several 0x00 bytes while Citadel does any necessary
+ * work. */
+ citadel_is_awake = *response == be32_to_cpu(0xdfdfdfde);
+
+ if (citadel_is_awake) {
+ /* Wait for the reply bit to go high */
+ ret = citadel_wait_cmd_done(citadel);
+ /* Reset CS in the case of unexpected bytes. */
+ if (ret)
+ citadel_is_awake = 0;
+ }
+
+ /* TODO: If there's no data, how do we disable the CS? */
+ if (!dg->len || !citadel_is_awake) {
+ /* For now, just transfer one more byte and throw it away */
+ is_read = 1;
+ dg->len = 1;
+ ignore_result = 1;
+ }
+
+ /* Now we can transfer the data */
+ spi_xfer.cs_change = 0;
+ spi_xfer.len = dg->len;
+ if (is_read) {
+ spi_xfer.rx_buf = citadel->rx_buf;
+ spi_xfer.tx_buf = NULL;
+ } else {
+ spi_xfer.rx_buf = NULL;
+ spi_xfer.tx_buf = citadel->tx_buf;
+
+ if (copy_from_user(citadel->tx_buf,
+ (void __user *)dg->buf, dg->len)) {
+ ret = -EFAULT;
+ goto exit;
+ }
+ }
+
+ spi_message_init(&m);
+ spi_message_add_tail(&spi_xfer, &m);
+ ret = spi_sync_locked(spi, &m);
+ if (ret)
+ goto exit;
+
+ /* This condition typically happens when citadel is asleep. Toggling
+ * CS should be sufficient to wake up citadel, but some time needs to
+ * pass before it will be ready. */
+ if (!citadel_is_awake)
+ ret = -EAGAIN;
+
+ if (ignore_result)
+ goto exit;
+
+ if (is_read && copy_to_user((void __user *)dg->buf,
+ citadel->rx_buf, dg->len))
+ ret = -EFAULT;
+
+exit:
+ spi_bus_unlock(spi->master);
+
+ return ret;
+}
+
+static int citadel_reset(struct citadel_data *citadel)
+{
+ /* Synchronize with the datagrams by locking the SPI bus */
+ struct spi_device *spi = citadel->spi;
+ spi_bus_lock(spi->master);
+
+ /* Assert reset for at least 3ms after VDDIOM is stable; 10ms is safe */
+ gpio_set_value(citadel->ctdl_rst, 1);
+ msleep(10);
+
+ /* Clear reset and wait for Citadel to become functional */
+ gpio_set_value(citadel->ctdl_rst, 0);
+ msleep(100);
+
+ spi_bus_unlock(spi->master);
+ return 0;
+}
+
+static long
+citadel_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
+{
+ u32 tmp;
+ struct citadel_data *citadel = filp->private_data;
+ struct citadel_ioc_tpm_datagram dg;
+
+ /* check magic */
+ if (_IOC_TYPE(cmd) != CITADEL_IOC_MAGIC)
+ return -ENOTTY;
+ switch (cmd) {
+ case CITADEL_IOC_TPM_DATAGRAM:
+ tmp = _IOC_SIZE(cmd);
+ if (tmp != sizeof(dg))
+ return -EINVAL;
+
+ if (copy_from_user(&dg, (void __user *)arg, sizeof(dg)))
+ return -EFAULT;
+
+ if (dg.len > MAX_DATA_SIZE)
+ return -E2BIG;
+
+ /* translate to spi_message, execute */
+ return citadel_tpm_datagram(citadel, &dg);
+ case CITADEL_IOC_RESET:
+ return citadel_reset(citadel);
+ }
+ return -EINVAL;
+}
+
+static ssize_t
+citadel_read(struct file *filp, char __user *buf, size_t count, loff_t *f_pos)
+{
+ int ret;
+ size_t c = 0;
+ size_t len;
+ struct citadel_data *citadel = filp->private_data;
+
+ while (count) {
+ len = count > MAX_DATA_SIZE ? MAX_DATA_SIZE : count;
+ ret = spi_read(citadel->spi, citadel->rx_buf, len);
+ if (ret)
+ return ret;
+ ret = copy_to_user(buf + c, citadel->rx_buf, len);
+ if (ret)
+ return -EFAULT;
+ c += len;
+ count -= len;
+ }
+
+ return c;
+}
+
+static ssize_t
+citadel_write(struct file *filp, const char __user *buf,
+ size_t count, loff_t *f_pos)
+{
+ int ret;
+ size_t c = 0;
+ size_t len;
+ struct citadel_data *citadel = filp->private_data;
+
+ while (count) {
+ len = count > MAX_DATA_SIZE ? MAX_DATA_SIZE : count;
+ ret = copy_from_user(citadel->tx_buf, buf + c, len);
+ if (ret)
+ return -EFAULT;
+ ret = spi_write(citadel->spi, citadel->tx_buf, len);
+ if (ret)
+ return ret;
+ c += len;
+ count -= len;
+ }
+
+ return c;
+}
+
+static int citadel_open(struct inode *inode, struct file *filp)
+{
+ struct citadel_data *citadel;
+
+ citadel = container_of(inode->i_cdev, struct citadel_data, cdev);
+
+ /* we only support 1 user at the same time */
+ if (!atomic_add_unless(&citadel->users, 1, 1))
+ return -EBUSY;
+
+ filp->private_data = citadel;
+ nonseekable_open(inode, filp);
+
+ return 0;
+}
+
+static unsigned int citadel_poll(struct file *filp, poll_table *wait)
+{
+ struct citadel_data *citadel = filp->private_data;
+ poll_wait(filp, &citadel->waitq, wait);
+ return gpio_get_value(citadel->ctdl_ap_irq) ? POLLIN : 0;
+}
+
+static int citadel_release(struct inode *inode, struct file *filp)
+{
+ struct citadel_data *citadel = filp->private_data;
+
+ atomic_dec(&citadel->users);
+
+ return 0;
+}
+
+static const struct file_operations citadel_fops = {
+ .owner = THIS_MODULE,
+ .write = citadel_write,
+ .read = citadel_read,
+ .open = citadel_open,
+ .poll = citadel_poll,
+ .release = citadel_release,
+ .unlocked_ioctl = citadel_ioctl,
+ .llseek = no_llseek,
+};
+
+#ifdef CONFIG_OF
+static const struct of_device_id citadel_dt_ids[] = {
+ { .compatible = "google,citadel" },
+ {},
+};
+MODULE_DEVICE_TABLE(of, citadel_dt_ids);
+#endif
+
+static irqreturn_t citadel_irq_handler(int irq, void *handle)
+{
+ struct citadel_data *citadel = (struct citadel_data *)handle;
+ wake_up_interruptible(&citadel->waitq);
+ return IRQ_HANDLED;
+}
+
+static int citadel_request_named_gpio(struct citadel_data *citadel,
+ const char *label, int *gpio)
+{
+ struct device *dev = &citadel->spi->dev;
+ struct device_node *np = dev->of_node;
+ int rc = of_get_named_gpio(np, label, 0);
+
+ if (rc < 0) {
+ dev_err(dev, "failed to get '%s'\n", label);
+ return rc;
+ }
+ *gpio = rc;
+
+ rc = devm_gpio_request(dev, *gpio, label);
+ if (rc) {
+ dev_err(dev, "failed to request gpio %d\n", *gpio);
+ return rc;
+ }
+ dev_dbg(dev, "%s %d\n", label, *gpio);
+
+ return 0;
+}
+
+static int citadel_probe(struct spi_device *spi)
+{
+ struct device *dev;
+ struct cdev cdev;
+ struct citadel_data *citadel;
+ int ret;
+ dev_t devt;
+ u32 minor;
+
+ /* use chip select as minor */
+ minor = (u32)spi->chip_select;
+ if (minor >= CITADEL_MAX_DEVICES) {
+ dev_err(&spi->dev, "minor %u out of boundaries\n", minor);
+ return -ENXIO;
+ }
+
+ citadel = kmalloc(sizeof(*citadel), GFP_KERNEL);
+ if (!citadel)
+ return -ENOMEM;
+
+ /* init citadel structure */
+ citadel->tx_buf = (void *)__get_free_pages(GFP_KERNEL,
+ get_order(MAX_DATA_SIZE));
+ citadel->rx_buf = (void *)__get_free_pages(GFP_KERNEL,
+ get_order(MAX_DATA_SIZE));
+ if (!citadel->tx_buf || !citadel->rx_buf) {
+ ret = -ENOMEM;
+ goto free_citadel;
+ }
+ init_waitqueue_head(&citadel->waitq);
+ atomic_set(&citadel->users, 0);
+ citadel->spi = spi;
+ devt = MKDEV(MAJOR(citadel_devt), minor);
+ citadel->devt = devt;
+ spi_set_drvdata(spi, citadel);
+
+ /* setup ctdl_ap_irq */
+ ret = citadel_request_named_gpio(citadel, "citadel,ctdl_ap_irq",
+ &citadel->ctdl_ap_irq);
+ if (ret) {
+ dev_err(&spi->dev,
+ "citadel_request_named_gpio "
+ "citadel,ctdl_ap_irq failed.\n");
+ goto free_citadel;
+ }
+
+ ret = devm_request_irq(&citadel->spi->dev,
+ gpio_to_irq(citadel->ctdl_ap_irq),
+ citadel_irq_handler,
+ IRQF_NO_SUSPEND | IRQF_TRIGGER_RISING |
+ IRQF_ONESHOT,
+ dev_name(&spi->dev),
+ citadel);
+ if (ret) {
+ dev_err(&spi->dev,
+ "devm_request_irq citadel,ctdl_ap_irq failed.\n");
+ goto free_citadel;
+ }
+
+ enable_irq_wake(gpio_to_irq(citadel->ctdl_ap_irq));
+
+ /* setup ctdl_rst */
+ ret = citadel_request_named_gpio(citadel, "citadel,ctdl_rst",
+ &citadel->ctdl_rst);
+ if (ret) {
+ dev_err(&spi->dev,
+ "citadel_request_named_gpio "
+ "citadel,ctdl_rst failed.\n");
+ goto free_citadel;
+ }
+
+ gpio_direction_output(citadel->ctdl_rst, 0);
+
+ /* create the device */
+ dev = device_create(citadel_class, &spi->dev, devt, NULL,
+ "citadel%u", minor);
+ if (IS_ERR(dev)) {
+ ret = PTR_ERR(dev);
+ goto free_citadel;
+ }
+
+ cdev_init(&citadel->cdev, &citadel_fops);
+ cdev.owner = THIS_MODULE;
+ ret = cdev_add(&citadel->cdev, citadel_devt, 1);
+ if (ret)
+ goto destroy_device;
+
+ return 0;
+
+destroy_device:
+ device_destroy(citadel_class, devt);
+free_citadel:
+ free_pages((unsigned long)citadel->rx_buf, get_order(MAX_DATA_SIZE));
+ free_pages((unsigned long)citadel->tx_buf, get_order(MAX_DATA_SIZE));
+ kfree(citadel);
+ return ret;
+}
+
+static int citadel_remove(struct spi_device *spi)
+{
+ struct citadel_data *citadel = spi_get_drvdata(spi);
+
+ cdev_del(&citadel->cdev);
+ device_destroy(citadel_class, citadel->devt);
+ free_pages((unsigned long)citadel->rx_buf, get_order(MAX_DATA_SIZE));
+ free_pages((unsigned long)citadel->tx_buf, get_order(MAX_DATA_SIZE));
+ kfree(citadel);
+
+ return 0;
+}
+
+static struct spi_driver citadel_spi_driver = {
+ .driver = {
+ .name = "citadel",
+ .of_match_table = of_match_ptr(citadel_dt_ids),
+ },
+ .probe = citadel_probe,
+ .remove = citadel_remove,
+};
+
+static int __init citadel_init(void)
+{
+ int ret;
+
+ ret = alloc_chrdev_region(&citadel_devt, 0, CITADEL_MAX_DEVICES,
+ "citadel");
+ if (ret) {
+ pr_err("%s: failed to alloc cdev region %d\n", __func__, ret);
+ return ret;
+ }
+
+ citadel_class = class_create(THIS_MODULE, "citadel");
+ if (IS_ERR(citadel_class)) {
+ unregister_chrdev_region(citadel_devt, 1);
+ return PTR_ERR(citadel_class);
+ }
+
+ ret = spi_register_driver(&citadel_spi_driver);
+ if (ret < 0) {
+ class_destroy(citadel_class);
+ unregister_chrdev_region(citadel_devt, 1);
+ }
+ return ret;
+}
+
+static void __exit citadel_exit(void)
+{
+ spi_unregister_driver(&citadel_spi_driver);
+ class_destroy(citadel_class);
+ unregister_chrdev_region(citadel_devt, 1);
+}
+
+module_init(citadel_init);
+module_exit(citadel_exit);
+
+MODULE_AUTHOR("Fernando Lugo, <flugo@google.com>");
+MODULE_DESCRIPTION("CITADEL TPM driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/misc/cs40l20.h b/drivers/misc/cs40l20.h
new file mode 100644
index 0000000..8aca0fd
--- /dev/null
+++ b/drivers/misc/cs40l20.h
@@ -0,0 +1,706 @@
+/*
+ * cs40l20.h -- CS40L20 Haptics Driver
+ *
+ * Copyright 2017 Cirrus Logic, Inc.
+ *
+ * Author: Jeff LaBundy <jeff.labundy@cirrus.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#ifndef __CS40L20_H__
+#define __CS40L20_H__
+
+#include <linux/regmap.h>
+
+#define CS40L20_FIRSTREG 0x00000000
+#define CS40L20_LASTREG 0x03804FE8
+#define CS40L20_DEVID 0x00000000
+#define CS40L20_REVID 0x00000004
+#define CS40L20_FABID 0x00000008
+#define CS40L20_RELID 0x0000000C
+#define CS40L20_OTPID 0x00000010
+#define CS40L20_SFT_RESET 0x00000020
+#define CS40L20_TEST_KEY_CTL 0x00000040
+#define CS40L20_USER_KEY_CTL 0x00000044
+#define CS40L20_CTRL_ASYNC1 0x00000054
+#define CS40L20_OTP_MEM0 0x00000400
+#define CS40L20_OTP_MEM31 0x0000047C
+#define CS40L20_OTP_CTRL0 0x00000500
+#define CS40L20_OTP_CTRL1 0x00000504
+#define CS40L20_OTP_CTRL3 0x00000508
+#define CS40L20_OTP_CTRL4 0x0000050C
+#define CS40L20_OTP_CTRL5 0x00000510
+#define CS40L20_OTP_CTRL6 0x00000514
+#define CS40L20_OTP_CTRL7 0x00000518
+#define CS40L20_OTP_CTRL8 0x0000051C
+#define CS40L20_PWR_CTRL1 0x00002014
+#define CS40L20_PWR_CTRL2 0x00002018
+#define CS40L20_PWR_CTRL3 0x0000201C
+#define CS40L20_CTRL_OVRRIDE 0x00002020
+#define CS40L20_AMP_OUT_MUTE 0x00002024
+#define CS40L20_PROTECT_REL_ERR_IGN 0x00002034
+#define CS40L20_LRCK_PAD_CONTROL 0x00002418
+#define CS40L20_SDIN_PAD_CONTROL 0x00002420
+#define CS40L20_GPIO_PAD_CONTROL 0x0000242C
+#define CS40L20_JTAG_CONTROL 0x00002438
+#define CS40L20_PWRMGT_CTL 0x00002900
+#define CS40L20_WAKESRC_CTL 0x00002904
+#define CS40L20_PWRMGT_STS 0x00002908
+#define CS40L20_PWRMGT_TEST 0x0000290C
+#define CS40L20_PLL_CLK_CTRL 0x00002C04
+#define CS40L20_DSP_CLK_CTRL 0x00002C08
+#define CS40L20_GLOBAL_CLK_CTRL 0x00002C0C
+#define CS40L20_DATA_FS_SEL 0x00002C10
+#define CS40L20_PLL_MISC_CTRL 0x00003014
+#define CS40L20_MDSYNC_EN 0x00003400
+#define CS40L20_MDSYNC_TX_ID 0x00003408
+#define CS40L20_MDSYNC_PWR_CTRL 0x0000340C
+#define CS40L20_MDSYNC_DATA_TX 0x00003410
+#define CS40L20_MDSYNC_TX_STATUS 0x00003414
+#define CS40L20_MDSYNC_DATA_RX 0x0000341C
+#define CS40L20_MDSYNC_RX_STATUS 0x00003420
+#define CS40L20_MDSYNC_ERR_STATUS 0x00003424
+#define CS40L20_MDSYNC_SYNC_PTE2 0x00003528
+#define CS40L20_MDSYNC_SYNC_PTE3 0x0000352C
+#define CS40L20_MDSYNC_SYNC_MSM_STATUS 0x0000353C
+#define CS40L20_BSTCVRT_VCTRL1 0x00003800
+#define CS40L20_BSTCVRT_VCTRL2 0x00003804
+#define CS40L20_BSTCVRT_PEAK_CUR 0x00003808
+#define CS40L20_BSTCVRT_SFT_RAMP 0x0000380C
+#define CS40L20_BSTCVRT_COEFF 0x00003810
+#define CS40L20_BSTCVRT_SLOPE_LBST 0x00003814
+#define CS40L20_BSTCVRT_SW_FREQ 0x00003818
+#define CS40L20_BSTCVRT_DCM_CTRL 0x0000381C
+#define CS40L20_BSTCVRT_DCM_MODE_FORCE 0x00003820
+#define CS40L20_BSTCVRT_OVERVOLT_CTRL 0x00003830
+#define CS40L20_VI_VOL_POL 0x00004000
+#define CS40L20_DTEMP_WARN_THLD 0x00004220
+#define CS40L20_DTEMP_CFG 0x00004224
+#define CS40L20_DTEMP_EN 0x00004308
+#define CS40L20_VPVBST_FS_SEL 0x00004400
+#define CS40L20_SP_ENABLES 0x00004800
+#define CS40L20_SP_RATE_CTRL 0x00004804
+#define CS40L20_SP_FORMAT 0x00004808
+#define CS40L20_SP_HIZ_CTRL 0x0000480C
+#define CS40L20_SP_FRAME_TX_SLOT 0x00004810
+#define CS40L20_SP_FRAME_RX_SLOT 0x00004820
+#define CS40L20_SP_TX_WL 0x00004830
+#define CS40L20_SP_RX_WL 0x00004840
+#define CS40L20_DAC_PCM1_SRC 0x00004C00
+#define CS40L20_ASP_TX1_SRC 0x00004C20
+#define CS40L20_ASP_TX2_SRC 0x00004C24
+#define CS40L20_ASP_TX3_SRC 0x00004C28
+#define CS40L20_ASP_TX4_SRC 0x00004C2C
+#define CS40L20_DSP1_RX1_SRC 0x00004C40
+#define CS40L20_DSP1_RX2_SRC 0x00004C44
+#define CS40L20_DSP1_RX3_SRC 0x00004C48
+#define CS40L20_DSP1_RX4_SRC 0x00004C4C
+#define CS40L20_DSP1_RX5_SRC 0x00004C50
+#define CS40L20_DSP1_RX6_SRC 0x00004C54
+#define CS40L20_DSP1_RX7_SRC 0x00004C58
+#define CS40L20_DSP1_RX8_SRC 0x00004C5C
+#define CS40L20_NGATE1_SRC 0x00004C60
+#define CS40L20_NGATE2_SRC 0x00004C64
+#define CS40L20_AMP_DIG_VOL_CTRL 0x00006000
+#define CS40L20_VPBR_CFG 0x00006404
+#define CS40L20_VBBR_CFG 0x00006408
+#define CS40L20_VPBR_STATUS 0x0000640C
+#define CS40L20_VBBR_STATUS 0x00006410
+#define CS40L20_OVERTEMP_CFG 0x00006414
+#define CS40L20_AMP_ERR_VOL 0x00006418
+#define CS40L20_VOL_STATUS_TO_DSP 0x00006450
+#define CS40L20_CLASSH_CFG 0x00006800
+#define CS40L20_WKFET_CFG 0x00006804
+#define CS40L20_NG_CFG 0x00006808
+#define CS40L20_AMP_GAIN_CTRL 0x00006C04
+#define CS40L20_DAC_MSM_CFG 0x00007400
+#define CS40L20_IRQ1_CFG 0x00010000
+#define CS40L20_IRQ1_STATUS 0x00010004
+#define CS40L20_IRQ1_STATUS1 0x00010010
+#define CS40L20_IRQ1_STATUS2 0x00010014
+#define CS40L20_IRQ1_STATUS3 0x00010018
+#define CS40L20_IRQ1_STATUS4 0x0001001C
+#define CS40L20_IRQ1_RAW_STATUS1 0x00010090
+#define CS40L20_IRQ1_RAW_STATUS2 0x00010094
+#define CS40L20_IRQ1_RAW_STATUS3 0x00010098
+#define CS40L20_IRQ1_RAW_STATUS4 0x0001009C
+#define CS40L20_IRQ1_MASK1 0x00010110
+#define CS40L20_IRQ1_MASK2 0x00010114
+#define CS40L20_IRQ1_MASK3 0x00010118
+#define CS40L20_IRQ1_MASK4 0x0001011C
+#define CS40L20_IRQ1_FRC1 0x00010190
+#define CS40L20_IRQ1_FRC2 0x00010194
+#define CS40L20_IRQ1_FRC3 0x00010198
+#define CS40L20_IRQ1_FRC4 0x0001019C
+#define CS40L20_IRQ1_EDGE1 0x00010210
+#define CS40L20_IRQ1_EDGE4 0x0001021C
+#define CS40L20_IRQ1_POL1 0x00010290
+#define CS40L20_IRQ1_POL2 0x00010294
+#define CS40L20_IRQ1_POL3 0x00010298
+#define CS40L20_IRQ1_POL4 0x0001029C
+#define CS40L20_IRQ1_DB3 0x00010318
+#define CS40L20_IRQ2_CFG 0x00010800
+#define CS40L20_IRQ2_STATUS 0x00010804
+#define CS40L20_IRQ2_STATUS1 0x00010810
+#define CS40L20_IRQ2_STATUS2 0x00010814
+#define CS40L20_IRQ2_STATUS3 0x00010818
+#define CS40L20_IRQ2_STATUS4 0x0001081C
+#define CS40L20_IRQ2_RAW_STATUS1 0x00010890
+#define CS40L20_IRQ2_RAW_STATUS2 0x00010894
+#define CS40L20_IRQ2_RAW_STATUS3 0x00010898
+#define CS40L20_IRQ2_RAW_STATUS4 0x0001089C
+#define CS40L20_IRQ2_MASK1 0x00010910
+#define CS40L20_IRQ2_MASK2 0x00010914
+#define CS40L20_IRQ2_MASK3 0x00010918
+#define CS40L20_IRQ2_MASK4 0x0001091C
+#define CS40L20_IRQ2_FRC1 0x00010990
+#define CS40L20_IRQ2_FRC2 0x00010994
+#define CS40L20_IRQ2_FRC3 0x00010998
+#define CS40L20_IRQ2_FRC4 0x0001099C
+#define CS40L20_IRQ2_EDGE1 0x00010A10
+#define CS40L20_IRQ2_EDGE4 0x00010A1C
+#define CS40L20_IRQ2_POL1 0x00010A90
+#define CS40L20_IRQ2_POL2 0x00010A94
+#define CS40L20_IRQ2_POL3 0x00010A98
+#define CS40L20_IRQ2_POL4 0x00010A9C
+#define CS40L20_IRQ2_DB3 0x00010B18
+#define CS40L20_GPIO_STATUS1 0x00011000
+#define CS40L20_GPIO1_CTRL1 0x00011008
+#define CS40L20_GPIO2_CTRL1 0x0001100C
+#define CS40L20_GPIO3_CTRL1 0x00011010
+#define CS40L20_GPIO4_CTRL1 0x00011014
+#define CS40L20_MIXER_NGATE_CFG 0x00012000
+#define CS40L20_MIXER_NGATE_CH1_CFG 0x00012004
+#define CS40L20_MIXER_NGATE_CH2_CFG 0x00012008
+#define CS40L20_DSP_MBOX_1 0x00013000
+#define CS40L20_DSP_MBOX_2 0x00013004
+#define CS40L20_DSP_MBOX_3 0x00013008
+#define CS40L20_DSP_MBOX_4 0x0001300C
+#define CS40L20_DSP_MBOX_5 0x00013010
+#define CS40L20_DSP_MBOX_6 0x00013014
+#define CS40L20_DSP_MBOX_7 0x00013018
+#define CS40L20_DSP_MBOX_8 0x0001301C
+#define CS40L20_DSP_VIRT1_MBOX_1 0x00013020
+#define CS40L20_DSP_VIRT1_MBOX_2 0x00013024
+#define CS40L20_DSP_VIRT1_MBOX_3 0x00013028
+#define CS40L20_DSP_VIRT1_MBOX_4 0x0001302C
+#define CS40L20_DSP_VIRT1_MBOX_5 0x00013030
+#define CS40L20_DSP_VIRT1_MBOX_6 0x00013034
+#define CS40L20_DSP_VIRT1_MBOX_7 0x00013038
+#define CS40L20_DSP_VIRT1_MBOX_8 0x0001303C
+#define CS40L20_DSP_VIRT2_MBOX_1 0x00013040
+#define CS40L20_DSP_VIRT2_MBOX_2 0x00013044
+#define CS40L20_DSP_VIRT2_MBOX_3 0x00013048
+#define CS40L20_DSP_VIRT2_MBOX_4 0x0001304C
+#define CS40L20_DSP_VIRT2_MBOX_5 0x00013050
+#define CS40L20_DSP_VIRT2_MBOX_6 0x00013054
+#define CS40L20_DSP_VIRT2_MBOX_7 0x00013058
+#define CS40L20_DSP_VIRT2_MBOX_8 0x0001305C
+#define CS40L20_CLOCK_DETECT_1 0x00014000
+#define CS40L20_TIMER1_CONTROL 0x00015000
+#define CS40L20_TIMER1_COUNT_PRESET 0x00015004
+#define CS40L20_TIMER1_START_STOP 0x0001500C
+#define CS40L20_TIMER1_STATUS 0x00015010
+#define CS40L20_TIMER1_COUNT_READBACK 0x00015014
+#define CS40L20_TIMER1_DSP_CLK_CFG 0x00015018
+#define CS40L20_TIMER1_DSP_CLK_STATUS 0x0001501C
+#define CS40L20_TIMER2_CONTROL 0x00015100
+#define CS40L20_TIMER2_COUNT_PRESET 0x00015104
+#define CS40L20_TIMER2_START_STOP 0x0001510C
+#define CS40L20_TIMER2_STATUS 0x00015110
+#define CS40L20_TIMER2_COUNT_READBACK 0x00015114
+#define CS40L20_TIMER2_DSP_CLK_CFG 0x00015118
+#define CS40L20_TIMER2_DSP_CLK_STATUS 0x0001511C
+#define CS40L20_DFT_JTAG_CONTROL 0x00016000
+#define CS40L20_DIE_STS1 0x00017040
+#define CS40L20_DIE_STS2 0x00017044
+#define CS40L20_TEMP_CAL1 0x00017048
+#define CS40L20_TEMP_CAL2 0x0001704C
+#define CS40L20_DSP1_XMEM_PACK_0 0x02000000
+#define CS40L20_DSP1_XMEM_PACK_3068 0x02002FF0
+#define CS40L20_DSP1_XMEM_UNPACK32_0 0x02400000
+#define CS40L20_DSP1_XMEM_UNPACK32_2046 0x02401FF8
+#define CS40L20_DSP1_TIMESTAMP_COUNT 0x025C0800
+#define CS40L20_DSP1_SYS_ID 0x025E0000
+#define CS40L20_DSP1_SYS_VERSION 0x025E0004
+#define CS40L20_DSP1_SYS_CORE_ID 0x025E0008
+#define CS40L20_DSP1_SYS_AHB_ADDR 0x025E000C
+#define CS40L20_DSP1_SYS_XSRAM_SIZE 0x025E0010
+#define CS40L20_DSP1_SYS_YSRAM_SIZE 0x025E0018
+#define CS40L20_DSP1_SYS_PSRAM_SIZE 0x025E0020
+#define CS40L20_DSP1_SYS_PM_BOOT_SIZE 0x025E0028
+#define CS40L20_DSP1_SYS_FEATURES 0x025E002C
+#define CS40L20_DSP1_SYS_FIR_FILTERS 0x025E0030
+#define CS40L20_DSP1_SYS_LMS_FILTERS 0x025E0034
+#define CS40L20_DSP1_SYS_XM_BANK_SIZE 0x025E0038
+#define CS40L20_DSP1_SYS_YM_BANK_SIZE 0x025E003C
+#define CS40L20_DSP1_SYS_PM_BANK_SIZE 0x025E0040
+#define CS40L20_DSP1_AHBM_WIN0_CTRL0 0x025E2000
+#define CS40L20_DSP1_AHBM_WIN0_CTRL1 0x025E2004
+#define CS40L20_DSP1_AHBM_WIN1_CTRL0 0x025E2008
+#define CS40L20_DSP1_AHBM_WIN1_CTRL1 0x025E200C
+#define CS40L20_DSP1_AHBM_WIN2_CTRL0 0x025E2010
+#define CS40L20_DSP1_AHBM_WIN2_CTRL1 0x025E2014
+#define CS40L20_DSP1_AHBM_WIN3_CTRL0 0x025E2018
+#define CS40L20_DSP1_AHBM_WIN3_CTRL1 0x025E201C
+#define CS40L20_DSP1_AHBM_WIN4_CTRL0 0x025E2020
+#define CS40L20_DSP1_AHBM_WIN4_CTRL1 0x025E2024
+#define CS40L20_DSP1_AHBM_WIN5_CTRL0 0x025E2028
+#define CS40L20_DSP1_AHBM_WIN5_CTRL1 0x025E202C
+#define CS40L20_DSP1_AHBM_WIN6_CTRL0 0x025E2030
+#define CS40L20_DSP1_AHBM_WIN6_CTRL1 0x025E2034
+#define CS40L20_DSP1_AHBM_WIN7_CTRL0 0x025E2038
+#define CS40L20_DSP1_AHBM_WIN7_CTRL1 0x025E203C
+#define CS40L20_DSP1_AHBM_WIN_DBG_CTRL0 0x025E2040
+#define CS40L20_DSP1_AHBM_WIN_DBG_CTRL1 0x025E2044
+#define CS40L20_DSP1_XMEM_UNPACK24_0 0x02800000
+#define CS40L20_DSP1_XMEM_UNPACK24_4093 0x02803FF4
+#define CS40L20_DSP1_CTRL_BASE 0x02B80000
+#define CS40L20_DSP1_CORE_SOFT_RESET 0x02B80010
+#define CS40L20_DSP1_DEBUG 0x02B80040
+#define CS40L20_DSP1_TIMER_CTRL 0x02B80048
+#define CS40L20_DSP1_STREAM_ARB_CTRL 0x02B80050
+#define CS40L20_DSP1_RX1_RATE 0x02B80080
+#define CS40L20_DSP1_RX2_RATE 0x02B80088
+#define CS40L20_DSP1_RX3_RATE 0x02B80090
+#define CS40L20_DSP1_RX4_RATE 0x02B80098
+#define CS40L20_DSP1_RX5_RATE 0x02B800A0
+#define CS40L20_DSP1_RX6_RATE 0x02B800A8
+#define CS40L20_DSP1_RX7_RATE 0x02B800B0
+#define CS40L20_DSP1_RX8_RATE 0x02B800B8
+#define CS40L20_DSP1_TX1_RATE 0x02B80280
+#define CS40L20_DSP1_TX2_RATE 0x02B80288
+#define CS40L20_DSP1_TX3_RATE 0x02B80290
+#define CS40L20_DSP1_TX4_RATE 0x02B80298
+#define CS40L20_DSP1_TX5_RATE 0x02B802A0
+#define CS40L20_DSP1_TX6_RATE 0x02B802A8
+#define CS40L20_DSP1_TX7_RATE 0x02B802B0
+#define CS40L20_DSP1_TX8_RATE 0x02B802B8
+#define CS40L20_DSP1_NMI_CTRL1 0x02B80480
+#define CS40L20_DSP1_NMI_CTRL2 0x02B80488
+#define CS40L20_DSP1_NMI_CTRL3 0x02B80490
+#define CS40L20_DSP1_NMI_CTRL4 0x02B80498
+#define CS40L20_DSP1_NMI_CTRL5 0x02B804A0
+#define CS40L20_DSP1_NMI_CTRL6 0x02B804A8
+#define CS40L20_DSP1_NMI_CTRL7 0x02B804B0
+#define CS40L20_DSP1_NMI_CTRL8 0x02B804B8
+#define CS40L20_DSP1_RESUME_CTRL 0x02B80500
+#define CS40L20_DSP1_IRQ1_CTRL 0x02B80508
+#define CS40L20_DSP1_IRQ2_CTRL 0x02B80510
+#define CS40L20_DSP1_IRQ3_CTRL 0x02B80518
+#define CS40L20_DSP1_IRQ4_CTRL 0x02B80520
+#define CS40L20_DSP1_IRQ5_CTRL 0x02B80528
+#define CS40L20_DSP1_IRQ6_CTRL 0x02B80530
+#define CS40L20_DSP1_IRQ7_CTRL 0x02B80538
+#define CS40L20_DSP1_IRQ8_CTRL 0x02B80540
+#define CS40L20_DSP1_IRQ9_CTRL 0x02B80548
+#define CS40L20_DSP1_IRQ10_CTRL 0x02B80550
+#define CS40L20_DSP1_IRQ11_CTRL 0x02B80558
+#define CS40L20_DSP1_IRQ12_CTRL 0x02B80560
+#define CS40L20_DSP1_IRQ13_CTRL 0x02B80568
+#define CS40L20_DSP1_IRQ14_CTRL 0x02B80570
+#define CS40L20_DSP1_IRQ15_CTRL 0x02B80578
+#define CS40L20_DSP1_IRQ16_CTRL 0x02B80580
+#define CS40L20_DSP1_IRQ17_CTRL 0x02B80588
+#define CS40L20_DSP1_IRQ18_CTRL 0x02B80590
+#define CS40L20_DSP1_IRQ19_CTRL 0x02B80598
+#define CS40L20_DSP1_IRQ20_CTRL 0x02B805A0
+#define CS40L20_DSP1_IRQ21_CTRL 0x02B805A8
+#define CS40L20_DSP1_IRQ22_CTRL 0x02B805B0
+#define CS40L20_DSP1_IRQ23_CTRL 0x02B805B8
+#define CS40L20_DSP1_SCRATCH1 0x02B805C0
+#define CS40L20_DSP1_SCRATCH2 0x02B805C8
+#define CS40L20_DSP1_SCRATCH3 0x02B805D0
+#define CS40L20_DSP1_SCRATCH4 0x02B805D8
+#define CS40L20_DSP1_CCM_CORE_CTRL 0x02BC1000
+#define CS40L20_DSP1_CCM_CLK_OVERRIDE 0x02BC1008
+#define CS40L20_DSP1_XM_MSTR_EN 0x02BC2000
+#define CS40L20_DSP1_XM_CORE_PRI 0x02BC2008
+#define CS40L20_DSP1_XM_AHB_PACK_PL_PRI 0x02BC2010
+#define CS40L20_DSP1_XM_AHB_UP_PL_PRI 0x02BC2018
+#define CS40L20_DSP1_XM_ACCEL_PL0_PRI 0x02BC2020
+#define CS40L20_DSP1_XM_NPL0_PRI 0x02BC2078
+#define CS40L20_DSP1_YM_MSTR_EN 0x02BC20C0
+#define CS40L20_DSP1_YM_CORE_PRI 0x02BC20C8
+#define CS40L20_DSP1_YM_AHB_PACK_PL_PRI 0x02BC20D0
+#define CS40L20_DSP1_YM_AHB_UP_PL_PRI 0x02BC20D8
+#define CS40L20_DSP1_YM_ACCEL_PL0_PRI 0x02BC20E0
+#define CS40L20_DSP1_YM_NPL0_PRI 0x02BC2138
+#define CS40L20_DSP1_PM_MSTR_EN 0x02BC2180
+#define CS40L20_DSP1_PM_PATCH0_ADDR 0x02BC2188
+#define CS40L20_DSP1_PM_PATCH0_EN 0x02BC218C
+#define CS40L20_DSP1_PM_PATCH0_DATA_LO 0x02BC2190
+#define CS40L20_DSP1_PM_PATCH0_DATA_HI 0x02BC2194
+#define CS40L20_DSP1_PM_PATCH1_ADDR 0x02BC2198
+#define CS40L20_DSP1_PM_PATCH1_EN 0x02BC219C
+#define CS40L20_DSP1_PM_PATCH1_DATA_LO 0x02BC21A0
+#define CS40L20_DSP1_PM_PATCH1_DATA_HI 0x02BC21A4
+#define CS40L20_DSP1_PM_PATCH2_ADDR 0x02BC21A8
+#define CS40L20_DSP1_PM_PATCH2_EN 0x02BC21AC
+#define CS40L20_DSP1_PM_PATCH2_DATA_LO 0x02BC21B0
+#define CS40L20_DSP1_PM_PATCH2_DATA_HI 0x02BC21B4
+#define CS40L20_DSP1_PM_PATCH3_ADDR 0x02BC21B8
+#define CS40L20_DSP1_PM_PATCH3_EN 0x02BC21BC
+#define CS40L20_DSP1_PM_PATCH3_DATA_LO 0x02BC21C0
+#define CS40L20_DSP1_PM_PATCH3_DATA_HI 0x02BC21C4
+#define CS40L20_DSP1_PM_PATCH4_ADDR 0x02BC21C8
+#define CS40L20_DSP1_PM_PATCH4_EN 0x02BC21CC
+#define CS40L20_DSP1_PM_PATCH4_DATA_LO 0x02BC21D0
+#define CS40L20_DSP1_PM_PATCH4_DATA_HI 0x02BC21D4
+#define CS40L20_DSP1_PM_PATCH5_ADDR 0x02BC21D8
+#define CS40L20_DSP1_PM_PATCH5_EN 0x02BC21DC
+#define CS40L20_DSP1_PM_PATCH5_DATA_LO 0x02BC21E0
+#define CS40L20_DSP1_PM_PATCH5_DATA_HI 0x02BC21E4
+#define CS40L20_DSP1_PM_PATCH6_ADDR 0x02BC21E8
+#define CS40L20_DSP1_PM_PATCH6_EN 0x02BC21EC
+#define CS40L20_DSP1_PM_PATCH6_DATA_LO 0x02BC21F0
+#define CS40L20_DSP1_PM_PATCH6_DATA_HI 0x02BC21F4
+#define CS40L20_DSP1_PM_PATCH7_ADDR 0x02BC21F8
+#define CS40L20_DSP1_PM_PATCH7_EN 0x02BC21FC
+#define CS40L20_DSP1_PM_PATCH7_DATA_LO 0x02BC2200
+#define CS40L20_DSP1_PM_PATCH7_DATA_HI 0x02BC2204
+#define CS40L20_DSP1_MPU_XM_ACCESS0 0x02BC3000
+#define CS40L20_DSP1_MPU_YM_ACCESS0 0x02BC3004
+#define CS40L20_DSP1_MPU_WNDW_ACCESS0 0x02BC3008
+#define CS40L20_DSP1_MPU_XREG_ACCESS0 0x02BC300C
+#define CS40L20_DSP1_MPU_YREG_ACCESS0 0x02BC3014
+#define CS40L20_DSP1_MPU_XM_ACCESS1 0x02BC3018
+#define CS40L20_DSP1_MPU_YM_ACCESS1 0x02BC301C
+#define CS40L20_DSP1_MPU_WNDW_ACCESS1 0x02BC3020
+#define CS40L20_DSP1_MPU_XREG_ACCESS1 0x02BC3024
+#define CS40L20_DSP1_MPU_YREG_ACCESS1 0x02BC302C
+#define CS40L20_DSP1_MPU_XM_ACCESS2 0x02BC3030
+#define CS40L20_DSP1_MPU_YM_ACCESS2 0x02BC3034
+#define CS40L20_DSP1_MPU_WNDW_ACCESS2 0x02BC3038
+#define CS40L20_DSP1_MPU_XREG_ACCESS2 0x02BC303C
+#define CS40L20_DSP1_MPU_YREG_ACCESS2 0x02BC3044
+#define CS40L20_DSP1_MPU_XM_ACCESS3 0x02BC3048
+#define CS40L20_DSP1_MPU_YM_ACCESS3 0x02BC304C
+#define CS40L20_DSP1_MPU_WNDW_ACCESS3 0x02BC3050
+#define CS40L20_DSP1_MPU_XREG_ACCESS3 0x02BC3054
+#define CS40L20_DSP1_MPU_YREG_ACCESS3 0x02BC305C
+#define CS40L20_DSP1_MPU_XM_VIO_ADDR 0x02BC3100
+#define CS40L20_DSP1_MPU_XM_VIO_STATUS 0x02BC3104
+#define CS40L20_DSP1_MPU_YM_VIO_ADDR 0x02BC3108
+#define CS40L20_DSP1_MPU_YM_VIO_STATUS 0x02BC310C
+#define CS40L20_DSP1_MPU_PM_VIO_ADDR 0x02BC3110
+#define CS40L20_DSP1_MPU_PM_VIO_STATUS 0x02BC3114
+#define CS40L20_DSP1_MPU_LOCK_CONFIG 0x02BC3140
+#define CS40L20_DSP1_MPU_WDT_RST_CTRL 0x02BC3180
+#define CS40L20_DSP1_STRMARB_MSTR0_CFG0 0x02BC5000
+#define CS40L20_DSP1_STRMARB_MSTR0_CFG1 0x02BC5004
+#define CS40L20_DSP1_STRMARB_MSTR0_CFG2 0x02BC5008
+#define CS40L20_DSP1_STRMARB_MSTR1_CFG0 0x02BC5010
+#define CS40L20_DSP1_STRMARB_MSTR1_CFG1 0x02BC5014
+#define CS40L20_DSP1_STRMARB_MSTR1_CFG2 0x02BC5018
+#define CS40L20_DSP1_STRMARB_MSTR2_CFG0 0x02BC5020
+#define CS40L20_DSP1_STRMARB_MSTR2_CFG1 0x02BC5024
+#define CS40L20_DSP1_STRMARB_MSTR2_CFG2 0x02BC5028
+#define CS40L20_DSP1_STRMARB_MSTR3_CFG0 0x02BC5030
+#define CS40L20_DSP1_STRMARB_MSTR3_CFG1 0x02BC5034
+#define CS40L20_DSP1_STRMARB_MSTR3_CFG2 0x02BC5038
+#define CS40L20_DSP1_STRMARB_MSTR4_CFG0 0x02BC5040
+#define CS40L20_DSP1_STRMARB_MSTR4_CFG1 0x02BC5044
+#define CS40L20_DSP1_STRMARB_MSTR4_CFG2 0x02BC5048
+#define CS40L20_DSP1_STRMARB_MSTR5_CFG0 0x02BC5050
+#define CS40L20_DSP1_STRMARB_MSTR5_CFG1 0x02BC5054
+#define CS40L20_DSP1_STRMARB_MSTR5_CFG2 0x02BC5058
+#define CS40L20_DSP1_STRMARB_MSTR6_CFG0 0x02BC5060
+#define CS40L20_DSP1_STRMARB_MSTR6_CFG1 0x02BC5064
+#define CS40L20_DSP1_STRMARB_MSTR6_CFG2 0x02BC5068
+#define CS40L20_DSP1_STRMARB_MSTR7_CFG0 0x02BC5070
+#define CS40L20_DSP1_STRMARB_MSTR7_CFG1 0x02BC5074
+#define CS40L20_DSP1_STRMARB_MSTR7_CFG2 0x02BC5078
+#define CS40L20_DSP1_STRMARB_TX0_CFG0 0x02BC5200
+#define CS40L20_DSP1_STRMARB_TX0_CFG1 0x02BC5204
+#define CS40L20_DSP1_STRMARB_TX1_CFG0 0x02BC5208
+#define CS40L20_DSP1_STRMARB_TX1_CFG1 0x02BC520C
+#define CS40L20_DSP1_STRMARB_TX2_CFG0 0x02BC5210
+#define CS40L20_DSP1_STRMARB_TX2_CFG1 0x02BC5214
+#define CS40L20_DSP1_STRMARB_TX3_CFG0 0x02BC5218
+#define CS40L20_DSP1_STRMARB_TX3_CFG1 0x02BC521C
+#define CS40L20_DSP1_STRMARB_TX4_CFG0 0x02BC5220
+#define CS40L20_DSP1_STRMARB_TX4_CFG1 0x02BC5224
+#define CS40L20_DSP1_STRMARB_TX5_CFG0 0x02BC5228
+#define CS40L20_DSP1_STRMARB_TX5_CFG1 0x02BC522C
+#define CS40L20_DSP1_STRMARB_TX6_CFG0 0x02BC5230
+#define CS40L20_DSP1_STRMARB_TX6_CFG1 0x02BC5234
+#define CS40L20_DSP1_STRMARB_TX7_CFG0 0x02BC5238
+#define CS40L20_DSP1_STRMARB_TX7_CFG1 0x02BC523C
+#define CS40L20_DSP1_STRMARB_RX0_CFG0 0x02BC5400
+#define CS40L20_DSP1_STRMARB_RX0_CFG1 0x02BC5404
+#define CS40L20_DSP1_STRMARB_RX1_CFG0 0x02BC5408
+#define CS40L20_DSP1_STRMARB_RX1_CFG1 0x02BC540C
+#define CS40L20_DSP1_STRMARB_RX2_CFG0 0x02BC5410
+#define CS40L20_DSP1_STRMARB_RX2_CFG1 0x02BC5414
+#define CS40L20_DSP1_STRMARB_RX3_CFG0 0x02BC5418
+#define CS40L20_DSP1_STRMARB_RX3_CFG1 0x02BC541C
+#define CS40L20_DSP1_STRMARB_RX4_CFG0 0x02BC5420
+#define CS40L20_DSP1_STRMARB_RX4_CFG1 0x02BC5424
+#define CS40L20_DSP1_STRMARB_RX5_CFG0 0x02BC5428
+#define CS40L20_DSP1_STRMARB_RX5_CFG1 0x02BC542C
+#define CS40L20_DSP1_STRMARB_RX6_CFG0 0x02BC5430
+#define CS40L20_DSP1_STRMARB_RX6_CFG1 0x02BC5434
+#define CS40L20_DSP1_STRMARB_RX7_CFG0 0x02BC5438
+#define CS40L20_DSP1_STRMARB_RX7_CFG1 0x02BC543C
+#define CS40L20_DSP1_STRMARB_IRQ0_CFG0 0x02BC5600
+#define CS40L20_DSP1_STRMARB_IRQ0_CFG1 0x02BC5604
+#define CS40L20_DSP1_STRMARB_IRQ0_CFG2 0x02BC5608
+#define CS40L20_DSP1_STRMARB_IRQ1_CFG0 0x02BC5610
+#define CS40L20_DSP1_STRMARB_IRQ1_CFG1 0x02BC5614
+#define CS40L20_DSP1_STRMARB_IRQ1_CFG2 0x02BC5618
+#define CS40L20_DSP1_STRMARB_IRQ2_CFG0 0x02BC5620
+#define CS40L20_DSP1_STRMARB_IRQ2_CFG1 0x02BC5624
+#define CS40L20_DSP1_STRMARB_IRQ2_CFG2 0x02BC5628
+#define CS40L20_DSP1_STRMARB_IRQ3_CFG0 0x02BC5630
+#define CS40L20_DSP1_STRMARB_IRQ3_CFG1 0x02BC5634
+#define CS40L20_DSP1_STRMARB_IRQ3_CFG2 0x02BC5638
+#define CS40L20_DSP1_STRMARB_IRQ4_CFG0 0x02BC5640
+#define CS40L20_DSP1_STRMARB_IRQ4_CFG1 0x02BC5644
+#define CS40L20_DSP1_STRMARB_IRQ4_CFG2 0x02BC5648
+#define CS40L20_DSP1_STRMARB_IRQ5_CFG0 0x02BC5650
+#define CS40L20_DSP1_STRMARB_IRQ5_CFG1 0x02BC5654
+#define CS40L20_DSP1_STRMARB_IRQ5_CFG2 0x02BC5658
+#define CS40L20_DSP1_STRMARB_IRQ6_CFG0 0x02BC5660
+#define CS40L20_DSP1_STRMARB_IRQ6_CFG1 0x02BC5664
+#define CS40L20_DSP1_STRMARB_IRQ6_CFG2 0x02BC5668
+#define CS40L20_DSP1_STRMARB_IRQ7_CFG0 0x02BC5670
+#define CS40L20_DSP1_STRMARB_IRQ7_CFG1 0x02BC5674
+#define CS40L20_DSP1_STRMARB_IRQ7_CFG2 0x02BC5678
+#define CS40L20_DSP1_STRMARB_RESYNC_MSK 0x02BC5A00
+#define CS40L20_DSP1_STRMARB_ERR_STATUS 0x02BC5A08
+#define CS40L20_DSP1_INTPCTL_RES_STATIC 0x02BC6000
+#define CS40L20_DSP1_INTPCTL_RES_DYN 0x02BC6004
+#define CS40L20_DSP1_INTPCTL_NMI_CTRL 0x02BC6008
+#define CS40L20_DSP1_INTPCTL_IRQ_INV 0x02BC6010
+#define CS40L20_DSP1_INTPCTL_IRQ_MODE 0x02BC6014
+#define CS40L20_DSP1_INTPCTL_IRQ_EN 0x02BC6018
+#define CS40L20_DSP1_INTPCTL_IRQ_MSK 0x02BC601C
+#define CS40L20_DSP1_INTPCTL_IRQ_FLUSH 0x02BC6020
+#define CS40L20_DSP1_INTPCTL_IRQ_MSKCLR 0x02BC6024
+#define CS40L20_DSP1_INTPCTL_IRQ_FRC 0x02BC6028
+#define CS40L20_DSP1_INTPCTL_IRQ_MSKSET 0x02BC602C
+#define CS40L20_DSP1_INTPCTL_IRQ_ERR 0x02BC6030
+#define CS40L20_DSP1_INTPCTL_IRQ_PEND 0x02BC6034
+#define CS40L20_DSP1_INTPCTL_IRQ_GEN 0x02BC6038
+#define CS40L20_DSP1_INTPCTL_TESTBITS 0x02BC6040
+#define CS40L20_DSP1_WDT_CONTROL 0x02BC7000
+#define CS40L20_DSP1_WDT_STATUS 0x02BC7008
+#define CS40L20_DSP1_YMEM_PACK_0 0x02C00000
+#define CS40L20_DSP1_YMEM_PACK_1532 0x02C017F0
+#define CS40L20_DSP1_YMEM_UNPACK32_0 0x03000000
+#define CS40L20_DSP1_YMEM_UNPACK32_1022 0x03000FF8
+#define CS40L20_DSP1_YMEM_UNPACK24_0 0x03400000
+#define CS40L20_DSP1_YMEM_UNPACK24_2045 0x03401FF4
+#define CS40L20_DSP1_PMEM_0 0x03800000
+#define CS40L20_DSP1_PMEM_5114 0x03804FE8
+
+/* trim fields unpacked from OTP memory */
+#define CS40L20_OTP_TRIM_1 0x0000208C
+#define CS40L20_OTP_TRIM_2 0x00002090
+#define CS40L20_OTP_TRIM_3 0x00003010
+#define CS40L20_OTP_TRIM_4 0x0000300C
+#define CS40L20_OTP_TRIM_5 0x0000394C
+#define CS40L20_OTP_TRIM_6 0x00003950
+#define CS40L20_OTP_TRIM_7 0x00003954
+#define CS40L20_OTP_TRIM_8 0x00003958
+#define CS40L20_OTP_TRIM_9 0x0000395C
+#define CS40L20_OTP_TRIM_10 0x0000416C
+#define CS40L20_OTP_TRIM_11 0x00004160
+#define CS40L20_OTP_TRIM_12 0x00004170
+#define CS40L20_OTP_TRIM_13 0x00004360
+#define CS40L20_OTP_TRIM_14 0x00004448
+#define CS40L20_OTP_TRIM_15 0x0000444C
+#define CS40L20_OTP_TRIM_16 0x00006E30
+#define CS40L20_OTP_TRIM_17 0x00006E34
+#define CS40L20_OTP_TRIM_18 0x00006E38
+#define CS40L20_OTP_TRIM_19 0x00006E3C
+#define CS40L20_OTP_TRIM_20 0x00006E40
+#define CS40L20_OTP_TRIM_21 0x00006E44
+#define CS40L20_OTP_TRIM_22 0x00006E48
+#define CS40L20_OTP_TRIM_23 0x00006E4C
+#define CS40L20_OTP_TRIM_24 0x00006E50
+#define CS40L20_OTP_TRIM_25 0x00006E54
+#define CS40L20_OTP_TRIM_26 0x00006E58
+#define CS40L20_OTP_TRIM_27 0x00006E5C
+#define CS40L20_OTP_TRIM_28 0x00006E60
+#define CS40L20_OTP_TRIM_29 0x00006E64
+#define CS40L20_OTP_TRIM_30 0x00007418
+#define CS40L20_OTP_TRIM_31 0x0000741C
+#define CS40L20_OTP_TRIM_32 0x00007434
+#define CS40L20_OTP_TRIM_33 0x00007068
+#define CS40L20_OTP_TRIM_34 0x0000410C
+#define CS40L20_OTP_TRIM_35 0x0000400C
+#define CS40L20_OTP_TRIM_36 0x00002030
+
+#define CS40L20_GLOBAL_EN_MASK 0x00000001
+#define CS40L20_GLOBAL_EN_SHIFT 0
+
+#define CS40L20_GP2_CTRL_MASK 0x07000000
+#define CS40L20_GP2_CTRL_SHIFT 24
+#define CS40L20_GP2_CTRL_MCLK 0x3
+#define CS40L20_MEM_RDY_MASK 0x00000002
+#define CS40L20_MEM_RDY_SHIFT 1
+
+#define CS40L20_PLL_REFCLK_SEL_MASK 0x00000007
+#define CS40L20_PLL_REFCLK_SEL_SHIFT 0
+#define CS40L20_PLL_REFCLK_SEL_MCLK 0x5
+
+#define CS40L20_DAC_PCM1_SRC_MASK 0x7F
+#define CS40L20_DAC_PCM1_SRC_SHIFT 0
+#define CS40L20_DAC_PCM1_SRC_DSP1TX1 0x32
+
+#define CS40L20_DSP1_RXN_SRC_MASK 0x7F
+#define CS40L20_DSP1_RXN_SRC_SHIFT 0
+#define CS40L20_DSP1_RXN_SRC_VMON 0x18
+#define CS40L20_DSP1_RXN_SRC_IMON 0x19
+
+#define CS40L20_DSP1_RESET_MASK 0x200
+#define CS40L20_DSP1_RESET_SHIFT 9
+
+#define CS40L20_DSP1_EN_MASK 0x1
+#define CS40L20_DSP1_EN_SHIFT 0
+
+#define CS40L20_BST_IPK_MASK 0x0000007F
+#define CS40L20_BST_IPK_SHIFT 0
+
+#define CS40L20_BST_K1_MASK 0x000000FF
+#define CS40L20_BST_K1_SHIFT 0
+#define CS40L20_BST_K2_MASK 0x0000FF00
+#define CS40L20_BST_K2_SHIFT 8
+
+#define CS40L20_BST_SLOPE_MASK 0x0000FF00
+#define CS40L20_BST_SLOPE_SHIFT 8
+
+#define CS40L20_BST_LBST_VAL_MASK 0x00000003
+#define CS40L20_BST_LBST_VAL_SHIFT 0
+#define CS40L20_AMP_VOL_PCM_MASK 0x00003FF8
+#define CS40L20_AMP_VOL_PCM_SHIFT 3
+
+#define CS40L20_OTP_BOOT_ERR 0x80000000
+#define CS40L20_OTP_BOOT_DONE 0x00000002
+
+#define CS40L20_DEVID_L20 0x035A40
+#define CS40L20_DEVID_L25 0x40A250
+#define CS40L20_DEVID_L25A 0x40A25A
+#define CS40L20_DEVID_L25B 0x40A25B
+
+#define CS40L20_REVID_A0 0xA0
+#define CS40L20_REVID_B0 0xB0
+#define CS40L20_REVID_B1 0xB1
+
+#define CS40L20_MPU_UNLOCK_CODE1 0x5555
+#define CS40L20_MPU_UNLOCK_CODE2 0xAAAA
+
+#define CS40L20_XM_UNPACKED_TYPE 0x05
+#define CS40L20_YM_UNPACKED_TYPE 0x06
+#define CS40L20_PM_PACKED_TYPE 0x10
+#define CS40L20_XM_PACKED_TYPE 0x11
+#define CS40L20_YM_PACKED_TYPE 0x12
+#define CS40L20_ALGO_INFO_TYPE 0xF2
+
+#define CS40L20_FW_FILE_HEADER_SIZE 40
+#define CS40L20_FW_DBLK_OFFSET_SIZE 3
+#define CS40L20_FW_DBLK_TYPE_SIZE 1
+#define CS40L20_FW_DBLK_LENGTH_SIZE 4
+
+#define CS40L20_WT_FILE_HEADER_SIZE 16
+#define CS40L20_WT_DBLK_OFFSET_SIZE 2
+#define CS40L20_WT_DBLK_TYPE_SIZE 2
+#define CS40L20_WT_ALGO_ID_SIZE 4
+#define CS40L20_WT_ALGO_REV_SIZE 4
+#define CS40L20_WT_SAMPLE_RATE_SIZE 4
+#define CS40L20_WT_DBLK_LENGTH_SIZE 4
+
+#define CS40L20_XM_FW_ID (CS40L20_DSP1_XMEM_UNPACK24_0 + 12)
+#define CS40L20_XM_NUM_ALGOS (CS40L20_DSP1_XMEM_UNPACK24_0 + 36)
+#define CS40L20_ALGO_ID_OFFSET 0
+#define CS40L20_ALGO_REV_OFFSET 4
+#define CS40L20_ALGO_XM_BASE_OFFSET 8
+#define CS40L20_ALGO_XM_SIZE_OFFSET 12
+#define CS40L20_ALGO_YM_BASE_OFFSET 16
+#define CS40L20_ALGO_YM_SIZE_OFFSET 20
+#define CS40L20_ALGO_ENTRY_SIZE 24
+#define CS40L20_ALGO_LIST_TERM 0xBEDEAD
+#define CS40L20_ALGO_REV_MASK 0xFF0000
+
+#define CS40L20_NUM_ALGOS_MAX 16
+#define CS40L20_FW_REV_MIN 0x030000
+
+#define CS40L20_ALGO_ID_SIZE 4
+#define CS40L20_COEFF_COUNT_SIZE 4
+#define CS40L20_COEFF_OFFSET_SIZE 2
+#define CS40L20_COEFF_TYPE_SIZE 2
+#define CS40L20_COEFF_LENGTH_SIZE 4
+
+#define CS40L20_TIMEOUT_MS_MAX 0x02AAAA /* ~174 sec */
+#define CS40L20_HALO_STATE_RUNNING 0x2
+
+#define CS40L20_MBOX_TRIGGERINDEX CS40L20_DSP_VIRT1_MBOX_1
+#define CS40L20_MBOX_TRIGGER_MS CS40L20_DSP_VIRT1_MBOX_2
+#define CS40L20_MBOX_STIMULUS_MODE CS40L20_DSP_VIRT1_MBOX_3
+
+#define CS40L20_INDEX_MASK 0x7FFF
+#define CS40L20_INDEX_DIAG 0xFFFF
+
+#define CS40L20_DIAG_STATE_INIT 0x00
+#define CS40L20_DIAG_STATE_RUN 0x01
+#define CS40L20_DIAG_STATE_DONE 0x02
+#define CS40L20_DIAG_STATE_DELAY_MS 500
+#define CS40L20_DSP_TIMEOUT_COUNT 10 /* 1 count = 10 ms */
+#define CS40L20_OTP_TIMEOUT_COUNT 10
+#define CS40L20_NUM_OTP_MAPS 2
+#define CS40L20_NUM_OTP_WORDS 32
+
+#define CS40L20_TEST_KEY_UNLOCK_CODE1 0x55
+#define CS40L20_TEST_KEY_UNLOCK_CODE2 0xAA
+#define CS40L20_TEST_KEY_RELOCK_CODE1 0xCC
+#define CS40L20_TEST_KEY_RELOCK_CODE2 0x33
+
+#define CS40L20_DIG_SCALE_MAX 816 /* -102 dB */
+
+#define CS40L20_MAX_WLEN 4096
+
+#define CS40L20_DEVICE_NAME "vibrator"
+
+#define CS40L20_FW_NAME_A0 "cs40l20.wmfw"
+#define CS40L20_FW_NAME_B1 "cs40l25a.wmfw"
+
+#define CS40L20_WT_NAME "cs40l20.bin"
+bool cs40l20_readable_reg(struct device *dev, unsigned int reg);
+bool cs40l20_precious_reg(struct device *dev, unsigned int reg);
+
+struct cs40l20_trim {
+ u32 reg;
+ u8 shift;
+ u8 size;
+};
+
+struct cs40l20_otp_desc {
+ u32 id;
+ u8 row_start;
+ u8 col_start;
+ u32 num_trims;
+ const struct cs40l20_trim *trim_table;
+};
+
+struct cs40l20_coeff_desc {
+ unsigned int parent_id;
+ unsigned int block_offset;
+ unsigned int block_type;
+ unsigned char name[32];
+ unsigned int reg;
+ struct list_head list;
+};
+
+extern const unsigned char cs40l20_bst_k1_table[4][5];
+extern const unsigned char cs40l20_bst_k2_table[4][5];
+extern const unsigned char cs40l20_bst_slope_table[4];
+
+extern const struct cs40l20_otp_desc cs40l20_otp_map[CS40L20_NUM_OTP_MAPS];
+
+#endif /*__CS40L20_H__*/
diff --git a/drivers/misc/cs40l2x-tables.c b/drivers/misc/cs40l2x-tables.c
new file mode 100644
index 0000000..210e3dc
--- /dev/null
+++ b/drivers/misc/cs40l2x-tables.c
@@ -0,0 +1,958 @@
+/*
+ * cs40l2x-tables.c -- CS40L20/CS40L25/CS40L25A/CS40L25B Haptics Driver
+ *
+ * Copyright 2018 Cirrus Logic, Inc.
+ *
+ * Author: Jeff LaBundy <jeff.labundy@cirrus.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#include "cs40l2x.h"
+
+const unsigned char cs40l2x_bst_k1_table[4][5] = {
+ {0x24, 0x32, 0x32, 0x4F, 0x57},
+ {0x24, 0x32, 0x32, 0x4F, 0x57},
+ {0x40, 0x32, 0x32, 0x4F, 0x57},
+ {0x40, 0x32, 0x32, 0x4F, 0x57}
+};
+
+const unsigned char cs40l2x_bst_k2_table[4][5] = {
+ {0x24, 0x49, 0x66, 0xA3, 0xEA},
+ {0x24, 0x49, 0x66, 0xA3, 0xEA},
+ {0x48, 0x49, 0x66, 0xA3, 0xEA},
+ {0x48, 0x49, 0x66, 0xA3, 0xEA}
+};
+
+const unsigned char cs40l2x_bst_slope_table[4] = {0x75, 0x6B, 0x3B, 0x28};
+
+bool cs40l2x_readable_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case CS40L2X_DEVID:
+ case CS40L2X_REVID:
+ case CS40L2X_FABID:
+ case CS40L2X_RELID:
+ case CS40L2X_OTPID:
+ case CS40L2X_CTRL_ASYNC1:
+ case CS40L2X_OTP_CTRL0:
+ case CS40L2X_OTP_CTRL3:
+ case CS40L2X_OTP_CTRL4:
+ case CS40L2X_OTP_CTRL5:
+ case CS40L2X_OTP_CTRL6:
+ case CS40L2X_OTP_CTRL7:
+ case CS40L2X_OTP_CTRL8:
+ case CS40L2X_PWR_CTRL1:
+ case CS40L2X_PWR_CTRL2:
+ case CS40L2X_PWR_CTRL3:
+ case CS40L2X_CTRL_OVRRIDE:
+ case CS40L2X_AMP_OUT_MUTE:
+ case CS40L2X_PROTECT_REL_ERR_IGN:
+ case CS40L2X_LRCK_PAD_CONTROL:
+ case CS40L2X_SDIN_PAD_CONTROL:
+ case CS40L2X_GPIO_PAD_CONTROL:
+ case CS40L2X_JTAG_CONTROL:
+ case CS40L2X_PWRMGT_CTL:
+ case CS40L2X_WAKESRC_CTL:
+ case CS40L2X_PWRMGT_STS:
+ case CS40L2X_PWRMGT_TEST:
+ case CS40L2X_PLL_CLK_CTRL:
+ case CS40L2X_DSP_CLK_CTRL:
+ case CS40L2X_GLOBAL_CLK_CTRL:
+ case CS40L2X_DATA_FS_SEL:
+ case CS40L2X_PLL_LOOP_PARAM:
+ case CS40L2X_PLL_MISC_CTRL:
+ case CS40L2X_MDSYNC_EN:
+ case CS40L2X_MDSYNC_TX_ID:
+ case CS40L2X_MDSYNC_PWR_CTRL:
+ case CS40L2X_MDSYNC_DATA_TX:
+ case CS40L2X_MDSYNC_TX_STATUS:
+ case CS40L2X_MDSYNC_DATA_RX:
+ case CS40L2X_MDSYNC_RX_STATUS:
+ case CS40L2X_MDSYNC_ERR_STATUS:
+ case CS40L2X_MDSYNC_SYNC_PTE2:
+ case CS40L2X_MDSYNC_SYNC_PTE3:
+ case CS40L2X_MDSYNC_SYNC_MSM_STATUS:
+ case CS40L2X_BSTCVRT_VCTRL1:
+ case CS40L2X_BSTCVRT_VCTRL2:
+ case CS40L2X_BSTCVRT_PEAK_CUR:
+ case CS40L2X_BSTCVRT_SFT_RAMP:
+ case CS40L2X_BSTCVRT_COEFF:
+ case CS40L2X_BSTCVRT_SLOPE_LBST:
+ case CS40L2X_BSTCVRT_SW_FREQ:
+ case CS40L2X_BSTCVRT_DCM_CTRL:
+ case CS40L2X_BSTCVRT_DCM_MODE_FORCE:
+ case CS40L2X_BSTCVRT_OVERVOLT_CTRL:
+ case CS40L2X_VI_VOL_POL:
+ case CS40L2X_DTEMP_WARN_THLD:
+ case CS40L2X_DTEMP_CFG:
+ case CS40L2X_DTEMP_EN:
+ case CS40L2X_VPVBST_FS_SEL:
+ case CS40L2X_SP_ENABLES:
+ case CS40L2X_SP_RATE_CTRL:
+ case CS40L2X_SP_FORMAT:
+ case CS40L2X_SP_HIZ_CTRL:
+ case CS40L2X_SP_FRAME_TX_SLOT:
+ case CS40L2X_SP_FRAME_RX_SLOT:
+ case CS40L2X_SP_TX_WL:
+ case CS40L2X_SP_RX_WL:
+ case CS40L2X_DAC_PCM1_SRC:
+ case CS40L2X_ASP_TX1_SRC:
+ case CS40L2X_ASP_TX2_SRC:
+ case CS40L2X_ASP_TX3_SRC:
+ case CS40L2X_ASP_TX4_SRC:
+ case CS40L2X_DSP1_RX1_SRC:
+ case CS40L2X_DSP1_RX2_SRC:
+ case CS40L2X_DSP1_RX3_SRC:
+ case CS40L2X_DSP1_RX4_SRC:
+ case CS40L2X_DSP1_RX5_SRC:
+ case CS40L2X_DSP1_RX6_SRC:
+ case CS40L2X_DSP1_RX7_SRC:
+ case CS40L2X_DSP1_RX8_SRC:
+ case CS40L2X_NGATE1_SRC:
+ case CS40L2X_NGATE2_SRC:
+ case CS40L2X_AMP_DIG_VOL_CTRL:
+ case CS40L2X_VPBR_CFG:
+ case CS40L2X_VBBR_CFG:
+ case CS40L2X_VPBR_STATUS:
+ case CS40L2X_VBBR_STATUS:
+ case CS40L2X_OVERTEMP_CFG:
+ case CS40L2X_AMP_ERR_VOL:
+ case CS40L2X_VOL_STATUS_TO_DSP:
+ case CS40L2X_CLASSH_CFG:
+ case CS40L2X_WKFET_CFG:
+ case CS40L2X_NG_CFG:
+ case CS40L2X_AMP_GAIN_CTRL:
+ case CS40L2X_DAC_MSM_CFG:
+ case CS40L2X_IRQ1_CFG:
+ case CS40L2X_IRQ1_STATUS:
+ case CS40L2X_IRQ1_STATUS1:
+ case CS40L2X_IRQ1_STATUS2:
+ case CS40L2X_IRQ1_STATUS3:
+ case CS40L2X_IRQ1_STATUS4:
+ case CS40L2X_IRQ1_RAW_STATUS1:
+ case CS40L2X_IRQ1_RAW_STATUS2:
+ case CS40L2X_IRQ1_RAW_STATUS3:
+ case CS40L2X_IRQ1_RAW_STATUS4:
+ case CS40L2X_IRQ1_MASK1:
+ case CS40L2X_IRQ1_MASK2:
+ case CS40L2X_IRQ1_MASK3:
+ case CS40L2X_IRQ1_MASK4:
+ case CS40L2X_IRQ1_FRC1:
+ case CS40L2X_IRQ1_FRC2:
+ case CS40L2X_IRQ1_FRC3:
+ case CS40L2X_IRQ1_FRC4:
+ case CS40L2X_IRQ1_EDGE1:
+ case CS40L2X_IRQ1_EDGE4:
+ case CS40L2X_IRQ1_POL1:
+ case CS40L2X_IRQ1_POL2:
+ case CS40L2X_IRQ1_POL3:
+ case CS40L2X_IRQ1_POL4:
+ case CS40L2X_IRQ1_DB3:
+ case CS40L2X_IRQ2_CFG:
+ case CS40L2X_IRQ2_STATUS:
+ case CS40L2X_IRQ2_STATUS1:
+ case CS40L2X_IRQ2_STATUS2:
+ case CS40L2X_IRQ2_STATUS3:
+ case CS40L2X_IRQ2_STATUS4:
+ case CS40L2X_IRQ2_RAW_STATUS1:
+ case CS40L2X_IRQ2_RAW_STATUS2:
+ case CS40L2X_IRQ2_RAW_STATUS3:
+ case CS40L2X_IRQ2_RAW_STATUS4:
+ case CS40L2X_IRQ2_MASK1:
+ case CS40L2X_IRQ2_MASK2:
+ case CS40L2X_IRQ2_MASK3:
+ case CS40L2X_IRQ2_MASK4:
+ case CS40L2X_IRQ2_FRC1:
+ case CS40L2X_IRQ2_FRC2:
+ case CS40L2X_IRQ2_FRC3:
+ case CS40L2X_IRQ2_FRC4:
+ case CS40L2X_IRQ2_EDGE1:
+ case CS40L2X_IRQ2_EDGE4:
+ case CS40L2X_IRQ2_POL1:
+ case CS40L2X_IRQ2_POL2:
+ case CS40L2X_IRQ2_POL3:
+ case CS40L2X_IRQ2_POL4:
+ case CS40L2X_IRQ2_DB3:
+ case CS40L2X_GPIO_STATUS1:
+ case CS40L2X_GPIO1_CTRL1:
+ case CS40L2X_GPIO2_CTRL1:
+ case CS40L2X_GPIO3_CTRL1:
+ case CS40L2X_GPIO4_CTRL1:
+ case CS40L2X_MIXER_NGATE_CFG:
+ case CS40L2X_MIXER_NGATE_CH1_CFG:
+ case CS40L2X_MIXER_NGATE_CH2_CFG:
+ case CS40L2X_CLOCK_DETECT_1:
+ case CS40L2X_TIMER1_CONTROL:
+ case CS40L2X_TIMER1_COUNT_PRESET:
+ case CS40L2X_TIMER1_STATUS:
+ case CS40L2X_TIMER1_COUNT_READBACK:
+ case CS40L2X_TIMER1_DSP_CLK_CFG:
+ case CS40L2X_TIMER1_DSP_CLK_STATUS:
+ case CS40L2X_TIMER2_CONTROL:
+ case CS40L2X_TIMER2_COUNT_PRESET:
+ case CS40L2X_TIMER2_STATUS:
+ case CS40L2X_TIMER2_COUNT_READBACK:
+ case CS40L2X_TIMER2_DSP_CLK_CFG:
+ case CS40L2X_TIMER2_DSP_CLK_STATUS:
+ case CS40L2X_DFT_JTAG_CONTROL:
+ case CS40L2X_DIE_STS1:
+ case CS40L2X_DIE_STS2:
+ case CS40L2X_TEMP_CAL1:
+ case CS40L2X_TEMP_CAL2:
+ case CS40L2X_DSP1_TIMESTAMP_COUNT:
+ case CS40L2X_DSP1_SYS_ID:
+ case CS40L2X_DSP1_SYS_VERSION:
+ case CS40L2X_DSP1_SYS_CORE_ID:
+ case CS40L2X_DSP1_SYS_AHB_ADDR:
+ case CS40L2X_DSP1_SYS_XSRAM_SIZE:
+ case CS40L2X_DSP1_SYS_YSRAM_SIZE:
+ case CS40L2X_DSP1_SYS_PSRAM_SIZE:
+ case CS40L2X_DSP1_SYS_PM_BOOT_SIZE:
+ case CS40L2X_DSP1_SYS_FEATURES:
+ case CS40L2X_DSP1_SYS_FIR_FILTERS:
+ case CS40L2X_DSP1_SYS_LMS_FILTERS:
+ case CS40L2X_DSP1_SYS_XM_BANK_SIZE:
+ case CS40L2X_DSP1_SYS_YM_BANK_SIZE:
+ case CS40L2X_DSP1_SYS_PM_BANK_SIZE:
+ case CS40L2X_DSP1_AHBM_WIN0_CTRL0:
+ case CS40L2X_DSP1_AHBM_WIN0_CTRL1:
+ case CS40L2X_DSP1_AHBM_WIN1_CTRL0:
+ case CS40L2X_DSP1_AHBM_WIN1_CTRL1:
+ case CS40L2X_DSP1_AHBM_WIN2_CTRL0:
+ case CS40L2X_DSP1_AHBM_WIN2_CTRL1:
+ case CS40L2X_DSP1_AHBM_WIN3_CTRL0:
+ case CS40L2X_DSP1_AHBM_WIN3_CTRL1:
+ case CS40L2X_DSP1_AHBM_WIN4_CTRL0:
+ case CS40L2X_DSP1_AHBM_WIN4_CTRL1:
+ case CS40L2X_DSP1_AHBM_WIN5_CTRL0:
+ case CS40L2X_DSP1_AHBM_WIN5_CTRL1:
+ case CS40L2X_DSP1_AHBM_WIN6_CTRL0:
+ case CS40L2X_DSP1_AHBM_WIN6_CTRL1:
+ case CS40L2X_DSP1_AHBM_WIN7_CTRL0:
+ case CS40L2X_DSP1_AHBM_WIN7_CTRL1:
+ case CS40L2X_DSP1_AHBM_WIN_DBG_CTRL0:
+ case CS40L2X_DSP1_AHBM_WIN_DBG_CTRL1:
+ case CS40L2X_DSP1_DEBUG:
+ case CS40L2X_DSP1_TIMER_CTRL:
+ case CS40L2X_DSP1_RX1_RATE:
+ case CS40L2X_DSP1_RX2_RATE:
+ case CS40L2X_DSP1_RX3_RATE:
+ case CS40L2X_DSP1_RX4_RATE:
+ case CS40L2X_DSP1_RX5_RATE:
+ case CS40L2X_DSP1_RX6_RATE:
+ case CS40L2X_DSP1_RX7_RATE:
+ case CS40L2X_DSP1_RX8_RATE:
+ case CS40L2X_DSP1_TX1_RATE:
+ case CS40L2X_DSP1_TX2_RATE:
+ case CS40L2X_DSP1_TX3_RATE:
+ case CS40L2X_DSP1_TX4_RATE:
+ case CS40L2X_DSP1_TX5_RATE:
+ case CS40L2X_DSP1_TX6_RATE:
+ case CS40L2X_DSP1_TX7_RATE:
+ case CS40L2X_DSP1_TX8_RATE:
+ case CS40L2X_DSP1_NMI_CTRL1:
+ case CS40L2X_DSP1_NMI_CTRL2:
+ case CS40L2X_DSP1_NMI_CTRL3:
+ case CS40L2X_DSP1_NMI_CTRL4:
+ case CS40L2X_DSP1_NMI_CTRL5:
+ case CS40L2X_DSP1_NMI_CTRL6:
+ case CS40L2X_DSP1_NMI_CTRL7:
+ case CS40L2X_DSP1_NMI_CTRL8:
+ case CS40L2X_DSP1_RESUME_CTRL:
+ case CS40L2X_DSP1_IRQ1_CTRL:
+ case CS40L2X_DSP1_IRQ2_CTRL:
+ case CS40L2X_DSP1_IRQ3_CTRL:
+ case CS40L2X_DSP1_IRQ4_CTRL:
+ case CS40L2X_DSP1_IRQ5_CTRL:
+ case CS40L2X_DSP1_IRQ6_CTRL:
+ case CS40L2X_DSP1_IRQ7_CTRL:
+ case CS40L2X_DSP1_IRQ8_CTRL:
+ case CS40L2X_DSP1_IRQ9_CTRL:
+ case CS40L2X_DSP1_IRQ10_CTRL:
+ case CS40L2X_DSP1_IRQ11_CTRL:
+ case CS40L2X_DSP1_IRQ12_CTRL:
+ case CS40L2X_DSP1_IRQ13_CTRL:
+ case CS40L2X_DSP1_IRQ14_CTRL:
+ case CS40L2X_DSP1_IRQ15_CTRL:
+ case CS40L2X_DSP1_IRQ16_CTRL:
+ case CS40L2X_DSP1_IRQ17_CTRL:
+ case CS40L2X_DSP1_IRQ18_CTRL:
+ case CS40L2X_DSP1_IRQ19_CTRL:
+ case CS40L2X_DSP1_IRQ20_CTRL:
+ case CS40L2X_DSP1_IRQ21_CTRL:
+ case CS40L2X_DSP1_IRQ22_CTRL:
+ case CS40L2X_DSP1_IRQ23_CTRL:
+ case CS40L2X_DSP1_SCRATCH1:
+ case CS40L2X_DSP1_SCRATCH2:
+ case CS40L2X_DSP1_SCRATCH3:
+ case CS40L2X_DSP1_SCRATCH4:
+ case CS40L2X_DSP1_CCM_CORE_CTRL:
+ case CS40L2X_DSP1_CCM_CLK_OVERRIDE:
+ case CS40L2X_DSP1_XM_MSTR_EN:
+ case CS40L2X_DSP1_XM_CORE_PRI:
+ case CS40L2X_DSP1_XM_AHB_PACK_PL_PRI:
+ case CS40L2X_DSP1_XM_AHB_UP_PL_PRI:
+ case CS40L2X_DSP1_XM_ACCEL_PL0_PRI:
+ case CS40L2X_DSP1_XM_NPL0_PRI:
+ case CS40L2X_DSP1_YM_MSTR_EN:
+ case CS40L2X_DSP1_YM_CORE_PRI:
+ case CS40L2X_DSP1_YM_AHB_PACK_PL_PRI:
+ case CS40L2X_DSP1_YM_AHB_UP_PL_PRI:
+ case CS40L2X_DSP1_YM_ACCEL_PL0_PRI:
+ case CS40L2X_DSP1_YM_NPL0_PRI:
+ case CS40L2X_DSP1_PM_MSTR_EN:
+ case CS40L2X_DSP1_PM_PATCH0_ADDR:
+ case CS40L2X_DSP1_PM_PATCH0_EN:
+ case CS40L2X_DSP1_PM_PATCH0_DATA_LO:
+ case CS40L2X_DSP1_PM_PATCH0_DATA_HI:
+ case CS40L2X_DSP1_PM_PATCH1_ADDR:
+ case CS40L2X_DSP1_PM_PATCH1_EN:
+ case CS40L2X_DSP1_PM_PATCH1_DATA_LO:
+ case CS40L2X_DSP1_PM_PATCH1_DATA_HI:
+ case CS40L2X_DSP1_PM_PATCH2_ADDR:
+ case CS40L2X_DSP1_PM_PATCH2_EN:
+ case CS40L2X_DSP1_PM_PATCH2_DATA_LO:
+ case CS40L2X_DSP1_PM_PATCH2_DATA_HI:
+ case CS40L2X_DSP1_PM_PATCH3_ADDR:
+ case CS40L2X_DSP1_PM_PATCH3_EN:
+ case CS40L2X_DSP1_PM_PATCH3_DATA_LO:
+ case CS40L2X_DSP1_PM_PATCH3_DATA_HI:
+ case CS40L2X_DSP1_PM_PATCH4_ADDR:
+ case CS40L2X_DSP1_PM_PATCH4_EN:
+ case CS40L2X_DSP1_PM_PATCH4_DATA_LO:
+ case CS40L2X_DSP1_PM_PATCH4_DATA_HI:
+ case CS40L2X_DSP1_PM_PATCH5_ADDR:
+ case CS40L2X_DSP1_PM_PATCH5_EN:
+ case CS40L2X_DSP1_PM_PATCH5_DATA_LO:
+ case CS40L2X_DSP1_PM_PATCH5_DATA_HI:
+ case CS40L2X_DSP1_PM_PATCH6_ADDR:
+ case CS40L2X_DSP1_PM_PATCH6_EN:
+ case CS40L2X_DSP1_PM_PATCH6_DATA_LO:
+ case CS40L2X_DSP1_PM_PATCH6_DATA_HI:
+ case CS40L2X_DSP1_PM_PATCH7_ADDR:
+ case CS40L2X_DSP1_PM_PATCH7_EN:
+ case CS40L2X_DSP1_PM_PATCH7_DATA_LO:
+ case CS40L2X_DSP1_PM_PATCH7_DATA_HI:
+ case CS40L2X_DSP1_MPU_XM_ACCESS0:
+ case CS40L2X_DSP1_MPU_YM_ACCESS0:
+ case CS40L2X_DSP1_MPU_WNDW_ACCESS0:
+ case CS40L2X_DSP1_MPU_XREG_ACCESS0:
+ case CS40L2X_DSP1_MPU_YREG_ACCESS0:
+ case CS40L2X_DSP1_MPU_XM_ACCESS1:
+ case CS40L2X_DSP1_MPU_YM_ACCESS1:
+ case CS40L2X_DSP1_MPU_WNDW_ACCESS1:
+ case CS40L2X_DSP1_MPU_XREG_ACCESS1:
+ case CS40L2X_DSP1_MPU_YREG_ACCESS1:
+ case CS40L2X_DSP1_MPU_XM_ACCESS2:
+ case CS40L2X_DSP1_MPU_YM_ACCESS2:
+ case CS40L2X_DSP1_MPU_WNDW_ACCESS2:
+ case CS40L2X_DSP1_MPU_XREG_ACCESS2:
+ case CS40L2X_DSP1_MPU_YREG_ACCESS2:
+ case CS40L2X_DSP1_MPU_XM_ACCESS3:
+ case CS40L2X_DSP1_MPU_YM_ACCESS3:
+ case CS40L2X_DSP1_MPU_WNDW_ACCESS3:
+ case CS40L2X_DSP1_MPU_XREG_ACCESS3:
+ case CS40L2X_DSP1_MPU_YREG_ACCESS3:
+ case CS40L2X_DSP1_MPU_XM_VIO_ADDR:
+ case CS40L2X_DSP1_MPU_XM_VIO_STATUS:
+ case CS40L2X_DSP1_MPU_YM_VIO_ADDR:
+ case CS40L2X_DSP1_MPU_YM_VIO_STATUS:
+ case CS40L2X_DSP1_MPU_PM_VIO_ADDR:
+ case CS40L2X_DSP1_MPU_PM_VIO_STATUS:
+ case CS40L2X_DSP1_MPU_LOCK_CONFIG:
+ case CS40L2X_DSP1_MPU_WDT_RST_CTRL:
+ case CS40L2X_DSP1_STRMARB_MSTR0_CFG0:
+ case CS40L2X_DSP1_STRMARB_MSTR0_CFG1:
+ case CS40L2X_DSP1_STRMARB_MSTR0_CFG2:
+ case CS40L2X_DSP1_STRMARB_MSTR1_CFG0:
+ case CS40L2X_DSP1_STRMARB_MSTR1_CFG1:
+ case CS40L2X_DSP1_STRMARB_MSTR1_CFG2:
+ case CS40L2X_DSP1_STRMARB_MSTR2_CFG0:
+ case CS40L2X_DSP1_STRMARB_MSTR2_CFG1:
+ case CS40L2X_DSP1_STRMARB_MSTR2_CFG2:
+ case CS40L2X_DSP1_STRMARB_MSTR3_CFG0:
+ case CS40L2X_DSP1_STRMARB_MSTR3_CFG1:
+ case CS40L2X_DSP1_STRMARB_MSTR3_CFG2:
+ case CS40L2X_DSP1_STRMARB_MSTR4_CFG0:
+ case CS40L2X_DSP1_STRMARB_MSTR4_CFG1:
+ case CS40L2X_DSP1_STRMARB_MSTR4_CFG2:
+ case CS40L2X_DSP1_STRMARB_MSTR5_CFG0:
+ case CS40L2X_DSP1_STRMARB_MSTR5_CFG1:
+ case CS40L2X_DSP1_STRMARB_MSTR5_CFG2:
+ case CS40L2X_DSP1_STRMARB_MSTR6_CFG0:
+ case CS40L2X_DSP1_STRMARB_MSTR6_CFG1:
+ case CS40L2X_DSP1_STRMARB_MSTR6_CFG2:
+ case CS40L2X_DSP1_STRMARB_MSTR7_CFG0:
+ case CS40L2X_DSP1_STRMARB_MSTR7_CFG1:
+ case CS40L2X_DSP1_STRMARB_MSTR7_CFG2:
+ case CS40L2X_DSP1_STRMARB_TX0_CFG0:
+ case CS40L2X_DSP1_STRMARB_TX0_CFG1:
+ case CS40L2X_DSP1_STRMARB_TX1_CFG0:
+ case CS40L2X_DSP1_STRMARB_TX1_CFG1:
+ case CS40L2X_DSP1_STRMARB_TX2_CFG0:
+ case CS40L2X_DSP1_STRMARB_TX2_CFG1:
+ case CS40L2X_DSP1_STRMARB_TX3_CFG0:
+ case CS40L2X_DSP1_STRMARB_TX3_CFG1:
+ case CS40L2X_DSP1_STRMARB_TX4_CFG0:
+ case CS40L2X_DSP1_STRMARB_TX4_CFG1:
+ case CS40L2X_DSP1_STRMARB_TX5_CFG0:
+ case CS40L2X_DSP1_STRMARB_TX5_CFG1:
+ case CS40L2X_DSP1_STRMARB_TX6_CFG0:
+ case CS40L2X_DSP1_STRMARB_TX6_CFG1:
+ case CS40L2X_DSP1_STRMARB_TX7_CFG0:
+ case CS40L2X_DSP1_STRMARB_TX7_CFG1:
+ case CS40L2X_DSP1_STRMARB_RX0_CFG0:
+ case CS40L2X_DSP1_STRMARB_RX0_CFG1:
+ case CS40L2X_DSP1_STRMARB_RX1_CFG0:
+ case CS40L2X_DSP1_STRMARB_RX1_CFG1:
+ case CS40L2X_DSP1_STRMARB_RX2_CFG0:
+ case CS40L2X_DSP1_STRMARB_RX2_CFG1:
+ case CS40L2X_DSP1_STRMARB_RX3_CFG0:
+ case CS40L2X_DSP1_STRMARB_RX3_CFG1:
+ case CS40L2X_DSP1_STRMARB_RX4_CFG0:
+ case CS40L2X_DSP1_STRMARB_RX4_CFG1:
+ case CS40L2X_DSP1_STRMARB_RX5_CFG0:
+ case CS40L2X_DSP1_STRMARB_RX5_CFG1:
+ case CS40L2X_DSP1_STRMARB_RX6_CFG0:
+ case CS40L2X_DSP1_STRMARB_RX6_CFG1:
+ case CS40L2X_DSP1_STRMARB_RX7_CFG0:
+ case CS40L2X_DSP1_STRMARB_RX7_CFG1:
+ case CS40L2X_DSP1_STRMARB_IRQ0_CFG0:
+ case CS40L2X_DSP1_STRMARB_IRQ0_CFG1:
+ case CS40L2X_DSP1_STRMARB_IRQ0_CFG2:
+ case CS40L2X_DSP1_STRMARB_IRQ1_CFG0:
+ case CS40L2X_DSP1_STRMARB_IRQ1_CFG1:
+ case CS40L2X_DSP1_STRMARB_IRQ1_CFG2:
+ case CS40L2X_DSP1_STRMARB_IRQ2_CFG0:
+ case CS40L2X_DSP1_STRMARB_IRQ2_CFG1:
+ case CS40L2X_DSP1_STRMARB_IRQ2_CFG2:
+ case CS40L2X_DSP1_STRMARB_IRQ3_CFG0:
+ case CS40L2X_DSP1_STRMARB_IRQ3_CFG1:
+ case CS40L2X_DSP1_STRMARB_IRQ3_CFG2:
+ case CS40L2X_DSP1_STRMARB_IRQ4_CFG0:
+ case CS40L2X_DSP1_STRMARB_IRQ4_CFG1:
+ case CS40L2X_DSP1_STRMARB_IRQ4_CFG2:
+ case CS40L2X_DSP1_STRMARB_IRQ5_CFG0:
+ case CS40L2X_DSP1_STRMARB_IRQ5_CFG1:
+ case CS40L2X_DSP1_STRMARB_IRQ5_CFG2:
+ case CS40L2X_DSP1_STRMARB_IRQ6_CFG0:
+ case CS40L2X_DSP1_STRMARB_IRQ6_CFG1:
+ case CS40L2X_DSP1_STRMARB_IRQ6_CFG2:
+ case CS40L2X_DSP1_STRMARB_IRQ7_CFG0:
+ case CS40L2X_DSP1_STRMARB_IRQ7_CFG1:
+ case CS40L2X_DSP1_STRMARB_IRQ7_CFG2:
+ case CS40L2X_DSP1_STRMARB_RESYNC_MSK:
+ case CS40L2X_DSP1_STRMARB_ERR_STATUS:
+ case CS40L2X_DSP1_INTPCTL_RES_STATIC:
+ case CS40L2X_DSP1_INTPCTL_RES_DYN:
+ case CS40L2X_DSP1_INTPCTL_NMI_CTRL:
+ case CS40L2X_DSP1_INTPCTL_IRQ_INV:
+ case CS40L2X_DSP1_INTPCTL_IRQ_MODE:
+ case CS40L2X_DSP1_INTPCTL_IRQ_EN:
+ case CS40L2X_DSP1_INTPCTL_IRQ_MSK:
+ case CS40L2X_DSP1_INTPCTL_IRQ_ERR:
+ case CS40L2X_DSP1_INTPCTL_IRQ_PEND:
+ case CS40L2X_DSP1_INTPCTL_TESTBITS:
+ case CS40L2X_DSP1_WDT_CONTROL:
+ case CS40L2X_DSP1_WDT_STATUS:
+ case CS40L2X_OTP_TRIM_1:
+ case CS40L2X_OTP_TRIM_2:
+ case CS40L2X_OTP_TRIM_3:
+ case CS40L2X_OTP_TRIM_4:
+ case CS40L2X_OTP_TRIM_5:
+ case CS40L2X_OTP_TRIM_6:
+ case CS40L2X_OTP_TRIM_7:
+ case CS40L2X_OTP_TRIM_8:
+ case CS40L2X_OTP_TRIM_9:
+ case CS40L2X_OTP_TRIM_10:
+ case CS40L2X_OTP_TRIM_11:
+ case CS40L2X_OTP_TRIM_12:
+ case CS40L2X_OTP_TRIM_13:
+ case CS40L2X_OTP_TRIM_14:
+ case CS40L2X_OTP_TRIM_15:
+ case CS40L2X_OTP_TRIM_16:
+ case CS40L2X_OTP_TRIM_17:
+ case CS40L2X_OTP_TRIM_18:
+ case CS40L2X_OTP_TRIM_19:
+ case CS40L2X_OTP_TRIM_20:
+ case CS40L2X_OTP_TRIM_21:
+ case CS40L2X_OTP_TRIM_22:
+ case CS40L2X_OTP_TRIM_23:
+ case CS40L2X_OTP_TRIM_24:
+ case CS40L2X_OTP_TRIM_25:
+ case CS40L2X_OTP_TRIM_26:
+ case CS40L2X_OTP_TRIM_27:
+ case CS40L2X_OTP_TRIM_28:
+ case CS40L2X_OTP_TRIM_29:
+ case CS40L2X_OTP_TRIM_30:
+ case CS40L2X_OTP_TRIM_31:
+ case CS40L2X_OTP_TRIM_32:
+ case CS40L2X_OTP_TRIM_33:
+ case CS40L2X_OTP_TRIM_34:
+ case CS40L2X_OTP_TRIM_35:
+ case CS40L2X_OTP_TRIM_36:
+ case CS40L2X_OTP_MEM0 ... CS40L2X_OTP_MEM31:
+ case CS40L2X_DSP1_XMEM_PACK_0 ... CS40L2X_DSP1_XMEM_PACK_3068:
+ case CS40L2X_DSP1_XMEM_UNPACK32_0 ... CS40L2X_DSP1_XMEM_UNPACK32_2046:
+ case CS40L2X_DSP1_XMEM_UNPACK24_0 ... CS40L2X_DSP1_XMEM_UNPACK24_4093:
+ case CS40L2X_DSP1_YMEM_PACK_0 ... CS40L2X_DSP1_YMEM_PACK_1532:
+ case CS40L2X_DSP1_YMEM_UNPACK32_0 ... CS40L2X_DSP1_YMEM_UNPACK32_1022:
+ case CS40L2X_DSP1_YMEM_UNPACK24_0 ... CS40L2X_DSP1_YMEM_UNPACK24_2045:
+ case CS40L2X_DSP1_PMEM_0 ... CS40L2X_DSP1_PMEM_5114:
+ return true;
+ default:
+ return false;
+ }
+}
+
+bool cs40l2x_precious_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case CS40L2X_OTP_MEM0 ... CS40L2X_OTP_MEM31:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static const struct cs40l2x_trim cs40l2x_trim_table_c[] = {
+ /* addr shift size */
+ {0x00002030, 0, 4}, /*TRIM_OSC_FREQ_TRIM*/
+ {0x00002030, 7, 1}, /*TRIM_OSC_TRIM_DONE*/
+ {0x0000208C, 24, 6}, /*TST_DIGREG_VREF_TRIM*/
+ {0x00002090, 14, 4}, /*TST_REF_TRIM*/
+ {0x00002090, 10, 4}, /*TST_REF_TEMPCO_TRIM*/
+ {0x0000300C, 11, 4}, /*PLL_LDOA_TST_VREF_TRIM*/
+ {0x0000394C, 23, 2}, /*BST_ATEST_CM_VOFF*/
+ {0x00003950, 0, 7}, /*BST_ATRIM_IADC_OFFSET*/
+ {0x00003950, 8, 7}, /*BST_ATRIM_IADC_GAIN1*/
+ {0x00003950, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET1*/
+ {0x00003950, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN1*/
+ {0x00003954, 0, 7}, /*BST_ATRIM_IADC_OFFSET2*/
+ {0x00003954, 8, 7}, /*BST_ATRIM_IADC_GAIN2*/
+ {0x00003954, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET2*/
+ {0x00003954, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN2*/
+ {0x00003958, 0, 7}, /*BST_ATRIM_IADC_OFFSET3*/
+ {0x00003958, 8, 7}, /*BST_ATRIM_IADC_GAIN3*/
+ {0x00003958, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET3*/
+ {0x00003958, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN3*/
+ {0x0000395C, 0, 7}, /*BST_ATRIM_IADC_OFFSET4*/
+ {0x0000395C, 8, 7}, /*BST_ATRIM_IADC_GAIN4*/
+ {0x0000395C, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET4*/
+ {0x0000395C, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN4*/
+ {0x0000416C, 0, 8}, /*VMON_GAIN_OTP_VAL*/
+ {0x00004160, 0, 7}, /*VMON_OFFSET_OTP_VAL*/
+ {0x0000416C, 8, 8}, /*IMON_GAIN_OTP_VAL*/
+ {0x00004160, 16, 10}, /*IMON_OFFSET_OTP_VAL*/
+ {0x0000416C, 16, 12}, /*VMON_CM_GAIN_OTP_VAL*/
+ {0x0000416C, 28, 1}, /*VMON_CM_GAIN_SIGN_OTP_VAL*/
+ {0x00004170, 0, 6}, /*IMON_CAL_TEMPCO_OTP_VAL*/
+ {0x00004170, 6, 1}, /*IMON_CAL_TEMPCO_SIGN_OTP*/
+ {0x00004170, 8, 6}, /*IMON_CAL_TEMPCO2_OTP_VAL*/
+ {0x00004170, 14, 1}, /*IMON_CAL_TEMPCO2_DN_UPB_OTP_VAL*/
+ {0x00004170, 16, 9}, /*IMON_CAL_TEMPCO_TBASE_OTP_VAL*/
+ {0x00004360, 0, 5}, /*TEMP_GAIN_OTP_VAL*/
+ {0x00004360, 6, 9}, /*TEMP_OFFSET_OTP_VAL*/
+ {0x00004448, 0, 8}, /*VP_SARADC_OFFSET*/
+ {0x00004448, 8, 8}, /*VP_GAIN_INDEX*/
+ {0x00004448, 16, 8}, /*VBST_SARADC_OFFSET*/
+ {0x00004448, 24, 8}, /*VBST_GAIN_INDEX*/
+ {0x0000444C, 0, 3}, /*ANA_SELINVREF*/
+ {0x00006E30, 0, 5}, /*GAIN_ERR_COEFF_0*/
+ {0x00006E30, 8, 5}, /*GAIN_ERR_COEFF_1*/
+ {0x00006E30, 16, 5}, /*GAIN_ERR_COEFF_2*/
+ {0x00006E30, 24, 5}, /*GAIN_ERR_COEFF_3*/
+ {0x00006E34, 0, 5}, /*GAIN_ERR_COEFF_4*/
+ {0x00006E34, 8, 5}, /*GAIN_ERR_COEFF_5*/
+ {0x00006E34, 16, 5}, /*GAIN_ERR_COEFF_6*/
+ {0x00006E34, 24, 5}, /*GAIN_ERR_COEFF_7*/
+ {0x00006E38, 0, 5}, /*GAIN_ERR_COEFF_8*/
+ {0x00006E38, 8, 5}, /*GAIN_ERR_COEFF_9*/
+ {0x00006E38, 16, 5}, /*GAIN_ERR_COEFF_10*/
+ {0x00006E38, 24, 5}, /*GAIN_ERR_COEFF_11*/
+ {0x00006E3C, 0, 5}, /*GAIN_ERR_COEFF_12*/
+ {0x00006E3C, 8, 5}, /*GAIN_ERR_COEFF_13*/
+ {0x00006E3C, 16, 5}, /*GAIN_ERR_COEFF_14*/
+ {0x00006E3C, 24, 5}, /*GAIN_ERR_COEFF_15*/
+ {0x00006E40, 0, 5}, /*GAIN_ERR_COEFF_16*/
+ {0x00006E40, 8, 5}, /*GAIN_ERR_COEFF_17*/
+ {0x00006E40, 16, 5}, /*GAIN_ERR_COEFF_18*/
+ {0x00006E40, 24, 5}, /*GAIN_ERR_COEFF_19*/
+ {0x00006E44, 0, 5}, /*GAIN_ERR_COEFF_20*/
+ {0x00006E48, 0, 10}, /*VOFF_GAIN_0*/
+ {0x00006E48, 10, 10}, /*VOFF_GAIN_1*/
+ {0x00006E48, 20, 10}, /*VOFF_GAIN_2*/
+ {0x00006E4C, 0, 10}, /*VOFF_GAIN_3*/
+ {0x00006E4C, 10, 10}, /*VOFF_GAIN_4*/
+ {0x00006E4C, 20, 10}, /*VOFF_GAIN_5*/
+ {0x00006E50, 0, 10}, /*VOFF_GAIN_6*/
+ {0x00006E50, 10, 10}, /*VOFF_GAIN_7*/
+ {0x00006E50, 20, 10}, /*VOFF_GAIN_8*/
+ {0x00006E54, 0, 10}, /*VOFF_GAIN_9*/
+ {0x00006E54, 10, 10}, /*VOFF_GAIN_10*/
+ {0x00006E54, 20, 10}, /*VOFF_GAIN_11*/
+ {0x00006E58, 0, 10}, /*VOFF_GAIN_12*/
+ {0x00006E58, 10, 10}, /*VOFF_GAIN_13*/
+ {0x00006E58, 20, 10}, /*VOFF_GAIN_14*/
+ {0x00006E5C, 0, 10}, /*VOFF_GAIN_15*/
+ {0x00006E5C, 10, 10}, /*VOFF_GAIN_16*/
+ {0x00006E5C, 20, 10}, /*VOFF_GAIN_17*/
+ {0x00006E60, 0, 10}, /*VOFF_GAIN_18*/
+ {0x00006E60, 10, 10}, /*VOFF_GAIN_19*/
+ {0x00006E60, 20, 10}, /*VOFF_GAIN_20*/
+ {0x00006E64, 0, 10}, /*VOFF_INT1*/
+ {0x00007418, 7, 5}, /*DS_SPK_INT1_CAP_TRIM*/
+ {0x0000741C, 0, 5}, /*DS_SPK_INT2_CAP_TRIM*/
+ {0x0000741C, 11, 4}, /*DS_SPK_LPF_CAP_TRIM*/
+ {0x0000741C, 19, 4}, /*DS_SPK_QUAN_CAP_TRIM*/
+ {0x00007434, 17, 1}, /*FORCE_CAL*/
+ {0x00007434, 18, 7}, /*CAL_OVERRIDE*/
+ {0x00007068, 0, 9}, /*MODIX*/
+ {0x0000410C, 7, 1}, /*VIMON_DLY_NOT_COMB*/
+ {0x0000400C, 0, 7}, /*VIMON_DLY*/
+ {0x00000000, 0, 1}, /*extra bit*/
+ {0x00017040, 0, 8}, /*X_COORDINATE*/
+ {0x00017040, 8, 8}, /*Y_COORDINATE*/
+ {0x00017040, 16, 8}, /*WAFER_ID*/
+ {0x00017040, 24, 8}, /*DVS*/
+ {0x00017044, 0, 24}, /*LOT_NUMBER*/
+};
+
+static const struct cs40l2x_trim cs40l2x_trim_table_d[] = {
+ /* addr shift size */
+ {0x00002030, 0, 4}, /*TRIM_OSC_FREQ_TRIM*/
+ {0x00002030, 7, 1}, /*TRIM_OSC_TRIM_DONE*/
+ {0x0000208C, 24, 6}, /*TST_DIGREG_VREF_TRIM*/
+ {0x00002090, 14, 4}, /*TST_REF_TRIM*/
+ {0x00002090, 10, 4}, /*TST_REF_TEMPCO_TRIM*/
+ {0x0000300C, 11, 4}, /*PLL_LDOA_TST_VREF_TRIM*/
+ {0x0000394C, 23, 2}, /*BST_ATEST_CM_VOFF*/
+ {0x00003950, 0, 7}, /*BST_ATRIM_IADC_OFFSET*/
+ {0x00003950, 8, 7}, /*BST_ATRIM_IADC_GAIN1*/
+ {0x00003950, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET1*/
+ {0x00003950, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN1*/
+ {0x00003954, 0, 7}, /*BST_ATRIM_IADC_OFFSET2*/
+ {0x00003954, 8, 7}, /*BST_ATRIM_IADC_GAIN2*/
+ {0x00003954, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET2*/
+ {0x00003954, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN2*/
+ {0x00003958, 0, 7}, /*BST_ATRIM_IADC_OFFSET3*/
+ {0x00003958, 8, 7}, /*BST_ATRIM_IADC_GAIN3*/
+ {0x00003958, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET3*/
+ {0x00003958, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN3*/
+ {0x0000395C, 0, 7}, /*BST_ATRIM_IADC_OFFSET4*/
+ {0x0000395C, 8, 7}, /*BST_ATRIM_IADC_GAIN4*/
+ {0x0000395C, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET4*/
+ {0x0000395C, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN4*/
+ {0x0000416C, 0, 8}, /*VMON_GAIN_OTP_VAL*/
+ {0x00004160, 0, 7}, /*VMON_OFFSET_OTP_VAL*/
+ {0x0000416C, 8, 8}, /*IMON_GAIN_OTP_VAL*/
+ {0x00004160, 16, 10}, /*IMON_OFFSET_OTP_VAL*/
+ {0x0000416C, 16, 12}, /*VMON_CM_GAIN_OTP_VAL*/
+ {0x0000416C, 28, 1}, /*VMON_CM_GAIN_SIGN_OTP_VAL*/
+ {0x00004170, 0, 6}, /*IMON_CAL_TEMPCO_OTP_VAL*/
+ {0x00004170, 6, 1}, /*IMON_CAL_TEMPCO_SIGN_OTP*/
+ {0x00004170, 8, 6}, /*IMON_CAL_TEMPCO2_OTP_VAL*/
+ {0x00004170, 14, 1}, /*IMON_CAL_TEMPCO2_DN_UPB_OTP_VAL*/
+ {0x00004170, 16, 9}, /*IMON_CAL_TEMPCO_TBASE_OTP_VAL*/
+ {0x00004360, 0, 5}, /*TEMP_GAIN_OTP_VAL*/
+ {0x00004360, 6, 9}, /*TEMP_OFFSET_OTP_VAL*/
+ {0x00004448, 0, 8}, /*VP_SARADC_OFFSET*/
+ {0x00004448, 8, 8}, /*VP_GAIN_INDEX*/
+ {0x00004448, 16, 8}, /*VBST_SARADC_OFFSET*/
+ {0x00004448, 24, 8}, /*VBST_GAIN_INDEX*/
+ {0x0000444C, 0, 3}, /*ANA_SELINVREF*/
+ {0x00006E30, 0, 5}, /*GAIN_ERR_COEFF_0*/
+ {0x00006E30, 8, 5}, /*GAIN_ERR_COEFF_1*/
+ {0x00006E30, 16, 5}, /*GAIN_ERR_COEFF_2*/
+ {0x00006E30, 24, 5}, /*GAIN_ERR_COEFF_3*/
+ {0x00006E34, 0, 5}, /*GAIN_ERR_COEFF_4*/
+ {0x00006E34, 8, 5}, /*GAIN_ERR_COEFF_5*/
+ {0x00006E34, 16, 5}, /*GAIN_ERR_COEFF_6*/
+ {0x00006E34, 24, 5}, /*GAIN_ERR_COEFF_7*/
+ {0x00006E38, 0, 5}, /*GAIN_ERR_COEFF_8*/
+ {0x00006E38, 8, 5}, /*GAIN_ERR_COEFF_9*/
+ {0x00006E38, 16, 5}, /*GAIN_ERR_COEFF_10*/
+ {0x00006E38, 24, 5}, /*GAIN_ERR_COEFF_11*/
+ {0x00006E3C, 0, 5}, /*GAIN_ERR_COEFF_12*/
+ {0x00006E3C, 8, 5}, /*GAIN_ERR_COEFF_13*/
+ {0x00006E3C, 16, 5}, /*GAIN_ERR_COEFF_14*/
+ {0x00006E3C, 24, 5}, /*GAIN_ERR_COEFF_15*/
+ {0x00006E40, 0, 5}, /*GAIN_ERR_COEFF_16*/
+ {0x00006E40, 8, 5}, /*GAIN_ERR_COEFF_17*/
+ {0x00006E40, 16, 5}, /*GAIN_ERR_COEFF_18*/
+ {0x00006E40, 24, 5}, /*GAIN_ERR_COEFF_19*/
+ {0x00006E44, 0, 5}, /*GAIN_ERR_COEFF_20*/
+ {0x00006E48, 0, 10}, /*VOFF_GAIN_0*/
+ {0x00006E48, 10, 10}, /*VOFF_GAIN_1*/
+ {0x00006E48, 20, 10}, /*VOFF_GAIN_2*/
+ {0x00006E4C, 0, 10}, /*VOFF_GAIN_3*/
+ {0x00006E4C, 10, 10}, /*VOFF_GAIN_4*/
+ {0x00006E4C, 20, 10}, /*VOFF_GAIN_5*/
+ {0x00006E50, 0, 10}, /*VOFF_GAIN_6*/
+ {0x00006E50, 10, 10}, /*VOFF_GAIN_7*/
+ {0x00006E50, 20, 10}, /*VOFF_GAIN_8*/
+ {0x00006E54, 0, 10}, /*VOFF_GAIN_9*/
+ {0x00006E54, 10, 10}, /*VOFF_GAIN_10*/
+ {0x00006E54, 20, 10}, /*VOFF_GAIN_11*/
+ {0x00006E58, 0, 10}, /*VOFF_GAIN_12*/
+ {0x00006E58, 10, 10}, /*VOFF_GAIN_13*/
+ {0x00006E58, 20, 10}, /*VOFF_GAIN_14*/
+ {0x00006E5C, 0, 10}, /*VOFF_GAIN_15*/
+ {0x00006E5C, 10, 10}, /*VOFF_GAIN_16*/
+ {0x00006E5C, 20, 10}, /*VOFF_GAIN_17*/
+ {0x00006E60, 0, 10}, /*VOFF_GAIN_18*/
+ {0x00006E60, 10, 10}, /*VOFF_GAIN_19*/
+ {0x00006E60, 20, 10}, /*VOFF_GAIN_20*/
+ {0x00006E64, 0, 10}, /*VOFF_INT1*/
+ {0x00007418, 7, 5}, /*DS_SPK_INT1_CAP_TRIM*/
+ {0x0000741C, 0, 5}, /*DS_SPK_INT2_CAP_TRIM*/
+ {0x0000741C, 11, 4}, /*DS_SPK_LPF_CAP_TRIM*/
+ {0x0000741C, 19, 4}, /*DS_SPK_QUAN_CAP_TRIM*/
+ {0x00007434, 17, 1}, /*FORCE_CAL*/
+ {0x00007434, 18, 7}, /*CAL_OVERRIDE*/
+ {0x00007068, 0, 9}, /*MODIX*/
+ {0x0000410C, 7, 1}, /*VIMON_DLY_NOT_COMB*/
+ {0x0000400C, 0, 7}, /*VIMON_DLY*/
+ {0x00004000, 11, 1}, /*VMON_POL*/
+ {0x00017040, 0, 8}, /*X_COORDINATE*/
+ {0x00017040, 8, 8}, /*Y_COORDINATE*/
+ {0x00017040, 16, 8}, /*WAFER_ID*/
+ {0x00017040, 24, 8}, /*DVS*/
+ {0x00017044, 0, 24}, /*LOT_NUMBER*/
+};
+
+static const struct cs40l2x_trim cs40l2x_trim_table_e[] = {
+ /* addr shift size */
+ {0x00002030, 0, 4}, /*TRIM_OSC_FREQ_TRIM*/
+ {0x00002030, 7, 1}, /*TRIM_OSC_TRIM_DONE*/
+ {0x0000208C, 24, 6}, /*TST_DIGREG_VREF_TRIM*/
+ {0x00002090, 14, 4}, /*TST_REF_TRIM*/
+ {0x00002090, 10, 4}, /*TST_REF_TEMPCO_TRIM*/
+ {0x0000300C, 11, 4}, /*PLL_LDOA_TST_VREF_TRIM*/
+ {0x0000394C, 23, 2}, /*BST_ATEST_CM_VOFF*/
+ {0x00003950, 0, 7}, /*BST_ATRIM_IADC_OFFSET*/
+ {0x00003950, 8, 7}, /*BST_ATRIM_IADC_GAIN1*/
+ {0x00003950, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET1*/
+ {0x00003950, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN1*/
+ {0x00003954, 0, 7}, /*BST_ATRIM_IADC_OFFSET2*/
+ {0x00003954, 8, 7}, /*BST_ATRIM_IADC_GAIN2*/
+ {0x00003954, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET2*/
+ {0x00003954, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN2*/
+ {0x00003958, 0, 7}, /*BST_ATRIM_IADC_OFFSET3*/
+ {0x00003958, 8, 7}, /*BST_ATRIM_IADC_GAIN3*/
+ {0x00003958, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET3*/
+ {0x00003958, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN3*/
+ {0x0000395C, 0, 7}, /*BST_ATRIM_IADC_OFFSET4*/
+ {0x0000395C, 8, 7}, /*BST_ATRIM_IADC_GAIN4*/
+ {0x0000395C, 16, 8}, /*BST_ATRIM_IPKCOMP_OFFSET4*/
+ {0x0000395C, 24, 8}, /*BST_ATRIM_IPKCOMP_GAIN4*/
+ {0x0000416C, 0, 8}, /*VMON_GAIN_OTP_VAL*/
+ {0x00004160, 0, 7}, /*VMON_OFFSET_OTP_VAL*/
+ {0x0000416C, 8, 8}, /*IMON_GAIN_OTP_VAL*/
+ {0x00004160, 16, 10}, /*IMON_OFFSET_OTP_VAL*/
+ {0x0000416C, 16, 12}, /*VMON_CM_GAIN_OTP_VAL*/
+ {0x0000416C, 28, 1}, /*VMON_CM_GAIN_SIGN_OTP_VAL*/
+ {0x00004170, 0, 6}, /*IMON_CAL_TEMPCO_OTP_VAL*/
+ {0x00004170, 6, 1}, /*IMON_CAL_TEMPCO_SIGN_OTP*/
+ {0x00004170, 8, 6}, /*IMON_CAL_TEMPCO2_OTP_VAL*/
+ {0x00004170, 14, 1}, /*IMON_CAL_TEMPCO2_DN_UPB_OTP_VAL*/
+ {0x00004170, 16, 9}, /*IMON_CAL_TEMPCO_TBASE_OTP_VAL*/
+ {0x00004360, 0, 5}, /*TEMP_GAIN_OTP_VAL*/
+ {0x00004360, 6, 9}, /*TEMP_OFFSET_OTP_VAL*/
+ {0x00004448, 0, 8}, /*VP_SARADC_OFFSET*/
+ {0x00004448, 8, 8}, /*VP_GAIN_INDEX*/
+ {0x00004448, 16, 8}, /*VBST_SARADC_OFFSET*/
+ {0x00004448, 24, 8}, /*VBST_GAIN_INDEX*/
+ {0x0000444C, 0, 3}, /*ANA_SELINVREF*/
+ {0x00006E30, 0, 5}, /*GAIN_ERR_COEFF_0*/
+ {0x00006E30, 8, 5}, /*GAIN_ERR_COEFF_1*/
+ {0x00006E30, 16, 5}, /*GAIN_ERR_COEFF_2*/
+ {0x00006E30, 24, 5}, /*GAIN_ERR_COEFF_3*/
+ {0x00006E34, 0, 5}, /*GAIN_ERR_COEFF_4*/
+ {0x00006E34, 8, 5}, /*GAIN_ERR_COEFF_5*/
+ {0x00006E34, 16, 5}, /*GAIN_ERR_COEFF_6*/
+ {0x00006E34, 24, 5}, /*GAIN_ERR_COEFF_7*/
+ {0x00006E38, 0, 5}, /*GAIN_ERR_COEFF_8*/
+ {0x00006E38, 8, 5}, /*GAIN_ERR_COEFF_9*/
+ {0x00006E38, 16, 5}, /*GAIN_ERR_COEFF_10*/
+ {0x00006E38, 24, 5}, /*GAIN_ERR_COEFF_11*/
+ {0x00006E3C, 0, 5}, /*GAIN_ERR_COEFF_12*/
+ {0x00006E3C, 8, 5}, /*GAIN_ERR_COEFF_13*/
+ {0x00006E3C, 16, 5}, /*GAIN_ERR_COEFF_14*/
+ {0x00006E3C, 24, 5}, /*GAIN_ERR_COEFF_15*/
+ {0x00006E40, 0, 5}, /*GAIN_ERR_COEFF_16*/
+ {0x00006E40, 8, 5}, /*GAIN_ERR_COEFF_17*/
+ {0x00006E40, 16, 5}, /*GAIN_ERR_COEFF_18*/
+ {0x00006E40, 24, 5}, /*GAIN_ERR_COEFF_19*/
+ {0x00006E44, 0, 5}, /*GAIN_ERR_COEFF_20*/
+ {0x00006E48, 0, 10}, /*VOFF_GAIN_0*/
+ {0x00006E48, 10, 10}, /*VOFF_GAIN_1*/
+ {0x00006E48, 20, 10}, /*VOFF_GAIN_2*/
+ {0x00006E4C, 0, 10}, /*VOFF_GAIN_3*/
+ {0x00006E4C, 10, 10}, /*VOFF_GAIN_4*/
+ {0x00006E4C, 20, 10}, /*VOFF_GAIN_5*/
+ {0x00006E50, 0, 10}, /*VOFF_GAIN_6*/
+ {0x00006E50, 10, 10}, /*VOFF_GAIN_7*/
+ {0x00006E50, 20, 10}, /*VOFF_GAIN_8*/
+ {0x00006E54, 0, 10}, /*VOFF_GAIN_9*/
+ {0x00006E54, 10, 10}, /*VOFF_GAIN_10*/
+ {0x00006E54, 20, 10}, /*VOFF_GAIN_11*/
+ {0x00006E58, 0, 10}, /*VOFF_GAIN_12*/
+ {0x00006E58, 10, 10}, /*VOFF_GAIN_13*/
+ {0x00006E58, 20, 10}, /*VOFF_GAIN_14*/
+ {0x00006E5C, 0, 10}, /*VOFF_GAIN_15*/
+ {0x00006E5C, 10, 10}, /*VOFF_GAIN_16*/
+ {0x00006E5C, 20, 10}, /*VOFF_GAIN_17*/
+ {0x00006E60, 0, 10}, /*VOFF_GAIN_18*/
+ {0x00006E60, 10, 10}, /*VOFF_GAIN_19*/
+ {0x00006E60, 20, 10}, /*VOFF_GAIN_20*/
+ {0x00006E64, 0, 10}, /*VOFF_INT1*/
+ {0x00007418, 7, 5}, /*DS_SPK_INT1_CAP_TRIM*/
+ {0x0000741C, 0, 5}, /*DS_SPK_INT2_CAP_TRIM*/
+ {0x0000741C, 11, 4}, /*DS_SPK_LPF_CAP_TRIM*/
+ {0x0000741C, 19, 4}, /*DS_SPK_QUAN_CAP_TRIM*/
+ {0x00007434, 17, 1}, /*FORCE_CAL*/
+ {0x00007434, 18, 7}, /*CAL_OVERRIDE*/
+ {0x00007068, 0, 9}, /*MODIX*/
+ {0x0000410C, 7, 1}, /*VIMON_DLY_NOT_COMB*/
+ {0x0000400C, 0, 7}, /*VIMON_DLY*/
+ {0x00004000, 11, 1}, /*VMON_POL*/
+ {0x00017040, 0, 8}, /*X_COORDINATE*/
+ {0x00017040, 8, 8}, /*Y_COORDINATE*/
+ {0x00017040, 16, 8}, /*WAFER_ID*/
+ {0x00017040, 24, 8}, /*DVS*/
+ {0x00017044, 0, 24}, /*LOT_NUMBER*/
+ {0x00003010, 2, 6}, /*PLL_DCO_CAL_TRIM*/
+};
+
+const struct cs40l2x_otp_desc cs40l2x_otp_map[CS40L2X_NUM_OTP_MAPS] = {
+ {
+ .id = 0xC,
+ .row_start = 2,
+ .col_start = 16,
+ .num_trims = ARRAY_SIZE(cs40l2x_trim_table_c),
+ .trim_table = cs40l2x_trim_table_c,
+ },
+ {
+ .id = 0xD,
+ .row_start = 2,
+ .col_start = 16,
+ .num_trims = ARRAY_SIZE(cs40l2x_trim_table_d),
+ .trim_table = cs40l2x_trim_table_d,
+ },
+ {
+ .id = 0xE,
+ .row_start = 2,
+ .col_start = 16,
+ .num_trims = ARRAY_SIZE(cs40l2x_trim_table_e),
+ .trim_table = cs40l2x_trim_table_e,
+ },
+};
+
+const unsigned int cs40l2x_pbq_dig_scale[CS40L2X_PBQ_SCALE_MAX + 1] = {
+ 0,
+ 320, /* 1% */
+ 272,
+ 244,
+ 224,
+ 208,
+ 195,
+ 185,
+ 176,
+ 167,
+ 160, /* 10% */
+ 153,
+ 147,
+ 142,
+ 137,
+ 132,
+ 127,
+ 123,
+ 119,
+ 115,
+ 112, /* 20% */
+ 108,
+ 105,
+ 102,
+ 99,
+ 96,
+ 94,
+ 91,
+ 88,
+ 86,
+ 84, /* 30% */
+ 81,
+ 79,
+ 77,
+ 75,
+ 73,
+ 71,
+ 69,
+ 67,
+ 65,
+ 64, /* 40% */
+ 62,
+ 60,
+ 59,
+ 57,
+ 55,
+ 54,
+ 52,
+ 51,
+ 50,
+ 48, /* 50% */
+ 47,
+ 45,
+ 44,
+ 43,
+ 42,
+ 40,
+ 39,
+ 38,
+ 37,
+ 35, /* 60% */
+ 34,
+ 33,
+ 32,
+ 31,
+ 30,
+ 29,
+ 28,
+ 27,
+ 26,
+ 25, /* 70% */
+ 24,
+ 23,
+ 22,
+ 21,
+ 20,
+ 19,
+ 18,
+ 17,
+ 16,
+ 16, /* 80% */
+ 15,
+ 14,
+ 13,
+ 12,
+ 11,
+ 10,
+ 10,
+ 9,
+ 8,
+ 7, /* 90% */
+ 7,
+ 6,
+ 5,
+ 4,
+ 4,
+ 3,
+ 2,
+ 1,
+ 1,
+ 0, /* 100% */
+};
diff --git a/drivers/misc/cs40l2x.c b/drivers/misc/cs40l2x.c
new file mode 100644
index 0000000..eba52b9
--- /dev/null
+++ b/drivers/misc/cs40l2x.c
@@ -0,0 +1,3107 @@
+/*
+ * cs40l2x.c -- CS40L20/CS40L25/CS40L25A/CS40L25B Haptics Driver
+ *
+ * Copyright 2018 Cirrus Logic, Inc.
+ *
+ * Author: Jeff LaBundy <jeff.labundy@cirrus.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/moduleparam.h>
+#include <linux/version.h>
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/i2c.h>
+#include <linux/slab.h>
+#include <linux/list.h>
+#include <linux/string.h>
+#include <linux/workqueue.h>
+#include <linux/regulator/consumer.h>
+#include <linux/regmap.h>
+#include <linux/sysfs.h>
+#include <linux/firmware.h>
+#include <linux/gpio.h>
+#include <linux/gpio/consumer.h>
+#include <linux/delay.h>
+#include <linux/hrtimer.h>
+#include <linux/of_device.h>
+#include <linux/leds.h>
+#include <linux/platform_device.h>
+#include <linux/platform_data/cs40l2x.h>
+
+#include "cs40l2x.h"
+
+struct cs40l2x_private {
+ struct device *dev;
+ struct regmap *regmap;
+ struct regulator_bulk_data supplies[2];
+ unsigned int num_supplies;
+ unsigned int devid;
+ unsigned int revid;
+ struct work_struct vibe_start_work;
+ struct work_struct vibe_pbq_work;
+ struct work_struct vibe_stop_work;
+ struct workqueue_struct *vibe_workqueue;
+ struct mutex lock;
+ unsigned int cp_trigger_index;
+ unsigned int cp_trailer_index;
+ unsigned int num_waves;
+ unsigned int vibegen_id;
+ unsigned int vibegen_rev;
+ unsigned int wt_limit_xm;
+ unsigned int wt_limit_ym;
+ bool vibe_init_success;
+ struct gpio_desc *reset_gpio;
+ struct cs40l2x_platform_data pdata;
+ struct list_head coeff_desc_head;
+ unsigned char diag_state;
+ unsigned int diag_dig_scale;
+ unsigned int f0_measured;
+ unsigned int redc_measured;
+ struct cs40l2x_pbq_pair pbq_pairs[CS40L2X_PBQ_DEPTH_MAX];
+ struct hrtimer pbq_timer;
+ unsigned int pbq_depth;
+ unsigned int pbq_index;
+ unsigned int pbq_state;
+ unsigned int pbq_cp_dig_scale;
+ int pbq_repeat;
+ int pbq_remain;
+ struct led_classdev led_dev;
+};
+
+static const char * const cs40l2x_supplies[] = {
+ "VA",
+ "VP",
+};
+
+static const char * const cs40l2x_part_nums[] = {
+ "CS40L20",
+ "CS40L25",
+ "CS40L25A",
+ "CS40L25B",
+};
+
+static struct cs40l2x_private *cs40l2x_get_private(struct device *dev)
+{
+ return dev_get_drvdata(dev);
+}
+
+static ssize_t cs40l2x_cp_trigger_index_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", cs40l2x->cp_trigger_index);
+}
+
+static ssize_t cs40l2x_cp_trigger_index_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int index;
+
+ ret = kstrtou32(buf, 10, &index);
+ if (ret)
+ return -EINVAL;
+
+ if ((index & CS40L2X_INDEX_MASK) >= cs40l2x->num_waves
+ && index != CS40L2X_INDEX_PBQ
+ && index != CS40L2X_INDEX_DIAG)
+ return -EINVAL;
+
+ cs40l2x->cp_trigger_index = index;
+
+ return count;
+}
+
+static ssize_t cs40l2x_cp_trigger_queue_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ ssize_t len = 0;
+ int i;
+
+ if (cs40l2x->pbq_depth == 0)
+ return -ENODATA;
+
+ mutex_lock(&cs40l2x->lock);
+
+ for (i = 0; i < cs40l2x->pbq_depth; i++) {
+ switch (cs40l2x->pbq_pairs[i].tag) {
+ case CS40L2X_PBQ_TAG_SILENCE:
+ len += snprintf(buf + len, PAGE_SIZE - len, "%d",
+ cs40l2x->pbq_pairs[i].mag);
+ break;
+ case CS40L2X_PBQ_TAG_START:
+ len += snprintf(buf + len, PAGE_SIZE - len, "!!");
+ break;
+ case CS40L2X_PBQ_TAG_STOP:
+ len += snprintf(buf + len, PAGE_SIZE - len, "%d!!",
+ cs40l2x->pbq_pairs[i].repeat);
+ break;
+ default:
+ len += snprintf(buf + len, PAGE_SIZE - len, "%d.%d",
+ cs40l2x->pbq_pairs[i].tag,
+ cs40l2x->pbq_pairs[i].mag);
+ }
+
+ if (i < (cs40l2x->pbq_depth - 1))
+ len += snprintf(buf + len, PAGE_SIZE - len, ", ");
+ }
+
+ switch (cs40l2x->pbq_repeat) {
+ case -1:
+ len += snprintf(buf + len, PAGE_SIZE - len, ", ~\n");
+ break;
+ case 0:
+ len += snprintf(buf + len, PAGE_SIZE - len, "\n");
+ break;
+ default:
+ len += snprintf(buf + len, PAGE_SIZE - len, ", %d!\n",
+ cs40l2x->pbq_repeat);
+ }
+
+ mutex_unlock(&cs40l2x->lock);
+
+ return len;
+}
+
+static ssize_t cs40l2x_cp_trigger_queue_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ char *pbq_str_alloc, *pbq_str, *pbq_str_tok;
+ char *pbq_seg_alloc, *pbq_seg, *pbq_seg_tok;
+ size_t pbq_seg_len;
+ unsigned int pbq_depth = 0;
+ unsigned int val, num_empty;
+ int pbq_marker = -1;
+ int ret;
+
+ pbq_str_alloc = kzalloc(count, GFP_KERNEL);
+ if (!pbq_str_alloc)
+ return -ENOMEM;
+
+ pbq_seg_alloc = kzalloc(CS40L2X_PBQ_SEG_LEN_MAX + 1, GFP_KERNEL);
+ if (!pbq_seg_alloc) {
+ kfree(pbq_str_alloc);
+ return -ENOMEM;
+ }
+
+ mutex_lock(&cs40l2x->lock);
+
+ cs40l2x->pbq_depth = 0;
+ cs40l2x->pbq_repeat = 0;
+
+ pbq_str = pbq_str_alloc;
+ strlcpy(pbq_str, buf, count);
+
+ pbq_str_tok = strsep(&pbq_str, ",");
+
+ while (pbq_str_tok) {
+ pbq_seg = pbq_seg_alloc;
+ pbq_seg_len = strlcpy(pbq_seg, strim(pbq_str_tok),
+ CS40L2X_PBQ_SEG_LEN_MAX + 1);
+ if (pbq_seg_len > CS40L2X_PBQ_SEG_LEN_MAX) {
+ ret = -E2BIG;
+ goto err_mutex;
+ }
+
+ /* waveform specifier */
+ if (strnchr(pbq_seg, CS40L2X_PBQ_SEG_LEN_MAX, '.')) {
+ /* index */
+ pbq_seg_tok = strsep(&pbq_seg, ".");
+
+ ret = kstrtou32(pbq_seg_tok, 10, &val);
+ if (ret) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+ if (val == 0 || val >= cs40l2x->num_waves) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+ cs40l2x->pbq_pairs[pbq_depth].tag = val;
+
+ /* scale */
+ pbq_seg_tok = strsep(&pbq_seg, ".");
+
+ ret = kstrtou32(pbq_seg_tok, 10, &val);
+ if (ret) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+ if (val == 0 || val > CS40L2X_PBQ_SCALE_MAX) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+ cs40l2x->pbq_pairs[pbq_depth++].mag = val;
+
+ /* repetition specifier */
+ } else if (strnchr(pbq_seg, CS40L2X_PBQ_SEG_LEN_MAX, '!')) {
+ val = 0;
+ num_empty = 0;
+ pbq_seg_tok = strsep(&pbq_seg, "!");
+
+ while (pbq_seg_tok) {
+ if (strnlen(pbq_seg_tok,
+ CS40L2X_PBQ_SEG_LEN_MAX)) {
+ ret = kstrtou32(pbq_seg_tok, 10, &val);
+ if (ret) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+ if (val > CS40L2X_PBQ_REPEAT_MAX) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+ } else {
+ num_empty++;
+ }
+
+ pbq_seg_tok = strsep(&pbq_seg, "!");
+ }
+ /* number of empty tokens reveals specifier type */
+ switch (num_empty) {
+ case 1: /* outer loop: "n!" or "!n" */
+ if (cs40l2x->pbq_repeat) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+ cs40l2x->pbq_repeat = val;
+ break;
+
+ case 2: /* inner loop stop: "n!!" or "!!n" */
+ if (pbq_marker < 0) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+
+ cs40l2x->pbq_pairs[pbq_depth].tag =
+ CS40L2X_PBQ_TAG_STOP;
+ cs40l2x->pbq_pairs[pbq_depth].mag = pbq_marker;
+ cs40l2x->pbq_pairs[pbq_depth++].repeat = val;
+ pbq_marker = -1;
+ break;
+
+ case 3: /* inner loop start: "!!" */
+ if (pbq_marker >= 0) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+
+ cs40l2x->pbq_pairs[pbq_depth].tag =
+ CS40L2X_PBQ_TAG_START;
+ pbq_marker = pbq_depth++;
+ break;
+
+ default:
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+
+ /* loop specifier */
+ } else if (strnchr(pbq_seg, CS40L2X_PBQ_SEG_LEN_MAX, '~')) {
+ if (cs40l2x->pbq_repeat) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+ cs40l2x->pbq_repeat = -1;
+
+ /* duration specifier */
+ } else {
+ cs40l2x->pbq_pairs[pbq_depth].tag =
+ CS40L2X_PBQ_TAG_SILENCE;
+
+ ret = kstrtou32(pbq_seg, 10, &val);
+ if (ret) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+ if (val > CS40L2X_PBQ_DELAY_MAX) {
+ ret = -EINVAL;
+ goto err_mutex;
+ }
+ cs40l2x->pbq_pairs[pbq_depth++].mag = val;
+ }
+
+ if (pbq_depth == CS40L2X_PBQ_DEPTH_MAX) {
+ ret = -E2BIG;
+ goto err_mutex;
+ }
+
+ pbq_str_tok = strsep(&pbq_str, ",");
+ }
+
+ cs40l2x->pbq_depth = pbq_depth;
+ ret = count;
+
+err_mutex:
+ mutex_unlock(&cs40l2x->lock);
+
+ kfree(pbq_str_alloc);
+ kfree(pbq_seg_alloc);
+
+ return ret;
+}
+
+static unsigned int cs40l2x_dsp_reg(struct cs40l2x_private *cs40l2x,
+ const char *coeff_name, const unsigned char block_type)
+{
+ struct cs40l2x_coeff_desc *coeff_desc;
+
+ list_for_each_entry(coeff_desc, &cs40l2x->coeff_desc_head, list) {
+ if (strcmp(coeff_desc->name, coeff_name))
+ continue;
+ if (coeff_desc->block_type != block_type)
+ continue;
+
+ return coeff_desc->reg;
+ }
+
+ /* return an identifiable register that is known to be read-only */
+ return CS40L2X_DEVID;
+}
+
+static ssize_t cs40l2x_gpio1_enable_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "GPIO_ENABLE",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read GPIO1 configuration\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t cs40l2x_gpio1_enable_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ ret = kstrtou32(buf, 10, &val);
+ if (ret)
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "GPIO_ENABLE",
+ CS40L2X_XM_UNPACKED_TYPE),
+ val ? CS40L2X_GPIO1_ENABLED : CS40L2X_GPIO1_DISABLED);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to write GPIO1 configuration\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static ssize_t cs40l2x_gpio1_rise_index_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "INDEXBUTTONPRESS",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read index\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t cs40l2x_gpio1_rise_index_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int index;
+
+ ret = kstrtou32(buf, 10, &index);
+ if (ret)
+ return -EINVAL;
+
+ if (index > (cs40l2x->num_waves - 1))
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "INDEXBUTTONPRESS",
+ CS40L2X_XM_UNPACKED_TYPE), index);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to write index\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static ssize_t cs40l2x_gpio1_fall_index_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "INDEXBUTTONRELEASE",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read index\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t cs40l2x_gpio1_fall_index_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int index;
+
+ ret = kstrtou32(buf, 10, &index);
+ if (ret)
+ return -EINVAL;
+
+ if (index > (cs40l2x->num_waves - 1))
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "INDEXBUTTONRELEASE",
+ CS40L2X_XM_UNPACKED_TYPE), index);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to write index\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static ssize_t cs40l2x_gpio1_fall_timeout_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "PRESS_RELEASE_TIMEOUT",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read GPIO1 falling-edge timeout\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t cs40l2x_gpio1_fall_timeout_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ ret = kstrtou32(buf, 10, &val);
+ if (ret)
+ return -EINVAL;
+
+ if (val > CS40L2X_PR_TIMEOUT_MAX)
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "PRESS_RELEASE_TIMEOUT",
+ CS40L2X_XM_UNPACKED_TYPE), val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to write GPIO1 falling-edge timeout\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static ssize_t cs40l2x_standby_timeout_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "EVENT_TIMEOUT",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read standby timeout\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t cs40l2x_standby_timeout_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ ret = kstrtou32(buf, 10, &val);
+ if (ret)
+ return -EINVAL;
+
+ if (val > CS40L2X_EVENT_TIMEOUT_MAX)
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "EVENT_TIMEOUT",
+ CS40L2X_XM_UNPACKED_TYPE), val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to write standby timeout\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static ssize_t cs40l2x_f0_measured_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ unsigned char diag_state;
+ unsigned int f0_measured;
+
+ mutex_lock(&cs40l2x->lock);
+
+ diag_state = cs40l2x->diag_state;
+ f0_measured = cs40l2x->f0_measured;
+
+ mutex_unlock(&cs40l2x->lock);
+
+ if (diag_state != CS40L2X_DIAG_STATE_DONE)
+ return -ENODATA;
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", f0_measured);
+}
+
+static ssize_t cs40l2x_f0_stored_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "F0_STORED",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read stored f0\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t cs40l2x_f0_stored_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ ret = kstrtou32(buf, 10, &val);
+ if (ret)
+ return -EINVAL;
+
+ if (cs40l2x->pdata.f0_min > 0 && val < cs40l2x->pdata.f0_min)
+ return -EINVAL;
+
+ if (cs40l2x->pdata.f0_max > 0 && val > cs40l2x->pdata.f0_max)
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "F0_STORED",
+ CS40L2X_XM_UNPACKED_TYPE), val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to store f0\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static ssize_t cs40l2x_redc_measured_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ unsigned char diag_state;
+ unsigned int redc_measured;
+
+ mutex_lock(&cs40l2x->lock);
+
+ diag_state = cs40l2x->diag_state;
+ redc_measured = cs40l2x->redc_measured;
+
+ mutex_unlock(&cs40l2x->lock);
+
+ if (diag_state != CS40L2X_DIAG_STATE_DONE)
+ return -ENODATA;
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", redc_measured);
+}
+
+static ssize_t cs40l2x_redc_stored_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "REDC_STORED",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read stored ReDC\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t cs40l2x_redc_stored_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ ret = kstrtou32(buf, 10, &val);
+ if (ret)
+ return -EINVAL;
+
+ if (cs40l2x->pdata.redc_min > 0 && val < cs40l2x->pdata.redc_min)
+ return -EINVAL;
+
+ if (cs40l2x->pdata.redc_max > 0 && val > cs40l2x->pdata.redc_max)
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "REDC_STORED",
+ CS40L2X_XM_UNPACKED_TYPE), val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to store ReDC\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static ssize_t cs40l2x_comp_enable_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "COMPENSATION_ENABLE",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read compensation state\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t cs40l2x_comp_enable_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ ret = kstrtou32(buf, 10, &val);
+ if (ret)
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "COMPENSATION_ENABLE",
+ CS40L2X_XM_UNPACKED_TYPE),
+ val ? CS40L2X_COMP_ENABLED : CS40L2X_COMP_DISABLED);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to write compensation state\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static int cs40l2x_dig_scale_get(struct cs40l2x_private *cs40l2x,
+ unsigned int *dig_scale)
+{
+ int ret;
+ unsigned int val;
+
+ if (!mutex_is_locked(&cs40l2x->lock))
+ return -EACCES;
+
+ ret = regmap_read(cs40l2x->regmap, CS40L2X_AMP_DIG_VOL_CTRL, &val);
+ if (ret)
+ return ret;
+
+ *dig_scale = (CS40L2X_DIG_SCALE_ZERO - ((val & CS40L2X_AMP_VOL_PCM_MASK)
+ >> CS40L2X_AMP_VOL_PCM_SHIFT)) & CS40L2X_DIG_SCALE_MASK;
+
+ return 0;
+}
+
+static int cs40l2x_dig_scale_set(struct cs40l2x_private *cs40l2x,
+ unsigned int dig_scale)
+{
+ if (!mutex_is_locked(&cs40l2x->lock))
+ return -EACCES;
+
+ if (dig_scale == CS40L2X_DIG_SCALE_RESET)
+ return -EINVAL;
+
+ return regmap_update_bits(cs40l2x->regmap,
+ CS40L2X_AMP_DIG_VOL_CTRL,
+ CS40L2X_AMP_VOL_PCM_MASK,
+ ((CS40L2X_DIG_SCALE_ZERO - dig_scale)
+ & CS40L2X_DIG_SCALE_MASK)
+ << CS40L2X_AMP_VOL_PCM_SHIFT);
+}
+
+static ssize_t cs40l2x_dig_scale_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int dig_scale;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = cs40l2x_dig_scale_get(cs40l2x, &dig_scale);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read digital scale\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", dig_scale);
+}
+
+static ssize_t cs40l2x_dig_scale_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int dig_scale;
+
+ ret = kstrtou32(buf, 10, &dig_scale);
+ if (ret)
+ return -EINVAL;
+
+ if (dig_scale > CS40L2X_DIG_SCALE_MAX)
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = cs40l2x_dig_scale_set(cs40l2x, dig_scale);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to write digital scale\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static int cs40l2x_gpio1_dig_scale_get(struct cs40l2x_private *cs40l2x,
+ unsigned int *dig_scale)
+{
+ int ret;
+ unsigned int val;
+
+ if (!mutex_is_locked(&cs40l2x->lock))
+ return -EACCES;
+
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "GAIN_CONTROL",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ if (ret)
+ return ret;
+
+ *dig_scale = (val & CS40L2X_GAIN_CTRL_GPIO_MASK)
+ >> CS40L2X_GAIN_CTRL_GPIO_SHIFT;
+
+ return 0;
+}
+
+static int cs40l2x_gpio1_dig_scale_set(struct cs40l2x_private *cs40l2x,
+ unsigned int dig_scale)
+{
+ if (!mutex_is_locked(&cs40l2x->lock))
+ return -EACCES;
+
+ if (dig_scale == CS40L2X_DIG_SCALE_RESET)
+ return -EINVAL;
+
+ return regmap_update_bits(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "GAIN_CONTROL",
+ CS40L2X_XM_UNPACKED_TYPE),
+ CS40L2X_GAIN_CTRL_GPIO_MASK,
+ dig_scale << CS40L2X_GAIN_CTRL_GPIO_SHIFT);
+}
+
+static ssize_t cs40l2x_gpio1_dig_scale_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int dig_scale;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = cs40l2x_gpio1_dig_scale_get(cs40l2x, &dig_scale);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read digital scale\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", dig_scale);
+}
+
+static ssize_t cs40l2x_gpio1_dig_scale_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int dig_scale;
+
+ ret = kstrtou32(buf, 10, &dig_scale);
+ if (ret)
+ return -EINVAL;
+
+ if (dig_scale > CS40L2X_DIG_SCALE_MAX)
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = cs40l2x_gpio1_dig_scale_set(cs40l2x, dig_scale);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to write digital scale\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static int cs40l2x_cp_dig_scale_get(struct cs40l2x_private *cs40l2x,
+ unsigned int *dig_scale)
+{
+ int ret;
+ unsigned int val;
+
+ if (!mutex_is_locked(&cs40l2x->lock))
+ return -EACCES;
+
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "GAIN_CONTROL",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ if (ret)
+ return ret;
+
+ *dig_scale = (val & CS40L2X_GAIN_CTRL_TRIG_MASK)
+ >> CS40L2X_GAIN_CTRL_TRIG_SHIFT;
+
+ return 0;
+}
+
+static int cs40l2x_cp_dig_scale_set(struct cs40l2x_private *cs40l2x,
+ unsigned int dig_scale)
+{
+ if (!mutex_is_locked(&cs40l2x->lock))
+ return -EACCES;
+
+ if (dig_scale == CS40L2X_DIG_SCALE_RESET)
+ return -EINVAL;
+
+ return regmap_update_bits(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "GAIN_CONTROL",
+ CS40L2X_XM_UNPACKED_TYPE),
+ CS40L2X_GAIN_CTRL_TRIG_MASK,
+ dig_scale << CS40L2X_GAIN_CTRL_TRIG_SHIFT);
+}
+
+static ssize_t cs40l2x_cp_dig_scale_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int dig_scale;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = cs40l2x_cp_dig_scale_get(cs40l2x, &dig_scale);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read digital scale\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", dig_scale);
+}
+
+static ssize_t cs40l2x_cp_dig_scale_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int dig_scale;
+
+ ret = kstrtou32(buf, 10, &dig_scale);
+ if (ret)
+ return -EINVAL;
+
+ if (dig_scale > CS40L2X_DIG_SCALE_MAX)
+ return -EINVAL;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = cs40l2x_cp_dig_scale_set(cs40l2x, dig_scale);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to write digital scale\n");
+ return ret;
+ }
+
+ return count;
+}
+
+static ssize_t cs40l2x_heartbeat_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+ int ret;
+ unsigned int val;
+
+ mutex_lock(&cs40l2x->lock);
+ ret = regmap_read(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "HALO_HEARTBEAT",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ mutex_unlock(&cs40l2x->lock);
+
+ if (ret) {
+ pr_err("Failed to read heartbeat\n");
+ return ret;
+ }
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", val);
+}
+
+static ssize_t cs40l2x_num_waves_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct cs40l2x_private *cs40l2x = cs40l2x_get_private(dev);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", cs40l2x->num_waves);
+}
+
+static DEVICE_ATTR(cp_trigger_index, 0660, cs40l2x_cp_trigger_index_show,
+ cs40l2x_cp_trigger_index_store);
+static DEVICE_ATTR(cp_trigger_queue, 0660, cs40l2x_cp_trigger_queue_show,
+ cs40l2x_cp_trigger_queue_store);
+static DEVICE_ATTR(gpio1_enable, 0660, cs40l2x_gpio1_enable_show,
+ cs40l2x_gpio1_enable_store);
+static DEVICE_ATTR(gpio1_rise_index, 0660, cs40l2x_gpio1_rise_index_show,
+ cs40l2x_gpio1_rise_index_store);
+static DEVICE_ATTR(gpio1_fall_index, 0660, cs40l2x_gpio1_fall_index_show,
+ cs40l2x_gpio1_fall_index_store);
+static DEVICE_ATTR(gpio1_fall_timeout, 0660, cs40l2x_gpio1_fall_timeout_show,
+ cs40l2x_gpio1_fall_timeout_store);
+static DEVICE_ATTR(standby_timeout, 0660, cs40l2x_standby_timeout_show,
+ cs40l2x_standby_timeout_store);
+static DEVICE_ATTR(f0_measured, 0660, cs40l2x_f0_measured_show, NULL);
+static DEVICE_ATTR(f0_stored, 0660, cs40l2x_f0_stored_show,
+ cs40l2x_f0_stored_store);
+static DEVICE_ATTR(redc_measured, 0660, cs40l2x_redc_measured_show, NULL);
+static DEVICE_ATTR(redc_stored, 0660, cs40l2x_redc_stored_show,
+ cs40l2x_redc_stored_store);
+static DEVICE_ATTR(comp_enable, 0660, cs40l2x_comp_enable_show,
+ cs40l2x_comp_enable_store);
+static DEVICE_ATTR(dig_scale, 0660, cs40l2x_dig_scale_show,
+ cs40l2x_dig_scale_store);
+static DEVICE_ATTR(gpio1_dig_scale, 0660, cs40l2x_gpio1_dig_scale_show,
+ cs40l2x_gpio1_dig_scale_store);
+static DEVICE_ATTR(cp_dig_scale, 0660, cs40l2x_cp_dig_scale_show,
+ cs40l2x_cp_dig_scale_store);
+static DEVICE_ATTR(heartbeat, 0660, cs40l2x_heartbeat_show, NULL);
+static DEVICE_ATTR(num_waves, 0660, cs40l2x_num_waves_show, NULL);
+
+static struct attribute *cs40l2x_dev_attrs[] = {
+ &dev_attr_cp_trigger_index.attr,
+ &dev_attr_cp_trigger_queue.attr,
+ &dev_attr_gpio1_enable.attr,
+ &dev_attr_gpio1_rise_index.attr,
+ &dev_attr_gpio1_fall_index.attr,
+ &dev_attr_gpio1_fall_timeout.attr,
+ &dev_attr_standby_timeout.attr,
+ &dev_attr_f0_measured.attr,
+ &dev_attr_f0_stored.attr,
+ &dev_attr_redc_measured.attr,
+ &dev_attr_redc_stored.attr,
+ &dev_attr_comp_enable.attr,
+ &dev_attr_dig_scale.attr,
+ &dev_attr_gpio1_dig_scale.attr,
+ &dev_attr_cp_dig_scale.attr,
+ &dev_attr_heartbeat.attr,
+ &dev_attr_num_waves.attr,
+ NULL,
+};
+
+static struct attribute_group cs40l2x_dev_attr_group = {
+ .attrs = cs40l2x_dev_attrs,
+};
+
+static int cs40l2x_stop_playback(struct cs40l2x_private *cs40l2x)
+{
+ int ret, i;
+
+ if (!mutex_is_locked(&cs40l2x->lock))
+ return -EACCES;
+
+ for (i = 0; i < CS40L2X_ENDPLAYBACK_RETRIES; i++) {
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "ENDPLAYBACK",
+ CS40L2X_XM_UNPACKED_TYPE),
+ CS40L2X_ENDPLAYBACK_REQ);
+ if (!ret)
+ return 0;
+
+ usleep_range(10000, 10100);
+ }
+
+ dev_err(cs40l2x->dev, "Parking device in reset\n");
+ gpiod_set_value_cansleep(cs40l2x->reset_gpio, 0);
+
+ return -EIO;
+}
+
+static int cs40l2x_pbq_cancel(struct cs40l2x_private *cs40l2x)
+{
+ int ret;
+
+ /* this function expects to be called from a locked worker function */
+ if (!mutex_is_locked(&cs40l2x->lock))
+ return -EACCES;
+
+ hrtimer_cancel(&cs40l2x->pbq_timer);
+
+ ret = cs40l2x_stop_playback(cs40l2x);
+ if (ret)
+ return ret;
+
+ ret = cs40l2x_cp_dig_scale_set(cs40l2x, cs40l2x->pbq_cp_dig_scale);
+ if (ret)
+ return ret;
+
+ cs40l2x->pbq_state = CS40L2X_PBQ_STATE_IDLE;
+
+ return 0;
+}
+
+static int cs40l2x_pbq_pair_launch(struct cs40l2x_private *cs40l2x)
+{
+ unsigned int tag, mag, cp_dig_scale;
+ int ret, i;
+
+ /* this function expects to be called from a locked worker function */
+ if (!mutex_is_locked(&cs40l2x->lock))
+ return -EACCES;
+
+ do {
+ /* restart queue as necessary */
+ if (cs40l2x->pbq_index == cs40l2x->pbq_depth) {
+ cs40l2x->pbq_index = 0;
+ for (i = 0; i < cs40l2x->pbq_depth; i++)
+ cs40l2x->pbq_pairs[i].remain =
+ cs40l2x->pbq_pairs[i].repeat;
+
+ switch (cs40l2x->pbq_remain) {
+ case -1:
+ /* loop until stopped */
+ break;
+ case 0:
+ /* queue is finished */
+ cs40l2x->cp_trailer_index = 0;
+ cs40l2x->pbq_state = CS40L2X_PBQ_STATE_IDLE;
+
+ ret = cs40l2x_cp_dig_scale_set(cs40l2x,
+ cs40l2x->pbq_cp_dig_scale);
+ return ret;
+ default:
+ /* loop once more */
+ cs40l2x->pbq_remain--;
+ }
+ }
+
+ tag = cs40l2x->pbq_pairs[cs40l2x->pbq_index].tag;
+ mag = cs40l2x->pbq_pairs[cs40l2x->pbq_index].mag;
+
+ switch (tag) {
+ case CS40L2X_PBQ_TAG_SILENCE:
+ hrtimer_start(&cs40l2x->pbq_timer,
+ ktime_set(mag / 1000,
+ (mag % 1000) * 1000000),
+ HRTIMER_MODE_REL);
+ cs40l2x->pbq_state = CS40L2X_PBQ_STATE_SILENT;
+ cs40l2x->pbq_index++;
+ break;
+ case CS40L2X_PBQ_TAG_START:
+ cs40l2x->pbq_index++;
+ break;
+ case CS40L2X_PBQ_TAG_STOP:
+ if (cs40l2x->pbq_pairs[cs40l2x->pbq_index].remain) {
+ cs40l2x->pbq_pairs[cs40l2x->pbq_index].remain--;
+ cs40l2x->pbq_index = mag;
+ } else {
+ cs40l2x->pbq_index++;
+ }
+ break;
+ default:
+ cp_dig_scale = cs40l2x->pbq_cp_dig_scale
+ + cs40l2x_pbq_dig_scale[mag];
+ if (cp_dig_scale > CS40L2X_DIG_SCALE_MAX)
+ cp_dig_scale = CS40L2X_DIG_SCALE_MAX;
+
+ ret = cs40l2x_cp_dig_scale_set(cs40l2x, cp_dig_scale);
+ if (ret)
+ return ret;
+
+ ret = regmap_write(cs40l2x->regmap,
+ CS40L2X_MBOX_TRIGGERINDEX, tag);
+ if (ret)
+ return ret;
+
+ hrtimer_start(&cs40l2x->pbq_timer,
+ ktime_set(0, CS40L2X_PBQ_POLL_NS),
+ HRTIMER_MODE_REL);
+
+ cs40l2x->pbq_state = CS40L2X_PBQ_STATE_PLAYING;
+ cs40l2x->pbq_index++;
+ }
+
+ } while (tag == CS40L2X_PBQ_TAG_START || tag == CS40L2X_PBQ_TAG_STOP);
+
+ return 0;
+}
+
+static void cs40l2x_vibe_pbq_worker(struct work_struct *work)
+{
+ struct cs40l2x_private *cs40l2x =
+ container_of(work, struct cs40l2x_private, vibe_pbq_work);
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+ unsigned int val;
+ int ret;
+
+ mutex_lock(&cs40l2x->lock);
+
+ switch (cs40l2x->pbq_state) {
+ case CS40L2X_PBQ_STATE_IDLE:
+ /* queue may have been canceled */
+ break;
+
+ case CS40L2X_PBQ_STATE_PLAYING:
+ ret = regmap_read(regmap, cs40l2x_dsp_reg(cs40l2x, "STATUS",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ if (ret) {
+ dev_err(dev, "Failed to capture playback status\n");
+ goto err_mutex;
+ }
+
+ if (val != CS40L2X_STATUS_IDLE) {
+ hrtimer_start(&cs40l2x->pbq_timer,
+ ktime_set(0, CS40L2X_PBQ_POLL_NS),
+ HRTIMER_MODE_REL);
+ goto err_mutex;
+ }
+
+ ret = cs40l2x_pbq_pair_launch(cs40l2x);
+ if (ret)
+ dev_err(dev, "Failed to continue playback queue\n");
+ break;
+
+ case CS40L2X_PBQ_STATE_SILENT:
+ ret = cs40l2x_pbq_pair_launch(cs40l2x);
+ if (ret)
+ dev_err(dev, "Failed to continue playback queue\n");
+ break;
+
+ default:
+ dev_err(dev, "Unexpected playback queue state: %d\n",
+ cs40l2x->pbq_state);
+ }
+
+err_mutex:
+ mutex_unlock(&cs40l2x->lock);
+}
+
+static enum hrtimer_restart cs40l2x_pbq_timer(struct hrtimer *timer)
+{
+ struct cs40l2x_private *cs40l2x =
+ container_of(timer, struct cs40l2x_private, pbq_timer);
+
+ queue_work(cs40l2x->vibe_workqueue, &cs40l2x->vibe_pbq_work);
+
+ return HRTIMER_NORESTART;
+}
+
+static int cs40l2x_diag_capture(struct cs40l2x_private *cs40l2x)
+{
+ struct regmap *regmap = cs40l2x->regmap;
+ int ret;
+
+ /* this function expects to be called from a locked worker function */
+ if (!mutex_is_locked(&cs40l2x->lock))
+ return -EACCES;
+
+ if (cs40l2x->diag_state != CS40L2X_DIAG_STATE_RUN)
+ return -ENODATA;
+
+ ret = regmap_read(regmap, cs40l2x_dsp_reg(cs40l2x, "F0",
+ CS40L2X_XM_UNPACKED_TYPE), &cs40l2x->f0_measured);
+ if (ret)
+ return ret;
+
+ ret = regmap_read(regmap, cs40l2x_dsp_reg(cs40l2x, "REDC",
+ CS40L2X_XM_UNPACKED_TYPE), &cs40l2x->redc_measured);
+ if (ret)
+ return ret;
+
+ cs40l2x->diag_state = CS40L2X_DIAG_STATE_DONE;
+
+ return 0;
+}
+
+static void cs40l2x_vibe_start_worker(struct work_struct *work)
+{
+ struct cs40l2x_private *cs40l2x =
+ container_of(work, struct cs40l2x_private, vibe_start_work);
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+ int ret, i;
+
+ mutex_lock(&cs40l2x->lock);
+
+ /* handle interruption of special cases */
+ switch (cs40l2x->cp_trailer_index) {
+ case CS40L2X_INDEX_PBQ:
+ ret = cs40l2x_pbq_cancel(cs40l2x);
+ if (ret)
+ dev_err(dev, "Failed to cancel playback queue\n");
+ break;
+
+ case CS40L2X_INDEX_DIAG:
+ cs40l2x->diag_state = CS40L2X_DIAG_STATE_INIT;
+ goto err_mutex;
+ }
+
+ cs40l2x->cp_trailer_index = cs40l2x->cp_trigger_index;
+
+ switch (cs40l2x->cp_trailer_index) {
+ case CS40L2X_INDEX_VIBE:
+ case CS40L2X_INDEX_CONT_MIN ... CS40L2X_INDEX_CONT_MAX:
+ case CS40L2X_INDEX_DIAG:
+ pm_stay_awake(dev);
+ }
+
+ switch (cs40l2x->cp_trailer_index) {
+ case CS40L2X_INDEX_VIBE:
+ case CS40L2X_INDEX_CONT_MIN ... CS40L2X_INDEX_CONT_MAX:
+ ret = regmap_write(regmap, CS40L2X_MBOX_TRIGGER_MS,
+ cs40l2x->cp_trailer_index & CS40L2X_INDEX_MASK);
+ if (ret)
+ dev_err(dev, "Failed to start playback\n");
+
+ cs40l2x->led_dev.brightness = LED_FULL;
+ break;
+
+ case CS40L2X_INDEX_CLICK_MIN ... CS40L2X_INDEX_CLICK_MAX:
+ ret = regmap_write(regmap, CS40L2X_MBOX_TRIGGERINDEX,
+ cs40l2x->cp_trailer_index);
+ if (ret)
+ dev_err(dev, "Failed to start playback\n");
+
+ cs40l2x->led_dev.brightness = LED_FULL;
+ break;
+
+ case CS40L2X_INDEX_PBQ:
+ cs40l2x->pbq_cp_dig_scale = CS40L2X_DIG_SCALE_RESET;
+ ret = cs40l2x_cp_dig_scale_get(cs40l2x,
+ &cs40l2x->pbq_cp_dig_scale);
+ if (ret) {
+ dev_err(dev, "Failed to read digital scale\n");
+ goto err_mutex;
+ }
+
+ cs40l2x->pbq_index = 0;
+ cs40l2x->pbq_remain = cs40l2x->pbq_repeat;
+ for (i = 0; i < cs40l2x->pbq_depth; i++)
+ cs40l2x->pbq_pairs[i].remain =
+ cs40l2x->pbq_pairs[i].repeat;
+ ret = cs40l2x_pbq_pair_launch(cs40l2x);
+ if (ret)
+ dev_err(dev, "Failed to launch playback queue\n");
+ break;
+
+ case CS40L2X_INDEX_DIAG:
+ cs40l2x->diag_state = CS40L2X_DIAG_STATE_INIT;
+ cs40l2x->diag_dig_scale = CS40L2X_DIG_SCALE_RESET;
+
+ ret = cs40l2x_dig_scale_get(cs40l2x, &cs40l2x->diag_dig_scale);
+ if (ret) {
+ dev_err(dev, "Failed to read digital scale\n");
+ goto err_mutex;
+ }
+
+ ret = cs40l2x_dig_scale_set(cs40l2x, 0);
+ if (ret) {
+ dev_err(dev, "Failed to reset digital scale\n");
+ goto err_mutex;
+ }
+
+ ret = regmap_write(regmap,
+ cs40l2x_dsp_reg(cs40l2x, "CLOSED_LOOP",
+ CS40L2X_XM_UNPACKED_TYPE), 0);
+ if (ret) {
+ dev_err(dev, "Failed to disable closed-loop mode\n");
+ goto err_mutex;
+ }
+
+ ret = regmap_write(regmap, CS40L2X_MBOX_STIMULUS_MODE, 1);
+ if (ret) {
+ dev_err(dev, "Failed to enable stimulus mode\n");
+ goto err_mutex;
+ }
+
+ msleep(CS40L2X_DIAG_STATE_DELAY_MS);
+
+ ret = regmap_write(regmap,
+ cs40l2x_dsp_reg(cs40l2x, "CLOSED_LOOP",
+ CS40L2X_XM_UNPACKED_TYPE), 1);
+ if (ret) {
+ dev_err(dev, "Failed to enable closed-loop mode\n");
+ goto err_mutex;
+ }
+ cs40l2x->diag_state = CS40L2X_DIAG_STATE_RUN;
+ cs40l2x->led_dev.brightness = LED_FULL;
+
+ break;
+
+ default:
+ dev_err(dev, "Invalid wavetable index\n");
+ }
+
+err_mutex:
+ mutex_unlock(&cs40l2x->lock);
+}
+
+static void cs40l2x_vibe_stop_worker(struct work_struct *work)
+{
+ struct cs40l2x_private *cs40l2x =
+ container_of(work, struct cs40l2x_private, vibe_stop_work);
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+ int ret;
+
+ mutex_lock(&cs40l2x->lock);
+
+ switch (cs40l2x->cp_trailer_index) {
+ case CS40L2X_INDEX_VIBE:
+ case CS40L2X_INDEX_CONT_MIN ... CS40L2X_INDEX_CONT_MAX:
+ ret = cs40l2x_stop_playback(cs40l2x);
+ if (ret)
+ dev_err(dev, "Failed to stop playback\n");
+ pm_relax(dev);
+ break;
+
+ case CS40L2X_INDEX_CLICK_MIN ... CS40L2X_INDEX_CLICK_MAX:
+ ret = cs40l2x_stop_playback(cs40l2x);
+ if (ret)
+ dev_err(dev, "Failed to stop playback\n");
+ break;
+
+ case CS40L2X_INDEX_PBQ:
+ ret = cs40l2x_pbq_cancel(cs40l2x);
+ if (ret)
+ dev_err(dev, "Failed to cancel playback queue\n");
+ break;
+
+ case CS40L2X_INDEX_DIAG:
+ ret = cs40l2x_diag_capture(cs40l2x);
+ if (ret)
+ dev_err(dev, "Failed to capture f0 and ReDC\n");
+
+ ret = regmap_write(regmap, CS40L2X_MBOX_STIMULUS_MODE, 0);
+ if (ret)
+ dev_err(dev, "Failed to disable stimulus mode\n");
+
+ ret = cs40l2x_dig_scale_set(cs40l2x, cs40l2x->diag_dig_scale);
+ if (ret)
+ dev_err(dev, "Failed to restore digital scale\n");
+ pm_relax(dev);
+ break;
+
+ default:
+ dev_err(dev, "Invalid wavetable index\n");
+ }
+
+ cs40l2x->cp_trailer_index = 0;
+ cs40l2x->led_dev.brightness = LED_OFF;
+
+ mutex_unlock(&cs40l2x->lock);
+}
+
+/* vibration callback for LED device */
+static void cs40l2x_vibe_brightness_set(struct led_classdev *led_cdev,
+ enum led_brightness brightness)
+{
+ struct cs40l2x_private *cs40l2x =
+ container_of(led_cdev, struct cs40l2x_private, led_dev);
+
+ if (!cs40l2x->vibe_workqueue || !cs40l2x->vibe_init_success) {
+ dev_err(cs40l2x->dev,
+ "Failed to set vibe when it's not ready\n");
+ return;
+ }
+
+ if (brightness == LED_OFF)
+ queue_work(cs40l2x->vibe_workqueue, &cs40l2x->vibe_stop_work);
+ else
+ queue_work(cs40l2x->vibe_workqueue, &cs40l2x->vibe_start_work);
+}
+
+static void cs40l2x_create_led(struct cs40l2x_private *cs40l2x)
+{
+ int ret;
+ struct led_classdev *led_dev = &cs40l2x->led_dev;
+ struct device *dev = cs40l2x->dev;
+
+ led_dev->name = CS40L2X_DEVICE_NAME;
+ led_dev->max_brightness = LED_FULL;
+ led_dev->brightness_set = cs40l2x_vibe_brightness_set;
+ led_dev->default_trigger = "transient";
+ led_dev->flags = LED_BRIGHTNESS_FAST;
+
+ ret = led_classdev_register(dev, led_dev);
+ if (ret) {
+ dev_err(dev, "Failed to register LED device: %d\n", ret);
+ return;
+ }
+
+ ret = sysfs_create_group(&cs40l2x->dev->kobj, &cs40l2x_dev_attr_group);
+ if (ret) {
+ dev_err(dev, "Failed to create sysfs group: %d\n", ret);
+ return;
+ }
+}
+
+static void cs40l2x_vibe_init(struct cs40l2x_private *cs40l2x)
+{
+ struct hrtimer *pbq_timer = &cs40l2x->pbq_timer;
+ int ret;
+
+ cs40l2x->vibe_workqueue =
+ alloc_ordered_workqueue("vibe_workqueue", WQ_HIGHPRI);
+ if (!cs40l2x->vibe_workqueue) {
+ dev_err(cs40l2x->dev, "Failed to allocate workqueue\n");
+ return;
+ }
+
+ INIT_WORK(&cs40l2x->vibe_start_work, cs40l2x_vibe_start_worker);
+ INIT_WORK(&cs40l2x->vibe_pbq_work, cs40l2x_vibe_pbq_worker);
+ INIT_WORK(&cs40l2x->vibe_stop_work, cs40l2x_vibe_stop_worker);
+
+ hrtimer_init(pbq_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+ pbq_timer->function = cs40l2x_pbq_timer;
+
+ ret = device_init_wakeup(cs40l2x->dev, true);
+ if (ret) {
+ dev_err(cs40l2x->dev, "Failed to initialize wakeup source\n");
+ return;
+ }
+
+ cs40l2x->vibe_init_success = true;
+}
+
+static int cs40l2x_coeff_init(struct cs40l2x_private *cs40l2x)
+{
+ int ret, i;
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+ struct cs40l2x_coeff_desc *coeff_desc;
+ unsigned int val, num_algos, algo_id, algo_rev;
+ unsigned int xm_base, xm_size, ym_base, ym_size;
+ unsigned int reg = CS40L2X_XM_FW_ID;
+
+ ret = regmap_read(regmap, CS40L2X_XM_NUM_ALGOS, &num_algos);
+ if (ret) {
+ dev_err(dev, "Failed to read number of algorithms\n");
+ return ret;
+ }
+
+ if (num_algos > CS40L2X_NUM_ALGOS_MAX) {
+ dev_err(dev, "Invalid number of algorithms\n");
+ return -EINVAL;
+ }
+
+ /* add one extra iteration to account for system algorithm */
+ for (i = 0; i < (num_algos + 1); i++) {
+ ret = regmap_read(regmap,
+ reg + CS40L2X_ALGO_ID_OFFSET, &algo_id);
+ if (ret) {
+ dev_err(dev, "Failed to read algo. %d ID\n", i);
+ return ret;
+ }
+
+ ret = regmap_read(regmap,
+ reg + CS40L2X_ALGO_REV_OFFSET, &algo_rev);
+ if (ret) {
+ dev_err(dev, "Failed to read algo. %d revision\n", i);
+ return ret;
+ }
+
+ /* discern firmware revision from system algorithm */
+ if (i == 0) {
+ if (algo_rev < CS40L2X_FW_REV_MIN) {
+ dev_err(dev,
+ "Invalid firmware revision: %d.%d.%d\n",
+ (algo_rev & 0xFF0000) >> 16,
+ (algo_rev & 0xFF00) >> 8,
+ algo_rev & 0xFF);
+ return -EINVAL;
+ }
+ dev_info(dev, "Firmware revision %d.%d.%d\n",
+ (algo_rev & 0xFF0000) >> 16,
+ (algo_rev & 0xFF00) >> 8,
+ algo_rev & 0xFF);
+ }
+
+ ret = regmap_read(regmap,
+ reg + CS40L2X_ALGO_XM_BASE_OFFSET, &xm_base);
+ if (ret) {
+ dev_err(dev, "Failed to read algo. %d XM_BASE\n", i);
+ return ret;
+ }
+
+ ret = regmap_read(regmap,
+ reg + CS40L2X_ALGO_XM_SIZE_OFFSET, &xm_size);
+ if (ret) {
+ dev_err(dev, "Failed to read algo. %d XM_SIZE\n", i);
+ return ret;
+ }
+
+ ret = regmap_read(regmap,
+ reg + CS40L2X_ALGO_YM_BASE_OFFSET, &ym_base);
+ if (ret) {
+ dev_err(dev, "Failed to read algo. %d YM_BASE\n", i);
+ return ret;
+ }
+
+ ret = regmap_read(regmap,
+ reg + CS40L2X_ALGO_YM_SIZE_OFFSET, &ym_size);
+ if (ret) {
+ dev_err(dev, "Failed to read algo. %d YM_SIZE\n", i);
+ return ret;
+ }
+
+ list_for_each_entry(coeff_desc,
+ &cs40l2x->coeff_desc_head, list) {
+
+ if (coeff_desc->parent_id != algo_id)
+ continue;
+
+ switch (coeff_desc->block_type) {
+ case CS40L2X_XM_UNPACKED_TYPE:
+ coeff_desc->reg = CS40L2X_DSP1_XMEM_UNPACK24_0
+ + xm_base * 4
+ + coeff_desc->block_offset * 4;
+ if (!strcmp(coeff_desc->name, "WAVETABLE")) {
+ cs40l2x->wt_limit_xm = (xm_size
+ - coeff_desc->block_offset) * 4;
+ cs40l2x->vibegen_id = algo_id;
+ cs40l2x->vibegen_rev = algo_rev;
+ }
+ break;
+ case CS40L2X_YM_UNPACKED_TYPE:
+ coeff_desc->reg = CS40L2X_DSP1_YMEM_UNPACK24_0
+ + ym_base * 4
+ + coeff_desc->block_offset * 4;
+ if (!strcmp(coeff_desc->name, "WAVETABLEYM"))
+ cs40l2x->wt_limit_ym = (ym_size
+ - coeff_desc->block_offset) * 4;
+ break;
+ }
+
+ dev_dbg(dev, "Found control %s at 0x%08X\n",
+ coeff_desc->name, coeff_desc->reg);
+ }
+
+ /* system algo. contains one extra register (num. algos.) */
+ if (i)
+ reg += CS40L2X_ALGO_ENTRY_SIZE;
+ else
+ reg += (CS40L2X_ALGO_ENTRY_SIZE + 4);
+ }
+
+ ret = regmap_read(regmap, reg, &val);
+ if (ret) {
+ dev_err(dev, "Failed to read list terminator\n");
+ return ret;
+ }
+
+ if (val != CS40L2X_ALGO_LIST_TERM) {
+ dev_err(dev, "Invalid list terminator: 0x%X\n", val);
+ return -EINVAL;
+ }
+
+ dev_info(dev, "Maximum wavetable size: %d bytes (XM), %d bytes (YM)\n",
+ cs40l2x->wt_limit_xm / 4 * 3,
+ cs40l2x->wt_limit_ym / 4 * 3);
+
+ return 0;
+}
+
+static void cs40l2x_dsp_start(struct cs40l2x_private *cs40l2x)
+{
+ int ret;
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+ unsigned int val;
+ int dsp_timeout = CS40L2X_DSP_TIMEOUT_COUNT;
+
+ if (cs40l2x->pdata.gpio1_mode != CS40L2X_GPIO1_MODE_DEF_ON) {
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "GPIO_ENABLE",
+ CS40L2X_XM_UNPACKED_TYPE),
+ CS40L2X_GPIO1_DISABLED);
+ if (ret) {
+ dev_err(dev, "Failed to pre-configure GPIO1\n");
+ return;
+ }
+ }
+
+ switch (cs40l2x->revid) {
+ case CS40L2X_REVID_A0:
+ ret = regmap_update_bits(regmap, CS40L2X_PWR_CTRL1,
+ CS40L2X_GLOBAL_EN_MASK,
+ 1 << CS40L2X_GLOBAL_EN_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to enable device\n");
+ return;
+ }
+
+ ret = regmap_update_bits(regmap, CS40L2X_DSP1_CCM_CORE_CTRL,
+ CS40L2X_DSP1_RESET_MASK |
+ CS40L2X_DSP1_EN_MASK,
+ (1 << CS40L2X_DSP1_RESET_SHIFT) |
+ (1 << CS40L2X_DSP1_EN_SHIFT));
+ if (ret) {
+ dev_err(dev, "Failed to start DSP\n");
+ return;
+ }
+ break;
+ default:
+ ret = regmap_update_bits(regmap, CS40L2X_PWRMGT_CTL,
+ CS40L2X_MEM_RDY_MASK,
+ 1 << CS40L2X_MEM_RDY_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to set memory ready flag\n");
+ return;
+ }
+
+ ret = regmap_update_bits(regmap, CS40L2X_DSP1_CCM_CORE_CTRL,
+ CS40L2X_DSP1_RESET_MASK,
+ 1 << CS40L2X_DSP1_RESET_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to restart DSP\n");
+ return;
+ }
+ }
+
+ while (dsp_timeout > 0) {
+ usleep_range(10000, 10100);
+
+ ret = regmap_read(regmap, cs40l2x_dsp_reg(cs40l2x, "HALO_STATE",
+ CS40L2X_XM_UNPACKED_TYPE), &val);
+ if (ret) {
+ dev_err(dev, "Failed to read DSP status\n");
+ return;
+ }
+
+ if (val == CS40L2X_HALO_STATE_RUNNING)
+ break;
+
+ dsp_timeout--;
+ }
+
+ if (dsp_timeout == 0) {
+ dev_err(dev, "Timed out with DSP status = %d\n", val);
+ return;
+ }
+
+ ret = regmap_write(regmap, cs40l2x_dsp_reg(cs40l2x, "TIMEOUT_MS",
+ CS40L2X_XM_UNPACKED_TYPE), CS40L2X_TIMEOUT_MS_MAX);
+ if (ret) {
+ dev_err(dev, "Failed to extend playback timeout\n");
+ return;
+ }
+
+ ret = regmap_read(regmap, cs40l2x_dsp_reg(cs40l2x, "NUMBEROFWAVES",
+ CS40L2X_XM_UNPACKED_TYPE), &cs40l2x->num_waves);
+ if (ret) {
+ dev_err(dev, "Failed to count wavetable entries\n");
+ return;
+ }
+
+ if (cs40l2x->num_waves == 0) {
+ dev_err(dev, "Wavetable is empty\n");
+ return;
+ }
+
+ if (cs40l2x->pdata.f0_default) {
+ ret = regmap_write(regmap,
+ cs40l2x_dsp_reg(cs40l2x, "F0_STORED",
+ CS40L2X_XM_UNPACKED_TYPE),
+ cs40l2x->pdata.f0_default);
+ if (ret) {
+ dev_err(dev, "Failed to write default f0\n");
+ return;
+ }
+ }
+
+ if (cs40l2x->pdata.redc_default) {
+ ret = regmap_write(regmap,
+ cs40l2x_dsp_reg(cs40l2x, "REDC_STORED",
+ CS40L2X_XM_UNPACKED_TYPE),
+ cs40l2x->pdata.redc_default);
+ if (ret) {
+ dev_err(dev, "Failed to write default ReDC\n");
+ return;
+ }
+ }
+
+ if (cs40l2x->pdata.gpio1_rise_index > 0
+ && cs40l2x->pdata.gpio1_rise_index
+ < cs40l2x->num_waves) {
+ ret = regmap_write(regmap,
+ cs40l2x_dsp_reg(cs40l2x, "INDEXBUTTONPRESS",
+ CS40L2X_XM_UNPACKED_TYPE),
+ cs40l2x->pdata.gpio1_rise_index);
+ if (ret) {
+ dev_err(dev,
+ "Failed to write default gpio1_rise_index\n");
+ return;
+ }
+ } else if (cs40l2x->pdata.gpio1_rise_index >= cs40l2x->num_waves) {
+ dev_warn(dev, "Ignored default gpio1_rise_index\n");
+ }
+
+ if (cs40l2x->pdata.gpio1_fall_index > 0
+ && cs40l2x->pdata.gpio1_fall_index
+ < cs40l2x->num_waves) {
+ ret = regmap_write(regmap,
+ cs40l2x_dsp_reg(cs40l2x, "INDEXBUTTONRELEASE",
+ CS40L2X_XM_UNPACKED_TYPE),
+ cs40l2x->pdata.gpio1_fall_index);
+ if (ret) {
+ dev_err(dev,
+ "Failed to write default gpio1_fall_index\n");
+ return;
+ }
+ } else if (cs40l2x->pdata.gpio1_fall_index >= cs40l2x->num_waves) {
+ dev_warn(dev, "Ignored default gpio1_fall_index\n");
+ }
+
+ if (cs40l2x->pdata.gpio1_fall_timeout > 0
+ && (cs40l2x->pdata.gpio1_fall_timeout
+ & CS40L2X_PDATA_MASK)
+ <= CS40L2X_PR_TIMEOUT_MAX) {
+ ret = regmap_write(regmap,
+ cs40l2x_dsp_reg(cs40l2x,
+ "PRESS_RELEASE_TIMEOUT",
+ CS40L2X_XM_UNPACKED_TYPE),
+ cs40l2x->pdata.gpio1_fall_timeout
+ & CS40L2X_PDATA_MASK);
+ if (ret) {
+ dev_err(dev,
+ "Failed to write default gpio1_fall_timeout\n");
+ return;
+ }
+ } else if ((cs40l2x->pdata.gpio1_fall_timeout
+ & CS40L2X_PDATA_MASK) > CS40L2X_PR_TIMEOUT_MAX) {
+ dev_warn(dev, "Ignored default gpio1_fall_timeout\n");
+ }
+
+ dev_info(dev, "Normal-mode haptics successfully started\n");
+
+ cs40l2x_vibe_init(cs40l2x);
+}
+
+static int cs40l2x_raw_write(struct cs40l2x_private *cs40l2x, unsigned int reg,
+ const void *val, size_t val_len, size_t limit)
+{
+ int ret = 0, i;
+
+ /* split "val" into smaller writes not to exceed "limit" in length */
+ for (i = 0; i < val_len; i += limit) {
+ ret = regmap_raw_write(cs40l2x->regmap, (reg + i), (val + i),
+ (val_len - i) > limit ? limit : (val_len - i));
+ if (ret)
+ break;
+ }
+
+ return ret;
+}
+
+static void cs40l2x_waveform_load(const struct firmware *fw, void *context)
+{
+ int ret;
+ struct cs40l2x_private *cs40l2x = (struct cs40l2x_private *)context;
+ struct device *dev = cs40l2x->dev;
+ unsigned int pos = CS40L2X_WT_FILE_HEADER_SIZE;
+ unsigned int block_type, block_length;
+ unsigned int algo_id, algo_rev;
+
+ if (!fw)
+ goto skip_loading;
+
+ if (memcmp(fw->data, "WMDR", 4)) {
+ dev_err(dev, "Failed to recognize waveform file\n");
+ goto err_rls_fw;
+ }
+
+ while (pos < fw->size) {
+
+ /* block offset is not used here */
+ pos += CS40L2X_WT_DBLK_OFFSET_SIZE;
+
+ block_type = fw->data[pos]
+ + (fw->data[pos + 1] << 8);
+ pos += CS40L2X_WT_DBLK_TYPE_SIZE;
+
+ algo_id = fw->data[pos]
+ + (fw->data[pos + 1] << 8)
+ + (fw->data[pos + 2] << 16)
+ + (fw->data[pos + 3] << 24);
+ pos += CS40L2X_WT_ALGO_ID_SIZE;
+
+ algo_rev = fw->data[pos]
+ + (fw->data[pos + 1] << 8)
+ + (fw->data[pos + 2] << 16)
+ + (fw->data[pos + 3] << 24);
+ pos += CS40L2X_WT_ALGO_REV_SIZE;
+
+ /* sample rate is not used here */
+ pos += CS40L2X_WT_SAMPLE_RATE_SIZE;
+
+ block_length = fw->data[pos]
+ + (fw->data[pos + 1] << 8)
+ + (fw->data[pos + 2] << 16)
+ + (fw->data[pos + 3] << 24);
+ pos += CS40L2X_WT_DBLK_LENGTH_SIZE;
+
+ switch (block_type) {
+ case CS40L2X_XM_UNPACKED_TYPE:
+ if (algo_id != cs40l2x->vibegen_id) {
+ dev_err(dev, "Invalid algo. ID: 0x%06X\n",
+ algo_id);
+ goto err_rls_fw;
+ }
+
+ if (((algo_rev >> 8) & CS40L2X_ALGO_REV_MASK)
+ != (cs40l2x->vibegen_rev
+ & CS40L2X_ALGO_REV_MASK)) {
+ dev_err(dev, "Invalid algo. rev.: %d.%d.%d\n",
+ (algo_rev & 0xFF000000) >> 24,
+ (algo_rev & 0xFF0000) >> 16,
+ (algo_rev & 0xFF00) >> 8);
+ goto err_rls_fw;
+ }
+
+ if (block_length > cs40l2x->wt_limit_xm) {
+ dev_err(dev,
+ "Wavetable too large: %d bytes (XM)\n",
+ block_length / 4 * 3);
+ goto err_rls_fw;
+ }
+
+ ret = cs40l2x_raw_write(cs40l2x,
+ cs40l2x_dsp_reg(cs40l2x, "WAVETABLE",
+ CS40L2X_XM_UNPACKED_TYPE),
+ &fw->data[pos], block_length,
+ CS40L2X_MAX_WLEN);
+ if (ret) {
+ dev_err(dev,
+ "Failed to write XM_UNPACKED memory\n");
+ goto err_rls_fw;
+ }
+ break;
+ case CS40L2X_YM_UNPACKED_TYPE:
+ if (algo_id != cs40l2x->vibegen_id) {
+ dev_err(dev, "Invalid algo. ID: 0x%06X\n",
+ algo_id);
+ goto err_rls_fw;
+ }
+
+ if (((algo_rev >> 8) & CS40L2X_ALGO_REV_MASK)
+ != (cs40l2x->vibegen_rev
+ & CS40L2X_ALGO_REV_MASK)) {
+ dev_err(dev, "Invalid algo. rev.: %d.%d.%d\n",
+ (algo_rev & 0xFF000000) >> 24,
+ (algo_rev & 0xFF0000) >> 16,
+ (algo_rev & 0xFF00) >> 8);
+ goto err_rls_fw;
+ }
+
+ if (block_length > cs40l2x->wt_limit_ym) {
+ dev_err(dev,
+ "Wavetable too large: %d bytes (YM)\n",
+ block_length / 4 * 3);
+ goto err_rls_fw;
+ }
+
+ ret = cs40l2x_raw_write(cs40l2x,
+ cs40l2x_dsp_reg(cs40l2x, "WAVETABLEYM",
+ CS40L2X_YM_UNPACKED_TYPE),
+ &fw->data[pos], block_length,
+ CS40L2X_MAX_WLEN);
+ if (ret) {
+ dev_err(dev,
+ "Failed to write YM_UNPACKED memory\n");
+ goto err_rls_fw;
+ }
+ break;
+ }
+
+ pos += block_length;
+ }
+
+skip_loading:
+ cs40l2x_dsp_start(cs40l2x);
+err_rls_fw:
+ release_firmware(fw);
+}
+
+static int cs40l2x_algo_parse(struct cs40l2x_private *cs40l2x,
+ const unsigned char *data)
+{
+ struct cs40l2x_coeff_desc *coeff_desc;
+ unsigned int pos = 0;
+ unsigned int algo_id, algo_desc_length, coeff_count;
+ unsigned int block_offset, block_type, block_length;
+ unsigned char algo_name_length;
+ int i;
+
+ /* record algorithm ID */
+ algo_id = *(data + pos)
+ + (*(data + pos + 1) << 8)
+ + (*(data + pos + 2) << 16)
+ + (*(data + pos + 3) << 24);
+ pos += CS40L2X_ALGO_ID_SIZE;
+
+ /* skip past algorithm name */
+ algo_name_length = *(data + pos);
+ pos += ((algo_name_length / 4) * 4) + 4;
+
+ /* skip past algorithm description */
+ algo_desc_length = *(data + pos)
+ + (*(data + pos + 1) << 8);
+ pos += ((algo_desc_length / 4) * 4) + 4;
+
+ /* record coefficient count */
+ coeff_count = *(data + pos)
+ + (*(data + pos + 1) << 8)
+ + (*(data + pos + 2) << 16)
+ + (*(data + pos + 3) << 24);
+ pos += CS40L2X_COEFF_COUNT_SIZE;
+
+ for (i = 0; i < coeff_count; i++) {
+ block_offset = *(data + pos)
+ + (*(data + pos + 1) << 8);
+ pos += CS40L2X_COEFF_OFFSET_SIZE;
+
+ block_type = *(data + pos)
+ + (*(data + pos + 1) << 8);
+ pos += CS40L2X_COEFF_TYPE_SIZE;
+
+ block_length = *(data + pos)
+ + (*(data + pos + 1) << 8)
+ + (*(data + pos + 2) << 16)
+ + (*(data + pos + 3) << 24);
+ pos += CS40L2X_COEFF_LENGTH_SIZE;
+
+ coeff_desc = devm_kzalloc(cs40l2x->dev, sizeof(*coeff_desc),
+ GFP_KERNEL);
+ if (!coeff_desc)
+ return -ENOMEM;
+
+ coeff_desc->parent_id = algo_id;
+ coeff_desc->block_offset = block_offset;
+ coeff_desc->block_type = block_type;
+
+ memcpy(coeff_desc->name, data + pos + 1, *(data + pos));
+ coeff_desc->name[*(data + pos)] = '\0';
+
+ list_add(&coeff_desc->list, &cs40l2x->coeff_desc_head);
+
+ pos += block_length;
+ }
+
+ return 0;
+}
+
+static void cs40l2x_firmware_load(const struct firmware *fw, void *context)
+{
+ int ret;
+ struct cs40l2x_private *cs40l2x = (struct cs40l2x_private *)context;
+ struct device *dev = cs40l2x->dev;
+ unsigned int pos = CS40L2X_FW_FILE_HEADER_SIZE;
+ unsigned int block_offset, block_length;
+ unsigned char block_type;
+
+ if (!fw) {
+ dev_err(dev, "Failed to request firmware file\n");
+ return;
+ }
+
+ if (memcmp(fw->data, "WMFW", 4)) {
+ dev_err(dev, "Failed to recognize firmware file\n");
+ goto err_rls_fw;
+ }
+
+ while (pos < fw->size) {
+
+ block_offset = fw->data[pos]
+ + (fw->data[pos + 1] << 8)
+ + (fw->data[pos + 2] << 16);
+ pos += CS40L2X_FW_DBLK_OFFSET_SIZE;
+
+ block_type = fw->data[pos];
+ pos += CS40L2X_FW_DBLK_TYPE_SIZE;
+
+ block_length = fw->data[pos]
+ + (fw->data[pos + 1] << 8)
+ + (fw->data[pos + 2] << 16)
+ + (fw->data[pos + 3] << 24);
+ pos += CS40L2X_FW_DBLK_LENGTH_SIZE;
+
+ switch (block_type) {
+ case CS40L2X_PM_PACKED_TYPE:
+ ret = cs40l2x_raw_write(cs40l2x,
+ CS40L2X_DSP1_PMEM_0
+ + block_offset * 5,
+ &fw->data[pos], block_length,
+ CS40L2X_MAX_WLEN);
+ if (ret) {
+ dev_err(dev,
+ "Failed to write PM_PACKED memory\n");
+ goto err_rls_fw;
+ }
+ break;
+ case CS40L2X_XM_PACKED_TYPE:
+ ret = cs40l2x_raw_write(cs40l2x,
+ CS40L2X_DSP1_XMEM_PACK_0
+ + block_offset * 3,
+ &fw->data[pos], block_length,
+ CS40L2X_MAX_WLEN);
+ if (ret) {
+ dev_err(dev,
+ "Failed to write XM_PACKED memory\n");
+ goto err_rls_fw;
+ }
+ break;
+ case CS40L2X_YM_PACKED_TYPE:
+ ret = cs40l2x_raw_write(cs40l2x,
+ CS40L2X_DSP1_YMEM_PACK_0
+ + block_offset * 3,
+ &fw->data[pos], block_length,
+ CS40L2X_MAX_WLEN);
+ if (ret) {
+ dev_err(dev,
+ "Failed to write YM_PACKED memory\n");
+ goto err_rls_fw;
+ }
+ break;
+ case CS40L2X_ALGO_INFO_TYPE:
+ ret = cs40l2x_algo_parse(cs40l2x, &fw->data[pos]);
+ if (ret) {
+ dev_err(dev,
+ "Failed to parse algorithm: %d\n", ret);
+ goto err_rls_fw;
+ }
+ break;
+ }
+
+ pos += block_length;
+ }
+
+ ret = cs40l2x_coeff_init(cs40l2x);
+ if (ret)
+ goto err_rls_fw;
+
+ request_firmware_nowait(THIS_MODULE, FW_ACTION_HOTPLUG, CS40L2X_WT_NAME,
+ dev, GFP_KERNEL, cs40l2x, cs40l2x_waveform_load);
+err_rls_fw:
+ release_firmware(fw);
+}
+
+static int cs40l2x_boost_config(struct cs40l2x_private *cs40l2x,
+ int boost_ind, int boost_cap, int boost_ipk)
+{
+ int ret;
+ unsigned char bst_lbst_val, bst_cbst_range, bst_ipk_scaled;
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+
+ switch (boost_ind) {
+ case 1000: /* 1.0 uH */
+ bst_lbst_val = 0;
+ break;
+ case 1200: /* 1.2 uH */
+ bst_lbst_val = 1;
+ break;
+ case 1500: /* 1.5 uH */
+ bst_lbst_val = 2;
+ break;
+ case 2200: /* 2.2 uH */
+ bst_lbst_val = 3;
+ break;
+ default:
+ dev_err(dev, "Invalid boost inductor value: %d nH\n",
+ boost_ind);
+ return -EINVAL;
+ }
+
+ switch (boost_cap) {
+ case 0 ... 19:
+ bst_cbst_range = 0;
+ break;
+ case 20 ... 50:
+ bst_cbst_range = 1;
+ break;
+ case 51 ... 100:
+ bst_cbst_range = 2;
+ break;
+ case 101 ... 200:
+ bst_cbst_range = 3;
+ break;
+ default: /* 201 uF and greater */
+ bst_cbst_range = 4;
+ }
+
+ ret = regmap_update_bits(regmap, CS40L2X_BSTCVRT_COEFF,
+ CS40L2X_BST_K1_MASK,
+ cs40l2x_bst_k1_table[bst_lbst_val][bst_cbst_range]
+ << CS40L2X_BST_K1_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to write boost K1 coefficient\n");
+ return ret;
+ }
+
+ ret = regmap_update_bits(regmap, CS40L2X_BSTCVRT_COEFF,
+ CS40L2X_BST_K2_MASK,
+ cs40l2x_bst_k2_table[bst_lbst_val][bst_cbst_range]
+ << CS40L2X_BST_K2_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to write boost K2 coefficient\n");
+ return ret;
+ }
+
+ ret = regmap_update_bits(regmap, CS40L2X_BSTCVRT_SLOPE_LBST,
+ CS40L2X_BST_SLOPE_MASK,
+ cs40l2x_bst_slope_table[bst_lbst_val]
+ << CS40L2X_BST_SLOPE_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to write boost slope coefficient\n");
+ return ret;
+ }
+
+ ret = regmap_update_bits(regmap, CS40L2X_BSTCVRT_SLOPE_LBST,
+ CS40L2X_BST_LBST_VAL_MASK,
+ bst_lbst_val << CS40L2X_BST_LBST_VAL_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to write boost inductor value\n");
+ return ret;
+ }
+
+ if ((boost_ipk < 1600) || (boost_ipk > 4500)) {
+ dev_err(dev, "Invalid boost inductor peak current: %d mA\n",
+ boost_ipk);
+ return -EINVAL;
+ }
+ bst_ipk_scaled = ((boost_ipk - 1600) / 50) + 0x10;
+
+ ret = regmap_update_bits(regmap, CS40L2X_BSTCVRT_PEAK_CUR,
+ CS40L2X_BST_IPK_MASK,
+ bst_ipk_scaled << CS40L2X_BST_IPK_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to write boost inductor peak current\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static const struct reg_sequence cs40l2x_mpu_config[] = {
+ {CS40L2X_DSP1_MPU_LOCK_CONFIG, CS40L2X_MPU_UNLOCK_CODE1},
+ {CS40L2X_DSP1_MPU_LOCK_CONFIG, CS40L2X_MPU_UNLOCK_CODE2},
+ {CS40L2X_DSP1_MPU_XM_ACCESS0, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_YM_ACCESS0, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_WNDW_ACCESS0, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_XREG_ACCESS0, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_YREG_ACCESS0, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_WNDW_ACCESS1, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_XREG_ACCESS1, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_YREG_ACCESS1, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_WNDW_ACCESS2, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_XREG_ACCESS2, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_YREG_ACCESS2, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_WNDW_ACCESS3, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_XREG_ACCESS3, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_YREG_ACCESS3, 0xFFFFFFFF},
+ {CS40L2X_DSP1_MPU_LOCK_CONFIG, 0x00000000}
+};
+
+static int cs40l2x_dsp_load(struct cs40l2x_private *cs40l2x)
+{
+ int ret;
+ unsigned int revid = cs40l2x->revid;
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+
+ switch (revid) {
+ case CS40L2X_REVID_A0:
+ case CS40L2X_REVID_B0:
+ ret = regmap_multi_reg_write(regmap, cs40l2x_mpu_config,
+ ARRAY_SIZE(cs40l2x_mpu_config));
+ if (ret) {
+ dev_err(dev, "Failed to configure MPU\n");
+ return ret;
+ }
+
+ request_firmware_nowait(THIS_MODULE, FW_ACTION_HOTPLUG,
+ CS40L2X_FW_NAME_A0, dev, GFP_KERNEL, cs40l2x,
+ cs40l2x_firmware_load);
+ return 0;
+ case CS40L2X_REVID_B1:
+ request_firmware_nowait(THIS_MODULE, FW_ACTION_HOTPLUG,
+ CS40L2X_FW_NAME_B1, dev, GFP_KERNEL, cs40l2x,
+ cs40l2x_firmware_load);
+ return 0;
+ default:
+ dev_err(dev, "No firmware defined for revision %02X\n", revid);
+ return -EINVAL;
+ }
+}
+
+static int cs40l2x_init(struct cs40l2x_private *cs40l2x)
+{
+ int ret;
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+
+ /* REFCLK configuration is handled by revision B1 ROM */
+ if (cs40l2x->pdata.refclk_gpio2 &&
+ (cs40l2x->revid < CS40L2X_REVID_B1)) {
+ ret = regmap_update_bits(regmap, CS40L2X_GPIO_PAD_CONTROL,
+ CS40L2X_GP2_CTRL_MASK,
+ CS40L2X_GP2_CTRL_MCLK
+ << CS40L2X_GP2_CTRL_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to select GPIO2 function\n");
+ return ret;
+ }
+
+ ret = regmap_update_bits(regmap, CS40L2X_PLL_CLK_CTRL,
+ CS40L2X_PLL_REFCLK_SEL_MASK,
+ CS40L2X_PLL_REFCLK_SEL_MCLK
+ << CS40L2X_PLL_REFCLK_SEL_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to select clock source\n");
+ return ret;
+ }
+ }
+
+ ret = cs40l2x_boost_config(cs40l2x, cs40l2x->pdata.boost_ind,
+ cs40l2x->pdata.boost_cap, cs40l2x->pdata.boost_ipk);
+ if (ret)
+ return ret;
+
+ ret = regmap_update_bits(regmap, CS40L2X_DAC_PCM1_SRC,
+ CS40L2X_DAC_PCM1_SRC_MASK,
+ CS40L2X_DAC_PCM1_SRC_DSP1TX1
+ << CS40L2X_DAC_PCM1_SRC_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to route DSP to amplifier\n");
+ return ret;
+ }
+
+ ret = regmap_update_bits(regmap, CS40L2X_DSP1_RX2_SRC,
+ CS40L2X_DSP1_RXn_SRC_MASK,
+ CS40L2X_DSP1_RXn_SRC_VMON
+ << CS40L2X_DSP1_RXn_SRC_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to route voltage monitor to DSP\n");
+ return ret;
+ }
+
+ ret = regmap_update_bits(regmap, CS40L2X_DSP1_RX3_SRC,
+ CS40L2X_DSP1_RXn_SRC_MASK,
+ CS40L2X_DSP1_RXn_SRC_IMON
+ << CS40L2X_DSP1_RXn_SRC_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to route current monitor to DSP\n");
+ return ret;
+ }
+
+ ret = regmap_update_bits(regmap, CS40L2X_DSP1_RX4_SRC,
+ CS40L2X_DSP1_RXn_SRC_MASK,
+ CS40L2X_DSP1_RXn_SRC_VPMON
+ << CS40L2X_DSP1_RXn_SRC_SHIFT);
+ if (ret) {
+ dev_err(dev, "Failed to route battery monitor to DSP\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static int cs40l2x_otp_unpack(struct cs40l2x_private *cs40l2x)
+{
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+ struct cs40l2x_trim trim;
+ unsigned char row_offset, col_offset;
+ unsigned int val, otp_map;
+ unsigned int *otp_mem;
+ int ret, i;
+
+ otp_mem = kmalloc_array(CS40L2X_NUM_OTP_WORDS,
+ sizeof(*otp_mem), GFP_KERNEL);
+ if (!otp_mem)
+ return -ENOMEM;
+
+ ret = regmap_read(cs40l2x->regmap, CS40L2X_OTPID, &val);
+ if (ret) {
+ dev_err(dev, "Failed to read OTP ID\n");
+ goto err_otp_unpack;
+ }
+
+ /* hard matching against known OTP IDs */
+ for (i = 0; i < CS40L2X_NUM_OTP_MAPS; i++) {
+ if (cs40l2x_otp_map[i].id == val) {
+ otp_map = i;
+ break;
+ }
+ }
+
+ /* reject unrecognized IDs, including untrimmed devices (OTP ID = 0) */
+ if (i == CS40L2X_NUM_OTP_MAPS) {
+ dev_err(dev, "Unrecognized OTP ID: 0x%01X\n", val);
+ ret = -ENODEV;
+ goto err_otp_unpack;
+ }
+
+ dev_dbg(dev, "Found OTP ID: 0x%01X\n", val);
+
+ ret = regmap_bulk_read(regmap, CS40L2X_OTP_MEM0, otp_mem,
+ CS40L2X_NUM_OTP_WORDS);
+ if (ret) {
+ dev_err(dev, "Failed to read OTP contents\n");
+ goto err_otp_unpack;
+ }
+
+ ret = regmap_write(regmap, CS40L2X_TEST_KEY_CTL,
+ CS40L2X_TEST_KEY_UNLOCK_CODE1);
+ if (ret) {
+ dev_err(dev, "Failed to unlock test space (step 1 of 2)\n");
+ goto err_otp_unpack;
+ }
+
+ ret = regmap_write(regmap, CS40L2X_TEST_KEY_CTL,
+ CS40L2X_TEST_KEY_UNLOCK_CODE2);
+ if (ret) {
+ dev_err(dev, "Failed to unlock test space (step 2 of 2)\n");
+ goto err_otp_unpack;
+ }
+
+ row_offset = cs40l2x_otp_map[otp_map].row_start;
+ col_offset = cs40l2x_otp_map[otp_map].col_start;
+
+ for (i = 0; i < cs40l2x_otp_map[otp_map].num_trims; i++) {
+ trim = cs40l2x_otp_map[otp_map].trim_table[i];
+
+ if (col_offset + trim.size - 1 > 31) {
+ /* trim straddles word boundary */
+ val = (otp_mem[row_offset] &
+ GENMASK(31, col_offset)) >> col_offset;
+ val |= (otp_mem[row_offset + 1] &
+ GENMASK(col_offset + trim.size - 33, 0))
+ << (32 - col_offset);
+ } else {
+ /* trim does not straddle word boundary */
+ val = (otp_mem[row_offset] &
+ GENMASK(col_offset + trim.size - 1,
+ col_offset)) >> col_offset;
+ }
+
+ /* advance column marker and wrap if necessary */
+ col_offset += trim.size;
+ if (col_offset > 31) {
+ col_offset -= 32;
+ row_offset++;
+ }
+
+ /* skip blank trims */
+ if (trim.reg == 0)
+ continue;
+
+ ret = regmap_update_bits(regmap, trim.reg,
+ GENMASK(trim.shift + trim.size - 1, trim.shift),
+ val << trim.shift);
+ if (ret) {
+ dev_err(dev, "Failed to write trim %d\n", i + 1);
+ goto err_otp_unpack;
+ }
+
+ dev_dbg(dev, "Trim %d: wrote 0x%X to 0x%08X bits [%d:%d]\n",
+ i + 1, val, trim.reg,
+ trim.shift + trim.size - 1, trim.shift);
+ }
+
+ ret = regmap_write(regmap, CS40L2X_TEST_KEY_CTL,
+ CS40L2X_TEST_KEY_RELOCK_CODE1);
+ if (ret) {
+ dev_err(dev, "Failed to lock test space (step 1 of 2)\n");
+ goto err_otp_unpack;
+ }
+
+ ret = regmap_write(regmap, CS40L2X_TEST_KEY_CTL,
+ CS40L2X_TEST_KEY_RELOCK_CODE2);
+ if (ret) {
+ dev_err(dev, "Failed to lock test space (step 2 of 2)\n");
+ goto err_otp_unpack;
+ }
+
+ ret = 0;
+
+err_otp_unpack:
+ kfree(otp_mem);
+
+ return ret;
+}
+
+static int cs40l2x_handle_of_data(struct i2c_client *i2c_client,
+ struct cs40l2x_platform_data *pdata)
+{
+ struct device_node *np = i2c_client->dev.of_node;
+ struct device *dev = &i2c_client->dev;
+ int ret;
+ unsigned int out_val;
+
+ if (!np)
+ return 0;
+
+ ret = of_property_read_u32(np, "cirrus,boost-ind-nanohenry", &out_val);
+ if (ret) {
+ dev_err(dev, "Boost inductor value not specified\n");
+ return -EINVAL;
+ }
+ pdata->boost_ind = out_val;
+
+ ret = of_property_read_u32(np, "cirrus,boost-cap-microfarad", &out_val);
+ if (ret) {
+ dev_err(dev, "Boost capacitance not specified\n");
+ return -EINVAL;
+ }
+ pdata->boost_cap = out_val;
+
+ ret = of_property_read_u32(np, "cirrus,boost-ipk-milliamp", &out_val);
+ if (ret) {
+ dev_err(dev, "Boost inductor peak current not specified\n");
+ return -EINVAL;
+ }
+ pdata->boost_ipk = out_val;
+
+ pdata->refclk_gpio2 = of_property_read_bool(np, "cirrus,refclk-gpio2");
+
+ ret = of_property_read_u32(np, "cirrus,f0-default", &out_val);
+ if (!ret)
+ pdata->f0_default = out_val;
+
+ ret = of_property_read_u32(np, "cirrus,f0-min", &out_val);
+ if (!ret)
+ pdata->f0_min = out_val;
+
+ ret = of_property_read_u32(np, "cirrus,f0-max", &out_val);
+ if (!ret)
+ pdata->f0_max = out_val;
+
+ ret = of_property_read_u32(np, "cirrus,redc-default", &out_val);
+ if (!ret)
+ pdata->redc_default = out_val;
+
+ ret = of_property_read_u32(np, "cirrus,redc-min", &out_val);
+ if (!ret)
+ pdata->redc_min = out_val;
+
+ ret = of_property_read_u32(np, "cirrus,redc-max", &out_val);
+ if (!ret)
+ pdata->redc_max = out_val;
+
+ ret = of_property_read_u32(np, "cirrus,gpio1-rise-index", &out_val);
+ if (!ret)
+ pdata->gpio1_rise_index = out_val;
+
+ ret = of_property_read_u32(np, "cirrus,gpio1-fall-index", &out_val);
+ if (!ret)
+ pdata->gpio1_fall_index = out_val;
+
+ ret = of_property_read_u32(np, "cirrus,gpio1-fall-timeout", &out_val);
+ if (!ret)
+ pdata->gpio1_fall_timeout = out_val | CS40L2X_PDATA_PRESENT;
+
+ ret = of_property_read_u32(np, "cirrus,gpio1-mode", &out_val);
+ if (!ret) {
+ if (out_val > CS40L2X_GPIO1_MODE_MAX)
+ dev_warn(dev, "Ignored default gpio1_mode\n");
+ else
+ pdata->gpio1_mode = out_val;
+ }
+
+ return 0;
+}
+
+static int cs40l2x_basic_mode_exit(struct cs40l2x_private *cs40l2x)
+{
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+ unsigned int val;
+ int shutdown_timeout = CS40L2X_BASIC_TIMEOUT_COUNT;
+ int ret;
+
+ ret = regmap_read(regmap, CS40L2X_BASIC_AMP_STATUS, &val);
+ if (ret) {
+ dev_err(dev, "Failed to read basic-mode status\n");
+ return ret;
+ }
+
+ if (val != CS40L2X_BASIC_BOOT_DONE) {
+ dev_err(dev, "Unexpected basic-mode status: %02X\n", val);
+ return -EIO;
+ }
+
+ ret = regmap_write(regmap, CS40L2X_BASIC_SHUTDOWNREQUEST, 1);
+ if (ret) {
+ dev_err(dev, "Failed to write shutdown request\n");
+ return ret;
+ }
+
+ while (shutdown_timeout > 0) {
+ usleep_range(10000, 10100);
+
+ ret = regmap_read(regmap, CS40L2X_BASIC_SHUTDOWNREQUEST, &val);
+ if (ret) {
+ dev_err(dev, "Failed to read shutdown request\n");
+ return ret;
+ }
+
+ if (!val)
+ break;
+
+ shutdown_timeout--;
+ }
+
+ if (shutdown_timeout == 0) {
+ dev_err(dev, "Timed out waiting for basic-mode shutdown\n");
+ return -ETIME;
+ }
+
+ ret = regmap_read(regmap, CS40L2X_BASIC_STATEMACHINE, &val);
+ if (ret) {
+ dev_err(dev, "Failed to read basic-mode state\n");
+ return ret;
+ }
+
+ if (val != CS40L2X_BASIC_SHUTDOWN) {
+ dev_err(dev, "Unexpected basic-mode state: %02X\n", val);
+ return -EBUSY;
+ }
+
+ dev_info(dev, "Basic-mode haptics successfully stopped\n");
+
+ return 0;
+}
+
+static const struct reg_sequence cs40l2x_rev_a0_errata[] = {
+ {CS40L2X_OTP_TRIM_30, 0x9091A1C8},
+ {CS40L2X_PLL_LOOP_PARAM, 0x000C1837},
+ {CS40L2X_PLL_MISC_CTRL, 0x03008E0E},
+ {CS40L2X_BSTCVRT_DCM_CTRL, 0x00000051},
+ {CS40L2X_CTRL_ASYNC1, 0x00000004},
+ {CS40L2X_IRQ1_DB3, 0x00000000},
+ {CS40L2X_IRQ2_DB3, 0x00000000},
+};
+
+static const struct reg_sequence cs40l2x_rev_b0_errata[] = {
+ {CS40L2X_PLL_LOOP_PARAM, 0x000C1837},
+ {CS40L2X_PLL_MISC_CTRL, 0x03008E0E},
+ {CS40L2X_TEST_KEY_CTL, CS40L2X_TEST_KEY_UNLOCK_CODE1},
+ {CS40L2X_TEST_KEY_CTL, CS40L2X_TEST_KEY_UNLOCK_CODE2},
+ {CS40L2X_OTP_TRIM_12, 0x002F0065},
+ {CS40L2X_OTP_TRIM_13, 0x00002B4F},
+ {CS40L2X_TEST_KEY_CTL, CS40L2X_TEST_KEY_RELOCK_CODE1},
+ {CS40L2X_TEST_KEY_CTL, CS40L2X_TEST_KEY_RELOCK_CODE2},
+};
+
+static const struct reg_sequence cs40l2x_basic_mode_revert[] = {
+ {CS40L2X_PWR_CTRL1, 0x00000000},
+ {CS40L2X_PWR_CTRL2, 0x00003321},
+ {CS40L2X_LRCK_PAD_CONTROL, 0x00000007},
+ {CS40L2X_SDIN_PAD_CONTROL, 0x00000007},
+ {CS40L2X_GPIO_PAD_CONTROL, 0x00000000},
+ {CS40L2X_AMP_DIG_VOL_CTRL, 0x00008000},
+ {CS40L2X_IRQ2_MASK1, 0xFFFFFFFF},
+ {CS40L2X_IRQ2_MASK2, 0xFFFFFFFF},
+};
+
+static int cs40l2x_part_num_resolve(struct cs40l2x_private *cs40l2x)
+{
+ struct regmap *regmap = cs40l2x->regmap;
+ struct device *dev = cs40l2x->dev;
+ unsigned int val, devid, revid;
+ unsigned int part_num_index;
+ int otp_timeout = CS40L2X_OTP_TIMEOUT_COUNT;
+ int ret;
+
+ while (otp_timeout > 0) {
+ usleep_range(10000, 10100);
+
+ ret = regmap_read(regmap, CS40L2X_IRQ1_STATUS4, &val);
+ if (ret) {
+ dev_err(dev, "Failed to read OTP boot status\n");
+ return ret;
+ }
+
+ if (val & CS40L2X_OTP_BOOT_DONE)
+ break;
+
+ otp_timeout--;
+ }
+
+ if (otp_timeout == 0) {
+ dev_err(dev, "Timed out waiting for OTP boot\n");
+ return -ETIME;
+ }
+
+ ret = regmap_read(regmap, CS40L2X_IRQ1_STATUS3, &val);
+ if (ret) {
+ dev_err(dev, "Failed to read OTP error status\n");
+ return ret;
+ }
+
+ if (val & CS40L2X_OTP_BOOT_ERR) {
+ dev_err(dev, "Encountered fatal OTP error\n");
+ return -EIO;
+ }
+
+ ret = regmap_read(regmap, CS40L2X_DEVID, &devid);
+ if (ret) {
+ dev_err(dev, "Failed to read device ID\n");
+ return ret;
+ }
+
+ ret = regmap_read(regmap, CS40L2X_REVID, &revid);
+ if (ret) {
+ dev_err(dev, "Failed to read revision ID\n");
+ return ret;
+ }
+
+ switch (devid) {
+ case CS40L2X_DEVID_L20:
+ part_num_index = 0;
+ if (revid != CS40L2X_REVID_A0)
+ goto err_revid;
+
+ ret = regmap_register_patch(regmap, cs40l2x_rev_a0_errata,
+ ARRAY_SIZE(cs40l2x_rev_a0_errata));
+ if (ret) {
+ dev_err(dev, "Failed to apply revision %02X errata\n",
+ revid);
+ return ret;
+ }
+
+ ret = cs40l2x_otp_unpack(cs40l2x);
+ if (ret)
+ return ret;
+ break;
+ case CS40L2X_DEVID_L25:
+ part_num_index = 1;
+ if (revid != CS40L2X_REVID_B0)
+ goto err_revid;
+
+ ret = regmap_register_patch(regmap, cs40l2x_rev_b0_errata,
+ ARRAY_SIZE(cs40l2x_rev_b0_errata));
+ if (ret) {
+ dev_err(dev, "Failed to apply revision %02X errata\n",
+ revid);
+ return ret;
+ }
+ break;
+ case CS40L2X_DEVID_L25A:
+ case CS40L2X_DEVID_L25B:
+ part_num_index = devid - CS40L2X_DEVID_L25A + 2;
+ if (revid < CS40L2X_REVID_B1)
+ goto err_revid;
+
+ ret = cs40l2x_basic_mode_exit(cs40l2x);
+ if (ret)
+ return ret;
+
+ ret = regmap_multi_reg_write(regmap, cs40l2x_basic_mode_revert,
+ ARRAY_SIZE(cs40l2x_basic_mode_revert));
+ if (ret) {
+ dev_err(dev, "Failed to revert basic-mode fields\n");
+ return ret;
+ }
+
+ ret = regmap_register_patch(regmap, cs40l2x_rev_b0_errata,
+ ARRAY_SIZE(cs40l2x_rev_b0_errata));
+ if (ret) {
+ dev_err(dev, "Failed to apply revision %02X errata\n",
+ revid);
+ return ret;
+ }
+ break;
+ default:
+ dev_err(dev, "Unrecognized device ID: 0x%06X\n", devid);
+ return -ENODEV;
+ }
+
+ dev_info(dev, "Cirrus Logic %s revision %02X\n",
+ cs40l2x_part_nums[part_num_index], revid);
+ cs40l2x->devid = devid;
+ cs40l2x->revid = revid;
+
+ return 0;
+err_revid:
+ dev_err(dev, "Unexpected revision ID for %s: %02X\n",
+ cs40l2x_part_nums[part_num_index], revid);
+ return -ENODEV;
+}
+
+static struct regmap_config cs40l2x_regmap = {
+ .reg_bits = 32,
+ .val_bits = 32,
+ .reg_stride = 4,
+ .reg_format_endian = REGMAP_ENDIAN_BIG,
+ .val_format_endian = REGMAP_ENDIAN_BIG,
+ .max_register = CS40L2X_LASTREG,
+ .precious_reg = cs40l2x_precious_reg,
+ .readable_reg = cs40l2x_readable_reg,
+ .cache_type = REGCACHE_NONE,
+};
+
+static int cs40l2x_i2c_probe(struct i2c_client *i2c_client,
+ const struct i2c_device_id *id)
+{
+ int ret, i;
+ struct cs40l2x_private *cs40l2x;
+ struct device *dev = &i2c_client->dev;
+ struct cs40l2x_platform_data *pdata = dev_get_platdata(dev);
+
+ cs40l2x = devm_kzalloc(dev, sizeof(struct cs40l2x_private), GFP_KERNEL);
+ if (!cs40l2x)
+ return -ENOMEM;
+
+ cs40l2x->dev = dev;
+ dev_set_drvdata(dev, cs40l2x);
+ i2c_set_clientdata(i2c_client, cs40l2x);
+
+ mutex_init(&cs40l2x->lock);
+
+ INIT_LIST_HEAD(&cs40l2x->coeff_desc_head);
+
+ cs40l2x->regmap = devm_regmap_init_i2c(i2c_client, &cs40l2x_regmap);
+ if (IS_ERR(cs40l2x->regmap)) {
+ ret = PTR_ERR(cs40l2x->regmap);
+ dev_err(dev, "Failed to allocate register map: %d\n", ret);
+ return ret;
+ }
+
+ for (i = 0; i < ARRAY_SIZE(cs40l2x_supplies); i++)
+ cs40l2x->supplies[i].supply = cs40l2x_supplies[i];
+
+ cs40l2x->num_supplies = ARRAY_SIZE(cs40l2x_supplies);
+
+ ret = devm_regulator_bulk_get(dev, cs40l2x->num_supplies,
+ cs40l2x->supplies);
+ if (ret) {
+ dev_err(dev, "Failed to request core supplies: %d\n", ret);
+ return ret;
+ }
+
+ if (pdata) {
+ cs40l2x->pdata = *pdata;
+ } else {
+ pdata = devm_kzalloc(dev, sizeof(struct cs40l2x_platform_data),
+ GFP_KERNEL);
+ if (!pdata)
+ return -ENOMEM;
+
+ if (i2c_client->dev.of_node) {
+ ret = cs40l2x_handle_of_data(i2c_client, pdata);
+ if (ret)
+ return ret;
+
+ }
+ cs40l2x->pdata = *pdata;
+ }
+
+ ret = regulator_bulk_enable(cs40l2x->num_supplies, cs40l2x->supplies);
+ if (ret) {
+ dev_err(dev, "Failed to enable core supplies: %d\n", ret);
+ return ret;
+ }
+
+ cs40l2x->reset_gpio = devm_gpiod_get_optional(dev, "reset",
+ GPIOD_OUT_LOW);
+ if (IS_ERR(cs40l2x->reset_gpio))
+ return PTR_ERR(cs40l2x->reset_gpio);
+
+ /* satisfy reset pulse width specification (with margin) */
+ usleep_range(2000, 2100);
+
+ gpiod_set_value_cansleep(cs40l2x->reset_gpio, 1);
+
+ /* satisfy control port delay specification (with margin) */
+ usleep_range(1000, 1100);
+
+ ret = cs40l2x_part_num_resolve(cs40l2x);
+ if (ret)
+ goto err;
+
+ ret = cs40l2x_init(cs40l2x);
+ if (ret)
+ goto err;
+
+ ret = cs40l2x_dsp_load(cs40l2x);
+ if (ret)
+ goto err;
+
+ cs40l2x_create_led(cs40l2x);
+
+ return 0;
+err:
+ gpiod_set_value_cansleep(cs40l2x->reset_gpio, 0);
+
+ regulator_bulk_disable(cs40l2x->num_supplies, cs40l2x->supplies);
+
+ return ret;
+}
+
+static int cs40l2x_i2c_remove(struct i2c_client *i2c_client)
+{
+ struct cs40l2x_private *cs40l2x = i2c_get_clientdata(i2c_client);
+
+ if (cs40l2x->vibe_init_success) {
+ led_classdev_unregister(&cs40l2x->led_dev);
+
+ sysfs_remove_group(&cs40l2x->dev->kobj,
+ &cs40l2x_dev_attr_group);
+
+ hrtimer_cancel(&cs40l2x->pbq_timer);
+
+ cancel_work_sync(&cs40l2x->vibe_start_work);
+ cancel_work_sync(&cs40l2x->vibe_pbq_work);
+ cancel_work_sync(&cs40l2x->vibe_stop_work);
+
+ destroy_workqueue(cs40l2x->vibe_workqueue);
+
+ device_init_wakeup(cs40l2x->dev, false);
+ }
+
+ gpiod_set_value_cansleep(cs40l2x->reset_gpio, 0);
+
+ regulator_bulk_disable(cs40l2x->num_supplies, cs40l2x->supplies);
+
+ mutex_destroy(&cs40l2x->lock);
+
+ return 0;
+}
+
+static int __maybe_unused cs40l2x_suspend(struct device *dev)
+{
+ struct cs40l2x_private *cs40l2x = dev_get_drvdata(dev);
+ int ret = 0;
+
+ mutex_lock(&cs40l2x->lock);
+
+ if (cs40l2x->pdata.gpio1_mode == CS40L2X_GPIO1_MODE_AUTO) {
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "GPIO_ENABLE",
+ CS40L2X_XM_UNPACKED_TYPE),
+ CS40L2X_GPIO1_ENABLED);
+ if (ret)
+ dev_err(dev, "Failed to enable GPIO1 upon suspend\n");
+ }
+
+ mutex_unlock(&cs40l2x->lock);
+
+ return ret;
+}
+
+static int __maybe_unused cs40l2x_resume(struct device *dev)
+{
+ struct cs40l2x_private *cs40l2x = dev_get_drvdata(dev);
+ int ret = 0;
+
+ mutex_lock(&cs40l2x->lock);
+
+ if (cs40l2x->pdata.gpio1_mode == CS40L2X_GPIO1_MODE_AUTO) {
+ ret = regmap_write(cs40l2x->regmap,
+ cs40l2x_dsp_reg(cs40l2x, "GPIO_ENABLE",
+ CS40L2X_XM_UNPACKED_TYPE),
+ CS40L2X_GPIO1_DISABLED);
+ if (ret)
+ dev_err(dev, "Failed to disable GPIO1 upon resume\n");
+ }
+
+ mutex_unlock(&cs40l2x->lock);
+
+ return ret;
+}
+
+static SIMPLE_DEV_PM_OPS(cs40l2x_pm_ops, cs40l2x_suspend, cs40l2x_resume);
+
+static const struct of_device_id cs40l2x_of_match[] = {
+ { .compatible = "cirrus,cs40l20" },
+ { .compatible = "cirrus,cs40l25" },
+ { .compatible = "cirrus,cs40l25a" },
+ { .compatible = "cirrus,cs40l25b" },
+ { }
+};
+
+MODULE_DEVICE_TABLE(of, cs40l2x_of_match);
+
+static const struct i2c_device_id cs40l2x_id[] = {
+ { "cs40l20", 0 },
+ { "cs40l25", 1 },
+ { "cs40l25a", 2 },
+ { "cs40l25b", 3 },
+ { }
+};
+
+MODULE_DEVICE_TABLE(i2c, cs40l2x_id);
+
+static struct i2c_driver cs40l2x_i2c_driver = {
+ .driver = {
+ .name = "cs40l2x",
+ .of_match_table = cs40l2x_of_match,
+ .pm = &cs40l2x_pm_ops,
+ },
+ .id_table = cs40l2x_id,
+ .probe = cs40l2x_i2c_probe,
+ .remove = cs40l2x_i2c_remove,
+};
+
+module_i2c_driver(cs40l2x_i2c_driver);
+
+MODULE_DESCRIPTION("CS40L20/CS40L25/CS40L25A/CS40L25B Haptics Driver");
+MODULE_AUTHOR("Jeff LaBundy, Cirrus Logic Inc, <jeff.labundy@cirrus.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/misc/cs40l2x.h b/drivers/misc/cs40l2x.h
new file mode 100644
index 0000000..a854e63
--- /dev/null
+++ b/drivers/misc/cs40l2x.h
@@ -0,0 +1,805 @@
+/*
+ * cs40l2x.h -- CS40L20/CS40L25/CS40L25A/CS40L25B Haptics Driver
+ *
+ * Copyright 2018 Cirrus Logic, Inc.
+ *
+ * Author: Jeff LaBundy <jeff.labundy@cirrus.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ */
+
+#ifndef __CS40L2X_H__
+#define __CS40L2X_H__
+
+#include <linux/regmap.h>
+
+#define CS40L2X_FIRSTREG 0x00000000
+#define CS40L2X_LASTREG 0x03804FE8
+#define CS40L2X_DEVID 0x00000000
+#define CS40L2X_REVID 0x00000004
+#define CS40L2X_FABID 0x00000008
+#define CS40L2X_RELID 0x0000000C
+#define CS40L2X_OTPID 0x00000010
+#define CS40L2X_SFT_RESET 0x00000020
+#define CS40L2X_TEST_KEY_CTL 0x00000040
+#define CS40L2X_USER_KEY_CTL 0x00000044
+#define CS40L2X_CTRL_ASYNC1 0x00000054
+#define CS40L2X_OTP_MEM0 0x00000400
+#define CS40L2X_OTP_MEM31 0x0000047C
+#define CS40L2X_OTP_CTRL0 0x00000500
+#define CS40L2X_OTP_CTRL1 0x00000504
+#define CS40L2X_OTP_CTRL3 0x00000508
+#define CS40L2X_OTP_CTRL4 0x0000050C
+#define CS40L2X_OTP_CTRL5 0x00000510
+#define CS40L2X_OTP_CTRL6 0x00000514
+#define CS40L2X_OTP_CTRL7 0x00000518
+#define CS40L2X_OTP_CTRL8 0x0000051C
+#define CS40L2X_PWR_CTRL1 0x00002014
+#define CS40L2X_PWR_CTRL2 0x00002018
+#define CS40L2X_PWR_CTRL3 0x0000201C
+#define CS40L2X_CTRL_OVRRIDE 0x00002020
+#define CS40L2X_AMP_OUT_MUTE 0x00002024
+#define CS40L2X_PROTECT_REL_ERR_IGN 0x00002034
+#define CS40L2X_LRCK_PAD_CONTROL 0x00002418
+#define CS40L2X_SDIN_PAD_CONTROL 0x00002420
+#define CS40L2X_GPIO_PAD_CONTROL 0x0000242C
+#define CS40L2X_JTAG_CONTROL 0x00002438
+#define CS40L2X_PWRMGT_CTL 0x00002900
+#define CS40L2X_WAKESRC_CTL 0x00002904
+#define CS40L2X_PWRMGT_STS 0x00002908
+#define CS40L2X_PWRMGT_TEST 0x0000290C
+#define CS40L2X_PLL_CLK_CTRL 0x00002C04
+#define CS40L2X_DSP_CLK_CTRL 0x00002C08
+#define CS40L2X_GLOBAL_CLK_CTRL 0x00002C0C
+#define CS40L2X_DATA_FS_SEL 0x00002C10
+#define CS40L2X_PLL_LOOP_PARAM 0x00003008
+#define CS40L2X_PLL_MISC_CTRL 0x00003014
+#define CS40L2X_MDSYNC_EN 0x00003400
+#define CS40L2X_MDSYNC_TX_ID 0x00003408
+#define CS40L2X_MDSYNC_PWR_CTRL 0x0000340C
+#define CS40L2X_MDSYNC_DATA_TX 0x00003410
+#define CS40L2X_MDSYNC_TX_STATUS 0x00003414
+#define CS40L2X_MDSYNC_DATA_RX 0x0000341C
+#define CS40L2X_MDSYNC_RX_STATUS 0x00003420
+#define CS40L2X_MDSYNC_ERR_STATUS 0x00003424
+#define CS40L2X_MDSYNC_SYNC_PTE2 0x00003528
+#define CS40L2X_MDSYNC_SYNC_PTE3 0x0000352C
+#define CS40L2X_MDSYNC_SYNC_MSM_STATUS 0x0000353C
+#define CS40L2X_BSTCVRT_VCTRL1 0x00003800
+#define CS40L2X_BSTCVRT_VCTRL2 0x00003804
+#define CS40L2X_BSTCVRT_PEAK_CUR 0x00003808
+#define CS40L2X_BSTCVRT_SFT_RAMP 0x0000380C
+#define CS40L2X_BSTCVRT_COEFF 0x00003810
+#define CS40L2X_BSTCVRT_SLOPE_LBST 0x00003814
+#define CS40L2X_BSTCVRT_SW_FREQ 0x00003818
+#define CS40L2X_BSTCVRT_DCM_CTRL 0x0000381C
+#define CS40L2X_BSTCVRT_DCM_MODE_FORCE 0x00003820
+#define CS40L2X_BSTCVRT_OVERVOLT_CTRL 0x00003830
+#define CS40L2X_VI_VOL_POL 0x00004000
+#define CS40L2X_DTEMP_WARN_THLD 0x00004220
+#define CS40L2X_DTEMP_CFG 0x00004224
+#define CS40L2X_DTEMP_EN 0x00004308
+#define CS40L2X_VPVBST_FS_SEL 0x00004400
+#define CS40L2X_SP_ENABLES 0x00004800
+#define CS40L2X_SP_RATE_CTRL 0x00004804
+#define CS40L2X_SP_FORMAT 0x00004808
+#define CS40L2X_SP_HIZ_CTRL 0x0000480C
+#define CS40L2X_SP_FRAME_TX_SLOT 0x00004810
+#define CS40L2X_SP_FRAME_RX_SLOT 0x00004820
+#define CS40L2X_SP_TX_WL 0x00004830
+#define CS40L2X_SP_RX_WL 0x00004840
+#define CS40L2X_DAC_PCM1_SRC 0x00004C00
+#define CS40L2X_ASP_TX1_SRC 0x00004C20
+#define CS40L2X_ASP_TX2_SRC 0x00004C24
+#define CS40L2X_ASP_TX3_SRC 0x00004C28
+#define CS40L2X_ASP_TX4_SRC 0x00004C2C
+#define CS40L2X_DSP1_RX1_SRC 0x00004C40
+#define CS40L2X_DSP1_RX2_SRC 0x00004C44
+#define CS40L2X_DSP1_RX3_SRC 0x00004C48
+#define CS40L2X_DSP1_RX4_SRC 0x00004C4C
+#define CS40L2X_DSP1_RX5_SRC 0x00004C50
+#define CS40L2X_DSP1_RX6_SRC 0x00004C54
+#define CS40L2X_DSP1_RX7_SRC 0x00004C58
+#define CS40L2X_DSP1_RX8_SRC 0x00004C5C
+#define CS40L2X_NGATE1_SRC 0x00004C60
+#define CS40L2X_NGATE2_SRC 0x00004C64
+#define CS40L2X_AMP_DIG_VOL_CTRL 0x00006000
+#define CS40L2X_VPBR_CFG 0x00006404
+#define CS40L2X_VBBR_CFG 0x00006408
+#define CS40L2X_VPBR_STATUS 0x0000640C
+#define CS40L2X_VBBR_STATUS 0x00006410
+#define CS40L2X_OVERTEMP_CFG 0x00006414
+#define CS40L2X_AMP_ERR_VOL 0x00006418
+#define CS40L2X_VOL_STATUS_TO_DSP 0x00006450
+#define CS40L2X_CLASSH_CFG 0x00006800
+#define CS40L2X_WKFET_CFG 0x00006804
+#define CS40L2X_NG_CFG 0x00006808
+#define CS40L2X_AMP_GAIN_CTRL 0x00006C04
+#define CS40L2X_DAC_MSM_CFG 0x00007400
+#define CS40L2X_IRQ1_CFG 0x00010000
+#define CS40L2X_IRQ1_STATUS 0x00010004
+#define CS40L2X_IRQ1_STATUS1 0x00010010
+#define CS40L2X_IRQ1_STATUS2 0x00010014
+#define CS40L2X_IRQ1_STATUS3 0x00010018
+#define CS40L2X_IRQ1_STATUS4 0x0001001C
+#define CS40L2X_IRQ1_RAW_STATUS1 0x00010090
+#define CS40L2X_IRQ1_RAW_STATUS2 0x00010094
+#define CS40L2X_IRQ1_RAW_STATUS3 0x00010098
+#define CS40L2X_IRQ1_RAW_STATUS4 0x0001009C
+#define CS40L2X_IRQ1_MASK1 0x00010110
+#define CS40L2X_IRQ1_MASK2 0x00010114
+#define CS40L2X_IRQ1_MASK3 0x00010118
+#define CS40L2X_IRQ1_MASK4 0x0001011C
+#define CS40L2X_IRQ1_FRC1 0x00010190
+#define CS40L2X_IRQ1_FRC2 0x00010194
+#define CS40L2X_IRQ1_FRC3 0x00010198
+#define CS40L2X_IRQ1_FRC4 0x0001019C
+#define CS40L2X_IRQ1_EDGE1 0x00010210
+#define CS40L2X_IRQ1_EDGE4 0x0001021C
+#define CS40L2X_IRQ1_POL1 0x00010290
+#define CS40L2X_IRQ1_POL2 0x00010294
+#define CS40L2X_IRQ1_POL3 0x00010298
+#define CS40L2X_IRQ1_POL4 0x0001029C
+#define CS40L2X_IRQ1_DB3 0x00010318
+#define CS40L2X_IRQ2_CFG 0x00010800
+#define CS40L2X_IRQ2_STATUS 0x00010804
+#define CS40L2X_IRQ2_STATUS1 0x00010810
+#define CS40L2X_IRQ2_STATUS2 0x00010814
+#define CS40L2X_IRQ2_STATUS3 0x00010818
+#define CS40L2X_IRQ2_STATUS4 0x0001081C
+#define CS40L2X_IRQ2_RAW_STATUS1 0x00010890
+#define CS40L2X_IRQ2_RAW_STATUS2 0x00010894
+#define CS40L2X_IRQ2_RAW_STATUS3 0x00010898
+#define CS40L2X_IRQ2_RAW_STATUS4 0x0001089C
+#define CS40L2X_IRQ2_MASK1 0x00010910
+#define CS40L2X_IRQ2_MASK2 0x00010914
+#define CS40L2X_IRQ2_MASK3 0x00010918
+#define CS40L2X_IRQ2_MASK4 0x0001091C
+#define CS40L2X_IRQ2_FRC1 0x00010990
+#define CS40L2X_IRQ2_FRC2 0x00010994
+#define CS40L2X_IRQ2_FRC3 0x00010998
+#define CS40L2X_IRQ2_FRC4 0x0001099C
+#define CS40L2X_IRQ2_EDGE1 0x00010A10
+#define CS40L2X_IRQ2_EDGE4 0x00010A1C
+#define CS40L2X_IRQ2_POL1 0x00010A90
+#define CS40L2X_IRQ2_POL2 0x00010A94
+#define CS40L2X_IRQ2_POL3 0x00010A98
+#define CS40L2X_IRQ2_POL4 0x00010A9C
+#define CS40L2X_IRQ2_DB3 0x00010B18
+#define CS40L2X_GPIO_STATUS1 0x00011000
+#define CS40L2X_GPIO1_CTRL1 0x00011008
+#define CS40L2X_GPIO2_CTRL1 0x0001100C
+#define CS40L2X_GPIO3_CTRL1 0x00011010
+#define CS40L2X_GPIO4_CTRL1 0x00011014
+#define CS40L2X_MIXER_NGATE_CFG 0x00012000
+#define CS40L2X_MIXER_NGATE_CH1_CFG 0x00012004
+#define CS40L2X_MIXER_NGATE_CH2_CFG 0x00012008
+#define CS40L2X_DSP_MBOX_1 0x00013000
+#define CS40L2X_DSP_MBOX_2 0x00013004
+#define CS40L2X_DSP_MBOX_3 0x00013008
+#define CS40L2X_DSP_MBOX_4 0x0001300C
+#define CS40L2X_DSP_MBOX_5 0x00013010
+#define CS40L2X_DSP_MBOX_6 0x00013014
+#define CS40L2X_DSP_MBOX_7 0x00013018
+#define CS40L2X_DSP_MBOX_8 0x0001301C
+#define CS40L2X_DSP_VIRT1_MBOX_1 0x00013020
+#define CS40L2X_DSP_VIRT1_MBOX_2 0x00013024
+#define CS40L2X_DSP_VIRT1_MBOX_3 0x00013028
+#define CS40L2X_DSP_VIRT1_MBOX_4 0x0001302C
+#define CS40L2X_DSP_VIRT1_MBOX_5 0x00013030
+#define CS40L2X_DSP_VIRT1_MBOX_6 0x00013034
+#define CS40L2X_DSP_VIRT1_MBOX_7 0x00013038
+#define CS40L2X_DSP_VIRT1_MBOX_8 0x0001303C
+#define CS40L2X_DSP_VIRT2_MBOX_1 0x00013040
+#define CS40L2X_DSP_VIRT2_MBOX_2 0x00013044
+#define CS40L2X_DSP_VIRT2_MBOX_3 0x00013048
+#define CS40L2X_DSP_VIRT2_MBOX_4 0x0001304C
+#define CS40L2X_DSP_VIRT2_MBOX_5 0x00013050
+#define CS40L2X_DSP_VIRT2_MBOX_6 0x00013054
+#define CS40L2X_DSP_VIRT2_MBOX_7 0x00013058
+#define CS40L2X_DSP_VIRT2_MBOX_8 0x0001305C
+#define CS40L2X_CLOCK_DETECT_1 0x00014000
+#define CS40L2X_TIMER1_CONTROL 0x00015000
+#define CS40L2X_TIMER1_COUNT_PRESET 0x00015004
+#define CS40L2X_TIMER1_START_STOP 0x0001500C
+#define CS40L2X_TIMER1_STATUS 0x00015010
+#define CS40L2X_TIMER1_COUNT_READBACK 0x00015014
+#define CS40L2X_TIMER1_DSP_CLK_CFG 0x00015018
+#define CS40L2X_TIMER1_DSP_CLK_STATUS 0x0001501C
+#define CS40L2X_TIMER2_CONTROL 0x00015100
+#define CS40L2X_TIMER2_COUNT_PRESET 0x00015104
+#define CS40L2X_TIMER2_START_STOP 0x0001510C
+#define CS40L2X_TIMER2_STATUS 0x00015110
+#define CS40L2X_TIMER2_COUNT_READBACK 0x00015114
+#define CS40L2X_TIMER2_DSP_CLK_CFG 0x00015118
+#define CS40L2X_TIMER2_DSP_CLK_STATUS 0x0001511C
+#define CS40L2X_DFT_JTAG_CONTROL 0x00016000
+#define CS40L2X_DIE_STS1 0x00017040
+#define CS40L2X_DIE_STS2 0x00017044
+#define CS40L2X_TEMP_CAL1 0x00017048
+#define CS40L2X_TEMP_CAL2 0x0001704C
+#define CS40L2X_DSP1_XMEM_PACK_0 0x02000000
+#define CS40L2X_DSP1_XMEM_PACK_3068 0x02002FF0
+#define CS40L2X_DSP1_XMEM_UNPACK32_0 0x02400000
+#define CS40L2X_DSP1_XMEM_UNPACK32_2046 0x02401FF8
+#define CS40L2X_DSP1_TIMESTAMP_COUNT 0x025C0800
+#define CS40L2X_DSP1_SYS_ID 0x025E0000
+#define CS40L2X_DSP1_SYS_VERSION 0x025E0004
+#define CS40L2X_DSP1_SYS_CORE_ID 0x025E0008
+#define CS40L2X_DSP1_SYS_AHB_ADDR 0x025E000C
+#define CS40L2X_DSP1_SYS_XSRAM_SIZE 0x025E0010
+#define CS40L2X_DSP1_SYS_YSRAM_SIZE 0x025E0018
+#define CS40L2X_DSP1_SYS_PSRAM_SIZE 0x025E0020
+#define CS40L2X_DSP1_SYS_PM_BOOT_SIZE 0x025E0028
+#define CS40L2X_DSP1_SYS_FEATURES 0x025E002C
+#define CS40L2X_DSP1_SYS_FIR_FILTERS 0x025E0030
+#define CS40L2X_DSP1_SYS_LMS_FILTERS 0x025E0034
+#define CS40L2X_DSP1_SYS_XM_BANK_SIZE 0x025E0038
+#define CS40L2X_DSP1_SYS_YM_BANK_SIZE 0x025E003C
+#define CS40L2X_DSP1_SYS_PM_BANK_SIZE 0x025E0040
+#define CS40L2X_DSP1_AHBM_WIN0_CTRL0 0x025E2000
+#define CS40L2X_DSP1_AHBM_WIN0_CTRL1 0x025E2004
+#define CS40L2X_DSP1_AHBM_WIN1_CTRL0 0x025E2008
+#define CS40L2X_DSP1_AHBM_WIN1_CTRL1 0x025E200C
+#define CS40L2X_DSP1_AHBM_WIN2_CTRL0 0x025E2010
+#define CS40L2X_DSP1_AHBM_WIN2_CTRL1 0x025E2014
+#define CS40L2X_DSP1_AHBM_WIN3_CTRL0 0x025E2018
+#define CS40L2X_DSP1_AHBM_WIN3_CTRL1 0x025E201C
+#define CS40L2X_DSP1_AHBM_WIN4_CTRL0 0x025E2020
+#define CS40L2X_DSP1_AHBM_WIN4_CTRL1 0x025E2024
+#define CS40L2X_DSP1_AHBM_WIN5_CTRL0 0x025E2028
+#define CS40L2X_DSP1_AHBM_WIN5_CTRL1 0x025E202C
+#define CS40L2X_DSP1_AHBM_WIN6_CTRL0 0x025E2030
+#define CS40L2X_DSP1_AHBM_WIN6_CTRL1 0x025E2034
+#define CS40L2X_DSP1_AHBM_WIN7_CTRL0 0x025E2038
+#define CS40L2X_DSP1_AHBM_WIN7_CTRL1 0x025E203C
+#define CS40L2X_DSP1_AHBM_WIN_DBG_CTRL0 0x025E2040
+#define CS40L2X_DSP1_AHBM_WIN_DBG_CTRL1 0x025E2044
+#define CS40L2X_DSP1_XMEM_UNPACK24_0 0x02800000
+#define CS40L2X_DSP1_XMEM_UNPACK24_4093 0x02803FF4
+#define CS40L2X_DSP1_CTRL_BASE 0x02B80000
+#define CS40L2X_DSP1_CORE_SOFT_RESET 0x02B80010
+#define CS40L2X_DSP1_DEBUG 0x02B80040
+#define CS40L2X_DSP1_TIMER_CTRL 0x02B80048
+#define CS40L2X_DSP1_STREAM_ARB_CTRL 0x02B80050
+#define CS40L2X_DSP1_RX1_RATE 0x02B80080
+#define CS40L2X_DSP1_RX2_RATE 0x02B80088
+#define CS40L2X_DSP1_RX3_RATE 0x02B80090
+#define CS40L2X_DSP1_RX4_RATE 0x02B80098
+#define CS40L2X_DSP1_RX5_RATE 0x02B800A0
+#define CS40L2X_DSP1_RX6_RATE 0x02B800A8
+#define CS40L2X_DSP1_RX7_RATE 0x02B800B0
+#define CS40L2X_DSP1_RX8_RATE 0x02B800B8
+#define CS40L2X_DSP1_TX1_RATE 0x02B80280
+#define CS40L2X_DSP1_TX2_RATE 0x02B80288
+#define CS40L2X_DSP1_TX3_RATE 0x02B80290
+#define CS40L2X_DSP1_TX4_RATE 0x02B80298
+#define CS40L2X_DSP1_TX5_RATE 0x02B802A0
+#define CS40L2X_DSP1_TX6_RATE 0x02B802A8
+#define CS40L2X_DSP1_TX7_RATE 0x02B802B0
+#define CS40L2X_DSP1_TX8_RATE 0x02B802B8
+#define CS40L2X_DSP1_NMI_CTRL1 0x02B80480
+#define CS40L2X_DSP1_NMI_CTRL2 0x02B80488
+#define CS40L2X_DSP1_NMI_CTRL3 0x02B80490
+#define CS40L2X_DSP1_NMI_CTRL4 0x02B80498
+#define CS40L2X_DSP1_NMI_CTRL5 0x02B804A0
+#define CS40L2X_DSP1_NMI_CTRL6 0x02B804A8
+#define CS40L2X_DSP1_NMI_CTRL7 0x02B804B0
+#define CS40L2X_DSP1_NMI_CTRL8 0x02B804B8
+#define CS40L2X_DSP1_RESUME_CTRL 0x02B80500
+#define CS40L2X_DSP1_IRQ1_CTRL 0x02B80508
+#define CS40L2X_DSP1_IRQ2_CTRL 0x02B80510
+#define CS40L2X_DSP1_IRQ3_CTRL 0x02B80518
+#define CS40L2X_DSP1_IRQ4_CTRL 0x02B80520
+#define CS40L2X_DSP1_IRQ5_CTRL 0x02B80528
+#define CS40L2X_DSP1_IRQ6_CTRL 0x02B80530
+#define CS40L2X_DSP1_IRQ7_CTRL 0x02B80538
+#define CS40L2X_DSP1_IRQ8_CTRL 0x02B80540
+#define CS40L2X_DSP1_IRQ9_CTRL 0x02B80548
+#define CS40L2X_DSP1_IRQ10_CTRL 0x02B80550
+#define CS40L2X_DSP1_IRQ11_CTRL 0x02B80558
+#define CS40L2X_DSP1_IRQ12_CTRL 0x02B80560
+#define CS40L2X_DSP1_IRQ13_CTRL 0x02B80568
+#define CS40L2X_DSP1_IRQ14_CTRL 0x02B80570
+#define CS40L2X_DSP1_IRQ15_CTRL 0x02B80578
+#define CS40L2X_DSP1_IRQ16_CTRL 0x02B80580
+#define CS40L2X_DSP1_IRQ17_CTRL 0x02B80588
+#define CS40L2X_DSP1_IRQ18_CTRL 0x02B80590
+#define CS40L2X_DSP1_IRQ19_CTRL 0x02B80598
+#define CS40L2X_DSP1_IRQ20_CTRL 0x02B805A0
+#define CS40L2X_DSP1_IRQ21_CTRL 0x02B805A8
+#define CS40L2X_DSP1_IRQ22_CTRL 0x02B805B0
+#define CS40L2X_DSP1_IRQ23_CTRL 0x02B805B8
+#define CS40L2X_DSP1_SCRATCH1 0x02B805C0
+#define CS40L2X_DSP1_SCRATCH2 0x02B805C8
+#define CS40L2X_DSP1_SCRATCH3 0x02B805D0
+#define CS40L2X_DSP1_SCRATCH4 0x02B805D8
+#define CS40L2X_DSP1_CCM_CORE_CTRL 0x02BC1000
+#define CS40L2X_DSP1_CCM_CLK_OVERRIDE 0x02BC1008
+#define CS40L2X_DSP1_XM_MSTR_EN 0x02BC2000
+#define CS40L2X_DSP1_XM_CORE_PRI 0x02BC2008
+#define CS40L2X_DSP1_XM_AHB_PACK_PL_PRI 0x02BC2010
+#define CS40L2X_DSP1_XM_AHB_UP_PL_PRI 0x02BC2018
+#define CS40L2X_DSP1_XM_ACCEL_PL0_PRI 0x02BC2020
+#define CS40L2X_DSP1_XM_NPL0_PRI 0x02BC2078
+#define CS40L2X_DSP1_YM_MSTR_EN 0x02BC20C0
+#define CS40L2X_DSP1_YM_CORE_PRI 0x02BC20C8
+#define CS40L2X_DSP1_YM_AHB_PACK_PL_PRI 0x02BC20D0
+#define CS40L2X_DSP1_YM_AHB_UP_PL_PRI 0x02BC20D8
+#define CS40L2X_DSP1_YM_ACCEL_PL0_PRI 0x02BC20E0
+#define CS40L2X_DSP1_YM_NPL0_PRI 0x02BC2138
+#define CS40L2X_DSP1_PM_MSTR_EN 0x02BC2180
+#define CS40L2X_DSP1_PM_PATCH0_ADDR 0x02BC2188
+#define CS40L2X_DSP1_PM_PATCH0_EN 0x02BC218C
+#define CS40L2X_DSP1_PM_PATCH0_DATA_LO 0x02BC2190
+#define CS40L2X_DSP1_PM_PATCH0_DATA_HI 0x02BC2194
+#define CS40L2X_DSP1_PM_PATCH1_ADDR 0x02BC2198
+#define CS40L2X_DSP1_PM_PATCH1_EN 0x02BC219C
+#define CS40L2X_DSP1_PM_PATCH1_DATA_LO 0x02BC21A0
+#define CS40L2X_DSP1_PM_PATCH1_DATA_HI 0x02BC21A4
+#define CS40L2X_DSP1_PM_PATCH2_ADDR 0x02BC21A8
+#define CS40L2X_DSP1_PM_PATCH2_EN 0x02BC21AC
+#define CS40L2X_DSP1_PM_PATCH2_DATA_LO 0x02BC21B0
+#define CS40L2X_DSP1_PM_PATCH2_DATA_HI 0x02BC21B4
+#define CS40L2X_DSP1_PM_PATCH3_ADDR 0x02BC21B8
+#define CS40L2X_DSP1_PM_PATCH3_EN 0x02BC21BC
+#define CS40L2X_DSP1_PM_PATCH3_DATA_LO 0x02BC21C0
+#define CS40L2X_DSP1_PM_PATCH3_DATA_HI 0x02BC21C4
+#define CS40L2X_DSP1_PM_PATCH4_ADDR 0x02BC21C8
+#define CS40L2X_DSP1_PM_PATCH4_EN 0x02BC21CC
+#define CS40L2X_DSP1_PM_PATCH4_DATA_LO 0x02BC21D0
+#define CS40L2X_DSP1_PM_PATCH4_DATA_HI 0x02BC21D4
+#define CS40L2X_DSP1_PM_PATCH5_ADDR 0x02BC21D8
+#define CS40L2X_DSP1_PM_PATCH5_EN 0x02BC21DC
+#define CS40L2X_DSP1_PM_PATCH5_DATA_LO 0x02BC21E0
+#define CS40L2X_DSP1_PM_PATCH5_DATA_HI 0x02BC21E4
+#define CS40L2X_DSP1_PM_PATCH6_ADDR 0x02BC21E8
+#define CS40L2X_DSP1_PM_PATCH6_EN 0x02BC21EC
+#define CS40L2X_DSP1_PM_PATCH6_DATA_LO 0x02BC21F0
+#define CS40L2X_DSP1_PM_PATCH6_DATA_HI 0x02BC21F4
+#define CS40L2X_DSP1_PM_PATCH7_ADDR 0x02BC21F8
+#define CS40L2X_DSP1_PM_PATCH7_EN 0x02BC21FC
+#define CS40L2X_DSP1_PM_PATCH7_DATA_LO 0x02BC2200
+#define CS40L2X_DSP1_PM_PATCH7_DATA_HI 0x02BC2204
+#define CS40L2X_DSP1_MPU_XM_ACCESS0 0x02BC3000
+#define CS40L2X_DSP1_MPU_YM_ACCESS0 0x02BC3004
+#define CS40L2X_DSP1_MPU_WNDW_ACCESS0 0x02BC3008
+#define CS40L2X_DSP1_MPU_XREG_ACCESS0 0x02BC300C
+#define CS40L2X_DSP1_MPU_YREG_ACCESS0 0x02BC3014
+#define CS40L2X_DSP1_MPU_XM_ACCESS1 0x02BC3018
+#define CS40L2X_DSP1_MPU_YM_ACCESS1 0x02BC301C
+#define CS40L2X_DSP1_MPU_WNDW_ACCESS1 0x02BC3020
+#define CS40L2X_DSP1_MPU_XREG_ACCESS1 0x02BC3024
+#define CS40L2X_DSP1_MPU_YREG_ACCESS1 0x02BC302C
+#define CS40L2X_DSP1_MPU_XM_ACCESS2 0x02BC3030
+#define CS40L2X_DSP1_MPU_YM_ACCESS2 0x02BC3034
+#define CS40L2X_DSP1_MPU_WNDW_ACCESS2 0x02BC3038
+#define CS40L2X_DSP1_MPU_XREG_ACCESS2 0x02BC303C
+#define CS40L2X_DSP1_MPU_YREG_ACCESS2 0x02BC3044
+#define CS40L2X_DSP1_MPU_XM_ACCESS3 0x02BC3048
+#define CS40L2X_DSP1_MPU_YM_ACCESS3 0x02BC304C
+#define CS40L2X_DSP1_MPU_WNDW_ACCESS3 0x02BC3050
+#define CS40L2X_DSP1_MPU_XREG_ACCESS3 0x02BC3054
+#define CS40L2X_DSP1_MPU_YREG_ACCESS3 0x02BC305C
+#define CS40L2X_DSP1_MPU_XM_VIO_ADDR 0x02BC3100
+#define CS40L2X_DSP1_MPU_XM_VIO_STATUS 0x02BC3104
+#define CS40L2X_DSP1_MPU_YM_VIO_ADDR 0x02BC3108
+#define CS40L2X_DSP1_MPU_YM_VIO_STATUS 0x02BC310C
+#define CS40L2X_DSP1_MPU_PM_VIO_ADDR 0x02BC3110
+#define CS40L2X_DSP1_MPU_PM_VIO_STATUS 0x02BC3114
+#define CS40L2X_DSP1_MPU_LOCK_CONFIG 0x02BC3140
+#define CS40L2X_DSP1_MPU_WDT_RST_CTRL 0x02BC3180
+#define CS40L2X_DSP1_STRMARB_MSTR0_CFG0 0x02BC5000
+#define CS40L2X_DSP1_STRMARB_MSTR0_CFG1 0x02BC5004
+#define CS40L2X_DSP1_STRMARB_MSTR0_CFG2 0x02BC5008
+#define CS40L2X_DSP1_STRMARB_MSTR1_CFG0 0x02BC5010
+#define CS40L2X_DSP1_STRMARB_MSTR1_CFG1 0x02BC5014
+#define CS40L2X_DSP1_STRMARB_MSTR1_CFG2 0x02BC5018
+#define CS40L2X_DSP1_STRMARB_MSTR2_CFG0 0x02BC5020
+#define CS40L2X_DSP1_STRMARB_MSTR2_CFG1 0x02BC5024
+#define CS40L2X_DSP1_STRMARB_MSTR2_CFG2 0x02BC5028
+#define CS40L2X_DSP1_STRMARB_MSTR3_CFG0 0x02BC5030
+#define CS40L2X_DSP1_STRMARB_MSTR3_CFG1 0x02BC5034
+#define CS40L2X_DSP1_STRMARB_MSTR3_CFG2 0x02BC5038
+#define CS40L2X_DSP1_STRMARB_MSTR4_CFG0 0x02BC5040
+#define CS40L2X_DSP1_STRMARB_MSTR4_CFG1 0x02BC5044
+#define CS40L2X_DSP1_STRMARB_MSTR4_CFG2 0x02BC5048
+#define CS40L2X_DSP1_STRMARB_MSTR5_CFG0 0x02BC5050
+#define CS40L2X_DSP1_STRMARB_MSTR5_CFG1 0x02BC5054
+#define CS40L2X_DSP1_STRMARB_MSTR5_CFG2 0x02BC5058
+#define CS40L2X_DSP1_STRMARB_MSTR6_CFG0 0x02BC5060
+#define CS40L2X_DSP1_STRMARB_MSTR6_CFG1 0x02BC5064
+#define CS40L2X_DSP1_STRMARB_MSTR6_CFG2 0x02BC5068
+#define CS40L2X_DSP1_STRMARB_MSTR7_CFG0 0x02BC5070
+#define CS40L2X_DSP1_STRMARB_MSTR7_CFG1 0x02BC5074
+#define CS40L2X_DSP1_STRMARB_MSTR7_CFG2 0x02BC5078
+#define CS40L2X_DSP1_STRMARB_TX0_CFG0 0x02BC5200
+#define CS40L2X_DSP1_STRMARB_TX0_CFG1 0x02BC5204
+#define CS40L2X_DSP1_STRMARB_TX1_CFG0 0x02BC5208
+#define CS40L2X_DSP1_STRMARB_TX1_CFG1 0x02BC520C
+#define CS40L2X_DSP1_STRMARB_TX2_CFG0 0x02BC5210
+#define CS40L2X_DSP1_STRMARB_TX2_CFG1 0x02BC5214
+#define CS40L2X_DSP1_STRMARB_TX3_CFG0 0x02BC5218
+#define CS40L2X_DSP1_STRMARB_TX3_CFG1 0x02BC521C
+#define CS40L2X_DSP1_STRMARB_TX4_CFG0 0x02BC5220
+#define CS40L2X_DSP1_STRMARB_TX4_CFG1 0x02BC5224
+#define CS40L2X_DSP1_STRMARB_TX5_CFG0 0x02BC5228
+#define CS40L2X_DSP1_STRMARB_TX5_CFG1 0x02BC522C
+#define CS40L2X_DSP1_STRMARB_TX6_CFG0 0x02BC5230
+#define CS40L2X_DSP1_STRMARB_TX6_CFG1 0x02BC5234
+#define CS40L2X_DSP1_STRMARB_TX7_CFG0 0x02BC5238
+#define CS40L2X_DSP1_STRMARB_TX7_CFG1 0x02BC523C
+#define CS40L2X_DSP1_STRMARB_RX0_CFG0 0x02BC5400
+#define CS40L2X_DSP1_STRMARB_RX0_CFG1 0x02BC5404
+#define CS40L2X_DSP1_STRMARB_RX1_CFG0 0x02BC5408
+#define CS40L2X_DSP1_STRMARB_RX1_CFG1 0x02BC540C
+#define CS40L2X_DSP1_STRMARB_RX2_CFG0 0x02BC5410
+#define CS40L2X_DSP1_STRMARB_RX2_CFG1 0x02BC5414
+#define CS40L2X_DSP1_STRMARB_RX3_CFG0 0x02BC5418
+#define CS40L2X_DSP1_STRMARB_RX3_CFG1 0x02BC541C
+#define CS40L2X_DSP1_STRMARB_RX4_CFG0 0x02BC5420
+#define CS40L2X_DSP1_STRMARB_RX4_CFG1 0x02BC5424
+#define CS40L2X_DSP1_STRMARB_RX5_CFG0 0x02BC5428
+#define CS40L2X_DSP1_STRMARB_RX5_CFG1 0x02BC542C
+#define CS40L2X_DSP1_STRMARB_RX6_CFG0 0x02BC5430
+#define CS40L2X_DSP1_STRMARB_RX6_CFG1 0x02BC5434
+#define CS40L2X_DSP1_STRMARB_RX7_CFG0 0x02BC5438
+#define CS40L2X_DSP1_STRMARB_RX7_CFG1 0x02BC543C
+#define CS40L2X_DSP1_STRMARB_IRQ0_CFG0 0x02BC5600
+#define CS40L2X_DSP1_STRMARB_IRQ0_CFG1 0x02BC5604
+#define CS40L2X_DSP1_STRMARB_IRQ0_CFG2 0x02BC5608
+#define CS40L2X_DSP1_STRMARB_IRQ1_CFG0 0x02BC5610
+#define CS40L2X_DSP1_STRMARB_IRQ1_CFG1 0x02BC5614
+#define CS40L2X_DSP1_STRMARB_IRQ1_CFG2 0x02BC5618
+#define CS40L2X_DSP1_STRMARB_IRQ2_CFG0 0x02BC5620
+#define CS40L2X_DSP1_STRMARB_IRQ2_CFG1 0x02BC5624
+#define CS40L2X_DSP1_STRMARB_IRQ2_CFG2 0x02BC5628
+#define CS40L2X_DSP1_STRMARB_IRQ3_CFG0 0x02BC5630
+#define CS40L2X_DSP1_STRMARB_IRQ3_CFG1 0x02BC5634
+#define CS40L2X_DSP1_STRMARB_IRQ3_CFG2 0x02BC5638
+#define CS40L2X_DSP1_STRMARB_IRQ4_CFG0 0x02BC5640
+#define CS40L2X_DSP1_STRMARB_IRQ4_CFG1 0x02BC5644
+#define CS40L2X_DSP1_STRMARB_IRQ4_CFG2 0x02BC5648
+#define CS40L2X_DSP1_STRMARB_IRQ5_CFG0 0x02BC5650
+#define CS40L2X_DSP1_STRMARB_IRQ5_CFG1 0x02BC5654
+#define CS40L2X_DSP1_STRMARB_IRQ5_CFG2 0x02BC5658
+#define CS40L2X_DSP1_STRMARB_IRQ6_CFG0 0x02BC5660
+#define CS40L2X_DSP1_STRMARB_IRQ6_CFG1 0x02BC5664
+#define CS40L2X_DSP1_STRMARB_IRQ6_CFG2 0x02BC5668
+#define CS40L2X_DSP1_STRMARB_IRQ7_CFG0 0x02BC5670
+#define CS40L2X_DSP1_STRMARB_IRQ7_CFG1 0x02BC5674
+#define CS40L2X_DSP1_STRMARB_IRQ7_CFG2 0x02BC5678
+#define CS40L2X_DSP1_STRMARB_RESYNC_MSK 0x02BC5A00
+#define CS40L2X_DSP1_STRMARB_ERR_STATUS 0x02BC5A08
+#define CS40L2X_DSP1_INTPCTL_RES_STATIC 0x02BC6000
+#define CS40L2X_DSP1_INTPCTL_RES_DYN 0x02BC6004
+#define CS40L2X_DSP1_INTPCTL_NMI_CTRL 0x02BC6008
+#define CS40L2X_DSP1_INTPCTL_IRQ_INV 0x02BC6010
+#define CS40L2X_DSP1_INTPCTL_IRQ_MODE 0x02BC6014
+#define CS40L2X_DSP1_INTPCTL_IRQ_EN 0x02BC6018
+#define CS40L2X_DSP1_INTPCTL_IRQ_MSK 0x02BC601C
+#define CS40L2X_DSP1_INTPCTL_IRQ_FLUSH 0x02BC6020
+#define CS40L2X_DSP1_INTPCTL_IRQ_MSKCLR 0x02BC6024
+#define CS40L2X_DSP1_INTPCTL_IRQ_FRC 0x02BC6028
+#define CS40L2X_DSP1_INTPCTL_IRQ_MSKSET 0x02BC602C
+#define CS40L2X_DSP1_INTPCTL_IRQ_ERR 0x02BC6030
+#define CS40L2X_DSP1_INTPCTL_IRQ_PEND 0x02BC6034
+#define CS40L2X_DSP1_INTPCTL_IRQ_GEN 0x02BC6038
+#define CS40L2X_DSP1_INTPCTL_TESTBITS 0x02BC6040
+#define CS40L2X_DSP1_WDT_CONTROL 0x02BC7000
+#define CS40L2X_DSP1_WDT_STATUS 0x02BC7008
+#define CS40L2X_DSP1_YMEM_PACK_0 0x02C00000
+#define CS40L2X_DSP1_YMEM_PACK_1532 0x02C017F0
+#define CS40L2X_DSP1_YMEM_UNPACK32_0 0x03000000
+#define CS40L2X_DSP1_YMEM_UNPACK32_1022 0x03000FF8
+#define CS40L2X_DSP1_YMEM_UNPACK24_0 0x03400000
+#define CS40L2X_DSP1_YMEM_UNPACK24_2045 0x03401FF4
+#define CS40L2X_DSP1_PMEM_0 0x03800000
+#define CS40L2X_DSP1_PMEM_5114 0x03804FE8
+
+/* trim fields unpacked from OTP memory */
+#define CS40L2X_OTP_TRIM_1 0x0000208C
+#define CS40L2X_OTP_TRIM_2 0x00002090
+#define CS40L2X_OTP_TRIM_3 0x00003010
+#define CS40L2X_OTP_TRIM_4 0x0000300C
+#define CS40L2X_OTP_TRIM_5 0x0000394C
+#define CS40L2X_OTP_TRIM_6 0x00003950
+#define CS40L2X_OTP_TRIM_7 0x00003954
+#define CS40L2X_OTP_TRIM_8 0x00003958
+#define CS40L2X_OTP_TRIM_9 0x0000395C
+#define CS40L2X_OTP_TRIM_10 0x0000416C
+#define CS40L2X_OTP_TRIM_11 0x00004160
+#define CS40L2X_OTP_TRIM_12 0x00004170
+#define CS40L2X_OTP_TRIM_13 0x00004360
+#define CS40L2X_OTP_TRIM_14 0x00004448
+#define CS40L2X_OTP_TRIM_15 0x0000444C
+#define CS40L2X_OTP_TRIM_16 0x00006E30
+#define CS40L2X_OTP_TRIM_17 0x00006E34
+#define CS40L2X_OTP_TRIM_18 0x00006E38
+#define CS40L2X_OTP_TRIM_19 0x00006E3C
+#define CS40L2X_OTP_TRIM_20 0x00006E40
+#define CS40L2X_OTP_TRIM_21 0x00006E44
+#define CS40L2X_OTP_TRIM_22 0x00006E48
+#define CS40L2X_OTP_TRIM_23 0x00006E4C
+#define CS40L2X_OTP_TRIM_24 0x00006E50
+#define CS40L2X_OTP_TRIM_25 0x00006E54
+#define CS40L2X_OTP_TRIM_26 0x00006E58
+#define CS40L2X_OTP_TRIM_27 0x00006E5C
+#define CS40L2X_OTP_TRIM_28 0x00006E60
+#define CS40L2X_OTP_TRIM_29 0x00006E64
+#define CS40L2X_OTP_TRIM_30 0x00007418
+#define CS40L2X_OTP_TRIM_31 0x0000741C
+#define CS40L2X_OTP_TRIM_32 0x00007434
+#define CS40L2X_OTP_TRIM_33 0x00007068
+#define CS40L2X_OTP_TRIM_34 0x0000410C
+#define CS40L2X_OTP_TRIM_35 0x0000400C
+#define CS40L2X_OTP_TRIM_36 0x00002030
+
+/* basic mode (ROM) fields */
+#define CS40L2X_BASIC_HALO_STATE 0x02800154
+#define CS40L2X_BASIC_HALO_HEARTBEAT 0x02800158
+#define CS40L2X_BASIC_STATEMACHINE 0X0280015C
+#define CS40L2X_BASIC_F0_STORED 0X02800174
+#define CS40L2X_BASIC_GPIO_TRIGGER_EDGE 0x02800178
+#define CS40L2X_BASIC_BUZZ_FREQ 0x0280017C
+#define CS40L2X_BASIC_BUZZ_LEVEL 0x02800180
+#define CS40L2X_BASIC_BUZZ_DURATION 0x02800184
+#define CS40L2X_BASIC_BUZZ_TRIGGER 0x02800188
+#define CS40L2X_BASIC_AMP_STATUS 0x0280018C
+#define CS40L2X_BASIC_SHUTDOWNREQUEST 0x02800190
+
+/* basic mode (ROM) states */
+#define CS40L2X_BASIC_IDLE 0
+#define CS40L2X_BASIC_PLAY_TONE 1
+#define CS40L2X_BASIC_CONTINUE_TONE 2
+#define CS40L2X_BASIC_SHUTTINGDOWN 3
+#define CS40L2X_BASIC_SHUTDOWN 4
+
+/* basic mode (ROM) status flags */
+#define CS40L2X_BASIC_NO_STATUS 0x00
+#define CS40L2X_BASIC_BOOT_DONE 0x01
+#define CS40L2X_BASIC_OTP_ERROR 0x02
+#define CS40L2X_BASIC_AMP_ERROR 0x04
+#define CS40L2X_BASIC_TEMP_RISE_WARN 0x08
+#define CS40L2X_BASIC_TEMP_ERROR 0x10
+#define CS40L2X_BASIC_SA_ERROR 0x20
+
+#define CS40L2X_GLOBAL_EN_MASK 0x00000001
+#define CS40L2X_GLOBAL_EN_SHIFT 0
+
+#define CS40L2X_GP2_CTRL_MASK 0x07000000
+#define CS40L2X_GP2_CTRL_SHIFT 24
+#define CS40L2X_GP2_CTRL_MCLK 0x3
+
+#define CS40L2X_MEM_RDY_MASK 0x00000002
+#define CS40L2X_MEM_RDY_SHIFT 1
+
+#define CS40L2X_PLL_FORCE_EN_MASK 0x00010000
+#define CS40L2X_PLL_FORCE_EN_SHIFT 16
+
+#define CS40L2X_PLL_REFCLK_SEL_MASK 0x00000007
+#define CS40L2X_PLL_REFCLK_SEL_SHIFT 0
+#define CS40L2X_PLL_REFCLK_SEL_MCLK 0x5
+
+#define CS40L2X_DAC_PCM1_SRC_MASK 0x7F
+#define CS40L2X_DAC_PCM1_SRC_SHIFT 0
+#define CS40L2X_DAC_PCM1_SRC_DSP1TX1 0x32
+
+#define CS40L2X_DSP1_RXn_SRC_MASK 0x7F
+#define CS40L2X_DSP1_RXn_SRC_SHIFT 0
+#define CS40L2X_DSP1_RXn_SRC_VMON 0x18
+#define CS40L2X_DSP1_RXn_SRC_IMON 0x19
+#define CS40L2X_DSP1_RXn_SRC_VPMON 0x28
+
+#define CS40L2X_DSP1_RESET_MASK 0x200
+#define CS40L2X_DSP1_RESET_SHIFT 9
+
+#define CS40L2X_DSP1_EN_MASK 0x1
+#define CS40L2X_DSP1_EN_SHIFT 0
+
+#define CS40L2X_BST_IPK_MASK 0x0000007F
+#define CS40L2X_BST_IPK_SHIFT 0
+
+#define CS40L2X_BST_K1_MASK 0x000000FF
+#define CS40L2X_BST_K1_SHIFT 0
+#define CS40L2X_BST_K2_MASK 0x0000FF00
+#define CS40L2X_BST_K2_SHIFT 8
+
+#define CS40L2X_BST_SLOPE_MASK 0x0000FF00
+#define CS40L2X_BST_SLOPE_SHIFT 8
+
+#define CS40L2X_BST_LBST_VAL_MASK 0x00000003
+#define CS40L2X_BST_LBST_VAL_SHIFT 0
+
+#define CS40L2X_AMP_VOL_PCM_MASK 0x00003FF8
+#define CS40L2X_AMP_VOL_PCM_SHIFT 3
+
+#define CS40L2X_GAIN_CTRL_GPIO_MASK 0x00FFC000
+#define CS40L2X_GAIN_CTRL_GPIO_SHIFT 14
+
+#define CS40L2X_GAIN_CTRL_TRIG_MASK 0x00003FF0
+#define CS40L2X_GAIN_CTRL_TRIG_SHIFT 4
+
+#define CS40L2X_OTP_BOOT_ERR 0x80000000
+#define CS40L2X_OTP_BOOT_DONE 0x00000002
+
+#define CS40L2X_DEVID_L20 0x035A40
+#define CS40L2X_DEVID_L25 0x40A250
+#define CS40L2X_DEVID_L25A 0x40A25A
+#define CS40L2X_DEVID_L25B 0x40A25B
+
+#define CS40L2X_REVID_A0 0xA0
+#define CS40L2X_REVID_B0 0xB0
+#define CS40L2X_REVID_B1 0xB1
+
+#define CS40L2X_MPU_UNLOCK_CODE1 0x5555
+#define CS40L2X_MPU_UNLOCK_CODE2 0xAAAA
+
+#define CS40L2X_XM_UNPACKED_TYPE 0x05
+#define CS40L2X_YM_UNPACKED_TYPE 0x06
+#define CS40L2X_PM_PACKED_TYPE 0x10
+#define CS40L2X_XM_PACKED_TYPE 0x11
+#define CS40L2X_YM_PACKED_TYPE 0x12
+#define CS40L2X_ALGO_INFO_TYPE 0xF2
+
+#define CS40L2X_FW_FILE_HEADER_SIZE 40
+#define CS40L2X_FW_DBLK_OFFSET_SIZE 3
+#define CS40L2X_FW_DBLK_TYPE_SIZE 1
+#define CS40L2X_FW_DBLK_LENGTH_SIZE 4
+
+#define CS40L2X_WT_FILE_HEADER_SIZE 16
+#define CS40L2X_WT_DBLK_OFFSET_SIZE 2
+#define CS40L2X_WT_DBLK_TYPE_SIZE 2
+#define CS40L2X_WT_ALGO_ID_SIZE 4
+#define CS40L2X_WT_ALGO_REV_SIZE 4
+#define CS40L2X_WT_SAMPLE_RATE_SIZE 4
+#define CS40L2X_WT_DBLK_LENGTH_SIZE 4
+
+#define CS40L2X_XM_FW_ID (CS40L2X_DSP1_XMEM_UNPACK24_0 + 12)
+#define CS40L2X_XM_NUM_ALGOS (CS40L2X_DSP1_XMEM_UNPACK24_0 + 36)
+#define CS40L2X_ALGO_ID_OFFSET 0
+#define CS40L2X_ALGO_REV_OFFSET 4
+#define CS40L2X_ALGO_XM_BASE_OFFSET 8
+#define CS40L2X_ALGO_XM_SIZE_OFFSET 12
+#define CS40L2X_ALGO_YM_BASE_OFFSET 16
+#define CS40L2X_ALGO_YM_SIZE_OFFSET 20
+#define CS40L2X_ALGO_ENTRY_SIZE 24
+#define CS40L2X_ALGO_LIST_TERM 0xBEDEAD
+#define CS40L2X_ALGO_REV_MASK 0xFF0000
+
+#define CS40L2X_NUM_ALGOS_MAX 16
+#define CS40L2X_FW_REV_MIN 0x040200
+
+#define CS40L2X_ALGO_ID_SIZE 4
+#define CS40L2X_COEFF_COUNT_SIZE 4
+#define CS40L2X_COEFF_OFFSET_SIZE 2
+#define CS40L2X_COEFF_TYPE_SIZE 2
+#define CS40L2X_COEFF_LENGTH_SIZE 4
+
+#define CS40L2X_TIMEOUT_MS_MAX 0x02AAAA /* ~174 sec */
+#define CS40L2X_PR_TIMEOUT_MAX 0x7FFFFF
+#define CS40L2X_EVENT_TIMEOUT_MAX CS40L2X_PR_TIMEOUT_MAX
+
+#define CS40L2X_HALO_STATE_RUNNING 0x2
+#define CS40L2X_STATUS_IDLE 0xFFFFFF
+
+#define CS40L2X_MBOX_TRIGGERINDEX CS40L2X_DSP_VIRT1_MBOX_1
+#define CS40L2X_MBOX_TRIGGER_MS CS40L2X_DSP_VIRT1_MBOX_2
+#define CS40L2X_MBOX_STIMULUS_MODE CS40L2X_DSP_VIRT1_MBOX_3
+
+#define CS40L2X_INDEX_MASK 0x7FFF
+#define CS40L2X_INDEX_VIBE 0x0000
+#define CS40L2X_INDEX_CLICK_MIN 0x0001
+#define CS40L2X_INDEX_CLICK_MAX 0x7FFF
+#define CS40L2X_INDEX_CONT_MIN 0x8000
+#define CS40L2X_INDEX_CONT_MAX 0xFFFD
+#define CS40L2X_INDEX_PBQ 0xFFFE
+#define CS40L2X_INDEX_DIAG 0xFFFF
+
+#define CS40L2X_PBQ_SEG_LEN_MAX 10
+#define CS40L2X_PBQ_DEPTH_MAX 256
+#define CS40L2X_PBQ_SCALE_MAX 100
+#define CS40L2X_PBQ_DELAY_MAX 10000
+#define CS40L2X_PBQ_REPEAT_MAX 16
+#define CS40L2X_PBQ_POLL_NS 5000000
+#define CS40L2X_PBQ_STATE_IDLE 0x00
+#define CS40L2X_PBQ_STATE_PLAYING 0x01
+#define CS40L2X_PBQ_STATE_SILENT 0x02
+#define CS40L2X_PBQ_TAG_SILENCE 0x0000
+#define CS40L2X_PBQ_TAG_START 0x8000
+#define CS40L2X_PBQ_TAG_STOP 0x8001
+
+#define CS40L2X_DIAG_STATE_INIT 0x00
+#define CS40L2X_DIAG_STATE_RUN 0x01
+#define CS40L2X_DIAG_STATE_DONE 0x02
+#define CS40L2X_DIAG_STATE_DELAY_MS 500
+
+#define CS40L2X_DSP_TIMEOUT_COUNT 10 /* 1 count = 10 ms */
+#define CS40L2X_OTP_TIMEOUT_COUNT 10
+#define CS40L2X_BASIC_TIMEOUT_COUNT 10
+
+#define CS40L2X_NUM_OTP_MAPS 3
+#define CS40L2X_NUM_OTP_WORDS 32
+
+#define CS40L2X_TEST_KEY_UNLOCK_CODE1 0x55
+#define CS40L2X_TEST_KEY_UNLOCK_CODE2 0xAA
+#define CS40L2X_TEST_KEY_RELOCK_CODE1 0xCC
+#define CS40L2X_TEST_KEY_RELOCK_CODE2 0x33
+
+#define CS40L2X_DIG_SCALE_MAX 816 /* -102 dB */
+#define CS40L2X_DIG_SCALE_ZERO 0x800
+#define CS40L2X_DIG_SCALE_MASK 0x7FF
+#define CS40L2X_DIG_SCALE_RESET 0x80000000
+
+#define CS40L2X_MAX_WLEN 4096
+
+#define CS40L2X_DEVICE_NAME "vibrator"
+
+#define CS40L2X_FW_NAME_A0 "cs40l20.wmfw"
+#define CS40L2X_FW_NAME_B1 "cs40l25a.wmfw"
+
+#define CS40L2X_WT_NAME "cs40l20.bin"
+
+#define CS40L2X_PDATA_PRESENT 0x80000000
+#define CS40L2X_PDATA_MASK ~CS40L2X_PDATA_PRESENT
+
+#define CS40L2X_GPIO1_MODE_DEF_ON 0
+#define CS40L2X_GPIO1_MODE_DEF_OFF 1
+#define CS40L2X_GPIO1_MODE_AUTO 2
+#define CS40L2X_GPIO1_MODE_MAX CS40L2X_GPIO1_MODE_AUTO
+
+#define CS40L2X_GPIO1_ENABLED 1
+#define CS40L2X_GPIO1_DISABLED 0
+
+#define CS40L2X_COMP_ENABLED 1
+#define CS40L2X_COMP_DISABLED 0
+
+#define CS40L2X_ENDPLAYBACK_RETRIES 10
+#define CS40L2X_ENDPLAYBACK_REQ 1
+#define CS40L2X_ENDPLAYBACK_ACK 0
+
+bool cs40l2x_readable_reg(struct device *dev, unsigned int reg);
+bool cs40l2x_precious_reg(struct device *dev, unsigned int reg);
+
+struct cs40l2x_trim {
+ unsigned int reg;
+ unsigned char shift;
+ unsigned char size;
+};
+
+struct cs40l2x_otp_desc {
+ unsigned int id;
+ unsigned char row_start;
+ unsigned char col_start;
+ unsigned int num_trims;
+ const struct cs40l2x_trim *trim_table;
+};
+
+struct cs40l2x_coeff_desc {
+ unsigned int parent_id;
+ unsigned int block_offset;
+ unsigned int block_type;
+ unsigned char name[32];
+ unsigned int reg;
+ struct list_head list;
+};
+
+struct cs40l2x_pbq_pair {
+ unsigned int tag;
+ unsigned int mag;
+ unsigned int repeat;
+ unsigned int remain;
+};
+
+extern const unsigned char cs40l2x_bst_k1_table[4][5];
+extern const unsigned char cs40l2x_bst_k2_table[4][5];
+extern const unsigned char cs40l2x_bst_slope_table[4];
+
+extern const struct cs40l2x_otp_desc cs40l2x_otp_map[CS40L2X_NUM_OTP_MAPS];
+
+extern const unsigned int cs40l2x_pbq_dig_scale[CS40L2X_PBQ_SCALE_MAX + 1];
+
+#endif /*__CS40L2X_H__*/
diff --git a/drivers/misc/cxl/main.c b/drivers/misc/cxl/main.c
index cc1706a..a078d49 100644
--- a/drivers/misc/cxl/main.c
+++ b/drivers/misc/cxl/main.c
@@ -293,7 +293,7 @@ int cxl_adapter_context_get(struct cxl *adapter)
int rc;
rc = atomic_inc_unless_negative(&adapter->contexts_num);
- return rc >= 0 ? 0 : -EBUSY;
+ return rc ? 0 : -EBUSY;
}
void cxl_adapter_context_put(struct cxl *adapter)
diff --git a/drivers/misc/genwqe/card_base.h b/drivers/misc/genwqe/card_base.h
index cb851c1..159f35b 100644
--- a/drivers/misc/genwqe/card_base.h
+++ b/drivers/misc/genwqe/card_base.h
@@ -404,7 +404,7 @@ struct genwqe_file {
struct file *filp;
struct fasync_struct *async_queue;
- struct task_struct *owner;
+ struct pid *opener;
struct list_head list; /* entry in list of open files */
spinlock_t map_lock; /* lock for dma_mappings */
diff --git a/drivers/misc/genwqe/card_dev.c b/drivers/misc/genwqe/card_dev.c
index 7f1b282..c0012ca 100644
--- a/drivers/misc/genwqe/card_dev.c
+++ b/drivers/misc/genwqe/card_dev.c
@@ -52,7 +52,7 @@ static void genwqe_add_file(struct genwqe_dev *cd, struct genwqe_file *cfile)
{
unsigned long flags;
- cfile->owner = current;
+ cfile->opener = get_pid(task_tgid(current));
spin_lock_irqsave(&cd->file_lock, flags);
list_add(&cfile->list, &cd->file_list);
spin_unlock_irqrestore(&cd->file_lock, flags);
@@ -65,6 +65,7 @@ static int genwqe_del_file(struct genwqe_dev *cd, struct genwqe_file *cfile)
spin_lock_irqsave(&cd->file_lock, flags);
list_del(&cfile->list);
spin_unlock_irqrestore(&cd->file_lock, flags);
+ put_pid(cfile->opener);
return 0;
}
@@ -275,7 +276,7 @@ static int genwqe_kill_fasync(struct genwqe_dev *cd, int sig)
return files;
}
-static int genwqe_force_sig(struct genwqe_dev *cd, int sig)
+static int genwqe_terminate(struct genwqe_dev *cd)
{
unsigned int files = 0;
unsigned long flags;
@@ -283,7 +284,7 @@ static int genwqe_force_sig(struct genwqe_dev *cd, int sig)
spin_lock_irqsave(&cd->file_lock, flags);
list_for_each_entry(cfile, &cd->file_list, list) {
- force_sig(sig, cfile->owner);
+ kill_pid(cfile->opener, SIGKILL, 1);
files++;
}
spin_unlock_irqrestore(&cd->file_lock, flags);
@@ -1356,7 +1357,7 @@ static int genwqe_inform_and_stop_processes(struct genwqe_dev *cd)
dev_warn(&pci_dev->dev,
"[%s] send SIGKILL and wait ...\n", __func__);
- rc = genwqe_force_sig(cd, SIGKILL); /* force terminate */
+ rc = genwqe_terminate(cd);
if (rc) {
/* Give kill_timout more seconds to end processes */
for (i = 0; (i < genwqe_kill_timeout) &&
diff --git a/drivers/misc/genwqe/card_utils.c b/drivers/misc/genwqe/card_utils.c
index fc2794b..466a9b7 100644
--- a/drivers/misc/genwqe/card_utils.c
+++ b/drivers/misc/genwqe/card_utils.c
@@ -217,7 +217,7 @@ u32 genwqe_crc32(u8 *buff, size_t len, u32 init)
void *__genwqe_alloc_consistent(struct genwqe_dev *cd, size_t size,
dma_addr_t *dma_handle)
{
- if (get_order(size) > MAX_ORDER)
+ if (get_order(size) >= MAX_ORDER)
return NULL;
return dma_zalloc_coherent(&cd->pci_dev->dev, size, dma_handle,
diff --git a/drivers/misc/google-easel-comm-dma.c b/drivers/misc/google-easel-comm-dma.c
new file mode 100644
index 0000000..35d81e2
--- /dev/null
+++ b/drivers/misc/google-easel-comm-dma.c
@@ -0,0 +1,751 @@
+/*
+ * Android/Easel coprocessor communication DMA routines.
+ *
+ * Copyright 2016 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+/* #define DEBUG */
+
+#include <uapi/linux/google-easel-comm.h>
+#include "google-easel-comm-shared.h"
+#include "google-easel-comm-private.h"
+
+#include <linux/uaccess.h>
+#include <linux/completion.h>
+#include <linux/device.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/miscdevice.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+
+/*
+ * Server receives DMA scatter-gather list from client. Store this and
+ * wake up the local process handling the DMA request.
+ */
+void easelcomm_handle_cmd_dma_sg(
+ struct easelcomm_service *service, char *command_args,
+ size_t command_arg_len)
+{
+ struct easelcomm_dma_sg_header *sg_header;
+ void *cmd_sg;
+ struct easelcomm_message_metadata *msg_metadata;
+
+ if (WARN_ON(command_arg_len < sizeof(struct easelcomm_dma_sg_header)))
+ return;
+
+ sg_header = (struct easelcomm_dma_sg_header *) command_args;
+ command_args += sizeof(struct easelcomm_dma_sg_header);
+ command_arg_len -= sizeof(struct easelcomm_dma_sg_header);
+
+ if (WARN_ON(command_arg_len < sg_header->scatterlist_size))
+ return;
+
+ if (sg_header->dma_dir == EASELCOMM_DMA_DIR_TO_CLIENT) {
+ msg_metadata =
+ easelcomm_find_local_message(
+ service, sg_header->message_id);
+ } else {
+ msg_metadata =
+ easelcomm_find_remote_message(
+ service, sg_header->message_id);
+ }
+
+ if (!msg_metadata) {
+ dev_err(easelcomm_miscdev.this_device,
+ "DMA_SG msg %u:%s%llu not found\n",
+ service->service_id,
+ sg_header->dma_dir == EASELCOMM_DMA_DIR_TO_CLIENT ?
+ "l" : "r", sg_header->message_id);
+ return;
+ }
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "recv cmd DMA_SG msg %u:%s%llu size=%u\n",
+ service->service_id,
+ easelcomm_msgid_prefix(msg_metadata), sg_header->message_id,
+ sg_header->scatterlist_size);
+
+ if (WARN_ON(msg_metadata->dma_xfer.sg_remote))
+ goto out;
+
+ msg_metadata->dma_xfer.sg_remote_size = sg_header->scatterlist_size;
+
+ if (sg_header->scatterlist_size) {
+ cmd_sg = command_args;
+ msg_metadata->dma_xfer.sg_remote =
+ kmalloc(sg_header->scatterlist_size, GFP_KERNEL);
+ if (WARN_ON(!msg_metadata->dma_xfer.sg_remote)) {
+ msg_metadata->dma_xfer.sg_remote_size = 0;
+ goto out;
+ }
+
+ memcpy(msg_metadata->dma_xfer.sg_remote, cmd_sg,
+ sg_header->scatterlist_size);
+ }
+
+ /* Let the local waiter know the remote scatterlist ready. */
+ complete(&msg_metadata->dma_xfer.sg_remote_ready);
+
+out:
+ easelcomm_drop_reference(service, msg_metadata, false);
+}
+EXPORT_SYMBOL(easelcomm_handle_cmd_dma_sg);
+
+/*
+ * Client receives DMA transfer instructions from server. Server has chosen
+ * single- vs. multi-block as necessary, and has provided either the single
+ * block server-side source or destination address, or the multi-block
+ * linked-list address. Wake up the local process to perform the DMA
+ * transfer.
+ */
+void easelcomm_handle_cmd_dma_xfer(
+ struct easelcomm_service *service, char *command_args,
+ size_t command_arg_len)
+{
+ struct easelcomm_dma_xfer_arg *dma_xfer;
+ struct easelcomm_message_metadata *msg_metadata;
+
+ if (WARN_ON(command_arg_len <
+ sizeof(struct easelcomm_dma_xfer_arg)))
+ return;
+
+ dma_xfer = (struct easelcomm_dma_xfer_arg *) command_args;
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "recv cmd DMA_XFER msg %u:%s%llu type=%u saddr=%llx\n",
+ service->service_id,
+ dma_xfer->dma_dir == EASELCOMM_DMA_DIR_TO_SERVER ? "l" : "r",
+ dma_xfer->message_id, dma_xfer->xfer_type,
+ dma_xfer->server_addr);
+
+ if (dma_xfer->dma_dir == EASELCOMM_DMA_DIR_TO_SERVER) {
+ msg_metadata =
+ easelcomm_find_local_message(
+ service, dma_xfer->message_id);
+ } else {
+ msg_metadata =
+ easelcomm_find_remote_message(
+ service, dma_xfer->message_id);
+ }
+
+ if (!msg_metadata) {
+ dev_err(easelcomm_miscdev.this_device,
+ "DMA_XFER msg %u:%s%llu not found\n",
+ service->service_id,
+ dma_xfer->dma_dir == EASELCOMM_DMA_DIR_TO_SERVER ?
+ "l" : "r", dma_xfer->message_id);
+ return;
+ }
+
+ msg_metadata->dma_xfer.xfer_type = dma_xfer->xfer_type;
+ msg_metadata->dma_xfer.server_addr = dma_xfer->server_addr;
+
+ /* Let the local waiter know the DMA request is received */
+ complete(&msg_metadata->dma_xfer.xfer_ready);
+ easelcomm_drop_reference(service, msg_metadata, false);
+}
+EXPORT_SYMBOL(easelcomm_handle_cmd_dma_xfer);
+
+/*
+ * Server receives DMA done indication from client.
+ * Safe to unmap buffers and return to waiting DMA originator or receiver.
+ */
+void easelcomm_handle_cmd_dma_done(
+ struct easelcomm_service *service, char *command_args,
+ size_t command_arg_len)
+{
+ struct easelcomm_dma_done_arg *dma_done_arg;
+ struct easelcomm_message_metadata *msg_metadata;
+
+ if (WARN_ON(command_arg_len < sizeof(struct easelcomm_dma_done_arg)))
+ return;
+
+ dma_done_arg = (struct easelcomm_dma_done_arg *) command_args;
+ dev_dbg(easelcomm_miscdev.this_device,
+ "recv cmd DMA_DONE msg %u:%s%llu err=%u\n",
+ service->service_id,
+ dma_done_arg->dma_dir == EASELCOMM_DMA_DIR_TO_CLIENT ?
+ "l" : "r", dma_done_arg->message_id, dma_done_arg->errcode);
+
+ if (dma_done_arg->dma_dir == EASELCOMM_DMA_DIR_TO_CLIENT) {
+ msg_metadata =
+ easelcomm_find_local_message(
+ service, dma_done_arg->message_id);
+ } else {
+ msg_metadata =
+ easelcomm_find_remote_message(
+ service, dma_done_arg->message_id);
+ }
+
+ if (!msg_metadata) {
+ dev_err(easelcomm_miscdev.this_device,
+ "CMD_DMA_DONE msg %u:%s%llu not found\n",
+ service->service_id,
+ dma_done_arg->dma_dir == EASELCOMM_DMA_DIR_TO_CLIENT ?
+ "l" : "r", dma_done_arg->message_id);
+ return;
+ }
+
+ msg_metadata->dma_xfer.errcode = dma_done_arg->errcode;
+ /*
+ * Wakeup local waiter waiting on remote SG (if this is an abort) or
+ * DMA completion.
+ */
+ complete(&msg_metadata->dma_xfer.sg_remote_ready);
+ complete(&msg_metadata->dma_xfer.xfer_done);
+ easelcomm_drop_reference(service, msg_metadata, false);
+}
+EXPORT_SYMBOL(easelcomm_handle_cmd_dma_done);
+
+static void *easelcomm_create_dma_scatterlist(
+ struct easelcomm_kbuf_desc *buf_desc, uint32_t *scatterlist_size,
+ void **sglocaldata, enum easelcomm_dma_direction dma_dir)
+{
+ return easelcomm_hw_build_scatterlist(buf_desc, scatterlist_size,
+ sglocaldata, dma_dir);
+}
+
+/*
+ * Client sends its local DMA dest scatter-gather list to the server, which
+ * will use this info to choose a single- or multi-block transfer, and build
+ * an MNH DMA Linked List structure based on the local and remote
+ * scatter-gather lists, if needed.
+ *
+ * If the client is discarding the DMA transfer then a zero-length
+ * scatter-gather list is sent.
+ */
+static int easelcomm_send_dma_scatterlist(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata,
+ enum easelcomm_dma_direction dma_dir)
+{
+ struct easelcomm_dma_sg_header sg_header;
+ int ret;
+
+ sg_header.message_id = msg_metadata->msg->desc.message_id;
+ sg_header.dma_dir = dma_dir;
+ sg_header.scatterlist_size = msg_metadata->dma_xfer.sg_local_size;
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "send cmd DMA_SG msg %u:%s%llu size=%u\n",
+ service->service_id, easelcomm_msgid_prefix(msg_metadata),
+ msg_metadata->msg->desc.message_id,
+ msg_metadata->dma_xfer.sg_local_size);
+ ret = easelcomm_start_cmd(
+ service, EASELCOMM_CMD_DMA_SG,
+ sizeof(struct easelcomm_dma_sg_header) +
+ msg_metadata->dma_xfer.sg_local_size);
+ if (ret)
+ return ret;
+
+ ret = easelcomm_append_cmd_args(
+ service, &sg_header, sizeof(struct easelcomm_dma_sg_header));
+ if (ret)
+ return ret;
+
+ /* If not discarding locally append the scatterlist */
+ if (msg_metadata->dma_xfer.sg_local_size) {
+ ret = easelcomm_append_cmd_args(
+ service, msg_metadata->dma_xfer.sg_local,
+ msg_metadata->dma_xfer.sg_local_size);
+ if (ret)
+ return ret;
+ }
+
+ return easelcomm_send_cmd(service);
+}
+
+/* Server sends DMA_XFER command to client */
+static int easelcomm_send_dma_xfer(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata,
+ enum easelcomm_dma_direction dma_dir)
+{
+ struct easelcomm_dma_xfer_arg dma_xfer;
+ int ret;
+
+ dma_xfer.message_id = msg_metadata->msg->desc.message_id;
+ dma_xfer.dma_dir = dma_dir;
+ dma_xfer.xfer_type = msg_metadata->dma_xfer.xfer_type;
+ dma_xfer.server_addr = msg_metadata->dma_xfer.server_addr;
+
+ ret = easelcomm_start_cmd(
+ service, EASELCOMM_CMD_DMA_XFER,
+ sizeof(struct easelcomm_dma_xfer_arg));
+ if (ret)
+ return ret;
+
+ ret = easelcomm_append_cmd_args(
+ service, &dma_xfer, sizeof(struct easelcomm_dma_xfer_arg));
+ if (ret)
+ return ret;
+ dev_dbg(easelcomm_miscdev.this_device, "send cmd DMA_XFER msg %u:%s%llu type=%u saddr=%llx\n",
+ service->service_id,
+ easelcomm_msgid_prefix(msg_metadata),
+ msg_metadata->msg->desc.message_id,
+ msg_metadata->dma_xfer.xfer_type,
+ msg_metadata->dma_xfer.server_addr);
+ return easelcomm_send_cmd(service);
+}
+
+/*
+ * Server-side DMA handling. When the client sends its scatterlist, inspect
+ * the client and server lists to see if the DMA can be a single-block
+ * transfer or whether it must be a multi-block transfer. Construct the
+ * multi-block linked list if needed. Tell the client to proceed with a
+ * single-block or multi-block transfer, wait for DMA done command from
+ * client, and clean up.
+ */
+static int easelcomm_server_handle_dma_request(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata,
+ enum easelcomm_dma_direction dma_dir)
+{
+ void *mblk_ll_data = NULL;
+ bool discard = false;
+ int ret;
+
+ /* Wait for client to send its scatterlist, if not already */
+ ret = wait_for_completion_interruptible(
+ &msg_metadata->dma_xfer.sg_remote_ready);
+ if (ret || msg_metadata->dma_xfer.aborting)
+ goto abort;
+
+ if (!msg_metadata->dma_xfer.sg_remote_size) {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "client discards DMA for msg %u:%s%llu\n",
+ service->service_id,
+ easelcomm_msgid_prefix(msg_metadata),
+ msg_metadata->msg->desc.message_id);
+ return 0;
+ }
+
+ /* If discarding locally then send an abort to client. */
+ if (msg_metadata->dma_xfer.sg_local_size == 0) {
+ discard = true;
+ goto abort;
+ }
+
+ /*
+ * Choose SBLK vs. MBLK DMA based on whether scatterlists need more
+ * than one block.
+ */
+ if (easelcomm_hw_scatterlist_block_count(
+ msg_metadata->dma_xfer.sg_local_size) == 1 &&
+ easelcomm_hw_scatterlist_block_count(
+ msg_metadata->dma_xfer.sg_remote_size) == 1) {
+ /*
+ * Single-block DMA. server_addr for the DMA_XFER command
+ * is the Easel-side starting physical address.
+ */
+ msg_metadata->dma_xfer.xfer_type = EASELCOMM_DMA_XFER_SBLK;
+ msg_metadata->dma_xfer.server_addr =
+ easelcomm_hw_scatterlist_sblk_addr(
+ msg_metadata->dma_xfer.sg_local);
+ } else {
+ /*
+ * Multiple-block DMA. server_addr is the physical address
+ * of the Linked List we create here.
+ */
+ msg_metadata->dma_xfer.xfer_type = EASELCOMM_DMA_XFER_MBLK;
+
+ /* Build DMA Linked List for the transfer*/
+ if (dma_dir == EASELCOMM_DMA_DIR_TO_SERVER)
+ ret = easelcomm_hw_easel_build_ll(
+ msg_metadata->dma_xfer.sg_remote,
+ msg_metadata->dma_xfer.sg_local,
+ &mblk_ll_data);
+ else
+ ret = easelcomm_hw_easel_build_ll(
+ msg_metadata->dma_xfer.sg_local,
+ msg_metadata->dma_xfer.sg_remote,
+ &mblk_ll_data);
+
+ /* If error, tell client to abort DMA transfer */
+ if (ret < 0 || !mblk_ll_data)
+ goto abort;
+ msg_metadata->dma_xfer.server_addr =
+ easelcomm_hw_easel_ll_addr(mblk_ll_data);
+ }
+
+ /* Send DMA_XFER to client */
+ ret = easelcomm_send_dma_xfer(service, msg_metadata, dma_dir);
+ if (ret)
+ goto abort;
+
+ /* Wait for client to send a DMA Done for this message */
+ ret = wait_for_completion_interruptible(
+ &msg_metadata->dma_xfer.xfer_done);
+ if (ret || msg_metadata->dma_xfer.aborting)
+ goto abort;
+
+ ret = msg_metadata->dma_xfer.errcode;
+ goto out;
+
+abort:
+ dev_dbg(easelcomm_miscdev.this_device,
+ "aborting DMA for msg %u:%s%llu\n",
+ service->service_id, easelcomm_msgid_prefix(msg_metadata),
+ msg_metadata->msg->desc.message_id);
+ msg_metadata->dma_xfer.aborting = true;
+ /* Send a DMA abort to remote */
+ msg_metadata->dma_xfer.xfer_type = EASELCOMM_DMA_XFER_ABORT;
+ ret = easelcomm_send_dma_xfer(service, msg_metadata, dma_dir);
+ if (ret)
+ dev_err(easelcomm_miscdev.this_device,
+ "send DMA abort for msg %u:%s%llu failed: %d\n",
+ service->service_id,
+ easelcomm_msgid_prefix(msg_metadata),
+ msg_metadata->msg->desc.message_id, ret);
+ ret = discard ? 0 : -EIO;
+
+ /*
+ * TODO-LATER: Need to make sure DMA hardware no longer refers to
+ * LL before destroy. Related: Ensure DMA hardware no longer
+ * accessing the locally-pinned memory pages prior to SG unmap.
+ */
+
+out:
+ /* Destroy LL if allocated for MBLK DMA */
+ if (mblk_ll_data)
+ easelcomm_hw_easel_destroy_ll(mblk_ll_data);
+ return ret;
+}
+
+/* Client sends DMA Done command to server to let it know it can clean up. */
+static int easelcomm_send_dma_done(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata,
+ enum easelcomm_dma_direction dma_dir, int32_t errcode)
+{
+ struct easelcomm_dma_done_arg dma_done_arg;
+ int ret = 0;
+
+ /* Send DMA Done command to server */
+ dma_done_arg.message_id =
+ msg_metadata->msg->desc.message_id;
+ dma_done_arg.dma_dir = dma_dir;
+ dma_done_arg.errcode = errcode;
+ ret = easelcomm_start_cmd(
+ service, EASELCOMM_CMD_DMA_DONE,
+ sizeof(struct easelcomm_dma_done_arg));
+ if (ret)
+ return ret;
+ ret = easelcomm_append_cmd_args(
+ service, &dma_done_arg, sizeof(struct easelcomm_dma_done_arg));
+ if (ret)
+ return ret;
+ dev_dbg(easelcomm_miscdev.this_device,
+ "send cmd DMA_DONE msg %u:%s%llu errcode=%d\n",
+ service->service_id, easelcomm_msgid_prefix(msg_metadata),
+ msg_metadata->msg->desc.message_id, errcode);
+ ret = easelcomm_send_cmd(service);
+ return ret;
+}
+
+/* Client performs single-block DMA transfer */
+static int easelcomm_client_perform_dma_sblk(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata,
+ enum easelcomm_dma_direction dir)
+{
+ uint64_t client_addr = easelcomm_hw_scatterlist_sblk_addr(
+ msg_metadata->dma_xfer.sg_local);
+
+ dev_dbg(easelcomm_miscdev.this_device, "sblk dma msg %u:%s%llu dir=%d size=%u local=%llx saddr=%llx\n",
+ service->service_id, easelcomm_msgid_prefix(msg_metadata),
+ msg_metadata->msg->desc.message_id, dir,
+ msg_metadata->msg->desc.dma_buf_size, client_addr,
+ msg_metadata->dma_xfer.server_addr);
+ return easelcomm_hw_ap_dma_sblk_transfer(
+ client_addr, msg_metadata->dma_xfer.server_addr,
+ msg_metadata->msg->desc.dma_buf_size,
+ dir == EASELCOMM_DMA_DIR_TO_SERVER);
+}
+
+/* Client performs multi-block DMA transfer */
+static int easelcomm_client_perform_dma_mblk(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata,
+ enum easelcomm_dma_direction dir)
+{
+ dev_dbg(easelcomm_miscdev.this_device, "mblk dma msg %u:%s%llu dir=%d size=%u ll=%llx\n",
+ service->service_id, easelcomm_msgid_prefix(msg_metadata),
+ msg_metadata->msg->desc.message_id, dir,
+ msg_metadata->msg->desc.dma_buf_size,
+ msg_metadata->dma_xfer.server_addr);
+ return easelcomm_hw_ap_dma_mblk_transfer(
+ msg_metadata->dma_xfer.server_addr,
+ dir == EASELCOMM_DMA_DIR_TO_SERVER);
+}
+
+/*
+ * Client handles DMA transfer. Send local scatterlist to server, wait for
+ * instructions back on how to proceed with single- or multi-block transfer,
+ * perform the transfer, and send the DMA done command to server.
+ */
+static int easelcomm_client_handle_dma_request(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata,
+ enum easelcomm_dma_direction dma_dir,
+ int timeout_ms)
+{
+ int ret, ret2;
+
+ /* Send scatterlist to server in a DMA_SG command */
+ ret = easelcomm_send_dma_scatterlist(
+ service, msg_metadata, dma_dir);
+ if (ret)
+ return ret;
+
+ /* If sent a DMA discard then done */
+ if (!msg_metadata->dma_xfer.sg_local_size)
+ return 0;
+
+ /* Wait for server to return the DMA request info */
+ ret = wait_for_completion_interruptible_timeout(
+ &msg_metadata->dma_xfer.xfer_ready,
+ msecs_to_jiffies(timeout_ms));
+ if (ret == -ERESTARTSYS) {
+ dev_info(easelcomm_miscdev.this_device,
+ "Wait for DMA_XFER from server interrupted\n");
+ goto abort;
+ } else if (ret == 0) {
+ dev_err(easelcomm_miscdev.this_device,
+ "Wait for DMA_XFER from server timed out\n");
+ goto abort;
+ } else if (msg_metadata->dma_xfer.aborting)
+ goto abort;
+
+ /*
+ * If server discards/aborts the DMA request then done. If
+ * this is a client DMA send then silently discard. If the
+ * client is receiving a DMA buffer then return an error on
+ * the read.
+ */
+ if (msg_metadata->dma_xfer.xfer_type == EASELCOMM_DMA_XFER_ABORT)
+ return dma_dir == EASELCOMM_DMA_DIR_TO_CLIENT ? -EIO : 0;
+
+ /* Sanity check that list is properly terminated */
+ if (easelcomm_hw_verify_scatterlist(&msg_metadata->dma_xfer)) {
+ dev_err(easelcomm_miscdev.this_device,
+ "sg fails verification\n");
+ goto abort;
+ }
+
+ /* Single-Block or Multi-Block DMA? */
+ if (msg_metadata->dma_xfer.xfer_type == EASELCOMM_DMA_XFER_SBLK) {
+ /* Perform the SBLK DMA transfer */
+ ret = easelcomm_client_perform_dma_sblk(
+ service, msg_metadata, dma_dir);
+ } else {
+ /* Perform the MBLK DMA transfer */
+ ret = easelcomm_client_perform_dma_mblk(
+ service, msg_metadata, dma_dir);
+ }
+
+ /* Tell server DMA transfer is done */
+ ret2 = easelcomm_send_dma_done(service, msg_metadata, dma_dir, ret);
+ return ret || ret2;
+
+ /*
+ * We are aborting the transfer before it started (local flush or
+ * signal received)
+ */
+abort:
+ msg_metadata->dma_xfer.aborting = true;
+ /* send abort to server */
+ msg_metadata->dma_xfer.xfer_type = EASELCOMM_DMA_XFER_ABORT;
+ easelcomm_send_dma_done(service, msg_metadata, dma_dir, -EIO);
+ return -EIO;
+}
+
+/* RECVDMA ioctl processing. */
+int easelcomm_receive_dma(
+ struct easelcomm_service *service,
+ struct easelcomm_kbuf_desc *buf_desc)
+{
+ struct easelcomm_message_metadata *msg_metadata;
+ enum easelcomm_dma_direction dma_dir;
+ int ret = 0;
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "RECVDMA msg %u:r%llu buf_type=%d buf_size=%u timeout_ms=%d dma_buf_fd=%d dma_buf_off=%u dma_buf_width=%u dma_buf_stride=%u\n",
+ service->service_id, buf_desc->message_id,
+ buf_desc->buf_type, buf_desc->buf_size,
+ buf_desc->wait.timeout_ms,
+ buf_desc->dma_buf_fd,
+ buf_desc->dma_buf_off,
+ buf_desc->dma_buf_width,
+ buf_desc->dma_buf_stride);
+
+ msg_metadata =
+ easelcomm_find_remote_message(service, buf_desc->message_id);
+ if (!msg_metadata) {
+ dev_err(easelcomm_miscdev.this_device, "RECVDMA msg r%llu not found\n",
+ buf_desc->message_id);
+ return -EINVAL;
+ }
+
+ if (buf_desc->buf_size &&
+ buf_desc->buf_size != msg_metadata->msg->desc.dma_buf_size) {
+ dev_err(easelcomm_miscdev.this_device,
+ "RECVDMA descriptor buffer size %u doesn't match msg r%llu size %u\n",
+ buf_desc->buf_size, buf_desc->message_id,
+ msg_metadata->msg->desc.dma_buf_size);
+ ret = -EINVAL;
+ goto out;
+ }
+
+ /* If the message doesn't even request a DMA transfer then skip it. */
+
+ if (!msg_metadata->msg->desc.dma_buf_size)
+ goto out;
+
+ /* If it's a user buffer, check valid range and writable. */
+ if (buf_desc->buf_type == EASELCOMM_DMA_BUFFER_USER) {
+ if (!access_ok(VERIFY_WRITE, buf_desc->buf,
+ buf_desc->buf_size)) {
+ ret = -EFAULT;
+ goto out;
+ }
+ }
+ dma_dir = easelcomm_is_client() ?
+ EASELCOMM_DMA_DIR_TO_CLIENT : EASELCOMM_DMA_DIR_TO_SERVER;
+
+ /*
+ * If the DMA transfer is not being discarded locally then generate
+ * the local DMA scatter-gather list.
+ */
+ msg_metadata->dma_xfer.sg_local_size = 0;
+ msg_metadata->dma_xfer.sg_local_localdata = NULL;
+
+ if (buf_desc->buf_size) {
+ msg_metadata->dma_xfer.sg_local =
+ easelcomm_create_dma_scatterlist(
+ buf_desc,
+ &msg_metadata->dma_xfer.sg_local_size,
+ &msg_metadata->dma_xfer.sg_local_localdata,
+ dma_dir);
+ if (!msg_metadata->dma_xfer.sg_local) {
+ ret = -ENOMEM;
+ msg_metadata->dma_xfer.sg_local_size = 0;
+ goto out;
+ }
+ }
+
+ if (easelcomm_is_client())
+ ret = easelcomm_client_handle_dma_request(
+ service, msg_metadata, dma_dir,
+ buf_desc->wait.timeout_ms);
+ else
+ ret = easelcomm_server_handle_dma_request(
+ service, msg_metadata, dma_dir);
+
+ /* Free contents of struct mnh_sg_list */
+ if (msg_metadata->dma_xfer.sg_local_localdata)
+ easelcomm_hw_destroy_scatterlist(
+ msg_metadata->dma_xfer.sg_local_localdata);
+
+out:
+ /* If no reply needed then done with the remote message. */
+ easelcomm_drop_reference(service, msg_metadata,
+ !msg_metadata->msg->desc.need_reply);
+ return ret;
+}
+EXPORT_SYMBOL(easelcomm_receive_dma);
+
+/* SENDDMA ioctl processing. */
+int easelcomm_send_dma(
+ struct easelcomm_service *service,
+ struct easelcomm_kbuf_desc *buf_desc)
+{
+ struct easelcomm_message_metadata *msg_metadata;
+ enum easelcomm_dma_direction dma_dir;
+ int ret = 0;
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "SENDDMA msg %u:l%llu buf_type=%d buf_size=%u timeout_ms=%d dma_buf_fd=%d dma_buf_off=%u dma_buf_width=%u dma_buf_stride=%u\n",
+ service->service_id, buf_desc->message_id,
+ buf_desc->buf_type, buf_desc->buf_size,
+ buf_desc->wait.timeout_ms,
+ buf_desc->dma_buf_fd, buf_desc->dma_buf_off,
+ buf_desc->dma_buf_width, buf_desc->dma_buf_stride);
+
+ msg_metadata =
+ easelcomm_find_local_message(service, buf_desc->message_id);
+ if (!msg_metadata) {
+ dev_err(easelcomm_miscdev.this_device, "SENDDMA msg l%llu not found\n",
+ buf_desc->message_id);
+ return -EINVAL;
+ }
+
+ if (buf_desc->buf_size &&
+ buf_desc->buf_size != msg_metadata->msg->desc.dma_buf_size) {
+ dev_err(easelcomm_miscdev.this_device,
+ "SENDDMA descriptor buffer size %u doesn't match message l%llu size %u\n",
+ buf_desc->buf_size, buf_desc->message_id,
+ msg_metadata->msg->desc.dma_buf_size);
+ ret = -EINVAL;
+ goto out;
+ }
+
+ /* If it's a user buffer, check valid range and readable. */
+ if (buf_desc->buf_type == EASELCOMM_DMA_BUFFER_USER) {
+ if (!access_ok(VERIFY_READ, buf_desc->buf,
+ buf_desc->buf_size)) {
+ ret = -EFAULT;
+ goto out;
+ }
+ }
+
+ dma_dir = easelcomm_is_client() ?
+ EASELCOMM_DMA_DIR_TO_SERVER : EASELCOMM_DMA_DIR_TO_CLIENT;
+
+ /*
+ * If the DMA transfer is not being discarded locally then generate
+ * the local DMA scatter-gather list.
+ */
+ msg_metadata->dma_xfer.sg_local_size = 0;
+ msg_metadata->dma_xfer.sg_local_localdata = NULL;
+ if (buf_desc->buf_size) {
+ msg_metadata->dma_xfer.sg_local =
+ easelcomm_create_dma_scatterlist(
+ buf_desc,
+ &msg_metadata->dma_xfer.sg_local_size,
+ &msg_metadata->dma_xfer.sg_local_localdata,
+ dma_dir);
+ if (!msg_metadata->dma_xfer.sg_local) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ }
+
+ if (msg_metadata->msg->desc.dma_buf_size) {
+ if (easelcomm_is_client())
+ ret = easelcomm_client_handle_dma_request(
+ service, msg_metadata, dma_dir,
+ buf_desc->wait.timeout_ms);
+ else
+ ret = easelcomm_server_handle_dma_request(
+ service, msg_metadata, dma_dir);
+ }
+
+ /* Free contents of struct mnh_sg_list */
+ if (msg_metadata->dma_xfer.sg_local_localdata)
+ easelcomm_hw_destroy_scatterlist(
+ msg_metadata->dma_xfer.sg_local_localdata);
+
+out:
+ /* If no reply needed then done with local message. */
+ easelcomm_drop_reference(service, msg_metadata,
+ !msg_metadata->msg->desc.need_reply);
+ return ret;
+}
+EXPORT_SYMBOL(easelcomm_send_dma);
diff --git a/drivers/misc/google-easel-comm-hw.c b/drivers/misc/google-easel-comm-hw.c
new file mode 100644
index 0000000..ac0a8cb
--- /dev/null
+++ b/drivers/misc/google-easel-comm-hw.c
@@ -0,0 +1,666 @@
+/*
+ * Android coprocessor communication hardware access functions
+ *
+ * Copyright 2016 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+/* #define DEBUG */
+
+#ifdef CONFIG_GOOGLE_EASEL_AP
+#define EASELCOMM_AP 1
+#else
+#define EASELCOMM_EASEL 1
+#endif
+
+#ifdef EASELCOMM_EASEL
+#include <linux/mnh_pcie_ep.h>
+#else
+#include "mnh/mnh-pcie.h"
+#include "mnh/mnh-sm.h"
+#include "mnh/hw-mnh-regs.h"
+#endif
+
+#include "google-easel-comm-shared.h"
+#include "google-easel-comm-private.h"
+
+#include <linux/io.h>
+#include <linux/compiler.h>
+#include <linux/completion.h>
+#include <linux/dma-mapping.h>
+#include <linux/io.h>
+#include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/workqueue.h>
+
+#ifdef EASELCOMM_AP
+/* Use only one DMA channel for app requests */
+#define APP_DMA_CHAN 1
+
+/* timeout for waiting for bootstrap MSI after hotplug */
+#define BOOTSTRAP_TIMEOUT_MS 2000
+
+/* Signaled when server sents bootstrap MSI */
+static DECLARE_COMPLETION(bootstrap_done);
+
+/*
+ * Mutex serializes thread context access to app_dma* data structures below.
+ * Client thread performing the app DMA transfer holds this for the duration
+ * of the h/w-layer transfer operation.
+ * The DMA IRQ callback accesses these without locking.
+ */
+static struct mutex app_dma_mutex;
+/* ISR signals app DMA done */
+static DECLARE_COMPLETION(app_dma_done);
+/* Status of last DMA transfer */
+static enum mnh_dma_trans_status_t app_dma_status;
+
+/* MNH BAR4 window size (and addressing granularity) 4MB */
+#define MNH_BAR4_MASK ((4 * 1024 * 1024) - 1)
+
+/* DMA address of Easel cmdchan buffer */
+static uint64_t easel_cmdchan_dma_addr;
+/* Offset between iATU inbound 4MB-aligned window and actual buffer start */
+static uint32_t easel_cmdchan_pcie_offset;
+#endif
+
+/* Common AP/Easel defines */
+/* Local cmdchan buffer cpu virtual address */
+static void *local_cmdchan_cpu_addr;
+/* Local cmdchan buffer DMA address */
+static dma_addr_t local_cmdchan_dma_addr;
+
+/* Command channel data ready, call up to generic layer. */
+static void easelcomm_hw_cmdchan_data_ready(struct work_struct *work)
+{
+ easelcomm_cmd_channel_data_handler();
+}
+static DECLARE_WORK(cmdchan_data, easelcomm_hw_cmdchan_data_ready);
+
+/* Command channel remote wrapped received, call up to generic layer. */
+static void easelcomm_hw_cmdchan_wrap(struct work_struct *work)
+{
+ easelcomm_cmd_channel_wrap_handler();
+}
+static DECLARE_WORK(cmdchan_wrap, easelcomm_hw_cmdchan_wrap);
+
+/* AP/client cmdchan setup. */
+int easelcomm_hw_ap_setup_cmdchans(void)
+{
+#ifdef EASELCOMM_AP
+ struct mnh_inb_window inbound;
+ struct mnh_outb_region outbound;
+ int ret;
+
+ /* Command channel size shall not exceed the translation region size */
+ BUILD_BUG_ON_MSG(
+ EASELCOMM_CMD_CHANNEL_LOCAL_SIZE > HW_MNH_PCIE_OUTBOUND_SIZE,
+ "Easelcomm command channel size exceeds translation region");
+
+ ret = mnh_get_rb_base(&easel_cmdchan_dma_addr);
+ if (WARN_ON(ret))
+ return ret;
+ pr_debug("easelcomm: cmdchans: ap virt %p dma %pad easel dma %llx",
+ local_cmdchan_cpu_addr, &local_cmdchan_dma_addr,
+ easel_cmdchan_dma_addr);
+
+ /* Inbound region 0 BAR4 match map to Easel cmdchan phys addr */
+ inbound.region = 0;
+ inbound.mode = BAR_MATCH;
+ inbound.bar = 4;
+ /* Align target address for 4MB BAR size */
+ /*
+ * TODO: Want a carveout dedicated for this so no chance of 4MB
+ * boundary crossed. b/32782918.
+ */
+ inbound.target_mth_address =
+ easel_cmdchan_dma_addr & ~(MNH_BAR4_MASK);
+ easel_cmdchan_pcie_offset = easel_cmdchan_dma_addr & (MNH_BAR4_MASK);
+ ret = mnh_set_inbound_iatu(&inbound);
+ if (WARN_ON(ret))
+ return ret;
+
+ /* Outbound region 1 map to AP cmdchan DMA addr */
+ outbound.region = 1;
+ outbound.base_address = HW_MNH_PCIE_OUTBOUND_BASE;
+ outbound.limit_address = HW_MNH_PCIE_OUTBOUND_BASE +
+ EASELCOMM_CMD_CHANNEL_LOCAL_SIZE - 1;
+ outbound.target_pcie_address = local_cmdchan_dma_addr;
+ ret = mnh_set_outbound_iatu(&outbound);
+ if (WARN_ON(ret))
+ return ret;
+ return 0;
+#else
+ return -EIO;
+#endif
+}
+EXPORT_SYMBOL(easelcomm_hw_ap_setup_cmdchans);
+
+/*
+ * Send "remote producer wrote new data to the local command channel ring
+ * buffer" interrupt.
+ */
+int easelcomm_hw_send_data_interrupt(void)
+{
+ int ret;
+
+#ifdef EASELCOMM_AP
+ ret = mnh_send_irq(IRQ_MSG_SENT);
+#else
+ ret = mnh_send_msi(MSG_SEND_M);
+#endif
+ WARN_ON(ret);
+ return ret;
+}
+EXPORT_SYMBOL(easelcomm_hw_send_data_interrupt);
+
+/*
+ * Send "remote consumer caught up with local producer and is ready to wrap the
+ * remote command channel ring buffer" interrupt.
+ */
+int easelcomm_hw_send_wrap_interrupt(void)
+{
+ int ret;
+#ifdef EASELCOMM_AP
+ ret = mnh_send_irq(IRQ_APPDEFINED_1);
+#else
+ ret = mnh_send_msi(APPDEFINED_1_M);
+#endif
+ WARN_ON(ret);
+ return ret;
+}
+EXPORT_SYMBOL(easelcomm_hw_send_wrap_interrupt);
+
+#ifdef EASELCOMM_EASEL
+/* EP/server MNH API IRQ callback */
+static int easelcomm_hw_easel_irq_callback(struct mnh_pcie_irq *irq)
+{
+ if (irq->msi_irq == MSG_SEND_I)
+ schedule_work(&cmdchan_data);
+ else if (irq->msi_irq == APPDEFINED_1_I)
+ schedule_work(&cmdchan_wrap);
+ else
+ pr_info("easelcomm: mnh msi %u ignored\n",
+ irq->msi_irq);
+ return 0;
+}
+
+/* EP/server MNH DMA IRQ callback, not currently used */
+static int easelcomm_hw_easel_dma_callback(struct mnh_dma_irq *irq)
+{
+ return 0;
+}
+
+/*
+ * Set the command channel ring buffer address in the MNH cluster register
+ * that the AP/client reads from, and send the MSI informing the ap that
+ * the value is setup.
+ */
+static int easelcomm_hw_easel_advertise_cmdchan(uint64_t buffer_dma_addr)
+{
+ int ret;
+
+ pr_debug("easelcomm: advertising buffer dma=%llx\n", buffer_dma_addr);
+ ret = mnh_set_rb_base(buffer_dma_addr);
+ WARN_ON(ret);
+ return ret;
+}
+
+/* Module init time actions for EP/server */
+int easelcomm_hw_init(void)
+{
+ int ret;
+
+ /* dma alloc a buffer for now, may want dedicated carveout */
+ /* temporary: alloc twice as much, align to boundary */
+ local_cmdchan_cpu_addr =
+ mnh_alloc_coherent(EASELCOMM_CMD_CHANNEL_LOCAL_SIZE,
+ &local_cmdchan_dma_addr);
+ pr_debug("easelcomm: cmdchan v=%p d=%pad\n",
+ local_cmdchan_cpu_addr, &local_cmdchan_dma_addr);
+
+ /* "PCIe" is immediately ready for server, no hotplug in/probe wait */
+ ret = easelcomm_init_pcie_ready(local_cmdchan_cpu_addr);
+ WARN_ON(ret);
+
+ /* Register EP-specific IRQ callbacks */
+ ret = mnh_reg_irq_callback(
+ &easelcomm_hw_easel_irq_callback,
+ &easelcomm_hw_easel_dma_callback);
+ WARN_ON(ret);
+
+ ret = easelcomm_hw_easel_advertise_cmdchan(
+ (uint64_t)local_cmdchan_dma_addr);
+ return ret;
+}
+EXPORT_SYMBOL(easelcomm_hw_init);
+#endif
+
+#ifdef EASELCOMM_AP
+/* AP/client MNH API MSI callback */
+static int easelcomm_hw_ap_msi_callback(uint32_t msi)
+{
+ pr_debug("easelcomm: msi %u\n", msi);
+
+ switch (msi) {
+ case MSI_BOOTSTRAP_SET:
+ complete(&bootstrap_done);
+ break;
+ case MSI_MSG_SEND:
+ schedule_work(&cmdchan_data);
+ break;
+ case MSI_PET_WATCHDOG:
+ pr_debug("easelcomm: ignore MSI_PET_WATCHDOG\n");
+ break;
+ case MSI_APPDEFINED_1:
+ schedule_work(&cmdchan_wrap);
+ break;
+ default:
+ pr_warn("easelcomm: MSI %u ignored\n", msi);
+ }
+
+ return 0;
+}
+
+/* AP/client MNH API DMA IRQ callback */
+static int easelcomm_hw_ap_dma_callback(
+ uint8_t chan, enum mnh_dma_chan_dir_t dir,
+ enum mnh_dma_trans_status_t status)
+{
+ pr_debug("easelcomm: DMA done chan=%u dir=%u status=%u\n",
+ chan, dir, status);
+ if (chan == APP_DMA_CHAN) {
+ app_dma_status = status;
+ complete(&app_dma_done);
+ }
+ return 0;
+}
+
+/* AP/client PCIe ready, EP enumerated, can now use MNH host driver. */
+static int easelcomm_hw_ap_pcie_ready(unsigned long bootstrap_timeout_jiffies)
+{
+ int ret = 0;
+ uint64_t temp_rb_base_val;
+
+ /* Only allocate ringbuffer once for AP */
+ if (local_cmdchan_cpu_addr == NULL) {
+ local_cmdchan_cpu_addr = mnh_alloc_coherent(
+ EASELCOMM_CMD_CHANNEL_LOCAL_SIZE,
+ &local_cmdchan_dma_addr);
+ if (IS_ERR_OR_NULL(local_cmdchan_cpu_addr)) {
+ pr_warn("%s: failed to alloc ringbuffer\n", __func__);
+ return -ENOMEM;
+ }
+ }
+
+ ret = easelcomm_init_pcie_ready(local_cmdchan_cpu_addr);
+ WARN_ON(ret);
+
+ /* Register AP-specific IRQ callbacks */
+ ret = mnh_reg_irq_callback(
+ &easelcomm_hw_ap_msi_callback, NULL,
+ &easelcomm_hw_ap_dma_callback);
+ WARN_ON(ret);
+
+ /*
+ * Wasel is booting in parallel, poll whether it already sent
+ * bootstrap done MSI with ringbuffer base setup, prior to us
+ * being ready to handle the IRQ.
+ */
+ ret = mnh_get_rb_base(&temp_rb_base_val);
+ if (WARN_ON(ret)) {
+ pr_err("%s: mnh_get_rb_base failed (%d)\n", __func__, ret);
+ mnh_reg_irq_callback(NULL, NULL, NULL);
+ return ret;
+ } else if (!temp_rb_base_val) {
+ /* wait for bootstrap completion */
+ ret = wait_for_completion_timeout(&bootstrap_done,
+ bootstrap_timeout_jiffies);
+ if (!ret) {
+ pr_err("%s: timeout waiting for bootstrap msi\n",
+ __func__);
+ mnh_reg_irq_callback(NULL, NULL, NULL);
+ return -ETIMEDOUT;
+ }
+ }
+
+ ret = easelcomm_client_remote_cmdchan_ready_handler();
+ if (ret)
+ pr_warn("%s: remote_cmdchan_ready_handler returns %d\n",
+ __func__, ret);
+
+ return ret;
+}
+
+/* Callback on MNH host driver hotplug in/out events. */
+static int easelcomm_hw_ap_hotplug_callback(enum mnh_hotplug_event_t event,
+ void *param)
+{
+ int ret = 0;
+ unsigned long timeout_ms = (unsigned long)param;
+ static enum mnh_hotplug_event_t state = MNH_HOTPLUG_OUT;
+
+ if (state == event)
+ return 0;
+
+ switch (event) {
+ case MNH_HOTPLUG_IN:
+ pr_debug("%s: mnh hotplug in\n", __func__);
+ if (!timeout_ms)
+ timeout_ms = BOOTSTRAP_TIMEOUT_MS;
+ ret = easelcomm_hw_ap_pcie_ready(msecs_to_jiffies(timeout_ms));
+ break;
+ case MNH_HOTPLUG_OUT:
+ pr_debug("%s: mnh hotplug out\n", __func__);
+ reinit_completion(&bootstrap_done);
+ /* Unregister IRQ callbacks first */
+ ret = mnh_reg_irq_callback(NULL, NULL, NULL);
+ WARN_ON(ret);
+ /* Call hotplug out callback implemented by easelcomm layer */
+ easelcomm_pcie_hotplug_out();
+ break;
+ default:
+ ret = -EINVAL;
+ break;
+ }
+
+ if (!ret)
+ state = event;
+
+ return ret;
+}
+
+/* Module init time actions for AP/client */
+int easelcomm_hw_init(void)
+{
+ int ret;
+
+ mutex_init(&app_dma_mutex);
+
+ local_cmdchan_cpu_addr = NULL;
+
+ ret = mnh_sm_reg_hotplug_callback(&easelcomm_hw_ap_hotplug_callback);
+ if (WARN_ON(ret))
+ return ret;
+
+ return 0;
+}
+EXPORT_SYMBOL(easelcomm_hw_init);
+#endif
+
+/* Read remote ringbuffer memory */
+int easelcomm_hw_remote_read(
+ void *local_addr, size_t len, uint64_t remote_offset)
+{
+ int ret;
+
+#ifdef EASELCOMM_AP
+ ret = mnh_ddr_read(easel_cmdchan_pcie_offset + (uint32_t)remote_offset,
+ len, local_addr);
+#else
+ ret = mnh_pcie_read(local_addr, len, remote_offset);
+#endif
+ WARN_ON(ret);
+ return ret;
+}
+EXPORT_SYMBOL(easelcomm_hw_remote_read);
+
+/* Write remote ringbuffer memory. */
+int easelcomm_hw_remote_write(
+ void *local_addr, size_t len, uint64_t remote_offset)
+{
+ int ret;
+
+#ifdef EASELCOMM_AP
+ ret = mnh_ddr_write(easel_cmdchan_pcie_offset + (uint32_t)remote_offset,
+ len, local_addr);
+#else
+ ret = mnh_pcie_write(local_addr, len, remote_offset);
+#endif
+ WARN_ON(ret);
+ return ret;
+}
+EXPORT_SYMBOL(easelcomm_hw_remote_write);
+
+/* Build an MNH scatter-gather list */
+void *easelcomm_hw_build_scatterlist(struct easelcomm_kbuf_desc *buf_desc,
+ uint32_t *scatterlist_size,
+ void **sglocaldata, enum easelcomm_dma_direction dma_dir)
+{
+ int n_ents_used = 0;
+ struct mnh_sg_entry *sg_ents;
+ struct mnh_sg_list *local_sg_info;
+ int ret;
+ bool to_easel = (dma_dir == EASELCOMM_DMA_DIR_TO_SERVER);
+
+ local_sg_info = kmalloc(sizeof(struct mnh_sg_list), GFP_KERNEL);
+ *sglocaldata = local_sg_info;
+ if (!local_sg_info) {
+ *scatterlist_size = 0;
+ return NULL;
+ }
+ /*
+ * Initialize dma_buf related pointers to NULL; if dma_buf is used,
+ * they will become non-zero by mnh_sg_retrieve_from_dma_buf().
+ * easelcomm_hw_destroy_scatterlist() will use this information
+ * to decide how to release the scatterlist.
+ */
+ local_sg_info->dma_buf = NULL;
+ local_sg_info->attach = NULL;
+ local_sg_info->sg_table = NULL;
+
+ local_sg_info->dir = to_easel ? DMA_AP2EP : DMA_EP2AP;
+
+ switch (buf_desc->buf_type) {
+ case EASELCOMM_DMA_BUFFER_UNUSED:
+ pr_err("%s: DMA buffer not used.\n", __func__);
+ ret = -EINVAL;
+ break;
+ case EASELCOMM_DMA_BUFFER_USER:
+ ret = mnh_sg_build(buf_desc->buf, buf_desc->buf_size,
+ &sg_ents, local_sg_info);
+ break;
+ case EASELCOMM_DMA_BUFFER_DMA_BUF:
+ ret = mnh_sg_retrieve_from_dma_buf(buf_desc->dma_buf_fd,
+ buf_desc->dma_buf_off,
+ buf_desc->buf_size,
+ buf_desc->dma_buf_width,
+ buf_desc->dma_buf_stride,
+ &sg_ents, local_sg_info);
+ break;
+ default:
+ pr_err("%s: Unknown DMA buffer type %d.\n",
+ __func__, buf_desc->buf_type);
+ ret = -EINVAL;
+ break;
+ }
+
+ if (ret < 0) {
+ kfree(local_sg_info);
+ *sglocaldata = NULL;
+ *scatterlist_size = 0;
+ return NULL;
+ }
+
+ n_ents_used = local_sg_info->length;
+ *scatterlist_size = sizeof(struct mnh_sg_entry) * n_ents_used;
+ return sg_ents;
+}
+EXPORT_SYMBOL(easelcomm_hw_build_scatterlist);
+
+int easelcomm_hw_verify_scatterlist(struct easelcomm_dma_xfer_info *xfer)
+{
+ return mnh_sg_verify(xfer->sg_local, xfer->sg_local_size,
+ xfer->sg_local_localdata);
+}
+
+/*
+ * Return the number of scatter-gather entries in the MNH SG list. Used to
+ * determine whether both sides require only 1 block and can use single-block
+ * DMA for the transfer.
+ */
+int easelcomm_hw_scatterlist_block_count(uint32_t scatterlist_size)
+{
+ if (scatterlist_size == 0)
+ return 0;
+ scatterlist_size /= sizeof(struct mnh_sg_entry);
+ return scatterlist_size - 1; /* subtract the terminator entry */
+}
+EXPORT_SYMBOL(easelcomm_hw_scatterlist_block_count);
+
+/*
+ * Return the physical/DMA address of the (server-side) single entry in the
+ * SG list. Used to determine the address needed by the client side for the
+ * single-block DMA transfer. It must alreayd have been determined that the
+ * SG list has only one entry via easelcomm_hw_scatterlist_block_count()
+ * above.
+ */
+uint64_t easelcomm_hw_scatterlist_sblk_addr(void *sgent)
+{
+ return (uint64_t)(((struct mnh_sg_entry *)sgent)->paddr);
+}
+EXPORT_SYMBOL(easelcomm_hw_scatterlist_sblk_addr);
+
+/* Destroy the MNH SG local mapping data */
+void easelcomm_hw_destroy_scatterlist(void *sglocaldata)
+{
+ struct mnh_sg_list *sg_local_data = (struct mnh_sg_list *)sglocaldata;
+
+ if (sglocaldata) {
+ if (sg_local_data->dma_buf == NULL) {
+ /* Destroy sgl created by mnh_sg_build() */
+ mnh_sg_destroy(sg_local_data);
+ } else {
+ /* Release sgl retrieved from dma_buf framework */
+ mnh_sg_release_from_dma_buf(sg_local_data);
+ }
+ kfree(sg_local_data);
+ }
+}
+EXPORT_SYMBOL(easelcomm_hw_destroy_scatterlist);
+
+/* Server builds an MNH DMA linked list for a multi-block transfer */
+int easelcomm_hw_easel_build_ll(
+ void *src_sg, void *dest_sg, void **ll_data)
+{
+#ifdef EASELCOMM_EASEL
+ struct mnh_dma_ll *mnh_ll;
+ int ret;
+
+ *ll_data = NULL;
+ mnh_ll = kmalloc(sizeof(struct mnh_dma_ll), GFP_KERNEL);
+ if (!mnh_ll)
+ return -ENOMEM;
+ ret = mnh_ll_build(src_sg, dest_sg, mnh_ll);
+ if (ret) {
+ kfree(mnh_ll);
+ return ret;
+ }
+
+ *ll_data = mnh_ll;
+ return 0;
+#else
+ return -EIO;
+#endif
+}
+EXPORT_SYMBOL(easelcomm_hw_easel_build_ll);
+
+/*
+ * Server returns Linked List DMA start address to send to client to
+ * initiate a multi-block DMA transfer.
+ */
+uint64_t easelcomm_hw_easel_ll_addr(void *ll_data)
+{
+#ifdef EASELCOMM_EASEL
+ return (uint64_t)((struct mnh_dma_ll *)ll_data)->dma[0];
+#else
+ return 0;
+#endif
+}
+EXPORT_SYMBOL(easelcomm_hw_easel_ll_addr);
+
+/* Server destroys an MNH DMA Linked List. */
+int easelcomm_hw_easel_destroy_ll(void *ll_data)
+{
+#ifdef EASELCOMM_EASEL
+ mnh_ll_destroy((struct mnh_dma_ll *)ll_data);
+ kfree(ll_data);
+ return 0;
+#else
+ return -EIO;
+#endif
+}
+EXPORT_SYMBOL(easelcomm_hw_easel_destroy_ll);
+
+/* Client performs a single-block DMA transfer */
+int easelcomm_hw_ap_dma_sblk_transfer(
+ uint64_t ap_daddr, uint64_t easel_daddr, uint32_t xfer_len,
+ bool to_easel)
+{
+#ifdef EASELCOMM_AP
+ enum mnh_dma_chan_dir_t dir = to_easel ? DMA_AP2EP : DMA_EP2AP;
+ struct mnh_dma_element_t blk;
+ int ret;
+
+ blk.src_addr = to_easel ? ap_daddr : easel_daddr;
+ blk.dst_addr = to_easel ? easel_daddr : ap_daddr;
+ blk.len = xfer_len;
+
+ mutex_lock(&app_dma_mutex);
+ reinit_completion(&app_dma_done);
+ ret = mnh_dma_sblk_start(APP_DMA_CHAN, dir, &blk);
+ if (WARN_ON(ret)) {
+ mutex_unlock(&app_dma_mutex);
+ return -EIO;
+ }
+
+ ret = wait_for_completion_interruptible(&app_dma_done);
+ if (WARN_ON(ret)) {
+ /* Ensure DMA aborted before returning */
+ ret = mnh_dma_abort(APP_DMA_CHAN, dir);
+ WARN_ON(ret);
+ mutex_unlock(&app_dma_mutex);
+ return ret;
+ }
+
+ mutex_unlock(&app_dma_mutex);
+ return app_dma_status == DMA_DONE ? 0 : -EIO;
+#else
+ return -EIO;
+#endif
+}
+EXPORT_SYMBOL(easelcomm_hw_ap_dma_sblk_transfer);
+
+/* Client performs a multi-block DMA transfer */
+int easelcomm_hw_ap_dma_mblk_transfer(uint64_t ll_paddr, bool to_easel)
+{
+#ifdef EASELCOMM_AP
+ enum mnh_dma_chan_dir_t dir = to_easel ? DMA_AP2EP : DMA_EP2AP;
+ int ret;
+
+ mutex_lock(&app_dma_mutex);
+ reinit_completion(&app_dma_done);
+ ret = mnh_dma_mblk_start(APP_DMA_CHAN, dir, &ll_paddr);
+ if (WARN_ON(ret)) {
+ mutex_unlock(&app_dma_mutex);
+ return -EIO;
+ }
+ ret = wait_for_completion_interruptible(&app_dma_done);
+ if (WARN_ON(ret)) {
+ /* Ensure DMA aborted before returning */
+ ret = mnh_dma_abort(APP_DMA_CHAN, dir);
+ WARN_ON(ret);
+ mutex_unlock(&app_dma_mutex);
+ return ret;
+ }
+ mutex_unlock(&app_dma_mutex);
+ return app_dma_status == DMA_DONE ? 0 : -EIO;
+#else
+ return -EIO;
+#endif
+}
+EXPORT_SYMBOL(easelcomm_hw_ap_dma_mblk_transfer);
diff --git a/drivers/misc/google-easel-comm-private.h b/drivers/misc/google-easel-comm-private.h
new file mode 100644
index 0000000..6a2e856
--- /dev/null
+++ b/drivers/misc/google-easel-comm-private.h
@@ -0,0 +1,321 @@
+/*
+ * Android/Easel coprocessor communication hardware access to the PCIe/EP
+ * driver on the local side of the link (AP or Easel), used by the common and
+ * link-partner-specific code.
+ *
+ * Copyright 2016 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef _GOOGLE_EASEL_COMM_PRIVATE_H
+#define _GOOGLE_EASEL_COMM_PRIVATE_H
+
+#include "google-easel-comm-shared.h"
+
+#include <linux/completion.h>
+#include <linux/list.h>
+#include <linux/miscdevice.h>
+#include <linux/mutex.h>
+#include <linux/spinlock.h>
+#include <linux/types.h>
+
+/*
+ * Message types used to set the general type of processing for the message
+ * and in which local data structures the message exists:
+ * local: message originated locally with a local message ID, stored in the
+ * local message list for the associated Easel service.
+ * remote non-reply: message received from remote with a remote message ID,
+ * stored in the remote message list, and also appears in the receive
+ * message queue until returned by a receiveMessage() call.
+ * remote reply: message from remote that is processed as a reply to a local
+ * message, does not go into the receiveMessage() queue.
+ */
+enum easelcomm_msg_type {
+ TYPE_LOCAL = 0, /* local/outgoing */
+ TYPE_REMOTE_NONREPLY, /* remote/incoming general */
+ TYPE_REMOTE_REPLY, /* remote/incoming reply */
+};
+
+/*
+ * Message metadata specific to DMA transfer handling. Messages with no DMA
+ * transfer have default values for these.
+ */
+struct easelcomm_dma_xfer_info {
+ /*
+ * The local MNH driver scatter-gather list, or NULL if discarding.
+ * This points to an array of mnh_sg_entry. Content of this is
+ * to be used to combine with remote scatter-gather list to be a
+ * DMA Linked List.
+ */
+ void *sg_local;
+ /* Size in bytes of the local scatter-gather list, zero if discard */
+ uint32_t sg_local_size;
+ /*
+ * Local data private to the MNH layer associated with the SG list.
+ * This points to mnh_sg_list. It marks information of
+ * the buffer for DMA transfer, and is used to collect local MNH
+ * driver scatter-gather list (sg_local).
+ */
+ void *sg_local_localdata;
+ /*
+ * Easel/server keeps the remote scatter-gather list received from the
+ * client (in a DMA_SG command) here for processing. The server is in
+ * charge of building the DMA Linked List for a multi-block transfer or
+ * deciding to use a single-block transfer based on the SG lists.
+ */
+ void *sg_remote;
+ /* Size in bytes of the remote SG list */
+ uint32_t sg_remote_size;
+ /* Informs server remote SG received from client */
+ struct completion sg_remote_ready;
+ /*
+ * Server determines DMA transfer type (SBLK/MBLK/ABORT) and sends to
+ * client to perform (or discard) the transfer. Sent with DMA_XFER
+ * command.
+ */
+ uint32_t xfer_type;
+ /*
+ * The server's address applicable to the DMA transfer command:
+ * The Multi-Block Linked List physical address (in Easel memory) or
+ * the Single-Block server-side physical address of the transfer
+ * source or destination address).
+ */
+ uint64_t server_addr;
+ /*
+ * Informs client the DMA transfer request ready (DMA_XFER command
+ * received from server), ready to proceed (or discard).
+ */
+ struct completion xfer_ready;
+ /*
+ * Informs server DMA transfer is complete, DMA_DONE command received
+ * from client after transfer is complete or aborted. Server can
+ * clean up message and DMA data structures and return to caller.
+ */
+ struct completion xfer_done;
+ /*
+ * DMA request is being aborted, either the remote sent an abort
+ * or locally a service flush or signal was received.
+ */
+ bool aborting;
+ /* DMA transfer error code (-errno) if non-zero */
+ int32_t errcode;
+};
+
+/*
+ * Message metadata, one for each local/outgoing and remote/incoming message
+ * in progress on the local system. Concurrency protection for all fields is
+ * provided through the spinlock for the Easel service for which the message
+ * is created.
+ */
+struct easelcomm_message_metadata {
+ /*
+ * Message reference count, > 0 means kernel code is currently
+ * executing that holds a pointer to this message (metadata), or a
+ * data structure (other than the local/remote message lists)
+ * currently points to this message (namely a reply message pointed to
+ * by the replied-to message). Zero reference count means the message
+ * can be safely removed from the local/remote list and freed (with the
+ * service lock held), which may happen outside the normal message
+ * lifecycle on a service flush.
+ */
+ int reference_count;
+ /*
+ * Set to true when the message is to be freed once the reference count
+ * goes to 0.
+ */
+ bool free_message;
+ /*
+ * True if message is marked to be flushed, lets DMA handling or
+ * reply waiter know to bail and let the message be flushed.
+ */
+ bool flushing;
+ /* what type of message: local, remote reply, remote non-reply */
+ enum easelcomm_msg_type msg_type;
+ /* local or remote message list linkage */
+ struct list_head list;
+ /*
+ * True if message is in the receiveMessage queue (a remote non-reply
+ * not yet picked up by a receiveMessage() call).
+ */
+ bool queued;
+ /* receiveMessage queue list linkage, if message is queued */
+ struct list_head rcvq_list;
+ /* signals reply message received, if message needs a reply */
+ struct completion reply_received;
+ /* pointer to reply message metadata once received, if need reply */
+ struct easelcomm_message_metadata *reply_metadata;
+ /* DMA info, if message includes a DMA transfer */
+ struct easelcomm_dma_xfer_info dma_xfer;
+ /* points to the Easel message desc and data */
+ struct easelcomm_kmsg *msg;
+};
+
+/* forward declare for service/user state cross-references */
+struct easelcomm_user_state;
+
+/* Per-Easel-service data. */
+struct easelcomm_service {
+ /* service ID (same as easelcom_service array index) */
+ unsigned int service_id;
+ /* protects access to all mutable fields below and in messages */
+ struct mutex lock;
+ /* currently registered user */
+ struct easelcomm_user_state *user;
+ /* true if service is being closed by local */
+ bool shutdown_local;
+ /* true if service was closed by remote */
+ bool shutdown_remote;
+ /* next local/outgoing message ID */
+ easelcomm_msgid_t next_id;
+ /* list of in-progress local/outgoing messages */
+ struct list_head local_list;
+ /* remote non-reply messages not yet returned by receiveMessage */
+ struct list_head receivemsg_queue;
+ /* signals new entry added to receivemsg_queue */
+ struct completion receivemsg_queue_new;
+ /* in-progress remote messages (reply and non-reply) */
+ struct list_head remote_list;
+ /* flush service complete on remote, ready to resume service */
+ struct completion flush_done;
+};
+
+/*
+ * Are we running as AP/client or Easel/server? Most decisions are made
+ * at runtime instead of compile time for code prettiness.
+ */
+#ifdef CONFIG_GOOGLE_EASEL_AP
+#define easelcomm_is_client() (true)
+#define easelcomm_is_server() (false)
+#else
+#define easelcomm_is_client() (false)
+#define easelcomm_is_server() (true)
+#endif
+
+/* the easelcomm-{client,server} miscdevice for dev_*() messages */
+extern struct miscdevice easelcomm_miscdev;
+
+/* debug message util prefixes message ID by local or remote marker */
+static inline char *easelcomm_msgid_prefix(
+ struct easelcomm_message_metadata *msg_metadata)
+{
+ return msg_metadata->msg_type == TYPE_LOCAL ? "l" : "r";
+}
+
+/* DMA functions called by main */
+
+/* Handle DMA_SG command */
+extern void easelcomm_handle_cmd_dma_sg(
+ struct easelcomm_service *service, char *command_args,
+ size_t command_arg_len);
+/* Handle DMA_XFER command */
+extern void easelcomm_handle_cmd_dma_xfer(
+ struct easelcomm_service *service, char *command_args,
+ size_t command_arg_len);
+/* Handle DMA_DONE command */
+extern void easelcomm_handle_cmd_dma_done(
+ struct easelcomm_service *service, char *command_args,
+ size_t command_arg_len);
+/* Handle RECVDMA ioctl */
+extern int easelcomm_receive_dma(
+ struct easelcomm_service *service,
+ struct easelcomm_kbuf_desc *buf_desc);
+/* Handle SENDDMA ioctl */
+extern int easelcomm_send_dma(
+ struct easelcomm_service *service,
+ struct easelcomm_kbuf_desc *buf_desc);
+
+/* Main functions called by DMA functions */
+
+/* Drop a reference to a message, optionally mark for freeing */
+extern void easelcomm_drop_reference(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata, bool mark_free);
+/* Find a local/outgoing message by message ID */
+extern struct easelcomm_message_metadata *easelcomm_find_local_message(
+ struct easelcomm_service *service, easelcomm_msgid_t message_id);
+/* Find a remote/incoming message by message ID */
+extern struct easelcomm_message_metadata *easelcomm_find_remote_message(
+ struct easelcomm_service *service, easelcomm_msgid_t message_id);
+/* Start a new command to be sent to remote */
+extern int easelcomm_start_cmd(
+ struct easelcomm_service *service, int command_code,
+ size_t command_arg_len);
+/* Add command arguments to the in-progress command */
+extern int easelcomm_append_cmd_args(
+ struct easelcomm_service *service, void *cmd_args,
+ size_t cmd_args_len);
+/* Finish and send command to remote */
+extern int easelcomm_send_cmd(struct easelcomm_service *service);
+/* Start and send a new command with no arguments */
+extern int easelcomm_send_cmd_noargs(
+ struct easelcomm_service *service, int command_code);
+
+/* Easel MNH coprocessor/PCIe hardware access functions */
+
+/* register IRQ callbacks, etc. */
+extern int easelcomm_hw_init(void);
+/* AP retrieve Easel buffer address and setup EP iATU for both directions */
+extern int easelcomm_hw_ap_setup_cmdchans(void);
+
+/* send command channel data ready interrupt to remote */
+extern int easelcomm_hw_send_data_interrupt(void);
+/* send command channel wraparound interrupt to remote */
+extern int easelcomm_hw_send_wrap_interrupt(void);
+
+/* read remote memory */
+extern int easelcomm_hw_remote_read(
+ void *local_addr, size_t len, uint64_t remote_offset);
+/* write remote memory */
+extern int easelcomm_hw_remote_write(
+ void *local_addr, size_t len, uint64_t remote_offset);
+
+/* build an MNH DMA scatter-gather list */
+extern void *easelcomm_hw_build_scatterlist(
+ struct easelcomm_kbuf_desc *buf_desc,
+ uint32_t *scatterlist_size,
+ void **sglocaldata, enum easelcomm_dma_direction dma_dir);
+/* get block count from SG list, for determing if multi- or single-block */
+extern int easelcomm_hw_scatterlist_block_count(uint32_t scatterlist_size);
+/* get start address from SG list for single-block transfer */
+extern uint64_t easelcomm_hw_scatterlist_sblk_addr(void *sglist);
+/* destroy (and unmap pages pinned by) a scatter-gather list */
+extern void easelcomm_hw_destroy_scatterlist(void *sglocaldata);
+/* sanity check scatterlists */
+extern int easelcomm_hw_verify_scatterlist(
+ struct easelcomm_dma_xfer_info *xfer);
+
+/*
+ * Easel builds multi-block DMA Linked List from local and remote SG lists.
+ *
+ * ll_data returns an opaque pointer passed to other calls.
+ */
+extern int easelcomm_hw_easel_build_ll(
+ void *src_sg, void *dest_sg, void **ll_data);
+/* Easel returns LL DMA address needed for multi-block transfer */
+extern uint64_t easelcomm_hw_easel_ll_addr(void *ll_data);
+/* Easel destroys MBLK DMA Linked List */
+extern int easelcomm_hw_easel_destroy_ll(void *ll_data);
+/* AP performs a single-block DMA transfer */
+extern int easelcomm_hw_ap_dma_sblk_transfer(
+ uint64_t ap_paddr, uint64_t easel_paddr, uint32_t xfer_len,
+ bool to_easel);
+/* AP performs a multi-block DMA transfer */
+extern int easelcomm_hw_ap_dma_mblk_transfer(uint64_t ll_paddr, bool to_easel);
+
+/* callbacks to main code from hardware-specific functions */
+
+/* PCIe ready, local cmdchan buffer alloc'ed, init upper layer */
+int easelcomm_init_pcie_ready(void *local_cmdchan_buffer);
+/* Called when PCIe EP has been hotplug out */
+int easelcomm_pcie_hotplug_out(void);
+/* Handle command channel data received interrupt (MSG_SENT) */
+extern void easelcomm_cmd_channel_data_handler(void);
+/* Handle command channel wrapround interrupt (APPDEFINED_1) */
+extern void easelcomm_cmd_channel_wrap_handler(void);
+/* AP handles BOOTSTRAP_SET interrupt, Easel command channel is ready */
+extern int easelcomm_client_remote_cmdchan_ready_handler(void);
+
+#endif /* _GOOGLE_EASEL_COMM_PRIVATE_H */
diff --git a/drivers/misc/google-easel-comm-shared.h b/drivers/misc/google-easel-comm-shared.h
new file mode 100644
index 0000000..4fbffc2
--- /dev/null
+++ b/drivers/misc/google-easel-comm-shared.h
@@ -0,0 +1,137 @@
+/*
+ * Android/Easel coprocessor communication kernel-layer data structures
+ * agreed upon by the drivers on both sides of the link (AP and Easel).
+ *
+ * Copyright 2016 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef _GOOGLE_EASEL_COMM_SHARED_H
+#define _GOOGLE_EASEL_COMM_SHARED_H
+
+#include <uapi/linux/google-easel-comm.h>
+#include <linux/compiler.h>
+#include <linux/types.h>
+
+/*
+ * Each side of the link allocates a command channel buffer in physically
+ * contiguous memory (such that any portions can be addressed in a single iATU
+ * region or single-block DMA transfer). These are sometimes called
+ * "ringbuffers", but in the current implementation flow control is performed
+ * at each wraparound, effectively implementing a FIFO.
+ *
+ * Each driver uses PCIe remote writes to its peer's buffer to send commands
+ * to the peer, and local memory access in the kernel virtual address space to
+ * read the commands sent from the peer.
+ *
+ * Multi-byte integers in driver-layer data structures are sent across the link
+ * in native format, which is assumed to be little-endian; endian conversion
+ * would be needed for big endian systems. It is assumed structure packing
+ * rules are compatible on both sides of the link.
+ */
+
+/* Remote (Easel) command channel buffer size 1 MB */
+#define EASELCOMM_CMD_CHANNEL_REMOTE_SIZE (1024 * 1024)
+/* Local (AP) command channel buffer size 100 KB */
+#define EASELCOMM_CMD_CHANNEL_LOCAL_SIZE (100 * 1024)
+
+
+/*
+ * Command channel header at the top of each command channel buffer contains
+ * the following info.
+ *
+ * Offset 0: Next producer sequence number (uint64_t 8 bytes Little Endian).
+ * The producer (the remote peer) bumps this value each time a new command is
+ * written to the buffer. The consumer (the local driver) reads and processes
+ * commands out of the buffer until its own "next sequence number" catches up
+ * with this value. The initial value is zero. Sequence numbers cannot be
+ * the CMD_BUFFER_WRAP_MARKER value defined below, which is reserved for
+ * marking the buffer wraparound point.
+ */
+#define EASELCOMM_CMDCHAN_PRODUCER_SEQNBR 0
+
+/* Alternatively can use this data structure for local access */
+struct easelcomm_cmd_channel_header {
+ uint64_t producer_seqnbr_next; /* next cmd seq# to be produced */
+};
+
+/* Size of command channel header in bytes */
+#define EASELCOMM_CMDCHAN_HEADER_SIZE 8
+
+/* easelcomm driver command codes */
+enum {
+ EASELCOMM_CMD_LINK_INIT = 0, /* link partner init */
+ EASELCOMM_CMD_LINK_SHUTDOWN, /* link partner shutdown */
+ EASELCOMM_CMD_ACK_SHUTDOWN, /* link shutdown acknowledge */
+ EASELCOMM_CMD_SEND_MSG, /* send message */
+ EASELCOMM_CMD_DMA_SG, /* DMA scatter-gather list */
+ EASELCOMM_CMD_DMA_XFER, /* DMA transfer request */
+ EASELCOMM_CMD_DMA_DONE, /* inform DMA transfer done */
+ EASELCOMM_CMD_FLUSH_SERVICE, /* flush service messages */
+ EASELCOMM_CMD_FLUSH_SERVICE_DONE, /* flush service done */
+ EASELCOMM_CMD_CLOSE_SERVICE, /* remote service handler closed */
+};
+
+/* Command channel producer wraparound marked by this invalid sequence # */
+#define CMD_BUFFER_WRAP_MARKER (0xFFFFFFFFULL)
+
+/* Command header, precedes command-specific params (if any) */
+struct easelcomm_cmd_header {
+ uint64_t sequence_nbr; /* seq# of command, or wrap marker */
+ uint32_t service_id; /* destination Easel service ID */
+ uint32_t command_code; /* easelcomm driver command code */
+ uint32_t command_arg_len; /* # of bytes of command args that follow */
+};
+
+/* Kernel-layer Easel message descriptor plus appended message data */
+struct easelcomm_kmsg {
+ struct easelcomm_kmsg_desc desc;
+ char message_data; /* message data starts here */
+};
+
+/* SEND_MSG command argument is a struct easelcomm_msg */
+
+/*
+ * DMA direction, used in part to tell whether a mesasge id in a DMA_SG or
+ * DMA_DONE argument refers to a client-allocated ID or a server-
+ * allocated ID, and therefore also whether the associated message is in the
+ * remote/incoming or local/outgoing message list maintained by the local
+ * driver.
+ */
+enum easelcomm_dma_direction {
+ EASELCOMM_DMA_DIR_TO_CLIENT,
+ EASELCOMM_DMA_DIR_TO_SERVER,
+};
+
+/* DMA_SG command argument header */
+struct easelcomm_dma_sg_header {
+ easelcomm_msgid_t message_id; /* which message this SG list is for */
+ uint32_t dma_dir; /* which DMA direction */
+ uint32_t scatterlist_size; /* length of scatterlist that follows */
+};
+/* Above header is followed by a MNH driver scatter-gather list */
+
+/* DMA_XFER command argument */
+enum {
+ EASELCOMM_DMA_XFER_SBLK, /* single-block DMA request */
+ EASELCOMM_DMA_XFER_MBLK, /* multi-block DMA request */
+ EASELCOMM_DMA_XFER_ABORT, /* abort DMA request, error or discard */
+};
+struct easelcomm_dma_xfer_arg {
+ easelcomm_msgid_t message_id; /* local or remote message ID */
+ uint32_t dma_dir; /* DMA direction tells local/remote */
+ uint32_t xfer_type; /* DMA transfer type */
+ uint64_t server_addr; /* MBLK LL addr or SBLK dest paddr */
+};
+
+/* DMA_DONE command argument */
+struct easelcomm_dma_done_arg {
+ easelcomm_msgid_t message_id; /* local or remote message ID */
+ uint32_t dma_dir; /* DMA direction tells local/remote */
+ int32_t errcode; /* DMA error code or zero if OK */
+};
+
+#endif /* _GOOGLE_EASEL_COMM_SHARED_H */
diff --git a/drivers/misc/google-easel-comm.c b/drivers/misc/google-easel-comm.c
new file mode 100644
index 0000000..44b103f
--- /dev/null
+++ b/drivers/misc/google-easel-comm.c
@@ -0,0 +1,2266 @@
+/*
+ * Android/Easel coprocessor communication.
+ *
+ * Copyright 2016 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+/* #define DEBUG */
+
+#include <uapi/linux/google-easel-comm.h>
+#include "google-easel-comm-shared.h"
+#include "google-easel-comm-private.h"
+
+#include <linux/compat.h>
+#include <linux/completion.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/fs.h>
+#include <linux/io.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/miscdevice.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/notifier.h>
+#include <linux/reboot.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/uaccess.h>
+
+/* retry delay in msec for retrying flush of a message being processed */
+#define MSG_FLUSH_RETRY_DELAY 10
+
+/* Service for link status update */
+static struct easelcomm_service link_service;
+
+/* Mutex to guard easelcomm_service_list */
+struct mutex service_mutex;
+
+/* Per-Easel-service state */
+static struct easelcomm_service *easelcomm_service_list[
+ EASELCOMM_SERVICE_COUNT];
+
+/* Local command channel local state */
+struct easelcomm_cmd_channel_local {
+ char *buffer; /* start of command buffer */
+ char *readp; /* ptr to next entry to read */
+ uint64_t consumer_seqnbr_next; /* next cmd seq# to be consumed */
+};
+static struct easelcomm_cmd_channel_local cmd_channel_local;
+
+/* Remote command channel local state */
+struct easelcomm_cmd_channel_remote {
+ /* Protects access to all mutable fields below */
+ struct mutex mutex;
+ /* offset of next entry to write */
+ uint64_t write_offset;
+ /* next command sequence number to write */
+ uint64_t write_seqnbr;
+ /* remote consumer has caught up, ready to wrap channel buffer */
+ struct completion wrap_ready;
+ /* remote channel is initialized */
+ bool initialized;
+ /* channel initialization or link shutdown received from remote */
+ struct completion init_state_changed;
+};
+static struct easelcomm_cmd_channel_remote cmd_channel_remote;
+
+/* Per-file-descriptor (user) state */
+struct easelcomm_user_state {
+ /* Which Easel service is registered for this fd, or NULL if none */
+ struct easelcomm_service *service;
+};
+
+/* max delay in msec waiting for remote to wrap command channel */
+#define CMDCHAN_WRAP_DONE_TIMEOUT_MS 1000
+/* max delay in msec waiting for remote to ack link shutdown */
+#define LINK_SHUTDOWN_ACK_TIMEOUT 500
+/* max delay in msec waiting for remote to return flush done */
+#define SERVICE_FLUSH_DONE_TIMEOUT 5000
+
+static bool easelcomm_up; /* is easelcomm up and running? */
+/* used to wait for remote peer to ack link shutdown */
+static DECLARE_COMPLETION(easelcomm_link_peer_shutdown);
+
+static int easelcomm_open(struct inode *inode, struct file *file);
+static int easelcomm_release(struct inode *inode, struct file *file);
+static long easelcomm_ioctl(
+ struct file *file, unsigned int cmd, unsigned long arg);
+
+#ifdef CONFIG_COMPAT
+/* 32-bit userspace on 64-bit OS conversion defines */
+struct easelcomm_compat_kbuf_desc_legacy {
+ easelcomm_msgid_t message_id; /* ID of message for this transfer */
+ compat_uptr_t __user buf; /* local buffer source or dest */
+ int dma_buf_fd; /* fd of local dma_buf */
+ int buf_type; /* e.g. EASELCOMM_DMA_BUFFER_DMA_BUF */
+ uint32_t buf_size; /* size of the local buffer */
+};
+
+/*
+ * dma_buf* specification example:
+ * ADDR CONTENTS
+ * -------
+ * 0x00 | 0xFF |
+ * 0x01 | 0x0A |
+ * 0x02 | 0x0A |
+ * 0x03 | 0xFF |
+ * 0x04 | 0x0B |
+ * 0x05 | 0x0B |
+ * 0x06 | 0xFF |
+ * 0x07 | 0xFF |
+ * ---------
+ * offset=1 size=4 width=2 stride=3 will transfer: 0A 0A 0B 0B
+ * Note that if we are transferring a simple contiguous block of memory,
+ * then stride == size == width.
+ */
+struct easelcomm_compat_kbuf_desc {
+ easelcomm_msgid_t message_id; /* ID of message for this transfer */
+ compat_uptr_t __user buf; /* local buffer source or dest */
+ int dma_buf_fd; /* fd of local dma_buf */
+ int buf_type; /* e.g. EASELCOMM_DMA_BUFFER_DMA_BUF */
+ uint32_t buf_size; /* size of the local buffer */
+ uint32_t dma_buf_off; /* offset into the local dma_buf */
+ uint32_t dma_buf_width; /* width of region */
+ uint32_t dma_buf_stride; /* stride width */
+};
+
+static long easelcomm_compat_ioctl(
+ struct file *file, unsigned int cmd, unsigned long arg);
+#else
+#define easelcomm_compat_ioctl NULL
+#endif
+
+static const struct file_operations easelcomm_fops = {
+ .owner = THIS_MODULE,
+ .open = easelcomm_open,
+ .release = easelcomm_release,
+ .unlocked_ioctl = easelcomm_ioctl,
+ .compat_ioctl = easelcomm_compat_ioctl,
+ .llseek = no_llseek,
+};
+
+struct miscdevice easelcomm_miscdev = {
+ .minor = MISC_DYNAMIC_MINOR,
+#ifdef CONFIG_GOOGLE_EASEL_AP
+ .name = "easelcomm-client",
+#else
+ .name = "easelcomm-server",
+#endif
+ .fops = &easelcomm_fops,
+};
+EXPORT_SYMBOL(easelcomm_miscdev);
+
+/* Dump message info for debugging. */
+static void easelcomm_dump_message(
+ struct easelcomm_message_metadata *msg_metadata)
+{
+ struct easelcomm_kmsg *msg = msg_metadata->msg;
+
+ dev_dbg(easelcomm_miscdev.this_device, "msg: id=%s%llu msgsize=%u dmasize=%u needreply=%d inreplyto=%llu replycode=%d\n",
+ easelcomm_msgid_prefix(msg_metadata),
+ msg->desc.message_id, msg->desc.message_size,
+ msg->desc.dma_buf_size, msg->desc.need_reply,
+ msg->desc.in_reply_to, msg->desc.replycode);
+}
+
+/*
+ * Add an additional refcount bump on a message for which a refcount is
+ * already held. This function is for use by callers that hold a valid
+ * message pointer for which a refcount is already held, in order to reflect
+ * another reference being created, which will be dropped by another context
+ * later. At present this is used only to hand off incoming reply messages to
+ * waiters, where the additional refcount reflects the fact that the original
+ * message contains a pointer to the reply message.
+ */
+static void easelcomm_grab_reference(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata)
+{
+ if (!msg_metadata)
+ return;
+ mutex_lock(&service->lock);
+ msg_metadata->reference_count++;
+ mutex_unlock(&service->lock);
+}
+
+/*
+ * Find a local message (for the specified service) by ID.
+ * Returns a pointer to the metadata with the reference count bumped.
+ */
+struct easelcomm_message_metadata *easelcomm_find_local_message(
+ struct easelcomm_service *service, easelcomm_msgid_t message_id)
+{
+ struct easelcomm_message_metadata *msg_metadata = NULL;
+ struct easelcomm_message_metadata *msg_cursor;
+
+ mutex_lock(&service->lock);
+ list_for_each_entry(msg_cursor, &service->local_list,
+ list) {
+ if (msg_cursor->msg->desc.message_id == message_id) {
+ msg_metadata = msg_cursor;
+ msg_metadata->reference_count++;
+ break;
+ }
+ }
+ mutex_unlock(&service->lock);
+ return msg_metadata;
+}
+EXPORT_SYMBOL(easelcomm_find_local_message);
+
+/*
+ * Find a remote message (for the specified service) by ID.
+ * Returns a pointer to the metadata with the reference count bumped.
+ */
+struct easelcomm_message_metadata *easelcomm_find_remote_message(
+ struct easelcomm_service *service, easelcomm_msgid_t message_id)
+{
+ struct easelcomm_message_metadata *msg_metadata = NULL;
+ struct easelcomm_message_metadata *msg_cursor;
+
+ mutex_lock(&service->lock);
+ list_for_each_entry(msg_cursor, &service->remote_list, list) {
+ if (msg_cursor->msg->desc.message_id == message_id) {
+ msg_metadata = msg_cursor;
+ msg_metadata->reference_count++;
+ break;
+ }
+ }
+ mutex_unlock(&service->lock);
+ return msg_metadata;
+}
+EXPORT_SYMBOL(easelcomm_find_remote_message);
+
+/*
+ * Return a pointer to the reply message for the specified original message.
+ * Responsibility for the existing reference count held for the reply message
+ * (due to the linkage from the original message) now transfers to the caller,
+ * which must drop the reference when appropriate.
+ */
+static struct easelcomm_message_metadata *easelcomm_grab_reply_message(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *orig_msg_metadata)
+{
+ struct easelcomm_message_metadata *reply_msg_metadata;
+
+ if (!orig_msg_metadata)
+ return NULL;
+ mutex_lock(&service->lock);
+ reply_msg_metadata = orig_msg_metadata->reply_metadata;
+ orig_msg_metadata->reply_metadata = NULL;
+ mutex_unlock(&service->lock);
+ return reply_msg_metadata;
+}
+
+/*
+ * Add a new message and metadata for the specified service. Depending on the
+ * message type, the message will be placed in the local or remote list and
+ * potentially the receiveMessage queue. The reference count for the message
+ * starts at 1 due to the returned pointer.
+ */
+static struct easelcomm_message_metadata *easelcomm_add_metadata(
+ struct easelcomm_service *service,
+ enum easelcomm_msg_type msg_type, struct easelcomm_kmsg *msg)
+{
+ struct easelcomm_message_metadata *msg_metadata =
+ kmalloc(sizeof(struct easelcomm_message_metadata), GFP_KERNEL);
+ if (!msg_metadata)
+ return NULL;
+ msg_metadata->msg_type = msg_type;
+ INIT_LIST_HEAD(&msg_metadata->list);
+ msg_metadata->dma_xfer.sg_local = NULL;
+ msg_metadata->dma_xfer.sg_local_size = 0;
+ msg_metadata->dma_xfer.sg_remote = NULL;
+ msg_metadata->dma_xfer.sg_remote_size = 0;
+ init_completion(&msg_metadata->dma_xfer.sg_remote_ready);
+ init_completion(&msg_metadata->dma_xfer.xfer_ready);
+ init_completion(&msg_metadata->dma_xfer.xfer_done);
+ msg_metadata->dma_xfer.aborting = false;
+ msg_metadata->msg = msg;
+ msg_metadata->reference_count = 1;
+ msg_metadata->queued = false;
+ msg_metadata->flushing = false;
+ msg_metadata->free_message = false;
+ init_completion(&msg_metadata->reply_received);
+ msg_metadata->reply_metadata = NULL;
+
+ mutex_lock(&service->lock);
+ switch (msg_metadata->msg_type) {
+ case TYPE_LOCAL:
+ /* add to the local list */
+ list_add_tail(&msg_metadata->list, &service->local_list);
+ break;
+ case TYPE_REMOTE_REPLY:
+ case TYPE_REMOTE_NONREPLY:
+ /* add to the remote list */
+ list_add_tail(&msg_metadata->list, &service->remote_list);
+
+ if (msg_metadata->msg_type == TYPE_REMOTE_NONREPLY) {
+ /*
+ * Also add to the receiveMessage queue and wakeup
+ * waiter.
+ */
+ list_add_tail(&msg_metadata->rcvq_list,
+ &service->receivemsg_queue);
+ msg_metadata->queued = true;
+ complete(&service->receivemsg_queue_new);
+ }
+ break;
+ default:
+ WARN_ON(1);
+ }
+ mutex_unlock(&service->lock);
+ return msg_metadata;
+}
+
+/*
+ * Free message, remove from lists and receiveMessage queue. Called when
+ * message reference count drops to zero, and either: the message is marked
+ * for freeing when no longer referenced, or: the message is being flushed.
+ * The lock for the containing service must be held. This method is only for
+ * use by reference counting and message flushing routines, not for general use.
+ */
+static void easelcomm_free_message(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata)
+{
+ if (!msg_metadata)
+ return;
+
+ WARN_ON(msg_metadata->reference_count);
+ /* remove from local or remote list */
+ list_del(&msg_metadata->list);
+ /* If queued remove from receive queue */
+ if (msg_metadata->queued)
+ list_del(&msg_metadata->rcvq_list);
+ kfree(msg_metadata->msg);
+ vfree(msg_metadata->dma_xfer.sg_local); /* was allocated by vmalloc */
+ kfree(msg_metadata->dma_xfer.sg_remote);
+ kfree(msg_metadata);
+}
+
+/*
+ * Decrement message refcount, optionally mark for freeing. The message is
+ * freed if it is now, or has previously been, marked for freeing when
+ * unreferenced and the reference count is dropped to zero. Once this method
+ * is called the supplied message pointer is no longer valid and must not be
+ * dereferenced. Any code that needs to reference the same message again
+ * needs to find it by message ID.
+ *
+ * A reference count must be held whenever a pointer to the message is created
+ * and valid (apart from local or remote message list linkage, which is always
+ * present until the message is freed). (And except briefly while maintaining
+ * message state and the service lock is held.) A reference count of zero
+ * indicates the message is not actively being manipulated by a kernel code
+ * path and no pointers to it existing apart from the local or remote message
+ * list for its associated service -- if not in the process of being freed then
+ * it is expected something will later grab a reference to the message and
+ * resume handling it.
+ */
+void easelcomm_drop_reference(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata, bool mark_free)
+{
+ if (!msg_metadata)
+ return;
+
+ mutex_lock(&service->lock);
+ if (!WARN_ON(!msg_metadata->reference_count))
+ msg_metadata->reference_count--;
+
+ if (mark_free)
+ msg_metadata->free_message = true;
+
+ if (!msg_metadata->reference_count && msg_metadata->free_message)
+ easelcomm_free_message(service, msg_metadata);
+ mutex_unlock(&service->lock);
+}
+EXPORT_SYMBOL(easelcomm_drop_reference);
+
+/*
+ * Attempt to flush a message. If a reference count is held on the message or
+ * the DMA transfer for the message is in progress then the message is not
+ * flushed. If a DMA transfer is active then a DMA abort is initiated.
+ * Function returns true if need to retry flushing the message later due to
+ * the above, or false if now flushed.
+ */
+static bool easelcomm_flush_message(
+ struct easelcomm_service *service,
+ struct easelcomm_message_metadata *msg_metadata)
+{
+ easelcomm_msgid_t msgid;
+ bool ret = false;
+
+ if (!msg_metadata || !msg_metadata->msg)
+ return false;
+
+ msgid = msg_metadata->msg->desc.message_id;
+ /* Mark message for flushing in case message is still in progress */
+ msg_metadata->flushing = true;
+
+ /*
+ * If we may be working on a DMA transfer then trigger an abort. If
+ * a local process is working on it then a refcount will be held, and
+ * we'll need to retry after dropping the mutex, letting the
+ * process handle the DMA abort and then drop the refcount.
+ */
+ if (msg_metadata->msg->desc.dma_buf_size &&
+ !msg_metadata->dma_xfer.aborting) {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "flush abort DMA msg %u:%s%llu\n",
+ service->service_id,
+ easelcomm_msgid_prefix(msg_metadata), msgid);
+ msg_metadata->dma_xfer.aborting = true;
+ complete(&msg_metadata->dma_xfer.sg_remote_ready);
+ complete(&msg_metadata->dma_xfer.xfer_ready);
+ complete(&msg_metadata->dma_xfer.xfer_done);
+ }
+
+ if (msg_metadata->reference_count) {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "not flushing msg %u:%s%llu refcnt=%d\n",
+ service->service_id,
+ easelcomm_msgid_prefix(msg_metadata), msgid,
+ msg_metadata->reference_count);
+ ret = true;
+ } else {
+ easelcomm_free_message(service, msg_metadata);
+ dev_dbg(easelcomm_miscdev.this_device,
+ "flushed msg %u:%s%llu\n",
+ service->service_id,
+ easelcomm_msgid_prefix(msg_metadata), msgid);
+ ret = false;
+ }
+
+ return ret;
+}
+
+/* Flush all messages (local or remote) kept locally for the service. */
+static void easelcomm_flush_local_service(struct easelcomm_service *service)
+{
+ struct easelcomm_message_metadata *msg_cursor;
+ struct easelcomm_message_metadata *msg_temp;
+ int i;
+ bool need_retry = true;
+
+#define MAX_FLUSH_TRIES 4
+
+ for (i = 0; need_retry && i < MAX_FLUSH_TRIES; i++) {
+ mutex_lock(&service->lock);
+ need_retry = false;
+ list_for_each_entry_safe(
+ msg_cursor, msg_temp, &service->local_list, list) {
+ need_retry |= easelcomm_flush_message(
+ service, msg_cursor);
+ }
+ list_for_each_entry_safe(
+ msg_cursor, msg_temp, &service->remote_list, list) {
+ need_retry |= easelcomm_flush_message(
+ service, msg_cursor);
+ }
+ mutex_unlock(&service->lock);
+
+ if (!need_retry)
+ break;
+
+ /* sleep, let refcount holders abort DMA/reply wait */
+ msleep(MSG_FLUSH_RETRY_DELAY);
+ }
+
+ if (need_retry)
+ dev_err(easelcomm_miscdev.this_device,
+ "svc %u failed to flush messages after %d tries\n",
+ service->service_id, MAX_FLUSH_TRIES);
+}
+
+/*
+ * Wake up any message of a service waiting for reply.
+ */
+static void __force_complete_reply_waiter(struct easelcomm_service *service)
+{
+ struct easelcomm_message_metadata *msg_cursor;
+
+ mutex_lock(&service->lock);
+ list_for_each_entry(msg_cursor, &service->local_list, list) {
+ if (msg_cursor->msg->desc.need_reply)
+ complete(&msg_cursor->reply_received);
+ }
+ mutex_unlock(&service->lock);
+}
+
+/*
+ * Handle local-side processing of shutting down an Easel service. Easel
+ * services are shutdown when the local side handling process closes its fd
+ * or issues the shutdown ioctl, or the remote side sends a CLOSE_SERVICE
+ * command (meaning the remote side is doing one of those things).
+ *
+ * Flush local state if local side initiated the shutdown, wakeup the
+ * receiveMessage() waiter.
+ */
+static void easelcomm_handle_service_shutdown(
+ struct easelcomm_service *service, bool shutdown_local)
+{
+ dev_dbg(easelcomm_miscdev.this_device, "svc %u shutdown from %s\n",
+ service->service_id, shutdown_local ? "local" : "remote");
+
+ /*
+ * If local side shutting down then flush local state, just in case.
+ * If remote is closing then local side may still have messages in
+ * progress, just signal the receivemessage() caller to return
+ * shutdown status.
+ */
+ if (shutdown_local)
+ easelcomm_flush_local_service(service);
+
+ mutex_lock(&service->lock);
+ if (shutdown_local)
+ service->shutdown_local = true;
+ else
+ service->shutdown_remote = true;
+ mutex_unlock(&service->lock);
+ /* Wakeup any receiveMessage() waiter so they can return to user */
+ complete(&service->receivemsg_queue_new);
+
+ /* Wakeup flush_done waiter so they can return to user */
+ complete(&service->flush_done);
+
+ /* Wakeup any sendMessageReceiveReply() waiter to return to user */
+ __force_complete_reply_waiter(service);
+}
+
+/*
+ * Shutdown easelcomm local activity, mark link down.
+ */
+static void easelcomm_stop_local(void)
+{
+ int i;
+
+ if (!easelcomm_up)
+ return;
+
+ dev_dbg(easelcomm_miscdev.this_device, "stopping\n");
+ /* Set local shutdown flag, disallow further activity */
+ easelcomm_up = false;
+
+ for (i = 0; i < EASELCOMM_SERVICE_COUNT; i++) {
+ struct easelcomm_service *service = easelcomm_service_list[i];
+
+ if (service != NULL)
+ easelcomm_handle_service_shutdown(service, true);
+ }
+}
+
+/*
+ * Shutdown easelcomm communications. async is true if emergency shutdown (on
+ * panic), don't wait for sync from other side.
+ */
+static void easelcomm_stop(bool async)
+{
+ int ret;
+
+ if (!easelcomm_up)
+ return;
+
+ easelcomm_send_cmd_noargs(
+ &link_service,
+ EASELCOMM_CMD_LINK_SHUTDOWN);
+ if (!async) {
+ ret = wait_for_completion_interruptible_timeout(
+ &easelcomm_link_peer_shutdown,
+ msecs_to_jiffies(LINK_SHUTDOWN_ACK_TIMEOUT));
+ if (ret <= 0)
+ dev_warn(easelcomm_miscdev.this_device,
+ "error or timeout on peer link shutdown ack\n");
+ }
+ easelcomm_stop_local();
+}
+
+/*
+ * LINK_SHUTDOWN command received from remote, stop local side and send
+ * ACK_SHUTDOWN back to remote.
+ */
+static void easelcomm_handle_cmd_link_shutdown(void)
+{
+ dev_dbg(easelcomm_miscdev.this_device,
+ "recv cmd LINK_SHUTDOWN\n");
+ easelcomm_stop_local();
+ dev_dbg(easelcomm_miscdev.this_device,
+ "send cmd ACK_SHUTDOWN\n");
+ easelcomm_send_cmd_noargs(
+ &link_service,
+ EASELCOMM_CMD_ACK_SHUTDOWN);
+}
+
+/* ACK_SHUTDOWN command received from remote, wakeup waiter. */
+static void easelcomm_handle_cmd_ack_shutdown(void)
+{
+ dev_dbg(easelcomm_miscdev.this_device,
+ "recv cmd ACK_SHUTDOWN\n");
+ complete(&easelcomm_link_peer_shutdown);
+}
+
+/*
+ * Handle SEND_MSG command from remote. Add a remote message, link a reply
+ * to original message and wakeup waiter.
+ */
+static void easelcomm_handle_cmd_send_msg(
+ struct easelcomm_service *service, char *command_args,
+ size_t command_arg_len)
+{
+ struct easelcomm_kmsg *cmd_msg;
+ struct easelcomm_kmsg *new_msg;
+ enum easelcomm_msg_type msg_type;
+ struct easelcomm_message_metadata *msg_metadata;
+ bool discard_message = false;
+
+ if (WARN_ON(command_arg_len < sizeof(struct easelcomm_kmsg_desc)))
+ return;
+ cmd_msg = (struct easelcomm_kmsg *) command_args;
+ if (WARN_ON(command_arg_len != sizeof(struct easelcomm_kmsg_desc) +
+ cmd_msg->desc.message_size))
+ return;
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "recv cmd SEND_MSG msg %u:r%llu\n",
+ service->service_id, cmd_msg->desc.message_id);
+
+ new_msg = kmalloc(command_arg_len, GFP_KERNEL);
+ if (WARN_ON(!new_msg))
+ return;
+ memcpy(new_msg, cmd_msg, command_arg_len);
+ msg_type = new_msg->desc.in_reply_to ?
+ TYPE_REMOTE_REPLY : TYPE_REMOTE_NONREPLY;
+ msg_metadata = easelcomm_add_metadata(service, msg_type, new_msg);
+ if (WARN_ON(!msg_metadata)) {
+ kfree(new_msg);
+ return;
+ }
+ easelcomm_dump_message(msg_metadata);
+
+ /* If a reply then link to the original message and wakeup waiter. */
+ if (msg_type == TYPE_REMOTE_REPLY) {
+ struct easelcomm_message_metadata *orig_msg_metadata =
+ easelcomm_find_local_message(
+ service, msg_metadata->msg->desc.in_reply_to);
+
+ if (orig_msg_metadata) {
+ orig_msg_metadata->reply_metadata =
+ msg_metadata;
+ /*
+ * Bump reference count for this new ref, waiter will
+ * drop.
+ */
+ easelcomm_grab_reference(service, msg_metadata);
+ complete(&orig_msg_metadata->reply_received);
+ easelcomm_drop_reference(
+ service, orig_msg_metadata, false);
+ } else {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "reply msg %u:r%llu reply-to msg %u:l%llu not found\n",
+ service->service_id,
+ msg_metadata->msg->desc.message_id,
+ service->service_id,
+ msg_metadata->msg->desc.in_reply_to);
+ discard_message = true;
+ }
+ }
+
+ easelcomm_drop_reference(service, msg_metadata, discard_message);
+}
+
+/*
+ * Remote side of link indicates command channel ready. Both server and
+ * client call this when handling any command.
+ * In addition, server calls this when a LINK_INIT command is received from
+ * the client.
+ *
+ * Init stuff and wakeup anybody waiting for the channel to be ready (to send
+ * a command).
+ */
+static void easelcomm_cmd_channel_remote_set_ready(void)
+{
+ struct easelcomm_cmd_channel_remote *channel =
+ &cmd_channel_remote;
+
+ mutex_lock(&channel->mutex);
+ /* Return right away if channel is already initialized */
+ if (channel->initialized) {
+ mutex_unlock(&channel->mutex);
+ return;
+ }
+ /* Next write offset starts at top after header, cmd seq# zero */
+ channel->write_offset =
+ (sizeof(struct easelcomm_cmd_channel_header) + 7) & ~0x7;
+ channel->write_seqnbr = 0;
+ /* mark channel initialized, wakeup waiters. */
+ channel->initialized = true;
+ complete_all(&channel->init_state_changed);
+ easelcomm_up = true;
+ mutex_unlock(&channel->mutex);
+}
+
+/*
+ * LINK_INIT command received from client, init local state for remote command
+ * channel.
+ */
+static void easelcomm_handle_cmd_link_init(
+ char *command_args, size_t command_arg_len)
+{
+ dev_dbg(easelcomm_miscdev.this_device, "recv cmd LINK_INIT\n");
+ easelcomm_cmd_channel_remote_set_ready();
+}
+
+/*
+ * Wait for remote command channel ready and initialized, prior to sending a
+ * command. Returns 0 if command channel ready and channel mutex is held, or
+ * non-zero if signal received while waiting, channel mutex not held.
+ */
+static int easelcomm_wait_channel_initialized(
+ struct easelcomm_cmd_channel_remote *channel)
+{
+ int ret = 0;
+
+ mutex_lock(&channel->mutex);
+ while (!channel->initialized) {
+ mutex_unlock(&channel->mutex);
+ ret = wait_for_completion_interruptible(
+ &channel->init_state_changed);
+ if (ret)
+ return ret;
+ mutex_lock(&channel->mutex);
+ }
+
+ return ret;
+}
+
+/* CLOSE_SERVICE command received from remote, shut down service. */
+static void easelcomm_handle_cmd_close_service(
+ struct easelcomm_service *service)
+{
+ dev_dbg(easelcomm_miscdev.this_device,
+ "recv cmd CLOSE_SERVICE svc %u\n",
+ service->service_id);
+ easelcomm_handle_service_shutdown(service, false);
+}
+
+/*
+ * FLUSH_SERVICE command received from remote, flush local state for service
+ * and send FLUSH_SERVICE_DONE back to remote.
+ */
+static void easelcomm_handle_cmd_flush_service(
+ struct easelcomm_service *service)
+{
+ dev_dbg(easelcomm_miscdev.this_device,
+ "recv cmd FLUSH_SERVICE svc %u\n",
+ service->service_id);
+ easelcomm_flush_local_service(service);
+ dev_dbg(easelcomm_miscdev.this_device,
+ "send cmd FLUSH_SERVICE_DONE svc %u\n",
+ service->service_id);
+ easelcomm_send_cmd_noargs(service, EASELCOMM_CMD_FLUSH_SERVICE_DONE);
+}
+
+/* FLUSH_SERVICE_DONE command received from remote, wakeup waiter. */
+static void easelcomm_handle_cmd_flush_service_done(
+ struct easelcomm_service *service)
+{
+ dev_dbg(easelcomm_miscdev.this_device,
+ "recv cmd FLUSH_SERVICE_DONE svc %u\n", service->service_id);
+ complete(&service->flush_done);
+}
+
+static void easelcomm_init_service(struct easelcomm_service *service, int id)
+{
+ if (service == NULL) {
+ dev_err(easelcomm_miscdev.this_device, "service is null\n");
+ return;
+ }
+ service->service_id = id;
+ service->user = NULL;
+ service->shutdown_local = false;
+ service->shutdown_remote = false;
+ mutex_init(&service->lock);
+ INIT_LIST_HEAD(&service->local_list);
+ INIT_LIST_HEAD(&service->receivemsg_queue);
+ init_completion(&service->receivemsg_queue_new);
+ INIT_LIST_HEAD(&service->remote_list);
+ init_completion(&service->flush_done);
+}
+
+/* Caller of this function must hold service_mutex. */
+static struct easelcomm_service *easelcomm_create_service(int id)
+{
+ struct easelcomm_service *service;
+
+ if (easelcomm_service_list[id] != NULL)
+ return easelcomm_service_list[id];
+
+ service = kzalloc(sizeof(struct easelcomm_service), GFP_KERNEL);
+ if (service == NULL) {
+ dev_err(easelcomm_miscdev.this_device,
+ "could not allocate service %u\n", id);
+ return NULL;
+ }
+ easelcomm_init_service(service, id);
+ easelcomm_service_list[id] = service;
+ return service;
+}
+
+/* Command received from remote, dispatch. */
+static void easelcomm_handle_command(struct easelcomm_cmd_header *cmdhdr)
+{
+ struct easelcomm_service *service;
+ char *cmdargs = (char *)cmdhdr + sizeof(struct easelcomm_cmd_header);
+
+ /*
+ * Any command can inform the server that remote is ready, not just
+ * LINK_INIT.
+ */
+ if (!easelcomm_is_client())
+ easelcomm_cmd_channel_remote_set_ready();
+
+ switch(cmdhdr->command_code) {
+ case EASELCOMM_CMD_LINK_INIT:
+ easelcomm_handle_cmd_link_init(
+ cmdargs, cmdhdr->command_arg_len);
+ return;
+ case EASELCOMM_CMD_LINK_SHUTDOWN:
+ easelcomm_handle_cmd_link_shutdown();
+ return;
+ case EASELCOMM_CMD_ACK_SHUTDOWN:
+ easelcomm_handle_cmd_ack_shutdown();
+ return;
+ default:
+ break;
+ }
+
+ if (cmdhdr->service_id >= EASELCOMM_SERVICE_COUNT) {
+ dev_err(easelcomm_miscdev.this_device,
+ "invalid service ID %u received\n",
+ cmdhdr->service_id);
+ return;
+ }
+ service = easelcomm_service_list[cmdhdr->service_id];
+
+ /*
+ * If service has not been initialized,
+ * initialize the service here without userspace ownership.
+ * Userspace ownership could be claimed later.
+ */
+ if (service == NULL) {
+ mutex_lock(&service_mutex);
+ service = easelcomm_create_service(cmdhdr->service_id);
+ mutex_unlock(&service_mutex);
+ if (service == NULL) {
+ dev_err(easelcomm_miscdev.this_device,
+ "could not handle cmd %d for service %u\n",
+ cmdhdr->command_code, cmdhdr->service_id);
+ return;
+ }
+ }
+
+ switch (cmdhdr->command_code) {
+ case EASELCOMM_CMD_SEND_MSG:
+ easelcomm_handle_cmd_send_msg(
+ service, cmdargs, cmdhdr->command_arg_len);
+ break;
+ case EASELCOMM_CMD_DMA_SG:
+ easelcomm_handle_cmd_dma_sg(
+ service, cmdargs, cmdhdr->command_arg_len);
+ break;
+ case EASELCOMM_CMD_DMA_XFER:
+ easelcomm_handle_cmd_dma_xfer(
+ service, cmdargs, cmdhdr->command_arg_len);
+ break;
+ case EASELCOMM_CMD_DMA_DONE:
+ easelcomm_handle_cmd_dma_done(
+ service, cmdargs, cmdhdr->command_arg_len);
+ break;
+ case EASELCOMM_CMD_FLUSH_SERVICE:
+ easelcomm_handle_cmd_flush_service(service);
+ break;
+ case EASELCOMM_CMD_FLUSH_SERVICE_DONE:
+ easelcomm_handle_cmd_flush_service_done(service);
+ break;
+ case EASELCOMM_CMD_CLOSE_SERVICE:
+ easelcomm_handle_cmd_close_service(service);
+ break;
+ default:
+ dev_err(easelcomm_miscdev.this_device,
+ "svc %u invalid command code %u received\n",
+ cmdhdr->service_id, cmdhdr->command_code);
+ }
+}
+
+/*
+ * Bump the consumer sequence number in the local command channel state,
+ * indicating that we've consumed messages prior to the new number in
+ * sequence. The sequence number cannot be the reserved "command buffer
+ * wrapped" marker; bump twice if needed to avoid.
+ *
+ * Not locked, but there's only one thread handling local command channel
+ * data, in worker context.
+ */
+static void easelcomm_cmd_channel_bump_consumer_seqnbr(
+ struct easelcomm_cmd_channel_local *channel)
+{
+ channel->consumer_seqnbr_next++;
+
+ if (channel->consumer_seqnbr_next == CMD_BUFFER_WRAP_MARKER)
+ channel->consumer_seqnbr_next++;
+}
+
+/*
+ * Interrupt for new local command channel data ready was received. This
+ * function is called on a workqueue worker. Process any new command channel
+ * data found.
+ *
+ * Not locked, but there's only one thread handling local command channel
+ * data, in worker context.
+ */
+void easelcomm_cmd_channel_data_handler(void)
+{
+ struct easelcomm_cmd_channel_local *channel =
+ &cmd_channel_local;
+ struct easelcomm_cmd_channel_header *channel_buf_hdr =
+ (struct easelcomm_cmd_channel_header *)
+ channel->buffer;
+
+ /* While we haven't caught up to producer in sequence #s processed. */
+ while (channel->consumer_seqnbr_next !=
+ channel_buf_hdr->producer_seqnbr_next) {
+ struct easelcomm_cmd_header *cmdhdr =
+ (struct easelcomm_cmd_header *)channel->readp;
+ uint32_t saved_cmd_len;
+
+ dev_dbg(easelcomm_miscdev.this_device, "cmdchan consumer loop prodseq=%llu consseq=%llu off=%lx\n",
+ channel_buf_hdr->producer_seqnbr_next,
+ channel->consumer_seqnbr_next,
+ channel->readp - channel->buffer);
+
+ /*
+ * If producer dropped a wrap marker at the current position
+ * then wrap our read pointer and let the producer know we
+ * wrapped and both sides are ready to continue at the top of
+ * the buffer.
+ */
+ if (cmdhdr->sequence_nbr == CMD_BUFFER_WRAP_MARKER) {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "cmdchan consumer wrap at off=%lx\n",
+ channel->readp - channel->buffer);
+ channel->readp =
+ channel->buffer +
+ sizeof(struct easelcomm_cmd_channel_header);
+ /* Send consumer wrapped IRQ to remote */
+ easelcomm_hw_send_wrap_interrupt();
+ /* Wrapping consumes a seqnbr */
+ easelcomm_cmd_channel_bump_consumer_seqnbr(channel);
+ continue;
+ }
+
+ saved_cmd_len = READ_ONCE(cmdhdr->command_arg_len);
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "cmdchan recv cmd seq=%llu svc=%u cmd=%u len=%u off=%lx\n",
+ cmdhdr->sequence_nbr, cmdhdr->service_id,
+ cmdhdr->command_code, saved_cmd_len,
+ channel->readp - channel->buffer);
+ if (sizeof(struct easelcomm_cmd_header) +
+ saved_cmd_len >
+ EASELCOMM_CMD_CHANNEL_LOCAL_SIZE) {
+ dev_err(easelcomm_miscdev.this_device,
+ "command channel corruption detected: seq=%llu svc=%u cmd=%u len=%u off=%lx\n",
+ cmdhdr->sequence_nbr, cmdhdr->service_id,
+ cmdhdr->command_code, saved_cmd_len,
+ channel->readp - channel->buffer);
+ break;
+ }
+
+ if (cmdhdr->sequence_nbr !=
+ channel->consumer_seqnbr_next) {
+ dev_err(easelcomm_miscdev.this_device,
+ "command channel corruption detected: expected seq# %llu, got %llu\n",
+ channel->consumer_seqnbr_next,
+ cmdhdr->sequence_nbr);
+ break;
+ }
+
+ /* Process the command. */
+ easelcomm_handle_command(cmdhdr);
+
+ /* Post-process double check */
+ if (saved_cmd_len != cmdhdr->command_arg_len) {
+ dev_err(easelcomm_miscdev.this_device,
+ "command channel corruption detected: off=%lx expected len %u got %u\n",
+ channel->readp - channel->buffer,
+ saved_cmd_len, cmdhdr->command_arg_len);
+ break;
+ }
+
+ channel->readp += sizeof(struct easelcomm_cmd_header) +
+ saved_cmd_len;
+
+ /* 8-byte-align next entry pointer */
+ if ((uintptr_t)channel->readp & 0x7)
+ channel->readp += 8 - ((uintptr_t)channel->readp & 0x7);
+ easelcomm_cmd_channel_bump_consumer_seqnbr(channel);
+ }
+ dev_dbg(easelcomm_miscdev.this_device, "cmdchan consumer exiting prodseq=%llu consseq=%llu off=%lx\n",
+ channel_buf_hdr->producer_seqnbr_next,
+ channel->consumer_seqnbr_next,
+ channel->readp - channel->buffer);
+}
+EXPORT_SYMBOL(easelcomm_cmd_channel_data_handler);
+
+/*
+ * Interrupt from remote indicating remote has followed our channel buffer
+ * wrap received. This function called in workqueue worker context. Wakeup
+ * the command producer that is waiting on the remote to follow the wrap.
+ */
+void easelcomm_cmd_channel_wrap_handler(void)
+{
+ struct easelcomm_cmd_channel_remote *channel =
+ &cmd_channel_remote;
+ dev_dbg(easelcomm_miscdev.this_device,
+ "cmdchan remote wrap IRQ received\n");
+ complete(&channel->wrap_ready);
+}
+EXPORT_SYMBOL(easelcomm_cmd_channel_wrap_handler);
+
+/* Device file open. Allocate a user state structure. */
+static int easelcomm_open(struct inode *inode, struct file *file)
+{
+ struct easelcomm_user_state *user_state;
+
+ if (!easelcomm_up)
+ return -ENODEV;
+
+ user_state = kzalloc(sizeof(struct easelcomm_user_state), GFP_KERNEL);
+ if (!user_state)
+ return -ENOMEM;
+ file->private_data = user_state;
+ return 0;
+}
+
+/* Initiate service shutdown from fd release or ioctl. */
+static void easelcomm_initiate_service_shutdown(
+ struct easelcomm_service *service)
+{
+ bool already_shutdown;
+
+ mutex_lock(&service->lock);
+ already_shutdown = service->shutdown_local;
+ mutex_unlock(&service->lock);
+
+ if (already_shutdown) {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "svc %u already shutdown\n",
+ service->service_id);
+ return;
+ }
+
+ dev_dbg(easelcomm_miscdev.this_device, "svc %u initiate shutdown\n",
+ service->service_id);
+ easelcomm_handle_service_shutdown(service, true);
+ dev_dbg(easelcomm_miscdev.this_device,
+ "svc %u send cmd CLOSE_SERVICE\n", service->service_id);
+ easelcomm_send_cmd_noargs(service, EASELCOMM_CMD_CLOSE_SERVICE);
+}
+
+/* Device file descriptor release, initiate service shutdown and cleanup */
+static int easelcomm_release(struct inode *inode, struct file *file)
+{
+ struct easelcomm_user_state *user_state = file->private_data;
+
+ if (user_state) {
+ struct easelcomm_service *service = user_state->service;
+
+ if (service) {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "svc %u release\n", service->service_id);
+
+ if (easelcomm_up)
+ easelcomm_initiate_service_shutdown(service);
+ mutex_lock(&service->lock);
+ service->user = NULL;
+ mutex_unlock(&service->lock);
+ }
+
+ kfree(file->private_data);
+ file->private_data = NULL;
+ }
+ return 0;
+}
+
+/*
+ * Bump the producer next sequence number in the remote command channel buffer
+ * header. After this, the remote side can then see that entries prior to
+ * that sequence number are present. The sequence number cannot be the
+ * reserved "command buffer wrapped" marker; bump twice to avoid if needed.
+ *
+ * Call with remote channel producer mutex held.
+ */
+static int easelcomm_cmd_channel_bump_producer_seqnbr(
+ struct easelcomm_cmd_channel_remote *channel)
+{
+ int ret;
+
+ channel->write_seqnbr++;
+ if (channel->write_seqnbr == CMD_BUFFER_WRAP_MARKER)
+ channel->write_seqnbr++;
+ ret = easelcomm_hw_remote_write(
+ &channel->write_seqnbr, sizeof(uint64_t),
+ (uint64_t)EASELCOMM_CMDCHAN_PRODUCER_SEQNBR);
+ return ret;
+}
+
+/*
+ * Wrap around remote's buffer. If producer does not receive a signal that
+ * consumer has wrapped around within predefined time, it will retry sending
+ * wrap command if enabled.
+ *
+ * Note: this retry mechanism is done by writing the wrap marker to the top
+ * of remote's cmd channel, which could, in the worst case, overwrite the
+ * unconsumed command at the top of cmd channel, causing all unconsumed
+ * commands behind it to be discarded.
+ *
+ * Called with remote channel mutex held.
+ * Caller is responsible for releasing the mutex.
+ *
+ * @channel: the pointer to remote channel
+ *
+ * @Return: 0 - sent wrap and remote completed wrapping
+ * negative - on error
+ */
+static int easelcomm_cmd_channel_send_wrap(
+ struct easelcomm_cmd_channel_remote *channel)
+{
+ int ret = 0;
+ int attempted = 0;
+ long remaining;
+ uint64_t wrap_marker = CMD_BUFFER_WRAP_MARKER;
+
+ {
+ if (attempted > 0) {
+ dev_warn(easelcomm_miscdev.this_device,
+ "%s: retrying after %d attempts",
+ __func__, attempted);
+ }
+ attempted++;
+
+ /* Write the "buffer wrapped" marker in place of sequence # */
+ dev_dbg(easelcomm_miscdev.this_device, "cmdchan producer wrap at off=%llx seq=%llu\n",
+ channel->write_offset, channel->write_seqnbr);
+ ret = easelcomm_hw_remote_write(
+ &wrap_marker, sizeof(wrap_marker),
+ channel->write_offset);
+ if (ret)
+ goto exit;
+
+ /*
+ * Bump the producer seqnbr for the wrap marker (so consumer
+ * knows its there) and send IRQ to remote to consume the new
+ * data.
+ *
+ * Consumer will process entries up to and including the wrap
+ * marker and then send a "wrapped" interrupt that wakes up
+ * the wrap_ready completion.
+ */
+ ret = easelcomm_cmd_channel_bump_producer_seqnbr(channel);
+ if (ret)
+ goto exit;
+ ret = easelcomm_hw_send_data_interrupt();
+ if (ret)
+ goto exit;
+
+ channel->write_offset =
+ (sizeof(struct easelcomm_cmd_channel_header) + 7)
+ & ~0x7;
+ /* Wait for remote to catch up. IRQ from remote wakes this. */
+ remaining = wait_for_completion_interruptible_timeout(
+ &channel->wrap_ready,
+ msecs_to_jiffies(CMDCHAN_WRAP_DONE_TIMEOUT_MS));
+ if (remaining > 0) {
+ ret = 0;
+ dev_dbg(easelcomm_miscdev.this_device,
+ "cmdchan wrap completed, new off=%llx\n",
+ channel->write_offset);
+ goto exit;
+ }
+ if (remaining < 0) {
+ /* waiting was interrupted */
+ dev_err(easelcomm_miscdev.this_device,
+ "remote channel did not catch up: reason interrupted off=%llx seq=%llu\n",
+ channel->write_offset, channel->write_seqnbr);
+ ret = remaining;
+ goto exit;
+ }
+ /* remaining jiffies is 0; timed out */
+ dev_err(easelcomm_miscdev.this_device,
+ "remote channel did not catch up: reason timeout off=%llx seq=%llu\n",
+ channel->write_offset, channel->write_seqnbr);
+ ret = -ETIMEDOUT;
+ }
+
+exit:
+ return ret;
+}
+
+/*
+ * Start the process of writing a new command to the remote command channel.
+ * Wait for channel init'ed and grab the mutex. Wraparound the buffer if
+ * needed. Write the command header.
+ * If no error then return zero and channel mutex is locked on return.
+ * If error then return non-zero and channel mutex is unlocked.
+ */
+int easelcomm_start_cmd(
+ struct easelcomm_service *service, int command_code,
+ size_t command_arg_len)
+{
+ struct easelcomm_cmd_channel_remote *channel =
+ &cmd_channel_remote;
+ struct easelcomm_cmd_header cmdhdr;
+ size_t cmdbuf_size =
+ sizeof(struct easelcomm_cmd_header) + command_arg_len;
+ int ret;
+
+ /*
+ * If never enough room for the command plus an 8-byte wrap marker,
+ * bail.
+ */
+ if (cmdbuf_size > EASELCOMM_CMD_CHANNEL_REMOTE_SIZE - 8 -
+ sizeof(struct easelcomm_cmd_channel_header))
+ return -EINVAL;
+
+ ret = easelcomm_wait_channel_initialized(channel);
+ /* If no error then channel->mutex is locked */
+ if (ret)
+ return ret;
+
+ /* Choose a spot for the new entry, wrap around if needed. */
+ if (channel->write_offset + cmdbuf_size >
+ EASELCOMM_CMD_CHANNEL_REMOTE_SIZE - 8 -
+ sizeof(struct easelcomm_cmd_channel_header)) {
+ ret = easelcomm_cmd_channel_send_wrap(channel);
+ if (ret)
+ goto error;
+ }
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "cmdchan producer sending cmd seq#%llu svc=%u cmd=%u arglen=%zu off=%llx\n",
+ channel->write_seqnbr, service->service_id, command_code,
+ command_arg_len, channel->write_offset);
+ cmdhdr.service_id = service->service_id;
+ cmdhdr.sequence_nbr = channel->write_seqnbr;
+ cmdhdr.command_code = command_code;
+ cmdhdr.command_arg_len = (uint32_t)command_arg_len;
+
+ /*
+ * Send the command header. Subsequent calls to
+ * easelcomm_append_cmd_args() and easelcomm_send_cmd() will finish
+ * the in-progress command.
+ */
+ ret = easelcomm_hw_remote_write(
+ &cmdhdr, sizeof(cmdhdr), channel->write_offset);
+ if (ret)
+ goto error;
+
+ /* Advance write offset */
+ channel->write_offset += sizeof(cmdhdr);
+ return 0;
+
+error:
+ mutex_unlock(&channel->mutex);
+ return ret;
+}
+EXPORT_SYMBOL(easelcomm_start_cmd);
+
+/*
+ * Append arguments to the in-progress command being sent.
+ * Call with remote channel producer mutex held.
+ */
+int easelcomm_append_cmd_args(
+ struct easelcomm_service *service, void *cmd_args,
+ size_t cmd_args_len)
+{
+ struct easelcomm_cmd_channel_remote *channel =
+ &cmd_channel_remote;
+ int ret = easelcomm_hw_remote_write(
+ cmd_args, cmd_args_len, channel->write_offset);
+ if (ret) {
+ mutex_unlock(&channel->mutex);
+ return ret;
+ }
+ channel->write_offset += cmd_args_len;
+ return 0;
+}
+EXPORT_SYMBOL(easelcomm_append_cmd_args);
+
+/* Start and send a command with no arguments. */
+int easelcomm_send_cmd_noargs(
+ struct easelcomm_service *service, int command_code)
+{
+ int ret;
+
+ ret = easelcomm_start_cmd(service, command_code, 0);
+ if (ret)
+ return ret;
+ return easelcomm_send_cmd(service);
+}
+
+/*
+ * Finish sending an in-progress command to the remote.
+ */
+int easelcomm_send_cmd(struct easelcomm_service *service)
+{
+ struct easelcomm_cmd_channel_remote *channel =
+ &cmd_channel_remote;
+ int ret;
+
+ /*
+ * Bump the producer sequence number. Remote may now see and process
+ * the new entry as soon as this value is visible to it.
+ */
+ ret = easelcomm_cmd_channel_bump_producer_seqnbr(channel);
+ if (ret)
+ goto out;
+ /* Send data ready IRQ to remote */
+ ret = easelcomm_hw_send_data_interrupt();
+ if (ret)
+ goto out;
+
+ /* Bump write offset for next command, 8-byte-integer-aligned */
+ if (channel->write_offset & 0x7)
+ channel->write_offset += 8 - (channel->write_offset & 0x7);
+
+out:
+ mutex_unlock(&channel->mutex);
+ return ret;
+}
+EXPORT_SYMBOL(easelcomm_send_cmd);
+
+/*
+ * Return next message ID in sequence for this service. Message ID zero is
+ * not used, bump twice to avoid if needed.
+ */
+static easelcomm_msgid_t easelcomm_next_msgid(
+ struct easelcomm_service *service)
+{
+ easelcomm_msgid_t next_id;
+
+ mutex_lock(&service->lock);
+ next_id = ++service->next_id;
+ if (!next_id)
+ next_id = ++service->next_id;
+ mutex_unlock(&service->lock);
+ return next_id;
+}
+
+/*
+ * Handle SENDMSG ioctl, the message descriptor send part of easelcomm
+ * sendMessage*() and sendReply() calls.
+ */
+static int easelcomm_send_message_ioctl(
+ struct easelcomm_service *service,
+ struct easelcomm_kmsg_desc __user *pmsg_desc)
+{
+ struct easelcomm_kmsg_desc *msg_desc;
+ struct easelcomm_message_metadata *msg_metadata = NULL;
+ int ret = 0;
+
+ msg_desc = kmalloc(sizeof(struct easelcomm_kmsg_desc), GFP_KERNEL);
+ if (!msg_desc)
+ return -ENOMEM;
+ if (copy_from_user(msg_desc, (void __user *) pmsg_desc,
+ sizeof(struct easelcomm_kmsg_desc))) {
+ ret = -EFAULT;
+ goto out_freedesc;
+ }
+ if (msg_desc->message_size > EASELCOMM_MAX_MESSAGE_SIZE) {
+ dev_err(easelcomm_miscdev.this_device,
+ "%s: svc:%u msg size %d is larger than max size %d\n",
+ __func__,
+ service->service_id,
+ msg_desc->message_size,
+ EASELCOMM_MAX_MESSAGE_SIZE);
+ ret = -EINVAL;
+ goto out_freedesc;
+ }
+
+ /* Assign a message ID to the new message and add to local list. */
+ msg_desc->message_id = easelcomm_next_msgid(service);
+ msg_metadata = easelcomm_add_metadata(service, TYPE_LOCAL,
+ (struct easelcomm_kmsg *)msg_desc);
+ if (!msg_metadata) {
+ ret = -ENOMEM;
+ goto out_freedesc;
+ }
+ /*
+ * After a successful call to the above, msg_desc is pointed to by
+ * msg_metadata and the memory is managed along with the other
+ * metadata.
+ */
+
+ dev_dbg(easelcomm_miscdev.this_device, "SENDMSG msg %u:l%llu\n",
+ service->service_id, msg_desc->message_id);
+ easelcomm_dump_message(msg_metadata);
+
+ /* Copy the updated msg desc with message ID to user. */
+ if (copy_to_user((void __user *) pmsg_desc, msg_desc,
+ sizeof(struct easelcomm_kmsg_desc))) {
+ easelcomm_drop_reference(service, msg_metadata, true);
+ return -EFAULT;
+ }
+
+ /*
+ * If this is a reply then we are done with the replied-to remote
+ * message.
+ */
+ if (msg_metadata->msg->desc.in_reply_to) {
+ struct easelcomm_message_metadata *orig_msg_metadata =
+ easelcomm_find_remote_message(
+ service, msg_metadata->msg->desc.in_reply_to);
+
+ if (!orig_msg_metadata) {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "msg %u:l%llu replied-to msg r%llu not found\n",
+ service->service_id, msg_desc->message_id,
+ msg_metadata->msg->desc.in_reply_to);
+ } else {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "msg %u:l%llu is reply to msg r%llu\n",
+ service->service_id, msg_desc->message_id,
+ msg_metadata->msg->desc.in_reply_to);
+ easelcomm_drop_reference(
+ service, orig_msg_metadata, true);
+ }
+ }
+ easelcomm_drop_reference(service, msg_metadata, false);
+ return 0;
+
+out_freedesc:
+ kfree(msg_desc);
+ return ret;
+}
+
+/* Handle RECVMSG ioctl / easelcomm receiveMessage() call. */
+static int easelcomm_wait_message(
+ struct easelcomm_service *service,
+ struct easelcomm_kmsg_desc __user *msgp)
+{
+ struct easelcomm_message_metadata *msg_metadata = NULL;
+ struct easelcomm_kmsg_desc msg_temp;
+ int ret = 0;
+ bool has_timeout = false;
+ long remaining = 0;
+
+ dev_dbg(easelcomm_miscdev.this_device, "WAITMSG svc %u\n",
+ service->service_id);
+
+ if (copy_from_user(&msg_temp, (void __user *) msgp,
+ sizeof(struct easelcomm_kmsg_desc)))
+ return -EFAULT;
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "%s: timeout_ms=%d\n", __func__, msg_temp.wait.timeout_ms);
+
+ if (msg_temp.wait.timeout_ms >= 0) {
+ has_timeout = true;
+ remaining = (long)msecs_to_jiffies(
+ (unsigned int)msg_temp.wait.timeout_ms);
+ dev_dbg(easelcomm_miscdev.this_device, "%s: remaining=%ld\n",
+ __func__, remaining);
+ }
+
+ while (!msg_metadata) {
+ mutex_lock(&service->lock);
+ if (service->shutdown_local ||
+ (easelcomm_is_client() && service->shutdown_remote)) {
+ /* Service shutdown initiated locally or remotely. */
+ dev_dbg(easelcomm_miscdev.this_device,
+ "WAITMSG svc %u returning shutdown local=%d remote=%d\n",
+ service->service_id, service->shutdown_local,
+ service->shutdown_remote);
+ service->shutdown_remote = false;
+ mutex_unlock(&service->lock);
+ return -ESHUTDOWN;
+ }
+ /* grab first entry off receiveMessage queue. */
+ msg_metadata = list_first_entry_or_null(
+ &service->receivemsg_queue,
+ struct easelcomm_message_metadata, rcvq_list);
+ if (msg_metadata)
+ break;
+ mutex_unlock(&service->lock);
+ if (has_timeout) {
+ remaining = wait_for_completion_interruptible_timeout(
+ &service->receivemsg_queue_new,
+ (unsigned long)remaining);
+ if (remaining == 0)
+ ret = -ETIMEDOUT;
+ if (remaining < 0)
+ ret = remaining;
+ } else {
+ ret = wait_for_completion_interruptible(
+ &service->receivemsg_queue_new);
+ }
+ if (ret)
+ return ret;
+ }
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "WAITMSG svc %u returning msg %u:r%llu\n",
+ service->service_id, service->service_id,
+ msg_metadata->msg->desc.message_id);
+ /* remove from receive queue, grab a ref while copy to user */
+ list_del(&msg_metadata->rcvq_list);
+ msg_metadata->queued = false;
+ msg_metadata->reference_count++;
+ mutex_unlock(&service->lock);
+ /* Copy the message desc to the caller */
+ if (copy_to_user(msgp, msg_metadata->msg,
+ sizeof(struct easelcomm_kmsg_desc)))
+ ret = -EFAULT;
+
+ easelcomm_drop_reference(service, msg_metadata, false);
+ return ret;
+}
+
+/*
+ * Handle WAITREPLY ioctl / reply waiting part of easelcomm
+ * sendMessageReceiveReply() call.
+ */
+static int easelcomm_wait_reply(
+ struct easelcomm_service *service,
+ struct easelcomm_kmsg_desc __user *pmsg_desc)
+{
+ struct easelcomm_kmsg_desc orig_msg_desc;
+ struct easelcomm_message_metadata *orig_msg_metadata;
+ struct easelcomm_message_metadata *msg_metadata = NULL;
+ int ret = 0;
+ bool has_timeout = false;
+ long remaining = 0;
+
+ if (copy_from_user(&orig_msg_desc, (void __user *) pmsg_desc,
+ sizeof(struct easelcomm_kmsg_desc)))
+ return -EFAULT;
+
+ dev_dbg(easelcomm_miscdev.this_device, "%s: timeout_ms=%d\n",
+ __func__, orig_msg_desc.wait.timeout_ms);
+
+ if (orig_msg_desc.wait.timeout_ms >= 0) {
+ has_timeout = true;
+ remaining = (long)msecs_to_jiffies(
+ (unsigned int)orig_msg_desc.wait.timeout_ms);
+ dev_dbg(easelcomm_miscdev.this_device, "%s: remaining=%ld\n",
+ __func__, remaining);
+ }
+
+ /* Grab a reference to the original message. */
+ orig_msg_metadata = easelcomm_find_local_message(
+ service, orig_msg_desc.message_id);
+ if (!orig_msg_metadata) {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "WAITREPLY msg %u:l%llu not found\n",
+ service->service_id, orig_msg_desc.message_id);
+ return -EINVAL;
+ }
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "WAITREPLY msg %u:l%llu\n",
+ service->service_id, orig_msg_desc.message_id);
+
+ if (has_timeout) {
+ remaining = wait_for_completion_interruptible_timeout(
+ &orig_msg_metadata->reply_received,
+ (unsigned long)remaining);
+ if (remaining == 0)
+ ret = -ETIMEDOUT;
+ if (remaining < 0)
+ ret = remaining;
+ } else {
+ ret = wait_for_completion_interruptible(
+ &orig_msg_metadata->reply_received);
+ }
+ if (ret)
+ goto out;
+ if (orig_msg_metadata->flushing) {
+ dev_dbg(easelcomm_miscdev.this_device,
+ "WAITREPLY msg %u:l%llu flushed\n",
+ service->service_id, orig_msg_desc.message_id);
+ goto out;
+ }
+
+ msg_metadata = easelcomm_grab_reply_message(
+ service, orig_msg_metadata);
+
+ /*
+ * If completion was waken up on receiving the reply,
+ * msg_metadata really shouldn't be null.
+ * If completion was waken up on receiving shutdown, we should
+ * return to user immediately.
+ */
+ if (!msg_metadata) {
+ dev_err(easelcomm_miscdev.this_device,
+ "WAITREPLY msg %u:l%llu no matching reply\n",
+ service->service_id, orig_msg_desc.message_id);
+ /*
+ * Here we return -ESHUTDOWN. See above comment and
+ * easelcomm_handle_service_shutdown().
+ */
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+
+ /* Copy the reply message descriptor to the caller */
+ if (copy_to_user(pmsg_desc, &msg_metadata->msg->desc,
+ sizeof(struct easelcomm_kmsg_desc))) {
+ ret = -EFAULT;
+ goto out;
+ }
+ dev_dbg(easelcomm_miscdev.this_device,
+ "WAITREPLY msg %u:l%llu returning msg r%llu\n",
+ service->service_id, orig_msg_desc.message_id,
+ msg_metadata->msg->desc.message_id);
+ ret = 0;
+
+out:
+ /* Discard reply message if error retrieving it. */
+ if (msg_metadata)
+ easelcomm_drop_reference(service, msg_metadata, ret);
+ /* Always discard the original message, we're done with it. */
+ easelcomm_drop_reference(service, orig_msg_metadata, true);
+ return ret;
+}
+
+/* Initiate a flush of service on both sides of the link. */
+static void easelcomm_flush_service(struct easelcomm_service *service)
+{
+ int ret;
+
+ easelcomm_flush_local_service(service);
+ dev_dbg(easelcomm_miscdev.this_device,
+ "send cmd FLUSH_SERVICE svc %u\n",
+ service->service_id);
+ ret = easelcomm_send_cmd_noargs(service, EASELCOMM_CMD_FLUSH_SERVICE);
+ if (ret)
+ return;
+ ret = wait_for_completion_interruptible_timeout(
+ &service->flush_done,
+ msecs_to_jiffies(SERVICE_FLUSH_DONE_TIMEOUT));
+ if (ret <= 0)
+ dev_err(easelcomm_miscdev.this_device,
+ "service flush done wait aborted svc %u\n",
+ service->service_id);
+}
+
+/* Handle REGISTER ioctl, register an open fd for an Easel service. */
+static int easelcomm_register(struct easelcomm_user_state *user_state,
+ unsigned int service_id) {
+ struct easelcomm_service *service;
+
+ if (service_id >= EASELCOMM_SERVICE_COUNT)
+ return -EINVAL;
+
+ mutex_lock(&service_mutex);
+ service = easelcomm_service_list[service_id];
+ if (service == NULL) {
+ service = easelcomm_create_service(service_id);
+ if (service == NULL) {
+ mutex_unlock(&service_mutex);
+ dev_err(easelcomm_miscdev.this_device,
+ "could not create service %u\n",
+ service_id);
+ return -ENOMEM;
+ }
+ }
+ mutex_unlock(&service_mutex);
+
+ mutex_lock(&service->lock);
+ if (service->user) {
+ mutex_unlock(&service->lock);
+ return -EBUSY;
+ }
+ user_state->service = service;
+ service->user = user_state;
+ service->shutdown_local = false;
+ service->shutdown_remote = false;
+ mutex_unlock(&service->lock);
+ dev_dbg(easelcomm_miscdev.this_device, "REGISTER svc %u\n",
+ service_id);
+
+ /*
+ * New client flushes any messages generated from activity by previous
+ * clients.
+ *
+ * New server handles any client messages that were waiting for the
+ * server to startup and process, does not flush at start/restart.
+ * Server can explicitly call the flush ioctl if wanted, but with the
+ * understanding any clients starting at the same time may have their
+ * traffic dropped.
+ */
+ if (easelcomm_is_client())
+ easelcomm_flush_service(service);
+ return 0;
+}
+
+/*
+ * Handle READDATA ioctl, the read message data part of easelcomm
+ * receiveMessage() and sendMessageReceiveReply() calls.
+ */
+static int easelcomm_read_msgdata(
+ struct easelcomm_service *service,
+ struct easelcomm_kbuf_desc *buf_desc)
+{
+ struct easelcomm_message_metadata *msg_metadata;
+ int ret = 0;
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "READDATA msg %u:r%llu buf=%p\n",
+ service->service_id, buf_desc->message_id, buf_desc->buf);
+ msg_metadata =
+ easelcomm_find_remote_message(service, buf_desc->message_id);
+ if (!msg_metadata) {
+ dev_err(easelcomm_miscdev.this_device,
+ "READDATA msg %u:r%llu not found\n",
+ service->service_id, buf_desc->message_id);
+ return -EINVAL;
+ }
+
+ if (buf_desc->buf_size &&
+ buf_desc->buf_size != msg_metadata->msg->desc.message_size) {
+ dev_err(easelcomm_miscdev.this_device,
+ "READDATA descriptor buffer size %u doesn't match message %u:r%llu size %u\n",
+ buf_desc->buf_size, service->service_id,
+ buf_desc->message_id,
+ msg_metadata->msg->desc.message_size);
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (buf_desc->buf_size) {
+ if (copy_to_user(buf_desc->buf,
+ &msg_metadata->msg->message_data,
+ buf_desc->buf_size)) {
+ ret = -EFAULT;
+ goto out;
+ }
+ }
+
+ /*
+ * If no DMA transfer and no reply needed then we're done with this
+ * message.
+ */
+ if (!msg_metadata->msg->desc.dma_buf_size &&
+ !msg_metadata->msg->desc.need_reply) {
+ easelcomm_drop_reference(service, msg_metadata, true);
+ return 0;
+ }
+
+out:
+ easelcomm_drop_reference(service, msg_metadata, false);
+ return ret;
+}
+
+/*
+ * Handle WRITEDATA ioctl, the message data write portion of easelcomm
+ * sendMessage*() and sendReply() calls.
+ */
+static int easelcomm_write_msgdata(
+ struct easelcomm_service *service, struct easelcomm_kbuf_desc *buf_desc)
+{
+ struct easelcomm_message_metadata *msg_metadata;
+ char *databuf = NULL;
+ bool discard_message = true;
+ int ret;
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "WRITEDATA msg %u:l%llu buf=%p\n",
+ service->service_id, buf_desc->message_id, buf_desc->buf);
+
+ msg_metadata =
+ easelcomm_find_local_message(service, buf_desc->message_id);
+ if (!msg_metadata) {
+ dev_err(easelcomm_miscdev.this_device,
+ "WRITEDATA msg %u:l%llu not found\n",
+ service->service_id, buf_desc->message_id);
+ return -EINVAL;
+ }
+
+ if (buf_desc->buf_size &&
+ buf_desc->buf_size != msg_metadata->msg->desc.message_size) {
+ dev_err(easelcomm_miscdev.this_device,
+ "WRITEDATA descriptor buffer size %u doesn't match message %u:l%llu size %u\n",
+ buf_desc->buf_size, service->service_id,
+ buf_desc->message_id,
+ msg_metadata->msg->desc.message_size);
+ ret = -EINVAL;
+ goto out;
+ }
+
+ if (buf_desc->buf_size) {
+ databuf = kmalloc(buf_desc->buf_size, GFP_KERNEL);
+ if (!databuf) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ if (copy_from_user(
+ databuf, buf_desc->buf, buf_desc->buf_size)) {
+ ret = -EFAULT;
+ goto out;
+ }
+ }
+
+ dev_dbg(easelcomm_miscdev.this_device,
+ "send cmd SEND_MSG msg %u:l%llu\n",
+ service->service_id, msg_metadata->msg->desc.message_id);
+ ret = easelcomm_start_cmd(
+ service, EASELCOMM_CMD_SEND_MSG,
+ sizeof(struct easelcomm_kmsg_desc) + buf_desc->buf_size);
+ if (ret)
+ goto out;
+ ret = easelcomm_append_cmd_args(
+ service, &msg_metadata->msg->desc,
+ sizeof(struct easelcomm_kmsg_desc));
+ if (ret)
+ goto out;
+ if (buf_desc->buf_size) {
+ ret = easelcomm_append_cmd_args(
+ service, databuf, buf_desc->buf_size);
+ if (ret)
+ goto out;
+ }
+ ret = easelcomm_send_cmd(service);
+ if (ret)
+ goto out;
+
+ /*
+ * If no DMA transfer and no reply needed then we're done with this
+ * message.
+ */
+ discard_message = !msg_metadata->msg->desc.dma_buf_size &&
+ !msg_metadata->msg->desc.need_reply;
+
+out:
+ /*
+ * If success and no DMA transfer and no reply needed, or if error,
+ * then free the message.
+ */
+ easelcomm_drop_reference(service, msg_metadata, discard_message);
+ kfree(databuf);
+ return ret;
+}
+
+/*
+ * Client calls this upon receipt of BOOTSTRAP_READY IRQ
+ * from server. Call MNH host driver to setup iATU mappings for the command
+ * channels and send LINK_INIT.
+ */
+int easelcomm_client_remote_cmdchan_ready_handler(void)
+{
+ int ret;
+
+ /* Sync up client and server sides using MNH host driver APIs */
+ ret = easelcomm_hw_ap_setup_cmdchans();
+ if (ret)
+ return ret;
+ easelcomm_cmd_channel_remote_set_ready();
+ /* Client can now send link init to server */
+ dev_dbg(easelcomm_miscdev.this_device, "send cmd LINK_INIT\n");
+ return easelcomm_send_cmd_noargs(
+ &link_service,
+ EASELCOMM_CMD_LINK_INIT);
+}
+EXPORT_SYMBOL(easelcomm_client_remote_cmdchan_ready_handler);
+
+/*
+ * Handle ioctls that take a struct easelcomm_kbuf_desc * argument, convert
+ * field layout and buf field pointer for 32-bit compat if needed.
+ */
+static int __easelcomm_ioctl_kbuf_desc(
+ struct easelcomm_service *service, unsigned int cmd,
+ struct easelcomm_kbuf_desc *kbuf_desc) {
+ int ret;
+
+ switch (cmd) {
+ case EASELCOMM_IOC_WRITEDATA:
+ case EASELCOMM_IOC_WRITEDATA_LEGACY:
+ ret = easelcomm_write_msgdata(service, kbuf_desc);
+ break;
+ case EASELCOMM_IOC_READDATA:
+ case EASELCOMM_IOC_READDATA_LEGACY:
+ ret = easelcomm_read_msgdata(service, kbuf_desc);
+ break;
+ case EASELCOMM_IOC_SENDDMA:
+ case EASELCOMM_IOC_SENDDMA_LEGACY:
+ ret = easelcomm_send_dma(service, kbuf_desc);
+ break;
+ case EASELCOMM_IOC_RECVDMA:
+ case EASELCOMM_IOC_RECVDMA_LEGACY:
+ ret = easelcomm_receive_dma(service, kbuf_desc);
+ break;
+ default:
+ ret = -EINVAL;
+ }
+
+ return ret;
+}
+
+static int easelcomm_ioctl_kbuf_desc(
+ struct easelcomm_service *service, unsigned int cmd,
+ unsigned long arg, bool compat)
+{
+ struct easelcomm_kbuf_desc kbuf_desc;
+
+ memset(&kbuf_desc, 0, sizeof(kbuf_desc));
+
+ if (compat) {
+#ifdef CONFIG_COMPAT
+ struct easelcomm_compat_kbuf_desc compat_kbuf_desc;
+
+ if (copy_from_user(
+ &compat_kbuf_desc, (void __user *) arg,
+ sizeof(struct easelcomm_compat_kbuf_desc)))
+ return -EFAULT;
+
+ kbuf_desc.message_id = compat_kbuf_desc.message_id;
+ kbuf_desc.buf = compat_ptr(compat_kbuf_desc.buf);
+ kbuf_desc.buf_size = compat_kbuf_desc.buf_size;
+ kbuf_desc.buf_type = compat_kbuf_desc.buf_type;
+ kbuf_desc.wait.timeout_ms = -1;
+ kbuf_desc.dma_buf_fd = compat_kbuf_desc.dma_buf_fd;
+ kbuf_desc.dma_buf_off = compat_kbuf_desc.dma_buf_off;
+ kbuf_desc.dma_buf_width = compat_kbuf_desc.dma_buf_width;
+ kbuf_desc.dma_buf_stride = compat_kbuf_desc.dma_buf_stride;
+#else
+ return -EINVAL;
+#endif
+ } else {
+ if (copy_from_user(
+ &kbuf_desc, (void __user *) arg,
+ sizeof(struct easelcomm_kbuf_desc)))
+ return -EFAULT;
+ }
+
+ return __easelcomm_ioctl_kbuf_desc(service, cmd, &kbuf_desc);
+}
+
+static int easelcomm_ioctl_kbuf_desc_legacy(
+ struct easelcomm_service *service, unsigned int cmd,
+ unsigned long arg, bool compat)
+{
+ struct easelcomm_kbuf_desc kbuf_desc;
+
+ memset(&kbuf_desc, 0, sizeof(kbuf_desc));
+
+ if (compat) {
+#ifdef CONFIG_COMPAT
+ struct easelcomm_compat_kbuf_desc_legacy
+ compat_kbuf_desc_legacy;
+
+ if (copy_from_user(&compat_kbuf_desc_legacy,
+ (void __user *) arg,
+ sizeof(compat_kbuf_desc_legacy)))
+ return -EFAULT;
+
+ kbuf_desc.message_id = compat_kbuf_desc_legacy.message_id;
+ kbuf_desc.buf = compat_ptr(compat_kbuf_desc_legacy.buf);
+ kbuf_desc.buf_size = compat_kbuf_desc_legacy.buf_size;
+ kbuf_desc.buf_type = compat_kbuf_desc_legacy.buf_type;
+ kbuf_desc.wait.timeout_ms = -1;
+ kbuf_desc.dma_buf_fd = compat_kbuf_desc_legacy.dma_buf_fd;
+ kbuf_desc.dma_buf_width = compat_kbuf_desc_legacy.buf_size;
+ kbuf_desc.dma_buf_stride = compat_kbuf_desc_legacy.buf_size;
+#else
+ return -EINVAL;
+#endif
+ } else {
+ struct easelcomm_kbuf_desc_legacy kbuf_desc_legacy;
+
+ if (copy_from_user(&kbuf_desc_legacy, (void __user *) arg,
+ sizeof(kbuf_desc_legacy)))
+ return -EFAULT;
+
+ kbuf_desc.message_id = kbuf_desc_legacy.message_id;
+ kbuf_desc.buf = kbuf_desc_legacy.buf;
+ kbuf_desc.buf_size = kbuf_desc_legacy.buf_size;
+ kbuf_desc.buf_type = kbuf_desc_legacy.buf_type;
+ kbuf_desc.wait.timeout_ms = kbuf_desc_legacy.wait.timeout_ms;
+ kbuf_desc.dma_buf_fd = kbuf_desc_legacy.dma_buf_fd;
+ kbuf_desc.dma_buf_width = kbuf_desc_legacy.buf_size;
+ kbuf_desc.dma_buf_stride = kbuf_desc_legacy.buf_size;
+ }
+
+ return __easelcomm_ioctl_kbuf_desc(service, cmd, &kbuf_desc);
+}
+
+/* 32-bit and 64-bit userspace ioctl handling, dispatch. */
+static long easelcomm_ioctl_common(
+ struct file *file, unsigned int cmd, unsigned long arg, bool compat)
+{
+ int ret = 0;
+ struct easelcomm_user_state *user_state = file->private_data;
+ struct easelcomm_service *service;
+
+ if (_IOC_TYPE(cmd) != EASELCOMM_IOC_MAGIC)
+ return -EINVAL;
+
+ if (WARN_ON(!user_state))
+ return -EINVAL;
+
+ if (!easelcomm_up)
+ return -ESHUTDOWN;
+
+ /* REGISTER is the only ioctl that doesn't need a service registered. */
+ if (cmd == EASELCOMM_IOC_REGISTER)
+ return easelcomm_register(user_state, (unsigned int) arg);
+
+ service = user_state->service;
+ if (!service) {
+ dev_err(easelcomm_miscdev.this_device,
+ "user has not registered a service for the file\n");
+ return -EINVAL;
+ }
+
+ switch (cmd) {
+ case EASELCOMM_IOC_SENDMSG:
+ ret = easelcomm_send_message_ioctl(
+ service, (struct easelcomm_kmsg_desc *)arg);
+ break;
+ case EASELCOMM_IOC_WRITEDATA:
+ case EASELCOMM_IOC_READDATA:
+ case EASELCOMM_IOC_SENDDMA:
+ case EASELCOMM_IOC_RECVDMA:
+ /* Convert 32-bit kbuf_desc argument if needed and dispatch. */
+ ret = easelcomm_ioctl_kbuf_desc(service, cmd, arg, compat);
+ break;
+ case EASELCOMM_IOC_WRITEDATA_LEGACY:
+ case EASELCOMM_IOC_READDATA_LEGACY:
+ case EASELCOMM_IOC_SENDDMA_LEGACY:
+ case EASELCOMM_IOC_RECVDMA_LEGACY:
+ /* Convert 32-bit kbuf_desc argument if needed and dispatch. */
+ ret = easelcomm_ioctl_kbuf_desc_legacy(service, cmd, arg,
+ compat);
+ break;
+ case EASELCOMM_IOC_WAITMSG:
+ ret = easelcomm_wait_message(
+ service, (struct easelcomm_kmsg_desc *)arg);
+ break;
+ case EASELCOMM_IOC_WAITREPLY:
+ ret = easelcomm_wait_reply(
+ service, (struct easelcomm_kmsg_desc *)arg);
+ break;
+ case EASELCOMM_IOC_SHUTDOWN:
+ easelcomm_initiate_service_shutdown(service);
+ break;
+ case EASELCOMM_IOC_FLUSH:
+ easelcomm_flush_service(service);
+ break;
+ default:
+ ret = -ENOIOCTLCMD;
+ break;
+ }
+
+ return ret;
+}
+
+/* 64-bit ioctl handling. */
+static long easelcomm_ioctl(
+ struct file *file, unsigned int cmd, unsigned long arg)
+{
+ return easelcomm_ioctl_common(file, cmd, arg, false);
+}
+
+#ifdef CONFIG_COMPAT
+/* 32-bit ioctl handling. */
+static long easelcomm_compat_ioctl(
+ struct file *file, unsigned int cmd, unsigned long arg)
+{
+ unsigned int cmd_nr = _IOC_NR(cmd);
+ int ret;
+
+ /* Fixup pointer argument size for ioctls 1..7 */
+ if ((cmd_nr >= 1 && cmd_nr <= 7) &&
+ _IOC_SIZE(cmd) == sizeof(compat_uptr_t)) {
+ cmd &= ~(_IOC_SIZEMASK << _IOC_SIZESHIFT);
+ cmd |= sizeof(void *) << _IOC_SIZESHIFT;
+ }
+
+ switch (cmd) {
+ case EASELCOMM_IOC_SENDMSG:
+ case EASELCOMM_IOC_WAITMSG:
+ case EASELCOMM_IOC_WAITREPLY:
+ case EASELCOMM_IOC_WRITEDATA:
+ case EASELCOMM_IOC_READDATA:
+ case EASELCOMM_IOC_SENDDMA:
+ case EASELCOMM_IOC_RECVDMA:
+ case EASELCOMM_IOC_WRITEDATA_LEGACY:
+ case EASELCOMM_IOC_READDATA_LEGACY:
+ case EASELCOMM_IOC_SENDDMA_LEGACY:
+ case EASELCOMM_IOC_RECVDMA_LEGACY:
+ /* pointer argument, fixup */
+ ret = easelcomm_ioctl_common(file, cmd,
+ (unsigned long)compat_ptr(arg), true);
+ break;
+
+ case EASELCOMM_IOC_REGISTER:
+ case EASELCOMM_IOC_SHUTDOWN:
+ case EASELCOMM_IOC_FLUSH:
+ /* no ioctl argument conversion needed */
+ ret = easelcomm_ioctl_common(file, cmd, arg, true);
+ break;
+
+ default:
+ ret = -ENOIOCTLCMD;
+ break;
+ }
+
+ return ret;
+}
+#endif /* CONFIG_COMPAT */
+
+/* Init local state for remote command channel. */
+static void easelcomm_init_cmd_channel_remote(
+ struct easelcomm_cmd_channel_remote *channel)
+{
+ mutex_init(&channel->mutex);
+ channel->initialized = false;
+ init_completion(&channel->init_state_changed);
+ init_completion(&channel->wrap_ready);
+ /* Write pointer starts after the channel header */
+ channel->write_offset =
+ (sizeof(struct easelcomm_cmd_channel_header) + 7) & ~0x7;
+ channel->write_seqnbr = 0;
+}
+
+/* Init local command channel. */
+static int easelcomm_init_cmd_channel_local(
+ struct easelcomm_cmd_channel_local *channel, void *cmdchan_buffer)
+{
+ channel->buffer = cmdchan_buffer;
+ if (!channel->buffer)
+ return -ENOMEM;
+ channel->readp = channel->buffer +
+ sizeof(struct easelcomm_cmd_channel_header);
+
+ /* next sequence # to be produced/consumed starts at zero */
+ channel->consumer_seqnbr_next = 0;
+ ((struct easelcomm_cmd_channel_header *)
+ channel->buffer)->producer_seqnbr_next = 0;
+ return 0;
+}
+
+/*
+ * Callback from h/w layer when PCIe ready (which is immediate for server,
+ * happens on first PCIe host driver probe on client). The h/w layer has
+ * allocated the local command channel buffer (which needs the host PCIe
+ * driver on client), the pointer to which is passed as an argument. Can now
+ * init our local command channel state, and do remote too.
+ */
+int easelcomm_init_pcie_ready(void *local_cmdchan_buffer)
+{
+ int ret;
+
+ ret = easelcomm_init_cmd_channel_local(
+ &cmd_channel_local, local_cmdchan_buffer);
+ easelcomm_init_cmd_channel_remote(&cmd_channel_remote);
+ return ret;
+}
+EXPORT_SYMBOL(easelcomm_init_pcie_ready);
+
+/*
+ * Callback for AP/client from h/w layer when PCIe EP is off.
+ */
+int easelcomm_pcie_hotplug_out(void)
+{
+ struct easelcomm_cmd_channel_remote *channel = &cmd_channel_remote;
+
+ /*
+ * PCIe link is already down, so no need to call easelcomm_stop() to
+ * notify remote, but only call easelcomm_stop_local().
+ */
+ easelcomm_stop_local();
+
+ mutex_lock(&channel->mutex);
+ channel->initialized = false;
+ mutex_unlock(&channel->mutex);
+ return 0;
+}
+EXPORT_SYMBOL(easelcomm_pcie_hotplug_out);
+
+static int easelcomm_notify_sys(
+ struct notifier_block *this, unsigned long code, void *unused)
+{
+ if (code == SYS_RESTART || code == SYS_HALT || code == SYS_POWER_OFF)
+ easelcomm_stop(false);
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block easelcomm_notifier = {
+ .notifier_call = easelcomm_notify_sys,
+};
+
+static int __init easelcomm_init(void)
+{
+ int ret;
+ int i;
+
+ ret = misc_register(&easelcomm_miscdev);
+ if (ret) {
+ dev_err(easelcomm_miscdev.this_device,
+ "misc_register failed\n");
+ return ret;
+ }
+ dev_info(easelcomm_miscdev.this_device,
+ "registered at misc device minor %d\n",
+ easelcomm_miscdev.minor);
+
+ mutex_init(&service_mutex);
+
+ for (i = 0; i < EASELCOMM_SERVICE_COUNT; i++) {
+ easelcomm_service_list[i] = NULL;
+ }
+
+ /* Initializes link_service. */
+ easelcomm_init_service(&link_service, EASELCOMM_SERVICE_COUNT);
+
+ easelcomm_hw_init();
+
+ /* Shut down easelcomm on reboot */
+ ret = register_reboot_notifier(&easelcomm_notifier);
+ if (ret)
+ dev_warn(easelcomm_miscdev.this_device,
+ "register reboot notifier failed\n");
+
+ return 0;
+}
+
+static void __exit easelcomm_exit(void)
+{
+ int i;
+ easelcomm_stop(false);
+ kfree(cmd_channel_local.buffer);
+ misc_deregister(&easelcomm_miscdev);
+ /*
+ * TODO(cjluo): Some of these service struct
+ * could be freed when unregistered.
+ */
+ for (i = 0; i < EASELCOMM_SERVICE_COUNT; i++) {
+ if (easelcomm_service_list[i] != NULL) {
+ kfree(easelcomm_service_list[i]);
+ easelcomm_service_list[i] = NULL;
+ }
+ }
+}
+
+module_init(easelcomm_init);
+module_exit(easelcomm_exit);
+
+MODULE_AUTHOR("Google Inc.");
+MODULE_DESCRIPTION("Easel coprocessor communication driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/misc/hmc6352.c b/drivers/misc/hmc6352.c
index 90520d7..9cde4c5 100644
--- a/drivers/misc/hmc6352.c
+++ b/drivers/misc/hmc6352.c
@@ -27,6 +27,7 @@
#include <linux/err.h>
#include <linux/delay.h>
#include <linux/sysfs.h>
+#include <linux/nospec.h>
static DEFINE_MUTEX(compass_mutex);
@@ -50,6 +51,7 @@ static int compass_store(struct device *dev, const char *buf, size_t count,
return ret;
if (val >= strlen(map))
return -EINVAL;
+ val = array_index_nospec(val, strlen(map));
mutex_lock(&compass_mutex);
ret = compass_command(c, map[val]);
mutex_unlock(&compass_mutex);
diff --git a/drivers/misc/ibmasm/ibmasmfs.c b/drivers/misc/ibmasm/ibmasmfs.c
index 520f584..65ad7e5 100644
--- a/drivers/misc/ibmasm/ibmasmfs.c
+++ b/drivers/misc/ibmasm/ibmasmfs.c
@@ -507,35 +507,14 @@ static int remote_settings_file_close(struct inode *inode, struct file *file)
static ssize_t remote_settings_file_read(struct file *file, char __user *buf, size_t count, loff_t *offset)
{
void __iomem *address = (void __iomem *)file->private_data;
- unsigned char *page;
- int retval;
int len = 0;
unsigned int value;
-
- if (*offset < 0)
- return -EINVAL;
- if (count == 0 || count > 1024)
- return 0;
- if (*offset != 0)
- return 0;
-
- page = (unsigned char *)__get_free_page(GFP_KERNEL);
- if (!page)
- return -ENOMEM;
+ char lbuf[20];
value = readl(address);
- len = sprintf(page, "%d\n", value);
+ len = snprintf(lbuf, sizeof(lbuf), "%d\n", value);
- if (copy_to_user(buf, page, len)) {
- retval = -EFAULT;
- goto exit;
- }
- *offset += len;
- retval = len;
-
-exit:
- free_page((unsigned long)page);
- return retval;
+ return simple_read_from_buffer(buf, count, offset, lbuf, len);
}
static ssize_t remote_settings_file_write(struct file *file, const char __user *ubuff, size_t count, loff_t *offset)
diff --git a/drivers/misc/lkdtm_bugs.c b/drivers/misc/lkdtm_bugs.c
index bb3bb8e..f36674e 100644
--- a/drivers/misc/lkdtm_bugs.c
+++ b/drivers/misc/lkdtm_bugs.c
@@ -66,7 +66,7 @@ void lkdtm_WARNING(void)
void lkdtm_EXCEPTION(void)
{
- *((int *) 0) = 0;
+ *((volatile int *) 0) = 0;
}
void lkdtm_LOOP(void)
diff --git a/drivers/misc/mei/bus-fixup.c b/drivers/misc/mei/bus-fixup.c
index 75b9d4a..371f5f6 100644
--- a/drivers/misc/mei/bus-fixup.c
+++ b/drivers/misc/mei/bus-fixup.c
@@ -178,7 +178,7 @@ static int mei_nfc_if_version(struct mei_cl *cl,
ret = 0;
bytes_recv = __mei_cl_recv(cl, (u8 *)reply, if_version_length);
- if (bytes_recv < if_version_length) {
+ if (bytes_recv < 0 || bytes_recv < if_version_length) {
dev_err(bus->dev, "Could not read IF version\n");
ret = -EIO;
goto err;
diff --git a/drivers/misc/mei/hbm.c b/drivers/misc/mei/hbm.c
index dd7f15a..ae4a570 100644
--- a/drivers/misc/mei/hbm.c
+++ b/drivers/misc/mei/hbm.c
@@ -1137,15 +1137,18 @@ int mei_hbm_dispatch(struct mei_device *dev, struct mei_msg_hdr *hdr)
props_res = (struct hbm_props_response *)mei_msg;
- if (props_res->status) {
+ if (props_res->status == MEI_HBMS_CLIENT_NOT_FOUND) {
+ dev_dbg(dev->dev, "hbm: properties response: %d CLIENT_NOT_FOUND\n",
+ props_res->me_addr);
+ } else if (props_res->status) {
dev_err(dev->dev, "hbm: properties response: wrong status = %d %s\n",
props_res->status,
mei_hbm_status_str(props_res->status));
return -EPROTO;
+ } else {
+ mei_hbm_me_cl_add(dev, props_res);
}
- mei_hbm_me_cl_add(dev, props_res);
-
/* request property for the next client */
if (mei_hbm_prop_req(dev, props_res->me_addr + 1))
return -EIO;
diff --git a/drivers/misc/mei/main.c b/drivers/misc/mei/main.c
index 60f5a8d..8904491 100644
--- a/drivers/misc/mei/main.c
+++ b/drivers/misc/mei/main.c
@@ -304,7 +304,6 @@ static ssize_t mei_write(struct file *file, const char __user *ubuf,
goto out;
}
- *offset = 0;
cb = mei_cl_alloc_cb(cl, length, MEI_FOP_WRITE, file);
if (!cb) {
rets = -ENOMEM;
diff --git a/drivers/misc/mei/pci-me.c b/drivers/misc/mei/pci-me.c
index f9c6ec4..013a7b3 100644
--- a/drivers/misc/mei/pci-me.c
+++ b/drivers/misc/mei/pci-me.c
@@ -229,8 +229,11 @@ static int mei_me_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
if (!pci_dev_run_wake(pdev))
mei_me_set_pm_domain(dev);
- if (mei_pg_is_enabled(dev))
+ if (mei_pg_is_enabled(dev)) {
pm_runtime_put_noidle(&pdev->dev);
+ if (hw->d0i3_supported)
+ pm_runtime_allow(&pdev->dev);
+ }
dev_dbg(&pdev->dev, "initialization successful.\n");
diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index ddc9e4b..56efa9d 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -370,11 +370,10 @@ int scif_bind(scif_epd_t epd, u16 pn)
goto scif_bind_exit;
}
} else {
- pn = scif_get_new_port();
- if (!pn) {
- ret = -ENOSPC;
+ ret = scif_get_new_port();
+ if (ret < 0)
goto scif_bind_exit;
- }
+ pn = ret;
}
ep->state = SCIFEP_BOUND;
@@ -648,13 +647,12 @@ int __scif_connect(scif_epd_t epd, struct scif_port_id *dst, bool non_block)
err = -EISCONN;
break;
case SCIFEP_UNBOUND:
- ep->port.port = scif_get_new_port();
- if (!ep->port.port) {
- err = -ENOSPC;
- } else {
- ep->port.node = scif_info.nodeid;
- ep->conn_async_state = ASYNC_CONN_IDLE;
- }
+ err = scif_get_new_port();
+ if (err < 0)
+ break;
+ ep->port.port = err;
+ ep->port.node = scif_info.nodeid;
+ ep->conn_async_state = ASYNC_CONN_IDLE;
/* Fall through */
case SCIFEP_BOUND:
/*
diff --git a/drivers/misc/mic/scif/scif_rma.c b/drivers/misc/mic/scif/scif_rma.c
index f806a44..32ab0f4 100644
--- a/drivers/misc/mic/scif/scif_rma.c
+++ b/drivers/misc/mic/scif/scif_rma.c
@@ -414,7 +414,7 @@ static int scif_create_remote_lookup(struct scif_dev *remote_dev,
if (err)
goto error_window;
err = scif_map_page(&window->num_pages_lookup.lookup[j],
- vmalloc_dma_phys ?
+ vmalloc_num_pages ?
vmalloc_to_page(&window->num_pages[i]) :
virt_to_page(&window->num_pages[i]),
remote_dev);
diff --git a/drivers/misc/mnh/Kconfig b/drivers/misc/mnh/Kconfig
new file mode 100644
index 0000000..931b9fc
--- /dev/null
+++ b/drivers/misc/mnh/Kconfig
@@ -0,0 +1,57 @@
+#
+# MNH PCIE Host driver configuration
+#
+config MNH_PCIE_HOST
+ tristate "MNH PCIe Host-side driver"
+ depends on PCI
+ default "n"
+ ---help---
+ Include support for MNH PCIe Host-side driver
+
+
+config MNH_SM_HOST
+ tristate "MNH Easel State Machine Host-side driver"
+ depends on PCI
+ default "n"
+ ---help---
+ Include support for MNH State Machine Host-side driver. This driver
+ controls the power-on and power-down sequences including firmware
+ download. The state manager is controlled by a userspace service
+ to control state transitions.
+
+config MNH_THERMAL_HOST
+ tristate "MNH Easel Thermal Sensor Device Host-side driver"
+ depends on MNH_SM_HOST
+ default "n"
+ ---help---
+ Include support for MNH Thermal Sensor Device Host-side driver. This
+ driver adds support for all thermal zone sensors on the Easel device and
+ exposes them to the kernel thermal framework.
+
+config MNH_PCIE_MULTIPLE_MSI
+ tristate "MNH PCIE Multiple MSI support"
+ depends on MNH_PCIE_HOST
+ default "n"
+ ---help---
+ Include support for multiple MSIs in MNH PCIe Host driver
+
+config MNH_SIG
+ tristate "MNH signing engine"
+ depends on MNH_SM_HOST
+ default "n"
+ ---help---
+ Adds support for authentication of MNH images
+
+config MNH_SIG_FORCE_OTA
+ tristate "MNH signature enforcement - OTA"
+ depends on MNH_SIG
+ default "n"
+ ---help---
+ Enforce firmware signing for images delivered through an OTA
+
+config MNH_SIG_FORCE_PS
+ tristate "MNH signature enforcement - Play Store"
+ depends on MNH_SIG
+ default "y"
+ ---help---
+ Enforce firmware signing for images delivered through the Play Store
diff --git a/drivers/misc/mnh/Makefile b/drivers/misc/mnh/Makefile
new file mode 100644
index 0000000..c09a6f4
--- /dev/null
+++ b/drivers/misc/mnh/Makefile
@@ -0,0 +1,13 @@
+#
+# Makefile for MNH PCIE/DMA devices
+#
+
+obj-$(CONFIG_MNH_PCIE_HOST) += mnh-pcie.o
+obj-$(CONFIG_MNH_SM_HOST) += mnh-sm.o mnh-sm-ion.o mnh-pwr.o mnh-mipi.o mnh-ddr.o mnh-clk.o
+obj-$(CONFIG_MNH_THERMAL_HOST) += mnh-thermal.o mnh-efuse.o
+obj-$(CONFIG_MNH_SIG) += mnh-crypto.o mnh-crypto-sysfs.o
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-pointer-bool-conversion \
+
+
diff --git a/drivers/misc/mnh/hw-mnh-regs.h b/drivers/misc/mnh/hw-mnh-regs.h
new file mode 100644
index 0000000..74733f2
--- /dev/null
+++ b/drivers/misc/mnh/hw-mnh-regs.h
@@ -0,0 +1,174 @@
+/**
+ *
+ * MNH PCIe/DMA HW Register Address header file
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef __HW_MNH_PCIE_REGS_H_
+#define __HW_MNH_PCIE_REGS_H_
+
+#define HW_MNH_PCIE_CONFIG_ADDR_START 0x00200000
+#define HW_MNH_PCIE_CONFIG_ADDR_END 0x00200FFF
+#define HW_MNH_PCIE_BAR_2_ADDR_START 0x00000000
+#define HW_MNH_PCIE_BAR_2_ADDR_END 0x3FFFFFFF
+#define HW_MNH_PCIE_BAR_4_ADDR_START 0x40000000
+#define HW_MNH_PCIE_BAR_4_ADDR_END 0x7FFFFFFF
+#define HW_MNH_PCIE_CLUSTER_ADDR_START 0x040C0000
+#define HW_MNH_PCIE_CLUSTER_ADDR_END 0x040C0FFF
+#define HW_MNH_PCIE_OUTBOUND_BASE 0x80000000
+#define HW_MNH_PCIE_OUTBOUND_SIZE 0x00019000 /* 100kB */
+#define HW_MNH_PCIE_OUTBOUND_END \
+ (HW_MNH_PCIE_OUTBOUND_BASE + HW_MNH_PCIE_OUTBOUND_SIZE - 1)
+
+#define HW_MNH_PCIE_BAR2_R1_ADDR_START 0x00000000 /* Start of ROM */
+#define HW_MNH_PCIE_BAR2_R1_ADDR_END 0x0011FFFF /* End of SRAM */
+#define HW_MNH_PCIE_BAR2_R2_ADDR_START 0x04000000 /* Start of Peripheral */
+#define HW_MHH_PCIE_BAR2_R2_ADDR_END 0x047FFFFF /* End of Peripheral */
+
+#define HW_MNH_PCIE_CLUSTER_ADDR_OFFSET \
+ (HW_MNH_PCIE_CLUSTER_ADDR_START - HW_MNH_PCIE_BAR_2_ADDR_START)
+
+/* PCIE Cluster register value */
+#define HW_MNH_PCIE_SW_INTR_TRIGG 0x0
+#define HW_MNH_PCIE_SW_INTR_TRIGG_MASK_RW 0x80000000 /*31*/
+#define HW_MNH_PCIE_SW_INTR_EN 0x4
+#define HW_MNH_PCIE_MSI_PEND_REG 0x1C
+#define HW_MNH_PCIE_GP_0 0x80 /* magic number handshake */
+#define HW_MNH_PCIE_GP_1 0x84 /* ring buffer address in MNH */
+#define HW_MNH_PCIE_GP_2 0x88 /* DMA Status from AP to EP */
+#define HW_MNH_PCIE_GP_3 0x8C /* MSI vector */
+#define HW_MNH_PCIE_GP_4 0x90
+#define HW_MNH_PCIE_GP_5 0x94
+#define HW_MNH_PCIE_GP_6 0x98
+#define HW_MNH_PCIE_GP_7 0x9C
+
+/* IATU Registers */
+#define IATU_VIEWPORT 0x900
+#define IATU_LWR_BASE_ADDR 0x90C
+#define IATU_UPPER_BASE_ADDR 0x910
+#define IATU_LIMIT_ADDR 0x914
+#define IATU_LWR_TARGET_ADDR 0x918
+#define IATU_UPPER_TARGET_ADDR 0x91C
+#define IATU_REGION_CTRL_1 0x904
+#define IATU_REGION_CTRL_2 0x908
+
+#define IATU_ENABLE 0x80000000
+#define IATU_DMA_BYPASS 0x08000000
+#define IATU_OUTBOUND 0x0
+#define IATU_INBOUND 0x80000000
+#define IATU_BAR_MODE 0xC0000000
+
+/* MSI related register values */
+#define MSI_CAP_ID_NEXT_CTRL_REG 0x50
+#define MULTIPLE_MSG_ENABLE_MASK 0x700000
+#define MSI_CAP_OFF_04H_REG 0x54
+#define MSI_CAP_OFF_08H_REG 0x58
+#define MSI_CAP_OFF_0CH_REG 0x5C
+#define MSI_CAP_OFF_10H_REG 0x60
+
+/* DMA related register values */
+#define DMA_CTRL_OFF 0x978 /* DMA Number of Channels Register */
+#define DMA_CTRL_OFF_MASK_WR_CHAN 0x07
+#define DMA_CTRL_OFF_MASK_RD_CHAN 0xE0000
+
+#define DMA_WRITE_ENGINE_EN_OFF 0x97C/* DMA Write Engine Enable Register */
+#define DMA_WRITE_ENGINE_EN_OFF_MASK_ENABLE 0x01
+
+#define DMA_WRITE_DOORBELL_OFF 0x980 /* DMA Write Doorbell register */
+#define DMA_WRITE_DOORBELL_OFF_MASK_CH_NUM 0x07
+#define DMA_WRITE_DOORBELL_OFF_MASK_WR_STOP 0x80000000
+
+#define DMA_READ_ENGINE_EN_OFF 0x99C /*DMA Read Engine Enable Register */
+#define DMA_READ_ENGINE_EN_OFF_MASK_ENABLE 0x01
+
+#define DMA_READ_DOORBELL_OFF 0x9A0 /* DMA Read Doorbell register */
+#define DMA_READ_DOORBELL_OFF_MASK_RD_STOP 0x80000000
+#define DMA_READ_DOORBELL_OFF_MASK_CH_NUM 0x07
+
+#define DMA_WRITE_INT_STATUS_OFF 0x9BC
+#define DMA_WRITE_INT_STATUS_OFF_ABORT_STATUS 0xFF0000 /*23:16*/
+#define DMA_WRITE_INT_STATUS_OFF_DONE_STATUS 0xFF/*7:0 */
+
+#define DMA_WRITE_INT_MASK_OFF 0x9C4
+#define DMA_WRITE_INT_CLEAR_OFF 0x9C8
+#define DMA_WRITE_ERR_STATUS_OFF 0x9CC
+#define DMA_WRITE_ERR_STATUS_OFF_MASK_APP_READ_ERR 0xFF /* 7:0 */
+#define DMA_WRITE_ERR_STATUS_OFF_MASK_LL_FETCH_ERR 0xFF0000 /* 23:16 */
+
+#define DMA_WRITE_DONE_IMWR_LOW_OFF 0x9d0
+#define DMA_WRITE_DONE_IMWR_HIGH_OFF 0x9d4
+#define DMA_WRITE_ABORT_IMWR_LOW_OFF 0x9d8
+#define DMA_WRITE_ABORT_IMWR_HIGH_OFF 0x9dc
+#define DMA_WRITE_CH01_IMWR_DATA_OFF 0x9E0
+#define DMA_WRITE_CH23_IMWR_DATA_OFF 0x9E4
+
+#define DMA_READ_INT_STATUS_OFF 0xA10
+#define DMA_READ_INT_STATUS_OFF_ABORT_STATUS 0xFF0000 /*23:16*/
+#define DMA_READ_INT_STATUS_OFF_DONE_STATUS 0xFF/*7:0 */
+
+#define DMA_READ_INT_MASK_OFF 0xA18
+#define DMA_READ_INT_CLEAR_OFF 0xA1C
+
+#define DMA_READ_ERR_STATUS_OFF 0xA24
+#define DMA_READ_ERR_STATUS_OFF_MASK_APP_READ_ERR 0xFF /* 7:0 */
+#define DMA_READ_ERR_STATUS_OFF_MASK_LL_FETCH_ERR 0xFF0000 /* 23:16 */
+#define DMA_READ_ERR_STATUS_HIGH_OFF 0xA28
+#define DMA_READ_ERR_STATUS_HIGH_OFF_MASK_DATA_POISON 0xFF000000 /* 31:24 */
+#define DMA_READ_ERR_STATUS_HIGH_OFF_CPL_TIMEOUT 0xFF0000 /*23:16 */
+#define DMA_READ_ERR_STATUS_HIGH_OFF_CPL_ABORT 0xFF00 /* 15:8 */
+#define DMA_READ_ERR_STATUS_HIGH_OFF_UNSUPPORTED 0xFF /*7:0 */
+
+#define DMA_READ_DONE_IMWR_LOW_OFF 0xa3c
+#define DMA_READ_DONE_IMWR_HIGH_OFF 0xa40
+#define DMA_READ_ABORT_IMWR_LOW_OFF 0xa44
+#define DMA_READ_ABORT_IMWR_HIGH_OFF 0xa48
+#define DMA_READ_CH01_IMWR_DATA_OFF 0xA4C
+#define DMA_READ_CH23_IMWR_DATA_OFF 0xA50
+
+#define DMA_READ_LINKED_LIST_ERR_EN_OFF 0xa34
+#define DMA_WRITE_LINKED_LIST_ERR_EN_OFF 0xa00
+
+
+#define DMA_VIEWPORT_SEL_OFF 0xA6C
+#define DMA_VIEWPORT_SEL_OFF_MASK_CHAN_DIR 0x80000000
+#define DMA_VIEWPORT_SEL_OFF_MASK_CHAN_NUM 0x7 /* 2:1 */
+
+#define DMA_CH_CONTROL1_OFF 0xA70
+#define DMA_CH_CONTROL1_OFF_RESERVED 0x4000000
+#define DMA_CH_CONTROL1_MASK_LLE 0x200
+#define DMA_CH_CONTROL1_MASK_CCS 0x100
+#define DMA_CH_CONTROL1_MASK_CS 0x60 /* 6:5 */
+#define DMA_CH_CONTROL1_MASK_RIE 0x10 /* 4 */
+#define DMA_CH_CONTROL1_MASK_LIE 0x8 /* 3 */
+#define DMA_CH_CONTROL1_MASK_LLP 0x4 /* 2 */
+#define DMA_CH_CONTROL1_MASK_TCB 0x2 /* 1 */
+
+#define DMA_CH_CONTROL2_OFF 0xA74
+
+
+#define DMA_TRANSFER_SIZE_OFF 0xA78
+
+#define DMA_SAR_LOW_OFF 0xA7C
+#define DMA_SAR_HIGH_OFF 0xA80
+#define DMA_DAR_LOW_OFF 0xA84
+#define DMA_DAR_HIGH_OFF 0xA88
+
+#define DMA_LLP_LOW_OFF 0xA8C
+#define DMA_LLP_HIGH_OFF 0xA90
+
+#define DMA_READ_DONE_MASK 0xFF /* 7:0 */
+#define DMA_READ_ABORT_MASK 0xFF00 /* 15:8 */
+#define DMA_WRITE_DONE_MASK 0xFF0000 /* 16:23 */
+#define DMA_WRITE_ABORT_MASK 0xFF000000 /* 24:31 */
+
+#endif
diff --git a/drivers/misc/mnh/mnh-clk.c b/drivers/misc/mnh/mnh-clk.c
new file mode 100644
index 0000000..4bb51cf
--- /dev/null
+++ b/drivers/misc/mnh/mnh-clk.c
@@ -0,0 +1,1073 @@
+/*
+ *
+ * MNH Clock Driver
+ * Copyright (c) 2016-2017, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/sysfs.h>
+#include <linux/device.h>
+#include "mnh-hwio.h"
+#include "mnh-hwio-bases.h"
+#include "mnh-hwio-scu.h"
+#include "mnh-clk.h"
+#include "mnh-ddr.h"
+
+#define SCU_IN(...) \
+ HW_IN(HWIO_SCU_BASE_ADDR, SCU, __VA_ARGS__)
+#define SCU_INf(...) \
+ HW_INf(HWIO_SCU_BASE_ADDR, SCU, __VA_ARGS__)
+#define SCU_INxf(...) \
+ HW_INxf(HWIO_SCU_BASE_ADDR, SCU, __VA_ARGS__)
+#define SCU_OUT(...) \
+ HW_OUT(HWIO_SCU_BASE_ADDR, SCU, __VA_ARGS__)
+#define SCU_OUTf(...) \
+ HW_OUTf(HWIO_SCU_BASE_ADDR, SCU, __VA_ARGS__)
+
+#define PLL_UNLOCK 0x4CD9
+#define LP4_LPC_FREQ_SWITCH 0x8A
+#define REF_CLK_KHZ 19200
+#define SYS200_CLK_KHZ 200000
+#define PLL_LOCK_TIMEOUT 100
+
+enum mnh_clk_type {
+ CPU_CLK = 0,
+ IPU_CLK
+};
+
+enum mnh_lpddr_lpc_rsp_type {
+ LPC_CMD_NOERR = 0,
+ LPC_CMD_ERR = 1
+};
+
+enum mnh_ipu_clk_src {
+ CPU_IPU_PLL = 0,
+ IPU_PLL
+};
+
+struct freq_reg_table {
+ char *freq_str;
+ int fbdiv;
+ int postdiv1;
+ int postdiv2;
+ int clk_div;
+};
+
+/* SYS200 frequency calculation table */
+static struct freq_reg_table sys200_reg_table[] = {
+ {"200", 104, 2, 1, 0}, /* CPU_CLK */
+ {"100", 104, 2, 1, 1} /* IPU_CLK */
+};
+
+/* CPU clock frequency calculation table */
+static struct freq_reg_table cpu_reg_table[] = {
+ {"200", 125, 6, 2, 0},
+ {"400", 125, 6, 1, 0},
+ {"600", 125, 4, 1, 0},
+ {"800", 125, 3, 1, 0},
+ {"950", 99, 2, 1, 0}
+};
+
+/* IPU clock frequency calculation table */
+static struct freq_reg_table ipu_reg_table[] = {
+ {"100", 125, 6, 2, 1},
+ {"200", 125, 6, 1, 1},
+ {"300", 125, 4, 1, 1},
+ {"400", 125, 6, 1, 0},
+ {"425", 133, 6, 1, 0}
+};
+
+struct mnh_clk {
+ struct device *dev;
+};
+
+static struct mnh_clk *mnh_clk;
+
+/* If IPU clock is driven by CPU_IPU PLL
+ * calculate IPU divider based on CPU clk and divider
+ * CPU CLK < 850: IPU_CLK_DIV = ((CPU_CLK_DIV+1)*2-1)
+ * CPU CLK > 850: IPU_CLK_DIV = ((CPU_CLK_DIV+1)*2)
+ */
+static inline int mnh_clk_get_ipu_div(int cpu_freq, int cpu_div)
+{
+ if (cpu_freq < 850)
+ return ((cpu_div + 1) * 2) - 1;
+ else
+ return (cpu_div + 1) * 2;
+}
+
+/* Frequency calculation by PLL configuration
+ * @cfg: struct freq_reg_table with pll config information.
+ * Return: freq MHz.
+ *
+ * This returns current frequency in MHz unit
+ */
+static int mnh_freq_get_by_pll(struct freq_reg_table cfg, int sys200)
+{
+ uint32_t freq_khz;
+
+ if (sys200)
+ freq_khz = (SYS200_CLK_KHZ) / (cfg.clk_div + 1);
+ else
+ freq_khz = (REF_CLK_KHZ * cfg.fbdiv) /
+ ((cfg.postdiv1 * cfg.postdiv2) * (cfg.clk_div + 1));
+
+ return (freq_khz / 1000);
+}
+
+static int mnh_get_cpu_pllcfg(struct freq_reg_table *pllcfg)
+{
+ int sys200;
+
+ sys200 = SCU_INf(CCU_CLK_CTL, CPU_IPU_SYS200_MODE);
+
+ pllcfg->fbdiv = SCU_INf(CPU_IPU_PLL_INTGR_DIV, FBDIV);
+ pllcfg->postdiv1 = SCU_INf(CPU_IPU_PLL_INTGR_DIV, POSTDIV1);
+ pllcfg->postdiv2 = SCU_INf(CPU_IPU_PLL_INTGR_DIV, POSTDIV2);
+ pllcfg->clk_div = SCU_INf(CCU_CLK_DIV, CPU_CLK_DIV);
+
+ return mnh_freq_get_by_pll(*pllcfg, sys200);
+}
+
+static int mnh_get_ipu_pllcfg(struct freq_reg_table *pllcfg)
+{
+ int sys200, clk_src;
+
+ clk_src = SCU_INf(CCU_CLK_CTL, IPU_CLK_SRC);
+ sys200 = SCU_INf(CCU_CLK_CTL, CPU_IPU_SYS200_MODE);
+ if (sys200 && (clk_src == IPU_PLL))
+ sys200 = 0;
+
+ if (clk_src == CPU_IPU_PLL) {
+ pllcfg->fbdiv = SCU_INf(CPU_IPU_PLL_INTGR_DIV, FBDIV);
+ pllcfg->postdiv1 = SCU_INf(CPU_IPU_PLL_INTGR_DIV, POSTDIV1);
+ pllcfg->postdiv2 = SCU_INf(CPU_IPU_PLL_INTGR_DIV, POSTDIV2);
+ } else {
+ pllcfg->fbdiv = SCU_INf(IPU_PLL_INTGR_DIV, FBDIV);
+ pllcfg->postdiv1 = SCU_INf(IPU_PLL_INTGR_DIV, POSTDIV1);
+ pllcfg->postdiv2 = SCU_INf(IPU_PLL_INTGR_DIV, POSTDIV2);
+ }
+ pllcfg->clk_div = SCU_INf(CCU_CLK_DIV, IPU_CLK_DIV);
+
+ return mnh_freq_get_by_pll(*pllcfg, sys200);
+}
+
+static int mnh_get_cpu_freq(void)
+{
+ struct freq_reg_table pllcfg;
+
+ return mnh_get_cpu_pllcfg(&pllcfg);
+}
+
+static int mnh_get_ipu_freq(void)
+{
+ struct freq_reg_table pllcfg;
+
+ return mnh_get_ipu_pllcfg(&pllcfg);
+}
+
+int mnh_cpu_freq_to_index(void)
+{
+ int fbdiv, postdiv1, postdiv2, clk_div;
+ int sys200, i;
+
+ sys200 = SCU_INf(CCU_CLK_CTL, CPU_IPU_SYS200_MODE);
+ if (sys200)
+ return 0;
+
+ fbdiv = SCU_INf(CPU_IPU_PLL_INTGR_DIV, FBDIV);
+ postdiv1 = SCU_INf(CPU_IPU_PLL_INTGR_DIV, POSTDIV1);
+ postdiv2 = SCU_INf(CPU_IPU_PLL_INTGR_DIV, POSTDIV2);
+ clk_div = SCU_INf(CCU_CLK_DIV, CPU_CLK_DIV);
+
+ for (i = 0; i < ARRAY_SIZE(cpu_reg_table); i++) {
+ if ((fbdiv == cpu_reg_table[i].fbdiv) &&
+ (postdiv1 == cpu_reg_table[i].postdiv1) &&
+ (postdiv2 == cpu_reg_table[i].postdiv2) &&
+ (clk_div == cpu_reg_table[i].clk_div))
+ return i;
+ }
+
+ return -EINVAL;
+}
+
+int mnh_ipu_freq_to_index(void)
+{
+ int fbdiv, postdiv1, postdiv2, clk_div;
+ int sys200, ipu_clk_src;
+ int i;
+ const struct freq_reg_table *table = ipu_reg_table;
+
+ sys200 = SCU_INf(CCU_CLK_CTL, CPU_IPU_SYS200_MODE);
+ if (sys200)
+ return 0;
+
+ ipu_clk_src = SCU_INf(CCU_CLK_CTL, IPU_CLK_SRC);
+ if (ipu_clk_src == CPU_IPU_PLL)
+ return mnh_cpu_freq_to_index();
+
+ fbdiv = SCU_INf(IPU_PLL_INTGR_DIV, FBDIV);
+ postdiv1 = SCU_INf(IPU_PLL_INTGR_DIV, POSTDIV1);
+ postdiv2 = SCU_INf(IPU_PLL_INTGR_DIV, POSTDIV2);
+ clk_div = SCU_INf(CCU_CLK_DIV, IPU_CLK_DIV);
+
+ for (i = 0; i < ARRAY_SIZE(ipu_reg_table); i++) {
+ if ((fbdiv == table[i].fbdiv) &&
+ (postdiv1 == table[i].postdiv1) &&
+ (postdiv2 == table[i].postdiv2) &&
+ (clk_div == table[i].clk_div))
+ return i;
+ }
+
+ return -EINVAL;
+}
+
+/**
+ * CPU clock controller
+ * @index: int with frquency table index info.
+ * Return: 0 on success, an error code otherwise.
+ *
+ * 1PLL(CPU_IPU PLL) for CPU/IPU clocks. Since CPU and IPU clocks are derived
+ * from same PLL in this mode, there would be restriction on achedivable clock
+ * frequencies for CPU and IPU clocks. IPU clock would be half of CPU clock. Any
+ * frequency changes is achieved by changing FBDIV(integer feedback division) of
+ * the PLL(PLL output = FBDIV * REFCLK frequency).
+ * Default CPU_CLK_DIV : 1, IPU_CLK_DIV: 2
+ * CLK = (REFCLK FREQ * FBDIV) / ((POSTDIV1 * POSTDIV2) * (CLK_DIV + 1))
+ */
+int mnh_cpu_freq_change(int index)
+{
+ int lock = 0;
+ int timeout = 0;
+ int sys200, ipu_div = 0;
+ const struct freq_reg_table *pllcfg;
+ bool use_sys200;
+ int ipu_clk_src;
+ int cpu_freq, old_cpu_freq;
+ int clk_div;
+
+ if (!mnh_clk)
+ return -ENODEV;
+
+ if (index < CPU_FREQ_MIN || index > CPU_FREQ_MAX)
+ return -EINVAL;
+
+ dev_dbg(mnh_clk->dev, "%s: %d\n", __func__, index);
+
+ /* Get PLL config from reg table */
+ pllcfg = &cpu_reg_table[index];
+
+ /* Get current SYS200 mode */
+ sys200 = SCU_INf(CCU_CLK_CTL, CPU_IPU_SYS200_MODE);
+
+ /* Calculate frequency from PLL config */
+ old_cpu_freq = mnh_get_cpu_freq();
+ cpu_freq = mnh_freq_get_by_pll(*pllcfg, 0);
+
+ /* Check to see if we need to change the frequency */
+ if (old_cpu_freq == cpu_freq) {
+ dev_dbg(mnh_clk->dev, "%s: already at desired frequency\n",
+ __func__);
+ return 0;
+ }
+
+ /* Check to see if we should use SYS200 mode */
+ if (cpu_freq == 200) {
+ use_sys200 = true;
+ clk_div = 0;
+ } else {
+ use_sys200 = false;
+ clk_div = pllcfg->clk_div;
+ }
+
+ /* Unlock PLL access */
+ SCU_OUTf(PLL_PASSCODE, PASSCODE, PLL_UNLOCK);
+
+ /* Switch to SYS200 mode */
+ SCU_OUTf(CCU_CLK_CTL, CPU_IPU_SYS200_MODE, 0x1);
+
+ /* Read current IPU clock source */
+ ipu_clk_src = SCU_INf(CCU_CLK_CTL, IPU_CLK_SRC);
+
+ /* Latch current settings going to PLL */
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FRZ_PLL_IN, 1);
+
+ /* Configure CPU PLL */
+ if (use_sys200) {
+ /* Power down PLL and PLL output */
+ SCU_OUTf(CPU_IPU_PLL_CTRL, PD, 1);
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FOUTPOSTDIVPD, 1);
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FRZ_PLL_IN, 0);
+ } else {
+ /* Configure FBDIV first and set POSTDIV1, POSTDIV2
+ * Compute dividers based on REF_FREQ_SEL hardware strap
+ * Check FBDIV * REFCLK is witin VCO range (950-3800MHz)
+ */
+ SCU_OUTf(CPU_IPU_PLL_INTGR_DIV, FBDIV, pllcfg->fbdiv);
+ SCU_OUTf(CPU_IPU_PLL_INTGR_DIV, POSTDIV1, pllcfg->postdiv1);
+ SCU_OUTf(CPU_IPU_PLL_INTGR_DIV, POSTDIV2, pllcfg->postdiv2);
+
+ /* Set FOUTPOSTDIVPD = 1 to avoid glitches to output */
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FOUTPOSTDIVPD, 1);
+
+ /* Apply the updated PLL configurations */
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FRZ_PLL_IN, 0);
+
+ /* Power up PLL */
+ SCU_OUTf(CPU_IPU_PLL_CTRL, PD, 0);
+
+ /* Wait for minimum 128REFCLK, 6.7usec for 19.2MHz refclk
+ * before checking PLL lock
+ */
+ udelay(7);
+
+ /* Check PLL is locked */
+ do {
+ lock = SCU_INf(CPU_IPU_PLL_STS, LOCK);
+ } while ((++timeout < PLL_LOCK_TIMEOUT) && (lock != 1));
+
+ if ((timeout >= PLL_LOCK_TIMEOUT) && (lock != 1))
+ goto fail_cpu_pll_lock;
+
+ /* Set FOUTPOSTDIVPD = 0 to ensure clk output is un-gated */
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FOUTPOSTDIVPD, 0);
+ }
+
+ /* Configure CPU_CLK_DIV */
+ SCU_OUTf(CCU_CLK_DIV, CPU_CLK_DIV, clk_div);
+
+ /* If IPU clock is driven by CPU_IPU PLL,
+ * configure IPU divider based on CPU divider value
+ * to make sure IPU clock does not go over its limit
+ */
+ if (ipu_clk_src == CPU_IPU_PLL) {
+ ipu_div = mnh_clk_get_ipu_div(cpu_freq, clk_div);
+ SCU_OUTf(CCU_CLK_DIV, IPU_CLK_DIV, ipu_div);
+ }
+
+ /* Set final CPU clock source */
+ SCU_OUTf(CCU_CLK_CTL, CPU_IPU_SYS200_MODE, use_sys200);
+
+ /* Lock PLL access */
+ SCU_OUTf(PLL_PASSCODE, PASSCODE, 0);
+
+ return 0;
+
+fail_cpu_pll_lock:
+ dev_err(mnh_clk->dev, "%s: failed to lock CPU PLL\n", __func__);
+
+ /* Power down the PLL */
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FRZ_PLL_IN, 1);
+ SCU_OUTf(CPU_IPU_PLL_CTRL, PD, 1);
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FOUTPOSTDIVPD, 1);
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FRZ_PLL_IN, 0);
+
+ /* Lock PLL access */
+ SCU_OUTf(PLL_PASSCODE, PASSCODE, 0);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(mnh_cpu_freq_change);
+
+/**
+ * IPU clock controller
+ * @index: int with frquency table index info.
+ * Return: 0 on success, an error code otherwise.
+ *
+ * Until IPU clock is configured, IPU clock is driven from PCIe or CPU_IPU PLL,
+ * and once it is configured by driver, IPU PLL is used to control IPU clock.
+ * To turn off IPU PLL, CPU frequency needs to be set to 200MHz to put
+ * both CPU and IPU into SYS200 mode.
+ */
+int mnh_ipu_freq_change(int index)
+{
+ int timeout = 0;
+ int lock = 0;
+ const struct freq_reg_table *pllcfg;
+ bool use_cpu_clk_src;
+ int ipu_freq, old_ipu_freq, cpu_freq;
+ int clk_div;
+ int ipu_clken;
+
+ if (!mnh_clk)
+ return -ENODEV;
+
+ if (index < IPU_FREQ_MIN || index > IPU_FREQ_MAX)
+ return -EINVAL;
+
+ dev_dbg(mnh_clk->dev, "%s: %d\n", __func__, index);
+
+ /* Get PLL config from reg table */
+ pllcfg = &ipu_reg_table[index];
+
+ /* Calculate frequency from PLL config */
+ old_ipu_freq = mnh_get_ipu_freq();
+ cpu_freq = mnh_get_cpu_freq();
+ ipu_freq = mnh_freq_get_by_pll(*pllcfg, 0);
+
+ /* Check to see if we need to change the frequency */
+ if (old_ipu_freq == ipu_freq) {
+ dev_dbg(mnh_clk->dev, "%s: already at desired frequency\n",
+ __func__);
+ return 0;
+ }
+
+ /* TODO: determine if we can get to IPU frequency from CPU frequency */
+ if ((cpu_freq == 200) && (ipu_freq == 100)) {
+ use_cpu_clk_src = true;
+ clk_div = 1;
+ } else if ((cpu_freq == 200) && (ipu_freq == 200)) {
+ use_cpu_clk_src = true;
+ clk_div = 0;
+ } else {
+ use_cpu_clk_src = false;
+ clk_div = pllcfg->clk_div;
+ }
+
+ /* Disable IPU_PLL clock output */
+ ipu_clken = SCU_INf(CCU_CLK_CTL, IPU_CLKEN);
+ SCU_OUTf(CCU_CLK_CTL, IPU_CLKEN, 0x0);
+
+ /* Unlock PLL access */
+ SCU_OUTf(PLL_PASSCODE, PASSCODE, PLL_UNLOCK);
+
+ /* Switch to stable clock before freq switch to avoid glitches */
+ SCU_OUTf(CCU_CLK_CTL, IPU_CLK_SRC, CPU_IPU_PLL);
+
+ /* Set FRZ_PLL_IN=1 to latch the current settings going to PLL */
+ SCU_OUTf(IPU_PLL_CTRL, FRZ_PLL_IN, 1);
+
+ /* Configure IPU PLL */
+ if (use_cpu_clk_src) {
+ SCU_OUTf(IPU_PLL_CTRL, PD, 1);
+ SCU_OUTf(IPU_PLL_CTRL, FOUTPOSTDIVPD, 1);
+ SCU_OUTf(IPU_PLL_CTRL, FRZ_PLL_IN, 0);
+ } else {
+ /* Configure FBDIV first and set POSTDIV1, POSTDIV2 */
+ SCU_OUTf(IPU_PLL_INTGR_DIV, FBDIV, pllcfg->fbdiv);
+ SCU_OUTf(IPU_PLL_INTGR_DIV, POSTDIV1, pllcfg->postdiv1);
+ SCU_OUTf(IPU_PLL_INTGR_DIV, POSTDIV2, pllcfg->postdiv2);
+
+ /* Set FOUTPOSTDIVPD = 1 to avoid glitches to output */
+ SCU_OUTf(IPU_PLL_CTRL, FOUTPOSTDIVPD, 1);
+
+ /* Apply the updated PLL configurations */
+ SCU_OUTf(IPU_PLL_CTRL, FRZ_PLL_IN, 0);
+
+ /* Power up PLL */
+ SCU_OUTf(IPU_PLL_CTRL, PD, 0);
+
+ /* Wait for minimum 128REFCLK, 6.7usec for 19.2MHz refclk
+ * before checking PLL lock
+ */
+ udelay(7);
+
+ /* Check PLL is locked */
+ do {
+ lock = SCU_INf(IPU_PLL_STS, LOCK);
+ } while ((++timeout < PLL_LOCK_TIMEOUT) && (lock != 1));
+
+ if ((timeout >= PLL_LOCK_TIMEOUT) && (lock != 1))
+ goto fail_ipu_pll_lock;
+
+ /* Set FOUTPOSTDIVPD = 0 to ensure clk output is un-gated */
+ SCU_OUTf(IPU_PLL_CTRL, FOUTPOSTDIVPD, 0);
+ }
+
+ /* Configure IPU_CLK_DIV */
+ SCU_OUTf(CCU_CLK_DIV, IPU_CLK_DIV, clk_div);
+
+ /* Go back to IPU PLL output */
+ if (!use_cpu_clk_src)
+ SCU_OUTf(CCU_CLK_CTL, IPU_CLK_SRC, IPU_PLL);
+
+ /* Lock PLL access */
+ SCU_OUTf(PLL_PASSCODE, PASSCODE, 0);
+
+ /* Enable IPU_PLL clock output */
+ SCU_OUTf(CCU_CLK_CTL, IPU_CLKEN, ipu_clken);
+
+ return 0;
+
+fail_ipu_pll_lock:
+ dev_err(mnh_clk->dev, "%s: failed to lock IPU PLL\n", __func__);
+
+ /* Power down IPU PLL */
+ SCU_OUTf(IPU_PLL_CTRL, FRZ_PLL_IN, 1);
+ SCU_OUTf(IPU_PLL_CTRL, PD, 1);
+ SCU_OUTf(IPU_PLL_CTRL, FOUTPOSTDIVPD, 1);
+ SCU_OUTf(IPU_PLL_CTRL, FRZ_PLL_IN, 0);
+
+ /* Lock PLL access */
+ SCU_OUTf(PLL_PASSCODE, PASSCODE, 0);
+
+ return -EBUSY;
+}
+EXPORT_SYMBOL_GPL(mnh_ipu_freq_change);
+
+/**
+ * LPDDR clock control driver
+ * @index: int with frquency table index info.
+ * Return: 0 on success, an error code otherwise.
+ *
+ * LPDDR clock is controlled by LPC instead of using direct PLL configuration.
+ * Precondition:
+ * LPDDR refclk pll should be enabled at cold boot and resume
+ * LPDDR FSPx registers should be configured at cold boot and resume
+ */
+int mnh_lpddr_freq_change(int index)
+{
+ int status = 0;
+ int timeout = 0;
+ enum mnh_lpddr_freq_type ddr_freq;
+
+ if (!mnh_clk)
+ return -ENODEV;
+
+ dev_dbg(mnh_clk->dev, "%s: %d\n", __func__, index);
+
+ if (index < LPDDR_FREQ_MIN || index > LPDDR_FREQ_MAX)
+ return -EINVAL;
+
+ /* Check the requested FSP is already in use */
+ ddr_freq = SCU_INf(LPDDR4_LOW_POWER_STS, LPDDR4_CUR_FSP);
+ if (ddr_freq == index) {
+ dev_dbg(mnh_clk->dev, "%s: requested fsp%d is in use\n",
+ __func__, index);
+ return 0;
+ }
+
+ /* Power up LPDDR PLL if the FSP setting uses it */
+ if (!SCU_INxf(LPDDR4_FSP_SETTING, index, FSP_SYS200_MODE))
+ mnh_lpddr_sys200_mode(false);
+
+ /* Disable LPC SW override */
+ SCU_OUTf(LPDDR4_LOW_POWER_CFG, LP4_FSP_SW_OVERRIDE, 0);
+
+ /* Configure FSP index */
+ SCU_OUTf(LPDDR4_LOW_POWER_CFG, LPC_FREQ_CHG_COPY_NUM, index);
+
+ /* Configure LPC cmd for frequency switch */
+ SCU_OUTf(LPDDR4_LOW_POWER_CFG, LPC_EXT_CMD, LP4_LPC_FREQ_SWITCH);
+
+ /* Initiate LPC cmd to LPDDR controller */
+ dev_dbg(mnh_clk->dev, "%s: lpddr freq switching from fsp%d to fsp%d\n",
+ __func__, ddr_freq, index);
+ SCU_OUTf(LPDDR4_LOW_POWER_CFG, LPC_EXT_CMD_REQ, 1);
+
+ /* Wait until LPC cmd process is done */
+ do {
+ status = SCU_INf(LPDDR4_LOW_POWER_STS, LPC_CMD_DONE);
+ } while ((++timeout < PLL_LOCK_TIMEOUT) && (status != 1));
+
+ /* Clear LPC cmd status */
+ SCU_OUTf(LPDDR4_LOW_POWER_STS, LPC_CMD_DONE, 1);
+
+ /* Check LPC error status */
+ if (SCU_INf(LPDDR4_LOW_POWER_STS, LPC_CMD_RSP) == LPC_CMD_ERR) {
+ /* Clear error status */
+ SCU_OUTf(LPDDR4_LOW_POWER_STS, LPC_CMD_RSP, 1);
+ dev_err(mnh_clk->dev, "Failed to process lpc cmd:0x%x\n",
+ LP4_LPC_FREQ_SWITCH);
+ return -EIO;
+ }
+
+ /* Check FSPx switch status */
+ if (SCU_INf(LPDDR4_LOW_POWER_STS, LPDDR4_CUR_FSP) != index) {
+ dev_err(mnh_clk->dev, "Failed to switch to fsp%d\n", index);
+ return -EIO;
+ }
+
+ /* Power down LPDDR PLL if the FSP setting doesn't use it */
+ if (SCU_INxf(LPDDR4_FSP_SETTING, index, FSP_SYS200_MODE))
+ mnh_lpddr_sys200_mode(true);
+
+ mnh_ddr_clr_int_status(mnh_clk->dev);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_lpddr_freq_change);
+
+/**
+ * LPDDR clock control driver
+ * Return: 0 on success, an error code otherwise.
+ *
+ * LPDDR clock is derived from sys200 clk instead of separate lpddr clk
+ */
+int mnh_lpddr_sys200_mode(bool enable)
+{
+ int lock;
+ int timeout = 0;
+
+ /* Switch lpddr to SYS200 mode */
+ SCU_OUTf(CCU_CLK_CTL, LP4_AXI_SYS200_MODE, 0x1);
+
+ dev_dbg(mnh_clk->dev, "%s: %d\n", __func__, enable);
+ /* Unlock PLL access */
+ SCU_OUTf(PLL_PASSCODE, PASSCODE, PLL_UNLOCK);
+
+ if (enable) {
+ /* Power down LPDDR PLL */
+ SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, PD, 1);
+ SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, FOUTPOSTDIVPD, 1);
+ SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, BYPASS, 1);
+ } else {
+ /* Power up LPDDR PLL */
+ SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, FRZ_PLL_IN, 1);
+ SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, PD, 0);
+ SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, FOUTPOSTDIVPD, 0);
+ SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, BYPASS, 0);
+ SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, FRZ_PLL_IN, 0);
+
+ /* Check PLL is locked */
+ do {
+ lock = SCU_INf(LPDDR4_REFCLK_PLL_STS, LOCK);
+ } while ((++timeout < PLL_LOCK_TIMEOUT) && (lock != 1));
+ }
+
+ /* Lock PLL */
+ SCU_OUTf(PLL_PASSCODE, PASSCODE, 0);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_lpddr_sys200_mode);
+
+/**
+ * CPU and IPU SYS200 clock control driver
+ * Return: 0 on success, an error code otherwise.
+ *
+ * CPU and IPU clock is derived from sys200 clk instead of separate plls
+ */
+int mnh_cpu_ipu_sys200_mode(void)
+{
+ int ipu_clk_src;
+
+ if (!mnh_clk)
+ return -ENODEV;
+
+ dev_dbg(mnh_clk->dev, "%s\n", __func__);
+
+ /* Unlock PLL access */
+ SCU_OUTf(PLL_PASSCODE, PASSCODE, PLL_UNLOCK);
+
+ /* Switch to SYS200 mode */
+ SCU_OUTf(CCU_CLK_CTL, CPU_IPU_SYS200_MODE, 0x1);
+
+ /* Read current IPU clock source */
+ ipu_clk_src =
+ SCU_INf(CCU_CLK_CTL, IPU_CLK_SRC);
+
+ if (ipu_clk_src == IPU_PLL) {
+ /* Change clk source to CPU_IPU_PLL */
+ SCU_OUTf(CCU_CLK_CTL, IPU_CLK_SRC, CPU_IPU_PLL);
+
+ /* IPU: Latch current settings to go into PLL */
+ SCU_OUTf(IPU_PLL_CTRL, FRZ_PLL_IN, 1);
+
+ /* IPU: Power down PLL */
+ SCU_OUTf(IPU_PLL_CTRL, PD, 1);
+ SCU_OUTf(IPU_PLL_CTRL, FOUTPOSTDIVPD, 1);
+
+ /* IPU: Apply PLL configurations */
+ SCU_OUTf(IPU_PLL_CTRL, FRZ_PLL_IN, 0);
+ }
+
+ /* Configure IPU_CLK_DIV */
+ SCU_OUTf(CCU_CLK_DIV, IPU_CLK_DIV, sys200_reg_table[IPU_CLK].clk_div);
+
+ /* Configure CPU_CLK_DIV */
+ SCU_OUTf(CCU_CLK_DIV, CPU_CLK_DIV, sys200_reg_table[CPU_CLK].clk_div);
+
+ /* CPU_IPU: Latch current settings to go into PLL */
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FRZ_PLL_IN, 1);
+
+ /* CPU_IPU: Power down PLL */
+ SCU_OUTf(CPU_IPU_PLL_CTRL, PD, 1);
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FOUTPOSTDIVPD, 1);
+
+ /* CPU_IPU: Apply PLL configurations */
+ SCU_OUTf(CPU_IPU_PLL_CTRL, FRZ_PLL_IN, 0);
+
+ /* Lock PLL access */
+ SCU_OUTf(PLL_PASSCODE, PASSCODE, 0);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_cpu_ipu_sys200_mode);
+
+/**
+ * Set the SCU clock gating at init
+ * Return: 0 on success, an error code otherwise.
+ *
+ * enable == 1 sets the SCU to enable clock gating in general when CPU enters
+ * L2 WFI state.
+ */
+int mnh_clock_init_gating(bool enabled)
+{
+ if (!mnh_clk)
+ return -ENODEV;
+
+ dev_dbg(mnh_clk->dev, "%s:%d\n", __func__, __LINE__);
+
+ /* Add periph clk gates */
+ SCU_OUTf(RSTC, PERI_DMA_RST, enabled);
+
+ SCU_OUTf(PERIPH_CLK_CTRL, PERI_DMA_CLKEN_SW, !enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, HALT_BTROM_PD_EN, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, HALT_BTSRAM_PD_EN, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, BTROM_SLP, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, BTSRAM_DS, enabled);
+ SCU_OUTf(CCU_CLK_CTL, HALT_BTSRAMCG_EN, enabled);
+ SCU_OUTf(CCU_CLK_CTL, HALT_BTROMCG_EN, enabled);
+
+ SCU_OUTf(CCU_CLK_CTL, HALT_LP4_PLL_BYPCLK_CG_EN, enabled);
+ SCU_OUTf(CCU_CLK_CTL, LP4PHY_PLL_BYPASS_CLKEN, enabled);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_clock_init_gating);
+
+int mnh_bypass_clock_gating(bool enabled)
+{
+ if (!mnh_clk)
+ return -ENODEV;
+
+ dev_dbg(mnh_clk->dev, "%s:%d\n", __func__, __LINE__);
+
+ if (enabled) {
+ SCU_OUTf(CCU_CLK_CTL, HALT_CPUCG_EN, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, HALT_CPUMEM_PD_EN, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, CPU_L2MEM_DS, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, CPU_L1MEM_DS, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, HALT_LP4CMEM_PD_EN, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, LP4C_MEM_DS, enabled);
+ } else {
+ SCU_OUTf(MEM_PWR_MGMNT, CPU_L2MEM_DS, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, CPU_L1MEM_DS, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, HALT_CPUMEM_PD_EN, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, LP4C_MEM_DS, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, HALT_LP4CMEM_PD_EN, enabled);
+ SCU_OUTf(CCU_CLK_CTL, HALT_CPUCG_EN, enabled);
+ }
+
+ SCU_OUTf(RSTC, WDT_RST, enabled);
+
+ SCU_OUTf(PERIPH_CLK_CTRL, PVT_CLKEN, !enabled);
+ SCU_OUTf(PERIPH_CLK_CTRL, WDT_CLKEN_SW, !enabled);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_bypass_clock_gating);
+
+/**
+ * Set the SCU clock gating for bypass mode
+ * Return: 0 on success, an error code otherwise.
+ *
+ * enable == 1 sets the SCU to enable clock gating in general when CPU enters
+ * L2 WFI state.
+ */
+int mnh_ipu_clock_gating(bool enabled)
+{
+ if (!mnh_clk)
+ return -ENOENT;
+
+ dev_dbg(mnh_clk->dev, "%s\n", __func__);
+
+ if (enabled) {
+ SCU_OUTf(CCU_CLK_CTL, IPU_CLKEN, !enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, IPU_MEM_DS, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, IPU_MEM_SD, enabled);
+ } else {
+ SCU_OUTf(MEM_PWR_MGMNT, IPU_MEM_SD, enabled);
+ SCU_OUTf(MEM_PWR_MGMNT, IPU_MEM_DS, enabled);
+ SCU_OUTf(CCU_CLK_CTL, IPU_CLKEN, !enabled);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_ipu_clock_gating);
+
+int mnh_ipu_reset(void)
+{
+ dev_dbg(mnh_clk->dev, "%s\n", __func__);
+ if (!mnh_clk)
+ return -ENOENT;
+
+ SCU_OUTf(RSTC, IPU_RST, 1);
+ SCU_OUTf(RSTC, IPU_RST, 0);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_ipu_reset);
+
+static ssize_t cpu_freq_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%dMHz\n", mnh_get_cpu_freq());
+}
+
+static ssize_t cpu_freq_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ int i, err;
+
+ dev_dbg(mnh_clk->dev, "%s: %s\n", __func__, buf);
+ for (i = 0; i < ARRAY_SIZE(cpu_reg_table); i++) {
+ if (!strncmp(buf, cpu_reg_table[i].freq_str,
+ strlen(cpu_reg_table[i].freq_str))) {
+ err = mnh_cpu_freq_change(i);
+ if (!err)
+ return count;
+ else
+ return -EIO;
+ }
+ }
+
+ dev_err(mnh_clk->dev, "invalid freq: %s\n", buf);
+ return -EINVAL;
+
+}
+static DEVICE_ATTR_RW(cpu_freq);
+
+static ssize_t ipu_freq_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%dMHz\n", mnh_get_ipu_freq());
+}
+
+static ssize_t ipu_freq_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ int i, err;
+
+ dev_dbg(mnh_clk->dev, "%s: %s\n", __func__, buf);
+ for (i = 0; i < ARRAY_SIZE(ipu_reg_table); i++) {
+ if (!strncmp(buf, ipu_reg_table[i].freq_str,
+ strlen(ipu_reg_table[i].freq_str))) {
+ err = mnh_ipu_freq_change(i);
+ if (!err)
+ return count;
+ else
+ return -EIO;
+ }
+ }
+
+ dev_err(mnh_clk->dev, "invalid freq: %s\n", buf);
+ return -EINVAL;
+}
+static DEVICE_ATTR_RW(ipu_freq);
+
+static ssize_t lpddr_freq_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ uint32_t var = SCU_INf(LPDDR4_LOW_POWER_STS,
+ LPDDR4_CUR_FSP);
+
+ dev_dbg(mnh_clk->dev, "%s: %d\n", __func__, var);
+ return snprintf(buf, PAGE_SIZE, "FSP%d\n", var);
+}
+
+static ssize_t lpddr_freq_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ int var;
+ int ret;
+
+ ret = kstrtoint(buf, 10, &var);
+ if (ret < 0)
+ return ret;
+
+ dev_dbg(mnh_clk->dev, "%s: %d\n", __func__, var);
+ if (!mnh_lpddr_freq_change(var))
+ return count;
+ else
+ return -EIO;
+}
+static DEVICE_ATTR_RW(lpddr_freq);
+
+static ssize_t ipu_clk_src_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ int ipu_clk_src = SCU_INf(CCU_CLK_CTL, IPU_CLK_SRC);
+
+ return snprintf(buf, PAGE_SIZE, "%s\n",
+ (ipu_clk_src == CPU_IPU_PLL) ? "CPU_IPU":"IPU");
+}
+static DEVICE_ATTR_RO(ipu_clk_src);
+
+static ssize_t sys200_mode_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ int sys200, clk_src;
+
+ clk_src = SCU_INf(CCU_CLK_CTL, IPU_CLK_SRC);
+ sys200 = SCU_INf(CCU_CLK_CTL, CPU_IPU_SYS200_MODE);
+ if (sys200 && (clk_src == IPU_PLL))
+ sys200 = 0;
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", sys200);
+}
+
+static ssize_t sys200_mode_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ bool var;
+ int ret;
+
+ ret = kstrtobool(buf, &var);
+ if (ret < 0)
+ return ret;
+
+ if (var) {
+ dev_dbg(mnh_clk->dev, "%s: %d\n", __func__, var);
+ if (!mnh_cpu_ipu_sys200_mode())
+ return count;
+ }
+
+ return -EIO;
+}
+static DEVICE_ATTR_RW(sys200_mode);
+
+static ssize_t ipu_clock_gating_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ int clk_gated;
+
+ clk_gated = !SCU_INf(CCU_CLK_CTL, IPU_CLKEN);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", clk_gated);
+}
+
+static ssize_t ipu_clock_gating_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ bool var;
+ int ret;
+
+ ret = kstrtobool(buf, &var);
+ if (ret < 0)
+ return ret;
+
+ dev_dbg(mnh_clk->dev, "%s: %d\n", __func__, var);
+ if (!mnh_ipu_clock_gating(var))
+ return count;
+
+ return -EIO;
+}
+
+static DEVICE_ATTR_RW(ipu_clock_gating);
+
+static ssize_t bypass_clock_gating_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ int clk_gated;
+
+ clk_gated = SCU_INf(CCU_CLK_CTL, HALT_CPUCG_EN);
+
+ return snprintf(buf, PAGE_SIZE, "%d\n", clk_gated);
+}
+
+static ssize_t bypass_clock_gating_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ bool var;
+ int ret;
+
+ ret = kstrtobool(buf, &var);
+ if (ret < 0)
+ return ret;
+
+ dev_dbg(mnh_clk->dev, "%s: %d\n", __func__, var);
+ if (!mnh_bypass_clock_gating(var))
+ return count;
+
+ return -EIO;
+}
+static DEVICE_ATTR_RW(bypass_clock_gating);
+
+static struct attribute *mnh_clk_attrs[] = {
+ &dev_attr_cpu_freq.attr,
+ &dev_attr_ipu_freq.attr,
+ &dev_attr_lpddr_freq.attr,
+ &dev_attr_ipu_clk_src.attr,
+ &dev_attr_sys200_mode.attr,
+ &dev_attr_ipu_clock_gating.attr,
+ &dev_attr_bypass_clock_gating.attr,
+ NULL
+};
+
+static struct attribute_group mnh_clk_attr_group = {
+ .name = "mnh_clk",
+ .attrs = mnh_clk_attrs
+};
+
+static int init_sysfs(struct device *dev, struct kobject *sysfs_kobj)
+{
+ int ret;
+
+ ret = sysfs_create_group(sysfs_kobj, &mnh_clk_attr_group);
+ if (ret) {
+ dev_err(dev, "Failed to create sysfs group (%d)\n", ret);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void clean_sysfs(struct kobject *sysfs_kobj)
+{
+ sysfs_remove_group(sysfs_kobj, &mnh_clk_attr_group);
+}
+
+int mnh_clk_init(struct device *dev, uint32_t baseadress)
+{
+ dev_dbg(dev, "%s\n", __func__);
+
+ mnh_clk = devm_kzalloc(dev, sizeof(*mnh_clk), GFP_KERNEL);
+ if (!mnh_clk)
+ return -ENOMEM;
+
+ mnh_clk->dev = dev;
+
+ init_sysfs(dev, &dev->kobj);
+
+ return 0;
+}
+
+void mnh_clk_clean(struct device *dev)
+{
+ clean_sysfs(&dev->kobj);
+}
diff --git a/drivers/misc/mnh/mnh-clk.h b/drivers/misc/mnh/mnh-clk.h
new file mode 100644
index 0000000..0d17063
--- /dev/null
+++ b/drivers/misc/mnh/mnh-clk.h
@@ -0,0 +1,157 @@
+/*
+ *
+ * MNH Clock Driver
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef __MNH_CLK
+#define __MNH_CLK
+
+#include <linux/device.h>
+
+enum mnh_cpu_freq_type {
+ CPU_FREQ_MIN = 0,
+ CPU_FREQ_200 = CPU_FREQ_MIN,
+ CPU_FREQ_400,
+ CPU_FREQ_600,
+ CPU_FREQ_800,
+ CPU_FREQ_950,
+ CPU_FREQ_MAX = CPU_FREQ_950
+};
+
+enum mnh_ipu_freq_type {
+ IPU_FREQ_MIN = 0,
+ IPU_FREQ_100 = IPU_FREQ_MIN,
+ IPU_FREQ_200,
+ IPU_FREQ_300,
+ IPU_FREQ_400,
+ IPU_FREQ_425,
+ IPU_FREQ_MAX = IPU_FREQ_425
+};
+
+enum mnh_lpddr_freq_type {
+ LPDDR_FREQ_MIN = 0,
+ LPDDR_FREQ_FSP0 = LPDDR_FREQ_MIN,
+ LPDDR_FREQ_FSP1,
+ LPDDR_FREQ_FSP2,
+ LPDDR_FREQ_FSP3,
+ LPDDR_FREQ_MAX = LPDDR_FREQ_FSP3,
+ LPDDR_FREQ_SW = 7
+};
+
+/**
+ * CPU clock controller
+ * @index: int with frequency table index info.
+ * Return: 0 on success, an error code otherwise.
+ *
+ * 1PLL(CPU_IPU PLL) for CPU/IPU clocks. Since CPU and IPU clocks are derived
+ * from same PLL in this mode, there would be restriction on achedivable clock
+ * frequencies for CPU and IPU clocks. IPU clock would be half of CPU clock. Any
+ * frequency changes is achieved by changing FBDIV(integer feedback division) of
+ * the PLL(PLL output = FBDIV * REFCLK frequency).
+ * Default CPU_CLK_DIV : 1, IPU_CLK_DIV: 2
+ * CLK = (REFCLK FREQ * FBDIV) / ((POSTDIV1 * POSTDIV2) * (CLK_DIV + 1))
+ */
+int mnh_cpu_freq_change(int index);
+
+/**
+ * IPU clock controller
+ * @index: int with frequency table index info.
+ * Return: 0 on success, an error code otherwise.
+ *
+ * Until IPU clock is configured, IPU clock is driven from PCIe or CPU_IPU PLL,
+ * and once it is configured by driver, IPU PLL is used to control IPU clock.
+ * To turn off IPU PLL, CPU frequency needs to be set to 200MHz to put
+ * both CPU and IPU into SYS200 mode.
+ */
+int mnh_ipu_freq_change(int index);
+
+/**
+ * LPDDR clock control driver
+ * @index: int with frequency table index info.
+ * Return: 0 on success, an error code otherwise.
+ *
+ * LPDDR clock is controlled by LPC instead of using direct PLL configuration.
+ * Precondition:
+ * LPDDR refclk pll should be enabled at cold boot and resume
+ * LPDDR FSPx registers should be configured at cold boot and resume
+ */
+int mnh_lpddr_freq_change(int index);
+
+/**
+ * Enable LPDDR sys200 mode
+ * @enable: bool flag
+ * Return: 0 on success, an error code otherwise.
+ *
+ * LPDDR clock is derived from sys200 clk instead of separate lpddr clk
+ */
+int mnh_lpddr_sys200_mode(bool enable);
+
+/**
+ * Enable CPU/IPU sys200 mode
+ * @enable: bool flag
+ * Return: 0 on success, an error code otherwise.
+ *
+ * CPU and IPU clocks are derived from sys200 clk instead of CPU_IPU PLL
+ */
+int mnh_cpu_ipu_sys200_mode(void);
+
+/**
+ * Initial clock gating
+ * @enable: bool flag
+ * Return: 0 on success, an error code otherwise.
+ *
+ * Enable or disable clock gating of unused peripherals during boot
+ */
+int mnh_clock_init_gating(bool enabled);
+
+/**
+ * Bypass mode clock gating
+ * @enable: bool flag
+ * Return: 0 on success, an error code otherwise.
+ *
+ * Enable or disable clock gating on a subset of peripherals that are not used
+ * during bypass mode
+ */
+int mnh_bypass_clock_gating(bool enabled);
+
+/**
+ * IPU clock gating
+ * @enable: bool flag
+ * Return: 0 on success, an error code otherwise.
+ *
+ * Enable or disable IPU clock gating for power savings
+ */
+int mnh_ipu_clock_gating(bool enabled);
+
+/**
+ * IPU reset
+ * Return: 0 on success, an error code otherwise.
+ *
+ * Assert, and then deassert reset to IPU
+ */
+int mnh_ipu_reset(void);
+
+/**
+ * Setup interface for controlling clocks
+ * @dev: device structure.
+ * Return 0 on success, errcode on failures.
+ */
+int mnh_clk_init(struct device *dev, uint32_t baseadress);
+
+/**
+ * Clean up the interface
+ */
+void mnh_clk_clean(struct device *dev);
+
+#endif /* __MNH_CLK */
diff --git a/drivers/misc/mnh/mnh-crypto-sysfs.c b/drivers/misc/mnh/mnh-crypto-sysfs.c
new file mode 100644
index 0000000..421e448
--- /dev/null
+++ b/drivers/misc/mnh/mnh-crypto-sysfs.c
@@ -0,0 +1,97 @@
+/*
+ * MNH Firmware for firmware authentication
+ *
+ * Copyright 2017 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/device.h>
+#include <linux/miscdevice.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include "mnh-crypto.h"
+
+#define MISC_DEVICE_NAME "mnh_crypto"
+
+static ssize_t verify_img_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count);
+
+static void mnh_crypto_decode_error(struct device *dev, int err);
+
+/* Device attributes */
+DEVICE_ATTR_WO(verify_img);
+
+static struct attribute *mnh_crypto_attrs[] = {
+ &dev_attr_verify_img.attr,
+ NULL
+};
+ATTRIBUTE_GROUPS(mnh_crypto);
+
+static struct miscdevice mnh_crypto_miscdevice = {
+ .minor = MISC_DYNAMIC_MINOR,
+ .name = MISC_DEVICE_NAME,
+ .groups = mnh_crypto_groups,
+};
+
+/* echo -n "easel/Image">verify_img */
+static ssize_t verify_img_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int err;
+
+ err = mnh_crypto_verify_fw(dev, buf);
+ if (!err) {
+ dev_info(dev, "%s: Signature verfied OK\n", __func__);
+ return count;
+ }
+
+ mnh_crypto_decode_error(dev, err);
+ return err;
+}
+
+/* create sysfs group */
+int mnh_crypto_config_sysfs(void)
+{
+ int err;
+ struct device *dev;
+
+ err = misc_register(&mnh_crypto_miscdevice);
+ if (err) {
+ pr_err("%s: failed to create misc device (%d)\n",
+ __func__, err);
+ return err;
+ }
+ dev = mnh_crypto_miscdevice.this_device;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_crypto_config_sysfs);
+
+static void mnh_crypto_decode_error(struct device *dev, int err)
+{
+ /* Decode return code information */
+ if (err == -ENOKEY)
+ dev_err(dev, "%s: Required key not available\n", __func__);
+ else if (err == -EKEYEXPIRED)
+ dev_err(dev, "%s: Key has expired\n", __func__);
+ else if (err == -EKEYREVOKED)
+ dev_err(dev, "%s: Key has been revoked\n", __func__);
+ else if (err == -EKEYREJECTED)
+ dev_err(dev, "%s: Key was rejected by service\n", __func__);
+ else if (err == -ENOMEM)
+ dev_err(dev, "%s: Out of memory\n", __func__);
+ else if (err == -ECANCELED)
+ dev_err(dev, "%s: Operation canceled\n", __func__);
+ else
+ dev_err(dev, "%s: Unknown error code %d\n", __func__, err);
+}
diff --git a/drivers/misc/mnh/mnh-crypto.c b/drivers/misc/mnh/mnh-crypto.c
new file mode 100644
index 0000000..38d9ba1
--- /dev/null
+++ b/drivers/misc/mnh/mnh-crypto.c
@@ -0,0 +1,177 @@
+/*
+ * MNH Firmware for firmware authentication
+ *
+ * Copyright 2017 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/types.h>
+#include <crypto/pkcs7.h>
+#include <crypto/public_key.h>
+#include <keys/asymmetric-type.h>
+#include <keys/system_keyring.h>
+#include <linux/device.h>
+#include <linux/firmware.h>
+#include <linux/kernel.h>
+#include <linux/key.h>
+#include <linux/slab.h>
+#include "mnh-crypto.h"
+#include "mnh-sm.h"
+
+#define PKEY_ID_PKCS7 2
+#define ESL_SIG_STRING "~Module signature appended~\n"
+
+/**
+ * Verify signature of a firmware blob in the filesystem
+ * @dev: Device node
+ * @path: path of the firmware file to be verified
+ * RETURNS:
+ * 0 success
+ * -EBADMSG PKCS#7 signature with non-detached data
+ * -ECANCELED invalid PKCS#7 message or signature block is missing
+ * -EKEYREJECTED key had a usage restriction, or signature failed to match
+ * -ENOPKG crypto module could not be found
+ * -ENOKEY PKCS#7 signature not signed with a trusted key
+ */
+int mnh_crypto_verify_fw(struct device *dev, const char *path)
+{
+ struct cert_info *info;
+ const struct firmware *img;
+ int err = 0;
+
+ /* load firmware from filesystem */
+ dev_dbg(dev, "%s loading firmware: %s\n", __func__, path);
+ err = request_firmware(&img, path, dev);
+ if (err) {
+ dev_err(dev, "%s request firmware failed - %d\n",
+ __func__, err);
+ return err;
+ }
+ dev_dbg(dev, "%s firmware loaded\n", __func__);
+
+ info = kzalloc(sizeof(struct cert_info),
+ GFP_KERNEL);
+ if (info == NULL) {
+ release_firmware(img);
+ return -ENOMEM;
+ }
+
+ info->img = img->data;
+ info->img_len = img->size;
+ err = mnh_crypto_verify(dev, info);
+
+ kfree(info);
+ release_firmware(img);
+
+ return err;
+}
+EXPORT_SYMBOL_GPL(mnh_crypto_verify_fw);
+
+/**
+ * Check proper framing of image, signature, header and end-marker
+ * and perform signature check.
+ * @dev: Device node
+ * @info: data structure including buffer address and size
+ * RETURNS:
+ * 0 success
+ * -EBADMSG PKCS#7 signature with non-detached data
+ * -ECANCELED invalid PKCS#7 message or signature block is missing
+ * -EKEYREJECTED key had a usage restriction, or signature failed to match
+ * -ENOPKG crypto module could not be found
+ * -ENOKEY PKCS#7 signature not signed with a trusted key
+ */
+int mnh_crypto_verify(struct device *dev, struct cert_info *info)
+{
+ int err = -ENOKEY;
+ const unsigned long marker_len = sizeof(ESL_SIG_STRING) - 1;
+ const void *img = info->img;
+
+ dev_dbg(dev, "%s starting signature check\n", __func__);
+ /* check if signature string is present at the end of the image */
+ dev_dbg(dev, "%s - buffer addr: %pP | buffer size: %lu\n", __func__,
+ info->img, info->img_len);
+ if (memcmp(img + info->img_len - marker_len,
+ ESL_SIG_STRING, marker_len) == 0) {
+ dev_dbg(dev, "%s ESL_SIG_STRING found, proceeding\n",
+ __func__);
+ info->img_len -= marker_len;
+ /* verify the signature against the available keys */
+ err = mnh_crypto_verify_sig(dev, info);
+ info->cert_size += marker_len;
+ if (!err) {
+ info->cert = FW_IMAGE_CERT_OK;
+ dev_dbg(dev, "%s Image signature verified OK\n",
+ __func__);
+ } else {
+ dev_err(dev, "%s Image signature error: %d\n",
+ __func__, err);
+ info->cert = FW_IMAGE_TAINTED;
+ }
+ } else {
+ info->cert = FW_IMAGE_NO_CERT;
+ dev_err(dev, "%s ESL_SIG_STRING not found, aborting\n",
+ __func__);
+ err = -ECANCELED;
+ }
+ return err;
+}
+EXPORT_SYMBOL_GPL(mnh_crypto_verify);
+
+/*
+ * mnh_crypto_verify_sig checks the validity of the image's signature type
+ * and validates the signature of the image against the available public
+ * keys in the system keychain. Function populates info->cert & ->cert_size.
+ * RETURNS:
+ * 0 success
+ * -EBADMSG PKCS#7 signature with non-detached data
+ * -ECANCELED invalid PKCS#7 message
+ * -EKEYREJECTED key had a usage restriction, or signature failed to match
+ * -ENOPKG crypto module could not be found
+ * -ENOKEY PKCS#7 signature not signed with a trusted key
+ */
+int mnh_crypto_verify_sig(struct device *dev, struct cert_info *info)
+{
+ struct esl_img_signature esl_sig;
+ size_t sig_len;
+ size_t imglen = info->img_len;
+ const void *esl_img = info->img;
+
+ dev_dbg(dev, "mnh %s image size %zu\n", __func__, imglen);
+ if (imglen <= sizeof(esl_sig))
+ return -ECANCELED;
+ /* this structure is appended at the end of the file */
+ memcpy(&esl_sig, esl_img + (imglen - sizeof(esl_sig)),
+ sizeof(esl_sig));
+ /* strip out the esl_img_signature structure at the end */
+ imglen -= sizeof(esl_sig);
+ /* length of the actual signature block */
+ sig_len = be32_to_cpu(esl_sig.sig_len);
+ /* check signature type */
+ if (esl_sig.id_type != PKEY_ID_PKCS7) {
+ dev_err(dev, "mnh %s Image not signed with PKCS#7 message\n",
+ __func__);
+ return -ECANCELED;
+ }
+ /* strip out the signature, all that is left is the payload */
+ if (imglen <= sig_len)
+ return -ECANCELED;
+ imglen -= sig_len;
+ dev_dbg(dev, "mnh %s sizes: payload %zu | signature %zu | ident %zu\n",
+ __func__, imglen, sig_len, sizeof(esl_sig));
+
+ /* total size of signature and header information */
+ info->cert_size = info->img_len - imglen;
+
+ return verify_pkcs7_signature(esl_img, imglen,
+ esl_img + imglen, sig_len,
+ NULL, VERIFYING_MODULE_SIGNATURE,
+ NULL, NULL);
+}
diff --git a/drivers/misc/mnh/mnh-crypto.h b/drivers/misc/mnh/mnh-crypto.h
new file mode 100644
index 0000000..ad7acee
--- /dev/null
+++ b/drivers/misc/mnh/mnh-crypto.h
@@ -0,0 +1,60 @@
+/*
+ * MNH Firmware for firmware authentication
+ *
+ * Copyright 2017 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ */
+
+#ifndef __MNH_CRYPTO
+#define __MNH_CRYPTO
+
+/*
+ * Signed firmware image blob as generated by signing tool
+ * +--------------+------------+-------------+--------------+
+ * | firmware | signature | esl_img_ | ESL_SIG |
+ * | image | | signature | _STRING |
+ * +--------------+------------+-------------+--------------+
+ * <- cert_info.img_len ->
+ * <- cert_info.cert_size ->
+ */
+struct esl_img_signature {
+ __u8 algo; /* Public key crypto algorithm */
+ __u8 hash; /* Digest algorithm */
+ __u8 id_type; /* Key identifier type */
+ __u8 signer_len; /* Length of signer's name */
+ __u8 key_id_len; /* Length of key identifier */
+ __u8 __pad[3]; /* */
+ __be32 sig_len; /* Length of the signature data */
+};
+
+enum cert_state {
+ FW_IMAGE_TAINTED = 0, /* initialization value */
+ FW_IMAGE_NO_CERT, /* firmware has no valid certificate */
+ FW_IMAGE_CERT_OK, /* firmware's signature verified correct */
+};
+
+struct cert_info {
+ const __u8 *img; /* pointer to the image file in memory */
+ unsigned long img_len; /* length of the image file */
+ enum cert_state cert; /* signature checked out */
+ size_t cert_size; /* length of the appended signature */
+};
+
+#if IS_ENABLED(CONFIG_MNH_SIG)
+int mnh_crypto_config_sysfs(void);
+#else
+static inline int mnh_crypto_config_sysfs(void) { return 0; }
+#endif
+int mnh_crypto_verify_fw(struct device *dev, const char *path);
+int mnh_crypto_verify(struct device *dev, struct cert_info *info);
+int mnh_crypto_verify_sig(struct device *dev, struct cert_info *info);
+
+#endif /* __MNH_CRYPTO */
diff --git a/drivers/misc/mnh/mnh-ddr-33-100-400-600.h b/drivers/misc/mnh/mnh-ddr-33-100-400-600.h
new file mode 100644
index 0000000..f5b42e9
--- /dev/null
+++ b/drivers/misc/mnh/mnh-ddr-33-100-400-600.h
@@ -0,0 +1,1920 @@
+/* regconfig tool: v1.4 */
+#ifndef _MNH_DDR_33_100_400_600_
+#define _MNH_DDR_33_100_400_600_
+#include "mnh-ddr.h"
+static const struct mnh_ddr_reg_config mnh_ddr_33_100_400_600 = {
+ {
+ 0x131b007d /* fsp 0 */,
+ 0x1311007d /* fsp 1 */,
+ 0x0422007d /* fsp 2 */,
+ 0x0311007d /* fsp 3 */
+ },
+ {
+ 0x00000b00 /* ctl 0 */,
+ 0x00000000 /* ctl 1 */,
+ 0x00000000 /* ctl 2 */,
+ 0x00000000 /* ctl 3 */,
+ 0x00000000 /* ctl 4 */,
+ 0x00000000 /* ctl 5 */,
+ 0x00003416 /* ctl 6 */,
+ 0x000208d6 /* ctl 7 */,
+ 0x00000005 /* ctl 8 */,
+ 0x00000086 /* ctl 9 */,
+ 0x00009c40 /* ctl 10 */,
+ 0x00061a80 /* ctl 11 */,
+ 0x00000005 /* ctl 12 */,
+ 0x00000190 /* ctl 13 */,
+ 0x00027100 /* ctl 14 */,
+ 0x00186a00 /* ctl 15 */,
+ 0x00000005 /* ctl 16 */,
+ 0x00000640 /* ctl 17 */,
+ 0x0003a9e1 /* ctl 18 */,
+ 0x0024a2c1 /* ctl 19 */,
+ 0x00000005 /* ctl 20 */,
+ 0x01000961 /* ctl 21 */,
+ 0x01010000 /* ctl 22 */,
+ 0x00030100 /* ctl 23 */,
+ 0x03020100 /* ctl 24 */,
+ 0x0000000e /* ctl 25 */,
+ 0x00000022 /* ctl 26 */,
+ 0x040c040c /* ctl 27 */,
+ 0x0c30081c /* ctl 28 */,
+ 0x07040804 /* ctl 29 */,
+ 0x03040903 /* ctl 30 */,
+ 0x09090d04 /* ctl 31 */,
+ 0x31080804 /* ctl 32 */,
+ 0x200f0922 /* ctl 33 */,
+ 0x0e33490d /* ctl 34 */,
+ 0x08203116 /* ctl 35 */,
+ 0x00000a0a /* ctl 36 */,
+ 0x04001248 /* ctl 37 */,
+ 0x0a0a0804 /* ctl 38 */,
+ 0x040036d8 /* ctl 39 */,
+ 0x0a0a0804 /* ctl 40 */,
+ 0x0600db60 /* ctl 41 */,
+ 0x0d0d0a06 /* ctl 42 */,
+ 0x0a014931 /* ctl 43 */,
+ 0x0203040a /* ctl 44 */,
+ 0x04040400 /* ctl 45 */,
+ 0x16100f05 /* ctl 46 */,
+ 0x000a0817 /* ctl 47 */,
+ 0x0a140011 /* ctl 48 */,
+ 0x0201010a /* ctl 49 */,
+ 0x00010002 /* ctl 50 */,
+ 0x2d1f0908 /* ctl 51 */,
+ 0x11050404 /* ctl 52 */,
+ 0x0101001a /* ctl 53 */,
+ 0x01000000 /* ctl 54 */,
+ 0x00000903 /* ctl 55 */,
+ 0x001a003f /* ctl 56 */,
+ 0x006800c1 /* ctl 57 */,
+ 0x009d030a /* ctl 58 */,
+ 0x00000490 /* ctl 59 */,
+ 0x00030003 /* ctl 60 */,
+ 0x000a0006 /* ctl 61 */,
+ 0x23180a0a /* ctl 62 */,
+ 0x0a000001 /* ctl 63 */,
+ 0x00010103 /* ctl 64 */,
+ 0x01030a00 /* ctl 65 */,
+ 0x0c000002 /* ctl 66 */,
+ 0x00030103 /* ctl 67 */,
+ 0x01031100 /* ctl 68 */,
+ 0x000a0003 /* ctl 69 */,
+ 0x001c0003 /* ctl 70 */,
+ 0x006e0006 /* ctl 71 */,
+ 0x00a6000a /* ctl 72 */,
+ 0x03050505 /* ctl 73 */,
+ 0x03010302 /* ctl 74 */,
+ 0x03050505 /* ctl 75 */,
+ 0x03010302 /* ctl 76 */,
+ 0x0c050605 /* ctl 77 */,
+ 0x03020602 /* ctl 78 */,
+ 0x13070a07 /* ctl 79 */,
+ 0x03030a02 /* ctl 80 */,
+ 0x03010000 /* ctl 81 */,
+ 0x00010000 /* ctl 82 */,
+ 0x00000000 /* ctl 83 */,
+ 0x00000000 /* ctl 84 */,
+ 0x00000000 /* ctl 85 */,
+ 0x01000000 /* ctl 86 */,
+ 0x80104002 /* ctl 87 */,
+ 0x00040003 /* ctl 88 */,
+ 0x00040005 /* ctl 89 */,
+ 0x00030000 /* ctl 90 */,
+ 0x00050004 /* ctl 91 */,
+ 0x00000004 /* ctl 92 */,
+ 0x00040003 /* ctl 93 */,
+ 0x00040005 /* ctl 94 */,
+ 0x00030000 /* ctl 95 */,
+ 0x00050004 /* ctl 96 */,
+ 0x00000004 /* ctl 97 */,
+ 0x01f803f0 /* ctl 98 */,
+ 0x0c100000 /* ctl 99 */,
+ 0x00000608 /* ctl 100 */,
+ 0x185030a0 /* ctl 101 */,
+ 0x49040000 /* ctl 102 */,
+ 0x00002482 /* ctl 103 */,
+ 0x00000000 /* ctl 104 */,
+ 0x00000000 /* ctl 105 */,
+ 0x00000000 /* ctl 106 */,
+ 0x00000000 /* ctl 107 */,
+ 0x00000000 /* ctl 108 */,
+ 0x00000000 /* ctl 109 */,
+ 0x03000000 /* ctl 110 */,
+ 0x06030303 /* ctl 111 */,
+ 0x00030a03 /* ctl 112 */,
+ 0x02030200 /* ctl 113 */,
+ 0x00070703 /* ctl 114 */,
+ 0x03020302 /* ctl 115 */,
+ 0x02000707 /* ctl 116 */,
+ 0x07030203 /* ctl 117 */,
+ 0x03020007 /* ctl 118 */,
+ 0x07070302 /* ctl 119 */,
+ 0x0000030f /* ctl 120 */,
+ 0x00070004 /* ctl 121 */,
+ 0x00000000 /* ctl 122 */,
+ 0x00000000 /* ctl 123 */,
+ 0x00000000 /* ctl 124 */,
+ 0x00000000 /* ctl 125 */,
+ 0x00000000 /* ctl 126 */,
+ 0x00000000 /* ctl 127 */,
+ 0x00000000 /* ctl 128 */,
+ 0x00000000 /* ctl 129 */,
+ 0x0400c001 /* ctl 130 */,
+ 0xc00400c0 /* ctl 131 */,
+ 0x00c00400 /* ctl 132 */,
+ 0x01000400 /* ctl 133 */,
+ 0x00000c00 /* ctl 134 */,
+ 0x00000000 /* ctl 135 */,
+ 0x00000001 /* ctl 136 */,
+ 0x00000002 /* ctl 137 */,
+ 0x00000003 /* ctl 138 */,
+ 0x0000100e /* ctl 139 */,
+ 0x00000000 /* ctl 140 */,
+ 0x00000000 /* ctl 141 */,
+ 0x00000000 /* ctl 142 */,
+ 0x00000000 /* ctl 143 */,
+ 0x000e0000 /* ctl 144 */,
+ 0x00110007 /* ctl 145 */,
+ 0x00110404 /* ctl 146 */,
+ 0x00140028 /* ctl 147 */,
+ 0x04040032 /* ctl 148 */,
+ 0x00a00032 /* ctl 149 */,
+ 0x00c80050 /* ctl 150 */,
+ 0x00c80606 /* ctl 151 */,
+ 0x007900f1 /* ctl 152 */,
+ 0x0a0a012d /* ctl 153 */,
+ 0x0000012d /* ctl 154 */,
+ 0x00000000 /* ctl 155 */,
+ 0x00000000 /* ctl 156 */,
+ 0x00000000 /* ctl 157 */,
+ 0x00060000 /* ctl 158 */,
+ 0x12260006 /* ctl 159 */,
+ 0x31002446 /* ctl 160 */,
+ 0x00313131 /* ctl 161 */,
+ 0x00000000 /* ctl 162 */,
+ 0x4d4d4d4d /* ctl 163 */,
+ 0x4d4d4dc0 /* ctl 164 */,
+ 0x0000004d /* ctl 165 */,
+ 0x00000000 /* ctl 166 */,
+ 0x06060606 /* ctl 167 */,
+ 0x01000000 /* ctl 168 */,
+ 0x00000001 /* ctl 169 */,
+ 0x00000000 /* ctl 170 */,
+ 0x01000000 /* ctl 171 */,
+ 0x00000001 /* ctl 172 */,
+ 0x00000000 /* ctl 173 */,
+ 0x00000000 /* ctl 174 */,
+ 0x00000000 /* ctl 175 */,
+ 0x00000000 /* ctl 176 */,
+ 0x00000000 /* ctl 177 */,
+ 0x00000000 /* ctl 178 */,
+ 0x00000000 /* ctl 179 */,
+ 0x00000000 /* ctl 180 */,
+ 0x00000000 /* ctl 181 */,
+ 0x00000000 /* ctl 182 */,
+ 0x11000000 /* ctl 183 */,
+ 0x000c1815 /* ctl 184 */,
+ 0x00000000 /* ctl 185 */,
+ 0x00000000 /* ctl 186 */,
+ 0x00000000 /* ctl 187 */,
+ 0x00000000 /* ctl 188 */,
+ 0x00000000 /* ctl 189 */,
+ 0x00000000 /* ctl 190 */,
+ 0x00000000 /* ctl 191 */,
+ 0x00000000 /* ctl 192 */,
+ 0x00000000 /* ctl 193 */,
+ 0x00000000 /* ctl 194 */,
+ 0x00000000 /* ctl 195 */,
+ 0x00000000 /* ctl 196 */,
+ 0x00000000 /* ctl 197 */,
+ 0x00000000 /* ctl 198 */,
+ 0x00000000 /* ctl 199 */,
+ 0x00000000 /* ctl 200 */,
+ 0x00000000 /* ctl 201 */,
+ 0x00000000 /* ctl 202 */,
+ 0x00020003 /* ctl 203 */,
+ 0x00400100 /* ctl 204 */,
+ 0x00080043 /* ctl 205 */,
+ 0x01000200 /* ctl 206 */,
+ 0x00c80040 /* ctl 207 */,
+ 0x00020008 /* ctl 208 */,
+ 0x00400100 /* ctl 209 */,
+ 0x00180320 /* ctl 210 */,
+ 0x01000200 /* ctl 211 */,
+ 0x04b10040 /* ctl 212 */,
+ 0x00000025 /* ctl 213 */,
+ 0x000a0004 /* ctl 214 */,
+ 0x003d0028 /* ctl 215 */,
+ 0x02010000 /* ctl 216 */,
+ 0xff0b0000 /* ctl 217 */,
+ 0x010101ff /* ctl 218 */,
+ 0x01010101 /* ctl 219 */,
+ 0x00010b01 /* ctl 220 */,
+ 0x01010000 /* ctl 221 */,
+ 0x00000004 /* ctl 222 */,
+ 0x01000001 /* ctl 223 */,
+ 0x02020202 /* ctl 224 */,
+ 0x00000001 /* ctl 225 */,
+ 0x00000000 /* ctl 226 */,
+ 0x00000000 /* ctl 227 */,
+ 0x00000000 /* ctl 228 */,
+ 0x00000000 /* ctl 229 */,
+ 0x00000000 /* ctl 230 */,
+ 0x00000000 /* ctl 231 */,
+ 0x00000000 /* ctl 232 */,
+ 0x00000000 /* ctl 233 */,
+ 0x00000000 /* ctl 234 */,
+ 0x00000000 /* ctl 235 */,
+ 0x00000000 /* ctl 236 */,
+ 0x00000000 /* ctl 237 */,
+ 0x00000000 /* ctl 238 */,
+ 0x00000000 /* ctl 239 */,
+ 0x00000000 /* ctl 240 */,
+ 0x00000000 /* ctl 241 */,
+ 0x00000000 /* ctl 242 */,
+ 0x00000000 /* ctl 243 */,
+ 0x00000000 /* ctl 244 */,
+ 0x00000000 /* ctl 245 */,
+ 0x00000000 /* ctl 246 */,
+ 0x00000000 /* ctl 247 */,
+ 0x00010001 /* ctl 248 */,
+ 0x01010001 /* ctl 249 */,
+ 0x00010101 /* ctl 250 */,
+ 0x08080300 /* ctl 251 */,
+ 0x08000a08 /* ctl 252 */,
+ 0x000b0b08 /* ctl 253 */,
+ 0x03010100 /* ctl 254 */,
+ 0x00000005 /* ctl 255 */,
+ 0x00000000 /* ctl 256 */,
+ 0x00010000 /* ctl 257 */,
+ 0x00280d00 /* ctl 258 */,
+ 0x00000001 /* ctl 259 */,
+ 0x00000001 /* ctl 260 */,
+ 0x00000000 /* ctl 261 */,
+ 0x00000000 /* ctl 262 */,
+ 0x00000000 /* ctl 263 */,
+ 0x00000000 /* ctl 264 */,
+ 0x00000000 /* ctl 265 */,
+ 0x00000000 /* ctl 266 */,
+ 0x00000000 /* ctl 267 */,
+ 0x00000000 /* ctl 268 */,
+ 0x00000000 /* ctl 269 */,
+ 0x00000000 /* ctl 270 */,
+ 0x01000000 /* ctl 271 */,
+ 0x00000001 /* ctl 272 */,
+ 0x00010100 /* ctl 273 */,
+ 0x00000000 /* ctl 274 */,
+ 0x00000000 /* ctl 275 */,
+ 0x00000000 /* ctl 276 */,
+ 0x00000000 /* ctl 277 */,
+ 0x00000000 /* ctl 278 */,
+ 0x00000000 /* ctl 279 */,
+ 0x00000000 /* ctl 280 */,
+ 0x00000000 /* ctl 281 */,
+ 0x00000000 /* ctl 282 */,
+ 0x00000000 /* ctl 283 */,
+ 0x00000000 /* ctl 284 */,
+ 0x00000000 /* ctl 285 */,
+ 0x00000000 /* ctl 286 */,
+ 0x00000000 /* ctl 287 */,
+ 0x00000000 /* ctl 288 */,
+ 0x00000000 /* ctl 289 */,
+ 0x00000000 /* ctl 290 */,
+ 0x00000000 /* ctl 291 */,
+ 0x00000000 /* ctl 292 */,
+ 0x00000000 /* ctl 293 */,
+ 0x00000000 /* ctl 294 */,
+ 0x000556aa /* ctl 295 */,
+ 0x000aaaaa /* ctl 296 */,
+ 0x000aa955 /* ctl 297 */,
+ 0x00055555 /* ctl 298 */,
+ 0x000b3133 /* ctl 299 */,
+ 0x0004cd33 /* ctl 300 */,
+ 0x0004cecc /* ctl 301 */,
+ 0x000b32cc /* ctl 302 */,
+ 0x00010300 /* ctl 303 */,
+ 0x00000100 /* ctl 304 */,
+ 0x00000000 /* ctl 305 */,
+ 0x00000000 /* ctl 306 */,
+ 0x00000000 /* ctl 307 */,
+ 0x00000000 /* ctl 308 */,
+ 0x00000000 /* ctl 309 */,
+ 0x00000000 /* ctl 310 */,
+ 0x00000000 /* ctl 311 */,
+ 0x00000000 /* ctl 312 */,
+ 0x00000000 /* ctl 313 */,
+ 0x00000000 /* ctl 314 */,
+ 0x00000101 /* ctl 315 */,
+ 0x00010100 /* ctl 316 */,
+ 0x00000001 /* ctl 317 */,
+ 0x00000000 /* ctl 318 */,
+ 0x0003ffff /* ctl 319 */,
+ 0x00000000 /* ctl 320 */,
+ 0x0003ffff /* ctl 321 */,
+ 0x00000000 /* ctl 322 */,
+ 0x0003ffff /* ctl 323 */,
+ 0x00000000 /* ctl 324 */,
+ 0x0003ffff /* ctl 325 */,
+ 0x00000000 /* ctl 326 */,
+ 0x0003ffff /* ctl 327 */,
+ 0x00000000 /* ctl 328 */,
+ 0x0003ffff /* ctl 329 */,
+ 0x00000000 /* ctl 330 */,
+ 0x0003ffff /* ctl 331 */,
+ 0x00000000 /* ctl 332 */,
+ 0x0003ffff /* ctl 333 */,
+ 0x00000000 /* ctl 334 */,
+ 0x0003ffff /* ctl 335 */,
+ 0x00000000 /* ctl 336 */,
+ 0x0003ffff /* ctl 337 */,
+ 0x00000000 /* ctl 338 */,
+ 0x0003ffff /* ctl 339 */,
+ 0x00000000 /* ctl 340 */,
+ 0x0003ffff /* ctl 341 */,
+ 0x00000000 /* ctl 342 */,
+ 0x0003ffff /* ctl 343 */,
+ 0x00000000 /* ctl 344 */,
+ 0x0003ffff /* ctl 345 */,
+ 0x00000000 /* ctl 346 */,
+ 0x0003ffff /* ctl 347 */,
+ 0x00000000 /* ctl 348 */,
+ 0x0003ffff /* ctl 349 */,
+ 0x00000000 /* ctl 350 */,
+ 0x0003ffff /* ctl 351 */,
+ 0x00000000 /* ctl 352 */,
+ 0x0003ffff /* ctl 353 */,
+ 0x00000000 /* ctl 354 */,
+ 0x0003ffff /* ctl 355 */,
+ 0x00000000 /* ctl 356 */,
+ 0x0003ffff /* ctl 357 */,
+ 0x00000000 /* ctl 358 */,
+ 0x0003ffff /* ctl 359 */,
+ 0x00000000 /* ctl 360 */,
+ 0x0003ffff /* ctl 361 */,
+ 0x00000000 /* ctl 362 */,
+ 0x0003ffff /* ctl 363 */,
+ 0x00000000 /* ctl 364 */,
+ 0x0003ffff /* ctl 365 */,
+ 0x00000000 /* ctl 366 */,
+ 0x0003ffff /* ctl 367 */,
+ 0x00000000 /* ctl 368 */,
+ 0x0003ffff /* ctl 369 */,
+ 0x00000000 /* ctl 370 */,
+ 0x0003ffff /* ctl 371 */,
+ 0x00000000 /* ctl 372 */,
+ 0x0003ffff /* ctl 373 */,
+ 0x00000000 /* ctl 374 */,
+ 0x0003ffff /* ctl 375 */,
+ 0x00000000 /* ctl 376 */,
+ 0x0003ffff /* ctl 377 */,
+ 0x00000000 /* ctl 378 */,
+ 0x0003ffff /* ctl 379 */,
+ 0x00000000 /* ctl 380 */,
+ 0x0303ffff /* ctl 381 */,
+ 0xffffffff /* ctl 382 */,
+ 0x00030000 /* ctl 383 */,
+ 0xffffffff /* ctl 384 */,
+ 0x00030000 /* ctl 385 */,
+ 0xffffffff /* ctl 386 */,
+ 0x00030000 /* ctl 387 */,
+ 0xffffffff /* ctl 388 */,
+ 0x00030000 /* ctl 389 */,
+ 0xffffffff /* ctl 390 */,
+ 0x00030000 /* ctl 391 */,
+ 0xffffffff /* ctl 392 */,
+ 0x00030000 /* ctl 393 */,
+ 0xffffffff /* ctl 394 */,
+ 0x00030000 /* ctl 395 */,
+ 0xffffffff /* ctl 396 */,
+ 0x00030000 /* ctl 397 */,
+ 0xffffffff /* ctl 398 */,
+ 0x00030000 /* ctl 399 */,
+ 0xffffffff /* ctl 400 */,
+ 0x00030000 /* ctl 401 */,
+ 0xffffffff /* ctl 402 */,
+ 0x00030000 /* ctl 403 */,
+ 0xffffffff /* ctl 404 */,
+ 0x00030000 /* ctl 405 */,
+ 0xffffffff /* ctl 406 */,
+ 0x00030000 /* ctl 407 */,
+ 0xffffffff /* ctl 408 */,
+ 0x00030000 /* ctl 409 */,
+ 0xffffffff /* ctl 410 */,
+ 0x00030000 /* ctl 411 */,
+ 0xffffffff /* ctl 412 */,
+ 0x00030000 /* ctl 413 */,
+ 0xffffffff /* ctl 414 */,
+ 0x00030000 /* ctl 415 */,
+ 0xffffffff /* ctl 416 */,
+ 0x00030000 /* ctl 417 */,
+ 0xffffffff /* ctl 418 */,
+ 0x00030000 /* ctl 419 */,
+ 0xffffffff /* ctl 420 */,
+ 0x00030000 /* ctl 421 */,
+ 0xffffffff /* ctl 422 */,
+ 0x00030000 /* ctl 423 */,
+ 0xffffffff /* ctl 424 */,
+ 0x00030000 /* ctl 425 */,
+ 0xffffffff /* ctl 426 */,
+ 0x00030000 /* ctl 427 */,
+ 0xffffffff /* ctl 428 */,
+ 0x00030000 /* ctl 429 */,
+ 0xffffffff /* ctl 430 */,
+ 0x00030000 /* ctl 431 */,
+ 0xffffffff /* ctl 432 */,
+ 0x00030000 /* ctl 433 */,
+ 0xffffffff /* ctl 434 */,
+ 0x00030000 /* ctl 435 */,
+ 0xffffffff /* ctl 436 */,
+ 0x00030000 /* ctl 437 */,
+ 0xffffffff /* ctl 438 */,
+ 0x00030000 /* ctl 439 */,
+ 0xffffffff /* ctl 440 */,
+ 0x00030000 /* ctl 441 */,
+ 0xffffffff /* ctl 442 */,
+ 0x00030000 /* ctl 443 */,
+ 0xffffffff /* ctl 444 */,
+ 0x00000000 /* ctl 445 */,
+ 0x02020200 /* ctl 446 */,
+ 0x00640002 /* ctl 447 */,
+ 0x01010101 /* ctl 448 */,
+ 0x00006401 /* ctl 449 */,
+ 0x00000000 /* ctl 450 */,
+ 0x15140000 /* ctl 451 */,
+ 0x0000221c /* ctl 452 */,
+ 0x0001f808 /* ctl 453 */,
+ 0x00000000 /* ctl 454 */,
+ 0x00000000 /* ctl 455 */,
+ 0x000001f8 /* ctl 456 */,
+ 0x000013b0 /* ctl 457 */,
+ 0x06080205 /* ctl 458 */,
+ 0x00000000 /* ctl 459 */,
+ 0x00000000 /* ctl 460 */,
+ 0x00000608 /* ctl 461 */,
+ 0x00003c50 /* ctl 462 */,
+ 0x18500205 /* ctl 463 */,
+ 0x00000000 /* ctl 464 */,
+ 0x00000000 /* ctl 465 */,
+ 0x00001850 /* ctl 466 */,
+ 0x0000f320 /* ctl 467 */,
+ 0x2482060c /* ctl 468 */,
+ 0x00000000 /* ctl 469 */,
+ 0x00000000 /* ctl 470 */,
+ 0x00002482 /* ctl 471 */,
+ 0x00016d14 /* ctl 472 */,
+ 0x02020a15 /* ctl 473 */,
+ 0x03020202 /* ctl 474 */,
+ 0x00001803 /* ctl 475 */,
+ 0x00000000 /* ctl 476 */,
+ 0x00000000 /* ctl 477 */,
+ 0x00001403 /* ctl 478 */,
+ 0x000007d0 /* ctl 479 */,
+ 0x00000000 /* ctl 480 */,
+ 0x00000000 /* ctl 481 */,
+ 0x00030000 /* ctl 482 */,
+ 0x00070019 /* ctl 483 */,
+ 0x0008001a /* ctl 484 */,
+ 0x000e0020 /* ctl 485 */,
+ 0x00120024 /* ctl 486 */,
+ 0x00000000 /* ctl 487 */,
+ 0x00000000 /* ctl 488 */,
+ 0x02000000 /* ctl 489 */,
+ 0x05020202 /* ctl 490 */,
+ 0x0d010501 /* ctl 491 */,
+ 0x05091705 /* ctl 492 */,
+ 0x01010001 /* ctl 493 */,
+ 0x01010101 /* ctl 494 */,
+ 0x01000101 /* ctl 495 */,
+ 0x01000100 /* ctl 496 */,
+ 0x00010001 /* ctl 497 */,
+ 0x00010002 /* ctl 498 */,
+ 0x00020100 /* ctl 499 */,
+ 0x00020002 /* ctl 500 */,
+ 0x00000000 /* ctl 501 */,
+ 0x00000400 /* ctl 502 */,
+ 0x000c0018 /* ctl 503 */,
+ 0x00300059 /* ctl 504 */,
+ 0x0049017e /* ctl 505 */,
+ 0x00400241 /* ctl 506 */,
+ 0x00120103 /* ctl 507 */,
+ 0x00000002 /* ctl 508 */,
+ 0x00000200 /* ctl 509 */,
+ 0x00000200 /* ctl 510 */,
+ 0x00000200 /* ctl 511 */,
+ 0x00000200 /* ctl 512 */,
+ 0x00000200 /* ctl 513 */,
+ 0x00000200 /* ctl 514 */,
+ 0x00000200 /* ctl 515 */,
+ 0x00000200 /* ctl 516 */,
+ 0x00000200 /* ctl 517 */,
+ 0x00000200 /* ctl 518 */,
+ 0x00000200 /* ctl 519 */,
+ 0x00000200 /* ctl 520 */,
+ 0x00000200 /* ctl 521 */,
+ 0x00000200 /* ctl 522 */,
+ 0x00000200 /* ctl 523 */,
+ 0x00000200 /* ctl 524 */,
+ 0x00000001 /* ctl 525 */,
+ 0x000f0000 /* ctl 526 */,
+ 0x000f000f /* ctl 527 */,
+ 0x000f000f /* ctl 528 */,
+ 0x000f000f /* ctl 529 */,
+ 0x000f000f /* ctl 530 */,
+ 0x000f000f /* ctl 531 */,
+ 0x000f000f /* ctl 532 */,
+ 0x000f000f /* ctl 533 */,
+ 0x000f000f /* ctl 534 */,
+ 0x000f000f /* ctl 535 */,
+ 0x000f000f /* ctl 536 */,
+ 0x000f000f /* ctl 537 */,
+ 0x000f000f /* ctl 538 */,
+ 0x000f000f /* ctl 539 */,
+ 0x000f000f /* ctl 540 */,
+ 0x000f000f /* ctl 541 */,
+ 0x000f000f /* ctl 542 */,
+ 0x000f000f /* ctl 543 */,
+ 0x000f000f /* ctl 544 */,
+ 0x000f000f /* ctl 545 */,
+ 0x000f000f /* ctl 546 */,
+ 0x000f000f /* ctl 547 */,
+ 0x000f000f /* ctl 548 */,
+ 0x000f000f /* ctl 549 */,
+ 0x000f000f /* ctl 550 */,
+ 0x000f000f /* ctl 551 */,
+ 0x000f000f /* ctl 552 */,
+ 0x000f000f /* ctl 553 */,
+ 0x000f000f /* ctl 554 */,
+ 0x000f000f /* ctl 555 */,
+ 0x000f000f /* ctl 556 */,
+ 0x000f000f /* ctl 557 */,
+ 0x0000000f /* ctl 558 */
+ },
+ {
+ 0x00000b00 /* pi 0 */,
+ 0x00000101 /* pi 1 */,
+ 0x00640000 /* pi 2 */,
+ 0x00049040 /* pi 3 */,
+ 0x000001f8 /* pi 4 */,
+ 0x0000c100 /* pi 5 */,
+ 0x00000608 /* pi 6 */,
+ 0x00030a00 /* pi 7 */,
+ 0x00001850 /* pi 8 */,
+ 0x00049040 /* pi 9 */,
+ 0x00002482 /* pi 10 */,
+ 0x000001f8 /* pi 11 */,
+ 0x00000200 /* pi 12 */,
+ 0x00000200 /* pi 13 */,
+ 0x00000200 /* pi 14 */,
+ 0x00000200 /* pi 15 */,
+ 0x00000608 /* pi 16 */,
+ 0x00000200 /* pi 17 */,
+ 0x00000200 /* pi 18 */,
+ 0x00000200 /* pi 19 */,
+ 0x00000200 /* pi 20 */,
+ 0x00001850 /* pi 21 */,
+ 0x00000200 /* pi 22 */,
+ 0x00000200 /* pi 23 */,
+ 0x00000200 /* pi 24 */,
+ 0x00000200 /* pi 25 */,
+ 0x00002482 /* pi 26 */,
+ 0x00000200 /* pi 27 */,
+ 0x00000200 /* pi 28 */,
+ 0x00000200 /* pi 29 */,
+ 0x00000200 /* pi 30 */,
+ 0x00010000 /* pi 31 */,
+ 0x0000000f /* pi 32 */,
+ 0x01000003 /* pi 33 */,
+ 0x00000000 /* pi 34 */,
+ 0x00000000 /* pi 35 */,
+ 0x00000000 /* pi 36 */,
+ 0x00000000 /* pi 37 */,
+ 0x00000000 /* pi 38 */,
+ 0x00000000 /* pi 39 */,
+ 0x00000000 /* pi 40 */,
+ 0x00000000 /* pi 41 */,
+ 0x00000000 /* pi 42 */,
+ 0x00000000 /* pi 43 */,
+ 0x00000000 /* pi 44 */,
+ 0x00000000 /* pi 45 */,
+ 0x00000000 /* pi 46 */,
+ 0x00000000 /* pi 47 */,
+ 0x00000000 /* pi 48 */,
+ 0x01000000 /* pi 49 */,
+ 0x29030001 /* pi 50 */,
+ 0x0847382a /* pi 51 */,
+ 0x00010002 /* pi 52 */,
+ 0x00000009 /* pi 53 */,
+ 0x000000fc /* pi 54 */,
+ 0x0000001a /* pi 55 */,
+ 0x00000304 /* pi 56 */,
+ 0x00000068 /* pi 57 */,
+ 0x00000c28 /* pi 58 */,
+ 0x0000009d /* pi 59 */,
+ 0x00001241 /* pi 60 */,
+ 0x00000000 /* pi 61 */,
+ 0x00000000 /* pi 62 */,
+ 0x00000000 /* pi 63 */,
+ 0x00000000 /* pi 64 */,
+ 0x00000000 /* pi 65 */,
+ 0x00000000 /* pi 66 */,
+ 0x00000000 /* pi 67 */,
+ 0x00000000 /* pi 68 */,
+ 0x00000000 /* pi 69 */,
+ 0x01000000 /* pi 70 */,
+ 0x04040401 /* pi 71 */,
+ 0x0a000004 /* pi 72 */,
+ 0x01010128 /* pi 73 */,
+ 0x00000001 /* pi 74 */,
+ 0x00000000 /* pi 75 */,
+ 0x00030003 /* pi 76 */,
+ 0x00000018 /* pi 77 */,
+ 0x00000000 /* pi 78 */,
+ 0x00000000 /* pi 79 */,
+ 0x00060002 /* pi 80 */,
+ 0x00010001 /* pi 81 */,
+ 0x01010001 /* pi 82 */,
+ 0x02010100 /* pi 83 */,
+ 0x03030002 /* pi 84 */,
+ 0x04040705 /* pi 85 */,
+ 0x0000120a /* pi 86 */,
+ 0x00000000 /* pi 87 */,
+ 0x00000000 /* pi 88 */,
+ 0x00000000 /* pi 89 */,
+ 0x001e0303 /* pi 90 */,
+ 0x000007d0 /* pi 91 */,
+ 0x01010300 /* pi 92 */,
+ 0x01010101 /* pi 93 */,
+ 0x00000101 /* pi 94 */,
+ 0x00000000 /* pi 95 */,
+ 0x00000000 /* pi 96 */,
+ 0x01000000 /* pi 97 */,
+ 0x00000101 /* pi 98 */,
+ 0x55555a5a /* pi 99 */,
+ 0x55555a5a /* pi 100 */,
+ 0x55555a5a /* pi 101 */,
+ 0x55555a5a /* pi 102 */,
+ 0x05050001 /* pi 103 */,
+ 0x0200150c /* pi 104 */,
+ 0x000a0602 /* pi 105 */,
+ 0x170d0505 /* pi 106 */,
+ 0x02020202 /* pi 107 */,
+ 0x00000000 /* pi 108 */,
+ 0x00000003 /* pi 109 */,
+ 0x00170300 /* pi 110 */,
+ 0x00070019 /* pi 111 */,
+ 0x0008001a /* pi 112 */,
+ 0x000e0020 /* pi 113 */,
+ 0x00120024 /* pi 114 */,
+ 0x00000000 /* pi 115 */,
+ 0x00000000 /* pi 116 */,
+ 0x01010100 /* pi 117 */,
+ 0x00000001 /* pi 118 */,
+ 0x010a140a /* pi 119 */,
+ 0x00010011 /* pi 120 */,
+ 0x00020032 /* pi 121 */,
+ 0x000200c8 /* pi 122 */,
+ 0x010a012d /* pi 123 */,
+ 0x00120100 /* pi 124 */,
+ 0x01000012 /* pi 125 */,
+ 0x00330033 /* pi 126 */,
+ 0x00c90100 /* pi 127 */,
+ 0x010000c9 /* pi 128 */,
+ 0x012e012e /* pi 129 */,
+ 0x02044d4d /* pi 130 */,
+ 0x06051001 /* pi 131 */,
+ 0x02080706 /* pi 132 */,
+ 0x00c00002 /* pi 133 */,
+ 0x00c01000 /* pi 134 */,
+ 0x00c01000 /* pi 135 */,
+ 0x00c01000 /* pi 136 */,
+ 0x04041000 /* pi 137 */,
+ 0x0a020100 /* pi 138 */,
+ 0x01001510 /* pi 139 */,
+ 0x00001901 /* pi 140 */,
+ 0x001d0019 /* pi 141 */,
+ 0x005f0026 /* pi 142 */,
+ 0x4d030000 /* pi 143 */,
+ 0x0102044d /* pi 144 */,
+ 0x34000000 /* pi 145 */,
+ 0x00000000 /* pi 146 */,
+ 0x00000000 /* pi 147 */,
+ 0x01010000 /* pi 148 */,
+ 0x00000101 /* pi 149 */,
+ 0x31000600 /* pi 150 */,
+ 0x064d4d00 /* pi 151 */,
+ 0x00310006 /* pi 152 */,
+ 0x26064d4d /* pi 153 */,
+ 0x4d003112 /* pi 154 */,
+ 0x2446064d /* pi 155 */,
+ 0x4d4d0031 /* pi 156 */,
+ 0x00060006 /* pi 157 */,
+ 0x4d4d0031 /* pi 158 */,
+ 0x31000600 /* pi 159 */,
+ 0x064d4d00 /* pi 160 */,
+ 0x00311226 /* pi 161 */,
+ 0x46064d4d /* pi 162 */,
+ 0x4d003124 /* pi 163 */,
+ 0x0000064d /* pi 164 */,
+ 0x00001101 /* pi 165 */,
+ 0x00c80032 /* pi 166 */,
+ 0x010a012d /* pi 167 */,
+ 0x080c020a /* pi 168 */,
+ 0x00020d11 /* pi 169 */,
+ 0x00020002 /* pi 170 */,
+ 0x00020002 /* pi 171 */,
+ 0x00000000 /* pi 172 */,
+ 0x00000000 /* pi 173 */,
+ 0x04000000 /* pi 174 */,
+ 0x00080100 /* pi 175 */,
+ 0x000001f8 /* pi 176 */,
+ 0x000013b0 /* pi 177 */,
+ 0x00000608 /* pi 178 */,
+ 0x00003c50 /* pi 179 */,
+ 0x00001850 /* pi 180 */,
+ 0x0000f320 /* pi 181 */,
+ 0x00002482 /* pi 182 */,
+ 0x00016d14 /* pi 183 */,
+ 0x00010000 /* pi 184 */,
+ 0x02000101 /* pi 185 */,
+ 0x01030001 /* pi 186 */,
+ 0x00010400 /* pi 187 */,
+ 0x06000105 /* pi 188 */,
+ 0x01070001 /* pi 189 */,
+ 0x00000000 /* pi 190 */,
+ 0x00000000 /* pi 191 */
+ },
+ {
+ 0x76543210 /* phy 0 */,
+ 0x0004f008 /* phy 1 */,
+ 0x00020133 /* phy 2 */,
+ 0x00000000 /* phy 3 */,
+ 0x00000000 /* phy 4 */,
+ 0x00010000 /* phy 5 */,
+ 0x01cece0e /* phy 6 */,
+ 0x02cece0e /* phy 7 */,
+ 0x00010000 /* phy 8 */,
+ 0x00000004 /* phy 9 */,
+ 0x00000001 /* phy 10 */,
+ 0x00000000 /* phy 11 */,
+ 0x00000000 /* phy 12 */,
+ 0x00000100 /* phy 13 */,
+ 0x001700c0 /* phy 14 */,
+ 0x020100cc /* phy 15 */,
+ 0x00030066 /* phy 16 */,
+ 0x00000000 /* phy 17 */,
+ 0x00000001 /* phy 18 */,
+ 0x00000000 /* phy 19 */,
+ 0x00000000 /* phy 20 */,
+ 0x00000000 /* phy 21 */,
+ 0x00000000 /* phy 22 */,
+ 0x00000000 /* phy 23 */,
+ 0x00000000 /* phy 24 */,
+ 0x04080000 /* phy 25 */,
+ 0x04080400 /* phy 26 */,
+ 0x10000000 /* phy 27 */,
+ 0x0c008007 /* phy 28 */,
+ 0x00000100 /* phy 29 */,
+ 0x00000100 /* phy 30 */,
+ 0x55555555 /* phy 31 */,
+ 0xaaaaaaaa /* phy 32 */,
+ 0x55555555 /* phy 33 */,
+ 0xaaaaaaaa /* phy 34 */,
+ 0x00015555 /* phy 35 */,
+ 0x00000000 /* phy 36 */,
+ 0x00000000 /* phy 37 */,
+ 0x00000000 /* phy 38 */,
+ 0x00000000 /* phy 39 */,
+ 0x00000000 /* phy 40 */,
+ 0x00000000 /* phy 41 */,
+ 0x00000000 /* phy 42 */,
+ 0x00000000 /* phy 43 */,
+ 0x00000000 /* phy 44 */,
+ 0x00000000 /* phy 45 */,
+ 0x00000000 /* phy 46 */,
+ 0x00000000 /* phy 47 */,
+ 0x00000000 /* phy 48 */,
+ 0x00000000 /* phy 49 */,
+ 0x00000000 /* phy 50 */,
+ 0x00000000 /* phy 51 */,
+ 0x00000000 /* phy 52 */,
+ 0x00000000 /* phy 53 */,
+ 0x00000000 /* phy 54 */,
+ 0x00000104 /* phy 55 */,
+ 0x00000020 /* phy 56 */,
+ 0x00000000 /* phy 57 */,
+ 0x00000000 /* phy 58 */,
+ 0x00000000 /* phy 59 */,
+ 0x00000000 /* phy 60 */,
+ 0x00000000 /* phy 61 */,
+ 0x00000000 /* phy 62 */,
+ 0x00000000 /* phy 63 */,
+ 0x02800280 /* phy 64 */,
+ 0x02800280 /* phy 65 */,
+ 0x02800280 /* phy 66 */,
+ 0x02800280 /* phy 67 */,
+ 0x00000280 /* phy 68 */,
+ 0x00000000 /* phy 69 */,
+ 0x00000000 /* phy 70 */,
+ 0x00000000 /* phy 71 */,
+ 0x00000000 /* phy 72 */,
+ 0x00000000 /* phy 73 */,
+ 0x00800080 /* phy 74 */,
+ 0x00800080 /* phy 75 */,
+ 0x00800080 /* phy 76 */,
+ 0x00800080 /* phy 77 */,
+ 0x00800080 /* phy 78 */,
+ 0x00800080 /* phy 79 */,
+ 0x00800080 /* phy 80 */,
+ 0x00800080 /* phy 81 */,
+ 0x00800080 /* phy 82 */,
+ 0x00010019 /* phy 83 */,
+ 0x000001d0 /* phy 84 */,
+ 0x00000000 /* phy 85 */,
+ 0x00000200 /* phy 86 */,
+ 0x00000004 /* phy 87 */,
+ 0x51816152 /* phy 88 */,
+ 0xc0c08161 /* phy 89 */,
+ 0x00010000 /* phy 90 */,
+ 0x02001000 /* phy 91 */,
+ 0x0c0432ff /* phy 92 */,
+ 0x000f0c18 /* phy 93 */,
+ 0x01000140 /* phy 94 */,
+ 0x00000c20 /* phy 95 */,
+ 0x00000000 /* phy 96 */,
+ 0x00000000 /* phy 97 */,
+ 0x00000000 /* phy 98 */,
+ 0x00000000 /* phy 99 */,
+ 0x00000000 /* phy 100 */,
+ 0x00000000 /* phy 101 */,
+ 0x00000000 /* phy 102 */,
+ 0x00000000 /* phy 103 */,
+ 0x00000000 /* phy 104 */,
+ 0x00000000 /* phy 105 */,
+ 0x00000000 /* phy 106 */,
+ 0x00000000 /* phy 107 */,
+ 0x00000000 /* phy 108 */,
+ 0x00000000 /* phy 109 */,
+ 0x00000000 /* phy 110 */,
+ 0x00000000 /* phy 111 */,
+ 0x00000000 /* phy 112 */,
+ 0x00000000 /* phy 113 */,
+ 0x00000000 /* phy 114 */,
+ 0x00000000 /* phy 115 */,
+ 0x00000000 /* phy 116 */,
+ 0x00000000 /* phy 117 */,
+ 0x00000000 /* phy 118 */,
+ 0x00000000 /* phy 119 */,
+ 0x00000000 /* phy 120 */,
+ 0x00000000 /* phy 121 */,
+ 0x00000000 /* phy 122 */,
+ 0x00000000 /* phy 123 */,
+ 0x00000000 /* phy 124 */,
+ 0x00000000 /* phy 125 */,
+ 0x00000000 /* phy 126 */,
+ 0x00000000 /* phy 127 */,
+ 0x76543210 /* phy 128 */,
+ 0x0004f008 /* phy 129 */,
+ 0x00020133 /* phy 130 */,
+ 0x00000000 /* phy 131 */,
+ 0x00000000 /* phy 132 */,
+ 0x00010000 /* phy 133 */,
+ 0x01cece0e /* phy 134 */,
+ 0x02cece0e /* phy 135 */,
+ 0x00010000 /* phy 136 */,
+ 0x00000004 /* phy 137 */,
+ 0x00000001 /* phy 138 */,
+ 0x00000000 /* phy 139 */,
+ 0x00000000 /* phy 140 */,
+ 0x00000100 /* phy 141 */,
+ 0x001700c0 /* phy 142 */,
+ 0x020100cc /* phy 143 */,
+ 0x00030066 /* phy 144 */,
+ 0x00000000 /* phy 145 */,
+ 0x00000001 /* phy 146 */,
+ 0x00000000 /* phy 147 */,
+ 0x00000000 /* phy 148 */,
+ 0x00000000 /* phy 149 */,
+ 0x00000000 /* phy 150 */,
+ 0x00000000 /* phy 151 */,
+ 0x00000000 /* phy 152 */,
+ 0x04080000 /* phy 153 */,
+ 0x04080400 /* phy 154 */,
+ 0x10000000 /* phy 155 */,
+ 0x0c008007 /* phy 156 */,
+ 0x00000100 /* phy 157 */,
+ 0x00000100 /* phy 158 */,
+ 0x55555555 /* phy 159 */,
+ 0xaaaaaaaa /* phy 160 */,
+ 0x55555555 /* phy 161 */,
+ 0xaaaaaaaa /* phy 162 */,
+ 0x00005555 /* phy 163 */,
+ 0x00000000 /* phy 164 */,
+ 0x00000000 /* phy 165 */,
+ 0x00000000 /* phy 166 */,
+ 0x00000000 /* phy 167 */,
+ 0x00000000 /* phy 168 */,
+ 0x00000000 /* phy 169 */,
+ 0x00000000 /* phy 170 */,
+ 0x00000000 /* phy 171 */,
+ 0x00000000 /* phy 172 */,
+ 0x00000000 /* phy 173 */,
+ 0x00000000 /* phy 174 */,
+ 0x00000000 /* phy 175 */,
+ 0x00000000 /* phy 176 */,
+ 0x00000000 /* phy 177 */,
+ 0x00000000 /* phy 178 */,
+ 0x00000000 /* phy 179 */,
+ 0x00000000 /* phy 180 */,
+ 0x00000000 /* phy 181 */,
+ 0x00000000 /* phy 182 */,
+ 0x00000104 /* phy 183 */,
+ 0x00000020 /* phy 184 */,
+ 0x00000000 /* phy 185 */,
+ 0x00000000 /* phy 186 */,
+ 0x00000000 /* phy 187 */,
+ 0x00000000 /* phy 188 */,
+ 0x00000000 /* phy 189 */,
+ 0x00000000 /* phy 190 */,
+ 0x00000000 /* phy 191 */,
+ 0x02800280 /* phy 192 */,
+ 0x02800280 /* phy 193 */,
+ 0x02800280 /* phy 194 */,
+ 0x02800280 /* phy 195 */,
+ 0x00000280 /* phy 196 */,
+ 0x00000000 /* phy 197 */,
+ 0x00000000 /* phy 198 */,
+ 0x00000000 /* phy 199 */,
+ 0x00000000 /* phy 200 */,
+ 0x00000000 /* phy 201 */,
+ 0x00800080 /* phy 202 */,
+ 0x00800080 /* phy 203 */,
+ 0x00800080 /* phy 204 */,
+ 0x00800080 /* phy 205 */,
+ 0x00800080 /* phy 206 */,
+ 0x00800080 /* phy 207 */,
+ 0x00800080 /* phy 208 */,
+ 0x00800080 /* phy 209 */,
+ 0x00800080 /* phy 210 */,
+ 0x00010019 /* phy 211 */,
+ 0x000001d0 /* phy 212 */,
+ 0x00000000 /* phy 213 */,
+ 0x00000200 /* phy 214 */,
+ 0x00000004 /* phy 215 */,
+ 0x51816152 /* phy 216 */,
+ 0xc0c08161 /* phy 217 */,
+ 0x00010000 /* phy 218 */,
+ 0x02001000 /* phy 219 */,
+ 0x0c0432ff /* phy 220 */,
+ 0x000f0c18 /* phy 221 */,
+ 0x01000140 /* phy 222 */,
+ 0x00000c20 /* phy 223 */,
+ 0x00000000 /* phy 224 */,
+ 0x00000000 /* phy 225 */,
+ 0x00000000 /* phy 226 */,
+ 0x00000000 /* phy 227 */,
+ 0x00000000 /* phy 228 */,
+ 0x00000000 /* phy 229 */,
+ 0x00000000 /* phy 230 */,
+ 0x00000000 /* phy 231 */,
+ 0x00000000 /* phy 232 */,
+ 0x00000000 /* phy 233 */,
+ 0x00000000 /* phy 234 */,
+ 0x00000000 /* phy 235 */,
+ 0x00000000 /* phy 236 */,
+ 0x00000000 /* phy 237 */,
+ 0x00000000 /* phy 238 */,
+ 0x00000000 /* phy 239 */,
+ 0x00000000 /* phy 240 */,
+ 0x00000000 /* phy 241 */,
+ 0x00000000 /* phy 242 */,
+ 0x00000000 /* phy 243 */,
+ 0x00000000 /* phy 244 */,
+ 0x00000000 /* phy 245 */,
+ 0x00000000 /* phy 246 */,
+ 0x00000000 /* phy 247 */,
+ 0x00000000 /* phy 248 */,
+ 0x00000000 /* phy 249 */,
+ 0x00000000 /* phy 250 */,
+ 0x00000000 /* phy 251 */,
+ 0x00000000 /* phy 252 */,
+ 0x00000000 /* phy 253 */,
+ 0x00000000 /* phy 254 */,
+ 0x00000000 /* phy 255 */,
+ 0x76543210 /* phy 256 */,
+ 0x0004f008 /* phy 257 */,
+ 0x00020133 /* phy 258 */,
+ 0x00000000 /* phy 259 */,
+ 0x00000000 /* phy 260 */,
+ 0x00010000 /* phy 261 */,
+ 0x01cece0e /* phy 262 */,
+ 0x02cece0e /* phy 263 */,
+ 0x00010000 /* phy 264 */,
+ 0x00000004 /* phy 265 */,
+ 0x00000001 /* phy 266 */,
+ 0x00000000 /* phy 267 */,
+ 0x00000000 /* phy 268 */,
+ 0x00000100 /* phy 269 */,
+ 0x001700c0 /* phy 270 */,
+ 0x020100cc /* phy 271 */,
+ 0x00030066 /* phy 272 */,
+ 0x00000000 /* phy 273 */,
+ 0x00000001 /* phy 274 */,
+ 0x00000000 /* phy 275 */,
+ 0x00000000 /* phy 276 */,
+ 0x00000000 /* phy 277 */,
+ 0x00000000 /* phy 278 */,
+ 0x00000000 /* phy 279 */,
+ 0x00000000 /* phy 280 */,
+ 0x04080000 /* phy 281 */,
+ 0x04080400 /* phy 282 */,
+ 0x10000000 /* phy 283 */,
+ 0x0c008007 /* phy 284 */,
+ 0x00000100 /* phy 285 */,
+ 0x00000100 /* phy 286 */,
+ 0x55555555 /* phy 287 */,
+ 0xaaaaaaaa /* phy 288 */,
+ 0x55555555 /* phy 289 */,
+ 0xaaaaaaaa /* phy 290 */,
+ 0x00015555 /* phy 291 */,
+ 0x00000000 /* phy 292 */,
+ 0x00000000 /* phy 293 */,
+ 0x00000000 /* phy 294 */,
+ 0x00000000 /* phy 295 */,
+ 0x00000000 /* phy 296 */,
+ 0x00000000 /* phy 297 */,
+ 0x00000000 /* phy 298 */,
+ 0x00000000 /* phy 299 */,
+ 0x00000000 /* phy 300 */,
+ 0x00000000 /* phy 301 */,
+ 0x00000000 /* phy 302 */,
+ 0x00000000 /* phy 303 */,
+ 0x00000000 /* phy 304 */,
+ 0x00000000 /* phy 305 */,
+ 0x00000000 /* phy 306 */,
+ 0x00000000 /* phy 307 */,
+ 0x00000000 /* phy 308 */,
+ 0x00000000 /* phy 309 */,
+ 0x00000000 /* phy 310 */,
+ 0x00000104 /* phy 311 */,
+ 0x00000020 /* phy 312 */,
+ 0x00000000 /* phy 313 */,
+ 0x00000000 /* phy 314 */,
+ 0x00000000 /* phy 315 */,
+ 0x00000000 /* phy 316 */,
+ 0x00000000 /* phy 317 */,
+ 0x00000000 /* phy 318 */,
+ 0x00000000 /* phy 319 */,
+ 0x02800280 /* phy 320 */,
+ 0x02800280 /* phy 321 */,
+ 0x02800280 /* phy 322 */,
+ 0x02800280 /* phy 323 */,
+ 0x00000280 /* phy 324 */,
+ 0x00000000 /* phy 325 */,
+ 0x00000000 /* phy 326 */,
+ 0x00000000 /* phy 327 */,
+ 0x00000000 /* phy 328 */,
+ 0x00000000 /* phy 329 */,
+ 0x00800080 /* phy 330 */,
+ 0x00800080 /* phy 331 */,
+ 0x00800080 /* phy 332 */,
+ 0x00800080 /* phy 333 */,
+ 0x00800080 /* phy 334 */,
+ 0x00800080 /* phy 335 */,
+ 0x00800080 /* phy 336 */,
+ 0x00800080 /* phy 337 */,
+ 0x00800080 /* phy 338 */,
+ 0x00010019 /* phy 339 */,
+ 0x000001d0 /* phy 340 */,
+ 0x00000000 /* phy 341 */,
+ 0x00000200 /* phy 342 */,
+ 0x00000004 /* phy 343 */,
+ 0x51816152 /* phy 344 */,
+ 0xc0c08161 /* phy 345 */,
+ 0x00010000 /* phy 346 */,
+ 0x02001000 /* phy 347 */,
+ 0x0c0432ff /* phy 348 */,
+ 0x000f0c18 /* phy 349 */,
+ 0x01000140 /* phy 350 */,
+ 0x00000c20 /* phy 351 */,
+ 0x00000000 /* phy 352 */,
+ 0x00000000 /* phy 353 */,
+ 0x00000000 /* phy 354 */,
+ 0x00000000 /* phy 355 */,
+ 0x00000000 /* phy 356 */,
+ 0x00000000 /* phy 357 */,
+ 0x00000000 /* phy 358 */,
+ 0x00000000 /* phy 359 */,
+ 0x00000000 /* phy 360 */,
+ 0x00000000 /* phy 361 */,
+ 0x00000000 /* phy 362 */,
+ 0x00000000 /* phy 363 */,
+ 0x00000000 /* phy 364 */,
+ 0x00000000 /* phy 365 */,
+ 0x00000000 /* phy 366 */,
+ 0x00000000 /* phy 367 */,
+ 0x00000000 /* phy 368 */,
+ 0x00000000 /* phy 369 */,
+ 0x00000000 /* phy 370 */,
+ 0x00000000 /* phy 371 */,
+ 0x00000000 /* phy 372 */,
+ 0x00000000 /* phy 373 */,
+ 0x00000000 /* phy 374 */,
+ 0x00000000 /* phy 375 */,
+ 0x00000000 /* phy 376 */,
+ 0x00000000 /* phy 377 */,
+ 0x00000000 /* phy 378 */,
+ 0x00000000 /* phy 379 */,
+ 0x00000000 /* phy 380 */,
+ 0x00000000 /* phy 381 */,
+ 0x00000000 /* phy 382 */,
+ 0x00000000 /* phy 383 */,
+ 0x76543210 /* phy 384 */,
+ 0x0004f008 /* phy 385 */,
+ 0x00020133 /* phy 386 */,
+ 0x00000000 /* phy 387 */,
+ 0x00000000 /* phy 388 */,
+ 0x00010000 /* phy 389 */,
+ 0x01cece0e /* phy 390 */,
+ 0x02cece0e /* phy 391 */,
+ 0x00010000 /* phy 392 */,
+ 0x00000004 /* phy 393 */,
+ 0x00000001 /* phy 394 */,
+ 0x00000000 /* phy 395 */,
+ 0x00000000 /* phy 396 */,
+ 0x00000100 /* phy 397 */,
+ 0x001700c0 /* phy 398 */,
+ 0x020100cc /* phy 399 */,
+ 0x00030066 /* phy 400 */,
+ 0x00000000 /* phy 401 */,
+ 0x00000001 /* phy 402 */,
+ 0x00000000 /* phy 403 */,
+ 0x00000000 /* phy 404 */,
+ 0x00000000 /* phy 405 */,
+ 0x00000000 /* phy 406 */,
+ 0x00000000 /* phy 407 */,
+ 0x00000000 /* phy 408 */,
+ 0x04080000 /* phy 409 */,
+ 0x04080400 /* phy 410 */,
+ 0x10000000 /* phy 411 */,
+ 0x0c008007 /* phy 412 */,
+ 0x00000100 /* phy 413 */,
+ 0x00000100 /* phy 414 */,
+ 0x55555555 /* phy 415 */,
+ 0xaaaaaaaa /* phy 416 */,
+ 0x55555555 /* phy 417 */,
+ 0xaaaaaaaa /* phy 418 */,
+ 0x00005555 /* phy 419 */,
+ 0x00000000 /* phy 420 */,
+ 0x00000000 /* phy 421 */,
+ 0x00000000 /* phy 422 */,
+ 0x00000000 /* phy 423 */,
+ 0x00000000 /* phy 424 */,
+ 0x00000000 /* phy 425 */,
+ 0x00000000 /* phy 426 */,
+ 0x00000000 /* phy 427 */,
+ 0x00000000 /* phy 428 */,
+ 0x00000000 /* phy 429 */,
+ 0x00000000 /* phy 430 */,
+ 0x00000000 /* phy 431 */,
+ 0x00000000 /* phy 432 */,
+ 0x00000000 /* phy 433 */,
+ 0x00000000 /* phy 434 */,
+ 0x00000000 /* phy 435 */,
+ 0x00000000 /* phy 436 */,
+ 0x00000000 /* phy 437 */,
+ 0x00000000 /* phy 438 */,
+ 0x00000104 /* phy 439 */,
+ 0x00000020 /* phy 440 */,
+ 0x00000000 /* phy 441 */,
+ 0x00000000 /* phy 442 */,
+ 0x00000000 /* phy 443 */,
+ 0x00000000 /* phy 444 */,
+ 0x00000000 /* phy 445 */,
+ 0x00000000 /* phy 446 */,
+ 0x00000000 /* phy 447 */,
+ 0x02800280 /* phy 448 */,
+ 0x02800280 /* phy 449 */,
+ 0x02800280 /* phy 450 */,
+ 0x02800280 /* phy 451 */,
+ 0x00000280 /* phy 452 */,
+ 0x00000000 /* phy 453 */,
+ 0x00000000 /* phy 454 */,
+ 0x00000000 /* phy 455 */,
+ 0x00000000 /* phy 456 */,
+ 0x00000000 /* phy 457 */,
+ 0x00800080 /* phy 458 */,
+ 0x00800080 /* phy 459 */,
+ 0x00800080 /* phy 460 */,
+ 0x00800080 /* phy 461 */,
+ 0x00800080 /* phy 462 */,
+ 0x00800080 /* phy 463 */,
+ 0x00800080 /* phy 464 */,
+ 0x00800080 /* phy 465 */,
+ 0x00800080 /* phy 466 */,
+ 0x00010019 /* phy 467 */,
+ 0x000001d0 /* phy 468 */,
+ 0x00000000 /* phy 469 */,
+ 0x00000200 /* phy 470 */,
+ 0x00000004 /* phy 471 */,
+ 0x51816152 /* phy 472 */,
+ 0xc0c08161 /* phy 473 */,
+ 0x00010000 /* phy 474 */,
+ 0x02001000 /* phy 475 */,
+ 0x0c0432ff /* phy 476 */,
+ 0x000f0c18 /* phy 477 */,
+ 0x01000140 /* phy 478 */,
+ 0x00000c20 /* phy 479 */,
+ 0x00000000 /* phy 480 */,
+ 0x00000000 /* phy 481 */,
+ 0x00000000 /* phy 482 */,
+ 0x00000000 /* phy 483 */,
+ 0x00000000 /* phy 484 */,
+ 0x00000000 /* phy 485 */,
+ 0x00000000 /* phy 486 */,
+ 0x00000000 /* phy 487 */,
+ 0x00000000 /* phy 488 */,
+ 0x00000000 /* phy 489 */,
+ 0x00000000 /* phy 490 */,
+ 0x00000000 /* phy 491 */,
+ 0x00000000 /* phy 492 */,
+ 0x00000000 /* phy 493 */,
+ 0x00000000 /* phy 494 */,
+ 0x00000000 /* phy 495 */,
+ 0x00000000 /* phy 496 */,
+ 0x00000000 /* phy 497 */,
+ 0x00000000 /* phy 498 */,
+ 0x00000000 /* phy 499 */,
+ 0x00000000 /* phy 500 */,
+ 0x00000000 /* phy 501 */,
+ 0x00000000 /* phy 502 */,
+ 0x00000000 /* phy 503 */,
+ 0x00000000 /* phy 504 */,
+ 0x00000000 /* phy 505 */,
+ 0x00000000 /* phy 506 */,
+ 0x00000000 /* phy 507 */,
+ 0x00000000 /* phy 508 */,
+ 0x00000000 /* phy 509 */,
+ 0x00000000 /* phy 510 */,
+ 0x00000000 /* phy 511 */,
+ 0x00000000 /* phy 512 */,
+ 0x00000000 /* phy 513 */,
+ 0x00000000 /* phy 514 */,
+ 0x00000000 /* phy 515 */,
+ 0x00000000 /* phy 516 */,
+ 0x00000000 /* phy 517 */,
+ 0x00000000 /* phy 518 */,
+ 0x00010000 /* phy 519 */,
+ 0x00000200 /* phy 520 */,
+ 0x00000000 /* phy 521 */,
+ 0x00000000 /* phy 522 */,
+ 0x00000000 /* phy 523 */,
+ 0x00400320 /* phy 524 */,
+ 0x00000040 /* phy 525 */,
+ 0x00000098 /* phy 526 */,
+ 0x03000098 /* phy 527 */,
+ 0x00000200 /* phy 528 */,
+ 0x00000000 /* phy 529 */,
+ 0x00000000 /* phy 530 */,
+ 0x00000000 /* phy 531 */,
+ 0x0000002a /* phy 532 */,
+ 0x00000015 /* phy 533 */,
+ 0x00000015 /* phy 534 */,
+ 0x0000002a /* phy 535 */,
+ 0x00000033 /* phy 536 */,
+ 0x0000000c /* phy 537 */,
+ 0x0000000c /* phy 538 */,
+ 0x00000033 /* phy 539 */,
+ 0x00000010 /* phy 540 */,
+ 0x00030000 /* phy 541 */,
+ 0x00000003 /* phy 542 */,
+ 0x000300ce /* phy 543 */,
+ 0x03000300 /* phy 544 */,
+ 0x03000300 /* phy 545 */,
+ 0x00000300 /* phy 546 */,
+ 0xff020010 /* phy 547 */,
+ 0x00000332 /* phy 548 */,
+ 0x00000000 /* phy 549 */,
+ 0x00000000 /* phy 550 */,
+ 0x00000000 /* phy 551 */,
+ 0x00000000 /* phy 552 */,
+ 0x00000000 /* phy 553 */,
+ 0x00000000 /* phy 554 */,
+ 0x00000000 /* phy 555 */,
+ 0x00000000 /* phy 556 */,
+ 0x00000000 /* phy 557 */,
+ 0x00000000 /* phy 558 */,
+ 0x00000000 /* phy 559 */,
+ 0x00000000 /* phy 560 */,
+ 0x00000000 /* phy 561 */,
+ 0x00000000 /* phy 562 */,
+ 0x00000000 /* phy 563 */,
+ 0x00000000 /* phy 564 */,
+ 0x00000000 /* phy 565 */,
+ 0x00000000 /* phy 566 */,
+ 0x00000000 /* phy 567 */,
+ 0x00000000 /* phy 568 */,
+ 0x00000000 /* phy 569 */,
+ 0x00000000 /* phy 570 */,
+ 0x00000000 /* phy 571 */,
+ 0x00000000 /* phy 572 */,
+ 0x00000000 /* phy 573 */,
+ 0x00000000 /* phy 574 */,
+ 0x00000000 /* phy 575 */,
+ 0x00000000 /* phy 576 */,
+ 0x00000000 /* phy 577 */,
+ 0x00000000 /* phy 578 */,
+ 0x00000000 /* phy 579 */,
+ 0x00000000 /* phy 580 */,
+ 0x00000000 /* phy 581 */,
+ 0x00000000 /* phy 582 */,
+ 0x00000000 /* phy 583 */,
+ 0x00000000 /* phy 584 */,
+ 0x00000000 /* phy 585 */,
+ 0x00000000 /* phy 586 */,
+ 0x00000000 /* phy 587 */,
+ 0x00000000 /* phy 588 */,
+ 0x00000000 /* phy 589 */,
+ 0x00000000 /* phy 590 */,
+ 0x00000000 /* phy 591 */,
+ 0x00000000 /* phy 592 */,
+ 0x00000000 /* phy 593 */,
+ 0x00000000 /* phy 594 */,
+ 0x00000000 /* phy 595 */,
+ 0x00000000 /* phy 596 */,
+ 0x00000000 /* phy 597 */,
+ 0x00000000 /* phy 598 */,
+ 0x00000000 /* phy 599 */,
+ 0x00000000 /* phy 600 */,
+ 0x00000000 /* phy 601 */,
+ 0x00000000 /* phy 602 */,
+ 0x00000000 /* phy 603 */,
+ 0x00000000 /* phy 604 */,
+ 0x00000000 /* phy 605 */,
+ 0x00000000 /* phy 606 */,
+ 0x00000000 /* phy 607 */,
+ 0x00000000 /* phy 608 */,
+ 0x00000000 /* phy 609 */,
+ 0x00000000 /* phy 610 */,
+ 0x00000000 /* phy 611 */,
+ 0x00000000 /* phy 612 */,
+ 0x00000000 /* phy 613 */,
+ 0x00000000 /* phy 614 */,
+ 0x00000000 /* phy 615 */,
+ 0x00000000 /* phy 616 */,
+ 0x00000000 /* phy 617 */,
+ 0x00000000 /* phy 618 */,
+ 0x00000000 /* phy 619 */,
+ 0x00000000 /* phy 620 */,
+ 0x00000000 /* phy 621 */,
+ 0x00000000 /* phy 622 */,
+ 0x00000000 /* phy 623 */,
+ 0x00000000 /* phy 624 */,
+ 0x00000000 /* phy 625 */,
+ 0x00000000 /* phy 626 */,
+ 0x00000000 /* phy 627 */,
+ 0x00000000 /* phy 628 */,
+ 0x00000000 /* phy 629 */,
+ 0x00000000 /* phy 630 */,
+ 0x00000000 /* phy 631 */,
+ 0x00000000 /* phy 632 */,
+ 0x00000000 /* phy 633 */,
+ 0x00000000 /* phy 634 */,
+ 0x00000000 /* phy 635 */,
+ 0x00000000 /* phy 636 */,
+ 0x00000000 /* phy 637 */,
+ 0x00000000 /* phy 638 */,
+ 0x00000000 /* phy 639 */,
+ 0x00000000 /* phy 640 */,
+ 0x00000000 /* phy 641 */,
+ 0x00000000 /* phy 642 */,
+ 0x00000000 /* phy 643 */,
+ 0x00000000 /* phy 644 */,
+ 0x00000000 /* phy 645 */,
+ 0x00000000 /* phy 646 */,
+ 0x00010000 /* phy 647 */,
+ 0x00000200 /* phy 648 */,
+ 0x00000000 /* phy 649 */,
+ 0x00000000 /* phy 650 */,
+ 0x00000000 /* phy 651 */,
+ 0x00400320 /* phy 652 */,
+ 0x00000040 /* phy 653 */,
+ 0x0000dcba /* phy 654 */,
+ 0x0300dcba /* phy 655 */,
+ 0x00000200 /* phy 656 */,
+ 0x00000000 /* phy 657 */,
+ 0x00000000 /* phy 658 */,
+ 0x00000000 /* phy 659 */,
+ 0x0000002a /* phy 660 */,
+ 0x00000015 /* phy 661 */,
+ 0x00000015 /* phy 662 */,
+ 0x0000002a /* phy 663 */,
+ 0x00000033 /* phy 664 */,
+ 0x0000000c /* phy 665 */,
+ 0x0000000c /* phy 666 */,
+ 0x00000033 /* phy 667 */,
+ 0x00005432 /* phy 668 */,
+ 0x000f0000 /* phy 669 */,
+ 0x0000000f /* phy 670 */,
+ 0x000300ce /* phy 671 */,
+ 0x03000300 /* phy 672 */,
+ 0x03000300 /* phy 673 */,
+ 0x00000300 /* phy 674 */,
+ 0xff020010 /* phy 675 */,
+ 0x00000332 /* phy 676 */,
+ 0x00000000 /* phy 677 */,
+ 0x00000000 /* phy 678 */,
+ 0x00000000 /* phy 679 */,
+ 0x00000000 /* phy 680 */,
+ 0x00000000 /* phy 681 */,
+ 0x00000000 /* phy 682 */,
+ 0x00000000 /* phy 683 */,
+ 0x00000000 /* phy 684 */,
+ 0x00000000 /* phy 685 */,
+ 0x00000000 /* phy 686 */,
+ 0x00000000 /* phy 687 */,
+ 0x00000000 /* phy 688 */,
+ 0x00000000 /* phy 689 */,
+ 0x00000000 /* phy 690 */,
+ 0x00000000 /* phy 691 */,
+ 0x00000000 /* phy 692 */,
+ 0x00000000 /* phy 693 */,
+ 0x00000000 /* phy 694 */,
+ 0x00000000 /* phy 695 */,
+ 0x00000000 /* phy 696 */,
+ 0x00000000 /* phy 697 */,
+ 0x00000000 /* phy 698 */,
+ 0x00000000 /* phy 699 */,
+ 0x00000000 /* phy 700 */,
+ 0x00000000 /* phy 701 */,
+ 0x00000000 /* phy 702 */,
+ 0x00000000 /* phy 703 */,
+ 0x00000000 /* phy 704 */,
+ 0x00000000 /* phy 705 */,
+ 0x00000000 /* phy 706 */,
+ 0x00000000 /* phy 707 */,
+ 0x00000000 /* phy 708 */,
+ 0x00000000 /* phy 709 */,
+ 0x00000000 /* phy 710 */,
+ 0x00000000 /* phy 711 */,
+ 0x00000000 /* phy 712 */,
+ 0x00000000 /* phy 713 */,
+ 0x00000000 /* phy 714 */,
+ 0x00000000 /* phy 715 */,
+ 0x00000000 /* phy 716 */,
+ 0x00000000 /* phy 717 */,
+ 0x00000000 /* phy 718 */,
+ 0x00000000 /* phy 719 */,
+ 0x00000000 /* phy 720 */,
+ 0x00000000 /* phy 721 */,
+ 0x00000000 /* phy 722 */,
+ 0x00000000 /* phy 723 */,
+ 0x00000000 /* phy 724 */,
+ 0x00000000 /* phy 725 */,
+ 0x00000000 /* phy 726 */,
+ 0x00000000 /* phy 727 */,
+ 0x00000000 /* phy 728 */,
+ 0x00000000 /* phy 729 */,
+ 0x00000000 /* phy 730 */,
+ 0x00000000 /* phy 731 */,
+ 0x00000000 /* phy 732 */,
+ 0x00000000 /* phy 733 */,
+ 0x00000000 /* phy 734 */,
+ 0x00000000 /* phy 735 */,
+ 0x00000000 /* phy 736 */,
+ 0x00000000 /* phy 737 */,
+ 0x00000000 /* phy 738 */,
+ 0x00000000 /* phy 739 */,
+ 0x00000000 /* phy 740 */,
+ 0x00000000 /* phy 741 */,
+ 0x00000000 /* phy 742 */,
+ 0x00000000 /* phy 743 */,
+ 0x00000000 /* phy 744 */,
+ 0x00000000 /* phy 745 */,
+ 0x00000000 /* phy 746 */,
+ 0x00000000 /* phy 747 */,
+ 0x00000000 /* phy 748 */,
+ 0x00000000 /* phy 749 */,
+ 0x00000000 /* phy 750 */,
+ 0x00000000 /* phy 751 */,
+ 0x00000000 /* phy 752 */,
+ 0x00000000 /* phy 753 */,
+ 0x00000000 /* phy 754 */,
+ 0x00000000 /* phy 755 */,
+ 0x00000000 /* phy 756 */,
+ 0x00000000 /* phy 757 */,
+ 0x00000000 /* phy 758 */,
+ 0x00000000 /* phy 759 */,
+ 0x00000000 /* phy 760 */,
+ 0x00000000 /* phy 761 */,
+ 0x00000000 /* phy 762 */,
+ 0x00000000 /* phy 763 */,
+ 0x00000000 /* phy 764 */,
+ 0x00000000 /* phy 765 */,
+ 0x00000000 /* phy 766 */,
+ 0x00000000 /* phy 767 */,
+ 0x00000000 /* phy 768 */,
+ 0x00000000 /* phy 769 */,
+ 0x00000000 /* phy 770 */,
+ 0x00000000 /* phy 771 */,
+ 0x00000000 /* phy 772 */,
+ 0x00000000 /* phy 773 */,
+ 0x00000000 /* phy 774 */,
+ 0x00010000 /* phy 775 */,
+ 0x00000200 /* phy 776 */,
+ 0x00000000 /* phy 777 */,
+ 0x00000000 /* phy 778 */,
+ 0x00000000 /* phy 779 */,
+ 0x00400320 /* phy 780 */,
+ 0x00000040 /* phy 781 */,
+ 0x00000098 /* phy 782 */,
+ 0x03000098 /* phy 783 */,
+ 0x00000200 /* phy 784 */,
+ 0x00000000 /* phy 785 */,
+ 0x00000000 /* phy 786 */,
+ 0x00000000 /* phy 787 */,
+ 0x0000002a /* phy 788 */,
+ 0x00000015 /* phy 789 */,
+ 0x00000015 /* phy 790 */,
+ 0x0000002a /* phy 791 */,
+ 0x00000033 /* phy 792 */,
+ 0x0000000c /* phy 793 */,
+ 0x0000000c /* phy 794 */,
+ 0x00000033 /* phy 795 */,
+ 0x00000010 /* phy 796 */,
+ 0x00030000 /* phy 797 */,
+ 0x00000003 /* phy 798 */,
+ 0x000300ce /* phy 799 */,
+ 0x03000300 /* phy 800 */,
+ 0x03000300 /* phy 801 */,
+ 0x00000300 /* phy 802 */,
+ 0xff020010 /* phy 803 */,
+ 0x00000332 /* phy 804 */,
+ 0x00000000 /* phy 805 */,
+ 0x00000000 /* phy 806 */,
+ 0x00000000 /* phy 807 */,
+ 0x00000000 /* phy 808 */,
+ 0x00000000 /* phy 809 */,
+ 0x00000000 /* phy 810 */,
+ 0x00000000 /* phy 811 */,
+ 0x00000000 /* phy 812 */,
+ 0x00000000 /* phy 813 */,
+ 0x00000000 /* phy 814 */,
+ 0x00000000 /* phy 815 */,
+ 0x00000000 /* phy 816 */,
+ 0x00000000 /* phy 817 */,
+ 0x00000000 /* phy 818 */,
+ 0x00000000 /* phy 819 */,
+ 0x00000000 /* phy 820 */,
+ 0x00000000 /* phy 821 */,
+ 0x00000000 /* phy 822 */,
+ 0x00000000 /* phy 823 */,
+ 0x00000000 /* phy 824 */,
+ 0x00000000 /* phy 825 */,
+ 0x00000000 /* phy 826 */,
+ 0x00000000 /* phy 827 */,
+ 0x00000000 /* phy 828 */,
+ 0x00000000 /* phy 829 */,
+ 0x00000000 /* phy 830 */,
+ 0x00000000 /* phy 831 */,
+ 0x00000000 /* phy 832 */,
+ 0x00000000 /* phy 833 */,
+ 0x00000000 /* phy 834 */,
+ 0x00000000 /* phy 835 */,
+ 0x00000000 /* phy 836 */,
+ 0x00000000 /* phy 837 */,
+ 0x00000000 /* phy 838 */,
+ 0x00000000 /* phy 839 */,
+ 0x00000000 /* phy 840 */,
+ 0x00000000 /* phy 841 */,
+ 0x00000000 /* phy 842 */,
+ 0x00000000 /* phy 843 */,
+ 0x00000000 /* phy 844 */,
+ 0x00000000 /* phy 845 */,
+ 0x00000000 /* phy 846 */,
+ 0x00000000 /* phy 847 */,
+ 0x00000000 /* phy 848 */,
+ 0x00000000 /* phy 849 */,
+ 0x00000000 /* phy 850 */,
+ 0x00000000 /* phy 851 */,
+ 0x00000000 /* phy 852 */,
+ 0x00000000 /* phy 853 */,
+ 0x00000000 /* phy 854 */,
+ 0x00000000 /* phy 855 */,
+ 0x00000000 /* phy 856 */,
+ 0x00000000 /* phy 857 */,
+ 0x00000000 /* phy 858 */,
+ 0x00000000 /* phy 859 */,
+ 0x00000000 /* phy 860 */,
+ 0x00000000 /* phy 861 */,
+ 0x00000000 /* phy 862 */,
+ 0x00000000 /* phy 863 */,
+ 0x00000000 /* phy 864 */,
+ 0x00000000 /* phy 865 */,
+ 0x00000000 /* phy 866 */,
+ 0x00000000 /* phy 867 */,
+ 0x00000000 /* phy 868 */,
+ 0x00000000 /* phy 869 */,
+ 0x00000000 /* phy 870 */,
+ 0x00000000 /* phy 871 */,
+ 0x00000000 /* phy 872 */,
+ 0x00000000 /* phy 873 */,
+ 0x00000000 /* phy 874 */,
+ 0x00000000 /* phy 875 */,
+ 0x00000000 /* phy 876 */,
+ 0x00000000 /* phy 877 */,
+ 0x00000000 /* phy 878 */,
+ 0x00000000 /* phy 879 */,
+ 0x00000000 /* phy 880 */,
+ 0x00000000 /* phy 881 */,
+ 0x00000000 /* phy 882 */,
+ 0x00000000 /* phy 883 */,
+ 0x00000000 /* phy 884 */,
+ 0x00000000 /* phy 885 */,
+ 0x00000000 /* phy 886 */,
+ 0x00000000 /* phy 887 */,
+ 0x00000000 /* phy 888 */,
+ 0x00000000 /* phy 889 */,
+ 0x00000000 /* phy 890 */,
+ 0x00000000 /* phy 891 */,
+ 0x00000000 /* phy 892 */,
+ 0x00000000 /* phy 893 */,
+ 0x00000000 /* phy 894 */,
+ 0x00000000 /* phy 895 */,
+ 0x00000000 /* phy 896 */,
+ 0x00000000 /* phy 897 */,
+ 0x00000000 /* phy 898 */,
+ 0x00000000 /* phy 899 */,
+ 0x00000000 /* phy 900 */,
+ 0x00000000 /* phy 901 */,
+ 0x00000000 /* phy 902 */,
+ 0x00010000 /* phy 903 */,
+ 0x00000200 /* phy 904 */,
+ 0x00000000 /* phy 905 */,
+ 0x00000000 /* phy 906 */,
+ 0x00000000 /* phy 907 */,
+ 0x00400320 /* phy 908 */,
+ 0x00000040 /* phy 909 */,
+ 0x0000dcba /* phy 910 */,
+ 0x0300dcba /* phy 911 */,
+ 0x00000200 /* phy 912 */,
+ 0x00000000 /* phy 913 */,
+ 0x00000000 /* phy 914 */,
+ 0x00000000 /* phy 915 */,
+ 0x0000002a /* phy 916 */,
+ 0x00000015 /* phy 917 */,
+ 0x00000015 /* phy 918 */,
+ 0x0000002a /* phy 919 */,
+ 0x00000033 /* phy 920 */,
+ 0x0000000c /* phy 921 */,
+ 0x0000000c /* phy 922 */,
+ 0x00000033 /* phy 923 */,
+ 0x00005432 /* phy 924 */,
+ 0x000f0000 /* phy 925 */,
+ 0x0000000f /* phy 926 */,
+ 0x000300ce /* phy 927 */,
+ 0x03000300 /* phy 928 */,
+ 0x03000300 /* phy 929 */,
+ 0x00000300 /* phy 930 */,
+ 0xff020010 /* phy 931 */,
+ 0x00000332 /* phy 932 */,
+ 0x00000000 /* phy 933 */,
+ 0x00000000 /* phy 934 */,
+ 0x00000000 /* phy 935 */,
+ 0x00000000 /* phy 936 */,
+ 0x00000000 /* phy 937 */,
+ 0x00000000 /* phy 938 */,
+ 0x00000000 /* phy 939 */,
+ 0x00000000 /* phy 940 */,
+ 0x00000000 /* phy 941 */,
+ 0x00000000 /* phy 942 */,
+ 0x00000000 /* phy 943 */,
+ 0x00000000 /* phy 944 */,
+ 0x00000000 /* phy 945 */,
+ 0x00000000 /* phy 946 */,
+ 0x00000000 /* phy 947 */,
+ 0x00000000 /* phy 948 */,
+ 0x00000000 /* phy 949 */,
+ 0x00000000 /* phy 950 */,
+ 0x00000000 /* phy 951 */,
+ 0x00000000 /* phy 952 */,
+ 0x00000000 /* phy 953 */,
+ 0x00000000 /* phy 954 */,
+ 0x00000000 /* phy 955 */,
+ 0x00000000 /* phy 956 */,
+ 0x00000000 /* phy 957 */,
+ 0x00000000 /* phy 958 */,
+ 0x00000000 /* phy 959 */,
+ 0x00000000 /* phy 960 */,
+ 0x00000000 /* phy 961 */,
+ 0x00000000 /* phy 962 */,
+ 0x00000000 /* phy 963 */,
+ 0x00000000 /* phy 964 */,
+ 0x00000000 /* phy 965 */,
+ 0x00000000 /* phy 966 */,
+ 0x00000000 /* phy 967 */,
+ 0x00000000 /* phy 968 */,
+ 0x00000000 /* phy 969 */,
+ 0x00000000 /* phy 970 */,
+ 0x00000000 /* phy 971 */,
+ 0x00000000 /* phy 972 */,
+ 0x00000000 /* phy 973 */,
+ 0x00000000 /* phy 974 */,
+ 0x00000000 /* phy 975 */,
+ 0x00000000 /* phy 976 */,
+ 0x00000000 /* phy 977 */,
+ 0x00000000 /* phy 978 */,
+ 0x00000000 /* phy 979 */,
+ 0x00000000 /* phy 980 */,
+ 0x00000000 /* phy 981 */,
+ 0x00000000 /* phy 982 */,
+ 0x00000000 /* phy 983 */,
+ 0x00000000 /* phy 984 */,
+ 0x00000000 /* phy 985 */,
+ 0x00000000 /* phy 986 */,
+ 0x00000000 /* phy 987 */,
+ 0x00000000 /* phy 988 */,
+ 0x00000000 /* phy 989 */,
+ 0x00000000 /* phy 990 */,
+ 0x00000000 /* phy 991 */,
+ 0x00000000 /* phy 992 */,
+ 0x00000000 /* phy 993 */,
+ 0x00000000 /* phy 994 */,
+ 0x00000000 /* phy 995 */,
+ 0x00000000 /* phy 996 */,
+ 0x00000000 /* phy 997 */,
+ 0x00000000 /* phy 998 */,
+ 0x00000000 /* phy 999 */,
+ 0x00000000 /* phy 1000 */,
+ 0x00000000 /* phy 1001 */,
+ 0x00000000 /* phy 1002 */,
+ 0x00000000 /* phy 1003 */,
+ 0x00000000 /* phy 1004 */,
+ 0x00000000 /* phy 1005 */,
+ 0x00000000 /* phy 1006 */,
+ 0x00000000 /* phy 1007 */,
+ 0x00000000 /* phy 1008 */,
+ 0x00000000 /* phy 1009 */,
+ 0x00000000 /* phy 1010 */,
+ 0x00000000 /* phy 1011 */,
+ 0x00000000 /* phy 1012 */,
+ 0x00000000 /* phy 1013 */,
+ 0x00000000 /* phy 1014 */,
+ 0x00000000 /* phy 1015 */,
+ 0x00000000 /* phy 1016 */,
+ 0x00000000 /* phy 1017 */,
+ 0x00000000 /* phy 1018 */,
+ 0x00000000 /* phy 1019 */,
+ 0x00000000 /* phy 1020 */,
+ 0x00000000 /* phy 1021 */,
+ 0x00000000 /* phy 1022 */,
+ 0x00000000 /* phy 1023 */,
+ 0x00000000 /* phy 1024 */,
+ 0x00000100 /* phy 1025 */,
+ 0x05000000 /* phy 1026 */,
+ 0x01000000 /* phy 1027 */,
+ 0x00040003 /* phy 1028 */,
+ 0x00320100 /* phy 1029 */,
+ 0x02005502 /* phy 1030 */,
+ 0x00000000 /* phy 1031 */,
+ 0x00000000 /* phy 1032 */,
+ 0x00000000 /* phy 1033 */,
+ 0x00000004 /* phy 1034 */,
+ 0x00000000 /* phy 1035 */,
+ 0x01010100 /* phy 1036 */,
+ 0x00000200 /* phy 1037 */,
+ 0x01000000 /* phy 1038 */,
+ 0x00000001 /* phy 1039 */,
+ 0x00000000 /* phy 1040 */,
+ 0x00000000 /* phy 1041 */,
+ 0x00000000 /* phy 1042 */,
+ 0x00000000 /* phy 1043 */,
+ 0x0000ffff /* phy 1044 */,
+ 0x05221502 /* phy 1045 */,
+ 0x00000522 /* phy 1046 */,
+ 0x00000000 /* phy 1047 */,
+ 0x000f1f01 /* phy 1048 */,
+ 0x0f1f0f1f /* phy 1049 */,
+ 0x0f1f0f1f /* phy 1050 */,
+ 0x03000003 /* phy 1051 */,
+ 0x00000300 /* phy 1052 */,
+ 0x0b221b02 /* phy 1053 */,
+ 0x09240b22 /* phy 1054 */,
+ 0x00000000 /* phy 1055 */,
+ 0x00000000 /* phy 1056 */,
+ 0x05030000 /* phy 1057 */,
+ 0x14000001 /* phy 1058 */,
+ 0x63c0ce00 /* phy 1059 */,
+ 0x0000000e /* phy 1060 */,
+ 0x001f0fc0 /* phy 1061 */,
+ 0x003c37c0 /* phy 1062 */,
+ 0x1f0fc0ce /* phy 1063 */,
+ 0x3c37c0ce /* phy 1064 */,
+ 0x00024410 /* phy 1065 */,
+ 0x00004410 /* phy 1066 */,
+ 0x00004410 /* phy 1067 */,
+ 0x00024410 /* phy 1068 */,
+ 0x00004410 /* phy 1069 */,
+ 0x000fc0ff /* phy 1070 */,
+ 0x00004410 /* phy 1071 */,
+ 0x000fc0ff /* phy 1072 */,
+ 0x00004410 /* phy 1073 */,
+ 0x1f0fc0ce /* phy 1074 */,
+ 0x00004410 /* phy 1075 */,
+ 0x00004011 /* phy 1076 */,
+ 0x00004410 /* phy 1077 */,
+ 0x00000000 /* phy 1078 */,
+ 0x00000000 /* phy 1079 */,
+ 0x00000000 /* phy 1080 */,
+ 0x00000000 /* phy 1081 */,
+ 0x00000078 /* phy 1082 */,
+ 0x00000000 /* phy 1083 */,
+ 0x00010108 /* phy 1084 */,
+ 0x00000000 /* phy 1085 */,
+ 0x03000000 /* phy 1086 */,
+ 0x00000008 /* phy 1087 */,
+ 0x00000000 /* phy 1088 */,
+ 0x00000000 /* phy 1089 */,
+ 0x00000000 /* phy 1090 */,
+ 0x00000000 /* phy 1091 */,
+ 0x00000001 /* phy 1092 */,
+ 0x00000000 /* phy 1093 */,
+ 0x00000000 /* phy 1094 */,
+ 0x01000000 /* phy 1095 */,
+ 0x00000001 /* phy 1096 */,
+ 0x00000000 /* phy 1097 */,
+ 0x07010000 /* phy 1098 */,
+ 0x00000204 /* phy 1099 */,
+ 0x00000000 /* phy 1100 */
+ },
+ {
+ { 83, 0x00020119 } /* setA */,
+ { 90, 0x01020000 } /* setA */,
+ { 92, 0x0c053eff } /* setA */,
+ { 211, 0x00020119 } /* setA */,
+ { 218, 0x01020000 } /* setA */,
+ { 220, 0x0c053eff } /* setA */,
+ { 339, 0x00020119 } /* setA */,
+ { 346, 0x01020000 } /* setA */,
+ { 348, 0x0c053eff } /* setA */,
+ { 467, 0x00020119 } /* setA */,
+ { 474, 0x01020000 } /* setA */,
+ { 476, 0x0c053eff } /* setA */,
+ { 548, 0x0000033e } /* setA */,
+ { 676, 0x0000033e } /* setA */,
+ { 804, 0x0000033e } /* setA */,
+ { 932, 0x0000033e } /* setA */,
+ { 1045, 0x01221102 } /* setA */,
+ { 1046, 0x00000122 } /* setA */,
+ { 1048, 0x000f1f00 } /* setA */,
+ { 0xFFFFFFFF, 0xFFFFFFFF } /* termination */
+ },
+ {
+ { 83, 0x0003011a } /* setB */,
+ { 85, 0x01000000 } /* setB */,
+ { 90, 0x02030000 } /* setB */,
+ { 92, 0x0c073eff } /* setB */,
+ { 211, 0x0003011a } /* setB */,
+ { 213, 0x01000000 } /* setB */,
+ { 218, 0x02030000 } /* setB */,
+ { 220, 0x0c073eff } /* setB */,
+ { 339, 0x0003011a } /* setB */,
+ { 341, 0x01000000 } /* setB */,
+ { 346, 0x02030000 } /* setB */,
+ { 348, 0x0c073eff } /* setB */,
+ { 467, 0x0003011a } /* setB */,
+ { 469, 0x01000000 } /* setB */,
+ { 474, 0x02030000 } /* setB */,
+ { 476, 0x0c073eff } /* setB */,
+ { 548, 0x0000033e } /* setB */,
+ { 676, 0x0000033e } /* setB */,
+ { 804, 0x0000033e } /* setB */,
+ { 932, 0x0000033e } /* setB */,
+ { 1045, 0x01221102 } /* setB */,
+ { 1046, 0x00000122 } /* setB */,
+ { 1048, 0x000f1f00 } /* setB */,
+ { 0xFFFFFFFF, 0xFFFFFFFF } /* termination */
+ },
+};
+#endif
+
diff --git a/drivers/misc/mnh/mnh-ddr.c b/drivers/misc/mnh/mnh-ddr.c
new file mode 100644
index 0000000..d1c32d3
--- /dev/null
+++ b/drivers/misc/mnh/mnh-ddr.c
@@ -0,0 +1,651 @@
+/*
+*
+* MNH DDR Driver
+* Copyright (c) 2016-2017, Intel Corporation.
+*
+* This program is free software; you can redistribute it and/or modify it
+* under the terms and conditions of the GNU General Public License,
+* version 2, as published by the Free Software Foundation.
+*
+* This program is distributed in the hope it will be useful, but WITHOUT
+* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+* more details.
+*
+*/
+
+#include "mnh-clk.h"
+#include "mnh-hwio.h"
+#include "mnh-hwio-bases.h"
+#include "mnh-hwio-ddr-ctl.h"
+#include "mnh-hwio-ddr-pi.h"
+#include "mnh-hwio-ddr-phy.h"
+#include "mnh-hwio-scu.h"
+#include "mnh-ddr-33-100-400-600.h"
+#include "mnh-ddr.h"
+#include "mnh-pcie.h"
+#include "mnh-sm.h"
+
+#define MNH_DDR_CTL_IN(reg) \
+ HW_IN(HWIO_DDR_CTL_BASE_ADDR, DDR_CTL, reg)
+#define MNH_DDR_CTL_INf(reg, fld) \
+ HW_INf(HWIO_DDR_CTL_BASE_ADDR, DDR_CTL, reg, fld)
+#define MNH_DDR_CTL_OUT(reg, val) \
+ HW_OUT(HWIO_DDR_CTL_BASE_ADDR, DDR_CTL, reg, val)
+#define MNH_DDR_CTL_OUTf(reg, fld, val) \
+ HW_OUTf(HWIO_DDR_CTL_BASE_ADDR, DDR_CTL, reg, fld, val)
+
+#define MNH_DDR_PI_INf(reg, fld) \
+ HW_INf(HWIO_DDR_PI_BASE_ADDR, DDR_PI, reg, fld)
+#define MNH_DDR_PI_OUTf(reg, fld, val) \
+ HW_OUTf(HWIO_DDR_PI_BASE_ADDR, DDR_PI, reg, fld, val)
+#define MNH_DDR_PI_OUT(reg, val) \
+ HW_OUT(HWIO_DDR_PI_BASE_ADDR, DDR_PI, reg, val)
+
+#define MNH_DDR_PHY_INf(reg, fld) \
+ HW_INf(HWIO_DDR_PHY_BASE_ADDR, DDR_PHY, reg, fld)
+#define MNH_DDR_PHY_OUTf(reg, fld, val) \
+ HW_OUTf(HWIO_DDR_PHY_BASE_ADDR, DDR_PHY, reg, fld, val)
+#define MNH_DDR_PHY_OUT(reg, val) \
+ HW_OUT(HWIO_DDR_PHY_BASE_ADDR, DDR_PHY, reg, val)
+
+#define MNH_SCU_IN(reg) \
+ HW_IN(HWIO_SCU_BASE_ADDR, SCU, reg)
+#define MNH_SCU_INf(reg, fld) \
+ HW_INf(HWIO_SCU_BASE_ADDR, SCU, reg, fld)
+#define MNH_SCU_INx(reg, inst) \
+ HW_INx(HWIO_SCU_BASE_ADDR, SCU, reg, inst)
+#define MNH_SCU_INxf(reg, inst, fld) \
+ HW_INxf(HWIO_SCU_BASE_ADDR, SCU, reg, inst, fld)
+#define MNH_SCU_OUTf(reg, fld, val) \
+ HW_OUTf(HWIO_SCU_BASE_ADDR, SCU, reg, fld, val)
+#define MNH_SCU_OUT(reg, val) \
+ HW_OUT(HWIO_SCU_BASE_ADDR, SCU, reg, val)
+#define MNH_SCU_OUTx(reg, inst, val) \
+ HW_OUTx(HWIO_SCU_BASE_ADDR, SCU, reg, inst, val)
+
+#define MNH_RSTC_INf(fld) \
+ HW_INf(HWIO_SCU_BASE_ADDR, SCU, RSTC, fld)
+#define MNH_RSTC_OUTf(fld, val) \
+ HW_OUTf(HWIO_SCU_BASE_ADDR, SCU, RSTC, fld, val)
+
+#define WRITE_DDR_REG_CONFIG(_state, ddrblock, regindex) \
+do { \
+ if (_state->ddrblock[regindex]) { \
+ mnh_reg_write(_state->ddrblock##_base + \
+ (regindex * sizeof(u32)), \
+ _state->ddrblock[regindex]); \
+ } \
+} while (0)
+
+#define WRITE_DDR_PHY_CONFIG(_state, fsp, regindex) \
+do { \
+ if (_state->phy[fsp][regindex]) { \
+ mnh_reg_write(_state->phy_base + (regindex * sizeof(u32)), \
+ _state->phy[fsp][regindex]); \
+ } \
+} while (0)
+
+#define WRITE_SET_ELEMENT(_state, regindex, regvalue) \
+ mnh_reg_write(_state->phy_base + (regindex * sizeof(u32)),\
+ regvalue)
+
+#define WRITE_SCU_FSP(_state, fsp) \
+do { \
+ _state->fsps[fsp] &= 0xFFFFFF00;\
+ _state->fsps[fsp] |= 0x7d;\
+ MNH_SCU_OUTx(LPDDR4_FSP_SETTING, fsp, _state->fsps[fsp]); \
+} while (0)
+
+#define SAVE_CURRENT_FSP(dev, _state) \
+do { \
+ _state->suspend_fsp = \
+ MNH_SCU_INf(LPDDR4_LOW_POWER_STS, LPDDR4_CUR_FSP); \
+ dev_dbg(dev, "%s: saved fsp: %d\n", __func__, _state->suspend_fsp); \
+} while (0)
+
+#define SAVED_FSP(_state) _state->suspend_fsp
+
+#define WRITE_CLK_FROM_FSP(dev, _state, fsp) \
+do { \
+ if (fsp < (MNH_DDR_NUM_FSPS)) { \
+ MNH_SCU_OUTf(CCU_CLK_DIV, LPDDR4_REFCLK_DIV, \
+ MNH_SCU_INxf(LPDDR4_FSP_SETTING, fsp, \
+ FSP_LPDDR4_REFCLK_DIV)); \
+ MNH_SCU_OUTf(CCU_CLK_DIV, AXI_FABRIC_CLK_DIV, \
+ MNH_SCU_INxf(LPDDR4_FSP_SETTING, fsp, \
+ FSP_AXI_FABRIC_CLK_DIV)); \
+ MNH_SCU_OUTf(CCU_CLK_DIV, PCIE_AXI_CLK_DIV, \
+ MNH_SCU_INxf(LPDDR4_FSP_SETTING, fsp, \
+ FSP_PCIE_AXI_CLK_DIV)); \
+ MNH_SCU_OUTf(CCU_CLK_CTL, LP4_AXI_SYS200_MODE, \
+ MNH_SCU_INxf(LPDDR4_FSP_SETTING, fsp, \
+ FSP_SYS200_MODE)); \
+ } else \
+ dev_err(dev, "%s: invalid fsp 0x%x", __func__, fsp); \
+} while (0)
+
+#define SAVE_DDR_REG_CONFIG(_state, ddrblock, regindex) \
+do { \
+ _state->ddrblock[regindex] = \
+ mnh_reg_read(_state->ddrblock##_base + \
+ (regindex * sizeof(u32))); \
+} while (0)
+
+#define SAVE_DDR_PHY_REG_CONFIG(_state, fsp, regindex) \
+do { \
+ _state->phy[fsp][regindex] = \
+ mnh_reg_read(_state->phy_base + (regindex * sizeof(u32))); \
+} while (0)
+
+#define CLR_START(_state, ddrblock) (_state->ddrblock[0] &= (0xFFFFFFFE))
+
+/* timeout for training all FSPs */
+#define TRAINING_TIMEOUT msecs_to_jiffies(45)
+
+#define LP_CMD_EXIT_LP 0x81
+#define LP_CMD_DSRPD 0xFE
+
+/* INT status bits */
+#define MR_WRITE_SBIT 26
+#define MR_READ_SBIT 23
+#define BIST_SBIT 6
+#define LP_CMD_SBIT 5
+#define INIT_DONE_SBIT 4
+
+/* read entire int_status */
+u64 mnh_ddr_int_status(struct device *dev)
+{
+ u64 int_stat = ((u64)MNH_DDR_CTL_IN(228) << 32) | MNH_DDR_CTL_IN(227);
+ return int_stat;
+}
+EXPORT_SYMBOL(mnh_ddr_int_status);
+
+/* clear entire int_status */
+int mnh_ddr_clr_int_status(struct device *dev)
+{
+ u64 stat = 0;
+
+ MNH_DDR_CTL_OUT(230, 0x0F);
+ MNH_DDR_CTL_OUT(229, 0xFFFFFFFF);
+ stat = mnh_ddr_int_status(dev);
+ if (stat) {
+ dev_err(dev, "%s: int stat not all clear: %llx\n", __func__,
+ stat);
+ return -EIO;
+ }
+ return 0;
+}
+EXPORT_SYMBOL(mnh_ddr_clr_int_status);
+
+/* read single bit in int_status */
+static u32 mnh_ddr_int_status_bit(u8 sbit)
+{
+ u32 status = 0;
+ u32 upper = 0;
+ const u32 max_int_status_bit = 35;
+ const u32 first_upper_bit = 32;
+
+ if (sbit > max_int_status_bit)
+ return -EINVAL;
+
+ /*
+ * docs refer to int status by bit numbers 0-35,
+ * but we're only reading 32 bits at a time.
+ */
+ upper = (sbit >= first_upper_bit) ? 1 : 0;
+ sbit -= (upper) ? first_upper_bit : 0;
+
+ status = (upper) ? MNH_DDR_CTL_IN(228) : MNH_DDR_CTL_IN(227);
+ status &= (1 << sbit);
+ return status;
+}
+
+/* clear single bit in int_status */
+static int mnh_ddr_clr_int_status_bit(struct device *dev, u8 sbit)
+{
+ const u32 max_int_status_bit = 35;
+ const u32 first_upper_bit = 32;
+
+ if (sbit > max_int_status_bit)
+ return -EINVAL;
+
+ if (sbit >= first_upper_bit)
+ MNH_DDR_CTL_OUT(230, 1 << (sbit - first_upper_bit));
+ else
+ MNH_DDR_CTL_OUT(229, 1 << sbit);
+
+ if (mnh_ddr_int_status_bit(sbit)) {
+ dev_err(dev, "%s: bit %d is still set.\n", __func__, sbit);
+ return -EIO;
+ }
+ return 0;
+}
+
+static int mnh_ddr_send_lp_cmd(struct device *dev, u8 cmd)
+{
+ u32 timeout = 100000;
+
+ dev_dbg(dev, "%s sending cmd: 0x%x\n", __func__, cmd);
+ MNH_DDR_CTL_OUTf(112, LP_CMD, cmd);
+
+ while (!mnh_ddr_int_status_bit(LP_CMD_SBIT) && --timeout)
+ udelay(1);
+
+ if (!mnh_ddr_int_status_bit(LP_CMD_SBIT))
+ return -ETIMEDOUT;
+
+ return mnh_ddr_clr_int_status_bit(dev, LP_CMD_SBIT);
+}
+
+static void mnh_ddr_enable_lp(void)
+{
+ MNH_DDR_CTL_OUTf(124, LP_AUTO_SR_MC_GATE_IDLE, 0xFF);
+ MNH_DDR_CTL_OUTf(122, LP_AUTO_MEM_GATE_EN, 0x4);
+ MNH_DDR_CTL_OUTf(122, LP_AUTO_ENTRY_EN, 0x4);
+ MNH_DDR_CTL_OUTf(122, LP_AUTO_EXIT_EN, 0xF);
+}
+
+static void mnh_ddr_disable_lp(struct device *dev)
+{
+ MNH_DDR_CTL_OUTf(124, LP_AUTO_SR_MC_GATE_IDLE, 0x00);
+ MNH_DDR_CTL_OUTf(122, LP_AUTO_MEM_GATE_EN, 0x0);
+ MNH_DDR_CTL_OUTf(122, LP_AUTO_ENTRY_EN, 0x0);
+ MNH_DDR_CTL_OUTf(122, LP_AUTO_EXIT_EN, 0x0);
+ mnh_ddr_send_lp_cmd(dev, LP_CMD_EXIT_LP);
+}
+
+static void mnh_ddr_init_internal_state(struct mnh_ddr_internal_state *_state,
+ const struct mnh_ddr_reg_config *cfg)
+{
+ _state->ctl_base = HWIO_DDR_CTL_BASE_ADDR;
+ _state->pi_base = HWIO_DDR_PI_BASE_ADDR;
+ _state->phy_base = HWIO_DDR_PHY_BASE_ADDR;
+
+ memcpy(&(_state->fsps[0]),
+ &(cfg->fsps[0]),
+ MNH_DDR_NUM_FSPS * sizeof(u32));
+
+ memcpy(&(_state->ctl[0]),
+ &(cfg->ctl[0]),
+ MNH_DDR_NUM_CTL_REG * sizeof(u32));
+
+ memcpy(&(_state->phy[0][0]),
+ &(cfg->phy[0]),
+ MNH_DDR_NUM_PHY_REG * sizeof(u32));
+
+ memcpy(&(_state->pi[0]),
+ &(cfg->pi[0]),
+ MNH_DDR_NUM_PI_REG * sizeof(u32));
+
+ _state->suspend_fsp = 0;
+ _state->tref[0] = cfg->ctl[56] & 0xFFFF;
+ _state->tref[1] = cfg->ctl[57] & 0xFFFF;
+ _state->tref[2] = cfg->ctl[58] & 0xFFFF;
+ _state->tref[3] = cfg->ctl[59] & 0xFFFF;
+}
+
+static void mnh_ddr_init_clocks(struct mnh_ddr_data *data)
+{
+ struct device *dev = &data->pdev->dev;
+ struct mnh_ddr_internal_state *_state = &data->_state;
+
+ int timeout = 0;
+
+ /* MNH_PLL_PASSCODE_SET */
+ MNH_SCU_OUTf(PLL_PASSCODE, PASSCODE, 0x4CD9);
+
+ MNH_SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, FRZ_PLL_IN, 1);
+ MNH_SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, PD, 0);
+ MNH_SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, FOUTPOSTDIVPD, 0);
+ MNH_SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, FRZ_PLL_IN, 0);
+
+ dev_dbg(dev, "%s waiting for lpddr4 pll lock", __func__);
+ while ((timeout < 20) && (!MNH_SCU_INf(LPDDR4_REFCLK_PLL_STS, LOCK))) {
+ udelay(1);
+ timeout++;
+ }
+
+ if (timeout == 20)
+ dev_err(dev, "%s lpddr4 pll lock failed", __func__);
+ else
+ dev_dbg(dev, "%s lpddr4 pll locked after %d iterations",
+ __func__, timeout);
+
+ WRITE_SCU_FSP(_state, 0);
+ WRITE_SCU_FSP(_state, 1);
+ WRITE_SCU_FSP(_state, 2);
+ WRITE_SCU_FSP(_state, 3);
+
+ WRITE_CLK_FROM_FSP(dev, _state, SAVED_FSP(_state));
+ dev_dbg(dev, "%s lpddr4 pll locked", __func__);
+ MNH_SCU_OUTf(LPDDR4_LOW_POWER_CFG, LP4_FSP_SW_OVERRIDE, 0);
+ /* MNH_PLL_PASSCODE_CLR */
+ MNH_SCU_OUTf(PLL_PASSCODE, PASSCODE, 0x0);
+}
+
+static void mnh_ddr_pull_config(struct mnh_ddr_data *data)
+{
+ struct mnh_ddr_internal_state *_state = &data->_state;
+
+ int index, fsp;
+ for (index = 0; index < MNH_DDR_NUM_CTL_REG; index++)
+ SAVE_DDR_REG_CONFIG(_state, ctl, index);
+ CLR_START(_state, ctl);
+
+ for (index = 0; index < MNH_DDR_NUM_PI_REG; index++)
+ SAVE_DDR_REG_CONFIG(_state, pi, index);
+ CLR_START(_state, pi);
+
+ for (fsp = 0; fsp < MNH_DDR_NUM_FSPS; fsp++) {
+ MNH_DDR_PHY_OUTf(1025, PHY_FREQ_SEL_INDEX, fsp);
+ for (index = 0; index < MNH_DDR_NUM_PHY_REG; index++)
+ SAVE_DDR_PHY_REG_CONFIG(_state, fsp, index);
+ }
+}
+
+int mnh_ddr_suspend(struct mnh_ddr_data *data, struct gpio_desc *iso_n)
+{
+ struct device *dev = &data->pdev->dev;
+ struct mnh_ddr_internal_state *_state = &data->_state;
+
+ mnh_ddr_disable_lp(dev);
+
+ dev_dbg(dev, "%s: tref 0x%04x 0x%04x 0x%04x 0x%04x\n",
+ __func__, MNH_DDR_CTL_INf(56, TREF_F0),
+ MNH_DDR_CTL_INf(57, TREF_F1), MNH_DDR_CTL_INf(58, TREF_F2),
+ MNH_DDR_CTL_INf(59, TREF_F3));
+
+ /*
+ * restore hot tref settings, rather than what was
+ * saved from the MNH to handle the case where
+ * we're suspending in a cold temp and resuming at hot.
+ * If we're not actually hot, the MNH side will adjust
+ * the rate downward.
+ */
+ MNH_DDR_CTL_OUTf(56, TREF_F0, _state->tref[0]);
+ MNH_DDR_CTL_OUTf(57, TREF_F1, _state->tref[1]);
+ MNH_DDR_CTL_OUTf(58, TREF_F2, _state->tref[2]);
+ MNH_DDR_CTL_OUTf(59, TREF_F3, _state->tref[3]);
+
+ /* resume to fsp3 */
+ mnh_lpddr_freq_change(LPDDR_FREQ_FSP3);
+ SAVE_CURRENT_FSP(dev, _state);
+ mnh_ddr_pull_config(data);
+
+ mnh_ddr_send_lp_cmd(dev, LP_CMD_DSRPD);
+ dev_dbg(dev, "%s LP_STATE is 0x%x", __func__,
+ MNH_DDR_CTL_INf(121, LP_STATE));
+
+ /* Enable clock gating */
+ MNH_SCU_OUTf(CCU_CLK_CTL, HALT_LP4CG_EN, 0);
+ MNH_SCU_OUTf(CCU_CLK_CTL, HALT_LP4_PLL_BYPCLK_CG_EN, 0);
+ MNH_SCU_OUTf(CCU_CLK_CTL, LP4PHY_PLL_BYPASS_CLKEN, 0);
+ MNH_SCU_OUTf(CCU_CLK_CTL, LP4_REFCLKEN, 0);
+
+ udelay(1);
+ gpiod_set_value_cansleep(iso_n, 0);
+ udelay(1);
+
+ dev_dbg(dev, "%s done.", __func__);
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_ddr_suspend);
+
+int mnh_ddr_resume(struct mnh_ddr_data *data, struct gpio_desc *iso_n)
+{
+ struct device *dev = &data->pdev->dev;
+ struct mnh_ddr_internal_state *_state = &data->_state;
+
+ int index, fsp;
+ int timeout = 0;
+
+ mnh_ddr_init_clocks(data);
+
+ for (index = 0; index < MNH_DDR_NUM_CTL_REG; index++)
+ WRITE_DDR_REG_CONFIG(_state, ctl, index);
+
+ MNH_DDR_CTL_OUTf(23, DFIBUS_FREQ_INIT, SAVED_FSP(_state));
+ MNH_DDR_CTL_OUTf(23, DFIBUS_BOOT_FREQ, 0);
+
+ MNH_DDR_CTL_OUTf(23, PHY_INDEP_TRAIN_MODE, 0);
+ MNH_DDR_CTL_OUTf(23, CDNS_INTRL0, 1);
+
+ for (index = 0; index < MNH_DDR_NUM_PI_REG; index++)
+ WRITE_DDR_REG_CONFIG(_state, pi, index);
+
+ for (fsp = 0; fsp < MNH_DDR_NUM_FSPS; fsp++) {
+ MNH_DDR_PHY_OUTf(1025, PHY_FREQ_SEL_MULTICAST_EN, 0);
+ MNH_DDR_PHY_OUTf(1025, PHY_FREQ_SEL_INDEX, fsp);
+
+ for (index = 0; index < MNH_DDR_NUM_PHY_REG; index++) {
+ if (index != 1025)
+ WRITE_DDR_PHY_CONFIG(_state, fsp, index);
+ }
+ MNH_DDR_PHY_OUTf(1084, PHY_CAL_CLK_SELECT_0, 0x4);
+ }
+
+ MNH_DDR_CTL_OUTf(81, PWRUP_SREFRESH_EXIT, 1);
+
+ while ((timeout < 10) && (MNH_DDR_CTL_INf(450, MEM_RST_VALID) == 0)) {
+ udelay(1000);
+ timeout++;
+ }
+
+ if (timeout == 10)
+ dev_err(dev, "%s: failed to see reset valid\n", __func__);
+
+ udelay(1000);
+ /* extra delay after getting reset valid */
+ udelay(1000);
+ MNH_DDR_PHY_OUTf(1051, PHY_SET_DFI_INPUT_RST_PAD, 1);
+
+ gpiod_set_value_cansleep(iso_n, 1);
+ udelay(1000);
+
+ MNH_DDR_CTL_OUTf(00, START, 1);
+
+ dev_dbg(dev, "%s waiting for init done.", __func__);
+
+ timeout = 0;
+ while ((timeout < 1000) && (!mnh_ddr_int_status_bit(INIT_DONE_SBIT))) {
+ udelay(1);
+ timeout++;
+ }
+
+ if (!mnh_ddr_int_status_bit(INIT_DONE_SBIT)) {
+ dev_err(dev, "%s time out on init done %llx.\n",
+ __func__, mnh_ddr_int_status(dev));
+ return -ETIMEDOUT;
+ }
+
+ /* need to clear PWRUP_SREFRESH_EXIT to clear interrupt status bit 0 */
+ MNH_DDR_CTL_OUTf(81, PWRUP_SREFRESH_EXIT, 0);
+ dev_dbg(dev, "%s got init done %llx.\n", __func__,
+ mnh_ddr_int_status(dev));
+ mnh_ddr_clr_int_status(dev);
+ mnh_lpddr_freq_change(SAVED_FSP(_state));
+
+ dev_dbg(dev, "%s: tref 0x%04x 0x%04x 0x%04x 0x%04x\n",
+ __func__, MNH_DDR_CTL_INf(56, TREF_F0),
+ MNH_DDR_CTL_INf(57, TREF_F1), MNH_DDR_CTL_INf(58, TREF_F2),
+ MNH_DDR_CTL_INf(59, TREF_F3));
+
+ mnh_ddr_enable_lp();
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_ddr_resume);
+
+int mnh_ddr_po_init(struct mnh_ddr_data *data, struct gpio_desc *iso_n)
+{
+ struct device *dev = &data->pdev->dev;
+ struct mnh_ddr_internal_state *_state = &data->_state;
+
+ int index, setindex;
+ unsigned long timeout;
+ const struct mnh_ddr_reg_config *cfg = &mnh_ddr_33_100_400_600;
+
+ mnh_ddr_init_internal_state(_state, cfg);
+
+ dev_dbg(dev, "%s start.", __func__);
+
+ /* deassert iso_n */
+ gpiod_set_value_cansleep(iso_n, 1);
+
+ mnh_ddr_init_clocks(data);
+
+ for (index = 0; index < MNH_DDR_NUM_CTL_REG; index++)
+ WRITE_DDR_REG_CONFIG(_state, ctl, index);
+
+ /* Make sure DRAM will request refresh rate adjustments */
+ MNH_DDR_CTL_OUTf(164, MR13_DATA_0, 0xD0);
+
+ for (index = 0; index < MNH_DDR_NUM_PI_REG; index++)
+ WRITE_DDR_REG_CONFIG(_state, pi, index);
+
+ MNH_DDR_PHY_OUTf(1025, PHY_FREQ_SEL_MULTICAST_EN, 1);
+ MNH_DDR_PHY_OUTf(1025, PHY_FREQ_SEL_INDEX, 0);
+
+ for (index = 0; index < MNH_DDR_NUM_PHY_REG; index++)
+ WRITE_DDR_PHY_CONFIG(_state, 0, index);
+
+ MNH_DDR_PHY_OUTf(1025, PHY_FREQ_SEL_MULTICAST_EN, 0);
+ MNH_DDR_PHY_OUTf(1025, PHY_FREQ_SEL_INDEX, 1);
+
+ /* a */
+ setindex = 0;
+ while ((setindex < MNH_DDR_PHY_SET_SIZE) &&
+ (cfg->phy_setA[setindex][0] != 0xFFFFFFFF)) {
+ WRITE_SET_ELEMENT(_state,
+ cfg->phy_setA[setindex][0],
+ cfg->phy_setA[setindex][1]);
+ setindex++;
+ }
+
+ MNH_DDR_PHY_OUTf(1025, PHY_FREQ_SEL_INDEX, 2);
+
+ /* b */
+ setindex = 0;
+ while ((setindex < MNH_DDR_PHY_SET_SIZE) &&
+ (cfg->phy_setB[setindex][0] != 0xFFFFFFFF)) {
+ WRITE_SET_ELEMENT(_state,
+ cfg->phy_setB[setindex][0],
+ cfg->phy_setB[setindex][1]);
+ setindex++;
+ }
+
+ dev_dbg(dev, "%s begin training,", __func__);
+ MNH_DDR_PI_OUTf(00, PI_START, 1);
+ MNH_DDR_CTL_OUTf(00, START, 1);
+
+ timeout = jiffies + TRAINING_TIMEOUT;
+ while (time_before(jiffies, timeout) &&
+ (!mnh_ddr_int_status_bit(INIT_DONE_SBIT)))
+ usleep_range(100, 200);
+
+ if (!mnh_ddr_int_status_bit(INIT_DONE_SBIT)) {
+ dev_err(dev, "%s timed out on init done.\n", __func__);
+ return -ETIMEDOUT;
+ }
+
+ dev_dbg(dev, "%s got init done %llx.\n", __func__,
+ mnh_ddr_int_status(dev));
+
+ mnh_ddr_clr_int_status(dev);
+ MNH_DDR_CTL_OUTf(165, MR_FSP_DATA_VALID_F0_0, 1);
+ MNH_DDR_CTL_OUTf(165, MR_FSP_DATA_VALID_F1_0, 1);
+ MNH_DDR_CTL_OUTf(165, MR_FSP_DATA_VALID_F2_0, 1);
+ MNH_DDR_CTL_OUTf(166, MR_FSP_DATA_VALID_F3_0, 1);
+
+ dev_dbg(dev, "%s: tref 0x%04x 0x%04x 0x%04x 0x%04x\n",
+ __func__, MNH_DDR_CTL_INf(56, TREF_F0),
+ MNH_DDR_CTL_INf(57, TREF_F1), MNH_DDR_CTL_INf(58, TREF_F2),
+ MNH_DDR_CTL_INf(59, TREF_F3));
+
+ mnh_ddr_enable_lp();
+
+ /* Enable FSP3 => 2400 */
+ mnh_lpddr_freq_change(LPDDR_FREQ_FSP3);
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_ddr_po_init);
+
+u32 mnh_ddr_mbist(struct mnh_ddr_data *data, enum mnh_ddr_bist_type bist_type)
+{
+ struct device *dev = &data->pdev->dev;
+
+ u32 result = 0;
+ u32 timeout = 1000000;
+ const u32 pattern[] = {
+ 0x55555555, 0x33333333, 0x0f0f0f0f, 0x00ff00ff };
+
+ u32 old_in_order_accept;
+
+ switch (bist_type) {
+ case MOVI1_3N:
+ case LIMITED_MOVI1_3N:
+ dev_info(dev, "%s: type %d\n", __func__, bist_type);
+ break;
+ default:
+ dev_err(dev, "%s: type %d is not supported\n",
+ __func__, bist_type);
+ return 0;
+ }
+
+ mnh_ddr_disable_lp(dev);
+
+ old_in_order_accept = MNH_DDR_CTL_INf(223, IN_ORDER_ACCEPT);
+ MNH_DDR_CTL_OUTf(223, IN_ORDER_ACCEPT, 1);
+
+ /* 512MB */
+ MNH_DDR_CTL_OUTf(171, ADDR_SPACE, 0x1D);
+ MNH_DDR_CTL_OUTf(173, BIST_START_ADDRESS, 0);
+
+ MNH_DDR_CTL_OUTf(171, BIST_DATA_CHECK, 1);
+ MNH_DDR_CTL_OUTf(172, BIST_ADDR_CHECK, 0);
+
+ /* if limited movi, use these patterns */
+ MNH_DDR_CTL_OUTf(178, BIST_DATA_PATTERN, pattern[0]);
+ MNH_DDR_CTL_OUTf(179, BIST_DATA_PATTERN, pattern[1]);
+ MNH_DDR_CTL_OUTf(180, BIST_DATA_PATTERN, pattern[2]);
+ MNH_DDR_CTL_OUTf(181, BIST_DATA_PATTERN, pattern[3]);
+
+ MNH_DDR_CTL_OUTf(175, BIST_DATA_MASK, 0);
+
+ MNH_DDR_CTL_OUTf(177, BIST_TEST_MODE, bist_type);
+
+ MNH_DDR_CTL_OUTf(171, BIST_GO, 1);
+ dev_info(dev, "%s: waiting to finish BIST\n", __func__);
+
+ while (!mnh_ddr_int_status_bit(BIST_SBIT) && --timeout)
+ msleep(20);
+
+ if (!mnh_ddr_int_status_bit(BIST_SBIT)) {
+ dev_err(dev, "%s: BIST timedout: %llx\n",
+ __func__, mnh_ddr_int_status(dev));
+ } else {
+ result = MNH_DDR_CTL_INf(171, BIST_RESULT);
+ dev_info(dev, "%s: result 0x%02x\n", __func__, result);
+ }
+
+ MNH_DDR_CTL_OUTf(171, BIST_GO, 0);
+ mnh_ddr_clr_int_status(dev);
+
+ MNH_DDR_CTL_OUTf(223, IN_ORDER_ACCEPT,
+ old_in_order_accept);
+
+ mnh_ddr_enable_lp();
+ return result;
+}
+EXPORT_SYMBOL(mnh_ddr_mbist);
+
+int mnh_ddr_platform_init(struct platform_device *pdev,
+ struct mnh_ddr_data *data)
+{
+ data->pdev = pdev;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_ddr_platform_init);
diff --git a/drivers/misc/mnh/mnh-ddr.h b/drivers/misc/mnh/mnh-ddr.h
new file mode 100644
index 0000000..372ca55
--- /dev/null
+++ b/drivers/misc/mnh/mnh-ddr.h
@@ -0,0 +1,76 @@
+/*
+*
+* MNH DDR Driver
+* Copyright (c) 2016-2017, Intel Corporation.
+*
+* This program is free software; you can redistribute it and/or modify it
+* under the terms and conditions of the GNU General Public License,
+* version 2, as published by the Free Software Foundation.
+*
+* This program is distributed in the hope it will be useful, but WITHOUT
+* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+* more details.
+*
+*/
+#ifndef __MNH_DDR_H__
+#define __MNH_DDR_H__
+
+#include <linux/types.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/gpio/consumer.h>
+#include <linux/platform_device.h>
+
+#define MNH_DDR_NUM_CTL_REG (558 + 1)
+#define MNH_DDR_NUM_PHY_REG (1100 + 1)
+#define MNH_DDR_NUM_PI_REG (191 + 1)
+
+#define MNH_DDR_NUM_FSPS (4)
+#define MNH_DDR_NUM_BASES (3)
+
+/* arbitrary but sufficient size for phy deltas */
+#define MNH_DDR_PHY_SET_SIZE 32
+/* need reg index for setA/B */
+#define MNH_DDR_PHY_SET_ELEMS 2
+struct mnh_ddr_reg_config {
+ u32 fsps[MNH_DDR_NUM_FSPS];
+ u32 ctl[MNH_DDR_NUM_CTL_REG];
+ u32 pi[MNH_DDR_NUM_PI_REG];
+ u32 phy[MNH_DDR_NUM_PHY_REG];
+ u32 phy_setA[MNH_DDR_PHY_SET_SIZE][MNH_DDR_PHY_SET_ELEMS];
+ u32 phy_setB[MNH_DDR_PHY_SET_SIZE][MNH_DDR_PHY_SET_ELEMS];
+};
+
+struct mnh_ddr_internal_state {
+ u32 ctl_base;
+ u32 ctl[MNH_DDR_NUM_CTL_REG];
+ u32 pi_base;
+ u32 pi[MNH_DDR_NUM_PI_REG];
+ u32 phy_base;
+ u32 phy[MNH_DDR_NUM_FSPS][MNH_DDR_NUM_PHY_REG];
+ u32 fsps[MNH_DDR_NUM_FSPS];
+ u32 suspend_fsp;
+ u32 tref[MNH_DDR_NUM_FSPS];
+};
+
+enum mnh_ddr_bist_type {
+ MOVI1_3N = 0,
+ LIMITED_MOVI1_3N,
+};
+
+struct mnh_ddr_data {
+ struct platform_device *pdev;
+ struct mnh_ddr_internal_state _state;
+};
+
+int mnh_ddr_platform_init(struct platform_device *pdev,
+ struct mnh_ddr_data *data);
+int mnh_ddr_po_init(struct mnh_ddr_data *data, struct gpio_desc *iso_n);
+int mnh_ddr_resume(struct mnh_ddr_data *data, struct gpio_desc *iso_n);
+int mnh_ddr_suspend(struct mnh_ddr_data *data, struct gpio_desc *iso_n);
+int mnh_ddr_clr_int_status(struct device *dev);
+u64 mnh_ddr_int_status(struct device *dev);
+u32 mnh_ddr_mbist(struct mnh_ddr_data *data, enum mnh_ddr_bist_type bist_type);
+
+#endif /* __MNH_DDR_H__ */
diff --git a/drivers/misc/mnh/mnh-efuse.c b/drivers/misc/mnh/mnh-efuse.c
new file mode 100644
index 0000000..298bacd
--- /dev/null
+++ b/drivers/misc/mnh/mnh-efuse.c
@@ -0,0 +1,111 @@
+/*
+ *
+ * MNH eFuse Driver
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#include <linux/delay.h>
+#include "mnh-hwio.h"
+#include "mnh-hwio-bases.h"
+#include "mnh-hwio-scu.h"
+#include "mnh-efuse.h"
+
+static uint32_t initialized;
+static uint32_t rows[MNH_EFUSE_ROWS] = { 0, 0, 0, 0, 0, 0, 0, 0 };
+
+static const uintptr_t scu_base = HWIO_SCU_BASE_ADDR;
+#define SCU_IN(reg) HW_IN(scu_base, SCU, reg)
+#define SCU_INf(reg, fld) HW_INf(scu_base, SCU, reg, fld)
+#define SCU_OUT(reg, val) HW_OUT(scu_base, SCU, reg, val)
+#define SCU_OUTf(reg, fld, val) HW_OUTf(scu_base, SCU, reg, fld, val)
+
+#define EFUSE_IN(reg) HW_IN(scu_base, SCU, EFUSE_##reg)
+#define EFUSE_INf(reg, fld) HW_INf(scu_base, SCU, EFUSE_##reg, fld)
+#define EFUSE_OUTf(reg, fld, val) HW_OUTf(scu_base, SCU, EFUSE_##reg, fld, val)
+
+#define MNH_EFUSE_T_SUR_PD 800
+#define MNH_EFUSE_T_RD 100
+#define MNH_EFUSE_T_HR_PS_CS 50
+
+static void mnh_efuse_read_raw(void)
+{
+#ifdef MNH_FULL_EMULATION
+ /* populate dummy fuse data */
+ rows[0] = 0x12345678;
+ rows[1] = 0x12345678;
+ rows[2] = 0x9abcdef0;
+ rows[3] = 0x9abcdef0;
+ rows[4] = 0x11223344;
+ rows[5] = 0x11223344;
+ rows[6] = 0x55667788;
+ rows[7] = 0x55667788;
+#else
+ uint32_t row = 0;
+
+ EFUSE_OUTf(PASSCODE, PASSCODE, 0x9DC4);
+ EFUSE_OUTf(CTRL, PD, 0);
+ /*
+ * PS should always be low except for write/programming mode
+ * according to efuse datasheet
+ */
+ EFUSE_OUTf(CTRL, PS, 0);
+ ndelay(MNH_EFUSE_T_SUR_PD);
+ /* looking at the efuse datasheet, I reordered these from HAS */
+ EFUSE_OUTf(ACC, CSB, 0);
+ EFUSE_OUTf(CTRL, PGENB, 1);
+ EFUSE_OUTf(CTRL, LOAD, 1);
+
+ for (row = 0; row < MNH_EFUSE_ROWS; row++) {
+ EFUSE_OUTf(ACC, ZEROED, 0);
+ EFUSE_OUTf(ACC, ROW, row);
+ EFUSE_OUTf(ACC, STROBE, 1);
+ ndelay(MNH_EFUSE_T_RD);
+ EFUSE_OUTf(ACC, STROBE, 0);
+ ndelay(MNH_EFUSE_T_RD/2);
+ rows[row] = EFUSE_IN(READ_DATA);
+ }
+
+ EFUSE_OUTf(ACC, CSB, 1);
+ ndelay(MNH_EFUSE_T_HR_PS_CS);
+
+ EFUSE_OUTf(CTRL, PD, 1);
+ EFUSE_OUTf(PASSCODE, PASSCODE, 0);
+#endif /* MNH_FULL_EMULATION */
+ initialized = 1;
+}
+
+uint32_t mnh_efuse_read(const struct mnh_efuse_addr *addr)
+{
+ uint32_t high, low, row1, row2, mask, value = 0;
+
+ if (!initialized)
+ mnh_efuse_read_raw();
+
+ if (addr) {
+ high = addr->bit_high;
+ low = addr->bit_low;
+ row1 = addr->row1;
+ row2 = addr->row2;
+
+ if ((high < MNH_EFUSE_ROW_WIDTH) &&
+ (low <= high) &&
+ (row1 < MNH_EFUSE_ROWS) &&
+ (row2 < MNH_EFUSE_ROWS)) {
+ mask = (1 << ((high - low) + 1)) - 1;
+ value = ((rows[row1] | rows[row2]) >> low) & mask;
+ } else {
+ pr_err("%s: address is malformed!\n", __func__);
+ }
+ }
+ return value;
+}
diff --git a/drivers/misc/mnh/mnh-efuse.h b/drivers/misc/mnh/mnh-efuse.h
new file mode 100644
index 0000000..6ece0ac
--- /dev/null
+++ b/drivers/misc/mnh/mnh-efuse.h
@@ -0,0 +1,34 @@
+/*
+ *
+ * MNH eFuse Driver
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef __MNH_EFUSE_H__
+#define __MNH_EFUSE_H__
+
+
+#define MNH_EFUSE_ROWS 8
+#define MNH_EFUSE_ROW_WIDTH 32
+
+struct mnh_efuse_addr {
+ uint32_t row1;
+ uint32_t row2;
+ /* zero based, inclusive */
+ uint32_t bit_high;
+ uint32_t bit_low;
+};
+
+uint32_t mnh_efuse_read(const struct mnh_efuse_addr *addr);
+
+#endif /* __MNH_EFUSE_H__ */
diff --git a/drivers/misc/mnh/mnh-hwio-bases.h b/drivers/misc/mnh/mnh-hwio-bases.h
new file mode 100644
index 0000000..e654d3d
--- /dev/null
+++ b/drivers/misc/mnh/mnh-hwio-bases.h
@@ -0,0 +1,74 @@
+/*
+ * Copyright (c) 2016, Intel Corporation. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this
+ * list of conditions and the following disclaimer.
+ *
+ * Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * Neither the name of Intel nor the names of its contributors may be used
+ * to endorse or promote products derived from this software without specific
+ * prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+
+#ifndef __MNH_HWIO_BASES_H_
+#define __MNH_HWIO_BASES_H_
+
+#define MNH_BAD_ADDR (0xFFFFFFFF)
+
+/* silicon addresses */
+#define HWIO_ROM_BASE_ADDR (0x00000000)
+#define HWIO_SRAM_BASE_ADDR (0x00100000)
+#define HWIO_PCIE_EP_BASE_ADDR (0x00200000)
+#define HWIO_SCU_BASE_ADDR (0x04003000)
+#define HWIO_CPU_BASE_ADDR (0x04004000)
+
+#define HWIO_DDR_CTL_BASE_ADDR (0x04008000)
+#define HWIO_DDR_PI_BASE_ADDR (0x04009000)
+#define HWIO_DDR_PHY_BASE_ADDR (0x0400A000)
+
+#define HWIO_WDT_BASE_ADDR (0x04040000)
+#define HWIO_TIMER_BASE_ADDR (0x04041000)
+#define HWIO_SPIS_BASE_ADDR (0x04042000)
+#define HWIO_SPIM_BASE_ADDR (0x04043000)
+#define HWIO_GPIO_BASE_ADDR (0x04044000)
+#define HWIO_I2C_BASE_ADDR(blki) (0x04045000 + (blki * 0x1000))
+#define HWIO_UART_BASE_ADDR(blki) (0x04049000 + (blki * 0x1000))
+#define HWIO_PMON_BASE_ADDR (0x0404C000)
+#define HWIO_AXI_BASE_ADDR (0x0404D000)
+
+#define HWIO_PCIE_PHY_BASE_ADDR (0x04080000)
+#define HWIO_PCIE_SS_BASE_ADDR (0x040C0000)
+#define HWIO_MIPI_TX_BASE_ADDR(blki) (0x04010000 + (blki * 0x1000))
+#define HWIO_MIPI_RX_BASE_ADDR(blki) (0x04012000 + (blki * 0x1000))
+#define HWIO_MIPI_TOP_BASE_ADDR (0x04015000)
+
+
+#define HWIO_PCIE_EP_TYPE0_HDR_BASE_ADDR (HWIO_PCIE_EP_BASE_ADDR)
+#define HWIO_PCIE_EP_SPCIE_CAP_BASE_ADDR (HWIO_PCIE_EP_BASE_ADDR + 0x148)
+#define HWIO_PCIE_EP_L1SUB_CAP_BASE_ADDR (HWIO_PCIE_EP_BASE_ADDR + 0x170)
+#define HWIO_PCIE_EP_PM_CAP_BASE_ADDR (HWIO_PCIE_EP_BASE_ADDR + 0x40)
+#define HWIO_PCIE_EP_PCIE_CAP_BASE_ADDR (HWIO_PCIE_EP_BASE_ADDR + 0x70)
+#define HWIO_PCIE_EP_AER_CAP_BASE_ADDR (HWIO_PCIE_EP_BASE_ADDR + 0x100)
+#define HWIO_PCIE_EP_MSI_CAP_BASE_ADDR (HWIO_PCIE_EP_BASE_ADDR + 0x50)
+#define HWIO_PCIE_EP_LTR_CAP_BASE_ADDR (HWIO_PCIE_EP_BASE_ADDR + 0x168)
+#define HWIO_PCIE_EP_PORT_LOGIC_BASE_ADDR (HWIO_PCIE_EP_BASE_ADDR + 0x700)
+
+#endif /* __MNH_HWIO_BASES_H_ */
diff --git a/drivers/misc/mnh/mnh-hwio-ddr-ctl.h b/drivers/misc/mnh/mnh-hwio-ddr-ctl.h
new file mode 100644
index 0000000..7ffefc5
--- /dev/null
+++ b/drivers/misc/mnh/mnh-hwio-ddr-ctl.h
@@ -0,0 +1,5887 @@
+/* auto generated: Friday, August 26th, 2016 11:42:15am */
+/*
+ * Copyright (c) 2016, Intel Corporation. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this
+ * list of conditions and the following disclaimer.
+ *
+ * Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * Neither the name of Intel nor the names of its contributors may be used
+ * to endorse or promote products derived from this software without specific
+ * prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef __MNH_HWIO_DDR_CTL_
+#define __MNH_HWIO_DDR_CTL_
+
+#define HWIO_DDR_CTL_00_REGOFF 0x0
+#define HWIO_DDR_CTL_00_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_00_REGOFF)
+#define HWIO_DDR_CTL_00_START_FLDMASK (0x1)
+#define HWIO_DDR_CTL_00_START_FLDSHFT (0)
+#define HWIO_DDR_CTL_00_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_00_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_00_DRAM_CLASS_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_00_DRAM_CLASS_FLDSHFT (8)
+#define HWIO_DDR_CTL_00_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_00_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_00_VERSION_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_00_VERSION_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_01_REGOFF 0x4
+#define HWIO_DDR_CTL_01_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_01_REGOFF)
+#define HWIO_DDR_CTL_01_MAX_ROW_REG_FLDMASK (0xf)
+#define HWIO_DDR_CTL_01_MAX_ROW_REG_FLDSHFT (0)
+#define HWIO_DDR_CTL_01_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_01_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_01_MAX_COL_REG_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_01_MAX_COL_REG_FLDSHFT (8)
+#define HWIO_DDR_CTL_01_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_01_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_01_MAX_CS_REG_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_01_MAX_CS_REG_FLDSHFT (16)
+#define HWIO_DDR_CTL_01_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_01_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_01_READ_DATA_FIFO_DEPTH_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_01_READ_DATA_FIFO_DEPTH_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_02_REGOFF 0x8
+#define HWIO_DDR_CTL_02_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_02_REGOFF)
+#define HWIO_DDR_CTL_02_READ_DATA_FIFO_PTR_WIDTH_FLDMASK (0xff)
+#define HWIO_DDR_CTL_02_READ_DATA_FIFO_PTR_WIDTH_FLDSHFT (0)
+#define HWIO_DDR_CTL_02_WRITE_DATA_FIFO_DEPTH_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_02_WRITE_DATA_FIFO_DEPTH_FLDSHFT (8)
+#define HWIO_DDR_CTL_02_WRITE_DATA_FIFO_PTR_WIDTH_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_02_WRITE_DATA_FIFO_PTR_WIDTH_FLDSHFT (16)
+#define HWIO_DDR_CTL_02_MEMCD_RMODW_FIFO_DEPTH_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_02_MEMCD_RMODW_FIFO_DEPTH_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_03_REGOFF 0xc
+#define HWIO_DDR_CTL_03_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_03_REGOFF)
+#define HWIO_DDR_CTL_03_MEMCD_RMODW_FIFO_PTR_WIDTH_FLDMASK (0xff)
+#define HWIO_DDR_CTL_03_MEMCD_RMODW_FIFO_PTR_WIDTH_FLDSHFT (0)
+#define HWIO_DDR_CTL_03_ASYNC_CDC_STAGES_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_03_ASYNC_CDC_STAGES_FLDSHFT (8)
+#define HWIO_DDR_CTL_03_AXI0_CMDFIFO_LOG2_DEPTH_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_03_AXI0_CMDFIFO_LOG2_DEPTH_FLDSHFT (16)
+#define HWIO_DDR_CTL_03_AXI0_RDFIFO_LOG2_DEPTH_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_03_AXI0_RDFIFO_LOG2_DEPTH_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_04_REGOFF 0x10
+#define HWIO_DDR_CTL_04_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_04_REGOFF)
+#define HWIO_DDR_CTL_04_OBSOLETE0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_04_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_04_AXI0_WRCMD_PROC_FIFO_LOG2_DEPTH_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_04_AXI0_WRCMD_PROC_FIFO_LOG2_DEPTH_FLDSHFT (8)
+#define HWIO_DDR_CTL_04_AXI1_CMDFIFO_LOG2_DEPTH_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_04_AXI1_CMDFIFO_LOG2_DEPTH_FLDSHFT (16)
+#define HWIO_DDR_CTL_04_AXI1_RDFIFO_LOG2_DEPTH_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_04_AXI1_RDFIFO_LOG2_DEPTH_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_05_REGOFF 0x14
+#define HWIO_DDR_CTL_05_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_05_REGOFF)
+#define HWIO_DDR_CTL_05_OBSOLETE0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_05_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_05_AXI1_WRCMD_PROC_FIFO_LOG2_DEPTH_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_05_AXI1_WRCMD_PROC_FIFO_LOG2_DEPTH_FLDSHFT (8)
+#define HWIO_DDR_CTL_05_DFS_CLOSE_BANKS_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_05_DFS_CLOSE_BANKS_FLDSHFT (16)
+#define HWIO_DDR_CTL_05_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_05_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_CTL_05_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_05_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_06_REGOFF 0x18
+#define HWIO_DDR_CTL_06_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_06_REGOFF)
+#define HWIO_DDR_CTL_06_TINIT_F0_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_06_TINIT_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_06_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_06_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_07_REGOFF 0x1c
+#define HWIO_DDR_CTL_07_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_07_REGOFF)
+#define HWIO_DDR_CTL_07_TINIT3_F0_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_07_TINIT3_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_07_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_07_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_08_REGOFF 0x20
+#define HWIO_DDR_CTL_08_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_08_REGOFF)
+#define HWIO_DDR_CTL_08_TINIT4_F0_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_08_TINIT4_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_08_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_08_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_09_REGOFF 0x24
+#define HWIO_DDR_CTL_09_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_09_REGOFF)
+#define HWIO_DDR_CTL_09_TINIT5_F0_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_09_TINIT5_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_09_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_09_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_10_REGOFF 0x28
+#define HWIO_DDR_CTL_10_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_10_REGOFF)
+#define HWIO_DDR_CTL_10_TINIT_F1_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_10_TINIT_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_10_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_10_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_11_REGOFF 0x2c
+#define HWIO_DDR_CTL_11_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_11_REGOFF)
+#define HWIO_DDR_CTL_11_TINIT3_F1_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_11_TINIT3_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_11_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_11_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_12_REGOFF 0x30
+#define HWIO_DDR_CTL_12_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_12_REGOFF)
+#define HWIO_DDR_CTL_12_TINIT4_F1_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_12_TINIT4_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_12_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_12_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_13_REGOFF 0x34
+#define HWIO_DDR_CTL_13_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_13_REGOFF)
+#define HWIO_DDR_CTL_13_TINIT5_F1_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_13_TINIT5_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_13_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_13_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_14_REGOFF 0x38
+#define HWIO_DDR_CTL_14_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_14_REGOFF)
+#define HWIO_DDR_CTL_14_TINIT_F2_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_14_TINIT_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_14_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_14_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_15_REGOFF 0x3c
+#define HWIO_DDR_CTL_15_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_15_REGOFF)
+#define HWIO_DDR_CTL_15_TINIT3_F2_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_15_TINIT3_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_15_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_15_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_16_REGOFF 0x40
+#define HWIO_DDR_CTL_16_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_16_REGOFF)
+#define HWIO_DDR_CTL_16_TINIT4_F2_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_16_TINIT4_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_16_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_16_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_17_REGOFF 0x44
+#define HWIO_DDR_CTL_17_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_17_REGOFF)
+#define HWIO_DDR_CTL_17_TINIT5_F2_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_17_TINIT5_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_17_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_17_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_18_REGOFF 0x48
+#define HWIO_DDR_CTL_18_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_18_REGOFF)
+#define HWIO_DDR_CTL_18_TINIT_F3_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_18_TINIT_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_18_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_18_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_19_REGOFF 0x4c
+#define HWIO_DDR_CTL_19_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_19_REGOFF)
+#define HWIO_DDR_CTL_19_TINIT3_F3_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_19_TINIT3_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_19_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_19_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_20_REGOFF 0x50
+#define HWIO_DDR_CTL_20_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_20_REGOFF)
+#define HWIO_DDR_CTL_20_TINIT4_F3_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_20_TINIT4_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_20_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_20_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_21_REGOFF 0x54
+#define HWIO_DDR_CTL_21_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_21_REGOFF)
+#define HWIO_DDR_CTL_21_TINIT5_F3_FLDMASK (0xffffff)
+#define HWIO_DDR_CTL_21_TINIT5_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_21_NO_AUTO_MRR_INIT_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_21_NO_AUTO_MRR_INIT_FLDSHFT (24)
+#define HWIO_DDR_CTL_21_RESERVED_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_21_RESERVED_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_22_REGOFF 0x58
+#define HWIO_DDR_CTL_22_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_22_REGOFF)
+#define HWIO_DDR_CTL_22_MRR_ERROR_STATUS_FLDMASK (0x3)
+#define HWIO_DDR_CTL_22_MRR_ERROR_STATUS_FLDSHFT (0)
+#define HWIO_DDR_CTL_22_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_CTL_22_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_CTL_22_DFI_INV_DATA_CS_FLDMASK (0x100)
+#define HWIO_DDR_CTL_22_DFI_INV_DATA_CS_FLDSHFT (8)
+#define HWIO_DDR_CTL_22_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_22_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_22_NO_MRW_BT_INIT_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_22_NO_MRW_BT_INIT_FLDSHFT (16)
+#define HWIO_DDR_CTL_22_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_22_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_22_NO_MRW_INIT_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_22_NO_MRW_INIT_FLDSHFT (24)
+#define HWIO_DDR_CTL_22_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_22_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_23_REGOFF 0x5c
+#define HWIO_DDR_CTL_23_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_23_REGOFF)
+#define HWIO_DDR_CTL_23_CDNS_INTRL0_FLDMASK (0x1)
+#define HWIO_DDR_CTL_23_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_23_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_23_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_23_PHY_INDEP_TRAIN_MODE_FLDMASK (0x100)
+#define HWIO_DDR_CTL_23_PHY_INDEP_TRAIN_MODE_FLDSHFT (8)
+#define HWIO_DDR_CTL_23_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_23_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_23_DFIBUS_FREQ_INIT_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_23_DFIBUS_FREQ_INIT_FLDSHFT (16)
+#define HWIO_DDR_CTL_23_CDNS_INTRL2_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_23_CDNS_INTRL2_FLDSHFT (18)
+#define HWIO_DDR_CTL_23_DFIBUS_BOOT_FREQ_FLDMASK (0x3000000)
+#define HWIO_DDR_CTL_23_DFIBUS_BOOT_FREQ_FLDSHFT (24)
+#define HWIO_DDR_CTL_23_CDNS_INTRL3_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_23_CDNS_INTRL3_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_24_REGOFF 0x60
+#define HWIO_DDR_CTL_24_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_24_REGOFF)
+#define HWIO_DDR_CTL_24_DFIBUS_FREQ_F0_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_24_DFIBUS_FREQ_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_24_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_24_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_24_DFIBUS_FREQ_F1_FLDMASK (0x1f00)
+#define HWIO_DDR_CTL_24_DFIBUS_FREQ_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_24_CDNS_INTRL1_FLDMASK (0xe000)
+#define HWIO_DDR_CTL_24_CDNS_INTRL1_FLDSHFT (13)
+#define HWIO_DDR_CTL_24_DFIBUS_FREQ_F2_FLDMASK (0x1f0000)
+#define HWIO_DDR_CTL_24_DFIBUS_FREQ_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_24_CDNS_INTRL2_FLDMASK (0xe00000)
+#define HWIO_DDR_CTL_24_CDNS_INTRL2_FLDSHFT (21)
+#define HWIO_DDR_CTL_24_DFIBUS_FREQ_F3_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_24_DFIBUS_FREQ_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_24_CDNS_INTRL3_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_24_CDNS_INTRL3_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_25_REGOFF 0x64
+#define HWIO_DDR_CTL_25_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_25_REGOFF)
+#define HWIO_DDR_CTL_25_TRST_PWRON_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_25_TRST_PWRON_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_26_REGOFF 0x68
+#define HWIO_DDR_CTL_26_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_26_REGOFF)
+#define HWIO_DDR_CTL_26_CKE_INACTIVE_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_26_CKE_INACTIVE_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_27_REGOFF 0x6c
+#define HWIO_DDR_CTL_27_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_27_REGOFF)
+#define HWIO_DDR_CTL_27_CASLAT_LIN_F0_FLDMASK (0x7f)
+#define HWIO_DDR_CTL_27_CASLAT_LIN_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_27_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_CTL_27_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_CTL_27_WRLAT_F0_FLDMASK (0x7f00)
+#define HWIO_DDR_CTL_27_WRLAT_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_27_CDNS_INTRL1_FLDMASK (0x8000)
+#define HWIO_DDR_CTL_27_CDNS_INTRL1_FLDSHFT (15)
+#define HWIO_DDR_CTL_27_CASLAT_LIN_F1_FLDMASK (0x7f0000)
+#define HWIO_DDR_CTL_27_CASLAT_LIN_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_27_CDNS_INTRL2_FLDMASK (0x800000)
+#define HWIO_DDR_CTL_27_CDNS_INTRL2_FLDSHFT (23)
+#define HWIO_DDR_CTL_27_WRLAT_F1_FLDMASK (0x7f000000)
+#define HWIO_DDR_CTL_27_WRLAT_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_27_CDNS_INTRL3_FLDMASK (0x80000000)
+#define HWIO_DDR_CTL_27_CDNS_INTRL3_FLDSHFT (31)
+
+#define HWIO_DDR_CTL_28_REGOFF 0x70
+#define HWIO_DDR_CTL_28_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_28_REGOFF)
+#define HWIO_DDR_CTL_28_CASLAT_LIN_F2_FLDMASK (0x7f)
+#define HWIO_DDR_CTL_28_CASLAT_LIN_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_28_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_CTL_28_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_CTL_28_WRLAT_F2_FLDMASK (0x7f00)
+#define HWIO_DDR_CTL_28_WRLAT_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_28_CDNS_INTRL1_FLDMASK (0x8000)
+#define HWIO_DDR_CTL_28_CDNS_INTRL1_FLDSHFT (15)
+#define HWIO_DDR_CTL_28_CASLAT_LIN_F3_FLDMASK (0x7f0000)
+#define HWIO_DDR_CTL_28_CASLAT_LIN_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_28_CDNS_INTRL2_FLDMASK (0x800000)
+#define HWIO_DDR_CTL_28_CDNS_INTRL2_FLDSHFT (23)
+#define HWIO_DDR_CTL_28_WRLAT_F3_FLDMASK (0x7f000000)
+#define HWIO_DDR_CTL_28_WRLAT_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_28_CDNS_INTRL3_FLDMASK (0x80000000)
+#define HWIO_DDR_CTL_28_CDNS_INTRL3_FLDSHFT (31)
+
+#define HWIO_DDR_CTL_29_REGOFF 0x74
+#define HWIO_DDR_CTL_29_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_29_REGOFF)
+#define HWIO_DDR_CTL_29_TBST_INT_INTERVAL_FLDMASK (0x7)
+#define HWIO_DDR_CTL_29_TBST_INT_INTERVAL_FLDSHFT (0)
+#define HWIO_DDR_CTL_29_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_29_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_29_TCCD_FLDMASK (0x1f00)
+#define HWIO_DDR_CTL_29_TCCD_FLDSHFT (8)
+#define HWIO_DDR_CTL_29_CDNS_INTRL1_FLDMASK (0xe000)
+#define HWIO_DDR_CTL_29_CDNS_INTRL1_FLDSHFT (13)
+#define HWIO_DDR_CTL_29_TRRD_F0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_29_TRRD_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_29_TRC_F0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_29_TRC_F0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_30_REGOFF 0x78
+#define HWIO_DDR_CTL_30_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_30_REGOFF)
+#define HWIO_DDR_CTL_30_TRAS_MIN_F0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_30_TRAS_MIN_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_30_TWTR_F0_FLDMASK (0x3f00)
+#define HWIO_DDR_CTL_30_TWTR_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_30_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_CTL_30_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_CTL_30_TRP_F0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_30_TRP_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_30_TFAW_F0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_30_TFAW_F0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_31_REGOFF 0x7c
+#define HWIO_DDR_CTL_31_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_31_REGOFF)
+#define HWIO_DDR_CTL_31_TRRD_F1_FLDMASK (0xff)
+#define HWIO_DDR_CTL_31_TRRD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_31_TRC_F1_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_31_TRC_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_31_TRAS_MIN_F1_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_31_TRAS_MIN_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_31_TWTR_F1_FLDMASK (0x3f000000)
+#define HWIO_DDR_CTL_31_TWTR_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_31_RESERVED_FLDMASK (0xc0000000)
+#define HWIO_DDR_CTL_31_RESERVED_FLDSHFT (30)
+
+#define HWIO_DDR_CTL_32_REGOFF 0x80
+#define HWIO_DDR_CTL_32_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_32_REGOFF)
+#define HWIO_DDR_CTL_32_TRP_F1_FLDMASK (0xff)
+#define HWIO_DDR_CTL_32_TRP_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_32_TFAW_F1_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_32_TFAW_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_32_TRRD_F2_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_32_TRRD_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_32_TRC_F2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_32_TRC_F2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_33_REGOFF 0x84
+#define HWIO_DDR_CTL_33_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_33_REGOFF)
+#define HWIO_DDR_CTL_33_TRAS_MIN_F2_FLDMASK (0xff)
+#define HWIO_DDR_CTL_33_TRAS_MIN_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_33_TWTR_F2_FLDMASK (0x3f00)
+#define HWIO_DDR_CTL_33_TWTR_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_33_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_CTL_33_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_CTL_33_TRP_F2_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_33_TRP_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_33_TFAW_F2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_33_TFAW_F2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_34_REGOFF 0x88
+#define HWIO_DDR_CTL_34_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_34_REGOFF)
+#define HWIO_DDR_CTL_34_TRRD_F3_FLDMASK (0xff)
+#define HWIO_DDR_CTL_34_TRRD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_34_TRC_F3_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_34_TRC_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_34_TRAS_MIN_F3_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_34_TRAS_MIN_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_34_TWTR_F3_FLDMASK (0x3f000000)
+#define HWIO_DDR_CTL_34_TWTR_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_34_RESERVED_FLDMASK (0xc0000000)
+#define HWIO_DDR_CTL_34_RESERVED_FLDSHFT (30)
+
+#define HWIO_DDR_CTL_35_REGOFF 0x8c
+#define HWIO_DDR_CTL_35_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_35_REGOFF)
+#define HWIO_DDR_CTL_35_TRP_F3_FLDMASK (0xff)
+#define HWIO_DDR_CTL_35_TRP_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_35_TFAW_F3_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_35_TFAW_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_35_TCCDMW_FLDMASK (0x3f0000)
+#define HWIO_DDR_CTL_35_TCCDMW_FLDSHFT (16)
+#define HWIO_DDR_CTL_35_RESERVED_FLDMASK (0xc00000)
+#define HWIO_DDR_CTL_35_RESERVED_FLDSHFT (22)
+#define HWIO_DDR_CTL_35_TRTP_F0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_35_TRTP_F0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_36_REGOFF 0x90
+#define HWIO_DDR_CTL_36_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_36_REGOFF)
+#define HWIO_DDR_CTL_36_TMRD_F0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_36_TMRD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_36_TMOD_F0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_36_TMOD_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_36_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_36_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_37_REGOFF 0x94
+#define HWIO_DDR_CTL_37_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_37_REGOFF)
+#define HWIO_DDR_CTL_37_TRAS_MAX_F0_FLDMASK (0x1ffff)
+#define HWIO_DDR_CTL_37_TRAS_MAX_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_37_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_37_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_CTL_37_TCKE_F0_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_37_TCKE_F0_FLDSHFT (24)
+#define HWIO_DDR_CTL_37_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_37_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_38_REGOFF 0x98
+#define HWIO_DDR_CTL_38_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_38_REGOFF)
+#define HWIO_DDR_CTL_38_TCKESR_F0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_38_TCKESR_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_38_TRTP_F1_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_38_TRTP_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_38_TMRD_F1_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_38_TMRD_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_38_TMOD_F1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_38_TMOD_F1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_39_REGOFF 0x9c
+#define HWIO_DDR_CTL_39_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_39_REGOFF)
+#define HWIO_DDR_CTL_39_TRAS_MAX_F1_FLDMASK (0x1ffff)
+#define HWIO_DDR_CTL_39_TRAS_MAX_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_39_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_39_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_CTL_39_TCKE_F1_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_39_TCKE_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_39_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_39_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_40_REGOFF 0xa0
+#define HWIO_DDR_CTL_40_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_40_REGOFF)
+#define HWIO_DDR_CTL_40_TCKESR_F1_FLDMASK (0xff)
+#define HWIO_DDR_CTL_40_TCKESR_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_40_TRTP_F2_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_40_TRTP_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_40_TMRD_F2_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_40_TMRD_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_40_TMOD_F2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_40_TMOD_F2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_41_REGOFF 0xa4
+#define HWIO_DDR_CTL_41_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_41_REGOFF)
+#define HWIO_DDR_CTL_41_TRAS_MAX_F2_FLDMASK (0x1ffff)
+#define HWIO_DDR_CTL_41_TRAS_MAX_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_41_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_41_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_CTL_41_TCKE_F2_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_41_TCKE_F2_FLDSHFT (24)
+#define HWIO_DDR_CTL_41_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_41_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_42_REGOFF 0xa8
+#define HWIO_DDR_CTL_42_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_42_REGOFF)
+#define HWIO_DDR_CTL_42_TCKESR_F2_FLDMASK (0xff)
+#define HWIO_DDR_CTL_42_TCKESR_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_42_TRTP_F3_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_42_TRTP_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_42_TMRD_F3_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_42_TMRD_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_42_TMOD_F3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_42_TMOD_F3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_43_REGOFF 0xac
+#define HWIO_DDR_CTL_43_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_43_REGOFF)
+#define HWIO_DDR_CTL_43_TRAS_MAX_F3_FLDMASK (0x1ffff)
+#define HWIO_DDR_CTL_43_TRAS_MAX_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_43_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_43_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_CTL_43_TCKE_F3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_43_TCKE_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_43_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_43_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_44_REGOFF 0xb0
+#define HWIO_DDR_CTL_44_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_44_REGOFF)
+#define HWIO_DDR_CTL_44_TCKESR_F3_FLDMASK (0xff)
+#define HWIO_DDR_CTL_44_TCKESR_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_44_TPPD_FLDMASK (0x700)
+#define HWIO_DDR_CTL_44_TPPD_FLDSHFT (8)
+#define HWIO_DDR_CTL_44_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_CTL_44_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_CTL_44_CDNS_INTRL2_FLDMASK (0x70000)
+#define HWIO_DDR_CTL_44_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_44_CDNS_INTRL1_FLDMASK (0xf80000)
+#define HWIO_DDR_CTL_44_CDNS_INTRL1_FLDSHFT (19)
+#define HWIO_DDR_CTL_44_CDNS_INTRL3_FLDMASK (0x7000000)
+#define HWIO_DDR_CTL_44_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_44_RESERVED4_FLDMASK (0xf8000000)
+#define HWIO_DDR_CTL_44_RESERVED4_FLDSHFT (27)
+
+#define HWIO_DDR_CTL_45_REGOFF 0xb4
+#define HWIO_DDR_CTL_45_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_45_REGOFF)
+#define HWIO_DDR_CTL_45_WRITEINTERP_FLDMASK (0x1)
+#define HWIO_DDR_CTL_45_WRITEINTERP_FLDSHFT (0)
+#define HWIO_DDR_CTL_45_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_45_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_45_TRCD_F0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_45_TRCD_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_45_TWR_F0_FLDMASK (0x3f0000)
+#define HWIO_DDR_CTL_45_TWR_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_45_CDNS_INTRL1_FLDMASK (0xc00000)
+#define HWIO_DDR_CTL_45_CDNS_INTRL1_FLDSHFT (22)
+#define HWIO_DDR_CTL_45_TRCD_F1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_45_TRCD_F1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_46_REGOFF 0xb8
+#define HWIO_DDR_CTL_46_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_46_REGOFF)
+#define HWIO_DDR_CTL_46_TWR_F1_FLDMASK (0x3f)
+#define HWIO_DDR_CTL_46_TWR_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_46_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_CTL_46_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_CTL_46_TRCD_F2_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_46_TRCD_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_46_TWR_F2_FLDMASK (0x3f0000)
+#define HWIO_DDR_CTL_46_TWR_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_46_CDNS_INTRL1_FLDMASK (0xc00000)
+#define HWIO_DDR_CTL_46_CDNS_INTRL1_FLDSHFT (22)
+#define HWIO_DDR_CTL_46_TRCD_F3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_46_TRCD_F3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_47_REGOFF 0xbc
+#define HWIO_DDR_CTL_47_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_47_REGOFF)
+#define HWIO_DDR_CTL_47_TWR_F3_FLDMASK (0x3f)
+#define HWIO_DDR_CTL_47_TWR_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_47_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_CTL_47_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_CTL_47_TMRR_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_47_TMRR_FLDSHFT (8)
+#define HWIO_DDR_CTL_47_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_47_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_47_TCACKEL_FLDMASK (0x1f0000)
+#define HWIO_DDR_CTL_47_TCACKEL_FLDSHFT (16)
+#define HWIO_DDR_CTL_47_CDNS_INTRL2_FLDMASK (0xe00000)
+#define HWIO_DDR_CTL_47_CDNS_INTRL2_FLDSHFT (21)
+#define HWIO_DDR_CTL_47_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_47_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_48_REGOFF 0xc0
+#define HWIO_DDR_CTL_48_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_48_REGOFF)
+#define HWIO_DDR_CTL_48_TCAENT_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_48_TCAENT_FLDSHFT (0)
+#define HWIO_DDR_CTL_48_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_48_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_48_TCAMRD_FLDMASK (0x3f0000)
+#define HWIO_DDR_CTL_48_TCAMRD_FLDSHFT (16)
+#define HWIO_DDR_CTL_48_CDNS_INTRL1_FLDMASK (0xc00000)
+#define HWIO_DDR_CTL_48_CDNS_INTRL1_FLDSHFT (22)
+#define HWIO_DDR_CTL_48_TCAEXT_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_48_TCAEXT_FLDSHFT (24)
+#define HWIO_DDR_CTL_48_CDNS_INTRL2_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_48_CDNS_INTRL2_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_49_REGOFF 0xc4
+#define HWIO_DDR_CTL_49_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_49_REGOFF)
+#define HWIO_DDR_CTL_49_TCACKEH_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_49_TCACKEH_FLDSHFT (0)
+#define HWIO_DDR_CTL_49_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_49_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_49_TMRZ_F0_FLDMASK (0x1f00)
+#define HWIO_DDR_CTL_49_TMRZ_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_49_CDNS_INTRL1_FLDMASK (0xe000)
+#define HWIO_DDR_CTL_49_CDNS_INTRL1_FLDSHFT (13)
+#define HWIO_DDR_CTL_49_TMRZ_F1_FLDMASK (0x1f0000)
+#define HWIO_DDR_CTL_49_TMRZ_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_49_CDNS_INTRL2_FLDMASK (0xe00000)
+#define HWIO_DDR_CTL_49_CDNS_INTRL2_FLDSHFT (21)
+#define HWIO_DDR_CTL_49_TMRZ_F2_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_49_TMRZ_F2_FLDSHFT (24)
+#define HWIO_DDR_CTL_49_CDNS_INTRL3_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_49_CDNS_INTRL3_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_50_REGOFF 0xc8
+#define HWIO_DDR_CTL_50_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_50_REGOFF)
+#define HWIO_DDR_CTL_50_TMRZ_F3_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_50_TMRZ_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_50_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_50_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_50_AP_FLDMASK (0x100)
+#define HWIO_DDR_CTL_50_AP_FLDSHFT (8)
+#define HWIO_DDR_CTL_50_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_50_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_50_CONCURRENTAP_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_50_CONCURRENTAP_FLDSHFT (16)
+#define HWIO_DDR_CTL_50_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_50_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_50_TRAS_LOCKOUT_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_50_TRAS_LOCKOUT_FLDSHFT (24)
+#define HWIO_DDR_CTL_50_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_50_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_51_REGOFF 0xcc
+#define HWIO_DDR_CTL_51_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_51_REGOFF)
+#define HWIO_DDR_CTL_51_TDAL_F0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_51_TDAL_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_51_TDAL_F1_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_51_TDAL_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_51_TDAL_F2_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_51_TDAL_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_51_TDAL_F3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_51_TDAL_F3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_52_REGOFF 0xd0
+#define HWIO_DDR_CTL_52_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_52_REGOFF)
+#define HWIO_DDR_CTL_52_BSTLEN_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_52_BSTLEN_FLDSHFT (0)
+#define HWIO_DDR_CTL_52_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_52_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_52_TRP_AB_F0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_52_TRP_AB_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_52_TRP_AB_F1_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_52_TRP_AB_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_52_TRP_AB_F2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_52_TRP_AB_F2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_53_REGOFF 0xd4
+#define HWIO_DDR_CTL_53_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_53_REGOFF)
+#define HWIO_DDR_CTL_53_TRP_AB_F3_FLDMASK (0xff)
+#define HWIO_DDR_CTL_53_TRP_AB_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_53_REG_DIMM_ENABLE_FLDMASK (0x100)
+#define HWIO_DDR_CTL_53_REG_DIMM_ENABLE_FLDSHFT (8)
+#define HWIO_DDR_CTL_53_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_53_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_53_OPTIMAL_RMODW_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_53_OPTIMAL_RMODW_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_53_CDNS_INTRL1_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_53_CDNS_INTRL1_FLDSHFT (17)
+#define HWIO_DDR_CTL_53_CDNS_INTRL3_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_53_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_53_CDNS_INTRL2_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_53_CDNS_INTRL2_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_54_REGOFF 0xd8
+#define HWIO_DDR_CTL_54_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_54_REGOFF)
+#define HWIO_DDR_CTL_54_NO_MEMORY_DM_FLDMASK (0x1)
+#define HWIO_DDR_CTL_54_NO_MEMORY_DM_FLDSHFT (0)
+#define HWIO_DDR_CTL_54_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_54_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_54_AREFRESH_FLDMASK (0x100)
+#define HWIO_DDR_CTL_54_AREFRESH_FLDSHFT (8)
+#define HWIO_DDR_CTL_54_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_54_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_54_CDNS_INTRL2_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_54_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_54_CDNS_INTRL3_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_54_CDNS_INTRL3_FLDSHFT (17)
+#define HWIO_DDR_CTL_54_TREF_ENABLE_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_54_TREF_ENABLE_FLDSHFT (24)
+#define HWIO_DDR_CTL_54_RESERVED4_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_54_RESERVED4_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_55_REGOFF 0xdc
+#define HWIO_DDR_CTL_55_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_55_REGOFF)
+#define HWIO_DDR_CTL_55_CDNS_INTRL0_FLDMASK (0x7)
+#define HWIO_DDR_CTL_55_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_55_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_55_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_55_TRFC_F0_FLDMASK (0x3ff00)
+#define HWIO_DDR_CTL_55_TRFC_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_55_CDNS_INTRL1_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_55_CDNS_INTRL1_FLDSHFT (18)
+#define HWIO_DDR_CTL_55_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_55_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_56_REGOFF 0xe0
+#define HWIO_DDR_CTL_56_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_56_REGOFF)
+#define HWIO_DDR_CTL_56_TREF_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_56_TREF_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_56_TRFC_F1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_56_TRFC_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_56_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_56_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_57_REGOFF 0xe4
+#define HWIO_DDR_CTL_57_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_57_REGOFF)
+#define HWIO_DDR_CTL_57_TREF_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_57_TREF_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_57_TRFC_F2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_57_TRFC_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_57_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_57_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_58_REGOFF 0xe8
+#define HWIO_DDR_CTL_58_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_58_REGOFF)
+#define HWIO_DDR_CTL_58_TREF_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_58_TREF_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_58_TRFC_F3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_58_TRFC_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_58_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_58_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_59_REGOFF 0xec
+#define HWIO_DDR_CTL_59_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_59_REGOFF)
+#define HWIO_DDR_CTL_59_TREF_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_59_TREF_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_59_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_59_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_60_REGOFF 0xf0
+#define HWIO_DDR_CTL_60_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_60_REGOFF)
+#define HWIO_DDR_CTL_60_TPDEX_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_60_TPDEX_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_60_TPDEX_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_60_TPDEX_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_61_REGOFF 0xf4
+#define HWIO_DDR_CTL_61_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_61_REGOFF)
+#define HWIO_DDR_CTL_61_TPDEX_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_61_TPDEX_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_61_TPDEX_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_61_TPDEX_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_62_REGOFF 0xf8
+#define HWIO_DDR_CTL_62_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_62_REGOFF)
+#define HWIO_DDR_CTL_62_TMRRI_F0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_62_TMRRI_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_62_TMRRI_F1_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_62_TMRRI_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_62_TMRRI_F2_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_62_TMRRI_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_62_TMRRI_F3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_62_TMRRI_F3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_63_REGOFF 0xfc
+#define HWIO_DDR_CTL_63_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_63_REGOFF)
+#define HWIO_DDR_CTL_63_TCSCKE_F0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_63_TCSCKE_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_63_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_63_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_63_TCKELCS_F0_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_63_TCKELCS_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_63_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_63_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_63_TCKEHCS_F0_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_63_TCKEHCS_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_63_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_63_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_63_TMRWCKEL_F0_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_63_TMRWCKEL_F0_FLDSHFT (24)
+#define HWIO_DDR_CTL_63_CDNS_INTRL3_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_63_CDNS_INTRL3_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_64_REGOFF 0x100
+#define HWIO_DDR_CTL_64_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_64_REGOFF)
+#define HWIO_DDR_CTL_64_TZQCKE_F0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_64_TZQCKE_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_64_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_64_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_64_CA_DEFAULT_VAL_F0_FLDMASK (0x100)
+#define HWIO_DDR_CTL_64_CA_DEFAULT_VAL_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_64_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_64_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_64_TCSCKE_F1_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_64_TCSCKE_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_64_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_64_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_64_TCKELCS_F1_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_64_TCKELCS_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_64_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_64_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_65_REGOFF 0x104
+#define HWIO_DDR_CTL_65_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_65_REGOFF)
+#define HWIO_DDR_CTL_65_TCKEHCS_F1_FLDMASK (0xf)
+#define HWIO_DDR_CTL_65_TCKEHCS_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_65_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_65_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_65_TMRWCKEL_F1_FLDMASK (0x1f00)
+#define HWIO_DDR_CTL_65_TMRWCKEL_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_65_CDNS_INTRL1_FLDMASK (0xe000)
+#define HWIO_DDR_CTL_65_CDNS_INTRL1_FLDSHFT (13)
+#define HWIO_DDR_CTL_65_TZQCKE_F1_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_65_TZQCKE_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_65_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_65_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_65_CA_DEFAULT_VAL_F1_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_65_CA_DEFAULT_VAL_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_65_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_65_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_66_REGOFF 0x108
+#define HWIO_DDR_CTL_66_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_66_REGOFF)
+#define HWIO_DDR_CTL_66_TCSCKE_F2_FLDMASK (0xf)
+#define HWIO_DDR_CTL_66_TCSCKE_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_66_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_66_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_66_TCKELCS_F2_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_66_TCKELCS_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_66_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_66_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_66_TCKEHCS_F2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_66_TCKEHCS_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_66_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_66_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_66_TMRWCKEL_F2_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_66_TMRWCKEL_F2_FLDSHFT (24)
+#define HWIO_DDR_CTL_66_CDNS_INTRL3_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_66_CDNS_INTRL3_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_67_REGOFF 0x10c
+#define HWIO_DDR_CTL_67_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_67_REGOFF)
+#define HWIO_DDR_CTL_67_TZQCKE_F2_FLDMASK (0xf)
+#define HWIO_DDR_CTL_67_TZQCKE_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_67_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_67_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_67_CA_DEFAULT_VAL_F2_FLDMASK (0x100)
+#define HWIO_DDR_CTL_67_CA_DEFAULT_VAL_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_67_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_67_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_67_TCSCKE_F3_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_67_TCSCKE_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_67_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_67_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_67_TCKELCS_F3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_67_TCKELCS_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_67_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_67_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_68_REGOFF 0x110
+#define HWIO_DDR_CTL_68_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_68_REGOFF)
+#define HWIO_DDR_CTL_68_TCKEHCS_F3_FLDMASK (0xf)
+#define HWIO_DDR_CTL_68_TCKEHCS_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_68_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_68_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_68_TMRWCKEL_F3_FLDMASK (0x1f00)
+#define HWIO_DDR_CTL_68_TMRWCKEL_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_68_CDNS_INTRL1_FLDMASK (0xe000)
+#define HWIO_DDR_CTL_68_CDNS_INTRL1_FLDSHFT (13)
+#define HWIO_DDR_CTL_68_TZQCKE_F3_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_68_TZQCKE_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_68_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_68_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_68_CA_DEFAULT_VAL_F3_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_68_CA_DEFAULT_VAL_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_68_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_68_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_69_REGOFF 0x114
+#define HWIO_DDR_CTL_69_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_69_REGOFF)
+#define HWIO_DDR_CTL_69_TXSR_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_69_TXSR_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_69_TXSNR_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_69_TXSNR_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_70_REGOFF 0x118
+#define HWIO_DDR_CTL_70_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_70_REGOFF)
+#define HWIO_DDR_CTL_70_TXSR_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_70_TXSR_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_70_TXSNR_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_70_TXSNR_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_71_REGOFF 0x11c
+#define HWIO_DDR_CTL_71_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_71_REGOFF)
+#define HWIO_DDR_CTL_71_TXSR_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_71_TXSR_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_71_TXSNR_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_71_TXSNR_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_72_REGOFF 0x120
+#define HWIO_DDR_CTL_72_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_72_REGOFF)
+#define HWIO_DDR_CTL_72_TXSR_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_72_TXSR_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_72_TXSNR_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_72_TXSNR_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_73_REGOFF 0x124
+#define HWIO_DDR_CTL_73_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_73_REGOFF)
+#define HWIO_DDR_CTL_73_TCKELCMD_F0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_73_TCKELCMD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_73_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_73_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_73_TCKEHCMD_F0_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_73_TCKEHCMD_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_73_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_73_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_73_TCKCKEL_F0_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_73_TCKCKEL_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_73_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_73_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_73_TSR_F0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_73_TSR_F0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_74_REGOFF 0x128
+#define HWIO_DDR_CTL_74_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_74_REGOFF)
+#define HWIO_DDR_CTL_74_TESCKE_F0_FLDMASK (0x7)
+#define HWIO_DDR_CTL_74_TESCKE_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_74_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_74_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_74_TCKELPD_F0_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_74_TCKELPD_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_74_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_74_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_74_TCSCKEH_F0_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_74_TCSCKEH_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_74_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_74_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_74_TCMDCKE_F0_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_74_TCMDCKE_F0_FLDSHFT (24)
+#define HWIO_DDR_CTL_74_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_74_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_75_REGOFF 0x12c
+#define HWIO_DDR_CTL_75_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_75_REGOFF)
+#define HWIO_DDR_CTL_75_TCKELCMD_F1_FLDMASK (0xf)
+#define HWIO_DDR_CTL_75_TCKELCMD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_75_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_75_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_75_TCKEHCMD_F1_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_75_TCKEHCMD_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_75_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_75_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_75_TCKCKEL_F1_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_75_TCKCKEL_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_75_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_75_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_75_TSR_F1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_75_TSR_F1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_76_REGOFF 0x130
+#define HWIO_DDR_CTL_76_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_76_REGOFF)
+#define HWIO_DDR_CTL_76_TESCKE_F1_FLDMASK (0x7)
+#define HWIO_DDR_CTL_76_TESCKE_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_76_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_76_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_76_TCKELPD_F1_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_76_TCKELPD_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_76_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_76_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_76_TCSCKEH_F1_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_76_TCSCKEH_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_76_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_76_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_76_TCMDCKE_F1_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_76_TCMDCKE_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_76_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_76_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_77_REGOFF 0x134
+#define HWIO_DDR_CTL_77_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_77_REGOFF)
+#define HWIO_DDR_CTL_77_TCKELCMD_F2_FLDMASK (0xf)
+#define HWIO_DDR_CTL_77_TCKELCMD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_77_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_77_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_77_TCKEHCMD_F2_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_77_TCKEHCMD_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_77_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_77_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_77_TCKCKEL_F2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_77_TCKCKEL_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_77_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_77_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_77_TSR_F2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_77_TSR_F2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_78_REGOFF 0x138
+#define HWIO_DDR_CTL_78_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_78_REGOFF)
+#define HWIO_DDR_CTL_78_TESCKE_F2_FLDMASK (0x7)
+#define HWIO_DDR_CTL_78_TESCKE_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_78_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_78_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_78_TCKELPD_F2_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_78_TCKELPD_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_78_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_78_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_78_TCSCKEH_F2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_78_TCSCKEH_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_78_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_78_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_78_TCMDCKE_F2_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_78_TCMDCKE_F2_FLDSHFT (24)
+#define HWIO_DDR_CTL_78_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_78_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_79_REGOFF 0x13c
+#define HWIO_DDR_CTL_79_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_79_REGOFF)
+#define HWIO_DDR_CTL_79_TCKELCMD_F3_FLDMASK (0xf)
+#define HWIO_DDR_CTL_79_TCKELCMD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_79_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_79_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_79_TCKEHCMD_F3_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_79_TCKEHCMD_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_79_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_79_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_79_TCKCKEL_F3_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_79_TCKCKEL_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_79_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_79_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_79_TSR_F3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_79_TSR_F3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_80_REGOFF 0x140
+#define HWIO_DDR_CTL_80_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_80_REGOFF)
+#define HWIO_DDR_CTL_80_TESCKE_F3_FLDMASK (0x7)
+#define HWIO_DDR_CTL_80_TESCKE_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_80_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_80_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_80_TCKELPD_F3_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_80_TCKELPD_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_80_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_80_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_80_TCSCKEH_F3_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_80_TCSCKEH_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_80_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_80_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_80_TCMDCKE_F3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_80_TCMDCKE_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_80_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_80_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_81_REGOFF 0x144
+#define HWIO_DDR_CTL_81_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_81_REGOFF)
+#define HWIO_DDR_CTL_81_PWRUP_SREFRESH_EXIT_FLDMASK (0x1)
+#define HWIO_DDR_CTL_81_PWRUP_SREFRESH_EXIT_FLDSHFT (0)
+#define HWIO_DDR_CTL_81_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_81_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_81_CDNS_INTRL1_FLDMASK (0x100)
+#define HWIO_DDR_CTL_81_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_81_CDNS_INTRL2_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_81_CDNS_INTRL2_FLDSHFT (9)
+#define HWIO_DDR_CTL_81_ENABLE_QUICK_SREFRESH_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_81_ENABLE_QUICK_SREFRESH_FLDSHFT (16)
+#define HWIO_DDR_CTL_81_CDNS_INTRL3_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_81_CDNS_INTRL3_FLDSHFT (17)
+#define HWIO_DDR_CTL_81_CKE_DELAY_FLDMASK (0x7000000)
+#define HWIO_DDR_CTL_81_CKE_DELAY_FLDSHFT (24)
+#define HWIO_DDR_CTL_81_RESERVED4_FLDMASK (0xf8000000)
+#define HWIO_DDR_CTL_81_RESERVED4_FLDSHFT (27)
+
+#define HWIO_DDR_CTL_82_REGOFF 0x148
+#define HWIO_DDR_CTL_82_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_82_REGOFF)
+#define HWIO_DDR_CTL_82_CDNS_INTRL0_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_82_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_82_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_82_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_82_DFS_STATUS_FLDMASK (0x300)
+#define HWIO_DDR_CTL_82_DFS_STATUS_FLDSHFT (8)
+#define HWIO_DDR_CTL_82_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_82_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_82_DFS_ZQ_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_82_DFS_ZQ_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_82_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_82_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_82_DFS_CALVL_EN_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_82_DFS_CALVL_EN_FLDSHFT (24)
+#define HWIO_DDR_CTL_82_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_82_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_83_REGOFF 0x14c
+#define HWIO_DDR_CTL_83_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_83_REGOFF)
+#define HWIO_DDR_CTL_83_DFS_WRLVL_EN_FLDMASK (0x1)
+#define HWIO_DDR_CTL_83_DFS_WRLVL_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_83_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_83_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_83_DFS_RDLVL_EN_FLDMASK (0x100)
+#define HWIO_DDR_CTL_83_DFS_RDLVL_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_83_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_83_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_83_DFS_RDLVL_GATE_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_83_DFS_RDLVL_GATE_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_83_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_83_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_83_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_83_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_84_REGOFF 0x150
+#define HWIO_DDR_CTL_84_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_84_REGOFF)
+#define HWIO_DDR_CTL_84_DFS_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_84_DFS_PROMOTE_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_84_DFS_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_84_DFS_PROMOTE_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_85_REGOFF 0x154
+#define HWIO_DDR_CTL_85_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_85_REGOFF)
+#define HWIO_DDR_CTL_85_DFS_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_85_DFS_PROMOTE_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_85_DFS_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_85_DFS_PROMOTE_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_86_REGOFF 0x158
+#define HWIO_DDR_CTL_86_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_86_REGOFF)
+#define HWIO_DDR_CTL_86_ZQ_CALSTART_STATUS_FLDMASK (0x3)
+#define HWIO_DDR_CTL_86_ZQ_CALSTART_STATUS_FLDSHFT (0)
+#define HWIO_DDR_CTL_86_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_CTL_86_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_CTL_86_ZQ_CALLATCH_STATUS_FLDMASK (0x300)
+#define HWIO_DDR_CTL_86_ZQ_CALLATCH_STATUS_FLDSHFT (8)
+#define HWIO_DDR_CTL_86_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_86_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_86_ZQ_CALINIT_CS_CL_STATUS_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_86_ZQ_CALINIT_CS_CL_STATUS_FLDSHFT (16)
+#define HWIO_DDR_CTL_86_CDNS_INTRL2_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_86_CDNS_INTRL2_FLDSHFT (18)
+#define HWIO_DDR_CTL_86_CDNS_INTRL3_FLDMASK (0x7000000)
+#define HWIO_DDR_CTL_86_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_86_RESERVED4_FLDMASK (0xf8000000)
+#define HWIO_DDR_CTL_86_RESERVED4_FLDSHFT (27)
+
+#define HWIO_DDR_CTL_87_REGOFF 0x15c
+#define HWIO_DDR_CTL_87_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_87_REGOFF)
+#define HWIO_DDR_CTL_87_CDNS_INTRL0_FLDMASK (0x7)
+#define HWIO_DDR_CTL_87_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_87_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_87_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_87_CDNS_INTRL1_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_87_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_87_CDNS_INTRL2_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_87_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_87_CDNS_INTRL3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_87_CDNS_INTRL3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_88_REGOFF 0x160
+#define HWIO_DDR_CTL_88_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_88_REGOFF)
+#define HWIO_DDR_CTL_88_UPD_CTRLUPD_NORM_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_88_UPD_CTRLUPD_NORM_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_88_UPD_CTRLUPD_HIGH_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_88_UPD_CTRLUPD_HIGH_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_89_REGOFF 0x164
+#define HWIO_DDR_CTL_89_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_89_REGOFF)
+#define HWIO_DDR_CTL_89_UPD_CTRLUPD_TIMEOUT_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_89_UPD_CTRLUPD_TIMEOUT_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_89_UPD_CTRLUPD_SW_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_89_UPD_CTRLUPD_SW_PROMOTE_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_90_REGOFF 0x168
+#define HWIO_DDR_CTL_90_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_90_REGOFF)
+#define HWIO_DDR_CTL_90_UPD_PHYUPD_DFI_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_90_UPD_PHYUPD_DFI_PROMOTE_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_90_UPD_CTRLUPD_NORM_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_90_UPD_CTRLUPD_NORM_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_91_REGOFF 0x16c
+#define HWIO_DDR_CTL_91_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_91_REGOFF)
+#define HWIO_DDR_CTL_91_UPD_CTRLUPD_HIGH_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_91_UPD_CTRLUPD_HIGH_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_91_UPD_CTRLUPD_TIMEOUT_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_91_UPD_CTRLUPD_TIMEOUT_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_92_REGOFF 0x170
+#define HWIO_DDR_CTL_92_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_92_REGOFF)
+#define HWIO_DDR_CTL_92_UPD_CTRLUPD_SW_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_92_UPD_CTRLUPD_SW_PROMOTE_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_92_UPD_PHYUPD_DFI_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_92_UPD_PHYUPD_DFI_PROMOTE_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_93_REGOFF 0x174
+#define HWIO_DDR_CTL_93_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_93_REGOFF)
+#define HWIO_DDR_CTL_93_UPD_CTRLUPD_NORM_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_93_UPD_CTRLUPD_NORM_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_93_UPD_CTRLUPD_HIGH_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_93_UPD_CTRLUPD_HIGH_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_94_REGOFF 0x178
+#define HWIO_DDR_CTL_94_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_94_REGOFF)
+#define HWIO_DDR_CTL_94_UPD_CTRLUPD_TIMEOUT_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_94_UPD_CTRLUPD_TIMEOUT_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_94_UPD_CTRLUPD_SW_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_94_UPD_CTRLUPD_SW_PROMOTE_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_95_REGOFF 0x17c
+#define HWIO_DDR_CTL_95_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_95_REGOFF)
+#define HWIO_DDR_CTL_95_UPD_PHYUPD_DFI_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_95_UPD_PHYUPD_DFI_PROMOTE_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_95_UPD_CTRLUPD_NORM_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_95_UPD_CTRLUPD_NORM_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_96_REGOFF 0x180
+#define HWIO_DDR_CTL_96_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_96_REGOFF)
+#define HWIO_DDR_CTL_96_UPD_CTRLUPD_HIGH_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_96_UPD_CTRLUPD_HIGH_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_96_UPD_CTRLUPD_TIMEOUT_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_96_UPD_CTRLUPD_TIMEOUT_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_97_REGOFF 0x184
+#define HWIO_DDR_CTL_97_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_97_REGOFF)
+#define HWIO_DDR_CTL_97_UPD_CTRLUPD_SW_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_97_UPD_CTRLUPD_SW_PROMOTE_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_97_UPD_PHYUPD_DFI_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_97_UPD_PHYUPD_DFI_PROMOTE_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_98_REGOFF 0x188
+#define HWIO_DDR_CTL_98_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_98_REGOFF)
+#define HWIO_DDR_CTL_98_TDFI_PHYMSTR_MAX_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_98_TDFI_PHYMSTR_MAX_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_98_TDFI_PHYMSTR_RESP_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_98_TDFI_PHYMSTR_RESP_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_99_REGOFF 0x18c
+#define HWIO_DDR_CTL_99_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_99_REGOFF)
+#define HWIO_DDR_CTL_99_PHYMSTR_DFI_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_99_PHYMSTR_DFI_PROMOTE_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_99_TDFI_PHYMSTR_MAX_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_99_TDFI_PHYMSTR_MAX_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_100_REGOFF 0x190
+#define HWIO_DDR_CTL_100_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_100_REGOFF)
+#define HWIO_DDR_CTL_100_TDFI_PHYMSTR_RESP_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_100_TDFI_PHYMSTR_RESP_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_100_PHYMSTR_DFI_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_100_PHYMSTR_DFI_PROMOTE_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_101_REGOFF 0x194
+#define HWIO_DDR_CTL_101_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_101_REGOFF)
+#define HWIO_DDR_CTL_101_TDFI_PHYMSTR_MAX_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_101_TDFI_PHYMSTR_MAX_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_101_TDFI_PHYMSTR_RESP_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_101_TDFI_PHYMSTR_RESP_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_102_REGOFF 0x198
+#define HWIO_DDR_CTL_102_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_102_REGOFF)
+#define HWIO_DDR_CTL_102_PHYMSTR_DFI_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_102_PHYMSTR_DFI_PROMOTE_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_102_TDFI_PHYMSTR_MAX_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_102_TDFI_PHYMSTR_MAX_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_103_REGOFF 0x19c
+#define HWIO_DDR_CTL_103_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_103_REGOFF)
+#define HWIO_DDR_CTL_103_TDFI_PHYMSTR_RESP_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_103_TDFI_PHYMSTR_RESP_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_103_PHYMSTR_DFI_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_103_PHYMSTR_DFI_PROMOTE_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_104_REGOFF 0x1a0
+#define HWIO_DDR_CTL_104_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_104_REGOFF)
+#define HWIO_DDR_CTL_104_PHYMSTR_NO_AREF_FLDMASK (0x1)
+#define HWIO_DDR_CTL_104_PHYMSTR_NO_AREF_FLDSHFT (0)
+#define HWIO_DDR_CTL_104_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_104_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_104_PHYMSTR_ERROR_STATUS_FLDMASK (0x300)
+#define HWIO_DDR_CTL_104_PHYMSTR_ERROR_STATUS_FLDSHFT (8)
+#define HWIO_DDR_CTL_104_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_104_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_104_MRR_TEMPCHK_NORM_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_104_MRR_TEMPCHK_NORM_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_105_REGOFF 0x1a4
+#define HWIO_DDR_CTL_105_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_105_REGOFF)
+#define HWIO_DDR_CTL_105_MRR_TEMPCHK_HIGH_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_105_MRR_TEMPCHK_HIGH_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_105_MRR_TEMPCHK_TIMEOUT_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_105_MRR_TEMPCHK_TIMEOUT_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_106_REGOFF 0x1a8
+#define HWIO_DDR_CTL_106_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_106_REGOFF)
+#define HWIO_DDR_CTL_106_MRR_TEMPCHK_NORM_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_106_MRR_TEMPCHK_NORM_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_106_MRR_TEMPCHK_HIGH_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_106_MRR_TEMPCHK_HIGH_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_107_REGOFF 0x1ac
+#define HWIO_DDR_CTL_107_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_107_REGOFF)
+#define HWIO_DDR_CTL_107_MRR_TEMPCHK_TIMEOUT_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_107_MRR_TEMPCHK_TIMEOUT_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_107_MRR_TEMPCHK_NORM_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_107_MRR_TEMPCHK_NORM_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_108_REGOFF 0x1b0
+#define HWIO_DDR_CTL_108_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_108_REGOFF)
+#define HWIO_DDR_CTL_108_MRR_TEMPCHK_HIGH_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_108_MRR_TEMPCHK_HIGH_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_108_MRR_TEMPCHK_TIMEOUT_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_108_MRR_TEMPCHK_TIMEOUT_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_109_REGOFF 0x1b4
+#define HWIO_DDR_CTL_109_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_109_REGOFF)
+#define HWIO_DDR_CTL_109_MRR_TEMPCHK_NORM_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_109_MRR_TEMPCHK_NORM_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_109_MRR_TEMPCHK_HIGH_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_109_MRR_TEMPCHK_HIGH_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_110_REGOFF 0x1b8
+#define HWIO_DDR_CTL_110_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_110_REGOFF)
+#define HWIO_DDR_CTL_110_MRR_TEMPCHK_TIMEOUT_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_110_MRR_TEMPCHK_TIMEOUT_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_110_LOWPOWER_REFRESH_ENABLE_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_110_LOWPOWER_REFRESH_ENABLE_FLDSHFT (16)
+#define HWIO_DDR_CTL_110_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_110_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_CTL_110_CKSRE_F0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_110_CKSRE_F0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_111_REGOFF 0x1bc
+#define HWIO_DDR_CTL_111_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_111_REGOFF)
+#define HWIO_DDR_CTL_111_CKSRX_F0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_111_CKSRX_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_111_CKSRE_F1_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_111_CKSRE_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_111_CKSRX_F1_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_111_CKSRX_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_111_CKSRE_F2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_111_CKSRE_F2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_112_REGOFF 0x1c0
+#define HWIO_DDR_CTL_112_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_112_REGOFF)
+#define HWIO_DDR_CTL_112_CKSRX_F2_FLDMASK (0xff)
+#define HWIO_DDR_CTL_112_CKSRX_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_112_CKSRE_F3_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_112_CKSRE_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_112_CKSRX_F3_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_112_CKSRX_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_112_LP_CMD_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_112_LP_CMD_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_113_REGOFF 0x1c4
+#define HWIO_DDR_CTL_113_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_113_REGOFF)
+#define HWIO_DDR_CTL_113_LPI_PD_WAKEUP_F0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_113_LPI_PD_WAKEUP_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_113_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_113_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_113_LPI_SR_WAKEUP_F0_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_113_LPI_SR_WAKEUP_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_113_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_113_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_113_LPI_SR_MCCLK_GATE_WAKEUP_F0_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_113_LPI_SR_MCCLK_GATE_WAKEUP_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_113_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_113_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_113_LPI_SRPD_LITE_WAKEUP_F0_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_113_LPI_SRPD_LITE_WAKEUP_F0_FLDSHFT (24)
+#define HWIO_DDR_CTL_113_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_113_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_114_REGOFF 0x1c8
+#define HWIO_DDR_CTL_114_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_114_REGOFF)
+#define HWIO_DDR_CTL_114_LPI_SRPD_DEEP_WAKEUP_F0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_114_LPI_SRPD_DEEP_WAKEUP_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_114_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_114_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_114_LPI_SRPD_DEEP_MCCLK_GATE_WAKEUP_F0_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_114_LPI_SRPD_DEEP_MCCLK_GATE_WAKEUP_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_114_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_114_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_114_LPI_TIMER_WAKEUP_F0_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_114_LPI_TIMER_WAKEUP_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_114_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_114_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_114_LPI_PD_WAKEUP_F1_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_114_LPI_PD_WAKEUP_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_114_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_114_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_115_REGOFF 0x1cc
+#define HWIO_DDR_CTL_115_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_115_REGOFF)
+#define HWIO_DDR_CTL_115_LPI_SR_WAKEUP_F1_FLDMASK (0xf)
+#define HWIO_DDR_CTL_115_LPI_SR_WAKEUP_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_115_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_115_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_115_LPI_SR_MCCLK_GATE_WAKEUP_F1_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_115_LPI_SR_MCCLK_GATE_WAKEUP_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_115_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_115_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_115_LPI_SRPD_LITE_WAKEUP_F1_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_115_LPI_SRPD_LITE_WAKEUP_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_115_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_115_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_115_LPI_SRPD_DEEP_WAKEUP_F1_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_115_LPI_SRPD_DEEP_WAKEUP_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_115_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_115_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_116_REGOFF 0x1d0
+#define HWIO_DDR_CTL_116_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_116_REGOFF)
+#define HWIO_DDR_CTL_116_LPI_SRPD_DEEP_MCCLK_GATE_WAKEUP_F1_FLDMASK (0xf)
+#define HWIO_DDR_CTL_116_LPI_SRPD_DEEP_MCCLK_GATE_WAKEUP_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_116_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_116_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_116_LPI_TIMER_WAKEUP_F1_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_116_LPI_TIMER_WAKEUP_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_116_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_116_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_116_LPI_PD_WAKEUP_F2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_116_LPI_PD_WAKEUP_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_116_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_116_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_116_LPI_SR_WAKEUP_F2_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_116_LPI_SR_WAKEUP_F2_FLDSHFT (24)
+#define HWIO_DDR_CTL_116_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_116_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_117_REGOFF 0x1d4
+#define HWIO_DDR_CTL_117_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_117_REGOFF)
+#define HWIO_DDR_CTL_117_LPI_SR_MCCLK_GATE_WAKEUP_F2_FLDMASK (0xf)
+#define HWIO_DDR_CTL_117_LPI_SR_MCCLK_GATE_WAKEUP_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_117_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_117_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_117_LPI_SRPD_LITE_WAKEUP_F2_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_117_LPI_SRPD_LITE_WAKEUP_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_117_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_117_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_117_LPI_SRPD_DEEP_WAKEUP_F2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_117_LPI_SRPD_DEEP_WAKEUP_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_117_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_117_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_117_LPI_SRPD_DEEP_MCCLK_GATE_WAKEUP_F2_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_117_LPI_SRPD_DEEP_MCCLK_GATE_WAKEUP_F2_FLDSHFT (24)
+#define HWIO_DDR_CTL_117_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_117_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_118_REGOFF 0x1d8
+#define HWIO_DDR_CTL_118_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_118_REGOFF)
+#define HWIO_DDR_CTL_118_LPI_TIMER_WAKEUP_F2_FLDMASK (0xf)
+#define HWIO_DDR_CTL_118_LPI_TIMER_WAKEUP_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_118_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_118_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_118_LPI_PD_WAKEUP_F3_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_118_LPI_PD_WAKEUP_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_118_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_118_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_118_LPI_SR_WAKEUP_F3_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_118_LPI_SR_WAKEUP_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_118_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_118_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_118_LPI_SR_MCCLK_GATE_WAKEUP_F3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_118_LPI_SR_MCCLK_GATE_WAKEUP_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_118_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_118_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_119_REGOFF 0x1dc
+#define HWIO_DDR_CTL_119_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_119_REGOFF)
+#define HWIO_DDR_CTL_119_LPI_SRPD_LITE_WAKEUP_F3_FLDMASK (0xf)
+#define HWIO_DDR_CTL_119_LPI_SRPD_LITE_WAKEUP_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_119_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_119_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_119_LPI_SRPD_DEEP_WAKEUP_F3_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_119_LPI_SRPD_DEEP_WAKEUP_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_119_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_119_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_119_LPI_SRPD_DEEP_MCCLK_GATE_WAKEUP_F3_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_119_LPI_SRPD_DEEP_MCCLK_GATE_WAKEUP_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_119_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_119_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_119_LPI_TIMER_WAKEUP_F3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_119_LPI_TIMER_WAKEUP_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_119_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_119_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_120_REGOFF 0x1e0
+#define HWIO_DDR_CTL_120_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_120_REGOFF)
+#define HWIO_DDR_CTL_120_LPI_WAKEUP_EN_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_120_LPI_WAKEUP_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_120_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_120_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_120_LPI_TIMER_COUNT_FLDMASK (0xfff00)
+#define HWIO_DDR_CTL_120_LPI_TIMER_COUNT_FLDSHFT (8)
+#define HWIO_DDR_CTL_120_CDNS_INTRL1_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_120_CDNS_INTRL1_FLDSHFT (20)
+#define HWIO_DDR_CTL_120_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_120_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_121_REGOFF 0x1e4
+#define HWIO_DDR_CTL_121_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_121_REGOFF)
+#define HWIO_DDR_CTL_121_LPI_WAKEUP_TIMEOUT_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_121_LPI_WAKEUP_TIMEOUT_FLDSHFT (0)
+#define HWIO_DDR_CTL_121_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_121_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_121_TDFI_LP_RESP_FLDMASK (0x70000)
+#define HWIO_DDR_CTL_121_TDFI_LP_RESP_FLDSHFT (16)
+#define HWIO_DDR_CTL_121_CDNS_INTRL1_FLDMASK (0xf80000)
+#define HWIO_DDR_CTL_121_CDNS_INTRL1_FLDSHFT (19)
+#define HWIO_DDR_CTL_121_LP_STATE_FLDMASK (0x7f000000)
+#define HWIO_DDR_CTL_121_LP_STATE_FLDSHFT (24)
+#define HWIO_DDR_CTL_121_CDNS_INTRL2_FLDMASK (0x80000000)
+#define HWIO_DDR_CTL_121_CDNS_INTRL2_FLDSHFT (31)
+
+#define HWIO_DDR_CTL_122_REGOFF 0x1e8
+#define HWIO_DDR_CTL_122_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_122_REGOFF)
+#define HWIO_DDR_CTL_122_LP_AUTO_ENTRY_EN_FLDMASK (0xf)
+#define HWIO_DDR_CTL_122_LP_AUTO_ENTRY_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_122_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_122_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_122_LP_AUTO_EXIT_EN_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_122_LP_AUTO_EXIT_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_122_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_122_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_122_LP_AUTO_MEM_GATE_EN_FLDMASK (0x70000)
+#define HWIO_DDR_CTL_122_LP_AUTO_MEM_GATE_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_122_CDNS_INTRL2_FLDMASK (0xf80000)
+#define HWIO_DDR_CTL_122_CDNS_INTRL2_FLDSHFT (19)
+#define HWIO_DDR_CTL_122_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_122_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_123_REGOFF 0x1ec
+#define HWIO_DDR_CTL_123_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_123_REGOFF)
+#define HWIO_DDR_CTL_123_LP_AUTO_PD_IDLE_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_123_LP_AUTO_PD_IDLE_FLDSHFT (0)
+#define HWIO_DDR_CTL_123_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_123_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_123_LP_AUTO_SRPD_LITE_IDLE_FLDMASK (0xfff0000)
+#define HWIO_DDR_CTL_123_LP_AUTO_SRPD_LITE_IDLE_FLDSHFT (16)
+#define HWIO_DDR_CTL_123_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_123_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_124_REGOFF 0x1f0
+#define HWIO_DDR_CTL_124_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_124_REGOFF)
+#define HWIO_DDR_CTL_124_LP_AUTO_SR_IDLE_FLDMASK (0xff)
+#define HWIO_DDR_CTL_124_LP_AUTO_SR_IDLE_FLDSHFT (0)
+#define HWIO_DDR_CTL_124_LP_AUTO_SR_MC_GATE_IDLE_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_124_LP_AUTO_SR_MC_GATE_IDLE_FLDSHFT (8)
+#define HWIO_DDR_CTL_124_HW_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_124_HW_PROMOTE_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_125_REGOFF 0x1f4
+#define HWIO_DDR_CTL_125_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_125_REGOFF)
+#define HWIO_DDR_CTL_125_HW_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_125_HW_PROMOTE_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_125_HW_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_125_HW_PROMOTE_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_126_REGOFF 0x1f8
+#define HWIO_DDR_CTL_126_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_126_REGOFF)
+#define HWIO_DDR_CTL_126_HW_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_126_HW_PROMOTE_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_126_LPC_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_126_LPC_PROMOTE_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_127_REGOFF 0x1fc
+#define HWIO_DDR_CTL_127_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_127_REGOFF)
+#define HWIO_DDR_CTL_127_LPC_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_127_LPC_PROMOTE_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_127_LPC_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_127_LPC_PROMOTE_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_128_REGOFF 0x200
+#define HWIO_DDR_CTL_128_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_128_REGOFF)
+#define HWIO_DDR_CTL_128_LPC_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_128_LPC_PROMOTE_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_128_LPC_SR_CTRLUPD_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_128_LPC_SR_CTRLUPD_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_128_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_128_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_CTL_128_LPC_SR_PHYUPD_EN_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_128_LPC_SR_PHYUPD_EN_FLDSHFT (24)
+#define HWIO_DDR_CTL_128_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_128_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_129_REGOFF 0x204
+#define HWIO_DDR_CTL_129_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_129_REGOFF)
+#define HWIO_DDR_CTL_129_LPC_SR_PHYMSTR_EN_FLDMASK (0x1)
+#define HWIO_DDR_CTL_129_LPC_SR_PHYMSTR_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_129_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_129_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_129_CDNS_INTRL1_FLDMASK (0x100)
+#define HWIO_DDR_CTL_129_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_129_CDNS_INTRL2_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_129_CDNS_INTRL2_FLDSHFT (9)
+#define HWIO_DDR_CTL_129_LPC_SR_ZQ_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_129_LPC_SR_ZQ_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_129_RESERVED4_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_129_RESERVED4_FLDSHFT (17)
+#define HWIO_DDR_CTL_129_CDNS_INTRL3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_129_CDNS_INTRL3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_130_REGOFF 0x208
+#define HWIO_DDR_CTL_130_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_130_REGOFF)
+#define HWIO_DDR_CTL_130_DFS_ENABLE_FLDMASK (0x1)
+#define HWIO_DDR_CTL_130_DFS_ENABLE_FLDSHFT (0)
+#define HWIO_DDR_CTL_130_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_130_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_130_TDFI_INIT_START_F0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_130_TDFI_INIT_START_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_130_TDFI_INIT_COMPLETE_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_130_TDFI_INIT_COMPLETE_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_131_REGOFF 0x20c
+#define HWIO_DDR_CTL_131_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_131_REGOFF)
+#define HWIO_DDR_CTL_131_TDFI_INIT_START_F1_FLDMASK (0xff)
+#define HWIO_DDR_CTL_131_TDFI_INIT_START_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_131_TDFI_INIT_COMPLETE_F1_FLDMASK (0xffff00)
+#define HWIO_DDR_CTL_131_TDFI_INIT_COMPLETE_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_131_TDFI_INIT_START_F2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_131_TDFI_INIT_START_F2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_132_REGOFF 0x210
+#define HWIO_DDR_CTL_132_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_132_REGOFF)
+#define HWIO_DDR_CTL_132_TDFI_INIT_COMPLETE_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_132_TDFI_INIT_COMPLETE_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_132_TDFI_INIT_START_F3_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_132_TDFI_INIT_START_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_132_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_132_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_133_REGOFF 0x214
+#define HWIO_DDR_CTL_133_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_133_REGOFF)
+#define HWIO_DDR_CTL_133_TDFI_INIT_COMPLETE_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_133_TDFI_INIT_COMPLETE_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_133_CURRENT_REG_COPY_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_133_CURRENT_REG_COPY_FLDSHFT (16)
+#define HWIO_DDR_CTL_133_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_133_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_133_DFS_PHY_REG_WRITE_EN_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_133_DFS_PHY_REG_WRITE_EN_FLDSHFT (24)
+#define HWIO_DDR_CTL_133_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_133_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_134_REGOFF 0x218
+#define HWIO_DDR_CTL_134_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_134_REGOFF)
+#define HWIO_DDR_CTL_134_DFS_PHY_REG_WRITE_ADDR_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_134_DFS_PHY_REG_WRITE_ADDR_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_135_REGOFF 0x21c
+#define HWIO_DDR_CTL_135_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_135_REGOFF)
+#define HWIO_DDR_CTL_135_DFS_PHY_REG_WRITE_DATA_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_135_DFS_PHY_REG_WRITE_DATA_F0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_136_REGOFF 0x220
+#define HWIO_DDR_CTL_136_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_136_REGOFF)
+#define HWIO_DDR_CTL_136_DFS_PHY_REG_WRITE_DATA_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_136_DFS_PHY_REG_WRITE_DATA_F1_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_137_REGOFF 0x224
+#define HWIO_DDR_CTL_137_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_137_REGOFF)
+#define HWIO_DDR_CTL_137_DFS_PHY_REG_WRITE_DATA_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_137_DFS_PHY_REG_WRITE_DATA_F2_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_138_REGOFF 0x228
+#define HWIO_DDR_CTL_138_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_138_REGOFF)
+#define HWIO_DDR_CTL_138_DFS_PHY_REG_WRITE_DATA_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_138_DFS_PHY_REG_WRITE_DATA_F3_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_139_REGOFF 0x22c
+#define HWIO_DDR_CTL_139_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_139_REGOFF)
+#define HWIO_DDR_CTL_139_DFS_PHY_REG_WRITE_MASK_FLDMASK (0xf)
+#define HWIO_DDR_CTL_139_DFS_PHY_REG_WRITE_MASK_FLDSHFT (0)
+#define HWIO_DDR_CTL_139_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_139_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_139_DFS_PHY_REG_WRITE_WAIT_FLDMASK (0xffff00)
+#define HWIO_DDR_CTL_139_DFS_PHY_REG_WRITE_WAIT_FLDSHFT (8)
+#define HWIO_DDR_CTL_139_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_139_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_140_REGOFF 0x230
+#define HWIO_DDR_CTL_140_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_140_REGOFF)
+#define HWIO_DDR_CTL_140_WRITE_MODEREG_FLDMASK (0x7ffffff)
+#define HWIO_DDR_CTL_140_WRITE_MODEREG_FLDSHFT (0)
+#define HWIO_DDR_CTL_140_RESERVED_FLDMASK (0xf8000000)
+#define HWIO_DDR_CTL_140_RESERVED_FLDSHFT (27)
+
+#define HWIO_DDR_CTL_141_REGOFF 0x234
+#define HWIO_DDR_CTL_141_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_141_REGOFF)
+#define HWIO_DDR_CTL_141_MRW_STATUS_FLDMASK (0xff)
+#define HWIO_DDR_CTL_141_MRW_STATUS_FLDSHFT (0)
+#define HWIO_DDR_CTL_141_READ_MODEREG_FLDMASK (0x1ffff00)
+#define HWIO_DDR_CTL_141_READ_MODEREG_FLDSHFT (8)
+#define HWIO_DDR_CTL_141_RESERVED_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_141_RESERVED_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_142_REGOFF 0x238
+#define HWIO_DDR_CTL_142_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_142_REGOFF)
+#define HWIO_DDR_CTL_142_PERIPHERAL_MRR_DATA_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_142_PERIPHERAL_MRR_DATA_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_143_REGOFF 0x23c
+#define HWIO_DDR_CTL_143_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_143_REGOFF)
+#define HWIO_DDR_CTL_143_PERIPHERAL_MRR_DATA_FLDMASK (0xff)
+#define HWIO_DDR_CTL_143_PERIPHERAL_MRR_DATA_FLDSHFT (0)
+#define HWIO_DDR_CTL_143_AUTO_TEMPCHK_VAL_0_FLDMASK (0xffff00)
+#define HWIO_DDR_CTL_143_AUTO_TEMPCHK_VAL_0_FLDSHFT (8)
+#define HWIO_DDR_CTL_143_DISABLE_UPDATE_TVRCG_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_143_DISABLE_UPDATE_TVRCG_FLDSHFT (24)
+#define HWIO_DDR_CTL_143_RESERVED_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_143_RESERVED_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_144_REGOFF 0x240
+#define HWIO_DDR_CTL_144_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_144_REGOFF)
+#define HWIO_DDR_CTL_144_MRW_DFS_UPDATE_FRC_FLDMASK (0x3)
+#define HWIO_DDR_CTL_144_MRW_DFS_UPDATE_FRC_FLDSHFT (0)
+#define HWIO_DDR_CTL_144_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_CTL_144_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_CTL_144_OBSOLETE1_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_144_OBSOLETE1_FLDSHFT (8)
+#define HWIO_DDR_CTL_144_TVRCG_ENABLE_F0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_144_TVRCG_ENABLE_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_144_CDNS_INTRL1_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_144_CDNS_INTRL1_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_145_REGOFF 0x244
+#define HWIO_DDR_CTL_145_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_145_REGOFF)
+#define HWIO_DDR_CTL_145_TVRCG_DISABLE_F0_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_145_TVRCG_DISABLE_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_145_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_145_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_145_TFC_F0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_145_TFC_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_145_CDNS_INTRL1_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_145_CDNS_INTRL1_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_146_REGOFF 0x248
+#define HWIO_DDR_CTL_146_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_146_REGOFF)
+#define HWIO_DDR_CTL_146_TCKFSPE_F0_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_146_TCKFSPE_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_146_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_146_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_146_TCKFSPX_F0_FLDMASK (0x1f00)
+#define HWIO_DDR_CTL_146_TCKFSPX_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_146_CDNS_INTRL1_FLDMASK (0xe000)
+#define HWIO_DDR_CTL_146_CDNS_INTRL1_FLDSHFT (13)
+#define HWIO_DDR_CTL_146_TVREF_LONG_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_146_TVREF_LONG_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_147_REGOFF 0x24c
+#define HWIO_DDR_CTL_147_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_147_REGOFF)
+#define HWIO_DDR_CTL_147_TVRCG_ENABLE_F1_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_147_TVRCG_ENABLE_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_147_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_147_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_147_TVRCG_DISABLE_F1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_147_TVRCG_DISABLE_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_147_CDNS_INTRL1_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_147_CDNS_INTRL1_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_148_REGOFF 0x250
+#define HWIO_DDR_CTL_148_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_148_REGOFF)
+#define HWIO_DDR_CTL_148_TFC_F1_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_148_TFC_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_148_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_148_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_148_TCKFSPE_F1_FLDMASK (0x1f0000)
+#define HWIO_DDR_CTL_148_TCKFSPE_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_148_CDNS_INTRL1_FLDMASK (0xe00000)
+#define HWIO_DDR_CTL_148_CDNS_INTRL1_FLDSHFT (21)
+#define HWIO_DDR_CTL_148_TCKFSPX_F1_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_148_TCKFSPX_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_148_CDNS_INTRL2_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_148_CDNS_INTRL2_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_149_REGOFF 0x254
+#define HWIO_DDR_CTL_149_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_149_REGOFF)
+#define HWIO_DDR_CTL_149_TVREF_LONG_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_149_TVREF_LONG_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_149_TVRCG_ENABLE_F2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_149_TVRCG_ENABLE_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_149_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_149_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_150_REGOFF 0x258
+#define HWIO_DDR_CTL_150_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_150_REGOFF)
+#define HWIO_DDR_CTL_150_TVRCG_DISABLE_F2_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_150_TVRCG_DISABLE_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_150_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_150_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_150_TFC_F2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_150_TFC_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_150_CDNS_INTRL1_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_150_CDNS_INTRL1_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_151_REGOFF 0x25c
+#define HWIO_DDR_CTL_151_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_151_REGOFF)
+#define HWIO_DDR_CTL_151_TCKFSPE_F2_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_151_TCKFSPE_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_151_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_151_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_151_TCKFSPX_F2_FLDMASK (0x1f00)
+#define HWIO_DDR_CTL_151_TCKFSPX_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_151_CDNS_INTRL1_FLDMASK (0xe000)
+#define HWIO_DDR_CTL_151_CDNS_INTRL1_FLDSHFT (13)
+#define HWIO_DDR_CTL_151_TVREF_LONG_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_151_TVREF_LONG_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_152_REGOFF 0x260
+#define HWIO_DDR_CTL_152_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_152_REGOFF)
+#define HWIO_DDR_CTL_152_TVRCG_ENABLE_F3_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_152_TVRCG_ENABLE_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_152_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_152_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_152_TVRCG_DISABLE_F3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_152_TVRCG_DISABLE_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_152_CDNS_INTRL1_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_152_CDNS_INTRL1_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_153_REGOFF 0x264
+#define HWIO_DDR_CTL_153_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_153_REGOFF)
+#define HWIO_DDR_CTL_153_TFC_F3_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_153_TFC_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_153_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_153_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_153_TCKFSPE_F3_FLDMASK (0x1f0000)
+#define HWIO_DDR_CTL_153_TCKFSPE_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_153_CDNS_INTRL1_FLDMASK (0xe00000)
+#define HWIO_DDR_CTL_153_CDNS_INTRL1_FLDSHFT (21)
+#define HWIO_DDR_CTL_153_TCKFSPX_F3_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_153_TCKFSPX_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_153_CDNS_INTRL2_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_153_CDNS_INTRL2_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_154_REGOFF 0x268
+#define HWIO_DDR_CTL_154_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_154_REGOFF)
+#define HWIO_DDR_CTL_154_TVREF_LONG_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_154_TVREF_LONG_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_154_MRR_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_154_MRR_PROMOTE_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_155_REGOFF 0x26c
+#define HWIO_DDR_CTL_155_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_155_REGOFF)
+#define HWIO_DDR_CTL_155_MRR_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_155_MRR_PROMOTE_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_155_MRR_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_155_MRR_PROMOTE_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_156_REGOFF 0x270
+#define HWIO_DDR_CTL_156_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_156_REGOFF)
+#define HWIO_DDR_CTL_156_MRR_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_156_MRR_PROMOTE_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_156_MRW_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_156_MRW_PROMOTE_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_157_REGOFF 0x274
+#define HWIO_DDR_CTL_157_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_157_REGOFF)
+#define HWIO_DDR_CTL_157_MRW_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_157_MRW_PROMOTE_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_157_MRW_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_157_MRW_PROMOTE_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_158_REGOFF 0x278
+#define HWIO_DDR_CTL_158_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_158_REGOFF)
+#define HWIO_DDR_CTL_158_MRW_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_158_MRW_PROMOTE_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_158_MR1_DATA_F0_0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_158_MR1_DATA_F0_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_158_MR2_DATA_F0_0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_158_MR2_DATA_F0_0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_159_REGOFF 0x27c
+#define HWIO_DDR_CTL_159_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_159_REGOFF)
+#define HWIO_DDR_CTL_159_MR1_DATA_F1_0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_159_MR1_DATA_F1_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_159_MR2_DATA_F1_0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_159_MR2_DATA_F1_0_FLDSHFT (8)
+#define HWIO_DDR_CTL_159_MR1_DATA_F2_0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_159_MR1_DATA_F2_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_159_MR2_DATA_F2_0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_159_MR2_DATA_F2_0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_160_REGOFF 0x280
+#define HWIO_DDR_CTL_160_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_160_REGOFF)
+#define HWIO_DDR_CTL_160_MR1_DATA_F3_0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_160_MR1_DATA_F3_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_160_MR2_DATA_F3_0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_160_MR2_DATA_F3_0_FLDSHFT (8)
+#define HWIO_DDR_CTL_160_MRSINGLE_DATA_0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_160_MRSINGLE_DATA_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_160_MR3_DATA_F0_0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_160_MR3_DATA_F0_0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_161_REGOFF 0x284
+#define HWIO_DDR_CTL_161_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_161_REGOFF)
+#define HWIO_DDR_CTL_161_MR3_DATA_F1_0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_161_MR3_DATA_F1_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_161_MR3_DATA_F2_0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_161_MR3_DATA_F2_0_FLDSHFT (8)
+#define HWIO_DDR_CTL_161_MR3_DATA_F3_0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_161_MR3_DATA_F3_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_161_MR8_DATA_0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_161_MR8_DATA_0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_162_REGOFF 0x288
+#define HWIO_DDR_CTL_162_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_162_REGOFF)
+#define HWIO_DDR_CTL_162_MR11_DATA_F0_0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_162_MR11_DATA_F0_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_162_MR11_DATA_F1_0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_162_MR11_DATA_F1_0_FLDSHFT (8)
+#define HWIO_DDR_CTL_162_MR11_DATA_F2_0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_162_MR11_DATA_F2_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_162_MR11_DATA_F3_0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_162_MR11_DATA_F3_0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_163_REGOFF 0x28c
+#define HWIO_DDR_CTL_163_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_163_REGOFF)
+#define HWIO_DDR_CTL_163_MR12_DATA_F0_0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_163_MR12_DATA_F0_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_163_MR12_DATA_F1_0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_163_MR12_DATA_F1_0_FLDSHFT (8)
+#define HWIO_DDR_CTL_163_MR12_DATA_F2_0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_163_MR12_DATA_F2_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_163_MR12_DATA_F3_0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_163_MR12_DATA_F3_0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_164_REGOFF 0x290
+#define HWIO_DDR_CTL_164_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_164_REGOFF)
+#define HWIO_DDR_CTL_164_MR13_DATA_0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_164_MR13_DATA_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_164_MR14_DATA_F0_0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_164_MR14_DATA_F0_0_FLDSHFT (8)
+#define HWIO_DDR_CTL_164_MR14_DATA_F1_0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_164_MR14_DATA_F1_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_164_MR14_DATA_F2_0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_164_MR14_DATA_F2_0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_165_REGOFF 0x294
+#define HWIO_DDR_CTL_165_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_165_REGOFF)
+#define HWIO_DDR_CTL_165_MR14_DATA_F3_0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_165_MR14_DATA_F3_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_165_MR_FSP_DATA_VALID_F0_0_FLDMASK (0x100)
+#define HWIO_DDR_CTL_165_MR_FSP_DATA_VALID_F0_0_FLDSHFT (8)
+#define HWIO_DDR_CTL_165_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_165_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_165_MR_FSP_DATA_VALID_F1_0_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_165_MR_FSP_DATA_VALID_F1_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_165_CDNS_INTRL1_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_165_CDNS_INTRL1_FLDSHFT (17)
+#define HWIO_DDR_CTL_165_MR_FSP_DATA_VALID_F2_0_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_165_MR_FSP_DATA_VALID_F2_0_FLDSHFT (24)
+#define HWIO_DDR_CTL_165_CDNS_INTRL2_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_165_CDNS_INTRL2_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_166_REGOFF 0x298
+#define HWIO_DDR_CTL_166_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_166_REGOFF)
+#define HWIO_DDR_CTL_166_MR_FSP_DATA_VALID_F3_0_FLDMASK (0x1)
+#define HWIO_DDR_CTL_166_MR_FSP_DATA_VALID_F3_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_166_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_166_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_166_MR16_DATA_0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_166_MR16_DATA_0_FLDSHFT (8)
+#define HWIO_DDR_CTL_166_MR17_DATA_0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_166_MR17_DATA_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_166_MR20_DATA_0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_166_MR20_DATA_0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_167_REGOFF 0x29c
+#define HWIO_DDR_CTL_167_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_167_REGOFF)
+#define HWIO_DDR_CTL_167_MR22_DATA_F0_0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_167_MR22_DATA_F0_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_167_MR22_DATA_F1_0_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_167_MR22_DATA_F1_0_FLDSHFT (8)
+#define HWIO_DDR_CTL_167_MR22_DATA_F2_0_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_167_MR22_DATA_F2_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_167_MR22_DATA_F3_0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_167_MR22_DATA_F3_0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_168_REGOFF 0x2a0
+#define HWIO_DDR_CTL_168_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_168_REGOFF)
+#define HWIO_DDR_CTL_168_RL3_SUPPORT_EN_FLDMASK (0x1)
+#define HWIO_DDR_CTL_168_RL3_SUPPORT_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_168_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_168_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_168_CDNS_INTRL1_FLDMASK (0x100)
+#define HWIO_DDR_CTL_168_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_168_CDNS_INTRL3_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_168_CDNS_INTRL3_FLDSHFT (9)
+#define HWIO_DDR_CTL_168_CDNS_INTRL2_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_168_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_168_RESERVED4_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_168_RESERVED4_FLDSHFT (17)
+#define HWIO_DDR_CTL_168_FSP_PHY_UPDATE_MRW_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_168_FSP_PHY_UPDATE_MRW_FLDSHFT (24)
+#define HWIO_DDR_CTL_168_RESERVED5_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_168_RESERVED5_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_169_REGOFF 0x2a4
+#define HWIO_DDR_CTL_169_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_169_REGOFF)
+#define HWIO_DDR_CTL_169_DFS_ALWAYS_WRITE_FSP_FLDMASK (0x1)
+#define HWIO_DDR_CTL_169_DFS_ALWAYS_WRITE_FSP_FLDSHFT (0)
+#define HWIO_DDR_CTL_169_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_169_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_169_FSP_STATUS_FLDMASK (0x100)
+#define HWIO_DDR_CTL_169_FSP_STATUS_FLDSHFT (8)
+#define HWIO_DDR_CTL_169_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_169_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_169_FSP_OP_CURRENT_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_169_FSP_OP_CURRENT_FLDSHFT (16)
+#define HWIO_DDR_CTL_169_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_169_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_169_FSP_WR_CURRENT_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_169_FSP_WR_CURRENT_FLDSHFT (24)
+#define HWIO_DDR_CTL_169_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_169_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_170_REGOFF 0x2a8
+#define HWIO_DDR_CTL_170_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_170_REGOFF)
+#define HWIO_DDR_CTL_170_FSP0_FRC_VALID_FLDMASK (0x1)
+#define HWIO_DDR_CTL_170_FSP0_FRC_VALID_FLDSHFT (0)
+#define HWIO_DDR_CTL_170_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_170_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_170_FSP1_FRC_VALID_FLDMASK (0x100)
+#define HWIO_DDR_CTL_170_FSP1_FRC_VALID_FLDSHFT (8)
+#define HWIO_DDR_CTL_170_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_170_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_170_FSP0_FRC_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_170_FSP0_FRC_FLDSHFT (16)
+#define HWIO_DDR_CTL_170_CDNS_INTRL2_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_170_CDNS_INTRL2_FLDSHFT (18)
+#define HWIO_DDR_CTL_170_FSP1_FRC_FLDMASK (0x3000000)
+#define HWIO_DDR_CTL_170_FSP1_FRC_FLDSHFT (24)
+#define HWIO_DDR_CTL_170_CDNS_INTRL3_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_170_CDNS_INTRL3_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_171_REGOFF 0x2ac
+#define HWIO_DDR_CTL_171_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_171_REGOFF)
+#define HWIO_DDR_CTL_171_BIST_GO_FLDMASK (0x1)
+#define HWIO_DDR_CTL_171_BIST_GO_FLDSHFT (0)
+#define HWIO_DDR_CTL_171_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_171_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_171_BIST_RESULT_FLDMASK (0x300)
+#define HWIO_DDR_CTL_171_BIST_RESULT_FLDSHFT (8)
+#define HWIO_DDR_CTL_171_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_171_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_171_ADDR_SPACE_FLDMASK (0x3f0000)
+#define HWIO_DDR_CTL_171_ADDR_SPACE_FLDSHFT (16)
+#define HWIO_DDR_CTL_171_CDNS_INTRL2_FLDMASK (0xc00000)
+#define HWIO_DDR_CTL_171_CDNS_INTRL2_FLDSHFT (22)
+#define HWIO_DDR_CTL_171_BIST_DATA_CHECK_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_171_BIST_DATA_CHECK_FLDSHFT (24)
+#define HWIO_DDR_CTL_171_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_171_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_172_REGOFF 0x2b0
+#define HWIO_DDR_CTL_172_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_172_REGOFF)
+#define HWIO_DDR_CTL_172_BIST_ADDR_CHECK_FLDMASK (0x1)
+#define HWIO_DDR_CTL_172_BIST_ADDR_CHECK_FLDSHFT (0)
+#define HWIO_DDR_CTL_172_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_172_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_172_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_CTL_172_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_CTL_173_REGOFF 0x2b4
+#define HWIO_DDR_CTL_173_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_173_REGOFF)
+#define HWIO_DDR_CTL_173_BIST_START_ADDRESS_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_173_BIST_START_ADDRESS_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_174_REGOFF 0x2b8
+#define HWIO_DDR_CTL_174_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_174_REGOFF)
+#define HWIO_DDR_CTL_174_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_174_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_175_REGOFF 0x2bc
+#define HWIO_DDR_CTL_175_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_175_REGOFF)
+#define HWIO_DDR_CTL_175_BIST_DATA_MASK_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_175_BIST_DATA_MASK_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_176_REGOFF 0x2c0
+#define HWIO_DDR_CTL_176_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_176_REGOFF)
+#define HWIO_DDR_CTL_176_BIST_DATA_MASK_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_176_BIST_DATA_MASK_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_177_REGOFF 0x2c4
+#define HWIO_DDR_CTL_177_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_177_REGOFF)
+#define HWIO_DDR_CTL_177_BIST_TEST_MODE_FLDMASK (0x7)
+#define HWIO_DDR_CTL_177_BIST_TEST_MODE_FLDSHFT (0)
+#define HWIO_DDR_CTL_177_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_177_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_177_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_CTL_177_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_CTL_178_REGOFF 0x2c8
+#define HWIO_DDR_CTL_178_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_178_REGOFF)
+#define HWIO_DDR_CTL_178_BIST_DATA_PATTERN_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_178_BIST_DATA_PATTERN_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_179_REGOFF 0x2cc
+#define HWIO_DDR_CTL_179_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_179_REGOFF)
+#define HWIO_DDR_CTL_179_BIST_DATA_PATTERN_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_179_BIST_DATA_PATTERN_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_180_REGOFF 0x2d0
+#define HWIO_DDR_CTL_180_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_180_REGOFF)
+#define HWIO_DDR_CTL_180_BIST_DATA_PATTERN_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_180_BIST_DATA_PATTERN_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_181_REGOFF 0x2d4
+#define HWIO_DDR_CTL_181_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_181_REGOFF)
+#define HWIO_DDR_CTL_181_BIST_DATA_PATTERN_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_181_BIST_DATA_PATTERN_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_182_REGOFF 0x2d8
+#define HWIO_DDR_CTL_182_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_182_REGOFF)
+#define HWIO_DDR_CTL_182_BIST_RET_STATE_EXIT_FLDMASK (0x1)
+#define HWIO_DDR_CTL_182_BIST_RET_STATE_EXIT_FLDSHFT (0)
+#define HWIO_DDR_CTL_182_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_182_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_182_BIST_RET_STATE_FLDMASK (0x100)
+#define HWIO_DDR_CTL_182_BIST_RET_STATE_FLDSHFT (8)
+#define HWIO_DDR_CTL_182_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_182_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_182_BIST_ERR_STOP_FLDMASK (0xfff0000)
+#define HWIO_DDR_CTL_182_BIST_ERR_STOP_FLDSHFT (16)
+#define HWIO_DDR_CTL_182_CDNS_INTRL2_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_182_CDNS_INTRL2_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_183_REGOFF 0x2dc
+#define HWIO_DDR_CTL_183_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_183_REGOFF)
+#define HWIO_DDR_CTL_183_BIST_ERR_COUNT_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_183_BIST_ERR_COUNT_FLDSHFT (0)
+#define HWIO_DDR_CTL_183_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_183_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_183_LONG_COUNT_MASK_FLDMASK (0x1f0000)
+#define HWIO_DDR_CTL_183_LONG_COUNT_MASK_FLDSHFT (16)
+#define HWIO_DDR_CTL_183_CDNS_INTRL1_FLDMASK (0xe00000)
+#define HWIO_DDR_CTL_183_CDNS_INTRL1_FLDSHFT (21)
+#define HWIO_DDR_CTL_183_AREF_NORM_THRESHOLD_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_183_AREF_NORM_THRESHOLD_FLDSHFT (24)
+#define HWIO_DDR_CTL_183_CDNS_INTRL2_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_183_CDNS_INTRL2_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_184_REGOFF 0x2e0
+#define HWIO_DDR_CTL_184_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_184_REGOFF)
+#define HWIO_DDR_CTL_184_AREF_HIGH_THRESHOLD_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_184_AREF_HIGH_THRESHOLD_FLDSHFT (0)
+#define HWIO_DDR_CTL_184_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_184_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_184_AREF_MAX_DEFICIT_FLDMASK (0x1f00)
+#define HWIO_DDR_CTL_184_AREF_MAX_DEFICIT_FLDSHFT (8)
+#define HWIO_DDR_CTL_184_CDNS_INTRL1_FLDMASK (0xe000)
+#define HWIO_DDR_CTL_184_CDNS_INTRL1_FLDSHFT (13)
+#define HWIO_DDR_CTL_184_AREF_MAX_CREDIT_FLDMASK (0x1f0000)
+#define HWIO_DDR_CTL_184_AREF_MAX_CREDIT_FLDSHFT (16)
+#define HWIO_DDR_CTL_184_CDNS_INTRL2_FLDMASK (0xe00000)
+#define HWIO_DDR_CTL_184_CDNS_INTRL2_FLDSHFT (21)
+#define HWIO_DDR_CTL_184_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_184_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_185_REGOFF 0x2e4
+#define HWIO_DDR_CTL_185_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_185_REGOFF)
+#define HWIO_DDR_CTL_185_ZQ_CALSTART_NORM_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_185_ZQ_CALSTART_NORM_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_185_ZQ_CALSTART_HIGH_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_185_ZQ_CALSTART_HIGH_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_186_REGOFF 0x2e8
+#define HWIO_DDR_CTL_186_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_186_REGOFF)
+#define HWIO_DDR_CTL_186_ZQ_CALLATCH_HIGH_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_186_ZQ_CALLATCH_HIGH_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_186_ZQ_CS_NORM_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_186_ZQ_CS_NORM_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_187_REGOFF 0x2ec
+#define HWIO_DDR_CTL_187_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_187_REGOFF)
+#define HWIO_DDR_CTL_187_ZQ_CS_HIGH_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_187_ZQ_CS_HIGH_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_187_ZQ_CALSTART_TIMEOUT_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_187_ZQ_CALSTART_TIMEOUT_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_188_REGOFF 0x2f0
+#define HWIO_DDR_CTL_188_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_188_REGOFF)
+#define HWIO_DDR_CTL_188_ZQ_CALLATCH_TIMEOUT_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_188_ZQ_CALLATCH_TIMEOUT_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_188_ZQ_CS_TIMEOUT_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_188_ZQ_CS_TIMEOUT_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_189_REGOFF 0x2f4
+#define HWIO_DDR_CTL_189_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_189_REGOFF)
+#define HWIO_DDR_CTL_189_ZQ_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_189_ZQ_PROMOTE_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_189_ZQ_CALSTART_NORM_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_189_ZQ_CALSTART_NORM_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_190_REGOFF 0x2f8
+#define HWIO_DDR_CTL_190_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_190_REGOFF)
+#define HWIO_DDR_CTL_190_ZQ_CALSTART_HIGH_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_190_ZQ_CALSTART_HIGH_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_190_ZQ_CALLATCH_HIGH_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_190_ZQ_CALLATCH_HIGH_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_191_REGOFF 0x2fc
+#define HWIO_DDR_CTL_191_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_191_REGOFF)
+#define HWIO_DDR_CTL_191_ZQ_CS_NORM_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_191_ZQ_CS_NORM_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_191_ZQ_CS_HIGH_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_191_ZQ_CS_HIGH_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_192_REGOFF 0x300
+#define HWIO_DDR_CTL_192_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_192_REGOFF)
+#define HWIO_DDR_CTL_192_ZQ_CALSTART_TIMEOUT_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_192_ZQ_CALSTART_TIMEOUT_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_192_ZQ_CALLATCH_TIMEOUT_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_192_ZQ_CALLATCH_TIMEOUT_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_193_REGOFF 0x304
+#define HWIO_DDR_CTL_193_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_193_REGOFF)
+#define HWIO_DDR_CTL_193_ZQ_CS_TIMEOUT_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_193_ZQ_CS_TIMEOUT_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_193_ZQ_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_193_ZQ_PROMOTE_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_194_REGOFF 0x308
+#define HWIO_DDR_CTL_194_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_194_REGOFF)
+#define HWIO_DDR_CTL_194_ZQ_CALSTART_NORM_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_194_ZQ_CALSTART_NORM_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_194_ZQ_CALSTART_HIGH_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_194_ZQ_CALSTART_HIGH_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_195_REGOFF 0x30c
+#define HWIO_DDR_CTL_195_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_195_REGOFF)
+#define HWIO_DDR_CTL_195_ZQ_CALLATCH_HIGH_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_195_ZQ_CALLATCH_HIGH_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_195_ZQ_CS_NORM_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_195_ZQ_CS_NORM_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_196_REGOFF 0x310
+#define HWIO_DDR_CTL_196_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_196_REGOFF)
+#define HWIO_DDR_CTL_196_ZQ_CS_HIGH_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_196_ZQ_CS_HIGH_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_196_ZQ_CALSTART_TIMEOUT_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_196_ZQ_CALSTART_TIMEOUT_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_197_REGOFF 0x314
+#define HWIO_DDR_CTL_197_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_197_REGOFF)
+#define HWIO_DDR_CTL_197_ZQ_CALLATCH_TIMEOUT_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_197_ZQ_CALLATCH_TIMEOUT_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_197_ZQ_CS_TIMEOUT_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_197_ZQ_CS_TIMEOUT_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_198_REGOFF 0x318
+#define HWIO_DDR_CTL_198_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_198_REGOFF)
+#define HWIO_DDR_CTL_198_ZQ_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_198_ZQ_PROMOTE_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_198_ZQ_CALSTART_NORM_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_198_ZQ_CALSTART_NORM_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_199_REGOFF 0x31c
+#define HWIO_DDR_CTL_199_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_199_REGOFF)
+#define HWIO_DDR_CTL_199_ZQ_CALSTART_HIGH_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_199_ZQ_CALSTART_HIGH_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_199_ZQ_CALLATCH_HIGH_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_199_ZQ_CALLATCH_HIGH_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_200_REGOFF 0x320
+#define HWIO_DDR_CTL_200_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_200_REGOFF)
+#define HWIO_DDR_CTL_200_ZQ_CS_NORM_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_200_ZQ_CS_NORM_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_200_ZQ_CS_HIGH_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_200_ZQ_CS_HIGH_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_201_REGOFF 0x324
+#define HWIO_DDR_CTL_201_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_201_REGOFF)
+#define HWIO_DDR_CTL_201_ZQ_CALSTART_TIMEOUT_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_201_ZQ_CALSTART_TIMEOUT_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_201_ZQ_CALLATCH_TIMEOUT_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_201_ZQ_CALLATCH_TIMEOUT_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_202_REGOFF 0x328
+#define HWIO_DDR_CTL_202_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_202_REGOFF)
+#define HWIO_DDR_CTL_202_ZQ_CS_TIMEOUT_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_202_ZQ_CS_TIMEOUT_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_202_ZQ_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_202_ZQ_PROMOTE_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_203_REGOFF 0x32c
+#define HWIO_DDR_CTL_203_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_203_REGOFF)
+#define HWIO_DDR_CTL_203_CDNS_INTRL0_FLDMASK (0x7)
+#define HWIO_DDR_CTL_203_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_203_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_203_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_203_ZQINIT_F0_FLDMASK (0xfff00)
+#define HWIO_DDR_CTL_203_ZQINIT_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_203_CDNS_INTRL1_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_203_CDNS_INTRL1_FLDSHFT (20)
+#define HWIO_DDR_CTL_203_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_203_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_204_REGOFF 0x330
+#define HWIO_DDR_CTL_204_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_204_REGOFF)
+#define HWIO_DDR_CTL_204_ZQCL_F0_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_204_ZQCL_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_204_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_204_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_204_ZQCS_F0_FLDMASK (0xfff0000)
+#define HWIO_DDR_CTL_204_ZQCS_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_204_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_204_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_205_REGOFF 0x334
+#define HWIO_DDR_CTL_205_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_205_REGOFF)
+#define HWIO_DDR_CTL_205_TZQCAL_F0_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_205_TZQCAL_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_205_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_205_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_205_TZQLAT_F0_FLDMASK (0x3f0000)
+#define HWIO_DDR_CTL_205_TZQLAT_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_205_CDNS_INTRL1_FLDMASK (0xc00000)
+#define HWIO_DDR_CTL_205_CDNS_INTRL1_FLDSHFT (22)
+#define HWIO_DDR_CTL_205_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_205_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_206_REGOFF 0x338
+#define HWIO_DDR_CTL_206_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_206_REGOFF)
+#define HWIO_DDR_CTL_206_ZQINIT_F1_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_206_ZQINIT_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_206_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_206_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_206_ZQCL_F1_FLDMASK (0xfff0000)
+#define HWIO_DDR_CTL_206_ZQCL_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_206_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_206_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_207_REGOFF 0x33c
+#define HWIO_DDR_CTL_207_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_207_REGOFF)
+#define HWIO_DDR_CTL_207_ZQCS_F1_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_207_ZQCS_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_207_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_207_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_207_TZQCAL_F1_FLDMASK (0xfff0000)
+#define HWIO_DDR_CTL_207_TZQCAL_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_207_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_207_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_208_REGOFF 0x340
+#define HWIO_DDR_CTL_208_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_208_REGOFF)
+#define HWIO_DDR_CTL_208_TZQLAT_F1_FLDMASK (0x3f)
+#define HWIO_DDR_CTL_208_TZQLAT_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_208_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_CTL_208_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_CTL_208_ZQINIT_F2_FLDMASK (0xfff00)
+#define HWIO_DDR_CTL_208_ZQINIT_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_208_CDNS_INTRL1_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_208_CDNS_INTRL1_FLDSHFT (20)
+#define HWIO_DDR_CTL_208_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_208_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_209_REGOFF 0x344
+#define HWIO_DDR_CTL_209_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_209_REGOFF)
+#define HWIO_DDR_CTL_209_ZQCL_F2_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_209_ZQCL_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_209_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_209_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_209_ZQCS_F2_FLDMASK (0xfff0000)
+#define HWIO_DDR_CTL_209_ZQCS_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_209_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_209_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_210_REGOFF 0x348
+#define HWIO_DDR_CTL_210_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_210_REGOFF)
+#define HWIO_DDR_CTL_210_TZQCAL_F2_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_210_TZQCAL_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_210_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_210_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_210_TZQLAT_F2_FLDMASK (0x3f0000)
+#define HWIO_DDR_CTL_210_TZQLAT_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_210_CDNS_INTRL1_FLDMASK (0xc00000)
+#define HWIO_DDR_CTL_210_CDNS_INTRL1_FLDSHFT (22)
+#define HWIO_DDR_CTL_210_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_210_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_211_REGOFF 0x34c
+#define HWIO_DDR_CTL_211_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_211_REGOFF)
+#define HWIO_DDR_CTL_211_ZQINIT_F3_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_211_ZQINIT_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_211_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_211_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_211_ZQCL_F3_FLDMASK (0xfff0000)
+#define HWIO_DDR_CTL_211_ZQCL_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_211_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_211_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_212_REGOFF 0x350
+#define HWIO_DDR_CTL_212_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_212_REGOFF)
+#define HWIO_DDR_CTL_212_ZQCS_F3_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_212_ZQCS_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_212_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_212_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_212_TZQCAL_F3_FLDMASK (0xfff0000)
+#define HWIO_DDR_CTL_212_TZQCAL_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_212_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_212_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_213_REGOFF 0x354
+#define HWIO_DDR_CTL_213_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_213_REGOFF)
+#define HWIO_DDR_CTL_213_TZQLAT_F3_FLDMASK (0x3f)
+#define HWIO_DDR_CTL_213_TZQLAT_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_213_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_CTL_213_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_CTL_213_ZQ_REQ_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_213_ZQ_REQ_FLDSHFT (8)
+#define HWIO_DDR_CTL_213_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_213_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_213_ZQ_REQ_PENDING_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_213_ZQ_REQ_PENDING_FLDSHFT (16)
+#define HWIO_DDR_CTL_213_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_213_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_213_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_213_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_214_REGOFF 0x358
+#define HWIO_DDR_CTL_214_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_214_REGOFF)
+#define HWIO_DDR_CTL_214_ZQRESET_F0_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_214_ZQRESET_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_214_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_214_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_214_ZQRESET_F1_FLDMASK (0xfff0000)
+#define HWIO_DDR_CTL_214_ZQRESET_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_214_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_214_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_215_REGOFF 0x35c
+#define HWIO_DDR_CTL_215_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_215_REGOFF)
+#define HWIO_DDR_CTL_215_ZQRESET_F2_FLDMASK (0xfff)
+#define HWIO_DDR_CTL_215_ZQRESET_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_215_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_215_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_CTL_215_ZQRESET_F3_FLDMASK (0xfff0000)
+#define HWIO_DDR_CTL_215_ZQRESET_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_215_CDNS_INTRL1_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_215_CDNS_INTRL1_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_216_REGOFF 0x360
+#define HWIO_DDR_CTL_216_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_216_REGOFF)
+#define HWIO_DDR_CTL_216_NO_ZQ_INIT_FLDMASK (0x1)
+#define HWIO_DDR_CTL_216_NO_ZQ_INIT_FLDSHFT (0)
+#define HWIO_DDR_CTL_216_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_216_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_216_BANK_DIFF_FLDMASK (0x300)
+#define HWIO_DDR_CTL_216_BANK_DIFF_FLDSHFT (8)
+#define HWIO_DDR_CTL_216_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_216_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_216_ROW_DIFF_FLDMASK (0x70000)
+#define HWIO_DDR_CTL_216_ROW_DIFF_FLDSHFT (16)
+#define HWIO_DDR_CTL_216_CDNS_INTRL2_FLDMASK (0xf80000)
+#define HWIO_DDR_CTL_216_CDNS_INTRL2_FLDSHFT (19)
+#define HWIO_DDR_CTL_216_COL_DIFF_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_216_COL_DIFF_FLDSHFT (24)
+#define HWIO_DDR_CTL_216_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_216_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_217_REGOFF 0x364
+#define HWIO_DDR_CTL_217_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_217_REGOFF)
+#define HWIO_DDR_CTL_217_BANK_START_BIT_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_217_BANK_START_BIT_FLDSHFT (0)
+#define HWIO_DDR_CTL_217_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_217_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_217_BANK_ADDR_INTLV_EN_FLDMASK (0x100)
+#define HWIO_DDR_CTL_217_BANK_ADDR_INTLV_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_217_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_217_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_217_APREBIT_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_217_APREBIT_FLDSHFT (16)
+#define HWIO_DDR_CTL_217_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_217_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_217_AGE_COUNT_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_217_AGE_COUNT_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_218_REGOFF 0x368
+#define HWIO_DDR_CTL_218_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_218_REGOFF)
+#define HWIO_DDR_CTL_218_COMMAND_AGE_COUNT_FLDMASK (0xff)
+#define HWIO_DDR_CTL_218_COMMAND_AGE_COUNT_FLDSHFT (0)
+#define HWIO_DDR_CTL_218_ADDR_CMP_EN_FLDMASK (0x100)
+#define HWIO_DDR_CTL_218_ADDR_CMP_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_218_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_218_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_218_CDNS_INTRL2_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_218_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_218_CDNS_INTRL1_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_218_CDNS_INTRL1_FLDSHFT (17)
+#define HWIO_DDR_CTL_218_BANK_SPLIT_EN_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_218_BANK_SPLIT_EN_FLDSHFT (24)
+#define HWIO_DDR_CTL_218_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_218_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_219_REGOFF 0x36c
+#define HWIO_DDR_CTL_219_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_219_REGOFF)
+#define HWIO_DDR_CTL_219_PLACEMENT_EN_FLDMASK (0x1)
+#define HWIO_DDR_CTL_219_PLACEMENT_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_219_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_219_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_219_PRIORITY_EN_FLDMASK (0x100)
+#define HWIO_DDR_CTL_219_PRIORITY_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_219_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_219_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_219_RW_SAME_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_219_RW_SAME_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_219_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_219_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_219_RW_SAME_PAGE_EN_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_219_RW_SAME_PAGE_EN_FLDSHFT (24)
+#define HWIO_DDR_CTL_219_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_219_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_220_REGOFF 0x370
+#define HWIO_DDR_CTL_220_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_220_REGOFF)
+#define HWIO_DDR_CTL_220_DISABLE_RW_GROUP_W_BNK_CONFLICT_FLDMASK (0x3)
+#define HWIO_DDR_CTL_220_DISABLE_RW_GROUP_W_BNK_CONFLICT_FLDSHFT (0)
+#define HWIO_DDR_CTL_220_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_CTL_220_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_CTL_220_NUM_Q_ENTRIES_ACT_DISABLE_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_220_NUM_Q_ENTRIES_ACT_DISABLE_FLDSHFT (8)
+#define HWIO_DDR_CTL_220_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_220_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_220_SWAP_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_220_SWAP_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_220_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_220_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_220_DISABLE_RD_INTERLEAVE_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_220_DISABLE_RD_INTERLEAVE_FLDSHFT (24)
+#define HWIO_DDR_CTL_220_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_220_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_221_REGOFF 0x374
+#define HWIO_DDR_CTL_221_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_221_REGOFF)
+#define HWIO_DDR_CTL_221_INHIBIT_DRAM_CMD_FLDMASK (0x3)
+#define HWIO_DDR_CTL_221_INHIBIT_DRAM_CMD_FLDSHFT (0)
+#define HWIO_DDR_CTL_221_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_CTL_221_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_CTL_221_REDUC_FLDMASK (0x100)
+#define HWIO_DDR_CTL_221_REDUC_FLDSHFT (8)
+#define HWIO_DDR_CTL_221_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_221_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_221_MEMDATA_RATIO_0_FLDMASK (0x70000)
+#define HWIO_DDR_CTL_221_MEMDATA_RATIO_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_221_CDNS_INTRL2_FLDMASK (0xf80000)
+#define HWIO_DDR_CTL_221_CDNS_INTRL2_FLDSHFT (19)
+#define HWIO_DDR_CTL_221_CDNS_INTRL3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_221_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_221_RESERVED4_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_221_RESERVED4_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_222_REGOFF 0x378
+#define HWIO_DDR_CTL_222_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_222_REGOFF)
+#define HWIO_DDR_CTL_222_CDNS_INTRL0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_222_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_222_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_222_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_222_CDNS_INTRL1_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_222_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_222_CDNS_INTRL3_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_222_CDNS_INTRL3_FLDSHFT (12)
+#define HWIO_DDR_CTL_222_CDNS_INTRL2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_222_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_222_RESERVED4_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_222_RESERVED4_FLDSHFT (20)
+#define HWIO_DDR_CTL_222_Q_FULLNESS_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_222_Q_FULLNESS_FLDSHFT (24)
+#define HWIO_DDR_CTL_222_RESERVED5_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_222_RESERVED5_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_223_REGOFF 0x37c
+#define HWIO_DDR_CTL_223_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_223_REGOFF)
+#define HWIO_DDR_CTL_223_IN_ORDER_ACCEPT_FLDMASK (0x1)
+#define HWIO_DDR_CTL_223_IN_ORDER_ACCEPT_FLDSHFT (0)
+#define HWIO_DDR_CTL_223_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_223_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_223_CONTROLLER_BUSY_FLDMASK (0x100)
+#define HWIO_DDR_CTL_223_CONTROLLER_BUSY_FLDSHFT (8)
+#define HWIO_DDR_CTL_223_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_223_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_223_CTRLUPD_REQ_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_223_CTRLUPD_REQ_FLDSHFT (16)
+#define HWIO_DDR_CTL_223_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_223_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_223_CTRLUPD_REQ_PER_AREF_EN_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_223_CTRLUPD_REQ_PER_AREF_EN_FLDSHFT (24)
+#define HWIO_DDR_CTL_223_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_223_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_224_REGOFF 0x380
+#define HWIO_DDR_CTL_224_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_224_REGOFF)
+#define HWIO_DDR_CTL_224_PREAMBLE_SUPPORT_F0_FLDMASK (0x3)
+#define HWIO_DDR_CTL_224_PREAMBLE_SUPPORT_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_224_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_CTL_224_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_CTL_224_PREAMBLE_SUPPORT_F1_FLDMASK (0x300)
+#define HWIO_DDR_CTL_224_PREAMBLE_SUPPORT_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_224_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_224_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_224_PREAMBLE_SUPPORT_F2_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_224_PREAMBLE_SUPPORT_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_224_CDNS_INTRL2_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_224_CDNS_INTRL2_FLDSHFT (18)
+#define HWIO_DDR_CTL_224_PREAMBLE_SUPPORT_F3_FLDMASK (0x3000000)
+#define HWIO_DDR_CTL_224_PREAMBLE_SUPPORT_F3_FLDSHFT (24)
+#define HWIO_DDR_CTL_224_CDNS_INTRL3_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_224_CDNS_INTRL3_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_225_REGOFF 0x384
+#define HWIO_DDR_CTL_225_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_225_REGOFF)
+#define HWIO_DDR_CTL_225_RD_PREAMBLE_TRAINING_EN_FLDMASK (0x1)
+#define HWIO_DDR_CTL_225_RD_PREAMBLE_TRAINING_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_225_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_225_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_225_WR_DBI_EN_FLDMASK (0x100)
+#define HWIO_DDR_CTL_225_WR_DBI_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_225_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_225_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_225_RD_DBI_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_225_RD_DBI_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_225_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_225_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_225_DFI_ERROR_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_225_DFI_ERROR_FLDSHFT (24)
+#define HWIO_DDR_CTL_225_CDNS_INTRL3_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_225_CDNS_INTRL3_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_226_REGOFF 0x388
+#define HWIO_DDR_CTL_226_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_226_REGOFF)
+#define HWIO_DDR_CTL_226_DFI_ERROR_INFO_FLDMASK (0xfffff)
+#define HWIO_DDR_CTL_226_DFI_ERROR_INFO_FLDSHFT (0)
+#define HWIO_DDR_CTL_226_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_226_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_CTL_226_CDNS_INTRL1_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_226_CDNS_INTRL1_FLDSHFT (24)
+#define HWIO_DDR_CTL_226_CDNS_INTRL2_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_226_CDNS_INTRL2_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_227_REGOFF 0x38c
+#define HWIO_DDR_CTL_227_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_227_REGOFF)
+#define HWIO_DDR_CTL_227_INT_STATUS_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_227_INT_STATUS_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_228_REGOFF 0x390
+#define HWIO_DDR_CTL_228_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_228_REGOFF)
+#define HWIO_DDR_CTL_228_INT_STATUS_FLDMASK (0xf)
+#define HWIO_DDR_CTL_228_INT_STATUS_FLDSHFT (0)
+#define HWIO_DDR_CTL_228_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_228_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_228_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_CTL_228_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_CTL_229_REGOFF 0x394
+#define HWIO_DDR_CTL_229_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_229_REGOFF)
+#define HWIO_DDR_CTL_229_INT_ACK_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_229_INT_ACK_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_230_REGOFF 0x398
+#define HWIO_DDR_CTL_230_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_230_REGOFF)
+#define HWIO_DDR_CTL_230_INT_ACK_FLDMASK (0x7)
+#define HWIO_DDR_CTL_230_INT_ACK_FLDSHFT (0)
+#define HWIO_DDR_CTL_230_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_230_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_230_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_CTL_230_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_CTL_231_REGOFF 0x39c
+#define HWIO_DDR_CTL_231_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_231_REGOFF)
+#define HWIO_DDR_CTL_231_INT_MASK_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_231_INT_MASK_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_232_REGOFF 0x3a0
+#define HWIO_DDR_CTL_232_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_232_REGOFF)
+#define HWIO_DDR_CTL_232_INT_MASK_FLDMASK (0xf)
+#define HWIO_DDR_CTL_232_INT_MASK_FLDSHFT (0)
+#define HWIO_DDR_CTL_232_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_232_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_232_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_CTL_232_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_CTL_233_REGOFF 0x3a4
+#define HWIO_DDR_CTL_233_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_233_REGOFF)
+#define HWIO_DDR_CTL_233_OUT_OF_RANGE_ADDR_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_233_OUT_OF_RANGE_ADDR_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_234_REGOFF 0x3a8
+#define HWIO_DDR_CTL_234_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_234_REGOFF)
+#define HWIO_DDR_CTL_234_OBSOLETE0_FLDMASK (0xff)
+#define HWIO_DDR_CTL_234_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_234_OUT_OF_RANGE_LENGTH_FLDMASK (0xfff00)
+#define HWIO_DDR_CTL_234_OUT_OF_RANGE_LENGTH_FLDSHFT (8)
+#define HWIO_DDR_CTL_234_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_234_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_CTL_234_OUT_OF_RANGE_TYPE_FLDMASK (0x7f000000)
+#define HWIO_DDR_CTL_234_OUT_OF_RANGE_TYPE_FLDSHFT (24)
+#define HWIO_DDR_CTL_234_CDNS_INTRL1_FLDMASK (0x80000000)
+#define HWIO_DDR_CTL_234_CDNS_INTRL1_FLDSHFT (31)
+
+#define HWIO_DDR_CTL_235_REGOFF 0x3ac
+#define HWIO_DDR_CTL_235_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_235_REGOFF)
+#define HWIO_DDR_CTL_235_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_235_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_236_REGOFF 0x3b0
+#define HWIO_DDR_CTL_236_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_236_REGOFF)
+#define HWIO_DDR_CTL_236_BIST_EXP_DATA_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_236_BIST_EXP_DATA_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_237_REGOFF 0x3b4
+#define HWIO_DDR_CTL_237_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_237_REGOFF)
+#define HWIO_DDR_CTL_237_BIST_EXP_DATA_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_237_BIST_EXP_DATA_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_238_REGOFF 0x3b8
+#define HWIO_DDR_CTL_238_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_238_REGOFF)
+#define HWIO_DDR_CTL_238_BIST_EXP_DATA_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_238_BIST_EXP_DATA_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_239_REGOFF 0x3bc
+#define HWIO_DDR_CTL_239_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_239_REGOFF)
+#define HWIO_DDR_CTL_239_BIST_EXP_DATA_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_239_BIST_EXP_DATA_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_240_REGOFF 0x3c0
+#define HWIO_DDR_CTL_240_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_240_REGOFF)
+#define HWIO_DDR_CTL_240_BIST_FAIL_DATA_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_240_BIST_FAIL_DATA_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_241_REGOFF 0x3c4
+#define HWIO_DDR_CTL_241_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_241_REGOFF)
+#define HWIO_DDR_CTL_241_BIST_FAIL_DATA_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_241_BIST_FAIL_DATA_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_242_REGOFF 0x3c8
+#define HWIO_DDR_CTL_242_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_242_REGOFF)
+#define HWIO_DDR_CTL_242_BIST_FAIL_DATA_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_242_BIST_FAIL_DATA_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_243_REGOFF 0x3cc
+#define HWIO_DDR_CTL_243_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_243_REGOFF)
+#define HWIO_DDR_CTL_243_BIST_FAIL_DATA_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_243_BIST_FAIL_DATA_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_244_REGOFF 0x3d0
+#define HWIO_DDR_CTL_244_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_244_REGOFF)
+#define HWIO_DDR_CTL_244_BIST_FAIL_ADDR_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_244_BIST_FAIL_ADDR_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_245_REGOFF 0x3d4
+#define HWIO_DDR_CTL_245_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_245_REGOFF)
+#define HWIO_DDR_CTL_245_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_245_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_246_REGOFF 0x3d8
+#define HWIO_DDR_CTL_246_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_246_REGOFF)
+#define HWIO_DDR_CTL_246_PORT_CMD_ERROR_ADDR_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_246_PORT_CMD_ERROR_ADDR_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_247_REGOFF 0x3dc
+#define HWIO_DDR_CTL_247_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_247_REGOFF)
+#define HWIO_DDR_CTL_247_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_247_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_247_PORT_CMD_ERROR_TYPE_FLDMASK (0x70000)
+#define HWIO_DDR_CTL_247_PORT_CMD_ERROR_TYPE_FLDSHFT (16)
+#define HWIO_DDR_CTL_247_RESERVED_FLDMASK (0xf80000)
+#define HWIO_DDR_CTL_247_RESERVED_FLDSHFT (19)
+#define HWIO_DDR_CTL_247_TODTL_2CMD_F0_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_247_TODTL_2CMD_F0_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_248_REGOFF 0x3e0
+#define HWIO_DDR_CTL_248_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_248_REGOFF)
+#define HWIO_DDR_CTL_248_TODTH_WR_F0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_248_TODTH_WR_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_248_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_248_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_248_TODTL_2CMD_F1_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_248_TODTL_2CMD_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_248_TODTH_WR_F1_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_248_TODTH_WR_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_248_CDNS_INTRL1_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_248_CDNS_INTRL1_FLDSHFT (20)
+#define HWIO_DDR_CTL_248_TODTL_2CMD_F2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_248_TODTL_2CMD_F2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_249_REGOFF 0x3e4
+#define HWIO_DDR_CTL_249_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_249_REGOFF)
+#define HWIO_DDR_CTL_249_TODTH_WR_F2_FLDMASK (0xf)
+#define HWIO_DDR_CTL_249_TODTH_WR_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_249_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_249_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_249_TODTL_2CMD_F3_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_249_TODTL_2CMD_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_249_TODTH_WR_F3_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_249_TODTH_WR_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_249_CDNS_INTRL1_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_249_CDNS_INTRL1_FLDSHFT (20)
+#define HWIO_DDR_CTL_249_ODT_EN_F0_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_249_ODT_EN_F0_FLDSHFT (24)
+#define HWIO_DDR_CTL_249_CDNS_INTRL2_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_249_CDNS_INTRL2_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_250_REGOFF 0x3e8
+#define HWIO_DDR_CTL_250_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_250_REGOFF)
+#define HWIO_DDR_CTL_250_ODT_EN_F1_FLDMASK (0x1)
+#define HWIO_DDR_CTL_250_ODT_EN_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_250_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_250_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_250_ODT_EN_F2_FLDMASK (0x100)
+#define HWIO_DDR_CTL_250_ODT_EN_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_250_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_250_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_250_ODT_EN_F3_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_250_ODT_EN_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_250_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_250_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_250_EN_ODT_ASSERT_EXCEPT_RD_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_250_EN_ODT_ASSERT_EXCEPT_RD_FLDSHFT (24)
+#define HWIO_DDR_CTL_250_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_250_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_251_REGOFF 0x3ec
+#define HWIO_DDR_CTL_251_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_251_REGOFF)
+#define HWIO_DDR_CTL_251_CTLR_DISABLE_ODT_ON_ZQ_FLDMASK (0x1)
+#define HWIO_DDR_CTL_251_CTLR_DISABLE_ODT_ON_ZQ_FLDSHFT (0)
+#define HWIO_DDR_CTL_251_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_251_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_251_CDNS_INTRL1_FLDMASK (0x3f00)
+#define HWIO_DDR_CTL_251_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_251_CDNS_INTRL2_FLDMASK (0xc000)
+#define HWIO_DDR_CTL_251_CDNS_INTRL2_FLDSHFT (14)
+#define HWIO_DDR_CTL_251_RW2MRW_DLY_F0_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_251_RW2MRW_DLY_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_251_CDNS_INTRL3_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_251_CDNS_INTRL3_FLDSHFT (20)
+#define HWIO_DDR_CTL_251_RW2MRW_DLY_F1_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_251_RW2MRW_DLY_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_251_RESERVED4_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_251_RESERVED4_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_252_REGOFF 0x3f0
+#define HWIO_DDR_CTL_252_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_252_REGOFF)
+#define HWIO_DDR_CTL_252_RW2MRW_DLY_F2_FLDMASK (0xf)
+#define HWIO_DDR_CTL_252_RW2MRW_DLY_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_252_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_252_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_252_RW2MRW_DLY_F3_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_252_RW2MRW_DLY_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_252_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_252_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_252_R2R_SAMECS_DLY_FLDMASK (0x1f0000)
+#define HWIO_DDR_CTL_252_R2R_SAMECS_DLY_FLDSHFT (16)
+#define HWIO_DDR_CTL_252_CDNS_INTRL2_FLDMASK (0xe00000)
+#define HWIO_DDR_CTL_252_CDNS_INTRL2_FLDSHFT (21)
+#define HWIO_DDR_CTL_252_R2W_SAMECS_DLY_F0_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_252_R2W_SAMECS_DLY_F0_FLDSHFT (24)
+#define HWIO_DDR_CTL_252_CDNS_INTRL3_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_252_CDNS_INTRL3_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_253_REGOFF 0x3f4
+#define HWIO_DDR_CTL_253_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_253_REGOFF)
+#define HWIO_DDR_CTL_253_R2W_SAMECS_DLY_F1_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_253_R2W_SAMECS_DLY_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_253_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_253_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_253_R2W_SAMECS_DLY_F2_FLDMASK (0x1f00)
+#define HWIO_DDR_CTL_253_R2W_SAMECS_DLY_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_253_CDNS_INTRL1_FLDMASK (0xe000)
+#define HWIO_DDR_CTL_253_CDNS_INTRL1_FLDSHFT (13)
+#define HWIO_DDR_CTL_253_R2W_SAMECS_DLY_F3_FLDMASK (0x1f0000)
+#define HWIO_DDR_CTL_253_R2W_SAMECS_DLY_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_253_CDNS_INTRL2_FLDMASK (0xe00000)
+#define HWIO_DDR_CTL_253_CDNS_INTRL2_FLDSHFT (21)
+#define HWIO_DDR_CTL_253_W2R_SAMECS_DLY_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_253_W2R_SAMECS_DLY_FLDSHFT (24)
+#define HWIO_DDR_CTL_253_CDNS_INTRL3_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_253_CDNS_INTRL3_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_254_REGOFF 0x3f8
+#define HWIO_DDR_CTL_254_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_254_REGOFF)
+#define HWIO_DDR_CTL_254_W2W_SAMECS_DLY_FLDMASK (0x1f)
+#define HWIO_DDR_CTL_254_W2W_SAMECS_DLY_FLDSHFT (0)
+#define HWIO_DDR_CTL_254_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_CTL_254_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_CTL_254_TDQSCK_MAX_F0_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_254_TDQSCK_MAX_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_254_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_254_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_254_TDQSCK_MAX_F1_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_254_TDQSCK_MAX_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_254_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_254_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_254_TDQSCK_MAX_F2_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_254_TDQSCK_MAX_F2_FLDSHFT (24)
+#define HWIO_DDR_CTL_254_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_254_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_255_REGOFF 0x3fc
+#define HWIO_DDR_CTL_255_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_255_REGOFF)
+#define HWIO_DDR_CTL_255_TDQSCK_MAX_F3_FLDMASK (0xf)
+#define HWIO_DDR_CTL_255_TDQSCK_MAX_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_255_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_255_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_255_SW_LEVELING_MODE_FLDMASK (0x700)
+#define HWIO_DDR_CTL_255_SW_LEVELING_MODE_FLDSHFT (8)
+#define HWIO_DDR_CTL_255_CDNS_INTRL1_FLDMASK (0xf800)
+#define HWIO_DDR_CTL_255_CDNS_INTRL1_FLDSHFT (11)
+#define HWIO_DDR_CTL_255_SWLVL_LOAD_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_255_SWLVL_LOAD_FLDSHFT (16)
+#define HWIO_DDR_CTL_255_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_255_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_255_SWLVL_START_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_255_SWLVL_START_FLDSHFT (24)
+#define HWIO_DDR_CTL_255_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_255_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_256_REGOFF 0x400
+#define HWIO_DDR_CTL_256_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_256_REGOFF)
+#define HWIO_DDR_CTL_256_SWLVL_EXIT_FLDMASK (0x1)
+#define HWIO_DDR_CTL_256_SWLVL_EXIT_FLDSHFT (0)
+#define HWIO_DDR_CTL_256_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_256_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_256_SWLVL_OP_DONE_FLDMASK (0x100)
+#define HWIO_DDR_CTL_256_SWLVL_OP_DONE_FLDSHFT (8)
+#define HWIO_DDR_CTL_256_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_256_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_256_SWLVL_RESP_0_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_256_SWLVL_RESP_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_256_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_256_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_256_SWLVL_RESP_1_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_256_SWLVL_RESP_1_FLDSHFT (24)
+#define HWIO_DDR_CTL_256_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_256_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_257_REGOFF 0x404
+#define HWIO_DDR_CTL_257_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_257_REGOFF)
+#define HWIO_DDR_CTL_257_SWLVL_RESP_2_FLDMASK (0x1)
+#define HWIO_DDR_CTL_257_SWLVL_RESP_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_257_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_257_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_257_SWLVL_RESP_3_FLDMASK (0x100)
+#define HWIO_DDR_CTL_257_SWLVL_RESP_3_FLDSHFT (8)
+#define HWIO_DDR_CTL_257_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_257_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_257_PHYUPD_APPEND_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_257_PHYUPD_APPEND_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_257_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_257_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_257_WRLVL_REQ_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_257_WRLVL_REQ_FLDSHFT (24)
+#define HWIO_DDR_CTL_257_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_257_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_258_REGOFF 0x408
+#define HWIO_DDR_CTL_258_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_258_REGOFF)
+#define HWIO_DDR_CTL_258_WRLVL_CS_FLDMASK (0x1)
+#define HWIO_DDR_CTL_258_WRLVL_CS_FLDSHFT (0)
+#define HWIO_DDR_CTL_258_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_258_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_258_WLDQSEN_FLDMASK (0x3f00)
+#define HWIO_DDR_CTL_258_WLDQSEN_FLDSHFT (8)
+#define HWIO_DDR_CTL_258_CDNS_INTRL1_FLDMASK (0xc000)
+#define HWIO_DDR_CTL_258_CDNS_INTRL1_FLDSHFT (14)
+#define HWIO_DDR_CTL_258_WLMRD_FLDMASK (0x3f0000)
+#define HWIO_DDR_CTL_258_WLMRD_FLDSHFT (16)
+#define HWIO_DDR_CTL_258_CDNS_INTRL2_FLDMASK (0xc00000)
+#define HWIO_DDR_CTL_258_CDNS_INTRL2_FLDSHFT (22)
+#define HWIO_DDR_CTL_258_WRLVL_EN_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_258_WRLVL_EN_FLDSHFT (24)
+#define HWIO_DDR_CTL_258_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_258_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_259_REGOFF 0x40c
+#define HWIO_DDR_CTL_259_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_259_REGOFF)
+#define HWIO_DDR_CTL_259_DFI_PHY_WRLVL_MODE_FLDMASK (0x1)
+#define HWIO_DDR_CTL_259_DFI_PHY_WRLVL_MODE_FLDSHFT (0)
+#define HWIO_DDR_CTL_259_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_259_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_259_WRLVL_PERIODIC_FLDMASK (0x100)
+#define HWIO_DDR_CTL_259_WRLVL_PERIODIC_FLDSHFT (8)
+#define HWIO_DDR_CTL_259_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_259_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_259_WRLVL_ON_SREF_EXIT_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_259_WRLVL_ON_SREF_EXIT_FLDSHFT (16)
+#define HWIO_DDR_CTL_259_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_259_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_259_WRLVL_RESP_MASK_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_259_WRLVL_RESP_MASK_FLDSHFT (24)
+#define HWIO_DDR_CTL_259_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_259_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_260_REGOFF 0x410
+#define HWIO_DDR_CTL_260_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_260_REGOFF)
+#define HWIO_DDR_CTL_260_WRLVL_AREF_EN_FLDMASK (0x1)
+#define HWIO_DDR_CTL_260_WRLVL_AREF_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_260_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_260_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_260_WRLVL_ERROR_STATUS_FLDMASK (0x300)
+#define HWIO_DDR_CTL_260_WRLVL_ERROR_STATUS_FLDSHFT (8)
+#define HWIO_DDR_CTL_260_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_260_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_260_WRLVL_NORM_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_260_WRLVL_NORM_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_261_REGOFF 0x414
+#define HWIO_DDR_CTL_261_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_261_REGOFF)
+#define HWIO_DDR_CTL_261_WRLVL_HIGH_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_261_WRLVL_HIGH_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_261_WRLVL_TIMEOUT_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_261_WRLVL_TIMEOUT_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_262_REGOFF 0x418
+#define HWIO_DDR_CTL_262_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_262_REGOFF)
+#define HWIO_DDR_CTL_262_WRLVL_SW_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_262_WRLVL_SW_PROMOTE_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_262_WRLVL_DFI_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_262_WRLVL_DFI_PROMOTE_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_263_REGOFF 0x41c
+#define HWIO_DDR_CTL_263_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_263_REGOFF)
+#define HWIO_DDR_CTL_263_WRLVL_NORM_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_263_WRLVL_NORM_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_263_WRLVL_HIGH_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_263_WRLVL_HIGH_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_264_REGOFF 0x420
+#define HWIO_DDR_CTL_264_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_264_REGOFF)
+#define HWIO_DDR_CTL_264_WRLVL_TIMEOUT_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_264_WRLVL_TIMEOUT_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_264_WRLVL_SW_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_264_WRLVL_SW_PROMOTE_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_265_REGOFF 0x424
+#define HWIO_DDR_CTL_265_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_265_REGOFF)
+#define HWIO_DDR_CTL_265_WRLVL_DFI_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_265_WRLVL_DFI_PROMOTE_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_265_WRLVL_NORM_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_265_WRLVL_NORM_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_266_REGOFF 0x428
+#define HWIO_DDR_CTL_266_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_266_REGOFF)
+#define HWIO_DDR_CTL_266_WRLVL_HIGH_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_266_WRLVL_HIGH_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_266_WRLVL_TIMEOUT_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_266_WRLVL_TIMEOUT_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_267_REGOFF 0x42c
+#define HWIO_DDR_CTL_267_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_267_REGOFF)
+#define HWIO_DDR_CTL_267_WRLVL_SW_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_267_WRLVL_SW_PROMOTE_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_267_WRLVL_DFI_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_267_WRLVL_DFI_PROMOTE_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_268_REGOFF 0x430
+#define HWIO_DDR_CTL_268_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_268_REGOFF)
+#define HWIO_DDR_CTL_268_WRLVL_NORM_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_268_WRLVL_NORM_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_268_WRLVL_HIGH_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_268_WRLVL_HIGH_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_269_REGOFF 0x434
+#define HWIO_DDR_CTL_269_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_269_REGOFF)
+#define HWIO_DDR_CTL_269_WRLVL_TIMEOUT_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_269_WRLVL_TIMEOUT_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_269_WRLVL_SW_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_269_WRLVL_SW_PROMOTE_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_270_REGOFF 0x438
+#define HWIO_DDR_CTL_270_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_270_REGOFF)
+#define HWIO_DDR_CTL_270_WRLVL_DFI_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_270_WRLVL_DFI_PROMOTE_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_270_RDLVL_REQ_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_270_RDLVL_REQ_FLDSHFT (16)
+#define HWIO_DDR_CTL_270_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_270_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_CTL_270_RDLVL_GATE_REQ_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_270_RDLVL_GATE_REQ_FLDSHFT (24)
+#define HWIO_DDR_CTL_270_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_270_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_271_REGOFF 0x43c
+#define HWIO_DDR_CTL_271_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_271_REGOFF)
+#define HWIO_DDR_CTL_271_RDLVL_CS_FLDMASK (0x1)
+#define HWIO_DDR_CTL_271_RDLVL_CS_FLDSHFT (0)
+#define HWIO_DDR_CTL_271_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_271_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_271_RDLVL_SEQ_EN_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_271_RDLVL_SEQ_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_271_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_271_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_271_RDLVL_GATE_SEQ_EN_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_271_RDLVL_GATE_SEQ_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_271_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_271_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_271_DFI_PHY_RDLVL_MODE_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_271_DFI_PHY_RDLVL_MODE_FLDSHFT (24)
+#define HWIO_DDR_CTL_271_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_271_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_272_REGOFF 0x440
+#define HWIO_DDR_CTL_272_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_272_REGOFF)
+#define HWIO_DDR_CTL_272_DFI_PHY_RDLVL_GATE_MODE_FLDMASK (0x1)
+#define HWIO_DDR_CTL_272_DFI_PHY_RDLVL_GATE_MODE_FLDSHFT (0)
+#define HWIO_DDR_CTL_272_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_272_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_272_RDLVL_PERIODIC_FLDMASK (0x100)
+#define HWIO_DDR_CTL_272_RDLVL_PERIODIC_FLDSHFT (8)
+#define HWIO_DDR_CTL_272_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_272_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_272_RDLVL_ON_SREF_EXIT_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_272_RDLVL_ON_SREF_EXIT_FLDSHFT (16)
+#define HWIO_DDR_CTL_272_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_272_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_272_RDLVL_GATE_PERIODIC_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_272_RDLVL_GATE_PERIODIC_FLDSHFT (24)
+#define HWIO_DDR_CTL_272_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_272_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_273_REGOFF 0x444
+#define HWIO_DDR_CTL_273_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_273_REGOFF)
+#define HWIO_DDR_CTL_273_RDLVL_GATE_ON_SREF_EXIT_FLDMASK (0x1)
+#define HWIO_DDR_CTL_273_RDLVL_GATE_ON_SREF_EXIT_FLDSHFT (0)
+#define HWIO_DDR_CTL_273_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_273_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_273_RDLVL_AREF_EN_FLDMASK (0x100)
+#define HWIO_DDR_CTL_273_RDLVL_AREF_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_273_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_273_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_273_RDLVL_GATE_AREF_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_273_RDLVL_GATE_AREF_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_273_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_273_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_273_CDNS_INTRL3_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_273_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_273_RESERVED4_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_273_RESERVED4_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_274_REGOFF 0x448
+#define HWIO_DDR_CTL_274_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_274_REGOFF)
+#define HWIO_DDR_CTL_274_RDLVL_NORM_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_274_RDLVL_NORM_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_274_RDLVL_HIGH_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_274_RDLVL_HIGH_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_275_REGOFF 0x44c
+#define HWIO_DDR_CTL_275_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_275_REGOFF)
+#define HWIO_DDR_CTL_275_RDLVL_TIMEOUT_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_275_RDLVL_TIMEOUT_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_275_RDLVL_SW_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_275_RDLVL_SW_PROMOTE_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_276_REGOFF 0x450
+#define HWIO_DDR_CTL_276_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_276_REGOFF)
+#define HWIO_DDR_CTL_276_RDLVL_DFI_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_276_RDLVL_DFI_PROMOTE_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_276_RDLVL_GATE_NORM_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_276_RDLVL_GATE_NORM_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_277_REGOFF 0x454
+#define HWIO_DDR_CTL_277_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_277_REGOFF)
+#define HWIO_DDR_CTL_277_RDLVL_GATE_HIGH_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_277_RDLVL_GATE_HIGH_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_277_RDLVL_GATE_TIMEOUT_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_277_RDLVL_GATE_TIMEOUT_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_278_REGOFF 0x458
+#define HWIO_DDR_CTL_278_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_278_REGOFF)
+#define HWIO_DDR_CTL_278_RDLVL_GATE_SW_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_278_RDLVL_GATE_SW_PROMOTE_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_278_RDLVL_GATE_DFI_PROMOTE_THRESHOLD_F0_FLDMASK \
+ (0xffff0000)
+#define HWIO_DDR_CTL_278_RDLVL_GATE_DFI_PROMOTE_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_279_REGOFF 0x45c
+#define HWIO_DDR_CTL_279_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_279_REGOFF)
+#define HWIO_DDR_CTL_279_RDLVL_NORM_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_279_RDLVL_NORM_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_279_RDLVL_HIGH_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_279_RDLVL_HIGH_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_280_REGOFF 0x460
+#define HWIO_DDR_CTL_280_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_280_REGOFF)
+#define HWIO_DDR_CTL_280_RDLVL_TIMEOUT_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_280_RDLVL_TIMEOUT_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_280_RDLVL_SW_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_280_RDLVL_SW_PROMOTE_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_281_REGOFF 0x464
+#define HWIO_DDR_CTL_281_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_281_REGOFF)
+#define HWIO_DDR_CTL_281_RDLVL_DFI_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_281_RDLVL_DFI_PROMOTE_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_281_RDLVL_GATE_NORM_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_281_RDLVL_GATE_NORM_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_282_REGOFF 0x468
+#define HWIO_DDR_CTL_282_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_282_REGOFF)
+#define HWIO_DDR_CTL_282_RDLVL_GATE_HIGH_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_282_RDLVL_GATE_HIGH_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_282_RDLVL_GATE_TIMEOUT_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_282_RDLVL_GATE_TIMEOUT_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_283_REGOFF 0x46c
+#define HWIO_DDR_CTL_283_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_283_REGOFF)
+#define HWIO_DDR_CTL_283_RDLVL_GATE_SW_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_283_RDLVL_GATE_SW_PROMOTE_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_283_RDLVL_GATE_DFI_PROMOTE_THRESHOLD_F1_FLDMASK \
+ (0xffff0000)
+#define HWIO_DDR_CTL_283_RDLVL_GATE_DFI_PROMOTE_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_284_REGOFF 0x470
+#define HWIO_DDR_CTL_284_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_284_REGOFF)
+#define HWIO_DDR_CTL_284_RDLVL_NORM_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_284_RDLVL_NORM_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_284_RDLVL_HIGH_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_284_RDLVL_HIGH_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_285_REGOFF 0x474
+#define HWIO_DDR_CTL_285_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_285_REGOFF)
+#define HWIO_DDR_CTL_285_RDLVL_TIMEOUT_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_285_RDLVL_TIMEOUT_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_285_RDLVL_SW_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_285_RDLVL_SW_PROMOTE_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_286_REGOFF 0x478
+#define HWIO_DDR_CTL_286_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_286_REGOFF)
+#define HWIO_DDR_CTL_286_RDLVL_DFI_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_286_RDLVL_DFI_PROMOTE_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_286_RDLVL_GATE_NORM_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_286_RDLVL_GATE_NORM_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_287_REGOFF 0x47c
+#define HWIO_DDR_CTL_287_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_287_REGOFF)
+#define HWIO_DDR_CTL_287_RDLVL_GATE_HIGH_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_287_RDLVL_GATE_HIGH_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_287_RDLVL_GATE_TIMEOUT_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_287_RDLVL_GATE_TIMEOUT_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_288_REGOFF 0x480
+#define HWIO_DDR_CTL_288_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_288_REGOFF)
+#define HWIO_DDR_CTL_288_RDLVL_GATE_SW_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_288_RDLVL_GATE_SW_PROMOTE_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_288_RDLVL_GATE_DFI_PROMOTE_THRESHOLD_F2_FLDMASK \
+ (0xffff0000)
+#define HWIO_DDR_CTL_288_RDLVL_GATE_DFI_PROMOTE_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_289_REGOFF 0x484
+#define HWIO_DDR_CTL_289_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_289_REGOFF)
+#define HWIO_DDR_CTL_289_RDLVL_NORM_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_289_RDLVL_NORM_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_289_RDLVL_HIGH_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_289_RDLVL_HIGH_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_290_REGOFF 0x488
+#define HWIO_DDR_CTL_290_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_290_REGOFF)
+#define HWIO_DDR_CTL_290_RDLVL_TIMEOUT_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_290_RDLVL_TIMEOUT_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_290_RDLVL_SW_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_290_RDLVL_SW_PROMOTE_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_291_REGOFF 0x48c
+#define HWIO_DDR_CTL_291_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_291_REGOFF)
+#define HWIO_DDR_CTL_291_RDLVL_DFI_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_291_RDLVL_DFI_PROMOTE_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_291_RDLVL_GATE_NORM_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_291_RDLVL_GATE_NORM_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_292_REGOFF 0x490
+#define HWIO_DDR_CTL_292_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_292_REGOFF)
+#define HWIO_DDR_CTL_292_RDLVL_GATE_HIGH_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_292_RDLVL_GATE_HIGH_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_292_RDLVL_GATE_TIMEOUT_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_292_RDLVL_GATE_TIMEOUT_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_293_REGOFF 0x494
+#define HWIO_DDR_CTL_293_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_293_REGOFF)
+#define HWIO_DDR_CTL_293_RDLVL_GATE_SW_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_293_RDLVL_GATE_SW_PROMOTE_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_293_RDLVL_GATE_DFI_PROMOTE_THRESHOLD_F3_FLDMASK \
+ (0xffff0000)
+#define HWIO_DDR_CTL_293_RDLVL_GATE_DFI_PROMOTE_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_294_REGOFF 0x498
+#define HWIO_DDR_CTL_294_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_294_REGOFF)
+#define HWIO_DDR_CTL_294_CALVL_REQ_FLDMASK (0x1)
+#define HWIO_DDR_CTL_294_CALVL_REQ_FLDSHFT (0)
+#define HWIO_DDR_CTL_294_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_294_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_294_CALVL_CS_FLDMASK (0x100)
+#define HWIO_DDR_CTL_294_CALVL_CS_FLDSHFT (8)
+#define HWIO_DDR_CTL_294_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_294_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_294_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_294_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_295_REGOFF 0x49c
+#define HWIO_DDR_CTL_295_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_295_REGOFF)
+#define HWIO_DDR_CTL_295_CALVL_PAT_0_FLDMASK (0xfffff)
+#define HWIO_DDR_CTL_295_CALVL_PAT_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_295_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_295_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_CTL_295_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_295_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_296_REGOFF 0x4a0
+#define HWIO_DDR_CTL_296_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_296_REGOFF)
+#define HWIO_DDR_CTL_296_CALVL_BG_PAT_0_FLDMASK (0xfffff)
+#define HWIO_DDR_CTL_296_CALVL_BG_PAT_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_296_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_296_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_CTL_296_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_296_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_297_REGOFF 0x4a4
+#define HWIO_DDR_CTL_297_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_297_REGOFF)
+#define HWIO_DDR_CTL_297_CALVL_PAT_1_FLDMASK (0xfffff)
+#define HWIO_DDR_CTL_297_CALVL_PAT_1_FLDSHFT (0)
+#define HWIO_DDR_CTL_297_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_297_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_CTL_297_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_297_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_298_REGOFF 0x4a8
+#define HWIO_DDR_CTL_298_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_298_REGOFF)
+#define HWIO_DDR_CTL_298_CALVL_BG_PAT_1_FLDMASK (0xfffff)
+#define HWIO_DDR_CTL_298_CALVL_BG_PAT_1_FLDSHFT (0)
+#define HWIO_DDR_CTL_298_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_298_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_CTL_298_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_298_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_299_REGOFF 0x4ac
+#define HWIO_DDR_CTL_299_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_299_REGOFF)
+#define HWIO_DDR_CTL_299_CALVL_PAT_2_FLDMASK (0xfffff)
+#define HWIO_DDR_CTL_299_CALVL_PAT_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_299_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_299_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_CTL_299_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_299_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_300_REGOFF 0x4b0
+#define HWIO_DDR_CTL_300_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_300_REGOFF)
+#define HWIO_DDR_CTL_300_CALVL_BG_PAT_2_FLDMASK (0xfffff)
+#define HWIO_DDR_CTL_300_CALVL_BG_PAT_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_300_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_300_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_CTL_300_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_300_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_301_REGOFF 0x4b4
+#define HWIO_DDR_CTL_301_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_301_REGOFF)
+#define HWIO_DDR_CTL_301_CALVL_PAT_3_FLDMASK (0xfffff)
+#define HWIO_DDR_CTL_301_CALVL_PAT_3_FLDSHFT (0)
+#define HWIO_DDR_CTL_301_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_301_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_CTL_301_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_301_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_302_REGOFF 0x4b8
+#define HWIO_DDR_CTL_302_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_302_REGOFF)
+#define HWIO_DDR_CTL_302_CALVL_BG_PAT_3_FLDMASK (0xfffff)
+#define HWIO_DDR_CTL_302_CALVL_BG_PAT_3_FLDSHFT (0)
+#define HWIO_DDR_CTL_302_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_302_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_CTL_302_CDNS_INTRL1_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_302_CDNS_INTRL1_FLDSHFT (24)
+#define HWIO_DDR_CTL_302_CDNS_INTRL2_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_302_CDNS_INTRL2_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_303_REGOFF 0x4bc
+#define HWIO_DDR_CTL_303_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_303_REGOFF)
+#define HWIO_DDR_CTL_303_CDNS_INTRL0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_303_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_303_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_303_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_303_CALVL_SEQ_EN_FLDMASK (0x300)
+#define HWIO_DDR_CTL_303_CALVL_SEQ_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_303_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_303_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_303_DFI_PHY_CALVL_MODE_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_303_DFI_PHY_CALVL_MODE_FLDSHFT (16)
+#define HWIO_DDR_CTL_303_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_303_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_303_CALVL_PERIODIC_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_303_CALVL_PERIODIC_FLDSHFT (24)
+#define HWIO_DDR_CTL_303_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_303_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_304_REGOFF 0x4c0
+#define HWIO_DDR_CTL_304_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_304_REGOFF)
+#define HWIO_DDR_CTL_304_CALVL_ON_SREF_EXIT_FLDMASK (0x1)
+#define HWIO_DDR_CTL_304_CALVL_ON_SREF_EXIT_FLDSHFT (0)
+#define HWIO_DDR_CTL_304_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_304_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_304_CALVL_AREF_EN_FLDMASK (0x100)
+#define HWIO_DDR_CTL_304_CALVL_AREF_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_304_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_304_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_304_CALVL_NORM_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_304_CALVL_NORM_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_305_REGOFF 0x4c4
+#define HWIO_DDR_CTL_305_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_305_REGOFF)
+#define HWIO_DDR_CTL_305_CALVL_HIGH_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_305_CALVL_HIGH_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_305_CALVL_TIMEOUT_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_305_CALVL_TIMEOUT_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_306_REGOFF 0x4c8
+#define HWIO_DDR_CTL_306_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_306_REGOFF)
+#define HWIO_DDR_CTL_306_CALVL_SW_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_306_CALVL_SW_PROMOTE_THRESHOLD_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_306_CALVL_DFI_PROMOTE_THRESHOLD_F0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_306_CALVL_DFI_PROMOTE_THRESHOLD_F0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_307_REGOFF 0x4cc
+#define HWIO_DDR_CTL_307_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_307_REGOFF)
+#define HWIO_DDR_CTL_307_CALVL_NORM_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_307_CALVL_NORM_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_307_CALVL_HIGH_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_307_CALVL_HIGH_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_308_REGOFF 0x4d0
+#define HWIO_DDR_CTL_308_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_308_REGOFF)
+#define HWIO_DDR_CTL_308_CALVL_TIMEOUT_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_308_CALVL_TIMEOUT_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_308_CALVL_SW_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_308_CALVL_SW_PROMOTE_THRESHOLD_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_309_REGOFF 0x4d4
+#define HWIO_DDR_CTL_309_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_309_REGOFF)
+#define HWIO_DDR_CTL_309_CALVL_DFI_PROMOTE_THRESHOLD_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_309_CALVL_DFI_PROMOTE_THRESHOLD_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_309_CALVL_NORM_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_309_CALVL_NORM_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_310_REGOFF 0x4d8
+#define HWIO_DDR_CTL_310_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_310_REGOFF)
+#define HWIO_DDR_CTL_310_CALVL_HIGH_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_310_CALVL_HIGH_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_310_CALVL_TIMEOUT_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_310_CALVL_TIMEOUT_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_311_REGOFF 0x4dc
+#define HWIO_DDR_CTL_311_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_311_REGOFF)
+#define HWIO_DDR_CTL_311_CALVL_SW_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_311_CALVL_SW_PROMOTE_THRESHOLD_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_311_CALVL_DFI_PROMOTE_THRESHOLD_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_311_CALVL_DFI_PROMOTE_THRESHOLD_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_312_REGOFF 0x4e0
+#define HWIO_DDR_CTL_312_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_312_REGOFF)
+#define HWIO_DDR_CTL_312_CALVL_NORM_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_312_CALVL_NORM_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_312_CALVL_HIGH_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_312_CALVL_HIGH_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_313_REGOFF 0x4e4
+#define HWIO_DDR_CTL_313_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_313_REGOFF)
+#define HWIO_DDR_CTL_313_CALVL_TIMEOUT_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_313_CALVL_TIMEOUT_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_313_CALVL_SW_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_313_CALVL_SW_PROMOTE_THRESHOLD_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_314_REGOFF 0x4e8
+#define HWIO_DDR_CTL_314_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_314_REGOFF)
+#define HWIO_DDR_CTL_314_CALVL_DFI_PROMOTE_THRESHOLD_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_314_CALVL_DFI_PROMOTE_THRESHOLD_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_314_AXI0_ALL_STROBES_USED_ENABLE_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_314_AXI0_ALL_STROBES_USED_ENABLE_FLDSHFT (16)
+#define HWIO_DDR_CTL_314_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_314_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_CTL_314_AXI0_FIXED_PORT_PRIORITY_ENABLE_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_314_AXI0_FIXED_PORT_PRIORITY_ENABLE_FLDSHFT (24)
+#define HWIO_DDR_CTL_314_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_314_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_315_REGOFF 0x4ec
+#define HWIO_DDR_CTL_315_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_315_REGOFF)
+#define HWIO_DDR_CTL_315_AXI0_R_PRIORITY_FLDMASK (0x3)
+#define HWIO_DDR_CTL_315_AXI0_R_PRIORITY_FLDSHFT (0)
+#define HWIO_DDR_CTL_315_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_CTL_315_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_CTL_315_AXI0_W_PRIORITY_FLDMASK (0x300)
+#define HWIO_DDR_CTL_315_AXI0_W_PRIORITY_FLDSHFT (8)
+#define HWIO_DDR_CTL_315_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_315_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_315_AXI0_FIFO_TYPE_REG_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_315_AXI0_FIFO_TYPE_REG_FLDSHFT (16)
+#define HWIO_DDR_CTL_315_CDNS_INTRL2_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_315_CDNS_INTRL2_FLDSHFT (18)
+#define HWIO_DDR_CTL_315_AXI1_ALL_STROBES_USED_ENABLE_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_315_AXI1_ALL_STROBES_USED_ENABLE_FLDSHFT (24)
+#define HWIO_DDR_CTL_315_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_315_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_316_REGOFF 0x4f0
+#define HWIO_DDR_CTL_316_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_316_REGOFF)
+#define HWIO_DDR_CTL_316_AXI1_FIXED_PORT_PRIORITY_ENABLE_FLDMASK (0x1)
+#define HWIO_DDR_CTL_316_AXI1_FIXED_PORT_PRIORITY_ENABLE_FLDSHFT (0)
+#define HWIO_DDR_CTL_316_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_316_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_316_AXI1_R_PRIORITY_FLDMASK (0x300)
+#define HWIO_DDR_CTL_316_AXI1_R_PRIORITY_FLDSHFT (8)
+#define HWIO_DDR_CTL_316_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_316_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_316_AXI1_W_PRIORITY_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_316_AXI1_W_PRIORITY_FLDSHFT (16)
+#define HWIO_DDR_CTL_316_CDNS_INTRL2_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_316_CDNS_INTRL2_FLDSHFT (18)
+#define HWIO_DDR_CTL_316_AXI1_FIFO_TYPE_REG_FLDMASK (0x3000000)
+#define HWIO_DDR_CTL_316_AXI1_FIFO_TYPE_REG_FLDSHFT (24)
+#define HWIO_DDR_CTL_316_CDNS_INTRL3_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_316_CDNS_INTRL3_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_317_REGOFF 0x4f4
+#define HWIO_DDR_CTL_317_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_317_REGOFF)
+#define HWIO_DDR_CTL_317_PORT_ADDR_PROTECTION_EN_FLDMASK (0x1)
+#define HWIO_DDR_CTL_317_PORT_ADDR_PROTECTION_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_317_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_317_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_317_AXI0_ADDRESS_RANGE_ENABLE_FLDMASK (0x100)
+#define HWIO_DDR_CTL_317_AXI0_ADDRESS_RANGE_ENABLE_FLDSHFT (8)
+#define HWIO_DDR_CTL_317_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_317_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_317_AXI1_ADDRESS_RANGE_ENABLE_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_317_AXI1_ADDRESS_RANGE_ENABLE_FLDSHFT (16)
+#define HWIO_DDR_CTL_317_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_317_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_317_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_317_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_318_REGOFF 0x4f8
+#define HWIO_DDR_CTL_318_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_318_REGOFF)
+#define HWIO_DDR_CTL_318_AXI0_START_ADDR_0_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_318_AXI0_START_ADDR_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_318_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_318_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_318_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_318_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_319_REGOFF 0x4fc
+#define HWIO_DDR_CTL_319_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_319_REGOFF)
+#define HWIO_DDR_CTL_319_AXI0_END_ADDR_0_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_319_AXI0_END_ADDR_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_319_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_319_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_319_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_319_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_320_REGOFF 0x500
+#define HWIO_DDR_CTL_320_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_320_REGOFF)
+#define HWIO_DDR_CTL_320_AXI0_START_ADDR_1_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_320_AXI0_START_ADDR_1_FLDSHFT (0)
+#define HWIO_DDR_CTL_320_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_320_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_320_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_320_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_321_REGOFF 0x504
+#define HWIO_DDR_CTL_321_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_321_REGOFF)
+#define HWIO_DDR_CTL_321_AXI0_END_ADDR_1_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_321_AXI0_END_ADDR_1_FLDSHFT (0)
+#define HWIO_DDR_CTL_321_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_321_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_321_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_321_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_322_REGOFF 0x508
+#define HWIO_DDR_CTL_322_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_322_REGOFF)
+#define HWIO_DDR_CTL_322_AXI0_START_ADDR_2_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_322_AXI0_START_ADDR_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_322_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_322_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_322_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_322_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_323_REGOFF 0x50c
+#define HWIO_DDR_CTL_323_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_323_REGOFF)
+#define HWIO_DDR_CTL_323_AXI0_END_ADDR_2_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_323_AXI0_END_ADDR_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_323_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_323_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_323_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_323_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_324_REGOFF 0x510
+#define HWIO_DDR_CTL_324_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_324_REGOFF)
+#define HWIO_DDR_CTL_324_AXI0_START_ADDR_3_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_324_AXI0_START_ADDR_3_FLDSHFT (0)
+#define HWIO_DDR_CTL_324_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_324_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_324_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_324_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_325_REGOFF 0x514
+#define HWIO_DDR_CTL_325_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_325_REGOFF)
+#define HWIO_DDR_CTL_325_AXI0_END_ADDR_3_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_325_AXI0_END_ADDR_3_FLDSHFT (0)
+#define HWIO_DDR_CTL_325_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_325_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_325_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_325_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_326_REGOFF 0x518
+#define HWIO_DDR_CTL_326_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_326_REGOFF)
+#define HWIO_DDR_CTL_326_AXI0_START_ADDR_4_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_326_AXI0_START_ADDR_4_FLDSHFT (0)
+#define HWIO_DDR_CTL_326_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_326_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_326_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_326_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_327_REGOFF 0x51c
+#define HWIO_DDR_CTL_327_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_327_REGOFF)
+#define HWIO_DDR_CTL_327_AXI0_END_ADDR_4_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_327_AXI0_END_ADDR_4_FLDSHFT (0)
+#define HWIO_DDR_CTL_327_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_327_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_327_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_327_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_328_REGOFF 0x520
+#define HWIO_DDR_CTL_328_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_328_REGOFF)
+#define HWIO_DDR_CTL_328_AXI0_START_ADDR_5_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_328_AXI0_START_ADDR_5_FLDSHFT (0)
+#define HWIO_DDR_CTL_328_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_328_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_328_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_328_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_329_REGOFF 0x524
+#define HWIO_DDR_CTL_329_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_329_REGOFF)
+#define HWIO_DDR_CTL_329_AXI0_END_ADDR_5_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_329_AXI0_END_ADDR_5_FLDSHFT (0)
+#define HWIO_DDR_CTL_329_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_329_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_329_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_329_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_330_REGOFF 0x528
+#define HWIO_DDR_CTL_330_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_330_REGOFF)
+#define HWIO_DDR_CTL_330_AXI0_START_ADDR_6_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_330_AXI0_START_ADDR_6_FLDSHFT (0)
+#define HWIO_DDR_CTL_330_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_330_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_330_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_330_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_331_REGOFF 0x52c
+#define HWIO_DDR_CTL_331_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_331_REGOFF)
+#define HWIO_DDR_CTL_331_AXI0_END_ADDR_6_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_331_AXI0_END_ADDR_6_FLDSHFT (0)
+#define HWIO_DDR_CTL_331_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_331_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_331_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_331_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_332_REGOFF 0x530
+#define HWIO_DDR_CTL_332_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_332_REGOFF)
+#define HWIO_DDR_CTL_332_AXI0_START_ADDR_7_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_332_AXI0_START_ADDR_7_FLDSHFT (0)
+#define HWIO_DDR_CTL_332_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_332_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_332_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_332_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_333_REGOFF 0x534
+#define HWIO_DDR_CTL_333_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_333_REGOFF)
+#define HWIO_DDR_CTL_333_AXI0_END_ADDR_7_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_333_AXI0_END_ADDR_7_FLDSHFT (0)
+#define HWIO_DDR_CTL_333_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_333_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_333_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_333_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_334_REGOFF 0x538
+#define HWIO_DDR_CTL_334_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_334_REGOFF)
+#define HWIO_DDR_CTL_334_AXI0_START_ADDR_8_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_334_AXI0_START_ADDR_8_FLDSHFT (0)
+#define HWIO_DDR_CTL_334_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_334_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_334_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_334_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_335_REGOFF 0x53c
+#define HWIO_DDR_CTL_335_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_335_REGOFF)
+#define HWIO_DDR_CTL_335_AXI0_END_ADDR_8_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_335_AXI0_END_ADDR_8_FLDSHFT (0)
+#define HWIO_DDR_CTL_335_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_335_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_335_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_335_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_336_REGOFF 0x540
+#define HWIO_DDR_CTL_336_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_336_REGOFF)
+#define HWIO_DDR_CTL_336_AXI0_START_ADDR_9_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_336_AXI0_START_ADDR_9_FLDSHFT (0)
+#define HWIO_DDR_CTL_336_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_336_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_336_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_336_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_337_REGOFF 0x544
+#define HWIO_DDR_CTL_337_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_337_REGOFF)
+#define HWIO_DDR_CTL_337_AXI0_END_ADDR_9_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_337_AXI0_END_ADDR_9_FLDSHFT (0)
+#define HWIO_DDR_CTL_337_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_337_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_337_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_337_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_338_REGOFF 0x548
+#define HWIO_DDR_CTL_338_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_338_REGOFF)
+#define HWIO_DDR_CTL_338_AXI0_START_ADDR_10_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_338_AXI0_START_ADDR_10_FLDSHFT (0)
+#define HWIO_DDR_CTL_338_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_338_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_338_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_338_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_339_REGOFF 0x54c
+#define HWIO_DDR_CTL_339_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_339_REGOFF)
+#define HWIO_DDR_CTL_339_AXI0_END_ADDR_10_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_339_AXI0_END_ADDR_10_FLDSHFT (0)
+#define HWIO_DDR_CTL_339_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_339_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_339_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_339_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_340_REGOFF 0x550
+#define HWIO_DDR_CTL_340_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_340_REGOFF)
+#define HWIO_DDR_CTL_340_AXI0_START_ADDR_11_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_340_AXI0_START_ADDR_11_FLDSHFT (0)
+#define HWIO_DDR_CTL_340_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_340_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_340_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_340_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_341_REGOFF 0x554
+#define HWIO_DDR_CTL_341_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_341_REGOFF)
+#define HWIO_DDR_CTL_341_AXI0_END_ADDR_11_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_341_AXI0_END_ADDR_11_FLDSHFT (0)
+#define HWIO_DDR_CTL_341_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_341_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_341_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_341_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_342_REGOFF 0x558
+#define HWIO_DDR_CTL_342_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_342_REGOFF)
+#define HWIO_DDR_CTL_342_AXI0_START_ADDR_12_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_342_AXI0_START_ADDR_12_FLDSHFT (0)
+#define HWIO_DDR_CTL_342_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_342_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_342_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_342_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_343_REGOFF 0x55c
+#define HWIO_DDR_CTL_343_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_343_REGOFF)
+#define HWIO_DDR_CTL_343_AXI0_END_ADDR_12_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_343_AXI0_END_ADDR_12_FLDSHFT (0)
+#define HWIO_DDR_CTL_343_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_343_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_343_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_343_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_344_REGOFF 0x560
+#define HWIO_DDR_CTL_344_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_344_REGOFF)
+#define HWIO_DDR_CTL_344_AXI0_START_ADDR_13_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_344_AXI0_START_ADDR_13_FLDSHFT (0)
+#define HWIO_DDR_CTL_344_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_344_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_344_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_344_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_345_REGOFF 0x564
+#define HWIO_DDR_CTL_345_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_345_REGOFF)
+#define HWIO_DDR_CTL_345_AXI0_END_ADDR_13_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_345_AXI0_END_ADDR_13_FLDSHFT (0)
+#define HWIO_DDR_CTL_345_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_345_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_345_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_345_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_346_REGOFF 0x568
+#define HWIO_DDR_CTL_346_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_346_REGOFF)
+#define HWIO_DDR_CTL_346_AXI0_START_ADDR_14_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_346_AXI0_START_ADDR_14_FLDSHFT (0)
+#define HWIO_DDR_CTL_346_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_346_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_346_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_346_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_347_REGOFF 0x56c
+#define HWIO_DDR_CTL_347_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_347_REGOFF)
+#define HWIO_DDR_CTL_347_AXI0_END_ADDR_14_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_347_AXI0_END_ADDR_14_FLDSHFT (0)
+#define HWIO_DDR_CTL_347_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_347_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_347_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_347_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_348_REGOFF 0x570
+#define HWIO_DDR_CTL_348_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_348_REGOFF)
+#define HWIO_DDR_CTL_348_AXI0_START_ADDR_15_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_348_AXI0_START_ADDR_15_FLDSHFT (0)
+#define HWIO_DDR_CTL_348_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_348_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_348_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_348_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_349_REGOFF 0x574
+#define HWIO_DDR_CTL_349_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_349_REGOFF)
+#define HWIO_DDR_CTL_349_AXI0_END_ADDR_15_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_349_AXI0_END_ADDR_15_FLDSHFT (0)
+#define HWIO_DDR_CTL_349_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_349_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_349_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_349_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_350_REGOFF 0x578
+#define HWIO_DDR_CTL_350_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_350_REGOFF)
+#define HWIO_DDR_CTL_350_AXI1_START_ADDR_0_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_350_AXI1_START_ADDR_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_350_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_350_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_350_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_350_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_351_REGOFF 0x57c
+#define HWIO_DDR_CTL_351_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_351_REGOFF)
+#define HWIO_DDR_CTL_351_AXI1_END_ADDR_0_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_351_AXI1_END_ADDR_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_351_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_351_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_351_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_351_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_352_REGOFF 0x580
+#define HWIO_DDR_CTL_352_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_352_REGOFF)
+#define HWIO_DDR_CTL_352_AXI1_START_ADDR_1_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_352_AXI1_START_ADDR_1_FLDSHFT (0)
+#define HWIO_DDR_CTL_352_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_352_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_352_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_352_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_353_REGOFF 0x584
+#define HWIO_DDR_CTL_353_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_353_REGOFF)
+#define HWIO_DDR_CTL_353_AXI1_END_ADDR_1_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_353_AXI1_END_ADDR_1_FLDSHFT (0)
+#define HWIO_DDR_CTL_353_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_353_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_353_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_353_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_354_REGOFF 0x588
+#define HWIO_DDR_CTL_354_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_354_REGOFF)
+#define HWIO_DDR_CTL_354_AXI1_START_ADDR_2_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_354_AXI1_START_ADDR_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_354_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_354_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_354_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_354_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_355_REGOFF 0x58c
+#define HWIO_DDR_CTL_355_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_355_REGOFF)
+#define HWIO_DDR_CTL_355_AXI1_END_ADDR_2_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_355_AXI1_END_ADDR_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_355_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_355_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_355_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_355_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_356_REGOFF 0x590
+#define HWIO_DDR_CTL_356_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_356_REGOFF)
+#define HWIO_DDR_CTL_356_AXI1_START_ADDR_3_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_356_AXI1_START_ADDR_3_FLDSHFT (0)
+#define HWIO_DDR_CTL_356_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_356_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_356_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_356_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_357_REGOFF 0x594
+#define HWIO_DDR_CTL_357_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_357_REGOFF)
+#define HWIO_DDR_CTL_357_AXI1_END_ADDR_3_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_357_AXI1_END_ADDR_3_FLDSHFT (0)
+#define HWIO_DDR_CTL_357_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_357_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_357_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_357_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_358_REGOFF 0x598
+#define HWIO_DDR_CTL_358_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_358_REGOFF)
+#define HWIO_DDR_CTL_358_AXI1_START_ADDR_4_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_358_AXI1_START_ADDR_4_FLDSHFT (0)
+#define HWIO_DDR_CTL_358_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_358_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_358_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_358_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_359_REGOFF 0x59c
+#define HWIO_DDR_CTL_359_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_359_REGOFF)
+#define HWIO_DDR_CTL_359_AXI1_END_ADDR_4_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_359_AXI1_END_ADDR_4_FLDSHFT (0)
+#define HWIO_DDR_CTL_359_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_359_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_359_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_359_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_360_REGOFF 0x5a0
+#define HWIO_DDR_CTL_360_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_360_REGOFF)
+#define HWIO_DDR_CTL_360_AXI1_START_ADDR_5_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_360_AXI1_START_ADDR_5_FLDSHFT (0)
+#define HWIO_DDR_CTL_360_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_360_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_360_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_360_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_361_REGOFF 0x5a4
+#define HWIO_DDR_CTL_361_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_361_REGOFF)
+#define HWIO_DDR_CTL_361_AXI1_END_ADDR_5_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_361_AXI1_END_ADDR_5_FLDSHFT (0)
+#define HWIO_DDR_CTL_361_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_361_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_361_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_361_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_362_REGOFF 0x5a8
+#define HWIO_DDR_CTL_362_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_362_REGOFF)
+#define HWIO_DDR_CTL_362_AXI1_START_ADDR_6_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_362_AXI1_START_ADDR_6_FLDSHFT (0)
+#define HWIO_DDR_CTL_362_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_362_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_362_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_362_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_363_REGOFF 0x5ac
+#define HWIO_DDR_CTL_363_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_363_REGOFF)
+#define HWIO_DDR_CTL_363_AXI1_END_ADDR_6_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_363_AXI1_END_ADDR_6_FLDSHFT (0)
+#define HWIO_DDR_CTL_363_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_363_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_363_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_363_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_364_REGOFF 0x5b0
+#define HWIO_DDR_CTL_364_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_364_REGOFF)
+#define HWIO_DDR_CTL_364_AXI1_START_ADDR_7_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_364_AXI1_START_ADDR_7_FLDSHFT (0)
+#define HWIO_DDR_CTL_364_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_364_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_364_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_364_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_365_REGOFF 0x5b4
+#define HWIO_DDR_CTL_365_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_365_REGOFF)
+#define HWIO_DDR_CTL_365_AXI1_END_ADDR_7_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_365_AXI1_END_ADDR_7_FLDSHFT (0)
+#define HWIO_DDR_CTL_365_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_365_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_365_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_365_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_366_REGOFF 0x5b8
+#define HWIO_DDR_CTL_366_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_366_REGOFF)
+#define HWIO_DDR_CTL_366_AXI1_START_ADDR_8_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_366_AXI1_START_ADDR_8_FLDSHFT (0)
+#define HWIO_DDR_CTL_366_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_366_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_366_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_366_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_367_REGOFF 0x5bc
+#define HWIO_DDR_CTL_367_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_367_REGOFF)
+#define HWIO_DDR_CTL_367_AXI1_END_ADDR_8_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_367_AXI1_END_ADDR_8_FLDSHFT (0)
+#define HWIO_DDR_CTL_367_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_367_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_367_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_367_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_368_REGOFF 0x5c0
+#define HWIO_DDR_CTL_368_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_368_REGOFF)
+#define HWIO_DDR_CTL_368_AXI1_START_ADDR_9_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_368_AXI1_START_ADDR_9_FLDSHFT (0)
+#define HWIO_DDR_CTL_368_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_368_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_368_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_368_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_369_REGOFF 0x5c4
+#define HWIO_DDR_CTL_369_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_369_REGOFF)
+#define HWIO_DDR_CTL_369_AXI1_END_ADDR_9_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_369_AXI1_END_ADDR_9_FLDSHFT (0)
+#define HWIO_DDR_CTL_369_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_369_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_369_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_369_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_370_REGOFF 0x5c8
+#define HWIO_DDR_CTL_370_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_370_REGOFF)
+#define HWIO_DDR_CTL_370_AXI1_START_ADDR_10_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_370_AXI1_START_ADDR_10_FLDSHFT (0)
+#define HWIO_DDR_CTL_370_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_370_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_370_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_370_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_371_REGOFF 0x5cc
+#define HWIO_DDR_CTL_371_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_371_REGOFF)
+#define HWIO_DDR_CTL_371_AXI1_END_ADDR_10_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_371_AXI1_END_ADDR_10_FLDSHFT (0)
+#define HWIO_DDR_CTL_371_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_371_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_371_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_371_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_372_REGOFF 0x5d0
+#define HWIO_DDR_CTL_372_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_372_REGOFF)
+#define HWIO_DDR_CTL_372_AXI1_START_ADDR_11_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_372_AXI1_START_ADDR_11_FLDSHFT (0)
+#define HWIO_DDR_CTL_372_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_372_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_372_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_372_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_373_REGOFF 0x5d4
+#define HWIO_DDR_CTL_373_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_373_REGOFF)
+#define HWIO_DDR_CTL_373_AXI1_END_ADDR_11_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_373_AXI1_END_ADDR_11_FLDSHFT (0)
+#define HWIO_DDR_CTL_373_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_373_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_373_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_373_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_374_REGOFF 0x5d8
+#define HWIO_DDR_CTL_374_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_374_REGOFF)
+#define HWIO_DDR_CTL_374_AXI1_START_ADDR_12_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_374_AXI1_START_ADDR_12_FLDSHFT (0)
+#define HWIO_DDR_CTL_374_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_374_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_374_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_374_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_375_REGOFF 0x5dc
+#define HWIO_DDR_CTL_375_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_375_REGOFF)
+#define HWIO_DDR_CTL_375_AXI1_END_ADDR_12_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_375_AXI1_END_ADDR_12_FLDSHFT (0)
+#define HWIO_DDR_CTL_375_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_375_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_375_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_375_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_376_REGOFF 0x5e0
+#define HWIO_DDR_CTL_376_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_376_REGOFF)
+#define HWIO_DDR_CTL_376_AXI1_START_ADDR_13_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_376_AXI1_START_ADDR_13_FLDSHFT (0)
+#define HWIO_DDR_CTL_376_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_376_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_376_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_376_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_377_REGOFF 0x5e4
+#define HWIO_DDR_CTL_377_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_377_REGOFF)
+#define HWIO_DDR_CTL_377_AXI1_END_ADDR_13_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_377_AXI1_END_ADDR_13_FLDSHFT (0)
+#define HWIO_DDR_CTL_377_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_377_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_377_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_377_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_378_REGOFF 0x5e8
+#define HWIO_DDR_CTL_378_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_378_REGOFF)
+#define HWIO_DDR_CTL_378_AXI1_START_ADDR_14_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_378_AXI1_START_ADDR_14_FLDSHFT (0)
+#define HWIO_DDR_CTL_378_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_378_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_378_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_378_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_379_REGOFF 0x5ec
+#define HWIO_DDR_CTL_379_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_379_REGOFF)
+#define HWIO_DDR_CTL_379_AXI1_END_ADDR_14_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_379_AXI1_END_ADDR_14_FLDSHFT (0)
+#define HWIO_DDR_CTL_379_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_379_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_379_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_379_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_380_REGOFF 0x5f0
+#define HWIO_DDR_CTL_380_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_380_REGOFF)
+#define HWIO_DDR_CTL_380_AXI1_START_ADDR_15_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_380_AXI1_START_ADDR_15_FLDSHFT (0)
+#define HWIO_DDR_CTL_380_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_380_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_380_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_380_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_381_REGOFF 0x5f4
+#define HWIO_DDR_CTL_381_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_381_REGOFF)
+#define HWIO_DDR_CTL_381_AXI1_END_ADDR_15_FLDMASK (0x3ffff)
+#define HWIO_DDR_CTL_381_AXI1_END_ADDR_15_FLDSHFT (0)
+#define HWIO_DDR_CTL_381_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_381_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_381_AXI0_RANGE_PROT_BITS_0_FLDMASK (0x3000000)
+#define HWIO_DDR_CTL_381_AXI0_RANGE_PROT_BITS_0_FLDSHFT (24)
+#define HWIO_DDR_CTL_381_CDNS_INTRL1_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_381_CDNS_INTRL1_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_382_REGOFF 0x5f8
+#define HWIO_DDR_CTL_382_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_382_REGOFF)
+#define HWIO_DDR_CTL_382_AXI0_RANGE_RID_CHECK_BITS_0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_382_AXI0_RANGE_RID_CHECK_BITS_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_382_AXI0_RANGE_WID_CHECK_BITS_0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_382_AXI0_RANGE_WID_CHECK_BITS_0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_383_REGOFF 0x5fc
+#define HWIO_DDR_CTL_383_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_383_REGOFF)
+#define HWIO_DDR_CTL_383_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_383_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_383_AXI0_RANGE_PROT_BITS_1_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_383_AXI0_RANGE_PROT_BITS_1_FLDSHFT (16)
+#define HWIO_DDR_CTL_383_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_383_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_383_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_383_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_384_REGOFF 0x600
+#define HWIO_DDR_CTL_384_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_384_REGOFF)
+#define HWIO_DDR_CTL_384_AXI0_RANGE_RID_CHECK_BITS_1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_384_AXI0_RANGE_RID_CHECK_BITS_1_FLDSHFT (0)
+#define HWIO_DDR_CTL_384_AXI0_RANGE_WID_CHECK_BITS_1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_384_AXI0_RANGE_WID_CHECK_BITS_1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_385_REGOFF 0x604
+#define HWIO_DDR_CTL_385_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_385_REGOFF)
+#define HWIO_DDR_CTL_385_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_385_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_385_AXI0_RANGE_PROT_BITS_2_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_385_AXI0_RANGE_PROT_BITS_2_FLDSHFT (16)
+#define HWIO_DDR_CTL_385_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_385_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_385_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_385_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_386_REGOFF 0x608
+#define HWIO_DDR_CTL_386_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_386_REGOFF)
+#define HWIO_DDR_CTL_386_AXI0_RANGE_RID_CHECK_BITS_2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_386_AXI0_RANGE_RID_CHECK_BITS_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_386_AXI0_RANGE_WID_CHECK_BITS_2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_386_AXI0_RANGE_WID_CHECK_BITS_2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_387_REGOFF 0x60c
+#define HWIO_DDR_CTL_387_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_387_REGOFF)
+#define HWIO_DDR_CTL_387_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_387_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_387_AXI0_RANGE_PROT_BITS_3_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_387_AXI0_RANGE_PROT_BITS_3_FLDSHFT (16)
+#define HWIO_DDR_CTL_387_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_387_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_387_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_387_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_388_REGOFF 0x610
+#define HWIO_DDR_CTL_388_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_388_REGOFF)
+#define HWIO_DDR_CTL_388_AXI0_RANGE_RID_CHECK_BITS_3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_388_AXI0_RANGE_RID_CHECK_BITS_3_FLDSHFT (0)
+#define HWIO_DDR_CTL_388_AXI0_RANGE_WID_CHECK_BITS_3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_388_AXI0_RANGE_WID_CHECK_BITS_3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_389_REGOFF 0x614
+#define HWIO_DDR_CTL_389_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_389_REGOFF)
+#define HWIO_DDR_CTL_389_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_389_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_389_AXI0_RANGE_PROT_BITS_4_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_389_AXI0_RANGE_PROT_BITS_4_FLDSHFT (16)
+#define HWIO_DDR_CTL_389_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_389_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_389_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_389_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_390_REGOFF 0x618
+#define HWIO_DDR_CTL_390_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_390_REGOFF)
+#define HWIO_DDR_CTL_390_AXI0_RANGE_RID_CHECK_BITS_4_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_390_AXI0_RANGE_RID_CHECK_BITS_4_FLDSHFT (0)
+#define HWIO_DDR_CTL_390_AXI0_RANGE_WID_CHECK_BITS_4_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_390_AXI0_RANGE_WID_CHECK_BITS_4_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_391_REGOFF 0x61c
+#define HWIO_DDR_CTL_391_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_391_REGOFF)
+#define HWIO_DDR_CTL_391_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_391_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_391_AXI0_RANGE_PROT_BITS_5_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_391_AXI0_RANGE_PROT_BITS_5_FLDSHFT (16)
+#define HWIO_DDR_CTL_391_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_391_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_391_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_391_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_392_REGOFF 0x620
+#define HWIO_DDR_CTL_392_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_392_REGOFF)
+#define HWIO_DDR_CTL_392_AXI0_RANGE_RID_CHECK_BITS_5_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_392_AXI0_RANGE_RID_CHECK_BITS_5_FLDSHFT (0)
+#define HWIO_DDR_CTL_392_AXI0_RANGE_WID_CHECK_BITS_5_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_392_AXI0_RANGE_WID_CHECK_BITS_5_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_393_REGOFF 0x624
+#define HWIO_DDR_CTL_393_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_393_REGOFF)
+#define HWIO_DDR_CTL_393_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_393_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_393_AXI0_RANGE_PROT_BITS_6_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_393_AXI0_RANGE_PROT_BITS_6_FLDSHFT (16)
+#define HWIO_DDR_CTL_393_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_393_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_393_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_393_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_394_REGOFF 0x628
+#define HWIO_DDR_CTL_394_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_394_REGOFF)
+#define HWIO_DDR_CTL_394_AXI0_RANGE_RID_CHECK_BITS_6_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_394_AXI0_RANGE_RID_CHECK_BITS_6_FLDSHFT (0)
+#define HWIO_DDR_CTL_394_AXI0_RANGE_WID_CHECK_BITS_6_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_394_AXI0_RANGE_WID_CHECK_BITS_6_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_395_REGOFF 0x62c
+#define HWIO_DDR_CTL_395_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_395_REGOFF)
+#define HWIO_DDR_CTL_395_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_395_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_395_AXI0_RANGE_PROT_BITS_7_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_395_AXI0_RANGE_PROT_BITS_7_FLDSHFT (16)
+#define HWIO_DDR_CTL_395_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_395_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_395_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_395_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_396_REGOFF 0x630
+#define HWIO_DDR_CTL_396_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_396_REGOFF)
+#define HWIO_DDR_CTL_396_AXI0_RANGE_RID_CHECK_BITS_7_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_396_AXI0_RANGE_RID_CHECK_BITS_7_FLDSHFT (0)
+#define HWIO_DDR_CTL_396_AXI0_RANGE_WID_CHECK_BITS_7_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_396_AXI0_RANGE_WID_CHECK_BITS_7_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_397_REGOFF 0x634
+#define HWIO_DDR_CTL_397_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_397_REGOFF)
+#define HWIO_DDR_CTL_397_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_397_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_397_AXI0_RANGE_PROT_BITS_8_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_397_AXI0_RANGE_PROT_BITS_8_FLDSHFT (16)
+#define HWIO_DDR_CTL_397_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_397_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_397_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_397_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_398_REGOFF 0x638
+#define HWIO_DDR_CTL_398_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_398_REGOFF)
+#define HWIO_DDR_CTL_398_AXI0_RANGE_RID_CHECK_BITS_8_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_398_AXI0_RANGE_RID_CHECK_BITS_8_FLDSHFT (0)
+#define HWIO_DDR_CTL_398_AXI0_RANGE_WID_CHECK_BITS_8_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_398_AXI0_RANGE_WID_CHECK_BITS_8_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_399_REGOFF 0x63c
+#define HWIO_DDR_CTL_399_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_399_REGOFF)
+#define HWIO_DDR_CTL_399_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_399_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_399_AXI0_RANGE_PROT_BITS_9_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_399_AXI0_RANGE_PROT_BITS_9_FLDSHFT (16)
+#define HWIO_DDR_CTL_399_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_399_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_399_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_399_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_400_REGOFF 0x640
+#define HWIO_DDR_CTL_400_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_400_REGOFF)
+#define HWIO_DDR_CTL_400_AXI0_RANGE_RID_CHECK_BITS_9_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_400_AXI0_RANGE_RID_CHECK_BITS_9_FLDSHFT (0)
+#define HWIO_DDR_CTL_400_AXI0_RANGE_WID_CHECK_BITS_9_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_400_AXI0_RANGE_WID_CHECK_BITS_9_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_401_REGOFF 0x644
+#define HWIO_DDR_CTL_401_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_401_REGOFF)
+#define HWIO_DDR_CTL_401_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_401_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_401_AXI0_RANGE_PROT_BITS_10_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_401_AXI0_RANGE_PROT_BITS_10_FLDSHFT (16)
+#define HWIO_DDR_CTL_401_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_401_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_401_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_401_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_402_REGOFF 0x648
+#define HWIO_DDR_CTL_402_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_402_REGOFF)
+#define HWIO_DDR_CTL_402_AXI0_RANGE_RID_CHECK_BITS_10_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_402_AXI0_RANGE_RID_CHECK_BITS_10_FLDSHFT (0)
+#define HWIO_DDR_CTL_402_AXI0_RANGE_WID_CHECK_BITS_10_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_402_AXI0_RANGE_WID_CHECK_BITS_10_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_403_REGOFF 0x64c
+#define HWIO_DDR_CTL_403_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_403_REGOFF)
+#define HWIO_DDR_CTL_403_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_403_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_403_AXI0_RANGE_PROT_BITS_11_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_403_AXI0_RANGE_PROT_BITS_11_FLDSHFT (16)
+#define HWIO_DDR_CTL_403_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_403_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_403_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_403_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_404_REGOFF 0x650
+#define HWIO_DDR_CTL_404_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_404_REGOFF)
+#define HWIO_DDR_CTL_404_AXI0_RANGE_RID_CHECK_BITS_11_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_404_AXI0_RANGE_RID_CHECK_BITS_11_FLDSHFT (0)
+#define HWIO_DDR_CTL_404_AXI0_RANGE_WID_CHECK_BITS_11_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_404_AXI0_RANGE_WID_CHECK_BITS_11_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_405_REGOFF 0x654
+#define HWIO_DDR_CTL_405_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_405_REGOFF)
+#define HWIO_DDR_CTL_405_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_405_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_405_AXI0_RANGE_PROT_BITS_12_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_405_AXI0_RANGE_PROT_BITS_12_FLDSHFT (16)
+#define HWIO_DDR_CTL_405_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_405_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_405_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_405_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_406_REGOFF 0x658
+#define HWIO_DDR_CTL_406_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_406_REGOFF)
+#define HWIO_DDR_CTL_406_AXI0_RANGE_RID_CHECK_BITS_12_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_406_AXI0_RANGE_RID_CHECK_BITS_12_FLDSHFT (0)
+#define HWIO_DDR_CTL_406_AXI0_RANGE_WID_CHECK_BITS_12_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_406_AXI0_RANGE_WID_CHECK_BITS_12_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_407_REGOFF 0x65c
+#define HWIO_DDR_CTL_407_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_407_REGOFF)
+#define HWIO_DDR_CTL_407_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_407_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_407_AXI0_RANGE_PROT_BITS_13_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_407_AXI0_RANGE_PROT_BITS_13_FLDSHFT (16)
+#define HWIO_DDR_CTL_407_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_407_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_407_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_407_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_408_REGOFF 0x660
+#define HWIO_DDR_CTL_408_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_408_REGOFF)
+#define HWIO_DDR_CTL_408_AXI0_RANGE_RID_CHECK_BITS_13_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_408_AXI0_RANGE_RID_CHECK_BITS_13_FLDSHFT (0)
+#define HWIO_DDR_CTL_408_AXI0_RANGE_WID_CHECK_BITS_13_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_408_AXI0_RANGE_WID_CHECK_BITS_13_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_409_REGOFF 0x664
+#define HWIO_DDR_CTL_409_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_409_REGOFF)
+#define HWIO_DDR_CTL_409_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_409_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_409_AXI0_RANGE_PROT_BITS_14_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_409_AXI0_RANGE_PROT_BITS_14_FLDSHFT (16)
+#define HWIO_DDR_CTL_409_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_409_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_409_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_409_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_410_REGOFF 0x668
+#define HWIO_DDR_CTL_410_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_410_REGOFF)
+#define HWIO_DDR_CTL_410_AXI0_RANGE_RID_CHECK_BITS_14_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_410_AXI0_RANGE_RID_CHECK_BITS_14_FLDSHFT (0)
+#define HWIO_DDR_CTL_410_AXI0_RANGE_WID_CHECK_BITS_14_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_410_AXI0_RANGE_WID_CHECK_BITS_14_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_411_REGOFF 0x66c
+#define HWIO_DDR_CTL_411_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_411_REGOFF)
+#define HWIO_DDR_CTL_411_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_411_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_411_AXI0_RANGE_PROT_BITS_15_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_411_AXI0_RANGE_PROT_BITS_15_FLDSHFT (16)
+#define HWIO_DDR_CTL_411_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_411_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_411_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_411_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_412_REGOFF 0x670
+#define HWIO_DDR_CTL_412_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_412_REGOFF)
+#define HWIO_DDR_CTL_412_AXI0_RANGE_RID_CHECK_BITS_15_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_412_AXI0_RANGE_RID_CHECK_BITS_15_FLDSHFT (0)
+#define HWIO_DDR_CTL_412_AXI0_RANGE_WID_CHECK_BITS_15_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_412_AXI0_RANGE_WID_CHECK_BITS_15_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_413_REGOFF 0x674
+#define HWIO_DDR_CTL_413_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_413_REGOFF)
+#define HWIO_DDR_CTL_413_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_413_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_413_AXI1_RANGE_PROT_BITS_0_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_413_AXI1_RANGE_PROT_BITS_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_413_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_413_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_413_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_413_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_414_REGOFF 0x678
+#define HWIO_DDR_CTL_414_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_414_REGOFF)
+#define HWIO_DDR_CTL_414_AXI1_RANGE_RID_CHECK_BITS_0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_414_AXI1_RANGE_RID_CHECK_BITS_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_414_AXI1_RANGE_WID_CHECK_BITS_0_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_414_AXI1_RANGE_WID_CHECK_BITS_0_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_415_REGOFF 0x67c
+#define HWIO_DDR_CTL_415_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_415_REGOFF)
+#define HWIO_DDR_CTL_415_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_415_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_415_AXI1_RANGE_PROT_BITS_1_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_415_AXI1_RANGE_PROT_BITS_1_FLDSHFT (16)
+#define HWIO_DDR_CTL_415_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_415_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_415_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_415_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_416_REGOFF 0x680
+#define HWIO_DDR_CTL_416_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_416_REGOFF)
+#define HWIO_DDR_CTL_416_AXI1_RANGE_RID_CHECK_BITS_1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_416_AXI1_RANGE_RID_CHECK_BITS_1_FLDSHFT (0)
+#define HWIO_DDR_CTL_416_AXI1_RANGE_WID_CHECK_BITS_1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_416_AXI1_RANGE_WID_CHECK_BITS_1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_417_REGOFF 0x684
+#define HWIO_DDR_CTL_417_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_417_REGOFF)
+#define HWIO_DDR_CTL_417_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_417_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_417_AXI1_RANGE_PROT_BITS_2_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_417_AXI1_RANGE_PROT_BITS_2_FLDSHFT (16)
+#define HWIO_DDR_CTL_417_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_417_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_417_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_417_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_418_REGOFF 0x688
+#define HWIO_DDR_CTL_418_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_418_REGOFF)
+#define HWIO_DDR_CTL_418_AXI1_RANGE_RID_CHECK_BITS_2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_418_AXI1_RANGE_RID_CHECK_BITS_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_418_AXI1_RANGE_WID_CHECK_BITS_2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_418_AXI1_RANGE_WID_CHECK_BITS_2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_419_REGOFF 0x68c
+#define HWIO_DDR_CTL_419_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_419_REGOFF)
+#define HWIO_DDR_CTL_419_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_419_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_419_AXI1_RANGE_PROT_BITS_3_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_419_AXI1_RANGE_PROT_BITS_3_FLDSHFT (16)
+#define HWIO_DDR_CTL_419_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_419_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_419_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_419_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_420_REGOFF 0x690
+#define HWIO_DDR_CTL_420_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_420_REGOFF)
+#define HWIO_DDR_CTL_420_AXI1_RANGE_RID_CHECK_BITS_3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_420_AXI1_RANGE_RID_CHECK_BITS_3_FLDSHFT (0)
+#define HWIO_DDR_CTL_420_AXI1_RANGE_WID_CHECK_BITS_3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_420_AXI1_RANGE_WID_CHECK_BITS_3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_421_REGOFF 0x694
+#define HWIO_DDR_CTL_421_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_421_REGOFF)
+#define HWIO_DDR_CTL_421_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_421_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_421_AXI1_RANGE_PROT_BITS_4_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_421_AXI1_RANGE_PROT_BITS_4_FLDSHFT (16)
+#define HWIO_DDR_CTL_421_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_421_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_421_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_421_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_422_REGOFF 0x698
+#define HWIO_DDR_CTL_422_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_422_REGOFF)
+#define HWIO_DDR_CTL_422_AXI1_RANGE_RID_CHECK_BITS_4_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_422_AXI1_RANGE_RID_CHECK_BITS_4_FLDSHFT (0)
+#define HWIO_DDR_CTL_422_AXI1_RANGE_WID_CHECK_BITS_4_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_422_AXI1_RANGE_WID_CHECK_BITS_4_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_423_REGOFF 0x69c
+#define HWIO_DDR_CTL_423_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_423_REGOFF)
+#define HWIO_DDR_CTL_423_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_423_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_423_AXI1_RANGE_PROT_BITS_5_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_423_AXI1_RANGE_PROT_BITS_5_FLDSHFT (16)
+#define HWIO_DDR_CTL_423_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_423_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_423_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_423_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_424_REGOFF 0x6a0
+#define HWIO_DDR_CTL_424_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_424_REGOFF)
+#define HWIO_DDR_CTL_424_AXI1_RANGE_RID_CHECK_BITS_5_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_424_AXI1_RANGE_RID_CHECK_BITS_5_FLDSHFT (0)
+#define HWIO_DDR_CTL_424_AXI1_RANGE_WID_CHECK_BITS_5_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_424_AXI1_RANGE_WID_CHECK_BITS_5_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_425_REGOFF 0x6a4
+#define HWIO_DDR_CTL_425_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_425_REGOFF)
+#define HWIO_DDR_CTL_425_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_425_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_425_AXI1_RANGE_PROT_BITS_6_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_425_AXI1_RANGE_PROT_BITS_6_FLDSHFT (16)
+#define HWIO_DDR_CTL_425_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_425_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_425_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_425_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_426_REGOFF 0x6a8
+#define HWIO_DDR_CTL_426_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_426_REGOFF)
+#define HWIO_DDR_CTL_426_AXI1_RANGE_RID_CHECK_BITS_6_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_426_AXI1_RANGE_RID_CHECK_BITS_6_FLDSHFT (0)
+#define HWIO_DDR_CTL_426_AXI1_RANGE_WID_CHECK_BITS_6_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_426_AXI1_RANGE_WID_CHECK_BITS_6_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_427_REGOFF 0x6ac
+#define HWIO_DDR_CTL_427_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_427_REGOFF)
+#define HWIO_DDR_CTL_427_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_427_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_427_AXI1_RANGE_PROT_BITS_7_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_427_AXI1_RANGE_PROT_BITS_7_FLDSHFT (16)
+#define HWIO_DDR_CTL_427_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_427_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_427_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_427_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_428_REGOFF 0x6b0
+#define HWIO_DDR_CTL_428_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_428_REGOFF)
+#define HWIO_DDR_CTL_428_AXI1_RANGE_RID_CHECK_BITS_7_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_428_AXI1_RANGE_RID_CHECK_BITS_7_FLDSHFT (0)
+#define HWIO_DDR_CTL_428_AXI1_RANGE_WID_CHECK_BITS_7_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_428_AXI1_RANGE_WID_CHECK_BITS_7_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_429_REGOFF 0x6b4
+#define HWIO_DDR_CTL_429_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_429_REGOFF)
+#define HWIO_DDR_CTL_429_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_429_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_429_AXI1_RANGE_PROT_BITS_8_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_429_AXI1_RANGE_PROT_BITS_8_FLDSHFT (16)
+#define HWIO_DDR_CTL_429_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_429_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_429_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_429_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_430_REGOFF 0x6b8
+#define HWIO_DDR_CTL_430_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_430_REGOFF)
+#define HWIO_DDR_CTL_430_AXI1_RANGE_RID_CHECK_BITS_8_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_430_AXI1_RANGE_RID_CHECK_BITS_8_FLDSHFT (0)
+#define HWIO_DDR_CTL_430_AXI1_RANGE_WID_CHECK_BITS_8_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_430_AXI1_RANGE_WID_CHECK_BITS_8_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_431_REGOFF 0x6bc
+#define HWIO_DDR_CTL_431_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_431_REGOFF)
+#define HWIO_DDR_CTL_431_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_431_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_431_AXI1_RANGE_PROT_BITS_9_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_431_AXI1_RANGE_PROT_BITS_9_FLDSHFT (16)
+#define HWIO_DDR_CTL_431_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_431_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_431_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_431_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_432_REGOFF 0x6c0
+#define HWIO_DDR_CTL_432_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_432_REGOFF)
+#define HWIO_DDR_CTL_432_AXI1_RANGE_RID_CHECK_BITS_9_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_432_AXI1_RANGE_RID_CHECK_BITS_9_FLDSHFT (0)
+#define HWIO_DDR_CTL_432_AXI1_RANGE_WID_CHECK_BITS_9_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_432_AXI1_RANGE_WID_CHECK_BITS_9_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_433_REGOFF 0x6c4
+#define HWIO_DDR_CTL_433_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_433_REGOFF)
+#define HWIO_DDR_CTL_433_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_433_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_433_AXI1_RANGE_PROT_BITS_10_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_433_AXI1_RANGE_PROT_BITS_10_FLDSHFT (16)
+#define HWIO_DDR_CTL_433_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_433_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_433_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_433_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_434_REGOFF 0x6c8
+#define HWIO_DDR_CTL_434_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_434_REGOFF)
+#define HWIO_DDR_CTL_434_AXI1_RANGE_RID_CHECK_BITS_10_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_434_AXI1_RANGE_RID_CHECK_BITS_10_FLDSHFT (0)
+#define HWIO_DDR_CTL_434_AXI1_RANGE_WID_CHECK_BITS_10_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_434_AXI1_RANGE_WID_CHECK_BITS_10_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_435_REGOFF 0x6cc
+#define HWIO_DDR_CTL_435_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_435_REGOFF)
+#define HWIO_DDR_CTL_435_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_435_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_435_AXI1_RANGE_PROT_BITS_11_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_435_AXI1_RANGE_PROT_BITS_11_FLDSHFT (16)
+#define HWIO_DDR_CTL_435_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_435_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_435_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_435_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_436_REGOFF 0x6d0
+#define HWIO_DDR_CTL_436_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_436_REGOFF)
+#define HWIO_DDR_CTL_436_AXI1_RANGE_RID_CHECK_BITS_11_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_436_AXI1_RANGE_RID_CHECK_BITS_11_FLDSHFT (0)
+#define HWIO_DDR_CTL_436_AXI1_RANGE_WID_CHECK_BITS_11_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_436_AXI1_RANGE_WID_CHECK_BITS_11_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_437_REGOFF 0x6d4
+#define HWIO_DDR_CTL_437_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_437_REGOFF)
+#define HWIO_DDR_CTL_437_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_437_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_437_AXI1_RANGE_PROT_BITS_12_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_437_AXI1_RANGE_PROT_BITS_12_FLDSHFT (16)
+#define HWIO_DDR_CTL_437_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_437_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_437_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_437_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_438_REGOFF 0x6d8
+#define HWIO_DDR_CTL_438_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_438_REGOFF)
+#define HWIO_DDR_CTL_438_AXI1_RANGE_RID_CHECK_BITS_12_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_438_AXI1_RANGE_RID_CHECK_BITS_12_FLDSHFT (0)
+#define HWIO_DDR_CTL_438_AXI1_RANGE_WID_CHECK_BITS_12_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_438_AXI1_RANGE_WID_CHECK_BITS_12_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_439_REGOFF 0x6dc
+#define HWIO_DDR_CTL_439_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_439_REGOFF)
+#define HWIO_DDR_CTL_439_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_439_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_439_AXI1_RANGE_PROT_BITS_13_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_439_AXI1_RANGE_PROT_BITS_13_FLDSHFT (16)
+#define HWIO_DDR_CTL_439_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_439_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_439_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_439_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_440_REGOFF 0x6e0
+#define HWIO_DDR_CTL_440_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_440_REGOFF)
+#define HWIO_DDR_CTL_440_AXI1_RANGE_RID_CHECK_BITS_13_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_440_AXI1_RANGE_RID_CHECK_BITS_13_FLDSHFT (0)
+#define HWIO_DDR_CTL_440_AXI1_RANGE_WID_CHECK_BITS_13_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_440_AXI1_RANGE_WID_CHECK_BITS_13_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_441_REGOFF 0x6e4
+#define HWIO_DDR_CTL_441_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_441_REGOFF)
+#define HWIO_DDR_CTL_441_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_441_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_441_AXI1_RANGE_PROT_BITS_14_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_441_AXI1_RANGE_PROT_BITS_14_FLDSHFT (16)
+#define HWIO_DDR_CTL_441_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_441_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_441_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_441_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_442_REGOFF 0x6e8
+#define HWIO_DDR_CTL_442_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_442_REGOFF)
+#define HWIO_DDR_CTL_442_AXI1_RANGE_RID_CHECK_BITS_14_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_442_AXI1_RANGE_RID_CHECK_BITS_14_FLDSHFT (0)
+#define HWIO_DDR_CTL_442_AXI1_RANGE_WID_CHECK_BITS_14_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_442_AXI1_RANGE_WID_CHECK_BITS_14_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_443_REGOFF 0x6ec
+#define HWIO_DDR_CTL_443_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_443_REGOFF)
+#define HWIO_DDR_CTL_443_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_443_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_443_AXI1_RANGE_PROT_BITS_15_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_443_AXI1_RANGE_PROT_BITS_15_FLDSHFT (16)
+#define HWIO_DDR_CTL_443_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_443_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_443_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_443_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_444_REGOFF 0x6f0
+#define HWIO_DDR_CTL_444_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_444_REGOFF)
+#define HWIO_DDR_CTL_444_AXI1_RANGE_RID_CHECK_BITS_15_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_444_AXI1_RANGE_RID_CHECK_BITS_15_FLDSHFT (0)
+#define HWIO_DDR_CTL_444_AXI1_RANGE_WID_CHECK_BITS_15_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_444_AXI1_RANGE_WID_CHECK_BITS_15_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_445_REGOFF 0x6f4
+#define HWIO_DDR_CTL_445_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_445_REGOFF)
+#define HWIO_DDR_CTL_445_OBSOLETE0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_445_OBSOLETE0_FLDSHFT (0)
+#define HWIO_DDR_CTL_445_WEIGHTED_ROUND_ROBIN_LATENCY_CONTROL_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_445_WEIGHTED_ROUND_ROBIN_LATENCY_CONTROL_FLDSHFT (16)
+#define HWIO_DDR_CTL_445_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_445_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_CTL_445_WEIGHTED_ROUND_ROBIN_WEIGHT_SHARING_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_445_WEIGHTED_ROUND_ROBIN_WEIGHT_SHARING_FLDSHFT (24)
+#define HWIO_DDR_CTL_445_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_445_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_446_REGOFF 0x6f8
+#define HWIO_DDR_CTL_446_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_446_REGOFF)
+#define HWIO_DDR_CTL_446_WRR_PARAM_VALUE_ERR_FLDMASK (0xf)
+#define HWIO_DDR_CTL_446_WRR_PARAM_VALUE_ERR_FLDSHFT (0)
+#define HWIO_DDR_CTL_446_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_446_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_446_AXI0_PRIORITY0_RELATIVE_PRIORITY_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_446_AXI0_PRIORITY0_RELATIVE_PRIORITY_FLDSHFT (8)
+#define HWIO_DDR_CTL_446_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_446_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_446_AXI0_PRIORITY1_RELATIVE_PRIORITY_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_446_AXI0_PRIORITY1_RELATIVE_PRIORITY_FLDSHFT (16)
+#define HWIO_DDR_CTL_446_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_446_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_446_AXI0_PRIORITY2_RELATIVE_PRIORITY_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_446_AXI0_PRIORITY2_RELATIVE_PRIORITY_FLDSHFT (24)
+#define HWIO_DDR_CTL_446_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_446_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_447_REGOFF 0x6fc
+#define HWIO_DDR_CTL_447_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_447_REGOFF)
+#define HWIO_DDR_CTL_447_AXI0_PRIORITY3_RELATIVE_PRIORITY_FLDMASK (0xf)
+#define HWIO_DDR_CTL_447_AXI0_PRIORITY3_RELATIVE_PRIORITY_FLDSHFT (0)
+#define HWIO_DDR_CTL_447_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_447_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_447_AXI0_PORT_ORDERING_FLDMASK (0x100)
+#define HWIO_DDR_CTL_447_AXI0_PORT_ORDERING_FLDSHFT (8)
+#define HWIO_DDR_CTL_447_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_447_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_447_AXI0_PRIORITY_RELAX_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_447_AXI0_PRIORITY_RELAX_FLDSHFT (16)
+#define HWIO_DDR_CTL_447_CDNS_INTRL2_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_447_CDNS_INTRL2_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_448_REGOFF 0x700
+#define HWIO_DDR_CTL_448_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_448_REGOFF)
+#define HWIO_DDR_CTL_448_AXI1_PRIORITY0_RELATIVE_PRIORITY_FLDMASK (0xf)
+#define HWIO_DDR_CTL_448_AXI1_PRIORITY0_RELATIVE_PRIORITY_FLDSHFT (0)
+#define HWIO_DDR_CTL_448_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_448_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_448_AXI1_PRIORITY1_RELATIVE_PRIORITY_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_448_AXI1_PRIORITY1_RELATIVE_PRIORITY_FLDSHFT (8)
+#define HWIO_DDR_CTL_448_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_448_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_448_AXI1_PRIORITY2_RELATIVE_PRIORITY_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_448_AXI1_PRIORITY2_RELATIVE_PRIORITY_FLDSHFT (16)
+#define HWIO_DDR_CTL_448_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_448_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_448_AXI1_PRIORITY3_RELATIVE_PRIORITY_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_448_AXI1_PRIORITY3_RELATIVE_PRIORITY_FLDSHFT (24)
+#define HWIO_DDR_CTL_448_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_448_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_449_REGOFF 0x704
+#define HWIO_DDR_CTL_449_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_449_REGOFF)
+#define HWIO_DDR_CTL_449_AXI1_PORT_ORDERING_FLDMASK (0x1)
+#define HWIO_DDR_CTL_449_AXI1_PORT_ORDERING_FLDSHFT (0)
+#define HWIO_DDR_CTL_449_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_449_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_449_AXI1_PRIORITY_RELAX_FLDMASK (0x3ff00)
+#define HWIO_DDR_CTL_449_AXI1_PRIORITY_RELAX_FLDSHFT (8)
+#define HWIO_DDR_CTL_449_CDNS_INTRL1_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_449_CDNS_INTRL1_FLDSHFT (18)
+#define HWIO_DDR_CTL_449_CKE_STATUS_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_449_CKE_STATUS_FLDSHFT (24)
+#define HWIO_DDR_CTL_449_CDNS_INTRL2_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_449_CDNS_INTRL2_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_450_REGOFF 0x708
+#define HWIO_DDR_CTL_450_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_450_REGOFF)
+#define HWIO_DDR_CTL_450_MEM_RST_VALID_FLDMASK (0x1)
+#define HWIO_DDR_CTL_450_MEM_RST_VALID_FLDSHFT (0)
+#define HWIO_DDR_CTL_450_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_450_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_450_DLL_RST_DELAY_FLDMASK (0xffff00)
+#define HWIO_DDR_CTL_450_DLL_RST_DELAY_FLDSHFT (8)
+#define HWIO_DDR_CTL_450_DLL_RST_ADJ_DLY_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_450_DLL_RST_ADJ_DLY_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_451_REGOFF 0x70c
+#define HWIO_DDR_CTL_451_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_451_REGOFF)
+#define HWIO_DDR_CTL_451_TDFI_PHY_WRLAT_FLDMASK (0x7f)
+#define HWIO_DDR_CTL_451_TDFI_PHY_WRLAT_FLDSHFT (0)
+#define HWIO_DDR_CTL_451_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_CTL_451_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_CTL_451_UPDATE_ERROR_STATUS_FLDMASK (0x7f00)
+#define HWIO_DDR_CTL_451_UPDATE_ERROR_STATUS_FLDSHFT (8)
+#define HWIO_DDR_CTL_451_CDNS_INTRL1_FLDMASK (0x8000)
+#define HWIO_DDR_CTL_451_CDNS_INTRL1_FLDSHFT (15)
+#define HWIO_DDR_CTL_451_TDFI_PHY_RDLAT_F0_FLDMASK (0x7f0000)
+#define HWIO_DDR_CTL_451_TDFI_PHY_RDLAT_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_451_CDNS_INTRL2_FLDMASK (0x800000)
+#define HWIO_DDR_CTL_451_CDNS_INTRL2_FLDSHFT (23)
+#define HWIO_DDR_CTL_451_TDFI_PHY_RDLAT_F1_FLDMASK (0x7f000000)
+#define HWIO_DDR_CTL_451_TDFI_PHY_RDLAT_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_451_CDNS_INTRL3_FLDMASK (0x80000000)
+#define HWIO_DDR_CTL_451_CDNS_INTRL3_FLDSHFT (31)
+
+#define HWIO_DDR_CTL_452_REGOFF 0x710
+#define HWIO_DDR_CTL_452_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_452_REGOFF)
+#define HWIO_DDR_CTL_452_TDFI_PHY_RDLAT_F2_FLDMASK (0x7f)
+#define HWIO_DDR_CTL_452_TDFI_PHY_RDLAT_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_452_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_CTL_452_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_CTL_452_TDFI_PHY_RDLAT_F3_FLDMASK (0x7f00)
+#define HWIO_DDR_CTL_452_TDFI_PHY_RDLAT_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_452_CDNS_INTRL1_FLDMASK (0x8000)
+#define HWIO_DDR_CTL_452_CDNS_INTRL1_FLDSHFT (15)
+#define HWIO_DDR_CTL_452_TDFI_RDDATA_EN_FLDMASK (0x7f0000)
+#define HWIO_DDR_CTL_452_TDFI_RDDATA_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_452_CDNS_INTRL2_FLDMASK (0x800000)
+#define HWIO_DDR_CTL_452_CDNS_INTRL2_FLDSHFT (23)
+#define HWIO_DDR_CTL_452_DRAM_CLK_DISABLE_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_452_DRAM_CLK_DISABLE_FLDSHFT (24)
+#define HWIO_DDR_CTL_452_CDNS_INTRL3_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_452_CDNS_INTRL3_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_453_REGOFF 0x714
+#define HWIO_DDR_CTL_453_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_453_REGOFF)
+#define HWIO_DDR_CTL_453_TDFI_CTRLUPD_MIN_FLDMASK (0xf)
+#define HWIO_DDR_CTL_453_TDFI_CTRLUPD_MIN_FLDSHFT (0)
+#define HWIO_DDR_CTL_453_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_453_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_453_TDFI_CTRLUPD_MAX_F0_FLDMASK (0xffff00)
+#define HWIO_DDR_CTL_453_TDFI_CTRLUPD_MAX_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_453_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_453_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_454_REGOFF 0x718
+#define HWIO_DDR_CTL_454_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_454_REGOFF)
+#define HWIO_DDR_CTL_454_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_454_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_455_REGOFF 0x71c
+#define HWIO_DDR_CTL_455_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_455_REGOFF)
+#define HWIO_DDR_CTL_455_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_455_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_456_REGOFF 0x720
+#define HWIO_DDR_CTL_456_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_456_REGOFF)
+#define HWIO_DDR_CTL_456_TDFI_PHYUPD_RESP_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_456_TDFI_PHYUPD_RESP_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_456_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_456_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_457_REGOFF 0x724
+#define HWIO_DDR_CTL_457_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_457_REGOFF)
+#define HWIO_DDR_CTL_457_TDFI_CTRLUPD_INTERVAL_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_457_TDFI_CTRLUPD_INTERVAL_F0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_458_REGOFF 0x728
+#define HWIO_DDR_CTL_458_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_458_REGOFF)
+#define HWIO_DDR_CTL_458_RDLAT_ADJ_F0_FLDMASK (0x7f)
+#define HWIO_DDR_CTL_458_RDLAT_ADJ_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_458_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_CTL_458_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_CTL_458_WRLAT_ADJ_F0_FLDMASK (0x7f00)
+#define HWIO_DDR_CTL_458_WRLAT_ADJ_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_458_CDNS_INTRL1_FLDMASK (0x8000)
+#define HWIO_DDR_CTL_458_CDNS_INTRL1_FLDSHFT (15)
+#define HWIO_DDR_CTL_458_TDFI_CTRLUPD_MAX_F1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_458_TDFI_CTRLUPD_MAX_F1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_459_REGOFF 0x72c
+#define HWIO_DDR_CTL_459_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_459_REGOFF)
+#define HWIO_DDR_CTL_459_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_459_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_460_REGOFF 0x730
+#define HWIO_DDR_CTL_460_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_460_REGOFF)
+#define HWIO_DDR_CTL_460_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_460_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_461_REGOFF 0x734
+#define HWIO_DDR_CTL_461_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_461_REGOFF)
+#define HWIO_DDR_CTL_461_TDFI_PHYUPD_RESP_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_461_TDFI_PHYUPD_RESP_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_461_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_461_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_462_REGOFF 0x738
+#define HWIO_DDR_CTL_462_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_462_REGOFF)
+#define HWIO_DDR_CTL_462_TDFI_CTRLUPD_INTERVAL_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_462_TDFI_CTRLUPD_INTERVAL_F1_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_463_REGOFF 0x73c
+#define HWIO_DDR_CTL_463_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_463_REGOFF)
+#define HWIO_DDR_CTL_463_RDLAT_ADJ_F1_FLDMASK (0x7f)
+#define HWIO_DDR_CTL_463_RDLAT_ADJ_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_463_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_CTL_463_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_CTL_463_WRLAT_ADJ_F1_FLDMASK (0x7f00)
+#define HWIO_DDR_CTL_463_WRLAT_ADJ_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_463_CDNS_INTRL1_FLDMASK (0x8000)
+#define HWIO_DDR_CTL_463_CDNS_INTRL1_FLDSHFT (15)
+#define HWIO_DDR_CTL_463_TDFI_CTRLUPD_MAX_F2_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_463_TDFI_CTRLUPD_MAX_F2_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_464_REGOFF 0x740
+#define HWIO_DDR_CTL_464_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_464_REGOFF)
+#define HWIO_DDR_CTL_464_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_464_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_465_REGOFF 0x744
+#define HWIO_DDR_CTL_465_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_465_REGOFF)
+#define HWIO_DDR_CTL_465_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_465_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_466_REGOFF 0x748
+#define HWIO_DDR_CTL_466_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_466_REGOFF)
+#define HWIO_DDR_CTL_466_TDFI_PHYUPD_RESP_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_466_TDFI_PHYUPD_RESP_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_466_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_466_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_467_REGOFF 0x74c
+#define HWIO_DDR_CTL_467_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_467_REGOFF)
+#define HWIO_DDR_CTL_467_TDFI_CTRLUPD_INTERVAL_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_467_TDFI_CTRLUPD_INTERVAL_F2_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_468_REGOFF 0x750
+#define HWIO_DDR_CTL_468_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_468_REGOFF)
+#define HWIO_DDR_CTL_468_RDLAT_ADJ_F2_FLDMASK (0x7f)
+#define HWIO_DDR_CTL_468_RDLAT_ADJ_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_468_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_CTL_468_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_CTL_468_WRLAT_ADJ_F2_FLDMASK (0x7f00)
+#define HWIO_DDR_CTL_468_WRLAT_ADJ_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_468_CDNS_INTRL1_FLDMASK (0x8000)
+#define HWIO_DDR_CTL_468_CDNS_INTRL1_FLDSHFT (15)
+#define HWIO_DDR_CTL_468_TDFI_CTRLUPD_MAX_F3_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_468_TDFI_CTRLUPD_MAX_F3_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_469_REGOFF 0x754
+#define HWIO_DDR_CTL_469_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_469_REGOFF)
+#define HWIO_DDR_CTL_469_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_469_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_470_REGOFF 0x758
+#define HWIO_DDR_CTL_470_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_470_REGOFF)
+#define HWIO_DDR_CTL_470_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_470_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_471_REGOFF 0x75c
+#define HWIO_DDR_CTL_471_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_471_REGOFF)
+#define HWIO_DDR_CTL_471_TDFI_PHYUPD_RESP_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_471_TDFI_PHYUPD_RESP_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_471_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_471_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_472_REGOFF 0x760
+#define HWIO_DDR_CTL_472_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_472_REGOFF)
+#define HWIO_DDR_CTL_472_TDFI_CTRLUPD_INTERVAL_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_472_TDFI_CTRLUPD_INTERVAL_F3_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_473_REGOFF 0x764
+#define HWIO_DDR_CTL_473_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_473_REGOFF)
+#define HWIO_DDR_CTL_473_RDLAT_ADJ_F3_FLDMASK (0x7f)
+#define HWIO_DDR_CTL_473_RDLAT_ADJ_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_473_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_CTL_473_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_CTL_473_WRLAT_ADJ_F3_FLDMASK (0x7f00)
+#define HWIO_DDR_CTL_473_WRLAT_ADJ_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_473_CDNS_INTRL1_FLDMASK (0x8000)
+#define HWIO_DDR_CTL_473_CDNS_INTRL1_FLDSHFT (15)
+#define HWIO_DDR_CTL_473_TDFI_CTRL_DELAY_F0_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_473_TDFI_CTRL_DELAY_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_473_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_473_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_473_TDFI_CTRL_DELAY_F1_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_473_TDFI_CTRL_DELAY_F1_FLDSHFT (24)
+#define HWIO_DDR_CTL_473_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_473_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_474_REGOFF 0x768
+#define HWIO_DDR_CTL_474_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_474_REGOFF)
+#define HWIO_DDR_CTL_474_TDFI_CTRL_DELAY_F2_FLDMASK (0xf)
+#define HWIO_DDR_CTL_474_TDFI_CTRL_DELAY_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_474_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_474_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_474_TDFI_CTRL_DELAY_F3_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_474_TDFI_CTRL_DELAY_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_474_CDNS_INTRL1_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_474_CDNS_INTRL1_FLDSHFT (12)
+#define HWIO_DDR_CTL_474_TDFI_DRAM_CLK_DISABLE_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_474_TDFI_DRAM_CLK_DISABLE_FLDSHFT (16)
+#define HWIO_DDR_CTL_474_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_474_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_474_TDFI_DRAM_CLK_ENABLE_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_474_TDFI_DRAM_CLK_ENABLE_FLDSHFT (24)
+#define HWIO_DDR_CTL_474_CDNS_INTRL3_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_474_CDNS_INTRL3_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_475_REGOFF 0x76c
+#define HWIO_DDR_CTL_475_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_475_REGOFF)
+#define HWIO_DDR_CTL_475_TDFI_WRLVL_EN_FLDMASK (0xff)
+#define HWIO_DDR_CTL_475_TDFI_WRLVL_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_475_TDFI_WRLVL_WW_FLDMASK (0x3ff00)
+#define HWIO_DDR_CTL_475_TDFI_WRLVL_WW_FLDSHFT (8)
+#define HWIO_DDR_CTL_475_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_475_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_475_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_475_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_476_REGOFF 0x770
+#define HWIO_DDR_CTL_476_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_476_REGOFF)
+#define HWIO_DDR_CTL_476_TDFI_WRLVL_RESP_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_476_TDFI_WRLVL_RESP_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_477_REGOFF 0x774
+#define HWIO_DDR_CTL_477_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_477_REGOFF)
+#define HWIO_DDR_CTL_477_TDFI_WRLVL_MAX_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_477_TDFI_WRLVL_MAX_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_478_REGOFF 0x778
+#define HWIO_DDR_CTL_478_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_478_REGOFF)
+#define HWIO_DDR_CTL_478_TDFI_RDLVL_EN_FLDMASK (0xff)
+#define HWIO_DDR_CTL_478_TDFI_RDLVL_EN_FLDSHFT (0)
+#define HWIO_DDR_CTL_478_TDFI_RDLVL_RR_FLDMASK (0x3ff00)
+#define HWIO_DDR_CTL_478_TDFI_RDLVL_RR_FLDSHFT (8)
+#define HWIO_DDR_CTL_478_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_478_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_CTL_478_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_478_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_479_REGOFF 0x77c
+#define HWIO_DDR_CTL_479_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_479_REGOFF)
+#define HWIO_DDR_CTL_479_TDFI_RDLVL_RESP_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_479_TDFI_RDLVL_RESP_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_480_REGOFF 0x780
+#define HWIO_DDR_CTL_480_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_480_REGOFF)
+#define HWIO_DDR_CTL_480_RDLVL_RESP_MASK_FLDMASK (0xff)
+#define HWIO_DDR_CTL_480_RDLVL_RESP_MASK_FLDSHFT (0)
+#define HWIO_DDR_CTL_480_RDLVL_EN_FLDMASK (0x100)
+#define HWIO_DDR_CTL_480_RDLVL_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_480_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_480_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_480_RDLVL_GATE_EN_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_480_RDLVL_GATE_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_480_CDNS_INTRL1_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_480_CDNS_INTRL1_FLDSHFT (17)
+#define HWIO_DDR_CTL_480_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_480_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_481_REGOFF 0x784
+#define HWIO_DDR_CTL_481_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_481_REGOFF)
+#define HWIO_DDR_CTL_481_TDFI_RDLVL_MAX_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_481_TDFI_RDLVL_MAX_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_482_REGOFF 0x788
+#define HWIO_DDR_CTL_482_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_482_REGOFF)
+#define HWIO_DDR_CTL_482_RDLVL_ERROR_STATUS_FLDMASK (0x3)
+#define HWIO_DDR_CTL_482_RDLVL_ERROR_STATUS_FLDSHFT (0)
+#define HWIO_DDR_CTL_482_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_CTL_482_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_CTL_482_RDLVL_GATE_ERROR_STATUS_FLDMASK (0x300)
+#define HWIO_DDR_CTL_482_RDLVL_GATE_ERROR_STATUS_FLDSHFT (8)
+#define HWIO_DDR_CTL_482_CDNS_INTRL1_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_482_CDNS_INTRL1_FLDSHFT (10)
+#define HWIO_DDR_CTL_482_TDFI_CALVL_EN_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_482_TDFI_CALVL_EN_FLDSHFT (16)
+#define HWIO_DDR_CTL_482_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_482_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_483_REGOFF 0x78c
+#define HWIO_DDR_CTL_483_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_483_REGOFF)
+#define HWIO_DDR_CTL_483_TDFI_CALVL_CC_F0_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_483_TDFI_CALVL_CC_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_483_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_483_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_483_TDFI_CALVL_CAPTURE_F0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_483_TDFI_CALVL_CAPTURE_F0_FLDSHFT (16)
+#define HWIO_DDR_CTL_483_CDNS_INTRL1_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_483_CDNS_INTRL1_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_484_REGOFF 0x790
+#define HWIO_DDR_CTL_484_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_484_REGOFF)
+#define HWIO_DDR_CTL_484_TDFI_CALVL_CC_F1_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_484_TDFI_CALVL_CC_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_484_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_484_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_484_TDFI_CALVL_CAPTURE_F1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_484_TDFI_CALVL_CAPTURE_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_484_CDNS_INTRL1_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_484_CDNS_INTRL1_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_485_REGOFF 0x794
+#define HWIO_DDR_CTL_485_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_485_REGOFF)
+#define HWIO_DDR_CTL_485_TDFI_CALVL_CC_F2_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_485_TDFI_CALVL_CC_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_485_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_485_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_485_TDFI_CALVL_CAPTURE_F2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_485_TDFI_CALVL_CAPTURE_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_485_CDNS_INTRL1_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_485_CDNS_INTRL1_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_486_REGOFF 0x798
+#define HWIO_DDR_CTL_486_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_486_REGOFF)
+#define HWIO_DDR_CTL_486_TDFI_CALVL_CC_F3_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_486_TDFI_CALVL_CC_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_486_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_486_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_486_TDFI_CALVL_CAPTURE_F3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_486_TDFI_CALVL_CAPTURE_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_486_CDNS_INTRL1_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_486_CDNS_INTRL1_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_487_REGOFF 0x79c
+#define HWIO_DDR_CTL_487_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_487_REGOFF)
+#define HWIO_DDR_CTL_487_TDFI_CALVL_RESP_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_487_TDFI_CALVL_RESP_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_488_REGOFF 0x7a0
+#define HWIO_DDR_CTL_488_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_488_REGOFF)
+#define HWIO_DDR_CTL_488_TDFI_CALVL_MAX_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_488_TDFI_CALVL_MAX_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_489_REGOFF 0x7a4
+#define HWIO_DDR_CTL_489_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_489_REGOFF)
+#define HWIO_DDR_CTL_489_CALVL_RESP_MASK_FLDMASK (0x1)
+#define HWIO_DDR_CTL_489_CALVL_RESP_MASK_FLDSHFT (0)
+#define HWIO_DDR_CTL_489_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_489_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_489_CALVL_EN_FLDMASK (0x100)
+#define HWIO_DDR_CTL_489_CALVL_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_489_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_489_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_489_CALVL_ERROR_STATUS_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_489_CALVL_ERROR_STATUS_FLDSHFT (16)
+#define HWIO_DDR_CTL_489_CDNS_INTRL2_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_489_CDNS_INTRL2_FLDSHFT (20)
+#define HWIO_DDR_CTL_489_TDFI_PHY_WRDATA_F0_FLDMASK (0x7000000)
+#define HWIO_DDR_CTL_489_TDFI_PHY_WRDATA_F0_FLDSHFT (24)
+#define HWIO_DDR_CTL_489_CDNS_INTRL3_FLDMASK (0xf8000000)
+#define HWIO_DDR_CTL_489_CDNS_INTRL3_FLDSHFT (27)
+
+#define HWIO_DDR_CTL_490_REGOFF 0x7a8
+#define HWIO_DDR_CTL_490_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_490_REGOFF)
+#define HWIO_DDR_CTL_490_TDFI_PHY_WRDATA_F1_FLDMASK (0x7)
+#define HWIO_DDR_CTL_490_TDFI_PHY_WRDATA_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_490_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_490_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_490_TDFI_PHY_WRDATA_F2_FLDMASK (0x700)
+#define HWIO_DDR_CTL_490_TDFI_PHY_WRDATA_F2_FLDSHFT (8)
+#define HWIO_DDR_CTL_490_CDNS_INTRL1_FLDMASK (0xf800)
+#define HWIO_DDR_CTL_490_CDNS_INTRL1_FLDSHFT (11)
+#define HWIO_DDR_CTL_490_TDFI_PHY_WRDATA_F3_FLDMASK (0x70000)
+#define HWIO_DDR_CTL_490_TDFI_PHY_WRDATA_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_490_CDNS_INTRL2_FLDMASK (0xf80000)
+#define HWIO_DDR_CTL_490_CDNS_INTRL2_FLDSHFT (19)
+#define HWIO_DDR_CTL_490_TDFI_RDCSLAT_F0_FLDMASK (0x7f000000)
+#define HWIO_DDR_CTL_490_TDFI_RDCSLAT_F0_FLDSHFT (24)
+#define HWIO_DDR_CTL_490_CDNS_INTRL3_FLDMASK (0x80000000)
+#define HWIO_DDR_CTL_490_CDNS_INTRL3_FLDSHFT (31)
+
+#define HWIO_DDR_CTL_491_REGOFF 0x7ac
+#define HWIO_DDR_CTL_491_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_491_REGOFF)
+#define HWIO_DDR_CTL_491_TDFI_WRCSLAT_F0_FLDMASK (0x7f)
+#define HWIO_DDR_CTL_491_TDFI_WRCSLAT_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_491_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_CTL_491_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_CTL_491_TDFI_RDCSLAT_F1_FLDMASK (0x7f00)
+#define HWIO_DDR_CTL_491_TDFI_RDCSLAT_F1_FLDSHFT (8)
+#define HWIO_DDR_CTL_491_CDNS_INTRL1_FLDMASK (0x8000)
+#define HWIO_DDR_CTL_491_CDNS_INTRL1_FLDSHFT (15)
+#define HWIO_DDR_CTL_491_TDFI_WRCSLAT_F1_FLDMASK (0x7f0000)
+#define HWIO_DDR_CTL_491_TDFI_WRCSLAT_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_491_CDNS_INTRL2_FLDMASK (0x800000)
+#define HWIO_DDR_CTL_491_CDNS_INTRL2_FLDSHFT (23)
+#define HWIO_DDR_CTL_491_TDFI_RDCSLAT_F2_FLDMASK (0x7f000000)
+#define HWIO_DDR_CTL_491_TDFI_RDCSLAT_F2_FLDSHFT (24)
+#define HWIO_DDR_CTL_491_CDNS_INTRL3_FLDMASK (0x80000000)
+#define HWIO_DDR_CTL_491_CDNS_INTRL3_FLDSHFT (31)
+
+#define HWIO_DDR_CTL_492_REGOFF 0x7b0
+#define HWIO_DDR_CTL_492_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_492_REGOFF)
+#define HWIO_DDR_CTL_492_TDFI_WRCSLAT_F2_FLDMASK (0x7f)
+#define HWIO_DDR_CTL_492_TDFI_WRCSLAT_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_492_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_CTL_492_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_CTL_492_TDFI_RDCSLAT_F3_FLDMASK (0x7f00)
+#define HWIO_DDR_CTL_492_TDFI_RDCSLAT_F3_FLDSHFT (8)
+#define HWIO_DDR_CTL_492_CDNS_INTRL1_FLDMASK (0x8000)
+#define HWIO_DDR_CTL_492_CDNS_INTRL1_FLDSHFT (15)
+#define HWIO_DDR_CTL_492_TDFI_WRCSLAT_F3_FLDMASK (0x7f0000)
+#define HWIO_DDR_CTL_492_TDFI_WRCSLAT_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_492_CDNS_INTRL2_FLDMASK (0x800000)
+#define HWIO_DDR_CTL_492_CDNS_INTRL2_FLDSHFT (23)
+#define HWIO_DDR_CTL_492_TDFI_WRDATA_DELAY_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_492_TDFI_WRDATA_DELAY_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_493_REGOFF 0x7b4
+#define HWIO_DDR_CTL_493_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_493_REGOFF)
+#define HWIO_DDR_CTL_493_EN_1T_TIMING_FLDMASK (0x1)
+#define HWIO_DDR_CTL_493_EN_1T_TIMING_FLDSHFT (0)
+#define HWIO_DDR_CTL_493_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_493_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_493_DISABLE_MEMORY_MASKED_WRITE_FLDMASK (0x100)
+#define HWIO_DDR_CTL_493_DISABLE_MEMORY_MASKED_WRITE_FLDSHFT (8)
+#define HWIO_DDR_CTL_493_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_493_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_493_BL_ON_FLY_ENABLE_FLDMASK (0x10000)
+#define HWIO_DDR_CTL_493_BL_ON_FLY_ENABLE_FLDSHFT (16)
+#define HWIO_DDR_CTL_493_CDNS_INTRL2_FLDMASK (0xfe0000)
+#define HWIO_DDR_CTL_493_CDNS_INTRL2_FLDSHFT (17)
+#define HWIO_DDR_CTL_493_CDNS_INTRL3_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_493_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_493_RESERVED4_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_493_RESERVED4_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_494_REGOFF 0x7b8
+#define HWIO_DDR_CTL_494_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_494_REGOFF)
+#define HWIO_DDR_CTL_494_CDNS_INTRL0_FLDMASK (0x7)
+#define HWIO_DDR_CTL_494_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_494_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_494_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_494_CDNS_INTRL1_FLDMASK (0x700)
+#define HWIO_DDR_CTL_494_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_494_RESERVED4_FLDMASK (0xf800)
+#define HWIO_DDR_CTL_494_RESERVED4_FLDSHFT (11)
+#define HWIO_DDR_CTL_494_CDNS_INTRL2_FLDMASK (0x70000)
+#define HWIO_DDR_CTL_494_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_494_RESERVED5_FLDMASK (0xf80000)
+#define HWIO_DDR_CTL_494_RESERVED5_FLDSHFT (19)
+#define HWIO_DDR_CTL_494_CDNS_INTRL3_FLDMASK (0x7000000)
+#define HWIO_DDR_CTL_494_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_494_RESERVED6_FLDMASK (0xf8000000)
+#define HWIO_DDR_CTL_494_RESERVED6_FLDSHFT (27)
+
+#define HWIO_DDR_CTL_495_REGOFF 0x7bc
+#define HWIO_DDR_CTL_495_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_495_REGOFF)
+#define HWIO_DDR_CTL_495_CDNS_INTRL0_FLDMASK (0x7)
+#define HWIO_DDR_CTL_495_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_495_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_CTL_495_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_CTL_495_CDNS_INTRL1_FLDMASK (0x700)
+#define HWIO_DDR_CTL_495_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_495_RESERVED4_FLDMASK (0xf800)
+#define HWIO_DDR_CTL_495_RESERVED4_FLDSHFT (11)
+#define HWIO_DDR_CTL_495_CDNS_INTRL2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_495_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_495_RESERVED5_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_495_RESERVED5_FLDSHFT (20)
+#define HWIO_DDR_CTL_495_CDNS_INTRL3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_495_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_495_RESERVED6_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_495_RESERVED6_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_496_REGOFF 0x7c0
+#define HWIO_DDR_CTL_496_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_496_REGOFF)
+#define HWIO_DDR_CTL_496_CDNS_INTRL0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_496_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_496_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_496_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_496_CDNS_INTRL1_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_496_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_496_RESERVED4_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_496_RESERVED4_FLDSHFT (12)
+#define HWIO_DDR_CTL_496_CDNS_INTRL2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_496_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_496_RESERVED5_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_496_RESERVED5_FLDSHFT (20)
+#define HWIO_DDR_CTL_496_CDNS_INTRL3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_496_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_496_RESERVED6_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_496_RESERVED6_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_497_REGOFF 0x7c4
+#define HWIO_DDR_CTL_497_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_497_REGOFF)
+#define HWIO_DDR_CTL_497_CDNS_INTRL0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_497_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_497_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_497_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_497_CDNS_INTRL1_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_497_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_497_RESERVED4_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_497_RESERVED4_FLDSHFT (12)
+#define HWIO_DDR_CTL_497_CDNS_INTRL2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_497_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_497_RESERVED5_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_497_RESERVED5_FLDSHFT (20)
+#define HWIO_DDR_CTL_497_CDNS_INTRL3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_497_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_497_RESERVED6_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_497_RESERVED6_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_498_REGOFF 0x7c8
+#define HWIO_DDR_CTL_498_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_498_REGOFF)
+#define HWIO_DDR_CTL_498_CDNS_INTRL0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_498_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_498_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_498_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_498_CDNS_INTRL1_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_498_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_498_RESERVED4_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_498_RESERVED4_FLDSHFT (12)
+#define HWIO_DDR_CTL_498_CDNS_INTRL2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_498_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_498_RESERVED5_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_498_RESERVED5_FLDSHFT (20)
+#define HWIO_DDR_CTL_498_CDNS_INTRL3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_498_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_498_RESERVED6_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_498_RESERVED6_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_499_REGOFF 0x7cc
+#define HWIO_DDR_CTL_499_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_499_REGOFF)
+#define HWIO_DDR_CTL_499_CDNS_INTRL0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_499_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_499_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_499_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_499_CDNS_INTRL1_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_499_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_499_RESERVED4_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_499_RESERVED4_FLDSHFT (12)
+#define HWIO_DDR_CTL_499_CDNS_INTRL2_FLDMASK (0xf0000)
+#define HWIO_DDR_CTL_499_CDNS_INTRL2_FLDSHFT (16)
+#define HWIO_DDR_CTL_499_RESERVED5_FLDMASK (0xf00000)
+#define HWIO_DDR_CTL_499_RESERVED5_FLDSHFT (20)
+#define HWIO_DDR_CTL_499_CDNS_INTRL3_FLDMASK (0xf000000)
+#define HWIO_DDR_CTL_499_CDNS_INTRL3_FLDSHFT (24)
+#define HWIO_DDR_CTL_499_RESERVED6_FLDMASK (0xf0000000)
+#define HWIO_DDR_CTL_499_RESERVED6_FLDSHFT (28)
+
+#define HWIO_DDR_CTL_500_REGOFF 0x7d0
+#define HWIO_DDR_CTL_500_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_500_REGOFF)
+#define HWIO_DDR_CTL_500_CDNS_INTRL0_FLDMASK (0xf)
+#define HWIO_DDR_CTL_500_CDNS_INTRL0_FLDSHFT (0)
+#define HWIO_DDR_CTL_500_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_500_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_500_CDNS_INTRL1_FLDMASK (0xf00)
+#define HWIO_DDR_CTL_500_CDNS_INTRL1_FLDSHFT (8)
+#define HWIO_DDR_CTL_500_CDNS_INTRL2_FLDMASK (0xf000)
+#define HWIO_DDR_CTL_500_CDNS_INTRL2_FLDSHFT (12)
+#define HWIO_DDR_CTL_500_SRAM_READ_LATENCY_FLDMASK (0x30000)
+#define HWIO_DDR_CTL_500_SRAM_READ_LATENCY_FLDSHFT (16)
+#define HWIO_DDR_CTL_500_CDNS_INTRL3_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_500_CDNS_INTRL3_FLDSHFT (18)
+#define HWIO_DDR_CTL_500_AXI0_WR_ARRAY_LOG2_DEPTH_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_500_AXI0_WR_ARRAY_LOG2_DEPTH_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_501_REGOFF 0x7d4
+#define HWIO_DDR_CTL_501_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_501_REGOFF)
+#define HWIO_DDR_CTL_501_AXI0_TRANS_WRFIFO_LOG2_DEPTH_FLDMASK (0xff)
+#define HWIO_DDR_CTL_501_AXI0_TRANS_WRFIFO_LOG2_DEPTH_FLDSHFT (0)
+#define HWIO_DDR_CTL_501_AXI1_WR_ARRAY_LOG2_DEPTH_FLDMASK (0xff00)
+#define HWIO_DDR_CTL_501_AXI1_WR_ARRAY_LOG2_DEPTH_FLDSHFT (8)
+#define HWIO_DDR_CTL_501_AXI1_TRANS_WRFIFO_LOG2_DEPTH_FLDMASK (0xff0000)
+#define HWIO_DDR_CTL_501_AXI1_TRANS_WRFIFO_LOG2_DEPTH_FLDSHFT (16)
+#define HWIO_DDR_CTL_501_PBR_EN_FLDMASK (0x1000000)
+#define HWIO_DDR_CTL_501_PBR_EN_FLDSHFT (24)
+#define HWIO_DDR_CTL_501_RESERVED_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_501_RESERVED_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_502_REGOFF 0x7d8
+#define HWIO_DDR_CTL_502_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_502_REGOFF)
+#define HWIO_DDR_CTL_502_PBR_NUMERIC_ORDER_FLDMASK (0x1)
+#define HWIO_DDR_CTL_502_PBR_NUMERIC_ORDER_FLDSHFT (0)
+#define HWIO_DDR_CTL_502_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_502_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_502_TRFC_PB_F0_FLDMASK (0x3ff00)
+#define HWIO_DDR_CTL_502_TRFC_PB_F0_FLDSHFT (8)
+#define HWIO_DDR_CTL_502_CDNS_INTRL1_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_502_CDNS_INTRL1_FLDSHFT (18)
+#define HWIO_DDR_CTL_502_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_502_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_503_REGOFF 0x7dc
+#define HWIO_DDR_CTL_503_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_503_REGOFF)
+#define HWIO_DDR_CTL_503_TREFI_PB_F0_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_503_TREFI_PB_F0_FLDSHFT (0)
+#define HWIO_DDR_CTL_503_TRFC_PB_F1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_503_TRFC_PB_F1_FLDSHFT (16)
+#define HWIO_DDR_CTL_503_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_503_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_504_REGOFF 0x7e0
+#define HWIO_DDR_CTL_504_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_504_REGOFF)
+#define HWIO_DDR_CTL_504_TREFI_PB_F1_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_504_TREFI_PB_F1_FLDSHFT (0)
+#define HWIO_DDR_CTL_504_TRFC_PB_F2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_504_TRFC_PB_F2_FLDSHFT (16)
+#define HWIO_DDR_CTL_504_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_504_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_505_REGOFF 0x7e4
+#define HWIO_DDR_CTL_505_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_505_REGOFF)
+#define HWIO_DDR_CTL_505_TREFI_PB_F2_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_505_TREFI_PB_F2_FLDSHFT (0)
+#define HWIO_DDR_CTL_505_TRFC_PB_F3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_CTL_505_TRFC_PB_F3_FLDSHFT (16)
+#define HWIO_DDR_CTL_505_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_CTL_505_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_CTL_506_REGOFF 0x7e8
+#define HWIO_DDR_CTL_506_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_506_REGOFF)
+#define HWIO_DDR_CTL_506_TREFI_PB_F3_FLDMASK (0xffff)
+#define HWIO_DDR_CTL_506_TREFI_PB_F3_FLDSHFT (0)
+#define HWIO_DDR_CTL_506_PBR_MAX_BANK_WAIT_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_506_PBR_MAX_BANK_WAIT_FLDSHFT (16)
+
+#define HWIO_DDR_CTL_507_REGOFF 0x7ec
+#define HWIO_DDR_CTL_507_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_507_REGOFF)
+#define HWIO_DDR_CTL_507_PBR_BANK_SELECT_DELAY_FLDMASK (0xf)
+#define HWIO_DDR_CTL_507_PBR_BANK_SELECT_DELAY_FLDSHFT (0)
+#define HWIO_DDR_CTL_507_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_CTL_507_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_CTL_507_PBR_CONT_REQ_EN_FLDMASK (0x100)
+#define HWIO_DDR_CTL_507_PBR_CONT_REQ_EN_FLDSHFT (8)
+#define HWIO_DDR_CTL_507_CDNS_INTRL1_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_507_CDNS_INTRL1_FLDSHFT (9)
+#define HWIO_DDR_CTL_507_AREF_PBR_CONT_EN_THRESHOLD_FLDMASK (0x1f0000)
+#define HWIO_DDR_CTL_507_AREF_PBR_CONT_EN_THRESHOLD_FLDSHFT (16)
+#define HWIO_DDR_CTL_507_CDNS_INTRL2_FLDMASK (0xe00000)
+#define HWIO_DDR_CTL_507_CDNS_INTRL2_FLDSHFT (21)
+#define HWIO_DDR_CTL_507_AREF_PBR_CONT_DIS_THRESHOLD_FLDMASK (0x1f000000)
+#define HWIO_DDR_CTL_507_AREF_PBR_CONT_DIS_THRESHOLD_FLDSHFT (24)
+#define HWIO_DDR_CTL_507_CDNS_INTRL3_FLDMASK (0xe0000000)
+#define HWIO_DDR_CTL_507_CDNS_INTRL3_FLDSHFT (29)
+
+#define HWIO_DDR_CTL_508_REGOFF 0x7f0
+#define HWIO_DDR_CTL_508_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_508_REGOFF)
+#define HWIO_DDR_CTL_508_WR_ORDER_REQ_FLDMASK (0x3)
+#define HWIO_DDR_CTL_508_WR_ORDER_REQ_FLDSHFT (0)
+#define HWIO_DDR_CTL_508_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_CTL_508_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_CTL_508_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_CTL_508_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_CTL_509_REGOFF 0x7f4
+#define HWIO_DDR_CTL_509_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_509_REGOFF)
+#define HWIO_DDR_CTL_509_TDFI_PHYUPD_TYPE0_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_509_TDFI_PHYUPD_TYPE0_F0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_510_REGOFF 0x7f8
+#define HWIO_DDR_CTL_510_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_510_REGOFF)
+#define HWIO_DDR_CTL_510_TDFI_PHYUPD_TYPE1_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_510_TDFI_PHYUPD_TYPE1_F0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_511_REGOFF 0x7fc
+#define HWIO_DDR_CTL_511_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_511_REGOFF)
+#define HWIO_DDR_CTL_511_TDFI_PHYUPD_TYPE2_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_511_TDFI_PHYUPD_TYPE2_F0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_512_REGOFF 0x800
+#define HWIO_DDR_CTL_512_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_512_REGOFF)
+#define HWIO_DDR_CTL_512_TDFI_PHYUPD_TYPE3_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_512_TDFI_PHYUPD_TYPE3_F0_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_513_REGOFF 0x804
+#define HWIO_DDR_CTL_513_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_513_REGOFF)
+#define HWIO_DDR_CTL_513_TDFI_PHYUPD_TYPE0_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_513_TDFI_PHYUPD_TYPE0_F1_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_514_REGOFF 0x808
+#define HWIO_DDR_CTL_514_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_514_REGOFF)
+#define HWIO_DDR_CTL_514_TDFI_PHYUPD_TYPE1_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_514_TDFI_PHYUPD_TYPE1_F1_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_515_REGOFF 0x80c
+#define HWIO_DDR_CTL_515_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_515_REGOFF)
+#define HWIO_DDR_CTL_515_TDFI_PHYUPD_TYPE2_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_515_TDFI_PHYUPD_TYPE2_F1_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_516_REGOFF 0x810
+#define HWIO_DDR_CTL_516_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_516_REGOFF)
+#define HWIO_DDR_CTL_516_TDFI_PHYUPD_TYPE3_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_516_TDFI_PHYUPD_TYPE3_F1_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_517_REGOFF 0x814
+#define HWIO_DDR_CTL_517_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_517_REGOFF)
+#define HWIO_DDR_CTL_517_TDFI_PHYUPD_TYPE0_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_517_TDFI_PHYUPD_TYPE0_F2_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_518_REGOFF 0x818
+#define HWIO_DDR_CTL_518_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_518_REGOFF)
+#define HWIO_DDR_CTL_518_TDFI_PHYUPD_TYPE1_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_518_TDFI_PHYUPD_TYPE1_F2_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_519_REGOFF 0x81c
+#define HWIO_DDR_CTL_519_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_519_REGOFF)
+#define HWIO_DDR_CTL_519_TDFI_PHYUPD_TYPE2_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_519_TDFI_PHYUPD_TYPE2_F2_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_520_REGOFF 0x820
+#define HWIO_DDR_CTL_520_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_520_REGOFF)
+#define HWIO_DDR_CTL_520_TDFI_PHYUPD_TYPE3_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_520_TDFI_PHYUPD_TYPE3_F2_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_521_REGOFF 0x824
+#define HWIO_DDR_CTL_521_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_521_REGOFF)
+#define HWIO_DDR_CTL_521_TDFI_PHYUPD_TYPE0_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_521_TDFI_PHYUPD_TYPE0_F3_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_522_REGOFF 0x828
+#define HWIO_DDR_CTL_522_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_522_REGOFF)
+#define HWIO_DDR_CTL_522_TDFI_PHYUPD_TYPE1_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_522_TDFI_PHYUPD_TYPE1_F3_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_523_REGOFF 0x82c
+#define HWIO_DDR_CTL_523_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_523_REGOFF)
+#define HWIO_DDR_CTL_523_TDFI_PHYUPD_TYPE2_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_523_TDFI_PHYUPD_TYPE2_F3_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_524_REGOFF 0x830
+#define HWIO_DDR_CTL_524_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_524_REGOFF)
+#define HWIO_DDR_CTL_524_TDFI_PHYUPD_TYPE3_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_CTL_524_TDFI_PHYUPD_TYPE3_F3_FLDSHFT (0)
+
+#define HWIO_DDR_CTL_525_REGOFF 0x834
+#define HWIO_DDR_CTL_525_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_525_REGOFF)
+#define HWIO_DDR_CTL_525_CTRLUPD_AREF_HP_ENABLE_FLDMASK (0x1)
+#define HWIO_DDR_CTL_525_CTRLUPD_AREF_HP_ENABLE_FLDSHFT (0)
+#define HWIO_DDR_CTL_525_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_CTL_525_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_CTL_525_OUT_OF_RANGE_SOURCE_ID_FLDMASK (0x3ff00)
+#define HWIO_DDR_CTL_525_OUT_OF_RANGE_SOURCE_ID_FLDSHFT (8)
+#define HWIO_DDR_CTL_525_CDNS_INTRL1_FLDMASK (0xfc0000)
+#define HWIO_DDR_CTL_525_CDNS_INTRL1_FLDSHFT (18)
+#define HWIO_DDR_CTL_525_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_CTL_525_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_CTL_526_REGOFF 0x838
+#define HWIO_DDR_CTL_526_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_526_REGOFF)
+#define HWIO_DDR_CTL_526_PORT_CMD_ERROR_ID_FLDMASK (0x3ff)
+#define HWIO_DDR_CTL_526_PORT_CMD_ERROR_ID_FLDSHFT (0)
+#define HWIO_DDR_CTL_526_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_CTL_526_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_CTL_526_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_0_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_526_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_526_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_526_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_527_REGOFF 0x83c
+#define HWIO_DDR_CTL_527_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_527_REGOFF)
+#define HWIO_DDR_CTL_527_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_0_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_527_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_527_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_527_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_527_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_1_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_527_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_1_FLDSHFT (16)
+#define HWIO_DDR_CTL_527_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_527_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_528_REGOFF 0x840
+#define HWIO_DDR_CTL_528_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_528_REGOFF)
+#define HWIO_DDR_CTL_528_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_1_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_528_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_1_FLDSHFT (0)
+#define HWIO_DDR_CTL_528_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_528_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_528_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_2_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_528_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_2_FLDSHFT (16)
+#define HWIO_DDR_CTL_528_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_528_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_529_REGOFF 0x844
+#define HWIO_DDR_CTL_529_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_529_REGOFF)
+#define HWIO_DDR_CTL_529_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_2_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_529_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_529_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_529_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_529_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_3_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_529_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_3_FLDSHFT (16)
+#define HWIO_DDR_CTL_529_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_529_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_530_REGOFF 0x848
+#define HWIO_DDR_CTL_530_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_530_REGOFF)
+#define HWIO_DDR_CTL_530_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_3_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_530_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_3_FLDSHFT (0)
+#define HWIO_DDR_CTL_530_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_530_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_530_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_4_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_530_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_4_FLDSHFT (16)
+#define HWIO_DDR_CTL_530_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_530_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_531_REGOFF 0x84c
+#define HWIO_DDR_CTL_531_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_531_REGOFF)
+#define HWIO_DDR_CTL_531_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_4_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_531_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_4_FLDSHFT (0)
+#define HWIO_DDR_CTL_531_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_531_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_531_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_5_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_531_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_5_FLDSHFT (16)
+#define HWIO_DDR_CTL_531_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_531_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_532_REGOFF 0x850
+#define HWIO_DDR_CTL_532_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_532_REGOFF)
+#define HWIO_DDR_CTL_532_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_5_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_532_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_5_FLDSHFT (0)
+#define HWIO_DDR_CTL_532_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_532_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_532_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_6_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_532_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_6_FLDSHFT (16)
+#define HWIO_DDR_CTL_532_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_532_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_533_REGOFF 0x854
+#define HWIO_DDR_CTL_533_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_533_REGOFF)
+#define HWIO_DDR_CTL_533_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_6_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_533_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_6_FLDSHFT (0)
+#define HWIO_DDR_CTL_533_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_533_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_533_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_7_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_533_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_7_FLDSHFT (16)
+#define HWIO_DDR_CTL_533_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_533_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_534_REGOFF 0x858
+#define HWIO_DDR_CTL_534_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_534_REGOFF)
+#define HWIO_DDR_CTL_534_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_7_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_534_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_7_FLDSHFT (0)
+#define HWIO_DDR_CTL_534_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_534_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_534_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_8_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_534_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_8_FLDSHFT (16)
+#define HWIO_DDR_CTL_534_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_534_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_535_REGOFF 0x85c
+#define HWIO_DDR_CTL_535_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_535_REGOFF)
+#define HWIO_DDR_CTL_535_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_8_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_535_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_8_FLDSHFT (0)
+#define HWIO_DDR_CTL_535_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_535_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_535_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_9_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_535_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_9_FLDSHFT (16)
+#define HWIO_DDR_CTL_535_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_535_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_536_REGOFF 0x860
+#define HWIO_DDR_CTL_536_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_536_REGOFF)
+#define HWIO_DDR_CTL_536_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_9_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_536_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_9_FLDSHFT (0)
+#define HWIO_DDR_CTL_536_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_536_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_536_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_10_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_536_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_10_FLDSHFT (16)
+#define HWIO_DDR_CTL_536_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_536_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_537_REGOFF 0x864
+#define HWIO_DDR_CTL_537_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_537_REGOFF)
+#define HWIO_DDR_CTL_537_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_10_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_537_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_10_FLDSHFT (0)
+#define HWIO_DDR_CTL_537_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_537_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_537_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_11_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_537_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_11_FLDSHFT (16)
+#define HWIO_DDR_CTL_537_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_537_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_538_REGOFF 0x868
+#define HWIO_DDR_CTL_538_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_538_REGOFF)
+#define HWIO_DDR_CTL_538_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_11_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_538_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_11_FLDSHFT (0)
+#define HWIO_DDR_CTL_538_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_538_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_538_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_12_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_538_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_12_FLDSHFT (16)
+#define HWIO_DDR_CTL_538_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_538_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_539_REGOFF 0x86c
+#define HWIO_DDR_CTL_539_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_539_REGOFF)
+#define HWIO_DDR_CTL_539_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_12_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_539_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_12_FLDSHFT (0)
+#define HWIO_DDR_CTL_539_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_539_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_539_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_13_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_539_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_13_FLDSHFT (16)
+#define HWIO_DDR_CTL_539_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_539_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_540_REGOFF 0x870
+#define HWIO_DDR_CTL_540_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_540_REGOFF)
+#define HWIO_DDR_CTL_540_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_13_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_540_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_13_FLDSHFT (0)
+#define HWIO_DDR_CTL_540_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_540_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_540_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_14_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_540_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_14_FLDSHFT (16)
+#define HWIO_DDR_CTL_540_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_540_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_541_REGOFF 0x874
+#define HWIO_DDR_CTL_541_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_541_REGOFF)
+#define HWIO_DDR_CTL_541_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_14_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_541_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_14_FLDSHFT (0)
+#define HWIO_DDR_CTL_541_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_541_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_541_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_15_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_541_AXI0_RANGE_RID_CHECK_BITS_ID_LOOKUP_15_FLDSHFT (16)
+#define HWIO_DDR_CTL_541_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_541_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_542_REGOFF 0x878
+#define HWIO_DDR_CTL_542_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_542_REGOFF)
+#define HWIO_DDR_CTL_542_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_15_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_542_AXI0_RANGE_WID_CHECK_BITS_ID_LOOKUP_15_FLDSHFT (0)
+#define HWIO_DDR_CTL_542_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_542_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_542_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_0_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_542_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_0_FLDSHFT (16)
+#define HWIO_DDR_CTL_542_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_542_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_543_REGOFF 0x87c
+#define HWIO_DDR_CTL_543_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_543_REGOFF)
+#define HWIO_DDR_CTL_543_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_0_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_543_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_0_FLDSHFT (0)
+#define HWIO_DDR_CTL_543_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_543_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_543_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_1_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_543_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_1_FLDSHFT (16)
+#define HWIO_DDR_CTL_543_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_543_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_544_REGOFF 0x880
+#define HWIO_DDR_CTL_544_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_544_REGOFF)
+#define HWIO_DDR_CTL_544_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_1_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_544_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_1_FLDSHFT (0)
+#define HWIO_DDR_CTL_544_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_544_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_544_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_2_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_544_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_2_FLDSHFT (16)
+#define HWIO_DDR_CTL_544_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_544_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_545_REGOFF 0x884
+#define HWIO_DDR_CTL_545_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_545_REGOFF)
+#define HWIO_DDR_CTL_545_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_2_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_545_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_2_FLDSHFT (0)
+#define HWIO_DDR_CTL_545_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_545_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_545_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_3_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_545_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_3_FLDSHFT (16)
+#define HWIO_DDR_CTL_545_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_545_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_546_REGOFF 0x888
+#define HWIO_DDR_CTL_546_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_546_REGOFF)
+#define HWIO_DDR_CTL_546_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_3_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_546_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_3_FLDSHFT (0)
+#define HWIO_DDR_CTL_546_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_546_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_546_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_4_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_546_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_4_FLDSHFT (16)
+#define HWIO_DDR_CTL_546_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_546_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_547_REGOFF 0x88c
+#define HWIO_DDR_CTL_547_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_547_REGOFF)
+#define HWIO_DDR_CTL_547_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_4_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_547_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_4_FLDSHFT (0)
+#define HWIO_DDR_CTL_547_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_547_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_547_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_5_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_547_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_5_FLDSHFT (16)
+#define HWIO_DDR_CTL_547_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_547_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_548_REGOFF 0x890
+#define HWIO_DDR_CTL_548_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_548_REGOFF)
+#define HWIO_DDR_CTL_548_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_5_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_548_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_5_FLDSHFT (0)
+#define HWIO_DDR_CTL_548_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_548_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_548_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_6_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_548_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_6_FLDSHFT (16)
+#define HWIO_DDR_CTL_548_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_548_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_549_REGOFF 0x894
+#define HWIO_DDR_CTL_549_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_549_REGOFF)
+#define HWIO_DDR_CTL_549_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_6_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_549_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_6_FLDSHFT (0)
+#define HWIO_DDR_CTL_549_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_549_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_549_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_7_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_549_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_7_FLDSHFT (16)
+#define HWIO_DDR_CTL_549_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_549_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_550_REGOFF 0x898
+#define HWIO_DDR_CTL_550_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_550_REGOFF)
+#define HWIO_DDR_CTL_550_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_7_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_550_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_7_FLDSHFT (0)
+#define HWIO_DDR_CTL_550_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_550_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_550_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_8_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_550_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_8_FLDSHFT (16)
+#define HWIO_DDR_CTL_550_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_550_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_551_REGOFF 0x89c
+#define HWIO_DDR_CTL_551_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_551_REGOFF)
+#define HWIO_DDR_CTL_551_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_8_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_551_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_8_FLDSHFT (0)
+#define HWIO_DDR_CTL_551_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_551_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_551_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_9_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_551_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_9_FLDSHFT (16)
+#define HWIO_DDR_CTL_551_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_551_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_552_REGOFF 0x8a0
+#define HWIO_DDR_CTL_552_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_552_REGOFF)
+#define HWIO_DDR_CTL_552_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_9_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_552_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_9_FLDSHFT (0)
+#define HWIO_DDR_CTL_552_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_552_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_552_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_10_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_552_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_10_FLDSHFT (16)
+#define HWIO_DDR_CTL_552_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_552_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_553_REGOFF 0x8a4
+#define HWIO_DDR_CTL_553_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_553_REGOFF)
+#define HWIO_DDR_CTL_553_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_10_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_553_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_10_FLDSHFT (0)
+#define HWIO_DDR_CTL_553_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_553_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_553_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_11_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_553_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_11_FLDSHFT (16)
+#define HWIO_DDR_CTL_553_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_553_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_554_REGOFF 0x8a8
+#define HWIO_DDR_CTL_554_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_554_REGOFF)
+#define HWIO_DDR_CTL_554_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_11_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_554_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_11_FLDSHFT (0)
+#define HWIO_DDR_CTL_554_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_554_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_554_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_12_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_554_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_12_FLDSHFT (16)
+#define HWIO_DDR_CTL_554_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_554_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_555_REGOFF 0x8ac
+#define HWIO_DDR_CTL_555_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_555_REGOFF)
+#define HWIO_DDR_CTL_555_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_12_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_555_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_12_FLDSHFT (0)
+#define HWIO_DDR_CTL_555_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_555_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_555_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_13_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_555_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_13_FLDSHFT (16)
+#define HWIO_DDR_CTL_555_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_555_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_556_REGOFF 0x8b0
+#define HWIO_DDR_CTL_556_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_556_REGOFF)
+#define HWIO_DDR_CTL_556_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_13_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_556_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_13_FLDSHFT (0)
+#define HWIO_DDR_CTL_556_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_556_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_556_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_14_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_556_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_14_FLDSHFT (16)
+#define HWIO_DDR_CTL_556_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_556_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_557_REGOFF 0x8b4
+#define HWIO_DDR_CTL_557_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_557_REGOFF)
+#define HWIO_DDR_CTL_557_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_14_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_557_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_14_FLDSHFT (0)
+#define HWIO_DDR_CTL_557_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_557_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_557_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_15_FLDMASK \
+ (0x1ff0000)
+#define HWIO_DDR_CTL_557_AXI1_RANGE_RID_CHECK_BITS_ID_LOOKUP_15_FLDSHFT (16)
+#define HWIO_DDR_CTL_557_CDNS_INTRL1_FLDMASK (0xfe000000)
+#define HWIO_DDR_CTL_557_CDNS_INTRL1_FLDSHFT (25)
+
+#define HWIO_DDR_CTL_558_REGOFF 0x8b8
+#define HWIO_DDR_CTL_558_ADDR(bAddr, regX) (bAddr + HWIO_DDR_CTL_558_REGOFF)
+#define HWIO_DDR_CTL_558_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_15_FLDMASK (0x1ff)
+#define HWIO_DDR_CTL_558_AXI1_RANGE_WID_CHECK_BITS_ID_LOOKUP_15_FLDSHFT (0)
+#define HWIO_DDR_CTL_558_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_CTL_558_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_CTL_558_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_CTL_558_OBSOLETE1_FLDSHFT (16)
+
+#endif /* __MNH_HWIO_DDR_CTL_ */
diff --git a/drivers/misc/mnh/mnh-hwio-ddr-phy.h b/drivers/misc/mnh/mnh-hwio-ddr-phy.h
new file mode 100644
index 0000000..bc42298
--- /dev/null
+++ b/drivers/misc/mnh/mnh-hwio-ddr-phy.h
@@ -0,0 +1,9251 @@
+/* auto generated: Friday, August 26th, 2016 11:42:18am */
+/*
+ * Copyright (c) 2016, Intel Corporation. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this
+ * list of conditions and the following disclaimer.
+ *
+ * Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * Neither the name of Intel nor the names of its contributors may be used
+ * to endorse or promote products derived from this software without specific
+ * prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef __MNH_HWIO_DDR_PHY_
+#define __MNH_HWIO_DDR_PHY_
+
+#define HWIO_DDR_PHY_00_REGOFF 0x0
+#define HWIO_DDR_PHY_00_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_00_REGOFF)
+#define HWIO_DDR_PHY_00_PHY_DQ_DM_SWIZZLE0_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_00_PHY_DQ_DM_SWIZZLE0_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_01_REGOFF 0x4
+#define HWIO_DDR_PHY_01_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_01_REGOFF)
+#define HWIO_DDR_PHY_01_PHY_DQ_DM_SWIZZLE1_0_FLDMASK (0xf)
+#define HWIO_DDR_PHY_01_PHY_DQ_DM_SWIZZLE1_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_01_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_01_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_01_PHY_CLK_WR_BYPASS_SLAVE_DELAY_0_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_01_PHY_CLK_WR_BYPASS_SLAVE_DELAY_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_01_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_01_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_01_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_01_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_02_REGOFF 0x8
+#define HWIO_DDR_PHY_02_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_02_REGOFF)
+#define HWIO_DDR_PHY_02_PHY_RDDQS_GATE_BYPASS_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_02_PHY_RDDQS_GATE_BYPASS_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_02_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_02_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_02_PHY_BYPASS_TWO_CYC_PREAMBLE_0_FLDMASK (0x30000)
+#define HWIO_DDR_PHY_02_PHY_BYPASS_TWO_CYC_PREAMBLE_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_02_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_02_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_02_PHY_CLK_BYPASS_OVERRIDE_0_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_02_PHY_CLK_BYPASS_OVERRIDE_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_02_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_02_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_03_REGOFF 0xc
+#define HWIO_DDR_PHY_03_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_03_REGOFF)
+#define HWIO_DDR_PHY_03_PHY_SW_WRDQ0_SHIFT_0_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_03_PHY_SW_WRDQ0_SHIFT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_03_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_03_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_03_PHY_SW_WRDQ1_SHIFT_0_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_03_PHY_SW_WRDQ1_SHIFT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_03_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_03_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_03_PHY_SW_WRDQ2_SHIFT_0_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_03_PHY_SW_WRDQ2_SHIFT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_03_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_03_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_03_PHY_SW_WRDQ3_SHIFT_0_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_03_PHY_SW_WRDQ3_SHIFT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_03_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_03_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_04_REGOFF 0x10
+#define HWIO_DDR_PHY_04_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_04_REGOFF)
+#define HWIO_DDR_PHY_04_PHY_SW_WRDQ4_SHIFT_0_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_04_PHY_SW_WRDQ4_SHIFT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_04_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_04_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_04_PHY_SW_WRDQ5_SHIFT_0_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_04_PHY_SW_WRDQ5_SHIFT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_04_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_04_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_04_PHY_SW_WRDQ6_SHIFT_0_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_04_PHY_SW_WRDQ6_SHIFT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_04_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_04_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_04_PHY_SW_WRDQ7_SHIFT_0_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_04_PHY_SW_WRDQ7_SHIFT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_04_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_04_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_05_REGOFF 0x14
+#define HWIO_DDR_PHY_05_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_05_REGOFF)
+#define HWIO_DDR_PHY_05_PHY_SW_WRDM_SHIFT_0_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_05_PHY_SW_WRDM_SHIFT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_05_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_05_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_05_PHY_SW_WRDQS_SHIFT_0_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_05_PHY_SW_WRDQS_SHIFT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_05_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_05_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_05_PHY_DQ_TSEL_ENABLE_0_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_05_PHY_DQ_TSEL_ENABLE_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_05_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_05_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_05_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_05_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_06_REGOFF 0x18
+#define HWIO_DDR_PHY_06_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_06_REGOFF)
+#define HWIO_DDR_PHY_06_PHY_DQ_TSEL_SELECT_0_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_06_PHY_DQ_TSEL_SELECT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_06_PHY_DQS_TSEL_ENABLE_0_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_06_PHY_DQS_TSEL_ENABLE_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_06_RESERVED_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_06_RESERVED_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_07_REGOFF 0x1c
+#define HWIO_DDR_PHY_07_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_07_REGOFF)
+#define HWIO_DDR_PHY_07_PHY_DQS_TSEL_SELECT_0_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_07_PHY_DQS_TSEL_SELECT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_07_PHY_TWO_CYC_PREAMBLE_0_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_07_PHY_TWO_CYC_PREAMBLE_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_07_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_07_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_08_REGOFF 0x20
+#define HWIO_DDR_PHY_08_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_08_REGOFF)
+#define HWIO_DDR_PHY_08_PHY_DBI_MODE_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_08_PHY_DBI_MODE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_08_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_08_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_08_PHY_LP4_BOOT_RDDATA_EN_IE_DLY_0_FLDMASK (0x300)
+#define HWIO_DDR_PHY_08_PHY_LP4_BOOT_RDDATA_EN_IE_DLY_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_08_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_08_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_08_PHY_LP4_BOOT_RDDATA_EN_DLY_0_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_08_PHY_LP4_BOOT_RDDATA_EN_DLY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_08_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_08_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_08_PHY_LP4_BOOT_RDDATA_EN_TSEL_DLY_0_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_08_PHY_LP4_BOOT_RDDATA_EN_TSEL_DLY_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_08_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_08_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_09_REGOFF 0x24
+#define HWIO_DDR_PHY_09_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_09_REGOFF)
+#define HWIO_DDR_PHY_09_PHY_LP4_BOOT_RPTR_UPDATE_0_FLDMASK (0xf)
+#define HWIO_DDR_PHY_09_PHY_LP4_BOOT_RPTR_UPDATE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_09_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_09_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_09_PHY_LP4_BOOT_RDDQS_LATENCY_ADJUST_0_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_09_PHY_LP4_BOOT_RDDQS_LATENCY_ADJUST_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_09_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_09_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_09_PHY_LPBK_CONTROL_0_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_09_PHY_LPBK_CONTROL_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_09_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_09_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_10_REGOFF 0x28
+#define HWIO_DDR_PHY_10_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_10_REGOFF)
+#define HWIO_DDR_PHY_10_PHY_LPBK_DFX_TIMEOUT_EN_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_10_PHY_LPBK_DFX_TIMEOUT_EN_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_10_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_10_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_10_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_10_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_11_REGOFF 0x2c
+#define HWIO_DDR_PHY_11_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_11_REGOFF)
+#define HWIO_DDR_PHY_11_PHY_AUTO_TIMING_MARGIN_CONTROL_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_11_PHY_AUTO_TIMING_MARGIN_CONTROL_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_12_REGOFF 0x30
+#define HWIO_DDR_PHY_12_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_12_REGOFF)
+#define HWIO_DDR_PHY_12_PHY_AUTO_TIMING_MARGIN_OBS_0_FLDMASK (0xfffffff)
+#define HWIO_DDR_PHY_12_PHY_AUTO_TIMING_MARGIN_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_12_RESERVED_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_12_RESERVED_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_13_REGOFF 0x34
+#define HWIO_DDR_PHY_13_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_13_REGOFF)
+#define HWIO_DDR_PHY_13_PHY_SLICE_PWR_RDC_DISABLE_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_13_PHY_SLICE_PWR_RDC_DISABLE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_13_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_13_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_13_PHY_PRBS_PATTERN_START_0_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_13_PHY_PRBS_PATTERN_START_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_13_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_13_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_13_PHY_PRBS_PATTERN_MASK_0_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_13_PHY_PRBS_PATTERN_MASK_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_13_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_13_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_14_REGOFF 0x38
+#define HWIO_DDR_PHY_14_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_14_REGOFF)
+#define HWIO_DDR_PHY_14_PHY_RDDQS_DQ_BYPASS_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_14_PHY_RDDQS_DQ_BYPASS_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_14_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_14_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_14_PHY_GATE_ERROR_DELAY_SELECT_0_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_14_PHY_GATE_ERROR_DELAY_SELECT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_14_RESERVED1_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_14_RESERVED1_FLDSHFT (21)
+#define HWIO_DDR_PHY_14_SC_PHY_SNAP_OBS_REGS_0_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_14_SC_PHY_SNAP_OBS_REGS_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_14_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_14_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_15_REGOFF 0x3c
+#define HWIO_DDR_PHY_15_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_15_REGOFF)
+#define HWIO_DDR_PHY_15_PHY_GATE_SMPL1_SLAVE_DELAY_0_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_15_PHY_GATE_SMPL1_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_15_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_15_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_15_PHY_LPDDR_0_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_15_PHY_LPDDR_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_15_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_15_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_15_PHY_LPDDR_TYPE_0_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_15_PHY_LPDDR_TYPE_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_15_RESERVED2_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_15_RESERVED2_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_16_REGOFF 0x40
+#define HWIO_DDR_PHY_16_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_16_REGOFF)
+#define HWIO_DDR_PHY_16_PHY_GATE_SMPL2_SLAVE_DELAY_0_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_16_PHY_GATE_SMPL2_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_16_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_16_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_16_ON_FLY_GATE_ADJUST_EN_0_FLDMASK (0x30000)
+#define HWIO_DDR_PHY_16_ON_FLY_GATE_ADJUST_EN_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_16_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_16_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_16_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_16_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_17_REGOFF 0x44
+#define HWIO_DDR_PHY_17_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_17_REGOFF)
+#define HWIO_DDR_PHY_17_PHY_GATE_TRACKING_OBS_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_17_PHY_GATE_TRACKING_OBS_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_18_REGOFF 0x48
+#define HWIO_DDR_PHY_18_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_18_REGOFF)
+#define HWIO_DDR_PHY_18_PHY_LP4_PST_AMBLE_0_FLDMASK (0x3)
+#define HWIO_DDR_PHY_18_PHY_LP4_PST_AMBLE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_18_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_18_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_18_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_18_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_19_REGOFF 0x4c
+#define HWIO_DDR_PHY_19_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_19_REGOFF)
+#define HWIO_DDR_PHY_19_PHY_LP4_RDLVL_PATT8_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_19_PHY_LP4_RDLVL_PATT8_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_20_REGOFF 0x50
+#define HWIO_DDR_PHY_20_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_20_REGOFF)
+#define HWIO_DDR_PHY_20_PHY_LP4_RDLVL_PATT9_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_20_PHY_LP4_RDLVL_PATT9_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_21_REGOFF 0x54
+#define HWIO_DDR_PHY_21_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_21_REGOFF)
+#define HWIO_DDR_PHY_21_PHY_LP4_RDLVL_PATT10_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_21_PHY_LP4_RDLVL_PATT10_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_22_REGOFF 0x58
+#define HWIO_DDR_PHY_22_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_22_REGOFF)
+#define HWIO_DDR_PHY_22_PHY_LP4_RDLVL_PATT11_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_22_PHY_LP4_RDLVL_PATT11_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_23_REGOFF 0x5c
+#define HWIO_DDR_PHY_23_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_23_REGOFF)
+#define HWIO_DDR_PHY_23_PHY_SLAVE_LOOP_CNT_UPDATE_0_FLDMASK (0x7)
+#define HWIO_DDR_PHY_23_PHY_SLAVE_LOOP_CNT_UPDATE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_23_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_23_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_23_PHY_SW_FIFO_PTR_RST_DISABLE_0_FLDMASK (0x100)
+#define HWIO_DDR_PHY_23_PHY_SW_FIFO_PTR_RST_DISABLE_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_23_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_23_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_23_PHY_MASTER_DLY_LOCK_OBS_SELECT_0_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_23_PHY_MASTER_DLY_LOCK_OBS_SELECT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_23_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_23_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_23_PHY_RDDQ_ENC_OBS_SELECT_0_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_23_PHY_RDDQ_ENC_OBS_SELECT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_23_RESERVED3_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_23_RESERVED3_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_24_REGOFF 0x60
+#define HWIO_DDR_PHY_24_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_24_REGOFF)
+#define HWIO_DDR_PHY_24_PHY_RDDQS_DQ_ENC_OBS_SELECT_0_FLDMASK (0xf)
+#define HWIO_DDR_PHY_24_PHY_RDDQS_DQ_ENC_OBS_SELECT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_24_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_24_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_24_PHY_WR_ENC_OBS_SELECT_0_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_24_PHY_WR_ENC_OBS_SELECT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_24_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_24_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_24_PHY_WR_SHIFT_OBS_SELECT_0_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_24_PHY_WR_SHIFT_OBS_SELECT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_24_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_24_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_24_PHY_FIFO_PTR_OBS_SELECT_0_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_24_PHY_FIFO_PTR_OBS_SELECT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_24_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_24_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_25_REGOFF 0x64
+#define HWIO_DDR_PHY_25_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_25_REGOFF)
+#define HWIO_DDR_PHY_25_PHY_LVL_DEBUG_MODE_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_25_PHY_LVL_DEBUG_MODE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_25_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_25_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_25_SC_PHY_LVL_DEBUG_CONT_0_FLDMASK (0x100)
+#define HWIO_DDR_PHY_25_SC_PHY_LVL_DEBUG_CONT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_25_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_25_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_25_PHY_WRLVL_CAPTURE_CNT_0_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_25_PHY_WRLVL_CAPTURE_CNT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_25_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_25_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PHY_25_PHY_WRLVL_UPDT_WAIT_CNT_0_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_25_PHY_WRLVL_UPDT_WAIT_CNT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_25_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_25_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_26_REGOFF 0x68
+#define HWIO_DDR_PHY_26_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_26_REGOFF)
+#define HWIO_DDR_PHY_26_PHY_GTLVL_CAPTURE_CNT_0_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_26_PHY_GTLVL_CAPTURE_CNT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_26_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_26_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_26_PHY_GTLVL_UPDT_WAIT_CNT_0_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_26_PHY_GTLVL_UPDT_WAIT_CNT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_26_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_26_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_26_PHY_RDLVL_CAPTURE_CNT_0_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_26_PHY_RDLVL_CAPTURE_CNT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_26_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_26_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PHY_26_PHY_RDLVL_UPDT_WAIT_CNT_0_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_26_PHY_RDLVL_UPDT_WAIT_CNT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_26_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_26_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_27_REGOFF 0x6c
+#define HWIO_DDR_PHY_27_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_27_REGOFF)
+#define HWIO_DDR_PHY_27_PHY_RDLVL_OP_MODE_0_FLDMASK (0x3)
+#define HWIO_DDR_PHY_27_PHY_RDLVL_OP_MODE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_27_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_27_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_27_PHY_RDLVL_RDDQS_DQ_OBS_SELECT_0_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_27_PHY_RDLVL_RDDQS_DQ_OBS_SELECT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_27_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_27_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_27_PHY_RDLVL_DATA_MASK_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_27_PHY_RDLVL_DATA_MASK_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_27_PHY_WDQLVL_BURST_CNT_0_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_27_PHY_WDQLVL_BURST_CNT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_27_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_27_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_28_REGOFF 0x70
+#define HWIO_DDR_PHY_28_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_28_REGOFF)
+#define HWIO_DDR_PHY_28_PHY_WDQLVL_PATT_0_FLDMASK (0x7)
+#define HWIO_DDR_PHY_28_PHY_WDQLVL_PATT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_28_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_28_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_28_PHY_WDQLVL_DQDM_SLV_DLY_JUMP_OFFSET_0_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_28_PHY_WDQLVL_DQDM_SLV_DLY_JUMP_OFFSET_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_28_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_28_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_28_PHY_WDQLVL_UPDT_WAIT_CNT_0_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_28_PHY_WDQLVL_UPDT_WAIT_CNT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_28_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_28_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_29_REGOFF 0x74
+#define HWIO_DDR_PHY_29_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_29_REGOFF)
+#define HWIO_DDR_PHY_29_PHY_WDQLVL_DQDM_OBS_SELECT_0_FLDMASK (0xf)
+#define HWIO_DDR_PHY_29_PHY_WDQLVL_DQDM_OBS_SELECT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_29_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_29_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_29_PHY_WDQLVL_QTR_DLY_STEP_0_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_29_PHY_WDQLVL_QTR_DLY_STEP_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_29_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_29_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_29_SC_PHY_WDQLVL_CLR_PREV_RESULTS_0_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_29_SC_PHY_WDQLVL_CLR_PREV_RESULTS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_29_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_29_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PHY_29_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_29_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_30_REGOFF 0x78
+#define HWIO_DDR_PHY_30_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_30_REGOFF)
+#define HWIO_DDR_PHY_30_PHY_WDQLVL_DATADM_MASK_0_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_30_PHY_WDQLVL_DATADM_MASK_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_30_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_30_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_30_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_30_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_31_REGOFF 0x7c
+#define HWIO_DDR_PHY_31_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_31_REGOFF)
+#define HWIO_DDR_PHY_31_PHY_USER_PATT0_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_31_PHY_USER_PATT0_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_32_REGOFF 0x80
+#define HWIO_DDR_PHY_32_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_32_REGOFF)
+#define HWIO_DDR_PHY_32_PHY_USER_PATT1_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_32_PHY_USER_PATT1_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_33_REGOFF 0x84
+#define HWIO_DDR_PHY_33_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_33_REGOFF)
+#define HWIO_DDR_PHY_33_PHY_USER_PATT2_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_33_PHY_USER_PATT2_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_34_REGOFF 0x88
+#define HWIO_DDR_PHY_34_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_34_REGOFF)
+#define HWIO_DDR_PHY_34_PHY_USER_PATT3_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_34_PHY_USER_PATT3_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_35_REGOFF 0x8c
+#define HWIO_DDR_PHY_35_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_35_REGOFF)
+#define HWIO_DDR_PHY_35_PHY_USER_PATT4_0_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_35_PHY_USER_PATT4_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_35_PHY_CALVL_VREF_DRIVING_SLICE_0_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_35_PHY_CALVL_VREF_DRIVING_SLICE_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_35_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_35_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_35_SC_PHY_MANUAL_CLEAR_0_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_35_SC_PHY_MANUAL_CLEAR_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_35_RESERVED1_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_35_RESERVED1_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_36_REGOFF 0x90
+#define HWIO_DDR_PHY_36_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_36_REGOFF)
+#define HWIO_DDR_PHY_36_PHY_FIFO_PTR_OBS_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_36_PHY_FIFO_PTR_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_36_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_36_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_37_REGOFF 0x94
+#define HWIO_DDR_PHY_37_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_37_REGOFF)
+#define HWIO_DDR_PHY_37_PHY_LPBK_RESULT_OBS_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_37_PHY_LPBK_RESULT_OBS_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_38_REGOFF 0x98
+#define HWIO_DDR_PHY_38_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_38_REGOFF)
+#define HWIO_DDR_PHY_38_PHY_LPBK_ERROR_COUNT_OBS_0_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_38_PHY_LPBK_ERROR_COUNT_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_38_PHY_MASTER_DLY_LOCK_OBS_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_38_PHY_MASTER_DLY_LOCK_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_38_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_38_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_39_REGOFF 0x9c
+#define HWIO_DDR_PHY_39_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_39_REGOFF)
+#define HWIO_DDR_PHY_39_PHY_RDDQ_SLV_DLY_ENC_OBS_0_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_39_PHY_RDDQ_SLV_DLY_ENC_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_39_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_39_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_39_PHY_RDDQS_BASE_SLV_DLY_ENC_OBS_0_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_39_PHY_RDDQS_BASE_SLV_DLY_ENC_OBS_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_39_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_39_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_39_PHY_RDDQS_DQ_RISE_ADDER_SLV_DLY_ENC_OBS_0_FLDMASK \
+ (0xff0000)
+#define HWIO_DDR_PHY_39_PHY_RDDQS_DQ_RISE_ADDER_SLV_DLY_ENC_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_39_PHY_RDDQS_DQ_FALL_ADDER_SLV_DLY_ENC_OBS_0_FLDMASK \
+ (0xff000000)
+#define HWIO_DDR_PHY_39_PHY_RDDQS_DQ_FALL_ADDER_SLV_DLY_ENC_OBS_0_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_40_REGOFF 0xa0
+#define HWIO_DDR_PHY_40_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_40_REGOFF)
+#define HWIO_DDR_PHY_40_PHY_RDDQS_GATE_SLV_DLY_ENC_OBS_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_40_PHY_RDDQS_GATE_SLV_DLY_ENC_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_40_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_40_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_40_PHY_WRDQS_BASE_SLV_DLY_ENC_OBS_0_FLDMASK (0x7f0000)
+#define HWIO_DDR_PHY_40_PHY_WRDQS_BASE_SLV_DLY_ENC_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_40_RESERVED1_FLDMASK (0x800000)
+#define HWIO_DDR_PHY_40_RESERVED1_FLDSHFT (23)
+#define HWIO_DDR_PHY_40_PHY_WRDQ_BASE_SLV_DLY_ENC_OBS_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_40_PHY_WRDQ_BASE_SLV_DLY_ENC_OBS_0_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_41_REGOFF 0xa4
+#define HWIO_DDR_PHY_41_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_41_REGOFF)
+#define HWIO_DDR_PHY_41_PHY_WR_ADDER_SLV_DLY_ENC_OBS_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_41_PHY_WR_ADDER_SLV_DLY_ENC_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_41_PHY_WR_SHIFT_OBS_0_FLDMASK (0x700)
+#define HWIO_DDR_PHY_41_PHY_WR_SHIFT_OBS_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_41_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_41_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_41_PHY_WRLVL_HARD0_DELAY_OBS_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_41_PHY_WRLVL_HARD0_DELAY_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_41_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_41_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_42_REGOFF 0xa8
+#define HWIO_DDR_PHY_42_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_42_REGOFF)
+#define HWIO_DDR_PHY_42_PHY_WRLVL_HARD1_DELAY_OBS_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_42_PHY_WRLVL_HARD1_DELAY_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_42_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_42_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_42_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_42_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_43_REGOFF 0xac
+#define HWIO_DDR_PHY_43_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_43_REGOFF)
+#define HWIO_DDR_PHY_43_PHY_WRLVL_STATUS_OBS_0_FLDMASK (0x1ffff)
+#define HWIO_DDR_PHY_43_PHY_WRLVL_STATUS_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_43_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_43_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_43_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_43_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_44_REGOFF 0xb0
+#define HWIO_DDR_PHY_44_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_44_REGOFF)
+#define HWIO_DDR_PHY_44_PHY_GATE_SMPL1_SLV_DLY_ENC_OBS_0_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_44_PHY_GATE_SMPL1_SLV_DLY_ENC_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_44_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_44_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_44_PHY_GATE_SMPL2_SLV_DLY_ENC_OBS_0_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_44_PHY_GATE_SMPL2_SLV_DLY_ENC_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_44_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_44_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_45_REGOFF 0xb4
+#define HWIO_DDR_PHY_45_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_45_REGOFF)
+#define HWIO_DDR_PHY_45_PHY_GTLVL_HARD0_DELAY_OBS_0_FLDMASK (0x3fff)
+#define HWIO_DDR_PHY_45_PHY_GTLVL_HARD0_DELAY_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_45_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_PHY_45_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_PHY_45_PHY_GTLVL_HARD1_DELAY_OBS_0_FLDMASK (0x3fff0000)
+#define HWIO_DDR_PHY_45_PHY_GTLVL_HARD1_DELAY_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_45_RESERVED1_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_45_RESERVED1_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_46_REGOFF 0xb8
+#define HWIO_DDR_PHY_46_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_46_REGOFF)
+#define HWIO_DDR_PHY_46_PHY_GTLVL_STATUS_OBS_0_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_46_PHY_GTLVL_STATUS_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_46_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_46_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_46_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_46_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_47_REGOFF 0xbc
+#define HWIO_DDR_PHY_47_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_47_REGOFF)
+#define HWIO_DDR_PHY_47_PHY_RDLVL_RDDQS_DQ_LE_DLY_OBS_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_47_PHY_RDLVL_RDDQS_DQ_LE_DLY_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_47_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_47_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_47_PHY_RDLVL_RDDQS_DQ_TE_DLY_OBS_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_47_PHY_RDLVL_RDDQS_DQ_TE_DLY_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_47_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_47_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_48_REGOFF 0xc0
+#define HWIO_DDR_PHY_48_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_48_REGOFF)
+#define HWIO_DDR_PHY_48_PHY_RDLVL_RDDQS_DQ_NUM_WINDOWS_OBS_0_FLDMASK (0x3)
+#define HWIO_DDR_PHY_48_PHY_RDLVL_RDDQS_DQ_NUM_WINDOWS_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_48_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_48_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_48_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_48_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_49_REGOFF 0xc4
+#define HWIO_DDR_PHY_49_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_49_REGOFF)
+#define HWIO_DDR_PHY_49_PHY_RDLVL_STATUS_OBS_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_49_PHY_RDLVL_STATUS_OBS_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_50_REGOFF 0xc8
+#define HWIO_DDR_PHY_50_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_50_REGOFF)
+#define HWIO_DDR_PHY_50_PHY_WDQLVL_DQDM_LE_DLY_OBS_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_50_PHY_WDQLVL_DQDM_LE_DLY_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_50_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_50_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_50_PHY_WDQLVL_DQDM_TE_DLY_OBS_0_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_50_PHY_WDQLVL_DQDM_TE_DLY_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_50_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_50_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_51_REGOFF 0xcc
+#define HWIO_DDR_PHY_51_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_51_REGOFF)
+#define HWIO_DDR_PHY_51_PHY_WDQLVL_STATUS_OBS_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_51_PHY_WDQLVL_STATUS_OBS_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_52_REGOFF 0xd0
+#define HWIO_DDR_PHY_52_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_52_REGOFF)
+#define HWIO_DDR_PHY_52_PHY_DDL_MODE_0_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_52_PHY_DDL_MODE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_52_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_52_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_52_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_52_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_53_REGOFF 0xd4
+#define HWIO_DDR_PHY_53_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_53_REGOFF)
+#define HWIO_DDR_PHY_53_PHY_DDL_TEST_OBS_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_53_PHY_DDL_TEST_OBS_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_54_REGOFF 0xd8
+#define HWIO_DDR_PHY_54_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_54_REGOFF)
+#define HWIO_DDR_PHY_54_PHY_DDL_TEST_MSTR_DLY_OBS_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_54_PHY_DDL_TEST_MSTR_DLY_OBS_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_55_REGOFF 0xdc
+#define HWIO_DDR_PHY_55_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_55_REGOFF)
+#define HWIO_DDR_PHY_55_PHY_DDL_TRACK_UPD_THRESHOLD_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_55_PHY_DDL_TRACK_UPD_THRESHOLD_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_55_PHY_LP4_WDQS_OE_EXTEND_0_FLDMASK (0x100)
+#define HWIO_DDR_PHY_55_PHY_LP4_WDQS_OE_EXTEND_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_55_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_55_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_55_SC_PHY_RX_CAL_START_0_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_55_SC_PHY_RX_CAL_START_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_55_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_55_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_55_PHY_RX_CAL_OVERRIDE_0_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_55_PHY_RX_CAL_OVERRIDE_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_55_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_55_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_56_REGOFF 0xe0
+#define HWIO_DDR_PHY_56_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_56_REGOFF)
+#define HWIO_DDR_PHY_56_PHY_RX_CAL_SAMPLE_WAIT_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_56_PHY_RX_CAL_SAMPLE_WAIT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_56_PHY_RX_CAL_DQ0_0_FLDMASK (0xfff00)
+#define HWIO_DDR_PHY_56_PHY_RX_CAL_DQ0_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_56_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_56_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_56_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_56_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_57_REGOFF 0xe4
+#define HWIO_DDR_PHY_57_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_57_REGOFF)
+#define HWIO_DDR_PHY_57_PHY_RX_CAL_DQ1_0_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_57_PHY_RX_CAL_DQ1_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_57_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_57_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_57_PHY_RX_CAL_DQ2_0_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_57_PHY_RX_CAL_DQ2_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_57_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_57_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_58_REGOFF 0xe8
+#define HWIO_DDR_PHY_58_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_58_REGOFF)
+#define HWIO_DDR_PHY_58_PHY_RX_CAL_DQ3_0_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_58_PHY_RX_CAL_DQ3_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_58_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_58_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_58_PHY_RX_CAL_DQ4_0_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_58_PHY_RX_CAL_DQ4_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_58_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_58_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_59_REGOFF 0xec
+#define HWIO_DDR_PHY_59_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_59_REGOFF)
+#define HWIO_DDR_PHY_59_PHY_RX_CAL_DQ5_0_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_59_PHY_RX_CAL_DQ5_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_59_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_59_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_59_PHY_RX_CAL_DQ6_0_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_59_PHY_RX_CAL_DQ6_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_59_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_59_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_60_REGOFF 0xf0
+#define HWIO_DDR_PHY_60_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_60_REGOFF)
+#define HWIO_DDR_PHY_60_PHY_RX_CAL_DQ7_0_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_60_PHY_RX_CAL_DQ7_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_60_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_60_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_60_PHY_RX_CAL_DM_0_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_60_PHY_RX_CAL_DM_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_60_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_60_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_61_REGOFF 0xf4
+#define HWIO_DDR_PHY_61_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_61_REGOFF)
+#define HWIO_DDR_PHY_61_PHY_RX_CAL_DQS_0_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_61_PHY_RX_CAL_DQS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_61_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_61_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_61_PHY_RX_CAL_FDBK_0_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_61_PHY_RX_CAL_FDBK_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_61_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_61_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_62_REGOFF 0xf8
+#define HWIO_DDR_PHY_62_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_62_REGOFF)
+#define HWIO_DDR_PHY_62_PHY_RX_CAL_OBS_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_62_PHY_RX_CAL_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_62_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_62_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_62_PHY_RX_CAL_LOCK_OBS_0_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_62_PHY_RX_CAL_LOCK_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_62_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_62_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_63_REGOFF 0xfc
+#define HWIO_DDR_PHY_63_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_63_REGOFF)
+#define HWIO_DDR_PHY_63_PHY_RX_CAL_DISABLE_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_63_PHY_RX_CAL_DISABLE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_63_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_63_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_63_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_63_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_64_REGOFF 0x100
+#define HWIO_DDR_PHY_64_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_64_REGOFF)
+#define HWIO_DDR_PHY_64_PHY_CLK_WRDQ0_SLAVE_DELAY_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_64_PHY_CLK_WRDQ0_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_64_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_64_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_64_PHY_CLK_WRDQ1_SLAVE_DELAY_0_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_64_PHY_CLK_WRDQ1_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_64_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_64_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_65_REGOFF 0x104
+#define HWIO_DDR_PHY_65_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_65_REGOFF)
+#define HWIO_DDR_PHY_65_PHY_CLK_WRDQ2_SLAVE_DELAY_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_65_PHY_CLK_WRDQ2_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_65_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_65_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_65_PHY_CLK_WRDQ3_SLAVE_DELAY_0_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_65_PHY_CLK_WRDQ3_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_65_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_65_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_66_REGOFF 0x108
+#define HWIO_DDR_PHY_66_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_66_REGOFF)
+#define HWIO_DDR_PHY_66_PHY_CLK_WRDQ4_SLAVE_DELAY_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_66_PHY_CLK_WRDQ4_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_66_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_66_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_66_PHY_CLK_WRDQ5_SLAVE_DELAY_0_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_66_PHY_CLK_WRDQ5_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_66_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_66_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_67_REGOFF 0x10c
+#define HWIO_DDR_PHY_67_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_67_REGOFF)
+#define HWIO_DDR_PHY_67_PHY_CLK_WRDQ6_SLAVE_DELAY_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_67_PHY_CLK_WRDQ6_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_67_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_67_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_67_PHY_CLK_WRDQ7_SLAVE_DELAY_0_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_67_PHY_CLK_WRDQ7_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_67_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_67_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_68_REGOFF 0x110
+#define HWIO_DDR_PHY_68_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_68_REGOFF)
+#define HWIO_DDR_PHY_68_PHY_CLK_WRDM_SLAVE_DELAY_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_68_PHY_CLK_WRDM_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_68_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_68_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_68_PHY_CLK_WRDQS_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_68_PHY_CLK_WRDQS_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_68_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_68_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_69_REGOFF 0x114
+#define HWIO_DDR_PHY_69_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_69_REGOFF)
+#define HWIO_DDR_PHY_69_PHY_WRLVL_THRESHOLD_ADJUST_0_FLDMASK (0x3)
+#define HWIO_DDR_PHY_69_PHY_WRLVL_THRESHOLD_ADJUST_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_69_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_69_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_69_PHY_RDDQ0_SLAVE_DELAY_0_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_69_PHY_RDDQ0_SLAVE_DELAY_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_69_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_69_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_69_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_69_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_70_REGOFF 0x118
+#define HWIO_DDR_PHY_70_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_70_REGOFF)
+#define HWIO_DDR_PHY_70_PHY_RDDQ1_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_70_PHY_RDDQ1_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_70_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_70_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_70_PHY_RDDQ2_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_70_PHY_RDDQ2_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_70_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_70_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_71_REGOFF 0x11c
+#define HWIO_DDR_PHY_71_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_71_REGOFF)
+#define HWIO_DDR_PHY_71_PHY_RDDQ3_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_71_PHY_RDDQ3_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_71_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_71_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_71_PHY_RDDQ4_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_71_PHY_RDDQ4_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_71_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_71_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_72_REGOFF 0x120
+#define HWIO_DDR_PHY_72_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_72_REGOFF)
+#define HWIO_DDR_PHY_72_PHY_RDDQ5_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_72_PHY_RDDQ5_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_72_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_72_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_72_PHY_RDDQ6_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_72_PHY_RDDQ6_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_72_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_72_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_73_REGOFF 0x124
+#define HWIO_DDR_PHY_73_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_73_REGOFF)
+#define HWIO_DDR_PHY_73_PHY_RDDQ7_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_73_PHY_RDDQ7_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_73_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_73_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_73_PHY_RDDM_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_73_PHY_RDDM_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_73_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_73_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_74_REGOFF 0x128
+#define HWIO_DDR_PHY_74_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_74_REGOFF)
+#define HWIO_DDR_PHY_74_PHY_RDDQS_DQ0_RISE_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_74_PHY_RDDQS_DQ0_RISE_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_74_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_74_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_74_PHY_RDDQS_DQ0_FALL_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_74_PHY_RDDQS_DQ0_FALL_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_74_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_74_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_75_REGOFF 0x12c
+#define HWIO_DDR_PHY_75_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_75_REGOFF)
+#define HWIO_DDR_PHY_75_PHY_RDDQS_DQ1_RISE_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_75_PHY_RDDQS_DQ1_RISE_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_75_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_75_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_75_PHY_RDDQS_DQ1_FALL_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_75_PHY_RDDQS_DQ1_FALL_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_75_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_75_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_76_REGOFF 0x130
+#define HWIO_DDR_PHY_76_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_76_REGOFF)
+#define HWIO_DDR_PHY_76_PHY_RDDQS_DQ2_RISE_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_76_PHY_RDDQS_DQ2_RISE_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_76_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_76_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_76_PHY_RDDQS_DQ2_FALL_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_76_PHY_RDDQS_DQ2_FALL_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_76_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_76_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_77_REGOFF 0x134
+#define HWIO_DDR_PHY_77_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_77_REGOFF)
+#define HWIO_DDR_PHY_77_PHY_RDDQS_DQ3_RISE_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_77_PHY_RDDQS_DQ3_RISE_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_77_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_77_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_77_PHY_RDDQS_DQ3_FALL_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_77_PHY_RDDQS_DQ3_FALL_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_77_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_77_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_78_REGOFF 0x138
+#define HWIO_DDR_PHY_78_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_78_REGOFF)
+#define HWIO_DDR_PHY_78_PHY_RDDQS_DQ4_RISE_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_78_PHY_RDDQS_DQ4_RISE_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_78_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_78_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_78_PHY_RDDQS_DQ4_FALL_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_78_PHY_RDDQS_DQ4_FALL_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_78_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_78_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_79_REGOFF 0x13c
+#define HWIO_DDR_PHY_79_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_79_REGOFF)
+#define HWIO_DDR_PHY_79_PHY_RDDQS_DQ5_RISE_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_79_PHY_RDDQS_DQ5_RISE_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_79_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_79_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_79_PHY_RDDQS_DQ5_FALL_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_79_PHY_RDDQS_DQ5_FALL_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_79_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_79_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_80_REGOFF 0x140
+#define HWIO_DDR_PHY_80_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_80_REGOFF)
+#define HWIO_DDR_PHY_80_PHY_RDDQS_DQ6_RISE_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_80_PHY_RDDQS_DQ6_RISE_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_80_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_80_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_80_PHY_RDDQS_DQ6_FALL_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_80_PHY_RDDQS_DQ6_FALL_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_80_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_80_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_81_REGOFF 0x144
+#define HWIO_DDR_PHY_81_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_81_REGOFF)
+#define HWIO_DDR_PHY_81_PHY_RDDQS_DQ7_RISE_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_81_PHY_RDDQS_DQ7_RISE_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_81_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_81_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_81_PHY_RDDQS_DQ7_FALL_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_81_PHY_RDDQS_DQ7_FALL_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_81_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_81_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_82_REGOFF 0x148
+#define HWIO_DDR_PHY_82_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_82_REGOFF)
+#define HWIO_DDR_PHY_82_PHY_RDDQS_DM_RISE_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_82_PHY_RDDQS_DM_RISE_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_82_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_82_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_82_PHY_RDDQS_DM_FALL_SLAVE_DELAY_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_82_PHY_RDDQS_DM_FALL_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_82_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_82_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_83_REGOFF 0x14c
+#define HWIO_DDR_PHY_83_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_83_REGOFF)
+#define HWIO_DDR_PHY_83_PHY_RDDQS_GATE_SLAVE_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_83_PHY_RDDQS_GATE_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_83_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_83_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_83_PHY_RDDQS_LATENCY_ADJUST_0_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_83_PHY_RDDQS_LATENCY_ADJUST_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_83_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_83_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PHY_83_PHY_WRITE_PATH_LAT_ADD_0_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_83_PHY_WRITE_PATH_LAT_ADD_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_83_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_83_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_84_REGOFF 0x150
+#define HWIO_DDR_PHY_84_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_84_REGOFF)
+#define HWIO_DDR_PHY_84_PHY_WRLVL_DELAY_EARLY_THRESHOLD_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_84_PHY_WRLVL_DELAY_EARLY_THRESHOLD_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_84_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_84_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_84_PHY_WRLVL_DELAY_PERIOD_THRESHOLD_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_84_PHY_WRLVL_DELAY_PERIOD_THRESHOLD_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_84_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_84_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_85_REGOFF 0x154
+#define HWIO_DDR_PHY_85_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_85_REGOFF)
+#define HWIO_DDR_PHY_85_PHY_WRLVL_EARLY_FORCE_0_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_85_PHY_WRLVL_EARLY_FORCE_0_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_85_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_85_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_85_PHY_GTLVL_RDDQS_SLV_DLY_START_0_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_85_PHY_GTLVL_RDDQS_SLV_DLY_START_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_85_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_85_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_85_PHY_GTLVL_LAT_ADJ_START_0_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_85_PHY_GTLVL_LAT_ADJ_START_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_85_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_85_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_86_REGOFF 0x158
+#define HWIO_DDR_PHY_86_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_86_REGOFF)
+#define HWIO_DDR_PHY_86_PHY_WDQLVL_DQDM_SLV_DLY_START_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_86_PHY_WDQLVL_DQDM_SLV_DLY_START_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_86_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_86_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_86_PHY_RDLVL_RDDQS_DQ_SLV_DLY_START_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_86_PHY_RDLVL_RDDQS_DQ_SLV_DLY_START_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_86_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_86_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_87_REGOFF 0x15c
+#define HWIO_DDR_PHY_87_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_87_REGOFF)
+#define HWIO_DDR_PHY_87_PHY_FDBK_PWR_CTRL_0_FLDMASK (0x7)
+#define HWIO_DDR_PHY_87_PHY_FDBK_PWR_CTRL_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_87_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_87_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_87_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_87_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_88_REGOFF 0x160
+#define HWIO_DDR_PHY_88_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_88_REGOFF)
+#define HWIO_DDR_PHY_88_PHY_DQ_OE_TIMING_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_88_PHY_DQ_OE_TIMING_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_88_PHY_DQ_TSEL_RD_TIMING_0_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_88_PHY_DQ_TSEL_RD_TIMING_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_88_PHY_DQ_TSEL_WR_TIMING_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_88_PHY_DQ_TSEL_WR_TIMING_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_88_PHY_DQS_OE_TIMING_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_88_PHY_DQS_OE_TIMING_0_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_89_REGOFF 0x164
+#define HWIO_DDR_PHY_89_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_89_REGOFF)
+#define HWIO_DDR_PHY_89_PHY_DQS_TSEL_RD_TIMING_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_89_PHY_DQS_TSEL_RD_TIMING_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_89_PHY_DQS_TSEL_WR_TIMING_0_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_89_PHY_DQS_TSEL_WR_TIMING_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_89_PHY_DQ_IE_TIMING_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_89_PHY_DQ_IE_TIMING_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_89_PHY_DQS_IE_TIMING_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_89_PHY_DQS_IE_TIMING_0_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_90_REGOFF 0x168
+#define HWIO_DDR_PHY_90_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_90_REGOFF)
+#define HWIO_DDR_PHY_90_PHY_RDDATA_EN_IE_DLY_0_FLDMASK (0x3)
+#define HWIO_DDR_PHY_90_PHY_RDDATA_EN_IE_DLY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_90_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_90_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_90_PHY_IE_MODE_0_FLDMASK (0x300)
+#define HWIO_DDR_PHY_90_PHY_IE_MODE_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_90_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_90_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_90_PHY_RDDATA_EN_DLY_0_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_90_PHY_RDDATA_EN_DLY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_90_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_90_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_90_PHY_RDDATA_EN_TSEL_DLY_0_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_90_PHY_RDDATA_EN_TSEL_DLY_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_90_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_90_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_91_REGOFF 0x16c
+#define HWIO_DDR_PHY_91_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_91_REGOFF)
+#define HWIO_DDR_PHY_91_PHY_SW_MASTER_MODE_0_FLDMASK (0xf)
+#define HWIO_DDR_PHY_91_PHY_SW_MASTER_MODE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_91_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_91_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_91_PHY_MASTER_DELAY_START_0_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_91_PHY_MASTER_DELAY_START_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_91_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_91_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_91_PHY_MASTER_DELAY_STEP_0_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_91_PHY_MASTER_DELAY_STEP_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_91_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_91_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_92_REGOFF 0x170
+#define HWIO_DDR_PHY_92_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_92_REGOFF)
+#define HWIO_DDR_PHY_92_PHY_MASTER_DELAY_WAIT_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_92_PHY_MASTER_DELAY_WAIT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_92_PHY_MASTER_DELAY_HALF_MEASURE_0_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_92_PHY_MASTER_DELAY_HALF_MEASURE_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_92_PHY_RPTR_UPDATE_0_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_92_PHY_RPTR_UPDATE_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_92_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_92_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_92_PHY_WRLVL_DLY_STEP_0_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_92_PHY_WRLVL_DLY_STEP_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_92_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_92_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_93_REGOFF 0x174
+#define HWIO_DDR_PHY_93_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_93_REGOFF)
+#define HWIO_DDR_PHY_93_PHY_WRLVL_RESP_WAIT_CNT_0_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_93_PHY_WRLVL_RESP_WAIT_CNT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_93_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_93_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_93_PHY_GTLVL_DLY_STEP_0_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_93_PHY_GTLVL_DLY_STEP_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_93_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_93_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_93_PHY_GTLVL_RESP_WAIT_CNT_0_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_93_PHY_GTLVL_RESP_WAIT_CNT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_93_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_93_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_93_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_93_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_94_REGOFF 0x178
+#define HWIO_DDR_PHY_94_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_94_REGOFF)
+#define HWIO_DDR_PHY_94_PHY_GTLVL_BACK_STEP_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_94_PHY_GTLVL_BACK_STEP_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_94_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_94_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_94_PHY_GTLVL_FINAL_STEP_0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_94_PHY_GTLVL_FINAL_STEP_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_94_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_94_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_95_REGOFF 0x17c
+#define HWIO_DDR_PHY_95_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_95_REGOFF)
+#define HWIO_DDR_PHY_95_PHY_WDQLVL_DLY_STEP_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_95_PHY_WDQLVL_DLY_STEP_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_95_PHY_RDLVL_DLY_STEP_0_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_95_PHY_RDLVL_DLY_STEP_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_95_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_95_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_95_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_95_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_96_REGOFF 0x180
+#define HWIO_DDR_PHY_96_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_96_REGOFF)
+#define HWIO_DDR_PHY_96_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_96_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_97_REGOFF 0x184
+#define HWIO_DDR_PHY_97_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_97_REGOFF)
+#define HWIO_DDR_PHY_97_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_97_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_98_REGOFF 0x188
+#define HWIO_DDR_PHY_98_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_98_REGOFF)
+#define HWIO_DDR_PHY_98_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_98_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_99_REGOFF 0x18c
+#define HWIO_DDR_PHY_99_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_99_REGOFF)
+#define HWIO_DDR_PHY_99_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_99_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_100_REGOFF 0x190
+#define HWIO_DDR_PHY_100_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_100_REGOFF)
+#define HWIO_DDR_PHY_100_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_100_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_101_REGOFF 0x194
+#define HWIO_DDR_PHY_101_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_101_REGOFF)
+#define HWIO_DDR_PHY_101_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_101_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_102_REGOFF 0x198
+#define HWIO_DDR_PHY_102_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_102_REGOFF)
+#define HWIO_DDR_PHY_102_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_102_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_103_REGOFF 0x19c
+#define HWIO_DDR_PHY_103_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_103_REGOFF)
+#define HWIO_DDR_PHY_103_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_103_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_104_REGOFF 0x1a0
+#define HWIO_DDR_PHY_104_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_104_REGOFF)
+#define HWIO_DDR_PHY_104_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_104_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_105_REGOFF 0x1a4
+#define HWIO_DDR_PHY_105_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_105_REGOFF)
+#define HWIO_DDR_PHY_105_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_105_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_106_REGOFF 0x1a8
+#define HWIO_DDR_PHY_106_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_106_REGOFF)
+#define HWIO_DDR_PHY_106_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_106_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_107_REGOFF 0x1ac
+#define HWIO_DDR_PHY_107_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_107_REGOFF)
+#define HWIO_DDR_PHY_107_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_107_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_108_REGOFF 0x1b0
+#define HWIO_DDR_PHY_108_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_108_REGOFF)
+#define HWIO_DDR_PHY_108_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_108_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_109_REGOFF 0x1b4
+#define HWIO_DDR_PHY_109_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_109_REGOFF)
+#define HWIO_DDR_PHY_109_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_109_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_110_REGOFF 0x1b8
+#define HWIO_DDR_PHY_110_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_110_REGOFF)
+#define HWIO_DDR_PHY_110_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_110_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_111_REGOFF 0x1bc
+#define HWIO_DDR_PHY_111_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_111_REGOFF)
+#define HWIO_DDR_PHY_111_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_111_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_112_REGOFF 0x1c0
+#define HWIO_DDR_PHY_112_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_112_REGOFF)
+#define HWIO_DDR_PHY_112_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_112_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_113_REGOFF 0x1c4
+#define HWIO_DDR_PHY_113_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_113_REGOFF)
+#define HWIO_DDR_PHY_113_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_113_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_114_REGOFF 0x1c8
+#define HWIO_DDR_PHY_114_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_114_REGOFF)
+#define HWIO_DDR_PHY_114_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_114_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_115_REGOFF 0x1cc
+#define HWIO_DDR_PHY_115_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_115_REGOFF)
+#define HWIO_DDR_PHY_115_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_115_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_116_REGOFF 0x1d0
+#define HWIO_DDR_PHY_116_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_116_REGOFF)
+#define HWIO_DDR_PHY_116_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_116_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_117_REGOFF 0x1d4
+#define HWIO_DDR_PHY_117_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_117_REGOFF)
+#define HWIO_DDR_PHY_117_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_117_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_118_REGOFF 0x1d8
+#define HWIO_DDR_PHY_118_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_118_REGOFF)
+#define HWIO_DDR_PHY_118_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_118_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_119_REGOFF 0x1dc
+#define HWIO_DDR_PHY_119_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_119_REGOFF)
+#define HWIO_DDR_PHY_119_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_119_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_120_REGOFF 0x1e0
+#define HWIO_DDR_PHY_120_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_120_REGOFF)
+#define HWIO_DDR_PHY_120_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_120_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_121_REGOFF 0x1e4
+#define HWIO_DDR_PHY_121_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_121_REGOFF)
+#define HWIO_DDR_PHY_121_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_121_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_122_REGOFF 0x1e8
+#define HWIO_DDR_PHY_122_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_122_REGOFF)
+#define HWIO_DDR_PHY_122_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_122_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_123_REGOFF 0x1ec
+#define HWIO_DDR_PHY_123_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_123_REGOFF)
+#define HWIO_DDR_PHY_123_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_123_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_124_REGOFF 0x1f0
+#define HWIO_DDR_PHY_124_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_124_REGOFF)
+#define HWIO_DDR_PHY_124_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_124_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_125_REGOFF 0x1f4
+#define HWIO_DDR_PHY_125_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_125_REGOFF)
+#define HWIO_DDR_PHY_125_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_125_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_126_REGOFF 0x1f8
+#define HWIO_DDR_PHY_126_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_126_REGOFF)
+#define HWIO_DDR_PHY_126_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_126_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_127_REGOFF 0x1fc
+#define HWIO_DDR_PHY_127_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_127_REGOFF)
+#define HWIO_DDR_PHY_127_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_127_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_128_REGOFF 0x200
+#define HWIO_DDR_PHY_128_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_128_REGOFF)
+#define HWIO_DDR_PHY_128_PHY_DQ_DM_SWIZZLE0_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_128_PHY_DQ_DM_SWIZZLE0_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_129_REGOFF 0x204
+#define HWIO_DDR_PHY_129_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_129_REGOFF)
+#define HWIO_DDR_PHY_129_PHY_DQ_DM_SWIZZLE1_1_FLDMASK (0xf)
+#define HWIO_DDR_PHY_129_PHY_DQ_DM_SWIZZLE1_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_129_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_129_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_129_PHY_CLK_WR_BYPASS_SLAVE_DELAY_1_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_129_PHY_CLK_WR_BYPASS_SLAVE_DELAY_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_129_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_129_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_129_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_129_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_130_REGOFF 0x208
+#define HWIO_DDR_PHY_130_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_130_REGOFF)
+#define HWIO_DDR_PHY_130_PHY_RDDQS_GATE_BYPASS_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_130_PHY_RDDQS_GATE_BYPASS_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_130_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_130_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_130_PHY_BYPASS_TWO_CYC_PREAMBLE_1_FLDMASK (0x30000)
+#define HWIO_DDR_PHY_130_PHY_BYPASS_TWO_CYC_PREAMBLE_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_130_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_130_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_130_PHY_CLK_BYPASS_OVERRIDE_1_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_130_PHY_CLK_BYPASS_OVERRIDE_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_130_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_130_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_131_REGOFF 0x20c
+#define HWIO_DDR_PHY_131_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_131_REGOFF)
+#define HWIO_DDR_PHY_131_PHY_SW_WRDQ0_SHIFT_1_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_131_PHY_SW_WRDQ0_SHIFT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_131_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_131_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_131_PHY_SW_WRDQ1_SHIFT_1_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_131_PHY_SW_WRDQ1_SHIFT_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_131_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_131_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_131_PHY_SW_WRDQ2_SHIFT_1_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_131_PHY_SW_WRDQ2_SHIFT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_131_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_131_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_131_PHY_SW_WRDQ3_SHIFT_1_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_131_PHY_SW_WRDQ3_SHIFT_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_131_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_131_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_132_REGOFF 0x210
+#define HWIO_DDR_PHY_132_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_132_REGOFF)
+#define HWIO_DDR_PHY_132_PHY_SW_WRDQ4_SHIFT_1_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_132_PHY_SW_WRDQ4_SHIFT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_132_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_132_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_132_PHY_SW_WRDQ5_SHIFT_1_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_132_PHY_SW_WRDQ5_SHIFT_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_132_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_132_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_132_PHY_SW_WRDQ6_SHIFT_1_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_132_PHY_SW_WRDQ6_SHIFT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_132_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_132_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_132_PHY_SW_WRDQ7_SHIFT_1_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_132_PHY_SW_WRDQ7_SHIFT_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_132_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_132_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_133_REGOFF 0x214
+#define HWIO_DDR_PHY_133_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_133_REGOFF)
+#define HWIO_DDR_PHY_133_PHY_SW_WRDM_SHIFT_1_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_133_PHY_SW_WRDM_SHIFT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_133_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_133_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_133_PHY_SW_WRDQS_SHIFT_1_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_133_PHY_SW_WRDQS_SHIFT_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_133_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_133_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_133_PHY_DQ_TSEL_ENABLE_1_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_133_PHY_DQ_TSEL_ENABLE_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_133_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_133_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_133_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_133_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_134_REGOFF 0x218
+#define HWIO_DDR_PHY_134_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_134_REGOFF)
+#define HWIO_DDR_PHY_134_PHY_DQ_TSEL_SELECT_1_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_134_PHY_DQ_TSEL_SELECT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_134_PHY_DQS_TSEL_ENABLE_1_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_134_PHY_DQS_TSEL_ENABLE_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_134_RESERVED_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_134_RESERVED_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_135_REGOFF 0x21c
+#define HWIO_DDR_PHY_135_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_135_REGOFF)
+#define HWIO_DDR_PHY_135_PHY_DQS_TSEL_SELECT_1_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_135_PHY_DQS_TSEL_SELECT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_135_PHY_TWO_CYC_PREAMBLE_1_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_135_PHY_TWO_CYC_PREAMBLE_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_135_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_135_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_136_REGOFF 0x220
+#define HWIO_DDR_PHY_136_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_136_REGOFF)
+#define HWIO_DDR_PHY_136_PHY_DBI_MODE_1_FLDMASK (0x1)
+#define HWIO_DDR_PHY_136_PHY_DBI_MODE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_136_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_136_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_136_PHY_LP4_BOOT_RDDATA_EN_IE_DLY_1_FLDMASK (0x300)
+#define HWIO_DDR_PHY_136_PHY_LP4_BOOT_RDDATA_EN_IE_DLY_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_136_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_136_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_136_PHY_LP4_BOOT_RDDATA_EN_DLY_1_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_136_PHY_LP4_BOOT_RDDATA_EN_DLY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_136_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_136_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_136_PHY_LP4_BOOT_RDDATA_EN_TSEL_DLY_1_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_136_PHY_LP4_BOOT_RDDATA_EN_TSEL_DLY_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_136_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_136_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_137_REGOFF 0x224
+#define HWIO_DDR_PHY_137_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_137_REGOFF)
+#define HWIO_DDR_PHY_137_PHY_LP4_BOOT_RPTR_UPDATE_1_FLDMASK (0xf)
+#define HWIO_DDR_PHY_137_PHY_LP4_BOOT_RPTR_UPDATE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_137_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_137_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_137_PHY_LP4_BOOT_RDDQS_LATENCY_ADJUST_1_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_137_PHY_LP4_BOOT_RDDQS_LATENCY_ADJUST_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_137_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_137_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_137_PHY_LPBK_CONTROL_1_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_137_PHY_LPBK_CONTROL_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_137_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_137_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_138_REGOFF 0x228
+#define HWIO_DDR_PHY_138_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_138_REGOFF)
+#define HWIO_DDR_PHY_138_PHY_LPBK_DFX_TIMEOUT_EN_1_FLDMASK (0x1)
+#define HWIO_DDR_PHY_138_PHY_LPBK_DFX_TIMEOUT_EN_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_138_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_138_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_138_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_138_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_139_REGOFF 0x22c
+#define HWIO_DDR_PHY_139_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_139_REGOFF)
+#define HWIO_DDR_PHY_139_PHY_AUTO_TIMING_MARGIN_CONTROL_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_139_PHY_AUTO_TIMING_MARGIN_CONTROL_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_140_REGOFF 0x230
+#define HWIO_DDR_PHY_140_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_140_REGOFF)
+#define HWIO_DDR_PHY_140_PHY_AUTO_TIMING_MARGIN_OBS_1_FLDMASK (0xfffffff)
+#define HWIO_DDR_PHY_140_PHY_AUTO_TIMING_MARGIN_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_140_RESERVED_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_140_RESERVED_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_141_REGOFF 0x234
+#define HWIO_DDR_PHY_141_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_141_REGOFF)
+#define HWIO_DDR_PHY_141_PHY_SLICE_PWR_RDC_DISABLE_1_FLDMASK (0x1)
+#define HWIO_DDR_PHY_141_PHY_SLICE_PWR_RDC_DISABLE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_141_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_141_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_141_PHY_PRBS_PATTERN_START_1_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_141_PHY_PRBS_PATTERN_START_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_141_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_141_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_141_PHY_PRBS_PATTERN_MASK_1_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_141_PHY_PRBS_PATTERN_MASK_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_141_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_141_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_142_REGOFF 0x238
+#define HWIO_DDR_PHY_142_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_142_REGOFF)
+#define HWIO_DDR_PHY_142_PHY_RDDQS_DQ_BYPASS_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_142_PHY_RDDQS_DQ_BYPASS_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_142_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_142_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_142_PHY_GATE_ERROR_DELAY_SELECT_1_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_142_PHY_GATE_ERROR_DELAY_SELECT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_142_RESERVED1_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_142_RESERVED1_FLDSHFT (21)
+#define HWIO_DDR_PHY_142_SC_PHY_SNAP_OBS_REGS_1_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_142_SC_PHY_SNAP_OBS_REGS_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_142_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_142_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_143_REGOFF 0x23c
+#define HWIO_DDR_PHY_143_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_143_REGOFF)
+#define HWIO_DDR_PHY_143_PHY_GATE_SMPL1_SLAVE_DELAY_1_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_143_PHY_GATE_SMPL1_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_143_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_143_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_143_PHY_LPDDR_1_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_143_PHY_LPDDR_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_143_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_143_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_143_PHY_LPDDR_TYPE_1_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_143_PHY_LPDDR_TYPE_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_143_RESERVED2_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_143_RESERVED2_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_144_REGOFF 0x240
+#define HWIO_DDR_PHY_144_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_144_REGOFF)
+#define HWIO_DDR_PHY_144_PHY_GATE_SMPL2_SLAVE_DELAY_1_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_144_PHY_GATE_SMPL2_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_144_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_144_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_144_ON_FLY_GATE_ADJUST_EN_1_FLDMASK (0x30000)
+#define HWIO_DDR_PHY_144_ON_FLY_GATE_ADJUST_EN_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_144_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_144_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_144_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_144_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_145_REGOFF 0x244
+#define HWIO_DDR_PHY_145_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_145_REGOFF)
+#define HWIO_DDR_PHY_145_PHY_GATE_TRACKING_OBS_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_145_PHY_GATE_TRACKING_OBS_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_146_REGOFF 0x248
+#define HWIO_DDR_PHY_146_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_146_REGOFF)
+#define HWIO_DDR_PHY_146_PHY_LP4_PST_AMBLE_1_FLDMASK (0x3)
+#define HWIO_DDR_PHY_146_PHY_LP4_PST_AMBLE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_146_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_146_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_146_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_146_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_147_REGOFF 0x24c
+#define HWIO_DDR_PHY_147_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_147_REGOFF)
+#define HWIO_DDR_PHY_147_PHY_LP4_RDLVL_PATT8_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_147_PHY_LP4_RDLVL_PATT8_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_148_REGOFF 0x250
+#define HWIO_DDR_PHY_148_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_148_REGOFF)
+#define HWIO_DDR_PHY_148_PHY_LP4_RDLVL_PATT9_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_148_PHY_LP4_RDLVL_PATT9_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_149_REGOFF 0x254
+#define HWIO_DDR_PHY_149_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_149_REGOFF)
+#define HWIO_DDR_PHY_149_PHY_LP4_RDLVL_PATT10_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_149_PHY_LP4_RDLVL_PATT10_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_150_REGOFF 0x258
+#define HWIO_DDR_PHY_150_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_150_REGOFF)
+#define HWIO_DDR_PHY_150_PHY_LP4_RDLVL_PATT11_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_150_PHY_LP4_RDLVL_PATT11_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_151_REGOFF 0x25c
+#define HWIO_DDR_PHY_151_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_151_REGOFF)
+#define HWIO_DDR_PHY_151_PHY_SLAVE_LOOP_CNT_UPDATE_1_FLDMASK (0x7)
+#define HWIO_DDR_PHY_151_PHY_SLAVE_LOOP_CNT_UPDATE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_151_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_151_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_151_PHY_SW_FIFO_PTR_RST_DISABLE_1_FLDMASK (0x100)
+#define HWIO_DDR_PHY_151_PHY_SW_FIFO_PTR_RST_DISABLE_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_151_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_151_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_151_PHY_MASTER_DLY_LOCK_OBS_SELECT_1_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_151_PHY_MASTER_DLY_LOCK_OBS_SELECT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_151_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_151_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_151_PHY_RDDQ_ENC_OBS_SELECT_1_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_151_PHY_RDDQ_ENC_OBS_SELECT_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_151_RESERVED3_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_151_RESERVED3_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_152_REGOFF 0x260
+#define HWIO_DDR_PHY_152_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_152_REGOFF)
+#define HWIO_DDR_PHY_152_PHY_RDDQS_DQ_ENC_OBS_SELECT_1_FLDMASK (0xf)
+#define HWIO_DDR_PHY_152_PHY_RDDQS_DQ_ENC_OBS_SELECT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_152_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_152_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_152_PHY_WR_ENC_OBS_SELECT_1_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_152_PHY_WR_ENC_OBS_SELECT_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_152_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_152_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_152_PHY_WR_SHIFT_OBS_SELECT_1_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_152_PHY_WR_SHIFT_OBS_SELECT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_152_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_152_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_152_PHY_FIFO_PTR_OBS_SELECT_1_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_152_PHY_FIFO_PTR_OBS_SELECT_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_152_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_152_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_153_REGOFF 0x264
+#define HWIO_DDR_PHY_153_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_153_REGOFF)
+#define HWIO_DDR_PHY_153_PHY_LVL_DEBUG_MODE_1_FLDMASK (0x1)
+#define HWIO_DDR_PHY_153_PHY_LVL_DEBUG_MODE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_153_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_153_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_153_SC_PHY_LVL_DEBUG_CONT_1_FLDMASK (0x100)
+#define HWIO_DDR_PHY_153_SC_PHY_LVL_DEBUG_CONT_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_153_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_153_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_153_PHY_WRLVL_CAPTURE_CNT_1_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_153_PHY_WRLVL_CAPTURE_CNT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_153_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_153_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PHY_153_PHY_WRLVL_UPDT_WAIT_CNT_1_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_153_PHY_WRLVL_UPDT_WAIT_CNT_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_153_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_153_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_154_REGOFF 0x268
+#define HWIO_DDR_PHY_154_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_154_REGOFF)
+#define HWIO_DDR_PHY_154_PHY_GTLVL_CAPTURE_CNT_1_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_154_PHY_GTLVL_CAPTURE_CNT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_154_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_154_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_154_PHY_GTLVL_UPDT_WAIT_CNT_1_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_154_PHY_GTLVL_UPDT_WAIT_CNT_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_154_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_154_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_154_PHY_RDLVL_CAPTURE_CNT_1_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_154_PHY_RDLVL_CAPTURE_CNT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_154_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_154_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PHY_154_PHY_RDLVL_UPDT_WAIT_CNT_1_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_154_PHY_RDLVL_UPDT_WAIT_CNT_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_154_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_154_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_155_REGOFF 0x26c
+#define HWIO_DDR_PHY_155_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_155_REGOFF)
+#define HWIO_DDR_PHY_155_PHY_RDLVL_OP_MODE_1_FLDMASK (0x3)
+#define HWIO_DDR_PHY_155_PHY_RDLVL_OP_MODE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_155_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_155_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_155_PHY_RDLVL_RDDQS_DQ_OBS_SELECT_1_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_155_PHY_RDLVL_RDDQS_DQ_OBS_SELECT_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_155_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_155_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_155_PHY_RDLVL_DATA_MASK_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_155_PHY_RDLVL_DATA_MASK_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_155_PHY_WDQLVL_BURST_CNT_1_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_155_PHY_WDQLVL_BURST_CNT_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_155_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_155_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_156_REGOFF 0x270
+#define HWIO_DDR_PHY_156_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_156_REGOFF)
+#define HWIO_DDR_PHY_156_PHY_WDQLVL_PATT_1_FLDMASK (0x7)
+#define HWIO_DDR_PHY_156_PHY_WDQLVL_PATT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_156_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_156_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_156_PHY_WDQLVL_DQDM_SLV_DLY_JUMP_OFFSET_1_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_156_PHY_WDQLVL_DQDM_SLV_DLY_JUMP_OFFSET_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_156_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_156_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_156_PHY_WDQLVL_UPDT_WAIT_CNT_1_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_156_PHY_WDQLVL_UPDT_WAIT_CNT_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_156_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_156_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_157_REGOFF 0x274
+#define HWIO_DDR_PHY_157_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_157_REGOFF)
+#define HWIO_DDR_PHY_157_PHY_WDQLVL_DQDM_OBS_SELECT_1_FLDMASK (0xf)
+#define HWIO_DDR_PHY_157_PHY_WDQLVL_DQDM_OBS_SELECT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_157_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_157_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_157_PHY_WDQLVL_QTR_DLY_STEP_1_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_157_PHY_WDQLVL_QTR_DLY_STEP_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_157_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_157_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_157_SC_PHY_WDQLVL_CLR_PREV_RESULTS_1_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_157_SC_PHY_WDQLVL_CLR_PREV_RESULTS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_157_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_157_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PHY_157_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_157_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_158_REGOFF 0x278
+#define HWIO_DDR_PHY_158_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_158_REGOFF)
+#define HWIO_DDR_PHY_158_PHY_WDQLVL_DATADM_MASK_1_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_158_PHY_WDQLVL_DATADM_MASK_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_158_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_158_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_158_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_158_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_159_REGOFF 0x27c
+#define HWIO_DDR_PHY_159_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_159_REGOFF)
+#define HWIO_DDR_PHY_159_PHY_USER_PATT0_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_159_PHY_USER_PATT0_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_160_REGOFF 0x280
+#define HWIO_DDR_PHY_160_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_160_REGOFF)
+#define HWIO_DDR_PHY_160_PHY_USER_PATT1_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_160_PHY_USER_PATT1_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_161_REGOFF 0x284
+#define HWIO_DDR_PHY_161_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_161_REGOFF)
+#define HWIO_DDR_PHY_161_PHY_USER_PATT2_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_161_PHY_USER_PATT2_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_162_REGOFF 0x288
+#define HWIO_DDR_PHY_162_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_162_REGOFF)
+#define HWIO_DDR_PHY_162_PHY_USER_PATT3_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_162_PHY_USER_PATT3_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_163_REGOFF 0x28c
+#define HWIO_DDR_PHY_163_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_163_REGOFF)
+#define HWIO_DDR_PHY_163_PHY_USER_PATT4_1_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_163_PHY_USER_PATT4_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_163_PHY_CALVL_VREF_DRIVING_SLICE_1_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_163_PHY_CALVL_VREF_DRIVING_SLICE_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_163_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_163_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_163_SC_PHY_MANUAL_CLEAR_1_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_163_SC_PHY_MANUAL_CLEAR_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_163_RESERVED1_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_163_RESERVED1_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_164_REGOFF 0x290
+#define HWIO_DDR_PHY_164_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_164_REGOFF)
+#define HWIO_DDR_PHY_164_PHY_FIFO_PTR_OBS_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_164_PHY_FIFO_PTR_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_164_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_164_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_165_REGOFF 0x294
+#define HWIO_DDR_PHY_165_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_165_REGOFF)
+#define HWIO_DDR_PHY_165_PHY_LPBK_RESULT_OBS_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_165_PHY_LPBK_RESULT_OBS_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_166_REGOFF 0x298
+#define HWIO_DDR_PHY_166_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_166_REGOFF)
+#define HWIO_DDR_PHY_166_PHY_LPBK_ERROR_COUNT_OBS_1_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_166_PHY_LPBK_ERROR_COUNT_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_166_PHY_MASTER_DLY_LOCK_OBS_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_166_PHY_MASTER_DLY_LOCK_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_166_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_166_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_167_REGOFF 0x29c
+#define HWIO_DDR_PHY_167_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_167_REGOFF)
+#define HWIO_DDR_PHY_167_PHY_RDDQ_SLV_DLY_ENC_OBS_1_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_167_PHY_RDDQ_SLV_DLY_ENC_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_167_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_167_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_167_PHY_RDDQS_BASE_SLV_DLY_ENC_OBS_1_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_167_PHY_RDDQS_BASE_SLV_DLY_ENC_OBS_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_167_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_167_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_167_PHY_RDDQS_DQ_RISE_ADDER_SLV_DLY_ENC_OBS_1_FLDMASK \
+ (0xff0000)
+#define HWIO_DDR_PHY_167_PHY_RDDQS_DQ_RISE_ADDER_SLV_DLY_ENC_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_167_PHY_RDDQS_DQ_FALL_ADDER_SLV_DLY_ENC_OBS_1_FLDMASK \
+ (0xff000000)
+#define HWIO_DDR_PHY_167_PHY_RDDQS_DQ_FALL_ADDER_SLV_DLY_ENC_OBS_1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_168_REGOFF 0x2a0
+#define HWIO_DDR_PHY_168_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_168_REGOFF)
+#define HWIO_DDR_PHY_168_PHY_RDDQS_GATE_SLV_DLY_ENC_OBS_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_168_PHY_RDDQS_GATE_SLV_DLY_ENC_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_168_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_168_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_168_PHY_WRDQS_BASE_SLV_DLY_ENC_OBS_1_FLDMASK (0x7f0000)
+#define HWIO_DDR_PHY_168_PHY_WRDQS_BASE_SLV_DLY_ENC_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_168_RESERVED1_FLDMASK (0x800000)
+#define HWIO_DDR_PHY_168_RESERVED1_FLDSHFT (23)
+#define HWIO_DDR_PHY_168_PHY_WRDQ_BASE_SLV_DLY_ENC_OBS_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_168_PHY_WRDQ_BASE_SLV_DLY_ENC_OBS_1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_169_REGOFF 0x2a4
+#define HWIO_DDR_PHY_169_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_169_REGOFF)
+#define HWIO_DDR_PHY_169_PHY_WR_ADDER_SLV_DLY_ENC_OBS_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_169_PHY_WR_ADDER_SLV_DLY_ENC_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_169_PHY_WR_SHIFT_OBS_1_FLDMASK (0x700)
+#define HWIO_DDR_PHY_169_PHY_WR_SHIFT_OBS_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_169_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_169_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_169_PHY_WRLVL_HARD0_DELAY_OBS_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_169_PHY_WRLVL_HARD0_DELAY_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_169_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_169_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_170_REGOFF 0x2a8
+#define HWIO_DDR_PHY_170_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_170_REGOFF)
+#define HWIO_DDR_PHY_170_PHY_WRLVL_HARD1_DELAY_OBS_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_170_PHY_WRLVL_HARD1_DELAY_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_170_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_170_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_170_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_170_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_171_REGOFF 0x2ac
+#define HWIO_DDR_PHY_171_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_171_REGOFF)
+#define HWIO_DDR_PHY_171_PHY_WRLVL_STATUS_OBS_1_FLDMASK (0x1ffff)
+#define HWIO_DDR_PHY_171_PHY_WRLVL_STATUS_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_171_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_171_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_171_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_171_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_172_REGOFF 0x2b0
+#define HWIO_DDR_PHY_172_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_172_REGOFF)
+#define HWIO_DDR_PHY_172_PHY_GATE_SMPL1_SLV_DLY_ENC_OBS_1_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_172_PHY_GATE_SMPL1_SLV_DLY_ENC_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_172_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_172_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_172_PHY_GATE_SMPL2_SLV_DLY_ENC_OBS_1_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_172_PHY_GATE_SMPL2_SLV_DLY_ENC_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_172_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_172_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_173_REGOFF 0x2b4
+#define HWIO_DDR_PHY_173_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_173_REGOFF)
+#define HWIO_DDR_PHY_173_PHY_GTLVL_HARD0_DELAY_OBS_1_FLDMASK (0x3fff)
+#define HWIO_DDR_PHY_173_PHY_GTLVL_HARD0_DELAY_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_173_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_PHY_173_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_PHY_173_PHY_GTLVL_HARD1_DELAY_OBS_1_FLDMASK (0x3fff0000)
+#define HWIO_DDR_PHY_173_PHY_GTLVL_HARD1_DELAY_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_173_RESERVED1_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_173_RESERVED1_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_174_REGOFF 0x2b8
+#define HWIO_DDR_PHY_174_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_174_REGOFF)
+#define HWIO_DDR_PHY_174_PHY_GTLVL_STATUS_OBS_1_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_174_PHY_GTLVL_STATUS_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_174_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_174_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_174_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_174_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_175_REGOFF 0x2bc
+#define HWIO_DDR_PHY_175_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_175_REGOFF)
+#define HWIO_DDR_PHY_175_PHY_RDLVL_RDDQS_DQ_LE_DLY_OBS_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_175_PHY_RDLVL_RDDQS_DQ_LE_DLY_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_175_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_175_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_175_PHY_RDLVL_RDDQS_DQ_TE_DLY_OBS_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_175_PHY_RDLVL_RDDQS_DQ_TE_DLY_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_175_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_175_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_176_REGOFF 0x2c0
+#define HWIO_DDR_PHY_176_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_176_REGOFF)
+#define HWIO_DDR_PHY_176_PHY_RDLVL_RDDQS_DQ_NUM_WINDOWS_OBS_1_FLDMASK (0x3)
+#define HWIO_DDR_PHY_176_PHY_RDLVL_RDDQS_DQ_NUM_WINDOWS_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_176_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_176_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_176_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_176_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_177_REGOFF 0x2c4
+#define HWIO_DDR_PHY_177_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_177_REGOFF)
+#define HWIO_DDR_PHY_177_PHY_RDLVL_STATUS_OBS_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_177_PHY_RDLVL_STATUS_OBS_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_178_REGOFF 0x2c8
+#define HWIO_DDR_PHY_178_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_178_REGOFF)
+#define HWIO_DDR_PHY_178_PHY_WDQLVL_DQDM_LE_DLY_OBS_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_178_PHY_WDQLVL_DQDM_LE_DLY_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_178_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_178_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_178_PHY_WDQLVL_DQDM_TE_DLY_OBS_1_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_178_PHY_WDQLVL_DQDM_TE_DLY_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_178_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_178_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_179_REGOFF 0x2cc
+#define HWIO_DDR_PHY_179_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_179_REGOFF)
+#define HWIO_DDR_PHY_179_PHY_WDQLVL_STATUS_OBS_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_179_PHY_WDQLVL_STATUS_OBS_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_180_REGOFF 0x2d0
+#define HWIO_DDR_PHY_180_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_180_REGOFF)
+#define HWIO_DDR_PHY_180_PHY_DDL_MODE_1_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_180_PHY_DDL_MODE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_180_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_180_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_180_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_180_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_181_REGOFF 0x2d4
+#define HWIO_DDR_PHY_181_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_181_REGOFF)
+#define HWIO_DDR_PHY_181_PHY_DDL_TEST_OBS_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_181_PHY_DDL_TEST_OBS_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_182_REGOFF 0x2d8
+#define HWIO_DDR_PHY_182_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_182_REGOFF)
+#define HWIO_DDR_PHY_182_PHY_DDL_TEST_MSTR_DLY_OBS_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_182_PHY_DDL_TEST_MSTR_DLY_OBS_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_183_REGOFF 0x2dc
+#define HWIO_DDR_PHY_183_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_183_REGOFF)
+#define HWIO_DDR_PHY_183_PHY_DDL_TRACK_UPD_THRESHOLD_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_183_PHY_DDL_TRACK_UPD_THRESHOLD_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_183_PHY_LP4_WDQS_OE_EXTEND_1_FLDMASK (0x100)
+#define HWIO_DDR_PHY_183_PHY_LP4_WDQS_OE_EXTEND_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_183_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_183_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_183_SC_PHY_RX_CAL_START_1_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_183_SC_PHY_RX_CAL_START_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_183_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_183_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_183_PHY_RX_CAL_OVERRIDE_1_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_183_PHY_RX_CAL_OVERRIDE_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_183_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_183_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_184_REGOFF 0x2e0
+#define HWIO_DDR_PHY_184_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_184_REGOFF)
+#define HWIO_DDR_PHY_184_PHY_RX_CAL_SAMPLE_WAIT_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_184_PHY_RX_CAL_SAMPLE_WAIT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_184_PHY_RX_CAL_DQ0_1_FLDMASK (0xfff00)
+#define HWIO_DDR_PHY_184_PHY_RX_CAL_DQ0_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_184_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_184_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_184_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_184_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_185_REGOFF 0x2e4
+#define HWIO_DDR_PHY_185_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_185_REGOFF)
+#define HWIO_DDR_PHY_185_PHY_RX_CAL_DQ1_1_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_185_PHY_RX_CAL_DQ1_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_185_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_185_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_185_PHY_RX_CAL_DQ2_1_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_185_PHY_RX_CAL_DQ2_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_185_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_185_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_186_REGOFF 0x2e8
+#define HWIO_DDR_PHY_186_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_186_REGOFF)
+#define HWIO_DDR_PHY_186_PHY_RX_CAL_DQ3_1_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_186_PHY_RX_CAL_DQ3_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_186_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_186_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_186_PHY_RX_CAL_DQ4_1_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_186_PHY_RX_CAL_DQ4_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_186_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_186_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_187_REGOFF 0x2ec
+#define HWIO_DDR_PHY_187_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_187_REGOFF)
+#define HWIO_DDR_PHY_187_PHY_RX_CAL_DQ5_1_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_187_PHY_RX_CAL_DQ5_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_187_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_187_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_187_PHY_RX_CAL_DQ6_1_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_187_PHY_RX_CAL_DQ6_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_187_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_187_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_188_REGOFF 0x2f0
+#define HWIO_DDR_PHY_188_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_188_REGOFF)
+#define HWIO_DDR_PHY_188_PHY_RX_CAL_DQ7_1_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_188_PHY_RX_CAL_DQ7_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_188_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_188_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_188_PHY_RX_CAL_DM_1_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_188_PHY_RX_CAL_DM_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_188_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_188_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_189_REGOFF 0x2f4
+#define HWIO_DDR_PHY_189_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_189_REGOFF)
+#define HWIO_DDR_PHY_189_PHY_RX_CAL_DQS_1_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_189_PHY_RX_CAL_DQS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_189_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_189_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_189_PHY_RX_CAL_FDBK_1_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_189_PHY_RX_CAL_FDBK_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_189_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_189_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_190_REGOFF 0x2f8
+#define HWIO_DDR_PHY_190_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_190_REGOFF)
+#define HWIO_DDR_PHY_190_PHY_RX_CAL_OBS_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_190_PHY_RX_CAL_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_190_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_190_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_190_PHY_RX_CAL_LOCK_OBS_1_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_190_PHY_RX_CAL_LOCK_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_190_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_190_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_191_REGOFF 0x2fc
+#define HWIO_DDR_PHY_191_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_191_REGOFF)
+#define HWIO_DDR_PHY_191_PHY_RX_CAL_DISABLE_1_FLDMASK (0x1)
+#define HWIO_DDR_PHY_191_PHY_RX_CAL_DISABLE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_191_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_191_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_191_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_191_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_192_REGOFF 0x300
+#define HWIO_DDR_PHY_192_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_192_REGOFF)
+#define HWIO_DDR_PHY_192_PHY_CLK_WRDQ0_SLAVE_DELAY_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_192_PHY_CLK_WRDQ0_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_192_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_192_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_192_PHY_CLK_WRDQ1_SLAVE_DELAY_1_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_192_PHY_CLK_WRDQ1_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_192_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_192_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_193_REGOFF 0x304
+#define HWIO_DDR_PHY_193_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_193_REGOFF)
+#define HWIO_DDR_PHY_193_PHY_CLK_WRDQ2_SLAVE_DELAY_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_193_PHY_CLK_WRDQ2_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_193_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_193_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_193_PHY_CLK_WRDQ3_SLAVE_DELAY_1_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_193_PHY_CLK_WRDQ3_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_193_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_193_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_194_REGOFF 0x308
+#define HWIO_DDR_PHY_194_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_194_REGOFF)
+#define HWIO_DDR_PHY_194_PHY_CLK_WRDQ4_SLAVE_DELAY_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_194_PHY_CLK_WRDQ4_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_194_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_194_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_194_PHY_CLK_WRDQ5_SLAVE_DELAY_1_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_194_PHY_CLK_WRDQ5_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_194_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_194_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_195_REGOFF 0x30c
+#define HWIO_DDR_PHY_195_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_195_REGOFF)
+#define HWIO_DDR_PHY_195_PHY_CLK_WRDQ6_SLAVE_DELAY_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_195_PHY_CLK_WRDQ6_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_195_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_195_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_195_PHY_CLK_WRDQ7_SLAVE_DELAY_1_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_195_PHY_CLK_WRDQ7_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_195_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_195_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_196_REGOFF 0x310
+#define HWIO_DDR_PHY_196_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_196_REGOFF)
+#define HWIO_DDR_PHY_196_PHY_CLK_WRDM_SLAVE_DELAY_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_196_PHY_CLK_WRDM_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_196_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_196_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_196_PHY_CLK_WRDQS_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_196_PHY_CLK_WRDQS_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_196_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_196_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_197_REGOFF 0x314
+#define HWIO_DDR_PHY_197_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_197_REGOFF)
+#define HWIO_DDR_PHY_197_PHY_WRLVL_THRESHOLD_ADJUST_1_FLDMASK (0x3)
+#define HWIO_DDR_PHY_197_PHY_WRLVL_THRESHOLD_ADJUST_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_197_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_197_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_197_PHY_RDDQ0_SLAVE_DELAY_1_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_197_PHY_RDDQ0_SLAVE_DELAY_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_197_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_197_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_197_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_197_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_198_REGOFF 0x318
+#define HWIO_DDR_PHY_198_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_198_REGOFF)
+#define HWIO_DDR_PHY_198_PHY_RDDQ1_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_198_PHY_RDDQ1_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_198_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_198_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_198_PHY_RDDQ2_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_198_PHY_RDDQ2_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_198_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_198_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_199_REGOFF 0x31c
+#define HWIO_DDR_PHY_199_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_199_REGOFF)
+#define HWIO_DDR_PHY_199_PHY_RDDQ3_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_199_PHY_RDDQ3_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_199_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_199_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_199_PHY_RDDQ4_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_199_PHY_RDDQ4_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_199_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_199_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_200_REGOFF 0x320
+#define HWIO_DDR_PHY_200_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_200_REGOFF)
+#define HWIO_DDR_PHY_200_PHY_RDDQ5_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_200_PHY_RDDQ5_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_200_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_200_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_200_PHY_RDDQ6_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_200_PHY_RDDQ6_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_200_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_200_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_201_REGOFF 0x324
+#define HWIO_DDR_PHY_201_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_201_REGOFF)
+#define HWIO_DDR_PHY_201_PHY_RDDQ7_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_201_PHY_RDDQ7_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_201_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_201_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_201_PHY_RDDM_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_201_PHY_RDDM_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_201_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_201_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_202_REGOFF 0x328
+#define HWIO_DDR_PHY_202_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_202_REGOFF)
+#define HWIO_DDR_PHY_202_PHY_RDDQS_DQ0_RISE_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_202_PHY_RDDQS_DQ0_RISE_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_202_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_202_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_202_PHY_RDDQS_DQ0_FALL_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_202_PHY_RDDQS_DQ0_FALL_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_202_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_202_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_203_REGOFF 0x32c
+#define HWIO_DDR_PHY_203_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_203_REGOFF)
+#define HWIO_DDR_PHY_203_PHY_RDDQS_DQ1_RISE_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_203_PHY_RDDQS_DQ1_RISE_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_203_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_203_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_203_PHY_RDDQS_DQ1_FALL_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_203_PHY_RDDQS_DQ1_FALL_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_203_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_203_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_204_REGOFF 0x330
+#define HWIO_DDR_PHY_204_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_204_REGOFF)
+#define HWIO_DDR_PHY_204_PHY_RDDQS_DQ2_RISE_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_204_PHY_RDDQS_DQ2_RISE_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_204_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_204_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_204_PHY_RDDQS_DQ2_FALL_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_204_PHY_RDDQS_DQ2_FALL_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_204_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_204_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_205_REGOFF 0x334
+#define HWIO_DDR_PHY_205_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_205_REGOFF)
+#define HWIO_DDR_PHY_205_PHY_RDDQS_DQ3_RISE_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_205_PHY_RDDQS_DQ3_RISE_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_205_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_205_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_205_PHY_RDDQS_DQ3_FALL_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_205_PHY_RDDQS_DQ3_FALL_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_205_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_205_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_206_REGOFF 0x338
+#define HWIO_DDR_PHY_206_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_206_REGOFF)
+#define HWIO_DDR_PHY_206_PHY_RDDQS_DQ4_RISE_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_206_PHY_RDDQS_DQ4_RISE_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_206_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_206_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_206_PHY_RDDQS_DQ4_FALL_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_206_PHY_RDDQS_DQ4_FALL_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_206_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_206_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_207_REGOFF 0x33c
+#define HWIO_DDR_PHY_207_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_207_REGOFF)
+#define HWIO_DDR_PHY_207_PHY_RDDQS_DQ5_RISE_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_207_PHY_RDDQS_DQ5_RISE_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_207_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_207_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_207_PHY_RDDQS_DQ5_FALL_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_207_PHY_RDDQS_DQ5_FALL_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_207_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_207_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_208_REGOFF 0x340
+#define HWIO_DDR_PHY_208_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_208_REGOFF)
+#define HWIO_DDR_PHY_208_PHY_RDDQS_DQ6_RISE_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_208_PHY_RDDQS_DQ6_RISE_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_208_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_208_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_208_PHY_RDDQS_DQ6_FALL_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_208_PHY_RDDQS_DQ6_FALL_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_208_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_208_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_209_REGOFF 0x344
+#define HWIO_DDR_PHY_209_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_209_REGOFF)
+#define HWIO_DDR_PHY_209_PHY_RDDQS_DQ7_RISE_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_209_PHY_RDDQS_DQ7_RISE_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_209_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_209_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_209_PHY_RDDQS_DQ7_FALL_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_209_PHY_RDDQS_DQ7_FALL_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_209_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_209_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_210_REGOFF 0x348
+#define HWIO_DDR_PHY_210_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_210_REGOFF)
+#define HWIO_DDR_PHY_210_PHY_RDDQS_DM_RISE_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_210_PHY_RDDQS_DM_RISE_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_210_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_210_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_210_PHY_RDDQS_DM_FALL_SLAVE_DELAY_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_210_PHY_RDDQS_DM_FALL_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_210_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_210_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_211_REGOFF 0x34c
+#define HWIO_DDR_PHY_211_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_211_REGOFF)
+#define HWIO_DDR_PHY_211_PHY_RDDQS_GATE_SLAVE_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_211_PHY_RDDQS_GATE_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_211_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_211_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_211_PHY_RDDQS_LATENCY_ADJUST_1_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_211_PHY_RDDQS_LATENCY_ADJUST_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_211_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_211_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PHY_211_PHY_WRITE_PATH_LAT_ADD_1_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_211_PHY_WRITE_PATH_LAT_ADD_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_211_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_211_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_212_REGOFF 0x350
+#define HWIO_DDR_PHY_212_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_212_REGOFF)
+#define HWIO_DDR_PHY_212_PHY_WRLVL_DELAY_EARLY_THRESHOLD_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_212_PHY_WRLVL_DELAY_EARLY_THRESHOLD_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_212_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_212_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_212_PHY_WRLVL_DELAY_PERIOD_THRESHOLD_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_212_PHY_WRLVL_DELAY_PERIOD_THRESHOLD_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_212_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_212_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_213_REGOFF 0x354
+#define HWIO_DDR_PHY_213_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_213_REGOFF)
+#define HWIO_DDR_PHY_213_PHY_WRLVL_EARLY_FORCE_0_1_FLDMASK (0x1)
+#define HWIO_DDR_PHY_213_PHY_WRLVL_EARLY_FORCE_0_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_213_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_213_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_213_PHY_GTLVL_RDDQS_SLV_DLY_START_1_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_213_PHY_GTLVL_RDDQS_SLV_DLY_START_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_213_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_213_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_213_PHY_GTLVL_LAT_ADJ_START_1_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_213_PHY_GTLVL_LAT_ADJ_START_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_213_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_213_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_214_REGOFF 0x358
+#define HWIO_DDR_PHY_214_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_214_REGOFF)
+#define HWIO_DDR_PHY_214_PHY_WDQLVL_DQDM_SLV_DLY_START_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_214_PHY_WDQLVL_DQDM_SLV_DLY_START_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_214_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_214_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_214_PHY_RDLVL_RDDQS_DQ_SLV_DLY_START_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_214_PHY_RDLVL_RDDQS_DQ_SLV_DLY_START_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_214_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_214_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_215_REGOFF 0x35c
+#define HWIO_DDR_PHY_215_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_215_REGOFF)
+#define HWIO_DDR_PHY_215_PHY_FDBK_PWR_CTRL_1_FLDMASK (0x7)
+#define HWIO_DDR_PHY_215_PHY_FDBK_PWR_CTRL_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_215_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_215_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_215_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_215_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_216_REGOFF 0x360
+#define HWIO_DDR_PHY_216_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_216_REGOFF)
+#define HWIO_DDR_PHY_216_PHY_DQ_OE_TIMING_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_216_PHY_DQ_OE_TIMING_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_216_PHY_DQ_TSEL_RD_TIMING_1_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_216_PHY_DQ_TSEL_RD_TIMING_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_216_PHY_DQ_TSEL_WR_TIMING_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_216_PHY_DQ_TSEL_WR_TIMING_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_216_PHY_DQS_OE_TIMING_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_216_PHY_DQS_OE_TIMING_1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_217_REGOFF 0x364
+#define HWIO_DDR_PHY_217_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_217_REGOFF)
+#define HWIO_DDR_PHY_217_PHY_DQS_TSEL_RD_TIMING_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_217_PHY_DQS_TSEL_RD_TIMING_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_217_PHY_DQS_TSEL_WR_TIMING_1_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_217_PHY_DQS_TSEL_WR_TIMING_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_217_PHY_DQ_IE_TIMING_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_217_PHY_DQ_IE_TIMING_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_217_PHY_DQS_IE_TIMING_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_217_PHY_DQS_IE_TIMING_1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_218_REGOFF 0x368
+#define HWIO_DDR_PHY_218_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_218_REGOFF)
+#define HWIO_DDR_PHY_218_PHY_RDDATA_EN_IE_DLY_1_FLDMASK (0x3)
+#define HWIO_DDR_PHY_218_PHY_RDDATA_EN_IE_DLY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_218_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_218_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_218_PHY_IE_MODE_1_FLDMASK (0x300)
+#define HWIO_DDR_PHY_218_PHY_IE_MODE_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_218_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_218_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_218_PHY_RDDATA_EN_DLY_1_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_218_PHY_RDDATA_EN_DLY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_218_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_218_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_218_PHY_RDDATA_EN_TSEL_DLY_1_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_218_PHY_RDDATA_EN_TSEL_DLY_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_218_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_218_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_219_REGOFF 0x36c
+#define HWIO_DDR_PHY_219_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_219_REGOFF)
+#define HWIO_DDR_PHY_219_PHY_SW_MASTER_MODE_1_FLDMASK (0xf)
+#define HWIO_DDR_PHY_219_PHY_SW_MASTER_MODE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_219_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_219_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_219_PHY_MASTER_DELAY_START_1_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_219_PHY_MASTER_DELAY_START_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_219_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_219_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_219_PHY_MASTER_DELAY_STEP_1_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_219_PHY_MASTER_DELAY_STEP_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_219_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_219_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_220_REGOFF 0x370
+#define HWIO_DDR_PHY_220_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_220_REGOFF)
+#define HWIO_DDR_PHY_220_PHY_MASTER_DELAY_WAIT_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_220_PHY_MASTER_DELAY_WAIT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_220_PHY_MASTER_DELAY_HALF_MEASURE_1_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_220_PHY_MASTER_DELAY_HALF_MEASURE_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_220_PHY_RPTR_UPDATE_1_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_220_PHY_RPTR_UPDATE_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_220_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_220_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_220_PHY_WRLVL_DLY_STEP_1_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_220_PHY_WRLVL_DLY_STEP_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_220_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_220_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_221_REGOFF 0x374
+#define HWIO_DDR_PHY_221_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_221_REGOFF)
+#define HWIO_DDR_PHY_221_PHY_WRLVL_RESP_WAIT_CNT_1_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_221_PHY_WRLVL_RESP_WAIT_CNT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_221_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_221_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_221_PHY_GTLVL_DLY_STEP_1_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_221_PHY_GTLVL_DLY_STEP_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_221_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_221_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_221_PHY_GTLVL_RESP_WAIT_CNT_1_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_221_PHY_GTLVL_RESP_WAIT_CNT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_221_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_221_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_221_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_221_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_222_REGOFF 0x378
+#define HWIO_DDR_PHY_222_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_222_REGOFF)
+#define HWIO_DDR_PHY_222_PHY_GTLVL_BACK_STEP_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_222_PHY_GTLVL_BACK_STEP_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_222_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_222_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_222_PHY_GTLVL_FINAL_STEP_1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_222_PHY_GTLVL_FINAL_STEP_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_222_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_222_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_223_REGOFF 0x37c
+#define HWIO_DDR_PHY_223_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_223_REGOFF)
+#define HWIO_DDR_PHY_223_PHY_WDQLVL_DLY_STEP_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_223_PHY_WDQLVL_DLY_STEP_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_223_PHY_RDLVL_DLY_STEP_1_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_223_PHY_RDLVL_DLY_STEP_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_223_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_223_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_223_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_223_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_224_REGOFF 0x380
+#define HWIO_DDR_PHY_224_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_224_REGOFF)
+#define HWIO_DDR_PHY_224_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_224_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_225_REGOFF 0x384
+#define HWIO_DDR_PHY_225_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_225_REGOFF)
+#define HWIO_DDR_PHY_225_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_225_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_226_REGOFF 0x388
+#define HWIO_DDR_PHY_226_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_226_REGOFF)
+#define HWIO_DDR_PHY_226_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_226_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_227_REGOFF 0x38c
+#define HWIO_DDR_PHY_227_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_227_REGOFF)
+#define HWIO_DDR_PHY_227_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_227_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_228_REGOFF 0x390
+#define HWIO_DDR_PHY_228_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_228_REGOFF)
+#define HWIO_DDR_PHY_228_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_228_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_229_REGOFF 0x394
+#define HWIO_DDR_PHY_229_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_229_REGOFF)
+#define HWIO_DDR_PHY_229_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_229_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_230_REGOFF 0x398
+#define HWIO_DDR_PHY_230_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_230_REGOFF)
+#define HWIO_DDR_PHY_230_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_230_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_231_REGOFF 0x39c
+#define HWIO_DDR_PHY_231_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_231_REGOFF)
+#define HWIO_DDR_PHY_231_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_231_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_232_REGOFF 0x3a0
+#define HWIO_DDR_PHY_232_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_232_REGOFF)
+#define HWIO_DDR_PHY_232_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_232_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_233_REGOFF 0x3a4
+#define HWIO_DDR_PHY_233_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_233_REGOFF)
+#define HWIO_DDR_PHY_233_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_233_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_234_REGOFF 0x3a8
+#define HWIO_DDR_PHY_234_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_234_REGOFF)
+#define HWIO_DDR_PHY_234_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_234_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_235_REGOFF 0x3ac
+#define HWIO_DDR_PHY_235_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_235_REGOFF)
+#define HWIO_DDR_PHY_235_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_235_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_236_REGOFF 0x3b0
+#define HWIO_DDR_PHY_236_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_236_REGOFF)
+#define HWIO_DDR_PHY_236_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_236_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_237_REGOFF 0x3b4
+#define HWIO_DDR_PHY_237_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_237_REGOFF)
+#define HWIO_DDR_PHY_237_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_237_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_238_REGOFF 0x3b8
+#define HWIO_DDR_PHY_238_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_238_REGOFF)
+#define HWIO_DDR_PHY_238_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_238_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_239_REGOFF 0x3bc
+#define HWIO_DDR_PHY_239_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_239_REGOFF)
+#define HWIO_DDR_PHY_239_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_239_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_240_REGOFF 0x3c0
+#define HWIO_DDR_PHY_240_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_240_REGOFF)
+#define HWIO_DDR_PHY_240_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_240_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_241_REGOFF 0x3c4
+#define HWIO_DDR_PHY_241_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_241_REGOFF)
+#define HWIO_DDR_PHY_241_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_241_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_242_REGOFF 0x3c8
+#define HWIO_DDR_PHY_242_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_242_REGOFF)
+#define HWIO_DDR_PHY_242_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_242_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_243_REGOFF 0x3cc
+#define HWIO_DDR_PHY_243_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_243_REGOFF)
+#define HWIO_DDR_PHY_243_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_243_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_244_REGOFF 0x3d0
+#define HWIO_DDR_PHY_244_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_244_REGOFF)
+#define HWIO_DDR_PHY_244_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_244_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_245_REGOFF 0x3d4
+#define HWIO_DDR_PHY_245_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_245_REGOFF)
+#define HWIO_DDR_PHY_245_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_245_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_246_REGOFF 0x3d8
+#define HWIO_DDR_PHY_246_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_246_REGOFF)
+#define HWIO_DDR_PHY_246_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_246_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_247_REGOFF 0x3dc
+#define HWIO_DDR_PHY_247_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_247_REGOFF)
+#define HWIO_DDR_PHY_247_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_247_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_248_REGOFF 0x3e0
+#define HWIO_DDR_PHY_248_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_248_REGOFF)
+#define HWIO_DDR_PHY_248_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_248_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_249_REGOFF 0x3e4
+#define HWIO_DDR_PHY_249_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_249_REGOFF)
+#define HWIO_DDR_PHY_249_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_249_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_250_REGOFF 0x3e8
+#define HWIO_DDR_PHY_250_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_250_REGOFF)
+#define HWIO_DDR_PHY_250_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_250_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_251_REGOFF 0x3ec
+#define HWIO_DDR_PHY_251_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_251_REGOFF)
+#define HWIO_DDR_PHY_251_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_251_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_252_REGOFF 0x3f0
+#define HWIO_DDR_PHY_252_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_252_REGOFF)
+#define HWIO_DDR_PHY_252_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_252_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_253_REGOFF 0x3f4
+#define HWIO_DDR_PHY_253_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_253_REGOFF)
+#define HWIO_DDR_PHY_253_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_253_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_254_REGOFF 0x3f8
+#define HWIO_DDR_PHY_254_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_254_REGOFF)
+#define HWIO_DDR_PHY_254_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_254_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_255_REGOFF 0x3fc
+#define HWIO_DDR_PHY_255_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_255_REGOFF)
+#define HWIO_DDR_PHY_255_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_255_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_256_REGOFF 0x400
+#define HWIO_DDR_PHY_256_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_256_REGOFF)
+#define HWIO_DDR_PHY_256_PHY_DQ_DM_SWIZZLE0_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_256_PHY_DQ_DM_SWIZZLE0_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_257_REGOFF 0x404
+#define HWIO_DDR_PHY_257_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_257_REGOFF)
+#define HWIO_DDR_PHY_257_PHY_DQ_DM_SWIZZLE1_2_FLDMASK (0xf)
+#define HWIO_DDR_PHY_257_PHY_DQ_DM_SWIZZLE1_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_257_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_257_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_257_PHY_CLK_WR_BYPASS_SLAVE_DELAY_2_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_257_PHY_CLK_WR_BYPASS_SLAVE_DELAY_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_257_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_257_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_257_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_257_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_258_REGOFF 0x408
+#define HWIO_DDR_PHY_258_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_258_REGOFF)
+#define HWIO_DDR_PHY_258_PHY_RDDQS_GATE_BYPASS_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_258_PHY_RDDQS_GATE_BYPASS_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_258_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_258_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_258_PHY_BYPASS_TWO_CYC_PREAMBLE_2_FLDMASK (0x30000)
+#define HWIO_DDR_PHY_258_PHY_BYPASS_TWO_CYC_PREAMBLE_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_258_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_258_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_258_PHY_CLK_BYPASS_OVERRIDE_2_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_258_PHY_CLK_BYPASS_OVERRIDE_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_258_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_258_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_259_REGOFF 0x40c
+#define HWIO_DDR_PHY_259_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_259_REGOFF)
+#define HWIO_DDR_PHY_259_PHY_SW_WRDQ0_SHIFT_2_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_259_PHY_SW_WRDQ0_SHIFT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_259_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_259_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_259_PHY_SW_WRDQ1_SHIFT_2_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_259_PHY_SW_WRDQ1_SHIFT_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_259_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_259_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_259_PHY_SW_WRDQ2_SHIFT_2_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_259_PHY_SW_WRDQ2_SHIFT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_259_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_259_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_259_PHY_SW_WRDQ3_SHIFT_2_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_259_PHY_SW_WRDQ3_SHIFT_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_259_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_259_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_260_REGOFF 0x410
+#define HWIO_DDR_PHY_260_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_260_REGOFF)
+#define HWIO_DDR_PHY_260_PHY_SW_WRDQ4_SHIFT_2_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_260_PHY_SW_WRDQ4_SHIFT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_260_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_260_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_260_PHY_SW_WRDQ5_SHIFT_2_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_260_PHY_SW_WRDQ5_SHIFT_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_260_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_260_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_260_PHY_SW_WRDQ6_SHIFT_2_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_260_PHY_SW_WRDQ6_SHIFT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_260_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_260_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_260_PHY_SW_WRDQ7_SHIFT_2_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_260_PHY_SW_WRDQ7_SHIFT_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_260_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_260_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_261_REGOFF 0x414
+#define HWIO_DDR_PHY_261_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_261_REGOFF)
+#define HWIO_DDR_PHY_261_PHY_SW_WRDM_SHIFT_2_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_261_PHY_SW_WRDM_SHIFT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_261_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_261_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_261_PHY_SW_WRDQS_SHIFT_2_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_261_PHY_SW_WRDQS_SHIFT_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_261_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_261_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_261_PHY_DQ_TSEL_ENABLE_2_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_261_PHY_DQ_TSEL_ENABLE_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_261_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_261_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_261_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_261_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_262_REGOFF 0x418
+#define HWIO_DDR_PHY_262_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_262_REGOFF)
+#define HWIO_DDR_PHY_262_PHY_DQ_TSEL_SELECT_2_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_262_PHY_DQ_TSEL_SELECT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_262_PHY_DQS_TSEL_ENABLE_2_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_262_PHY_DQS_TSEL_ENABLE_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_262_RESERVED_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_262_RESERVED_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_263_REGOFF 0x41c
+#define HWIO_DDR_PHY_263_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_263_REGOFF)
+#define HWIO_DDR_PHY_263_PHY_DQS_TSEL_SELECT_2_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_263_PHY_DQS_TSEL_SELECT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_263_PHY_TWO_CYC_PREAMBLE_2_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_263_PHY_TWO_CYC_PREAMBLE_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_263_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_263_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_264_REGOFF 0x420
+#define HWIO_DDR_PHY_264_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_264_REGOFF)
+#define HWIO_DDR_PHY_264_PHY_DBI_MODE_2_FLDMASK (0x1)
+#define HWIO_DDR_PHY_264_PHY_DBI_MODE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_264_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_264_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_264_PHY_LP4_BOOT_RDDATA_EN_IE_DLY_2_FLDMASK (0x300)
+#define HWIO_DDR_PHY_264_PHY_LP4_BOOT_RDDATA_EN_IE_DLY_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_264_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_264_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_264_PHY_LP4_BOOT_RDDATA_EN_DLY_2_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_264_PHY_LP4_BOOT_RDDATA_EN_DLY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_264_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_264_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_264_PHY_LP4_BOOT_RDDATA_EN_TSEL_DLY_2_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_264_PHY_LP4_BOOT_RDDATA_EN_TSEL_DLY_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_264_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_264_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_265_REGOFF 0x424
+#define HWIO_DDR_PHY_265_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_265_REGOFF)
+#define HWIO_DDR_PHY_265_PHY_LP4_BOOT_RPTR_UPDATE_2_FLDMASK (0xf)
+#define HWIO_DDR_PHY_265_PHY_LP4_BOOT_RPTR_UPDATE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_265_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_265_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_265_PHY_LP4_BOOT_RDDQS_LATENCY_ADJUST_2_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_265_PHY_LP4_BOOT_RDDQS_LATENCY_ADJUST_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_265_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_265_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_265_PHY_LPBK_CONTROL_2_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_265_PHY_LPBK_CONTROL_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_265_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_265_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_266_REGOFF 0x428
+#define HWIO_DDR_PHY_266_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_266_REGOFF)
+#define HWIO_DDR_PHY_266_PHY_LPBK_DFX_TIMEOUT_EN_2_FLDMASK (0x1)
+#define HWIO_DDR_PHY_266_PHY_LPBK_DFX_TIMEOUT_EN_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_266_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_266_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_266_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_266_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_267_REGOFF 0x42c
+#define HWIO_DDR_PHY_267_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_267_REGOFF)
+#define HWIO_DDR_PHY_267_PHY_AUTO_TIMING_MARGIN_CONTROL_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_267_PHY_AUTO_TIMING_MARGIN_CONTROL_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_268_REGOFF 0x430
+#define HWIO_DDR_PHY_268_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_268_REGOFF)
+#define HWIO_DDR_PHY_268_PHY_AUTO_TIMING_MARGIN_OBS_2_FLDMASK (0xfffffff)
+#define HWIO_DDR_PHY_268_PHY_AUTO_TIMING_MARGIN_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_268_RESERVED_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_268_RESERVED_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_269_REGOFF 0x434
+#define HWIO_DDR_PHY_269_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_269_REGOFF)
+#define HWIO_DDR_PHY_269_PHY_SLICE_PWR_RDC_DISABLE_2_FLDMASK (0x1)
+#define HWIO_DDR_PHY_269_PHY_SLICE_PWR_RDC_DISABLE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_269_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_269_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_269_PHY_PRBS_PATTERN_START_2_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_269_PHY_PRBS_PATTERN_START_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_269_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_269_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_269_PHY_PRBS_PATTERN_MASK_2_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_269_PHY_PRBS_PATTERN_MASK_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_269_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_269_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_270_REGOFF 0x438
+#define HWIO_DDR_PHY_270_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_270_REGOFF)
+#define HWIO_DDR_PHY_270_PHY_RDDQS_DQ_BYPASS_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_270_PHY_RDDQS_DQ_BYPASS_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_270_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_270_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_270_PHY_GATE_ERROR_DELAY_SELECT_2_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_270_PHY_GATE_ERROR_DELAY_SELECT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_270_RESERVED1_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_270_RESERVED1_FLDSHFT (21)
+#define HWIO_DDR_PHY_270_SC_PHY_SNAP_OBS_REGS_2_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_270_SC_PHY_SNAP_OBS_REGS_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_270_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_270_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_271_REGOFF 0x43c
+#define HWIO_DDR_PHY_271_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_271_REGOFF)
+#define HWIO_DDR_PHY_271_PHY_GATE_SMPL1_SLAVE_DELAY_2_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_271_PHY_GATE_SMPL1_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_271_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_271_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_271_PHY_LPDDR_2_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_271_PHY_LPDDR_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_271_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_271_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_271_PHY_LPDDR_TYPE_2_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_271_PHY_LPDDR_TYPE_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_271_RESERVED2_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_271_RESERVED2_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_272_REGOFF 0x440
+#define HWIO_DDR_PHY_272_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_272_REGOFF)
+#define HWIO_DDR_PHY_272_PHY_GATE_SMPL2_SLAVE_DELAY_2_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_272_PHY_GATE_SMPL2_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_272_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_272_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_272_ON_FLY_GATE_ADJUST_EN_2_FLDMASK (0x30000)
+#define HWIO_DDR_PHY_272_ON_FLY_GATE_ADJUST_EN_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_272_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_272_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_272_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_272_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_273_REGOFF 0x444
+#define HWIO_DDR_PHY_273_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_273_REGOFF)
+#define HWIO_DDR_PHY_273_PHY_GATE_TRACKING_OBS_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_273_PHY_GATE_TRACKING_OBS_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_274_REGOFF 0x448
+#define HWIO_DDR_PHY_274_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_274_REGOFF)
+#define HWIO_DDR_PHY_274_PHY_LP4_PST_AMBLE_2_FLDMASK (0x3)
+#define HWIO_DDR_PHY_274_PHY_LP4_PST_AMBLE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_274_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_274_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_274_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_274_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_275_REGOFF 0x44c
+#define HWIO_DDR_PHY_275_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_275_REGOFF)
+#define HWIO_DDR_PHY_275_PHY_LP4_RDLVL_PATT8_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_275_PHY_LP4_RDLVL_PATT8_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_276_REGOFF 0x450
+#define HWIO_DDR_PHY_276_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_276_REGOFF)
+#define HWIO_DDR_PHY_276_PHY_LP4_RDLVL_PATT9_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_276_PHY_LP4_RDLVL_PATT9_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_277_REGOFF 0x454
+#define HWIO_DDR_PHY_277_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_277_REGOFF)
+#define HWIO_DDR_PHY_277_PHY_LP4_RDLVL_PATT10_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_277_PHY_LP4_RDLVL_PATT10_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_278_REGOFF 0x458
+#define HWIO_DDR_PHY_278_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_278_REGOFF)
+#define HWIO_DDR_PHY_278_PHY_LP4_RDLVL_PATT11_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_278_PHY_LP4_RDLVL_PATT11_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_279_REGOFF 0x45c
+#define HWIO_DDR_PHY_279_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_279_REGOFF)
+#define HWIO_DDR_PHY_279_PHY_SLAVE_LOOP_CNT_UPDATE_2_FLDMASK (0x7)
+#define HWIO_DDR_PHY_279_PHY_SLAVE_LOOP_CNT_UPDATE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_279_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_279_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_279_PHY_SW_FIFO_PTR_RST_DISABLE_2_FLDMASK (0x100)
+#define HWIO_DDR_PHY_279_PHY_SW_FIFO_PTR_RST_DISABLE_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_279_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_279_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_279_PHY_MASTER_DLY_LOCK_OBS_SELECT_2_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_279_PHY_MASTER_DLY_LOCK_OBS_SELECT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_279_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_279_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_279_PHY_RDDQ_ENC_OBS_SELECT_2_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_279_PHY_RDDQ_ENC_OBS_SELECT_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_279_RESERVED3_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_279_RESERVED3_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_280_REGOFF 0x460
+#define HWIO_DDR_PHY_280_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_280_REGOFF)
+#define HWIO_DDR_PHY_280_PHY_RDDQS_DQ_ENC_OBS_SELECT_2_FLDMASK (0xf)
+#define HWIO_DDR_PHY_280_PHY_RDDQS_DQ_ENC_OBS_SELECT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_280_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_280_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_280_PHY_WR_ENC_OBS_SELECT_2_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_280_PHY_WR_ENC_OBS_SELECT_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_280_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_280_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_280_PHY_WR_SHIFT_OBS_SELECT_2_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_280_PHY_WR_SHIFT_OBS_SELECT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_280_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_280_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_280_PHY_FIFO_PTR_OBS_SELECT_2_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_280_PHY_FIFO_PTR_OBS_SELECT_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_280_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_280_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_281_REGOFF 0x464
+#define HWIO_DDR_PHY_281_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_281_REGOFF)
+#define HWIO_DDR_PHY_281_PHY_LVL_DEBUG_MODE_2_FLDMASK (0x1)
+#define HWIO_DDR_PHY_281_PHY_LVL_DEBUG_MODE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_281_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_281_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_281_SC_PHY_LVL_DEBUG_CONT_2_FLDMASK (0x100)
+#define HWIO_DDR_PHY_281_SC_PHY_LVL_DEBUG_CONT_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_281_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_281_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_281_PHY_WRLVL_CAPTURE_CNT_2_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_281_PHY_WRLVL_CAPTURE_CNT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_281_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_281_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PHY_281_PHY_WRLVL_UPDT_WAIT_CNT_2_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_281_PHY_WRLVL_UPDT_WAIT_CNT_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_281_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_281_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_282_REGOFF 0x468
+#define HWIO_DDR_PHY_282_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_282_REGOFF)
+#define HWIO_DDR_PHY_282_PHY_GTLVL_CAPTURE_CNT_2_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_282_PHY_GTLVL_CAPTURE_CNT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_282_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_282_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_282_PHY_GTLVL_UPDT_WAIT_CNT_2_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_282_PHY_GTLVL_UPDT_WAIT_CNT_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_282_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_282_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_282_PHY_RDLVL_CAPTURE_CNT_2_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_282_PHY_RDLVL_CAPTURE_CNT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_282_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_282_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PHY_282_PHY_RDLVL_UPDT_WAIT_CNT_2_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_282_PHY_RDLVL_UPDT_WAIT_CNT_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_282_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_282_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_283_REGOFF 0x46c
+#define HWIO_DDR_PHY_283_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_283_REGOFF)
+#define HWIO_DDR_PHY_283_PHY_RDLVL_OP_MODE_2_FLDMASK (0x3)
+#define HWIO_DDR_PHY_283_PHY_RDLVL_OP_MODE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_283_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_283_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_283_PHY_RDLVL_RDDQS_DQ_OBS_SELECT_2_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_283_PHY_RDLVL_RDDQS_DQ_OBS_SELECT_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_283_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_283_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_283_PHY_RDLVL_DATA_MASK_2_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_283_PHY_RDLVL_DATA_MASK_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_283_PHY_WDQLVL_BURST_CNT_2_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_283_PHY_WDQLVL_BURST_CNT_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_283_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_283_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_284_REGOFF 0x470
+#define HWIO_DDR_PHY_284_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_284_REGOFF)
+#define HWIO_DDR_PHY_284_PHY_WDQLVL_PATT_2_FLDMASK (0x7)
+#define HWIO_DDR_PHY_284_PHY_WDQLVL_PATT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_284_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_284_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_284_PHY_WDQLVL_DQDM_SLV_DLY_JUMP_OFFSET_2_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_284_PHY_WDQLVL_DQDM_SLV_DLY_JUMP_OFFSET_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_284_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_284_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_284_PHY_WDQLVL_UPDT_WAIT_CNT_2_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_284_PHY_WDQLVL_UPDT_WAIT_CNT_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_284_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_284_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_285_REGOFF 0x474
+#define HWIO_DDR_PHY_285_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_285_REGOFF)
+#define HWIO_DDR_PHY_285_PHY_WDQLVL_DQDM_OBS_SELECT_2_FLDMASK (0xf)
+#define HWIO_DDR_PHY_285_PHY_WDQLVL_DQDM_OBS_SELECT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_285_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_285_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_285_PHY_WDQLVL_QTR_DLY_STEP_2_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_285_PHY_WDQLVL_QTR_DLY_STEP_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_285_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_285_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_285_SC_PHY_WDQLVL_CLR_PREV_RESULTS_2_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_285_SC_PHY_WDQLVL_CLR_PREV_RESULTS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_285_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_285_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PHY_285_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_285_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_286_REGOFF 0x478
+#define HWIO_DDR_PHY_286_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_286_REGOFF)
+#define HWIO_DDR_PHY_286_PHY_WDQLVL_DATADM_MASK_2_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_286_PHY_WDQLVL_DATADM_MASK_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_286_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_286_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_286_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_286_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_287_REGOFF 0x47c
+#define HWIO_DDR_PHY_287_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_287_REGOFF)
+#define HWIO_DDR_PHY_287_PHY_USER_PATT0_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_287_PHY_USER_PATT0_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_288_REGOFF 0x480
+#define HWIO_DDR_PHY_288_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_288_REGOFF)
+#define HWIO_DDR_PHY_288_PHY_USER_PATT1_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_288_PHY_USER_PATT1_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_289_REGOFF 0x484
+#define HWIO_DDR_PHY_289_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_289_REGOFF)
+#define HWIO_DDR_PHY_289_PHY_USER_PATT2_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_289_PHY_USER_PATT2_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_290_REGOFF 0x488
+#define HWIO_DDR_PHY_290_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_290_REGOFF)
+#define HWIO_DDR_PHY_290_PHY_USER_PATT3_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_290_PHY_USER_PATT3_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_291_REGOFF 0x48c
+#define HWIO_DDR_PHY_291_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_291_REGOFF)
+#define HWIO_DDR_PHY_291_PHY_USER_PATT4_2_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_291_PHY_USER_PATT4_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_291_PHY_CALVL_VREF_DRIVING_SLICE_2_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_291_PHY_CALVL_VREF_DRIVING_SLICE_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_291_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_291_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_291_SC_PHY_MANUAL_CLEAR_2_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_291_SC_PHY_MANUAL_CLEAR_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_291_RESERVED1_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_291_RESERVED1_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_292_REGOFF 0x490
+#define HWIO_DDR_PHY_292_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_292_REGOFF)
+#define HWIO_DDR_PHY_292_PHY_FIFO_PTR_OBS_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_292_PHY_FIFO_PTR_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_292_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_292_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_293_REGOFF 0x494
+#define HWIO_DDR_PHY_293_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_293_REGOFF)
+#define HWIO_DDR_PHY_293_PHY_LPBK_RESULT_OBS_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_293_PHY_LPBK_RESULT_OBS_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_294_REGOFF 0x498
+#define HWIO_DDR_PHY_294_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_294_REGOFF)
+#define HWIO_DDR_PHY_294_PHY_LPBK_ERROR_COUNT_OBS_2_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_294_PHY_LPBK_ERROR_COUNT_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_294_PHY_MASTER_DLY_LOCK_OBS_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_294_PHY_MASTER_DLY_LOCK_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_294_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_294_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_295_REGOFF 0x49c
+#define HWIO_DDR_PHY_295_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_295_REGOFF)
+#define HWIO_DDR_PHY_295_PHY_RDDQ_SLV_DLY_ENC_OBS_2_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_295_PHY_RDDQ_SLV_DLY_ENC_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_295_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_295_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_295_PHY_RDDQS_BASE_SLV_DLY_ENC_OBS_2_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_295_PHY_RDDQS_BASE_SLV_DLY_ENC_OBS_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_295_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_295_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_295_PHY_RDDQS_DQ_RISE_ADDER_SLV_DLY_ENC_OBS_2_FLDMASK \
+ (0xff0000)
+#define HWIO_DDR_PHY_295_PHY_RDDQS_DQ_RISE_ADDER_SLV_DLY_ENC_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_295_PHY_RDDQS_DQ_FALL_ADDER_SLV_DLY_ENC_OBS_2_FLDMASK \
+ (0xff000000)
+#define HWIO_DDR_PHY_295_PHY_RDDQS_DQ_FALL_ADDER_SLV_DLY_ENC_OBS_2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_296_REGOFF 0x4a0
+#define HWIO_DDR_PHY_296_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_296_REGOFF)
+#define HWIO_DDR_PHY_296_PHY_RDDQS_GATE_SLV_DLY_ENC_OBS_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_296_PHY_RDDQS_GATE_SLV_DLY_ENC_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_296_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_296_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_296_PHY_WRDQS_BASE_SLV_DLY_ENC_OBS_2_FLDMASK (0x7f0000)
+#define HWIO_DDR_PHY_296_PHY_WRDQS_BASE_SLV_DLY_ENC_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_296_RESERVED1_FLDMASK (0x800000)
+#define HWIO_DDR_PHY_296_RESERVED1_FLDSHFT (23)
+#define HWIO_DDR_PHY_296_PHY_WRDQ_BASE_SLV_DLY_ENC_OBS_2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_296_PHY_WRDQ_BASE_SLV_DLY_ENC_OBS_2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_297_REGOFF 0x4a4
+#define HWIO_DDR_PHY_297_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_297_REGOFF)
+#define HWIO_DDR_PHY_297_PHY_WR_ADDER_SLV_DLY_ENC_OBS_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_297_PHY_WR_ADDER_SLV_DLY_ENC_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_297_PHY_WR_SHIFT_OBS_2_FLDMASK (0x700)
+#define HWIO_DDR_PHY_297_PHY_WR_SHIFT_OBS_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_297_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_297_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_297_PHY_WRLVL_HARD0_DELAY_OBS_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_297_PHY_WRLVL_HARD0_DELAY_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_297_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_297_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_298_REGOFF 0x4a8
+#define HWIO_DDR_PHY_298_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_298_REGOFF)
+#define HWIO_DDR_PHY_298_PHY_WRLVL_HARD1_DELAY_OBS_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_298_PHY_WRLVL_HARD1_DELAY_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_298_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_298_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_298_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_298_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_299_REGOFF 0x4ac
+#define HWIO_DDR_PHY_299_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_299_REGOFF)
+#define HWIO_DDR_PHY_299_PHY_WRLVL_STATUS_OBS_2_FLDMASK (0x1ffff)
+#define HWIO_DDR_PHY_299_PHY_WRLVL_STATUS_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_299_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_299_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_299_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_299_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_300_REGOFF 0x4b0
+#define HWIO_DDR_PHY_300_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_300_REGOFF)
+#define HWIO_DDR_PHY_300_PHY_GATE_SMPL1_SLV_DLY_ENC_OBS_2_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_300_PHY_GATE_SMPL1_SLV_DLY_ENC_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_300_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_300_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_300_PHY_GATE_SMPL2_SLV_DLY_ENC_OBS_2_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_300_PHY_GATE_SMPL2_SLV_DLY_ENC_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_300_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_300_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_301_REGOFF 0x4b4
+#define HWIO_DDR_PHY_301_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_301_REGOFF)
+#define HWIO_DDR_PHY_301_PHY_GTLVL_HARD0_DELAY_OBS_2_FLDMASK (0x3fff)
+#define HWIO_DDR_PHY_301_PHY_GTLVL_HARD0_DELAY_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_301_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_PHY_301_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_PHY_301_PHY_GTLVL_HARD1_DELAY_OBS_2_FLDMASK (0x3fff0000)
+#define HWIO_DDR_PHY_301_PHY_GTLVL_HARD1_DELAY_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_301_RESERVED1_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_301_RESERVED1_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_302_REGOFF 0x4b8
+#define HWIO_DDR_PHY_302_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_302_REGOFF)
+#define HWIO_DDR_PHY_302_PHY_GTLVL_STATUS_OBS_2_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_302_PHY_GTLVL_STATUS_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_302_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_302_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_302_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_302_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_303_REGOFF 0x4bc
+#define HWIO_DDR_PHY_303_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_303_REGOFF)
+#define HWIO_DDR_PHY_303_PHY_RDLVL_RDDQS_DQ_LE_DLY_OBS_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_303_PHY_RDLVL_RDDQS_DQ_LE_DLY_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_303_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_303_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_303_PHY_RDLVL_RDDQS_DQ_TE_DLY_OBS_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_303_PHY_RDLVL_RDDQS_DQ_TE_DLY_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_303_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_303_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_304_REGOFF 0x4c0
+#define HWIO_DDR_PHY_304_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_304_REGOFF)
+#define HWIO_DDR_PHY_304_PHY_RDLVL_RDDQS_DQ_NUM_WINDOWS_OBS_2_FLDMASK (0x3)
+#define HWIO_DDR_PHY_304_PHY_RDLVL_RDDQS_DQ_NUM_WINDOWS_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_304_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_304_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_304_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_304_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_305_REGOFF 0x4c4
+#define HWIO_DDR_PHY_305_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_305_REGOFF)
+#define HWIO_DDR_PHY_305_PHY_RDLVL_STATUS_OBS_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_305_PHY_RDLVL_STATUS_OBS_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_306_REGOFF 0x4c8
+#define HWIO_DDR_PHY_306_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_306_REGOFF)
+#define HWIO_DDR_PHY_306_PHY_WDQLVL_DQDM_LE_DLY_OBS_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_306_PHY_WDQLVL_DQDM_LE_DLY_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_306_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_306_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_306_PHY_WDQLVL_DQDM_TE_DLY_OBS_2_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_306_PHY_WDQLVL_DQDM_TE_DLY_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_306_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_306_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_307_REGOFF 0x4cc
+#define HWIO_DDR_PHY_307_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_307_REGOFF)
+#define HWIO_DDR_PHY_307_PHY_WDQLVL_STATUS_OBS_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_307_PHY_WDQLVL_STATUS_OBS_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_308_REGOFF 0x4d0
+#define HWIO_DDR_PHY_308_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_308_REGOFF)
+#define HWIO_DDR_PHY_308_PHY_DDL_MODE_2_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_308_PHY_DDL_MODE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_308_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_308_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_308_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_308_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_309_REGOFF 0x4d4
+#define HWIO_DDR_PHY_309_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_309_REGOFF)
+#define HWIO_DDR_PHY_309_PHY_DDL_TEST_OBS_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_309_PHY_DDL_TEST_OBS_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_310_REGOFF 0x4d8
+#define HWIO_DDR_PHY_310_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_310_REGOFF)
+#define HWIO_DDR_PHY_310_PHY_DDL_TEST_MSTR_DLY_OBS_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_310_PHY_DDL_TEST_MSTR_DLY_OBS_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_311_REGOFF 0x4dc
+#define HWIO_DDR_PHY_311_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_311_REGOFF)
+#define HWIO_DDR_PHY_311_PHY_DDL_TRACK_UPD_THRESHOLD_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_311_PHY_DDL_TRACK_UPD_THRESHOLD_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_311_PHY_LP4_WDQS_OE_EXTEND_2_FLDMASK (0x100)
+#define HWIO_DDR_PHY_311_PHY_LP4_WDQS_OE_EXTEND_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_311_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_311_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_311_SC_PHY_RX_CAL_START_2_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_311_SC_PHY_RX_CAL_START_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_311_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_311_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_311_PHY_RX_CAL_OVERRIDE_2_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_311_PHY_RX_CAL_OVERRIDE_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_311_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_311_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_312_REGOFF 0x4e0
+#define HWIO_DDR_PHY_312_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_312_REGOFF)
+#define HWIO_DDR_PHY_312_PHY_RX_CAL_SAMPLE_WAIT_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_312_PHY_RX_CAL_SAMPLE_WAIT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_312_PHY_RX_CAL_DQ0_2_FLDMASK (0xfff00)
+#define HWIO_DDR_PHY_312_PHY_RX_CAL_DQ0_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_312_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_312_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_312_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_312_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_313_REGOFF 0x4e4
+#define HWIO_DDR_PHY_313_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_313_REGOFF)
+#define HWIO_DDR_PHY_313_PHY_RX_CAL_DQ1_2_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_313_PHY_RX_CAL_DQ1_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_313_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_313_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_313_PHY_RX_CAL_DQ2_2_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_313_PHY_RX_CAL_DQ2_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_313_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_313_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_314_REGOFF 0x4e8
+#define HWIO_DDR_PHY_314_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_314_REGOFF)
+#define HWIO_DDR_PHY_314_PHY_RX_CAL_DQ3_2_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_314_PHY_RX_CAL_DQ3_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_314_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_314_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_314_PHY_RX_CAL_DQ4_2_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_314_PHY_RX_CAL_DQ4_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_314_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_314_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_315_REGOFF 0x4ec
+#define HWIO_DDR_PHY_315_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_315_REGOFF)
+#define HWIO_DDR_PHY_315_PHY_RX_CAL_DQ5_2_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_315_PHY_RX_CAL_DQ5_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_315_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_315_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_315_PHY_RX_CAL_DQ6_2_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_315_PHY_RX_CAL_DQ6_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_315_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_315_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_316_REGOFF 0x4f0
+#define HWIO_DDR_PHY_316_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_316_REGOFF)
+#define HWIO_DDR_PHY_316_PHY_RX_CAL_DQ7_2_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_316_PHY_RX_CAL_DQ7_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_316_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_316_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_316_PHY_RX_CAL_DM_2_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_316_PHY_RX_CAL_DM_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_316_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_316_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_317_REGOFF 0x4f4
+#define HWIO_DDR_PHY_317_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_317_REGOFF)
+#define HWIO_DDR_PHY_317_PHY_RX_CAL_DQS_2_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_317_PHY_RX_CAL_DQS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_317_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_317_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_317_PHY_RX_CAL_FDBK_2_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_317_PHY_RX_CAL_FDBK_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_317_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_317_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_318_REGOFF 0x4f8
+#define HWIO_DDR_PHY_318_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_318_REGOFF)
+#define HWIO_DDR_PHY_318_PHY_RX_CAL_OBS_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_318_PHY_RX_CAL_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_318_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_318_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_318_PHY_RX_CAL_LOCK_OBS_2_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_318_PHY_RX_CAL_LOCK_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_318_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_318_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_319_REGOFF 0x4fc
+#define HWIO_DDR_PHY_319_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_319_REGOFF)
+#define HWIO_DDR_PHY_319_PHY_RX_CAL_DISABLE_2_FLDMASK (0x1)
+#define HWIO_DDR_PHY_319_PHY_RX_CAL_DISABLE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_319_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_319_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_319_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_319_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_320_REGOFF 0x500
+#define HWIO_DDR_PHY_320_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_320_REGOFF)
+#define HWIO_DDR_PHY_320_PHY_CLK_WRDQ0_SLAVE_DELAY_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_320_PHY_CLK_WRDQ0_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_320_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_320_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_320_PHY_CLK_WRDQ1_SLAVE_DELAY_2_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_320_PHY_CLK_WRDQ1_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_320_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_320_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_321_REGOFF 0x504
+#define HWIO_DDR_PHY_321_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_321_REGOFF)
+#define HWIO_DDR_PHY_321_PHY_CLK_WRDQ2_SLAVE_DELAY_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_321_PHY_CLK_WRDQ2_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_321_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_321_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_321_PHY_CLK_WRDQ3_SLAVE_DELAY_2_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_321_PHY_CLK_WRDQ3_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_321_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_321_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_322_REGOFF 0x508
+#define HWIO_DDR_PHY_322_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_322_REGOFF)
+#define HWIO_DDR_PHY_322_PHY_CLK_WRDQ4_SLAVE_DELAY_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_322_PHY_CLK_WRDQ4_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_322_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_322_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_322_PHY_CLK_WRDQ5_SLAVE_DELAY_2_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_322_PHY_CLK_WRDQ5_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_322_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_322_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_323_REGOFF 0x50c
+#define HWIO_DDR_PHY_323_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_323_REGOFF)
+#define HWIO_DDR_PHY_323_PHY_CLK_WRDQ6_SLAVE_DELAY_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_323_PHY_CLK_WRDQ6_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_323_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_323_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_323_PHY_CLK_WRDQ7_SLAVE_DELAY_2_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_323_PHY_CLK_WRDQ7_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_323_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_323_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_324_REGOFF 0x510
+#define HWIO_DDR_PHY_324_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_324_REGOFF)
+#define HWIO_DDR_PHY_324_PHY_CLK_WRDM_SLAVE_DELAY_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_324_PHY_CLK_WRDM_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_324_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_324_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_324_PHY_CLK_WRDQS_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_324_PHY_CLK_WRDQS_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_324_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_324_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_325_REGOFF 0x514
+#define HWIO_DDR_PHY_325_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_325_REGOFF)
+#define HWIO_DDR_PHY_325_PHY_WRLVL_THRESHOLD_ADJUST_2_FLDMASK (0x3)
+#define HWIO_DDR_PHY_325_PHY_WRLVL_THRESHOLD_ADJUST_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_325_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_325_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_325_PHY_RDDQ0_SLAVE_DELAY_2_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_325_PHY_RDDQ0_SLAVE_DELAY_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_325_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_325_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_325_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_325_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_326_REGOFF 0x518
+#define HWIO_DDR_PHY_326_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_326_REGOFF)
+#define HWIO_DDR_PHY_326_PHY_RDDQ1_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_326_PHY_RDDQ1_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_326_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_326_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_326_PHY_RDDQ2_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_326_PHY_RDDQ2_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_326_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_326_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_327_REGOFF 0x51c
+#define HWIO_DDR_PHY_327_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_327_REGOFF)
+#define HWIO_DDR_PHY_327_PHY_RDDQ3_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_327_PHY_RDDQ3_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_327_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_327_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_327_PHY_RDDQ4_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_327_PHY_RDDQ4_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_327_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_327_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_328_REGOFF 0x520
+#define HWIO_DDR_PHY_328_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_328_REGOFF)
+#define HWIO_DDR_PHY_328_PHY_RDDQ5_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_328_PHY_RDDQ5_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_328_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_328_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_328_PHY_RDDQ6_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_328_PHY_RDDQ6_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_328_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_328_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_329_REGOFF 0x524
+#define HWIO_DDR_PHY_329_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_329_REGOFF)
+#define HWIO_DDR_PHY_329_PHY_RDDQ7_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_329_PHY_RDDQ7_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_329_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_329_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_329_PHY_RDDM_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_329_PHY_RDDM_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_329_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_329_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_330_REGOFF 0x528
+#define HWIO_DDR_PHY_330_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_330_REGOFF)
+#define HWIO_DDR_PHY_330_PHY_RDDQS_DQ0_RISE_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_330_PHY_RDDQS_DQ0_RISE_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_330_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_330_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_330_PHY_RDDQS_DQ0_FALL_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_330_PHY_RDDQS_DQ0_FALL_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_330_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_330_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_331_REGOFF 0x52c
+#define HWIO_DDR_PHY_331_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_331_REGOFF)
+#define HWIO_DDR_PHY_331_PHY_RDDQS_DQ1_RISE_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_331_PHY_RDDQS_DQ1_RISE_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_331_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_331_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_331_PHY_RDDQS_DQ1_FALL_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_331_PHY_RDDQS_DQ1_FALL_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_331_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_331_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_332_REGOFF 0x530
+#define HWIO_DDR_PHY_332_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_332_REGOFF)
+#define HWIO_DDR_PHY_332_PHY_RDDQS_DQ2_RISE_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_332_PHY_RDDQS_DQ2_RISE_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_332_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_332_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_332_PHY_RDDQS_DQ2_FALL_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_332_PHY_RDDQS_DQ2_FALL_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_332_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_332_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_333_REGOFF 0x534
+#define HWIO_DDR_PHY_333_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_333_REGOFF)
+#define HWIO_DDR_PHY_333_PHY_RDDQS_DQ3_RISE_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_333_PHY_RDDQS_DQ3_RISE_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_333_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_333_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_333_PHY_RDDQS_DQ3_FALL_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_333_PHY_RDDQS_DQ3_FALL_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_333_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_333_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_334_REGOFF 0x538
+#define HWIO_DDR_PHY_334_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_334_REGOFF)
+#define HWIO_DDR_PHY_334_PHY_RDDQS_DQ4_RISE_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_334_PHY_RDDQS_DQ4_RISE_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_334_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_334_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_334_PHY_RDDQS_DQ4_FALL_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_334_PHY_RDDQS_DQ4_FALL_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_334_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_334_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_335_REGOFF 0x53c
+#define HWIO_DDR_PHY_335_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_335_REGOFF)
+#define HWIO_DDR_PHY_335_PHY_RDDQS_DQ5_RISE_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_335_PHY_RDDQS_DQ5_RISE_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_335_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_335_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_335_PHY_RDDQS_DQ5_FALL_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_335_PHY_RDDQS_DQ5_FALL_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_335_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_335_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_336_REGOFF 0x540
+#define HWIO_DDR_PHY_336_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_336_REGOFF)
+#define HWIO_DDR_PHY_336_PHY_RDDQS_DQ6_RISE_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_336_PHY_RDDQS_DQ6_RISE_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_336_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_336_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_336_PHY_RDDQS_DQ6_FALL_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_336_PHY_RDDQS_DQ6_FALL_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_336_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_336_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_337_REGOFF 0x544
+#define HWIO_DDR_PHY_337_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_337_REGOFF)
+#define HWIO_DDR_PHY_337_PHY_RDDQS_DQ7_RISE_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_337_PHY_RDDQS_DQ7_RISE_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_337_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_337_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_337_PHY_RDDQS_DQ7_FALL_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_337_PHY_RDDQS_DQ7_FALL_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_337_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_337_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_338_REGOFF 0x548
+#define HWIO_DDR_PHY_338_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_338_REGOFF)
+#define HWIO_DDR_PHY_338_PHY_RDDQS_DM_RISE_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_338_PHY_RDDQS_DM_RISE_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_338_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_338_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_338_PHY_RDDQS_DM_FALL_SLAVE_DELAY_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_338_PHY_RDDQS_DM_FALL_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_338_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_338_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_339_REGOFF 0x54c
+#define HWIO_DDR_PHY_339_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_339_REGOFF)
+#define HWIO_DDR_PHY_339_PHY_RDDQS_GATE_SLAVE_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_339_PHY_RDDQS_GATE_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_339_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_339_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_339_PHY_RDDQS_LATENCY_ADJUST_2_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_339_PHY_RDDQS_LATENCY_ADJUST_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_339_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_339_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PHY_339_PHY_WRITE_PATH_LAT_ADD_2_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_339_PHY_WRITE_PATH_LAT_ADD_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_339_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_339_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_340_REGOFF 0x550
+#define HWIO_DDR_PHY_340_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_340_REGOFF)
+#define HWIO_DDR_PHY_340_PHY_WRLVL_DELAY_EARLY_THRESHOLD_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_340_PHY_WRLVL_DELAY_EARLY_THRESHOLD_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_340_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_340_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_340_PHY_WRLVL_DELAY_PERIOD_THRESHOLD_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_340_PHY_WRLVL_DELAY_PERIOD_THRESHOLD_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_340_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_340_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_341_REGOFF 0x554
+#define HWIO_DDR_PHY_341_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_341_REGOFF)
+#define HWIO_DDR_PHY_341_PHY_WRLVL_EARLY_FORCE_0_2_FLDMASK (0x1)
+#define HWIO_DDR_PHY_341_PHY_WRLVL_EARLY_FORCE_0_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_341_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_341_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_341_PHY_GTLVL_RDDQS_SLV_DLY_START_2_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_341_PHY_GTLVL_RDDQS_SLV_DLY_START_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_341_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_341_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_341_PHY_GTLVL_LAT_ADJ_START_2_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_341_PHY_GTLVL_LAT_ADJ_START_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_341_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_341_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_342_REGOFF 0x558
+#define HWIO_DDR_PHY_342_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_342_REGOFF)
+#define HWIO_DDR_PHY_342_PHY_WDQLVL_DQDM_SLV_DLY_START_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_342_PHY_WDQLVL_DQDM_SLV_DLY_START_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_342_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_342_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_342_PHY_RDLVL_RDDQS_DQ_SLV_DLY_START_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_342_PHY_RDLVL_RDDQS_DQ_SLV_DLY_START_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_342_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_342_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_343_REGOFF 0x55c
+#define HWIO_DDR_PHY_343_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_343_REGOFF)
+#define HWIO_DDR_PHY_343_PHY_FDBK_PWR_CTRL_2_FLDMASK (0x7)
+#define HWIO_DDR_PHY_343_PHY_FDBK_PWR_CTRL_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_343_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_343_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_343_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_343_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_344_REGOFF 0x560
+#define HWIO_DDR_PHY_344_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_344_REGOFF)
+#define HWIO_DDR_PHY_344_PHY_DQ_OE_TIMING_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_344_PHY_DQ_OE_TIMING_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_344_PHY_DQ_TSEL_RD_TIMING_2_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_344_PHY_DQ_TSEL_RD_TIMING_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_344_PHY_DQ_TSEL_WR_TIMING_2_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_344_PHY_DQ_TSEL_WR_TIMING_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_344_PHY_DQS_OE_TIMING_2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_344_PHY_DQS_OE_TIMING_2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_345_REGOFF 0x564
+#define HWIO_DDR_PHY_345_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_345_REGOFF)
+#define HWIO_DDR_PHY_345_PHY_DQS_TSEL_RD_TIMING_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_345_PHY_DQS_TSEL_RD_TIMING_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_345_PHY_DQS_TSEL_WR_TIMING_2_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_345_PHY_DQS_TSEL_WR_TIMING_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_345_PHY_DQ_IE_TIMING_2_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_345_PHY_DQ_IE_TIMING_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_345_PHY_DQS_IE_TIMING_2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_345_PHY_DQS_IE_TIMING_2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_346_REGOFF 0x568
+#define HWIO_DDR_PHY_346_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_346_REGOFF)
+#define HWIO_DDR_PHY_346_PHY_RDDATA_EN_IE_DLY_2_FLDMASK (0x3)
+#define HWIO_DDR_PHY_346_PHY_RDDATA_EN_IE_DLY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_346_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_346_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_346_PHY_IE_MODE_2_FLDMASK (0x300)
+#define HWIO_DDR_PHY_346_PHY_IE_MODE_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_346_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_346_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_346_PHY_RDDATA_EN_DLY_2_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_346_PHY_RDDATA_EN_DLY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_346_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_346_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_346_PHY_RDDATA_EN_TSEL_DLY_2_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_346_PHY_RDDATA_EN_TSEL_DLY_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_346_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_346_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_347_REGOFF 0x56c
+#define HWIO_DDR_PHY_347_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_347_REGOFF)
+#define HWIO_DDR_PHY_347_PHY_SW_MASTER_MODE_2_FLDMASK (0xf)
+#define HWIO_DDR_PHY_347_PHY_SW_MASTER_MODE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_347_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_347_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_347_PHY_MASTER_DELAY_START_2_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_347_PHY_MASTER_DELAY_START_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_347_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_347_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_347_PHY_MASTER_DELAY_STEP_2_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_347_PHY_MASTER_DELAY_STEP_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_347_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_347_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_348_REGOFF 0x570
+#define HWIO_DDR_PHY_348_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_348_REGOFF)
+#define HWIO_DDR_PHY_348_PHY_MASTER_DELAY_WAIT_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_348_PHY_MASTER_DELAY_WAIT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_348_PHY_MASTER_DELAY_HALF_MEASURE_2_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_348_PHY_MASTER_DELAY_HALF_MEASURE_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_348_PHY_RPTR_UPDATE_2_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_348_PHY_RPTR_UPDATE_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_348_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_348_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_348_PHY_WRLVL_DLY_STEP_2_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_348_PHY_WRLVL_DLY_STEP_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_348_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_348_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_349_REGOFF 0x574
+#define HWIO_DDR_PHY_349_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_349_REGOFF)
+#define HWIO_DDR_PHY_349_PHY_WRLVL_RESP_WAIT_CNT_2_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_349_PHY_WRLVL_RESP_WAIT_CNT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_349_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_349_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_349_PHY_GTLVL_DLY_STEP_2_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_349_PHY_GTLVL_DLY_STEP_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_349_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_349_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_349_PHY_GTLVL_RESP_WAIT_CNT_2_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_349_PHY_GTLVL_RESP_WAIT_CNT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_349_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_349_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_349_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_349_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_350_REGOFF 0x578
+#define HWIO_DDR_PHY_350_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_350_REGOFF)
+#define HWIO_DDR_PHY_350_PHY_GTLVL_BACK_STEP_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_350_PHY_GTLVL_BACK_STEP_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_350_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_350_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_350_PHY_GTLVL_FINAL_STEP_2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_350_PHY_GTLVL_FINAL_STEP_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_350_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_350_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_351_REGOFF 0x57c
+#define HWIO_DDR_PHY_351_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_351_REGOFF)
+#define HWIO_DDR_PHY_351_PHY_WDQLVL_DLY_STEP_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_351_PHY_WDQLVL_DLY_STEP_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_351_PHY_RDLVL_DLY_STEP_2_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_351_PHY_RDLVL_DLY_STEP_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_351_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_351_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_351_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_351_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_352_REGOFF 0x580
+#define HWIO_DDR_PHY_352_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_352_REGOFF)
+#define HWIO_DDR_PHY_352_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_352_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_353_REGOFF 0x584
+#define HWIO_DDR_PHY_353_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_353_REGOFF)
+#define HWIO_DDR_PHY_353_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_353_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_354_REGOFF 0x588
+#define HWIO_DDR_PHY_354_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_354_REGOFF)
+#define HWIO_DDR_PHY_354_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_354_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_355_REGOFF 0x58c
+#define HWIO_DDR_PHY_355_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_355_REGOFF)
+#define HWIO_DDR_PHY_355_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_355_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_356_REGOFF 0x590
+#define HWIO_DDR_PHY_356_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_356_REGOFF)
+#define HWIO_DDR_PHY_356_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_356_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_357_REGOFF 0x594
+#define HWIO_DDR_PHY_357_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_357_REGOFF)
+#define HWIO_DDR_PHY_357_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_357_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_358_REGOFF 0x598
+#define HWIO_DDR_PHY_358_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_358_REGOFF)
+#define HWIO_DDR_PHY_358_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_358_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_359_REGOFF 0x59c
+#define HWIO_DDR_PHY_359_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_359_REGOFF)
+#define HWIO_DDR_PHY_359_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_359_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_360_REGOFF 0x5a0
+#define HWIO_DDR_PHY_360_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_360_REGOFF)
+#define HWIO_DDR_PHY_360_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_360_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_361_REGOFF 0x5a4
+#define HWIO_DDR_PHY_361_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_361_REGOFF)
+#define HWIO_DDR_PHY_361_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_361_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_362_REGOFF 0x5a8
+#define HWIO_DDR_PHY_362_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_362_REGOFF)
+#define HWIO_DDR_PHY_362_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_362_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_363_REGOFF 0x5ac
+#define HWIO_DDR_PHY_363_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_363_REGOFF)
+#define HWIO_DDR_PHY_363_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_363_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_364_REGOFF 0x5b0
+#define HWIO_DDR_PHY_364_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_364_REGOFF)
+#define HWIO_DDR_PHY_364_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_364_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_365_REGOFF 0x5b4
+#define HWIO_DDR_PHY_365_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_365_REGOFF)
+#define HWIO_DDR_PHY_365_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_365_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_366_REGOFF 0x5b8
+#define HWIO_DDR_PHY_366_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_366_REGOFF)
+#define HWIO_DDR_PHY_366_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_366_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_367_REGOFF 0x5bc
+#define HWIO_DDR_PHY_367_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_367_REGOFF)
+#define HWIO_DDR_PHY_367_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_367_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_368_REGOFF 0x5c0
+#define HWIO_DDR_PHY_368_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_368_REGOFF)
+#define HWIO_DDR_PHY_368_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_368_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_369_REGOFF 0x5c4
+#define HWIO_DDR_PHY_369_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_369_REGOFF)
+#define HWIO_DDR_PHY_369_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_369_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_370_REGOFF 0x5c8
+#define HWIO_DDR_PHY_370_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_370_REGOFF)
+#define HWIO_DDR_PHY_370_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_370_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_371_REGOFF 0x5cc
+#define HWIO_DDR_PHY_371_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_371_REGOFF)
+#define HWIO_DDR_PHY_371_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_371_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_372_REGOFF 0x5d0
+#define HWIO_DDR_PHY_372_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_372_REGOFF)
+#define HWIO_DDR_PHY_372_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_372_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_373_REGOFF 0x5d4
+#define HWIO_DDR_PHY_373_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_373_REGOFF)
+#define HWIO_DDR_PHY_373_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_373_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_374_REGOFF 0x5d8
+#define HWIO_DDR_PHY_374_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_374_REGOFF)
+#define HWIO_DDR_PHY_374_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_374_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_375_REGOFF 0x5dc
+#define HWIO_DDR_PHY_375_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_375_REGOFF)
+#define HWIO_DDR_PHY_375_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_375_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_376_REGOFF 0x5e0
+#define HWIO_DDR_PHY_376_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_376_REGOFF)
+#define HWIO_DDR_PHY_376_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_376_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_377_REGOFF 0x5e4
+#define HWIO_DDR_PHY_377_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_377_REGOFF)
+#define HWIO_DDR_PHY_377_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_377_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_378_REGOFF 0x5e8
+#define HWIO_DDR_PHY_378_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_378_REGOFF)
+#define HWIO_DDR_PHY_378_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_378_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_379_REGOFF 0x5ec
+#define HWIO_DDR_PHY_379_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_379_REGOFF)
+#define HWIO_DDR_PHY_379_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_379_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_380_REGOFF 0x5f0
+#define HWIO_DDR_PHY_380_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_380_REGOFF)
+#define HWIO_DDR_PHY_380_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_380_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_381_REGOFF 0x5f4
+#define HWIO_DDR_PHY_381_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_381_REGOFF)
+#define HWIO_DDR_PHY_381_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_381_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_382_REGOFF 0x5f8
+#define HWIO_DDR_PHY_382_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_382_REGOFF)
+#define HWIO_DDR_PHY_382_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_382_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_383_REGOFF 0x5fc
+#define HWIO_DDR_PHY_383_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_383_REGOFF)
+#define HWIO_DDR_PHY_383_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_383_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_384_REGOFF 0x600
+#define HWIO_DDR_PHY_384_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_384_REGOFF)
+#define HWIO_DDR_PHY_384_PHY_DQ_DM_SWIZZLE0_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_384_PHY_DQ_DM_SWIZZLE0_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_385_REGOFF 0x604
+#define HWIO_DDR_PHY_385_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_385_REGOFF)
+#define HWIO_DDR_PHY_385_PHY_DQ_DM_SWIZZLE1_3_FLDMASK (0xf)
+#define HWIO_DDR_PHY_385_PHY_DQ_DM_SWIZZLE1_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_385_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_385_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_385_PHY_CLK_WR_BYPASS_SLAVE_DELAY_3_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_385_PHY_CLK_WR_BYPASS_SLAVE_DELAY_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_385_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_385_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_385_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_385_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_386_REGOFF 0x608
+#define HWIO_DDR_PHY_386_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_386_REGOFF)
+#define HWIO_DDR_PHY_386_PHY_RDDQS_GATE_BYPASS_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_386_PHY_RDDQS_GATE_BYPASS_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_386_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_386_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_386_PHY_BYPASS_TWO_CYC_PREAMBLE_3_FLDMASK (0x30000)
+#define HWIO_DDR_PHY_386_PHY_BYPASS_TWO_CYC_PREAMBLE_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_386_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_386_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_386_PHY_CLK_BYPASS_OVERRIDE_3_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_386_PHY_CLK_BYPASS_OVERRIDE_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_386_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_386_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_387_REGOFF 0x60c
+#define HWIO_DDR_PHY_387_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_387_REGOFF)
+#define HWIO_DDR_PHY_387_PHY_SW_WRDQ0_SHIFT_3_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_387_PHY_SW_WRDQ0_SHIFT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_387_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_387_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_387_PHY_SW_WRDQ1_SHIFT_3_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_387_PHY_SW_WRDQ1_SHIFT_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_387_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_387_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_387_PHY_SW_WRDQ2_SHIFT_3_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_387_PHY_SW_WRDQ2_SHIFT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_387_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_387_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_387_PHY_SW_WRDQ3_SHIFT_3_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_387_PHY_SW_WRDQ3_SHIFT_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_387_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_387_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_388_REGOFF 0x610
+#define HWIO_DDR_PHY_388_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_388_REGOFF)
+#define HWIO_DDR_PHY_388_PHY_SW_WRDQ4_SHIFT_3_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_388_PHY_SW_WRDQ4_SHIFT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_388_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_388_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_388_PHY_SW_WRDQ5_SHIFT_3_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_388_PHY_SW_WRDQ5_SHIFT_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_388_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_388_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_388_PHY_SW_WRDQ6_SHIFT_3_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_388_PHY_SW_WRDQ6_SHIFT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_388_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_388_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_388_PHY_SW_WRDQ7_SHIFT_3_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_388_PHY_SW_WRDQ7_SHIFT_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_388_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_388_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_389_REGOFF 0x614
+#define HWIO_DDR_PHY_389_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_389_REGOFF)
+#define HWIO_DDR_PHY_389_PHY_SW_WRDM_SHIFT_3_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_389_PHY_SW_WRDM_SHIFT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_389_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_389_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_389_PHY_SW_WRDQS_SHIFT_3_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_389_PHY_SW_WRDQS_SHIFT_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_389_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_389_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_389_PHY_DQ_TSEL_ENABLE_3_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_389_PHY_DQ_TSEL_ENABLE_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_389_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_389_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_389_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_389_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_390_REGOFF 0x618
+#define HWIO_DDR_PHY_390_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_390_REGOFF)
+#define HWIO_DDR_PHY_390_PHY_DQ_TSEL_SELECT_3_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_390_PHY_DQ_TSEL_SELECT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_390_PHY_DQS_TSEL_ENABLE_3_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_390_PHY_DQS_TSEL_ENABLE_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_390_RESERVED_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_390_RESERVED_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_391_REGOFF 0x61c
+#define HWIO_DDR_PHY_391_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_391_REGOFF)
+#define HWIO_DDR_PHY_391_PHY_DQS_TSEL_SELECT_3_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_391_PHY_DQS_TSEL_SELECT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_391_PHY_TWO_CYC_PREAMBLE_3_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_391_PHY_TWO_CYC_PREAMBLE_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_391_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_391_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_392_REGOFF 0x620
+#define HWIO_DDR_PHY_392_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_392_REGOFF)
+#define HWIO_DDR_PHY_392_PHY_DBI_MODE_3_FLDMASK (0x1)
+#define HWIO_DDR_PHY_392_PHY_DBI_MODE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_392_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_392_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_392_PHY_LP4_BOOT_RDDATA_EN_IE_DLY_3_FLDMASK (0x300)
+#define HWIO_DDR_PHY_392_PHY_LP4_BOOT_RDDATA_EN_IE_DLY_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_392_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_392_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_392_PHY_LP4_BOOT_RDDATA_EN_DLY_3_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_392_PHY_LP4_BOOT_RDDATA_EN_DLY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_392_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_392_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_392_PHY_LP4_BOOT_RDDATA_EN_TSEL_DLY_3_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_392_PHY_LP4_BOOT_RDDATA_EN_TSEL_DLY_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_392_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_392_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_393_REGOFF 0x624
+#define HWIO_DDR_PHY_393_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_393_REGOFF)
+#define HWIO_DDR_PHY_393_PHY_LP4_BOOT_RPTR_UPDATE_3_FLDMASK (0xf)
+#define HWIO_DDR_PHY_393_PHY_LP4_BOOT_RPTR_UPDATE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_393_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_393_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_393_PHY_LP4_BOOT_RDDQS_LATENCY_ADJUST_3_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_393_PHY_LP4_BOOT_RDDQS_LATENCY_ADJUST_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_393_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_393_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_393_PHY_LPBK_CONTROL_3_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_393_PHY_LPBK_CONTROL_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_393_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_393_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_394_REGOFF 0x628
+#define HWIO_DDR_PHY_394_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_394_REGOFF)
+#define HWIO_DDR_PHY_394_PHY_LPBK_DFX_TIMEOUT_EN_3_FLDMASK (0x1)
+#define HWIO_DDR_PHY_394_PHY_LPBK_DFX_TIMEOUT_EN_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_394_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_394_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_394_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_394_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_395_REGOFF 0x62c
+#define HWIO_DDR_PHY_395_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_395_REGOFF)
+#define HWIO_DDR_PHY_395_PHY_AUTO_TIMING_MARGIN_CONTROL_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_395_PHY_AUTO_TIMING_MARGIN_CONTROL_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_396_REGOFF 0x630
+#define HWIO_DDR_PHY_396_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_396_REGOFF)
+#define HWIO_DDR_PHY_396_PHY_AUTO_TIMING_MARGIN_OBS_3_FLDMASK (0xfffffff)
+#define HWIO_DDR_PHY_396_PHY_AUTO_TIMING_MARGIN_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_396_RESERVED_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_396_RESERVED_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_397_REGOFF 0x634
+#define HWIO_DDR_PHY_397_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_397_REGOFF)
+#define HWIO_DDR_PHY_397_PHY_SLICE_PWR_RDC_DISABLE_3_FLDMASK (0x1)
+#define HWIO_DDR_PHY_397_PHY_SLICE_PWR_RDC_DISABLE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_397_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_397_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_397_PHY_PRBS_PATTERN_START_3_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_397_PHY_PRBS_PATTERN_START_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_397_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_397_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_397_PHY_PRBS_PATTERN_MASK_3_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_397_PHY_PRBS_PATTERN_MASK_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_397_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_397_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_398_REGOFF 0x638
+#define HWIO_DDR_PHY_398_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_398_REGOFF)
+#define HWIO_DDR_PHY_398_PHY_RDDQS_DQ_BYPASS_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_398_PHY_RDDQS_DQ_BYPASS_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_398_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_398_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_398_PHY_GATE_ERROR_DELAY_SELECT_3_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_398_PHY_GATE_ERROR_DELAY_SELECT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_398_RESERVED1_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_398_RESERVED1_FLDSHFT (21)
+#define HWIO_DDR_PHY_398_SC_PHY_SNAP_OBS_REGS_3_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_398_SC_PHY_SNAP_OBS_REGS_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_398_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_398_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_399_REGOFF 0x63c
+#define HWIO_DDR_PHY_399_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_399_REGOFF)
+#define HWIO_DDR_PHY_399_PHY_GATE_SMPL1_SLAVE_DELAY_3_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_399_PHY_GATE_SMPL1_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_399_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_399_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_399_PHY_LPDDR_3_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_399_PHY_LPDDR_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_399_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_399_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_399_PHY_LPDDR_TYPE_3_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_399_PHY_LPDDR_TYPE_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_399_RESERVED2_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_399_RESERVED2_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_400_REGOFF 0x640
+#define HWIO_DDR_PHY_400_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_400_REGOFF)
+#define HWIO_DDR_PHY_400_PHY_GATE_SMPL2_SLAVE_DELAY_3_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_400_PHY_GATE_SMPL2_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_400_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_400_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_400_ON_FLY_GATE_ADJUST_EN_3_FLDMASK (0x30000)
+#define HWIO_DDR_PHY_400_ON_FLY_GATE_ADJUST_EN_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_400_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_400_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_400_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_400_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_401_REGOFF 0x644
+#define HWIO_DDR_PHY_401_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_401_REGOFF)
+#define HWIO_DDR_PHY_401_PHY_GATE_TRACKING_OBS_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_401_PHY_GATE_TRACKING_OBS_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_402_REGOFF 0x648
+#define HWIO_DDR_PHY_402_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_402_REGOFF)
+#define HWIO_DDR_PHY_402_PHY_LP4_PST_AMBLE_3_FLDMASK (0x3)
+#define HWIO_DDR_PHY_402_PHY_LP4_PST_AMBLE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_402_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_402_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_402_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_402_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_403_REGOFF 0x64c
+#define HWIO_DDR_PHY_403_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_403_REGOFF)
+#define HWIO_DDR_PHY_403_PHY_LP4_RDLVL_PATT8_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_403_PHY_LP4_RDLVL_PATT8_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_404_REGOFF 0x650
+#define HWIO_DDR_PHY_404_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_404_REGOFF)
+#define HWIO_DDR_PHY_404_PHY_LP4_RDLVL_PATT9_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_404_PHY_LP4_RDLVL_PATT9_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_405_REGOFF 0x654
+#define HWIO_DDR_PHY_405_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_405_REGOFF)
+#define HWIO_DDR_PHY_405_PHY_LP4_RDLVL_PATT10_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_405_PHY_LP4_RDLVL_PATT10_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_406_REGOFF 0x658
+#define HWIO_DDR_PHY_406_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_406_REGOFF)
+#define HWIO_DDR_PHY_406_PHY_LP4_RDLVL_PATT11_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_406_PHY_LP4_RDLVL_PATT11_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_407_REGOFF 0x65c
+#define HWIO_DDR_PHY_407_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_407_REGOFF)
+#define HWIO_DDR_PHY_407_PHY_SLAVE_LOOP_CNT_UPDATE_3_FLDMASK (0x7)
+#define HWIO_DDR_PHY_407_PHY_SLAVE_LOOP_CNT_UPDATE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_407_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_407_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_407_PHY_SW_FIFO_PTR_RST_DISABLE_3_FLDMASK (0x100)
+#define HWIO_DDR_PHY_407_PHY_SW_FIFO_PTR_RST_DISABLE_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_407_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_407_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_407_PHY_MASTER_DLY_LOCK_OBS_SELECT_3_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_407_PHY_MASTER_DLY_LOCK_OBS_SELECT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_407_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_407_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_407_PHY_RDDQ_ENC_OBS_SELECT_3_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_407_PHY_RDDQ_ENC_OBS_SELECT_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_407_RESERVED3_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_407_RESERVED3_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_408_REGOFF 0x660
+#define HWIO_DDR_PHY_408_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_408_REGOFF)
+#define HWIO_DDR_PHY_408_PHY_RDDQS_DQ_ENC_OBS_SELECT_3_FLDMASK (0xf)
+#define HWIO_DDR_PHY_408_PHY_RDDQS_DQ_ENC_OBS_SELECT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_408_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_408_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_408_PHY_WR_ENC_OBS_SELECT_3_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_408_PHY_WR_ENC_OBS_SELECT_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_408_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_408_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_408_PHY_WR_SHIFT_OBS_SELECT_3_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_408_PHY_WR_SHIFT_OBS_SELECT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_408_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_408_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_408_PHY_FIFO_PTR_OBS_SELECT_3_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_408_PHY_FIFO_PTR_OBS_SELECT_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_408_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_408_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_409_REGOFF 0x664
+#define HWIO_DDR_PHY_409_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_409_REGOFF)
+#define HWIO_DDR_PHY_409_PHY_LVL_DEBUG_MODE_3_FLDMASK (0x1)
+#define HWIO_DDR_PHY_409_PHY_LVL_DEBUG_MODE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_409_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_409_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_409_SC_PHY_LVL_DEBUG_CONT_3_FLDMASK (0x100)
+#define HWIO_DDR_PHY_409_SC_PHY_LVL_DEBUG_CONT_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_409_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_409_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_409_PHY_WRLVL_CAPTURE_CNT_3_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_409_PHY_WRLVL_CAPTURE_CNT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_409_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_409_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PHY_409_PHY_WRLVL_UPDT_WAIT_CNT_3_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_409_PHY_WRLVL_UPDT_WAIT_CNT_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_409_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_409_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_410_REGOFF 0x668
+#define HWIO_DDR_PHY_410_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_410_REGOFF)
+#define HWIO_DDR_PHY_410_PHY_GTLVL_CAPTURE_CNT_3_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_410_PHY_GTLVL_CAPTURE_CNT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_410_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_410_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_410_PHY_GTLVL_UPDT_WAIT_CNT_3_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_410_PHY_GTLVL_UPDT_WAIT_CNT_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_410_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_410_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_410_PHY_RDLVL_CAPTURE_CNT_3_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_410_PHY_RDLVL_CAPTURE_CNT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_410_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_410_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PHY_410_PHY_RDLVL_UPDT_WAIT_CNT_3_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_410_PHY_RDLVL_UPDT_WAIT_CNT_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_410_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_410_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_411_REGOFF 0x66c
+#define HWIO_DDR_PHY_411_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_411_REGOFF)
+#define HWIO_DDR_PHY_411_PHY_RDLVL_OP_MODE_3_FLDMASK (0x3)
+#define HWIO_DDR_PHY_411_PHY_RDLVL_OP_MODE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_411_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_411_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_411_PHY_RDLVL_RDDQS_DQ_OBS_SELECT_3_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_411_PHY_RDLVL_RDDQS_DQ_OBS_SELECT_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_411_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_411_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_411_PHY_RDLVL_DATA_MASK_3_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_411_PHY_RDLVL_DATA_MASK_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_411_PHY_WDQLVL_BURST_CNT_3_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_411_PHY_WDQLVL_BURST_CNT_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_411_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_411_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_412_REGOFF 0x670
+#define HWIO_DDR_PHY_412_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_412_REGOFF)
+#define HWIO_DDR_PHY_412_PHY_WDQLVL_PATT_3_FLDMASK (0x7)
+#define HWIO_DDR_PHY_412_PHY_WDQLVL_PATT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_412_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_412_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_412_PHY_WDQLVL_DQDM_SLV_DLY_JUMP_OFFSET_3_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_412_PHY_WDQLVL_DQDM_SLV_DLY_JUMP_OFFSET_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_412_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_412_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_412_PHY_WDQLVL_UPDT_WAIT_CNT_3_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_412_PHY_WDQLVL_UPDT_WAIT_CNT_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_412_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_412_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_413_REGOFF 0x674
+#define HWIO_DDR_PHY_413_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_413_REGOFF)
+#define HWIO_DDR_PHY_413_PHY_WDQLVL_DQDM_OBS_SELECT_3_FLDMASK (0xf)
+#define HWIO_DDR_PHY_413_PHY_WDQLVL_DQDM_OBS_SELECT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_413_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_413_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_413_PHY_WDQLVL_QTR_DLY_STEP_3_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_413_PHY_WDQLVL_QTR_DLY_STEP_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_413_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_413_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_413_SC_PHY_WDQLVL_CLR_PREV_RESULTS_3_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_413_SC_PHY_WDQLVL_CLR_PREV_RESULTS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_413_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_413_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PHY_413_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_413_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_414_REGOFF 0x678
+#define HWIO_DDR_PHY_414_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_414_REGOFF)
+#define HWIO_DDR_PHY_414_PHY_WDQLVL_DATADM_MASK_3_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_414_PHY_WDQLVL_DATADM_MASK_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_414_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_414_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_414_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_414_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_415_REGOFF 0x67c
+#define HWIO_DDR_PHY_415_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_415_REGOFF)
+#define HWIO_DDR_PHY_415_PHY_USER_PATT0_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_415_PHY_USER_PATT0_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_416_REGOFF 0x680
+#define HWIO_DDR_PHY_416_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_416_REGOFF)
+#define HWIO_DDR_PHY_416_PHY_USER_PATT1_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_416_PHY_USER_PATT1_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_417_REGOFF 0x684
+#define HWIO_DDR_PHY_417_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_417_REGOFF)
+#define HWIO_DDR_PHY_417_PHY_USER_PATT2_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_417_PHY_USER_PATT2_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_418_REGOFF 0x688
+#define HWIO_DDR_PHY_418_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_418_REGOFF)
+#define HWIO_DDR_PHY_418_PHY_USER_PATT3_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_418_PHY_USER_PATT3_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_419_REGOFF 0x68c
+#define HWIO_DDR_PHY_419_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_419_REGOFF)
+#define HWIO_DDR_PHY_419_PHY_USER_PATT4_3_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_419_PHY_USER_PATT4_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_419_PHY_CALVL_VREF_DRIVING_SLICE_3_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_419_PHY_CALVL_VREF_DRIVING_SLICE_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_419_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_419_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_419_SC_PHY_MANUAL_CLEAR_3_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_419_SC_PHY_MANUAL_CLEAR_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_419_RESERVED1_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_419_RESERVED1_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_420_REGOFF 0x690
+#define HWIO_DDR_PHY_420_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_420_REGOFF)
+#define HWIO_DDR_PHY_420_PHY_FIFO_PTR_OBS_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_420_PHY_FIFO_PTR_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_420_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_420_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_421_REGOFF 0x694
+#define HWIO_DDR_PHY_421_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_421_REGOFF)
+#define HWIO_DDR_PHY_421_PHY_LPBK_RESULT_OBS_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_421_PHY_LPBK_RESULT_OBS_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_422_REGOFF 0x698
+#define HWIO_DDR_PHY_422_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_422_REGOFF)
+#define HWIO_DDR_PHY_422_PHY_LPBK_ERROR_COUNT_OBS_3_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_422_PHY_LPBK_ERROR_COUNT_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_422_PHY_MASTER_DLY_LOCK_OBS_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_422_PHY_MASTER_DLY_LOCK_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_422_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_422_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_423_REGOFF 0x69c
+#define HWIO_DDR_PHY_423_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_423_REGOFF)
+#define HWIO_DDR_PHY_423_PHY_RDDQ_SLV_DLY_ENC_OBS_3_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_423_PHY_RDDQ_SLV_DLY_ENC_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_423_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_423_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_423_PHY_RDDQS_BASE_SLV_DLY_ENC_OBS_3_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_423_PHY_RDDQS_BASE_SLV_DLY_ENC_OBS_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_423_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_423_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_423_PHY_RDDQS_DQ_RISE_ADDER_SLV_DLY_ENC_OBS_3_FLDMASK \
+ (0xff0000)
+#define HWIO_DDR_PHY_423_PHY_RDDQS_DQ_RISE_ADDER_SLV_DLY_ENC_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_423_PHY_RDDQS_DQ_FALL_ADDER_SLV_DLY_ENC_OBS_3_FLDMASK \
+ (0xff000000)
+#define HWIO_DDR_PHY_423_PHY_RDDQS_DQ_FALL_ADDER_SLV_DLY_ENC_OBS_3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_424_REGOFF 0x6a0
+#define HWIO_DDR_PHY_424_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_424_REGOFF)
+#define HWIO_DDR_PHY_424_PHY_RDDQS_GATE_SLV_DLY_ENC_OBS_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_424_PHY_RDDQS_GATE_SLV_DLY_ENC_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_424_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_424_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_424_PHY_WRDQS_BASE_SLV_DLY_ENC_OBS_3_FLDMASK (0x7f0000)
+#define HWIO_DDR_PHY_424_PHY_WRDQS_BASE_SLV_DLY_ENC_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_424_RESERVED1_FLDMASK (0x800000)
+#define HWIO_DDR_PHY_424_RESERVED1_FLDSHFT (23)
+#define HWIO_DDR_PHY_424_PHY_WRDQ_BASE_SLV_DLY_ENC_OBS_3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_424_PHY_WRDQ_BASE_SLV_DLY_ENC_OBS_3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_425_REGOFF 0x6a4
+#define HWIO_DDR_PHY_425_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_425_REGOFF)
+#define HWIO_DDR_PHY_425_PHY_WR_ADDER_SLV_DLY_ENC_OBS_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_425_PHY_WR_ADDER_SLV_DLY_ENC_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_425_PHY_WR_SHIFT_OBS_3_FLDMASK (0x700)
+#define HWIO_DDR_PHY_425_PHY_WR_SHIFT_OBS_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_425_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_425_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_425_PHY_WRLVL_HARD0_DELAY_OBS_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_425_PHY_WRLVL_HARD0_DELAY_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_425_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_425_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_426_REGOFF 0x6a8
+#define HWIO_DDR_PHY_426_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_426_REGOFF)
+#define HWIO_DDR_PHY_426_PHY_WRLVL_HARD1_DELAY_OBS_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_426_PHY_WRLVL_HARD1_DELAY_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_426_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_426_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_426_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_426_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_427_REGOFF 0x6ac
+#define HWIO_DDR_PHY_427_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_427_REGOFF)
+#define HWIO_DDR_PHY_427_PHY_WRLVL_STATUS_OBS_3_FLDMASK (0x1ffff)
+#define HWIO_DDR_PHY_427_PHY_WRLVL_STATUS_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_427_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_427_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_427_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_427_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_428_REGOFF 0x6b0
+#define HWIO_DDR_PHY_428_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_428_REGOFF)
+#define HWIO_DDR_PHY_428_PHY_GATE_SMPL1_SLV_DLY_ENC_OBS_3_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_428_PHY_GATE_SMPL1_SLV_DLY_ENC_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_428_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_428_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_428_PHY_GATE_SMPL2_SLV_DLY_ENC_OBS_3_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_428_PHY_GATE_SMPL2_SLV_DLY_ENC_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_428_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_428_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_429_REGOFF 0x6b4
+#define HWIO_DDR_PHY_429_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_429_REGOFF)
+#define HWIO_DDR_PHY_429_PHY_GTLVL_HARD0_DELAY_OBS_3_FLDMASK (0x3fff)
+#define HWIO_DDR_PHY_429_PHY_GTLVL_HARD0_DELAY_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_429_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_PHY_429_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_PHY_429_PHY_GTLVL_HARD1_DELAY_OBS_3_FLDMASK (0x3fff0000)
+#define HWIO_DDR_PHY_429_PHY_GTLVL_HARD1_DELAY_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_429_RESERVED1_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_429_RESERVED1_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_430_REGOFF 0x6b8
+#define HWIO_DDR_PHY_430_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_430_REGOFF)
+#define HWIO_DDR_PHY_430_PHY_GTLVL_STATUS_OBS_3_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_430_PHY_GTLVL_STATUS_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_430_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_430_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_430_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_430_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_431_REGOFF 0x6bc
+#define HWIO_DDR_PHY_431_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_431_REGOFF)
+#define HWIO_DDR_PHY_431_PHY_RDLVL_RDDQS_DQ_LE_DLY_OBS_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_431_PHY_RDLVL_RDDQS_DQ_LE_DLY_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_431_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_431_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_431_PHY_RDLVL_RDDQS_DQ_TE_DLY_OBS_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_431_PHY_RDLVL_RDDQS_DQ_TE_DLY_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_431_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_431_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_432_REGOFF 0x6c0
+#define HWIO_DDR_PHY_432_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_432_REGOFF)
+#define HWIO_DDR_PHY_432_PHY_RDLVL_RDDQS_DQ_NUM_WINDOWS_OBS_3_FLDMASK (0x3)
+#define HWIO_DDR_PHY_432_PHY_RDLVL_RDDQS_DQ_NUM_WINDOWS_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_432_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_432_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_432_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_432_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_433_REGOFF 0x6c4
+#define HWIO_DDR_PHY_433_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_433_REGOFF)
+#define HWIO_DDR_PHY_433_PHY_RDLVL_STATUS_OBS_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_433_PHY_RDLVL_STATUS_OBS_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_434_REGOFF 0x6c8
+#define HWIO_DDR_PHY_434_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_434_REGOFF)
+#define HWIO_DDR_PHY_434_PHY_WDQLVL_DQDM_LE_DLY_OBS_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_434_PHY_WDQLVL_DQDM_LE_DLY_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_434_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_434_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_434_PHY_WDQLVL_DQDM_TE_DLY_OBS_3_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_434_PHY_WDQLVL_DQDM_TE_DLY_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_434_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_434_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_435_REGOFF 0x6cc
+#define HWIO_DDR_PHY_435_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_435_REGOFF)
+#define HWIO_DDR_PHY_435_PHY_WDQLVL_STATUS_OBS_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_435_PHY_WDQLVL_STATUS_OBS_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_436_REGOFF 0x6d0
+#define HWIO_DDR_PHY_436_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_436_REGOFF)
+#define HWIO_DDR_PHY_436_PHY_DDL_MODE_3_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_436_PHY_DDL_MODE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_436_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_436_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_436_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_436_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_437_REGOFF 0x6d4
+#define HWIO_DDR_PHY_437_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_437_REGOFF)
+#define HWIO_DDR_PHY_437_PHY_DDL_TEST_OBS_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_437_PHY_DDL_TEST_OBS_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_438_REGOFF 0x6d8
+#define HWIO_DDR_PHY_438_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_438_REGOFF)
+#define HWIO_DDR_PHY_438_PHY_DDL_TEST_MSTR_DLY_OBS_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_438_PHY_DDL_TEST_MSTR_DLY_OBS_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_439_REGOFF 0x6dc
+#define HWIO_DDR_PHY_439_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_439_REGOFF)
+#define HWIO_DDR_PHY_439_PHY_DDL_TRACK_UPD_THRESHOLD_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_439_PHY_DDL_TRACK_UPD_THRESHOLD_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_439_PHY_LP4_WDQS_OE_EXTEND_3_FLDMASK (0x100)
+#define HWIO_DDR_PHY_439_PHY_LP4_WDQS_OE_EXTEND_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_439_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_439_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_439_SC_PHY_RX_CAL_START_3_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_439_SC_PHY_RX_CAL_START_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_439_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_439_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_439_PHY_RX_CAL_OVERRIDE_3_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_439_PHY_RX_CAL_OVERRIDE_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_439_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_439_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_440_REGOFF 0x6e0
+#define HWIO_DDR_PHY_440_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_440_REGOFF)
+#define HWIO_DDR_PHY_440_PHY_RX_CAL_SAMPLE_WAIT_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_440_PHY_RX_CAL_SAMPLE_WAIT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_440_PHY_RX_CAL_DQ0_3_FLDMASK (0xfff00)
+#define HWIO_DDR_PHY_440_PHY_RX_CAL_DQ0_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_440_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_440_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_440_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_440_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_441_REGOFF 0x6e4
+#define HWIO_DDR_PHY_441_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_441_REGOFF)
+#define HWIO_DDR_PHY_441_PHY_RX_CAL_DQ1_3_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_441_PHY_RX_CAL_DQ1_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_441_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_441_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_441_PHY_RX_CAL_DQ2_3_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_441_PHY_RX_CAL_DQ2_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_441_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_441_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_442_REGOFF 0x6e8
+#define HWIO_DDR_PHY_442_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_442_REGOFF)
+#define HWIO_DDR_PHY_442_PHY_RX_CAL_DQ3_3_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_442_PHY_RX_CAL_DQ3_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_442_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_442_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_442_PHY_RX_CAL_DQ4_3_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_442_PHY_RX_CAL_DQ4_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_442_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_442_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_443_REGOFF 0x6ec
+#define HWIO_DDR_PHY_443_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_443_REGOFF)
+#define HWIO_DDR_PHY_443_PHY_RX_CAL_DQ5_3_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_443_PHY_RX_CAL_DQ5_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_443_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_443_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_443_PHY_RX_CAL_DQ6_3_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_443_PHY_RX_CAL_DQ6_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_443_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_443_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_444_REGOFF 0x6f0
+#define HWIO_DDR_PHY_444_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_444_REGOFF)
+#define HWIO_DDR_PHY_444_PHY_RX_CAL_DQ7_3_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_444_PHY_RX_CAL_DQ7_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_444_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_444_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_444_PHY_RX_CAL_DM_3_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_444_PHY_RX_CAL_DM_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_444_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_444_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_445_REGOFF 0x6f4
+#define HWIO_DDR_PHY_445_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_445_REGOFF)
+#define HWIO_DDR_PHY_445_PHY_RX_CAL_DQS_3_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_445_PHY_RX_CAL_DQS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_445_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_445_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_445_PHY_RX_CAL_FDBK_3_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_445_PHY_RX_CAL_FDBK_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_445_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_445_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_446_REGOFF 0x6f8
+#define HWIO_DDR_PHY_446_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_446_REGOFF)
+#define HWIO_DDR_PHY_446_PHY_RX_CAL_OBS_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_446_PHY_RX_CAL_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_446_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_446_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_446_PHY_RX_CAL_LOCK_OBS_3_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_446_PHY_RX_CAL_LOCK_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_446_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_446_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_447_REGOFF 0x6fc
+#define HWIO_DDR_PHY_447_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_447_REGOFF)
+#define HWIO_DDR_PHY_447_PHY_RX_CAL_DISABLE_3_FLDMASK (0x1)
+#define HWIO_DDR_PHY_447_PHY_RX_CAL_DISABLE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_447_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_447_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_447_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_447_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_448_REGOFF 0x700
+#define HWIO_DDR_PHY_448_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_448_REGOFF)
+#define HWIO_DDR_PHY_448_PHY_CLK_WRDQ0_SLAVE_DELAY_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_448_PHY_CLK_WRDQ0_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_448_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_448_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_448_PHY_CLK_WRDQ1_SLAVE_DELAY_3_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_448_PHY_CLK_WRDQ1_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_448_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_448_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_449_REGOFF 0x704
+#define HWIO_DDR_PHY_449_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_449_REGOFF)
+#define HWIO_DDR_PHY_449_PHY_CLK_WRDQ2_SLAVE_DELAY_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_449_PHY_CLK_WRDQ2_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_449_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_449_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_449_PHY_CLK_WRDQ3_SLAVE_DELAY_3_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_449_PHY_CLK_WRDQ3_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_449_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_449_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_450_REGOFF 0x708
+#define HWIO_DDR_PHY_450_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_450_REGOFF)
+#define HWIO_DDR_PHY_450_PHY_CLK_WRDQ4_SLAVE_DELAY_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_450_PHY_CLK_WRDQ4_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_450_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_450_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_450_PHY_CLK_WRDQ5_SLAVE_DELAY_3_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_450_PHY_CLK_WRDQ5_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_450_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_450_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_451_REGOFF 0x70c
+#define HWIO_DDR_PHY_451_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_451_REGOFF)
+#define HWIO_DDR_PHY_451_PHY_CLK_WRDQ6_SLAVE_DELAY_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_451_PHY_CLK_WRDQ6_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_451_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_451_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_451_PHY_CLK_WRDQ7_SLAVE_DELAY_3_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_451_PHY_CLK_WRDQ7_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_451_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_451_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_452_REGOFF 0x710
+#define HWIO_DDR_PHY_452_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_452_REGOFF)
+#define HWIO_DDR_PHY_452_PHY_CLK_WRDM_SLAVE_DELAY_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_452_PHY_CLK_WRDM_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_452_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_452_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_452_PHY_CLK_WRDQS_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_452_PHY_CLK_WRDQS_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_452_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_452_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_453_REGOFF 0x714
+#define HWIO_DDR_PHY_453_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_453_REGOFF)
+#define HWIO_DDR_PHY_453_PHY_WRLVL_THRESHOLD_ADJUST_3_FLDMASK (0x3)
+#define HWIO_DDR_PHY_453_PHY_WRLVL_THRESHOLD_ADJUST_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_453_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_453_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_453_PHY_RDDQ0_SLAVE_DELAY_3_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_453_PHY_RDDQ0_SLAVE_DELAY_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_453_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_453_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_453_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_453_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_454_REGOFF 0x718
+#define HWIO_DDR_PHY_454_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_454_REGOFF)
+#define HWIO_DDR_PHY_454_PHY_RDDQ1_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_454_PHY_RDDQ1_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_454_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_454_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_454_PHY_RDDQ2_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_454_PHY_RDDQ2_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_454_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_454_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_455_REGOFF 0x71c
+#define HWIO_DDR_PHY_455_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_455_REGOFF)
+#define HWIO_DDR_PHY_455_PHY_RDDQ3_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_455_PHY_RDDQ3_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_455_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_455_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_455_PHY_RDDQ4_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_455_PHY_RDDQ4_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_455_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_455_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_456_REGOFF 0x720
+#define HWIO_DDR_PHY_456_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_456_REGOFF)
+#define HWIO_DDR_PHY_456_PHY_RDDQ5_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_456_PHY_RDDQ5_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_456_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_456_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_456_PHY_RDDQ6_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_456_PHY_RDDQ6_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_456_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_456_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_457_REGOFF 0x724
+#define HWIO_DDR_PHY_457_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_457_REGOFF)
+#define HWIO_DDR_PHY_457_PHY_RDDQ7_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_457_PHY_RDDQ7_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_457_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_457_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_457_PHY_RDDM_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_457_PHY_RDDM_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_457_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_457_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_458_REGOFF 0x728
+#define HWIO_DDR_PHY_458_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_458_REGOFF)
+#define HWIO_DDR_PHY_458_PHY_RDDQS_DQ0_RISE_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_458_PHY_RDDQS_DQ0_RISE_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_458_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_458_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_458_PHY_RDDQS_DQ0_FALL_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_458_PHY_RDDQS_DQ0_FALL_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_458_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_458_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_459_REGOFF 0x72c
+#define HWIO_DDR_PHY_459_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_459_REGOFF)
+#define HWIO_DDR_PHY_459_PHY_RDDQS_DQ1_RISE_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_459_PHY_RDDQS_DQ1_RISE_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_459_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_459_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_459_PHY_RDDQS_DQ1_FALL_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_459_PHY_RDDQS_DQ1_FALL_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_459_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_459_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_460_REGOFF 0x730
+#define HWIO_DDR_PHY_460_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_460_REGOFF)
+#define HWIO_DDR_PHY_460_PHY_RDDQS_DQ2_RISE_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_460_PHY_RDDQS_DQ2_RISE_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_460_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_460_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_460_PHY_RDDQS_DQ2_FALL_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_460_PHY_RDDQS_DQ2_FALL_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_460_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_460_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_461_REGOFF 0x734
+#define HWIO_DDR_PHY_461_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_461_REGOFF)
+#define HWIO_DDR_PHY_461_PHY_RDDQS_DQ3_RISE_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_461_PHY_RDDQS_DQ3_RISE_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_461_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_461_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_461_PHY_RDDQS_DQ3_FALL_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_461_PHY_RDDQS_DQ3_FALL_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_461_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_461_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_462_REGOFF 0x738
+#define HWIO_DDR_PHY_462_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_462_REGOFF)
+#define HWIO_DDR_PHY_462_PHY_RDDQS_DQ4_RISE_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_462_PHY_RDDQS_DQ4_RISE_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_462_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_462_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_462_PHY_RDDQS_DQ4_FALL_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_462_PHY_RDDQS_DQ4_FALL_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_462_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_462_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_463_REGOFF 0x73c
+#define HWIO_DDR_PHY_463_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_463_REGOFF)
+#define HWIO_DDR_PHY_463_PHY_RDDQS_DQ5_RISE_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_463_PHY_RDDQS_DQ5_RISE_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_463_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_463_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_463_PHY_RDDQS_DQ5_FALL_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_463_PHY_RDDQS_DQ5_FALL_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_463_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_463_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_464_REGOFF 0x740
+#define HWIO_DDR_PHY_464_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_464_REGOFF)
+#define HWIO_DDR_PHY_464_PHY_RDDQS_DQ6_RISE_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_464_PHY_RDDQS_DQ6_RISE_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_464_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_464_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_464_PHY_RDDQS_DQ6_FALL_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_464_PHY_RDDQS_DQ6_FALL_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_464_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_464_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_465_REGOFF 0x744
+#define HWIO_DDR_PHY_465_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_465_REGOFF)
+#define HWIO_DDR_PHY_465_PHY_RDDQS_DQ7_RISE_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_465_PHY_RDDQS_DQ7_RISE_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_465_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_465_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_465_PHY_RDDQS_DQ7_FALL_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_465_PHY_RDDQS_DQ7_FALL_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_465_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_465_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_466_REGOFF 0x748
+#define HWIO_DDR_PHY_466_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_466_REGOFF)
+#define HWIO_DDR_PHY_466_PHY_RDDQS_DM_RISE_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_466_PHY_RDDQS_DM_RISE_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_466_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_466_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_466_PHY_RDDQS_DM_FALL_SLAVE_DELAY_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_466_PHY_RDDQS_DM_FALL_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_466_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_466_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_467_REGOFF 0x74c
+#define HWIO_DDR_PHY_467_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_467_REGOFF)
+#define HWIO_DDR_PHY_467_PHY_RDDQS_GATE_SLAVE_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_467_PHY_RDDQS_GATE_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_467_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_467_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_467_PHY_RDDQS_LATENCY_ADJUST_3_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_467_PHY_RDDQS_LATENCY_ADJUST_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_467_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_467_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PHY_467_PHY_WRITE_PATH_LAT_ADD_3_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_467_PHY_WRITE_PATH_LAT_ADD_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_467_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_467_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_468_REGOFF 0x750
+#define HWIO_DDR_PHY_468_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_468_REGOFF)
+#define HWIO_DDR_PHY_468_PHY_WRLVL_DELAY_EARLY_THRESHOLD_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_468_PHY_WRLVL_DELAY_EARLY_THRESHOLD_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_468_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_468_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_468_PHY_WRLVL_DELAY_PERIOD_THRESHOLD_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_468_PHY_WRLVL_DELAY_PERIOD_THRESHOLD_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_468_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_468_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_469_REGOFF 0x754
+#define HWIO_DDR_PHY_469_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_469_REGOFF)
+#define HWIO_DDR_PHY_469_PHY_WRLVL_EARLY_FORCE_0_3_FLDMASK (0x1)
+#define HWIO_DDR_PHY_469_PHY_WRLVL_EARLY_FORCE_0_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_469_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_469_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_469_PHY_GTLVL_RDDQS_SLV_DLY_START_3_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_469_PHY_GTLVL_RDDQS_SLV_DLY_START_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_469_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_469_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_469_PHY_GTLVL_LAT_ADJ_START_3_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_469_PHY_GTLVL_LAT_ADJ_START_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_469_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_469_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_470_REGOFF 0x758
+#define HWIO_DDR_PHY_470_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_470_REGOFF)
+#define HWIO_DDR_PHY_470_PHY_WDQLVL_DQDM_SLV_DLY_START_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_470_PHY_WDQLVL_DQDM_SLV_DLY_START_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_470_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_470_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_470_PHY_RDLVL_RDDQS_DQ_SLV_DLY_START_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_470_PHY_RDLVL_RDDQS_DQ_SLV_DLY_START_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_470_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_470_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_471_REGOFF 0x75c
+#define HWIO_DDR_PHY_471_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_471_REGOFF)
+#define HWIO_DDR_PHY_471_PHY_FDBK_PWR_CTRL_3_FLDMASK (0x7)
+#define HWIO_DDR_PHY_471_PHY_FDBK_PWR_CTRL_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_471_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_471_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_471_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_471_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_472_REGOFF 0x760
+#define HWIO_DDR_PHY_472_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_472_REGOFF)
+#define HWIO_DDR_PHY_472_PHY_DQ_OE_TIMING_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_472_PHY_DQ_OE_TIMING_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_472_PHY_DQ_TSEL_RD_TIMING_3_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_472_PHY_DQ_TSEL_RD_TIMING_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_472_PHY_DQ_TSEL_WR_TIMING_3_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_472_PHY_DQ_TSEL_WR_TIMING_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_472_PHY_DQS_OE_TIMING_3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_472_PHY_DQS_OE_TIMING_3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_473_REGOFF 0x764
+#define HWIO_DDR_PHY_473_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_473_REGOFF)
+#define HWIO_DDR_PHY_473_PHY_DQS_TSEL_RD_TIMING_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_473_PHY_DQS_TSEL_RD_TIMING_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_473_PHY_DQS_TSEL_WR_TIMING_3_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_473_PHY_DQS_TSEL_WR_TIMING_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_473_PHY_DQ_IE_TIMING_3_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_473_PHY_DQ_IE_TIMING_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_473_PHY_DQS_IE_TIMING_3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_473_PHY_DQS_IE_TIMING_3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_474_REGOFF 0x768
+#define HWIO_DDR_PHY_474_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_474_REGOFF)
+#define HWIO_DDR_PHY_474_PHY_RDDATA_EN_IE_DLY_3_FLDMASK (0x3)
+#define HWIO_DDR_PHY_474_PHY_RDDATA_EN_IE_DLY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_474_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_474_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_474_PHY_IE_MODE_3_FLDMASK (0x300)
+#define HWIO_DDR_PHY_474_PHY_IE_MODE_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_474_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_474_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_474_PHY_RDDATA_EN_DLY_3_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_474_PHY_RDDATA_EN_DLY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_474_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_474_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_474_PHY_RDDATA_EN_TSEL_DLY_3_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_474_PHY_RDDATA_EN_TSEL_DLY_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_474_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_474_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_475_REGOFF 0x76c
+#define HWIO_DDR_PHY_475_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_475_REGOFF)
+#define HWIO_DDR_PHY_475_PHY_SW_MASTER_MODE_3_FLDMASK (0xf)
+#define HWIO_DDR_PHY_475_PHY_SW_MASTER_MODE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_475_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_475_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_475_PHY_MASTER_DELAY_START_3_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_475_PHY_MASTER_DELAY_START_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_475_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_475_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_475_PHY_MASTER_DELAY_STEP_3_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_475_PHY_MASTER_DELAY_STEP_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_475_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_475_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_476_REGOFF 0x770
+#define HWIO_DDR_PHY_476_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_476_REGOFF)
+#define HWIO_DDR_PHY_476_PHY_MASTER_DELAY_WAIT_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_476_PHY_MASTER_DELAY_WAIT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_476_PHY_MASTER_DELAY_HALF_MEASURE_3_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_476_PHY_MASTER_DELAY_HALF_MEASURE_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_476_PHY_RPTR_UPDATE_3_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_476_PHY_RPTR_UPDATE_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_476_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_476_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_476_PHY_WRLVL_DLY_STEP_3_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_476_PHY_WRLVL_DLY_STEP_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_476_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_476_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_477_REGOFF 0x774
+#define HWIO_DDR_PHY_477_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_477_REGOFF)
+#define HWIO_DDR_PHY_477_PHY_WRLVL_RESP_WAIT_CNT_3_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_477_PHY_WRLVL_RESP_WAIT_CNT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_477_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_477_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_477_PHY_GTLVL_DLY_STEP_3_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_477_PHY_GTLVL_DLY_STEP_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_477_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_477_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_477_PHY_GTLVL_RESP_WAIT_CNT_3_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_477_PHY_GTLVL_RESP_WAIT_CNT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_477_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_477_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_477_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_477_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_478_REGOFF 0x778
+#define HWIO_DDR_PHY_478_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_478_REGOFF)
+#define HWIO_DDR_PHY_478_PHY_GTLVL_BACK_STEP_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_478_PHY_GTLVL_BACK_STEP_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_478_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_478_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_478_PHY_GTLVL_FINAL_STEP_3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_478_PHY_GTLVL_FINAL_STEP_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_478_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_478_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_479_REGOFF 0x77c
+#define HWIO_DDR_PHY_479_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_479_REGOFF)
+#define HWIO_DDR_PHY_479_PHY_WDQLVL_DLY_STEP_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_479_PHY_WDQLVL_DLY_STEP_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_479_PHY_RDLVL_DLY_STEP_3_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_479_PHY_RDLVL_DLY_STEP_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_479_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_479_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_479_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_479_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_480_REGOFF 0x780
+#define HWIO_DDR_PHY_480_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_480_REGOFF)
+#define HWIO_DDR_PHY_480_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_480_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_481_REGOFF 0x784
+#define HWIO_DDR_PHY_481_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_481_REGOFF)
+#define HWIO_DDR_PHY_481_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_481_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_482_REGOFF 0x788
+#define HWIO_DDR_PHY_482_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_482_REGOFF)
+#define HWIO_DDR_PHY_482_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_482_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_483_REGOFF 0x78c
+#define HWIO_DDR_PHY_483_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_483_REGOFF)
+#define HWIO_DDR_PHY_483_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_483_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_484_REGOFF 0x790
+#define HWIO_DDR_PHY_484_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_484_REGOFF)
+#define HWIO_DDR_PHY_484_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_484_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_485_REGOFF 0x794
+#define HWIO_DDR_PHY_485_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_485_REGOFF)
+#define HWIO_DDR_PHY_485_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_485_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_486_REGOFF 0x798
+#define HWIO_DDR_PHY_486_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_486_REGOFF)
+#define HWIO_DDR_PHY_486_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_486_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_487_REGOFF 0x79c
+#define HWIO_DDR_PHY_487_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_487_REGOFF)
+#define HWIO_DDR_PHY_487_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_487_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_488_REGOFF 0x7a0
+#define HWIO_DDR_PHY_488_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_488_REGOFF)
+#define HWIO_DDR_PHY_488_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_488_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_489_REGOFF 0x7a4
+#define HWIO_DDR_PHY_489_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_489_REGOFF)
+#define HWIO_DDR_PHY_489_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_489_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_490_REGOFF 0x7a8
+#define HWIO_DDR_PHY_490_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_490_REGOFF)
+#define HWIO_DDR_PHY_490_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_490_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_491_REGOFF 0x7ac
+#define HWIO_DDR_PHY_491_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_491_REGOFF)
+#define HWIO_DDR_PHY_491_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_491_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_492_REGOFF 0x7b0
+#define HWIO_DDR_PHY_492_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_492_REGOFF)
+#define HWIO_DDR_PHY_492_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_492_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_493_REGOFF 0x7b4
+#define HWIO_DDR_PHY_493_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_493_REGOFF)
+#define HWIO_DDR_PHY_493_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_493_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_494_REGOFF 0x7b8
+#define HWIO_DDR_PHY_494_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_494_REGOFF)
+#define HWIO_DDR_PHY_494_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_494_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_495_REGOFF 0x7bc
+#define HWIO_DDR_PHY_495_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_495_REGOFF)
+#define HWIO_DDR_PHY_495_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_495_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_496_REGOFF 0x7c0
+#define HWIO_DDR_PHY_496_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_496_REGOFF)
+#define HWIO_DDR_PHY_496_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_496_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_497_REGOFF 0x7c4
+#define HWIO_DDR_PHY_497_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_497_REGOFF)
+#define HWIO_DDR_PHY_497_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_497_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_498_REGOFF 0x7c8
+#define HWIO_DDR_PHY_498_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_498_REGOFF)
+#define HWIO_DDR_PHY_498_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_498_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_499_REGOFF 0x7cc
+#define HWIO_DDR_PHY_499_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_499_REGOFF)
+#define HWIO_DDR_PHY_499_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_499_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_500_REGOFF 0x7d0
+#define HWIO_DDR_PHY_500_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_500_REGOFF)
+#define HWIO_DDR_PHY_500_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_500_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_501_REGOFF 0x7d4
+#define HWIO_DDR_PHY_501_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_501_REGOFF)
+#define HWIO_DDR_PHY_501_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_501_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_502_REGOFF 0x7d8
+#define HWIO_DDR_PHY_502_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_502_REGOFF)
+#define HWIO_DDR_PHY_502_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_502_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_503_REGOFF 0x7dc
+#define HWIO_DDR_PHY_503_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_503_REGOFF)
+#define HWIO_DDR_PHY_503_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_503_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_504_REGOFF 0x7e0
+#define HWIO_DDR_PHY_504_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_504_REGOFF)
+#define HWIO_DDR_PHY_504_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_504_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_505_REGOFF 0x7e4
+#define HWIO_DDR_PHY_505_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_505_REGOFF)
+#define HWIO_DDR_PHY_505_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_505_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_506_REGOFF 0x7e8
+#define HWIO_DDR_PHY_506_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_506_REGOFF)
+#define HWIO_DDR_PHY_506_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_506_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_507_REGOFF 0x7ec
+#define HWIO_DDR_PHY_507_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_507_REGOFF)
+#define HWIO_DDR_PHY_507_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_507_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_508_REGOFF 0x7f0
+#define HWIO_DDR_PHY_508_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_508_REGOFF)
+#define HWIO_DDR_PHY_508_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_508_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_509_REGOFF 0x7f4
+#define HWIO_DDR_PHY_509_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_509_REGOFF)
+#define HWIO_DDR_PHY_509_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_509_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_510_REGOFF 0x7f8
+#define HWIO_DDR_PHY_510_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_510_REGOFF)
+#define HWIO_DDR_PHY_510_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_510_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_511_REGOFF 0x7fc
+#define HWIO_DDR_PHY_511_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_511_REGOFF)
+#define HWIO_DDR_PHY_511_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_511_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_512_REGOFF 0x800
+#define HWIO_DDR_PHY_512_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_512_REGOFF)
+#define HWIO_DDR_PHY_512_PHY_ADR0_SW_WRADDR_SHIFT_0_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_512_PHY_ADR0_SW_WRADDR_SHIFT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_512_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_512_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_512_PHY_ADR1_SW_WRADDR_SHIFT_0_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_512_PHY_ADR1_SW_WRADDR_SHIFT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_512_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_512_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_512_PHY_ADR2_SW_WRADDR_SHIFT_0_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_512_PHY_ADR2_SW_WRADDR_SHIFT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_512_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_512_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_512_PHY_ADR3_SW_WRADDR_SHIFT_0_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_512_PHY_ADR3_SW_WRADDR_SHIFT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_512_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_512_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_513_REGOFF 0x804
+#define HWIO_DDR_PHY_513_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_513_REGOFF)
+#define HWIO_DDR_PHY_513_PHY_ADR4_SW_WRADDR_SHIFT_0_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_513_PHY_ADR4_SW_WRADDR_SHIFT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_513_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_513_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_513_PHY_ADR5_SW_WRADDR_SHIFT_0_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_513_PHY_ADR5_SW_WRADDR_SHIFT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_513_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_513_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_513_PHY_ADR_CLK_WR_BYPASS_SLAVE_DELAY_0_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_513_PHY_ADR_CLK_WR_BYPASS_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_513_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_513_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_514_REGOFF 0x808
+#define HWIO_DDR_PHY_514_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_514_REGOFF)
+#define HWIO_DDR_PHY_514_PHY_ADR_CLK_BYPASS_OVERRIDE_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_514_PHY_ADR_CLK_BYPASS_OVERRIDE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_514_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_514_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_514_SC_PHY_ADR_MANUAL_CLEAR_0_FLDMASK (0x700)
+#define HWIO_DDR_PHY_514_SC_PHY_ADR_MANUAL_CLEAR_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_514_RESERVED1_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_514_RESERVED1_FLDSHFT (11)
+#define HWIO_DDR_PHY_514_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_514_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_515_REGOFF 0x80c
+#define HWIO_DDR_PHY_515_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_515_REGOFF)
+#define HWIO_DDR_PHY_515_PHY_ADR_LPBK_RESULT_OBS_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_515_PHY_ADR_LPBK_RESULT_OBS_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_516_REGOFF 0x810
+#define HWIO_DDR_PHY_516_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_516_REGOFF)
+#define HWIO_DDR_PHY_516_PHY_ADR_LPBK_ERROR_COUNT_OBS_0_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_516_PHY_ADR_LPBK_ERROR_COUNT_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_516_PHY_ADR_MASTER_DLY_LOCK_OBS_SELECT_0_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_516_PHY_ADR_MASTER_DLY_LOCK_OBS_SELECT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_516_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_516_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_516_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_516_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_517_REGOFF 0x814
+#define HWIO_DDR_PHY_517_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_517_REGOFF)
+#define HWIO_DDR_PHY_517_PHY_ADR_MASTER_DLY_LOCK_OBS_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_517_PHY_ADR_MASTER_DLY_LOCK_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_517_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_517_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_517_PHY_ADR_BASE_SLV_DLY_ENC_OBS_0_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_517_PHY_ADR_BASE_SLV_DLY_ENC_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_517_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_517_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_518_REGOFF 0x818
+#define HWIO_DDR_PHY_518_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_518_REGOFF)
+#define HWIO_DDR_PHY_518_PHY_ADR_ADDER_SLV_DLY_ENC_OBS_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_518_PHY_ADR_ADDER_SLV_DLY_ENC_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_518_PHY_ADR_SLAVE_LOOP_CNT_UPDATE_0_FLDMASK (0x700)
+#define HWIO_DDR_PHY_518_PHY_ADR_SLAVE_LOOP_CNT_UPDATE_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_518_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_518_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_518_PHY_ADR_SLV_DLY_ENC_OBS_SELECT_0_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_518_PHY_ADR_SLV_DLY_ENC_OBS_SELECT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_518_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_518_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_518_SC_PHY_ADR_SNAP_OBS_REGS_0_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_518_SC_PHY_ADR_SNAP_OBS_REGS_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_518_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_518_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_519_REGOFF 0x81c
+#define HWIO_DDR_PHY_519_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_519_REGOFF)
+#define HWIO_DDR_PHY_519_PHY_ADR_TSEL_ENABLE_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_519_PHY_ADR_TSEL_ENABLE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_519_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_519_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_519_PHY_ADR_LPBK_CONTROL_0_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_519_PHY_ADR_LPBK_CONTROL_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_519_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_519_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_519_PHY_ADR_PRBS_PATTERN_START_0_FLDMASK (0x7f0000)
+#define HWIO_DDR_PHY_519_PHY_ADR_PRBS_PATTERN_START_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_519_RESERVED2_FLDMASK (0x800000)
+#define HWIO_DDR_PHY_519_RESERVED2_FLDSHFT (23)
+#define HWIO_DDR_PHY_519_PHY_ADR_PRBS_PATTERN_MASK_0_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_519_PHY_ADR_PRBS_PATTERN_MASK_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_519_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_519_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_520_REGOFF 0x820
+#define HWIO_DDR_PHY_520_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_520_REGOFF)
+#define HWIO_DDR_PHY_520_PHY_ADR_PWR_RDC_DISABLE_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_520_PHY_ADR_PWR_RDC_DISABLE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_520_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_520_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_520_PHY_ADR_TYPE_0_FLDMASK (0x300)
+#define HWIO_DDR_PHY_520_PHY_ADR_TYPE_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_520_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_520_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_520_PHY_ADR_WRADDR_SHIFT_OBS_0_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_520_PHY_ADR_WRADDR_SHIFT_OBS_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_520_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_520_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_520_PHY_ADR_IE_MODE_0_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_520_PHY_ADR_IE_MODE_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_520_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_520_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_521_REGOFF 0x824
+#define HWIO_DDR_PHY_521_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_521_REGOFF)
+#define HWIO_DDR_PHY_521_PHY_ADR_DDL_MODE_0_FLDMASK (0x7fff)
+#define HWIO_DDR_PHY_521_PHY_ADR_DDL_MODE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_521_RESERVED_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_521_RESERVED_FLDSHFT (15)
+#define HWIO_DDR_PHY_521_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_521_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_522_REGOFF 0x828
+#define HWIO_DDR_PHY_522_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_522_REGOFF)
+#define HWIO_DDR_PHY_522_PHY_ADR_DDL_TEST_OBS_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_522_PHY_ADR_DDL_TEST_OBS_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_523_REGOFF 0x82c
+#define HWIO_DDR_PHY_523_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_523_REGOFF)
+#define HWIO_DDR_PHY_523_PHY_ADR_DDL_TEST_MSTR_DLY_OBS_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_523_PHY_ADR_DDL_TEST_MSTR_DLY_OBS_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_524_REGOFF 0x830
+#define HWIO_DDR_PHY_524_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_524_REGOFF)
+#define HWIO_DDR_PHY_524_PHY_ADR_CALVL_START_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_524_PHY_ADR_CALVL_START_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_524_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_524_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_524_PHY_ADR_CALVL_COARSE_DLY_0_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_524_PHY_ADR_CALVL_COARSE_DLY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_524_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_524_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_525_REGOFF 0x834
+#define HWIO_DDR_PHY_525_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_525_REGOFF)
+#define HWIO_DDR_PHY_525_PHY_ADR_CALVL_QTR_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_525_PHY_ADR_CALVL_QTR_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_525_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_525_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_525_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_525_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_526_REGOFF 0x838
+#define HWIO_DDR_PHY_526_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_526_REGOFF)
+#define HWIO_DDR_PHY_526_PHY_ADR_CALVL_SWIZZLE0_0_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_526_PHY_ADR_CALVL_SWIZZLE0_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_526_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_526_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_527_REGOFF 0x83c
+#define HWIO_DDR_PHY_527_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_527_REGOFF)
+#define HWIO_DDR_PHY_527_PHY_ADR_CALVL_SWIZZLE1_0_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_527_PHY_ADR_CALVL_SWIZZLE1_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_527_PHY_ADR_CALVL_RANK_CTRL_0_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_527_PHY_ADR_CALVL_RANK_CTRL_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_527_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_527_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_528_REGOFF 0x840
+#define HWIO_DDR_PHY_528_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_528_REGOFF)
+#define HWIO_DDR_PHY_528_PHY_ADR_CALVL_NUM_PATTERNS_0_FLDMASK (0x3)
+#define HWIO_DDR_PHY_528_PHY_ADR_CALVL_NUM_PATTERNS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_528_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_528_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_528_PHY_ADR_CALVL_CAPTURE_CNT_0_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_528_PHY_ADR_CALVL_CAPTURE_CNT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_528_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_528_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_528_PHY_ADR_CALVL_RESP_WAIT_CNT_0_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_528_PHY_ADR_CALVL_RESP_WAIT_CNT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_528_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_528_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_528_PHY_ADR_CALVL_DEBUG_MODE_0_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_528_PHY_ADR_CALVL_DEBUG_MODE_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_528_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_528_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_529_REGOFF 0x844
+#define HWIO_DDR_PHY_529_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_529_REGOFF)
+#define HWIO_DDR_PHY_529_SC_PHY_ADR_CALVL_DEBUG_CONT_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_529_SC_PHY_ADR_CALVL_DEBUG_CONT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_529_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_529_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_529_SC_PHY_ADR_CALVL_ERROR_CLR_0_FLDMASK (0x100)
+#define HWIO_DDR_PHY_529_SC_PHY_ADR_CALVL_ERROR_CLR_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_529_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_529_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_529_PHY_ADR_CALVL_OBS_SELECT_0_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_529_PHY_ADR_CALVL_OBS_SELECT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_529_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_529_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_529_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_529_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_530_REGOFF 0x848
+#define HWIO_DDR_PHY_530_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_530_REGOFF)
+#define HWIO_DDR_PHY_530_PHY_ADR_CALVL_OBS0_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_530_PHY_ADR_CALVL_OBS0_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_531_REGOFF 0x84c
+#define HWIO_DDR_PHY_531_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_531_REGOFF)
+#define HWIO_DDR_PHY_531_PHY_ADR_CALVL_OBS1_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_531_PHY_ADR_CALVL_OBS1_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_532_REGOFF 0x850
+#define HWIO_DDR_PHY_532_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_532_REGOFF)
+#define HWIO_DDR_PHY_532_PHY_ADR_CALVL_FG_0_0_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_532_PHY_ADR_CALVL_FG_0_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_532_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_532_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_532_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_532_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_533_REGOFF 0x854
+#define HWIO_DDR_PHY_533_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_533_REGOFF)
+#define HWIO_DDR_PHY_533_PHY_ADR_CALVL_BG_0_0_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_533_PHY_ADR_CALVL_BG_0_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_533_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_533_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_533_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_533_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_534_REGOFF 0x858
+#define HWIO_DDR_PHY_534_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_534_REGOFF)
+#define HWIO_DDR_PHY_534_PHY_ADR_CALVL_FG_1_0_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_534_PHY_ADR_CALVL_FG_1_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_534_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_534_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_534_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_534_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_535_REGOFF 0x85c
+#define HWIO_DDR_PHY_535_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_535_REGOFF)
+#define HWIO_DDR_PHY_535_PHY_ADR_CALVL_BG_1_0_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_535_PHY_ADR_CALVL_BG_1_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_535_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_535_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_535_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_535_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_536_REGOFF 0x860
+#define HWIO_DDR_PHY_536_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_536_REGOFF)
+#define HWIO_DDR_PHY_536_PHY_ADR_CALVL_FG_2_0_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_536_PHY_ADR_CALVL_FG_2_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_536_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_536_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_536_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_536_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_537_REGOFF 0x864
+#define HWIO_DDR_PHY_537_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_537_REGOFF)
+#define HWIO_DDR_PHY_537_PHY_ADR_CALVL_BG_2_0_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_537_PHY_ADR_CALVL_BG_2_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_537_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_537_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_537_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_537_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_538_REGOFF 0x868
+#define HWIO_DDR_PHY_538_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_538_REGOFF)
+#define HWIO_DDR_PHY_538_PHY_ADR_CALVL_FG_3_0_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_538_PHY_ADR_CALVL_FG_3_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_538_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_538_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_538_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_538_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_539_REGOFF 0x86c
+#define HWIO_DDR_PHY_539_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_539_REGOFF)
+#define HWIO_DDR_PHY_539_PHY_ADR_CALVL_BG_3_0_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_539_PHY_ADR_CALVL_BG_3_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_539_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_539_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_539_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_539_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_540_REGOFF 0x870
+#define HWIO_DDR_PHY_540_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_540_REGOFF)
+#define HWIO_DDR_PHY_540_PHY_ADR_ADDR_SEL_0_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_540_PHY_ADR_ADDR_SEL_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_540_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_540_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_541_REGOFF 0x874
+#define HWIO_DDR_PHY_541_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_541_REGOFF)
+#define HWIO_DDR_PHY_541_PHY_ADR_LP4_BOOT_SLV_DELAY_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_541_PHY_ADR_LP4_BOOT_SLV_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_541_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_541_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_541_PHY_ADR_BIT_MASK_0_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_541_PHY_ADR_BIT_MASK_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_541_RESERVED1_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_541_RESERVED1_FLDSHFT (22)
+#define HWIO_DDR_PHY_541_PHY_ADR_SEG_MASK_0_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_541_PHY_ADR_SEG_MASK_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_541_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_541_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_542_REGOFF 0x878
+#define HWIO_DDR_PHY_542_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_542_REGOFF)
+#define HWIO_DDR_PHY_542_PHY_ADR_CALVL_TRAIN_MASK_0_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_542_PHY_ADR_CALVL_TRAIN_MASK_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_542_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_542_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_542_PHY_ADR_SW_TXIO_CTRL_0_FLDMASK (0x3f00)
+#define HWIO_DDR_PHY_542_PHY_ADR_SW_TXIO_CTRL_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_542_RESERVED1_FLDMASK (0xc000)
+#define HWIO_DDR_PHY_542_RESERVED1_FLDSHFT (14)
+#define HWIO_DDR_PHY_542_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_542_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_543_REGOFF 0x87c
+#define HWIO_DDR_PHY_543_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_543_REGOFF)
+#define HWIO_DDR_PHY_543_PHY_ADR_TSEL_SELECT_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_543_PHY_ADR_TSEL_SELECT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_543_PHY_ADR0_CLK_WR_SLAVE_DELAY_0_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_543_PHY_ADR0_CLK_WR_SLAVE_DELAY_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_543_RESERVED_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_543_RESERVED_FLDSHFT (19)
+#define HWIO_DDR_PHY_543_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_543_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_544_REGOFF 0x880
+#define HWIO_DDR_PHY_544_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_544_REGOFF)
+#define HWIO_DDR_PHY_544_PHY_ADR1_CLK_WR_SLAVE_DELAY_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_544_PHY_ADR1_CLK_WR_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_544_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_544_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_544_PHY_ADR2_CLK_WR_SLAVE_DELAY_0_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_544_PHY_ADR2_CLK_WR_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_544_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_544_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_545_REGOFF 0x884
+#define HWIO_DDR_PHY_545_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_545_REGOFF)
+#define HWIO_DDR_PHY_545_PHY_ADR3_CLK_WR_SLAVE_DELAY_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_545_PHY_ADR3_CLK_WR_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_545_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_545_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_545_PHY_ADR4_CLK_WR_SLAVE_DELAY_0_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_545_PHY_ADR4_CLK_WR_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_545_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_545_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_546_REGOFF 0x888
+#define HWIO_DDR_PHY_546_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_546_REGOFF)
+#define HWIO_DDR_PHY_546_PHY_ADR5_CLK_WR_SLAVE_DELAY_0_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_546_PHY_ADR5_CLK_WR_SLAVE_DELAY_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_546_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_546_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_546_PHY_ADR_SW_MASTER_MODE_0_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_546_PHY_ADR_SW_MASTER_MODE_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_546_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_546_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PHY_546_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_546_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_547_REGOFF 0x88c
+#define HWIO_DDR_PHY_547_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_547_REGOFF)
+#define HWIO_DDR_PHY_547_PHY_ADR_MASTER_DELAY_START_0_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_547_PHY_ADR_MASTER_DELAY_START_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_547_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_547_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_547_PHY_ADR_MASTER_DELAY_STEP_0_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_547_PHY_ADR_MASTER_DELAY_STEP_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_547_RESERVED1_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_547_RESERVED1_FLDSHFT (22)
+#define HWIO_DDR_PHY_547_PHY_ADR_MASTER_DELAY_WAIT_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_547_PHY_ADR_MASTER_DELAY_WAIT_0_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_548_REGOFF 0x890
+#define HWIO_DDR_PHY_548_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_548_REGOFF)
+#define HWIO_DDR_PHY_548_PHY_ADR_MASTER_DELAY_HALF_MEASURE_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_548_PHY_ADR_MASTER_DELAY_HALF_MEASURE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_548_PHY_ADR_CALVL_DLY_STEP_0_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_548_PHY_ADR_CALVL_DLY_STEP_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_548_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_548_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_548_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_548_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_549_REGOFF 0x894
+#define HWIO_DDR_PHY_549_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_549_REGOFF)
+#define HWIO_DDR_PHY_549_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_549_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_550_REGOFF 0x898
+#define HWIO_DDR_PHY_550_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_550_REGOFF)
+#define HWIO_DDR_PHY_550_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_550_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_551_REGOFF 0x89c
+#define HWIO_DDR_PHY_551_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_551_REGOFF)
+#define HWIO_DDR_PHY_551_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_551_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_552_REGOFF 0x8a0
+#define HWIO_DDR_PHY_552_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_552_REGOFF)
+#define HWIO_DDR_PHY_552_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_552_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_553_REGOFF 0x8a4
+#define HWIO_DDR_PHY_553_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_553_REGOFF)
+#define HWIO_DDR_PHY_553_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_553_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_554_REGOFF 0x8a8
+#define HWIO_DDR_PHY_554_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_554_REGOFF)
+#define HWIO_DDR_PHY_554_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_554_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_555_REGOFF 0x8ac
+#define HWIO_DDR_PHY_555_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_555_REGOFF)
+#define HWIO_DDR_PHY_555_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_555_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_556_REGOFF 0x8b0
+#define HWIO_DDR_PHY_556_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_556_REGOFF)
+#define HWIO_DDR_PHY_556_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_556_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_557_REGOFF 0x8b4
+#define HWIO_DDR_PHY_557_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_557_REGOFF)
+#define HWIO_DDR_PHY_557_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_557_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_558_REGOFF 0x8b8
+#define HWIO_DDR_PHY_558_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_558_REGOFF)
+#define HWIO_DDR_PHY_558_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_558_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_559_REGOFF 0x8bc
+#define HWIO_DDR_PHY_559_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_559_REGOFF)
+#define HWIO_DDR_PHY_559_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_559_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_560_REGOFF 0x8c0
+#define HWIO_DDR_PHY_560_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_560_REGOFF)
+#define HWIO_DDR_PHY_560_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_560_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_561_REGOFF 0x8c4
+#define HWIO_DDR_PHY_561_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_561_REGOFF)
+#define HWIO_DDR_PHY_561_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_561_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_562_REGOFF 0x8c8
+#define HWIO_DDR_PHY_562_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_562_REGOFF)
+#define HWIO_DDR_PHY_562_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_562_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_563_REGOFF 0x8cc
+#define HWIO_DDR_PHY_563_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_563_REGOFF)
+#define HWIO_DDR_PHY_563_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_563_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_564_REGOFF 0x8d0
+#define HWIO_DDR_PHY_564_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_564_REGOFF)
+#define HWIO_DDR_PHY_564_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_564_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_565_REGOFF 0x8d4
+#define HWIO_DDR_PHY_565_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_565_REGOFF)
+#define HWIO_DDR_PHY_565_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_565_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_566_REGOFF 0x8d8
+#define HWIO_DDR_PHY_566_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_566_REGOFF)
+#define HWIO_DDR_PHY_566_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_566_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_567_REGOFF 0x8dc
+#define HWIO_DDR_PHY_567_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_567_REGOFF)
+#define HWIO_DDR_PHY_567_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_567_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_568_REGOFF 0x8e0
+#define HWIO_DDR_PHY_568_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_568_REGOFF)
+#define HWIO_DDR_PHY_568_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_568_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_569_REGOFF 0x8e4
+#define HWIO_DDR_PHY_569_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_569_REGOFF)
+#define HWIO_DDR_PHY_569_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_569_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_570_REGOFF 0x8e8
+#define HWIO_DDR_PHY_570_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_570_REGOFF)
+#define HWIO_DDR_PHY_570_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_570_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_571_REGOFF 0x8ec
+#define HWIO_DDR_PHY_571_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_571_REGOFF)
+#define HWIO_DDR_PHY_571_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_571_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_572_REGOFF 0x8f0
+#define HWIO_DDR_PHY_572_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_572_REGOFF)
+#define HWIO_DDR_PHY_572_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_572_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_573_REGOFF 0x8f4
+#define HWIO_DDR_PHY_573_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_573_REGOFF)
+#define HWIO_DDR_PHY_573_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_573_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_574_REGOFF 0x8f8
+#define HWIO_DDR_PHY_574_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_574_REGOFF)
+#define HWIO_DDR_PHY_574_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_574_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_575_REGOFF 0x8fc
+#define HWIO_DDR_PHY_575_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_575_REGOFF)
+#define HWIO_DDR_PHY_575_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_575_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_576_REGOFF 0x900
+#define HWIO_DDR_PHY_576_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_576_REGOFF)
+#define HWIO_DDR_PHY_576_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_576_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_577_REGOFF 0x904
+#define HWIO_DDR_PHY_577_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_577_REGOFF)
+#define HWIO_DDR_PHY_577_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_577_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_578_REGOFF 0x908
+#define HWIO_DDR_PHY_578_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_578_REGOFF)
+#define HWIO_DDR_PHY_578_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_578_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_579_REGOFF 0x90c
+#define HWIO_DDR_PHY_579_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_579_REGOFF)
+#define HWIO_DDR_PHY_579_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_579_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_580_REGOFF 0x910
+#define HWIO_DDR_PHY_580_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_580_REGOFF)
+#define HWIO_DDR_PHY_580_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_580_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_581_REGOFF 0x914
+#define HWIO_DDR_PHY_581_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_581_REGOFF)
+#define HWIO_DDR_PHY_581_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_581_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_582_REGOFF 0x918
+#define HWIO_DDR_PHY_582_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_582_REGOFF)
+#define HWIO_DDR_PHY_582_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_582_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_583_REGOFF 0x91c
+#define HWIO_DDR_PHY_583_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_583_REGOFF)
+#define HWIO_DDR_PHY_583_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_583_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_584_REGOFF 0x920
+#define HWIO_DDR_PHY_584_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_584_REGOFF)
+#define HWIO_DDR_PHY_584_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_584_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_585_REGOFF 0x924
+#define HWIO_DDR_PHY_585_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_585_REGOFF)
+#define HWIO_DDR_PHY_585_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_585_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_586_REGOFF 0x928
+#define HWIO_DDR_PHY_586_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_586_REGOFF)
+#define HWIO_DDR_PHY_586_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_586_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_587_REGOFF 0x92c
+#define HWIO_DDR_PHY_587_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_587_REGOFF)
+#define HWIO_DDR_PHY_587_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_587_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_588_REGOFF 0x930
+#define HWIO_DDR_PHY_588_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_588_REGOFF)
+#define HWIO_DDR_PHY_588_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_588_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_589_REGOFF 0x934
+#define HWIO_DDR_PHY_589_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_589_REGOFF)
+#define HWIO_DDR_PHY_589_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_589_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_590_REGOFF 0x938
+#define HWIO_DDR_PHY_590_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_590_REGOFF)
+#define HWIO_DDR_PHY_590_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_590_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_591_REGOFF 0x93c
+#define HWIO_DDR_PHY_591_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_591_REGOFF)
+#define HWIO_DDR_PHY_591_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_591_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_592_REGOFF 0x940
+#define HWIO_DDR_PHY_592_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_592_REGOFF)
+#define HWIO_DDR_PHY_592_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_592_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_593_REGOFF 0x944
+#define HWIO_DDR_PHY_593_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_593_REGOFF)
+#define HWIO_DDR_PHY_593_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_593_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_594_REGOFF 0x948
+#define HWIO_DDR_PHY_594_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_594_REGOFF)
+#define HWIO_DDR_PHY_594_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_594_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_595_REGOFF 0x94c
+#define HWIO_DDR_PHY_595_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_595_REGOFF)
+#define HWIO_DDR_PHY_595_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_595_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_596_REGOFF 0x950
+#define HWIO_DDR_PHY_596_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_596_REGOFF)
+#define HWIO_DDR_PHY_596_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_596_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_597_REGOFF 0x954
+#define HWIO_DDR_PHY_597_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_597_REGOFF)
+#define HWIO_DDR_PHY_597_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_597_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_598_REGOFF 0x958
+#define HWIO_DDR_PHY_598_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_598_REGOFF)
+#define HWIO_DDR_PHY_598_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_598_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_599_REGOFF 0x95c
+#define HWIO_DDR_PHY_599_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_599_REGOFF)
+#define HWIO_DDR_PHY_599_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_599_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_600_REGOFF 0x960
+#define HWIO_DDR_PHY_600_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_600_REGOFF)
+#define HWIO_DDR_PHY_600_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_600_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_601_REGOFF 0x964
+#define HWIO_DDR_PHY_601_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_601_REGOFF)
+#define HWIO_DDR_PHY_601_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_601_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_602_REGOFF 0x968
+#define HWIO_DDR_PHY_602_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_602_REGOFF)
+#define HWIO_DDR_PHY_602_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_602_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_603_REGOFF 0x96c
+#define HWIO_DDR_PHY_603_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_603_REGOFF)
+#define HWIO_DDR_PHY_603_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_603_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_604_REGOFF 0x970
+#define HWIO_DDR_PHY_604_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_604_REGOFF)
+#define HWIO_DDR_PHY_604_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_604_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_605_REGOFF 0x974
+#define HWIO_DDR_PHY_605_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_605_REGOFF)
+#define HWIO_DDR_PHY_605_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_605_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_606_REGOFF 0x978
+#define HWIO_DDR_PHY_606_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_606_REGOFF)
+#define HWIO_DDR_PHY_606_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_606_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_607_REGOFF 0x97c
+#define HWIO_DDR_PHY_607_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_607_REGOFF)
+#define HWIO_DDR_PHY_607_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_607_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_608_REGOFF 0x980
+#define HWIO_DDR_PHY_608_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_608_REGOFF)
+#define HWIO_DDR_PHY_608_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_608_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_609_REGOFF 0x984
+#define HWIO_DDR_PHY_609_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_609_REGOFF)
+#define HWIO_DDR_PHY_609_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_609_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_610_REGOFF 0x988
+#define HWIO_DDR_PHY_610_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_610_REGOFF)
+#define HWIO_DDR_PHY_610_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_610_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_611_REGOFF 0x98c
+#define HWIO_DDR_PHY_611_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_611_REGOFF)
+#define HWIO_DDR_PHY_611_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_611_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_612_REGOFF 0x990
+#define HWIO_DDR_PHY_612_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_612_REGOFF)
+#define HWIO_DDR_PHY_612_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_612_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_613_REGOFF 0x994
+#define HWIO_DDR_PHY_613_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_613_REGOFF)
+#define HWIO_DDR_PHY_613_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_613_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_614_REGOFF 0x998
+#define HWIO_DDR_PHY_614_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_614_REGOFF)
+#define HWIO_DDR_PHY_614_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_614_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_615_REGOFF 0x99c
+#define HWIO_DDR_PHY_615_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_615_REGOFF)
+#define HWIO_DDR_PHY_615_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_615_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_616_REGOFF 0x9a0
+#define HWIO_DDR_PHY_616_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_616_REGOFF)
+#define HWIO_DDR_PHY_616_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_616_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_617_REGOFF 0x9a4
+#define HWIO_DDR_PHY_617_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_617_REGOFF)
+#define HWIO_DDR_PHY_617_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_617_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_618_REGOFF 0x9a8
+#define HWIO_DDR_PHY_618_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_618_REGOFF)
+#define HWIO_DDR_PHY_618_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_618_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_619_REGOFF 0x9ac
+#define HWIO_DDR_PHY_619_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_619_REGOFF)
+#define HWIO_DDR_PHY_619_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_619_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_620_REGOFF 0x9b0
+#define HWIO_DDR_PHY_620_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_620_REGOFF)
+#define HWIO_DDR_PHY_620_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_620_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_621_REGOFF 0x9b4
+#define HWIO_DDR_PHY_621_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_621_REGOFF)
+#define HWIO_DDR_PHY_621_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_621_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_622_REGOFF 0x9b8
+#define HWIO_DDR_PHY_622_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_622_REGOFF)
+#define HWIO_DDR_PHY_622_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_622_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_623_REGOFF 0x9bc
+#define HWIO_DDR_PHY_623_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_623_REGOFF)
+#define HWIO_DDR_PHY_623_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_623_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_624_REGOFF 0x9c0
+#define HWIO_DDR_PHY_624_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_624_REGOFF)
+#define HWIO_DDR_PHY_624_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_624_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_625_REGOFF 0x9c4
+#define HWIO_DDR_PHY_625_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_625_REGOFF)
+#define HWIO_DDR_PHY_625_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_625_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_626_REGOFF 0x9c8
+#define HWIO_DDR_PHY_626_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_626_REGOFF)
+#define HWIO_DDR_PHY_626_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_626_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_627_REGOFF 0x9cc
+#define HWIO_DDR_PHY_627_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_627_REGOFF)
+#define HWIO_DDR_PHY_627_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_627_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_628_REGOFF 0x9d0
+#define HWIO_DDR_PHY_628_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_628_REGOFF)
+#define HWIO_DDR_PHY_628_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_628_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_629_REGOFF 0x9d4
+#define HWIO_DDR_PHY_629_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_629_REGOFF)
+#define HWIO_DDR_PHY_629_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_629_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_630_REGOFF 0x9d8
+#define HWIO_DDR_PHY_630_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_630_REGOFF)
+#define HWIO_DDR_PHY_630_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_630_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_631_REGOFF 0x9dc
+#define HWIO_DDR_PHY_631_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_631_REGOFF)
+#define HWIO_DDR_PHY_631_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_631_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_632_REGOFF 0x9e0
+#define HWIO_DDR_PHY_632_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_632_REGOFF)
+#define HWIO_DDR_PHY_632_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_632_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_633_REGOFF 0x9e4
+#define HWIO_DDR_PHY_633_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_633_REGOFF)
+#define HWIO_DDR_PHY_633_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_633_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_634_REGOFF 0x9e8
+#define HWIO_DDR_PHY_634_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_634_REGOFF)
+#define HWIO_DDR_PHY_634_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_634_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_635_REGOFF 0x9ec
+#define HWIO_DDR_PHY_635_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_635_REGOFF)
+#define HWIO_DDR_PHY_635_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_635_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_636_REGOFF 0x9f0
+#define HWIO_DDR_PHY_636_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_636_REGOFF)
+#define HWIO_DDR_PHY_636_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_636_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_637_REGOFF 0x9f4
+#define HWIO_DDR_PHY_637_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_637_REGOFF)
+#define HWIO_DDR_PHY_637_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_637_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_638_REGOFF 0x9f8
+#define HWIO_DDR_PHY_638_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_638_REGOFF)
+#define HWIO_DDR_PHY_638_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_638_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_639_REGOFF 0x9fc
+#define HWIO_DDR_PHY_639_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_639_REGOFF)
+#define HWIO_DDR_PHY_639_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_639_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_640_REGOFF 0xa00
+#define HWIO_DDR_PHY_640_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_640_REGOFF)
+#define HWIO_DDR_PHY_640_PHY_ADR0_SW_WRADDR_SHIFT_1_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_640_PHY_ADR0_SW_WRADDR_SHIFT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_640_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_640_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_640_PHY_ADR1_SW_WRADDR_SHIFT_1_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_640_PHY_ADR1_SW_WRADDR_SHIFT_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_640_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_640_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_640_PHY_ADR2_SW_WRADDR_SHIFT_1_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_640_PHY_ADR2_SW_WRADDR_SHIFT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_640_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_640_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_640_PHY_ADR3_SW_WRADDR_SHIFT_1_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_640_PHY_ADR3_SW_WRADDR_SHIFT_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_640_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_640_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_641_REGOFF 0xa04
+#define HWIO_DDR_PHY_641_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_641_REGOFF)
+#define HWIO_DDR_PHY_641_PHY_ADR4_SW_WRADDR_SHIFT_1_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_641_PHY_ADR4_SW_WRADDR_SHIFT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_641_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_641_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_641_PHY_ADR5_SW_WRADDR_SHIFT_1_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_641_PHY_ADR5_SW_WRADDR_SHIFT_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_641_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_641_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_641_PHY_ADR_CLK_WR_BYPASS_SLAVE_DELAY_1_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_641_PHY_ADR_CLK_WR_BYPASS_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_641_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_641_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_642_REGOFF 0xa08
+#define HWIO_DDR_PHY_642_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_642_REGOFF)
+#define HWIO_DDR_PHY_642_PHY_ADR_CLK_BYPASS_OVERRIDE_1_FLDMASK (0x1)
+#define HWIO_DDR_PHY_642_PHY_ADR_CLK_BYPASS_OVERRIDE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_642_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_642_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_642_SC_PHY_ADR_MANUAL_CLEAR_1_FLDMASK (0x700)
+#define HWIO_DDR_PHY_642_SC_PHY_ADR_MANUAL_CLEAR_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_642_RESERVED1_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_642_RESERVED1_FLDSHFT (11)
+#define HWIO_DDR_PHY_642_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_642_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_643_REGOFF 0xa0c
+#define HWIO_DDR_PHY_643_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_643_REGOFF)
+#define HWIO_DDR_PHY_643_PHY_ADR_LPBK_RESULT_OBS_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_643_PHY_ADR_LPBK_RESULT_OBS_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_644_REGOFF 0xa10
+#define HWIO_DDR_PHY_644_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_644_REGOFF)
+#define HWIO_DDR_PHY_644_PHY_ADR_LPBK_ERROR_COUNT_OBS_1_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_644_PHY_ADR_LPBK_ERROR_COUNT_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_644_PHY_ADR_MASTER_DLY_LOCK_OBS_SELECT_1_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_644_PHY_ADR_MASTER_DLY_LOCK_OBS_SELECT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_644_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_644_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_644_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_644_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_645_REGOFF 0xa14
+#define HWIO_DDR_PHY_645_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_645_REGOFF)
+#define HWIO_DDR_PHY_645_PHY_ADR_MASTER_DLY_LOCK_OBS_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_645_PHY_ADR_MASTER_DLY_LOCK_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_645_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_645_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_645_PHY_ADR_BASE_SLV_DLY_ENC_OBS_1_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_645_PHY_ADR_BASE_SLV_DLY_ENC_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_645_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_645_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_646_REGOFF 0xa18
+#define HWIO_DDR_PHY_646_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_646_REGOFF)
+#define HWIO_DDR_PHY_646_PHY_ADR_ADDER_SLV_DLY_ENC_OBS_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_646_PHY_ADR_ADDER_SLV_DLY_ENC_OBS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_646_PHY_ADR_SLAVE_LOOP_CNT_UPDATE_1_FLDMASK (0x700)
+#define HWIO_DDR_PHY_646_PHY_ADR_SLAVE_LOOP_CNT_UPDATE_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_646_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_646_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_646_PHY_ADR_SLV_DLY_ENC_OBS_SELECT_1_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_646_PHY_ADR_SLV_DLY_ENC_OBS_SELECT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_646_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_646_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_646_SC_PHY_ADR_SNAP_OBS_REGS_1_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_646_SC_PHY_ADR_SNAP_OBS_REGS_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_646_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_646_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_647_REGOFF 0xa1c
+#define HWIO_DDR_PHY_647_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_647_REGOFF)
+#define HWIO_DDR_PHY_647_PHY_ADR_TSEL_ENABLE_1_FLDMASK (0x1)
+#define HWIO_DDR_PHY_647_PHY_ADR_TSEL_ENABLE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_647_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_647_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_647_PHY_ADR_LPBK_CONTROL_1_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_647_PHY_ADR_LPBK_CONTROL_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_647_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_647_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_647_PHY_ADR_PRBS_PATTERN_START_1_FLDMASK (0x7f0000)
+#define HWIO_DDR_PHY_647_PHY_ADR_PRBS_PATTERN_START_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_647_RESERVED2_FLDMASK (0x800000)
+#define HWIO_DDR_PHY_647_RESERVED2_FLDSHFT (23)
+#define HWIO_DDR_PHY_647_PHY_ADR_PRBS_PATTERN_MASK_1_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_647_PHY_ADR_PRBS_PATTERN_MASK_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_647_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_647_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_648_REGOFF 0xa20
+#define HWIO_DDR_PHY_648_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_648_REGOFF)
+#define HWIO_DDR_PHY_648_PHY_ADR_PWR_RDC_DISABLE_1_FLDMASK (0x1)
+#define HWIO_DDR_PHY_648_PHY_ADR_PWR_RDC_DISABLE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_648_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_648_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_648_PHY_ADR_TYPE_1_FLDMASK (0x300)
+#define HWIO_DDR_PHY_648_PHY_ADR_TYPE_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_648_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_648_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_648_PHY_ADR_WRADDR_SHIFT_OBS_1_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_648_PHY_ADR_WRADDR_SHIFT_OBS_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_648_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_648_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_648_PHY_ADR_IE_MODE_1_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_648_PHY_ADR_IE_MODE_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_648_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_648_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_649_REGOFF 0xa24
+#define HWIO_DDR_PHY_649_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_649_REGOFF)
+#define HWIO_DDR_PHY_649_PHY_ADR_DDL_MODE_1_FLDMASK (0x7fff)
+#define HWIO_DDR_PHY_649_PHY_ADR_DDL_MODE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_649_RESERVED_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_649_RESERVED_FLDSHFT (15)
+#define HWIO_DDR_PHY_649_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_649_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_650_REGOFF 0xa28
+#define HWIO_DDR_PHY_650_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_650_REGOFF)
+#define HWIO_DDR_PHY_650_PHY_ADR_DDL_TEST_OBS_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_650_PHY_ADR_DDL_TEST_OBS_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_651_REGOFF 0xa2c
+#define HWIO_DDR_PHY_651_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_651_REGOFF)
+#define HWIO_DDR_PHY_651_PHY_ADR_DDL_TEST_MSTR_DLY_OBS_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_651_PHY_ADR_DDL_TEST_MSTR_DLY_OBS_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_652_REGOFF 0xa30
+#define HWIO_DDR_PHY_652_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_652_REGOFF)
+#define HWIO_DDR_PHY_652_PHY_ADR_CALVL_START_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_652_PHY_ADR_CALVL_START_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_652_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_652_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_652_PHY_ADR_CALVL_COARSE_DLY_1_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_652_PHY_ADR_CALVL_COARSE_DLY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_652_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_652_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_653_REGOFF 0xa34
+#define HWIO_DDR_PHY_653_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_653_REGOFF)
+#define HWIO_DDR_PHY_653_PHY_ADR_CALVL_QTR_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_653_PHY_ADR_CALVL_QTR_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_653_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_653_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_653_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_653_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_654_REGOFF 0xa38
+#define HWIO_DDR_PHY_654_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_654_REGOFF)
+#define HWIO_DDR_PHY_654_PHY_ADR_CALVL_SWIZZLE0_1_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_654_PHY_ADR_CALVL_SWIZZLE0_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_654_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_654_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_655_REGOFF 0xa3c
+#define HWIO_DDR_PHY_655_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_655_REGOFF)
+#define HWIO_DDR_PHY_655_PHY_ADR_CALVL_SWIZZLE1_1_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_655_PHY_ADR_CALVL_SWIZZLE1_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_655_PHY_ADR_CALVL_RANK_CTRL_1_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_655_PHY_ADR_CALVL_RANK_CTRL_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_655_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_655_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_656_REGOFF 0xa40
+#define HWIO_DDR_PHY_656_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_656_REGOFF)
+#define HWIO_DDR_PHY_656_PHY_ADR_CALVL_NUM_PATTERNS_1_FLDMASK (0x3)
+#define HWIO_DDR_PHY_656_PHY_ADR_CALVL_NUM_PATTERNS_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_656_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_656_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_656_PHY_ADR_CALVL_CAPTURE_CNT_1_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_656_PHY_ADR_CALVL_CAPTURE_CNT_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_656_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_656_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_656_PHY_ADR_CALVL_RESP_WAIT_CNT_1_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_656_PHY_ADR_CALVL_RESP_WAIT_CNT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_656_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_656_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_656_PHY_ADR_CALVL_DEBUG_MODE_1_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_656_PHY_ADR_CALVL_DEBUG_MODE_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_656_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_656_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_657_REGOFF 0xa44
+#define HWIO_DDR_PHY_657_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_657_REGOFF)
+#define HWIO_DDR_PHY_657_SC_PHY_ADR_CALVL_DEBUG_CONT_1_FLDMASK (0x1)
+#define HWIO_DDR_PHY_657_SC_PHY_ADR_CALVL_DEBUG_CONT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_657_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_657_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_657_SC_PHY_ADR_CALVL_ERROR_CLR_1_FLDMASK (0x100)
+#define HWIO_DDR_PHY_657_SC_PHY_ADR_CALVL_ERROR_CLR_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_657_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_657_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_657_PHY_ADR_CALVL_OBS_SELECT_1_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_657_PHY_ADR_CALVL_OBS_SELECT_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_657_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_657_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_657_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_657_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_658_REGOFF 0xa48
+#define HWIO_DDR_PHY_658_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_658_REGOFF)
+#define HWIO_DDR_PHY_658_PHY_ADR_CALVL_OBS0_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_658_PHY_ADR_CALVL_OBS0_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_659_REGOFF 0xa4c
+#define HWIO_DDR_PHY_659_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_659_REGOFF)
+#define HWIO_DDR_PHY_659_PHY_ADR_CALVL_OBS1_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_659_PHY_ADR_CALVL_OBS1_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_660_REGOFF 0xa50
+#define HWIO_DDR_PHY_660_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_660_REGOFF)
+#define HWIO_DDR_PHY_660_PHY_ADR_CALVL_FG_0_1_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_660_PHY_ADR_CALVL_FG_0_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_660_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_660_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_660_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_660_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_661_REGOFF 0xa54
+#define HWIO_DDR_PHY_661_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_661_REGOFF)
+#define HWIO_DDR_PHY_661_PHY_ADR_CALVL_BG_0_1_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_661_PHY_ADR_CALVL_BG_0_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_661_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_661_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_661_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_661_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_662_REGOFF 0xa58
+#define HWIO_DDR_PHY_662_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_662_REGOFF)
+#define HWIO_DDR_PHY_662_PHY_ADR_CALVL_FG_1_1_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_662_PHY_ADR_CALVL_FG_1_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_662_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_662_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_662_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_662_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_663_REGOFF 0xa5c
+#define HWIO_DDR_PHY_663_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_663_REGOFF)
+#define HWIO_DDR_PHY_663_PHY_ADR_CALVL_BG_1_1_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_663_PHY_ADR_CALVL_BG_1_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_663_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_663_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_663_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_663_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_664_REGOFF 0xa60
+#define HWIO_DDR_PHY_664_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_664_REGOFF)
+#define HWIO_DDR_PHY_664_PHY_ADR_CALVL_FG_2_1_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_664_PHY_ADR_CALVL_FG_2_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_664_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_664_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_664_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_664_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_665_REGOFF 0xa64
+#define HWIO_DDR_PHY_665_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_665_REGOFF)
+#define HWIO_DDR_PHY_665_PHY_ADR_CALVL_BG_2_1_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_665_PHY_ADR_CALVL_BG_2_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_665_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_665_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_665_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_665_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_666_REGOFF 0xa68
+#define HWIO_DDR_PHY_666_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_666_REGOFF)
+#define HWIO_DDR_PHY_666_PHY_ADR_CALVL_FG_3_1_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_666_PHY_ADR_CALVL_FG_3_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_666_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_666_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_666_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_666_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_667_REGOFF 0xa6c
+#define HWIO_DDR_PHY_667_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_667_REGOFF)
+#define HWIO_DDR_PHY_667_PHY_ADR_CALVL_BG_3_1_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_667_PHY_ADR_CALVL_BG_3_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_667_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_667_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_667_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_667_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_668_REGOFF 0xa70
+#define HWIO_DDR_PHY_668_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_668_REGOFF)
+#define HWIO_DDR_PHY_668_PHY_ADR_ADDR_SEL_1_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_668_PHY_ADR_ADDR_SEL_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_668_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_668_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_669_REGOFF 0xa74
+#define HWIO_DDR_PHY_669_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_669_REGOFF)
+#define HWIO_DDR_PHY_669_PHY_ADR_LP4_BOOT_SLV_DELAY_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_669_PHY_ADR_LP4_BOOT_SLV_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_669_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_669_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_669_PHY_ADR_BIT_MASK_1_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_669_PHY_ADR_BIT_MASK_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_669_RESERVED1_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_669_RESERVED1_FLDSHFT (22)
+#define HWIO_DDR_PHY_669_PHY_ADR_SEG_MASK_1_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_669_PHY_ADR_SEG_MASK_1_FLDSHFT (24)
+#define HWIO_DDR_PHY_669_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_669_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_670_REGOFF 0xa78
+#define HWIO_DDR_PHY_670_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_670_REGOFF)
+#define HWIO_DDR_PHY_670_PHY_ADR_CALVL_TRAIN_MASK_1_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_670_PHY_ADR_CALVL_TRAIN_MASK_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_670_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_670_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_670_PHY_ADR_SW_TXIO_CTRL_1_FLDMASK (0x3f00)
+#define HWIO_DDR_PHY_670_PHY_ADR_SW_TXIO_CTRL_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_670_RESERVED1_FLDMASK (0xc000)
+#define HWIO_DDR_PHY_670_RESERVED1_FLDSHFT (14)
+#define HWIO_DDR_PHY_670_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_670_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_671_REGOFF 0xa7c
+#define HWIO_DDR_PHY_671_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_671_REGOFF)
+#define HWIO_DDR_PHY_671_PHY_ADR_TSEL_SELECT_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_671_PHY_ADR_TSEL_SELECT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_671_PHY_ADR0_CLK_WR_SLAVE_DELAY_1_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_671_PHY_ADR0_CLK_WR_SLAVE_DELAY_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_671_RESERVED_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_671_RESERVED_FLDSHFT (19)
+#define HWIO_DDR_PHY_671_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_671_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_672_REGOFF 0xa80
+#define HWIO_DDR_PHY_672_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_672_REGOFF)
+#define HWIO_DDR_PHY_672_PHY_ADR1_CLK_WR_SLAVE_DELAY_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_672_PHY_ADR1_CLK_WR_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_672_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_672_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_672_PHY_ADR2_CLK_WR_SLAVE_DELAY_1_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_672_PHY_ADR2_CLK_WR_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_672_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_672_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_673_REGOFF 0xa84
+#define HWIO_DDR_PHY_673_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_673_REGOFF)
+#define HWIO_DDR_PHY_673_PHY_ADR3_CLK_WR_SLAVE_DELAY_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_673_PHY_ADR3_CLK_WR_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_673_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_673_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_673_PHY_ADR4_CLK_WR_SLAVE_DELAY_1_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_673_PHY_ADR4_CLK_WR_SLAVE_DELAY_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_673_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_673_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_674_REGOFF 0xa88
+#define HWIO_DDR_PHY_674_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_674_REGOFF)
+#define HWIO_DDR_PHY_674_PHY_ADR5_CLK_WR_SLAVE_DELAY_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_674_PHY_ADR5_CLK_WR_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_674_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_674_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_674_PHY_ADR_SW_MASTER_MODE_1_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_674_PHY_ADR_SW_MASTER_MODE_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_674_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_674_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PHY_674_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_674_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_675_REGOFF 0xa8c
+#define HWIO_DDR_PHY_675_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_675_REGOFF)
+#define HWIO_DDR_PHY_675_PHY_ADR_MASTER_DELAY_START_1_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_675_PHY_ADR_MASTER_DELAY_START_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_675_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_675_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_675_PHY_ADR_MASTER_DELAY_STEP_1_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_675_PHY_ADR_MASTER_DELAY_STEP_1_FLDSHFT (16)
+#define HWIO_DDR_PHY_675_RESERVED1_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_675_RESERVED1_FLDSHFT (22)
+#define HWIO_DDR_PHY_675_PHY_ADR_MASTER_DELAY_WAIT_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_675_PHY_ADR_MASTER_DELAY_WAIT_1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_676_REGOFF 0xa90
+#define HWIO_DDR_PHY_676_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_676_REGOFF)
+#define HWIO_DDR_PHY_676_PHY_ADR_MASTER_DELAY_HALF_MEASURE_1_FLDMASK (0xff)
+#define HWIO_DDR_PHY_676_PHY_ADR_MASTER_DELAY_HALF_MEASURE_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_676_PHY_ADR_CALVL_DLY_STEP_1_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_676_PHY_ADR_CALVL_DLY_STEP_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_676_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_676_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_676_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_676_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_677_REGOFF 0xa94
+#define HWIO_DDR_PHY_677_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_677_REGOFF)
+#define HWIO_DDR_PHY_677_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_677_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_678_REGOFF 0xa98
+#define HWIO_DDR_PHY_678_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_678_REGOFF)
+#define HWIO_DDR_PHY_678_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_678_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_679_REGOFF 0xa9c
+#define HWIO_DDR_PHY_679_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_679_REGOFF)
+#define HWIO_DDR_PHY_679_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_679_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_680_REGOFF 0xaa0
+#define HWIO_DDR_PHY_680_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_680_REGOFF)
+#define HWIO_DDR_PHY_680_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_680_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_681_REGOFF 0xaa4
+#define HWIO_DDR_PHY_681_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_681_REGOFF)
+#define HWIO_DDR_PHY_681_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_681_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_682_REGOFF 0xaa8
+#define HWIO_DDR_PHY_682_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_682_REGOFF)
+#define HWIO_DDR_PHY_682_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_682_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_683_REGOFF 0xaac
+#define HWIO_DDR_PHY_683_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_683_REGOFF)
+#define HWIO_DDR_PHY_683_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_683_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_684_REGOFF 0xab0
+#define HWIO_DDR_PHY_684_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_684_REGOFF)
+#define HWIO_DDR_PHY_684_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_684_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_685_REGOFF 0xab4
+#define HWIO_DDR_PHY_685_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_685_REGOFF)
+#define HWIO_DDR_PHY_685_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_685_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_686_REGOFF 0xab8
+#define HWIO_DDR_PHY_686_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_686_REGOFF)
+#define HWIO_DDR_PHY_686_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_686_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_687_REGOFF 0xabc
+#define HWIO_DDR_PHY_687_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_687_REGOFF)
+#define HWIO_DDR_PHY_687_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_687_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_688_REGOFF 0xac0
+#define HWIO_DDR_PHY_688_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_688_REGOFF)
+#define HWIO_DDR_PHY_688_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_688_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_689_REGOFF 0xac4
+#define HWIO_DDR_PHY_689_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_689_REGOFF)
+#define HWIO_DDR_PHY_689_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_689_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_690_REGOFF 0xac8
+#define HWIO_DDR_PHY_690_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_690_REGOFF)
+#define HWIO_DDR_PHY_690_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_690_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_691_REGOFF 0xacc
+#define HWIO_DDR_PHY_691_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_691_REGOFF)
+#define HWIO_DDR_PHY_691_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_691_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_692_REGOFF 0xad0
+#define HWIO_DDR_PHY_692_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_692_REGOFF)
+#define HWIO_DDR_PHY_692_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_692_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_693_REGOFF 0xad4
+#define HWIO_DDR_PHY_693_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_693_REGOFF)
+#define HWIO_DDR_PHY_693_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_693_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_694_REGOFF 0xad8
+#define HWIO_DDR_PHY_694_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_694_REGOFF)
+#define HWIO_DDR_PHY_694_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_694_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_695_REGOFF 0xadc
+#define HWIO_DDR_PHY_695_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_695_REGOFF)
+#define HWIO_DDR_PHY_695_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_695_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_696_REGOFF 0xae0
+#define HWIO_DDR_PHY_696_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_696_REGOFF)
+#define HWIO_DDR_PHY_696_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_696_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_697_REGOFF 0xae4
+#define HWIO_DDR_PHY_697_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_697_REGOFF)
+#define HWIO_DDR_PHY_697_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_697_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_698_REGOFF 0xae8
+#define HWIO_DDR_PHY_698_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_698_REGOFF)
+#define HWIO_DDR_PHY_698_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_698_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_699_REGOFF 0xaec
+#define HWIO_DDR_PHY_699_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_699_REGOFF)
+#define HWIO_DDR_PHY_699_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_699_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_700_REGOFF 0xaf0
+#define HWIO_DDR_PHY_700_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_700_REGOFF)
+#define HWIO_DDR_PHY_700_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_700_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_701_REGOFF 0xaf4
+#define HWIO_DDR_PHY_701_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_701_REGOFF)
+#define HWIO_DDR_PHY_701_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_701_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_702_REGOFF 0xaf8
+#define HWIO_DDR_PHY_702_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_702_REGOFF)
+#define HWIO_DDR_PHY_702_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_702_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_703_REGOFF 0xafc
+#define HWIO_DDR_PHY_703_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_703_REGOFF)
+#define HWIO_DDR_PHY_703_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_703_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_704_REGOFF 0xb00
+#define HWIO_DDR_PHY_704_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_704_REGOFF)
+#define HWIO_DDR_PHY_704_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_704_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_705_REGOFF 0xb04
+#define HWIO_DDR_PHY_705_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_705_REGOFF)
+#define HWIO_DDR_PHY_705_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_705_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_706_REGOFF 0xb08
+#define HWIO_DDR_PHY_706_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_706_REGOFF)
+#define HWIO_DDR_PHY_706_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_706_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_707_REGOFF 0xb0c
+#define HWIO_DDR_PHY_707_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_707_REGOFF)
+#define HWIO_DDR_PHY_707_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_707_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_708_REGOFF 0xb10
+#define HWIO_DDR_PHY_708_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_708_REGOFF)
+#define HWIO_DDR_PHY_708_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_708_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_709_REGOFF 0xb14
+#define HWIO_DDR_PHY_709_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_709_REGOFF)
+#define HWIO_DDR_PHY_709_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_709_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_710_REGOFF 0xb18
+#define HWIO_DDR_PHY_710_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_710_REGOFF)
+#define HWIO_DDR_PHY_710_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_710_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_711_REGOFF 0xb1c
+#define HWIO_DDR_PHY_711_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_711_REGOFF)
+#define HWIO_DDR_PHY_711_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_711_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_712_REGOFF 0xb20
+#define HWIO_DDR_PHY_712_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_712_REGOFF)
+#define HWIO_DDR_PHY_712_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_712_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_713_REGOFF 0xb24
+#define HWIO_DDR_PHY_713_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_713_REGOFF)
+#define HWIO_DDR_PHY_713_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_713_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_714_REGOFF 0xb28
+#define HWIO_DDR_PHY_714_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_714_REGOFF)
+#define HWIO_DDR_PHY_714_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_714_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_715_REGOFF 0xb2c
+#define HWIO_DDR_PHY_715_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_715_REGOFF)
+#define HWIO_DDR_PHY_715_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_715_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_716_REGOFF 0xb30
+#define HWIO_DDR_PHY_716_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_716_REGOFF)
+#define HWIO_DDR_PHY_716_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_716_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_717_REGOFF 0xb34
+#define HWIO_DDR_PHY_717_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_717_REGOFF)
+#define HWIO_DDR_PHY_717_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_717_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_718_REGOFF 0xb38
+#define HWIO_DDR_PHY_718_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_718_REGOFF)
+#define HWIO_DDR_PHY_718_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_718_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_719_REGOFF 0xb3c
+#define HWIO_DDR_PHY_719_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_719_REGOFF)
+#define HWIO_DDR_PHY_719_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_719_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_720_REGOFF 0xb40
+#define HWIO_DDR_PHY_720_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_720_REGOFF)
+#define HWIO_DDR_PHY_720_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_720_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_721_REGOFF 0xb44
+#define HWIO_DDR_PHY_721_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_721_REGOFF)
+#define HWIO_DDR_PHY_721_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_721_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_722_REGOFF 0xb48
+#define HWIO_DDR_PHY_722_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_722_REGOFF)
+#define HWIO_DDR_PHY_722_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_722_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_723_REGOFF 0xb4c
+#define HWIO_DDR_PHY_723_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_723_REGOFF)
+#define HWIO_DDR_PHY_723_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_723_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_724_REGOFF 0xb50
+#define HWIO_DDR_PHY_724_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_724_REGOFF)
+#define HWIO_DDR_PHY_724_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_724_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_725_REGOFF 0xb54
+#define HWIO_DDR_PHY_725_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_725_REGOFF)
+#define HWIO_DDR_PHY_725_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_725_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_726_REGOFF 0xb58
+#define HWIO_DDR_PHY_726_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_726_REGOFF)
+#define HWIO_DDR_PHY_726_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_726_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_727_REGOFF 0xb5c
+#define HWIO_DDR_PHY_727_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_727_REGOFF)
+#define HWIO_DDR_PHY_727_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_727_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_728_REGOFF 0xb60
+#define HWIO_DDR_PHY_728_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_728_REGOFF)
+#define HWIO_DDR_PHY_728_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_728_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_729_REGOFF 0xb64
+#define HWIO_DDR_PHY_729_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_729_REGOFF)
+#define HWIO_DDR_PHY_729_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_729_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_730_REGOFF 0xb68
+#define HWIO_DDR_PHY_730_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_730_REGOFF)
+#define HWIO_DDR_PHY_730_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_730_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_731_REGOFF 0xb6c
+#define HWIO_DDR_PHY_731_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_731_REGOFF)
+#define HWIO_DDR_PHY_731_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_731_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_732_REGOFF 0xb70
+#define HWIO_DDR_PHY_732_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_732_REGOFF)
+#define HWIO_DDR_PHY_732_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_732_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_733_REGOFF 0xb74
+#define HWIO_DDR_PHY_733_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_733_REGOFF)
+#define HWIO_DDR_PHY_733_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_733_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_734_REGOFF 0xb78
+#define HWIO_DDR_PHY_734_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_734_REGOFF)
+#define HWIO_DDR_PHY_734_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_734_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_735_REGOFF 0xb7c
+#define HWIO_DDR_PHY_735_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_735_REGOFF)
+#define HWIO_DDR_PHY_735_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_735_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_736_REGOFF 0xb80
+#define HWIO_DDR_PHY_736_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_736_REGOFF)
+#define HWIO_DDR_PHY_736_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_736_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_737_REGOFF 0xb84
+#define HWIO_DDR_PHY_737_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_737_REGOFF)
+#define HWIO_DDR_PHY_737_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_737_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_738_REGOFF 0xb88
+#define HWIO_DDR_PHY_738_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_738_REGOFF)
+#define HWIO_DDR_PHY_738_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_738_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_739_REGOFF 0xb8c
+#define HWIO_DDR_PHY_739_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_739_REGOFF)
+#define HWIO_DDR_PHY_739_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_739_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_740_REGOFF 0xb90
+#define HWIO_DDR_PHY_740_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_740_REGOFF)
+#define HWIO_DDR_PHY_740_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_740_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_741_REGOFF 0xb94
+#define HWIO_DDR_PHY_741_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_741_REGOFF)
+#define HWIO_DDR_PHY_741_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_741_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_742_REGOFF 0xb98
+#define HWIO_DDR_PHY_742_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_742_REGOFF)
+#define HWIO_DDR_PHY_742_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_742_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_743_REGOFF 0xb9c
+#define HWIO_DDR_PHY_743_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_743_REGOFF)
+#define HWIO_DDR_PHY_743_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_743_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_744_REGOFF 0xba0
+#define HWIO_DDR_PHY_744_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_744_REGOFF)
+#define HWIO_DDR_PHY_744_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_744_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_745_REGOFF 0xba4
+#define HWIO_DDR_PHY_745_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_745_REGOFF)
+#define HWIO_DDR_PHY_745_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_745_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_746_REGOFF 0xba8
+#define HWIO_DDR_PHY_746_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_746_REGOFF)
+#define HWIO_DDR_PHY_746_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_746_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_747_REGOFF 0xbac
+#define HWIO_DDR_PHY_747_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_747_REGOFF)
+#define HWIO_DDR_PHY_747_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_747_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_748_REGOFF 0xbb0
+#define HWIO_DDR_PHY_748_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_748_REGOFF)
+#define HWIO_DDR_PHY_748_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_748_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_749_REGOFF 0xbb4
+#define HWIO_DDR_PHY_749_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_749_REGOFF)
+#define HWIO_DDR_PHY_749_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_749_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_750_REGOFF 0xbb8
+#define HWIO_DDR_PHY_750_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_750_REGOFF)
+#define HWIO_DDR_PHY_750_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_750_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_751_REGOFF 0xbbc
+#define HWIO_DDR_PHY_751_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_751_REGOFF)
+#define HWIO_DDR_PHY_751_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_751_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_752_REGOFF 0xbc0
+#define HWIO_DDR_PHY_752_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_752_REGOFF)
+#define HWIO_DDR_PHY_752_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_752_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_753_REGOFF 0xbc4
+#define HWIO_DDR_PHY_753_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_753_REGOFF)
+#define HWIO_DDR_PHY_753_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_753_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_754_REGOFF 0xbc8
+#define HWIO_DDR_PHY_754_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_754_REGOFF)
+#define HWIO_DDR_PHY_754_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_754_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_755_REGOFF 0xbcc
+#define HWIO_DDR_PHY_755_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_755_REGOFF)
+#define HWIO_DDR_PHY_755_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_755_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_756_REGOFF 0xbd0
+#define HWIO_DDR_PHY_756_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_756_REGOFF)
+#define HWIO_DDR_PHY_756_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_756_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_757_REGOFF 0xbd4
+#define HWIO_DDR_PHY_757_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_757_REGOFF)
+#define HWIO_DDR_PHY_757_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_757_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_758_REGOFF 0xbd8
+#define HWIO_DDR_PHY_758_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_758_REGOFF)
+#define HWIO_DDR_PHY_758_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_758_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_759_REGOFF 0xbdc
+#define HWIO_DDR_PHY_759_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_759_REGOFF)
+#define HWIO_DDR_PHY_759_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_759_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_760_REGOFF 0xbe0
+#define HWIO_DDR_PHY_760_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_760_REGOFF)
+#define HWIO_DDR_PHY_760_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_760_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_761_REGOFF 0xbe4
+#define HWIO_DDR_PHY_761_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_761_REGOFF)
+#define HWIO_DDR_PHY_761_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_761_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_762_REGOFF 0xbe8
+#define HWIO_DDR_PHY_762_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_762_REGOFF)
+#define HWIO_DDR_PHY_762_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_762_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_763_REGOFF 0xbec
+#define HWIO_DDR_PHY_763_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_763_REGOFF)
+#define HWIO_DDR_PHY_763_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_763_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_764_REGOFF 0xbf0
+#define HWIO_DDR_PHY_764_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_764_REGOFF)
+#define HWIO_DDR_PHY_764_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_764_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_765_REGOFF 0xbf4
+#define HWIO_DDR_PHY_765_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_765_REGOFF)
+#define HWIO_DDR_PHY_765_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_765_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_766_REGOFF 0xbf8
+#define HWIO_DDR_PHY_766_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_766_REGOFF)
+#define HWIO_DDR_PHY_766_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_766_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_767_REGOFF 0xbfc
+#define HWIO_DDR_PHY_767_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_767_REGOFF)
+#define HWIO_DDR_PHY_767_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_767_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_768_REGOFF 0xc00
+#define HWIO_DDR_PHY_768_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_768_REGOFF)
+#define HWIO_DDR_PHY_768_PHY_ADR0_SW_WRADDR_SHIFT_2_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_768_PHY_ADR0_SW_WRADDR_SHIFT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_768_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_768_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_768_PHY_ADR1_SW_WRADDR_SHIFT_2_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_768_PHY_ADR1_SW_WRADDR_SHIFT_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_768_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_768_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_768_PHY_ADR2_SW_WRADDR_SHIFT_2_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_768_PHY_ADR2_SW_WRADDR_SHIFT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_768_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_768_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_768_PHY_ADR3_SW_WRADDR_SHIFT_2_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_768_PHY_ADR3_SW_WRADDR_SHIFT_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_768_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_768_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_769_REGOFF 0xc04
+#define HWIO_DDR_PHY_769_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_769_REGOFF)
+#define HWIO_DDR_PHY_769_PHY_ADR4_SW_WRADDR_SHIFT_2_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_769_PHY_ADR4_SW_WRADDR_SHIFT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_769_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_769_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_769_PHY_ADR5_SW_WRADDR_SHIFT_2_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_769_PHY_ADR5_SW_WRADDR_SHIFT_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_769_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_769_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_769_PHY_ADR_CLK_WR_BYPASS_SLAVE_DELAY_2_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_769_PHY_ADR_CLK_WR_BYPASS_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_769_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_769_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_770_REGOFF 0xc08
+#define HWIO_DDR_PHY_770_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_770_REGOFF)
+#define HWIO_DDR_PHY_770_PHY_ADR_CLK_BYPASS_OVERRIDE_2_FLDMASK (0x1)
+#define HWIO_DDR_PHY_770_PHY_ADR_CLK_BYPASS_OVERRIDE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_770_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_770_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_770_SC_PHY_ADR_MANUAL_CLEAR_2_FLDMASK (0x700)
+#define HWIO_DDR_PHY_770_SC_PHY_ADR_MANUAL_CLEAR_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_770_RESERVED1_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_770_RESERVED1_FLDSHFT (11)
+#define HWIO_DDR_PHY_770_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_770_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_771_REGOFF 0xc0c
+#define HWIO_DDR_PHY_771_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_771_REGOFF)
+#define HWIO_DDR_PHY_771_PHY_ADR_LPBK_RESULT_OBS_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_771_PHY_ADR_LPBK_RESULT_OBS_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_772_REGOFF 0xc10
+#define HWIO_DDR_PHY_772_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_772_REGOFF)
+#define HWIO_DDR_PHY_772_PHY_ADR_LPBK_ERROR_COUNT_OBS_2_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_772_PHY_ADR_LPBK_ERROR_COUNT_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_772_PHY_ADR_MASTER_DLY_LOCK_OBS_SELECT_2_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_772_PHY_ADR_MASTER_DLY_LOCK_OBS_SELECT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_772_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_772_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_772_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_772_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_773_REGOFF 0xc14
+#define HWIO_DDR_PHY_773_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_773_REGOFF)
+#define HWIO_DDR_PHY_773_PHY_ADR_MASTER_DLY_LOCK_OBS_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_773_PHY_ADR_MASTER_DLY_LOCK_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_773_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_773_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_773_PHY_ADR_BASE_SLV_DLY_ENC_OBS_2_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_773_PHY_ADR_BASE_SLV_DLY_ENC_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_773_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_773_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_774_REGOFF 0xc18
+#define HWIO_DDR_PHY_774_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_774_REGOFF)
+#define HWIO_DDR_PHY_774_PHY_ADR_ADDER_SLV_DLY_ENC_OBS_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_774_PHY_ADR_ADDER_SLV_DLY_ENC_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_774_PHY_ADR_SLAVE_LOOP_CNT_UPDATE_2_FLDMASK (0x700)
+#define HWIO_DDR_PHY_774_PHY_ADR_SLAVE_LOOP_CNT_UPDATE_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_774_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_774_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_774_PHY_ADR_SLV_DLY_ENC_OBS_SELECT_2_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_774_PHY_ADR_SLV_DLY_ENC_OBS_SELECT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_774_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_774_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_774_SC_PHY_ADR_SNAP_OBS_REGS_2_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_774_SC_PHY_ADR_SNAP_OBS_REGS_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_774_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_774_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_775_REGOFF 0xc1c
+#define HWIO_DDR_PHY_775_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_775_REGOFF)
+#define HWIO_DDR_PHY_775_PHY_ADR_TSEL_ENABLE_2_FLDMASK (0x1)
+#define HWIO_DDR_PHY_775_PHY_ADR_TSEL_ENABLE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_775_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_775_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_775_PHY_ADR_LPBK_CONTROL_2_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_775_PHY_ADR_LPBK_CONTROL_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_775_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_775_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_775_PHY_ADR_PRBS_PATTERN_START_2_FLDMASK (0x7f0000)
+#define HWIO_DDR_PHY_775_PHY_ADR_PRBS_PATTERN_START_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_775_RESERVED2_FLDMASK (0x800000)
+#define HWIO_DDR_PHY_775_RESERVED2_FLDSHFT (23)
+#define HWIO_DDR_PHY_775_PHY_ADR_PRBS_PATTERN_MASK_2_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_775_PHY_ADR_PRBS_PATTERN_MASK_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_775_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_775_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_776_REGOFF 0xc20
+#define HWIO_DDR_PHY_776_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_776_REGOFF)
+#define HWIO_DDR_PHY_776_PHY_ADR_PWR_RDC_DISABLE_2_FLDMASK (0x1)
+#define HWIO_DDR_PHY_776_PHY_ADR_PWR_RDC_DISABLE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_776_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_776_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_776_PHY_ADR_TYPE_2_FLDMASK (0x300)
+#define HWIO_DDR_PHY_776_PHY_ADR_TYPE_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_776_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_776_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_776_PHY_ADR_WRADDR_SHIFT_OBS_2_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_776_PHY_ADR_WRADDR_SHIFT_OBS_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_776_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_776_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_776_PHY_ADR_IE_MODE_2_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_776_PHY_ADR_IE_MODE_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_776_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_776_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_777_REGOFF 0xc24
+#define HWIO_DDR_PHY_777_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_777_REGOFF)
+#define HWIO_DDR_PHY_777_PHY_ADR_DDL_MODE_2_FLDMASK (0x7fff)
+#define HWIO_DDR_PHY_777_PHY_ADR_DDL_MODE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_777_RESERVED_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_777_RESERVED_FLDSHFT (15)
+#define HWIO_DDR_PHY_777_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_777_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_778_REGOFF 0xc28
+#define HWIO_DDR_PHY_778_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_778_REGOFF)
+#define HWIO_DDR_PHY_778_PHY_ADR_DDL_TEST_OBS_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_778_PHY_ADR_DDL_TEST_OBS_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_779_REGOFF 0xc2c
+#define HWIO_DDR_PHY_779_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_779_REGOFF)
+#define HWIO_DDR_PHY_779_PHY_ADR_DDL_TEST_MSTR_DLY_OBS_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_779_PHY_ADR_DDL_TEST_MSTR_DLY_OBS_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_780_REGOFF 0xc30
+#define HWIO_DDR_PHY_780_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_780_REGOFF)
+#define HWIO_DDR_PHY_780_PHY_ADR_CALVL_START_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_780_PHY_ADR_CALVL_START_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_780_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_780_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_780_PHY_ADR_CALVL_COARSE_DLY_2_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_780_PHY_ADR_CALVL_COARSE_DLY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_780_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_780_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_781_REGOFF 0xc34
+#define HWIO_DDR_PHY_781_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_781_REGOFF)
+#define HWIO_DDR_PHY_781_PHY_ADR_CALVL_QTR_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_781_PHY_ADR_CALVL_QTR_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_781_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_781_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_781_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_781_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_782_REGOFF 0xc38
+#define HWIO_DDR_PHY_782_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_782_REGOFF)
+#define HWIO_DDR_PHY_782_PHY_ADR_CALVL_SWIZZLE0_2_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_782_PHY_ADR_CALVL_SWIZZLE0_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_782_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_782_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_783_REGOFF 0xc3c
+#define HWIO_DDR_PHY_783_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_783_REGOFF)
+#define HWIO_DDR_PHY_783_PHY_ADR_CALVL_SWIZZLE1_2_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_783_PHY_ADR_CALVL_SWIZZLE1_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_783_PHY_ADR_CALVL_RANK_CTRL_2_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_783_PHY_ADR_CALVL_RANK_CTRL_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_783_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_783_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_784_REGOFF 0xc40
+#define HWIO_DDR_PHY_784_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_784_REGOFF)
+#define HWIO_DDR_PHY_784_PHY_ADR_CALVL_NUM_PATTERNS_2_FLDMASK (0x3)
+#define HWIO_DDR_PHY_784_PHY_ADR_CALVL_NUM_PATTERNS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_784_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_784_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_784_PHY_ADR_CALVL_CAPTURE_CNT_2_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_784_PHY_ADR_CALVL_CAPTURE_CNT_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_784_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_784_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_784_PHY_ADR_CALVL_RESP_WAIT_CNT_2_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_784_PHY_ADR_CALVL_RESP_WAIT_CNT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_784_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_784_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_784_PHY_ADR_CALVL_DEBUG_MODE_2_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_784_PHY_ADR_CALVL_DEBUG_MODE_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_784_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_784_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_785_REGOFF 0xc44
+#define HWIO_DDR_PHY_785_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_785_REGOFF)
+#define HWIO_DDR_PHY_785_SC_PHY_ADR_CALVL_DEBUG_CONT_2_FLDMASK (0x1)
+#define HWIO_DDR_PHY_785_SC_PHY_ADR_CALVL_DEBUG_CONT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_785_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_785_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_785_SC_PHY_ADR_CALVL_ERROR_CLR_2_FLDMASK (0x100)
+#define HWIO_DDR_PHY_785_SC_PHY_ADR_CALVL_ERROR_CLR_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_785_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_785_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_785_PHY_ADR_CALVL_OBS_SELECT_2_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_785_PHY_ADR_CALVL_OBS_SELECT_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_785_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_785_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_785_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_785_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_786_REGOFF 0xc48
+#define HWIO_DDR_PHY_786_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_786_REGOFF)
+#define HWIO_DDR_PHY_786_PHY_ADR_CALVL_OBS0_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_786_PHY_ADR_CALVL_OBS0_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_787_REGOFF 0xc4c
+#define HWIO_DDR_PHY_787_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_787_REGOFF)
+#define HWIO_DDR_PHY_787_PHY_ADR_CALVL_OBS1_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_787_PHY_ADR_CALVL_OBS1_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_788_REGOFF 0xc50
+#define HWIO_DDR_PHY_788_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_788_REGOFF)
+#define HWIO_DDR_PHY_788_PHY_ADR_CALVL_FG_0_2_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_788_PHY_ADR_CALVL_FG_0_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_788_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_788_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_788_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_788_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_789_REGOFF 0xc54
+#define HWIO_DDR_PHY_789_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_789_REGOFF)
+#define HWIO_DDR_PHY_789_PHY_ADR_CALVL_BG_0_2_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_789_PHY_ADR_CALVL_BG_0_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_789_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_789_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_789_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_789_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_790_REGOFF 0xc58
+#define HWIO_DDR_PHY_790_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_790_REGOFF)
+#define HWIO_DDR_PHY_790_PHY_ADR_CALVL_FG_1_2_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_790_PHY_ADR_CALVL_FG_1_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_790_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_790_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_790_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_790_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_791_REGOFF 0xc5c
+#define HWIO_DDR_PHY_791_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_791_REGOFF)
+#define HWIO_DDR_PHY_791_PHY_ADR_CALVL_BG_1_2_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_791_PHY_ADR_CALVL_BG_1_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_791_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_791_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_791_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_791_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_792_REGOFF 0xc60
+#define HWIO_DDR_PHY_792_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_792_REGOFF)
+#define HWIO_DDR_PHY_792_PHY_ADR_CALVL_FG_2_2_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_792_PHY_ADR_CALVL_FG_2_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_792_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_792_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_792_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_792_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_793_REGOFF 0xc64
+#define HWIO_DDR_PHY_793_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_793_REGOFF)
+#define HWIO_DDR_PHY_793_PHY_ADR_CALVL_BG_2_2_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_793_PHY_ADR_CALVL_BG_2_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_793_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_793_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_793_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_793_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_794_REGOFF 0xc68
+#define HWIO_DDR_PHY_794_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_794_REGOFF)
+#define HWIO_DDR_PHY_794_PHY_ADR_CALVL_FG_3_2_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_794_PHY_ADR_CALVL_FG_3_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_794_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_794_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_794_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_794_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_795_REGOFF 0xc6c
+#define HWIO_DDR_PHY_795_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_795_REGOFF)
+#define HWIO_DDR_PHY_795_PHY_ADR_CALVL_BG_3_2_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_795_PHY_ADR_CALVL_BG_3_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_795_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_795_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_795_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_795_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_796_REGOFF 0xc70
+#define HWIO_DDR_PHY_796_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_796_REGOFF)
+#define HWIO_DDR_PHY_796_PHY_ADR_ADDR_SEL_2_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_796_PHY_ADR_ADDR_SEL_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_796_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_796_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_797_REGOFF 0xc74
+#define HWIO_DDR_PHY_797_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_797_REGOFF)
+#define HWIO_DDR_PHY_797_PHY_ADR_LP4_BOOT_SLV_DELAY_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_797_PHY_ADR_LP4_BOOT_SLV_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_797_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_797_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_797_PHY_ADR_BIT_MASK_2_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_797_PHY_ADR_BIT_MASK_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_797_RESERVED1_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_797_RESERVED1_FLDSHFT (22)
+#define HWIO_DDR_PHY_797_PHY_ADR_SEG_MASK_2_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_797_PHY_ADR_SEG_MASK_2_FLDSHFT (24)
+#define HWIO_DDR_PHY_797_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_797_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_798_REGOFF 0xc78
+#define HWIO_DDR_PHY_798_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_798_REGOFF)
+#define HWIO_DDR_PHY_798_PHY_ADR_CALVL_TRAIN_MASK_2_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_798_PHY_ADR_CALVL_TRAIN_MASK_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_798_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_798_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_798_PHY_ADR_SW_TXIO_CTRL_2_FLDMASK (0x3f00)
+#define HWIO_DDR_PHY_798_PHY_ADR_SW_TXIO_CTRL_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_798_RESERVED1_FLDMASK (0xc000)
+#define HWIO_DDR_PHY_798_RESERVED1_FLDSHFT (14)
+#define HWIO_DDR_PHY_798_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_798_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_799_REGOFF 0xc7c
+#define HWIO_DDR_PHY_799_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_799_REGOFF)
+#define HWIO_DDR_PHY_799_PHY_ADR_TSEL_SELECT_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_799_PHY_ADR_TSEL_SELECT_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_799_PHY_ADR0_CLK_WR_SLAVE_DELAY_2_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_799_PHY_ADR0_CLK_WR_SLAVE_DELAY_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_799_RESERVED_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_799_RESERVED_FLDSHFT (19)
+#define HWIO_DDR_PHY_799_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_799_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_800_REGOFF 0xc80
+#define HWIO_DDR_PHY_800_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_800_REGOFF)
+#define HWIO_DDR_PHY_800_PHY_ADR1_CLK_WR_SLAVE_DELAY_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_800_PHY_ADR1_CLK_WR_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_800_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_800_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_800_PHY_ADR2_CLK_WR_SLAVE_DELAY_2_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_800_PHY_ADR2_CLK_WR_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_800_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_800_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_801_REGOFF 0xc84
+#define HWIO_DDR_PHY_801_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_801_REGOFF)
+#define HWIO_DDR_PHY_801_PHY_ADR3_CLK_WR_SLAVE_DELAY_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_801_PHY_ADR3_CLK_WR_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_801_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_801_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_801_PHY_ADR4_CLK_WR_SLAVE_DELAY_2_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_801_PHY_ADR4_CLK_WR_SLAVE_DELAY_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_801_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_801_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_802_REGOFF 0xc88
+#define HWIO_DDR_PHY_802_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_802_REGOFF)
+#define HWIO_DDR_PHY_802_PHY_ADR5_CLK_WR_SLAVE_DELAY_2_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_802_PHY_ADR5_CLK_WR_SLAVE_DELAY_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_802_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_802_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_802_PHY_ADR_SW_MASTER_MODE_2_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_802_PHY_ADR_SW_MASTER_MODE_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_802_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_802_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PHY_802_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_802_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_803_REGOFF 0xc8c
+#define HWIO_DDR_PHY_803_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_803_REGOFF)
+#define HWIO_DDR_PHY_803_PHY_ADR_MASTER_DELAY_START_2_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_803_PHY_ADR_MASTER_DELAY_START_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_803_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_803_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_803_PHY_ADR_MASTER_DELAY_STEP_2_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_803_PHY_ADR_MASTER_DELAY_STEP_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_803_RESERVED1_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_803_RESERVED1_FLDSHFT (22)
+#define HWIO_DDR_PHY_803_PHY_ADR_MASTER_DELAY_WAIT_2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_803_PHY_ADR_MASTER_DELAY_WAIT_2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_804_REGOFF 0xc90
+#define HWIO_DDR_PHY_804_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_804_REGOFF)
+#define HWIO_DDR_PHY_804_PHY_ADR_MASTER_DELAY_HALF_MEASURE_2_FLDMASK (0xff)
+#define HWIO_DDR_PHY_804_PHY_ADR_MASTER_DELAY_HALF_MEASURE_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_804_PHY_ADR_CALVL_DLY_STEP_2_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_804_PHY_ADR_CALVL_DLY_STEP_2_FLDSHFT (8)
+#define HWIO_DDR_PHY_804_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_804_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_804_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_804_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_805_REGOFF 0xc94
+#define HWIO_DDR_PHY_805_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_805_REGOFF)
+#define HWIO_DDR_PHY_805_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_805_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_806_REGOFF 0xc98
+#define HWIO_DDR_PHY_806_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_806_REGOFF)
+#define HWIO_DDR_PHY_806_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_806_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_807_REGOFF 0xc9c
+#define HWIO_DDR_PHY_807_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_807_REGOFF)
+#define HWIO_DDR_PHY_807_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_807_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_808_REGOFF 0xca0
+#define HWIO_DDR_PHY_808_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_808_REGOFF)
+#define HWIO_DDR_PHY_808_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_808_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_809_REGOFF 0xca4
+#define HWIO_DDR_PHY_809_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_809_REGOFF)
+#define HWIO_DDR_PHY_809_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_809_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_810_REGOFF 0xca8
+#define HWIO_DDR_PHY_810_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_810_REGOFF)
+#define HWIO_DDR_PHY_810_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_810_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_811_REGOFF 0xcac
+#define HWIO_DDR_PHY_811_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_811_REGOFF)
+#define HWIO_DDR_PHY_811_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_811_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_812_REGOFF 0xcb0
+#define HWIO_DDR_PHY_812_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_812_REGOFF)
+#define HWIO_DDR_PHY_812_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_812_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_813_REGOFF 0xcb4
+#define HWIO_DDR_PHY_813_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_813_REGOFF)
+#define HWIO_DDR_PHY_813_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_813_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_814_REGOFF 0xcb8
+#define HWIO_DDR_PHY_814_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_814_REGOFF)
+#define HWIO_DDR_PHY_814_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_814_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_815_REGOFF 0xcbc
+#define HWIO_DDR_PHY_815_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_815_REGOFF)
+#define HWIO_DDR_PHY_815_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_815_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_816_REGOFF 0xcc0
+#define HWIO_DDR_PHY_816_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_816_REGOFF)
+#define HWIO_DDR_PHY_816_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_816_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_817_REGOFF 0xcc4
+#define HWIO_DDR_PHY_817_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_817_REGOFF)
+#define HWIO_DDR_PHY_817_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_817_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_818_REGOFF 0xcc8
+#define HWIO_DDR_PHY_818_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_818_REGOFF)
+#define HWIO_DDR_PHY_818_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_818_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_819_REGOFF 0xccc
+#define HWIO_DDR_PHY_819_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_819_REGOFF)
+#define HWIO_DDR_PHY_819_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_819_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_820_REGOFF 0xcd0
+#define HWIO_DDR_PHY_820_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_820_REGOFF)
+#define HWIO_DDR_PHY_820_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_820_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_821_REGOFF 0xcd4
+#define HWIO_DDR_PHY_821_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_821_REGOFF)
+#define HWIO_DDR_PHY_821_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_821_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_822_REGOFF 0xcd8
+#define HWIO_DDR_PHY_822_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_822_REGOFF)
+#define HWIO_DDR_PHY_822_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_822_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_823_REGOFF 0xcdc
+#define HWIO_DDR_PHY_823_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_823_REGOFF)
+#define HWIO_DDR_PHY_823_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_823_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_824_REGOFF 0xce0
+#define HWIO_DDR_PHY_824_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_824_REGOFF)
+#define HWIO_DDR_PHY_824_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_824_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_825_REGOFF 0xce4
+#define HWIO_DDR_PHY_825_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_825_REGOFF)
+#define HWIO_DDR_PHY_825_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_825_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_826_REGOFF 0xce8
+#define HWIO_DDR_PHY_826_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_826_REGOFF)
+#define HWIO_DDR_PHY_826_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_826_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_827_REGOFF 0xcec
+#define HWIO_DDR_PHY_827_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_827_REGOFF)
+#define HWIO_DDR_PHY_827_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_827_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_828_REGOFF 0xcf0
+#define HWIO_DDR_PHY_828_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_828_REGOFF)
+#define HWIO_DDR_PHY_828_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_828_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_829_REGOFF 0xcf4
+#define HWIO_DDR_PHY_829_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_829_REGOFF)
+#define HWIO_DDR_PHY_829_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_829_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_830_REGOFF 0xcf8
+#define HWIO_DDR_PHY_830_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_830_REGOFF)
+#define HWIO_DDR_PHY_830_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_830_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_831_REGOFF 0xcfc
+#define HWIO_DDR_PHY_831_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_831_REGOFF)
+#define HWIO_DDR_PHY_831_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_831_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_832_REGOFF 0xd00
+#define HWIO_DDR_PHY_832_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_832_REGOFF)
+#define HWIO_DDR_PHY_832_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_832_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_833_REGOFF 0xd04
+#define HWIO_DDR_PHY_833_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_833_REGOFF)
+#define HWIO_DDR_PHY_833_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_833_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_834_REGOFF 0xd08
+#define HWIO_DDR_PHY_834_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_834_REGOFF)
+#define HWIO_DDR_PHY_834_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_834_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_835_REGOFF 0xd0c
+#define HWIO_DDR_PHY_835_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_835_REGOFF)
+#define HWIO_DDR_PHY_835_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_835_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_836_REGOFF 0xd10
+#define HWIO_DDR_PHY_836_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_836_REGOFF)
+#define HWIO_DDR_PHY_836_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_836_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_837_REGOFF 0xd14
+#define HWIO_DDR_PHY_837_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_837_REGOFF)
+#define HWIO_DDR_PHY_837_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_837_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_838_REGOFF 0xd18
+#define HWIO_DDR_PHY_838_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_838_REGOFF)
+#define HWIO_DDR_PHY_838_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_838_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_839_REGOFF 0xd1c
+#define HWIO_DDR_PHY_839_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_839_REGOFF)
+#define HWIO_DDR_PHY_839_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_839_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_840_REGOFF 0xd20
+#define HWIO_DDR_PHY_840_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_840_REGOFF)
+#define HWIO_DDR_PHY_840_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_840_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_841_REGOFF 0xd24
+#define HWIO_DDR_PHY_841_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_841_REGOFF)
+#define HWIO_DDR_PHY_841_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_841_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_842_REGOFF 0xd28
+#define HWIO_DDR_PHY_842_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_842_REGOFF)
+#define HWIO_DDR_PHY_842_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_842_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_843_REGOFF 0xd2c
+#define HWIO_DDR_PHY_843_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_843_REGOFF)
+#define HWIO_DDR_PHY_843_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_843_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_844_REGOFF 0xd30
+#define HWIO_DDR_PHY_844_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_844_REGOFF)
+#define HWIO_DDR_PHY_844_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_844_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_845_REGOFF 0xd34
+#define HWIO_DDR_PHY_845_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_845_REGOFF)
+#define HWIO_DDR_PHY_845_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_845_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_846_REGOFF 0xd38
+#define HWIO_DDR_PHY_846_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_846_REGOFF)
+#define HWIO_DDR_PHY_846_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_846_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_847_REGOFF 0xd3c
+#define HWIO_DDR_PHY_847_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_847_REGOFF)
+#define HWIO_DDR_PHY_847_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_847_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_848_REGOFF 0xd40
+#define HWIO_DDR_PHY_848_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_848_REGOFF)
+#define HWIO_DDR_PHY_848_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_848_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_849_REGOFF 0xd44
+#define HWIO_DDR_PHY_849_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_849_REGOFF)
+#define HWIO_DDR_PHY_849_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_849_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_850_REGOFF 0xd48
+#define HWIO_DDR_PHY_850_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_850_REGOFF)
+#define HWIO_DDR_PHY_850_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_850_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_851_REGOFF 0xd4c
+#define HWIO_DDR_PHY_851_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_851_REGOFF)
+#define HWIO_DDR_PHY_851_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_851_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_852_REGOFF 0xd50
+#define HWIO_DDR_PHY_852_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_852_REGOFF)
+#define HWIO_DDR_PHY_852_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_852_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_853_REGOFF 0xd54
+#define HWIO_DDR_PHY_853_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_853_REGOFF)
+#define HWIO_DDR_PHY_853_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_853_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_854_REGOFF 0xd58
+#define HWIO_DDR_PHY_854_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_854_REGOFF)
+#define HWIO_DDR_PHY_854_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_854_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_855_REGOFF 0xd5c
+#define HWIO_DDR_PHY_855_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_855_REGOFF)
+#define HWIO_DDR_PHY_855_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_855_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_856_REGOFF 0xd60
+#define HWIO_DDR_PHY_856_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_856_REGOFF)
+#define HWIO_DDR_PHY_856_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_856_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_857_REGOFF 0xd64
+#define HWIO_DDR_PHY_857_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_857_REGOFF)
+#define HWIO_DDR_PHY_857_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_857_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_858_REGOFF 0xd68
+#define HWIO_DDR_PHY_858_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_858_REGOFF)
+#define HWIO_DDR_PHY_858_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_858_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_859_REGOFF 0xd6c
+#define HWIO_DDR_PHY_859_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_859_REGOFF)
+#define HWIO_DDR_PHY_859_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_859_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_860_REGOFF 0xd70
+#define HWIO_DDR_PHY_860_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_860_REGOFF)
+#define HWIO_DDR_PHY_860_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_860_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_861_REGOFF 0xd74
+#define HWIO_DDR_PHY_861_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_861_REGOFF)
+#define HWIO_DDR_PHY_861_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_861_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_862_REGOFF 0xd78
+#define HWIO_DDR_PHY_862_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_862_REGOFF)
+#define HWIO_DDR_PHY_862_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_862_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_863_REGOFF 0xd7c
+#define HWIO_DDR_PHY_863_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_863_REGOFF)
+#define HWIO_DDR_PHY_863_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_863_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_864_REGOFF 0xd80
+#define HWIO_DDR_PHY_864_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_864_REGOFF)
+#define HWIO_DDR_PHY_864_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_864_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_865_REGOFF 0xd84
+#define HWIO_DDR_PHY_865_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_865_REGOFF)
+#define HWIO_DDR_PHY_865_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_865_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_866_REGOFF 0xd88
+#define HWIO_DDR_PHY_866_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_866_REGOFF)
+#define HWIO_DDR_PHY_866_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_866_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_867_REGOFF 0xd8c
+#define HWIO_DDR_PHY_867_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_867_REGOFF)
+#define HWIO_DDR_PHY_867_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_867_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_868_REGOFF 0xd90
+#define HWIO_DDR_PHY_868_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_868_REGOFF)
+#define HWIO_DDR_PHY_868_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_868_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_869_REGOFF 0xd94
+#define HWIO_DDR_PHY_869_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_869_REGOFF)
+#define HWIO_DDR_PHY_869_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_869_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_870_REGOFF 0xd98
+#define HWIO_DDR_PHY_870_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_870_REGOFF)
+#define HWIO_DDR_PHY_870_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_870_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_871_REGOFF 0xd9c
+#define HWIO_DDR_PHY_871_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_871_REGOFF)
+#define HWIO_DDR_PHY_871_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_871_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_872_REGOFF 0xda0
+#define HWIO_DDR_PHY_872_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_872_REGOFF)
+#define HWIO_DDR_PHY_872_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_872_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_873_REGOFF 0xda4
+#define HWIO_DDR_PHY_873_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_873_REGOFF)
+#define HWIO_DDR_PHY_873_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_873_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_874_REGOFF 0xda8
+#define HWIO_DDR_PHY_874_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_874_REGOFF)
+#define HWIO_DDR_PHY_874_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_874_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_875_REGOFF 0xdac
+#define HWIO_DDR_PHY_875_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_875_REGOFF)
+#define HWIO_DDR_PHY_875_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_875_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_876_REGOFF 0xdb0
+#define HWIO_DDR_PHY_876_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_876_REGOFF)
+#define HWIO_DDR_PHY_876_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_876_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_877_REGOFF 0xdb4
+#define HWIO_DDR_PHY_877_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_877_REGOFF)
+#define HWIO_DDR_PHY_877_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_877_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_878_REGOFF 0xdb8
+#define HWIO_DDR_PHY_878_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_878_REGOFF)
+#define HWIO_DDR_PHY_878_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_878_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_879_REGOFF 0xdbc
+#define HWIO_DDR_PHY_879_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_879_REGOFF)
+#define HWIO_DDR_PHY_879_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_879_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_880_REGOFF 0xdc0
+#define HWIO_DDR_PHY_880_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_880_REGOFF)
+#define HWIO_DDR_PHY_880_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_880_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_881_REGOFF 0xdc4
+#define HWIO_DDR_PHY_881_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_881_REGOFF)
+#define HWIO_DDR_PHY_881_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_881_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_882_REGOFF 0xdc8
+#define HWIO_DDR_PHY_882_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_882_REGOFF)
+#define HWIO_DDR_PHY_882_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_882_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_883_REGOFF 0xdcc
+#define HWIO_DDR_PHY_883_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_883_REGOFF)
+#define HWIO_DDR_PHY_883_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_883_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_884_REGOFF 0xdd0
+#define HWIO_DDR_PHY_884_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_884_REGOFF)
+#define HWIO_DDR_PHY_884_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_884_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_885_REGOFF 0xdd4
+#define HWIO_DDR_PHY_885_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_885_REGOFF)
+#define HWIO_DDR_PHY_885_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_885_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_886_REGOFF 0xdd8
+#define HWIO_DDR_PHY_886_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_886_REGOFF)
+#define HWIO_DDR_PHY_886_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_886_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_887_REGOFF 0xddc
+#define HWIO_DDR_PHY_887_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_887_REGOFF)
+#define HWIO_DDR_PHY_887_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_887_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_888_REGOFF 0xde0
+#define HWIO_DDR_PHY_888_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_888_REGOFF)
+#define HWIO_DDR_PHY_888_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_888_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_889_REGOFF 0xde4
+#define HWIO_DDR_PHY_889_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_889_REGOFF)
+#define HWIO_DDR_PHY_889_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_889_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_890_REGOFF 0xde8
+#define HWIO_DDR_PHY_890_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_890_REGOFF)
+#define HWIO_DDR_PHY_890_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_890_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_891_REGOFF 0xdec
+#define HWIO_DDR_PHY_891_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_891_REGOFF)
+#define HWIO_DDR_PHY_891_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_891_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_892_REGOFF 0xdf0
+#define HWIO_DDR_PHY_892_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_892_REGOFF)
+#define HWIO_DDR_PHY_892_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_892_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_893_REGOFF 0xdf4
+#define HWIO_DDR_PHY_893_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_893_REGOFF)
+#define HWIO_DDR_PHY_893_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_893_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_894_REGOFF 0xdf8
+#define HWIO_DDR_PHY_894_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_894_REGOFF)
+#define HWIO_DDR_PHY_894_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_894_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_895_REGOFF 0xdfc
+#define HWIO_DDR_PHY_895_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_895_REGOFF)
+#define HWIO_DDR_PHY_895_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_895_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_896_REGOFF 0xe00
+#define HWIO_DDR_PHY_896_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_896_REGOFF)
+#define HWIO_DDR_PHY_896_PHY_ADR0_SW_WRADDR_SHIFT_3_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_896_PHY_ADR0_SW_WRADDR_SHIFT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_896_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_896_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_896_PHY_ADR1_SW_WRADDR_SHIFT_3_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_896_PHY_ADR1_SW_WRADDR_SHIFT_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_896_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_896_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_896_PHY_ADR2_SW_WRADDR_SHIFT_3_FLDMASK (0x1f0000)
+#define HWIO_DDR_PHY_896_PHY_ADR2_SW_WRADDR_SHIFT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_896_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_896_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PHY_896_PHY_ADR3_SW_WRADDR_SHIFT_3_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_896_PHY_ADR3_SW_WRADDR_SHIFT_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_896_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_896_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_897_REGOFF 0xe04
+#define HWIO_DDR_PHY_897_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_897_REGOFF)
+#define HWIO_DDR_PHY_897_PHY_ADR4_SW_WRADDR_SHIFT_3_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_897_PHY_ADR4_SW_WRADDR_SHIFT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_897_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_897_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_897_PHY_ADR5_SW_WRADDR_SHIFT_3_FLDMASK (0x1f00)
+#define HWIO_DDR_PHY_897_PHY_ADR5_SW_WRADDR_SHIFT_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_897_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_897_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PHY_897_PHY_ADR_CLK_WR_BYPASS_SLAVE_DELAY_3_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_897_PHY_ADR_CLK_WR_BYPASS_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_897_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_897_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_898_REGOFF 0xe08
+#define HWIO_DDR_PHY_898_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_898_REGOFF)
+#define HWIO_DDR_PHY_898_PHY_ADR_CLK_BYPASS_OVERRIDE_3_FLDMASK (0x1)
+#define HWIO_DDR_PHY_898_PHY_ADR_CLK_BYPASS_OVERRIDE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_898_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_898_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_898_SC_PHY_ADR_MANUAL_CLEAR_3_FLDMASK (0x700)
+#define HWIO_DDR_PHY_898_SC_PHY_ADR_MANUAL_CLEAR_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_898_RESERVED1_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_898_RESERVED1_FLDSHFT (11)
+#define HWIO_DDR_PHY_898_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_898_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_899_REGOFF 0xe0c
+#define HWIO_DDR_PHY_899_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_899_REGOFF)
+#define HWIO_DDR_PHY_899_PHY_ADR_LPBK_RESULT_OBS_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_899_PHY_ADR_LPBK_RESULT_OBS_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_900_REGOFF 0xe10
+#define HWIO_DDR_PHY_900_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_900_REGOFF)
+#define HWIO_DDR_PHY_900_PHY_ADR_LPBK_ERROR_COUNT_OBS_3_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_900_PHY_ADR_LPBK_ERROR_COUNT_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_900_PHY_ADR_MASTER_DLY_LOCK_OBS_SELECT_3_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_900_PHY_ADR_MASTER_DLY_LOCK_OBS_SELECT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_900_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_900_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_900_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_900_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_901_REGOFF 0xe14
+#define HWIO_DDR_PHY_901_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_901_REGOFF)
+#define HWIO_DDR_PHY_901_PHY_ADR_MASTER_DLY_LOCK_OBS_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_901_PHY_ADR_MASTER_DLY_LOCK_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_901_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_901_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_901_PHY_ADR_BASE_SLV_DLY_ENC_OBS_3_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_901_PHY_ADR_BASE_SLV_DLY_ENC_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_901_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_901_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_902_REGOFF 0xe18
+#define HWIO_DDR_PHY_902_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_902_REGOFF)
+#define HWIO_DDR_PHY_902_PHY_ADR_ADDER_SLV_DLY_ENC_OBS_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_902_PHY_ADR_ADDER_SLV_DLY_ENC_OBS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_902_PHY_ADR_SLAVE_LOOP_CNT_UPDATE_3_FLDMASK (0x700)
+#define HWIO_DDR_PHY_902_PHY_ADR_SLAVE_LOOP_CNT_UPDATE_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_902_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_902_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_902_PHY_ADR_SLV_DLY_ENC_OBS_SELECT_3_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_902_PHY_ADR_SLV_DLY_ENC_OBS_SELECT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_902_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_902_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_902_SC_PHY_ADR_SNAP_OBS_REGS_3_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_902_SC_PHY_ADR_SNAP_OBS_REGS_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_902_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_902_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_903_REGOFF 0xe1c
+#define HWIO_DDR_PHY_903_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_903_REGOFF)
+#define HWIO_DDR_PHY_903_PHY_ADR_TSEL_ENABLE_3_FLDMASK (0x1)
+#define HWIO_DDR_PHY_903_PHY_ADR_TSEL_ENABLE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_903_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_903_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_903_PHY_ADR_LPBK_CONTROL_3_FLDMASK (0x7f00)
+#define HWIO_DDR_PHY_903_PHY_ADR_LPBK_CONTROL_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_903_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_903_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PHY_903_PHY_ADR_PRBS_PATTERN_START_3_FLDMASK (0x7f0000)
+#define HWIO_DDR_PHY_903_PHY_ADR_PRBS_PATTERN_START_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_903_RESERVED2_FLDMASK (0x800000)
+#define HWIO_DDR_PHY_903_RESERVED2_FLDSHFT (23)
+#define HWIO_DDR_PHY_903_PHY_ADR_PRBS_PATTERN_MASK_3_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_903_PHY_ADR_PRBS_PATTERN_MASK_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_903_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_903_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_904_REGOFF 0xe20
+#define HWIO_DDR_PHY_904_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_904_REGOFF)
+#define HWIO_DDR_PHY_904_PHY_ADR_PWR_RDC_DISABLE_3_FLDMASK (0x1)
+#define HWIO_DDR_PHY_904_PHY_ADR_PWR_RDC_DISABLE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_904_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_904_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_904_PHY_ADR_TYPE_3_FLDMASK (0x300)
+#define HWIO_DDR_PHY_904_PHY_ADR_TYPE_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_904_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_904_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_904_PHY_ADR_WRADDR_SHIFT_OBS_3_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_904_PHY_ADR_WRADDR_SHIFT_OBS_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_904_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_904_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_904_PHY_ADR_IE_MODE_3_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_904_PHY_ADR_IE_MODE_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_904_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_904_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_905_REGOFF 0xe24
+#define HWIO_DDR_PHY_905_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_905_REGOFF)
+#define HWIO_DDR_PHY_905_PHY_ADR_DDL_MODE_3_FLDMASK (0x7fff)
+#define HWIO_DDR_PHY_905_PHY_ADR_DDL_MODE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_905_RESERVED_FLDMASK (0x8000)
+#define HWIO_DDR_PHY_905_RESERVED_FLDSHFT (15)
+#define HWIO_DDR_PHY_905_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_905_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_906_REGOFF 0xe28
+#define HWIO_DDR_PHY_906_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_906_REGOFF)
+#define HWIO_DDR_PHY_906_PHY_ADR_DDL_TEST_OBS_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_906_PHY_ADR_DDL_TEST_OBS_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_907_REGOFF 0xe2c
+#define HWIO_DDR_PHY_907_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_907_REGOFF)
+#define HWIO_DDR_PHY_907_PHY_ADR_DDL_TEST_MSTR_DLY_OBS_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_907_PHY_ADR_DDL_TEST_MSTR_DLY_OBS_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_908_REGOFF 0xe30
+#define HWIO_DDR_PHY_908_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_908_REGOFF)
+#define HWIO_DDR_PHY_908_PHY_ADR_CALVL_START_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_908_PHY_ADR_CALVL_START_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_908_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_908_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_908_PHY_ADR_CALVL_COARSE_DLY_3_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_908_PHY_ADR_CALVL_COARSE_DLY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_908_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_908_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_909_REGOFF 0xe34
+#define HWIO_DDR_PHY_909_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_909_REGOFF)
+#define HWIO_DDR_PHY_909_PHY_ADR_CALVL_QTR_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_909_PHY_ADR_CALVL_QTR_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_909_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_909_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_909_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_909_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_910_REGOFF 0xe38
+#define HWIO_DDR_PHY_910_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_910_REGOFF)
+#define HWIO_DDR_PHY_910_PHY_ADR_CALVL_SWIZZLE0_3_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_910_PHY_ADR_CALVL_SWIZZLE0_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_910_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_910_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_911_REGOFF 0xe3c
+#define HWIO_DDR_PHY_911_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_911_REGOFF)
+#define HWIO_DDR_PHY_911_PHY_ADR_CALVL_SWIZZLE1_3_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_911_PHY_ADR_CALVL_SWIZZLE1_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_911_PHY_ADR_CALVL_RANK_CTRL_3_FLDMASK (0x3000000)
+#define HWIO_DDR_PHY_911_PHY_ADR_CALVL_RANK_CTRL_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_911_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_911_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_912_REGOFF 0xe40
+#define HWIO_DDR_PHY_912_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_912_REGOFF)
+#define HWIO_DDR_PHY_912_PHY_ADR_CALVL_NUM_PATTERNS_3_FLDMASK (0x3)
+#define HWIO_DDR_PHY_912_PHY_ADR_CALVL_NUM_PATTERNS_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_912_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_912_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_912_PHY_ADR_CALVL_CAPTURE_CNT_3_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_912_PHY_ADR_CALVL_CAPTURE_CNT_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_912_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_912_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_912_PHY_ADR_CALVL_RESP_WAIT_CNT_3_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_912_PHY_ADR_CALVL_RESP_WAIT_CNT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_912_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_912_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PHY_912_PHY_ADR_CALVL_DEBUG_MODE_3_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_912_PHY_ADR_CALVL_DEBUG_MODE_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_912_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_912_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_913_REGOFF 0xe44
+#define HWIO_DDR_PHY_913_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_913_REGOFF)
+#define HWIO_DDR_PHY_913_SC_PHY_ADR_CALVL_DEBUG_CONT_3_FLDMASK (0x1)
+#define HWIO_DDR_PHY_913_SC_PHY_ADR_CALVL_DEBUG_CONT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_913_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_913_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_913_SC_PHY_ADR_CALVL_ERROR_CLR_3_FLDMASK (0x100)
+#define HWIO_DDR_PHY_913_SC_PHY_ADR_CALVL_ERROR_CLR_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_913_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_913_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_913_PHY_ADR_CALVL_OBS_SELECT_3_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_913_PHY_ADR_CALVL_OBS_SELECT_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_913_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_913_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PHY_913_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_913_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_914_REGOFF 0xe48
+#define HWIO_DDR_PHY_914_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_914_REGOFF)
+#define HWIO_DDR_PHY_914_PHY_ADR_CALVL_OBS0_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_914_PHY_ADR_CALVL_OBS0_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_915_REGOFF 0xe4c
+#define HWIO_DDR_PHY_915_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_915_REGOFF)
+#define HWIO_DDR_PHY_915_PHY_ADR_CALVL_OBS1_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_915_PHY_ADR_CALVL_OBS1_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_916_REGOFF 0xe50
+#define HWIO_DDR_PHY_916_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_916_REGOFF)
+#define HWIO_DDR_PHY_916_PHY_ADR_CALVL_FG_0_3_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_916_PHY_ADR_CALVL_FG_0_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_916_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_916_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_916_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_916_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_917_REGOFF 0xe54
+#define HWIO_DDR_PHY_917_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_917_REGOFF)
+#define HWIO_DDR_PHY_917_PHY_ADR_CALVL_BG_0_3_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_917_PHY_ADR_CALVL_BG_0_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_917_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_917_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_917_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_917_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_918_REGOFF 0xe58
+#define HWIO_DDR_PHY_918_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_918_REGOFF)
+#define HWIO_DDR_PHY_918_PHY_ADR_CALVL_FG_1_3_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_918_PHY_ADR_CALVL_FG_1_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_918_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_918_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_918_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_918_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_919_REGOFF 0xe5c
+#define HWIO_DDR_PHY_919_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_919_REGOFF)
+#define HWIO_DDR_PHY_919_PHY_ADR_CALVL_BG_1_3_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_919_PHY_ADR_CALVL_BG_1_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_919_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_919_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_919_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_919_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_920_REGOFF 0xe60
+#define HWIO_DDR_PHY_920_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_920_REGOFF)
+#define HWIO_DDR_PHY_920_PHY_ADR_CALVL_FG_2_3_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_920_PHY_ADR_CALVL_FG_2_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_920_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_920_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_920_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_920_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_921_REGOFF 0xe64
+#define HWIO_DDR_PHY_921_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_921_REGOFF)
+#define HWIO_DDR_PHY_921_PHY_ADR_CALVL_BG_2_3_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_921_PHY_ADR_CALVL_BG_2_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_921_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_921_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_921_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_921_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_922_REGOFF 0xe68
+#define HWIO_DDR_PHY_922_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_922_REGOFF)
+#define HWIO_DDR_PHY_922_PHY_ADR_CALVL_FG_3_3_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_922_PHY_ADR_CALVL_FG_3_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_922_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_922_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_922_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_922_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_923_REGOFF 0xe6c
+#define HWIO_DDR_PHY_923_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_923_REGOFF)
+#define HWIO_DDR_PHY_923_PHY_ADR_CALVL_BG_3_3_FLDMASK (0xfffff)
+#define HWIO_DDR_PHY_923_PHY_ADR_CALVL_BG_3_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_923_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_923_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PHY_923_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_923_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_924_REGOFF 0xe70
+#define HWIO_DDR_PHY_924_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_924_REGOFF)
+#define HWIO_DDR_PHY_924_PHY_ADR_ADDR_SEL_3_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_924_PHY_ADR_ADDR_SEL_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_924_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_924_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_925_REGOFF 0xe74
+#define HWIO_DDR_PHY_925_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_925_REGOFF)
+#define HWIO_DDR_PHY_925_PHY_ADR_LP4_BOOT_SLV_DELAY_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_925_PHY_ADR_LP4_BOOT_SLV_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_925_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_925_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_925_PHY_ADR_BIT_MASK_3_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_925_PHY_ADR_BIT_MASK_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_925_RESERVED1_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_925_RESERVED1_FLDSHFT (22)
+#define HWIO_DDR_PHY_925_PHY_ADR_SEG_MASK_3_FLDMASK (0x3f000000)
+#define HWIO_DDR_PHY_925_PHY_ADR_SEG_MASK_3_FLDSHFT (24)
+#define HWIO_DDR_PHY_925_RESERVED2_FLDMASK (0xc0000000)
+#define HWIO_DDR_PHY_925_RESERVED2_FLDSHFT (30)
+
+#define HWIO_DDR_PHY_926_REGOFF 0xe78
+#define HWIO_DDR_PHY_926_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_926_REGOFF)
+#define HWIO_DDR_PHY_926_PHY_ADR_CALVL_TRAIN_MASK_3_FLDMASK (0x3f)
+#define HWIO_DDR_PHY_926_PHY_ADR_CALVL_TRAIN_MASK_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_926_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PHY_926_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PHY_926_PHY_ADR_SW_TXIO_CTRL_3_FLDMASK (0x3f00)
+#define HWIO_DDR_PHY_926_PHY_ADR_SW_TXIO_CTRL_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_926_RESERVED1_FLDMASK (0xc000)
+#define HWIO_DDR_PHY_926_RESERVED1_FLDSHFT (14)
+#define HWIO_DDR_PHY_926_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_926_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_927_REGOFF 0xe7c
+#define HWIO_DDR_PHY_927_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_927_REGOFF)
+#define HWIO_DDR_PHY_927_PHY_ADR_TSEL_SELECT_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_927_PHY_ADR_TSEL_SELECT_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_927_PHY_ADR0_CLK_WR_SLAVE_DELAY_3_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_927_PHY_ADR0_CLK_WR_SLAVE_DELAY_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_927_RESERVED_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_927_RESERVED_FLDSHFT (19)
+#define HWIO_DDR_PHY_927_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_927_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_928_REGOFF 0xe80
+#define HWIO_DDR_PHY_928_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_928_REGOFF)
+#define HWIO_DDR_PHY_928_PHY_ADR1_CLK_WR_SLAVE_DELAY_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_928_PHY_ADR1_CLK_WR_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_928_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_928_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_928_PHY_ADR2_CLK_WR_SLAVE_DELAY_3_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_928_PHY_ADR2_CLK_WR_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_928_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_928_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_929_REGOFF 0xe84
+#define HWIO_DDR_PHY_929_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_929_REGOFF)
+#define HWIO_DDR_PHY_929_PHY_ADR3_CLK_WR_SLAVE_DELAY_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_929_PHY_ADR3_CLK_WR_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_929_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_929_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_929_PHY_ADR4_CLK_WR_SLAVE_DELAY_3_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_929_PHY_ADR4_CLK_WR_SLAVE_DELAY_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_929_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_929_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_930_REGOFF 0xe88
+#define HWIO_DDR_PHY_930_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_930_REGOFF)
+#define HWIO_DDR_PHY_930_PHY_ADR5_CLK_WR_SLAVE_DELAY_3_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_930_PHY_ADR5_CLK_WR_SLAVE_DELAY_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_930_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_930_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_930_PHY_ADR_SW_MASTER_MODE_3_FLDMASK (0xf0000)
+#define HWIO_DDR_PHY_930_PHY_ADR_SW_MASTER_MODE_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_930_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_930_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PHY_930_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_930_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_931_REGOFF 0xe8c
+#define HWIO_DDR_PHY_931_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_931_REGOFF)
+#define HWIO_DDR_PHY_931_PHY_ADR_MASTER_DELAY_START_3_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_931_PHY_ADR_MASTER_DELAY_START_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_931_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_931_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_931_PHY_ADR_MASTER_DELAY_STEP_3_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_931_PHY_ADR_MASTER_DELAY_STEP_3_FLDSHFT (16)
+#define HWIO_DDR_PHY_931_RESERVED1_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_931_RESERVED1_FLDSHFT (22)
+#define HWIO_DDR_PHY_931_PHY_ADR_MASTER_DELAY_WAIT_3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_931_PHY_ADR_MASTER_DELAY_WAIT_3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_932_REGOFF 0xe90
+#define HWIO_DDR_PHY_932_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_932_REGOFF)
+#define HWIO_DDR_PHY_932_PHY_ADR_MASTER_DELAY_HALF_MEASURE_3_FLDMASK (0xff)
+#define HWIO_DDR_PHY_932_PHY_ADR_MASTER_DELAY_HALF_MEASURE_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_932_PHY_ADR_CALVL_DLY_STEP_3_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_932_PHY_ADR_CALVL_DLY_STEP_3_FLDSHFT (8)
+#define HWIO_DDR_PHY_932_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_932_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_932_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_932_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_933_REGOFF 0xe94
+#define HWIO_DDR_PHY_933_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_933_REGOFF)
+#define HWIO_DDR_PHY_933_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_933_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_934_REGOFF 0xe98
+#define HWIO_DDR_PHY_934_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_934_REGOFF)
+#define HWIO_DDR_PHY_934_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_934_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_935_REGOFF 0xe9c
+#define HWIO_DDR_PHY_935_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_935_REGOFF)
+#define HWIO_DDR_PHY_935_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_935_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_936_REGOFF 0xea0
+#define HWIO_DDR_PHY_936_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_936_REGOFF)
+#define HWIO_DDR_PHY_936_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_936_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_937_REGOFF 0xea4
+#define HWIO_DDR_PHY_937_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_937_REGOFF)
+#define HWIO_DDR_PHY_937_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_937_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_938_REGOFF 0xea8
+#define HWIO_DDR_PHY_938_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_938_REGOFF)
+#define HWIO_DDR_PHY_938_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_938_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_939_REGOFF 0xeac
+#define HWIO_DDR_PHY_939_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_939_REGOFF)
+#define HWIO_DDR_PHY_939_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_939_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_940_REGOFF 0xeb0
+#define HWIO_DDR_PHY_940_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_940_REGOFF)
+#define HWIO_DDR_PHY_940_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_940_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_941_REGOFF 0xeb4
+#define HWIO_DDR_PHY_941_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_941_REGOFF)
+#define HWIO_DDR_PHY_941_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_941_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_942_REGOFF 0xeb8
+#define HWIO_DDR_PHY_942_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_942_REGOFF)
+#define HWIO_DDR_PHY_942_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_942_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_943_REGOFF 0xebc
+#define HWIO_DDR_PHY_943_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_943_REGOFF)
+#define HWIO_DDR_PHY_943_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_943_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_944_REGOFF 0xec0
+#define HWIO_DDR_PHY_944_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_944_REGOFF)
+#define HWIO_DDR_PHY_944_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_944_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_945_REGOFF 0xec4
+#define HWIO_DDR_PHY_945_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_945_REGOFF)
+#define HWIO_DDR_PHY_945_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_945_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_946_REGOFF 0xec8
+#define HWIO_DDR_PHY_946_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_946_REGOFF)
+#define HWIO_DDR_PHY_946_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_946_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_947_REGOFF 0xecc
+#define HWIO_DDR_PHY_947_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_947_REGOFF)
+#define HWIO_DDR_PHY_947_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_947_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_948_REGOFF 0xed0
+#define HWIO_DDR_PHY_948_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_948_REGOFF)
+#define HWIO_DDR_PHY_948_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_948_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_949_REGOFF 0xed4
+#define HWIO_DDR_PHY_949_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_949_REGOFF)
+#define HWIO_DDR_PHY_949_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_949_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_950_REGOFF 0xed8
+#define HWIO_DDR_PHY_950_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_950_REGOFF)
+#define HWIO_DDR_PHY_950_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_950_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_951_REGOFF 0xedc
+#define HWIO_DDR_PHY_951_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_951_REGOFF)
+#define HWIO_DDR_PHY_951_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_951_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_952_REGOFF 0xee0
+#define HWIO_DDR_PHY_952_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_952_REGOFF)
+#define HWIO_DDR_PHY_952_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_952_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_953_REGOFF 0xee4
+#define HWIO_DDR_PHY_953_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_953_REGOFF)
+#define HWIO_DDR_PHY_953_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_953_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_954_REGOFF 0xee8
+#define HWIO_DDR_PHY_954_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_954_REGOFF)
+#define HWIO_DDR_PHY_954_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_954_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_955_REGOFF 0xeec
+#define HWIO_DDR_PHY_955_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_955_REGOFF)
+#define HWIO_DDR_PHY_955_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_955_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_956_REGOFF 0xef0
+#define HWIO_DDR_PHY_956_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_956_REGOFF)
+#define HWIO_DDR_PHY_956_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_956_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_957_REGOFF 0xef4
+#define HWIO_DDR_PHY_957_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_957_REGOFF)
+#define HWIO_DDR_PHY_957_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_957_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_958_REGOFF 0xef8
+#define HWIO_DDR_PHY_958_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_958_REGOFF)
+#define HWIO_DDR_PHY_958_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_958_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_959_REGOFF 0xefc
+#define HWIO_DDR_PHY_959_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_959_REGOFF)
+#define HWIO_DDR_PHY_959_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_959_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_960_REGOFF 0xf00
+#define HWIO_DDR_PHY_960_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_960_REGOFF)
+#define HWIO_DDR_PHY_960_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_960_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_961_REGOFF 0xf04
+#define HWIO_DDR_PHY_961_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_961_REGOFF)
+#define HWIO_DDR_PHY_961_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_961_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_962_REGOFF 0xf08
+#define HWIO_DDR_PHY_962_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_962_REGOFF)
+#define HWIO_DDR_PHY_962_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_962_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_963_REGOFF 0xf0c
+#define HWIO_DDR_PHY_963_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_963_REGOFF)
+#define HWIO_DDR_PHY_963_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_963_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_964_REGOFF 0xf10
+#define HWIO_DDR_PHY_964_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_964_REGOFF)
+#define HWIO_DDR_PHY_964_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_964_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_965_REGOFF 0xf14
+#define HWIO_DDR_PHY_965_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_965_REGOFF)
+#define HWIO_DDR_PHY_965_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_965_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_966_REGOFF 0xf18
+#define HWIO_DDR_PHY_966_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_966_REGOFF)
+#define HWIO_DDR_PHY_966_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_966_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_967_REGOFF 0xf1c
+#define HWIO_DDR_PHY_967_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_967_REGOFF)
+#define HWIO_DDR_PHY_967_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_967_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_968_REGOFF 0xf20
+#define HWIO_DDR_PHY_968_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_968_REGOFF)
+#define HWIO_DDR_PHY_968_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_968_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_969_REGOFF 0xf24
+#define HWIO_DDR_PHY_969_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_969_REGOFF)
+#define HWIO_DDR_PHY_969_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_969_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_970_REGOFF 0xf28
+#define HWIO_DDR_PHY_970_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_970_REGOFF)
+#define HWIO_DDR_PHY_970_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_970_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_971_REGOFF 0xf2c
+#define HWIO_DDR_PHY_971_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_971_REGOFF)
+#define HWIO_DDR_PHY_971_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_971_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_972_REGOFF 0xf30
+#define HWIO_DDR_PHY_972_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_972_REGOFF)
+#define HWIO_DDR_PHY_972_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_972_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_973_REGOFF 0xf34
+#define HWIO_DDR_PHY_973_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_973_REGOFF)
+#define HWIO_DDR_PHY_973_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_973_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_974_REGOFF 0xf38
+#define HWIO_DDR_PHY_974_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_974_REGOFF)
+#define HWIO_DDR_PHY_974_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_974_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_975_REGOFF 0xf3c
+#define HWIO_DDR_PHY_975_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_975_REGOFF)
+#define HWIO_DDR_PHY_975_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_975_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_976_REGOFF 0xf40
+#define HWIO_DDR_PHY_976_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_976_REGOFF)
+#define HWIO_DDR_PHY_976_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_976_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_977_REGOFF 0xf44
+#define HWIO_DDR_PHY_977_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_977_REGOFF)
+#define HWIO_DDR_PHY_977_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_977_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_978_REGOFF 0xf48
+#define HWIO_DDR_PHY_978_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_978_REGOFF)
+#define HWIO_DDR_PHY_978_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_978_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_979_REGOFF 0xf4c
+#define HWIO_DDR_PHY_979_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_979_REGOFF)
+#define HWIO_DDR_PHY_979_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_979_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_980_REGOFF 0xf50
+#define HWIO_DDR_PHY_980_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_980_REGOFF)
+#define HWIO_DDR_PHY_980_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_980_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_981_REGOFF 0xf54
+#define HWIO_DDR_PHY_981_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_981_REGOFF)
+#define HWIO_DDR_PHY_981_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_981_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_982_REGOFF 0xf58
+#define HWIO_DDR_PHY_982_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_982_REGOFF)
+#define HWIO_DDR_PHY_982_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_982_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_983_REGOFF 0xf5c
+#define HWIO_DDR_PHY_983_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_983_REGOFF)
+#define HWIO_DDR_PHY_983_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_983_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_984_REGOFF 0xf60
+#define HWIO_DDR_PHY_984_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_984_REGOFF)
+#define HWIO_DDR_PHY_984_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_984_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_985_REGOFF 0xf64
+#define HWIO_DDR_PHY_985_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_985_REGOFF)
+#define HWIO_DDR_PHY_985_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_985_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_986_REGOFF 0xf68
+#define HWIO_DDR_PHY_986_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_986_REGOFF)
+#define HWIO_DDR_PHY_986_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_986_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_987_REGOFF 0xf6c
+#define HWIO_DDR_PHY_987_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_987_REGOFF)
+#define HWIO_DDR_PHY_987_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_987_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_988_REGOFF 0xf70
+#define HWIO_DDR_PHY_988_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_988_REGOFF)
+#define HWIO_DDR_PHY_988_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_988_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_989_REGOFF 0xf74
+#define HWIO_DDR_PHY_989_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_989_REGOFF)
+#define HWIO_DDR_PHY_989_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_989_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_990_REGOFF 0xf78
+#define HWIO_DDR_PHY_990_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_990_REGOFF)
+#define HWIO_DDR_PHY_990_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_990_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_991_REGOFF 0xf7c
+#define HWIO_DDR_PHY_991_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_991_REGOFF)
+#define HWIO_DDR_PHY_991_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_991_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_992_REGOFF 0xf80
+#define HWIO_DDR_PHY_992_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_992_REGOFF)
+#define HWIO_DDR_PHY_992_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_992_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_993_REGOFF 0xf84
+#define HWIO_DDR_PHY_993_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_993_REGOFF)
+#define HWIO_DDR_PHY_993_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_993_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_994_REGOFF 0xf88
+#define HWIO_DDR_PHY_994_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_994_REGOFF)
+#define HWIO_DDR_PHY_994_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_994_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_995_REGOFF 0xf8c
+#define HWIO_DDR_PHY_995_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_995_REGOFF)
+#define HWIO_DDR_PHY_995_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_995_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_996_REGOFF 0xf90
+#define HWIO_DDR_PHY_996_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_996_REGOFF)
+#define HWIO_DDR_PHY_996_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_996_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_997_REGOFF 0xf94
+#define HWIO_DDR_PHY_997_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_997_REGOFF)
+#define HWIO_DDR_PHY_997_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_997_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_998_REGOFF 0xf98
+#define HWIO_DDR_PHY_998_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_998_REGOFF)
+#define HWIO_DDR_PHY_998_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_998_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_999_REGOFF 0xf9c
+#define HWIO_DDR_PHY_999_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_999_REGOFF)
+#define HWIO_DDR_PHY_999_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_999_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1000_REGOFF 0xfa0
+#define HWIO_DDR_PHY_1000_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1000_REGOFF)
+#define HWIO_DDR_PHY_1000_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1000_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1001_REGOFF 0xfa4
+#define HWIO_DDR_PHY_1001_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1001_REGOFF)
+#define HWIO_DDR_PHY_1001_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1001_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1002_REGOFF 0xfa8
+#define HWIO_DDR_PHY_1002_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1002_REGOFF)
+#define HWIO_DDR_PHY_1002_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1002_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1003_REGOFF 0xfac
+#define HWIO_DDR_PHY_1003_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1003_REGOFF)
+#define HWIO_DDR_PHY_1003_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1003_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1004_REGOFF 0xfb0
+#define HWIO_DDR_PHY_1004_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1004_REGOFF)
+#define HWIO_DDR_PHY_1004_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1004_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1005_REGOFF 0xfb4
+#define HWIO_DDR_PHY_1005_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1005_REGOFF)
+#define HWIO_DDR_PHY_1005_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1005_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1006_REGOFF 0xfb8
+#define HWIO_DDR_PHY_1006_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1006_REGOFF)
+#define HWIO_DDR_PHY_1006_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1006_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1007_REGOFF 0xfbc
+#define HWIO_DDR_PHY_1007_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1007_REGOFF)
+#define HWIO_DDR_PHY_1007_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1007_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1008_REGOFF 0xfc0
+#define HWIO_DDR_PHY_1008_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1008_REGOFF)
+#define HWIO_DDR_PHY_1008_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1008_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1009_REGOFF 0xfc4
+#define HWIO_DDR_PHY_1009_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1009_REGOFF)
+#define HWIO_DDR_PHY_1009_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1009_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1010_REGOFF 0xfc8
+#define HWIO_DDR_PHY_1010_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1010_REGOFF)
+#define HWIO_DDR_PHY_1010_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1010_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1011_REGOFF 0xfcc
+#define HWIO_DDR_PHY_1011_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1011_REGOFF)
+#define HWIO_DDR_PHY_1011_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1011_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1012_REGOFF 0xfd0
+#define HWIO_DDR_PHY_1012_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1012_REGOFF)
+#define HWIO_DDR_PHY_1012_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1012_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1013_REGOFF 0xfd4
+#define HWIO_DDR_PHY_1013_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1013_REGOFF)
+#define HWIO_DDR_PHY_1013_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1013_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1014_REGOFF 0xfd8
+#define HWIO_DDR_PHY_1014_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1014_REGOFF)
+#define HWIO_DDR_PHY_1014_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1014_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1015_REGOFF 0xfdc
+#define HWIO_DDR_PHY_1015_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1015_REGOFF)
+#define HWIO_DDR_PHY_1015_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1015_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1016_REGOFF 0xfe0
+#define HWIO_DDR_PHY_1016_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1016_REGOFF)
+#define HWIO_DDR_PHY_1016_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1016_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1017_REGOFF 0xfe4
+#define HWIO_DDR_PHY_1017_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1017_REGOFF)
+#define HWIO_DDR_PHY_1017_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1017_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1018_REGOFF 0xfe8
+#define HWIO_DDR_PHY_1018_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1018_REGOFF)
+#define HWIO_DDR_PHY_1018_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1018_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1019_REGOFF 0xfec
+#define HWIO_DDR_PHY_1019_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1019_REGOFF)
+#define HWIO_DDR_PHY_1019_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1019_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1020_REGOFF 0xff0
+#define HWIO_DDR_PHY_1020_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1020_REGOFF)
+#define HWIO_DDR_PHY_1020_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1020_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1021_REGOFF 0xff4
+#define HWIO_DDR_PHY_1021_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1021_REGOFF)
+#define HWIO_DDR_PHY_1021_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1021_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1022_REGOFF 0xff8
+#define HWIO_DDR_PHY_1022_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1022_REGOFF)
+#define HWIO_DDR_PHY_1022_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1022_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1023_REGOFF 0xffc
+#define HWIO_DDR_PHY_1023_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1023_REGOFF)
+#define HWIO_DDR_PHY_1023_OBSOLETE0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1023_OBSOLETE0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1024_REGOFF 0x1000
+#define HWIO_DDR_PHY_1024_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1024_REGOFF)
+#define HWIO_DDR_PHY_1024_PHY_FREQ_SEL_FLDMASK (0x7)
+#define HWIO_DDR_PHY_1024_PHY_FREQ_SEL_FLDSHFT (0)
+#define HWIO_DDR_PHY_1024_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PHY_1024_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PHY_1024_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_1024_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_1025_REGOFF 0x1004
+#define HWIO_DDR_PHY_1025_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1025_REGOFF)
+#define HWIO_DDR_PHY_1025_PHY_FREQ_SEL_FROM_REGIF_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1025_PHY_FREQ_SEL_FROM_REGIF_FLDSHFT (0)
+#define HWIO_DDR_PHY_1025_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1025_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1025_PHY_FREQ_SEL_MULTICAST_EN_FLDMASK (0x100)
+#define HWIO_DDR_PHY_1025_PHY_FREQ_SEL_MULTICAST_EN_FLDSHFT (8)
+#define HWIO_DDR_PHY_1025_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_1025_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_1025_PHY_FREQ_SEL_INDEX_FLDMASK (0x30000)
+#define HWIO_DDR_PHY_1025_PHY_FREQ_SEL_INDEX_FLDSHFT (16)
+#define HWIO_DDR_PHY_1025_RESERVED2_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_1025_RESERVED2_FLDSHFT (18)
+#define HWIO_DDR_PHY_1025_PHY_SW_GRP_SHIFT_0_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_1025_PHY_SW_GRP_SHIFT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_1025_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1025_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1026_REGOFF 0x1008
+#define HWIO_DDR_PHY_1026_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1026_REGOFF)
+#define HWIO_DDR_PHY_1026_PHY_SW_GRP_SHIFT_1_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_1026_PHY_SW_GRP_SHIFT_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_1026_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_1026_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_1026_PHY_GRP_BYPASS_SLAVE_DELAY_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_1026_PHY_GRP_BYPASS_SLAVE_DELAY_FLDSHFT (8)
+#define HWIO_DDR_PHY_1026_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_1026_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_1026_PHY_SW_GRP_BYPASS_SHIFT_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_1026_PHY_SW_GRP_BYPASS_SHIFT_FLDSHFT (24)
+#define HWIO_DDR_PHY_1026_RESERVED2_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1026_RESERVED2_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1027_REGOFF 0x100c
+#define HWIO_DDR_PHY_1027_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1027_REGOFF)
+#define HWIO_DDR_PHY_1027_PHY_GRP_BYPASS_OVERRIDE_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1027_PHY_GRP_BYPASS_OVERRIDE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1027_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1027_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1027_SC_PHY_MANUAL_UPDATE_FLDMASK (0x100)
+#define HWIO_DDR_PHY_1027_SC_PHY_MANUAL_UPDATE_FLDSHFT (8)
+#define HWIO_DDR_PHY_1027_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_1027_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_1027_PHY_LP4_BOOT_DISABLE_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1027_PHY_LP4_BOOT_DISABLE_FLDSHFT (16)
+#define HWIO_DDR_PHY_1027_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1027_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PHY_1027_PHY_CSLVL_ENABLE_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_1027_PHY_CSLVL_ENABLE_FLDSHFT (24)
+#define HWIO_DDR_PHY_1027_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_1027_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_1028_REGOFF 0x1010
+#define HWIO_DDR_PHY_1028_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1028_REGOFF)
+#define HWIO_DDR_PHY_1028_PHY_CSLVL_CS_MAP_FLDMASK (0x3)
+#define HWIO_DDR_PHY_1028_PHY_CSLVL_CS_MAP_FLDSHFT (0)
+#define HWIO_DDR_PHY_1028_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PHY_1028_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PHY_1028_PHY_CSLVL_START_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_1028_PHY_CSLVL_START_FLDSHFT (8)
+#define HWIO_DDR_PHY_1028_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_1028_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_1028_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1028_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1029_REGOFF 0x1014
+#define HWIO_DDR_PHY_1029_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1029_REGOFF)
+#define HWIO_DDR_PHY_1029_PHY_CSLVL_QTR_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_1029_PHY_CSLVL_QTR_FLDSHFT (0)
+#define HWIO_DDR_PHY_1029_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_1029_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_1029_PHY_CSLVL_COARSE_CHK_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_1029_PHY_CSLVL_COARSE_CHK_FLDSHFT (16)
+#define HWIO_DDR_PHY_1029_RESERVED1_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_1029_RESERVED1_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_1030_REGOFF 0x1018
+#define HWIO_DDR_PHY_1030_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1030_REGOFF)
+#define HWIO_DDR_PHY_1030_PHY_CSLVL_CAPTURE_CNT_FLDMASK (0xf)
+#define HWIO_DDR_PHY_1030_PHY_CSLVL_CAPTURE_CNT_FLDSHFT (0)
+#define HWIO_DDR_PHY_1030_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_1030_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_1030_PHY_CSLVL_COARSE_DLY_FLDMASK (0x7ff00)
+#define HWIO_DDR_PHY_1030_PHY_CSLVL_COARSE_DLY_FLDSHFT (8)
+#define HWIO_DDR_PHY_1030_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_1030_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_1030_PHY_CSLVL_COARSE_CAPTURE_CNT_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_1030_PHY_CSLVL_COARSE_CAPTURE_CNT_FLDSHFT (24)
+#define HWIO_DDR_PHY_1030_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_1030_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_1031_REGOFF 0x101c
+#define HWIO_DDR_PHY_1031_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1031_REGOFF)
+#define HWIO_DDR_PHY_1031_PHY_CSLVL_DEBUG_MODE_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1031_PHY_CSLVL_DEBUG_MODE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1031_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1031_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1031_SC_PHY_CSLVL_DEBUG_CONT_FLDMASK (0x100)
+#define HWIO_DDR_PHY_1031_SC_PHY_CSLVL_DEBUG_CONT_FLDSHFT (8)
+#define HWIO_DDR_PHY_1031_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_1031_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_1031_SC_PHY_CSLVL_ERROR_CLR_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1031_SC_PHY_CSLVL_ERROR_CLR_FLDSHFT (16)
+#define HWIO_DDR_PHY_1031_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1031_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PHY_1031_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1031_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1032_REGOFF 0x1020
+#define HWIO_DDR_PHY_1032_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1032_REGOFF)
+#define HWIO_DDR_PHY_1032_PHY_CSLVL_OBS0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1032_PHY_CSLVL_OBS0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1033_REGOFF 0x1024
+#define HWIO_DDR_PHY_1033_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1033_REGOFF)
+#define HWIO_DDR_PHY_1033_PHY_CSLVL_OBS1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1033_PHY_CSLVL_OBS1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1034_REGOFF 0x1028
+#define HWIO_DDR_PHY_1034_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1034_REGOFF)
+#define HWIO_DDR_PHY_1034_PHY_CALVL_CS_MAP_FLDMASK (0xf)
+#define HWIO_DDR_PHY_1034_PHY_CALVL_CS_MAP_FLDSHFT (0)
+#define HWIO_DDR_PHY_1034_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_1034_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_1034_PHY_GRP_SLV_DLY_ENC_OBS_SELECT_FLDMASK (0x3ff00)
+#define HWIO_DDR_PHY_1034_PHY_GRP_SLV_DLY_ENC_OBS_SELECT_FLDSHFT (8)
+#define HWIO_DDR_PHY_1034_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_1034_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PHY_1034_PHY_GRP_SHIFT_OBS_SELECT_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_1034_PHY_GRP_SHIFT_OBS_SELECT_FLDSHFT (24)
+#define HWIO_DDR_PHY_1034_RESERVED2_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1034_RESERVED2_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1035_REGOFF 0x102c
+#define HWIO_DDR_PHY_1035_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1035_REGOFF)
+#define HWIO_DDR_PHY_1035_PHY_GRP_SLV_DLY_ENC_OBS_FLDMASK (0x3ff)
+#define HWIO_DDR_PHY_1035_PHY_GRP_SLV_DLY_ENC_OBS_FLDSHFT (0)
+#define HWIO_DDR_PHY_1035_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_1035_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_1035_PHY_GRP_SHIFT_OBS_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_1035_PHY_GRP_SHIFT_OBS_FLDSHFT (16)
+#define HWIO_DDR_PHY_1035_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_1035_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_1035_PHY_ADRCTL_SLAVE_LOOP_CNT_UPDATE_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_1035_PHY_ADRCTL_SLAVE_LOOP_CNT_UPDATE_FLDSHFT (24)
+#define HWIO_DDR_PHY_1035_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_1035_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_1036_REGOFF 0x1030
+#define HWIO_DDR_PHY_1036_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1036_REGOFF)
+#define HWIO_DDR_PHY_1036_PHY_ADRCTL_SNAP_OBS_REGS_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1036_PHY_ADRCTL_SNAP_OBS_REGS_FLDSHFT (0)
+#define HWIO_DDR_PHY_1036_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1036_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1036_PHY_DFI_PHYUPD_TYPE_FLDMASK (0x300)
+#define HWIO_DDR_PHY_1036_PHY_DFI_PHYUPD_TYPE_FLDSHFT (8)
+#define HWIO_DDR_PHY_1036_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_1036_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_1036_PHY_ADRCTL_LPDDR_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1036_PHY_ADRCTL_LPDDR_FLDSHFT (16)
+#define HWIO_DDR_PHY_1036_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1036_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PHY_1036_PHY_LP4_ACTIVE_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_1036_PHY_LP4_ACTIVE_FLDSHFT (24)
+#define HWIO_DDR_PHY_1036_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_1036_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_1037_REGOFF 0x1034
+#define HWIO_DDR_PHY_1037_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1037_REGOFF)
+#define HWIO_DDR_PHY_1037_PHY_LPDDR3_CS_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1037_PHY_LPDDR3_CS_FLDSHFT (0)
+#define HWIO_DDR_PHY_1037_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1037_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1037_PHY_CALVL_RESULT_MASK_FLDMASK (0x300)
+#define HWIO_DDR_PHY_1037_PHY_CALVL_RESULT_MASK_FLDSHFT (8)
+#define HWIO_DDR_PHY_1037_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_1037_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_1037_SC_PHY_UPDATE_CLK_CAL_VALUES_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1037_SC_PHY_UPDATE_CLK_CAL_VALUES_FLDSHFT (16)
+#define HWIO_DDR_PHY_1037_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1037_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PHY_1037_PHY_CONTINUOUS_CLK_CAL_UPDATE_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_1037_PHY_CONTINUOUS_CLK_CAL_UPDATE_FLDSHFT (24)
+#define HWIO_DDR_PHY_1037_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_1037_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_1038_REGOFF 0x1038
+#define HWIO_DDR_PHY_1038_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1038_REGOFF)
+#define HWIO_DDR_PHY_1038_PHY_SW_TXIO_CTRL_0_FLDMASK (0xf)
+#define HWIO_DDR_PHY_1038_PHY_SW_TXIO_CTRL_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_1038_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_1038_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_1038_PHY_SW_TXIO_CTRL_1_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_1038_PHY_SW_TXIO_CTRL_1_FLDSHFT (8)
+#define HWIO_DDR_PHY_1038_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_1038_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_1038_PHY_MEMCLK_SW_TXIO_CTRL_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1038_PHY_MEMCLK_SW_TXIO_CTRL_FLDSHFT (16)
+#define HWIO_DDR_PHY_1038_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1038_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PHY_1038_PHY_CA_SW_TXPWR_CTRL_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_1038_PHY_CA_SW_TXPWR_CTRL_FLDSHFT (24)
+#define HWIO_DDR_PHY_1038_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_1038_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_1039_REGOFF 0x103c
+#define HWIO_DDR_PHY_1039_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1039_REGOFF)
+#define HWIO_DDR_PHY_1039_PHY_MEMCLK_SW_TXPWR_CTRL_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1039_PHY_MEMCLK_SW_TXPWR_CTRL_FLDSHFT (0)
+#define HWIO_DDR_PHY_1039_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1039_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1039_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_1039_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_1040_REGOFF 0x1040
+#define HWIO_DDR_PHY_1040_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1040_REGOFF)
+#define HWIO_DDR_PHY_1040_PHY_USER_DEF_REG_AC_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1040_PHY_USER_DEF_REG_AC_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1041_REGOFF 0x1044
+#define HWIO_DDR_PHY_1041_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1041_REGOFF)
+#define HWIO_DDR_PHY_1041_PHY_USER_DEF_REG_AC_1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1041_PHY_USER_DEF_REG_AC_1_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1042_REGOFF 0x1048
+#define HWIO_DDR_PHY_1042_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1042_REGOFF)
+#define HWIO_DDR_PHY_1042_PHY_USER_DEF_REG_AC_2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1042_PHY_USER_DEF_REG_AC_2_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1043_REGOFF 0x104c
+#define HWIO_DDR_PHY_1043_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1043_REGOFF)
+#define HWIO_DDR_PHY_1043_PHY_USER_DEF_REG_AC_3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1043_PHY_USER_DEF_REG_AC_3_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1044_REGOFF 0x1050
+#define HWIO_DDR_PHY_1044_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1044_REGOFF)
+#define HWIO_DDR_PHY_1044_PHY_PLL_WAIT_FLDMASK (0xff)
+#define HWIO_DDR_PHY_1044_PHY_PLL_WAIT_FLDSHFT (0)
+#define HWIO_DDR_PHY_1044_PHY_PLL_WAIT_TOP_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_1044_PHY_PLL_WAIT_TOP_FLDSHFT (8)
+#define HWIO_DDR_PHY_1044_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_1044_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_1045_REGOFF 0x1054
+#define HWIO_DDR_PHY_1045_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1045_REGOFF)
+#define HWIO_DDR_PHY_1045_PHY_PLL_CTRL_FLDMASK (0x1fff)
+#define HWIO_DDR_PHY_1045_PHY_PLL_CTRL_FLDSHFT (0)
+#define HWIO_DDR_PHY_1045_RESERVED_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_1045_RESERVED_FLDSHFT (13)
+#define HWIO_DDR_PHY_1045_PHY_PLL_CTRL_TOP_FLDMASK (0x1fff0000)
+#define HWIO_DDR_PHY_1045_PHY_PLL_CTRL_TOP_FLDSHFT (16)
+#define HWIO_DDR_PHY_1045_RESERVED1_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1045_RESERVED1_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1046_REGOFF 0x1058
+#define HWIO_DDR_PHY_1046_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1046_REGOFF)
+#define HWIO_DDR_PHY_1046_PHY_PLL_CTRL_CA_FLDMASK (0x1fff)
+#define HWIO_DDR_PHY_1046_PHY_PLL_CTRL_CA_FLDSHFT (0)
+#define HWIO_DDR_PHY_1046_RESERVED_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_1046_RESERVED_FLDSHFT (13)
+#define HWIO_DDR_PHY_1046_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_1046_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_1047_REGOFF 0x105c
+#define HWIO_DDR_PHY_1047_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1047_REGOFF)
+#define HWIO_DDR_PHY_1047_PHY_PLL_BYPASS_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_1047_PHY_PLL_BYPASS_FLDSHFT (0)
+#define HWIO_DDR_PHY_1047_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_1047_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_1047_OBSOLETE1_FLDMASK (0xffffff00)
+#define HWIO_DDR_PHY_1047_OBSOLETE1_FLDSHFT (8)
+
+#define HWIO_DDR_PHY_1048_REGOFF 0x1060
+#define HWIO_DDR_PHY_1048_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1048_REGOFF)
+#define HWIO_DDR_PHY_1048_PHY_LOW_FREQ_SEL_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1048_PHY_LOW_FREQ_SEL_FLDSHFT (0)
+#define HWIO_DDR_PHY_1048_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1048_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1048_PHY_PAD_VREF_CTRL_DQ_0_FLDMASK (0xfff00)
+#define HWIO_DDR_PHY_1048_PHY_PAD_VREF_CTRL_DQ_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_1048_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PHY_1048_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PHY_1048_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1048_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1049_REGOFF 0x1064
+#define HWIO_DDR_PHY_1049_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1049_REGOFF)
+#define HWIO_DDR_PHY_1049_PHY_PAD_VREF_CTRL_DQ_1_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_1049_PHY_PAD_VREF_CTRL_DQ_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_1049_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_1049_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_1049_PHY_PAD_VREF_CTRL_DQ_2_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_1049_PHY_PAD_VREF_CTRL_DQ_2_FLDSHFT (16)
+#define HWIO_DDR_PHY_1049_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_1049_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_1050_REGOFF 0x1068
+#define HWIO_DDR_PHY_1050_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1050_REGOFF)
+#define HWIO_DDR_PHY_1050_PHY_PAD_VREF_CTRL_DQ_3_FLDMASK (0xfff)
+#define HWIO_DDR_PHY_1050_PHY_PAD_VREF_CTRL_DQ_3_FLDSHFT (0)
+#define HWIO_DDR_PHY_1050_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_1050_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PHY_1050_PHY_PAD_VREF_CTRL_AC_FLDMASK (0xfff0000)
+#define HWIO_DDR_PHY_1050_PHY_PAD_VREF_CTRL_AC_FLDSHFT (16)
+#define HWIO_DDR_PHY_1050_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_1050_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_1051_REGOFF 0x106c
+#define HWIO_DDR_PHY_1051_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1051_REGOFF)
+#define HWIO_DDR_PHY_1051_PHY_CSLVL_DLY_STEP_FLDMASK (0xf)
+#define HWIO_DDR_PHY_1051_PHY_CSLVL_DLY_STEP_FLDSHFT (0)
+#define HWIO_DDR_PHY_1051_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_1051_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_1051_PHY_SET_DFI_INPUT_RST_PAD_FLDMASK (0x100)
+#define HWIO_DDR_PHY_1051_PHY_SET_DFI_INPUT_RST_PAD_FLDSHFT (8)
+#define HWIO_DDR_PHY_1051_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_1051_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_1051_PHY_GRP_SLAVE_DELAY_0_FLDMASK (0x7ff0000)
+#define HWIO_DDR_PHY_1051_PHY_GRP_SLAVE_DELAY_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_1051_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_1051_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_1052_REGOFF 0x1070
+#define HWIO_DDR_PHY_1052_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1052_REGOFF)
+#define HWIO_DDR_PHY_1052_PHY_GRP_SLAVE_DELAY_1_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_1052_PHY_GRP_SLAVE_DELAY_1_FLDSHFT (0)
+#define HWIO_DDR_PHY_1052_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_1052_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_1052_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_1052_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_1053_REGOFF 0x1074
+#define HWIO_DDR_PHY_1053_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1053_REGOFF)
+#define HWIO_DDR_PHY_1053_PHY_LP4_BOOT_PLL_CTRL_FLDMASK (0x1fff)
+#define HWIO_DDR_PHY_1053_PHY_LP4_BOOT_PLL_CTRL_FLDSHFT (0)
+#define HWIO_DDR_PHY_1053_RESERVED_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_1053_RESERVED_FLDSHFT (13)
+#define HWIO_DDR_PHY_1053_PHY_LP4_BOOT_PLL_CTRL_CA_FLDMASK (0x1fff0000)
+#define HWIO_DDR_PHY_1053_PHY_LP4_BOOT_PLL_CTRL_CA_FLDSHFT (16)
+#define HWIO_DDR_PHY_1053_RESERVED1_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1053_RESERVED1_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1054_REGOFF 0x1078
+#define HWIO_DDR_PHY_1054_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1054_REGOFF)
+#define HWIO_DDR_PHY_1054_PHY_LP4_BOOT_TOP_PLL_CTRL_FLDMASK (0x1fff)
+#define HWIO_DDR_PHY_1054_PHY_LP4_BOOT_TOP_PLL_CTRL_FLDSHFT (0)
+#define HWIO_DDR_PHY_1054_RESERVED_FLDMASK (0xe000)
+#define HWIO_DDR_PHY_1054_RESERVED_FLDSHFT (13)
+#define HWIO_DDR_PHY_1054_PHY_PLL_CTRL_OVERRIDE_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_1054_PHY_PLL_CTRL_OVERRIDE_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_1055_REGOFF 0x107c
+#define HWIO_DDR_PHY_1055_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1055_REGOFF)
+#define HWIO_DDR_PHY_1055_PHY_PLL_OBS_0_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_1055_PHY_PLL_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_1055_PHY_PLL_OBS_1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_1055_PHY_PLL_OBS_1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_1056_REGOFF 0x1080
+#define HWIO_DDR_PHY_1056_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1056_REGOFF)
+#define HWIO_DDR_PHY_1056_PHY_PLL_OBS_2_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_1056_PHY_PLL_OBS_2_FLDSHFT (0)
+#define HWIO_DDR_PHY_1056_PHY_PLL_OBS_3_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_1056_PHY_PLL_OBS_3_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_1057_REGOFF 0x1084
+#define HWIO_DDR_PHY_1057_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1057_REGOFF)
+#define HWIO_DDR_PHY_1057_PHY_PLL_OBS_4_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_1057_PHY_PLL_OBS_4_FLDSHFT (0)
+#define HWIO_DDR_PHY_1057_PHY_PLL_TESTOUT_SEL_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_1057_PHY_PLL_TESTOUT_SEL_FLDSHFT (16)
+#define HWIO_DDR_PHY_1057_RESERVED_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_1057_RESERVED_FLDSHFT (19)
+#define HWIO_DDR_PHY_1057_PHY_TCKSRE_WAIT_FLDMASK (0xf000000)
+#define HWIO_DDR_PHY_1057_PHY_TCKSRE_WAIT_FLDSHFT (24)
+#define HWIO_DDR_PHY_1057_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PHY_1057_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PHY_1058_REGOFF 0x1088
+#define HWIO_DDR_PHY_1058_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1058_REGOFF)
+#define HWIO_DDR_PHY_1058_PHY_LP4_BOOT_LOW_FREQ_SEL_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1058_PHY_LP4_BOOT_LOW_FREQ_SEL_FLDSHFT (0)
+#define HWIO_DDR_PHY_1058_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1058_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1058_PHY_LP_WAKEUP_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_1058_PHY_LP_WAKEUP_FLDSHFT (8)
+#define HWIO_DDR_PHY_1058_PHY_LS_IDLE_EN_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1058_PHY_LS_IDLE_EN_FLDSHFT (16)
+#define HWIO_DDR_PHY_1058_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1058_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_1058_PHY_LP_CTRLUPD_CNTR_CFG_FLDMASK (0x1f000000)
+#define HWIO_DDR_PHY_1058_PHY_LP_CTRLUPD_CNTR_CFG_FLDSHFT (24)
+#define HWIO_DDR_PHY_1058_RESERVED2_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1058_RESERVED2_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1059_REGOFF 0x108c
+#define HWIO_DDR_PHY_1059_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1059_REGOFF)
+#define HWIO_DDR_PHY_1059_PHY_TDFI_PHY_WRDELAY_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1059_PHY_TDFI_PHY_WRDELAY_FLDSHFT (0)
+#define HWIO_DDR_PHY_1059_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1059_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1059_PHY_PAD_FDBK_DRIVE_FLDMASK (0x7fffff00)
+#define HWIO_DDR_PHY_1059_PHY_PAD_FDBK_DRIVE_FLDSHFT (8)
+#define HWIO_DDR_PHY_1059_RESERVED1_FLDMASK (0x80000000)
+#define HWIO_DDR_PHY_1059_RESERVED1_FLDSHFT (31)
+
+#define HWIO_DDR_PHY_1060_REGOFF 0x1090
+#define HWIO_DDR_PHY_1060_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1060_REGOFF)
+#define HWIO_DDR_PHY_1060_PHY_PAD_FDBK_DRIVE2_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_1060_PHY_PAD_FDBK_DRIVE2_FLDSHFT (0)
+#define HWIO_DDR_PHY_1060_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_1060_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_1061_REGOFF 0x1094
+#define HWIO_DDR_PHY_1061_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1061_REGOFF)
+#define HWIO_DDR_PHY_1061_PHY_PAD_DATA_DRIVE_FLDMASK (0x1fffff)
+#define HWIO_DDR_PHY_1061_PHY_PAD_DATA_DRIVE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1061_RESERVED_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_1061_RESERVED_FLDSHFT (21)
+#define HWIO_DDR_PHY_1061_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1061_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1062_REGOFF 0x1098
+#define HWIO_DDR_PHY_1062_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1062_REGOFF)
+#define HWIO_DDR_PHY_1062_PHY_PAD_DQS_DRIVE_FLDMASK (0x7fffff)
+#define HWIO_DDR_PHY_1062_PHY_PAD_DQS_DRIVE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1062_RESERVED_FLDMASK (0x800000)
+#define HWIO_DDR_PHY_1062_RESERVED_FLDSHFT (23)
+#define HWIO_DDR_PHY_1062_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1062_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1063_REGOFF 0x109c
+#define HWIO_DDR_PHY_1063_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1063_REGOFF)
+#define HWIO_DDR_PHY_1063_PHY_PAD_ADDR_DRIVE_FLDMASK (0x1fffffff)
+#define HWIO_DDR_PHY_1063_PHY_PAD_ADDR_DRIVE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1063_RESERVED_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1063_RESERVED_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1064_REGOFF 0x10a0
+#define HWIO_DDR_PHY_1064_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1064_REGOFF)
+#define HWIO_DDR_PHY_1064_PHY_PAD_CLK_DRIVE_FLDMASK (0x7fffffff)
+#define HWIO_DDR_PHY_1064_PHY_PAD_CLK_DRIVE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1064_RESERVED_FLDMASK (0x80000000)
+#define HWIO_DDR_PHY_1064_RESERVED_FLDSHFT (31)
+
+#define HWIO_DDR_PHY_1065_REGOFF 0x10a4
+#define HWIO_DDR_PHY_1065_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1065_REGOFF)
+#define HWIO_DDR_PHY_1065_PHY_PAD_FDBK_TERM_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_1065_PHY_PAD_FDBK_TERM_FLDSHFT (0)
+#define HWIO_DDR_PHY_1065_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_1065_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_1065_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1065_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1066_REGOFF 0x10a8
+#define HWIO_DDR_PHY_1066_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1066_REGOFF)
+#define HWIO_DDR_PHY_1066_PHY_PAD_DATA_TERM_FLDMASK (0x1ffff)
+#define HWIO_DDR_PHY_1066_PHY_PAD_DATA_TERM_FLDSHFT (0)
+#define HWIO_DDR_PHY_1066_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1066_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_1066_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1066_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1067_REGOFF 0x10ac
+#define HWIO_DDR_PHY_1067_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1067_REGOFF)
+#define HWIO_DDR_PHY_1067_PHY_PAD_DQS_TERM_FLDMASK (0x1ffff)
+#define HWIO_DDR_PHY_1067_PHY_PAD_DQS_TERM_FLDSHFT (0)
+#define HWIO_DDR_PHY_1067_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1067_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_1067_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1067_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1068_REGOFF 0x10b0
+#define HWIO_DDR_PHY_1068_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1068_REGOFF)
+#define HWIO_DDR_PHY_1068_PHY_PAD_ADDR_TERM_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_1068_PHY_PAD_ADDR_TERM_FLDSHFT (0)
+#define HWIO_DDR_PHY_1068_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_1068_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_1068_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1068_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1069_REGOFF 0x10b4
+#define HWIO_DDR_PHY_1069_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1069_REGOFF)
+#define HWIO_DDR_PHY_1069_PHY_PAD_CLK_TERM_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_1069_PHY_PAD_CLK_TERM_FLDSHFT (0)
+#define HWIO_DDR_PHY_1069_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_1069_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_1069_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1069_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1070_REGOFF 0x10b8
+#define HWIO_DDR_PHY_1070_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1070_REGOFF)
+#define HWIO_DDR_PHY_1070_PHY_PAD_CKE_DRIVE_FLDMASK (0x1fffffff)
+#define HWIO_DDR_PHY_1070_PHY_PAD_CKE_DRIVE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1070_RESERVED_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1070_RESERVED_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1071_REGOFF 0x10bc
+#define HWIO_DDR_PHY_1071_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1071_REGOFF)
+#define HWIO_DDR_PHY_1071_PHY_PAD_CKE_TERM_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_1071_PHY_PAD_CKE_TERM_FLDSHFT (0)
+#define HWIO_DDR_PHY_1071_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_1071_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_1071_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1071_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1072_REGOFF 0x10c0
+#define HWIO_DDR_PHY_1072_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1072_REGOFF)
+#define HWIO_DDR_PHY_1072_PHY_PAD_RST_DRIVE_FLDMASK (0x1fffffff)
+#define HWIO_DDR_PHY_1072_PHY_PAD_RST_DRIVE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1072_RESERVED_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1072_RESERVED_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1073_REGOFF 0x10c4
+#define HWIO_DDR_PHY_1073_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1073_REGOFF)
+#define HWIO_DDR_PHY_1073_PHY_PAD_RST_TERM_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_1073_PHY_PAD_RST_TERM_FLDSHFT (0)
+#define HWIO_DDR_PHY_1073_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_1073_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_1073_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1073_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1074_REGOFF 0x10c8
+#define HWIO_DDR_PHY_1074_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1074_REGOFF)
+#define HWIO_DDR_PHY_1074_PHY_PAD_CS_DRIVE_FLDMASK (0x1fffffff)
+#define HWIO_DDR_PHY_1074_PHY_PAD_CS_DRIVE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1074_RESERVED_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1074_RESERVED_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1075_REGOFF 0x10cc
+#define HWIO_DDR_PHY_1075_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1075_REGOFF)
+#define HWIO_DDR_PHY_1075_PHY_PAD_CS_TERM_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_1075_PHY_PAD_CS_TERM_FLDSHFT (0)
+#define HWIO_DDR_PHY_1075_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_1075_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_1075_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1075_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1076_REGOFF 0x10d0
+#define HWIO_DDR_PHY_1076_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1076_REGOFF)
+#define HWIO_DDR_PHY_1076_PHY_PAD_ODT_DRIVE_FLDMASK (0x1fffffff)
+#define HWIO_DDR_PHY_1076_PHY_PAD_ODT_DRIVE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1076_RESERVED_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1076_RESERVED_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1077_REGOFF 0x10d4
+#define HWIO_DDR_PHY_1077_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1077_REGOFF)
+#define HWIO_DDR_PHY_1077_PHY_PAD_ODT_TERM_FLDMASK (0x3ffff)
+#define HWIO_DDR_PHY_1077_PHY_PAD_ODT_TERM_FLDSHFT (0)
+#define HWIO_DDR_PHY_1077_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PHY_1077_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PHY_1077_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1077_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1078_REGOFF 0x10d8
+#define HWIO_DDR_PHY_1078_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1078_REGOFF)
+#define HWIO_DDR_PHY_1078_PHY_ADRCTL_RX_CAL_FLDMASK (0x3fff)
+#define HWIO_DDR_PHY_1078_PHY_ADRCTL_RX_CAL_FLDSHFT (0)
+#define HWIO_DDR_PHY_1078_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_PHY_1078_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_PHY_1078_PHY_ADRCTL_LP3_RX_CAL_FLDMASK (0x1fff0000)
+#define HWIO_DDR_PHY_1078_PHY_ADRCTL_LP3_RX_CAL_FLDSHFT (16)
+#define HWIO_DDR_PHY_1078_RESERVED1_FLDMASK (0xe0000000)
+#define HWIO_DDR_PHY_1078_RESERVED1_FLDSHFT (29)
+
+#define HWIO_DDR_PHY_1079_REGOFF 0x10dc
+#define HWIO_DDR_PHY_1079_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1079_REGOFF)
+#define HWIO_DDR_PHY_1079_PHY_TST_CLK_PAD_CTRL_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1079_PHY_TST_CLK_PAD_CTRL_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1080_REGOFF 0x10e0
+#define HWIO_DDR_PHY_1080_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1080_REGOFF)
+#define HWIO_DDR_PHY_1080_PHY_TST_CLK_PAD_CTRL2_FLDMASK (0x7fffff)
+#define HWIO_DDR_PHY_1080_PHY_TST_CLK_PAD_CTRL2_FLDSHFT (0)
+#define HWIO_DDR_PHY_1080_RESERVED_FLDMASK (0x800000)
+#define HWIO_DDR_PHY_1080_RESERVED_FLDSHFT (23)
+#define HWIO_DDR_PHY_1080_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1080_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1081_REGOFF 0x10e4
+#define HWIO_DDR_PHY_1081_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1081_REGOFF)
+#define HWIO_DDR_PHY_1081_PHY_TST_CLK_PAD_CTRL3_FLDMASK (0x1fffff)
+#define HWIO_DDR_PHY_1081_PHY_TST_CLK_PAD_CTRL3_FLDSHFT (0)
+#define HWIO_DDR_PHY_1081_RESERVED_FLDMASK (0xe00000)
+#define HWIO_DDR_PHY_1081_RESERVED_FLDSHFT (21)
+#define HWIO_DDR_PHY_1081_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1081_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1082_REGOFF 0x10e8
+#define HWIO_DDR_PHY_1082_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1082_REGOFF)
+#define HWIO_DDR_PHY_1082_PHY_CAL_MODE_0_FLDMASK (0x1ff)
+#define HWIO_DDR_PHY_1082_PHY_CAL_MODE_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_1082_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_1082_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PHY_1082_PHY_CAL_CLEAR_0_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1082_PHY_CAL_CLEAR_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_1082_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1082_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_1082_PHY_CAL_START_0_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_1082_PHY_CAL_START_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_1082_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_1082_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_1083_REGOFF 0x10ec
+#define HWIO_DDR_PHY_1083_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1083_REGOFF)
+#define HWIO_DDR_PHY_1083_PHY_CAL_INTERVAL_COUNT_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1083_PHY_CAL_INTERVAL_COUNT_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1084_REGOFF 0x10f0
+#define HWIO_DDR_PHY_1084_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1084_REGOFF)
+#define HWIO_DDR_PHY_1084_PHY_CAL_SAMPLE_WAIT_0_FLDMASK (0xff)
+#define HWIO_DDR_PHY_1084_PHY_CAL_SAMPLE_WAIT_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_1084_PHY_LP4_BOOT_CAL_CLK_SELECT_0_FLDMASK (0x700)
+#define HWIO_DDR_PHY_1084_PHY_LP4_BOOT_CAL_CLK_SELECT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_1084_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_1084_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_1084_PHY_CAL_CLK_SELECT_0_FLDMASK (0x70000)
+#define HWIO_DDR_PHY_1084_PHY_CAL_CLK_SELECT_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_1084_RESERVED1_FLDMASK (0xf80000)
+#define HWIO_DDR_PHY_1084_RESERVED1_FLDSHFT (19)
+#define HWIO_DDR_PHY_1084_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1084_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1085_REGOFF 0x10f4
+#define HWIO_DDR_PHY_1085_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1085_REGOFF)
+#define HWIO_DDR_PHY_1085_PHY_CAL_RESULT_OBS_0_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_1085_PHY_CAL_RESULT_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_1085_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1085_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1086_REGOFF 0x10f8
+#define HWIO_DDR_PHY_1086_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1086_REGOFF)
+#define HWIO_DDR_PHY_1086_PHY_CAL_RESULT2_OBS_0_FLDMASK (0xffffff)
+#define HWIO_DDR_PHY_1086_PHY_CAL_RESULT2_OBS_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_1086_PHY_CAL_CPTR_CNT_0_FLDMASK (0x7f000000)
+#define HWIO_DDR_PHY_1086_PHY_CAL_CPTR_CNT_0_FLDSHFT (24)
+#define HWIO_DDR_PHY_1086_RESERVED_FLDMASK (0x80000000)
+#define HWIO_DDR_PHY_1086_RESERVED_FLDSHFT (31)
+
+#define HWIO_DDR_PHY_1087_REGOFF 0x10fc
+#define HWIO_DDR_PHY_1087_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1087_REGOFF)
+#define HWIO_DDR_PHY_1087_PHY_CAL_SETTLING_PRD_0_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_1087_PHY_CAL_SETTLING_PRD_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_1087_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_1087_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_1087_PHY_CAL_PU_FINE_ADJ_0_FLDMASK (0xff00)
+#define HWIO_DDR_PHY_1087_PHY_CAL_PU_FINE_ADJ_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_1087_PHY_CAL_PD_FINE_ADJ_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PHY_1087_PHY_CAL_PD_FINE_ADJ_0_FLDSHFT (16)
+#define HWIO_DDR_PHY_1087_PHY_CAL_RCV_FINE_ADJ_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1087_PHY_CAL_RCV_FINE_ADJ_0_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1088_REGOFF 0x1100
+#define HWIO_DDR_PHY_1088_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1088_REGOFF)
+#define HWIO_DDR_PHY_1088_PHY_CAL_DBG_CFG_0_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1088_PHY_CAL_DBG_CFG_0_FLDSHFT (0)
+#define HWIO_DDR_PHY_1088_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1088_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1088_SC_PHY_PAD_DBG_CONT_0_FLDMASK (0x100)
+#define HWIO_DDR_PHY_1088_SC_PHY_PAD_DBG_CONT_0_FLDSHFT (8)
+#define HWIO_DDR_PHY_1088_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PHY_1088_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PHY_1088_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_1088_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_1089_REGOFF 0x1104
+#define HWIO_DDR_PHY_1089_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1089_REGOFF)
+#define HWIO_DDR_PHY_1089_PHY_CAL_RESULT3_OBS_0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1089_PHY_CAL_RESULT3_OBS_0_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1090_REGOFF 0x1108
+#define HWIO_DDR_PHY_1090_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1090_REGOFF)
+#define HWIO_DDR_PHY_1090_PHY_PAD_ATB_CTRL_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_1090_PHY_PAD_ATB_CTRL_FLDSHFT (0)
+#define HWIO_DDR_PHY_1090_PHY_ADRCTL_MANUAL_UPDATE_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1090_PHY_ADRCTL_MANUAL_UPDATE_FLDSHFT (16)
+#define HWIO_DDR_PHY_1090_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1090_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_1090_PHY_AC_LPBK_ERR_CLEAR_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_1090_PHY_AC_LPBK_ERR_CLEAR_FLDSHFT (24)
+#define HWIO_DDR_PHY_1090_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_1090_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_1091_REGOFF 0x110c
+#define HWIO_DDR_PHY_1091_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1091_REGOFF)
+#define HWIO_DDR_PHY_1091_PHY_AC_LPBK_OBS_SELECT_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1091_PHY_AC_LPBK_OBS_SELECT_FLDSHFT (0)
+#define HWIO_DDR_PHY_1091_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1091_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1091_PHY_AC_LPBK_ENABLE_FLDMASK (0x300)
+#define HWIO_DDR_PHY_1091_PHY_AC_LPBK_ENABLE_FLDSHFT (8)
+#define HWIO_DDR_PHY_1091_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_1091_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_1091_PHY_AC_LPBK_CONTROL_FLDMASK (0x1ff0000)
+#define HWIO_DDR_PHY_1091_PHY_AC_LPBK_CONTROL_FLDSHFT (16)
+#define HWIO_DDR_PHY_1091_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_1091_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_1092_REGOFF 0x1110
+#define HWIO_DDR_PHY_1092_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1092_REGOFF)
+#define HWIO_DDR_PHY_1092_PHY_AC_PRBS_PATTERN_START_FLDMASK (0x7f)
+#define HWIO_DDR_PHY_1092_PHY_AC_PRBS_PATTERN_START_FLDSHFT (0)
+#define HWIO_DDR_PHY_1092_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_PHY_1092_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_PHY_1092_PHY_AC_PRBS_PATTERN_MASK_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_1092_PHY_AC_PRBS_PATTERN_MASK_FLDSHFT (8)
+#define HWIO_DDR_PHY_1092_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_1092_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_1092_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PHY_1092_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PHY_1093_REGOFF 0x1114
+#define HWIO_DDR_PHY_1093_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1093_REGOFF)
+#define HWIO_DDR_PHY_1093_PHY_AC_LPBK_RESULT_OBS_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1093_PHY_AC_LPBK_RESULT_OBS_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1094_REGOFF 0x1118
+#define HWIO_DDR_PHY_1094_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1094_REGOFF)
+#define HWIO_DDR_PHY_1094_PHY_AC_CLK_LPBK_OBS_SELECT_FLDMASK (0x1)
+#define HWIO_DDR_PHY_1094_PHY_AC_CLK_LPBK_OBS_SELECT_FLDSHFT (0)
+#define HWIO_DDR_PHY_1094_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PHY_1094_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PHY_1094_PHY_AC_CLK_LPBK_ENABLE_FLDMASK (0x300)
+#define HWIO_DDR_PHY_1094_PHY_AC_CLK_LPBK_ENABLE_FLDSHFT (8)
+#define HWIO_DDR_PHY_1094_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_1094_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PHY_1094_PHY_AC_CLK_LPBK_CONTROL_FLDMASK (0x3f0000)
+#define HWIO_DDR_PHY_1094_PHY_AC_CLK_LPBK_CONTROL_FLDSHFT (16)
+#define HWIO_DDR_PHY_1094_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PHY_1094_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PHY_1094_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1094_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1095_REGOFF 0x111c
+#define HWIO_DDR_PHY_1095_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1095_REGOFF)
+#define HWIO_DDR_PHY_1095_PHY_AC_CLK_LPBK_RESULT_OBS_FLDMASK (0xffff)
+#define HWIO_DDR_PHY_1095_PHY_AC_CLK_LPBK_RESULT_OBS_FLDSHFT (0)
+#define HWIO_DDR_PHY_1095_PHY_AC_PWR_RDC_DISABLE_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1095_PHY_AC_PWR_RDC_DISABLE_FLDSHFT (16)
+#define HWIO_DDR_PHY_1095_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1095_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PHY_1095_PHY_LPDDR4_CONNECT_FLDMASK (0x1000000)
+#define HWIO_DDR_PHY_1095_PHY_LPDDR4_CONNECT_FLDSHFT (24)
+#define HWIO_DDR_PHY_1095_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PHY_1095_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PHY_1096_REGOFF 0x1120
+#define HWIO_DDR_PHY_1096_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1096_REGOFF)
+#define HWIO_DDR_PHY_1096_PHY_CALVL_DEVICE_MAP_FLDMASK (0x1f)
+#define HWIO_DDR_PHY_1096_PHY_CALVL_DEVICE_MAP_FLDSHFT (0)
+#define HWIO_DDR_PHY_1096_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PHY_1096_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PHY_1096_PHY_ADRCTL_MSTR_DLY_ENC_SEL_FLDMASK (0xf00)
+#define HWIO_DDR_PHY_1096_PHY_ADRCTL_MSTR_DLY_ENC_SEL_FLDSHFT (8)
+#define HWIO_DDR_PHY_1096_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PHY_1096_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PHY_1096_PHY_CS_DLY_UPT_PER_AC_SLICE_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1096_PHY_CS_DLY_UPT_PER_AC_SLICE_FLDSHFT (16)
+#define HWIO_DDR_PHY_1096_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1096_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PHY_1096_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PHY_1096_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PHY_1097_REGOFF 0x1124
+#define HWIO_DDR_PHY_1097_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1097_REGOFF)
+#define HWIO_DDR_PHY_1097_PHY_DDL_AC_ENABLE_FLDMASK (0xffffffff)
+#define HWIO_DDR_PHY_1097_PHY_DDL_AC_ENABLE_FLDSHFT (0)
+
+#define HWIO_DDR_PHY_1098_REGOFF 0x1128
+#define HWIO_DDR_PHY_1098_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1098_REGOFF)
+#define HWIO_DDR_PHY_1098_PHY_DDL_AC_MODE_FLDMASK (0x7ff)
+#define HWIO_DDR_PHY_1098_PHY_DDL_AC_MODE_FLDSHFT (0)
+#define HWIO_DDR_PHY_1098_RESERVED_FLDMASK (0xf800)
+#define HWIO_DDR_PHY_1098_RESERVED_FLDSHFT (11)
+#define HWIO_DDR_PHY_1098_PHY_PAD_BACKGROUND_CAL_FLDMASK (0x10000)
+#define HWIO_DDR_PHY_1098_PHY_PAD_BACKGROUND_CAL_FLDSHFT (16)
+#define HWIO_DDR_PHY_1098_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PHY_1098_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PHY_1098_PHY_INIT_UPDATE_CONFIG_FLDMASK (0x7000000)
+#define HWIO_DDR_PHY_1098_PHY_INIT_UPDATE_CONFIG_FLDSHFT (24)
+#define HWIO_DDR_PHY_1098_RESERVED2_FLDMASK (0xf8000000)
+#define HWIO_DDR_PHY_1098_RESERVED2_FLDSHFT (27)
+
+#define HWIO_DDR_PHY_1099_REGOFF 0x112c
+#define HWIO_DDR_PHY_1099_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1099_REGOFF)
+#define HWIO_DDR_PHY_1099_PHY_DDL_TRACK_UPD_THRESHOLD_AC_FLDMASK (0xff)
+#define HWIO_DDR_PHY_1099_PHY_DDL_TRACK_UPD_THRESHOLD_AC_FLDSHFT (0)
+#define HWIO_DDR_PHY_1099_PHY_DLL_RST_EN_FLDMASK (0x300)
+#define HWIO_DDR_PHY_1099_PHY_DLL_RST_EN_FLDSHFT (8)
+#define HWIO_DDR_PHY_1099_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PHY_1099_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PHY_1099_PHY_AC_INIT_COMPLETE_OBS_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PHY_1099_PHY_AC_INIT_COMPLETE_OBS_FLDSHFT (16)
+#define HWIO_DDR_PHY_1099_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PHY_1099_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PHY_1100_REGOFF 0x1130
+#define HWIO_DDR_PHY_1100_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PHY_1100_REGOFF)
+#define HWIO_DDR_PHY_1100_PHY_DS_INIT_COMPLETE_OBS_FLDMASK (0xf)
+#define HWIO_DDR_PHY_1100_PHY_DS_INIT_COMPLETE_OBS_FLDSHFT (0)
+#define HWIO_DDR_PHY_1100_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PHY_1100_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PHY_1100_PHY_UPDATE_MASK_FLDMASK (0x100)
+#define HWIO_DDR_PHY_1100_PHY_UPDATE_MASK_FLDSHFT (8)
+#define HWIO_DDR_PHY_1100_RESERVED1_FLDMASK (0xfffffe00)
+#define HWIO_DDR_PHY_1100_RESERVED1_FLDSHFT (9)
+
+#endif /* __MNH_HWIO_DDR_PHY_ */
diff --git a/drivers/misc/mnh/mnh-hwio-ddr-pi.h b/drivers/misc/mnh/mnh-hwio-ddr-pi.h
new file mode 100644
index 0000000..c133c61
--- /dev/null
+++ b/drivers/misc/mnh/mnh-hwio-ddr-pi.h
@@ -0,0 +1,2291 @@
+/* auto generated: Friday, August 26th, 2016 11:42:19am */
+/*
+ * Copyright (c) 2016, Intel Corporation. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this
+ * list of conditions and the following disclaimer.
+ *
+ * Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * Neither the name of Intel nor the names of its contributors may be used
+ * to endorse or promote products derived from this software without specific
+ * prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef __MNH_HWIO_DDR_PI_
+#define __MNH_HWIO_DDR_PI_
+
+#define HWIO_DDR_PI_00_REGOFF 0x0
+#define HWIO_DDR_PI_00_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_00_REGOFF)
+#define HWIO_DDR_PI_00_PI_START_FLDMASK (0x1)
+#define HWIO_DDR_PI_00_PI_START_FLDSHFT (0)
+#define HWIO_DDR_PI_00_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_00_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_00_PI_DRAM_CLASS_FLDMASK (0xf00)
+#define HWIO_DDR_PI_00_PI_DRAM_CLASS_FLDSHFT (8)
+#define HWIO_DDR_PI_00_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PI_00_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PI_00_PI_VERSION_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_00_PI_VERSION_FLDSHFT (16)
+
+#define HWIO_DDR_PI_01_REGOFF 0x4
+#define HWIO_DDR_PI_01_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_01_REGOFF)
+#define HWIO_DDR_PI_01_PI_NORMAL_LVL_SEQ_FLDMASK (0x1)
+#define HWIO_DDR_PI_01_PI_NORMAL_LVL_SEQ_FLDSHFT (0)
+#define HWIO_DDR_PI_01_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_01_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_01_PI_INIT_LVL_EN_FLDMASK (0x100)
+#define HWIO_DDR_PI_01_PI_INIT_LVL_EN_FLDSHFT (8)
+#define HWIO_DDR_PI_01_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_01_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_01_PI_NOTCARE_PHYUPD_FLDMASK (0x10000)
+#define HWIO_DDR_PI_01_PI_NOTCARE_PHYUPD_FLDSHFT (16)
+#define HWIO_DDR_PI_01_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_01_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_01_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_01_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PI_02_REGOFF 0x8
+#define HWIO_DDR_PI_02_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_02_REGOFF)
+#define HWIO_DDR_PI_02_PI_TCMD_GAP_FLDMASK (0xffff)
+#define HWIO_DDR_PI_02_PI_TCMD_GAP_FLDSHFT (0)
+#define HWIO_DDR_PI_02_PI_MASTER_ACK_DURATION_MIN_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_02_PI_MASTER_ACK_DURATION_MIN_FLDSHFT (16)
+#define HWIO_DDR_PI_02_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_02_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_03_REGOFF 0xc
+#define HWIO_DDR_PI_03_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_03_REGOFF)
+#define HWIO_DDR_PI_03_PI_TDFI_PHYMSTR_MAX_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_03_PI_TDFI_PHYMSTR_MAX_F0_FLDSHFT (0)
+
+#define HWIO_DDR_PI_04_REGOFF 0x10
+#define HWIO_DDR_PI_04_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_04_REGOFF)
+#define HWIO_DDR_PI_04_PI_TDFI_PHYMSTR_RESP_F0_FLDMASK (0xffff)
+#define HWIO_DDR_PI_04_PI_TDFI_PHYMSTR_RESP_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_04_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_04_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_05_REGOFF 0x14
+#define HWIO_DDR_PI_05_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_05_REGOFF)
+#define HWIO_DDR_PI_05_PI_TDFI_PHYMSTR_MAX_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_05_PI_TDFI_PHYMSTR_MAX_F1_FLDSHFT (0)
+
+#define HWIO_DDR_PI_06_REGOFF 0x18
+#define HWIO_DDR_PI_06_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_06_REGOFF)
+#define HWIO_DDR_PI_06_PI_TDFI_PHYMSTR_RESP_F1_FLDMASK (0xffff)
+#define HWIO_DDR_PI_06_PI_TDFI_PHYMSTR_RESP_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_06_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_06_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_07_REGOFF 0x1c
+#define HWIO_DDR_PI_07_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_07_REGOFF)
+#define HWIO_DDR_PI_07_PI_TDFI_PHYMSTR_MAX_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_07_PI_TDFI_PHYMSTR_MAX_F2_FLDSHFT (0)
+
+#define HWIO_DDR_PI_08_REGOFF 0x20
+#define HWIO_DDR_PI_08_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_08_REGOFF)
+#define HWIO_DDR_PI_08_PI_TDFI_PHYMSTR_RESP_F2_FLDMASK (0xffff)
+#define HWIO_DDR_PI_08_PI_TDFI_PHYMSTR_RESP_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_08_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_08_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_09_REGOFF 0x24
+#define HWIO_DDR_PI_09_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_09_REGOFF)
+#define HWIO_DDR_PI_09_PI_TDFI_PHYMSTR_MAX_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_09_PI_TDFI_PHYMSTR_MAX_F3_FLDSHFT (0)
+
+#define HWIO_DDR_PI_10_REGOFF 0x28
+#define HWIO_DDR_PI_10_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_10_REGOFF)
+#define HWIO_DDR_PI_10_PI_TDFI_PHYMSTR_RESP_F3_FLDMASK (0xffff)
+#define HWIO_DDR_PI_10_PI_TDFI_PHYMSTR_RESP_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_10_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_10_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_11_REGOFF 0x2c
+#define HWIO_DDR_PI_11_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_11_REGOFF)
+#define HWIO_DDR_PI_11_PI_TDFI_PHYUPD_RESP_F0_FLDMASK (0xfffff)
+#define HWIO_DDR_PI_11_PI_TDFI_PHYUPD_RESP_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_11_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_11_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PI_11_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_11_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_12_REGOFF 0x30
+#define HWIO_DDR_PI_12_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_12_REGOFF)
+#define HWIO_DDR_PI_12_PI_TDFI_PHYUPD_TYPE0_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_12_PI_TDFI_PHYUPD_TYPE0_F0_FLDSHFT (0)
+
+#define HWIO_DDR_PI_13_REGOFF 0x34
+#define HWIO_DDR_PI_13_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_13_REGOFF)
+#define HWIO_DDR_PI_13_PI_TDFI_PHYUPD_TYPE1_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_13_PI_TDFI_PHYUPD_TYPE1_F0_FLDSHFT (0)
+
+#define HWIO_DDR_PI_14_REGOFF 0x38
+#define HWIO_DDR_PI_14_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_14_REGOFF)
+#define HWIO_DDR_PI_14_PI_TDFI_PHYUPD_TYPE2_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_14_PI_TDFI_PHYUPD_TYPE2_F0_FLDSHFT (0)
+
+#define HWIO_DDR_PI_15_REGOFF 0x3c
+#define HWIO_DDR_PI_15_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_15_REGOFF)
+#define HWIO_DDR_PI_15_PI_TDFI_PHYUPD_TYPE3_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_15_PI_TDFI_PHYUPD_TYPE3_F0_FLDSHFT (0)
+
+#define HWIO_DDR_PI_16_REGOFF 0x40
+#define HWIO_DDR_PI_16_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_16_REGOFF)
+#define HWIO_DDR_PI_16_PI_TDFI_PHYUPD_RESP_F1_FLDMASK (0xfffff)
+#define HWIO_DDR_PI_16_PI_TDFI_PHYUPD_RESP_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_16_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_16_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PI_16_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_16_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_17_REGOFF 0x44
+#define HWIO_DDR_PI_17_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_17_REGOFF)
+#define HWIO_DDR_PI_17_PI_TDFI_PHYUPD_TYPE0_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_17_PI_TDFI_PHYUPD_TYPE0_F1_FLDSHFT (0)
+
+#define HWIO_DDR_PI_18_REGOFF 0x48
+#define HWIO_DDR_PI_18_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_18_REGOFF)
+#define HWIO_DDR_PI_18_PI_TDFI_PHYUPD_TYPE1_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_18_PI_TDFI_PHYUPD_TYPE1_F1_FLDSHFT (0)
+
+#define HWIO_DDR_PI_19_REGOFF 0x4c
+#define HWIO_DDR_PI_19_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_19_REGOFF)
+#define HWIO_DDR_PI_19_PI_TDFI_PHYUPD_TYPE2_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_19_PI_TDFI_PHYUPD_TYPE2_F1_FLDSHFT (0)
+
+#define HWIO_DDR_PI_20_REGOFF 0x50
+#define HWIO_DDR_PI_20_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_20_REGOFF)
+#define HWIO_DDR_PI_20_PI_TDFI_PHYUPD_TYPE3_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_20_PI_TDFI_PHYUPD_TYPE3_F1_FLDSHFT (0)
+
+#define HWIO_DDR_PI_21_REGOFF 0x54
+#define HWIO_DDR_PI_21_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_21_REGOFF)
+#define HWIO_DDR_PI_21_PI_TDFI_PHYUPD_RESP_F2_FLDMASK (0xfffff)
+#define HWIO_DDR_PI_21_PI_TDFI_PHYUPD_RESP_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_21_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_21_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PI_21_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_21_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_22_REGOFF 0x58
+#define HWIO_DDR_PI_22_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_22_REGOFF)
+#define HWIO_DDR_PI_22_PI_TDFI_PHYUPD_TYPE0_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_22_PI_TDFI_PHYUPD_TYPE0_F2_FLDSHFT (0)
+
+#define HWIO_DDR_PI_23_REGOFF 0x5c
+#define HWIO_DDR_PI_23_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_23_REGOFF)
+#define HWIO_DDR_PI_23_PI_TDFI_PHYUPD_TYPE1_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_23_PI_TDFI_PHYUPD_TYPE1_F2_FLDSHFT (0)
+
+#define HWIO_DDR_PI_24_REGOFF 0x60
+#define HWIO_DDR_PI_24_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_24_REGOFF)
+#define HWIO_DDR_PI_24_PI_TDFI_PHYUPD_TYPE2_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_24_PI_TDFI_PHYUPD_TYPE2_F2_FLDSHFT (0)
+
+#define HWIO_DDR_PI_25_REGOFF 0x64
+#define HWIO_DDR_PI_25_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_25_REGOFF)
+#define HWIO_DDR_PI_25_PI_TDFI_PHYUPD_TYPE3_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_25_PI_TDFI_PHYUPD_TYPE3_F2_FLDSHFT (0)
+
+#define HWIO_DDR_PI_26_REGOFF 0x68
+#define HWIO_DDR_PI_26_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_26_REGOFF)
+#define HWIO_DDR_PI_26_PI_TDFI_PHYUPD_RESP_F3_FLDMASK (0xfffff)
+#define HWIO_DDR_PI_26_PI_TDFI_PHYUPD_RESP_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_26_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_26_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PI_26_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_26_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_27_REGOFF 0x6c
+#define HWIO_DDR_PI_27_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_27_REGOFF)
+#define HWIO_DDR_PI_27_PI_TDFI_PHYUPD_TYPE0_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_27_PI_TDFI_PHYUPD_TYPE0_F3_FLDSHFT (0)
+
+#define HWIO_DDR_PI_28_REGOFF 0x70
+#define HWIO_DDR_PI_28_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_28_REGOFF)
+#define HWIO_DDR_PI_28_PI_TDFI_PHYUPD_TYPE1_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_28_PI_TDFI_PHYUPD_TYPE1_F3_FLDSHFT (0)
+
+#define HWIO_DDR_PI_29_REGOFF 0x74
+#define HWIO_DDR_PI_29_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_29_REGOFF)
+#define HWIO_DDR_PI_29_PI_TDFI_PHYUPD_TYPE2_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_29_PI_TDFI_PHYUPD_TYPE2_F3_FLDSHFT (0)
+
+#define HWIO_DDR_PI_30_REGOFF 0x78
+#define HWIO_DDR_PI_30_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_30_REGOFF)
+#define HWIO_DDR_PI_30_PI_TDFI_PHYUPD_TYPE3_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_30_PI_TDFI_PHYUPD_TYPE3_F3_FLDSHFT (0)
+
+#define HWIO_DDR_PI_31_REGOFF 0x7c
+#define HWIO_DDR_PI_31_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_31_REGOFF)
+#define HWIO_DDR_PI_31_PI_CONTROL_ERROR_STATUS_FLDMASK (0x1ff)
+#define HWIO_DDR_PI_31_PI_CONTROL_ERROR_STATUS_FLDSHFT (0)
+#define HWIO_DDR_PI_31_RESERVED_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_31_RESERVED_FLDSHFT (9)
+#define HWIO_DDR_PI_31_PI_EXIT_AFTER_INIT_CALVL_FLDMASK (0x10000)
+#define HWIO_DDR_PI_31_PI_EXIT_AFTER_INIT_CALVL_FLDSHFT (16)
+#define HWIO_DDR_PI_31_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_31_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PI_31_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_31_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_32_REGOFF 0x80
+#define HWIO_DDR_PI_32_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_32_REGOFF)
+#define HWIO_DDR_PI_32_PI_FREQ_MAP_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_32_PI_FREQ_MAP_FLDSHFT (0)
+
+#define HWIO_DDR_PI_33_REGOFF 0x84
+#define HWIO_DDR_PI_33_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_33_REGOFF)
+#define HWIO_DDR_PI_33_PI_INIT_WORK_FREQ_FLDMASK (0x1f)
+#define HWIO_DDR_PI_33_PI_INIT_WORK_FREQ_FLDSHFT (0)
+#define HWIO_DDR_PI_33_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PI_33_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PI_33_PI_INIT_DFS_CALVL_ONLY_FLDMASK (0x100)
+#define HWIO_DDR_PI_33_PI_INIT_DFS_CALVL_ONLY_FLDSHFT (8)
+#define HWIO_DDR_PI_33_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_33_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_33_PI_POWER_ON_SEQ_BYPASS_ARRAY_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_33_PI_POWER_ON_SEQ_BYPASS_ARRAY_FLDSHFT (16)
+#define HWIO_DDR_PI_33_PI_POWER_ON_SEQ_END_ARRAY_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_33_PI_POWER_ON_SEQ_END_ARRAY_FLDSHFT (24)
+
+#define HWIO_DDR_PI_34_REGOFF 0x88
+#define HWIO_DDR_PI_34_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_34_REGOFF)
+#define HWIO_DDR_PI_34_PI_SEQ1_PAT_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_34_PI_SEQ1_PAT_FLDSHFT (0)
+#define HWIO_DDR_PI_34_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_34_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_35_REGOFF 0x8c
+#define HWIO_DDR_PI_35_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_35_REGOFF)
+#define HWIO_DDR_PI_35_PI_SEQ1_PAT_MASK_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_35_PI_SEQ1_PAT_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_35_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_35_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_36_REGOFF 0x90
+#define HWIO_DDR_PI_36_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_36_REGOFF)
+#define HWIO_DDR_PI_36_PI_SEQ2_PAT_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_36_PI_SEQ2_PAT_FLDSHFT (0)
+#define HWIO_DDR_PI_36_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_36_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_37_REGOFF 0x94
+#define HWIO_DDR_PI_37_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_37_REGOFF)
+#define HWIO_DDR_PI_37_PI_SEQ2_PAT_MASK_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_37_PI_SEQ2_PAT_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_37_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_37_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_38_REGOFF 0x98
+#define HWIO_DDR_PI_38_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_38_REGOFF)
+#define HWIO_DDR_PI_38_PI_SEQ3_PAT_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_38_PI_SEQ3_PAT_FLDSHFT (0)
+#define HWIO_DDR_PI_38_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_38_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_39_REGOFF 0x9c
+#define HWIO_DDR_PI_39_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_39_REGOFF)
+#define HWIO_DDR_PI_39_PI_SEQ3_PAT_MASK_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_39_PI_SEQ3_PAT_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_39_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_39_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_40_REGOFF 0xa0
+#define HWIO_DDR_PI_40_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_40_REGOFF)
+#define HWIO_DDR_PI_40_PI_SEQ4_PAT_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_40_PI_SEQ4_PAT_FLDSHFT (0)
+#define HWIO_DDR_PI_40_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_40_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_41_REGOFF 0xa4
+#define HWIO_DDR_PI_41_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_41_REGOFF)
+#define HWIO_DDR_PI_41_PI_SEQ4_PAT_MASK_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_41_PI_SEQ4_PAT_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_41_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_41_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_42_REGOFF 0xa8
+#define HWIO_DDR_PI_42_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_42_REGOFF)
+#define HWIO_DDR_PI_42_PI_SEQ5_PAT_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_42_PI_SEQ5_PAT_FLDSHFT (0)
+#define HWIO_DDR_PI_42_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_42_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_43_REGOFF 0xac
+#define HWIO_DDR_PI_43_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_43_REGOFF)
+#define HWIO_DDR_PI_43_PI_SEQ5_PAT_MASK_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_43_PI_SEQ5_PAT_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_43_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_43_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_44_REGOFF 0xb0
+#define HWIO_DDR_PI_44_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_44_REGOFF)
+#define HWIO_DDR_PI_44_PI_SEQ6_PAT_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_44_PI_SEQ6_PAT_FLDSHFT (0)
+#define HWIO_DDR_PI_44_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_44_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_45_REGOFF 0xb4
+#define HWIO_DDR_PI_45_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_45_REGOFF)
+#define HWIO_DDR_PI_45_PI_SEQ6_PAT_MASK_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_45_PI_SEQ6_PAT_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_45_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_45_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_46_REGOFF 0xb8
+#define HWIO_DDR_PI_46_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_46_REGOFF)
+#define HWIO_DDR_PI_46_PI_SEQ7_PAT_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_46_PI_SEQ7_PAT_FLDSHFT (0)
+#define HWIO_DDR_PI_46_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_46_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_47_REGOFF 0xbc
+#define HWIO_DDR_PI_47_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_47_REGOFF)
+#define HWIO_DDR_PI_47_PI_SEQ7_PAT_MASK_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_47_PI_SEQ7_PAT_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_47_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_47_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_48_REGOFF 0xc0
+#define HWIO_DDR_PI_48_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_48_REGOFF)
+#define HWIO_DDR_PI_48_PI_SEQ8_PAT_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_48_PI_SEQ8_PAT_FLDSHFT (0)
+#define HWIO_DDR_PI_48_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_48_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_49_REGOFF 0xc4
+#define HWIO_DDR_PI_49_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_49_REGOFF)
+#define HWIO_DDR_PI_49_PI_SEQ8_PAT_MASK_FLDMASK (0xffffff)
+#define HWIO_DDR_PI_49_PI_SEQ8_PAT_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_49_PI_WDT_DISABLE_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_49_PI_WDT_DISABLE_FLDSHFT (24)
+#define HWIO_DDR_PI_49_RESERVED_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_49_RESERVED_FLDSHFT (25)
+
+#define HWIO_DDR_PI_50_REGOFF 0xc8
+#define HWIO_DDR_PI_50_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_50_REGOFF)
+#define HWIO_DDR_PI_50_PI_SW_RST_N_FLDMASK (0x1)
+#define HWIO_DDR_PI_50_PI_SW_RST_N_FLDSHFT (0)
+#define HWIO_DDR_PI_50_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_50_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_50_RESERVED1_FLDMASK (0x100)
+#define HWIO_DDR_PI_50_RESERVED1_FLDSHFT (8)
+#define HWIO_DDR_PI_50_RESERVED2_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_50_RESERVED2_FLDSHFT (9)
+#define HWIO_DDR_PI_50_PI_CS_MAP_FLDMASK (0x30000)
+#define HWIO_DDR_PI_50_PI_CS_MAP_FLDSHFT (16)
+#define HWIO_DDR_PI_50_RESERVED3_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_50_RESERVED3_FLDSHFT (18)
+#define HWIO_DDR_PI_50_PI_TDELAY_RDWR_2_BUS_IDLE_F0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_50_PI_TDELAY_RDWR_2_BUS_IDLE_F0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_51_REGOFF 0xcc
+#define HWIO_DDR_PI_51_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_51_REGOFF)
+#define HWIO_DDR_PI_51_PI_TDELAY_RDWR_2_BUS_IDLE_F1_FLDMASK (0xff)
+#define HWIO_DDR_PI_51_PI_TDELAY_RDWR_2_BUS_IDLE_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_51_PI_TDELAY_RDWR_2_BUS_IDLE_F2_FLDMASK (0xff00)
+#define HWIO_DDR_PI_51_PI_TDELAY_RDWR_2_BUS_IDLE_F2_FLDSHFT (8)
+#define HWIO_DDR_PI_51_PI_TDELAY_RDWR_2_BUS_IDLE_F3_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_51_PI_TDELAY_RDWR_2_BUS_IDLE_F3_FLDSHFT (16)
+#define HWIO_DDR_PI_51_PI_TMRR_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_51_PI_TMRR_FLDSHFT (24)
+#define HWIO_DDR_PI_51_RESERVED_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_51_RESERVED_FLDSHFT (28)
+
+#define HWIO_DDR_PI_52_REGOFF 0xd0
+#define HWIO_DDR_PI_52_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_52_REGOFF)
+#define HWIO_DDR_PI_52_PI_PREAMBLE_SUPPORT_FLDMASK (0x3)
+#define HWIO_DDR_PI_52_PI_PREAMBLE_SUPPORT_FLDSHFT (0)
+#define HWIO_DDR_PI_52_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_52_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_52_PI_AREFRESH_FLDMASK (0x100)
+#define HWIO_DDR_PI_52_PI_AREFRESH_FLDSHFT (8)
+#define HWIO_DDR_PI_52_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_52_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_52_PI_MCAREF_FORWARD_ONLY_FLDMASK (0x10000)
+#define HWIO_DDR_PI_52_PI_MCAREF_FORWARD_ONLY_FLDSHFT (16)
+#define HWIO_DDR_PI_52_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_52_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_52_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_52_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PI_53_REGOFF 0xd4
+#define HWIO_DDR_PI_53_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_53_REGOFF)
+#define HWIO_DDR_PI_53_PI_TRFC_F0_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_53_PI_TRFC_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_53_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_53_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_53_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_53_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_54_REGOFF 0xd8
+#define HWIO_DDR_PI_54_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_54_REGOFF)
+#define HWIO_DDR_PI_54_PI_TREF_F0_FLDMASK (0xfffff)
+#define HWIO_DDR_PI_54_PI_TREF_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_54_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_54_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PI_54_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_54_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_55_REGOFF 0xdc
+#define HWIO_DDR_PI_55_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_55_REGOFF)
+#define HWIO_DDR_PI_55_PI_TRFC_F1_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_55_PI_TRFC_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_55_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_55_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_55_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_55_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_56_REGOFF 0xe0
+#define HWIO_DDR_PI_56_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_56_REGOFF)
+#define HWIO_DDR_PI_56_PI_TREF_F1_FLDMASK (0xfffff)
+#define HWIO_DDR_PI_56_PI_TREF_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_56_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_56_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PI_56_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_56_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_57_REGOFF 0xe4
+#define HWIO_DDR_PI_57_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_57_REGOFF)
+#define HWIO_DDR_PI_57_PI_TRFC_F2_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_57_PI_TRFC_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_57_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_57_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_57_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_57_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_58_REGOFF 0xe8
+#define HWIO_DDR_PI_58_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_58_REGOFF)
+#define HWIO_DDR_PI_58_PI_TREF_F2_FLDMASK (0xfffff)
+#define HWIO_DDR_PI_58_PI_TREF_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_58_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_58_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PI_58_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_58_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_59_REGOFF 0xec
+#define HWIO_DDR_PI_59_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_59_REGOFF)
+#define HWIO_DDR_PI_59_PI_TRFC_F3_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_59_PI_TRFC_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_59_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_59_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_59_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_59_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_60_REGOFF 0xf0
+#define HWIO_DDR_PI_60_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_60_REGOFF)
+#define HWIO_DDR_PI_60_PI_TREF_F3_FLDMASK (0xfffff)
+#define HWIO_DDR_PI_60_PI_TREF_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_60_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_60_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PI_60_RESERVED1_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_60_RESERVED1_FLDSHFT (24)
+#define HWIO_DDR_PI_60_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_60_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PI_61_REGOFF 0xf4
+#define HWIO_DDR_PI_61_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_61_REGOFF)
+#define HWIO_DDR_PI_61_PI_FREQ_CHANGE_REG_COPY_FLDMASK (0x1f)
+#define HWIO_DDR_PI_61_PI_FREQ_CHANGE_REG_COPY_FLDSHFT (0)
+#define HWIO_DDR_PI_61_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PI_61_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PI_61_PI_FREQ_SEL_FROM_REGIF_FLDMASK (0x100)
+#define HWIO_DDR_PI_61_PI_FREQ_SEL_FROM_REGIF_FLDSHFT (8)
+#define HWIO_DDR_PI_61_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_61_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_61_PI_SWLVL_LOAD_FLDMASK (0x10000)
+#define HWIO_DDR_PI_61_PI_SWLVL_LOAD_FLDSHFT (16)
+#define HWIO_DDR_PI_61_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_61_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_61_PI_SWLVL_OP_DONE_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_61_PI_SWLVL_OP_DONE_FLDSHFT (24)
+#define HWIO_DDR_PI_61_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_61_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_62_REGOFF 0xf8
+#define HWIO_DDR_PI_62_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_62_REGOFF)
+#define HWIO_DDR_PI_62_PI_SW_WRLVL_RESP_0_FLDMASK (0x1)
+#define HWIO_DDR_PI_62_PI_SW_WRLVL_RESP_0_FLDSHFT (0)
+#define HWIO_DDR_PI_62_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_62_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_62_PI_SW_WRLVL_RESP_1_FLDMASK (0x100)
+#define HWIO_DDR_PI_62_PI_SW_WRLVL_RESP_1_FLDSHFT (8)
+#define HWIO_DDR_PI_62_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_62_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_62_PI_SW_WRLVL_RESP_2_FLDMASK (0x10000)
+#define HWIO_DDR_PI_62_PI_SW_WRLVL_RESP_2_FLDSHFT (16)
+#define HWIO_DDR_PI_62_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_62_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_62_PI_SW_WRLVL_RESP_3_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_62_PI_SW_WRLVL_RESP_3_FLDSHFT (24)
+#define HWIO_DDR_PI_62_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_62_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_63_REGOFF 0xfc
+#define HWIO_DDR_PI_63_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_63_REGOFF)
+#define HWIO_DDR_PI_63_PI_SW_RDLVL_RESP_0_FLDMASK (0x3)
+#define HWIO_DDR_PI_63_PI_SW_RDLVL_RESP_0_FLDSHFT (0)
+#define HWIO_DDR_PI_63_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_63_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_63_PI_SW_RDLVL_RESP_1_FLDMASK (0x300)
+#define HWIO_DDR_PI_63_PI_SW_RDLVL_RESP_1_FLDSHFT (8)
+#define HWIO_DDR_PI_63_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_63_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_63_PI_SW_RDLVL_RESP_2_FLDMASK (0x30000)
+#define HWIO_DDR_PI_63_PI_SW_RDLVL_RESP_2_FLDSHFT (16)
+#define HWIO_DDR_PI_63_RESERVED2_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_63_RESERVED2_FLDSHFT (18)
+#define HWIO_DDR_PI_63_PI_SW_RDLVL_RESP_3_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_63_PI_SW_RDLVL_RESP_3_FLDSHFT (24)
+#define HWIO_DDR_PI_63_RESERVED3_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_63_RESERVED3_FLDSHFT (26)
+
+#define HWIO_DDR_PI_64_REGOFF 0x100
+#define HWIO_DDR_PI_64_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_64_REGOFF)
+#define HWIO_DDR_PI_64_PI_SW_CALVL_RESP_0_FLDMASK (0x3)
+#define HWIO_DDR_PI_64_PI_SW_CALVL_RESP_0_FLDSHFT (0)
+#define HWIO_DDR_PI_64_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_64_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_64_PI_SW_LEVELING_MODE_FLDMASK (0x700)
+#define HWIO_DDR_PI_64_PI_SW_LEVELING_MODE_FLDSHFT (8)
+#define HWIO_DDR_PI_64_RESERVED1_FLDMASK (0xf800)
+#define HWIO_DDR_PI_64_RESERVED1_FLDSHFT (11)
+#define HWIO_DDR_PI_64_PI_SWLVL_START_FLDMASK (0x10000)
+#define HWIO_DDR_PI_64_PI_SWLVL_START_FLDSHFT (16)
+#define HWIO_DDR_PI_64_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_64_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_64_PI_SWLVL_EXIT_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_64_PI_SWLVL_EXIT_FLDSHFT (24)
+#define HWIO_DDR_PI_64_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_64_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_65_REGOFF 0x104
+#define HWIO_DDR_PI_65_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_65_REGOFF)
+#define HWIO_DDR_PI_65_PI_SWLVL_WR_SLICE_0_FLDMASK (0x1)
+#define HWIO_DDR_PI_65_PI_SWLVL_WR_SLICE_0_FLDSHFT (0)
+#define HWIO_DDR_PI_65_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_65_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_65_PI_SWLVL_RD_SLICE_0_FLDMASK (0x100)
+#define HWIO_DDR_PI_65_PI_SWLVL_RD_SLICE_0_FLDSHFT (8)
+#define HWIO_DDR_PI_65_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_65_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_65_PI_SWLVL_VREF_UPDATE_SLICE_0_FLDMASK (0x10000)
+#define HWIO_DDR_PI_65_PI_SWLVL_VREF_UPDATE_SLICE_0_FLDSHFT (16)
+#define HWIO_DDR_PI_65_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_65_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_65_PI_SW_WDQLVL_RESP_0_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_65_PI_SW_WDQLVL_RESP_0_FLDSHFT (24)
+#define HWIO_DDR_PI_65_RESERVED3_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_65_RESERVED3_FLDSHFT (26)
+
+#define HWIO_DDR_PI_66_REGOFF 0x108
+#define HWIO_DDR_PI_66_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_66_REGOFF)
+#define HWIO_DDR_PI_66_PI_SWLVL_WR_SLICE_1_FLDMASK (0x1)
+#define HWIO_DDR_PI_66_PI_SWLVL_WR_SLICE_1_FLDSHFT (0)
+#define HWIO_DDR_PI_66_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_66_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_66_PI_SWLVL_RD_SLICE_1_FLDMASK (0x100)
+#define HWIO_DDR_PI_66_PI_SWLVL_RD_SLICE_1_FLDSHFT (8)
+#define HWIO_DDR_PI_66_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_66_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_66_PI_SWLVL_VREF_UPDATE_SLICE_1_FLDMASK (0x10000)
+#define HWIO_DDR_PI_66_PI_SWLVL_VREF_UPDATE_SLICE_1_FLDSHFT (16)
+#define HWIO_DDR_PI_66_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_66_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_66_PI_SW_WDQLVL_RESP_1_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_66_PI_SW_WDQLVL_RESP_1_FLDSHFT (24)
+#define HWIO_DDR_PI_66_RESERVED3_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_66_RESERVED3_FLDSHFT (26)
+
+#define HWIO_DDR_PI_67_REGOFF 0x10c
+#define HWIO_DDR_PI_67_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_67_REGOFF)
+#define HWIO_DDR_PI_67_PI_SWLVL_WR_SLICE_2_FLDMASK (0x1)
+#define HWIO_DDR_PI_67_PI_SWLVL_WR_SLICE_2_FLDSHFT (0)
+#define HWIO_DDR_PI_67_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_67_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_67_PI_SWLVL_RD_SLICE_2_FLDMASK (0x100)
+#define HWIO_DDR_PI_67_PI_SWLVL_RD_SLICE_2_FLDSHFT (8)
+#define HWIO_DDR_PI_67_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_67_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_67_PI_SWLVL_VREF_UPDATE_SLICE_2_FLDMASK (0x10000)
+#define HWIO_DDR_PI_67_PI_SWLVL_VREF_UPDATE_SLICE_2_FLDSHFT (16)
+#define HWIO_DDR_PI_67_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_67_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_67_PI_SW_WDQLVL_RESP_2_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_67_PI_SW_WDQLVL_RESP_2_FLDSHFT (24)
+#define HWIO_DDR_PI_67_RESERVED3_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_67_RESERVED3_FLDSHFT (26)
+
+#define HWIO_DDR_PI_68_REGOFF 0x110
+#define HWIO_DDR_PI_68_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_68_REGOFF)
+#define HWIO_DDR_PI_68_PI_SWLVL_WR_SLICE_3_FLDMASK (0x1)
+#define HWIO_DDR_PI_68_PI_SWLVL_WR_SLICE_3_FLDSHFT (0)
+#define HWIO_DDR_PI_68_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_68_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_68_PI_SWLVL_RD_SLICE_3_FLDMASK (0x100)
+#define HWIO_DDR_PI_68_PI_SWLVL_RD_SLICE_3_FLDSHFT (8)
+#define HWIO_DDR_PI_68_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_68_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_68_PI_SWLVL_VREF_UPDATE_SLICE_3_FLDMASK (0x10000)
+#define HWIO_DDR_PI_68_PI_SWLVL_VREF_UPDATE_SLICE_3_FLDSHFT (16)
+#define HWIO_DDR_PI_68_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_68_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_68_PI_SW_WDQLVL_RESP_3_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_68_PI_SW_WDQLVL_RESP_3_FLDSHFT (24)
+#define HWIO_DDR_PI_68_RESERVED3_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_68_RESERVED3_FLDSHFT (26)
+
+#define HWIO_DDR_PI_69_REGOFF 0x114
+#define HWIO_DDR_PI_69_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_69_REGOFF)
+#define HWIO_DDR_PI_69_PI_SW_WDQLVL_VREF_FLDMASK (0x7f)
+#define HWIO_DDR_PI_69_PI_SW_WDQLVL_VREF_FLDSHFT (0)
+#define HWIO_DDR_PI_69_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_PI_69_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_PI_69_PI_SWLVL_SM2_START_FLDMASK (0x100)
+#define HWIO_DDR_PI_69_PI_SWLVL_SM2_START_FLDSHFT (8)
+#define HWIO_DDR_PI_69_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_69_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_69_PI_SWLVL_SM2_WR_FLDMASK (0x10000)
+#define HWIO_DDR_PI_69_PI_SWLVL_SM2_WR_FLDSHFT (16)
+#define HWIO_DDR_PI_69_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_69_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_69_PI_SWLVL_SM2_RD_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_69_PI_SWLVL_SM2_RD_FLDSHFT (24)
+#define HWIO_DDR_PI_69_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_69_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_70_REGOFF 0x118
+#define HWIO_DDR_PI_70_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_70_REGOFF)
+#define HWIO_DDR_PI_70_PI_SEQUENTIAL_LVL_REQ_FLDMASK (0x1)
+#define HWIO_DDR_PI_70_PI_SEQUENTIAL_LVL_REQ_FLDSHFT (0)
+#define HWIO_DDR_PI_70_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_70_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_70_PI_DFS_PERIOD_EN_FLDMASK (0x100)
+#define HWIO_DDR_PI_70_PI_DFS_PERIOD_EN_FLDSHFT (8)
+#define HWIO_DDR_PI_70_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_70_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_70_PI_SRE_PERIOD_EN_FLDMASK (0x10000)
+#define HWIO_DDR_PI_70_PI_SRE_PERIOD_EN_FLDSHFT (16)
+#define HWIO_DDR_PI_70_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_70_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_70_PI_DFI40_POLARITY_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_70_PI_DFI40_POLARITY_FLDSHFT (24)
+#define HWIO_DDR_PI_70_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_70_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_71_REGOFF 0x11c
+#define HWIO_DDR_PI_71_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_71_REGOFF)
+#define HWIO_DDR_PI_71_PI_16BIT_DRAM_CONNECT_FLDMASK (0x1)
+#define HWIO_DDR_PI_71_PI_16BIT_DRAM_CONNECT_FLDSHFT (0)
+#define HWIO_DDR_PI_71_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_71_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_71_PI_TDFI_CTRL_DELAY_F0_FLDMASK (0xf00)
+#define HWIO_DDR_PI_71_PI_TDFI_CTRL_DELAY_F0_FLDSHFT (8)
+#define HWIO_DDR_PI_71_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PI_71_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PI_71_PI_TDFI_CTRL_DELAY_F1_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_71_PI_TDFI_CTRL_DELAY_F1_FLDSHFT (16)
+#define HWIO_DDR_PI_71_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_71_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PI_71_PI_TDFI_CTRL_DELAY_F2_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_71_PI_TDFI_CTRL_DELAY_F2_FLDSHFT (24)
+#define HWIO_DDR_PI_71_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_71_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PI_72_REGOFF 0x120
+#define HWIO_DDR_PI_72_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_72_REGOFF)
+#define HWIO_DDR_PI_72_PI_TDFI_CTRL_DELAY_F3_FLDMASK (0xf)
+#define HWIO_DDR_PI_72_PI_TDFI_CTRL_DELAY_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_72_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PI_72_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PI_72_PI_WRLVL_REQ_FLDMASK (0x100)
+#define HWIO_DDR_PI_72_PI_WRLVL_REQ_FLDSHFT (8)
+#define HWIO_DDR_PI_72_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_72_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_72_PI_WRLVL_CS_FLDMASK (0x10000)
+#define HWIO_DDR_PI_72_PI_WRLVL_CS_FLDSHFT (16)
+#define HWIO_DDR_PI_72_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_72_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_72_PI_WLDQSEN_FLDMASK (0x3f000000)
+#define HWIO_DDR_PI_72_PI_WLDQSEN_FLDSHFT (24)
+#define HWIO_DDR_PI_72_RESERVED3_FLDMASK (0xc0000000)
+#define HWIO_DDR_PI_72_RESERVED3_FLDSHFT (30)
+
+#define HWIO_DDR_PI_73_REGOFF 0x124
+#define HWIO_DDR_PI_73_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_73_REGOFF)
+#define HWIO_DDR_PI_73_PI_WLMRD_FLDMASK (0x3f)
+#define HWIO_DDR_PI_73_PI_WLMRD_FLDSHFT (0)
+#define HWIO_DDR_PI_73_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PI_73_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PI_73_PI_WRLVL_EN_F0_FLDMASK (0x300)
+#define HWIO_DDR_PI_73_PI_WRLVL_EN_F0_FLDSHFT (8)
+#define HWIO_DDR_PI_73_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_73_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_73_PI_WRLVL_EN_F1_FLDMASK (0x30000)
+#define HWIO_DDR_PI_73_PI_WRLVL_EN_F1_FLDSHFT (16)
+#define HWIO_DDR_PI_73_RESERVED2_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_73_RESERVED2_FLDSHFT (18)
+#define HWIO_DDR_PI_73_PI_WRLVL_EN_F2_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_73_PI_WRLVL_EN_F2_FLDSHFT (24)
+#define HWIO_DDR_PI_73_RESERVED3_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_73_RESERVED3_FLDSHFT (26)
+
+#define HWIO_DDR_PI_74_REGOFF 0x128
+#define HWIO_DDR_PI_74_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_74_REGOFF)
+#define HWIO_DDR_PI_74_PI_WRLVL_EN_F3_FLDMASK (0x3)
+#define HWIO_DDR_PI_74_PI_WRLVL_EN_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_74_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_74_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_74_PI_WRLVL_INTERVAL_FLDMASK (0xffff00)
+#define HWIO_DDR_PI_74_PI_WRLVL_INTERVAL_FLDSHFT (8)
+#define HWIO_DDR_PI_74_PI_WRLVL_PERIODIC_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_74_PI_WRLVL_PERIODIC_FLDSHFT (24)
+#define HWIO_DDR_PI_74_RESERVED1_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_74_RESERVED1_FLDSHFT (25)
+
+#define HWIO_DDR_PI_75_REGOFF 0x12c
+#define HWIO_DDR_PI_75_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_75_REGOFF)
+#define HWIO_DDR_PI_75_PI_WRLVL_ON_SREF_EXIT_FLDMASK (0x1)
+#define HWIO_DDR_PI_75_PI_WRLVL_ON_SREF_EXIT_FLDSHFT (0)
+#define HWIO_DDR_PI_75_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_75_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_75_PI_WRLVL_DISABLE_DFS_FLDMASK (0x100)
+#define HWIO_DDR_PI_75_PI_WRLVL_DISABLE_DFS_FLDSHFT (8)
+#define HWIO_DDR_PI_75_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_75_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_75_PI_WRLVL_RESP_MASK_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_75_PI_WRLVL_RESP_MASK_FLDSHFT (16)
+#define HWIO_DDR_PI_75_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_75_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PI_75_PI_WRLVL_ROTATE_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_75_PI_WRLVL_ROTATE_FLDSHFT (24)
+#define HWIO_DDR_PI_75_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_75_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_76_REGOFF 0x130
+#define HWIO_DDR_PI_76_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_76_REGOFF)
+#define HWIO_DDR_PI_76_PI_WRLVL_CS_MAP_FLDMASK (0x3)
+#define HWIO_DDR_PI_76_PI_WRLVL_CS_MAP_FLDSHFT (0)
+#define HWIO_DDR_PI_76_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_76_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_76_PI_WRLVL_ERROR_STATUS_FLDMASK (0x300)
+#define HWIO_DDR_PI_76_PI_WRLVL_ERROR_STATUS_FLDSHFT (8)
+#define HWIO_DDR_PI_76_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_76_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_76_PI_TDFI_WRLVL_EN_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_76_PI_TDFI_WRLVL_EN_FLDSHFT (16)
+#define HWIO_DDR_PI_76_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_76_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PI_77_REGOFF 0x134
+#define HWIO_DDR_PI_77_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_77_REGOFF)
+#define HWIO_DDR_PI_77_PI_TDFI_WRLVL_WW_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_77_PI_TDFI_WRLVL_WW_FLDSHFT (0)
+#define HWIO_DDR_PI_77_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_77_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_77_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_77_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_78_REGOFF 0x138
+#define HWIO_DDR_PI_78_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_78_REGOFF)
+#define HWIO_DDR_PI_78_PI_TDFI_WRLVL_RESP_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_78_PI_TDFI_WRLVL_RESP_FLDSHFT (0)
+
+#define HWIO_DDR_PI_79_REGOFF 0x13c
+#define HWIO_DDR_PI_79_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_79_REGOFF)
+#define HWIO_DDR_PI_79_PI_TDFI_WRLVL_MAX_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_79_PI_TDFI_WRLVL_MAX_FLDSHFT (0)
+
+#define HWIO_DDR_PI_80_REGOFF 0x140
+#define HWIO_DDR_PI_80_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_80_REGOFF)
+#define HWIO_DDR_PI_80_PI_WRLVL_STROBE_NUM_FLDMASK (0x1f)
+#define HWIO_DDR_PI_80_PI_WRLVL_STROBE_NUM_FLDSHFT (0)
+#define HWIO_DDR_PI_80_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PI_80_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PI_80_RESERVED1_FLDMASK (0x100)
+#define HWIO_DDR_PI_80_RESERVED1_FLDSHFT (8)
+#define HWIO_DDR_PI_80_RESERVED3_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_80_RESERVED3_FLDSHFT (9)
+#define HWIO_DDR_PI_80_RESERVED2_FLDMASK (0x1f0000)
+#define HWIO_DDR_PI_80_RESERVED2_FLDSHFT (16)
+#define HWIO_DDR_PI_80_RESERVED4_FLDMASK (0xe00000)
+#define HWIO_DDR_PI_80_RESERVED4_FLDSHFT (21)
+#define HWIO_DDR_PI_80_PI_TODTL_2CMD_F0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_80_PI_TODTL_2CMD_F0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_81_REGOFF 0x144
+#define HWIO_DDR_PI_81_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_81_REGOFF)
+#define HWIO_DDR_PI_81_PI_ODT_EN_F0_FLDMASK (0x1)
+#define HWIO_DDR_PI_81_PI_ODT_EN_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_81_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_81_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_81_PI_TODTL_2CMD_F1_FLDMASK (0xff00)
+#define HWIO_DDR_PI_81_PI_TODTL_2CMD_F1_FLDSHFT (8)
+#define HWIO_DDR_PI_81_PI_ODT_EN_F1_FLDMASK (0x10000)
+#define HWIO_DDR_PI_81_PI_ODT_EN_F1_FLDSHFT (16)
+#define HWIO_DDR_PI_81_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_81_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PI_81_PI_TODTL_2CMD_F2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_81_PI_TODTL_2CMD_F2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_82_REGOFF 0x148
+#define HWIO_DDR_PI_82_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_82_REGOFF)
+#define HWIO_DDR_PI_82_PI_ODT_EN_F2_FLDMASK (0x1)
+#define HWIO_DDR_PI_82_PI_ODT_EN_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_82_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_82_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_82_PI_TODTL_2CMD_F3_FLDMASK (0xff00)
+#define HWIO_DDR_PI_82_PI_TODTL_2CMD_F3_FLDSHFT (8)
+#define HWIO_DDR_PI_82_PI_ODT_EN_F3_FLDMASK (0x10000)
+#define HWIO_DDR_PI_82_PI_ODT_EN_F3_FLDSHFT (16)
+#define HWIO_DDR_PI_82_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_82_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PI_82_PI_TODTH_WR_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_82_PI_TODTH_WR_FLDSHFT (24)
+#define HWIO_DDR_PI_82_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_82_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PI_83_REGOFF 0x14c
+#define HWIO_DDR_PI_83_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_83_REGOFF)
+#define HWIO_DDR_PI_83_PI_TODTH_RD_FLDMASK (0xf)
+#define HWIO_DDR_PI_83_PI_TODTH_RD_FLDSHFT (0)
+#define HWIO_DDR_PI_83_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PI_83_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PI_83_PI_ODT_RD_MAP_CS0_FLDMASK (0x300)
+#define HWIO_DDR_PI_83_PI_ODT_RD_MAP_CS0_FLDSHFT (8)
+#define HWIO_DDR_PI_83_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_83_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_83_PI_ODT_WR_MAP_CS0_FLDMASK (0x30000)
+#define HWIO_DDR_PI_83_PI_ODT_WR_MAP_CS0_FLDSHFT (16)
+#define HWIO_DDR_PI_83_RESERVED2_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_83_RESERVED2_FLDSHFT (18)
+#define HWIO_DDR_PI_83_PI_ODT_RD_MAP_CS1_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_83_PI_ODT_RD_MAP_CS1_FLDSHFT (24)
+#define HWIO_DDR_PI_83_RESERVED3_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_83_RESERVED3_FLDSHFT (26)
+
+#define HWIO_DDR_PI_84_REGOFF 0x150
+#define HWIO_DDR_PI_84_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_84_REGOFF)
+#define HWIO_DDR_PI_84_PI_ODT_WR_MAP_CS1_FLDMASK (0x3)
+#define HWIO_DDR_PI_84_PI_ODT_WR_MAP_CS1_FLDSHFT (0)
+#define HWIO_DDR_PI_84_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_84_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_84_PI_EN_ODT_ASSERT_EXCEPT_RD_FLDMASK (0x100)
+#define HWIO_DDR_PI_84_PI_EN_ODT_ASSERT_EXCEPT_RD_FLDSHFT (8)
+#define HWIO_DDR_PI_84_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_84_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_84_PI_WR_TO_ODTH_F0_FLDMASK (0x3f0000)
+#define HWIO_DDR_PI_84_PI_WR_TO_ODTH_F0_FLDSHFT (16)
+#define HWIO_DDR_PI_84_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PI_84_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PI_84_PI_WR_TO_ODTH_F1_FLDMASK (0x3f000000)
+#define HWIO_DDR_PI_84_PI_WR_TO_ODTH_F1_FLDSHFT (24)
+#define HWIO_DDR_PI_84_RESERVED3_FLDMASK (0xc0000000)
+#define HWIO_DDR_PI_84_RESERVED3_FLDSHFT (30)
+
+#define HWIO_DDR_PI_85_REGOFF 0x154
+#define HWIO_DDR_PI_85_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_85_REGOFF)
+#define HWIO_DDR_PI_85_PI_WR_TO_ODTH_F2_FLDMASK (0x3f)
+#define HWIO_DDR_PI_85_PI_WR_TO_ODTH_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_85_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PI_85_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PI_85_PI_WR_TO_ODTH_F3_FLDMASK (0x3f00)
+#define HWIO_DDR_PI_85_PI_WR_TO_ODTH_F3_FLDSHFT (8)
+#define HWIO_DDR_PI_85_RESERVED1_FLDMASK (0xc000)
+#define HWIO_DDR_PI_85_RESERVED1_FLDSHFT (14)
+#define HWIO_DDR_PI_85_PI_RD_TO_ODTH_F0_FLDMASK (0x3f0000)
+#define HWIO_DDR_PI_85_PI_RD_TO_ODTH_F0_FLDSHFT (16)
+#define HWIO_DDR_PI_85_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PI_85_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PI_85_PI_RD_TO_ODTH_F1_FLDMASK (0x3f000000)
+#define HWIO_DDR_PI_85_PI_RD_TO_ODTH_F1_FLDSHFT (24)
+#define HWIO_DDR_PI_85_RESERVED3_FLDMASK (0xc0000000)
+#define HWIO_DDR_PI_85_RESERVED3_FLDSHFT (30)
+
+#define HWIO_DDR_PI_86_REGOFF 0x158
+#define HWIO_DDR_PI_86_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_86_REGOFF)
+#define HWIO_DDR_PI_86_PI_RD_TO_ODTH_F2_FLDMASK (0x3f)
+#define HWIO_DDR_PI_86_PI_RD_TO_ODTH_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_86_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PI_86_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PI_86_PI_RD_TO_ODTH_F3_FLDMASK (0x3f00)
+#define HWIO_DDR_PI_86_PI_RD_TO_ODTH_F3_FLDSHFT (8)
+#define HWIO_DDR_PI_86_RESERVED1_FLDMASK (0xc000)
+#define HWIO_DDR_PI_86_RESERVED1_FLDSHFT (14)
+#define HWIO_DDR_PI_86_PI_RDLVL_REQ_FLDMASK (0x10000)
+#define HWIO_DDR_PI_86_PI_RDLVL_REQ_FLDSHFT (16)
+#define HWIO_DDR_PI_86_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_86_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_86_PI_RDLVL_GATE_REQ_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_86_PI_RDLVL_GATE_REQ_FLDSHFT (24)
+#define HWIO_DDR_PI_86_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_86_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_87_REGOFF 0x15c
+#define HWIO_DDR_PI_87_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_87_REGOFF)
+#define HWIO_DDR_PI_87_PI_RDLVL_CS_FLDMASK (0x1)
+#define HWIO_DDR_PI_87_PI_RDLVL_CS_FLDSHFT (0)
+#define HWIO_DDR_PI_87_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_87_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_87_PI_RDLVL_SEQ_EN_FLDMASK (0xf00)
+#define HWIO_DDR_PI_87_PI_RDLVL_SEQ_EN_FLDSHFT (8)
+#define HWIO_DDR_PI_87_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PI_87_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PI_87_PI_RDLVL_GATE_SEQ_EN_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_87_PI_RDLVL_GATE_SEQ_EN_FLDSHFT (16)
+#define HWIO_DDR_PI_87_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_87_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PI_87_PI_RDLVL_PERIODIC_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_87_PI_RDLVL_PERIODIC_FLDSHFT (24)
+#define HWIO_DDR_PI_87_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_87_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_88_REGOFF 0x160
+#define HWIO_DDR_PI_88_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_88_REGOFF)
+#define HWIO_DDR_PI_88_PI_RDLVL_ON_SREF_EXIT_FLDMASK (0x1)
+#define HWIO_DDR_PI_88_PI_RDLVL_ON_SREF_EXIT_FLDSHFT (0)
+#define HWIO_DDR_PI_88_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_88_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_88_PI_RDLVL_DISABLE_DFS_FLDMASK (0x100)
+#define HWIO_DDR_PI_88_PI_RDLVL_DISABLE_DFS_FLDSHFT (8)
+#define HWIO_DDR_PI_88_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_88_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_88_PI_RDLVL_GATE_PERIODIC_FLDMASK (0x10000)
+#define HWIO_DDR_PI_88_PI_RDLVL_GATE_PERIODIC_FLDSHFT (16)
+#define HWIO_DDR_PI_88_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_88_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_88_PI_RDLVL_GATE_ON_SREF_EXIT_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_88_PI_RDLVL_GATE_ON_SREF_EXIT_FLDSHFT (24)
+#define HWIO_DDR_PI_88_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_88_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_89_REGOFF 0x164
+#define HWIO_DDR_PI_89_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_89_REGOFF)
+#define HWIO_DDR_PI_89_PI_RDLVL_GATE_DISABLE_DFS_FLDMASK (0x1)
+#define HWIO_DDR_PI_89_PI_RDLVL_GATE_DISABLE_DFS_FLDSHFT (0)
+#define HWIO_DDR_PI_89_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_89_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_89_RESERVED1_FLDMASK (0x100)
+#define HWIO_DDR_PI_89_RESERVED1_FLDSHFT (8)
+#define HWIO_DDR_PI_89_RESERVED2_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_89_RESERVED2_FLDSHFT (9)
+#define HWIO_DDR_PI_89_PI_RDLVL_ROTATE_FLDMASK (0x10000)
+#define HWIO_DDR_PI_89_PI_RDLVL_ROTATE_FLDSHFT (16)
+#define HWIO_DDR_PI_89_RESERVED3_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_89_RESERVED3_FLDSHFT (17)
+#define HWIO_DDR_PI_89_PI_RDLVL_GATE_ROTATE_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_89_PI_RDLVL_GATE_ROTATE_FLDSHFT (24)
+#define HWIO_DDR_PI_89_RESERVED4_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_89_RESERVED4_FLDSHFT (25)
+
+#define HWIO_DDR_PI_90_REGOFF 0x168
+#define HWIO_DDR_PI_90_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_90_REGOFF)
+#define HWIO_DDR_PI_90_PI_RDLVL_CS_MAP_FLDMASK (0x3)
+#define HWIO_DDR_PI_90_PI_RDLVL_CS_MAP_FLDSHFT (0)
+#define HWIO_DDR_PI_90_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_90_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_90_PI_RDLVL_GATE_CS_MAP_FLDMASK (0x300)
+#define HWIO_DDR_PI_90_PI_RDLVL_GATE_CS_MAP_FLDSHFT (8)
+#define HWIO_DDR_PI_90_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_90_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_90_PI_TDFI_RDLVL_RR_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_90_PI_TDFI_RDLVL_RR_FLDSHFT (16)
+#define HWIO_DDR_PI_90_RESERVED2_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_90_RESERVED2_FLDSHFT (26)
+
+#define HWIO_DDR_PI_91_REGOFF 0x16c
+#define HWIO_DDR_PI_91_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_91_REGOFF)
+#define HWIO_DDR_PI_91_PI_TDFI_RDLVL_RESP_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_91_PI_TDFI_RDLVL_RESP_FLDSHFT (0)
+
+#define HWIO_DDR_PI_92_REGOFF 0x170
+#define HWIO_DDR_PI_92_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_92_REGOFF)
+#define HWIO_DDR_PI_92_PI_RDLVL_RESP_MASK_FLDMASK (0xf)
+#define HWIO_DDR_PI_92_PI_RDLVL_RESP_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_92_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PI_92_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PI_92_PI_TDFI_RDLVL_EN_FLDMASK (0xff00)
+#define HWIO_DDR_PI_92_PI_TDFI_RDLVL_EN_FLDSHFT (8)
+#define HWIO_DDR_PI_92_PI_RDLVL_EN_F0_FLDMASK (0x30000)
+#define HWIO_DDR_PI_92_PI_RDLVL_EN_F0_FLDSHFT (16)
+#define HWIO_DDR_PI_92_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_92_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PI_92_PI_RDLVL_GATE_EN_F0_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_92_PI_RDLVL_GATE_EN_F0_FLDSHFT (24)
+#define HWIO_DDR_PI_92_RESERVED2_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_92_RESERVED2_FLDSHFT (26)
+
+#define HWIO_DDR_PI_93_REGOFF 0x174
+#define HWIO_DDR_PI_93_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_93_REGOFF)
+#define HWIO_DDR_PI_93_PI_RDLVL_EN_F1_FLDMASK (0x3)
+#define HWIO_DDR_PI_93_PI_RDLVL_EN_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_93_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_93_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_93_PI_RDLVL_GATE_EN_F1_FLDMASK (0x300)
+#define HWIO_DDR_PI_93_PI_RDLVL_GATE_EN_F1_FLDSHFT (8)
+#define HWIO_DDR_PI_93_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_93_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_93_PI_RDLVL_EN_F2_FLDMASK (0x30000)
+#define HWIO_DDR_PI_93_PI_RDLVL_EN_F2_FLDSHFT (16)
+#define HWIO_DDR_PI_93_RESERVED2_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_93_RESERVED2_FLDSHFT (18)
+#define HWIO_DDR_PI_93_PI_RDLVL_GATE_EN_F2_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_93_PI_RDLVL_GATE_EN_F2_FLDSHFT (24)
+#define HWIO_DDR_PI_93_RESERVED3_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_93_RESERVED3_FLDSHFT (26)
+
+#define HWIO_DDR_PI_94_REGOFF 0x178
+#define HWIO_DDR_PI_94_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_94_REGOFF)
+#define HWIO_DDR_PI_94_PI_RDLVL_EN_F3_FLDMASK (0x3)
+#define HWIO_DDR_PI_94_PI_RDLVL_EN_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_94_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_94_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_94_PI_RDLVL_GATE_EN_F3_FLDMASK (0x300)
+#define HWIO_DDR_PI_94_PI_RDLVL_GATE_EN_F3_FLDSHFT (8)
+#define HWIO_DDR_PI_94_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_94_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_94_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_94_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PI_95_REGOFF 0x17c
+#define HWIO_DDR_PI_95_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_95_REGOFF)
+#define HWIO_DDR_PI_95_PI_TDFI_RDLVL_MAX_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_95_PI_TDFI_RDLVL_MAX_FLDSHFT (0)
+
+#define HWIO_DDR_PI_96_REGOFF 0x180
+#define HWIO_DDR_PI_96_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_96_REGOFF)
+#define HWIO_DDR_PI_96_PI_RDLVL_ERROR_STATUS_FLDMASK (0x3)
+#define HWIO_DDR_PI_96_PI_RDLVL_ERROR_STATUS_FLDSHFT (0)
+#define HWIO_DDR_PI_96_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_96_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_96_PI_RDLVL_INTERVAL_FLDMASK (0xffff00)
+#define HWIO_DDR_PI_96_PI_RDLVL_INTERVAL_FLDSHFT (8)
+#define HWIO_DDR_PI_96_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_96_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_97_REGOFF 0x184
+#define HWIO_DDR_PI_97_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_97_REGOFF)
+#define HWIO_DDR_PI_97_PI_RDLVL_GATE_INTERVAL_FLDMASK (0xffff)
+#define HWIO_DDR_PI_97_PI_RDLVL_GATE_INTERVAL_FLDSHFT (0)
+#define HWIO_DDR_PI_97_PI_RDLVL_PATTERN_START_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_97_PI_RDLVL_PATTERN_START_FLDSHFT (16)
+#define HWIO_DDR_PI_97_RESERVED_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_97_RESERVED_FLDSHFT (20)
+#define HWIO_DDR_PI_97_PI_RDLVL_PATTERN_NUM_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_97_PI_RDLVL_PATTERN_NUM_FLDSHFT (24)
+#define HWIO_DDR_PI_97_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_97_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PI_98_REGOFF 0x188
+#define HWIO_DDR_PI_98_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_98_REGOFF)
+#define HWIO_DDR_PI_98_PI_RDLVL_STROBE_NUM_FLDMASK (0x1f)
+#define HWIO_DDR_PI_98_PI_RDLVL_STROBE_NUM_FLDSHFT (0)
+#define HWIO_DDR_PI_98_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PI_98_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PI_98_PI_RDLVL_GATE_STROBE_NUM_FLDMASK (0x1f00)
+#define HWIO_DDR_PI_98_PI_RDLVL_GATE_STROBE_NUM_FLDSHFT (8)
+#define HWIO_DDR_PI_98_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PI_98_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PI_98_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_98_OBSOLETE2_FLDSHFT (16)
+
+#define HWIO_DDR_PI_99_REGOFF 0x18c
+#define HWIO_DDR_PI_99_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_99_REGOFF)
+#define HWIO_DDR_PI_99_PI_LPDDR4_RDLVL_PATTERN_8_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_99_PI_LPDDR4_RDLVL_PATTERN_8_FLDSHFT (0)
+
+#define HWIO_DDR_PI_100_REGOFF 0x190
+#define HWIO_DDR_PI_100_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_100_REGOFF)
+#define HWIO_DDR_PI_100_PI_LPDDR4_RDLVL_PATTERN_9_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_100_PI_LPDDR4_RDLVL_PATTERN_9_FLDSHFT (0)
+
+#define HWIO_DDR_PI_101_REGOFF 0x194
+#define HWIO_DDR_PI_101_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_101_REGOFF)
+#define HWIO_DDR_PI_101_PI_LPDDR4_RDLVL_PATTERN_10_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_101_PI_LPDDR4_RDLVL_PATTERN_10_FLDSHFT (0)
+
+#define HWIO_DDR_PI_102_REGOFF 0x198
+#define HWIO_DDR_PI_102_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_102_REGOFF)
+#define HWIO_DDR_PI_102_PI_LPDDR4_RDLVL_PATTERN_11_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_102_PI_LPDDR4_RDLVL_PATTERN_11_FLDSHFT (0)
+
+#define HWIO_DDR_PI_103_REGOFF 0x19c
+#define HWIO_DDR_PI_103_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_103_REGOFF)
+#define HWIO_DDR_PI_103_PI_RD_PREAMBLE_TRAINING_EN_FLDMASK (0x1)
+#define HWIO_DDR_PI_103_PI_RD_PREAMBLE_TRAINING_EN_FLDSHFT (0)
+#define HWIO_DDR_PI_103_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_103_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_103_PI_REG_DIMM_ENABLE_FLDMASK (0x100)
+#define HWIO_DDR_PI_103_PI_REG_DIMM_ENABLE_FLDSHFT (8)
+#define HWIO_DDR_PI_103_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_103_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_103_PI_RDLAT_ADJ_F0_FLDMASK (0x7f0000)
+#define HWIO_DDR_PI_103_PI_RDLAT_ADJ_F0_FLDSHFT (16)
+#define HWIO_DDR_PI_103_RESERVED2_FLDMASK (0x800000)
+#define HWIO_DDR_PI_103_RESERVED2_FLDSHFT (23)
+#define HWIO_DDR_PI_103_PI_RDLAT_ADJ_F1_FLDMASK (0x7f000000)
+#define HWIO_DDR_PI_103_PI_RDLAT_ADJ_F1_FLDSHFT (24)
+#define HWIO_DDR_PI_103_RESERVED3_FLDMASK (0x80000000)
+#define HWIO_DDR_PI_103_RESERVED3_FLDSHFT (31)
+
+#define HWIO_DDR_PI_104_REGOFF 0x1a0
+#define HWIO_DDR_PI_104_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_104_REGOFF)
+#define HWIO_DDR_PI_104_PI_RDLAT_ADJ_F2_FLDMASK (0x7f)
+#define HWIO_DDR_PI_104_PI_RDLAT_ADJ_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_104_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_PI_104_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_PI_104_PI_RDLAT_ADJ_F3_FLDMASK (0x7f00)
+#define HWIO_DDR_PI_104_PI_RDLAT_ADJ_F3_FLDSHFT (8)
+#define HWIO_DDR_PI_104_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PI_104_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PI_104_PI_TDFI_RDDATA_EN_FLDMASK (0x7f0000)
+#define HWIO_DDR_PI_104_PI_TDFI_RDDATA_EN_FLDSHFT (16)
+#define HWIO_DDR_PI_104_RESERVED2_FLDMASK (0x800000)
+#define HWIO_DDR_PI_104_RESERVED2_FLDSHFT (23)
+#define HWIO_DDR_PI_104_PI_WRLAT_ADJ_F0_FLDMASK (0x7f000000)
+#define HWIO_DDR_PI_104_PI_WRLAT_ADJ_F0_FLDSHFT (24)
+#define HWIO_DDR_PI_104_RESERVED3_FLDMASK (0x80000000)
+#define HWIO_DDR_PI_104_RESERVED3_FLDSHFT (31)
+
+#define HWIO_DDR_PI_105_REGOFF 0x1a4
+#define HWIO_DDR_PI_105_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_105_REGOFF)
+#define HWIO_DDR_PI_105_PI_WRLAT_ADJ_F1_FLDMASK (0x7f)
+#define HWIO_DDR_PI_105_PI_WRLAT_ADJ_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_105_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_PI_105_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_PI_105_PI_WRLAT_ADJ_F2_FLDMASK (0x7f00)
+#define HWIO_DDR_PI_105_PI_WRLAT_ADJ_F2_FLDSHFT (8)
+#define HWIO_DDR_PI_105_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PI_105_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PI_105_PI_WRLAT_ADJ_F3_FLDMASK (0x7f0000)
+#define HWIO_DDR_PI_105_PI_WRLAT_ADJ_F3_FLDSHFT (16)
+#define HWIO_DDR_PI_105_RESERVED2_FLDMASK (0x800000)
+#define HWIO_DDR_PI_105_RESERVED2_FLDSHFT (23)
+#define HWIO_DDR_PI_105_PI_TDFI_PHY_WRLAT_FLDMASK (0x7f000000)
+#define HWIO_DDR_PI_105_PI_TDFI_PHY_WRLAT_FLDSHFT (24)
+#define HWIO_DDR_PI_105_RESERVED3_FLDMASK (0x80000000)
+#define HWIO_DDR_PI_105_RESERVED3_FLDSHFT (31)
+
+#define HWIO_DDR_PI_106_REGOFF 0x1a8
+#define HWIO_DDR_PI_106_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_106_REGOFF)
+#define HWIO_DDR_PI_106_PI_TDFI_RDCSLAT_F0_FLDMASK (0x7f)
+#define HWIO_DDR_PI_106_PI_TDFI_RDCSLAT_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_106_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_PI_106_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_PI_106_PI_TDFI_RDCSLAT_F1_FLDMASK (0x7f00)
+#define HWIO_DDR_PI_106_PI_TDFI_RDCSLAT_F1_FLDSHFT (8)
+#define HWIO_DDR_PI_106_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PI_106_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PI_106_PI_TDFI_RDCSLAT_F2_FLDMASK (0x7f0000)
+#define HWIO_DDR_PI_106_PI_TDFI_RDCSLAT_F2_FLDSHFT (16)
+#define HWIO_DDR_PI_106_RESERVED2_FLDMASK (0x800000)
+#define HWIO_DDR_PI_106_RESERVED2_FLDSHFT (23)
+#define HWIO_DDR_PI_106_PI_TDFI_RDCSLAT_F3_FLDMASK (0x7f000000)
+#define HWIO_DDR_PI_106_PI_TDFI_RDCSLAT_F3_FLDSHFT (24)
+#define HWIO_DDR_PI_106_RESERVED3_FLDMASK (0x80000000)
+#define HWIO_DDR_PI_106_RESERVED3_FLDSHFT (31)
+
+#define HWIO_DDR_PI_107_REGOFF 0x1ac
+#define HWIO_DDR_PI_107_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_107_REGOFF)
+#define HWIO_DDR_PI_107_PI_TDFI_PHY_WRDATA_F0_FLDMASK (0x7)
+#define HWIO_DDR_PI_107_PI_TDFI_PHY_WRDATA_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_107_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PI_107_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PI_107_PI_TDFI_PHY_WRDATA_F1_FLDMASK (0x700)
+#define HWIO_DDR_PI_107_PI_TDFI_PHY_WRDATA_F1_FLDSHFT (8)
+#define HWIO_DDR_PI_107_RESERVED1_FLDMASK (0xf800)
+#define HWIO_DDR_PI_107_RESERVED1_FLDSHFT (11)
+#define HWIO_DDR_PI_107_PI_TDFI_PHY_WRDATA_F2_FLDMASK (0x70000)
+#define HWIO_DDR_PI_107_PI_TDFI_PHY_WRDATA_F2_FLDSHFT (16)
+#define HWIO_DDR_PI_107_RESERVED2_FLDMASK (0xf80000)
+#define HWIO_DDR_PI_107_RESERVED2_FLDSHFT (19)
+#define HWIO_DDR_PI_107_PI_TDFI_PHY_WRDATA_F3_FLDMASK (0x7000000)
+#define HWIO_DDR_PI_107_PI_TDFI_PHY_WRDATA_F3_FLDSHFT (24)
+#define HWIO_DDR_PI_107_RESERVED3_FLDMASK (0xf8000000)
+#define HWIO_DDR_PI_107_RESERVED3_FLDSHFT (27)
+
+#define HWIO_DDR_PI_108_REGOFF 0x1b0
+#define HWIO_DDR_PI_108_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_108_REGOFF)
+#define HWIO_DDR_PI_108_PI_CALVL_REQ_FLDMASK (0x1)
+#define HWIO_DDR_PI_108_PI_CALVL_REQ_FLDSHFT (0)
+#define HWIO_DDR_PI_108_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_108_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_108_PI_CALVL_CS_FLDMASK (0x100)
+#define HWIO_DDR_PI_108_PI_CALVL_CS_FLDSHFT (8)
+#define HWIO_DDR_PI_108_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_108_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_108_RESERVED2_FLDMASK (0x10000)
+#define HWIO_DDR_PI_108_RESERVED2_FLDSHFT (16)
+#define HWIO_DDR_PI_108_RESERVED4_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_108_RESERVED4_FLDSHFT (17)
+#define HWIO_DDR_PI_108_RESERVED3_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_108_RESERVED3_FLDSHFT (24)
+#define HWIO_DDR_PI_108_RESERVED5_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_108_RESERVED5_FLDSHFT (28)
+
+#define HWIO_DDR_PI_109_REGOFF 0x1b4
+#define HWIO_DDR_PI_109_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_109_REGOFF)
+#define HWIO_DDR_PI_109_PI_CALVL_SEQ_EN_FLDMASK (0x3)
+#define HWIO_DDR_PI_109_PI_CALVL_SEQ_EN_FLDSHFT (0)
+#define HWIO_DDR_PI_109_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_109_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_109_PI_CALVL_PERIODIC_FLDMASK (0x100)
+#define HWIO_DDR_PI_109_PI_CALVL_PERIODIC_FLDSHFT (8)
+#define HWIO_DDR_PI_109_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_109_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_109_PI_CALVL_ON_SREF_EXIT_FLDMASK (0x10000)
+#define HWIO_DDR_PI_109_PI_CALVL_ON_SREF_EXIT_FLDSHFT (16)
+#define HWIO_DDR_PI_109_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_109_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_109_PI_CALVL_DISABLE_DFS_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_109_PI_CALVL_DISABLE_DFS_FLDSHFT (24)
+#define HWIO_DDR_PI_109_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_109_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_110_REGOFF 0x1b8
+#define HWIO_DDR_PI_110_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_110_REGOFF)
+#define HWIO_DDR_PI_110_PI_CALVL_ROTATE_FLDMASK (0x1)
+#define HWIO_DDR_PI_110_PI_CALVL_ROTATE_FLDSHFT (0)
+#define HWIO_DDR_PI_110_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_110_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_110_PI_CALVL_CS_MAP_FLDMASK (0x300)
+#define HWIO_DDR_PI_110_PI_CALVL_CS_MAP_FLDSHFT (8)
+#define HWIO_DDR_PI_110_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_110_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_110_PI_TDFI_CALVL_EN_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_110_PI_TDFI_CALVL_EN_FLDSHFT (16)
+#define HWIO_DDR_PI_110_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_110_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PI_111_REGOFF 0x1bc
+#define HWIO_DDR_PI_111_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_111_REGOFF)
+#define HWIO_DDR_PI_111_PI_TDFI_CALVL_CC_F0_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_111_PI_TDFI_CALVL_CC_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_111_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_111_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_111_PI_TDFI_CALVL_CAPTURE_F0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_111_PI_TDFI_CALVL_CAPTURE_F0_FLDSHFT (16)
+#define HWIO_DDR_PI_111_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_111_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PI_112_REGOFF 0x1c0
+#define HWIO_DDR_PI_112_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_112_REGOFF)
+#define HWIO_DDR_PI_112_PI_TDFI_CALVL_CC_F1_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_112_PI_TDFI_CALVL_CC_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_112_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_112_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_112_PI_TDFI_CALVL_CAPTURE_F1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_112_PI_TDFI_CALVL_CAPTURE_F1_FLDSHFT (16)
+#define HWIO_DDR_PI_112_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_112_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PI_113_REGOFF 0x1c4
+#define HWIO_DDR_PI_113_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_113_REGOFF)
+#define HWIO_DDR_PI_113_PI_TDFI_CALVL_CC_F2_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_113_PI_TDFI_CALVL_CC_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_113_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_113_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_113_PI_TDFI_CALVL_CAPTURE_F2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_113_PI_TDFI_CALVL_CAPTURE_F2_FLDSHFT (16)
+#define HWIO_DDR_PI_113_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_113_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PI_114_REGOFF 0x1c8
+#define HWIO_DDR_PI_114_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_114_REGOFF)
+#define HWIO_DDR_PI_114_PI_TDFI_CALVL_CC_F3_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_114_PI_TDFI_CALVL_CC_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_114_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_114_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_114_PI_TDFI_CALVL_CAPTURE_F3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_114_PI_TDFI_CALVL_CAPTURE_F3_FLDSHFT (16)
+#define HWIO_DDR_PI_114_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_114_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PI_115_REGOFF 0x1cc
+#define HWIO_DDR_PI_115_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_115_REGOFF)
+#define HWIO_DDR_PI_115_PI_TDFI_CALVL_RESP_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_115_PI_TDFI_CALVL_RESP_FLDSHFT (0)
+
+#define HWIO_DDR_PI_116_REGOFF 0x1d0
+#define HWIO_DDR_PI_116_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_116_REGOFF)
+#define HWIO_DDR_PI_116_PI_TDFI_CALVL_MAX_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_116_PI_TDFI_CALVL_MAX_FLDSHFT (0)
+
+#define HWIO_DDR_PI_117_REGOFF 0x1d4
+#define HWIO_DDR_PI_117_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_117_REGOFF)
+#define HWIO_DDR_PI_117_PI_CALVL_RESP_MASK_FLDMASK (0x1)
+#define HWIO_DDR_PI_117_PI_CALVL_RESP_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_117_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_117_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_117_PI_CALVL_EN_F0_FLDMASK (0x300)
+#define HWIO_DDR_PI_117_PI_CALVL_EN_F0_FLDSHFT (8)
+#define HWIO_DDR_PI_117_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_117_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_117_PI_CALVL_EN_F1_FLDMASK (0x30000)
+#define HWIO_DDR_PI_117_PI_CALVL_EN_F1_FLDSHFT (16)
+#define HWIO_DDR_PI_117_RESERVED2_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_117_RESERVED2_FLDSHFT (18)
+#define HWIO_DDR_PI_117_PI_CALVL_EN_F2_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_117_PI_CALVL_EN_F2_FLDSHFT (24)
+#define HWIO_DDR_PI_117_RESERVED3_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_117_RESERVED3_FLDSHFT (26)
+
+#define HWIO_DDR_PI_118_REGOFF 0x1d8
+#define HWIO_DDR_PI_118_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_118_REGOFF)
+#define HWIO_DDR_PI_118_PI_CALVL_EN_F3_FLDMASK (0x3)
+#define HWIO_DDR_PI_118_PI_CALVL_EN_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_118_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_118_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_118_PI_CALVL_ERROR_STATUS_FLDMASK (0x300)
+#define HWIO_DDR_PI_118_PI_CALVL_ERROR_STATUS_FLDSHFT (8)
+#define HWIO_DDR_PI_118_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_118_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_118_PI_CALVL_INTERVAL_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_118_PI_CALVL_INTERVAL_FLDSHFT (16)
+
+#define HWIO_DDR_PI_119_REGOFF 0x1dc
+#define HWIO_DDR_PI_119_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_119_REGOFF)
+#define HWIO_DDR_PI_119_PI_TCACKEL_FLDMASK (0x1f)
+#define HWIO_DDR_PI_119_PI_TCACKEL_FLDSHFT (0)
+#define HWIO_DDR_PI_119_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PI_119_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PI_119_PI_TCAMRD_FLDMASK (0x3f00)
+#define HWIO_DDR_PI_119_PI_TCAMRD_FLDSHFT (8)
+#define HWIO_DDR_PI_119_RESERVED1_FLDMASK (0xc000)
+#define HWIO_DDR_PI_119_RESERVED1_FLDSHFT (14)
+#define HWIO_DDR_PI_119_PI_TCACKEH_FLDMASK (0x1f0000)
+#define HWIO_DDR_PI_119_PI_TCACKEH_FLDSHFT (16)
+#define HWIO_DDR_PI_119_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PI_119_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PI_119_PI_TMRZ_F0_FLDMASK (0x1f000000)
+#define HWIO_DDR_PI_119_PI_TMRZ_F0_FLDSHFT (24)
+#define HWIO_DDR_PI_119_RESERVED3_FLDMASK (0xe0000000)
+#define HWIO_DDR_PI_119_RESERVED3_FLDSHFT (29)
+
+#define HWIO_DDR_PI_120_REGOFF 0x1e0
+#define HWIO_DDR_PI_120_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_120_REGOFF)
+#define HWIO_DDR_PI_120_PI_TCAENT_F0_FLDMASK (0x3fff)
+#define HWIO_DDR_PI_120_PI_TCAENT_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_120_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_PI_120_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_PI_120_PI_TMRZ_F1_FLDMASK (0x1f0000)
+#define HWIO_DDR_PI_120_PI_TMRZ_F1_FLDSHFT (16)
+#define HWIO_DDR_PI_120_RESERVED1_FLDMASK (0xe00000)
+#define HWIO_DDR_PI_120_RESERVED1_FLDSHFT (21)
+#define HWIO_DDR_PI_120_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_120_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_121_REGOFF 0x1e4
+#define HWIO_DDR_PI_121_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_121_REGOFF)
+#define HWIO_DDR_PI_121_PI_TCAENT_F1_FLDMASK (0x3fff)
+#define HWIO_DDR_PI_121_PI_TCAENT_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_121_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_PI_121_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_PI_121_PI_TMRZ_F2_FLDMASK (0x1f0000)
+#define HWIO_DDR_PI_121_PI_TMRZ_F2_FLDSHFT (16)
+#define HWIO_DDR_PI_121_RESERVED1_FLDMASK (0xe00000)
+#define HWIO_DDR_PI_121_RESERVED1_FLDSHFT (21)
+#define HWIO_DDR_PI_121_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_121_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_122_REGOFF 0x1e8
+#define HWIO_DDR_PI_122_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_122_REGOFF)
+#define HWIO_DDR_PI_122_PI_TCAENT_F2_FLDMASK (0x3fff)
+#define HWIO_DDR_PI_122_PI_TCAENT_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_122_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_PI_122_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_PI_122_PI_TMRZ_F3_FLDMASK (0x1f0000)
+#define HWIO_DDR_PI_122_PI_TMRZ_F3_FLDSHFT (16)
+#define HWIO_DDR_PI_122_RESERVED1_FLDMASK (0xe00000)
+#define HWIO_DDR_PI_122_RESERVED1_FLDSHFT (21)
+#define HWIO_DDR_PI_122_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_122_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_123_REGOFF 0x1ec
+#define HWIO_DDR_PI_123_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_123_REGOFF)
+#define HWIO_DDR_PI_123_PI_TCAENT_F3_FLDMASK (0x3fff)
+#define HWIO_DDR_PI_123_PI_TCAENT_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_123_RESERVED_FLDMASK (0xc000)
+#define HWIO_DDR_PI_123_RESERVED_FLDSHFT (14)
+#define HWIO_DDR_PI_123_PI_TCAEXT_FLDMASK (0x1f0000)
+#define HWIO_DDR_PI_123_PI_TCAEXT_FLDSHFT (16)
+#define HWIO_DDR_PI_123_RESERVED1_FLDMASK (0xe00000)
+#define HWIO_DDR_PI_123_RESERVED1_FLDSHFT (21)
+#define HWIO_DDR_PI_123_PI_CA_TRAIN_VREF_EN_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_123_PI_CA_TRAIN_VREF_EN_FLDSHFT (24)
+#define HWIO_DDR_PI_123_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_123_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PI_124_REGOFF 0x1f0
+#define HWIO_DDR_PI_124_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_124_REGOFF)
+#define HWIO_DDR_PI_124_PI_TDFI_CACSCA_F0_FLDMASK (0x1f)
+#define HWIO_DDR_PI_124_PI_TDFI_CACSCA_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_124_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PI_124_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PI_124_PI_TDFI_CASEL_F0_FLDMASK (0x1f00)
+#define HWIO_DDR_PI_124_PI_TDFI_CASEL_F0_FLDSHFT (8)
+#define HWIO_DDR_PI_124_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PI_124_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PI_124_PI_TVREF_SHORT_F0_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_124_PI_TVREF_SHORT_F0_FLDSHFT (16)
+#define HWIO_DDR_PI_124_RESERVED2_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_124_RESERVED2_FLDSHFT (26)
+
+#define HWIO_DDR_PI_125_REGOFF 0x1f4
+#define HWIO_DDR_PI_125_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_125_REGOFF)
+#define HWIO_DDR_PI_125_PI_TVREF_LONG_F0_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_125_PI_TVREF_LONG_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_125_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_125_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_125_PI_TDFI_CACSCA_F1_FLDMASK (0x1f0000)
+#define HWIO_DDR_PI_125_PI_TDFI_CACSCA_F1_FLDSHFT (16)
+#define HWIO_DDR_PI_125_RESERVED1_FLDMASK (0xe00000)
+#define HWIO_DDR_PI_125_RESERVED1_FLDSHFT (21)
+#define HWIO_DDR_PI_125_PI_TDFI_CASEL_F1_FLDMASK (0x1f000000)
+#define HWIO_DDR_PI_125_PI_TDFI_CASEL_F1_FLDSHFT (24)
+#define HWIO_DDR_PI_125_RESERVED2_FLDMASK (0xe0000000)
+#define HWIO_DDR_PI_125_RESERVED2_FLDSHFT (29)
+
+#define HWIO_DDR_PI_126_REGOFF 0x1f8
+#define HWIO_DDR_PI_126_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_126_REGOFF)
+#define HWIO_DDR_PI_126_PI_TVREF_SHORT_F1_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_126_PI_TVREF_SHORT_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_126_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_126_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_126_PI_TVREF_LONG_F1_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_126_PI_TVREF_LONG_F1_FLDSHFT (16)
+#define HWIO_DDR_PI_126_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_126_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PI_127_REGOFF 0x1fc
+#define HWIO_DDR_PI_127_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_127_REGOFF)
+#define HWIO_DDR_PI_127_PI_TDFI_CACSCA_F2_FLDMASK (0x1f)
+#define HWIO_DDR_PI_127_PI_TDFI_CACSCA_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_127_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PI_127_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PI_127_PI_TDFI_CASEL_F2_FLDMASK (0x1f00)
+#define HWIO_DDR_PI_127_PI_TDFI_CASEL_F2_FLDSHFT (8)
+#define HWIO_DDR_PI_127_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PI_127_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PI_127_PI_TVREF_SHORT_F2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_127_PI_TVREF_SHORT_F2_FLDSHFT (16)
+#define HWIO_DDR_PI_127_RESERVED2_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_127_RESERVED2_FLDSHFT (26)
+
+#define HWIO_DDR_PI_128_REGOFF 0x200
+#define HWIO_DDR_PI_128_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_128_REGOFF)
+#define HWIO_DDR_PI_128_PI_TVREF_LONG_F2_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_128_PI_TVREF_LONG_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_128_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_128_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_128_PI_TDFI_CACSCA_F3_FLDMASK (0x1f0000)
+#define HWIO_DDR_PI_128_PI_TDFI_CACSCA_F3_FLDSHFT (16)
+#define HWIO_DDR_PI_128_RESERVED1_FLDMASK (0xe00000)
+#define HWIO_DDR_PI_128_RESERVED1_FLDSHFT (21)
+#define HWIO_DDR_PI_128_PI_TDFI_CASEL_F3_FLDMASK (0x1f000000)
+#define HWIO_DDR_PI_128_PI_TDFI_CASEL_F3_FLDSHFT (24)
+#define HWIO_DDR_PI_128_RESERVED2_FLDMASK (0xe0000000)
+#define HWIO_DDR_PI_128_RESERVED2_FLDSHFT (29)
+
+#define HWIO_DDR_PI_129_REGOFF 0x204
+#define HWIO_DDR_PI_129_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_129_REGOFF)
+#define HWIO_DDR_PI_129_PI_TVREF_SHORT_F3_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_129_PI_TVREF_SHORT_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_129_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_129_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_129_PI_TVREF_LONG_F3_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_129_PI_TVREF_LONG_F3_FLDSHFT (16)
+#define HWIO_DDR_PI_129_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_129_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PI_130_REGOFF 0x208
+#define HWIO_DDR_PI_130_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_130_REGOFF)
+#define HWIO_DDR_PI_130_PI_CALVL_VREF_INITIAL_START_POINT_FLDMASK (0x7f)
+#define HWIO_DDR_PI_130_PI_CALVL_VREF_INITIAL_START_POINT_FLDSHFT (0)
+#define HWIO_DDR_PI_130_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_PI_130_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_PI_130_PI_CALVL_VREF_INITIAL_STOP_POINT_FLDMASK (0x7f00)
+#define HWIO_DDR_PI_130_PI_CALVL_VREF_INITIAL_STOP_POINT_FLDSHFT (8)
+#define HWIO_DDR_PI_130_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PI_130_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PI_130_PI_CALVL_VREF_INITIAL_STEPSIZE_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_130_PI_CALVL_VREF_INITIAL_STEPSIZE_FLDSHFT (16)
+#define HWIO_DDR_PI_130_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_130_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PI_130_PI_CALVL_VREF_NORMAL_STEPSIZE_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_130_PI_CALVL_VREF_NORMAL_STEPSIZE_FLDSHFT (24)
+#define HWIO_DDR_PI_130_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_130_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PI_131_REGOFF 0x20c
+#define HWIO_DDR_PI_131_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_131_REGOFF)
+#define HWIO_DDR_PI_131_PI_CALVL_VREF_DELTA_FLDMASK (0xf)
+#define HWIO_DDR_PI_131_PI_CALVL_VREF_DELTA_FLDSHFT (0)
+#define HWIO_DDR_PI_131_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PI_131_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PI_131_PI_TDFI_INIT_START_MIN_FLDMASK (0xff00)
+#define HWIO_DDR_PI_131_PI_TDFI_INIT_START_MIN_FLDSHFT (8)
+#define HWIO_DDR_PI_131_PI_TDFI_INIT_COMPLETE_MIN_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_131_PI_TDFI_INIT_COMPLETE_MIN_FLDSHFT (16)
+#define HWIO_DDR_PI_131_PI_TDFI_CALVL_STROBE_F0_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_131_PI_TDFI_CALVL_STROBE_F0_FLDSHFT (24)
+#define HWIO_DDR_PI_131_RESERVED1_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_131_RESERVED1_FLDSHFT (28)
+
+#define HWIO_DDR_PI_132_REGOFF 0x210
+#define HWIO_DDR_PI_132_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_132_REGOFF)
+#define HWIO_DDR_PI_132_PI_TDFI_CALVL_STROBE_F1_FLDMASK (0xf)
+#define HWIO_DDR_PI_132_PI_TDFI_CALVL_STROBE_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_132_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PI_132_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PI_132_PI_TDFI_CALVL_STROBE_F2_FLDMASK (0xf00)
+#define HWIO_DDR_PI_132_PI_TDFI_CALVL_STROBE_F2_FLDSHFT (8)
+#define HWIO_DDR_PI_132_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PI_132_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PI_132_PI_TDFI_CALVL_STROBE_F3_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_132_PI_TDFI_CALVL_STROBE_F3_FLDSHFT (16)
+#define HWIO_DDR_PI_132_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_132_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PI_132_PI_TCKCKEH_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_132_PI_TCKCKEH_FLDSHFT (24)
+#define HWIO_DDR_PI_132_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_132_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PI_133_REGOFF 0x214
+#define HWIO_DDR_PI_133_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_133_REGOFF)
+#define HWIO_DDR_PI_133_PI_CALVL_STROBE_NUM_FLDMASK (0x1f)
+#define HWIO_DDR_PI_133_PI_CALVL_STROBE_NUM_FLDSHFT (0)
+#define HWIO_DDR_PI_133_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PI_133_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PI_133_PI_SW_CA_TRAIN_VREF_FLDMASK (0x7f00)
+#define HWIO_DDR_PI_133_PI_SW_CA_TRAIN_VREF_FLDSHFT (8)
+#define HWIO_DDR_PI_133_RESERVED1_FLDMASK (0x8000)
+#define HWIO_DDR_PI_133_RESERVED1_FLDSHFT (15)
+#define HWIO_DDR_PI_133_PI_TDFI_INIT_START_F0_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_133_PI_TDFI_INIT_START_F0_FLDSHFT (16)
+#define HWIO_DDR_PI_133_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_133_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PI_134_REGOFF 0x218
+#define HWIO_DDR_PI_134_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_134_REGOFF)
+#define HWIO_DDR_PI_134_PI_TDFI_INIT_COMPLETE_F0_FLDMASK (0xffff)
+#define HWIO_DDR_PI_134_PI_TDFI_INIT_COMPLETE_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_134_PI_TDFI_INIT_START_F1_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_134_PI_TDFI_INIT_START_F1_FLDSHFT (16)
+#define HWIO_DDR_PI_134_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_134_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_135_REGOFF 0x21c
+#define HWIO_DDR_PI_135_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_135_REGOFF)
+#define HWIO_DDR_PI_135_PI_TDFI_INIT_COMPLETE_F1_FLDMASK (0xffff)
+#define HWIO_DDR_PI_135_PI_TDFI_INIT_COMPLETE_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_135_PI_TDFI_INIT_START_F2_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_135_PI_TDFI_INIT_START_F2_FLDSHFT (16)
+#define HWIO_DDR_PI_135_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_135_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_136_REGOFF 0x220
+#define HWIO_DDR_PI_136_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_136_REGOFF)
+#define HWIO_DDR_PI_136_PI_TDFI_INIT_COMPLETE_F2_FLDMASK (0xffff)
+#define HWIO_DDR_PI_136_PI_TDFI_INIT_COMPLETE_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_136_PI_TDFI_INIT_START_F3_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_136_PI_TDFI_INIT_START_F3_FLDSHFT (16)
+#define HWIO_DDR_PI_136_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_136_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_137_REGOFF 0x224
+#define HWIO_DDR_PI_137_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_137_REGOFF)
+#define HWIO_DDR_PI_137_PI_TDFI_INIT_COMPLETE_F3_FLDMASK (0xffff)
+#define HWIO_DDR_PI_137_PI_TDFI_INIT_COMPLETE_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_137_PI_CLKDISABLE_2_INIT_START_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_137_PI_CLKDISABLE_2_INIT_START_FLDSHFT (16)
+#define HWIO_DDR_PI_137_PI_INIT_STARTORCOMPLETE_2_CLKDISABLE_FLDMASK \
+ (0xff000000)
+#define HWIO_DDR_PI_137_PI_INIT_STARTORCOMPLETE_2_CLKDISABLE_FLDSHFT (24)
+
+#define HWIO_DDR_PI_138_REGOFF 0x228
+#define HWIO_DDR_PI_138_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_138_REGOFF)
+#define HWIO_DDR_PI_138_PI_DRAM_CLK_DISABLE_DEASSERT_SEL_FLDMASK (0x1)
+#define HWIO_DDR_PI_138_PI_DRAM_CLK_DISABLE_DEASSERT_SEL_FLDSHFT (0)
+#define HWIO_DDR_PI_138_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_138_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_138_PI_REFRESH_BETWEEN_SEGMENT_DISABLE_FLDMASK (0x100)
+#define HWIO_DDR_PI_138_PI_REFRESH_BETWEEN_SEGMENT_DISABLE_FLDSHFT (8)
+#define HWIO_DDR_PI_138_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_138_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_138_PI_TCKEHDQS_F0_FLDMASK (0x3f0000)
+#define HWIO_DDR_PI_138_PI_TCKEHDQS_F0_FLDSHFT (16)
+#define HWIO_DDR_PI_138_RESERVED2_FLDMASK (0xc00000)
+#define HWIO_DDR_PI_138_RESERVED2_FLDSHFT (22)
+#define HWIO_DDR_PI_138_PI_TCKEHDQS_F1_FLDMASK (0x3f000000)
+#define HWIO_DDR_PI_138_PI_TCKEHDQS_F1_FLDSHFT (24)
+#define HWIO_DDR_PI_138_RESERVED3_FLDMASK (0xc0000000)
+#define HWIO_DDR_PI_138_RESERVED3_FLDSHFT (30)
+
+#define HWIO_DDR_PI_139_REGOFF 0x22c
+#define HWIO_DDR_PI_139_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_139_REGOFF)
+#define HWIO_DDR_PI_139_PI_TCKEHDQS_F2_FLDMASK (0x3f)
+#define HWIO_DDR_PI_139_PI_TCKEHDQS_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_139_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PI_139_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PI_139_PI_TCKEHDQS_F3_FLDMASK (0x3f00)
+#define HWIO_DDR_PI_139_PI_TCKEHDQS_F3_FLDSHFT (8)
+#define HWIO_DDR_PI_139_RESERVED1_FLDMASK (0xc000)
+#define HWIO_DDR_PI_139_RESERVED1_FLDSHFT (14)
+#define HWIO_DDR_PI_139_PI_MC_DFS_PI_SET_VREF_ENABLE_FLDMASK (0x10000)
+#define HWIO_DDR_PI_139_PI_MC_DFS_PI_SET_VREF_ENABLE_FLDSHFT (16)
+#define HWIO_DDR_PI_139_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_139_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_139_PI_WDQLVL_VREF_EN_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_139_PI_WDQLVL_VREF_EN_FLDSHFT (24)
+#define HWIO_DDR_PI_139_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_139_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_140_REGOFF 0x230
+#define HWIO_DDR_PI_140_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_140_REGOFF)
+#define HWIO_DDR_PI_140_PI_WDQLVL_BST_NUM_FLDMASK (0x7)
+#define HWIO_DDR_PI_140_PI_WDQLVL_BST_NUM_FLDSHFT (0)
+#define HWIO_DDR_PI_140_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PI_140_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PI_140_PI_TDFI_WDQLVL_WR_F0_FLDMASK (0x3ff00)
+#define HWIO_DDR_PI_140_PI_TDFI_WDQLVL_WR_F0_FLDSHFT (8)
+#define HWIO_DDR_PI_140_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_140_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PI_140_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_140_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_141_REGOFF 0x234
+#define HWIO_DDR_PI_141_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_141_REGOFF)
+#define HWIO_DDR_PI_141_PI_TDFI_WDQLVL_WR_F1_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_141_PI_TDFI_WDQLVL_WR_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_141_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_141_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_141_PI_TDFI_WDQLVL_WR_F2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_141_PI_TDFI_WDQLVL_WR_F2_FLDSHFT (16)
+#define HWIO_DDR_PI_141_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_141_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PI_142_REGOFF 0x238
+#define HWIO_DDR_PI_142_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_142_REGOFF)
+#define HWIO_DDR_PI_142_PI_TDFI_WDQLVL_WR_F3_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_142_PI_TDFI_WDQLVL_WR_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_142_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_142_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_142_PI_TDFI_WDQLVL_RW_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_142_PI_TDFI_WDQLVL_RW_FLDSHFT (16)
+#define HWIO_DDR_PI_142_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_142_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PI_143_REGOFF 0x23c
+#define HWIO_DDR_PI_143_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_143_REGOFF)
+#define HWIO_DDR_PI_143_PI_WDQLVL_RESP_MASK_FLDMASK (0xf)
+#define HWIO_DDR_PI_143_PI_WDQLVL_RESP_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_143_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PI_143_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PI_143_PI_WDQLVL_ROTATE_FLDMASK (0x100)
+#define HWIO_DDR_PI_143_PI_WDQLVL_ROTATE_FLDSHFT (8)
+#define HWIO_DDR_PI_143_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_143_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_143_PI_WDQLVL_CS_MAP_FLDMASK (0x30000)
+#define HWIO_DDR_PI_143_PI_WDQLVL_CS_MAP_FLDSHFT (16)
+#define HWIO_DDR_PI_143_RESERVED2_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_143_RESERVED2_FLDSHFT (18)
+#define HWIO_DDR_PI_143_PI_WDQLVL_VREF_INITIAL_START_POINT_FLDMASK (0x7f000000)
+#define HWIO_DDR_PI_143_PI_WDQLVL_VREF_INITIAL_START_POINT_FLDSHFT (24)
+#define HWIO_DDR_PI_143_RESERVED3_FLDMASK (0x80000000)
+#define HWIO_DDR_PI_143_RESERVED3_FLDSHFT (31)
+
+#define HWIO_DDR_PI_144_REGOFF 0x240
+#define HWIO_DDR_PI_144_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_144_REGOFF)
+#define HWIO_DDR_PI_144_PI_WDQLVL_VREF_INITIAL_STOP_POINT_FLDMASK (0x7f)
+#define HWIO_DDR_PI_144_PI_WDQLVL_VREF_INITIAL_STOP_POINT_FLDSHFT (0)
+#define HWIO_DDR_PI_144_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_PI_144_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_PI_144_PI_WDQLVL_VREF_INITIAL_STEPSIZE_FLDMASK (0x1f00)
+#define HWIO_DDR_PI_144_PI_WDQLVL_VREF_INITIAL_STEPSIZE_FLDSHFT (8)
+#define HWIO_DDR_PI_144_RESERVED1_FLDMASK (0xe000)
+#define HWIO_DDR_PI_144_RESERVED1_FLDSHFT (13)
+#define HWIO_DDR_PI_144_PI_WDQLVL_VREF_NORMAL_STEPSIZE_FLDMASK (0x1f0000)
+#define HWIO_DDR_PI_144_PI_WDQLVL_VREF_NORMAL_STEPSIZE_FLDSHFT (16)
+#define HWIO_DDR_PI_144_RESERVED2_FLDMASK (0xe00000)
+#define HWIO_DDR_PI_144_RESERVED2_FLDSHFT (21)
+#define HWIO_DDR_PI_144_PI_WDQLVL_VREF_DELTA_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_144_PI_WDQLVL_VREF_DELTA_FLDSHFT (24)
+#define HWIO_DDR_PI_144_RESERVED3_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_144_RESERVED3_FLDSHFT (28)
+
+#define HWIO_DDR_PI_145_REGOFF 0x244
+#define HWIO_DDR_PI_145_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_145_REGOFF)
+#define HWIO_DDR_PI_145_PI_WDQLVL_PERIODIC_FLDMASK (0x1)
+#define HWIO_DDR_PI_145_PI_WDQLVL_PERIODIC_FLDSHFT (0)
+#define HWIO_DDR_PI_145_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_145_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_145_PI_WDQLVL_REQ_FLDMASK (0x100)
+#define HWIO_DDR_PI_145_PI_WDQLVL_REQ_FLDSHFT (8)
+#define HWIO_DDR_PI_145_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_145_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_145_PI_WDQLVL_CS_FLDMASK (0x10000)
+#define HWIO_DDR_PI_145_PI_WDQLVL_CS_FLDSHFT (16)
+#define HWIO_DDR_PI_145_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_145_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_145_PI_TDFI_WDQLVL_EN_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_145_PI_TDFI_WDQLVL_EN_FLDSHFT (24)
+
+#define HWIO_DDR_PI_146_REGOFF 0x248
+#define HWIO_DDR_PI_146_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_146_REGOFF)
+#define HWIO_DDR_PI_146_PI_TDFI_WDQLVL_RESP_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_146_PI_TDFI_WDQLVL_RESP_FLDSHFT (0)
+
+#define HWIO_DDR_PI_147_REGOFF 0x24c
+#define HWIO_DDR_PI_147_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_147_REGOFF)
+#define HWIO_DDR_PI_147_PI_TDFI_WDQLVL_MAX_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_147_PI_TDFI_WDQLVL_MAX_FLDSHFT (0)
+
+#define HWIO_DDR_PI_148_REGOFF 0x250
+#define HWIO_DDR_PI_148_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_148_REGOFF)
+#define HWIO_DDR_PI_148_PI_WDQLVL_INTERVAL_FLDMASK (0xffff)
+#define HWIO_DDR_PI_148_PI_WDQLVL_INTERVAL_FLDSHFT (0)
+#define HWIO_DDR_PI_148_PI_WDQLVL_EN_F0_FLDMASK (0x30000)
+#define HWIO_DDR_PI_148_PI_WDQLVL_EN_F0_FLDSHFT (16)
+#define HWIO_DDR_PI_148_RESERVED_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_148_RESERVED_FLDSHFT (18)
+#define HWIO_DDR_PI_148_PI_WDQLVL_EN_F1_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_148_PI_WDQLVL_EN_F1_FLDSHFT (24)
+#define HWIO_DDR_PI_148_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_148_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PI_149_REGOFF 0x254
+#define HWIO_DDR_PI_149_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_149_REGOFF)
+#define HWIO_DDR_PI_149_PI_WDQLVL_EN_F2_FLDMASK (0x3)
+#define HWIO_DDR_PI_149_PI_WDQLVL_EN_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_149_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_149_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_149_PI_WDQLVL_EN_F3_FLDMASK (0x300)
+#define HWIO_DDR_PI_149_PI_WDQLVL_EN_F3_FLDSHFT (8)
+#define HWIO_DDR_PI_149_RESERVED1_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_149_RESERVED1_FLDSHFT (10)
+#define HWIO_DDR_PI_149_PI_WDQLVL_ON_SREF_EXIT_FLDMASK (0x10000)
+#define HWIO_DDR_PI_149_PI_WDQLVL_ON_SREF_EXIT_FLDSHFT (16)
+#define HWIO_DDR_PI_149_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_149_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_149_PI_WDQLVL_DISABLE_DFS_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_149_PI_WDQLVL_DISABLE_DFS_FLDSHFT (24)
+#define HWIO_DDR_PI_149_RESERVED3_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_149_RESERVED3_FLDSHFT (25)
+
+#define HWIO_DDR_PI_150_REGOFF 0x258
+#define HWIO_DDR_PI_150_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_150_REGOFF)
+#define HWIO_DDR_PI_150_PI_WDQLVL_ERROR_STATUS_FLDMASK (0x3)
+#define HWIO_DDR_PI_150_PI_WDQLVL_ERROR_STATUS_FLDSHFT (0)
+#define HWIO_DDR_PI_150_RESERVED_FLDMASK (0xfc)
+#define HWIO_DDR_PI_150_RESERVED_FLDSHFT (2)
+#define HWIO_DDR_PI_150_PI_MR1_DATA_F0_0_FLDMASK (0xff00)
+#define HWIO_DDR_PI_150_PI_MR1_DATA_F0_0_FLDSHFT (8)
+#define HWIO_DDR_PI_150_PI_MR2_DATA_F0_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_150_PI_MR2_DATA_F0_0_FLDSHFT (16)
+#define HWIO_DDR_PI_150_PI_MR3_DATA_F0_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_150_PI_MR3_DATA_F0_0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_151_REGOFF 0x25c
+#define HWIO_DDR_PI_151_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_151_REGOFF)
+#define HWIO_DDR_PI_151_PI_MR11_DATA_F0_0_FLDMASK (0xff)
+#define HWIO_DDR_PI_151_PI_MR11_DATA_F0_0_FLDSHFT (0)
+#define HWIO_DDR_PI_151_PI_MR12_DATA_F0_0_FLDMASK (0xff00)
+#define HWIO_DDR_PI_151_PI_MR12_DATA_F0_0_FLDSHFT (8)
+#define HWIO_DDR_PI_151_PI_MR14_DATA_F0_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_151_PI_MR14_DATA_F0_0_FLDSHFT (16)
+#define HWIO_DDR_PI_151_PI_MR22_DATA_F0_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_151_PI_MR22_DATA_F0_0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_152_REGOFF 0x260
+#define HWIO_DDR_PI_152_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_152_REGOFF)
+#define HWIO_DDR_PI_152_PI_MR1_DATA_F1_0_FLDMASK (0xff)
+#define HWIO_DDR_PI_152_PI_MR1_DATA_F1_0_FLDSHFT (0)
+#define HWIO_DDR_PI_152_PI_MR2_DATA_F1_0_FLDMASK (0xff00)
+#define HWIO_DDR_PI_152_PI_MR2_DATA_F1_0_FLDSHFT (8)
+#define HWIO_DDR_PI_152_PI_MR3_DATA_F1_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_152_PI_MR3_DATA_F1_0_FLDSHFT (16)
+#define HWIO_DDR_PI_152_PI_MR11_DATA_F1_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_152_PI_MR11_DATA_F1_0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_153_REGOFF 0x264
+#define HWIO_DDR_PI_153_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_153_REGOFF)
+#define HWIO_DDR_PI_153_PI_MR12_DATA_F1_0_FLDMASK (0xff)
+#define HWIO_DDR_PI_153_PI_MR12_DATA_F1_0_FLDSHFT (0)
+#define HWIO_DDR_PI_153_PI_MR14_DATA_F1_0_FLDMASK (0xff00)
+#define HWIO_DDR_PI_153_PI_MR14_DATA_F1_0_FLDSHFT (8)
+#define HWIO_DDR_PI_153_PI_MR22_DATA_F1_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_153_PI_MR22_DATA_F1_0_FLDSHFT (16)
+#define HWIO_DDR_PI_153_PI_MR1_DATA_F2_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_153_PI_MR1_DATA_F2_0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_154_REGOFF 0x268
+#define HWIO_DDR_PI_154_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_154_REGOFF)
+#define HWIO_DDR_PI_154_PI_MR2_DATA_F2_0_FLDMASK (0xff)
+#define HWIO_DDR_PI_154_PI_MR2_DATA_F2_0_FLDSHFT (0)
+#define HWIO_DDR_PI_154_PI_MR3_DATA_F2_0_FLDMASK (0xff00)
+#define HWIO_DDR_PI_154_PI_MR3_DATA_F2_0_FLDSHFT (8)
+#define HWIO_DDR_PI_154_PI_MR11_DATA_F2_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_154_PI_MR11_DATA_F2_0_FLDSHFT (16)
+#define HWIO_DDR_PI_154_PI_MR12_DATA_F2_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_154_PI_MR12_DATA_F2_0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_155_REGOFF 0x26c
+#define HWIO_DDR_PI_155_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_155_REGOFF)
+#define HWIO_DDR_PI_155_PI_MR14_DATA_F2_0_FLDMASK (0xff)
+#define HWIO_DDR_PI_155_PI_MR14_DATA_F2_0_FLDSHFT (0)
+#define HWIO_DDR_PI_155_PI_MR22_DATA_F2_0_FLDMASK (0xff00)
+#define HWIO_DDR_PI_155_PI_MR22_DATA_F2_0_FLDSHFT (8)
+#define HWIO_DDR_PI_155_PI_MR1_DATA_F3_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_155_PI_MR1_DATA_F3_0_FLDSHFT (16)
+#define HWIO_DDR_PI_155_PI_MR2_DATA_F3_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_155_PI_MR2_DATA_F3_0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_156_REGOFF 0x270
+#define HWIO_DDR_PI_156_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_156_REGOFF)
+#define HWIO_DDR_PI_156_PI_MR3_DATA_F3_0_FLDMASK (0xff)
+#define HWIO_DDR_PI_156_PI_MR3_DATA_F3_0_FLDSHFT (0)
+#define HWIO_DDR_PI_156_PI_MR11_DATA_F3_0_FLDMASK (0xff00)
+#define HWIO_DDR_PI_156_PI_MR11_DATA_F3_0_FLDSHFT (8)
+#define HWIO_DDR_PI_156_PI_MR12_DATA_F3_0_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_156_PI_MR12_DATA_F3_0_FLDSHFT (16)
+#define HWIO_DDR_PI_156_PI_MR14_DATA_F3_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_156_PI_MR14_DATA_F3_0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_157_REGOFF 0x274
+#define HWIO_DDR_PI_157_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_157_REGOFF)
+#define HWIO_DDR_PI_157_PI_MR22_DATA_F3_0_FLDMASK (0xff)
+#define HWIO_DDR_PI_157_PI_MR22_DATA_F3_0_FLDSHFT (0)
+#define HWIO_DDR_PI_157_PI_MR13_DATA_0_FLDMASK (0xff00)
+#define HWIO_DDR_PI_157_PI_MR13_DATA_0_FLDSHFT (8)
+#define HWIO_DDR_PI_157_PI_MR1_DATA_F0_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_157_PI_MR1_DATA_F0_1_FLDSHFT (16)
+#define HWIO_DDR_PI_157_PI_MR2_DATA_F0_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_157_PI_MR2_DATA_F0_1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_158_REGOFF 0x278
+#define HWIO_DDR_PI_158_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_158_REGOFF)
+#define HWIO_DDR_PI_158_PI_MR3_DATA_F0_1_FLDMASK (0xff)
+#define HWIO_DDR_PI_158_PI_MR3_DATA_F0_1_FLDSHFT (0)
+#define HWIO_DDR_PI_158_PI_MR11_DATA_F0_1_FLDMASK (0xff00)
+#define HWIO_DDR_PI_158_PI_MR11_DATA_F0_1_FLDSHFT (8)
+#define HWIO_DDR_PI_158_PI_MR12_DATA_F0_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_158_PI_MR12_DATA_F0_1_FLDSHFT (16)
+#define HWIO_DDR_PI_158_PI_MR14_DATA_F0_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_158_PI_MR14_DATA_F0_1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_159_REGOFF 0x27c
+#define HWIO_DDR_PI_159_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_159_REGOFF)
+#define HWIO_DDR_PI_159_PI_MR22_DATA_F0_1_FLDMASK (0xff)
+#define HWIO_DDR_PI_159_PI_MR22_DATA_F0_1_FLDSHFT (0)
+#define HWIO_DDR_PI_159_PI_MR1_DATA_F1_1_FLDMASK (0xff00)
+#define HWIO_DDR_PI_159_PI_MR1_DATA_F1_1_FLDSHFT (8)
+#define HWIO_DDR_PI_159_PI_MR2_DATA_F1_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_159_PI_MR2_DATA_F1_1_FLDSHFT (16)
+#define HWIO_DDR_PI_159_PI_MR3_DATA_F1_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_159_PI_MR3_DATA_F1_1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_160_REGOFF 0x280
+#define HWIO_DDR_PI_160_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_160_REGOFF)
+#define HWIO_DDR_PI_160_PI_MR11_DATA_F1_1_FLDMASK (0xff)
+#define HWIO_DDR_PI_160_PI_MR11_DATA_F1_1_FLDSHFT (0)
+#define HWIO_DDR_PI_160_PI_MR12_DATA_F1_1_FLDMASK (0xff00)
+#define HWIO_DDR_PI_160_PI_MR12_DATA_F1_1_FLDSHFT (8)
+#define HWIO_DDR_PI_160_PI_MR14_DATA_F1_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_160_PI_MR14_DATA_F1_1_FLDSHFT (16)
+#define HWIO_DDR_PI_160_PI_MR22_DATA_F1_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_160_PI_MR22_DATA_F1_1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_161_REGOFF 0x284
+#define HWIO_DDR_PI_161_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_161_REGOFF)
+#define HWIO_DDR_PI_161_PI_MR1_DATA_F2_1_FLDMASK (0xff)
+#define HWIO_DDR_PI_161_PI_MR1_DATA_F2_1_FLDSHFT (0)
+#define HWIO_DDR_PI_161_PI_MR2_DATA_F2_1_FLDMASK (0xff00)
+#define HWIO_DDR_PI_161_PI_MR2_DATA_F2_1_FLDSHFT (8)
+#define HWIO_DDR_PI_161_PI_MR3_DATA_F2_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_161_PI_MR3_DATA_F2_1_FLDSHFT (16)
+#define HWIO_DDR_PI_161_PI_MR11_DATA_F2_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_161_PI_MR11_DATA_F2_1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_162_REGOFF 0x288
+#define HWIO_DDR_PI_162_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_162_REGOFF)
+#define HWIO_DDR_PI_162_PI_MR12_DATA_F2_1_FLDMASK (0xff)
+#define HWIO_DDR_PI_162_PI_MR12_DATA_F2_1_FLDSHFT (0)
+#define HWIO_DDR_PI_162_PI_MR14_DATA_F2_1_FLDMASK (0xff00)
+#define HWIO_DDR_PI_162_PI_MR14_DATA_F2_1_FLDSHFT (8)
+#define HWIO_DDR_PI_162_PI_MR22_DATA_F2_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_162_PI_MR22_DATA_F2_1_FLDSHFT (16)
+#define HWIO_DDR_PI_162_PI_MR1_DATA_F3_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_162_PI_MR1_DATA_F3_1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_163_REGOFF 0x28c
+#define HWIO_DDR_PI_163_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_163_REGOFF)
+#define HWIO_DDR_PI_163_PI_MR2_DATA_F3_1_FLDMASK (0xff)
+#define HWIO_DDR_PI_163_PI_MR2_DATA_F3_1_FLDSHFT (0)
+#define HWIO_DDR_PI_163_PI_MR3_DATA_F3_1_FLDMASK (0xff00)
+#define HWIO_DDR_PI_163_PI_MR3_DATA_F3_1_FLDSHFT (8)
+#define HWIO_DDR_PI_163_PI_MR11_DATA_F3_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_163_PI_MR11_DATA_F3_1_FLDSHFT (16)
+#define HWIO_DDR_PI_163_PI_MR12_DATA_F3_1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_163_PI_MR12_DATA_F3_1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_164_REGOFF 0x290
+#define HWIO_DDR_PI_164_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_164_REGOFF)
+#define HWIO_DDR_PI_164_PI_MR14_DATA_F3_1_FLDMASK (0xff)
+#define HWIO_DDR_PI_164_PI_MR14_DATA_F3_1_FLDSHFT (0)
+#define HWIO_DDR_PI_164_PI_MR22_DATA_F3_1_FLDMASK (0xff00)
+#define HWIO_DDR_PI_164_PI_MR22_DATA_F3_1_FLDSHFT (8)
+#define HWIO_DDR_PI_164_PI_MR13_DATA_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_164_PI_MR13_DATA_1_FLDSHFT (16)
+#define HWIO_DDR_PI_164_PI_BANK_DIFF_FLDMASK (0x3000000)
+#define HWIO_DDR_PI_164_PI_BANK_DIFF_FLDSHFT (24)
+#define HWIO_DDR_PI_164_RESERVED_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_164_RESERVED_FLDSHFT (26)
+
+#define HWIO_DDR_PI_165_REGOFF 0x294
+#define HWIO_DDR_PI_165_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_165_REGOFF)
+#define HWIO_DDR_PI_165_PI_ROW_DIFF_FLDMASK (0x7)
+#define HWIO_DDR_PI_165_PI_ROW_DIFF_FLDSHFT (0)
+#define HWIO_DDR_PI_165_RESERVED_FLDMASK (0xf8)
+#define HWIO_DDR_PI_165_RESERVED_FLDSHFT (3)
+#define HWIO_DDR_PI_165_PI_TFC_F0_FLDMASK (0x3ff00)
+#define HWIO_DDR_PI_165_PI_TFC_F0_FLDSHFT (8)
+#define HWIO_DDR_PI_165_RESERVED1_FLDMASK (0xfc0000)
+#define HWIO_DDR_PI_165_RESERVED1_FLDSHFT (18)
+#define HWIO_DDR_PI_165_OBSOLETE2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_165_OBSOLETE2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_166_REGOFF 0x298
+#define HWIO_DDR_PI_166_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_166_REGOFF)
+#define HWIO_DDR_PI_166_PI_TFC_F1_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_166_PI_TFC_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_166_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_166_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_166_PI_TFC_F2_FLDMASK (0x3ff0000)
+#define HWIO_DDR_PI_166_PI_TFC_F2_FLDSHFT (16)
+#define HWIO_DDR_PI_166_RESERVED1_FLDMASK (0xfc000000)
+#define HWIO_DDR_PI_166_RESERVED1_FLDSHFT (26)
+
+#define HWIO_DDR_PI_167_REGOFF 0x29c
+#define HWIO_DDR_PI_167_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_167_REGOFF)
+#define HWIO_DDR_PI_167_PI_TFC_F3_FLDMASK (0x3ff)
+#define HWIO_DDR_PI_167_PI_TFC_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_167_RESERVED_FLDMASK (0xfc00)
+#define HWIO_DDR_PI_167_RESERVED_FLDSHFT (10)
+#define HWIO_DDR_PI_167_PI_TMRD_F0_FLDMASK (0x3f0000)
+#define HWIO_DDR_PI_167_PI_TMRD_F0_FLDSHFT (16)
+#define HWIO_DDR_PI_167_RESERVED1_FLDMASK (0xc00000)
+#define HWIO_DDR_PI_167_RESERVED1_FLDSHFT (22)
+#define HWIO_DDR_PI_167_PI_TMRW_F0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_167_PI_TMRW_F0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_168_REGOFF 0x2a0
+#define HWIO_DDR_PI_168_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_168_REGOFF)
+#define HWIO_DDR_PI_168_PI_TMRD_F1_FLDMASK (0x3f)
+#define HWIO_DDR_PI_168_PI_TMRD_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_168_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PI_168_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PI_168_PI_TMRW_F1_FLDMASK (0xff00)
+#define HWIO_DDR_PI_168_PI_TMRW_F1_FLDSHFT (8)
+#define HWIO_DDR_PI_168_PI_TMRD_F2_FLDMASK (0x3f0000)
+#define HWIO_DDR_PI_168_PI_TMRD_F2_FLDSHFT (16)
+#define HWIO_DDR_PI_168_RESERVED1_FLDMASK (0xc00000)
+#define HWIO_DDR_PI_168_RESERVED1_FLDSHFT (22)
+#define HWIO_DDR_PI_168_PI_TMRW_F2_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_168_PI_TMRW_F2_FLDSHFT (24)
+
+#define HWIO_DDR_PI_169_REGOFF 0x2a4
+#define HWIO_DDR_PI_169_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_169_REGOFF)
+#define HWIO_DDR_PI_169_PI_TMRD_F3_FLDMASK (0x3f)
+#define HWIO_DDR_PI_169_PI_TMRD_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_169_RESERVED_FLDMASK (0xc0)
+#define HWIO_DDR_PI_169_RESERVED_FLDSHFT (6)
+#define HWIO_DDR_PI_169_PI_TMRW_F3_FLDMASK (0xff00)
+#define HWIO_DDR_PI_169_PI_TMRW_F3_FLDSHFT (8)
+#define HWIO_DDR_PI_169_RESERVED2_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_169_RESERVED2_FLDSHFT (16)
+#define HWIO_DDR_PI_169_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_169_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PI_169_RESERVED3_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_169_RESERVED3_FLDSHFT (24)
+#define HWIO_DDR_PI_169_RESERVED4_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_169_RESERVED4_FLDSHFT (28)
+
+#define HWIO_DDR_PI_170_REGOFF 0x2a8
+#define HWIO_DDR_PI_170_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_170_REGOFF)
+#define HWIO_DDR_PI_170_RESERVED0_FLDMASK (0xf)
+#define HWIO_DDR_PI_170_RESERVED0_FLDSHFT (0)
+#define HWIO_DDR_PI_170_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PI_170_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PI_170_RESERVED1_FLDMASK (0xf00)
+#define HWIO_DDR_PI_170_RESERVED1_FLDSHFT (8)
+#define HWIO_DDR_PI_170_RESERVED4_FLDMASK (0xf000)
+#define HWIO_DDR_PI_170_RESERVED4_FLDSHFT (12)
+#define HWIO_DDR_PI_170_RESERVED2_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_170_RESERVED2_FLDSHFT (16)
+#define HWIO_DDR_PI_170_RESERVED5_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_170_RESERVED5_FLDSHFT (20)
+#define HWIO_DDR_PI_170_RESERVED3_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_170_RESERVED3_FLDSHFT (24)
+#define HWIO_DDR_PI_170_RESERVED6_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_170_RESERVED6_FLDSHFT (28)
+
+#define HWIO_DDR_PI_171_REGOFF 0x2ac
+#define HWIO_DDR_PI_171_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_171_REGOFF)
+#define HWIO_DDR_PI_171_RESERVED0_FLDMASK (0xf)
+#define HWIO_DDR_PI_171_RESERVED0_FLDSHFT (0)
+#define HWIO_DDR_PI_171_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PI_171_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PI_171_RESERVED1_FLDMASK (0xf00)
+#define HWIO_DDR_PI_171_RESERVED1_FLDSHFT (8)
+#define HWIO_DDR_PI_171_RESERVED4_FLDMASK (0xf000)
+#define HWIO_DDR_PI_171_RESERVED4_FLDSHFT (12)
+#define HWIO_DDR_PI_171_RESERVED2_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_171_RESERVED2_FLDSHFT (16)
+#define HWIO_DDR_PI_171_RESERVED5_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_171_RESERVED5_FLDSHFT (20)
+#define HWIO_DDR_PI_171_RESERVED3_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_171_RESERVED3_FLDSHFT (24)
+#define HWIO_DDR_PI_171_RESERVED6_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_171_RESERVED6_FLDSHFT (28)
+
+#define HWIO_DDR_PI_172_REGOFF 0x2b0
+#define HWIO_DDR_PI_172_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_172_REGOFF)
+#define HWIO_DDR_PI_172_PI_INT_STATUS_FLDMASK (0x1ffff)
+#define HWIO_DDR_PI_172_PI_INT_STATUS_FLDSHFT (0)
+#define HWIO_DDR_PI_172_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_172_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PI_172_OBSOLETE1_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_172_OBSOLETE1_FLDSHFT (24)
+
+#define HWIO_DDR_PI_173_REGOFF 0x2b4
+#define HWIO_DDR_PI_173_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_173_REGOFF)
+#define HWIO_DDR_PI_173_PI_INT_ACK_FLDMASK (0xffff)
+#define HWIO_DDR_PI_173_PI_INT_ACK_FLDSHFT (0)
+#define HWIO_DDR_PI_173_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_173_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_174_REGOFF 0x2b8
+#define HWIO_DDR_PI_174_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_174_REGOFF)
+#define HWIO_DDR_PI_174_PI_INT_MASK_FLDMASK (0x1ffff)
+#define HWIO_DDR_PI_174_PI_INT_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_174_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_174_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PI_174_PI_BSTLEN_FLDMASK (0x1f000000)
+#define HWIO_DDR_PI_174_PI_BSTLEN_FLDSHFT (24)
+#define HWIO_DDR_PI_174_RESERVED1_FLDMASK (0xe0000000)
+#define HWIO_DDR_PI_174_RESERVED1_FLDSHFT (29)
+
+#define HWIO_DDR_PI_175_REGOFF 0x2bc
+#define HWIO_DDR_PI_175_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_175_REGOFF)
+#define HWIO_DDR_PI_175_PI_LONG_COUNT_MASK_FLDMASK (0x1f)
+#define HWIO_DDR_PI_175_PI_LONG_COUNT_MASK_FLDSHFT (0)
+#define HWIO_DDR_PI_175_RESERVED_FLDMASK (0xe0)
+#define HWIO_DDR_PI_175_RESERVED_FLDSHFT (5)
+#define HWIO_DDR_PI_175_PI_CTRLUPD_REQ_PER_AREF_EN_FLDMASK (0x100)
+#define HWIO_DDR_PI_175_PI_CTRLUPD_REQ_PER_AREF_EN_FLDSHFT (8)
+#define HWIO_DDR_PI_175_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_175_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_175_PI_TDFI_CTRLUPD_MIN_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_175_PI_TDFI_CTRLUPD_MIN_FLDSHFT (16)
+#define HWIO_DDR_PI_175_RESERVED2_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_175_RESERVED2_FLDSHFT (20)
+#define HWIO_DDR_PI_175_OBSOLETE3_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_175_OBSOLETE3_FLDSHFT (24)
+
+#define HWIO_DDR_PI_176_REGOFF 0x2c0
+#define HWIO_DDR_PI_176_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_176_REGOFF)
+#define HWIO_DDR_PI_176_PI_TDFI_CTRLUPD_MAX_F0_FLDMASK (0xffff)
+#define HWIO_DDR_PI_176_PI_TDFI_CTRLUPD_MAX_F0_FLDSHFT (0)
+#define HWIO_DDR_PI_176_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_176_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_177_REGOFF 0x2c4
+#define HWIO_DDR_PI_177_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_177_REGOFF)
+#define HWIO_DDR_PI_177_PI_TDFI_CTRLUPD_INTERVAL_F0_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_177_PI_TDFI_CTRLUPD_INTERVAL_F0_FLDSHFT (0)
+
+#define HWIO_DDR_PI_178_REGOFF 0x2c8
+#define HWIO_DDR_PI_178_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_178_REGOFF)
+#define HWIO_DDR_PI_178_PI_TDFI_CTRLUPD_MAX_F1_FLDMASK (0xffff)
+#define HWIO_DDR_PI_178_PI_TDFI_CTRLUPD_MAX_F1_FLDSHFT (0)
+#define HWIO_DDR_PI_178_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_178_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_179_REGOFF 0x2cc
+#define HWIO_DDR_PI_179_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_179_REGOFF)
+#define HWIO_DDR_PI_179_PI_TDFI_CTRLUPD_INTERVAL_F1_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_179_PI_TDFI_CTRLUPD_INTERVAL_F1_FLDSHFT (0)
+
+#define HWIO_DDR_PI_180_REGOFF 0x2d0
+#define HWIO_DDR_PI_180_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_180_REGOFF)
+#define HWIO_DDR_PI_180_PI_TDFI_CTRLUPD_MAX_F2_FLDMASK (0xffff)
+#define HWIO_DDR_PI_180_PI_TDFI_CTRLUPD_MAX_F2_FLDSHFT (0)
+#define HWIO_DDR_PI_180_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_180_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_181_REGOFF 0x2d4
+#define HWIO_DDR_PI_181_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_181_REGOFF)
+#define HWIO_DDR_PI_181_PI_TDFI_CTRLUPD_INTERVAL_F2_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_181_PI_TDFI_CTRLUPD_INTERVAL_F2_FLDSHFT (0)
+
+#define HWIO_DDR_PI_182_REGOFF 0x2d8
+#define HWIO_DDR_PI_182_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_182_REGOFF)
+#define HWIO_DDR_PI_182_PI_TDFI_CTRLUPD_MAX_F3_FLDMASK (0xffff)
+#define HWIO_DDR_PI_182_PI_TDFI_CTRLUPD_MAX_F3_FLDSHFT (0)
+#define HWIO_DDR_PI_182_OBSOLETE1_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_182_OBSOLETE1_FLDSHFT (16)
+
+#define HWIO_DDR_PI_183_REGOFF 0x2dc
+#define HWIO_DDR_PI_183_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_183_REGOFF)
+#define HWIO_DDR_PI_183_PI_TDFI_CTRLUPD_INTERVAL_F3_FLDMASK (0xffffffff)
+#define HWIO_DDR_PI_183_PI_TDFI_CTRLUPD_INTERVAL_F3_FLDSHFT (0)
+
+#define HWIO_DDR_PI_184_REGOFF 0x2e0
+#define HWIO_DDR_PI_184_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_184_REGOFF)
+#define HWIO_DDR_PI_184_PI_UPDATE_ERROR_STATUS_FLDMASK (0x7f)
+#define HWIO_DDR_PI_184_PI_UPDATE_ERROR_STATUS_FLDSHFT (0)
+#define HWIO_DDR_PI_184_RESERVED_FLDMASK (0x80)
+#define HWIO_DDR_PI_184_RESERVED_FLDSHFT (7)
+#define HWIO_DDR_PI_184_PI_MONITOR_SRC_SEL_0_FLDMASK (0xf00)
+#define HWIO_DDR_PI_184_PI_MONITOR_SRC_SEL_0_FLDSHFT (8)
+#define HWIO_DDR_PI_184_RESERVED1_FLDMASK (0xf000)
+#define HWIO_DDR_PI_184_RESERVED1_FLDSHFT (12)
+#define HWIO_DDR_PI_184_PI_MONITOR_CAP_SEL_0_FLDMASK (0x10000)
+#define HWIO_DDR_PI_184_PI_MONITOR_CAP_SEL_0_FLDSHFT (16)
+#define HWIO_DDR_PI_184_RESERVED2_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_184_RESERVED2_FLDSHFT (17)
+#define HWIO_DDR_PI_184_PI_MONITOR_0_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_184_PI_MONITOR_0_FLDSHFT (24)
+
+#define HWIO_DDR_PI_185_REGOFF 0x2e4
+#define HWIO_DDR_PI_185_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_185_REGOFF)
+#define HWIO_DDR_PI_185_PI_MONITOR_SRC_SEL_1_FLDMASK (0xf)
+#define HWIO_DDR_PI_185_PI_MONITOR_SRC_SEL_1_FLDSHFT (0)
+#define HWIO_DDR_PI_185_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PI_185_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PI_185_PI_MONITOR_CAP_SEL_1_FLDMASK (0x100)
+#define HWIO_DDR_PI_185_PI_MONITOR_CAP_SEL_1_FLDSHFT (8)
+#define HWIO_DDR_PI_185_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_185_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_185_PI_MONITOR_1_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_185_PI_MONITOR_1_FLDSHFT (16)
+#define HWIO_DDR_PI_185_PI_MONITOR_SRC_SEL_2_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_185_PI_MONITOR_SRC_SEL_2_FLDSHFT (24)
+#define HWIO_DDR_PI_185_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_185_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PI_186_REGOFF 0x2e8
+#define HWIO_DDR_PI_186_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_186_REGOFF)
+#define HWIO_DDR_PI_186_PI_MONITOR_CAP_SEL_2_FLDMASK (0x1)
+#define HWIO_DDR_PI_186_PI_MONITOR_CAP_SEL_2_FLDSHFT (0)
+#define HWIO_DDR_PI_186_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_186_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_186_PI_MONITOR_2_FLDMASK (0xff00)
+#define HWIO_DDR_PI_186_PI_MONITOR_2_FLDSHFT (8)
+#define HWIO_DDR_PI_186_PI_MONITOR_SRC_SEL_3_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_186_PI_MONITOR_SRC_SEL_3_FLDSHFT (16)
+#define HWIO_DDR_PI_186_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_186_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PI_186_PI_MONITOR_CAP_SEL_3_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_186_PI_MONITOR_CAP_SEL_3_FLDSHFT (24)
+#define HWIO_DDR_PI_186_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_186_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PI_187_REGOFF 0x2ec
+#define HWIO_DDR_PI_187_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_187_REGOFF)
+#define HWIO_DDR_PI_187_PI_MONITOR_3_FLDMASK (0xff)
+#define HWIO_DDR_PI_187_PI_MONITOR_3_FLDSHFT (0)
+#define HWIO_DDR_PI_187_PI_MONITOR_SRC_SEL_4_FLDMASK (0xf00)
+#define HWIO_DDR_PI_187_PI_MONITOR_SRC_SEL_4_FLDSHFT (8)
+#define HWIO_DDR_PI_187_RESERVED_FLDMASK (0xf000)
+#define HWIO_DDR_PI_187_RESERVED_FLDSHFT (12)
+#define HWIO_DDR_PI_187_PI_MONITOR_CAP_SEL_4_FLDMASK (0x10000)
+#define HWIO_DDR_PI_187_PI_MONITOR_CAP_SEL_4_FLDSHFT (16)
+#define HWIO_DDR_PI_187_RESERVED1_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_187_RESERVED1_FLDSHFT (17)
+#define HWIO_DDR_PI_187_PI_MONITOR_4_FLDMASK (0xff000000)
+#define HWIO_DDR_PI_187_PI_MONITOR_4_FLDSHFT (24)
+
+#define HWIO_DDR_PI_188_REGOFF 0x2f0
+#define HWIO_DDR_PI_188_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_188_REGOFF)
+#define HWIO_DDR_PI_188_PI_MONITOR_SRC_SEL_5_FLDMASK (0xf)
+#define HWIO_DDR_PI_188_PI_MONITOR_SRC_SEL_5_FLDSHFT (0)
+#define HWIO_DDR_PI_188_RESERVED_FLDMASK (0xf0)
+#define HWIO_DDR_PI_188_RESERVED_FLDSHFT (4)
+#define HWIO_DDR_PI_188_PI_MONITOR_CAP_SEL_5_FLDMASK (0x100)
+#define HWIO_DDR_PI_188_PI_MONITOR_CAP_SEL_5_FLDSHFT (8)
+#define HWIO_DDR_PI_188_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_188_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_188_PI_MONITOR_5_FLDMASK (0xff0000)
+#define HWIO_DDR_PI_188_PI_MONITOR_5_FLDSHFT (16)
+#define HWIO_DDR_PI_188_PI_MONITOR_SRC_SEL_6_FLDMASK (0xf000000)
+#define HWIO_DDR_PI_188_PI_MONITOR_SRC_SEL_6_FLDSHFT (24)
+#define HWIO_DDR_PI_188_RESERVED2_FLDMASK (0xf0000000)
+#define HWIO_DDR_PI_188_RESERVED2_FLDSHFT (28)
+
+#define HWIO_DDR_PI_189_REGOFF 0x2f4
+#define HWIO_DDR_PI_189_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_189_REGOFF)
+#define HWIO_DDR_PI_189_PI_MONITOR_CAP_SEL_6_FLDMASK (0x1)
+#define HWIO_DDR_PI_189_PI_MONITOR_CAP_SEL_6_FLDSHFT (0)
+#define HWIO_DDR_PI_189_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_189_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_189_PI_MONITOR_6_FLDMASK (0xff00)
+#define HWIO_DDR_PI_189_PI_MONITOR_6_FLDSHFT (8)
+#define HWIO_DDR_PI_189_PI_MONITOR_SRC_SEL_7_FLDMASK (0xf0000)
+#define HWIO_DDR_PI_189_PI_MONITOR_SRC_SEL_7_FLDSHFT (16)
+#define HWIO_DDR_PI_189_RESERVED1_FLDMASK (0xf00000)
+#define HWIO_DDR_PI_189_RESERVED1_FLDSHFT (20)
+#define HWIO_DDR_PI_189_PI_MONITOR_CAP_SEL_7_FLDMASK (0x1000000)
+#define HWIO_DDR_PI_189_PI_MONITOR_CAP_SEL_7_FLDSHFT (24)
+#define HWIO_DDR_PI_189_RESERVED2_FLDMASK (0xfe000000)
+#define HWIO_DDR_PI_189_RESERVED2_FLDSHFT (25)
+
+#define HWIO_DDR_PI_190_REGOFF 0x2f8
+#define HWIO_DDR_PI_190_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_190_REGOFF)
+#define HWIO_DDR_PI_190_PI_MONITOR_7_FLDMASK (0xff)
+#define HWIO_DDR_PI_190_PI_MONITOR_7_FLDSHFT (0)
+#define HWIO_DDR_PI_190_PI_MONITOR_STROBE_FLDMASK (0xff00)
+#define HWIO_DDR_PI_190_PI_MONITOR_STROBE_FLDSHFT (8)
+#define HWIO_DDR_PI_190_PI_DLL_LOCK_FLDMASK (0x10000)
+#define HWIO_DDR_PI_190_PI_DLL_LOCK_FLDSHFT (16)
+#define HWIO_DDR_PI_190_RESERVED_FLDMASK (0xfe0000)
+#define HWIO_DDR_PI_190_RESERVED_FLDSHFT (17)
+#define HWIO_DDR_PI_190_PI_FREQ_NUMBER_STATUS_FLDMASK (0x1f000000)
+#define HWIO_DDR_PI_190_PI_FREQ_NUMBER_STATUS_FLDSHFT (24)
+#define HWIO_DDR_PI_190_RESERVED1_FLDMASK (0xe0000000)
+#define HWIO_DDR_PI_190_RESERVED1_FLDSHFT (29)
+
+#define HWIO_DDR_PI_191_REGOFF 0x2fc
+#define HWIO_DDR_PI_191_ADDR(bAddr, regX) (bAddr + HWIO_DDR_PI_191_REGOFF)
+#define HWIO_DDR_PI_191_RESERVED0_FLDMASK (0x1)
+#define HWIO_DDR_PI_191_RESERVED0_FLDSHFT (0)
+#define HWIO_DDR_PI_191_RESERVED_FLDMASK (0xfe)
+#define HWIO_DDR_PI_191_RESERVED_FLDSHFT (1)
+#define HWIO_DDR_PI_191_PI_PHYMSTR_TYPE_FLDMASK (0x100)
+#define HWIO_DDR_PI_191_PI_PHYMSTR_TYPE_FLDSHFT (8)
+#define HWIO_DDR_PI_191_RESERVED1_FLDMASK (0xfe00)
+#define HWIO_DDR_PI_191_RESERVED1_FLDSHFT (9)
+#define HWIO_DDR_PI_191_OBSOLETE2_FLDMASK (0xffff0000)
+#define HWIO_DDR_PI_191_OBSOLETE2_FLDSHFT (16)
+
+#endif /* __MNH_HWIO_DDR_PI_ */
diff --git a/drivers/misc/mnh/mnh-hwio-mipi-rx.h b/drivers/misc/mnh/mnh-hwio-mipi-rx.h
new file mode 100644
index 0000000..1afc869
--- /dev/null
+++ b/drivers/misc/mnh/mnh-hwio-mipi-rx.h
@@ -0,0 +1,772 @@
+/* auto generated: Monday, August 15th, 2016 12:26:49pm */
+/*
+ * Copyright (c) 2016, Intel Corporation. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this
+ * list of conditions and the following disclaimer.
+ *
+ * Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * Neither the name of Intel nor the names of its contributors may be used
+ * to endorse or promote products derived from this software without specific
+ * prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef __MNH_HWIO_MIPI_RX_
+#define __MNH_HWIO_MIPI_RX_
+
+#define HWIO_MIPI_RX_VERSION_REGOFF 0x0
+#define HWIO_MIPI_RX_VERSION_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_VERSION_REGOFF)
+#define HWIO_MIPI_RX_VERSION_VERSION_FLDMASK (0xffffffff)
+#define HWIO_MIPI_RX_VERSION_VERSION_FLDSHFT (0)
+
+#define HWIO_MIPI_RX_N_LANES_REGOFF 0x4
+#define HWIO_MIPI_RX_N_LANES_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_N_LANES_REGOFF)
+#define HWIO_MIPI_RX_N_LANES_N_LANES_FLDMASK (0x7)
+#define HWIO_MIPI_RX_N_LANES_N_LANES_FLDSHFT (0)
+#define HWIO_MIPI_RX_N_LANES_RESERVED_31_3_FLDMASK (0xfffffff8)
+#define HWIO_MIPI_RX_N_LANES_RESERVED_31_3_FLDSHFT (3)
+
+#define HWIO_MIPI_RX_CSI2_RESETN_REGOFF 0x8
+#define HWIO_MIPI_RX_CSI2_RESETN_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_CSI2_RESETN_REGOFF)
+#define HWIO_MIPI_RX_CSI2_RESETN_CSI2_RESETN_FLDMASK (0x1)
+#define HWIO_MIPI_RX_CSI2_RESETN_CSI2_RESETN_FLDSHFT (0)
+#define HWIO_MIPI_RX_CSI2_RESETN_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_RX_CSI2_RESETN_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_RX_INT_ST_MAIN_REGOFF 0xc
+#define HWIO_MIPI_RX_INT_ST_MAIN_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_ST_MAIN_REGOFF)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_PHY_FATAL_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_PHY_FATAL_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_PKT_FATAL_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_PKT_FATAL_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_FRAME_FATAL_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_FRAME_FATAL_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_ST_MAIN_RESERVED_15_3_FLDMASK (0xfff8)
+#define HWIO_MIPI_RX_INT_ST_MAIN_RESERVED_15_3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_PHY_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_PHY_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_PKT_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_PKT_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_LINE_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_ST_MAIN_STATUS_INT_LINE_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_ST_MAIN_RESERVED_19_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_ST_MAIN_RESERVED_19_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_ST_MAIN_RESERVED_31_20_FLDMASK (0xfff00000)
+#define HWIO_MIPI_RX_INT_ST_MAIN_RESERVED_31_20_FLDSHFT (20)
+
+#define HWIO_MIPI_RX_DATA_IDS_1_REGOFF 0x10
+#define HWIO_MIPI_RX_DATA_IDS_1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_DATA_IDS_1_REGOFF)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI0_DT_FLDMASK (0x3f)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI0_DT_FLDSHFT (0)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI0_VC_FLDMASK (0xc0)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI0_VC_FLDSHFT (6)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI1_DT_FLDMASK (0x3f00)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI1_DT_FLDSHFT (8)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI1_VC_FLDMASK (0xc000)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI1_VC_FLDSHFT (14)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI2_DT_FLDMASK (0x3f0000)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI2_DT_FLDSHFT (16)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI2_VC_FLDMASK (0xc00000)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI2_VC_FLDSHFT (22)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI3_DT_FLDMASK (0x3f000000)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI3_DT_FLDSHFT (24)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI3_VC_FLDMASK (0xc0000000)
+#define HWIO_MIPI_RX_DATA_IDS_1_DI3_VC_FLDSHFT (30)
+
+#define HWIO_MIPI_RX_DATA_IDS_2_REGOFF 0x14
+#define HWIO_MIPI_RX_DATA_IDS_2_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_DATA_IDS_2_REGOFF)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI4_DT_FLDMASK (0x3f)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI4_DT_FLDSHFT (0)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI4_VC_FLDMASK (0xc0)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI4_VC_FLDSHFT (6)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI5_DT_FLDMASK (0x3f00)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI5_DT_FLDSHFT (8)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI5_VC_FLDMASK (0xc000)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI5_VC_FLDSHFT (14)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI6_DT_FLDMASK (0x3f0000)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI6_DT_FLDSHFT (16)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI6_VC_FLDMASK (0xc00000)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI6_VC_FLDSHFT (22)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI7_DT_FLDMASK (0x3f000000)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI7_DT_FLDSHFT (24)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI7_VC_FLDMASK (0xc0000000)
+#define HWIO_MIPI_RX_DATA_IDS_2_DI7_VC_FLDSHFT (30)
+
+#define HWIO_MIPI_RX_PHY_SHUTDOWNZ_REGOFF 0x40
+#define HWIO_MIPI_RX_PHY_SHUTDOWNZ_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_PHY_SHUTDOWNZ_REGOFF)
+#define HWIO_MIPI_RX_PHY_SHUTDOWNZ_PHY_SHUTDOWNZ_FLDMASK (0x1)
+#define HWIO_MIPI_RX_PHY_SHUTDOWNZ_PHY_SHUTDOWNZ_FLDSHFT (0)
+#define HWIO_MIPI_RX_PHY_SHUTDOWNZ_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_RX_PHY_SHUTDOWNZ_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_RX_DPHY_RSTZ_REGOFF 0x44
+#define HWIO_MIPI_RX_DPHY_RSTZ_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_DPHY_RSTZ_REGOFF)
+#define HWIO_MIPI_RX_DPHY_RSTZ_DPHY_RSTZ_FLDMASK (0x1)
+#define HWIO_MIPI_RX_DPHY_RSTZ_DPHY_RSTZ_FLDSHFT (0)
+#define HWIO_MIPI_RX_DPHY_RSTZ_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_RX_DPHY_RSTZ_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_RX_PHY_RX_REGOFF 0x48
+#define HWIO_MIPI_RX_PHY_RX_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_PHY_RX_REGOFF)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXULPSESC_0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXULPSESC_0_FLDSHFT (0)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXULPSESC_1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXULPSESC_1_FLDSHFT (1)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXULPSESC_2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXULPSESC_2_FLDSHFT (2)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXULPSESC_3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXULPSESC_3_FLDSHFT (3)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_5_FLDSHFT (5)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_6_FLDSHFT (6)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_7_FLDSHFT (7)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_15_8_FLDMASK (0xff00)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_15_8_FLDSHFT (8)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXULPSCLKNOT_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXULPSCLKNOT_FLDSHFT (16)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXCLKACTIVEHS_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_PHY_RX_PHY_RXCLKACTIVEHS_FLDSHFT (17)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_31_18_FLDMASK (0xfffc0000)
+#define HWIO_MIPI_RX_PHY_RX_RESERVED_31_18_FLDSHFT (18)
+
+#define HWIO_MIPI_RX_PHY_STOPSTATE_REGOFF 0x4c
+#define HWIO_MIPI_RX_PHY_STOPSTATE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_PHY_STOPSTATE_REGOFF)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_PHY_STOPSTATEDATA_0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_PHY_STOPSTATEDATA_0_FLDSHFT (0)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_PHY_STOPSTATEDATA_1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_PHY_STOPSTATEDATA_1_FLDSHFT (1)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_PHY_STOPSTATEDATA_2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_PHY_STOPSTATEDATA_2_FLDSHFT (2)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_PHY_STOPSTATEDATA_3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_PHY_STOPSTATEDATA_3_FLDSHFT (3)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_5_FLDSHFT (5)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_6_FLDSHFT (6)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_7_FLDSHFT (7)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_15_8_FLDMASK (0xff00)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_15_8_FLDSHFT (8)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_PHY_STOPSTATECLK_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_PHY_STOPSTATECLK_FLDSHFT (16)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_31_17_FLDMASK (0xfffe0000)
+#define HWIO_MIPI_RX_PHY_STOPSTATE_RESERVED_31_17_FLDSHFT (17)
+
+#define HWIO_MIPI_RX_PHY_TEST_CTRL0_REGOFF 0x50
+#define HWIO_MIPI_RX_PHY_TEST_CTRL0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_PHY_TEST_CTRL0_REGOFF)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL0_PHY_TESTCLR_FLDMASK (0x1)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL0_PHY_TESTCLR_FLDSHFT (0)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL0_PHY_TESTCLK_FLDMASK (0x2)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL0_PHY_TESTCLK_FLDSHFT (1)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL0_RESERVED_31_2_FLDMASK (0xfffffffc)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL0_RESERVED_31_2_FLDSHFT (2)
+
+#define HWIO_MIPI_RX_PHY_TEST_CTRL1_REGOFF 0x54
+#define HWIO_MIPI_RX_PHY_TEST_CTRL1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_PHY_TEST_CTRL1_REGOFF)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL1_PHY_TESTDIN_FLDMASK (0xff)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL1_PHY_TESTDIN_FLDSHFT (0)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL1_PHY_TESTDOUT_FLDMASK (0xff00)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL1_PHY_TESTDOUT_FLDSHFT (8)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL1_PHY_TESTEN_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL1_PHY_TESTEN_FLDSHFT (16)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL1_RESERVED_31_17_FLDMASK (0xfffe0000)
+#define HWIO_MIPI_RX_PHY_TEST_CTRL1_RESERVED_31_17_FLDSHFT (17)
+
+#define HWIO_MIPI_RX_PHY_CAL_REGOFF 0xcc
+#define HWIO_MIPI_RX_PHY_CAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_PHY_CAL_REGOFF)
+#define HWIO_MIPI_RX_PHY_CAL_RXSKEWCALHS_FLDMASK (0x1)
+#define HWIO_MIPI_RX_PHY_CAL_RXSKEWCALHS_FLDSHFT (0)
+#define HWIO_MIPI_RX_PHY_CAL_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_RX_PHY_CAL_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_REGOFF 0xe0
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_ST_PHY_FATAL_REGOFF)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_PHY_ERRSOTSYNCHS_0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_PHY_ERRSOTSYNCHS_0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_PHY_ERRSOTSYNCHS_1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_PHY_ERRSOTSYNCHS_1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_PHY_ERRSOTSYNCHS_2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_PHY_ERRSOTSYNCHS_2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_PHY_ERRSOTSYNCHS_3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_PHY_ERRSOTSYNCHS_3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_RESERVED_4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_RESERVED_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_RESERVED_5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_RESERVED_5_FLDSHFT (5)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_RESERVED_6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_RESERVED_6_FLDSHFT (6)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_RESERVED_7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_RESERVED_7_FLDSHFT (7)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_RESERVED_31_8_FLDMASK (0xffffff00)
+#define HWIO_MIPI_RX_INT_ST_PHY_FATAL_RESERVED_31_8_FLDSHFT (8)
+
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_REGOFF 0xe4
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_MSK_PHY_FATAL_REGOFF)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_MASK_PHY_ERRSOTSYNCHS_0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_MASK_PHY_ERRSOTSYNCHS_0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_MASK_PHY_ERRSOTSYNCHS_1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_MASK_PHY_ERRSOTSYNCHS_1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_MASK_PHY_ERRSOTSYNCHS_2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_MASK_PHY_ERRSOTSYNCHS_2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_MASK_PHY_ERRSOTSYNCHS_3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_MASK_PHY_ERRSOTSYNCHS_3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_RESERVED_4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_RESERVED_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_RESERVED_5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_RESERVED_5_FLDSHFT (5)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_RESERVED_6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_RESERVED_6_FLDSHFT (6)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_RESERVED_7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_RESERVED_7_FLDSHFT (7)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_RESERVED_31_8_FLDMASK (0xffffff00)
+#define HWIO_MIPI_RX_INT_MSK_PHY_FATAL_RESERVED_31_8_FLDSHFT (8)
+
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_REGOFF 0xe8
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_REGOFF)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_FORCE_PHY_ERRSOTSYNCHS_0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_FORCE_PHY_ERRSOTSYNCHS_0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_FORCE_PHY_ERRSOTSYNCHS_1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_FORCE_PHY_ERRSOTSYNCHS_1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_FORCE_PHY_ERRSOTSYNCHS_2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_FORCE_PHY_ERRSOTSYNCHS_2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_FORCE_PHY_ERRSOTSYNCHS_3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_FORCE_PHY_ERRSOTSYNCHS_3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_RESERVED_4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_RESERVED_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_RESERVED_5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_RESERVED_5_FLDSHFT (5)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_RESERVED_6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_RESERVED_6_FLDSHFT (6)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_RESERVED_7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_RESERVED_7_FLDSHFT (7)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_RESERVED_31_8_FLDMASK (0xffffff00)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FATAL_RESERVED_31_8_FLDSHFT (8)
+
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_REGOFF 0xf0
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_ST_PKT_FATAL_REGOFF)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_VC0_ERR_CRC_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_VC0_ERR_CRC_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_VC1_ERR_CRC_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_VC1_ERR_CRC_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_VC2_ERR_CRC_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_VC2_ERR_CRC_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_VC3_ERR_CRC_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_VC3_ERR_CRC_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_RESERVED_15_4_FLDMASK (0xfff0)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_RESERVED_15_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_ERR_ECC_DOUBLE_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_ERR_ECC_DOUBLE_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_RESERVED_31_17_FLDMASK (0xfffe0000)
+#define HWIO_MIPI_RX_INT_ST_PKT_FATAL_RESERVED_31_17_FLDSHFT (17)
+
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_REGOFF 0xf4
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_MSK_PKT_FATAL_REGOFF)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_MASK_VC0_ERR_CRC_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_MASK_VC0_ERR_CRC_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_MASK_VC1_ERR_CRC_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_MASK_VC1_ERR_CRC_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_MASK_VC2_ERR_CRC_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_MASK_VC2_ERR_CRC_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_MASK_VC3_ERR_CRC_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_MASK_VC3_ERR_CRC_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_RESERVED_15_4_FLDMASK (0xfff0)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_RESERVED_15_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_MASK_ERR_ECC_DOUBLE_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_MASK_ERR_ECC_DOUBLE_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_RESERVED_31_17_FLDMASK (0xfffe0000)
+#define HWIO_MIPI_RX_INT_MSK_PKT_FATAL_RESERVED_31_17_FLDSHFT (17)
+
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_REGOFF 0xf8
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_REGOFF)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_FORCE_VC0_ERR_CRC_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_FORCE_VC0_ERR_CRC_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_FORCE_VC1_ERR_CRC_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_FORCE_VC1_ERR_CRC_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_FORCE_VC2_ERR_CRC_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_FORCE_VC2_ERR_CRC_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_FORCE_VC3_ERR_CRC_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_FORCE_VC3_ERR_CRC_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_RESERVED_15_4_FLDMASK (0xfff0)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_RESERVED_15_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_FORCE_ERR_ECC_DOUBLE_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_FORCE_ERR_ECC_DOUBLE_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_RESERVED_31_17_FLDMASK (0xfffe0000)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FATAL_RESERVED_31_17_FLDSHFT (17)
+
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_REGOFF 0x100
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_ST_FRAME_FATAL_REGOFF)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_BNDRY_MATCH_VC0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_BNDRY_MATCH_VC0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_BNDRY_MATCH_VC1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_BNDRY_MATCH_VC1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_BNDRY_MATCH_VC2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_BNDRY_MATCH_VC2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_BNDRY_MATCH_VC3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_BNDRY_MATCH_VC3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_RESERVED_7_4_FLDMASK (0xf0)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_RESERVED_7_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_SEQ_VC0_FLDMASK (0x100)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_SEQ_VC0_FLDSHFT (8)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_SEQ_VC1_FLDMASK (0x200)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_SEQ_VC1_FLDSHFT (9)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_SEQ_VC2_FLDMASK (0x400)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_SEQ_VC2_FLDSHFT (10)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_SEQ_VC3_FLDMASK (0x800)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_F_SEQ_VC3_FLDSHFT (11)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_RESERVED_15_12_FLDMASK (0xf000)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_RESERVED_15_12_FLDSHFT (12)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_FRAME_DATA_VC0_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_FRAME_DATA_VC0_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_FRAME_DATA_VC1_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_FRAME_DATA_VC1_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_FRAME_DATA_VC2_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_FRAME_DATA_VC2_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_FRAME_DATA_VC3_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_ERR_FRAME_DATA_VC3_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_RESERVED_31_20_FLDMASK (0xfff00000)
+#define HWIO_MIPI_RX_INT_ST_FRAME_FATAL_RESERVED_31_20_FLDSHFT (20)
+
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_REGOFF 0x104
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_REGOFF)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_BNDRY_MATCH_VC0_FLDMASK \
+ (0x1)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_BNDRY_MATCH_VC0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_BNDRY_MATCH_VC1_FLDMASK \
+ (0x2)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_BNDRY_MATCH_VC1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_BNDRY_MATCH_VC2_FLDMASK \
+ (0x4)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_BNDRY_MATCH_VC2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_BNDRY_MATCH_VC3_FLDMASK \
+ (0x8)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_BNDRY_MATCH_VC3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_RESERVED_7_4_FLDMASK (0xf0)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_RESERVED_7_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_SEQ_VC0_FLDMASK (0x100)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_SEQ_VC0_FLDSHFT (8)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_SEQ_VC1_FLDMASK (0x200)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_SEQ_VC1_FLDSHFT (9)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_SEQ_VC2_FLDMASK (0x400)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_SEQ_VC2_FLDSHFT (10)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_SEQ_VC3_FLDMASK (0x800)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_F_SEQ_VC3_FLDSHFT (11)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_RESERVED_15_12_FLDMASK (0xf000)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_RESERVED_15_12_FLDSHFT (12)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_FRAME_DATA_VC0_FLDMASK \
+ (0x10000)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_FRAME_DATA_VC0_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_FRAME_DATA_VC1_FLDMASK \
+ (0x20000)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_FRAME_DATA_VC1_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_FRAME_DATA_VC2_FLDMASK \
+ (0x40000)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_FRAME_DATA_VC2_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_FRAME_DATA_VC3_FLDMASK \
+ (0x80000)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_MASK_ERR_FRAME_DATA_VC3_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_RESERVED_31_20_FLDMASK (0xfff00000)
+#define HWIO_MIPI_RX_INT_MSK_FRAME_FATAL_RESERVED_31_20_FLDSHFT (20)
+
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_REGOFF 0x108
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_REGOFF)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_BNDRY_MATCH_VC0_FLDMASK \
+ (0x1)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_BNDRY_MATCH_VC0_FLDSHFT \
+ (0)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_BNDRY_MATCH_VC1_FLDMASK \
+ (0x2)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_BNDRY_MATCH_VC1_FLDSHFT \
+ (1)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_BNDRY_MATCH_VC2_FLDMASK \
+ (0x4)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_BNDRY_MATCH_VC2_FLDSHFT \
+ (2)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_BNDRY_MATCH_VC3_FLDMASK \
+ (0x8)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_BNDRY_MATCH_VC3_FLDSHFT \
+ (3)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_RESERVED_7_4_FLDMASK (0xf0)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_RESERVED_7_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_SEQ_VC0_FLDMASK (0x100)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_SEQ_VC0_FLDSHFT (8)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_SEQ_VC1_FLDMASK (0x200)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_SEQ_VC1_FLDSHFT (9)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_SEQ_VC2_FLDMASK (0x400)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_SEQ_VC2_FLDSHFT (10)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_SEQ_VC3_FLDMASK (0x800)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_F_SEQ_VC3_FLDSHFT (11)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_RESERVED_15_12_FLDMASK (0xf000)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_RESERVED_15_12_FLDSHFT (12)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_FRAME_DATA_VC0_FLDMASK \
+ (0x10000)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_FRAME_DATA_VC0_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_FRAME_DATA_VC1_FLDMASK \
+ (0x20000)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_FRAME_DATA_VC1_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_FRAME_DATA_VC2_FLDMASK \
+ (0x40000)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_FRAME_DATA_VC2_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_FRAME_DATA_VC3_FLDMASK \
+ (0x80000)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_FORCE_ERR_FRAME_DATA_VC3_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_RESERVED_31_20_FLDMASK (0xfff00000)
+#define HWIO_MIPI_RX_INT_FORCE_FRAME_FATAL_RESERVED_31_20_FLDSHFT (20)
+
+#define HWIO_MIPI_RX_INT_ST_PHY_REGOFF 0x110
+#define HWIO_MIPI_RX_INT_ST_PHY_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_ST_PHY_REGOFF)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRSOTHS_0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRSOTHS_0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRSOTHS_1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRSOTHS_1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRSOTHS_2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRSOTHS_2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRSOTHS_3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRSOTHS_3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_5_FLDSHFT (5)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_6_FLDSHFT (6)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_7_FLDSHFT (7)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_15_8_FLDMASK (0xff00)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_15_8_FLDSHFT (8)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRESC_0_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRESC_0_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRESC_1_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRESC_1_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRESC_2_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRESC_2_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRESC_3_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_ST_PHY_PHY_ERRESC_3_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_20_FLDMASK (0x100000)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_20_FLDSHFT (20)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_21_FLDMASK (0x200000)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_21_FLDSHFT (21)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_22_FLDMASK (0x400000)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_22_FLDSHFT (22)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_23_FLDMASK (0x800000)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_23_FLDSHFT (23)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_31_24_FLDMASK (0xff000000)
+#define HWIO_MIPI_RX_INT_ST_PHY_RESERVED_31_24_FLDSHFT (24)
+
+#define HWIO_MIPI_RX_INT_MSK_PHY_REGOFF 0x114
+#define HWIO_MIPI_RX_INT_MSK_PHY_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_MSK_PHY_REGOFF)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRSOTHS_0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRSOTHS_0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRSOTHS_1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRSOTHS_1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRSOTHS_2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRSOTHS_2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRSOTHS_3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRSOTHS_3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_5_FLDSHFT (5)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_6_FLDSHFT (6)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_7_FLDSHFT (7)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_15_8_FLDMASK (0xff00)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_15_8_FLDSHFT (8)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRESC_0_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRESC_0_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRESC_1_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRESC_1_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRESC_2_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRESC_2_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRESC_3_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_MSK_PHY_MASK_PHY_ERRESC_3_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_20_FLDMASK (0x100000)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_20_FLDSHFT (20)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_21_FLDMASK (0x200000)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_21_FLDSHFT (21)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_22_FLDMASK (0x400000)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_22_FLDSHFT (22)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_23_FLDMASK (0x800000)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_23_FLDSHFT (23)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_31_24_FLDMASK (0xff000000)
+#define HWIO_MIPI_RX_INT_MSK_PHY_RESERVED_31_24_FLDSHFT (24)
+
+#define HWIO_MIPI_RX_INT_FORCE_PHY_REGOFF 0x118
+#define HWIO_MIPI_RX_INT_FORCE_PHY_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_FORCE_PHY_REGOFF)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRSOTHS_0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRSOTHS_0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRSOTHS_1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRSOTHS_1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRSOTHS_2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRSOTHS_2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRSOTHS_3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRSOTHS_3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_5_FLDSHFT (5)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_6_FLDSHFT (6)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_7_FLDSHFT (7)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_15_8_FLDMASK (0xff00)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_15_8_FLDSHFT (8)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRESC_0_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRESC_0_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRESC_1_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRESC_1_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRESC_2_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRESC_2_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRESC_3_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_FORCE_PHY_ERRESC_3_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_20_FLDMASK (0x100000)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_20_FLDSHFT (20)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_21_FLDMASK (0x200000)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_21_FLDSHFT (21)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_22_FLDMASK (0x400000)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_22_FLDSHFT (22)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_23_FLDMASK (0x800000)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_23_FLDSHFT (23)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_31_24_FLDMASK (0xff000000)
+#define HWIO_MIPI_RX_INT_FORCE_PHY_RESERVED_31_24_FLDSHFT (24)
+
+#define HWIO_MIPI_RX_INT_ST_PKT_REGOFF 0x120
+#define HWIO_MIPI_RX_INT_ST_PKT_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_ST_PKT_REGOFF)
+#define HWIO_MIPI_RX_INT_ST_PKT_ERR_ID_VC0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_ST_PKT_ERR_ID_VC0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_ST_PKT_ERR_ID_VC1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_ST_PKT_ERR_ID_VC1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_ST_PKT_ERR_ID_VC2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_ST_PKT_ERR_ID_VC2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_ST_PKT_ERR_ID_VC3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_ST_PKT_ERR_ID_VC3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_ST_PKT_RESERVED_15_4_FLDMASK (0xfff0)
+#define HWIO_MIPI_RX_INT_ST_PKT_RESERVED_15_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_ST_PKT_VC0_ERR_ECC_CORRECTED_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_ST_PKT_VC0_ERR_ECC_CORRECTED_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_ST_PKT_VC1_ERR_ECC_CORRECTED_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_ST_PKT_VC1_ERR_ECC_CORRECTED_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_ST_PKT_VC2_ERR_ECC_CORRECTED_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_ST_PKT_VC2_ERR_ECC_CORRECTED_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_ST_PKT_VC3_ERR_ECC_CORRECTED_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_ST_PKT_VC3_ERR_ECC_CORRECTED_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_ST_PKT_RESERVED_31_20_FLDMASK (0xfff00000)
+#define HWIO_MIPI_RX_INT_ST_PKT_RESERVED_31_20_FLDSHFT (20)
+
+#define HWIO_MIPI_RX_INT_MSK_PKT_REGOFF 0x124
+#define HWIO_MIPI_RX_INT_MSK_PKT_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_MSK_PKT_REGOFF)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_ERR_ID_VC0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_ERR_ID_VC0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_ERR_ID_VC1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_ERR_ID_VC1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_ERR_ID_VC2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_ERR_ID_VC2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_ERR_ID_VC3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_ERR_ID_VC3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_MSK_PKT_RESERVED_15_4_FLDMASK (0xfff0)
+#define HWIO_MIPI_RX_INT_MSK_PKT_RESERVED_15_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_VC0_ERR_ECC_CORRECTED_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_VC0_ERR_ECC_CORRECTED_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_VC1_ERR_ECC_CORRECTED_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_VC1_ERR_ECC_CORRECTED_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_VC2_ERR_ECC_CORRECTED_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_VC2_ERR_ECC_CORRECTED_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_VC3_ERR_ECC_CORRECTED_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_MSK_PKT_MASK_VC3_ERR_ECC_CORRECTED_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_MSK_PKT_RESERVED_31_20_FLDMASK (0xfff00000)
+#define HWIO_MIPI_RX_INT_MSK_PKT_RESERVED_31_20_FLDSHFT (20)
+
+#define HWIO_MIPI_RX_INT_FORCE_PKT_REGOFF 0x128
+#define HWIO_MIPI_RX_INT_FORCE_PKT_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_FORCE_PKT_REGOFF)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_ERR_ID_VC0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_ERR_ID_VC0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_ERR_ID_VC1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_ERR_ID_VC1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_ERR_ID_VC2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_ERR_ID_VC2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_ERR_ID_VC3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_ERR_ID_VC3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_RESERVED_15_4_FLDMASK (0xfff0)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_RESERVED_15_4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_VC0_ERR_ECC_CORRECTED_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_VC0_ERR_ECC_CORRECTED_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_VC1_ERR_ECC_CORRECTED_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_VC1_ERR_ECC_CORRECTED_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_VC2_ERR_ECC_CORRECTED_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_VC2_ERR_ECC_CORRECTED_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_VC3_ERR_ECC_CORRECTED_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_FORCE_VC3_ERR_ECC_CORRECTED_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_RESERVED_31_20_FLDMASK (0xfff00000)
+#define HWIO_MIPI_RX_INT_FORCE_PKT_RESERVED_31_20_FLDSHFT (20)
+
+#define HWIO_MIPI_RX_INT_ST_LINE_REGOFF 0x130
+#define HWIO_MIPI_RX_INT_ST_LINE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_ST_LINE_REGOFF)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI5_FLDSHFT (5)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI6_FLDSHFT (6)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_BNDRY_MATCH_DI7_FLDSHFT (7)
+#define HWIO_MIPI_RX_INT_ST_LINE_RESERVED_15_8_FLDMASK (0xff00)
+#define HWIO_MIPI_RX_INT_ST_LINE_RESERVED_15_8_FLDSHFT (8)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI0_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI0_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI1_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI1_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI2_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI2_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI3_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI3_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI4_FLDMASK (0x100000)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI4_FLDSHFT (20)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI5_FLDMASK (0x200000)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI5_FLDSHFT (21)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI6_FLDMASK (0x400000)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI6_FLDSHFT (22)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI7_FLDMASK (0x800000)
+#define HWIO_MIPI_RX_INT_ST_LINE_ERR_L_SEQ_DI7_FLDSHFT (23)
+#define HWIO_MIPI_RX_INT_ST_LINE_RESERVED_31_24_FLDMASK (0xff000000)
+#define HWIO_MIPI_RX_INT_ST_LINE_RESERVED_31_24_FLDSHFT (24)
+
+#define HWIO_MIPI_RX_INT_MSK_LINE_REGOFF 0x134
+#define HWIO_MIPI_RX_INT_MSK_LINE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_MSK_LINE_REGOFF)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI5_FLDSHFT (5)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI6_FLDSHFT (6)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_BNDRY_MATCH_DI7_FLDSHFT (7)
+#define HWIO_MIPI_RX_INT_MSK_LINE_RESERVED_15_8_FLDMASK (0xff00)
+#define HWIO_MIPI_RX_INT_MSK_LINE_RESERVED_15_8_FLDSHFT (8)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI0_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI0_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI1_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI1_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI2_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI2_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI3_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI3_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI4_FLDMASK (0x100000)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI4_FLDSHFT (20)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI5_FLDMASK (0x200000)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI5_FLDSHFT (21)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI6_FLDMASK (0x400000)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI6_FLDSHFT (22)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI7_FLDMASK (0x800000)
+#define HWIO_MIPI_RX_INT_MSK_LINE_MASK_ERR_L_SEQ_DI7_FLDSHFT (23)
+#define HWIO_MIPI_RX_INT_MSK_LINE_RESERVED_31_24_FLDMASK (0xff000000)
+#define HWIO_MIPI_RX_INT_MSK_LINE_RESERVED_31_24_FLDSHFT (24)
+
+#define HWIO_MIPI_RX_INT_FORCE_LINE_REGOFF 0x138
+#define HWIO_MIPI_RX_INT_FORCE_LINE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_RX_INT_FORCE_LINE_REGOFF)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI0_FLDMASK (0x1)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI0_FLDSHFT (0)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI1_FLDMASK (0x2)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI1_FLDSHFT (1)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI2_FLDMASK (0x4)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI2_FLDSHFT (2)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI3_FLDMASK (0x8)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI3_FLDSHFT (3)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI4_FLDMASK (0x10)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI4_FLDSHFT (4)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI5_FLDMASK (0x20)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI5_FLDSHFT (5)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI6_FLDMASK (0x40)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI6_FLDSHFT (6)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI7_FLDMASK (0x80)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_BNDRY_MATCH_DI7_FLDSHFT (7)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_RESERVED_15_8_FLDMASK (0xff00)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_RESERVED_15_8_FLDSHFT (8)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI0_FLDMASK (0x10000)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI0_FLDSHFT (16)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI1_FLDMASK (0x20000)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI1_FLDSHFT (17)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI2_FLDMASK (0x40000)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI2_FLDSHFT (18)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI3_FLDMASK (0x80000)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI3_FLDSHFT (19)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI4_FLDMASK (0x100000)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI4_FLDSHFT (20)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI5_FLDMASK (0x200000)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI5_FLDSHFT (21)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI6_FLDMASK (0x400000)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI6_FLDSHFT (22)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI7_FLDMASK (0x800000)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_FORCE_ERR_L_SEQ_DI7_FLDSHFT (23)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_RESERVED_31_24_FLDMASK (0xff000000)
+#define HWIO_MIPI_RX_INT_FORCE_LINE_RESERVED_31_24_FLDSHFT (24)
+
+#endif /* __MNH_HWIO_MIPI_RX_ */
diff --git a/drivers/misc/mnh/mnh-hwio-mipi-top.h b/drivers/misc/mnh/mnh-hwio-mipi-top.h
new file mode 100644
index 0000000..ae0a75d
--- /dev/null
+++ b/drivers/misc/mnh/mnh-hwio-mipi-top.h
@@ -0,0 +1,644 @@
+/* auto generated: Monday, August 15th, 2016 12:26:49pm */
+/*
+ * Copyright (c) 2016, Intel Corporation. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this
+ * list of conditions and the following disclaimer.
+ *
+ * Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * Neither the name of Intel nor the names of its contributors may be used
+ * to endorse or promote products derived from this software without specific
+ * prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef __MNH_HWIO_MIPI_TOP_
+#define __MNH_HWIO_MIPI_TOP_
+
+#define HWIO_MIPI_TOP_TX0_MODE_REGOFF 0x0
+#define HWIO_MIPI_TOP_TX0_MODE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_MODE_REGOFF)
+#define HWIO_MIPI_TOP_TX0_MODE_RSVD0_FLDMASK (0xffffff80)
+#define HWIO_MIPI_TOP_TX0_MODE_RSVD0_FLDSHFT (7)
+#define HWIO_MIPI_TOP_TX0_MODE_TX0_FORCE_OFF_FLDMASK (0x40)
+#define HWIO_MIPI_TOP_TX0_MODE_TX0_FORCE_OFF_FLDSHFT (6)
+#define HWIO_MIPI_TOP_TX0_MODE_TX0_CUR_BYP_SEL_FLDMASK (0x38)
+#define HWIO_MIPI_TOP_TX0_MODE_TX0_CUR_BYP_SEL_FLDSHFT (3)
+#define HWIO_MIPI_TOP_TX0_MODE_TX0_BYP_SEL_FLDMASK (0x6)
+#define HWIO_MIPI_TOP_TX0_MODE_TX0_BYP_SEL_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX0_MODE_TX0_FUNC_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX0_MODE_TX0_FUNC_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_BYPINT_REGOFF 0x4
+#define HWIO_MIPI_TOP_TX0_BYPINT_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_BYPINT_REGOFF)
+#define HWIO_MIPI_TOP_TX0_BYPINT_RSVD0_FLDMASK (0xfffffffc)
+#define HWIO_MIPI_TOP_TX0_BYPINT_RSVD0_FLDSHFT (2)
+#define HWIO_MIPI_TOP_TX0_BYPINT_TX0_INT_EN_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_TX0_BYPINT_TX0_INT_EN_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX0_BYPINT_TX0_BYP_OF_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX0_BYPINT_TX0_BYP_OF_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_REGOFF 0x8
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_REGOFF)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_RSVD0_FLDMASK (0xe0000000)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_RSVD0_FLDSHFT (29)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_GMP_CNTRL_FLDMASK (0x18000000)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_GMP_CNTRL_FLDSHFT (27)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_CPBIAS_CNTRL_FLDMASK (0x7f00000)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_CPBIAS_CNTRL_FLDSHFT (20)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_VCO_CNTRL_FLDMASK (0xfc000)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_VCO_CNTRL_FLDSHFT (14)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_N_FLDMASK (0x3c00)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_N_FLDSHFT (10)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_M_FLDMASK (0x3ff)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG0_M_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG1_REGOFF 0x0C
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG1_REGOFF)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG1_RSVD0_FLDMASK (0xfffff000)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG1_RSVD0_FLDSHFT (12)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG1_PROP_CNTRL_FLDMASK (0xfc0)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG1_PROP_CNTRL_FLDSHFT (6)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG1_INT_CNTRL_FLDMASK (0x3f)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CONFIG1_INT_CNTRL_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_REGOFF 0x10
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_REGOFF)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_RSVD0_FLDMASK (0xffffff00)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_RSVD0_FLDSHFT (8)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_CLK_SEL_FLDMASK (0xc0)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_CLK_SEL_FLDSHFT (6)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_FORCE_LOCK_FLDMASK (0x20)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_FORCE_LOCK_FLDSHFT (5)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_GP_CLK_EN_FLDMASK (0x10)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_GP_CLK_EN_FLDSHFT (4)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_FORCEPLL_FLDMASK (0x8)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_FORCEPLL_FLDSHFT (3)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_UPDATEPLL_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_UPDATEPLL_FLDSHFT (2)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_SHADOW_CLEAR_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_SHADOW_CLEAR_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_PLL_SHADOW_CONTROL_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_CNTRL_PLL_SHADOW_CONTROL_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_STATUS_REGOFF 0x14
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_STATUS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_DPHY_PLL_STATUS_REGOFF)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_STATUS_RSVD0_FLDMASK (0xfffffffc)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_STATUS_RSVD0_FLDSHFT (2)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_STATUS_PLL_SHADOW_CONTROL_BUS_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_STATUS_PLL_SHADOW_CONTROL_BUS_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_STATUS_LOCK_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX0_DPHY_PLL_STATUS_LOCK_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_DPHY_CONFIG_REGOFF 0x18
+#define HWIO_MIPI_TOP_TX0_DPHY_CONFIG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_DPHY_CONFIG_REGOFF)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONFIG_RSVD0_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONFIG_RSVD0_FLDSHFT (16)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONFIG_CFGCLKFREQRANGE_FLDMASK (0xff00)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONFIG_CFGCLKFREQRANGE_FLDSHFT (8)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONFIG_RSVD1_FLDMASK (0x80)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONFIG_RSVD1_FLDSHFT (7)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONFIG_HSFREQRANGE_FLDMASK (0x7f)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONFIG_HSFREQRANGE_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_REGOFF 0x1C
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_DPHY_CONTROL_REGOFF)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_RSVD0_FLDMASK (0xfffffff0)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_RSVD0_FLDSHFT (4)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_PHY_SHADOW_CLEAR_FLDMASK (0x8)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_PHY_SHADOW_CLEAR_FLDSHFT (3)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_PHY_SHADOW_CONTROL_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_PHY_SHADOW_CONTROL_FLDSHFT (2)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_RSTZ_N_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_RSTZ_N_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_SHUTDOWNZ_N_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX0_DPHY_CONTROL_SHUTDOWNZ_N_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_REGOFF 0x20
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_DPHY_IOTEST_REGOFF)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_RSVD0_FLDMASK (0xffff8000)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_RSVD0_FLDSHFT (15)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_BISTOK_FLDMASK (0x4000)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_BISTOK_FLDSHFT (14)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_BISTDONE_FLDMASK (0x2000)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_BISTDONE_FLDSHFT (13)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_BISTON_FLDMASK (0x1000)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_BISTON_FLDSHFT (12)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_CONT_DATA_FLDMASK (0xffe)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_CONT_DATA_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_CONT_EN_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX0_DPHY_IOTEST_CONT_EN_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_DPHY_ESCAPE_REGOFF 0x24
+#define HWIO_MIPI_TOP_TX0_DPHY_ESCAPE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_DPHY_ESCAPE_REGOFF)
+#define HWIO_MIPI_TOP_TX0_DPHY_ESCAPE_TX0_DPHY_ESCAPE_FLDMASK (0xffffffff)
+#define HWIO_MIPI_TOP_TX0_DPHY_ESCAPE_TX0_DPHY_ESCAPE_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_MODE_REGOFF 0x28
+#define HWIO_MIPI_TOP_TX1_MODE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_MODE_REGOFF)
+#define HWIO_MIPI_TOP_TX1_MODE_RSVD0_FLDMASK (0xffffff80)
+#define HWIO_MIPI_TOP_TX1_MODE_RSVD0_FLDSHFT (7)
+#define HWIO_MIPI_TOP_TX1_MODE_TX1_FORCE_OFF_FLDMASK (0x40)
+#define HWIO_MIPI_TOP_TX1_MODE_TX1_FORCE_OFF_FLDSHFT (6)
+#define HWIO_MIPI_TOP_TX1_MODE_TX1_CUR_BYP_SEL_FLDMASK (0x38)
+#define HWIO_MIPI_TOP_TX1_MODE_TX1_CUR_BYP_SEL_FLDSHFT (3)
+#define HWIO_MIPI_TOP_TX1_MODE_TX1_BYP_SEL_FLDMASK (0x6)
+#define HWIO_MIPI_TOP_TX1_MODE_TX1_BYP_SEL_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX1_MODE_TX1_FUNC_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX1_MODE_TX1_FUNC_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_BYPINT_REGOFF 0x2C
+#define HWIO_MIPI_TOP_TX1_BYPINT_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_BYPINT_REGOFF)
+#define HWIO_MIPI_TOP_TX1_BYPINT_RSVD0_FLDMASK (0xfffffffc)
+#define HWIO_MIPI_TOP_TX1_BYPINT_RSVD0_FLDSHFT (2)
+#define HWIO_MIPI_TOP_TX1_BYPINT_TX1_INT_EN_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_TX1_BYPINT_TX1_INT_EN_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX1_BYPINT_TX1_BYP_OF_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX1_BYPINT_TX1_BYP_OF_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_REGOFF 0x30
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_REGOFF)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_RSVD0_FLDMASK (0xe0000000)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_RSVD0_FLDSHFT (29)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_GMP_CNTRL_FLDMASK (0x18000000)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_GMP_CNTRL_FLDSHFT (27)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_CPBIAS_CNTRL_FLDMASK (0x7f00000)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_CPBIAS_CNTRL_FLDSHFT (20)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_VCO_CNTRL_FLDMASK (0xfc000)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_VCO_CNTRL_FLDSHFT (14)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_N_FLDMASK (0x3c00)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_N_FLDSHFT (10)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_M_FLDMASK (0x3ff)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG0_M_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG1_REGOFF 0x34
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG1_REGOFF)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG1_RSVD0_FLDMASK (0xfffff000)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG1_RSVD0_FLDSHFT (12)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG1_PROP_CNTRL_FLDMASK (0xfc0)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG1_PROP_CNTRL_FLDSHFT (6)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG1_INT_CNTRL_FLDMASK (0x3f)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CONFIG1_INT_CNTRL_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_REGOFF 0x38
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_REGOFF)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_RSVD0_FLDMASK (0xffffff00)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_RSVD0_FLDSHFT (8)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_CLK_SEL_FLDMASK (0xc0)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_CLK_SEL_FLDSHFT (6)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_FORCE_LOCK_FLDMASK (0x20)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_FORCE_LOCK_FLDSHFT (5)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_GP_CLK_EN_FLDMASK (0x10)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_GP_CLK_EN_FLDSHFT (4)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_FORCEPLL_FLDMASK (0x8)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_FORCEPLL_FLDSHFT (3)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_UPDATEPLL_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_UPDATEPLL_FLDSHFT (2)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_SHADOW_CLEAR_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_SHADOW_CLEAR_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_PLL_SHADOW_CONTROL_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_CNTRL_PLL_SHADOW_CONTROL_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_STATUS_REGOFF 0x3C
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_STATUS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_DPHY_PLL_STATUS_REGOFF)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_STATUS_RSVD0_FLDMASK (0xfffffffc)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_STATUS_RSVD0_FLDSHFT (2)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_STATUS_PLL_SHADOW_CONTROL_BUS_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_STATUS_PLL_SHADOW_CONTROL_BUS_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_STATUS_LOCK_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX1_DPHY_PLL_STATUS_LOCK_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_DPHY_CONFIG_REGOFF 0x40
+#define HWIO_MIPI_TOP_TX1_DPHY_CONFIG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_DPHY_CONFIG_REGOFF)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONFIG_RSVD0_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONFIG_RSVD0_FLDSHFT (16)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONFIG_CFGCLKFREQRANGE_FLDMASK (0xff00)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONFIG_CFGCLKFREQRANGE_FLDSHFT (8)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONFIG_RSVD1_FLDMASK (0x80)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONFIG_RSVD1_FLDSHFT (7)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONFIG_HSFREQRANGE_FLDMASK (0x7f)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONFIG_HSFREQRANGE_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_REGOFF 0x44
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_DPHY_CONTROL_REGOFF)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_RSVD0_FLDMASK (0xfffffff0)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_RSVD0_FLDSHFT (4)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_PHY_SHADOW_CLEAR_FLDMASK (0x8)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_PHY_SHADOW_CLEAR_FLDSHFT (3)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_PHY_SHADOW_CONTROL_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_PHY_SHADOW_CONTROL_FLDSHFT (2)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_RSTZ_N_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_RSTZ_N_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_SHUTDOWNZ_N_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX1_DPHY_CONTROL_SHUTDOWNZ_N_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_REGOFF 0x48
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_DPHY_IOTEST_REGOFF)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_RSVD0_FLDMASK (0xffff8000)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_RSVD0_FLDSHFT (15)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_BISTOK_FLDMASK (0x4000)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_BISTOK_FLDSHFT (14)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_BISTDONE_FLDMASK (0x2000)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_BISTDONE_FLDSHFT (13)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_BISTON_FLDMASK (0x1000)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_BISTON_FLDSHFT (12)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_CONT_DATA_FLDMASK (0xffe)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_CONT_DATA_FLDSHFT (1)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_CONT_EN_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_TX1_DPHY_IOTEST_CONT_EN_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_DPHY_ESCAPE_REGOFF 0x4C
+#define HWIO_MIPI_TOP_TX1_DPHY_ESCAPE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_DPHY_ESCAPE_REGOFF)
+#define HWIO_MIPI_TOP_TX1_DPHY_ESCAPE_tx1_dphy_escape_FLDMASK (0xffffffff)
+#define HWIO_MIPI_TOP_TX1_DPHY_ESCAPE_tx1_dphy_escape_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX0_MODE_REGOFF 0x50
+#define HWIO_MIPI_TOP_RX0_MODE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX0_MODE_REGOFF)
+#define HWIO_MIPI_TOP_RX0_MODE_RSVD0_FLDMASK (0xff000000)
+#define HWIO_MIPI_TOP_RX0_MODE_RSVD0_FLDSHFT (24)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_ECC_FLDMASK (0xff0000)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_ECC_FLDSHFT (16)
+#define HWIO_MIPI_TOP_RX0_MODE_RSVD1_FLDMASK (0xf800)
+#define HWIO_MIPI_TOP_RX0_MODE_RSVD1_FLDSHFT (11)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_VC_EN_FLDMASK (0x780)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_VC_EN_FLDSHFT (7)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_FORCE_OFF_FLDMASK (0x40)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_FORCE_OFF_FLDSHFT (6)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_CUR_BYP_MODE_FLDMASK (0x38)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_CUR_BYP_MODE_FLDSHFT (3)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_BYP_TX1_EN_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_BYP_TX1_EN_FLDSHFT (2)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_BYP_TX0_EN_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_BYP_TX0_EN_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_IPU_EN_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX0_MODE_RX0_IPU_EN_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_REGOFF 0x54
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_REGOFF)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_RSVD0_FLDMASK (0xe0000000)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_RSVD0_FLDSHFT (29)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_GMP_CNTRL_FLDMASK (0x18000000)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_GMP_CNTRL_FLDSHFT (27)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_CPBIAS_CNTRL_FLDMASK (0x7f00000)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_CPBIAS_CNTRL_FLDSHFT (20)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_VCO_CNTRL_FLDMASK (0xfc000)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_VCO_CNTRL_FLDSHFT (14)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_N_FLDMASK (0x3c00)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_N_FLDSHFT (10)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_M_FLDMASK (0x3ff)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG0_M_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG1_REGOFF 0x58
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG1_REGOFF)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG1_RSVD0_FLDMASK (0xfffff000)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG1_RSVD0_FLDSHFT (12)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG1_PROP_CNTRL_FLDMASK (0xfc0)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG1_PROP_CNTRL_FLDSHFT (6)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG1_INT_CNTRL_FLDMASK (0x3f)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CONFIG1_INT_CNTRL_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_REGOFF 0x5C
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_REGOFF)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_RSVD0_FLDMASK (0xffffff00)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_RSVD0_FLDSHFT (8)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_CLK_SEL_FLDMASK (0xc0)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_CLK_SEL_FLDSHFT (6)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_FORCE_LOCK_FLDMASK (0x20)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_FORCE_LOCK_FLDSHFT (5)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_GP_CLK_EN_FLDMASK (0x10)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_GP_CLK_EN_FLDSHFT (4)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_FORCEPLL_FLDMASK (0x8)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_FORCEPLL_FLDSHFT (3)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_UPDATEPLL_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_UPDATEPLL_FLDSHFT (2)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_SHADOW_CLEAR_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_SHADOW_CLEAR_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_PLL_SHADOW_CONTROL_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_CNTRL_PLL_SHADOW_CONTROL_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_STATUS_REGOFF 0x60
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_STATUS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX0_DPHY_PLL_STATUS_REGOFF)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_STATUS_RSVD0_FLDMASK (0xfffffffc)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_STATUS_RSVD0_FLDSHFT (2)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_STATUS_PLL_SHADOW_CONTROL_BUS_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_STATUS_PLL_SHADOW_CONTROL_BUS_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_STATUS_LOCK_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX0_DPHY_PLL_STATUS_LOCK_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX0_DPHY_CONFIG_REGOFF 0x64
+#define HWIO_MIPI_TOP_RX0_DPHY_CONFIG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX0_DPHY_CONFIG_REGOFF)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONFIG_RSVD0_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONFIG_RSVD0_FLDSHFT (16)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONFIG_CFGCLKFREQRANGE_FLDMASK (0xff00)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONFIG_CFGCLKFREQRANGE_FLDSHFT (8)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONFIG_RSVD1_FLDMASK (0x80)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONFIG_RSVD1_FLDSHFT (7)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONFIG_HSFREQRANGE_FLDMASK (0x7f)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONFIG_HSFREQRANGE_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX0_DPHY_CONTROL_REGOFF 0x68
+#define HWIO_MIPI_TOP_RX0_DPHY_CONTROL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX0_DPHY_CONTROL_REGOFF)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONTROL_RSVD0_FLDMASK (0xfffffff8)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONTROL_RSVD0_FLDSHFT (3)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONTROL_PHY_SHADOW_CONTROL_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONTROL_PHY_SHADOW_CONTROL_FLDSHFT (2)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONTROL_RSTZ_N_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONTROL_RSTZ_N_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONTROL_SHUTDOWNZ_N_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX0_DPHY_CONTROL_SHUTDOWNZ_N_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_REGOFF 0x6C
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX0_DPHY_IOTEST_REGOFF)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_RSVD0_FLDMASK (0xffff8000)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_RSVD0_FLDSHFT (15)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_BISTOK_FLDMASK (0x4000)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_BISTOK_FLDSHFT (14)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_BISTDONE_FLDMASK (0x2000)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_BISTDONE_FLDSHFT (13)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_BISTON_FLDMASK (0x1000)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_BISTON_FLDSHFT (12)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_CONT_DATA_FLDMASK (0xffe)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_CONT_DATA_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_CONT_EN_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX0_DPHY_IOTEST_CONT_EN_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX0_DPHY_ESCAPE_REGOFF 0x70
+#define HWIO_MIPI_TOP_RX0_DPHY_ESCAPE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX0_DPHY_ESCAPE_REGOFF)
+#define HWIO_MIPI_TOP_RX0_DPHY_ESCAPE_rx0_dphy_escape_FLDMASK (0xffffffff)
+#define HWIO_MIPI_TOP_RX0_DPHY_ESCAPE_rx0_dphy_escape_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX1_MODE_REGOFF 0x74
+#define HWIO_MIPI_TOP_RX1_MODE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX1_MODE_REGOFF)
+#define HWIO_MIPI_TOP_RX1_MODE_RSVD0_FLDMASK (0xff000000)
+#define HWIO_MIPI_TOP_RX1_MODE_RSVD0_FLDSHFT (24)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_ECC_FLDMASK (0xff0000)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_ECC_FLDSHFT (16)
+#define HWIO_MIPI_TOP_RX1_MODE_RSVD1_FLDMASK (0xf800)
+#define HWIO_MIPI_TOP_RX1_MODE_RSVD1_FLDSHFT (11)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_VC_EN_FLDMASK (0x780)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_VC_EN_FLDSHFT (7)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_FORCE_OFF_FLDMASK (0x40)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_FORCE_OFF_FLDSHFT (6)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_CUR_BYP_MODE_FLDMASK (0x38)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_CUR_BYP_MODE_FLDSHFT (3)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_BYP_TX1_EN_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_BYP_TX1_EN_FLDSHFT (2)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_BYP_TX0_EN_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_BYP_TX0_EN_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_IPU_EN_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX1_MODE_RX1_IPU_EN_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX1_DPHY_CONFIG_REGOFF 0x78
+#define HWIO_MIPI_TOP_RX1_DPHY_CONFIG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX1_DPHY_CONFIG_REGOFF)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONFIG_RSVD0_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONFIG_RSVD0_FLDSHFT (16)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONFIG_CFGCLKFREQRANGE_FLDMASK (0xff00)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONFIG_CFGCLKFREQRANGE_FLDSHFT (8)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONFIG_RSVD1_FLDMASK (0x80)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONFIG_RSVD1_FLDSHFT (7)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONFIG_HSFREQRANGE_FLDMASK (0x7f)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONFIG_HSFREQRANGE_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX1_DPHY_CONTROL_REGOFF 0x7C
+#define HWIO_MIPI_TOP_RX1_DPHY_CONTROL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX1_DPHY_CONTROL_REGOFF)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONTROL_RSVD0_FLDMASK (0xfffffff8)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONTROL_RSVD0_FLDSHFT (3)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONTROL_PHY_SHADOW_CONTROL_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONTROL_PHY_SHADOW_CONTROL_FLDSHFT (2)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONTROL_RSTZ_N_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONTROL_RSTZ_N_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONTROL_SHUTDOWNZ_N_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX1_DPHY_CONTROL_SHUTDOWNZ_N_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_REGOFF 0x80
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX1_DPHY_IOTEST_REGOFF)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_RSVD0_FLDMASK (0xffff8000)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_RSVD0_FLDSHFT (15)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_BISTOK_FLDMASK (0x4000)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_BISTOK_FLDSHFT (14)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_BISTDONE_FLDMASK (0x2000)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_BISTDONE_FLDSHFT (13)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_BISTON_FLDMASK (0x1000)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_BISTON_FLDSHFT (12)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_CONT_DATA_FLDMASK (0xffe)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_CONT_DATA_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_CONT_EN_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX1_DPHY_IOTEST_CONT_EN_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX1_DPHY_ESCAPE_REGOFF 0x84
+#define HWIO_MIPI_TOP_RX1_DPHY_ESCAPE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX1_DPHY_ESCAPE_REGOFF)
+#define HWIO_MIPI_TOP_RX1_DPHY_ESCAPE_rx1_dphy_escape_FLDMASK (0xffffffff)
+#define HWIO_MIPI_TOP_RX1_DPHY_ESCAPE_rx1_dphy_escape_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX2_MODE_REGOFF 0x88
+#define HWIO_MIPI_TOP_RX2_MODE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX2_MODE_REGOFF)
+#define HWIO_MIPI_TOP_RX2_MODE_RSVD0_FLDMASK (0xff000000)
+#define HWIO_MIPI_TOP_RX2_MODE_RSVD0_FLDSHFT (24)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_ECC_FLDMASK (0xff0000)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_ECC_FLDSHFT (16)
+#define HWIO_MIPI_TOP_RX2_MODE_RSVD1_FLDMASK (0xf800)
+#define HWIO_MIPI_TOP_RX2_MODE_RSVD1_FLDSHFT (11)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_VC_EN_FLDMASK (0x780)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_VC_EN_FLDSHFT (7)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_FORCE_OFF_FLDMASK (0x40)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_FORCE_OFF_FLDSHFT (6)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_CUR_BYP_MODE_FLDMASK (0x38)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_CUR_BYP_MODE_FLDSHFT (3)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_BYP_TX1_EN_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_BYP_TX1_EN_FLDSHFT (2)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_BYP_TX0_EN_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_BYP_TX0_EN_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_IPU_EN_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX2_MODE_RX2_IPU_EN_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX2_DPHY_CONFIG_REGOFF 0x8C
+#define HWIO_MIPI_TOP_RX2_DPHY_CONFIG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX2_DPHY_CONFIG_REGOFF)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONFIG_RSVD0_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONFIG_RSVD0_FLDSHFT (16)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONFIG_CFGCLKFREQRANGE_FLDMASK (0xff00)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONFIG_CFGCLKFREQRANGE_FLDSHFT (8)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONFIG_RSVD1_FLDMASK (0x80)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONFIG_RSVD1_FLDSHFT (7)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONFIG_HSFREQRANGE_FLDMASK (0x7f)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONFIG_HSFREQRANGE_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX2_DPHY_CONTROL_REGOFF 0x90
+#define HWIO_MIPI_TOP_RX2_DPHY_CONTROL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX2_DPHY_CONTROL_REGOFF)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONTROL_RSVD0_FLDMASK (0xfffffff8)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONTROL_RSVD0_FLDSHFT (3)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONTROL_PHY_SHADOW_CONTROL_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONTROL_PHY_SHADOW_CONTROL_FLDSHFT (2)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONTROL_RSTZ_N_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONTROL_RSTZ_N_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONTROL_SHUTDOWNZ_N_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX2_DPHY_CONTROL_SHUTDOWNZ_N_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_REGOFF 0x94
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX2_DPHY_IOTEST_REGOFF)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_RSVD0_FLDMASK (0xffff8000)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_RSVD0_FLDSHFT (15)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_BISTOK_FLDMASK (0x4000)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_BISTOK_FLDSHFT (14)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_BISTDONE_FLDMASK (0x2000)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_BISTDONE_FLDSHFT (13)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_BISTON_FLDMASK (0x1000)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_BISTON_FLDSHFT (12)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_CONT_DATA_FLDMASK (0xffe)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_CONT_DATA_FLDSHFT (1)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_CONT_EN_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_RX2_DPHY_IOTEST_CONT_EN_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_RX2_DPHY_ESCAPE_REGOFF 0x98
+#define HWIO_MIPI_TOP_RX2_DPHY_ESCAPE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_RX2_DPHY_ESCAPE_REGOFF)
+#define HWIO_MIPI_TOP_RX2_DPHY_ESCAPE_rx2_dphy_escape_FLDMASK (0xffffffff)
+#define HWIO_MIPI_TOP_RX2_DPHY_ESCAPE_rx2_dphy_escape_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_IPU_VC_EN_REGOFF 0x9C
+#define HWIO_MIPI_TOP_TX0_IPU_VC_EN_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_IPU_VC_EN_REGOFF)
+#define HWIO_MIPI_TOP_TX0_IPU_VC_EN_RSVD0_FLDMASK (0xfffffff0)
+#define HWIO_MIPI_TOP_TX0_IPU_VC_EN_RSVD0_FLDSHFT (4)
+#define HWIO_MIPI_TOP_TX0_IPU_VC_EN_TX0_IPU_VC_EN_FLDMASK (0xf)
+#define HWIO_MIPI_TOP_TX0_IPU_VC_EN_TX0_IPU_VC_EN_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_IPU_VC_EN_REGOFF 0x100
+#define HWIO_MIPI_TOP_TX1_IPU_VC_EN_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_IPU_VC_EN_REGOFF)
+#define HWIO_MIPI_TOP_TX1_IPU_VC_EN_RSVD0_FLDMASK (0xfffffff0)
+#define HWIO_MIPI_TOP_TX1_IPU_VC_EN_RSVD0_FLDSHFT (4)
+#define HWIO_MIPI_TOP_TX1_IPU_VC_EN_TX1_IPU_VC_EN_FLDMASK (0xf)
+#define HWIO_MIPI_TOP_TX1_IPU_VC_EN_TX1_IPU_VC_EN_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_REGOFF 0x104
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_CSI_CLK_CTRL_REGOFF)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_RSVD0_FLDMASK (0xfffffe00)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_RSVD0_FLDSHFT (9)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_MIPI_DBG_SEL_FLDMASK (0x1c0)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_MIPI_DBG_SEL_FLDSHFT (6)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_MIPI_SW_RST_FLDMASK (0x20)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_MIPI_SW_RST_FLDSHFT (5)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_CSI2_RX2_CG_FLDMASK (0x10)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_CSI2_RX2_CG_FLDSHFT (4)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_CSI2_RX1_CG_FLDMASK (0x8)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_CSI2_RX1_CG_FLDSHFT (3)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_CSI2_RX0_CG_FLDMASK (0x4)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_CSI2_RX0_CG_FLDSHFT (2)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_CSI2_TX1_CG_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_CSI2_TX1_CG_FLDSHFT (1)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_CSI2_TX0_CG_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_CSI_CLK_CTRL_CSI2_TX0_CG_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_FRAME_TIMING_REGOFF 0x108
+#define HWIO_MIPI_TOP_TX0_FRAME_TIMING_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_FRAME_TIMING_REGOFF)
+#define HWIO_MIPI_TOP_TX0_FRAME_TIMING_RSVD0_FLDMASK (0xff000000)
+#define HWIO_MIPI_TOP_TX0_FRAME_TIMING_RSVD0_FLDSHFT (24)
+#define HWIO_MIPI_TOP_TX0_FRAME_TIMING_TX_FRAME_ACT_PERIOD_FLDMASK (0xffffff)
+#define HWIO_MIPI_TOP_TX0_FRAME_TIMING_TX_FRAME_ACT_PERIOD_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_VBLANK_TIMING_REGOFF 0x10C
+#define HWIO_MIPI_TOP_TX0_VBLANK_TIMING_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_VBLANK_TIMING_REGOFF)
+#define HWIO_MIPI_TOP_TX0_VBLANK_TIMING_RSVD0_FLDMASK (0xff000000)
+#define HWIO_MIPI_TOP_TX0_VBLANK_TIMING_RSVD0_FLDSHFT (24)
+#define HWIO_MIPI_TOP_TX0_VBLANK_TIMING_TX_FRAME_BLANK_PERIOD_FLDMASK (0xffffff)
+#define HWIO_MIPI_TOP_TX0_VBLANK_TIMING_TX_FRAME_BLANK_PERIOD_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX0_LINE_TIMING_REGOFF 0x110
+#define HWIO_MIPI_TOP_TX0_LINE_TIMING_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX0_LINE_TIMING_REGOFF)
+#define HWIO_MIPI_TOP_TX0_LINE_TIMING_TX_LINE_BLANK_PERIOD_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TOP_TX0_LINE_TIMING_TX_LINE_BLANK_PERIOD_FLDSHFT (16)
+#define HWIO_MIPI_TOP_TX0_LINE_TIMING_TX_LINE_ACT_PERIOD_FLDMASK (0xffff)
+#define HWIO_MIPI_TOP_TX0_LINE_TIMING_TX_LINE_ACT_PERIOD_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_FRAME_TIMING_REGOFF 0x114
+#define HWIO_MIPI_TOP_TX1_FRAME_TIMING_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_FRAME_TIMING_REGOFF)
+#define HWIO_MIPI_TOP_TX1_FRAME_TIMING_RSVD0_FLDMASK (0xff000000)
+#define HWIO_MIPI_TOP_TX1_FRAME_TIMING_RSVD0_FLDSHFT (24)
+#define HWIO_MIPI_TOP_TX1_FRAME_TIMING_TX_FRAME_ACT_PERIOD_FLDMASK (0xffffff)
+#define HWIO_MIPI_TOP_TX1_FRAME_TIMING_TX_FRAME_ACT_PERIOD_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_VBLANK_TIMING_REGOFF 0x118
+#define HWIO_MIPI_TOP_TX1_VBLANK_TIMING_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_VBLANK_TIMING_REGOFF)
+#define HWIO_MIPI_TOP_TX1_VBLANK_TIMING_RSVD0_FLDMASK (0xff000000)
+#define HWIO_MIPI_TOP_TX1_VBLANK_TIMING_RSVD0_FLDSHFT (24)
+#define HWIO_MIPI_TOP_TX1_VBLANK_TIMING_TX_FRAME_BLANK_PERIOD_FLDMASK (0xffffff)
+#define HWIO_MIPI_TOP_TX1_VBLANK_TIMING_TX_FRAME_BLANK_PERIOD_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_TX1_LINE_TIMING_REGOFF 0x11C
+#define HWIO_MIPI_TOP_TX1_LINE_TIMING_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_TX1_LINE_TIMING_REGOFF)
+#define HWIO_MIPI_TOP_TX1_LINE_TIMING_TX_LINE_BLANK_PERIOD_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TOP_TX1_LINE_TIMING_TX_LINE_BLANK_PERIOD_FLDSHFT (16)
+#define HWIO_MIPI_TOP_TX1_LINE_TIMING_TX_LINE_ACT_PERIOD_FLDMASK (0xffff)
+#define HWIO_MIPI_TOP_TX1_LINE_TIMING_TX_LINE_ACT_PERIOD_FLDSHFT (0)
+
+#define HWIO_MIPI_TOP_IDI_HALT_UD_EN_REGOFF 0x120
+#define HWIO_MIPI_TOP_IDI_HALT_UD_EN_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TOP_IDI_HALT_UD_EN_REGOFF)
+#define HWIO_MIPI_TOP_IDI_HALT_UD_EN_RSVD0_FLDMASK (0xfffffffc)
+#define HWIO_MIPI_TOP_IDI_HALT_UD_EN_RSVD0_FLDSHFT (2)
+#define HWIO_MIPI_TOP_IDI_HALT_UD_EN_TX1_IDI_HALT_EN_FLDMASK (0x2)
+#define HWIO_MIPI_TOP_IDI_HALT_UD_EN_TX1_IDI_HALT_EN_FLDSHFT (1)
+#define HWIO_MIPI_TOP_IDI_HALT_UD_EN_TX0_IDI_HALT_EN_FLDMASK (0x1)
+#define HWIO_MIPI_TOP_IDI_HALT_UD_EN_TX0_IDI_HALT_EN_FLDSHFT (0)
+
+#endif /* __MNH_HWIO_MIPI_TOP_ */
diff --git a/drivers/misc/mnh/mnh-hwio-mipi-tx.h b/drivers/misc/mnh/mnh-hwio-mipi-tx.h
new file mode 100644
index 0000000..45c49e7
--- /dev/null
+++ b/drivers/misc/mnh/mnh-hwio-mipi-tx.h
@@ -0,0 +1,486 @@
+/* auto generated: Monday, August 15th, 2016 12:26:49pm */
+/*
+ * Copyright (c) 2016, Intel Corporation. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this
+ * list of conditions and the following disclaimer.
+ *
+ * Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * Neither the name of Intel nor the names of its contributors may be used
+ * to endorse or promote products derived from this software without specific
+ * prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef __MNH_HWIO_MIPI_TX_
+#define __MNH_HWIO_MIPI_TX_
+
+#define HWIO_MIPI_TX_VERSION_REGOFF 0x0
+#define HWIO_MIPI_TX_VERSION_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VERSION_REGOFF)
+#define HWIO_MIPI_TX_VERSION_VERSION_FLDMASK (0xffffffff)
+#define HWIO_MIPI_TX_VERSION_VERSION_FLDSHFT (0)
+
+#define HWIO_MIPI_TX_CSI2_RESETN_REGOFF 0x4
+#define HWIO_MIPI_TX_CSI2_RESETN_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_CSI2_RESETN_REGOFF)
+#define HWIO_MIPI_TX_CSI2_RESETN_CSI2_RESETN_RW_FLDMASK (0x1)
+#define HWIO_MIPI_TX_CSI2_RESETN_CSI2_RESETN_RW_FLDSHFT (0)
+#define HWIO_MIPI_TX_CSI2_RESETN_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_TX_CSI2_RESETN_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_TX_INT_ST_MAIN_REGOFF 0x20
+#define HWIO_MIPI_TX_INT_ST_MAIN_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_INT_ST_MAIN_REGOFF)
+#define HWIO_MIPI_TX_INT_ST_MAIN_INT_ST_VPG_FLDMASK (0x1)
+#define HWIO_MIPI_TX_INT_ST_MAIN_INT_ST_VPG_FLDSHFT (0)
+#define HWIO_MIPI_TX_INT_ST_MAIN_INT_ST_IDI_FLDMASK (0x2)
+#define HWIO_MIPI_TX_INT_ST_MAIN_INT_ST_IDI_FLDSHFT (1)
+#define HWIO_MIPI_TX_INT_ST_MAIN_RESERVED_2_FLDMASK (0x4)
+#define HWIO_MIPI_TX_INT_ST_MAIN_RESERVED_2_FLDSHFT (2)
+#define HWIO_MIPI_TX_INT_ST_MAIN_INT_ST_PHY_FLDMASK (0x8)
+#define HWIO_MIPI_TX_INT_ST_MAIN_INT_ST_PHY_FLDSHFT (3)
+#define HWIO_MIPI_TX_INT_ST_MAIN_RESERVED_31_4_FLDMASK (0xfffffff0)
+#define HWIO_MIPI_TX_INT_ST_MAIN_RESERVED_31_4_FLDSHFT (4)
+
+#define HWIO_MIPI_TX_INT_ST_VPG_REGOFF 0x24
+#define HWIO_MIPI_TX_INT_ST_VPG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_INT_ST_VPG_REGOFF)
+#define HWIO_MIPI_TX_INT_ST_VPG_VPG_PKT_LOST_FLDMASK (0x1)
+#define HWIO_MIPI_TX_INT_ST_VPG_VPG_PKT_LOST_FLDSHFT (0)
+#define HWIO_MIPI_TX_INT_ST_VPG_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_TX_INT_ST_VPG_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_TX_INT_ST_IDI_REGOFF 0x28
+#define HWIO_MIPI_TX_INT_ST_IDI_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_INT_ST_IDI_REGOFF)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_ERRWC_FLDMASK (0x1)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_ERRWC_FLDSHFT (0)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC0_ERRF_SEQ_FLDMASK (0x2)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC0_ERRF_SEQ_FLDSHFT (1)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC1_ERRF_SEQ_FLDMASK (0x4)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC1_ERRF_SEQ_FLDSHFT (2)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC2_ERRF_SEQ_FLDMASK (0x8)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC2_ERRF_SEQ_FLDSHFT (3)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC3_ERRF_SEQ_FLDMASK (0x10)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC3_ERRF_SEQ_FLDSHFT (4)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC0_ERRL_SEQ_FLDMASK (0x20)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC0_ERRL_SEQ_FLDSHFT (5)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC1_ERRL_SEQ_FLDMASK (0x40)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC1_ERRL_SEQ_FLDSHFT (6)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC2_ERRL_SEQ_FLDMASK (0x80)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC2_ERRL_SEQ_FLDSHFT (7)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC3_ERRL_SEQ_FLDMASK (0x100)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_VC3_ERRL_SEQ_FLDSHFT (8)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_FIFO_OVERFLOW_FLDMASK (0x200)
+#define HWIO_MIPI_TX_INT_ST_IDI_IDI_FIFO_OVERFLOW_FLDSHFT (9)
+#define HWIO_MIPI_TX_INT_ST_IDI_RESERVED_31_10_FLDMASK (0xfffffc00)
+#define HWIO_MIPI_TX_INT_ST_IDI_RESERVED_31_10_FLDSHFT (10)
+
+#define HWIO_MIPI_TX_INT_ST_PHY_REGOFF 0x30
+#define HWIO_MIPI_TX_INT_ST_PHY_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_INT_ST_PHY_REGOFF)
+#define HWIO_MIPI_TX_INT_ST_PHY_TO_HS_TX_FLDMASK (0x1)
+#define HWIO_MIPI_TX_INT_ST_PHY_TO_HS_TX_FLDSHFT (0)
+#define HWIO_MIPI_TX_INT_ST_PHY_ERRCONTENTIONLP0_FLDMASK (0x2)
+#define HWIO_MIPI_TX_INT_ST_PHY_ERRCONTENTIONLP0_FLDSHFT (1)
+#define HWIO_MIPI_TX_INT_ST_PHY_ERRCONTENTIONLP1_FLDMASK (0x4)
+#define HWIO_MIPI_TX_INT_ST_PHY_ERRCONTENTIONLP1_FLDSHFT (2)
+#define HWIO_MIPI_TX_INT_ST_PHY_RESERVED_31_3_FLDMASK (0xfffffff8)
+#define HWIO_MIPI_TX_INT_ST_PHY_RESERVED_31_3_FLDSHFT (3)
+
+#define HWIO_MIPI_TX_INT_MASK_N_VPG_REGOFF 0x40
+#define HWIO_MIPI_TX_INT_MASK_N_VPG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_INT_MASK_N_VPG_REGOFF)
+#define HWIO_MIPI_TX_INT_MASK_N_VPG_MASK_VPG_PKT_LOST_FLDMASK (0x1)
+#define HWIO_MIPI_TX_INT_MASK_N_VPG_MASK_VPG_PKT_LOST_FLDSHFT (0)
+#define HWIO_MIPI_TX_INT_MASK_N_VPG_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_TX_INT_MASK_N_VPG_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_TX_INT_FORCE_VPG_REGOFF 0x44
+#define HWIO_MIPI_TX_INT_FORCE_VPG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_INT_FORCE_VPG_REGOFF)
+#define HWIO_MIPI_TX_INT_FORCE_VPG_FORCE_VPG_PKT_LOST_FLDMASK (0x1)
+#define HWIO_MIPI_TX_INT_FORCE_VPG_FORCE_VPG_PKT_LOST_FLDSHFT (0)
+#define HWIO_MIPI_TX_INT_FORCE_VPG_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_TX_INT_FORCE_VPG_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_REGOFF 0x48
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_INT_MASK_N_IDI_REGOFF)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_ERRWC_FLDMASK (0x1)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_ERRWC_FLDSHFT (0)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC0_ERRF_SEQ_FLDMASK (0x2)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC0_ERRF_SEQ_FLDSHFT (1)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC1_ERRF_SEQ_FLDMASK (0x4)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC1_ERRF_SEQ_FLDSHFT (2)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC2_ERRF_SEQ_FLDMASK (0x8)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC2_ERRF_SEQ_FLDSHFT (3)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC3_ERRF_SEQ_FLDMASK (0x10)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC3_ERRF_SEQ_FLDSHFT (4)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC0_ERRL_SEQ_FLDMASK (0x20)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC0_ERRL_SEQ_FLDSHFT (5)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC1_ERRL_SEQ_FLDMASK (0x40)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC1_ERRL_SEQ_FLDSHFT (6)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC2_ERRL_SEQ_FLDMASK (0x80)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC2_ERRL_SEQ_FLDSHFT (7)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC3_ERRL_SEQ_FLDMASK (0x100)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_VC3_ERRL_SEQ_FLDSHFT (8)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_FIFO_OVERFLOW_FLDMASK (0x200)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_MASK_IDI_FIFO_OVERFLOW_FLDSHFT (9)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_RESERVED_31_10_FLDMASK (0xfffffc00)
+#define HWIO_MIPI_TX_INT_MASK_N_IDI_RESERVED_31_10_FLDSHFT (10)
+
+#define HWIO_MIPI_TX_INT_FORCE_IDI_REGOFF 0x4c
+#define HWIO_MIPI_TX_INT_FORCE_IDI_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_INT_FORCE_IDI_REGOFF)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_ERRWC_FLDMASK (0x1)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_ERRWC_FLDSHFT (0)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC0_ERRF_SEQ_FLDMASK (0x2)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC0_ERRF_SEQ_FLDSHFT (1)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC1_ERRF_SEQ_FLDMASK (0x4)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC1_ERRF_SEQ_FLDSHFT (2)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC2_ERRF_SEQ_FLDMASK (0x8)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC2_ERRF_SEQ_FLDSHFT (3)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC3_ERRF_SEQ_FLDMASK (0x10)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC3_ERRF_SEQ_FLDSHFT (4)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC0_ERRL_SEQ_FLDMASK (0x20)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC0_ERRL_SEQ_FLDSHFT (5)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC1_ERRL_SEQ_FLDMASK (0x40)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC1_ERRL_SEQ_FLDSHFT (6)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC2_ERRL_SEQ_FLDMASK (0x80)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC2_ERRL_SEQ_FLDSHFT (7)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC3_ERRL_SEQ_FLDMASK (0x100)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_VC3_ERRL_SEQ_FLDSHFT (8)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_FIFO_OVERFLOW_FLDMASK (0x200)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_FORCE_IDI_FIFO_OVERFLOW_FLDSHFT (9)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_RESERVED_31_10_FLDMASK (0xfffffc00)
+#define HWIO_MIPI_TX_INT_FORCE_IDI_RESERVED_31_10_FLDSHFT (10)
+
+#define HWIO_MIPI_TX_INT_MASK_N_PHY_REGOFF 0x58
+#define HWIO_MIPI_TX_INT_MASK_N_PHY_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_INT_MASK_N_PHY_REGOFF)
+#define HWIO_MIPI_TX_INT_MASK_N_PHY_MASK_TO_HS_TX_FLDMASK (0x1)
+#define HWIO_MIPI_TX_INT_MASK_N_PHY_MASK_TO_HS_TX_FLDSHFT (0)
+#define HWIO_MIPI_TX_INT_MASK_N_PHY_MASK_ERRCONTENTIONLP0_FLDMASK (0x2)
+#define HWIO_MIPI_TX_INT_MASK_N_PHY_MASK_ERRCONTENTIONLP0_FLDSHFT (1)
+#define HWIO_MIPI_TX_INT_MASK_N_PHY_MASK_ERRCONTENTIONLP1_FLDMASK (0x4)
+#define HWIO_MIPI_TX_INT_MASK_N_PHY_MASK_ERRCONTENTIONLP1_FLDSHFT (2)
+#define HWIO_MIPI_TX_INT_MASK_N_PHY_RESERVED_31_3_FLDMASK (0xfffffff8)
+#define HWIO_MIPI_TX_INT_MASK_N_PHY_RESERVED_31_3_FLDSHFT (3)
+
+#define HWIO_MIPI_TX_INT_FORCE_PHY_REGOFF 0x5c
+#define HWIO_MIPI_TX_INT_FORCE_PHY_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_INT_FORCE_PHY_REGOFF)
+#define HWIO_MIPI_TX_INT_FORCE_PHY_FORCE_TO_HS_TX_FLDMASK (0x1)
+#define HWIO_MIPI_TX_INT_FORCE_PHY_FORCE_TO_HS_TX_FLDSHFT (0)
+#define HWIO_MIPI_TX_INT_FORCE_PHY_FORCE_ERRCONTENTIONLP0_FLDMASK (0x2)
+#define HWIO_MIPI_TX_INT_FORCE_PHY_FORCE_ERRCONTENTIONLP0_FLDSHFT (1)
+#define HWIO_MIPI_TX_INT_FORCE_PHY_FORCE_ERRCONTENTIONLP1_FLDMASK (0x4)
+#define HWIO_MIPI_TX_INT_FORCE_PHY_FORCE_ERRCONTENTIONLP1_FLDSHFT (2)
+#define HWIO_MIPI_TX_INT_FORCE_PHY_RESERVED_31_3_FLDMASK (0xfffffff8)
+#define HWIO_MIPI_TX_INT_FORCE_PHY_RESERVED_31_3_FLDSHFT (3)
+
+#define HWIO_MIPI_TX_VPG_CTRL_REGOFF 0x80
+#define HWIO_MIPI_TX_VPG_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_CTRL_REGOFF)
+#define HWIO_MIPI_TX_VPG_CTRL_VPG_EN_FLDMASK (0x1)
+#define HWIO_MIPI_TX_VPG_CTRL_VPG_EN_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_CTRL_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_TX_VPG_CTRL_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_TX_VPG_STATUS_REGOFF 0x84
+#define HWIO_MIPI_TX_VPG_STATUS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_STATUS_REGOFF)
+#define HWIO_MIPI_TX_VPG_STATUS_VPG_ACTIVE_FLDMASK (0x1)
+#define HWIO_MIPI_TX_VPG_STATUS_VPG_ACTIVE_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_STATUS_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_TX_VPG_STATUS_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_TX_VPG_MODE_CFG_REGOFF 0x88
+#define HWIO_MIPI_TX_VPG_MODE_CFG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_MODE_CFG_REGOFF)
+#define HWIO_MIPI_TX_VPG_MODE_CFG_VPG_MODE_FLDMASK (0x1)
+#define HWIO_MIPI_TX_VPG_MODE_CFG_VPG_MODE_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_MODE_CFG_RESERVED_15_1_FLDMASK (0xfffe)
+#define HWIO_MIPI_TX_VPG_MODE_CFG_RESERVED_15_1_FLDSHFT (1)
+#define HWIO_MIPI_TX_VPG_MODE_CFG_VPG_ORIENTATION_FLDMASK (0x10000)
+#define HWIO_MIPI_TX_VPG_MODE_CFG_VPG_ORIENTATION_FLDSHFT (16)
+#define HWIO_MIPI_TX_VPG_MODE_CFG_RESERVED_31_17_FLDMASK (0xfffe0000)
+#define HWIO_MIPI_TX_VPG_MODE_CFG_RESERVED_31_17_FLDSHFT (17)
+
+#define HWIO_MIPI_TX_VPG_PKT_CFG_REGOFF 0x8c
+#define HWIO_MIPI_TX_VPG_PKT_CFG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_PKT_CFG_REGOFF)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_VPG_DT_FLDMASK (0x3f)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_VPG_DT_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_VPG_VC_FLDMASK (0xc0)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_VPG_VC_FLDSHFT (6)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_VPG_HSYNC_PKT_EN_FLDMASK (0x100)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_VPG_HSYNC_PKT_EN_FLDSHFT (8)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_VPG_LINE_NUM_MODE_FLDMASK (0x600)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_VPG_LINE_NUM_MODE_FLDSHFT (9)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_VPG_FRAME_NUM_MODE_FLDMASK (0x800)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_VPG_FRAME_NUM_MODE_FLDSHFT (11)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_RESERVED_31_12_FLDMASK (0xfffff000)
+#define HWIO_MIPI_TX_VPG_PKT_CFG_RESERVED_31_12_FLDSHFT (12)
+
+#define HWIO_MIPI_TX_VPG_PKT_SIZE_REGOFF 0x90
+#define HWIO_MIPI_TX_VPG_PKT_SIZE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_PKT_SIZE_REGOFF)
+#define HWIO_MIPI_TX_VPG_PKT_SIZE_VPG_PKT_SIZE_FLDMASK (0x3fff)
+#define HWIO_MIPI_TX_VPG_PKT_SIZE_VPG_PKT_SIZE_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_PKT_SIZE_RESERVED_31_14_FLDMASK (0xffffc000)
+#define HWIO_MIPI_TX_VPG_PKT_SIZE_RESERVED_31_14_FLDSHFT (14)
+
+#define HWIO_MIPI_TX_VPG_HSA_TIME_REGOFF 0x94
+#define HWIO_MIPI_TX_VPG_HSA_TIME_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_HSA_TIME_REGOFF)
+#define HWIO_MIPI_TX_VPG_HSA_TIME_VPG_HSA_TIME_FLDMASK (0xfff)
+#define HWIO_MIPI_TX_VPG_HSA_TIME_VPG_HSA_TIME_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_HSA_TIME_RESERVED_31_12_FLDMASK (0xfffff000)
+#define HWIO_MIPI_TX_VPG_HSA_TIME_RESERVED_31_12_FLDSHFT (12)
+
+#define HWIO_MIPI_TX_VPG_HBP_TIME_REGOFF 0x98
+#define HWIO_MIPI_TX_VPG_HBP_TIME_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_HBP_TIME_REGOFF)
+#define HWIO_MIPI_TX_VPG_HBP_TIME_VPG_HBP_TIME_FLDMASK (0xfff)
+#define HWIO_MIPI_TX_VPG_HBP_TIME_VPG_HBP_TIME_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_HBP_TIME_RESERVED_31_12_FLDMASK (0xfffff000)
+#define HWIO_MIPI_TX_VPG_HBP_TIME_RESERVED_31_12_FLDSHFT (12)
+
+#define HWIO_MIPI_TX_VPG_HLINE_TIME_REGOFF 0x9c
+#define HWIO_MIPI_TX_VPG_HLINE_TIME_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_HLINE_TIME_REGOFF)
+#define HWIO_MIPI_TX_VPG_HLINE_TIME_VPG_HLINE_TIME_FLDMASK (0x7fff)
+#define HWIO_MIPI_TX_VPG_HLINE_TIME_VPG_HLINE_TIME_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_HLINE_TIME_RESERVED_31_15_FLDMASK (0xffff8000)
+#define HWIO_MIPI_TX_VPG_HLINE_TIME_RESERVED_31_15_FLDSHFT (15)
+
+#define HWIO_MIPI_TX_VPG_VSA_LINES_REGOFF 0xa0
+#define HWIO_MIPI_TX_VPG_VSA_LINES_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_VSA_LINES_REGOFF)
+#define HWIO_MIPI_TX_VPG_VSA_LINES_VPG_VSA_LINES_FLDMASK (0x3ff)
+#define HWIO_MIPI_TX_VPG_VSA_LINES_VPG_VSA_LINES_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_VSA_LINES_RESERVED_31_10_FLDMASK (0xfffffc00)
+#define HWIO_MIPI_TX_VPG_VSA_LINES_RESERVED_31_10_FLDSHFT (10)
+
+#define HWIO_MIPI_TX_VPG_VBP_LINES_REGOFF 0xa4
+#define HWIO_MIPI_TX_VPG_VBP_LINES_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_VBP_LINES_REGOFF)
+#define HWIO_MIPI_TX_VPG_VBP_LINES_VPG_VBP_LINES_FLDMASK (0x3ff)
+#define HWIO_MIPI_TX_VPG_VBP_LINES_VPG_VBP_LINES_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_VBP_LINES_RESERVED_31_10_FLDMASK (0xfffffc00)
+#define HWIO_MIPI_TX_VPG_VBP_LINES_RESERVED_31_10_FLDSHFT (10)
+
+#define HWIO_MIPI_TX_VPG_VFP_LINES_REGOFF 0xa8
+#define HWIO_MIPI_TX_VPG_VFP_LINES_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_VFP_LINES_REGOFF)
+#define HWIO_MIPI_TX_VPG_VFP_LINES_VPG_VFP_LINES_FLDMASK (0x3ff)
+#define HWIO_MIPI_TX_VPG_VFP_LINES_VPG_VFP_LINES_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_VFP_LINES_RESERVED_31_10_FLDMASK (0xfffffc00)
+#define HWIO_MIPI_TX_VPG_VFP_LINES_RESERVED_31_10_FLDSHFT (10)
+
+#define HWIO_MIPI_TX_VPG_ACT_LINES_REGOFF 0xac
+#define HWIO_MIPI_TX_VPG_ACT_LINES_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_ACT_LINES_REGOFF)
+#define HWIO_MIPI_TX_VPG_ACT_LINES_VPG_ACT_LINES_FLDMASK (0x3fff)
+#define HWIO_MIPI_TX_VPG_ACT_LINES_VPG_ACT_LINES_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_ACT_LINES_RESERVED_31_14_FLDMASK (0xffffc000)
+#define HWIO_MIPI_TX_VPG_ACT_LINES_RESERVED_31_14_FLDSHFT (14)
+
+#define HWIO_MIPI_TX_VPG_MAX_FRAME_NUM_REGOFF 0xb0
+#define HWIO_MIPI_TX_VPG_MAX_FRAME_NUM_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_MAX_FRAME_NUM_REGOFF)
+#define HWIO_MIPI_TX_VPG_MAX_FRAME_NUM_VPG_MAX_FRAME_NUM_FLDMASK (0xffff)
+#define HWIO_MIPI_TX_VPG_MAX_FRAME_NUM_VPG_MAX_FRAME_NUM_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_MAX_FRAME_NUM_RESERVED_31_16_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TX_VPG_MAX_FRAME_NUM_RESERVED_31_16_FLDSHFT (16)
+
+#define HWIO_MIPI_TX_VPG_START_LINE_NUM_REGOFF 0xb4
+#define HWIO_MIPI_TX_VPG_START_LINE_NUM_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_START_LINE_NUM_REGOFF)
+#define HWIO_MIPI_TX_VPG_START_LINE_NUM_VPG_START_LINE_NUM_FLDMASK (0xffff)
+#define HWIO_MIPI_TX_VPG_START_LINE_NUM_VPG_START_LINE_NUM_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_START_LINE_NUM_RESERVED_31_16_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TX_VPG_START_LINE_NUM_RESERVED_31_16_FLDSHFT (16)
+
+#define HWIO_MIPI_TX_VPG_STEP_LINE_NUM_REGOFF 0xb8
+#define HWIO_MIPI_TX_VPG_STEP_LINE_NUM_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_VPG_STEP_LINE_NUM_REGOFF)
+#define HWIO_MIPI_TX_VPG_STEP_LINE_NUM_VPG_STEP_LINE_NUM_FLDMASK (0xffff)
+#define HWIO_MIPI_TX_VPG_STEP_LINE_NUM_VPG_STEP_LINE_NUM_FLDSHFT (0)
+#define HWIO_MIPI_TX_VPG_STEP_LINE_NUM_RESERVED_31_16_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TX_VPG_STEP_LINE_NUM_RESERVED_31_16_FLDSHFT (16)
+
+#define HWIO_MIPI_TX_PHY_RSTZ_REGOFF 0xe0
+#define HWIO_MIPI_TX_PHY_RSTZ_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_PHY_RSTZ_REGOFF)
+#define HWIO_MIPI_TX_PHY_RSTZ_PHY_SHUTDOWNZ_FLDMASK (0x1)
+#define HWIO_MIPI_TX_PHY_RSTZ_PHY_SHUTDOWNZ_FLDSHFT (0)
+#define HWIO_MIPI_TX_PHY_RSTZ_PHY_RSTZ_FLDMASK (0x2)
+#define HWIO_MIPI_TX_PHY_RSTZ_PHY_RSTZ_FLDSHFT (1)
+#define HWIO_MIPI_TX_PHY_RSTZ_PHY_ENABLECLK_FLDMASK (0x4)
+#define HWIO_MIPI_TX_PHY_RSTZ_PHY_ENABLECLK_FLDSHFT (2)
+#define HWIO_MIPI_TX_PHY_RSTZ_PHY_FORCEPLL_FLDMASK (0x8)
+#define HWIO_MIPI_TX_PHY_RSTZ_PHY_FORCEPLL_FLDSHFT (3)
+#define HWIO_MIPI_TX_PHY_RSTZ_RESERVED_31_4_FLDMASK (0xfffffff0)
+#define HWIO_MIPI_TX_PHY_RSTZ_RESERVED_31_4_FLDSHFT (4)
+
+#define HWIO_MIPI_TX_PHY_IF_CFG_REGOFF 0xe4
+#define HWIO_MIPI_TX_PHY_IF_CFG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_PHY_IF_CFG_REGOFF)
+#define HWIO_MIPI_TX_PHY_IF_CFG_LANE_EN_NUM_FLDMASK (0x7)
+#define HWIO_MIPI_TX_PHY_IF_CFG_LANE_EN_NUM_FLDSHFT (0)
+#define HWIO_MIPI_TX_PHY_IF_CFG_RESERVED_7_3_FLDMASK (0xf8)
+#define HWIO_MIPI_TX_PHY_IF_CFG_RESERVED_7_3_FLDSHFT (3)
+#define HWIO_MIPI_TX_PHY_IF_CFG_PHY_STOP_WAIT_TIME_FLDMASK (0xff00)
+#define HWIO_MIPI_TX_PHY_IF_CFG_PHY_STOP_WAIT_TIME_FLDSHFT (8)
+#define HWIO_MIPI_TX_PHY_IF_CFG_RESERVED_31_16_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TX_PHY_IF_CFG_RESERVED_31_16_FLDSHFT (16)
+
+#define HWIO_MIPI_TX_LPCLK_CTRL_REGOFF 0xe8
+#define HWIO_MIPI_TX_LPCLK_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_LPCLK_CTRL_REGOFF)
+#define HWIO_MIPI_TX_LPCLK_CTRL_PHY_TXREQCLKHS_CON_FLDMASK (0x1)
+#define HWIO_MIPI_TX_LPCLK_CTRL_PHY_TXREQCLKHS_CON_FLDSHFT (0)
+#define HWIO_MIPI_TX_LPCLK_CTRL_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_TX_LPCLK_CTRL_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_REGOFF 0xec
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_PHY_ULPS_CTRL_REGOFF)
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_PHY_TXREQULPSCLK_FLDMASK (0x1)
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_PHY_TXREQULPSCLK_FLDSHFT (0)
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_PHY_TXEXITULPSCLK_FLDMASK (0x2)
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_PHY_TXEXITULPSCLK_FLDSHFT (1)
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_PHY_TXREQULPSLAN_FLDMASK (0x4)
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_PHY_TXREQULPSLAN_FLDSHFT (2)
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_PHY_TXEXITULPSLAN_FLDMASK (0x8)
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_PHY_TXEXITULPSLAN_FLDSHFT (3)
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_RESERVED_31_4_FLDMASK (0xfffffff0)
+#define HWIO_MIPI_TX_PHY_ULPS_CTRL_RESERVED_31_4_FLDSHFT (4)
+
+#define HWIO_MIPI_TX_CLKMGR_CFG_REGOFF 0xf0
+#define HWIO_MIPI_TX_CLKMGR_CFG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_CLKMGR_CFG_REGOFF)
+#define HWIO_MIPI_TX_CLKMGR_CFG_TX_ESC_CLK_DIVISION_FLDMASK (0xff)
+#define HWIO_MIPI_TX_CLKMGR_CFG_TX_ESC_CLK_DIVISION_FLDSHFT (0)
+#define HWIO_MIPI_TX_CLKMGR_CFG_TO_CLK_DIVISION_FLDMASK (0xff00)
+#define HWIO_MIPI_TX_CLKMGR_CFG_TO_CLK_DIVISION_FLDSHFT (8)
+#define HWIO_MIPI_TX_CLKMGR_CFG_RESERVED_31_16_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TX_CLKMGR_CFG_RESERVED_31_16_FLDSHFT (16)
+
+#define HWIO_MIPI_TX_PHY_TX_TRIGGERS_REGOFF 0xf4
+#define HWIO_MIPI_TX_PHY_TX_TRIGGERS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_PHY_TX_TRIGGERS_REGOFF)
+#define HWIO_MIPI_TX_PHY_TX_TRIGGERS_TX_TRIGGERS_FLDMASK (0x3)
+#define HWIO_MIPI_TX_PHY_TX_TRIGGERS_TX_TRIGGERS_FLDSHFT (0)
+#define HWIO_MIPI_TX_PHY_TX_TRIGGERS_RESERVED_6_2_FLDMASK (0x7c)
+#define HWIO_MIPI_TX_PHY_TX_TRIGGERS_RESERVED_6_2_FLDSHFT (2)
+#define HWIO_MIPI_TX_PHY_TX_TRIGGERS_TX_TRIGGERS_EN_FLDMASK (0x80)
+#define HWIO_MIPI_TX_PHY_TX_TRIGGERS_TX_TRIGGERS_EN_FLDSHFT (7)
+#define HWIO_MIPI_TX_PHY_TX_TRIGGERS_RESERVED_31_8_FLDMASK (0xffffff00)
+#define HWIO_MIPI_TX_PHY_TX_TRIGGERS_RESERVED_31_8_FLDSHFT (8)
+
+#define HWIO_MIPI_TX_PHY_CAL_REGOFF 0xf8
+#define HWIO_MIPI_TX_PHY_CAL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_PHY_CAL_REGOFF)
+#define HWIO_MIPI_TX_PHY_CAL_PHY_CAL_FLDMASK (0x1)
+#define HWIO_MIPI_TX_PHY_CAL_PHY_CAL_FLDSHFT (0)
+#define HWIO_MIPI_TX_PHY_CAL_RESERVED_31_1_FLDMASK (0xfffffffe)
+#define HWIO_MIPI_TX_PHY_CAL_RESERVED_31_1_FLDSHFT (1)
+
+#define HWIO_MIPI_TX_TO_CNT_CFG_REGOFF 0xfc
+#define HWIO_MIPI_TX_TO_CNT_CFG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_TO_CNT_CFG_REGOFF)
+#define HWIO_MIPI_TX_TO_CNT_CFG_HSTX_TO_CNT_FLDMASK (0xffff)
+#define HWIO_MIPI_TX_TO_CNT_CFG_HSTX_TO_CNT_FLDSHFT (0)
+#define HWIO_MIPI_TX_TO_CNT_CFG_RESERVED_31_16_FLDMASK (0xffff0000)
+#define HWIO_MIPI_TX_TO_CNT_CFG_RESERVED_31_16_FLDSHFT (16)
+
+#define HWIO_MIPI_TX_PHY_STATUS_REGOFF 0x110
+#define HWIO_MIPI_TX_PHY_STATUS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_PHY_STATUS_REGOFF)
+#define HWIO_MIPI_TX_PHY_STATUS_LANE_MAX_NUM_FLDMASK (0x7)
+#define HWIO_MIPI_TX_PHY_STATUS_LANE_MAX_NUM_FLDSHFT (0)
+#define HWIO_MIPI_TX_PHY_STATUS_PLL_LOCK_FLDMASK (0x8)
+#define HWIO_MIPI_TX_PHY_STATUS_PLL_LOCK_FLDSHFT (3)
+#define HWIO_MIPI_TX_PHY_STATUS_TXSTOPSTATE_CLK_FLDMASK (0x10)
+#define HWIO_MIPI_TX_PHY_STATUS_TXSTOPSTATE_CLK_FLDSHFT (4)
+#define HWIO_MIPI_TX_PHY_STATUS_TXULPSACTIVENOT_CLK_FLDMASK (0x20)
+#define HWIO_MIPI_TX_PHY_STATUS_TXULPSACTIVENOT_CLK_FLDSHFT (5)
+#define HWIO_MIPI_TX_PHY_STATUS_TXSTOPSTATE_L0_FLDMASK (0x40)
+#define HWIO_MIPI_TX_PHY_STATUS_TXSTOPSTATE_L0_FLDSHFT (6)
+#define HWIO_MIPI_TX_PHY_STATUS_TXULPSACTIVENOT_L0_FLDMASK (0x80)
+#define HWIO_MIPI_TX_PHY_STATUS_TXULPSACTIVENOT_L0_FLDSHFT (7)
+#define HWIO_MIPI_TX_PHY_STATUS_TXSTOPSTATE_L1_FLDMASK (0x100)
+#define HWIO_MIPI_TX_PHY_STATUS_TXSTOPSTATE_L1_FLDSHFT (8)
+#define HWIO_MIPI_TX_PHY_STATUS_TXULPSACTIVENOT_L1_FLDMASK (0x200)
+#define HWIO_MIPI_TX_PHY_STATUS_TXULPSACTIVENOT_L1_FLDSHFT (9)
+#define HWIO_MIPI_TX_PHY_STATUS_TXSTOPSTATE_L2_FLDMASK (0x400)
+#define HWIO_MIPI_TX_PHY_STATUS_TXSTOPSTATE_L2_FLDSHFT (10)
+#define HWIO_MIPI_TX_PHY_STATUS_TXULPSACTIVENOT_L2_FLDMASK (0x800)
+#define HWIO_MIPI_TX_PHY_STATUS_TXULPSACTIVENOT_L2_FLDSHFT (11)
+#define HWIO_MIPI_TX_PHY_STATUS_TXSTOPSTATE_L3_FLDMASK (0x1000)
+#define HWIO_MIPI_TX_PHY_STATUS_TXSTOPSTATE_L3_FLDSHFT (12)
+#define HWIO_MIPI_TX_PHY_STATUS_TXULPSACTIVENOT_L3_FLDMASK (0x2000)
+#define HWIO_MIPI_TX_PHY_STATUS_TXULPSACTIVENOT_L3_FLDSHFT (13)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_14_FLDMASK (0x4000)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_14_FLDSHFT (14)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_15_FLDMASK (0x8000)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_15_FLDSHFT (15)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_16_FLDMASK (0x10000)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_16_FLDSHFT (16)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_17_FLDMASK (0x20000)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_17_FLDSHFT (17)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_18_FLDMASK (0x40000)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_18_FLDSHFT (18)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_19_FLDMASK (0x80000)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_19_FLDSHFT (19)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_20_FLDMASK (0x100000)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_20_FLDSHFT (20)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_21_FLDMASK (0x200000)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_21_FLDSHFT (21)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_31_22_FLDMASK (0xffc00000)
+#define HWIO_MIPI_TX_PHY_STATUS_RESERVED_31_22_FLDSHFT (22)
+
+#define HWIO_MIPI_TX_PHY0_TST_CTRL0_REGOFF 0x114
+#define HWIO_MIPI_TX_PHY0_TST_CTRL0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_PHY0_TST_CTRL0_REGOFF)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL0_PHY0_TESTCLR_FLDMASK (0x1)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL0_PHY0_TESTCLR_FLDSHFT (0)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL0_PHY0_TESTCLK_FLDMASK (0x2)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL0_PHY0_TESTCLK_FLDSHFT (1)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL0_RESERVED_31_2_FLDMASK (0xfffffffc)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL0_RESERVED_31_2_FLDSHFT (2)
+
+#define HWIO_MIPI_TX_PHY0_TST_CTRL1_REGOFF 0x118
+#define HWIO_MIPI_TX_PHY0_TST_CTRL1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_MIPI_TX_PHY0_TST_CTRL1_REGOFF)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL1_PHY0_TESTDIN_FLDMASK (0xff)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL1_PHY0_TESTDIN_FLDSHFT (0)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL1_PHY0_TESTDOUT_FLDMASK (0xff00)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL1_PHY0_TESTDOUT_FLDSHFT (8)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL1_PHY0_TESTEN_FLDMASK (0x10000)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL1_PHY0_TESTEN_FLDSHFT (16)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL1_RESERVED_31_17_FLDMASK (0xfffe0000)
+#define HWIO_MIPI_TX_PHY0_TST_CTRL1_RESERVED_31_17_FLDSHFT (17)
+
+#endif /* __MNH_HWIO_MIPI_TX_ */
diff --git a/drivers/misc/mnh/mnh-hwio-pcie-ss.h b/drivers/misc/mnh/mnh-hwio-pcie-ss.h
new file mode 100644
index 0000000..24bb4ad
--- /dev/null
+++ b/drivers/misc/mnh/mnh-hwio-pcie-ss.h
@@ -0,0 +1,398 @@
+/* auto generated: Monday, August 15th, 2016 12:26:47pm */
+/*
+ * Copyright (c) 2016, Intel Corporation. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this
+ * list of conditions and the following disclaimer.
+ *
+ * Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * Neither the name of Intel nor the names of its contributors may be used
+ * to endorse or promote products derived from this software without specific
+ * prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef __MNH_HWIO_PCIE_SS_
+#define __MNH_HWIO_PCIE_SS_
+
+#define HWIO_PCIE_SS_PCIE_SW_INTR_TRIGG_REGOFF 0x0
+#define HWIO_PCIE_SS_PCIE_SW_INTR_TRIGG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_SW_INTR_TRIGG_REGOFF)
+#define HWIO_PCIE_SS_PCIE_SW_INTR_TRIGG_PCIE_SW_INTR_TRIGG_FLDMASK (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_SW_INTR_TRIGG_PCIE_SW_INTR_TRIGG_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_SW_INTR_EN_REGOFF 0x4
+#define HWIO_PCIE_SS_PCIE_SW_INTR_EN_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_SW_INTR_EN_REGOFF)
+#define HWIO_PCIE_SS_PCIE_SW_INTR_EN_RSVD0_FLDMASK (0x80000000)
+#define HWIO_PCIE_SS_PCIE_SW_INTR_EN_RSVD0_FLDSHFT (31)
+#define HWIO_PCIE_SS_PCIE_SW_INTR_EN_PCIE_SW_INTR_EN_FLDMASK (0x7fffffff)
+#define HWIO_PCIE_SS_PCIE_SW_INTR_EN_PCIE_SW_INTR_EN_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_REGOFF 0x8
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_SS_INTR_STS_REGOFF)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_RSVD0_FLDMASK (0xff000000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_RSVD0_FLDSHFT (24)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_TRGT_CPL_TIMEOUT_FLDMASK (0x800000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_TRGT_CPL_TIMEOUT_FLDSHFT (23)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_RADM_CPL_TIMEOUT_FLDMASK (0x400000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_RADM_CPL_TIMEOUT_FLDSHFT (22)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_PM_TURNOFF_FLDMASK (0x200000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_PM_TURNOFF_FLDSHFT (21)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_RADM_MSG_UNLOCK_FLDMASK (0x100000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_RADM_MSG_UNLOCK_FLDSHFT (20)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_RSVD1_FLDMASK (0x80000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_RSVD1_FLDSHFT (19)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_FATAL_ERR_FLDMASK (0x40000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_FATAL_ERR_FLDSHFT (18)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_NONFATAL_ERR_FLDMASK (0x20000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_NONFATAL_ERR_FLDSHFT (17)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_COR_ERR_FLDMASK (0x10000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_COR_ERR_FLDSHFT (16)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_RSVD2_FLDMASK (0xfe00)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_RSVD2_FLDSHFT (9)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_FRS_SENT_FLDMASK (0x100)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_FRS_SENT_FLDSHFT (8)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_DRS_SENT_FLDMASK (0x80)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_DRS_SENT_FLDSHFT (7)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_LTR_SENT_FLDMASK (0x40)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_LTR_SENT_FLDSHFT (6)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_VMSG0_RXD_FLDMASK (0x20)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_VMSG0_RXD_FLDSHFT (5)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_VMSG1_RXD_FLDMASK (0x10)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_VMSG1_RXD_FLDSHFT (4)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_VMSG_SENT_FLDMASK (0x8)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_VMSG_SENT_FLDSHFT (3)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_MSI_SENT_FLDMASK (0x4)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_MSI_SENT_FLDSHFT (2)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_LINK_REQ_RST_NOT_FLDMASK (0x2)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_LINK_REQ_RST_NOT_FLDSHFT (1)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_LINK_EQ_REQ_INT_FLDMASK (0x1)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_STS_PCIE_LINK_EQ_REQ_INT_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_REGOFF 0x0C
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_SS_INTR_EN_REGOFF)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_RSVD0_FLDMASK (0xff000000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_RSVD0_FLDSHFT (24)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_TRGT_CPL_TIMEOUT_IE_FLDMASK (0x800000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_TRGT_CPL_TIMEOUT_IE_FLDSHFT (23)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_RADM_CPL_TIMEOUT_IE_FLDMASK (0x400000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_RADM_CPL_TIMEOUT_IE_FLDSHFT (22)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_PM_TURNOFF_IE_FLDMASK (0x200000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_PM_TURNOFF_IE_FLDSHFT (21)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_RADM_MSG_UNLOCK_IE_FLDMASK (0x100000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_RADM_MSG_UNLOCK_IE_FLDSHFT (20)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_RSVD1_FLDMASK (0x80000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_RSVD1_FLDSHFT (19)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_FATAL_ERR_IE_FLDMASK (0x40000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_FATAL_ERR_IE_FLDSHFT (18)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_NONFATAL_ERR_IE_FLDMASK (0x20000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_NONFATAL_ERR_IE_FLDSHFT (17)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_COR_ERR_IE_FLDMASK (0x10000)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_COR_ERR_IE_FLDSHFT (16)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_RSVD2_FLDMASK (0xfe00)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_RSVD2_FLDSHFT (9)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_FRS_SENT_IE_FLDMASK (0x100)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_FRS_SENT_IE_FLDSHFT (8)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_DRS_SENT_IE_FLDMASK (0x80)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_DRS_SENT_IE_FLDSHFT (7)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_LTR_SENT_IE_FLDMASK (0x40)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_LTR_SENT_IE_FLDSHFT (6)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_VMSG0_RXD_IE_FLDMASK (0x20)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_VMSG0_RXD_IE_FLDSHFT (5)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_VMSG1_RXD_IE_FLDMASK (0x10)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_VMSG1_RXD_IE_FLDSHFT (4)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_VMSG_SENT_IE_FLDMASK (0x8)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_VMSG_SENT_IE_FLDSHFT (3)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_MSI_SENT_IE_FLDMASK (0x4)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_MSI_SENT_IE_FLDSHFT (2)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_LINK_REQ_RST_NOT_IE_FLDMASK (0x2)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_LINK_REQ_RST_NOT_IE_FLDSHFT (1)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_LINK_EQ_REQ_IE_FLDMASK (0x1)
+#define HWIO_PCIE_SS_PCIE_SS_INTR_EN_PCIE_LINK_EQ_REQ_IE_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_REGOFF 0x10
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_APP_CTRL_REGOFF)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_RSVD0_FLDMASK (0xffc00000)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_RSVD0_FLDSHFT (22)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_PHY_CMNREF_CLK_MODE_FLDMASK (0x200000)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_PHY_CMNREF_CLK_MODE_FLDSHFT (21)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_PHY_SRAM_LD_DONE_FLDMASK (0x100000)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_PHY_SRAM_LD_DONE_FLDSHFT (20)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_PHY_SRAM_BYPASS_FLDMASK (0x80000)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_PHY_SRAM_BYPASS_FLDSHFT (19)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_TEST_BYPASS_LP_FLDMASK (0x40000)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_TEST_BYPASS_LP_FLDSHFT (18)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_TX_LANE_FLIP_EN_FLDMASK (0x20000)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_TX_LANE_FLIP_EN_FLDSHFT (17)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_RX_LANE_FLIP_EN_FLDMASK (0x10000)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_RX_LANE_FLIP_EN_FLDSHFT (16)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_RSVD1_FLDMASK (0xc000)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_RSVD1_FLDSHFT (14)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_FRS_READY_FLDMASK (0x2000)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_FRS_READY_FLDSHFT (13)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_DRS_READY_FLDMASK (0x1000)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_DRS_READY_FLDSHFT (12)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_CLK_PM_EN_FLDMASK (0x800)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_CLK_PM_EN_FLDSHFT (11)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_XFER_PEND_FLDMASK (0x400)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_XFER_PEND_FLDSHFT (10)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_REQ_EXIT_L1_FLDMASK (0x200)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_REQ_EXIT_L1_FLDSHFT (9)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_READY_ENTRY_L23_FLDMASK (0x100)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_READY_ENTRY_L23_FLDSHFT (8)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_REQ_ENTRY_L1_FLDMASK (0x80)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_REQ_ENTRY_L1_FLDSHFT (7)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_SYS_AUX_PWR_DET_FLDMASK (0x40)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_SYS_AUX_PWR_DET_FLDSHFT (6)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_PM_XMT_PME_FLDMASK (0x20)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_PM_XMT_PME_FLDSHFT (5)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_PERST_N_FLDMASK (0x10)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_PERST_N_FLDSHFT (4)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_CLK_REQ_N_EN_FLDMASK (0x8)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_CLK_REQ_N_EN_FLDSHFT (3)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_INIT_RST_EN_FLDMASK (0x4)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_INIT_RST_EN_FLDSHFT (2)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_REQ_RETRY_EN_FLDMASK (0x2)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_REQ_RETRY_EN_FLDSHFT (1)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_LTSSM_EN_FLDMASK (0x1)
+#define HWIO_PCIE_SS_PCIE_APP_CTRL_PCIE_APP_LTSSM_EN_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_APP_STS_REGOFF 0x14
+#define HWIO_PCIE_SS_PCIE_APP_STS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_APP_STS_REGOFF)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PHY_SRAM_INIT_DONE_FLDMASK (0x80000000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PHY_SRAM_INIT_DONE_FLDSHFT (31)
+#define HWIO_PCIE_SS_PCIE_APP_STS_RSVD0_FLDMASK (0x40000000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_RSVD0_FLDSHFT (30)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_BRDG_SLV_XFER_PEND_FLDMASK (0x20000000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_BRDG_SLV_XFER_PEND_FLDSHFT (29)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_BRDG_DBI_XFER_PEND_FLDMASK (0x10000000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_BRDG_DBI_XFER_PEND_FLDSHFT (28)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_EDMA_XFER_PEND_FLDMASK (0x8000000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_EDMA_XFER_PEND_FLDSHFT (27)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_RADM_XFER_PEND_FLDMASK (0x4000000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_RADM_XFER_PEND_FLDSHFT (26)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_RADM_Q_NOTEMPTY_FLDMASK (0x2000000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_RADM_Q_NOTEMPTY_FLDSHFT (25)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_RADM_QOVRFLOW_FLDMASK (0x1000000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_RADM_QOVRFLOW_FLDSHFT (24)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_STATE_FLDMASK (0xe00000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_STATE_FLDSHFT (21)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_SMLH_LTSSM_STATE_FLDMASK (0x1f8000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_SMLH_LTSSM_STATE_FLDSHFT (15)
+#define HWIO_PCIE_SS_PCIE_APP_STS_RSVD1_FLDMASK (0x4000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_RSVD1_FLDSHFT (14)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_WAKE_EVENT_FLDMASK (0x2000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_WAKE_EVENT_FLDSHFT (13)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_L2_EXIT_FLDMASK (0x1000)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_L2_EXIT_FLDSHFT (12)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_LINKST_IN_L2_FLDMASK (0x800)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_LINKST_IN_L2_FLDSHFT (11)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_LINKST_IN_L1sub_FLDMASK (0x400)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_LINKST_IN_L1sub_FLDSHFT (10)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_LINKST_IN_L1_FLDMASK (0x200)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_LINKST_IN_L1_FLDSHFT (9)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_LINKST_IN_L0s_FLDMASK (0x100)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_LINKST_IN_L0s_FLDSHFT (8)
+#define HWIO_PCIE_SS_PCIE_APP_STS_RSVD2_FLDMASK (0xc0)
+#define HWIO_PCIE_SS_PCIE_APP_STS_RSVD2_FLDSHFT (6)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_RDLH_LINK_UP_FLDMASK (0x20)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_RDLH_LINK_UP_FLDSHFT (5)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_CURR_ST_FLDMASK (0x1c)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_PM_CURR_ST_FLDSHFT (2)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_SMLH_LINK_UP_FLDMASK (0x2)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_SMLH_LINK_UP_FLDSHFT (1)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_RADM_IDLE_FLDMASK (0x1)
+#define HWIO_PCIE_SS_PCIE_APP_STS_PCIE_RADM_IDLE_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_REGOFF 0x18
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_MSI_TRIG_REGOFF)
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_RSVD0_FLDMASK (0xffff0000)
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_RSVD0_FLDSHFT (16)
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_PCIE_VEN_MSI_TC_FLDMASK (0xe000)
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_PCIE_VEN_MSI_TC_FLDSHFT (13)
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_PCIE_VEN_MSI_VEC_FLDMASK (0x1f00)
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_PCIE_VEN_MSI_VEC_FLDSHFT (8)
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_RSVD1_FLDMASK (0xfe)
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_RSVD1_FLDSHFT (1)
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_PCIE_VEN_MSI_REQ_FLDMASK (0x1)
+#define HWIO_PCIE_SS_PCIE_MSI_TRIG_PCIE_VEN_MSI_REQ_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_MSI_PEND_REGOFF 0x1C
+#define HWIO_PCIE_SS_PCIE_MSI_PEND_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_MSI_PEND_REGOFF)
+#define HWIO_PCIE_SS_PCIE_MSI_PEND_PCIE_CFG_MSI_PEND_FLDMASK (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_MSI_PEND_PCIE_CFG_MSI_PEND_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_CURR_LTR_STS_REGOFF 0x20
+#define HWIO_PCIE_SS_PCIE_CURR_LTR_STS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_CURR_LTR_STS_REGOFF)
+#define HWIO_PCIE_SS_PCIE_CURR_LTR_STS_PCIE_CURR_LTR_STS_FLDMASK (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_CURR_LTR_STS_PCIE_CURR_LTR_STS_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_LTR_MSG_LATENCY_REGOFF 0x24
+#define HWIO_PCIE_SS_PCIE_LTR_MSG_LATENCY_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_LTR_MSG_LATENCY_REGOFF)
+#define HWIO_PCIE_SS_PCIE_LTR_MSG_LATENCY_PCIE_LTR_MSG_LATENCY_FLDMASK \
+ (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_LTR_MSG_LATENCY_PCIE_LTR_MSG_LATENCY_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_LTR_MSG_CTRL_REGOFF 0x28
+#define HWIO_PCIE_SS_PCIE_LTR_MSG_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_LTR_MSG_CTRL_REGOFF)
+#define HWIO_PCIE_SS_PCIE_LTR_MSG_CTRL_RSVD0_FLDMASK (0xfffffffe)
+#define HWIO_PCIE_SS_PCIE_LTR_MSG_CTRL_RSVD0_FLDSHFT (1)
+#define HWIO_PCIE_SS_PCIE_LTR_MSG_CTRL_PCIE_LTR_MSG_REQ_FLDMASK (0x1)
+#define HWIO_PCIE_SS_PCIE_LTR_MSG_CTRL_PCIE_LTR_MSG_REQ_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_RX_VMSG0_ID_REGOFF 0x30
+#define HWIO_PCIE_SS_PCIE_RX_VMSG0_ID_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_RX_VMSG0_ID_REGOFF)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG0_ID_RSVD0_FLDMASK (0xfffe0000)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG0_ID_RSVD0_FLDSHFT (17)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG0_ID_PCIE_RADM_MSG0_VALID_FLDMASK (0x10000)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG0_ID_PCIE_RADM_MSG0_VALID_FLDSHFT (16)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG0_ID_PCIE_RADM_MSG0_REQ_ID_FLDMASK (0xffff)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG0_ID_PCIE_RADM_MSG0_REQ_ID_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_RX_MSG0_PYLD_REG1_REGOFF 0x34
+#define HWIO_PCIE_SS_PCIE_RX_MSG0_PYLD_REG1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_RX_MSG0_PYLD_REG1_REGOFF)
+#define HWIO_PCIE_SS_PCIE_RX_MSG0_PYLD_REG1_PCIE_RX_MSG0_PYLD_REG1_FLDMASK \
+ (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_RX_MSG0_PYLD_REG1_PCIE_RX_MSG0_PYLD_REG1_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_RX_MSG0_PYLD_REG0_REGOFF 0x38
+#define HWIO_PCIE_SS_PCIE_RX_MSG0_PYLD_REG0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_RX_MSG0_PYLD_REG0_REGOFF)
+#define HWIO_PCIE_SS_PCIE_RX_MSG0_PYLD_REG0_PCIE_RX_MSG0_PYLD_REG0_FLDMASK \
+ (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_RX_MSG0_PYLD_REG0_PCIE_RX_MSG0_PYLD_REG0_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_RX_VMSG1_ID_REGOFF 0x40
+#define HWIO_PCIE_SS_PCIE_RX_VMSG1_ID_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_RX_VMSG1_ID_REGOFF)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG1_ID_RSVD0_FLDMASK (0xfffe0000)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG1_ID_RSVD0_FLDSHFT (17)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG1_ID_PCIE_RADM_MSG1_VALID_FLDMASK (0x10000)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG1_ID_PCIE_RADM_MSG1_VALID_FLDSHFT (16)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG1_ID_PCIE_RADM_MSG1_REQ_ID_FLDMASK (0xffff)
+#define HWIO_PCIE_SS_PCIE_RX_VMSG1_ID_PCIE_RADM_MSG1_REQ_ID_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_RX_MSG1_PYLD_REG1_REGOFF 0x44
+#define HWIO_PCIE_SS_PCIE_RX_MSG1_PYLD_REG1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_RX_MSG1_PYLD_REG1_REGOFF)
+#define HWIO_PCIE_SS_PCIE_RX_MSG1_PYLD_REG1_PCIE_RX_MSG1_PYLD_REG1_FLDMASK \
+ (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_RX_MSG1_PYLD_REG1_PCIE_RX_MSG1_PYLD_REG1_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_RX_MSG1_PYLD_REG0_REGOFF 0x48
+#define HWIO_PCIE_SS_PCIE_RX_MSG1_PYLD_REG0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_RX_MSG1_PYLD_REG0_REGOFF)
+#define HWIO_PCIE_SS_PCIE_RX_MSG1_PYLD_REG0_PCIE_RX_MSG1_PYLD_REG0_FLDMASK \
+ (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_RX_MSG1_PYLD_REG0_PCIE_RX_MSG1_PYLD_REG0_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_TX_MSG_PYLD_REG1_REGOFF 0x50
+#define HWIO_PCIE_SS_PCIE_TX_MSG_PYLD_REG1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_TX_MSG_PYLD_REG1_REGOFF)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_PYLD_REG1_PCIE_TX_MSG_PYLD_REG1_FLDMASK \
+ (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_PYLD_REG1_PCIE_TX_MSG_PYLD_REG1_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_TX_MSG_PYLD_REG0_REGOFF 0x54
+#define HWIO_PCIE_SS_PCIE_TX_MSG_PYLD_REG0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_TX_MSG_PYLD_REG0_REGOFF)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_PYLD_REG0_PCIE_TX_MSG_PYLD_REG0_FLDMASK \
+ (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_PYLD_REG0_PCIE_TX_MSG_PYLD_REG0_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_REGOFF 0x58
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_REGOFF)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_FMT_FLDMASK (0xc0000000)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_FMT_FLDSHFT (30)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_TYPE_FLDMASK (0x3e000000)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_TYPE_FLDSHFT (25)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_TC_FLDMASK (0x1c00000)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_TC_FLDSHFT (22)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_RSVD0_FLDMASK (0x200000)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_RSVD0_FLDSHFT (21)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_ATTR_FLDMASK (0x180000)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_ATTR_FLDSHFT (19)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_TD_FLDMASK (0x40000)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_TD_FLDSHFT (18)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_EP_FLDMASK (0x20000)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_EP_FLDSHFT (17)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_RSVD1_FLDMASK (0x1fc00)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_RSVD1_FLDSHFT (10)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_LEN_FLDMASK (0x3ff)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_FIELD_PCIE_TX_MSG_LEN_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_TX_MSG_REQ_REGOFF 0x5C
+#define HWIO_PCIE_SS_PCIE_TX_MSG_REQ_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_TX_MSG_REQ_REGOFF)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_REQ_PCIE_TX_MSG_TAG_FLDMASK (0xff000000)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_REQ_PCIE_TX_MSG_TAG_FLDSHFT (24)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_REQ_PCIE_TX_MSG_CODE_FLDMASK (0xff0000)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_REQ_PCIE_TX_MSG_CODE_FLDSHFT (16)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_REQ_RSVD0_FLDMASK (0xfffe)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_REQ_RSVD0_FLDSHFT (1)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_REQ_PCIE_TX_MSG_REQ_FLDMASK (0x1)
+#define HWIO_PCIE_SS_PCIE_TX_MSG_REQ_PCIE_TX_MSG_REQ_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_CORE_DEBUG_REG1_REGOFF 0x60
+#define HWIO_PCIE_SS_PCIE_CORE_DEBUG_REG1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_CORE_DEBUG_REG1_REGOFF)
+#define HWIO_PCIE_SS_PCIE_CORE_DEBUG_REG1_PCIE_CORE_DEBUG_REG1_FLDMASK \
+ (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_CORE_DEBUG_REG1_PCIE_CORE_DEBUG_REG1_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_CORE_DEBUG_REG0_REGOFF 0x64
+#define HWIO_PCIE_SS_PCIE_CORE_DEBUG_REG0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_CORE_DEBUG_REG0_REGOFF)
+#define HWIO_PCIE_SS_PCIE_CORE_DEBUG_REG0_PCIE_CORE_DEBUG_REG0_FLDMASK \
+ (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_CORE_DEBUG_REG0_PCIE_CORE_DEBUG_REG0_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_CORE_EIDEBUG_REG0_REGOFF 0x6C
+#define HWIO_PCIE_SS_PCIE_CORE_EIDEBUG_REG0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_PCIE_SS_PCIE_CORE_EIDEBUG_REG0_REGOFF)
+#define HWIO_PCIE_SS_PCIE_CORE_EIDEBUG_REG0_RSVD0_FLDMASK (0xfffc0000)
+#define HWIO_PCIE_SS_PCIE_CORE_EIDEBUG_REG0_RSVD0_FLDSHFT (18)
+#define HWIO_PCIE_SS_PCIE_CORE_EIDEBUG_REG0_PCIE_DEBUG_MUX_SEL_FLDMASK (0x30000)
+#define HWIO_PCIE_SS_PCIE_CORE_EIDEBUG_REG0_PCIE_DEBUG_MUX_SEL_FLDSHFT (16)
+#define HWIO_PCIE_SS_PCIE_CORE_EIDEBUG_REG0_PCIE_CORE_EIDEBUG_REG0_FLDMASK \
+ (0xffff)
+#define HWIO_PCIE_SS_PCIE_CORE_EIDEBUG_REG0_PCIE_CORE_EIDEBUG_REG0_FLDSHFT (0)
+
+#define HWIO_PCIE_SS_PCIE_GP_REGNUM 8
+#define HWIO_PCIE_SS_PCIE_GP_REGOFF 0x080
+#define HWIO_PCIE_SS_PCIE_GP_ADDR(bAddr, regX) \
+ (((regX >= 0) && (regX < HWIO_PCIE_SS_PCIE_GP_REGNUM)) ? \
+ (bAddr + HWIO_PCIE_SS_PCIE_GP_REGOFF + (regX * 4)) : MNH_BAD_ADDR)
+#define HWIO_PCIE_SS_PCIE_GP_PCIE_GP_FLDMASK (0xffffffff)
+#define HWIO_PCIE_SS_PCIE_GP_PCIE_GP_FLDSHFT (0)
+
+#endif /* __MNH_HWIO_PCIE_SS_ */
diff --git a/drivers/misc/mnh/mnh-hwio-scu.h b/drivers/misc/mnh/mnh-hwio-scu.h
new file mode 100644
index 0000000..965c450
--- /dev/null
+++ b/drivers/misc/mnh/mnh-hwio-scu.h
@@ -0,0 +1,988 @@
+/* manually add PCIE_POWER register details. Using GP 1 register */
+/* manual rework for FSP indexing */
+/* manual rework for EFUSE addressing and hwstrap */
+/* auto generated: Monday, August 15th, 2016 12:26:48pm */
+/*
+ * Copyright (c) 2016, Intel Corporation. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this
+ * list of conditions and the following disclaimer.
+ *
+ * Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * Neither the name of Intel nor the names of its contributors may be used
+ * to endorse or promote products derived from this software without specific
+ * prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef __MNH_HWIO_SCU_
+#define __MNH_HWIO_SCU_
+
+#define HWIO_SCU_ID_REGOFF 0x0
+#define HWIO_SCU_ID_ADDR(bAddr, regX) (bAddr + HWIO_SCU_ID_REGOFF)
+#define HWIO_SCU_ID_RSVD0_FLDMASK (0xff000000)
+#define HWIO_SCU_ID_RSVD0_FLDSHFT (24)
+#define HWIO_SCU_ID_MAJOR_REV_ID_FLDMASK (0xff0000)
+#define HWIO_SCU_ID_MAJOR_REV_ID_FLDSHFT (16)
+#define HWIO_SCU_ID_RSVD1_FLDMASK (0xff00)
+#define HWIO_SCU_ID_RSVD1_FLDSHFT (8)
+#define HWIO_SCU_ID_MINOR_REV_ID_FLDMASK (0xff)
+#define HWIO_SCU_ID_MINOR_REV_ID_FLDSHFT (0)
+
+#define HWIO_SCU_CACHE_SIZE_REGOFF 0x4
+#define HWIO_SCU_CACHE_SIZE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_CACHE_SIZE_REGOFF)
+#define HWIO_SCU_CACHE_SIZE_CPU_L2_CAHCE_SIZE_FLDMASK (0xffff0000)
+#define HWIO_SCU_CACHE_SIZE_CPU_L2_CAHCE_SIZE_FLDSHFT (16)
+#define HWIO_SCU_CACHE_SIZE_CPU_L1_DATA_CAHCE_SIZE_FLDMASK (0xff00)
+#define HWIO_SCU_CACHE_SIZE_CPU_L1_DATA_CAHCE_SIZE_FLDSHFT (8)
+#define HWIO_SCU_CACHE_SIZE_CPU_L1_CODE_CAHCE_SIZE_FLDMASK (0xff)
+#define HWIO_SCU_CACHE_SIZE_CPU_L1_CODE_CAHCE_SIZE_FLDSHFT (0)
+
+#define HWIO_SCU_LPDDR4_MEM_SIZE_REGOFF 0x8
+#define HWIO_SCU_LPDDR4_MEM_SIZE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_LPDDR4_MEM_SIZE_REGOFF)
+#define HWIO_SCU_LPDDR4_MEM_SIZE_LPDDR4_MEM_SIZE_FLDMASK (0xffffffff)
+#define HWIO_SCU_LPDDR4_MEM_SIZE_LPDDR4_MEM_SIZE_FLDSHFT (0)
+
+#define HWIO_SCU_HW_STRAP_REGOFF 0x0C
+#define HWIO_SCU_HW_STRAP_ADDR(bAddr, regX) (bAddr + HWIO_SCU_HW_STRAP_REGOFF)
+#define HWIO_SCU_HW_STRAP_RSVD_FLDMASK (0xffffff00)
+#define HWIO_SCU_HW_STRAP_RSVD_FLDSHFT (8)
+#define HWIO_SCU_HW_STRAP_PCIE_LANE_FLDMASK (0x80)
+#define HWIO_SCU_HW_STRAP_PCIE_LANE_FLDSHFT (7)
+#define HWIO_SCU_HW_STRAP_SPIS_MODE_FLDMASK (0x40)
+#define HWIO_SCU_HW_STRAP_SPIS_MODE_FLDSHFT (6)
+#define HWIO_SCU_HW_STRAP_BOOT_DEBUG_FLDMASK (0x20)
+#define HWIO_SCU_HW_STRAP_BOOT_DEBUG_FLDSHFT (5)
+#define HWIO_SCU_HW_STRAP_BIT4_FLDMASK (0x10)
+#define HWIO_SCU_HW_STRAP_BIT4_FLDSHFT (4)
+#define HWIO_SCU_HW_STRAP_REF_CLK_SEL_FLDMASK (0x08)
+#define HWIO_SCU_HW_STRAP_REF_CLK_SEL_FLDSHFT (3)
+#define HWIO_SCU_HW_STRAP_PCIE_CLK_SEL_FLDMASK (0x04)
+#define HWIO_SCU_HW_STRAP_PCIE_CLK_SEL_FLDSHFT (2)
+#define HWIO_SCU_HW_STRAP_READY_FLDMASK (0x02)
+#define HWIO_SCU_HW_STRAP_READY_FLDSHFT (1)
+#define HWIO_SCU_HW_STRAP_BOOT_MODE_FLDMASK (0x01)
+#define HWIO_SCU_HW_STRAP_BOOT_MODE_FLDSHFT (0)
+
+/* General Purpose Sticky */
+#define HWIO_SCU_GPS_REGNUM 4
+#define HWIO_SCU_GPS_REGOFF 0x10
+#define HWIO_SCU_GPS_ADDR(bAddr, regX) \
+ (((regX >= 0) && (regX < HWIO_SCU_GPS_REGNUM)) ? \
+ (bAddr + HWIO_SCU_GPS_REGOFF + (regX * 4)) : 0)
+#define HWIO_SCU_GPS_GPS_FLDMASK (0xffffffff)
+#define HWIO_SCU_GPS_GPS_FLDSHFT (0)
+
+/* General Purpose Scratchpad */
+/*
+ * Allocation:
+ * 0 = pcie init done (from AP)
+ * 1 = Power mode register (from Easel)
+ * 2 = boot args mask (from AP)
+ * 3 = debug boot trace low (from Easel)
+*/
+#define HWIO_SCU_GP_REGNUM 4
+#define HWIO_SCU_GP_REGOFF 0x20
+#define HWIO_SCU_GP_ADDR(bAddr, regX) \
+ (((regX >= 0) && (regX < HWIO_SCU_GP_REGNUM)) ? \
+ (bAddr + HWIO_SCU_GP_REGOFF + (regX * 4)) : 0)
+#define HWIO_SCU_GP_GP_FLDMASK (0xffffffff)
+#define HWIO_SCU_GP_GP_FLDSHFT (0)
+
+#define HWIO_SCU_GP_POWER_MODE_REGOFF 0x24
+#define HWIO_SCU_GP_POWER_MODE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_GP_POWER_MODE_REGOFF)
+#define HWIO_SCU_GP_POWER_MODE_PCIE_L1_2_EN_FLDMASK (0x2)
+#define HWIO_SCU_GP_POWER_MODE_PCIE_L1_2_EN_FLDSHFT (1)
+#define HWIO_SCU_GP_POWER_MODE_PCIE_CLKPM_EN_FLDMASK (0x1)
+#define HWIO_SCU_GP_POWER_MODE_PCIE_CLKPM_EN_FLDSHFT (0)
+
+#define HWIO_SCU_GP_BOOT_ARGS_REGOFF 0x28
+#define HWIO_SCU_GP_BOOT_ARGS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_GP_BOOT_ARGS_REGOFF)
+
+#define MNH_BOOT_TRACE (HWIO_SCU_GP_ADDR(HWIO_SCU_BASE_ADDR, 3))
+
+#define HWIO_SCU_GLOBAL_IRQ_STATUS_SET1_REGOFF 0x38
+#define HWIO_SCU_GLOBAL_IRQ_STATUS_SET1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_GLOBAL_IRQ_STATUS_SET1_REGOFF)
+#define HWIO_SCU_GLOBAL_IRQ_STATUS_SET1_GLOBAL_IRQ_STATUS_SET1_FLDMASK \
+ (0xffffffff)
+#define HWIO_SCU_GLOBAL_IRQ_STATUS_SET1_GLOBAL_IRQ_STATUS_SET1_FLDSHFT (0)
+
+#define HWIO_SCU_GLOBAL_IRQ_STATUS_SET0_REGOFF 0x3C
+#define HWIO_SCU_GLOBAL_IRQ_STATUS_SET0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_GLOBAL_IRQ_STATUS_SET0_REGOFF)
+#define HWIO_SCU_GLOBAL_IRQ_STATUS_SET0_GLOBAL_IRQ_STATUS_SET0_FLDMASK \
+ (0xffffffff)
+#define HWIO_SCU_GLOBAL_IRQ_STATUS_SET0_GLOBAL_IRQ_STATUS_SET0_FLDSHFT (0)
+
+#define HWIO_SCU_GLOBAL_IRQ_HLT_RST_EN_SET1_REGOFF 0x48
+#define HWIO_SCU_GLOBAL_IRQ_HLT_RST_EN_SET1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_GLOBAL_IRQ_HLT_RST_EN_SET1_REGOFF)
+#define HWIO_SCU_GLOBAL_IRQ_HLT_RST_EN_SET1_GLOBAL_IRQ_HLT_RST_EN_SET1_FLDMASK \
+ (0xffffffff)
+#define HWIO_SCU_GLOBAL_IRQ_HLT_RST_EN_SET1_GLOBAL_IRQ_HLT_RST_EN_SET1_FLDSHFT \
+ (0)
+
+#define HWIO_SCU_GLOBAL_IRQ_HLT_RST_EN_SET0_REGOFF 0x4C
+#define HWIO_SCU_GLOBAL_IRQ_HLT_RST_EN_SET0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_GLOBAL_IRQ_HLT_RST_EN_SET0_REGOFF)
+#define HWIO_SCU_GLOBAL_IRQ_HLT_RST_EN_SET0_GLOBAL_IRQ_HLT_RST_EN_SET0_FLDMASK \
+ (0xffffffff)
+#define HWIO_SCU_GLOBAL_IRQ_HLT_RST_EN_SET0_GLOBAL_IRQ_HLT_RST_EN_SET0_FLDSHFT \
+ (0)
+
+#define HWIO_SCU_GLOBAL_WAKE_EN_SET1_REGOFF 0x58
+#define HWIO_SCU_GLOBAL_WAKE_EN_SET1_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_GLOBAL_WAKE_EN_SET1_REGOFF)
+#define HWIO_SCU_GLOBAL_WAKE_EN_SET1_GLOBAL_WAKE_EN_SET1_FLDMASK (0xffffffff)
+#define HWIO_SCU_GLOBAL_WAKE_EN_SET1_GLOBAL_WAKE_EN_SET1_FLDSHFT (0)
+
+#define HWIO_SCU_GLOBAL_WAKE_EN_SET0_REGOFF 0x5C
+#define HWIO_SCU_GLOBAL_WAKE_EN_SET0_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_GLOBAL_WAKE_EN_SET0_REGOFF)
+#define HWIO_SCU_GLOBAL_WAKE_EN_SET0_GLOBAL_WAKE_EN_SET0_FLDMASK (0xffffffff)
+#define HWIO_SCU_GLOBAL_WAKE_EN_SET0_GLOBAL_WAKE_EN_SET0_FLDSHFT (0)
+
+#define HWIO_SCU_SCU_IRQ_STATUS_REGOFF 0x70
+#define HWIO_SCU_SCU_IRQ_STATUS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_SCU_IRQ_STATUS_REGOFF)
+#define HWIO_SCU_SCU_IRQ_STATUS_RSVD0_FLDMASK (0xffffff80)
+#define HWIO_SCU_SCU_IRQ_STATUS_RSVD0_FLDSHFT (7)
+#define HWIO_SCU_SCU_IRQ_STATUS_LP4_FREQ_CHG_REQ_FLDMASK (0x40)
+#define HWIO_SCU_SCU_IRQ_STATUS_LP4_FREQ_CHG_REQ_FLDSHFT (6)
+#define HWIO_SCU_SCU_IRQ_STATUS_LP4_LPC_CMD_DONE_FLDMASK (0x20)
+#define HWIO_SCU_SCU_IRQ_STATUS_LP4_LPC_CMD_DONE_FLDSHFT (5)
+#define HWIO_SCU_SCU_IRQ_STATUS_PVT_SENS4_DV_FLDMASK (0x10)
+#define HWIO_SCU_SCU_IRQ_STATUS_PVT_SENS4_DV_FLDSHFT (4)
+#define HWIO_SCU_SCU_IRQ_STATUS_PVT_SENS3_DV_FLDMASK (0x8)
+#define HWIO_SCU_SCU_IRQ_STATUS_PVT_SENS3_DV_FLDSHFT (3)
+#define HWIO_SCU_SCU_IRQ_STATUS_PVT_SENS2_DV_FLDMASK (0x4)
+#define HWIO_SCU_SCU_IRQ_STATUS_PVT_SENS2_DV_FLDSHFT (2)
+#define HWIO_SCU_SCU_IRQ_STATUS_PVT_SENS1_DV_FLDMASK (0x2)
+#define HWIO_SCU_SCU_IRQ_STATUS_PVT_SENS1_DV_FLDSHFT (1)
+#define HWIO_SCU_SCU_IRQ_STATUS_PVT_SENS0_DV_FLDMASK (0x1)
+#define HWIO_SCU_SCU_IRQ_STATUS_PVT_SENS0_DV_FLDSHFT (0)
+
+#define HWIO_SCU_SCU_IRQ_ENABLE_REGOFF 0x74
+#define HWIO_SCU_SCU_IRQ_ENABLE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_SCU_IRQ_ENABLE_REGOFF)
+#define HWIO_SCU_SCU_IRQ_ENABLE_RSVD0_FLDMASK (0xffffff80)
+#define HWIO_SCU_SCU_IRQ_ENABLE_RSVD0_FLDSHFT (7)
+#define HWIO_SCU_SCU_IRQ_ENABLE_LP4_FREQ_CHG_REQ_IE_FLDMASK (0x40)
+#define HWIO_SCU_SCU_IRQ_ENABLE_LP4_FREQ_CHG_REQ_IE_FLDSHFT (6)
+#define HWIO_SCU_SCU_IRQ_ENABLE_LP4_LPC_CMD_DONE_IE_FLDMASK (0x20)
+#define HWIO_SCU_SCU_IRQ_ENABLE_LP4_LPC_CMD_DONE_IE_FLDSHFT (5)
+#define HWIO_SCU_SCU_IRQ_ENABLE_PVT_SENS4_IE_FLDMASK (0x10)
+#define HWIO_SCU_SCU_IRQ_ENABLE_PVT_SENS4_IE_FLDSHFT (4)
+#define HWIO_SCU_SCU_IRQ_ENABLE_PVT_SENS3_IE_FLDMASK (0x8)
+#define HWIO_SCU_SCU_IRQ_ENABLE_PVT_SENS3_IE_FLDSHFT (3)
+#define HWIO_SCU_SCU_IRQ_ENABLE_PVT_SENS2_IE_FLDMASK (0x4)
+#define HWIO_SCU_SCU_IRQ_ENABLE_PVT_SENS2_IE_FLDSHFT (2)
+#define HWIO_SCU_SCU_IRQ_ENABLE_PVT_SENS1_IE_FLDMASK (0x2)
+#define HWIO_SCU_SCU_IRQ_ENABLE_PVT_SENS1_IE_FLDSHFT (1)
+#define HWIO_SCU_SCU_IRQ_ENABLE_PVT_SENS0_IE_FLDMASK (0x1)
+#define HWIO_SCU_SCU_IRQ_ENABLE_PVT_SENS0_IE_FLDSHFT (0)
+
+#define HWIO_SCU_RSTC_REGOFF 0x78
+#define HWIO_SCU_RSTC_ADDR(bAddr, regX) (bAddr + HWIO_SCU_RSTC_REGOFF)
+#define HWIO_SCU_RSTC_INTR_HALT_RST_SEL_FLDMASK (0x80000000)
+#define HWIO_SCU_RSTC_INTR_HALT_RST_SEL_FLDSHFT (31)
+#define HWIO_SCU_RSTC_RSVD0_FLDMASK (0x70000000)
+#define HWIO_SCU_RSTC_RSVD0_FLDSHFT (28)
+#define HWIO_SCU_RSTC_I2C3_RST_FLDMASK (0x8000000)
+#define HWIO_SCU_RSTC_I2C3_RST_FLDSHFT (27)
+#define HWIO_SCU_RSTC_I2C2_RST_FLDMASK (0x4000000)
+#define HWIO_SCU_RSTC_I2C2_RST_FLDSHFT (26)
+#define HWIO_SCU_RSTC_I2C1_RST_FLDMASK (0x2000000)
+#define HWIO_SCU_RSTC_I2C1_RST_FLDSHFT (25)
+#define HWIO_SCU_RSTC_I2C0_RST_FLDMASK (0x1000000)
+#define HWIO_SCU_RSTC_I2C0_RST_FLDSHFT (24)
+#define HWIO_SCU_RSTC_UART1_RST_FLDMASK (0x800000)
+#define HWIO_SCU_RSTC_UART1_RST_FLDSHFT (23)
+#define HWIO_SCU_RSTC_UART0_RST_FLDMASK (0x400000)
+#define HWIO_SCU_RSTC_UART0_RST_FLDSHFT (22)
+#define HWIO_SCU_RSTC_SPIS_RST_FLDMASK (0x200000)
+#define HWIO_SCU_RSTC_SPIS_RST_FLDSHFT (21)
+#define HWIO_SCU_RSTC_SPIM_RST_FLDMASK (0x100000)
+#define HWIO_SCU_RSTC_SPIM_RST_FLDSHFT (20)
+#define HWIO_SCU_RSTC_GPIO_RST_FLDMASK (0x80000)
+#define HWIO_SCU_RSTC_GPIO_RST_FLDSHFT (19)
+#define HWIO_SCU_RSTC_PERI_DMA_RST_FLDMASK (0x40000)
+#define HWIO_SCU_RSTC_PERI_DMA_RST_FLDSHFT (18)
+#define HWIO_SCU_RSTC_TIMER_RST_FLDMASK (0x20000)
+#define HWIO_SCU_RSTC_TIMER_RST_FLDSHFT (17)
+#define HWIO_SCU_RSTC_WDT_RST_FLDMASK (0x10000)
+#define HWIO_SCU_RSTC_WDT_RST_FLDSHFT (16)
+#define HWIO_SCU_RSTC_RSVD1_FLDMASK (0xe000)
+#define HWIO_SCU_RSTC_RSVD1_FLDSHFT (13)
+#define HWIO_SCU_RSTC_PMON_RST_FLDMASK (0x1000)
+#define HWIO_SCU_RSTC_PMON_RST_FLDSHFT (12)
+#define HWIO_SCU_RSTC_MIPIRXPHY_RST_FLDMASK (0x800)
+#define HWIO_SCU_RSTC_MIPIRXPHY_RST_FLDSHFT (11)
+#define HWIO_SCU_RSTC_MIPITXPHY_RST_FLDMASK (0x400)
+#define HWIO_SCU_RSTC_MIPITXPHY_RST_FLDSHFT (10)
+#define HWIO_SCU_RSTC_MIPIRXCTL_RST_FLDMASK (0x200)
+#define HWIO_SCU_RSTC_MIPIRXCTL_RST_FLDSHFT (9)
+#define HWIO_SCU_RSTC_MIPITXCTL_RST_FLDMASK (0x100)
+#define HWIO_SCU_RSTC_MIPITXCTL_RST_FLDSHFT (8)
+#define HWIO_SCU_RSTC_PCIE_PHY_RST_FLDMASK (0x80)
+#define HWIO_SCU_RSTC_PCIE_PHY_RST_FLDSHFT (7)
+#define HWIO_SCU_RSTC_PCIE_CTL_RST_FLDMASK (0x40)
+#define HWIO_SCU_RSTC_PCIE_CTL_RST_FLDSHFT (6)
+#define HWIO_SCU_RSTC_RSVD2_FLDMASK (0x20)
+#define HWIO_SCU_RSTC_RSVD2_FLDSHFT (5)
+#define HWIO_SCU_RSTC_LP4PHY_RST_FLDMASK (0x10)
+#define HWIO_SCU_RSTC_LP4PHY_RST_FLDSHFT (4)
+#define HWIO_SCU_RSTC_LP4CTL_RST_FLDMASK (0x8)
+#define HWIO_SCU_RSTC_LP4CTL_RST_FLDSHFT (3)
+#define HWIO_SCU_RSTC_IPU_RST_FLDMASK (0x4)
+#define HWIO_SCU_RSTC_IPU_RST_FLDSHFT (2)
+#define HWIO_SCU_RSTC_CPU_RST_FLDMASK (0x2)
+#define HWIO_SCU_RSTC_CPU_RST_FLDSHFT (1)
+#define HWIO_SCU_RSTC_SW_COLDRST_FLDMASK (0x1)
+#define HWIO_SCU_RSTC_SW_COLDRST_FLDSHFT (0)
+
+#define HWIO_SCU_RSTS_REGOFF 0x7C
+#define HWIO_SCU_RSTS_ADDR(bAddr, regX) (bAddr + HWIO_SCU_RSTS_REGOFF)
+#define HWIO_SCU_RSTS_RSVD0_FLDMASK (0xfffffc00)
+#define HWIO_SCU_RSTS_RSVD0_FLDSHFT (10)
+#define HWIO_SCU_RSTS_PVT3_ALARM_RESET_FLDMASK (0x200)
+#define HWIO_SCU_RSTS_PVT3_ALARM_RESET_FLDSHFT (9)
+#define HWIO_SCU_RSTS_PVT2_ALARM_RESET_FLDMASK (0x100)
+#define HWIO_SCU_RSTS_PVT2_ALARM_RESET_FLDSHFT (8)
+#define HWIO_SCU_RSTS_PVT1_ALARM_RESET_FLDMASK (0x80)
+#define HWIO_SCU_RSTS_PVT1_ALARM_RESET_FLDSHFT (7)
+#define HWIO_SCU_RSTS_PVT0_ALARM_RESET_FLDMASK (0x40)
+#define HWIO_SCU_RSTS_PVT0_ALARM_RESET_FLDSHFT (6)
+#define HWIO_SCU_RSTS_CPU_RST_FLDMASK (0x20)
+#define HWIO_SCU_RSTS_CPU_RST_FLDSHFT (5)
+#define HWIO_SCU_RSTS_INTR_RST_FLDMASK (0x10)
+#define HWIO_SCU_RSTS_INTR_RST_FLDSHFT (4)
+#define HWIO_SCU_RSTS_POR_RST_FLDMASK (0x8)
+#define HWIO_SCU_RSTS_POR_RST_FLDSHFT (3)
+#define HWIO_SCU_RSTS_WDT_WRST_FLDMASK (0x4)
+#define HWIO_SCU_RSTS_WDT_WRST_FLDSHFT (2)
+#define HWIO_SCU_RSTS_EXT_RST_FLDMASK (0x2)
+#define HWIO_SCU_RSTS_EXT_RST_FLDSHFT (1)
+#define HWIO_SCU_RSTS_SW_COLDRST_FLDMASK (0x1)
+#define HWIO_SCU_RSTS_SW_COLDRST_FLDSHFT (0)
+
+#define HWIO_SCU_CCU_CLK_DIV_REGOFF 0x80
+#define HWIO_SCU_CCU_CLK_DIV_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_CCU_CLK_DIV_REGOFF)
+#define HWIO_SCU_CCU_CLK_DIV_PVT_SENSOR_CLK_DIV_FLDMASK (0xfe000000)
+#define HWIO_SCU_CCU_CLK_DIV_PVT_SENSOR_CLK_DIV_FLDSHFT (25)
+#define HWIO_SCU_CCU_CLK_DIV_PCIE_AUX_CLK_DIV_FLDMASK (0x1f00000)
+#define HWIO_SCU_CCU_CLK_DIV_PCIE_AUX_CLK_DIV_FLDSHFT (20)
+#define HWIO_SCU_CCU_CLK_DIV_PCIE_AXI_CLK_DIV_FLDMASK (0xf0000)
+#define HWIO_SCU_CCU_CLK_DIV_PCIE_AXI_CLK_DIV_FLDSHFT (16)
+#define HWIO_SCU_CCU_CLK_DIV_AXI_FABRIC_CLK_DIV_FLDMASK (0xf000)
+#define HWIO_SCU_CCU_CLK_DIV_AXI_FABRIC_CLK_DIV_FLDSHFT (12)
+#define HWIO_SCU_CCU_CLK_DIV_LPDDR4_REFCLK_DIV_FLDMASK (0xf00)
+#define HWIO_SCU_CCU_CLK_DIV_LPDDR4_REFCLK_DIV_FLDSHFT (8)
+#define HWIO_SCU_CCU_CLK_DIV_IPU_CLK_DIV_FLDMASK (0xf0)
+#define HWIO_SCU_CCU_CLK_DIV_IPU_CLK_DIV_FLDSHFT (4)
+#define HWIO_SCU_CCU_CLK_DIV_CPU_CLK_DIV_FLDMASK (0xf)
+#define HWIO_SCU_CCU_CLK_DIV_CPU_CLK_DIV_FLDSHFT (0)
+
+#define HWIO_SCU_CCU_CLK_CTL_REGOFF 0x84
+#define HWIO_SCU_CCU_CLK_CTL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_CCU_CLK_CTL_REGOFF)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_CPUCG_EN_FLDMASK (0x80000000)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_CPUCG_EN_FLDSHFT (31)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_LP4CG_EN_FLDMASK (0x40000000)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_LP4CG_EN_FLDSHFT (30)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_BTROMCG_EN_FLDMASK (0x20000000)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_BTROMCG_EN_FLDSHFT (29)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_BTSRAMCG_EN_FLDMASK (0x10000000)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_BTSRAMCG_EN_FLDSHFT (28)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_AXICG_EN_FLDMASK (0x8000000)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_AXICG_EN_FLDSHFT (27)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_AHBCG_EN_FLDMASK (0x4000000)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_AHBCG_EN_FLDSHFT (26)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_LP4_PLL_BYPCLK_CG_EN_FLDMASK (0x2000000)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_LP4_PLL_BYPCLK_CG_EN_FLDSHFT (25)
+#define HWIO_SCU_CCU_CLK_CTL_LP4PHY_PLL_BYPASS_CLKEN_FLDMASK (0x1000000)
+#define HWIO_SCU_CCU_CLK_CTL_LP4PHY_PLL_BYPASS_CLKEN_FLDSHFT (24)
+#define HWIO_SCU_CCU_CLK_CTL_CPU_CLKEN_FLDMASK (0x800000)
+#define HWIO_SCU_CCU_CLK_CTL_CPU_CLKEN_FLDSHFT (23)
+#define HWIO_SCU_CCU_CLK_CTL_LP4_REFCLKEN_FLDMASK (0x400000)
+#define HWIO_SCU_CCU_CLK_CTL_LP4_REFCLKEN_FLDSHFT (22)
+#define HWIO_SCU_CCU_CLK_CTL_BTROM_CLKEN_FLDMASK (0x200000)
+#define HWIO_SCU_CCU_CLK_CTL_BTROM_CLKEN_FLDSHFT (21)
+#define HWIO_SCU_CCU_CLK_CTL_BTSRAM_CLKEN_FLDMASK (0x100000)
+#define HWIO_SCU_CCU_CLK_CTL_BTSRAM_CLKEN_FLDSHFT (20)
+#define HWIO_SCU_CCU_CLK_CTL_AXI_CLKEN_FLDMASK (0x80000)
+#define HWIO_SCU_CCU_CLK_CTL_AXI_CLKEN_FLDSHFT (19)
+#define HWIO_SCU_CCU_CLK_CTL_AHB_CLKEN_FLDMASK (0x40000)
+#define HWIO_SCU_CCU_CLK_CTL_AHB_CLKEN_FLDSHFT (18)
+#define HWIO_SCU_CCU_CLK_CTL_PCIE_AXI_CLKEN_FLDMASK (0x20000)
+#define HWIO_SCU_CCU_CLK_CTL_PCIE_AXI_CLKEN_FLDSHFT (17)
+#define HWIO_SCU_CCU_CLK_CTL_PCIE_AUX_CLKEN_FLDMASK (0x10000)
+#define HWIO_SCU_CCU_CLK_CTL_PCIE_AUX_CLKEN_FLDSHFT (16)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_CLKGATE_CNT_FLDMASK (0xf000)
+#define HWIO_SCU_CCU_CLK_CTL_HALT_CLKGATE_CNT_FLDSHFT (12)
+#define HWIO_SCU_CCU_CLK_CTL_WAKE_CLKEN_CNT_FLDMASK (0xf00)
+#define HWIO_SCU_CCU_CLK_CTL_WAKE_CLKEN_CNT_FLDSHFT (8)
+#define HWIO_SCU_CCU_CLK_CTL_PMON_CLKEN_FLDMASK (0x80)
+#define HWIO_SCU_CCU_CLK_CTL_PMON_CLKEN_FLDSHFT (7)
+#define HWIO_SCU_CCU_CLK_CTL_MIPI_REFCLKEN_FLDMASK (0x40)
+#define HWIO_SCU_CCU_CLK_CTL_MIPI_REFCLKEN_FLDSHFT (6)
+#define HWIO_SCU_CCU_CLK_CTL_IPU_CLKEN_FLDMASK (0x20)
+#define HWIO_SCU_CCU_CLK_CTL_IPU_CLKEN_FLDSHFT (5)
+#define HWIO_SCU_CCU_CLK_CTL_MIPI_TESTCLKEN_FLDMASK (0x10)
+#define HWIO_SCU_CCU_CLK_CTL_MIPI_TESTCLKEN_FLDSHFT (4)
+#define HWIO_SCU_CCU_CLK_CTL_RSVD0_FLDMASK (0x8)
+#define HWIO_SCU_CCU_CLK_CTL_RSVD0_FLDSHFT (3)
+#define HWIO_SCU_CCU_CLK_CTL_IPU_CLK_SRC_FLDMASK (0x4)
+#define HWIO_SCU_CCU_CLK_CTL_IPU_CLK_SRC_FLDSHFT (2)
+#define HWIO_SCU_CCU_CLK_CTL_CPU_IPU_SYS200_MODE_FLDMASK (0x2)
+#define HWIO_SCU_CCU_CLK_CTL_CPU_IPU_SYS200_MODE_FLDSHFT (1)
+#define HWIO_SCU_CCU_CLK_CTL_LP4_AXI_SYS200_MODE_FLDMASK (0x1)
+#define HWIO_SCU_CCU_CLK_CTL_LP4_AXI_SYS200_MODE_FLDSHFT (0)
+
+#define HWIO_SCU_PERIPH_CLK_CTRL_REGOFF 0x88
+#define HWIO_SCU_PERIPH_CLK_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_PERIPH_CLK_CTRL_REGOFF)
+#define HWIO_SCU_PERIPH_CLK_CTRL_PCIE_CLK_MODE_FLDMASK (0x80000000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_PCIE_CLK_MODE_FLDSHFT (31)
+#define HWIO_SCU_PERIPH_CLK_CTRL_RSVD0_FLDMASK (0x70000000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_RSVD0_FLDSHFT (28)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C3_CLK_MODE_FLDMASK (0x8000000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C3_CLK_MODE_FLDSHFT (27)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C2_CLK_MODE_FLDMASK (0x4000000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C2_CLK_MODE_FLDSHFT (26)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C1_CLK_MODE_FLDMASK (0x2000000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C1_CLK_MODE_FLDSHFT (25)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C0_CLK_MODE_FLDMASK (0x1000000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C0_CLK_MODE_FLDSHFT (24)
+#define HWIO_SCU_PERIPH_CLK_CTRL_UART1_CLK_MODE_FLDMASK (0x800000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_UART1_CLK_MODE_FLDSHFT (23)
+#define HWIO_SCU_PERIPH_CLK_CTRL_UART0_CLK_MODE_FLDMASK (0x400000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_UART0_CLK_MODE_FLDSHFT (22)
+#define HWIO_SCU_PERIPH_CLK_CTRL_SPIS_CLK_MODE_FLDMASK (0x200000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_SPIS_CLK_MODE_FLDSHFT (21)
+#define HWIO_SCU_PERIPH_CLK_CTRL_SPIM_CLK_MODE_FLDMASK (0x100000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_SPIM_CLK_MODE_FLDSHFT (20)
+#define HWIO_SCU_PERIPH_CLK_CTRL_RSVD1_FLDMASK (0xf0000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_RSVD1_FLDSHFT (16)
+#define HWIO_SCU_PERIPH_CLK_CTRL_PCIE_REFCLKEN_FLDMASK (0x8000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_PCIE_REFCLKEN_FLDSHFT (15)
+#define HWIO_SCU_PERIPH_CLK_CTRL_PVT_CLKEN_FLDMASK (0x4000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_PVT_CLKEN_FLDSHFT (14)
+#define HWIO_SCU_PERIPH_CLK_CTRL_PERIPH_CLKEN_FLDMASK (0x2000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_PERIPH_CLKEN_FLDSHFT (13)
+#define HWIO_SCU_PERIPH_CLK_CTRL_WDT_SPEED_UP_FLDMASK (0x1000)
+#define HWIO_SCU_PERIPH_CLK_CTRL_WDT_SPEED_UP_FLDSHFT (12)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C3_CLKEN_SW_FLDMASK (0x800)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C3_CLKEN_SW_FLDSHFT (11)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C2_CLKEN_SW_FLDMASK (0x400)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C2_CLKEN_SW_FLDSHFT (10)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C1_CLKEN_SW_FLDMASK (0x200)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C1_CLKEN_SW_FLDSHFT (9)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C0_CLKEN_SW_FLDMASK (0x100)
+#define HWIO_SCU_PERIPH_CLK_CTRL_I2C0_CLKEN_SW_FLDSHFT (8)
+#define HWIO_SCU_PERIPH_CLK_CTRL_UART1_CLKEN_SW_FLDMASK (0x80)
+#define HWIO_SCU_PERIPH_CLK_CTRL_UART1_CLKEN_SW_FLDSHFT (7)
+#define HWIO_SCU_PERIPH_CLK_CTRL_UART0_CLKEN_SW_FLDMASK (0x40)
+#define HWIO_SCU_PERIPH_CLK_CTRL_UART0_CLKEN_SW_FLDSHFT (6)
+#define HWIO_SCU_PERIPH_CLK_CTRL_SPIS_CLKEN_SW_FLDMASK (0x20)
+#define HWIO_SCU_PERIPH_CLK_CTRL_SPIS_CLKEN_SW_FLDSHFT (5)
+#define HWIO_SCU_PERIPH_CLK_CTRL_SPIM_CLKEN_SW_FLDMASK (0x10)
+#define HWIO_SCU_PERIPH_CLK_CTRL_SPIM_CLKEN_SW_FLDSHFT (4)
+#define HWIO_SCU_PERIPH_CLK_CTRL_GPIO_CLKEN_SW_FLDMASK (0x8)
+#define HWIO_SCU_PERIPH_CLK_CTRL_GPIO_CLKEN_SW_FLDSHFT (3)
+#define HWIO_SCU_PERIPH_CLK_CTRL_PERI_DMA_CLKEN_SW_FLDMASK (0x4)
+#define HWIO_SCU_PERIPH_CLK_CTRL_PERI_DMA_CLKEN_SW_FLDSHFT (2)
+#define HWIO_SCU_PERIPH_CLK_CTRL_TIMER_CLKEN_SW_FLDMASK (0x2)
+#define HWIO_SCU_PERIPH_CLK_CTRL_TIMER_CLKEN_SW_FLDSHFT (1)
+#define HWIO_SCU_PERIPH_CLK_CTRL_WDT_CLKEN_SW_FLDMASK (0x1)
+#define HWIO_SCU_PERIPH_CLK_CTRL_WDT_CLKEN_SW_FLDSHFT (0)
+
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_REGOFF 0x8C
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_PERIPH_TMR_CLK_CTL_REGOFF)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_RSVD0_FLDMASK (0xfff00000)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_RSVD0_FLDSHFT (20)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR3_EXTCLK_SE_FLDMASK (0x80000)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR3_EXTCLK_SE_FLDSHFT (19)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR2_EXTCLK_SE_FLDMASK (0x40000)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR2_EXTCLK_SE_FLDSHFT (18)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR1_EXTCLK_SE_FLDMASK (0x20000)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR1_EXTCLK_SE_FLDSHFT (17)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR0_EXTCLK_SE_FLDMASK (0x10000)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR0_EXTCLK_SE_FLDSHFT (16)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_RSVD1_FLDMASK (0xf000)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_RSVD1_FLDSHFT (12)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR3_EXTPAUSE_POL_FLDMASK (0x800)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR3_EXTPAUSE_POL_FLDSHFT (11)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR2_EXTPAUSE_POL_FLDMASK (0x400)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR2_EXTPAUSE_POL_FLDSHFT (10)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR1_EXTPAUSE_POL_FLDMASK (0x200)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR1_EXTPAUSE_POL_FLDSHFT (9)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR0_EXTPAUSE_POL_FLDMASK (0x100)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR0_EXTPAUSE_POL_FLDSHFT (8)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_RSVD2_FLDMASK (0xf0)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_RSVD2_FLDSHFT (4)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR3_EXTPAUSE_EN_FLDMASK (0x8)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR3_EXTPAUSE_EN_FLDSHFT (3)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR2_EXTPAUSE_EN_FLDMASK (0x4)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR2_EXTPAUSE_EN_FLDSHFT (2)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR1_EXTPAUSE_EN_FLDMASK (0x2)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR1_EXTPAUSE_EN_FLDSHFT (1)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR0_EXTPAUSE_EN_FLDMASK (0x1)
+#define HWIO_SCU_PERIPH_TMR_CLK_CTL_TMR0_EXTPAUSE_EN_FLDSHFT (0)
+
+#define HWIO_SCU_MEM_PWR_MGMNT_REGOFF 0x90
+#define HWIO_SCU_MEM_PWR_MGMNT_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_MEM_PWR_MGMNT_REGOFF)
+#define HWIO_SCU_MEM_PWR_MGMNT_HALT_CPUMEM_PD_EN_FLDMASK (0x80000000)
+#define HWIO_SCU_MEM_PWR_MGMNT_HALT_CPUMEM_PD_EN_FLDSHFT (31)
+#define HWIO_SCU_MEM_PWR_MGMNT_HALT_LP4CMEM_PD_EN_FLDMASK (0x40000000)
+#define HWIO_SCU_MEM_PWR_MGMNT_HALT_LP4CMEM_PD_EN_FLDSHFT (30)
+#define HWIO_SCU_MEM_PWR_MGMNT_HALT_BTROM_PD_EN_FLDMASK (0x20000000)
+#define HWIO_SCU_MEM_PWR_MGMNT_HALT_BTROM_PD_EN_FLDSHFT (29)
+#define HWIO_SCU_MEM_PWR_MGMNT_HALT_BTSRAM_PD_EN_FLDMASK (0x10000000)
+#define HWIO_SCU_MEM_PWR_MGMNT_HALT_BTSRAM_PD_EN_FLDSHFT (28)
+#define HWIO_SCU_MEM_PWR_MGMNT_RSVD0_FLDMASK (0xc000000)
+#define HWIO_SCU_MEM_PWR_MGMNT_RSVD0_FLDSHFT (26)
+#define HWIO_SCU_MEM_PWR_MGMNT_BTROM_SLP_FLDMASK (0x2000000)
+#define HWIO_SCU_MEM_PWR_MGMNT_BTROM_SLP_FLDSHFT (25)
+#define HWIO_SCU_MEM_PWR_MGMNT_RSVD1_FLDMASK (0x1000000)
+#define HWIO_SCU_MEM_PWR_MGMNT_RSVD1_FLDSHFT (24)
+#define HWIO_SCU_MEM_PWR_MGMNT_BTSRAM_DS_FLDMASK (0x800000)
+#define HWIO_SCU_MEM_PWR_MGMNT_BTSRAM_DS_FLDSHFT (23)
+#define HWIO_SCU_MEM_PWR_MGMNT_BTSRAM_SD_FLDMASK (0x400000)
+#define HWIO_SCU_MEM_PWR_MGMNT_BTSRAM_SD_FLDSHFT (22)
+#define HWIO_SCU_MEM_PWR_MGMNT_LP4C_MEM_DS_FLDMASK (0x200000)
+#define HWIO_SCU_MEM_PWR_MGMNT_LP4C_MEM_DS_FLDSHFT (21)
+#define HWIO_SCU_MEM_PWR_MGMNT_LP4C_MEM_SD_FLDMASK (0x100000)
+#define HWIO_SCU_MEM_PWR_MGMNT_LP4C_MEM_SD_FLDSHFT (20)
+#define HWIO_SCU_MEM_PWR_MGMNT_CPU_L2MEM_DS_FLDMASK (0x80000)
+#define HWIO_SCU_MEM_PWR_MGMNT_CPU_L2MEM_DS_FLDSHFT (19)
+#define HWIO_SCU_MEM_PWR_MGMNT_CPU_L2MEM_SD_FLDMASK (0x40000)
+#define HWIO_SCU_MEM_PWR_MGMNT_CPU_L2MEM_SD_FLDSHFT (18)
+#define HWIO_SCU_MEM_PWR_MGMNT_CPU_L1MEM_DS_FLDMASK (0x20000)
+#define HWIO_SCU_MEM_PWR_MGMNT_CPU_L1MEM_DS_FLDSHFT (17)
+#define HWIO_SCU_MEM_PWR_MGMNT_CPU_L1MEM_SD_FLDMASK (0x10000)
+#define HWIO_SCU_MEM_PWR_MGMNT_CPU_L1MEM_SD_FLDSHFT (16)
+#define HWIO_SCU_MEM_PWR_MGMNT_HALT_MEMPD_CNT_FLDMASK (0xf000)
+#define HWIO_SCU_MEM_PWR_MGMNT_HALT_MEMPD_CNT_FLDSHFT (12)
+#define HWIO_SCU_MEM_PWR_MGMNT_WAKE_MEMPU_CNT_FLDMASK (0xf00)
+#define HWIO_SCU_MEM_PWR_MGMNT_WAKE_MEMPU_CNT_FLDSHFT (8)
+#define HWIO_SCU_MEM_PWR_MGMNT_MIPITX_MEM_DS_FLDMASK (0x80)
+#define HWIO_SCU_MEM_PWR_MGMNT_MIPITX_MEM_DS_FLDSHFT (7)
+#define HWIO_SCU_MEM_PWR_MGMNT_MIPITX_MEM_SD_FLDMASK (0x40)
+#define HWIO_SCU_MEM_PWR_MGMNT_MIPITX_MEM_SD_FLDSHFT (6)
+#define HWIO_SCU_MEM_PWR_MGMNT_MIPIRX_MEM_DS_FLDMASK (0x20)
+#define HWIO_SCU_MEM_PWR_MGMNT_MIPIRX_MEM_DS_FLDSHFT (5)
+#define HWIO_SCU_MEM_PWR_MGMNT_MIPIRX_MEM_SD_FLDMASK (0x10)
+#define HWIO_SCU_MEM_PWR_MGMNT_MIPIRX_MEM_SD_FLDSHFT (4)
+#define HWIO_SCU_MEM_PWR_MGMNT_IPU_MEM_DS_FLDMASK (0x8)
+#define HWIO_SCU_MEM_PWR_MGMNT_IPU_MEM_DS_FLDSHFT (3)
+#define HWIO_SCU_MEM_PWR_MGMNT_IPU_MEM_SD_FLDMASK (0x4)
+#define HWIO_SCU_MEM_PWR_MGMNT_IPU_MEM_SD_FLDSHFT (2)
+#define HWIO_SCU_MEM_PWR_MGMNT_PCIE_MEM_DS_FLDMASK (0x2)
+#define HWIO_SCU_MEM_PWR_MGMNT_PCIE_MEM_DS_FLDSHFT (1)
+#define HWIO_SCU_MEM_PWR_MGMNT_PCIE_MEM_SD_FLDMASK (0x1)
+#define HWIO_SCU_MEM_PWR_MGMNT_PCIE_MEM_SD_FLDSHFT (0)
+
+#define HWIO_SCU_DBG_CONFIG_REGOFF 0x0A0
+#define HWIO_SCU_DBG_CONFIG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_DBG_CONFIG_REGOFF)
+#define HWIO_SCU_DBG_CONFIG_DBG_CONFIG_FLDMASK (0xffffffff)
+#define HWIO_SCU_DBG_CONFIG_DBG_CONFIG_FLDSHFT (0)
+
+#define HWIO_SCU_DBG_STATUS_REGOFF 0x0A4
+#define HWIO_SCU_DBG_STATUS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_DBG_STATUS_REGOFF)
+#define HWIO_SCU_DBG_STATUS_DBG_STATUS_FLDMASK (0xffffffff)
+#define HWIO_SCU_DBG_STATUS_DBG_STATUS_FLDSHFT (0)
+
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_REGOFF 0x0A8
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_LPDDR4_LOW_POWER_CFG_REGOFF)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LP4_FSP_SW_OVERRIDE_FLDMASK \
+ (0x80000000)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LP4_FSP_SW_OVERRIDE_FLDSHFT (31)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LP4_PI_FREQCHG_EN_FLDMASK (0x40000000)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LP4_PI_FREQCHG_EN_FLDSHFT (30)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LP4_FREQ_CHG_ACK_FLDMASK (0x20000000)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LP4_FREQ_CHG_ACK_FLDSHFT (29)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_RSVD0_FLDMASK (0x1fff0000)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_RSVD0_FLDSHFT (16)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LPC_EXT_CMD_REQ_FLDMASK (0x8000)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LPC_EXT_CMD_REQ_FLDSHFT (15)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_RSVD1_FLDMASK (0x7c00)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_RSVD1_FLDSHFT (10)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LPC_FREQ_CHG_COPY_NUM_FLDMASK (0x300)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LPC_FREQ_CHG_COPY_NUM_FLDSHFT (8)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LPC_EXT_CMD_FLDMASK (0xff)
+#define HWIO_SCU_LPDDR4_LOW_POWER_CFG_LPC_EXT_CMD_FLDSHFT (0)
+
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_REGOFF 0x0AC
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_LPDDR4_LOW_POWER_STS_REGOFF)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_RSVD0_FLDMASK (0xff800000)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_RSVD0_FLDSHFT (23)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_LPDDR4_REQ_FSP_FLDMASK (0x700000)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_LPDDR4_REQ_FSP_FLDSHFT (20)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_RSVD1_FLDMASK (0x80000)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_RSVD1_FLDSHFT (19)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_LPDDR4_CUR_FSP_FLDMASK (0x70000)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_LPDDR4_CUR_FSP_FLDSHFT (16)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_RSVD2_FLDMASK (0xfffc)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_RSVD2_FLDSHFT (2)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_LPC_CMD_DONE_FLDMASK (0x2)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_LPC_CMD_DONE_FLDSHFT (1)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_LPC_CMD_RSP_FLDMASK (0x1)
+#define HWIO_SCU_LPDDR4_LOW_POWER_STS_LPC_CMD_RSP_FLDSHFT (0)
+
+#define HWIO_SCU_EFUSE_PASSCODE_REGOFF 0x0B0
+#define HWIO_SCU_EFUSE_PASSCODE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_EFUSE_PASSCODE_REGOFF)
+#define HWIO_SCU_EFUSE_PASSCODE_RSVD0_FLDMASK (0xffff0000)
+#define HWIO_SCU_EFUSE_PASSCODE_RSVD0_FLDSHFT (16)
+#define HWIO_SCU_EFUSE_PASSCODE_PASSCODE_FLDMASK (0xffff)
+#define HWIO_SCU_EFUSE_PASSCODE_PASSCODE_FLDSHFT (0)
+
+#define HWIO_SCU_PLL_PASSCODE_REGOFF 0x0B4
+#define HWIO_SCU_PLL_PASSCODE_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_PLL_PASSCODE_REGOFF)
+#define HWIO_SCU_PLL_PASSCODE_RSVD0_FLDMASK (0xffff0000)
+#define HWIO_SCU_PLL_PASSCODE_RSVD0_FLDSHFT (16)
+#define HWIO_SCU_PLL_PASSCODE_PASSCODE_FLDMASK (0xffff)
+#define HWIO_SCU_PLL_PASSCODE_PASSCODE_FLDSHFT (0)
+
+#define HWIO_SCU_SOC_GLOBAL_CONTROL_REGOFF 0x0B8
+#define HWIO_SCU_SOC_GLOBAL_CONTROL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_SOC_GLOBAL_CONTROL_REGOFF)
+#define HWIO_SCU_SOC_GLOBAL_CONTROL_RSVD0_FLDMASK (0xfffffffe)
+#define HWIO_SCU_SOC_GLOBAL_CONTROL_RSVD0_FLDSHFT (1)
+#define HWIO_SCU_SOC_GLOBAL_CONTROL_GLOBAL_PAD_OUTPUT_DISABLE_FLDMASK (0x1)
+#define HWIO_SCU_SOC_GLOBAL_CONTROL_GLOBAL_PAD_OUTPUT_DISABLE_FLDSHFT (0)
+
+#define HWIO_SCU_PCIE_REFCLK_PLL_FRAC_DIV_REGOFF 0x0C0
+#define HWIO_SCU_PCIE_REFCLK_PLL_FRAC_DIV_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_PCIE_REFCLK_PLL_FRAC_DIV_REGOFF)
+#define HWIO_SCU_PCIE_REFCLK_PLL_FRAC_DIV_RSVD0_FLDMASK (0xff000000)
+#define HWIO_SCU_PCIE_REFCLK_PLL_FRAC_DIV_RSVD0_FLDSHFT (24)
+#define HWIO_SCU_PCIE_REFCLK_PLL_FRAC_DIV_FRAC_FLDMASK (0xffffff)
+#define HWIO_SCU_PCIE_REFCLK_PLL_FRAC_DIV_FRAC_FLDSHFT (0)
+
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_REGOFF 0x0C4
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_REGOFF)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_RSVD0_FLDMASK (0xc0000000)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_RSVD0_FLDSHFT (30)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_REFDIV_FLDMASK (0x3f000000)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_REFDIV_FLDSHFT (24)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_RSVD1_FLDMASK (0x800000)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_RSVD1_FLDSHFT (23)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_POSTDIV1_FLDMASK (0x700000)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_POSTDIV1_FLDSHFT (20)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_RSVD2_FLDMASK (0x80000)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_RSVD2_FLDSHFT (19)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_POSTDIV2_FLDMASK (0x70000)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_POSTDIV2_FLDSHFT (16)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_RSVD3_FLDMASK (0xf000)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_RSVD3_FLDSHFT (12)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_FBDIV_FLDMASK (0xfff)
+#define HWIO_SCU_PCIE_REFCLK_PLL_INTGR_DIV_FBDIV_FLDSHFT (0)
+
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_REGOFF 0x0C8
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_PCIE_REFCLK_PLL_CTRL_REGOFF)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_RSVD0_FLDMASK (0xfffe0000)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_RSVD0_FLDSHFT (17)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_FRZ_PLL_IN_FLDMASK (0x10000)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_FRZ_PLL_IN_FLDSHFT (16)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_RSVD1_FLDMASK (0xff80)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_RSVD1_FLDSHFT (7)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_PD_FLDMASK (0x40)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_PD_FLDSHFT (6)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_FOUTPOSTDIVPD_FLDMASK (0x20)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_FOUTPOSTDIVPD_FLDSHFT (5)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_FOUT4PHASEPD_FLDMASK (0x10)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_FOUT4PHASEPD_FLDSHFT (4)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_FOUTVCOPD_FLDMASK (0x8)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_FOUTVCOPD_FLDSHFT (3)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_DSMPD_FLDMASK (0x4)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_DSMPD_FLDSHFT (2)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_DACPD_FLDMASK (0x2)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_DACPD_FLDSHFT (1)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_BYPASS_FLDMASK (0x1)
+#define HWIO_SCU_PCIE_REFCLK_PLL_CTRL_BYPASS_FLDSHFT (0)
+
+#define HWIO_SCU_PCIE_REFCLK_PLL_STS_REGOFF 0x0CC
+#define HWIO_SCU_PCIE_REFCLK_PLL_STS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_PCIE_REFCLK_PLL_STS_REGOFF)
+#define HWIO_SCU_PCIE_REFCLK_PLL_STS_RSVD0_FLDMASK (0xfffffffe)
+#define HWIO_SCU_PCIE_REFCLK_PLL_STS_RSVD0_FLDSHFT (1)
+#define HWIO_SCU_PCIE_REFCLK_PLL_STS_LOCK_FLDMASK (0x1)
+#define HWIO_SCU_PCIE_REFCLK_PLL_STS_LOCK_FLDSHFT (0)
+
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_FRAC_DIV_REGOFF 0x0D0
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_FRAC_DIV_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_LPDDR4_REFCLK_PLL_FRAC_DIV_REGOFF)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_FRAC_DIV_RSVD0_FLDMASK (0xff000000)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_FRAC_DIV_RSVD0_FLDSHFT (24)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_FRAC_DIV_FRAC_FLDMASK (0xffffff)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_FRAC_DIV_FRAC_FLDSHFT (0)
+
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_REGOFF 0x0D4
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_REGOFF)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_RSVD0_FLDMASK (0xc0000000)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_RSVD0_FLDSHFT (30)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_REFDIV_FLDMASK (0x3f000000)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_REFDIV_FLDSHFT (24)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_RSVD1_FLDMASK (0x800000)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_RSVD1_FLDSHFT (23)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_POSTDIV1_FLDMASK (0x700000)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_POSTDIV1_FLDSHFT (20)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_RSVD2_FLDMASK (0x80000)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_RSVD2_FLDSHFT (19)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_POSTDIV2_FLDMASK (0x70000)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_POSTDIV2_FLDSHFT (16)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_RSVD3_FLDMASK (0xf000)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_RSVD3_FLDSHFT (12)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_FBDIV_FLDMASK (0xfff)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_INTGR_DIV_FBDIV_FLDSHFT (0)
+
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_REGOFF 0x0D8
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_REGOFF)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_RSVD0_FLDMASK (0xfffe0000)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_RSVD0_FLDSHFT (17)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_FRZ_PLL_IN_FLDMASK (0x10000)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_FRZ_PLL_IN_FLDSHFT (16)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_RSVD1_FLDMASK (0xff80)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_RSVD1_FLDSHFT (7)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_PD_FLDMASK (0x40)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_PD_FLDSHFT (6)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_FOUTPOSTDIVPD_FLDMASK (0x20)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_FOUTPOSTDIVPD_FLDSHFT (5)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_FOUT4PHASEPD_FLDMASK (0x10)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_FOUT4PHASEPD_FLDSHFT (4)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_FOUTVCOPD_FLDMASK (0x8)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_FOUTVCOPD_FLDSHFT (3)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_DSMPD_FLDMASK (0x4)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_DSMPD_FLDSHFT (2)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_DACPD_FLDMASK (0x2)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_DACPD_FLDSHFT (1)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_BYPASS_FLDMASK (0x1)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_CTRL_BYPASS_FLDSHFT (0)
+
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_STS_REGOFF 0x0DC
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_STS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_LPDDR4_REFCLK_PLL_STS_REGOFF)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_STS_RSVD0_FLDMASK (0xfffffffe)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_STS_RSVD0_FLDSHFT (1)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_STS_LOCK_FLDMASK (0x1)
+#define HWIO_SCU_LPDDR4_REFCLK_PLL_STS_LOCK_FLDSHFT (0)
+
+#define HWIO_SCU_CPU_IPU_PLL_FRAC_DIV_REGOFF 0x0E0
+#define HWIO_SCU_CPU_IPU_PLL_FRAC_DIV_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_CPU_IPU_PLL_FRAC_DIV_REGOFF)
+#define HWIO_SCU_CPU_IPU_PLL_FRAC_DIV_RSVD0_FLDMASK (0xff000000)
+#define HWIO_SCU_CPU_IPU_PLL_FRAC_DIV_RSVD0_FLDSHFT (24)
+#define HWIO_SCU_CPU_IPU_PLL_FRAC_DIV_FRAC_FLDMASK (0xffffff)
+#define HWIO_SCU_CPU_IPU_PLL_FRAC_DIV_FRAC_FLDSHFT (0)
+
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_REGOFF 0x0E4
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_REGOFF)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_RSVD0_FLDMASK (0xc0000000)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_RSVD0_FLDSHFT (30)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_REFDIV_FLDMASK (0x3f000000)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_REFDIV_FLDSHFT (24)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_RSVD1_FLDMASK (0x800000)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_RSVD1_FLDSHFT (23)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_POSTDIV1_FLDMASK (0x700000)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_POSTDIV1_FLDSHFT (20)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_RSVD2_FLDMASK (0x80000)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_RSVD2_FLDSHFT (19)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_POSTDIV2_FLDMASK (0x70000)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_POSTDIV2_FLDSHFT (16)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_RSVD3_FLDMASK (0xf000)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_RSVD3_FLDSHFT (12)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_FBDIV_FLDMASK (0xfff)
+#define HWIO_SCU_CPU_IPU_PLL_INTGR_DIV_FBDIV_FLDSHFT (0)
+
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_REGOFF 0x0E8
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_CPU_IPU_PLL_CTRL_REGOFF)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_RSVD0_FLDMASK (0xfffe0000)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_RSVD0_FLDSHFT (17)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_FRZ_PLL_IN_FLDMASK (0x10000)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_FRZ_PLL_IN_FLDSHFT (16)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_RSVD1_FLDMASK (0xff80)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_RSVD1_FLDSHFT (7)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_PD_FLDMASK (0x40)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_PD_FLDSHFT (6)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_FOUTPOSTDIVPD_FLDMASK (0x20)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_FOUTPOSTDIVPD_FLDSHFT (5)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_FOUT4PHASEPD_FLDMASK (0x10)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_FOUT4PHASEPD_FLDSHFT (4)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_FOUTVCOPD_FLDMASK (0x8)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_FOUTVCOPD_FLDSHFT (3)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_DSMPD_FLDMASK (0x4)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_DSMPD_FLDSHFT (2)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_DACPD_FLDMASK (0x2)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_DACPD_FLDSHFT (1)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_BYPASS_FLDMASK (0x1)
+#define HWIO_SCU_CPU_IPU_PLL_CTRL_BYPASS_FLDSHFT (0)
+
+#define HWIO_SCU_CPU_IPU_PLL_STS_REGOFF 0x0EC
+#define HWIO_SCU_CPU_IPU_PLL_STS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_CPU_IPU_PLL_STS_REGOFF)
+#define HWIO_SCU_CPU_IPU_PLL_STS_RSVD0_FLDMASK (0xfffffffe)
+#define HWIO_SCU_CPU_IPU_PLL_STS_RSVD0_FLDSHFT (1)
+#define HWIO_SCU_CPU_IPU_PLL_STS_LOCK_FLDMASK (0x1)
+#define HWIO_SCU_CPU_IPU_PLL_STS_LOCK_FLDSHFT (0)
+
+#define HWIO_SCU_LPDDR4_FSP_SETTING_REGNUM 4
+#define HWIO_SCU_LPDDR4_FSP_SETTING_REGOFF 0x0F0
+#define HWIO_SCU_LPDDR4_FSP_SETTING_ADDR(bAddr, regX) \
+ (((regX >= 0) && (regX < HWIO_SCU_LPDDR4_FSP_SETTING_REGNUM)) ? \
+ (bAddr + HWIO_SCU_LPDDR4_FSP_SETTING_REGOFF + (regX * 4)) : 0x0)
+
+#define HWIO_SCU_LPDDR4_FSP_SETTING_RSVD0_FLDMASK (0xc0000000)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_RSVD0_FLDSHFT (30)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_SW_CTRL_FLDMASK (0x20000000)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_SW_CTRL_FLDSHFT (29)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_SYS200_MODE_FLDMASK (0x10000000)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_SYS200_MODE_FLDSHFT (28)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_PCIE_AXI_CLK_DIV_FLDMASK (0xf000000)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_PCIE_AXI_CLK_DIV_FLDSHFT (24)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_AXI_FABRIC_CLK_DIV_FLDMASK (0xf00000)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_AXI_FABRIC_CLK_DIV_FLDSHFT (20)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_LPDDR4_REFCLK_DIV_FLDMASK (0xf0000)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_LPDDR4_REFCLK_DIV_FLDSHFT (16)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_RSVD1_FLDMASK (0xf000)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_RSVD1_FLDSHFT (12)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_FBDIV_FLDMASK (0xfff)
+#define HWIO_SCU_LPDDR4_FSP_SETTING_FSP_FBDIV_FLDSHFT (0)
+
+#define HWIO_SCU_PVT_CONTROL_REGNUM 4
+#define HWIO_SCU_PVT_CONTROL_REGOFF 0x100
+#define HWIO_SCU_PVT_CONTROL_ADDR(bAddr, regX) \
+ (((regX >= 0) && (regX < HWIO_SCU_PVT_CONTROL_REGNUM)) ? \
+ (bAddr + HWIO_SCU_PVT_CONTROL_REGOFF + (regX * 8)) : 0)
+#define HWIO_SCU_PVT_CONTROL_ALARM_ENA_FLDMASK (0x80000000)
+#define HWIO_SCU_PVT_CONTROL_ALARM_ENA_FLDSHFT (31)
+#define HWIO_SCU_PVT_CONTROL_TRIG_FLDMASK (0x7fe00000)
+#define HWIO_SCU_PVT_CONTROL_TRIG_FLDSHFT (21)
+#define HWIO_SCU_PVT_CONTROL_TRIM_FLDMASK (0x1f0000)
+#define HWIO_SCU_PVT_CONTROL_TRIM_FLDSHFT (16)
+#define HWIO_SCU_PVT_CONTROL_RSVD0_FLDMASK (0xffc0)
+#define HWIO_SCU_PVT_CONTROL_RSVD0_FLDSHFT (6)
+#define HWIO_SCU_PVT_CONTROL_PRECISION_FLDMASK (0x30)
+#define HWIO_SCU_PVT_CONTROL_PRECISION_FLDSHFT (4)
+#define HWIO_SCU_PVT_CONTROL_PSAMPLE_FLDMASK (0xc)
+#define HWIO_SCU_PVT_CONTROL_PSAMPLE_FLDSHFT (2)
+#define HWIO_SCU_PVT_CONTROL_VSAMPLE_FLDMASK (0x2)
+#define HWIO_SCU_PVT_CONTROL_VSAMPLE_FLDSHFT (1)
+#define HWIO_SCU_PVT_CONTROL_ENA_FLDMASK (0x1)
+#define HWIO_SCU_PVT_CONTROL_ENA_FLDSHFT (0)
+
+#define HWIO_SCU_PVT_DATA_REGNUM 4
+#define HWIO_SCU_PVT_DATA_REGOFF 0x104
+#define HWIO_SCU_PVT_DATA_ADDR(bAddr, regX) \
+ (((regX >= 0) && (regX < HWIO_SCU_PVT_DATA_REGNUM)) ? \
+ (bAddr + HWIO_SCU_PVT_DATA_REGOFF + (regX * 8)) : 0)
+#define HWIO_SCU_PVT_DATA_RSVD0_FLDMASK (0xfffe0000)
+#define HWIO_SCU_PVT_DATA_RSVD0_FLDSHFT (17)
+#define HWIO_SCU_PVT_DATA_DATAVALID_FLDMASK (0x10000)
+#define HWIO_SCU_PVT_DATA_DATAVALID_FLDSHFT (16)
+#define HWIO_SCU_PVT_DATA_RSVD1_FLDMASK (0xfc00)
+#define HWIO_SCU_PVT_DATA_RSVD1_FLDSHFT (10)
+#define HWIO_SCU_PVT_DATA_DATA_FLDMASK (0x3ff)
+#define HWIO_SCU_PVT_DATA_DATA_FLDSHFT (0)
+
+#define HWIO_SCU_EFUSE_CTRL_REGOFF 0x140
+#define HWIO_SCU_EFUSE_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_EFUSE_CTRL_REGOFF)
+#define HWIO_SCU_EFUSE_CTRL_RSVD0_FLDMASK (0xffffff80)
+#define HWIO_SCU_EFUSE_CTRL_RSVD0_FLDSHFT (7)
+#define HWIO_SCU_EFUSE_CTRL_LOAD_FLDMASK (0x40)
+#define HWIO_SCU_EFUSE_CTRL_LOAD_FLDSHFT (6)
+#define HWIO_SCU_EFUSE_CTRL_PGENB_FLDMASK (0x20)
+#define HWIO_SCU_EFUSE_CTRL_PGENB_FLDSHFT (5)
+#define HWIO_SCU_EFUSE_CTRL_PS_FLDMASK (0x10)
+#define HWIO_SCU_EFUSE_CTRL_PS_FLDSHFT (4)
+#define HWIO_SCU_EFUSE_CTRL_PD_FLDMASK (0x8)
+#define HWIO_SCU_EFUSE_CTRL_PD_FLDSHFT (3)
+#define HWIO_SCU_EFUSE_CTRL_MR_FLDMASK (0x4)
+#define HWIO_SCU_EFUSE_CTRL_MR_FLDSHFT (2)
+#define HWIO_SCU_EFUSE_CTRL_RSB_FLDMASK (0x2)
+#define HWIO_SCU_EFUSE_CTRL_RSB_FLDSHFT (1)
+#define HWIO_SCU_EFUSE_CTRL_RWL_FLDMASK (0x1)
+#define HWIO_SCU_EFUSE_CTRL_RWL_FLDSHFT (0)
+
+#define HWIO_SCU_EFUSE_ACC_REGOFF 0x144
+#define HWIO_SCU_EFUSE_ACC_ADDR(bAddr, regX) (bAddr + HWIO_SCU_EFUSE_ACC_REGOFF)
+#define HWIO_SCU_EFUSE_ACC_RSVD0_FLDMASK (0xff000000)
+#define HWIO_SCU_EFUSE_ACC_RSVD0_FLDSHFT (24)
+#define HWIO_SCU_EFUSE_ACC_A7_FLDMASK (0x800000)
+#define HWIO_SCU_EFUSE_ACC_A7_FLDSHFT (23)
+#define HWIO_SCU_EFUSE_ACC_A6_FLDMASK (0x400000)
+#define HWIO_SCU_EFUSE_ACC_A6_FLDSHFT (22)
+#define HWIO_SCU_EFUSE_ACC_A5_FLDMASK (0x200000)
+#define HWIO_SCU_EFUSE_ACC_A5_FLDSHFT (21)
+#define HWIO_SCU_EFUSE_ACC_A4_FLDMASK (0x100000)
+#define HWIO_SCU_EFUSE_ACC_A4_FLDSHFT (20)
+#define HWIO_SCU_EFUSE_ACC_A3_FLDMASK (0x80000)
+#define HWIO_SCU_EFUSE_ACC_A3_FLDSHFT (19)
+#define HWIO_SCU_EFUSE_ACC_ZEROED_FLDMASK \
+ (HWIO_SCU_EFUSE_ACC_A3_FLDMASK | \
+ HWIO_SCU_EFUSE_ACC_A4_FLDMASK | \
+ HWIO_SCU_EFUSE_ACC_A5_FLDMASK | \
+ HWIO_SCU_EFUSE_ACC_A6_FLDMASK | \
+ HWIO_SCU_EFUSE_ACC_A7_FLDMASK)
+#define HWIO_SCU_EFUSE_ACC_ZEROED_FLDSHFT HWIO_SCU_EFUSE_ACC_A3_FLDSHFT
+#define HWIO_SCU_EFUSE_ACC_A2_FLDMASK (0x40000)
+#define HWIO_SCU_EFUSE_ACC_A2_FLDSHFT (18)
+#define HWIO_SCU_EFUSE_ACC_A1_FLDMASK (0x20000)
+#define HWIO_SCU_EFUSE_ACC_A1_FLDSHFT (17)
+#define HWIO_SCU_EFUSE_ACC_A0_FLDMASK (0x10000)
+#define HWIO_SCU_EFUSE_ACC_A0_FLDSHFT (16)
+#define HWIO_SCU_EFUSE_ACC_ROW_FLDMASK \
+ (HWIO_SCU_EFUSE_ACC_A0_FLDMASK | \
+ HWIO_SCU_EFUSE_ACC_A1_FLDMASK | \
+ HWIO_SCU_EFUSE_ACC_A2_FLDMASK)
+#define HWIO_SCU_EFUSE_ACC_ROW_FLDSHFT HWIO_SCU_EFUSE_ACC_A0_FLDSHFT
+#define HWIO_SCU_EFUSE_ACC_RSVD1_FLDMASK (0xfffc)
+#define HWIO_SCU_EFUSE_ACC_RSVD1_FLDSHFT (2)
+#define HWIO_SCU_EFUSE_ACC_CSB_FLDMASK (0x2)
+#define HWIO_SCU_EFUSE_ACC_CSB_FLDSHFT (1)
+#define HWIO_SCU_EFUSE_ACC_STROBE_FLDMASK (0x1)
+#define HWIO_SCU_EFUSE_ACC_STROBE_FLDSHFT (0)
+
+#define HWIO_SCU_EFUSE_READ_DATA_REGOFF 0x148
+#define HWIO_SCU_EFUSE_READ_DATA_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_EFUSE_READ_DATA_REGOFF)
+#define HWIO_SCU_EFUSE_READ_DATA_READ_DATA_FLDMASK (0xffffffff)
+#define HWIO_SCU_EFUSE_READ_DATA_READ_DATA_FLDSHFT (0)
+
+#define HWIO_SCU_EFUSE_REDN_FLAG_REGOFF 0x14C
+#define HWIO_SCU_EFUSE_REDN_FLAG_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_EFUSE_REDN_FLAG_REGOFF)
+#define HWIO_SCU_EFUSE_REDN_FLAG_RSVD0_FLDMASK (0xfffffff0)
+#define HWIO_SCU_EFUSE_REDN_FLAG_RSVD0_FLDSHFT (4)
+#define HWIO_SCU_EFUSE_REDN_FLAG_RF3_FLDMASK (0x8)
+#define HWIO_SCU_EFUSE_REDN_FLAG_RF3_FLDSHFT (3)
+#define HWIO_SCU_EFUSE_REDN_FLAG_RF2_FLDMASK (0x4)
+#define HWIO_SCU_EFUSE_REDN_FLAG_RF2_FLDSHFT (2)
+#define HWIO_SCU_EFUSE_REDN_FLAG_RF1_FLDMASK (0x2)
+#define HWIO_SCU_EFUSE_REDN_FLAG_RF1_FLDSHFT (1)
+#define HWIO_SCU_EFUSE_REDN_FLAG_RF0_FLDMASK (0x1)
+#define HWIO_SCU_EFUSE_REDN_FLAG_RF0_FLDSHFT (0)
+
+#define HWIO_SCU_IPU_PLL_FRAC_DIV_REGOFF 0x1C0
+#define HWIO_SCU_IPU_PLL_FRAC_DIV_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_IPU_PLL_FRAC_DIV_REGOFF)
+#define HWIO_SCU_IPU_PLL_FRAC_DIV_RSVD0_FLDMASK (0xff000000)
+#define HWIO_SCU_IPU_PLL_FRAC_DIV_RSVD0_FLDSHFT (24)
+#define HWIO_SCU_IPU_PLL_FRAC_DIV_FRAC_FLDMASK (0xffffff)
+#define HWIO_SCU_IPU_PLL_FRAC_DIV_FRAC_FLDSHFT (0)
+
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_REGOFF 0x1C4
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_IPU_PLL_INTGR_DIV_REGOFF)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_RSVD0_FLDMASK (0xc0000000)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_RSVD0_FLDSHFT (30)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_REFDIV_FLDMASK (0x3f000000)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_REFDIV_FLDSHFT (24)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_RSVD1_FLDMASK (0x800000)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_RSVD1_FLDSHFT (23)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_POSTDIV1_FLDMASK (0x700000)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_POSTDIV1_FLDSHFT (20)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_RSVD2_FLDMASK (0x80000)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_RSVD2_FLDSHFT (19)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_POSTDIV2_FLDMASK (0x70000)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_POSTDIV2_FLDSHFT (16)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_RSVD3_FLDMASK (0xf000)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_RSVD3_FLDSHFT (12)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_FBDIV_FLDMASK (0xfff)
+#define HWIO_SCU_IPU_PLL_INTGR_DIV_FBDIV_FLDSHFT (0)
+
+#define HWIO_SCU_IPU_PLL_CTRL_REGOFF 0x1C8
+#define HWIO_SCU_IPU_PLL_CTRL_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_IPU_PLL_CTRL_REGOFF)
+#define HWIO_SCU_IPU_PLL_CTRL_RSVD0_FLDMASK (0xfffe0000)
+#define HWIO_SCU_IPU_PLL_CTRL_RSVD0_FLDSHFT (17)
+#define HWIO_SCU_IPU_PLL_CTRL_FRZ_PLL_IN_FLDMASK (0x10000)
+#define HWIO_SCU_IPU_PLL_CTRL_FRZ_PLL_IN_FLDSHFT (16)
+#define HWIO_SCU_IPU_PLL_CTRL_RSVD1_FLDMASK (0xff80)
+#define HWIO_SCU_IPU_PLL_CTRL_RSVD1_FLDSHFT (7)
+#define HWIO_SCU_IPU_PLL_CTRL_PD_FLDMASK (0x40)
+#define HWIO_SCU_IPU_PLL_CTRL_PD_FLDSHFT (6)
+#define HWIO_SCU_IPU_PLL_CTRL_FOUTPOSTDIVPD_FLDMASK (0x20)
+#define HWIO_SCU_IPU_PLL_CTRL_FOUTPOSTDIVPD_FLDSHFT (5)
+#define HWIO_SCU_IPU_PLL_CTRL_FOUT4PHASEPD_FLDMASK (0x10)
+#define HWIO_SCU_IPU_PLL_CTRL_FOUT4PHASEPD_FLDSHFT (4)
+#define HWIO_SCU_IPU_PLL_CTRL_FOUTVCOPD_FLDMASK (0x8)
+#define HWIO_SCU_IPU_PLL_CTRL_FOUTVCOPD_FLDSHFT (3)
+#define HWIO_SCU_IPU_PLL_CTRL_DSMPD_FLDMASK (0x4)
+#define HWIO_SCU_IPU_PLL_CTRL_DSMPD_FLDSHFT (2)
+#define HWIO_SCU_IPU_PLL_CTRL_DACPD_FLDMASK (0x2)
+#define HWIO_SCU_IPU_PLL_CTRL_DACPD_FLDSHFT (1)
+#define HWIO_SCU_IPU_PLL_CTRL_BYPASS_FLDMASK (0x1)
+#define HWIO_SCU_IPU_PLL_CTRL_BYPASS_FLDSHFT (0)
+
+#define HWIO_SCU_IPU_PLL_STS_REGOFF 0x1CC
+#define HWIO_SCU_IPU_PLL_STS_ADDR(bAddr, regX) \
+ (bAddr + HWIO_SCU_IPU_PLL_STS_REGOFF)
+#define HWIO_SCU_IPU_PLL_STS_RSVD0_FLDMASK (0xfffffffe)
+#define HWIO_SCU_IPU_PLL_STS_RSVD0_FLDSHFT (1)
+#define HWIO_SCU_IPU_PLL_STS_LOCK_FLDMASK (0x1)
+#define HWIO_SCU_IPU_PLL_STS_LOCK_FLDSHFT (0)
+
+
+#define HWIO_SCU_PIN_CFG_REGNUM 55
+#define HWIO_SCU_PIN_CFG_REGOFF 0x200
+#define HWIO_SCU_PIN_CFG_ADDR(bAddr, regX) \
+ (((regX >= 0) && (regX < HWIO_SCU_PIN_CFG_REGNUM)) ? \
+ (bAddr + HWIO_SCU_PIN_CFG_REGOFF + (regX * 4)) : 0x0)
+#define HWIO_SCU_PIN_CFG_RSVD0_FLDMASK (0xfffff800)
+#define HWIO_SCU_PIN_CFG_RSVD0_FLDSHFT (11)
+#define HWIO_SCU_PIN_CFG_PIN_SL_FLDMASK (0x400)
+#define HWIO_SCU_PIN_CFG_PIN_SL_FLDSHFT (10)
+#define HWIO_SCU_PIN_CFG_PIN_ST_FLDMASK (0x300)
+#define HWIO_SCU_PIN_CFG_PIN_ST_FLDSHFT (8)
+#define HWIO_SCU_PIN_CFG_PIN_DS_FLDMASK (0xc0)
+#define HWIO_SCU_PIN_CFG_PIN_DS_FLDSHFT (6)
+#define HWIO_SCU_PIN_CFG_PIN_HE_FLDMASK (0x20)
+#define HWIO_SCU_PIN_CFG_PIN_HE_FLDSHFT (5)
+#define HWIO_SCU_PIN_CFG_PIN_IE_FLDMASK (0x10)
+#define HWIO_SCU_PIN_CFG_PIN_IE_FLDSHFT (4)
+#define HWIO_SCU_PIN_CFG_PIN_PE_FLDMASK (0x8)
+#define HWIO_SCU_PIN_CFG_PIN_PE_FLDSHFT (3)
+#define HWIO_SCU_PIN_CFG_PIN_PS_FLDMASK (0x4)
+#define HWIO_SCU_PIN_CFG_PIN_PS_FLDSHFT (2)
+#define HWIO_SCU_PIN_CFG_PIN_MUXSEL_FLDMASK (0x3)
+#define HWIO_SCU_PIN_CFG_PIN_MUXSEL_FLDSHFT (0)
+
+#endif /* __MNH_HWIO_SCU_ */
diff --git a/drivers/misc/mnh/mnh-hwio.h b/drivers/misc/mnh/mnh-hwio.h
new file mode 100644
index 0000000..205d8eb
--- /dev/null
+++ b/drivers/misc/mnh/mnh-hwio.h
@@ -0,0 +1,92 @@
+/*
+ * Copyright (c) 2016, Intel Corporation. All rights reserved.
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are met:
+ *
+ * Redistributions of source code must retain the above copyright notice, this
+ * list of conditions and the following disclaimer.
+ *
+ * Redistributions in binary form must reproduce the above copyright notice,
+ * this list of conditions and the following disclaimer in the documentation
+ * and/or other materials provided with the distribution.
+ *
+ * Neither the name of Intel nor the names of its contributors may be used
+ * to endorse or promote products derived from this software without specific
+ * prior written permission.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+ * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
+ * LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+ * CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+ * SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+ * INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+ * CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ * ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+ * POSSIBILITY OF SUCH DAMAGE.
+ */
+#ifndef __MNH_HWIO_H_
+#define __MNH_HWIO_H_
+
+#include "mnh-hwio-bases.h"
+#include "mnh-pcie.h"
+
+static inline uint32_t mnh_reg_read(uint32_t addr)
+{
+ uint32_t data = 0;
+
+ mnh_config_read(addr, 4, &data);
+
+ return data;
+}
+
+static inline void mnh_reg_write(uint32_t addr, uint32_t value)
+{
+ mnh_config_write(addr, 4, value);
+}
+
+#define HW_IN(bAddr, mod, reg) \
+ mnh_reg_read(HWIO_##mod##_##reg##_ADDR(bAddr, 0))
+#define HW_INx(bAddr, mod, reg, inst) \
+ mnh_reg_read(HWIO_##mod##_##reg##_ADDR(bAddr, inst))
+#define HW_PRTx(bAddr, mod, reg, inst) \
+ printk(#mod" "#reg" "#inst" %p\n", \
+ HWIO_##mod##_##reg##_ADDR(bAddr, inst))
+
+#define HW_INf(bAddr, mod, reg, fld) \
+ ((mnh_reg_read(HWIO_##mod##_##reg##_ADDR(bAddr, 0)) & \
+ HWIO_##mod##_##reg##_##fld##_FLDMASK) >> \
+ HWIO_##mod##_##reg##_##fld##_FLDSHFT)
+#define HW_INxf(bAddr, mod, reg, inst, fld) \
+ ((mnh_reg_read(HWIO_##mod##_##reg##_ADDR(bAddr, inst)) & \
+ HWIO_##mod##_##reg##_##fld##_FLDMASK) >> \
+ HWIO_##mod##_##reg##_##fld##_FLDSHFT)
+
+#define HW_OUT(bAddr, mod, reg, val) \
+ mnh_reg_write(HWIO_##mod##_##reg##_ADDR(bAddr, 0), val)
+#define HW_PRT(bAddr, mod, reg) \
+ printk(#mod" "#reg" %p\n", \
+ HWIO_##mod##_##reg##_ADDR(bAddr, 0))
+
+#define HW_OUTx(bAddr, mod, reg, inst, val) \
+ mnh_reg_write(HWIO_##mod##_##reg##_ADDR(bAddr, inst), val)
+
+#define HW_OUTf(bAddr, mod, reg, fld, val) \
+ mnh_reg_write( \
+ HWIO_##mod##_##reg##_ADDR(bAddr, 0), \
+ (mnh_reg_read(HWIO_##mod##_##reg##_ADDR(bAddr, 0)) & \
+ ~HWIO_##mod##_##reg##_##fld##_FLDMASK) | \
+ ((val << HWIO_##mod##_##reg##_##fld##_FLDSHFT) & \
+ HWIO_##mod##_##reg##_##fld##_FLDMASK))
+
+#define HW_OUTxf(bAddr, mod, reg, inst, fld, val) \
+ mnh_reg_write( \
+ HWIO_##mod##_##reg##_ADDR(bAddr, inst), \
+ (mnh_reg_read(HWIO_##mod##_##reg##_ADDR(bAddr, inst)) & \
+ ~HWIO_##mod##_##reg##_##fld##_FLDMASK) | \
+ ((val << HWIO_##mod##_##reg##_##fld##_FLDSHFT) & \
+ HWIO_##mod##_##reg##_##fld##_FLDMASK))
+
+#endif /* __MNH_HWIO_H_ */
diff --git a/drivers/misc/mnh/mnh-mipi.c b/drivers/misc/mnh/mnh-mipi.c
new file mode 100644
index 0000000..aeed997
--- /dev/null
+++ b/drivers/misc/mnh/mnh-mipi.c
@@ -0,0 +1,1163 @@
+/*
+*
+* MNH MIPI APIs
+* Copyright (c) 2016, Intel Corporation.
+*
+* This program is free software; you can redistribute it and/or modify it
+* under the terms and conditions of the GNU General Public License,
+* version 2, as published by the Free Software Foundation.
+*
+* This program is distributed in the hope it will be useful, but WITHOUT
+* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+* more details.
+*
+*/
+
+#include <linux/delay.h>
+
+#include "mnh-hwio.h"
+#include "mnh-hwio-mipi-rx.h"
+#include "mnh-hwio-mipi-tx.h"
+#include "mnh-hwio-mipi-top.h"
+#include "mnh-hwio-bases.h"
+#include "mnh-mipi.h"
+
+#define REF_FREQ_KHZ 25000
+
+#define TOP_IN(reg) HW_IN(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, reg)
+#define TOP_MASK(reg, fld) HWIO_MIPI_TOP_##reg##_##fld##_FLDMASK
+#define TOP_OUTf(reg, fld, val) \
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, reg, fld, val)
+#define TOP_OUTf_LOCAL(reg, fld, val, curr) \
+ ((curr & ~HWIO_MIPI_TOP_##reg##_##fld##_FLDMASK) | \
+ ((val << HWIO_MIPI_TOP_##reg##_##fld##_FLDSHFT) & \
+ HWIO_MIPI_TOP_##reg##_##fld##_FLDMASK))
+
+#define TX_IN(reg) HW_IN(baddr, MIPI_TX, reg)
+#define TX_MASK(reg, fld) HWIO_MIPI_TX_##reg##_##fld##_FLDMASK
+
+#define RX_IN(reg) HW_IN(baddr, MIPI_RX, reg)
+#define RX_MASK(reg, fld) HWIO_MIPI_RX_##reg##_##fld##_FLDMASK
+
+struct mipi_rate_config {
+ uint32_t rate;
+ uint32_t freq_range_code;
+ uint32_t osc_freq_code;
+};
+
+struct mnh_mipi_vco_range_cntrl {
+ uint32_t freq_kHz;
+ uint8_t cntrl;
+};
+
+/* MIPI Timing override registers */
+struct mipi_dev_ovr_cfg {
+ uint32_t rate;
+ /* Timing override registers */
+ uint32_t reg_0x5A;
+ uint32_t reg_0x5B;
+ uint32_t reg_0x5C;
+ uint32_t reg_0x5D;
+ uint32_t reg_0x5E;
+ uint32_t reg_0x5F;
+ uint32_t reg_0x61;
+ uint32_t reg_0x62;
+ uint32_t reg_0x63;
+ uint32_t reg_0x64;
+ uint32_t reg_0x65;
+ /* PHY_STOP_WAIT_TIME */
+ uint32_t wait_time; /* Table A-4 High-speed exit */
+ bool use_ovrd;
+};
+
+/* must be in ascending order */
+static struct mipi_rate_config mipi_rate_configs[] = {
+ /* rate freq_range_code osc_freq_code */
+ { 80, 0x00, 0x1b6},
+ { 90, 0x10, 0x1b6},
+ { 100, 0x20, 0x1b6},
+ { 110, 0x30, 0x1b6},
+ { 120, 0x01, 0x1b6},
+ { 130, 0x11, 0x1b6},
+ { 140, 0x21, 0x1b6},
+ { 150, 0x31, 0x1b6},
+ { 160, 0x02, 0x1b6},
+ { 170, 0x12, 0x1b6},
+ { 180, 0x22, 0x1b6},
+ { 190, 0x32, 0x1b6},
+ { 205, 0x03, 0x1b6},
+ { 220, 0x13, 0x1b6},
+ { 235, 0x23, 0x1b6},
+ { 250, 0x33, 0x1b6},
+ { 275, 0x04, 0x1b6},
+ { 300, 0x14, 0x1b6},
+ { 325, 0x25, 0x1b6},
+ { 350, 0x35, 0x1b6},
+ { 400, 0x05, 0x1b6},
+ { 450, 0x16, 0x1b6},
+ { 500, 0x26, 0x1b6},
+ { 550, 0x37, 0x1b6},
+ { 600, 0x07, 0x1b6},
+ { 650, 0x18, 0x1b6},
+ { 700, 0x28, 0x1b6},
+ { 750, 0x39, 0x1b6},
+ { 800, 0x09, 0x1b6},
+ { 850, 0x19, 0x1b6},
+ { 900, 0x29, 0x1b6},
+ { 950, 0x3a, 0x1b6},
+ { 1000, 0x0a, 0x1b6},
+ { 1050, 0x1a, 0x1b6},
+ { 1100, 0x2a, 0x1b6},
+ { 1150, 0x3b, 0x1b6},
+ { 1200, 0x0b, 0x1b6},
+ { 1250, 0x1b, 0x1b6},
+ { 1300, 0x2b, 0x1b6},
+ { 1350, 0x3c, 0x1b6},
+ { 1400, 0x0c, 0x1b6},
+ { 1450, 0x1c, 0x1b6},
+ { 1500, 0x2c, 0x1b6},
+ { 1550, 0x3d, 0x10f},
+ { 1600, 0x0d, 0x118},
+ { 1650, 0x1d, 0x121},
+ { 1700, 0x2e, 0x12a},
+ { 1750, 0x3e, 0x132},
+ { 1800, 0x0e, 0x13b},
+ { 1850, 0x1e, 0x144},
+ { 1900, 0x2f, 0x14d},
+ { 1950, 0x3f, 0x155},
+ { 2000, 0x0f, 0x15e},
+ { 2050, 0x40, 0x167},
+ { 2100, 0x41, 0x170},
+ { 2150, 0x42, 0x178},
+ { 2200, 0x43, 0x181},
+ { 2250, 0x44, 0x18a},
+ { 2300, 0x45, 0x193},
+ { 2350, 0x46, 0x19b},
+ { 2400, 0x47, 0x1a4},
+ { 2450, 0x48, 0x1ad},
+ { 2500, 0x49, 0x1b6},
+};
+
+/* must be in ascending order */
+static struct mnh_mipi_vco_range_cntrl vco_range_cntrls[] = {
+ /* freq_kHz cntrl */
+ { 40000, 0x3f },
+ { 52500, 0x39 },
+ { 80000, 0x2f },
+ { 105000, 0x29 },
+ { 160000, 0x1f },
+ { 210000, 0x19 },
+ { 320000, 0x0f },
+ { 420000, 0x09 },
+ { 630000, 0x03 },
+ { 1100000, 0x01 },
+ { 1150000, 0x01 },
+};
+
+static struct mipi_dev_ovr_cfg mipi_dev_ovr_cfgs[] = {
+ /* rate 0x05A 0x05B 0x05C 0x05D 0x05E 0x05F 0x061 0x062 0x063 0x064 0x065 wait_time */
+ { 340, 0x57, 0x43, 0xC2, 0x44, 0x42, 0x8A, 0x43, 0xC3, 0x44, 0x43, 0x83, 0x14, true},
+ { 360, 0x57, 0x43, 0xC2, 0x44, 0x42, 0x8A, 0x43, 0xC3, 0x44, 0x43, 0x83, 0x14, false},
+ { 446, 0x57, 0x45, 0xC3, 0x9D, 0x43, 0x8E, 0x45, 0xC3, 0x9D, 0x44, 0x84, 0x17, true},
+ { 488, 0x58, 0x45, 0xC3, 0x03, 0x43, 0x90, 0x45, 0xC3, 0x03, 0x44, 0x85, 0x18, true},
+ { 550, 0x4C, 0x48, 0xC5, 0x03, 0x44, 0x91, 0x47, 0xC6, 0x03, 0x43, 0x85, 0x1A, true},
+ { 648, 0x4E, 0x48, 0xC5, 0x05, 0x47, 0x94, 0x48, 0xC5, 0x05, 0x47, 0x85, 0x1C, true},
+ { 685, 0x4C, 0x48, 0xC5, 0x03, 0x44, 0x91, 0x47, 0xC6, 0x03, 0x43, 0x85, 0x1D, true},
+ { 720, 0x4E, 0x49, 0xC5, 0x05, 0x48, 0x96, 0x49, 0xC6, 0x05, 0x48, 0x86, 0x1D, true},
+ { 984, 0x50, 0x4D, 0xC7, 0x05, 0x48, 0xA0, 0x4D, 0xC8, 0x05, 0x4A, 0x8A, 0x26, true},
+ { 1104, 0x5D, 0x4F, 0xC6, 0x09, 0x49, 0xA7, 0x4F, 0xC9, 0x07, 0x4B, 0x8B, 0x27, true},
+ { 1200, 0x53, 0x51, 0xCA, 0x09, 0x4E, 0xA7, 0x50, 0xCC, 0x07, 0x4B, 0x8D, 0x2B, true},
+ { 1296, 0x53, 0x51, 0xCA, 0x09, 0x4E, 0xA7, 0x50, 0xCC, 0x07, 0x4C, 0x8D, 0x2E, true},
+ { 1300, 0x53, 0x51, 0xCA, 0x09, 0x4E, 0xA7, 0x50, 0xCC, 0x07, 0x4C, 0x8D, 0x2E, true},
+ { 1368, 0x53, 0x51, 0xCA, 0x09, 0x4E, 0xA7, 0x50, 0xCC, 0x08, 0x4C, 0x8D, 0x2F, true},
+ { 1776, 0x58, 0x59, 0xCD, 0x0B, 0x55, 0xC0, 0x50, 0xCE, 0x0B, 0x52, 0x90, 0x36, true},
+ { 1850, 0x58, 0x59, 0xCD, 0x0B, 0x55, 0xC0, 0x59, 0xCE, 0x0B, 0x52, 0x98, 0x36, true},
+ { 1900, 0x58, 0x59, 0xCD, 0x0B, 0x55, 0xC0, 0x59, 0xCE, 0x0B, 0x52, 0x98, 0x1B, true},
+ { 2000, 0x58, 0x59, 0xCD, 0x0B, 0x55, 0xC0, 0x59, 0xCE, 0x0B, 0x52, 0x98, 0x1B, true},
+ { 2100, 0x9A, 0x9B, 0xD0, 0x0E, 0x50, 0xC0, 0x39, 0xD3, 0x0C, 0x4A, 0xA7, 0x1E, true},
+};
+
+static int mipi_debug;
+
+static void mnh_sm_mipi_rx_dphy_write_gen3(int command, int data,
+ uint32_t device)
+{
+ /* Write 4-bit testcode MSB */
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL0,
+ PHY_TESTCLK, 0);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL1,
+ PHY_TESTEN, 0);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL1,
+ PHY_TESTEN, 1);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL0,
+ PHY_TESTCLK, 1);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL1,
+ PHY_TESTDIN, 0);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL0,
+ PHY_TESTCLK, 0);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL1,
+ PHY_TESTEN, 0);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL1,
+ PHY_TESTDIN, ((command & 0xF00) >> 8));
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL0,
+ PHY_TESTCLK, 1);
+
+ /* Write 8-bit testcode LSB */
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL0,
+ PHY_TESTCLK, 0);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL1,
+ PHY_TESTEN, 1);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL0,
+ PHY_TESTCLK, 1);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL1,
+ PHY_TESTDIN, (command & 0xFF));
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL0,
+ PHY_TESTCLK, 0);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL1,
+ PHY_TESTEN, 0);
+
+ /* Write the data */
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL1,
+ PHY_TESTDIN, data);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL0,
+ PHY_TESTCLK, 1);
+ udelay(1);
+}
+
+static void mnh_sm_mipi_tx_dphy_write_gen3(int command, int data,
+ uint32_t device)
+{
+ /* Write 4-bit testcode MSB */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL0,
+ PHY0_TESTCLK, 0);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL1,
+ PHY0_TESTEN, 0);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL1,
+ PHY0_TESTEN, 1);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL0,
+ PHY0_TESTCLK, 1);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL1,
+ PHY0_TESTDIN, 0);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL0,
+ PHY0_TESTCLK, 0);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL1,
+ PHY0_TESTEN, 0);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL1,
+ PHY0_TESTDIN, ((command & 0xF00) >> 8));
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL0,
+ PHY0_TESTCLK, 1);
+
+ /* Write 8-bit testcode LSB */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL0,
+ PHY0_TESTCLK, 0);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL1,
+ PHY0_TESTEN, 1);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL0,
+ PHY0_TESTCLK, 1);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL1,
+ PHY0_TESTDIN, (command & 0xFF));
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL0,
+ PHY0_TESTCLK, 0);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL1,
+ PHY0_TESTEN, 0);
+
+ /* Write the data */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL1,
+ PHY0_TESTDIN, data);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL0,
+ PHY0_TESTCLK, 1);
+
+ udelay(1);
+}
+
+/* Bugfix for b/63578602 */
+static void mnh_mipi_gen3_lprxpon_wa(struct device *dev, u32 device, int enable)
+{
+ dev_dbg(dev, "%s: dev %d\n", __func__, device);
+ if (enable == 1) {
+ mnh_sm_mipi_rx_dphy_write_gen3(0x1AE, 0x07, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x1AD, 0xe0, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x305, 0x06, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x505, 0x06, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x705, 0x06, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x905, 0x06, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0xB05, 0x06, device);
+ } else {
+ mnh_sm_mipi_rx_dphy_write_gen3(0x1AE, 0x00, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x1AD, 0x00, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x305, 0x00, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x505, 0x00, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x705, 0x00, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x905, 0x00, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0xB05, 0x00, device);
+ }
+}
+
+static int mnh_mipi_gen3_lookup_freq_code(uint32_t rate)
+{
+ int i;
+
+ if ((rate < mipi_rate_configs[0].rate) ||
+ (rate > mipi_rate_configs[ARRAY_SIZE(mipi_rate_configs)-1].rate))
+ return -EINVAL;
+
+ for (i = 0; i < ARRAY_SIZE(mipi_rate_configs); i++) {
+ if (rate < mipi_rate_configs[i].rate)
+ return i - 1;
+ }
+
+ return i - 1;
+}
+
+static int mnh_mipi_gen3_lookup_device_cfg_idx(uint32_t rate)
+{
+ int i;
+
+ if ((rate < mipi_dev_ovr_cfgs[0].rate) ||
+ (rate > mipi_dev_ovr_cfgs[ARRAY_SIZE(mipi_dev_ovr_cfgs)-1].rate))
+ return -EINVAL;
+
+ for (i = 0; i < ARRAY_SIZE(mipi_dev_ovr_cfgs); i++) {
+ if (rate <= mipi_dev_ovr_cfgs[i].rate)
+ return i;
+ }
+
+ return i - 1;
+}
+
+static void mnh_mipi_gen3_host(struct device *dev, uint32_t device,
+ uint32_t rate)
+{
+ uint32_t code_index, freq_range_code, osc_freq_code;
+
+ /* only support devices 0-2 */
+ if (device > 2)
+ return;
+
+ dev_dbg(dev, "%s: dev %d, rate %d\n", __func__, device, rate);
+
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, N_LANES, N_LANES, 0x0);
+
+ /* clear interrupts */
+ HW_OUT(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, INT_MSK_PHY_FATAL,
+ 0xFFFFFFFF);
+ HW_OUT(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, INT_MSK_PKT_FATAL,
+ 0xFFFFFFFF);
+ HW_OUT(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, INT_MSK_FRAME_FATAL,
+ 0xFFFFFFFF);
+ HW_OUT(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, INT_MSK_PHY,
+ 0xFFFFFFFF);
+ HW_OUT(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, INT_MSK_PKT,
+ 0xFFFFFFFF);
+ HW_OUT(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, INT_MSK_LINE,
+ 0xFFFFFFFF);
+
+ /* enable clock to controller */
+ if (device == 0)
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, CSI_CLK_CTRL,
+ CSI2_RX0_CG, 0x0);
+ else if (device == 1)
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, CSI_CLK_CTRL,
+ CSI2_RX1_CG, 0x0);
+ else
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, CSI_CLK_CTRL,
+ CSI2_RX2_CG, 0x0);
+
+ /* disable the PHY before changing settings */
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_SHUTDOWNZ,
+ PHY_SHUTDOWNZ, 0x0);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, DPHY_RSTZ, DPHY_RSTZ,
+ 0x0);
+ HW_OUT(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL0, 0x1);
+ udelay(1);
+ HW_OUT(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_TEST_CTRL0, 0x0);
+
+ /* get the PHY settings */
+ code_index = mnh_mipi_gen3_lookup_freq_code(rate);
+ if (code_index < 0)
+ return;
+ freq_range_code = mipi_rate_configs[code_index].freq_range_code;
+ osc_freq_code = mipi_rate_configs[code_index].osc_freq_code;
+
+ /* update PHY configuration */
+ if (device == 0)
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, RX0_DPHY_CONFIG,
+ 0x8400 | freq_range_code);
+ else if (device == 1)
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, RX1_DPHY_CONFIG,
+ 0x8400 | freq_range_code);
+ else
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, RX2_DPHY_CONFIG,
+ 0x8400 | freq_range_code);
+
+ /* update PHY PLL settings */
+ mnh_sm_mipi_rx_dphy_write_gen3(0xe2, osc_freq_code & 0xFF, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0xe3, osc_freq_code >> 8, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0xe4, 0x01, device);
+ mnh_sm_mipi_rx_dphy_write_gen3(0x08, 0x20, device);
+ udelay(1);
+
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, N_LANES, N_LANES, 0x3);
+ udelay(1);
+
+ /* b/63578602 */
+ mnh_mipi_gen3_lprxpon_wa(dev, device, 1);
+
+ /* release reset */
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, PHY_SHUTDOWNZ,
+ PHY_SHUTDOWNZ, 0x1);
+ udelay(20);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, DPHY_RSTZ, DPHY_RSTZ,
+ 0x1);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(device), MIPI_RX, CSI2_RESETN,
+ CSI2_RESETN, 0x1);
+
+ udelay(30);
+
+ mnh_mipi_gen3_lprxpon_wa(dev, device, 0);
+}
+
+static uint8_t mnh_mipi_get_vco_cntrl(uint32_t rate)
+{
+ int i;
+ /* vco frequency is half the desired rate */
+ uint32_t freq_kHz = rate * 1000 / 2;
+
+ for (i = 0; i < ARRAY_SIZE(vco_range_cntrls); i++) {
+ if (freq_kHz < vco_range_cntrls[i].freq_kHz)
+ return vco_range_cntrls[i - 1].cntrl;
+ }
+
+ return vco_range_cntrls[i].cntrl;
+}
+
+/* M is required to be between 64 and 625 */
+/* N is required to be between 6 and 16 */
+static void mnh_mipi_get_pll_coeffs(uint32_t rate, uint16_t *m, uint8_t *n)
+{
+ uint32_t freq_kHz = rate * 1000 / 2; /* to not lose precision */
+ int vco_div_factor;
+ int est_ratio_scaled; /* M/N ratio scaled by 1000 to losing precision */
+ int iter_n, iter_m;
+ int best_error = INT_MAX;
+ int best_n = 0, best_m = 0;
+ int calc_ratio_scaled;
+
+ /* determine the vco division factor */
+ if (freq_kHz < 80000)
+ vco_div_factor = 8;
+ else if (freq_kHz < 160000)
+ vco_div_factor = 4;
+ else if (freq_kHz < 320000)
+ vco_div_factor = 2;
+ else
+ vco_div_factor = 1;
+
+ /* determine desired ratio, scaled by 1000 to avoid losing precision */
+ est_ratio_scaled = freq_kHz * 1000 * vco_div_factor / REF_FREQ_KHZ;
+
+ /* search through values of N and find ratio with least error */
+ for (iter_n = 4; iter_n <= 12; iter_n++) {
+ /* always round up so we don't get lower tx rate than we want */
+ iter_m = (est_ratio_scaled * iter_n + 999) / 1000;
+ calc_ratio_scaled = iter_m * 1000 / iter_n;
+
+ if (iter_m < 64)
+ continue;
+ else if (iter_m > 625)
+ break;
+
+ /* calculate the error given this M and N combo */
+ if ((calc_ratio_scaled - est_ratio_scaled) < best_error) {
+ best_n = iter_n;
+ best_m = iter_m;
+ best_error = calc_ratio_scaled - est_ratio_scaled;
+ }
+ }
+
+ /* assign the outputs */
+ *m = best_m;
+ *n = best_n;
+}
+
+static void mnh_mipi_gen3_device(struct device *dev, uint32_t device,
+ uint32_t rate)
+{
+ uint32_t code_index, freq_range_code, osc_freq_code;
+ struct mipi_dev_ovr_cfg *dev_ovr_cfg;
+ uint32_t mipi_dev_cfg_index;
+ unsigned long data;
+ uint8_t vco_cntrl;
+ uint16_t pll_m;
+ uint8_t pll_n;
+ int i = 0;
+
+ dev_dbg(dev, "%s: dev %d, rate %d\n", __func__, device, rate);
+
+ if (!mipi_debug) {
+ if (rate <= 360) {
+ dev_dbg(dev, "%s: %d config. default 360\n",
+ __func__, rate);
+ rate = 360;
+ } else if (rate <= 550) { /* debug mode */
+ dev_dbg(dev, "%s: %d config. default 550\n",
+ __func__, rate);
+ rate = 550;
+ } else if (rate <= 685) { /* front cam photo */
+ dev_dbg(dev, "%s: %d config. default 685\n",
+ __func__, rate);
+ rate = 685;
+ } else if (rate <= 720) { /* main cam 60fps */
+ dev_dbg(dev, "%s: %d config. default 720\n",
+ __func__, rate);
+ rate = 720;
+ } else if (rate <= 984) { /* main cam 30fps */
+ dev_dbg(dev, "%s: %d config. default 984\n",
+ __func__, rate);
+ rate = 984;
+ } else if (rate <= 1104) { /* main cam slo.mo. 120fps */
+ dev_dbg(dev, "%s: %d config. default 1104\n",
+ __func__, rate);
+ rate = 1104;
+ } else if (rate <= 1200) {
+ dev_dbg(dev, "%s: %d config. default 1200\n",
+ __func__, rate);
+ rate = 1200;
+ } else if (rate <= 1368) {
+ dev_dbg(dev, "%s: %d config. default 1368\n",
+ __func__, rate);
+ rate = 1368;
+ } else if (rate <= 1850) {
+ dev_dbg(dev, "%s: %d config. default 1850\n",
+ __func__, rate);
+ rate = 1850;
+ }
+ }
+
+ /* only support devices 0-1 */
+ if (device > 1)
+ return;
+
+ if (device == 0)
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, CSI_CLK_CTRL,
+ CSI2_TX0_CG, 0x0);
+ else
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, CSI_CLK_CTRL,
+ CSI2_TX1_CG, 0x0);
+
+ /* mipicsi_device_hw_init */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_RSTZ,
+ PHY_SHUTDOWNZ, 0x1);
+
+ /* mipicsi_device_dphy_reset */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_RSTZ, PHY_RSTZ,
+ 0x1);
+ udelay(1000);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_RSTZ, PHY_RSTZ,
+ 0x0);
+ HW_OUT(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, CSI2_RESETN, 0x0);
+ udelay(1);
+ HW_OUT(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, CSI2_RESETN, 0x1);
+ HW_OUT(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, INT_MASK_N_VPG,
+ 0xFFFFFFFF);
+ HW_OUT(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, INT_MASK_N_IDI,
+ 0xFFFFFFFF);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_RSTZ,
+ PHY_SHUTDOWNZ,
+ 0x0);
+
+ /* disable clock gating */
+ if (device == 0) {
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, TX0_DPHY_PLL_CNTRL,
+ PLL_SHADOW_CONTROL, 0x1);
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, TX0_DPHY_PLL_CNTRL,
+ CLK_SEL, 0x1);
+ } else {
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, TX1_DPHY_PLL_CNTRL,
+ PLL_SHADOW_CONTROL, 0x1);
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, TX1_DPHY_PLL_CNTRL,
+ CLK_SEL, 0x1);
+ }
+
+ /* disable the PHY before changing settings */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, CSI2_RESETN,
+ CSI2_RESETN_RW, 1);
+ HW_OUT(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_RSTZ, 0x0);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_RSTZ,
+ PHY_ENABLECLK, 1);
+ HW_OUT(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL0, 0x1);
+ udelay(1);
+ HW_OUT(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY0_TST_CTRL0, 0x0);
+
+ /* get the PHY settings */
+ code_index = mnh_mipi_gen3_lookup_freq_code(rate);
+ if (code_index < 0)
+ return;
+ freq_range_code = mipi_rate_configs[code_index].freq_range_code;
+ osc_freq_code = mipi_rate_configs[code_index].osc_freq_code;
+
+ /* update PHY configuration */
+ if (device == 0)
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, TX0_DPHY_CONFIG,
+ 0x8400 | freq_range_code);
+ else
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, TX1_DPHY_CONFIG,
+ 0x8400 | freq_range_code);
+
+ /* configure slew rate calibration */
+ if (rate > 1500) {
+ mnh_sm_mipi_tx_dphy_write_gen3(0x26B, 0x44, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x272, 0x00, device);
+ } else if (rate > 1000) {
+ mnh_sm_mipi_tx_dphy_write_gen3(0x270, 0xD0, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x271, 0x07, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x272, 0x10, device);
+ } else if (rate > 500) {
+ mnh_sm_mipi_tx_dphy_write_gen3(0x270, 0xE2, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x271, 0x04, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x272, 0x11, device);
+ } else {
+ mnh_sm_mipi_tx_dphy_write_gen3(0x270, 0x84, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x271, 0x03, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x272, 0x11, device);
+ }
+
+ /* get PLL coefficients and codes */
+ vco_cntrl = mnh_mipi_get_vco_cntrl(rate);
+ mnh_mipi_get_pll_coeffs(rate, &pll_m, &pll_n);
+
+ /* mipi device configuration settings from lookup table */
+ mipi_dev_cfg_index = mnh_mipi_gen3_lookup_device_cfg_idx(rate);
+ if (mipi_dev_cfg_index < 0)
+ return;
+ dev_ovr_cfg = &mipi_dev_ovr_cfgs[mipi_dev_cfg_index];
+ dev_dbg(dev, "%s: Device configuration index: %d\n", __func__,
+ mipi_dev_cfg_index);
+
+ /* adjust the values to meet the register definition */
+ pll_m -= 2;
+ pll_n -= 1;
+
+ /* configure PLL frequency multiplication and division factors */
+ mnh_sm_mipi_tx_dphy_write_gen3(0x17B, ((vco_cntrl & 0x3F) << 1) | 0x81,
+ device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x178, 0x80 | ((pll_n & 0xF) << 3),
+ device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x179, pll_m & 0xFF, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x17A, (pll_m >> 8) & 0x3, device);
+
+ /* configure rate-independent PLL settings */
+ mnh_sm_mipi_tx_dphy_write_gen3(0x15E, 0x10, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x162, 0x04, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x16E, 0x0C, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x173, 0x02, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x174, 0x00, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x175, 0x60, device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x176, 0x03, device);
+
+ /* TODO: Maybe do a read-modify-write */
+ if (rate <= 450)
+ mnh_sm_mipi_tx_dphy_write_gen3(0x1AC, 1 << 4, device);
+
+ /* Wait for 15ns */
+ udelay(1);
+
+ /* Timing register overrides */
+ if (!mipi_debug && dev_ovr_cfg->use_ovrd) {
+ mnh_sm_mipi_tx_dphy_write_gen3(0x5A, dev_ovr_cfg->reg_0x5A,
+ device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x5B, dev_ovr_cfg->reg_0x5B,
+ device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x5C, dev_ovr_cfg->reg_0x5C,
+ device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x5D, dev_ovr_cfg->reg_0x5D,
+ device);
+ /*
+ * mnh_sm_mipi_tx_dphy_write_gen3(0x5E, dev_ovr_cfg->reg_0x5E,
+ * device);
+ */
+ mnh_sm_mipi_tx_dphy_write_gen3(0x5F, dev_ovr_cfg->reg_0x5F,
+ device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x61, dev_ovr_cfg->reg_0x61,
+ device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x62, dev_ovr_cfg->reg_0x62,
+ device);
+ mnh_sm_mipi_tx_dphy_write_gen3(0x63, dev_ovr_cfg->reg_0x63,
+ device);
+ /*
+ * mnh_sm_mipi_tx_dphy_write_gen3(0x64, dev_ovr_cfg->reg_0x64,
+ * device);
+ */
+ mnh_sm_mipi_tx_dphy_write_gen3(0x65, dev_ovr_cfg->reg_0x65,
+ device);
+ }
+
+ /* Enable high speed drivers (required for Tlpx < 500ns) */
+ /* Clock lane driver */
+ mnh_sm_mipi_tx_dphy_write_gen3(0x304, (1 << 2) | (1 << 3), device);
+ /* Data lane 0 driver */
+ mnh_sm_mipi_tx_dphy_write_gen3(0x504, (1 << 2) | (1 << 3), device);
+ /* Data lane 1 driver */
+ mnh_sm_mipi_tx_dphy_write_gen3(0x704, (1 << 2) | (1 << 3), device);
+ /* Data lane 2 driver */
+ mnh_sm_mipi_tx_dphy_write_gen3(0x904, (1 << 2) | (1 << 3), device);
+ /* Data lane 3 driver */
+ mnh_sm_mipi_tx_dphy_write_gen3(0xB04, (1 << 2) | (1 << 3), device);
+
+ /* Enable lanes */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_IF_CFG,
+ LANE_EN_NUM, 3);
+
+ /* Table A-4 High-Speed Transition Times: Data HS-LP */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_IF_CFG,
+ PHY_STOP_WAIT_TIME, dev_ovr_cfg->wait_time);
+
+ /* enable the PHY */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_RSTZ,
+ PHY_ENABLECLK, 1);
+ udelay(1);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_RSTZ,
+ PHY_SHUTDOWNZ, 1);
+ udelay(1);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX, PHY_RSTZ, PHY_RSTZ, 1);
+
+ /* wait for data & clk lanes to reach the low-power state */
+ i = 0;
+ do {
+ data = HW_IN(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX,
+ PHY_STATUS);
+ udelay(10);
+ i++;
+ } while ((i < 40) && ((data & 0x1550) != 0x1550));
+
+ if ((i >= 100) && ((data & 0x1550) != 0x1550))
+ dev_err(dev,
+ "device %d could not drive lp state, status 0x%lx\n",
+ device, data);
+
+ /* INIT period in continuous clock mode is software controlled */
+ udelay(1000); /* relevant timing parameter, do not remove */
+
+ /* Enable ct clock, this takes immediate effect */
+ dev_dbg(dev, "%s: enabling ct. clock mode\n", __func__);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(device), MIPI_TX,
+ LPCLK_CTRL, PHY_TXREQCLKHS_CON, 1);
+}
+
+static int mnh_mipi_config_mux_sel(struct device *dev,
+ struct mnh_mipi_config *config)
+{
+
+ uint32_t rxdev, txdev, mode;
+ uint32_t vc_en_mask, byp_tx_en_mask, ipu_en_mask;
+ uint32_t tx_func_mask, tx_byp_sel_mask;
+ uint32_t rx_mode, tx_mode;
+
+ rxdev = config->rxdev;
+ txdev = config->txdev;
+ vc_en_mask = config->vc_en_mask;
+ mode = config->mode;
+
+ dev_dbg(dev, "%s: MUX rxdev %d, txdev %d, vc_en_mask 0x%1x, mode %d\n",
+ __func__, rxdev, txdev, vc_en_mask, mode);
+
+ /*
+ * 0. Upper layers should issue a MUX reset prior to configuring
+ * the MUX
+ * 1. Configure RX section of the MUX unless rxdev is MIPI_RX_IPU
+ * in that case we are in functional mode IPU -> Tx[n] and only
+ * the tx_mode register must be changed
+ * 2. Configure TX section of the MUX unless txdev is MIPI_TX_IPU
+ * in that case we are in functional mode Rx[n] -> IPU and only
+ * the rx_mode register must be changed
+ */
+
+ /*
+ * construct the rx mode register. This register is synchronized with
+ * the IDI domain when the RX clock is present. However, if the clock is
+ * not present, only the first write to the register will succeed.
+ * Therefore, use read-modify-write to construct register so we only
+ * need to write once.
+ */
+ if (rxdev == MIPI_RX0)
+ rx_mode = HW_IN(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, RX0_MODE);
+ else if (rxdev == MIPI_RX1)
+ rx_mode = HW_IN(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, RX1_MODE);
+ else if (rxdev == MIPI_RX2)
+ rx_mode = HW_IN(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, RX2_MODE);
+ else
+ rx_mode = 0; /* rx_mode register not required for IPU source */
+ /*
+ * it doesn't actually matter which RX# is used since this is just to
+ * create the mask. The actual register write occurs below.
+ */
+ rx_mode &= ~HWIO_MIPI_TOP_RX0_MODE_RX0_VC_EN_FLDMASK;
+ rx_mode &= ~HWIO_MIPI_TOP_RX0_MODE_RX0_BYP_TX0_EN_FLDMASK;
+ rx_mode &= ~HWIO_MIPI_TOP_RX0_MODE_RX0_BYP_TX1_EN_FLDMASK;
+ rx_mode &= ~HWIO_MIPI_TOP_RX0_MODE_RX0_IPU_EN_FLDMASK;
+
+ vc_en_mask <<= HWIO_MIPI_TOP_RX0_MODE_RX0_VC_EN_FLDSHFT;
+ vc_en_mask &= HWIO_MIPI_TOP_RX0_MODE_RX0_VC_EN_FLDMASK;
+
+ /* Bypass mode to one of the two Tx channels */
+ if (mode == MIPI_MODE_BYPASS) { /* no IPU, TXn_EN */
+ byp_tx_en_mask =
+ HWIO_MIPI_TOP_RX0_MODE_RX0_BYP_TX0_EN_FLDMASK << txdev;
+ ipu_en_mask = 0;
+ tx_func_mask = 0;
+ tx_byp_sel_mask = (1 + rxdev)
+ << HWIO_MIPI_TOP_TX0_MODE_TX0_BYP_SEL_FLDSHFT;
+ /* Bypass mode with IPU sink enabled */
+ } else if (mode == MIPI_MODE_BYPASS_W_IPU) {
+ byp_tx_en_mask =
+ HWIO_MIPI_TOP_RX0_MODE_RX0_BYP_TX0_EN_FLDMASK << txdev;
+ ipu_en_mask = 1 << HWIO_MIPI_TOP_RX0_MODE_RX0_IPU_EN_FLDSHFT;
+ tx_func_mask = 0;
+ tx_byp_sel_mask = (1 + rxdev)
+ << HWIO_MIPI_TOP_TX0_MODE_TX0_BYP_SEL_FLDSHFT;
+ /* Functional mode, direct bypass between Rx & TX disabled */
+ } else if (mode == MIPI_MODE_FUNCTIONAL) {
+ byp_tx_en_mask = 0;
+ ipu_en_mask = 1 << HWIO_MIPI_TOP_RX0_MODE_RX0_IPU_EN_FLDSHFT;
+ tx_func_mask = 1 << HWIO_MIPI_TOP_TX0_MODE_TX0_FUNC_FLDSHFT;
+ tx_byp_sel_mask = 0; /* do not care */
+ } else {
+ return -EINVAL;
+ }
+
+ rx_mode = (rx_mode | vc_en_mask | byp_tx_en_mask | ipu_en_mask);
+ tx_mode = (tx_func_mask | tx_byp_sel_mask);
+ dev_dbg(dev, "%s: rx_mode=%d | tx_mode=%d\n", __func__,
+ rx_mode, tx_mode);
+
+ switch (txdev) { /* Sink */
+ case MIPI_TX0:
+ switch (rxdev) { /* Source */
+ case MIPI_RX0: /* RX0 -> TX0 */
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, RX0_MODE, rx_mode);
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, TX0_MODE, tx_mode);
+ break;
+ case MIPI_RX1: /* RX1 -> TX0 */
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, RX1_MODE, rx_mode);
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, TX0_MODE, tx_mode);
+ break;
+ case MIPI_RX2: /* RX2 -> TX0 */
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, RX2_MODE, rx_mode);
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, TX0_MODE, tx_mode);
+ break;
+ case MIPI_RX_IPU: /* IPU -> TX0 */
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, TX0_MODE, tx_mode);
+ break;
+ default:
+ dev_err(dev, "%s: invalid rx device %d!\n", __func__,
+ rxdev);
+ return -EINVAL;
+ }
+ break;
+ case MIPI_TX1:
+ switch (rxdev) {
+ case MIPI_RX0: /* RX0 -> TX1 */
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, RX0_MODE, rx_mode);
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, TX1_MODE, tx_mode);
+ break;
+ case MIPI_RX1: /* RX1 -> TX1 */
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, RX1_MODE, rx_mode);
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, TX1_MODE, tx_mode);
+ break;
+ case MIPI_RX2: /* RX2 -> TX1 */
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, RX2_MODE, rx_mode);
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, TX1_MODE, tx_mode);
+ break;
+ case MIPI_RX_IPU:
+ /* Configure IPU -> Tx1 */
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, TX1_MODE, tx_mode);
+ break;
+ default:
+ dev_err(dev, "%s: invalid rx device %d!\n", __func__,
+ rxdev);
+ return -EINVAL;
+ }
+ break;
+ case MIPI_TX_IPU:
+ switch (rxdev) {
+ case MIPI_RX0: /* Rx0 -> IPU */
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, RX0_MODE, rx_mode);
+ break;
+ case MIPI_RX1:
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, RX1_MODE, rx_mode);
+ break;
+ case MIPI_RX2:
+ HW_OUT(HWIO_MIPI_TOP_BASE_ADDR,
+ MIPI_TOP, RX2_MODE, rx_mode);
+ break;
+ default:
+ dev_err(dev, "%s: invalid rx device %d!\n", __func__,
+ rxdev);
+ return -EINVAL;
+ }
+ break;
+ default:
+ dev_err(dev, "%s: invalid tx device %d!\n", __func__, txdev);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+int mnh_mipi_config(struct device *dev, struct mnh_mipi_config config)
+{
+ uint32_t txdev = config.txdev;
+ uint32_t rxdev = config.rxdev;
+ uint32_t rx_rate = config.rx_rate;
+ uint32_t tx_rate = config.tx_rate;
+ uint32_t vc_en_mask = config.vc_en_mask;
+
+ dev_dbg(dev, "%s: init rxdev %d, txdev %d, rx_rate %d, tx_rate %d, vc_en_mask 0x%1x\n",
+ __func__, rxdev, txdev, rx_rate, tx_rate, vc_en_mask);
+
+ /* configure rx / MIPI-source */
+ switch (rxdev) {
+ case MIPI_RX0:
+ case MIPI_RX1:
+ case MIPI_RX2:
+ dev_dbg(dev, "%s: configuring host controller Rx%d\n",
+ __func__, rxdev);
+ mnh_mipi_gen3_host(dev, rxdev, rx_rate);
+ break;
+ case MIPI_RX_IPU:
+ dev_dbg(dev, "%s: configuring IPU IDI Tx%d as MIPI source\n",
+ __func__, txdev);
+ break;
+ default:
+ dev_dbg(dev, "%s: Invalid MIPI input device\n", __func__);
+ break;
+ }
+
+ /* configure tx / MIPI-sink */
+ switch (txdev) {
+ case MIPI_TX0:
+ case MIPI_TX1:
+ dev_dbg(dev, "%s: configuring device controller Tx%d\n",
+ __func__, txdev);
+ mnh_mipi_gen3_device(dev, txdev, tx_rate);
+ break;
+ case MIPI_TX_IPU:
+ dev_dbg(dev, "%s: configuring IPU IDI Rx%d as MIPI sink\n",
+ __func__, rxdev);
+ break;
+ default:
+ dev_dbg(dev, "%s: Invalid MIPI output device\n", __func__);
+ break;
+ }
+
+ /* configure mux select */
+ mnh_mipi_config_mux_sel(dev, &config);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_mipi_config);
+
+int mnh_mipi_stop(struct device *dev, struct mnh_mipi_config config)
+{
+ uint32_t txdev = config.txdev;
+ uint32_t rxdev = config.rxdev;
+
+ dev_dbg(dev, "%s: stopping rxdev %d, txdev %d\n", __func__, rxdev,
+ txdev);
+
+ /* Shutdown host */
+ mnh_mipi_stop_host(dev, rxdev);
+
+ /* Shutdown device */
+ mnh_mipi_stop_device(dev, txdev);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_mipi_stop);
+
+void mnh_mipi_stop_device(struct device *dev, int txdev)
+{
+ /* shut down the PHY */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(txdev), MIPI_TX, PHY_RSTZ,
+ PHY_RSTZ, 0);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(txdev), MIPI_TX, PHY_RSTZ,
+ PHY_SHUTDOWNZ, 0);
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(txdev), MIPI_TX, PHY_RSTZ,
+ PHY_ENABLECLK, 0);
+ /* shut down the controller logic */
+ HW_OUTf(HWIO_MIPI_TX_BASE_ADDR(txdev), MIPI_TX, CSI2_RESETN,
+ CSI2_RESETN_RW, 0);
+ /* enable clock gating (disable clock) */
+ switch (txdev) {
+ case MIPI_TX0:
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, CSI_CLK_CTRL,
+ CSI2_TX0_CG, 0x1);
+ break;
+ case MIPI_TX1:
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, CSI_CLK_CTRL,
+ CSI2_TX1_CG, 0x1);
+ break;
+ default:
+ dev_err(dev, "%s invalid mipi device!\n", __func__);
+ break;
+ }
+}
+EXPORT_SYMBOL_GPL(mnh_mipi_stop_device);
+
+void mnh_mipi_stop_host(struct device *dev, int rxdev)
+{
+ /* shut down the PHY */
+ /* see 7.3.1: Rx-DPHY databook */
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(rxdev), MIPI_RX, PHY_SHUTDOWNZ,
+ PHY_SHUTDOWNZ, 0x0);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(rxdev), MIPI_RX, DPHY_RSTZ,
+ DPHY_RSTZ, 0x0);
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(rxdev), MIPI_RX, PHY_TEST_CTRL0,
+ PHY_TESTCLR, 0x1);
+ /* shut down the controller logic */
+ HW_OUTf(HWIO_MIPI_RX_BASE_ADDR(rxdev), MIPI_RX, CSI2_RESETN,
+ CSI2_RESETN, 0x0);
+ /* enable clock gating (disable clock) */
+ switch (rxdev) {
+ case MIPI_RX0:
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, CSI_CLK_CTRL,
+ CSI2_RX0_CG, 0x1);
+ break;
+ case MIPI_RX1:
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, CSI_CLK_CTRL,
+ CSI2_RX1_CG, 0x1);
+ break;
+ case MIPI_RX2:
+ HW_OUTf(HWIO_MIPI_TOP_BASE_ADDR, MIPI_TOP, CSI_CLK_CTRL,
+ CSI2_RX2_CG, 0x1);
+ break;
+ default:
+ dev_err(dev, "%s invalid mipi host!\n", __func__);
+ break;
+ }
+}
+EXPORT_SYMBOL_GPL(mnh_mipi_stop_host);
+
+void mnh_mipi_set_debug(int val)
+{
+ mipi_debug = val;
+}
+EXPORT_SYMBOL(mnh_mipi_set_debug);
+
+int mnh_mipi_get_top_interrupts(struct device *dev,
+ struct mipi_device_irq_st *int_status)
+{
+ u32 status = 0;
+
+ switch (int_status->dev) {
+ case 0:
+ status = TOP_IN(TX0_BYPINT);
+ if (status & TOP_MASK(TX0_BYPINT, TX0_BYP_OF)) {
+ TOP_OUTf(TX0_BYPINT, TX0_BYP_OF, 1);
+ dev_info(dev, "TX0_BYPINT BYP_OF occurred\n");
+ int_status->fifo_overflow = 1;
+ }
+ break;
+ case 1:
+ status = TOP_IN(TX1_BYPINT);
+ if (status & TOP_MASK(TX1_BYPINT, TX1_BYP_OF)) {
+ TOP_OUTf(TX1_BYPINT, TX1_BYP_OF, 1);
+ dev_info(dev, "TX1_BYPINT BYP_OF occurred\n");
+ int_status->fifo_overflow = 1;
+ }
+ break;
+ default:
+ break;
+ }
+ return status;
+}
+
+int mnh_mipi_get_device_interrupts(struct device *dev,
+ struct mipi_device_irq_st *int_status)
+{
+ uint32_t baddr = HWIO_MIPI_TX_BASE_ADDR(int_status->dev);
+
+ int_status->main = TX_IN(INT_ST_MAIN);
+ dev_info(dev, "MIPI device controller %d interrupt main: %x\n",
+ int_status->dev, int_status->main);
+
+ if (int_status->main & TX_MASK(INT_ST_MAIN, INT_ST_VPG)) {
+ int_status->vpg = TX_IN(INT_ST_VPG);
+ dev_info(dev, "CSI INT_ST_VPG: %x\n", int_status->vpg);
+ }
+ if (int_status->main & TX_MASK(INT_ST_MAIN, INT_ST_IDI)) {
+ int_status->idi = TX_IN(INT_ST_IDI);
+ dev_info(dev, "CSI INT_ST_IDI: %x\n", int_status->idi);
+ }
+ if (int_status->main & TX_MASK(INT_ST_MAIN, INT_ST_PHY)) {
+ int_status->phy = TX_IN(INT_ST_PHY);
+ dev_info(dev, "CSI INT_ST_PHY: %x\n", int_status->phy);
+ }
+ mnh_mipi_get_top_interrupts(dev, int_status);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_mipi_get_device_interrupts);
+
+int mnh_mipi_get_host_interrupts(struct device *dev,
+ struct mipi_host_irq_st *int_status)
+{
+
+ uint32_t baddr = HWIO_MIPI_RX_BASE_ADDR(int_status->dev);
+
+ int_status->main = RX_IN(INT_ST_MAIN);
+ dev_info(dev, "MIPI host controller %d interrupt main: %x\n",
+ int_status->dev, int_status->main);
+
+ if (int_status->main & RX_MASK(INT_ST_MAIN, STATUS_INT_PHY_FATAL)) {
+ int_status->phy_fatal = RX_IN(INT_ST_PHY_FATAL);
+ dev_info(dev, "CSI INT PHY FATAL: %x\n",
+ int_status->phy_fatal);
+ }
+
+ if (int_status->main & RX_MASK(INT_ST_MAIN, STATUS_INT_PKT_FATAL)) {
+ int_status->pkt_fatal = RX_IN(INT_ST_PKT_FATAL);
+ dev_info(dev, "CSI INT PKT FATAL: %x\n",
+ int_status->pkt_fatal);
+ }
+
+ if (int_status->main & RX_MASK(INT_ST_MAIN, STATUS_INT_FRAME_FATAL)) {
+ int_status->frame_fatal = RX_IN(INT_ST_FRAME_FATAL);
+ dev_info(dev, "CSI INT FRAME FATAL: %x\n",
+ int_status->frame_fatal);
+ }
+
+ if (int_status->main & RX_MASK(INT_ST_MAIN, STATUS_INT_PHY)) {
+ int_status->phy = RX_IN(INT_ST_PHY);
+ dev_info(dev, "CSI INT PHY: %x\n", int_status->phy);
+ }
+
+ if (int_status->main & RX_MASK(INT_ST_MAIN, STATUS_INT_PKT)) {
+ int_status->pkt = RX_IN(INT_ST_PKT);
+ dev_info(dev, "CSI INT PKT: %x\n", int_status->pkt);
+ }
+
+ if (int_status->main & RX_MASK(INT_ST_MAIN, STATUS_INT_LINE)) {
+ int_status->line = RX_IN(INT_ST_LINE);
+ dev_info(dev, "CSI INT LINE: %x\n", int_status->line);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_mipi_get_host_interrupts);
diff --git a/drivers/misc/mnh/mnh-mipi.h b/drivers/misc/mnh/mnh-mipi.h
new file mode 100644
index 0000000..0a3e87d
--- /dev/null
+++ b/drivers/misc/mnh/mnh-mipi.h
@@ -0,0 +1,65 @@
+/*
+ *
+ * MNH State Manager MIPI Driver
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef __MNH_MIPI
+#define __MNH_MIPI
+
+#include "mnh-sm.h"
+
+/* Interrupt status registers, R/C=read-clear, W/C=write-clear */
+struct mipi_device_irq_st {
+ int dev;
+ /* status of individual interrupt sources (R/C) */
+ uint32_t main;
+ /* interrupt group caused by video pattern gen. (R/C) */
+ uint32_t vpg;
+ /* interrupt group caused by IDI interface (R/C) */
+ uint32_t idi;
+ /* interrupt group caused by the PHY (R/C) */
+ uint32_t phy;
+ /* buffer overflow indicator in the bypass FIFOs in MIPI_TOP (W/C=1) */
+ uint32_t fifo_overflow;
+};
+
+struct mipi_host_irq_st {
+ int dev;
+ /* status of individual interrupt sources (R/C) */
+ uint32_t main;
+ /* fatal interruption caused by PHY (R/C) */
+ uint32_t phy_fatal;
+ /* fatal interruption related to packet construction (R/C) */
+ uint32_t pkt_fatal;
+ /* fatal interruption related to frame construction (R/C) */
+ uint32_t frame_fatal;
+ /* interruption caused by PHY (R/C) */
+ uint32_t phy;
+ /* interruption related to packet construction (R/C) */
+ uint32_t pkt;
+ /* interruption related to line construction (R/C) */
+ uint32_t line;
+};
+
+int mnh_mipi_config(struct device *dev, struct mnh_mipi_config cfg);
+int mnh_mipi_stop(struct device *dev, struct mnh_mipi_config cfg);
+void mnh_mipi_stop_device(struct device *dev, int txdev);
+void mnh_mipi_stop_host(struct device *dev, int rxdev);
+void mnh_mipi_set_debug(int val);
+int mnh_mipi_get_device_interrupts(struct device *dev,
+ struct mipi_device_irq_st *int_status);
+int mnh_mipi_get_host_interrupts(struct device *dev,
+ struct mipi_host_irq_st *int_status);
+
+#endif /* __MNH_MIPI */
diff --git a/drivers/misc/mnh/mnh-pcie.c b/drivers/misc/mnh/mnh-pcie.c
new file mode 100644
index 0000000..a51553e
--- /dev/null
+++ b/drivers/misc/mnh/mnh-pcie.c
@@ -0,0 +1,2217 @@
+/*
+ *
+ * MNH PCIe/DMA HOST Driver
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+/* #define DEBUG */
+#include <asm/dma-iommu.h>
+#include <linux/device.h>
+#include <linux/dma-mapping.h>
+#include <linux/sched.h>
+#include <linux/errno.h>
+#include <linux/io.h>
+#include <linux/iommu.h>
+#include <linux/module.h>
+#include <linux/kernel.h>
+#include <linux/delay.h>
+#include <linux/types.h>
+#include <linux/pci.h>
+#include <linux/platform_device.h>
+#include <linux/interrupt.h>
+#include <linux/mm.h>
+#include <linux/rwsem.h>
+#include <linux/scatterlist.h>
+#include <linux/pagemap.h>
+#include <linux/fs.h>
+#include <linux/firmware.h>
+#include <linux/dma-buf.h>
+
+#include "mnh-pcie.h"
+#include "hw-mnh-regs.h"
+
+#define COMBINE_SG 1
+
+#define DEV_MINOR 0
+#define DEVICENO 1
+#define MODULE_NAME "mnh_pci"
+#define PCI_DEVICE_ID_MNH 0x3140
+
+#define BAR_0 0
+#define BAR_2 2
+#define BAR_4 4
+#define BAR_MAX_NUM 6
+
+#define SUCCESS 0
+#define INIT_DONE 0x1
+
+#define SET_BIT(val, bitIndex) (val |= (1 << bitIndex))
+#define CLEAR_BIT(val, bitIndex) (val &= ~(1 << bitIndex))
+#define BIT_IS_SET(val, bitIndex) (val & (1 << bitIndex))
+#define MIN(a, b) (((a) < (b)) ? (a) : (b))
+
+
+#define UPPER(address) ((unsigned int)((address & 0xFFFFFFFF00000000) >> 32))
+#define LOWER(address) ((unsigned int)(address & 0x00000000FFFFFFFF))
+
+#define IS_NULL(ptr) ((ptr == NULL)?1:0)
+
+/* mnh_pci_tbl - PCI Device ID Table */
+static const struct pci_device_id mnh_pci_tbl[] = {
+ {PCI_DEVICE(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_MNH)},
+ {0, } /* terminate the list */
+};
+MODULE_DEVICE_TABLE(pci, mnh_pci_tbl);
+
+struct mnh_device {
+ struct pci_dev *pdev;
+ void __iomem *pcie_config;
+ void __iomem *config;
+ void __iomem *ddr;
+ irq_cb_t msg_cb;
+ irq_cb_t vm_cb;
+ irq_dma_cb_t dma_cb;
+ uint32_t ring_buf_addr;
+ uint32_t msi_num;
+ int irq;
+ uint8_t bar2_iatu_region; /* 0 - whole BAR2
+ 1 - ROM/SRAM
+ 2 - Peripheral Config */
+ uint32_t bar_size[BAR_MAX_NUM];
+ bool powered;
+};
+
+static struct mnh_device *mnh_dev;
+
+static void mnh_check_pci_resources(struct pci_dev *dev, int bar);
+static uint32_t mnh_check_iatu_bar2(uint32_t offset);
+
+void *mnh_alloc_coherent(size_t size, dma_addr_t *dma_addr)
+{
+ return dma_alloc_coherent(
+ &mnh_dev->pdev->dev, size, dma_addr, GFP_KERNEL);
+}
+EXPORT_SYMBOL(mnh_alloc_coherent);
+
+void mnh_free_coherent(size_t size, void *cpu_addr, dma_addr_t dma_addr)
+{
+ dma_free_coherent(
+ &mnh_dev->pdev->dev, size, cpu_addr, dma_addr);
+}
+EXPORT_SYMBOL(mnh_free_coherent);
+
+/**
+ * Map host memory for access by MNH PCIe host
+ * @param[in] cpu_addr cpu virtual address of the memory region
+ * @param[in] size size of the memory region in bytes
+ * @param[in] direction DMA direction DMA_TO_DEVICE, etc.
+ * @return DMA address returned by dma_map_single(), or zero for error
+ */
+dma_addr_t mnh_map_mem(
+ void *cpu_addr, size_t size, enum dma_data_direction direction)
+{
+ dma_addr_t dma_addr =
+ dma_map_single(&mnh_dev->pdev->dev, cpu_addr, size, direction);
+ if (dma_mapping_error(&mnh_dev->pdev->dev, dma_addr))
+ return 0;
+ return dma_addr;
+}
+EXPORT_SYMBOL(mnh_map_mem);
+
+/**
+ * Unmap host memory from MNH PCIe host access
+ * @param[in] dma_addr DMA address of the memory returned by mnh_map_mem()
+ * @param[in] size size of the memory region in bytes
+ * @param[in] direction DMA direction DMA_TO_DEVICE, etc.
+ */
+void mnh_unmap_mem(
+ dma_addr_t dma_addr, size_t size, enum dma_data_direction direction)
+{
+ dma_unmap_single(&mnh_dev->pdev->dev, dma_addr, size, direction);
+}
+EXPORT_SYMBOL(mnh_unmap_mem);
+
+/**
+ * API to read data from PCIE configuration space
+ * @param[in] offset offset into PCIE configuration space(BAR0)
+ * @param[in] len buffer size : supported size is 4
+ * @param[in] data data to be read into. Client must allocate
+ * the buffer for reading and pass the pointer
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ * an error happens during the process.(mnh_dma_chan_status_t)
+ */
+int mnh_pcie_config_read(uint32_t offset, uint32_t len, uint32_t *data)
+{
+
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ if (mnh_dev->pdev->current_state != PCI_D0)
+ return -EIO;
+
+ if (len != sizeof(uint32_t))
+ return -EINVAL; /* only 32bit access is supported */
+
+ if (offset > mnh_dev->bar_size[BAR_0] - sizeof(uint32_t))
+ return -EINVAL; /* address invalid */
+
+ if (WARN_ON(!mnh_dev->powered)) {
+ dev_err(&mnh_dev->pdev->dev,
+ "%s: cannot do pcie transfer while powering down\n",
+ __func__);
+ return -EIO;
+ }
+
+ pci_read_config_dword(mnh_dev->pdev, offset, data);
+
+ dev_dbg(&mnh_dev->pdev->dev, "Read PCIE Config [0x%08x]-0x%x\n",
+ offset, *data);
+
+ if (*data != 0xffffffff)
+ return 0;
+ else
+ return -EIO;
+}
+EXPORT_SYMBOL(mnh_pcie_config_read);
+
+/**
+ * API to write data to PCIE configuration space
+ * @param[in] offset offset into PCIE configuration space(BAR0)
+ * @param[in] len buffer size : supported size is 4
+ * @param[in] data data to be written
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_pcie_config_write(uint32_t offset, uint32_t len, uint32_t data)
+{
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ if (mnh_dev->pdev->current_state != PCI_D0)
+ return -EIO;
+
+ if (len != sizeof(uint32_t))
+ return -EINVAL; /* only 32bit access is supported */
+
+ if (offset > mnh_dev->bar_size[BAR_0] - sizeof(uint32_t))
+ return -EINVAL; /* address invalid */
+
+ dev_dbg(&mnh_dev->pdev->dev, "Write PCIE Config[0x%08x]-0x%x",
+ offset, data);
+
+ if (WARN_ON(!mnh_dev->powered)) {
+ dev_err(&mnh_dev->pdev->dev,
+ "%s: cannot do pcie transfer while powering down\n",
+ __func__);
+ return -EIO;
+ }
+
+ pci_write_config_dword(mnh_dev->pdev, offset, data);
+
+ return 0;
+
+}
+EXPORT_SYMBOL(mnh_pcie_config_write);
+
+/**
+ * API to read data from MNH configuration space
+ * @param[in] offset offset into MNH Address space(BAR2)
+ * @param[in] len buffer size : supported size is 1,2,4
+ * @param[in] data data to be read into. Client must allocate
+ * the buffer for reading and pass the pointer
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_config_read(uint32_t offset, uint32_t len, uint32_t *data)
+{
+ uint32_t new_offset;
+
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ if (mnh_dev->pdev->current_state != PCI_D0)
+ return -EIO;
+
+ if (offset > HW_MNH_PCIE_BAR_2_ADDR_END - len) {
+ dev_err(&mnh_dev->pdev->dev, "Addr Invalid: %x", offset);
+ return -EINVAL; /* address invalid */
+ }
+
+ new_offset = mnh_check_iatu_bar2(offset);
+
+ if (WARN_ON(!mnh_dev->powered)) {
+ dev_err(&mnh_dev->pdev->dev,
+ "%s: cannot do pcie transfer while powering down\n",
+ __func__);
+ return -EIO;
+ }
+
+ if (len == sizeof(uint32_t))
+ *data = ioread32(mnh_dev->config + new_offset);
+ else if (len == sizeof(uint16_t))
+ *data = ioread16(mnh_dev->config + new_offset);
+ else if (len == sizeof(uint8_t))
+ *data = ioread8(mnh_dev->config + new_offset);
+ else {
+ dev_err(&mnh_dev->pdev->dev, "%s: invalid len %d\n",
+ __func__, len);
+ return -EINVAL;
+ }
+
+ dev_dbg(&mnh_dev->pdev->dev, "Read Config[0x%08x] - 0x%x",
+ new_offset, *data);
+
+ if (*data != 0xffffffff)
+ return 0;
+ else
+ return -EIO;
+}
+EXPORT_SYMBOL(mnh_config_read);
+
+/**
+ * API to write data to MNH configuration space
+ * @param[in] offset offset into MNH Address space(BAR2)
+ * @param[in] len buffer size : supported size could be 4
+ * @param[in] data pointer to the data to be wrriten.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_config_write(uint32_t offset, uint32_t len, uint32_t data)
+{
+ uint32_t new_offset;
+
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ if (mnh_dev->pdev->current_state != PCI_D0)
+ return -EIO;
+
+ if (offset > HW_MNH_PCIE_BAR_2_ADDR_END - len)
+ return -EINVAL; /* address invalid */
+
+ new_offset = mnh_check_iatu_bar2(offset);
+
+ dev_dbg(&mnh_dev->pdev->dev, "Write Config[0x%08x] - 0x%x",
+ new_offset, data);
+
+ if (WARN_ON(!mnh_dev->powered)) {
+ dev_err(&mnh_dev->pdev->dev,
+ "%s: cannot do pcie transfer while powering down\n",
+ __func__);
+ return -EIO;
+ }
+
+ if (len == sizeof(uint32_t))
+ iowrite32(data, mnh_dev->config + new_offset);
+ else if (len == sizeof(uint16_t))
+ iowrite16(data, mnh_dev->config + new_offset);
+ else if (len == sizeof(uint8_t))
+ iowrite8(data, mnh_dev->config + new_offset);
+ else {
+ dev_err(&mnh_dev->pdev->dev, "%s: invalid len %d\n",
+ __func__, len);
+ return -EINVAL;
+ }
+
+ return 0;
+
+}
+EXPORT_SYMBOL(mnh_config_write);
+
+/**
+ * API to read data from MNH DDR space
+ * @param[in] offset offset into MNH DRAM space(BAR4)
+ * @param[in] len buffer size
+ * @param[in] data data to be read into. Client must allocate
+ * the buffer for reading and pass the pointer
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ * @note Usually DMA will be used to read/write from DDR space in MNH
+ * This maybe only used for debugging purpose
+ */
+int mnh_ddr_read(uint32_t offset, uint32_t len, void *data)
+{
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ if (mnh_dev->pdev->current_state != PCI_D0)
+ return -EIO;
+
+ if (len > mnh_dev->bar_size[BAR_4])
+ return -EINVAL;
+
+ if (offset > mnh_dev->bar_size[BAR_4] - len)
+ return -EINVAL; /* address invalid */
+
+ dev_dbg(&mnh_dev->pdev->dev, "Read DDR[0x%08x], len-%d, data-0x%0x",
+ offset, len, *(uint32_t *)data);
+
+ if (WARN_ON(!mnh_dev->powered)) {
+ dev_err(&mnh_dev->pdev->dev,
+ "%s: cannot do pcie transfer while powering down\n",
+ __func__);
+ return -EIO;
+ }
+
+ memcpy(data, mnh_dev->ddr + offset, len);
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_ddr_read);
+
+/**
+ * API to write data to MNH DDR space
+ * @param[in] offset offset into MNH DRAM space(BAR4)
+ * @param[in] len buffer size
+ * @param[in] data pointer to the data to be wrriten.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ * @note Usually DMA will be used to read/write from DDR space in MNH
+ * This maybe only used for debugging purpose
+ */
+int mnh_ddr_write(uint32_t offset, uint32_t len, void *data)
+{
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ if (mnh_dev->pdev->current_state != PCI_D0)
+ return -EIO;
+
+ if (len > mnh_dev->bar_size[BAR_4])
+ return -EINVAL;
+
+ if (offset > mnh_dev->bar_size[BAR_4] - len)
+ return -EINVAL; /* address invalid */
+
+ if (!data) {
+ dev_err(&mnh_dev->pdev->dev, "No Data\n");
+ return -EINVAL; /*data pointer is NULL */
+ }
+
+ dev_dbg(&mnh_dev->pdev->dev, "Write DDR[0x%08x], len-%d, data-0x%x",
+ offset, len, *(uint32_t *)data);
+
+ if (WARN_ON(!mnh_dev->powered)) {
+ dev_err(&mnh_dev->pdev->dev,
+ "%s: cannot do pcie transfer while powering down\n",
+ __func__);
+ return -EIO;
+ }
+
+ memcpy(mnh_dev->ddr + offset, data, len);
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_ddr_write);
+
+/**
+ * API to generate IRQ from AP to MNH
+ * @param[in] irq IRQ ID to be sent (mnh_irq_msg_t)
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_send_irq(enum mnh_irq_msg_t irq)
+{
+ uint32_t mask = 0, data = 0;
+
+ SET_BIT(mask, irq);
+
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ if (mnh_dev->pdev->current_state != PCI_D0)
+ return -EIO;
+
+ dev_dbg(&mnh_dev->pdev->dev, "Send IRQ to EP:%d", irq);
+
+ /* Update PCIE_SW_INTR_EN to enable the spcecific bit*/
+ mnh_config_read(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_SW_INTR_EN,
+ sizeof(uint32_t), &data);
+ data |= mask;
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_SW_INTR_EN,
+ sizeof(uint32_t), data);
+ dev_dbg(&mnh_dev->pdev->dev, "Write sw_intr_en: 0x%x", data);
+
+ /* trigger the interrupt write */
+ mask |= HW_MNH_PCIE_SW_INTR_TRIGG_MASK_RW;
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_SW_INTR_TRIGG,
+ sizeof(uint32_t), mask);
+ dev_dbg(&mnh_dev->pdev->dev, "write sw_intr_trigg: 0x%x", mask);
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_send_irq);
+
+
+/** API to register IRQ callback to receive msg delivery from MNH
+ * @param[in] msg_cb interrupt handler for MSI received from MNH, pass NULL
+ * to de-register
+ * @param[in] vm_cb interrupt handler for Vendor Message received from MNH,
+ * pass NULL to de-register
+ * @param[in] dma_cb interrupt handler for DMA message(DMA_DONE/ABORT) received
+ * from MNH, pass NULL to de-register
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_reg_irq_callback(irq_cb_t msg_cb, irq_cb_t vm_cb, irq_dma_cb_t dma_cb)
+{
+ if (!mnh_dev)
+ return -ENODEV;
+
+ mnh_dev->msg_cb = msg_cb;
+ mnh_dev->vm_cb = vm_cb;
+ mnh_dev->dma_cb = dma_cb;
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_reg_irq_callback);
+
+/**
+ * API to send Vendor specific message from AP to MNH
+ * @param[in] msg Vendor message to be sent include msg_id
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_send_vendor_msg(struct mnh_pcie_vm msg)
+{
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ dev_dbg(&mnh_dev->pdev->dev, "Send Vendor Msg: NOT SUPPORTED\n");
+ return 0;
+}
+EXPORT_SYMBOL(mnh_send_vendor_msg);
+
+
+/**
+ * API to get the base ring buffer address of MNH
+ * @param[out] rb_base ring buffer address TODO: Need to change 64bit
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_get_rb_base(uint64_t *rb_base)
+{
+ uint32_t data;
+
+ if (mnh_config_read(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_1,
+ sizeof(uint32_t), &data) != SUCCESS) {
+ dev_err(&mnh_dev->pdev->dev, "Not able to read RB base\n");
+ return -EINVAL;
+ }
+ *rb_base = data;
+
+ dev_dbg(&mnh_dev->pdev->dev, "RB base: %llx", *rb_base);
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_get_rb_base);
+
+/**
+ * API to set outbound region
+ * @param outb[in] - iATU outbound region information
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_set_outbound_iatu(struct mnh_outb_region *outb)
+{
+ uint32_t data, upper, lower;
+ int size = sizeof(uint32_t);
+
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ dev_dbg(&mnh_dev->pdev->dev, "Set outbound IATU\n");
+
+ if ((outb->region > 0xF) ||
+ (outb->base_address > HW_MNH_PCIE_OUTBOUND_BASE) ||
+ (outb->limit_address > HW_MNH_PCIE_OUTBOUND_END))
+ return -EINVAL; /* address out of range */
+
+ data = IATU_OUTBOUND | outb->region;
+ mnh_pcie_config_write(IATU_VIEWPORT, size, data);
+ upper = UPPER(outb->base_address);
+ lower = LOWER(outb->base_address);
+ mnh_pcie_config_write(IATU_LWR_BASE_ADDR, size, lower);
+ mnh_pcie_config_write(IATU_UPPER_BASE_ADDR, size, upper);
+ data = outb->limit_address;
+ mnh_pcie_config_write(IATU_LIMIT_ADDR, size, data);
+ upper = UPPER(outb->target_pcie_address);
+ lower = LOWER(outb->target_pcie_address);
+ mnh_pcie_config_write(IATU_LWR_TARGET_ADDR, size, lower);
+ mnh_pcie_config_write(IATU_UPPER_TARGET_ADDR, size, upper);
+ mnh_pcie_config_write(IATU_REGION_CTRL_1, size, IATU_MEM);
+ mnh_pcie_config_write(IATU_REGION_CTRL_2, size,
+ IATU_ENABLE | IATU_DMA_BYPASS);
+ udelay(1);
+ mnh_pcie_config_read(IATU_REGION_CTRL_2, size, &data);
+ if (data != (IATU_ENABLE | IATU_DMA_BYPASS))
+ dev_err(&mnh_dev->pdev->dev, "Set outbound IATU Fail\n");
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_set_outbound_iatu);
+
+int mnh_set_inbound_iatu(struct mnh_inb_window *inb)
+{
+ uint32_t data, upper, lower;
+ int size = sizeof(uint32_t);
+
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ dev_dbg(&mnh_dev->pdev->dev, "Set inbound IATU\n");
+
+ if (inb->mode == BAR_MATCH) {
+ if ((inb->region > 0xF) || (inb->bar > 5))
+ return -EINVAL;
+ data = IATU_INBOUND | inb->region;
+ mnh_pcie_config_write(IATU_VIEWPORT, size, data);
+ upper = UPPER(inb->target_mth_address);
+ lower = LOWER(inb->target_mth_address);
+ mnh_pcie_config_write(IATU_LWR_TARGET_ADDR, size, lower);
+ mnh_pcie_config_write(IATU_UPPER_TARGET_ADDR, size, upper);
+ data = IATU_MEM;
+ mnh_pcie_config_write(IATU_REGION_CTRL_1, size, data);
+ data = IATU_BAR_MODE | (inb->bar << 8);
+ mnh_pcie_config_write(IATU_REGION_CTRL_2, size, data);
+ udelay(1);
+ mnh_pcie_config_read(IATU_REGION_CTRL_2, size, &data);
+ } else {
+ if ((inb->region > 0xF) ||
+ (inb->target_mth_address >
+ HW_MNH_PCIE_OUTBOUND_BASE) ||
+ ((inb->target_mth_address
+ + inb->limit_pcie_address)
+ > HW_MNH_PCIE_OUTBOUND_BASE) ||
+ (inb->memmode > 0xf))
+ return -EINVAL; /* address out of range */
+ data = IATU_INBOUND | inb->region;
+ mnh_pcie_config_write(IATU_VIEWPORT, size, data);
+ upper = UPPER(inb->base_pcie_address);
+ lower = LOWER(inb->base_pcie_address);
+ mnh_pcie_config_write(IATU_LWR_BASE_ADDR, size, lower);
+ mnh_pcie_config_write(IATU_UPPER_BASE_ADDR, size, upper);
+ data = inb->limit_pcie_address;
+ mnh_pcie_config_write(IATU_LIMIT_ADDR, size, data);
+ upper = UPPER(inb->target_mth_address);
+ lower = LOWER(inb->target_mth_address);
+ mnh_pcie_config_write(IATU_LWR_TARGET_ADDR, size, lower);
+ mnh_pcie_config_write(IATU_UPPER_TARGET_ADDR, size, upper);
+ mnh_pcie_config_write(IATU_REGION_CTRL_1, size, inb->memmode);
+ mnh_pcie_config_write(IATU_REGION_CTRL_2, size, IATU_ENABLE);
+ udelay(1);
+ mnh_pcie_config_read(IATU_REGION_CTRL_2, size, &data);
+ if (data != IATU_ENABLE)
+ dev_err(&mnh_dev->pdev->dev, "Set inbound IATU Fail\n");
+ }
+ return 0;
+}
+EXPORT_SYMBOL(mnh_set_inbound_iatu);
+
+/**
+ * API to abort DMA transfer on specific channel
+ * @param[in] chan The channel number for DMA transfer abort
+ * @param[in] dir Direction of DMA channel, READ or WRITE
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_dma_abort(uint8_t chan, enum mnh_dma_chan_dir_t dir)
+{
+ uint32_t data;
+ uint8_t len = sizeof(data);
+ struct mnh_dma_state_info_t state;
+
+ dev_dbg(&mnh_dev->pdev->dev, "DMA ABORT: DIR:%d CH:%d\n", dir, chan);
+
+ mnh_dma_get_status(chan, dir, &state);
+ if(state.status != DMA_CHAN_RUNNING) {
+ dev_dbg(&mnh_dev->pdev->dev, "CH not running! - %d\n",
+ state.status);
+ return -EINVAL;
+ }
+
+ /* update doorbell register */
+ if (dir == DMA_EP2AP) { /* write */
+ data = DMA_WRITE_DOORBELL_OFF_MASK_WR_STOP;
+ data |= DMA_WRITE_DOORBELL_OFF_MASK_CH_NUM & chan;
+ mnh_pcie_config_write(DMA_WRITE_DOORBELL_OFF, len, data);
+ } else if (dir == DMA_AP2EP) { /* read */
+ data = DMA_READ_DOORBELL_OFF_MASK_RD_STOP;
+ data |= DMA_READ_DOORBELL_OFF_MASK_CH_NUM & chan;
+ mnh_pcie_config_write(DMA_READ_DOORBELL_OFF, len, data);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_dma_abort);
+
+
+/**
+ * API to resume DMA transfer on specific channel
+ * @param[in] chan The channel number for DMA transfer resume
+ * @param[in] dir Direction of DMA channel, READ or WRITE
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_dma_resume(uint8_t chan, enum mnh_dma_chan_dir_t dir)
+{
+ struct mnh_dma_state_info_t state;
+
+ /* Get the chan state */
+ mnh_dma_get_status(chan, dir, &state);
+
+ if (state.status != DMA_CHAN_STOPPED) {
+ dev_err(&mnh_dev->pdev->dev,
+ "CH[#: %d, dir: %d] is not STOPPED, can't resume\n",
+ chan, dir);
+ return -EINVAL;
+ }
+
+ /*TODO: LL list check */
+ /*TODO: Get the current src and dst pos and do single block update */
+ dev_dbg(&mnh_dev->pdev->dev, "DMA RESUME/ABORT NOT SUPPORTED YET\n");
+
+ return -EINVAL;
+}
+EXPORT_SYMBOL(mnh_dma_resume);
+
+/**
+ * API to read/write single block on specific channel.
+ * @param[in] chan The channel number for DMA transfer to be performed
+ * @param[in] dir The channel direction (read/write)
+ * @param[in] blk One dma element info which include SAR(Source Address),
+ * DAR(Destination Address) and transfer size.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_dma_sblk_start(uint8_t chan, enum mnh_dma_chan_dir_t dir,
+ struct mnh_dma_element_t *blk)
+{
+ uint32_t data = 0;
+ uint8_t size = sizeof(data);
+
+ if (dir == DMA_EP2AP) { /* write */
+ dev_dbg(&mnh_dev->pdev->dev, "DMA WRITE[EP2AP]: CH%d\n", chan);
+ data = DMA_WRITE_ENGINE_EN_OFF_MASK_ENABLE;
+ mnh_pcie_config_write(DMA_WRITE_ENGINE_EN_OFF, size, data);
+ mnh_pcie_config_write(DMA_WRITE_INT_MASK_OFF, size, 0);
+ } else if (dir == DMA_AP2EP) { /* read */
+ dev_dbg(&mnh_dev->pdev->dev, "DMA READ[AP2EP]: CH%d\n", chan);
+ data = DMA_READ_ENGINE_EN_OFF_MASK_ENABLE;
+ mnh_pcie_config_write(DMA_READ_ENGINE_EN_OFF, size, data);
+ mnh_pcie_config_write(DMA_READ_INT_MASK_OFF, size, 0);
+ } else
+ return -EINVAL;
+
+ data = DMA_VIEWPORT_SEL_OFF_MASK_CHAN_NUM & chan;
+ data |= dir ? DMA_VIEWPORT_SEL_OFF_MASK_CHAN_DIR:0;
+ mnh_pcie_config_write(DMA_VIEWPORT_SEL_OFF, size, data);
+ data = DMA_CH_CONTROL1_MASK_LIE;
+ data |= DMA_CH_CONTROL1_MASK_RIE;
+ data |= DMA_CH_CONTROL1_OFF_RESERVED;
+ mnh_pcie_config_write(DMA_CH_CONTROL1_OFF, size, data);
+ data = blk->len;
+ mnh_pcie_config_write(DMA_TRANSFER_SIZE_OFF, size, data);
+ data = LOWER(blk->src_addr);
+ mnh_pcie_config_write(DMA_SAR_LOW_OFF, size, data);
+ data = UPPER(blk->src_addr);
+ mnh_pcie_config_write(DMA_SAR_HIGH_OFF, size, data);
+ data = LOWER(blk->dst_addr);
+ mnh_pcie_config_write(DMA_DAR_LOW_OFF, size, data);
+ data = UPPER(blk->dst_addr);
+ mnh_pcie_config_write(DMA_DAR_HIGH_OFF, size, data);
+
+ if (dir == DMA_EP2AP) { /* write */
+ data = DMA_WRITE_DOORBELL_OFF_MASK_CH_NUM & chan;
+ mnh_pcie_config_write(DMA_WRITE_DOORBELL_OFF, size, data);
+ } else if (dir == DMA_AP2EP) { /* read */
+ data = DMA_READ_DOORBELL_OFF_MASK_CH_NUM & chan;
+ mnh_pcie_config_write(DMA_READ_DOORBELL_OFF, size, data);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_dma_sblk_start);
+
+
+static enum dma_data_direction mnh_to_dma_dir(enum mnh_dma_chan_dir_t mnh_dir)
+{
+ return (mnh_dir == DMA_AP2EP) ? DMA_TO_DEVICE : DMA_FROM_DEVICE;
+}
+
+/**
+ * Convert Linux scatterlist to array of entries used by PCIe EP DMA engine
+ * @param[in] sc_list Scatter gather list for DMA buffer
+ * @param[in] count Number of entries in scatterlist
+ * @param[in] off Offset, in bytes, from which to start transfer.
+ * @param[in] size Size, in bytes, of transfer. This represents actual data.
+ * @param[in] width Width, in bytes. Represents a line of data.
+ * @param[in] stride Stride, in bytes. Must be equal or greater to width.
+ * @param[out] sg Array generated dma addresses and length.
+ * @param[in] maxsg Allocated max array number of the sg
+ * @return a count of sg entries used on success
+ * -EINVAL if exceeding maxsg
+ */
+static int scatterlist_to_mnh_sg(struct scatterlist *sc_list, int count,
+ uint32_t off, uint32_t size, uint32_t width, uint32_t stride,
+ struct mnh_sg_entry *sg, size_t maxsg)
+{
+ struct scatterlist *in_sg;
+ int i, u;
+ uint32_t sc_base = 0; /* Base of an sc segment */
+ uint32_t sc_end = 0; /* End of the sc segment */
+ uint32_t sc_off = off; /* Offset where data is written */
+ uint32_t size_rem = size; /* Overall remaining size */
+ uint32_t width_rem = width; /* Current line remaining width */
+ uint32_t in_sc_off = 0; /* Used to calculate paddr */
+ uint32_t in_sc_rem = 0; /* Leftover size in an sc segment */
+
+ if (stride < width) {
+ dev_err(&mnh_dev->pdev->dev,
+ "stride (%0d )must be greater or equal to width (%0d)\n",
+ stride, width);
+ return -EINVAL;
+ }
+
+ if (width > size) {
+ dev_err(&mnh_dev->pdev->dev,
+ "width(%0d) must be less or equal to size(%0d)\n",
+ width, size);
+ return -EINVAL;
+ }
+
+ i = 0; /* iterator of *sc_list */
+ u = 0; /* iterator of *sg */
+
+ for_each_sg(sc_list, in_sg, count, i) {
+ /* Last entry is reserved for the NULL terminator */
+ if (u >= (maxsg - 1)) {
+ dev_err(&mnh_dev->pdev->dev, "maxsg exceeded\n");
+ return -EINVAL;
+ }
+
+ sc_end = sc_base + sg_dma_len(in_sg);
+ while ((sc_off < sc_end) && size_rem) {
+ in_sc_off = sc_off - sc_base;
+ in_sc_rem = sc_end - sc_off;
+ sg[u].paddr = sg_dma_address(in_sg) + in_sc_off;
+ sg[u].size = width_rem < in_sc_rem ? width_rem :
+ in_sc_rem;
+ width_rem -= sg[u].size;
+ size_rem -= sg[u].size;
+ sc_off += sg[u].size;
+
+ if (width_rem == 0) {
+ width_rem = size_rem > width ? width : size_rem;
+ sc_off += stride - width;
+ }
+ dev_dbg(&mnh_dev->pdev->dev, "sg[%d] : Address %pa , length %zu\n",
+ u, &sg[u].paddr, sg[u].size);
+
+#ifdef COMBINE_SG
+ if ((u > 0) && (sg[u-1].paddr + sg[u-1].size ==
+ sg[u].paddr)) {
+ sg[u-1].size = sg[u-1].size + sg[u].size;
+ sg[u].size = 0;
+ } else {
+ u++;
+ }
+#else
+ u++;
+#endif
+ }
+
+ sc_base = sc_end;
+
+ if (!size_rem)
+ break;
+ }
+
+ if (size_rem) {
+ dev_dbg(&mnh_dev->pdev->dev,
+ "%s: transfer oob: off %u, size %u, sc list size %u, size_rem %u\n",
+ __func__, off, size, sc_end, size_rem);
+ }
+
+ /* Zero out the list terminator entry.
+ * mnh dma engine looks at sg.paddr=0 for end of chain */
+ memset(&sg[u], 0, sizeof(sg[0]));
+ u++; /* Count of entries includes list terminator entry */
+
+ dev_dbg(&mnh_dev->pdev->dev, "SGL with %d/%d entries\n", u, i);
+
+ return u;
+}
+
+/**
+ * Import dma_buf (from ION buffer)
+ * @param[in] fd Handle of dma_buf passed from user
+ * @param[out] sgl pointer of Scatter gather list which has information of
+ * scatter gather list and num of its entries
+ * @return 0 on SUCCESS
+ * negative on failure
+ */
+static int mnh_sg_import_dma_buf(int fd, struct mnh_sg_list *sgl)
+{
+ int ret;
+
+ sgl->dma_buf = dma_buf_get(fd);
+ if (IS_ERR(sgl->dma_buf)) {
+ ret = PTR_ERR(sgl->dma_buf);
+ dev_err(&mnh_dev->pdev->dev,
+ "%s: failed to get dma_buf, err %d\n",
+ __func__, ret);
+ return ret;
+ }
+
+ sgl->attach = dma_buf_attach(sgl->dma_buf, &mnh_dev->pdev->dev);
+ if (IS_ERR(sgl->attach)) {
+ ret = PTR_ERR(sgl->attach);
+ dev_err(&mnh_dev->pdev->dev,
+ "%s: failed to attach dma_buf, err %d\n",
+ __func__, ret);
+ goto err_put;
+ }
+
+ sgl->sg_table = dma_buf_map_attachment(sgl->attach,
+ mnh_to_dma_dir(sgl->dir));
+ if (IS_ERR(sgl->sg_table)) {
+ ret = PTR_ERR(sgl->sg_table);
+ dev_err(&mnh_dev->pdev->dev,
+ "%s: failed to map dma_buf, err %d\n",
+ __func__, ret);
+ goto err_detach;
+ }
+
+ return 0;
+
+err_detach:
+ dma_buf_detach(sgl->dma_buf, sgl->attach);
+err_put:
+ dma_buf_put(sgl->dma_buf);
+ return ret;
+}
+
+/**
+ * API to build a scatter-gather list for multi-block DMA transfer for a
+ * dma_buf
+ * @param[in] fd Handle of dma_buf passed from user
+ * @param[in] off Offset within DMA buffer from which transfer should start.
+ * @param[in] size Size, in bytes, of transfer.
+ * @param[out] sg Array of maxsg pointers to struct mnh_sg_entry, allocated
+ * and filled out by this routine.
+ * @param[out] sgl pointer of Scatter gather list which has information of
+ * scatter gather list and num of its entries.
+ * @return 0 on SUCCESS
+ * negative on failure
+ */
+int mnh_sg_retrieve_from_dma_buf(int fd, uint32_t off, uint32_t size,
+ uint32_t width, uint32_t stride, struct mnh_sg_entry **sg,
+ struct mnh_sg_list *sgl)
+{
+ int ret;
+ size_t maxsg;
+
+ /* Retrieve sg_table from dma_buf framework */
+ ret = mnh_sg_import_dma_buf(fd, sgl);
+ if (ret)
+ return ret;
+
+ /* Use sg_table->sgl as our sc_list */
+ sgl->sc_list = sgl->sg_table->sgl;
+ sgl->n_num = sgl->sg_table->nents;
+
+ /*
+ * The driver assumes either ION userspace code (e.g. Camera HAL)
+ * or dma_buf provider has handled cache correctly.
+ */
+ /* dma_sync_sg_for_device(&mnh_dev->pdev->dev, sgl->sc_list,
+ sgl->n_num, mnh_to_dma_dir(sgl->dir)); */
+
+ /*
+ * Allocate enough for one entry per sc_list entry, plus end of list.
+ * Normally the maximum number of DMA entries would be simply,
+ * equal or less than "sgl->num+1" (including null entry). However we
+ * add the term "(size-1)/width" to account that striding may introduce
+ * more DMA entries per sgl segment. Consider the following example:
+ * sgl->n_num=1 sgl->len=1024 size=16 width=4 stride=8 offset=0
+ * Theoretically would need max_sg = 5 (4 dma entries + 1 null)
+ * With the (size-1)/width the formula now gives us:
+ * max_sg=1+(15/4)+1=5 which is the correct number of entries.
+ * On extreme cases the formula will allocate almost twice the elements,
+ * needed, but this is not a common case.
+ */
+ maxsg = sgl->n_num + ((size-1)/width) + 1;
+ *sg = vmalloc(maxsg * sizeof(struct mnh_sg_entry));
+ if (!(*sg)) {
+ mnh_sg_release_from_dma_buf(sgl);
+ return -ENOMEM;
+ }
+
+ dev_dbg(&mnh_dev->pdev->dev,
+ "Enter %s: n_num:%d maxsg:%zu off:%d size:%d width:%d stride:%d\n",
+ __func__, sgl->n_num, maxsg, off, size, width, stride);
+
+ /* Convert sc_list to a Synopsys compatible linked-list */
+ sgl->length = scatterlist_to_mnh_sg(sgl->sc_list, sgl->n_num, off, size,
+ width, stride, *sg, maxsg);
+ if (IS_ERR(&sgl->length)) {
+ vfree((*sg));
+ *sg = NULL;
+ ret = PTR_ERR(&sgl->length);
+ mnh_sg_release_from_dma_buf(sgl);
+ return ret;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_sg_retrieve_from_dma_buf);
+
+/**
+ * API to release a scatter-gather list for a dma_buf
+ * @param[in] *sgl pointer to the scatter gather list that was built during
+ * mnh_sg_retrieve_from_dma_buf
+ * @return 0 for SUCCESS
+ */
+int mnh_sg_release_from_dma_buf(struct mnh_sg_list *sgl)
+{
+ /* dma_sync_sg_for_cpu(&mnh_dev->pdev->dev, sgl->sc_list,
+ sgl->n_num, mnh_to_dma_dir(sgl->dir)); */
+ dma_buf_unmap_attachment(sgl->attach, sgl->sg_table,
+ mnh_to_dma_dir(sgl->dir));
+ sgl->sc_list = NULL;
+ sgl->n_num = 0;
+ sgl->length = 0;
+ dma_buf_detach(sgl->dma_buf, sgl->attach);
+ dma_buf_put(sgl->dma_buf);
+ return 0;
+}
+EXPORT_SYMBOL(mnh_sg_release_from_dma_buf);
+
+/**
+ * API to build Scatter Gather list to do Multi-block DMA transfer for a user
+ * buffer
+ * @param[in] dmadest Starting virtual addr of the DMA destination
+ * @param[in] size Totalsize of the transfer in bytes
+ * @param[out] sg Array of maxsg pointers to struct mnh_sg_entry, allocated
+ * and filled out by this routine.
+ * @param[out] sgl pointer of Scatter gather list which has information of
+ * page list, scatter gather list and num of its entries.
+ * @return The number of sg[] entries filled out by the routine, negative if
+ * overflow or sg[] not allocated.
+ */
+int mnh_sg_build(void *dmadest, size_t size, struct mnh_sg_entry **sg,
+ struct mnh_sg_list *sgl)
+{
+ int i, u, fp_offset, count;
+ int n_num, p_num;
+ int first_page, last_page;
+ size_t maxsg;
+
+ /*
+ * Allocate enough for one entry per page, perhaps needing 1 more due
+ * to crossing a page boundary, plus end of list.
+ */
+ maxsg = (size / PAGE_SIZE) + 3;
+ *sg = vmalloc(maxsg * sizeof(struct mnh_sg_entry));
+ if (!(*sg))
+ return -ENOMEM;
+
+ /* page num calculation */
+ first_page = ((unsigned long) dmadest & PAGE_MASK) >> PAGE_SHIFT;
+ last_page = (((unsigned long) dmadest + size - 1) & PAGE_MASK) >> PAGE_SHIFT;
+ fp_offset = (unsigned long) dmadest & ~PAGE_MASK;
+ p_num = last_page - first_page + 1;
+
+ dev_dbg(&mnh_dev->pdev->dev,
+ "Enter mnh_sg_build:p_num:%d, maxsg:%zu\n",
+ p_num, maxsg);
+
+ sgl->mypage = vzalloc(p_num * sizeof(struct page *));
+ if (!sgl->mypage) {
+ vfree((*sg));
+ *sg = NULL;
+ sgl->n_num = 0;
+ sgl->length = 0;
+ return -EINVAL;
+ }
+ sgl->sc_list = vzalloc(p_num * sizeof(struct scatterlist));
+ if (!sgl->sc_list) {
+ vfree((*sg));
+ *sg = NULL;
+ vfree(sgl->mypage);
+ sgl->mypage = NULL;
+ sgl->n_num = 0;
+ sgl->length = 0;
+ return -EINVAL;
+ }
+
+ down_read(¤t->mm->mmap_sem);
+ n_num = get_user_pages((unsigned long) dmadest, p_num,
+ FOLL_WRITE | FOLL_FORCE, sgl->mypage, NULL);
+ up_read(¤t->mm->mmap_sem);
+ if (n_num < 0) {
+ dev_err(&mnh_dev->pdev->dev, "fail to get user_pages\n");
+ goto free_mem;
+ }
+ if (n_num < maxsg) {
+ sg_init_table(sgl->sc_list, n_num);
+ if (n_num == 1) {
+ sg_set_page(sgl->sc_list, *(sgl->mypage),
+ size, fp_offset);
+ } else {
+ sg_set_page(sgl->sc_list, *(sgl->mypage),
+ PAGE_SIZE - fp_offset, fp_offset);
+ for (i = 1; i < n_num-1; i++) {
+ sg_set_page(sgl->sc_list + i, *(sgl->mypage + i),
+ PAGE_SIZE, 0);
+ }
+ sg_set_page(sgl->sc_list + n_num-1, *(sgl->mypage + n_num-1),
+ size - (PAGE_SIZE - fp_offset)
+ - ((n_num-2)*PAGE_SIZE), 0);
+ }
+
+ count = dma_map_sg(&mnh_dev->pdev->dev, sgl->sc_list,
+ n_num, mnh_to_dma_dir(sgl->dir));
+
+ u = scatterlist_to_mnh_sg(sgl->sc_list, count, /*off*/0,
+ /*size*/size, /*width*/size, /*stride*/size,
+ *sg, maxsg);
+ if (u < 0)
+ goto unmap_sg;
+ } else {
+ dev_err(&mnh_dev->pdev->dev, "maxsg exceeded\n");
+ goto release_page;
+ }
+ sgl->n_num = n_num;
+ sgl->length = u;
+
+ return 0;
+
+unmap_sg:
+ dma_unmap_sg(&mnh_dev->pdev->dev,
+ sgl->sc_list, n_num, mnh_to_dma_dir(sgl->dir));
+release_page:
+ for (i = 0; i < sgl->n_num; i++)
+ put_page(*(sgl->mypage + i));
+free_mem:
+ vfree((*sg));
+ *sg = NULL;
+ vfree(sgl->mypage);
+ sgl->mypage = NULL;
+ vfree(sgl->sc_list);
+ sgl->sc_list = NULL;
+ sgl->n_num = 0;
+ sgl->length = 0;
+
+ return -EINVAL;
+}
+EXPORT_SYMBOL(mnh_sg_build);
+
+/**
+ * API to release scatter gather list for a user buffer
+ * @param[in] *sgl pointer to the scatter gather list that was built during
+ * mnh_sg_build
+ * @return 0 for SUCCESS
+ */
+int mnh_sg_destroy(struct mnh_sg_list *sgl)
+{
+ int i;
+ struct page *page;
+
+ dma_unmap_sg(&mnh_dev->pdev->dev, sgl->sc_list,
+ sgl->n_num, mnh_to_dma_dir(sgl->dir));
+ for (i = 0; i < sgl->n_num; i++) {
+ page = *(sgl->mypage + i);
+ /* Mark page as dirty before releasing the pages. */
+ if (!PageReserved(page))
+ SetPageDirty(page);
+ put_page(page);
+ }
+ vfree(sgl->mypage);
+ sgl->mypage = NULL;
+ vfree(sgl->sc_list);
+ sgl->sc_list = NULL;
+ sgl->n_num = 0;
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_sg_destroy);
+
+int mnh_sg_verify(struct mnh_sg_entry *sg, size_t size, struct mnh_sg_list *sgl)
+{
+ int i;
+ size_t len = size / sizeof(struct mnh_sg_entry);
+
+ /* At least one entry plus null terminator required */
+ if (len < 2) {
+ dev_err(&mnh_dev->pdev->dev,
+ "Invalid SG list length %zu\n", len);
+ return -EINVAL;
+ }
+ if (sg == NULL) {
+ dev_err(&mnh_dev->pdev->dev, "No SG list\n");
+ return -EINVAL;
+ }
+ for (i = 0; i < len - 1; i++) {
+ if (sg[i].paddr == 0) {
+ dev_err(&mnh_dev->pdev->dev, "Early list end\n");
+ return -EINVAL;
+ }
+ if (sg[i].size == 0) {
+ dev_err(&mnh_dev->pdev->dev, "Invalid entry size\n");
+ return -EINVAL;
+ }
+ }
+ /* Verify terminator */
+ if (sg[i].paddr != 0) {
+ dev_err(&mnh_dev->pdev->dev, "Missing list terminator\n");
+ return -EINVAL;
+ }
+ return 0;
+}
+EXPORT_SYMBOL(mnh_sg_verify);
+
+/**
+ * API to read/write multi blocks on specific channel.
+ * Multi block read/write are based on Linked List(Transfer List) built by MNH.
+ * @param[in] chan The channel number for DMA transfer to be performed
+ * @param[in] dir The channel direction (read/write)
+ * @param[in] start_addr Physical start_addr(in MNH) where transfer list is
+ * stored.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_dma_mblk_start(uint8_t chan, enum mnh_dma_chan_dir_t dir,
+ phys_addr_t *start_addr)
+{
+ uint32_t data = 0;
+ uint8_t size = sizeof(data);
+ uint64_t addr = *start_addr;
+
+ if (dir == DMA_EP2AP) { /* write */
+ dev_dbg(&mnh_dev->pdev->dev, "DMA WRITE M[EP2AP]:CH%d\n", chan);
+ data = DMA_WRITE_ENGINE_EN_OFF_MASK_ENABLE;
+ mnh_pcie_config_write(DMA_WRITE_ENGINE_EN_OFF, size, data);
+ mnh_pcie_config_write(DMA_WRITE_INT_MASK_OFF, size, 0);
+ } else if (dir == DMA_AP2EP) { /* read */
+ dev_dbg(&mnh_dev->pdev->dev, "DMA READ M[AP2EP]:CH%d\n", chan);
+ data = DMA_READ_ENGINE_EN_OFF_MASK_ENABLE;
+ mnh_pcie_config_write(DMA_READ_ENGINE_EN_OFF, size, data);
+ mnh_pcie_config_write(DMA_READ_INT_MASK_OFF, size, 0);
+ }
+
+ data = DMA_VIEWPORT_SEL_OFF_MASK_CHAN_NUM & chan;
+ data |= dir ? DMA_VIEWPORT_SEL_OFF_MASK_CHAN_DIR:0;
+ mnh_pcie_config_write(DMA_VIEWPORT_SEL_OFF, size, data);
+ data = DMA_CH_CONTROL1_MASK_LLE;
+ data |= DMA_CH_CONTROL1_MASK_CCS;
+ data |= DMA_CH_CONTROL1_OFF_RESERVED;
+ mnh_pcie_config_write(DMA_CH_CONTROL1_OFF, size, data);
+ data = LOWER(addr);
+ mnh_pcie_config_write(DMA_LLP_LOW_OFF, size, data);
+ data = UPPER(addr);
+ mnh_pcie_config_write(DMA_LLP_HIGH_OFF, size, data);
+
+ mnh_pcie_config_write(DMA_READ_LINKED_LIST_ERR_EN_OFF, size, 0xF00F);
+ mnh_pcie_config_write(DMA_WRITE_LINKED_LIST_ERR_EN_OFF, size, 0xF00F);
+
+ if (dir == DMA_EP2AP) { /* write */
+ data = DMA_WRITE_DOORBELL_OFF_MASK_CH_NUM & chan;
+ mnh_pcie_config_write(DMA_WRITE_DOORBELL_OFF, size, data);
+ } else if (dir == DMA_AP2EP) { /* read */
+ data = DMA_READ_DOORBELL_OFF_MASK_CH_NUM & chan;
+ mnh_pcie_config_write(DMA_READ_DOORBELL_OFF, size, data);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_dma_mblk_start);
+
+/**
+ * API to get the current status of chan
+ * Multi block DMA write is to transfer data from MNH to AP. Multi block write
+ * are based on Linked List descriptor. Caller specify the source and
+ * destination address of each element through linked list(blk_llist).
+ * @param[in] chan The write channel on which the transfer is performed.
+ * @param[in] dir The direction of channel specified on chan.
+ * @param[out] info chan information includes channel status, transferred
+ * size, error status, and etc.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+
+int mnh_dma_get_status(uint8_t chan, enum mnh_dma_chan_dir_t dir,
+ struct mnh_dma_state_info_t *info)
+{
+ uint32_t data;
+ uint32_t data_hi;
+ uint32_t err = (0x1 << chan);
+
+ /* Select CH */
+ data = dir ? DMA_VIEWPORT_SEL_OFF_MASK_CHAN_DIR:0;
+ data |= DMA_VIEWPORT_SEL_OFF_MASK_CHAN_NUM & chan;
+ mnh_pcie_config_write(DMA_VIEWPORT_SEL_OFF, sizeof(data), data);
+
+ /* Get Status */
+ mnh_pcie_config_read(DMA_CH_CONTROL1_OFF, sizeof(data), &data);
+ info->status = (data & DMA_CH_CONTROL1_MASK_CS) >> 5;
+
+ mnh_pcie_config_read(DMA_TRANSFER_SIZE_OFF, sizeof(data), &data);
+ info->xferred = data;
+
+ info->err = 0;
+ if (dir == DMA_EP2AP) { /* write */
+ mnh_pcie_config_read(DMA_WRITE_ERR_STATUS_OFF, sizeof(data), &data);
+ if (err & (data & DMA_WRITE_ERR_STATUS_OFF_MASK_APP_READ_ERR))
+ info->err = DMA_ERR_WR;
+ else if (err & (data & DMA_WRITE_ERR_STATUS_OFF_MASK_LL_FETCH_ERR))
+ info->err = DMA_ERR_FETCH_LL;
+ } else if (dir == DMA_AP2EP) { /*read */
+ mnh_pcie_config_read(DMA_READ_ERR_STATUS_OFF, sizeof(data),
+ &data);
+ mnh_pcie_config_read(DMA_READ_ERR_STATUS_HIGH_OFF,
+ sizeof(data), &data_hi);
+ if (err & (data & DMA_READ_ERR_STATUS_OFF_MASK_APP_READ_ERR))
+ info->err = DMA_ERR_RD;
+ else if (err & (data & DMA_READ_ERR_STATUS_OFF_MASK_LL_FETCH_ERR))
+ info->err = DMA_ERR_FETCH_LL;
+ else if (err & (data & DMA_READ_ERR_STATUS_HIGH_OFF_UNSUPPORTED))
+ info->err = DMA_ERR_UNSUPPORTED_RQST;
+ else if (err & (data & DMA_READ_ERR_STATUS_HIGH_OFF_CPL_ABORT))
+ info->err = DMA_ERR_COMPLETER_ABORT;
+ else if (err & (data & DMA_READ_ERR_STATUS_HIGH_OFF_CPL_TIMEOUT))
+ info->err = DMA_ERR_CPL_TIME_OUT;
+ else if (err & (data & DMA_READ_ERR_STATUS_HIGH_OFF_MASK_DATA_POISON))
+ info->err = DMA_ERR_DATA_POISONING;
+ }
+
+ dev_dbg(&mnh_dev->pdev->dev,
+ "DMA Status[ch:%d, dir:%d] - , status:%d, xferred:%lld, err:%d\n",
+ chan, dir, info->status, info->xferred, info->err);
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_dma_get_status);
+
+
+/************************************************
+ LOCAL APIS
+*************************************************/
+
+
+/**
+ * mnh_check_pci_resources - MNH PCIE interrupt handler
+ * @irq: Linux interrupt number
+ * @ptr: Pointer to interrupt-specific data
+ *
+ * Handles MNH interrupts signaled using MSI
+ */
+
+static void mnh_check_pci_resources(struct pci_dev *dev, int bar)
+{
+ resource_size_t start, len, end;
+ unsigned long flags;
+
+ dev_dbg(&dev->dev, "PCI RESOURCES CHECK:%d\n", bar);
+
+ start = pci_resource_start(dev, bar);
+ len = pci_resource_len(dev, bar);
+ end = pci_resource_end(dev, bar);
+ flags = pci_resource_flags(dev, bar);
+
+ mnh_dev->bar_size[bar] = len;
+
+ if (bar == BAR_2) {
+ if (len == 0x8000000) /* 128M */
+ mnh_dev->bar2_iatu_region = 0;
+ else if (len == 0x800000 || len == 0x400000) /* 8M or 4M */
+ mnh_dev->bar2_iatu_region = 1;
+ else {
+ dev_err(&dev->dev, "Not supported BAR2 size!!\n");
+ mnh_dev->bar2_iatu_region = 0;
+ }
+ dev_dbg(&dev->dev, "bar2_iatu_region :%d",
+ mnh_dev->bar2_iatu_region);
+ }
+
+ dev_dbg(&dev->dev,
+ "PCI:BAR[%d]:start[%llx],end[%llx],len[%llx],flag[%lx]",
+ bar, start, end, len, flags);
+
+ if ((flags & IORESOURCE_MEM) || (flags & IORESOURCE_MEM_64))
+ dev_dbg(&dev->dev, " flags : %s\n", "MEMORY");
+ else if (flags & IORESOURCE_IO)
+ dev_dbg(&dev->dev, " flags: %s\n", "IO");
+ else
+ dev_dbg(&dev->dev, " flags: %lu\n", flags);
+}
+
+/**
+ * mnh_check_iatu_bar2 - IATU Bar2 reprogramming
+ * @offset: register address
+ *
+ * Based on the register offset, reprogram IATU BAR2.
+ * Due to limitation of BAR2 size, we need to remap the IATU either for
+ * SRAM or Cluster register region.
+ */
+static uint32_t mnh_check_iatu_bar2(uint32_t offset)
+{
+ uint8_t new_region;
+ uint64_t start_addr;
+ uint32_t new_offset;
+ struct mnh_inb_window iatu;
+
+ if (!mnh_dev || !mnh_dev->pdev)
+ return -ENODEV;
+
+ if (mnh_dev->bar2_iatu_region == 0) /* No need to reprogram the IATU */
+ return offset;
+
+ if (offset < HW_MNH_PCIE_BAR2_R2_ADDR_START) {
+ new_region = 1;
+ start_addr = HW_MNH_PCIE_BAR2_R1_ADDR_START;
+ new_offset = offset;
+ } else {
+ new_region = 2;
+ start_addr = HW_MNH_PCIE_BAR2_R2_ADDR_START;
+ new_offset = offset - HW_MNH_PCIE_BAR2_R2_ADDR_START;
+ }
+
+ if (mnh_dev->bar2_iatu_region != new_region) {
+ dev_dbg(&mnh_dev->pdev->dev, "IATU BAR2 reprogram - region#%d",
+ new_region);
+
+ iatu.mode = BAR_MATCH;
+ iatu.bar = 2;
+ iatu.region = 1;
+ iatu.target_mth_address = start_addr;
+ iatu.limit_pcie_address = 0xfff;
+ iatu.base_pcie_address = 0x3fffffff00000000;
+ mnh_set_inbound_iatu(&iatu);
+ mnh_dev->bar2_iatu_region = new_region;
+ }
+
+ return new_offset;
+}
+
+/**
+ * mnh_dma_send_cb - DMA Callback notify
+ * @chan: DMA channel being used
+ * @dir: DMA direction
+ * @status: DMA transfer status(result)
+ *
+ * Upon DMA transfer finished, send out the result back to Client and EP
+ */
+static int mnh_dma_send_cb(uint8_t chan, enum mnh_dma_chan_dir_t dir,
+ enum mnh_dma_trans_status_t status)
+{
+
+ uint32_t ep_irq = 0;
+ uint32_t status_shift = 0, dir_shift = 0;
+
+ mnh_config_read(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET +
+ HW_MNH_PCIE_GP_2, sizeof(uint32_t),
+ &ep_irq);
+
+ dir_shift = (dir == DMA_AP2EP) ? 0 : 16;
+ status_shift = (status == DMA_DONE) ? 0 : 8;
+ ep_irq |= 1 << chan << dir_shift << status_shift;
+
+ /* Send IRQ to EP */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET +
+ HW_MNH_PCIE_GP_2, sizeof(uint32_t),
+ ep_irq);
+ mnh_send_irq(IRQ_DMA_STATUS);
+
+ /* Send CB to Client */
+ if (!IS_NULL(mnh_dev->dma_cb))
+ mnh_dev->dma_cb(chan, dir, status);
+
+ return 0;
+}
+
+/**
+ * API to initialize DMA
+ * perform DMA initialization.
+ * This includes reset MNH DMA controller logic and check enabled channel
+ * and set the MSI address and data.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+static int mnh_dma_init(void)
+{
+ uint32_t low_addr, high_addr, data, msi_data;
+ uint8_t msi_en = 0;
+ uint32_t read_imwr = 0, write_imwr = 0;
+ int size = sizeof(uint32_t);
+
+ /* Read MSI addr */
+ mnh_pcie_config_read(MSI_CAP_ID_NEXT_CTRL_REG, size, &data);
+ msi_en = (data & MULTIPLE_MSG_ENABLE_MASK) >> 20;
+ mnh_pcie_config_read(MSI_CAP_OFF_04H_REG, size, &low_addr);
+ mnh_pcie_config_read(MSI_CAP_OFF_08H_REG, size, &high_addr);
+ mnh_pcie_config_read(MSI_CAP_OFF_0CH_REG, size, &msi_data);
+ mnh_pcie_config_read(MSI_CAP_OFF_10H_REG, size, &data);
+
+ /* Update DMA MSI addr information */
+ mnh_pcie_config_write(DMA_WRITE_DONE_IMWR_LOW_OFF, size, low_addr);
+ mnh_pcie_config_write(DMA_WRITE_DONE_IMWR_HIGH_OFF, size, high_addr);
+ mnh_pcie_config_write(DMA_WRITE_ABORT_IMWR_LOW_OFF, size, low_addr);
+ mnh_pcie_config_write(DMA_WRITE_ABORT_IMWR_HIGH_OFF, size, high_addr);
+
+ mnh_pcie_config_write(DMA_READ_DONE_IMWR_LOW_OFF, size, low_addr);
+ mnh_pcie_config_write(DMA_READ_DONE_IMWR_HIGH_OFF, size, high_addr);
+ mnh_pcie_config_write(DMA_READ_ABORT_IMWR_LOW_OFF, size, low_addr);
+ mnh_pcie_config_write(DMA_READ_ABORT_IMWR_HIGH_OFF, size, high_addr);
+
+ /* Set DMA MSI */
+ msi_data = msi_data & (0xFFFFFFF << msi_en);
+ dev_dbg(&mnh_dev->pdev->dev, " msi_data: 0x%x, msi_en:%d",
+ msi_data, msi_en);
+ if (mnh_dev->msi_num > 1) {
+ dev_dbg(&mnh_dev->pdev->dev, "%d MSI\n", mnh_dev->msi_num);
+ read_imwr = (msi_data | MSI_DMA_READ) |
+ (msi_data | MSI_DMA_READ) << 16;
+ write_imwr = (msi_data | MSI_DMA_WRITE) |
+ (msi_data | MSI_DMA_WRITE) << 16;
+ } else if (mnh_dev->msi_num == 1) {
+ dev_dbg(&mnh_dev->pdev->dev, "1 MSI\n");
+ read_imwr = msi_data | msi_data << 16;
+ write_imwr = read_imwr;
+ }
+
+ mnh_pcie_config_write(DMA_READ_CH01_IMWR_DATA_OFF, size, read_imwr);
+ mnh_pcie_config_write(DMA_READ_CH23_IMWR_DATA_OFF, size, read_imwr);
+ mnh_pcie_config_write(DMA_WRITE_CH01_IMWR_DATA_OFF, size, write_imwr);
+ mnh_pcie_config_write(DMA_WRITE_CH23_IMWR_DATA_OFF, size, write_imwr);
+
+ return 0;
+}
+
+#ifdef CONFIG_MNH_PCIE_MULTIPLE_MSI
+
+/**
+ * mnh_pci_dma_irq_handler - MNH PCIE DMA interrupt handler
+ * @irq: Linux interrupt number
+ * @ptr: Pointer to interrupt-specific data
+ */
+static irqreturn_t mnh_pci_dma_irq_handler(int irq, void *ptr)
+{
+
+ struct mnh_device *dev = (struct mnh_device *)ptr;
+ uint32_t data, done, abort;
+ uint8_t chan, i = 0;
+ uint8_t len = sizeof(data);
+ enum mnh_dma_chan_dir_t dir;
+ enum mnh_dma_trans_status_t status = -1;
+
+ dev_dbg(&dev->pdev->dev, "DMA IRQ Handler: %d\n", irq);
+
+ if (irq < dev->pdev->irq + MSI_DMA_READ ||
+ irq > dev->pdev->irq + MSI_DMA_WRITE) {
+ dev_err(&dev->pdev->dev, "Invalid DMA IRQ : %d\n", irq);
+ return IRQ_HANDLED;
+ }
+
+ if (irq == dev->pdev->irq + MSI_DMA_READ) {
+ dir = DMA_AP2EP;
+ mnh_pcie_config_read(DMA_READ_INT_STATUS_OFF, len, &data);
+ done = data & DMA_READ_INT_STATUS_OFF_DONE_STATUS;
+ abort = (data & DMA_READ_INT_STATUS_OFF_ABORT_STATUS) >> 16;
+ /* Clear interrupt */
+ mnh_pcie_config_write(DMA_READ_INT_CLEAR_OFF, len, data);
+
+ } else if (irq == dev->pdev->irq + MSI_DMA_WRITE) {
+ dir = DMA_EP2AP;
+ mnh_pcie_config_read(DMA_WRITE_INT_STATUS_OFF, len, &data);
+ done = data & DMA_WRITE_INT_STATUS_OFF_DONE_STATUS;
+ abort = (data & DMA_WRITE_INT_STATUS_OFF_ABORT_STATUS) >> 16;
+
+ /* Clear interrupt */
+ mnh_pcie_config_write(DMA_WRITE_INT_CLEAR_OFF, len, data);
+ } else {
+ dev_err(&dev->pdev->dev, "Invalid DMA IRQ : %d\n", irq);
+ return IRQ_HANDLED;
+ }
+
+ for (i = 0; i < DMA_MAX_CHAN; i++) {
+ chan = 1 << i;
+ if (done & chan) {
+ status = DMA_DONE;
+ dev_dbg(&dev->pdev->dev, "DMA DONE:ch[%d], dir[%d]",
+ i, dir);
+ mnh_dma_send_cb(i, dir, status);
+ }
+ if (abort & chan) {
+ status = DMA_ABORT;
+ dev_dbg(&dev->pdev->dev, "DMA ABORT:ch[%d], dir[%d]",
+ i, dir);
+ mnh_dma_send_cb(i, dir, status);
+ }
+ }
+ return IRQ_HANDLED;
+}
+
+/**
+ * mnh_pci_irq_handler - MNH PCIE interrupt handler
+ * @irq: Linux interrupt number
+ * @ptr: Pointer to interrupt-specific data
+ *
+ * Handles MNH interrupts signaled using MSI
+ */
+static irqreturn_t mnh_pci_irq_handler(int irq, void *ptr)
+{
+ struct mnh_device *dev = (struct mnh_device *)ptr;
+ uint32_t local_irq;
+
+ dev_dbg(&dev->pdev->dev, "irq_handler: %d\n", irq);
+
+ if ((irq < dev->pdev->irq + MSI_GENERIC_START) ||
+ (irq > dev->pdev->irq + MSI_GENERIC_END)) {
+ dev_err(&dev->pdev->dev, "Invalid IRQ : %d\n", irq);
+ return IRQ_HANDLED;
+ }
+
+ if (irq == dev->pdev->irq + MSI_BOOTSTRAP_SET) {
+ /* Read ring_buf_addr */
+ mnh_config_read(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET +
+ HW_MNH_PCIE_GP_1, sizeof(uint32_t),
+ &mnh_dev->ring_buf_addr);
+ dev_dbg(&dev->pdev->dev, "MNH ring buffer addr: 0x%x\n",
+ mnh_dev->ring_buf_addr);
+ }
+
+
+ /* If MSG Callback is set, then call the callback to notify the MSI
+ has been delivered */
+ if (!IS_NULL(dev->msg_cb)) {
+ local_irq = irq - dev->pdev->irq;
+ dev->msg_cb(local_irq);
+ dev_dbg(&dev->pdev->dev, "MSI sent out to client:0x%x\n",
+ local_irq);
+ }
+
+ return IRQ_HANDLED;
+}
+#endif
+/**
+ * mnh_pci_irq_single_handler - MNH PCIE interrupt handler for single MSI
+ * @irq: Linux interrupt number
+ * @ptr: Pointer to interrupt-specific data
+ *
+ * Handles MNH/DMA interrrupts when single MSI is being used
+ */
+static irqreturn_t mnh_pci_irq_single_handler(int irq, void *ptr)
+{
+ struct mnh_device *dev = (struct mnh_device *)ptr;
+ uint32_t msi_vector;
+ uint32_t data1, data2, done, abort;
+ uint8_t chan, i = 0;
+ uint8_t len = sizeof(uint32_t);
+ enum mnh_dma_chan_dir_t dir;
+ enum mnh_dma_trans_status_t status = -1;
+
+ dev_dbg(&dev->pdev->dev, "irq_single_handler: %d\n", irq);
+
+ /* Check DMA interrupt */
+ mnh_pcie_config_read(DMA_READ_INT_STATUS_OFF, len, &data1);
+ mnh_pcie_config_read(DMA_WRITE_INT_STATUS_OFF, len, &data2);
+ if (data1) {
+ dev_dbg(&dev->pdev->dev, "DMA Read IRQ found -0x%x\n", data1);
+ dir = DMA_AP2EP;
+ done = data1 & DMA_READ_INT_STATUS_OFF_DONE_STATUS;
+ abort = (data1 & DMA_READ_INT_STATUS_OFF_ABORT_STATUS) >> 16;
+ /* Clear interrupt */
+ mnh_pcie_config_write(DMA_READ_INT_CLEAR_OFF, len, data1);
+ } else if (data2) {
+ dev_dbg(&dev->pdev->dev, "DMA Write IRQ found -0x%x\n", data2);
+ dir = DMA_EP2AP;
+ done = data2 & DMA_WRITE_INT_STATUS_OFF_DONE_STATUS;
+ abort = (data2 & DMA_WRITE_INT_STATUS_OFF_ABORT_STATUS) >> 16;
+ /* Clear interrupt */
+ mnh_pcie_config_write(DMA_WRITE_INT_CLEAR_OFF, len, data2);
+ }
+
+ if (data1 || data2) {
+ for (i = 0; i < DMA_MAX_CHAN; i++) {
+ chan = 1 << i;
+ if (done & chan) {
+ status = DMA_DONE;
+ dev_dbg(&dev->pdev->dev, "DMA DONE:ch[%d], dir[%d]",
+ i, dir);
+ mnh_dma_send_cb(i, dir, status);
+ }
+ if (abort & chan) {
+ status = DMA_ABORT;
+ dev_dbg(&dev->pdev->dev, "DMA ABORT:ch[%d], dir[%d]",
+ i, dir);
+ mnh_dma_send_cb(i, dir, status);
+ }
+ }
+ return IRQ_HANDLED;
+ }
+
+
+ mnh_config_read(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET +
+ HW_MNH_PCIE_GP_3, sizeof(uint32_t), &msi_vector);
+ dev_dbg(&dev->pdev->dev, "msi_vector: 0x%x\n", msi_vector);
+
+ /* Check MSI generic interrupt */
+ if ((msi_vector >= MSI_GENERIC_START) ||
+ (msi_vector <= MSI_GENERIC_END)) {
+ if (msi_vector == MSI_BOOTSTRAP_SET) {
+ /* Read ring_buf_addr */
+ mnh_config_read(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET +
+ HW_MNH_PCIE_GP_1, sizeof(uint32_t),
+ &mnh_dev->ring_buf_addr);
+ dev_dbg(&dev->pdev->dev, "MNH ring buffer addr: 0x%x\n",
+ mnh_dev->ring_buf_addr);
+ }
+ /* If MSG Callback is set, then call the callback to notify
+ the MSI has been delivered */
+ if (!IS_NULL(dev->msg_cb)) {
+ dev->msg_cb(msi_vector);
+ dev_dbg(&dev->pdev->dev, "MSI sent out to client:0x%x\n",
+ msi_vector);
+ }
+ } else if (msi_vector != 0)
+ dev_err(&dev->pdev->dev, "Invalid IRQ : %d\n", msi_vector);
+
+ return IRQ_HANDLED;
+
+}
+
+/**
+ * mnh_irq_init - Setting up MNH PCIE MSI and register irq handler
+ * @pdev: PCIE dev
+ * @return returns msi number initialized successfully
+ */
+static int mnh_irq_init(struct pci_dev *pdev)
+{
+
+#ifdef CONFIG_MNH_PCIE_MULTIPLE_MSI
+ int err, vector, i;
+ uint32_t msinum = MSI_DMA_WRITE + 1;
+
+ vector = pci_enable_msi_range(pdev, 1, msinum);
+ dev_err(&pdev->dev, "vector :%d , msi_num:%d, irq:%d\n",
+ vector, msinum, pdev->irq);
+ if (vector < msinum) {
+ dev_err(&pdev->dev, "failed to enable MSI range :%d, err:%d\n",
+ msinum, vector);
+ if (vector >= 1) {
+ /* Fall back to 1 MSI */
+ err = request_threaded_irq(pdev->irq, NULL,
+ mnh_pci_irq_single_handler,
+ IRQF_SHARED | IRQF_ONESHOT,
+ MODULE_NAME, mnh_dev);
+ if (err) {
+ dev_err(&pdev->dev,
+ "fail to req msi %d: err:%d\n",
+ pdev->irq, err);
+ goto error;
+ }
+ return 1;
+ }
+ goto error;
+ }
+
+
+ /* MSI IRQs request */
+ for (i = MSI_GENERIC_START; i <= MSI_GENERIC_END; i++) {
+ err = request_threaded_irq(pdev->irq + i,
+ NULL, mnh_pci_irq_handler,
+ IRQF_SHARED | IRQF_ONESHOT, MODULE_NAME, mnh_dev);
+ if (err) {
+ dev_err(&pdev->dev, "failed to req MSI:%d, err:%d\n",
+ pdev->irq + i, err);
+ goto free_irq;
+ }
+ dev_err(&pdev->dev, "request irq:%d\n", pdev->irq+i);
+ }
+
+ /* DMA READ MSI IRQ request */
+ err = request_threaded_irq(pdev->irq + MSI_DMA_READ,
+ NULL, mnh_pci_dma_irq_handler,
+ IRQF_SHARED | IRQF_ONESHOT, MODULE_NAME, mnh_dev);
+ dev_err(&pdev->dev, "request irq:%d\n", pdev->irq+MSI_DMA_READ);
+ if (err) {
+ dev_err(&pdev->dev, "request DMA read irq failure. irq = %d\n",
+ pdev->irq + MSI_DMA_READ);
+ goto free_irq;
+ }
+
+ /* DMA WRITE MSI IRQ request */
+ err = request_threaded_irq(pdev->irq + MSI_DMA_WRITE,
+ NULL, mnh_pci_dma_irq_handler,
+ IRQF_SHARED | IRQF_ONESHOT, MODULE_NAME, mnh_dev);
+ dev_err(&pdev->dev, "request irq:%d\n", pdev->irq+MSI_DMA_WRITE);
+ if (err) {
+ dev_err(&pdev->dev, "request DMA write irq failure. irq = %d\n",
+ pdev->irq + MSI_DMA_WRITE);
+ goto free_dma_irq;
+ }
+
+ return msinum;
+
+free_dma_irq:
+ free_irq(pdev->irq + MSI_DMA_READ, mnh_dev);
+free_irq:
+
+ for (--i; i >= MSI_GENERIC_START; i--)
+ free_irq(pdev->irq + i, mnh_dev);
+
+#else
+ int err;
+
+ err = pci_enable_msi(pdev);
+ if (err) {
+ dev_err(&pdev->dev, "fail to allocate msi\n");
+ return 0;
+ }
+
+ err = request_threaded_irq(pdev->irq, NULL,
+ mnh_pci_irq_single_handler,
+ IRQF_SHARED | IRQF_ONESHOT,
+ MODULE_NAME, mnh_dev);
+
+ if (err) {
+ dev_err(&pdev->dev, "fail to req msi %d: err:%d\n",
+ pdev->irq, err);
+ goto error;
+ }
+
+ return 1;
+
+#endif
+
+error:
+ dev_err(&pdev->dev, "Error setting up irq\n");
+ pci_disable_msi(pdev);
+ return 0;
+}
+
+static int mnh_irq_deinit(struct pci_dev *pdev)
+{
+ int i;
+
+ dev_dbg(&mnh_dev->pdev->dev, "IRQ deinitializing..\n");
+
+ if (mnh_dev->msi_num == 1) {
+ dev_dbg(&mnh_dev->pdev->dev, "free-irq : %d", pdev->irq);
+ free_irq(pdev->irq, mnh_dev);
+ pci_disable_msi(pdev);
+ } else if (mnh_dev->msi_num > 1) {
+ for (i = MSI_GENERIC_START; i <= MSI_GENERIC_END; i++) {
+ dev_dbg(&mnh_dev->pdev->dev, "free-irq:%d",
+ pdev->irq + i);
+ free_irq(pdev->irq + i, mnh_dev);
+ }
+ free_irq(pdev->irq + MSI_DMA_READ, mnh_dev);
+ free_irq(pdev->irq + MSI_DMA_WRITE, mnh_dev);
+ pci_disable_msi(pdev);
+ }
+
+ return 0;
+
+}
+
+static void setup_smmu(struct pci_dev *pdev)
+{
+ struct dma_iommu_mapping *mapping;
+ int atomic_ctx = 1;
+ int bypass_enable = 1;
+ int ret;
+
+/* Following stolen from msm_11ad.c */
+#define SMMU_BASE 0x10000000 /* Device address range base */
+#define SMMU_SIZE 0x40000000 /* Device address range size */
+
+ mapping = arm_iommu_create_mapping(&platform_bus_type,
+ SMMU_BASE, SMMU_SIZE);
+
+ if (IS_ERR_OR_NULL(mapping)) {
+ ret = PTR_ERR(mapping) ?: -ENODEV;
+ dev_err(&pdev->dev,
+ "Failed to create IOMMU mapping (%d)\n", ret);
+ return;
+ }
+
+ ret = iommu_domain_set_attr(
+ mapping->domain, DOMAIN_ATTR_ATOMIC, &atomic_ctx);
+ if (ret) {
+ dev_err(&pdev->dev,
+ "Set atomic attribute to SMMU failed (%d)\n", ret);
+ }
+
+ ret = iommu_domain_set_attr(mapping->domain,
+ DOMAIN_ATTR_S1_BYPASS,
+ &bypass_enable);
+ if (ret) {
+ dev_err(&pdev->dev,
+ "Set bypass attribute to SMMU failed (%d)\n", ret);
+ }
+
+ ret = arm_iommu_attach_device(&pdev->dev, mapping);
+ if (ret) {
+ dev_err(&pdev->dev,
+ "arm_iommu_attach_device failed (%d)\n", ret);
+ return;
+ }
+
+ dev_info(&pdev->dev, "attached to IOMMU\n");
+}
+
+/*
+ * mnh_pci_fixup - Fixup routine before probe
+ *
+ * @pdev: PCI device structure
+ */
+static void mnh_pci_fixup(struct pci_dev *pdev)
+{
+ dev_dbg(&pdev->dev, "MNH PCI Entering mnh_pci_fixup\n");
+ if (pdev->class == PCI_CLASS_NOT_DEFINED) {
+ dev_info(&pdev->dev, "setting pcie class\n");
+ pdev->class = PCI_CLASS_MEMORY_OTHER;
+ }
+}
+
+int mnh_pci_init_resume(void)
+{
+ int err;
+ int i = 0;
+ uint32_t magic;
+ struct mnh_inb_window iatu;
+ struct pci_dev *pdev = mnh_dev->pdev;
+
+ /* enable pci dev */
+ err = pci_enable_device(pdev);
+ if (err)
+ dev_err(&pdev->dev, "failed to enable pci device.\n");
+
+ /* set PCI host mastering */
+ pci_set_master(pdev);
+
+ if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(64)) ||
+ dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64))) {
+
+ err = dma_set_mask(&pdev->dev, DMA_BIT_MASK(32));
+ if (err)
+ err = dma_set_coherent_mask(&pdev->dev,
+ DMA_BIT_MASK(32));
+ }
+ if (err)
+ dev_err(&pdev->dev, "No usable DMA configuration, aborting\n");
+
+ for (i = 0; i < BAR_MAX_NUM; i++)
+ mnh_check_pci_resources(pdev, i);
+
+ mnh_dma_init();
+
+ /* Programing the inbound IATU */
+ iatu.mode = BAR_MATCH;
+ iatu.bar = 2;
+ iatu.region = 1;
+ iatu.target_mth_address = HW_MNH_PCIE_BAR_2_ADDR_START;
+ iatu.limit_pcie_address = 0xfff;
+ iatu.base_pcie_address = 0x3fffffff00000000;
+ mnh_set_inbound_iatu(&iatu);
+
+ iatu.mode = BAR_MATCH;
+ iatu.bar = 4;
+ iatu.region = 2;
+ iatu.target_mth_address = HW_MNH_PCIE_BAR_4_ADDR_START;
+ iatu.limit_pcie_address = 0xfff;
+ iatu.base_pcie_address = 0x3ffffff00000000;
+ mnh_set_inbound_iatu(&iatu);
+
+ err = mnh_pcie_config_read(0, sizeof(uint32_t), &magic);
+ if (err) {
+ dev_err(&pdev->dev, "failed to read magic register: 0x%08x (%d)\n",
+ magic, err);
+ }
+
+ dev_dbg(&pdev->dev, "MNH PCIe reinitialization successful.\n");
+
+ return 0;
+}
+
+/**
+ * mnh_pci_probe - Device Initialization Routine
+ *
+ * @pdev: PCI device structure
+ * @ent: entry in kcs_pci_tbl
+ *
+ * Return: 0 on success, <0 on failure.
+ */
+static int mnh_pci_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
+{
+ int err;
+ int i = 0;
+ uint32_t magic;
+ struct mnh_inb_window iatu;
+
+ dev_dbg(&pdev->dev, "MNH PCI Device found [%04x:%04x] (rev %x)\n",
+ (int)pdev->vendor, (int)pdev->device, (int)pdev->revision);
+
+ mnh_dev = kzalloc(sizeof(struct mnh_device), GFP_KERNEL);
+ if (mnh_dev == NULL) {
+ err = -ENOMEM;
+ goto end;
+ }
+
+ mnh_dev->pdev = pdev;
+
+ /* probe is called after pcie has enumerated, so set powered flag */
+ mnh_dev->powered = true;
+
+ /* enable pci dev */
+ err = pci_enable_device(pdev);
+ if (err) {
+ dev_err(&pdev->dev, "failed to enable pci device.\n");
+ goto free_device;
+ }
+
+ /* set PCI host mastering */
+ pci_set_master(pdev);
+
+ /* pci request regions */
+ err = pci_request_regions(pdev, MODULE_NAME);
+ dev_dbg(&pdev->dev, "return value of request_region:%x", err);
+
+ if (err) {
+ dev_err(&pdev->dev, "failed to get pci regions.\n");
+ goto disable_device;
+ }
+
+
+ if (dma_set_mask(&pdev->dev, DMA_BIT_MASK(64)) ||
+ dma_set_coherent_mask(&pdev->dev, DMA_BIT_MASK(64))) {
+
+ err = dma_set_mask(&pdev->dev, DMA_BIT_MASK(32));
+ if (err)
+ err = dma_set_coherent_mask(&pdev->dev,
+ DMA_BIT_MASK(32));
+ }
+ if (err) {
+ dev_err(&pdev->dev, "No usable DMA configuration, aborting\n");
+ goto release_regions;
+ }
+
+ for (i = 0; i < BAR_MAX_NUM; i++)
+ mnh_check_pci_resources(pdev, i);
+
+ /* mapping PCIE configuration memory */
+ mnh_dev->pcie_config = pci_ioremap_bar(pdev, BAR_0);
+
+ if (!mnh_dev->pcie_config) {
+ dev_err(&pdev->dev, "mapping BAR0/1 device memory failure.\n");
+ err = -ENOMEM;
+ goto release_regions;
+ }
+
+ /* mapping Peripharal configuration memory */
+ mnh_dev->config = pci_ioremap_bar(pdev, BAR_2);
+ if (!mnh_dev->config) {
+ dev_err(&pdev->dev, "mapping BAR2/3 memory failure.\n");
+ err = -ENOMEM;
+ goto unmap_bar;
+ }
+
+ /* mapping Peripharal configuration memory */
+ mnh_dev->ddr = pci_ioremap_bar(pdev, BAR_4);
+ if (!mnh_dev->ddr) {
+ dev_err(&pdev->dev, "mapping BAR4/5 memory failure.\n");
+ err = -ENOMEM;
+ goto unmap_bar;
+ }
+
+
+
+/* IRQ handling */
+#if CONFIG_PCI_MSI
+ mnh_dev->msi_num = mnh_irq_init(pdev);
+#endif
+
+ pci_set_drvdata(pdev, (void *)mnh_dev);
+
+ mnh_dma_init();
+
+ /* Programing the inbound IATU */
+ iatu.mode = BAR_MATCH;
+ iatu.bar = 2;
+ iatu.region = 1;
+ iatu.target_mth_address = HW_MNH_PCIE_BAR_2_ADDR_START;
+ iatu.limit_pcie_address = 0xfff;
+ iatu.base_pcie_address = 0x3fffffff00000000;
+ mnh_set_inbound_iatu(&iatu);
+
+ iatu.mode = BAR_MATCH;
+ iatu.bar = 4;
+ iatu.region = 2;
+ iatu.target_mth_address = HW_MNH_PCIE_BAR_4_ADDR_START;
+ iatu.limit_pcie_address = 0xfff;
+ iatu.base_pcie_address = 0x3ffffff00000000;
+ mnh_set_inbound_iatu(&iatu);
+
+ setup_smmu(pdev);
+
+ err = mnh_pcie_config_read(0, sizeof(uint32_t), &magic);
+ if (err) {
+ dev_err(&pdev->dev, "failed to read magic register: 0x%08x (%d)\n",
+ magic, err);
+ }
+
+ dev_dbg(&pdev->dev, "MNH PCIe initialization successful.\n");
+
+ return 0;
+
+unmap_bar:
+ if (mnh_dev->pcie_config)
+ pci_iounmap(pdev, mnh_dev->pcie_config);
+ if (mnh_dev->config)
+ pci_iounmap(pdev, mnh_dev->config);
+ if (mnh_dev->ddr)
+ pci_iounmap(pdev, mnh_dev->ddr);
+release_regions:
+ pci_release_regions(pdev);
+disable_device:
+ pci_disable_device(pdev);
+free_device:
+ pci_set_drvdata(pdev, NULL);
+ kfree(mnh_dev);
+end:
+ dev_err(&pdev->dev, "initialization failed.\n");
+ return err;
+}
+
+/**
+ * mnh_pci_remove - Device Removal Routine
+ *
+ * @pdev: PCI device structure
+ *
+ * mnh_pci_remove is called by the PCI subsystem to alert the driver
+ * that it should release a PCI device.
+ */
+static void mnh_pci_remove(struct pci_dev *pdev)
+{
+ struct mnh_device *dev;
+
+ dev = pci_get_drvdata(pdev);
+ if (!dev) {
+ dev_err(&pdev->dev, "get Drvdata fail\n");
+ return;
+ }
+
+ mnh_irq_deinit(pdev);
+
+ if (dev->pcie_config) {
+ dev_err(&pdev->dev, "ioummap pcie_config\n");
+ pci_iounmap(pdev, dev->pcie_config);
+ }
+ if (dev->config) {
+ dev_err(&pdev->dev, "ioummap config\n");
+ pci_iounmap(pdev, dev->config);
+ }
+ if (dev->ddr) {
+ dev_err(&pdev->dev, "ioummap ddr\n");
+ pci_iounmap(pdev, dev->ddr);
+ }
+
+ pci_set_drvdata(pdev, NULL);
+ kfree(dev);
+
+ pci_release_regions(pdev);
+ pci_disable_device(pdev);
+
+ dev_dbg(&pdev->dev, "MNH PCIe driver is removed\n");
+}
+
+int mnh_pci_suspend(void)
+{
+ struct pci_dev *pdev = mnh_dev->pdev;
+ struct device *dev = &mnh_dev->pdev->dev;
+
+ dev_dbg(dev, "%s: enter\n", __func__);
+
+ /* disable IRQs */
+#ifdef CONFIG_MNH_PCIE_MULTIPLE_MSI
+ {
+ int i;
+
+ for (i = MSI_GENERIC_START; i <= MSI_GENERIC_END; i++)
+ disable_irq(pdev->irq + i);
+ disable_irq(pdev->irq + MSI_DMA_READ);
+ disable_irq(pdev->irq + MSI_DMA_WRITE);
+ }
+#else
+ disable_irq(pdev->irq);
+#endif
+
+ mnh_dev->powered = false;
+
+ dev_dbg(dev, "%s: exit\n", __func__);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(mnh_pci_suspend);
+
+int mnh_pci_resume(void)
+{
+ struct pci_dev *pdev = mnh_dev->pdev;
+ struct device *dev = &mnh_dev->pdev->dev;
+ int ret = 0;
+
+ dev_dbg(dev, "%s: enter\n", __func__);
+
+ mnh_dev->powered = true;
+
+ ret = mnh_pci_init_resume();
+ if (ret) {
+ dev_err(dev, "%s: mnh_pci_init_resume failed (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ /* enable IRQs */
+#ifdef CONFIG_MNH_PCIE_MULTIPLE_MSI
+ {
+ int i;
+
+ for (i = MSI_GENERIC_START; i <= MSI_GENERIC_END; i++)
+ enable_irq(pdev->irq + i);
+ enable_irq(pdev->irq + MSI_DMA_READ);
+ enable_irq(pdev->irq + MSI_DMA_WRITE);
+ }
+#else
+ enable_irq(pdev->irq);
+#endif
+
+ dev_dbg(dev, "%s: exit\n", __func__);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(mnh_pci_resume);
+
+#if 0 /* TODO: implement error handlers below */
+static const struct pci_error_handlers mnh_pci_err_handler = {
+ .error_detected = mnh_pci_error_detected,
+ .slot_reset = mnh_pci_slot_reset,
+ .resume = mnh_pci_resume
+};
+#endif
+
+/*
+ * PCI driver structure
+ */
+static struct pci_driver mnh_pci_driver = {
+ .name = MODULE_NAME,
+ .id_table = mnh_pci_tbl,
+ .probe = mnh_pci_probe,
+ .remove = mnh_pci_remove,
+ .shutdown = mnh_pci_remove,
+};
+
+module_pci_driver(mnh_pci_driver);
+
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL,
+ PCI_DEVICE_ID_MNH,
+ mnh_pci_fixup);
+
+MODULE_AUTHOR("Intel Corporation");
+MODULE_DESCRIPTION("MNH PCI HOST DRIVER");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/misc/mnh/mnh-pcie.h b/drivers/misc/mnh/mnh-pcie.h
new file mode 100644
index 0000000..007c0c6
--- /dev/null
+++ b/drivers/misc/mnh/mnh-pcie.h
@@ -0,0 +1,484 @@
+/*
+ *
+ * MNH PCIe/DMA HOST Driver
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef __MNH_PCIE_HOST
+#define __MNH_PCIE_HOST
+#include <linux/dma-mapping.h>
+#include <linux/types.h>
+
+
+/*******************************************************************************
+ *
+ * Data structure
+ *
+ ******************************************************************************/
+
+#define DMA_MAX_CHAN 4
+
+/**
+ * DMA channel direction type
+ */
+enum mnh_dma_chan_dir_t {
+ DMA_EP2AP = 0, /**< EP2AP(WRITE_CH):To write data to AP from EP */
+ DMA_AP2EP /**< AP2EP(READ_CH):To read data from AP and send to EP*/
+
+};
+
+/**
+ * DMA transfer status type
+ * This is to indicate if DMA is transfer is successful or not
+ */
+enum mnh_dma_trans_status_t {
+ DMA_DONE = 0, /**< DONE: DMA DONE interrupt */
+ DMA_ABORT/**< ABORT: DMA ABORT interrupt */
+};
+
+/**
+ * DMA Channel status type
+ */
+enum mnh_dma_chan_status_t {
+ DMA_CHAN_UNKNOWN = 0, /**< Unknown */
+ DMA_CHAN_RUNNING, /**< Channel is running */
+ DMA_CHAN_HALTED, /**< Channel is halted */
+ DMA_CHAN_STOPPED, /**< Channel is stopped */
+ DMA_CHAN_QUEUING /**< Queuing. Not a real HW status */
+};
+
+/**
+ * DMA hardware error types.
+ * Defined in DMA_READ_ERR_STATUS_LOW_OFF/HIGH_OFF
+ */
+enum mnh_dma_error_t {
+ DMA_ERR_NONE, /**< No DMA error found */
+ DMA_ERR_WR, /**< The DMA Write Channel has received an error
+ * response from the AHB/AXI bus (or RTRGT1 interface
+ * when the AHB/AXI Bridge is not used) while reading
+ * data from it. It's fatal error. */
+ DMA_ERR_RD, /**< The DMA Read Channel has received an error response
+ * from the AHB/AXI bus (or RTRGT1 interface when the
+ * AHB/AXI Bridge is not used) while writing data to
+ * it. It's fatal error.*/
+ DMA_ERR_FETCH_LL,/**< The DMA Write/Read Channel has received an error
+ * response from the AHB/AXI bus (or RTRGT1 interface
+ * when the AHB/AXI Bridge is not used) while reading
+ * a Linked List Element from local memory. It's fatal
+ * error. */
+ DMA_ERR_UNSUPPORTED_RQST,
+ /**< The DMA Read Channel has received a PCIe
+ * Unsupported Request CPL status from the remote
+ * device in response to the MRd Request.*/
+ DMA_ERR_COMPLETER_ABORT,
+ /**< The DMA Read Channel has received a PCIe Completer
+ *Abort CPL status from the remote device in response
+ *to the MRd Request. Non-fatal error.
+ */
+ DMA_ERR_CPL_TIME_OUT,
+ /**< The DMA Read Channel has timed-out while waiting
+ * for the remote device to respond to the MRd Request,
+ * or a malformed CplD has been received. Non-fatal
+ * error. */
+ DMA_ERR_DATA_POISONING,
+ /**< The DMA Read Channel has detected data poisoning
+ * in the CPL from the remote device in response to the
+ * MRd Request. Non-fatal error. */
+};
+
+/**
+ * IRQ sent from AP to MNH
+ */
+enum mnh_irq_msg_t {
+ IRQ_INVAL = 0,
+ IRQ_START = 1,
+ IRQ_MSG_SENT = IRQ_START, /**< IRQ Msg generated interrupt */
+ IRQ_DMA_STATUS, /**< IRQ for DMA done/abort */
+ IRQ_APPDEFINED_1, /**< Application-defined */
+
+ /* Add additional IRQ above here */
+ IRQ_END = IRQ_APPDEFINED_1,
+};
+
+/* Interrupt(MSI) from MNH to AP */
+enum mnh_msi_msg_t {
+ MSI_INVAL = 0,
+ MSI_GENERIC_START,
+ MSI_MSG_SEND = MSI_GENERIC_START,
+ MSI_PET_WATCHDOG,
+ MSI_CRASH_DUMP,
+ MSI_BOOTSTRAP_SET,
+ MSI_APPDEFINED_1,
+ /* Add additional MSIs above here */
+ MSI_GENERIC_END = MSI_APPDEFINED_1,
+
+ MSI_DMA_READ = 8,
+ MSI_DMA_WRITE,
+ /* Add additional DMA related MSIs below here */
+
+ MSI_MAX = 31
+};
+
+/** Vendor message between AP to MNH */
+struct mnh_pcie_vm {
+ u32 vm; /**< vendor defined message */
+};
+
+/**
+ * Data structure defined for DMA block element information
+ */
+struct mnh_dma_element_t {
+ uint32_t len; /**< DMA transfer size */
+ uint64_t src_addr; /**< SAR - DMA Source Address */
+ uint64_t dst_addr; /**< DAR - DMA Destination Address */
+};
+
+/**
+ * Structure used for DMA Channel status
+ */
+struct mnh_dma_state_info_t {
+ uint8_t status; /**< dma chan status mnh_dma_chan_status_t) */
+ uint32_t err; /**< hw error status if there is any (mnh_dma_error_t) */
+ uint64_t xferred; /**< size of data have been transferred on ch */
+};
+
+/**
+ * Structure used for mnh_set_outbound() API
+ */
+struct mnh_outb_region {
+ uint8_t region; /**< iATU region to be programmed */
+ uint64_t base_address; /**< start of src buffer */
+ uint32_t limit_address; /**< end of src buffer */
+ uint64_t target_pcie_address; /**< dest address */
+};
+
+/* TODO: REMOVE BELOW INBOUND enum/structure LATER AFTER BRING UP */
+/** enum value used for mode setting of mth_set_inbound() API */
+enum mth_iatu_mode_t {
+ ADDR_MATCH = 0,
+ BAR_MATCH
+};
+
+/** memmode values for struct mnh_inb_window */
+#define IATU_MEM 0x0
+#define IATU_IO 0x02
+
+/** structure used for mth_set_inbound() API */
+struct mnh_inb_window {
+ uint8_t mode; /**< type of region */
+ uint32_t bar; /**< BAR ignored in addr mode */
+ uint8_t region; /**< iATU region to be programmed */
+ uint8_t memmode;
+ uint64_t base_pcie_address; /**< start of src buffer
+ *ignored in BAR mode */
+ uint32_t limit_pcie_address; /**< end of src buffer
+ *ignored in BAR mode*/
+ uint64_t target_mth_address; /**< dest address */
+};
+
+/**
+ * Structure of scatter gather list entry
+ */
+struct mnh_sg_entry {
+ phys_addr_t paddr; /**< Physical address */
+ size_t size; /**< size of entry */
+};
+
+enum mnh_hotplug_event_t {
+ MNH_HOTPLUG_IN = 0,
+ MNH_HOTPLUG_OUT,
+};
+
+struct mnh_sg_list {
+ struct page **mypage;
+ struct scatterlist *sc_list;
+ int n_num;
+ int length;
+ enum mnh_dma_chan_dir_t dir;
+ struct dma_buf *dma_buf;
+ struct dma_buf_attachment *attach;
+ struct sg_table *sg_table;
+};
+
+typedef int (*irq_cb_t)(uint32_t irq);
+typedef int (*irq_dma_cb_t)(uint8_t chan, enum mnh_dma_chan_dir_t dir,
+ enum mnh_dma_trans_status_t status);
+typedef int (*hotplug_cb_t)(enum mnh_hotplug_event_t event, void *param);
+
+/*******************************************************************************
+ *
+ * APIs
+ *
+ ******************************************************************************/
+
+
+/**
+ * API to read data from PCIE configuration space
+ * @param[in] offset offset into PCIE configuration space(BAR0)
+ * @param[in] len buffer size : supported size is 4
+ * @param[in] data data to be read into. Client must allocate
+ * the buffer for reading and pass the pointer
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ * an error happens during the process.(mnh_dma_chan_status_t)
+ */
+int mnh_pcie_config_read(uint32_t offset, uint32_t len, uint32_t *data);
+
+/**
+ * API to write data to PCIE configuration space
+ * @param[in] offset offset into PCIE configuration space(BAR0)
+ * @param[in] len buffer size : supported size is 4
+ * @param[in] data data to be written
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_pcie_config_write(uint32_t offset, uint32_t len, uint32_t data);
+
+/**
+ * API to read data from MNH configuration space
+ * @param[in] offset offset into MNH Address space(BAR2)
+ * @param[in] len buffer size : supported size is 1,2,4
+ * @param[in] data data to be read into. Client must allocate
+ * the buffer for reading and pass the pointer
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_config_read(uint32_t offset, uint32_t len, uint32_t *data);
+
+/**
+ * API to write data to MNH configuration space
+ * @param[in] offset offset into MNH Address space(BAR2)
+ * @param[in] len buffer size : supported size could be 1,2,4
+ * @param[in] data pointer to the data to be wrriten.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_config_write(uint32_t offset, uint32_t len, uint32_t data);
+
+/**
+ * API to read data from MNH DDR space
+ * @param[in] offset offset into MNH DRAM space(BAR4)
+ * @param[in] len buffer size
+ * @param[in] data data to be read into. Client must allocate
+ * the buffer for reading and pass the pointer
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ * @note Usually DMA will be used to read/write from DDR space in MNH
+ * This maybe only used for debugging purpose
+ */
+int mnh_ddr_read(uint32_t offset, uint32_t len, void *data);
+
+/**
+ * API to write data to MNH DDR space
+ * @param[in] offset offset into MNH DRAM space(BAR4)
+ * @param[in] len buffer size
+ * @param[in] data pointer to the data to be wrriten.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ * @note Usually DMA will be used to read/write from DDR space in MNH
+ * This maybe only used for debugging purpose
+ */
+int mnh_ddr_write(uint32_t offset, uint32_t len, void *data);
+
+/**
+ * API to generate IRQ from AP to MNH
+ * @param[in] irq IRQ ID to be sent (mnh_irq_msg_t)
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_send_irq(enum mnh_irq_msg_t irq);
+
+/**
+ * API to register IRQ callback to receive msg delivery from MNH
+ * @param[in] msg_cb interrupt handler for MSI received from MNH, pass NULL
+ * to de-register
+ * @param[in] vm_cb interrupt handler for Vendor Message received from MNH,
+ * pass NULL to de-register
+ * @param[in] dma_cb interrupt handler for DMA message(DMA_DONE/ABORT) received
+ * from MNH, pass NULL to de-register
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_reg_irq_callback(irq_cb_t msg_cb, irq_cb_t vm_cb, irq_dma_cb_t dma_cb);
+
+/**
+ * API to send Vendor specific message from AP to MNH
+ * @param[in] msg Vendor message to be sent include msg_id
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_send_vendor_msg(struct mnh_pcie_vm msg);
+
+/**
+ * API to get the base ring buffer address of MNH
+ * @param[out] rb_base ring buffer address
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_get_rb_base(uint64_t *rb_base);
+
+/**
+ * API to set outbound region
+ * @param outb[in] - iATU outbound region information
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_set_outbound_iatu(struct mnh_outb_region *outb);
+
+/**
+ * API to set inbound region
+ * @param outb[in] - iATU outbound region information
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_set_inbound_iatu(struct mnh_inb_window *inb);
+
+/**
+ * API to abort DMA transfer on specific channel
+ * @param[in] chan The channel number for DMA transfer abort
+ * @param[in] dir Direction of DMA channel, READ or WRITE
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_dma_abort(uint8_t chan, enum mnh_dma_chan_dir_t dir);
+
+/**
+ * API to resume DMA transfer on specific channel
+ * @param[in] chan The channel number for DMA transfer resume
+ * @param[in] dir Direction of DMA channel, READ or WRITE
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_dma_resume(uint8_t chan, enum mnh_dma_chan_dir_t dir);
+
+/**
+ * API to read/write single block on specific channel.
+ * @param[in] chan The channel number for DMA transfer to be performed
+ * @param[in] dir The channel direction (read/write)
+ * @param[in] blk one dma element info which include SAR(Source Address),
+ * DAR(Destination Address) and transfer size.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_dma_sblk_start(uint8_t chan, enum mnh_dma_chan_dir_t dir,
+ struct mnh_dma_element_t *blk);
+
+/**
+ * API to build Scatter Gather list to do Multi-block DMA transfer for a user
+ * buffer
+ * @param[in] dmadest Starting virtual addr of the DMA destination
+ * @param[in] size Totalsize of the transfer in bytes
+ * @param[out] sg Array of maxsg pointers to struct mnh_sg_entry, allocated
+ * and filled out by this routine.
+ * @param[out] sgl pointer of Scatter gather list which has information of
+ * page list, scatter gather list and num of its entries.
+ * @return The number of sg[] entries filled out by the routine, negative if
+ * overflow or sg[] not allocated.
+ */
+int mnh_sg_build(void *dmadest, size_t size, struct mnh_sg_entry **sg,
+ struct mnh_sg_list *sgl);
+
+/**
+ * API to release scatter gather list for a user buffer
+ * @param[in] *sgl pointer to the scatter gather list that was built during
+ * mnh_sg_build
+ * @return 0 for SUCCESS
+ */
+int mnh_sg_destroy(struct mnh_sg_list *sgl);
+
+/**
+ * API to verify that a scatter gather list is valid
+ * @param[in] *sg pointer to the scatter gather list that was built during
+ * mnh_sg_build
+ * @param[in] size size of the list in bytes
+ * @param[in] sgl pointer to the scatter gather local data, if any
+ * @return 0 for SUCCESS
+ */
+int mnh_sg_verify(struct mnh_sg_entry *sg, size_t size,
+ struct mnh_sg_list *sgl);
+
+/**
+ * API to build a scatter-gather list for multi-block DMA transfer for a
+ * dma_buf
+ * @param[in] fd Handle of dma_buf passed from user
+ * @param[in] off Offset within DMA buffer from which transfer should start.
+ * @param[in] size Size, in bytes, of transfer.
+ * @param[in] width Width in bytes of transfer.
+ * @param[in] stride Stride in bytes. Must be greater or equal to width.
+ * @param[out] sg Array of maxsg pointers to struct mnh_sg_entry, allocated
+ * and filled out by this routine.
+ * @param[out] sgl pointer of Scatter gather list which has information of
+ * scatter gather list and num of its entries.
+ * @return 0 on SUCCESS
+ * negative on failure
+ */
+int mnh_sg_retrieve_from_dma_buf(int fd, uint32_t off, uint32_t size,
+ uint32_t width, uint32_t stride, struct mnh_sg_entry **sg,
+ struct mnh_sg_list *sgl);
+
+/**
+ * API to release a scatter-gather list for a dma_buf
+ * @param[in] *sgl pointer to the scatter gather list that was built during
+ * mnh_sg_retrieve_from_dma_buf
+ * @return 0 for SUCCESS
+ */
+int mnh_sg_release_from_dma_buf(struct mnh_sg_list *sgl);
+
+/**
+ * API to read/write multi blocks on specific channel.
+ * Multi block read/write are based on Linked List(Transfer List) built by MNH.
+ * @param[in] chan The channel number for DMA transfer to be performed
+ * @param[in] dir The channel direction (read/write)
+ * @param[in] start_addr Physical start_addr(in MNH) where transfer list is
+ * stored.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+int mnh_dma_mblk_start(uint8_t chan, enum mnh_dma_chan_dir_t dir,
+ phys_addr_t *start_addr);
+
+/**
+ * API to get the current status of a specified channel
+ * @param[in] chan The channel number
+ * @param[in] dir The direction of channel.
+ * @param[out] info chan information includes channel status, transferred
+ * size, error status, and etc.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+
+int mnh_dma_get_status(uint8_t chan, enum mnh_dma_chan_dir_t dir,
+ struct mnh_dma_state_info_t *info);
+
+/**
+ * Allocate DMA coherent host memory for shared PCIe/CPU access. Must be
+ * called in a sleepable context.
+ * @param[in] size requested size of the memory region in bytes
+ * @param[out] dma_addr returns the DMA address of the allocated memory
+ * @return cpu virtual address of the allocated memory
+ */
+void *mnh_alloc_coherent(size_t size, dma_addr_t *dma_addr);
+
+/**
+ * Free DMA coherent memory allocated by mnh_alloc_coherent()
+ * @param[in] size size of the memory region in bytes
+ * @param[in] cpu_addr cpu virtual address of the memory region
+ * @param[in] dma_addr DMA address of the memory region
+ */
+void mnh_free_coherent(size_t size, void *cpu_addr, dma_addr_t dma_addr);
+
+/**
+ * Map host memory for access by MNH PCIe host
+ * @param[in] cpu_addr cpu virtual address of the memory region
+ * @param[in] size size of the memory region in bytes
+ * @param[in] direction DMA direction DMA_TO_DEVICE, etc.
+ * @return DMA address returned by dma_map_single(), or zero for error
+ */
+dma_addr_t mnh_map_mem(
+ void *cpu_addr, size_t size, enum dma_data_direction direction);
+
+/**
+ * Unmap host memory from MNH PCIe host access
+ * @param[in] dma_addr DMA address of the memory returned by mnh_map_mem()
+ * @param[in] size size of the memory region in bytes
+ * @param[in] direction DMA direction DMA_TO_DEVICE, etc.
+ */
+void mnh_unmap_mem(
+ dma_addr_t dma_addr, size_t size, enum dma_data_direction direction);
+
+int mnh_pci_suspend(void);
+int mnh_pci_resume(void);
+#endif /* __MNH_PCIE_HOST */
diff --git a/drivers/misc/mnh/mnh-pwr.c b/drivers/misc/mnh/mnh-pwr.c
new file mode 100644
index 0000000..47edbfb
--- /dev/null
+++ b/drivers/misc/mnh/mnh-pwr.c
@@ -0,0 +1,871 @@
+/*
+ *
+ * MNH PWR APIs
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+/* #define DEBUG */
+
+#include <linux/clk.h>
+#include <linux/device.h>
+#include <linux/gpio/consumer.h>
+#include <linux/gpio/machine.h>
+#include <linux/interrupt.h>
+#include <linux/msm_pcie.h>
+#include <linux/regulator/consumer.h>
+
+#include "mnh-pcie.h"
+#include "mnh-pwr.h"
+#include "mnh-sm.h"
+
+#define MNH_PCIE_RC_INDEX 1
+#define MNH_PCIE_VENDOR_ID 0x8086
+#define MNH_PCIE_DEVICE_ID 0x3140
+
+#define PM_OPT_SUSPEND MSM_PCIE_CONFIG_LINKDOWN
+#define PM_OPT_RESUME MSM_PCIE_CONFIG_NO_CFG_FREE
+
+struct mnh_pwr_data {
+ struct platform_device *pdev;
+ struct device *dev;
+
+ /* regulators */
+ struct regulator *smps1_supply;
+ struct regulator *smps2_supply;
+ struct regulator *ldo2_supply;
+ struct regulator *ldo1_supply;
+ struct notifier_block smps1_nb;
+ struct notifier_block smps2_nb;
+ struct notifier_block ldo1_nb;
+ struct notifier_block ldo2_nb;
+
+ /* clocks */
+ struct clk *ref_clk;
+ bool ref_clk_enabled;
+
+ /* pins */
+ struct gpio_desc *soc_pwr_good_pin;
+
+ /* pcie device */
+ struct pci_dev *pcidev;
+ struct pci_saved_state *pristine_state;
+ struct msm_pcie_register_event pci_link_event;
+
+ struct work_struct shutdown_work;
+
+ bool pcie_failure;
+
+ enum mnh_pwr_state state;
+
+ struct mutex lock;
+};
+
+static struct mnh_pwr_data *mnh_pwr;
+
+static void mnh_pwr_shutdown_work(struct work_struct *data)
+{
+ dev_err(mnh_pwr->dev, "%s: begin emergency power down\n", __func__);
+
+ mnh_pwr_set_state(MNH_PWR_S4);
+ mnh_sm_pwr_error_cb();
+}
+
+static int mnh_pwr_smps1_notifier_cb(struct notifier_block *nb,
+ unsigned long event, void *cookie)
+{
+ dev_dbg(mnh_pwr->dev, "%s: received event %ld\n", __func__, event);
+
+ mnh_pwr->pcie_failure = true;
+
+ /* force emergency shutdown if regulator output has failed */
+ if (event & REGULATOR_EVENT_FAIL) {
+ dev_err(mnh_pwr->dev,
+ "%s: smps1 supply has failed, forcing shutdown\n",
+ __func__);
+
+ if (mnh_pwr->state != MNH_PWR_S4)
+ schedule_work(&mnh_pwr->shutdown_work);
+ }
+
+ return 0;
+}
+
+static int mnh_pwr_smps2_notifier_cb(struct notifier_block *nb,
+ unsigned long event, void *cookie)
+{
+ dev_dbg(mnh_pwr->dev, "%s: received event %ld\n", __func__, event);
+
+ mnh_pwr->pcie_failure = true;
+
+ /* force emergency shutdown if regulator output has failed */
+ if (event & REGULATOR_EVENT_FAIL) {
+ dev_err(mnh_pwr->dev,
+ "%s: smps2 supply has failed, forcing shutdown\n",
+ __func__);
+
+ if (mnh_pwr->state != MNH_PWR_S4)
+ schedule_work(&mnh_pwr->shutdown_work);
+ }
+
+ return 0;
+}
+
+static int mnh_pwr_ldo1_notifier_cb(struct notifier_block *nb,
+ unsigned long event, void *cookie)
+{
+ dev_dbg(mnh_pwr->dev, "%s: received event %ld\n", __func__, event);
+
+ mnh_pwr->pcie_failure = true;
+
+ /* force emergency shutdown if regulator output has failed */
+ if (event & REGULATOR_EVENT_FAIL) {
+ dev_err(mnh_pwr->dev,
+ "%s: ldo1 supply has failed, forcing shutdown\n",
+ __func__);
+
+ if (mnh_pwr->state != MNH_PWR_S4)
+ schedule_work(&mnh_pwr->shutdown_work);
+ }
+
+ return 0;
+}
+
+static int mnh_pwr_ldo2_notifier_cb(struct notifier_block *nb,
+ unsigned long event, void *cookie)
+{
+ dev_dbg(mnh_pwr->dev, "%s: received event %ld\n", __func__, event);
+
+ mnh_pwr->pcie_failure = true;
+
+ /* force emergency shutdown if regulator output has failed */
+ if (event & REGULATOR_EVENT_FAIL) {
+ dev_err(mnh_pwr->dev,
+ "%s: ldo2 supply has failed, forcing shutdown\n",
+ __func__);
+
+ if (mnh_pwr->state != MNH_PWR_S4)
+ schedule_work(&mnh_pwr->shutdown_work);
+ }
+
+ return 0;
+}
+
+void mnh_pwr_pcie_link_state_cb(struct msm_pcie_notify *notify)
+{
+ struct mnh_pwr_data *mnh_pwr = notify->data;
+
+ switch (notify->event) {
+ case MSM_PCIE_EVENT_LINKDOWN:
+ dev_err(mnh_pwr->dev,
+ "%s: PCIe link is down, forcing power down\n",
+ __func__);
+
+ mnh_pwr->pcie_failure = true;
+
+ /* force emergency shutdown */
+ schedule_work(&mnh_pwr->shutdown_work);
+ break;
+ default:
+ dev_err(mnh_pwr->dev,
+ "%s: received invalid pcie link state event (%d)\n",
+ __func__, notify->event);
+ break;
+ }
+}
+
+static int mnh_pwr_pcie_enumerate(void)
+{
+ struct pci_dev *pcidev = NULL;
+ int ret;
+
+ /* enumerate PCIE */
+ ret = msm_pcie_enumerate(MNH_PCIE_RC_INDEX);
+ if (ret < 0) {
+ dev_err(mnh_pwr->dev, "%s: pcie enumeration failed (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ /* search for PCIE device in our domain */
+ do {
+ pcidev = pci_get_device(MNH_PCIE_VENDOR_ID, MNH_PCIE_DEVICE_ID, pcidev);
+ if (!pcidev)
+ break;
+
+ if (pci_domain_nr(pcidev->bus) == MNH_PCIE_RC_INDEX)
+ break;
+ } while (true);
+ if (!pcidev) {
+ dev_err(mnh_pwr->dev, "%s: could not find mnh device\n",
+ __func__);
+ return -ENODEV;
+ }
+
+ /* save current state in pcidev */
+ ret = pci_save_state(pcidev);
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: pci_save_state failed (%d)\n",
+ __func__, ret);
+ pci_dev_put(pcidev);
+ return ret;
+ }
+
+ /* store saved state so we can recall it after resume */
+ mnh_pwr->pristine_state = pci_store_saved_state(pcidev);
+ if (!mnh_pwr->pristine_state) {
+ dev_err(mnh_pwr->dev,
+ "%s: pci_store_saved_state failed\n",
+ __func__);
+ pci_dev_put(pcidev);
+ return ret;
+ }
+
+ /* save device to driver struct */
+ mnh_pwr->pcidev = pcidev;
+
+ return 0;
+}
+
+static int mnh_pwr_pcie_suspend(void)
+{
+ struct pci_dev *pcidev = mnh_pwr->pcidev;
+ int ret;
+
+ if (!pcidev)
+ return -ENODEV;
+
+ /* suspend the driver state */
+ ret = mnh_pci_suspend();
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: mnh_pci_suspend failed (%d)\n",
+ __func__, ret);
+ }
+
+ if (mnh_pwr->pcie_failure) {
+ /* call the platform driver to update link status */
+ ret = msm_pcie_pm_control(MSM_PCIE_SUSPEND, pcidev->bus->number,
+ pcidev, NULL,
+ PM_OPT_SUSPEND | MSM_PCIE_CONFIG_NO_CFG_RESTORE);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: msm_pcie_pm_control(suspend) failed (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ mnh_pwr->pcie_failure = false;
+ } else {
+ /* prepare the root complex and endpoint for going to suspend */
+ ret = pci_prepare_to_sleep(pcidev);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: pci_prepare_to_sleep failed (%d)\n",
+ __func__, ret);
+ }
+
+ /* call the platform driver to suspend PCIe link */
+ ret = msm_pcie_pm_control(MSM_PCIE_SUSPEND, pcidev->bus->number,
+ pcidev, NULL, PM_OPT_SUSPEND);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: msm_pcie_pm_control(suspend) failed (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int mnh_pwr_pcie_resume(void)
+{
+ struct pci_dev *pcidev = mnh_pwr->pcidev;
+ int ret;
+
+ /* check for valid pcidev */
+ if (!pcidev) {
+ /* enumerate pci device */
+ ret = mnh_pwr_pcie_enumerate();
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: mnh_pwr_pcie_enumerate failed (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ /* save the newly enumerated device to the local copy */
+ pcidev = mnh_pwr->pcidev;
+
+ /* register for link down events so we can handle them */
+ mnh_pwr->pci_link_event.events = MSM_PCIE_EVENT_LINKDOWN;
+ mnh_pwr->pci_link_event.user = pcidev;
+ mnh_pwr->pci_link_event.callback = mnh_pwr_pcie_link_state_cb;
+ mnh_pwr->pci_link_event.notify.data = mnh_pwr;
+ ret = msm_pcie_register_event(&mnh_pwr->pci_link_event);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: msm_pcie_register_event failed (%d)\n",
+ __func__, ret);
+ }
+ } else {
+ ret = msm_pcie_pm_control(MSM_PCIE_RESUME, pcidev->bus->number,
+ pcidev, NULL, PM_OPT_RESUME);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: msm_pcie_pm_control(resume) failed (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ /* prepare the root complex and endpoint */
+ ret = pci_back_from_sleep(pcidev);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: pci_back_from_sleep failed (%d)\n",
+ __func__, ret);
+ goto fail_pcie_resume_awake;
+ }
+
+ /* load the saved state in the device buffer */
+ ret = pci_load_saved_state(pcidev, mnh_pwr->pristine_state);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: pci_load_saved_state failed (%d)\n",
+ __func__, ret);
+ goto fail_pcie_resume_load_state;
+ }
+
+ /* apply the saved state to the device */
+ pci_restore_state(pcidev);
+
+ /* resume the driver state */
+ ret = mnh_pci_resume();
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: mnh_pci_resume failed (%d)\n",
+ __func__, ret);
+ goto fail_pcie_resume_mnh_init;
+ }
+ }
+
+ mnh_pwr->pcie_failure = false;
+
+ return 0;
+
+fail_pcie_resume_mnh_init:
+fail_pcie_resume_load_state:
+ pci_prepare_to_sleep(pcidev);
+fail_pcie_resume_awake:
+ msm_pcie_pm_control(MSM_PCIE_SUSPEND, pcidev->bus->number, pcidev,
+ NULL, PM_OPT_SUSPEND);
+
+ return ret;
+}
+
+static int mnh_pwr_down(void)
+{
+ int ret;
+
+ if ((mnh_pwr->state != MNH_PWR_S3) &&
+ (mnh_sm_get_boot_mode() == MNH_BOOT_MODE_PCIE)) {
+ /* suspend pcie link */
+ ret = mnh_pwr_pcie_suspend();
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to suspend pcie link (%d)\n",
+ __func__, ret);
+ goto fail_pwr_down_pcie;
+ }
+ } else {
+ /* assert reset */
+ msm_pcie_set_reset(MNH_PCIE_RC_INDEX, true);
+ }
+
+ /* deassert soc_pwr_good */
+ gpiod_set_value_cansleep(mnh_pwr->soc_pwr_good_pin, 0);
+
+ /* disable clocks */
+ if (mnh_pwr->ref_clk_enabled) {
+ clk_disable_unprepare(mnh_pwr->ref_clk);
+ mnh_pwr->ref_clk_enabled = false;
+ }
+
+ /* disable supplies: smps2 -> smps1 -> ldo1 -> ldo2 */
+ if (regulator_is_enabled(mnh_pwr->smps2_supply)) {
+ ret = regulator_disable(mnh_pwr->smps2_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to disable smps2 (%d)\n",
+ __func__, ret);
+ goto fail_pwr_down_smps2;
+ }
+ }
+
+ if (regulator_is_enabled(mnh_pwr->smps1_supply)) {
+ ret = regulator_disable(mnh_pwr->smps1_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to disable smps1 (%d)\n",
+ __func__, ret);
+ goto fail_pwr_down_smps1;
+ }
+ }
+
+ if (regulator_is_enabled(mnh_pwr->ldo1_supply)) {
+ ret = regulator_disable(mnh_pwr->ldo1_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to disable ldo1 (%d)\n",
+ __func__, ret);
+ goto fail_pwr_down_ldo1;
+ }
+ }
+
+ if (regulator_is_enabled(mnh_pwr->ldo2_supply)) {
+ ret = regulator_disable(mnh_pwr->ldo2_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to disable ldo2 (%d)\n",
+ __func__, ret);
+ goto fail_pwr_down_ldo2;
+ }
+ }
+
+ mnh_pwr->state = MNH_PWR_S4;
+
+ return 0;
+
+fail_pwr_down_pcie:
+ gpiod_set_value_cansleep(mnh_pwr->soc_pwr_good_pin, 0);
+ if (mnh_pwr->ref_clk_enabled) {
+ clk_disable_unprepare(mnh_pwr->ref_clk);
+ mnh_pwr->ref_clk_enabled = false;
+ }
+ if (regulator_is_enabled(mnh_pwr->smps2_supply))
+ regulator_disable(mnh_pwr->smps2_supply);
+fail_pwr_down_smps2:
+ if (regulator_is_enabled(mnh_pwr->smps1_supply))
+ regulator_disable(mnh_pwr->smps1_supply);
+fail_pwr_down_smps1:
+ if (regulator_is_enabled(mnh_pwr->ldo1_supply))
+ regulator_disable(mnh_pwr->ldo1_supply);
+fail_pwr_down_ldo1:
+ if (regulator_is_enabled(mnh_pwr->ldo2_supply))
+ regulator_disable(mnh_pwr->ldo2_supply);
+fail_pwr_down_ldo2:
+
+ dev_err(mnh_pwr->dev,
+ "%s: force shutdown because of powerdown failure (%d)\n",
+ __func__, ret);
+
+ mnh_pwr->state = MNH_PWR_S4;
+
+ return ret;
+}
+
+static int mnh_pwr_suspend(void)
+{
+ int ret;
+
+ if (mnh_sm_get_boot_mode() == MNH_BOOT_MODE_PCIE) {
+ /* suspend pcie link */
+ ret = mnh_pwr_pcie_suspend();
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to suspend pcie link (%d)\n",
+ __func__, ret);
+ goto fail_pwr_suspend_pcie;
+ }
+ } else {
+ /* assert reset */
+ msm_pcie_set_reset(MNH_PCIE_RC_INDEX, true);
+ }
+
+ /* deassert soc_pwr_good */
+ gpiod_set_value_cansleep(mnh_pwr->soc_pwr_good_pin, 0);
+
+ /* disable clocks */
+ if (mnh_pwr->ref_clk_enabled) {
+ clk_disable_unprepare(mnh_pwr->ref_clk);
+ mnh_pwr->ref_clk_enabled = false;
+ }
+
+ /* disable core supplies */
+ ret = regulator_disable(mnh_pwr->smps1_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: failed to disable smps1 (%d)\n", __func__, ret);
+ goto fail_pwr_suspend_regulators;
+ }
+
+ regulator_disable(mnh_pwr->ldo1_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: failed to disable ldo1 (%d)\n", __func__, ret);
+ goto fail_pwr_suspend_regulators;
+ }
+
+ mnh_pwr->state = MNH_PWR_S3;
+
+ return 0;
+
+fail_pwr_suspend_pcie:
+ gpiod_set_value_cansleep(mnh_pwr->soc_pwr_good_pin, 0);
+ if (mnh_pwr->ref_clk_enabled) {
+ clk_disable_unprepare(mnh_pwr->ref_clk);
+ mnh_pwr->ref_clk_enabled = false;
+ }
+fail_pwr_suspend_regulators:
+ if (regulator_is_enabled(mnh_pwr->smps2_supply))
+ regulator_disable(mnh_pwr->smps2_supply);
+ if (regulator_is_enabled(mnh_pwr->smps1_supply))
+ regulator_disable(mnh_pwr->smps1_supply);
+ if (regulator_is_enabled(mnh_pwr->ldo1_supply))
+ regulator_disable(mnh_pwr->ldo1_supply);
+ if (regulator_is_enabled(mnh_pwr->ldo2_supply))
+ regulator_disable(mnh_pwr->ldo2_supply);
+
+ dev_err(mnh_pwr->dev,
+ "%s: force shutdown because of suspend failure (%d)\n",
+ __func__, ret);
+
+ mnh_pwr->state = MNH_PWR_S4;
+
+ return ret;
+}
+
+static int mnh_pwr_partial(void)
+{
+ int ret;
+
+ /* disable DRAM core supply */
+ if (regulator_is_enabled(mnh_pwr->smps2_supply)) {
+ ret = regulator_disable(mnh_pwr->smps2_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to disable smps2 (%d)\n",
+ __func__, ret);
+ }
+ }
+
+ /* disable DRAM I/O supply */
+ if (regulator_is_enabled(mnh_pwr->ldo2_supply)) {
+ ret = regulator_disable(mnh_pwr->ldo2_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to disable ldo2 (%d)\n",
+ __func__, ret);
+ }
+ }
+
+ mnh_pwr->state = MNH_PWR_S1;
+
+ return 0;
+}
+
+static int mnh_pwr_up(enum mnh_pwr_state next_state)
+{
+ int ret;
+
+ /* enable supplies */
+ /* ldo2 -> ldo1 -> smps1 -> smps2 */
+ if (mnh_pwr->state == MNH_PWR_S4) {
+ ret = regulator_enable(mnh_pwr->ldo2_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: failed to enable ldo2 (%d)\n",
+ __func__, ret);
+ goto fail_pwr_up_regulators;
+ }
+ }
+
+ ret = regulator_enable(mnh_pwr->ldo1_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to enable ldo1 (%d)\n",
+ __func__, ret);
+ goto fail_pwr_up_regulators;
+ }
+
+ ret = regulator_enable(mnh_pwr->smps1_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to enable ldo1 (%d)\n",
+ __func__, ret);
+ goto fail_pwr_up_regulators;
+ }
+
+ if (mnh_pwr->state == MNH_PWR_S4) {
+ ret = regulator_enable(mnh_pwr->smps2_supply);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: failed to enable smps2 (%d)\n",
+ __func__, ret);
+ goto fail_pwr_up_regulators;
+ }
+ }
+
+ /* turn on clocks */
+ if (!mnh_pwr->ref_clk_enabled) {
+ ret = clk_prepare_enable(mnh_pwr->ref_clk);
+ if (ret) {
+ dev_err(mnh_pwr->dev,
+ "%s: failed to enable ref clk (%d)\n",
+ __func__, ret);
+ goto fail_pwr_up_ref_clk;
+ }
+ mnh_pwr->ref_clk_enabled = true;
+ }
+
+ /* assert soc_pwr_good */
+ gpiod_set_value_cansleep(mnh_pwr->soc_pwr_good_pin, 1);
+
+ /* give the PLLs some time to initialize */
+ udelay(60);
+
+ if (mnh_sm_get_boot_mode() == MNH_BOOT_MODE_PCIE) {
+ /* resume pcie link */
+ ret = mnh_pwr_pcie_resume();
+ if (ret) {
+ dev_err(mnh_pwr->dev, "%s: failed to resume pcie link (%d)\n",
+ __func__, ret);
+ goto fail_pwr_up_pcie;
+ }
+ } else {
+ /* deassert reset */
+ msm_pcie_set_reset(MNH_PCIE_RC_INDEX, false);
+ }
+
+ mnh_pwr->state = next_state;
+
+ return 0;
+
+fail_pwr_up_pcie:
+ gpiod_set_value_cansleep(mnh_pwr->soc_pwr_good_pin, 0);
+ if (mnh_pwr->ref_clk_enabled) {
+ clk_disable_unprepare(mnh_pwr->ref_clk);
+ mnh_pwr->ref_clk_enabled = false;
+ }
+fail_pwr_up_ref_clk:
+fail_pwr_up_regulators:
+ if (regulator_is_enabled(mnh_pwr->smps2_supply))
+ regulator_disable(mnh_pwr->smps2_supply);
+ if (regulator_is_enabled(mnh_pwr->smps1_supply))
+ regulator_disable(mnh_pwr->smps1_supply);
+ if (regulator_is_enabled(mnh_pwr->ldo1_supply))
+ regulator_disable(mnh_pwr->ldo1_supply);
+ if (regulator_is_enabled(mnh_pwr->ldo2_supply))
+ regulator_disable(mnh_pwr->ldo2_supply);
+
+ dev_err(mnh_pwr->dev,
+ "%s: force shutdown because of power up failure (%d)\n",
+ __func__, ret);
+
+ mnh_pwr->state = MNH_PWR_S4;
+
+ return ret;
+}
+
+static int mnh_pwr_get_resources(void)
+{
+ struct platform_device *pdev = mnh_pwr->pdev;
+ struct device *dev = mnh_pwr->dev;
+ int ret;
+
+ /* request supplies */
+ mnh_pwr->smps1_supply = devm_regulator_get(&pdev->dev, "s2mpb04_smps1");
+ if (IS_ERR(mnh_pwr->smps1_supply)) {
+ dev_err(dev, "%s: failed to get smps1 supply (%ld)\n",
+ __func__, PTR_ERR(mnh_pwr->smps1_supply));
+ return PTR_ERR(mnh_pwr->smps1_supply);
+ }
+
+ mnh_pwr->smps2_supply = devm_regulator_get(&pdev->dev, "s2mpb04_smps2");
+ if (IS_ERR(mnh_pwr->smps2_supply)) {
+ dev_err(dev, "%s: failed to get smps2 supply (%ld)\n",
+ __func__, PTR_ERR(mnh_pwr->smps2_supply));
+ return PTR_ERR(mnh_pwr->smps2_supply);
+ }
+
+ mnh_pwr->ldo1_supply = devm_regulator_get(&pdev->dev,
+ "s2mpb04_ldo1");
+ if (IS_ERR(mnh_pwr->ldo1_supply)) {
+ dev_err(dev, "%s: failed to get ldo1 supply (%ld)\n",
+ __func__, PTR_ERR(mnh_pwr->ldo1_supply));
+ return PTR_ERR(mnh_pwr->ldo1_supply);
+ }
+
+ mnh_pwr->ldo2_supply = devm_regulator_get(&pdev->dev,
+ "s2mpb04_ldo2");
+ if (IS_ERR(mnh_pwr->ldo2_supply)) {
+ dev_err(dev, "%s: failed to get ldo2 supply (%ld)\n",
+ __func__, PTR_ERR(mnh_pwr->ldo2_supply));
+ return PTR_ERR(mnh_pwr->ldo2_supply);
+ }
+
+ /* register the notifier for each of the supplies */
+ mnh_pwr->smps1_nb.notifier_call = mnh_pwr_smps1_notifier_cb;
+ ret = devm_regulator_register_notifier(mnh_pwr->smps1_supply,
+ &mnh_pwr->smps1_nb);
+ if (ret) {
+ dev_err(dev,
+ "%s: failed to register notifier block for smps1 supply (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ mnh_pwr->smps2_nb.notifier_call = mnh_pwr_smps2_notifier_cb;
+ ret = devm_regulator_register_notifier(mnh_pwr->smps2_supply,
+ &mnh_pwr->smps2_nb);
+ if (ret) {
+ dev_err(dev,
+ "%s: failed to register notifier block for smps2 supply (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ mnh_pwr->ldo1_nb.notifier_call = mnh_pwr_ldo1_notifier_cb;
+ ret = devm_regulator_register_notifier(mnh_pwr->ldo1_supply,
+ &mnh_pwr->ldo1_nb);
+ if (ret) {
+ dev_err(dev,
+ "%s: failed to register notifier block for ldo1 supply (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ mnh_pwr->ldo2_nb.notifier_call = mnh_pwr_ldo2_notifier_cb;
+ ret = devm_regulator_register_notifier(mnh_pwr->ldo2_supply,
+ &mnh_pwr->ldo2_nb);
+ if (ret) {
+ dev_err(dev,
+ "%s: failed to register notifier block for ldo2 supply (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ /* request gpio descriptors */
+ mnh_pwr->soc_pwr_good_pin = devm_gpiod_get(&pdev->dev, "soc-pwr-good",
+ GPIOD_OUT_LOW);
+ if (IS_ERR(mnh_pwr->soc_pwr_good_pin)) {
+ dev_err(dev, "%s: could not get soc_pwr_good gpio (%ld)\n",
+ __func__, PTR_ERR(mnh_pwr->soc_pwr_good_pin));
+ return PTR_ERR(mnh_pwr->soc_pwr_good_pin);
+ }
+
+ /* request clocks */
+ mnh_pwr->ref_clk = devm_clk_get(&pdev->dev, "ref_clk");
+ if (IS_ERR(mnh_pwr->ref_clk)) {
+ dev_err(dev, "%s: could not get ref clk (%ld)\n", __func__,
+ PTR_ERR(mnh_pwr->ref_clk));
+ return PTR_ERR(mnh_pwr->ref_clk);
+ }
+
+ return 0;
+}
+
+int mnh_pwr_set_state(enum mnh_pwr_state system_state)
+{
+ int ret = 0;
+ enum mnh_pwr_state curr_state = mnh_pwr_get_state();
+
+ dev_dbg(mnh_pwr->dev, "%s req: %d, current: %d\n", __func__,
+ system_state, curr_state);
+
+ mutex_lock(&mnh_pwr->lock);
+
+ if (system_state != mnh_pwr->state) {
+ switch (system_state) {
+ case MNH_PWR_S0:
+ ret = mnh_pwr_up(system_state);
+ break;
+ case MNH_PWR_S1:
+ ret = mnh_pwr_partial();
+ break;
+ case MNH_PWR_S3:
+ ret = mnh_pwr_suspend();
+ break;
+ case MNH_PWR_S4:
+ ret = mnh_pwr_down();
+ break;
+ default:
+ dev_err(mnh_pwr->dev, "%s: invalid state %d\n",
+ __func__, system_state);
+ ret = -EINVAL;
+ break;
+ }
+
+ if (ret)
+ dev_err(mnh_pwr->dev,
+ "%s: state transition failed (%d)\n",
+ __func__, ret);
+ else
+ dev_dbg(mnh_pwr->dev, "%s done with state: %d\n",
+ __func__, system_state);
+ } else {
+ dev_dbg(mnh_pwr->dev, "%s: no state change needed\n", __func__);
+ }
+
+ mutex_unlock(&mnh_pwr->lock);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(mnh_pwr_set_state);
+
+enum mnh_pwr_state mnh_pwr_get_state(void)
+{
+ enum mnh_pwr_state curr_state;
+
+ mutex_lock(&mnh_pwr->lock);
+ curr_state = mnh_pwr->state;
+ mutex_unlock(&mnh_pwr->lock);
+
+ return curr_state;
+}
+EXPORT_SYMBOL_GPL(mnh_pwr_get_state);
+
+int mnh_pwr_init(struct platform_device *pdev, struct device *dev)
+{
+ int ret;
+
+ /* allocate memory for mnh_pwr_data struct */
+ mnh_pwr = devm_kzalloc(dev, sizeof(struct mnh_pwr_data), GFP_KERNEL);
+ if (!mnh_pwr)
+ return -ENOMEM;
+
+ /* save a local copy of the device struct */
+ mnh_pwr->pdev = pdev;
+ mnh_pwr->dev = dev;
+
+ /* initialize power state to powered down */
+ mnh_pwr->state = MNH_PWR_S4;
+
+ /* get platform resources */
+ ret = mnh_pwr_get_resources();
+ if (ret) {
+ dev_err(dev, "%s: failed to get platform resources (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ /* initialize some structures */
+ INIT_WORK(&mnh_pwr->shutdown_work, mnh_pwr_shutdown_work);
+ mutex_init(&mnh_pwr->lock);
+
+ /* power on the device to enumerate PCIe */
+ ret = mnh_pwr_up(MNH_PWR_S0);
+ if (ret) {
+ dev_err(dev, "%s: failed initial power up (%d)", __func__, ret);
+ return ret;
+ }
+
+ /* power down the device */
+ ret = mnh_pwr_down();
+ if (ret) {
+ dev_err(dev, "%s: failed initial power down (%d)", __func__,
+ ret);
+ return ret;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_pwr_init);
diff --git a/drivers/misc/mnh/mnh-pwr.h b/drivers/misc/mnh/mnh-pwr.h
new file mode 100644
index 0000000..049ffd0
--- /dev/null
+++ b/drivers/misc/mnh/mnh-pwr.h
@@ -0,0 +1,37 @@
+/*
+ *
+ * MNH PWR APIs
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef __MNH_PWR_H__
+#define __MNH_PWR_H__
+
+#include <linux/types.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/gpio.h>
+#include <linux/platform_device.h>
+
+enum mnh_pwr_state {
+ MNH_PWR_S0 = 0, /* active mode */
+ MNH_PWR_S1 = 1, /* partial active mode */
+ MNH_PWR_S3 = 3, /* suspend mode */
+ MNH_PWR_S4 = 4, /* power off */
+};
+
+int mnh_pwr_set_state(enum mnh_pwr_state system_state);
+enum mnh_pwr_state mnh_pwr_get_state(void);
+int mnh_pwr_init(struct platform_device *pdev, struct device *dev);
+
+#endif /* __MNH_PWR_H__ */
diff --git a/drivers/misc/mnh/mnh-sm-ion.c b/drivers/misc/mnh/mnh-sm-ion.c
new file mode 100644
index 0000000..561d6dd
--- /dev/null
+++ b/drivers/misc/mnh/mnh-sm-ion.c
@@ -0,0 +1,349 @@
+/*
+ * MNH Firmware on ION buffer
+ *
+ * Copyright 2017 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ */
+
+#include <linux/errno.h>
+#include <linux/firmware.h>
+#include <linux/msm_ion.h>
+#include "hw-mnh-regs.h"
+#include "mnh-sm.h"
+
+#define MSM_ION_EASEL_MEM_HEAP_NAME "easel_mem"
+
+/**
+ * mnh_ion_fw_copy - Copy data to ION buffer and flush cache
+ * Return 0 on success
+ * -ENOMEM if exceeding totoal buffer size
+ * mnh_ion_fw_copy must be called in slot incrementing order:
+ * 1. MNH_FW_SLOT_SBL, 2. MNH_FW_SLOT_KERNEL,
+ * 3. MNH_FW_SLOT_DTB, 4. MNH_FW_SLOT_RAMDISK
+ */
+static int mnh_ion_fw_copy(struct mnh_ion *ion, int slot,
+ uint64_t ep_addr, size_t size,
+ const u8 *data)
+{
+ dma_addr_t ion_addr; /* Base bus address of ION buffer */
+ unsigned long offset; /* offset inside ION buffer */
+ void *buf;
+
+ ion_addr = ion->sg->sgl->dma_address;
+ offset = (slot == 0) ? 0 : ion->fw_array[slot-1].ap_offs +
+ ion->fw_array[slot-1].size;
+
+ buf = ion->vaddr + offset;
+
+ if (offset + size >= ion->total_size) {
+ dev_err(ion->device,
+ "%s: slot=%d offset=0x%lx + size=0x%zx >= total=0x%zx\n",
+ __func__, slot, offset, size, ion->total_size);
+ return -ENOMEM;
+ }
+
+ ion->fw_array[slot].ep_addr = ep_addr;
+ ion->fw_array[slot].size = size;
+ ion->fw_array[slot].ap_addr = ion_addr + offset;
+ ion->fw_array[slot].ap_offs = offset;
+
+ memcpy(buf, data, size);
+
+ msm_ion_do_cache_op(ion->client, ion->handle, buf, size,
+ ION_IOC_CLEAN_CACHES);
+
+ return 0;
+}
+
+/**
+ * mnh_ion_fw_copy_by_name - Request firmware from disk and copy to ION
+ * Returns 0 on success
+ * negative on error
+ */
+static int mnh_ion_fw_copy_by_name(struct mnh_ion *ion, int slot,
+ uint64_t ep_addr, const char *name)
+{
+ int err = 0;
+ const struct firmware *fw_img;
+
+ dev_dbg(ion->device, "%s: request %s for slot %d\n",
+ __func__, name, slot);
+
+ err = request_firmware(&fw_img, name, ion->device);
+ if (err) {
+ dev_err(ion->device, "%s: request %s failed (%d)\n",
+ __func__, name, err);
+ return err;
+ }
+ err = mnh_ion_fw_copy(ion, slot, ep_addr, fw_img->size, fw_img->data);
+ if (err) {
+ dev_err(ion->device, "%s: copy %s to ION failed (%d)\n",
+ __func__, name, err);
+ /* Do not return yet */
+ }
+ release_firmware(fw_img);
+
+ return err;
+}
+
+/**
+ * mnh_ion_fw_copy_try_ion_sec - Try update from secondary ION buffer
+ * Returns 0 on success
+ * negative no update at this slot, or failed to copy
+ */
+static int mnh_ion_fw_copy_try_ion_sec(struct mnh_ion *ion,
+ struct mnh_ion *ion_sec, int slot,
+ uint64_t ep_addr)
+{
+ int err = 0;
+
+ if (!ion_sec || !ion)
+ return -EINVAL;
+
+ if (ion_sec->fw_array[slot].size <= 0)
+ return -EINVAL;
+
+ dev_dbg(ion->device, "%s: Using update buffer for slot %d\n",
+ __func__, slot);
+
+ err = mnh_ion_fw_copy(ion, slot, ep_addr,
+ ion_sec->fw_array[slot].size,
+ ion_sec->vaddr +
+ ion_sec->fw_array[slot].ap_offs);
+
+ return err;
+}
+
+/**
+ * mnh_ion_fw_update_request - For each slot, try to update from secondary
+ * ION buffer first. If failed, try update that slot from disk instead.
+ * Returns 0 on success
+ * negative on error
+ */
+static int mnh_ion_fw_update_request(struct mnh_ion *ion,
+ struct mnh_ion *ion_sec)
+{
+ int err;
+
+ err = mnh_ion_fw_copy_try_ion_sec(ion, ion_sec, MNH_FW_SLOT_SBL,
+ HW_MNH_SBL_DOWNLOAD);
+ if (err) {
+ err = mnh_ion_fw_copy_by_name(ion, MNH_FW_SLOT_SBL,
+ HW_MNH_SBL_DOWNLOAD,
+ "easel/fip.bin");
+ if (err)
+ return err;
+ }
+
+ err = mnh_ion_fw_copy_try_ion_sec(ion, ion_sec, MNH_FW_SLOT_KERNEL,
+ HW_MNH_KERNEL_DOWNLOAD);
+ if (err) {
+ err = mnh_ion_fw_copy_by_name(ion, MNH_FW_SLOT_KERNEL,
+ HW_MNH_KERNEL_DOWNLOAD,
+ "easel/Image");
+ if (err)
+ return err;
+ }
+
+ err = mnh_ion_fw_copy_try_ion_sec(ion, ion_sec, MNH_FW_SLOT_DTB,
+ HW_MNH_DT_DOWNLOAD);
+ if (err) {
+ err = mnh_ion_fw_copy_by_name(ion, MNH_FW_SLOT_DTB,
+ HW_MNH_DT_DOWNLOAD,
+ "easel/mnh.dtb");
+ if (err)
+ return err;
+ }
+
+ err = mnh_ion_fw_copy_try_ion_sec(ion, ion_sec, MNH_FW_SLOT_RAMDISK,
+ HW_MNH_RAMDISK_DOWNLOAD);
+ if (err) {
+ err = mnh_ion_fw_copy_by_name(ion, MNH_FW_SLOT_RAMDISK,
+ HW_MNH_RAMDISK_DOWNLOAD,
+ "easel/ramdisk.img");
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
+/**
+ * mnh_ion_fw_request - For each slot, update firmware from disk
+ * Returns 0 on success
+ * negative on error
+ */
+static int mnh_ion_fw_request(struct mnh_ion *ion)
+{
+ int err;
+
+ err = mnh_ion_fw_copy_by_name(ion, MNH_FW_SLOT_SBL,
+ HW_MNH_SBL_DOWNLOAD,
+ "easel/fip.bin");
+ if (err)
+ return err;
+
+ err = mnh_ion_fw_copy_by_name(ion, MNH_FW_SLOT_KERNEL,
+ HW_MNH_KERNEL_DOWNLOAD,
+ "easel/Image");
+ if (err)
+ return err;
+
+ err = mnh_ion_fw_copy_by_name(ion, MNH_FW_SLOT_DTB,
+ HW_MNH_DT_DOWNLOAD,
+ "easel/mnh.dtb");
+ if (err)
+ return err;
+
+ err = mnh_ion_fw_copy_by_name(ion, MNH_FW_SLOT_RAMDISK,
+ HW_MNH_RAMDISK_DOWNLOAD,
+ "easel/ramdisk.img");
+ if (err)
+ return err;
+
+ return 0;
+}
+
+int mnh_ion_stage_firmware(struct mnh_ion *ion)
+{
+ int ret;
+
+ if (!ion || !ion->client)
+ return -ENODEV;
+
+ ion->is_fw_ready = false;
+ ret = mnh_ion_fw_request(ion);
+ if (ret)
+ return ret;
+
+ ion->is_fw_ready = true;
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_ion_stage_firmware);
+
+int mnh_ion_stage_firmware_update(struct mnh_ion *ion, struct mnh_ion *ion_sec)
+{
+ int err;
+ int i;
+
+ dev_dbg(ion->device, "%s: staging update\n", __func__);
+ if (!ion || !ion_sec || !ion->client ||
+ !ion_sec->client || !ion_sec->is_fw_ready)
+ return -ENODEV;
+
+ ion->is_fw_ready = false;
+ err = mnh_ion_fw_update_request(ion, ion_sec);
+ if (err)
+ return err;
+
+ for (i = 0; i < MAX_NR_MNH_FW_SLOTS; i++) {
+ ion->fw_array[i].cert_size =
+ ion_sec->fw_array[i].cert_size;
+ }
+ ion->is_fw_ready = true;
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_ion_stage_firmware_update);
+
+long mnh_ion_create_buffer(struct mnh_ion *ion, size_t size,
+ enum ion_heap_ids heap_id)
+{
+ long err = 0;
+ struct device *dev = ion->device;
+
+ if (!ion)
+ return -EINVAL;
+
+ dev_dbg(dev, "%s: Claiming %zuMB ION buffer...\n",
+ __func__, size >> 20);
+
+ /* Create ION client */
+ ion->client = msm_ion_client_create(MSM_ION_EASEL_MEM_HEAP_NAME);
+ if (IS_ERR(ion->client)) {
+ err = PTR_ERR(ion->client);
+ dev_err(dev, "%s: Ion cannot create client \"%s\" (%ld)\n",
+ __func__, MSM_ION_EASEL_MEM_HEAP_NAME, err);
+ return err;
+ }
+
+ /* Allocate an ION buffer */
+ ion->handle = ion_alloc(ion->client, size, PAGE_SIZE /* align */,
+ ION_HEAP(heap_id),
+ ION_FLAG_CACHED);
+ if (IS_ERR(ion->handle)) {
+ err = PTR_ERR(ion->handle);
+ dev_err(dev, "%s: Ion cannot allocate buffer (%ld)\n",
+ __func__, err);
+ goto fail_ion_alloc;
+ }
+
+ /* Map ION buffer to kernel space */
+ ion->vaddr = ion_map_kernel(ion->client, ion->handle);
+ if (IS_ERR(ion->vaddr)) {
+ err = PTR_ERR(ion->vaddr);
+ dev_err(dev, "%s: Cannot map ion buffer to kernel (%ld)\n",
+ __func__, err);
+ goto fail_ion_map_kernel;
+ }
+
+ /* Map ION buffer to IOMMU and get sg_table */
+ ion->sg = ion_sg_table(ion->client, ion->handle);
+ if (IS_ERR(ion->sg)) {
+ err = PTR_ERR(ion->sg);
+ dev_err(dev, "%s: Cannot get ion sg_table (%ld)\n",
+ __func__, err);
+ goto fail_ion_sg_table;
+ }
+
+ dev_dbg(dev, "sg->nents=%d\n", ion->sg->nents);
+ dev_dbg(dev, "sg->orig_nents=%d\n", ion->sg->orig_nents);
+ dev_dbg(dev, "sgl->dma_address=%pa\n", &(ion->sg->sgl->dma_address));
+ dev_dbg(dev, "sgl->length=0x%x\n", ion->sg->sgl->length);
+
+ ion->total_size = size;
+
+ return 0;
+
+fail_ion_sg_table:
+ ion_unmap_kernel(ion->client, ion->handle);
+ ion->vaddr = NULL;
+fail_ion_map_kernel:
+ ion_free(ion->client, ion->handle);
+ ion->handle = NULL;
+fail_ion_alloc:
+ ion_client_destroy(ion->client);
+ ion->client = NULL;
+ return err;
+}
+EXPORT_SYMBOL(mnh_ion_create_buffer);
+
+void mnh_ion_destroy_buffer(struct mnh_ion *ion)
+{
+ if (!ion || !(ion->client))
+ return;
+
+ dev_dbg(ion->device, "%s: Freeing %zuMB ION buffer...\n",
+ __func__, ion->total_size >> 20);
+
+ if (!IS_ERR_OR_NULL(ion->handle)) {
+ ion_unmap_kernel(ion->client, ion->handle);
+ ion->vaddr = NULL;
+ ion_free(ion->client, ion->handle);
+ ion->handle = NULL;
+ }
+ ion_client_destroy(ion->client);
+ ion->client = NULL;
+
+ ion->is_fw_ready = false;
+}
+EXPORT_SYMBOL(mnh_ion_destroy_buffer);
diff --git a/drivers/misc/mnh/mnh-sm.c b/drivers/misc/mnh/mnh-sm.c
new file mode 100644
index 0000000..7ed048d
--- /dev/null
+++ b/drivers/misc/mnh/mnh-sm.c
@@ -0,0 +1,2836 @@
+/*
+ *
+ * MNH State Manager HOST Driver
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#include <linux/atomic.h>
+#include <linux/debugfs.h>
+#include <linux/device.h>
+#include <linux/delay.h>
+#include <linux/firmware.h>
+#include <linux/fs.h>
+#include <linux/gpio/consumer.h>
+#include <linux/gpio/machine.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/ion.h>
+#include <linux/kernel.h>
+#include <linux/kobject.h>
+#include <linux/ktime.h>
+#include <linux/kthread.h>
+#include <linux/miscdevice.h>
+#include <linux/module.h>
+#include <linux/msm_ion.h>
+#include <linux/mutex.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/stat.h>
+#include <linux/string.h>
+#include <linux/sysfs.h>
+#include <linux/types.h>
+#include <linux/uaccess.h>
+
+#include "hw-mnh-regs.h"
+#include "mnh-clk.h"
+#include "mnh-crypto.h"
+#include "mnh-ddr.h"
+#include "mnh-hwio.h"
+#include "mnh-hwio-bases.h"
+#include "mnh-hwio-pcie-ss.h"
+#include "mnh-hwio-scu.h"
+#include "mnh-mipi.h"
+#include "mnh-pcie.h"
+#include "mnh-pwr.h"
+#include "mnh-sm.h"
+
+#define MNH_SCU_INf(reg, fld) \
+HW_INf(HWIO_SCU_BASE_ADDR, SCU, reg, fld)
+#define MNH_SCU_INx(reg, inst) \
+HW_INx(HWIO_SCU_BASE_ADDR, SCU, reg, inst)
+#define MNH_SCU_OUTf(reg, fld, val) \
+HW_OUTf(HWIO_SCU_BASE_ADDR, SCU, reg, fld, val)
+#define MNH_SCU_OUT(reg, val) \
+HW_OUT(HWIO_SCU_BASE_ADDR, SCU, reg, val)
+#define MNH_SCU_OUTx(reg, inst, val) \
+HW_OUTx(HWIO_SCU_BASE_ADDR, SCU, reg, inst, val)
+
+#define MNH_PCIE_IN(reg) \
+HW_IN(HWIO_PCIE_SS_BASE_ADDR, PCIE_SS, reg)
+#define MNH_PCIE_INf(reg, fld) \
+HW_INf(HWIO_PCIE_SS_BASE_ADDR, PCIE_SS, reg, fld)
+#define MNH_PCIE_OUTf(reg, fld, val) \
+HW_OUTf(HWIO_PCIE_SS_BASE_ADDR, PCIE_SS, reg, fld, val)
+#define MNH_PCIE_OUT(reg, val) \
+HW_OUT(HWIO_PCIE_SS_BASE_ADDR, PCIE_SS, reg, val)
+#define MNH_PCIE_OUTx(reg, inst, val) \
+HW_OUTx(HWIO_PCIE_SS_BASE_ADDR, PCIE_SS, reg, inst, val)
+
+#define MAX_STR_COPY 32
+#define SUCCESS 0
+#define DEVICE_NAME "mnh_sm"
+
+#define SGL_SIZE 64
+
+#define INIT_DONE 0x1
+#define INIT_RESUME 0x2
+
+/* Timeout for waiting for PCIe firmware download */
+#define PCIE_FIRMWARE_DOWNLOAD_TIMEOUT msecs_to_jiffies(500)
+
+/* Timeout for waiting for MNH to be powered */
+#define POWERED_COMPLETE_TIMEOUT msecs_to_jiffies(5000)
+
+/* Timeout for waiting for MNH to suspend after issuing request */
+#define SUSPEND_COMPLETE_TIMEOUT msecs_to_jiffies(5000)
+
+/* Timeout for waiting for MNH set_state to complete */
+#define STATE_CHANGE_COMPLETE_TIMEOUT msecs_to_jiffies(5000)
+
+/* Timeout for MNH_HOTPLUG_IN when uart is enabled (in ms) */
+#define HOTPLUG_IN_LOOSE_TIMEOUT_MS 15000
+
+/* PCIe */
+#define MNH_PCIE_CHAN_0 0
+
+/*
+ * ALLOW_PARTIAL_ACTIVE: set to 1 to allow partial active mode where
+ * PCIe link is up, but Easel is not fully booted.
+ */
+#define ALLOW_PARTIAL_ACTIVE 0
+
+/* Firmware update mask, defines userspace updatable slots */
+#define FW_SLOT_UPDATE_MASK ((1 << MNH_FW_SLOT_RAMDISK) |\
+ (1 << MNH_FW_SLOT_SBL) |\
+ (1 << MNH_FW_SLOT_KERNEL) |\
+ (1 << MNH_FW_SLOT_DTB))
+
+#define MIN(a, b) (((a) < (b)) ? (a) : (b))
+
+/*
+ * The maximum timeout value to stay awake.
+ *
+ * For most use cases, this will not be the only wakeup source,
+ * because when Easel not used, application should have suspended Easel.
+ *
+ * For some rare background tasks, such as firmware update, or killing
+ * Easel services in background, we want to stay awake for no more than
+ * the timeout value, because at that time we may be the only wakeup
+ * source.
+ *
+ * Driver automatically releases wakelock after timeout value, or
+ * releases wakelock whenever Easel is not active.
+ *
+ * All power states change goes through mnh_sm_set_state(), so
+ * it will be the central place to request to stay awake.
+ */
+#define MNH_SM_WAKEUP_SOURCE_TIMEOUT_PRODUCTION (10000)
+
+enum fw_image_state {
+ FW_IMAGE_NONE = 0,
+ FW_IMAGE_DOWNLOADING,
+ FW_IMAGE_DOWNLOAD_SUCCESS,
+ FW_IMAGE_DOWNLOAD_FAIL
+};
+
+enum mnh_ddr_status {
+ MNH_DDR_OFF = 0, /* powered off */
+ MNH_DDR_ACTIVE, /* powered on, active */
+ MNH_DDR_SELF_REFRESH, /* powered on, in self-refresh */
+};
+
+enum fw_image_partition_index {
+ FW_PART_PRI = 0, /* Carveout region, stores final FW for easel */
+ FW_PART_SEC, /* Temp allocation for buf sharing with PS app */
+ FW_PART_TER, /* Temp allocation to protect shared buffer */
+ FW_PART_MAX
+};
+
+enum fw_update_status {
+ FW_UPD_NONE = 0, /* no update available */
+ FW_UPD_INIT, /* update service requested firmware buffer */
+ FW_UPD_RECEIVED, /* service uploaded new FW, authentication pending */
+ FW_UPD_VERIFIED /* fw signature check successful */
+};
+
+struct mnh_sm_state_stat {
+ uint32_t counter; /* accumulative */
+ ktime_t duration; /* accumulative */
+ ktime_t last_entry;
+ ktime_t last_exit;
+};
+
+struct mnh_sm_device {
+ struct platform_device *pdev;
+ struct miscdevice *misc_dev;
+ struct device *dev;
+ atomic_t cdev_ctr;
+ struct device *chardev;
+ struct class *dev_class;
+ uint32_t *firmware_buf[2];
+ size_t firmware_buf_size[2];
+ enum fw_image_state image_loaded;
+ int state;
+
+ /* flag for when driver has completed initialization */
+ bool initialized;
+
+ /* mutex to synchronize state transitions */
+ struct mutex lock;
+
+ /* kernel thread for state transitions from ioctls */
+ struct kthread_worker worker;
+ struct kthread_work set_state_work;
+ struct task_struct *thread;
+ int next_state;
+
+ /* completion used to signal when mnh is powered */
+ struct completion powered_complete;
+ bool powered;
+
+ /* completion used for signaling kthread work is complete */
+ struct completion work_complete;
+ int work_ret;
+
+ /* pins used for boot and power state transitions */
+ struct gpio_desc *boot_mode_gpio;
+ struct gpio_desc *ready_gpio;
+
+ /* irq for ready_gpio */
+ unsigned int ready_irq;
+
+ /* completion used for synchronizing with secondary bootloader */
+ struct completion suspend_complete;
+
+ /* size of the SBL, PBL copies it from DDR to SRAM */
+ uint32_t sbl_size;
+
+ /* resume address of kernel, updated before suspending */
+ uint32_t resume_addr;
+
+ /* state of the ddr channel */
+ enum mnh_ddr_status ddr_status;
+
+ /* mnh-ddr data */
+ struct mnh_ddr_data mnh_ddr_data;
+
+ /* pin used for ddr pad isolation */
+ struct gpio_desc *ddr_pad_iso_n_pin;
+
+ /* flag to know if firmware has already been downloaded */
+ bool firmware_downloaded;
+
+ /* flag indicating a valid update buffer */
+ enum fw_update_status update_status;
+
+ /* node to carveout memory, owned by mnh_sm_device */
+ struct mnh_ion *ion[FW_PART_MAX];
+
+ /* firmware version */
+ char mnh_fw_ver[FW_VER_SIZE];
+
+ /* serialize fw update requests */
+ struct mutex fw_update_lock;
+
+ /* debugfs */
+ struct dentry *dbgfs_dir;
+
+ /* power state stats */
+ struct mnh_sm_state_stat state_stats[MNH_STATE_MAX];
+};
+
+static struct mnh_sm_device *mnh_sm_dev;
+static int mnh_mipi_debug;
+static int mnh_freeze_state;
+
+/*
+ * bit [0] - bootloader console enabled
+ * bit [1] - kernel console enabled
+ * bit [2] - kernel stdout-to-console enabled
+ */
+enum {
+ MNH_BOOTARGS_BL_CONSOLE_ENABLE = 1 << 0,
+ MNH_BOOTARGS_KERNEL_CONSOLE_ENABLE = 1 << 1,
+ MNH_BOOTARGS_STDOUT_CONSOLE_ENABLE = 1 << 2,
+};
+#define MNH_UART_ENABLE (MNH_BOOTARGS_BL_CONSOLE_ENABLE | \
+ MNH_BOOTARGS_KERNEL_CONSOLE_ENABLE | \
+ MNH_BOOTARGS_STDOUT_CONSOLE_ENABLE)
+static unsigned int mnh_boot_args;
+
+/* 32-bit flag mask to SCU for boot power options */
+enum {
+ MNH_POWER_MODE_CLKPM_ENABLE = 1 << 0,
+ MNH_POWER_MODE_L1_2_ENABLE = 1 << 1,
+ MNH_POWER_MODE_AXI_CG_ENABLE = 1 << 2,
+};
+static uint32_t mnh_power_mode = MNH_POWER_MODE_CLKPM_ENABLE;
+
+/* flag for boot mode options */
+static enum mnh_boot_mode mnh_boot_mode = MNH_BOOT_MODE_PCIE;
+
+/* callback when easel enters and leaves the active state */
+static hotplug_cb_t mnh_hotplug_cb;
+
+static int mnh_sm_get_val_from_buf(const char *buf, unsigned long *val)
+{
+ uint8_t *token;
+ const char *delim = ";";
+
+ token = strsep((char **)&buf, delim);
+ if ((token) && (!(kstrtoul(token, 0, val))))
+ return 0;
+ else
+ return -EINVAL;
+}
+
+static int mnh_sm_get_token_val_from_buf(char **buf, unsigned long *val)
+{
+ uint8_t *token;
+ const char *delim = ";";
+
+ token = strsep(buf, delim);
+ if ((token) && (!(kstrtoul(token, 0, val))))
+ return 0;
+ else
+ return -EINVAL;
+}
+
+static ssize_t stage_fw_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ dev_dbg(mnh_sm_dev->dev, "Entering mnh_sm_stage_fw_show...\n");
+
+ return scnprintf(buf, MAX_STR_COPY, "%s\n",
+ mnh_sm_dev->ion[FW_PART_PRI]->is_fw_ready
+ ? "Firmware staged to ION"
+ : "Firmware not staged");
+
+ return 0;
+}
+
+static ssize_t stage_fw_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+
+ unsigned long val = 0;
+ int ret;
+
+ dev_dbg(mnh_sm_dev->dev, "Entering mnh_sm_stage_fw_store...\n");
+
+ ret = mnh_sm_get_val_from_buf(buf, &val);
+ if (!ret) {
+ if (val)
+ mnh_ion_stage_firmware(mnh_sm_dev->ion[FW_PART_PRI]);
+ return count;
+ }
+
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_RW(stage_fw);
+
+static ssize_t poweron_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ ssize_t strlen = 0;
+
+ dev_dbg(dev, "Entering mnh_sm_poweron_show...\n");
+ mnh_sm_set_state(MNH_STATE_ACTIVE);
+ return strlen;
+}
+
+static DEVICE_ATTR_RO(poweron);
+
+static ssize_t dump_scu_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ int i;
+
+ for (i = 0; i < HWIO_SCU_PIN_CFG_REGNUM; i++) {
+ dev_info(mnh_sm_dev->dev,
+ "PIN_CFG%d: 0x%08x\n", i,
+ MNH_SCU_INx(PIN_CFG, i));
+ }
+
+ return 0;
+}
+static DEVICE_ATTR_RO(dump_scu);
+
+static ssize_t poweroff_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ ssize_t strlen = 0;
+
+ dev_dbg(dev, "Entering mnh_sm_poweroff_show...\n");
+ mnh_sm_set_state(MNH_STATE_OFF);
+
+ return strlen;
+}
+
+static DEVICE_ATTR_RO(poweroff);
+
+static ssize_t state_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ dev_dbg(dev, "Entering mnh_sm_state_show...\n");
+
+ return scnprintf(buf, MAX_STR_COPY, "%d\n", mnh_sm_dev->state);
+}
+
+static ssize_t state_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ unsigned long val = 0;
+ int ret;
+
+ dev_dbg(dev, "Entering mnh_sm_state_store...\n");
+
+ ret = mnh_sm_get_val_from_buf(buf, &val);
+ mnh_sm_set_state((int)val);
+ return count;
+}
+
+static DEVICE_ATTR_RW(state);
+
+static int dma_callback(uint8_t chan, enum mnh_dma_chan_dir_t dir,
+ enum mnh_dma_trans_status_t status)
+{
+ dev_dbg(mnh_sm_dev->dev, "DMA_CALLBACK: ch:%d, dir:%s, status:%s\n",
+ chan, (dir == DMA_AP2EP)?"READ(AP2EP)":"WRITE(EP2AP)",
+ (status == DMA_DONE)?"DONE":"ABORT");
+
+ if (chan == MNH_PCIE_CHAN_0 && dir == DMA_AP2EP) {
+ if (mnh_sm_dev->image_loaded == FW_IMAGE_DOWNLOADING) {
+ if (status == DMA_DONE)
+ mnh_sm_dev->image_loaded =
+ FW_IMAGE_DOWNLOAD_SUCCESS;
+ else if (status == DMA_ABORT)
+ mnh_sm_dev->image_loaded =
+ FW_IMAGE_DOWNLOAD_FAIL;
+ }
+ } else {
+ dev_err(mnh_sm_dev->dev, "DMA_CALLBACK: incorrect channel and direction");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int mnh_firmware_waitdownloaded(void)
+{
+ unsigned long timeout = jiffies + PCIE_FIRMWARE_DOWNLOAD_TIMEOUT;
+
+ do {
+ if (mnh_sm_dev->image_loaded == FW_IMAGE_DOWNLOAD_SUCCESS) {
+ dev_dbg(mnh_sm_dev->dev, "Firmware loaded!\n");
+ return 0;
+ } else if (mnh_sm_dev->image_loaded == FW_IMAGE_DOWNLOAD_FAIL) {
+ dev_err(mnh_sm_dev->dev, "Firmware load fail!\n");
+ return -EIO;
+ }
+ usleep_range(2000, 5000);
+ } while (time_before(jiffies, timeout));
+
+ dev_err(mnh_sm_dev->dev, "Fail to Download Firmware, timeout!!\n");
+ return -EIO;
+}
+
+/**
+ * mnh_alloc_firmware_buf - Allocate temporary kernel buffer for mnh firmware
+ *
+ * Attempts allocation of a large, contiguous kernel buffer. If the initial
+ * attempt weren't successful, try a smaller size until success.
+ *
+ * @dev: Device to allocate memory for.
+ * @buf: Address of pointer to allocated memory.
+ *
+ * Returns size of allocated memory in bytes.
+ */
+static size_t mnh_alloc_firmware_buf(struct device *dev, uint32_t **buf)
+{
+ size_t size = IMG_DOWNLOAD_MAX_SIZE;
+
+ while (size > 0) {
+ *buf = devm_kmalloc(dev, size, GFP_KERNEL);
+ if (*buf)
+ break;
+
+ size >>= 1;
+ }
+
+ return size;
+}
+
+/**
+ * mnh_free_firmware_buf - Free buffer allocated by mnh_alloc_firmware_buf()
+ *
+ * @dev: Device to free memory for.
+ * @buf: Pointer to allocated memory.
+ */
+static inline void mnh_free_firmware_buf(struct device *dev, uint32_t *buf)
+{
+ devm_kfree(dev, buf);
+}
+
+static int mnh_transfer_firmware_contig(size_t fw_size, dma_addr_t src_addr,
+ uint64_t dst_addr)
+{
+ struct mnh_dma_element_t dma_blk;
+ int err = -EINVAL;
+
+ dma_blk.dst_addr = dst_addr;
+ dma_blk.len = fw_size;
+ dma_blk.src_addr = src_addr;
+
+ dev_dbg(mnh_sm_dev->dev,
+ "FW download - AP(:0x%llx) to EP(:0x%llx), size(%d)\n",
+ dma_blk.src_addr, dma_blk.dst_addr, dma_blk.len);
+
+ mnh_sm_dev->image_loaded = FW_IMAGE_DOWNLOADING;
+ mnh_dma_sblk_start(MNH_PCIE_CHAN_0, DMA_AP2EP, &dma_blk);
+
+ err = mnh_firmware_waitdownloaded();
+
+ return err;
+}
+
+static int mnh_transfer_firmware(size_t fw_size, const uint8_t *fw_data,
+ uint64_t dst_addr)
+{
+ uint32_t *buf;
+ size_t buf_size;
+ int buf_index = 0;
+ struct mnh_dma_element_t dma_blk;
+ int err = -EINVAL;
+ size_t sent = 0, size = 0, remaining;
+
+ remaining = fw_size;
+
+ while (remaining > 0) {
+ buf = mnh_sm_dev->firmware_buf[buf_index];
+ buf_size = mnh_sm_dev->firmware_buf_size[buf_index];
+
+ size = MIN(remaining, buf_size);
+
+ memcpy(buf, fw_data + sent, size);
+
+ if (mnh_sm_dev->image_loaded == FW_IMAGE_DOWNLOADING) {
+ err = mnh_firmware_waitdownloaded();
+ mnh_unmap_mem(dma_blk.src_addr, size, DMA_TO_DEVICE);
+ if (err)
+ break;
+ }
+
+ dma_blk.dst_addr = dst_addr + sent;
+ dma_blk.len = size;
+ dma_blk.src_addr = mnh_map_mem(buf, size, DMA_TO_DEVICE);
+
+ if (!dma_blk.src_addr) {
+ dev_err(mnh_sm_dev->dev,
+ "Could not map dma buffer for FW download\n");
+ return -ENOMEM;
+ }
+
+ dev_dbg(mnh_sm_dev->dev, "FW download - AP(:0x%llx) to EP(:0x%llx), size(%d)\n",
+ dma_blk.src_addr, dma_blk.dst_addr, dma_blk.len);
+
+ mnh_sm_dev->image_loaded = FW_IMAGE_DOWNLOADING;
+ mnh_dma_sblk_start(MNH_PCIE_CHAN_0, DMA_AP2EP, &dma_blk);
+
+ sent += size;
+ remaining -= size;
+ buf_index = (buf_index + 1) & 0x1;
+ dev_dbg(mnh_sm_dev->dev, "Sent:%zd, Remaining:%zd\n",
+ sent, remaining);
+ }
+
+ if (mnh_sm_dev->image_loaded == FW_IMAGE_DOWNLOADING) {
+ err = mnh_firmware_waitdownloaded();
+ mnh_unmap_mem(dma_blk.src_addr, size, DMA_TO_DEVICE);
+ }
+
+ return err;
+}
+
+/*
+ * Returns:
+ * 0 if all firmware slots verify correct
+ * >1 bitmask indicating firmware slots with wrong signature
+ * bit 0: SBL
+ * bit 1: Kernel
+ * bit 2: DTB
+ * bit 3: Ramdisk
+ * -ENODEV in case ION buffer is not available
+ */
+int mnh_verify_firmware_ion(struct mnh_ion *ion)
+{
+#if IS_ENABLED(CONFIG_MNH_SIG)
+ int err = 0, ret = 0;
+ int i;
+
+ struct cert_info info = {0};
+
+ if (!ion)
+ return -ENODEV;
+
+ for (i = 0; i < MAX_NR_MNH_FW_SLOTS; i++) {
+ dev_dbg(mnh_sm_dev->dev,
+ "Slot %d size %zu\n", i, ion->fw_array[i].size);
+ if (ion->fw_array[i].size > 0) {
+ dev_dbg(mnh_sm_dev->dev,
+ "Signature check of slot[%d]...\n", i);
+
+ info.img = ion->vaddr + ion->fw_array[i].ap_offs;
+ info.img_len = ion->fw_array[i].size;
+
+ err = mnh_crypto_verify(mnh_sm_dev->dev, &info);
+
+ ion->fw_array[i].cert = info.cert;
+ ion->fw_array[i].cert_size = info.cert_size;
+ if (!err) {
+ dev_dbg(mnh_sm_dev->dev,
+ "->Sig. check PASS for slot[%d]\n", i);
+ dev_dbg(mnh_sm_dev->dev,
+ "->Img size %zu, total img size %zu\n",
+ ion->fw_array[i].size -
+ ion->fw_array[i].cert_size,
+ ion->fw_array[i].size);
+ } else {
+ dev_err(mnh_sm_dev->dev,
+ "->Sig. check failed for slot[%d]=%d\n",
+ i,
+ ion->fw_array[i].cert);
+ ret |= (1 << i);
+ }
+ }
+ }
+ return ret;
+#else
+ return 0;
+#endif
+}
+
+int mnh_transfer_slot_ion(struct mnh_ion *ion, int slot)
+{
+ int err;
+ uint32_t sbl_size, sbl_offset;
+
+ dev_dbg(mnh_sm_dev->dev, "ION downloading fw[%d]...\n", slot);
+ if (slot == MNH_FW_SLOT_SBL) {
+ /* extract SBL from fip.bin image */
+ memcpy(&sbl_size, (uint8_t *)(ion->vaddr
+ + ion->fw_array[slot].ap_offs
+ + FIP_IMG_SBL_SIZE_OFFSET),
+ sizeof(sbl_size));
+ memcpy(&sbl_offset, (uint8_t *)(ion->vaddr
+ + ion->fw_array[slot].ap_offs
+ + FIP_IMG_SBL_ADDR_OFFSET),
+ sizeof(sbl_offset));
+ dev_dbg(mnh_sm_dev->dev,
+ "SBL offset %d, SBL size %d\n",
+ sbl_offset, sbl_size);
+ mnh_sm_dev->sbl_size = sbl_size;
+
+ err = mnh_transfer_firmware_contig(
+ sbl_size,
+ ion->fw_array[slot].ap_addr + sbl_offset,
+ ion->fw_array[slot].ep_addr);
+ } else {
+ err = mnh_transfer_firmware_contig(
+ ion->fw_array[slot].size -
+ ion->fw_array[slot].cert_size,
+ ion->fw_array[slot].ap_addr,
+ ion->fw_array[slot].ep_addr);
+ }
+ return err;
+}
+
+int mnh_download_firmware_ion(struct mnh_ion *ion[FW_PART_MAX])
+{
+ int err;
+ int i;
+
+ if (!ion)
+ return -ENODEV;
+
+ if (!ion[FW_PART_PRI]->is_fw_ready)
+ return -EAGAIN;
+
+ mnh_sm_dev->image_loaded = FW_IMAGE_NONE;
+
+ /* Register DMA callback */
+ err = mnh_reg_irq_callback(0, 0, dma_callback);
+ if (err) {
+ dev_err(mnh_sm_dev->dev,
+ "register irq callback failed - %d\n", err);
+ return err;
+ }
+
+ for (i = 0; i < MAX_NR_MNH_FW_SLOTS; i++) {
+#if IS_ENABLED(CONFIG_MNH_SIG_FORCE_OTA)
+ if (ion[FW_PART_PRI]->fw_array[i].cert != FW_IMAGE_CERT_OK) {
+ dev_err(mnh_sm_dev->dev,
+ "Firmware tainted, upload aborted - %d\n",
+ ion[FW_PART_PRI]->fw_array[i].cert);
+ mnh_reg_irq_callback(NULL, NULL, NULL);
+ return -EKEYREJECTED;
+ }
+#endif
+ /* only upload from primary ION buffer */
+ err = mnh_transfer_slot_ion(ion[FW_PART_PRI], i);
+ if (err) {
+ /* Unregister DMA callback */
+ dev_err(mnh_sm_dev->dev,
+ "failed FW transfer of slot %d: %d\n", i, err);
+ mnh_reg_irq_callback(NULL, NULL, NULL);
+ return err;
+ }
+ }
+
+ /* Configure sbl addresses and size */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_4, 4,
+ HW_MNH_SBL_DOWNLOAD);
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_5, 4,
+ HW_MNH_SBL_DOWNLOAD_EXE);
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_6, 4,
+ mnh_sm_dev->sbl_size);
+
+ /* Configure post sbl entry address */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_7, 4,
+ HW_MNH_KERNEL_DOWNLOAD);
+
+ /* sbl needs this for its own operation and arg0 for kernel */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_3, 4,
+ HW_MNH_DT_DOWNLOAD);
+ /* Unregister DMA callback */
+ mnh_reg_irq_callback(NULL, NULL, NULL);
+
+ mnh_sm_dev->firmware_downloaded = true;
+
+ return 0;
+}
+
+int mnh_download_firmware_legacy(void)
+{
+ const struct firmware *dt_img, *kernel_img, *ram_img;
+ const struct firmware *fip_img;
+ int err;
+ uint32_t size, addr;
+ int i;
+
+ mnh_sm_dev->image_loaded = FW_IMAGE_NONE;
+
+ /* Register DMA callback */
+ err = mnh_reg_irq_callback(0, 0, dma_callback);
+ if (err) {
+ dev_err(mnh_sm_dev->dev, "register irq callback failed - %d\n",
+ err);
+ return err;
+ }
+
+ err = request_firmware(&fip_img, "easel/fip.bin", mnh_sm_dev->dev);
+ if (err) {
+ dev_err(mnh_sm_dev->dev, "request fip_image failed - %d\n",
+ err);
+ return -EIO;
+ }
+
+ err = request_firmware(&kernel_img, "easel/Image", mnh_sm_dev->dev);
+ if (err) {
+ dev_err(mnh_sm_dev->dev, "request kernel failed - %d\n", err);
+ goto free_fip;
+ }
+
+ err = request_firmware(&dt_img, "easel/mnh.dtb", mnh_sm_dev->dev);
+ if (err) {
+ dev_err(mnh_sm_dev->dev, "request kernel failed - %d\n", err);
+ goto free_kernel;
+ }
+
+ err = request_firmware(&ram_img, "easel/ramdisk.img", mnh_sm_dev->dev);
+ if (err) {
+ dev_err(mnh_sm_dev->dev, "request kernel failed - %d\n", err);
+ goto free_dt;
+ }
+
+ /* get double buffers for transferring firmware */
+ for (i = 0; i < 2; i++) {
+ mnh_sm_dev->firmware_buf_size[i] =
+ mnh_alloc_firmware_buf(mnh_sm_dev->dev,
+ &mnh_sm_dev->firmware_buf[i]);
+ if (!mnh_sm_dev->firmware_buf_size[i]) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: could not allocate a buffer for firmware transfers\n",
+ __func__);
+ goto free_ramdisk;
+ }
+ dev_dbg(mnh_sm_dev->dev,
+ "%s: firmware_buf_size[%d]=%zu",
+ __func__, i, mnh_sm_dev->firmware_buf_size[i]);
+ }
+
+ /* DMA transfer for SBL */
+ memcpy(&size, (uint8_t *)(fip_img->data + FIP_IMG_SBL_SIZE_OFFSET),
+ sizeof(size));
+ memcpy(&addr, (uint8_t *)(fip_img->data + FIP_IMG_SBL_ADDR_OFFSET),
+ sizeof(addr));
+ mnh_sm_dev->sbl_size = size;
+
+ dev_dbg(mnh_sm_dev->dev, "sbl data addr :0x%x", addr);
+ dev_dbg(mnh_sm_dev->dev, "DOWNLOADING SBL...size:0x%x\n", size);
+ if (mnh_transfer_firmware(size, fip_img->data + addr,
+ HW_MNH_SBL_DOWNLOAD))
+ goto fail_downloading;
+
+ /* DMA transfer for device tree */
+ dev_dbg(mnh_sm_dev->dev, "DOWNLOADING DT...size:%zd\n", dt_img->size);
+ if (mnh_transfer_firmware(dt_img->size, dt_img->data,
+ HW_MNH_DT_DOWNLOAD))
+ goto fail_downloading;
+
+ /* DMA transfer for ramdisk */
+ dev_dbg(mnh_sm_dev->dev, "DOWNLOADING RAMDISK...size:%zd\n",
+ ram_img->size);
+ if (mnh_transfer_firmware(ram_img->size, ram_img->data,
+ HW_MNH_RAMDISK_DOWNLOAD))
+ goto fail_downloading;
+
+ /* DMA transfer for Kernel image */
+ dev_dbg(mnh_sm_dev->dev, "DOWNLOADING KERNEL...size:%zd\n",
+ kernel_img->size);
+ if (mnh_transfer_firmware(kernel_img->size, kernel_img->data,
+ HW_MNH_KERNEL_DOWNLOAD))
+ goto fail_downloading;
+
+ /* Configure sbl addresses and size */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_4, 4,
+ HW_MNH_SBL_DOWNLOAD);
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_5, 4,
+ HW_MNH_SBL_DOWNLOAD_EXE);
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_6, 4,
+ size);
+
+ /* Configure post sbl entry address */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_7, 4,
+ HW_MNH_KERNEL_DOWNLOAD);
+
+ /* sbl needs this for its own operation and arg0 for kernel */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_3, 4,
+ HW_MNH_DT_DOWNLOAD);
+
+ for (i = 0; i < 2; i++) {
+ mnh_free_firmware_buf(mnh_sm_dev->dev,
+ mnh_sm_dev->firmware_buf[i]);
+ mnh_sm_dev->firmware_buf_size[i] = 0;
+ mnh_sm_dev->firmware_buf[i] = NULL;
+ }
+
+ release_firmware(fip_img);
+ release_firmware(kernel_img);
+ release_firmware(dt_img);
+ release_firmware(ram_img);
+
+ /* Unregister DMA callback */
+ mnh_reg_irq_callback(NULL, NULL, NULL);
+
+ mnh_sm_dev->firmware_downloaded = true;
+
+ return 0;
+
+fail_downloading:
+ dev_err(mnh_sm_dev->dev, "FW downloading fails\n");
+ mnh_sm_dev->image_loaded = FW_IMAGE_DOWNLOAD_FAIL;
+free_ramdisk:
+ for (i = 0; i < 2; i++) {
+ mnh_free_firmware_buf(mnh_sm_dev->dev,
+ mnh_sm_dev->firmware_buf[i]);
+ mnh_sm_dev->firmware_buf_size[i] = 0;
+ mnh_sm_dev->firmware_buf[i] = NULL;
+ }
+ release_firmware(ram_img);
+free_dt:
+ release_firmware(dt_img);
+free_kernel:
+ release_firmware(kernel_img);
+free_fip:
+ release_firmware(fip_img);
+
+ /* Unregister DMA callback */
+ mnh_reg_irq_callback(NULL, NULL, NULL);
+ return -EIO;
+}
+
+int mnh_download_firmware(void)
+{
+ int err;
+
+ /* Prefer to download from ION buffer if it's ready */
+ err = mnh_download_firmware_ion(mnh_sm_dev->ion);
+ if (!err) {
+ dev_dbg(mnh_sm_dev->dev,
+ "%s: ION download successful\n", __func__);
+ return 0;
+ }
+#if !IS_ENABLED(CONFIG_MNH_SIG_FORCE_OTA)
+ /* Otherwise fall back to legacy mode */
+ dev_err(mnh_sm_dev->dev, "%s: Fallback to legacy mode\n", __func__);
+ return mnh_download_firmware_legacy();
+#else
+ return -EKEYREJECTED;
+#endif
+}
+
+/**
+ * Verify the integrity of the images in the secondary ion buffer
+ * and populate the fw_array structure. Use tertiary buffer only
+ * if signature verification is enabled.
+ * @return 0 if success or -1 on failure to verify the image integrity
+ */
+int mnh_validate_update_buf(struct mnh_update_configs configs)
+{
+ int i, slot;
+ int err_mask = 0;
+ struct mnh_update_config config;
+ enum fw_image_partition_index fw_idx;
+
+#if IS_ENABLED(CONFIG_MNH_SIG)
+ fw_idx = FW_PART_TER;
+#else
+ fw_idx = FW_PART_SEC;
+#endif
+
+ if (!mnh_sm_dev->ion[FW_PART_SEC])
+ return -ENOMEM;
+
+#if IS_ENABLED(CONFIG_MNH_SIG)
+ /* buffer to prevent future data corruption from userspace */
+ if (mnh_sm_dev->ion[FW_PART_TER])
+ mnh_ion_destroy_buffer(mnh_sm_dev->ion[FW_PART_TER]);
+ if (mnh_ion_create_buffer(mnh_sm_dev->ion[FW_PART_TER],
+ MNH_ION_BUFFER_SIZE,
+ ION_SYSTEM_HEAP_ID)) {
+ dev_err(mnh_sm_dev->dev, "%s: cannot claim ION buffer\n",
+ __func__);
+ return -ENOMEM;
+ }
+ memcpy(mnh_sm_dev->ion[FW_PART_TER]->vaddr,
+ mnh_sm_dev->ion[FW_PART_SEC]->vaddr,
+ MNH_ION_BUFFER_SIZE);
+ mnh_ion_destroy_buffer(mnh_sm_dev->ion[FW_PART_SEC]);
+#endif
+ /* Initialize update buffer struct */
+ for (i = 0; i < MAX_NR_MNH_FW_SLOTS; i++) {
+ mnh_sm_dev->ion[fw_idx]->fw_array[i].ap_offs = 0;
+ mnh_sm_dev->ion[fw_idx]->fw_array[i].size = 0;
+ }
+
+ /* Parse update slots */
+ for (i = 0; i < MAX_NR_MNH_FW_SLOTS; i++) {
+ config = configs.config[i];
+ slot = config.slot_type;
+ if ((slot < 0) || (slot >= MAX_NR_MNH_FW_SLOTS))
+ return -EINVAL;
+
+ if (!((config.size > 0) &&
+ (config.size < MNH_ION_BUFFER_SIZE) &&
+ (config.offset >= 0) &&
+ (config.offset < (MNH_ION_BUFFER_SIZE - config.size))))
+ return -EINVAL;
+
+ /* only configure slots masked by FW_SLOT_UPDATE_MASK */
+ if (FW_SLOT_UPDATE_MASK & (1 << slot)) {
+ mnh_sm_dev->ion[fw_idx]->fw_array[slot].ap_offs =
+ config.offset;
+ mnh_sm_dev->ion[fw_idx]->fw_array[slot].size =
+ config.size;
+ }
+ }
+
+ /* Signature verification */
+ err_mask = mnh_verify_firmware_ion(mnh_sm_dev->ion[fw_idx]);
+
+#if IS_ENABLED(CONFIG_MNH_SIG_FORCE_PS)
+ /* all update slots must verify for the update to be accepted */
+ if (err_mask) {
+ dev_err(mnh_sm_dev->dev, "%s: Update firmware tainted: %d\n",
+ __func__, err_mask);
+ mnh_sm_dev->ion[fw_idx]->is_fw_ready = false;
+ return err_mask;
+ }
+#endif
+ dev_info(mnh_sm_dev->dev, "%s: Update firmware validated\n",
+ __func__);
+
+ mnh_sm_dev->ion[fw_idx]->is_fw_ready = true;
+
+ return 0;
+}
+
+static ssize_t download_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ ssize_t strlen = 0;
+
+ dev_dbg(mnh_sm_dev->dev, "MNH PM mnh_pm_download_show...\n");
+
+ mnh_sm_set_state(MNH_STATE_ACTIVE);
+
+ return strlen;
+}
+
+static DEVICE_ATTR_RO(download);
+
+static ssize_t suspend_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ ssize_t strlen = 0;
+
+ mnh_sm_set_state(MNH_STATE_SUSPEND);
+
+ return strlen;
+}
+
+static DEVICE_ATTR_RO(suspend);
+
+int mnh_update_fw_version(struct mnh_ion *ion)
+{
+ if ((ion != NULL) && (ion->fw_array[MNH_FW_SLOT_RAMDISK].size <
+ sizeof(mnh_sm_dev->mnh_fw_ver)))
+ return -EINVAL;
+ memcpy(mnh_sm_dev->mnh_fw_ver,
+ (uint8_t *)(ion->vaddr +
+ ion->fw_array[MNH_FW_SLOT_RAMDISK].ap_offs +
+ ion->fw_array[MNH_FW_SLOT_RAMDISK].size -
+ ion->fw_array[MNH_FW_SLOT_RAMDISK].cert_size -
+ sizeof(mnh_sm_dev->mnh_fw_ver)),
+ sizeof(mnh_sm_dev->mnh_fw_ver));
+ dev_dbg(mnh_sm_dev->dev,
+ "%s: Easel firmware version: %s size %zu\n", __func__,
+ mnh_sm_dev->mnh_fw_ver, sizeof(mnh_sm_dev->mnh_fw_ver));
+ return sizeof(mnh_sm_dev->mnh_fw_ver);
+}
+
+int mnh_resume_firmware(void)
+{
+ dev_dbg(mnh_sm_dev->dev, "%s sbl dl:0x%x ex:0x%x size:0x%x\n", __func__,
+ HW_MNH_SBL_DOWNLOAD, HW_MNH_SBL_DOWNLOAD_EXE,
+ mnh_sm_dev->sbl_size);
+ dev_dbg(mnh_sm_dev->dev, "%s resume:0x%x dt:0x%x\n", __func__,
+ mnh_sm_dev->resume_addr, HW_MNH_DT_DOWNLOAD);
+
+ /* Configure sbl addresses and size */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_4, 4,
+ HW_MNH_SBL_DOWNLOAD);
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_5, 4,
+ HW_MNH_SBL_DOWNLOAD_EXE);
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_6, 4,
+ mnh_sm_dev->sbl_size);
+
+ /* Configure resume entry address */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_7, 4,
+ mnh_sm_dev->resume_addr);
+
+ /* sbl needs this for its own operation and arg0 for kernel */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_3, 4,
+ HW_MNH_DT_DOWNLOAD);
+
+ /* Configure boot command to resume */
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET + HW_MNH_PCIE_GP_0,
+ sizeof(uint32_t), INIT_RESUME);
+
+ return 0;
+}
+
+static ssize_t resume_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ ssize_t strlen = 0;
+
+ mnh_sm_set_state(MNH_STATE_ACTIVE);
+
+ return strlen;
+}
+
+static DEVICE_ATTR_RO(resume);
+
+static ssize_t reset_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ ssize_t strlen = 0;
+ return strlen;
+}
+
+static DEVICE_ATTR_RO(reset);
+
+static ssize_t cpu_clk_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ unsigned long val = 0;
+ int ret;
+
+ dev_dbg(mnh_sm_dev->dev, "Entering mnh_sm_cpu_clk_store...\n");
+
+ if (count > 10)
+ return -EINVAL;
+
+ ret = mnh_sm_get_val_from_buf(buf, &val);
+ if (!ret && (val >= CPU_FREQ_MIN) && (val <= CPU_FREQ_MAX)) {
+ mnh_cpu_freq_change(val);
+ return count;
+ }
+
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_WO(cpu_clk);
+
+static ssize_t debug_mipi_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ dev_dbg(mnh_sm_dev->dev, "Entering mnh_sm_debug_mipi_show...\n");
+
+ return scnprintf(buf, MAX_STR_COPY, "%d\n", mnh_mipi_debug);
+}
+
+static ssize_t debug_mipi_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ unsigned long val = 0;
+ int ret;
+
+ dev_dbg(mnh_sm_dev->dev, "Entering mnh_sm_debug_mipi_store...\n");
+
+ ret = mnh_sm_get_val_from_buf(buf, &val);
+ if (!ret) {
+ mnh_mipi_debug = val;
+ mnh_mipi_set_debug(val);
+ return count;
+ }
+
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_RW(debug_mipi);
+
+static ssize_t freeze_state_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ dev_dbg(mnh_sm_dev->dev, "Entering mnh_sm_freeze_state_show...\n");
+
+ return scnprintf(buf, MAX_STR_COPY, "%d\n", mnh_freeze_state);
+}
+
+static ssize_t freeze_state_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ unsigned long val = 0;
+ int ret;
+
+ dev_dbg(mnh_sm_dev->dev, "Entering mnh_sm_freeze_state_store...\n");
+
+ ret = mnh_sm_get_val_from_buf(buf, &val);
+ if (!ret) {
+ mnh_freeze_state = val;
+ return count;
+ }
+
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_RW(freeze_state);
+
+/* temporary sysfs hook to test mipi shutdown of the available channels */
+static ssize_t mipi_stop_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ struct mnh_mipi_config cfg;
+ unsigned long val = 0;
+ int ret;
+
+ dev_dbg(mnh_sm_dev->dev, "%s: Entering mnh_sm_mipi_store...\n",
+ __func__);
+
+ ret = mnh_sm_get_val_from_buf(buf, &val);
+ if (!ret) {
+ if (val < 3) {
+ cfg.rxdev = cfg.txdev = val;
+ dev_dbg(mnh_sm_dev->dev,
+ "%s: Shutting down mipi tx dev %d, rx dev %d\n",
+ __func__, cfg.txdev, cfg.rxdev);
+ mnh_mipi_stop(mnh_sm_dev->dev, cfg);
+ } else {
+ dev_dbg(mnh_sm_dev->dev, "%s: Invalid MIPI channel\n",
+ __func__);
+ }
+ return count;
+ }
+ dev_err(mnh_sm_dev->dev, "%s: Usage: echo\"<channel>\">mipi_stop\n",
+ __func__);
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_WO(mipi_stop);
+
+static ssize_t boot_args_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ dev_dbg(mnh_sm_dev->dev, "Entering mnh_sm_boot_args_show...\n");
+
+ return scnprintf(buf, MAX_STR_COPY, "%d\n", mnh_boot_args);
+}
+
+static ssize_t boot_args_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ unsigned long val = 0;
+ int ret;
+
+ dev_dbg(mnh_sm_dev->dev, "Entering mnh_sm_boot_args_store...\n");
+
+ ret = mnh_sm_get_val_from_buf(buf, &val);
+ if (!ret) {
+ mnh_boot_args = val;
+ return count;
+ }
+
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_RW(boot_args);
+
+static ssize_t boot_trace_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ int err;
+ uint32_t val;
+
+ err = mnh_config_read(MNH_BOOT_TRACE, sizeof(val), &val);
+ if (!err)
+ return scnprintf(buf, MAX_STR_COPY, "%x\n", val);
+ return scnprintf(buf, MAX_STR_COPY, "Unavailable\n");
+}
+
+static DEVICE_ATTR_RO(boot_trace);
+
+static ssize_t config_read_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ unsigned long addr = 0;
+ uint32_t data;
+ int ret;
+
+ dev_dbg(mnh_sm_dev->dev, "Entering config_read_store...\n");
+
+ ret = mnh_sm_get_val_from_buf(buf, &addr);
+ if (ret)
+ return ret;
+
+ /* reads 4 bytes config register at a time */
+ ret = mnh_config_read(addr, 4, &data);
+ if (ret) {
+ dev_err(mnh_sm_dev->dev, "Cannot read offset 0x%lx\n", addr);
+ } else {
+ dev_info(mnh_sm_dev->dev, "Config [0x%lx] = 0x%08x\n",
+ addr, data);
+ }
+
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_WO(config_read);
+
+static ssize_t mem_read_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ unsigned long addr = 0;
+ uint32_t data;
+ int ret;
+
+ dev_dbg(mnh_sm_dev->dev, "Entering mem_read_store...\n");
+
+ ret = mnh_sm_get_val_from_buf(buf, &addr);
+ if (ret)
+ return ret;
+
+ /* reads 1 byte memory at a time */
+ ret = mnh_ddr_read(addr, 1, &data);
+ if (ret) {
+ dev_err(mnh_sm_dev->dev, "Cannot read offset 0x%lx\n", addr);
+ } else {
+ dev_info(mnh_sm_dev->dev, "Offset [0x%lx] = 0x%08x\n",
+ addr, data);
+ }
+
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_WO(mem_read);
+
+static ssize_t enable_uart_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ return scnprintf(buf, MAX_STR_COPY, "%d\n", mnh_boot_args &
+ MNH_UART_ENABLE);
+}
+
+static ssize_t enable_uart_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ unsigned long val = 0;
+ int ret;
+
+ ret = mnh_sm_get_val_from_buf(buf, &val);
+ if (!ret) {
+ if (val)
+ mnh_boot_args |= MNH_UART_ENABLE;
+ else
+ mnh_boot_args &= ~MNH_UART_ENABLE;
+ return count;
+ }
+
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_RW(enable_uart);
+
+static ssize_t power_mode_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ return scnprintf(buf, MAX_STR_COPY, "%d\n", mnh_power_mode);
+}
+
+static ssize_t power_mode_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ int val = 0;
+ int ret;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (!ret) {
+ mnh_power_mode = val;
+ return count;
+ }
+
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_RW(power_mode);
+
+static ssize_t fw_ver_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ return scnprintf(buf, MAX_STR_COPY, "%s\n", mnh_sm_dev->mnh_fw_ver);
+}
+
+static DEVICE_ATTR_RO(fw_ver);
+
+static int mnh_sm_read_mipi_interrupts(void)
+{
+ int i, int_event = 0;
+ struct mipi_device_irq_st dev_ints;
+ struct mipi_host_irq_st host_ints;
+
+ memset((void *)&dev_ints, 0, sizeof(dev_ints));
+ memset((void *)&host_ints, 0, sizeof(host_ints));
+
+ dev_info(mnh_sm_dev->dev, "%s: Querying MIPI interrupts\n", __func__);
+ /* Read device & top interrupts */
+ for (i = 0; i <= 1; i++) {
+ dev_ints.dev = i;
+ mnh_mipi_get_device_interrupts(mnh_sm_dev->dev, &dev_ints);
+ if (dev_ints.main || dev_ints.fifo_overflow)
+ int_event = 1;
+ }
+ /* Read host interrupts */
+ for (i = 0; i <= 2; i++) {
+ host_ints.dev = i;
+ mnh_mipi_get_host_interrupts(mnh_sm_dev->dev, &host_ints);
+ if (host_ints.main)
+ int_event = 1;
+ }
+
+ return int_event;
+}
+
+static ssize_t mipi_int_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ return scnprintf(buf, MAX_STR_COPY, "%d\n",
+ mnh_sm_read_mipi_interrupts());
+}
+
+static DEVICE_ATTR_RO(mipi_int);
+
+static ssize_t spi_boot_mode_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ return scnprintf(buf, MAX_STR_COPY, "%d\n", mnh_boot_mode);
+}
+
+static ssize_t spi_boot_mode_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int val = 0;
+ int ret;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (ret || (val < MNH_BOOT_MODE_PCIE) || (val > MNH_BOOT_MODE_SPI))
+ return -EINVAL;
+
+ if (val != mnh_boot_mode) {
+ mnh_pwr_set_state(MNH_PWR_S4);
+ gpiod_set_value(mnh_sm_dev->boot_mode_gpio, val);
+ mnh_boot_mode = val;
+
+ if (val == MNH_BOOT_MODE_SPI)
+ mnh_pwr_set_state(MNH_PWR_S0);
+ }
+
+ return count;
+}
+
+static DEVICE_ATTR_RW(spi_boot_mode);
+
+static ssize_t error_event_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ return scnprintf(buf, MAX_STR_COPY, "%d\n", 0);
+}
+
+static DEVICE_ATTR_RO(error_event);
+
+static ssize_t mipi_config_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ unsigned long val = 0;
+ int ret;
+ struct mnh_mipi_config cfg = {.vc_en_mask = MNH_MIPI_VC_ALL_EN_MASK};
+
+ ret = mnh_sm_get_token_val_from_buf((char **)&buf, &val);
+ if (ret)
+ goto mipi_config_arg_fail;
+ cfg.rxdev = val;
+
+ ret = mnh_sm_get_token_val_from_buf((char **)&buf, &val);
+ if (ret)
+ goto mipi_config_arg_fail;
+ cfg.rx_rate = val;
+
+ ret = mnh_sm_get_token_val_from_buf((char **)&buf, &val);
+ if (ret)
+ goto mipi_config_arg_fail;
+ cfg.txdev = val;
+
+ ret = mnh_sm_get_token_val_from_buf((char **)&buf, &val);
+ if (ret)
+ goto mipi_config_arg_fail;
+ cfg.tx_rate = val;
+
+ mnh_mipi_stop(mnh_sm_dev->dev, cfg);
+ mnh_mipi_config(mnh_sm_dev->dev, cfg);
+
+ return count;
+
+mipi_config_arg_fail:
+ dev_err(dev,
+ "%s: invalid argument; expecting \"{rx_device};{rx_rate};{tx_device};{tx_rate}\"\n",
+ __func__);
+
+ return -EINVAL;
+}
+
+static DEVICE_ATTR_WO(mipi_config);
+
+static ssize_t ddr_mbist_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int val = 0;
+ int ret;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (ret || ((val != LIMITED_MOVI1_3N) && (val != MOVI1_3N)))
+ return -EINVAL;
+
+ mnh_pwr_set_state(MNH_PWR_S0);
+ mnh_ddr_po_init(&mnh_sm_dev->mnh_ddr_data,
+ mnh_sm_dev->ddr_pad_iso_n_pin);
+ mnh_ddr_mbist(&mnh_sm_dev->mnh_ddr_data, val);
+ mnh_pwr_set_state(MNH_PWR_S4);
+
+ return count;
+}
+
+static DEVICE_ATTR_WO(ddr_mbist);
+
+#if IS_ENABLED(CONFIG_MNH_SIG)
+/* issue signature verification of the firmware in memory and print results */
+static ssize_t verify_fw_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ int err, i;
+
+ dev_info(dev, "Starting signature check\n");
+ err = mnh_verify_firmware_ion(mnh_sm_dev->ion[FW_PART_PRI]);
+ for (i = 0; i < MAX_NR_MNH_FW_SLOTS; i++) {
+ dev_info(dev, "Signature check slot[%d]:%d\n", i,
+ mnh_sm_dev->ion[FW_PART_PRI]->fw_array[i].cert);
+ }
+ return scnprintf(buf, MAX_STR_COPY, "%d", err);
+}
+
+static DEVICE_ATTR_RO(verify_fw);
+#endif
+
+static struct attribute *mnh_sm_attrs[] = {
+ &dev_attr_stage_fw.attr,
+ &dev_attr_poweron.attr,
+ &dev_attr_dump_scu.attr,
+ &dev_attr_poweroff.attr,
+ &dev_attr_state.attr,
+ &dev_attr_download.attr,
+ &dev_attr_suspend.attr,
+ &dev_attr_resume.attr,
+ &dev_attr_reset.attr,
+ &dev_attr_cpu_clk.attr,
+ &dev_attr_debug_mipi.attr,
+ &dev_attr_freeze_state.attr,
+ &dev_attr_mipi_stop.attr,
+ &dev_attr_boot_args.attr,
+ &dev_attr_enable_uart.attr,
+ &dev_attr_power_mode.attr,
+ &dev_attr_mipi_int.attr,
+ &dev_attr_spi_boot_mode.attr,
+ &dev_attr_boot_trace.attr,
+ &dev_attr_config_read.attr,
+ &dev_attr_mem_read.attr,
+ &dev_attr_error_event.attr,
+ &dev_attr_fw_ver.attr,
+ &dev_attr_mipi_config.attr,
+ &dev_attr_ddr_mbist.attr,
+#if IS_ENABLED(CONFIG_MNH_SIG)
+ &dev_attr_verify_fw.attr,
+#endif
+ NULL
+};
+ATTRIBUTE_GROUPS(mnh_sm);
+
+static void mnh_sm_state_stats_init(void)
+{
+ mnh_sm_dev->state_stats[mnh_sm_dev->state].counter++;
+ mnh_sm_dev->state_stats[mnh_sm_dev->state].last_entry =
+ ktime_get_boottime();
+}
+
+static void mnh_sm_record_state_change(int prev_state, int new_state)
+{
+ ktime_t time;
+
+ if (new_state != prev_state) {
+ time = ktime_get_boottime();
+ mnh_sm_dev->state_stats[new_state].counter++;
+ mnh_sm_dev->state_stats[new_state].last_entry = time;
+ mnh_sm_dev->state_stats[prev_state].last_exit = time;
+ time = ktime_sub(mnh_sm_dev->state_stats[prev_state].last_exit,
+ mnh_sm_dev->state_stats[prev_state].last_entry);
+ mnh_sm_dev->state_stats[prev_state].duration = ktime_add(
+ mnh_sm_dev->state_stats[prev_state].duration, time);
+ }
+}
+
+static ssize_t mnh_sm_dbgfs_read_powerstats(struct file *fp, char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ const char * const states[MNH_STATE_MAX] = {"OFF", "ACTIVE", "SUSPEND"};
+ char *buf;
+ int pos = 0;
+ int ret, i;
+ ktime_t last_entry;
+ ktime_t last_exit;
+ ktime_t adjusted_duration;
+ ktime_t partial_duration;
+
+
+#define BUFFER_MAX_LEN (4096)
+#define POWER_DEBUGFS_HEADER "Easel Subsystem Power Stats\n"
+#define PRINT_FORMAT "%s\n"\
+ "\tCumulative count: %d\n"\
+ "\tCumulative duration msec: %lld\n"\
+ "\tLast entry timestamp msec: %lld\n"\
+ "\tLast exit timestamp msec: %lld\n"
+
+
+ buf = kzalloc(BUFFER_MAX_LEN, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ pos += scnprintf(buf + pos, BUFFER_MAX_LEN - pos, POWER_DEBUGFS_HEADER);
+
+ mutex_lock(&mnh_sm_dev->lock);
+
+ for (i = 0; i < MNH_STATE_MAX; i++) {
+ adjusted_duration = mnh_sm_dev->state_stats[i].duration;
+ last_entry = mnh_sm_dev->state_stats[i].last_entry;
+ last_exit = mnh_sm_dev->state_stats[i].last_exit;
+
+ /* adjust duration for current state */
+ if (mnh_sm_dev->state == i &&
+ ktime_after(last_entry, last_exit)) {
+ partial_duration = ktime_sub(ktime_get_boottime(),
+ last_entry);
+ adjusted_duration = ktime_add(adjusted_duration,
+ partial_duration);
+ }
+
+ pos += scnprintf(buf + pos, BUFFER_MAX_LEN - pos,
+ PRINT_FORMAT, states[i],
+ mnh_sm_dev->state_stats[i].counter,
+ ktime_to_ms(adjusted_duration),
+ ktime_to_ms(last_entry),
+ ktime_to_ms(last_exit));
+ }
+
+ mutex_unlock(&mnh_sm_dev->lock);
+#undef BUFFER_MAX_LEN
+#undef POWER_DEBUGFS_HEADER
+#undef PRINT_FORMAT
+
+ ret = simple_read_from_buffer(ubuf, cnt, ppos, buf, pos);
+ kfree(buf);
+
+ return ret;
+}
+
+static const struct file_operations mnh_sm_dbgfs_fops_powerstats = {
+ .open = simple_open,
+ .read = mnh_sm_dbgfs_read_powerstats,
+};
+
+static void mnh_sm_dbgfs_deregister(void)
+{
+ if (!mnh_sm_dev->dbgfs_dir)
+ return;
+
+ debugfs_remove_recursive(mnh_sm_dev->dbgfs_dir);
+ mnh_sm_dev->dbgfs_dir = NULL;
+}
+
+static int mnh_sm_dbgfs_register(const char *name)
+{
+ struct dentry *dir, *f;
+
+ dir = debugfs_create_dir(name, NULL);
+
+ if (!dir) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: failed to create debug dir\n",
+ __func__);
+ return -ENOMEM;
+ }
+
+ mnh_sm_dev->dbgfs_dir = dir;
+
+ f = debugfs_create_file("power_stats", 0444, dir,
+ NULL, &mnh_sm_dbgfs_fops_powerstats);
+
+ if (!f) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: failed to create powerstats\n",
+ __func__);
+
+ mnh_sm_dbgfs_deregister();
+ return -ENODEV;
+ } else {
+ return 0;
+ }
+}
+
+/*******************************************************************************
+ *
+ * APIs
+ *
+ ******************************************************************************/
+
+/**
+ * API to register hotplug callback to receive MNH up/down notifications
+ * @param[in] hotplug_cb handler for hotplug in/out events
+ * @return 0
+ */
+int mnh_sm_reg_hotplug_callback(hotplug_cb_t hotplug_cb)
+{
+ mnh_hotplug_cb = hotplug_cb;
+ return 0;
+}
+EXPORT_SYMBOL(mnh_sm_reg_hotplug_callback);
+
+static int mnh_sm_hotplug_callback(enum mnh_hotplug_event_t event)
+{
+ if (!mnh_hotplug_cb)
+ return -EFAULT;
+
+ if ((event == MNH_HOTPLUG_IN) && (mnh_boot_args & MNH_UART_ENABLE)) {
+ dev_info(mnh_sm_dev->dev,
+ "%s: allow %d secs for MNH_HOTPLUG_IN\n",
+ __func__, HOTPLUG_IN_LOOSE_TIMEOUT_MS / 1000);
+ return mnh_hotplug_cb(event,
+ (void *)HOTPLUG_IN_LOOSE_TIMEOUT_MS);
+ }
+
+ return mnh_hotplug_cb(event, NULL);
+}
+
+/**
+ * API to initialize Power and clocks to MNH.
+ * @param[in] Structure argument to configure power and clock component.
+ * This structure will be populated within the kernel module.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+static int mnh_sm_poweron(void)
+{
+ int ret;
+
+ /* Initialize MNH Power */
+ ret = mnh_pwr_set_state(MNH_PWR_S0);
+ if (ret) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: failed to power on device (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+/**
+ * API to power monette hill.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+static int mnh_sm_poweroff(void)
+{
+ mnh_pwr_set_state(MNH_PWR_S4);
+ mnh_sm_dev->ddr_status = MNH_DDR_OFF;
+ mnh_sm_dev->firmware_downloaded = false;
+ return 0;
+}
+
+static int mnh_sm_config_ddr(void)
+{
+ int ret;
+
+ /* Initialize DDR */
+ ret = mnh_ddr_po_init(&mnh_sm_dev->mnh_ddr_data,
+ mnh_sm_dev->ddr_pad_iso_n_pin);
+ if (ret) {
+ dev_err(mnh_sm_dev->dev, "%s: ddr training failed (%d)\n",
+ __func__, ret);
+ return ret;
+ }
+
+ mnh_sm_dev->ddr_status = MNH_DDR_ACTIVE;
+ return 0;
+}
+
+static int mnh_sm_resume_ddr(void)
+{
+ /* deassert pad isolation, take ddr out of self-refresh mode */
+ mnh_ddr_resume(&mnh_sm_dev->mnh_ddr_data,
+ mnh_sm_dev->ddr_pad_iso_n_pin);
+ mnh_sm_dev->ddr_status = MNH_DDR_ACTIVE;
+ return 0;
+}
+
+static int mnh_sm_suspend_ddr(void)
+{
+ /* put ddr into self-refresh mode, assert pad isolation */
+ mnh_ddr_suspend(&mnh_sm_dev->mnh_ddr_data,
+ mnh_sm_dev->ddr_pad_iso_n_pin);
+ mnh_sm_dev->ddr_status = MNH_DDR_SELF_REFRESH;
+ return 0;
+}
+
+/**
+ * API to download the binary images(SBL, UBoot, Kernel, Ramdisk) for mnh.
+ * The location of the binaries will be located in the AP file system.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+static int mnh_sm_download(void)
+{
+ uint32_t magic;
+ int ret;
+ ktime_t start, end;
+ static int iter;
+
+ iter++;
+ start = ktime_get();
+ ret = mnh_download_firmware();
+ end = ktime_get();
+ dev_dbg(mnh_sm_dev->dev, "iter:%d took %d ms to download firmware\n",
+ iter, (unsigned)ktime_to_ms(ktime_sub(end, start)));
+
+ if (ret) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: firmware download failed\n", __func__);
+ return ret;
+ }
+
+ /* set the boot_args mask */
+ MNH_SCU_OUT(GP_BOOT_ARGS, mnh_boot_args);
+
+ /* set the default power mode */
+ MNH_SCU_OUT(GP_POWER_MODE, mnh_power_mode);
+
+ /*
+ * Magic number setting to notify MNH that PCIE initialization
+ * is done on Host side
+ */
+ if (mnh_config_read(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET +
+ HW_MNH_PCIE_GP_0,
+ sizeof(uint32_t), &magic) == SUCCESS &&
+ magic == 0) {
+ mnh_config_write(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET +
+ HW_MNH_PCIE_GP_0,
+ sizeof(uint32_t), INIT_DONE);
+ } else {
+ dev_err(mnh_sm_dev->dev,
+ "Read GP0 register fail or GP0 is not 0:%d",
+ magic);
+ }
+
+ return 0;
+}
+
+/**
+ * API to put MNH in suspend state. In suspend mode the DDR will be isolated
+ * and put in self refresh while the CPU is powered down.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+static int mnh_sm_suspend(void)
+{
+ unsigned long timeout;
+ int ret;
+
+ dev_dbg(mnh_sm_dev->dev, "%s: waiting for suspend signal\n", __func__);
+
+ /* Wait until MNH is ready to go to suspend */
+ timeout = wait_for_completion_timeout(&mnh_sm_dev->suspend_complete,
+ SUSPEND_COMPLETE_TIMEOUT);
+ if (!timeout) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: timeout waiting for device to suspend kernel\n",
+ __func__);
+ return -ETIMEDOUT;
+ }
+
+ dev_dbg(mnh_sm_dev->dev, "%s: suspend successful.\n", __func__);
+
+ /* Read resume entry address */
+ ret = mnh_config_read(HW_MNH_PCIE_CLUSTER_ADDR_OFFSET +
+ HW_MNH_PCIE_GP_7, sizeof(uint32_t),
+ &mnh_sm_dev->resume_addr);
+ if (ret != SUCCESS) {
+ dev_err(mnh_sm_dev->dev, "%s: resume entry addr read failed\n",
+ __func__);
+ return -EIO;
+ }
+ dev_dbg(mnh_sm_dev->dev, "%s: resume entry: 0x%x\n", __func__,
+ mnh_sm_dev->resume_addr);
+
+ mnh_sm_suspend_ddr();
+
+ /* Suspend MNH power */
+ mnh_pwr_set_state(MNH_PWR_S3);
+ return 0;
+}
+
+/**
+ * API to put MNH into active state.
+ * The resume call flow should be similar to normal bootflow except for DDR
+ * initializations. Since the binaries are already saved on the DDR while MNH
+ * is in suspend, ESM will not need to download the binaries again during
+ * resume.
+ * @return 0 if success or -EINVAL or -EFATAL on failure
+ */
+static int mnh_sm_resume(void)
+{
+ /* set the boot_args mask */
+ MNH_SCU_OUT(GP_BOOT_ARGS, mnh_boot_args);
+
+ /* set the default power mode */
+ MNH_SCU_OUT(GP_POWER_MODE, mnh_power_mode);
+
+ mnh_resume_firmware();
+ return 0;
+}
+
+/**
+ * API to obtain the state of monette hill.
+ * @return the power states of mnh
+ */
+int mnh_sm_get_state(void)
+{
+ int state;
+
+ if (!mnh_sm_dev)
+ return MNH_STATE_OFF;
+
+ mutex_lock(&mnh_sm_dev->lock);
+ state = mnh_sm_dev->state;
+ mutex_unlock(&mnh_sm_dev->lock);
+
+ return state;
+}
+EXPORT_SYMBOL(mnh_sm_get_state);
+
+static void mnh_sm_print_boot_trace(struct device (*dev))
+{
+ int err;
+ uint32_t val;
+
+ err = mnh_config_read(MNH_BOOT_TRACE, sizeof(val), &val);
+
+ if (err) {
+ dev_err(dev,
+ "%s: failed reading MNH_BOOT_TRACE (%d)\n",
+ __func__, err);
+ return;
+ }
+
+ dev_info(dev, "%s: MNH_BOOT_TRACE = 0x%x\n", __func__, val);
+}
+
+static void mnh_sm_enable_ready_irq(bool enable)
+{
+ /* irqs are automatically enabled during request_irq */
+ static bool is_enabled = true;
+
+ if (enable && !is_enabled) {
+ enable_irq(mnh_sm_dev->ready_irq);
+ is_enabled = true;
+ } else if (!enable && is_enabled) {
+ disable_irq(mnh_sm_dev->ready_irq);
+ is_enabled = false;
+ } else {
+ dev_warn(mnh_sm_dev->dev, "%s: ready_irq already %s\n",
+ __func__, is_enabled ? "enabled" : "disabled");
+ }
+}
+
+/*
+ * NOTE (b/64372955): Put Easel into a low-power mode for MIPI bypass. Ideally,
+ * this would be done from Easel kernel, but if the Easel kernel fails for some
+ * reason, we can disable a lot of clocks on Easel to reduce power.
+ */
+__maybe_unused static void mnh_sm_enter_low_power_mode(void)
+{
+ /* disable clocks for unused system components */
+ MNH_SCU_OUTf(CCU_CLK_CTL, CPU_CLKEN, 0);
+ MNH_SCU_OUTf(CCU_CLK_CTL, BTSRAM_CLKEN, 0);
+ MNH_SCU_OUTf(CCU_CLK_CTL, BTROM_CLKEN, 0);
+ MNH_SCU_OUTf(CCU_CLK_CTL, LP4_REFCLKEN, 0);
+ MNH_SCU_OUTf(CCU_CLK_CTL, IPU_CLKEN, 0);
+
+ /* disable clocks for unused peripherals */
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, PVT_CLKEN, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, PCIE_REFCLKEN, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, I2C3_CLKEN_SW, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, I2C2_CLKEN_SW, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, I2C1_CLKEN_SW, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, I2C0_CLKEN_SW, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, UART1_CLKEN_SW, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, UART0_CLKEN_SW, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, SPIS_CLKEN_SW, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, SPIM_CLKEN_SW, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, PERI_DMA_CLKEN_SW, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, TIMER_CLKEN_SW, 0);
+ MNH_SCU_OUTf(PERIPH_CLK_CTRL, WDT_CLKEN_SW, 0);
+
+ /* reset unused peripherals */
+ MNH_SCU_OUTf(RSTC, I2C3_RST, 1);
+ MNH_SCU_OUTf(RSTC, I2C2_RST, 1);
+ MNH_SCU_OUTf(RSTC, I2C1_RST, 1);
+ MNH_SCU_OUTf(RSTC, I2C0_RST, 1);
+ MNH_SCU_OUTf(RSTC, UART1_RST, 1);
+ MNH_SCU_OUTf(RSTC, UART0_RST, 1);
+ MNH_SCU_OUTf(RSTC, SPIS_RST, 1);
+ MNH_SCU_OUTf(RSTC, SPIM_RST, 1);
+ MNH_SCU_OUTf(RSTC, PERI_DMA_RST, 1);
+ MNH_SCU_OUTf(RSTC, TIMER_RST, 1);
+ MNH_SCU_OUTf(RSTC, WDT_RST, 1);
+ MNH_SCU_OUTf(RSTC, PMON_RST, 1);
+ MNH_SCU_OUTf(RSTC, MIPIRXPHY_RST, 1);
+ MNH_SCU_OUTf(RSTC, MIPITXPHY_RST, 1);
+ MNH_SCU_OUTf(RSTC, LP4PHY_RST, 1);
+ MNH_SCU_OUTf(RSTC, LP4CTL_RST, 1);
+ MNH_SCU_OUTf(RSTC, IPU_RST, 1);
+ MNH_SCU_OUTf(RSTC, CPU_RST, 1);
+
+ /* Shutdown unused memories */
+ MNH_SCU_OUTf(MEM_PWR_MGMNT, BTROM_SLP, 1);
+ MNH_SCU_OUTf(MEM_PWR_MGMNT, BTSRAM_SD, 1);
+ MNH_SCU_OUTf(MEM_PWR_MGMNT, LP4C_MEM_SD, 1);
+ MNH_SCU_OUTf(MEM_PWR_MGMNT, CPU_L2MEM_SD, 1);
+ MNH_SCU_OUTf(MEM_PWR_MGMNT, CPU_L1MEM_SD, 1);
+ MNH_SCU_OUTf(MEM_PWR_MGMNT, IPU_MEM_SD, 1);
+
+ /* switch to SYS200 clock */
+ MNH_SCU_OUTf(CCU_CLK_DIV, LPDDR4_REFCLK_DIV, 0xB);
+ MNH_SCU_OUTf(CCU_CLK_DIV, AXI_FABRIC_CLK_DIV, 1);
+ MNH_SCU_OUTf(CCU_CLK_DIV, PCIE_AXI_CLK_DIV, 3);
+ MNH_SCU_OUTf(CCU_CLK_CTL, CPU_IPU_SYS200_MODE, 1);
+ MNH_SCU_OUTf(CCU_CLK_CTL, LP4_AXI_SYS200_MODE, 1);
+ MNH_SCU_OUTf(LPDDR4_LOW_POWER_CFG, LP4_FSP_SW_OVERRIDE, 1);
+
+ /* disable PLLs */
+ MNH_SCU_OUTf(PLL_PASSCODE, PASSCODE, 0x4CD9);
+ MNH_SCU_OUTf(IPU_PLL_CTRL, FRZ_PLL_IN, 1);
+ MNH_SCU_OUTf(IPU_PLL_CTRL, PD, 1);
+ MNH_SCU_OUTf(IPU_PLL_CTRL, FOUTPOSTDIVPD, 1);
+ MNH_SCU_OUTf(IPU_PLL_CTRL, FRZ_PLL_IN, 0);
+ MNH_SCU_OUTf(CPU_IPU_PLL_CTRL, FRZ_PLL_IN, 1);
+ MNH_SCU_OUTf(CPU_IPU_PLL_CTRL, PD, 1);
+ MNH_SCU_OUTf(CPU_IPU_PLL_CTRL, FOUTPOSTDIVPD, 1);
+ MNH_SCU_OUTf(CPU_IPU_PLL_CTRL, FRZ_PLL_IN, 0);
+ MNH_SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, FRZ_PLL_IN, 1);
+ MNH_SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, PD, 1);
+ MNH_SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, FOUTPOSTDIVPD, 1);
+ MNH_SCU_OUTf(LPDDR4_REFCLK_PLL_CTRL, FRZ_PLL_IN, 0);
+ MNH_SCU_OUTf(PLL_PASSCODE, PASSCODE, 0x0);
+
+ /* enable pad isolation to the DRAM */
+ gpiod_set_value_cansleep(mnh_sm_dev->ddr_pad_iso_n_pin, 0);
+
+ /* disable DRAM power */
+ mnh_pwr_set_state(MNH_PWR_S1);
+
+ mnh_sm_dev->ddr_status = MNH_DDR_OFF;
+ mnh_sm_dev->firmware_downloaded = false;
+}
+
+static int mnh_sm_set_state_locked(int state)
+{
+ struct device *dev = mnh_sm_dev->dev;
+ int ret = 0;
+ enum fw_image_partition_index fw_idx;
+
+#if IS_ENABLED(CONFIG_MNH_SIG)
+ fw_idx = FW_PART_TER;
+#else
+ fw_idx = FW_PART_SEC;
+#endif
+
+ if (state == mnh_sm_dev->state) {
+ dev_dbg(dev, "%s: already in state %d\n", __func__, state);
+ return 0;
+ }
+
+ switch (state) {
+ case MNH_STATE_OFF:
+ mnh_sm_hotplug_callback(MNH_HOTPLUG_OUT);
+
+ /* toggle powered flag and clear completion */
+ mnh_sm_dev->powered = false;
+ reinit_completion(&mnh_sm_dev->powered_complete);
+
+ ret = mnh_sm_poweroff();
+
+ /* enable pad isolation to the DRAM */
+ gpiod_set_value_cansleep(mnh_sm_dev->ddr_pad_iso_n_pin, 0);
+
+ mnh_sm_enable_ready_irq(false);
+ break;
+ case MNH_STATE_ACTIVE:
+ /* stage firmware copy to ION if valid update was received */
+ if (mnh_sm_dev->update_status == FW_UPD_VERIFIED) {
+ dev_info(mnh_sm_dev->dev,
+ "%s: staging firmware update",
+ __func__);
+ mnh_ion_stage_firmware_update(
+ mnh_sm_dev->ion[FW_PART_PRI],
+ mnh_sm_dev->ion[fw_idx]
+ );
+ mnh_ion_destroy_buffer(mnh_sm_dev->ion[fw_idx]);
+ mnh_update_fw_version(mnh_sm_dev->ion[FW_PART_PRI]);
+ mnh_sm_dev->update_status = FW_UPD_NONE;
+ } else if (mnh_sm_dev->ion &&
+ !mnh_sm_dev->ion[FW_PART_PRI]->is_fw_ready) {
+ /* Request firmware and stage them to carveout buf. */
+ dev_info(mnh_sm_dev->dev, "%s: staging firmware\n",
+ __func__);
+ mnh_ion_stage_firmware(mnh_sm_dev->ion[FW_PART_PRI]);
+
+ /* Extract firmware version from ramdisk image */
+ mnh_update_fw_version(mnh_sm_dev->ion[FW_PART_PRI]);
+ }
+
+ mnh_sm_enable_ready_irq(true);
+ ret = mnh_sm_poweron();
+ if (ret)
+ break;
+
+ /* toggle powered flag and notify any waiting threads */
+ mnh_sm_dev->powered = true;
+ complete(&mnh_sm_dev->powered_complete);
+
+ /* make sure ddr is configured */
+ if (mnh_sm_dev->ddr_status == MNH_DDR_OFF)
+ ret = mnh_sm_config_ddr();
+ else if (mnh_sm_dev->ddr_status == MNH_DDR_SELF_REFRESH)
+ ret = mnh_sm_resume_ddr();
+ if (ret)
+ break;
+
+ /* use max CPU frequency for fast boot */
+ mnh_cpu_freq_change(CPU_FREQ_950);
+ mnh_ipu_freq_change(IPU_FREQ_425);
+
+ /* have we downloaded firmware already? */
+ if (mnh_sm_dev->firmware_downloaded)
+ ret = mnh_sm_resume();
+ else
+ ret = mnh_sm_download();
+ if (ret)
+ break;
+
+ ret = mnh_sm_hotplug_callback(MNH_HOTPLUG_IN);
+ if (ret)
+ mnh_sm_print_boot_trace(mnh_sm_dev->dev);
+
+ break;
+ case MNH_STATE_SUSPEND:
+ ret = mnh_sm_set_state_locked(MNH_STATE_ACTIVE);
+ if (!ret) {
+ mnh_sm_hotplug_callback(MNH_HOTPLUG_OUT);
+
+ /* toggle powered flag and clear completion */
+ mnh_sm_dev->powered = false;
+ reinit_completion(&mnh_sm_dev->powered_complete);
+
+ ret = mnh_sm_suspend();
+
+ mnh_sm_enable_ready_irq(false);
+ }
+ break;
+ default:
+ dev_err(mnh_sm_dev->dev,
+ "%s: invalid state %d\n", __func__, state);
+ ret = -EINVAL;
+ }
+
+ if (ret) {
+ /*
+ * NOTE (b/64372955): the minimum requirements for MIPI bypass
+ * are power and PCIe. A successful boot is only necessary for
+ * easelcomm communication. Return a special code to indicate
+ * that we reached the minimum requirements, but couldn't
+ * establish host communication. Also, disable a few clocks on
+ * Easel to reduce power.
+ */
+#if ALLOW_PARTIAL_ACTIVE
+ if ((state == MNH_STATE_ACTIVE) && (mnh_sm_dev->powered)) {
+ dev_warn(mnh_sm_dev->dev,
+ "%s: failed to fully transition to state %d (%d), allow partial active\n",
+ __func__, state, ret);
+ mnh_sm_enter_low_power_mode();
+ ret = -EHOSTUNREACH;
+ } else {
+#endif
+ dev_err(mnh_sm_dev->dev,
+ "%s: failed to transition to state %d (%d)\n",
+ __func__, state, ret);
+
+ if (state == MNH_STATE_ACTIVE) {
+ mnh_sm_dev->powered = false;
+ reinit_completion(
+ &mnh_sm_dev->powered_complete);
+ mnh_sm_poweroff();
+ mnh_sm_enable_ready_irq(false);
+ mnh_sm_dev->state = MNH_STATE_OFF;
+ }
+
+ return ret;
+#if ALLOW_PARTIAL_ACTIVE
+ }
+#endif
+ }
+
+ mnh_sm_dev->state = state;
+
+ return ret;
+}
+
+/**
+ * API to set the state of monette hill.
+ * @param[in] Set the power states of mnh
+ */
+int mnh_sm_set_state(int state)
+{
+ int ret;
+ int prev_state;
+
+ if (!mnh_sm_dev)
+ return -ENODEV;
+
+ dev_info(mnh_sm_dev->dev, "%s: request state %d\n", __func__, state);
+
+ if (mnh_freeze_state) {
+ dev_info(mnh_sm_dev->dev,
+ "%s: ignoring requested state %d because freeze_state is set\n",
+ __func__, mnh_sm_dev->next_state);
+ return -EBUSY;
+ }
+
+ mutex_lock(&mnh_sm_dev->lock);
+
+ prev_state = mnh_sm_dev->state;
+
+ /* on boot/resume, hold wakelock with timeout */
+ if (state == MNH_STATE_ACTIVE)
+ pm_wakeup_event(mnh_sm_dev->dev,
+ MNH_SM_WAKEUP_SOURCE_TIMEOUT_PRODUCTION);
+
+ ret = mnh_sm_set_state_locked(state);
+
+ /* release wakelock immediately if ended up not active */
+ if (mnh_sm_dev->state != MNH_STATE_ACTIVE)
+ pm_relax(mnh_sm_dev->dev);
+
+ /* record state change */
+ mnh_sm_record_state_change(prev_state, mnh_sm_dev->state);
+
+ mutex_unlock(&mnh_sm_dev->lock);
+
+ if (ret)
+ return ret;
+
+ dev_info(mnh_sm_dev->dev, "%s: finished state %d\n", __func__,
+ mnh_sm_dev->state);
+
+ return 0;
+}
+EXPORT_SYMBOL(mnh_sm_set_state);
+
+int mnh_sm_is_present(void)
+{
+ return (mnh_sm_dev != NULL) ? 1 : 0;
+}
+EXPORT_SYMBOL(mnh_sm_is_present);
+
+int mnh_sm_mipi_config(struct mnh_mipi_config config)
+{
+ if (!mnh_sm_dev)
+ return -ENODEV;
+
+ return mnh_mipi_config(mnh_sm_dev->dev, config);
+}
+EXPORT_SYMBOL_GPL(mnh_sm_mipi_config);
+
+int mnh_sm_pwr_error_cb(void)
+{
+ dev_err(mnh_sm_dev->dev,
+ "%s: observed mnh-pwr error, switching state to off\n",
+ __func__);
+ sysfs_notify(&mnh_sm_dev->dev->kobj, NULL, "error_event");
+ return mnh_sm_set_state(MNH_STATE_OFF);
+}
+EXPORT_SYMBOL(mnh_sm_pwr_error_cb);
+
+enum mnh_boot_mode mnh_sm_get_boot_mode(void)
+{
+ return mnh_boot_mode;
+}
+EXPORT_SYMBOL(mnh_sm_get_boot_mode);
+
+static int mnh_sm_open(struct inode *inode, struct file *filp)
+{
+ int dev_ctr;
+
+ if (!mnh_sm_dev)
+ return -ENODEV;
+
+ if (!mnh_sm_dev->initialized)
+ return -EBUSY;
+
+ filp->private_data = mnh_sm_dev; /* for other methods */
+
+ dev_ctr = atomic_inc_return(&mnh_sm_dev->cdev_ctr);
+ dev_dbg(mnh_sm_dev->dev, "%s: opening mnh_sm: mnh_sm_dev->cdev_ctr %d\n",
+ __func__, dev_ctr);
+
+ /* only stage fw transf. when the first handle to the cdev is opened */
+ if (dev_ctr == 1) {
+ if (mnh_sm_dev->ion &&
+ !mnh_sm_dev->ion[FW_PART_PRI]->is_fw_ready) {
+ /* Request firmware and stage them to carveout buf. */
+ dev_dbg(mnh_sm_dev->dev, "%s: staging firmware\n",
+ __func__);
+ mnh_ion_stage_firmware(mnh_sm_dev->ion[FW_PART_PRI]);
+ /* Extract firmware version from ramdisk image */
+ mnh_update_fw_version(mnh_sm_dev->ion[FW_PART_PRI]);
+ }
+ }
+
+ /* perform signature check on firmware images */
+#if IS_ENABLED(CONFIG_MNH_SIG_FORCE_OTA)
+ mnh_verify_firmware_ion(mnh_sm_dev->ion[FW_PART_PRI]);
+#endif
+ return 0;
+}
+
+static int mnh_sm_close(struct inode *inode, struct file *filp)
+{
+ if (!mnh_sm_dev)
+ return -ENODEV;
+
+ if (!mnh_sm_dev->initialized)
+ return -EBUSY;
+
+ /* Only shut mnh down when there is no active handle to the cdev */
+ if (atomic_dec_and_test(&mnh_sm_dev->cdev_ctr)) {
+ mnh_sm_set_state(MNH_STATE_OFF);
+ dev_dbg(mnh_sm_dev->dev, "%s: closing mnh_sm\n", __func__);
+ }
+
+ return 0;
+}
+
+static void mnh_sm_set_state_work(struct kthread_work *data)
+{
+ mnh_sm_dev->work_ret = mnh_sm_set_state(mnh_sm_dev->next_state);
+ complete(&mnh_sm_dev->work_complete);
+}
+
+static int mnh_sm_wait_for_power(void)
+{
+ unsigned long timeout;
+
+ if (mnh_sm_dev->powered)
+ return 0;
+
+ timeout = wait_for_completion_timeout(&mnh_sm_dev->powered_complete,
+ POWERED_COMPLETE_TIMEOUT);
+ if (!timeout) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: timeout waiting for mnh to power on\n",
+ __func__);
+ return -ETIMEDOUT;
+ }
+
+ return 0;
+}
+
+static int mnh_sm_wait_for_state(int state)
+{
+ unsigned long timeout;
+
+ if (mnh_sm_dev->state == state)
+ return 0;
+
+ if (mnh_sm_dev->next_state != state)
+ return -EINVAL;
+
+ timeout = wait_for_completion_timeout(&mnh_sm_dev->work_complete,
+ STATE_CHANGE_COMPLETE_TIMEOUT);
+ if (!timeout) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: timeout waiting for state transition\n",
+ __func__);
+ return -ETIMEDOUT;
+ }
+
+ return mnh_sm_dev->work_ret;
+}
+
+static long mnh_sm_ioctl(struct file *file, unsigned int cmd,
+ unsigned long arg)
+{
+ int err = 0;
+ int fd = -1; /* handle for ion buffer sharing */
+ struct mnh_mipi_config mipi_config;
+ struct mnh_update_configs update_configs;
+
+ /*
+ * the direction is a bitmask, and VERIFY_WRITE catches R/W
+ * transfers. `Type' is user-oriented, while
+ * access_ok is kernel-oriented, so the concept of "read" and
+ * "write" is reversed
+ */
+ if (_IOC_DIR(cmd) & _IOC_READ)
+ err = !access_ok(VERIFY_WRITE, (void __user *)arg,
+ _IOC_SIZE(cmd));
+ else if (_IOC_DIR(cmd) & _IOC_WRITE)
+ err = !access_ok(VERIFY_READ, (void __user *)arg,
+ _IOC_SIZE(cmd));
+ if (err) {
+ dev_err(mnh_sm_dev->dev, "%s: access error!\n", __func__);
+ return -EFAULT;
+ }
+
+ /*
+ * extract the type and number bitfields, and don't decode
+ * wrong cmds: return ENOTTY (inappropriate ioctl) before access_ok( )
+ */
+
+ if (_IOC_TYPE(cmd) != MNH_SM_IOC_MAGIC) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: invalid ioctl type %c (0x%08x)\n",
+ __func__, _IOC_TYPE(cmd), cmd);
+ return -ENOTTY;
+ }
+
+ switch (cmd) {
+ case MNH_SM_IOC_GET_STATE:
+ err = copy_to_user((void __user *)arg, &mnh_sm_dev->state,
+ sizeof(mnh_sm_dev->state));
+ if (err) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: failed to copy to userspace (%d)\n",
+ __func__, err);
+ return err;
+ }
+ break;
+ case MNH_SM_IOC_SET_STATE:
+ mnh_sm_dev->next_state = (int)arg;
+ reinit_completion(&mnh_sm_dev->work_complete);
+ kthread_queue_work(&mnh_sm_dev->worker,
+ &mnh_sm_dev->set_state_work);
+ break;
+ case MNH_SM_IOC_WAIT_FOR_POWER:
+ err = mnh_sm_wait_for_power();
+ break;
+ case MNH_SM_IOC_WAIT_FOR_STATE:
+ err = mnh_sm_wait_for_state((int)arg);
+ break;
+ case MNH_SM_IOC_CONFIG_MIPI:
+ case MNH_SM_IOC_STOP_MIPI:
+ err = copy_from_user(&mipi_config, (void __user *)arg,
+ sizeof(struct mnh_mipi_config));
+ if (err) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: failed to copy mipi config from userspace (%d)\n",
+ __func__, err);
+ return err;
+ }
+ if (!mnh_sm_dev->powered)
+ return -EIO;
+ if (cmd == MNH_SM_IOC_CONFIG_MIPI)
+ mnh_mipi_config(mnh_sm_dev->dev, mipi_config);
+ else
+ mnh_mipi_stop(mnh_sm_dev->dev, mipi_config);
+ break;
+ case MNH_SM_IOC_GET_UPDATE_BUF:
+ mutex_lock(&mnh_sm_dev->fw_update_lock);
+ mnh_sm_dev->ion[FW_PART_SEC]->is_fw_ready = false;
+ if (mnh_sm_dev->ion[FW_PART_SEC]) {
+ if (mnh_ion_create_buffer(mnh_sm_dev->ion[FW_PART_SEC],
+ MNH_ION_BUFFER_SIZE,
+ ION_SYSTEM_HEAP_ID))
+ dev_warn(mnh_sm_dev->dev,
+ "%s: cannot claim ION buffer\n",
+ __func__);
+ }
+ fd = ion_share_dma_buf_fd(mnh_sm_dev->ion[FW_PART_SEC]->client,
+ mnh_sm_dev->ion[FW_PART_SEC]->handle);
+ err = copy_to_user((void __user *)arg, &fd, sizeof(fd));
+ if (err) {
+ mnh_ion_destroy_buffer(mnh_sm_dev->ion[FW_PART_SEC]);
+ mnh_sm_dev->update_status = FW_UPD_NONE;
+ dev_err(mnh_sm_dev->dev,
+ "%s: failed to copy to userspace (%d)\n",
+ __func__, err);
+ mutex_unlock(&mnh_sm_dev->fw_update_lock);
+ return err;
+ }
+ mnh_sm_dev->update_status = FW_UPD_INIT;
+ mutex_unlock(&mnh_sm_dev->fw_update_lock);
+ break;
+ case MNH_SM_IOC_POST_UPDATE_BUF:
+ mutex_lock(&mnh_sm_dev->fw_update_lock);
+ err = copy_from_user(&update_configs, (void __user *)arg,
+ sizeof(update_configs));
+ if (err) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: failed to copy from userspace (%d)\n",
+ __func__, err);
+ mutex_unlock(&mnh_sm_dev->fw_update_lock);
+ return err;
+ }
+ if (mnh_sm_dev->update_status == FW_UPD_INIT) {
+ mnh_sm_dev->update_status = FW_UPD_RECEIVED;
+ err = mnh_validate_update_buf(update_configs);
+ if (err) {
+ mnh_ion_destroy_buffer(
+ mnh_sm_dev->ion[FW_PART_SEC]);
+ mnh_sm_dev->update_status = FW_UPD_NONE;
+ dev_err(mnh_sm_dev->dev,
+ "%s: failed to validate slots (%d)\n",
+ __func__, err);
+ mutex_unlock(&mnh_sm_dev->fw_update_lock);
+ return err;
+ }
+ mnh_sm_dev->update_status = FW_UPD_VERIFIED;
+ }
+ mutex_unlock(&mnh_sm_dev->fw_update_lock);
+ break;
+ case MNH_SM_IOC_GET_FW_VER:
+ err = copy_to_user((void __user *)arg, &mnh_sm_dev->mnh_fw_ver,
+ sizeof(mnh_sm_dev->mnh_fw_ver));
+ if (err) {
+ dev_err(mnh_sm_dev->dev,
+ "%s: failed to copy to userspace (%d)\n",
+ __func__, err);
+ return err;
+ }
+ break;
+ default:
+ dev_err(mnh_sm_dev->dev,
+ "%s: unknown ioctl %c, dir=%d, #%d (0x%08x)\n",
+ __func__, _IOC_TYPE(cmd), _IOC_DIR(cmd), _IOC_NR(cmd),
+ cmd);
+ break;
+ }
+
+ return err;
+}
+
+static irqreturn_t mnh_sm_ready_irq_handler(int irq, void *cookie)
+{
+ if (gpiod_get_value(mnh_sm_dev->ready_gpio)) {
+ dev_info(mnh_sm_dev->dev, "%s: mnh device is ready to boot\n",
+ __func__);
+ reinit_completion(&mnh_sm_dev->suspend_complete);
+ } else {
+ dev_info(mnh_sm_dev->dev, "%s: mnh device is ready to suspend\n",
+ __func__);
+ complete(&mnh_sm_dev->suspend_complete);
+ }
+
+ return IRQ_HANDLED;
+}
+
+static long mnh_sm_compat_ioctl(struct file *file, unsigned int cmd,
+ unsigned long arg)
+{
+ int ret;
+
+ switch (_IOC_NR(cmd)) {
+ case _IOC_NR(MNH_SM_IOC_STOP_MIPI):
+ case _IOC_NR(MNH_SM_IOC_CONFIG_MIPI):
+ case _IOC_NR(MNH_SM_IOC_GET_STATE):
+ case _IOC_NR(MNH_SM_IOC_POST_UPDATE_BUF):
+ case _IOC_NR(MNH_SM_IOC_GET_UPDATE_BUF):
+ cmd &= ~(_IOC_SIZEMASK << _IOC_SIZESHIFT);
+ cmd |= sizeof(void *) << _IOC_SIZESHIFT;
+ ret = mnh_sm_ioctl(file, cmd, (unsigned long)compat_ptr(arg));
+ break;
+ default:
+ ret = mnh_sm_ioctl(file, cmd, arg);
+ break;
+ }
+
+ return ret;
+}
+
+static const struct file_operations mnh_sm_fops = {
+ .open = mnh_sm_open,
+ .unlocked_ioctl = mnh_sm_ioctl,
+ .compat_ioctl = mnh_sm_compat_ioctl,
+ .release = mnh_sm_close
+};
+
+static struct miscdevice mnh_sm_miscdevice = {
+ .minor = MISC_DYNAMIC_MINOR,
+ .name = DEVICE_NAME,
+ .fops = &mnh_sm_fops,
+ .groups = mnh_sm_groups,
+};
+
+static int mnh_sm_probe(struct platform_device *pdev)
+{
+ struct device *dev;
+ struct sched_param param = { .sched_priority = 5 };
+
+ int error = 0;
+ int i;
+
+ pr_debug("%s: MNH SM initializing...\n", __func__);
+
+ /* create char driver */
+ error = misc_register(&mnh_sm_miscdevice);
+ if (error) {
+ pr_err("%s: failed to create char device (%d)\n",
+ __func__, error);
+ return error;
+ }
+ dev = mnh_sm_miscdevice.this_device;
+
+ /* allocate device memory */
+ mnh_sm_dev = devm_kzalloc(dev, sizeof(struct mnh_sm_device),
+ GFP_KERNEL);
+ if (mnh_sm_dev == NULL) {
+ error = -ENOMEM;
+ goto fail_probe_0;
+ }
+
+ /* add device data to platform device */
+ mnh_sm_dev->pdev = pdev;
+ mnh_sm_dev->misc_dev = &mnh_sm_miscdevice;
+ mnh_sm_dev->dev = dev;
+
+ /* initialize kthread work queue */
+ kthread_init_worker(&mnh_sm_dev->worker);
+ kthread_init_work(&mnh_sm_dev->set_state_work, mnh_sm_set_state_work);
+ mnh_sm_dev->thread = kthread_run(kthread_worker_fn,
+ &mnh_sm_dev->worker,
+ "set_state_work");
+ if (IS_ERR(mnh_sm_dev->thread)) {
+ dev_err(dev, "%s: unable to start work thread\n", __func__);
+ mnh_sm_dev->thread = NULL;
+ error = -ENOMEM;
+ goto fail_probe_1;
+ }
+ sched_setscheduler(mnh_sm_dev->thread, SCHED_FIFO, ¶m);
+
+ /* initialize driver structures */
+ mutex_init(&mnh_sm_dev->lock);
+ mutex_init(&mnh_sm_dev->fw_update_lock);
+ init_completion(&mnh_sm_dev->powered_complete);
+ init_completion(&mnh_sm_dev->work_complete);
+ init_completion(&mnh_sm_dev->suspend_complete);
+
+ /* Allocate ion buffer structures */
+ for (i = 0; i < FW_PART_MAX; i++) {
+ mnh_sm_dev->ion[i] = devm_kzalloc(dev,
+ sizeof(struct mnh_ion),
+ GFP_KERNEL);
+ if (mnh_sm_dev->ion[i])
+ mnh_sm_dev->ion[i]->device = dev;
+ }
+
+ /* Allocate primary ION buffer (from carveout region) */
+ if (mnh_sm_dev->ion[FW_PART_PRI]) {
+ if (mnh_ion_create_buffer(mnh_sm_dev->ion[FW_PART_PRI],
+ MNH_ION_BUFFER_SIZE,
+ ION_GOOGLE_HEAP_ID))
+ dev_warn(dev, "%s: cannot claim ION buffer\n",
+ __func__);
+ }
+
+ /* get boot mode gpio */
+ mnh_sm_dev->boot_mode_gpio = devm_gpiod_get(&pdev->dev, "boot-mode",
+ GPIOD_OUT_LOW);
+ if (IS_ERR(mnh_sm_dev->boot_mode_gpio)) {
+ dev_err(dev, "%s: could not get boot_mode gpio (%ld)\n",
+ __func__, PTR_ERR(mnh_sm_dev->boot_mode_gpio));
+ error = PTR_ERR(mnh_sm_dev->boot_mode_gpio);
+ goto fail_probe_2;
+ }
+
+ /* get ready gpio */
+ mnh_sm_dev->ready_gpio = devm_gpiod_get(&pdev->dev, "ready", GPIOD_IN);
+ if (IS_ERR(mnh_sm_dev->ready_gpio)) {
+ dev_err(dev, "%s: could not get ready gpio (%ld)\n",
+ __func__, PTR_ERR(mnh_sm_dev->ready_gpio));
+ error = PTR_ERR(mnh_sm_dev->ready_gpio);
+ goto fail_probe_2;
+ }
+ mnh_sm_dev->ready_irq = gpiod_to_irq(mnh_sm_dev->ready_gpio);
+
+ /* request ready gpio irq */
+ irq_set_status_flags(mnh_sm_dev->ready_irq, IRQ_DISABLE_UNLAZY);
+ error = devm_request_threaded_irq(dev, mnh_sm_dev->ready_irq, NULL,
+ mnh_sm_ready_irq_handler,
+ IRQF_TRIGGER_FALLING |
+ IRQF_TRIGGER_RISING | IRQF_ONESHOT,
+ "mnh-ready", mnh_sm_dev);
+ if (error) {
+ dev_err(dev, "%s: could not get ready irq (%d)\n", __func__,
+ error);
+ goto fail_probe_2;
+ }
+ mnh_sm_enable_ready_irq(false);
+
+ /* request ddr pad isolation pin */
+ mnh_sm_dev->ddr_pad_iso_n_pin = devm_gpiod_get(&pdev->dev,
+ "ddr-pad-iso-n",
+ GPIOD_OUT_HIGH);
+ if (IS_ERR(mnh_sm_dev->ddr_pad_iso_n_pin)) {
+ dev_err(dev, "%s: could not get ddr_pad_iso_n gpio (%ld)\n",
+ __func__, PTR_ERR(mnh_sm_dev->ddr_pad_iso_n_pin));
+ if (PTR_ERR(mnh_sm_dev->ddr_pad_iso_n_pin) == -EPROBE_DEFER)
+ error = -ENODEV;
+ else
+ error = PTR_ERR(mnh_sm_dev->ddr_pad_iso_n_pin);
+ goto fail_probe_2;
+ }
+
+ /* initialize mnh-pwr and get resources there */
+ mnh_sm_enable_ready_irq(true);
+ error = mnh_pwr_init(pdev, dev);
+ mnh_sm_enable_ready_irq(false);
+ if (error) {
+ dev_err(dev, "failed to initialize mnh-pwr (%d)\n", error);
+ goto fail_probe_2;
+ }
+
+ /* initialize mnh-crypto */
+ mnh_crypto_config_sysfs();
+
+ /* initialize mnh-clk driver */
+ error = mnh_clk_init(dev, HWIO_SCU_BASE_ADDR);
+ if (error) {
+ dev_err(dev, "failed to initialize mnh-clk (%d)\n", error);
+ goto fail_probe_2;
+ }
+
+ /* initialize mnh-ddr driver */
+ error = mnh_ddr_platform_init(pdev, &mnh_sm_dev->mnh_ddr_data);
+ if (error) {
+ dev_err(dev, "failed to initialize mnh-ddr (%d)\n", error);
+ goto fail_probe_2;
+ }
+
+ /* register debug fs */
+ error = mnh_sm_dbgfs_register(DEVICE_NAME);
+ if (error) {
+ dev_err(dev, "failed to register debugfs (%d)\n", error);
+ goto fail_probe_2;
+ }
+
+ /* init state stats */
+ mnh_sm_state_stats_init();
+
+ /* initialize device wakeup source */
+ device_init_wakeup(mnh_sm_dev->dev, /*enable=*/true);
+
+ mnh_sm_dev->initialized = true;
+ dev_info(dev, "MNH SM initialized successfully\n");
+
+ return 0;
+
+fail_probe_2:
+ for (i = 0; i < FW_PART_MAX; i++) {
+ mnh_ion_destroy_buffer(mnh_sm_dev->ion[i]);
+ devm_kfree(dev, mnh_sm_dev->ion[i]);
+ mnh_sm_dev->ion[i] = NULL;
+ }
+fail_probe_1:
+ devm_kfree(dev, mnh_sm_dev);
+ mnh_sm_dev = NULL;
+fail_probe_0:
+ misc_deregister(&mnh_sm_miscdevice);
+
+ return error;
+}
+
+static const struct of_device_id mnh_sm_ids[] = {
+ { .compatible = "intel,mnh-sm", },
+ { },
+};
+
+MODULE_DEVICE_TABLE(of, esm_dt_ids);
+
+#ifdef CONFIG_SUSPEND
+
+static int mnh_sm_pm_ops_suspend(struct device *dev)
+{
+ dev_dbg(dev, "%s: enter\n", __func__);
+ if (mnh_sm_get_state() == MNH_STATE_ACTIVE) {
+ dev_info(dev, "system about to suspend; power off now\n");
+ mnh_sm_pwr_error_cb();
+ }
+ return 0;
+}
+
+static int mnh_sm_pm_ops_resume(struct device *dev)
+{
+ dev_dbg(dev, "%s: enter\n", __func__);
+ return 0;
+}
+
+#else /* CONFIG_SUSPEND */
+
+#define mnh_sm_pm_ops_suspend NULL
+#define mnh_sm_pm_ops_resume NULL
+
+#endif /* CONFIG_SUSPEND */
+
+static const struct dev_pm_ops mnh_sm_dev_pm_ops = {
+ .suspend = mnh_sm_pm_ops_suspend,
+ .resume = mnh_sm_pm_ops_resume,
+};
+
+static struct platform_driver mnh_sm = {
+ .probe = mnh_sm_probe,
+ .driver = {
+ .name = DEVICE_NAME,
+ .of_match_table = mnh_sm_ids,
+ .probe_type = PROBE_PREFER_ASYNCHRONOUS,
+#ifdef CONFIG_PM
+ .pm = &mnh_sm_dev_pm_ops,
+#endif /* CONFIG_PM */
+ },
+};
+
+/* Make sure mnh_sm driver is initialized after s2mpb04-gpio */
+static int __init mnh_sm_init(void)
+{
+ return platform_driver_register(&mnh_sm);
+}
+late_initcall(mnh_sm_init);
+
+MODULE_AUTHOR("Intel Corporation");
+MODULE_DESCRIPTION("MNH State Manager HOST DRIVER");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/misc/mnh/mnh-sm.h b/drivers/misc/mnh/mnh-sm.h
new file mode 100644
index 0000000..d96b25b
--- /dev/null
+++ b/drivers/misc/mnh/mnh-sm.h
@@ -0,0 +1,139 @@
+/*
+ *
+ * MNH State Manager HOST Driver
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef __MNH_SM_HOST
+#define __MNH_SM_HOST
+
+#include <linux/msm_ion.h>
+#include <uapi/linux/mnh-sm.h>
+#include "mnh-crypto.h"
+#include "mnh-mipi.h"
+#include "mnh-pcie.h"
+
+/* Firmware download address */
+#define HW_MNH_DRAM_BASE 0x40000000
+#define HW_MNH_SBL_DOWNLOAD 0x40000000
+#define HW_MNH_SBL_DOWNLOAD_EXE 0x00101000
+#define HW_MNH_KERNEL_DOWNLOAD 0x40080000
+#define HW_MNH_DT_DOWNLOAD 0x40880000
+#define HW_MNH_RAMDISK_DOWNLOAD 0x40890000
+/* Firmware download related definitions */
+#define IMG_DOWNLOAD_MAX_SIZE (4000 * 1024)
+#define FIP_IMG_SBL_SIZE_OFFSET 0x28
+#define FIP_IMG_SBL_ADDR_OFFSET 0x20
+
+enum mnh_boot_mode {
+ MNH_BOOT_MODE_PCIE,
+ MNH_BOOT_MODE_SPI,
+};
+
+struct mnh_ion_fw_conf {
+ dma_addr_t ap_addr; /* AP side addr (dma) */
+ unsigned long ap_offs; /* Slot's offset in the ion buffer */
+ uint64_t ep_addr; /* EP side addr (phys) */
+ size_t size; /* size of firmware */
+ enum cert_state cert; /* signature status */
+ size_t cert_size; /* length of the signature */
+};
+
+struct mnh_ion {
+ struct ion_client *client;
+ struct ion_handle *handle;
+ struct sg_table *sg;
+ void *vaddr;
+ size_t total_size; /* size in bytes */
+ bool is_fw_ready; /* whether fw loaded in ION */
+ struct mnh_ion_fw_conf fw_array[MAX_NR_MNH_FW_SLOTS];
+ struct device *device; /* Linux basic device associated with parent */
+};
+
+/** API to register hotplug callback to receive MNH up/down notifications
+ * @param[in] hotplug_cb handler for hotplug in/out events
+ * @return 0
+ */
+int mnh_sm_reg_hotplug_callback(hotplug_cb_t hotplug_cb);
+
+/**
+ * API to obtain the state of monette hill.
+ * @return the power states of mnh.
+ */
+int mnh_sm_get_state(void);
+
+/**
+ * API to set the state of monette hill.
+ * @param[in] Set the power states of mnh
+ */
+int mnh_sm_set_state(int state);
+
+int mnh_sm_is_present(void);
+
+/**
+ * API to claim ION buffer for mnh.
+ * @param[in] dev Linux device structure assocatiated with mnh_sm
+ * @param[in] ion Pre-allocated mnh_ion struct
+ * @param[in] size Size of ION buffer to claim (in bytes)
+ * @param[in] heap_id Heap ID to be used for the allocation
+ *
+ * @return 0 on success
+ * @return -negative on failure
+ */
+long mnh_ion_create_buffer(struct mnh_ion *ion, size_t size,
+ enum ion_heap_ids heap_id);
+
+/**
+ * API to release previously claimed ION buffer.
+ * @param[in] ion Pre-allocated mnh_ion struct
+ * Caller is responsible for freeing ion.
+ */
+void mnh_ion_destroy_buffer(struct mnh_ion *ion);
+
+/**
+ * API to request MNH firmware and stage them to ION buffer.
+ * @param[in] ion Pre-allocated mnh_ion struct
+ * Caller is responsible for freeing ion.
+ *
+ * @return 0 on success
+ * @return -negative on failure
+ */
+int mnh_ion_stage_firmware(struct mnh_ion *ion);
+
+/**
+ * API to stage Play Store firmware update to ION buffer.
+ * @param[in] ion Pre-allocated mnh_ion struct
+ * Caller is responsible for freeing ion.
+ * @param[in] ion_sec Temporary allocated buffer holding
+ * firmware updated from Play Store channel
+ *
+ * @return 0 on success
+ * @return -negative on failure
+ */
+int mnh_ion_stage_firmware_update(struct mnh_ion *ion,
+ struct mnh_ion *ion_sec);
+
+/*
+ * Callback from mnh-pwr when there is a failure event.
+ *
+ * @return 0 on success
+ * @return -errno on failure
+ */
+int mnh_sm_pwr_error_cb(void);
+
+int mnh_sm_mipi_config(struct mnh_mipi_config config);
+
+enum mnh_boot_mode mnh_sm_get_boot_mode(void);
+
+#endif /* __MNH_SM_HOST */
+
diff --git a/drivers/misc/mnh/mnh-thermal.c b/drivers/misc/mnh/mnh-thermal.c
new file mode 100644
index 0000000..2fb75e9
--- /dev/null
+++ b/drivers/misc/mnh/mnh-thermal.c
@@ -0,0 +1,456 @@
+/*
+*
+* MNH Thermal Driver
+* Copyright (c) 2016, Intel Corporation.
+*
+* This program is free software; you can redistribute it and/or modify it
+* under the terms and conditions of the GNU General Public License,
+* version 2, as published by the Free Software Foundation.
+*
+* This program is distributed in the hope it will be useful, but WITHOUT
+* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+* FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+* more details.
+*
+*/
+
+/* #define DEBUG */
+
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/err.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/reset.h>
+#include <linux/thermal.h>
+#include <linux/timer.h>
+
+#include "mnh-efuse.h"
+#include "mnh-hwio.h"
+#include "mnh-hwio-bases.h"
+#include "mnh-hwio-scu.h"
+#include "mnh-sm.h"
+#include "mnh-thermal.h"
+
+#define DEVICE_NAME "mnh_thermal"
+
+/* PVT operation mode configuration */
+static struct mnh_thermal_op_param mnh_op_mode_table[] = {
+ {0, 0}, /* temperature: vsample b0, psample b00 */
+ {1, 0}, /* voltage: vsample b1, psample b00 */
+ {0, 1}, /* LVT: vsample b0, psample b01 */
+ {0, 2}, /* HVT: vsample b0, psample b10 */
+ {0, 3} /* SVT: vsample b0, psample b11 */
+};
+
+/* PVT dts trim efuse addresses */
+static const struct mnh_efuse_addr mnh_thermal_trim_addr[] = {
+ /* id0, ipu1 */
+ {
+ .row1 = 2,
+ .row2 = 3,
+ .bit_high = 9,
+ .bit_low = 0,
+ },
+ /* id1, ipu2 */
+ {
+ .row1 = 2,
+ .row2 = 3,
+ .bit_high = 19,
+ .bit_low = 10,
+ },
+ /* id2, cpu */
+ {
+ .row1 = 2,
+ .row2 = 3,
+ .bit_high = 29,
+ .bit_low = 20,
+ },
+ /* id3, lpddr */
+ {
+ .row1 = 0,
+ .row2 = 1,
+ .bit_high = 9,
+ .bit_low = 0,
+ }
+};
+
+/**
+ * This checks mnh sm state to see mnh is out of reset
+ * Return: 0 if mnh is out of reset, an error code otherwise.
+ */
+static int check_mnh_hw_init(void)
+{
+ int state = mnh_sm_get_state();
+
+ if ((state == MNH_STATE_OFF) || (state == MNH_STATE_SUSPEND))
+ return -EIO;
+
+ return 0;
+}
+
+static void config_dts_trim(struct mnh_thermal_device *mnh_dev)
+{
+ uint32_t i, value;
+ uint32_t slope, offset;
+
+ for (i = 0; i < MNH_NUM_PVT_SENSORS; i++) {
+ value = mnh_efuse_read(&mnh_thermal_trim_addr[i]);
+
+ /* Make sure code is 10bit */
+ value = value & 0x3ff;
+
+ /* Calculate slope and offset */
+ slope = ((unsigned)value >> API_BITS_OFFSET) &
+ API_BITS_SLOPE_MASK;
+ if (slope >= 16)
+ slope = slope - 32;
+
+ offset = value & API_BITS_OFFSET_MASK;
+ if (offset >= 16)
+ offset = offset - 32;
+
+ dev_dbg(mnh_dev->dev, "%s: pvt%d efuse:0x%x, s:%d, o:%d\n",
+ __func__, i, value, slope, offset);
+
+ mnh_dev->sensors[i]->slope = slope;
+ mnh_dev->sensors[i]->offset = offset;
+ }
+}
+
+/**
+ * Configure 1.2MHz clock to PVT sensor and wait for 2usecs
+ * until the clock is ticking if enabled
+ */
+static void config_pvt_clk_en(struct mnh_thermal_device *mnh_dev, bool clk_en)
+{
+ dev_dbg(mnh_dev->dev, "%s: enable %d\n", __func__, clk_en);
+
+ HW_OUTf(mnh_dev->regs, SCU, PERIPH_CLK_CTRL, PVT_CLKEN, clk_en);
+ udelay(2);
+}
+
+/**
+ * Check pvt clock is enabled
+ * Return: 0 if pvt clk is off, 1 if pvt clk is on
+ */
+static uint32_t read_pvt_clk_en(struct mnh_thermal_device *mnh_dev)
+{
+ uint32_t val = HW_INf(mnh_dev->regs, SCU, PERIPH_CLK_CTRL, PVT_CLKEN);
+ return val;
+}
+
+/*
+ * Caculate PVT_DATA output to millicelsius.
+ *
+ * Step #1 : Calculate Code to DegC using ideal equation.
+ * DegC= API_Poly_N4 * Code^4 + API_Poly_N3 * Code^3 + API_Poly_N2 * Code^2
+ * + API_Poly_N1 * Code^1 + API_Poly_N0
+ * Step #2 : Calculate Error using slope and offset.
+ * Error= Slope * DegC + Offset
+ * Step #3 : Calculate Final_temp by removing the above error from DegC
+ * Final_Temp = DegC - Error
+ *
+ * Note: kernel doesn't support floaing point calculation, so convert all the
+ * predefined floating number to 64bit number. All variables and predefined
+ * constants with an underscore "_" carry an implicit E-15 factor.
+ */
+static int calculate_temp(struct mnh_thermal_device *mnh_dev, int code,
+ int slope, int offset)
+{
+ long n1_, n2_, n3_, n4_, n0_, n01234_, final_temp_, error_;
+ int temp_md;
+
+ dev_dbg(mnh_dev->dev, "%s: pvt data:%d, slope:%d, offset:%d\n",
+ __func__, code, slope, offset);
+
+ /* Make sure code is 10bit */
+ code = code & 0x3FF;
+
+ /* Calculate degree Celcius value
+ * n4: 55bits
+ * n3: 57bits
+ * n2: 55bits
+ * n1: 56bits
+ * n0: 59bits
+ */
+ n4_ = (long)code*code*code*code * API_POLY_N4;
+ n3_ = (long)code*code*code * API_POLY_N3 * N3_E15_MULTIPLIER;
+ n2_ = (long)code*code * API_POLY_N2 * N2_E15_MULTIPLIER;
+ n1_ = (long)code * API_POLY_N1 * N1_E15_MULTIPLIER;
+ n0_ = API_POLY_N0 * N0_E15_MULTIPLIER;
+ n01234_ = n4_ + n3_ + n2_ + n1_ + n0_;
+
+ /* slope = trim_slope * API_RES_SLOPE
+ * offset = trim_offset * API_RES_OFFSET
+ * error = slope * degC + offset
+ * n01234_ is 59bit, max slope is 5bit, which may overflow
+ * when multiplied without first down-scaling.
+ */
+ error_ = n01234_ / RES_SLOPE_DIVIDER * slope * API_RES_SLOPE
+ + (long)offset * RES_OFFSET_E15_MULTIPLIER * API_RES_OFFSET;
+
+ final_temp_ = n01234_ - error_;
+ temp_md = (final_temp_ + N12_ROUNDING_NUM) / N12_DIVIDER;
+
+ dev_dbg(mnh_dev->dev, "%s: deg:%ld, err:%ld, temp:%ld\n",
+ __func__, n01234_, error_, final_temp_);
+
+ return temp_md;
+}
+
+/* Read raw temperature data from sensors
+ * Enable 1.2MHz Clock to PVT Sensor
+ * Wait for 2 msecs for the clock to PVT sensor to start ticking
+ * Program VSAMPLE/PSAMPLE
+ * Enable PVT sensor access
+ * Wait for conversion cycle time or PVT_DATA.DATAVALID or
+ * wait for SCU interrupt
+ * Read PVT_DATA[x].DATA_OUT[9:0]
+ * Disable PVT sensor access
+ * Reset PVT_DATA.DATAVALID
+ * Disable 1.2MHz clock to PVT sensor
+*/
+static int mnh_thermal_get_data(void *data, int *data_out)
+{
+ struct mnh_thermal_sensor *sensor = (struct mnh_thermal_sensor *)data;
+ struct mnh_thermal_device *mnh_dev = sensor->mnh_dev;
+ u32 val = 0;
+ unsigned long timeout = jiffies + msecs_to_jiffies(500);
+ u32 op_mode, psample, vsample;
+ uint32_t mnh_scu_base = mnh_dev->regs;
+ uint32_t id = sensor->id;
+
+ op_mode = THERMAL_OPMODE_TEMP;
+ psample = mnh_op_mode_table[op_mode].psample;
+ vsample = mnh_op_mode_table[op_mode].vsample;
+ dev_dbg(mnh_dev->dev, "%s: pvt op:%d, vs:%d, ps:%d\n",
+ __func__, op_mode, vsample, psample);
+
+ /* Enable PVT clock */
+ if (!read_pvt_clk_en(mnh_dev))
+ config_pvt_clk_en(mnh_dev, 1);
+
+ /* Program VSAMPLE/PSAMPLE for temperature evaulation */
+ HW_OUTxf(mnh_scu_base, SCU, PVT_CONTROL, id, PSAMPLE, psample);
+ HW_OUTxf(mnh_scu_base, SCU, PVT_CONTROL, id, VSAMPLE, vsample);
+
+ /* Enable PVT sensor access */
+ HW_OUTxf(mnh_scu_base, SCU, PVT_CONTROL, id, ENA, 1);
+
+ /* Wait until PVT data conversion is finished in
+ * temperature and voltage mode.
+ * Conversion time: 376 PVT_SENSOR_CLK cycles
+ */
+ do {
+ val = HW_INxf(mnh_scu_base, SCU, PVT_DATA, id, DATAVALID);
+ if (val == 1)
+ break;
+ udelay(mnh_dev->wait_time_us);
+ } while (time_before(jiffies, timeout));
+
+ /* Read DATA_OUT from sensor */
+ val = HW_INxf(mnh_scu_base, SCU, PVT_DATA, id, DATA);
+
+ /* Disable PVT sensor access */
+ HW_OUTxf(mnh_scu_base, SCU, PVT_CONTROL, id, ENA, 0);
+
+ /* Reset DATAVALID bit */
+ HW_OUTxf(mnh_scu_base, SCU, PVT_DATA, id, DATAVALID, 1);
+
+ *data_out = val;
+
+ return 0;
+}
+
+/*
+ * This will read raw_temp_code and convert to DecC temperature.
+ */
+static int mnh_thermal_get_temp(void *data, int *temp_out)
+{
+ struct mnh_thermal_sensor *sensor = (struct mnh_thermal_sensor *)data;
+ struct mnh_thermal_device *mnh_dev = sensor->mnh_dev;
+ int temp_raw = 0;
+ uint32_t id = sensor->id;
+
+ if (check_mnh_hw_init())
+ return -EIO;
+
+ /* read dts trim data if it is not configured yet */
+ if (!mnh_dev->mnh_trim_init_done) {
+ config_dts_trim(mnh_dev);
+ mnh_dev->mnh_trim_init_done = 1;
+ }
+
+ /* Get raw temp code */
+ mnh_thermal_get_data(data, &temp_raw);
+
+ /* Convert raw code to milli DecC */
+ *temp_out = calculate_temp(mnh_dev, temp_raw,
+ mnh_dev->sensors[id]->slope, mnh_dev->sensors[id]->offset);
+
+ return 0;
+}
+
+static int mnh_get_cur_state(struct thermal_cooling_device *cooling_dev,
+ unsigned long *state)
+{
+ struct mnh_thermal_device *mnh_dev = cooling_dev->devdata;
+
+ if (!mnh_dev)
+ return -EINVAL;
+
+ *state = mnh_dev->throttle_state;
+
+ return 0;
+}
+
+static int mnh_get_max_state(struct thermal_cooling_device *cooling_dev,
+ unsigned long *state)
+{
+ struct mnh_thermal_device *mnh_dev = cooling_dev->devdata;
+
+ if (!mnh_dev)
+ return -EINVAL;
+
+ *state = MNH_MAX_THROTTLE_STATE;
+
+ return 0;
+}
+
+static int mnh_set_cur_state(struct thermal_cooling_device *cooling_dev,
+ unsigned long state)
+{
+ struct mnh_thermal_device *mnh_dev = cooling_dev->devdata;
+
+ if (!mnh_dev)
+ return -EINVAL;
+
+ if (state > MNH_MAX_THROTTLE_STATE)
+ state = MNH_MAX_THROTTLE_STATE;
+
+ mnh_dev->throttle_state = state;
+
+ return 0;
+}
+
+static const struct thermal_zone_of_device_ops mnh_of_thermal_ops = {
+ .get_temp = mnh_thermal_get_temp,
+};
+
+static const struct thermal_cooling_device_ops mnh_cooling_ops = {
+ .get_max_state = mnh_get_max_state,
+ .get_cur_state = mnh_get_cur_state,
+ .set_cur_state = mnh_set_cur_state,
+};
+
+static int mnh_thermal_probe(struct platform_device *pdev)
+{
+ struct mnh_thermal_device *mnh_dev;
+ unsigned int i;
+ int err;
+
+ dev_dbg(&pdev->dev, "%s: enter\n", __func__);
+
+ mnh_dev = devm_kzalloc(&pdev->dev, sizeof(*mnh_dev),
+ GFP_KERNEL);
+ if (!mnh_dev)
+ return -ENOMEM;
+
+ mnh_dev->pdev = pdev;
+ mnh_dev->dev = &pdev->dev;
+ mnh_dev->regs = HWIO_SCU_BASE_ADDR;
+ mnh_dev->wait_time_us = PVT_WAIT_US(MNH_PRECISION);
+ platform_set_drvdata(pdev, mnh_dev);
+
+ /* Register cooling device */
+ mnh_dev->cooling_dev = thermal_of_cooling_device_register(
+ dev_of_node(mnh_dev->dev), "mnh", mnh_dev,
+ &mnh_cooling_ops);
+
+ if (IS_ERR(mnh_dev->cooling_dev)) {
+ err = PTR_ERR(mnh_dev->cooling_dev);
+ dev_err(mnh_dev->dev, "%s: failed to register cooling device: %d\n",
+ __func__, err);
+ return err;
+ }
+
+ /* Initialize thermctl sensors */
+ for (i = 0; i < ARRAY_SIZE(mnh_dev->sensors); ++i) {
+ struct mnh_thermal_sensor *sensor =
+ devm_kzalloc(&pdev->dev, sizeof(*sensor),
+ GFP_KERNEL);
+
+ sensor->mnh_dev = mnh_dev;
+ sensor->id = i;
+ sensor->tzd = thermal_zone_of_sensor_register(&pdev->dev, i,
+ sensor, &mnh_of_thermal_ops);
+
+ if (IS_ERR(sensor->tzd)) {
+ err = PTR_ERR(sensor->tzd);
+ dev_err(mnh_dev->dev, "%s: failed to register sensor: %d\n",
+ __func__, err);
+ goto unregister_sensors;
+ }
+
+ mnh_dev->sensors[i] = sensor;
+ }
+
+ dev_info(&pdev->dev, "%s: initialized\n", __func__);
+
+ return 0;
+
+unregister_sensors:
+ while (i--)
+ thermal_zone_of_sensor_unregister(&pdev->dev,
+ mnh_dev->sensors[i]->tzd);
+
+ thermal_cooling_device_unregister(mnh_dev->cooling_dev);
+
+ return err;
+}
+
+static int mnh_thermal_remove(struct platform_device *pdev)
+{
+ struct mnh_thermal_device *mnh_dev = platform_get_drvdata(pdev);
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(mnh_dev->sensors); ++i) {
+ thermal_zone_of_sensor_unregister(&pdev->dev,
+ mnh_dev->sensors[i]->tzd);
+ }
+
+ thermal_cooling_device_unregister(mnh_dev->cooling_dev);
+
+ return 0;
+}
+
+/*
+ * of_device_id structure
+ */
+static const struct of_device_id mnh_thermal_of_match[] = {
+ { .compatible = "intel,mnh_thermal" },
+ { }
+};
+
+MODULE_DEVICE_TABLE(of, mnh_thermal_of_match);
+
+/*
+ * Platform driver structure
+ */
+static struct platform_driver mnh_thermal_driver = {
+ .probe = mnh_thermal_probe,
+ .remove = mnh_thermal_remove,
+ .driver = {
+ .name = DEVICE_NAME,
+ .owner = THIS_MODULE,
+ .of_match_table = mnh_thermal_of_match,
+ },
+};
+module_platform_driver(mnh_thermal_driver);
+
+MODULE_DESCRIPTION("MonetteHill Thermal Driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/misc/mnh/mnh-thermal.h b/drivers/misc/mnh/mnh-thermal.h
new file mode 100644
index 0000000..6b161bf
--- /dev/null
+++ b/drivers/misc/mnh/mnh-thermal.h
@@ -0,0 +1,107 @@
+#ifndef __MNH_THERMAL_H__
+#define __MNH_THERMAL_H__
+
+#include <linux/types.h>
+#include <linux/delay.h>
+#include <linux/device.h>
+
+#define MNH_NUM_PVT_SENSORS 4
+#define MNH_MAX_THROTTLE_STATE 4
+
+/* Thermal characterization data */
+#define API_TRIM_CODE 0xF
+#define MNH_PRECISION 0x1
+
+/* Thermal efuse data */
+#define NUM_BITS_EFUSE 10
+#define API_BITS_SLOPE 5
+#define API_BITS_OFFSET (NUM_BITS_EFUSE-API_BITS_SLOPE)
+
+/* Temperature calculation algorithm parameters */
+#define API_POLY_N4 (-16743) /* -1.6743e-11 : 15bits excluding sign */
+#define API_POLY_N3 (+81542) /* +8.1542e-08 : 17bits excluding sign */
+#define API_POLY_N2 (-18201) /* -1.8201e-04 : 15bits excluding sign */
+#define API_POLY_N1 (+31020) /* +3.1020e-01 : 15bits excluding sign */
+#define API_POLY_N0 (-48380) /* -4.8380e+01 : 16bits excluding sign */
+
+#define N3_E15_MULTIPLIER 1000 /* 1e3*/
+#define N2_E15_MULTIPLIER 10000000 /* 1e7 */
+#define N1_E15_MULTIPLIER 10000000000 /* 1e10 */
+#define N0_E15_MULTIPLIER 1000000000000 /* 1e12 */
+
+#define API_RES_SLOPE 78125 /* 0.0078125 */
+#define API_RES_OFFSET 5 /* 0.5 */
+
+#define RES_SLOPE_DIVIDER 10000000 /* 1e7 */
+#define RES_OFFSET_E15_MULTIPLIER 100000000000000 /* 1e14 */
+
+#define API_BITS_SLOPE_MASK 0x1F
+#define API_BITS_OFFSET_MASK 0x1F
+
+#define N12_DIVIDER 1000000000000 /* 1e12 */
+#define N12_ROUNDING_NUM 500000000000 /* 5e11 */
+
+/* PVT Data conversion wait time calculation
+ * PVT_FREQ: 1.15MHZ
+ * PVT_WAIT_CYCLE: 376 + marginal cycles
+ * PVT_WAIT_TIME: ((1/PVT_FREQ)*PVT_WAIT_CYCLE*PRECISION)
+ * Average samples: (1<<(PRECISION*2))
+ * PVT data conversion time:
+ * temp/vol mode: 376 PVT_SENSOR_CLK cycles
+ * process mode: 4 PVT_SENSOR_CLK cycles
+ */
+#define PVT_FREQ_KHZ 1150
+#define PVT_FREQ_MS_UNIT 1000000
+#define PVT_FREQ_US_UNIT 1000
+#define PVT_CYCLE (PVT_FREQ_MS_UNIT/PVT_FREQ_KHZ)
+#define PVT_WAIT_CYCLE 390
+#define PVT_AVG_SAMPLES(p) (1<<(p*2))
+#define PVT_WAIT_MS(p) (((PVT_CYCLE*PVT_WAIT_CYCLE*PVT_AVG_SAMPLES(p)) \
+ /PVT_FREQ_MS_UNIT)+1)
+#define PVT_WAIT_US(p) ((PVT_CYCLE*PVT_WAIT_CYCLE*PVT_AVG_SAMPLES(p)) \
+ /PVT_FREQ_US_UNIT)
+
+struct mnh_thermal_device {
+ struct platform_device *pdev;
+ struct device *dev;
+ struct reset_control *reset;
+ uint32_t regs;
+ uint32_t wait_time_us;
+ uint32_t mnh_trim_init_done;
+ uint32_t throttle_state;
+ struct mnh_thermal_sensor *sensors[MNH_NUM_PVT_SENSORS];
+ struct thermal_cooling_device *cooling_dev;
+};
+
+struct mnh_thermal_sensor {
+ struct mnh_thermal_device *mnh_dev;
+ struct thermal_zone_device *tzd;
+ uint32_t id;
+ uint32_t slope;
+ uint32_t offset;
+ uint32_t alarm_temp;
+};
+
+/* PVT Sensor operation mode */
+enum mnh_thermal_op_mode {
+ THERMAL_OPMODE_TEMP, /* Temperature evaluation */
+ THERMAL_OPMODE_VOL, /* Voltage evaluation */
+ THERMAL_OPMODE_LVT, /* Process LVT evaluation */
+ THERMAL_OPMODE_HVT, /* Process HVT evaluation */
+ THERMAL_OPMODE_SVT, /* Process SVT evaluation */
+ THERMAL_OPMODE_MAX = THERMAL_OPMODE_SVT
+};
+
+struct mnh_thermal_op_param {
+ int vsample;
+ int psample;
+};
+
+struct mnh_thermal_efuse_param {
+ uint8_t row1;
+ uint8_t row2;
+ uint8_t bh;
+ uint8_t bl;
+};
+
+#endif /* __MNH_THERMAL_H__ */
diff --git a/drivers/misc/qseecom.c b/drivers/misc/qseecom.c
index 9d6069b..d671454 100644
--- a/drivers/misc/qseecom.c
+++ b/drivers/misc/qseecom.c
@@ -1,7 +1,7 @@
/*
* QTI Secure Execution Environment Communicator (QSEECOM) driver
*
- * Copyright (c) 2012-2018, The Linux Foundation. All rights reserved.
+ * Copyright (c) 2012-2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -6967,31 +6967,6 @@ static inline long qseecom_ioctl(struct file *file,
pr_err("failed qseecom_register_listener: %d\n", ret);
break;
}
- case QSEECOM_IOCTL_SET_ICE_INFO: {
- struct qseecom_ice_data_t ice_data;
-
- ret = copy_from_user(&ice_data, argp, sizeof(ice_data));
- if (ret) {
- pr_err("copy_from_user failed\n");
- return -EFAULT;
- }
- qcom_ice_set_fde_flag(ice_data.flag);
- break;
- }
-
- case QSEECOM_IOCTL_SET_ENCDEC_INFO: {
- struct qseecom_encdec_conf_t conf;
-
- ret = copy_from_user(&conf, argp, sizeof(conf));
- if (ret) {
- pr_err("copy_from_user failed\n");
- return -EFAULT;
- }
- ret = qcom_ice_set_fde_conf(conf.start_sector, conf.fs_size,
- conf.index, conf.mode);
- break;
- }
-
case QSEECOM_IOCTL_UNREGISTER_LISTENER_REQ: {
if ((data->listener.id == 0) ||
(data->type != QSEECOM_LISTENER_SERVICE)) {
diff --git a/drivers/misc/sgi-gru/grukdump.c b/drivers/misc/sgi-gru/grukdump.c
index 313da31..1540a77 100644
--- a/drivers/misc/sgi-gru/grukdump.c
+++ b/drivers/misc/sgi-gru/grukdump.c
@@ -27,6 +27,9 @@
#include <linux/delay.h>
#include <linux/bitops.h>
#include <asm/uv/uv_hub.h>
+
+#include <linux/nospec.h>
+
#include "gru.h"
#include "grutables.h"
#include "gruhandles.h"
@@ -196,6 +199,7 @@ int gru_dump_chiplet_request(unsigned long arg)
/* Currently, only dump by gid is implemented */
if (req.gid >= gru_max_gids)
return -EINVAL;
+ req.gid = array_index_nospec(req.gid, gru_max_gids);
gru = GID_TO_GRU(req.gid);
ubuf = req.buf;
diff --git a/drivers/misc/st21nfc.c b/drivers/misc/st21nfc.c
new file mode 100644
index 0000000..54fac42
--- /dev/null
+++ b/drivers/misc/st21nfc.c
@@ -0,0 +1,756 @@
+/*
+ * Copyright (C) 2016 ST Microelectronics S.A.
+ * Copyright (C) 2010 Stollmann E+V GmbH
+ * Copyright (C) 2010 Trusted Logic S.A.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/fs.h>
+#include <linux/version.h>
+#include <linux/slab.h>
+#include <linux/init.h>
+#include <linux/list.h>
+#include <linux/i2c.h>
+#include <linux/irq.h>
+#include <linux/jiffies.h>
+#include <linux/uaccess.h>
+#include <linux/delay.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/platform_device.h>
+#include <linux/gpio.h>
+#include <linux/poll.h>
+#include <linux/miscdevice.h>
+#include <linux/spinlock.h>
+#include "st21nfc.h"
+#include <linux/of_gpio.h>
+
+
+#define MAX_BUFFER_SIZE 260
+
+#define DRIVER_VERSION "2.0.0"
+
+/* define the active state of the WAKEUP pin */
+#define ST21_IRQ_ACTIVE_HIGH 1
+#define ST21_IRQ_ACTIVE_LOW 0
+
+/* prototypes */
+static irqreturn_t st21nfc_dev_irq_handler(int irq, void *dev_id);
+/*
+ * The platform data member 'polarity_mode' defines
+ * how the wakeup pin is configured and handled.
+ * it can take the following values :
+ * IRQF_TRIGGER_RISING
+ * IRQF_TRIGGER_FALLING
+ * IRQF_TRIGGER_HIGH
+ * IRQF_TRIGGER_LOW
+ */
+
+struct st21nfc_platform {
+ struct mutex read_mutex;
+ struct i2c_client *client;
+ unsigned int irq_gpio;
+ unsigned int reset_gpio;
+ unsigned int ena_gpio;
+ unsigned int polarity_mode;
+ /* either 0 (low-active) or 1 (high-active) */
+ unsigned int active_polarity;
+};
+
+static bool irqIsAttached;
+static bool device_open; /* Is device open? */
+
+struct st21nfc_dev {
+ wait_queue_head_t read_wq;
+ struct miscdevice st21nfc_device;
+ bool irq_enabled;
+ struct st21nfc_platform platform_data;
+ spinlock_t irq_enabled_lock;
+};
+
+static int st21nfc_loc_set_polaritymode(struct st21nfc_dev *st21nfc_dev,
+ int mode)
+{
+
+ struct i2c_client *client = st21nfc_dev->platform_data.client;
+ unsigned int irq_type;
+ int ret;
+
+ st21nfc_dev->platform_data.polarity_mode = mode;
+ /* setup irq_flags */
+ switch (mode) {
+ case IRQF_TRIGGER_RISING:
+ irq_type = IRQ_TYPE_EDGE_RISING;
+ st21nfc_dev->platform_data.active_polarity = 1;
+ break;
+ case IRQF_TRIGGER_FALLING:
+ irq_type = IRQ_TYPE_EDGE_FALLING;
+ st21nfc_dev->platform_data.active_polarity = 0;
+ break;
+ case IRQF_TRIGGER_HIGH:
+ irq_type = IRQ_TYPE_LEVEL_HIGH;
+ st21nfc_dev->platform_data.active_polarity = 1;
+ break;
+ case IRQF_TRIGGER_LOW:
+ irq_type = IRQ_TYPE_LEVEL_LOW;
+ st21nfc_dev->platform_data.active_polarity = 0;
+ break;
+ default:
+ irq_type = IRQF_TRIGGER_FALLING;
+ st21nfc_dev->platform_data.active_polarity = 0;
+ break;
+ }
+ if (irqIsAttached) {
+ free_irq(client->irq, st21nfc_dev);
+ irqIsAttached = false;
+ }
+ ret = irq_set_irq_type(client->irq, irq_type);
+ if (ret) {
+ pr_err("%s : set_irq_type failed\n", __FILE__);
+ return -ENODEV;
+ }
+ /* request irq. the irq is set whenever the chip has data available
+ * for reading. it is cleared when all data has been read.
+ */
+ pr_debug("%s : requesting IRQ %d\n", __func__, client->irq);
+ st21nfc_dev->irq_enabled = true;
+
+ ret = request_irq(client->irq, st21nfc_dev_irq_handler,
+ st21nfc_dev->platform_data.polarity_mode,
+ client->name, st21nfc_dev);
+ if (!ret)
+ irqIsAttached = true;
+
+ return ret;
+}
+
+static void st21nfc_disable_irq(struct st21nfc_dev *st21nfc_dev)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&st21nfc_dev->irq_enabled_lock, flags);
+ if (st21nfc_dev->irq_enabled) {
+ disable_irq_nosync(st21nfc_dev->platform_data.client->irq);
+ st21nfc_dev->irq_enabled = false;
+ }
+ spin_unlock_irqrestore(&st21nfc_dev->irq_enabled_lock, flags);
+}
+
+static irqreturn_t st21nfc_dev_irq_handler(int irq, void *dev_id)
+{
+ struct st21nfc_dev *st21nfc_dev = dev_id;
+
+ st21nfc_disable_irq(st21nfc_dev);
+
+ /* Wake up waiting readers */
+ wake_up(&st21nfc_dev->read_wq);
+
+ return IRQ_HANDLED;
+}
+
+static ssize_t st21nfc_dev_read(struct file *filp, char __user *buf,
+ size_t count, loff_t *offset)
+{
+ struct st21nfc_dev *st21nfc_dev = container_of(filp->private_data,
+ struct st21nfc_dev,
+ st21nfc_device);
+ char tmp[MAX_BUFFER_SIZE];
+ int ret;
+
+ if (count > MAX_BUFFER_SIZE)
+ count = MAX_BUFFER_SIZE;
+
+ pr_debug("%s : reading %zu bytes.\n", __func__, count);
+
+ mutex_lock(&st21nfc_dev->platform_data.read_mutex);
+
+ /* Read data */
+ ret = i2c_master_recv(st21nfc_dev->platform_data.client, tmp, count);
+ mutex_unlock(&st21nfc_dev->platform_data.read_mutex);
+
+ if (ret < 0) {
+ pr_err("%s: i2c_master_recv returned %d\n", __func__, ret);
+ return ret;
+ }
+ if (ret > count) {
+ pr_err("%s: received too many bytes from i2c (%d)\n",
+ __func__, ret);
+ return -EIO;
+ }
+ if (copy_to_user(buf, tmp, ret)) {
+ pr_warn("%s : failed to copy to user space\n", __func__);
+ return -EFAULT;
+ }
+ return ret;
+}
+
+static ssize_t st21nfc_dev_write(struct file *filp, const char __user *buf,
+ size_t count, loff_t *offset)
+{
+ struct st21nfc_dev *st21nfc_dev;
+ char tmp[MAX_BUFFER_SIZE];
+ int ret = count;
+
+ st21nfc_dev = container_of(filp->private_data,
+ struct st21nfc_dev, st21nfc_device);
+ pr_debug("%s: st21nfc_dev ptr %p\n", __func__, st21nfc_dev);
+
+ if (count > MAX_BUFFER_SIZE)
+ count = MAX_BUFFER_SIZE;
+
+ if (copy_from_user(tmp, buf, count)) {
+ pr_err("%s : failed to copy from user space\n", __func__);
+ return -EFAULT;
+ }
+
+ pr_debug("%s : writing %zu bytes.\n", __func__, count);
+ /* Write data */
+ ret = i2c_master_send(st21nfc_dev->platform_data.client, tmp, count);
+ if (ret != count) {
+ pr_err("%s : i2c_master_send returned %d\n", __func__, ret);
+ ret = -EIO;
+ }
+ return ret;
+}
+
+static int st21nfc_dev_open(struct inode *inode, struct file *filp)
+{
+ int ret = 0;
+ struct st21nfc_dev *st21nfc_dev = NULL;
+
+ if (device_open) {
+ ret = -EBUSY;
+ pr_err("%s : device already opened ret= %d\n", __func__, ret);
+ } else {
+ device_open = true;
+ st21nfc_dev = container_of(filp->private_data,
+ struct st21nfc_dev,
+ st21nfc_device);
+
+ pr_debug("%s : device_open = %d", __func__, device_open);
+ pr_debug("%s : %d,%d ", __func__, imajor(inode), iminor(inode));
+
+ pr_debug("%s: st21nfc_dev ptr %p\n", __func__, st21nfc_dev);
+
+ }
+ return ret;
+}
+
+static int st21nfc_release(struct inode *inode, struct file *file)
+{
+ device_open = false;
+ pr_debug("%s : device_open = %d\n", __func__, device_open);
+
+ return 0;
+}
+
+static long st21nfc_dev_ioctl(struct file *filp, unsigned int cmd,
+ unsigned long arg)
+{
+ struct st21nfc_dev *st21nfc_dev = container_of(filp->private_data,
+ struct st21nfc_dev,
+ st21nfc_device);
+
+ int ret = 0;
+
+ switch (cmd) {
+
+ case ST21NFC_SET_POLARITY_FALLING:
+ pr_info(" ### ST21NFC_SET_POLARITY_FALLING ###");
+ st21nfc_loc_set_polaritymode(st21nfc_dev, IRQF_TRIGGER_FALLING);
+ break;
+
+ case ST21NFC_SET_POLARITY_RISING:
+ pr_info(" ### ST21NFC_SET_POLARITY_RISING ###");
+ st21nfc_loc_set_polaritymode(st21nfc_dev, IRQF_TRIGGER_RISING);
+ break;
+
+ case ST21NFC_SET_POLARITY_LOW:
+ pr_info(" ### ST21NFC_SET_POLARITY_LOW ###");
+ st21nfc_loc_set_polaritymode(st21nfc_dev, IRQF_TRIGGER_LOW);
+ break;
+
+ case ST21NFC_SET_POLARITY_HIGH:
+ pr_info(" ### ST21NFC_SET_POLARITY_HIGH ###");
+ st21nfc_loc_set_polaritymode(st21nfc_dev, IRQF_TRIGGER_HIGH);
+ break;
+
+ case ST21NFC_PULSE_RESET:
+#ifdef ST21_USES_SINGLE_RESET
+ pr_info("%s Pulse Request\n", __func__);
+ if (st21nfc_dev->platform_data.reset_gpio != 0) {
+ /* pulse low for 20 millisecs */
+ gpio_set_value(st21nfc_dev->platform_data.reset_gpio,
+ 0);
+ msleep(20);
+ gpio_set_value(st21nfc_dev->platform_data.reset_gpio,
+ 1);
+ pr_info("%s done Pulse Request\n", __func__);
+ }
+ break;
+#else
+ /* Double pulse is done to exit Quick boot mode.*/
+ pr_info("%s Double Pulse Request\n", __func__);
+ if (st21nfc_dev->platform_data.reset_gpio != 0) {
+ /* pulse low for 20 millisecs */
+ gpio_set_value(st21nfc_dev->platform_data.reset_gpio,
+ 0);
+ msleep(20);
+ gpio_set_value(st21nfc_dev->platform_data.reset_gpio,
+ 1);
+ msleep(20);
+ /* pulse low for 20 millisecs */
+ gpio_set_value(st21nfc_dev->platform_data.reset_gpio,
+ 0);
+ msleep(20);
+ gpio_set_value(st21nfc_dev->platform_data.reset_gpio,
+ 1);
+ pr_info("%s done Double Pulse Request\n", __func__);
+ }
+ break;
+#endif
+
+ case ST21NFC_GET_WAKEUP:
+ /* deliver state of Wake_up_pin as return value of ioctl */
+ ret = gpio_get_value(st21nfc_dev->platform_data.irq_gpio);
+ /*
+ * ret shall be equal to 1 if gpio level equals to polarity.
+ * Warning: depending on gpio_get_value implementation,
+ * it can returns a value different than 1 in case of high level
+ */
+ if (((ret == 0)
+ && (st21nfc_dev->platform_data.active_polarity == 0))
+ || ((ret > 0)
+ && (st21nfc_dev->platform_data.active_polarity == 1))) {
+ ret = 1;
+ } else {
+ ret = 0;
+ }
+ pr_debug("%s get gpio result %d\n", __func__, ret);
+ break;
+ case ST21NFC_GET_POLARITY:
+ ret = st21nfc_dev->platform_data.polarity_mode;
+ pr_debug("%s get polarity %d\n", __func__, ret);
+ break;
+ case ST21NFC_RECOVERY:
+ /* For ST21NFCD usage only */
+ pr_info("%s Recovery Request\n", __func__);
+ if (st21nfc_dev->platform_data.reset_gpio != 0) {
+ /* pulse low for 20 millisecs */
+ gpio_set_value(st21nfc_dev->platform_data.reset_gpio,
+ 0);
+ msleep(20);
+ /* during the reset, force IRQ OUT as DH output instead of
+ * input in normal usage
+ */
+ ret = gpio_direction_output(
+ st21nfc_dev->platform_data.irq_gpio, 1);
+ if (ret) {
+ pr_err("%s : gpio_direction_output failed\n",
+ __FILE__);
+ ret = -ENODEV;
+ break;
+ }
+ gpio_set_value(st21nfc_dev->platform_data.irq_gpio, 1);
+ msleep(20);
+ gpio_set_value(st21nfc_dev->platform_data.reset_gpio,
+ 1);
+ pr_info("%s done Pulse Request\n", __func__);
+ }
+ msleep(20);
+ gpio_set_value(st21nfc_dev->platform_data.irq_gpio, 0);
+ msleep(20);
+ gpio_set_value(st21nfc_dev->platform_data.irq_gpio, 1);
+ msleep(20);
+ gpio_set_value(st21nfc_dev->platform_data.irq_gpio, 0);
+ msleep(20);
+ pr_info("%s Recovery procedure finished\n", __func__);
+ ret = gpio_direction_input(
+ st21nfc_dev->platform_data.irq_gpio);
+ if (ret) {
+ pr_err("%s : gpio_direction_input failed\n", __FILE__);
+ ret = -ENODEV;
+ }
+ break;
+ default:
+ pr_err("%s bad ioctl %u\n", __func__, cmd);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static unsigned int st21nfc_poll(struct file *file, poll_table *wait)
+{
+ struct st21nfc_dev *st21nfc_dev = container_of(file->private_data,
+ struct st21nfc_dev,
+ st21nfc_device);
+ unsigned int mask = 0;
+ int pinlev = 0;
+
+ /* wait for Wake_up_pin == high */
+ pr_debug("%s call poll_Wait\n", __func__);
+ poll_wait(file, &st21nfc_dev->read_wq, wait);
+
+ pinlev = gpio_get_value(st21nfc_dev->platform_data.irq_gpio);
+
+ if (((pinlev == 0) && (st21nfc_dev->platform_data.active_polarity == 0))
+ || ((pinlev > 0)
+ && (st21nfc_dev->platform_data.active_polarity == 1))) {
+ pr_debug("%s return ready\n", __func__);
+ mask = POLLIN | POLLRDNORM; /* signal data avail */
+ st21nfc_disable_irq(st21nfc_dev);
+ } else {
+ /* Wake_up_pin is low. Activate ISR */
+ if (!st21nfc_dev->irq_enabled) {
+ pr_debug("%s enable irq\n", __func__);
+ st21nfc_dev->irq_enabled = true;
+ enable_irq(st21nfc_dev->platform_data.client->irq);
+ } else {
+ pr_debug("%s irq already enabled\n", __func__);
+ }
+
+ }
+ return mask;
+}
+
+static const struct file_operations st21nfc_dev_fops = {
+ .owner = THIS_MODULE,
+ .llseek = no_llseek,
+ .read = st21nfc_dev_read,
+ .write = st21nfc_dev_write,
+ .open = st21nfc_dev_open,
+ .poll = st21nfc_poll,
+ .release = st21nfc_release,
+
+ .unlocked_ioctl = st21nfc_dev_ioctl,
+#ifdef CONFIG_COMPAT
+ .compat_ioctl = st21nfc_dev_ioctl
+#endif
+};
+
+static ssize_t st21nfc_show_i2c_addr(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+
+ if (client != NULL)
+ return snprintf(buf, PAGE_SIZE, "0x%.2x\n", client->addr);
+ return 0;
+} /* st21nfc_show_i2c_addr() */
+
+static ssize_t st21nfc_change_i2c_addr(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+
+ struct st21nfc_dev *data = dev_get_drvdata(dev);
+ long new_addr = 0;
+
+ if (data != NULL && data->platform_data.client != NULL) {
+ if (!kstrtol(buf, 10, &new_addr)) {
+ mutex_lock(&data->platform_data.read_mutex);
+ data->platform_data.client->addr = new_addr;
+ mutex_unlock(&data->platform_data.read_mutex);
+ return count;
+ }
+ return -EINVAL;
+ }
+ return 0;
+} /* st21nfc_change_i2c_addr() */
+
+static ssize_t st21nfc_version(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%s\n", DRIVER_VERSION);
+} /* st21nfc_version */
+
+static DEVICE_ATTR(i2c_addr, 0644, st21nfc_show_i2c_addr,
+ st21nfc_change_i2c_addr);
+
+static DEVICE_ATTR(version, 0444, st21nfc_version, NULL);
+
+static struct attribute *st21nfc_attrs[] = {
+ &dev_attr_i2c_addr.attr,
+ &dev_attr_version.attr,
+ NULL,
+};
+
+static struct attribute_group st21nfc_attr_grp = {
+ .attrs = st21nfc_attrs,
+};
+
+static int nfc_parse_dt(struct device *dev, struct st21nfc_platform_data *pdata)
+{
+#ifdef CONFIG_OF
+ struct device_node *np = dev->of_node;
+
+ np = of_find_compatible_node(NULL, NULL, "st,st21nfc");
+ if (IS_ERR_OR_NULL(np)) {
+ pr_err("%s: cannot find compatible node \"%s\"",
+ __func__, "st,st21nfc");
+ return -ENODEV;
+ }
+
+ pdata->reset_gpio = of_get_named_gpio(np, "st,reset_gpio", 0);
+ if ((!gpio_is_valid(pdata->reset_gpio))) {
+ pr_err("%s: fail to get reset_gpio\n", __func__);
+ return -EINVAL;
+ }
+ pdata->irq_gpio = of_get_named_gpio(np, "st,irq_gpio", 0);
+ if ((!gpio_is_valid(pdata->irq_gpio))) {
+ pr_err("%s: fail to get irq_gpio\n", __func__);
+ return -EINVAL;
+ }
+
+ pdata->polarity_mode = 0;
+ pr_err("%s : get reset_gpio[%d], irq_gpio[%d], polarity_mode[%d]\n",
+ __func__, pdata->reset_gpio, pdata->irq_gpio,
+ pdata->polarity_mode);
+#endif
+ return 0;
+}
+
+static int st21nfc_probe(struct i2c_client *client,
+ const struct i2c_device_id *id)
+{
+ int ret;
+ struct st21nfc_platform_data *platform_data;
+ struct st21nfc_dev *st21nfc_dev;
+
+ dev_info(&client->dev, "st21nfc_probe\n");
+
+ if (client->dev.of_node) {
+ platform_data = devm_kzalloc(&client->dev,
+ sizeof(struct st21nfc_platform_data), GFP_KERNEL);
+ if (!platform_data)
+ return -ENOMEM;
+ dev_info(&client->dev, "%s: Parse st21nfc DTS\n", __func__);
+ ret = nfc_parse_dt(&client->dev, platform_data);
+ if (ret)
+ return ret;
+ } else {
+ platform_data = client->dev.platform_data;
+ pr_info("%s : No st21nfc DTS\n", __func__);
+ }
+ if (!platform_data) {
+ dev_err(&client->dev, "%s: no platform data\n", __func__);
+ return -EINVAL;
+ }
+ dev_dbg(&client->dev,
+ "nfc-nci probe: %s, inside nfc-nci flags = %x\n",
+ __func__, client->flags);
+
+ if (platform_data == NULL) {
+ dev_err(&client->dev, "nfc-nci probe: failed\n");
+ return -ENODEV;
+ }
+
+ if (!i2c_check_functionality(client->adapter, I2C_FUNC_I2C)) {
+ pr_err("%s : need I2C_FUNC_I2C\n", __func__);
+ return -ENODEV;
+ }
+
+ client->adapter->timeout = msecs_to_jiffies(3 * 10); /* 30ms */
+ client->adapter->retries = 0;
+
+ st21nfc_dev = kzalloc(sizeof(*st21nfc_dev), GFP_KERNEL);
+ if (st21nfc_dev == NULL) {
+ ret = -ENOMEM;
+ goto err_exit;
+ }
+
+ dev_dbg(&client->dev, "%s : dev_cb_addr %p\n", __func__, st21nfc_dev);
+
+ /* store for later use */
+ st21nfc_dev->platform_data.irq_gpio = platform_data->irq_gpio;
+ st21nfc_dev->platform_data.ena_gpio = platform_data->ena_gpio;
+ st21nfc_dev->platform_data.reset_gpio = platform_data->reset_gpio;
+ st21nfc_dev->platform_data.polarity_mode = platform_data->polarity_mode;
+ st21nfc_dev->platform_data.client = client;
+
+ ret = gpio_request(platform_data->irq_gpio, "irq_gpio");
+ if (ret) {
+ pr_err("%s : gpio_request failed\n", __FILE__);
+ ret = -ENODEV;
+ goto err_free_buffer;
+ }
+
+ ret = gpio_direction_input(platform_data->irq_gpio);
+ if (ret) {
+ pr_err("%s : gpio_direction_input failed\n", __FILE__);
+ ret = -ENODEV;
+ goto err_free_buffer;
+ }
+ /* initialize irqIsAttached variable */
+ irqIsAttached = false;
+
+ /* initialize device_open variable */
+ device_open = 0;
+
+ /* handle optional RESET */
+ if (platform_data->reset_gpio != 0) {
+ ret = gpio_request(platform_data->reset_gpio, "reset_gpio");
+ if (ret) {
+ pr_err("%s : reset gpio_request failed\n", __FILE__);
+ ret = -ENODEV;
+ goto err_free_buffer;
+ }
+ ret = gpio_direction_output(platform_data->reset_gpio, 1);
+ if (ret) {
+ pr_err("%s : reset gpio_direction_output failed\n",
+ __FILE__);
+ ret = -ENODEV;
+ goto err_free_buffer;
+ }
+ /* low active */
+ gpio_set_value(st21nfc_dev->platform_data.reset_gpio, 1);
+ }
+
+ client->irq = gpio_to_irq(platform_data->irq_gpio);
+
+ enable_irq_wake(client->irq);
+ /* init mutex and queues */
+ init_waitqueue_head(&st21nfc_dev->read_wq);
+ mutex_init(&st21nfc_dev->platform_data.read_mutex);
+ spin_lock_init(&st21nfc_dev->irq_enabled_lock);
+ dev_dbg(&client->dev, "%s : debug irq_gpio = %d, client-irq = %d\n",
+ __func__, platform_data->irq_gpio, client->irq);
+ st21nfc_dev->st21nfc_device.minor = MISC_DYNAMIC_MINOR;
+ st21nfc_dev->st21nfc_device.name = "st21nfc";
+ st21nfc_dev->st21nfc_device.fops = &st21nfc_dev_fops;
+ st21nfc_dev->st21nfc_device.parent = &client->dev;
+
+ i2c_set_clientdata(client, st21nfc_dev);
+ ret = misc_register(&st21nfc_dev->st21nfc_device);
+ if (ret) {
+ pr_err("%s : misc_register failed\n", __FILE__);
+ goto err_misc_register;
+ }
+
+ if (sysfs_create_group(&client->dev.kobj, &st21nfc_attr_grp)) {
+ pr_err("%s : sysfs_create_group failed\n", __FILE__);
+ goto err_request_irq_failed;
+ }
+ st21nfc_disable_irq(st21nfc_dev);
+ return 0;
+
+err_request_irq_failed:
+ misc_deregister(&st21nfc_dev->st21nfc_device);
+err_misc_register:
+ mutex_destroy(&st21nfc_dev->platform_data.read_mutex);
+err_free_buffer:
+ kfree(st21nfc_dev);
+err_exit:
+ gpio_free(platform_data->irq_gpio);
+ if (platform_data->ena_gpio != 0)
+ gpio_free(platform_data->ena_gpio);
+ return ret;
+}
+
+static int st21nfc_remove(struct i2c_client *client)
+{
+ struct st21nfc_dev *st21nfc_dev;
+
+ st21nfc_dev = i2c_get_clientdata(client);
+ free_irq(client->irq, st21nfc_dev);
+ misc_deregister(&st21nfc_dev->st21nfc_device);
+ mutex_destroy(&st21nfc_dev->platform_data.read_mutex);
+ gpio_free(st21nfc_dev->platform_data.irq_gpio);
+ if (st21nfc_dev->platform_data.ena_gpio != 0)
+ gpio_free(st21nfc_dev->platform_data.ena_gpio);
+ kfree(st21nfc_dev);
+
+ return 0;
+}
+
+static const struct i2c_device_id st21nfc_id[] = {
+ {"st21nfc", 0},
+ {}
+};
+
+static const struct of_device_id st21nfc_of_match[] = {
+ /* it's same as the compatible of nfc in dts. */
+ { .compatible = "st,st21nfc", },
+ {}
+};
+
+/* MODULE_DEVICE_TABLE(of, st21nfc_of_match); */
+static struct i2c_driver st21nfc_driver = {
+ .id_table = st21nfc_id,
+ .driver = {
+ .name = "st21nfc",
+ .owner = THIS_MODULE,
+ .of_match_table = st21nfc_of_match,
+ },
+ .probe = st21nfc_probe,
+ .remove = st21nfc_remove,
+};
+
+static const struct of_device_id st21nfc_board_of_match[] = {
+ { .compatible = "st,st21nfc", },
+ {}
+};
+
+static int st21nfc_platform_probe(struct platform_device *pdev)
+{
+ pr_err("st21nfc_platform_probe pr_err\n");
+ return 0;
+}
+
+static int st21nfc_platform_remove(struct platform_device *pdev)
+{
+ pr_err("st21nfc_platform_remove\n");
+ return 0;
+}
+
+static struct platform_driver st21nfc_platform_driver = {
+ .probe = st21nfc_platform_probe,
+ .remove = st21nfc_platform_remove,
+ .driver = {
+ .name = "st21nfc",
+ .owner = THIS_MODULE,
+ .of_match_table = st21nfc_board_of_match,
+ },
+};
+
+/*
+ * module load/unload record keeping
+ */
+
+static int __init st21nfc_dev_init(void)
+{
+ pr_info("%s: Loading st21nfc driver\n", __func__);
+ /* add by wuling to fix compilation error */
+ platform_driver_register(&st21nfc_platform_driver);
+ return i2c_add_driver(&st21nfc_driver);
+}
+
+module_init(st21nfc_dev_init);
+
+static void __exit st21nfc_dev_exit(void)
+{
+ pr_debug("Unloading st21nfc driver\n");
+ i2c_del_driver(&st21nfc_driver);
+}
+
+module_exit(st21nfc_dev_exit);
+
+MODULE_AUTHOR("Norbert Kawulski");
+MODULE_DESCRIPTION("NFC ST21NFC driver");
+MODULE_VERSION(DRIVER_VERSION);
+MODULE_LICENSE("GPL");
diff --git a/drivers/misc/st21nfc.h b/drivers/misc/st21nfc.h
new file mode 100644
index 0000000..68a55338
--- /dev/null
+++ b/drivers/misc/st21nfc.h
@@ -0,0 +1,44 @@
+/*
+ * Copyright (C) 2013 ST Microelectronics S.A.
+ * Copyright (C) 2010 Stollmann E+V GmbH
+ * Copyright (C) 2010 Trusted Logic S.A.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#define ST21NFC_MAGIC 0xEA
+
+#define ST21NFC_NAME "st21nfc"
+/*
+ * ST21NFC power control via ioctl
+ * ST21NFC_GET_WAKEUP : poll gpio-level for Wakeup pin
+ * PN544_SET_PWR(1): power on
+ * PN544_SET_PWR(>1): power on with firmware download enabled
+ */
+#define ST21NFC_GET_WAKEUP _IOR(ST21NFC_MAGIC, 0x01, unsigned int)
+#define ST21NFC_PULSE_RESET _IOR(ST21NFC_MAGIC, 0x02, unsigned int)
+#define ST21NFC_SET_POLARITY_RISING _IOR(ST21NFC_MAGIC, 0x03, unsigned int)
+#define ST21NFC_SET_POLARITY_FALLING _IOR(ST21NFC_MAGIC, 0x04, unsigned int)
+#define ST21NFC_SET_POLARITY_HIGH _IOR(ST21NFC_MAGIC, 0x05, unsigned int)
+#define ST21NFC_SET_POLARITY_LOW _IOR(ST21NFC_MAGIC, 0x06, unsigned int)
+#define ST21NFC_GET_POLARITY _IOR(ST21NFC_MAGIC, 0x07, unsigned int)
+#define ST21NFC_RECOVERY _IOR(ST21NFC_MAGIC, 0x08, unsigned int)
+
+
+struct st21nfc_platform_data {
+ unsigned int irq_gpio;
+ unsigned int ena_gpio;
+ unsigned int reset_gpio;
+ unsigned int polarity_mode;
+};
diff --git a/drivers/misc/st33/Kconfig b/drivers/misc/st33/Kconfig
new file mode 100644
index 0000000..6ecd5b9
--- /dev/null
+++ b/drivers/misc/st33/Kconfig
@@ -0,0 +1,9 @@
+#
+# ST33 SPI driver
+#
+config ST33
+ tristate "ST33"
+ depends on SPI
+ default "n"
+ ---help---
+ Include support for ST33 SPI driver
diff --git a/drivers/misc/st33/Makefile b/drivers/misc/st33/Makefile
new file mode 100644
index 0000000..6a6dc12
--- /dev/null
+++ b/drivers/misc/st33/Makefile
@@ -0,0 +1,5 @@
+#
+# Makefile for ST33
+#
+
+obj-$(CONFIG_ST33) += st33.o
diff --git a/drivers/misc/st33/st33.c b/drivers/misc/st33/st33.c
new file mode 100644
index 0000000..7bc8d2b
--- /dev/null
+++ b/drivers/misc/st33/st33.c
@@ -0,0 +1,628 @@
+/*
+ * Copyright (C) 2017 ST Microelectronics S.A.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/fs.h>
+#include <linux/version.h>
+#include <linux/slab.h>
+#include <linux/init.h>
+#include <linux/list.h>
+#include <linux/spi/spi.h>
+#include <linux/irq.h>
+#include <linux/jiffies.h>
+#include <linux/uaccess.h>
+#include <linux/delay.h>
+#include <linux/sched.h>
+#include <linux/interrupt.h>
+#include <linux/io.h>
+#include <linux/platform_device.h>
+#include <linux/gpio.h>
+#include <linux/poll.h>
+#include <linux/miscdevice.h>
+#include <linux/spinlock.h>
+#include "st33.h"
+#include <linux/of_gpio.h>
+
+#define SPIDEV_MAJOR 153 /* assigned */
+#define N_SPI_MINORS 32 /* ... up to 256 */
+
+//static DECLARE_BITMAP(minors, N_SPI_MINORS);
+
+#define NCI_MAX_SIZE 255
+#define NDLC_SIZE 1
+#define NDLC_CRC_SIZE 2
+#define MAX_BUFFER_SIZE (NCI_MAX_SIZE + NDLC_SIZE + NDLC_CRC_SIZE)
+
+#define DRIVER_VERSION "1.0.0"
+
+#define DEBUG
+#define INFO
+#define ERROR
+
+#ifdef DEBUG
+#define DBG_MSG(msg...) {pr_info(msg);}
+#else
+#define DBG_MSG(msg...) {}
+#endif
+#ifdef INFO
+#define INFO_MSG(msg...) {pr_info(msg);}
+#else
+#define INFO_MSG(msg...) {}
+#endif
+#ifdef ERROR
+#define ERR_MSG(msg...) {pr_err(msg);}
+#else
+#define ERR_MSG(msg...) {}
+#endif
+
+/* prototypes */
+static irqreturn_t st33_dev_irq_handler(int irq, void *dev_id);
+
+struct st33_data {
+ wait_queue_head_t read_wq;
+ struct spi_device *spi;
+ struct miscdevice st33_device;
+ bool irq_enabled;
+ struct mutex read_mutex;
+ unsigned int irq_gpio;
+ unsigned int pow_gpio;
+ unsigned int polarity_mode;
+ spinlock_t irq_lock;
+ unsigned char *rx_buffer;
+ unsigned int trans_speed;
+};
+
+/* Global Variables */
+static bool irqIsAttached;
+static bool device_open; /* Is device open? */
+
+static int st33_set_polarity_mode(struct st33_data *st33_dev)
+{
+ struct spi_device *spi = st33_dev->spi;
+ unsigned int irq_type;
+ unsigned int polarity_mode;
+ int ret;
+
+ DBG_MSG("Entry : %s\n",__func__);
+
+ irq_type = IRQ_TYPE_EDGE_FALLING;
+ polarity_mode = IRQF_TRIGGER_FALLING;
+
+ if (irqIsAttached) {
+ free_irq(spi->irq, st33_dev);
+ irqIsAttached = false;
+ }
+ ret = irq_set_irq_type(spi->irq, irq_type);
+ if (ret) {
+ ERR_MSG("%s : set_irq_type failed (ret=%d)\n", __func__, ret);
+ return -ENODEV;
+ }
+ /* request irq. the irq is set whenever the chip has data available
+ * for reading. it is cleared when all data has been read.
+ */
+ DBG_MSG("%s : requesting IRQ %d\n", __func__, spi->irq);
+ st33_dev->irq_enabled = true;
+
+ ret = request_irq(spi->irq, st33_dev_irq_handler,
+ polarity_mode,
+ st33_dev->st33_device.name,
+ st33_dev);
+
+ if (ret) {
+ ERR_MSG("%s : request_irq failed (ret=%d)\n", __func__, ret);
+ return -ENODEV;
+ }
+
+ irqIsAttached = true;
+
+ DBG_MSG("Exit : %s\n",__func__);
+
+ return ret;
+}
+
+static void st33_disable_irq(struct st33_data *st33_dev)
+{
+ unsigned long flags;
+ DBG_MSG("Entry : %s\n",__func__);
+
+ spin_lock_irqsave(&st33_dev->irq_lock, flags);
+ if (st33_dev->irq_enabled) {
+ disable_irq_nosync(st33_dev->spi->irq);
+ st33_dev->irq_enabled = false;
+ }
+ spin_unlock_irqrestore(&st33_dev->irq_lock, flags);
+ DBG_MSG("Exit : %s\n",__func__);
+}
+
+static irqreturn_t st33_dev_irq_handler(int irq, void *dev_id)
+{
+ struct st33_data *st33_dev = dev_id;
+
+ DBG_MSG("Entry : %s\n",__func__);
+
+ st33_disable_irq(st33_dev);
+
+ /* Wake up waiting readers */
+ wake_up(&st33_dev->read_wq);
+
+ DBG_MSG("Exit : %s\n",__func__);
+
+ return IRQ_HANDLED;
+}
+
+static ssize_t st33_dev_read(struct file *filp, char __user *buf,
+ size_t count, loff_t *offset)
+{
+ struct st33_data *st33_dev = container_of(filp->private_data,
+ struct st33_data,
+ st33_device);
+ struct spi_message msg;
+ struct spi_transfer xfer = {
+ .rx_buf = st33_dev->rx_buffer,
+ .len = count,
+ };
+ int ret;
+
+ DBG_MSG("Entry : %s (reading %zu bytes)\n",__func__,count);
+
+ if (count > MAX_BUFFER_SIZE)
+ count = MAX_BUFFER_SIZE;
+
+ /* Read data */
+ mutex_lock(&st33_dev->read_mutex);
+ xfer.speed_hz = st33_dev->trans_speed;
+ spi_message_init(&msg);
+ spi_message_add_tail(&xfer, &msg);
+ ret = spi_sync(st33_dev->spi, &msg);
+ mutex_unlock(&st33_dev->read_mutex);
+
+ if (ret < 0) {
+ ERR_MSG("Exit %s with ERROR: spi_sync returned %d\n", __func__, ret);
+ return ret;
+ }
+ if (copy_to_user(buf, st33_dev->rx_buffer, count)) {
+ ERR_MSG("Exit %s with ERROR: failed to copy to user space\n", __func__);
+ return -EFAULT;
+ }
+
+ DBG_MSG("Exit : %s (ret=%d)\n",__func__,ret);
+ ret = count;
+ return ret;
+}
+
+static ssize_t st33_dev_write(struct file *filp, const char __user *buf,
+ size_t count, loff_t *offset)
+{
+ struct st33_data *st33_dev = container_of(filp->private_data,
+ struct st33_data,
+ st33_device);
+ unsigned char buffer[MAX_BUFFER_SIZE];
+ int ret = 0;
+ struct spi_message msg;
+ struct spi_transfer xfer = {
+ .tx_buf = buffer,
+ .rx_buf = st33_dev->rx_buffer,
+ .len = count,
+ };
+
+ DBG_MSG("Entry : %s\n",__func__);
+
+ if (count > MAX_BUFFER_SIZE)
+ count = MAX_BUFFER_SIZE;
+
+ if (copy_from_user(buffer, buf, count)) {
+ ERR_MSG("Exit %s with ERROR: failed to copy from user space\n", __func__);
+ return -EFAULT;
+ }
+
+ pr_debug("%s : writing %zu bytes (speed=%d).\n", __func__, count,st33_dev->trans_speed);
+ /* Write data */
+ xfer.speed_hz = st33_dev->trans_speed;
+ spi_message_init(&msg);
+ spi_message_add_tail(&xfer, &msg);
+ ret = spi_sync(st33_dev->spi, &msg);
+
+ if (ret < 0) {
+ ERR_MSG("%s ERROR: spi_sync returned %d\n", __func__, ret);
+ }
+
+ ret = count;
+
+ DBG_MSG("Exit : %s (ret=%d)\n",__func__,ret);
+ return ret;
+}
+
+static int st33_dev_open(struct inode *inode, struct file *filp)
+{
+ int ret = 0;
+ struct st33_data *st33_dev = NULL;
+
+ DBG_MSG("Entry : %s\n",__func__);
+
+ if (device_open) {
+ ret = -EBUSY;
+ ERR_MSG("%s : device already opened ret= %d\n", __func__, ret);
+ }
+ else {
+ device_open = true;
+ st33_dev = container_of(filp->private_data,
+ struct st33_data,
+ st33_device);
+ DBG_MSG("%s : device_open = %d", __func__, device_open);
+ DBG_MSG("%s : Major No: %d, Minor No: %d ", __func__, imajor(inode), iminor(inode));
+ DBG_MSG("%s : st33_dev ptr %p\n", __func__, st33_dev);
+ }
+
+ DBG_MSG("Exit : %s\n",__func__);
+
+ return ret;
+}
+
+static int st33_release(struct inode *inode, struct file *file)
+{
+ device_open = false;
+ DBG_MSG("Entry : %s\n",__func__);
+
+ return 0;
+}
+
+static long st33_dev_ioctl(struct file *filp, unsigned int cmd,
+ unsigned long arg)
+{
+ struct st33_data *st33_dev = container_of(filp->private_data,
+ struct st33_data,
+ st33_device);
+
+ int ret = 0;
+ unsigned int tmp;
+
+ DBG_MSG("Entry : %s (cmd=%d)\n",__func__,cmd);
+
+ switch (cmd) {
+
+ case ST33_POW_ON:
+ INFO_MSG("%s Power On Request\n", __func__);
+ gpio_set_value(st33_dev->pow_gpio,1);
+ break;
+
+ case ST33_POW_OFF:
+ INFO_MSG("%s Power Off Request\n", __func__);
+ gpio_set_value(st33_dev->pow_gpio,0);
+ break;
+
+ case ST33_GET_SPI_SPEED_HZ:
+ INFO_MSG("%s Get SPI Speed request\n", __func__);
+ ret = st33_dev->trans_speed;
+ break;
+ case ST33_SET_SPI_SPEED_HZ:
+ ret = __get_user(tmp, (__u32 __user *)arg);
+ INFO_MSG("%s Set SPI Speed request (%d Hz)\n", __func__, tmp);
+ if (ret == 0) {
+ if ( (tmp >= ST33_SPI_FREQ_MIN) && (tmp <= ST33_SPI_FREQ_MAX) ) {
+ st33_dev->trans_speed = tmp;
+ INFO_MSG("%s Set SPI Speed to %d Hz (max)\n",__func__, tmp);
+ }
+ }
+ break;
+
+ case ST33_GET_IRQ_LEVEL:
+ /* deliver state of Wake_up_pin as return value of ioctl */
+ ret = gpio_get_value(st33_dev->irq_gpio);
+ /* change the result as the polarity is active low */
+ if (ret > 0)
+ ret = 0;
+ else if (ret == 0)
+ ret = 1;
+ INFO_MSG("%s : gpio level is %d\n", __func__,ret);
+ break;
+
+ default:
+ ERR_MSG("%s : bad ioctl %u\n", __func__, cmd);
+ ret = -EINVAL;
+ break;
+ }
+
+ return ret;
+}
+
+static unsigned int st33_poll(struct file *filp, poll_table *wait)
+{
+ struct st33_data *st33_dev = container_of(filp->private_data,
+ struct st33_data,
+ st33_device);
+ unsigned int mask = 0;
+ int pinlev = 0;
+
+ DBG_MSG("Entry : %s\n",__func__);
+
+ /* wait for Wake_up_pin == high */
+ poll_wait(filp, &st33_dev->read_wq, wait);
+
+ pinlev = gpio_get_value(st33_dev->irq_gpio);
+
+ if (pinlev == 0) {
+ DBG_MSG("%s detected IRQ pin to 0, disabling IRQ\n", __func__);
+ mask = POLLIN | POLLRDNORM; /* signal data available */
+ st33_disable_irq(st33_dev);
+ } else {
+ /* Wake_up_pin is high. Activate ISR */
+ if (!st33_dev->irq_enabled) {
+ DBG_MSG("%s enable irq\n", __func__);
+ st33_dev->irq_enabled = true;
+ enable_irq(st33_dev->spi->irq);
+ } else {
+ DBG_MSG("%s irq already enabled\n", __func__);
+ }
+ }
+
+ DBG_MSG("Exit : %s\n",__func__);
+
+ return mask;
+}
+
+static const struct file_operations st33_dev_fops = {
+ .owner = THIS_MODULE,
+ .write = st33_dev_write,
+ .read = st33_dev_read,
+ .unlocked_ioctl = st33_dev_ioctl,
+ .open = st33_dev_open,
+ .poll = st33_poll,
+ .release = st33_release,
+ .llseek = no_llseek,
+#ifdef CONFIG_COMPAT
+ .compat_ioctl = st33_dev_ioctl
+#endif
+};
+
+static int st33_parse_dt(struct device *dev, struct st33_platform_data *pdata)
+{
+ struct device_node *np = dev->of_node;
+
+ DBG_MSG("Entry : %s\n",__func__);
+
+ np = of_find_compatible_node(NULL, NULL, "st,st33");
+ if (IS_ERR_OR_NULL(np)) {
+ ERR_MSG("%s: cannot find compatible node \"%s\"", __func__, "st,st33");
+ return -ENODEV;
+ }
+
+ pdata->pow_gpio = of_get_named_gpio(np, "st,pow_gpio", 0);
+ if ((!gpio_is_valid(pdata->pow_gpio))) {
+ pr_err("[dsc]%s: fail to get pow_gpio\n", __func__);
+ return -EINVAL;
+ }
+
+ pdata->irq_gpio = of_get_named_gpio(np, "st,irq_gpio", 0);
+ if ((!gpio_is_valid(pdata->irq_gpio))) {
+ ERR_MSG("%s: fail to get irq_gpio\n", __func__);
+ return -EINVAL;
+ }
+
+ DBG_MSG("Exit : %s (get irq_gpio[%d])\n", __func__, pdata->irq_gpio);
+
+ return 0;
+}
+
+/**
+ * \ingroup spi_driver
+ * \brief Probe ST33 SPI device and create device entry for user space.
+ *
+ * \param[in] spi_device*
+ *
+ * \retval 0 if ok.
+ *
+*/
+static int st33_probe(struct spi_device *spi)
+{
+ int ret = -1;
+ struct st33_data *st33_dev = NULL;
+ struct st33_platform_data platform_data;
+
+ DBG_MSG("Entry : %s, chip select=%d, bus number=%d\n",
+ __func__, spi->chip_select, spi->master->bus_num);
+
+ /* check that dts file is correct */
+ ret = st33_parse_dt(&spi->dev, &platform_data);
+ if (ret) {
+ ERR_MSG("%s - Failed to parse DT\n", __func__);
+ return ret;
+ }
+
+ /* Allocate driver memory */
+ st33_dev = kzalloc(sizeof(*st33_dev), GFP_KERNEL);
+ if (!st33_dev) {
+ ERR_MSG("%s : Failed to allocate memory for module data\n", __func__);
+ return -ENOMEM;
+ }
+
+ st33_dev->irq_gpio = platform_data.irq_gpio;
+ st33_dev->pow_gpio = platform_data.pow_gpio;
+
+ /* GPIO IRQ initialization (Input) */
+ ret = gpio_request(st33_dev->irq_gpio, "irq_gpio");
+ if (ret) {
+ ERR_MSG("%s : gpio_request failed (gpio %d)\n", __func__, st33_dev->irq_gpio);
+ return -ENODEV;
+ }
+
+ ret = gpio_direction_input(st33_dev->irq_gpio);
+ if (ret) {
+ ERR_MSG("%s : gpio_direction_input failed (gpio %d)\n", __func__, st33_dev->irq_gpio);
+ return -ENODEV;
+ }
+
+ /* GPIO POW initialization (Output, 0)*/
+ ret = gpio_request(st33_dev->pow_gpio, "pow_gpio");
+ if (ret) {
+ ERR_MSG("%s : pow_gpio gpio_request failed\n", __func__);
+ return -ENOMEM;
+ }
+ ret = gpio_direction_output(st33_dev->pow_gpio, 1);
+ if (ret) {
+ pr_err("%s : pow_gpio gpio_direction_output failed\n",__func__);
+ return -ENOMEM;
+ }
+ gpio_set_value(st33_dev->pow_gpio, 0);
+
+ /* Initialize the driver data */
+ spi->bits_per_word = 8;
+ spi->mode = SPI_MODE_0;
+ spi->max_speed_hz = ST33_SPI_FREQ_MAX;
+ st33_dev->trans_speed = ST33_SPI_FREQ_MAX;
+ ret = spi_setup(spi);
+ if (ret) {
+ ERR_MSG("%s : spi_setup failed\n", __func__);
+ return ret;
+ }
+
+ st33_dev->spi = spi;
+ st33_dev -> st33_device.minor = MISC_DYNAMIC_MINOR;
+ st33_dev -> st33_device.name = ST33_NAME;
+ st33_dev -> st33_device.fops = &st33_dev_fops;
+ st33_dev -> st33_device.parent = &spi->dev;
+ st33_dev->rx_buffer = kzalloc(MAX_BUFFER_SIZE, GFP_KERNEL);
+
+ if (!st33_dev->rx_buffer) {
+ ERR_MSG("%s : Failed to allocate rx_buffer\n", __func__);
+ return -ENOMEM;
+ }
+ spi_set_drvdata(spi, st33_dev);
+
+ irqIsAttached = false;
+ device_open = 0;
+
+ spi->irq = gpio_to_irq(st33_dev->irq_gpio);
+ enable_irq_wake(st33_dev->spi->irq);
+
+ /* init mutex and queues */
+ init_waitqueue_head(&st33_dev->read_wq);
+ mutex_init(&st33_dev->read_mutex);
+
+ spin_lock_init(&st33_dev->irq_lock);
+
+ ret = misc_register(&st33_dev->st33_device);
+ if (ret) {
+ ERR_MSG("%s : misc_register failed\n", __func__);
+ return ret;
+ }
+
+ st33_set_polarity_mode(st33_dev);
+
+ st33_disable_irq(st33_dev);
+
+ DBG_MSG("Exit : %s\n",__func__);
+
+ return 0;
+}
+
+/**
+ * \ingroup spi_driver
+ * \brief Called to release the resources when the device is removed.
+ *
+ * \param[in] spi_device*
+ *
+ * \retval 0
+ *
+*/
+
+static int st33_remove(struct spi_device *spi)
+{
+ struct st33_data *st33_dev = spi_get_drvdata(spi);
+
+ DBG_MSG("Entry : %s\n",__func__);
+
+ free_irq(st33_dev->spi->irq, st33_dev);
+ gpio_free(st33_dev->irq_gpio);
+
+ mutex_destroy(&st33_dev->read_mutex);
+ misc_deregister(&st33_dev->st33_device);
+
+ kfree(st33_dev->rx_buffer);
+ st33_dev->rx_buffer = NULL;
+
+ if (st33_dev) {
+ kfree(st33_dev);
+ }
+
+ DBG_MSG("Exit : %s\n",__func__);
+
+ return 0;
+}
+
+static struct of_device_id st33_of_match[] = {
+ { .compatible = "st,st33", }, //it's same as the compatible of ese in dts.
+ {}
+};
+
+static struct spi_driver st33_driver = {
+ .driver = {
+ .name = ST33_NAME,
+ .owner = THIS_MODULE,
+ .bus = &spi_bus_type,
+ .of_match_table = st33_of_match,
+ },
+ .probe = st33_probe,
+ .remove = st33_remove,
+};
+
+/**
+ * \ingroup spi_driver
+ * \brief Module init interface
+ *
+ * \param[in] void
+ *
+ * \retval spi_register_driver(&st33_driver)
+ *
+*/
+static int __init st33_dev_init(void)
+{
+ int ret = 0;
+ INFO_MSG("\nEntry : %s, loading st33 driver\n", __func__);
+
+ ret = spi_register_driver(&st33_driver);
+
+ INFO_MSG("\nExit : %s, spi_register_driver ret = %d\n", __func__,ret);
+ return ret;
+}
+
+module_init(st33_dev_init);
+
+/**
+ * \ingroup spi_driver
+ * \brief Module exit interface
+ *
+ * \param[in] void
+ *
+ * \retval void
+ *
+*/
+
+static void __exit st33_dev_exit(void)
+{
+ DBG_MSG("Entry : %s\n",__func__);
+ spi_unregister_driver(&st33_driver);
+ DBG_MSG("Exit : %s\n",__func__);
+}
+
+module_exit(st33_dev_exit);
+
+MODULE_AUTHOR("STMicroelectronics");
+MODULE_DESCRIPTION("ST33 SPI driver");
+MODULE_LICENSE("GPL");
+MODULE_VERSION(DRIVER_VERSION);
+
+/** @} */
diff --git a/drivers/misc/st33/st33.h b/drivers/misc/st33/st33.h
new file mode 100644
index 0000000..302b2a0
--- /dev/null
+++ b/drivers/misc/st33/st33.h
@@ -0,0 +1,34 @@
+/*
+ * Copyright (C) 2017 ST Microelectronics S.A.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#define ST33_MAGIC 0xEA
+#define ST33_NAME "st33"
+#define ST33_SPI_FREQ_MIN 100000 /* 100KHz */
+#define ST33_SPI_FREQ_MAX 10000000 /* 3MHz */
+
+
+struct st33_platform_data {
+ unsigned int irq_gpio;
+ unsigned int pow_gpio;
+};
+
+#define ST33_GET_IRQ_LEVEL _IOR(ST33_MAGIC, 0x01, unsigned int)
+#define ST33_POW_ON _IOR(ST33_MAGIC, 0x02, unsigned int)
+#define ST33_POW_OFF _IOR(ST33_MAGIC, 0x03, unsigned int)
+#define ST33_RECOVERY _IOR(ST33_MAGIC, 0x04, unsigned int)
+#define ST33_GET_SPI_SPEED_HZ _IOR(ST33_MAGIC, 0x10, unsigned int)
+#define ST33_SET_SPI_SPEED_HZ _IOR(ST33_MAGIC, 0x11, unsigned int)
+
diff --git a/drivers/misc/ti-st/st_kim.c b/drivers/misc/ti-st/st_kim.c
index bf0d770..9be6433 100644
--- a/drivers/misc/ti-st/st_kim.c
+++ b/drivers/misc/ti-st/st_kim.c
@@ -756,14 +756,14 @@ static int kim_probe(struct platform_device *pdev)
err = gpio_request(kim_gdata->nshutdown, "kim");
if (unlikely(err)) {
pr_err(" gpio %d request failed ", kim_gdata->nshutdown);
- return err;
+ goto err_sysfs_group;
}
/* Configure nShutdown GPIO as output=0 */
err = gpio_direction_output(kim_gdata->nshutdown, 0);
if (unlikely(err)) {
pr_err(" unable to configure gpio %d", kim_gdata->nshutdown);
- return err;
+ goto err_sysfs_group;
}
/* get reference of pdev for request_firmware
*/
diff --git a/drivers/misc/tsl2550.c b/drivers/misc/tsl2550.c
index 87a1337..eb57610 100644
--- a/drivers/misc/tsl2550.c
+++ b/drivers/misc/tsl2550.c
@@ -177,7 +177,7 @@ static int tsl2550_calculate_lux(u8 ch0, u8 ch1)
} else
lux = 0;
else
- return -EAGAIN;
+ return 0;
/* LUX range check */
return lux > TSL2550_MAX_LUX ? TSL2550_MAX_LUX : lux;
diff --git a/drivers/misc/vmw_balloon.c b/drivers/misc/vmw_balloon.c
index 1e688bf..5e9122cd 100644
--- a/drivers/misc/vmw_balloon.c
+++ b/drivers/misc/vmw_balloon.c
@@ -45,6 +45,7 @@
#include <linux/seq_file.h>
#include <linux/vmw_vmci_defs.h>
#include <linux/vmw_vmci_api.h>
+#include <linux/io.h>
#include <asm/hypervisor.h>
MODULE_AUTHOR("VMware, Inc.");
@@ -341,7 +342,13 @@ static bool vmballoon_send_start(struct vmballoon *b, unsigned long req_caps)
success = false;
}
- if (b->capabilities & VMW_BALLOON_BATCHED_2M_CMDS)
+ /*
+ * 2MB pages are only supported with batching. If batching is for some
+ * reason disabled, do not use 2MB pages, since otherwise the legacy
+ * mechanism is used with 2MB pages, causing a failure.
+ */
+ if ((b->capabilities & VMW_BALLOON_BATCHED_2M_CMDS) &&
+ (b->capabilities & VMW_BALLOON_BATCHED_CMDS))
b->supported_page_sizes = 2;
else
b->supported_page_sizes = 1;
@@ -450,7 +457,7 @@ static int vmballoon_send_lock_page(struct vmballoon *b, unsigned long pfn,
pfn32 = (u32)pfn;
if (pfn32 != pfn)
- return -1;
+ return -EINVAL;
STATS_INC(b->stats.lock[false]);
@@ -460,14 +467,14 @@ static int vmballoon_send_lock_page(struct vmballoon *b, unsigned long pfn,
pr_debug("%s - ppn %lx, hv returns %ld\n", __func__, pfn, status);
STATS_INC(b->stats.lock_fail[false]);
- return 1;
+ return -EIO;
}
static int vmballoon_send_batched_lock(struct vmballoon *b,
unsigned int num_pages, bool is_2m_pages, unsigned int *target)
{
unsigned long status;
- unsigned long pfn = page_to_pfn(b->page);
+ unsigned long pfn = PHYS_PFN(virt_to_phys(b->batch_page));
STATS_INC(b->stats.lock[is_2m_pages]);
@@ -515,7 +522,7 @@ static bool vmballoon_send_batched_unlock(struct vmballoon *b,
unsigned int num_pages, bool is_2m_pages, unsigned int *target)
{
unsigned long status;
- unsigned long pfn = page_to_pfn(b->page);
+ unsigned long pfn = PHYS_PFN(virt_to_phys(b->batch_page));
STATS_INC(b->stats.unlock[is_2m_pages]);
@@ -576,15 +583,9 @@ static void vmballoon_pop(struct vmballoon *b)
}
}
- if (b->batch_page) {
- vunmap(b->batch_page);
- b->batch_page = NULL;
- }
-
- if (b->page) {
- __free_page(b->page);
- b->page = NULL;
- }
+ /* Clearing the batch_page unconditionally has no adverse effect */
+ free_page((unsigned long)b->batch_page);
+ b->batch_page = NULL;
}
/*
@@ -603,11 +604,12 @@ static int vmballoon_lock_page(struct vmballoon *b, unsigned int num_pages,
locked = vmballoon_send_lock_page(b, page_to_pfn(page), &hv_status,
target);
- if (locked > 0) {
+ if (locked) {
STATS_INC(b->stats.refused_alloc[false]);
- if (hv_status == VMW_BALLOON_ERROR_RESET ||
- hv_status == VMW_BALLOON_ERROR_PPN_NOTNEEDED) {
+ if (locked == -EIO &&
+ (hv_status == VMW_BALLOON_ERROR_RESET ||
+ hv_status == VMW_BALLOON_ERROR_PPN_NOTNEEDED)) {
vmballoon_free_page(page, false);
return -EIO;
}
@@ -623,7 +625,7 @@ static int vmballoon_lock_page(struct vmballoon *b, unsigned int num_pages,
} else {
vmballoon_free_page(page, false);
}
- return -EIO;
+ return locked;
}
/* track allocated page */
@@ -991,16 +993,13 @@ static const struct vmballoon_ops vmballoon_batched_ops = {
static bool vmballoon_init_batching(struct vmballoon *b)
{
- b->page = alloc_page(VMW_PAGE_ALLOC_NOSLEEP);
- if (!b->page)
+ struct page *page;
+
+ page = alloc_page(GFP_KERNEL | __GFP_ZERO);
+ if (!page)
return false;
- b->batch_page = vmap(&b->page, 1, VM_MAP, PAGE_KERNEL);
- if (!b->batch_page) {
- __free_page(b->page);
- return false;
- }
-
+ b->batch_page = page_address(page);
return true;
}
@@ -1038,29 +1037,30 @@ static void vmballoon_vmci_cleanup(struct vmballoon *b)
*/
static int vmballoon_vmci_init(struct vmballoon *b)
{
- int error = 0;
+ unsigned long error, dummy;
- if ((b->capabilities & VMW_BALLOON_SIGNALLED_WAKEUP_CMD) != 0) {
- error = vmci_doorbell_create(&b->vmci_doorbell,
- VMCI_FLAG_DELAYED_CB,
- VMCI_PRIVILEGE_FLAG_RESTRICTED,
- vmballoon_doorbell, b);
+ if ((b->capabilities & VMW_BALLOON_SIGNALLED_WAKEUP_CMD) == 0)
+ return 0;
- if (error == VMCI_SUCCESS) {
- VMWARE_BALLOON_CMD(VMCI_DOORBELL_SET,
- b->vmci_doorbell.context,
- b->vmci_doorbell.resource, error);
- STATS_INC(b->stats.doorbell_set);
- }
- }
+ error = vmci_doorbell_create(&b->vmci_doorbell, VMCI_FLAG_DELAYED_CB,
+ VMCI_PRIVILEGE_FLAG_RESTRICTED,
+ vmballoon_doorbell, b);
- if (error != 0) {
- vmballoon_vmci_cleanup(b);
+ if (error != VMCI_SUCCESS)
+ goto fail;
- return -EIO;
- }
+ error = VMWARE_BALLOON_CMD(VMCI_DOORBELL_SET, b->vmci_doorbell.context,
+ b->vmci_doorbell.resource, dummy);
+
+ STATS_INC(b->stats.doorbell_set);
+
+ if (error != VMW_BALLOON_SUCCESS)
+ goto fail;
return 0;
+fail:
+ vmballoon_vmci_cleanup(b);
+ return -EIO;
}
/*
@@ -1298,7 +1298,14 @@ static int __init vmballoon_init(void)
return 0;
}
-module_init(vmballoon_init);
+
+/*
+ * Using late_initcall() instead of module_init() allows the balloon to use the
+ * VMCI doorbell even when the balloon is built into the kernel. Otherwise the
+ * VMCI is probed only after the balloon is initialized. If the balloon is used
+ * as a module, late_initcall() is equivalent to module_init().
+ */
+late_initcall(vmballoon_init);
static void __exit vmballoon_exit(void)
{
diff --git a/drivers/misc/vmw_vmci/vmci_driver.c b/drivers/misc/vmw_vmci/vmci_driver.c
index d7eaf1e..003bfba 100644
--- a/drivers/misc/vmw_vmci/vmci_driver.c
+++ b/drivers/misc/vmw_vmci/vmci_driver.c
@@ -113,5 +113,5 @@ module_exit(vmci_drv_exit);
MODULE_AUTHOR("VMware, Inc.");
MODULE_DESCRIPTION("VMware Virtual Machine Communication Interface.");
-MODULE_VERSION("1.1.5.0-k");
+MODULE_VERSION("1.1.6.0-k");
MODULE_LICENSE("GPL v2");
diff --git a/drivers/misc/vmw_vmci/vmci_queue_pair.c b/drivers/misc/vmw_vmci/vmci_queue_pair.c
index f735ab4..5927db04 100644
--- a/drivers/misc/vmw_vmci/vmci_queue_pair.c
+++ b/drivers/misc/vmw_vmci/vmci_queue_pair.c
@@ -755,7 +755,7 @@ static int qp_host_get_user_memory(u64 produce_uva,
retval = get_user_pages_fast((uintptr_t) produce_uva,
produce_q->kernel_if->num_pages, 1,
produce_q->kernel_if->u.h.header_page);
- if (retval < produce_q->kernel_if->num_pages) {
+ if (retval < (int)produce_q->kernel_if->num_pages) {
pr_debug("get_user_pages_fast(produce) failed (retval=%d)",
retval);
qp_release_pages(produce_q->kernel_if->u.h.header_page,
@@ -767,7 +767,7 @@ static int qp_host_get_user_memory(u64 produce_uva,
retval = get_user_pages_fast((uintptr_t) consume_uva,
consume_q->kernel_if->num_pages, 1,
consume_q->kernel_if->u.h.header_page);
- if (retval < consume_q->kernel_if->num_pages) {
+ if (retval < (int)consume_q->kernel_if->num_pages) {
pr_debug("get_user_pages_fast(consume) failed (retval=%d)",
retval);
qp_release_pages(consume_q->kernel_if->u.h.header_page,
diff --git a/drivers/misc/vmw_vmci/vmci_resource.c b/drivers/misc/vmw_vmci/vmci_resource.c
index 9a53a30..f116460 100644
--- a/drivers/misc/vmw_vmci/vmci_resource.c
+++ b/drivers/misc/vmw_vmci/vmci_resource.c
@@ -56,7 +56,8 @@ static struct vmci_resource *vmci_resource_lookup(struct vmci_handle handle,
if (r->type == type &&
rid == handle.resource &&
- (cid == handle.context || cid == VMCI_INVALID_ID)) {
+ (cid == handle.context || cid == VMCI_INVALID_ID ||
+ handle.context == VMCI_INVALID_ID)) {
resource = r;
break;
}
diff --git a/drivers/mmc/core/core.c b/drivers/mmc/core/core.c
index 60f8a6d..edd9714 100644
--- a/drivers/mmc/core/core.c
+++ b/drivers/mmc/core/core.c
@@ -2896,7 +2896,7 @@ int mmc_regulator_set_ocr(struct mmc_host *mmc,
unsigned short vdd_bit)
{
int result = 0;
- int min_uV, max_uV;
+ int min_uV = 0, max_uV = 0;
if (vdd_bit) {
mmc_ocrbitnum_to_vdd(vdd_bit, &min_uV, &max_uV);
diff --git a/drivers/mmc/core/mmc.c b/drivers/mmc/core/mmc.c
index 99165f7..3b55544 100644
--- a/drivers/mmc/core/mmc.c
+++ b/drivers/mmc/core/mmc.c
@@ -29,6 +29,7 @@
#include "sd_ops.h"
#define DEFAULT_CMD6_TIMEOUT_MS 500
+#define MIN_CACHE_EN_TIMEOUT_MS 1600
static const unsigned int tran_exp[] = {
10000, 100000, 1000000, 10000000,
@@ -554,9 +555,7 @@ static int mmc_decode_ext_csd(struct mmc_card *card, u8 *ext_csd)
card->cid.year += 16;
/* check whether the eMMC card supports BKOPS */
- if (!mmc_card_broken_hpi(card) &&
- (ext_csd[EXT_CSD_BKOPS_SUPPORT] & 0x1) &&
- card->ext_csd.hpi) {
+ if (ext_csd[EXT_CSD_BKOPS_SUPPORT] & 0x1) {
card->ext_csd.bkops = 1;
card->ext_csd.bkops_en = ext_csd[EXT_CSD_BKOPS_EN];
card->ext_csd.raw_bkops_status =
@@ -2198,23 +2197,27 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
if (err) {
pr_warn("%s: Enabling HPI failed\n",
mmc_hostname(card->host));
+ card->ext_csd.hpi_en = 0;
err = 0;
- } else
+ } else {
card->ext_csd.hpi_en = 1;
+ }
}
/*
- * If cache size is higher than 0, this indicates
- * the existence of cache and it can be turned on.
- * If HPI is not supported then cache shouldn't be enabled.
+ * If cache size is higher than 0, this indicates the existence of cache
+ * and it can be turned on. Note that some eMMCs from Micron has been
+ * reported to need ~800 ms timeout, while enabling the cache after
+ * sudden power failure tests. Let's extend the timeout to a minimum of
+ * DEFAULT_CACHE_EN_TIMEOUT_MS and do it for all cards.
*/
- if (!mmc_card_broken_hpi(card) &&
- card->ext_csd.cache_size > 0) {
- if (card->ext_csd.hpi_en &&
- (!(card->quirks & MMC_QUIRK_CACHE_DISABLE))) {
+ if (card->ext_csd.cache_size > 0) {
+ unsigned int timeout_ms = MIN_CACHE_EN_TIMEOUT_MS;
+
+ timeout_ms = max(card->ext_csd.generic_cmd6_time, timeout_ms);
+ if (!(card->quirks & MMC_QUIRK_CACHE_DISABLE)) {
err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
- EXT_CSD_CACHE_CTRL, 1,
- card->ext_csd.generic_cmd6_time);
+ EXT_CSD_CACHE_CTRL, 1, timeout_ms);
if (err && err != -EBADMSG) {
pr_err("%s: %s: fail on CACHE_CTRL ON %d\n",
mmc_hostname(host), __func__, err);
@@ -2232,12 +2235,12 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
} else {
card->ext_csd.cache_ctrl = 1;
}
+
/* enable cache barrier if supported by the device */
if (card->ext_csd.cache_ctrl &&
card->ext_csd.barrier_support) {
err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
- EXT_CSD_BARRIER_CTRL, 1,
- card->ext_csd.generic_cmd6_time);
+ EXT_CSD_BARRIER_CTRL, 1, timeout_ms);
if (err && err != -EBADMSG) {
pr_err("%s: %s: mmc_switch() for BARRIER_CTRL fails %d\n",
mmc_hostname(host), __func__,
@@ -2263,8 +2266,7 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
* we want to avoid cache.
*/
err = mmc_switch(card, EXT_CSD_CMD_SET_NORMAL,
- EXT_CSD_CACHE_CTRL, 0,
- card->ext_csd.generic_cmd6_time);
+ EXT_CSD_CACHE_CTRL, 0, timeout_ms);
if (err) {
pr_err("%s: %s: fail on CACHE_CTRL OFF %d\n",
mmc_hostname(host), __func__, err);
@@ -2272,6 +2274,7 @@ static int mmc_init_card(struct mmc_host *host, u32 ocr,
}
}
}
+
/*
* The mandatory minimum values are defined for packed command.
* read: 5, write: 3
diff --git a/drivers/mmc/core/pwrseq_simple.c b/drivers/mmc/core/pwrseq_simple.c
index 1304160..8cd9ddf 100644
--- a/drivers/mmc/core/pwrseq_simple.c
+++ b/drivers/mmc/core/pwrseq_simple.c
@@ -39,14 +39,18 @@ static void mmc_pwrseq_simple_set_gpios_value(struct mmc_pwrseq_simple *pwrseq,
struct gpio_descs *reset_gpios = pwrseq->reset_gpios;
if (!IS_ERR(reset_gpios)) {
- int i;
- int values[reset_gpios->ndescs];
+ int i, *values;
+ int nvalues = reset_gpios->ndescs;
- for (i = 0; i < reset_gpios->ndescs; i++)
+ values = kmalloc_array(nvalues, sizeof(int), GFP_KERNEL);
+ if (!values)
+ return;
+
+ for (i = 0; i < nvalues; i++)
values[i] = value;
- gpiod_set_array_value_cansleep(
- reset_gpios->ndescs, reset_gpios->desc, values);
+ gpiod_set_array_value_cansleep(nvalues, reset_gpios->desc, values);
+ kfree(values);
}
}
diff --git a/drivers/mmc/core/sdio_cis.c b/drivers/mmc/core/sdio_cis.c
index 5d7f198..acb9f5ec 100644
--- a/drivers/mmc/core/sdio_cis.c
+++ b/drivers/mmc/core/sdio_cis.c
@@ -55,7 +55,7 @@ static int cistpl_vers_1(struct mmc_card *card, struct sdio_func *func,
for (i = 0; i < nr_strings; i++) {
buffer[i] = string;
- strlcpy(string, buf, strlen(buf) + 1);
+ strcpy(string, buf);
string += strlen(string) + 1;
buf += strlen(buf) + 1;
}
diff --git a/drivers/mmc/host/dw_mmc.c b/drivers/mmc/host/dw_mmc.c
index d382dbd..e10a00d 100644
--- a/drivers/mmc/host/dw_mmc.c
+++ b/drivers/mmc/host/dw_mmc.c
@@ -981,8 +981,8 @@ static void dw_mci_ctrl_thld(struct dw_mci *host, struct mmc_data *data)
* It's used when HS400 mode is enabled.
*/
if (data->flags & MMC_DATA_WRITE &&
- !(host->timing != MMC_TIMING_MMC_HS400))
- return;
+ host->timing != MMC_TIMING_MMC_HS400)
+ goto disable;
if (data->flags & MMC_DATA_WRITE)
enable = SDMMC_CARD_WR_THR_EN;
@@ -990,7 +990,8 @@ static void dw_mci_ctrl_thld(struct dw_mci *host, struct mmc_data *data)
enable = SDMMC_CARD_RD_THR_EN;
if (host->timing != MMC_TIMING_MMC_HS200 &&
- host->timing != MMC_TIMING_UHS_SDR104)
+ host->timing != MMC_TIMING_UHS_SDR104 &&
+ host->timing != MMC_TIMING_MMC_HS400)
goto disable;
blksz_depth = blksz / (1 << host->data_shift);
@@ -1163,6 +1164,8 @@ static void dw_mci_setup_bus(struct dw_mci_slot *slot, bool force_clkinit)
if (host->state == STATE_WAITING_CMD11_DONE)
sdmmc_cmd_bits |= SDMMC_CMD_VOLT_SWITCH;
+ slot->mmc->actual_clock = 0;
+
if (!clock) {
mci_writel(host, CLKENA, 0);
mci_send_cmd(slot, sdmmc_cmd_bits, 0);
@@ -1208,6 +1211,8 @@ static void dw_mci_setup_bus(struct dw_mci_slot *slot, bool force_clkinit)
/* keep the last clock value that was requested from core */
slot->__clk_old = clock;
+ slot->mmc->actual_clock = div ? ((host->bus_hz / div) >> 1) :
+ host->bus_hz;
}
host->current_speed = clock;
diff --git a/drivers/mmc/host/omap.c b/drivers/mmc/host/omap.c
index be3c49f..a4bf14e 100644
--- a/drivers/mmc/host/omap.c
+++ b/drivers/mmc/host/omap.c
@@ -104,6 +104,7 @@ struct mmc_omap_slot {
unsigned int vdd;
u16 saved_con;
u16 bus_mode;
+ u16 power_mode;
unsigned int fclk_freq;
struct tasklet_struct cover_tasklet;
@@ -1157,7 +1158,7 @@ static void mmc_omap_set_ios(struct mmc_host *mmc, struct mmc_ios *ios)
struct mmc_omap_slot *slot = mmc_priv(mmc);
struct mmc_omap_host *host = slot->host;
int i, dsor;
- int clk_enabled;
+ int clk_enabled, init_stream;
mmc_omap_select_slot(slot, 0);
@@ -1167,6 +1168,7 @@ static void mmc_omap_set_ios(struct mmc_host *mmc, struct mmc_ios *ios)
slot->vdd = ios->vdd;
clk_enabled = 0;
+ init_stream = 0;
switch (ios->power_mode) {
case MMC_POWER_OFF:
mmc_omap_set_power(slot, 0, ios->vdd);
@@ -1174,13 +1176,17 @@ static void mmc_omap_set_ios(struct mmc_host *mmc, struct mmc_ios *ios)
case MMC_POWER_UP:
/* Cannot touch dsor yet, just power up MMC */
mmc_omap_set_power(slot, 1, ios->vdd);
+ slot->power_mode = ios->power_mode;
goto exit;
case MMC_POWER_ON:
mmc_omap_fclk_enable(host, 1);
clk_enabled = 1;
dsor |= 1 << 11;
+ if (slot->power_mode != MMC_POWER_ON)
+ init_stream = 1;
break;
}
+ slot->power_mode = ios->power_mode;
if (slot->bus_mode != ios->bus_mode) {
if (slot->pdata->set_bus_mode != NULL)
@@ -1196,7 +1202,7 @@ static void mmc_omap_set_ios(struct mmc_host *mmc, struct mmc_ios *ios)
for (i = 0; i < 2; i++)
OMAP_MMC_WRITE(host, CON, dsor);
slot->saved_con = dsor;
- if (ios->power_mode == MMC_POWER_ON) {
+ if (init_stream) {
/* worst case at 400kHz, 80 cycles makes 200 microsecs */
int usecs = 250;
@@ -1234,6 +1240,7 @@ static int mmc_omap_new_slot(struct mmc_omap_host *host, int id)
slot->host = host;
slot->mmc = mmc;
slot->id = id;
+ slot->power_mode = MMC_POWER_UNDEFINED;
slot->pdata = &host->pdata->slots[id];
host->slots[id] = slot;
diff --git a/drivers/mmc/host/omap_hsmmc.c b/drivers/mmc/host/omap_hsmmc.c
index a082aa3..009242b 100644
--- a/drivers/mmc/host/omap_hsmmc.c
+++ b/drivers/mmc/host/omap_hsmmc.c
@@ -2105,7 +2105,6 @@ static int omap_hsmmc_probe(struct platform_device *pdev)
mmc->max_blk_size = 512; /* Block Length at max can be 1024 */
mmc->max_blk_count = 0xFFFF; /* No. of Blocks is 16 bits */
mmc->max_req_size = mmc->max_blk_size * mmc->max_blk_count;
- mmc->max_seg_size = mmc->max_req_size;
mmc->caps |= MMC_CAP_MMC_HIGHSPEED | MMC_CAP_SD_HIGHSPEED |
MMC_CAP_WAIT_WHILE_BUSY | MMC_CAP_ERASE;
@@ -2135,6 +2134,17 @@ static int omap_hsmmc_probe(struct platform_device *pdev)
goto err_irq;
}
+ /*
+ * Limit the maximum segment size to the lower of the request size
+ * and the DMA engine device segment size limits. In reality, with
+ * 32-bit transfers, the DMA engine can do longer segments than this
+ * but there is no way to represent that in the DMA model - if we
+ * increase this figure here, we get warnings from the DMA API debug.
+ */
+ mmc->max_seg_size = min3(mmc->max_req_size,
+ dma_get_max_seg_size(host->rx_chan->device->dev),
+ dma_get_max_seg_size(host->tx_chan->device->dev));
+
/* Request IRQ for MMC operations */
ret = devm_request_irq(&pdev->dev, host->irq, omap_hsmmc_irq, 0,
mmc_hostname(mmc), host);
@@ -2215,6 +2225,7 @@ static int omap_hsmmc_remove(struct platform_device *pdev)
dma_release_channel(host->tx_chan);
dma_release_channel(host->rx_chan);
+ dev_pm_clear_wake_irq(host->dev);
pm_runtime_dont_use_autosuspend(host->dev);
pm_runtime_put_sync(host->dev);
pm_runtime_disable(host->dev);
diff --git a/drivers/mmc/host/sdhci-iproc.c b/drivers/mmc/host/sdhci-iproc.c
index 50dd6bd..524c8e0 100644
--- a/drivers/mmc/host/sdhci-iproc.c
+++ b/drivers/mmc/host/sdhci-iproc.c
@@ -33,6 +33,8 @@ struct sdhci_iproc_host {
const struct sdhci_iproc_data *data;
u32 shadow_cmd;
u32 shadow_blk;
+ bool is_cmd_shadowed;
+ bool is_blk_shadowed;
};
#define REG_OFFSET_IN_BITS(reg) ((reg) << 3 & 0x18)
@@ -48,8 +50,22 @@ static inline u32 sdhci_iproc_readl(struct sdhci_host *host, int reg)
static u16 sdhci_iproc_readw(struct sdhci_host *host, int reg)
{
- u32 val = sdhci_iproc_readl(host, (reg & ~3));
- u16 word = val >> REG_OFFSET_IN_BITS(reg) & 0xffff;
+ struct sdhci_pltfm_host *pltfm_host = sdhci_priv(host);
+ struct sdhci_iproc_host *iproc_host = sdhci_pltfm_priv(pltfm_host);
+ u32 val;
+ u16 word;
+
+ if ((reg == SDHCI_TRANSFER_MODE) && iproc_host->is_cmd_shadowed) {
+ /* Get the saved transfer mode */
+ val = iproc_host->shadow_cmd;
+ } else if ((reg == SDHCI_BLOCK_SIZE || reg == SDHCI_BLOCK_COUNT) &&
+ iproc_host->is_blk_shadowed) {
+ /* Get the saved block info */
+ val = iproc_host->shadow_blk;
+ } else {
+ val = sdhci_iproc_readl(host, (reg & ~3));
+ }
+ word = val >> REG_OFFSET_IN_BITS(reg) & 0xffff;
return word;
}
@@ -105,13 +121,15 @@ static void sdhci_iproc_writew(struct sdhci_host *host, u16 val, int reg)
if (reg == SDHCI_COMMAND) {
/* Write the block now as we are issuing a command */
- if (iproc_host->shadow_blk != 0) {
+ if (iproc_host->is_blk_shadowed) {
sdhci_iproc_writel(host, iproc_host->shadow_blk,
SDHCI_BLOCK_SIZE);
- iproc_host->shadow_blk = 0;
+ iproc_host->is_blk_shadowed = false;
}
oldval = iproc_host->shadow_cmd;
- } else if (reg == SDHCI_BLOCK_SIZE || reg == SDHCI_BLOCK_COUNT) {
+ iproc_host->is_cmd_shadowed = false;
+ } else if ((reg == SDHCI_BLOCK_SIZE || reg == SDHCI_BLOCK_COUNT) &&
+ iproc_host->is_blk_shadowed) {
/* Block size and count are stored in shadow reg */
oldval = iproc_host->shadow_blk;
} else {
@@ -123,9 +141,11 @@ static void sdhci_iproc_writew(struct sdhci_host *host, u16 val, int reg)
if (reg == SDHCI_TRANSFER_MODE) {
/* Save the transfer mode until the command is issued */
iproc_host->shadow_cmd = newval;
+ iproc_host->is_cmd_shadowed = true;
} else if (reg == SDHCI_BLOCK_SIZE || reg == SDHCI_BLOCK_COUNT) {
/* Save the block info until the command is issued */
iproc_host->shadow_blk = newval;
+ iproc_host->is_blk_shadowed = true;
} else {
/* Command or other regular 32-bit write */
sdhci_iproc_writel(host, newval, reg & ~3);
@@ -176,7 +196,6 @@ static const struct sdhci_iproc_data iproc_data = {
.caps1 = SDHCI_DRIVER_TYPE_C |
SDHCI_DRIVER_TYPE_D |
SDHCI_SUPPORT_DDR50,
- .mmc_caps = MMC_CAP_1_8V_DDR,
};
static const struct sdhci_pltfm_data sdhci_bcm2835_pltfm_data = {
diff --git a/drivers/mmc/host/sdhci-pci-o2micro.c b/drivers/mmc/host/sdhci-pci-o2micro.c
index d48f031..e417e42 100644
--- a/drivers/mmc/host/sdhci-pci-o2micro.c
+++ b/drivers/mmc/host/sdhci-pci-o2micro.c
@@ -334,6 +334,9 @@ int sdhci_pci_o2_probe(struct sdhci_pci_chip *chip)
pci_write_config_byte(chip->pdev, O2_SD_LOCK_WP, scratch);
break;
case PCI_DEVICE_ID_O2_SEABIRD0:
+ if (chip->pdev->revision == 0x01)
+ chip->quirks |= SDHCI_QUIRK_DELAY_AFTER_POWER;
+ /* fall through */
case PCI_DEVICE_ID_O2_SEABIRD1:
/* UnLock WP */
ret = pci_read_config_byte(chip->pdev,
diff --git a/drivers/mmc/host/sdhci-tegra.c b/drivers/mmc/host/sdhci-tegra.c
index 20b6ff5..088a3ae 100644
--- a/drivers/mmc/host/sdhci-tegra.c
+++ b/drivers/mmc/host/sdhci-tegra.c
@@ -350,7 +350,8 @@ static const struct sdhci_pltfm_data sdhci_tegra30_pdata = {
SDHCI_QUIRK_NO_HISPD_BIT |
SDHCI_QUIRK_BROKEN_ADMA_ZEROLEN_DESC |
SDHCI_QUIRK_CAP_CLOCK_BASE_BROKEN,
- .quirks2 = SDHCI_QUIRK2_PRESET_VALUE_BROKEN,
+ .quirks2 = SDHCI_QUIRK2_PRESET_VALUE_BROKEN |
+ SDHCI_QUIRK2_BROKEN_HS200,
.ops = &tegra_sdhci_ops,
};
diff --git a/drivers/mmc/host/sdhci.c b/drivers/mmc/host/sdhci.c
index 36aecd2..4d33122 100644
--- a/drivers/mmc/host/sdhci.c
+++ b/drivers/mmc/host/sdhci.c
@@ -4299,14 +4299,21 @@ int sdhci_setup_host(struct sdhci_host *host)
!(mmc->caps2 & MMC_CAP2_NONHOTPLUG) && !host->mmc->extcon)
mmc->caps |= MMC_CAP_NEEDS_POLL;
- /* If vqmmc regulator and no 1.8V signalling, then there's no UHS */
if (!IS_ERR(mmc->supply.vqmmc)) {
ret = regulator_enable(mmc->supply.vqmmc);
+
+ /* If vqmmc provides no 1.8V signalling, then there's no UHS */
if (!regulator_is_supported_voltage(mmc->supply.vqmmc, 1700000,
1950000))
host->caps1 &= ~(SDHCI_SUPPORT_SDR104 |
SDHCI_SUPPORT_SDR50 |
SDHCI_SUPPORT_DDR50);
+
+ /* In eMMC case vqmmc might be a fixed 1.8V regulator */
+ if (!regulator_is_supported_voltage(mmc->supply.vqmmc, 2700000,
+ 3600000))
+ host->flags &= ~SDHCI_SIGNALING_330;
+
if (ret) {
pr_warn("%s: Failed to enable vqmmc regulator: %d\n",
mmc_hostname(mmc), ret);
diff --git a/drivers/mtd/chips/cfi_cmdset_0001.c b/drivers/mtd/chips/cfi_cmdset_0001.c
index 5e1b68c..e1b603c 100644
--- a/drivers/mtd/chips/cfi_cmdset_0001.c
+++ b/drivers/mtd/chips/cfi_cmdset_0001.c
@@ -45,6 +45,7 @@
#define I82802AB 0x00ad
#define I82802AC 0x00ac
#define PF38F4476 0x881c
+#define M28F00AP30 0x8963
/* STMicroelectronics chips */
#define M50LPW080 0x002F
#define M50FLW080A 0x0080
@@ -375,6 +376,17 @@ static void cfi_fixup_major_minor(struct cfi_private *cfi,
extp->MinorVersion = '1';
}
+static int cfi_is_micron_28F00AP30(struct cfi_private *cfi, struct flchip *chip)
+{
+ /*
+ * Micron(was Numonyx) 1Gbit bottom boot are buggy w.r.t
+ * Erase Supend for their small Erase Blocks(0x8000)
+ */
+ if (cfi->mfr == CFI_MFR_INTEL && cfi->id == M28F00AP30)
+ return 1;
+ return 0;
+}
+
static inline struct cfi_pri_intelext *
read_pri_intelext(struct map_info *map, __u16 adr)
{
@@ -831,21 +843,30 @@ static int chip_ready (struct map_info *map, struct flchip *chip, unsigned long
(mode == FL_WRITING && (cfip->SuspendCmdSupport & 1))))
goto sleep;
+ /* Do not allow suspend iff read/write to EB address */
+ if ((adr & chip->in_progress_block_mask) ==
+ chip->in_progress_block_addr)
+ goto sleep;
+
+ /* do not suspend small EBs, buggy Micron Chips */
+ if (cfi_is_micron_28F00AP30(cfi, chip) &&
+ (chip->in_progress_block_mask == ~(0x8000-1)))
+ goto sleep;
/* Erase suspend */
- map_write(map, CMD(0xB0), adr);
+ map_write(map, CMD(0xB0), chip->in_progress_block_addr);
/* If the flash has finished erasing, then 'erase suspend'
* appears to make some (28F320) flash devices switch to
* 'read' mode. Make sure that we switch to 'read status'
* mode so we get the right data. --rmk
*/
- map_write(map, CMD(0x70), adr);
+ map_write(map, CMD(0x70), chip->in_progress_block_addr);
chip->oldstate = FL_ERASING;
chip->state = FL_ERASE_SUSPENDING;
chip->erase_suspended = 1;
for (;;) {
- status = map_read(map, adr);
+ status = map_read(map, chip->in_progress_block_addr);
if (map_word_andequal(map, status, status_OK, status_OK))
break;
@@ -1041,8 +1062,8 @@ static void put_chip(struct map_info *map, struct flchip *chip, unsigned long ad
sending the 0x70 (Read Status) command to an erasing
chip and expecting it to be ignored, that's what we
do. */
- map_write(map, CMD(0xd0), adr);
- map_write(map, CMD(0x70), adr);
+ map_write(map, CMD(0xd0), chip->in_progress_block_addr);
+ map_write(map, CMD(0x70), chip->in_progress_block_addr);
chip->oldstate = FL_READY;
chip->state = FL_ERASING;
break;
@@ -1933,6 +1954,8 @@ static int __xipram do_erase_oneblock(struct map_info *map, struct flchip *chip,
map_write(map, CMD(0xD0), adr);
chip->state = FL_ERASING;
chip->erase_suspended = 0;
+ chip->in_progress_block_addr = adr;
+ chip->in_progress_block_mask = ~(len - 1);
ret = INVAL_CACHE_AND_WAIT(map, chip, adr,
adr, len,
diff --git a/drivers/mtd/chips/cfi_cmdset_0002.c b/drivers/mtd/chips/cfi_cmdset_0002.c
index 9dca881..de35a2a 100644
--- a/drivers/mtd/chips/cfi_cmdset_0002.c
+++ b/drivers/mtd/chips/cfi_cmdset_0002.c
@@ -42,7 +42,7 @@
#define AMD_BOOTLOC_BUG
#define FORCE_WORD_WRITE 0
-#define MAX_WORD_RETRIES 3
+#define MAX_RETRIES 3
#define SST49LF004B 0x0060
#define SST49LF040B 0x0050
@@ -812,9 +812,10 @@ static int get_chip(struct map_info *map, struct flchip *chip, unsigned long adr
(mode == FL_WRITING && (cfip->EraseSuspend & 0x2))))
goto sleep;
- /* We could check to see if we're trying to access the sector
- * that is currently being erased. However, no user will try
- * anything like that so we just wait for the timeout. */
+ /* Do not allow suspend iff read/write to EB address */
+ if ((adr & chip->in_progress_block_mask) ==
+ chip->in_progress_block_addr)
+ goto sleep;
/* Erase suspend */
/* It's harmless to issue the Erase-Suspend and Erase-Resume
@@ -1642,7 +1643,7 @@ static int __xipram do_write_oneword(struct map_info *map, struct flchip *chip,
map_write( map, CMD(0xF0), chip->start );
/* FIXME - should have reset delay before continuing */
- if (++retry_cnt <= MAX_WORD_RETRIES)
+ if (++retry_cnt <= MAX_RETRIES)
goto retry;
ret = -EIO;
@@ -1875,7 +1876,7 @@ static int __xipram do_write_buffer(struct map_info *map, struct flchip *chip,
if (time_after(jiffies, timeo) && !chip_ready(map, adr))
break;
- if (chip_ready(map, adr)) {
+ if (chip_good(map, adr, datum)) {
xip_enable(map, chip, adr);
goto op_done;
}
@@ -2101,7 +2102,7 @@ static int do_panic_write_oneword(struct map_info *map, struct flchip *chip,
map_write(map, CMD(0xF0), chip->start);
/* FIXME - should have reset delay before continuing */
- if (++retry_cnt <= MAX_WORD_RETRIES)
+ if (++retry_cnt <= MAX_RETRIES)
goto retry;
ret = -EIO;
@@ -2236,6 +2237,7 @@ static int __xipram do_erase_chip(struct map_info *map, struct flchip *chip)
unsigned long int adr;
DECLARE_WAITQUEUE(wait, current);
int ret = 0;
+ int retry_cnt = 0;
adr = cfi->addr_unlock1;
@@ -2253,6 +2255,7 @@ static int __xipram do_erase_chip(struct map_info *map, struct flchip *chip)
ENABLE_VPP(map);
xip_disable(map, chip, adr);
+ retry:
cfi_send_gen_cmd(0xAA, cfi->addr_unlock1, chip->start, map, cfi, cfi->device_type, NULL);
cfi_send_gen_cmd(0x55, cfi->addr_unlock2, chip->start, map, cfi, cfi->device_type, NULL);
cfi_send_gen_cmd(0x80, cfi->addr_unlock1, chip->start, map, cfi, cfi->device_type, NULL);
@@ -2263,6 +2266,7 @@ static int __xipram do_erase_chip(struct map_info *map, struct flchip *chip)
chip->state = FL_ERASING;
chip->erase_suspended = 0;
chip->in_progress_block_addr = adr;
+ chip->in_progress_block_mask = ~(map->size - 1);
INVALIDATE_CACHE_UDELAY(map, chip,
adr, map->size,
@@ -2288,12 +2292,13 @@ static int __xipram do_erase_chip(struct map_info *map, struct flchip *chip)
chip->erase_suspended = 0;
}
- if (chip_ready(map, adr))
+ if (chip_good(map, adr, map_word_ff(map)))
break;
if (time_after(jiffies, timeo)) {
printk(KERN_WARNING "MTD %s(): software timeout\n",
__func__ );
+ ret = -EIO;
break;
}
@@ -2301,12 +2306,15 @@ static int __xipram do_erase_chip(struct map_info *map, struct flchip *chip)
UDELAY(map, chip, adr, 1000000/HZ);
}
/* Did we succeed? */
- if (!chip_good(map, adr, map_word_ff(map))) {
+ if (ret) {
/* reset on all failures. */
map_write( map, CMD(0xF0), chip->start );
/* FIXME - should have reset delay before continuing */
- ret = -EIO;
+ if (++retry_cnt <= MAX_RETRIES) {
+ ret = 0;
+ goto retry;
+ }
}
chip->state = FL_READY;
@@ -2325,6 +2333,7 @@ static int __xipram do_erase_oneblock(struct map_info *map, struct flchip *chip,
unsigned long timeo = jiffies + HZ;
DECLARE_WAITQUEUE(wait, current);
int ret = 0;
+ int retry_cnt = 0;
adr += chip->start;
@@ -2342,6 +2351,7 @@ static int __xipram do_erase_oneblock(struct map_info *map, struct flchip *chip,
ENABLE_VPP(map);
xip_disable(map, chip, adr);
+ retry:
cfi_send_gen_cmd(0xAA, cfi->addr_unlock1, chip->start, map, cfi, cfi->device_type, NULL);
cfi_send_gen_cmd(0x55, cfi->addr_unlock2, chip->start, map, cfi, cfi->device_type, NULL);
cfi_send_gen_cmd(0x80, cfi->addr_unlock1, chip->start, map, cfi, cfi->device_type, NULL);
@@ -2352,6 +2362,7 @@ static int __xipram do_erase_oneblock(struct map_info *map, struct flchip *chip,
chip->state = FL_ERASING;
chip->erase_suspended = 0;
chip->in_progress_block_addr = adr;
+ chip->in_progress_block_mask = ~(len - 1);
INVALIDATE_CACHE_UDELAY(map, chip,
adr, len,
@@ -2377,7 +2388,7 @@ static int __xipram do_erase_oneblock(struct map_info *map, struct flchip *chip,
chip->erase_suspended = 0;
}
- if (chip_ready(map, adr)) {
+ if (chip_good(map, adr, map_word_ff(map))) {
xip_enable(map, chip, adr);
break;
}
@@ -2386,6 +2397,7 @@ static int __xipram do_erase_oneblock(struct map_info *map, struct flchip *chip,
xip_enable(map, chip, adr);
printk(KERN_WARNING "MTD %s(): software timeout\n",
__func__ );
+ ret = -EIO;
break;
}
@@ -2393,12 +2405,15 @@ static int __xipram do_erase_oneblock(struct map_info *map, struct flchip *chip,
UDELAY(map, chip, adr, 1000000/HZ);
}
/* Did we succeed? */
- if (!chip_good(map, adr, map_word_ff(map))) {
+ if (ret) {
/* reset on all failures. */
map_write( map, CMD(0xF0), chip->start );
/* FIXME - should have reset delay before continuing */
- ret = -EIO;
+ if (++retry_cnt <= MAX_RETRIES) {
+ ret = 0;
+ goto retry;
+ }
}
chip->state = FL_READY;
@@ -2528,7 +2543,7 @@ static int cfi_atmel_unlock(struct mtd_info *mtd, loff_t ofs, uint64_t len)
struct ppb_lock {
struct flchip *chip;
- loff_t offset;
+ unsigned long adr;
int locked;
};
@@ -2546,8 +2561,9 @@ static int __maybe_unused do_ppb_xxlock(struct map_info *map,
unsigned long timeo;
int ret;
+ adr += chip->start;
mutex_lock(&chip->mutex);
- ret = get_chip(map, chip, adr + chip->start, FL_LOCKING);
+ ret = get_chip(map, chip, adr, FL_LOCKING);
if (ret) {
mutex_unlock(&chip->mutex);
return ret;
@@ -2565,8 +2581,8 @@ static int __maybe_unused do_ppb_xxlock(struct map_info *map,
if (thunk == DO_XXLOCK_ONEBLOCK_LOCK) {
chip->state = FL_LOCKING;
- map_write(map, CMD(0xA0), chip->start + adr);
- map_write(map, CMD(0x00), chip->start + adr);
+ map_write(map, CMD(0xA0), adr);
+ map_write(map, CMD(0x00), adr);
} else if (thunk == DO_XXLOCK_ONEBLOCK_UNLOCK) {
/*
* Unlocking of one specific sector is not supported, so we
@@ -2604,7 +2620,7 @@ static int __maybe_unused do_ppb_xxlock(struct map_info *map,
map_write(map, CMD(0x00), chip->start);
chip->state = FL_READY;
- put_chip(map, chip, adr + chip->start);
+ put_chip(map, chip, adr);
mutex_unlock(&chip->mutex);
return ret;
@@ -2661,9 +2677,9 @@ static int __maybe_unused cfi_ppb_unlock(struct mtd_info *mtd, loff_t ofs,
* sectors shall be unlocked, so lets keep their locking
* status at "unlocked" (locked=0) for the final re-locking.
*/
- if ((adr < ofs) || (adr >= (ofs + len))) {
+ if ((offset < ofs) || (offset >= (ofs + len))) {
sect[sectors].chip = &cfi->chips[chipnum];
- sect[sectors].offset = offset;
+ sect[sectors].adr = adr;
sect[sectors].locked = do_ppb_xxlock(
map, &cfi->chips[chipnum], adr, 0,
DO_XXLOCK_ONEBLOCK_GETLOCK);
@@ -2677,6 +2693,8 @@ static int __maybe_unused cfi_ppb_unlock(struct mtd_info *mtd, loff_t ofs,
i++;
if (adr >> cfi->chipshift) {
+ if (offset >= (ofs + len))
+ break;
adr = 0;
chipnum++;
@@ -2707,7 +2725,7 @@ static int __maybe_unused cfi_ppb_unlock(struct mtd_info *mtd, loff_t ofs,
*/
for (i = 0; i < sectors; i++) {
if (sect[i].locked)
- do_ppb_xxlock(map, sect[i].chip, sect[i].offset, 0,
+ do_ppb_xxlock(map, sect[i].chip, sect[i].adr, 0,
DO_XXLOCK_ONEBLOCK_LOCK);
}
diff --git a/drivers/mtd/devices/Kconfig b/drivers/mtd/devices/Kconfig
index e0f0c06..858a5e9 100644
--- a/drivers/mtd/devices/Kconfig
+++ b/drivers/mtd/devices/Kconfig
@@ -209,7 +209,7 @@
config MTD_DOCG3
tristate "M-Systems Disk-On-Chip G3"
select BCH
- select BCH_CONST_PARAMS
+ select BCH_CONST_PARAMS if !MTD_NAND_BCH
select BITREVERSE
---help---
This provides an MTD device driver for the M-Systems DiskOnChip
diff --git a/drivers/mtd/devices/m25p80.c b/drivers/mtd/devices/m25p80.c
index 9cf7fcd..16a7df2 100644
--- a/drivers/mtd/devices/m25p80.c
+++ b/drivers/mtd/devices/m25p80.c
@@ -172,7 +172,8 @@ static ssize_t m25p80_read(struct spi_nor *nor, loff_t from, size_t len,
t[1].rx_buf = buf;
t[1].rx_nbits = m25p80_rx_nbits(nor);
- t[1].len = min(len, spi_max_transfer_size(spi));
+ t[1].len = min3(len, spi_max_transfer_size(spi),
+ spi_max_message_size(spi) - t[0].len);
spi_message_add_tail(&t[1], &m);
ret = spi_sync(spi, &m);
diff --git a/drivers/mtd/maps/solutionengine.c b/drivers/mtd/maps/solutionengine.c
index bb580bc..c07f21b 100644
--- a/drivers/mtd/maps/solutionengine.c
+++ b/drivers/mtd/maps/solutionengine.c
@@ -59,9 +59,9 @@ static int __init init_soleng_maps(void)
return -ENXIO;
}
}
- printk(KERN_NOTICE "Solution Engine: Flash at 0x%08lx, EPROM at 0x%08lx\n",
- soleng_flash_map.phys & 0x1fffffff,
- soleng_eprom_map.phys & 0x1fffffff);
+ printk(KERN_NOTICE "Solution Engine: Flash at 0x%pap, EPROM at 0x%pap\n",
+ &soleng_flash_map.phys,
+ &soleng_eprom_map.phys);
flash_mtd->owner = THIS_MODULE;
eprom_mtd = do_map_probe("map_rom", &soleng_eprom_map);
diff --git a/drivers/mtd/mtdchar.c b/drivers/mtd/mtdchar.c
index b4092ea..95b6a66 100644
--- a/drivers/mtd/mtdchar.c
+++ b/drivers/mtd/mtdchar.c
@@ -160,8 +160,12 @@ static ssize_t mtdchar_read(struct file *file, char __user *buf, size_t count,
pr_debug("MTD_read\n");
- if (*ppos + count > mtd->size)
- count = mtd->size - *ppos;
+ if (*ppos + count > mtd->size) {
+ if (*ppos < mtd->size)
+ count = mtd->size - *ppos;
+ else
+ count = 0;
+ }
if (!count)
return 0;
@@ -246,7 +250,7 @@ static ssize_t mtdchar_write(struct file *file, const char __user *buf, size_t c
pr_debug("MTD_write\n");
- if (*ppos == mtd->size)
+ if (*ppos >= mtd->size)
return -ENOSPC;
if (*ppos + count > mtd->size)
diff --git a/drivers/mtd/nand/fsl_ifc_nand.c b/drivers/mtd/nand/fsl_ifc_nand.c
index 2f6b552..4c3b986 100644
--- a/drivers/mtd/nand/fsl_ifc_nand.c
+++ b/drivers/mtd/nand/fsl_ifc_nand.c
@@ -372,9 +372,16 @@ static void fsl_ifc_cmdfunc(struct mtd_info *mtd, unsigned int command,
case NAND_CMD_READID:
case NAND_CMD_PARAM: {
+ /*
+ * For READID, read 8 bytes that are currently used.
+ * For PARAM, read all 3 copies of 256-bytes pages.
+ */
+ int len = 8;
int timing = IFC_FIR_OP_RB;
- if (command == NAND_CMD_PARAM)
+ if (command == NAND_CMD_PARAM) {
timing = IFC_FIR_OP_RBCD;
+ len = 256 * 3;
+ }
ifc_out32((IFC_FIR_OP_CW0 << IFC_NAND_FIR0_OP0_SHIFT) |
(IFC_FIR_OP_UA << IFC_NAND_FIR0_OP1_SHIFT) |
@@ -384,12 +391,8 @@ static void fsl_ifc_cmdfunc(struct mtd_info *mtd, unsigned int command,
&ifc->ifc_nand.nand_fcr0);
ifc_out32(column, &ifc->ifc_nand.row3);
- /*
- * although currently it's 8 bytes for READID, we always read
- * the maximum 256 bytes(for PARAM)
- */
- ifc_out32(256, &ifc->ifc_nand.nand_fbcr);
- ifc_nand_ctrl->read_bytes = 256;
+ ifc_out32(len, &ifc->ifc_nand.nand_fbcr);
+ ifc_nand_ctrl->read_bytes = len;
set_addr(mtd, 0, 0, 0);
fsl_ifc_run_command(mtd);
diff --git a/drivers/mtd/nand/mxc_nand.c b/drivers/mtd/nand/mxc_nand.c
index 0c84ee8..5c44eb5 100644
--- a/drivers/mtd/nand/mxc_nand.c
+++ b/drivers/mtd/nand/mxc_nand.c
@@ -48,7 +48,7 @@
#define NFC_V1_V2_CONFIG (host->regs + 0x0a)
#define NFC_V1_V2_ECC_STATUS_RESULT (host->regs + 0x0c)
#define NFC_V1_V2_RSLTMAIN_AREA (host->regs + 0x0e)
-#define NFC_V1_V2_RSLTSPARE_AREA (host->regs + 0x10)
+#define NFC_V21_RSLTSPARE_AREA (host->regs + 0x10)
#define NFC_V1_V2_WRPROT (host->regs + 0x12)
#define NFC_V1_UNLOCKSTART_BLKADDR (host->regs + 0x14)
#define NFC_V1_UNLOCKEND_BLKADDR (host->regs + 0x16)
@@ -1121,6 +1121,9 @@ static void preset_v2(struct mtd_info *mtd)
writew(config1, NFC_V1_V2_CONFIG1);
/* preset operation */
+ /* spare area size in 16-bit half-words */
+ writew(mtd->oobsize / 2, NFC_V21_RSLTSPARE_AREA);
+
/* Unlock the internal RAM Buffer */
writew(0x2, NFC_V1_V2_CONFIG);
diff --git a/drivers/mtd/nand/qcom_nandc.c b/drivers/mtd/nand/qcom_nandc.c
index 6f0fd15..9f6c9a3 100644
--- a/drivers/mtd/nand/qcom_nandc.c
+++ b/drivers/mtd/nand/qcom_nandc.c
@@ -142,15 +142,15 @@
#define NAND_VERSION_MINOR_SHIFT 16
/* NAND OP_CMDs */
-#define PAGE_READ 0x2
-#define PAGE_READ_WITH_ECC 0x3
-#define PAGE_READ_WITH_ECC_SPARE 0x4
-#define PROGRAM_PAGE 0x6
-#define PAGE_PROGRAM_WITH_ECC 0x7
-#define PROGRAM_PAGE_SPARE 0x9
-#define BLOCK_ERASE 0xa
-#define FETCH_ID 0xb
-#define RESET_DEVICE 0xd
+#define OP_PAGE_READ 0x2
+#define OP_PAGE_READ_WITH_ECC 0x3
+#define OP_PAGE_READ_WITH_ECC_SPARE 0x4
+#define OP_PROGRAM_PAGE 0x6
+#define OP_PAGE_PROGRAM_WITH_ECC 0x7
+#define OP_PROGRAM_PAGE_SPARE 0x9
+#define OP_BLOCK_ERASE 0xa
+#define OP_FETCH_ID 0xb
+#define OP_RESET_DEVICE 0xd
/* Default Value for NAND_DEV_CMD_VLD */
#define NAND_DEV_CMD_VLD_VAL (READ_START_VLD | WRITE_START_VLD | \
@@ -425,11 +425,11 @@ static void update_rw_regs(struct qcom_nand_host *host, int num_cw, bool read)
if (read) {
if (host->use_ecc)
- cmd = PAGE_READ_WITH_ECC | PAGE_ACC | LAST_PAGE;
+ cmd = OP_PAGE_READ_WITH_ECC | PAGE_ACC | LAST_PAGE;
else
- cmd = PAGE_READ | PAGE_ACC | LAST_PAGE;
+ cmd = OP_PAGE_READ | PAGE_ACC | LAST_PAGE;
} else {
- cmd = PROGRAM_PAGE | PAGE_ACC | LAST_PAGE;
+ cmd = OP_PROGRAM_PAGE | PAGE_ACC | LAST_PAGE;
}
if (host->use_ecc) {
@@ -662,7 +662,7 @@ static int nandc_param(struct qcom_nand_host *host)
* in use. we configure the controller to perform a raw read of 512
* bytes to read onfi params
*/
- nandc_set_reg(nandc, NAND_FLASH_CMD, PAGE_READ | PAGE_ACC | LAST_PAGE);
+ nandc_set_reg(nandc, NAND_FLASH_CMD, OP_PAGE_READ | PAGE_ACC | LAST_PAGE);
nandc_set_reg(nandc, NAND_ADDR0, 0);
nandc_set_reg(nandc, NAND_ADDR1, 0);
nandc_set_reg(nandc, NAND_DEV0_CFG0, 0 << CW_PER_PAGE
@@ -715,7 +715,7 @@ static int erase_block(struct qcom_nand_host *host, int page_addr)
struct qcom_nand_controller *nandc = get_qcom_nand_controller(chip);
nandc_set_reg(nandc, NAND_FLASH_CMD,
- BLOCK_ERASE | PAGE_ACC | LAST_PAGE);
+ OP_BLOCK_ERASE | PAGE_ACC | LAST_PAGE);
nandc_set_reg(nandc, NAND_ADDR0, page_addr);
nandc_set_reg(nandc, NAND_ADDR1, 0);
nandc_set_reg(nandc, NAND_DEV0_CFG0,
@@ -746,7 +746,7 @@ static int read_id(struct qcom_nand_host *host, int column)
if (column == -1)
return 0;
- nandc_set_reg(nandc, NAND_FLASH_CMD, FETCH_ID);
+ nandc_set_reg(nandc, NAND_FLASH_CMD, OP_FETCH_ID);
nandc_set_reg(nandc, NAND_ADDR0, column);
nandc_set_reg(nandc, NAND_ADDR1, 0);
nandc_set_reg(nandc, NAND_FLASH_CHIP_SELECT, DM_EN);
@@ -766,7 +766,7 @@ static int reset(struct qcom_nand_host *host)
struct nand_chip *chip = &host->chip;
struct qcom_nand_controller *nandc = get_qcom_nand_controller(chip);
- nandc_set_reg(nandc, NAND_FLASH_CMD, RESET_DEVICE);
+ nandc_set_reg(nandc, NAND_FLASH_CMD, OP_RESET_DEVICE);
nandc_set_reg(nandc, NAND_EXEC_CMD, 1);
write_reg_dma(nandc, NAND_FLASH_CMD, 1);
@@ -2008,6 +2008,9 @@ static int qcom_nand_host_init(struct qcom_nand_controller *nandc,
nand_set_flash_node(chip, dn);
mtd->name = devm_kasprintf(dev, GFP_KERNEL, "qcom_nand.%d", host->cs);
+ if (!mtd->name)
+ return -ENOMEM;
+
mtd->owner = THIS_MODULE;
mtd->dev.parent = dev;
diff --git a/drivers/mtd/spi-nor/Kconfig b/drivers/mtd/spi-nor/Kconfig
index 4a682ee..b4f6cadd 100644
--- a/drivers/mtd/spi-nor/Kconfig
+++ b/drivers/mtd/spi-nor/Kconfig
@@ -31,7 +31,7 @@
config SPI_ATMEL_QUADSPI
tristate "Atmel Quad SPI Controller"
- depends on ARCH_AT91 || (ARM && COMPILE_TEST)
+ depends on ARCH_AT91 || (ARM && COMPILE_TEST && !ARCH_EBSA110)
depends on OF && HAS_IOMEM
help
This enables support for the Quad SPI controller in master mode.
diff --git a/drivers/mtd/spi-nor/spi-nor.c b/drivers/mtd/spi-nor/spi-nor.c
index 21dde52..fe97d9f 100644
--- a/drivers/mtd/spi-nor/spi-nor.c
+++ b/drivers/mtd/spi-nor/spi-nor.c
@@ -858,6 +858,12 @@ static const struct flash_info spi_nor_ids[] = {
/* ISSI */
{ "is25cd512", INFO(0x7f9d20, 0, 32 * 1024, 2, SECT_4K) },
+ { "is25wp032", INFO(0x9d7016, 0, 64 * 1024, 64,
+ SECT_4K | SPI_NOR_DUAL_READ | SPI_NOR_QUAD_READ) },
+ { "is25wp064", INFO(0x9d7017, 0, 64 * 1024, 128,
+ SECT_4K | SPI_NOR_DUAL_READ | SPI_NOR_QUAD_READ) },
+ { "is25wp128", INFO(0x9d7018, 0, 64 * 1024, 256,
+ SECT_4K | SPI_NOR_DUAL_READ | SPI_NOR_QUAD_READ) },
/* Macronix */
{ "mx25l512e", INFO(0xc22010, 0, 64 * 1024, 1, SECT_4K) },
@@ -1000,6 +1006,7 @@ static const struct flash_info spi_nor_ids[] = {
},
{ "w25q80", INFO(0xef5014, 0, 64 * 1024, 16, SECT_4K) },
{ "w25q80bl", INFO(0xef4014, 0, 64 * 1024, 16, SECT_4K) },
+ { "w25q80ew", INFO(0xef6014, 0, 64 * 1024, 16, SECT_4K) },
{ "w25q128", INFO(0xef4018, 0, 64 * 1024, 256, SECT_4K) },
{ "w25q256", INFO(0xef4019, 0, 64 * 1024, 512, SECT_4K) },
diff --git a/drivers/mtd/ubi/build.c b/drivers/mtd/ubi/build.c
index 68902b8..541c179 100644
--- a/drivers/mtd/ubi/build.c
+++ b/drivers/mtd/ubi/build.c
@@ -1137,6 +1137,9 @@ int ubi_detach_mtd_dev(int ubi_num, int anyway)
*/
get_device(&ubi->dev);
+#ifdef CONFIG_MTD_UBI_FASTMAP
+ cancel_work_sync(&ubi->fm_work);
+#endif
ubi_debugfs_exit_dev(ubi);
uif_close(ubi);
diff --git a/drivers/mtd/ubi/eba.c b/drivers/mtd/ubi/eba.c
index 388e46b..d0884bd 100644
--- a/drivers/mtd/ubi/eba.c
+++ b/drivers/mtd/ubi/eba.c
@@ -490,6 +490,82 @@ int ubi_eba_unmap_leb(struct ubi_device *ubi, struct ubi_volume *vol,
return err;
}
+#ifdef CONFIG_MTD_UBI_FASTMAP
+/**
+ * check_mapping - check and fixup a mapping
+ * @ubi: UBI device description object
+ * @vol: volume description object
+ * @lnum: logical eraseblock number
+ * @pnum: physical eraseblock number
+ *
+ * Checks whether a given mapping is valid. Fastmap cannot track LEB unmap
+ * operations, if such an operation is interrupted the mapping still looks
+ * good, but upon first read an ECC is reported to the upper layer.
+ * Normaly during the full-scan at attach time this is fixed, for Fastmap
+ * we have to deal with it while reading.
+ * If the PEB behind a LEB shows this symthom we change the mapping to
+ * %UBI_LEB_UNMAPPED and schedule the PEB for erasure.
+ *
+ * Returns 0 on success, negative error code in case of failure.
+ */
+static int check_mapping(struct ubi_device *ubi, struct ubi_volume *vol, int lnum,
+ int *pnum)
+{
+ int err;
+ struct ubi_vid_io_buf *vidb;
+
+ if (!ubi->fast_attach)
+ return 0;
+
+ vidb = ubi_alloc_vid_buf(ubi, GFP_NOFS);
+ if (!vidb)
+ return -ENOMEM;
+
+ err = ubi_io_read_vid_hdr(ubi, *pnum, vidb, 0);
+ if (err > 0 && err != UBI_IO_BITFLIPS) {
+ int torture = 0;
+
+ switch (err) {
+ case UBI_IO_FF:
+ case UBI_IO_FF_BITFLIPS:
+ case UBI_IO_BAD_HDR:
+ case UBI_IO_BAD_HDR_EBADMSG:
+ break;
+ default:
+ ubi_assert(0);
+ }
+
+ if (err == UBI_IO_BAD_HDR_EBADMSG || err == UBI_IO_FF_BITFLIPS)
+ torture = 1;
+
+ down_read(&ubi->fm_eba_sem);
+ vol->eba_tbl->entries[lnum].pnum = UBI_LEB_UNMAPPED;
+ up_read(&ubi->fm_eba_sem);
+ ubi_wl_put_peb(ubi, vol->vol_id, lnum, *pnum, torture);
+
+ *pnum = UBI_LEB_UNMAPPED;
+ } else if (err < 0) {
+ ubi_err(ubi, "unable to read VID header back from PEB %i: %i",
+ *pnum, err);
+
+ goto out_free;
+ }
+
+ err = 0;
+
+out_free:
+ ubi_free_vid_buf(vidb);
+
+ return err;
+}
+#else
+static int check_mapping(struct ubi_device *ubi, struct ubi_volume *vol, int lnum,
+ int *pnum)
+{
+ return 0;
+}
+#endif
+
/**
* ubi_eba_read_leb - read data.
* @ubi: UBI device description object
@@ -522,7 +598,13 @@ int ubi_eba_read_leb(struct ubi_device *ubi, struct ubi_volume *vol, int lnum,
return err;
pnum = vol->eba_tbl->entries[lnum].pnum;
- if (pnum < 0) {
+ if (pnum >= 0) {
+ err = check_mapping(ubi, vol, lnum, &pnum);
+ if (err < 0)
+ goto out_unlock;
+ }
+
+ if (pnum == UBI_LEB_UNMAPPED) {
/*
* The logical eraseblock is not mapped, fill the whole buffer
* with 0xFF bytes. The exception is static volumes for which
@@ -931,6 +1013,12 @@ int ubi_eba_write_leb(struct ubi_device *ubi, struct ubi_volume *vol, int lnum,
pnum = vol->eba_tbl->entries[lnum].pnum;
if (pnum >= 0) {
+ err = check_mapping(ubi, vol, lnum, &pnum);
+ if (err < 0)
+ goto out;
+ }
+
+ if (pnum >= 0) {
dbg_eba("write %d bytes at offset %d of LEB %d:%d, PEB %d",
len, offset, vol_id, lnum, pnum);
diff --git a/drivers/mtd/ubi/wl.c b/drivers/mtd/ubi/wl.c
index 668b462..a8f74d9 100644
--- a/drivers/mtd/ubi/wl.c
+++ b/drivers/mtd/ubi/wl.c
@@ -1505,6 +1505,7 @@ int ubi_thread(void *u)
}
dbg_wl("background thread \"%s\" is killed", ubi->bgt_name);
+ ubi->thread_enabled = 0;
return 0;
}
@@ -1514,9 +1515,6 @@ int ubi_thread(void *u)
*/
static void shutdown_work(struct ubi_device *ubi)
{
-#ifdef CONFIG_MTD_UBI_FASTMAP
- flush_work(&ubi->fm_work);
-#endif
while (!list_empty(&ubi->works)) {
struct ubi_work *wrk;
@@ -1617,8 +1615,10 @@ int ubi_wl_init(struct ubi_device *ubi, struct ubi_attach_info *ai)
cond_resched();
e = kmem_cache_alloc(ubi_wl_entry_slab, GFP_KERNEL);
- if (!e)
+ if (!e) {
+ err = -ENOMEM;
goto out_free;
+ }
e->pnum = aeb->pnum;
e->ec = aeb->ec;
@@ -1637,8 +1637,10 @@ int ubi_wl_init(struct ubi_device *ubi, struct ubi_attach_info *ai)
cond_resched();
e = kmem_cache_alloc(ubi_wl_entry_slab, GFP_KERNEL);
- if (!e)
+ if (!e) {
+ err = -ENOMEM;
goto out_free;
+ }
e->pnum = aeb->pnum;
e->ec = aeb->ec;
diff --git a/drivers/net/appletalk/ipddp.c b/drivers/net/appletalk/ipddp.c
index 2e464965..4e98e5a 100644
--- a/drivers/net/appletalk/ipddp.c
+++ b/drivers/net/appletalk/ipddp.c
@@ -284,8 +284,12 @@ static int ipddp_ioctl(struct net_device *dev, struct ifreq *ifr, int cmd)
case SIOCFINDIPDDPRT:
spin_lock_bh(&ipddp_route_lock);
rp = __ipddp_find_route(&rcp);
- if (rp)
- memcpy(&rcp2, rp, sizeof(rcp2));
+ if (rp) {
+ memset(&rcp2, 0, sizeof(rcp2));
+ rcp2.ip = rp->ip;
+ rcp2.at = rp->at;
+ rcp2.flags = rp->flags;
+ }
spin_unlock_bh(&ipddp_route_lock);
if (rp) {
diff --git a/drivers/net/bonding/bond_3ad.c b/drivers/net/bonding/bond_3ad.c
index 6dcc42d..1e2ee97 100644
--- a/drivers/net/bonding/bond_3ad.c
+++ b/drivers/net/bonding/bond_3ad.c
@@ -2050,6 +2050,9 @@ void bond_3ad_unbind_slave(struct slave *slave)
aggregator->aggregator_identifier);
/* Tell the partner that this port is not suitable for aggregation */
+ port->actor_oper_port_state &= ~AD_STATE_SYNCHRONIZATION;
+ port->actor_oper_port_state &= ~AD_STATE_COLLECTING;
+ port->actor_oper_port_state &= ~AD_STATE_DISTRIBUTING;
port->actor_oper_port_state &= ~AD_STATE_AGGREGATION;
__update_lacpdu_from_port(port);
ad_lacpdu_send(port);
diff --git a/drivers/net/bonding/bond_alb.c b/drivers/net/bonding/bond_alb.c
index 551f0f8..91d8a48 100644
--- a/drivers/net/bonding/bond_alb.c
+++ b/drivers/net/bonding/bond_alb.c
@@ -450,7 +450,7 @@ static void rlb_update_client(struct rlb_client_info *client_info)
{
int i;
- if (!client_info->slave)
+ if (!client_info->slave || !is_valid_ether_addr(client_info->mac_dst))
return;
for (i = 0; i < RLB_ARP_BURST_SIZE; i++) {
@@ -944,6 +944,10 @@ static void alb_send_lp_vid(struct slave *slave, u8 mac_addr[],
skb->priority = TC_PRIO_CONTROL;
skb->dev = slave->dev;
+ netdev_dbg(slave->bond->dev,
+ "Send learning packet: dev %s mac %pM vlan %d\n",
+ slave->dev->name, mac_addr, vid);
+
if (vid)
__vlan_hwaccel_put_tag(skb, vlan_proto, vid);
@@ -966,14 +970,13 @@ static void alb_send_learning_packets(struct slave *slave, u8 mac_addr[],
*/
rcu_read_lock();
netdev_for_each_all_upper_dev_rcu(bond->dev, upper, iter) {
- if (is_vlan_dev(upper) && vlan_get_encap_level(upper) == 0) {
- if (strict_match &&
- ether_addr_equal_64bits(mac_addr,
- upper->dev_addr)) {
+ if (is_vlan_dev(upper) &&
+ bond->nest_level == vlan_get_encap_level(upper) - 1) {
+ if (upper->addr_assign_type == NET_ADDR_STOLEN) {
alb_send_lp_vid(slave, mac_addr,
vlan_dev_vlan_proto(upper),
vlan_dev_vlan_id(upper));
- } else if (!strict_match) {
+ } else {
alb_send_lp_vid(slave, upper->dev_addr,
vlan_dev_vlan_proto(upper),
vlan_dev_vlan_id(upper));
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index 513457a..389d1db 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -216,6 +216,7 @@ static struct rtnl_link_stats64 *bond_get_stats(struct net_device *bond_dev,
static void bond_slave_arr_handler(struct work_struct *work);
static bool bond_time_in_interval(struct bonding *bond, unsigned long last_act,
int mod);
+static void bond_netdev_notify_work(struct work_struct *work);
/*---------------------------- General routines -----------------------------*/
@@ -384,20 +385,15 @@ static int bond_update_speed_duplex(struct slave *slave)
slave->duplex = DUPLEX_UNKNOWN;
res = __ethtool_get_link_ksettings(slave_dev, &ecmd);
- if (res < 0) {
- slave->link = BOND_LINK_DOWN;
+ if (res < 0)
return 1;
- }
- if (ecmd.base.speed == 0 || ecmd.base.speed == ((__u32)-1)) {
- slave->link = BOND_LINK_DOWN;
+ if (ecmd.base.speed == 0 || ecmd.base.speed == ((__u32)-1))
return 1;
- }
switch (ecmd.base.duplex) {
case DUPLEX_FULL:
case DUPLEX_HALF:
break;
default:
- slave->link = BOND_LINK_DOWN;
return 1;
}
@@ -1255,6 +1251,8 @@ static struct slave *bond_alloc_slave(struct bonding *bond)
return NULL;
}
}
+ INIT_DELAYED_WORK(&slave->notify_work, bond_netdev_notify_work);
+
return slave;
}
@@ -1262,6 +1260,7 @@ static void bond_free_slave(struct slave *slave)
{
struct bonding *bond = bond_get_bond_by_slave(slave);
+ cancel_delayed_work_sync(&slave->notify_work);
if (BOND_MODE(bond) == BOND_MODE_8023AD)
kfree(SLAVE_AD_INFO(slave));
@@ -1283,39 +1282,26 @@ static void bond_fill_ifslave(struct slave *slave, struct ifslave *info)
info->link_failure_count = slave->link_failure_count;
}
-static void bond_netdev_notify(struct net_device *dev,
- struct netdev_bonding_info *info)
-{
- rtnl_lock();
- netdev_bonding_info_change(dev, info);
- rtnl_unlock();
-}
-
static void bond_netdev_notify_work(struct work_struct *_work)
{
- struct netdev_notify_work *w =
- container_of(_work, struct netdev_notify_work, work.work);
+ struct slave *slave = container_of(_work, struct slave,
+ notify_work.work);
- bond_netdev_notify(w->dev, &w->bonding_info);
- dev_put(w->dev);
- kfree(w);
+ if (rtnl_trylock()) {
+ struct netdev_bonding_info binfo;
+
+ bond_fill_ifslave(slave, &binfo.slave);
+ bond_fill_ifbond(slave->bond, &binfo.master);
+ netdev_bonding_info_change(slave->dev, &binfo);
+ rtnl_unlock();
+ } else {
+ queue_delayed_work(slave->bond->wq, &slave->notify_work, 1);
+ }
}
void bond_queue_slave_event(struct slave *slave)
{
- struct bonding *bond = slave->bond;
- struct netdev_notify_work *nnw = kzalloc(sizeof(*nnw), GFP_ATOMIC);
-
- if (!nnw)
- return;
-
- dev_hold(slave->dev);
- nnw->dev = slave->dev;
- bond_fill_ifslave(slave, &nnw->bonding_info.slave);
- bond_fill_ifbond(bond, &nnw->bonding_info.master);
- INIT_DELAYED_WORK(&nnw->work, bond_netdev_notify_work);
-
- queue_delayed_work(slave->bond->wq, &nnw->work, 0);
+ queue_delayed_work(slave->bond->wq, &slave->notify_work, 0);
}
void bond_lower_state_changed(struct slave *slave)
@@ -1536,7 +1522,9 @@ int bond_enslave(struct net_device *bond_dev, struct net_device *slave_dev)
new_slave->delay = 0;
new_slave->link_failure_count = 0;
- bond_update_speed_duplex(new_slave);
+ if (bond_update_speed_duplex(new_slave) &&
+ bond_needs_speed_duplex(bond))
+ new_slave->link = BOND_LINK_DOWN;
new_slave->last_rx = jiffies -
(msecs_to_jiffies(bond->params.arp_interval) + 1);
@@ -1654,8 +1642,7 @@ int bond_enslave(struct net_device *bond_dev, struct net_device *slave_dev)
} /* switch(bond_mode) */
#ifdef CONFIG_NET_POLL_CONTROLLER
- slave_dev->npinfo = bond->dev->npinfo;
- if (slave_dev->npinfo) {
+ if (bond->dev->npinfo) {
if (slave_enable_netpoll(new_slave)) {
netdev_info(bond_dev, "master_dev is using netpoll, but new slave device does not support netpoll\n");
res = -EBUSY;
@@ -1686,6 +1673,8 @@ int bond_enslave(struct net_device *bond_dev, struct net_device *slave_dev)
goto err_upper_unlink;
}
+ bond->nest_level = dev_get_nest_level(bond_dev) + 1;
+
/* If the mode uses primary, then the following is handled by
* bond_change_active_slave().
*/
@@ -1733,6 +1722,7 @@ int bond_enslave(struct net_device *bond_dev, struct net_device *slave_dev)
if (bond_mode_uses_xmit_hash(bond))
bond_update_slave_arr(bond, NULL);
+
netdev_info(bond_dev, "Enslaving %s as %s interface with %s link\n",
slave_dev->name,
bond_is_active_slave(new_slave) ? "an active" : "a backup",
@@ -2139,7 +2129,15 @@ static void bond_miimon_commit(struct bonding *bond)
continue;
case BOND_LINK_UP:
- bond_update_speed_duplex(slave);
+ if (bond_update_speed_duplex(slave) &&
+ bond_needs_speed_duplex(bond)) {
+ slave->link = BOND_LINK_DOWN;
+ if (net_ratelimit())
+ netdev_warn(bond->dev,
+ "failed to get link speed/duplex for %s\n",
+ slave->dev->name);
+ continue;
+ }
bond_set_slave_link_state(slave, BOND_LINK_UP,
BOND_SLAVE_NOTIFY_NOW);
slave->last_link_up = jiffies;
@@ -3354,6 +3352,13 @@ static void bond_fold_stats(struct rtnl_link_stats64 *_res,
}
}
+static int bond_get_nest_level(struct net_device *bond_dev)
+{
+ struct bonding *bond = netdev_priv(bond_dev);
+
+ return bond->nest_level;
+}
+
static struct rtnl_link_stats64 *bond_get_stats(struct net_device *bond_dev,
struct rtnl_link_stats64 *stats)
{
@@ -3362,7 +3367,7 @@ static struct rtnl_link_stats64 *bond_get_stats(struct net_device *bond_dev,
struct list_head *iter;
struct slave *slave;
- spin_lock(&bond->stats_lock);
+ spin_lock_nested(&bond->stats_lock, bond_get_nest_level(bond_dev));
memcpy(stats, &bond->bond_stats, sizeof(*stats));
rcu_read_lock();
@@ -4158,6 +4163,7 @@ static const struct net_device_ops bond_netdev_ops = {
.ndo_neigh_setup = bond_neigh_setup,
.ndo_vlan_rx_add_vid = bond_vlan_rx_add_vid,
.ndo_vlan_rx_kill_vid = bond_vlan_rx_kill_vid,
+ .ndo_get_lock_subclass = bond_get_nest_level,
#ifdef CONFIG_NET_POLL_CONTROLLER
.ndo_netpoll_setup = bond_netpoll_setup,
.ndo_netpoll_cleanup = bond_netpoll_cleanup,
@@ -4650,6 +4656,7 @@ static int bond_init(struct net_device *bond_dev)
if (!bond->wq)
return -ENOMEM;
+ bond->nest_level = SINGLE_DEPTH_NESTING;
netdev_lockdep_set_classes(bond_dev);
list_add_tail(&bond->bond_list, &bn->dev_list);
diff --git a/drivers/net/bonding/bond_netlink.c b/drivers/net/bonding/bond_netlink.c
index b8df0f5..3f320f4 100644
--- a/drivers/net/bonding/bond_netlink.c
+++ b/drivers/net/bonding/bond_netlink.c
@@ -628,8 +628,7 @@ static int bond_fill_info(struct sk_buff *skb,
goto nla_put_failure;
if (nla_put(skb, IFLA_BOND_AD_ACTOR_SYSTEM,
- sizeof(bond->params.ad_actor_system),
- &bond->params.ad_actor_system))
+ ETH_ALEN, &bond->params.ad_actor_system))
goto nla_put_failure;
}
if (!bond_3ad_get_active_agg_info(bond, &info)) {
diff --git a/drivers/net/bonding/bond_options.c b/drivers/net/bonding/bond_options.c
index 577e57c..473da3b 100644
--- a/drivers/net/bonding/bond_options.c
+++ b/drivers/net/bonding/bond_options.c
@@ -1114,6 +1114,7 @@ static int bond_option_primary_set(struct bonding *bond,
slave->dev->name);
rcu_assign_pointer(bond->primary_slave, slave);
strcpy(bond->params.primary, slave->dev->name);
+ bond->force_primary = true;
bond_select_active_slave(bond);
goto out;
}
diff --git a/drivers/net/can/dev.c b/drivers/net/can/dev.c
index 8d6208c..ff3d9fc 100644
--- a/drivers/net/can/dev.c
+++ b/drivers/net/can/dev.c
@@ -453,6 +453,34 @@ void can_put_echo_skb(struct sk_buff *skb, struct net_device *dev,
}
EXPORT_SYMBOL_GPL(can_put_echo_skb);
+struct sk_buff *__can_get_echo_skb(struct net_device *dev, unsigned int idx, u8 *len_ptr)
+{
+ struct can_priv *priv = netdev_priv(dev);
+ struct sk_buff *skb = priv->echo_skb[idx];
+ struct canfd_frame *cf;
+
+ if (idx >= priv->echo_skb_max) {
+ netdev_err(dev, "%s: BUG! Trying to access can_priv::echo_skb out of bounds (%u/max %u)\n",
+ __func__, idx, priv->echo_skb_max);
+ return NULL;
+ }
+
+ if (!skb) {
+ netdev_err(dev, "%s: BUG! Trying to echo non existing skb: can_priv::echo_skb[%u]\n",
+ __func__, idx);
+ return NULL;
+ }
+
+ /* Using "struct canfd_frame::len" for the frame
+ * length is supported on both CAN and CANFD frames.
+ */
+ cf = (struct canfd_frame *)skb->data;
+ *len_ptr = cf->len;
+ priv->echo_skb[idx] = NULL;
+
+ return skb;
+}
+
/*
* Get the skb from the stack and loop it back locally
*
@@ -462,22 +490,16 @@ EXPORT_SYMBOL_GPL(can_put_echo_skb);
*/
unsigned int can_get_echo_skb(struct net_device *dev, unsigned int idx)
{
- struct can_priv *priv = netdev_priv(dev);
+ struct sk_buff *skb;
+ u8 len;
- BUG_ON(idx >= priv->echo_skb_max);
+ skb = __can_get_echo_skb(dev, idx, &len);
+ if (!skb)
+ return 0;
- if (priv->echo_skb[idx]) {
- struct sk_buff *skb = priv->echo_skb[idx];
- struct can_frame *cf = (struct can_frame *)skb->data;
- u8 dlc = cf->can_dlc;
+ netif_rx(skb);
- netif_rx(priv->echo_skb[idx]);
- priv->echo_skb[idx] = NULL;
-
- return dlc;
- }
-
- return 0;
+ return len;
}
EXPORT_SYMBOL_GPL(can_get_echo_skb);
diff --git a/drivers/net/can/mscan/mpc5xxx_can.c b/drivers/net/can/mscan/mpc5xxx_can.c
index c7427bd..2949a38 100644
--- a/drivers/net/can/mscan/mpc5xxx_can.c
+++ b/drivers/net/can/mscan/mpc5xxx_can.c
@@ -86,6 +86,11 @@ static u32 mpc52xx_can_get_clock(struct platform_device *ofdev,
return 0;
}
cdm = of_iomap(np_cdm, 0);
+ if (!cdm) {
+ of_node_put(np_cdm);
+ dev_err(&ofdev->dev, "can't map clock node!\n");
+ return 0;
+ }
if (in_8(&cdm->ipb_clk_sel) & 0x1)
freq *= 2;
diff --git a/drivers/net/can/rcar/rcar_can.c b/drivers/net/can/rcar/rcar_can.c
index 788459f..9b2c3bd 100644
--- a/drivers/net/can/rcar/rcar_can.c
+++ b/drivers/net/can/rcar/rcar_can.c
@@ -24,6 +24,9 @@
#define RCAR_CAN_DRV_NAME "rcar_can"
+#define RCAR_SUPPORTED_CLOCKS (BIT(CLKR_CLKP1) | BIT(CLKR_CLKP2) | \
+ BIT(CLKR_CLKEXT))
+
/* Mailbox configuration:
* mailbox 60 - 63 - Rx FIFO mailboxes
* mailbox 56 - 59 - Tx FIFO mailboxes
@@ -789,7 +792,7 @@ static int rcar_can_probe(struct platform_device *pdev)
goto fail_clk;
}
- if (clock_select >= ARRAY_SIZE(clock_names)) {
+ if (!(BIT(clock_select) & RCAR_SUPPORTED_CLOCKS)) {
err = -EINVAL;
dev_err(&pdev->dev, "invalid CAN clock selected\n");
goto fail_clk;
diff --git a/drivers/net/can/usb/ems_usb.c b/drivers/net/can/usb/ems_usb.c
index b003582..d0846ae 100644
--- a/drivers/net/can/usb/ems_usb.c
+++ b/drivers/net/can/usb/ems_usb.c
@@ -1071,6 +1071,7 @@ static void ems_usb_disconnect(struct usb_interface *intf)
usb_free_urb(dev->intr_urb);
kfree(dev->intr_in_buffer);
+ kfree(dev->tx_msg_buffer);
}
}
diff --git a/drivers/net/can/usb/kvaser_usb.c b/drivers/net/can/usb/kvaser_usb.c
index c9d61a6..3a75352 100644
--- a/drivers/net/can/usb/kvaser_usb.c
+++ b/drivers/net/can/usb/kvaser_usb.c
@@ -1179,7 +1179,7 @@ static void kvaser_usb_rx_can_msg(const struct kvaser_usb *dev,
skb = alloc_can_skb(priv->netdev, &cf);
if (!skb) {
- stats->tx_dropped++;
+ stats->rx_dropped++;
return;
}
diff --git a/drivers/net/can/xilinx_can.c b/drivers/net/can/xilinx_can.c
index c71a035..e680bab 100644
--- a/drivers/net/can/xilinx_can.c
+++ b/drivers/net/can/xilinx_can.c
@@ -2,6 +2,7 @@
*
* Copyright (C) 2012 - 2014 Xilinx, Inc.
* Copyright (C) 2009 PetaLogix. All rights reserved.
+ * Copyright (C) 2017 Sandvik Mining and Construction Oy
*
* Description:
* This driver is developed for Axi CAN IP and for Zynq CANPS Controller.
@@ -25,8 +26,10 @@
#include <linux/module.h>
#include <linux/netdevice.h>
#include <linux/of.h>
+#include <linux/of_device.h>
#include <linux/platform_device.h>
#include <linux/skbuff.h>
+#include <linux/spinlock.h>
#include <linux/string.h>
#include <linux/types.h>
#include <linux/can/dev.h>
@@ -101,7 +104,7 @@ enum xcan_reg {
#define XCAN_INTR_ALL (XCAN_IXR_TXOK_MASK | XCAN_IXR_BSOFF_MASK |\
XCAN_IXR_WKUP_MASK | XCAN_IXR_SLP_MASK | \
XCAN_IXR_RXNEMP_MASK | XCAN_IXR_ERROR_MASK | \
- XCAN_IXR_ARBLST_MASK | XCAN_IXR_RXOK_MASK)
+ XCAN_IXR_RXOFLW_MASK | XCAN_IXR_ARBLST_MASK)
/* CAN register bit shift - XCAN_<REG>_<BIT>_SHIFT */
#define XCAN_BTR_SJW_SHIFT 7 /* Synchronous jump width */
@@ -118,6 +121,7 @@ enum xcan_reg {
/**
* struct xcan_priv - This definition define CAN driver instance
* @can: CAN private data structure.
+ * @tx_lock: Lock for synchronizing TX interrupt handling
* @tx_head: Tx CAN packets ready to send on the queue
* @tx_tail: Tx CAN packets successfully sended on the queue
* @tx_max: Maximum number packets the driver can send
@@ -132,6 +136,7 @@ enum xcan_reg {
*/
struct xcan_priv {
struct can_priv can;
+ spinlock_t tx_lock;
unsigned int tx_head;
unsigned int tx_tail;
unsigned int tx_max;
@@ -159,6 +164,11 @@ static const struct can_bittiming_const xcan_bittiming_const = {
.brp_inc = 1,
};
+#define XCAN_CAP_WATERMARK 0x0001
+struct xcan_devtype_data {
+ unsigned int caps;
+};
+
/**
* xcan_write_reg_le - Write a value to the device register little endian
* @priv: Driver private data structure
@@ -238,6 +248,10 @@ static int set_reset_mode(struct net_device *ndev)
usleep_range(500, 10000);
}
+ /* reset clears FIFOs */
+ priv->tx_head = 0;
+ priv->tx_tail = 0;
+
return 0;
}
@@ -392,6 +406,7 @@ static int xcan_start_xmit(struct sk_buff *skb, struct net_device *ndev)
struct net_device_stats *stats = &ndev->stats;
struct can_frame *cf = (struct can_frame *)skb->data;
u32 id, dlc, data[2] = {0, 0};
+ unsigned long flags;
if (can_dropped_invalid_skb(ndev, skb))
return NETDEV_TX_OK;
@@ -439,6 +454,9 @@ static int xcan_start_xmit(struct sk_buff *skb, struct net_device *ndev)
data[1] = be32_to_cpup((__be32 *)(cf->data + 4));
can_put_echo_skb(skb, ndev, priv->tx_head % priv->tx_max);
+
+ spin_lock_irqsave(&priv->tx_lock, flags);
+
priv->tx_head++;
/* Write the Frame to Xilinx CAN TX FIFO */
@@ -454,10 +472,16 @@ static int xcan_start_xmit(struct sk_buff *skb, struct net_device *ndev)
stats->tx_bytes += cf->can_dlc;
}
+ /* Clear TX-FIFO-empty interrupt for xcan_tx_interrupt() */
+ if (priv->tx_max > 1)
+ priv->write_reg(priv, XCAN_ICR_OFFSET, XCAN_IXR_TXFEMP_MASK);
+
/* Check if the TX buffer is full */
if ((priv->tx_head - priv->tx_tail) == priv->tx_max)
netif_stop_queue(ndev);
+ spin_unlock_irqrestore(&priv->tx_lock, flags);
+
return NETDEV_TX_OK;
}
@@ -530,6 +554,123 @@ static int xcan_rx(struct net_device *ndev)
}
/**
+ * xcan_current_error_state - Get current error state from HW
+ * @ndev: Pointer to net_device structure
+ *
+ * Checks the current CAN error state from the HW. Note that this
+ * only checks for ERROR_PASSIVE and ERROR_WARNING.
+ *
+ * Return:
+ * ERROR_PASSIVE or ERROR_WARNING if either is active, ERROR_ACTIVE
+ * otherwise.
+ */
+static enum can_state xcan_current_error_state(struct net_device *ndev)
+{
+ struct xcan_priv *priv = netdev_priv(ndev);
+ u32 status = priv->read_reg(priv, XCAN_SR_OFFSET);
+
+ if ((status & XCAN_SR_ESTAT_MASK) == XCAN_SR_ESTAT_MASK)
+ return CAN_STATE_ERROR_PASSIVE;
+ else if (status & XCAN_SR_ERRWRN_MASK)
+ return CAN_STATE_ERROR_WARNING;
+ else
+ return CAN_STATE_ERROR_ACTIVE;
+}
+
+/**
+ * xcan_set_error_state - Set new CAN error state
+ * @ndev: Pointer to net_device structure
+ * @new_state: The new CAN state to be set
+ * @cf: Error frame to be populated or NULL
+ *
+ * Set new CAN error state for the device, updating statistics and
+ * populating the error frame if given.
+ */
+static void xcan_set_error_state(struct net_device *ndev,
+ enum can_state new_state,
+ struct can_frame *cf)
+{
+ struct xcan_priv *priv = netdev_priv(ndev);
+ u32 ecr = priv->read_reg(priv, XCAN_ECR_OFFSET);
+ u32 txerr = ecr & XCAN_ECR_TEC_MASK;
+ u32 rxerr = (ecr & XCAN_ECR_REC_MASK) >> XCAN_ESR_REC_SHIFT;
+
+ priv->can.state = new_state;
+
+ if (cf) {
+ cf->can_id |= CAN_ERR_CRTL;
+ cf->data[6] = txerr;
+ cf->data[7] = rxerr;
+ }
+
+ switch (new_state) {
+ case CAN_STATE_ERROR_PASSIVE:
+ priv->can.can_stats.error_passive++;
+ if (cf)
+ cf->data[1] = (rxerr > 127) ?
+ CAN_ERR_CRTL_RX_PASSIVE :
+ CAN_ERR_CRTL_TX_PASSIVE;
+ break;
+ case CAN_STATE_ERROR_WARNING:
+ priv->can.can_stats.error_warning++;
+ if (cf)
+ cf->data[1] |= (txerr > rxerr) ?
+ CAN_ERR_CRTL_TX_WARNING :
+ CAN_ERR_CRTL_RX_WARNING;
+ break;
+ case CAN_STATE_ERROR_ACTIVE:
+ if (cf)
+ cf->data[1] |= CAN_ERR_CRTL_ACTIVE;
+ break;
+ default:
+ /* non-ERROR states are handled elsewhere */
+ WARN_ON(1);
+ break;
+ }
+}
+
+/**
+ * xcan_update_error_state_after_rxtx - Update CAN error state after RX/TX
+ * @ndev: Pointer to net_device structure
+ *
+ * If the device is in a ERROR-WARNING or ERROR-PASSIVE state, check if
+ * the performed RX/TX has caused it to drop to a lesser state and set
+ * the interface state accordingly.
+ */
+static void xcan_update_error_state_after_rxtx(struct net_device *ndev)
+{
+ struct xcan_priv *priv = netdev_priv(ndev);
+ enum can_state old_state = priv->can.state;
+ enum can_state new_state;
+
+ /* changing error state due to successful frame RX/TX can only
+ * occur from these states
+ */
+ if (old_state != CAN_STATE_ERROR_WARNING &&
+ old_state != CAN_STATE_ERROR_PASSIVE)
+ return;
+
+ new_state = xcan_current_error_state(ndev);
+
+ if (new_state != old_state) {
+ struct sk_buff *skb;
+ struct can_frame *cf;
+
+ skb = alloc_can_err_skb(ndev, &cf);
+
+ xcan_set_error_state(ndev, new_state, skb ? cf : NULL);
+
+ if (skb) {
+ struct net_device_stats *stats = &ndev->stats;
+
+ stats->rx_packets++;
+ stats->rx_bytes += cf->can_dlc;
+ netif_rx(skb);
+ }
+ }
+}
+
+/**
* xcan_err_interrupt - error frame Isr
* @ndev: net_device pointer
* @isr: interrupt status register value
@@ -544,16 +685,12 @@ static void xcan_err_interrupt(struct net_device *ndev, u32 isr)
struct net_device_stats *stats = &ndev->stats;
struct can_frame *cf;
struct sk_buff *skb;
- u32 err_status, status, txerr = 0, rxerr = 0;
+ u32 err_status;
skb = alloc_can_err_skb(ndev, &cf);
err_status = priv->read_reg(priv, XCAN_ESR_OFFSET);
priv->write_reg(priv, XCAN_ESR_OFFSET, err_status);
- txerr = priv->read_reg(priv, XCAN_ECR_OFFSET) & XCAN_ECR_TEC_MASK;
- rxerr = ((priv->read_reg(priv, XCAN_ECR_OFFSET) &
- XCAN_ECR_REC_MASK) >> XCAN_ESR_REC_SHIFT);
- status = priv->read_reg(priv, XCAN_SR_OFFSET);
if (isr & XCAN_IXR_BSOFF_MASK) {
priv->can.state = CAN_STATE_BUS_OFF;
@@ -563,28 +700,10 @@ static void xcan_err_interrupt(struct net_device *ndev, u32 isr)
can_bus_off(ndev);
if (skb)
cf->can_id |= CAN_ERR_BUSOFF;
- } else if ((status & XCAN_SR_ESTAT_MASK) == XCAN_SR_ESTAT_MASK) {
- priv->can.state = CAN_STATE_ERROR_PASSIVE;
- priv->can.can_stats.error_passive++;
- if (skb) {
- cf->can_id |= CAN_ERR_CRTL;
- cf->data[1] = (rxerr > 127) ?
- CAN_ERR_CRTL_RX_PASSIVE :
- CAN_ERR_CRTL_TX_PASSIVE;
- cf->data[6] = txerr;
- cf->data[7] = rxerr;
- }
- } else if (status & XCAN_SR_ERRWRN_MASK) {
- priv->can.state = CAN_STATE_ERROR_WARNING;
- priv->can.can_stats.error_warning++;
- if (skb) {
- cf->can_id |= CAN_ERR_CRTL;
- cf->data[1] |= (txerr > rxerr) ?
- CAN_ERR_CRTL_TX_WARNING :
- CAN_ERR_CRTL_RX_WARNING;
- cf->data[6] = txerr;
- cf->data[7] = rxerr;
- }
+ } else {
+ enum can_state new_state = xcan_current_error_state(ndev);
+
+ xcan_set_error_state(ndev, new_state, skb ? cf : NULL);
}
/* Check for Arbitration lost interrupt */
@@ -600,7 +719,6 @@ static void xcan_err_interrupt(struct net_device *ndev, u32 isr)
if (isr & XCAN_IXR_RXOFLW_MASK) {
stats->rx_over_errors++;
stats->rx_errors++;
- priv->write_reg(priv, XCAN_SRR_OFFSET, XCAN_SRR_RESET_MASK);
if (skb) {
cf->can_id |= CAN_ERR_CRTL;
cf->data[1] |= CAN_ERR_CRTL_RX_OVERFLOW;
@@ -709,26 +827,20 @@ static int xcan_rx_poll(struct napi_struct *napi, int quota)
isr = priv->read_reg(priv, XCAN_ISR_OFFSET);
while ((isr & XCAN_IXR_RXNEMP_MASK) && (work_done < quota)) {
- if (isr & XCAN_IXR_RXOK_MASK) {
- priv->write_reg(priv, XCAN_ICR_OFFSET,
- XCAN_IXR_RXOK_MASK);
- work_done += xcan_rx(ndev);
- } else {
- priv->write_reg(priv, XCAN_ICR_OFFSET,
- XCAN_IXR_RXNEMP_MASK);
- break;
- }
+ work_done += xcan_rx(ndev);
priv->write_reg(priv, XCAN_ICR_OFFSET, XCAN_IXR_RXNEMP_MASK);
isr = priv->read_reg(priv, XCAN_ISR_OFFSET);
}
- if (work_done)
+ if (work_done) {
can_led_event(ndev, CAN_LED_EVENT_RX);
+ xcan_update_error_state_after_rxtx(ndev);
+ }
if (work_done < quota) {
napi_complete(napi);
ier = priv->read_reg(priv, XCAN_IER_OFFSET);
- ier |= (XCAN_IXR_RXOK_MASK | XCAN_IXR_RXNEMP_MASK);
+ ier |= XCAN_IXR_RXNEMP_MASK;
priv->write_reg(priv, XCAN_IER_OFFSET, ier);
}
return work_done;
@@ -743,18 +855,71 @@ static void xcan_tx_interrupt(struct net_device *ndev, u32 isr)
{
struct xcan_priv *priv = netdev_priv(ndev);
struct net_device_stats *stats = &ndev->stats;
+ unsigned int frames_in_fifo;
+ int frames_sent = 1; /* TXOK => at least 1 frame was sent */
+ unsigned long flags;
+ int retries = 0;
- while ((priv->tx_head - priv->tx_tail > 0) &&
- (isr & XCAN_IXR_TXOK_MASK)) {
+ /* Synchronize with xmit as we need to know the exact number
+ * of frames in the FIFO to stay in sync due to the TXFEMP
+ * handling.
+ * This also prevents a race between netif_wake_queue() and
+ * netif_stop_queue().
+ */
+ spin_lock_irqsave(&priv->tx_lock, flags);
+
+ frames_in_fifo = priv->tx_head - priv->tx_tail;
+
+ if (WARN_ON_ONCE(frames_in_fifo == 0)) {
+ /* clear TXOK anyway to avoid getting back here */
priv->write_reg(priv, XCAN_ICR_OFFSET, XCAN_IXR_TXOK_MASK);
+ spin_unlock_irqrestore(&priv->tx_lock, flags);
+ return;
+ }
+
+ /* Check if 2 frames were sent (TXOK only means that at least 1
+ * frame was sent).
+ */
+ if (frames_in_fifo > 1) {
+ WARN_ON(frames_in_fifo > priv->tx_max);
+
+ /* Synchronize TXOK and isr so that after the loop:
+ * (1) isr variable is up-to-date at least up to TXOK clear
+ * time. This avoids us clearing a TXOK of a second frame
+ * but not noticing that the FIFO is now empty and thus
+ * marking only a single frame as sent.
+ * (2) No TXOK is left. Having one could mean leaving a
+ * stray TXOK as we might process the associated frame
+ * via TXFEMP handling as we read TXFEMP *after* TXOK
+ * clear to satisfy (1).
+ */
+ while ((isr & XCAN_IXR_TXOK_MASK) && !WARN_ON(++retries == 100)) {
+ priv->write_reg(priv, XCAN_ICR_OFFSET, XCAN_IXR_TXOK_MASK);
+ isr = priv->read_reg(priv, XCAN_ISR_OFFSET);
+ }
+
+ if (isr & XCAN_IXR_TXFEMP_MASK) {
+ /* nothing in FIFO anymore */
+ frames_sent = frames_in_fifo;
+ }
+ } else {
+ /* single frame in fifo, just clear TXOK */
+ priv->write_reg(priv, XCAN_ICR_OFFSET, XCAN_IXR_TXOK_MASK);
+ }
+
+ while (frames_sent--) {
can_get_echo_skb(ndev, priv->tx_tail %
priv->tx_max);
priv->tx_tail++;
stats->tx_packets++;
- isr = priv->read_reg(priv, XCAN_ISR_OFFSET);
}
- can_led_event(ndev, CAN_LED_EVENT_TX);
+
netif_wake_queue(ndev);
+
+ spin_unlock_irqrestore(&priv->tx_lock, flags);
+
+ can_led_event(ndev, CAN_LED_EVENT_TX);
+ xcan_update_error_state_after_rxtx(ndev);
}
/**
@@ -773,6 +938,7 @@ static irqreturn_t xcan_interrupt(int irq, void *dev_id)
struct net_device *ndev = (struct net_device *)dev_id;
struct xcan_priv *priv = netdev_priv(ndev);
u32 isr, ier;
+ u32 isr_errors;
/* Get the interrupt status from Xilinx CAN */
isr = priv->read_reg(priv, XCAN_ISR_OFFSET);
@@ -791,18 +957,17 @@ static irqreturn_t xcan_interrupt(int irq, void *dev_id)
xcan_tx_interrupt(ndev, isr);
/* Check for the type of error interrupt and Processing it */
- if (isr & (XCAN_IXR_ERROR_MASK | XCAN_IXR_RXOFLW_MASK |
- XCAN_IXR_BSOFF_MASK | XCAN_IXR_ARBLST_MASK)) {
- priv->write_reg(priv, XCAN_ICR_OFFSET, (XCAN_IXR_ERROR_MASK |
- XCAN_IXR_RXOFLW_MASK | XCAN_IXR_BSOFF_MASK |
- XCAN_IXR_ARBLST_MASK));
+ isr_errors = isr & (XCAN_IXR_ERROR_MASK | XCAN_IXR_RXOFLW_MASK |
+ XCAN_IXR_BSOFF_MASK | XCAN_IXR_ARBLST_MASK);
+ if (isr_errors) {
+ priv->write_reg(priv, XCAN_ICR_OFFSET, isr_errors);
xcan_err_interrupt(ndev, isr);
}
/* Check for the type of receive interrupt and Processing it */
- if (isr & (XCAN_IXR_RXNEMP_MASK | XCAN_IXR_RXOK_MASK)) {
+ if (isr & XCAN_IXR_RXNEMP_MASK) {
ier = priv->read_reg(priv, XCAN_IER_OFFSET);
- ier &= ~(XCAN_IXR_RXNEMP_MASK | XCAN_IXR_RXOK_MASK);
+ ier &= ~XCAN_IXR_RXNEMP_MASK;
priv->write_reg(priv, XCAN_IER_OFFSET, ier);
napi_schedule(&priv->napi);
}
@@ -819,13 +984,9 @@ static irqreturn_t xcan_interrupt(int irq, void *dev_id)
static void xcan_chip_stop(struct net_device *ndev)
{
struct xcan_priv *priv = netdev_priv(ndev);
- u32 ier;
/* Disable interrupts and leave the can in configuration mode */
- ier = priv->read_reg(priv, XCAN_IER_OFFSET);
- ier &= ~XCAN_INTR_ALL;
- priv->write_reg(priv, XCAN_IER_OFFSET, ier);
- priv->write_reg(priv, XCAN_SRR_OFFSET, XCAN_SRR_RESET_MASK);
+ set_reset_mode(ndev);
priv->can.state = CAN_STATE_STOPPED;
}
@@ -958,10 +1119,15 @@ static const struct net_device_ops xcan_netdev_ops = {
*/
static int __maybe_unused xcan_suspend(struct device *dev)
{
- if (!device_may_wakeup(dev))
- return pm_runtime_force_suspend(dev);
+ struct net_device *ndev = dev_get_drvdata(dev);
- return 0;
+ if (netif_running(ndev)) {
+ netif_stop_queue(ndev);
+ netif_device_detach(ndev);
+ xcan_chip_stop(ndev);
+ }
+
+ return pm_runtime_force_suspend(dev);
}
/**
@@ -973,11 +1139,27 @@ static int __maybe_unused xcan_suspend(struct device *dev)
*/
static int __maybe_unused xcan_resume(struct device *dev)
{
- if (!device_may_wakeup(dev))
- return pm_runtime_force_resume(dev);
+ struct net_device *ndev = dev_get_drvdata(dev);
+ int ret;
+
+ ret = pm_runtime_force_resume(dev);
+ if (ret) {
+ dev_err(dev, "pm_runtime_force_resume failed on resume\n");
+ return ret;
+ }
+
+ if (netif_running(ndev)) {
+ ret = xcan_chip_start(ndev);
+ if (ret) {
+ dev_err(dev, "xcan_chip_start failed on resume\n");
+ return ret;
+ }
+
+ netif_device_attach(ndev);
+ netif_start_queue(ndev);
+ }
return 0;
-
}
/**
@@ -992,14 +1174,6 @@ static int __maybe_unused xcan_runtime_suspend(struct device *dev)
struct net_device *ndev = dev_get_drvdata(dev);
struct xcan_priv *priv = netdev_priv(ndev);
- if (netif_running(ndev)) {
- netif_stop_queue(ndev);
- netif_device_detach(ndev);
- }
-
- priv->write_reg(priv, XCAN_MSR_OFFSET, XCAN_MSR_SLEEP_MASK);
- priv->can.state = CAN_STATE_SLEEPING;
-
clk_disable_unprepare(priv->bus_clk);
clk_disable_unprepare(priv->can_clk);
@@ -1018,7 +1192,6 @@ static int __maybe_unused xcan_runtime_resume(struct device *dev)
struct net_device *ndev = dev_get_drvdata(dev);
struct xcan_priv *priv = netdev_priv(ndev);
int ret;
- u32 isr, status;
ret = clk_prepare_enable(priv->bus_clk);
if (ret) {
@@ -1032,27 +1205,6 @@ static int __maybe_unused xcan_runtime_resume(struct device *dev)
return ret;
}
- priv->write_reg(priv, XCAN_SRR_OFFSET, XCAN_SRR_RESET_MASK);
- isr = priv->read_reg(priv, XCAN_ISR_OFFSET);
- status = priv->read_reg(priv, XCAN_SR_OFFSET);
-
- if (netif_running(ndev)) {
- if (isr & XCAN_IXR_BSOFF_MASK) {
- priv->can.state = CAN_STATE_BUS_OFF;
- priv->write_reg(priv, XCAN_SRR_OFFSET,
- XCAN_SRR_RESET_MASK);
- } else if ((status & XCAN_SR_ESTAT_MASK) ==
- XCAN_SR_ESTAT_MASK) {
- priv->can.state = CAN_STATE_ERROR_PASSIVE;
- } else if (status & XCAN_SR_ERRWRN_MASK) {
- priv->can.state = CAN_STATE_ERROR_WARNING;
- } else {
- priv->can.state = CAN_STATE_ERROR_ACTIVE;
- }
- netif_device_attach(ndev);
- netif_start_queue(ndev);
- }
-
return 0;
}
@@ -1061,6 +1213,18 @@ static const struct dev_pm_ops xcan_dev_pm_ops = {
SET_RUNTIME_PM_OPS(xcan_runtime_suspend, xcan_runtime_resume, NULL)
};
+static const struct xcan_devtype_data xcan_zynq_data = {
+ .caps = XCAN_CAP_WATERMARK,
+};
+
+/* Match table for OF platform binding */
+static const struct of_device_id xcan_of_match[] = {
+ { .compatible = "xlnx,zynq-can-1.0", .data = &xcan_zynq_data },
+ { .compatible = "xlnx,axi-can-1.00.a", },
+ { /* end of list */ },
+};
+MODULE_DEVICE_TABLE(of, xcan_of_match);
+
/**
* xcan_probe - Platform registration call
* @pdev: Handle to the platform device structure
@@ -1075,8 +1239,10 @@ static int xcan_probe(struct platform_device *pdev)
struct resource *res; /* IO mem resources */
struct net_device *ndev;
struct xcan_priv *priv;
+ const struct of_device_id *of_id;
+ int caps = 0;
void __iomem *addr;
- int ret, rx_max, tx_max;
+ int ret, rx_max, tx_max, tx_fifo_depth;
/* Get the virtual base address for the device */
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
@@ -1086,7 +1252,8 @@ static int xcan_probe(struct platform_device *pdev)
goto err;
}
- ret = of_property_read_u32(pdev->dev.of_node, "tx-fifo-depth", &tx_max);
+ ret = of_property_read_u32(pdev->dev.of_node, "tx-fifo-depth",
+ &tx_fifo_depth);
if (ret < 0)
goto err;
@@ -1094,6 +1261,30 @@ static int xcan_probe(struct platform_device *pdev)
if (ret < 0)
goto err;
+ of_id = of_match_device(xcan_of_match, &pdev->dev);
+ if (of_id) {
+ const struct xcan_devtype_data *devtype_data = of_id->data;
+
+ if (devtype_data)
+ caps = devtype_data->caps;
+ }
+
+ /* There is no way to directly figure out how many frames have been
+ * sent when the TXOK interrupt is processed. If watermark programming
+ * is supported, we can have 2 frames in the FIFO and use TXFEMP
+ * to determine if 1 or 2 frames have been sent.
+ * Theoretically we should be able to use TXFWMEMP to determine up
+ * to 3 frames, but it seems that after putting a second frame in the
+ * FIFO, with watermark at 2 frames, it can happen that TXFWMEMP (less
+ * than 2 frames in FIFO) is set anyway with no TXOK (a frame was
+ * sent), which is not a sensible state - possibly TXFWMEMP is not
+ * completely synchronized with the rest of the bits?
+ */
+ if (caps & XCAN_CAP_WATERMARK)
+ tx_max = min(tx_fifo_depth, 2);
+ else
+ tx_max = 1;
+
/* Create a CAN device instance */
ndev = alloc_candev(sizeof(struct xcan_priv), tx_max);
if (!ndev)
@@ -1108,6 +1299,7 @@ static int xcan_probe(struct platform_device *pdev)
CAN_CTRLMODE_BERR_REPORTING;
priv->reg_base = addr;
priv->tx_max = tx_max;
+ spin_lock_init(&priv->tx_lock);
/* Get IRQ for the device */
ndev->irq = platform_get_irq(pdev, 0);
@@ -1172,9 +1364,9 @@ static int xcan_probe(struct platform_device *pdev)
pm_runtime_put(&pdev->dev);
- netdev_dbg(ndev, "reg_base=0x%p irq=%d clock=%d, tx fifo depth:%d\n",
+ netdev_dbg(ndev, "reg_base=0x%p irq=%d clock=%d, tx fifo depth: actual %d, using %d\n",
priv->reg_base, ndev->irq, priv->can.clock.freq,
- priv->tx_max);
+ tx_fifo_depth, priv->tx_max);
return 0;
@@ -1208,14 +1400,6 @@ static int xcan_remove(struct platform_device *pdev)
return 0;
}
-/* Match table for OF platform binding */
-static const struct of_device_id xcan_of_match[] = {
- { .compatible = "xlnx,zynq-can-1.0", },
- { .compatible = "xlnx,axi-can-1.00.a", },
- { /* end of list */ },
-};
-MODULE_DEVICE_TABLE(of, xcan_of_match);
-
static struct platform_driver xcan_driver = {
.probe = xcan_probe,
.remove = xcan_remove,
diff --git a/drivers/net/dsa/b53/b53_common.c b/drivers/net/dsa/b53/b53_common.c
index c26debc..7152595 100644
--- a/drivers/net/dsa/b53/b53_common.c
+++ b/drivers/net/dsa/b53/b53_common.c
@@ -1516,6 +1516,18 @@ static const struct b53_chip_data b53_switch_chips[] = {
.duplex_reg = B53_DUPLEX_STAT_FE,
},
{
+ .chip_id = BCM5389_DEVICE_ID,
+ .dev_name = "BCM5389",
+ .vlans = 4096,
+ .enabled_ports = 0x1f,
+ .arl_entries = 4,
+ .cpu_port = B53_CPU_PORT,
+ .vta_regs = B53_VTA_REGS,
+ .duplex_reg = B53_DUPLEX_STAT_GE,
+ .jumbo_pm_reg = B53_JUMBO_PORT_MASK,
+ .jumbo_size_reg = B53_JUMBO_MAX_SIZE,
+ },
+ {
.chip_id = BCM5395_DEVICE_ID,
.dev_name = "BCM5395",
.vlans = 4096,
@@ -1825,6 +1837,7 @@ int b53_switch_detect(struct b53_device *dev)
else
dev->chip_id = BCM5365_DEVICE_ID;
break;
+ case BCM5389_DEVICE_ID:
case BCM5395_DEVICE_ID:
case BCM5397_DEVICE_ID:
case BCM5398_DEVICE_ID:
diff --git a/drivers/net/dsa/b53/b53_mdio.c b/drivers/net/dsa/b53/b53_mdio.c
index 477a16b..6f47ff1 100644
--- a/drivers/net/dsa/b53/b53_mdio.c
+++ b/drivers/net/dsa/b53/b53_mdio.c
@@ -285,6 +285,7 @@ static const struct b53_io_ops b53_mdio_ops = {
#define B53_BRCM_OUI_1 0x0143bc00
#define B53_BRCM_OUI_2 0x03625c00
#define B53_BRCM_OUI_3 0x00406000
+#define B53_BRCM_OUI_4 0x01410c00
static int b53_mdio_probe(struct mdio_device *mdiodev)
{
@@ -311,7 +312,8 @@ static int b53_mdio_probe(struct mdio_device *mdiodev)
*/
if ((phy_id & 0xfffffc00) != B53_BRCM_OUI_1 &&
(phy_id & 0xfffffc00) != B53_BRCM_OUI_2 &&
- (phy_id & 0xfffffc00) != B53_BRCM_OUI_3) {
+ (phy_id & 0xfffffc00) != B53_BRCM_OUI_3 &&
+ (phy_id & 0xfffffc00) != B53_BRCM_OUI_4) {
dev_err(&mdiodev->dev, "Unsupported device: 0x%08x\n", phy_id);
return -ENODEV;
}
@@ -360,6 +362,7 @@ static const struct of_device_id b53_of_match[] = {
{ .compatible = "brcm,bcm53125" },
{ .compatible = "brcm,bcm53128" },
{ .compatible = "brcm,bcm5365" },
+ { .compatible = "brcm,bcm5389" },
{ .compatible = "brcm,bcm5395" },
{ .compatible = "brcm,bcm5397" },
{ .compatible = "brcm,bcm5398" },
diff --git a/drivers/net/dsa/b53/b53_priv.h b/drivers/net/dsa/b53/b53_priv.h
index f192a67..68ab20b 100644
--- a/drivers/net/dsa/b53/b53_priv.h
+++ b/drivers/net/dsa/b53/b53_priv.h
@@ -47,6 +47,7 @@ struct b53_io_ops {
enum {
BCM5325_DEVICE_ID = 0x25,
BCM5365_DEVICE_ID = 0x65,
+ BCM5389_DEVICE_ID = 0x89,
BCM5395_DEVICE_ID = 0x95,
BCM5397_DEVICE_ID = 0x97,
BCM5398_DEVICE_ID = 0x98,
diff --git a/drivers/net/dsa/bcm_sf2.c b/drivers/net/dsa/bcm_sf2.c
index 2ce7ae9..c2cd540 100644
--- a/drivers/net/dsa/bcm_sf2.c
+++ b/drivers/net/dsa/bcm_sf2.c
@@ -744,7 +744,6 @@ static int bcm_sf2_sw_suspend(struct dsa_switch *ds)
static int bcm_sf2_sw_resume(struct dsa_switch *ds)
{
struct bcm_sf2_priv *priv = bcm_sf2_to_priv(ds);
- unsigned int port;
int ret;
ret = bcm_sf2_sw_rst(priv);
@@ -756,12 +755,7 @@ static int bcm_sf2_sw_resume(struct dsa_switch *ds)
if (priv->hw_params.num_gphy == 1)
bcm_sf2_gphy_enable_set(ds, true);
- for (port = 0; port < DSA_MAX_PORTS; port++) {
- if ((1 << port) & ds->enabled_port_mask)
- bcm_sf2_port_setup(ds, port, NULL);
- else if (dsa_is_cpu_port(ds, port))
- bcm_sf2_imp_setup(ds, port);
- }
+ ds->ops->setup(ds);
return 0;
}
@@ -1135,10 +1129,10 @@ static int bcm_sf2_sw_remove(struct platform_device *pdev)
{
struct bcm_sf2_priv *priv = platform_get_drvdata(pdev);
- /* Disable all ports and interrupts */
priv->wol_ports_mask = 0;
- bcm_sf2_sw_suspend(priv->dev->ds);
dsa_unregister_switch(priv->dev->ds);
+ /* Disable all ports and interrupts */
+ bcm_sf2_sw_suspend(priv->dev->ds);
bcm_sf2_mdio_unregister(priv);
return 0;
diff --git a/drivers/net/dsa/mv88e6060.c b/drivers/net/dsa/mv88e6060.c
index 7ce36db..a3607d0 100644
--- a/drivers/net/dsa/mv88e6060.c
+++ b/drivers/net/dsa/mv88e6060.c
@@ -114,8 +114,7 @@ static int mv88e6060_switch_reset(struct dsa_switch *ds)
/* Reset the switch. */
REG_WRITE(REG_GLOBAL, GLOBAL_ATU_CONTROL,
GLOBAL_ATU_CONTROL_SWRESET |
- GLOBAL_ATU_CONTROL_ATUSIZE_1024 |
- GLOBAL_ATU_CONTROL_ATE_AGE_5MIN);
+ GLOBAL_ATU_CONTROL_LEARNDIS);
/* Wait up to one second for reset to complete. */
timeout = jiffies + 1 * HZ;
@@ -140,13 +139,10 @@ static int mv88e6060_setup_global(struct dsa_switch *ds)
*/
REG_WRITE(REG_GLOBAL, GLOBAL_CONTROL, GLOBAL_CONTROL_MAX_FRAME_1536);
- /* Enable automatic address learning, set the address
- * database size to 1024 entries, and set the default aging
- * time to 5 minutes.
+ /* Disable automatic address learning.
*/
REG_WRITE(REG_GLOBAL, GLOBAL_ATU_CONTROL,
- GLOBAL_ATU_CONTROL_ATUSIZE_1024 |
- GLOBAL_ATU_CONTROL_ATE_AGE_5MIN);
+ GLOBAL_ATU_CONTROL_LEARNDIS);
return 0;
}
diff --git a/drivers/net/dsa/qca8k.c b/drivers/net/dsa/qca8k.c
index b3df70d..7f64a76 100644
--- a/drivers/net/dsa/qca8k.c
+++ b/drivers/net/dsa/qca8k.c
@@ -474,7 +474,7 @@ qca8k_set_pad_ctrl(struct qca8k_priv *priv, int port, int mode)
static void
qca8k_port_set_status(struct qca8k_priv *priv, int port, int enable)
{
- u32 mask = QCA8K_PORT_STATUS_TXMAC;
+ u32 mask = QCA8K_PORT_STATUS_TXMAC | QCA8K_PORT_STATUS_RXMAC;
/* Port 0 and 6 have no internal PHY */
if ((port > 0) && (port < 6))
@@ -491,6 +491,7 @@ qca8k_setup(struct dsa_switch *ds)
{
struct qca8k_priv *priv = (struct qca8k_priv *)ds->priv;
int ret, i, phy_mode = -1;
+ u32 mask;
/* Make sure that port 0 is the cpu port */
if (!dsa_is_cpu_port(ds, 0)) {
@@ -516,7 +517,10 @@ qca8k_setup(struct dsa_switch *ds)
if (ret < 0)
return ret;
- /* Enable CPU Port */
+ /* Enable CPU Port, force it to maximum bandwidth and full-duplex */
+ mask = QCA8K_PORT_STATUS_SPEED_1000 | QCA8K_PORT_STATUS_TXFLOW |
+ QCA8K_PORT_STATUS_RXFLOW | QCA8K_PORT_STATUS_DUPLEX;
+ qca8k_write(priv, QCA8K_REG_PORT_STATUS(QCA8K_CPU_PORT), mask);
qca8k_reg_set(priv, QCA8K_REG_GLOBAL_FW_CTRL0,
QCA8K_GLOBAL_FW_CTRL0_CPU_PORT_EN);
qca8k_port_set_status(priv, QCA8K_CPU_PORT, 1);
@@ -585,6 +589,47 @@ qca8k_setup(struct dsa_switch *ds)
return 0;
}
+static void
+qca8k_adjust_link(struct dsa_switch *ds, int port, struct phy_device *phy)
+{
+ struct qca8k_priv *priv = ds->priv;
+ u32 reg;
+
+ /* Force fixed-link setting for CPU port, skip others. */
+ if (!phy_is_pseudo_fixed_link(phy))
+ return;
+
+ /* Set port speed */
+ switch (phy->speed) {
+ case 10:
+ reg = QCA8K_PORT_STATUS_SPEED_10;
+ break;
+ case 100:
+ reg = QCA8K_PORT_STATUS_SPEED_100;
+ break;
+ case 1000:
+ reg = QCA8K_PORT_STATUS_SPEED_1000;
+ break;
+ default:
+ dev_dbg(priv->dev, "port%d link speed %dMbps not supported.\n",
+ port, phy->speed);
+ return;
+ }
+
+ /* Set duplex mode */
+ if (phy->duplex == DUPLEX_FULL)
+ reg |= QCA8K_PORT_STATUS_DUPLEX;
+
+ /* Force flow control */
+ if (dsa_is_cpu_port(ds, port))
+ reg |= QCA8K_PORT_STATUS_RXFLOW | QCA8K_PORT_STATUS_TXFLOW;
+
+ /* Force link down before changing MAC options */
+ qca8k_port_set_status(priv, port, 0);
+ qca8k_write(priv, QCA8K_REG_PORT_STATUS(port), reg);
+ qca8k_port_set_status(priv, port, 1);
+}
+
static int
qca8k_phy_read(struct dsa_switch *ds, int phy, int regnum)
{
@@ -914,6 +959,7 @@ qca8k_get_tag_protocol(struct dsa_switch *ds)
static struct dsa_switch_ops qca8k_switch_ops = {
.get_tag_protocol = qca8k_get_tag_protocol,
.setup = qca8k_setup,
+ .adjust_link = qca8k_adjust_link,
.get_strings = qca8k_get_strings,
.phy_read = qca8k_phy_read,
.phy_write = qca8k_phy_write,
@@ -946,6 +992,7 @@ qca8k_sw_probe(struct mdio_device *mdiodev)
return -ENOMEM;
priv->bus = mdiodev->bus;
+ priv->dev = &mdiodev->dev;
/* read the switches ID register */
id = qca8k_read(priv, QCA8K_REG_MASK_CTRL);
@@ -1018,6 +1065,7 @@ static SIMPLE_DEV_PM_OPS(qca8k_pm_ops,
qca8k_suspend, qca8k_resume);
static const struct of_device_id qca8k_of_match[] = {
+ { .compatible = "qca,qca8334" },
{ .compatible = "qca,qca8337" },
{ /* sentinel */ },
};
diff --git a/drivers/net/dsa/qca8k.h b/drivers/net/dsa/qca8k.h
index 2014647..9c22bc3 100644
--- a/drivers/net/dsa/qca8k.h
+++ b/drivers/net/dsa/qca8k.h
@@ -51,8 +51,10 @@
#define QCA8K_GOL_MAC_ADDR0 0x60
#define QCA8K_GOL_MAC_ADDR1 0x64
#define QCA8K_REG_PORT_STATUS(_i) (0x07c + (_i) * 4)
-#define QCA8K_PORT_STATUS_SPEED GENMASK(2, 0)
-#define QCA8K_PORT_STATUS_SPEED_S 0
+#define QCA8K_PORT_STATUS_SPEED GENMASK(1, 0)
+#define QCA8K_PORT_STATUS_SPEED_10 0
+#define QCA8K_PORT_STATUS_SPEED_100 0x1
+#define QCA8K_PORT_STATUS_SPEED_1000 0x2
#define QCA8K_PORT_STATUS_TXMAC BIT(2)
#define QCA8K_PORT_STATUS_RXMAC BIT(3)
#define QCA8K_PORT_STATUS_TXFLOW BIT(4)
@@ -167,6 +169,7 @@ struct qca8k_priv {
struct ar8xxx_port_status port_sts[QCA8K_NUM_PORTS];
struct dsa_switch *ds;
struct mutex reg_mutex;
+ struct device *dev;
};
struct qca8k_mib_desc {
diff --git a/drivers/net/ethernet/3com/Kconfig b/drivers/net/ethernet/3com/Kconfig
index 5b7658b..5c3ef9f 100644
--- a/drivers/net/ethernet/3com/Kconfig
+++ b/drivers/net/ethernet/3com/Kconfig
@@ -32,7 +32,7 @@
config 3C515
tristate "3c515 ISA \"Fast EtherLink\""
- depends on ISA && ISA_DMA_API
+ depends on ISA && ISA_DMA_API && !PPC32
---help---
If you have a 3Com ISA EtherLink XL "Corkscrew" 3c515 Fast Ethernet
network card, say Y here.
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c
index e13c9cd..bcd9931 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_com.c
@@ -331,6 +331,7 @@ static int ena_com_init_io_sq(struct ena_com_dev *ena_dev,
memset(&io_sq->desc_addr, 0x0, sizeof(struct ena_com_io_desc_addr));
+ io_sq->dma_addr_bits = ena_dev->dma_addr_bits;
io_sq->desc_entry_size =
(io_sq->direction == ENA_COM_IO_QUEUE_DIRECTION_TX) ?
sizeof(struct ena_eth_io_tx_desc) :
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index 0d9ce08..0c29887 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -422,7 +422,7 @@ static inline int ena_alloc_rx_page(struct ena_ring *rx_ring,
return -ENOMEM;
}
- dma = dma_map_page(rx_ring->dev, page, 0, PAGE_SIZE,
+ dma = dma_map_page(rx_ring->dev, page, 0, ENA_PAGE_SIZE,
DMA_FROM_DEVICE);
if (unlikely(dma_mapping_error(rx_ring->dev, dma))) {
u64_stats_update_begin(&rx_ring->syncp);
@@ -439,7 +439,7 @@ static inline int ena_alloc_rx_page(struct ena_ring *rx_ring,
rx_info->page_offset = 0;
ena_buf = &rx_info->ena_buf;
ena_buf->paddr = dma;
- ena_buf->len = PAGE_SIZE;
+ ena_buf->len = ENA_PAGE_SIZE;
return 0;
}
@@ -456,7 +456,7 @@ static void ena_free_rx_page(struct ena_ring *rx_ring,
return;
}
- dma_unmap_page(rx_ring->dev, ena_buf->paddr, PAGE_SIZE,
+ dma_unmap_page(rx_ring->dev, ena_buf->paddr, ENA_PAGE_SIZE,
DMA_FROM_DEVICE);
__free_page(page);
@@ -849,10 +849,10 @@ static struct sk_buff *ena_rx_skb(struct ena_ring *rx_ring,
do {
dma_unmap_page(rx_ring->dev,
dma_unmap_addr(&rx_info->ena_buf, paddr),
- PAGE_SIZE, DMA_FROM_DEVICE);
+ ENA_PAGE_SIZE, DMA_FROM_DEVICE);
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, rx_info->page,
- rx_info->page_offset, len, PAGE_SIZE);
+ rx_info->page_offset, len, ENA_PAGE_SIZE);
netif_dbg(rx_ring->adapter, rx_status, rx_ring->netdev,
"rx skb updated. len %d. data_len %d\n",
@@ -1482,8 +1482,6 @@ static int ena_up_complete(struct ena_adapter *adapter)
if (rc)
return rc;
- ena_init_napi(adapter);
-
ena_change_mtu(adapter->netdev, adapter->netdev->mtu);
ena_refill_all_rx_bufs(adapter);
@@ -1643,6 +1641,13 @@ static int ena_up(struct ena_adapter *adapter)
ena_setup_io_intr(adapter);
+ /* napi poll functions should be initialized before running
+ * request_irq(), to handle a rare condition where there is a pending
+ * interrupt, causing the ISR to fire immediately while the poll
+ * function wasn't set yet, causing a null dereference
+ */
+ ena_init_napi(adapter);
+
rc = ena_request_io_irq(adapter);
if (rc)
goto err_req_irq;
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.h b/drivers/net/ethernet/amazon/ena/ena_netdev.h
index c5eaf76..008f2d5 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.h
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.h
@@ -321,4 +321,15 @@ void ena_dump_stats_to_buf(struct ena_adapter *adapter, u8 *buf);
int ena_get_sset_count(struct net_device *netdev, int sset);
+/* The ENA buffer length fields is 16 bit long. So when PAGE_SIZE == 64kB the
+ * driver passas 0.
+ * Since the max packet size the ENA handles is ~9kB limit the buffer length to
+ * 16kB.
+ */
+#if PAGE_SIZE > SZ_16K
+#define ENA_PAGE_SIZE SZ_16K
+#else
+#define ENA_PAGE_SIZE PAGE_SIZE
+#endif
+
#endif /* !(ENA_H) */
diff --git a/drivers/net/ethernet/amd/Kconfig b/drivers/net/ethernet/amd/Kconfig
index 0038709..ec59425 100644
--- a/drivers/net/ethernet/amd/Kconfig
+++ b/drivers/net/ethernet/amd/Kconfig
@@ -44,7 +44,7 @@
config LANCE
tristate "AMD LANCE and PCnet (AT1500 and NE2100) support"
- depends on ISA && ISA_DMA_API && !ARM
+ depends on ISA && ISA_DMA_API && !ARM && !PPC32
---help---
If you have a network (Ethernet) card of this type, say Y here.
Some LinkSys cards are of this type.
@@ -138,7 +138,7 @@
config NI65
tristate "NI6510 support"
- depends on ISA && ISA_DMA_API && !ARM
+ depends on ISA && ISA_DMA_API && !ARM && !PPC32
---help---
If you have a network (Ethernet) card of this type, say Y here.
diff --git a/drivers/net/ethernet/amd/declance.c b/drivers/net/ethernet/amd/declance.c
index b799c7a..9e80a76 100644
--- a/drivers/net/ethernet/amd/declance.c
+++ b/drivers/net/ethernet/amd/declance.c
@@ -1030,6 +1030,7 @@ static int dec_lance_probe(struct device *bdev, const int type)
int i, ret;
unsigned long esar_base;
unsigned char *esar;
+ const char *desc;
if (dec_lance_debug && version_printed++ == 0)
printk(version);
@@ -1215,19 +1216,20 @@ static int dec_lance_probe(struct device *bdev, const int type)
*/
switch (type) {
case ASIC_LANCE:
- printk("%s: IOASIC onboard LANCE", name);
+ desc = "IOASIC onboard LANCE";
break;
case PMAD_LANCE:
- printk("%s: PMAD-AA", name);
+ desc = "PMAD-AA";
break;
case PMAX_LANCE:
- printk("%s: PMAX onboard LANCE", name);
+ desc = "PMAX onboard LANCE";
break;
}
for (i = 0; i < 6; i++)
dev->dev_addr[i] = esar[i * 4];
- printk(", addr = %pM, irq = %d\n", dev->dev_addr, dev->irq);
+ printk("%s: %s, addr = %pM, irq = %d\n",
+ name, desc, dev->dev_addr, dev->irq);
dev->netdev_ops = &lance_netdev_ops;
dev->watchdog_timeo = 5*HZ;
diff --git a/drivers/net/ethernet/amd/sunlance.c b/drivers/net/ethernet/amd/sunlance.c
index 9b56b40..3153465 100644
--- a/drivers/net/ethernet/amd/sunlance.c
+++ b/drivers/net/ethernet/amd/sunlance.c
@@ -1419,7 +1419,7 @@ static int sparc_lance_probe_one(struct platform_device *op,
prop = of_get_property(nd, "tpe-link-test?", NULL);
if (!prop)
- goto no_link_test;
+ goto node_put;
if (strcmp(prop, "true")) {
printk(KERN_NOTICE "SunLance: warning: overriding option "
@@ -1428,6 +1428,8 @@ static int sparc_lance_probe_one(struct platform_device *op,
"to ecd@skynet.be\n");
auxio_set_lte(AUXIO_LTE_ON);
}
+node_put:
+ of_node_put(nd);
no_link_test:
lp->auto_select = 1;
lp->tpe = 0;
diff --git a/drivers/net/ethernet/amd/xgbe/xgbe-mdio.c b/drivers/net/ethernet/amd/xgbe/xgbe-mdio.c
index 84c5d29..6848358 100644
--- a/drivers/net/ethernet/amd/xgbe/xgbe-mdio.c
+++ b/drivers/net/ethernet/amd/xgbe/xgbe-mdio.c
@@ -877,14 +877,14 @@ static void xgbe_phy_adjust_link(struct xgbe_prv_data *pdata)
if (pdata->tx_pause != pdata->phy.tx_pause) {
new_state = 1;
- pdata->hw_if.config_tx_flow_control(pdata);
pdata->tx_pause = pdata->phy.tx_pause;
+ pdata->hw_if.config_tx_flow_control(pdata);
}
if (pdata->rx_pause != pdata->phy.rx_pause) {
new_state = 1;
- pdata->hw_if.config_rx_flow_control(pdata);
pdata->rx_pause = pdata->phy.rx_pause;
+ pdata->hw_if.config_rx_flow_control(pdata);
}
/* Speed support */
diff --git a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
index fca2e42..de4b5d2 100644
--- a/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
+++ b/drivers/net/ethernet/apm/xgene/xgene_enet_main.c
@@ -29,9 +29,6 @@
#define RES_RING_CSR 1
#define RES_RING_CMD 2
-static const struct of_device_id xgene_enet_of_match[];
-static const struct acpi_device_id xgene_enet_acpi_match[];
-
static void xgene_enet_init_bufpool(struct xgene_enet_desc_ring *buf_pool)
{
struct xgene_enet_raw_desc16 *raw_desc;
diff --git a/drivers/net/ethernet/atheros/atl1c/atl1c_main.c b/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
index a3200ea..85e7177 100644
--- a/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
+++ b/drivers/net/ethernet/atheros/atl1c/atl1c_main.c
@@ -1678,6 +1678,7 @@ static struct sk_buff *atl1c_alloc_skb(struct atl1c_adapter *adapter)
skb = build_skb(page_address(page) + adapter->rx_page_offset,
adapter->rx_frag_size);
if (likely(skb)) {
+ skb_reserve(skb, NET_SKB_PAD);
adapter->rx_page_offset += adapter->rx_frag_size;
if (adapter->rx_page_offset >= PAGE_SIZE)
adapter->rx_page = NULL;
diff --git a/drivers/net/ethernet/broadcom/bcm63xx_enet.c b/drivers/net/ethernet/broadcom/bcm63xx_enet.c
index 08d91ef..c407840 100644
--- a/drivers/net/ethernet/broadcom/bcm63xx_enet.c
+++ b/drivers/net/ethernet/broadcom/bcm63xx_enet.c
@@ -1063,7 +1063,8 @@ static int bcm_enet_open(struct net_device *dev)
val = enet_readl(priv, ENET_CTL_REG);
val |= ENET_CTL_ENABLE_MASK;
enet_writel(priv, val, ENET_CTL_REG);
- enet_dma_writel(priv, ENETDMA_CFG_EN_MASK, ENETDMA_CFG_REG);
+ if (priv->dma_has_sram)
+ enet_dma_writel(priv, ENETDMA_CFG_EN_MASK, ENETDMA_CFG_REG);
enet_dmac_writel(priv, priv->dma_chan_en_mask,
ENETDMAC_CHANCFG, priv->rx_chan);
@@ -1790,7 +1791,9 @@ static int bcm_enet_probe(struct platform_device *pdev)
ret = PTR_ERR(priv->mac_clk);
goto out;
}
- clk_prepare_enable(priv->mac_clk);
+ ret = clk_prepare_enable(priv->mac_clk);
+ if (ret)
+ goto out_put_clk_mac;
/* initialize default and fetch platform data */
priv->rx_ring_size = BCMENET_DEF_RX_DESC;
@@ -1822,9 +1825,11 @@ static int bcm_enet_probe(struct platform_device *pdev)
if (IS_ERR(priv->phy_clk)) {
ret = PTR_ERR(priv->phy_clk);
priv->phy_clk = NULL;
- goto out_put_clk_mac;
+ goto out_disable_clk_mac;
}
- clk_prepare_enable(priv->phy_clk);
+ ret = clk_prepare_enable(priv->phy_clk);
+ if (ret)
+ goto out_put_clk_phy;
}
/* do minimal hardware init to be able to probe mii bus */
@@ -1915,13 +1920,16 @@ static int bcm_enet_probe(struct platform_device *pdev)
out_uninit_hw:
/* turn off mdc clock */
enet_writel(priv, 0, ENET_MIISC_REG);
- if (priv->phy_clk) {
+ if (priv->phy_clk)
clk_disable_unprepare(priv->phy_clk);
- clk_put(priv->phy_clk);
- }
-out_put_clk_mac:
+out_put_clk_phy:
+ if (priv->phy_clk)
+ clk_put(priv->phy_clk);
+
+out_disable_clk_mac:
clk_disable_unprepare(priv->mac_clk);
+out_put_clk_mac:
clk_put(priv->mac_clk);
out:
free_netdev(dev);
@@ -2766,7 +2774,9 @@ static int bcm_enetsw_probe(struct platform_device *pdev)
ret = PTR_ERR(priv->mac_clk);
goto out_unmap;
}
- clk_enable(priv->mac_clk);
+ ret = clk_prepare_enable(priv->mac_clk);
+ if (ret)
+ goto out_put_clk;
priv->rx_chan = 0;
priv->tx_chan = 1;
@@ -2787,7 +2797,7 @@ static int bcm_enetsw_probe(struct platform_device *pdev)
ret = register_netdev(dev);
if (ret)
- goto out_put_clk;
+ goto out_disable_clk;
netif_carrier_off(dev);
platform_set_drvdata(pdev, dev);
@@ -2796,6 +2806,9 @@ static int bcm_enetsw_probe(struct platform_device *pdev)
return 0;
+out_disable_clk:
+ clk_disable_unprepare(priv->mac_clk);
+
out_put_clk:
clk_put(priv->mac_clk);
@@ -2827,6 +2840,9 @@ static int bcm_enetsw_remove(struct platform_device *pdev)
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
release_mem_region(res->start, resource_size(res));
+ clk_disable_unprepare(priv->mac_clk);
+ clk_put(priv->mac_clk);
+
free_netdev(dev);
return 0;
}
diff --git a/drivers/net/ethernet/broadcom/bcmsysport.c b/drivers/net/ethernet/broadcom/bcmsysport.c
index 91fbba5..16dc9ac 100644
--- a/drivers/net/ethernet/broadcom/bcmsysport.c
+++ b/drivers/net/ethernet/broadcom/bcmsysport.c
@@ -828,14 +828,22 @@ static void bcm_sysport_resume_from_wol(struct bcm_sysport_priv *priv)
{
u32 reg;
- /* Stop monitoring MPD interrupt */
- intrl2_0_mask_set(priv, INTRL2_0_MPD);
-
/* Clear the MagicPacket detection logic */
reg = umac_readl(priv, UMAC_MPD_CTRL);
reg &= ~MPD_EN;
umac_writel(priv, reg, UMAC_MPD_CTRL);
+ reg = intrl2_0_readl(priv, INTRL2_CPU_STATUS);
+ if (reg & INTRL2_0_MPD)
+ netdev_info(priv->netdev, "Wake-on-LAN (MPD) interrupt!\n");
+
+ if (reg & INTRL2_0_BRCM_MATCH_TAG) {
+ reg = rxchk_readl(priv, RXCHK_BRCM_TAG_MATCH_STATUS) &
+ RXCHK_BRCM_TAG_MATCH_MASK;
+ netdev_info(priv->netdev,
+ "Wake-on-LAN (filters 0x%02x) interrupt!\n", reg);
+ }
+
netif_dbg(priv, wol, priv->netdev, "resumed from WOL\n");
}
@@ -868,11 +876,6 @@ static irqreturn_t bcm_sysport_rx_isr(int irq, void *dev_id)
if (priv->irq0_stat & INTRL2_0_TX_RING_FULL)
bcm_sysport_tx_reclaim_all(priv);
- if (priv->irq0_stat & INTRL2_0_MPD) {
- netdev_info(priv->netdev, "Wake-on-LAN interrupt!\n");
- bcm_sysport_resume_from_wol(priv);
- }
-
return IRQ_HANDLED;
}
@@ -1901,9 +1904,6 @@ static int bcm_sysport_suspend_to_wol(struct bcm_sysport_priv *priv)
/* UniMAC receive needs to be turned on */
umac_enable_set(priv, CMD_RX_EN, 1);
- /* Enable the interrupt wake-up source */
- intrl2_0_mask_clear(priv, INTRL2_0_MPD);
-
netif_dbg(priv, wol, ndev, "entered WOL mode\n");
return 0;
diff --git a/drivers/net/ethernet/broadcom/bgmac.c b/drivers/net/ethernet/broadcom/bgmac.c
index 49f4cafe..86a32fe 100644
--- a/drivers/net/ethernet/broadcom/bgmac.c
+++ b/drivers/net/ethernet/broadcom/bgmac.c
@@ -529,7 +529,8 @@ static void bgmac_dma_tx_ring_free(struct bgmac *bgmac,
int i;
for (i = 0; i < BGMAC_TX_RING_SLOTS; i++) {
- int len = dma_desc[i].ctl1 & BGMAC_DESC_CTL1_LEN;
+ u32 ctl1 = le32_to_cpu(dma_desc[i].ctl1);
+ unsigned int len = ctl1 & BGMAC_DESC_CTL1_LEN;
slot = &ring->slots[i];
dev_kfree_skb(slot->skb);
diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x.h b/drivers/net/ethernet/broadcom/bnx2x/bnx2x.h
index 7dd7490..162b809 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x.h
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x.h
@@ -1278,6 +1278,7 @@ enum sp_rtnl_flag {
BNX2X_SP_RTNL_TX_STOP,
BNX2X_SP_RTNL_GET_DRV_VERSION,
BNX2X_SP_RTNL_CHANGE_UDP_PORT,
+ BNX2X_SP_RTNL_UPDATE_SVID,
};
enum bnx2x_iov_flag {
@@ -1529,6 +1530,7 @@ struct bnx2x {
struct link_vars link_vars;
u32 link_cnt;
struct bnx2x_link_report_data last_reported_link;
+ bool force_link_down;
struct mdio_if_info mdio;
@@ -2186,6 +2188,13 @@ void bnx2x_igu_clear_sb_gen(struct bnx2x *bp, u8 func, u8 idu_sb_id,
#define PMF_DMAE_C(bp) (BP_PORT(bp) * MAX_DMAE_C_PER_PORT + \
E1HVN_MAX)
+/* Following is the DMAE channel number allocation for the clients.
+ * MFW: OCBB/OCSD implementations use DMAE channels 14/15 respectively.
+ * Driver: 0-3 and 8-11 (for PF dmae operations)
+ * 4 and 12 (for stats requests)
+ */
+#define BNX2X_FW_DMAE_C 13 /* Channel for FW DMAE operations */
+
/* PCIE link and speed */
#define PCICFG_LINK_WIDTH 0x1f00000
#define PCICFG_LINK_WIDTH_SHIFT 20
diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c
index 31287ce..2cd1dcd 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c
@@ -1265,6 +1265,11 @@ void __bnx2x_link_report(struct bnx2x *bp)
{
struct bnx2x_link_report_data cur_data;
+ if (bp->force_link_down) {
+ bp->link_vars.link_up = 0;
+ return;
+ }
+
/* reread mf_cfg */
if (IS_PF(bp) && !CHIP_IS_E1(bp))
bnx2x_read_mf_cfg(bp);
@@ -2822,6 +2827,7 @@ int bnx2x_nic_load(struct bnx2x *bp, int load_mode)
bp->pending_max = 0;
}
+ bp->force_link_down = false;
if (bp->port.pmf) {
rc = bnx2x_initial_phy_init(bp, load_mode);
if (rc)
diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_ethtool.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_ethtool.c
index 5f19427..8aecd8e 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_ethtool.c
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_ethtool.c
@@ -3367,14 +3367,18 @@ static int bnx2x_set_rss_flags(struct bnx2x *bp, struct ethtool_rxnfc *info)
DP(BNX2X_MSG_ETHTOOL,
"rss re-configured, UDP 4-tupple %s\n",
udp_rss_requested ? "enabled" : "disabled");
- return bnx2x_rss(bp, &bp->rss_conf_obj, false, true);
+ if (bp->state == BNX2X_STATE_OPEN)
+ return bnx2x_rss(bp, &bp->rss_conf_obj, false,
+ true);
} else if ((info->flow_type == UDP_V6_FLOW) &&
(bp->rss_conf_obj.udp_rss_v6 != udp_rss_requested)) {
bp->rss_conf_obj.udp_rss_v6 = udp_rss_requested;
DP(BNX2X_MSG_ETHTOOL,
"rss re-configured, UDP 4-tupple %s\n",
udp_rss_requested ? "enabled" : "disabled");
- return bnx2x_rss(bp, &bp->rss_conf_obj, false, true);
+ if (bp->state == BNX2X_STATE_OPEN)
+ return bnx2x_rss(bp, &bp->rss_conf_obj, false,
+ true);
}
return 0;
@@ -3488,7 +3492,10 @@ static int bnx2x_set_rxfh(struct net_device *dev, const u32 *indir,
bp->rss_conf_obj.ind_table[i] = indir[i] + bp->fp->cl_id;
}
- return bnx2x_config_rss_eth(bp, false);
+ if (bp->state == BNX2X_STATE_OPEN)
+ return bnx2x_config_rss_eth(bp, false);
+
+ return 0;
}
/**
diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_link.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_link.c
index 1fb8010..912900d 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_link.c
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_link.c
@@ -594,7 +594,7 @@ static void bnx2x_ets_e3b0_nig_disabled(const struct link_params *params,
* slots for the highest priority.
*/
REG_WR(bp, (port) ? NIG_REG_P1_TX_ARB_NUM_STRICT_ARB_SLOTS :
- NIG_REG_P1_TX_ARB_NUM_STRICT_ARB_SLOTS, 0x100);
+ NIG_REG_P0_TX_ARB_NUM_STRICT_ARB_SLOTS, 0x100);
/* Mapping between the CREDIT_WEIGHT registers and actual client
* numbers
*/
diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c
index 554c408..4bc2c80 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c
@@ -2925,6 +2925,10 @@ static void bnx2x_handle_update_svid_cmd(struct bnx2x *bp)
func_params.f_obj = &bp->func_obj;
func_params.cmd = BNX2X_F_CMD_SWITCH_UPDATE;
+ /* Prepare parameters for function state transitions */
+ __set_bit(RAMROD_COMP_WAIT, &func_params.ramrod_flags);
+ __set_bit(RAMROD_RETRY, &func_params.ramrod_flags);
+
if (IS_MF_UFP(bp) || IS_MF_BD(bp)) {
int func = BP_ABS_FUNC(bp);
u32 val;
@@ -4301,7 +4305,8 @@ static void bnx2x_attn_int_deasserted3(struct bnx2x *bp, u32 attn)
bnx2x_handle_eee_event(bp);
if (val & DRV_STATUS_OEM_UPDATE_SVID)
- bnx2x_handle_update_svid_cmd(bp);
+ bnx2x_schedule_sp_rtnl(bp,
+ BNX2X_SP_RTNL_UPDATE_SVID, 0);
if (bp->link_vars.periodic_flags &
PERIODIC_FLAGS_LINK_EVENT) {
@@ -8462,6 +8467,7 @@ int bnx2x_set_vlan_one(struct bnx2x *bp, u16 vlan,
/* Fill a user request section if needed */
if (!test_bit(RAMROD_CONT, ramrod_flags)) {
ramrod_param.user_req.u.vlan.vlan = vlan;
+ __set_bit(BNX2X_VLAN, &ramrod_param.user_req.vlan_mac_flags);
/* Set the command: ADD or DEL */
if (set)
ramrod_param.user_req.cmd = BNX2X_VLAN_MAC_ADD;
@@ -8482,6 +8488,27 @@ int bnx2x_set_vlan_one(struct bnx2x *bp, u16 vlan,
return rc;
}
+static int bnx2x_del_all_vlans(struct bnx2x *bp)
+{
+ struct bnx2x_vlan_mac_obj *vlan_obj = &bp->sp_objs[0].vlan_obj;
+ unsigned long ramrod_flags = 0, vlan_flags = 0;
+ struct bnx2x_vlan_entry *vlan;
+ int rc;
+
+ __set_bit(RAMROD_COMP_WAIT, &ramrod_flags);
+ __set_bit(BNX2X_VLAN, &vlan_flags);
+ rc = vlan_obj->delete_all(bp, vlan_obj, &vlan_flags, &ramrod_flags);
+ if (rc)
+ return rc;
+
+ /* Mark that hw forgot all entries */
+ list_for_each_entry(vlan, &bp->vlan_reg, link)
+ vlan->hw = false;
+ bp->vlan_cnt = 0;
+
+ return 0;
+}
+
int bnx2x_del_all_macs(struct bnx2x *bp,
struct bnx2x_vlan_mac_obj *mac_obj,
int mac_type, bool wait_for_comp)
@@ -9320,6 +9347,17 @@ void bnx2x_chip_cleanup(struct bnx2x *bp, int unload_mode, bool keep_link)
BNX2X_ERR("Failed to schedule DEL commands for UC MACs list: %d\n",
rc);
+ /* The whole *vlan_obj structure may be not initialized if VLAN
+ * filtering offload is not supported by hardware. Currently this is
+ * true for all hardware covered by CHIP_IS_E1x().
+ */
+ if (!CHIP_IS_E1x(bp)) {
+ /* Remove all currently configured VLANs */
+ rc = bnx2x_del_all_vlans(bp);
+ if (rc < 0)
+ BNX2X_ERR("Failed to delete all VLANs\n");
+ }
+
/* Disable LLH */
if (!CHIP_IS_E1(bp))
REG_WR(bp, NIG_REG_LLH0_FUNC_EN + port*8, 0);
@@ -10279,6 +10317,12 @@ static void bnx2x_sp_rtnl_task(struct work_struct *work)
bp->sp_rtnl_state = 0;
smp_mb();
+ /* Immediately indicate link as down */
+ bp->link_vars.link_up = 0;
+ bp->force_link_down = true;
+ netif_carrier_off(bp->dev);
+ BNX2X_ERR("Indicating link is down due to Tx-timeout\n");
+
bnx2x_nic_unload(bp, UNLOAD_NORMAL, true);
bnx2x_nic_load(bp, LOAD_NORMAL);
@@ -10336,6 +10380,9 @@ static void bnx2x_sp_rtnl_task(struct work_struct *work)
&bp->sp_rtnl_state))
bnx2x_update_mng_version(bp);
+ if (test_and_clear_bit(BNX2X_SP_RTNL_UPDATE_SVID, &bp->sp_rtnl_state))
+ bnx2x_handle_update_svid_cmd(bp);
+
if (test_and_clear_bit(BNX2X_SP_RTNL_CHANGE_UDP_PORT,
&bp->sp_rtnl_state)) {
if (bnx2x_udp_port_update(bp)) {
@@ -11727,8 +11774,10 @@ static void bnx2x_get_fcoe_info(struct bnx2x *bp)
* If maximum allowed number of connections is zero -
* disable the feature.
*/
- if (!bp->cnic_eth_dev.max_fcoe_conn)
+ if (!bp->cnic_eth_dev.max_fcoe_conn) {
bp->flags |= NO_FCOE_FLAG;
+ eth_zero_addr(bp->fip_mac);
+ }
}
static void bnx2x_get_cnic_info(struct bnx2x *bp)
@@ -12999,13 +13048,6 @@ static void bnx2x_vlan_configure(struct bnx2x *bp, bool set_rx_mode)
int bnx2x_vlan_reconfigure_vid(struct bnx2x *bp)
{
- struct bnx2x_vlan_entry *vlan;
-
- /* The hw forgot all entries after reload */
- list_for_each_entry(vlan, &bp->vlan_reg, link)
- vlan->hw = false;
- bp->vlan_cnt = 0;
-
/* Don't set rx mode here. Our caller will do it. */
bnx2x_vlan_configure(bp, false);
diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c
index cea6bdc..bf760c9 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.c
@@ -6149,6 +6149,7 @@ static inline int bnx2x_func_send_start(struct bnx2x *bp,
rdata->sd_vlan_tag = cpu_to_le16(start_params->sd_vlan_tag);
rdata->path_id = BP_PATH(bp);
rdata->network_cos_mode = start_params->network_cos_mode;
+ rdata->dmae_cmd_id = BNX2X_FW_DMAE_C;
rdata->vxlan_dst_port = cpu_to_le16(start_params->vxlan_dst_port);
rdata->geneve_dst_port = cpu_to_le16(start_params->geneve_dst_port);
diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.h b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.h
index 0bf2fd4..7a6e82d 100644
--- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.h
+++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_sp.h
@@ -265,6 +265,7 @@ enum {
BNX2X_ETH_MAC,
BNX2X_ISCSI_ETH_MAC,
BNX2X_NETQ_ETH_MAC,
+ BNX2X_VLAN,
BNX2X_DONT_CONSUME_CAM_CREDIT,
BNX2X_DONT_CONSUME_CAM_CREDIT_DEST,
};
@@ -272,7 +273,8 @@ enum {
#define BNX2X_VLAN_MAC_CMP_MASK (1 << BNX2X_UC_LIST_MAC | \
1 << BNX2X_ETH_MAC | \
1 << BNX2X_ISCSI_ETH_MAC | \
- 1 << BNX2X_NETQ_ETH_MAC)
+ 1 << BNX2X_NETQ_ETH_MAC | \
+ 1 << BNX2X_VLAN)
#define BNX2X_VLAN_MAC_CMP_FLAGS(flags) \
((flags) & BNX2X_VLAN_MAC_CMP_MASK)
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 3aa993b..a036f70 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -1666,8 +1666,11 @@ static int bnxt_poll_work(struct bnxt *bp, struct bnxt_napi *bnapi, int budget)
if (TX_CMP_TYPE(txcmp) == CMP_TYPE_TX_L2_CMP) {
tx_pkts++;
/* return full budget so NAPI will complete. */
- if (unlikely(tx_pkts > bp->tx_wake_thresh))
+ if (unlikely(tx_pkts > bp->tx_wake_thresh)) {
rx_pkts = budget;
+ raw_cons = NEXT_RAW_CMP(raw_cons);
+ break;
+ }
} else if ((TX_CMP_TYPE(txcmp) & 0x30) == 0x10) {
rc = bnxt_rx_pkt(bp, bnapi, &raw_cons, &agg_event);
if (likely(rc >= 0))
@@ -1685,7 +1688,7 @@ static int bnxt_poll_work(struct bnxt *bp, struct bnxt_napi *bnapi, int budget)
}
raw_cons = NEXT_RAW_CMP(raw_cons);
- if (rx_pkts == budget)
+ if (rx_pkts && rx_pkts == budget)
break;
}
@@ -1797,8 +1800,12 @@ static int bnxt_poll(struct napi_struct *napi, int budget)
while (1) {
work_done += bnxt_poll_work(bp, bnapi, budget - work_done);
- if (work_done >= budget)
+ if (work_done >= budget) {
+ if (!budget)
+ BNXT_CP_DB_REARM(cpr->cp_doorbell,
+ cpr->cp_raw_cons);
break;
+ }
if (!bnxt_has_work(bp, cpr)) {
napi_complete(napi);
@@ -3401,6 +3408,9 @@ static int bnxt_hwrm_vnic_set_tpa(struct bnxt *bp, u16 vnic_id, u32 tpa_flags)
struct bnxt_vnic_info *vnic = &bp->vnic_info[vnic_id];
struct hwrm_vnic_tpa_cfg_input req = {0};
+ if (vnic->fw_vnic_id == INVALID_HW_RING_ID)
+ return 0;
+
bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_VNIC_TPA_CFG, -1, -1);
if (tpa_flags) {
@@ -5254,6 +5264,9 @@ static int bnxt_update_link(struct bnxt *bp, bool chng_link_state)
}
mutex_unlock(&bp->hwrm_cmd_lock);
+ if (!BNXT_SINGLE_PF(bp))
+ return 0;
+
diff = link_info->support_auto_speeds ^ link_info->advertising;
if ((link_info->support_auto_speeds | diff) !=
link_info->support_auto_speeds) {
@@ -5554,7 +5567,7 @@ static int __bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init)
rc = bnxt_request_irq(bp);
if (rc) {
netdev_err(bp->dev, "bnxt_request_irq err: %x\n", rc);
- goto open_err;
+ goto open_err_irq;
}
}
@@ -5567,7 +5580,9 @@ static int __bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init)
}
if (link_re_init) {
+ mutex_lock(&bp->link_lock);
rc = bnxt_update_phy_setting(bp);
+ mutex_unlock(&bp->link_lock);
if (rc)
netdev_warn(bp->dev, "failed to update phy settings\n");
}
@@ -5587,6 +5602,8 @@ static int __bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init)
open_err:
bnxt_disable_napi(bp);
+
+open_err_irq:
bnxt_del_napi(bp);
open_err_free_mem:
@@ -6215,30 +6232,28 @@ static void bnxt_sp_task(struct work_struct *work)
if (test_and_clear_bit(BNXT_PERIODIC_STATS_SP_EVENT, &bp->sp_event))
bnxt_hwrm_port_qstats(bp);
- /* These functions below will clear BNXT_STATE_IN_SP_TASK. They
- * must be the last functions to be called before exiting.
- */
if (test_and_clear_bit(BNXT_LINK_CHNG_SP_EVENT, &bp->sp_event)) {
- int rc = 0;
+ int rc;
+ mutex_lock(&bp->link_lock);
if (test_and_clear_bit(BNXT_LINK_SPEED_CHNG_SP_EVENT,
&bp->sp_event))
bnxt_hwrm_phy_qcaps(bp);
- bnxt_rtnl_lock_sp(bp);
- if (test_bit(BNXT_STATE_OPEN, &bp->state))
- rc = bnxt_update_link(bp, true);
- bnxt_rtnl_unlock_sp(bp);
+ rc = bnxt_update_link(bp, true);
+ mutex_unlock(&bp->link_lock);
if (rc)
netdev_err(bp->dev, "SP task can't update link (rc: %x)\n",
rc);
}
if (test_and_clear_bit(BNXT_HWRM_PORT_MODULE_SP_EVENT, &bp->sp_event)) {
- bnxt_rtnl_lock_sp(bp);
- if (test_bit(BNXT_STATE_OPEN, &bp->state))
- bnxt_get_port_module_status(bp);
- bnxt_rtnl_unlock_sp(bp);
+ mutex_lock(&bp->link_lock);
+ bnxt_get_port_module_status(bp);
+ mutex_unlock(&bp->link_lock);
}
+ /* These functions below will clear BNXT_STATE_IN_SP_TASK. They
+ * must be the last functions to be called before exiting.
+ */
if (test_and_clear_bit(BNXT_RESET_TASK_SP_EVENT, &bp->sp_event))
bnxt_reset(bp, false);
@@ -6773,6 +6788,7 @@ static int bnxt_probe_phy(struct bnxt *bp)
rc);
return rc;
}
+ mutex_init(&bp->link_lock);
rc = bnxt_update_link(bp, false);
if (rc) {
@@ -6856,11 +6872,11 @@ int bnxt_get_max_rings(struct bnxt *bp, int *max_rx, int *max_tx, bool shared)
int rx, tx, cp;
_bnxt_get_max_rings(bp, &rx, &tx, &cp);
+ *max_rx = rx;
+ *max_tx = tx;
if (!rx || !tx || !cp)
return -ENOMEM;
- *max_rx = rx;
- *max_tx = tx;
return bnxt_trim_rings(bp, max_rx, max_tx, cp, shared);
}
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
index 666bc06..017c10c 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
@@ -1109,6 +1109,10 @@ struct bnxt {
unsigned long *ntp_fltr_bmap;
int ntp_fltr_count;
+ /* To protect link related settings during link changes and
+ * ethtool settings changes.
+ */
+ struct mutex link_lock;
struct bnxt_link_info link_info;
struct ethtool_eee eee;
u32 lpi_tmr_lo;
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
index cde4b96..3a352f7 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c
@@ -793,6 +793,7 @@ static int bnxt_get_link_ksettings(struct net_device *dev,
u32 ethtool_speed;
ethtool_link_ksettings_zero_link_mode(lk_ksettings, supported);
+ mutex_lock(&bp->link_lock);
bnxt_fw_to_ethtool_support_spds(link_info, lk_ksettings);
ethtool_link_ksettings_zero_link_mode(lk_ksettings, advertising);
@@ -840,6 +841,7 @@ static int bnxt_get_link_ksettings(struct net_device *dev,
base->port = PORT_FIBRE;
}
base->phy_address = link_info->phy_addr;
+ mutex_unlock(&bp->link_lock);
return 0;
}
@@ -926,6 +928,7 @@ static int bnxt_set_link_ksettings(struct net_device *dev,
if (!BNXT_SINGLE_PF(bp))
return -EOPNOTSUPP;
+ mutex_lock(&bp->link_lock);
if (base->autoneg == AUTONEG_ENABLE) {
BNXT_ETHTOOL_TO_FW_SPDS(fw_advertising, lk_ksettings,
advertising);
@@ -970,6 +973,7 @@ static int bnxt_set_link_ksettings(struct net_device *dev,
rc = bnxt_hwrm_set_link_setting(bp, set_pause, false);
set_setting_exit:
+ mutex_unlock(&bp->link_lock);
return rc;
}
diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.h b/drivers/net/ethernet/broadcom/genet/bcmgenet.h
index db7f289..3f8858d 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet.h
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.h
@@ -185,6 +185,9 @@ struct bcmgenet_mib_counters {
#define UMAC_MAC1 0x010
#define UMAC_MAX_FRAME_LEN 0x014
+#define UMAC_MODE 0x44
+#define MODE_LINK_STATUS (1 << 5)
+
#define UMAC_EEE_CTRL 0x064
#define EN_LPI_RX_PAUSE (1 << 0)
#define EN_LPI_TX_PFC (1 << 1)
diff --git a/drivers/net/ethernet/broadcom/genet/bcmmii.c b/drivers/net/ethernet/broadcom/genet/bcmmii.c
index 2f92819..9bd90a7 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmmii.c
+++ b/drivers/net/ethernet/broadcom/genet/bcmmii.c
@@ -167,8 +167,14 @@ void bcmgenet_mii_setup(struct net_device *dev)
static int bcmgenet_fixed_phy_link_update(struct net_device *dev,
struct fixed_phy_status *status)
{
- if (dev && dev->phydev && status)
- status->link = dev->phydev->link;
+ struct bcmgenet_priv *priv;
+ u32 reg;
+
+ if (dev && dev->phydev && status) {
+ priv = netdev_priv(dev);
+ reg = bcmgenet_umac_readl(priv, UMAC_MODE);
+ status->link = !!(reg & MODE_LINK_STATUS);
+ }
return 0;
}
@@ -477,7 +483,7 @@ static int bcmgenet_mii_of_init(struct bcmgenet_priv *priv)
if (!compat)
return -ENOMEM;
- priv->mdio_dn = of_find_compatible_node(dn, NULL, compat);
+ priv->mdio_dn = of_get_compatible_child(dn, compat);
kfree(compat);
if (!priv->mdio_dn) {
dev_err(kdev, "unable to find MDIO bus node\n");
diff --git a/drivers/net/ethernet/broadcom/tg3.c b/drivers/net/ethernet/broadcom/tg3.c
index 795a133..c069a04 100644
--- a/drivers/net/ethernet/broadcom/tg3.c
+++ b/drivers/net/ethernet/broadcom/tg3.c
@@ -8720,14 +8720,15 @@ static void tg3_free_consistent(struct tg3 *tp)
tg3_mem_rx_release(tp);
tg3_mem_tx_release(tp);
- /* Protect tg3_get_stats64() from reading freed tp->hw_stats. */
- tg3_full_lock(tp, 0);
+ /* tp->hw_stats can be referenced safely:
+ * 1. under rtnl_lock
+ * 2. or under tp->lock if TG3_FLAG_INIT_COMPLETE is set.
+ */
if (tp->hw_stats) {
dma_free_coherent(&tp->pdev->dev, sizeof(struct tg3_hw_stats),
tp->hw_stats, tp->stats_mapping);
tp->hw_stats = NULL;
}
- tg3_full_unlock(tp);
}
/*
@@ -9275,6 +9276,15 @@ static int tg3_chip_reset(struct tg3 *tp)
tg3_restore_clk(tp);
+ /* Increase the core clock speed to fix tx timeout issue for 5762
+ * with 100Mbps link speed.
+ */
+ if (tg3_asic_rev(tp) == ASIC_REV_5762) {
+ val = tr32(TG3_CPMU_CLCK_ORIDE_ENABLE);
+ tw32(TG3_CPMU_CLCK_ORIDE_ENABLE, val |
+ TG3_CPMU_MAC_ORIDE_ENABLE);
+ }
+
/* Reprobe ASF enable state. */
tg3_flag_clear(tp, ENABLE_ASF);
tp->phy_flags &= ~(TG3_PHYFLG_1G_ON_VAUX_OK |
@@ -12379,6 +12389,7 @@ static int tg3_set_ringparam(struct net_device *dev, struct ethtool_ringparam *e
{
struct tg3 *tp = netdev_priv(dev);
int i, irq_sync = 0, err = 0;
+ bool reset_phy = false;
if ((ering->rx_pending > tp->rx_std_ring_mask) ||
(ering->rx_jumbo_pending > tp->rx_jmb_ring_mask) ||
@@ -12410,7 +12421,13 @@ static int tg3_set_ringparam(struct net_device *dev, struct ethtool_ringparam *e
if (netif_running(dev)) {
tg3_halt(tp, RESET_KIND_SHUTDOWN, 1);
- err = tg3_restart_hw(tp, false);
+ /* Reset PHY to avoid PHY lock up */
+ if (tg3_asic_rev(tp) == ASIC_REV_5717 ||
+ tg3_asic_rev(tp) == ASIC_REV_5719 ||
+ tg3_asic_rev(tp) == ASIC_REV_5720)
+ reset_phy = true;
+
+ err = tg3_restart_hw(tp, reset_phy);
if (!err)
tg3_netif_start(tp);
}
@@ -12444,6 +12461,7 @@ static int tg3_set_pauseparam(struct net_device *dev, struct ethtool_pauseparam
{
struct tg3 *tp = netdev_priv(dev);
int err = 0;
+ bool reset_phy = false;
if (tp->link_config.autoneg == AUTONEG_ENABLE)
tg3_warn_mgmt_link_flap(tp);
@@ -12534,7 +12552,13 @@ static int tg3_set_pauseparam(struct net_device *dev, struct ethtool_pauseparam
if (netif_running(dev)) {
tg3_halt(tp, RESET_KIND_SHUTDOWN, 1);
- err = tg3_restart_hw(tp, false);
+ /* Reset PHY to avoid PHY lock up */
+ if (tg3_asic_rev(tp) == ASIC_REV_5717 ||
+ tg3_asic_rev(tp) == ASIC_REV_5719 ||
+ tg3_asic_rev(tp) == ASIC_REV_5720)
+ reset_phy = true;
+
+ err = tg3_restart_hw(tp, reset_phy);
if (!err)
tg3_netif_start(tp);
}
@@ -14161,7 +14185,7 @@ static struct rtnl_link_stats64 *tg3_get_stats64(struct net_device *dev,
struct tg3 *tp = netdev_priv(dev);
spin_lock_bh(&tp->lock);
- if (!tp->hw_stats) {
+ if (!tp->hw_stats || !tg3_flag(tp, INIT_COMPLETE)) {
*stats = tp->net_stats_prev;
spin_unlock_bh(&tp->lock);
return stats;
diff --git a/drivers/net/ethernet/brocade/bna/bnad_ethtool.c b/drivers/net/ethernet/brocade/bna/bnad_ethtool.c
index 31f61a7..9473d12 100644
--- a/drivers/net/ethernet/brocade/bna/bnad_ethtool.c
+++ b/drivers/net/ethernet/brocade/bna/bnad_ethtool.c
@@ -541,8 +541,8 @@ bnad_get_strings(struct net_device *netdev, u32 stringset, u8 *string)
for (i = 0; i < BNAD_ETHTOOL_STATS_NUM; i++) {
BUG_ON(!(strlen(bnad_net_stats_strings[i]) <
ETH_GSTRING_LEN));
- memcpy(string, bnad_net_stats_strings[i],
- ETH_GSTRING_LEN);
+ strncpy(string, bnad_net_stats_strings[i],
+ ETH_GSTRING_LEN);
string += ETH_GSTRING_LEN;
}
bmap = bna_tx_rid_mask(&bnad->bna);
diff --git a/drivers/net/ethernet/cadence/macb.c b/drivers/net/ethernet/cadence/macb.c
index ec09fce..a0d6402 100644
--- a/drivers/net/ethernet/cadence/macb.c
+++ b/drivers/net/ethernet/cadence/macb.c
@@ -517,7 +517,7 @@ static int macb_halt_tx(struct macb *bp)
if (!(status & MACB_BIT(TGO)))
return 0;
- usleep_range(10, 250);
+ udelay(250);
} while (time_before(halt_time, timeout));
return -ETIMEDOUT;
@@ -1737,6 +1737,7 @@ static void macb_configure_dma(struct macb *bp)
else
dmacfg &= ~GEM_BIT(TXCOEN);
+ dmacfg &= ~GEM_BIT(ADDR64);
#ifdef CONFIG_ARCH_DMA_ADDR_T_64BIT
dmacfg |= GEM_BIT(ADDR64);
#endif
@@ -2861,6 +2862,13 @@ static const struct macb_config at91sam9260_config = {
.init = macb_init,
};
+static const struct macb_config sama5d3macb_config = {
+ .caps = MACB_CAPS_SG_DISABLED
+ | MACB_CAPS_USRIO_HAS_CLKEN | MACB_CAPS_USRIO_DEFAULT_IS_MII_GMII,
+ .clk_init = macb_clk_init,
+ .init = macb_init,
+};
+
static const struct macb_config pc302gem_config = {
.caps = MACB_CAPS_SG_DISABLED | MACB_CAPS_GIGABIT_MODE_AVAILABLE,
.dma_burst_length = 16,
@@ -2925,6 +2933,7 @@ static const struct of_device_id macb_dt_ids[] = {
{ .compatible = "cdns,gem", .data = &pc302gem_config },
{ .compatible = "atmel,sama5d2-gem", .data = &sama5d2_config },
{ .compatible = "atmel,sama5d3-gem", .data = &sama5d3_config },
+ { .compatible = "atmel,sama5d3-macb", .data = &sama5d3macb_config },
{ .compatible = "atmel,sama5d4-gem", .data = &sama5d4_config },
{ .compatible = "cdns,at91rm9200-emac", .data = &emac_config },
{ .compatible = "cdns,emac", .data = &emac_config },
diff --git a/drivers/net/ethernet/cavium/thunder/nic_main.c b/drivers/net/ethernet/cavium/thunder/nic_main.c
index 6677b96..da142f6 100644
--- a/drivers/net/ethernet/cavium/thunder/nic_main.c
+++ b/drivers/net/ethernet/cavium/thunder/nic_main.c
@@ -1371,6 +1371,9 @@ static void nic_remove(struct pci_dev *pdev)
{
struct nicpf *nic = pci_get_drvdata(pdev);
+ if (!nic)
+ return;
+
if (nic->flags & NIC_SRIOV_ENABLED)
pci_disable_sriov(pdev);
diff --git a/drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c b/drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c
index 43da891..ddd1ec8 100644
--- a/drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c
+++ b/drivers/net/ethernet/chelsio/cxgb3/cxgb3_main.c
@@ -50,6 +50,7 @@
#include <linux/stringify.h>
#include <linux/sched.h>
#include <linux/slab.h>
+#include <linux/nospec.h>
#include <asm/uaccess.h>
#include "common.h"
@@ -2149,6 +2150,8 @@ static int cxgb_extension_ioctl(struct net_device *dev, void __user *useraddr)
return -EPERM;
if (copy_from_user(&t, useraddr, sizeof(t)))
return -EFAULT;
+ if (t.cmd != CHELSIO_SET_QSET_PARAMS)
+ return -EINVAL;
if (t.qset_idx >= SGE_QSETS)
return -EINVAL;
if (!in_range(t.intr_lat, 0, M_NEWTIMER) ||
@@ -2248,6 +2251,9 @@ static int cxgb_extension_ioctl(struct net_device *dev, void __user *useraddr)
if (copy_from_user(&t, useraddr, sizeof(t)))
return -EFAULT;
+ if (t.cmd != CHELSIO_GET_QSET_PARAMS)
+ return -EINVAL;
+
/* Display qsets for all ports when offload enabled */
if (test_bit(OFFLOAD_DEVMAP_BIT, &adapter->open_device_map)) {
q1 = 0;
@@ -2259,6 +2265,7 @@ static int cxgb_extension_ioctl(struct net_device *dev, void __user *useraddr)
if (t.qset_idx >= nqsets)
return -EINVAL;
+ t.qset_idx = array_index_nospec(t.qset_idx, nqsets);
q = &adapter->params.sge.qset[q1 + t.qset_idx];
t.rspq_size = q->rspq_size;
@@ -2292,6 +2299,8 @@ static int cxgb_extension_ioctl(struct net_device *dev, void __user *useraddr)
return -EBUSY;
if (copy_from_user(&edata, useraddr, sizeof(edata)))
return -EFAULT;
+ if (edata.cmd != CHELSIO_SET_QSET_NUM)
+ return -EINVAL;
if (edata.val < 1 ||
(edata.val > 1 && !(adapter->flags & USING_MSIX)))
return -EINVAL;
@@ -2332,6 +2341,8 @@ static int cxgb_extension_ioctl(struct net_device *dev, void __user *useraddr)
return -EPERM;
if (copy_from_user(&t, useraddr, sizeof(t)))
return -EFAULT;
+ if (t.cmd != CHELSIO_LOAD_FW)
+ return -EINVAL;
/* Check t.len sanity ? */
fw_data = memdup_user(useraddr + sizeof(t), t.len);
if (IS_ERR(fw_data))
@@ -2355,6 +2366,8 @@ static int cxgb_extension_ioctl(struct net_device *dev, void __user *useraddr)
return -EBUSY;
if (copy_from_user(&m, useraddr, sizeof(m)))
return -EFAULT;
+ if (m.cmd != CHELSIO_SETMTUTAB)
+ return -EINVAL;
if (m.nmtus != NMTUS)
return -EINVAL;
if (m.mtus[0] < 81) /* accommodate SACK */
@@ -2396,6 +2409,8 @@ static int cxgb_extension_ioctl(struct net_device *dev, void __user *useraddr)
return -EBUSY;
if (copy_from_user(&m, useraddr, sizeof(m)))
return -EFAULT;
+ if (m.cmd != CHELSIO_SET_PM)
+ return -EINVAL;
if (!is_power_of_2(m.rx_pg_sz) ||
!is_power_of_2(m.tx_pg_sz))
return -EINVAL; /* not power of 2 */
@@ -2429,6 +2444,8 @@ static int cxgb_extension_ioctl(struct net_device *dev, void __user *useraddr)
return -EIO; /* need the memory controllers */
if (copy_from_user(&t, useraddr, sizeof(t)))
return -EFAULT;
+ if (t.cmd != CHELSIO_GET_MEM)
+ return -EINVAL;
if ((t.addr & 7) || (t.len & 7))
return -EINVAL;
if (t.mem_id == MEM_CM)
@@ -2481,6 +2498,8 @@ static int cxgb_extension_ioctl(struct net_device *dev, void __user *useraddr)
return -EAGAIN;
if (copy_from_user(&t, useraddr, sizeof(t)))
return -EFAULT;
+ if (t.cmd != CHELSIO_SET_TRACE_FILTER)
+ return -EINVAL;
tp = (const struct trace_params *)&t.sip;
if (t.config_tx)
diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
index 3ec32d7..c71a52a 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c
@@ -274,7 +274,7 @@ static void dcb_tx_queue_prio_enable(struct net_device *dev, int enable)
"Can't %s DCB Priority on port %d, TX Queue %d: err=%d\n",
enable ? "set" : "unset", pi->port_id, i, -err);
else
- txq->dcb_prio = value;
+ txq->dcb_prio = enable ? value : 0;
}
}
@@ -836,8 +836,6 @@ static int setup_fw_sge_queues(struct adapter *adap)
err = t4_sge_alloc_rxq(adap, &s->fw_evtq, true, adap->port[0],
adap->msi_idx, NULL, fwevtq_handler, NULL, -1);
- if (err)
- t4_free_sge_resources(adap);
return err;
}
@@ -4940,6 +4938,13 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
if (err)
goto out_free_dev;
+ err = setup_fw_sge_queues(adapter);
+ if (err) {
+ dev_err(adapter->pdev_dev,
+ "FW sge queue allocation failed, err %d", err);
+ goto out_free_dev;
+ }
+
/*
* The card is now ready to go. If any errors occur during device
* registration we do not fail the whole card but rather proceed only
@@ -4983,7 +4988,6 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
}
print_adapter_info(adapter);
- setup_fw_sge_queues(adapter);
return 0;
sriov:
@@ -5035,6 +5039,7 @@ static int init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
#endif
out_free_dev:
+ t4_free_sge_resources(adapter);
free_some_resources(adapter);
if (adapter->flags & USING_MSIX)
free_msix_info(adapter);
diff --git a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.c b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.c
index 2471ff4..23d6c44 100644
--- a/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.c
+++ b/drivers/net/ethernet/chelsio/cxgb4/cxgb4_uld.c
@@ -342,6 +342,7 @@ static void free_queues_uld(struct adapter *adap, unsigned int uld_type)
{
struct sge_uld_rxq_info *rxq_info = adap->sge.uld_rxq_info[uld_type];
+ adap->sge.uld_rxq_info[uld_type] = NULL;
kfree(rxq_info->rspq_id);
kfree(rxq_info->uldrxq);
kfree(rxq_info);
diff --git a/drivers/net/ethernet/cirrus/Kconfig b/drivers/net/ethernet/cirrus/Kconfig
index 5ab9129..ec0b545 100644
--- a/drivers/net/ethernet/cirrus/Kconfig
+++ b/drivers/net/ethernet/cirrus/Kconfig
@@ -19,6 +19,7 @@
config CS89x0
tristate "CS89x0 support"
depends on ISA || EISA || ARM
+ depends on !PPC32
---help---
Support for CS89x0 chipset based Ethernet cards. If you have a
network (Ethernet) card of this type, say Y and read the file
diff --git a/drivers/net/ethernet/cisco/enic/enic_clsf.c b/drivers/net/ethernet/cisco/enic/enic_clsf.c
index 3c677ed..4d9014d 100644
--- a/drivers/net/ethernet/cisco/enic/enic_clsf.c
+++ b/drivers/net/ethernet/cisco/enic/enic_clsf.c
@@ -78,7 +78,6 @@ void enic_rfs_flw_tbl_init(struct enic *enic)
enic->rfs_h.max = enic->config.num_arfs;
enic->rfs_h.free = enic->rfs_h.max;
enic->rfs_h.toclean = 0;
- enic_rfs_timer_start(enic);
}
void enic_rfs_flw_tbl_free(struct enic *enic)
@@ -87,7 +86,6 @@ void enic_rfs_flw_tbl_free(struct enic *enic)
enic_rfs_timer_stop(enic);
spin_lock_bh(&enic->rfs_h.lock);
- enic->rfs_h.free = 0;
for (i = 0; i < (1 << ENIC_RFS_FLW_BITSHIFT); i++) {
struct hlist_head *hhead;
struct hlist_node *tmp;
@@ -98,6 +96,7 @@ void enic_rfs_flw_tbl_free(struct enic *enic)
enic_delfltr(enic, n->fltr_id);
hlist_del(&n->node);
kfree(n);
+ enic->rfs_h.free++;
}
}
spin_unlock_bh(&enic->rfs_h.lock);
diff --git a/drivers/net/ethernet/cisco/enic/enic_main.c b/drivers/net/ethernet/cisco/enic/enic_main.c
index 48f82ab..07282eb 100644
--- a/drivers/net/ethernet/cisco/enic/enic_main.c
+++ b/drivers/net/ethernet/cisco/enic/enic_main.c
@@ -1708,7 +1708,7 @@ static int enic_open(struct net_device *netdev)
{
struct enic *enic = netdev_priv(netdev);
unsigned int i;
- int err;
+ int err, ret;
err = enic_request_intr(enic);
if (err) {
@@ -1726,6 +1726,8 @@ static int enic_open(struct net_device *netdev)
}
for (i = 0; i < enic->rq_count; i++) {
+ /* enable rq before updating rq desc */
+ vnic_rq_enable(&enic->rq[i]);
vnic_rq_fill(&enic->rq[i], enic_rq_alloc_buf);
/* Need at least one buffer on ring to get going */
if (vnic_rq_desc_used(&enic->rq[i]) == 0) {
@@ -1737,8 +1739,6 @@ static int enic_open(struct net_device *netdev)
for (i = 0; i < enic->wq_count; i++)
vnic_wq_enable(&enic->wq[i]);
- for (i = 0; i < enic->rq_count; i++)
- vnic_rq_enable(&enic->rq[i]);
if (!enic_is_dynamic(enic) && !enic_is_sriov_vf(enic))
enic_dev_add_station_addr(enic);
@@ -1760,13 +1760,16 @@ static int enic_open(struct net_device *netdev)
vnic_intr_unmask(&enic->intr[i]);
enic_notify_timer_start(enic);
- enic_rfs_flw_tbl_init(enic);
+ enic_rfs_timer_start(enic);
return 0;
err_out_free_rq:
- for (i = 0; i < enic->rq_count; i++)
- vnic_rq_clean(&enic->rq[i], enic_free_rq_buf);
+ for (i = 0; i < enic->rq_count; i++) {
+ ret = vnic_rq_disable(&enic->rq[i]);
+ if (!ret)
+ vnic_rq_clean(&enic->rq[i], enic_free_rq_buf);
+ }
enic_dev_notify_unset(enic);
err_out_free_intr:
enic_unset_affinity_hint(enic);
@@ -1838,10 +1841,32 @@ static int enic_stop(struct net_device *netdev)
return 0;
}
+static int _enic_change_mtu(struct net_device *netdev, int new_mtu)
+{
+ bool running = netif_running(netdev);
+ int err = 0;
+
+ ASSERT_RTNL();
+ if (running) {
+ err = enic_stop(netdev);
+ if (err)
+ return err;
+ }
+
+ netdev->mtu = new_mtu;
+
+ if (running) {
+ err = enic_open(netdev);
+ if (err)
+ return err;
+ }
+
+ return 0;
+}
+
static int enic_change_mtu(struct net_device *netdev, int new_mtu)
{
struct enic *enic = netdev_priv(netdev);
- int running = netif_running(netdev);
if (new_mtu < ENIC_MIN_MTU || new_mtu > ENIC_MAX_MTU)
return -EINVAL;
@@ -1849,20 +1874,12 @@ static int enic_change_mtu(struct net_device *netdev, int new_mtu)
if (enic_is_dynamic(enic) || enic_is_sriov_vf(enic))
return -EOPNOTSUPP;
- if (running)
- enic_stop(netdev);
-
- netdev->mtu = new_mtu;
-
if (netdev->mtu > enic->port_mtu)
netdev_warn(netdev,
- "interface MTU (%d) set higher than port MTU (%d)\n",
- netdev->mtu, enic->port_mtu);
+ "interface MTU (%d) set higher than port MTU (%d)\n",
+ netdev->mtu, enic->port_mtu);
- if (running)
- enic_open(netdev);
-
- return 0;
+ return _enic_change_mtu(netdev, new_mtu);
}
static void enic_change_mtu_work(struct work_struct *work)
@@ -1870,47 +1887,9 @@ static void enic_change_mtu_work(struct work_struct *work)
struct enic *enic = container_of(work, struct enic, change_mtu_work);
struct net_device *netdev = enic->netdev;
int new_mtu = vnic_dev_mtu(enic->vdev);
- int err;
- unsigned int i;
-
- new_mtu = max_t(int, ENIC_MIN_MTU, min_t(int, ENIC_MAX_MTU, new_mtu));
rtnl_lock();
-
- /* Stop RQ */
- del_timer_sync(&enic->notify_timer);
-
- for (i = 0; i < enic->rq_count; i++)
- napi_disable(&enic->napi[i]);
-
- vnic_intr_mask(&enic->intr[0]);
- enic_synchronize_irqs(enic);
- err = vnic_rq_disable(&enic->rq[0]);
- if (err) {
- rtnl_unlock();
- netdev_err(netdev, "Unable to disable RQ.\n");
- return;
- }
- vnic_rq_clean(&enic->rq[0], enic_free_rq_buf);
- vnic_cq_clean(&enic->cq[0]);
- vnic_intr_clean(&enic->intr[0]);
-
- /* Fill RQ with new_mtu-sized buffers */
- netdev->mtu = new_mtu;
- vnic_rq_fill(&enic->rq[0], enic_rq_alloc_buf);
- /* Need at least one buffer on ring to get going */
- if (vnic_rq_desc_used(&enic->rq[0]) == 0) {
- rtnl_unlock();
- netdev_err(netdev, "Unable to alloc receive buffers.\n");
- return;
- }
-
- /* Start RQ */
- vnic_rq_enable(&enic->rq[0]);
- napi_enable(&enic->napi[0]);
- vnic_intr_unmask(&enic->intr[0]);
- enic_notify_timer_start(enic);
-
+ (void)_enic_change_mtu(netdev, new_mtu);
rtnl_unlock();
netdev_info(netdev, "interface MTU set as %d\n", netdev->mtu);
@@ -2537,11 +2516,11 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
pci_set_master(pdev);
/* Query PCI controller on system for DMA addressing
- * limitation for the device. Try 64-bit first, and
+ * limitation for the device. Try 47-bit first, and
* fail to 32-bit.
*/
- err = pci_set_dma_mask(pdev, DMA_BIT_MASK(64));
+ err = pci_set_dma_mask(pdev, DMA_BIT_MASK(47));
if (err) {
err = pci_set_dma_mask(pdev, DMA_BIT_MASK(32));
if (err) {
@@ -2555,10 +2534,10 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
goto err_out_release_regions;
}
} else {
- err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64));
+ err = pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(47));
if (err) {
dev_err(dev, "Unable to obtain %u-bit DMA "
- "for consistent allocations, aborting\n", 64);
+ "for consistent allocations, aborting\n", 47);
goto err_out_release_regions;
}
using_dac = 1;
@@ -2688,6 +2667,7 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
enic->notify_timer.function = enic_notify_timer;
enic->notify_timer.data = (unsigned long)enic;
+ enic_rfs_flw_tbl_init(enic);
enic_set_rx_coal_setting(enic);
INIT_WORK(&enic->reset, enic_reset);
INIT_WORK(&enic->tx_hang_reset, enic_tx_hang_reset);
@@ -2700,7 +2680,6 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
*/
enic->port_mtu = enic->config.mtu;
- (void)enic_change_mtu(netdev, enic->port_mtu);
err = enic_set_mac_addr(netdev, enic->mac_addr);
if (err) {
@@ -2750,6 +2729,7 @@ static int enic_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
netdev->features |= NETIF_F_HIGHDMA;
netdev->priv_flags |= IFF_UNICAST_FLT;
+ netdev->mtu = enic->port_mtu;
err = register_netdev(netdev);
if (err) {
diff --git a/drivers/net/ethernet/emulex/benet/be_cmds.c b/drivers/net/ethernet/emulex/benet/be_cmds.c
index 30e8550..8887dd3 100644
--- a/drivers/net/ethernet/emulex/benet/be_cmds.c
+++ b/drivers/net/ethernet/emulex/benet/be_cmds.c
@@ -4500,7 +4500,7 @@ int be_cmd_get_profile_config(struct be_adapter *adapter,
port_res->max_vfs += le16_to_cpu(pcie->num_vfs);
}
}
- return status;
+ goto err;
}
pcie = be_get_pcie_desc(resp->func_param, desc_count,
diff --git a/drivers/net/ethernet/faraday/ftmac100.c b/drivers/net/ethernet/faraday/ftmac100.c
index dce5f7b..05e1f92 100644
--- a/drivers/net/ethernet/faraday/ftmac100.c
+++ b/drivers/net/ethernet/faraday/ftmac100.c
@@ -865,11 +865,10 @@ static irqreturn_t ftmac100_interrupt(int irq, void *dev_id)
struct net_device *netdev = dev_id;
struct ftmac100 *priv = netdev_priv(netdev);
- if (likely(netif_running(netdev))) {
- /* Disable interrupts for polling */
- ftmac100_disable_all_int(priv);
+ /* Disable interrupts for polling */
+ ftmac100_disable_all_int(priv);
+ if (likely(netif_running(netdev)))
napi_schedule(&priv->napi);
- }
return IRQ_HANDLED;
}
diff --git a/drivers/net/ethernet/freescale/fec_main.c b/drivers/net/ethernet/freescale/fec_main.c
index fe00f71..051ecc7 100644
--- a/drivers/net/ethernet/freescale/fec_main.c
+++ b/drivers/net/ethernet/freescale/fec_main.c
@@ -1152,7 +1152,7 @@ static void fec_enet_timeout_work(struct work_struct *work)
napi_disable(&fep->napi);
netif_tx_lock_bh(ndev);
fec_restart(ndev);
- netif_wake_queue(ndev);
+ netif_tx_wake_all_queues(ndev);
netif_tx_unlock_bh(ndev);
napi_enable(&fep->napi);
}
@@ -1267,7 +1267,7 @@ fec_enet_tx_queue(struct net_device *ndev, u16 queue_id)
/* Since we have freed up a buffer, the ring is no longer full
*/
- if (netif_queue_stopped(ndev)) {
+ if (netif_tx_queue_stopped(nq)) {
entries_free = fec_enet_get_free_txdesc_num(txq);
if (entries_free >= txq->tx_wake_threshold)
netif_tx_wake_queue(nq);
@@ -1744,7 +1744,7 @@ static void fec_enet_adjust_link(struct net_device *ndev)
napi_disable(&fep->napi);
netif_tx_lock_bh(ndev);
fec_restart(ndev);
- netif_wake_queue(ndev);
+ netif_tx_wake_all_queues(ndev);
netif_tx_unlock_bh(ndev);
napi_enable(&fep->napi);
}
@@ -2247,7 +2247,7 @@ static int fec_enet_set_pauseparam(struct net_device *ndev,
napi_disable(&fep->napi);
netif_tx_lock_bh(ndev);
fec_restart(ndev);
- netif_wake_queue(ndev);
+ netif_tx_wake_all_queues(ndev);
netif_tx_unlock_bh(ndev);
napi_enable(&fep->napi);
}
diff --git a/drivers/net/ethernet/freescale/fman/fman.c b/drivers/net/ethernet/freescale/fman/fman.c
index dafd9e1..380c4a2 100644
--- a/drivers/net/ethernet/freescale/fman/fman.c
+++ b/drivers/net/ethernet/freescale/fman/fman.c
@@ -2817,7 +2817,7 @@ static struct fman *read_dts_node(struct platform_device *of_dev)
if (!muram_node) {
dev_err(&of_dev->dev, "%s: could not find MURAM node\n",
__func__);
- goto fman_node_put;
+ goto fman_free;
}
err = of_address_to_resource(muram_node, 0,
@@ -2826,11 +2826,10 @@ static struct fman *read_dts_node(struct platform_device *of_dev)
of_node_put(muram_node);
dev_err(&of_dev->dev, "%s: of_address_to_resource() = %d\n",
__func__, err);
- goto fman_node_put;
+ goto fman_free;
}
of_node_put(muram_node);
- of_node_put(fm_node);
err = devm_request_irq(&of_dev->dev, irq, fman_irq, 0, "fman", fman);
if (err < 0) {
diff --git a/drivers/net/ethernet/freescale/fman/fman_dtsec.c b/drivers/net/ethernet/freescale/fman/fman_dtsec.c
index c88918c..641b916 100644
--- a/drivers/net/ethernet/freescale/fman/fman_dtsec.c
+++ b/drivers/net/ethernet/freescale/fman/fman_dtsec.c
@@ -1036,7 +1036,7 @@ int dtsec_add_hash_mac_address(struct fman_mac *dtsec, enet_addr_t *eth_addr)
set_bucket(dtsec->regs, bucket, true);
/* Create element to be added to the driver hash table */
- hash_entry = kmalloc(sizeof(*hash_entry), GFP_KERNEL);
+ hash_entry = kmalloc(sizeof(*hash_entry), GFP_ATOMIC);
if (!hash_entry)
return -ENOMEM;
hash_entry->addr = addr;
diff --git a/drivers/net/ethernet/freescale/fs_enet/fs_enet-main.c b/drivers/net/ethernet/freescale/fs_enet/fs_enet-main.c
index 4b86260..8b66551 100644
--- a/drivers/net/ethernet/freescale/fs_enet/fs_enet-main.c
+++ b/drivers/net/ethernet/freescale/fs_enet/fs_enet-main.c
@@ -613,9 +613,11 @@ static int fs_enet_start_xmit(struct sk_buff *skb, struct net_device *dev)
return NETDEV_TX_OK;
}
-static void fs_timeout(struct net_device *dev)
+static void fs_timeout_work(struct work_struct *work)
{
- struct fs_enet_private *fep = netdev_priv(dev);
+ struct fs_enet_private *fep = container_of(work, struct fs_enet_private,
+ timeout_work);
+ struct net_device *dev = fep->ndev;
unsigned long flags;
int wake = 0;
@@ -627,7 +629,6 @@ static void fs_timeout(struct net_device *dev)
phy_stop(dev->phydev);
(*fep->ops->stop)(dev);
(*fep->ops->restart)(dev);
- phy_start(dev->phydev);
}
phy_start(dev->phydev);
@@ -639,6 +640,13 @@ static void fs_timeout(struct net_device *dev)
netif_wake_queue(dev);
}
+static void fs_timeout(struct net_device *dev)
+{
+ struct fs_enet_private *fep = netdev_priv(dev);
+
+ schedule_work(&fep->timeout_work);
+}
+
/*-----------------------------------------------------------------------------
* generic link-change handler - should be sufficient for most cases
*-----------------------------------------------------------------------------*/
@@ -759,6 +767,7 @@ static int fs_enet_close(struct net_device *dev)
netif_stop_queue(dev);
netif_carrier_off(dev);
napi_disable(&fep->napi);
+ cancel_work_sync(&fep->timeout_work);
phy_stop(dev->phydev);
spin_lock_irqsave(&fep->lock, flags);
@@ -1033,6 +1042,7 @@ static int fs_enet_probe(struct platform_device *ofdev)
ndev->netdev_ops = &fs_enet_netdev_ops;
ndev->watchdog_timeo = 2 * HZ;
+ INIT_WORK(&fep->timeout_work, fs_timeout_work);
netif_napi_add(ndev, &fep->napi, fs_enet_napi, fpi->napi_weight);
ndev->ethtool_ops = &fs_ethtool_ops;
diff --git a/drivers/net/ethernet/freescale/fs_enet/fs_enet.h b/drivers/net/ethernet/freescale/fs_enet/fs_enet.h
index fee24c8..0e4e024 100644
--- a/drivers/net/ethernet/freescale/fs_enet/fs_enet.h
+++ b/drivers/net/ethernet/freescale/fs_enet/fs_enet.h
@@ -124,6 +124,7 @@ struct fs_enet_private {
spinlock_t lock; /* during all ops except TX pckt processing */
spinlock_t tx_lock; /* during fs_start_xmit and fs_tx */
struct fs_platform_info *fpi;
+ struct work_struct timeout_work;
const struct fs_ops *ops;
int rx_ring, tx_ring;
dma_addr_t ring_mem_addr;
diff --git a/drivers/net/ethernet/freescale/gianfar.c b/drivers/net/ethernet/freescale/gianfar.c
index e3b41ba..60bd1b3 100644
--- a/drivers/net/ethernet/freescale/gianfar.c
+++ b/drivers/net/ethernet/freescale/gianfar.c
@@ -2935,7 +2935,7 @@ static irqreturn_t gfar_transmit(int irq, void *grp_id)
static bool gfar_add_rx_frag(struct gfar_rx_buff *rxb, u32 lstatus,
struct sk_buff *skb, bool first)
{
- unsigned int size = lstatus & BD_LENGTH_MASK;
+ int size = lstatus & BD_LENGTH_MASK;
struct page *page = rxb->page;
bool last = !!(lstatus & BD_LFLAG(RXBD_LAST));
@@ -2950,11 +2950,16 @@ static bool gfar_add_rx_frag(struct gfar_rx_buff *rxb, u32 lstatus,
if (last)
size -= skb->len;
- /* in case the last fragment consisted only of the FCS */
+ /* Add the last fragment if it contains something other than
+ * the FCS, otherwise drop it and trim off any part of the FCS
+ * that was already received.
+ */
if (size > 0)
skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, page,
rxb->page_offset + RXBUF_ALIGNMENT,
size, GFAR_RXB_TRUESIZE);
+ else if (size < 0)
+ pskb_trim(skb, skb->len + size);
}
/* try reuse page */
@@ -3070,9 +3075,6 @@ static void gfar_process_frame(struct net_device *ndev, struct sk_buff *skb)
if (ndev->features & NETIF_F_RXCSUM)
gfar_rx_checksum(skb, fcb);
- /* Tell the skb what kind of packet this is */
- skb->protocol = eth_type_trans(skb, ndev);
-
/* There's need to check for NETIF_F_HW_VLAN_CTAG_RX here.
* Even if vlan rx accel is disabled, on some chips
* RXFCB_VLN is pseudo randomly set.
@@ -3143,13 +3145,15 @@ int gfar_clean_rx_ring(struct gfar_priv_rx_q *rx_queue, int rx_work_limit)
continue;
}
+ gfar_process_frame(ndev, skb);
+
/* Increment the number of packets */
total_pkts++;
total_bytes += skb->len;
skb_record_rx_queue(skb, rx_queue->qindex);
- gfar_process_frame(ndev, skb);
+ skb->protocol = eth_type_trans(skb, ndev);
/* Send the packet up the stack */
napi_gro_receive(&rx_queue->grp->napi_rx, skb);
diff --git a/drivers/net/ethernet/hisilicon/hip04_eth.c b/drivers/net/ethernet/hisilicon/hip04_eth.c
index 3977889..b5d18d9 100644
--- a/drivers/net/ethernet/hisilicon/hip04_eth.c
+++ b/drivers/net/ethernet/hisilicon/hip04_eth.c
@@ -922,10 +922,8 @@ static int hip04_mac_probe(struct platform_device *pdev)
}
ret = register_netdev(ndev);
- if (ret) {
- free_netdev(ndev);
+ if (ret)
goto alloc_fail;
- }
return 0;
diff --git a/drivers/net/ethernet/hisilicon/hns/hnae.c b/drivers/net/ethernet/hisilicon/hns/hnae.c
index b6ed818..06bc863 100644
--- a/drivers/net/ethernet/hisilicon/hns/hnae.c
+++ b/drivers/net/ethernet/hisilicon/hns/hnae.c
@@ -80,7 +80,7 @@ static void hnae_unmap_buffer(struct hnae_ring *ring, struct hnae_desc_cb *cb)
if (cb->type == DESC_TYPE_SKB)
dma_unmap_single(ring_to_dev(ring), cb->dma, cb->length,
ring_to_dma_dir(ring));
- else
+ else if (cb->length)
dma_unmap_page(ring_to_dev(ring), cb->dma, cb->length,
ring_to_dma_dir(ring));
}
diff --git a/drivers/net/ethernet/hisilicon/hns/hnae.h b/drivers/net/ethernet/hisilicon/hns/hnae.h
index e093cbf..f9d6845 100644
--- a/drivers/net/ethernet/hisilicon/hns/hnae.h
+++ b/drivers/net/ethernet/hisilicon/hns/hnae.h
@@ -213,10 +213,10 @@ struct hnae_desc_cb {
/* priv data for the desc, e.g. skb when use with ip stack*/
void *priv;
- u16 page_offset;
- u16 reuse_flag;
+ u32 page_offset;
+ u32 length; /* length of the buffer */
- u16 length; /* length of the buffer */
+ u16 reuse_flag;
/* desc type, used by the ring user to mark the type of the priv data */
u16 type;
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_ae_adapt.c b/drivers/net/ethernet/hisilicon/hns/hns_ae_adapt.c
index b7c8433..0b4d90c 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_ae_adapt.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_ae_adapt.c
@@ -290,6 +290,9 @@ void hns_ae_stop(struct hnae_handle *handle)
hns_ae_ring_enable_all(handle, 0);
+ /* clean rx fbd. */
+ hns_rcb_wait_fbd_clean(handle->qs, handle->q_num, RCB_INT_FLAG_RX);
+
(void)hns_mac_vm_config_bc_en(mac_cb, 0, false);
}
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_gmac.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_gmac.c
index 02a03bc..9596747 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_gmac.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_gmac.c
@@ -67,11 +67,14 @@ static void hns_gmac_enable(void *mac_drv, enum mac_commom_mode mode)
struct mac_driver *drv = (struct mac_driver *)mac_drv;
/*enable GE rX/tX */
- if ((mode == MAC_COMM_MODE_TX) || (mode == MAC_COMM_MODE_RX_AND_TX))
+ if (mode == MAC_COMM_MODE_TX || mode == MAC_COMM_MODE_RX_AND_TX)
dsaf_set_dev_bit(drv, GMAC_PORT_EN_REG, GMAC_PORT_TX_EN_B, 1);
- if ((mode == MAC_COMM_MODE_RX) || (mode == MAC_COMM_MODE_RX_AND_TX))
+ if (mode == MAC_COMM_MODE_RX || mode == MAC_COMM_MODE_RX_AND_TX) {
+ /* enable rx pcs */
+ dsaf_set_dev_bit(drv, GMAC_PCS_RX_EN_REG, 0, 0);
dsaf_set_dev_bit(drv, GMAC_PORT_EN_REG, GMAC_PORT_RX_EN_B, 1);
+ }
}
static void hns_gmac_disable(void *mac_drv, enum mac_commom_mode mode)
@@ -79,11 +82,14 @@ static void hns_gmac_disable(void *mac_drv, enum mac_commom_mode mode)
struct mac_driver *drv = (struct mac_driver *)mac_drv;
/*disable GE rX/tX */
- if ((mode == MAC_COMM_MODE_TX) || (mode == MAC_COMM_MODE_RX_AND_TX))
+ if (mode == MAC_COMM_MODE_TX || mode == MAC_COMM_MODE_RX_AND_TX)
dsaf_set_dev_bit(drv, GMAC_PORT_EN_REG, GMAC_PORT_TX_EN_B, 0);
- if ((mode == MAC_COMM_MODE_RX) || (mode == MAC_COMM_MODE_RX_AND_TX))
+ if (mode == MAC_COMM_MODE_RX || mode == MAC_COMM_MODE_RX_AND_TX) {
+ /* disable rx pcs */
+ dsaf_set_dev_bit(drv, GMAC_PCS_RX_EN_REG, 0, 1);
dsaf_set_dev_bit(drv, GMAC_PORT_EN_REG, GMAC_PORT_RX_EN_B, 0);
+ }
}
/* hns_gmac_get_en - get port enable
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c
index ec8c738..b6429be 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_mac.c
@@ -724,6 +724,17 @@ static void hns_mac_register_phy(struct hns_mac_cb *mac_cb)
mac_cb->mac_id, addr);
}
+static void hns_mac_remove_phydev(struct hns_mac_cb *mac_cb)
+{
+ if (!to_acpi_device_node(mac_cb->fw_port) || !mac_cb->phy_dev)
+ return;
+
+ phy_device_remove(mac_cb->phy_dev);
+ phy_device_free(mac_cb->phy_dev);
+
+ mac_cb->phy_dev = NULL;
+}
+
#define MAC_MEDIA_TYPE_MAX_LEN 16
static const struct {
@@ -1030,7 +1041,11 @@ void hns_mac_uninit(struct dsaf_device *dsaf_dev)
int max_port_num = hns_mac_get_max_port_num(dsaf_dev);
for (i = 0; i < max_port_num; i++) {
+ if (!dsaf_dev->mac_cb[i])
+ continue;
+
dsaf_dev->misc_op->cpld_reset_led(dsaf_dev->mac_cb[i]);
+ hns_mac_remove_phydev(dsaf_dev->mac_cb[i]);
dsaf_dev->mac_cb[i] = NULL;
}
}
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c
index 8ea3d95..5bb019d 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_main.c
@@ -2163,9 +2163,9 @@ void hns_dsaf_update_stats(struct dsaf_device *dsaf_dev, u32 node_num)
DSAF_INODE_LOCAL_ADDR_FALSE_NUM_0_REG + 0x80 * (u64)node_num);
hw_stats->vlan_drop += dsaf_read_dev(dsaf_dev,
- DSAF_INODE_SW_VLAN_TAG_DISC_0_REG + 0x80 * (u64)node_num);
+ DSAF_INODE_SW_VLAN_TAG_DISC_0_REG + 4 * (u64)node_num);
hw_stats->stp_drop += dsaf_read_dev(dsaf_dev,
- DSAF_INODE_IN_DATA_STP_DISC_0_REG + 0x80 * (u64)node_num);
+ DSAF_INODE_IN_DATA_STP_DISC_0_REG + 4 * (u64)node_num);
/* pfc pause frame statistics stored in dsaf inode*/
if ((node_num < DSAF_SERVICE_NW_NUM) && !is_ver1) {
@@ -2282,237 +2282,237 @@ void hns_dsaf_get_regs(struct dsaf_device *ddev, u32 port, void *data)
DSAF_INODE_BD_ORDER_STATUS_0_REG + j * 4);
p[223 + i] = dsaf_read_dev(ddev,
DSAF_INODE_SW_VLAN_TAG_DISC_0_REG + j * 4);
- p[224 + i] = dsaf_read_dev(ddev,
+ p[226 + i] = dsaf_read_dev(ddev,
DSAF_INODE_IN_DATA_STP_DISC_0_REG + j * 4);
}
- p[227] = dsaf_read_dev(ddev, DSAF_INODE_GE_FC_EN_0_REG + port * 4);
+ p[229] = dsaf_read_dev(ddev, DSAF_INODE_GE_FC_EN_0_REG + port * 4);
for (i = 0; i < DSAF_INODE_NUM / DSAF_COMM_CHN; i++) {
j = i * DSAF_COMM_CHN + port;
- p[228 + i] = dsaf_read_dev(ddev,
+ p[230 + i] = dsaf_read_dev(ddev,
DSAF_INODE_VC0_IN_PKT_NUM_0_REG + j * 4);
}
- p[231] = dsaf_read_dev(ddev,
- DSAF_INODE_VC1_IN_PKT_NUM_0_REG + port * 4);
+ p[233] = dsaf_read_dev(ddev,
+ DSAF_INODE_VC1_IN_PKT_NUM_0_REG + port * 0x80);
/* dsaf inode registers */
for (i = 0; i < HNS_DSAF_SBM_NUM(ddev) / DSAF_COMM_CHN; i++) {
j = i * DSAF_COMM_CHN + port;
- p[232 + i] = dsaf_read_dev(ddev,
+ p[234 + i] = dsaf_read_dev(ddev,
DSAF_SBM_CFG_REG_0_REG + j * 0x80);
- p[235 + i] = dsaf_read_dev(ddev,
+ p[237 + i] = dsaf_read_dev(ddev,
DSAF_SBM_BP_CFG_0_XGE_REG_0_REG + j * 0x80);
- p[238 + i] = dsaf_read_dev(ddev,
+ p[240 + i] = dsaf_read_dev(ddev,
DSAF_SBM_BP_CFG_1_REG_0_REG + j * 0x80);
- p[241 + i] = dsaf_read_dev(ddev,
+ p[243 + i] = dsaf_read_dev(ddev,
DSAF_SBM_BP_CFG_2_XGE_REG_0_REG + j * 0x80);
- p[244 + i] = dsaf_read_dev(ddev,
+ p[246 + i] = dsaf_read_dev(ddev,
DSAF_SBM_FREE_CNT_0_0_REG + j * 0x80);
- p[245 + i] = dsaf_read_dev(ddev,
+ p[249 + i] = dsaf_read_dev(ddev,
DSAF_SBM_FREE_CNT_1_0_REG + j * 0x80);
- p[248 + i] = dsaf_read_dev(ddev,
+ p[252 + i] = dsaf_read_dev(ddev,
DSAF_SBM_BP_CNT_0_0_REG + j * 0x80);
- p[251 + i] = dsaf_read_dev(ddev,
+ p[255 + i] = dsaf_read_dev(ddev,
DSAF_SBM_BP_CNT_1_0_REG + j * 0x80);
- p[254 + i] = dsaf_read_dev(ddev,
+ p[258 + i] = dsaf_read_dev(ddev,
DSAF_SBM_BP_CNT_2_0_REG + j * 0x80);
- p[257 + i] = dsaf_read_dev(ddev,
+ p[261 + i] = dsaf_read_dev(ddev,
DSAF_SBM_BP_CNT_3_0_REG + j * 0x80);
- p[260 + i] = dsaf_read_dev(ddev,
+ p[264 + i] = dsaf_read_dev(ddev,
DSAF_SBM_INER_ST_0_REG + j * 0x80);
- p[263 + i] = dsaf_read_dev(ddev,
+ p[267 + i] = dsaf_read_dev(ddev,
DSAF_SBM_MIB_REQ_FAILED_TC_0_REG + j * 0x80);
- p[266 + i] = dsaf_read_dev(ddev,
+ p[270 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_INPORT_CNT_0_REG + j * 0x80);
- p[269 + i] = dsaf_read_dev(ddev,
+ p[273 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_DROP_CNT_0_REG + j * 0x80);
- p[272 + i] = dsaf_read_dev(ddev,
+ p[276 + i] = dsaf_read_dev(ddev,
DSAF_SBM_INF_OUTPORT_CNT_0_REG + j * 0x80);
- p[275 + i] = dsaf_read_dev(ddev,
+ p[279 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_INPORT_TC0_CNT_0_REG + j * 0x80);
- p[278 + i] = dsaf_read_dev(ddev,
+ p[282 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_INPORT_TC1_CNT_0_REG + j * 0x80);
- p[281 + i] = dsaf_read_dev(ddev,
+ p[285 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_INPORT_TC2_CNT_0_REG + j * 0x80);
- p[284 + i] = dsaf_read_dev(ddev,
+ p[288 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_INPORT_TC3_CNT_0_REG + j * 0x80);
- p[287 + i] = dsaf_read_dev(ddev,
+ p[291 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_INPORT_TC4_CNT_0_REG + j * 0x80);
- p[290 + i] = dsaf_read_dev(ddev,
+ p[294 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_INPORT_TC5_CNT_0_REG + j * 0x80);
- p[293 + i] = dsaf_read_dev(ddev,
+ p[297 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_INPORT_TC6_CNT_0_REG + j * 0x80);
- p[296 + i] = dsaf_read_dev(ddev,
+ p[300 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_INPORT_TC7_CNT_0_REG + j * 0x80);
- p[299 + i] = dsaf_read_dev(ddev,
+ p[303 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_REQ_CNT_0_REG + j * 0x80);
- p[302 + i] = dsaf_read_dev(ddev,
+ p[306 + i] = dsaf_read_dev(ddev,
DSAF_SBM_LNK_RELS_CNT_0_REG + j * 0x80);
- p[305 + i] = dsaf_read_dev(ddev,
+ p[309 + i] = dsaf_read_dev(ddev,
DSAF_SBM_BP_CFG_3_REG_0_REG + j * 0x80);
- p[308 + i] = dsaf_read_dev(ddev,
+ p[312 + i] = dsaf_read_dev(ddev,
DSAF_SBM_BP_CFG_4_REG_0_REG + j * 0x80);
}
/* dsaf onode registers */
for (i = 0; i < DSAF_XOD_NUM; i++) {
- p[311 + i] = dsaf_read_dev(ddev,
+ p[315 + i] = dsaf_read_dev(ddev,
DSAF_XOD_ETS_TSA_TC0_TC3_CFG_0_REG + i * 0x90);
- p[319 + i] = dsaf_read_dev(ddev,
+ p[323 + i] = dsaf_read_dev(ddev,
DSAF_XOD_ETS_TSA_TC4_TC7_CFG_0_REG + i * 0x90);
- p[327 + i] = dsaf_read_dev(ddev,
+ p[331 + i] = dsaf_read_dev(ddev,
DSAF_XOD_ETS_BW_TC0_TC3_CFG_0_REG + i * 0x90);
- p[335 + i] = dsaf_read_dev(ddev,
+ p[339 + i] = dsaf_read_dev(ddev,
DSAF_XOD_ETS_BW_TC4_TC7_CFG_0_REG + i * 0x90);
- p[343 + i] = dsaf_read_dev(ddev,
+ p[347 + i] = dsaf_read_dev(ddev,
DSAF_XOD_ETS_BW_OFFSET_CFG_0_REG + i * 0x90);
- p[351 + i] = dsaf_read_dev(ddev,
+ p[355 + i] = dsaf_read_dev(ddev,
DSAF_XOD_ETS_TOKEN_CFG_0_REG + i * 0x90);
}
- p[359] = dsaf_read_dev(ddev, DSAF_XOD_PFS_CFG_0_0_REG + port * 0x90);
- p[360] = dsaf_read_dev(ddev, DSAF_XOD_PFS_CFG_1_0_REG + port * 0x90);
- p[361] = dsaf_read_dev(ddev, DSAF_XOD_PFS_CFG_2_0_REG + port * 0x90);
+ p[363] = dsaf_read_dev(ddev, DSAF_XOD_PFS_CFG_0_0_REG + port * 0x90);
+ p[364] = dsaf_read_dev(ddev, DSAF_XOD_PFS_CFG_1_0_REG + port * 0x90);
+ p[365] = dsaf_read_dev(ddev, DSAF_XOD_PFS_CFG_2_0_REG + port * 0x90);
for (i = 0; i < DSAF_XOD_BIG_NUM / DSAF_COMM_CHN; i++) {
j = i * DSAF_COMM_CHN + port;
- p[362 + i] = dsaf_read_dev(ddev,
+ p[366 + i] = dsaf_read_dev(ddev,
DSAF_XOD_GNT_L_0_REG + j * 0x90);
- p[365 + i] = dsaf_read_dev(ddev,
+ p[369 + i] = dsaf_read_dev(ddev,
DSAF_XOD_GNT_H_0_REG + j * 0x90);
- p[368 + i] = dsaf_read_dev(ddev,
+ p[372 + i] = dsaf_read_dev(ddev,
DSAF_XOD_CONNECT_STATE_0_REG + j * 0x90);
- p[371 + i] = dsaf_read_dev(ddev,
+ p[375 + i] = dsaf_read_dev(ddev,
DSAF_XOD_RCVPKT_CNT_0_REG + j * 0x90);
- p[374 + i] = dsaf_read_dev(ddev,
+ p[378 + i] = dsaf_read_dev(ddev,
DSAF_XOD_RCVTC0_CNT_0_REG + j * 0x90);
- p[377 + i] = dsaf_read_dev(ddev,
+ p[381 + i] = dsaf_read_dev(ddev,
DSAF_XOD_RCVTC1_CNT_0_REG + j * 0x90);
- p[380 + i] = dsaf_read_dev(ddev,
+ p[384 + i] = dsaf_read_dev(ddev,
DSAF_XOD_RCVTC2_CNT_0_REG + j * 0x90);
- p[383 + i] = dsaf_read_dev(ddev,
+ p[387 + i] = dsaf_read_dev(ddev,
DSAF_XOD_RCVTC3_CNT_0_REG + j * 0x90);
- p[386 + i] = dsaf_read_dev(ddev,
+ p[390 + i] = dsaf_read_dev(ddev,
DSAF_XOD_RCVVC0_CNT_0_REG + j * 0x90);
- p[389 + i] = dsaf_read_dev(ddev,
+ p[393 + i] = dsaf_read_dev(ddev,
DSAF_XOD_RCVVC1_CNT_0_REG + j * 0x90);
}
- p[392] = dsaf_read_dev(ddev,
- DSAF_XOD_XGE_RCVIN0_CNT_0_REG + port * 0x90);
- p[393] = dsaf_read_dev(ddev,
- DSAF_XOD_XGE_RCVIN1_CNT_0_REG + port * 0x90);
- p[394] = dsaf_read_dev(ddev,
- DSAF_XOD_XGE_RCVIN2_CNT_0_REG + port * 0x90);
- p[395] = dsaf_read_dev(ddev,
- DSAF_XOD_XGE_RCVIN3_CNT_0_REG + port * 0x90);
p[396] = dsaf_read_dev(ddev,
- DSAF_XOD_XGE_RCVIN4_CNT_0_REG + port * 0x90);
+ DSAF_XOD_XGE_RCVIN0_CNT_0_REG + port * 0x90);
p[397] = dsaf_read_dev(ddev,
- DSAF_XOD_XGE_RCVIN5_CNT_0_REG + port * 0x90);
+ DSAF_XOD_XGE_RCVIN1_CNT_0_REG + port * 0x90);
p[398] = dsaf_read_dev(ddev,
- DSAF_XOD_XGE_RCVIN6_CNT_0_REG + port * 0x90);
+ DSAF_XOD_XGE_RCVIN2_CNT_0_REG + port * 0x90);
p[399] = dsaf_read_dev(ddev,
- DSAF_XOD_XGE_RCVIN7_CNT_0_REG + port * 0x90);
+ DSAF_XOD_XGE_RCVIN3_CNT_0_REG + port * 0x90);
p[400] = dsaf_read_dev(ddev,
- DSAF_XOD_PPE_RCVIN0_CNT_0_REG + port * 0x90);
+ DSAF_XOD_XGE_RCVIN4_CNT_0_REG + port * 0x90);
p[401] = dsaf_read_dev(ddev,
- DSAF_XOD_PPE_RCVIN1_CNT_0_REG + port * 0x90);
+ DSAF_XOD_XGE_RCVIN5_CNT_0_REG + port * 0x90);
p[402] = dsaf_read_dev(ddev,
- DSAF_XOD_ROCEE_RCVIN0_CNT_0_REG + port * 0x90);
+ DSAF_XOD_XGE_RCVIN6_CNT_0_REG + port * 0x90);
p[403] = dsaf_read_dev(ddev,
- DSAF_XOD_ROCEE_RCVIN1_CNT_0_REG + port * 0x90);
+ DSAF_XOD_XGE_RCVIN7_CNT_0_REG + port * 0x90);
p[404] = dsaf_read_dev(ddev,
+ DSAF_XOD_PPE_RCVIN0_CNT_0_REG + port * 0x90);
+ p[405] = dsaf_read_dev(ddev,
+ DSAF_XOD_PPE_RCVIN1_CNT_0_REG + port * 0x90);
+ p[406] = dsaf_read_dev(ddev,
+ DSAF_XOD_ROCEE_RCVIN0_CNT_0_REG + port * 0x90);
+ p[407] = dsaf_read_dev(ddev,
+ DSAF_XOD_ROCEE_RCVIN1_CNT_0_REG + port * 0x90);
+ p[408] = dsaf_read_dev(ddev,
DSAF_XOD_FIFO_STATUS_0_REG + port * 0x90);
/* dsaf voq registers */
for (i = 0; i < DSAF_VOQ_NUM / DSAF_COMM_CHN; i++) {
j = (i * DSAF_COMM_CHN + port) * 0x90;
- p[405 + i] = dsaf_read_dev(ddev,
+ p[409 + i] = dsaf_read_dev(ddev,
DSAF_VOQ_ECC_INVERT_EN_0_REG + j);
- p[408 + i] = dsaf_read_dev(ddev,
+ p[412 + i] = dsaf_read_dev(ddev,
DSAF_VOQ_SRAM_PKT_NUM_0_REG + j);
- p[411 + i] = dsaf_read_dev(ddev, DSAF_VOQ_IN_PKT_NUM_0_REG + j);
- p[414 + i] = dsaf_read_dev(ddev,
+ p[415 + i] = dsaf_read_dev(ddev, DSAF_VOQ_IN_PKT_NUM_0_REG + j);
+ p[418 + i] = dsaf_read_dev(ddev,
DSAF_VOQ_OUT_PKT_NUM_0_REG + j);
- p[417 + i] = dsaf_read_dev(ddev,
+ p[421 + i] = dsaf_read_dev(ddev,
DSAF_VOQ_ECC_ERR_ADDR_0_REG + j);
- p[420 + i] = dsaf_read_dev(ddev, DSAF_VOQ_BP_STATUS_0_REG + j);
- p[423 + i] = dsaf_read_dev(ddev, DSAF_VOQ_SPUP_IDLE_0_REG + j);
- p[426 + i] = dsaf_read_dev(ddev,
+ p[424 + i] = dsaf_read_dev(ddev, DSAF_VOQ_BP_STATUS_0_REG + j);
+ p[427 + i] = dsaf_read_dev(ddev, DSAF_VOQ_SPUP_IDLE_0_REG + j);
+ p[430 + i] = dsaf_read_dev(ddev,
DSAF_VOQ_XGE_XOD_REQ_0_0_REG + j);
- p[429 + i] = dsaf_read_dev(ddev,
+ p[433 + i] = dsaf_read_dev(ddev,
DSAF_VOQ_XGE_XOD_REQ_1_0_REG + j);
- p[432 + i] = dsaf_read_dev(ddev,
+ p[436 + i] = dsaf_read_dev(ddev,
DSAF_VOQ_PPE_XOD_REQ_0_REG + j);
- p[435 + i] = dsaf_read_dev(ddev,
+ p[439 + i] = dsaf_read_dev(ddev,
DSAF_VOQ_ROCEE_XOD_REQ_0_REG + j);
- p[438 + i] = dsaf_read_dev(ddev,
+ p[442 + i] = dsaf_read_dev(ddev,
DSAF_VOQ_BP_ALL_THRD_0_REG + j);
}
/* dsaf tbl registers */
- p[441] = dsaf_read_dev(ddev, DSAF_TBL_CTRL_0_REG);
- p[442] = dsaf_read_dev(ddev, DSAF_TBL_INT_MSK_0_REG);
- p[443] = dsaf_read_dev(ddev, DSAF_TBL_INT_SRC_0_REG);
- p[444] = dsaf_read_dev(ddev, DSAF_TBL_INT_STS_0_REG);
- p[445] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_ADDR_0_REG);
- p[446] = dsaf_read_dev(ddev, DSAF_TBL_LINE_ADDR_0_REG);
- p[447] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_HIGH_0_REG);
- p[448] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_LOW_0_REG);
- p[449] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_MCAST_CFG_4_0_REG);
- p[450] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_MCAST_CFG_3_0_REG);
- p[451] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_MCAST_CFG_2_0_REG);
- p[452] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_MCAST_CFG_1_0_REG);
- p[453] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_MCAST_CFG_0_0_REG);
- p[454] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_UCAST_CFG_0_REG);
- p[455] = dsaf_read_dev(ddev, DSAF_TBL_LIN_CFG_0_REG);
- p[456] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RDATA_HIGH_0_REG);
- p[457] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RDATA_LOW_0_REG);
- p[458] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RAM_RDATA4_0_REG);
- p[459] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RAM_RDATA3_0_REG);
- p[460] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RAM_RDATA2_0_REG);
- p[461] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RAM_RDATA1_0_REG);
- p[462] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RAM_RDATA0_0_REG);
- p[463] = dsaf_read_dev(ddev, DSAF_TBL_LIN_RDATA_0_REG);
+ p[445] = dsaf_read_dev(ddev, DSAF_TBL_CTRL_0_REG);
+ p[446] = dsaf_read_dev(ddev, DSAF_TBL_INT_MSK_0_REG);
+ p[447] = dsaf_read_dev(ddev, DSAF_TBL_INT_SRC_0_REG);
+ p[448] = dsaf_read_dev(ddev, DSAF_TBL_INT_STS_0_REG);
+ p[449] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_ADDR_0_REG);
+ p[450] = dsaf_read_dev(ddev, DSAF_TBL_LINE_ADDR_0_REG);
+ p[451] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_HIGH_0_REG);
+ p[452] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_LOW_0_REG);
+ p[453] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_MCAST_CFG_4_0_REG);
+ p[454] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_MCAST_CFG_3_0_REG);
+ p[455] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_MCAST_CFG_2_0_REG);
+ p[456] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_MCAST_CFG_1_0_REG);
+ p[457] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_MCAST_CFG_0_0_REG);
+ p[458] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_UCAST_CFG_0_REG);
+ p[459] = dsaf_read_dev(ddev, DSAF_TBL_LIN_CFG_0_REG);
+ p[460] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RDATA_HIGH_0_REG);
+ p[461] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RDATA_LOW_0_REG);
+ p[462] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RAM_RDATA4_0_REG);
+ p[463] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RAM_RDATA3_0_REG);
+ p[464] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RAM_RDATA2_0_REG);
+ p[465] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RAM_RDATA1_0_REG);
+ p[466] = dsaf_read_dev(ddev, DSAF_TBL_TCAM_RAM_RDATA0_0_REG);
+ p[467] = dsaf_read_dev(ddev, DSAF_TBL_LIN_RDATA_0_REG);
for (i = 0; i < DSAF_SW_PORT_NUM; i++) {
j = i * 0x8;
- p[464 + 2 * i] = dsaf_read_dev(ddev,
+ p[468 + 2 * i] = dsaf_read_dev(ddev,
DSAF_TBL_DA0_MIS_INFO1_0_REG + j);
- p[465 + 2 * i] = dsaf_read_dev(ddev,
+ p[469 + 2 * i] = dsaf_read_dev(ddev,
DSAF_TBL_DA0_MIS_INFO0_0_REG + j);
}
- p[480] = dsaf_read_dev(ddev, DSAF_TBL_SA_MIS_INFO2_0_REG);
- p[481] = dsaf_read_dev(ddev, DSAF_TBL_SA_MIS_INFO1_0_REG);
- p[482] = dsaf_read_dev(ddev, DSAF_TBL_SA_MIS_INFO0_0_REG);
- p[483] = dsaf_read_dev(ddev, DSAF_TBL_PUL_0_REG);
- p[484] = dsaf_read_dev(ddev, DSAF_TBL_OLD_RSLT_0_REG);
- p[485] = dsaf_read_dev(ddev, DSAF_TBL_OLD_SCAN_VAL_0_REG);
- p[486] = dsaf_read_dev(ddev, DSAF_TBL_DFX_CTRL_0_REG);
- p[487] = dsaf_read_dev(ddev, DSAF_TBL_DFX_STAT_0_REG);
- p[488] = dsaf_read_dev(ddev, DSAF_TBL_DFX_STAT_2_0_REG);
- p[489] = dsaf_read_dev(ddev, DSAF_TBL_LKUP_NUM_I_0_REG);
- p[490] = dsaf_read_dev(ddev, DSAF_TBL_LKUP_NUM_O_0_REG);
- p[491] = dsaf_read_dev(ddev, DSAF_TBL_UCAST_BCAST_MIS_INFO_0_0_REG);
+ p[484] = dsaf_read_dev(ddev, DSAF_TBL_SA_MIS_INFO2_0_REG);
+ p[485] = dsaf_read_dev(ddev, DSAF_TBL_SA_MIS_INFO1_0_REG);
+ p[486] = dsaf_read_dev(ddev, DSAF_TBL_SA_MIS_INFO0_0_REG);
+ p[487] = dsaf_read_dev(ddev, DSAF_TBL_PUL_0_REG);
+ p[488] = dsaf_read_dev(ddev, DSAF_TBL_OLD_RSLT_0_REG);
+ p[489] = dsaf_read_dev(ddev, DSAF_TBL_OLD_SCAN_VAL_0_REG);
+ p[490] = dsaf_read_dev(ddev, DSAF_TBL_DFX_CTRL_0_REG);
+ p[491] = dsaf_read_dev(ddev, DSAF_TBL_DFX_STAT_0_REG);
+ p[492] = dsaf_read_dev(ddev, DSAF_TBL_DFX_STAT_2_0_REG);
+ p[493] = dsaf_read_dev(ddev, DSAF_TBL_LKUP_NUM_I_0_REG);
+ p[494] = dsaf_read_dev(ddev, DSAF_TBL_LKUP_NUM_O_0_REG);
+ p[495] = dsaf_read_dev(ddev, DSAF_TBL_UCAST_BCAST_MIS_INFO_0_0_REG);
/* dsaf other registers */
- p[492] = dsaf_read_dev(ddev, DSAF_INODE_FIFO_WL_0_REG + port * 0x4);
- p[493] = dsaf_read_dev(ddev, DSAF_ONODE_FIFO_WL_0_REG + port * 0x4);
- p[494] = dsaf_read_dev(ddev, DSAF_XGE_GE_WORK_MODE_0_REG + port * 0x4);
- p[495] = dsaf_read_dev(ddev,
+ p[496] = dsaf_read_dev(ddev, DSAF_INODE_FIFO_WL_0_REG + port * 0x4);
+ p[497] = dsaf_read_dev(ddev, DSAF_ONODE_FIFO_WL_0_REG + port * 0x4);
+ p[498] = dsaf_read_dev(ddev, DSAF_XGE_GE_WORK_MODE_0_REG + port * 0x4);
+ p[499] = dsaf_read_dev(ddev,
DSAF_XGE_APP_RX_LINK_UP_0_REG + port * 0x4);
- p[496] = dsaf_read_dev(ddev, DSAF_NETPORT_CTRL_SIG_0_REG + port * 0x4);
- p[497] = dsaf_read_dev(ddev, DSAF_XGE_CTRL_SIG_CFG_0_REG + port * 0x4);
+ p[500] = dsaf_read_dev(ddev, DSAF_NETPORT_CTRL_SIG_0_REG + port * 0x4);
+ p[501] = dsaf_read_dev(ddev, DSAF_XGE_CTRL_SIG_CFG_0_REG + port * 0x4);
if (!is_ver1)
- p[498] = dsaf_read_dev(ddev, DSAF_PAUSE_CFG_REG + port * 0x4);
+ p[502] = dsaf_read_dev(ddev, DSAF_PAUSE_CFG_REG + port * 0x4);
/* mark end of dsaf regs */
- for (i = 499; i < 504; i++)
+ for (i = 503; i < 504; i++)
p[i] = 0xdddddddd;
}
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_reg.h b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_reg.h
index 2cf8b1d..6f3ec2f 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_dsaf_reg.h
+++ b/drivers/net/ethernet/hisilicon/hns/hns_dsaf_reg.h
@@ -173,7 +173,7 @@
#define DSAF_INODE_IN_DATA_STP_DISC_0_REG 0x1A50
#define DSAF_INODE_GE_FC_EN_0_REG 0x1B00
#define DSAF_INODE_VC0_IN_PKT_NUM_0_REG 0x1B50
-#define DSAF_INODE_VC1_IN_PKT_NUM_0_REG 0x1C00
+#define DSAF_INODE_VC1_IN_PKT_NUM_0_REG 0x103C
#define DSAF_INODE_IN_PRIO_PAUSE_BASE_REG 0x1C00
#define DSAF_INODE_IN_PRIO_PAUSE_BASE_OFFSET 0x100
#define DSAF_INODE_IN_PRIO_PAUSE_OFFSET 0x50
@@ -400,11 +400,11 @@
#define RCB_ECC_ERR_ADDR4_REG 0x460
#define RCB_ECC_ERR_ADDR5_REG 0x464
-#define RCB_COM_SF_CFG_INTMASK_RING 0x480
-#define RCB_COM_SF_CFG_RING_STS 0x484
-#define RCB_COM_SF_CFG_RING 0x488
-#define RCB_COM_SF_CFG_INTMASK_BD 0x48C
-#define RCB_COM_SF_CFG_BD_RINT_STS 0x470
+#define RCB_COM_SF_CFG_INTMASK_RING 0x470
+#define RCB_COM_SF_CFG_RING_STS 0x474
+#define RCB_COM_SF_CFG_RING 0x478
+#define RCB_COM_SF_CFG_INTMASK_BD 0x47C
+#define RCB_COM_SF_CFG_BD_RINT_STS 0x480
#define RCB_COM_RCB_RD_BD_BUSY 0x490
#define RCB_COM_RCB_FBD_CRT_EN 0x494
#define RCB_COM_AXI_WR_ERR_INTMASK 0x498
@@ -528,6 +528,7 @@
#define GMAC_LD_LINK_COUNTER_REG 0x01D0UL
#define GMAC_LOOP_REG 0x01DCUL
#define GMAC_RECV_CONTROL_REG 0x01E0UL
+#define GMAC_PCS_RX_EN_REG 0x01E4UL
#define GMAC_VLAN_CODE_REG 0x01E8UL
#define GMAC_RX_OVERRUN_CNT_REG 0x01ECUL
#define GMAC_RX_LENGTHFIELD_ERR_CNT_REG 0x01F4UL
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_enet.c b/drivers/net/ethernet/hisilicon/hns/hns_enet.c
index 111e1aab..a2f7d08 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_enet.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_enet.c
@@ -39,9 +39,9 @@
#define SKB_TMP_LEN(SKB) \
(((SKB)->transport_header - (SKB)->mac_header) + tcp_hdrlen(SKB))
-static void fill_v2_desc(struct hnae_ring *ring, void *priv,
- int size, dma_addr_t dma, int frag_end,
- int buf_num, enum hns_desc_type type, int mtu)
+static void fill_v2_desc_hw(struct hnae_ring *ring, void *priv, int size,
+ int send_sz, dma_addr_t dma, int frag_end,
+ int buf_num, enum hns_desc_type type, int mtu)
{
struct hnae_desc *desc = &ring->desc[ring->next_to_use];
struct hnae_desc_cb *desc_cb = &ring->desc_cb[ring->next_to_use];
@@ -63,7 +63,7 @@ static void fill_v2_desc(struct hnae_ring *ring, void *priv,
desc_cb->type = type;
desc->addr = cpu_to_le64(dma);
- desc->tx.send_size = cpu_to_le16((u16)size);
+ desc->tx.send_size = cpu_to_le16((u16)send_sz);
/* config bd buffer end */
hnae_set_bit(rrcfv, HNSV2_TXD_VLD_B, 1);
@@ -132,6 +132,14 @@ static void fill_v2_desc(struct hnae_ring *ring, void *priv,
ring_ptr_move_fw(ring, next_to_use);
}
+static void fill_v2_desc(struct hnae_ring *ring, void *priv,
+ int size, dma_addr_t dma, int frag_end,
+ int buf_num, enum hns_desc_type type, int mtu)
+{
+ fill_v2_desc_hw(ring, priv, size, size, dma, frag_end,
+ buf_num, type, mtu);
+}
+
static const struct acpi_device_id hns_enet_acpi_match[] = {
{ "HISI00C1", 0 },
{ "HISI00C2", 0 },
@@ -288,15 +296,15 @@ static void fill_tso_desc(struct hnae_ring *ring, void *priv,
/* when the frag size is bigger than hardware, split this frag */
for (k = 0; k < frag_buf_num; k++)
- fill_v2_desc(ring, priv,
- (k == frag_buf_num - 1) ?
+ fill_v2_desc_hw(ring, priv, k == 0 ? size : 0,
+ (k == frag_buf_num - 1) ?
sizeoflast : BD_MAX_SEND_SIZE,
- dma + BD_MAX_SEND_SIZE * k,
- frag_end && (k == frag_buf_num - 1) ? 1 : 0,
- buf_num,
- (type == DESC_TYPE_SKB && !k) ?
+ dma + BD_MAX_SEND_SIZE * k,
+ frag_end && (k == frag_buf_num - 1) ? 1 : 0,
+ buf_num,
+ (type == DESC_TYPE_SKB && !k) ?
DESC_TYPE_SKB : DESC_TYPE_PAGE,
- mtu);
+ mtu);
}
netdev_tx_t hns_nic_net_xmit_hw(struct net_device *ndev,
@@ -529,7 +537,7 @@ static void hns_nic_reuse_page(struct sk_buff *skb, int i,
}
skb_add_rx_frag(skb, i, desc_cb->priv, desc_cb->page_offset + pull_len,
- size - pull_len, truesize - pull_len);
+ size - pull_len, truesize);
/* avoid re-using remote pages,flag default unreuse */
if (unlikely(page_to_nid(desc_cb->priv) != numa_node_id()))
@@ -1071,6 +1079,9 @@ int hns_nic_init_phy(struct net_device *ndev, struct hnae_handle *h)
if (h->phy_if == PHY_INTERFACE_MODE_XGMII)
phy_dev->autoneg = false;
+ if (h->phy_if == PHY_INTERFACE_MODE_SGMII)
+ phy_stop(phy_dev);
+
return 0;
}
@@ -1184,6 +1195,22 @@ static void hns_set_irq_affinity(struct hns_nic_priv *priv)
}
}
+static void hns_nic_free_irq(int q_num, struct hns_nic_priv *priv)
+{
+ int i;
+
+ for (i = 0; i < q_num * 2; i++) {
+ if (priv->ring_data[i].ring->irq_init_flag == RCB_IRQ_INITED) {
+ irq_set_affinity_hint(priv->ring_data[i].ring->irq,
+ NULL);
+ free_irq(priv->ring_data[i].ring->irq,
+ &priv->ring_data[i]);
+ priv->ring_data[i].ring->irq_init_flag =
+ RCB_IRQ_NOT_INITED;
+ }
+ }
+}
+
static int hns_nic_init_irq(struct hns_nic_priv *priv)
{
struct hnae_handle *h = priv->ae_handle;
@@ -1208,7 +1235,7 @@ static int hns_nic_init_irq(struct hns_nic_priv *priv)
if (ret) {
netdev_err(priv->netdev, "request irq(%d) fail\n",
rd->ring->irq);
- return ret;
+ goto out_free_irq;
}
disable_irq(rd->ring->irq);
rd->ring->irq_init_flag = RCB_IRQ_INITED;
@@ -1218,6 +1245,10 @@ static int hns_nic_init_irq(struct hns_nic_priv *priv)
hns_set_irq_affinity(priv);
return 0;
+
+out_free_irq:
+ hns_nic_free_irq(h->q_num, priv);
+ return ret;
}
static int hns_nic_net_up(struct net_device *ndev)
@@ -1227,6 +1258,9 @@ static int hns_nic_net_up(struct net_device *ndev)
int i, j;
int ret;
+ if (!test_bit(NIC_STATE_DOWN, &priv->state))
+ return 0;
+
ret = hns_nic_init_irq(priv);
if (ret != 0) {
netdev_err(ndev, "hns init irq failed! ret=%d\n", ret);
@@ -1262,6 +1296,7 @@ static int hns_nic_net_up(struct net_device *ndev)
for (j = i - 1; j >= 0; j--)
hns_nic_ring_close(ndev, j);
+ hns_nic_free_irq(h->q_num, priv);
set_bit(NIC_STATE_DOWN, &priv->state);
return ret;
@@ -1372,11 +1407,19 @@ static int hns_nic_net_stop(struct net_device *ndev)
}
static void hns_tx_timeout_reset(struct hns_nic_priv *priv);
+#define HNS_TX_TIMEO_LIMIT (40 * HZ)
static void hns_nic_net_timeout(struct net_device *ndev)
{
struct hns_nic_priv *priv = netdev_priv(ndev);
- hns_tx_timeout_reset(priv);
+ if (ndev->watchdog_timeo < HNS_TX_TIMEO_LIMIT) {
+ ndev->watchdog_timeo *= 2;
+ netdev_info(ndev, "watchdog_timo changed to %d.\n",
+ ndev->watchdog_timeo);
+ } else {
+ ndev->watchdog_timeo = HNS_NIC_TX_TIMEOUT;
+ hns_tx_timeout_reset(priv);
+ }
}
static int hns_nic_do_ioctl(struct net_device *netdev, struct ifreq *ifr,
@@ -1710,11 +1753,11 @@ static void hns_nic_service_task(struct work_struct *work)
= container_of(work, struct hns_nic_priv, service_task);
struct hnae_handle *h = priv->ae_handle;
+ hns_nic_reset_subtask(priv);
hns_nic_update_link_status(priv->netdev);
h->dev->ops->update_led_status(h);
hns_nic_update_stats(priv->netdev);
- hns_nic_reset_subtask(priv);
hns_nic_service_event_complete(priv);
}
@@ -1993,7 +2036,7 @@ static int hns_nic_dev_probe(struct platform_device *pdev)
switch (priv->enet_ver) {
case AE_VERSION_2:
- ndev->features |= NETIF_F_TSO | NETIF_F_TSO6;
+ ndev->features |= NETIF_F_TSO | NETIF_F_TSO6 | NETIF_F_NTUPLE;
ndev->hw_features |= NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
NETIF_F_RXCSUM | NETIF_F_SG | NETIF_F_GSO |
NETIF_F_GRO | NETIF_F_TSO | NETIF_F_TSO6;
diff --git a/drivers/net/ethernet/hisilicon/hns/hns_ethtool.c b/drivers/net/ethernet/hisilicon/hns/hns_ethtool.c
index 6be0cae..4cd1633 100644
--- a/drivers/net/ethernet/hisilicon/hns/hns_ethtool.c
+++ b/drivers/net/ethernet/hisilicon/hns/hns_ethtool.c
@@ -243,7 +243,9 @@ static int hns_nic_set_link_ksettings(struct net_device *net_dev,
}
if (h->dev->ops->adjust_link) {
+ netif_carrier_off(net_dev);
h->dev->ops->adjust_link(h, (int)speed, cmd->base.duplex);
+ netif_carrier_on(net_dev);
return 0;
}
diff --git a/drivers/net/ethernet/hp/hp100.c b/drivers/net/ethernet/hp/hp100.c
index 631dbc7..0988bf1 100644
--- a/drivers/net/ethernet/hp/hp100.c
+++ b/drivers/net/ethernet/hp/hp100.c
@@ -2636,7 +2636,7 @@ static int hp100_login_to_vg_hub(struct net_device *dev, u_short force_relogin)
/* Wait for link to drop */
time = jiffies + (HZ / 10);
do {
- if (~(hp100_inb(VG_LAN_CFG_1) & HP100_LINK_UP_ST))
+ if (!(hp100_inb(VG_LAN_CFG_1) & HP100_LINK_UP_ST))
break;
if (!in_interrupt())
schedule_timeout_interruptible(1);
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index b375ae9..4996228 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -1162,11 +1162,15 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
map_failed_frags:
last = i+1;
- for (i = 0; i < last; i++)
+ for (i = 1; i < last; i++)
dma_unmap_page(&adapter->vdev->dev, descs[i].fields.address,
descs[i].fields.flags_len & IBMVETH_BUF_LEN_MASK,
DMA_TO_DEVICE);
+ dma_unmap_single(&adapter->vdev->dev,
+ descs[0].fields.address,
+ descs[0].fields.flags_len & IBMVETH_BUF_LEN_MASK,
+ DMA_TO_DEVICE);
map_failed:
if (!firmware_has_feature(FW_FEATURE_CMO))
netdev_err(netdev, "tx: unable to map xmit buffer\n");
diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c
index 49094c9..897a87a 100644
--- a/drivers/net/ethernet/ibm/ibmvnic.c
+++ b/drivers/net/ethernet/ibm/ibmvnic.c
@@ -994,6 +994,7 @@ static int ibmvnic_poll(struct napi_struct *napi, int budget)
netdev_err(netdev, "rx error %x\n", next->rx_comp.rc);
/* free the entry */
next->rx_comp.first = 0;
+ dev_kfree_skb_any(rx_buff->skb);
remove_buff_from_pool(adapter, rx_buff);
break;
}
diff --git a/drivers/net/ethernet/intel/e1000/e1000_ethtool.c b/drivers/net/ethernet/intel/e1000/e1000_ethtool.c
index 975eeb8..2a81f6d 100644
--- a/drivers/net/ethernet/intel/e1000/e1000_ethtool.c
+++ b/drivers/net/ethernet/intel/e1000/e1000_ethtool.c
@@ -645,14 +645,14 @@ static int e1000_set_ringparam(struct net_device *netdev,
adapter->tx_ring = tx_old;
e1000_free_all_rx_resources(adapter);
e1000_free_all_tx_resources(adapter);
- kfree(tx_old);
- kfree(rx_old);
adapter->rx_ring = rxdr;
adapter->tx_ring = txdr;
err = e1000_up(adapter);
if (err)
goto err_setup;
}
+ kfree(tx_old);
+ kfree(rx_old);
clear_bit(__E1000_RESETTING, &adapter->flags);
return 0;
@@ -665,7 +665,8 @@ static int e1000_set_ringparam(struct net_device *netdev,
err_alloc_rx:
kfree(txdr);
err_alloc_tx:
- e1000_up(adapter);
+ if (netif_running(adapter->netdev))
+ e1000_up(adapter);
err_setup:
clear_bit(__E1000_RESETTING, &adapter->flags);
return err;
@@ -1825,11 +1826,12 @@ static void e1000_get_ethtool_stats(struct net_device *netdev,
{
struct e1000_adapter *adapter = netdev_priv(netdev);
int i;
- char *p = NULL;
const struct e1000_stats *stat = e1000_gstrings_stats;
e1000_update_stats(adapter);
- for (i = 0; i < E1000_GLOBAL_STATS_LEN; i++) {
+ for (i = 0; i < E1000_GLOBAL_STATS_LEN; i++, stat++) {
+ char *p;
+
switch (stat->type) {
case NETDEV_STATS:
p = (char *)netdev + stat->stat_offset;
@@ -1840,15 +1842,13 @@ static void e1000_get_ethtool_stats(struct net_device *netdev,
default:
WARN_ONCE(1, "Invalid E1000 stat type: %u index %d\n",
stat->type, i);
- break;
+ continue;
}
if (stat->sizeof_stat == sizeof(u64))
data[i] = *(u64 *)p;
else
data[i] = *(u32 *)p;
-
- stat++;
}
/* BUG_ON(i != E1000_STATS_LEN); */
}
diff --git a/drivers/net/ethernet/intel/e1000/e1000_main.c b/drivers/net/ethernet/intel/e1000/e1000_main.c
index dd112aa..39a09e1 100644
--- a/drivers/net/ethernet/intel/e1000/e1000_main.c
+++ b/drivers/net/ethernet/intel/e1000/e1000_main.c
@@ -521,8 +521,6 @@ void e1000_down(struct e1000_adapter *adapter)
struct net_device *netdev = adapter->netdev;
u32 rctl, tctl;
- netif_carrier_off(netdev);
-
/* disable receives in the hardware */
rctl = er32(RCTL);
ew32(RCTL, rctl & ~E1000_RCTL_EN);
@@ -538,6 +536,15 @@ void e1000_down(struct e1000_adapter *adapter)
E1000_WRITE_FLUSH();
msleep(10);
+ /* Set the carrier off after transmits have been disabled in the
+ * hardware, to avoid race conditions with e1000_watchdog() (which
+ * may be running concurrently to us, checking for the carrier
+ * bit to decide whether it should enable transmits again). Such
+ * a race condition would result into transmission being disabled
+ * in the hardware until the next IFF_DOWN+IFF_UP cycle.
+ */
+ netif_carrier_off(netdev);
+
napi_disable(&adapter->napi);
e1000_irq_disable(adapter);
diff --git a/drivers/net/ethernet/intel/e1000e/defines.h b/drivers/net/ethernet/intel/e1000e/defines.h
index afb7ebe..824fd44 100644
--- a/drivers/net/ethernet/intel/e1000e/defines.h
+++ b/drivers/net/ethernet/intel/e1000e/defines.h
@@ -400,6 +400,10 @@
#define E1000_ICR_RXDMT0 0x00000010 /* Rx desc min. threshold (0) */
#define E1000_ICR_RXO 0x00000040 /* Receiver Overrun */
#define E1000_ICR_RXT0 0x00000080 /* Rx timer intr (ring 0) */
+#define E1000_ICR_MDAC 0x00000200 /* MDIO Access Complete */
+#define E1000_ICR_SRPD 0x00010000 /* Small Receive Packet Detected */
+#define E1000_ICR_ACK 0x00020000 /* Receive ACK Frame Detected */
+#define E1000_ICR_MNG 0x00040000 /* Manageability Event Detected */
#define E1000_ICR_ECCER 0x00400000 /* Uncorrectable ECC Error */
/* If this bit asserted, the driver should claim the interrupt */
#define E1000_ICR_INT_ASSERTED 0x80000000
@@ -407,7 +411,7 @@
#define E1000_ICR_RXQ1 0x00200000 /* Rx Queue 1 Interrupt */
#define E1000_ICR_TXQ0 0x00400000 /* Tx Queue 0 Interrupt */
#define E1000_ICR_TXQ1 0x00800000 /* Tx Queue 1 Interrupt */
-#define E1000_ICR_OTHER 0x01000000 /* Other Interrupts */
+#define E1000_ICR_OTHER 0x01000000 /* Other Interrupt */
/* PBA ECC Register */
#define E1000_PBA_ECC_COUNTER_MASK 0xFFF00000 /* ECC counter mask */
@@ -431,12 +435,27 @@
E1000_IMS_RXSEQ | \
E1000_IMS_LSC)
+/* These are all of the events related to the OTHER interrupt.
+ */
+#define IMS_OTHER_MASK ( \
+ E1000_IMS_LSC | \
+ E1000_IMS_RXO | \
+ E1000_IMS_MDAC | \
+ E1000_IMS_SRPD | \
+ E1000_IMS_ACK | \
+ E1000_IMS_MNG)
+
/* Interrupt Mask Set */
#define E1000_IMS_TXDW E1000_ICR_TXDW /* Transmit desc written back */
#define E1000_IMS_LSC E1000_ICR_LSC /* Link Status Change */
#define E1000_IMS_RXSEQ E1000_ICR_RXSEQ /* Rx sequence error */
#define E1000_IMS_RXDMT0 E1000_ICR_RXDMT0 /* Rx desc min. threshold */
+#define E1000_IMS_RXO E1000_ICR_RXO /* Receiver Overrun */
#define E1000_IMS_RXT0 E1000_ICR_RXT0 /* Rx timer intr */
+#define E1000_IMS_MDAC E1000_ICR_MDAC /* MDIO Access Complete */
+#define E1000_IMS_SRPD E1000_ICR_SRPD /* Small Receive Packet */
+#define E1000_IMS_ACK E1000_ICR_ACK /* Receive ACK Frame Detected */
+#define E1000_IMS_MNG E1000_ICR_MNG /* Manageability Event */
#define E1000_IMS_ECCER E1000_ICR_ECCER /* Uncorrectable ECC Error */
#define E1000_IMS_RXQ0 E1000_ICR_RXQ0 /* Rx Queue 0 Interrupt */
#define E1000_IMS_RXQ1 E1000_ICR_RXQ1 /* Rx Queue 1 Interrupt */
diff --git a/drivers/net/ethernet/intel/e1000e/ich8lan.c b/drivers/net/ethernet/intel/e1000e/ich8lan.c
index 8a48656..dc7d671b 100644
--- a/drivers/net/ethernet/intel/e1000e/ich8lan.c
+++ b/drivers/net/ethernet/intel/e1000e/ich8lan.c
@@ -1364,9 +1364,6 @@ static s32 e1000_disable_ulp_lpt_lp(struct e1000_hw *hw, bool force)
* Checks to see of the link status of the hardware has changed. If a
* change in link status has been detected, then we read the PHY registers
* to get the current speed/duplex if link exists.
- *
- * Returns a negative error code (-E1000_ERR_*) or 0 (link down) or 1 (link
- * up).
**/
static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
{
@@ -1382,7 +1379,8 @@ static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
* Change or Rx Sequence Error interrupt.
*/
if (!mac->get_link_status)
- return 1;
+ return 0;
+ mac->get_link_status = false;
/* First we want to see if the MII Status Register reports
* link. If so, then we want to get the current speed/duplex
@@ -1390,12 +1388,12 @@ static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
*/
ret_val = e1000e_phy_has_link_generic(hw, 1, 0, &link);
if (ret_val)
- return ret_val;
+ goto out;
if (hw->mac.type == e1000_pchlan) {
ret_val = e1000_k1_gig_workaround_hv(hw, link);
if (ret_val)
- return ret_val;
+ goto out;
}
/* When connected at 10Mbps half-duplex, some parts are excessively
@@ -1430,7 +1428,7 @@ static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
ret_val = hw->phy.ops.acquire(hw);
if (ret_val)
- return ret_val;
+ goto out;
if (hw->mac.type == e1000_pch2lan)
emi_addr = I82579_RX_CONFIG;
@@ -1453,7 +1451,7 @@ static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
hw->phy.ops.release(hw);
if (ret_val)
- return ret_val;
+ goto out;
if (hw->mac.type == e1000_pch_spt) {
u16 data;
@@ -1462,14 +1460,14 @@ static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
if (speed == SPEED_1000) {
ret_val = hw->phy.ops.acquire(hw);
if (ret_val)
- return ret_val;
+ goto out;
ret_val = e1e_rphy_locked(hw,
PHY_REG(776, 20),
&data);
if (ret_val) {
hw->phy.ops.release(hw);
- return ret_val;
+ goto out;
}
ptr_gap = (data & (0x3FF << 2)) >> 2;
@@ -1483,18 +1481,18 @@ static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
}
hw->phy.ops.release(hw);
if (ret_val)
- return ret_val;
+ goto out;
} else {
ret_val = hw->phy.ops.acquire(hw);
if (ret_val)
- return ret_val;
+ goto out;
ret_val = e1e_wphy_locked(hw,
PHY_REG(776, 20),
0xC023);
hw->phy.ops.release(hw);
if (ret_val)
- return ret_val;
+ goto out;
}
}
@@ -1521,7 +1519,7 @@ static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
(hw->adapter->pdev->device == E1000_DEV_ID_PCH_I218_V3)) {
ret_val = e1000_k1_workaround_lpt_lp(hw, link);
if (ret_val)
- return ret_val;
+ goto out;
}
if ((hw->mac.type == e1000_pch_lpt) ||
(hw->mac.type == e1000_pch_spt)) {
@@ -1530,7 +1528,7 @@ static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
*/
ret_val = e1000_platform_pm_pch_lpt(hw, link);
if (ret_val)
- return ret_val;
+ goto out;
}
/* Clear link partner's EEE ability */
@@ -1550,9 +1548,7 @@ static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
}
if (!link)
- return 0; /* No link detected */
-
- mac->get_link_status = false;
+ goto out;
switch (hw->mac.type) {
case e1000_pch2lan:
@@ -1614,12 +1610,14 @@ static s32 e1000_check_for_copper_link_ich8lan(struct e1000_hw *hw)
* different link partner.
*/
ret_val = e1000e_config_fc_after_link_up(hw);
- if (ret_val) {
+ if (ret_val)
e_dbg("Error configuring flow control\n");
- return ret_val;
- }
- return 1;
+ return ret_val;
+
+out:
+ mac->get_link_status = true;
+ return ret_val;
}
static s32 e1000_get_variants_ich8lan(struct e1000_adapter *adapter)
diff --git a/drivers/net/ethernet/intel/e1000e/mac.c b/drivers/net/ethernet/intel/e1000e/mac.c
index f457c57..5bdc3a2d 100644
--- a/drivers/net/ethernet/intel/e1000e/mac.c
+++ b/drivers/net/ethernet/intel/e1000e/mac.c
@@ -410,9 +410,6 @@ void e1000e_clear_hw_cntrs_base(struct e1000_hw *hw)
* Checks to see of the link status of the hardware has changed. If a
* change in link status has been detected, then we read the PHY registers
* to get the current speed/duplex if link exists.
- *
- * Returns a negative error code (-E1000_ERR_*) or 0 (link down) or 1 (link
- * up).
**/
s32 e1000e_check_for_copper_link(struct e1000_hw *hw)
{
@@ -426,20 +423,16 @@ s32 e1000e_check_for_copper_link(struct e1000_hw *hw)
* Change or Rx Sequence Error interrupt.
*/
if (!mac->get_link_status)
- return 1;
+ return 0;
+ mac->get_link_status = false;
/* First we want to see if the MII Status Register reports
* link. If so, then we want to get the current speed/duplex
* of the PHY.
*/
ret_val = e1000e_phy_has_link_generic(hw, 1, 0, &link);
- if (ret_val)
- return ret_val;
-
- if (!link)
- return 0; /* No link detected */
-
- mac->get_link_status = false;
+ if (ret_val || !link)
+ goto out;
/* Check if there was DownShift, must be checked
* immediately after link-up
@@ -464,12 +457,14 @@ s32 e1000e_check_for_copper_link(struct e1000_hw *hw)
* different link partner.
*/
ret_val = e1000e_config_fc_after_link_up(hw);
- if (ret_val) {
+ if (ret_val)
e_dbg("Error configuring flow control\n");
- return ret_val;
- }
- return 1;
+ return ret_val;
+
+out:
+ mac->get_link_status = true;
+ return ret_val;
}
/**
diff --git a/drivers/net/ethernet/intel/e1000e/netdev.c b/drivers/net/ethernet/intel/e1000e/netdev.c
index 825ec8f..6855b33 100644
--- a/drivers/net/ethernet/intel/e1000e/netdev.c
+++ b/drivers/net/ethernet/intel/e1000e/netdev.c
@@ -1911,30 +1911,20 @@ static irqreturn_t e1000_msix_other(int __always_unused irq, void *data)
struct net_device *netdev = data;
struct e1000_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = &adapter->hw;
- u32 icr;
- bool enable = true;
+ u32 icr = er32(ICR);
- icr = er32(ICR);
- if (icr & E1000_ICR_RXO) {
- ew32(ICR, E1000_ICR_RXO);
- enable = false;
- /* napi poll will re-enable Other, make sure it runs */
- if (napi_schedule_prep(&adapter->napi)) {
- adapter->total_rx_bytes = 0;
- adapter->total_rx_packets = 0;
- __napi_schedule(&adapter->napi);
- }
- }
+ if (icr & adapter->eiac_mask)
+ ew32(ICS, (icr & adapter->eiac_mask));
+
if (icr & E1000_ICR_LSC) {
- ew32(ICR, E1000_ICR_LSC);
hw->mac.get_link_status = true;
/* guard against interrupt when we're going down */
if (!test_bit(__E1000_DOWN, &adapter->state))
mod_timer(&adapter->watchdog_timer, jiffies + 1);
}
- if (enable && !test_bit(__E1000_DOWN, &adapter->state))
- ew32(IMS, E1000_IMS_OTHER);
+ if (!test_bit(__E1000_DOWN, &adapter->state))
+ ew32(IMS, E1000_IMS_OTHER | IMS_OTHER_MASK);
return IRQ_HANDLED;
}
@@ -2037,7 +2027,6 @@ static void e1000_configure_msix(struct e1000_adapter *adapter)
hw->hw_addr + E1000_EITR_82574(vector));
else
writel(1, hw->hw_addr + E1000_EITR_82574(vector));
- adapter->eiac_mask |= E1000_IMS_OTHER;
/* Cause Tx interrupts on every write back */
ivar |= BIT(31);
@@ -2262,7 +2251,8 @@ static void e1000_irq_enable(struct e1000_adapter *adapter)
if (adapter->msix_entries) {
ew32(EIAC_82574, adapter->eiac_mask & E1000_EIAC_MASK_82574);
- ew32(IMS, adapter->eiac_mask | E1000_IMS_LSC);
+ ew32(IMS, adapter->eiac_mask | E1000_IMS_OTHER |
+ IMS_OTHER_MASK);
} else if ((hw->mac.type == e1000_pch_lpt) ||
(hw->mac.type == e1000_pch_spt)) {
ew32(IMS, IMS_ENABLE_MASK | E1000_IMS_ECCER);
@@ -2331,8 +2321,8 @@ static int e1000_alloc_ring_dma(struct e1000_adapter *adapter,
{
struct pci_dev *pdev = adapter->pdev;
- ring->desc = dma_alloc_coherent(&pdev->dev, ring->size, &ring->dma,
- GFP_KERNEL);
+ ring->desc = dma_zalloc_coherent(&pdev->dev, ring->size, &ring->dma,
+ GFP_KERNEL);
if (!ring->desc)
return -ENOMEM;
@@ -2705,8 +2695,7 @@ static int e1000e_poll(struct napi_struct *napi, int weight)
napi_complete_done(napi, work_done);
if (!test_bit(__E1000_DOWN, &adapter->state)) {
if (adapter->msix_entries)
- ew32(IMS, adapter->rx_ring->ims_val |
- E1000_IMS_OTHER);
+ ew32(IMS, adapter->rx_ring->ims_val);
else
e1000_irq_enable(adapter);
}
@@ -5085,7 +5074,7 @@ static bool e1000e_has_link(struct e1000_adapter *adapter)
case e1000_media_type_copper:
if (hw->mac.get_link_status) {
ret_val = hw->mac.ops.check_for_link(hw);
- link_active = ret_val > 0;
+ link_active = !hw->mac.get_link_status;
} else {
link_active = true;
}
diff --git a/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c b/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
index 0562938..ea5ea65 100644
--- a/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
+++ b/drivers/net/ethernet/intel/fm10k/fm10k_netdev.c
@@ -803,8 +803,12 @@ static int fm10k_update_vid(struct net_device *netdev, u16 vid, bool set)
if (vid >= VLAN_N_VID)
return -EINVAL;
- /* Verify we have permission to add VLANs */
- if (hw->mac.vlan_override)
+ /* Verify that we have permission to add VLANs. If this is a request
+ * to remove a VLAN, we still want to allow the user to remove the
+ * VLAN device. In that case, we need to clear the bit in the
+ * active_vlans bitmask.
+ */
+ if (set && hw->mac.vlan_override)
return -EACCES;
/* update active_vlans bitmask */
@@ -823,6 +827,12 @@ static int fm10k_update_vid(struct net_device *netdev, u16 vid, bool set)
rx_ring->vid &= ~FM10K_VLAN_CLEAR;
}
+ /* If our VLAN has been overridden, there is no reason to send VLAN
+ * removal requests as they will be silently ignored.
+ */
+ if (hw->mac.vlan_override)
+ return 0;
+
/* Do not remove default VLAN ID related entries from VLAN and MAC
* tables
*/
diff --git a/drivers/net/ethernet/intel/i40e/i40e_nvm.c b/drivers/net/ethernet/intel/i40e/i40e_nvm.c
index abe290b..8408682 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_nvm.c
+++ b/drivers/net/ethernet/intel/i40e/i40e_nvm.c
@@ -266,7 +266,7 @@ static i40e_status i40e_read_nvm_aq(struct i40e_hw *hw, u8 module_pointer,
* @offset: offset of the Shadow RAM word to read (0x000000 - 0x001FFF)
* @data: word read from the Shadow RAM
*
- * Reads one 16 bit word from the Shadow RAM using the GLNVM_SRCTL register.
+ * Reads one 16 bit word from the Shadow RAM using the AdminQ
**/
static i40e_status i40e_read_nvm_word_aq(struct i40e_hw *hw, u16 offset,
u16 *data)
@@ -280,27 +280,49 @@ static i40e_status i40e_read_nvm_word_aq(struct i40e_hw *hw, u16 offset,
}
/**
- * i40e_read_nvm_word - Reads Shadow RAM
+ * __i40e_read_nvm_word - Reads nvm word, assumes called does the locking
* @hw: pointer to the HW structure
* @offset: offset of the Shadow RAM word to read (0x000000 - 0x001FFF)
* @data: word read from the Shadow RAM
*
- * Reads one 16 bit word from the Shadow RAM using the GLNVM_SRCTL register.
+ * Reads one 16 bit word from the Shadow RAM.
+ *
+ * Do not use this function except in cases where the nvm lock is already
+ * taken via i40e_acquire_nvm().
+ **/
+static i40e_status __i40e_read_nvm_word(struct i40e_hw *hw,
+ u16 offset, u16 *data)
+{
+ i40e_status ret_code = 0;
+
+ if (hw->flags & I40E_HW_FLAG_AQ_SRCTL_ACCESS_ENABLE)
+ ret_code = i40e_read_nvm_word_aq(hw, offset, data);
+ else
+ ret_code = i40e_read_nvm_word_srctl(hw, offset, data);
+ return ret_code;
+}
+
+/**
+ * i40e_read_nvm_word - Reads nvm word and acquire lock if necessary
+ * @hw: pointer to the HW structure
+ * @offset: offset of the Shadow RAM word to read (0x000000 - 0x001FFF)
+ * @data: word read from the Shadow RAM
+ *
+ * Reads one 16 bit word from the Shadow RAM.
**/
i40e_status i40e_read_nvm_word(struct i40e_hw *hw, u16 offset,
u16 *data)
{
- enum i40e_status_code ret_code = 0;
+ i40e_status ret_code = 0;
ret_code = i40e_acquire_nvm(hw, I40E_RESOURCE_READ);
- if (!ret_code) {
- if (hw->flags & I40E_HW_FLAG_AQ_SRCTL_ACCESS_ENABLE) {
- ret_code = i40e_read_nvm_word_aq(hw, offset, data);
- } else {
- ret_code = i40e_read_nvm_word_srctl(hw, offset, data);
- }
- i40e_release_nvm(hw);
- }
+ if (ret_code)
+ return ret_code;
+
+ ret_code = __i40e_read_nvm_word(hw, offset, data);
+
+ i40e_release_nvm(hw);
+
return ret_code;
}
@@ -393,31 +415,25 @@ static i40e_status i40e_read_nvm_buffer_aq(struct i40e_hw *hw, u16 offset,
}
/**
- * i40e_read_nvm_buffer - Reads Shadow RAM buffer
+ * __i40e_read_nvm_buffer - Reads nvm buffer, caller must acquire lock
* @hw: pointer to the HW structure
* @offset: offset of the Shadow RAM word to read (0x000000 - 0x001FFF).
* @words: (in) number of words to read; (out) number of words actually read
* @data: words read from the Shadow RAM
*
* Reads 16 bit words (data buffer) from the SR using the i40e_read_nvm_srrd()
- * method. The buffer read is preceded by the NVM ownership take
- * and followed by the release.
+ * method.
**/
-i40e_status i40e_read_nvm_buffer(struct i40e_hw *hw, u16 offset,
- u16 *words, u16 *data)
+static i40e_status __i40e_read_nvm_buffer(struct i40e_hw *hw,
+ u16 offset, u16 *words,
+ u16 *data)
{
- enum i40e_status_code ret_code = 0;
+ i40e_status ret_code = 0;
- if (hw->flags & I40E_HW_FLAG_AQ_SRCTL_ACCESS_ENABLE) {
- ret_code = i40e_acquire_nvm(hw, I40E_RESOURCE_READ);
- if (!ret_code) {
- ret_code = i40e_read_nvm_buffer_aq(hw, offset, words,
- data);
- i40e_release_nvm(hw);
- }
- } else {
+ if (hw->flags & I40E_HW_FLAG_AQ_SRCTL_ACCESS_ENABLE)
+ ret_code = i40e_read_nvm_buffer_aq(hw, offset, words, data);
+ else
ret_code = i40e_read_nvm_buffer_srctl(hw, offset, words, data);
- }
return ret_code;
}
@@ -499,15 +515,15 @@ static i40e_status i40e_calc_nvm_checksum(struct i40e_hw *hw,
data = (u16 *)vmem.va;
/* read pointer to VPD area */
- ret_code = i40e_read_nvm_word(hw, I40E_SR_VPD_PTR, &vpd_module);
+ ret_code = __i40e_read_nvm_word(hw, I40E_SR_VPD_PTR, &vpd_module);
if (ret_code) {
ret_code = I40E_ERR_NVM_CHECKSUM;
goto i40e_calc_nvm_checksum_exit;
}
/* read pointer to PCIe Alt Auto-load module */
- ret_code = i40e_read_nvm_word(hw, I40E_SR_PCIE_ALT_AUTO_LOAD_PTR,
- &pcie_alt_module);
+ ret_code = __i40e_read_nvm_word(hw, I40E_SR_PCIE_ALT_AUTO_LOAD_PTR,
+ &pcie_alt_module);
if (ret_code) {
ret_code = I40E_ERR_NVM_CHECKSUM;
goto i40e_calc_nvm_checksum_exit;
@@ -521,7 +537,7 @@ static i40e_status i40e_calc_nvm_checksum(struct i40e_hw *hw,
if ((i % I40E_SR_SECTOR_SIZE_IN_WORDS) == 0) {
u16 words = I40E_SR_SECTOR_SIZE_IN_WORDS;
- ret_code = i40e_read_nvm_buffer(hw, i, &words, data);
+ ret_code = __i40e_read_nvm_buffer(hw, i, &words, data);
if (ret_code) {
ret_code = I40E_ERR_NVM_CHECKSUM;
goto i40e_calc_nvm_checksum_exit;
@@ -593,14 +609,19 @@ i40e_status i40e_validate_nvm_checksum(struct i40e_hw *hw,
u16 checksum_sr = 0;
u16 checksum_local = 0;
- ret_code = i40e_calc_nvm_checksum(hw, &checksum_local);
- if (ret_code)
- goto i40e_validate_nvm_checksum_exit;
-
- /* Do not use i40e_read_nvm_word() because we do not want to take
- * the synchronization semaphores twice here.
+ /* We must acquire the NVM lock in order to correctly synchronize the
+ * NVM accesses across multiple PFs. Without doing so it is possible
+ * for one of the PFs to read invalid data potentially indicating that
+ * the checksum is invalid.
*/
- i40e_read_nvm_word(hw, I40E_SR_SW_CHECKSUM_WORD, &checksum_sr);
+ ret_code = i40e_acquire_nvm(hw, I40E_RESOURCE_READ);
+ if (ret_code)
+ return ret_code;
+ ret_code = i40e_calc_nvm_checksum(hw, &checksum_local);
+ __i40e_read_nvm_word(hw, I40E_SR_SW_CHECKSUM_WORD, &checksum_sr);
+ i40e_release_nvm(hw);
+ if (ret_code)
+ return ret_code;
/* Verify read checksum from EEPROM is the same as
* calculated checksum
@@ -612,7 +633,6 @@ i40e_status i40e_validate_nvm_checksum(struct i40e_hw *hw,
if (checksum)
*checksum = checksum_local;
-i40e_validate_nvm_checksum_exit:
return ret_code;
}
@@ -986,6 +1006,7 @@ static i40e_status i40e_nvmupd_state_writing(struct i40e_hw *hw,
break;
case I40E_NVMUPD_CSUM_CON:
+ /* Assumes the caller has acquired the nvm */
status = i40e_update_nvm_checksum(hw);
if (status) {
*perrno = hw->aq.asq_last_status ?
@@ -1000,6 +1021,7 @@ static i40e_status i40e_nvmupd_state_writing(struct i40e_hw *hw,
break;
case I40E_NVMUPD_CSUM_LCB:
+ /* Assumes the caller has acquired the nvm */
status = i40e_update_nvm_checksum(hw);
if (status) {
*perrno = hw->aq.asq_last_status ?
diff --git a/drivers/net/ethernet/intel/i40e/i40e_prototype.h b/drivers/net/ethernet/intel/i40e/i40e_prototype.h
index 4660c5a..6b36411 100644
--- a/drivers/net/ethernet/intel/i40e/i40e_prototype.h
+++ b/drivers/net/ethernet/intel/i40e/i40e_prototype.h
@@ -311,8 +311,6 @@ i40e_status i40e_acquire_nvm(struct i40e_hw *hw,
void i40e_release_nvm(struct i40e_hw *hw);
i40e_status i40e_read_nvm_word(struct i40e_hw *hw, u16 offset,
u16 *data);
-i40e_status i40e_read_nvm_buffer(struct i40e_hw *hw, u16 offset,
- u16 *words, u16 *data);
i40e_status i40e_update_nvm_checksum(struct i40e_hw *hw);
i40e_status i40e_validate_nvm_checksum(struct i40e_hw *hw,
u16 *checksum);
diff --git a/drivers/net/ethernet/intel/igb/e1000_82575.c b/drivers/net/ethernet/intel/igb/e1000_82575.c
index 4a50870..a61447f 100644
--- a/drivers/net/ethernet/intel/igb/e1000_82575.c
+++ b/drivers/net/ethernet/intel/igb/e1000_82575.c
@@ -245,19 +245,7 @@ static s32 igb_init_phy_params_82575(struct e1000_hw *hw)
hw->bus.func = (rd32(E1000_STATUS) & E1000_STATUS_FUNC_MASK) >>
E1000_STATUS_FUNC_SHIFT;
- /* Make sure the PHY is in a good state. Several people have reported
- * firmware leaving the PHY's page select register set to something
- * other than the default of zero, which causes the PHY ID read to
- * access something other than the intended register.
- */
- ret_val = hw->phy.ops.reset(hw);
- if (ret_val) {
- hw_dbg("Error resetting the PHY.\n");
- goto out;
- }
-
/* Set phy->phy_addr and phy->id. */
- igb_write_phy_reg_82580(hw, I347AT4_PAGE_SELECT, 0);
ret_val = igb_get_phy_id_82575(hw);
if (ret_val)
return ret_val;
diff --git a/drivers/net/ethernet/intel/igb/e1000_i210.c b/drivers/net/ethernet/intel/igb/e1000_i210.c
index 07d48f2..6766081 100644
--- a/drivers/net/ethernet/intel/igb/e1000_i210.c
+++ b/drivers/net/ethernet/intel/igb/e1000_i210.c
@@ -862,6 +862,7 @@ s32 igb_pll_workaround_i210(struct e1000_hw *hw)
nvm_word = E1000_INVM_DEFAULT_AL;
tmp_nvm = nvm_word | E1000_INVM_PLL_WO_VAL;
igb_write_phy_reg_82580(hw, I347AT4_PAGE_SELECT, E1000_PHY_PLL_FREQ_PAGE);
+ phy_word = E1000_PHY_PLL_UNCONF;
for (i = 0; i < E1000_MAX_PLL_TRIES; i++) {
/* check current state directly from internal PHY */
igb_read_phy_reg_82580(hw, E1000_PHY_PLL_FREQ_REG, &phy_word);
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
index ad33622..0d2baec 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_common.c
@@ -1847,7 +1847,12 @@ s32 ixgbe_set_rar_generic(struct ixgbe_hw *hw, u32 index, u8 *addr, u32 vmdq,
if (enable_addr != 0)
rar_high |= IXGBE_RAH_AV;
+ /* Record lower 32 bits of MAC address and then make
+ * sure that write is flushed to hardware before writing
+ * the upper 16 bits and setting the valid bit.
+ */
IXGBE_WRITE_REG(hw, IXGBE_RAL(index), rar_low);
+ IXGBE_WRITE_FLUSH(hw);
IXGBE_WRITE_REG(hw, IXGBE_RAH(index), rar_high);
return 0;
@@ -1879,8 +1884,13 @@ s32 ixgbe_clear_rar_generic(struct ixgbe_hw *hw, u32 index)
rar_high = IXGBE_READ_REG(hw, IXGBE_RAH(index));
rar_high &= ~(0x0000FFFF | IXGBE_RAH_AV);
- IXGBE_WRITE_REG(hw, IXGBE_RAL(index), 0);
+ /* Clear the address valid bit and upper 16 bits of the address
+ * before clearing the lower bits. This way we aren't updating
+ * a live filter.
+ */
IXGBE_WRITE_REG(hw, IXGBE_RAH(index), rar_high);
+ IXGBE_WRITE_FLUSH(hw);
+ IXGBE_WRITE_REG(hw, IXGBE_RAL(index), 0);
/* clear VMDq pool/queue selection for this RAR */
hw->mac.ops.clear_vmdq(hw, index, IXGBE_CLEAR_VMDQ_ALL);
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c
index 77a60aa..8466f38 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_x550.c
@@ -1702,7 +1702,9 @@ static s32 ixgbe_get_link_capabilities_X550em(struct ixgbe_hw *hw,
*autoneg = false;
if (hw->phy.sfp_type == ixgbe_sfp_type_1g_sx_core0 ||
- hw->phy.sfp_type == ixgbe_sfp_type_1g_sx_core1) {
+ hw->phy.sfp_type == ixgbe_sfp_type_1g_sx_core1 ||
+ hw->phy.sfp_type == ixgbe_sfp_type_1g_lx_core0 ||
+ hw->phy.sfp_type == ixgbe_sfp_type_1g_lx_core1) {
*speed = IXGBE_LINK_SPEED_1GB_FULL;
return 0;
}
diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
index 1499ce2b..7560726 100644
--- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
+++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c
@@ -3419,6 +3419,10 @@ static void ixgbevf_tx_csum(struct ixgbevf_ring *tx_ring,
skb_checksum_help(skb);
goto no_csum;
}
+
+ if (first->protocol == htons(ETH_P_IP))
+ type_tucmd |= IXGBE_ADVTXD_TUCMD_IPV4;
+
/* update TX checksum flag */
first->tx_flags |= IXGBE_TX_FLAGS_CSUM;
vlan_macip_lens = skb_checksum_start_offset(skb) -
@@ -3729,6 +3733,7 @@ static int ixgbevf_set_mac(struct net_device *netdev, void *p)
return -EPERM;
ether_addr_copy(hw->mac.addr, addr->sa_data);
+ ether_addr_copy(hw->mac.perm_addr, addr->sa_data);
ether_addr_copy(netdev->dev_addr, addr->sa_data);
return 0;
diff --git a/drivers/net/ethernet/marvell/mvneta.c b/drivers/net/ethernet/marvell/mvneta.c
index fa46326..c92ffdf9 100644
--- a/drivers/net/ethernet/marvell/mvneta.c
+++ b/drivers/net/ethernet/marvell/mvneta.c
@@ -1080,6 +1080,7 @@ static void mvneta_port_up(struct mvneta_port *pp)
}
mvreg_write(pp, MVNETA_TXQ_CMD, q_map);
+ q_map = 0;
/* Enable all initialized RXQs. */
for (queue = 0; queue < rxq_number; queue++) {
struct mvneta_rx_queue *rxq = &pp->rxqs[queue];
@@ -3116,7 +3117,6 @@ static int mvneta_change_mtu(struct net_device *dev, int mtu)
on_each_cpu(mvneta_percpu_enable, pp, true);
mvneta_start_dev(pp);
- mvneta_port_up(pp);
netdev_update_features(dev);
diff --git a/drivers/net/ethernet/marvell/mvpp2.c b/drivers/net/ethernet/marvell/mvpp2.c
index 7e2ebfc..ff62dc7 100644
--- a/drivers/net/ethernet/marvell/mvpp2.c
+++ b/drivers/net/ethernet/marvell/mvpp2.c
@@ -29,6 +29,7 @@
#include <linux/clk.h>
#include <linux/hrtimer.h>
#include <linux/ktime.h>
+#include <linux/if_vlan.h>
#include <uapi/linux/ppp_defs.h>
#include <net/ip.h>
#include <net/ipv6.h>
@@ -4266,7 +4267,7 @@ static void mvpp2_txq_desc_put(struct mvpp2_tx_queue *txq)
}
/* Set Tx descriptors fields relevant for CSUM calculation */
-static u32 mvpp2_txq_desc_csum(int l3_offs, int l3_proto,
+static u32 mvpp2_txq_desc_csum(int l3_offs, __be16 l3_proto,
int ip_hdr_len, int l4_proto)
{
u32 command;
@@ -5019,14 +5020,15 @@ static u32 mvpp2_skb_tx_csum(struct mvpp2_port *port, struct sk_buff *skb)
if (skb->ip_summed == CHECKSUM_PARTIAL) {
int ip_hdr_len = 0;
u8 l4_proto;
+ __be16 l3_proto = vlan_get_protocol(skb);
- if (skb->protocol == htons(ETH_P_IP)) {
+ if (l3_proto == htons(ETH_P_IP)) {
struct iphdr *ip4h = ip_hdr(skb);
/* Calculate IPv4 checksum and L4 checksum */
ip_hdr_len = ip4h->ihl;
l4_proto = ip4h->protocol;
- } else if (skb->protocol == htons(ETH_P_IPV6)) {
+ } else if (l3_proto == htons(ETH_P_IPV6)) {
struct ipv6hdr *ip6h = ipv6_hdr(skb);
/* Read l4_protocol from one of IPv6 extra headers */
@@ -5038,7 +5040,7 @@ static u32 mvpp2_skb_tx_csum(struct mvpp2_port *port, struct sk_buff *skb)
}
return mvpp2_txq_desc_csum(skb_network_offset(skb),
- skb->protocol, ip_hdr_len, l4_proto);
+ l3_proto, ip_hdr_len, l4_proto);
}
return MVPP2_TXD_L4_CSUM_NOT | MVPP2_TXD_IP_CSUM_DISABLE;
diff --git a/drivers/net/ethernet/mellanox/mlx4/Kconfig b/drivers/net/ethernet/mellanox/mlx4/Kconfig
index 5098e7f..a0eb4e4 100644
--- a/drivers/net/ethernet/mellanox/mlx4/Kconfig
+++ b/drivers/net/ethernet/mellanox/mlx4/Kconfig
@@ -5,7 +5,7 @@
config MLX4_EN
tristate "Mellanox Technologies 1/10/40Gbit Ethernet support"
depends on MAY_USE_DEVLINK
- depends on PCI
+ depends on PCI && NETDEVICES && ETHERNET && INET
select MLX4_CORE
select PTP_1588_CLOCK
---help---
diff --git a/drivers/net/ethernet/mellanox/mlx4/alloc.c b/drivers/net/ethernet/mellanox/mlx4/alloc.c
index 249a458..b89a34f 100644
--- a/drivers/net/ethernet/mellanox/mlx4/alloc.c
+++ b/drivers/net/ethernet/mellanox/mlx4/alloc.c
@@ -339,7 +339,7 @@ void mlx4_zone_allocator_destroy(struct mlx4_zone_allocator *zone_alloc)
static u32 __mlx4_alloc_from_zone(struct mlx4_zone_entry *zone, int count,
int align, u32 skip_mask, u32 *puid)
{
- u32 uid;
+ u32 uid = 0;
u32 res;
struct mlx4_zone_allocator *zone_alloc = zone->allocator;
struct mlx4_zone_entry *curr_node;
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c
index 24977cc..8a9a332d 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_ethtool.c
@@ -970,6 +970,22 @@ static int mlx4_en_set_coalesce(struct net_device *dev,
if (!coal->tx_max_coalesced_frames_irq)
return -EINVAL;
+ if (coal->tx_coalesce_usecs > MLX4_EN_MAX_COAL_TIME ||
+ coal->rx_coalesce_usecs > MLX4_EN_MAX_COAL_TIME ||
+ coal->rx_coalesce_usecs_low > MLX4_EN_MAX_COAL_TIME ||
+ coal->rx_coalesce_usecs_high > MLX4_EN_MAX_COAL_TIME) {
+ netdev_info(dev, "%s: maximum coalesce time supported is %d usecs\n",
+ __func__, MLX4_EN_MAX_COAL_TIME);
+ return -ERANGE;
+ }
+
+ if (coal->tx_max_coalesced_frames > MLX4_EN_MAX_COAL_PKTS ||
+ coal->rx_max_coalesced_frames > MLX4_EN_MAX_COAL_PKTS) {
+ netdev_info(dev, "%s: maximum coalesced frames supported is %d\n",
+ __func__, MLX4_EN_MAX_COAL_PKTS);
+ return -ERANGE;
+ }
+
priv->rx_frames = (coal->rx_max_coalesced_frames ==
MLX4_EN_AUTO_CONF) ?
MLX4_EN_RX_COAL_TARGET :
@@ -1011,8 +1027,8 @@ static int mlx4_en_set_pauseparam(struct net_device *dev,
tx_pause = !!(pause->tx_pause);
rx_pause = !!(pause->rx_pause);
- rx_ppp = priv->prof->rx_ppp && !(tx_pause || rx_pause);
- tx_ppp = priv->prof->tx_ppp && !(tx_pause || rx_pause);
+ rx_ppp = (tx_pause || rx_pause) ? 0 : priv->prof->rx_ppp;
+ tx_ppp = (tx_pause || rx_pause) ? 0 : priv->prof->tx_ppp;
err = mlx4_SET_PORT_general(mdev->dev, priv->port,
priv->rx_skb_size + ETH_FCS_LEN,
diff --git a/drivers/net/ethernet/mellanox/mlx4/en_rx.c b/drivers/net/ethernet/mellanox/mlx4/en_rx.c
index bcbb80f..1a92cd7 100644
--- a/drivers/net/ethernet/mellanox/mlx4/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx4/en_rx.c
@@ -142,16 +142,17 @@ static void mlx4_en_free_frag(struct mlx4_en_priv *priv,
struct mlx4_en_rx_alloc *frags,
int i)
{
- const struct mlx4_en_frag_info *frag_info = &priv->frag_info[i];
- u32 next_frag_end = frags[i].page_offset + 2 * frag_info->frag_stride;
+ if (frags[i].page) {
+ const struct mlx4_en_frag_info *frag_info = &priv->frag_info[i];
+ u32 next_frag_end = frags[i].page_offset +
+ 2 * frag_info->frag_stride;
-
- if (next_frag_end > frags[i].page_size)
- dma_unmap_page(priv->ddev, frags[i].dma, frags[i].page_size,
- frag_info->dma_dir);
-
- if (frags[i].page)
+ if (next_frag_end > frags[i].page_size) {
+ dma_unmap_page(priv->ddev, frags[i].dma,
+ frags[i].page_size, frag_info->dma_dir);
+ }
put_page(frags[i].page);
+ }
}
static int mlx4_en_init_allocator(struct mlx4_en_priv *priv,
@@ -586,21 +587,28 @@ static int mlx4_en_complete_rx_desc(struct mlx4_en_priv *priv,
int length)
{
struct skb_frag_struct *skb_frags_rx = skb_shinfo(skb)->frags;
- struct mlx4_en_frag_info *frag_info;
int nr;
dma_addr_t dma;
/* Collect used fragments while replacing them in the HW descriptors */
for (nr = 0; nr < priv->num_frags; nr++) {
- frag_info = &priv->frag_info[nr];
+ struct mlx4_en_frag_info *frag_info = &priv->frag_info[nr];
+ u32 next_frag_end = frags[nr].page_offset +
+ 2 * frag_info->frag_stride;
+
if (length <= frag_info->frag_prefix_size)
break;
if (unlikely(!frags[nr].page))
goto fail;
dma = be64_to_cpu(rx_desc->data[nr].addr);
- dma_sync_single_for_cpu(priv->ddev, dma, frag_info->frag_size,
- DMA_FROM_DEVICE);
+ if (next_frag_end > frags[nr].page_size)
+ dma_unmap_page(priv->ddev, frags[nr].dma,
+ frags[nr].page_size, frag_info->dma_dir);
+ else
+ dma_sync_single_for_cpu(priv->ddev, dma,
+ frag_info->frag_size,
+ DMA_FROM_DEVICE);
/* Save page reference in skb */
__skb_frag_set_page(&skb_frags_rx[nr], frags[nr].page);
diff --git a/drivers/net/ethernet/mellanox/mlx4/eq.c b/drivers/net/ethernet/mellanox/mlx4/eq.c
index 0509996..e70d6fe 100644
--- a/drivers/net/ethernet/mellanox/mlx4/eq.c
+++ b/drivers/net/ethernet/mellanox/mlx4/eq.c
@@ -240,7 +240,8 @@ static void mlx4_set_eq_affinity_hint(struct mlx4_priv *priv, int vec)
struct mlx4_dev *dev = &priv->dev;
struct mlx4_eq *eq = &priv->eq_table.eq[vec];
- if (!eq->affinity_mask || cpumask_empty(eq->affinity_mask))
+ if (!cpumask_available(eq->affinity_mask) ||
+ cpumask_empty(eq->affinity_mask))
return;
hint_err = irq_set_affinity_hint(eq->irq, eq->affinity_mask);
diff --git a/drivers/net/ethernet/mellanox/mlx4/main.c b/drivers/net/ethernet/mellanox/mlx4/main.c
index 5411ca4..cb7c3ef 100644
--- a/drivers/net/ethernet/mellanox/mlx4/main.c
+++ b/drivers/net/ethernet/mellanox/mlx4/main.c
@@ -2983,6 +2983,7 @@ static int mlx4_init_port_info(struct mlx4_dev *dev, int port)
mlx4_err(dev, "Failed to create file for port %d\n", port);
devlink_port_unregister(&info->devlink_port);
info->port = -1;
+ return err;
}
sprintf(info->dev_mtu_name, "mlx4_port%d_mtu", port);
@@ -3004,9 +3005,10 @@ static int mlx4_init_port_info(struct mlx4_dev *dev, int port)
&info->port_attr);
devlink_port_unregister(&info->devlink_port);
info->port = -1;
+ return err;
}
- return err;
+ return 0;
}
static void mlx4_cleanup_port_info(struct mlx4_port_info *info)
diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4.h b/drivers/net/ethernet/mellanox/mlx4/mlx4.h
index 086920b6..df5d86f 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4.h
@@ -542,8 +542,8 @@ struct slave_list {
struct resource_allocator {
spinlock_t alloc_lock; /* protect quotas */
union {
- int res_reserved;
- int res_port_rsvd[MLX4_MAX_PORTS];
+ unsigned int res_reserved;
+ unsigned int res_port_rsvd[MLX4_MAX_PORTS];
};
union {
int res_free;
diff --git a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
index 18f221d..247d340 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
+++ b/drivers/net/ethernet/mellanox/mlx4/mlx4_en.h
@@ -141,6 +141,9 @@ enum {
#define MLX4_EN_TX_COAL_PKTS 16
#define MLX4_EN_TX_COAL_TIME 0x10
+#define MLX4_EN_MAX_COAL_PKTS U16_MAX
+#define MLX4_EN_MAX_COAL_TIME U16_MAX
+
#define MLX4_EN_RX_RATE_LOW 400000
#define MLX4_EN_RX_COAL_TIME_LOW 0
#define MLX4_EN_RX_RATE_HIGH 450000
@@ -543,8 +546,8 @@ struct mlx4_en_priv {
u16 rx_usecs_low;
u32 pkt_rate_high;
u16 rx_usecs_high;
- u16 sample_interval;
- u16 adaptive_rx_coal;
+ u32 sample_interval;
+ u32 adaptive_rx_coal;
u32 msg_enable;
u32 loopback_ok;
u32 validate_loopback;
diff --git a/drivers/net/ethernet/mellanox/mlx4/mr.c b/drivers/net/ethernet/mellanox/mlx4/mr.c
index 395b546..3637474 100644
--- a/drivers/net/ethernet/mellanox/mlx4/mr.c
+++ b/drivers/net/ethernet/mellanox/mlx4/mr.c
@@ -366,6 +366,7 @@ int mlx4_mr_hw_write_mpt(struct mlx4_dev *dev, struct mlx4_mr *mmr,
container_of((void *)mpt_entry, struct mlx4_cmd_mailbox,
buf);
+ (*mpt_entry)->lkey = 0;
err = mlx4_SW2HW_MPT(dev, mailbox, key);
}
diff --git a/drivers/net/ethernet/mellanox/mlx4/qp.c b/drivers/net/ethernet/mellanox/mlx4/qp.c
index 474ff36..71578d4 100644
--- a/drivers/net/ethernet/mellanox/mlx4/qp.c
+++ b/drivers/net/ethernet/mellanox/mlx4/qp.c
@@ -392,11 +392,11 @@ struct mlx4_qp *mlx4_qp_lookup(struct mlx4_dev *dev, u32 qpn)
struct mlx4_qp_table *qp_table = &mlx4_priv(dev)->qp_table;
struct mlx4_qp *qp;
- spin_lock(&qp_table->lock);
+ spin_lock_irq(&qp_table->lock);
qp = __mlx4_qp_lookup(dev, qpn);
- spin_unlock(&qp_table->lock);
+ spin_unlock_irq(&qp_table->lock);
return qp;
}
diff --git a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
index d6b06be..9d1a7d5 100644
--- a/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
+++ b/drivers/net/ethernet/mellanox/mlx4/resource_tracker.c
@@ -2916,7 +2916,7 @@ int mlx4_RST2INIT_QP_wrapper(struct mlx4_dev *dev, int slave,
u32 srqn = qp_get_srqn(qpc) & 0xffffff;
int use_srq = (qp_get_srqn(qpc) >> 24) & 1;
struct res_srq *srq;
- int local_qpn = be32_to_cpu(qpc->local_qpn) & 0xffffff;
+ int local_qpn = vhcr->in_modifier & 0xffffff;
err = adjust_qp_sched_queue(dev, slave, qpc, inbox);
if (err)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
index 4c3f1cb..1d5263c 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/cmd.c
@@ -784,6 +784,7 @@ static void cmd_work_handler(struct work_struct *work)
struct semaphore *sem;
unsigned long flags;
int alloc_ret;
+ int cmd_mode;
sem = ent->page_queue ? &cmd->pages_sem : &cmd->sem;
down(sem);
@@ -830,6 +831,7 @@ static void cmd_work_handler(struct work_struct *work)
set_signature(ent, !cmd->checksum_disabled);
dump_command(dev, ent, 1);
ent->ts1 = ktime_get_ns();
+ cmd_mode = cmd->mode;
if (ent->callback)
schedule_delayed_work(&ent->cb_timeout_work, cb_timeout);
@@ -854,7 +856,7 @@ static void cmd_work_handler(struct work_struct *work)
iowrite32be(1 << ent->idx, &dev->iseg->cmd_dbell);
mmiowb();
/* if not in polling don't use ent after this point */
- if (cmd->mode == CMD_MODE_POLLING) {
+ if (cmd_mode == CMD_MODE_POLLING) {
poll_timeout(ent);
/* make sure we read the descriptor after ownership is SW */
rmb();
@@ -963,7 +965,7 @@ static int mlx5_cmd_invoke(struct mlx5_core_dev *dev, struct mlx5_cmd_msg *in,
err = wait_func(dev, ent);
if (err == -ETIMEDOUT)
- goto out_free;
+ goto out;
ds = ent->ts2 - ent->ts1;
op = MLX5_GET(mbox_in, in->first.data, opcode);
@@ -1256,7 +1258,7 @@ static ssize_t outlen_write(struct file *filp, const char __user *buf,
{
struct mlx5_core_dev *dev = filp->private_data;
struct mlx5_cmd_debug *dbg = &dev->cmd.dbg;
- char outlen_str[8];
+ char outlen_str[8] = {0};
int outlen;
void *ptr;
int err;
@@ -1271,8 +1273,6 @@ static ssize_t outlen_write(struct file *filp, const char __user *buf,
if (copy_from_user(outlen_str, buf, count))
return -EFAULT;
- outlen_str[7] = 0;
-
err = sscanf(outlen_str, "%d", &outlen);
if (err < 0)
return err;
@@ -1428,6 +1428,7 @@ void mlx5_cmd_comp_handler(struct mlx5_core_dev *dev, u64 vec, bool forced)
mlx5_core_err(dev, "Command completion arrived after timeout (entry idx = %d).\n",
ent->idx);
free_ent(cmd, ent->idx);
+ free_cmd(ent);
}
continue;
}
@@ -1486,7 +1487,8 @@ void mlx5_cmd_comp_handler(struct mlx5_core_dev *dev, u64 vec, bool forced)
free_msg(dev, ent->in);
err = err ? err : ent->status;
- free_cmd(ent);
+ if (!forced)
+ free_cmd(ent);
callback(err, context);
} else {
complete(&ent->done);
@@ -1765,7 +1767,7 @@ int mlx5_cmd_init(struct mlx5_core_dev *dev)
cmd->checksum_disabled = 1;
cmd->max_reg_cmds = (1 << cmd->log_sz) - 1;
- cmd->bitmask = (1 << cmd->max_reg_cmds) - 1;
+ cmd->bitmask = (1UL << cmd->max_reg_cmds) - 1;
cmd->cmdif_rev = ioread32be(&dev->iseg->cmdif_rev_fw_sub) >> 16;
if (cmd->cmdif_rev > CMD_IF_REV) {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/dev.c b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
index a9dbc28..524fff2 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/dev.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/dev.c
@@ -288,16 +288,17 @@ void mlx5_remove_dev_by_protocol(struct mlx5_core_dev *dev, int protocol)
}
}
-static u16 mlx5_gen_pci_id(struct mlx5_core_dev *dev)
+static u32 mlx5_gen_pci_id(struct mlx5_core_dev *dev)
{
- return (u16)((dev->pdev->bus->number << 8) |
+ return (u32)((pci_domain_nr(dev->pdev->bus) << 16) |
+ (dev->pdev->bus->number << 8) |
PCI_SLOT(dev->pdev->devfn));
}
/* Must be called with intf_mutex held */
struct mlx5_core_dev *mlx5_get_next_phys_dev(struct mlx5_core_dev *dev)
{
- u16 pci_id = mlx5_gen_pci_id(dev);
+ u32 pci_id = mlx5_gen_pci_id(dev);
struct mlx5_core_dev *res = NULL;
struct mlx5_core_dev *tmp_dev;
struct mlx5_priv *priv;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
index a8cb387..4a51fc6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_arfs.c
@@ -383,14 +383,14 @@ static void arfs_may_expire_flow(struct mlx5e_priv *priv)
HLIST_HEAD(del_list);
spin_lock_bh(&priv->fs.arfs.arfs_lock);
mlx5e_for_each_arfs_rule(arfs_rule, htmp, priv->fs.arfs.arfs_tables, i, j) {
- if (quota++ > MLX5E_ARFS_EXPIRY_QUOTA)
- break;
if (!work_pending(&arfs_rule->arfs_work) &&
rps_may_expire_flow(priv->netdev,
arfs_rule->rxq, arfs_rule->flow_id,
arfs_rule->filter_id)) {
hlist_del_init(&arfs_rule->hlist);
hlist_add_head(&arfs_rule->hlist, &del_list);
+ if (quota++ > MLX5E_ARFS_EXPIRY_QUOTA)
+ break;
}
}
spin_unlock_bh(&priv->fs.arfs.arfs_lock);
@@ -715,6 +715,9 @@ int mlx5e_rx_flow_steer(struct net_device *dev, const struct sk_buff *skb,
skb->protocol != htons(ETH_P_IPV6))
return -EPROTONOSUPPORT;
+ if (skb->encapsulation)
+ return -EPROTONOSUPPORT;
+
arfs_t = arfs_get_table(arfs, arfs_get_ip_proto(skb), skb->protocol);
if (!arfs_t)
return -EPROTONOSUPPORT;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_clock.c b/drivers/net/ethernet/mellanox/mlx5/core/en_clock.c
index 1612ec0..f8b99d0 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_clock.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_clock.c
@@ -233,6 +233,7 @@ static void mlx5e_timestamp_init_config(struct mlx5e_tstamp *tstamp)
void mlx5e_timestamp_init(struct mlx5e_priv *priv)
{
struct mlx5e_tstamp *tstamp = &priv->tstamp;
+ u64 overflow_cycles;
u64 ns;
u64 frac = 0;
u32 dev_freq;
@@ -257,10 +258,17 @@ void mlx5e_timestamp_init(struct mlx5e_priv *priv)
/* Calculate period in seconds to call the overflow watchdog - to make
* sure counter is checked at least once every wrap around.
+ * The period is calculated as the minimum between max HW cycles count
+ * (The clock source mask) and max amount of cycles that can be
+ * multiplied by clock multiplier where the result doesn't exceed
+ * 64bits.
*/
- ns = cyclecounter_cyc2ns(&tstamp->cycles, tstamp->cycles.mask,
+ overflow_cycles = div64_u64(~0ULL >> 1, tstamp->cycles.mult);
+ overflow_cycles = min(overflow_cycles, tstamp->cycles.mask >> 1);
+
+ ns = cyclecounter_cyc2ns(&tstamp->cycles, overflow_cycles,
frac, &frac);
- do_div(ns, NSEC_PER_SEC / 2 / HZ);
+ do_div(ns, NSEC_PER_SEC / HZ);
tstamp->overflow_period = ns;
INIT_DELAYED_WORK(&tstamp->overflow_work, mlx5e_timestamp_overflow);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
index da1d73f..d5e8ac8 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_ethtool.c
@@ -1167,11 +1167,6 @@ static int mlx5e_get_ts_info(struct net_device *dev,
struct ethtool_ts_info *info)
{
struct mlx5e_priv *priv = netdev_priv(dev);
- int ret;
-
- ret = ethtool_op_get_ts_info(dev, info);
- if (ret)
- return ret;
info->phc_index = priv->tstamp.ptp ?
ptp_clock_index(priv->tstamp.ptp) : -1;
@@ -1179,9 +1174,9 @@ static int mlx5e_get_ts_info(struct net_device *dev,
if (!MLX5_CAP_GEN(priv->mdev, device_frequency_khz))
return 0;
- info->so_timestamping |= SOF_TIMESTAMPING_TX_HARDWARE |
- SOF_TIMESTAMPING_RX_HARDWARE |
- SOF_TIMESTAMPING_RAW_HARDWARE;
+ info->so_timestamping = SOF_TIMESTAMPING_TX_HARDWARE |
+ SOF_TIMESTAMPING_RX_HARDWARE |
+ SOF_TIMESTAMPING_RAW_HARDWARE;
info->tx_types = BIT(HWTSTAMP_TX_OFF) |
BIT(HWTSTAMP_TX_ON);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
index 23ccec4..a1f3556 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx_am.c
@@ -197,9 +197,15 @@ static int mlx5e_am_stats_compare(struct mlx5e_rx_am_stats *curr,
return (curr->bpms > prev->bpms) ? MLX5E_AM_STATS_BETTER :
MLX5E_AM_STATS_WORSE;
+ if (!prev->ppms)
+ return curr->ppms ? MLX5E_AM_STATS_BETTER :
+ MLX5E_AM_STATS_SAME;
+
if (IS_SIGNIFICANT_DIFF(curr->ppms, prev->ppms))
return (curr->ppms > prev->ppms) ? MLX5E_AM_STATS_BETTER :
MLX5E_AM_STATS_WORSE;
+ if (!prev->epms)
+ return MLX5E_AM_STATS_SAME;
if (IS_SIGNIFICANT_DIFF(curr->epms, prev->epms))
return (curr->epms < prev->epms) ? MLX5E_AM_STATS_BETTER :
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
index a8966e6..5d6eab1 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
@@ -1924,26 +1924,35 @@ int mlx5_eswitch_get_vport_stats(struct mlx5_eswitch *esw,
memset(vf_stats, 0, sizeof(*vf_stats));
vf_stats->rx_packets =
MLX5_GET_CTR(out, received_eth_unicast.packets) +
+ MLX5_GET_CTR(out, received_ib_unicast.packets) +
MLX5_GET_CTR(out, received_eth_multicast.packets) +
+ MLX5_GET_CTR(out, received_ib_multicast.packets) +
MLX5_GET_CTR(out, received_eth_broadcast.packets);
vf_stats->rx_bytes =
MLX5_GET_CTR(out, received_eth_unicast.octets) +
+ MLX5_GET_CTR(out, received_ib_unicast.octets) +
MLX5_GET_CTR(out, received_eth_multicast.octets) +
+ MLX5_GET_CTR(out, received_ib_multicast.octets) +
MLX5_GET_CTR(out, received_eth_broadcast.octets);
vf_stats->tx_packets =
MLX5_GET_CTR(out, transmitted_eth_unicast.packets) +
+ MLX5_GET_CTR(out, transmitted_ib_unicast.packets) +
MLX5_GET_CTR(out, transmitted_eth_multicast.packets) +
+ MLX5_GET_CTR(out, transmitted_ib_multicast.packets) +
MLX5_GET_CTR(out, transmitted_eth_broadcast.packets);
vf_stats->tx_bytes =
MLX5_GET_CTR(out, transmitted_eth_unicast.octets) +
+ MLX5_GET_CTR(out, transmitted_ib_unicast.octets) +
MLX5_GET_CTR(out, transmitted_eth_multicast.octets) +
+ MLX5_GET_CTR(out, transmitted_ib_multicast.octets) +
MLX5_GET_CTR(out, transmitted_eth_broadcast.octets);
vf_stats->multicast =
- MLX5_GET_CTR(out, received_eth_multicast.packets);
+ MLX5_GET_CTR(out, received_eth_multicast.packets) +
+ MLX5_GET_CTR(out, received_ib_multicast.packets);
vf_stats->broadcast =
MLX5_GET_CTR(out, received_eth_broadcast.packets);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
index 331a6ca..13dfc19 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fs_core.c
@@ -153,6 +153,7 @@ static void del_rule(struct fs_node *node);
static void del_flow_table(struct fs_node *node);
static void del_flow_group(struct fs_node *node);
static void del_fte(struct fs_node *node);
+static void cleanup_root_ns(struct mlx5_flow_root_namespace *root_ns);
static void tree_init_node(struct fs_node *node,
unsigned int refcount,
@@ -389,7 +390,7 @@ static void del_rule(struct fs_node *node)
}
if ((fte->action & MLX5_FLOW_CONTEXT_ACTION_FWD_DEST) &&
--fte->dests_size) {
- modify_mask = BIT(MLX5_SET_FTE_MODIFY_ENABLE_MASK_DESTINATION_LIST),
+ modify_mask = BIT(MLX5_SET_FTE_MODIFY_ENABLE_MASK_DESTINATION_LIST);
err = mlx5_cmd_update_fte(dev, ft,
fg->id,
modify_mask,
@@ -1690,24 +1691,28 @@ static int create_anchor_flow_table(struct mlx5_flow_steering *steering)
static int init_root_ns(struct mlx5_flow_steering *steering)
{
+ int err;
steering->root_ns = create_root_ns(steering, FS_FT_NIC_RX);
if (!steering->root_ns)
- goto cleanup;
+ return -ENOMEM;
- if (init_root_tree(steering, &root_fs, &steering->root_ns->ns.node))
- goto cleanup;
+ err = init_root_tree(steering, &root_fs, &steering->root_ns->ns.node);
+ if (err)
+ goto out_err;
set_prio_attrs(steering->root_ns);
- if (create_anchor_flow_table(steering))
- goto cleanup;
+ err = create_anchor_flow_table(steering);
+ if (err)
+ goto out_err;
return 0;
-cleanup:
- mlx5_cleanup_fs(steering->dev);
- return -ENOMEM;
+out_err:
+ cleanup_root_ns(steering->root_ns);
+ steering->root_ns = NULL;
+ return err;
}
static void clean_tree(struct fs_node *node)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/health.c b/drivers/net/ethernet/mellanox/mlx5/core/health.c
index 8beecd6..264f51b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/health.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/health.c
@@ -339,9 +339,17 @@ void mlx5_start_health_poll(struct mlx5_core_dev *dev)
add_timer(&health->timer);
}
-void mlx5_stop_health_poll(struct mlx5_core_dev *dev)
+void mlx5_stop_health_poll(struct mlx5_core_dev *dev, bool disable_health)
{
struct mlx5_core_health *health = &dev->priv.health;
+ unsigned long flags;
+
+ if (disable_health) {
+ spin_lock_irqsave(&health->wq_lock, flags);
+ set_bit(MLX5_DROP_NEW_HEALTH_WORK, &health->flags);
+ set_bit(MLX5_DROP_NEW_RECOVERY_WORK, &health->flags);
+ spin_unlock_irqrestore(&health->wq_lock, flags);
+ }
del_timer_sync(&health->timer);
}
@@ -361,10 +369,11 @@ void mlx5_drain_health_wq(struct mlx5_core_dev *dev)
void mlx5_drain_health_recovery(struct mlx5_core_dev *dev)
{
struct mlx5_core_health *health = &dev->priv.health;
+ unsigned long flags;
- spin_lock(&health->wq_lock);
+ spin_lock_irqsave(&health->wq_lock, flags);
set_bit(MLX5_DROP_NEW_RECOVERY_WORK, &health->flags);
- spin_unlock(&health->wq_lock);
+ spin_unlock_irqrestore(&health->wq_lock, flags);
cancel_delayed_work_sync(&dev->priv.health.recover_work);
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/main.c b/drivers/net/ethernet/mellanox/mlx5/core/main.c
index 3c183b8..d676088 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/main.c
@@ -787,8 +787,10 @@ static int mlx5_pci_init(struct mlx5_core_dev *dev, struct mlx5_priv *priv)
priv->numa_node = dev_to_node(&dev->pdev->dev);
priv->dbg_root = debugfs_create_dir(dev_name(&pdev->dev), mlx5_debugfs_root);
- if (!priv->dbg_root)
+ if (!priv->dbg_root) {
+ dev_err(&pdev->dev, "Cannot create debugfs dir, aborting\n");
return -ENOMEM;
+ }
err = mlx5_pci_enable_device(dev);
if (err) {
@@ -837,7 +839,7 @@ static void mlx5_pci_close(struct mlx5_core_dev *dev, struct mlx5_priv *priv)
pci_clear_master(dev->pdev);
release_bar(dev->pdev);
mlx5_pci_disable_device(dev);
- debugfs_remove(priv->dbg_root);
+ debugfs_remove_recursive(priv->dbg_root);
}
static int mlx5_init_once(struct mlx5_core_dev *dev, struct mlx5_priv *priv)
@@ -955,7 +957,7 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv,
if (err) {
dev_err(&dev->pdev->dev, "Firmware over %d MS in pre-initializing state, aborting\n",
FW_PRE_INIT_TIMEOUT_MILI);
- goto out;
+ goto out_err;
}
err = mlx5_cmd_init(dev);
@@ -1130,7 +1132,7 @@ static int mlx5_load_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv,
mlx5_cleanup_once(dev);
err_stop_poll:
- mlx5_stop_health_poll(dev);
+ mlx5_stop_health_poll(dev, boot);
if (mlx5_cmd_teardown_hca(dev)) {
dev_err(&dev->pdev->dev, "tear_down_hca failed, skip cleanup\n");
goto out_err;
@@ -1187,7 +1189,7 @@ static int mlx5_unload_one(struct mlx5_core_dev *dev, struct mlx5_priv *priv,
mlx5_disable_msix(dev);
if (cleanup)
mlx5_cleanup_once(dev);
- mlx5_stop_health_poll(dev);
+ mlx5_stop_health_poll(dev, cleanup);
err = mlx5_cmd_teardown_hca(dev);
if (err) {
dev_err(&dev->pdev->dev, "tear_down_hca failed, skip cleanup\n");
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/port.c b/drivers/net/ethernet/mellanox/mlx5/core/port.c
index 34e7184..43d7c83 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/port.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/port.c
@@ -575,7 +575,7 @@ EXPORT_SYMBOL_GPL(mlx5_set_port_prio_tc);
static int mlx5_set_port_qetcr_reg(struct mlx5_core_dev *mdev, u32 *in,
int inlen)
{
- u32 out[MLX5_ST_SZ_DW(qtct_reg)];
+ u32 out[MLX5_ST_SZ_DW(qetc_reg)];
if (!MLX5_CAP_GEN(mdev, ets))
return -ENOTSUPP;
@@ -587,7 +587,7 @@ static int mlx5_set_port_qetcr_reg(struct mlx5_core_dev *mdev, u32 *in,
static int mlx5_query_port_qetcr_reg(struct mlx5_core_dev *mdev, u32 *out,
int outlen)
{
- u32 in[MLX5_ST_SZ_DW(qtct_reg)];
+ u32 in[MLX5_ST_SZ_DW(qetc_reg)];
if (!MLX5_CAP_GEN(mdev, ets))
return -ENOTSUPP;
diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
index d50350c..22a5916e 100644
--- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
+++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.c
@@ -4187,10 +4187,6 @@ static int mlxsw_sp_netdevice_port_upper_event(struct net_device *dev,
if (netif_is_lag_port(dev) && is_vlan_dev(upper_dev) &&
!netif_is_lag_master(vlan_dev_real_dev(upper_dev)))
return -EINVAL;
- if (!info->linking)
- break;
- if (netdev_has_any_upper_dev(upper_dev))
- return -EINVAL;
break;
case NETDEV_CHANGEUPPER:
upper_dev = info->upper_dev;
@@ -4566,6 +4562,8 @@ static int mlxsw_sp_netdevice_vport_event(struct net_device *dev,
return -EINVAL;
if (!info->linking)
break;
+ if (netdev_has_any_upper_dev(upper_dev))
+ return -EINVAL;
/* We can't have multiple VLAN interfaces configured on
* the same port and being members in the same bridge.
*/
diff --git a/drivers/net/ethernet/natsemi/sonic.c b/drivers/net/ethernet/natsemi/sonic.c
index 612c7a4..2382154 100644
--- a/drivers/net/ethernet/natsemi/sonic.c
+++ b/drivers/net/ethernet/natsemi/sonic.c
@@ -71,7 +71,7 @@ static int sonic_open(struct net_device *dev)
for (i = 0; i < SONIC_NUM_RRS; i++) {
dma_addr_t laddr = dma_map_single(lp->device, skb_put(lp->rx_skb[i], SONIC_RBSIZE),
SONIC_RBSIZE, DMA_FROM_DEVICE);
- if (!laddr) {
+ if (dma_mapping_error(lp->device, laddr)) {
while(i > 0) { /* free any that were mapped successfully */
i--;
dma_unmap_single(lp->device, lp->rx_laddr[i], SONIC_RBSIZE, DMA_FROM_DEVICE);
diff --git a/drivers/net/ethernet/neterion/vxge/vxge-config.c b/drivers/net/ethernet/neterion/vxge/vxge-config.c
index 6223930..6f57b0b 100644
--- a/drivers/net/ethernet/neterion/vxge/vxge-config.c
+++ b/drivers/net/ethernet/neterion/vxge/vxge-config.c
@@ -808,7 +808,7 @@ __vxge_hw_vpath_fw_ver_get(struct __vxge_hw_virtualpath *vpath,
struct vxge_hw_device_date *fw_date = &hw_info->fw_date;
struct vxge_hw_device_version *flash_version = &hw_info->flash_version;
struct vxge_hw_device_date *flash_date = &hw_info->flash_date;
- u64 data0, data1 = 0, steer_ctrl = 0;
+ u64 data0 = 0, data1 = 0, steer_ctrl = 0;
enum vxge_hw_status status;
status = vxge_hw_vpath_fw_api(vpath,
diff --git a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
index 4ca82bd..2e8703d 100644
--- a/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
+++ b/drivers/net/ethernet/netronome/nfp/nfp_net_common.c
@@ -854,6 +854,8 @@ static int nfp_net_tx(struct sk_buff *skb, struct net_device *netdev)
netdev_tx_sent_queue(nd_q, txbuf->real_len);
+ skb_tx_timestamp(skb);
+
tx_ring->wr_p += nr_frags + 1;
if (nfp_net_tx_ring_should_stop(tx_ring))
nfp_net_tx_ring_stop(nd_q, tx_ring);
@@ -866,8 +868,6 @@ static int nfp_net_tx(struct sk_buff *skb, struct net_device *netdev)
tx_ring->wr_ptr_add = 0;
}
- skb_tx_timestamp(skb);
-
return NETDEV_TX_OK;
err_unmap:
@@ -990,7 +990,7 @@ static void nfp_net_tx_complete(struct nfp_net_tx_ring *tx_ring)
* @nn: NFP Net device
* @tx_ring: TX ring structure
*
- * Assumes that the device is stopped
+ * Assumes that the device is stopped, must be idempotent.
*/
static void
nfp_net_tx_ring_reset(struct nfp_net *nn, struct nfp_net_tx_ring *tx_ring)
@@ -1144,13 +1144,18 @@ static void nfp_net_rx_give_one(struct nfp_net_rx_ring *rx_ring,
* nfp_net_rx_ring_reset() - Reflect in SW state of freelist after disable
* @rx_ring: RX ring structure
*
- * Warning: Do *not* call if ring buffers were never put on the FW freelist
- * (i.e. device was not enabled)!
+ * Assumes that the device is stopped, must be idempotent.
*/
static void nfp_net_rx_ring_reset(struct nfp_net_rx_ring *rx_ring)
{
unsigned int wr_idx, last_idx;
+ /* wr_p == rd_p means ring was never fed FL bufs. RX rings are always
+ * kept at cnt - 1 FL bufs.
+ */
+ if (rx_ring->wr_p == 0 && rx_ring->rd_p == 0)
+ return;
+
/* Move the empty entry to the end of the list */
wr_idx = rx_ring->wr_p % rx_ring->cnt;
last_idx = rx_ring->cnt - 1;
@@ -1919,6 +1924,8 @@ static void nfp_net_vec_clear_ring_data(struct nfp_net *nn, unsigned int idx)
/**
* nfp_net_clear_config_and_disable() - Clear control BAR and disable NFP
* @nn: NFP Net device to reconfigure
+ *
+ * Warning: must be fully idempotent.
*/
static void nfp_net_clear_config_and_disable(struct nfp_net *nn)
{
diff --git a/drivers/net/ethernet/nuvoton/w90p910_ether.c b/drivers/net/ethernet/nuvoton/w90p910_ether.c
index 712d8bc..a296016 100644
--- a/drivers/net/ethernet/nuvoton/w90p910_ether.c
+++ b/drivers/net/ethernet/nuvoton/w90p910_ether.c
@@ -918,7 +918,7 @@ static const struct net_device_ops w90p910_ether_netdev_ops = {
.ndo_change_mtu = eth_change_mtu,
};
-static void __init get_mac_address(struct net_device *dev)
+static void get_mac_address(struct net_device *dev)
{
struct w90p910_ether *ether = netdev_priv(dev);
struct platform_device *pdev;
diff --git a/drivers/net/ethernet/qlogic/netxen/netxen_nic_init.c b/drivers/net/ethernet/qlogic/netxen/netxen_nic_init.c
index 7b43a3b..5cf5519 100644
--- a/drivers/net/ethernet/qlogic/netxen/netxen_nic_init.c
+++ b/drivers/net/ethernet/qlogic/netxen/netxen_nic_init.c
@@ -1125,7 +1125,8 @@ netxen_validate_firmware(struct netxen_adapter *adapter)
return -EINVAL;
}
val = nx_get_bios_version(adapter);
- netxen_rom_fast_read(adapter, NX_BIOS_VERSION_OFFSET, (int *)&bios);
+ if (netxen_rom_fast_read(adapter, NX_BIOS_VERSION_OFFSET, (int *)&bios))
+ return -EIO;
if ((__force u32)val != bios) {
dev_err(&pdev->dev, "%s: firmware bios is incompatible\n",
fw_name[fw_type]);
diff --git a/drivers/net/ethernet/qlogic/qed/qed_cxt.c b/drivers/net/ethernet/qlogic/qed/qed_cxt.c
index 457e304..f1956c4 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_cxt.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_cxt.c
@@ -54,7 +54,7 @@
#define ILT_CFG_REG(cli, reg) PSWRQ2_REG_ ## cli ## _ ## reg ## _RT_OFFSET
/* ILT entry structure */
-#define ILT_ENTRY_PHY_ADDR_MASK 0x000FFFFFFFFFFFULL
+#define ILT_ENTRY_PHY_ADDR_MASK (~0ULL >> 12)
#define ILT_ENTRY_PHY_ADDR_SHIFT 0
#define ILT_ENTRY_VALID_MASK 0x1ULL
#define ILT_ENTRY_VALID_SHIFT 52
diff --git a/drivers/net/ethernet/qlogic/qed/qed_dcbx.c b/drivers/net/ethernet/qlogic/qed/qed_dcbx.c
index 9d59cb8..7b6824e 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_dcbx.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_dcbx.c
@@ -677,9 +677,9 @@ qed_dcbx_get_local_lldp_params(struct qed_hwfn *p_hwfn,
p_local = &p_hwfn->p_dcbx_info->lldp_local[LLDP_NEAREST_BRIDGE];
memcpy(params->lldp_local.local_chassis_id, p_local->local_chassis_id,
- ARRAY_SIZE(p_local->local_chassis_id));
+ sizeof(p_local->local_chassis_id));
memcpy(params->lldp_local.local_port_id, p_local->local_port_id,
- ARRAY_SIZE(p_local->local_port_id));
+ sizeof(p_local->local_port_id));
}
static void
@@ -692,9 +692,9 @@ qed_dcbx_get_remote_lldp_params(struct qed_hwfn *p_hwfn,
p_remote = &p_hwfn->p_dcbx_info->lldp_remote[LLDP_NEAREST_BRIDGE];
memcpy(params->lldp_remote.peer_chassis_id, p_remote->peer_chassis_id,
- ARRAY_SIZE(p_remote->peer_chassis_id));
+ sizeof(p_remote->peer_chassis_id));
memcpy(params->lldp_remote.peer_port_id, p_remote->peer_port_id,
- ARRAY_SIZE(p_remote->peer_port_id));
+ sizeof(p_remote->peer_port_id));
}
static int
diff --git a/drivers/net/ethernet/qlogic/qed/qed_debug.c b/drivers/net/ethernet/qlogic/qed/qed_debug.c
index 68f19ca..245eb02 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_debug.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_debug.c
@@ -3039,10 +3039,10 @@ static u32 qed_grc_dump_big_ram(struct qed_hwfn *p_hwfn,
s_big_ram_defs[big_ram_id].num_of_blocks[dev_data->chip_id];
ram_size = total_blocks * BIG_RAM_BLOCK_SIZE_DWORDS;
- strncpy(type_name, s_big_ram_defs[big_ram_id].instance_name,
- strlen(s_big_ram_defs[big_ram_id].instance_name));
- strncpy(mem_name, s_big_ram_defs[big_ram_id].instance_name,
- strlen(s_big_ram_defs[big_ram_id].instance_name));
+ strscpy(type_name, s_big_ram_defs[big_ram_id].instance_name,
+ sizeof(type_name));
+ strscpy(mem_name, s_big_ram_defs[big_ram_id].instance_name,
+ sizeof(mem_name));
/* Dump memory header */
offset += qed_grc_dump_mem_hdr(p_hwfn,
diff --git a/drivers/net/ethernet/qlogic/qed/qed_hw.c b/drivers/net/ethernet/qlogic/qed/qed_hw.c
index 6e4fae9..944749c 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_hw.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_hw.c
@@ -34,6 +34,7 @@ struct qed_ptt {
struct list_head list_entry;
unsigned int idx;
struct pxp_ptt_entry pxp;
+ u8 hwfn_id;
};
struct qed_ptt_pool {
@@ -55,6 +56,7 @@ int qed_ptt_pool_alloc(struct qed_hwfn *p_hwfn)
p_pool->ptts[i].idx = i;
p_pool->ptts[i].pxp.offset = QED_BAR_INVALID_OFFSET;
p_pool->ptts[i].pxp.pretend.control = 0;
+ p_pool->ptts[i].hwfn_id = p_hwfn->my_id;
if (i >= RESERVED_PTT_MAX)
list_add(&p_pool->ptts[i].list_entry,
&p_pool->free_list);
@@ -169,6 +171,11 @@ static u32 qed_set_ptt(struct qed_hwfn *p_hwfn,
offset = hw_addr - win_hw_addr;
+ if (p_ptt->hwfn_id != p_hwfn->my_id)
+ DP_NOTICE(p_hwfn,
+ "ptt[%d] of hwfn[%02x] is used by hwfn[%02x]!\n",
+ p_ptt->idx, p_ptt->hwfn_id, p_hwfn->my_id);
+
/* Verify the address is within the window */
if (hw_addr < win_hw_addr ||
offset >= PXP_EXTERNAL_BAR_PF_WINDOW_SINGLE_SIZE) {
diff --git a/drivers/net/ethernet/qlogic/qed/qed_int.c b/drivers/net/ethernet/qlogic/qed/qed_int.c
index 2adedc6..fd19372 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_int.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_int.c
@@ -2135,6 +2135,8 @@ static int qed_int_attentions(struct qed_hwfn *p_hwfn)
*/
do {
index = p_sb_attn->sb_index;
+ /* finish reading index before the loop condition */
+ dma_rmb();
attn_bits = le32_to_cpu(p_sb_attn->atten_bits);
attn_acks = le32_to_cpu(p_sb_attn->atten_ack);
} while (index != p_sb_attn->sb_index);
diff --git a/drivers/net/ethernet/qlogic/qed/qed_l2.c b/drivers/net/ethernet/qlogic/qed/qed_l2.c
index ddd410a..715776e 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_l2.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_l2.c
@@ -313,7 +313,7 @@ qed_sp_update_mcast_bin(struct qed_hwfn *p_hwfn,
p_ramrod->common.update_approx_mcast_flg = 1;
for (i = 0; i < ETH_MULTICAST_MAC_BINS_IN_REGS; i++) {
- u32 *p_bins = (u32 *)p_params->bins;
+ u32 *p_bins = p_params->bins;
p_ramrod->approx_mcast.bins[i] = cpu_to_le32(p_bins[i]);
}
@@ -1182,8 +1182,8 @@ qed_sp_eth_filter_mcast(struct qed_hwfn *p_hwfn,
enum spq_mode comp_mode,
struct qed_spq_comp_cb *p_comp_data)
{
- unsigned long bins[ETH_MULTICAST_MAC_BINS_IN_REGS];
struct vport_update_ramrod_data *p_ramrod = NULL;
+ u32 bins[ETH_MULTICAST_MAC_BINS_IN_REGS];
struct qed_spq_entry *p_ent = NULL;
struct qed_sp_init_data init_data;
u8 abs_vport_id = 0;
@@ -1219,26 +1219,25 @@ qed_sp_eth_filter_mcast(struct qed_hwfn *p_hwfn,
/* explicitly clear out the entire vector */
memset(&p_ramrod->approx_mcast.bins, 0,
sizeof(p_ramrod->approx_mcast.bins));
- memset(bins, 0, sizeof(unsigned long) *
- ETH_MULTICAST_MAC_BINS_IN_REGS);
+ memset(bins, 0, sizeof(bins));
/* filter ADD op is explicit set op and it removes
* any existing filters for the vport
*/
if (p_filter_cmd->opcode == QED_FILTER_ADD) {
for (i = 0; i < p_filter_cmd->num_mc_addrs; i++) {
- u32 bit;
+ u32 bit, nbits;
bit = qed_mcast_bin_from_mac(p_filter_cmd->mac[i]);
- __set_bit(bit, bins);
+ nbits = sizeof(u32) * BITS_PER_BYTE;
+ bins[bit / nbits] |= 1 << (bit % nbits);
}
/* Convert to correct endianity */
for (i = 0; i < ETH_MULTICAST_MAC_BINS_IN_REGS; i++) {
struct vport_update_ramrod_mcast *p_ramrod_bins;
- u32 *p_bins = (u32 *)bins;
p_ramrod_bins = &p_ramrod->approx_mcast;
- p_ramrod_bins->bins[i] = cpu_to_le32(p_bins[i]);
+ p_ramrod_bins->bins[i] = cpu_to_le32(bins[i]);
}
}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_l2.h b/drivers/net/ethernet/qlogic/qed/qed_l2.h
index e495d62..14d0017 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_l2.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_l2.h
@@ -156,7 +156,7 @@ struct qed_sp_vport_update_params {
u8 anti_spoofing_en;
u8 update_accept_any_vlan_flg;
u8 accept_any_vlan;
- unsigned long bins[8];
+ u32 bins[8];
struct qed_rss_params *rss_params;
struct qed_filter_accept_flags accept_flags;
struct qed_sge_tpa_params *sge_tpa_params;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_main.c b/drivers/net/ethernet/qlogic/qed/qed_main.c
index 0b949c6..a769196 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_main.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_main.c
@@ -23,6 +23,7 @@
#include <linux/vmalloc.h>
#include <linux/qed/qed_if.h>
#include <linux/qed/qed_ll2_if.h>
+#include <linux/crash_dump.h>
#include "qed.h"
#include "qed_sriov.h"
@@ -501,8 +502,16 @@ static irqreturn_t qed_single_int(int irq, void *dev_instance)
/* Fastpath interrupts */
for (j = 0; j < 64; j++) {
if ((0x2ULL << j) & status) {
- hwfn->simd_proto_handler[j].func(
- hwfn->simd_proto_handler[j].token);
+ struct qed_simd_fp_handler *p_handler =
+ &hwfn->simd_proto_handler[j];
+
+ if (p_handler->func)
+ p_handler->func(p_handler->token);
+ else
+ DP_NOTICE(hwfn,
+ "Not calling fastpath handler as it is NULL [handler #%d, status 0x%llx]\n",
+ j, status);
+
status &= ~(0x2ULL << j);
rc = IRQ_HANDLED;
}
@@ -701,6 +710,14 @@ static int qed_slowpath_setup_int(struct qed_dev *cdev,
/* We want a minimum of one slowpath and one fastpath vector per hwfn */
cdev->int_params.in.min_msix_cnt = cdev->num_hwfns * 2;
+ if (is_kdump_kernel()) {
+ DP_INFO(cdev,
+ "Kdump kernel: Limit the max number of requested MSI-X vectors to %hd\n",
+ cdev->int_params.in.min_msix_cnt);
+ cdev->int_params.in.num_vectors =
+ cdev->int_params.in.min_msix_cnt;
+ }
+
rc = qed_set_int_mode(cdev, false);
if (rc) {
DP_ERR(cdev, "qed_slowpath_setup_int ERR\n");
@@ -1378,9 +1395,9 @@ static int qed_drain(struct qed_dev *cdev)
return -EBUSY;
}
rc = qed_mcp_drain(hwfn, ptt);
+ qed_ptt_release(hwfn, ptt);
if (rc)
return rc;
- qed_ptt_release(hwfn, ptt);
}
return 0;
diff --git a/drivers/net/ethernet/qlogic/qed/qed_mcp.c b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
index 8b7d2f9..e175fcd 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_mcp.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
@@ -97,18 +97,57 @@ int qed_mcp_free(struct qed_hwfn *p_hwfn)
return 0;
}
+/* Maximum of 1 sec to wait for the SHMEM ready indication */
+#define QED_MCP_SHMEM_RDY_MAX_RETRIES 20
+#define QED_MCP_SHMEM_RDY_ITER_MS 50
+
static int qed_load_mcp_offsets(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
{
struct qed_mcp_info *p_info = p_hwfn->mcp_info;
+ u8 cnt = QED_MCP_SHMEM_RDY_MAX_RETRIES;
+ u8 msec = QED_MCP_SHMEM_RDY_ITER_MS;
u32 drv_mb_offsize, mfw_mb_offsize;
u32 mcp_pf_id = MCP_PF_ID(p_hwfn);
p_info->public_base = qed_rd(p_hwfn, p_ptt, MISC_REG_SHARED_MEM_ADDR);
- if (!p_info->public_base)
- return 0;
+ if (!p_info->public_base) {
+ DP_NOTICE(p_hwfn,
+ "The address of the MCP scratch-pad is not configured\n");
+ return -EINVAL;
+ }
p_info->public_base |= GRCBASE_MCP;
+ /* Get the MFW MB address and number of supported messages */
+ mfw_mb_offsize = qed_rd(p_hwfn, p_ptt,
+ SECTION_OFFSIZE_ADDR(p_info->public_base,
+ PUBLIC_MFW_MB));
+ p_info->mfw_mb_addr = SECTION_ADDR(mfw_mb_offsize, mcp_pf_id);
+ p_info->mfw_mb_length = (u16)qed_rd(p_hwfn, p_ptt,
+ p_info->mfw_mb_addr +
+ offsetof(struct public_mfw_mb,
+ sup_msgs));
+
+ /* The driver can notify that there was an MCP reset, and might read the
+ * SHMEM values before the MFW has completed initializing them.
+ * To avoid this, the "sup_msgs" field in the MFW mailbox is used as a
+ * data ready indication.
+ */
+ while (!p_info->mfw_mb_length && --cnt) {
+ msleep(msec);
+ p_info->mfw_mb_length =
+ (u16)qed_rd(p_hwfn, p_ptt,
+ p_info->mfw_mb_addr +
+ offsetof(struct public_mfw_mb, sup_msgs));
+ }
+
+ if (!cnt) {
+ DP_NOTICE(p_hwfn,
+ "Failed to get the SHMEM ready notification after %d msec\n",
+ QED_MCP_SHMEM_RDY_MAX_RETRIES * msec);
+ return -EBUSY;
+ }
+
/* Calculate the driver and MFW mailbox address */
drv_mb_offsize = qed_rd(p_hwfn, p_ptt,
SECTION_OFFSIZE_ADDR(p_info->public_base,
@@ -118,13 +157,6 @@ static int qed_load_mcp_offsets(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
"drv_mb_offsiz = 0x%x, drv_mb_addr = 0x%x mcp_pf_id = 0x%x\n",
drv_mb_offsize, p_info->drv_mb_addr, mcp_pf_id);
- /* Set the MFW MB address */
- mfw_mb_offsize = qed_rd(p_hwfn, p_ptt,
- SECTION_OFFSIZE_ADDR(p_info->public_base,
- PUBLIC_MFW_MB));
- p_info->mfw_mb_addr = SECTION_ADDR(mfw_mb_offsize, mcp_pf_id);
- p_info->mfw_mb_length = (u16)qed_rd(p_hwfn, p_ptt, p_info->mfw_mb_addr);
-
/* Get the current driver mailbox sequence before sending
* the first command
*/
@@ -613,6 +645,7 @@ static void qed_mcp_handle_link_change(struct qed_hwfn *p_hwfn,
break;
default:
p_link->speed = 0;
+ p_link->link_up = 0;
}
if (p_link->link_up && p_link->speed)
@@ -1197,31 +1230,61 @@ qed_mcp_send_drv_version(struct qed_hwfn *p_hwfn,
return rc;
}
+/* A maximal 100 msec waiting time for the MCP to halt */
+#define QED_MCP_HALT_SLEEP_MS 10
+#define QED_MCP_HALT_MAX_RETRIES 10
+
int qed_mcp_halt(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
{
- u32 resp = 0, param = 0;
+ u32 resp = 0, param = 0, cpu_state, cnt = 0;
int rc;
rc = qed_mcp_cmd(p_hwfn, p_ptt, DRV_MSG_CODE_MCP_HALT, 0, &resp,
¶m);
- if (rc)
+ if (rc) {
DP_ERR(p_hwfn, "MCP response failure, aborting\n");
+ return rc;
+ }
- return rc;
+ do {
+ msleep(QED_MCP_HALT_SLEEP_MS);
+ cpu_state = qed_rd(p_hwfn, p_ptt, MCP_REG_CPU_STATE);
+ if (cpu_state & MCP_REG_CPU_STATE_SOFT_HALTED)
+ break;
+ } while (++cnt < QED_MCP_HALT_MAX_RETRIES);
+
+ if (cnt == QED_MCP_HALT_MAX_RETRIES) {
+ DP_NOTICE(p_hwfn,
+ "Failed to halt the MCP [CPU_MODE = 0x%08x, CPU_STATE = 0x%08x]\n",
+ qed_rd(p_hwfn, p_ptt, MCP_REG_CPU_MODE), cpu_state);
+ return -EBUSY;
+ }
+
+ return 0;
}
+#define QED_MCP_RESUME_SLEEP_MS 10
+
int qed_mcp_resume(struct qed_hwfn *p_hwfn, struct qed_ptt *p_ptt)
{
- u32 value, cpu_mode;
+ u32 cpu_mode, cpu_state;
qed_wr(p_hwfn, p_ptt, MCP_REG_CPU_STATE, 0xffffffff);
- value = qed_rd(p_hwfn, p_ptt, MCP_REG_CPU_MODE);
- value &= ~MCP_REG_CPU_MODE_SOFT_HALT;
- qed_wr(p_hwfn, p_ptt, MCP_REG_CPU_MODE, value);
cpu_mode = qed_rd(p_hwfn, p_ptt, MCP_REG_CPU_MODE);
+ cpu_mode &= ~MCP_REG_CPU_MODE_SOFT_HALT;
+ qed_wr(p_hwfn, p_ptt, MCP_REG_CPU_MODE, cpu_mode);
+ msleep(QED_MCP_RESUME_SLEEP_MS);
+ cpu_state = qed_rd(p_hwfn, p_ptt, MCP_REG_CPU_STATE);
- return (cpu_mode & MCP_REG_CPU_MODE_SOFT_HALT) ? -EAGAIN : 0;
+ if (cpu_state & MCP_REG_CPU_STATE_SOFT_HALTED) {
+ DP_NOTICE(p_hwfn,
+ "Failed to resume the MCP [CPU_MODE = 0x%08x, CPU_STATE = 0x%08x]\n",
+ cpu_mode, cpu_state);
+ return -EBUSY;
+ }
+
+ return 0;
}
int qed_mcp_set_led(struct qed_hwfn *p_hwfn,
diff --git a/drivers/net/ethernet/qlogic/qed/qed_reg_addr.h b/drivers/net/ethernet/qlogic/qed/qed_reg_addr.h
index b414a05..56be1d6 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_reg_addr.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_reg_addr.h
@@ -510,6 +510,7 @@
0
#define MCP_REG_CPU_STATE \
0xe05004UL
+#define MCP_REG_CPU_STATE_SOFT_HALTED (0x1UL << 10)
#define MCP_REG_CPU_EVENT_MASK \
0xe05008UL
#define PGLUE_B_REG_PF_BAR0_SIZE \
diff --git a/drivers/net/ethernet/qlogic/qed/qed_roce.c b/drivers/net/ethernet/qlogic/qed/qed_roce.c
index d9dcb0d..07783d1 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_roce.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_roce.c
@@ -1059,23 +1059,16 @@ static void qed_rdma_copy_gids(struct qed_rdma_qp *qp, __le32 *src_gid,
static enum roce_flavor qed_roce_mode_to_flavor(enum roce_mode roce_mode)
{
- enum roce_flavor flavor;
-
switch (roce_mode) {
case ROCE_V1:
- flavor = PLAIN_ROCE;
- break;
+ return PLAIN_ROCE;
case ROCE_V2_IPV4:
- flavor = RROCE_IPV4;
- break;
+ return RROCE_IPV4;
case ROCE_V2_IPV6:
- flavor = ROCE_V2_IPV6;
- break;
+ return RROCE_IPV6;
default:
- flavor = MAX_ROCE_MODE;
- break;
+ return MAX_ROCE_FLAVOR;
}
- return flavor;
}
static int qed_roce_alloc_cid(struct qed_hwfn *p_hwfn, u16 *cid)
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sp.h b/drivers/net/ethernet/qlogic/qed/qed_sp.h
index b2c08e4..bae7b7f 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sp.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_sp.h
@@ -132,6 +132,9 @@ struct qed_spq_entry {
enum spq_mode comp_mode;
struct qed_spq_comp_cb comp_cb;
struct qed_spq_comp_done comp_done; /* SPQ_MODE_EBLOCK */
+
+ /* Posted entry for unlimited list entry in EBLOCK mode */
+ struct qed_spq_entry *post_ent;
};
struct qed_eq {
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c b/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
index 2888eb0..ac69ff3 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_sp_commands.c
@@ -56,7 +56,7 @@ int qed_sp_init_request(struct qed_hwfn *p_hwfn,
case QED_SPQ_MODE_BLOCK:
if (!p_data->p_comp_data)
- return -EINVAL;
+ goto err;
p_ent->comp_cb.cookie = p_data->p_comp_data->cookie;
break;
@@ -71,7 +71,7 @@ int qed_sp_init_request(struct qed_hwfn *p_hwfn,
default:
DP_NOTICE(p_hwfn, "Unknown SPQE completion mode %d\n",
p_ent->comp_mode);
- return -EINVAL;
+ goto err;
}
DP_VERBOSE(p_hwfn, QED_MSG_SPQ,
@@ -85,6 +85,18 @@ int qed_sp_init_request(struct qed_hwfn *p_hwfn,
memset(&p_ent->ramrod, 0, sizeof(p_ent->ramrod));
return 0;
+
+err:
+ /* qed_spq_get_entry() can either get an entry from the free_pool,
+ * or, if no entries are left, allocate a new entry and add it to
+ * the unlimited_pending list.
+ */
+ if (p_ent->queue == &p_hwfn->p_spq->unlimited_pending)
+ kfree(p_ent);
+ else
+ qed_spq_return_entry(p_hwfn, p_ent);
+
+ return -EINVAL;
}
static enum tunnel_clss qed_tunn_get_clss_type(u8 type)
diff --git a/drivers/net/ethernet/qlogic/qed/qed_spq.c b/drivers/net/ethernet/qlogic/qed/qed_spq.c
index 9fbaf94..80c8c7f 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_spq.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_spq.c
@@ -595,6 +595,8 @@ static int qed_spq_add_entry(struct qed_hwfn *p_hwfn,
/* EBLOCK responsible to free the allocated p_ent */
if (p_ent->comp_mode != QED_SPQ_MODE_EBLOCK)
kfree(p_ent);
+ else
+ p_ent->post_ent = p_en2;
p_ent = p_en2;
}
@@ -678,6 +680,25 @@ static int qed_spq_pend_post(struct qed_hwfn *p_hwfn)
SPQ_HIGH_PRI_RESERVE_DEFAULT);
}
+/* Avoid overriding of SPQ entries when getting out-of-order completions, by
+ * marking the completions in a bitmap and increasing the chain consumer only
+ * for the first successive completed entries.
+ */
+static void qed_spq_comp_bmap_update(struct qed_hwfn *p_hwfn, __le16 echo)
+{
+ u16 pos = le16_to_cpu(echo) % SPQ_RING_SIZE;
+ struct qed_spq *p_spq = p_hwfn->p_spq;
+
+ __set_bit(pos, p_spq->p_comp_bitmap);
+ while (test_bit(p_spq->comp_bitmap_idx,
+ p_spq->p_comp_bitmap)) {
+ __clear_bit(p_spq->comp_bitmap_idx,
+ p_spq->p_comp_bitmap);
+ p_spq->comp_bitmap_idx++;
+ qed_chain_return_produced(&p_spq->chain);
+ }
+}
+
int qed_spq_post(struct qed_hwfn *p_hwfn,
struct qed_spq_entry *p_ent, u8 *fw_return_code)
{
@@ -728,11 +749,12 @@ int qed_spq_post(struct qed_hwfn *p_hwfn,
rc = qed_spq_block(p_hwfn, p_ent, fw_return_code);
if (p_ent->queue == &p_spq->unlimited_pending) {
- /* This is an allocated p_ent which does not need to
- * return to pool.
- */
+ struct qed_spq_entry *p_post_ent = p_ent->post_ent;
+
kfree(p_ent);
- return rc;
+
+ /* Return the entry which was actually posted */
+ p_ent = p_post_ent;
}
if (rc)
@@ -746,7 +768,7 @@ int qed_spq_post(struct qed_hwfn *p_hwfn,
spq_post_fail2:
spin_lock_bh(&p_spq->lock);
list_del(&p_ent->list);
- qed_chain_return_produced(&p_spq->chain);
+ qed_spq_comp_bmap_update(p_hwfn, p_ent->elem.hdr.echo);
spq_post_fail:
/* return to the free pool */
@@ -778,25 +800,8 @@ int qed_spq_completion(struct qed_hwfn *p_hwfn,
spin_lock_bh(&p_spq->lock);
list_for_each_entry_safe(p_ent, tmp, &p_spq->completion_pending, list) {
if (p_ent->elem.hdr.echo == echo) {
- u16 pos = le16_to_cpu(echo) % SPQ_RING_SIZE;
-
list_del(&p_ent->list);
-
- /* Avoid overriding of SPQ entries when getting
- * out-of-order completions, by marking the completions
- * in a bitmap and increasing the chain consumer only
- * for the first successive completed entries.
- */
- __set_bit(pos, p_spq->p_comp_bitmap);
-
- while (test_bit(p_spq->comp_bitmap_idx,
- p_spq->p_comp_bitmap)) {
- __clear_bit(p_spq->comp_bitmap_idx,
- p_spq->p_comp_bitmap);
- p_spq->comp_bitmap_idx++;
- qed_chain_return_produced(&p_spq->chain);
- }
-
+ qed_spq_comp_bmap_update(p_hwfn, echo);
p_spq->comp_count++;
found = p_ent;
break;
@@ -835,11 +840,9 @@ int qed_spq_completion(struct qed_hwfn *p_hwfn,
QED_MSG_SPQ,
"Got a completion without a callback function\n");
- if ((found->comp_mode != QED_SPQ_MODE_EBLOCK) ||
- (found->queue == &p_spq->unlimited_pending))
+ if (found->comp_mode != QED_SPQ_MODE_EBLOCK)
/* EBLOCK is responsible for returning its own entry into the
- * free list, unless it originally added the entry into the
- * unlimited pending list.
+ * free list.
*/
qed_spq_return_entry(p_hwfn, found);
diff --git a/drivers/net/ethernet/qlogic/qed/qed_sriov.c b/drivers/net/ethernet/qlogic/qed/qed_sriov.c
index 48bc5c1..6379bfe 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_sriov.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_sriov.c
@@ -2157,7 +2157,7 @@ qed_iov_vp_update_mcast_bin_param(struct qed_hwfn *p_hwfn,
p_data->update_approx_mcast_flg = 1;
memcpy(p_data->bins, p_mcast_tlv->bins,
- sizeof(unsigned long) * ETH_MULTICAST_MAC_BINS_IN_REGS);
+ sizeof(u32) * ETH_MULTICAST_MAC_BINS_IN_REGS);
*tlvs_mask |= 1 << QED_IOV_VP_UPDATE_MCAST;
}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_vf.c b/drivers/net/ethernet/qlogic/qed/qed_vf.c
index 0645124..9cc02b9 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_vf.c
+++ b/drivers/net/ethernet/qlogic/qed/qed_vf.c
@@ -295,7 +295,6 @@ static int qed_vf_pf_acquire(struct qed_hwfn *p_hwfn)
}
if (!p_iov->b_pre_fp_hsi &&
- ETH_HSI_VER_MINOR &&
(resp->pfdev_info.minor_fp_hsi < ETH_HSI_VER_MINOR)) {
DP_INFO(p_hwfn,
"PF is using older fastpath HSI; %02x.%02x is configured\n",
@@ -786,7 +785,7 @@ int qed_vf_pf_vport_update(struct qed_hwfn *p_hwfn,
resp_size += sizeof(struct pfvf_def_resp_tlv);
memcpy(p_mcast_tlv->bins, p_params->bins,
- sizeof(unsigned long) * ETH_MULTICAST_MAC_BINS_IN_REGS);
+ sizeof(u32) * ETH_MULTICAST_MAC_BINS_IN_REGS);
}
update_rx = p_params->accept_flags.update_rx_mode_config;
@@ -972,7 +971,7 @@ void qed_vf_pf_filter_mcast(struct qed_hwfn *p_hwfn,
u32 bit;
bit = qed_mcast_bin_from_mac(p_filter_cmd->mac[i]);
- __set_bit(bit, sp_params.bins);
+ sp_params.bins[bit / 32] |= 1 << (bit % 32);
}
}
diff --git a/drivers/net/ethernet/qlogic/qed/qed_vf.h b/drivers/net/ethernet/qlogic/qed/qed_vf.h
index 35db7a28..b962ef8 100644
--- a/drivers/net/ethernet/qlogic/qed/qed_vf.h
+++ b/drivers/net/ethernet/qlogic/qed/qed_vf.h
@@ -336,7 +336,12 @@ struct vfpf_vport_update_mcast_bin_tlv {
struct channel_tlv tl;
u8 padding[4];
- u64 bins[8];
+ /* There are only 256 approx bins, and in HSI they're divided into
+ * 32-bit values. As old VFs used to set-bit to the values on its side,
+ * the upper half of the array is never expected to contain any data.
+ */
+ u64 bins[4];
+ u64 obsolete_bins[4];
};
struct vfpf_vport_update_accept_param_tlv {
diff --git a/drivers/net/ethernet/qlogic/qla3xxx.c b/drivers/net/ethernet/qlogic/qla3xxx.c
index b09a6b8..355c5fb 100644
--- a/drivers/net/ethernet/qlogic/qla3xxx.c
+++ b/drivers/net/ethernet/qlogic/qla3xxx.c
@@ -380,8 +380,6 @@ static void fm93c56a_select(struct ql3_adapter *qdev)
qdev->eeprom_cmd_data = AUBURN_EEPROM_CS_1;
ql_write_nvram_reg(qdev, spir, ISP_NVRAM_MASK | qdev->eeprom_cmd_data);
- ql_write_nvram_reg(qdev, spir,
- ((ISP_NVRAM_MASK << 16) | qdev->eeprom_cmd_data));
}
/*
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h b/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h
index 49bad00..f1242ab 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic.h
@@ -1800,7 +1800,8 @@ struct qlcnic_hardware_ops {
int (*config_loopback) (struct qlcnic_adapter *, u8);
int (*clear_loopback) (struct qlcnic_adapter *, u8);
int (*config_promisc_mode) (struct qlcnic_adapter *, u32);
- void (*change_l2_filter) (struct qlcnic_adapter *, u64 *, u16);
+ void (*change_l2_filter)(struct qlcnic_adapter *adapter, u64 *addr,
+ u16 vlan, struct qlcnic_host_tx_ring *tx_ring);
int (*get_board_info) (struct qlcnic_adapter *);
void (*set_mac_filter_count) (struct qlcnic_adapter *);
void (*free_mac_list) (struct qlcnic_adapter *);
@@ -1824,22 +1825,44 @@ struct qlcnic_hardware_ops {
u32 (*get_cap_size)(void *, int);
void (*set_sys_info)(void *, int, u32);
void (*store_cap_mask)(void *, u32);
+ bool (*encap_rx_offload) (struct qlcnic_adapter *adapter);
+ bool (*encap_tx_offload) (struct qlcnic_adapter *adapter);
};
extern struct qlcnic_nic_template qlcnic_vf_ops;
-static inline bool qlcnic_encap_tx_offload(struct qlcnic_adapter *adapter)
+static inline bool qlcnic_83xx_encap_tx_offload(struct qlcnic_adapter *adapter)
{
return adapter->ahw->extra_capability[0] &
QLCNIC_83XX_FW_CAPAB_ENCAP_TX_OFFLOAD;
}
-static inline bool qlcnic_encap_rx_offload(struct qlcnic_adapter *adapter)
+static inline bool qlcnic_83xx_encap_rx_offload(struct qlcnic_adapter *adapter)
{
return adapter->ahw->extra_capability[0] &
QLCNIC_83XX_FW_CAPAB_ENCAP_RX_OFFLOAD;
}
+static inline bool qlcnic_82xx_encap_tx_offload(struct qlcnic_adapter *adapter)
+{
+ return false;
+}
+
+static inline bool qlcnic_82xx_encap_rx_offload(struct qlcnic_adapter *adapter)
+{
+ return false;
+}
+
+static inline bool qlcnic_encap_rx_offload(struct qlcnic_adapter *adapter)
+{
+ return adapter->ahw->hw_ops->encap_rx_offload(adapter);
+}
+
+static inline bool qlcnic_encap_tx_offload(struct qlcnic_adapter *adapter)
+{
+ return adapter->ahw->hw_ops->encap_tx_offload(adapter);
+}
+
static inline int qlcnic_start_firmware(struct qlcnic_adapter *adapter)
{
return adapter->nic_ops->start_firmware(adapter);
@@ -2042,9 +2065,10 @@ static inline int qlcnic_nic_set_promisc(struct qlcnic_adapter *adapter,
}
static inline void qlcnic_change_filter(struct qlcnic_adapter *adapter,
- u64 *addr, u16 id)
+ u64 *addr, u16 vlan,
+ struct qlcnic_host_tx_ring *tx_ring)
{
- adapter->ahw->hw_ops->change_l2_filter(adapter, addr, id);
+ adapter->ahw->hw_ops->change_l2_filter(adapter, addr, vlan, tx_ring);
}
static inline int qlcnic_get_board_info(struct qlcnic_adapter *adapter)
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c
index c3c28f0..35c5ac4 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.c
@@ -242,6 +242,8 @@ static struct qlcnic_hardware_ops qlcnic_83xx_hw_ops = {
.get_cap_size = qlcnic_83xx_get_cap_size,
.set_sys_info = qlcnic_83xx_set_sys_info,
.store_cap_mask = qlcnic_83xx_store_cap_mask,
+ .encap_rx_offload = qlcnic_83xx_encap_rx_offload,
+ .encap_tx_offload = qlcnic_83xx_encap_tx_offload,
};
static struct qlcnic_nic_template qlcnic_83xx_ops = {
@@ -2132,7 +2134,8 @@ int qlcnic_83xx_sre_macaddr_change(struct qlcnic_adapter *adapter, u8 *addr,
}
void qlcnic_83xx_change_l2_filter(struct qlcnic_adapter *adapter, u64 *addr,
- u16 vlan_id)
+ u16 vlan_id,
+ struct qlcnic_host_tx_ring *tx_ring)
{
u8 mac[ETH_ALEN];
memcpy(&mac, addr, ETH_ALEN);
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h
index 331ae2c..c8e012b 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_83xx_hw.h
@@ -550,7 +550,8 @@ int qlcnic_83xx_wrt_reg_indirect(struct qlcnic_adapter *, ulong, u32);
int qlcnic_83xx_nic_set_promisc(struct qlcnic_adapter *, u32);
int qlcnic_83xx_config_hw_lro(struct qlcnic_adapter *, int);
int qlcnic_83xx_config_rss(struct qlcnic_adapter *, int);
-void qlcnic_83xx_change_l2_filter(struct qlcnic_adapter *, u64 *, u16);
+void qlcnic_83xx_change_l2_filter(struct qlcnic_adapter *adapter, u64 *addr,
+ u16 vlan, struct qlcnic_host_tx_ring *ring);
int qlcnic_83xx_get_pci_info(struct qlcnic_adapter *, struct qlcnic_pci_info *);
int qlcnic_83xx_set_nic_info(struct qlcnic_adapter *, struct qlcnic_info *);
void qlcnic_83xx_initialize_nic(struct qlcnic_adapter *, int);
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h
index 4bb33af..56a3bd9 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_hw.h
@@ -173,7 +173,8 @@ int qlcnic_82xx_napi_add(struct qlcnic_adapter *adapter,
struct net_device *netdev);
void qlcnic_82xx_get_beacon_state(struct qlcnic_adapter *);
void qlcnic_82xx_change_filter(struct qlcnic_adapter *adapter,
- u64 *uaddr, u16 vlan_id);
+ u64 *uaddr, u16 vlan_id,
+ struct qlcnic_host_tx_ring *tx_ring);
int qlcnic_82xx_config_intr_coalesce(struct qlcnic_adapter *,
struct ethtool_coalesce *);
int qlcnic_82xx_set_rx_coalesce(struct qlcnic_adapter *);
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c
index fedd736..e361294 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c
@@ -268,13 +268,12 @@ static void qlcnic_add_lb_filter(struct qlcnic_adapter *adapter,
}
void qlcnic_82xx_change_filter(struct qlcnic_adapter *adapter, u64 *uaddr,
- u16 vlan_id)
+ u16 vlan_id, struct qlcnic_host_tx_ring *tx_ring)
{
struct cmd_desc_type0 *hwdesc;
struct qlcnic_nic_req *req;
struct qlcnic_mac_req *mac_req;
struct qlcnic_vlan_req *vlan_req;
- struct qlcnic_host_tx_ring *tx_ring = adapter->tx_ring;
u32 producer;
u64 word;
@@ -301,7 +300,8 @@ void qlcnic_82xx_change_filter(struct qlcnic_adapter *adapter, u64 *uaddr,
static void qlcnic_send_filter(struct qlcnic_adapter *adapter,
struct cmd_desc_type0 *first_desc,
- struct sk_buff *skb)
+ struct sk_buff *skb,
+ struct qlcnic_host_tx_ring *tx_ring)
{
struct vlan_ethhdr *vh = (struct vlan_ethhdr *)(skb->data);
struct ethhdr *phdr = (struct ethhdr *)(skb->data);
@@ -335,7 +335,7 @@ static void qlcnic_send_filter(struct qlcnic_adapter *adapter,
tmp_fil->vlan_id == vlan_id) {
if (jiffies > (QLCNIC_READD_AGE * HZ + tmp_fil->ftime))
qlcnic_change_filter(adapter, &src_addr,
- vlan_id);
+ vlan_id, tx_ring);
tmp_fil->ftime = jiffies;
return;
}
@@ -350,7 +350,7 @@ static void qlcnic_send_filter(struct qlcnic_adapter *adapter,
if (!fil)
return;
- qlcnic_change_filter(adapter, &src_addr, vlan_id);
+ qlcnic_change_filter(adapter, &src_addr, vlan_id, tx_ring);
fil->ftime = jiffies;
fil->vlan_id = vlan_id;
memcpy(fil->faddr, &src_addr, ETH_ALEN);
@@ -766,7 +766,7 @@ netdev_tx_t qlcnic_xmit_frame(struct sk_buff *skb, struct net_device *netdev)
}
if (adapter->drv_mac_learn)
- qlcnic_send_filter(adapter, first_desc, skb);
+ qlcnic_send_filter(adapter, first_desc, skb, tx_ring);
tx_ring->tx_stats.tx_bytes += skb->len;
tx_ring->tx_stats.xmit_called++;
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c
index 3ae3968..ebf5ead 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_main.c
@@ -632,6 +632,8 @@ static struct qlcnic_hardware_ops qlcnic_hw_ops = {
.get_cap_size = qlcnic_82xx_get_cap_size,
.set_sys_info = qlcnic_82xx_set_sys_info,
.store_cap_mask = qlcnic_82xx_store_cap_mask,
+ .encap_rx_offload = qlcnic_82xx_encap_rx_offload,
+ .encap_tx_offload = qlcnic_82xx_encap_tx_offload,
};
static int qlcnic_check_multi_tx_capability(struct qlcnic_adapter *adapter)
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sriov_common.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sriov_common.c
index 2f656f3..c58180f 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sriov_common.c
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sriov_common.c
@@ -77,6 +77,8 @@ static struct qlcnic_hardware_ops qlcnic_sriov_vf_hw_ops = {
.free_mac_list = qlcnic_sriov_vf_free_mac_list,
.enable_sds_intr = qlcnic_83xx_enable_sds_intr,
.disable_sds_intr = qlcnic_83xx_disable_sds_intr,
+ .encap_rx_offload = qlcnic_83xx_encap_rx_offload,
+ .encap_tx_offload = qlcnic_83xx_encap_tx_offload,
};
static struct qlcnic_nic_template qlcnic_sriov_vf_ops = {
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sysfs.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sysfs.c
index ccbb045..b53a18e 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sysfs.c
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_sysfs.c
@@ -1128,6 +1128,8 @@ static ssize_t qlcnic_83xx_sysfs_flash_write_handler(struct file *filp,
struct qlcnic_adapter *adapter = dev_get_drvdata(dev);
ret = kstrtoul(buf, 16, &data);
+ if (ret)
+ return ret;
switch (data) {
case QLC_83XX_FLASH_SECTOR_ERASE_CMD:
diff --git a/drivers/net/ethernet/qlogic/qlge/qlge_main.c b/drivers/net/ethernet/qlogic/qlge/qlge_main.c
index fd4a8e4..6a50754 100644
--- a/drivers/net/ethernet/qlogic/qlge/qlge_main.c
+++ b/drivers/net/ethernet/qlogic/qlge/qlge_main.c
@@ -2387,26 +2387,20 @@ static int qlge_update_hw_vlan_features(struct net_device *ndev,
return status;
}
-static netdev_features_t qlge_fix_features(struct net_device *ndev,
- netdev_features_t features)
-{
- int err;
-
- /* Update the behavior of vlan accel in the adapter */
- err = qlge_update_hw_vlan_features(ndev, features);
- if (err)
- return err;
-
- return features;
-}
-
static int qlge_set_features(struct net_device *ndev,
netdev_features_t features)
{
netdev_features_t changed = ndev->features ^ features;
+ int err;
- if (changed & NETIF_F_HW_VLAN_CTAG_RX)
+ if (changed & NETIF_F_HW_VLAN_CTAG_RX) {
+ /* Update the behavior of vlan accel in the adapter */
+ err = qlge_update_hw_vlan_features(ndev, features);
+ if (err)
+ return err;
+
qlge_vlan_mode(ndev, features);
+ }
return 0;
}
@@ -4719,7 +4713,6 @@ static const struct net_device_ops qlge_netdev_ops = {
.ndo_set_mac_address = qlge_set_mac_address,
.ndo_validate_addr = eth_validate_addr,
.ndo_tx_timeout = qlge_tx_timeout,
- .ndo_fix_features = qlge_fix_features,
.ndo_set_features = qlge_set_features,
.ndo_vlan_rx_add_vid = qlge_vlan_rx_add_vid,
.ndo_vlan_rx_kill_vid = qlge_vlan_rx_kill_vid,
diff --git a/drivers/net/ethernet/qualcomm/emac/emac-mac.c b/drivers/net/ethernet/qualcomm/emac/emac-mac.c
index f683bfb..9d223ff 100644
--- a/drivers/net/ethernet/qualcomm/emac/emac-mac.c
+++ b/drivers/net/ethernet/qualcomm/emac/emac-mac.c
@@ -1250,9 +1250,9 @@ void emac_mac_tx_process(struct emac_adapter *adpt, struct emac_tx_queue *tx_q)
while (tx_q->tpd.consume_idx != hw_consume_idx) {
tpbuf = GET_TPD_BUFFER(tx_q, tx_q->tpd.consume_idx);
if (tpbuf->dma_addr) {
- dma_unmap_single(adpt->netdev->dev.parent,
- tpbuf->dma_addr, tpbuf->length,
- DMA_TO_DEVICE);
+ dma_unmap_page(adpt->netdev->dev.parent,
+ tpbuf->dma_addr, tpbuf->length,
+ DMA_TO_DEVICE);
tpbuf->dma_addr = 0;
}
@@ -1409,9 +1409,11 @@ static void emac_tx_fill_tpd(struct emac_adapter *adpt,
tpbuf = GET_TPD_BUFFER(tx_q, tx_q->tpd.produce_idx);
tpbuf->length = mapped_len;
- tpbuf->dma_addr = dma_map_single(adpt->netdev->dev.parent,
- skb->data, tpbuf->length,
- DMA_TO_DEVICE);
+ tpbuf->dma_addr = dma_map_page(adpt->netdev->dev.parent,
+ virt_to_page(skb->data),
+ offset_in_page(skb->data),
+ tpbuf->length,
+ DMA_TO_DEVICE);
ret = dma_mapping_error(adpt->netdev->dev.parent,
tpbuf->dma_addr);
if (ret)
@@ -1427,9 +1429,12 @@ static void emac_tx_fill_tpd(struct emac_adapter *adpt,
if (mapped_len < len) {
tpbuf = GET_TPD_BUFFER(tx_q, tx_q->tpd.produce_idx);
tpbuf->length = len - mapped_len;
- tpbuf->dma_addr = dma_map_single(adpt->netdev->dev.parent,
- skb->data + mapped_len,
- tpbuf->length, DMA_TO_DEVICE);
+ tpbuf->dma_addr = dma_map_page(adpt->netdev->dev.parent,
+ virt_to_page(skb->data +
+ mapped_len),
+ offset_in_page(skb->data +
+ mapped_len),
+ tpbuf->length, DMA_TO_DEVICE);
ret = dma_mapping_error(adpt->netdev->dev.parent,
tpbuf->dma_addr);
if (ret)
diff --git a/drivers/net/ethernet/qualcomm/qca_spi.c b/drivers/net/ethernet/qualcomm/qca_spi.c
index 8bbb55f3..21f5465 100644
--- a/drivers/net/ethernet/qualcomm/qca_spi.c
+++ b/drivers/net/ethernet/qualcomm/qca_spi.c
@@ -635,7 +635,7 @@ qcaspi_netdev_open(struct net_device *dev)
return ret;
}
- netif_start_queue(qca->net_dev);
+ /* SPI thread takes care of TX queue */
return 0;
}
@@ -739,6 +739,9 @@ qcaspi_netdev_tx_timeout(struct net_device *dev)
qca->net_dev->stats.tx_errors++;
/* Trigger tx queue flush and QCA7000 reset */
qca->sync = QCASPI_SYNC_UNKNOWN;
+
+ if (qca->spi_thread)
+ wake_up_process(qca->spi_thread);
}
static int
@@ -865,22 +868,22 @@ qca_spi_probe(struct spi_device *spi)
if ((qcaspi_clkspeed < QCASPI_CLK_SPEED_MIN) ||
(qcaspi_clkspeed > QCASPI_CLK_SPEED_MAX)) {
- dev_info(&spi->dev, "Invalid clkspeed: %d\n",
- qcaspi_clkspeed);
+ dev_err(&spi->dev, "Invalid clkspeed: %d\n",
+ qcaspi_clkspeed);
return -EINVAL;
}
if ((qcaspi_burst_len < QCASPI_BURST_LEN_MIN) ||
(qcaspi_burst_len > QCASPI_BURST_LEN_MAX)) {
- dev_info(&spi->dev, "Invalid burst len: %d\n",
- qcaspi_burst_len);
+ dev_err(&spi->dev, "Invalid burst len: %d\n",
+ qcaspi_burst_len);
return -EINVAL;
}
if ((qcaspi_pluggable < QCASPI_PLUGGABLE_MIN) ||
(qcaspi_pluggable > QCASPI_PLUGGABLE_MAX)) {
- dev_info(&spi->dev, "Invalid pluggable: %d\n",
- qcaspi_pluggable);
+ dev_err(&spi->dev, "Invalid pluggable: %d\n",
+ qcaspi_pluggable);
return -EINVAL;
}
@@ -941,8 +944,8 @@ qca_spi_probe(struct spi_device *spi)
}
if (register_netdev(qcaspi_devs)) {
- dev_info(&spi->dev, "Unable to register net device %s\n",
- qcaspi_devs->name);
+ dev_err(&spi->dev, "Unable to register net device %s\n",
+ qcaspi_devs->name);
free_netdev(qcaspi_devs);
return -EFAULT;
}
diff --git a/drivers/net/ethernet/realtek/8139cp.c b/drivers/net/ethernet/realtek/8139cp.c
index 5297bf7..d608c2a 100644
--- a/drivers/net/ethernet/realtek/8139cp.c
+++ b/drivers/net/ethernet/realtek/8139cp.c
@@ -578,6 +578,7 @@ static irqreturn_t cp_interrupt (int irq, void *dev_instance)
struct cp_private *cp;
int handled = 0;
u16 status;
+ u16 mask;
if (unlikely(dev == NULL))
return IRQ_NONE;
@@ -585,6 +586,10 @@ static irqreturn_t cp_interrupt (int irq, void *dev_instance)
spin_lock(&cp->lock);
+ mask = cpr16(IntrMask);
+ if (!mask)
+ goto out_unlock;
+
status = cpr16(IntrStatus);
if (!status || (status == 0xFFFF))
goto out_unlock;
diff --git a/drivers/net/ethernet/realtek/8139too.c b/drivers/net/ethernet/realtek/8139too.c
index da4c2d8..1420dfb 100644
--- a/drivers/net/ethernet/realtek/8139too.c
+++ b/drivers/net/ethernet/realtek/8139too.c
@@ -2233,7 +2233,7 @@ static void rtl8139_poll_controller(struct net_device *dev)
struct rtl8139_private *tp = netdev_priv(dev);
const int irq = tp->pci_dev->irq;
- disable_irq(irq);
+ disable_irq_nosync(irq);
rtl8139_interrupt(irq, dev);
enable_irq(irq);
}
diff --git a/drivers/net/ethernet/realtek/r8169.c b/drivers/net/ethernet/realtek/r8169.c
index dbb6364..24754d3 100644
--- a/drivers/net/ethernet/realtek/r8169.c
+++ b/drivers/net/ethernet/realtek/r8169.c
@@ -329,6 +329,7 @@ static const struct pci_device_id rtl8169_pci_tbl[] = {
{ PCI_DEVICE(PCI_VENDOR_ID_REALTEK, 0x8161), 0, 0, RTL_CFG_1 },
{ PCI_DEVICE(PCI_VENDOR_ID_REALTEK, 0x8167), 0, 0, RTL_CFG_0 },
{ PCI_DEVICE(PCI_VENDOR_ID_REALTEK, 0x8168), 0, 0, RTL_CFG_1 },
+ { PCI_DEVICE(PCI_VENDOR_ID_NCUBE, 0x8168), 0, 0, RTL_CFG_1 },
{ PCI_DEVICE(PCI_VENDOR_ID_REALTEK, 0x8169), 0, 0, RTL_CFG_0 },
{ PCI_VENDOR_ID_DLINK, 0x4300,
PCI_VENDOR_ID_DLINK, 0x4b10, 0, 0, RTL_CFG_1 },
@@ -759,7 +760,7 @@ struct rtl8169_tc_offsets {
};
enum rtl_flag {
- RTL_FLAG_TASK_ENABLED,
+ RTL_FLAG_TASK_ENABLED = 0,
RTL_FLAG_TASK_SLOW_PENDING,
RTL_FLAG_TASK_RESET_PENDING,
RTL_FLAG_TASK_PHY_PENDING,
@@ -4861,6 +4862,9 @@ static void rtl_pll_power_down(struct rtl8169_private *tp)
static void rtl_pll_power_up(struct rtl8169_private *tp)
{
rtl_generic_op(tp, tp->pll_power_ops.up);
+
+ /* give MAC/PHY some time to resume */
+ msleep(20);
}
static void rtl_init_pll_power_ops(struct rtl8169_private *tp)
@@ -7555,17 +7559,15 @@ static int rtl8169_poll(struct napi_struct *napi, int budget)
struct rtl8169_private *tp = container_of(napi, struct rtl8169_private, napi);
struct net_device *dev = tp->dev;
u16 enable_mask = RTL_EVENT_NAPI | tp->event_slow;
- int work_done= 0;
+ int work_done;
u16 status;
status = rtl_get_events(tp);
rtl_ack_events(tp, status & ~tp->event_slow);
- if (status & RTL_EVENT_NAPI_RX)
- work_done = rtl_rx(dev, tp, (u32) budget);
+ work_done = rtl_rx(dev, tp, (u32) budget);
- if (status & RTL_EVENT_NAPI_TX)
- rtl_tx(dev, tp);
+ rtl_tx(dev, tp);
if (status & tp->event_slow) {
enable_mask &= ~tp->event_slow;
@@ -7633,7 +7635,8 @@ static int rtl8169_close(struct net_device *dev)
rtl8169_update_counters(dev);
rtl_lock_work(tp);
- clear_bit(RTL_FLAG_TASK_ENABLED, tp->wk.flags);
+ /* Clear all task flags */
+ bitmap_zero(tp->wk.flags, RTL_FLAG_MAX);
rtl8169_down(dev);
rtl_unlock_work(tp);
@@ -7816,7 +7819,9 @@ static void rtl8169_net_suspend(struct net_device *dev)
rtl_lock_work(tp);
napi_disable(&tp->napi);
- clear_bit(RTL_FLAG_TASK_ENABLED, tp->wk.flags);
+ /* Clear all task flags */
+ bitmap_zero(tp->wk.flags, RTL_FLAG_MAX);
+
rtl_unlock_work(tp);
rtl_pll_power_down(tp);
diff --git a/drivers/net/ethernet/renesas/ravb.h b/drivers/net/ethernet/renesas/ravb.h
index f110966..4103990 100644
--- a/drivers/net/ethernet/renesas/ravb.h
+++ b/drivers/net/ethernet/renesas/ravb.h
@@ -421,6 +421,7 @@ enum EIS_BIT {
EIS_CULF1 = 0x00000080,
EIS_TFFF = 0x00000100,
EIS_QFS = 0x00010000,
+ EIS_RESERVED = (GENMASK(31, 17) | GENMASK(15, 11)),
};
/* RIC0 */
@@ -465,6 +466,7 @@ enum RIS0_BIT {
RIS0_FRF15 = 0x00008000,
RIS0_FRF16 = 0x00010000,
RIS0_FRF17 = 0x00020000,
+ RIS0_RESERVED = GENMASK(31, 18),
};
/* RIC1 */
@@ -521,6 +523,7 @@ enum RIS2_BIT {
RIS2_QFF16 = 0x00010000,
RIS2_QFF17 = 0x00020000,
RIS2_RFFF = 0x80000000,
+ RIS2_RESERVED = GENMASK(30, 18),
};
/* TIC */
@@ -537,6 +540,7 @@ enum TIS_BIT {
TIS_FTF1 = 0x00000002, /* Undocumented? */
TIS_TFUF = 0x00000100,
TIS_TFWF = 0x00000200,
+ TIS_RESERVED = (GENMASK(31, 20) | GENMASK(15, 12) | GENMASK(7, 4))
};
/* ISS */
@@ -610,6 +614,7 @@ enum GIC_BIT {
enum GIS_BIT {
GIS_PTCF = 0x00000001, /* Undocumented? */
GIS_PTMF = 0x00000004,
+ GIS_RESERVED = GENMASK(15, 10),
};
/* GIE (R-Car Gen3 only) */
diff --git a/drivers/net/ethernet/renesas/ravb_main.c b/drivers/net/ethernet/renesas/ravb_main.c
index 10d3a9f..71836a7 100644
--- a/drivers/net/ethernet/renesas/ravb_main.c
+++ b/drivers/net/ethernet/renesas/ravb_main.c
@@ -717,10 +717,11 @@ static void ravb_error_interrupt(struct net_device *ndev)
u32 eis, ris2;
eis = ravb_read(ndev, EIS);
- ravb_write(ndev, ~EIS_QFS, EIS);
+ ravb_write(ndev, ~(EIS_QFS | EIS_RESERVED), EIS);
if (eis & EIS_QFS) {
ris2 = ravb_read(ndev, RIS2);
- ravb_write(ndev, ~(RIS2_QFF0 | RIS2_RFFF), RIS2);
+ ravb_write(ndev, ~(RIS2_QFF0 | RIS2_RFFF | RIS2_RESERVED),
+ RIS2);
/* Receive Descriptor Empty int */
if (ris2 & RIS2_QFF0)
@@ -773,7 +774,7 @@ static bool ravb_timestamp_interrupt(struct net_device *ndev)
u32 tis = ravb_read(ndev, TIS);
if (tis & TIS_TFUF) {
- ravb_write(ndev, ~TIS_TFUF, TIS);
+ ravb_write(ndev, ~(TIS_TFUF | TIS_RESERVED), TIS);
ravb_get_tx_tstamp(ndev);
return true;
}
@@ -908,7 +909,7 @@ static int ravb_poll(struct napi_struct *napi, int budget)
/* Processing RX Descriptor Ring */
if (ris0 & mask) {
/* Clear RX interrupt */
- ravb_write(ndev, ~mask, RIS0);
+ ravb_write(ndev, ~(mask | RIS0_RESERVED), RIS0);
if (ravb_rx(ndev, "a, q))
goto out;
}
@@ -916,7 +917,7 @@ static int ravb_poll(struct napi_struct *napi, int budget)
if (tis & mask) {
spin_lock_irqsave(&priv->lock, flags);
/* Clear TX interrupt */
- ravb_write(ndev, ~mask, TIS);
+ ravb_write(ndev, ~(mask | TIS_RESERVED), TIS);
ravb_tx_free(ndev, q, true);
netif_wake_subqueue(ndev, q);
mmiowb();
@@ -955,6 +956,13 @@ static void ravb_adjust_link(struct net_device *ndev)
struct ravb_private *priv = netdev_priv(ndev);
struct phy_device *phydev = ndev->phydev;
bool new_state = false;
+ unsigned long flags;
+
+ spin_lock_irqsave(&priv->lock, flags);
+
+ /* Disable TX and RX right over here, if E-MAC change is ignored */
+ if (priv->no_avb_link)
+ ravb_rcv_snd_disable(ndev);
if (phydev->link) {
if (phydev->duplex != priv->duplex) {
@@ -972,18 +980,21 @@ static void ravb_adjust_link(struct net_device *ndev)
ravb_modify(ndev, ECMR, ECMR_TXF, 0);
new_state = true;
priv->link = phydev->link;
- if (priv->no_avb_link)
- ravb_rcv_snd_enable(ndev);
}
} else if (priv->link) {
new_state = true;
priv->link = 0;
priv->speed = 0;
priv->duplex = -1;
- if (priv->no_avb_link)
- ravb_rcv_snd_disable(ndev);
}
+ /* Enable TX and RX right over here, if E-MAC change is ignored */
+ if (priv->no_avb_link && phydev->link)
+ ravb_rcv_snd_enable(ndev);
+
+ mmiowb();
+ spin_unlock_irqrestore(&priv->lock, flags);
+
if (new_state && netif_msg_link(priv))
phy_print_status(phydev);
}
@@ -1085,52 +1096,18 @@ static int ravb_get_link_ksettings(struct net_device *ndev,
static int ravb_set_link_ksettings(struct net_device *ndev,
const struct ethtool_link_ksettings *cmd)
{
- struct ravb_private *priv = netdev_priv(ndev);
- unsigned long flags;
- int error;
-
if (!ndev->phydev)
return -ENODEV;
- spin_lock_irqsave(&priv->lock, flags);
-
- /* Disable TX and RX */
- ravb_rcv_snd_disable(ndev);
-
- error = phy_ethtool_ksettings_set(ndev->phydev, cmd);
- if (error)
- goto error_exit;
-
- if (cmd->base.duplex == DUPLEX_FULL)
- priv->duplex = 1;
- else
- priv->duplex = 0;
-
- ravb_set_duplex(ndev);
-
-error_exit:
- mdelay(1);
-
- /* Enable TX and RX */
- ravb_rcv_snd_enable(ndev);
-
- mmiowb();
- spin_unlock_irqrestore(&priv->lock, flags);
-
- return error;
+ return phy_ethtool_ksettings_set(ndev->phydev, cmd);
}
static int ravb_nway_reset(struct net_device *ndev)
{
- struct ravb_private *priv = netdev_priv(ndev);
int error = -ENODEV;
- unsigned long flags;
- if (ndev->phydev) {
- spin_lock_irqsave(&priv->lock, flags);
+ if (ndev->phydev)
error = phy_start_aneg(ndev->phydev);
- spin_unlock_irqrestore(&priv->lock, flags);
- }
return error;
}
diff --git a/drivers/net/ethernet/renesas/ravb_ptp.c b/drivers/net/ethernet/renesas/ravb_ptp.c
index eede70ec..9e3222f 100644
--- a/drivers/net/ethernet/renesas/ravb_ptp.c
+++ b/drivers/net/ethernet/renesas/ravb_ptp.c
@@ -319,7 +319,7 @@ void ravb_ptp_interrupt(struct net_device *ndev)
}
}
- ravb_write(ndev, ~gis, GIS);
+ ravb_write(ndev, ~(gis | GIS_RESERVED), GIS);
}
void ravb_ptp_init(struct net_device *ndev, struct platform_device *pdev)
diff --git a/drivers/net/ethernet/renesas/sh_eth.c b/drivers/net/ethernet/renesas/sh_eth.c
index c8fd99b..c59e8fe 100644
--- a/drivers/net/ethernet/renesas/sh_eth.c
+++ b/drivers/net/ethernet/renesas/sh_eth.c
@@ -1743,8 +1743,15 @@ static void sh_eth_adjust_link(struct net_device *ndev)
{
struct sh_eth_private *mdp = netdev_priv(ndev);
struct phy_device *phydev = ndev->phydev;
+ unsigned long flags;
int new_state = 0;
+ spin_lock_irqsave(&mdp->lock, flags);
+
+ /* Disable TX and RX right over here, if E-MAC change is ignored */
+ if (mdp->cd->no_psr || mdp->no_ether_link)
+ sh_eth_rcv_snd_disable(ndev);
+
if (phydev->link) {
if (phydev->duplex != mdp->duplex) {
new_state = 1;
@@ -1763,18 +1770,21 @@ static void sh_eth_adjust_link(struct net_device *ndev)
sh_eth_modify(ndev, ECMR, ECMR_TXF, 0);
new_state = 1;
mdp->link = phydev->link;
- if (mdp->cd->no_psr || mdp->no_ether_link)
- sh_eth_rcv_snd_enable(ndev);
}
} else if (mdp->link) {
new_state = 1;
mdp->link = 0;
mdp->speed = 0;
mdp->duplex = -1;
- if (mdp->cd->no_psr || mdp->no_ether_link)
- sh_eth_rcv_snd_disable(ndev);
}
+ /* Enable TX and RX right over here, if E-MAC change is ignored */
+ if ((mdp->cd->no_psr || mdp->no_ether_link) && phydev->link)
+ sh_eth_rcv_snd_enable(ndev);
+
+ mmiowb();
+ spin_unlock_irqrestore(&mdp->lock, flags);
+
if (new_state && netif_msg_link(mdp))
phy_print_status(phydev);
}
@@ -1856,39 +1866,10 @@ static int sh_eth_get_link_ksettings(struct net_device *ndev,
static int sh_eth_set_link_ksettings(struct net_device *ndev,
const struct ethtool_link_ksettings *cmd)
{
- struct sh_eth_private *mdp = netdev_priv(ndev);
- unsigned long flags;
- int ret;
-
if (!ndev->phydev)
return -ENODEV;
- spin_lock_irqsave(&mdp->lock, flags);
-
- /* disable tx and rx */
- sh_eth_rcv_snd_disable(ndev);
-
- ret = phy_ethtool_ksettings_set(ndev->phydev, cmd);
- if (ret)
- goto error_exit;
-
- if (cmd->base.duplex == DUPLEX_FULL)
- mdp->duplex = 1;
- else
- mdp->duplex = 0;
-
- if (mdp->cd->set_duplex)
- mdp->cd->set_duplex(ndev);
-
-error_exit:
- mdelay(1);
-
- /* enable tx and rx */
- sh_eth_rcv_snd_enable(ndev);
-
- spin_unlock_irqrestore(&mdp->lock, flags);
-
- return ret;
+ return phy_ethtool_ksettings_set(ndev->phydev, cmd);
}
/* If it is ever necessary to increase SH_ETH_REG_DUMP_MAX_REGS, the
@@ -2079,18 +2060,10 @@ static void sh_eth_get_regs(struct net_device *ndev, struct ethtool_regs *regs,
static int sh_eth_nway_reset(struct net_device *ndev)
{
- struct sh_eth_private *mdp = netdev_priv(ndev);
- unsigned long flags;
- int ret;
-
if (!ndev->phydev)
return -ENODEV;
- spin_lock_irqsave(&mdp->lock, flags);
- ret = phy_start_aneg(ndev->phydev);
- spin_unlock_irqrestore(&mdp->lock, flags);
-
- return ret;
+ return phy_start_aneg(ndev->phydev);
}
static u32 sh_eth_get_msglevel(struct net_device *ndev)
diff --git a/drivers/net/ethernet/smsc/smsc911x.c b/drivers/net/ethernet/smsc/smsc911x.c
index 8b0016a..734caa7 100644
--- a/drivers/net/ethernet/smsc/smsc911x.c
+++ b/drivers/net/ethernet/smsc/smsc911x.c
@@ -2330,14 +2330,14 @@ static int smsc911x_drv_remove(struct platform_device *pdev)
pdata = netdev_priv(dev);
BUG_ON(!pdata);
BUG_ON(!pdata->ioaddr);
- WARN_ON(dev->phydev);
SMSC_TRACE(pdata, ifdown, "Stopping driver");
+ unregister_netdev(dev);
+
mdiobus_unregister(pdata->mii_bus);
mdiobus_free(pdata->mii_bus);
- unregister_netdev(dev);
res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
"smsc911x-memory");
if (!res)
diff --git a/drivers/net/ethernet/stmicro/stmmac/Kconfig b/drivers/net/ethernet/stmicro/stmmac/Kconfig
index 4b78168..0d03682 100644
--- a/drivers/net/ethernet/stmicro/stmmac/Kconfig
+++ b/drivers/net/ethernet/stmicro/stmmac/Kconfig
@@ -83,7 +83,7 @@
config DWMAC_SOCFPGA
tristate "SOCFPGA dwmac support"
default ARCH_SOCFPGA
- depends on OF && (ARCH_SOCFPGA || COMPILE_TEST)
+ depends on OF && (ARCH_SOCFPGA || ARCH_STRATIX10 || COMPILE_TEST)
select MFD_SYSCON
help
Support for ethernet controller on Altera SOCFPGA
diff --git a/drivers/net/ethernet/stmicro/stmmac/altr_tse_pcs.c b/drivers/net/ethernet/stmicro/stmmac/altr_tse_pcs.c
index 489ef14..6a9c9544 100644
--- a/drivers/net/ethernet/stmicro/stmmac/altr_tse_pcs.c
+++ b/drivers/net/ethernet/stmicro/stmmac/altr_tse_pcs.c
@@ -37,6 +37,7 @@
#define TSE_PCS_CONTROL_AN_EN_MASK BIT(12)
#define TSE_PCS_CONTROL_REG 0x00
#define TSE_PCS_CONTROL_RESTART_AN_MASK BIT(9)
+#define TSE_PCS_CTRL_AUTONEG_SGMII 0x1140
#define TSE_PCS_IF_MODE_REG 0x28
#define TSE_PCS_LINK_TIMER_0_REG 0x24
#define TSE_PCS_LINK_TIMER_1_REG 0x26
@@ -65,6 +66,7 @@
#define TSE_PCS_SW_RESET_TIMEOUT 100
#define TSE_PCS_USE_SGMII_AN_MASK BIT(1)
#define TSE_PCS_USE_SGMII_ENA BIT(0)
+#define TSE_PCS_IF_USE_SGMII 0x03
#define SGMII_ADAPTER_CTRL_REG 0x00
#define SGMII_ADAPTER_DISABLE 0x0001
@@ -101,7 +103,9 @@ int tse_pcs_init(void __iomem *base, struct tse_pcs *pcs)
{
int ret = 0;
- writew(TSE_PCS_USE_SGMII_ENA, base + TSE_PCS_IF_MODE_REG);
+ writew(TSE_PCS_IF_USE_SGMII, base + TSE_PCS_IF_MODE_REG);
+
+ writew(TSE_PCS_CTRL_AUTONEG_SGMII, base + TSE_PCS_CONTROL_REG);
writew(TSE_PCS_SGMII_LINK_TIMER_0, base + TSE_PCS_LINK_TIMER_0_REG);
writew(TSE_PCS_SGMII_LINK_TIMER_1, base + TSE_PCS_LINK_TIMER_1_REG);
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac-meson8b.c b/drivers/net/ethernet/stmicro/stmmac/dwmac-meson8b.c
index ffaed1f..f356a44 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac-meson8b.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac-meson8b.c
@@ -118,7 +118,7 @@ static int meson8b_init_clk(struct meson8b_dwmac *dwmac)
snprintf(clk_name, sizeof(clk_name), "%s#m250_sel", dev_name(dev));
init.name = clk_name;
init.ops = &clk_mux_ops;
- init.flags = 0;
+ init.flags = CLK_SET_RATE_PARENT;
init.parent_names = mux_parent_names;
init.num_parents = MUX_CLK_NUM_PARENTS;
@@ -146,7 +146,9 @@ static int meson8b_init_clk(struct meson8b_dwmac *dwmac)
dwmac->m250_div.shift = PRG_ETH0_CLK_M250_DIV_SHIFT;
dwmac->m250_div.width = PRG_ETH0_CLK_M250_DIV_WIDTH;
dwmac->m250_div.hw.init = &init;
- dwmac->m250_div.flags = CLK_DIVIDER_ONE_BASED | CLK_DIVIDER_ALLOW_ZERO;
+ dwmac->m250_div.flags = CLK_DIVIDER_ONE_BASED |
+ CLK_DIVIDER_ALLOW_ZERO |
+ CLK_DIVIDER_ROUND_CLOSEST;
dwmac->m250_div_clk = devm_clk_register(dev, &dwmac->m250_div.hw);
if (WARN_ON(IS_ERR(dwmac->m250_div_clk)))
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac-socfpga.c b/drivers/net/ethernet/stmicro/stmmac/dwmac-socfpga.c
index 0c420e9..c3a78c1 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac-socfpga.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac-socfpga.c
@@ -55,6 +55,7 @@ struct socfpga_dwmac {
struct device *dev;
struct regmap *sys_mgr_base_addr;
struct reset_control *stmmac_rst;
+ struct reset_control *stmmac_ocp_rst;
void __iomem *splitter_base;
bool f2h_ptp_ref_clk;
struct tse_pcs pcs;
@@ -262,8 +263,8 @@ static int socfpga_dwmac_set_phy_mode(struct socfpga_dwmac *dwmac)
val = SYSMGR_EMACGRP_CTRL_PHYSEL_ENUM_GMII_MII;
/* Assert reset to the enet controller before changing the phy mode */
- if (dwmac->stmmac_rst)
- reset_control_assert(dwmac->stmmac_rst);
+ reset_control_assert(dwmac->stmmac_ocp_rst);
+ reset_control_assert(dwmac->stmmac_rst);
regmap_read(sys_mgr_base_addr, reg_offset, &ctrl);
ctrl &= ~(SYSMGR_EMACGRP_CTRL_PHYSEL_MASK << reg_shift);
@@ -285,8 +286,8 @@ static int socfpga_dwmac_set_phy_mode(struct socfpga_dwmac *dwmac)
/* Deassert reset for the phy configuration to be sampled by
* the enet controller, and operation to start in requested mode
*/
- if (dwmac->stmmac_rst)
- reset_control_deassert(dwmac->stmmac_rst);
+ reset_control_deassert(dwmac->stmmac_ocp_rst);
+ reset_control_deassert(dwmac->stmmac_rst);
if (phymode == PHY_INTERFACE_MODE_SGMII) {
if (tse_pcs_init(dwmac->pcs.tse_pcs_base, &dwmac->pcs) != 0) {
dev_err(dwmac->dev, "Unable to initialize TSE PCS");
@@ -321,6 +322,15 @@ static int socfpga_dwmac_probe(struct platform_device *pdev)
goto err_remove_config_dt;
}
+ dwmac->stmmac_ocp_rst = devm_reset_control_get_optional(dev, "stmmaceth-ocp");
+ if (IS_ERR(dwmac->stmmac_ocp_rst)) {
+ ret = PTR_ERR(dwmac->stmmac_ocp_rst);
+ dev_err(dev, "error getting reset control of ocp %d\n", ret);
+ goto err_remove_config_dt;
+ }
+
+ reset_control_deassert(dwmac->stmmac_ocp_rst);
+
ret = socfpga_dwmac_parse_data(dwmac, dev);
if (ret) {
dev_err(dev, "Unable to parse OF data\n");
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index c212d1d..65ed02b 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -55,7 +55,7 @@
#include <linux/of_mdio.h>
#include "dwmac1000.h"
-#define STMMAC_ALIGN(x) L1_CACHE_ALIGN(x)
+#define STMMAC_ALIGN(x) __ALIGN_KERNEL(x, SMP_CACHE_BYTES)
#define TSO_MAX_BUFF_SIZE (SZ_16K - 1)
/* Module parameters */
@@ -1343,6 +1343,11 @@ static void stmmac_tx_clean(struct stmmac_priv *priv)
if (unlikely(status & tx_dma_own))
break;
+ /* Make sure descriptor fields are read after reading
+ * the own bit.
+ */
+ dma_rmb();
+
/* Just consider the last segment and ...*/
if (likely(!(status & tx_not_ls))) {
/* ... verify the status error condition */
@@ -2136,8 +2141,15 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
tcp_hdrlen(skb) / 4, (skb->len - proto_hdr_len));
/* If context desc is used to change MSS */
- if (mss_desc)
+ if (mss_desc) {
+ /* Make sure that first descriptor has been completely
+ * written, including its own bit. This is because MSS is
+ * actually before first descriptor, so we need to make
+ * sure that MSS's own bit is the last thing written.
+ */
+ dma_wmb();
priv->hw->desc->set_tx_owner(mss_desc);
+ }
/* The own bit must be the latest setting done when prepare the
* descriptor and then barrier is needed to make sure that
@@ -2187,7 +2199,8 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
unsigned int nopaged_len = skb_headlen(skb);
int i, csum_insertion = 0, is_jumbo = 0;
int nfrags = skb_shinfo(skb)->nr_frags;
- unsigned int entry, first_entry;
+ int entry;
+ unsigned int first_entry;
struct dma_desc *desc, *first;
unsigned int enh_desc;
unsigned int des;
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_mdio.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_mdio.c
index ec29585..2abeba4 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_mdio.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_mdio.c
@@ -216,7 +216,7 @@ static int stmmac_mdio_write_gmac4(struct mii_bus *bus, int phyaddr, int phyreg,
*/
int stmmac_mdio_reset(struct mii_bus *bus)
{
-#if defined(CONFIG_STMMAC_PLATFORM)
+#if IS_ENABLED(CONFIG_STMMAC_PLATFORM)
struct net_device *ndev = bus->priv;
struct stmmac_priv *priv = netdev_priv(ndev);
unsigned int mii_address = priv->hw->mii.addr;
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
index 56c8a23..eafc281 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_pci.c
@@ -183,7 +183,7 @@ static int stmmac_pci_probe(struct pci_dev *pdev,
return -ENOMEM;
/* Enable pci device */
- ret = pcim_enable_device(pdev);
+ ret = pci_enable_device(pdev);
if (ret) {
dev_err(&pdev->dev, "%s: ERROR: failed to enable device\n",
__func__);
@@ -232,9 +232,45 @@ static int stmmac_pci_probe(struct pci_dev *pdev,
static void stmmac_pci_remove(struct pci_dev *pdev)
{
stmmac_dvr_remove(&pdev->dev);
+ pci_disable_device(pdev);
}
-static SIMPLE_DEV_PM_OPS(stmmac_pm_ops, stmmac_suspend, stmmac_resume);
+static int stmmac_pci_suspend(struct device *dev)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ int ret;
+
+ ret = stmmac_suspend(dev);
+ if (ret)
+ return ret;
+
+ ret = pci_save_state(pdev);
+ if (ret)
+ return ret;
+
+ pci_disable_device(pdev);
+ pci_wake_from_d3(pdev, true);
+ return 0;
+}
+
+static int stmmac_pci_resume(struct device *dev)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ int ret;
+
+ pci_restore_state(pdev);
+ pci_set_power_state(pdev, PCI_D0);
+
+ ret = pci_enable_device(pdev);
+ if (ret)
+ return ret;
+
+ pci_set_master(pdev);
+
+ return stmmac_resume(dev);
+}
+
+static SIMPLE_DEV_PM_OPS(stmmac_pm_ops, stmmac_pci_suspend, stmmac_pci_resume);
#define STMMAC_VENDOR_ID 0x700
#define STMMAC_QUARK_ID 0x0937
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c
index 890e4b0..2019e16 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_platform.c
@@ -71,7 +71,7 @@ static int dwmac1000_validate_mcast_bins(int mcast_bins)
* Description:
* This function validates the number of Unicast address entries supported
* by a particular Synopsys 10/100/1000 controller. The Synopsys controller
- * supports 1, 32, 64, or 128 Unicast filter entries for it's Unicast filter
+ * supports 1..32, 64, or 128 Unicast filter entries for it's Unicast filter
* logic. This function validates a valid, supported configuration is
* selected, and defaults to 1 Unicast address if an unsupported
* configuration is selected.
@@ -81,8 +81,7 @@ static int dwmac1000_validate_ucast_entries(int ucast_entries)
int x = ucast_entries;
switch (x) {
- case 1:
- case 32:
+ case 1 ... 32:
case 64:
case 128:
break;
diff --git a/drivers/net/ethernet/sun/niu.c b/drivers/net/ethernet/sun/niu.c
index a2371aa..e45e2f1 100644
--- a/drivers/net/ethernet/sun/niu.c
+++ b/drivers/net/ethernet/sun/niu.c
@@ -3442,7 +3442,7 @@ static int niu_process_rx_pkt(struct napi_struct *napi, struct niu *np,
len = (val & RCR_ENTRY_L2_LEN) >>
RCR_ENTRY_L2_LEN_SHIFT;
- len -= ETH_FCS_LEN;
+ append_size = len + ETH_HLEN + ETH_FCS_LEN;
addr = (val & RCR_ENTRY_PKT_BUF_ADDR) <<
RCR_ENTRY_PKT_BUF_ADDR_SHIFT;
@@ -3452,7 +3452,6 @@ static int niu_process_rx_pkt(struct napi_struct *napi, struct niu *np,
RCR_ENTRY_PKTBUFSZ_SHIFT];
off = addr & ~PAGE_MASK;
- append_size = rcr_size;
if (num_rcr == 1) {
int ptype;
@@ -3465,7 +3464,7 @@ static int niu_process_rx_pkt(struct napi_struct *napi, struct niu *np,
else
skb_checksum_none_assert(skb);
} else if (!(val & RCR_ENTRY_MULTI))
- append_size = len - skb->len;
+ append_size = append_size - skb->len;
niu_rx_skb_append(skb, page, off, append_size, rcr_size);
if ((page->index + rp->rbr_block_size) - rcr_size == addr) {
diff --git a/drivers/net/ethernet/sun/sungem.c b/drivers/net/ethernet/sun/sungem.c
index d6ad0fb..920321b 100644
--- a/drivers/net/ethernet/sun/sungem.c
+++ b/drivers/net/ethernet/sun/sungem.c
@@ -59,8 +59,7 @@
#include <linux/sungem_phy.h>
#include "sungem.h"
-/* Stripping FCS is causing problems, disabled for now */
-#undef STRIP_FCS
+#define STRIP_FCS
#define DEFAULT_MSG (NETIF_MSG_DRV | \
NETIF_MSG_PROBE | \
@@ -434,7 +433,7 @@ static int gem_rxmac_reset(struct gem *gp)
writel(desc_dma & 0xffffffff, gp->regs + RXDMA_DBLOW);
writel(RX_RING_SIZE - 4, gp->regs + RXDMA_KICK);
val = (RXDMA_CFG_BASE | (RX_OFFSET << 10) |
- ((14 / 2) << 13) | RXDMA_CFG_FTHRESH_128);
+ (ETH_HLEN << 13) | RXDMA_CFG_FTHRESH_128);
writel(val, gp->regs + RXDMA_CFG);
if (readl(gp->regs + GREG_BIFCFG) & GREG_BIFCFG_M66EN)
writel(((5 & RXDMA_BLANK_IPKTS) |
@@ -759,7 +758,6 @@ static int gem_rx(struct gem *gp, int work_to_do)
struct net_device *dev = gp->dev;
int entry, drops, work_done = 0;
u32 done;
- __sum16 csum;
if (netif_msg_rx_status(gp))
printk(KERN_DEBUG "%s: rx interrupt, done: %d, rx_new: %d\n",
@@ -854,9 +852,13 @@ static int gem_rx(struct gem *gp, int work_to_do)
skb = copy_skb;
}
- csum = (__force __sum16)htons((status & RXDCTRL_TCPCSUM) ^ 0xffff);
- skb->csum = csum_unfold(csum);
- skb->ip_summed = CHECKSUM_COMPLETE;
+ if (likely(dev->features & NETIF_F_RXCSUM)) {
+ __sum16 csum;
+
+ csum = (__force __sum16)htons((status & RXDCTRL_TCPCSUM) ^ 0xffff);
+ skb->csum = csum_unfold(csum);
+ skb->ip_summed = CHECKSUM_COMPLETE;
+ }
skb->protocol = eth_type_trans(skb, gp->dev);
napi_gro_receive(&gp->napi, skb);
@@ -1754,7 +1756,7 @@ static void gem_init_dma(struct gem *gp)
writel(0, gp->regs + TXDMA_KICK);
val = (RXDMA_CFG_BASE | (RX_OFFSET << 10) |
- ((14 / 2) << 13) | RXDMA_CFG_FTHRESH_128);
+ (ETH_HLEN << 13) | RXDMA_CFG_FTHRESH_128);
writel(val, gp->regs + RXDMA_CFG);
writel(desc_dma >> 32, gp->regs + RXDMA_DBHI);
@@ -2972,8 +2974,8 @@ static int gem_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
pci_set_drvdata(pdev, dev);
/* We can do scatter/gather and HW checksum */
- dev->hw_features = NETIF_F_SG | NETIF_F_HW_CSUM;
- dev->features |= dev->hw_features | NETIF_F_RXCSUM;
+ dev->hw_features = NETIF_F_SG | NETIF_F_HW_CSUM | NETIF_F_RXCSUM;
+ dev->features = dev->hw_features;
if (pci_using_dac)
dev->features |= NETIF_F_HIGHDMA;
diff --git a/drivers/net/ethernet/sun/sunvnet.c b/drivers/net/ethernet/sun/sunvnet.c
index a2f9b47..e36c700 100644
--- a/drivers/net/ethernet/sun/sunvnet.c
+++ b/drivers/net/ethernet/sun/sunvnet.c
@@ -198,7 +198,7 @@ static struct vnet *vnet_new(const u64 *local_mac,
dev->ethtool_ops = &vnet_ethtool_ops;
dev->watchdog_timeo = VNET_TX_TIMEOUT;
- dev->hw_features = NETIF_F_TSO | NETIF_F_GSO | NETIF_F_GSO_SOFTWARE |
+ dev->hw_features = NETIF_F_TSO | NETIF_F_GSO | NETIF_F_ALL_TSO |
NETIF_F_HW_CSUM | NETIF_F_SG;
dev->features = dev->hw_features;
diff --git a/drivers/net/ethernet/ti/cpsw-phy-sel.c b/drivers/net/ethernet/ti/cpsw-phy-sel.c
index 1801364..0c1adad 100644
--- a/drivers/net/ethernet/ti/cpsw-phy-sel.c
+++ b/drivers/net/ethernet/ti/cpsw-phy-sel.c
@@ -177,12 +177,18 @@ void cpsw_phy_sel(struct device *dev, phy_interface_t phy_mode, int slave)
}
dev = bus_find_device(&platform_bus_type, NULL, node, match);
- of_node_put(node);
+ if (!dev) {
+ dev_err(dev, "unable to find platform device for %pOF\n", node);
+ goto out;
+ }
+
priv = dev_get_drvdata(dev);
priv->cpsw_phy_sel(priv, phy_mode, slave);
put_device(dev);
+out:
+ of_node_put(node);
}
EXPORT_SYMBOL_GPL(cpsw_phy_sel);
diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c
index 552de9c..d7cb205 100644
--- a/drivers/net/ethernet/ti/cpsw.c
+++ b/drivers/net/ethernet/ti/cpsw.c
@@ -124,7 +124,7 @@ do { \
#define RX_PRIORITY_MAPPING 0x76543210
#define TX_PRIORITY_MAPPING 0x33221100
-#define CPDMA_TX_PRIORITY_MAP 0x01234567
+#define CPDMA_TX_PRIORITY_MAP 0x76543210
#define CPSW_VLAN_AWARE BIT(1)
#define CPSW_ALE_VLAN_AWARE 1
@@ -1141,6 +1141,8 @@ static inline void cpsw_add_dual_emac_def_ale_entries(
cpsw_ale_add_ucast(cpsw->ale, priv->mac_addr,
HOST_PORT_NUM, ALE_VLAN |
ALE_SECURE, slave->port_vlan);
+ cpsw_ale_control_set(cpsw->ale, slave_port,
+ ALE_PORT_DROP_UNKNOWN_VLAN, 1);
}
static void soft_reset_slave(struct cpsw_slave *slave)
diff --git a/drivers/net/ethernet/ti/davinci_emac.c b/drivers/net/ethernet/ti/davinci_emac.c
index 481c7bf..413cf14 100644
--- a/drivers/net/ethernet/ti/davinci_emac.c
+++ b/drivers/net/ethernet/ti/davinci_emac.c
@@ -1387,6 +1387,10 @@ static int emac_devioctl(struct net_device *ndev, struct ifreq *ifrq, int cmd)
static int match_first_device(struct device *dev, void *data)
{
+ if (dev->parent && dev->parent->of_node)
+ return of_device_is_compatible(dev->parent->of_node,
+ "ti,davinci_mdio");
+
return !strncmp(dev_name(dev), "davinci_mdio", 12);
}
diff --git a/drivers/net/ethernet/xilinx/xilinx_axienet_mdio.c b/drivers/net/ethernet/xilinx/xilinx_axienet_mdio.c
index 63307ea..9beea13 100644
--- a/drivers/net/ethernet/xilinx/xilinx_axienet_mdio.c
+++ b/drivers/net/ethernet/xilinx/xilinx_axienet_mdio.c
@@ -217,6 +217,7 @@ int axienet_mdio_setup(struct axienet_local *lp, struct device_node *np)
ret = of_mdiobus_register(bus, np1);
if (ret) {
mdiobus_free(bus);
+ lp->mii_bus = NULL;
return ret;
}
return 0;
diff --git a/drivers/net/hamradio/bpqether.c b/drivers/net/hamradio/bpqether.c
index 622ab3a..f5e0983 100644
--- a/drivers/net/hamradio/bpqether.c
+++ b/drivers/net/hamradio/bpqether.c
@@ -89,10 +89,6 @@
static const char banner[] __initconst = KERN_INFO \
"AX.25: bpqether driver version 004\n";
-static char bcast_addr[6]={0xFF,0xFF,0xFF,0xFF,0xFF,0xFF};
-
-static char bpq_eth_addr[6];
-
static int bpq_rcv(struct sk_buff *, struct net_device *, struct packet_type *, struct net_device *);
static int bpq_device_event(struct notifier_block *, unsigned long, void *);
@@ -515,8 +511,8 @@ static int bpq_new_device(struct net_device *edev)
bpq->ethdev = edev;
bpq->axdev = ndev;
- memcpy(bpq->dest_addr, bcast_addr, sizeof(bpq_eth_addr));
- memcpy(bpq->acpt_addr, bcast_addr, sizeof(bpq_eth_addr));
+ eth_broadcast_addr(bpq->dest_addr);
+ eth_broadcast_addr(bpq->acpt_addr);
err = register_netdevice(ndev);
if (err)
diff --git a/drivers/net/hyperv/netvsc_drv.c b/drivers/net/hyperv/netvsc_drv.c
index 36a04e1..53602fdf 100644
--- a/drivers/net/hyperv/netvsc_drv.c
+++ b/drivers/net/hyperv/netvsc_drv.c
@@ -29,6 +29,7 @@
#include <linux/netdevice.h>
#include <linux/inetdevice.h>
#include <linux/etherdevice.h>
+#include <linux/pci.h>
#include <linux/skbuff.h>
#include <linux/if_vlan.h>
#include <linux/in.h>
@@ -1228,11 +1229,15 @@ static int netvsc_register_vf(struct net_device *vf_netdev)
{
struct net_device *ndev;
struct net_device_context *net_device_ctx;
+ struct device *pdev = vf_netdev->dev.parent;
struct netvsc_device *netvsc_dev;
if (vf_netdev->addr_len != ETH_ALEN)
return NOTIFY_DONE;
+ if (!pdev || !dev_is_pci(pdev) || dev_is_pf(pdev))
+ return NOTIFY_DONE;
+
/*
* We will use the MAC address to locate the synthetic interface to
* associate with the VF interface. If we don't find a matching
diff --git a/drivers/net/ieee802154/at86rf230.c b/drivers/net/ieee802154/at86rf230.c
index 9f10da6..ce3b7fb 100644
--- a/drivers/net/ieee802154/at86rf230.c
+++ b/drivers/net/ieee802154/at86rf230.c
@@ -941,7 +941,7 @@ at86rf230_xmit(struct ieee802154_hw *hw, struct sk_buff *skb)
static int
at86rf230_ed(struct ieee802154_hw *hw, u8 *level)
{
- BUG_ON(!level);
+ WARN_ON(!level);
*level = 0xbe;
return 0;
}
@@ -1117,8 +1117,7 @@ at86rf230_set_hw_addr_filt(struct ieee802154_hw *hw,
if (changed & IEEE802154_AFILT_SADDR_CHANGED) {
u16 addr = le16_to_cpu(filt->short_addr);
- dev_vdbg(&lp->spi->dev,
- "at86rf230_set_hw_addr_filt called for saddr\n");
+ dev_vdbg(&lp->spi->dev, "%s called for saddr\n", __func__);
__at86rf230_write(lp, RG_SHORT_ADDR_0, addr);
__at86rf230_write(lp, RG_SHORT_ADDR_1, addr >> 8);
}
@@ -1126,8 +1125,7 @@ at86rf230_set_hw_addr_filt(struct ieee802154_hw *hw,
if (changed & IEEE802154_AFILT_PANID_CHANGED) {
u16 pan = le16_to_cpu(filt->pan_id);
- dev_vdbg(&lp->spi->dev,
- "at86rf230_set_hw_addr_filt called for pan id\n");
+ dev_vdbg(&lp->spi->dev, "%s called for pan id\n", __func__);
__at86rf230_write(lp, RG_PAN_ID_0, pan);
__at86rf230_write(lp, RG_PAN_ID_1, pan >> 8);
}
@@ -1136,15 +1134,13 @@ at86rf230_set_hw_addr_filt(struct ieee802154_hw *hw,
u8 i, addr[8];
memcpy(addr, &filt->ieee_addr, 8);
- dev_vdbg(&lp->spi->dev,
- "at86rf230_set_hw_addr_filt called for IEEE addr\n");
+ dev_vdbg(&lp->spi->dev, "%s called for IEEE addr\n", __func__);
for (i = 0; i < 8; i++)
__at86rf230_write(lp, RG_IEEE_ADDR_0 + i, addr[i]);
}
if (changed & IEEE802154_AFILT_PANC_CHANGED) {
- dev_vdbg(&lp->spi->dev,
- "at86rf230_set_hw_addr_filt called for panc change\n");
+ dev_vdbg(&lp->spi->dev, "%s called for panc change\n", __func__);
if (filt->pan_coord)
at86rf230_write_subreg(lp, SR_AACK_I_AM_COORD, 1);
else
@@ -1248,7 +1244,6 @@ at86rf230_set_cca_mode(struct ieee802154_hw *hw,
return at86rf230_write_subreg(lp, SR_CCA_MODE, val);
}
-
static int
at86rf230_set_cca_ed_level(struct ieee802154_hw *hw, s32 mbm)
{
diff --git a/drivers/net/ieee802154/fakelb.c b/drivers/net/ieee802154/fakelb.c
index ec387ef..6853981 100644
--- a/drivers/net/ieee802154/fakelb.c
+++ b/drivers/net/ieee802154/fakelb.c
@@ -49,7 +49,7 @@ struct fakelb_phy {
static int fakelb_hw_ed(struct ieee802154_hw *hw, u8 *level)
{
- BUG_ON(!level);
+ WARN_ON(!level);
*level = 0xbe;
return 0;
diff --git a/drivers/net/ipvlan/ipvlan_main.c b/drivers/net/ipvlan/ipvlan_main.c
index dfbc4ef..b299277 100644
--- a/drivers/net/ipvlan/ipvlan_main.c
+++ b/drivers/net/ipvlan/ipvlan_main.c
@@ -63,10 +63,23 @@ static int ipvlan_set_port_mode(struct ipvl_port *port, u16 nval)
{
struct ipvl_dev *ipvlan;
struct net_device *mdev = port->dev;
- int err = 0;
+ unsigned int flags;
+ int err;
ASSERT_RTNL();
if (port->mode != nval) {
+ list_for_each_entry(ipvlan, &port->ipvlans, pnode) {
+ flags = ipvlan->dev->flags;
+ if (nval == IPVLAN_MODE_L3 || nval == IPVLAN_MODE_L3S) {
+ err = dev_change_flags(ipvlan->dev,
+ flags | IFF_NOARP);
+ } else {
+ err = dev_change_flags(ipvlan->dev,
+ flags & ~IFF_NOARP);
+ }
+ if (unlikely(err))
+ goto fail;
+ }
if (nval == IPVLAN_MODE_L3S) {
/* New mode is L3S */
err = ipvlan_register_nf_hook();
@@ -74,21 +87,28 @@ static int ipvlan_set_port_mode(struct ipvl_port *port, u16 nval)
mdev->l3mdev_ops = &ipvl_l3mdev_ops;
mdev->priv_flags |= IFF_L3MDEV_MASTER;
} else
- return err;
+ goto fail;
} else if (port->mode == IPVLAN_MODE_L3S) {
/* Old mode was L3S */
mdev->priv_flags &= ~IFF_L3MDEV_MASTER;
ipvlan_unregister_nf_hook();
mdev->l3mdev_ops = NULL;
}
- list_for_each_entry(ipvlan, &port->ipvlans, pnode) {
- if (nval == IPVLAN_MODE_L3 || nval == IPVLAN_MODE_L3S)
- ipvlan->dev->flags |= IFF_NOARP;
- else
- ipvlan->dev->flags &= ~IFF_NOARP;
- }
port->mode = nval;
}
+ return 0;
+
+fail:
+ /* Undo the flags changes that have been done so far. */
+ list_for_each_entry_continue_reverse(ipvlan, &port->ipvlans, pnode) {
+ flags = ipvlan->dev->flags;
+ if (port->mode == IPVLAN_MODE_L3 ||
+ port->mode == IPVLAN_MODE_L3S)
+ dev_change_flags(ipvlan->dev, flags | IFF_NOARP);
+ else
+ dev_change_flags(ipvlan->dev, flags & ~IFF_NOARP);
+ }
+
return err;
}
@@ -525,7 +545,8 @@ static int ipvlan_link_new(struct net *src_net, struct net_device *dev,
ipvlan->dev = dev;
ipvlan->port = port;
ipvlan->sfeatures = IPVLAN_FEATURES;
- ipvlan_adjust_mtu(ipvlan, phy_dev);
+ if (!tb[IFLA_MTU])
+ ipvlan_adjust_mtu(ipvlan, phy_dev);
INIT_LIST_HEAD(&ipvlan->addrs);
/* TODO Probably put random address here to be presented to the
diff --git a/drivers/net/macsec.c b/drivers/net/macsec.c
index 365a48c..653f0b1 100644
--- a/drivers/net/macsec.c
+++ b/drivers/net/macsec.c
@@ -744,6 +744,7 @@ static struct sk_buff *macsec_encrypt(struct sk_buff *skb,
sg_init_table(sg, ret);
ret = skb_to_sgvec(skb, sg, 0, skb->len);
if (unlikely(ret < 0)) {
+ aead_request_free(req);
macsec_txsa_put(tx_sa);
kfree_skb(skb);
return ERR_PTR(ret);
@@ -956,6 +957,7 @@ static struct sk_buff *macsec_decrypt(struct sk_buff *skb,
sg_init_table(sg, ret);
ret = skb_to_sgvec(skb, sg, 0, skb->len);
if (unlikely(ret < 0)) {
+ aead_request_free(req);
kfree_skb(skb);
return ERR_PTR(ret);
}
diff --git a/drivers/net/macvlan.c b/drivers/net/macvlan.c
index e8ad4d0..6237236 100644
--- a/drivers/net/macvlan.c
+++ b/drivers/net/macvlan.c
@@ -1384,7 +1384,7 @@ int macvlan_common_newlink(struct net *src_net, struct net_device *dev,
/* the macvlan port may be freed by macvlan_uninit when fail to register.
* so we destroy the macvlan port only when it's valid.
*/
- if (create && macvlan_port_get_rtnl(dev))
+ if (create && macvlan_port_get_rtnl(lowerdev))
macvlan_port_destroy(port->dev);
return err;
}
diff --git a/drivers/net/phy/bcm-cygnus.c b/drivers/net/phy/bcm-cygnus.c
index 49bbc68..9a7dca2 100644
--- a/drivers/net/phy/bcm-cygnus.c
+++ b/drivers/net/phy/bcm-cygnus.c
@@ -61,17 +61,17 @@ static int bcm_cygnus_afe_config(struct phy_device *phydev)
return rc;
/* make rcal=100, since rdb default is 000 */
- rc = bcm_phy_write_exp(phydev, MII_BRCM_CORE_EXPB1, 0x10);
+ rc = bcm_phy_write_exp_sel(phydev, MII_BRCM_CORE_EXPB1, 0x10);
if (rc < 0)
return rc;
/* CORE_EXPB0, Reset R_CAL/RC_CAL Engine */
- rc = bcm_phy_write_exp(phydev, MII_BRCM_CORE_EXPB0, 0x10);
+ rc = bcm_phy_write_exp_sel(phydev, MII_BRCM_CORE_EXPB0, 0x10);
if (rc < 0)
return rc;
/* CORE_EXPB0, Disable Reset R_CAL/RC_CAL Engine */
- rc = bcm_phy_write_exp(phydev, MII_BRCM_CORE_EXPB0, 0x00);
+ rc = bcm_phy_write_exp_sel(phydev, MII_BRCM_CORE_EXPB0, 0x00);
return 0;
}
diff --git a/drivers/net/phy/bcm-phy-lib.h b/drivers/net/phy/bcm-phy-lib.h
index b2091c8..ce16b26 100644
--- a/drivers/net/phy/bcm-phy-lib.h
+++ b/drivers/net/phy/bcm-phy-lib.h
@@ -14,11 +14,18 @@
#ifndef _LINUX_BCM_PHY_LIB_H
#define _LINUX_BCM_PHY_LIB_H
+#include <linux/brcmphy.h>
#include <linux/phy.h>
int bcm_phy_write_exp(struct phy_device *phydev, u16 reg, u16 val);
int bcm_phy_read_exp(struct phy_device *phydev, u16 reg);
+static inline int bcm_phy_write_exp_sel(struct phy_device *phydev,
+ u16 reg, u16 val)
+{
+ return bcm_phy_write_exp(phydev, reg | MII_BCM54XX_EXP_SEL_ER, val);
+}
+
int bcm_phy_write_misc(struct phy_device *phydev,
u16 reg, u16 chl, u16 value);
int bcm_phy_read_misc(struct phy_device *phydev,
diff --git a/drivers/net/phy/bcm7xxx.c b/drivers/net/phy/bcm7xxx.c
index 9636da0..caff474 100644
--- a/drivers/net/phy/bcm7xxx.c
+++ b/drivers/net/phy/bcm7xxx.c
@@ -48,10 +48,10 @@
static void r_rc_cal_reset(struct phy_device *phydev)
{
/* Reset R_CAL/RC_CAL Engine */
- bcm_phy_write_exp(phydev, 0x00b0, 0x0010);
+ bcm_phy_write_exp_sel(phydev, 0x00b0, 0x0010);
/* Disable Reset R_AL/RC_CAL Engine */
- bcm_phy_write_exp(phydev, 0x00b0, 0x0000);
+ bcm_phy_write_exp_sel(phydev, 0x00b0, 0x0000);
}
static int bcm7xxx_28nm_b0_afe_config_init(struct phy_device *phydev)
diff --git a/drivers/net/phy/dp83640.c b/drivers/net/phy/dp83640.c
index b88f7d6..482ea40 100644
--- a/drivers/net/phy/dp83640.c
+++ b/drivers/net/phy/dp83640.c
@@ -1205,6 +1205,23 @@ static void dp83640_remove(struct phy_device *phydev)
kfree(dp83640);
}
+static int dp83640_soft_reset(struct phy_device *phydev)
+{
+ int ret;
+
+ ret = genphy_soft_reset(phydev);
+ if (ret < 0)
+ return ret;
+
+ /* From DP83640 datasheet: "Software driver code must wait 3 us
+ * following a software reset before allowing further serial MII
+ * operations with the DP83640."
+ */
+ udelay(10); /* Taking udelay inaccuracy into account */
+
+ return 0;
+}
+
static int dp83640_config_init(struct phy_device *phydev)
{
struct dp83640_private *dp83640 = phydev->priv;
@@ -1498,6 +1515,7 @@ static struct phy_driver dp83640_driver = {
.flags = PHY_HAS_INTERRUPT,
.probe = dp83640_probe,
.remove = dp83640_remove,
+ .soft_reset = dp83640_soft_reset,
.config_init = dp83640_config_init,
.config_aneg = genphy_config_aneg,
.read_status = genphy_read_status,
diff --git a/drivers/net/phy/marvell.c b/drivers/net/phy/marvell.c
index c60c147..5203523 100644
--- a/drivers/net/phy/marvell.c
+++ b/drivers/net/phy/marvell.c
@@ -1610,7 +1610,7 @@ static struct phy_driver marvell_drivers[] = {
.flags = PHY_HAS_INTERRUPT,
.probe = marvell_probe,
.config_init = &m88e1145_config_init,
- .config_aneg = &marvell_config_aneg,
+ .config_aneg = &m88e1101_config_aneg,
.read_status = &genphy_read_status,
.ack_interrupt = &marvell_ack_interrupt,
.config_intr = &marvell_config_intr,
diff --git a/drivers/net/phy/mdio-mux-bcm-iproc.c b/drivers/net/phy/mdio-mux-bcm-iproc.c
index 0a5f62e..bd9f9b9 100644
--- a/drivers/net/phy/mdio-mux-bcm-iproc.c
+++ b/drivers/net/phy/mdio-mux-bcm-iproc.c
@@ -22,7 +22,7 @@
#include <linux/mdio-mux.h>
#include <linux/delay.h>
-#define MDIO_PARAM_OFFSET 0x00
+#define MDIO_PARAM_OFFSET 0x23c
#define MDIO_PARAM_MIIM_CYCLE 29
#define MDIO_PARAM_INTERNAL_SEL 25
#define MDIO_PARAM_BUS_ID 22
@@ -30,20 +30,22 @@
#define MDIO_PARAM_PHY_ID 16
#define MDIO_PARAM_PHY_DATA 0
-#define MDIO_READ_OFFSET 0x04
+#define MDIO_READ_OFFSET 0x240
#define MDIO_READ_DATA_MASK 0xffff
-#define MDIO_ADDR_OFFSET 0x08
+#define MDIO_ADDR_OFFSET 0x244
-#define MDIO_CTRL_OFFSET 0x0C
+#define MDIO_CTRL_OFFSET 0x248
#define MDIO_CTRL_WRITE_OP 0x1
#define MDIO_CTRL_READ_OP 0x2
-#define MDIO_STAT_OFFSET 0x10
+#define MDIO_STAT_OFFSET 0x24c
#define MDIO_STAT_DONE 1
#define BUS_MAX_ADDR 32
#define EXT_BUS_START_ADDR 16
+#define MDIO_REG_ADDR_SPACE_SIZE 0x250
+
struct iproc_mdiomux_desc {
void *mux_handle;
void __iomem *base;
@@ -169,6 +171,14 @@ static int mdio_mux_iproc_probe(struct platform_device *pdev)
md->dev = &pdev->dev;
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
+ if (res->start & 0xfff) {
+ /* For backward compatibility in case the
+ * base address is specified with an offset.
+ */
+ dev_info(&pdev->dev, "fix base address in dt-blob\n");
+ res->start &= ~0xfff;
+ res->end = res->start + MDIO_REG_ADDR_SPACE_SIZE - 1;
+ }
md->base = devm_ioremap_resource(&pdev->dev, res);
if (IS_ERR(md->base)) {
dev_err(&pdev->dev, "failed to ioremap register\n");
@@ -218,7 +228,7 @@ static int mdio_mux_iproc_probe(struct platform_device *pdev)
static int mdio_mux_iproc_remove(struct platform_device *pdev)
{
- struct iproc_mdiomux_desc *md = dev_get_platdata(&pdev->dev);
+ struct iproc_mdiomux_desc *md = platform_get_drvdata(pdev);
mdio_mux_uninit(md->mux_handle);
mdiobus_unregister(md->mii_bus);
diff --git a/drivers/net/phy/micrel.c b/drivers/net/phy/micrel.c
index 2032a6d..707190d 100644
--- a/drivers/net/phy/micrel.c
+++ b/drivers/net/phy/micrel.c
@@ -801,9 +801,6 @@ static struct phy_driver ksphy_driver[] = {
.read_status = genphy_read_status,
.ack_interrupt = kszphy_ack_interrupt,
.config_intr = kszphy_config_intr,
- .get_sset_count = kszphy_get_sset_count,
- .get_strings = kszphy_get_strings,
- .get_stats = kszphy_get_stats,
.suspend = genphy_suspend,
.resume = genphy_resume,
}, {
@@ -948,9 +945,6 @@ static struct phy_driver ksphy_driver[] = {
.read_status = genphy_read_status,
.ack_interrupt = kszphy_ack_interrupt,
.config_intr = kszphy_config_intr,
- .get_sset_count = kszphy_get_sset_count,
- .get_strings = kszphy_get_strings,
- .get_stats = kszphy_get_stats,
.suspend = genphy_suspend,
.resume = genphy_resume,
}, {
@@ -960,6 +954,7 @@ static struct phy_driver ksphy_driver[] = {
.features = (PHY_GBIT_FEATURES | SUPPORTED_Pause),
.flags = PHY_HAS_MAGICANEG | PHY_HAS_INTERRUPT,
.driver_data = &ksz9021_type,
+ .probe = kszphy_probe,
.config_init = ksz9021_config_init,
.config_aneg = genphy_config_aneg,
.read_status = genphy_read_status,
@@ -979,6 +974,7 @@ static struct phy_driver ksphy_driver[] = {
.features = (PHY_GBIT_FEATURES | SUPPORTED_Pause),
.flags = PHY_HAS_MAGICANEG | PHY_HAS_INTERRUPT,
.driver_data = &ksz9021_type,
+ .probe = kszphy_probe,
.config_init = ksz9031_config_init,
.config_aneg = genphy_config_aneg,
.read_status = ksz9031_read_status,
@@ -998,9 +994,6 @@ static struct phy_driver ksphy_driver[] = {
.config_init = kszphy_config_init,
.config_aneg = ksz8873mll_config_aneg,
.read_status = ksz8873mll_read_status,
- .get_sset_count = kszphy_get_sset_count,
- .get_strings = kszphy_get_strings,
- .get_stats = kszphy_get_stats,
.suspend = genphy_suspend,
.resume = genphy_resume,
}, {
@@ -1012,9 +1005,6 @@ static struct phy_driver ksphy_driver[] = {
.config_init = kszphy_config_init,
.config_aneg = genphy_config_aneg,
.read_status = genphy_read_status,
- .get_sset_count = kszphy_get_sset_count,
- .get_strings = kszphy_get_strings,
- .get_stats = kszphy_get_stats,
.suspend = genphy_suspend,
.resume = genphy_resume,
}, {
@@ -1026,9 +1016,6 @@ static struct phy_driver ksphy_driver[] = {
.config_init = kszphy_config_init,
.config_aneg = ksz8873mll_config_aneg,
.read_status = ksz8873mll_read_status,
- .get_sset_count = kszphy_get_sset_count,
- .get_strings = kszphy_get_strings,
- .get_stats = kszphy_get_stats,
.suspend = genphy_suspend,
.resume = genphy_resume,
} };
diff --git a/drivers/net/phy/phy.c b/drivers/net/phy/phy.c
index 4d21764..5fde8e3 100644
--- a/drivers/net/phy/phy.c
+++ b/drivers/net/phy/phy.c
@@ -598,7 +598,7 @@ static int phy_start_aneg_priv(struct phy_device *phydev, bool sync)
* negotiation may already be done and aneg interrupt may not be
* generated.
*/
- if (phy_interrupt_is_valid(phydev) && (phydev->state == PHY_AN)) {
+ if (phydev->irq != PHY_POLL && phydev->state == PHY_AN) {
err = phy_aneg_done(phydev);
if (err > 0) {
trigger = true;
diff --git a/drivers/net/phy/phy_device.c b/drivers/net/phy/phy_device.c
index bf02f8e..5048a6d 100644
--- a/drivers/net/phy/phy_device.c
+++ b/drivers/net/phy/phy_device.c
@@ -163,11 +163,8 @@ static int mdio_bus_phy_restore(struct device *dev)
if (ret < 0)
return ret;
- /* The PHY needs to renegotiate. */
- phydev->link = 0;
- phydev->state = PHY_UP;
-
- phy_start_machine(phydev);
+ if (phydev->attached_dev && phydev->adjust_link)
+ phy_start_machine(phydev);
return 0;
}
@@ -1579,23 +1576,17 @@ static int gen10g_resume(struct phy_device *phydev)
static int __set_phy_supported(struct phy_device *phydev, u32 max_speed)
{
- /* The default values for phydev->supported are provided by the PHY
- * driver "features" member, we want to reset to sane defaults first
- * before supporting higher speeds.
- */
- phydev->supported &= PHY_DEFAULT_FEATURES;
-
switch (max_speed) {
- default:
- return -ENOTSUPP;
- case SPEED_1000:
- phydev->supported |= PHY_1000BT_FEATURES;
+ case SPEED_10:
+ phydev->supported &= ~PHY_100BT_FEATURES;
/* fall through */
case SPEED_100:
- phydev->supported |= PHY_100BT_FEATURES;
- /* fall through */
- case SPEED_10:
- phydev->supported |= PHY_10BT_FEATURES;
+ phydev->supported &= ~PHY_1000BT_FEATURES;
+ break;
+ case SPEED_1000:
+ break;
+ default:
+ return -ENOTSUPP;
}
return 0;
diff --git a/drivers/net/phy/xilinx_gmii2rgmii.c b/drivers/net/phy/xilinx_gmii2rgmii.c
index 2e5150b..7a14e81 100644
--- a/drivers/net/phy/xilinx_gmii2rgmii.c
+++ b/drivers/net/phy/xilinx_gmii2rgmii.c
@@ -40,8 +40,11 @@ static int xgmiitorgmii_read_status(struct phy_device *phydev)
{
struct gmii2rgmii *priv = phydev->priv;
u16 val = 0;
+ int err;
- priv->phy_drv->read_status(phydev);
+ err = priv->phy_drv->read_status(phydev);
+ if (err < 0)
+ return err;
val = mdiobus_read(phydev->mdio.bus, priv->addr, XILINX_GMII2RGMII_REG);
val &= ~XILINX_GMII2RGMII_SPEED_MASK;
@@ -81,6 +84,11 @@ static int xgmiitorgmii_probe(struct mdio_device *mdiodev)
return -EPROBE_DEFER;
}
+ if (!priv->phy_dev->drv) {
+ dev_info(dev, "Attached phy not ready\n");
+ return -EPROBE_DEFER;
+ }
+
priv->addr = mdiodev->addr;
priv->phy_drv = priv->phy_dev->drv;
memcpy(&priv->conv_phy_drv, priv->phy_dev->drv,
diff --git a/drivers/net/ppp/pppoe.c b/drivers/net/ppp/pppoe.c
index dc36c2e..fa2c7bd 100644
--- a/drivers/net/ppp/pppoe.c
+++ b/drivers/net/ppp/pppoe.c
@@ -620,6 +620,10 @@ static int pppoe_connect(struct socket *sock, struct sockaddr *uservaddr,
lock_sock(sk);
error = -EINVAL;
+
+ if (sockaddr_len != sizeof(struct sockaddr_pppox))
+ goto end;
+
if (sp->sa_protocol != PX_PROTO_OE)
goto end;
diff --git a/drivers/net/rionet.c b/drivers/net/rionet.c
index a31f461..2c2604e 100644
--- a/drivers/net/rionet.c
+++ b/drivers/net/rionet.c
@@ -216,9 +216,9 @@ static int rionet_start_xmit(struct sk_buff *skb, struct net_device *ndev)
* it just report sending a packet to the target
* (without actual packet transfer).
*/
- dev_kfree_skb_any(skb);
ndev->stats.tx_packets++;
ndev->stats.tx_bytes += skb->len;
+ dev_kfree_skb_any(skb);
}
}
diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c
index 8673ef3..94b05dd 100644
--- a/drivers/net/team/team.c
+++ b/drivers/net/team/team.c
@@ -261,6 +261,17 @@ static void __team_option_inst_mark_removed_port(struct team *team,
}
}
+static bool __team_option_inst_tmp_find(const struct list_head *opts,
+ const struct team_option_inst *needle)
+{
+ struct team_option_inst *opt_inst;
+
+ list_for_each_entry(opt_inst, opts, tmp_list)
+ if (opt_inst == needle)
+ return true;
+ return false;
+}
+
static int __team_options_register(struct team *team,
const struct team_option *option,
size_t option_count)
@@ -978,8 +989,6 @@ static void team_port_disable(struct team *team,
team->en_port_count--;
team_queue_override_port_del(team, port);
team_adjust_ops(team);
- team_notify_peers(team);
- team_mcast_rejoin(team);
team_lower_state_changed(port);
}
@@ -993,7 +1002,8 @@ static void team_port_disable(struct team *team,
static void ___team_compute_features(struct team *team)
{
struct team_port *port;
- u32 vlan_features = TEAM_VLAN_FEATURES & NETIF_F_ALL_FOR_ALL;
+ netdev_features_t vlan_features = TEAM_VLAN_FEATURES &
+ NETIF_F_ALL_FOR_ALL;
netdev_features_t enc_features = TEAM_ENC_FEATURES;
unsigned short max_hard_header_len = ETH_HLEN;
unsigned int dst_release_flag = IFF_XMIT_DST_RELEASE |
@@ -1067,14 +1077,11 @@ static void team_port_leave(struct team *team, struct team_port *port)
}
#ifdef CONFIG_NET_POLL_CONTROLLER
-static int team_port_enable_netpoll(struct team *team, struct team_port *port)
+static int __team_port_enable_netpoll(struct team_port *port)
{
struct netpoll *np;
int err;
- if (!team->dev->npinfo)
- return 0;
-
np = kzalloc(sizeof(*np), GFP_KERNEL);
if (!np)
return -ENOMEM;
@@ -1088,6 +1095,14 @@ static int team_port_enable_netpoll(struct team *team, struct team_port *port)
return err;
}
+static int team_port_enable_netpoll(struct team_port *port)
+{
+ if (!port->team->dev->npinfo)
+ return 0;
+
+ return __team_port_enable_netpoll(port);
+}
+
static void team_port_disable_netpoll(struct team_port *port)
{
struct netpoll *np = port->np;
@@ -1102,7 +1117,7 @@ static void team_port_disable_netpoll(struct team_port *port)
kfree(np);
}
#else
-static int team_port_enable_netpoll(struct team *team, struct team_port *port)
+static int team_port_enable_netpoll(struct team_port *port)
{
return 0;
}
@@ -1154,6 +1169,11 @@ static int team_port_add(struct team *team, struct net_device *port_dev)
return -EBUSY;
}
+ if (dev == port_dev) {
+ netdev_err(dev, "Cannot enslave team device to itself\n");
+ return -EINVAL;
+ }
+
if (port_dev->features & NETIF_F_VLAN_CHALLENGED &&
vlan_uses_dev(dev)) {
netdev_err(dev, "Device %s is VLAN challenged and team device has VLAN set up\n",
@@ -1210,7 +1230,7 @@ static int team_port_add(struct team *team, struct net_device *port_dev)
goto err_vids_add;
}
- err = team_port_enable_netpoll(team, port);
+ err = team_port_enable_netpoll(port);
if (err) {
netdev_err(dev, "Failed to enable netpoll on device %s\n",
portname);
@@ -1908,7 +1928,7 @@ static int team_netpoll_setup(struct net_device *dev,
mutex_lock(&team->lock);
list_for_each_entry(port, &team->port_list, list) {
- err = team_port_enable_netpoll(team, port);
+ err = __team_port_enable_netpoll(port);
if (err) {
__team_netpoll_cleanup(team);
break;
@@ -2569,6 +2589,14 @@ static int team_nl_cmd_options_set(struct sk_buff *skb, struct genl_info *info)
if (err)
goto team_put;
opt_inst->changed = true;
+
+ /* dumb/evil user-space can send us duplicate opt,
+ * keep only the last one
+ */
+ if (__team_option_inst_tmp_find(&opt_inst_list,
+ opt_inst))
+ continue;
+
list_add(&opt_inst->tmp_list, &opt_inst_list);
}
if (!opt_found) {
diff --git a/drivers/net/tun.c b/drivers/net/tun.c
index 9babe04..cec816f 100644
--- a/drivers/net/tun.c
+++ b/drivers/net/tun.c
@@ -1574,7 +1574,9 @@ static void tun_setup(struct net_device *dev)
*/
static int tun_validate(struct nlattr *tb[], struct nlattr *data[])
{
- return -EINVAL;
+ /* NL_SET_ERR_MSG(extack,
+ "tun/tap creation via rtnetlink is not supported."); */
+ return -EOPNOTSUPP;
}
static struct rtnl_link_ops tun_link_ops __read_mostly = {
diff --git a/drivers/net/usb/asix_common.c b/drivers/net/usb/asix_common.c
index 125cff5..3dbb0646 100644
--- a/drivers/net/usb/asix_common.c
+++ b/drivers/net/usb/asix_common.c
@@ -575,6 +575,9 @@ int asix_set_wol(struct net_device *net, struct ethtool_wolinfo *wolinfo)
struct usbnet *dev = netdev_priv(net);
u8 opt = 0;
+ if (wolinfo->wolopts & ~(WAKE_PHY | WAKE_MAGIC))
+ return -EINVAL;
+
if (wolinfo->wolopts & WAKE_PHY)
opt |= AX_MONITOR_LINK;
if (wolinfo->wolopts & WAKE_MAGIC)
diff --git a/drivers/net/usb/asix_devices.c b/drivers/net/usb/asix_devices.c
index 32e9ec8..5be6b67 100644
--- a/drivers/net/usb/asix_devices.c
+++ b/drivers/net/usb/asix_devices.c
@@ -640,10 +640,12 @@ static void ax88772_restore_phy(struct usbnet *dev)
priv->presvd_phy_advertise);
/* Restore BMCR */
+ if (priv->presvd_phy_bmcr & BMCR_ANENABLE)
+ priv->presvd_phy_bmcr |= BMCR_ANRESTART;
+
asix_mdio_write_nopm(dev->net, dev->mii.phy_id, MII_BMCR,
priv->presvd_phy_bmcr);
- mii_nway_restart(&dev->mii);
priv->presvd_phy_advertise = 0;
priv->presvd_phy_bmcr = 0;
}
diff --git a/drivers/net/usb/ax88179_178a.c b/drivers/net/usb/ax88179_178a.c
index 8a6675d..559af8e 100644
--- a/drivers/net/usb/ax88179_178a.c
+++ b/drivers/net/usb/ax88179_178a.c
@@ -566,6 +566,9 @@ ax88179_set_wol(struct net_device *net, struct ethtool_wolinfo *wolinfo)
struct usbnet *dev = netdev_priv(net);
u8 opt = 0;
+ if (wolinfo->wolopts & ~(WAKE_PHY | WAKE_MAGIC))
+ return -EINVAL;
+
if (wolinfo->wolopts & WAKE_PHY)
opt |= AX_MONITOR_MODE_RWLC;
if (wolinfo->wolopts & WAKE_MAGIC)
diff --git a/drivers/net/usb/cdc_ether.c b/drivers/net/usb/cdc_ether.c
index 4fb4686..99424c8 100644
--- a/drivers/net/usb/cdc_ether.c
+++ b/drivers/net/usb/cdc_ether.c
@@ -530,6 +530,7 @@ static const struct driver_info wwan_info = {
#define REALTEK_VENDOR_ID 0x0bda
#define SAMSUNG_VENDOR_ID 0x04e8
#define LENOVO_VENDOR_ID 0x17ef
+#define LINKSYS_VENDOR_ID 0x13b1
#define NVIDIA_VENDOR_ID 0x0955
#define HP_VENDOR_ID 0x03f0
@@ -719,6 +720,15 @@ static const struct usb_device_id products[] = {
.driver_info = 0,
},
+#if IS_ENABLED(CONFIG_USB_RTL8152)
+/* Linksys USB3GIGV1 Ethernet Adapter */
+{
+ USB_DEVICE_AND_INTERFACE_INFO(LINKSYS_VENDOR_ID, 0x0041, USB_CLASS_COMM,
+ USB_CDC_SUBCLASS_ETHERNET, USB_CDC_PROTO_NONE),
+ .driver_info = 0,
+},
+#endif
+
/* Lenovo Thinkpad USB 3.0 Ethernet Adapters (based on Realtek RTL8153) */
{
USB_DEVICE_AND_INTERFACE_INFO(LENOVO_VENDOR_ID, 0x7205, USB_CLASS_COMM,
diff --git a/drivers/net/usb/cdc_mbim.c b/drivers/net/usb/cdc_mbim.c
index 3a98f37..4c8baba 100644
--- a/drivers/net/usb/cdc_mbim.c
+++ b/drivers/net/usb/cdc_mbim.c
@@ -608,7 +608,7 @@ static const struct driver_info cdc_mbim_info_ndp_to_end = {
*/
static const struct driver_info cdc_mbim_info_avoid_altsetting_toggle = {
.description = "CDC MBIM",
- .flags = FLAG_NO_SETINT | FLAG_MULTI_PACKET | FLAG_WWAN,
+ .flags = FLAG_NO_SETINT | FLAG_MULTI_PACKET | FLAG_WWAN | FLAG_SEND_ZLP,
.bind = cdc_mbim_bind,
.unbind = cdc_mbim_unbind,
.manage_power = cdc_mbim_manage_power,
diff --git a/drivers/net/usb/cdc_ncm.c b/drivers/net/usb/cdc_ncm.c
index feb61ea..7b15867 100644
--- a/drivers/net/usb/cdc_ncm.c
+++ b/drivers/net/usb/cdc_ncm.c
@@ -772,7 +772,7 @@ int cdc_ncm_bind_common(struct usbnet *dev, struct usb_interface *intf, u8 data_
int err;
u8 iface_no;
struct usb_cdc_parsed_header hdr;
- u16 curr_ntb_format;
+ __le16 curr_ntb_format;
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
if (!ctx)
@@ -890,7 +890,7 @@ int cdc_ncm_bind_common(struct usbnet *dev, struct usb_interface *intf, u8 data_
goto error2;
}
- if (curr_ntb_format == USB_CDC_NCM_NTB32_FORMAT) {
+ if (curr_ntb_format == cpu_to_le16(USB_CDC_NCM_NTB32_FORMAT)) {
dev_info(&intf->dev, "resetting NTB format to 16-bit");
err = usbnet_write_cmd(dev, USB_CDC_SET_NTB_FORMAT,
USB_TYPE_CLASS | USB_DIR_OUT
@@ -1124,7 +1124,7 @@ cdc_ncm_fill_tx_frame(struct usbnet *dev, struct sk_buff *skb, __le32 sign)
* accordingly. Otherwise, we should check here.
*/
if (ctx->drvflags & CDC_NCM_FLAG_NDP_TO_END)
- delayed_ndp_size = ctx->max_ndp_size;
+ delayed_ndp_size = ALIGN(ctx->max_ndp_size, ctx->tx_ndp_modulus);
else
delayed_ndp_size = 0;
@@ -1257,7 +1257,7 @@ cdc_ncm_fill_tx_frame(struct usbnet *dev, struct sk_buff *skb, __le32 sign)
/* If requested, put NDP at end of frame. */
if (ctx->drvflags & CDC_NCM_FLAG_NDP_TO_END) {
nth16 = (struct usb_cdc_ncm_nth16 *)skb_out->data;
- cdc_ncm_align_tail(skb_out, ctx->tx_ndp_modulus, 0, ctx->tx_max);
+ cdc_ncm_align_tail(skb_out, ctx->tx_ndp_modulus, 0, ctx->tx_max - ctx->max_ndp_size);
nth16->wNdpIndex = cpu_to_le16(skb_out->len);
memcpy(skb_put(skb_out, ctx->max_ndp_size), ctx->delayed_ndp16, ctx->max_ndp_size);
diff --git a/drivers/net/usb/hso.c b/drivers/net/usb/hso.c
index e7b5163..66ae647 100644
--- a/drivers/net/usb/hso.c
+++ b/drivers/net/usb/hso.c
@@ -2808,6 +2808,12 @@ static int hso_get_config_data(struct usb_interface *interface)
return -EIO;
}
+ /* check if we have a valid interface */
+ if (if_num > 16) {
+ kfree(config_data);
+ return -EINVAL;
+ }
+
switch (config_data[if_num]) {
case 0x0:
result = 0;
@@ -2878,10 +2884,18 @@ static int hso_probe(struct usb_interface *interface,
/* Get the interface/port specification from either driver_info or from
* the device itself */
- if (id->driver_info)
+ if (id->driver_info) {
+ /* if_num is controlled by the device, driver_info is a 0 terminated
+ * array. Make sure, the access is in bounds! */
+ for (i = 0; i <= if_num; ++i)
+ if (((u32 *)(id->driver_info))[i] == 0)
+ goto exit;
port_spec = ((u32 *)(id->driver_info))[if_num];
- else
+ } else {
port_spec = hso_get_config_data(interface);
+ if (port_spec < 0)
+ goto exit;
+ }
/* Check if we need to switch to alt interfaces prior to port
* configuration */
diff --git a/drivers/net/usb/ipheth.c b/drivers/net/usb/ipheth.c
index 76465b1..f1f8227 100644
--- a/drivers/net/usb/ipheth.c
+++ b/drivers/net/usb/ipheth.c
@@ -140,7 +140,6 @@ struct ipheth_device {
struct usb_device *udev;
struct usb_interface *intf;
struct net_device *net;
- struct sk_buff *tx_skb;
struct urb *tx_urb;
struct urb *rx_urb;
unsigned char *tx_buf;
@@ -229,6 +228,7 @@ static void ipheth_rcvbulk_callback(struct urb *urb)
case -ENOENT:
case -ECONNRESET:
case -ESHUTDOWN:
+ case -EPROTO:
return;
case 0:
break;
@@ -280,7 +280,6 @@ static void ipheth_sndbulk_callback(struct urb *urb)
dev_err(&dev->intf->dev, "%s: urb status: %d\n",
__func__, status);
- dev_kfree_skb_irq(dev->tx_skb);
netif_wake_queue(dev->net);
}
@@ -410,7 +409,7 @@ static int ipheth_tx(struct sk_buff *skb, struct net_device *net)
if (skb->len > IPHETH_BUF_SIZE) {
WARN(1, "%s: skb too large: %d bytes\n", __func__, skb->len);
dev->net->stats.tx_dropped++;
- dev_kfree_skb_irq(skb);
+ dev_kfree_skb_any(skb);
return NETDEV_TX_OK;
}
@@ -430,12 +429,11 @@ static int ipheth_tx(struct sk_buff *skb, struct net_device *net)
dev_err(&dev->intf->dev, "%s: usb_submit_urb: %d\n",
__func__, retval);
dev->net->stats.tx_errors++;
- dev_kfree_skb_irq(skb);
+ dev_kfree_skb_any(skb);
} else {
- dev->tx_skb = skb;
-
dev->net->stats.tx_packets++;
dev->net->stats.tx_bytes += skb->len;
+ dev_consume_skb_any(skb);
netif_stop_queue(net);
}
diff --git a/drivers/net/usb/lan78xx.c b/drivers/net/usb/lan78xx.c
index f5a9667..e143a7f 100644
--- a/drivers/net/usb/lan78xx.c
+++ b/drivers/net/usb/lan78xx.c
@@ -31,6 +31,7 @@
#include <linux/mdio.h>
#include <net/ip6_checksum.h>
#include <linux/microchipphy.h>
+#include <linux/of_net.h>
#include "lan78xx.h"
#define DRIVER_AUTHOR "WOOJUNG HUH <woojung.huh@microchip.com>"
@@ -1170,6 +1171,8 @@ static int lan78xx_link_reset(struct lan78xx_net *dev)
mod_timer(&dev->stat_monitor,
jiffies + STAT_UPDATE_TIMER);
}
+
+ tasklet_schedule(&dev->bh);
}
return ret;
@@ -1309,19 +1312,10 @@ static int lan78xx_set_wol(struct net_device *netdev,
if (ret < 0)
return ret;
- pdata->wol = 0;
- if (wol->wolopts & WAKE_UCAST)
- pdata->wol |= WAKE_UCAST;
- if (wol->wolopts & WAKE_MCAST)
- pdata->wol |= WAKE_MCAST;
- if (wol->wolopts & WAKE_BCAST)
- pdata->wol |= WAKE_BCAST;
- if (wol->wolopts & WAKE_MAGIC)
- pdata->wol |= WAKE_MAGIC;
- if (wol->wolopts & WAKE_PHY)
- pdata->wol |= WAKE_PHY;
- if (wol->wolopts & WAKE_ARP)
- pdata->wol |= WAKE_ARP;
+ if (wol->wolopts & ~WAKE_ALL)
+ return -EINVAL;
+
+ pdata->wol = wol->wolopts;
device_set_wakeup_enable(&dev->udev->dev, (bool)wol->wolopts);
@@ -1651,34 +1645,31 @@ static void lan78xx_init_mac_address(struct lan78xx_net *dev)
addr[5] = (addr_hi >> 8) & 0xFF;
if (!is_valid_ether_addr(addr)) {
- /* reading mac address from EEPROM or OTP */
- if ((lan78xx_read_eeprom(dev, EEPROM_MAC_OFFSET, ETH_ALEN,
- addr) == 0) ||
- (lan78xx_read_otp(dev, EEPROM_MAC_OFFSET, ETH_ALEN,
- addr) == 0)) {
- if (is_valid_ether_addr(addr)) {
- /* eeprom values are valid so use them */
- netif_dbg(dev, ifup, dev->net,
- "MAC address read from EEPROM");
- } else {
- /* generate random MAC */
- random_ether_addr(addr);
- netif_dbg(dev, ifup, dev->net,
- "MAC address set to random addr");
- }
-
- addr_lo = addr[0] | (addr[1] << 8) |
- (addr[2] << 16) | (addr[3] << 24);
- addr_hi = addr[4] | (addr[5] << 8);
-
- ret = lan78xx_write_reg(dev, RX_ADDRL, addr_lo);
- ret = lan78xx_write_reg(dev, RX_ADDRH, addr_hi);
+ if (!eth_platform_get_mac_address(&dev->udev->dev, addr)) {
+ /* valid address present in Device Tree */
+ netif_dbg(dev, ifup, dev->net,
+ "MAC address read from Device Tree");
+ } else if (((lan78xx_read_eeprom(dev, EEPROM_MAC_OFFSET,
+ ETH_ALEN, addr) == 0) ||
+ (lan78xx_read_otp(dev, EEPROM_MAC_OFFSET,
+ ETH_ALEN, addr) == 0)) &&
+ is_valid_ether_addr(addr)) {
+ /* eeprom values are valid so use them */
+ netif_dbg(dev, ifup, dev->net,
+ "MAC address read from EEPROM");
} else {
/* generate random MAC */
random_ether_addr(addr);
netif_dbg(dev, ifup, dev->net,
"MAC address set to random addr");
}
+
+ addr_lo = addr[0] | (addr[1] << 8) |
+ (addr[2] << 16) | (addr[3] << 24);
+ addr_hi = addr[4] | (addr[5] << 8);
+
+ ret = lan78xx_write_reg(dev, RX_ADDRL, addr_lo);
+ ret = lan78xx_write_reg(dev, RX_ADDRH, addr_hi);
}
ret = lan78xx_write_reg(dev, MAF_LO(0), addr_lo);
@@ -2023,6 +2014,10 @@ static int lan78xx_set_mac_addr(struct net_device *netdev, void *p)
ret = lan78xx_write_reg(dev, RX_ADDRL, addr_lo);
ret = lan78xx_write_reg(dev, RX_ADDRH, addr_hi);
+ /* Added to support MAC address changes */
+ ret = lan78xx_write_reg(dev, MAF_LO(0), addr_lo);
+ ret = lan78xx_write_reg(dev, MAF_HI(0), addr_hi | MAF_HI_VALID_);
+
return 0;
}
@@ -2964,6 +2959,7 @@ static void lan78xx_tx_bh(struct lan78xx_net *dev)
pkt_cnt = 0;
count = 0;
length = 0;
+ spin_lock_irqsave(&tqp->lock, flags);
for (skb = tqp->next; pkt_cnt < tqp->qlen; skb = skb->next) {
if (skb_is_gso(skb)) {
if (pkt_cnt) {
@@ -2972,7 +2968,8 @@ static void lan78xx_tx_bh(struct lan78xx_net *dev)
}
count = 1;
length = skb->len - TX_OVERHEAD;
- skb2 = skb_dequeue(tqp);
+ __skb_unlink(skb, tqp);
+ spin_unlock_irqrestore(&tqp->lock, flags);
goto gso_skb;
}
@@ -2981,6 +2978,7 @@ static void lan78xx_tx_bh(struct lan78xx_net *dev)
skb_totallen = skb->len + roundup(skb_totallen, sizeof(u32));
pkt_cnt++;
}
+ spin_unlock_irqrestore(&tqp->lock, flags);
/* copy to a single skb */
skb = alloc_skb(skb_totallen, GFP_ATOMIC);
diff --git a/drivers/net/usb/qmi_wwan.c b/drivers/net/usb/qmi_wwan.c
index 973e90f..134eb18 100644
--- a/drivers/net/usb/qmi_wwan.c
+++ b/drivers/net/usb/qmi_wwan.c
@@ -803,12 +803,17 @@ static const struct usb_device_id products[] = {
{QMI_FIXED_INTF(0x05c6, 0x9080, 8)},
{QMI_FIXED_INTF(0x05c6, 0x9083, 3)},
{QMI_FIXED_INTF(0x05c6, 0x9084, 4)},
+ {QMI_FIXED_INTF(0x05c6, 0x90b2, 3)}, /* ublox R410M */
{QMI_FIXED_INTF(0x05c6, 0x920d, 0)},
{QMI_FIXED_INTF(0x05c6, 0x920d, 5)},
{QMI_QUIRK_SET_DTR(0x05c6, 0x9625, 4)}, /* YUGA CLM920-NC5 */
{QMI_FIXED_INTF(0x0846, 0x68a2, 8)},
+ {QMI_FIXED_INTF(0x0846, 0x68d3, 8)}, /* Netgear Aircard 779S */
{QMI_FIXED_INTF(0x12d1, 0x140c, 1)}, /* Huawei E173 */
{QMI_FIXED_INTF(0x12d1, 0x14ac, 1)}, /* Huawei E1820 */
+ {QMI_FIXED_INTF(0x1435, 0xd181, 3)}, /* Wistron NeWeb D18Q1 */
+ {QMI_FIXED_INTF(0x1435, 0xd181, 4)}, /* Wistron NeWeb D18Q1 */
+ {QMI_FIXED_INTF(0x1435, 0xd181, 5)}, /* Wistron NeWeb D18Q1 */
{QMI_FIXED_INTF(0x16d8, 0x6003, 0)}, /* CMOTech 6003 */
{QMI_FIXED_INTF(0x16d8, 0x6007, 0)}, /* CMOTech CHE-628S */
{QMI_FIXED_INTF(0x16d8, 0x6008, 0)}, /* CMOTech CMU-301 */
@@ -885,6 +890,7 @@ static const struct usb_device_id products[] = {
{QMI_FIXED_INTF(0x19d2, 0x2002, 4)}, /* ZTE (Vodafone) K3765-Z */
{QMI_FIXED_INTF(0x2001, 0x7e19, 4)}, /* D-Link DWM-221 B1 */
{QMI_FIXED_INTF(0x2001, 0x7e35, 4)}, /* D-Link DWM-222 */
+ {QMI_FIXED_INTF(0x2020, 0x2033, 4)}, /* BroadMobi BM806U */
{QMI_FIXED_INTF(0x0f3d, 0x68a2, 8)}, /* Sierra Wireless MC7700 */
{QMI_FIXED_INTF(0x114f, 0x68a2, 8)}, /* Sierra Wireless MC7750 */
{QMI_FIXED_INTF(0x1199, 0x68a2, 8)}, /* Sierra Wireless MC7710 in QMI mode */
@@ -928,6 +934,7 @@ static const struct usb_device_id products[] = {
{QMI_FIXED_INTF(0x0b3c, 0xc00b, 4)}, /* Olivetti Olicard 500 */
{QMI_FIXED_INTF(0x1e2d, 0x0060, 4)}, /* Cinterion PLxx */
{QMI_FIXED_INTF(0x1e2d, 0x0053, 4)}, /* Cinterion PHxx,PXxx */
+ {QMI_FIXED_INTF(0x1e2d, 0x0063, 10)}, /* Cinterion ALASxx (1 RmNet) */
{QMI_FIXED_INTF(0x1e2d, 0x0082, 4)}, /* Cinterion PHxx,PXxx (2 RmNet) */
{QMI_FIXED_INTF(0x1e2d, 0x0082, 5)}, /* Cinterion PHxx,PXxx (2 RmNet) */
{QMI_FIXED_INTF(0x1e2d, 0x0083, 4)}, /* Cinterion PHxx,PXxx (1 RmNet + USB Audio)*/
@@ -940,11 +947,14 @@ static const struct usb_device_id products[] = {
{QMI_FIXED_INTF(0x413c, 0x81b3, 8)}, /* Dell Wireless 5809e Gobi(TM) 4G LTE Mobile Broadband Card (rev3) */
{QMI_FIXED_INTF(0x413c, 0x81b6, 8)}, /* Dell Wireless 5811e */
{QMI_FIXED_INTF(0x413c, 0x81b6, 10)}, /* Dell Wireless 5811e */
+ {QMI_FIXED_INTF(0x413c, 0x81d7, 0)}, /* Dell Wireless 5821e */
{QMI_FIXED_INTF(0x03f0, 0x4e1d, 8)}, /* HP lt4111 LTE/EV-DO/HSPA+ Gobi 4G Module */
+ {QMI_FIXED_INTF(0x03f0, 0x9d1d, 1)}, /* HP lt4120 Snapdragon X5 LTE */
{QMI_FIXED_INTF(0x22de, 0x9061, 3)}, /* WeTelecom WPD-600N */
- {QMI_FIXED_INTF(0x1e0e, 0x9001, 5)}, /* SIMCom 7230E */
+ {QMI_QUIRK_SET_DTR(0x1e0e, 0x9001, 5)}, /* SIMCom 7100E, 7230E, 7600E ++ */
{QMI_QUIRK_SET_DTR(0x2c7c, 0x0125, 4)}, /* Quectel EC25, EC20 R2.0 Mini PCIe */
{QMI_QUIRK_SET_DTR(0x2c7c, 0x0121, 4)}, /* Quectel EC21 Mini PCIe */
+ {QMI_QUIRK_SET_DTR(0x2c7c, 0x0191, 4)}, /* Quectel EG91 */
{QMI_FIXED_INTF(0x2c7c, 0x0296, 4)}, /* Quectel BG96 */
{QMI_QUIRK_SET_DTR(0x2c7c, 0x0306, 4)}, /* Quectel EP06 Mini PCIe */
@@ -1038,6 +1048,18 @@ static int qmi_wwan_probe(struct usb_interface *intf,
id->driver_info = (unsigned long)&qmi_wwan_info;
}
+ /* There are devices where the same interface number can be
+ * configured as different functions. We should only bind to
+ * vendor specific functions when matching on interface number
+ */
+ if (id->match_flags & USB_DEVICE_ID_MATCH_INT_NUMBER &&
+ desc->bInterfaceClass != USB_CLASS_VENDOR_SPEC) {
+ dev_dbg(&intf->dev,
+ "Rejecting interface number match for class %02x\n",
+ desc->bInterfaceClass);
+ return -ENODEV;
+ }
+
/* Quectel EC20 quirk where we've QMI on interface 4 instead of 0 */
if (quectel_ec20_detected(intf) && desc->bInterfaceNumber == 0) {
dev_dbg(&intf->dev, "Quectel EC20 quirk, skipping interface 0\n");
diff --git a/drivers/net/usb/r8152.c b/drivers/net/usb/r8152.c
index b2d7c7e..02e2956 100644
--- a/drivers/net/usb/r8152.c
+++ b/drivers/net/usb/r8152.c
@@ -519,6 +519,7 @@ enum rtl8152_flags {
#define VENDOR_ID_REALTEK 0x0bda
#define VENDOR_ID_SAMSUNG 0x04e8
#define VENDOR_ID_LENOVO 0x17ef
+#define VENDOR_ID_LINKSYS 0x13b1
#define VENDOR_ID_NVIDIA 0x0955
#define MCU_TYPE_PLA 0x0100
@@ -1692,7 +1693,7 @@ static int r8152_tx_agg_fill(struct r8152 *tp, struct tx_agg *agg)
tx_data += len;
agg->skb_len += len;
- agg->skb_num++;
+ agg->skb_num += skb_shinfo(skb)->gso_segs ?: 1;
dev_kfree_skb_any(skb);
@@ -3326,7 +3327,8 @@ static int rtl8152_close(struct net_device *netdev)
#ifdef CONFIG_PM_SLEEP
unregister_pm_notifier(&tp->pm_notifier);
#endif
- napi_disable(&tp->napi);
+ if (!test_bit(RTL8152_UNPLUG, &tp->flags))
+ napi_disable(&tp->napi);
clear_bit(WORK_ENABLE, &tp->flags);
usb_kill_urb(tp->intr_urb);
cancel_delayed_work_sync(&tp->schedule);
@@ -3774,6 +3776,9 @@ static int rtl8152_set_wol(struct net_device *dev, struct ethtool_wolinfo *wol)
if (!rtl_can_wakeup(tp))
return -EOPNOTSUPP;
+ if (wol->wolopts & ~WAKE_ANY)
+ return -EINVAL;
+
ret = usb_autopm_get_interface(tp->intf);
if (ret < 0)
goto out_set_wol;
@@ -4506,6 +4511,7 @@ static struct usb_device_id rtl8152_table[] = {
{REALTEK_USB_DEVICE(VENDOR_ID_SAMSUNG, 0xa101)},
{REALTEK_USB_DEVICE(VENDOR_ID_LENOVO, 0x7205)},
{REALTEK_USB_DEVICE(VENDOR_ID_LENOVO, 0x304f)},
+ {REALTEK_USB_DEVICE(VENDOR_ID_LINKSYS, 0x0041)},
{REALTEK_USB_DEVICE(VENDOR_ID_NVIDIA, 0x09ff)},
{}
};
diff --git a/drivers/net/usb/rtl8150.c b/drivers/net/usb/rtl8150.c
index dc4f7ea..9504800 100644
--- a/drivers/net/usb/rtl8150.c
+++ b/drivers/net/usb/rtl8150.c
@@ -681,7 +681,7 @@ static void rtl8150_set_multicast(struct net_device *netdev)
(netdev->flags & IFF_ALLMULTI)) {
rx_creg &= 0xfffe;
rx_creg |= 0x0002;
- dev_info(&netdev->dev, "%s: allmulti set\n", netdev->name);
+ dev_dbg(&netdev->dev, "%s: allmulti set\n", netdev->name);
} else {
/* ~RX_MULTICAST, ~RX_PROMISCUOUS */
rx_creg &= 0x00fc;
diff --git a/drivers/net/usb/smsc75xx.c b/drivers/net/usb/smsc75xx.c
index 4cb9b11..977d9c7 100644
--- a/drivers/net/usb/smsc75xx.c
+++ b/drivers/net/usb/smsc75xx.c
@@ -82,6 +82,9 @@ static bool turbo_mode = true;
module_param(turbo_mode, bool, 0644);
MODULE_PARM_DESC(turbo_mode, "Enable multiple frames per Rx transaction");
+static int smsc75xx_link_ok_nopm(struct usbnet *dev);
+static int smsc75xx_phy_gig_workaround(struct usbnet *dev);
+
static int __must_check __smsc75xx_read_reg(struct usbnet *dev, u32 index,
u32 *data, int in_pm)
{
@@ -728,6 +731,9 @@ static int smsc75xx_ethtool_set_wol(struct net_device *net,
struct smsc75xx_priv *pdata = (struct smsc75xx_priv *)(dev->data[0]);
int ret;
+ if (wolinfo->wolopts & ~SUPPORTED_WAKE)
+ return -EINVAL;
+
pdata->wolopts = wolinfo->wolopts & SUPPORTED_WAKE;
ret = device_set_wakeup_enable(&dev->udev->dev, pdata->wolopts);
@@ -852,6 +858,9 @@ static int smsc75xx_phy_initialize(struct usbnet *dev)
return -EIO;
}
+ /* phy workaround for gig link */
+ smsc75xx_phy_gig_workaround(dev);
+
smsc75xx_mdio_write(dev->net, dev->mii.phy_id, MII_ADVERTISE,
ADVERTISE_ALL | ADVERTISE_CSMA | ADVERTISE_PAUSE_CAP |
ADVERTISE_PAUSE_ASYM);
@@ -957,10 +966,11 @@ static int smsc75xx_set_features(struct net_device *netdev,
/* it's racing here! */
ret = smsc75xx_write_reg(dev, RFE_CTL, pdata->rfe_ctl);
- if (ret < 0)
+ if (ret < 0) {
netdev_warn(dev->net, "Error writing RFE_CTL\n");
-
- return ret;
+ return ret;
+ }
+ return 0;
}
static int smsc75xx_wait_ready(struct usbnet *dev, int in_pm)
@@ -989,6 +999,62 @@ static int smsc75xx_wait_ready(struct usbnet *dev, int in_pm)
return -EIO;
}
+static int smsc75xx_phy_gig_workaround(struct usbnet *dev)
+{
+ struct mii_if_info *mii = &dev->mii;
+ int ret = 0, timeout = 0;
+ u32 buf, link_up = 0;
+
+ /* Set the phy in Gig loopback */
+ smsc75xx_mdio_write(dev->net, mii->phy_id, MII_BMCR, 0x4040);
+
+ /* Wait for the link up */
+ do {
+ link_up = smsc75xx_link_ok_nopm(dev);
+ usleep_range(10000, 20000);
+ timeout++;
+ } while ((!link_up) && (timeout < 1000));
+
+ if (timeout >= 1000) {
+ netdev_warn(dev->net, "Timeout waiting for PHY link up\n");
+ return -EIO;
+ }
+
+ /* phy reset */
+ ret = smsc75xx_read_reg(dev, PMT_CTL, &buf);
+ if (ret < 0) {
+ netdev_warn(dev->net, "Failed to read PMT_CTL: %d\n", ret);
+ return ret;
+ }
+
+ buf |= PMT_CTL_PHY_RST;
+
+ ret = smsc75xx_write_reg(dev, PMT_CTL, buf);
+ if (ret < 0) {
+ netdev_warn(dev->net, "Failed to write PMT_CTL: %d\n", ret);
+ return ret;
+ }
+
+ timeout = 0;
+ do {
+ usleep_range(10000, 20000);
+ ret = smsc75xx_read_reg(dev, PMT_CTL, &buf);
+ if (ret < 0) {
+ netdev_warn(dev->net, "Failed to read PMT_CTL: %d\n",
+ ret);
+ return ret;
+ }
+ timeout++;
+ } while ((buf & PMT_CTL_PHY_RST) && (timeout < 100));
+
+ if (timeout >= 100) {
+ netdev_warn(dev->net, "timeout waiting for PHY Reset\n");
+ return -EIO;
+ }
+
+ return 0;
+}
+
static int smsc75xx_reset(struct usbnet *dev)
{
struct smsc75xx_priv *pdata = (struct smsc75xx_priv *)(dev->data[0]);
@@ -1455,6 +1521,7 @@ static void smsc75xx_unbind(struct usbnet *dev, struct usb_interface *intf)
{
struct smsc75xx_priv *pdata = (struct smsc75xx_priv *)(dev->data[0]);
if (pdata) {
+ cancel_work_sync(&pdata->set_multicast);
netif_dbg(dev, ifdown, dev->net, "free pdata\n");
kfree(pdata);
pdata = NULL;
diff --git a/drivers/net/usb/smsc95xx.c b/drivers/net/usb/smsc95xx.c
index 831aa33..e29f4c0 100644
--- a/drivers/net/usb/smsc95xx.c
+++ b/drivers/net/usb/smsc95xx.c
@@ -775,6 +775,9 @@ static int smsc95xx_ethtool_set_wol(struct net_device *net,
struct smsc95xx_priv *pdata = (struct smsc95xx_priv *)(dev->data[0]);
int ret;
+ if (wolinfo->wolopts & ~SUPPORTED_WAKE)
+ return -EINVAL;
+
pdata->wolopts = wolinfo->wolopts & SUPPORTED_WAKE;
ret = device_set_wakeup_enable(&dev->udev->dev, pdata->wolopts);
@@ -1587,6 +1590,8 @@ static int smsc95xx_suspend(struct usb_interface *intf, pm_message_t message)
return ret;
}
+ cancel_delayed_work_sync(&pdata->carrier_check);
+
if (pdata->suspend_flags) {
netdev_warn(dev->net, "error during last resume\n");
pdata->suspend_flags = 0;
@@ -1829,6 +1834,11 @@ static int smsc95xx_suspend(struct usb_interface *intf, pm_message_t message)
*/
if (ret && PMSG_IS_AUTO(message))
usbnet_resume(intf);
+
+ if (ret)
+ schedule_delayed_work(&pdata->carrier_check,
+ CARRIER_CHECK_DELAY);
+
return ret;
}
diff --git a/drivers/net/usb/sr9800.c b/drivers/net/usb/sr9800.c
index a50df0d..004c955 100644
--- a/drivers/net/usb/sr9800.c
+++ b/drivers/net/usb/sr9800.c
@@ -421,6 +421,9 @@ sr_set_wol(struct net_device *net, struct ethtool_wolinfo *wolinfo)
struct usbnet *dev = netdev_priv(net);
u8 opt = 0;
+ if (wolinfo->wolopts & ~(WAKE_PHY | WAKE_MAGIC))
+ return -EINVAL;
+
if (wolinfo->wolopts & WAKE_PHY)
opt |= SR_MONITOR_LINK;
if (wolinfo->wolopts & WAKE_MAGIC)
diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c
index 472ed6d..7118b82 100644
--- a/drivers/net/virtio_net.c
+++ b/drivers/net/virtio_net.c
@@ -1949,8 +1949,8 @@ static int virtnet_probe(struct virtio_device *vdev)
/* Assume link up if device can't report link status,
otherwise get link status from config. */
+ netif_carrier_off(dev);
if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
- netif_carrier_off(dev);
schedule_work(&vi->config_work);
} else {
vi->status = VIRTIO_NET_S_LINK_UP;
diff --git a/drivers/net/vmxnet3/vmxnet3_drv.c b/drivers/net/vmxnet3/vmxnet3_drv.c
index f809eed..c999b10 100644
--- a/drivers/net/vmxnet3/vmxnet3_drv.c
+++ b/drivers/net/vmxnet3/vmxnet3_drv.c
@@ -369,6 +369,11 @@ vmxnet3_tq_tx_complete(struct vmxnet3_tx_queue *tq,
gdesc = tq->comp_ring.base + tq->comp_ring.next2proc;
while (VMXNET3_TCD_GET_GEN(&gdesc->tcd) == tq->comp_ring.gen) {
+ /* Prevent any &gdesc->tcd field from being (speculatively)
+ * read before (&gdesc->tcd)->gen is read.
+ */
+ dma_rmb();
+
completed += vmxnet3_unmap_pkt(VMXNET3_TCD_GET_TXIDX(
&gdesc->tcd), tq, adapter->pdev,
adapter);
@@ -1099,6 +1104,11 @@ vmxnet3_tq_xmit(struct sk_buff *skb, struct vmxnet3_tx_queue *tq,
gdesc->txd.tci = skb_vlan_tag_get(skb);
}
+ /* Ensure that the write to (&gdesc->txd)->gen will be observed after
+ * all other writes to &gdesc->txd.
+ */
+ dma_wmb();
+
/* finally flips the GEN bit of the SOP desc. */
gdesc->dword[2] = cpu_to_le32(le32_to_cpu(gdesc->dword[2]) ^
VMXNET3_TXD_GEN);
@@ -1286,6 +1296,12 @@ vmxnet3_rq_rx_complete(struct vmxnet3_rx_queue *rq,
*/
break;
}
+
+ /* Prevent any rcd field from being (speculatively) read before
+ * rcd->gen is read.
+ */
+ dma_rmb();
+
BUG_ON(rcd->rqID != rq->qid && rcd->rqID != rq->qid2 &&
rcd->rqID != rq->dataRingQid);
idx = rcd->rxdIdx;
@@ -1515,6 +1531,12 @@ vmxnet3_rq_rx_complete(struct vmxnet3_rx_queue *rq,
ring->next2comp = idx;
num_to_alloc = vmxnet3_cmd_ring_desc_avail(ring);
ring = rq->rx_ring + ring_idx;
+
+ /* Ensure that the writes to rxd->gen bits will be observed
+ * after all other writes to rxd objects.
+ */
+ dma_wmb();
+
while (num_to_alloc) {
vmxnet3_getRxDesc(rxd, &ring->base[ring->next2fill].rxd,
&rxCmdDesc);
@@ -2675,7 +2697,7 @@ vmxnet3_set_mac_addr(struct net_device *netdev, void *p)
/* ==================== initialization and cleanup routines ============ */
static int
-vmxnet3_alloc_pci_resources(struct vmxnet3_adapter *adapter, bool *dma64)
+vmxnet3_alloc_pci_resources(struct vmxnet3_adapter *adapter)
{
int err;
unsigned long mmio_start, mmio_len;
@@ -2687,30 +2709,12 @@ vmxnet3_alloc_pci_resources(struct vmxnet3_adapter *adapter, bool *dma64)
return err;
}
- if (pci_set_dma_mask(pdev, DMA_BIT_MASK(64)) == 0) {
- if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64)) != 0) {
- dev_err(&pdev->dev,
- "pci_set_consistent_dma_mask failed\n");
- err = -EIO;
- goto err_set_mask;
- }
- *dma64 = true;
- } else {
- if (pci_set_dma_mask(pdev, DMA_BIT_MASK(32)) != 0) {
- dev_err(&pdev->dev,
- "pci_set_dma_mask failed\n");
- err = -EIO;
- goto err_set_mask;
- }
- *dma64 = false;
- }
-
err = pci_request_selected_regions(pdev, (1 << 2) - 1,
vmxnet3_driver_name);
if (err) {
dev_err(&pdev->dev,
"Failed to request region for adapter: error %d\n", err);
- goto err_set_mask;
+ goto err_enable_device;
}
pci_set_master(pdev);
@@ -2738,7 +2742,7 @@ vmxnet3_alloc_pci_resources(struct vmxnet3_adapter *adapter, bool *dma64)
iounmap(adapter->hw_addr0);
err_ioremap:
pci_release_selected_regions(pdev, (1 << 2) - 1);
-err_set_mask:
+err_enable_device:
pci_disable_device(pdev);
return err;
}
@@ -3246,7 +3250,7 @@ vmxnet3_probe_device(struct pci_dev *pdev,
#endif
};
int err;
- bool dma64 = false; /* stupid gcc */
+ bool dma64;
u32 ver;
struct net_device *netdev;
struct vmxnet3_adapter *adapter;
@@ -3292,6 +3296,24 @@ vmxnet3_probe_device(struct pci_dev *pdev,
adapter->rx_ring_size = VMXNET3_DEF_RX_RING_SIZE;
adapter->rx_ring2_size = VMXNET3_DEF_RX_RING2_SIZE;
+ if (pci_set_dma_mask(pdev, DMA_BIT_MASK(64)) == 0) {
+ if (pci_set_consistent_dma_mask(pdev, DMA_BIT_MASK(64)) != 0) {
+ dev_err(&pdev->dev,
+ "pci_set_consistent_dma_mask failed\n");
+ err = -EIO;
+ goto err_set_mask;
+ }
+ dma64 = true;
+ } else {
+ if (pci_set_dma_mask(pdev, DMA_BIT_MASK(32)) != 0) {
+ dev_err(&pdev->dev,
+ "pci_set_dma_mask failed\n");
+ err = -EIO;
+ goto err_set_mask;
+ }
+ dma64 = false;
+ }
+
spin_lock_init(&adapter->cmd_lock);
adapter->adapter_pa = dma_map_single(&adapter->pdev->dev, adapter,
sizeof(struct vmxnet3_adapter),
@@ -3299,7 +3321,7 @@ vmxnet3_probe_device(struct pci_dev *pdev,
if (dma_mapping_error(&adapter->pdev->dev, adapter->adapter_pa)) {
dev_err(&pdev->dev, "Failed to map dma\n");
err = -EFAULT;
- goto err_dma_map;
+ goto err_set_mask;
}
adapter->shared = dma_alloc_coherent(
&adapter->pdev->dev,
@@ -3350,7 +3372,7 @@ vmxnet3_probe_device(struct pci_dev *pdev,
}
#endif /* VMXNET3_RSS */
- err = vmxnet3_alloc_pci_resources(adapter, &dma64);
+ err = vmxnet3_alloc_pci_resources(adapter);
if (err < 0)
goto err_alloc_pci;
@@ -3492,7 +3514,7 @@ vmxnet3_probe_device(struct pci_dev *pdev,
err_alloc_shared:
dma_unmap_single(&adapter->pdev->dev, adapter->adapter_pa,
sizeof(struct vmxnet3_adapter), PCI_DMA_TODEVICE);
-err_dma_map:
+err_set_mask:
free_netdev(netdev);
return err;
}
diff --git a/drivers/net/wan/fsl_ucc_hdlc.c b/drivers/net/wan/fsl_ucc_hdlc.c
index a8bd68f..7a62316 100644
--- a/drivers/net/wan/fsl_ucc_hdlc.c
+++ b/drivers/net/wan/fsl_ucc_hdlc.c
@@ -161,7 +161,7 @@ static int uhdlc_init(struct ucc_hdlc_private *priv)
priv->ucc_pram_offset = qe_muram_alloc(sizeof(struct ucc_hdlc_param),
ALIGNMENT_OF_UCC_HDLC_PRAM);
- if (priv->ucc_pram_offset < 0) {
+ if (IS_ERR_VALUE(priv->ucc_pram_offset)) {
dev_err(priv->dev, "Can not allocate MURAM for hdlc parameter.\n");
ret = -ENOMEM;
goto free_tx_bd;
@@ -197,14 +197,14 @@ static int uhdlc_init(struct ucc_hdlc_private *priv)
/* Alloc riptr, tiptr */
riptr = qe_muram_alloc(32, 32);
- if (riptr < 0) {
+ if (IS_ERR_VALUE(riptr)) {
dev_err(priv->dev, "Cannot allocate MURAM mem for Receive internal temp data pointer\n");
ret = -ENOMEM;
goto free_tx_skbuff;
}
tiptr = qe_muram_alloc(32, 32);
- if (tiptr < 0) {
+ if (IS_ERR_VALUE(tiptr)) {
dev_err(priv->dev, "Cannot allocate MURAM mem for Transmit internal temp data pointer\n");
ret = -ENOMEM;
goto free_riptr;
diff --git a/drivers/net/wan/lmc/lmc_main.c b/drivers/net/wan/lmc/lmc_main.c
index 299140c..04b60ed 100644
--- a/drivers/net/wan/lmc/lmc_main.c
+++ b/drivers/net/wan/lmc/lmc_main.c
@@ -1372,7 +1372,7 @@ static irqreturn_t lmc_interrupt (int irq, void *dev_instance) /*fold00*/
case 0x001:
printk(KERN_WARNING "%s: Master Abort (naughty)\n", dev->name);
break;
- case 0x010:
+ case 0x002:
printk(KERN_WARNING "%s: Target Abort (not so naughty)\n", dev->name);
break;
default:
diff --git a/drivers/net/wan/x25_asy.c b/drivers/net/wan/x25_asy.c
index 1bc5e93..eb56bb5 100644
--- a/drivers/net/wan/x25_asy.c
+++ b/drivers/net/wan/x25_asy.c
@@ -488,8 +488,10 @@ static int x25_asy_open(struct net_device *dev)
/* Cleanup */
kfree(sl->xbuff);
+ sl->xbuff = NULL;
noxbuff:
kfree(sl->rbuff);
+ sl->rbuff = NULL;
norbuff:
return -ENOMEM;
}
diff --git a/drivers/net/wireless/ath/ath10k/ahb.c b/drivers/net/wireless/ath/ath10k/ahb.c
index 45226db..da770af 100644
--- a/drivers/net/wireless/ath/ath10k/ahb.c
+++ b/drivers/net/wireless/ath/ath10k/ahb.c
@@ -640,6 +640,7 @@ static int ath10k_ahb_hif_start(struct ath10k *ar)
{
ath10k_dbg(ar, ATH10K_DBG_BOOT, "boot ahb hif start\n");
+ napi_enable(&ar->napi);
ath10k_ce_enable_interrupts(ar);
ath10k_pci_enable_legacy_irq(ar);
@@ -692,7 +693,6 @@ static int ath10k_ahb_hif_power_up(struct ath10k *ar)
ath10k_err(ar, "could not wake up target CPU: %d\n", ret);
goto err_ce_deinit;
}
- napi_enable(&ar->napi);
return 0;
diff --git a/drivers/net/wireless/ath/ath10k/core.c b/drivers/net/wireless/ath/ath10k/core.c
index 65ad7a1..1e41d6c 100644
--- a/drivers/net/wireless/ath/ath10k/core.c
+++ b/drivers/net/wireless/ath/ath10k/core.c
@@ -698,7 +698,8 @@ static int ath10k_core_get_board_id_from_otp(struct ath10k *ar)
if ((result & ATH10K_BMI_BOARD_ID_STATUS_MASK) != 0 ||
(board_id == 0)) {
- ath10k_warn(ar, "board id is not exist in otp, ignore it\n");
+ ath10k_dbg(ar, ATH10K_DBG_BOOT,
+ "board id does not exist in otp, ignore it\n");
return -EOPNOTSUPP;
}
diff --git a/drivers/net/wireless/ath/ath10k/debug.c b/drivers/net/wireless/ath/ath10k/debug.c
index 0dadc60..b106a06 100644
--- a/drivers/net/wireless/ath/ath10k/debug.c
+++ b/drivers/net/wireless/ath/ath10k/debug.c
@@ -1,6 +1,7 @@
/*
* Copyright (c) 2005-2011 Atheros Communications Inc.
* Copyright (c) 2011-2013 Qualcomm Atheros, Inc.
+ * Copyright (c) 2018, The Linux Foundation. All rights reserved.
*
* Permission to use, copy, modify, and/or distribute this software for any
* purpose with or without fee is hereby granted, provided that the above
@@ -161,6 +162,8 @@ void ath10k_debug_print_hwfw_info(struct ath10k *ar)
void ath10k_debug_print_board_info(struct ath10k *ar)
{
char boardinfo[100];
+ const struct firmware *board;
+ u32 crc;
if (ar->id.bmi_ids_valid)
scnprintf(boardinfo, sizeof(boardinfo), "%d:%d",
@@ -168,11 +171,16 @@ void ath10k_debug_print_board_info(struct ath10k *ar)
else
scnprintf(boardinfo, sizeof(boardinfo), "N/A");
+ board = ar->normal_mode_fw.board;
+ if (!IS_ERR_OR_NULL(board))
+ crc = crc32_le(0, board->data, board->size);
+ else
+ crc = 0;
+
ath10k_info(ar, "board_file api %d bmi_id %s crc32 %08x",
ar->bd_api,
boardinfo,
- crc32_le(0, ar->normal_mode_fw.board->data,
- ar->normal_mode_fw.board->size));
+ crc);
}
void ath10k_debug_print_boot_info(struct ath10k *ar)
diff --git a/drivers/net/wireless/ath/ath10k/htt_rx.c b/drivers/net/wireless/ath/ath10k/htt_rx.c
index ba1fe61..a3c2180 100644
--- a/drivers/net/wireless/ath/ath10k/htt_rx.c
+++ b/drivers/net/wireless/ath/ath10k/htt_rx.c
@@ -214,11 +214,12 @@ int ath10k_htt_rx_ring_refill(struct ath10k *ar)
spin_lock_bh(&htt->rx_ring.lock);
ret = ath10k_htt_rx_ring_fill_n(htt, (htt->rx_ring.fill_level -
htt->rx_ring.fill_cnt));
- spin_unlock_bh(&htt->rx_ring.lock);
if (ret)
ath10k_htt_rx_ring_free(htt);
+ spin_unlock_bh(&htt->rx_ring.lock);
+
return ret;
}
@@ -230,7 +231,9 @@ void ath10k_htt_rx_free(struct ath10k_htt *htt)
skb_queue_purge(&htt->rx_in_ord_compl_q);
skb_queue_purge(&htt->tx_fetch_ind_q);
+ spin_lock_bh(&htt->rx_ring.lock);
ath10k_htt_rx_ring_free(htt);
+ spin_unlock_bh(&htt->rx_ring.lock);
dma_free_coherent(htt->ar->dev,
(htt->rx_ring.size *
diff --git a/drivers/net/wireless/ath/ath10k/mac.c b/drivers/net/wireless/ath/ath10k/mac.c
index a497bf3..fb632a4 100644
--- a/drivers/net/wireless/ath/ath10k/mac.c
+++ b/drivers/net/wireless/ath/ath10k/mac.c
@@ -3003,6 +3003,13 @@ static int ath10k_update_channel_list(struct ath10k *ar)
passive = channel->flags & IEEE80211_CHAN_NO_IR;
ch->passive = passive;
+ /* the firmware is ignoring the "radar" flag of the
+ * channel and is scanning actively using Probe Requests
+ * on "Radar detection"/DFS channels which are not
+ * marked as "available"
+ */
+ ch->passive |= ch->chan_radar;
+
ch->freq = channel->center_freq;
ch->band_center_freq1 = channel->center_freq;
ch->min_power = 0;
@@ -4960,7 +4967,9 @@ static int ath10k_add_interface(struct ieee80211_hw *hw,
}
ar->free_vdev_map &= ~(1LL << arvif->vdev_id);
+ spin_lock_bh(&ar->data_lock);
list_add(&arvif->list, &ar->arvifs);
+ spin_unlock_bh(&ar->data_lock);
/* It makes no sense to have firmware do keepalives. mac80211 already
* takes care of this with idle connection polling.
@@ -5111,7 +5120,9 @@ static int ath10k_add_interface(struct ieee80211_hw *hw,
err_vdev_delete:
ath10k_wmi_vdev_delete(ar, arvif->vdev_id);
ar->free_vdev_map |= 1LL << arvif->vdev_id;
+ spin_lock_bh(&ar->data_lock);
list_del(&arvif->list);
+ spin_unlock_bh(&ar->data_lock);
err:
if (arvif->beacon_buf) {
@@ -5157,7 +5168,9 @@ static void ath10k_remove_interface(struct ieee80211_hw *hw,
arvif->vdev_id, ret);
ar->free_vdev_map |= 1LL << arvif->vdev_id;
+ spin_lock_bh(&ar->data_lock);
list_del(&arvif->list);
+ spin_unlock_bh(&ar->data_lock);
if (arvif->vdev_type == WMI_VDEV_TYPE_AP ||
arvif->vdev_type == WMI_VDEV_TYPE_IBSS) {
@@ -5819,9 +5832,8 @@ static void ath10k_sta_rc_update_wk(struct work_struct *wk)
sta->addr, smps, err);
}
- if (changed & IEEE80211_RC_SUPP_RATES_CHANGED ||
- changed & IEEE80211_RC_NSS_CHANGED) {
- ath10k_dbg(ar, ATH10K_DBG_MAC, "mac update sta %pM supp rates/nss\n",
+ if (changed & IEEE80211_RC_SUPP_RATES_CHANGED) {
+ ath10k_dbg(ar, ATH10K_DBG_MAC, "mac update sta %pM supp rates\n",
sta->addr);
err = ath10k_station_assoc(ar, arvif->vif, sta, true);
@@ -6929,10 +6941,20 @@ static void ath10k_sta_rc_update(struct ieee80211_hw *hw,
{
struct ath10k *ar = hw->priv;
struct ath10k_sta *arsta = (struct ath10k_sta *)sta->drv_priv;
+ struct ath10k_vif *arvif = (void *)vif->drv_priv;
+ struct ath10k_peer *peer;
u32 bw, smps;
spin_lock_bh(&ar->data_lock);
+ peer = ath10k_peer_find(ar, arvif->vdev_id, sta->addr);
+ if (!peer) {
+ spin_unlock_bh(&ar->data_lock);
+ ath10k_warn(ar, "mac sta rc update failed to find peer %pM on vdev %i\n",
+ sta->addr, arvif->vdev_id);
+ return;
+ }
+
ath10k_dbg(ar, ATH10K_DBG_MAC,
"mac sta rc update for %pM changed %08x bw %d nss %d smps %d\n",
sta->addr, changed, sta->bandwidth, sta->rx_nss,
diff --git a/drivers/net/wireless/ath/ath10k/trace.h b/drivers/net/wireless/ath/ath10k/trace.h
index e0d00ce..5b974bb 100644
--- a/drivers/net/wireless/ath/ath10k/trace.h
+++ b/drivers/net/wireless/ath/ath10k/trace.h
@@ -152,10 +152,9 @@ TRACE_EVENT(ath10k_log_dbg_dump,
);
TRACE_EVENT(ath10k_wmi_cmd,
- TP_PROTO(struct ath10k *ar, int id, const void *buf, size_t buf_len,
- int ret),
+ TP_PROTO(struct ath10k *ar, int id, const void *buf, size_t buf_len),
- TP_ARGS(ar, id, buf, buf_len, ret),
+ TP_ARGS(ar, id, buf, buf_len),
TP_STRUCT__entry(
__string(device, dev_name(ar->dev))
@@ -163,7 +162,6 @@ TRACE_EVENT(ath10k_wmi_cmd,
__field(unsigned int, id)
__field(size_t, buf_len)
__dynamic_array(u8, buf, buf_len)
- __field(int, ret)
),
TP_fast_assign(
@@ -171,17 +169,15 @@ TRACE_EVENT(ath10k_wmi_cmd,
__assign_str(driver, dev_driver_string(ar->dev));
__entry->id = id;
__entry->buf_len = buf_len;
- __entry->ret = ret;
memcpy(__get_dynamic_array(buf), buf, buf_len);
),
TP_printk(
- "%s %s id %d len %zu ret %d",
+ "%s %s id %d len %zu",
__get_str(driver),
__get_str(device),
__entry->id,
- __entry->buf_len,
- __entry->ret
+ __entry->buf_len
)
);
diff --git a/drivers/net/wireless/ath/ath10k/wmi-tlv.c b/drivers/net/wireless/ath/ath10k/wmi-tlv.c
index 0e4d49a..642a441 100644
--- a/drivers/net/wireless/ath/ath10k/wmi-tlv.c
+++ b/drivers/net/wireless/ath/ath10k/wmi-tlv.c
@@ -1451,6 +1451,11 @@ static struct sk_buff *ath10k_wmi_tlv_op_gen_init(struct ath10k *ar)
cfg->keep_alive_pattern_size = __cpu_to_le32(0);
cfg->max_tdls_concurrent_sleep_sta = __cpu_to_le32(1);
cfg->max_tdls_concurrent_buffer_sta = __cpu_to_le32(1);
+ cfg->wmi_send_separate = __cpu_to_le32(0);
+ cfg->num_ocb_vdevs = __cpu_to_le32(0);
+ cfg->num_ocb_channels = __cpu_to_le32(0);
+ cfg->num_ocb_schedules = __cpu_to_le32(0);
+ cfg->host_capab = __cpu_to_le32(0);
ath10k_wmi_put_host_mem_chunks(ar, chunks);
@@ -1481,10 +1486,10 @@ ath10k_wmi_tlv_op_gen_start_scan(struct ath10k *ar,
bssid_len = arg->n_bssids * sizeof(struct wmi_mac_addr);
ie_len = roundup(arg->ie_len, 4);
len = (sizeof(*tlv) + sizeof(*cmd)) +
- (arg->n_channels ? sizeof(*tlv) + chan_len : 0) +
- (arg->n_ssids ? sizeof(*tlv) + ssid_len : 0) +
- (arg->n_bssids ? sizeof(*tlv) + bssid_len : 0) +
- (arg->ie_len ? sizeof(*tlv) + ie_len : 0);
+ sizeof(*tlv) + chan_len +
+ sizeof(*tlv) + ssid_len +
+ sizeof(*tlv) + bssid_len +
+ sizeof(*tlv) + ie_len;
skb = ath10k_wmi_alloc_skb(ar, len);
if (!skb)
diff --git a/drivers/net/wireless/ath/ath10k/wmi-tlv.h b/drivers/net/wireless/ath/ath10k/wmi-tlv.h
index b8aa600..2c94fe3 100644
--- a/drivers/net/wireless/ath/ath10k/wmi-tlv.h
+++ b/drivers/net/wireless/ath/ath10k/wmi-tlv.h
@@ -1227,6 +1227,11 @@ struct wmi_tlv_resource_config {
__le32 keep_alive_pattern_size;
__le32 max_tdls_concurrent_sleep_sta;
__le32 max_tdls_concurrent_buffer_sta;
+ __le32 wmi_send_separate;
+ __le32 num_ocb_vdevs;
+ __le32 num_ocb_channels;
+ __le32 num_ocb_schedules;
+ __le32 host_capab;
} __packed;
struct wmi_tlv_init_cmd {
diff --git a/drivers/net/wireless/ath/ath10k/wmi.c b/drivers/net/wireless/ath/ath10k/wmi.c
index e518b64..21aec5c 100644
--- a/drivers/net/wireless/ath/ath10k/wmi.c
+++ b/drivers/net/wireless/ath/ath10k/wmi.c
@@ -1711,8 +1711,8 @@ int ath10k_wmi_cmd_send_nowait(struct ath10k *ar, struct sk_buff *skb,
cmd_hdr->cmd_id = __cpu_to_le32(cmd);
memset(skb_cb, 0, sizeof(*skb_cb));
+ trace_ath10k_wmi_cmd(ar, cmd_id, skb->data, skb->len);
ret = ath10k_htc_send(&ar->htc, ar->wmi.eid, skb);
- trace_ath10k_wmi_cmd(ar, cmd_id, skb->data, skb->len, ret);
if (ret)
goto err_pull;
@@ -1822,6 +1822,12 @@ int ath10k_wmi_cmd_send(struct ath10k *ar, struct sk_buff *skb, u32 cmd_id)
if (ret)
dev_kfree_skb_any(skb);
+ if (ret == -EAGAIN) {
+ ath10k_warn(ar, "wmi command %d timeout, restarting hardware\n",
+ cmd_id);
+ queue_work(ar->workqueue, &ar->restart_work);
+ }
+
return ret;
}
diff --git a/drivers/net/wireless/ath/ath9k/hw.c b/drivers/net/wireless/ath/ath9k/hw.c
index a35f78b..951bac2 100644
--- a/drivers/net/wireless/ath/ath9k/hw.c
+++ b/drivers/net/wireless/ath/ath9k/hw.c
@@ -1603,6 +1603,10 @@ bool ath9k_hw_check_alive(struct ath_hw *ah)
int count = 50;
u32 reg, last_val;
+ /* Check if chip failed to wake up */
+ if (REG_READ(ah, AR_CFG) == 0xdeadbeef)
+ return false;
+
if (AR_SREV_9300(ah))
return !ath9k_hw_detect_mac_hang(ah);
@@ -2911,16 +2915,19 @@ void ath9k_hw_apply_txpower(struct ath_hw *ah, struct ath9k_channel *chan,
struct ath_regulatory *reg = ath9k_hw_regulatory(ah);
struct ieee80211_channel *channel;
int chan_pwr, new_pwr;
+ u16 ctl = NO_CTL;
if (!chan)
return;
+ if (!test)
+ ctl = ath9k_regd_get_ctl(reg, chan);
+
channel = chan->chan;
chan_pwr = min_t(int, channel->max_power * 2, MAX_RATE_POWER);
new_pwr = min_t(int, chan_pwr, reg->power_limit);
- ah->eep_ops->set_txpower(ah, chan,
- ath9k_regd_get_ctl(reg, chan),
+ ah->eep_ops->set_txpower(ah, chan, ctl,
get_antenna_gain(ah, chan), new_pwr, test);
}
diff --git a/drivers/net/wireless/ath/ath9k/xmit.c b/drivers/net/wireless/ath/ath9k/xmit.c
index e47286b..8a504af 100644
--- a/drivers/net/wireless/ath/ath9k/xmit.c
+++ b/drivers/net/wireless/ath/ath9k/xmit.c
@@ -84,7 +84,8 @@ static void ath_tx_status(struct ieee80211_hw *hw, struct sk_buff *skb)
struct ieee80211_tx_info *info = IEEE80211_SKB_CB(skb);
struct ieee80211_sta *sta = info->status.status_driver_data[0];
- if (info->flags & IEEE80211_TX_CTL_REQ_TX_STATUS) {
+ if (info->flags & (IEEE80211_TX_CTL_REQ_TX_STATUS |
+ IEEE80211_TX_STATUS_EOSP)) {
ieee80211_tx_status(hw, skb);
return;
}
diff --git a/drivers/net/wireless/ath/regd.h b/drivers/net/wireless/ath/regd.h
index 565d307..8553ab4 100644
--- a/drivers/net/wireless/ath/regd.h
+++ b/drivers/net/wireless/ath/regd.h
@@ -68,12 +68,14 @@ enum CountryCode {
CTRY_AUSTRALIA = 36,
CTRY_AUSTRIA = 40,
CTRY_AZERBAIJAN = 31,
+ CTRY_BAHAMAS = 44,
CTRY_BAHRAIN = 48,
CTRY_BANGLADESH = 50,
CTRY_BARBADOS = 52,
CTRY_BELARUS = 112,
CTRY_BELGIUM = 56,
CTRY_BELIZE = 84,
+ CTRY_BERMUDA = 60,
CTRY_BOLIVIA = 68,
CTRY_BOSNIA_HERZ = 70,
CTRY_BRAZIL = 76,
@@ -159,6 +161,7 @@ enum CountryCode {
CTRY_ROMANIA = 642,
CTRY_RUSSIA = 643,
CTRY_SAUDI_ARABIA = 682,
+ CTRY_SERBIA = 688,
CTRY_SERBIA_MONTENEGRO = 891,
CTRY_SINGAPORE = 702,
CTRY_SLOVAKIA = 703,
@@ -170,11 +173,13 @@ enum CountryCode {
CTRY_SWITZERLAND = 756,
CTRY_SYRIA = 760,
CTRY_TAIWAN = 158,
+ CTRY_TANZANIA = 834,
CTRY_THAILAND = 764,
CTRY_TRINIDAD_Y_TOBAGO = 780,
CTRY_TUNISIA = 788,
CTRY_TURKEY = 792,
CTRY_UAE = 784,
+ CTRY_UGANDA = 800,
CTRY_UKRAINE = 804,
CTRY_UNITED_KINGDOM = 826,
CTRY_UNITED_STATES = 840,
diff --git a/drivers/net/wireless/ath/regd_common.h b/drivers/net/wireless/ath/regd_common.h
index bdd2b4d..15bbd1e 100644
--- a/drivers/net/wireless/ath/regd_common.h
+++ b/drivers/net/wireless/ath/regd_common.h
@@ -35,6 +35,7 @@ enum EnumRd {
FRANCE_RES = 0x31,
FCC3_FCCA = 0x3A,
FCC3_WORLD = 0x3B,
+ FCC3_ETSIC = 0x3F,
ETSI1_WORLD = 0x37,
ETSI3_ETSIA = 0x32,
@@ -44,6 +45,7 @@ enum EnumRd {
ETSI4_ETSIC = 0x38,
ETSI5_WORLD = 0x39,
ETSI6_WORLD = 0x34,
+ ETSI8_WORLD = 0x3D,
ETSI_RESERVED = 0x33,
MKK1_MKKA = 0x40,
@@ -59,6 +61,7 @@ enum EnumRd {
MKK1_MKKA1 = 0x4A,
MKK1_MKKA2 = 0x4B,
MKK1_MKKC = 0x4C,
+ APL2_FCCA = 0x4D,
APL3_FCCA = 0x50,
APL1_WORLD = 0x52,
@@ -67,6 +70,7 @@ enum EnumRd {
APL1_ETSIC = 0x55,
APL2_ETSIC = 0x56,
APL5_WORLD = 0x58,
+ APL13_WORLD = 0x5A,
APL6_WORLD = 0x5B,
APL7_FCCA = 0x5C,
APL8_WORLD = 0x5D,
@@ -168,6 +172,7 @@ static struct reg_dmn_pair_mapping regDomainPairs[] = {
{FCC2_ETSIC, CTL_FCC, CTL_ETSI},
{FCC3_FCCA, CTL_FCC, CTL_FCC},
{FCC3_WORLD, CTL_FCC, CTL_ETSI},
+ {FCC3_ETSIC, CTL_FCC, CTL_ETSI},
{FCC4_FCCA, CTL_FCC, CTL_FCC},
{FCC5_FCCA, CTL_FCC, CTL_FCC},
{FCC6_FCCA, CTL_FCC, CTL_FCC},
@@ -179,6 +184,7 @@ static struct reg_dmn_pair_mapping regDomainPairs[] = {
{ETSI4_WORLD, CTL_ETSI, CTL_ETSI},
{ETSI5_WORLD, CTL_ETSI, CTL_ETSI},
{ETSI6_WORLD, CTL_ETSI, CTL_ETSI},
+ {ETSI8_WORLD, CTL_ETSI, CTL_ETSI},
/* XXX: For ETSI3_ETSIA, Was NO_CTL meant for the 2 GHz band ? */
{ETSI3_ETSIA, CTL_ETSI, CTL_ETSI},
@@ -188,9 +194,11 @@ static struct reg_dmn_pair_mapping regDomainPairs[] = {
{FCC1_FCCA, CTL_FCC, CTL_FCC},
{APL1_WORLD, CTL_FCC, CTL_ETSI},
{APL2_WORLD, CTL_FCC, CTL_ETSI},
+ {APL2_FCCA, CTL_FCC, CTL_FCC},
{APL3_WORLD, CTL_FCC, CTL_ETSI},
{APL4_WORLD, CTL_FCC, CTL_ETSI},
{APL5_WORLD, CTL_FCC, CTL_ETSI},
+ {APL13_WORLD, CTL_ETSI, CTL_ETSI},
{APL6_WORLD, CTL_ETSI, CTL_ETSI},
{APL8_WORLD, CTL_ETSI, CTL_ETSI},
{APL9_WORLD, CTL_ETSI, CTL_ETSI},
@@ -298,6 +306,7 @@ static struct country_code_to_enum_rd allCountries[] = {
{CTRY_AUSTRALIA2, FCC6_WORLD, "AU"},
{CTRY_AUSTRIA, ETSI1_WORLD, "AT"},
{CTRY_AZERBAIJAN, ETSI4_WORLD, "AZ"},
+ {CTRY_BAHAMAS, FCC3_WORLD, "BS"},
{CTRY_BAHRAIN, APL6_WORLD, "BH"},
{CTRY_BANGLADESH, NULL1_WORLD, "BD"},
{CTRY_BARBADOS, FCC2_WORLD, "BB"},
@@ -305,6 +314,7 @@ static struct country_code_to_enum_rd allCountries[] = {
{CTRY_BELGIUM, ETSI1_WORLD, "BE"},
{CTRY_BELGIUM2, ETSI4_WORLD, "BL"},
{CTRY_BELIZE, APL1_ETSIC, "BZ"},
+ {CTRY_BERMUDA, FCC3_FCCA, "BM"},
{CTRY_BOLIVIA, APL1_ETSIC, "BO"},
{CTRY_BOSNIA_HERZ, ETSI1_WORLD, "BA"},
{CTRY_BRAZIL, FCC3_WORLD, "BR"},
@@ -444,6 +454,7 @@ static struct country_code_to_enum_rd allCountries[] = {
{CTRY_ROMANIA, NULL1_WORLD, "RO"},
{CTRY_RUSSIA, NULL1_WORLD, "RU"},
{CTRY_SAUDI_ARABIA, NULL1_WORLD, "SA"},
+ {CTRY_SERBIA, ETSI1_WORLD, "RS"},
{CTRY_SERBIA_MONTENEGRO, ETSI1_WORLD, "CS"},
{CTRY_SINGAPORE, APL6_WORLD, "SG"},
{CTRY_SLOVAKIA, ETSI1_WORLD, "SK"},
@@ -455,10 +466,12 @@ static struct country_code_to_enum_rd allCountries[] = {
{CTRY_SWITZERLAND, ETSI1_WORLD, "CH"},
{CTRY_SYRIA, NULL1_WORLD, "SY"},
{CTRY_TAIWAN, APL3_FCCA, "TW"},
+ {CTRY_TANZANIA, APL1_WORLD, "TZ"},
{CTRY_THAILAND, FCC3_WORLD, "TH"},
{CTRY_TRINIDAD_Y_TOBAGO, FCC3_WORLD, "TT"},
{CTRY_TUNISIA, ETSI3_WORLD, "TN"},
{CTRY_TURKEY, ETSI3_WORLD, "TR"},
+ {CTRY_UGANDA, FCC3_WORLD, "UG"},
{CTRY_UKRAINE, NULL1_WORLD, "UA"},
{CTRY_UAE, NULL1_WORLD, "AE"},
{CTRY_UNITED_KINGDOM, ETSI1_WORLD, "GB"},
diff --git a/drivers/net/wireless/ath/wil6210/Makefile b/drivers/net/wireless/ath/wil6210/Makefile
index 94df1de..90bb9ac 100644
--- a/drivers/net/wireless/ath/wil6210/Makefile
+++ b/drivers/net/wireless/ath/wil6210/Makefile
@@ -29,3 +29,7 @@
MSM_11AD_PATH = drivers/platform/msm/msm_11ad
CFLAGS_wil_platform.o := -I$(MSM_11AD_PATH)
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-duplicate-decl-specifier \
diff --git a/drivers/net/wireless/ath/wil6210/fw_inc.c b/drivers/net/wireless/ath/wil6210/fw_inc.c
index 914c010..aef9cf2 100644
--- a/drivers/net/wireless/ath/wil6210/fw_inc.c
+++ b/drivers/net/wireless/ath/wil6210/fw_inc.c
@@ -533,7 +533,7 @@ static int wil_fw_process(struct wil6210_priv *wil, const void *data,
{
int rc = 0;
const struct wil_fw_record_head *hdr;
- size_t s, hdr_sz;
+ size_t s, hdr_sz = 0;
for (hdr = data;; hdr = (const void *)hdr + s, size -= s) {
if (size < sizeof(*hdr))
diff --git a/drivers/net/wireless/broadcom/b43/leds.c b/drivers/net/wireless/broadcom/b43/leds.c
index cb987c2..87131f6 100644
--- a/drivers/net/wireless/broadcom/b43/leds.c
+++ b/drivers/net/wireless/broadcom/b43/leds.c
@@ -131,7 +131,7 @@ static int b43_register_led(struct b43_wldev *dev, struct b43_led *led,
led->wl = dev->wl;
led->index = led_index;
led->activelow = activelow;
- strncpy(led->name, name, sizeof(led->name));
+ strlcpy(led->name, name, sizeof(led->name));
atomic_set(&led->state, 0);
led->led_dev.name = led->name;
diff --git a/drivers/net/wireless/broadcom/b43/phy_common.c b/drivers/net/wireless/broadcom/b43/phy_common.c
index 85f2ca9..ef3ffa5 100644
--- a/drivers/net/wireless/broadcom/b43/phy_common.c
+++ b/drivers/net/wireless/broadcom/b43/phy_common.c
@@ -616,7 +616,7 @@ struct b43_c32 b43_cordic(int theta)
u8 i;
s32 tmp;
s8 signx = 1;
- u32 angle = 0;
+ s32 angle = 0;
struct b43_c32 ret = { .i = 39797, .q = 0, };
while (theta > (180 << 16))
diff --git a/drivers/net/wireless/broadcom/b43legacy/leds.c b/drivers/net/wireless/broadcom/b43legacy/leds.c
index fd45653..bc92211 100644
--- a/drivers/net/wireless/broadcom/b43legacy/leds.c
+++ b/drivers/net/wireless/broadcom/b43legacy/leds.c
@@ -101,7 +101,7 @@ static int b43legacy_register_led(struct b43legacy_wldev *dev,
led->dev = dev;
led->index = led_index;
led->activelow = activelow;
- strncpy(led->name, name, sizeof(led->name));
+ strlcpy(led->name, name, sizeof(led->name));
led->led_dev.name = led->name;
led->led_dev.default_trigger = default_trigger;
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c
index 746f8c9..e69cf0e 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/bcmsdh.c
@@ -1099,6 +1099,7 @@ static const struct sdio_device_id brcmf_sdmmc_ids[] = {
BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_43340),
BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_43341),
BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_43362),
+ BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_43364),
BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_4335_4339),
BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_4339),
BRCMF_SDIO_DEVICE(SDIO_DEVICE_ID_BROADCOM_43430),
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c
index 54354a3..95b2d83 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/cfg80211.c
@@ -5999,7 +5999,8 @@ static int brcmf_construct_chaninfo(struct brcmf_cfg80211_info *cfg,
* for subsequent chanspecs.
*/
channel->flags = IEEE80211_CHAN_NO_HT40 |
- IEEE80211_CHAN_NO_80MHZ;
+ IEEE80211_CHAN_NO_80MHZ |
+ IEEE80211_CHAN_NO_160MHZ;
ch.bw = BRCMU_CHAN_BW_20;
cfg->d11inf.encchspec(&ch);
chaninfo = ch.chspec;
@@ -6798,7 +6799,7 @@ static void brcmf_cfg80211_reg_notifier(struct wiphy *wiphy,
int i;
/* ignore non-ISO3166 country codes */
- for (i = 0; i < sizeof(req->alpha2); i++)
+ for (i = 0; i < 2; i++)
if (req->alpha2[i] < 'A' || req->alpha2[i] > 'Z') {
brcmf_err("not a ISO3166 code (0x%02x 0x%02x)\n",
req->alpha2[0], req->alpha2[1]);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c
index d46f086..de52d82 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c
@@ -4229,6 +4229,13 @@ void brcmf_sdio_remove(struct brcmf_sdio *bus)
brcmf_dbg(TRACE, "Enter\n");
if (bus) {
+ /* Stop watchdog task */
+ if (bus->watchdog_tsk) {
+ send_sig(SIGTERM, bus->watchdog_tsk, 1);
+ kthread_stop(bus->watchdog_tsk);
+ bus->watchdog_tsk = NULL;
+ }
+
/* De-register interrupt handler */
brcmf_sdiod_intr_unregister(bus->sdiodev);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmutil/d11.c b/drivers/net/wireless/broadcom/brcm80211/brcmutil/d11.c
index d8b79cb..eb5db94 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmutil/d11.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmutil/d11.c
@@ -77,6 +77,8 @@ static u16 d11ac_bw(enum brcmu_chan_bw bw)
return BRCMU_CHSPEC_D11AC_BW_40;
case BRCMU_CHAN_BW_80:
return BRCMU_CHSPEC_D11AC_BW_80;
+ case BRCMU_CHAN_BW_160:
+ return BRCMU_CHSPEC_D11AC_BW_160;
default:
WARN_ON(1);
}
@@ -190,8 +192,41 @@ static void brcmu_d11ac_decchspec(struct brcmu_chan *ch)
break;
}
break;
- case BRCMU_CHSPEC_D11AC_BW_8080:
case BRCMU_CHSPEC_D11AC_BW_160:
+ ch->bw = BRCMU_CHAN_BW_160;
+ ch->sb = brcmu_maskget16(ch->chspec, BRCMU_CHSPEC_D11AC_SB_MASK,
+ BRCMU_CHSPEC_D11AC_SB_SHIFT);
+ switch (ch->sb) {
+ case BRCMU_CHAN_SB_LLL:
+ ch->control_ch_num -= CH_70MHZ_APART;
+ break;
+ case BRCMU_CHAN_SB_LLU:
+ ch->control_ch_num -= CH_50MHZ_APART;
+ break;
+ case BRCMU_CHAN_SB_LUL:
+ ch->control_ch_num -= CH_30MHZ_APART;
+ break;
+ case BRCMU_CHAN_SB_LUU:
+ ch->control_ch_num -= CH_10MHZ_APART;
+ break;
+ case BRCMU_CHAN_SB_ULL:
+ ch->control_ch_num += CH_10MHZ_APART;
+ break;
+ case BRCMU_CHAN_SB_ULU:
+ ch->control_ch_num += CH_30MHZ_APART;
+ break;
+ case BRCMU_CHAN_SB_UUL:
+ ch->control_ch_num += CH_50MHZ_APART;
+ break;
+ case BRCMU_CHAN_SB_UUU:
+ ch->control_ch_num += CH_70MHZ_APART;
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ break;
+ }
+ break;
+ case BRCMU_CHSPEC_D11AC_BW_8080:
default:
WARN_ON_ONCE(1);
break;
diff --git a/drivers/net/wireless/broadcom/brcm80211/include/brcmu_wifi.h b/drivers/net/wireless/broadcom/brcm80211/include/brcmu_wifi.h
index 7b9a779..75b2a04 100644
--- a/drivers/net/wireless/broadcom/brcm80211/include/brcmu_wifi.h
+++ b/drivers/net/wireless/broadcom/brcm80211/include/brcmu_wifi.h
@@ -29,6 +29,8 @@
#define CH_UPPER_SB 0x01
#define CH_LOWER_SB 0x02
#define CH_EWA_VALID 0x04
+#define CH_70MHZ_APART 14
+#define CH_50MHZ_APART 10
#define CH_30MHZ_APART 6
#define CH_20MHZ_APART 4
#define CH_10MHZ_APART 2
diff --git a/drivers/net/wireless/cnss/Kconfig b/drivers/net/wireless/cnss/Kconfig
index 0b37af6..0ae93e9 100644
--- a/drivers/net/wireless/cnss/Kconfig
+++ b/drivers/net/wireless/cnss/Kconfig
@@ -114,3 +114,9 @@
help
Enable WLAN_FEATURE_HOLD_RX_WAKELOCK which is required to take rx
wakelock when driver receives packets from fw.
+
+config WLAN_FEATURE_SARV1_TO_SARV2
+ bool "Enable conversion of SAR v1 to v2 feature"
+ help
+ Enable conversion to SAR V2 requests if BDF is configured for SAR V2.
+
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c
index f1231c0..92557cd 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c
@@ -327,8 +327,12 @@ struct ieee80211_regdomain *iwl_mvm_get_regdomain(struct wiphy *wiphy,
goto out;
}
- if (changed)
- *changed = (resp->status == MCC_RESP_NEW_CHAN_PROFILE);
+ if (changed) {
+ u32 status = le32_to_cpu(resp->status);
+
+ *changed = (status == MCC_RESP_NEW_CHAN_PROFILE ||
+ status == MCC_RESP_ILLEGAL);
+ }
regd = iwl_parse_nvm_mcc_info(mvm->trans->dev, mvm->cfg,
__le32_to_cpu(resp->n_channels),
@@ -2585,6 +2589,10 @@ static int iwl_mvm_mac_sta_state(struct ieee80211_hw *hw,
/* enable beacon filtering */
WARN_ON(iwl_mvm_enable_beacon_filter(mvm, vif, 0));
+
+ iwl_mvm_rs_rate_init(mvm, sta, mvmvif->phy_ctxt->channel->band,
+ false);
+
ret = 0;
} else if (old_state == IEEE80211_STA_AUTHORIZED &&
new_state == IEEE80211_STA_ASSOC) {
@@ -3972,10 +3980,6 @@ static void iwl_mvm_mac_sta_statistics(struct ieee80211_hw *hw,
sinfo->filled |= BIT(NL80211_STA_INFO_SIGNAL_AVG);
}
- if (!fw_has_capa(&mvm->fw->ucode_capa,
- IWL_UCODE_TLV_CAPA_RADIO_BEACON_STATS))
- return;
-
/* if beacon filtering isn't on mac80211 does it anyway */
if (!(vif->driver_flags & IEEE80211_VIF_BEACON_FILTER))
return;
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/nvm.c b/drivers/net/wireless/intel/iwlwifi/mvm/nvm.c
index eade099..e51aca8 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/nvm.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/nvm.c
@@ -739,9 +739,8 @@ iwl_mvm_update_mcc(struct iwl_mvm *mvm, const char *alpha2,
}
IWL_DEBUG_LAR(mvm,
- "MCC response status: 0x%x. new MCC: 0x%x ('%c%c') change: %d n_chans: %d\n",
- status, mcc, mcc >> 8, mcc & 0xff,
- !!(status == MCC_RESP_NEW_CHAN_PROFILE), n_channels);
+ "MCC response status: 0x%x. new MCC: 0x%x ('%c%c') n_chans: %d\n",
+ status, mcc, mcc >> 8, mcc & 0xff, n_channels);
exit:
iwl_free_resp(&cmd);
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rs.c b/drivers/net/wireless/intel/iwlwifi/mvm/rs.c
index 0aea476..f45c997 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/rs.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/rs.c
@@ -1207,7 +1207,11 @@ void iwl_mvm_rs_tx_status(struct iwl_mvm *mvm, struct ieee80211_sta *sta,
!(info->flags & IEEE80211_TX_STAT_AMPDU))
return;
- rs_rate_from_ucode_rate(tx_resp_hwrate, info->band, &tx_resp_rate);
+ if (rs_rate_from_ucode_rate(tx_resp_hwrate, info->band,
+ &tx_resp_rate)) {
+ WARN_ON_ONCE(1);
+ return;
+ }
#ifdef CONFIG_MAC80211_DEBUGFS
/* Disable last tx check if we are debugging with fixed rate but
@@ -1263,7 +1267,10 @@ void iwl_mvm_rs_tx_status(struct iwl_mvm *mvm, struct ieee80211_sta *sta,
*/
table = &lq_sta->lq;
lq_hwrate = le32_to_cpu(table->rs_table[0]);
- rs_rate_from_ucode_rate(lq_hwrate, info->band, &lq_rate);
+ if (rs_rate_from_ucode_rate(lq_hwrate, info->band, &lq_rate)) {
+ WARN_ON_ONCE(1);
+ return;
+ }
/* Here we actually compare this rate to the latest LQ command */
if (!rs_rate_equal(&tx_resp_rate, &lq_rate, allow_ant_mismatch)) {
@@ -1365,8 +1372,12 @@ void iwl_mvm_rs_tx_status(struct iwl_mvm *mvm, struct ieee80211_sta *sta,
/* Collect data for each rate used during failed TX attempts */
for (i = 0; i <= retries; ++i) {
lq_hwrate = le32_to_cpu(table->rs_table[i]);
- rs_rate_from_ucode_rate(lq_hwrate, info->band,
- &lq_rate);
+ if (rs_rate_from_ucode_rate(lq_hwrate, info->band,
+ &lq_rate)) {
+ WARN_ON_ONCE(1);
+ return;
+ }
+
/*
* Only collect stats if retried rate is in the same RS
* table as active/search.
@@ -2709,7 +2720,8 @@ static void rs_get_initial_rate(struct iwl_mvm *mvm,
struct ieee80211_sta *sta,
struct iwl_lq_sta *lq_sta,
enum nl80211_band band,
- struct rs_rate *rate)
+ struct rs_rate *rate,
+ bool init)
{
int i, nentries;
unsigned long active_rate;
@@ -2763,14 +2775,25 @@ static void rs_get_initial_rate(struct iwl_mvm *mvm,
*/
if (sta->vht_cap.vht_supported &&
best_rssi > IWL_RS_LOW_RSSI_THRESHOLD) {
- switch (sta->bandwidth) {
- case IEEE80211_STA_RX_BW_160:
- case IEEE80211_STA_RX_BW_80:
- case IEEE80211_STA_RX_BW_40:
+ /*
+ * In AP mode, when a new station associates, rs is initialized
+ * immediately upon association completion, before the phy
+ * context is updated with the association parameters, so the
+ * sta bandwidth might be wider than the phy context allows.
+ * To avoid this issue, always initialize rs with 20mhz
+ * bandwidth rate, and after authorization, when the phy context
+ * is already up-to-date, re-init rs with the correct bw.
+ */
+ u32 bw = init ? RATE_MCS_CHAN_WIDTH_20 : rs_bw_from_sta_bw(sta);
+
+ switch (bw) {
+ case RATE_MCS_CHAN_WIDTH_40:
+ case RATE_MCS_CHAN_WIDTH_80:
+ case RATE_MCS_CHAN_WIDTH_160:
initial_rates = rs_optimal_rates_vht;
nentries = ARRAY_SIZE(rs_optimal_rates_vht);
break;
- case IEEE80211_STA_RX_BW_20:
+ case RATE_MCS_CHAN_WIDTH_20:
initial_rates = rs_optimal_rates_vht_20mhz;
nentries = ARRAY_SIZE(rs_optimal_rates_vht_20mhz);
break;
@@ -2781,7 +2804,7 @@ static void rs_get_initial_rate(struct iwl_mvm *mvm,
active_rate = lq_sta->active_siso_rate;
rate->type = LQ_VHT_SISO;
- rate->bw = rs_bw_from_sta_bw(sta);
+ rate->bw = bw;
} else if (sta->ht_cap.ht_supported &&
best_rssi > IWL_RS_LOW_RSSI_THRESHOLD) {
initial_rates = rs_optimal_rates_ht;
@@ -2863,7 +2886,7 @@ static void rs_initialize_lq(struct iwl_mvm *mvm,
tbl = &(lq_sta->lq_info[active_tbl]);
rate = &tbl->rate;
- rs_get_initial_rate(mvm, sta, lq_sta, band, rate);
+ rs_get_initial_rate(mvm, sta, lq_sta, band, rate, init);
rs_init_optimal_rate(mvm, sta, lq_sta);
WARN_ON_ONCE(rate->ant != ANT_A && rate->ant != ANT_B);
@@ -3249,7 +3272,10 @@ static void rs_build_rates_table_from_fixed(struct iwl_mvm *mvm,
for (i = 0; i < num_rates; i++)
lq_cmd->rs_table[i] = ucode_rate_le32;
- rs_rate_from_ucode_rate(ucode_rate, band, &rate);
+ if (rs_rate_from_ucode_rate(ucode_rate, band, &rate)) {
+ WARN_ON_ONCE(1);
+ return;
+ }
if (is_mimo(&rate))
lq_cmd->mimo_delim = num_rates - 1;
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c b/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c
index a481eb4..c2bbc8c 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/rxmq.c
@@ -72,6 +72,7 @@ static inline int iwl_mvm_check_pn(struct iwl_mvm *mvm, struct sk_buff *skb,
struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data;
struct ieee80211_rx_status *stats = IEEE80211_SKB_RXCB(skb);
struct iwl_mvm_key_pn *ptk_pn;
+ int res;
u8 tid, keyidx;
u8 pn[IEEE80211_CCMP_PN_LEN];
u8 *extiv;
@@ -128,12 +129,13 @@ static inline int iwl_mvm_check_pn(struct iwl_mvm *mvm, struct sk_buff *skb,
pn[4] = extiv[1];
pn[5] = extiv[0];
- if (memcmp(pn, ptk_pn->q[queue].pn[tid],
- IEEE80211_CCMP_PN_LEN) <= 0)
+ res = memcmp(pn, ptk_pn->q[queue].pn[tid], IEEE80211_CCMP_PN_LEN);
+ if (res < 0)
+ return -1;
+ if (!res && !(stats->flag & RX_FLAG_ALLOW_SAME_PN))
return -1;
- if (!(stats->flag & RX_FLAG_AMSDU_MORE))
- memcpy(ptk_pn->q[queue].pn[tid], pn, IEEE80211_CCMP_PN_LEN);
+ memcpy(ptk_pn->q[queue].pn[tid], pn, IEEE80211_CCMP_PN_LEN);
stats->flag |= RX_FLAG_PN_VALIDATED;
return 0;
@@ -295,28 +297,21 @@ static void iwl_mvm_rx_csum(struct ieee80211_sta *sta,
}
/*
- * returns true if a packet outside BA session is a duplicate and
- * should be dropped
+ * returns true if a packet is a duplicate and should be dropped.
+ * Updates AMSDU PN tracking info
*/
-static bool iwl_mvm_is_nonagg_dup(struct ieee80211_sta *sta, int queue,
- struct ieee80211_rx_status *rx_status,
- struct ieee80211_hdr *hdr,
- struct iwl_rx_mpdu_desc *desc)
+static bool iwl_mvm_is_dup(struct ieee80211_sta *sta, int queue,
+ struct ieee80211_rx_status *rx_status,
+ struct ieee80211_hdr *hdr,
+ struct iwl_rx_mpdu_desc *desc)
{
struct iwl_mvm_sta *mvm_sta;
struct iwl_mvm_rxq_dup_data *dup_data;
- u8 baid, tid, sub_frame_idx;
+ u8 tid, sub_frame_idx;
if (WARN_ON(IS_ERR_OR_NULL(sta)))
return false;
- baid = (le32_to_cpu(desc->reorder_data) &
- IWL_RX_MPDU_REORDER_BAID_MASK) >>
- IWL_RX_MPDU_REORDER_BAID_SHIFT;
-
- if (baid != IWL_RX_REORDER_DATA_INVALID_BAID)
- return false;
-
mvm_sta = iwl_mvm_sta_from_mac80211(sta);
dup_data = &mvm_sta->dup_data[queue];
@@ -346,6 +341,12 @@ static bool iwl_mvm_is_nonagg_dup(struct ieee80211_sta *sta, int queue,
dup_data->last_sub_frame[tid] >= sub_frame_idx))
return true;
+ /* Allow same PN as the first subframe for following sub frames */
+ if (dup_data->last_seq[tid] == hdr->seq_ctrl &&
+ sub_frame_idx > dup_data->last_sub_frame[tid] &&
+ desc->mac_flags2 & IWL_RX_MPDU_MFLG2_AMSDU)
+ rx_status->flag |= RX_FLAG_ALLOW_SAME_PN;
+
dup_data->last_seq[tid] = hdr->seq_ctrl;
dup_data->last_sub_frame[tid] = sub_frame_idx;
@@ -882,7 +883,7 @@ void iwl_mvm_rx_mpdu_mq(struct iwl_mvm *mvm, struct napi_struct *napi,
if (ieee80211_is_data(hdr->frame_control))
iwl_mvm_rx_csum(sta, skb, desc);
- if (iwl_mvm_is_nonagg_dup(sta, queue, rx_status, hdr, desc)) {
+ if (iwl_mvm_is_dup(sta, queue, rx_status, hdr, desc)) {
kfree_skb(skb);
rcu_read_unlock();
return;
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/tx.c b/drivers/net/wireless/intel/iwlwifi/mvm/tx.c
index 7465d4d..bd7ff56 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/tx.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/tx.c
@@ -406,11 +406,11 @@ static void iwl_mvm_set_tx_cmd_crypto(struct iwl_mvm *mvm,
{
struct ieee80211_key_conf *keyconf = info->control.hw_key;
u8 *crypto_hdr = skb_frag->data + hdrlen;
+ enum iwl_tx_cmd_sec_ctrl type = TX_CMD_SEC_CCM;
u64 pn;
switch (keyconf->cipher) {
case WLAN_CIPHER_SUITE_CCMP:
- case WLAN_CIPHER_SUITE_CCMP_256:
iwl_mvm_set_tx_cmd_ccmp(info, tx_cmd);
iwl_mvm_set_tx_cmd_pn(info, crypto_hdr);
break;
@@ -434,13 +434,16 @@ static void iwl_mvm_set_tx_cmd_crypto(struct iwl_mvm *mvm,
break;
case WLAN_CIPHER_SUITE_GCMP:
case WLAN_CIPHER_SUITE_GCMP_256:
+ type = TX_CMD_SEC_GCMP;
+ /* Fall through */
+ case WLAN_CIPHER_SUITE_CCMP_256:
/* TODO: Taking the key from the table might introduce a race
* when PTK rekeying is done, having an old packets with a PN
* based on the old key but the message encrypted with a new
* one.
* Need to handle this.
*/
- tx_cmd->sec_ctl |= TX_CMD_SEC_GCMP | TX_CMD_SEC_KEY_FROM_TABLE;
+ tx_cmd->sec_ctl |= type | TX_CMD_SEC_KEY_FROM_TABLE;
tx_cmd->key[0] = keyconf->hw_key_idx;
iwl_mvm_set_tx_cmd_pn(info, crypto_hdr);
break;
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
index 6fe5546..e58a50d 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
@@ -898,6 +898,8 @@ int iwl_pcie_rx_init(struct iwl_trans *trans)
WQ_HIGHPRI | WQ_UNBOUND, 1);
INIT_WORK(&rba->rx_alloc, iwl_pcie_rx_allocator_work);
+ cancel_work_sync(&rba->rx_alloc);
+
spin_lock(&rba->lock);
atomic_set(&rba->req_pending, 0);
atomic_set(&rba->req_ready, 0);
@@ -1028,6 +1030,14 @@ void iwl_pcie_rx_free(struct iwl_trans *trans)
kfree(trans_pcie->rxq);
}
+static void iwl_pcie_rx_move_to_allocator(struct iwl_rxq *rxq,
+ struct iwl_rb_allocator *rba)
+{
+ spin_lock(&rba->lock);
+ list_splice_tail_init(&rxq->rx_used, &rba->rbd_empty);
+ spin_unlock(&rba->lock);
+}
+
/*
* iwl_pcie_rx_reuse_rbd - Recycle used RBDs
*
@@ -1059,9 +1069,7 @@ static void iwl_pcie_rx_reuse_rbd(struct iwl_trans *trans,
if ((rxq->used_count % RX_CLAIM_REQ_ALLOC) == RX_POST_REQ_ALLOC) {
/* Move the 2 RBDs to the allocator ownership.
Allocator has another 6 from pool for the request completion*/
- spin_lock(&rba->lock);
- list_splice_tail_init(&rxq->rx_used, &rba->rbd_empty);
- spin_unlock(&rba->lock);
+ iwl_pcie_rx_move_to_allocator(rxq, rba);
atomic_inc(&rba->req_pending);
queue_work(rba->alloc_wq, &rba->rx_alloc);
@@ -1231,10 +1239,18 @@ static void iwl_pcie_rx_handle(struct iwl_trans *trans, int queue)
IWL_DEBUG_RX(trans, "Q %d: HW = SW = %d\n", rxq->id, r);
while (i != r) {
+ struct iwl_rb_allocator *rba = &trans_pcie->rba;
struct iwl_rx_mem_buffer *rxb;
+ /* number of RBDs still waiting for page allocation */
+ u32 rb_pending_alloc =
+ atomic_read(&trans_pcie->rba.req_pending) *
+ RX_CLAIM_REQ_ALLOC;
- if (unlikely(rxq->used_count == rxq->queue_size / 2))
+ if (unlikely(rb_pending_alloc >= rxq->queue_size / 2 &&
+ !emergency)) {
+ iwl_pcie_rx_move_to_allocator(rxq, rba);
emergency = true;
+ }
if (trans->cfg->mq_rx_supported) {
/*
@@ -1277,17 +1293,13 @@ static void iwl_pcie_rx_handle(struct iwl_trans *trans, int queue)
iwl_pcie_rx_allocator_get(trans, rxq);
if (rxq->used_count % RX_CLAIM_REQ_ALLOC == 0 && !emergency) {
- struct iwl_rb_allocator *rba = &trans_pcie->rba;
-
/* Add the remaining empty RBDs for allocator use */
- spin_lock(&rba->lock);
- list_splice_tail_init(&rxq->rx_used, &rba->rbd_empty);
- spin_unlock(&rba->lock);
+ iwl_pcie_rx_move_to_allocator(rxq, rba);
} else if (emergency) {
count++;
if (count == 8) {
count = 0;
- if (rxq->used_count < rxq->queue_size / 3)
+ if (rb_pending_alloc < rxq->queue_size / 3)
emergency = false;
rxq->read = i;
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/trans.c b/drivers/net/wireless/intel/iwlwifi/pcie/trans.c
index fe32de2..e7b8730 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/trans.c
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/trans.c
@@ -1509,14 +1509,13 @@ static void iwl_pcie_set_interrupt_capa(struct pci_dev *pdev,
struct iwl_trans *trans)
{
struct iwl_trans_pcie *trans_pcie = IWL_TRANS_GET_PCIE_TRANS(trans);
- int max_irqs, num_irqs, i, ret, nr_online_cpus;
+ int max_irqs, num_irqs, i, ret;
u16 pci_cmd;
if (!trans->cfg->mq_rx_supported)
goto enable_msi;
- nr_online_cpus = num_online_cpus();
- max_irqs = min_t(u32, nr_online_cpus + 2, IWL_MAX_RX_HW_QUEUES);
+ max_irqs = min_t(u32, num_online_cpus() + 2, IWL_MAX_RX_HW_QUEUES);
for (i = 0; i < max_irqs; i++)
trans_pcie->msix_entries[i].entry = i;
@@ -1542,16 +1541,17 @@ static void iwl_pcie_set_interrupt_capa(struct pci_dev *pdev,
* Two interrupts less: non rx causes shared with FBQ and RSS.
* More than two interrupts: we will use fewer RSS queues.
*/
- if (num_irqs <= nr_online_cpus) {
+ if (num_irqs <= max_irqs - 2) {
trans_pcie->trans->num_rx_queues = num_irqs + 1;
trans_pcie->shared_vec_mask = IWL_SHARED_IRQ_NON_RX |
IWL_SHARED_IRQ_FIRST_RSS;
- } else if (num_irqs == nr_online_cpus + 1) {
+ } else if (num_irqs == max_irqs - 1) {
trans_pcie->trans->num_rx_queues = num_irqs;
trans_pcie->shared_vec_mask = IWL_SHARED_IRQ_NON_RX;
} else {
trans_pcie->trans->num_rx_queues = num_irqs - 1;
}
+ WARN_ON(trans_pcie->trans->num_rx_queues > IWL_MAX_RX_HW_QUEUES);
trans_pcie->alloc_vecs = num_irqs;
trans_pcie->msix_enabled = true;
diff --git a/drivers/net/wireless/mac80211_hwsim.c b/drivers/net/wireless/mac80211_hwsim.c
index 4182c37..780acf2 100644
--- a/drivers/net/wireless/mac80211_hwsim.c
+++ b/drivers/net/wireless/mac80211_hwsim.c
@@ -2569,9 +2569,6 @@ static int mac80211_hwsim_new_radio(struct genl_info *info,
IEEE80211_VHT_CAP_SHORT_GI_80 |
IEEE80211_VHT_CAP_SHORT_GI_160 |
IEEE80211_VHT_CAP_TXSTBC |
- IEEE80211_VHT_CAP_RXSTBC_1 |
- IEEE80211_VHT_CAP_RXSTBC_2 |
- IEEE80211_VHT_CAP_RXSTBC_3 |
IEEE80211_VHT_CAP_RXSTBC_4 |
IEEE80211_VHT_CAP_MAX_A_MPDU_LENGTH_EXPONENT_MASK;
sband->vht_cap.vht_mcs.rx_mcs_map =
@@ -2636,6 +2633,10 @@ static int mac80211_hwsim_new_radio(struct genl_info *info,
if (param->no_vif)
ieee80211_hw_set(hw, NO_AUTO_VIF);
+ tasklet_hrtimer_init(&data->beacon_timer,
+ mac80211_hwsim_beacon,
+ CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
+
err = ieee80211_register_hw(hw);
if (err < 0) {
printk(KERN_DEBUG "mac80211_hwsim: ieee80211_register_hw failed (%d)\n",
@@ -2660,16 +2661,11 @@ static int mac80211_hwsim_new_radio(struct genl_info *info,
data->debugfs,
data, &hwsim_simulate_radar);
- tasklet_hrtimer_init(&data->beacon_timer,
- mac80211_hwsim_beacon,
- CLOCK_MONOTONIC, HRTIMER_MODE_ABS);
-
spin_lock_bh(&hwsim_radio_lock);
list_add_tail(&data->list, &hwsim_radios);
spin_unlock_bh(&hwsim_radio_lock);
- if (idx > 0)
- hwsim_mcast_new_radio(idx, info, param);
+ hwsim_mcast_new_radio(idx, info, param);
return idx;
@@ -3084,8 +3080,10 @@ static int hwsim_new_radio_nl(struct sk_buff *msg, struct genl_info *info)
if (info->attrs[HWSIM_ATTR_REG_CUSTOM_REG]) {
u32 idx = nla_get_u32(info->attrs[HWSIM_ATTR_REG_CUSTOM_REG]);
- if (idx >= ARRAY_SIZE(hwsim_world_regdom_custom))
+ if (idx >= ARRAY_SIZE(hwsim_world_regdom_custom)) {
+ kfree(hwname);
return -EINVAL;
+ }
param.regd = hwsim_world_regdom_custom[idx];
}
@@ -3346,8 +3344,11 @@ static void __net_exit hwsim_exit_net(struct net *net)
continue;
list_del(&data->list);
- INIT_WORK(&data->destroy_work, destroy_radio);
- schedule_work(&data->destroy_work);
+ spin_unlock_bh(&hwsim_radio_lock);
+ mac80211_hwsim_del_radio(data, wiphy_name(data->hw->wiphy),
+ NULL);
+ spin_lock_bh(&hwsim_radio_lock);
+
}
spin_unlock_bh(&hwsim_radio_lock);
}
@@ -3402,16 +3403,16 @@ static int __init init_mac80211_hwsim(void)
if (err)
goto out_unregister_pernet;
+ err = hwsim_init_netlink();
+ if (err)
+ goto out_unregister_driver;
+
hwsim_class = class_create(THIS_MODULE, "mac80211_hwsim");
if (IS_ERR(hwsim_class)) {
err = PTR_ERR(hwsim_class);
- goto out_unregister_driver;
+ goto out_exit_netlink;
}
- err = hwsim_init_netlink();
- if (err < 0)
- goto out_unregister_driver;
-
for (i = 0; i < radios; i++) {
struct hwsim_new_radio_params param = { 0 };
@@ -3517,6 +3518,8 @@ static int __init init_mac80211_hwsim(void)
free_netdev(hwsim_mon);
out_free_radios:
mac80211_hwsim_free();
+out_exit_netlink:
+ hwsim_exit_netlink();
out_unregister_driver:
platform_driver_unregister(&mac80211_hwsim_driver);
out_unregister_pernet:
diff --git a/drivers/net/wireless/marvell/libertas/dev.h b/drivers/net/wireless/marvell/libertas/dev.h
index edf710b..3de1457 100644
--- a/drivers/net/wireless/marvell/libertas/dev.h
+++ b/drivers/net/wireless/marvell/libertas/dev.h
@@ -103,6 +103,7 @@ struct lbs_private {
u8 fw_ready;
u8 surpriseremoved;
u8 setup_fw_on_resume;
+ u8 power_up_on_resume;
int (*hw_host_to_card) (struct lbs_private *priv, u8 type, u8 *payload, u16 nb);
void (*reset_card) (struct lbs_private *priv);
int (*power_save) (struct lbs_private *priv);
diff --git a/drivers/net/wireless/marvell/libertas/if_sdio.c b/drivers/net/wireless/marvell/libertas/if_sdio.c
index 47f4a14..06a57c7 100644
--- a/drivers/net/wireless/marvell/libertas/if_sdio.c
+++ b/drivers/net/wireless/marvell/libertas/if_sdio.c
@@ -1341,15 +1341,23 @@ static void if_sdio_remove(struct sdio_func *func)
static int if_sdio_suspend(struct device *dev)
{
struct sdio_func *func = dev_to_sdio_func(dev);
- int ret;
struct if_sdio_card *card = sdio_get_drvdata(func);
+ struct lbs_private *priv = card->priv;
+ int ret;
mmc_pm_flag_t flags = sdio_get_host_pm_caps(func);
+ priv->power_up_on_resume = false;
/* If we're powered off anyway, just let the mmc layer remove the
* card. */
- if (!lbs_iface_active(card->priv))
- return -ENOSYS;
+ if (!lbs_iface_active(priv)) {
+ if (priv->fw_ready) {
+ priv->power_up_on_resume = true;
+ if_sdio_power_off(card);
+ }
+
+ return 0;
+ }
dev_info(dev, "%s: suspend: PM flags = 0x%x\n",
sdio_func_id(func), flags);
@@ -1357,9 +1365,18 @@ static int if_sdio_suspend(struct device *dev)
/* If we aren't being asked to wake on anything, we should bail out
* and let the SD stack power down the card.
*/
- if (card->priv->wol_criteria == EHS_REMOVE_WAKEUP) {
+ if (priv->wol_criteria == EHS_REMOVE_WAKEUP) {
dev_info(dev, "Suspend without wake params -- powering down card\n");
- return -ENOSYS;
+ if (priv->fw_ready) {
+ ret = lbs_suspend(priv);
+ if (ret)
+ return ret;
+
+ priv->power_up_on_resume = true;
+ if_sdio_power_off(card);
+ }
+
+ return 0;
}
if (!(flags & MMC_PM_KEEP_POWER)) {
@@ -1372,7 +1389,7 @@ static int if_sdio_suspend(struct device *dev)
if (ret)
return ret;
- ret = lbs_suspend(card->priv);
+ ret = lbs_suspend(priv);
if (ret)
return ret;
@@ -1387,6 +1404,11 @@ static int if_sdio_resume(struct device *dev)
dev_info(dev, "%s: resume: we're back\n", sdio_func_id(func));
+ if (card->priv->power_up_on_resume) {
+ if_sdio_power_on(card);
+ wait_event(card->pwron_waitq, card->priv->fw_ready);
+ }
+
ret = lbs_resume(card->priv);
return ret;
diff --git a/drivers/net/wireless/marvell/libertas/if_usb.c b/drivers/net/wireless/marvell/libertas/if_usb.c
index aba0c99..a605d56 100644
--- a/drivers/net/wireless/marvell/libertas/if_usb.c
+++ b/drivers/net/wireless/marvell/libertas/if_usb.c
@@ -468,8 +468,6 @@ static int __if_usb_submit_rx_urb(struct if_usb_card *cardp,
MRVDRV_ETH_RX_PACKET_BUFFER_SIZE, callbackfn,
cardp);
- cardp->rx_urb->transfer_flags |= URB_ZERO_PACKET;
-
lbs_deb_usb2(&cardp->udev->dev, "Pointer for rx_urb %p\n", cardp->rx_urb);
if ((ret = usb_submit_urb(cardp->rx_urb, GFP_ATOMIC))) {
lbs_deb_usbd(&cardp->udev->dev, "Submit Rx URB failed: %d\n", ret);
diff --git a/drivers/net/wireless/marvell/mwifiex/cfg80211.c b/drivers/net/wireless/marvell/mwifiex/cfg80211.c
index 48d51be..4da3541 100644
--- a/drivers/net/wireless/marvell/mwifiex/cfg80211.c
+++ b/drivers/net/wireless/marvell/mwifiex/cfg80211.c
@@ -1209,6 +1209,12 @@ mwifiex_cfg80211_change_virtual_intf(struct wiphy *wiphy,
priv->adapter->curr_iface_comb.p2p_intf--;
priv->adapter->curr_iface_comb.sta_intf++;
dev->ieee80211_ptr->iftype = type;
+ if (mwifiex_deinit_priv_params(priv))
+ return -1;
+ if (mwifiex_init_new_priv_params(priv, dev, type))
+ return -1;
+ if (mwifiex_sta_init_cmd(priv, false, false))
+ return -1;
break;
case NL80211_IFTYPE_ADHOC:
if (mwifiex_cfg80211_deinit_p2p(priv))
@@ -3079,8 +3085,10 @@ int mwifiex_del_virtual_intf(struct wiphy *wiphy, struct wireless_dev *wdev)
mwifiex_stop_net_dev_queue(priv->netdev, adapter);
- skb_queue_walk_safe(&priv->bypass_txq, skb, tmp)
+ skb_queue_walk_safe(&priv->bypass_txq, skb, tmp) {
+ skb_unlink(skb, &priv->bypass_txq);
mwifiex_write_data_complete(priv->adapter, skb, 0, -1);
+ }
if (netif_carrier_ok(priv->netdev))
netif_carrier_off(priv->netdev);
diff --git a/drivers/net/wireless/marvell/mwifiex/pcie.c b/drivers/net/wireless/marvell/mwifiex/pcie.c
index 1fdb86c..cb681b2 100644
--- a/drivers/net/wireless/marvell/mwifiex/pcie.c
+++ b/drivers/net/wireless/marvell/mwifiex/pcie.c
@@ -101,7 +101,6 @@ static int mwifiex_pcie_suspend(struct device *dev)
{
struct mwifiex_adapter *adapter;
struct pcie_service_card *card;
- int hs_actived;
struct pci_dev *pdev = to_pci_dev(dev);
if (pdev) {
@@ -117,7 +116,15 @@ static int mwifiex_pcie_suspend(struct device *dev)
adapter = card->adapter;
- hs_actived = mwifiex_enable_hs(adapter);
+ /* Enable the Host Sleep */
+ if (!mwifiex_enable_hs(adapter)) {
+ mwifiex_dbg(adapter, ERROR,
+ "cmd: failed to suspend\n");
+ adapter->hs_enabling = false;
+ return -EFAULT;
+ }
+
+ flush_workqueue(adapter->workqueue);
/* Indicate device suspended */
adapter->is_suspended = true;
@@ -1676,9 +1683,6 @@ static int mwifiex_pcie_process_cmd_complete(struct mwifiex_adapter *adapter)
if (!adapter->curr_cmd) {
if (adapter->ps_state == PS_STATE_SLEEP_CFM) {
- mwifiex_process_sleep_confirm_resp(adapter, skb->data,
- skb->len);
- mwifiex_pcie_enable_host_int(adapter);
if (mwifiex_write_reg(adapter,
PCIE_CPU_INT_EVENT,
CPU_INTR_SLEEP_CFM_DONE)) {
@@ -1691,6 +1695,9 @@ static int mwifiex_pcie_process_cmd_complete(struct mwifiex_adapter *adapter)
while (reg->sleep_cookie && (count++ < 10) &&
mwifiex_pcie_ok_to_access_hw(adapter))
usleep_range(50, 60);
+ mwifiex_pcie_enable_host_int(adapter);
+ mwifiex_process_sleep_confirm_resp(adapter, skb->data,
+ skb->len);
} else {
mwifiex_dbg(adapter, ERROR,
"There is no command but got cmdrsp\n");
@@ -2329,6 +2336,8 @@ static int mwifiex_process_pcie_int(struct mwifiex_adapter *adapter)
ret = mwifiex_pcie_process_cmd_complete(adapter);
if (ret)
return ret;
+ if (adapter->hs_activated)
+ return ret;
}
if (card->msi_enable) {
diff --git a/drivers/net/wireless/marvell/mwifiex/usb.c b/drivers/net/wireless/marvell/mwifiex/usb.c
index 73eb084..09185a1 100644
--- a/drivers/net/wireless/marvell/mwifiex/usb.c
+++ b/drivers/net/wireless/marvell/mwifiex/usb.c
@@ -624,6 +624,9 @@ static void mwifiex_usb_disconnect(struct usb_interface *intf)
MWIFIEX_FUNC_SHUTDOWN);
}
+ if (adapter->workqueue)
+ flush_workqueue(adapter->workqueue);
+
mwifiex_usb_free(card);
mwifiex_dbg(adapter, FATAL,
diff --git a/drivers/net/wireless/marvell/mwifiex/util.c b/drivers/net/wireless/marvell/mwifiex/util.c
index 18fbb96..d75756c 100644
--- a/drivers/net/wireless/marvell/mwifiex/util.c
+++ b/drivers/net/wireless/marvell/mwifiex/util.c
@@ -723,12 +723,14 @@ void mwifiex_hist_data_set(struct mwifiex_private *priv, u8 rx_rate, s8 snr,
s8 nflr)
{
struct mwifiex_histogram_data *phist_data = priv->hist_data;
+ s8 nf = -nflr;
+ s8 rssi = snr - nflr;
atomic_inc(&phist_data->num_samples);
atomic_inc(&phist_data->rx_rate[rx_rate]);
- atomic_inc(&phist_data->snr[snr]);
- atomic_inc(&phist_data->noise_flr[128 + nflr]);
- atomic_inc(&phist_data->sig_str[nflr - snr]);
+ atomic_inc(&phist_data->snr[snr + 128]);
+ atomic_inc(&phist_data->noise_flr[nf + 128]);
+ atomic_inc(&phist_data->sig_str[rssi + 128]);
}
/* function to reset histogram data during init/reset */
diff --git a/drivers/net/wireless/marvell/mwifiex/wmm.c b/drivers/net/wireless/marvell/mwifiex/wmm.c
index 0eb2465..dea2fe6 100644
--- a/drivers/net/wireless/marvell/mwifiex/wmm.c
+++ b/drivers/net/wireless/marvell/mwifiex/wmm.c
@@ -503,8 +503,10 @@ mwifiex_wmm_del_pkts_in_ralist_node(struct mwifiex_private *priv,
struct mwifiex_adapter *adapter = priv->adapter;
struct sk_buff *skb, *tmp;
- skb_queue_walk_safe(&ra_list->skb_head, skb, tmp)
+ skb_queue_walk_safe(&ra_list->skb_head, skb, tmp) {
+ skb_unlink(skb, &ra_list->skb_head);
mwifiex_write_data_complete(adapter, skb, 0, -1);
+ }
}
/*
@@ -600,11 +602,15 @@ mwifiex_clean_txrx(struct mwifiex_private *priv)
priv->adapter->if_ops.clean_pcie_ring(priv->adapter);
spin_unlock_irqrestore(&priv->wmm.ra_list_spinlock, flags);
- skb_queue_walk_safe(&priv->tdls_txq, skb, tmp)
+ skb_queue_walk_safe(&priv->tdls_txq, skb, tmp) {
+ skb_unlink(skb, &priv->tdls_txq);
mwifiex_write_data_complete(priv->adapter, skb, 0, -1);
+ }
- skb_queue_walk_safe(&priv->bypass_txq, skb, tmp)
+ skb_queue_walk_safe(&priv->bypass_txq, skb, tmp) {
+ skb_unlink(skb, &priv->bypass_txq);
mwifiex_write_data_complete(priv->adapter, skb, 0, -1);
+ }
atomic_set(&priv->adapter->bypass_tx_pending, 0);
idr_for_each(&priv->ack_status_frames, mwifiex_free_ack_frame, NULL);
diff --git a/drivers/net/wireless/realtek/rtlwifi/core.c b/drivers/net/wireless/realtek/rtlwifi/core.c
index 4da4e45..9526643 100644
--- a/drivers/net/wireless/realtek/rtlwifi/core.c
+++ b/drivers/net/wireless/realtek/rtlwifi/core.c
@@ -131,7 +131,6 @@ static void rtl_fw_do_work(const struct firmware *firmware, void *context,
firmware->size);
rtlpriv->rtlhal.wowlan_fwsize = firmware->size;
}
- rtlpriv->rtlhal.fwsize = firmware->size;
release_firmware(firmware);
}
diff --git a/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/rf.c b/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/rf.c
index ec2ea56..fdbd359 100644
--- a/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/rf.c
+++ b/drivers/net/wireless/realtek/rtlwifi/rtl8192cu/rf.c
@@ -304,9 +304,6 @@ static void _rtl92c_get_txpower_writeval_by_regulatory(struct ieee80211_hw *hw,
writeVal = 0x00000000;
if (rtlpriv->dm.dynamic_txhighpower_lvl == TXHIGHPWRLEVEL_BT1)
writeVal = writeVal - 0x06060606;
- else if (rtlpriv->dm.dynamic_txhighpower_lvl ==
- TXHIGHPWRLEVEL_BT2)
- writeVal = writeVal;
*(p_outwriteval + rf) = writeVal;
}
}
diff --git a/drivers/net/wireless/rndis_wlan.c b/drivers/net/wireless/rndis_wlan.c
index ae87b39..2e92872 100644
--- a/drivers/net/wireless/rndis_wlan.c
+++ b/drivers/net/wireless/rndis_wlan.c
@@ -2928,6 +2928,8 @@ static void rndis_wlan_auth_indication(struct usbnet *usbdev,
while (buflen >= sizeof(*auth_req)) {
auth_req = (void *)buf;
+ if (buflen < le32_to_cpu(auth_req->length))
+ return;
type = "unknown";
flags = le32_to_cpu(auth_req->flags);
pairwise_error = false;
diff --git a/drivers/net/wireless/rsi/rsi_91x_sdio.c b/drivers/net/wireless/rsi/rsi_91x_sdio.c
index 8428858..fc895b4 100644
--- a/drivers/net/wireless/rsi/rsi_91x_sdio.c
+++ b/drivers/net/wireless/rsi/rsi_91x_sdio.c
@@ -155,7 +155,6 @@ static void rsi_reset_card(struct sdio_func *pfunction)
int err;
struct mmc_card *card = pfunction->card;
struct mmc_host *host = card->host;
- s32 bit = (fls(host->ocr_avail) - 1);
u8 cmd52_resp;
u32 clock, resp, i;
u16 rca;
@@ -175,7 +174,6 @@ static void rsi_reset_card(struct sdio_func *pfunction)
msleep(20);
/* Initialize the SDIO card */
- host->ios.vdd = bit;
host->ios.chip_select = MMC_CS_DONTCARE;
host->ios.bus_mode = MMC_BUSMODE_OPENDRAIN;
host->ios.power_mode = MMC_POWER_UP;
diff --git a/drivers/net/wireless/st/cw1200/wsm.c b/drivers/net/wireless/st/cw1200/wsm.c
index ed93bf3..be4c22e 100644
--- a/drivers/net/wireless/st/cw1200/wsm.c
+++ b/drivers/net/wireless/st/cw1200/wsm.c
@@ -1805,16 +1805,18 @@ static int wsm_buf_reserve(struct wsm_buf *buf, size_t extra_size)
{
size_t pos = buf->data - buf->begin;
size_t size = pos + extra_size;
+ u8 *tmp;
size = round_up(size, FWLOAD_BLOCK_SIZE);
- buf->begin = krealloc(buf->begin, size, GFP_KERNEL | GFP_DMA);
- if (buf->begin) {
- buf->data = &buf->begin[pos];
- buf->end = &buf->begin[size];
- return 0;
- } else {
- buf->end = buf->data = buf->begin;
+ tmp = krealloc(buf->begin, size, GFP_KERNEL | GFP_DMA);
+ if (!tmp) {
+ wsm_buf_deinit(buf);
return -ENOMEM;
}
+
+ buf->begin = tmp;
+ buf->data = &buf->begin[pos];
+ buf->end = &buf->begin[size];
+ return 0;
}
diff --git a/drivers/net/wireless/ti/wlcore/rx.c b/drivers/net/wireless/ti/wlcore/rx.c
index b9e1404..7367f09 100644
--- a/drivers/net/wireless/ti/wlcore/rx.c
+++ b/drivers/net/wireless/ti/wlcore/rx.c
@@ -59,7 +59,7 @@ static u32 wlcore_rx_get_align_buf_size(struct wl1271 *wl, u32 pkt_len)
static void wl1271_rx_status(struct wl1271 *wl,
struct wl1271_rx_descriptor *desc,
struct ieee80211_rx_status *status,
- u8 beacon)
+ u8 beacon, u8 probe_rsp)
{
memset(status, 0, sizeof(struct ieee80211_rx_status));
@@ -106,6 +106,9 @@ static void wl1271_rx_status(struct wl1271 *wl,
}
}
+ if (beacon || probe_rsp)
+ status->boottime_ns = ktime_get_boot_ns();
+
if (beacon)
wlcore_set_pending_regdomain_ch(wl, (u16)desc->channel,
status->band);
@@ -194,7 +197,8 @@ static int wl1271_rx_handle_data(struct wl1271 *wl, u8 *data, u32 length,
if (ieee80211_is_data_present(hdr->frame_control))
is_data = 1;
- wl1271_rx_status(wl, desc, IEEE80211_SKB_RXCB(skb), beacon);
+ wl1271_rx_status(wl, desc, IEEE80211_SKB_RXCB(skb), beacon,
+ ieee80211_is_probe_resp(hdr->frame_control));
wlcore_hw_set_rx_csum(wl, desc, skb);
seq_num = (le16_to_cpu(hdr->seq_ctrl) & IEEE80211_SCTL_SEQ) >> 4;
diff --git a/drivers/net/wireless/ti/wlcore/sdio.c b/drivers/net/wireless/ti/wlcore/sdio.c
index 47fe7f9..6921cb0 100644
--- a/drivers/net/wireless/ti/wlcore/sdio.c
+++ b/drivers/net/wireless/ti/wlcore/sdio.c
@@ -404,6 +404,11 @@ static int wl1271_suspend(struct device *dev)
mmc_pm_flag_t sdio_flags;
int ret = 0;
+ if (!wl) {
+ dev_err(dev, "no wilink module was probed\n");
+ goto out;
+ }
+
dev_dbg(dev, "wl1271 suspend. wow_enabled: %d\n",
wl->wow_enabled);
diff --git a/drivers/net/xen-netback/hash.c b/drivers/net/xen-netback/hash.c
index 3c4c58b..3b6fb5b 100644
--- a/drivers/net/xen-netback/hash.c
+++ b/drivers/net/xen-netback/hash.c
@@ -332,20 +332,22 @@ u32 xenvif_set_hash_mapping_size(struct xenvif *vif, u32 size)
u32 xenvif_set_hash_mapping(struct xenvif *vif, u32 gref, u32 len,
u32 off)
{
- u32 *mapping = &vif->hash.mapping[off];
+ u32 *mapping = vif->hash.mapping;
struct gnttab_copy copy_op = {
.source.u.ref = gref,
.source.domid = vif->domid,
- .dest.u.gmfn = virt_to_gfn(mapping),
.dest.domid = DOMID_SELF,
- .dest.offset = xen_offset_in_page(mapping),
- .len = len * sizeof(u32),
+ .len = len * sizeof(*mapping),
.flags = GNTCOPY_source_gref
};
- if ((off + len > vif->hash.size) || copy_op.len > XEN_PAGE_SIZE)
+ if ((off + len < off) || (off + len > vif->hash.size) ||
+ len > XEN_PAGE_SIZE / sizeof(*mapping))
return XEN_NETIF_CTRL_STATUS_INVALID_PARAMETER;
+ copy_op.dest.u.gmfn = virt_to_gfn(mapping + off);
+ copy_op.dest.offset = xen_offset_in_page(mapping + off);
+
while (len-- != 0)
if (mapping[off++] >= vif->num_queues)
return XEN_NETIF_CTRL_STATUS_INVALID_PARAMETER;
diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index 1b28786..14ceeaa 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -86,7 +86,7 @@ struct netfront_cb {
/* IRQ name is queue name with "-tx" or "-rx" appended */
#define IRQ_NAME_SIZE (QUEUE_NAME_SIZE + 3)
-static DECLARE_WAIT_QUEUE_HEAD(module_unload_q);
+static DECLARE_WAIT_QUEUE_HEAD(module_wq);
struct netfront_stats {
u64 packets;
@@ -238,7 +238,7 @@ static void rx_refill_timeout(unsigned long data)
static int netfront_tx_slot_available(struct netfront_queue *queue)
{
return (queue->tx.req_prod_pvt - queue->tx.rsp_cons) <
- (NET_TX_RING_SIZE - MAX_SKB_FRAGS - 2);
+ (NET_TX_RING_SIZE - XEN_NETIF_NR_SLOTS_MIN - 1);
}
static void xennet_maybe_wake_tx(struct netfront_queue *queue)
@@ -350,6 +350,9 @@ static int xennet_open(struct net_device *dev)
unsigned int i = 0;
struct netfront_queue *queue = NULL;
+ if (!np->queues)
+ return -ENODEV;
+
for (i = 0; i < num_queues; ++i) {
queue = &np->queues[i];
napi_enable(&queue->napi);
@@ -786,7 +789,7 @@ static int xennet_get_responses(struct netfront_queue *queue,
RING_IDX cons = queue->rx.rsp_cons;
struct sk_buff *skb = xennet_get_rx_skb(queue, cons);
grant_ref_t ref = xennet_get_rx_ref(queue, cons);
- int max = MAX_SKB_FRAGS + (rx->status <= RX_COPY_THRESHOLD);
+ int max = XEN_NETIF_NR_SLOTS_MIN + (rx->status <= RX_COPY_THRESHOLD);
int slots = 1;
int err = 0;
unsigned long ret;
@@ -889,7 +892,6 @@ static RING_IDX xennet_fill_frags(struct netfront_queue *queue,
struct sk_buff *skb,
struct sk_buff_head *list)
{
- struct skb_shared_info *shinfo = skb_shinfo(skb);
RING_IDX cons = queue->rx.rsp_cons;
struct sk_buff *nskb;
@@ -898,15 +900,20 @@ static RING_IDX xennet_fill_frags(struct netfront_queue *queue,
RING_GET_RESPONSE(&queue->rx, ++cons);
skb_frag_t *nfrag = &skb_shinfo(nskb)->frags[0];
- if (shinfo->nr_frags == MAX_SKB_FRAGS) {
+ if (skb_shinfo(skb)->nr_frags == MAX_SKB_FRAGS) {
unsigned int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
- BUG_ON(pull_to <= skb_headlen(skb));
+ BUG_ON(pull_to < skb_headlen(skb));
__pskb_pull_tail(skb, pull_to - skb_headlen(skb));
}
- BUG_ON(shinfo->nr_frags >= MAX_SKB_FRAGS);
+ if (unlikely(skb_shinfo(skb)->nr_frags >= MAX_SKB_FRAGS)) {
+ queue->rx.rsp_cons = ++cons;
+ kfree_skb(nskb);
+ return ~0U;
+ }
- skb_add_rx_frag(skb, shinfo->nr_frags, skb_frag_page(nfrag),
+ skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags,
+ skb_frag_page(nfrag),
rx->offset, rx->status, PAGE_SIZE);
skb_shinfo(nskb)->nr_frags = 0;
@@ -1040,6 +1047,8 @@ static int xennet_poll(struct napi_struct *napi, int budget)
skb->len += rx->status;
i = xennet_fill_frags(queue, skb, &tmpq);
+ if (unlikely(i == ~0U))
+ goto err;
if (rx->flags & XEN_NETRXF_csum_blank)
skb->ip_summed = CHECKSUM_PARTIAL;
@@ -1346,6 +1355,11 @@ static struct net_device *xennet_create_dev(struct xenbus_device *dev)
netif_carrier_off(netdev);
xenbus_switch_state(dev, XenbusStateInitialising);
+ wait_event(module_wq,
+ xenbus_read_driver_state(dev->otherend) !=
+ XenbusStateClosed &&
+ xenbus_read_driver_state(dev->otherend) !=
+ XenbusStateUnknown);
return netdev;
exit:
@@ -1377,18 +1391,8 @@ static int netfront_probe(struct xenbus_device *dev,
#ifdef CONFIG_SYSFS
info->netdev->sysfs_groups[0] = &xennet_dev_group;
#endif
- err = register_netdev(info->netdev);
- if (err) {
- pr_warn("%s: register_netdev err=%d\n", __func__, err);
- goto fail;
- }
return 0;
-
- fail:
- xennet_free_netdev(netdev);
- dev_set_drvdata(&dev->dev, NULL);
- return err;
}
static void xennet_end_access(int ref, void *page)
@@ -1623,6 +1627,7 @@ static int xennet_init_queue(struct netfront_queue *queue)
{
unsigned short i;
int err = 0;
+ char *devid;
spin_lock_init(&queue->tx_lock);
spin_lock_init(&queue->rx_lock);
@@ -1630,8 +1635,9 @@ static int xennet_init_queue(struct netfront_queue *queue)
setup_timer(&queue->rx_refill_timer, rx_refill_timeout,
(unsigned long)queue);
- snprintf(queue->name, sizeof(queue->name), "%s-q%u",
- queue->info->netdev->name, queue->id);
+ devid = strrchr(queue->info->xbdev->nodename, '/') + 1;
+ snprintf(queue->name, sizeof(queue->name), "vif%s-q%u",
+ devid, queue->id);
/* Initialise tx_skbs as a free chain containing every entry. */
queue->tx_skb_freelist = 0;
@@ -1757,8 +1763,6 @@ static void xennet_destroy_queues(struct netfront_info *info)
{
unsigned int i;
- rtnl_lock();
-
for (i = 0; i < info->netdev->real_num_tx_queues; i++) {
struct netfront_queue *queue = &info->queues[i];
@@ -1767,8 +1771,6 @@ static void xennet_destroy_queues(struct netfront_info *info)
netif_napi_del(&queue->napi);
}
- rtnl_unlock();
-
kfree(info->queues);
info->queues = NULL;
}
@@ -1784,8 +1786,6 @@ static int xennet_create_queues(struct netfront_info *info,
if (!info->queues)
return -ENOMEM;
- rtnl_lock();
-
for (i = 0; i < *num_queues; i++) {
struct netfront_queue *queue = &info->queues[i];
@@ -1794,7 +1794,7 @@ static int xennet_create_queues(struct netfront_info *info,
ret = xennet_init_queue(queue);
if (ret < 0) {
- dev_warn(&info->netdev->dev,
+ dev_warn(&info->xbdev->dev,
"only created %d queues\n", i);
*num_queues = i;
break;
@@ -1808,10 +1808,8 @@ static int xennet_create_queues(struct netfront_info *info,
netif_set_real_num_tx_queues(info->netdev, *num_queues);
- rtnl_unlock();
-
if (*num_queues == 0) {
- dev_err(&info->netdev->dev, "no queues\n");
+ dev_err(&info->xbdev->dev, "no queues\n");
return -EINVAL;
}
return 0;
@@ -1850,9 +1848,10 @@ static int talk_to_netback(struct xenbus_device *dev,
err = xen_net_read_mac(dev, info->netdev->dev_addr);
if (err) {
xenbus_dev_fatal(dev, err, "parsing %s/mac", dev->nodename);
- goto out;
+ goto out_unlocked;
}
+ rtnl_lock();
if (info->queues)
xennet_destroy_queues(info);
@@ -1863,6 +1862,7 @@ static int talk_to_netback(struct xenbus_device *dev,
info->queues = NULL;
goto out;
}
+ rtnl_unlock();
/* Create shared ring, alloc event channel -- for each queue */
for (i = 0; i < num_queues; ++i) {
@@ -1959,8 +1959,11 @@ static int talk_to_netback(struct xenbus_device *dev,
xenbus_transaction_end(xbt, 1);
destroy_ring:
xennet_disconnect_backend(info);
+ rtnl_lock();
xennet_destroy_queues(info);
out:
+ rtnl_unlock();
+out_unlocked:
device_unregister(&dev->dev);
return err;
}
@@ -1992,6 +1995,15 @@ static int xennet_connect(struct net_device *dev)
/* talk_to_netback() sets the correct number of queues */
num_queues = dev->real_num_tx_queues;
+ if (dev->reg_state == NETREG_UNINITIALIZED) {
+ err = register_netdev(dev);
+ if (err) {
+ pr_warn("%s: register_netdev err=%d\n", __func__, err);
+ device_unregister(&np->xbdev->dev);
+ return err;
+ }
+ }
+
rtnl_lock();
netdev_update_features(dev);
rtnl_unlock();
@@ -2033,15 +2045,14 @@ static void netback_changed(struct xenbus_device *dev,
dev_dbg(&dev->dev, "%s\n", xenbus_strstate(backend_state));
+ wake_up_all(&module_wq);
+
switch (backend_state) {
case XenbusStateInitialising:
case XenbusStateInitialised:
case XenbusStateReconfiguring:
case XenbusStateReconfigured:
- break;
-
case XenbusStateUnknown:
- wake_up_all(&module_unload_q);
break;
case XenbusStateInitWait:
@@ -2057,12 +2068,10 @@ static void netback_changed(struct xenbus_device *dev,
break;
case XenbusStateClosed:
- wake_up_all(&module_unload_q);
if (dev->state == XenbusStateClosed)
break;
/* Missed the backend's CLOSING state -- fallthrough */
case XenbusStateClosing:
- wake_up_all(&module_unload_q);
xenbus_frontend_closed(dev);
break;
}
@@ -2170,14 +2179,14 @@ static int xennet_remove(struct xenbus_device *dev)
if (xenbus_read_driver_state(dev->otherend) != XenbusStateClosed) {
xenbus_switch_state(dev, XenbusStateClosing);
- wait_event(module_unload_q,
+ wait_event(module_wq,
xenbus_read_driver_state(dev->otherend) ==
XenbusStateClosing ||
xenbus_read_driver_state(dev->otherend) ==
XenbusStateUnknown);
xenbus_switch_state(dev, XenbusStateClosed);
- wait_event(module_unload_q,
+ wait_event(module_wq,
xenbus_read_driver_state(dev->otherend) ==
XenbusStateClosed ||
xenbus_read_driver_state(dev->otherend) ==
@@ -2186,10 +2195,14 @@ static int xennet_remove(struct xenbus_device *dev)
xennet_disconnect_backend(info);
- unregister_netdev(info->netdev);
+ if (info->netdev->reg_state == NETREG_REGISTERED)
+ unregister_netdev(info->netdev);
- if (info->queues)
+ if (info->queues) {
+ rtnl_lock();
xennet_destroy_queues(info);
+ rtnl_unlock();
+ }
xennet_free_netdev(info->netdev);
return 0;
diff --git a/drivers/nfc/Kconfig b/drivers/nfc/Kconfig
index e1c6f99..22a13d35 100644
--- a/drivers/nfc/Kconfig
+++ b/drivers/nfc/Kconfig
@@ -72,9 +72,19 @@
endmenu
config NFC_NQ
- tristate "QTI NCI based NFC Controller Driver for NQx"
- depends on I2C
- help
- This enables the NFC driver for NQx based devices.
- This is for i2c connected version. NCI protocol logic
- resides in the usermode and it has no other NFC dependencies.
+ tristate "QTI NCI based NFC Controller Driver for NQx"
+ depends on I2C
+ help
+ This enables the NFC driver for NQx based devices.
+ This is for i2c connected version. NCI protocol logic
+ resides in the usermode and it has no other NFC dependencies.
+
+config NFC_NQ_PN81A
+ tristate "NFC eSE driver for NXP PN81A using NFC_NQ"
+ depends on I2C
+ depends on SPI
+ depends on NFC_NQ
+ help
+ This enables the SPI driver for the NXP PN81A-based embedded
+ secure element using the NFC_NQ driver to control the power.
+
diff --git a/drivers/nfc/Makefile b/drivers/nfc/Makefile
index b691fd4..b0d43e6 100644
--- a/drivers/nfc/Makefile
+++ b/drivers/nfc/Makefile
@@ -18,3 +18,4 @@
obj-$(CONFIG_NFC_S3FWRN5) += s3fwrn5/
obj-$(CONFIG_NFC_ST95HF) += st95hf/
obj-$(CONFIG_NFC_NQ) += nq-nci.o
+obj-$(CONFIG_NFC_NQ_PN81A) += ese/
diff --git a/drivers/nfc/ese/Makefile b/drivers/nfc/ese/Makefile
new file mode 100644
index 0000000..1907c21
--- /dev/null
+++ b/drivers/nfc/ese/Makefile
@@ -0,0 +1,5 @@
+#
+# Makefile for NXP PN81A ESE driver
+#
+
+obj-$(CONFIG_NFC_NQ_PN81A) += pn81a.o
diff --git a/drivers/nfc/ese/pn81a.c b/drivers/nfc/ese/pn81a.c
new file mode 100644
index 0000000..4d2de15
--- /dev/null
+++ b/drivers/nfc/ese/pn81a.c
@@ -0,0 +1,390 @@
+/* drivers/nfc/ese/pn81a.c
+ *
+ * Copyright 2017 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/fs.h>
+
+#include <linux/delay.h>
+#include <linux/gpio.h>
+#include <linux/of_gpio.h>
+#include <linux/of_device.h>
+#include <linux/spi/spi.h>
+#include <linux/uaccess.h>
+
+#ifdef CONFIG_COMPAT
+#include <linux/compat.h>
+#endif
+
+/* Relies on struct nfcc_data and nqx_ese_pwr().
+ * This can easily be made generic is a function is exported
+ * that happily takes a void *.
+ */
+#include "../nq-nci.h"
+
+#define PN81A_MAX_BUF 258U
+
+struct ese_dev {
+ struct spi_device *spi;
+ struct device *nfcc_device;
+ struct nqx_dev *nfcc_data;
+ struct mutex mutex;
+ struct miscdevice device;
+ int gpio_clear_n;
+ const char *nfcc_name;
+};
+
+static long ese_clear_gpio(struct ese_dev *ese_dev, unsigned long arg)
+{
+ long r = 0;
+ int val = arg ? 1 : 0;
+ if (ese_dev->gpio_clear_n == -EINVAL) {
+ r = -ENODEV;
+ } else {
+ gpio_set_value(ese_dev->gpio_clear_n, val);
+ usleep_range(1000, 1100);
+ if (gpio_get_value(ese_dev->gpio_clear_n) != val) {
+ dev_err(&ese_dev->spi->dev,
+ "gpio_clear_n won't change to %d\n", val);
+ r = -EBUSY;
+ } else {
+ dev_info(&ese_dev->spi->dev,
+ "%s: gpio_clear_n changed: %d\n",
+ __func__, val);
+ }
+ }
+ return r;
+}
+
+#ifdef CONFIG_COMPAT
+static long ese_compat_ioctl(struct file *filp, unsigned int cmd,
+ unsigned long arg)
+{
+ long r = 0;
+ struct ese_dev *ese_dev = filp->private_data;
+ mutex_lock(&ese_dev->mutex);
+
+ arg = (compat_u64)arg;
+ switch (cmd) {
+ case ESE_SET_PWR:
+ nqx_ese_pwr(ese_dev->nfcc_data, arg);
+ break;
+ case ESE_GET_PWR:
+ nqx_ese_pwr(ese_dev->nfcc_data, 3);
+ break;
+ case ESE_CLEAR_GPIO:
+ r = ese_clear_gpio(ese_dev, arg);
+ break;
+ default:
+ r = -ENOTTY;
+ }
+ mutex_unlock(&ese_dev->mutex);
+ return r;
+}
+#endif
+
+static long ese_ioctl(struct file *filp, unsigned int cmd,
+ unsigned long arg)
+{
+ int r = 0;
+ struct ese_dev *ese_dev = filp->private_data;
+ mutex_lock(&ese_dev->mutex);
+
+ switch (cmd) {
+ case ESE_SET_PWR:
+ r = nqx_ese_pwr(ese_dev->nfcc_data, arg);
+ break;
+ case ESE_GET_PWR:
+ r = nqx_ese_pwr(ese_dev->nfcc_data, 3);
+ break;
+ case ESE_CLEAR_GPIO:
+ r = ese_clear_gpio(ese_dev, arg);
+ break;
+ default:
+ r = -ENOIOCTLCMD;
+ }
+ mutex_unlock(&ese_dev->mutex);
+ return r;
+}
+
+static int ese_open(struct inode *inode, struct file *filp)
+{
+ int pwr = 0;
+ struct ese_dev *ese_dev = container_of(filp->private_data,
+ struct ese_dev, device);
+
+ mutex_lock(&ese_dev->mutex);
+ /* Find the NFC parent device if it exists. */
+ if (ese_dev->nfcc_data == NULL) {
+ struct device *nfc_dev = bus_find_device_by_name(
+ &i2c_bus_type,
+ NULL,
+ ese_dev->nfcc_name);
+ if (!nfc_dev) {
+ dev_err(&ese_dev->spi->dev,
+ "%s: cannot find NFC controller '%s'\n",
+ __func__, ese_dev->nfcc_name);
+ goto err;
+ }
+ ese_dev->nfcc_data = dev_get_drvdata(nfc_dev);
+ if (!ese_dev->nfcc_data) {
+ dev_err(&ese_dev->spi->dev,
+ "%s: cannot find NFC controller device data\n",
+ __func__);
+ put_device(nfc_dev);
+ goto err;
+ }
+ dev_info(&ese_dev->spi->dev,
+ "%s: NFC controller found\n", __func__);
+ ese_dev->nfcc_device = nfc_dev;
+ }
+ mutex_unlock(&ese_dev->mutex);
+
+ if (nqx_claim_ese(ese_dev->nfcc_data, true) != 0)
+ return -EBUSY;
+ filp->private_data = ese_dev;
+ dev_dbg(&ese_dev->spi->dev,
+ "%s: major,minor: %d,%d\n",
+ __func__, imajor(inode), iminor(inode));
+
+ /* Note, opening and closing is treated wholly independently
+ * from power management. This ensures that the eSE can go to
+ * a deep power down state as dictated by the software stack.
+ */
+ pwr = ese_ioctl(filp, ESE_GET_PWR, 0);
+ dev_dbg(&ese_dev->spi->dev,
+ "%s: eSE opened (power: %d)\n", __func__, pwr);
+
+ return 0;
+
+err:
+ mutex_unlock(&ese_dev->mutex);
+ return -ENODEV;
+}
+
+static int ese_release(struct inode *ino, struct file *filp)
+{
+ struct ese_dev *ese_dev = filp->private_data;
+ int pwr = 0;
+
+ nqx_claim_ese(ese_dev->nfcc_data, false);
+
+ pwr = ese_ioctl(filp, ESE_GET_PWR, 0);
+
+ mutex_lock(&ese_dev->mutex);
+ dev_dbg(&ese_dev->spi->dev,
+ "%s: power: %d\n", __func__, pwr);
+ mutex_unlock(&ese_dev->mutex);
+ return 0;
+}
+
+static ssize_t ese_write(struct file *filp, const char __user *ubuf,
+ size_t len, loff_t *offset)
+{
+ struct ese_dev *ese_dev = filp->private_data;
+ ssize_t ret = -EFAULT;
+ size_t bytes = len;
+ char tx_buf[PN81A_MAX_BUF];
+
+ if (len > INT_MAX)
+ return -EINVAL;
+
+ mutex_lock(&ese_dev->mutex);
+ while (bytes > 0) {
+ size_t block = bytes < sizeof(tx_buf) ? bytes : sizeof(tx_buf);
+
+ memset(tx_buf, 0, sizeof(tx_buf));
+ if (copy_from_user(tx_buf, ubuf, block)) {
+ dev_dbg(&ese_dev->spi->dev, "failed to copy from user\n");
+ goto err;
+ }
+ ret = spi_write(ese_dev->spi, tx_buf, block);
+ if (ret < 0) {
+ dev_dbg(&ese_dev->spi->dev, "failed to write to SPI\n");
+ goto err;
+ }
+ ubuf += block;
+ bytes -= block;
+ }
+ ret = len;
+err:
+ mutex_unlock(&ese_dev->mutex);
+ return ret;
+}
+
+static ssize_t ese_read(struct file *filp, char __user *ubuf,
+ size_t len, loff_t *offset)
+{
+ struct ese_dev *ese_dev = filp->private_data;
+ ssize_t ret = -EFAULT;
+ size_t bytes = len;
+ char rx_buf[PN81A_MAX_BUF];
+
+ if (len > INT_MAX)
+ return -EINVAL;
+ mutex_lock(&ese_dev->mutex);
+ while (bytes > 0) {
+ size_t block = bytes < sizeof(rx_buf) ? bytes : sizeof(rx_buf);
+
+ memset(rx_buf, 0, sizeof(rx_buf));
+ ret = spi_read(ese_dev->spi, rx_buf, block);
+ if (ret < 0) {
+ dev_dbg(&ese_dev->spi->dev, "failed to read from SPI\n");
+ goto err;
+ }
+ if (copy_to_user(ubuf, rx_buf, block)) {
+ dev_dbg(&ese_dev->spi->dev, "failed to copy from user\n");
+ goto err;
+ }
+ ubuf += block;
+ bytes -= block;
+ }
+ ret = len;
+err:
+ mutex_unlock(&ese_dev->mutex);
+ return ret;
+}
+
+static const struct file_operations ese_dev_fops = {
+ .owner = THIS_MODULE,
+ .llseek = no_llseek,
+ .read = ese_read,
+ .write = ese_write,
+ .open = ese_open,
+ .release = ese_release,
+ .unlocked_ioctl = ese_ioctl,
+#ifdef CONFIG_COMPAT
+ .compat_ioctl = ese_compat_ioctl
+#endif
+};
+
+static int pn81a_probe(struct spi_device *spi)
+{
+ struct ese_dev *ese_dev;
+ struct device_node *np = dev_of_node(&spi->dev);
+ int ret;
+ int gpio;
+
+ dev_dbg(&spi->dev, "%s: called\n", __func__);
+
+ if (!np) {
+ dev_err(&spi->dev, "%s: device tree data missing\n", __func__);
+ return -EINVAL;
+ }
+
+ ese_dev = kzalloc(sizeof(*ese_dev), GFP_KERNEL);
+ if (ese_dev == NULL)
+ return -ENOMEM;
+
+ ese_dev->spi = spi;
+ ese_dev->device.minor = MISC_DYNAMIC_MINOR;
+ ese_dev->device.name = "pn81a";
+ ese_dev->device.fops = &ese_dev_fops;
+
+ spi->bits_per_word = 8;
+
+ gpio = of_get_named_gpio(np, "nxp,clear-n", 0);
+ if (!gpio_is_valid(gpio)) {
+ dev_warn(&spi->dev,
+ "%s: nxp,clear-n invalid or missing in device tree\n",
+ __func__);
+ gpio = -EINVAL;
+ }
+ ese_dev->gpio_clear_n = gpio;
+ if (ese_dev->gpio_clear_n != -EINVAL) {
+ ret = gpio_request(ese_dev->gpio_clear_n, "ese_clear_n");
+ if (ret) {
+ dev_warn(&spi->dev,
+ "%s: unable to request ese clear n gpio [%d]\n",
+ __func__,
+ gpio);
+ ese_dev->gpio_clear_n = -EINVAL;
+ goto skip_gpio;
+ }
+ ret = gpio_direction_output(ese_dev->gpio_clear_n, 1);
+ if (ret) {
+ dev_warn(&spi->dev,
+ "%s: failed to set direction for ese clear n[%d]\n",
+ __func__,
+ ese_dev->gpio_clear_n);
+ }
+ }
+
+skip_gpio:
+ mutex_init(&ese_dev->mutex);
+ ret = of_property_read_string(np, "nxp,nfcc", &ese_dev->nfcc_name);
+ if (ret < 0) {
+ dev_err(&spi->dev,
+ "%s: nxp,nfcc invalid or missing in device tree (%d)\n",
+ __func__, ret);
+ goto err;
+ }
+ dev_info(&spi->dev, "%s: device tree set '%s' as eSE power controller\n",
+ __func__, ese_dev->nfcc_name);
+
+ ret = misc_register(&ese_dev->device);
+ if (ret) {
+ dev_err(&spi->dev, "%s: misc_register failed\n", __func__);
+ goto err;
+ }
+ dev_info(&spi->dev, "%s: eSE is configured\n", __func__);
+ spi_set_drvdata(spi, ese_dev);
+
+ return 0;
+err:
+ mutex_destroy(&ese_dev->mutex);
+ kfree(ese_dev);
+ return ret;
+}
+
+static int pn81a_remove(struct spi_device *spi)
+{
+ struct ese_dev *ese_dev = spi_get_drvdata(spi);
+ int ret = 0;
+
+ if (!ese_dev) {
+ dev_err(&spi->dev,
+ "%s: device doesn't exist anymore\n", __func__);
+ ret = -ENODEV;
+ goto err;
+ }
+ /* If we have a NFC device, release it. */
+ if (ese_dev->nfcc_device) {
+ put_device(ese_dev->nfcc_device);
+ ese_dev->nfcc_data = NULL;
+ ese_dev->nfcc_device = NULL;
+ }
+ misc_deregister(&ese_dev->device);
+ mutex_destroy(&ese_dev->mutex);
+ if (ese_dev->gpio_clear_n != -EINVAL)
+ gpio_free(ese_dev->gpio_clear_n);
+ kfree(ese_dev);
+err:
+ return ret;
+}
+
+static const struct of_device_id pn81a_match_table[] = {
+ { .compatible = "nxp,pn81a" },
+ { }
+};
+MODULE_DEVICE_TABLE(of, pn81a_match_table);
+
+static struct spi_driver pn81a_driver = {
+ .driver = {
+ .name = "pn81a",
+ .of_match_table = pn81a_match_table,
+ },
+ .probe = pn81a_probe,
+ .remove = pn81a_remove,
+};
+module_spi_driver(pn81a_driver);
+
+MODULE_DESCRIPTION("PN81A eSE driver");
+MODULE_ALIAS("spi:pn81a");
+MODULE_AUTHOR("Google Inc");
+MODULE_LICENSE("GPL");
diff --git a/drivers/nfc/nfcmrvl/uart.c b/drivers/nfc/nfcmrvl/uart.c
index 6c0c301..1b11ded 100644
--- a/drivers/nfc/nfcmrvl/uart.c
+++ b/drivers/nfc/nfcmrvl/uart.c
@@ -73,10 +73,9 @@ static int nfcmrvl_uart_parse_dt(struct device_node *node,
struct device_node *matched_node;
int ret;
- matched_node = of_find_compatible_node(node, NULL, "marvell,nfc-uart");
+ matched_node = of_get_compatible_child(node, "marvell,nfc-uart");
if (!matched_node) {
- matched_node = of_find_compatible_node(node, NULL,
- "mrvl,nfc-uart");
+ matched_node = of_get_compatible_child(node, "mrvl,nfc-uart");
if (!matched_node)
return -ENODEV;
}
diff --git a/drivers/nfc/nq-nci.c b/drivers/nfc/nq-nci.c
index 0280d42..d126a88 100644
--- a/drivers/nfc/nq-nci.c
+++ b/drivers/nfc/nq-nci.c
@@ -35,6 +35,8 @@ struct nqx_platform_data {
unsigned int clkreq_gpio;
unsigned int firm_gpio;
unsigned int ese_gpio;
+ int i2c_postinit_cmd_len;
+ u8 *i2c_postinit_cmd;
const char *clk_src_name;
/* NFC_CLK pin voting state */
bool clk_pin_voting;
@@ -54,6 +56,7 @@ MODULE_DEVICE_TABLE(of, msm_match_table);
struct nqx_dev {
wait_queue_head_t read_wq;
struct mutex read_mutex;
+ struct mutex ese_status_mutex;
struct i2c_client *client;
struct miscdevice nqx_device;
union nqx_uinfo nqx_info;
@@ -63,6 +66,9 @@ struct nqx_dev {
unsigned int firm_gpio;
unsigned int clkreq_gpio;
unsigned int ese_gpio;
+ /* ese usage and gpio information */
+ unsigned int ese_gpio_stored;
+ bool ese_in_use; /* protected by ese_status_mutex */
/* NFC VEN pin state powered by Nfc */
bool nfc_ven_enabled;
/* NFC_IRQ state */
@@ -99,6 +105,7 @@ unsigned int disable_ctrl;
static void nqx_init_stat(struct nqx_dev *nqx_dev)
{
nqx_dev->count_irq = 0;
+ nqx_dev->ese_gpio_stored = 0;
}
static void nqx_disable_irq(struct nqx_dev *nqx_dev)
@@ -270,7 +277,7 @@ static ssize_t nfc_write(struct file *filp, const char __user *buf,
ret = i2c_master_send(nqx_dev->client, tmp, count);
if (ret != count) {
- dev_err(&nqx_dev->client->dev,
+ dev_dbg(&nqx_dev->client->dev,
"%s: failed to write %d\n", __func__, ret);
ret = -EIO;
goto out_free;
@@ -288,33 +295,34 @@ static ssize_t nfc_write(struct file *filp, const char __user *buf,
return ret;
}
-/**
- * nqx_standby_write()
- * @buf: pointer to data buffer
- * @len: # of bytes need to transfer
+/* Callback to indicate if secure element is in use.
+ * @claim: true/false indicating claim/release
*
- * write data buffer over I2C and retry
- * if NFCC is in stand by mode
- *
- * Return: # of bytes written or -ve value in case of error
+ * returns 0 in case request was successful, otherwise -1 if ese was forced low
+ * due to firmware download mode.
*/
-static int nqx_standby_write(struct nqx_dev *nqx_dev,
- const unsigned char *buf, size_t len)
+int nqx_claim_ese(struct nqx_dev *nqx_dev, bool claim)
{
- int ret = -EINVAL;
- int retry_cnt;
-
- for (retry_cnt = 1; retry_cnt <= MAX_RETRY_COUNT; retry_cnt++) {
- ret = i2c_master_send(nqx_dev->client, buf, len);
- if (ret < 0) {
- dev_err(&nqx_dev->client->dev,
- "%s: write failed, Maybe in Standby Mode - Retry(%d)\n",
- __func__, retry_cnt);
- usleep_range(1000, 1100);
- } else if (ret == len)
- break;
+ mutex_lock(&nqx_dev->ese_status_mutex);
+ if (claim) {
+ /* currently in download mode,
+ * we cannot start an ese transactionx
+ */
+ if (gpio_is_valid(nqx_dev->firm_gpio) &&
+ gpio_get_value(nqx_dev->firm_gpio)) {
+ dev_err(&nqx_dev->client->dev, "%s: claim ese failed\n",
+ __func__);
+ mutex_unlock(&nqx_dev->ese_status_mutex);
+ return -EBUSY;
+ }
+ dev_dbg(&nqx_dev->client->dev, "%s: ese claimed\n", __func__);
+ nqx_dev->ese_in_use = true;
+ } else {
+ dev_dbg(&nqx_dev->client->dev, "%s: ese released\n", __func__);
+ nqx_dev->ese_in_use = false;
}
- return ret;
+ mutex_unlock(&nqx_dev->ese_status_mutex);
+ return 0;
}
/*
@@ -323,12 +331,11 @@ static int nqx_standby_write(struct nqx_dev *nqx_dev,
* NFC OFF & eSE ON : NFC_EN high and eSE_pwr_req high.
* NFC OFF & eSE OFF : NFC_EN low and eSE_pwr_req low.
*/
-static int nqx_ese_pwr(struct nqx_dev *nqx_dev, unsigned long int arg)
+int nqx_ese_pwr(struct nqx_dev *nqx_dev, unsigned long int arg)
{
int r = -1;
- const unsigned char svdd_off_cmd_warn[] = {0x2F, 0x31, 0x01, 0x01};
- const unsigned char svdd_off_cmd_done[] = {0x2F, 0x31, 0x01, 0x00};
+ dev_dbg(&nqx_dev->client->dev, "%s: %lu\n", __func__, arg);
if (!gpio_is_valid(nqx_dev->ese_gpio)) {
dev_err(&nqx_dev->client->dev,
"%s: ese_gpio is not valid\n", __func__);
@@ -351,66 +358,30 @@ static int nqx_ese_pwr(struct nqx_dev *nqx_dev, unsigned long int arg)
nqx_dev->nfc_ven_enabled =
gpio_get_value(nqx_dev->en_gpio);
if (!nqx_dev->nfc_ven_enabled) {
+ dev_dbg(&nqx_dev->client->dev, "setting NFC VEN high\n");
gpio_set_value(nqx_dev->en_gpio, 1);
/* hardware dependent delay */
usleep_range(1000, 1100);
+ if (gpio_get_value(nqx_dev->en_gpio) == 0)
+ dev_dbg(&nqx_dev->client->dev, "NFC VEN still low\n");
}
+ dev_err(&nqx_dev->client->dev, "setting ese_gpio high\n");
gpio_set_value(nqx_dev->ese_gpio, 1);
- usleep_range(1000, 1100);
+ /* hardware dependent delay */
+ usleep_range(5000, 5100);
if (gpio_get_value(nqx_dev->ese_gpio)) {
dev_dbg(&nqx_dev->client->dev, "ese_gpio is enabled\n");
r = 0;
}
}
} else if (arg == 1) {
- if (nqx_dev->nfc_ven_enabled &&
- ((nqx_dev->nqx_info.info.chip_type == NFCC_NQ_220) ||
- (nqx_dev->nqx_info.info.chip_type == NFCC_PN66T))) {
- /**
- * Let's inform the CLF we're
- * powering off the eSE
- */
- r = nqx_standby_write(nqx_dev, svdd_off_cmd_warn,
- sizeof(svdd_off_cmd_warn));
- if (r < 0) {
- dev_err(&nqx_dev->client->dev,
- "%s: write failed after max retry\n",
- __func__);
- return -ENXIO;
- }
- dev_dbg(&nqx_dev->client->dev,
- "%s: svdd_off_cmd_warn sent\n", __func__);
-
- /* let's power down the eSE */
- gpio_set_value(nqx_dev->ese_gpio, 0);
- dev_dbg(&nqx_dev->client->dev,
- "%s: nqx_dev->ese_gpio set to 0\n", __func__);
-
- /**
- * Time needed for the SVDD capacitor
- * to get discharged
- */
- usleep_range(8000, 8100);
-
- /* Let's inform the CLF the eSE is now off */
- r = nqx_standby_write(nqx_dev, svdd_off_cmd_done,
- sizeof(svdd_off_cmd_done));
- if (r < 0) {
- dev_err(&nqx_dev->client->dev,
- "%s: write failed after max retry\n",
- __func__);
- return -ENXIO;
- }
- dev_dbg(&nqx_dev->client->dev,
- "%s: svdd_off_cmd_done sent\n", __func__);
- } else {
- /**
- * In case the NFC is off,
- * there's no need to send the i2c commands
- */
- gpio_set_value(nqx_dev->ese_gpio, 0);
- usleep_range(1000, 1100);
- }
+ r = 0;
+ /**
+ * In case the NFC is off,
+ * there's no need to send the i2c commands
+ */
+ gpio_set_value(nqx_dev->ese_gpio, 0);
+ dev_dbg(&nqx_dev->client->dev, "ese_gpio is disabled\n");
if (!gpio_get_value(nqx_dev->ese_gpio)) {
dev_dbg(&nqx_dev->client->dev, "ese_gpio is disabled\n");
@@ -461,6 +432,7 @@ int nfc_ioctl_power_states(struct file *filp, unsigned long arg)
int r = 0;
struct nqx_dev *nqx_dev = filp->private_data;
+ dev_dbg(&nqx_dev->client->dev, "%s: %lu\n", __func__, arg);
if (arg == 0) {
/*
* We are attempting a hardware reset so let us disable
@@ -471,10 +443,23 @@ int nfc_ioctl_power_states(struct file *filp, unsigned long arg)
dev_dbg(&nqx_dev->client->dev,
"gpio_set_value disable: %s: info: %p\n",
__func__, nqx_dev);
- if (gpio_is_valid(nqx_dev->firm_gpio)) {
+ mutex_lock(&nqx_dev->ese_status_mutex);
+ if (gpio_is_valid(nqx_dev->firm_gpio) &&
+ gpio_get_value(nqx_dev->firm_gpio)) {
+ if (gpio_is_valid(nqx_dev->ese_gpio) &&
+ gpio_get_value(nqx_dev->ese_gpio) !=
+ nqx_dev->ese_gpio_stored) {
+ dev_dbg(&nqx_dev->client->dev,
+ "%s: Restoring ese state %d\n",
+ __func__,
+ nqx_dev->ese_gpio_stored);
+ gpio_set_value(nqx_dev->ese_gpio,
+ nqx_dev->ese_gpio_stored);
+ }
gpio_set_value(nqx_dev->firm_gpio, 0);
usleep_range(10000, 10100);
}
+ mutex_unlock(&nqx_dev->ese_status_mutex);
if (gpio_is_valid(nqx_dev->ese_gpio)) {
if (!gpio_get_value(nqx_dev->ese_gpio)) {
@@ -494,9 +479,15 @@ int nfc_ioctl_power_states(struct file *filp, unsigned long arg)
if (r < 0)
dev_err(&nqx_dev->client->dev, "unable to disable clock\n");
}
+ if (gpio_is_valid(nqx_dev->clkreq_gpio)) {
+ r = nqx_clock_deselect(nqx_dev);
+ if (r < 0)
+ dev_err(&nqx_dev->client->dev, "unable to disable clock\n");
+ }
nqx_dev->nfc_ven_enabled = false;
} else if (arg == 1) {
nqx_enable_irq(nqx_dev);
+
dev_dbg(&nqx_dev->client->dev,
"gpio_set_value enable: %s: info: %p\n",
__func__, nqx_dev);
@@ -517,13 +508,24 @@ int nfc_ioctl_power_states(struct file *filp, unsigned long arg)
* We are switching to Dowload Mode, toggle the enable pin
* in order to set the NFCC in the new mode
*/
+ mutex_lock(&nqx_dev->ese_status_mutex);
if (gpio_is_valid(nqx_dev->ese_gpio)) {
- if (gpio_get_value(nqx_dev->ese_gpio)) {
+ if (nqx_dev->ese_in_use) {
dev_err(&nqx_dev->client->dev,
- "FW download forbidden while ese is on\n");
+ "FW download forbidden while ese is in use\n");
+ mutex_unlock(&nqx_dev->ese_status_mutex);
return -EBUSY; /* Device or resource busy */
}
+ nqx_dev->ese_gpio_stored =
+ gpio_get_value(nqx_dev->ese_gpio);
+ dev_err(&nqx_dev->client->dev,
+ "FW download backing up se state: %d\n",
+ nqx_dev->ese_gpio_stored);
+ if (nqx_dev->ese_gpio_stored == 1)
+ gpio_set_value(nqx_dev->ese_gpio, 0);
}
+ mutex_unlock(&nqx_dev->ese_status_mutex);
+
gpio_set_value(nqx_dev->en_gpio, 1);
usleep_range(10000, 10100);
if (gpio_is_valid(nqx_dev->firm_gpio)) {
@@ -648,131 +650,6 @@ static const struct file_operations nfc_dev_fops = {
#endif
};
-/* Check for availability of NQ_ NFC controller hardware */
-static int nfcc_hw_check(struct i2c_client *client, struct nqx_dev *nqx_dev)
-{
- int ret = 0;
-
- unsigned char raw_nci_reset_cmd[] = {0x20, 0x00, 0x01, 0x00};
- unsigned char raw_nci_init_cmd[] = {0x20, 0x01, 0x00};
- unsigned char nci_init_rsp[28];
- unsigned char nci_reset_rsp[6];
- unsigned char init_rsp_len = 0;
- unsigned int enable_gpio = nqx_dev->en_gpio;
-
- /* making sure that the NFCC starts in a clean state. */
- gpio_set_value(enable_gpio, 0);/* ULPM: Disable */
- /* hardware dependent delay */
- usleep_range(10000, 10100);
- gpio_set_value(enable_gpio, 1);/* HPD : Enable*/
- /* hardware dependent delay */
- usleep_range(10000, 10100);
-
- /* send NCI CORE RESET CMD with Keep Config parameters */
- ret = i2c_master_send(client, raw_nci_reset_cmd,
- sizeof(raw_nci_reset_cmd));
- if (ret < 0) {
- dev_err(&client->dev,
- "%s: - i2c_master_send Error\n", __func__);
- goto err_nfcc_hw_check;
- }
- /* hardware dependent delay */
- msleep(30);
-
- /* Read Response of RESET command */
- ret = i2c_master_recv(client, nci_reset_rsp,
- sizeof(nci_reset_rsp));
- if (ret < 0) {
- dev_err(&client->dev,
- "%s: - i2c_master_recv Error\n", __func__);
- goto err_nfcc_hw_check;
- }
- ret = nqx_standby_write(nqx_dev, raw_nci_init_cmd,
- sizeof(raw_nci_init_cmd));
- if (ret < 0) {
- dev_err(&client->dev,
- "%s: - i2c_master_send Error\n", __func__);
- goto err_nfcc_core_init_fail;
- }
- /* hardware dependent delay */
- msleep(30);
- /* Read Response of INIT command */
- ret = i2c_master_recv(client, nci_init_rsp,
- sizeof(nci_init_rsp));
- if (ret < 0) {
- dev_err(&client->dev,
- "%s: - i2c_master_recv Error\n", __func__);
- goto err_nfcc_core_init_fail;
- }
- init_rsp_len = 2 + nci_init_rsp[2]; /*payload + len*/
- if (init_rsp_len > PAYLOAD_HEADER_LENGTH) {
- nqx_dev->nqx_info.info.chip_type =
- nci_init_rsp[init_rsp_len - 3];
- nqx_dev->nqx_info.info.rom_version =
- nci_init_rsp[init_rsp_len - 2];
- nqx_dev->nqx_info.info.fw_major =
- nci_init_rsp[init_rsp_len - 1];
- nqx_dev->nqx_info.info.fw_minor =
- nci_init_rsp[init_rsp_len];
- }
- dev_dbg(&client->dev,
- "%s: - nq - reset cmd answer : NfcNciRx %x %x %x\n",
- __func__, nci_reset_rsp[0],
- nci_reset_rsp[1], nci_reset_rsp[2]);
-
- dev_dbg(&nqx_dev->client->dev, "NQ NFCC chip_type = %x\n",
- nqx_dev->nqx_info.info.chip_type);
- dev_dbg(&nqx_dev->client->dev, "NQ fw version = %x.%x.%x\n",
- nqx_dev->nqx_info.info.rom_version,
- nqx_dev->nqx_info.info.fw_major,
- nqx_dev->nqx_info.info.fw_minor);
-
- switch (nqx_dev->nqx_info.info.chip_type) {
- case NFCC_NQ_210:
- dev_dbg(&client->dev,
- "%s: ## NFCC == NQ210 ##\n", __func__);
- break;
- case NFCC_NQ_220:
- dev_dbg(&client->dev,
- "%s: ## NFCC == NQ220 ##\n", __func__);
- break;
- case NFCC_NQ_310:
- dev_dbg(&client->dev,
- "%s: ## NFCC == NQ310 ##\n", __func__);
- break;
- case NFCC_NQ_330:
- dev_dbg(&client->dev,
- "%s: ## NFCC == NQ330 ##\n", __func__);
- break;
- case NFCC_PN66T:
- dev_dbg(&client->dev,
- "%s: ## NFCC == PN66T ##\n", __func__);
- break;
- default:
- dev_err(&client->dev,
- "%s: - NFCC HW not Supported\n", __func__);
- break;
- }
-
- /*Disable NFC by default to save power on boot*/
- gpio_set_value(enable_gpio, 0);/* ULPM: Disable */
- ret = 0;
- goto done;
-
-err_nfcc_core_init_fail:
- dev_err(&client->dev,
- "%s: - nq - reset cmd answer : NfcNciRx %x %x %x\n",
- __func__, nci_reset_rsp[0],
- nci_reset_rsp[1], nci_reset_rsp[2]);
-
-err_nfcc_hw_check:
- ret = -ENXIO;
- dev_err(&client->dev,
- "%s: - NFCC HW not available\n", __func__);
-done:
- return ret;
-}
-
/*
* Routine to enable clock.
* this routine can be extended to select from multiple
@@ -847,6 +724,31 @@ static int nfc_parse_dt(struct device *dev, struct nqx_platform_data *pdata)
pdata->ese_gpio = -EINVAL;
}
+ /* First byte is the expected response length. */
+ /* E.g., to send a swp_svdd_cfg command to a pn5xx:
+ * qcom,nq-postinit-cmd = /bits/ 8
+ * <0x5 0x20 0x02 0x05 0x01 0xA0 0xF2 0x01 0x01>;
+ */
+ pdata->i2c_postinit_cmd_len = of_property_count_elems_of_size(np,
+ "qcom,nq-postinit-cmd",
+ sizeof(u8));
+ if (pdata->i2c_postinit_cmd_len > 1) {
+ pdata->i2c_postinit_cmd = devm_kzalloc(dev,
+ pdata->i2c_postinit_cmd_len, GFP_KERNEL);
+ if (!pdata->i2c_postinit_cmd)
+ pdata->i2c_postinit_cmd_len = 0;
+ }
+
+ if (pdata->i2c_postinit_cmd) {
+ r = of_property_read_u8_array(np, "qcom,nq-postinit-cmd",
+ pdata->i2c_postinit_cmd, pdata->i2c_postinit_cmd_len);
+ if (r) {
+ dev_warn(dev, "nq-postinit-cmd <OPTIONAL> error ");
+ dev_warn(dev, " getting from OF node\n");
+ devm_kfree(dev, pdata->i2c_postinit_cmd);
+ }
+ }
+
if (of_property_read_string(np, "qcom,clk-src", &pdata->clk_src_name))
pdata->clk_pin_voting = false;
else
@@ -1044,7 +946,6 @@ static int nqx_probe(struct i2c_client *client,
} else {
dev_err(&client->dev,
"%s: clkreq gpio not provided\n", __func__);
- goto err_ese_gpio;
}
nqx_dev->en_gpio = platform_data->en_gpio;
@@ -1056,6 +957,7 @@ static int nqx_probe(struct i2c_client *client,
/* init mutex and queues */
init_waitqueue_head(&nqx_dev->read_wq);
mutex_init(&nqx_dev->read_mutex);
+ mutex_init(&nqx_dev->ese_status_mutex);
spin_lock_init(&nqx_dev->irq_enabled_lock);
nqx_dev->nqx_device.minor = MISC_DYNAMIC_MINOR;
@@ -1078,19 +980,6 @@ static int nqx_probe(struct i2c_client *client,
}
nqx_disable_irq(nqx_dev);
- /*
- * To be efficient we need to test whether nfcc hardware is physically
- * present before attempting further hardware initialisation.
- *
- */
- r = nfcc_hw_check(client, nqx_dev);
- if (r) {
- /* make sure NFCC is not enabled */
- gpio_set_value(platform_data->en_gpio, 0);
- /* We don't think there is hardware switch NFC OFF */
- goto err_request_hw_check_failed;
- }
-
/* Register reboot notifier here */
r = register_reboot_notifier(&nfcc_notifier);
if (r) {
@@ -1133,6 +1022,7 @@ static int nqx_probe(struct i2c_client *client,
misc_deregister(&nqx_dev->nqx_device);
err_misc_register:
mutex_destroy(&nqx_dev->read_mutex);
+ mutex_destroy(&nqx_dev->ese_status_mutex);
err_clkreq_gpio:
gpio_free(platform_data->clkreq_gpio);
err_ese_gpio:
@@ -1150,8 +1040,12 @@ static int nqx_probe(struct i2c_client *client,
err_free_dev:
kfree(nqx_dev);
err_free_data:
- if (client->dev.of_node)
+ if (client->dev.of_node) {
+ if (platform_data->i2c_postinit_cmd)
+ devm_kfree(&client->dev,
+ platform_data->i2c_postinit_cmd);
devm_kfree(&client->dev, platform_data);
+ }
err_platform_data:
dev_err(&client->dev,
"%s: probing nqxx failed, check hardware\n",
@@ -1176,6 +1070,7 @@ static int nqx_remove(struct i2c_client *client)
free_irq(client->irq, nqx_dev);
misc_deregister(&nqx_dev->nqx_device);
mutex_destroy(&nqx_dev->read_mutex);
+ mutex_destroy(&nqx_dev->ese_status_mutex);
gpio_free(nqx_dev->clkreq_gpio);
/* optional gpio, not sure was configured in probe */
if (nqx_dev->ese_gpio > 0)
@@ -1184,6 +1079,9 @@ static int nqx_remove(struct i2c_client *client)
gpio_free(nqx_dev->irq_gpio);
gpio_free(nqx_dev->en_gpio);
kfree(nqx_dev->kbuf);
+ if (nqx_dev->pdata->i2c_postinit_cmd)
+ devm_kfree(&client->dev,
+ nqx_dev->pdata->i2c_postinit_cmd);
if (client->dev.of_node)
devm_kfree(&client->dev, nqx_dev->pdata);
diff --git a/drivers/nfc/nq-nci.h b/drivers/nfc/nq-nci.h
index 87715c2..d66f8e7 100644
--- a/drivers/nfc/nq-nci.h
+++ b/drivers/nfc/nq-nci.h
@@ -30,6 +30,7 @@
#define SET_RX_BLOCK _IOW(0xE9, 0x04, unsigned int)
#define SET_EMULATOR_TEST_POINT _IOW(0xE9, 0x05, unsigned int)
#define NFCC_INITIAL_CORE_RESET_NTF _IOW(0xE9, 0x10, unsigned int)
+#define ESE_CLEAR_GPIO _IOW(0xE9, 0x11, unsigned int)
#define NFC_RX_BUFFER_CNT_START (0x0)
#define PAYLOAD_HEADER_LENGTH (0x3)
@@ -51,4 +52,8 @@ enum nfcc_chip_variant {
NFCC_PN66T = 0x18, /**< NFCC PN66T */
NFCC_NOT_SUPPORTED = 0xFF /**< NFCC is not supported */
};
+
+struct nqx_dev;
+int nqx_claim_ese(struct nqx_dev *nqx_dev, bool claim);
+int nqx_ese_pwr(struct nqx_dev *nqx_dev, unsigned long int arg);
#endif
diff --git a/drivers/nfc/nxp-nci/firmware.c b/drivers/nfc/nxp-nci/firmware.c
index 5291797..553011f 100644
--- a/drivers/nfc/nxp-nci/firmware.c
+++ b/drivers/nfc/nxp-nci/firmware.c
@@ -24,7 +24,7 @@
#include <linux/completion.h>
#include <linux/firmware.h>
#include <linux/nfc.h>
-#include <linux/unaligned/access_ok.h>
+#include <asm/unaligned.h>
#include "nxp-nci.h"
diff --git a/drivers/nfc/nxp-nci/i2c.c b/drivers/nfc/nxp-nci/i2c.c
index 36099e5..06a157c6 100644
--- a/drivers/nfc/nxp-nci/i2c.c
+++ b/drivers/nfc/nxp-nci/i2c.c
@@ -36,7 +36,7 @@
#include <linux/of_gpio.h>
#include <linux/of_irq.h>
#include <linux/platform_data/nxp-nci.h>
-#include <linux/unaligned/access_ok.h>
+#include <asm/unaligned.h>
#include <net/nfc/nfc.h>
diff --git a/drivers/nfc/pn533/usb.c b/drivers/nfc/pn533/usb.c
index 33ed78b..3a897f5 100644
--- a/drivers/nfc/pn533/usb.c
+++ b/drivers/nfc/pn533/usb.c
@@ -71,7 +71,7 @@ static void pn533_recv_response(struct urb *urb)
struct sk_buff *skb = NULL;
if (!urb->status) {
- skb = alloc_skb(urb->actual_length, GFP_KERNEL);
+ skb = alloc_skb(urb->actual_length, GFP_ATOMIC);
if (!skb) {
nfc_err(&phy->udev->dev, "failed to alloc memory\n");
} else {
@@ -180,7 +180,7 @@ static int pn533_usb_send_frame(struct pn533 *dev,
if (dev->protocol_type == PN533_PROTO_REQ_RESP) {
/* request for response for sent packet directly */
- rc = pn533_submit_urb_for_response(phy, GFP_ATOMIC);
+ rc = pn533_submit_urb_for_response(phy, GFP_KERNEL);
if (rc)
goto error;
} else if (dev->protocol_type == PN533_PROTO_REQ_ACK_RESP) {
diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
index 24222a5d..da95bd8 100644
--- a/drivers/ntb/ntb_transport.c
+++ b/drivers/ntb/ntb_transport.c
@@ -996,6 +996,9 @@ static int ntb_transport_init_queue(struct ntb_transport_ctx *nt,
mw_base = nt->mw_vec[mw_num].phys_addr;
mw_size = nt->mw_vec[mw_num].phys_size;
+ if (max_mw_size && mw_size > max_mw_size)
+ mw_size = max_mw_size;
+
tx_size = (unsigned int)mw_size / num_qps_mw;
qp_offset = tx_size * (qp_num / mw_count);
diff --git a/drivers/nvdimm/bus.c b/drivers/nvdimm/bus.c
index 8311a93..5768a47 100644
--- a/drivers/nvdimm/bus.c
+++ b/drivers/nvdimm/bus.c
@@ -424,6 +424,8 @@ static void nd_async_device_register(void *d, async_cookie_t cookie)
put_device(dev);
}
put_device(dev);
+ if (dev->parent)
+ put_device(dev->parent);
}
static void nd_async_device_unregister(void *d, async_cookie_t cookie)
@@ -443,6 +445,8 @@ void __nd_device_register(struct device *dev)
if (!dev)
return;
dev->bus = &nvdimm_bus_type;
+ if (dev->parent)
+ get_device(dev->parent);
get_device(dev);
async_schedule_domain(nd_async_device_register, dev,
&nd_async_domain);
@@ -505,14 +509,18 @@ int nvdimm_revalidate_disk(struct gendisk *disk)
{
struct device *dev = disk_to_dev(disk)->parent;
struct nd_region *nd_region = to_nd_region(dev->parent);
- const char *pol = nd_region->ro ? "only" : "write";
+ int disk_ro = get_disk_ro(disk);
- if (nd_region->ro == get_disk_ro(disk))
+ /*
+ * Upgrade to read-only if the region is read-only preserve as
+ * read-only if the disk is already read-only.
+ */
+ if (disk_ro || nd_region->ro == disk_ro)
return 0;
- dev_info(dev, "%s read-%s, marking %s read-%s\n",
- dev_name(&nd_region->dev), pol, disk->disk_name, pol);
- set_disk_ro(disk, nd_region->ro);
+ dev_info(dev, "%s read-only, marking %s read-only\n",
+ dev_name(&nd_region->dev), disk->disk_name);
+ set_disk_ro(disk, 1);
return 0;
@@ -744,9 +752,9 @@ u32 nd_cmd_out_size(struct nvdimm *nvdimm, int cmd,
* overshoots the remainder by 4 bytes, assume it was
* including 'status'.
*/
- if (out_field[1] - 8 == remainder)
+ if (out_field[1] - 4 == remainder)
return remainder;
- return out_field[1] - 4;
+ return out_field[1] - 8;
} else if (cmd == ND_CMD_CALL) {
struct nd_cmd_pkg *pkg = (struct nd_cmd_pkg *) in_field;
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index c823e93..979c6ec 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -2042,7 +2042,8 @@ void nvme_kill_queues(struct nvme_ctrl *ctrl)
mutex_lock(&ctrl->namespaces_mutex);
/* Forcibly start all queues to avoid having stuck requests */
- blk_mq_start_hw_queues(ctrl->admin_q);
+ if (ctrl->admin_q)
+ blk_mq_start_hw_queues(ctrl->admin_q);
list_for_each_entry(ns, &ctrl->namespaces, list) {
/*
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index eef1a68..b634b89 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -583,8 +583,10 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
opts->discovery_nqn =
!(strcmp(opts->subsysnqn,
NVME_DISC_SUBSYS_NAME));
- if (opts->discovery_nqn)
+ if (opts->discovery_nqn) {
+ opts->kato = 0;
opts->nr_io_queues = 0;
+ }
break;
case NVMF_OPT_TRADDR:
p = match_strdup(args);
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index 8cc856e..1ac4cec 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -1034,17 +1034,15 @@ static int nvme_cmb_qdepth(struct nvme_dev *dev, int nr_io_queues,
static int nvme_alloc_sq_cmds(struct nvme_dev *dev, struct nvme_queue *nvmeq,
int qid, int depth)
{
- if (qid && dev->cmb && use_cmb_sqes && NVME_CMB_SQS(dev->cmbsz)) {
- unsigned offset = (qid - 1) * roundup(SQ_SIZE(depth),
- dev->ctrl.page_size);
- nvmeq->sq_dma_addr = dev->cmb_bus_addr + offset;
- nvmeq->sq_cmds_io = dev->cmb + offset;
- } else {
- nvmeq->sq_cmds = dma_alloc_coherent(dev->dev, SQ_SIZE(depth),
- &nvmeq->sq_dma_addr, GFP_KERNEL);
- if (!nvmeq->sq_cmds)
- return -ENOMEM;
- }
+
+ /* CMB SQEs will be mapped before creation */
+ if (qid && dev->cmb && use_cmb_sqes && NVME_CMB_SQS(dev->cmbsz))
+ return 0;
+
+ nvmeq->sq_cmds = dma_alloc_coherent(dev->dev, SQ_SIZE(depth),
+ &nvmeq->sq_dma_addr, GFP_KERNEL);
+ if (!nvmeq->sq_cmds)
+ return -ENOMEM;
return 0;
}
@@ -1117,26 +1115,36 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid)
struct nvme_dev *dev = nvmeq->dev;
int result;
+ if (qid && dev->cmb && use_cmb_sqes && NVME_CMB_SQS(dev->cmbsz)) {
+ unsigned offset = (qid - 1) * roundup(SQ_SIZE(nvmeq->q_depth),
+ dev->ctrl.page_size);
+ nvmeq->sq_dma_addr = dev->cmb_bus_addr + offset;
+ nvmeq->sq_cmds_io = dev->cmb + offset;
+ }
+
nvmeq->cq_vector = qid - 1;
result = adapter_alloc_cq(dev, qid, nvmeq);
if (result < 0)
- return result;
+ goto release_vector;
result = adapter_alloc_sq(dev, qid, nvmeq);
if (result < 0)
goto release_cq;
+ nvme_init_queue(nvmeq, qid);
result = queue_request_irq(nvmeq);
if (result < 0)
goto release_sq;
- nvme_init_queue(nvmeq, qid);
return result;
release_sq:
+ dev->online_queues--;
adapter_delete_sq(dev, qid);
release_cq:
adapter_delete_cq(dev, qid);
+ release_vector:
+ nvmeq->cq_vector = -1;
return result;
}
@@ -1245,6 +1253,7 @@ static int nvme_configure_admin_queue(struct nvme_dev *dev)
return result;
nvmeq->cq_vector = 0;
+ nvme_init_queue(nvmeq, 0);
result = queue_request_irq(nvmeq);
if (result) {
nvmeq->cq_vector = -1;
@@ -1384,11 +1393,9 @@ static inline void nvme_release_cmb(struct nvme_dev *dev)
if (dev->cmb) {
iounmap(dev->cmb);
dev->cmb = NULL;
- if (dev->cmbsz) {
- sysfs_remove_file_from_group(&dev->ctrl.device->kobj,
- &dev_attr_cmb.attr, NULL);
- dev->cmbsz = 0;
- }
+ sysfs_remove_file_from_group(&dev->ctrl.device->kobj,
+ &dev_attr_cmb.attr, NULL);
+ dev->cmbsz = 0;
}
}
@@ -1623,16 +1630,14 @@ static int nvme_pci_enable(struct nvme_dev *dev)
/*
* CMBs can currently only exist on >=1.2 PCIe devices. We only
- * populate sysfs if a CMB is implemented. Note that we add the
- * CMB attribute to the nvme_ctrl kobj which removes the need to remove
- * it on exit. Since nvme_dev_attrs_group has no name we can pass
- * NULL as final argument to sysfs_add_file_to_group.
+ * populate sysfs if a CMB is implemented. Since nvme_dev_attrs_group
+ * has no name we can pass NULL as final argument to
+ * sysfs_add_file_to_group.
*/
if (readl(dev->bar + NVME_REG_VS) >= NVME_VS(1, 2, 0)) {
dev->cmb = nvme_map_cmb(dev);
-
- if (dev->cmbsz) {
+ if (dev->cmb) {
if (sysfs_add_file_to_group(&dev->ctrl.device->kobj,
&dev_attr_cmb.attr, NULL))
dev_warn(dev->dev,
@@ -1773,7 +1778,6 @@ static void nvme_reset_work(struct work_struct *work)
if (result)
goto out;
- nvme_init_queue(dev->queues[0], 0);
result = nvme_alloc_admin_tags(dev);
if (result)
goto out;
diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index f791d46..cdb7752 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -166,11 +166,21 @@ static void nvmet_execute_get_log_page(struct nvmet_req *req)
nvmet_req_complete(req, status);
}
+static void copy_and_pad(char *dst, int dst_len, const char *src, int src_len)
+{
+ int len = min(src_len, dst_len);
+
+ memcpy(dst, src, len);
+ if (dst_len > len)
+ memset(dst + len, ' ', dst_len - len);
+}
+
static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
{
struct nvmet_ctrl *ctrl = req->sq->ctrl;
struct nvme_id_ctrl *id;
u16 status = 0;
+ const char model[] = "Linux";
id = kzalloc(sizeof(*id), GFP_KERNEL);
if (!id) {
@@ -183,13 +193,10 @@ static void nvmet_execute_identify_ctrl(struct nvmet_req *req)
id->ssvid = 0;
memset(id->sn, ' ', sizeof(id->sn));
- snprintf(id->sn, sizeof(id->sn), "%llx", ctrl->serial);
-
- memset(id->mn, ' ', sizeof(id->mn));
- strncpy((char *)id->mn, "Linux", sizeof(id->mn));
-
- memset(id->fr, ' ', sizeof(id->fr));
- strncpy((char *)id->fr, UTS_RELEASE, sizeof(id->fr));
+ bin2hex(id->sn, &ctrl->subsys->serial,
+ min(sizeof(ctrl->subsys->serial), sizeof(id->sn) / 2));
+ copy_and_pad(id->mn, sizeof(id->mn), model, sizeof(model) - 1);
+ copy_and_pad(id->fr, sizeof(id->fr), UTS_RELEASE, strlen(UTS_RELEASE));
id->rab = 6;
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index c89d68a..f12753e 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -491,9 +491,12 @@ bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
goto fail;
}
- /* either variant of SGLs is fine, as we don't support metadata */
- if (unlikely((flags & NVME_CMD_SGL_ALL) != NVME_CMD_SGL_METABUF &&
- (flags & NVME_CMD_SGL_ALL) != NVME_CMD_SGL_METASEG)) {
+ /*
+ * For fabrics, PSDT field shall describe metadata pointer (MPTR) that
+ * contains an address of a single contiguous physical buffer that is
+ * byte aligned.
+ */
+ if (unlikely((flags & NVME_CMD_SGL_ALL) != NVME_CMD_SGL_METABUF)) {
status = NVME_SC_INVALID_FIELD | NVME_SC_DNR;
goto fail;
}
@@ -575,6 +578,14 @@ static void nvmet_start_ctrl(struct nvmet_ctrl *ctrl)
}
ctrl->csts = NVME_CSTS_RDY;
+
+ /*
+ * Controllers that are not yet enabled should not really enforce the
+ * keep alive timeout, but we still want to track a timeout and cleanup
+ * in case a host died before it enabled the controller. Hence, simply
+ * reset the keep alive timer when the controller is enabled.
+ */
+ mod_delayed_work(system_wq, &ctrl->ka_work, ctrl->kato * HZ);
}
static void nvmet_clear_ctrl(struct nvmet_ctrl *ctrl)
@@ -740,9 +751,6 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
memcpy(ctrl->subsysnqn, subsysnqn, NVMF_NQN_SIZE);
memcpy(ctrl->hostnqn, hostnqn, NVMF_NQN_SIZE);
- /* generate a random serial number as our controllers are ephemeral: */
- get_random_bytes(&ctrl->serial, sizeof(ctrl->serial));
-
kref_init(&ctrl->ref);
ctrl->subsys = subsys;
@@ -901,6 +909,8 @@ struct nvmet_subsys *nvmet_subsys_alloc(const char *subsysnqn,
return NULL;
subsys->ver = NVME_VS(1, 2, 1); /* NVMe 1.2.1 */
+ /* generate a random serial number as our controllers are ephemeral: */
+ get_random_bytes(&subsys->serial, sizeof(subsys->serial));
switch (type) {
case NVME_NQN_NVME:
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 26b87dc8..0bc530cd 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -110,7 +110,6 @@ struct nvmet_ctrl {
struct mutex lock;
u64 cap;
- u64 serial;
u32 cc;
u32 csts;
@@ -151,6 +150,7 @@ struct nvmet_subsys {
u16 max_qid;
u64 ver;
+ u64 serial;
char *subsysnqn;
struct config_group group;
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 53bd325..486393f 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -65,6 +65,7 @@ struct nvmet_rdma_rsp {
struct nvmet_req req;
+ bool allocated;
u8 n_rdma;
u32 flags;
u32 invalidate_rkey;
@@ -167,11 +168,19 @@ nvmet_rdma_get_rsp(struct nvmet_rdma_queue *queue)
unsigned long flags;
spin_lock_irqsave(&queue->rsps_lock, flags);
- rsp = list_first_entry(&queue->free_rsps,
+ rsp = list_first_entry_or_null(&queue->free_rsps,
struct nvmet_rdma_rsp, free_list);
- list_del(&rsp->free_list);
+ if (likely(rsp))
+ list_del(&rsp->free_list);
spin_unlock_irqrestore(&queue->rsps_lock, flags);
+ if (unlikely(!rsp)) {
+ rsp = kmalloc(sizeof(*rsp), GFP_KERNEL);
+ if (unlikely(!rsp))
+ return NULL;
+ rsp->allocated = true;
+ }
+
return rsp;
}
@@ -180,6 +189,11 @@ nvmet_rdma_put_rsp(struct nvmet_rdma_rsp *rsp)
{
unsigned long flags;
+ if (rsp->allocated) {
+ kfree(rsp);
+ return;
+ }
+
spin_lock_irqsave(&rsp->queue->rsps_lock, flags);
list_add_tail(&rsp->free_list, &rsp->queue->free_rsps);
spin_unlock_irqrestore(&rsp->queue->rsps_lock, flags);
@@ -510,6 +524,7 @@ static void nvmet_rdma_send_done(struct ib_cq *cq, struct ib_wc *wc)
{
struct nvmet_rdma_rsp *rsp =
container_of(wc->wr_cqe, struct nvmet_rdma_rsp, send_cqe);
+ struct nvmet_rdma_queue *queue = cq->cq_context;
nvmet_rdma_release_rsp(rsp);
@@ -517,7 +532,7 @@ static void nvmet_rdma_send_done(struct ib_cq *cq, struct ib_wc *wc)
wc->status != IB_WC_WR_FLUSH_ERR)) {
pr_err("SEND for CQE 0x%p failed with status %s (%d).\n",
wc->wr_cqe, ib_wc_status_msg(wc->status), wc->status);
- nvmet_rdma_error_comp(rsp->queue);
+ nvmet_rdma_error_comp(queue);
}
}
@@ -755,6 +770,15 @@ static void nvmet_rdma_recv_done(struct ib_cq *cq, struct ib_wc *wc)
cmd->queue = queue;
rsp = nvmet_rdma_get_rsp(queue);
+ if (unlikely(!rsp)) {
+ /*
+ * we get here only under memory pressure,
+ * silently drop and have the host retry
+ * as we can't even fail it.
+ */
+ nvmet_rdma_post_recv(queue->dev, cmd);
+ return;
+ }
rsp->queue = queue;
rsp->cmd = cmd;
rsp->flags = 0;
diff --git a/drivers/nvmem/core.c b/drivers/nvmem/core.c
index 1b4d93e..824e282 100644
--- a/drivers/nvmem/core.c
+++ b/drivers/nvmem/core.c
@@ -1031,6 +1031,8 @@ static inline void *nvmem_cell_prepare_write_buffer(struct nvmem_cell *cell,
/* setup the first byte with lsb bits from nvmem */
rc = nvmem_reg_read(nvmem, cell->offset, &v, 1);
+ if (rc)
+ goto err;
*b++ |= GENMASK(bit_offset - 1, 0) & v;
/* setup rest of the byte if any */
@@ -1049,11 +1051,16 @@ static inline void *nvmem_cell_prepare_write_buffer(struct nvmem_cell *cell,
/* setup the last byte with msb bits from nvmem */
rc = nvmem_reg_read(nvmem,
cell->offset + cell->bytes - 1, &v, 1);
+ if (rc)
+ goto err;
*p |= GENMASK(7, (nbits + bit_offset) % BITS_PER_BYTE) & v;
}
return buf;
+err:
+ kfree(buf);
+ return ERR_PTR(rc);
}
/**
diff --git a/drivers/of/base.c b/drivers/of/base.c
index 23a6d36..88cec00 100644
--- a/drivers/of/base.c
+++ b/drivers/of/base.c
@@ -832,6 +832,31 @@ struct device_node *of_get_next_available_child(const struct device_node *node,
EXPORT_SYMBOL(of_get_next_available_child);
/**
+ * of_get_compatible_child - Find compatible child node
+ * @parent: parent node
+ * @compatible: compatible string
+ *
+ * Lookup child node whose compatible property contains the given compatible
+ * string.
+ *
+ * Returns a node pointer with refcount incremented, use of_node_put() on it
+ * when done; or NULL if not found.
+ */
+struct device_node *of_get_compatible_child(const struct device_node *parent,
+ const char *compatible)
+{
+ struct device_node *child;
+
+ for_each_child_of_node(parent, child) {
+ if (of_device_is_compatible(child, compatible))
+ break;
+ }
+
+ return child;
+}
+EXPORT_SYMBOL(of_get_compatible_child);
+
+/**
* of_get_child_by_name - Find the child node by name for a given parent
* @node: parent node
* @name: child name to look for.
@@ -2218,7 +2243,7 @@ void of_alias_scan(void * (*dt_alloc)(u64 size, u64 align))
continue;
/* Allocate an alias_prop with enough space for the stem */
- ap = dt_alloc(sizeof(*ap) + len + 1, 4);
+ ap = dt_alloc(sizeof(*ap) + len + 1, __alignof__(*ap));
if (!ap)
continue;
memset(ap, 0, sizeof(*ap) + len + 1);
diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
index 755b386..744f625 100644
--- a/drivers/of/fdt.c
+++ b/drivers/of/fdt.c
@@ -937,7 +937,7 @@ int __init early_init_dt_scan_chosen_stdout(void)
int offset;
const char *p, *q, *options = NULL;
int l;
- const struct earlycon_id *match;
+ const struct earlycon_id **p_match;
const void *fdt = initial_boot_params;
offset = fdt_path_offset(fdt, "/chosen");
@@ -964,7 +964,10 @@ int __init early_init_dt_scan_chosen_stdout(void)
return 0;
}
- for (match = __earlycon_table; match < __earlycon_table_end; match++) {
+ for (p_match = __earlycon_table; p_match < __earlycon_table_end;
+ p_match++) {
+ const struct earlycon_id *match = *p_match;
+
if (!match->compatible[0])
continue;
diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
index 2db1f7a..0b29ee9e 100644
--- a/drivers/of/of_numa.c
+++ b/drivers/of/of_numa.c
@@ -126,9 +126,14 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map)
distance = of_read_number(matrix, 1);
matrix++;
+ if ((nodea == nodeb && distance != LOCAL_DISTANCE) ||
+ (nodea != nodeb && distance <= LOCAL_DISTANCE)) {
+ pr_err("Invalid distance[node%d -> node%d] = %d\n",
+ nodea, nodeb, distance);
+ return -EINVAL;
+ }
+
numa_set_distance(nodea, nodeb, distance);
- pr_debug("distance[node%d -> node%d] = %d\n",
- nodea, nodeb, distance);
/* Set default distance of node B->A same as A->B */
if (nodeb > nodea)
diff --git a/drivers/of/unittest.c b/drivers/of/unittest.c
index 53c83d6..0a1ebbb 100644
--- a/drivers/of/unittest.c
+++ b/drivers/of/unittest.c
@@ -155,20 +155,20 @@ static void __init of_unittest_dynamic(void)
/* Add a new property - should pass*/
prop->name = "new-property";
prop->value = "new-property-data";
- prop->length = strlen(prop->value);
+ prop->length = strlen(prop->value) + 1;
unittest(of_add_property(np, prop) == 0, "Adding a new property failed\n");
/* Try to add an existing property - should fail */
prop++;
prop->name = "new-property";
prop->value = "new-property-data-should-fail";
- prop->length = strlen(prop->value);
+ prop->length = strlen(prop->value) + 1;
unittest(of_add_property(np, prop) != 0,
"Adding an existing property should have failed\n");
/* Try to modify an existing property - should pass */
prop->value = "modify-property-data-should-pass";
- prop->length = strlen(prop->value);
+ prop->length = strlen(prop->value) + 1;
unittest(of_update_property(np, prop) == 0,
"Updating an existing property should have passed\n");
@@ -176,7 +176,7 @@ static void __init of_unittest_dynamic(void)
prop++;
prop->name = "modify-property";
prop->value = "modify-missing-property-data-should-pass";
- prop->length = strlen(prop->value);
+ prop->length = strlen(prop->value) + 1;
unittest(of_update_property(np, prop) == 0,
"Updating a missing property should have passed\n");
@@ -548,6 +548,9 @@ static void __init of_unittest_parse_interrupts(void)
struct of_phandle_args args;
int i, rc;
+ if (of_irq_workarounds & OF_IMAP_OLDWORLD_MAC)
+ return;
+
np = of_find_node_by_path("/testcase-data/interrupts/interrupts0");
if (!np) {
pr_err("missing testcase data\n");
@@ -622,6 +625,9 @@ static void __init of_unittest_parse_interrupts_extended(void)
struct of_phandle_args args;
int i, rc;
+ if (of_irq_workarounds & OF_IMAP_OLDWORLD_MAC)
+ return;
+
np = of_find_node_by_path("/testcase-data/interrupts/interrupts-extended0");
if (!np) {
pr_err("missing testcase data\n");
@@ -778,15 +784,19 @@ static void __init of_unittest_platform_populate(void)
pdev = of_find_device_by_node(np);
unittest(pdev, "device 1 creation failed\n");
- irq = platform_get_irq(pdev, 0);
- unittest(irq == -EPROBE_DEFER, "device deferred probe failed - %d\n", irq);
+ if (!(of_irq_workarounds & OF_IMAP_OLDWORLD_MAC)) {
+ irq = platform_get_irq(pdev, 0);
+ unittest(irq == -EPROBE_DEFER,
+ "device deferred probe failed - %d\n", irq);
- /* Test that a parsing failure does not return -EPROBE_DEFER */
- np = of_find_node_by_path("/testcase-data/testcase-device2");
- pdev = of_find_device_by_node(np);
- unittest(pdev, "device 2 creation failed\n");
- irq = platform_get_irq(pdev, 0);
- unittest(irq < 0 && irq != -EPROBE_DEFER, "device parsing error failed - %d\n", irq);
+ /* Test that a parsing failure does not return -EPROBE_DEFER */
+ np = of_find_node_by_path("/testcase-data/testcase-device2");
+ pdev = of_find_device_by_node(np);
+ unittest(pdev, "device 2 creation failed\n");
+ irq = platform_get_irq(pdev, 0);
+ unittest(irq < 0 && irq != -EPROBE_DEFER,
+ "device parsing error failed - %d\n", irq);
+ }
np = of_find_node_by_path("/testcase-data/platform-tests");
unittest(np, "No testcase data in device tree\n");
diff --git a/drivers/parisc/lba_pci.c b/drivers/parisc/lba_pci.c
index 1cced1d..7e93858 100644
--- a/drivers/parisc/lba_pci.c
+++ b/drivers/parisc/lba_pci.c
@@ -1367,9 +1367,27 @@ lba_hw_init(struct lba_device *d)
WRITE_REG32(stat, d->hba.base_addr + LBA_ERROR_CONFIG);
}
- /* Set HF mode as the default (vs. -1 mode). */
+
+ /*
+ * Hard Fail vs. Soft Fail on PCI "Master Abort".
+ *
+ * "Master Abort" means the MMIO transaction timed out - usually due to
+ * the device not responding to an MMIO read. We would like HF to be
+ * enabled to find driver problems, though it means the system will
+ * crash with a HPMC.
+ *
+ * In SoftFail mode "~0L" is returned as a result of a timeout on the
+ * pci bus. This is like how PCI busses on x86 and most other
+ * architectures behave. In order to increase compatibility with
+ * existing (x86) PCI hardware and existing Linux drivers we enable
+ * Soft Faul mode on PA-RISC now too.
+ */
stat = READ_REG32(d->hba.base_addr + LBA_STAT_CTL);
+#if defined(ENABLE_HARDFAIL)
WRITE_REG32(stat | HF_ENABLE, d->hba.base_addr + LBA_STAT_CTL);
+#else
+ WRITE_REG32(stat & ~HF_ENABLE, d->hba.base_addr + LBA_STAT_CTL);
+#endif
/*
** Writing a zero to STAT_CTL.rf (bit 0) will clear reset signal
diff --git a/drivers/parport/parport_sunbpp.c b/drivers/parport/parport_sunbpp.c
index 01cf1c1..8de3295 100644
--- a/drivers/parport/parport_sunbpp.c
+++ b/drivers/parport/parport_sunbpp.c
@@ -286,12 +286,16 @@ static int bpp_probe(struct platform_device *op)
ops = kmemdup(&parport_sunbpp_ops, sizeof(struct parport_operations),
GFP_KERNEL);
- if (!ops)
+ if (!ops) {
+ err = -ENOMEM;
goto out_unmap;
+ }
dprintk(("register_port\n"));
- if (!(p = parport_register_port((unsigned long)base, irq, dma, ops)))
+ if (!(p = parport_register_port((unsigned long)base, irq, dma, ops))) {
+ err = -ENOMEM;
goto out_free_ops;
+ }
p->size = size;
p->dev = &op->dev;
diff --git a/drivers/pci/host/pci-aardvark.c b/drivers/pci/host/pci-aardvark.c
index 4fce494..1dbd09c 100644
--- a/drivers/pci/host/pci-aardvark.c
+++ b/drivers/pci/host/pci-aardvark.c
@@ -32,6 +32,7 @@
#define PCIE_CORE_DEV_CTRL_STATS_MAX_PAYLOAD_SZ_SHIFT 5
#define PCIE_CORE_DEV_CTRL_STATS_SNOOP_DISABLE (0 << 11)
#define PCIE_CORE_DEV_CTRL_STATS_MAX_RD_REQ_SIZE_SHIFT 12
+#define PCIE_CORE_DEV_CTRL_STATS_MAX_RD_REQ_SZ 0x2
#define PCIE_CORE_LINK_CTRL_STAT_REG 0xd0
#define PCIE_CORE_LINK_L0S_ENTRY BIT(0)
#define PCIE_CORE_LINK_TRAINING BIT(5)
@@ -175,8 +176,6 @@
#define PCIE_CONFIG_WR_TYPE0 0xa
#define PCIE_CONFIG_WR_TYPE1 0xb
-/* PCI_BDF shifts 8bit, so we need extra 4bit shift */
-#define PCIE_BDF(dev) (dev << 4)
#define PCIE_CONF_BUS(bus) (((bus) & 0xff) << 20)
#define PCIE_CONF_DEV(dev) (((dev) & 0x1f) << 15)
#define PCIE_CONF_FUNC(fun) (((fun) & 0x7) << 12)
@@ -298,7 +297,8 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie)
reg = PCIE_CORE_DEV_CTRL_STATS_RELAX_ORDER_DISABLE |
(7 << PCIE_CORE_DEV_CTRL_STATS_MAX_PAYLOAD_SZ_SHIFT) |
PCIE_CORE_DEV_CTRL_STATS_SNOOP_DISABLE |
- PCIE_CORE_DEV_CTRL_STATS_MAX_RD_REQ_SIZE_SHIFT;
+ (PCIE_CORE_DEV_CTRL_STATS_MAX_RD_REQ_SZ <<
+ PCIE_CORE_DEV_CTRL_STATS_MAX_RD_REQ_SIZE_SHIFT);
advk_writel(pcie, reg, PCIE_CORE_DEV_CTRL_STATS_REG);
/* Program PCIe Control 2 to disable strict ordering */
@@ -439,7 +439,7 @@ static int advk_pcie_rd_conf(struct pci_bus *bus, u32 devfn,
u32 reg;
int ret;
- if (PCI_SLOT(devfn) != 0) {
+ if ((bus->number == pcie->root_bus_nr) && PCI_SLOT(devfn) != 0) {
*val = 0xffffffff;
return PCIBIOS_DEVICE_NOT_FOUND;
}
@@ -458,7 +458,7 @@ static int advk_pcie_rd_conf(struct pci_bus *bus, u32 devfn,
advk_writel(pcie, reg, PIO_CTRL);
/* Program the address registers */
- reg = PCIE_BDF(devfn) | PCIE_CONF_REG(where);
+ reg = PCIE_CONF_ADDR(bus->number, devfn, where);
advk_writel(pcie, reg, PIO_ADDR_LS);
advk_writel(pcie, 0, PIO_ADDR_MS);
@@ -493,7 +493,7 @@ static int advk_pcie_wr_conf(struct pci_bus *bus, u32 devfn,
int offset;
int ret;
- if (PCI_SLOT(devfn) != 0)
+ if ((bus->number == pcie->root_bus_nr) && PCI_SLOT(devfn) != 0)
return PCIBIOS_DEVICE_NOT_FOUND;
if (where % size)
@@ -976,6 +976,7 @@ static int advk_pcie_probe(struct platform_device *pdev)
return -ENOMEM;
}
+ pci_bus_size_bridges(bus);
pci_bus_assign_resources(bus);
list_for_each_entry(child, &bus->children, node)
diff --git a/drivers/pci/host/pci-host-common.c b/drivers/pci/host/pci-host-common.c
index e3c48b5..5c90d7b 100644
--- a/drivers/pci/host/pci-host-common.c
+++ b/drivers/pci/host/pci-host-common.c
@@ -45,7 +45,7 @@ static int gen_pci_parse_request_of_pci_ranges(struct device *dev,
switch (resource_type(res)) {
case IORESOURCE_IO:
- err = pci_remap_iospace(res, iobase);
+ err = devm_pci_remap_iospace(dev, res, iobase);
if (err) {
dev_warn(dev, "error %d: failed to map resource %pR\n",
err, res);
diff --git a/drivers/pci/host/pci-hyperv.c b/drivers/pci/host/pci-hyperv.c
index d392a55..b4d8ccf 100644
--- a/drivers/pci/host/pci-hyperv.c
+++ b/drivers/pci/host/pci-hyperv.c
@@ -52,6 +52,8 @@
#include <linux/pci.h>
#include <linux/semaphore.h>
#include <linux/irqdomain.h>
+#include <linux/irq.h>
+
#include <asm/irqdomain.h>
#include <asm/apic.h>
#include <linux/msi.h>
diff --git a/drivers/pci/host/pci-msm.c b/drivers/pci/host/pci-msm.c
index cc68136..9cf3892 100644
--- a/drivers/pci/host/pci-msm.c
+++ b/drivers/pci/host/pci-msm.c
@@ -4961,28 +4961,44 @@ static int arch_setup_msi_irq_default(struct pci_dev *pdev,
struct msi_desc *desc, int nvec)
{
int irq;
+ int index;
+ int count;
+ int lastirq = -1;
struct msi_msg msg;
struct msm_pcie_dev_t *dev = PCIE_BUS_PRIV_DATA(pdev->bus);
- PCIE_DBG(dev, "RC%d\n", dev->rc_idx);
+ PCIE_DBG(dev, "RC%d nvec %d\n", dev->rc_idx, nvec);
- irq = msm_pcie_create_irq(dev);
+ /* create up to nvec consecutive irq mappings */
+ for (count = 0; count < nvec; count++) {
+ irq = msm_pcie_create_irq(dev);
- PCIE_DBG(dev, "IRQ %d is allocated.\n", irq);
+ if (irq < 0)
+ return irq;
+ else if ((lastirq != -1) && (irq != (lastirq + 1))) {
+ msm_pcie_destroy_irq(desc, irq);
+ break;
+ }
- if (irq < 0)
- return irq;
+ PCIE_DBG(dev, "irq %d allocated\n", irq);
- PCIE_DBG(dev, "irq %d allocated\n", irq);
+ lastirq = irq;
+ }
- irq_set_chip_data(irq, pdev);
- irq_set_msi_desc(irq, desc);
+ /* update irq mapping with descriptors, work backwards */
+ for (index = count; index > 0; index--, irq--) {
+ irq_set_msi_desc(irq, desc);
+ irq_set_chip_data(irq, pdev);
- /* write msi vector and data */
- msg.address_hi = 0;
- msg.address_lo = MSM_PCIE_MSI_PHY;
- msg.data = irq - irq_find_mapping(dev->irq_domain, 0);
- write_msi_msg(irq, &msg);
+ /* write msi vector and data */
+ msg.address_hi = 0;
+ msg.address_lo = MSM_PCIE_MSI_PHY;
+ msg.data = irq - irq_find_mapping(dev->irq_domain, index - 1);
+ write_msi_msg(irq, &msg);
+ }
+
+ if (count != nvec)
+ return count;
return 0;
}
@@ -5160,7 +5176,6 @@ static const struct irq_domain_ops msm_pcie_msi_ops = {
static int32_t msm_pcie_irq_init(struct msm_pcie_dev_t *dev)
{
int rc;
- int msi_start = 0;
struct device *pdev = &dev->pdev->dev;
PCIE_DBG(dev, "RC%d\n", dev->rc_idx);
@@ -5290,8 +5305,6 @@ static int32_t msm_pcie_irq_init(struct msm_pcie_dev_t *dev)
return PTR_ERR(dev->irq_domain);
}
-
- msi_start = irq_create_mapping(dev->irq_domain, 0);
}
return 0;
@@ -5519,7 +5532,7 @@ static void msm_pcie_config_link_pm_rc(struct msm_pcie_dev_t *dev,
{
bool child_l0s_enable = 0, child_l1_enable = 0, child_l1ss_enable = 0;
- if (!pdev->subordinate || !(&pdev->subordinate->devices)) {
+ if (!pdev->subordinate || list_empty(&pdev->subordinate->devices)) {
PCIE_DBG(dev,
"PCIe: RC%d: no device connected to root complex\n",
dev->rc_idx);
@@ -6253,7 +6266,7 @@ static int msm_pcie_pm_suspend(struct pci_dev *dev,
if (dev && !(options & MSM_PCIE_CONFIG_NO_CFG_RESTORE)
&& msm_pcie_confirm_linkup(pcie_dev, true, true,
- pcie_dev->conf)) {
+ pcie_dev->conf) && !pcie_dev->saved_state) {
ret = pci_save_state(dev);
pcie_dev->saved_state = pci_store_saved_state(dev);
}
@@ -6374,8 +6387,12 @@ static int msm_pcie_pm_resume(struct pci_dev *dev,
"RC%d: entry of PCI framework restore state\n",
pcie_dev->rc_idx);
- pci_load_and_free_saved_state(dev,
- &pcie_dev->saved_state);
+ if (options & MSM_PCIE_CONFIG_NO_CFG_FREE)
+ pci_load_saved_state(dev,
+ pcie_dev->saved_state);
+ else
+ pci_load_and_free_saved_state(dev,
+ &pcie_dev->saved_state);
pci_restore_state(dev);
PCIE_DBG(pcie_dev,
@@ -6818,3 +6835,23 @@ int msm_pcie_shadow_control(struct pci_dev *dev, bool enable)
return ret;
}
EXPORT_SYMBOL(msm_pcie_shadow_control);
+
+int msm_pcie_set_reset(u32 rc_idx, bool enable)
+{
+ struct msm_pcie_dev_t *dev = &msm_pcie_dev[rc_idx];
+
+ if (enable) {
+ PCIE_INFO(dev, "PCIe: Assert the reset of endpoint of RC%d.\n",
+ dev->rc_idx);
+ gpio_set_value(dev->gpio[MSM_PCIE_GPIO_PERST].num,
+ dev->gpio[MSM_PCIE_GPIO_PERST].on);
+ } else {
+ PCIE_INFO(dev, "PCIe: Release the reset of endpoint of RC%d.\n",
+ dev->rc_idx);
+ gpio_set_value(dev->gpio[MSM_PCIE_GPIO_PERST].num,
+ 1 - dev->gpio[MSM_PCIE_GPIO_PERST].on);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL(msm_pcie_set_reset);
diff --git a/drivers/pci/host/pci-mvebu.c b/drivers/pci/host/pci-mvebu.c
index 90e0b6f..23d7f73 100644
--- a/drivers/pci/host/pci-mvebu.c
+++ b/drivers/pci/host/pci-mvebu.c
@@ -1236,7 +1236,7 @@ static int mvebu_pcie_probe(struct platform_device *pdev)
pcie->realio.start = PCIBIOS_MIN_IO;
pcie->realio.end = min_t(resource_size_t,
IO_SPACE_LIMIT,
- resource_size(&pcie->io));
+ resource_size(&pcie->io) - 1);
} else
pcie->realio = pcie->io;
diff --git a/drivers/pci/host/pci-versatile.c b/drivers/pci/host/pci-versatile.c
index b7dc070..4096cce 100644
--- a/drivers/pci/host/pci-versatile.c
+++ b/drivers/pci/host/pci-versatile.c
@@ -89,7 +89,7 @@ static int versatile_pci_parse_request_of_pci_ranges(struct device *dev,
switch (resource_type(res)) {
case IORESOURCE_IO:
- err = pci_remap_iospace(res, iobase);
+ err = devm_pci_remap_iospace(dev, res, iobase);
if (err) {
dev_warn(dev, "error %d: failed to map resource %pR\n",
err, res);
diff --git a/drivers/pci/host/pcie-rcar.c b/drivers/pci/host/pcie-rcar.c
index 62700d1..d6196f7 100644
--- a/drivers/pci/host/pcie-rcar.c
+++ b/drivers/pci/host/pcie-rcar.c
@@ -1102,7 +1102,7 @@ static int rcar_pcie_parse_request_of_pci_ranges(struct rcar_pcie *pci)
struct resource *res = win->res;
if (resource_type(res) == IORESOURCE_IO) {
- err = pci_remap_iospace(res, iobase);
+ err = devm_pci_remap_iospace(dev, res, iobase);
if (err) {
dev_warn(dev, "error %d: failed to map resource %pR\n",
err, res);
diff --git a/drivers/pci/host/pcie-xilinx-nwl.c b/drivers/pci/host/pcie-xilinx-nwl.c
index 43eaa4a..94fdd29 100644
--- a/drivers/pci/host/pcie-xilinx-nwl.c
+++ b/drivers/pci/host/pcie-xilinx-nwl.c
@@ -532,7 +532,7 @@ static int nwl_pcie_init_irq_domain(struct nwl_pcie *pcie)
INTX_NUM,
&legacy_domain_ops,
pcie);
-
+ of_node_put(legacy_intc_node);
if (!pcie->legacy_irq_domain) {
dev_err(dev, "failed to create IRQ domain\n");
return -ENOMEM;
diff --git a/drivers/pci/host/pcie-xilinx.c b/drivers/pci/host/pcie-xilinx.c
index c8616fa..61332f4 100644
--- a/drivers/pci/host/pcie-xilinx.c
+++ b/drivers/pci/host/pcie-xilinx.c
@@ -527,6 +527,7 @@ static int xilinx_pcie_init_irq_domain(struct xilinx_pcie_port *port)
port->leg_domain = irq_domain_add_linear(pcie_intc_node, 4,
&intx_domain_ops,
port);
+ of_node_put(pcie_intc_node);
if (!port->leg_domain) {
dev_err(dev, "Failed to get a INTx IRQ domain\n");
return -ENODEV;
diff --git a/drivers/pci/hotplug/pci_hotplug_core.c b/drivers/pci/hotplug/pci_hotplug_core.c
index fea0b8b..0a3b3f7 100644
--- a/drivers/pci/hotplug/pci_hotplug_core.c
+++ b/drivers/pci/hotplug/pci_hotplug_core.c
@@ -455,8 +455,17 @@ int __pci_hp_register(struct hotplug_slot *slot, struct pci_bus *bus,
list_add(&slot->slot_list, &pci_hotplug_slot_list);
result = fs_add_slot(pci_slot);
+ if (result)
+ goto err_list_del;
+
kobject_uevent(&pci_slot->kobj, KOBJ_ADD);
dbg("Added slot %s to the list\n", name);
+ goto out;
+
+err_list_del:
+ list_del(&slot->slot_list);
+ pci_slot->hotplug = NULL;
+ pci_destroy_slot(pci_slot);
out:
mutex_unlock(&pci_hp_mutex);
return result;
diff --git a/drivers/pci/hotplug/pciehp.h b/drivers/pci/hotplug/pciehp.h
index 37d70b5..56c0b60 100644
--- a/drivers/pci/hotplug/pciehp.h
+++ b/drivers/pci/hotplug/pciehp.h
@@ -132,9 +132,10 @@ int pciehp_unconfigure_device(struct slot *p_slot);
void pciehp_queue_pushbutton_work(struct work_struct *work);
struct controller *pcie_init(struct pcie_device *dev);
int pcie_init_notification(struct controller *ctrl);
+void pcie_shutdown_notification(struct controller *ctrl);
int pciehp_enable_slot(struct slot *p_slot);
int pciehp_disable_slot(struct slot *p_slot);
-void pcie_enable_notification(struct controller *ctrl);
+void pcie_reenable_notification(struct controller *ctrl);
int pciehp_power_on_slot(struct slot *slot);
void pciehp_power_off_slot(struct slot *slot);
void pciehp_get_power_status(struct slot *slot, u8 *status);
diff --git a/drivers/pci/hotplug/pciehp_core.c b/drivers/pci/hotplug/pciehp_core.c
index 7d32fa33..a7485bc1 100644
--- a/drivers/pci/hotplug/pciehp_core.c
+++ b/drivers/pci/hotplug/pciehp_core.c
@@ -76,6 +76,12 @@ static int reset_slot(struct hotplug_slot *slot, int probe);
*/
static void release_slot(struct hotplug_slot *hotplug_slot)
{
+ struct slot *slot = hotplug_slot->private;
+
+ /* queued work needs hotplug_slot name */
+ cancel_delayed_work(&slot->work);
+ drain_workqueue(slot->wq);
+
kfree(hotplug_slot->ops);
kfree(hotplug_slot->info);
kfree(hotplug_slot);
@@ -278,6 +284,7 @@ static void pciehp_remove(struct pcie_device *dev)
{
struct controller *ctrl = get_service_data(dev);
+ pcie_shutdown_notification(ctrl);
cleanup_slot(ctrl);
pciehp_release_ctrl(ctrl);
}
@@ -297,7 +304,7 @@ static int pciehp_resume(struct pcie_device *dev)
ctrl = get_service_data(dev);
/* reinitialize the chipset's event detection logic */
- pcie_enable_notification(ctrl);
+ pcie_reenable_notification(ctrl);
slot = ctrl->slot;
diff --git a/drivers/pci/hotplug/pciehp_hpc.c b/drivers/pci/hotplug/pciehp_hpc.c
index d08dfc8..8b8b096 100644
--- a/drivers/pci/hotplug/pciehp_hpc.c
+++ b/drivers/pci/hotplug/pciehp_hpc.c
@@ -562,8 +562,6 @@ static irqreturn_t pciehp_isr(int irq, void *dev_id)
{
struct controller *ctrl = (struct controller *)dev_id;
struct pci_dev *pdev = ctrl_dev(ctrl);
- struct pci_bus *subordinate = pdev->subordinate;
- struct pci_dev *dev;
struct slot *slot = ctrl->slot;
u16 status, events;
u8 present;
@@ -611,14 +609,9 @@ static irqreturn_t pciehp_isr(int irq, void *dev_id)
wake_up(&ctrl->queue);
}
- if (subordinate) {
- list_for_each_entry(dev, &subordinate->devices, bus_list) {
- if (dev->ignore_hotplug) {
- ctrl_dbg(ctrl, "ignoring hotplug event %#06x (%s requested no hotplug)\n",
- events, pci_name(dev));
- return IRQ_HANDLED;
- }
- }
+ if (pdev->ignore_hotplug) {
+ ctrl_dbg(ctrl, "ignoring hotplug event %#06x\n", events);
+ return IRQ_HANDLED;
}
/* Check Attention Button Pressed */
@@ -673,7 +666,7 @@ static irqreturn_t pcie_isr(int irq, void *dev_id)
return handled;
}
-void pcie_enable_notification(struct controller *ctrl)
+static void pcie_enable_notification(struct controller *ctrl)
{
u16 cmd, mask;
@@ -711,6 +704,17 @@ void pcie_enable_notification(struct controller *ctrl)
pci_pcie_cap(ctrl->pcie->port) + PCI_EXP_SLTCTL, cmd);
}
+void pcie_reenable_notification(struct controller *ctrl)
+{
+ /*
+ * Clear both Presence and Data Link Layer Changed to make sure
+ * those events still fire after we have re-enabled them.
+ */
+ pcie_capability_write_word(ctrl->pcie->port, PCI_EXP_SLTSTA,
+ PCI_EXP_SLTSTA_PDC | PCI_EXP_SLTSTA_DLLSC);
+ pcie_enable_notification(ctrl);
+}
+
static void pcie_disable_notification(struct controller *ctrl)
{
u16 mask;
@@ -775,7 +779,7 @@ int pcie_init_notification(struct controller *ctrl)
return 0;
}
-static void pcie_shutdown_notification(struct controller *ctrl)
+void pcie_shutdown_notification(struct controller *ctrl)
{
if (ctrl->notification_enabled) {
pcie_disable_notification(ctrl);
@@ -810,7 +814,7 @@ static int pcie_init_slot(struct controller *ctrl)
static void pcie_cleanup_slot(struct controller *ctrl)
{
struct slot *slot = ctrl->slot;
- cancel_delayed_work(&slot->work);
+
destroy_workqueue(slot->wq);
kfree(slot);
}
@@ -891,7 +895,6 @@ struct controller *pcie_init(struct pcie_device *dev)
void pciehp_release_ctrl(struct controller *ctrl)
{
- pcie_shutdown_notification(ctrl);
pcie_cleanup_slot(ctrl);
kfree(ctrl);
}
diff --git a/drivers/pci/msi.c b/drivers/pci/msi.c
index 0e9a9db..37f393f 100644
--- a/drivers/pci/msi.c
+++ b/drivers/pci/msi.c
@@ -981,7 +981,6 @@ static int __pci_enable_msix(struct pci_dev *dev, struct msix_entry *entries,
}
}
}
- WARN_ON(!!dev->msix_enabled);
/* Check whether driver already requested for MSI irq */
if (dev->msi_enabled) {
@@ -1068,8 +1067,6 @@ static int __pci_enable_msi_range(struct pci_dev *dev, int minvec, int maxvec,
if (!pci_msi_supported(dev, minvec))
return -EINVAL;
- WARN_ON(!!dev->msi_enabled);
-
/* Check whether driver already requested MSI-X irqs */
if (dev->msix_enabled) {
dev_info(&dev->dev,
@@ -1080,6 +1077,9 @@ static int __pci_enable_msi_range(struct pci_dev *dev, int minvec, int maxvec,
if (maxvec < minvec)
return -ERANGE;
+ if (WARN_ON_ONCE(dev->msi_enabled))
+ return -EINVAL;
+
nvec = pci_msi_vec_count(dev);
if (nvec < 0)
return nvec;
@@ -1138,6 +1138,9 @@ static int __pci_enable_msix_range(struct pci_dev *dev,
if (maxvec < minvec)
return -ERANGE;
+ if (WARN_ON_ONCE(dev->msix_enabled))
+ return -EINVAL;
+
for (;;) {
if (affinity) {
nvec = irq_calc_affinity_vectors(dev->irq_affinity,
diff --git a/drivers/pci/pci-acpi.c b/drivers/pci/pci-acpi.c
index d966d47..d38d379 100644
--- a/drivers/pci/pci-acpi.c
+++ b/drivers/pci/pci-acpi.c
@@ -567,7 +567,7 @@ void acpi_pci_add_bus(struct pci_bus *bus)
union acpi_object *obj;
struct pci_host_bridge *bridge;
- if (acpi_pci_disabled || !bus->bridge)
+ if (acpi_pci_disabled || !bus->bridge || !ACPI_HANDLE(bus->bridge))
return;
acpi_pci_slot_enumerate(bus);
diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
index d81ad84..f11c382 100644
--- a/drivers/pci/pci-driver.c
+++ b/drivers/pci/pci-driver.c
@@ -1147,11 +1147,14 @@ static int pci_pm_runtime_suspend(struct device *dev)
int error;
/*
- * If pci_dev->driver is not set (unbound), the device should
- * always remain in D0 regardless of the runtime PM status
+ * If pci_dev->driver is not set (unbound), we leave the device in D0,
+ * but it may go to D3cold when the bridge above it runtime suspends.
+ * Save its config space in case that happens.
*/
- if (!pci_dev->driver)
+ if (!pci_dev->driver) {
+ pci_save_state(pci_dev);
return 0;
+ }
if (!pm || !pm->runtime_suspend)
return -ENOSYS;
@@ -1199,16 +1202,18 @@ static int pci_pm_runtime_resume(struct device *dev)
const struct dev_pm_ops *pm = dev->driver ? dev->driver->pm : NULL;
/*
- * If pci_dev->driver is not set (unbound), the device should
- * always remain in D0 regardless of the runtime PM status
+ * Restoring config space is necessary even if the device is not bound
+ * to a driver because although we left it in D0, it may have gone to
+ * D3cold when the bridge above it runtime suspended.
*/
+ pci_restore_standard_config(pci_dev);
+
if (!pci_dev->driver)
return 0;
if (!pm || !pm->runtime_resume)
return -ENOSYS;
- pci_restore_standard_config(pci_dev);
pci_fixup_device(pci_fixup_resume_early, pci_dev);
__pci_enable_wake(pci_dev, PCI_D0, true, false);
pci_fixup_device(pci_fixup_resume, pci_dev);
diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
index f9f4d1c..e5d8e2e 100644
--- a/drivers/pci/pci-sysfs.c
+++ b/drivers/pci/pci-sysfs.c
@@ -180,13 +180,16 @@ static ssize_t enable_store(struct device *dev, struct device_attribute *attr,
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
- if (!val) {
- if (pci_is_enabled(pdev))
- pci_disable_device(pdev);
- else
- result = -EIO;
- } else
+ device_lock(dev);
+ if (dev->driver)
+ result = -EBUSY;
+ else if (val)
result = pci_enable_device(pdev);
+ else if (pci_is_enabled(pdev))
+ pci_disable_device(pdev);
+ else
+ result = -EIO;
+ device_unlock(dev);
return result < 0 ? result : count;
}
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index a87c8e1..ccbbd4c 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -1114,12 +1114,12 @@ int pci_save_state(struct pci_dev *dev)
EXPORT_SYMBOL(pci_save_state);
static void pci_restore_config_dword(struct pci_dev *pdev, int offset,
- u32 saved_val, int retry)
+ u32 saved_val, int retry, bool force)
{
u32 val;
pci_read_config_dword(pdev, offset, &val);
- if (val == saved_val)
+ if (!force && val == saved_val)
return;
for (;;) {
@@ -1138,25 +1138,36 @@ static void pci_restore_config_dword(struct pci_dev *pdev, int offset,
}
static void pci_restore_config_space_range(struct pci_dev *pdev,
- int start, int end, int retry)
+ int start, int end, int retry,
+ bool force)
{
int index;
for (index = end; index >= start; index--)
pci_restore_config_dword(pdev, 4 * index,
pdev->saved_config_space[index],
- retry);
+ retry, force);
}
static void pci_restore_config_space(struct pci_dev *pdev)
{
if (pdev->hdr_type == PCI_HEADER_TYPE_NORMAL) {
- pci_restore_config_space_range(pdev, 10, 15, 0);
+ pci_restore_config_space_range(pdev, 10, 15, 0, false);
/* Restore BARs before the command register. */
- pci_restore_config_space_range(pdev, 4, 9, 10);
- pci_restore_config_space_range(pdev, 0, 3, 0);
+ pci_restore_config_space_range(pdev, 4, 9, 10, false);
+ pci_restore_config_space_range(pdev, 0, 3, 0, false);
+ } else if (pdev->hdr_type == PCI_HEADER_TYPE_BRIDGE) {
+ pci_restore_config_space_range(pdev, 12, 15, 0, false);
+
+ /*
+ * Force rewriting of prefetch registers to avoid S3 resume
+ * issues on Intel PCI bridges that occur when these
+ * registers are not explicitly written.
+ */
+ pci_restore_config_space_range(pdev, 9, 11, 0, true);
+ pci_restore_config_space_range(pdev, 0, 8, 0, false);
} else {
- pci_restore_config_space_range(pdev, 0, 15, 0);
+ pci_restore_config_space_range(pdev, 0, 15, 0, false);
}
}
@@ -3407,6 +3418,44 @@ void pci_unmap_iospace(struct resource *res)
#endif
}
+static void devm_pci_unmap_iospace(struct device *dev, void *ptr)
+{
+ struct resource **res = ptr;
+
+ pci_unmap_iospace(*res);
+}
+
+/**
+ * devm_pci_remap_iospace - Managed pci_remap_iospace()
+ * @dev: Generic device to remap IO address for
+ * @res: Resource describing the I/O space
+ * @phys_addr: physical address of range to be mapped
+ *
+ * Managed pci_remap_iospace(). Map is automatically unmapped on driver
+ * detach.
+ */
+int devm_pci_remap_iospace(struct device *dev, const struct resource *res,
+ phys_addr_t phys_addr)
+{
+ const struct resource **ptr;
+ int error;
+
+ ptr = devres_alloc(devm_pci_unmap_iospace, sizeof(*ptr), GFP_KERNEL);
+ if (!ptr)
+ return -ENOMEM;
+
+ error = pci_remap_iospace(res, phys_addr);
+ if (error) {
+ devres_free(ptr);
+ } else {
+ *ptr = res;
+ devres_add(dev, ptr);
+ }
+
+ return error;
+}
+EXPORT_SYMBOL(devm_pci_remap_iospace);
+
static void __pci_set_master(struct pci_dev *dev, bool enable)
{
u16 old_cmd, cmd;
@@ -3756,27 +3805,49 @@ int pci_wait_for_pending_transaction(struct pci_dev *dev)
}
EXPORT_SYMBOL(pci_wait_for_pending_transaction);
-/*
- * We should only need to wait 100ms after FLR, but some devices take longer.
- * Wait for up to 1000ms for config space to return something other than -1.
- * Intel IGD requires this when an LCD panel is attached. We read the 2nd
- * dword because VFs don't implement the 1st dword.
- */
static void pci_flr_wait(struct pci_dev *dev)
{
- int i = 0;
+ int delay = 1, timeout = 60000;
u32 id;
- do {
- msleep(100);
- pci_read_config_dword(dev, PCI_COMMAND, &id);
- } while (i++ < 10 && id == ~0);
+ /*
+ * Per PCIe r3.1, sec 6.6.2, a device must complete an FLR within
+ * 100ms, but may silently discard requests while the FLR is in
+ * progress. Wait 100ms before trying to access the device.
+ */
+ msleep(100);
- if (id == ~0)
- dev_warn(&dev->dev, "Failed to return from FLR\n");
- else if (i > 1)
- dev_info(&dev->dev, "Required additional %dms to return from FLR\n",
- (i - 1) * 100);
+ /*
+ * After 100ms, the device should not silently discard config
+ * requests, but it may still indicate that it needs more time by
+ * responding to them with CRS completions. The Root Port will
+ * generally synthesize ~0 data to complete the read (except when
+ * CRS SV is enabled and the read was for the Vendor ID; in that
+ * case it synthesizes 0x0001 data).
+ *
+ * Wait for the device to return a non-CRS completion. Read the
+ * Command register instead of Vendor ID so we don't have to
+ * contend with the CRS SV value.
+ */
+ pci_read_config_dword(dev, PCI_COMMAND, &id);
+ while (id == ~0) {
+ if (delay > timeout) {
+ dev_warn(&dev->dev, "not ready %dms after FLR; giving up\n",
+ 100 + delay - 1);
+ return;
+ }
+
+ if (delay > 1000)
+ dev_info(&dev->dev, "not ready %dms after FLR; waiting\n",
+ 100 + delay - 1);
+
+ msleep(delay);
+ delay *= 2;
+ pci_read_config_dword(dev, PCI_COMMAND, &id);
+ }
+
+ if (delay > 1000)
+ dev_info(&dev->dev, "ready %dms after FLR\n", 100 + delay - 1);
}
static int pcie_flr(struct pci_dev *dev, int probe)
diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 56340ab..16611cf 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -1363,6 +1363,10 @@ static void pci_configure_mps(struct pci_dev *dev)
if (!pci_is_pcie(dev) || !bridge || !pci_is_pcie(bridge))
return;
+ /* MPS and MRRS fields are of type 'RsvdP' for VFs, short-circuit out */
+ if (dev->is_virtfn)
+ return;
+
mps = pcie_get_mps(dev);
p_mps = pcie_get_mps(bridge);
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index fb177dc..dedb120 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -1634,8 +1634,8 @@ static void quirk_pcie_mch(struct pci_dev *pdev)
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_E7520_MCH, quirk_pcie_mch);
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_E7320_MCH, quirk_pcie_mch);
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_E7525_MCH, quirk_pcie_mch);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_HUAWEI, 0x1610, quirk_pcie_mch);
+DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_HUAWEI, 0x1610, PCI_CLASS_BRIDGE_PCI, 8, quirk_pcie_mch);
/*
* It's possible for the MSI to get corrupted if shpc and acpi
@@ -3124,7 +3124,11 @@ static void disable_igfx_irq(struct pci_dev *dev)
pci_iounmap(dev, regs);
}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0042, disable_igfx_irq);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0046, disable_igfx_irq);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x004a, disable_igfx_irq);
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0102, disable_igfx_irq);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0106, disable_igfx_irq);
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x010a, disable_igfx_irq);
DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0152, disable_igfx_irq);
@@ -3857,6 +3861,8 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9120,
quirk_dma_func1_alias);
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9123,
quirk_dma_func1_alias);
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9128,
+ quirk_dma_func1_alias);
/* https://bugzilla.kernel.org/show_bug.cgi?id=42679#c14 */
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9130,
quirk_dma_func1_alias);
@@ -3872,6 +3878,9 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9182,
/* https://bugzilla.kernel.org/show_bug.cgi?id=42679#c46 */
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x91a0,
quirk_dma_func1_alias);
+/* https://bugzilla.kernel.org/show_bug.cgi?id=42679#c127 */
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9220,
+ quirk_dma_func1_alias);
/* https://bugzilla.kernel.org/show_bug.cgi?id=42679#c49 */
DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MARVELL_EXT, 0x9230,
quirk_dma_func1_alias);
@@ -4220,11 +4229,24 @@ static int pci_quirk_qcom_rp_acs(struct pci_dev *dev, u16 acs_flags)
* 0xa290-0xa29f PCI Express Root port #{0-16}
* 0xa2e7-0xa2ee PCI Express Root port #{17-24}
*
+ * Mobile chipsets are also affected, 7th & 8th Generation
+ * Specification update confirms ACS errata 22, status no fix: (7th Generation
+ * Intel Processor Family I/O for U/Y Platforms and 8th Generation Intel
+ * Processor Family I/O for U Quad Core Platforms Specification Update,
+ * August 2017, Revision 002, Document#: 334660-002)[6]
+ * Device IDs from I/O datasheet: (7th Generation Intel Processor Family I/O
+ * for U/Y Platforms and 8th Generation Intel ® Processor Family I/O for U
+ * Quad Core Platforms, Vol 1 of 2, August 2017, Document#: 334658-003)[7]
+ *
+ * 0x9d10-0x9d1b PCI Express Root port #{1-12}
+ *
* [1] http://www.intel.com/content/www/us/en/chipsets/100-series-chipset-datasheet-vol-2.html
* [2] http://www.intel.com/content/www/us/en/chipsets/100-series-chipset-datasheet-vol-1.html
* [3] http://www.intel.com/content/www/us/en/chipsets/100-series-chipset-spec-update.html
* [4] http://www.intel.com/content/www/us/en/chipsets/200-series-chipset-pch-spec-update.html
* [5] http://www.intel.com/content/www/us/en/chipsets/200-series-chipset-pch-datasheet-vol-1.html
+ * [6] https://www.intel.com/content/www/us/en/processors/core/7th-gen-core-family-mobile-u-y-processor-lines-i-o-spec-update.html
+ * [7] https://www.intel.com/content/www/us/en/processors/core/7th-gen-core-family-mobile-u-y-processor-lines-i-o-datasheet-vol-1.html
*/
static bool pci_quirk_intel_spt_pch_acs_match(struct pci_dev *dev)
{
@@ -4234,6 +4256,7 @@ static bool pci_quirk_intel_spt_pch_acs_match(struct pci_dev *dev)
switch (dev->device) {
case 0xa110 ... 0xa11f: case 0xa167 ... 0xa16a: /* Sunrise Point */
case 0xa290 ... 0xa29f: case 0xa2e7 ... 0xa2ee: /* Union Point */
+ case 0x9d10 ... 0x9d1b: /* 7th & 8th Gen Mobile */
return true;
}
diff --git a/drivers/pcmcia/ricoh.h b/drivers/pcmcia/ricoh.h
index 01098c8..8ac7b13 100644
--- a/drivers/pcmcia/ricoh.h
+++ b/drivers/pcmcia/ricoh.h
@@ -119,6 +119,10 @@
#define RL5C4XX_MISC_CONTROL 0x2F /* 8 bit */
#define RL5C4XX_ZV_ENABLE 0x08
+/* Misc Control 3 Register */
+#define RL5C4XX_MISC3 0x00A2 /* 16 bit */
+#define RL5C47X_MISC3_CB_CLKRUN_DIS BIT(1)
+
#ifdef __YENTA_H
#define rl_misc(socket) ((socket)->private[0])
@@ -156,6 +160,35 @@ static void ricoh_set_zv(struct yenta_socket *socket)
}
}
+static void ricoh_set_clkrun(struct yenta_socket *socket, bool quiet)
+{
+ u16 misc3;
+
+ /*
+ * RL5C475II likely has this setting, too, however no datasheet
+ * is publicly available for this chip
+ */
+ if (socket->dev->device != PCI_DEVICE_ID_RICOH_RL5C476 &&
+ socket->dev->device != PCI_DEVICE_ID_RICOH_RL5C478)
+ return;
+
+ if (socket->dev->revision < 0x80)
+ return;
+
+ misc3 = config_readw(socket, RL5C4XX_MISC3);
+ if (misc3 & RL5C47X_MISC3_CB_CLKRUN_DIS) {
+ if (!quiet)
+ dev_dbg(&socket->dev->dev,
+ "CLKRUN feature already disabled\n");
+ } else if (disable_clkrun) {
+ if (!quiet)
+ dev_info(&socket->dev->dev,
+ "Disabling CLKRUN feature\n");
+ misc3 |= RL5C47X_MISC3_CB_CLKRUN_DIS;
+ config_writew(socket, RL5C4XX_MISC3, misc3);
+ }
+}
+
static void ricoh_save_state(struct yenta_socket *socket)
{
rl_misc(socket) = config_readw(socket, RL5C4XX_MISC);
@@ -172,6 +205,7 @@ static void ricoh_restore_state(struct yenta_socket *socket)
config_writew(socket, RL5C4XX_16BIT_IO_0, rl_io(socket));
config_writew(socket, RL5C4XX_16BIT_MEM_0, rl_mem(socket));
config_writew(socket, RL5C4XX_CONFIG, rl_config(socket));
+ ricoh_set_clkrun(socket, true);
}
@@ -197,6 +231,7 @@ static int ricoh_override(struct yenta_socket *socket)
config_writew(socket, RL5C4XX_CONFIG, config);
ricoh_set_zv(socket);
+ ricoh_set_clkrun(socket, false);
return 0;
}
diff --git a/drivers/pcmcia/yenta_socket.c b/drivers/pcmcia/yenta_socket.c
index 5d6d9b1..5034422 100644
--- a/drivers/pcmcia/yenta_socket.c
+++ b/drivers/pcmcia/yenta_socket.c
@@ -26,7 +26,8 @@
static bool disable_clkrun;
module_param(disable_clkrun, bool, 0444);
-MODULE_PARM_DESC(disable_clkrun, "If PC card doesn't function properly, please try this option");
+MODULE_PARM_DESC(disable_clkrun,
+ "If PC card doesn't function properly, please try this option (TI and Ricoh bridges only)");
static bool isa_probe = 1;
module_param(isa_probe, bool, 0444);
diff --git a/drivers/pinctrl/bcm/pinctrl-nsp-mux.c b/drivers/pinctrl/bcm/pinctrl-nsp-mux.c
index 35c1765..87618a4 100644
--- a/drivers/pinctrl/bcm/pinctrl-nsp-mux.c
+++ b/drivers/pinctrl/bcm/pinctrl-nsp-mux.c
@@ -460,8 +460,8 @@ static int nsp_pinmux_enable(struct pinctrl_dev *pctrl_dev,
const struct nsp_pin_function *func;
const struct nsp_pin_group *grp;
- if (grp_select > pinctrl->num_groups ||
- func_select > pinctrl->num_functions)
+ if (grp_select >= pinctrl->num_groups ||
+ func_select >= pinctrl->num_functions)
return -EINVAL;
func = &pinctrl->functions[func_select];
@@ -577,6 +577,8 @@ static int nsp_pinmux_probe(struct platform_device *pdev)
return PTR_ERR(pinctrl->base0);
res = platform_get_resource(pdev, IORESOURCE_MEM, 1);
+ if (!res)
+ return -EINVAL;
pinctrl->base1 = devm_ioremap_nocache(&pdev->dev, res->start,
resource_size(res));
if (!pinctrl->base1) {
diff --git a/drivers/pinctrl/freescale/pinctrl-imx1-core.c b/drivers/pinctrl/freescale/pinctrl-imx1-core.c
index a4e9f43..e2cca91 100644
--- a/drivers/pinctrl/freescale/pinctrl-imx1-core.c
+++ b/drivers/pinctrl/freescale/pinctrl-imx1-core.c
@@ -433,7 +433,7 @@ static void imx1_pinconf_group_dbg_show(struct pinctrl_dev *pctldev,
const char *name;
int i, ret;
- if (group > info->ngroups)
+ if (group >= info->ngroups)
return;
seq_puts(s, "\n");
diff --git a/drivers/pinctrl/intel/pinctrl-intel.c b/drivers/pinctrl/intel/pinctrl-intel.c
index df63b7d..15aeeb2 100644
--- a/drivers/pinctrl/intel/pinctrl-intel.c
+++ b/drivers/pinctrl/intel/pinctrl-intel.c
@@ -368,18 +368,6 @@ static void __intel_gpio_set_direction(void __iomem *padcfg0, bool input)
writel(value, padcfg0);
}
-static void intel_gpio_set_gpio_mode(void __iomem *padcfg0)
-{
- u32 value;
-
- /* Put the pad into GPIO mode */
- value = readl(padcfg0) & ~PADCFG0_PMODE_MASK;
- /* Disable SCI/SMI/NMI generation */
- value &= ~(PADCFG0_GPIROUTIOXAPIC | PADCFG0_GPIROUTSCI);
- value &= ~(PADCFG0_GPIROUTSMI | PADCFG0_GPIROUTNMI);
- writel(value, padcfg0);
-}
-
static int intel_gpio_request_enable(struct pinctrl_dev *pctldev,
struct pinctrl_gpio_range *range,
unsigned pin)
@@ -387,6 +375,7 @@ static int intel_gpio_request_enable(struct pinctrl_dev *pctldev,
struct intel_pinctrl *pctrl = pinctrl_dev_get_drvdata(pctldev);
void __iomem *padcfg0;
unsigned long flags;
+ u32 value;
raw_spin_lock_irqsave(&pctrl->lock, flags);
@@ -396,7 +385,13 @@ static int intel_gpio_request_enable(struct pinctrl_dev *pctldev,
}
padcfg0 = intel_get_padcfg(pctrl, pin, PADCFG0);
- intel_gpio_set_gpio_mode(padcfg0);
+ /* Put the pad into GPIO mode */
+ value = readl(padcfg0) & ~PADCFG0_PMODE_MASK;
+ /* Disable SCI/SMI/NMI generation */
+ value &= ~(PADCFG0_GPIROUTIOXAPIC | PADCFG0_GPIROUTSCI);
+ value &= ~(PADCFG0_GPIROUTSMI | PADCFG0_GPIROUTNMI);
+ writel(value, padcfg0);
+
/* Disable TX buffer and enable RX (this will be input) */
__intel_gpio_set_direction(padcfg0, true);
@@ -609,12 +604,17 @@ static int intel_gpio_get(struct gpio_chip *chip, unsigned offset)
{
struct intel_pinctrl *pctrl = gpiochip_get_data(chip);
void __iomem *reg;
+ u32 padcfg0;
reg = intel_get_padcfg(pctrl, offset, PADCFG0);
if (!reg)
return -EINVAL;
- return !!(readl(reg) & PADCFG0_GPIORXSTATE);
+ padcfg0 = readl(reg);
+ if (!(padcfg0 & PADCFG0_GPIOTXDIS))
+ return !!(padcfg0 & PADCFG0_GPIOTXSTATE);
+
+ return !!(padcfg0 & PADCFG0_GPIORXSTATE);
}
static void intel_gpio_set(struct gpio_chip *chip, unsigned offset, int value)
@@ -775,8 +775,6 @@ static int intel_gpio_irq_type(struct irq_data *d, unsigned type)
raw_spin_lock_irqsave(&pctrl->lock, flags);
- intel_gpio_set_gpio_mode(reg);
-
value = readl(reg);
value &= ~(PADCFG0_RXEVCFG_MASK | PADCFG0_RXINV);
diff --git a/drivers/pinctrl/meson/pinctrl-meson.c b/drivers/pinctrl/meson/pinctrl-meson.c
index 9443c9d..8e73641 100644
--- a/drivers/pinctrl/meson/pinctrl-meson.c
+++ b/drivers/pinctrl/meson/pinctrl-meson.c
@@ -274,8 +274,9 @@ static int meson_pinconf_set(struct pinctrl_dev *pcdev, unsigned int pin,
case PIN_CONFIG_BIAS_DISABLE:
dev_dbg(pc->dev, "pin %u: disable bias\n", pin);
- meson_calc_reg_and_bit(bank, pin, REG_PULL, ®, &bit);
- ret = regmap_update_bits(pc->reg_pull, reg,
+ meson_calc_reg_and_bit(bank, pin, REG_PULLEN, ®,
+ &bit);
+ ret = regmap_update_bits(pc->reg_pullen, reg,
BIT(bit), 0);
if (ret)
return ret;
diff --git a/drivers/pinctrl/pinctrl-at91-pio4.c b/drivers/pinctrl/pinctrl-at91-pio4.c
index 28bbc1b..88ba9c5 100644
--- a/drivers/pinctrl/pinctrl-at91-pio4.c
+++ b/drivers/pinctrl/pinctrl-at91-pio4.c
@@ -573,8 +573,10 @@ static int atmel_pctl_dt_node_to_map(struct pinctrl_dev *pctldev,
for_each_child_of_node(np_config, np) {
ret = atmel_pctl_dt_subnode_to_map(pctldev, np, map,
&reserved_maps, num_maps);
- if (ret < 0)
+ if (ret < 0) {
+ of_node_put(np);
break;
+ }
}
}
diff --git a/drivers/pinctrl/qcom/pinctrl-msm.c b/drivers/pinctrl/qcom/pinctrl-msm.c
index c84a526..4d321c1 100644
--- a/drivers/pinctrl/qcom/pinctrl-msm.c
+++ b/drivers/pinctrl/qcom/pinctrl-msm.c
@@ -490,6 +490,12 @@ static void msm_gpio_dbg_show_one(struct seq_file *s,
};
g = &pctrl->soc->groups[offset];
+
+ if (pctrl->pctrl->desc->pins[offset].no_read) {
+ seq_printf(s, "%s not readable", g->name);
+ return;
+ }
+
ctl_reg = readl(pctrl->regs + g->ctl_reg);
is_out = !!(ctl_reg & BIT(g->oe_bit));
@@ -884,14 +890,16 @@ static const struct irq_domain_ops msm_gpio_domain_ops = {
static struct irq_chip msm_dirconn_irq_chip;
-static void msm_gpio_dirconn_handler(struct irq_desc *desc)
+static bool msm_gpio_dirconn_handler(struct irq_desc *desc)
{
+ int res;
struct irq_data *irqd = irq_desc_get_handler_data(desc);
struct irq_chip *chip = irq_desc_get_chip(desc);
chained_irq_enter(chip, desc);
- generic_handle_irq(irqd->irq);
+ res = generic_handle_irq(irqd->irq);
chained_irq_exit(chip, desc);
+ return res == 1;
}
static void setup_pdc_gpio(struct irq_domain *domain,
@@ -1376,7 +1384,7 @@ static struct irq_chip msm_dirconn_irq_chip = {
| IRQCHIP_SET_TYPE_MASKED,
};
-static void msm_gpio_irq_handler(struct irq_desc *desc)
+static bool msm_gpio_irq_handler(struct irq_desc *desc)
{
struct gpio_chip *gc = irq_desc_get_handler_data(desc);
const struct msm_pingroup *g;
@@ -1386,6 +1394,7 @@ static void msm_gpio_irq_handler(struct irq_desc *desc)
int handled = 0;
u32 val;
int i;
+ bool ret;
chained_irq_enter(chip, desc);
@@ -1403,11 +1412,13 @@ static void msm_gpio_irq_handler(struct irq_desc *desc)
}
}
+ ret = (handled != 0);
/* No interrupts were flagged */
if (handled == 0)
- handle_bad_irq(desc);
+ ret = handle_bad_irq(desc);
chained_irq_exit(chip, desc);
+ return ret;
}
static void msm_gpio_setup_dir_connects(struct msm_pinctrl *pctrl)
diff --git a/drivers/pinctrl/qcom/pinctrl-sdm845-v2.c b/drivers/pinctrl/qcom/pinctrl-sdm845-v2.c
index bc8ec39..0aebae2 100644
--- a/drivers/pinctrl/qcom/pinctrl-sdm845-v2.c
+++ b/drivers/pinctrl/qcom/pinctrl-sdm845-v2.c
@@ -14,6 +14,7 @@
#include <linux/module.h>
#include <linux/of.h>
#include <linux/platform_device.h>
+#include <linux/err.h>
#include <linux/pinctrl/pinctrl.h>
#include "pinctrl-msm.h"
@@ -203,10 +204,10 @@ static const struct pinctrl_pin_desc sdm845_pins[] = {
PINCTRL_PIN(78, "GPIO_78"),
PINCTRL_PIN(79, "GPIO_79"),
PINCTRL_PIN(80, "GPIO_80"),
- PINCTRL_PIN(81, "GPIO_81"),
- PINCTRL_PIN(82, "GPIO_82"),
- PINCTRL_PIN(83, "GPIO_83"),
- PINCTRL_PIN(84, "GPIO_84"),
+ PINCTRL_PIN_NR(81, "GPIO_81"),
+ PINCTRL_PIN_NR(82, "GPIO_82"),
+ PINCTRL_PIN_NR(83, "GPIO_83"),
+ PINCTRL_PIN_NR(84, "GPIO_84"),
PINCTRL_PIN(85, "GPIO_85"),
PINCTRL_PIN(86, "GPIO_86"),
PINCTRL_PIN(87, "GPIO_87"),
diff --git a/drivers/pinctrl/qcom/pinctrl-sdm845.c b/drivers/pinctrl/qcom/pinctrl-sdm845.c
index ee5bd19..006aab5 100644
--- a/drivers/pinctrl/qcom/pinctrl-sdm845.c
+++ b/drivers/pinctrl/qcom/pinctrl-sdm845.c
@@ -203,10 +203,10 @@ static const struct pinctrl_pin_desc sdm845_pins[] = {
PINCTRL_PIN(78, "GPIO_78"),
PINCTRL_PIN(79, "GPIO_79"),
PINCTRL_PIN(80, "GPIO_80"),
- PINCTRL_PIN(81, "GPIO_81"),
- PINCTRL_PIN(82, "GPIO_82"),
- PINCTRL_PIN(83, "GPIO_83"),
- PINCTRL_PIN(84, "GPIO_84"),
+ PINCTRL_PIN_NR(81, "GPIO_81"),
+ PINCTRL_PIN_NR(82, "GPIO_82"),
+ PINCTRL_PIN_NR(83, "GPIO_83"),
+ PINCTRL_PIN_NR(84, "GPIO_84"),
PINCTRL_PIN(85, "GPIO_85"),
PINCTRL_PIN(86, "GPIO_86"),
PINCTRL_PIN(87, "GPIO_87"),
diff --git a/drivers/pinctrl/qcom/pinctrl-spmi-gpio.c b/drivers/pinctrl/qcom/pinctrl-spmi-gpio.c
index bdce49b..ac83345 100644
--- a/drivers/pinctrl/qcom/pinctrl-spmi-gpio.c
+++ b/drivers/pinctrl/qcom/pinctrl-spmi-gpio.c
@@ -411,31 +411,47 @@ static int pmic_gpio_config_get(struct pinctrl_dev *pctldev,
switch (param) {
case PIN_CONFIG_DRIVE_PUSH_PULL:
- arg = pad->buffer_type == PMIC_GPIO_OUT_BUF_CMOS;
+ if (pad->buffer_type != PMIC_GPIO_OUT_BUF_CMOS)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_DRIVE_OPEN_DRAIN:
- arg = pad->buffer_type == PMIC_GPIO_OUT_BUF_OPEN_DRAIN_NMOS;
+ if (pad->buffer_type != PMIC_GPIO_OUT_BUF_OPEN_DRAIN_NMOS)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_DRIVE_OPEN_SOURCE:
- arg = pad->buffer_type == PMIC_GPIO_OUT_BUF_OPEN_DRAIN_PMOS;
+ if (pad->buffer_type != PMIC_GPIO_OUT_BUF_OPEN_DRAIN_PMOS)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_BIAS_PULL_DOWN:
- arg = pad->pullup == PMIC_GPIO_PULL_DOWN;
+ if (pad->pullup != PMIC_GPIO_PULL_DOWN)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_BIAS_DISABLE:
- arg = pad->pullup = PMIC_GPIO_PULL_DISABLE;
+ if (pad->pullup != PMIC_GPIO_PULL_DISABLE)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_BIAS_PULL_UP:
- arg = pad->pullup == PMIC_GPIO_PULL_UP_30;
+ if (pad->pullup != PMIC_GPIO_PULL_UP_30)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_BIAS_HIGH_IMPEDANCE:
- arg = !pad->is_enabled;
+ if (pad->is_enabled)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_POWER_SOURCE:
arg = pad->power_source;
break;
case PIN_CONFIG_INPUT_ENABLE:
- arg = pad->input_enabled;
+ if (!pad->input_enabled)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_OUTPUT_ENABLE:
arg = pad->output_enabled;
diff --git a/drivers/pinctrl/qcom/pinctrl-spmi-mpp.c b/drivers/pinctrl/qcom/pinctrl-spmi-mpp.c
index 3e6ee7f..09be4a6 100644
--- a/drivers/pinctrl/qcom/pinctrl-spmi-mpp.c
+++ b/drivers/pinctrl/qcom/pinctrl-spmi-mpp.c
@@ -328,6 +328,8 @@ static int pmic_mpp_set_mux(struct pinctrl_dev *pctldev, unsigned function,
pad->function = function;
ret = pmic_mpp_write_mode_ctl(state, pad);
+ if (ret < 0)
+ return ret;
val = pad->is_enabled << PMIC_MPP_REG_MASTER_EN_SHIFT;
@@ -352,13 +354,12 @@ static int pmic_mpp_config_get(struct pinctrl_dev *pctldev,
switch (param) {
case PIN_CONFIG_BIAS_DISABLE:
- arg = pad->pullup == PMIC_MPP_PULL_UP_OPEN;
+ if (pad->pullup != PMIC_MPP_PULL_UP_OPEN)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_BIAS_PULL_UP:
switch (pad->pullup) {
- case PMIC_MPP_PULL_UP_OPEN:
- arg = 0;
- break;
case PMIC_MPP_PULL_UP_0P6KOHM:
arg = 600;
break;
@@ -373,13 +374,17 @@ static int pmic_mpp_config_get(struct pinctrl_dev *pctldev,
}
break;
case PIN_CONFIG_BIAS_HIGH_IMPEDANCE:
- arg = !pad->is_enabled;
+ if (pad->is_enabled)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_POWER_SOURCE:
arg = pad->power_source;
break;
case PIN_CONFIG_INPUT_ENABLE:
- arg = pad->input_enabled;
+ if (!pad->input_enabled)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_OUTPUT:
arg = pad->out_value;
@@ -391,7 +396,9 @@ static int pmic_mpp_config_get(struct pinctrl_dev *pctldev,
arg = pad->amux_input;
break;
case PMIC_MPP_CONF_PAIRED:
- arg = pad->paired;
+ if (!pad->paired)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_DRIVE_STRENGTH:
arg = pad->drive_strength;
@@ -538,6 +545,10 @@ static int pmic_mpp_config_set(struct pinctrl_dev *pctldev, unsigned int pin,
if (ret < 0)
return ret;
+ ret = pmic_mpp_write(state, pad, PMIC_MPP_REG_SINK_CTL, pad->drive_strength);
+ if (ret < 0)
+ return ret;
+
val = pad->is_enabled << PMIC_MPP_REG_MASTER_EN_SHIFT;
return pmic_mpp_write(state, pad, PMIC_MPP_REG_EN_CTL, val);
diff --git a/drivers/pinctrl/qcom/pinctrl-ssbi-gpio.c b/drivers/pinctrl/qcom/pinctrl-ssbi-gpio.c
index a562ed7e..db3fb23 100644
--- a/drivers/pinctrl/qcom/pinctrl-ssbi-gpio.c
+++ b/drivers/pinctrl/qcom/pinctrl-ssbi-gpio.c
@@ -260,22 +260,32 @@ static int pm8xxx_pin_config_get(struct pinctrl_dev *pctldev,
switch (param) {
case PIN_CONFIG_BIAS_DISABLE:
- arg = pin->bias == PM8XXX_GPIO_BIAS_NP;
+ if (pin->bias != PM8XXX_GPIO_BIAS_NP)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_BIAS_PULL_DOWN:
- arg = pin->bias == PM8XXX_GPIO_BIAS_PD;
+ if (pin->bias != PM8XXX_GPIO_BIAS_PD)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_BIAS_PULL_UP:
- arg = pin->bias <= PM8XXX_GPIO_BIAS_PU_1P5_30;
+ if (pin->bias > PM8XXX_GPIO_BIAS_PU_1P5_30)
+ return -EINVAL;
+ arg = 1;
break;
case PM8XXX_QCOM_PULL_UP_STRENGTH:
arg = pin->pull_up_strength;
break;
case PIN_CONFIG_BIAS_HIGH_IMPEDANCE:
- arg = pin->disable;
+ if (!pin->disable)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_INPUT_ENABLE:
- arg = pin->mode == PM8XXX_GPIO_MODE_INPUT;
+ if (pin->mode != PM8XXX_GPIO_MODE_INPUT)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_OUTPUT:
if (pin->mode & PM8XXX_GPIO_MODE_OUTPUT)
@@ -290,10 +300,14 @@ static int pm8xxx_pin_config_get(struct pinctrl_dev *pctldev,
arg = pin->output_strength;
break;
case PIN_CONFIG_DRIVE_PUSH_PULL:
- arg = !pin->open_drain;
+ if (pin->open_drain)
+ return -EINVAL;
+ arg = 1;
break;
case PIN_CONFIG_DRIVE_OPEN_DRAIN:
- arg = pin->open_drain;
+ if (!pin->open_drain)
+ return -EINVAL;
+ arg = 1;
break;
default:
return -EINVAL;
diff --git a/drivers/pinctrl/sh-pfc/pfc-r8a7796.c b/drivers/pinctrl/sh-pfc/pfc-r8a7796.c
index dc9b671..2971888 100644
--- a/drivers/pinctrl/sh-pfc/pfc-r8a7796.c
+++ b/drivers/pinctrl/sh-pfc/pfc-r8a7796.c
@@ -1,7 +1,7 @@
/*
* R8A7796 processor support - PFC hardware block.
*
- * Copyright (C) 2016 Renesas Electronics Corp.
+ * Copyright (C) 2016-2017 Renesas Electronics Corp.
*
* This file is based on the drivers/pinctrl/sh-pfc/pfc-r8a7795.c
*
@@ -476,7 +476,7 @@ FM(IP16_31_28) IP16_31_28 FM(IP17_31_28) IP17_31_28
#define MOD_SEL1_26 FM(SEL_TIMER_TMU_0) FM(SEL_TIMER_TMU_1)
#define MOD_SEL1_25_24 FM(SEL_SSP1_1_0) FM(SEL_SSP1_1_1) FM(SEL_SSP1_1_2) FM(SEL_SSP1_1_3)
#define MOD_SEL1_23_22_21 FM(SEL_SSP1_0_0) FM(SEL_SSP1_0_1) FM(SEL_SSP1_0_2) FM(SEL_SSP1_0_3) FM(SEL_SSP1_0_4) F_(0, 0) F_(0, 0) F_(0, 0)
-#define MOD_SEL1_20 FM(SEL_SSI_0) FM(SEL_SSI_1)
+#define MOD_SEL1_20 FM(SEL_SSI1_0) FM(SEL_SSI1_1)
#define MOD_SEL1_19 FM(SEL_SPEED_PULSE_0) FM(SEL_SPEED_PULSE_1)
#define MOD_SEL1_18_17 FM(SEL_SIMCARD_0) FM(SEL_SIMCARD_1) FM(SEL_SIMCARD_2) FM(SEL_SIMCARD_3)
#define MOD_SEL1_16 FM(SEL_SDHI2_0) FM(SEL_SDHI2_1)
@@ -1208,7 +1208,7 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_GPSR(IP13_11_8, HSCK0),
PINMUX_IPSR_MSEL(IP13_11_8, MSIOF1_SCK_D, SEL_MSIOF1_3),
PINMUX_IPSR_MSEL(IP13_11_8, AUDIO_CLKB_A, SEL_ADG_B_0),
- PINMUX_IPSR_MSEL(IP13_11_8, SSI_SDATA1_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP13_11_8, SSI_SDATA1_B, SEL_SSI1_1),
PINMUX_IPSR_MSEL(IP13_11_8, TS_SCK0_D, SEL_TSIF0_3),
PINMUX_IPSR_MSEL(IP13_11_8, STP_ISCLK_0_D, SEL_SSP1_0_3),
PINMUX_IPSR_MSEL(IP13_11_8, RIF0_CLK_C, SEL_DRIF0_2),
@@ -1216,14 +1216,14 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_GPSR(IP13_15_12, HRX0),
PINMUX_IPSR_MSEL(IP13_15_12, MSIOF1_RXD_D, SEL_MSIOF1_3),
- PINMUX_IPSR_MSEL(IP13_15_12, SSI_SDATA2_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP13_15_12, SSI_SDATA2_B, SEL_SSI2_1),
PINMUX_IPSR_MSEL(IP13_15_12, TS_SDEN0_D, SEL_TSIF0_3),
PINMUX_IPSR_MSEL(IP13_15_12, STP_ISEN_0_D, SEL_SSP1_0_3),
PINMUX_IPSR_MSEL(IP13_15_12, RIF0_D0_C, SEL_DRIF0_2),
PINMUX_IPSR_GPSR(IP13_19_16, HTX0),
PINMUX_IPSR_MSEL(IP13_19_16, MSIOF1_TXD_D, SEL_MSIOF1_3),
- PINMUX_IPSR_MSEL(IP13_19_16, SSI_SDATA9_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP13_19_16, SSI_SDATA9_B, SEL_SSI9_1),
PINMUX_IPSR_MSEL(IP13_19_16, TS_SDAT0_D, SEL_TSIF0_3),
PINMUX_IPSR_MSEL(IP13_19_16, STP_ISD_0_D, SEL_SSP1_0_3),
PINMUX_IPSR_MSEL(IP13_19_16, RIF0_D1_C, SEL_DRIF0_2),
@@ -1231,7 +1231,7 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_GPSR(IP13_23_20, HCTS0_N),
PINMUX_IPSR_MSEL(IP13_23_20, RX2_B, SEL_SCIF2_1),
PINMUX_IPSR_MSEL(IP13_23_20, MSIOF1_SYNC_D, SEL_MSIOF1_3),
- PINMUX_IPSR_MSEL(IP13_23_20, SSI_SCK9_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP13_23_20, SSI_SCK9_A, SEL_SSI9_0),
PINMUX_IPSR_MSEL(IP13_23_20, TS_SPSYNC0_D, SEL_TSIF0_3),
PINMUX_IPSR_MSEL(IP13_23_20, STP_ISSYNC_0_D, SEL_SSP1_0_3),
PINMUX_IPSR_MSEL(IP13_23_20, RIF0_SYNC_C, SEL_DRIF0_2),
@@ -1240,7 +1240,7 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_GPSR(IP13_27_24, HRTS0_N),
PINMUX_IPSR_MSEL(IP13_27_24, TX2_B, SEL_SCIF2_1),
PINMUX_IPSR_MSEL(IP13_27_24, MSIOF1_SS1_D, SEL_MSIOF1_3),
- PINMUX_IPSR_MSEL(IP13_27_24, SSI_WS9_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP13_27_24, SSI_WS9_A, SEL_SSI9_0),
PINMUX_IPSR_MSEL(IP13_27_24, STP_IVCXO27_0_D, SEL_SSP1_0_3),
PINMUX_IPSR_MSEL(IP13_27_24, BPFCLK_A, SEL_FM_0),
PINMUX_IPSR_GPSR(IP13_27_24, AUDIO_CLKOUT2_A),
@@ -1255,7 +1255,7 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_MSEL(IP14_3_0, RX5_A, SEL_SCIF5_0),
PINMUX_IPSR_MSEL(IP14_3_0, NFWP_N_A, SEL_NDF_0),
PINMUX_IPSR_MSEL(IP14_3_0, AUDIO_CLKA_C, SEL_ADG_A_2),
- PINMUX_IPSR_MSEL(IP14_3_0, SSI_SCK2_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP14_3_0, SSI_SCK2_A, SEL_SSI2_0),
PINMUX_IPSR_MSEL(IP14_3_0, STP_IVCXO27_0_C, SEL_SSP1_0_2),
PINMUX_IPSR_GPSR(IP14_3_0, AUDIO_CLKOUT3_A),
PINMUX_IPSR_MSEL(IP14_3_0, TCLK1_B, SEL_TIMER_TMU_1),
@@ -1264,7 +1264,7 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_MSEL(IP14_7_4, TX5_A, SEL_SCIF5_0),
PINMUX_IPSR_MSEL(IP14_7_4, MSIOF1_SS2_D, SEL_MSIOF1_3),
PINMUX_IPSR_MSEL(IP14_7_4, AUDIO_CLKC_A, SEL_ADG_C_0),
- PINMUX_IPSR_MSEL(IP14_7_4, SSI_WS2_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP14_7_4, SSI_WS2_A, SEL_SSI2_0),
PINMUX_IPSR_MSEL(IP14_7_4, STP_OPWM_0_D, SEL_SSP1_0_3),
PINMUX_IPSR_GPSR(IP14_7_4, AUDIO_CLKOUT_D),
PINMUX_IPSR_MSEL(IP14_7_4, SPEEDIN_B, SEL_SPEED_PULSE_1),
@@ -1292,10 +1292,10 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_MSEL(IP14_31_28, MSIOF1_SS2_F, SEL_MSIOF1_5),
/* IPSR15 */
- PINMUX_IPSR_MSEL(IP15_3_0, SSI_SDATA1_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP15_3_0, SSI_SDATA1_A, SEL_SSI1_0),
- PINMUX_IPSR_MSEL(IP15_7_4, SSI_SDATA2_A, SEL_SSI_0),
- PINMUX_IPSR_MSEL(IP15_7_4, SSI_SCK1_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP15_7_4, SSI_SDATA2_A, SEL_SSI2_0),
+ PINMUX_IPSR_MSEL(IP15_7_4, SSI_SCK1_B, SEL_SSI1_1),
PINMUX_IPSR_GPSR(IP15_11_8, SSI_SCK34),
PINMUX_IPSR_MSEL(IP15_11_8, MSIOF1_SS1_A, SEL_MSIOF1_0),
@@ -1381,11 +1381,11 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_MSEL(IP16_27_24, RIF1_D1_A, SEL_DRIF1_0),
PINMUX_IPSR_MSEL(IP16_27_24, RIF3_D1_A, SEL_DRIF3_0),
- PINMUX_IPSR_MSEL(IP16_31_28, SSI_SDATA9_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP16_31_28, SSI_SDATA9_A, SEL_SSI9_0),
PINMUX_IPSR_MSEL(IP16_31_28, HSCK2_B, SEL_HSCIF2_1),
PINMUX_IPSR_MSEL(IP16_31_28, MSIOF1_SS1_C, SEL_MSIOF1_2),
PINMUX_IPSR_MSEL(IP16_31_28, HSCK1_A, SEL_HSCIF1_0),
- PINMUX_IPSR_MSEL(IP16_31_28, SSI_WS1_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP16_31_28, SSI_WS1_B, SEL_SSI1_1),
PINMUX_IPSR_GPSR(IP16_31_28, SCK1),
PINMUX_IPSR_MSEL(IP16_31_28, STP_IVCXO27_1_A, SEL_SSP1_1_0),
PINMUX_IPSR_GPSR(IP16_31_28, SCK5_A),
@@ -1417,7 +1417,7 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_GPSR(IP17_19_16, USB1_PWEN),
PINMUX_IPSR_MSEL(IP17_19_16, SIM0_CLK_C, SEL_SIMCARD_2),
- PINMUX_IPSR_MSEL(IP17_19_16, SSI_SCK1_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP17_19_16, SSI_SCK1_A, SEL_SSI1_0),
PINMUX_IPSR_MSEL(IP17_19_16, TS_SCK0_E, SEL_TSIF0_4),
PINMUX_IPSR_MSEL(IP17_19_16, STP_ISCLK_0_E, SEL_SSP1_0_4),
PINMUX_IPSR_MSEL(IP17_19_16, FMCLK_B, SEL_FM_1),
@@ -1427,7 +1427,7 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_GPSR(IP17_23_20, USB1_OVC),
PINMUX_IPSR_MSEL(IP17_23_20, MSIOF1_SS2_C, SEL_MSIOF1_2),
- PINMUX_IPSR_MSEL(IP17_23_20, SSI_WS1_A, SEL_SSI_0),
+ PINMUX_IPSR_MSEL(IP17_23_20, SSI_WS1_A, SEL_SSI1_0),
PINMUX_IPSR_MSEL(IP17_23_20, TS_SDAT0_E, SEL_TSIF0_4),
PINMUX_IPSR_MSEL(IP17_23_20, STP_ISD_0_E, SEL_SSP1_0_4),
PINMUX_IPSR_MSEL(IP17_23_20, FMIN_B, SEL_FM_1),
@@ -1437,7 +1437,7 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_GPSR(IP17_27_24, USB30_PWEN),
PINMUX_IPSR_GPSR(IP17_27_24, AUDIO_CLKOUT_B),
- PINMUX_IPSR_MSEL(IP17_27_24, SSI_SCK2_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP17_27_24, SSI_SCK2_B, SEL_SSI2_1),
PINMUX_IPSR_MSEL(IP17_27_24, TS_SDEN1_D, SEL_TSIF1_3),
PINMUX_IPSR_MSEL(IP17_27_24, STP_ISEN_1_D, SEL_SSP1_1_2),
PINMUX_IPSR_MSEL(IP17_27_24, STP_OPWM_0_E, SEL_SSP1_0_4),
@@ -1449,7 +1449,7 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_GPSR(IP17_31_28, USB30_OVC),
PINMUX_IPSR_GPSR(IP17_31_28, AUDIO_CLKOUT1_B),
- PINMUX_IPSR_MSEL(IP17_31_28, SSI_WS2_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP17_31_28, SSI_WS2_B, SEL_SSI2_1),
PINMUX_IPSR_MSEL(IP17_31_28, TS_SPSYNC1_D, SEL_TSIF1_3),
PINMUX_IPSR_MSEL(IP17_31_28, STP_ISSYNC_1_D, SEL_SSP1_1_3),
PINMUX_IPSR_MSEL(IP17_31_28, STP_IVCXO27_0_E, SEL_SSP1_0_4),
@@ -1460,7 +1460,7 @@ static const u16 pinmux_data[] = {
/* IPSR18 */
PINMUX_IPSR_GPSR(IP18_3_0, GP6_30),
PINMUX_IPSR_GPSR(IP18_3_0, AUDIO_CLKOUT2_B),
- PINMUX_IPSR_MSEL(IP18_3_0, SSI_SCK9_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP18_3_0, SSI_SCK9_B, SEL_SSI9_1),
PINMUX_IPSR_MSEL(IP18_3_0, TS_SDEN0_E, SEL_TSIF0_4),
PINMUX_IPSR_MSEL(IP18_3_0, STP_ISEN_0_E, SEL_SSP1_0_4),
PINMUX_IPSR_MSEL(IP18_3_0, RIF2_D0_B, SEL_DRIF2_1),
@@ -1471,7 +1471,7 @@ static const u16 pinmux_data[] = {
PINMUX_IPSR_GPSR(IP18_7_4, GP6_31),
PINMUX_IPSR_GPSR(IP18_7_4, AUDIO_CLKOUT3_B),
- PINMUX_IPSR_MSEL(IP18_7_4, SSI_WS9_B, SEL_SSI_1),
+ PINMUX_IPSR_MSEL(IP18_7_4, SSI_WS9_B, SEL_SSI9_1),
PINMUX_IPSR_MSEL(IP18_7_4, TS_SPSYNC0_E, SEL_TSIF0_4),
PINMUX_IPSR_MSEL(IP18_7_4, STP_ISSYNC_0_E, SEL_SSP1_0_4),
PINMUX_IPSR_MSEL(IP18_7_4, RIF2_D1_B, SEL_DRIF2_1),
diff --git a/drivers/pinctrl/sunxi/pinctrl-sun8i-a83t.c b/drivers/pinctrl/sunxi/pinctrl-sun8i-a83t.c
index a7c81e9..383977ea3 100644
--- a/drivers/pinctrl/sunxi/pinctrl-sun8i-a83t.c
+++ b/drivers/pinctrl/sunxi/pinctrl-sun8i-a83t.c
@@ -568,7 +568,7 @@ static const struct sunxi_desc_pin sun8i_a83t_pins[] = {
SUNXI_PIN(SUNXI_PINCTRL_PIN(H, 11),
SUNXI_FUNCTION(0x0, "gpio_in"),
SUNXI_FUNCTION(0x1, "gpio_out"),
- SUNXI_FUNCTION_IRQ_BANK(0x6, 2, 1)), /* PH_EINT11 */
+ SUNXI_FUNCTION_IRQ_BANK(0x6, 2, 11)), /* PH_EINT11 */
};
static const struct sunxi_pinctrl_desc sun8i_a83t_pinctrl_data = {
diff --git a/drivers/platform/chrome/cros_ec_lpc.c b/drivers/platform/chrome/cros_ec_lpc.c
index f9a2454..6a25bfd 100644
--- a/drivers/platform/chrome/cros_ec_lpc.c
+++ b/drivers/platform/chrome/cros_ec_lpc.c
@@ -49,7 +49,6 @@ static int ec_response_timed_out(void)
static int cros_ec_pkt_xfer_lpc(struct cros_ec_device *ec,
struct cros_ec_command *msg)
{
- struct ec_host_request *request;
struct ec_host_response response;
u8 sum = 0;
int i;
@@ -62,8 +61,6 @@ static int cros_ec_pkt_xfer_lpc(struct cros_ec_device *ec,
for (i = 0; i < ret; i++)
outb(ec->dout[i], EC_LPC_ADDR_HOST_PACKET + i);
- request = (struct ec_host_request *)ec->dout;
-
/* Here we go */
outb(EC_COMMAND_PROTOCOL_3, EC_LPC_ADDR_HOST_CMD);
diff --git a/drivers/platform/msm/Makefile b/drivers/platform/msm/Makefile
index 06ff2ca..2861716 100644
--- a/drivers/platform/msm/Makefile
+++ b/drivers/platform/msm/Makefile
@@ -16,3 +16,7 @@
obj-$(CONFIG_SEEMP_CORE) += seemp_core/
obj-$(CONFIG_QCOM_GENI_SE) += qcom-geni-se.o
obj-$(CONFIG_MSM_EXT_DISPLAY) += msm_ext_display.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-enum-conversion \
diff --git a/drivers/platform/msm/ipa/ipa_clients/rndis_ipa_trace.h b/drivers/platform/msm/ipa/ipa_clients/rndis_ipa_trace.h
index bf66c1e..3719740 100644
--- a/drivers/platform/msm/ipa/ipa_clients/rndis_ipa_trace.h
+++ b/drivers/platform/msm/ipa/ipa_clients/rndis_ipa_trace.h
@@ -77,5 +77,5 @@ TRACE_EVENT(
/* This part must be outside protection */
#undef TRACE_INCLUDE_PATH
-#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_PATH ../../drivers/platform/msm/ipa/ipa_clients
#include <trace/define_trace.h>
diff --git a/drivers/platform/msm/ipa/ipa_v2/ipa_intf.c b/drivers/platform/msm/ipa/ipa_v2/ipa_intf.c
index f1f6b90..7a84cc4 100644
--- a/drivers/platform/msm/ipa/ipa_v2/ipa_intf.c
+++ b/drivers/platform/msm/ipa/ipa_v2/ipa_intf.c
@@ -414,6 +414,7 @@ static int wlan_msg_process(struct ipa_msg_meta *meta, void *buff)
msg_dup = kzalloc(sizeof(struct ipa_push_msg), GFP_KERNEL);
if (msg_dup == NULL) {
IPAERR("fail to alloc ipa_msg container\n");
+ mutex_unlock(&ipa_ctx->msg_wlan_client_lock);
return -ENOMEM;
}
msg_dup->meta = *meta;
@@ -422,6 +423,7 @@ static int wlan_msg_process(struct ipa_msg_meta *meta, void *buff)
if (data_dup == NULL) {
IPAERR("fail to alloc data_dup container\n");
kfree(msg_dup);
+ mutex_unlock(&ipa_ctx->msg_wlan_client_lock);
return -ENOMEM;
}
memcpy(data_dup, buff, meta->msg_len);
diff --git a/drivers/platform/msm/ipa/ipa_v3/Makefile b/drivers/platform/msm/ipa/ipa_v3/Makefile
index ed78342..8942634 100644
--- a/drivers/platform/msm/ipa/ipa_v3/Makefile
+++ b/drivers/platform/msm/ipa/ipa_v3/Makefile
@@ -17,3 +17,7 @@
endif
obj-$(CONFIG_RMNET_IPA3) += rmnet_ipa.o ipa_qmi_service_v01.o ipa_qmi_service.o rmnet_ipa_fd_ioctl.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-pointer-bool-conversion \
diff --git a/drivers/platform/msm/ipa/ipa_v3/ipa_trace.h b/drivers/platform/msm/ipa/ipa_v3/ipa_trace.h
index b67899b..d92f99e 100644
--- a/drivers/platform/msm/ipa/ipa_v3/ipa_trace.h
+++ b/drivers/platform/msm/ipa/ipa_v3/ipa_trace.h
@@ -149,5 +149,5 @@ TRACE_EVENT(
/* This part must be outside protection */
#undef TRACE_INCLUDE_PATH
-#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_PATH ../../drivers/platform/msm/ipa/ipa_v3
#include <trace/define_trace.h>
diff --git a/drivers/platform/msm/ipa/ipa_v3/ipahal/Makefile b/drivers/platform/msm/ipa/ipa_v3/ipahal/Makefile
index 869ee7e..e0cbb13 100644
--- a/drivers/platform/msm/ipa/ipa_v3/ipahal/Makefile
+++ b/drivers/platform/msm/ipa/ipa_v3/ipahal/Makefile
@@ -1,3 +1,7 @@
obj-$(CONFIG_IPA3) += ipa_hal.o
ipa_hal-y := ipahal.o ipahal_reg.o ipahal_fltrt.o ipahal_hw_stats.o ipahal_nat.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-shift-overflow \
diff --git a/drivers/platform/msm/ipa/ipa_v3/rmnet_ipa.c b/drivers/platform/msm/ipa/ipa_v3/rmnet_ipa.c
index 1123f87..37598d9 100644
--- a/drivers/platform/msm/ipa/ipa_v3/rmnet_ipa.c
+++ b/drivers/platform/msm/ipa/ipa_v3/rmnet_ipa.c
@@ -1107,9 +1107,9 @@ static int ipa3_wwan_change_mtu(struct net_device *dev, int new_mtu)
* later
* -EFAULT: Error while transmitting the skb
*/
-static int ipa3_wwan_xmit(struct sk_buff *skb, struct net_device *dev)
+static netdev_tx_t ipa3_wwan_xmit(struct sk_buff *skb, struct net_device *dev)
{
- int ret = 0;
+ netdev_tx_t ret = NETDEV_TX_OK;
bool qmap_check;
struct ipa3_wwan_private *wwan_ptr = netdev_priv(dev);
diff --git a/drivers/platform/x86/acer-wmi.c b/drivers/platform/x86/acer-wmi.c
index c29b9b6..1515c94 100644
--- a/drivers/platform/x86/acer-wmi.c
+++ b/drivers/platform/x86/acer-wmi.c
@@ -1856,7 +1856,7 @@ static acpi_status __init acer_wmi_get_handle_cb(acpi_handle ah, u32 level,
if (!strcmp(ctx, "SENR")) {
if (acpi_bus_get_device(ah, &dev))
return AE_OK;
- if (!strcmp(ACER_WMID_ACCEL_HID, acpi_device_hid(dev)))
+ if (strcmp(ACER_WMID_ACCEL_HID, acpi_device_hid(dev)))
return AE_OK;
} else
return AE_OK;
@@ -1877,8 +1877,7 @@ static int __init acer_wmi_get_handle(const char *name, const char *prop,
handle = NULL;
status = acpi_get_devices(prop, acer_wmi_get_handle_cb,
(void *)name, &handle);
-
- if (ACPI_SUCCESS(status)) {
+ if (ACPI_SUCCESS(status) && handle) {
*ah = handle;
return 0;
} else {
@@ -2247,8 +2246,8 @@ static int __init acer_wmi_init(void)
if (err)
return err;
err = acer_wmi_accel_setup();
- if (err)
- return err;
+ if (err && err != -ENODEV)
+ pr_warn("Cannot enable accelerometer\n");
}
err = platform_driver_register(&acer_platform_driver);
diff --git a/drivers/platform/x86/acerhdf.c b/drivers/platform/x86/acerhdf.c
index 2acdb0d..a0533e4 100644
--- a/drivers/platform/x86/acerhdf.c
+++ b/drivers/platform/x86/acerhdf.c
@@ -233,6 +233,7 @@ static const struct bios_settings bios_tbl[] = {
{"Gateway", "LT31", "v1.3201", 0x55, 0x58, {0x9e, 0x00}, 0},
{"Gateway", "LT31", "v1.3302", 0x55, 0x58, {0x9e, 0x00}, 0},
{"Gateway", "LT31", "v1.3303t", 0x55, 0x58, {0x9e, 0x00}, 0},
+ {"Gateway", "LT31", "v1.3307", 0x55, 0x58, {0x9e, 0x00}, 0},
/* Packard Bell */
{"Packard Bell", "DOA150", "v0.3104", 0x55, 0x58, {0x21, 0x00}, 0},
{"Packard Bell", "DOA150", "v0.3105", 0x55, 0x58, {0x20, 0x00}, 0},
diff --git a/drivers/platform/x86/alienware-wmi.c b/drivers/platform/x86/alienware-wmi.c
index 0056294..fe41993 100644
--- a/drivers/platform/x86/alienware-wmi.c
+++ b/drivers/platform/x86/alienware-wmi.c
@@ -518,6 +518,7 @@ static acpi_status alienware_wmax_command(struct wmax_basic_args *in_args,
if (obj && obj->type == ACPI_TYPE_INTEGER)
*out_data = (u32) obj->integer.value;
}
+ kfree(output.pointer);
return status;
}
diff --git a/drivers/platform/x86/asus-nb-wmi.c b/drivers/platform/x86/asus-nb-wmi.c
index 687cc5b..c857d2d 100644
--- a/drivers/platform/x86/asus-nb-wmi.c
+++ b/drivers/platform/x86/asus-nb-wmi.c
@@ -531,6 +531,7 @@ static const struct key_entry asus_nb_wmi_keymap[] = {
{ KE_KEY, 0xC4, { KEY_KBDILLUMUP } },
{ KE_KEY, 0xC5, { KEY_KBDILLUMDOWN } },
{ KE_IGNORE, 0xC6, }, /* Ambient Light Sensor notification */
+ { KE_KEY, 0xFA, { KEY_PROG2 } }, /* Lid flip action */
{ KE_END, 0},
};
diff --git a/drivers/platform/x86/asus-wireless.c b/drivers/platform/x86/asus-wireless.c
index 1871602..c098328 100644
--- a/drivers/platform/x86/asus-wireless.c
+++ b/drivers/platform/x86/asus-wireless.c
@@ -145,8 +145,10 @@ static int asus_wireless_remove(struct acpi_device *adev)
{
struct asus_wireless_data *data = acpi_driver_data(adev);
- if (data->wq)
+ if (data->wq) {
+ devm_led_classdev_unregister(&adev->dev, &data->led);
destroy_workqueue(data->wq);
+ }
return 0;
}
diff --git a/drivers/platform/x86/asus-wmi.c b/drivers/platform/x86/asus-wmi.c
index 8a1bfd4..ed27768 100644
--- a/drivers/platform/x86/asus-wmi.c
+++ b/drivers/platform/x86/asus-wmi.c
@@ -161,6 +161,16 @@ MODULE_LICENSE("GPL");
static const char * const ashs_ids[] = { "ATK4001", "ATK4002", NULL };
+static bool ashs_present(void)
+{
+ int i = 0;
+ while (ashs_ids[i]) {
+ if (acpi_dev_found(ashs_ids[i++]))
+ return true;
+ }
+ return false;
+}
+
struct bios_args {
u32 arg0;
u32 arg1;
@@ -966,6 +976,9 @@ static int asus_new_rfkill(struct asus_wmi *asus,
static void asus_wmi_rfkill_exit(struct asus_wmi *asus)
{
+ if (asus->driver->wlan_ctrl_by_user && ashs_present())
+ return;
+
asus_unregister_rfkill_notifier(asus, "\\_SB.PCI0.P0P5");
asus_unregister_rfkill_notifier(asus, "\\_SB.PCI0.P0P6");
asus_unregister_rfkill_notifier(asus, "\\_SB.PCI0.P0P7");
@@ -2062,16 +2075,6 @@ static int asus_wmi_fan_init(struct asus_wmi *asus)
return 0;
}
-static bool ashs_present(void)
-{
- int i = 0;
- while (ashs_ids[i]) {
- if (acpi_dev_found(ashs_ids[i++]))
- return true;
- }
- return false;
-}
-
/*
* WMI Driver
*/
diff --git a/drivers/platform/x86/intel_ips.c b/drivers/platform/x86/intel_ips.c
index 55663b3..58dcee5 100644
--- a/drivers/platform/x86/intel_ips.c
+++ b/drivers/platform/x86/intel_ips.c
@@ -68,6 +68,7 @@
#include <linux/module.h>
#include <linux/pci.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/seq_file.h>
#include <linux/string.h>
#include <linux/tick.h>
diff --git a/drivers/platform/x86/intel_punit_ipc.c b/drivers/platform/x86/intel_punit_ipc.c
index b5b8901..b7dfe062 100644
--- a/drivers/platform/x86/intel_punit_ipc.c
+++ b/drivers/platform/x86/intel_punit_ipc.c
@@ -17,6 +17,7 @@
#include <linux/bitops.h>
#include <linux/device.h>
#include <linux/interrupt.h>
+#include <linux/io.h>
#include <linux/platform_device.h>
#include <asm/intel_punit_ipc.h>
diff --git a/drivers/platform/x86/intel_telemetry_debugfs.c b/drivers/platform/x86/intel_telemetry_debugfs.c
index ef29f18..4069433 100644
--- a/drivers/platform/x86/intel_telemetry_debugfs.c
+++ b/drivers/platform/x86/intel_telemetry_debugfs.c
@@ -953,12 +953,16 @@ static int __init telemetry_debugfs_init(void)
debugfs_conf = (struct telemetry_debugfs_conf *)id->driver_data;
err = telemetry_pltconfig_valid();
- if (err < 0)
+ if (err < 0) {
+ pr_info("Invalid pltconfig, ensure IPC1 device is enabled in BIOS\n");
return -ENODEV;
+ }
err = telemetry_debugfs_check_evts();
- if (err < 0)
+ if (err < 0) {
+ pr_info("telemetry_debugfs_check_evts failed\n");
return -EINVAL;
+ }
#ifdef CONFIG_PM_SLEEP
diff --git a/drivers/platform/x86/toshiba_acpi.c b/drivers/platform/x86/toshiba_acpi.c
index 074bf2f..79a2289 100644
--- a/drivers/platform/x86/toshiba_acpi.c
+++ b/drivers/platform/x86/toshiba_acpi.c
@@ -34,6 +34,7 @@
#define TOSHIBA_ACPI_VERSION "0.24"
#define PROC_INTERFACE_VERSION 1
+#include <linux/compiler.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/moduleparam.h>
@@ -1687,7 +1688,7 @@ static const struct file_operations keys_proc_fops = {
.write = keys_proc_write,
};
-static int version_proc_show(struct seq_file *m, void *v)
+static int __maybe_unused version_proc_show(struct seq_file *m, void *v)
{
seq_printf(m, "driver: %s\n", TOSHIBA_ACPI_VERSION);
seq_printf(m, "proc_interface: %d\n", PROC_INTERFACE_VERSION);
diff --git a/drivers/power/reset/msm-poweroff.c b/drivers/power/reset/msm-poweroff.c
index 1a28e56..96e0bb3 100644
--- a/drivers/power/reset/msm-poweroff.c
+++ b/drivers/power/reset/msm-poweroff.c
@@ -25,6 +25,10 @@
#include <linux/delay.h>
#include <linux/input/qpnp-power-on.h>
#include <linux/of_address.h>
+#include <linux/kdebug.h>
+#include <linux/notifier.h>
+#include <linux/kallsyms.h>
+#include <linux/io.h>
#include <asm/cacheflush.h>
#include <asm/system_misc.h>
@@ -49,8 +53,19 @@
#define SCM_DLOAD_MINIDUMP 0X20
#define SCM_DLOAD_BOTHDUMPS (SCM_DLOAD_MINIDUMP | SCM_DLOAD_FULLDUMP)
+#define MAX_SZ_DIAG_ERR_MSG 100
+
+struct reboot_params {
+ u32 abnrst;
+ u32 xbl_log_addr;
+ u32 ddr_vendor;
+ u8 msg[0];
+};
+
static int restart_mode;
static void __iomem *restart_reason, *dload_type_addr;
+static struct reboot_params *reboot_params;
+static size_t rst_msg_size;
static bool scm_pmic_arbiter_disable_supported;
static bool scm_deassert_ps_hold_supported;
/* Download mode master kill-switch */
@@ -94,9 +109,70 @@ struct reset_attribute {
module_param_call(download_mode, dload_set, param_get_int,
&download_mode, 0644);
+static bool warm_reset;
+module_param(warm_reset, bool, 0644);
+MODULE_PARM_DESC(warm_reset, "Set 1 to override default cold-reset");
+
+static struct die_args *tombstone;
+
+void set_restart_msg(const char *msg)
+{
+ if (!reboot_params || rst_msg_size == 0)
+ return;
+
+ pr_info("%s: set restart msg = `%s'\r\n", __func__, msg?:"<null>");
+ memset_io(reboot_params->msg, 0, rst_msg_size);
+ memcpy_toio(reboot_params->msg, msg,
+ min(strlen(msg), rst_msg_size - 1));
+}
+EXPORT_SYMBOL(set_restart_msg);
+
+int die_notify(struct notifier_block *self,
+ unsigned long val, void *data)
+{
+ static struct die_args args;
+
+ memcpy(&args, data, sizeof(args));
+ tombstone = &args;
+ pr_debug("saving oops: %pK\n", (void *) tombstone);
+ return NOTIFY_DONE;
+}
+
+static struct notifier_block die_nb = {
+ .notifier_call = die_notify,
+};
+
static int panic_prep_restart(struct notifier_block *this,
unsigned long event, void *ptr)
{
+ char kernel_panic_msg[MAX_SZ_DIAG_ERR_MSG] = "Kernel Panic";
+
+ if (rst_msg_size <= 0)
+ goto out;
+
+ if (tombstone) { /* tamper the panic message for Oops */
+ char pc_symn[KSYM_NAME_LEN] = "<unknown>";
+ char lr_symn[KSYM_NAME_LEN] = "<unknown>";
+
+#if defined(CONFIG_ARM)
+ sprint_symbol(pc_symn, tombstone->regs->ARM_pc);
+ sprint_symbol(lr_symn, tombstone->regs->ARM_lr);
+#elif defined(CONFIG_ARM64)
+ sprint_symbol(pc_symn, tombstone->regs->pc);
+ sprint_symbol(lr_symn, tombstone->regs->regs[30]);
+#endif
+
+ snprintf(kernel_panic_msg, rst_msg_size - 1,
+ "KP: %s PC:%s LR:%s",
+ current->comm, pc_symn, lr_symn);
+ } else {
+ snprintf(kernel_panic_msg, rst_msg_size - 1,
+ "KP: %s", (char *)ptr);
+ }
+
+ set_restart_msg(kernel_panic_msg);
+
+out:
in_panic = 1;
return NOTIFY_DONE;
}
@@ -246,7 +322,11 @@ static void halt_spmi_pmic_arbiter(void)
static void msm_restart_prepare(const char *cmd)
{
- bool need_warm_reset = false;
+ bool need_warm_reset = warm_reset;
+
+ /* configure reset reason back to 0 before reset */
+ qpnp_pon_set_restart_reason(PON_RESTART_REASON_UNKNOWN);
+
#ifdef CONFIG_QCOM_DLOAD_MODE
/* Write download mode flags if we're panic'ing
* Write download mode flags if restart_mode says so
@@ -263,9 +343,8 @@ static void msm_restart_prepare(const char *cmd)
((cmd != NULL && cmd[0] != '\0') &&
!strcmp(cmd, "edl")))
need_warm_reset = true;
- } else {
- need_warm_reset = (get_dload_mode() ||
- (cmd != NULL && cmd[0] != '\0'));
+ } else if (get_dload_mode() || (cmd != NULL && cmd[0] != '\0')) {
+ need_warm_reset = true;
}
/* Hard reset the PMIC unless memory contents must be maintained. */
@@ -274,8 +353,14 @@ static void msm_restart_prepare(const char *cmd)
else
qpnp_pon_system_pwr_off(PON_POWER_OFF_HARD_RESET);
- if (cmd != NULL) {
- if (!strncmp(cmd, "bootloader", 10)) {
+ if (in_panic) {
+ qpnp_pon_set_restart_reason(PON_RESTART_REASON_PANIC);
+ } else if (cmd != NULL) {
+ if (!strncmp(cmd, "packout", strlen("packout"))) {
+ qpnp_pon_set_restart_reason(
+ PON_RESTART_REASON_OEM_PACKOUT);
+ __raw_writel(0x77665500, restart_reason);
+ } else if (!strncmp(cmd, "bootloader", 10)) {
qpnp_pon_set_restart_reason(
PON_RESTART_REASON_BOOTLOADER);
__raw_writel(0x77665500, restart_reason);
@@ -565,6 +650,42 @@ static struct attribute_group reset_attr_group = {
.attrs = reset_attrs,
};
+int restart_handler_init(void)
+{
+ struct device_node *np;
+ struct resource res;
+ u32 rst_info_size = 0;
+
+ np = of_find_compatible_node(NULL, NULL,
+ "msm-imem-restart_info");
+ if (!np) {
+ pr_err("unable to find DT imem restart info node\n");
+ return -ENOENT;
+ } else {
+ reboot_params = of_iomap(np, 0);
+ if (!reboot_params) {
+ pr_err("unable to map imem restart info offset\n");
+ return -ENOMEM;
+ }
+
+ of_address_to_resource(np, 0, &res);
+ rst_info_size = resource_size(&res);
+ if (rst_info_size == 0) {
+ pr_err("%s: Failed to find info_size.\n", __func__);
+ iounmap(reboot_params);
+ return -EINVAL;
+ }
+ }
+
+ rst_msg_size = (size_t) rst_info_size -
+ offsetof(struct reboot_params, msg);
+ rst_msg_size = min(rst_msg_size, (size_t)MAX_SZ_DIAG_ERR_MSG);
+
+ set_restart_msg("Unknown");
+ pr_debug("%s: default message is set\n", __func__);
+ return 0;
+}
+
static int msm_restart_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
@@ -572,10 +693,14 @@ static int msm_restart_probe(struct platform_device *pdev)
struct device_node *np;
int ret = 0;
+ if (restart_handler_init() < 0)
+ pr_err("restart_handler_init failure\n");
+
#ifdef CONFIG_QCOM_DLOAD_MODE
if (scm_is_call_available(SCM_SVC_BOOT, SCM_DLOAD_CMD) > 0)
scm_dload_supported = true;
+ register_die_notifier(&die_nb);
atomic_notifier_chain_register(&panic_notifier_list, &panic_blk);
np = of_find_compatible_node(NULL, NULL, DL_MODE_PROP);
if (!np) {
diff --git a/drivers/power/reset/vexpress-poweroff.c b/drivers/power/reset/vexpress-poweroff.c
index 102f95a..e9e749f 100644
--- a/drivers/power/reset/vexpress-poweroff.c
+++ b/drivers/power/reset/vexpress-poweroff.c
@@ -35,6 +35,7 @@ static void vexpress_reset_do(struct device *dev, const char *what)
}
static struct device *vexpress_power_off_device;
+static atomic_t vexpress_restart_nb_refcnt = ATOMIC_INIT(0);
static void vexpress_power_off(void)
{
@@ -99,10 +100,13 @@ static int _vexpress_register_restart_handler(struct device *dev)
int err;
vexpress_restart_device = dev;
- err = register_restart_handler(&vexpress_restart_nb);
- if (err) {
- dev_err(dev, "cannot register restart handler (err=%d)\n", err);
- return err;
+ if (atomic_inc_return(&vexpress_restart_nb_refcnt) == 1) {
+ err = register_restart_handler(&vexpress_restart_nb);
+ if (err) {
+ dev_err(dev, "cannot register restart handler (err=%d)\n", err);
+ atomic_dec(&vexpress_restart_nb_refcnt);
+ return err;
+ }
}
device_create_file(dev, &dev_attr_active);
diff --git a/drivers/power/supply/Kconfig b/drivers/power/supply/Kconfig
index a5cbb71..41360b4e 100644
--- a/drivers/power/supply/Kconfig
+++ b/drivers/power/supply/Kconfig
@@ -250,6 +250,15 @@
with MAX17042. This driver also supports max17047/50 chips which are
improved version of max17042.
+config BATTERY_MAX1720X
+ tristate "Maxim MAX17201/MAX17205 Fuel Gauge"
+ depends on I2C
+ help
+ MAX17201 is fuel-gauge systems for lithium-ion (Li+) batteries
+ in handheld and portable equipment. The MAX17201 is configured
+ to operate with a single lithium cell.
+ Say Y to include support for the MAX17201/MAX17205 Fuel Gauge.
+
config BATTERY_Z2
tristate "Z2 battery driver"
depends on I2C && MACH_ZIPIT2
@@ -521,6 +530,18 @@
usb_psy to disable CC detection on the event of any water/debris
detected at USBIN.
+config GOOGLE_CHARGER
+ tristate "Google Battery Charging Control"
+ help
+ Say Y here to choose the Google battery charging control.
+ The driver performs a multi CC-CV steps charging setting the Constant
+ Charge Current given battery temperature and voltage limits.
+
+config USB_OVERHEAT_MITIGATION
+ tristate "Google USB Overheat Mitigation"
+ help
+ Say Y here to enable Google USB overheat mitigation
+
source "drivers/power/supply/qcom/Kconfig"
endif # POWER_SUPPLY
diff --git a/drivers/power/supply/Makefile b/drivers/power/supply/Makefile
index f87bca1..68213a3 100644
--- a/drivers/power/supply/Makefile
+++ b/drivers/power/supply/Makefile
@@ -39,6 +39,7 @@
obj-$(CONFIG_BATTERY_DA9150) += da9150-fg.o
obj-$(CONFIG_BATTERY_MAX17040) += max17040_battery.o
obj-$(CONFIG_BATTERY_MAX17042) += max17042_battery.o
+obj-$(CONFIG_BATTERY_MAX1720X) += max1720x_battery.o
obj-$(CONFIG_BATTERY_Z2) += z2_battery.o
obj-$(CONFIG_BATTERY_RT5033) += rt5033_battery.o
obj-$(CONFIG_CHARGER_RT9455) += rt9455_charger.o
@@ -74,3 +75,5 @@
obj-$(CONFIG_ARCH_QCOM) += qcom/
obj-$(CONFIG_AXP288_CHARGER) += axp288_charger.o
obj-$(CONFIG_NX30P6093) += nx30p6093.o
+obj-$(CONFIG_GOOGLE_CHARGER) += google_charger.o
+obj-$(CONFIG_USB_OVERHEAT_MITIGATION) += overheat_mitigation.o
diff --git a/drivers/power/supply/bq2415x_charger.c b/drivers/power/supply/bq2415x_charger.c
index 73e2f0b..c4770a9 100644
--- a/drivers/power/supply/bq2415x_charger.c
+++ b/drivers/power/supply/bq2415x_charger.c
@@ -1569,6 +1569,11 @@ static int bq2415x_probe(struct i2c_client *client,
acpi_id =
acpi_match_device(client->dev.driver->acpi_match_table,
&client->dev);
+ if (!acpi_id) {
+ dev_err(&client->dev, "failed to match device name\n");
+ ret = -ENODEV;
+ goto error_1;
+ }
name = kasprintf(GFP_KERNEL, "%s-%d", acpi_id->id, num);
}
if (!name) {
diff --git a/drivers/power/supply/generic-adc-battery.c b/drivers/power/supply/generic-adc-battery.c
index edb36bf..f627b39 100644
--- a/drivers/power/supply/generic-adc-battery.c
+++ b/drivers/power/supply/generic-adc-battery.c
@@ -243,10 +243,10 @@ static int gab_probe(struct platform_device *pdev)
struct power_supply_desc *psy_desc;
struct power_supply_config psy_cfg = {};
struct gab_platform_data *pdata = pdev->dev.platform_data;
- enum power_supply_property *properties;
int ret = 0;
int chan;
- int index = 0;
+ int index = ARRAY_SIZE(gab_props);
+ bool any = false;
adc_bat = devm_kzalloc(&pdev->dev, sizeof(*adc_bat), GFP_KERNEL);
if (!adc_bat) {
@@ -280,8 +280,6 @@ static int gab_probe(struct platform_device *pdev)
}
memcpy(psy_desc->properties, gab_props, sizeof(gab_props));
- properties = (enum power_supply_property *)
- ((char *)psy_desc->properties + sizeof(gab_props));
/*
* getting channel from iio and copying the battery properties
@@ -295,15 +293,22 @@ static int gab_probe(struct platform_device *pdev)
adc_bat->channel[chan] = NULL;
} else {
/* copying properties for supported channels only */
- memcpy(properties + sizeof(*(psy_desc->properties)) * index,
- &gab_dyn_props[chan],
- sizeof(gab_dyn_props[chan]));
- index++;
+ int index2;
+
+ for (index2 = 0; index2 < index; index2++) {
+ if (psy_desc->properties[index2] ==
+ gab_dyn_props[chan])
+ break; /* already known */
+ }
+ if (index2 == index) /* really new */
+ psy_desc->properties[index++] =
+ gab_dyn_props[chan];
+ any = true;
}
}
/* none of the channels are supported so let's bail out */
- if (index == 0) {
+ if (!any) {
ret = -ENODEV;
goto second_mem_fail;
}
@@ -314,7 +319,7 @@ static int gab_probe(struct platform_device *pdev)
* as come channels may be not be supported by the device.So
* we need to take care of that.
*/
- psy_desc->num_properties = ARRAY_SIZE(gab_props) + index;
+ psy_desc->num_properties = index;
adc_bat->psy = power_supply_register(&pdev->dev, psy_desc, &psy_cfg);
if (IS_ERR(adc_bat->psy)) {
diff --git a/drivers/power/supply/google_charger.c b/drivers/power/supply/google_charger.c
new file mode 100644
index 0000000..c21bf91
--- /dev/null
+++ b/drivers/power/supply/google_charger.c
@@ -0,0 +1,1357 @@
+/*
+ * Copyright 2018 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/kernel.h>
+#include <linux/printk.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/power_supply.h>
+#include <linux/pm_wakeup.h>
+
+#ifdef CONFIG_DEBUG_FS
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+#endif
+
+#define CHG_TEMP_NB_LIMITS_MAX 10
+#define CHG_VOLT_NB_LIMITS_MAX 5
+#define CHG_DELAY_INIT_MS 250
+#define CHG_DELAY_INIT_DETECT_MS 1000
+
+#define DEFAULT_CHARGE_STOP_LEVEL 100
+#define DEFAULT_CHARGE_START_LEVEL 0
+
+struct chg_profile {
+ u32 update_interval;
+ u32 battery_capacity;
+ int temp_nb_limits;
+ s32 temp_limits[CHG_TEMP_NB_LIMITS_MAX];
+ int volt_nb_limits;
+ s32 volt_limits[CHG_VOLT_NB_LIMITS_MAX];
+ /* Array of constant current limits */
+ s32 *cccm_limits;
+ u32 cv_hw_resolution;
+ u32 cc_hw_resolution;
+ u32 fv_uv_margin_dpct;
+ u32 cv_range_accuracy;
+ u32 cv_otv_margin;
+ u32 cv_debounce_cnt;
+ u32 cv_update_interval;
+ u32 chg_cc_tolerance;
+ u32 cv_tier_ov_cnt;
+ u32 cv_tier_switch_cnt;
+};
+
+struct chg_drv {
+ struct device *device;
+ struct power_supply *chg_psy;
+ const char *chg_psy_name;
+ struct power_supply *usb_psy;
+ struct power_supply *wlc_psy;
+ const char *wlc_psy_name;
+ struct power_supply *bat_psy;
+ const char *bat_psy_name;
+ struct chg_profile chg_profile;
+ struct notifier_block psy_nb;
+
+ struct delayed_work init_work;
+ struct delayed_work chg_work;
+ struct wakeup_source chg_ws;
+
+ bool stop_charging;
+ int temp_idx;
+ int vbatt_idx;
+ int checked_cv_cnt;
+ int checked_ov_cnt;
+ int checked_tier_switch_cnt;
+
+ int chg_mode;
+ int fv_uv;
+ int disable_charging;
+ int disable_pwrsrc;
+ bool lowerdb_reached;
+ int charge_stop_level;
+ int charge_start_level;
+};
+
+/* Used as left operand also */
+#define CCCM_LIMITS(profile, ti, vi) \
+ profile->cccm_limits[(ti * profile->volt_nb_limits) + vi]
+
+static int psy_changed(struct notifier_block *nb,
+ unsigned long action, void *data)
+{
+ struct power_supply *psy = data;
+ struct chg_drv *chg_drv = container_of(nb, struct chg_drv, psy_nb);
+
+ pr_debug("name=%s evt=%lu\n", psy->desc->name, action);
+
+ if ((action != PSY_EVENT_PROP_CHANGED) ||
+ (psy == NULL) || (psy->desc == NULL) || (psy->desc->name == NULL))
+ return NOTIFY_OK;
+
+ if (action == PSY_EVENT_PROP_CHANGED &&
+ (!strcmp(psy->desc->name, chg_drv->chg_psy_name) ||
+ !strcmp(psy->desc->name, chg_drv->bat_psy_name) ||
+ !strcmp(psy->desc->name, "usb") ||
+ (chg_drv->wlc_psy_name &&
+ !strcmp(psy->desc->name, chg_drv->wlc_psy_name)))) {
+ cancel_delayed_work(&chg_drv->chg_work);
+ schedule_delayed_work(&chg_drv->chg_work, 0);
+ }
+ return NOTIFY_OK;
+}
+
+static char *psy_chgt_str[] = {
+ "Unknown", "N/A", "Trickle", "Fast", "Taper"
+};
+
+static char *psy_usb_type_str[] = {
+ "Unknown", "Battery", "UPS", "Mains", "USB", "USB_DCP",
+ "USB_CDP", "USB_ACA", "USB_HVDCP", "USB_HVDCP_3", "USB_PD",
+ "Wireless", "USB_FLOAT", "BMS", "Parallel", "Main", "Wipower",
+ "TYPEC", "TYPEC_UFP", "TYPEC_DFP"
+};
+
+#define PSY_GET_PROP(psy, psp) psy_get_prop(psy, psp, #psp)
+static inline int psy_get_prop(struct power_supply *psy,
+ enum power_supply_property psp, char *prop_name)
+{
+ union power_supply_propval val;
+ int ret = 0;
+
+ if (!psy)
+ return -EINVAL;
+ ret = power_supply_get_property(psy, psp, &val);
+ if (ret < 0) {
+ pr_err("failed to get %s from '%s', ret=%d\n",
+ prop_name, psy->desc->name, ret);
+ return -EINVAL;
+ }
+ pr_debug("get %s for '%s' => %d\n",
+ prop_name, psy->desc->name, val.intval);
+ return val.intval;
+}
+
+#define PSY_SET_PROP(psy, psp, val) psy_set_prop(psy, psp, val, #psp)
+static inline int psy_set_prop(struct power_supply *psy,
+ enum power_supply_property psp,
+ int intval, char *prop_name)
+{
+ union power_supply_propval val;
+ int ret = 0;
+
+ if (!psy)
+ return -EINVAL;
+ val.intval = intval;
+ pr_debug("set %s for '%s' to %d\n", prop_name, psy->desc->name, intval);
+ ret = power_supply_set_property(psy, psp, &val);
+ if (ret < 0) {
+ pr_err("failed to set %s for '%s', ret=%d\n",
+ prop_name, psy->desc->name, ret);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static inline void reset_chg_drv_state(struct chg_drv *chg_drv)
+{
+ chg_drv->temp_idx = -1;
+ chg_drv->vbatt_idx = -1;
+ chg_drv->fv_uv = -1;
+ chg_drv->checked_cv_cnt = 0;
+ chg_drv->checked_ov_cnt = 0;
+ chg_drv->checked_tier_switch_cnt = 0;
+ chg_drv->disable_charging = 0;
+ chg_drv->disable_pwrsrc = 0;
+ chg_drv->lowerdb_reached = true;
+ chg_drv->stop_charging = true;
+ PSY_SET_PROP(chg_drv->chg_psy,
+ POWER_SUPPLY_PROP_TAPER_CONTROL,
+ POWER_SUPPLY_TAPER_CONTROL_OFF);
+ PSY_SET_PROP(chg_drv->chg_psy, POWER_SUPPLY_PROP_CHARGE_DISABLE, 1);
+}
+
+static void pr_info_states(struct power_supply *chg_psy,
+ struct power_supply *usb_psy,
+ struct power_supply *wlc_psy,
+ int temp, int ibatt, int vbatt, int vchrg,
+ int chg_type, int fv_uv,
+ int soc, int usb_present, int wlc_online)
+{
+ int usb_type = PSY_GET_PROP(usb_psy, POWER_SUPPLY_PROP_REAL_TYPE);
+
+ pr_info("l=%d vb=%d vc=%d c=%d fv=%d t=%d s=%s usb=%d wlc=%d\n",
+ soc, vbatt / 1000, vchrg / 1000, ibatt / 1000,
+ fv_uv, temp, psy_chgt_str[chg_type],
+ usb_present, wlc_online);
+
+ if (usb_present)
+ pr_info("usbchg=%s usbv=%d usbc=%d usbMv=%d usbMc=%d\n",
+ psy_usb_type_str[usb_type],
+ PSY_GET_PROP(usb_psy,
+ POWER_SUPPLY_PROP_VOLTAGE_NOW) / 1000,
+ PSY_GET_PROP(usb_psy,
+ POWER_SUPPLY_PROP_INPUT_CURRENT_NOW)/1000,
+ PSY_GET_PROP(usb_psy,
+ POWER_SUPPLY_PROP_VOLTAGE_MAX) / 1000,
+ PSY_GET_PROP(usb_psy,
+ POWER_SUPPLY_PROP_CURRENT_MAX) / 1000);
+
+ if (wlc_online)
+ pr_info("wlcv=%d wlcc=%d wlcMv=%d wlcMc=%d wlct=%d\n",
+ PSY_GET_PROP(wlc_psy,
+ POWER_SUPPLY_PROP_VOLTAGE_NOW) / 1000,
+ PSY_GET_PROP(wlc_psy,
+ POWER_SUPPLY_PROP_CURRENT_NOW) / 1000,
+ PSY_GET_PROP(wlc_psy,
+ POWER_SUPPLY_PROP_VOLTAGE_MAX) / 1000,
+ PSY_GET_PROP(wlc_psy,
+ POWER_SUPPLY_PROP_CURRENT_MAX) / 1000,
+ PSY_GET_PROP(wlc_psy, POWER_SUPPLY_PROP_TEMP));
+}
+
+/* returns 1 if charging should be disabled given the current battery capacity
+ * given in percent, return 0 if charging should happen
+ */
+static int is_charging_disabled(struct chg_drv *chg_drv, int capacity)
+{
+ int disable_charging = 0;
+ int upperbd = chg_drv->charge_stop_level;
+ int lowerbd = chg_drv->charge_start_level;
+
+ if ((upperbd == DEFAULT_CHARGE_STOP_LEVEL) &&
+ (lowerbd == DEFAULT_CHARGE_START_LEVEL))
+ return 0;
+
+ if ((upperbd > lowerbd) &&
+ (upperbd <= DEFAULT_CHARGE_STOP_LEVEL) &&
+ (lowerbd >= DEFAULT_CHARGE_START_LEVEL)) {
+ if (chg_drv->lowerdb_reached && upperbd <= capacity) {
+ pr_info("%s: lowerbd=%d, upperbd=%d, capacity=%d, lowerdb_reached=1->0, charging off\n",
+ __func__, lowerbd, upperbd, capacity);
+ disable_charging = 1;
+ chg_drv->lowerdb_reached = false;
+ } else if (!chg_drv->lowerdb_reached && lowerbd < capacity) {
+ pr_info("%s: lowerbd=%d, upperbd=%d, capacity=%d, charging off\n",
+ __func__, lowerbd, upperbd, capacity);
+ disable_charging = 1;
+ } else if (!chg_drv->lowerdb_reached && capacity <= lowerbd) {
+ pr_info("%s: lowerbd=%d, upperbd=%d, capacity=%d, lowerdb_reached=0->1, charging on\n",
+ __func__, lowerbd, upperbd, capacity);
+ chg_drv->lowerdb_reached = true;
+ } else {
+ pr_info("%s: lowerbd=%d, upperbd=%d, capacity=%d, charging on\n",
+ __func__, lowerbd, upperbd, capacity);
+ }
+ }
+
+ return disable_charging;
+}
+
+
+/* 1. charge profile idx based on the battery temperature */
+static int msc_temp_idx(struct chg_profile *profile, int temp)
+{
+ int temp_idx = 0;
+
+ while (temp_idx < profile->temp_nb_limits - 1 &&
+ temp >= profile->temp_limits[temp_idx + 1])
+ temp_idx++;
+
+ return temp_idx;
+}
+
+/* 2. compute the step index given the battery voltage
+ * When selecting an index need to make sure that headroom for the tier voltage
+ * will allow to send to the battery _at least_ next tier max FCC current and
+ * well over charge termination current.
+ */
+static int msc_voltage_idx(struct chg_profile *profile, int vbatt)
+{
+ int vbatt_idx = 0;
+
+ while (vbatt_idx < profile->volt_nb_limits - 1 &&
+ vbatt > profile->volt_limits[vbatt_idx])
+ vbatt_idx++;
+
+ /* assumes that 3 times the hardware resolution is ok */
+ if (vbatt_idx != profile->volt_nb_limits - 1) {
+ const int vt = profile->volt_limits[vbatt_idx];
+ const int headr = profile->cv_hw_resolution * 3;
+
+ if ((vt - vbatt) < headr)
+ vbatt_idx += 1;
+ }
+
+ return vbatt_idx;
+}
+
+/* Cap to fv_uv_margin_pct of VTIER if needed */
+static int msc_round_fv_uv(struct chg_profile *profile, int vtier, int fv_uv)
+{
+ int result;
+ const unsigned int fv_uv_max = (vtier / 1000)
+ * profile->fv_uv_margin_dpct;
+
+ if (fv_uv_max != 0 && fv_uv > fv_uv_max)
+ fv_uv = fv_uv_max;
+
+ result = fv_uv - (fv_uv % profile->cv_hw_resolution);
+
+ if (fv_uv_max != 0)
+ pr_info("MSC_ROUND: fv_uv=%d vtier=%d fv_uv_max=%d -> %d\n",
+ fv_uv, vtier, fv_uv_max, result);
+
+ return result;
+}
+
+#define CHG_WORK_ERROR_RETRY_MS 1000
+static void chg_work(struct work_struct *work)
+{
+ struct chg_drv *chg_drv =
+ container_of(work, struct chg_drv, chg_work.work);
+ struct chg_profile *profile = &chg_drv->chg_profile;
+ union power_supply_propval val;
+ struct power_supply *chg_psy = chg_drv->chg_psy;
+ struct power_supply *usb_psy = chg_drv->usb_psy;
+ struct power_supply *wlc_psy = chg_drv->wlc_psy;
+ struct power_supply *bat_psy = chg_drv->bat_psy;
+ int temp, ibatt, vbatt, vchrg, soc, chg_type;
+ int usb_present, wlc_online = 0;
+ int vbatt_idx = chg_drv->vbatt_idx, fv_uv = chg_drv->fv_uv, temp_idx;
+ int update_interval = chg_drv->chg_profile.update_interval;
+ int batt_status;
+ bool rerun_work = false;
+ int disable_charging;
+ int disable_pwrsrc;
+
+ __pm_stay_awake(&chg_drv->chg_ws);
+ pr_debug("battery charging work item\n");
+
+ usb_present = PSY_GET_PROP(usb_psy, POWER_SUPPLY_PROP_PRESENT);
+ if (wlc_psy)
+ wlc_online = PSY_GET_PROP(wlc_psy, POWER_SUPPLY_PROP_ONLINE);
+
+ /* If no power source, disable charging and exit */
+ if (!usb_present && !wlc_online) {
+ pr_info("no power source detected, disabling charging\n");
+ reset_chg_drv_state(chg_drv);
+ goto exit_chg_work;
+ }
+
+ /* debug option */
+ if (chg_drv->chg_mode)
+ bat_psy = chg_psy;
+
+ temp = PSY_GET_PROP(bat_psy, POWER_SUPPLY_PROP_TEMP);
+ ibatt = PSY_GET_PROP(bat_psy, POWER_SUPPLY_PROP_CURRENT_NOW);
+ vbatt = PSY_GET_PROP(bat_psy, POWER_SUPPLY_PROP_VOLTAGE_NOW);
+ soc = PSY_GET_PROP(bat_psy, POWER_SUPPLY_PROP_CAPACITY);
+ vchrg = PSY_GET_PROP(chg_psy, POWER_SUPPLY_PROP_VOLTAGE_NOW);
+ chg_type = PSY_GET_PROP(chg_psy, POWER_SUPPLY_PROP_CHARGE_TYPE);
+
+ if (temp == -EINVAL || ibatt == -EINVAL || vbatt == -EINVAL ||
+ usb_present == -EINVAL || wlc_online == -EINVAL)
+ goto error_rerun;
+
+ pr_info_states(chg_psy, usb_psy, wlc_psy,
+ temp, ibatt, vbatt, vchrg,
+ chg_type, chg_drv->fv_uv,
+ soc, usb_present, wlc_online);
+
+ if (temp < profile->temp_limits[0] ||
+ temp > profile->temp_limits[profile->temp_nb_limits - 1]) {
+ if (!chg_drv->stop_charging) {
+ pr_info("batt. temp. off limits, disabling charging\n");
+ reset_chg_drv_state(chg_drv);
+ }
+ /* status will be discharging when disabled but we want to keep
+ * monitoring temperature to re-enable charging
+ */
+ rerun_work = true;
+ goto handle_rerun;
+ } else if (chg_drv->stop_charging) {
+ pr_info("batt. temp. ok, enabling charging\n");
+ PSY_SET_PROP(chg_psy, POWER_SUPPLY_PROP_CHARGE_DISABLE, 0);
+ chg_drv->stop_charging = false;
+ }
+
+ disable_charging = is_charging_disabled(chg_drv, soc);
+ if (disable_charging && soc > chg_drv->charge_stop_level)
+ disable_pwrsrc = 1;
+ else
+ disable_pwrsrc = 0;
+
+ if (disable_charging != chg_drv->disable_charging) {
+ pr_info("set disable_charging(%d)", disable_charging);
+ PSY_SET_PROP(chg_psy, POWER_SUPPLY_PROP_CHARGE_DISABLE,
+ disable_charging);
+ }
+ chg_drv->disable_charging = disable_charging;
+
+ if (disable_pwrsrc != chg_drv->disable_pwrsrc) {
+ pr_info("set disable_pwrsrc(%d)", disable_pwrsrc);
+ PSY_SET_PROP(chg_psy, POWER_SUPPLY_PROP_INPUT_SUSPEND,
+ disable_pwrsrc);
+ }
+ chg_drv->disable_pwrsrc = disable_pwrsrc;
+
+ /* no need to reschedule, battery psy event will reschedule work item */
+ if (chg_drv->disable_charging || chg_drv->disable_pwrsrc) {
+ rerun_work = false;
+ goto exit_chg_work;
+ }
+
+ /* Multi Step Chargings with compensation of IRDROP
+ * vbatt_idx = chg_drv->vbatt_idx, fv_uv = chg_drv->fv_uv
+ */
+ temp_idx = msc_temp_idx(profile, temp);
+ if (temp_idx != chg_drv->temp_idx || chg_drv->fv_uv == -1
+ || chg_drv->vbatt_idx == -1) {
+ /* seed voltage only when really needed */
+ if (chg_drv->vbatt_idx == -1)
+ vbatt_idx = msc_voltage_idx(profile, vbatt);
+
+ pr_info("MSC_SEED temp=%d vbatt=%d temp_idx:%d->%d, vbatt_idx:%d->%d\n",
+ temp, vbatt, chg_drv->temp_idx, temp_idx,
+ chg_drv->vbatt_idx, vbatt_idx);
+
+ /* Debounce tier switch only when not already switching */
+ if (chg_drv->checked_tier_switch_cnt == 0)
+ chg_drv->checked_cv_cnt = profile->cv_debounce_cnt;
+ } else if (ibatt > 0) {
+ /* Track battery voltage if discharging is due to system load,
+ * low ILIM or lack of headroom; stop charging work and reset
+ * chg_drv state() when discharging is due to disconnect.
+ * NOTE: POWER_SUPPLY_PROP_STATUS return *_DISCHARGING only on
+ * disconnect.
+ * NOTE: same vbat_idx will not change fv_uv
+ */
+ vbatt_idx = msc_voltage_idx(profile, vbatt);
+ update_interval = profile->cv_update_interval;
+
+ pr_info("MSC_DSG vbatt_idx:%d->%d vbatt=%d ibatt=%d fv_uv=%d cv_cnt=%d ov_cnt=%d\n",
+ chg_drv->vbatt_idx, vbatt_idx,
+ vbatt, ibatt, fv_uv,
+ chg_drv->checked_cv_cnt,
+ chg_drv->checked_ov_cnt);
+ } else if (chg_drv->vbatt_idx == profile->volt_nb_limits - 1) {
+ /* will not adjust charger voltage only in the configured
+ * last tier.
+ * NOTE: might not be the "real" last tier since can I have
+ * tiers with max charge current == 0.
+ * NOTE: should I use a voltage limit instead?
+ */
+ pr_info("MSC_LAST vbatt=%d ibatt=%d fv_uv=%d\n",
+ vbatt, ibatt, fv_uv);
+ } else {
+ const int vtier = profile->volt_limits[vbatt_idx];
+ const int utv_margin = profile->cv_range_accuracy;
+ const int otv_margin = profile->cv_otv_margin;
+ const int switch_cnt = profile->cv_tier_switch_cnt;
+ const int cc_next_max = CCCM_LIMITS(profile, temp_idx,
+ vbatt_idx + 1);
+
+ if ((vbatt - vtier) > otv_margin) {
+ /* OVER: vbatt over vtier for more than margin (usually 0) */
+ const int cc_max =
+ CCCM_LIMITS(profile, temp_idx, vbatt_idx);
+
+ /* pullback when over tier voltage, fast poll, penalty
+ * on TAPER_RAISE and no cv debounce (so will consider
+ * switching voltage tiers if the current is right).
+ * NOTE: lowering voltage might cause a small drop in
+ * current (we should remain under next tier)
+ */
+ fv_uv = msc_round_fv_uv(profile, vtier,
+ fv_uv - profile->cv_hw_resolution);
+ if (fv_uv < vtier)
+ fv_uv = vtier;
+
+ update_interval = profile->cv_update_interval;
+ chg_drv->checked_ov_cnt = profile->cv_tier_ov_cnt;
+ chg_drv->checked_cv_cnt = 0;
+
+ if (chg_drv->checked_tier_switch_cnt > 0) {
+ /* no pullback, next tier if already counting */
+ vbatt_idx = chg_drv->vbatt_idx + 1;
+
+ pr_info("MSC_VSWITCH vt=%d vb=%d ibatt=%d\n",
+ vtier, vbatt, ibatt);
+ } else if (-ibatt == cc_max) {
+ /* pullback, double penalty if at full current */
+ chg_drv->checked_ov_cnt *= 2;
+
+ pr_info("MSC_VOVER vt=%d vb=%d ibatt=%d fv_uv=%d->%d\n",
+ vtier, vbatt, ibatt,
+ chg_drv->fv_uv, fv_uv);
+ } else {
+ pr_info("MSC_PULLBACK vt=%d vb=%d ibatt=%d fv_uv=%d->%d\n",
+ vtier, vbatt, ibatt,
+ chg_drv->fv_uv, fv_uv);
+ }
+
+ /* NOTE: might get here after windup because algo will
+ * track the voltage drop caused from load as IRDROP.
+ * TODO: make sure that being current limited clear
+ * the taper condition.
+ */
+
+ } else if (chg_type == POWER_SUPPLY_CHARGE_TYPE_FAST) {
+ /* FAST: usual compensation (vchrg is vqcom)
+ * NOTE: there is a race in reading from charger and data here
+ * might not be consistent (b/110318684)
+ * NOTE: could add PID loop for management of thermals
+ */
+ if (vchrg > vbatt) {
+ fv_uv = msc_round_fv_uv(profile, vtier,
+ vtier + (vchrg - vbatt));
+ } else {
+ /* could keep it steady instead */
+ fv_uv = vtier;
+ }
+
+ /* no tier switch during fast charge */
+ if (chg_drv->checked_cv_cnt == 0)
+ chg_drv->checked_cv_cnt = 1;
+
+ pr_info("MSC_FAST vt=%d vb=%d fv_uv=%d->%d vchrg=%d cv_cnt=%d \n",
+ vtier, vbatt, chg_drv->fv_uv, fv_uv,
+ vchrg, chg_drv->checked_cv_cnt);
+
+ } else if (chg_type != POWER_SUPPLY_CHARGE_TYPE_TAPER) {
+ /* Not fast or taper: set checked_cv_cnt=0 to make sure we test
+ * for current and avoid early termination in case of lack of
+ * headroom (Vfloat ~= Vbatt)
+ * NOTE: this can cause early switch on low ilim
+ */
+ update_interval = profile->cv_update_interval;
+ chg_drv->checked_cv_cnt = 0;
+
+ pr_info("MSC_TYPE vt=%d vb=%d fv_uv=%d chg_type=%d\n",
+ vtier, vbatt, fv_uv, chg_type);
+
+ } else if (chg_drv->checked_cv_cnt + chg_drv->checked_ov_cnt) {
+ /* TAPER_COUNTDOWN: countdown to raise fv_uv and/or check
+ * for tier switch, will keep steady...
+ */
+ pr_info("MSC_DLY vt=%d vb=%d fv_uv=%d margin=%d cv_cnt=%d, ov_cnt=%d\n",
+ vtier, vbatt, fv_uv, profile->cv_range_accuracy,
+ chg_drv->checked_cv_cnt,
+ chg_drv->checked_ov_cnt);
+
+ update_interval = profile->cv_update_interval;
+ if (chg_drv->checked_cv_cnt)
+ chg_drv->checked_cv_cnt -= 1;
+ if (chg_drv->checked_ov_cnt)
+ chg_drv->checked_ov_cnt -= 1;
+
+ } else if ((vtier - vbatt) < utv_margin) {
+ /* TAPER_STEADY: close enough to tier, don't need to adjust */
+ update_interval = profile->cv_update_interval;
+
+ pr_info("MSC_STEADY vt=%d vb=%d fv_uv=%d margin=%d\n",
+ vtier, vbatt, fv_uv,
+ profile->cv_range_accuracy);
+ } else {
+ /* TAPER_RAISE: under tier vlim, raise one click & debounce
+ * taper (see above handling of "close enough")
+ */
+ fv_uv = msc_round_fv_uv(profile, vtier,
+ fv_uv + profile->cv_hw_resolution);
+ update_interval = profile->cv_update_interval;
+
+ /* debounce next taper voltage adjustment */
+ chg_drv->checked_cv_cnt = profile->cv_debounce_cnt;
+
+ pr_info("MSC_RAISE vt=%d vb=%d fv_uv=%d->%d\n",
+ vtier, vbatt, chg_drv->fv_uv, fv_uv);
+ }
+
+ if (chg_drv->checked_cv_cnt > 0) {
+ /* debounce period on tier switch */
+ pr_info("MSC_WAIT vt=%d vb=%d fv_uv=%d ibatt=%d cv_cnt=%d ov_cnt=%d\n",
+ vtier, vbatt, fv_uv, ibatt,
+ chg_drv->checked_cv_cnt,
+ chg_drv->checked_ov_cnt);
+ } else if (-ibatt > cc_next_max) {
+ /* current over next tier, reset tier switch count */
+ chg_drv->checked_tier_switch_cnt = 0;
+
+ pr_info("MSC_RSTC vt=%d vb=%d fv_uv=%d ibatt=%d cc_next_max=%d t_cnt=%d\n",
+ vtier, vbatt, fv_uv, ibatt, cc_next_max,
+ chg_drv->checked_tier_switch_cnt);
+ } else if (chg_drv->checked_tier_switch_cnt >= switch_cnt) {
+ /* next tier, fv_uv detemined at MSC_SET */
+ vbatt_idx = chg_drv->vbatt_idx + 1;
+
+ pr_info("MSC_NEXT tier vb=%d ibatt=%d vbatt_idx=%d->%d\n",
+ vbatt, ibatt, chg_drv->vbatt_idx, vbatt_idx);
+ } else {
+ /* current under next tier, increase tier switch count */
+ chg_drv->checked_tier_switch_cnt++;
+
+ pr_info("MSC_NYET ibatt=%d cc_next_max=%d t_cnt=%d\n",
+ ibatt, cc_next_max,
+ chg_drv->checked_tier_switch_cnt);
+ }
+
+ }
+
+ /* update fv or cc will change in last tier... */
+ if ((vbatt_idx != chg_drv->vbatt_idx) || (temp_idx != chg_drv->temp_idx)
+ || (fv_uv != chg_drv->fv_uv)) {
+ const int cc_max = CCCM_LIMITS(profile, temp_idx, vbatt_idx);
+ int rc;
+
+ /* need a new fv_uv only on a new voltage tier */
+ if (vbatt_idx != chg_drv->vbatt_idx) {
+ fv_uv = profile->volt_limits[vbatt_idx];
+ chg_drv->checked_tier_switch_cnt = 0;
+ chg_drv->checked_ov_cnt = 0;
+ }
+
+ pr_info("MSC_SET cv_cnt=%d ov_cnt=%d temp_idx:%d->%d, vbatt_idx:%d->%d, fv=%d->%d, cc_max=%d\n",
+ chg_drv->checked_cv_cnt, chg_drv->checked_ov_cnt,
+ chg_drv->temp_idx, temp_idx, chg_drv->vbatt_idx,
+ vbatt_idx, chg_drv->fv_uv, fv_uv, cc_max);
+
+ /* taper control on last tier with nonzero charge current */
+ if (vbatt_idx == (profile->volt_nb_limits - 1) ||
+ CCCM_LIMITS(profile, temp_idx, vbatt_idx + 1) == 0) {
+ PSY_SET_PROP(chg_drv->chg_psy,
+ POWER_SUPPLY_PROP_TAPER_CONTROL,
+ POWER_SUPPLY_TAPER_CONTROL_MODE_IMMEDIATE);
+ }
+
+ rc = PSY_SET_PROP(chg_psy,
+ POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT_MAX, cc_max);
+ if (rc == 0)
+ rc = PSY_SET_PROP(chg_psy,
+ POWER_SUPPLY_PROP_VOLTAGE_MAX, fv_uv);
+
+ if (rc != 0) {
+ pr_err("MSC_SET: error rc=%d\n", rc);
+ goto error_rerun;
+ }
+
+ chg_drv->vbatt_idx = vbatt_idx;
+ chg_drv->temp_idx = temp_idx;
+ chg_drv->fv_uv = fv_uv;
+ }
+
+ /* DISCHARGING only when not connected */
+ batt_status = PSY_GET_PROP(chg_psy, POWER_SUPPLY_PROP_STATUS);
+
+ switch (batt_status) {
+ case POWER_SUPPLY_STATUS_DISCHARGING:
+ rerun_work = false;
+ break;
+ case POWER_SUPPLY_STATUS_CHARGING:
+ case POWER_SUPPLY_STATUS_FULL:
+ case POWER_SUPPLY_STATUS_NOT_CHARGING:
+ rerun_work = true;
+ break;
+ case POWER_SUPPLY_STATUS_UNKNOWN:
+ pr_err("chg_work charging status UNKNOWN\n");
+ goto error_rerun;
+ default:
+ pr_err("chg_work invalid charging status %d\n", val.intval);
+ goto error_rerun;
+ }
+
+handle_rerun:
+ if (rerun_work) {
+ pr_debug("rerun battery charging work in %d ms\n",
+ update_interval);
+ schedule_delayed_work(&chg_drv->chg_work,
+ msecs_to_jiffies(update_interval));
+ } else {
+ pr_info("stop battery charging work: batt_status=%d\n",
+ batt_status);
+ reset_chg_drv_state(chg_drv);
+ }
+ goto exit_chg_work;
+
+error_rerun:
+ pr_err("error occurred, rerun battery charging work in %d ms\n",
+ CHG_WORK_ERROR_RETRY_MS);
+ schedule_delayed_work(&chg_drv->chg_work,
+ msecs_to_jiffies(CHG_WORK_ERROR_RETRY_MS));
+
+exit_chg_work:
+ __pm_relax(&chg_drv->chg_ws);
+}
+
+static void dump_profile(struct chg_profile *profile)
+{
+ char buff[256];
+ int ti, vi, count, len = sizeof(buff);
+
+ pr_info("Profile constant charge limits:\n");
+ count = 0;
+ for (vi = 0; vi < profile->volt_nb_limits; vi++) {
+ count += scnprintf(buff + count, len - count, " %4d",
+ profile->volt_limits[vi] / 1000);
+ }
+ pr_info("|T \\ V%s\n", buff);
+
+ for (ti = 0; ti < profile->temp_nb_limits - 1; ti++) {
+ count = 0;
+ count += scnprintf(buff + count, len - count, "|%2d:%2d",
+ profile->temp_limits[ti] / 10,
+ profile->temp_limits[ti + 1] / 10);
+ for (vi = 0; vi < profile->volt_nb_limits; vi++) {
+ count += scnprintf(buff + count, len - count, " %4d",
+ CCCM_LIMITS(profile, ti, vi) / 1000);
+ }
+ pr_info("%s\n", buff);
+ }
+}
+
+static int chg_init_chg_profile(struct chg_drv *chg_drv)
+{
+ struct device *dev = chg_drv->device;
+ struct device_node *node = dev->of_node;
+ struct chg_profile *profile = &chg_drv->chg_profile;
+ u32 cccm_array_size, ccm;
+ int ret = 0, vi, ti;
+
+ ret = of_property_read_u32(node, "google,chg-update-interval",
+ &profile->update_interval);
+ if (ret < 0) {
+ pr_err("cannot read chg-update-interval, ret=%d\n", ret);
+ return ret;
+ }
+
+ ret = of_property_read_u32(node, "google,chg-battery-capacity",
+ &profile->battery_capacity);
+ if (ret < 0) {
+ pr_err("cannot read chg-battery-capacity, ret=%d\n", ret);
+ return ret;
+ }
+
+ ret = of_property_read_u32(node, "google,cv-hw-resolution",
+ &profile->cv_hw_resolution);
+ if (ret < 0)
+ profile->cv_hw_resolution = 25000;
+
+ ret = of_property_read_u32(node, "google,cc-hw-resolution",
+ &profile->cc_hw_resolution);
+ if (ret < 0)
+ profile->cc_hw_resolution = 25000;
+
+ /* IEEE1725, default to 0, 1030 for 3% of VTIER */
+ ret = of_property_read_u32(node, "google,fv-uv-margin-dpct",
+ &profile->fv_uv_margin_dpct);
+ if (ret < 0)
+ profile->fv_uv_margin_dpct = 0;
+
+ ret = of_property_read_u32(node, "google,cv-range-accuracy",
+ &profile->cv_range_accuracy);
+ if (ret < 0)
+ profile->cv_range_accuracy = profile->cv_hw_resolution / 2;
+
+ /* allow being "a little" over tier voltage, experimental */
+ ret = of_property_read_u32(node, "google,cv-otv-margin",
+ &profile->cv_otv_margin);
+ if (ret < 0)
+ profile->cv_otv_margin = 0;
+
+ ret = of_property_read_u32(node, "google,cv-debounce-cnt",
+ &profile->cv_debounce_cnt);
+ if (ret < 0)
+ profile->cv_debounce_cnt = 3;
+
+ ret = of_property_read_u32(node, "google,cv-update-interval",
+ &profile->cv_update_interval);
+ if (ret < 0)
+ profile->cv_update_interval = 2000;
+
+ ret = of_property_read_u32(node, "google,cv-tier-ov-cnt",
+ &profile->cv_tier_ov_cnt);
+ if (ret < 0)
+ profile->cv_tier_ov_cnt = 10;
+
+ ret = of_property_read_u32(node, "google,cv-tier-switch-cnt",
+ &profile->cv_tier_switch_cnt);
+ if (ret < 0)
+ profile->cv_tier_switch_cnt = 3;
+
+ profile->temp_nb_limits =
+ of_property_count_elems_of_size(node, "google,chg-temp-limits",
+ sizeof(u32));
+ if (profile->temp_nb_limits <= 0) {
+ ret = profile->temp_nb_limits;
+ pr_err("cannot read chg-temp-limits, ret=%d\n", ret);
+ return ret;
+ }
+ if (profile->temp_nb_limits > CHG_TEMP_NB_LIMITS_MAX) {
+ pr_err("chg-temp-nb-limits exceeds driver max: %d\n",
+ CHG_TEMP_NB_LIMITS_MAX);
+ return -EINVAL;
+ }
+ ret = of_property_read_u32_array(node, "google,chg-temp-limits",
+ profile->temp_limits,
+ profile->temp_nb_limits);
+ if (ret < 0) {
+ pr_err("cannot read chg-temp-limits table, ret=%d\n", ret);
+ return ret;
+ }
+
+ profile->volt_nb_limits =
+ of_property_count_elems_of_size(node, "google,chg-cv-limits",
+ sizeof(u32));
+ if (profile->volt_nb_limits <= 0) {
+ ret = profile->volt_nb_limits;
+ pr_err("cannot read chg-cv-limits, ret=%d\n", ret);
+ return ret;
+ }
+ if (profile->volt_nb_limits > CHG_VOLT_NB_LIMITS_MAX) {
+ pr_err("chg-cv-nb-limits exceeds driver max: %d\n",
+ CHG_VOLT_NB_LIMITS_MAX);
+ return -EINVAL;
+ }
+ ret = of_property_read_u32_array(node, "google,chg-cv-limits",
+ profile->volt_limits,
+ profile->volt_nb_limits);
+ if (ret < 0) {
+ pr_err("cannot read chg-cv-limits table, ret=%d\n", ret);
+ return ret;
+ }
+ for (vi = 0; vi < profile->volt_nb_limits; vi++)
+ profile->volt_limits[vi] = profile->volt_limits[vi] /
+ profile->cv_hw_resolution * profile->cv_hw_resolution;
+
+ cccm_array_size =
+ (profile->temp_nb_limits - 1) * profile->volt_nb_limits;
+ profile->cccm_limits = devm_kzalloc(dev,
+ sizeof(s32) * cccm_array_size,
+ GFP_KERNEL);
+
+ ret = of_property_read_u32_array(node, "google,chg-cc-limits",
+ profile->cccm_limits, cccm_array_size);
+ if (ret < 0) {
+ pr_err("cannot read chg-cc-limits table, ret=%d\n", ret);
+ return ret;
+ }
+
+ ret = of_property_read_u32(node, "google,chg-cc-tolerance",
+ &profile->chg_cc_tolerance);
+ if (ret < 0)
+ profile->chg_cc_tolerance = 0;
+ else if (profile->chg_cc_tolerance > 250)
+ profile->chg_cc_tolerance = 250;
+
+ /* chg-battery-capacity is in mAh, chg-cc-limits relative to 100 */
+ for (ti = 0; ti < profile->temp_nb_limits - 1; ti++) {
+ for (vi = 0; vi < profile->volt_nb_limits; vi++) {
+ ccm = CCCM_LIMITS(profile, ti, vi);
+ ccm *= profile->battery_capacity * 10;
+ ccm = ccm * (1000 - profile->chg_cc_tolerance) / 1000;
+ // round to the nearest resolution the PMIC can handle
+ ccm = DIV_ROUND_CLOSEST(ccm, profile->cc_hw_resolution)
+ * profile->cc_hw_resolution;
+ CCCM_LIMITS(profile, ti, vi) = ccm;
+ }
+ }
+
+ pr_info("successfully read charging profile:\n");
+ dump_profile(profile);
+
+ return 0;
+}
+
+static ssize_t show_tier_ovc(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct chg_drv *chg_drv = dev_get_drvdata(dev);
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n",
+ chg_drv->chg_profile.cv_tier_ov_cnt);
+}
+
+static ssize_t set_tier_ovc(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct chg_drv *chg_drv = dev_get_drvdata(dev);
+ int ret = 0, val;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (ret < 0)
+ return ret;
+
+ if (val < 0)
+ return count;
+
+ chg_drv->chg_profile.cv_tier_ov_cnt = val;
+ return count;
+}
+
+static DEVICE_ATTR(tier_ovc, 0660, show_tier_ovc, set_tier_ovc);
+
+static ssize_t show_cv_update_interval(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct chg_drv *chg_drv = dev_get_drvdata(dev);
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n",
+ chg_drv->chg_profile.cv_update_interval);
+}
+
+static ssize_t set_cv_update_interval(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct chg_drv *chg_drv = dev_get_drvdata(dev);
+ int ret = 0, val;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (ret < 0)
+ return ret;
+
+ if (val < 1500)
+ return count;
+
+ chg_drv->chg_profile.cv_update_interval = val;
+ return count;
+}
+
+static DEVICE_ATTR(cv_update_interval, 0660,
+ show_cv_update_interval,
+ set_cv_update_interval);
+
+static ssize_t show_cv_range_accuracy(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct chg_drv *chg_drv = dev_get_drvdata(dev);
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n",
+ chg_drv->chg_profile.cv_range_accuracy);
+}
+
+static ssize_t set_cv_range_accuracy(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct chg_drv *chg_drv = dev_get_drvdata(dev);
+ int ret = 0, val;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (ret < 0)
+ return ret;
+
+ chg_drv->chg_profile.cv_range_accuracy = val;
+ return count;
+}
+
+static DEVICE_ATTR(cv_range_accuracy, 0660,
+ show_cv_range_accuracy,
+ set_cv_range_accuracy);
+
+static ssize_t show_charge_stop_level(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct chg_drv *chg_drv = dev_get_drvdata(dev);
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n", chg_drv->charge_stop_level);
+}
+
+static ssize_t set_charge_stop_level(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct chg_drv *chg_drv = dev_get_drvdata(dev);
+ int ret = 0, val;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (ret < 0)
+ return ret;
+
+ if (!chg_drv->bat_psy) {
+ pr_err("chg_drv->bat_psy is not ready");
+ return -ENODATA;
+ }
+
+ if ((val == chg_drv->charge_stop_level) ||
+ (val <= chg_drv->charge_start_level) ||
+ (val > DEFAULT_CHARGE_STOP_LEVEL))
+ return count;
+
+ chg_drv->charge_stop_level = val;
+ if (chg_drv->bat_psy)
+ power_supply_changed(chg_drv->bat_psy);
+ return count;
+}
+
+static DEVICE_ATTR(charge_stop_level, 0660,
+ show_charge_stop_level, set_charge_stop_level);
+
+static ssize_t
+show_charge_start_level(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct chg_drv *chg_drv = dev_get_drvdata(dev);
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n", chg_drv->charge_start_level);
+}
+
+static ssize_t set_charge_start_level(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct chg_drv *chg_drv = dev_get_drvdata(dev);
+ int ret = 0, val;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (ret < 0)
+ return ret;
+
+ if (!chg_drv->bat_psy) {
+ pr_err("chg_drv->bat_psy is not ready");
+ return -ENODATA;
+ }
+
+ if ((val == chg_drv->charge_start_level) ||
+ (val >= chg_drv->charge_stop_level) ||
+ (val < DEFAULT_CHARGE_START_LEVEL))
+ return count;
+
+ chg_drv->charge_start_level = val;
+ if (chg_drv->bat_psy)
+ power_supply_changed(chg_drv->bat_psy);
+ return count;
+}
+
+static DEVICE_ATTR(charge_start_level, 0660,
+ show_charge_start_level, set_charge_start_level);
+
+#ifdef CONFIG_DEBUG_FS
+
+/* use qcom VS maxim fg and more... */
+static int get_chg_mode(void *data, u64 *val)
+{
+ struct chg_drv *chg_drv = (struct chg_drv *)data;
+
+ *val = chg_drv->chg_mode;
+ return 0;
+}
+
+static int set_chg_mode(void *data, u64 val)
+{
+ struct chg_drv *chg_drv = (struct chg_drv *)data;
+
+ chg_drv->chg_mode = val;
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(chg_mode_fops, get_chg_mode, set_chg_mode, "%llu\n");
+
+/* allow vbatt to be over vtier by margin */
+static int get_cv_otv_margin(void *data, u64 *val)
+{
+ struct chg_drv *chg_drv = (struct chg_drv *)data;
+
+ *val = chg_drv->chg_profile.cv_otv_margin;
+ return 0;
+}
+
+static int set_cv_otv_margin(void *data, u64 val)
+{
+ struct chg_drv *chg_drv = (struct chg_drv *)data;
+
+ chg_drv->chg_profile.cv_otv_margin = val;
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(cv_otv_margin_fops, get_cv_otv_margin,
+ set_cv_otv_margin, "%llu\n");
+
+/* cap fv_uv to a percentage of vtier (1000 -> vtier, 1020 -> 2% over vtier) */
+static int get_fv_uv_margin(void *data, u64 *val)
+{
+ struct chg_drv *chg_drv = (struct chg_drv *)data;
+
+ *val = chg_drv->chg_profile.fv_uv_margin_dpct;
+ return 0;
+}
+
+static int set_fv_uv_margin(void *data, u64 val)
+{
+ struct chg_drv *chg_drv = (struct chg_drv *)data;
+
+ chg_drv->chg_profile.fv_uv_margin_dpct = val;
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(fv_uv_margin_fops, get_fv_uv_margin,
+ set_fv_uv_margin, "%llu\n");
+#endif
+
+static int init_debugfs(struct chg_drv *chg_drv)
+{
+#ifdef CONFIG_DEBUG_FS
+ struct dentry *de;
+
+ de = debugfs_create_dir("google_charger", 0);
+ if (de) {
+ debugfs_create_file("chg_mode", 0644, de,
+ chg_drv, &chg_mode_fops);
+ debugfs_create_file("cv_otv_margin", 0644, de,
+ chg_drv, &cv_otv_margin_fops);
+ debugfs_create_file("fv_uv_margin", 0644, de,
+ chg_drv, &fv_uv_margin_fops);
+ }
+#endif
+ return 0;
+
+}
+
+
+static void google_charger_init_work(struct work_struct *work)
+{
+ struct chg_drv *chg_drv = container_of(work, struct chg_drv,
+ init_work.work);
+ struct power_supply *chg_psy, *usb_psy, *wlc_psy = NULL, *bat_psy;
+ int ret = 0;
+
+ chg_psy = power_supply_get_by_name(chg_drv->chg_psy_name);
+ if (!chg_psy) {
+ pr_info("failed to get \"%s\" power supply\n",
+ chg_drv->chg_psy_name);
+ goto retry_init_work;
+ }
+
+ bat_psy = power_supply_get_by_name(chg_drv->bat_psy_name);
+ if (!bat_psy) {
+ pr_info("failed to get \"%s\" power supply\n",
+ chg_drv->bat_psy_name);
+ power_supply_put(chg_psy);
+ goto retry_init_work;
+ }
+
+ usb_psy = power_supply_get_by_name("usb");
+ if (!usb_psy) {
+ pr_info("failed to get \"usb\" power supply\n");
+ power_supply_put(chg_psy);
+ power_supply_put(bat_psy);
+ goto retry_init_work;
+ }
+
+ if (chg_drv->wlc_psy_name) {
+ wlc_psy = power_supply_get_by_name(chg_drv->wlc_psy_name);
+ if (!wlc_psy) {
+ pr_info("failed to get \"%s\" power supply\n",
+ chg_drv->wlc_psy_name);
+ power_supply_put(chg_psy);
+ power_supply_put(bat_psy);
+ power_supply_put(usb_psy);
+ goto retry_init_work;
+ }
+ }
+
+ chg_drv->chg_psy = chg_psy;
+ chg_drv->wlc_psy = wlc_psy;
+ chg_drv->usb_psy = usb_psy;
+ chg_drv->bat_psy = bat_psy;
+
+ chg_drv->charge_stop_level = DEFAULT_CHARGE_STOP_LEVEL;
+ chg_drv->charge_start_level = DEFAULT_CHARGE_START_LEVEL;
+
+ reset_chg_drv_state(chg_drv);
+
+ chg_drv->psy_nb.notifier_call = psy_changed;
+ ret = power_supply_reg_notifier(&chg_drv->psy_nb);
+ if (ret < 0)
+ pr_err("Cannot register power supply notifer, ret=%d\n", ret);
+
+ wakeup_source_init(&chg_drv->chg_ws, "google-charger");
+ pr_info("google_charger_init_work done\n");
+
+ /* catch state changes that happened before registering the notifier */
+ schedule_delayed_work(&chg_drv->chg_work,
+ msecs_to_jiffies(CHG_DELAY_INIT_DETECT_MS));
+ return;
+
+retry_init_work:
+ schedule_delayed_work(&chg_drv->init_work,
+ msecs_to_jiffies(CHG_DELAY_INIT_MS));
+}
+
+static int google_charger_probe(struct platform_device *pdev)
+{
+ const char *chg_psy_name, *bat_psy_name, *wlc_psy_name = NULL;
+ struct chg_drv *chg_drv;
+ int ret;
+
+ chg_drv = devm_kzalloc(&pdev->dev, sizeof(*chg_drv), GFP_KERNEL);
+ if (!chg_drv)
+ return -ENOMEM;
+
+ chg_drv->device = &pdev->dev;
+
+ ret = of_property_read_string(pdev->dev.of_node,
+ "google,chg-power-supply",
+ &chg_psy_name);
+ if (ret != 0) {
+ pr_err("cannot read google,chg-power-supply, ret=%d\n", ret);
+ return -EINVAL;
+ }
+ chg_drv->chg_psy_name =
+ devm_kstrdup(&pdev->dev, chg_psy_name, GFP_KERNEL);
+ if (!chg_drv->chg_psy_name)
+ return -ENOMEM;
+
+ ret = of_property_read_string(pdev->dev.of_node,
+ "google,bat-power-supply",
+ &bat_psy_name);
+ if (ret != 0) {
+ pr_err("cannot read google,bat-power-supply, ret=%d\n", ret);
+ return -EINVAL;
+ }
+ chg_drv->bat_psy_name =
+ devm_kstrdup(&pdev->dev, bat_psy_name, GFP_KERNEL);
+ if (!chg_drv->bat_psy_name)
+ return -ENOMEM;
+
+ ret = of_property_read_string(pdev->dev.of_node,
+ "google,wlc-power-supply",
+ &wlc_psy_name);
+ if (ret != 0)
+ pr_warn("google,wlc-power-supply not defined\n");
+ if (wlc_psy_name) {
+ chg_drv->wlc_psy_name =
+ devm_kstrdup(&pdev->dev, wlc_psy_name, GFP_KERNEL);
+ if (!chg_drv->wlc_psy_name)
+ return -ENOMEM;
+ }
+
+ ret = chg_init_chg_profile(chg_drv);
+ if (ret < 0) {
+ pr_err("cannot read charging profile from dt, ret=%d\n", ret);
+ return ret;
+ }
+
+ ret = device_create_file(&pdev->dev, &dev_attr_charge_stop_level);
+ if (ret != 0) {
+ pr_err("Failed to create charge_stop_level files, ret=%d\n",
+ ret);
+ return ret;
+ }
+
+ ret = device_create_file(&pdev->dev, &dev_attr_charge_start_level);
+ if (ret != 0) {
+ pr_err("Failed to create charge_start_level files, ret=%d\n",
+ ret);
+ return ret;
+ }
+
+ // TODO: move to debugfs
+ ret = device_create_file(&pdev->dev, &dev_attr_cv_range_accuracy);
+ if (ret != 0) {
+ pr_err("Failed to create cv_range_accuracy files, ret=%d\n",
+ ret);
+ return ret;
+ }
+
+ // TODO: move to debugfs
+ ret = device_create_file(&pdev->dev, &dev_attr_cv_update_interval);
+ if (ret != 0) {
+ pr_err("Failed to create cv_update_interval files, ret=%d\n",
+ ret);
+ return ret;
+ }
+
+ // TODO: move to debugfs
+ ret = device_create_file(&pdev->dev, &dev_attr_tier_ovc);
+ if (ret != 0) {
+ pr_err("Failed to create tier_ovc files, ret=%d\n", ret);
+ return ret;
+ }
+
+ /* debug */
+ init_debugfs(chg_drv);
+
+ INIT_DELAYED_WORK(&chg_drv->init_work, google_charger_init_work);
+ INIT_DELAYED_WORK(&chg_drv->chg_work, chg_work);
+ platform_set_drvdata(pdev, chg_drv);
+
+ schedule_delayed_work(&chg_drv->init_work,
+ msecs_to_jiffies(CHG_DELAY_INIT_MS));
+
+ return 0;
+}
+
+static int google_charger_remove(struct platform_device *pdev)
+{
+ struct chg_drv *chg_drv = platform_get_drvdata(pdev);
+
+ if (chg_drv) {
+ if (chg_drv->chg_psy)
+ power_supply_put(chg_drv->chg_psy);
+ if (chg_drv->bat_psy)
+ power_supply_put(chg_drv->bat_psy);
+ if (chg_drv->usb_psy)
+ power_supply_put(chg_drv->usb_psy);
+ if (chg_drv->wlc_psy)
+ power_supply_put(chg_drv->wlc_psy);
+ wakeup_source_trash(&chg_drv->chg_ws);
+ }
+
+ return 0;
+}
+
+static const struct of_device_id match_table[] = {
+ {.compatible = "google,charger"},
+ {},
+};
+
+static struct platform_driver charger_driver = {
+ .driver = {
+ .name = "google,charger",
+ .owner = THIS_MODULE,
+ .of_match_table = match_table,
+ },
+ .probe = google_charger_probe,
+ .remove = google_charger_remove,
+};
+
+static int __init google_charger_init(void)
+{
+ int ret;
+
+ ret = platform_driver_register(&charger_driver);
+ if (ret < 0) {
+ pr_err("device registration failed: %d\n", ret);
+ return ret;
+ }
+ return 0;
+}
+
+static void __init google_charger_exit(void)
+{
+ platform_driver_unregister(&charger_driver);
+ pr_info("unregistered platform driver\n");
+}
+
+module_init(google_charger_init);
+module_exit(google_charger_exit);
+MODULE_DESCRIPTION("Multi-step battery charger driver");
+MODULE_AUTHOR("Thierry Strudel <tstrudel@google.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/power/supply/max1720x_battery.c b/drivers/power/supply/max1720x_battery.c
new file mode 100644
index 0000000..9954028
--- /dev/null
+++ b/drivers/power/supply/max1720x_battery.c
@@ -0,0 +1,2144 @@
+/*
+ * Fuel gauge driver for Maxim 17201/17205
+ *
+ * Copyright (C) 2018 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": %s " fmt, __func__
+
+#include <linux/err.h>
+#include <linux/i2c.h>
+#include <linux/iio/consumer.h>
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/pm_runtime.h>
+#include <linux/power_supply.h>
+#include <linux/regmap.h>
+#include <linux/slab.h>
+#include <linux/time.h>
+
+#include <linux/cdev.h>
+#include <linux/device.h>
+#include <linux/fs.h> /* register_chrdev, unregister_chrdev */
+#include <linux/module.h>
+#include <linux/seq_file.h> /* seq_read, seq_lseek, single_release */
+
+#ifdef CONFIG_DEBUG_FS
+#include <linux/debugfs.h>
+#endif
+
+#define MAX1720X_TRECALL_MS 5
+#define MAX1720X_TPOR_MS 150
+#define MAX1720X_TICLR_MS 500
+#define MAX1720X_I2C_DRIVER_NAME "max1720x_fg_irq"
+#define MAX1720X_N_OF_HISTORY_PAGES 203
+#define MAX1720X_DELAY_INIT_MS 1000
+#define FULLCAPNOM_STABILIZE_CYCLES 5
+
+#define HISTORY_DEVICENAME "maxfg_history"
+
+enum max1720x_register {
+ /* ModelGauge m5 Register */
+ MAX1720X_STATUS = 0x00,
+ MAX1720X_VALRTTH = 0x01,
+ MAX1720X_TALRTTH = 0x02,
+ MAX1720X_SALRTTH = 0x03,
+ MAX1720X_ATRATE = 0x04,
+ MAX1720X_REPCAP = 0x05,
+ MAX1720X_REPSOC = 0x06,
+ MAX1720X_AGE = 0x07,
+ MAX1720X_TEMP = 0x08,
+ MAX1720X_VCELL = 0x09,
+ MAX1720X_CURRENT = 0x0A,
+ MAX1720X_AVGCURRENT = 0x0B,
+ MAX1720X_QRESIDUAL = 0x0C,
+ MAX1720X_MIXSOC = 0x0D,
+ MAX1720X_AVSOC = 0x0E,
+ MAX1720X_MIXCAP = 0x0F,
+
+ MAX1720X_FULLCAP = 0x10,
+ MAX1720X_TTE = 0x11,
+ MAX1720X_QRTABLE00 = 0x12,
+ MAX1720X_FULLSOCTHR = 0x13,
+ MAX1720X_RCELL = 0x14,
+ MAX1720X_RFAST = 0x15,
+ MAX1720X_AVGTA = 0x16,
+ MAX1720X_CYCLES = 0x17,
+ MAX1720X_DESIGNCAP = 0x18,
+ MAX1720X_AVGVCELL = 0x19,
+ MAX1720X_MAXMINTEMP = 0x1A,
+ MAX1720X_MAXMINVOLT = 0x1B,
+ MAX1720X_MAXMINCURR = 0x1C,
+ MAX1720X_CONFIG = 0x1D,
+ MAX1720X_ICHGTERM = 0x1E,
+ MAX1720X_AVCAP = 0x1F,
+
+ MAX1720X_TTF = 0x20,
+ MAX1720X_DEVNAME = 0x21,
+ MAX1720X_QRTABLE10 = 0x22,
+ MAX1720X_FULLCAPNOM = 0x23,
+ MAX1720X_AIN0 = 0x27,
+ MAX1720X_LEARNCFG = 0x28,
+ MAX1720X_FILTERCFG = 0x29,
+ MAX1720X_RELAXCFG = 0x2A,
+ MAX1720X_MISCCFG = 0x2B,
+ MAX1720X_TGAIN = 0x2C,
+ Max1720x_TOff = 0x2D,
+ MAX1720X_CGAIN = 0x2E,
+ MAX1720X_COFF = 0x2F,
+
+ MAX1720X_QRTABLE20 = 0x32,
+ MAX1720X_FULLCAPREP = 0x35,
+ MAX1720X_IAVGEMPTY = 0x36,
+ MAX1720X_RCOMP0 = 0x38,
+ MAX1720X_TEMPCO = 0x39,
+ MAX1720X_VEMPTY = 0x3A,
+ MAX1720X_FSTAT = 0x3D,
+ MAX1720X_TIMER = 0x3E,
+ MAX1720X_SHDNTIMER = 0x3F,
+
+ MAX1720X_QRTABLE30 = 0x42,
+ MAX1720X_DQACC = 0x45,
+ MAX1720X_DPACC = 0x46,
+ MAX1720X_VFREMCAP = 0x4A,
+ MAX1720X_QH = 0x4D,
+
+ MAX1720X_STATUS2 = 0xB0,
+ MAX1720X_IALRTTH = 0xB4,
+ MAX1720X_VSHDNCFG = 0xB8,
+ MAX1720X_AGEFORECAST = 0xB9,
+ MAX1720X_HIBCFG = 0xBA,
+ MAX1720X_CONFIG2 = 0xBB,
+ MAX1720X_VRIPPLE = 0xBC,
+ MAX1720X_PACKCFG = 0xBD,
+ MAX1720X_TIMERH = 0xBE,
+
+ MAX1720X_AVGCELL4 = 0xD1,
+ MAX1720X_AVGCELL3 = 0xD2,
+ MAX1720X_AVGCELL2 = 0xD3,
+ MAX1720X_AVGCELL1 = 0xD4,
+ MAX1720X_CELL4 = 0xD5,
+ MAX1720X_CELL3 = 0xD6,
+ MAX1720X_CELL2 = 0xD7,
+ MAX1720X_CELL1 = 0xD8,
+ MAX1720X_CELLX = 0xD9,
+ MAX1720X_BATT = 0xDA,
+ MAX1720X_ATQRESIDUAL = 0xDC,
+ MAX1720X_ATTTE = 0xDD,
+ MAX1720X_ATAVSOC = 0xDE,
+ MAX1720X_ATAVCAP = 0xDF,
+
+ /* Individual Registers */
+ MAX1720X_COMMAND = 0x60,
+ MAX1720X_COMMSTAT = 0x61,
+ MAX1720X_LOCK = 0x7F,
+ MAX1720X_ODSCTH = 0xF2,
+ MAX1720X_ODSCCFG = 0xF3,
+ MAX1720X_VFOCV = 0xFB,
+ MAX1720X_VFSOC = 0xFF,
+};
+
+enum max1720x_status_bits {
+ MAX1720X_STATUS_POR = BIT(1),
+ MAX1720X_STATUS_IMN = BIT(2),
+ MAX1720X_STATUS_BST = BIT(3),
+ MAX1720X_STATUS_IMX = BIT(6),
+ MAX1720X_STATUS_DSOCI = BIT(7),
+ MAX1720X_STATUS_VMN = BIT(8),
+ MAX1720X_STATUS_TMN = BIT(9),
+ MAX1720X_STATUS_SMN = BIT(10),
+ MAX1720X_STATUS_BI = BIT(11),
+ MAX1720X_STATUS_VMX = BIT(12),
+ MAX1720X_STATUS_TMX = BIT(13),
+ MAX1720X_STATUS_SMX = BIT(14),
+ MAX1720X_STATUS_BR = BIT(15),
+};
+
+enum max1720x_commstat_bits {
+ MAX1720X_COMMSTAT_NVBUSY = BIT(1),
+ MAX1720X_COMMSTAT_NVERROR = BIT(2),
+};
+
+enum max1720x_config_bits {
+ MAX1720X_CONFIG_BER = BIT(0),
+ MAX1720X_CONFIG_BEI = BIT(1),
+ MAX1720X_CONFIG_AEN = BIT(2),
+ MAX1720X_CONFIG_FTHRM = BIT(3),
+ MAX1720X_CONFIG_ETHRM = BIT(4),
+ MAX1720X_CONFIG_COMMSH = BIT(6),
+ MAX1720X_CONFIG_SHDN = BIT(7),
+ MAX1720X_CONFIG_TEX = BIT(8),
+ MAX1720X_CONFIG_TEN = BIT(9),
+ MAX1720X_CONFIG_AINSH = BIT(10),
+ MAX1720X_CONFIG_ALRTP = BIT(11),
+ MAX1720X_CONFIG_VS = BIT(12),
+ MAX1720X_CONFIG_TS = BIT(13),
+ MAX1720X_CONFIG_SS = BIT(14),
+};
+
+enum max1720x_nnvcfg0_bits {
+ MAX1720X_NNVCFG0_ENSBS = BIT(0),
+ MAX1720X_NNVCFG0_ENHCFG = BIT(1),
+ MAX1720X_NNVCFG0_ENAF = BIT(2),
+ MAX1720X_NNVCFG0_ENMC = BIT(3),
+ MAX1720X_NNVCFG0_ENDC = BIT(4),
+ MAX1720X_NNVCFG0_ENVE = BIT(5),
+ MAX1720X_NNVCFG0_ENCG = BIT(6),
+ MAX1720X_NNVCFG0_ENICT = BIT(7),
+ MAX1720X_NNVCFG0_ENLCFG = BIT(8),
+ MAX1720X_NNVCFG0_ENRCFG = BIT(9),
+ MAX1720X_NNVCFG0_ENFCFG = BIT(10),
+ MAX1720X_NNVCFG0_ENCFG = BIT(11),
+ MAX1720X_NNVCFG0_ENX = BIT(14),
+ MAX1720X_NNVCFG0_ENOCV = BIT(15),
+};
+
+enum max1720x_command_bits {
+ MAX1720X_COMMAND_FUEL_GAUGE_RESET = 0x0001,
+ MAX1720X_COMMAND_HARDWARE_RESET = 0x000F,
+ MAX1720X_COMMAND_QUERY_REMAINING_UPDATES = 0xE2FA,
+ MAX1720X_COMMAND_COPY_NV_BLOCK = 0xE904,
+ MAX1720X_COMMAND_HISTORY_RECALL_WRITE_0 = 0xE2FB,
+ MAX1720X_COMMAND_HISTORY_RECALL_WRITE_1 = 0xE2FC,
+ MAX1720X_COMMAND_HISTORY_RECALL_VALID_0 = 0xE2FC,
+ MAX1720X_COMMAND_HISTORY_RECALL_VALID_1 = 0xE2FD,
+ MAX1720X_COMMAND_HISTORY_RECALL_VALID_2 = 0xE2FE,
+ MAX1720X_READ_HISTORY_CMD_BASE = 0xE226,
+};
+
+/** Nonvolatile Register Memory Map */
+enum max1720x_nvram {
+ MAX1720X_NVRAM_START = 0x80,
+ MAX1720X_NUSER18C = 0x8C,
+ MAX1720X_NUSER18D = 0x8D,
+ MAX1720X_NODSCTH = 0x8E,
+ MAX1720X_NODSCCFG = 0x8F,
+ MAX1720X_NLEARNCFG = 0x9F,
+ MAX1720X_NMISCCFG = 0xB2,
+ MAX1720X_NHIBCFG = 0xB4,
+ MAX1720X_NCONVGCFG = 0xB7,
+ MAX1720X_NNVCFG0 = 0xB8,
+ MAX1720X_NUSER1C4 = 0xC4,
+ MAX1720X_NUSER1C5 = 0xC5,
+ MAX1720X_NCGAIN = 0xC8,
+ MAX1720X_NMANFCTRNAME0 = 0xCC,
+ MAX1720X_NMANFCTRNAME1 = 0xCD,
+ MAX1720X_NMANFCTRNAME2 = 0xCE,
+ MAX1720X_NRSENSE = 0xCF,
+ MAX1720X_NUSER1D0 = 0xD0,
+ MAX1720X_NUSER1D1 = 0xD1,
+ MAX1720X_NUSER1D4 = 0xD4,
+ MAX1720X_NMANFCTRDATE = 0xD6,
+ MAX1720X_NFIRSTUSED = 0xD7,
+ MAX1720X_NSERIALNUMBER0 = 0xD8,
+ MAX1720X_NSERIALNUMBER1 = 0xD9,
+ MAX1720X_NSERIALNUMBER2 = 0xDA,
+ MAX1720X_NDEVICENAME0 = 0xDB,
+ MAX1720X_NDEVICENAME1 = 0xDC,
+ MAX1720X_NDEVICENAME2 = 0xDD,
+ MAX1720X_NDEVICENAME3 = 0xDE,
+ MAX1720X_NDEVICENAME4 = 0xDF,
+ MAX1720X_NVRAM_END = 0xE0,
+ MAX1720X_NVRAM_HISTORY_WRITE_STATUS_START = 0xE1,
+ MAX1720X_NVRAM_HISTORY_WRITE_STATUS_END = 0xEA,
+ MAX1720X_NVRAM_HISTORY_VALID_STATUS_START = 0xEB,
+ MAX1720X_NVRAM_HISTORY_VALID_STATUS_END = 0xE4,
+ MAX1720X_NVRAM_REMAINING_UPDATES = 0xED,
+ MAX1720X_NVRAM_HISTORY_END = 0xEF,
+};
+
+#define BUCKET_COUNT 10
+
+const unsigned int max1720x_cycle_counter_addr[BUCKET_COUNT] = {
+ MAX1720X_NODSCTH,
+ MAX1720X_NODSCCFG,
+ MAX1720X_NMISCCFG,
+ MAX1720X_NHIBCFG,
+ MAX1720X_NMANFCTRNAME1,
+ MAX1720X_NMANFCTRNAME2,
+ MAX1720X_NFIRSTUSED,
+ MAX1720X_NDEVICENAME4,
+ MAX1720X_NUSER1C4,
+ MAX1720X_NUSER1C5,
+};
+
+#define NVRAM_U16_INDEX(reg) (reg - MAX1720X_NVRAM_START)
+#define NVRAM_BYTE_INDEX(reg) (NVRAM_U16_INDEX(reg) * sizeof(u16))
+#define NVRAM_BYTE_INDEX_FROM(reg, base) ((reg - base) * sizeof(u16))
+#define MAX1720X_NVRAM_U16_SIZE NVRAM_U16_INDEX(MAX1720X_NVRAM_END)
+#define MAX1720X_NVRAM_SIZE NVRAM_BYTE_INDEX(MAX1720X_NVRAM_END)
+
+#define MAX1720X_HISTORY_PAGE_SIZE \
+ (MAX1720X_NVRAM_HISTORY_END - MAX1720X_NVRAM_END + 1)
+
+#define MAX1720X_N_OF_HISTORY_FLAGS_REG \
+ (MAX1720X_NVRAM_HISTORY_END - \
+ MAX1720X_NVRAM_HISTORY_WRITE_STATUS_START + 1 + \
+ MAX1720X_NVRAM_HISTORY_WRITE_STATUS_END - \
+ MAX1720X_NVRAM_END + 1)
+#define MAX1720X_N_OF_QRTABLES 4
+
+struct max1720x_cyc_ctr_data {
+ u16 count[BUCKET_COUNT];
+ struct mutex lock;
+ int prev_soc;
+};
+
+
+struct max1720x_history {
+ loff_t history_index;
+ int history_count;
+ bool *page_status;
+ u16 *history;
+};
+
+struct max1720x_chip {
+ struct device *dev;
+ struct regmap *regmap;
+ struct power_supply *psy;
+ struct delayed_work init_work;
+ struct work_struct cycle_count_work;
+ struct i2c_client *primary;
+ struct i2c_client *secondary;
+ struct regmap *regmap_nvram;
+ struct device_node *batt_node;
+ struct iio_channel *iio_ch;
+ struct max1720x_cyc_ctr_data cyc_ctr;
+
+ /* history */
+ struct mutex history_lock;
+ int hcmajor;
+ struct cdev hcdev;
+ struct class *hcclass;
+ bool history_available;
+ bool history_added;
+
+ u16 RSense;
+ u16 RConfig;
+ bool init_complete;
+ bool resume_complete;
+ u16 health_status;
+ int fake_capacity;
+ int previous_qh;
+ int current_capacity;
+ int prev_charge_state;
+ char serial_number[25];
+ bool offmode_charger;
+ u32 convgcfg_hysteresis;
+ int nb_convgcfg;
+ int curr_convgcfg_idx;
+ s16 *temp_convgcfg;
+ u16 *convgcfg_values;
+ struct mutex convgcfg_lock;
+ unsigned int debug_irq_none_cnt;
+};
+
+static inline int max1720x_regmap_read(struct regmap *map,
+ unsigned int reg,
+ u16 *val,
+ const char *name)
+{
+ unsigned int tmp;
+ int rtn = regmap_read(map, reg, &tmp);
+
+ if (rtn)
+ pr_err("Failed to read %s\n", name);
+ else
+ *val = tmp;
+
+ return rtn;
+
+}
+#define REGMAP_READ(regmap, what, dst) \
+ max1720x_regmap_read(regmap, what, dst, #what)
+
+#define REGMAP_WRITE(regmap, what, data) \
+ do { \
+ int rtn; \
+ rtn = regmap_write(regmap, what, data); \
+ if (rtn) { \
+ pr_err("Failed to write %s\n", #what); \
+ } \
+ } while (0)
+
+static char *psy_status_str[] = {
+ "Unknown", "Charging", "Discharging", "NotCharging", "Full"
+};
+
+bool max1720x_is_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case MAX1720X_COMMAND:
+ case MAX1720X_COMMSTAT:
+ case MAX1720X_LOCK:
+ case MAX1720X_ODSCTH:
+ case MAX1720X_ODSCCFG:
+ case MAX1720X_VFOCV:
+ case MAX1720X_VFSOC:
+ case 0x00 ... 0x4F:
+ case 0xB0 ... 0xDF:
+ return true;
+ }
+ return false;
+}
+
+static const struct regmap_config max1720x_regmap_cfg = {
+ .reg_bits = 8,
+ .val_bits = 16,
+ .val_format_endian = REGMAP_ENDIAN_NATIVE,
+ .max_register = MAX1720X_VFSOC,
+ .readable_reg = max1720x_is_reg,
+ .volatile_reg = max1720x_is_reg,
+
+};
+
+bool max1720x_is_nvram_reg(struct device *dev, unsigned int reg)
+{
+ return (reg >= MAX1720X_NVRAM_START
+ && reg <= MAX1720X_NVRAM_HISTORY_END);
+}
+
+static const struct regmap_config max1720x_regmap_nvram_cfg = {
+ .reg_bits = 8,
+ .val_bits = 16,
+ .val_format_endian = REGMAP_ENDIAN_NATIVE,
+ .max_register = MAX1720X_NVRAM_HISTORY_END,
+ .readable_reg = max1720x_is_nvram_reg,
+ .volatile_reg = max1720x_is_nvram_reg,
+};
+
+static inline int reg_to_percentage(u16 val)
+{
+ /* LSB: 1/256% */
+ return val >> 8;
+}
+
+static inline int reg_to_twos_comp_int(u16 val)
+{
+ /* Convert u16 to twos complement */
+ return -(val & 0x8000) + (val & 0x7FFF);
+}
+
+static inline int reg_to_micro_amp_h(s16 val, u16 rsense)
+{
+ /* LSB: 5.0μVh/RSENSE ; Rsense LSB is 10μΩ */
+ return div_s64((s64) val * 500000, rsense);
+}
+
+static inline int reg_to_micro_volt(u16 val)
+{
+ /* LSB: 0.078125mV */
+ return div_u64((u64) val * 78125, 1000);
+}
+
+static inline int reg_to_micro_amp(s16 val, u16 rsense)
+{
+ /* LSB: 1.5625μV/RSENSE ; Rsense LSB is 10μΩ */
+ return div_s64((s64) val * 156250, rsense);
+}
+
+static inline int reg_to_deci_deg_cel(s16 val)
+{
+ /* LSB: 1/256°C */
+ return div_s64((s64) val * 10, 256);
+}
+
+static inline int reg_to_resistance_micro_ohms(s16 val, u16 rsense)
+{
+ /* LSB: 1/4096 Ohm */
+ return div_s64((s64) val * 1000 * rsense, 4096);
+}
+
+static inline int reg_to_cycles(s16 val)
+{
+ /* LSB: 16% of one cycle */
+ return DIV_ROUND_CLOSEST((int) val * 16, 100);
+}
+
+static inline int reg_to_seconds(s16 val)
+{
+ /* LSB: 5.625 seconds */
+ return DIV_ROUND_CLOSEST((int) val * 5625, 1000);
+}
+
+static void max1720x_read_log_write_status(struct max1720x_chip *chip,
+ u16 *buffer)
+{
+ int i;
+ u16 data = 0;
+
+ REGMAP_WRITE(chip->regmap, MAX1720X_COMMAND,
+ MAX1720X_COMMAND_HISTORY_RECALL_WRITE_0);
+ msleep(MAX1720X_TRECALL_MS);
+ for (i = MAX1720X_NVRAM_HISTORY_WRITE_STATUS_START;
+ i <= MAX1720X_NVRAM_HISTORY_END; i++) {
+ (void) REGMAP_READ(chip->regmap_nvram, i, &data);
+ *buffer++ = data;
+ }
+ REGMAP_WRITE(chip->regmap, MAX1720X_COMMAND,
+ MAX1720X_COMMAND_HISTORY_RECALL_WRITE_1);
+ msleep(MAX1720X_TRECALL_MS);
+ for (i = MAX1720X_NVRAM_END;
+ i <= MAX1720X_NVRAM_HISTORY_WRITE_STATUS_END; i++) {
+ (void) REGMAP_READ(chip->regmap_nvram, i, &data);
+ *buffer++ = data;
+ }
+}
+
+static void max1720x_read_log_valid_status(struct max1720x_chip *chip,
+ u16 *buffer)
+{
+ int i;
+ u16 data = 0;
+
+ REGMAP_WRITE(chip->regmap, MAX1720X_COMMAND,
+ MAX1720X_COMMAND_HISTORY_RECALL_VALID_0);
+ msleep(MAX1720X_TRECALL_MS);
+ for (i = MAX1720X_NVRAM_HISTORY_VALID_STATUS_START;
+ i <= MAX1720X_NVRAM_HISTORY_END; i++) {
+ (void) REGMAP_READ(chip->regmap_nvram, i, &data);
+ *buffer++ = data;
+ }
+ REGMAP_WRITE(chip->regmap, MAX1720X_COMMAND,
+ MAX1720X_COMMAND_HISTORY_RECALL_VALID_1);
+ msleep(MAX1720X_TRECALL_MS);
+ for (i = MAX1720X_NVRAM_END; i <= MAX1720X_NVRAM_HISTORY_END; i++) {
+ (void) REGMAP_READ(chip->regmap_nvram, i, &data);
+ *buffer++ = data;
+ }
+ REGMAP_WRITE(chip->regmap, MAX1720X_COMMAND,
+ MAX1720X_COMMAND_HISTORY_RECALL_VALID_2);
+ msleep(MAX1720X_TRECALL_MS);
+ for (i = MAX1720X_NVRAM_END;
+ i <= MAX1720X_NVRAM_HISTORY_VALID_STATUS_END; i++) {
+ (void) REGMAP_READ(chip->regmap_nvram, i, &data);
+ *buffer++ = data;
+ }
+}
+
+/* @return the number of pages or negative for error */
+static int get_battery_history_status(struct max1720x_chip *chip,
+ bool *page_status)
+{
+ u16 *write_status, *valid_status;
+ int i, addr_offset, bit_offset;
+ int valid_history_entry_count = 0;
+
+ write_status = kmalloc_array(MAX1720X_N_OF_HISTORY_FLAGS_REG,
+ sizeof(u16), GFP_KERNEL);
+ if (!write_status)
+ return -ENOMEM;
+
+ valid_status = kmalloc_array(MAX1720X_N_OF_HISTORY_FLAGS_REG,
+ sizeof(u16), GFP_KERNEL);
+ if (!valid_status) {
+ kfree(write_status);
+ return -ENOMEM;
+ }
+
+ max1720x_read_log_write_status(chip, write_status);
+ max1720x_read_log_valid_status(chip, valid_status);
+
+ /* Figure out the pages with valid history entry */
+ for (i = 0; i < MAX1720X_N_OF_HISTORY_PAGES; i++) {
+ addr_offset = i / 8;
+ bit_offset = i % 8;
+ page_status[i] =
+ ((write_status[addr_offset] & BIT(bit_offset)) ||
+ (write_status[addr_offset] & BIT(bit_offset + 8))) &&
+ ((valid_status[addr_offset] & BIT(bit_offset)) ||
+ (valid_status[addr_offset] & BIT(bit_offset + 8)));
+ if (page_status[i])
+ valid_history_entry_count++;
+ }
+
+ kfree(write_status);
+ kfree(valid_status);
+
+ return valid_history_entry_count;
+}
+
+static void get_battery_history(struct max1720x_chip *chip,
+ bool *page_status, u16 *history)
+{
+ int i, j, index = 0;
+ u16 data = 0;
+
+ for (i = 0; i < MAX1720X_N_OF_HISTORY_PAGES; i++) {
+ if (!page_status[i])
+ continue;
+ REGMAP_WRITE(chip->regmap, MAX1720X_COMMAND,
+ MAX1720X_READ_HISTORY_CMD_BASE + i);
+ msleep(MAX1720X_TRECALL_MS);
+ for (j = 0; j < MAX1720X_HISTORY_PAGE_SIZE; j++) {
+ (void) REGMAP_READ(chip->regmap_nvram,
+ MAX1720X_NVRAM_END + j,
+ &data);
+ history[index * MAX1720X_HISTORY_PAGE_SIZE + j] = data;
+ }
+ index++;
+ }
+}
+
+static int format_battery_history_entry(char *temp, int size, u16 *line)
+{
+ int length = 0, i;
+
+ for (i = 0; i < MAX1720X_HISTORY_PAGE_SIZE; i++) {
+ length += scnprintf(temp + length,
+ size - length, "%04x ",
+ line[i]);
+ }
+
+ if (length > 0)
+ temp[--length] = 0;
+ return length;
+}
+
+/* @return number of valid entries */
+static int max1720x_history_read(struct max1720x_chip *chip,
+ struct max1720x_history *hi)
+{
+ memset(hi, 0, sizeof(*hi));
+
+ hi->page_status = kcalloc(MAX1720X_N_OF_HISTORY_PAGES,
+ sizeof(bool), GFP_KERNEL);
+ if (!hi->page_status)
+ return -ENOMEM;
+
+ mutex_lock(&chip->history_lock);
+
+ hi->history_count = get_battery_history_status(chip, hi->page_status);
+ if (hi->history_count < 0) {
+ goto error_exit;
+ } else if (hi->history_count != 0) {
+ const int size = hi->history_count * MAX1720X_HISTORY_PAGE_SIZE;
+
+ hi->history = kmalloc_array(size, sizeof(u16), GFP_KERNEL);
+ if (!hi->history) {
+ hi->history_count = -ENOMEM;
+ goto error_exit;
+ }
+
+ get_battery_history(chip, hi->page_status, hi->history);
+ }
+
+ mutex_unlock(&chip->history_lock);
+ return hi->history_count;
+
+error_exit:
+ mutex_unlock(&chip->history_lock);
+ kfree(hi->page_status);
+ hi->page_status = NULL;
+ return hi->history_count;
+
+}
+
+static void max1720x_history_free(struct max1720x_history *hi)
+{
+ kfree(hi->page_status);
+ kfree(hi->history);
+
+ hi->history = NULL;
+ hi->page_status = NULL;
+}
+
+static enum power_supply_property max1720x_battery_props[] = {
+ POWER_SUPPLY_PROP_STATUS,
+ POWER_SUPPLY_PROP_HEALTH,
+ POWER_SUPPLY_PROP_CAPACITY,
+ POWER_SUPPLY_PROP_CHARGE_COUNTER,
+ POWER_SUPPLY_PROP_CHARGE_FULL,
+ POWER_SUPPLY_PROP_CHARGE_FULL_DESIGN,
+ POWER_SUPPLY_PROP_CURRENT_AVG,
+ POWER_SUPPLY_PROP_CURRENT_NOW,
+ POWER_SUPPLY_PROP_CYCLE_COUNT,
+ POWER_SUPPLY_PROP_PRESENT,
+ POWER_SUPPLY_PROP_RESISTANCE,
+ POWER_SUPPLY_PROP_TEMP,
+ POWER_SUPPLY_PROP_TIME_TO_EMPTY_AVG,
+ POWER_SUPPLY_PROP_TIME_TO_FULL_AVG,
+ POWER_SUPPLY_PROP_VOLTAGE_AVG,
+ POWER_SUPPLY_PROP_VOLTAGE_MAX_DESIGN,
+ POWER_SUPPLY_PROP_VOLTAGE_MIN_DESIGN,
+ POWER_SUPPLY_PROP_VOLTAGE_NOW,
+ POWER_SUPPLY_PROP_VOLTAGE_OCV,
+ POWER_SUPPLY_PROP_TECHNOLOGY,
+ POWER_SUPPLY_PROP_SERIAL_NUMBER,
+};
+
+static void max1720x_cycle_count_work(struct work_struct *work)
+{
+ int bucket, cnt, batt_soc;
+ u16 data;
+ struct max1720x_chip *chip = container_of(work, struct max1720x_chip,
+ cycle_count_work);
+
+ if (REGMAP_READ(chip->regmap, MAX1720X_REPSOC, &data))
+ return;
+ batt_soc = reg_to_percentage(data);
+
+ mutex_lock(&chip->cyc_ctr.lock);
+
+ if (chip->cyc_ctr.prev_soc != -1 &&
+ batt_soc >= 0 && batt_soc <= 100 &&
+ batt_soc > chip->cyc_ctr.prev_soc) {
+ for (cnt = batt_soc ; cnt > chip->cyc_ctr.prev_soc ; cnt--) {
+ bucket = cnt * BUCKET_COUNT / 100;
+ if (bucket >= BUCKET_COUNT)
+ bucket = BUCKET_COUNT - 1;
+ chip->cyc_ctr.count[bucket]++;
+ REGMAP_WRITE(chip->regmap_nvram,
+ max1720x_cycle_counter_addr[bucket],
+ chip->cyc_ctr.count[bucket]);
+ pr_debug("Stored count: prev_soc=%d, soc=%d bucket=%d count=%d\n",
+ chip->cyc_ctr.prev_soc, cnt, bucket,
+ chip->cyc_ctr.count[bucket]);
+ }
+ }
+ chip->cyc_ctr.prev_soc = batt_soc;
+
+ mutex_unlock(&chip->cyc_ctr.lock);
+}
+
+static void max1720x_restore_cycle_counter(struct max1720x_chip *chip)
+{
+ int i;
+ u16 data = 0;
+
+ mutex_lock(&chip->cyc_ctr.lock);
+
+ for (i = 0; i < BUCKET_COUNT; i++) {
+ if (!REGMAP_READ(chip->regmap_nvram,
+ max1720x_cycle_counter_addr[i],
+ &data)) {
+ chip->cyc_ctr.count[i] = data;
+ pr_debug("max1720x_cycle_counter[%d], addr=0x%02X, count=%d\n",
+ i, max1720x_cycle_counter_addr[i],
+ chip->cyc_ctr.count[i]);
+ }
+ }
+ chip->cyc_ctr.prev_soc = -1;
+
+ mutex_unlock(&chip->cyc_ctr.lock);
+}
+
+static ssize_t max1720x_get_cycle_counts_bins(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct power_supply *psy;
+ struct max1720x_chip *chip;
+ int len = 0, i;
+
+ psy = container_of(dev, struct power_supply, dev);
+ chip = power_supply_get_drvdata(psy);
+
+ mutex_lock(&chip->cyc_ctr.lock);
+
+ for (i = 0; i < BUCKET_COUNT; i++) {
+
+ len += scnprintf(buf + len, PAGE_SIZE - len, "%d",
+ chip->cyc_ctr.count[i]);
+
+ if (i == BUCKET_COUNT-1)
+ len += scnprintf(buf + len, PAGE_SIZE - len, "\n");
+ else
+ len += scnprintf(buf + len, PAGE_SIZE - len, " ");
+ }
+
+ mutex_unlock(&chip->cyc_ctr.lock);
+
+ return len;
+}
+
+static ssize_t max1720x_set_cycle_counts_bins(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct power_supply *psy;
+ struct max1720x_chip *chip;
+ int val[BUCKET_COUNT], i;
+
+ psy = container_of(dev, struct power_supply, dev);
+ chip = power_supply_get_drvdata(psy);
+
+ if (sscanf(buf, "%d %d %d %d %d %d %d %d %d %d",
+ &val[0], &val[1], &val[2], &val[3], &val[4],
+ &val[5], &val[6], &val[7], &val[8], &val[9]) != BUCKET_COUNT)
+ return -EINVAL;
+
+ mutex_lock(&chip->cyc_ctr.lock);
+ for (i = 0; i < BUCKET_COUNT; i++) {
+ if (val[i] >= 0 && val[i] < U16_MAX) {
+ chip->cyc_ctr.count[i] = val[i];
+ REGMAP_WRITE(chip->regmap_nvram,
+ max1720x_cycle_counter_addr[i],
+ chip->cyc_ctr.count[i]);
+ }
+ }
+ mutex_unlock(&chip->cyc_ctr.lock);
+
+ return count;
+}
+
+static DEVICE_ATTR(cycle_counts_bins, 0660,
+ max1720x_get_cycle_counts_bins,
+ max1720x_set_cycle_counts_bins);
+
+static ssize_t max1720x_get_offmode_charger(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct power_supply *psy = container_of(dev, struct power_supply, dev);
+ struct max1720x_chip *chip = power_supply_get_drvdata(psy);
+
+ return scnprintf(buf, PAGE_SIZE, "%hhd\n", chip->offmode_charger);
+}
+
+static ssize_t max1720x_set_offmode_charger(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct power_supply *psy = container_of(dev, struct power_supply, dev);
+ struct max1720x_chip *chip = power_supply_get_drvdata(psy);
+
+ if (kstrtobool(buf, &chip->offmode_charger))
+ return -EINVAL;
+
+ return count;
+}
+
+static DEVICE_ATTR(offmode_charger, 0660,
+ max1720x_get_offmode_charger,
+ max1720x_set_offmode_charger);
+
+static int max1720x_get_battery_soc(struct max1720x_chip *chip)
+{
+ u16 data;
+ int capacity, err;
+
+ if (chip->fake_capacity >= 0 && chip->fake_capacity <= 100)
+ return chip->fake_capacity;
+
+ err = REGMAP_READ(chip->regmap, MAX1720X_REPSOC, &data);
+ if (err)
+ return err;
+ capacity = reg_to_percentage(data);
+
+ if (capacity == 100 && chip->offmode_charger)
+ chip->fake_capacity = 100;
+
+ return capacity;
+}
+
+static void max1720x_prime_battery_qh_capacity(struct max1720x_chip *chip,
+ int status)
+{
+ u16 data = 0;
+
+ (void) REGMAP_READ(chip->regmap, MAX1720X_MIXCAP, &data);
+ chip->current_capacity = data;
+
+ REGMAP_WRITE(chip->regmap_nvram, MAX1720X_NUSER18C, ~data);
+ dev_info(chip->dev, "Capacity primed to %d on %s\n",
+ data, psy_status_str[status]);
+
+ (void) REGMAP_READ(chip->regmap, MAX1720X_QH, &data);
+ chip->previous_qh = reg_to_twos_comp_int(data);
+
+ REGMAP_WRITE(chip->regmap_nvram, MAX1720X_NUSER18D, data);
+ dev_info(chip->dev, "QH primed to %d on %s\n",
+ data, psy_status_str[status]);
+}
+
+static int max1720x_get_battery_status(struct max1720x_chip *chip)
+{
+ u16 data = 0;
+ int current_avg, ichgterm, fullsocthr;
+ int status = POWER_SUPPLY_STATUS_UNKNOWN, err;
+
+ err = REGMAP_READ(chip->regmap, MAX1720X_AVGCURRENT, &data);
+ if (err)
+ return err;
+ current_avg = -reg_to_micro_amp(data, chip->RSense);
+
+ err = REGMAP_READ(chip->regmap, MAX1720X_ICHGTERM, &data);
+ if (err)
+ return err;
+ ichgterm = reg_to_micro_amp(data, chip->RSense);
+
+ err = REGMAP_READ(chip->regmap, MAX1720X_FULLSOCTHR, &data);
+ if (err)
+ return err;
+ fullsocthr = reg_to_percentage(data);
+
+ if (current_avg < -ichgterm) {
+ status = POWER_SUPPLY_STATUS_CHARGING;
+ if (chip->prev_charge_state == POWER_SUPPLY_STATUS_DISCHARGING)
+ max1720x_prime_battery_qh_capacity(chip, status);
+ chip->prev_charge_state = POWER_SUPPLY_STATUS_CHARGING;
+ } else if (current_avg <= 0 &&
+ max1720x_get_battery_soc(chip) >= fullsocthr) {
+ status = POWER_SUPPLY_STATUS_FULL;
+ if (chip->prev_charge_state != POWER_SUPPLY_STATUS_FULL)
+ max1720x_prime_battery_qh_capacity(chip, status);
+ chip->prev_charge_state = POWER_SUPPLY_STATUS_FULL;
+ } else {
+ status = POWER_SUPPLY_STATUS_DISCHARGING;
+ chip->prev_charge_state = POWER_SUPPLY_STATUS_DISCHARGING;
+ }
+
+ return status;
+}
+
+static int max1720x_get_battery_health(struct max1720x_chip *chip)
+{
+ /* For health report what ever was recently alerted and clear it */
+
+ if (chip->health_status & MAX1720X_STATUS_VMX) {
+ chip->health_status &= ~MAX1720X_STATUS_VMX;
+ return POWER_SUPPLY_HEALTH_OVERVOLTAGE;
+ }
+
+ if (chip->health_status & MAX1720X_STATUS_TMN) {
+ chip->health_status &= ~MAX1720X_STATUS_TMN;
+ return POWER_SUPPLY_HEALTH_COLD;
+ }
+
+ if (chip->health_status & MAX1720X_STATUS_TMX) {
+ chip->health_status &= ~MAX1720X_STATUS_TMX;
+ return POWER_SUPPLY_HEALTH_OVERHEAT;
+ }
+
+ return POWER_SUPPLY_HEALTH_GOOD;
+}
+
+static int max1720x_set_battery_soc(struct max1720x_chip *chip,
+ const union power_supply_propval *val)
+{
+ chip->fake_capacity = val->intval;
+
+ if (chip->psy)
+ power_supply_changed(chip->psy);
+
+ return 0;
+}
+
+static int max1720x_update_battery_qh_based_capacity(struct max1720x_chip *chip)
+{
+ u16 data;
+ int current_qh, err = 0;
+
+ err = REGMAP_READ(chip->regmap, MAX1720X_QH, &data);
+ if (err)
+ return err;
+ current_qh = reg_to_twos_comp_int(data);
+
+ /* QH value accumulates as battery charges */
+ chip->current_capacity -= (chip->previous_qh - current_qh);
+ chip->previous_qh = current_qh;
+
+ return 0;
+}
+
+static void max1720x_restore_battery_qh_capacity(struct max1720x_chip *chip)
+{
+ u16 data = 0, nvram_capacity;
+ int current_qh, nvram_qh;
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NUSER18C, &data);
+ nvram_capacity = ~data;
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NUSER18D, &data);
+ nvram_qh = reg_to_twos_comp_int(data);
+
+ (void) REGMAP_READ(chip->regmap, MAX1720X_QH, &data);
+ current_qh = reg_to_twos_comp_int(data);
+
+ /* QH value accumulates as battery discharges */
+ chip->current_capacity = (int) nvram_capacity - (nvram_qh - current_qh);
+ dev_info(chip->dev, "Capacity restored to %d\n",
+ chip->current_capacity);
+ chip->previous_qh = current_qh;
+ dev_info(chip->dev, "QH value restored to %d\n",
+ chip->previous_qh);
+}
+
+static void max1720x_handle_update_nconvgcfg(struct max1720x_chip *chip,
+ int temp)
+{
+ int idx = -1;
+
+ if (chip->temp_convgcfg == NULL)
+ return;
+
+ if (temp <= chip->temp_convgcfg[0]) {
+ idx = 0;
+ } else if (temp > chip->temp_convgcfg[chip->nb_convgcfg - 1]) {
+ idx = chip->nb_convgcfg - 1;
+ } else {
+ for (idx = 1 ; idx < chip->nb_convgcfg; idx++) {
+ if (temp > chip->temp_convgcfg[idx - 1] &&
+ temp <= chip->temp_convgcfg[idx])
+ break;
+ }
+ }
+ mutex_lock(&chip->convgcfg_lock);
+ /* We want to switch to higher slot only if above temp + hysteresis
+ * but when temperature drops, we want to change at the level
+ */
+ if ((idx != chip->curr_convgcfg_idx) &&
+ (chip->curr_convgcfg_idx == -1 || idx < chip->curr_convgcfg_idx ||
+ temp >= chip->temp_convgcfg[chip->curr_convgcfg_idx] +
+ chip->convgcfg_hysteresis)) {
+ REGMAP_WRITE(chip->regmap_nvram, MAX1720X_NCONVGCFG,
+ chip->convgcfg_values[idx]);
+ chip->curr_convgcfg_idx = idx;
+ dev_info(chip->dev, "updating nConvgcfg to 0x%04x as temp is %d (idx:%d)\n",
+ chip->convgcfg_values[idx], temp, idx);
+ }
+ mutex_unlock(&chip->convgcfg_lock);
+}
+
+static int max1720x_get_property(struct power_supply *psy,
+ enum power_supply_property psp,
+ union power_supply_propval *val)
+{
+ struct max1720x_chip *chip = power_supply_get_drvdata(psy);
+ struct regmap *map = chip->regmap;
+ u16 data = 0;
+ int err = 0;
+
+ pm_runtime_get_sync(chip->dev);
+ if (!chip->init_complete || !chip->resume_complete) {
+ pm_runtime_put_sync(chip->dev);
+ return -EAGAIN;
+ }
+ pm_runtime_put_sync(chip->dev);
+
+ switch (psp) {
+ case POWER_SUPPLY_PROP_STATUS:
+ val->intval = max1720x_get_battery_status(chip);
+ if (val->intval < 0)
+ return val->intval;
+ break;
+ case POWER_SUPPLY_PROP_HEALTH:
+ val->intval = max1720x_get_battery_health(chip);
+ break;
+ case POWER_SUPPLY_PROP_CAPACITY:
+ val->intval = max1720x_get_battery_soc(chip);
+ if (val->intval < 0)
+ return val->intval;
+ break;
+ case POWER_SUPPLY_PROP_CHARGE_COUNTER:
+ err = max1720x_update_battery_qh_based_capacity(chip);
+ if (err < 0)
+ return err;
+ val->intval = reg_to_micro_amp_h(chip->current_capacity,
+ chip->RSense);
+ break;
+ case POWER_SUPPLY_PROP_CHARGE_FULL:
+ /*
+ * Snap charge_full to DESIGNCAP during early charge cycles to
+ * prevent large fluctuations in FULLCAPNOM. MAX1720X_CYCLES LSB
+ * is 16%
+ */
+ err = REGMAP_READ(map, MAX1720X_CYCLES, &data);
+ if (!err) {
+ if (reg_to_cycles(data) <= FULLCAPNOM_STABILIZE_CYCLES)
+ err = REGMAP_READ(map, MAX1720X_DESIGNCAP,
+ &data);
+ else
+ err = REGMAP_READ(map, MAX1720X_FULLCAPNOM,
+ &data);
+ val->intval = reg_to_micro_amp_h(data, chip->RSense);
+ }
+ break;
+ case POWER_SUPPLY_PROP_CHARGE_FULL_DESIGN:
+ err = REGMAP_READ(map, MAX1720X_DESIGNCAP, &data);
+ val->intval = reg_to_micro_amp_h(data, chip->RSense);
+ break;
+ case POWER_SUPPLY_PROP_CURRENT_AVG:
+ err = REGMAP_READ(map, MAX1720X_AVGCURRENT, &data);
+ /* current is positive value when flowing to device */
+ val->intval = -reg_to_micro_amp(data, chip->RSense);
+ break;
+ case POWER_SUPPLY_PROP_CURRENT_NOW:
+ err = REGMAP_READ(map, MAX1720X_CURRENT, &data);
+ /* current is positive value when flowing to device */
+ val->intval = -reg_to_micro_amp(data, chip->RSense);
+ break;
+ case POWER_SUPPLY_PROP_CYCLE_COUNT:
+ err = REGMAP_READ(map, MAX1720X_CYCLES, &data);
+ val->intval = reg_to_cycles(data);
+ break;
+ case POWER_SUPPLY_PROP_PRESENT:
+ err = REGMAP_READ(map, MAX1720X_STATUS, &data);
+ val->intval = (((u16) data) & MAX1720X_STATUS_BST) ? 0 : 1;
+ break;
+ case POWER_SUPPLY_PROP_RESISTANCE:
+ err = REGMAP_READ(map, MAX1720X_RCELL, &data);
+ val->intval = reg_to_resistance_micro_ohms(data, chip->RSense);
+ break;
+ case POWER_SUPPLY_PROP_TEMP:
+ REGMAP_READ(map, MAX1720X_TEMP, &data);
+ val->intval = reg_to_deci_deg_cel(data);
+ max1720x_handle_update_nconvgcfg(chip, val->intval);
+ break;
+ case POWER_SUPPLY_PROP_TIME_TO_EMPTY_AVG:
+ err = REGMAP_READ(map, MAX1720X_TTE, &data);
+ val->intval = reg_to_seconds(data);
+ break;
+ case POWER_SUPPLY_PROP_TIME_TO_FULL_AVG:
+ err = REGMAP_READ(map, MAX1720X_TTF, &data);
+ val->intval = reg_to_seconds(data);
+ break;
+ case POWER_SUPPLY_PROP_VOLTAGE_AVG:
+ err = REGMAP_READ(map, MAX1720X_AVGVCELL, &data);
+ val->intval = reg_to_micro_volt(data);
+ break;
+ case POWER_SUPPLY_PROP_VOLTAGE_MAX_DESIGN:
+ err = REGMAP_READ(map, MAX1720X_MAXMINVOLT, &data);
+ /* LSB: 20mV */
+ val->intval = ((data >> 8) & 0xFF) * 20000;
+ break;
+ case POWER_SUPPLY_PROP_VOLTAGE_MIN_DESIGN:
+ err = REGMAP_READ(map, MAX1720X_MAXMINVOLT, &data);
+ /* LSB: 20mV */
+ val->intval = (data & 0xFF) * 20000;
+ break;
+ case POWER_SUPPLY_PROP_VOLTAGE_NOW:
+ err = REGMAP_READ(map, MAX1720X_VCELL, &data);
+ val->intval = reg_to_micro_volt(data);
+ break;
+ case POWER_SUPPLY_PROP_VOLTAGE_OCV:
+ err = REGMAP_READ(map, MAX1720X_VFOCV, &data);
+ val->intval = reg_to_micro_volt(data);
+ break;
+ case POWER_SUPPLY_PROP_TECHNOLOGY:
+ val->intval = POWER_SUPPLY_TECHNOLOGY_LION;
+ break;
+ case POWER_SUPPLY_PROP_SERIAL_NUMBER:
+ val->strval = chip->serial_number;
+ break;
+ default:
+ return -EINVAL;
+ }
+ if (err < 0)
+ return err;
+ return 0;
+}
+
+static int max1720x_set_property(struct power_supply *psy,
+ enum power_supply_property psp,
+ const union power_supply_propval *val)
+{
+ struct max1720x_chip *chip = power_supply_get_drvdata(psy);
+ int rc = 0;
+
+ pm_runtime_get_sync(chip->dev);
+ if (!chip->init_complete || !chip->resume_complete) {
+ pm_runtime_put_sync(chip->dev);
+ return -EAGAIN;
+ }
+ pm_runtime_put_sync(chip->dev);
+
+ switch (psp) {
+ case POWER_SUPPLY_PROP_CAPACITY:
+ rc = max1720x_set_battery_soc(chip, val);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int max1720x_property_is_writeable(struct power_supply *psy,
+ enum power_supply_property psp)
+{
+ switch (psp) {
+ case POWER_SUPPLY_PROP_CAPACITY:
+ return 1;
+ default:
+ break;
+ }
+
+ return 0;
+}
+
+/*
+ * A fuel gauge reset resets only the fuel gauge operation without resetting IC
+ * hardware. This is useful for testing different configurations without writing
+ * nonvolatile memory.
+ */
+static void max1720x_fg_reset(struct max1720x_chip *chip)
+{
+ REGMAP_WRITE(chip->regmap, MAX1720X_CONFIG2,
+ MAX1720X_COMMAND_FUEL_GAUGE_RESET);
+ msleep(MAX1720X_TPOR_MS);
+}
+
+static irqreturn_t max1720x_fg_irq_thread_fn(int irq, void *obj)
+{
+ struct max1720x_chip *chip = obj;
+ u16 fg_status, fg_status_clr;
+ int err = 0;
+
+ if (!chip || irq != chip->primary->irq) {
+ WARN_ON_ONCE(1);
+ return IRQ_NONE;
+ }
+
+ pm_runtime_get_sync(chip->dev);
+ if (!chip->init_complete || !chip->resume_complete) {
+ pm_runtime_put_sync(chip->dev);
+ return -EAGAIN;
+ }
+ pm_runtime_put_sync(chip->dev);
+ err = REGMAP_READ(chip->regmap, MAX1720X_STATUS, &fg_status);
+ if (err)
+ return IRQ_NONE;
+ if (fg_status == 0) {
+ chip->debug_irq_none_cnt++;
+ pr_debug("spurius: fg_status=0 cnt=%d\n",
+ chip->debug_irq_none_cnt);
+ /* rate limit spurius interrupts */
+ msleep(MAX1720X_TICLR_MS);
+ return IRQ_HANDLED;
+ }
+
+ /* only used to report health */
+ chip->health_status |= fg_status;
+ /* write 0 to clear will loose interrupts when we don't write 1 to the
+ * bits that are not set. Just inverting fg_status cause an interrupt
+ * storm, only setting the bits marked as "host must clear" in the DS
+ * seems to work eg:
+ *
+ * fg_status_clr = fg_status
+ * fg_status_clr |= MAX1720X_STATUS_POR | MAX1720X_STATUS_DSOCI
+ * | MAX1720X_STATUS_BI;
+ *
+ * If the above logic is sound, we probably need to set also the bits
+ * that config mark as "host must clear". Maxim to confirm.
+ */
+ fg_status_clr = fg_status;
+
+ if (fg_status & MAX1720X_STATUS_POR) {
+ fg_status_clr &= ~MAX1720X_STATUS_POR;
+ pr_debug("POR is set\n");
+ }
+ if (fg_status & MAX1720X_STATUS_IMN)
+ pr_debug("IMN is set\n");
+
+ if (fg_status & MAX1720X_STATUS_BST)
+ pr_debug("BST is set\n");
+
+ if (fg_status & MAX1720X_STATUS_IMX)
+ pr_debug("IMX is set\n");
+
+ if (fg_status & MAX1720X_STATUS_DSOCI) {
+ fg_status_clr &= ~MAX1720X_STATUS_DSOCI;
+ pr_debug("DSOCI is set\n");
+ schedule_work(&chip->cycle_count_work);
+ }
+ if (fg_status & MAX1720X_STATUS_VMN) {
+ if (chip->RConfig & MAX1720X_CONFIG_VS)
+ fg_status_clr &= ~MAX1720X_STATUS_VMN;
+ pr_debug("VMN is set\n");
+ }
+ if (fg_status & MAX1720X_STATUS_TMN) {
+ if (chip->RConfig & MAX1720X_CONFIG_TS)
+ fg_status_clr &= ~MAX1720X_STATUS_TMN;
+ pr_debug("TMN is set\n");
+ }
+ if (fg_status & MAX1720X_STATUS_SMN) {
+ if (chip->RConfig & MAX1720X_CONFIG_SS)
+ fg_status_clr &= ~MAX1720X_STATUS_SMN;
+ pr_debug("SMN is set\n");
+ }
+ if (fg_status & MAX1720X_STATUS_BI)
+ pr_debug("BI is set\n");
+
+ if (fg_status & MAX1720X_STATUS_VMX) {
+ if (chip->RConfig & MAX1720X_CONFIG_VS)
+ fg_status_clr &= ~MAX1720X_STATUS_VMX;
+ pr_debug("VMX is set\n");
+ }
+ if (fg_status & MAX1720X_STATUS_TMX) {
+ if (chip->RConfig & MAX1720X_CONFIG_TS)
+ fg_status_clr &= ~MAX1720X_STATUS_TMX;
+ pr_debug("TMX is set\n");
+ }
+ if (fg_status & MAX1720X_STATUS_SMX) {
+ if (chip->RConfig & MAX1720X_CONFIG_SS)
+ fg_status_clr &= ~MAX1720X_STATUS_SMX;
+ pr_debug("SMX is set\n");
+ }
+
+ if (fg_status & MAX1720X_STATUS_BR)
+ pr_debug("BR is set\n");
+
+ REGMAP_WRITE(chip->regmap, MAX1720X_STATUS, fg_status_clr);
+
+ if (chip->psy)
+ power_supply_changed(chip->psy);
+
+ /* oneshot w/o filter will unmask on return but gauge will take up
+ * to 351 ms to clear ALRM1.
+ */
+ msleep(MAX1720X_TICLR_MS);
+
+ return IRQ_HANDLED;
+}
+
+
+static int max1720x_handle_dt_batt_id(struct max1720x_chip *chip)
+{
+ int ret, batt_id;
+ u32 batt_id_range = 20, batt_id_kohm;
+ const char *dt_iio_ch_name, *iio_ch_name;
+ struct device_node *node = chip->dev->of_node;
+ struct device_node *config_node, *child_node;
+
+ ret = of_property_read_string(node, "io-channel-names",
+ &dt_iio_ch_name);
+ if (ret == -EINVAL)
+ return 0;
+
+ if (ret) {
+ dev_warn(chip->dev, "failed to read io-channel-names\n");
+ /* Don't fail probe on that, just ignore error */
+ return 0;
+ }
+
+ iio_ch_name = devm_kstrdup(chip->dev, dt_iio_ch_name, GFP_KERNEL);
+ if (!iio_ch_name)
+ /* Don't fail probe on that, just ignore error */
+ return 0;
+
+ chip->iio_ch = iio_channel_get(chip->dev, iio_ch_name);
+ if (PTR_ERR(chip->iio_ch) == -EPROBE_DEFER) {
+ dev_warn(chip->dev, "iio_channel_get %s not ready\n",
+ iio_ch_name);
+ devm_kfree(chip->dev, (void *) iio_ch_name);
+ return -EPROBE_DEFER;
+ } else if (IS_ERR(chip->iio_ch)) {
+ dev_warn(chip->dev, "iio_channel_get %s error: %ld\n",
+ iio_ch_name, PTR_ERR(chip->iio_ch));
+ /* Don't fail probe on that, just ignore error */
+ devm_kfree(chip->dev, (void *) iio_ch_name);
+ return 0;
+ }
+
+ devm_kfree(chip->dev, (void *) iio_ch_name);
+ ret = iio_read_channel_processed(chip->iio_ch, &batt_id);
+ if (ret < 0) {
+ dev_warn(chip->dev, "Failed to read battery id: %d\n", ret);
+ return 0;
+ }
+
+ batt_id /= 1000;
+ dev_info(chip->dev, "device battery RID: %d\n", batt_id);
+
+ ret = of_property_read_u32(node, "maxim,batt-id-range-pct",
+ &batt_id_range);
+ if (ret && ret == -EINVAL)
+ dev_warn(chip->dev, "failed to read maxim,batt-id-range-pct\n");
+
+ config_node = of_find_node_by_name(node, "maxim,config");
+ if (!config_node) {
+ dev_warn(chip->dev, "Failed to find maxim,config setting\n");
+ return 0;
+ }
+
+ for_each_child_of_node(config_node, child_node) {
+ ret = of_property_read_u32(child_node, "maxim,batt-id-kohm",
+ &batt_id_kohm);
+ if (!ret &&
+ (batt_id < (batt_id_kohm * (100 + batt_id_range) / 100)) &&
+ (batt_id > (batt_id_kohm * (100 - batt_id_range) / 100))) {
+ chip->batt_node = child_node;
+ break;
+ }
+ }
+ if (!chip->batt_node)
+ dev_warn(chip->dev, "No child node found matching ID\n");
+
+ return 0;
+}
+
+static int max1720x_apply_regval_shadow(struct max1720x_chip *chip,
+ struct device_node *node,
+ u16 *nRAM, int nb)
+{
+ int ret, idx;
+ u16 *regs;
+
+ if (!node || nb <= 0)
+ return 0;
+
+ if (nb & 1) {
+ dev_warn(chip->dev, "%s maxim,n_regval u16 elems count is not even: %d\n",
+ node->name, nb);
+ return -EINVAL;
+ }
+
+ regs = kmalloc_array(nb, sizeof(u16), GFP_KERNEL);
+ if (!regs)
+ return -ENOMEM;
+
+ ret = of_property_read_u16_array(node, "maxim,n_regval", regs, nb);
+ if (ret) {
+ dev_warn(chip->dev, "failed to read maxim,n_regval: %d\n", ret);
+ goto shadow_out;
+ }
+
+ for (idx = 0; idx < nb; idx += 2) {
+ if ((regs[idx] >= MAX1720X_NVRAM_START) &&
+ (regs[idx] < MAX1720X_NVRAM_END)) {
+ nRAM[regs[idx] - MAX1720X_NVRAM_START] = regs[idx + 1];
+ }
+ }
+shadow_out:
+ kfree(regs);
+ return ret;
+}
+
+static int max1720x_handle_dt_shadow_config(struct max1720x_chip *chip)
+{
+ int ret = 0;
+ u16 *nRAM_current, *nRAM_updated;
+ int batt_cnt = 0, glob_cnt;
+
+ ret = max1720x_handle_dt_batt_id(chip);
+ if (ret)
+ return ret;
+
+ if (chip->batt_node)
+ batt_cnt = of_property_count_elems_of_size(chip->batt_node,
+ "maxim,n_regval",
+ sizeof(u16));
+ glob_cnt = of_property_count_elems_of_size(chip->dev->of_node,
+ "maxim,n_regval",
+ sizeof(u16));
+
+ nRAM_current = kmalloc_array(MAX1720X_NVRAM_U16_SIZE,
+ sizeof(u16), GFP_KERNEL);
+ if (!nRAM_current)
+ return -ENOMEM;
+
+ nRAM_updated = kmalloc_array(MAX1720X_NVRAM_U16_SIZE,
+ sizeof(u16), GFP_KERNEL);
+ if (!nRAM_updated) {
+ ret = -ENOMEM;
+ goto error_out;
+ }
+
+ ret = regmap_raw_read(chip->regmap_nvram, MAX1720X_NVRAM_START,
+ nRAM_current, MAX1720X_NVRAM_SIZE);
+ if (ret) {
+ dev_err(chip->dev,
+ "Failed to read config from shadow RAM\n");
+ goto error_out;
+ }
+ memcpy(nRAM_updated, nRAM_current, MAX1720X_NVRAM_SIZE);
+ if (chip->batt_node)
+ max1720x_apply_regval_shadow(chip, chip->batt_node,
+ nRAM_updated, batt_cnt);
+ max1720x_apply_regval_shadow(chip, chip->dev->of_node,
+ nRAM_updated, glob_cnt);
+
+ /* Ensure nCGain is not 0 if nNVCfg0.enCG is set */
+ if ((nRAM_updated[NVRAM_U16_INDEX(MAX1720X_NNVCFG0)] &
+ MAX1720X_NNVCFG0_ENCG) &&
+ ((nRAM_updated[NVRAM_U16_INDEX(MAX1720X_NCGAIN)] == 0) ||
+ (nRAM_updated[NVRAM_U16_INDEX(MAX1720X_NCGAIN)] == 0x0400)))
+ nRAM_updated[NVRAM_U16_INDEX(MAX1720X_NCGAIN)] = 0x4000;
+
+ if (memcmp(nRAM_updated, nRAM_current, MAX1720X_NVRAM_SIZE)) {
+ ret = regmap_raw_write(chip->regmap_nvram, MAX1720X_NVRAM_START,
+ nRAM_updated, MAX1720X_NVRAM_SIZE);
+ if (ret) {
+ dev_err(chip->dev,
+ "Failed to write config from shadow RAM\n");
+ goto error_out;
+ }
+ /* nConvgCfg change take effect without resetting the gauge */
+ nRAM_current[NVRAM_U16_INDEX(MAX1720X_NCONVGCFG)] =
+ nRAM_updated[NVRAM_U16_INDEX(MAX1720X_NCONVGCFG)];
+ if (memcmp(nRAM_updated, nRAM_current, MAX1720X_NVRAM_SIZE)) {
+ dev_info(chip->dev,
+ "DT config differs from shadow, resetting\n");
+ max1720x_fg_reset(chip);
+ }
+ }
+
+error_out:
+ kfree(nRAM_current);
+ kfree(nRAM_updated);
+
+ return ret;
+}
+
+static int max1720x_apply_regval_register(struct max1720x_chip *chip,
+ struct device_node *node)
+{
+ int cnt, ret = 0, idx, err;
+ u16 *regs, data;
+
+ cnt = of_property_count_elems_of_size(node, "maxim,r_regval",
+ sizeof(u16));
+ if (!node || cnt <= 0)
+ return 0;
+
+ if (cnt & 1) {
+ dev_warn(chip->dev, "%s maxim,r_regval u16 elems count is not even: %d\n",
+ node->name, cnt);
+ return -EINVAL;
+ }
+
+ regs = kmalloc_array(cnt, sizeof(u16), GFP_KERNEL);
+ if (!regs)
+ return -ENOMEM;
+
+ ret = of_property_read_u16_array(node, "maxim,r_regval", regs, cnt);
+ if (ret) {
+ dev_warn(chip->dev, "failed to read %s maxim,r_regval: %d\n",
+ node->name, ret);
+ goto register_out;
+ }
+
+ for (idx = 0; idx < cnt; idx += 2) {
+ if (max1720x_is_reg(chip->dev, regs[idx])) {
+ err = REGMAP_READ(chip->regmap, regs[idx], &data);
+ if (!err && data != regs[idx + 1])
+ REGMAP_WRITE(chip->regmap, regs[idx],
+ regs[idx + 1]);
+ }
+ }
+register_out:
+ kfree(regs);
+ return ret;
+}
+
+static int max1720x_handle_dt_register_config(struct max1720x_chip *chip)
+{
+ int ret = 0;
+
+ if (chip->batt_node)
+ ret = max1720x_apply_regval_register(chip, chip->batt_node);
+
+ if (ret)
+ return ret;
+
+ ret = max1720x_apply_regval_register(chip, chip->dev->of_node);
+
+ return ret;
+}
+
+static int max1720x_handle_dt_nconvgcfg(struct max1720x_chip *chip)
+{
+ int ret = 0, i;
+ struct device_node *node = chip->dev->of_node;
+
+ chip->curr_convgcfg_idx = -1;
+ mutex_init(&chip->convgcfg_lock);
+ chip->nb_convgcfg =
+ of_property_count_elems_of_size(node, "maxim,nconvgcfg-temp-limits",
+ sizeof(s16));
+ if (!chip->nb_convgcfg)
+ return 0;
+
+ ret = of_property_read_u32(node, "maxim,nconvgcfg-temp-hysteresis",
+ &chip->convgcfg_hysteresis);
+ if (ret < 0)
+ chip->convgcfg_hysteresis = 10;
+
+ if (chip->nb_convgcfg != of_property_count_elems_of_size(node,
+ "maxim,nconvgcfg-values",
+ sizeof(u16))) {
+ dev_warn(chip->dev, "%s maxim,nconvgcfg-values and maxim,nconvgcfg-temp-limits are missmatching number of elements\n",
+ node->name);
+ return -EINVAL;
+ }
+ chip->temp_convgcfg = devm_kmalloc_array(chip->dev, chip->nb_convgcfg,
+ sizeof(s16), GFP_KERNEL);
+ if (!chip->temp_convgcfg)
+ return -ENOMEM;
+
+ chip->convgcfg_values = devm_kmalloc_array(chip->dev, chip->nb_convgcfg,
+ sizeof(u16), GFP_KERNEL);
+ if (!chip->convgcfg_values) {
+ devm_kfree(chip->dev, chip->temp_convgcfg);
+ chip->temp_convgcfg = NULL;
+ return -ENOMEM;
+ }
+
+ ret = of_property_read_u16_array(node, "maxim,nconvgcfg-temp-limits",
+ (u16 *) chip->temp_convgcfg,
+ chip->nb_convgcfg);
+ if (ret) {
+ dev_warn(chip->dev, "failed to read maxim,nconvgcfg-temp-limits: %d\n",
+ ret);
+ goto error;
+ }
+
+ ret = of_property_read_u16_array(node, "maxim,nconvgcfg-values",
+ chip->convgcfg_values,
+ chip->nb_convgcfg);
+ if (ret) {
+ dev_warn(chip->dev, "failed to read maxim,nconvgcfg-values: %d\n",
+ ret);
+ goto error;
+ }
+ for (i = 1; i < chip->nb_convgcfg; i++) {
+ if (chip->temp_convgcfg[i] < chip->temp_convgcfg[i-1]) {
+ dev_warn(chip->dev, "nconvgcfg-temp-limits idx:%d < idx:%d\n",
+ i, i-1);
+ goto error;
+ }
+ if ((chip->temp_convgcfg[i] - chip->temp_convgcfg[i-1])
+ <= chip->convgcfg_hysteresis) {
+ dev_warn(chip->dev, "nconvgcfg-temp-hysteresis smaller than idx:%d, idx:%d\n",
+ i, i-1);
+ goto error;
+ }
+ }
+
+error:
+ if (ret) {
+ devm_kfree(chip->dev, chip->temp_convgcfg);
+ devm_kfree(chip->dev, chip->convgcfg_values);
+ chip->temp_convgcfg = NULL;
+ }
+
+ return ret;
+}
+
+#ifdef CONFIG_DEBUG_FS
+static int get_irq_none_cnt(void *data, u64 *val)
+{
+ struct max1720x_chip *chip = (struct max1720x_chip *)data;
+
+ *val = chip->debug_irq_none_cnt;
+ return 0;
+}
+
+static int set_irq_none_cnt(void *data, u64 val)
+{
+ struct max1720x_chip *chip = (struct max1720x_chip *)data;
+
+ if (val == 0)
+ chip->debug_irq_none_cnt = 0;
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(irq_none_cnt_fops, get_irq_none_cnt,
+ set_irq_none_cnt, "%llu\n");
+#endif
+
+static int init_debugfs(struct max1720x_chip *chip)
+{
+#ifdef CONFIG_DEBUG_FS
+ struct dentry *de;
+
+ de = debugfs_create_dir("max1720x", 0);
+ if (de)
+ debugfs_create_file("irq_none_cnt", 0644, de,
+ chip, &irq_none_cnt_fops);
+#endif
+ return 0;
+}
+
+static int max1720x_init_chip(struct max1720x_chip *chip)
+{
+ u16 data = 0;
+ int ret;
+
+ ret = max1720x_handle_dt_shadow_config(chip);
+ if (ret == -EPROBE_DEFER)
+ return ret;
+
+ ret = max1720x_handle_dt_register_config(chip);
+ if (ret == -EPROBE_DEFER)
+ return ret;
+
+ (void) max1720x_handle_dt_nconvgcfg(chip);
+
+ chip->fake_capacity = -EINVAL;
+ chip->init_complete = true;
+ chip->resume_complete = true;
+
+ ret = REGMAP_READ(chip->regmap, MAX1720X_STATUS, &data);
+ if (!ret && data & MAX1720X_STATUS_BR) {
+ dev_info(chip->dev, "Clearing Battery Removal bit\n");
+ regmap_update_bits(chip->regmap, MAX1720X_STATUS,
+ MAX1720X_STATUS_BR, 0x0);
+ }
+ if (!ret && data & MAX1720X_STATUS_BI) {
+ dev_info(chip->dev, "Clearing Battery Insertion bit\n");
+ regmap_update_bits(chip->regmap, MAX1720X_STATUS,
+ MAX1720X_STATUS_BI, 0x0);
+ }
+ if (!ret && data & MAX1720X_STATUS_POR) {
+ dev_info(chip->dev, "Clearing Power-On Reset bit\n");
+ regmap_update_bits(chip->regmap, MAX1720X_STATUS,
+ MAX1720X_STATUS_POR, 0x0);
+ }
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NRSENSE, &chip->RSense);
+ dev_info(chip->dev, "RSense value %d micro Ohm\n", chip->RSense * 10);
+ (void) REGMAP_READ(chip->regmap, MAX1720X_CONFIG, &chip->RConfig);
+ dev_info(chip->dev, "Config: 0x%04x\n", chip->RConfig);
+ (void) REGMAP_READ(chip->regmap, MAX1720X_ICHGTERM, &data);
+ dev_info(chip->dev, "IChgTerm: %d\n",
+ reg_to_micro_amp(data, chip->RSense));
+ (void) REGMAP_READ(chip->regmap, MAX1720X_VEMPTY, &data);
+ dev_info(chip->dev, "VEmpty: VE=%dmV VR=%dmV\n",
+ ((data >> 7) & 0x1ff) * 10, (data & 0x7f) * 40);
+
+ /*
+ * Capacity data is stored as complement so it will not be zero. Using
+ * zero case to detect new un-primed pack
+ */
+ ret = REGMAP_READ(chip->regmap_nvram, MAX1720X_NUSER18C, &data);
+ if (!ret && data == 0)
+ max1720x_prime_battery_qh_capacity(chip,
+ POWER_SUPPLY_STATUS_UNKNOWN);
+ else
+ max1720x_restore_battery_qh_capacity(chip);
+
+ chip->prev_charge_state = POWER_SUPPLY_STATUS_UNKNOWN;
+
+ init_debugfs(chip);
+
+ /* Handle any IRQ that might have been set before init */
+ max1720x_fg_irq_thread_fn(chip->primary->irq, chip);
+
+ return 0;
+}
+
+static void max1720x_set_serial_number(struct max1720x_chip *chip)
+{
+ u16 data0 = 0, data1 = 0, data2 = 0;
+ int date, count = 0, shift, err = 0;
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NMANFCTRNAME0, &data0);
+ if (data0 == 0x5357) /* "SW": SWD */
+ shift = 0;
+ else if (data0 == 0x4257) /* "BW": DSY */
+ shift = 8;
+ else
+ return;
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NSERIALNUMBER0, &data0);
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NSERIALNUMBER1, &data1);
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NSERIALNUMBER2, &data2);
+ count += scnprintf(chip->serial_number + count,
+ sizeof(chip->serial_number) - count,
+ "%02X%02X%02X",
+ data0 >> shift, data1 >> shift, data2 >> shift);
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NMANFCTRDATE, &data0);
+ date = (((((data0 >> 9) & 0x3f) + 1980) * 10000) +
+ ((data0 >> 5) & 0xf) * 100 + (data0 & 0x1F));
+ count += scnprintf(chip->serial_number + count,
+ sizeof(chip->serial_number) - count,
+ "%d", date);
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NMANFCTRNAME0, &data0);
+ count += scnprintf(chip->serial_number + count,
+ sizeof(chip->serial_number) - count,
+ "%c%c", data0 >> 8, data0 & 0xFF);
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NDEVICENAME0, &data0);
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NDEVICENAME1, &data1);
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NDEVICENAME2, &data2);
+ count += scnprintf(chip->serial_number + count,
+ sizeof(chip->serial_number) - count,
+ "%c%c%c",
+ data0 >> shift, data1 >> shift, data2 >> shift);
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NDEVICENAME3, &data0);
+ if (data0 >> 8 == 0)
+ data0 = ('?' << 8) | (data0 & 0xFF);
+ if ((data0 & 0xFF) == 0)
+ data0 = (data0 & 0xFF00) | '?';
+ count += scnprintf(chip->serial_number + count,
+ sizeof(chip->serial_number) - count,
+ "%c%c", data0 >> 8, data0 & 0xFF);
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NUSER1D1, &data0);
+ count += scnprintf(chip->serial_number + count,
+ sizeof(chip->serial_number) - count,
+ "%c", data0 >> 8);
+
+ (void) REGMAP_READ(chip->regmap_nvram, MAX1720X_NUSER1D0, &data0);
+ if (shift == 8) {
+ if (data0 >> 8 == 0xb1) {
+ count += scnprintf(chip->serial_number + count,
+ sizeof(chip->serial_number) - count,
+ "B1");
+ } else if (data0 >> 8 == 0xc1) {
+ count += scnprintf(chip->serial_number + count,
+ sizeof(chip->serial_number) - count,
+ "C1");
+ } else {
+ count += scnprintf(chip->serial_number + count,
+ sizeof(chip->serial_number) - count,
+ "??");
+ }
+ } else {
+ count += scnprintf(chip->serial_number + count,
+ sizeof(chip->serial_number) - count,
+ "%c%c", data0 >> 8, data0 & 0xFF);
+ }
+ if (err)
+ chip->serial_number[0] = '\0';
+}
+
+static struct power_supply_desc max1720x_psy_desc = {
+ .name = "maxfg",
+ .type = POWER_SUPPLY_TYPE_BATTERY,
+ .get_property = max1720x_get_property,
+ .set_property = max1720x_set_property,
+ .property_is_writeable = max1720x_property_is_writeable,
+ .properties = max1720x_battery_props,
+ .num_properties = ARRAY_SIZE(max1720x_battery_props),
+};
+
+
+static void *ct_seq_start(struct seq_file *s, loff_t *pos)
+{
+ struct max1720x_history *hi =
+ (struct max1720x_history *)s->private;
+
+ if (*pos >= hi->history_count)
+ return NULL;
+ hi->history_index = *pos;
+
+ return &hi->history_index;
+}
+
+static void *ct_seq_next(struct seq_file *s, void *v, loff_t *pos)
+{
+ loff_t *spos = (loff_t *)v;
+ struct max1720x_history *hi =
+ (struct max1720x_history *)s->private;
+
+ *pos = ++*spos;
+ if (*pos >= hi->history_count)
+ return NULL;
+
+ return spos;
+}
+
+static void ct_seq_stop(struct seq_file *s, void *v)
+{
+ /* iterator in hi, no need to free */
+}
+
+static int ct_seq_show(struct seq_file *s, void *v)
+{
+ char temp[96];
+ loff_t *spos = (loff_t *)v;
+ struct max1720x_history *hi =
+ (struct max1720x_history *)s->private;
+ const size_t offset = *spos * MAX1720X_HISTORY_PAGE_SIZE;
+
+ format_battery_history_entry(temp, sizeof(temp), &hi->history[offset]);
+ seq_printf(s, "%s\n", temp);
+
+ return 0;
+}
+
+static const struct seq_operations ct_seq_ops = {
+ .start = ct_seq_start,
+ .next = ct_seq_next,
+ .stop = ct_seq_stop,
+ .show = ct_seq_show
+};
+
+static int history_dev_open(struct inode *inode, struct file *file)
+{
+ struct max1720x_chip *chip =
+ container_of(inode->i_cdev, struct max1720x_chip, hcdev);
+ struct max1720x_history *hi;
+ int history_count;
+
+ hi = __seq_open_private(file, &ct_seq_ops, sizeof(*hi));
+ if (!hi)
+ return -ENOMEM;
+
+ history_count = max1720x_history_read(chip, hi);
+ if (history_count < 0) {
+ return history_count;
+ } else if (history_count == 0) {
+ dev_info(chip->dev,
+ "No battery history has been recorded\n");
+ }
+
+ return 0;
+}
+
+static int history_dev_release(struct inode *inode, struct file *file)
+{
+ struct max1720x_history *hi =
+ ((struct seq_file *)file->private_data)->private;
+
+ if (hi) {
+ max1720x_history_free(hi);
+ seq_release_private(inode, file);
+ }
+
+ return 0;
+}
+
+static const struct file_operations hdev_fops = {
+ .open = history_dev_open,
+ .owner = THIS_MODULE,
+ .read = seq_read,
+ .release = history_dev_release,
+};
+
+static void max1720x_cleanup_history(struct max1720x_chip *chip)
+{
+ if (chip->history_added)
+ cdev_del(&chip->hcdev);
+ if (chip->history_available)
+ device_destroy(chip->hcclass, chip->hcmajor);
+ if (chip->hcclass)
+ class_destroy(chip->hcclass);
+ if (chip->hcmajor != -1)
+ unregister_chrdev_region(chip->hcmajor, 1);
+}
+
+static int max1720x_init_history(struct max1720x_chip *chip)
+{
+ struct device *hcdev;
+
+ mutex_init(&chip->history_lock);
+
+ chip->hcmajor = -1;
+
+ /* cat /proc/devices */
+ if (alloc_chrdev_region(&chip->hcmajor, 0, 1, HISTORY_DEVICENAME) < 0)
+ goto no_history;
+ /* ls /sys/class */
+ chip->hcclass = class_create(THIS_MODULE, HISTORY_DEVICENAME);
+ if (chip->hcclass == NULL)
+ goto no_history;
+ /* ls /dev/ */
+ hcdev = device_create(chip->hcclass, NULL, chip->hcmajor, NULL,
+ HISTORY_DEVICENAME);
+ if (hcdev == NULL)
+ goto no_history;
+
+ chip->history_available = true;
+ cdev_init(&chip->hcdev, &hdev_fops);
+ if (cdev_add(&chip->hcdev, chip->hcmajor, 1) == -1)
+ goto no_history;
+
+ chip->history_added = true;
+ return 0;
+
+no_history:
+ max1720x_cleanup_history(chip);
+ return -ENODEV;
+}
+
+static void max1720x_init_work(struct work_struct *work)
+{
+ struct max1720x_chip *chip = container_of(work, struct max1720x_chip,
+ init_work.work);
+ int ret = max1720x_init_chip(chip);
+
+ if (ret == -EPROBE_DEFER) {
+ schedule_delayed_work(&chip->init_work,
+ msecs_to_jiffies(MAX1720X_DELAY_INIT_MS));
+ return;
+ }
+
+ (void)max1720x_init_history(chip);
+}
+
+static int max1720x_probe(struct i2c_client *client,
+ const struct i2c_device_id *id)
+{
+ struct max1720x_chip *chip;
+ struct device *dev = &client->dev;
+ struct power_supply_config psy_cfg = { };
+
+ int ret = 0;
+
+ chip = devm_kzalloc(dev, sizeof(*chip), GFP_KERNEL);
+ if (!chip)
+ return -ENOMEM;
+
+ chip->dev = dev;
+ i2c_set_clientdata(client, chip);
+ chip->primary = client;
+ chip->secondary = i2c_new_secondary_device(client, "nvram", 0xb);
+ if (chip->secondary == NULL) {
+ dev_err(dev, "Failed to initialize secondary i2c device\n");
+ return -EINVAL;
+ }
+ i2c_set_clientdata(chip->secondary, chip);
+ chip->regmap = devm_regmap_init_i2c(client, &max1720x_regmap_cfg);
+ if (IS_ERR(chip->regmap)) {
+ dev_err(dev, "Failed to initialize regmap\n");
+ ret = -EINVAL;
+ goto i2c_unregister;
+ }
+
+ chip->regmap_nvram =
+ devm_regmap_init_i2c(chip->secondary, &max1720x_regmap_nvram_cfg);
+ if (IS_ERR(chip->regmap_nvram)) {
+ dev_err(dev, "Failed to initialize nvram regmap\n");
+ ret = -EINVAL;
+ goto i2c_unregister;
+ }
+
+ mutex_init(&chip->cyc_ctr.lock);
+ max1720x_restore_cycle_counter(chip);
+
+ if (chip->primary->irq) {
+ ret = request_threaded_irq(chip->primary->irq, NULL,
+ max1720x_fg_irq_thread_fn,
+ IRQF_TRIGGER_LOW | IRQF_ONESHOT,
+ MAX1720X_I2C_DRIVER_NAME, chip);
+ if (ret != 0) {
+ dev_err(chip->dev, "Unable to register IRQ handler\n");
+ goto i2c_unregister;
+ }
+ enable_irq_wake(chip->primary->irq);
+ }
+
+ if (of_property_read_bool(dev->of_node, "maxim,psy-type-unknown"))
+ max1720x_psy_desc.type = POWER_SUPPLY_TYPE_UNKNOWN;
+
+ psy_cfg.drv_data = chip;
+ chip->psy = devm_power_supply_register(dev,
+ &max1720x_psy_desc, &psy_cfg);
+ if (IS_ERR(chip->psy)) {
+ dev_err(dev, "Couldn't register as power supply\n");
+ ret = PTR_ERR(chip->psy);
+ goto irq_unregister;
+ }
+
+ ret = device_create_file(&chip->psy->dev, &dev_attr_cycle_counts_bins);
+ if (ret) {
+ dev_err(dev, "Failed to create cycle_counts_bins attribute\n");
+ goto psy_unregister;
+ }
+
+ ret = device_create_file(&chip->psy->dev, &dev_attr_offmode_charger);
+ if (ret) {
+ dev_err(dev, "Failed to create offmode_charger attribute\n");
+ goto psy_unregister;
+ }
+
+ max1720x_set_serial_number(chip);
+
+ INIT_WORK(&chip->cycle_count_work, max1720x_cycle_count_work);
+ INIT_DELAYED_WORK(&chip->init_work, max1720x_init_work);
+ schedule_delayed_work(&chip->init_work,
+ msecs_to_jiffies(MAX1720X_DELAY_INIT_MS));
+
+ return 0;
+
+psy_unregister:
+ power_supply_unregister(chip->psy);
+irq_unregister:
+ free_irq(chip->primary->irq, chip);
+i2c_unregister:
+ i2c_unregister_device(chip->secondary);
+
+ return ret;
+}
+
+static int max1720x_remove(struct i2c_client *client)
+{
+ struct max1720x_chip *chip = i2c_get_clientdata(client);
+
+ max1720x_cleanup_history(chip);
+ cancel_delayed_work(&chip->init_work);
+ iio_channel_release(chip->iio_ch);
+ if (chip->primary->irq)
+ free_irq(chip->primary->irq, chip);
+ power_supply_unregister(chip->psy);
+ i2c_unregister_device(chip->secondary);
+
+ return 0;
+}
+
+static const struct of_device_id max1720x_of_match[] = {
+ { .compatible = "maxim,max1720x"},
+ {},
+};
+MODULE_DEVICE_TABLE(of, max1720x_of_match);
+
+static const struct i2c_device_id max1720x_id[] = {
+ {"max1720x", 0},
+ {}
+};
+MODULE_DEVICE_TABLE(i2c, max1720x_id);
+
+#ifdef CONFIG_PM_SLEEP
+static int max1720x_pm_suspend(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct max1720x_chip *chip = i2c_get_clientdata(client);
+
+ pm_runtime_get_sync(chip->dev);
+ chip->resume_complete = false;
+ pm_runtime_put_sync(chip->dev);
+
+ return 0;
+}
+
+static int max1720x_pm_resume(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct max1720x_chip *chip = i2c_get_clientdata(client);
+
+ pm_runtime_get_sync(chip->dev);
+ chip->resume_complete = true;
+ pm_runtime_put_sync(chip->dev);
+
+ return 0;
+}
+#endif
+static const struct dev_pm_ops max1720x_pm_ops = {
+ SET_LATE_SYSTEM_SLEEP_PM_OPS(max1720x_pm_suspend, max1720x_pm_resume)
+};
+
+static struct i2c_driver max1720x_i2c_driver = {
+ .driver = {
+ .name = "max1720x",
+ .of_match_table = max1720x_of_match,
+ .pm = &max1720x_pm_ops,
+ .probe_type = PROBE_PREFER_ASYNCHRONOUS,
+ },
+ .id_table = max1720x_id,
+ .probe = max1720x_probe,
+ .remove = max1720x_remove,
+};
+
+module_i2c_driver(max1720x_i2c_driver);
+MODULE_AUTHOR("Thierry Strudel <tstrudel@google.com>");
+MODULE_DESCRIPTION("MAX17201/MAX17205 Fuel Gauge");
+MODULE_LICENSE("GPL");
diff --git a/drivers/power/supply/olpc_battery.c b/drivers/power/supply/olpc_battery.c
index 9e29b13..1578386 100644
--- a/drivers/power/supply/olpc_battery.c
+++ b/drivers/power/supply/olpc_battery.c
@@ -427,14 +427,14 @@ static int olpc_bat_get_property(struct power_supply *psy,
if (ret)
return ret;
- val->intval = (s16)be16_to_cpu(ec_word) * 100 / 256;
+ val->intval = (s16)be16_to_cpu(ec_word) * 10 / 256;
break;
case POWER_SUPPLY_PROP_TEMP_AMBIENT:
ret = olpc_ec_cmd(EC_AMB_TEMP, NULL, 0, (void *)&ec_word, 2);
if (ret)
return ret;
- val->intval = (int)be16_to_cpu(ec_word) * 100 / 256;
+ val->intval = (int)be16_to_cpu(ec_word) * 10 / 256;
break;
case POWER_SUPPLY_PROP_CHARGE_COUNTER:
ret = olpc_ec_cmd(EC_BAT_ACR, NULL, 0, (void *)&ec_word, 2);
diff --git a/drivers/power/supply/overheat_mitigation.c b/drivers/power/supply/overheat_mitigation.c
new file mode 100644
index 0000000..39bd1e7
--- /dev/null
+++ b/drivers/power/supply/overheat_mitigation.c
@@ -0,0 +1,482 @@
+/*
+ * Copyright 2018 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/delay.h>
+#include <linux/platform_device.h>
+#include <linux/pmic-voter.h>
+#include <linux/power_supply.h>
+#include <linux/printk.h>
+#include <linux/thermal.h>
+#include <linux/time.h>
+#include <linux/pm_wakeup.h>
+
+#define USB_OVERHEAT_MITIGATION_VOTER "USB_OVERHEAT_MITIGATION_VOTER"
+
+static int fake_port_temp = -1;
+module_param_named(
+ fake_port_temp, fake_port_temp, int, 0600
+);
+
+static bool enable = true;
+module_param_named(
+ enable, enable, bool, 0600
+);
+
+struct overheat_info {
+ struct device *dev;
+ struct power_supply *usb_psy;
+ struct votable *usb_icl_votable;
+ struct votable *disable_power_role_switch;
+ struct notifier_block psy_nb;
+ struct delayed_work port_overheat_work;
+ struct wakeup_source overheat_ws;
+
+ bool usb_connected;
+ bool usb_online;
+ bool accessory_connected;
+ bool usb_replug;
+ bool overheat_mitigation;
+ bool overheat_work_running;
+
+ int begin_temp;
+ int clear_temp;
+ int overheat_work_delay_ms;
+ int polling_freq;
+ int monitor_accessory_s;
+ time_t accessory_connect_time;
+ int check_status;
+};
+
+#define PSY_GET_PROP(psy, psp) psy_get_prop(psy, psp, #psp)
+static inline int psy_get_prop(struct power_supply *psy,
+ enum power_supply_property psp,
+ char *prop_name)
+{
+ union power_supply_propval val;
+ int ret = 0;
+
+ if (!psy)
+ return -EINVAL;
+ pr_debug("get %s for '%s'...\n", prop_name, psy->desc->name);
+ ret = power_supply_get_property(psy, psp, &val);
+ if (ret < 0) {
+ pr_err("failed to get %s from '%s', ret=%d\n", prop_name,
+ psy->desc->name, ret);
+ return ret;
+ }
+ pr_debug("get %s for '%s' => %d\n", prop_name, psy->desc->name,
+ val.intval);
+ return val.intval;
+}
+
+static inline int get_dts_vars(struct overheat_info *ovh_info)
+{
+ struct device *dev = ovh_info->dev;
+ struct device_node *node = dev->of_node;
+ int ret;
+
+ ret = of_property_read_u32(node, "google,begin-mitigation-temp",
+ &ovh_info->begin_temp);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "cannot read begin-mitigation-temp, ret=%d\n", ret);
+ return ret;
+ }
+
+ ret = of_property_read_u32(node, "google,end-mitigation-temp",
+ &ovh_info->clear_temp);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "cannot read end-mitigation-temp, ret=%d\n", ret);
+ return ret;
+ }
+
+ ret = of_property_read_u32(node, "google,port-overheat-work-interval",
+ &ovh_info->overheat_work_delay_ms);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "cannot read port-overheat-work-interval, ret=%d\n",
+ ret);
+ return ret;
+ }
+
+ ret = of_property_read_u32(node, "google,polling-freq",
+ &ovh_info->polling_freq);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "cannot read polling-freq, ret=%d\n", ret);
+ return ret;
+ }
+
+ ret = of_property_read_u32(node, "google,accessory-monitoring-period",
+ &ovh_info->monitor_accessory_s);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "cannot read accessory-monitoring-period, ret=%d\n",
+ ret);
+ ovh_info->monitor_accessory_s = 5;
+ }
+
+ return 0;
+}
+
+static int suspend_usb(struct overheat_info *ovh_info)
+{
+ int ret;
+
+ ovh_info->usb_replug = false;
+
+ /* disable USB */
+ ret = vote(ovh_info->disable_power_role_switch,
+ USB_OVERHEAT_MITIGATION_VOTER, true, 0);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "Couldn't vote for disable_power_role_switch ret=%d\n",
+ ret);
+ return ret;
+ }
+
+ /* suspend charging */
+ ret = vote(ovh_info->usb_icl_votable,
+ USB_OVERHEAT_MITIGATION_VOTER, true, 0);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "Couldn't vote for USB ICL ret=%d\n", ret);
+ return ret;
+ }
+
+ ovh_info->overheat_mitigation = true;
+ return ret;
+}
+
+static int resume_usb(struct overheat_info *ovh_info)
+{
+ int ret;
+
+ /* enable charging */
+ ret = vote(ovh_info->usb_icl_votable,
+ USB_OVERHEAT_MITIGATION_VOTER, false, 0);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "Couldn't un-vote for USB ICL ret=%d\n", ret);
+ return ret;
+ }
+
+ /* enable USB */
+ ret = vote(ovh_info->disable_power_role_switch,
+ USB_OVERHEAT_MITIGATION_VOTER, false, 0);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "Couldn't un-vote for disable_power_role_switch ret=%d\n",
+ ret);
+ return ret;
+ }
+
+ ovh_info->overheat_mitigation = false;
+ ovh_info->usb_replug = false;
+ return ret;
+}
+
+static inline time_t get_seconds_since_boot(void)
+{
+ struct timespec boot;
+
+ getboottime(&boot);
+ return get_seconds() - boot.tv_sec;
+}
+
+/*
+ * Update usb_connected, accessory_connected, and usb_replug status in
+ * overheat_info struct.
+ */
+static int update_usb_status(struct overheat_info *ovh_info)
+{
+ int ret;
+ bool prev_state = ovh_info->usb_connected ||
+ ovh_info->accessory_connected;
+ bool prev_accessory_state = ovh_info->accessory_connected;
+ bool curr_state;
+ int *check_status = &ovh_info->check_status;
+
+ if (ovh_info->overheat_mitigation) {
+ // Only check USB status every polling_freq instances
+ *check_status = (*check_status + 1) % ovh_info->polling_freq;
+ if (*check_status > 0)
+ return 0;
+ ret = vote(ovh_info->disable_power_role_switch,
+ USB_OVERHEAT_MITIGATION_VOTER, false, 0);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "Couldn't un-vote for disable_power_role_switch ret=%d\n",
+ ret);
+ return ret;
+ }
+ msleep(200);
+ }
+
+ dev_dbg(ovh_info->dev, "Updating USB connected status\n");
+ /*
+ * Update USB online status so that port_overheat_work can check it
+ * before rescheduling.
+ */
+ ret = PSY_GET_PROP(ovh_info->usb_psy, POWER_SUPPLY_PROP_ONLINE);
+ if (ret < 0)
+ return ret;
+ ovh_info->usb_online = ret;
+
+ /*
+ * Update USB present status to determine if USB has been disconnected.
+ * If we use USB online status to determine replug, we will need to
+ * extend the delay between re-enabling CC detection and checking the
+ * USB online status.
+ */
+ ret = PSY_GET_PROP(ovh_info->usb_psy, POWER_SUPPLY_PROP_PRESENT);
+ if (ret < 0)
+ return ret;
+ ovh_info->usb_connected = ret;
+
+ ret = PSY_GET_PROP(ovh_info->usb_psy, POWER_SUPPLY_PROP_TYPEC_MODE);
+ if (ret < 0)
+ return ret;
+ ovh_info->accessory_connected = (ret == POWER_SUPPLY_TYPEC_SINK) ||
+ (ret == POWER_SUPPLY_TYPEC_SINK_POWERED_CABLE);
+
+ if (!prev_accessory_state && ovh_info->accessory_connected)
+ ovh_info->accessory_connect_time = get_seconds_since_boot();
+ curr_state = ovh_info->usb_connected || ovh_info->accessory_connected;
+
+ if (ovh_info->overheat_mitigation) {
+ ret = vote(ovh_info->disable_power_role_switch,
+ USB_OVERHEAT_MITIGATION_VOTER, true, 0);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "Couldn't vote for disable_power_role_switch ret=%d\n",
+ ret);
+ return ret;
+ }
+ }
+
+ if (curr_state != prev_state)
+ dev_info(ovh_info->dev,
+ "USB is %sconnected",
+ curr_state ? "" : "dis");
+
+ // USB should be disconnected for two cycles before replug is acked
+ if (ovh_info->overheat_mitigation && !curr_state && !prev_state)
+ ovh_info->usb_replug = true;
+
+ return 0;
+}
+
+static inline int get_usb_port_temp(struct overheat_info *ovh_info)
+{
+ int temp;
+
+ temp = PSY_GET_PROP(ovh_info->usb_psy, POWER_SUPPLY_PROP_TEMP);
+
+ if (fake_port_temp > 0)
+ temp = fake_port_temp;
+
+ if (ovh_info->overheat_mitigation || temp >= ovh_info->begin_temp)
+ dev_info(ovh_info->dev, "Overheat triggered: USB port temp is %d\n",
+ temp);
+ return temp;
+}
+
+static int psy_changed(struct notifier_block *nb, unsigned long action,
+ void *data)
+{
+ struct power_supply *psy = data;
+ struct overheat_info *ovh_info =
+ container_of(nb, struct overheat_info, psy_nb);
+
+ if ((action != PSY_EVENT_PROP_CHANGED) || (psy == NULL) ||
+ (psy->desc == NULL) || (psy->desc->name == NULL))
+ return NOTIFY_OK;
+
+ if (action == PSY_EVENT_PROP_CHANGED &&
+ !strcmp(psy->desc->name, "usb")) {
+ dev_dbg(ovh_info->dev, "name=usb evt=%lu\n", action);
+ if (!ovh_info->overheat_work_running)
+ schedule_delayed_work(&ovh_info->port_overheat_work, 0);
+ }
+ return NOTIFY_OK;
+}
+
+static bool should_check_accessory(struct overheat_info *ovh_info)
+{
+ time_t connected;
+ bool ret;
+
+ if (!ovh_info->accessory_connected)
+ return false;
+
+ connected =
+ get_seconds_since_boot() - ovh_info->accessory_connect_time;
+ ret = (connected < ovh_info->monitor_accessory_s);
+ if (!ret)
+ dev_info(ovh_info->dev,
+ "Stop monitoring: %d sec since USB accessory connected",
+ (int) connected);
+ return ret;
+}
+
+static void port_overheat_work(struct work_struct *work)
+{
+ struct overheat_info *ovh_info =
+ container_of(work, struct overheat_info,
+ port_overheat_work.work);
+ int temp = 0, ret = 0;
+
+ // Take a wake lock to ensure we poll the temp regularly
+ if (!ovh_info->overheat_work_running)
+ __pm_stay_awake(&ovh_info->overheat_ws);
+ ovh_info->overheat_work_running = true;
+
+ if (enable) {
+ temp = get_usb_port_temp(ovh_info);
+ if (temp < 0)
+ goto rerun;
+
+ // Check USB connection status if it's safe to do so
+ if (!ovh_info->overheat_mitigation ||
+ temp < ovh_info->clear_temp) {
+ ret = update_usb_status(ovh_info);
+ if (ret < 0)
+ goto rerun;
+ }
+ }
+
+ if (ovh_info->overheat_mitigation && (!enable ||
+ (temp < ovh_info->clear_temp && ovh_info->usb_replug))) {
+ dev_err(ovh_info->dev, "Port overheat mitigated\n");
+ resume_usb(ovh_info);
+ } else if (!ovh_info->overheat_mitigation &&
+ enable && temp > ovh_info->begin_temp) {
+ dev_err(ovh_info->dev, "Port overheat triggered\n");
+ suspend_usb(ovh_info);
+ goto rerun;
+ }
+
+ if (ovh_info->overheat_mitigation || ovh_info->usb_online ||
+ should_check_accessory(ovh_info))
+ goto rerun;
+ // Do not run again, USB port isn't overheated or registering as online
+ ovh_info->overheat_work_running = false;
+ __pm_relax(&ovh_info->overheat_ws);
+ return;
+
+rerun:
+ schedule_delayed_work(
+ &ovh_info->port_overheat_work,
+ msecs_to_jiffies(ovh_info->overheat_work_delay_ms));
+}
+
+static int ovh_probe(struct platform_device *pdev)
+{
+ int ret = 0;
+ struct overheat_info *ovh_info;
+ struct power_supply *usb_psy;
+ struct votable *usb_icl_votable;
+ struct votable *disable_power_role_switch;
+
+ usb_psy = power_supply_get_by_name("usb");
+ if (!usb_psy)
+ return -EPROBE_DEFER;
+
+ usb_icl_votable = find_votable("USB_ICL");
+ if (usb_icl_votable == NULL) {
+ pr_err("Couldn't find USB_ICL votable\n");
+ return -EPROBE_DEFER;
+ }
+
+ disable_power_role_switch = find_votable("DISABLE_POWER_ROLE_SWITCH");
+ if (disable_power_role_switch == NULL) {
+ pr_err("Couldn't find DISABLE_POWER_ROLE_SWITCH votable\n");
+ return -EPROBE_DEFER;
+ }
+
+ ovh_info = devm_kzalloc(&pdev->dev, sizeof(*ovh_info), GFP_KERNEL);
+ if (!ovh_info)
+ return -ENOMEM;
+
+ ovh_info->dev = &pdev->dev;
+ ovh_info->usb_icl_votable = usb_icl_votable;
+ ovh_info->disable_power_role_switch = disable_power_role_switch;
+ ovh_info->usb_psy = usb_psy;
+ ovh_info->overheat_mitigation = false;
+ ovh_info->usb_replug = false;
+ ovh_info->usb_online = false;
+ ovh_info->usb_connected = false;
+ ovh_info->overheat_work_running = false;
+
+ ret = get_dts_vars(ovh_info);
+ if (ret < 0)
+ return -ENODEV;
+
+ // initialize votables
+ vote(ovh_info->usb_icl_votable,
+ USB_OVERHEAT_MITIGATION_VOTER, false, 0);
+ vote(ovh_info->disable_power_role_switch,
+ USB_OVERHEAT_MITIGATION_VOTER, false, 0);
+
+ // register power supply change notifier
+ ovh_info->psy_nb.notifier_call = psy_changed;
+ ret = power_supply_reg_notifier(&ovh_info->psy_nb);
+ if (ret < 0) {
+ dev_err(ovh_info->dev,
+ "Cannot register power supply notifer, ret=%d\n", ret);
+ return ret;
+ }
+
+ platform_set_drvdata(pdev, ovh_info);
+ wakeup_source_init(&ovh_info->overheat_ws, "overheat_mitigation");
+ INIT_DELAYED_WORK(&ovh_info->port_overheat_work, port_overheat_work);
+ schedule_delayed_work(&ovh_info->port_overheat_work, 0);
+
+ return ret;
+}
+
+static int ovh_remove(struct platform_device *pdev)
+{
+ struct overheat_info *ovh_info = platform_get_drvdata(pdev);
+ if (ovh_info)
+ wakeup_source_trash(&ovh_info->overheat_ws);
+
+ return 0;
+}
+
+static const struct of_device_id match_table[] = {
+ {
+ .compatible = "google,overheat_mitigation",
+ },
+ {},
+};
+
+static struct platform_driver ovh_driver = {
+ .driver = {
+ .name = "google,overheat_mitigation",
+ .owner = THIS_MODULE,
+ .of_match_table = match_table,
+ },
+ .probe = ovh_probe,
+ .remove = ovh_remove,
+};
+
+module_platform_driver(ovh_driver);
+MODULE_DESCRIPTION("USB port overheat mitigation driver");
+MODULE_AUTHOR("Maggie White <maggiewhite@google.com>");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/power/supply/power_supply_core.c b/drivers/power/supply/power_supply_core.c
index 077d237..77b6885 100644
--- a/drivers/power/supply/power_supply_core.c
+++ b/drivers/power/supply/power_supply_core.c
@@ -14,6 +14,7 @@
#include <linux/types.h>
#include <linux/init.h>
#include <linux/slab.h>
+#include <linux/delay.h>
#include <linux/device.h>
#include <linux/notifier.h>
#include <linux/err.h>
@@ -141,8 +142,13 @@ static void power_supply_deferred_register_work(struct work_struct *work)
struct power_supply *psy = container_of(work, struct power_supply,
deferred_register_work.work);
- if (psy->dev.parent)
- mutex_lock(&psy->dev.parent->mutex);
+ if (psy->dev.parent) {
+ while (!mutex_trylock(&psy->dev.parent->mutex)) {
+ if (psy->removing)
+ return;
+ msleep(10);
+ }
+ }
psy_register_cooler(psy->dev.parent, psy);
power_supply_changed(psy);
@@ -948,6 +954,7 @@ EXPORT_SYMBOL_GPL(devm_power_supply_register_no_ws);
void power_supply_unregister(struct power_supply *psy)
{
WARN_ON(atomic_dec_return(&psy->use_cnt));
+ psy->removing = true;
cancel_work_sync(&psy->changed_work);
cancel_delayed_work_sync(&psy->deferred_register_work);
sysfs_remove_link(&psy->dev.kobj, "powers");
diff --git a/drivers/power/supply/power_supply_sysfs.c b/drivers/power/supply/power_supply_sysfs.c
index 3ab93fe..8318603 100644
--- a/drivers/power/supply/power_supply_sysfs.c
+++ b/drivers/power/supply/power_supply_sysfs.c
@@ -81,7 +81,7 @@ static ssize_t power_supply_show_property(struct device *dev,
"Non compliant",
};
static const char * const typec_pr_text[] = {
- "none", "dual power role", "sink", "source"
+ "none", "dual power role", "sink", "source", "source_1_5"
};
ssize_t ret = 0;
struct power_supply *psy = dev_get_drvdata(dev);
@@ -316,6 +316,7 @@ static struct device_attribute power_supply_attrs[] = {
POWER_SUPPLY_ATTR(cc_step),
POWER_SUPPLY_ATTR(cc_step_sel),
POWER_SUPPLY_ATTR(sw_jeita_enabled),
+ POWER_SUPPLY_ATTR(taper_control_enabled),
POWER_SUPPLY_ATTR(pd_voltage_max),
POWER_SUPPLY_ATTR(pd_voltage_min),
POWER_SUPPLY_ATTR(sdp_current_max),
@@ -335,6 +336,9 @@ static struct device_attribute power_supply_attrs[] = {
POWER_SUPPLY_ATTR(battery_info),
POWER_SUPPLY_ATTR(battery_info_id),
POWER_SUPPLY_ATTR(enable_jeita_detection),
+ POWER_SUPPLY_ATTR(otg_fastroleswap),
+ POWER_SUPPLY_ATTR(charge_disable),
+ POWER_SUPPLY_ATTR(in_explicit_contract),
/* Local extensions of type int64_t */
POWER_SUPPLY_ATTR(charge_counter_ext),
/* Properties of type `const char *' */
diff --git a/drivers/power/supply/qcom/Kconfig b/drivers/power/supply/qcom/Kconfig
index 1c8f116..ca89d96 100644
--- a/drivers/power/supply/qcom/Kconfig
+++ b/drivers/power/supply/qcom/Kconfig
@@ -18,6 +18,24 @@
the fuel gauge. The state of charge is reported through a BMS power
supply property and also sends uevents when the capacity is updated.
+config QPNP_FG_GEN3_DEBUG_BATT
+ bool "QPNP GEN3 FG debug battery"
+ default n
+ help
+ Say Y here to enable QCOM FG debug battery detection.
+ Disabling by default to avoid ID probe on every capacity update.
+
+config QPNP_FG_GEN3_LEGACY_CYCLE_COUNT
+ tristate "QPNP GEN3 fuel gauge cycle count bins"
+ depends on QPNP_FG_GEN3
+ default y
+ help
+ Say Y here to enable the GEN3 FG initial QCOM cycle count implementation.
+ When not selected then POWER_SUPPLY_PROP_CYCLE_COUNT exposes the linux
+ power supply battery cycle count which is not based on cycle bins.
+ When selected only 'bms' will expose POWER_SUPPLY_PROP_CYCLE_COUNT
+ reporting the bucket based value.
+
config SMB135X_CHARGER
tristate "SMB135X Battery Charger"
depends on I2C
@@ -115,6 +133,13 @@
module. It also allows userspace code to read diagnostics of voltage
and current measured during certain phases of the pulses.
+config CHARGER_P9221
+ tristate "IDT P9221 wireless power receiver driver"
+ depends on I2C
+ help
+ This driver provides support for the IDT P9221 wireless
+ power receiver.
+
config QPNP_VM_BMS
tristate "QPNP Voltage-Mode Battery Monitoring System driver"
depends on MFD_SPMI_PMIC
diff --git a/drivers/power/supply/qcom/Makefile b/drivers/power/supply/qcom/Makefile
index ffc1ce3..990ecdb 100644
--- a/drivers/power/supply/qcom/Makefile
+++ b/drivers/power/supply/qcom/Makefile
@@ -9,8 +9,14 @@
obj-$(CONFIG_SMB138X_CHARGER) += step-chg-jeita.o smb138x-charger.o smb-lib.o pmic-voter.o storm-watch.o battery.o
obj-$(CONFIG_QPNP_QG) += qpnp-qg.o pmic-voter.o qg-util.o qg-soc.o qg-sdam.o qg-battery-profile.o qg-profile-lib.o fg-alg.o
obj-$(CONFIG_QPNP_QNOVO) += qpnp-qnovo.o battery.o
+obj-$(CONFIG_CHARGER_P9221) += p9221_charger.o
obj-$(CONFIG_QPNP_TYPEC) += qpnp-typec.o
obj-$(CONFIG_QPNP_SMB5) += step-chg-jeita.o battery.o qpnp-smb5.o smb5-lib.o pmic-voter.o storm-watch.o schgm-flash.o
obj-$(CONFIG_SMB1390_CHARGE_PUMP) += smb1390-charger.o pmic-voter.o
obj-$(CONFIG_QPNP_VM_BMS) += qpnp-vm-bms.o batterydata-lib.o batterydata-interface.o
obj-$(CONFIG_QPNP_LINEAR_CHARGER) += qpnp-linear-charger.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-duplicate-decl-specifier \
+
diff --git a/drivers/power/supply/qcom/battery.c b/drivers/power/supply/qcom/battery.c
index e8d91aee..d08ab06 100644
--- a/drivers/power/supply/qcom/battery.c
+++ b/drivers/power/supply/qcom/battery.c
@@ -451,6 +451,12 @@ static void pl_taper_work(struct work_struct *work)
int rc;
int eff_fcc_ua;
int total_fcc_ua, master_fcc_ua, slave_fcc_ua = 0;
+ int tcm = POWER_SUPPLY_TAPER_CONTROL_MODE_STEPPER;
+
+ rc = power_supply_get_property(chip->batt_psy,
+ POWER_SUPPLY_PROP_TAPER_CONTROL, &pval);
+ if (rc == 0)
+ tcm = pval.intval;
chip->taper_work_running = true;
while (true) {
@@ -481,24 +487,34 @@ static void pl_taper_work(struct work_struct *work)
}
chip->charge_type = pval.intval;
- if (pval.intval == POWER_SUPPLY_CHARGE_TYPE_TAPER) {
+ if (pval.intval != POWER_SUPPLY_CHARGE_TYPE_TAPER) {
+ pl_dbg(chip, PR_PARALLEL,
+ "master is fast charging; waiting for next taper\n");
+ } else if (tcm == POWER_SUPPLY_TAPER_CONTROL_MODE_IMMEDIATE) {
+ pl_dbg(chip, PR_PARALLEL,
+ "master is taper charging: stop parallel\n");
+ vote(chip->pl_disable_votable, TAPER_END_VOTER,
+ true, 0);
+ goto done;
+ } else {
+
eff_fcc_ua = get_effective_result(chip->fcc_votable);
if (eff_fcc_ua < 0) {
pr_err("Couldn't get fcc, exiting taper work\n");
goto done;
}
+
eff_fcc_ua = eff_fcc_ua - TAPER_REDUCTION_UA;
if (eff_fcc_ua < 0) {
pr_err("Can't reduce FCC any more\n");
goto done;
}
- pl_dbg(chip, PR_PARALLEL, "master is taper charging; reducing FCC to %dua\n",
- eff_fcc_ua);
+ pl_dbg(chip, PR_PARALLEL,
+ "master is taper charging; reducing FCC to %dua\n",
+ eff_fcc_ua);
vote(chip->fcc_votable, TAPER_STEPPER_VOTER,
true, eff_fcc_ua);
- } else {
- pl_dbg(chip, PR_PARALLEL, "master is fast charging; waiting for next taper\n");
}
/* wait for the charger state to deglitch after FCC change */
msleep(PL_TAPER_WORK_DELAY_MS);
@@ -683,7 +699,8 @@ static int pl_disable_vote_callback(struct votable *votable,
struct pl_data *chip = data;
union power_supply_propval pval = {0, };
int master_fcc_ua = 0, total_fcc_ua = 0, slave_fcc_ua = 0;
- int rc = 0;
+ int rc = 0, charge_type;
+ int tcm = POWER_SUPPLY_TAPER_CONTROL_MODE_STEPPER;
bool disable = false;
if (!is_main_available(chip))
@@ -795,12 +812,21 @@ static int pl_disable_vote_callback(struct votable *votable,
* start the taper work if so
*/
rc = power_supply_get_property(chip->batt_psy,
+ POWER_SUPPLY_PROP_TAPER_CONTROL, &pval);
+ if (rc < 0)
+ pr_err("Couldn't get taper control state rc=%d\n", rc);
+ else
+ tcm = pval.intval;
+
+ rc = power_supply_get_property(chip->batt_psy,
POWER_SUPPLY_PROP_CHARGE_TYPE, &pval);
+ charge_type = pval.intval;
if (rc < 0) {
pr_err("Couldn't get batt charge type rc=%d\n", rc);
} else {
- if (pval.intval == POWER_SUPPLY_CHARGE_TYPE_TAPER
- && !chip->taper_work_running) {
+ if (charge_type == POWER_SUPPLY_CHARGE_TYPE_TAPER
+ && !chip->taper_work_running
+ && tcm != POWER_SUPPLY_TAPER_CONTROL_OFF) {
pl_dbg(chip, PR_PARALLEL,
"pl enabled in Taper scheduing work\n");
vote(chip->pl_awake_votable, TAPER_END_VOTER,
@@ -965,7 +991,8 @@ static bool is_parallel_available(struct pl_data *chip)
static void handle_main_charge_type(struct pl_data *chip)
{
union power_supply_propval pval = {0, };
- int rc;
+ int rc, charge_type;
+ int tcm = POWER_SUPPLY_TAPER_CONTROL_MODE_STEPPER;
rc = power_supply_get_property(chip->batt_psy,
POWER_SUPPLY_PROP_CHARGE_TYPE, &pval);
@@ -973,38 +1000,70 @@ static void handle_main_charge_type(struct pl_data *chip)
pr_err("Couldn't get batt charge type rc=%d\n", rc);
return;
}
+ charge_type = pval.intval;
+
+ rc = power_supply_get_property(chip->batt_psy,
+ POWER_SUPPLY_PROP_TAPER_CONTROL, &pval);
+ if (rc < 0) {
+ pr_err("Couldn't get taper control state rc=%d\n", rc);
+ return;
+ }
+ tcm = pval.intval;
/* not fast/not taper state to disables parallel */
- if ((pval.intval != POWER_SUPPLY_CHARGE_TYPE_FAST)
- && (pval.intval != POWER_SUPPLY_CHARGE_TYPE_TAPER)) {
+ if ((charge_type != POWER_SUPPLY_CHARGE_TYPE_FAST)
+ && (charge_type != POWER_SUPPLY_CHARGE_TYPE_TAPER)) {
vote(chip->pl_disable_votable, CHG_STATE_VOTER, true, 0);
- chip->charge_type = pval.intval;
+ chip->charge_type = charge_type;
return;
}
- /* handle taper charge entry */
+ /* handle taper charge entry
+ * taper_work will disable parallel with TAPER_END_VOTER after stepping
+ * down FCC or immediately.
+ */
if (chip->charge_type == POWER_SUPPLY_CHARGE_TYPE_FAST
- && (pval.intval == POWER_SUPPLY_CHARGE_TYPE_TAPER)) {
- chip->charge_type = pval.intval;
+ && charge_type == POWER_SUPPLY_CHARGE_TYPE_TAPER
+ && tcm != POWER_SUPPLY_TAPER_CONTROL_OFF) {
+ chip->charge_type = charge_type;
if (!chip->taper_work_running) {
- pl_dbg(chip, PR_PARALLEL, "taper entry scheduling work\n");
+ pl_dbg(chip, PR_PARALLEL, "start taper work (%d)\n",
+ tcm);
vote(chip->pl_awake_votable, TAPER_END_VOTER, true, 0);
queue_work(system_long_wq, &chip->pl_taper_work);
}
return;
}
- /* handle fast/taper charge entry */
- if (pval.intval == POWER_SUPPLY_CHARGE_TYPE_TAPER
- || pval.intval == POWER_SUPPLY_CHARGE_TYPE_FAST) {
- pl_dbg(chip, PR_PARALLEL, "chg_state enabling parallel\n");
+ /* re-enable parallel when in FAST mode or disable it immediately while
+ * in TAPER if taper control is enabled.
+ */
+ if (charge_type == POWER_SUPPLY_CHARGE_TYPE_FAST) {
+ pl_dbg(chip, PR_PARALLEL, "VOTER parallel enabled\n");
vote(chip->pl_disable_votable, CHG_STATE_VOTER, false, 0);
- chip->charge_type = pval.intval;
+ chip->charge_type = charge_type;
+ } else if (charge_type == POWER_SUPPLY_CHARGE_TYPE_TAPER) {
+ const bool state = (tcm
+ == POWER_SUPPLY_TAPER_CONTROL_MODE_IMMEDIATE);
+
+ pl_dbg(chip, PR_PARALLEL, "VOTER tce=%d dis_parallel stat=%d\n",
+ tcm, state);
+
+ /* when called with "false" (to enable parallel, DUH!) the
+ * callback logic will re-enable parallel charging and kick
+ * off pl_taper_work when the device is in TAPER. Here I take
+ * a shortcut and disable parallel immediately if in
+ * POWER_SUPPLY_TAPER_CONTROL_MODE_IMMEDIATE
+ * NOTE: I might not need the shortcut...
+ */
+
+ vote(chip->pl_disable_votable, CHG_STATE_VOTER, state, 0);
+ chip->charge_type = charge_type;
return;
}
/* remember the new state only if it isn't any of the above */
- chip->charge_type = pval.intval;
+ chip->charge_type = charge_type;
}
#define MIN_ICL_CHANGE_DELTA_UA 300000
diff --git a/drivers/power/supply/qcom/fg-core.h b/drivers/power/supply/qcom/fg-core.h
index f73e647..8df8d9a 100644
--- a/drivers/power/supply/qcom/fg-core.h
+++ b/drivers/power/supply/qcom/fg-core.h
@@ -273,6 +273,7 @@ struct fg_dt_props {
bool hold_soc_while_full;
bool linearize_soc;
bool auto_recharge_soc;
+ bool soc_irq_disable;
int cutoff_volt_mv;
int empty_volt_mv;
int vbatt_low_thr_mv;
diff --git a/drivers/power/supply/qcom/p9221_charger.c b/drivers/power/supply/qcom/p9221_charger.c
new file mode 100644
index 0000000..46f9f85
--- /dev/null
+++ b/drivers/power/supply/qcom/p9221_charger.c
@@ -0,0 +1,2448 @@
+/*
+ * P9221 Wireless Charger Driver
+ *
+ * Copyright (C) 2017 Google, Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+
+#include <linux/device.h>
+#include <linux/pm.h>
+#include <linux/gpio.h>
+#include <linux/interrupt.h>
+#include <linux/i2c.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/pm_runtime.h>
+#include <linux/of.h>
+#include <linux/of_gpio.h>
+#include <linux/kernel.h>
+#include <linux/delay.h>
+#include <linux/power_supply.h>
+#include <linux/pmic-voter.h>
+#include <linux/alarmtimer.h>
+#include "p9221_charger.h"
+
+#define P9221_TX_TIMEOUT_MS (20 * 1000)
+#define P9221_DCIN_TIMEOUT_MS (2 * 1000)
+#define P9221_VRECT_TIMEOUT_MS (2 * 1000)
+#define P9221_NOTIFIER_DELAY_MS 80
+#define P9221R5_ILIM_MAX_UA (1600 * 1000)
+#define P9221R5_OVER_CHECK_NUM 3
+
+#define OVC_LIMIT 1
+#define OVC_THRESHOLD 1400000
+#define OVC_BACKOFF_LIMIT 900000
+#define OVC_BACKOFF_AMOUNT 100000
+
+static void p9221_icl_ramp_reset(struct p9221_charger_data *charger);
+static void p9221_icl_ramp_start(struct p9221_charger_data *charger);
+
+static const u32 p9221_ov_set_lut[] = {
+ 17000000, 20000000, 15000000, 13000000,
+ 11000000, 11000000, 11000000, 11000000};
+
+static size_t p9221_hex_str(u8 *data, size_t len, char *buf, size_t max_buf,
+ bool msbfirst)
+{
+ int i;
+ int blen = 0;
+ u8 val;
+
+ for (i = 0; i < len; i++) {
+ if (msbfirst)
+ val = data[len - 1 - i];
+ else
+ val = data[i];
+ blen += scnprintf(buf + (i * 3), max_buf - (i * 3),
+ "%02x ", val);
+ }
+ return blen;
+}
+
+static int p9221_reg_read_n(struct p9221_charger_data *charger, u16 reg,
+ void *buf, size_t n)
+{
+ int ret;
+ struct i2c_msg msg[2];
+ u8 wbuf[2];
+
+ msg[0].addr = charger->client->addr;
+ msg[0].flags = charger->client->flags & I2C_M_TEN;
+ msg[0].len = 2;
+ msg[0].buf = wbuf;
+
+ wbuf[0] = (reg & 0xFF00) >> 8;
+ wbuf[1] = (reg & 0xFF);
+
+ msg[1].addr = charger->client->addr;
+ msg[1].flags = I2C_M_RD;
+ msg[1].len = n;
+ msg[1].buf = buf;
+
+ mutex_lock(&charger->io_lock);
+ ret = i2c_transfer(charger->client->adapter, msg, 2);
+ mutex_unlock(&charger->io_lock);
+
+ if (ret < 0) {
+ /*
+ * Treat -ENOTCONN as -ENODEV to suppress the get/set
+ * prop warnings.
+ */
+ int nret = (ret == -ENOTCONN) ? -ENODEV : ret;
+
+ dev_err(&charger->client->dev,
+ "i2c read error, reg:%x, ret:%d (%d)\n",
+ reg, ret, nret);
+ return nret;
+ }
+
+ return (ret == 2) ? 0 : -EIO;
+}
+
+static int p9221_reg_read_16(struct p9221_charger_data *charger, u16 reg,
+ u16 *val)
+{
+ u8 buf[2];
+ int ret;
+
+ ret = p9221_reg_read_n(charger, reg, buf, 2);
+ if (ret == 0)
+ *val = (buf[1] << 8) | buf[0];
+ return ret;
+}
+
+static int p9221_reg_read_8(struct p9221_charger_data *charger,
+ u16 reg, u8 *val)
+{
+ return p9221_reg_read_n(charger, reg, val, 1);
+}
+
+static bool p9221_reg_is_8_bit(struct p9221_charger_data *charger, u16 reg)
+{
+ switch (reg) {
+ case P9221_CHIP_REVISION_REG:
+ case P9221_CUSTOMER_ID_REG:
+ case P9221_COM_REG:
+ case P9221R5_VOUT_SET_REG:
+ case P9221R5_ILIM_SET_REG:
+ case P9221R5_CHARGE_STAT_REG:
+ case P9221R5_EPT_REG:
+ case P9221R5_SYSTEM_MODE_REG:
+ case P9221R5_COM_CHAN_RESET_REG:
+ case P9221R5_COM_CHAN_SEND_SIZE_REG:
+ case P9221R5_COM_CHAN_SEND_IDX_REG:
+ case P9221R5_COM_CHAN_RECV_SIZE_REG:
+ case P9221R5_COM_CHAN_RECV_IDX_REG:
+ case P9221R5_DEBUG_REG:
+ case P9221R5_EPP_Q_FACTOR_REG:
+ case P9221R5_EPP_TX_GUARANTEED_POWER_REG:
+ case P9221R5_EPP_TX_POTENTIAL_POWER_REG:
+ case P9221R5_EPP_TX_CAPABILITY_FLAGS_REG:
+ case P9221R5_EPP_RENEGOTIATION_REG:
+ case P9221R5_EPP_CUR_RPP_HEADER_REG:
+ case P9221R5_EPP_CUR_NEGOTIATED_POWER_REG:
+ case P9221R5_EPP_CUR_MAXIMUM_POWER_REG:
+ case P9221R5_EPP_CUR_FSK_MODULATION_REG:
+ case P9221R5_EPP_REQ_RPP_HEADER_REG:
+ case P9221R5_EPP_REQ_NEGOTIATED_POWER_REG:
+ case P9221R5_EPP_REQ_MAXIMUM_POWER_REG:
+ case P9221R5_EPP_REQ_FSK_MODULATION_REG:
+ case P9221R5_VRECT_TARGET_REG:
+ case P9221R5_VRECT_KNEE_REG:
+ case P9221R5_FOD_SECTION_REG:
+ case P9221R5_VRECT_ADJ_REG:
+ case P9221R5_ALIGN_X_ADC_REG:
+ case P9221R5_ALIGN_Y_ADC_REG:
+ case P9221R5_ASK_MODULATION_DEPTH_REG:
+ case P9221R5_OVSET_REG:
+ case P9221R5_EPP_TX_SPEC_REV_REG:
+ return true;
+ default:
+ return false;
+ }
+}
+
+/*
+ * Cook the value according to the register (R5+)
+ */
+static int p9221_cook_reg(struct p9221_charger_data *charger, u16 reg,
+ u16 raw_data, u32 *val)
+{
+ /* Do the appropriate conversion */
+ switch (reg) {
+ /* The following raw values */
+ case P9221R5_ALIGN_X_ADC_REG:
+ case P9221R5_ALIGN_Y_ADC_REG:
+ *val = raw_data;
+ break;
+
+ /* The following are 12-bit ADC raw values */
+ case P9221R5_VOUT_ADC_REG:
+ case P9221R5_IOUT_ADC_REG:
+ case P9221R5_DIE_TEMP_ADC_REG:
+ case P9221R5_EXT_TEMP_REG:
+ *val = raw_data & 0xFFF;
+ break;
+
+ /* The following are in 0.1 mill- and need to go to micro- */
+ case P9221R5_VOUT_SET_REG: /* 100mV -> uV */
+ raw_data *= 100;
+ /* Fall through */
+
+ /* The following are in milli- and need to go to micro- */
+ case P9221R5_IOUT_REG: /* mA -> uA */
+ case P9221R5_VRECT_REG: /* mV -> uV */
+ case P9221R5_VOUT_REG: /* mV -> uV */
+ /* Fall through */
+
+ /* The following are in kilo- and need to go to their base */
+ case P9221R5_OP_FREQ_REG: /* kHz -> Hz */
+ case P9221R5_TX_PINGFREQ_REG: /* kHz -> Hz */
+ *val = raw_data * 1000;
+ break;
+
+ case P9221R5_ILIM_SET_REG:
+ /* 100mA -> uA, 200mA offset */
+ *val = ((raw_data * 100) + 200) * 1000;
+ break;
+
+ case P9221R5_OVSET_REG:
+ /* uV */
+ raw_data &= P9221R5_OVSET_MASK;
+ *val = p9221_ov_set_lut[raw_data];
+ break;
+
+ default:
+ return -ENOENT;
+ }
+
+ return 0;
+}
+
+/*
+ * Read the reg and return the cooked value.
+ */
+static int p9221_reg_read_cooked(struct p9221_charger_data *charger,
+ u16 reg, u32 *val)
+{
+ int ret = 0;
+ u16 data = 0;
+
+ if (p9221_reg_is_8_bit(charger, reg)) {
+ u8 data8 = 0;
+ ret = p9221_reg_read_8(charger, reg, &data8);
+ data = data8;
+ } else {
+ ret = p9221_reg_read_16(charger, reg, &data);
+ }
+ if (ret)
+ return ret;
+
+ return p9221_cook_reg(charger, reg, data, val);
+}
+
+static int p9221_reg_write_n(struct p9221_charger_data *charger, u16 reg,
+ void *buf, size_t n)
+{
+ int ret;
+ u8 *data;
+ int datalen = 2 + n;
+
+ data = kmalloc(datalen, GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ data[0] = reg >> 8;
+ data[1] = reg & 0xFF;
+ memcpy(&data[2], buf, n);
+
+ mutex_lock(&charger->io_lock);
+ ret = i2c_master_send(charger->client, data, datalen);
+ mutex_unlock(&charger->io_lock);
+ kfree(data);
+
+ if (ret < datalen) {
+ /*
+ * Treat -ENOTCONN as -ENODEV to suppress the get/set
+ * prop warnings.
+ */
+ int nret = (ret == -ENOTCONN) ? -ENODEV : -EIO;
+
+ dev_err(&charger->client->dev,
+ "%s: i2c write error, reg: 0x%x, n: %zd ret: %d (%d)\n",
+ __func__, reg, n, ret, nret);
+ return nret;
+ }
+
+ return 0;
+}
+
+static int p9221_reg_write_16(struct p9221_charger_data *charger, u16 reg,
+ u16 val)
+{
+ return p9221_reg_write_n(charger, reg, &val, 2);
+}
+
+static int p9221_reg_write_8(struct p9221_charger_data *charger, u16 reg, u8 val)
+{
+ return p9221_reg_write_n(charger, reg, &val, 1);
+}
+
+/*
+ * Uncook the values and write to register
+ */
+static int p9221_reg_write_cooked(struct p9221_charger_data *charger,
+ u16 reg, u32 val)
+{
+ int ret = 0;
+ u16 data;
+ int i;
+
+ /* Do the appropriate conversion */
+ switch (reg) {
+ case P9221R5_ILIM_SET_REG:
+ /* uA -> 0.1A, offset 0.2A */
+ if ((val < 200000) || (val > 1600000))
+ return -EINVAL;
+ data = (val / (100 * 1000)) - 2;
+ break;
+
+ case P9221R5_VOUT_SET_REG:
+ /* uV -> 0.1V */
+ val /= 1000;
+ if (val < 3500 || val > charger->pdata->max_vout_mv)
+ return -EINVAL;
+ data = val / 100;
+ break;
+
+ case P9221R5_OVSET_REG:
+ /* uV */
+ for (i = 0; i < ARRAY_SIZE(p9221_ov_set_lut); i++) {
+ if (val == p9221_ov_set_lut[i])
+ break;
+ }
+ if (i == ARRAY_SIZE(p9221_ov_set_lut))
+ return -EINVAL;
+ data = i;
+ break;
+ default:
+ return -ENOENT;
+ }
+
+ if (p9221_reg_is_8_bit(charger, reg))
+ ret = p9221_reg_write_8(charger, reg, data);
+ else
+ ret = p9221_reg_write_16(charger, reg, data);
+
+ return ret;
+}
+
+static bool p9221_is_epp(struct p9221_charger_data *charger)
+{
+ int ret;
+ u32 vout_uv;
+ uint8_t reg;
+
+ ret = p9221_reg_read_8(charger, P9221R5_SYSTEM_MODE_REG, ®);
+ if (ret == 0)
+ return (reg & P9221R5_SYSTEM_MODE_EXTENDED_MASK) > 0;
+
+ dev_err(&charger->client->dev, "Could not read mode: %d\n",
+ ret);
+
+ /* Check based on power supply voltage */
+ ret = p9221_reg_read_cooked(charger, P9221R5_VOUT_ADC_REG, &vout_uv);
+ if (ret) {
+ dev_err(&charger->client->dev, "Could read VOUT_ADC, %d\n",
+ ret);
+ goto out;
+ }
+
+ dev_info(&charger->client->dev, "Voltage is %duV\n", vout_uv);
+ if (vout_uv > P9221_EPP_THRESHOLD_UV)
+ return true;
+
+out:
+ /* Default to BPP otherwise */
+ return false;
+}
+
+static void p9221_write_fod(struct p9221_charger_data *charger)
+{
+ bool epp = false;
+ u8 *fod = NULL;
+ int fod_count = charger->pdata->fod_num;
+ int ret;
+ int retries = 3;
+
+ if (!charger->pdata->fod_num && !charger->pdata->fod_epp_num)
+ goto no_fod;
+
+ /* Default to BPP FOD */
+ if (charger->pdata->fod_num)
+ fod = charger->pdata->fod;
+
+ if (p9221_is_epp(charger) && charger->pdata->fod_epp_num) {
+ fod = charger->pdata->fod_epp;
+ fod_count = charger->pdata->fod_epp_num;
+ epp = true;
+ }
+
+ if (!fod)
+ goto no_fod;
+
+ while (retries) {
+ char s[fod_count * 3 + 1];
+ u8 fod_read[fod_count];
+
+ dev_info(&charger->client->dev, "Writing %s FOD (n=%d reg=%02x try=%d)\n",
+ epp ? "EPP" : "BPP", fod_count, P9221R5_FOD_REG,
+ retries);
+
+ ret = p9221_reg_write_n(charger, P9221R5_FOD_REG, fod,
+ fod_count);
+ if (ret) {
+ dev_err(&charger->client->dev,
+ "Could not write FOD: %d\n", ret);
+ return;
+ }
+
+ /* Verify the FOD has been written properly */
+ ret = p9221_reg_read_n(charger, P9221R5_FOD_REG, fod_read,
+ fod_count);
+ if (ret) {
+ dev_err(&charger->client->dev,
+ "Could not read back FOD: %d\n", ret);
+ return;
+ }
+
+ if (memcmp(fod, fod_read, fod_count) == 0)
+ return;
+
+ p9221_hex_str(fod_read, fod_count, s, sizeof(s), 0);
+ dev_err(&charger->client->dev,
+ "FOD verify error, read: %s\n", s);
+
+ retries--;
+ msleep(100);
+ }
+
+no_fod:
+ dev_warn(&charger->client->dev, "FOD not set! bpp:%d epp:%d r:%d\n",
+ charger->pdata->fod_num, charger->pdata->fod_epp_num, retries);
+}
+
+static int p9221_set_cmd_reg(struct p9221_charger_data *charger, u8 cmd)
+{
+ u8 cur_cmd = 0;
+ int retry;
+ int ret;
+
+ for (retry = 0; retry < P9221_COM_CHAN_RETRIES; retry++) {
+ ret = p9221_reg_read_8(charger, P9221_COM_REG, &cur_cmd);
+ if (ret == 0 && cur_cmd == 0)
+ break;
+ msleep(25);
+ }
+
+ if (retry >= P9221_COM_CHAN_RETRIES) {
+ dev_err(&charger->client->dev,
+ "Failed to wait for cmd free %02x\n", cur_cmd);
+ return -EBUSY;
+ }
+
+ ret = p9221_reg_write_8(charger, P9221_COM_REG, cmd);
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Failed to set cmd reg %02x: %d\n", cmd, ret);
+
+ return ret;
+}
+
+static int p9221_send_data(struct p9221_charger_data *charger)
+{
+ int ret;
+
+ if (charger->tx_busy)
+ return -EBUSY;
+
+ if (!charger->tx_len || charger->tx_len > P9221R5_DATA_SEND_BUF_SIZE)
+ return -EINVAL;
+
+ charger->tx_busy = true;
+
+ mutex_lock(&charger->cmd_lock);
+
+ ret = p9221_reg_write_n(charger, P9221R5_DATA_SEND_BUF_START,
+ charger->tx_buf, charger->tx_len);
+ if (ret) {
+ dev_err(&charger->client->dev, "Failed to load tx %d\n", ret);
+ goto error;
+ }
+
+ ret = p9221_reg_write_8(charger, P9221R5_COM_CHAN_SEND_SIZE_REG,
+ charger->tx_len);
+ if (ret) {
+ dev_err(&charger->client->dev, "Failed to load txsz %d\n", ret);
+ goto error;
+ }
+
+ ret = p9221_set_cmd_reg(charger, P9221R5_COM_CCACTIVATE);
+ if (ret)
+ goto error;
+
+ mutex_unlock(&charger->cmd_lock);
+ return ret;
+
+error:
+ mutex_unlock(&charger->cmd_lock);
+ charger->tx_busy = false;
+ return ret;
+}
+
+static int p9221_send_csp(struct p9221_charger_data *charger, u8 stat)
+{
+ int ret;
+
+ dev_info(&charger->client->dev, "Send CSP status=%d\n", stat);
+
+ mutex_lock(&charger->cmd_lock);
+
+ ret = p9221_reg_write_8(charger, P9221R5_CHARGE_STAT_REG, stat);
+ if (ret == 0)
+ ret = p9221_set_cmd_reg(charger, P9221R5_COM_SENDCSP);
+
+ mutex_unlock(&charger->cmd_lock);
+ return ret;
+}
+
+static int p9221_send_eop(struct p9221_charger_data *charger, u8 reason)
+{
+ int ret;
+
+ dev_info(&charger->client->dev, "Send EOP reason=%d\n", reason);
+
+ mutex_lock(&charger->cmd_lock);
+
+ ret = p9221_reg_write_8(charger, P9221R5_EPT_REG, reason);
+ if (ret == 0)
+ ret = p9221_set_cmd_reg(charger, P9221R5_COM_SENDEPT);
+
+ mutex_unlock(&charger->cmd_lock);
+ return ret;
+}
+
+static int p9221_send_ccreset(struct p9221_charger_data *charger)
+{
+ int ret;
+
+ dev_info(&charger->client->dev, "Send CC reset\n");
+
+ mutex_lock(&charger->cmd_lock);
+
+ ret = p9221_reg_write_8(charger, P9221R5_COM_CHAN_RESET_REG,
+ P9221R5_COM_CHAN_CCRESET);
+ if (ret == 0)
+ ret = p9221_set_cmd_reg(charger, P9221R5_COM_CCACTIVATE);
+
+ mutex_unlock(&charger->cmd_lock);
+ return ret;
+}
+
+struct p9221_prop_reg_map_entry p9221_prop_reg_map[] = {
+ /* property register g, s */
+ {POWER_SUPPLY_PROP_CURRENT_NOW, P9221R5_IOUT_REG, 1, 0},
+ {POWER_SUPPLY_PROP_VOLTAGE_NOW, P9221R5_VOUT_REG, 1, 0},
+ {POWER_SUPPLY_PROP_VOLTAGE_MAX, P9221R5_VOUT_SET_REG, 1, 1},
+ {POWER_SUPPLY_PROP_TEMP, P9221R5_DIE_TEMP_ADC_REG, 1, 0},
+ {POWER_SUPPLY_PROP_CAPACITY, 0, 1, 1},
+ {POWER_SUPPLY_PROP_ONLINE, 0, 1, 1},
+};
+
+static struct p9221_prop_reg_map_entry *p9221_get_map_entry(
+ struct p9221_charger_data *charger,
+ enum power_supply_property prop, bool set)
+{
+ int i;
+ struct p9221_prop_reg_map_entry *p;
+ int map_size;
+
+ p = p9221_prop_reg_map;
+ map_size = ARRAY_SIZE(p9221_prop_reg_map);
+
+ for (i = 0; i < map_size; i++) {
+ if (p->prop == prop) {
+ if ((set && p->set) || (!set && p->get))
+ return p;
+ }
+ p++;
+ }
+ return NULL;
+}
+
+static int p9221_get_property_reg(struct p9221_charger_data *charger,
+ enum power_supply_property prop,
+ union power_supply_propval *val)
+{
+ int ret;
+ struct p9221_prop_reg_map_entry *p;
+ u32 data;
+
+ pm_runtime_get_sync(charger->dev);
+ if (!charger->resume_complete) {
+ pm_runtime_put_sync(charger->dev);
+ return -EAGAIN;
+ }
+ pm_runtime_put_sync(charger->dev);
+
+ p = p9221_get_map_entry(charger, prop, false);
+ if (p == NULL)
+ return -EINVAL;
+
+ if (!charger->online)
+ return -ENODEV;
+
+ ret = p9221_reg_read_cooked(charger, p->reg, &data);
+ if (ret)
+ return ret;
+
+ val->intval = data;
+ return 0;
+}
+
+static int p9221_set_property_reg(struct p9221_charger_data *charger,
+ enum power_supply_property prop,
+ const union power_supply_propval *val)
+{
+ struct p9221_prop_reg_map_entry *p;
+
+ p = p9221_get_map_entry(charger, prop, true);
+ if (p == NULL)
+ return -EINVAL;
+
+ if (!charger->online)
+ return -ENODEV;
+
+ return p9221_reg_write_cooked(charger, p->reg, val->intval);
+}
+
+static void p9221_abort_transfers(struct p9221_charger_data *charger)
+{
+ /* Abort all transfers */
+ cancel_delayed_work(&charger->tx_work);
+ charger->tx_busy = false;
+ charger->tx_done = true;
+ charger->rx_done = true;
+ charger->rx_len = 0;
+ sysfs_notify(&charger->dev->kobj, NULL, "txbusy");
+ sysfs_notify(&charger->dev->kobj, NULL, "txdone");
+ sysfs_notify(&charger->dev->kobj, NULL, "rxdone");
+}
+
+/*
+ * Put the default ICL back to BPP, reset OCP voter
+ * @pre charger && charger->dc_icl_votable && charger->client->dev
+ */
+static void p9221_vote_defaults(struct p9221_charger_data *charger)
+{
+ int ret;
+
+ if (!charger->dc_icl_votable) {
+ dev_err(&charger->client->dev,
+ "Could not vote DC_ICL - no votable\n");
+ return;
+ }
+
+ ret = vote(charger->dc_icl_votable, P9221_WLC_VOTER, true,
+ P9221_DC_ICL_BPP_UA);
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Could not vote DC_ICL %d\n", ret);
+
+ ret = vote(charger->dc_icl_votable, P9221_OCP_VOTER, true,
+ P9221_DC_ICL_EPP_UA);
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Could not reset OCP DC_ICL voter %d\n", ret);
+}
+
+static void p9221_set_offline(struct p9221_charger_data *charger)
+{
+ dev_info(&charger->client->dev, "Set offline\n");
+
+ charger->online = false;
+
+ /* Reset PP buf so we can get a new serial number next time around */
+ charger->pp_buf_valid = false;
+
+ p9221_abort_transfers(charger);
+ cancel_delayed_work(&charger->dcin_work);
+ p9221_icl_ramp_reset(charger);
+ del_timer(&charger->vrect_timer);
+
+ p9221_vote_defaults(charger);
+}
+
+static void p9221_tx_work(struct work_struct *work)
+{
+ struct p9221_charger_data *charger = container_of(work,
+ struct p9221_charger_data, tx_work.work);
+
+ dev_info(&charger->client->dev, "timeout waiting for tx complete\n");
+
+ charger->tx_busy = false;
+ charger->tx_done = true;
+ sysfs_notify(&charger->dev->kobj, NULL, "txbusy");
+ sysfs_notify(&charger->dev->kobj, NULL, "txdone");
+}
+
+static void p9221_vrect_timer_handler(unsigned long data)
+{
+ struct p9221_charger_data *charger = (struct p9221_charger_data *)data;
+
+ dev_info(&charger->client->dev,
+ "timeout waiting for VRECT, online=%d\n", charger->online);
+ pm_relax(charger->dev);
+}
+
+static void p9221_dcin_work(struct work_struct *work)
+{
+ struct p9221_charger_data *charger = container_of(work,
+ struct p9221_charger_data, dcin_work.work);
+
+ dev_info(&charger->client->dev,
+ "timeout waiting for dc-in, online=%d\n", charger->online);
+
+ if (charger->online)
+ p9221_set_offline(charger);
+
+ power_supply_changed(charger->wc_psy);
+
+ pm_relax(charger->dev);
+}
+
+static const char *p9221_get_tx_id_str(struct p9221_charger_data *charger)
+{
+ int ret;
+ uint32_t tx_id = 0;
+
+ if (!charger->online)
+ return NULL;
+
+ pm_runtime_get_sync(charger->dev);
+ if (!charger->resume_complete) {
+ pm_runtime_put_sync(charger->dev);
+ return NULL;
+ }
+ pm_runtime_put_sync(charger->dev);
+
+ if (p9221_is_epp(charger)) {
+ ret = p9221_reg_read_n(charger, P9221R5_PROP_TX_ID_REG,
+ &tx_id, sizeof(tx_id));
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Failed to read txid %d\n", ret);
+ } else {
+ /*
+ * If pp_buf_valid is true, we have received a serial
+ * number from the Tx, copy it to tx_id. (pp_buf_valid
+ * is left true here until we go offline as we may
+ * read this multiple times.)
+ */
+ if (charger->pp_buf_valid &&
+ sizeof(tx_id) <= P9221R5_MAX_PP_BUF_SIZE)
+ memcpy(&tx_id, &charger->pp_buf[1],
+ sizeof(tx_id));
+ }
+ scnprintf(charger->tx_id_str, sizeof(charger->tx_id_str),
+ "%08x", tx_id);
+ return charger->tx_id_str;
+}
+
+static int p9221_get_property(struct power_supply *psy,
+ enum power_supply_property prop,
+ union power_supply_propval *val)
+{
+ struct p9221_charger_data *charger = power_supply_get_drvdata(psy);
+ int ret = 0;
+
+ switch (prop) {
+ case POWER_SUPPLY_PROP_PRESENT:
+ val->intval = 1;
+ break;
+ case POWER_SUPPLY_PROP_ONLINE:
+ val->intval = charger->online && charger->enabled;
+ break;
+ case POWER_SUPPLY_PROP_SERIAL_NUMBER:
+ val->strval = p9221_get_tx_id_str(charger);
+ if (val->strval == NULL)
+ return -ENODATA;
+ break;
+ case POWER_SUPPLY_PROP_CAPACITY:
+ if (charger->last_capacity > 0)
+ val->intval = charger->last_capacity;
+ else
+ val->intval = 0;
+ break;
+ case POWER_SUPPLY_PROP_CURRENT_MAX:
+ if (!charger->dc_icl_votable)
+ return -EAGAIN;
+
+ ret = get_effective_result(charger->dc_icl_votable);
+ if (ret < 0)
+ break;
+
+ val->intval = ret;
+
+ /* success */
+ ret = 0;
+ break;
+ default:
+ ret = p9221_get_property_reg(charger, prop, val);
+ break;
+ }
+
+ if (ret)
+ dev_dbg(&charger->client->dev,
+ "Couldn't get prop %d, ret=%d\n", prop, ret);
+ return ret;
+}
+
+static int p9221_set_property(struct power_supply *psy,
+ enum power_supply_property prop,
+ const union power_supply_propval *val)
+{
+ struct p9221_charger_data *charger = power_supply_get_drvdata(psy);
+ int ret = 0;
+ bool changed = false;
+
+ switch (prop) {
+ case POWER_SUPPLY_PROP_ONLINE:
+ if ((val->intval < 0) || (val->intval > 1)) {
+ ret = -EINVAL;
+ break;
+ }
+
+ if (charger->enabled == val->intval)
+ break;
+
+ /*
+ * Asserting the enable line will automatically take bring
+ * us online if we are in field. De-asserting the enable
+ * line will automatically take us offline if we are in field.
+ * This is due to the fact that DC in will change state
+ * appropriately when we change the state of this line.
+ */
+ charger->enabled = val->intval;
+
+ dev_warn(&charger->client->dev, "Set enable %d\n",
+ charger->enabled);
+
+ if (charger->pdata->qien_gpio >= 0)
+ gpio_set_value(charger->pdata->qien_gpio,
+ charger->enabled ? 0 : 1);
+
+ changed = true;
+ break;
+
+ case POWER_SUPPLY_PROP_CAPACITY:
+ if (!charger->online || charger->last_capacity == val->intval)
+ break;
+
+ charger->last_capacity = val->intval;
+ ret = p9221_send_csp(charger, charger->last_capacity);
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Could send csp: %d\n", ret);
+ changed = true;
+
+ break;
+ case POWER_SUPPLY_PROP_CURRENT_MAX:
+ if (val->intval < 0) {
+ ret = -EINVAL;
+ break;
+ }
+
+ if (!charger->dc_icl_votable) {
+ ret = -EAGAIN;
+ break;
+ }
+
+ ret = vote(charger->dc_icl_votable, P9221_USER_VOTER, true,
+ val->intval);
+
+ changed = true;
+ break;
+ default:
+ ret = p9221_set_property_reg(charger, prop, val);
+ if (ret == 0)
+ changed = true;
+ break;
+ }
+
+ if (ret)
+ dev_dbg(&charger->client->dev,
+ "Couldn't set prop %d, ret=%d\n", prop, ret);
+
+ if (changed)
+ power_supply_changed(psy);
+
+ return ret;
+}
+
+static int p9221_prop_is_writeable(struct power_supply *psy,
+ enum power_supply_property prop)
+{
+ struct p9221_charger_data *charger = power_supply_get_drvdata(psy);
+
+ if (p9221_get_map_entry(charger, prop, true))
+ return 1;
+
+ return 0;
+}
+
+static int p9221_notifier_cb(struct notifier_block *nb, unsigned long event,
+ void *data)
+{
+ struct power_supply *psy = data;
+ struct p9221_charger_data *charger =
+ container_of(nb, struct p9221_charger_data, nb);
+
+ if (event != PSY_EVENT_PROP_CHANGED)
+ goto out;
+
+ if (strcmp(psy->desc->name, "dc") == 0) {
+ charger->dc_psy = psy;
+ charger->check_dc = true;
+ }
+
+ if (!charger->check_dc)
+ goto out;
+
+ pm_stay_awake(charger->dev);
+
+ if (!schedule_delayed_work(&charger->notifier_work,
+ msecs_to_jiffies(P9221_NOTIFIER_DELAY_MS)))
+ pm_relax(charger->dev);
+
+out:
+ return NOTIFY_OK;
+}
+
+static int p9221_clear_interrupts(struct p9221_charger_data *charger, u16 mask)
+{
+ int ret;
+
+ mutex_lock(&charger->cmd_lock);
+
+ ret = p9221_reg_write_16(charger, P9221R5_INT_CLEAR_REG, mask);
+ if (ret) {
+ dev_err(&charger->client->dev,
+ "Failed to clear INT reg: %d\n", ret);
+ goto out;
+ }
+
+ ret = p9221_set_cmd_reg(charger, P9221_COM_CLEAR_INT_MASK);
+ if (ret) {
+ dev_err(&charger->client->dev,
+ "Failed to reset INT: %d\n", ret);
+ }
+out:
+ mutex_unlock(&charger->cmd_lock);
+ return ret;
+}
+
+/*
+ * Enable interrupts on the P9221, note we don't really need to disable
+ * interrupts since when the device goes out of field, the P9221 is reset.
+ */
+static int p9221_enable_interrupts(struct p9221_charger_data *charger)
+{
+ u16 mask;
+ int ret;
+
+ dev_dbg(&charger->client->dev, "Enable interrupts\n");
+
+ mask = P9221R5_STAT_LIMIT_MASK | P9221R5_STAT_CC_MASK |
+ P9221_STAT_VRECT;
+
+ ret = p9221_clear_interrupts(charger, mask);
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Could not clear interrupts: %d\n", ret);
+
+ ret = p9221_reg_write_8(charger, P9221_INT_ENABLE_REG, mask);
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Could not enable interrupts: %d\n", ret);
+ return ret;
+}
+
+static int p9221_set_dc_icl(struct p9221_charger_data *charger)
+{
+ int icl;
+ int ret;
+
+ if (!charger->dc_icl_votable) {
+ charger->dc_icl_votable = find_votable("DC_ICL");
+ if (!charger->dc_icl_votable) {
+ dev_err(&charger->client->dev,
+ "Could not get votable: DC_ICL\n");
+ return -ENODEV;
+ }
+ }
+
+ /* Default to BPP ICL */
+ icl = P9221_DC_ICL_BPP_UA;
+
+ if (charger->icl_ramp)
+ icl = charger->icl_ramp_ua;
+
+ if (p9221_is_epp(charger))
+ icl = P9221_DC_ICL_EPP_UA;
+
+ dev_info(&charger->client->dev, "Setting ICL %duA ramp=%d\n", icl,
+ charger->icl_ramp);
+ ret = vote(charger->dc_icl_votable, P9221_WLC_VOTER, true, icl);
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Could not vote DC_ICL %d\n", ret);
+
+ /* Increase the IOUT limit */
+ ret = p9221_reg_write_cooked(charger, P9221R5_ILIM_SET_REG,
+ P9221R5_ILIM_MAX_UA);
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Could not set rx_iout limit reg: %d\n", ret);
+
+ return ret;
+}
+
+static enum alarmtimer_restart p9221_icl_ramp_alarm_cb(struct alarm *alarm,
+ ktime_t now)
+{
+ struct p9221_charger_data *charger =
+ container_of(alarm, struct p9221_charger_data,
+ icl_ramp_alarm);
+
+ dev_info(&charger->client->dev, "ICL ramp alarm, ramp=%d\n",
+ charger->icl_ramp);
+
+ /* Alarm is in atomic context, schedule work to complete the task */
+ pm_stay_awake(charger->dev);
+ schedule_delayed_work(&charger->icl_ramp_work, msecs_to_jiffies(100));
+
+ return ALARMTIMER_NORESTART;
+}
+
+static void p9221_icl_ramp_work(struct work_struct *work)
+{
+ struct p9221_charger_data *charger = container_of(work,
+ struct p9221_charger_data, icl_ramp_work.work);
+
+ pm_runtime_get_sync(charger->dev);
+ if (!charger->resume_complete) {
+ pm_runtime_put_sync(charger->dev);
+ schedule_delayed_work(&charger->icl_ramp_work,
+ msecs_to_jiffies(100));
+ dev_dbg(&charger->client->dev, "Ramp reschedule\n");
+ return;
+ }
+ pm_runtime_put_sync(charger->dev);
+
+ dev_info(&charger->client->dev, "ICL ramp work, ramp=%d\n",
+ charger->icl_ramp);
+
+ charger->icl_ramp = true;
+ p9221_set_dc_icl(charger);
+
+ pm_relax(charger->dev);
+}
+
+static void p9221_icl_ramp_reset(struct p9221_charger_data *charger)
+{
+ dev_info(&charger->client->dev, "ICL ramp reset, ramp=%d\n",
+ charger->icl_ramp);
+
+ charger->icl_ramp = false;
+
+ if (alarm_try_to_cancel(&charger->icl_ramp_alarm) < 0)
+ dev_warn(&charger->client->dev, "Couldn't cancel icl_ramp_alarm\n");
+ cancel_delayed_work(&charger->icl_ramp_work);
+}
+
+static void p9221_icl_ramp_start(struct p9221_charger_data *charger)
+{
+ /* Only ramp on BPP at this time */
+ if (p9221_is_epp(charger))
+ return;
+
+ p9221_icl_ramp_reset(charger);
+
+ dev_info(&charger->client->dev, "ICL ramp set alarm %dms, %dua, ramp=%d\n",
+ charger->icl_ramp_delay_ms, charger->icl_ramp_ua,
+ charger->icl_ramp);
+
+ alarm_start_relative(&charger->icl_ramp_alarm,
+ ms_to_ktime(charger->icl_ramp_delay_ms));
+}
+
+static void p9221_set_online(struct p9221_charger_data *charger)
+{
+ int ret;
+ u8 cid = 5;
+
+ dev_info(&charger->client->dev, "Set online\n");
+
+ charger->online = true;
+ charger->tx_busy = false;
+ charger->tx_done = true;
+ charger->rx_done = false;
+ charger->last_capacity = -1;
+
+ ret = p9221_reg_read_8(charger, P9221_CUSTOMER_ID_REG, &cid);
+ if (ret)
+ dev_err(&charger->client->dev, "Could not get ID: %d\n", ret);
+ else
+ charger->cust_id = cid;
+
+ dev_info(&charger->client->dev, "P9221 cid: %02x\n", charger->cust_id);
+
+ ret = p9221_enable_interrupts(charger);
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Could not enable interrupts: %d\n", ret);
+
+ /* NOTE: depends on _is_epp() which is not valid until DC_IN */
+ p9221_write_fod(charger);
+}
+
+/* 2 * P9221_NOTIFIER_DELAY_MS from VRECTON */
+static void p9221_notifier_check_dc(struct p9221_charger_data *charger)
+{
+ int ret;
+ union power_supply_propval prop;
+
+ charger->check_dc = false;
+
+ if (!charger->dc_psy)
+ return;
+
+ ret = power_supply_get_property(charger->dc_psy,
+ POWER_SUPPLY_PROP_PRESENT, &prop);
+ if (ret) {
+ dev_err(&charger->client->dev,
+ "Error getting charging status: %d\n", ret);
+ return;
+ }
+
+ dev_info(&charger->client->dev, "dc status is %d\n", prop.intval);
+
+ /*
+ * We now have confirmation from DC_IN, kill the timer, charger->online
+ * will be set by this function.
+ */
+ cancel_delayed_work(&charger->dcin_work);
+ del_timer(&charger->vrect_timer);
+
+ /*
+ * Always write FOD, check dc_icl, send CSP
+ */
+ if (prop.intval) {
+ p9221_set_dc_icl(charger);
+ p9221_write_fod(charger);
+ if (charger->last_capacity > 0)
+ p9221_send_csp(charger, charger->last_capacity);
+ p9221_icl_ramp_start(charger);
+ }
+
+ /* We may have already gone online during check_det */
+ if (charger->online == prop.intval)
+ goto out;
+
+ if (prop.intval)
+ p9221_set_online(charger);
+ else
+ p9221_set_offline(charger);
+
+out:
+ dev_info(&charger->client->dev, "trigger wc changed on:%d in:%d\n",
+ charger->online, prop.intval);
+ power_supply_changed(charger->wc_psy);
+}
+
+/* P9221_NOTIFIER_DELAY_MS from VRECTON */
+bool p9221_notifier_check_det(struct p9221_charger_data *charger)
+{
+ bool relax = true;
+
+ del_timer(&charger->vrect_timer);
+
+ if (charger->online)
+ goto done;
+
+ dev_info(&charger->client->dev, "detected wlc, trigger wc changed\n");
+ /* send out a FOD but is_epp() is still invalid */
+ p9221_set_online(charger);
+ power_supply_changed(charger->wc_psy);
+
+ /* Give the dc-in 2 seconds to come up. */
+ dev_info(&charger->client->dev, "start dc-in timer\n");
+ cancel_delayed_work_sync(&charger->dcin_work);
+ schedule_delayed_work(&charger->dcin_work,
+ msecs_to_jiffies(P9221_DCIN_TIMEOUT_MS));
+ relax = false;
+
+done:
+ charger->check_det = false;
+
+ return relax;
+}
+
+static void p9221_notifier_work(struct work_struct *work)
+{
+ struct p9221_charger_data *charger = container_of(work,
+ struct p9221_charger_data, notifier_work.work);
+ bool relax = true;
+
+ dev_info(&charger->client->dev, "Notifier work: on:%d dc:%d det:%d\n",
+ charger->online, charger->check_dc, charger->check_det);
+
+ if (charger->check_det)
+ relax = p9221_notifier_check_det(charger);
+
+ if (charger->check_dc)
+ p9221_notifier_check_dc(charger);
+
+ if (relax)
+ pm_relax(charger->dev);
+}
+
+static ssize_t p9221_add_reg_buffer(struct p9221_charger_data *charger,
+ char *buf, size_t count, u16 reg, int width,
+ bool cooked, const char *name, char *fmt)
+{
+ u32 val;
+ int ret;
+ int added = 0;
+
+ if (cooked)
+ ret = p9221_reg_read_cooked(charger, reg, &val);
+ else if (width == 16) {
+ u16 val16 = 0;
+ ret = p9221_reg_read_16(charger, reg, &val16);
+ val = val16;
+ } else {
+ u8 val8 = 0;
+ ret = p9221_reg_read_8(charger, reg, &val8);
+ val = val8;
+ }
+
+ added += scnprintf(buf + count, PAGE_SIZE - count, "%s", name);
+ count += added;
+ if (ret)
+ added += scnprintf(buf + count, PAGE_SIZE - count,
+ "err %d\n", ret);
+ else
+ added += scnprintf(buf + count, PAGE_SIZE - count, fmt, val);
+
+ return added;
+}
+
+static ssize_t p9221_show_version(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+ int count = 0;
+ int i;
+ int ret;
+ u8 val8 = 0;
+
+ if (!charger->online)
+ return -ENODEV;
+
+ count += p9221_add_reg_buffer(charger, buf, count, P9221_CHIP_ID_REG,
+ 16, 0, "chip id : ", "%04x\n");
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221_CHIP_REVISION_REG, 8, 0,
+ "chip rev : ", "%02x\n");
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221_CUSTOMER_ID_REG, 8, 0,
+ "cust id : ", "%02x\n");
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221_OTP_FW_MAJOR_REV_REG, 16, 0,
+ "otp fw maj : ", "%04x\n");
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221_OTP_FW_MINOR_REV_REG, 16, 0,
+ "otp fw min : ", "%04x\n");
+
+ count += scnprintf(buf + count, PAGE_SIZE - count, "otp fw date: ");
+ for (i = 0; i < P9221_OTP_FW_DATE_SIZE; i++) {
+ ret = p9221_reg_read_8(charger,
+ P9221_OTP_FW_DATE_REG + i, &val8);
+ if (val8)
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "%c", val8);
+ }
+
+ count += scnprintf(buf + count, PAGE_SIZE - count, "\notp fw time: ");
+ for (i = 0; i < P9221_OTP_FW_TIME_SIZE; i++) {
+ ret = p9221_reg_read_8(charger,
+ P9221_OTP_FW_TIME_REG + i, &val8);
+ if (val8)
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "%c", val8);
+ }
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221_SRAM_FW_MAJOR_REV_REG, 16, 0,
+ "\nram fw maj : ", "%04x\n");
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221_SRAM_FW_MINOR_REV_REG, 16, 0,
+ "ram fw min : ", "%04x\n");
+
+ count += scnprintf(buf + count, PAGE_SIZE - count, "ram fw date: ");
+ for (i = 0; i < P9221_SRAM_FW_DATE_SIZE; i++) {
+ ret = p9221_reg_read_8(charger,
+ P9221_SRAM_FW_DATE_REG + i, &val8);
+ if (val8)
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "%c", val8);
+ }
+
+ count += scnprintf(buf + count, PAGE_SIZE - count, "\nram fw time: ");
+ for (i = 0; i < P9221_SRAM_FW_TIME_SIZE; i++) {
+ ret = p9221_reg_read_8(charger,
+ P9221_SRAM_FW_TIME_REG + i, &val8);
+ if (val8)
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "%c", val8);
+ }
+
+ count += scnprintf(buf + count, PAGE_SIZE - count, "\n");
+ return count;
+}
+
+static DEVICE_ATTR(version, 0444, p9221_show_version, NULL);
+
+static ssize_t p9221_show_status(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+ int count = 0;
+ int ret;
+ u8 tmp[P9221R5_NUM_FOD];
+ uint32_t tx_id = 0;
+
+ if (!charger->online)
+ return -ENODEV;
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221_STATUS_REG, 16, 0,
+ "status : ", "%04x\n");
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221_INT_REG, 16, 0,
+ "int : ", "%04x\n");
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221_INT_ENABLE_REG, 16, 0,
+ "int_enable : ", "%04x\n");
+
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221R5_SYSTEM_MODE_REG, 8, 0,
+ "mode : ", "%02x\n");
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221R5_VOUT_REG, 16, 1,
+ "vout : ", "%d uV\n");
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221R5_VRECT_REG, 16, 1,
+ "vrect : ", "%d uV\n");
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221R5_IOUT_REG, 16, 1,
+ "iout : ", "%d uA\n");
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221R5_ILIM_SET_REG, 16, 1,
+ "ilim : ", "%d uA\n");
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221R5_OP_FREQ_REG, 16, 1,
+ "freq : ", "%d hz\n");
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "tx_busy : %d\n", charger->tx_busy);
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "tx_done : %d\n", charger->tx_done);
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "rx_done : %d\n", charger->rx_done);
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "tx_len : %d\n", charger->tx_len);
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "rx_len : %d\n", charger->rx_len);
+ p9221_reg_read_n(charger, P9221R5_PROP_TX_ID_REG, &tx_id,
+ sizeof(tx_id));
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "tx_id : %08x (%s)\n", tx_id,
+ p9221_get_tx_id_str(charger));
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221R5_ALIGN_X_ADC_REG, 8, 1,
+ "align_x : ", "%d\n");
+
+ count += p9221_add_reg_buffer(charger, buf, count,
+ P9221R5_ALIGN_Y_ADC_REG, 8, 1,
+ "align_y : ", "%d\n");
+
+ /* FOD Register */
+ ret = p9221_reg_read_n(charger, P9221R5_FOD_REG, tmp, P9221R5_NUM_FOD);
+ count += scnprintf(buf + count, PAGE_SIZE - count, "fod : ");
+ if (ret)
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "err %d\n", ret);
+ else {
+ count += p9221_hex_str(tmp, P9221R5_NUM_FOD, buf + count, count,
+ false);
+ count += scnprintf(buf + count, PAGE_SIZE - count, "\n");
+ }
+
+ /* Device tree FOD entries */
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "dt fod : (n=%d) ", charger->pdata->fod_num);
+ count += p9221_hex_str(charger->pdata->fod, charger->pdata->fod_num,
+ buf + count, PAGE_SIZE - count, false);
+
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "\ndt fod-epp : (n=%d) ",
+ charger->pdata->fod_epp_num);
+ count += p9221_hex_str(charger->pdata->fod_epp,
+ charger->pdata->fod_epp_num,
+ buf + count, PAGE_SIZE - count, false);
+
+ count += scnprintf(buf + count, PAGE_SIZE - count,
+ "\npp buf : (v=%d) ", charger->pp_buf_valid);
+ count += p9221_hex_str(charger->pp_buf, sizeof(charger->pp_buf),
+ buf + count, PAGE_SIZE - count, false);
+
+ count += scnprintf(buf + count, PAGE_SIZE - count, "\n");
+ return count;
+}
+
+static DEVICE_ATTR(status, 0444, p9221_show_status, NULL);
+
+static ssize_t p9221_show_count(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ return scnprintf(buf, PAGE_SIZE, "%u\n", charger->count);
+}
+
+static ssize_t p9221_store_count(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+ int ret;
+ u8 cnt;
+
+ ret = kstrtou8(buf, 0, &cnt);
+ if (ret < 0)
+ return ret;
+ charger->count = cnt;
+ return count;
+}
+
+static DEVICE_ATTR(count, 0644, p9221_show_count, p9221_store_count);
+
+static ssize_t p9221_show_icl_ramp_delay_ms(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n", charger->icl_ramp_delay_ms);
+}
+
+static ssize_t p9221_store_icl_ramp_delay_ms(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+ int ret;
+ u32 ms;
+
+ ret = kstrtou32(buf, 10, &ms);
+ if (ret < 0)
+ return ret;
+ charger->icl_ramp_delay_ms = ms;
+ return count;
+}
+
+static DEVICE_ATTR(icl_ramp_delay_ms, 0644,
+ p9221_show_icl_ramp_delay_ms,
+ p9221_store_icl_ramp_delay_ms);
+
+static ssize_t p9221_show_icl_ramp_ua(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ return scnprintf(buf, PAGE_SIZE, "%d\n", charger->icl_ramp_ua);
+}
+
+static ssize_t p9221_store_icl_ramp_ua(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+ int ret;
+ u32 ua;
+
+ ret = kstrtou32(buf, 10, &ua);
+ if (ret < 0)
+ return ret;
+ charger->icl_ramp_ua = ua;
+ return count;
+}
+
+static DEVICE_ATTR(icl_ramp_ua, 0644,
+ p9221_show_icl_ramp_ua, p9221_store_icl_ramp_ua);
+
+static ssize_t p9221_show_addr(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ return scnprintf(buf, PAGE_SIZE, "%04x\n", charger->addr);
+}
+
+static ssize_t p9221_store_addr(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+ int ret;
+ u16 addr;
+
+ ret = kstrtou16(buf, 16, &addr);
+ if (ret < 0)
+ return ret;
+ charger->addr = addr;
+ return count;
+}
+
+static DEVICE_ATTR(addr, 0644, p9221_show_addr, p9221_store_addr);
+
+static ssize_t p9221_show_data(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+ u8 reg[256];
+ int ret;
+ int i;
+ ssize_t len = 0;
+
+ if (!charger->count || (charger->addr > (0xFFFF - charger->count)))
+ return -EINVAL;
+
+ if (!charger->online)
+ return -ENODEV;
+
+ ret = p9221_reg_read_n(charger, charger->addr, reg, charger->count);
+ if (ret)
+ return ret;
+
+ for (i = 0; i < charger->count; i++) {
+ len += scnprintf(buf + len, PAGE_SIZE - len, "%02x: %02x\n",
+ charger->addr + i, reg[i]);
+ }
+ return len;
+}
+
+static ssize_t p9221_store_data(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+ u8 reg[256];
+ int i = 0;
+ int ret = 0;
+ char *data;
+
+ if (!charger->count || (charger->addr > (0xFFFF - charger->count)))
+ return -EINVAL;
+
+ if (!charger->online)
+ return -ENODEV;
+
+ data = kmalloc(strlen(buf) + 1, GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ strcpy(data, buf);
+ while(data && i < charger->count) {
+ char *d = strsep(&data, " ");
+ if (*d) {
+ ret = kstrtou8(d, 16, ®[i]);
+ if (ret)
+ break;
+ i++;
+ }
+ }
+ if ((i != charger->count) || ret) {
+ ret = -EINVAL;
+ goto out;
+ }
+
+ ret = p9221_reg_write_n(charger, charger->addr, reg, charger->count);
+ if (ret)
+ goto out;
+ ret = count;
+
+out:
+ kfree(data);
+ return ret;
+}
+
+static DEVICE_ATTR(data, 0644, p9221_show_data, p9221_store_data);
+
+static ssize_t p9221_store_ccreset(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+ int ret;
+
+ ret = p9221_send_ccreset(charger);
+ if (ret)
+ return ret;
+ return count;
+}
+
+static DEVICE_ATTR(ccreset, 0200, NULL, p9221_store_ccreset);
+
+static ssize_t p9221_show_rxdone(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ buf[0] = charger->rx_done ? '1' : '0';
+ buf[1] = 0;
+ return 1;
+}
+
+static DEVICE_ATTR(rxdone, 0444, p9221_show_rxdone, NULL);
+
+static ssize_t p9221_show_rxlen(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ return scnprintf(buf, PAGE_SIZE, "%hu\n", charger->rx_len);
+}
+
+static DEVICE_ATTR(rxlen, 0444, p9221_show_rxlen, NULL);
+
+static ssize_t p9221_show_txdone(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ buf[0] = charger->tx_done ? '1' : '0';
+ buf[1] = 0;
+ return 1;
+}
+
+static DEVICE_ATTR(txdone, 0444, p9221_show_txdone, NULL);
+
+static ssize_t p9221_show_txbusy(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ buf[0] = charger->tx_busy ? '1' : '0';
+ buf[1] = 0;
+ return 1;
+}
+
+static DEVICE_ATTR(txbusy, 0444, p9221_show_txbusy, NULL);
+
+static ssize_t p9221_store_txlen(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+ int ret;
+ u16 len;
+
+ ret = kstrtou16(buf, 16, &len);
+ if (ret < 0)
+ return ret;
+ if (ret > P9221R5_COM_CHAN_SEND_SIZE_REG)
+ return -EINVAL;
+
+ cancel_delayed_work_sync(&charger->tx_work);
+
+ charger->tx_len = len;
+ charger->tx_done = false;
+ ret = p9221_send_data(charger);
+ if (ret) {
+ charger->tx_done = true;
+ return ret;
+ }
+
+ schedule_delayed_work(&charger->tx_work,
+ msecs_to_jiffies(P9221_TX_TIMEOUT_MS));
+
+ return count;
+}
+
+static DEVICE_ATTR(txlen, 0200, NULL, p9221_store_txlen);
+
+static struct attribute *p9221_attributes[] = {
+ &dev_attr_version.attr,
+ &dev_attr_status.attr,
+ &dev_attr_addr.attr,
+ &dev_attr_count.attr,
+ &dev_attr_data.attr,
+ &dev_attr_ccreset.attr,
+ &dev_attr_txbusy.attr,
+ &dev_attr_txdone.attr,
+ &dev_attr_txlen.attr,
+ &dev_attr_rxlen.attr,
+ &dev_attr_rxdone.attr,
+ &dev_attr_icl_ramp_ua.attr,
+ &dev_attr_icl_ramp_delay_ms.attr,
+ NULL
+};
+
+static ssize_t p9221_rxdata_read(struct file *filp, struct kobject *kobj,
+ struct bin_attribute *bin_attr,
+ char *buf, loff_t pos, size_t size)
+{
+ struct p9221_charger_data *charger;
+ charger = dev_get_drvdata(container_of(kobj, struct device, kobj));
+
+ memcpy(buf, &charger->rx_buf[pos], size);
+ charger->rx_done = false;
+ return size;
+}
+
+static struct bin_attribute bin_attr_rxdata = {
+ .attr = {
+ .name = "rxdata",
+ .mode = 0400,
+ },
+ .read = p9221_rxdata_read,
+ .size = P9221R5_DATA_RECV_BUF_SIZE,
+};
+
+static ssize_t p9221_txdata_read(struct file *filp, struct kobject *kobj,
+ struct bin_attribute *bin_attr,
+ char *buf, loff_t pos, size_t size)
+{
+ struct p9221_charger_data *charger;
+ charger = dev_get_drvdata(container_of(kobj, struct device, kobj));
+
+ memcpy(buf, &charger->tx_buf[pos], size);
+ return size;
+}
+
+static ssize_t p9221_txdata_write(struct file *filp, struct kobject *kobj,
+ struct bin_attribute *bin_attr,
+ char *buf, loff_t pos, size_t size)
+{
+ struct p9221_charger_data *charger;
+ charger = dev_get_drvdata(container_of(kobj, struct device, kobj));
+
+ memcpy(&charger->tx_buf[pos], buf, size);
+ return size;
+}
+
+static struct bin_attribute bin_attr_txdata = {
+ .attr = {
+ .name = "txdata",
+ .mode = 0600,
+ },
+ .read = p9221_txdata_read,
+ .write = p9221_txdata_write,
+ .size = P9221R5_DATA_SEND_BUF_SIZE,
+};
+
+static struct bin_attribute *p9221_bin_attributes[] = {
+ &bin_attr_txdata,
+ &bin_attr_rxdata,
+ NULL,
+};
+
+static const struct attribute_group p9221_attr_group = {
+ .attrs = p9221_attributes,
+ .bin_attrs = p9221_bin_attributes,
+};
+
+static void print_current_samples(struct p9221_charger_data *charger,
+ u32 *iout_val, int count)
+{
+ int i;
+ char temp[P9221R5_OVER_CHECK_NUM * 9 + 1] = { 0 };
+
+ for (i = 0; i < count ; i++)
+ scnprintf(temp + i * 9, sizeof(temp) - i * 9,
+ "%08x ", iout_val[i]);
+
+ dev_info(&charger->client->dev, "OVER IOUT_SAMPLES: %s\n", temp);
+}
+
+/*
+ * Number of times to poll the status to see if the current limit condition
+ * was transient or not.
+ */
+static void p9221_over_handle(struct p9221_charger_data *charger,
+ u16 irq_src)
+{
+ u8 reason = 0;
+ int i;
+ int ret;
+ int ovc_count = 0;
+ u32 iout_val[P9221R5_OVER_CHECK_NUM] = { 0 };
+
+ dev_err(&charger->client->dev, "Received OVER INT: %02x\n", irq_src);
+
+ if (irq_src & P9221R5_STAT_OVV) {
+ reason = P9221_EOP_OVER_VOLT;
+ goto send_eop;
+ }
+
+ if (irq_src & P9221R5_STAT_OVT) {
+ reason = P9221_EOP_OVER_TEMP;
+ goto send_eop;
+ }
+
+ if ((irq_src & P9221R5_STAT_UV) && !(irq_src & P9221R5_STAT_OVC))
+ return;
+
+ /* Overcurrent, reduce ICL and poll to absorb any transients */
+
+ if (charger->dc_icl_votable) {
+ int icl;
+
+ icl = get_effective_result_locked(charger->dc_icl_votable);
+ if (icl < 0) {
+ dev_err(&charger->client->dev,
+ "Failed to read ICL (%d)\n", icl);
+ } else if (icl > OVC_BACKOFF_LIMIT) {
+ icl -= OVC_BACKOFF_AMOUNT;
+
+ ret = vote(charger->dc_icl_votable,
+ P9221_OCP_VOTER, true,
+ icl);
+ dev_err(&charger->client->dev,
+ "Reduced ICL to %d (%d)\n", icl, ret);
+ }
+ }
+
+ reason = P9221_EOP_OVER_CURRENT;
+ for (i = 0; i < P9221R5_OVER_CHECK_NUM; i++) {
+ ret = p9221_clear_interrupts(charger,
+ irq_src & P9221R5_STAT_LIMIT_MASK);
+ msleep(50);
+ if (ret)
+ continue;
+
+ ret = p9221_reg_read_cooked(charger, P9221R5_IOUT_REG,
+ &iout_val[i]);
+ if (ret) {
+ dev_err(&charger->client->dev,
+ "Failed to read IOUT[%d]: %d\n", i, ret);
+ continue;
+ } else if (iout_val[i] > OVC_THRESHOLD) {
+ ovc_count++;
+ }
+
+ ret = p9221_reg_read_16(charger, P9221_STATUS_REG, &irq_src);
+ if (ret) {
+ dev_err(&charger->client->dev,
+ "Failed to read status: %d\n", ret);
+ continue;
+ }
+
+ if ((irq_src & P9221R5_STAT_OVC) == 0) {
+ print_current_samples(charger, iout_val, i + 1);
+ dev_info(&charger->client->dev,
+ "OVER condition %04x cleared after %d tries\n",
+ irq_src, i);
+ return;
+ }
+
+ dev_err(&charger->client->dev,
+ "OVER status is still %04x, retry\n", irq_src);
+ }
+
+ if (ovc_count < OVC_LIMIT) {
+ print_current_samples(charger, iout_val,
+ P9221R5_OVER_CHECK_NUM);
+ dev_info(&charger->client->dev,
+ "ovc_threshold=%d, ovc_count=%d, ovc_limit=%d\n",
+ OVC_THRESHOLD, ovc_count, OVC_LIMIT);
+ return;
+ }
+
+send_eop:
+ dev_err(&charger->client->dev,
+ "OVER is %04x, sending EOP %d\n", irq_src, reason);
+
+ ret = p9221_send_eop(charger, reason);
+ if (ret)
+ dev_err(&charger->client->dev,
+ "Failed to send EOP %d: %d\n", reason, ret);
+}
+
+/* Handler for R5 and R7 chips */
+static void p9221_irq_handler(struct p9221_charger_data *charger, u16 irq_src)
+{
+ int res;
+
+ if (irq_src & P9221R5_STAT_LIMIT_MASK)
+ p9221_over_handle(charger, irq_src);
+
+ /* Receive complete */
+ if (irq_src & P9221R5_STAT_CCDATARCVD) {
+ uint8_t rxlen = 0;
+ res = p9221_reg_read_8(charger, P9221R5_COM_CHAN_RECV_SIZE_REG,
+ &rxlen);
+ if (res) {
+ dev_err(&charger->client->dev,
+ "Failed to read len: %d\n", res);
+ rxlen = 0;
+ }
+ if (rxlen) {
+ res = p9221_reg_read_n(charger,
+ P9221R5_DATA_RECV_BUF_START,
+ charger->rx_buf, rxlen);
+ if (res)
+ dev_err(&charger->client->dev,
+ "Failed to read len: %d\n", res);
+
+ charger->rx_len = rxlen;
+ charger->rx_done = true;
+ sysfs_notify(&charger->dev->kobj, NULL, "rxdone");
+ }
+ }
+
+ /* Send complete */
+ if (irq_src & P9221R5_STAT_CCSENDBUSY) {
+ charger->tx_busy = false;
+ charger->tx_done = true;
+ cancel_delayed_work(&charger->tx_work);
+ sysfs_notify(&charger->dev->kobj, NULL, "txbusy");
+ sysfs_notify(&charger->dev->kobj, NULL, "txdone");
+ }
+
+ /* Proprietary packet */
+ if (irq_src & P9221R5_STAT_PPRCVD) {
+ const size_t maxsz = sizeof(charger->pp_buf) * 3 + 1;
+ char s[maxsz];
+
+ res = p9221_reg_read_n(charger,
+ P9221R5_DATA_RECV_BUF_START,
+ charger->pp_buf,
+ sizeof(charger->pp_buf));
+ if (res)
+ dev_err(&charger->client->dev,
+ "Failed to read PP len: %d\n", res);
+
+ /* We only care about PP which come with 0x4F header */
+ charger->pp_buf_valid = (charger->pp_buf[0] == 0x4F);
+
+ p9221_hex_str(charger->pp_buf, sizeof(charger->pp_buf),
+ s, maxsz, false);
+ dev_info(&charger->client->dev, "Received PP: %s\n", s);
+ }
+
+ /* CC Reset complete */
+ if (irq_src & P9221R5_STAT_CCRESET)
+ p9221_abort_transfers(charger);
+}
+
+static irqreturn_t p9221_irq_thread(int irq, void *irq_data)
+{
+ struct p9221_charger_data *charger = irq_data;
+ int ret;
+ u16 irq_src = 0;
+
+ pm_runtime_get_sync(charger->dev);
+ if (!charger->resume_complete) {
+ pm_runtime_put_sync(charger->dev);
+ return -EAGAIN;
+ }
+ pm_runtime_put_sync(charger->dev);
+
+ ret = p9221_reg_read_16(charger, P9221_INT_REG, &irq_src);
+ if (ret) {
+ dev_err(&charger->client->dev,
+ "Failed to read INT reg: %d\n", ret);
+ goto out;
+ }
+
+ dev_info(&charger->client->dev, "INT: %04x\n", irq_src);
+ if (!irq_src)
+ goto out;
+
+ ret = p9221_clear_interrupts(charger, irq_src);
+ if (ret) {
+ dev_err(&charger->client->dev,
+ "Failed to clear INT reg: %d\n", ret);
+ goto out;
+ }
+
+ if (irq_src & P9221_STAT_VRECT) {
+ dev_info(&charger->client->dev,
+ "Received VRECTON, online=%d\n", charger->online);
+ if (!charger->online) {
+ charger->check_det = true;
+ pm_stay_awake(charger->dev);
+
+ if (!schedule_delayed_work(&charger->notifier_work,
+ msecs_to_jiffies(P9221_NOTIFIER_DELAY_MS))) {
+ pm_relax(charger->dev);
+ }
+ }
+ }
+
+ p9221_irq_handler(charger, irq_src);
+
+out:
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t p9221_irq_det_thread(int irq, void *irq_data)
+{
+ struct p9221_charger_data *charger = irq_data;
+
+ /* If we are already online, just ignore the interrupt. */
+ if (charger->online)
+ return IRQ_HANDLED;
+
+ /*
+ * This interrupt will wake the device if it's suspended,
+ * but it is not reliable enough to trigger the charging indicator.
+ * Give ourselves 2 seconds for the VRECTON interrupt to appear
+ * before we put up the charging indicator.
+ */
+ mod_timer(&charger->vrect_timer,
+ jiffies + msecs_to_jiffies(P9221_VRECT_TIMEOUT_MS));
+ pm_stay_awake(charger->dev);
+
+ return IRQ_HANDLED;
+}
+
+static int p9221_parse_dt(struct device *dev,
+ struct p9221_charger_platform_data *pdata)
+{
+ int ret = 0;
+ u32 data;
+ struct device_node *node = dev->of_node;
+
+ /* Enable */
+ ret = of_get_named_gpio(node, "idt,gpio_qien", 0);
+ pdata->qien_gpio = ret;
+ if (ret < 0)
+ dev_warn(dev, "unable to read idt,gpio_qien from dt: %d\n",
+ ret);
+ else
+ dev_info(dev, "enable gpio:%d", pdata->qien_gpio);
+
+ /* Main IRQ */
+ ret = of_get_named_gpio(node, "idt,irq_gpio", 0);
+ if (ret < 0) {
+ dev_err(dev, "unable to read idt,irq_gpio from dt: %d\n", ret);
+ return ret;
+ }
+ pdata->irq_gpio = ret;
+ pdata->irq_int = gpio_to_irq(pdata->irq_gpio);
+ dev_info(dev, "gpio:%d, gpio_irq:%d\n", pdata->irq_gpio,
+ pdata->irq_int);
+
+ /* Optional Detect IRQ */
+ ret = of_get_named_gpio(node, "idt,irq_det_gpio", 0);
+ pdata->irq_det_gpio = ret;
+ if (ret < 0) {
+ dev_warn(dev, "unable to read idt,irq_det_gpio from dt: %d\n",
+ ret);
+ } else {
+ pdata->irq_det_int = gpio_to_irq(pdata->irq_det_gpio);
+ dev_info(dev, "det gpio:%d, det gpio_irq:%d\n",
+ pdata->irq_det_gpio, pdata->irq_det_int);
+ }
+
+ /* Optional VOUT max */
+ pdata->max_vout_mv = P9221_MAX_VOUT_SET_MV_DEFAULT;
+ ret = of_property_read_u32(node, "max_vout_mv", &data);
+ if (ret == 0) {
+ if (data < 3500 || data > 29000)
+ dev_err(dev, "max_vout_mv out of range %d\n", data);
+ else
+ pdata->max_vout_mv = data;
+ }
+
+ /* Optional FOD data */
+ pdata->fod_num =
+ of_property_count_elems_of_size(node, "fod", sizeof(u8));
+ if (pdata->fod_num <= 0) {
+ dev_err(dev, "No dt fod provided (%d)\n", pdata->fod_num);
+ pdata->fod_num = 0;
+ } else {
+ if (pdata->fod_num > P9221R5_NUM_FOD) {
+ dev_err(dev,
+ "Incorrect num of FOD %d, using first %d\n",
+ pdata->fod_num, P9221R5_NUM_FOD);
+ pdata->fod_num = P9221R5_NUM_FOD;
+ }
+ ret = of_property_read_u8_array(node, "fod", pdata->fod,
+ pdata->fod_num);
+ if (ret == 0) {
+ char buf[pdata->fod_num * 3 + 1];
+
+ p9221_hex_str(pdata->fod, pdata->fod_num, buf,
+ pdata->fod_num * 3 + 1, false);
+ dev_info(dev, "dt fod: %s (%d)\n", buf, pdata->fod_num);
+ }
+ }
+
+ pdata->fod_epp_num =
+ of_property_count_elems_of_size(node, "fod_epp", sizeof(u8));
+ if (pdata->fod_epp_num <= 0) {
+ dev_err(dev, "No dt fod epp provided (%d)\n",
+ pdata->fod_epp_num);
+ pdata->fod_epp_num = 0;
+ } else {
+ if (pdata->fod_epp_num > P9221R5_NUM_FOD) {
+ dev_err(dev,
+ "Incorrect num of EPP FOD %d, using first %d\n",
+ pdata->fod_epp_num, P9221R5_NUM_FOD);
+ pdata->fod_epp_num = P9221R5_NUM_FOD;
+ }
+ ret = of_property_read_u8_array(node, "fod_epp", pdata->fod_epp,
+ pdata->fod_epp_num);
+ if (ret == 0) {
+ char buf[pdata->fod_epp_num * 3 + 1];
+
+ p9221_hex_str(pdata->fod_epp, pdata->fod_epp_num, buf,
+ pdata->fod_epp_num * 3 + 1, false);
+ dev_info(dev, "dt fod_epp: %s (%d)\n", buf,
+ pdata->fod_epp_num);
+ }
+ }
+
+ return 0;
+}
+
+static enum power_supply_property p9221_props[] = {
+ POWER_SUPPLY_PROP_PRESENT,
+ POWER_SUPPLY_PROP_ONLINE,
+ POWER_SUPPLY_PROP_CURRENT_NOW,
+ POWER_SUPPLY_PROP_CURRENT_MAX,
+ POWER_SUPPLY_PROP_VOLTAGE_NOW,
+ POWER_SUPPLY_PROP_VOLTAGE_MAX,
+ POWER_SUPPLY_PROP_TEMP,
+ POWER_SUPPLY_PROP_SERIAL_NUMBER,
+ POWER_SUPPLY_PROP_CAPACITY,
+};
+
+static const struct power_supply_desc p9221_psy_desc = {
+ .name = "wireless",
+ .type = POWER_SUPPLY_TYPE_WIRELESS,
+ .properties = p9221_props,
+ .num_properties = ARRAY_SIZE(p9221_props),
+ .get_property = p9221_get_property,
+ .set_property = p9221_set_property,
+ .property_is_writeable = p9221_prop_is_writeable,
+};
+
+static int p9221_charger_probe(struct i2c_client *client,
+ const struct i2c_device_id *id)
+{
+ struct device_node *of_node = client->dev.of_node;
+ struct p9221_charger_data *charger;
+ struct p9221_charger_platform_data *pdata = client->dev.platform_data;
+ struct power_supply_config psy_cfg = {};
+ int ret = 0;
+ u16 chip_id = 0;
+
+ ret = i2c_check_functionality(client->adapter,
+ I2C_FUNC_SMBUS_BYTE_DATA |
+ I2C_FUNC_SMBUS_WORD_DATA |
+ I2C_FUNC_SMBUS_I2C_BLOCK);
+ if (!ret) {
+ ret = i2c_get_functionality(client->adapter);
+ dev_err(&client->dev, "I2C adapter not compatible %x\n", ret);
+ return -ENOSYS;
+ }
+
+ if (of_node) {
+ pdata = devm_kzalloc(&client->dev, sizeof(*pdata), GFP_KERNEL);
+ if (!pdata) {
+ dev_err(&client->dev, "Failed to allocate pdata\n");
+ return -ENOMEM;
+ }
+ ret = p9221_parse_dt(&client->dev, pdata);
+ if (ret) {
+ dev_err(&client->dev, "Failed to parse dt\n");
+ return ret;
+ }
+ }
+
+ charger = devm_kzalloc(&client->dev, sizeof(*charger), GFP_KERNEL);
+ if (charger == NULL) {
+ dev_err(&client->dev, "Failed to allocate charger\n");
+ return -ENOMEM;
+ }
+ i2c_set_clientdata(client, charger);
+ charger->dev = &client->dev;
+ charger->client = client;
+ charger->pdata = pdata;
+ charger->resume_complete = true;
+ mutex_init(&charger->io_lock);
+ mutex_init(&charger->cmd_lock);
+ setup_timer(&charger->vrect_timer, p9221_vrect_timer_handler,
+ (unsigned long)charger);
+ INIT_DELAYED_WORK(&charger->dcin_work, p9221_dcin_work);
+ INIT_DELAYED_WORK(&charger->tx_work, p9221_tx_work);
+ INIT_DELAYED_WORK(&charger->icl_ramp_work, p9221_icl_ramp_work);
+ alarm_init(&charger->icl_ramp_alarm, ALARM_BOOTTIME,
+ p9221_icl_ramp_alarm_cb);
+
+ /* Default enable */
+ charger->enabled = true;
+ if (charger->pdata->qien_gpio >= 0)
+ gpio_direction_output(charger->pdata->qien_gpio, 0);
+
+ /* Default to R5+ */
+ charger->cust_id = 5;
+
+ psy_cfg.drv_data = charger;
+ psy_cfg.of_node = charger->dev->of_node;
+ charger->wc_psy = devm_power_supply_register(charger->dev,
+ &p9221_psy_desc,
+ &psy_cfg);
+ if (IS_ERR(charger->wc_psy)) {
+ dev_err(&client->dev, "Fail to register supply: %d\n", ret);
+ return PTR_ERR(charger->wc_psy);
+ }
+
+ /*
+ * Find the DC_ICL votable, we use this to limit the current that
+ * is taken from the wireless charger.
+ */
+ charger->dc_icl_votable = find_votable("DC_ICL");
+ if (!charger->dc_icl_votable)
+ dev_warn(&charger->client->dev, "Could not find DC_ICL votable\n");
+
+ charger->icl_ramp_ua = P9221_DC_ICL_BPP_RAMP_DEFAULT_UA;
+ charger->icl_ramp_delay_ms = P9221_DC_ICL_BPP_RAMP_DELAY_DEFAULT_MS;
+
+ /* Test to see if the charger is online */
+ ret = p9221_reg_read_16(charger, P9221_CHIP_ID_REG, &chip_id);
+ if (ret == 0 && chip_id == P9221_CHIP_ID) {
+ dev_info(&client->dev, "Charger online id:%04x\n", chip_id);
+ /* set charger->online=true, will ignore first VRECTON IRQ */
+ p9221_set_online(charger);
+ } else {
+ /* disconnected, (likely err!=0) vote for BPP */
+ p9221_vote_defaults(charger);
+ }
+
+ ret = devm_request_threaded_irq(
+ &client->dev, charger->pdata->irq_int, NULL,
+ p9221_irq_thread, IRQF_TRIGGER_LOW | IRQF_ONESHOT,
+ "p9221-irq", charger);
+ if (ret) {
+ dev_err(&client->dev, "Failed to request IRQ\n");
+ return ret;
+ }
+ device_init_wakeup(charger->dev, true);
+
+ /*
+ * We will receive a VRECTON after enabling IRQ if the device is
+ * if the device is already in-field when the driver is probed.
+ */
+ enable_irq_wake(charger->pdata->irq_int);
+
+ if (gpio_is_valid(charger->pdata->irq_det_gpio)) {
+ ret = devm_request_threaded_irq(
+ &client->dev, charger->pdata->irq_det_int, NULL,
+ p9221_irq_det_thread,
+ IRQF_TRIGGER_RISING | IRQF_ONESHOT, "p9221-irq-det",
+ charger);
+ if (ret) {
+ dev_err(&client->dev, "Failed to request IRQ_DET\n");
+ return ret;
+ }
+
+ ret = devm_gpio_request_one(&client->dev,
+ charger->pdata->irq_det_gpio,
+ GPIOF_DIR_IN, "p9221-det-gpio");
+ if (ret) {
+ dev_err(&client->dev, "Failed to request GPIO_DET\n");
+ return ret;
+ }
+ enable_irq_wake(charger->pdata->irq_det_int);
+ }
+
+ charger->last_capacity = -1;
+ charger->count = 1;
+ ret = sysfs_create_group(&charger->dev->kobj, &p9221_attr_group);
+ if (ret) {
+ dev_info(&client->dev, "sysfs_create_group failed\n");
+ }
+
+ /*
+ * Register notifier so we can detect changes on DC_IN
+ */
+ INIT_DELAYED_WORK(&charger->notifier_work, p9221_notifier_work);
+ charger->nb.notifier_call = p9221_notifier_cb;
+ ret = power_supply_reg_notifier(&charger->nb);
+ if (ret) {
+ dev_err(&client->dev, "Fail to register notifier: %d\n", ret);
+ return ret;
+ }
+
+ dev_info(&client->dev, "P9221 Charger Driver Loaded\n");
+
+ if (chip_id == P9221_CHIP_ID) {
+ charger->dc_psy = power_supply_get_by_name("dc");
+ if (charger->dc_psy)
+ power_supply_changed(charger->dc_psy);
+ }
+
+ return 0;
+}
+
+static int p9221_charger_remove(struct i2c_client *client)
+{
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ cancel_delayed_work_sync(&charger->dcin_work);
+ cancel_delayed_work_sync(&charger->tx_work);
+ cancel_delayed_work_sync(&charger->icl_ramp_work);
+ alarm_try_to_cancel(&charger->icl_ramp_alarm);
+ del_timer_sync(&charger->vrect_timer);
+ device_init_wakeup(charger->dev, false);
+ cancel_delayed_work_sync(&charger->notifier_work);
+ power_supply_unreg_notifier(&charger->nb);
+ mutex_destroy(&charger->io_lock);
+ return 0;
+}
+
+static const struct i2c_device_id p9221_charger_id_table[] = {
+ { "p9221", 0 },
+ { },
+};
+MODULE_DEVICE_TABLE(i2c, p9221_charger_id_table);
+
+#ifdef CONFIG_OF
+static struct of_device_id p9221_charger_match_table[] = {
+ { .compatible = "idt,p9221",},
+ {},
+};
+#else
+#define p9221_charger_match_table NULL
+#endif
+
+#ifdef CONFIG_PM_SLEEP
+static int p9221_pm_suspend(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ pm_runtime_get_sync(charger->dev);
+ charger->resume_complete = false;
+ pm_runtime_put_sync(charger->dev);
+
+ return 0;
+}
+
+static int p9221_pm_resume(struct device *dev)
+{
+ struct i2c_client *client = to_i2c_client(dev);
+ struct p9221_charger_data *charger = i2c_get_clientdata(client);
+
+ pm_runtime_get_sync(charger->dev);
+ charger->resume_complete = true;
+ pm_runtime_put_sync(charger->dev);
+
+ return 0;
+}
+#endif
+static const struct dev_pm_ops p9221_pm_ops = {
+ SET_LATE_SYSTEM_SLEEP_PM_OPS(p9221_pm_suspend, p9221_pm_resume)
+};
+
+static struct i2c_driver p9221_charger_driver = {
+ .driver = {
+ .name = "p9221",
+ .owner = THIS_MODULE,
+ .of_match_table = p9221_charger_match_table,
+ .pm = &p9221_pm_ops,
+ },
+ .probe = p9221_charger_probe,
+ .remove = p9221_charger_remove,
+ .id_table = p9221_charger_id_table,
+};
+
+static int __init p9221_charger_init(void)
+{
+ return i2c_add_driver(&p9221_charger_driver);
+}
+
+static void __exit p9221_charger_exit(void)
+{
+ i2c_del_driver(&p9221_charger_driver);
+}
+
+module_init(p9221_charger_init);
+module_exit(p9221_charger_exit);
+
+MODULE_DESCRIPTION("IDT P9221 Wireless Power Receiver Driver");
+MODULE_AUTHOR("Patrick Tjin <pattjin@google.com>");
+MODULE_LICENSE("GPL");
diff --git a/drivers/power/supply/qcom/p9221_charger.h b/drivers/power/supply/qcom/p9221_charger.h
new file mode 100644
index 0000000..f1a46e1
--- /dev/null
+++ b/drivers/power/supply/qcom/p9221_charger.h
@@ -0,0 +1,290 @@
+/*
+ * P9221 Wireless Charger Driver
+ *
+ * Copyright (C) 2017 Google, Inc.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+#ifndef __P9221_CHARGER_H__
+#define __P9221_CHARGER_H__
+
+#define P9221_WLC_VOTER "WLC_VOTER"
+#define P9221_USER_VOTER "WLC_USER_VOTER"
+#define P9221_OCP_VOTER "OCP_VOTER"
+#define P9221_DC_ICL_BPP_UA 700000
+#define P9221_DC_ICL_BPP_RAMP_DEFAULT_UA 900000
+#define P9221_DC_ICL_BPP_RAMP_DELAY_DEFAULT_MS (7 * 60 * 1000) /* 7 mins */
+#define P9221_DC_ICL_EPP_UA 1100000
+#define P9221_EPP_THRESHOLD_UV 7000000
+#define P9221_MAX_VOUT_SET_MV_DEFAULT 9000
+
+/*
+ * P9221 common registers
+ */
+#define P9221_CHIP_ID_REG 0x00
+#define P9221_CHIP_ID 0x9220
+#define P9221_CHIP_REVISION_REG 0x02
+#define P9221_CUSTOMER_ID_REG 0x03
+#define P9221R5_CUSTOMER_ID_VAL 0x05
+#define P9221_OTP_FW_MAJOR_REV_REG 0x04
+#define P9221_OTP_FW_MINOR_REV_REG 0x06
+#define P9221_OTP_FW_DATE_REG 0x08
+#define P9221_OTP_FW_DATE_SIZE 12
+#define P9221_OTP_FW_TIME_REG 0x14
+#define P9221_OTP_FW_TIME_SIZE 8
+#define P9221_SRAM_FW_MAJOR_REV_REG 0x1C
+#define P9221_SRAM_FW_MINOR_REV_REG 0x1E
+#define P9221_SRAM_FW_DATE_REG 0x20
+#define P9221_SRAM_FW_DATE_SIZE 12
+#define P9221_SRAM_FW_TIME_REG 0x2C
+#define P9221_SRAM_FW_TIME_SIZE 8
+#define P9221_STATUS_REG 0x34
+#define P9221_INT_REG 0x36
+#define P9221_INT_MASK 0xF7
+#define P9221_INT_ENABLE_REG 0x38
+#define P9221_COM_REG 0x4E
+
+
+/*
+ * P9221R5 unique registers
+ */
+#define P9221R5_INT_CLEAR_REG 0x3A
+#define P9221R5_VOUT_SET_REG 0x3C
+#define P9221R5_ILIM_SET_REG 0x3D
+#define P9221R5_ILIM_SET_MAX 0x0E /* 0x0E = 1.6A */
+#define P9221R5_CHARGE_STAT_REG 0x3E
+#define P9221R5_EPT_REG 0x3F
+#define P9221R5_VRECT_REG 0x40
+#define P9221R5_VOUT_REG 0x42
+#define P9221R5_IOUT_REG 0x44
+#define P9221R5_OP_FREQ_REG 0x48
+#define P9221R5_SYSTEM_MODE_REG 0x4C
+#define P9221R5_COM_CHAN_RESET_REG 0x50
+#define P9221R5_COM_CHAN_SEND_SIZE_REG 0x58
+#define P9221R5_COM_CHAN_SEND_IDX_REG 0x59
+#define P9221R5_COM_CHAN_RECV_SIZE_REG 0x5A
+#define P9221R5_COM_CHAN_RECV_IDX_REG 0x5B
+#define P9221R5_VRECT_ADC_REG 0x60
+#define P9221R5_VOUT_ADC_REG 0x62
+#define P9221R5_VOUT_ADC_MASK 0xFFF
+#define P9221R5_IOUT_ADC_REG 0x64
+#define P9221R5_IOUT_ADC_MASK 0xFFF
+#define P9221R5_DIE_TEMP_ADC_REG 0x66
+#define P9221R5_DIE_TEMP_ADC_MASK 0xFFF
+#define P9221R5_AC_PERIOD_REG 0x68
+#define P9221R5_TX_PINGFREQ_REG 0x6A
+#define P9221R5_EXT_TEMP_REG 0x6C
+#define P9221R5_EXT_TEMP_MASK 0xFFF
+#define P9221R5_FOD_REG 0x70
+#define P9221R5_NUM_FOD 16
+#define P9221R5_DEBUG_REG 0x80
+#define P9221R5_EPP_Q_FACTOR_REG 0x83
+#define P9221R5_EPP_TX_GUARANTEED_POWER_REG 0x84
+#define P9221R5_EPP_TX_POTENTIAL_POWER_REG 0x85
+#define P9221R5_EPP_TX_CAPABILITY_FLAGS_REG 0x86
+#define P9221R5_EPP_RENEGOTIATION_REG 0x87
+#define P9221R5_EPP_CUR_RPP_HEADER_REG 0x88
+#define P9221R5_EPP_CUR_NEGOTIATED_POWER_REG 0x89
+#define P9221R5_EPP_CUR_MAXIMUM_POWER_REG 0x8A
+#define P9221R5_EPP_CUR_FSK_MODULATION_REG 0x8B
+#define P9221R5_EPP_REQ_RPP_HEADER_REG 0x8C
+#define P9221R5_EPP_REQ_NEGOTIATED_POWER_REG 0x8D
+#define P9221R5_EPP_REQ_MAXIMUM_POWER_REG 0x8E
+#define P9221R5_EPP_REQ_FSK_MODULATION_REG 0x8F
+#define P9221R5_VRECT_TARGET_REG 0x90
+#define P9221R5_VRECT_KNEE_REG 0x92
+#define P9221R5_VRECT_CORRECTION_FACTOR_REG 0x93
+#define P9221R5_VRECT_MAX_CORRECTION_FACTOR_REG 0x94
+#define P9221R5_VRECT_MIN_CORRECTION_FACTOR_REG 0x96
+#define P9221R5_FOD_SECTION_REG 0x99
+#define P9221R5_VRECT_ADJ_REG 0x9E
+#define P9221R5_ALIGN_X_ADC_REG 0xA0
+#define P9221R5_ALIGN_Y_ADC_REG 0xA1
+#define P9221R5_ASK_MODULATION_DEPTH_REG 0xA2
+#define P9221R5_OVSET_REG 0xA3
+#define P9221R5_OVSET_MASK 0x7
+#define P9221R5_EPP_TX_SPEC_REV_REG 0xA9
+#define P9221R5_EPP_TX_MFG_CODE_REG 0xAA
+#define P9221R5_GP0_RESET_VOLT_REG 0xAC
+#define P9221R5_GP1_RESET_VOLT_REG 0xAE
+#define P9221R5_GP2_RESET_VOLT_REG 0xB0
+#define P9221R5_GP3_RESET_VOLT_REG 0xB2
+#define P9221R5_PROP_TX_ID_REG 0xB4
+#define P9221R5_PROP_TX_ID_SIZE 4
+#define P9221R5_DATA_SEND_BUF_START 0x100
+#define P9221R5_DATA_SEND_BUF_SIZE 0x80
+#define P9221R5_DATA_RECV_BUF_START 0x180
+#define P9221R5_DATA_RECV_BUF_SIZE 0x80
+#define P9221R5_MAX_PP_BUF_SIZE 16
+#define P9221R5_LAST_REG 0x1FF
+
+/*
+ * System Mode Mask (R5+/0x4C)
+ */
+#define P9221R5_SYSTEM_MODE_EXTENDED_MASK (1 << 3)
+
+/*
+ * Com Channel Commands
+ */
+#define P9221R5_COM_CHAN_CCRESET BIT(7)
+#define P9221_COM_CHAN_RETRIES 5
+
+/*
+ * End of Power packet types
+ */
+#define P9221_EOP_UNKNOWN 0x00
+#define P9221_EOP_EOC 0x01
+#define P9221_EOP_INTERNAL_FAULT 0x02
+#define P9221_EOP_OVER_TEMP 0x03
+#define P9221_EOP_OVER_VOLT 0x04
+#define P9221_EOP_OVER_CURRENT 0x05
+#define P9221_EOP_BATT_FAIL 0x06
+#define P9221_EOP_RECONFIG 0x07
+#define P9221_EOP_NO_RESPONSE 0x08
+#define P9221_EOP_NEGOTIATION_FAIL 0x0A
+#define P9221_EOP_RESTART_POWER 0x0B
+
+/*
+ * Command flags
+ */
+#define P9221R5_COM_RENEGOTIATE P9221_COM_RENEGOTIATE
+#define P9221R5_COM_SWITCH2RAM P9221_COM_SWITCH_TO_RAM_MASK
+#define P9221R5_COM_CLRINT P9221_COM_CLEAR_INT_MASK
+#define P9221R5_COM_SENDCSP P9221_COM_SEND_CHG_STAT_MASK
+#define P9221R5_COM_SENDEPT P9221_COM_SEND_EOP_MASK
+#define P9221R5_COM_LDOTGL P9221_COM_LDO_TOGGLE
+#define P9221R5_COM_CCACTIVATE BIT(0)
+
+#define P9221_COM_RENEGOTIATE BIT(7)
+#define P9221_COM_SWITCH_TO_RAM_MASK BIT(6)
+#define P9221_COM_CLEAR_INT_MASK BIT(5)
+#define P9221_COM_SEND_CHG_STAT_MASK BIT(4)
+#define P9221_COM_SEND_EOP_MASK BIT(3)
+#define P9221_COM_LDO_TOGGLE BIT(1)
+
+/*
+ * Interrupt/Status flags for P9221
+ */
+#define P9221_STAT_VOUT BIT(7)
+#define P9221_STAT_VRECT BIT(6)
+#define P9221_STAT_ACMISSING BIT(5)
+#define P9221_STAT_OV_TEMP BIT(2)
+#define P9221_STAT_OV_VOLT BIT(1)
+#define P9221_STAT_OV_CURRENT BIT(0)
+#define P9221_STAT_LIMIT_MASK (P9221_STAT_OV_TEMP | \
+ P9221_STAT_OV_VOLT | \
+ P9221_STAT_OV_CURRENT)
+/*
+ * Interrupt/Status flags for P9221R5
+ */
+#define P9221R5_STAT_CCRESET BIT(12)
+#define P9221R5_STAT_CCERROR BIT(11)
+#define P9221R5_STAT_PPRCVD BIT(10)
+#define P9221R5_STAT_CCDATARCVD BIT(9)
+#define P9221R5_STAT_CCSENDBUSY BIT(8)
+#define P9221R5_STAT_VOUTCHANGED BIT(7)
+#define P9221R5_STAT_VRECTON BIT(6)
+#define P9221R5_STAT_MODECHANGED BIT(5)
+#define P9221R5_STAT_UV BIT(3)
+#define P9221R5_STAT_OVT BIT(2)
+#define P9221R5_STAT_OVV BIT(1)
+#define P9221R5_STAT_OVC BIT(0)
+#define P9221R5_STAT_MASK 0x1FFF
+#define P9221R5_STAT_CC_MASK (P9221R5_STAT_CCRESET | \
+ P9221R5_STAT_PPRCVD | \
+ P9221R5_STAT_CCERROR | \
+ P9221R5_STAT_CCDATARCVD | \
+ P9221R5_STAT_CCSENDBUSY)
+#define P9221R5_STAT_LIMIT_MASK (P9221R5_STAT_UV | \
+ P9221R5_STAT_OVV | \
+ P9221R5_STAT_OVT | \
+ P9221R5_STAT_OVC)
+
+struct p9221_charger_platform_data {
+ int irq_gpio;
+ int irq_int;
+ int irq_det_gpio;
+ int irq_det_int;
+ int qien_gpio;
+ int max_vout_mv;
+ u8 fod[P9221R5_NUM_FOD];
+ u8 fod_epp[P9221R5_NUM_FOD];
+ int fod_num;
+ int fod_epp_num;
+};
+
+struct p9221_charger_data {
+ struct i2c_client *client;
+ struct p9221_charger_platform_data *pdata;
+ struct power_supply *wc_psy;
+ struct power_supply *dc_psy;
+ struct votable *dc_icl_votable;
+ struct notifier_block nb;
+ struct mutex io_lock;
+ struct mutex cmd_lock;
+ struct device *dev;
+ struct delayed_work notifier_work;
+ struct delayed_work dcin_work;
+ struct delayed_work tx_work;
+ struct delayed_work icl_ramp_work;
+ struct alarm icl_ramp_alarm;
+ struct timer_list vrect_timer;
+ struct bin_attribute bin;
+ int online;
+ bool enabled;
+ u16 addr;
+ u8 count;
+ u8 cust_id;
+ u8 pp_buf[P9221R5_MAX_PP_BUF_SIZE];
+ bool pp_buf_valid;
+ u8 rx_buf[P9221R5_DATA_RECV_BUF_SIZE];
+ u16 rx_len;
+ bool rx_done;
+ u8 tx_buf[P9221R5_DATA_SEND_BUF_SIZE];
+ u32 tx_id;
+ u8 tx_id_str[(sizeof(u32) * 2) + 1];
+ u16 tx_len;
+ bool tx_done;
+ bool tx_busy;
+ bool check_dc;
+ bool check_det;
+ int last_capacity;
+ bool resume_complete;
+ bool icl_ramp;
+ u32 icl_ramp_ua;
+ u32 icl_ramp_delay_ms;
+};
+
+struct p9221_prop_reg_map_entry {
+ enum power_supply_property prop;
+ u16 reg;
+ bool get;
+ bool set;
+};
+
+#define P9221_SHOW(name, reg, width, mask, format) \
+ static ssize_t p9221_show_##name(struct device *dev, \
+ struct device_attribute *attr, \
+ char *buf) \
+ { \
+ struct i2c_client *client = to_i2c_client(dev); \
+ struct p9221_charger_data *charger = \
+ i2c_get_clientdata(client); \
+ u##width val; \
+ int ret; \
+ \
+ ret = p9221_reg_read_##width(charger, reg, &val); \
+ if (ret) \
+ return ret; \
+ val &= mask; \
+ return snprintf(buf, PAGE_SIZE, format, val); \
+ }
+
+#endif /* __P9221_CHARGER_H__ */
diff --git a/drivers/power/supply/qcom/qpnp-fg-gen3.c b/drivers/power/supply/qcom/qpnp-fg-gen3.c
index eaf138c..8e8882e 100644
--- a/drivers/power/supply/qcom/qpnp-fg-gen3.c
+++ b/drivers/power/supply/qcom/qpnp-fg-gen3.c
@@ -657,11 +657,8 @@ static int fg_get_battery_temp(struct fg_chip *chip, int *val)
temp = ((buf[1] & BATT_TEMP_MSB_MASK) << 8) |
(buf[0] & BATT_TEMP_LSB_MASK);
- temp = DIV_ROUND_CLOSEST(temp, 4);
-
- /* Value is in Kelvin; Convert it to deciDegC */
- temp = (temp - 273) * 10;
- *val = temp;
+ /* Value is in 0.25Kelvin; Convert it to deciDegC */
+ *val = DIV_ROUND_CLOSEST((temp - 273*4) * 10, 4);
return 0;
}
@@ -822,6 +819,7 @@ static bool is_batt_empty(struct fg_chip *chip)
return ((vbatt_uv < chip->dt.cutoff_volt_mv * 1000) ? true : false);
}
+#ifdef CONFIG_QPNP_FG_GEN3_DEBUG_BATT
static int fg_get_debug_batt_id(struct fg_chip *chip, int *batt_id)
{
int rc;
@@ -878,6 +876,7 @@ static bool is_debug_batt_id(struct fg_chip *chip)
return false;
}
+#endif /* CONFIG_QPNP_FG_GEN3_DEBUG_BATT */
#define DEBUG_BATT_SOC 67
#define BATT_MISS_SOC 50
@@ -886,10 +885,12 @@ static int fg_get_prop_capacity(struct fg_chip *chip, int *val)
{
int rc, msoc;
+#ifdef CONFIG_QPNP_FG_GEN3_DEBUG_BATT
if (is_debug_batt_id(chip)) {
*val = DEBUG_BATT_SOC;
return 0;
}
+#endif
if (chip->fg_restarting) {
*val = chip->last_soc;
@@ -2588,6 +2589,86 @@ static void fg_ttf_update(struct fg_chip *chip)
schedule_delayed_work(&chip->ttf_work, msecs_to_jiffies(delay_ms));
}
+static ssize_t fg_get_cycle_counts_bins(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct fg_chip *chip = dev_get_drvdata(dev);
+ int rc = 0, i;
+ u8 data[2];
+ int length = 0;
+
+ mutex_lock(&chip->cyc_ctr.lock);
+ for (i = 0; i < BUCKET_COUNT; i++) {
+ rc = fg_sram_read(chip, CYCLE_COUNT_WORD + (i / 2),
+ CYCLE_COUNT_OFFSET + (i % 2) * 2, data, 2,
+ FG_IMA_DEFAULT);
+
+ if (rc < 0) {
+ pr_err("failed to read bucket %d rc=%d\n", i, rc);
+ chip->cyc_ctr.count[i] = 0;
+ } else
+ chip->cyc_ctr.count[i] = data[0] | data[1] << 8;
+
+ length += scnprintf(buf + length,
+ PAGE_SIZE - length, "%d",
+ chip->cyc_ctr.count[i]);
+
+ if (i == BUCKET_COUNT-1)
+ length += scnprintf(buf + length,
+ PAGE_SIZE - length, "\n");
+ else
+ length += scnprintf(buf + length,
+ PAGE_SIZE - length, " ");
+ }
+ mutex_unlock(&chip->cyc_ctr.lock);
+
+ return length;
+}
+
+static ssize_t fg_set_cycle_counts_bins(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct fg_chip *chip = dev_get_drvdata(dev);
+ int rc = 0, strval[BUCKET_COUNT], bucket;
+ u16 cyc_count;
+ u8 data[2];
+
+ if (sscanf(buf, "%d %d %d %d %d %d %d %d",
+ &strval[0], &strval[1], &strval[2], &strval[3],
+ &strval[4], &strval[5], &strval[6], &strval[7])
+ != BUCKET_COUNT)
+ return -EINVAL;
+
+ mutex_lock(&chip->cyc_ctr.lock);
+ for (bucket = 0; bucket < BUCKET_COUNT; bucket++) {
+ if (strval[bucket] > chip->cyc_ctr.count[bucket]) {
+ cyc_count = strval[bucket];
+ data[0] = cyc_count & 0xFF;
+ data[1] = cyc_count >> 8;
+
+ rc = fg_sram_write(chip,
+ CYCLE_COUNT_WORD + (bucket / 2),
+ CYCLE_COUNT_OFFSET + (bucket % 2) * 2,
+ data, 2, FG_IMA_DEFAULT);
+ if (rc < 0) {
+ pr_err("failed to write BATT_CYCLE[%d] rc=%d\n",
+ bucket, rc);
+ mutex_unlock(&chip->cyc_ctr.lock);
+ return rc;
+ }
+ chip->cyc_ctr.count[bucket] = cyc_count;
+ }
+ }
+ mutex_unlock(&chip->cyc_ctr.lock);
+
+ return count;
+}
+
+static DEVICE_ATTR(cycle_counts_bins, 0660,
+ fg_get_cycle_counts_bins, fg_set_cycle_counts_bins);
+
static void restore_cycle_counter(struct fg_chip *chip)
{
int rc = 0, i;
@@ -2714,12 +2795,25 @@ static int fg_get_cycle_count(struct fg_chip *chip)
if (!chip->cyc_ctr.en)
return 0;
+#ifdef CONFIG_QPNP_FG_GEN3_LEGACY_CYCLE_COUNT
if ((chip->cyc_ctr.id <= 0) || (chip->cyc_ctr.id > BUCKET_COUNT))
return -EINVAL;
mutex_lock(&chip->cyc_ctr.lock);
count = chip->cyc_ctr.count[chip->cyc_ctr.id - 1];
mutex_unlock(&chip->cyc_ctr.lock);
+#else
+ mutex_lock(&chip->cyc_ctr.lock);
+ {
+ int i;
+
+ count = 0;
+ for (i = 0 ; i < BUCKET_COUNT; i++)
+ count += chip->cyc_ctr.count[i];
+ count = DIV_ROUND_CLOSEST(count, 8);
+ }
+ mutex_unlock(&chip->cyc_ctr.lock);
+#endif
return count;
}
@@ -3850,9 +3944,11 @@ static int fg_psy_get_property(struct power_supply *psy,
case POWER_SUPPLY_PROP_CYCLE_COUNT:
pval->intval = fg_get_cycle_count(chip);
break;
+#ifdef CONFIG_QPNP_FG_GEN3_LEGACY_CYCLE_COUNT
case POWER_SUPPLY_PROP_CYCLE_COUNT_ID:
pval->intval = chip->cyc_ctr.id;
break;
+#endif
case POWER_SUPPLY_PROP_CHARGE_NOW_RAW:
rc = fg_get_charge_raw(chip, &pval->intval);
break;
@@ -3878,7 +3974,11 @@ static int fg_psy_get_property(struct power_supply *psy,
pval->intval = chip->soc_reporting_ready;
break;
case POWER_SUPPLY_PROP_DEBUG_BATTERY:
+#ifdef CONFIG_QPNP_FG_GEN3_DEBUG_BATT
pval->intval = is_debug_batt_id(chip);
+#else
+ pval->intval = 0;
+#endif
break;
case POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE:
rc = fg_get_sram_prop(chip, FG_SRAM_VBATT_FULL, &pval->intval);
@@ -3914,9 +4014,10 @@ static int fg_psy_set_property(struct power_supply *psy,
const union power_supply_propval *pval)
{
struct fg_chip *chip = power_supply_get_drvdata(psy);
- int rc = 0;
+ int rc = -EINVAL;
switch (psp) {
+#ifdef CONFIG_QPNP_FG_GEN3_LEGACY_CYCLE_COUNT
case POWER_SUPPLY_PROP_CYCLE_COUNT_ID:
if ((pval->intval > 0) && (pval->intval <= BUCKET_COUNT)) {
chip->cyc_ctr.id = pval->intval;
@@ -3926,6 +4027,7 @@ static int fg_psy_set_property(struct power_supply *psy,
return -EINVAL;
}
break;
+#endif
case POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE:
rc = fg_set_constant_chg_voltage(chip, pval->intval);
break;
@@ -4008,7 +4110,9 @@ static int fg_property_is_writeable(struct power_supply *psy,
enum power_supply_property psp)
{
switch (psp) {
+#ifdef CONFIG_QPNP_FG_GEN3_LEGACY_CYCLE_COUNT
case POWER_SUPPLY_PROP_CYCLE_COUNT_ID:
+#endif
case POWER_SUPPLY_PROP_CONSTANT_CHARGE_VOLTAGE:
case POWER_SUPPLY_PROP_CC_STEP:
case POWER_SUPPLY_PROP_CC_STEP_SEL:
@@ -4081,7 +4185,9 @@ static enum power_supply_property fg_psy_props[] = {
POWER_SUPPLY_PROP_CHARGE_FULL_DESIGN,
POWER_SUPPLY_PROP_VOLTAGE_MAX_DESIGN,
POWER_SUPPLY_PROP_CYCLE_COUNT,
+#ifdef CONFIG_QPNP_FG_GEN3_LEGACY_CYCLE_COUNT
POWER_SUPPLY_PROP_CYCLE_COUNT_ID,
+#endif
POWER_SUPPLY_PROP_CHARGE_NOW_RAW,
POWER_SUPPLY_PROP_CHARGE_NOW,
POWER_SUPPLY_PROP_CHARGE_FULL,
@@ -4372,6 +4478,7 @@ static int fg_hw_init(struct fg_chip *chip)
return rc;
}
+#ifdef CONFIG_QPNP_FG_GEN3_DEBUG_BATT
if (is_debug_batt_id(chip)) {
val = ESR_NO_PULL_DOWN;
rc = fg_masked_write(chip, BATT_INFO_ESR_PULL_DN_CFG(chip),
@@ -4381,6 +4488,7 @@ static int fg_hw_init(struct fg_chip *chip)
return rc;
}
}
+#endif
if (chip->dt.ki_coeff_low_dischg != -EINVAL) {
fg_encode(chip->sp, FG_SRAM_KI_COEFF_LOW_DISCHG,
@@ -5338,6 +5446,9 @@ static int fg_parse_dt(struct fg_chip *chip)
chip->dt.bmd_en_delay_ms = temp;
}
+ chip->dt.soc_irq_disable =
+ of_property_read_bool(node, "google,fg-soc-irq-disable");
+
return 0;
}
@@ -5391,6 +5502,7 @@ static struct thermal_zone_of_device_ops fg_gen3_tz_ops = {
.get_temp = fg_tz_get_temp,
};
+#define FG_DELAY_BATT_ID_MS 1000
static int fg_gen3_probe(struct platform_device *pdev)
{
struct fg_chip *chip;
@@ -5479,6 +5591,12 @@ static int fg_gen3_probe(struct platform_device *pdev)
goto exit;
}
+ rc = device_create_file(chip->dev, &dev_attr_cycle_counts_bins);
+ if (rc != 0) {
+ dev_err(chip->dev,
+ "Failed to create cycle_counts_bins files: %d\n", rc);
+ }
+
mutex_init(&chip->bus_lock);
mutex_init(&chip->sram_rw_lock);
mutex_init(&chip->cyc_ctr.lock);
@@ -5545,6 +5663,21 @@ static int fg_gen3_probe(struct platform_device *pdev)
if (fg_irqs[SOC_UPDATE_IRQ].irq)
disable_irq_nosync(fg_irqs[SOC_UPDATE_IRQ].irq);
+ if (chip->dt.soc_irq_disable) {
+ if (fg_irqs[MSOC_EMPTY_IRQ].irq)
+ disable_irq_nosync(fg_irqs[MSOC_EMPTY_IRQ].irq);
+ if (fg_irqs[MSOC_FULL_IRQ].irq)
+ disable_irq_nosync(fg_irqs[MSOC_FULL_IRQ].irq);
+ if (fg_irqs[MSOC_HIGH_IRQ].irq)
+ disable_irq_nosync(fg_irqs[MSOC_HIGH_IRQ].irq);
+ if (fg_irqs[MSOC_LOW_IRQ].irq)
+ disable_irq_nosync(fg_irqs[MSOC_LOW_IRQ].irq);
+ if (fg_irqs[MSOC_DELTA_IRQ].irq)
+ disable_irq_nosync(fg_irqs[MSOC_DELTA_IRQ].irq);
+ if (fg_irqs[BSOC_DELTA_IRQ].irq)
+ disable_irq_nosync(fg_irqs[BSOC_DELTA_IRQ].irq);
+ }
+
/* Keep BSOC_DELTA_IRQ disabled until we require it */
vote(chip->delta_bsoc_irq_en_votable, DELTA_BSOC_IRQ_VOTER, false, 0);
@@ -5583,7 +5716,8 @@ static int fg_gen3_probe(struct platform_device *pdev)
}
device_init_wakeup(chip->dev, true);
- schedule_delayed_work(&chip->profile_load_work, 0);
+ schedule_delayed_work(&chip->profile_load_work,
+ msecs_to_jiffies(FG_DELAY_BATT_ID_MS));
pr_debug("FG GEN3 driver probed successfully\n");
return 0;
diff --git a/drivers/power/supply/qcom/qpnp-smb2.c b/drivers/power/supply/qcom/qpnp-smb2.c
index 2b4fd7b..c2824ce 100644
--- a/drivers/power/supply/qcom/qpnp-smb2.c
+++ b/drivers/power/supply/qcom/qpnp-smb2.c
@@ -170,6 +170,7 @@ struct smb_dt_props {
bool hvdcp_disable;
bool auto_recharge_soc;
int wd_bark_time;
+ int batt_psy_is_bms;
};
struct smb2 {
@@ -179,7 +180,8 @@ struct smb2 {
bool bad_part;
};
-static int __debug_mask;
+static int __debug_mask =
+ PR_INTERRUPT | PR_REGISTER | PR_MISC | PR_PARALLEL | PR_OTG;
module_param_named(
debug_mask, __debug_mask, int, 0600
);
@@ -293,7 +295,33 @@ static int smb2_parse_dt(struct smb2 *chip)
chg->thermal_levels);
if (rc < 0) {
dev_err(chg->dev,
- "Couldn't read threm limits rc = %d\n", rc);
+ "Couldn't read therm limits rc = %d\n", rc);
+ return rc;
+ }
+ }
+
+ if (of_find_property(node, "google,wlc-thermal-mitigation",
+ &byte_len)) {
+ if (chg->thermal_levels &&
+ chg->thermal_levels != byte_len / sizeof(u32)) {
+ dev_err(chg->dev,
+ "google,wlc-thermal-mitigation size != qcom,thermal-mitigation\n");
+ return -EINVAL;
+ }
+
+ chg->wlc_thermal_mitigation = devm_kzalloc(chg->dev, byte_len,
+ GFP_KERNEL);
+ if (chg->wlc_thermal_mitigation == NULL)
+ return -ENOMEM;
+
+ chg->thermal_levels = byte_len / sizeof(u32);
+ rc = of_property_read_u32_array(node,
+ "google,wlc-thermal-mitigation",
+ chg->wlc_thermal_mitigation,
+ chg->thermal_levels);
+ if (rc < 0) {
+ dev_err(chg->dev,
+ "Couldn't read wlc therm limits rc = %d\n", rc);
return rc;
}
}
@@ -306,6 +334,7 @@ static int smb2_parse_dt(struct smb2 *chip)
chip->dt.hvdcp_disable = of_property_read_bool(node,
"qcom,hvdcp-disable");
+ chg->hvdcp_disable = chip->dt.hvdcp_disable;
of_property_read_u32(node, "qcom,chg-inhibit-threshold-mv",
&chip->dt.chg_inhibit_thr_mv);
@@ -331,6 +360,14 @@ static int smb2_parse_dt(struct smb2 *chip)
if (rc < 0)
chg->otg_delay_ms = OTG_DEFAULT_DEGLITCH_TIME_MS;
+ chip->dt.batt_psy_is_bms =
+ of_property_read_bool(node, "google,batt_psy_is_bms");
+
+ rc = of_property_read_string(node, "google,usb-port-tz-name",
+ &chg->usb_port_tz_name);
+ if (rc < 0)
+ pr_err("cannot read usb-port-tz-name, rc=%d\n", rc);
+
return 0;
}
@@ -346,6 +383,7 @@ static enum power_supply_property smb2_usb_props[] = {
POWER_SUPPLY_PROP_VOLTAGE_NOW,
POWER_SUPPLY_PROP_PD_CURRENT_MAX,
POWER_SUPPLY_PROP_CURRENT_MAX,
+ POWER_SUPPLY_PROP_TEMP,
POWER_SUPPLY_PROP_TYPE,
POWER_SUPPLY_PROP_TYPEC_MODE,
POWER_SUPPLY_PROP_TYPEC_POWER_ROLE,
@@ -364,7 +402,9 @@ static enum power_supply_property smb2_usb_props[] = {
POWER_SUPPLY_PROP_PD_VOLTAGE_MIN,
POWER_SUPPLY_PROP_SDP_CURRENT_MAX,
POWER_SUPPLY_PROP_CONNECTOR_TYPE,
+ POWER_SUPPLY_PROP_OTG_FASTROLESWAP,
POWER_SUPPLY_PROP_MOISTURE_DETECTED,
+ POWER_SUPPLY_PROP_PD_IN_EXPLICIT_CONTRACT,
};
static int smb2_usb_get_prop(struct power_supply *psy,
@@ -376,6 +416,9 @@ static int smb2_usb_get_prop(struct power_supply *psy,
int rc = 0;
switch (psp) {
+ case POWER_SUPPLY_PROP_PD_IN_EXPLICIT_CONTRACT:
+ val->intval = chg->in_explicit_contract;
+ break;
case POWER_SUPPLY_PROP_PRESENT:
if (chip->bad_part)
val->intval = 1;
@@ -409,7 +452,10 @@ static int smb2_usb_get_prop(struct power_supply *psy,
val->intval = get_client_vote(chg->usb_icl_votable, PD_VOTER);
break;
case POWER_SUPPLY_PROP_CURRENT_MAX:
- rc = smblib_get_prop_input_current_settled(chg, val);
+ rc = smblib_get_charge_current(chg, &val->intval);
+ break;
+ case POWER_SUPPLY_PROP_TEMP:
+ rc = smblib_get_prop_usb_port_temp(chg, val);
break;
case POWER_SUPPLY_PROP_TYPE:
val->intval = POWER_SUPPLY_TYPE_USB_PD;
@@ -486,6 +532,8 @@ static int smb2_usb_get_prop(struct power_supply *psy,
case POWER_SUPPLY_PROP_CONNECTOR_TYPE:
val->intval = chg->connector_type;
break;
+ case POWER_SUPPLY_PROP_OTG_FASTROLESWAP:
+ rc = smblib_get_prop_otg_fastroleswap(chg, val);
case POWER_SUPPLY_PROP_MOISTURE_DETECTED:
val->intval = get_client_vote(chg->disable_power_role_switch,
MOISTURE_VOTER);
@@ -517,17 +565,28 @@ static int smb2_usb_set_prop(struct power_supply *psy,
vote(chg->disable_power_role_switch, MOISTURE_VOTER,
val->intval > 0, 0);
break;
+ case POWER_SUPPLY_PROP_TYPEC_POWER_ROLE:
+ rc = smblib_set_prop_typec_power_role(chg, val);
+ break;
+ case POWER_SUPPLY_PROP_PD_IN_EXPLICIT_CONTRACT:
+ chg->in_explicit_contract = val->intval;
+ power_supply_changed(chg->usb_psy);
+ break;
default:
rc = -EINVAL;
break;
}
-
goto unlock;
}
switch (psp) {
+ case POWER_SUPPLY_PROP_PD_IN_EXPLICIT_CONTRACT:
+ chg->in_explicit_contract = val->intval;
+ power_supply_changed(chg->usb_psy);
+ break;
case POWER_SUPPLY_PROP_PD_CURRENT_MAX:
rc = smblib_set_prop_pd_current_max(chg, val);
+ power_supply_changed(chg->usb_psy);
break;
case POWER_SUPPLY_PROP_TYPEC_POWER_ROLE:
rc = smblib_set_prop_typec_power_role(chg, val);
@@ -560,6 +619,9 @@ static int smb2_usb_set_prop(struct power_supply *psy,
case POWER_SUPPLY_PROP_SDP_CURRENT_MAX:
rc = smblib_set_prop_sdp_current_max(chg, val);
break;
+ case POWER_SUPPLY_PROP_OTG_FASTROLESWAP:
+ rc = smblib_set_prop_otg_fastroleswap(chg, val);
+ break;
default:
pr_err("set prop %d is not supported\n", psp);
rc = -EINVAL;
@@ -576,6 +638,7 @@ static int smb2_usb_prop_is_writeable(struct power_supply *psy,
{
switch (psp) {
case POWER_SUPPLY_PROP_CTM_CURRENT_MAX:
+ case POWER_SUPPLY_PROP_SDP_CURRENT_MAX:
return 1;
default:
break;
@@ -882,7 +945,11 @@ static int smb2_dc_get_prop(struct power_supply *psy,
rc = smblib_get_prop_dc_current_max(chg, val);
break;
case POWER_SUPPLY_PROP_REAL_TYPE:
+#ifdef CONFIG_QPNP_SMB2_WIPOWER
val->intval = POWER_SUPPLY_TYPE_WIPOWER;
+#else
+ val->intval = POWER_SUPPLY_TYPE_UNKNOWN;
+#endif
break;
default:
return -EINVAL;
@@ -936,7 +1003,7 @@ static int smb2_dc_prop_is_writeable(struct power_supply *psy,
static const struct power_supply_desc dc_psy_desc = {
.name = "dc",
- .type = POWER_SUPPLY_TYPE_WIRELESS,
+ .type = POWER_SUPPLY_TYPE_UNKNOWN,
.properties = smb2_dc_props,
.num_properties = ARRAY_SIZE(smb2_dc_props),
.get_property = smb2_dc_get_prop,
@@ -948,6 +1015,9 @@ static int smb2_init_dc_psy(struct smb2 *chip)
{
struct power_supply_config dc_cfg = {};
struct smb_charger *chg = &chip->chg;
+#ifndef CONFIG_QPNP_SMB2_WIPOWER
+ int rc;
+#endif
dc_cfg.drv_data = chip;
dc_cfg.of_node = chg->dev->of_node;
@@ -955,10 +1025,29 @@ static int smb2_init_dc_psy(struct smb2 *chip)
&dc_psy_desc,
&dc_cfg);
if (IS_ERR(chg->dc_psy)) {
- pr_err("Couldn't register USB power supply\n");
+ pr_err("Couldn't register DC power supply\n");
return PTR_ERR(chg->dc_psy);
}
+#ifndef CONFIG_QPNP_SMB2_WIPOWER
+ rc = smblib_write(chg, WI_PWR_OPTIONS_REG, 0x00);
+ if (rc < 0) {
+ pr_err("Failed to set WI_PWR_OPTIONS_REG (%d)\n", rc);
+ return rc;
+ }
+ rc = smblib_write(chg, DCIN_AICL_REF_SEL_CFG_REG, 0x02);
+ if (rc < 0) {
+ pr_err("Failed to set DCIN_AICL_REF_SEL_CFG_REG (%d)\n", rc);
+ return rc;
+ }
+ rc = smblib_masked_write(chg, DCIN_AICL_OPTIONS_CFG_REG,
+ DCIN_AICL_ADC_EN_BIT, 0);
+ if (rc < 0) {
+ pr_err("Failed to set DCIN_AICL_OPTIONS_CFG_REG (%d)\n", rc);
+ return rc;
+ }
+#endif
+
return 0;
}
@@ -986,6 +1075,8 @@ static enum power_supply_property smb2_batt_props[] = {
POWER_SUPPLY_PROP_TECHNOLOGY,
POWER_SUPPLY_PROP_STEP_CHARGING_ENABLED,
POWER_SUPPLY_PROP_SW_JEITA_ENABLED,
+ POWER_SUPPLY_PROP_TAPER_CONTROL,
+ POWER_SUPPLY_PROP_CHARGE_DISABLE,
POWER_SUPPLY_PROP_CHARGE_DONE,
POWER_SUPPLY_PROP_PARALLEL_DISABLE,
POWER_SUPPLY_PROP_SET_SHIP_MODE,
@@ -995,6 +1086,10 @@ static enum power_supply_property smb2_batt_props[] = {
POWER_SUPPLY_PROP_CHARGE_CONTROL_LIMIT_MAX,
POWER_SUPPLY_PROP_CHARGE_CONTROL_LIMIT,
POWER_SUPPLY_PROP_CHARGE_COUNTER,
+ POWER_SUPPLY_PROP_INPUT_CURRENT_MAX,
+#ifndef CONFIG_QPNP_FG_GEN3_LEGACY_CYCLE_COUNT
+ POWER_SUPPLY_PROP_CYCLE_COUNT,
+#endif
};
static int smb2_batt_get_prop(struct power_supply *psy,
@@ -1006,6 +1101,12 @@ static int smb2_batt_get_prop(struct power_supply *psy,
union power_supply_propval pval = {0, };
switch (psp) {
+ case POWER_SUPPLY_PROP_ONLINE:
+ /** when type != POWER_SUPPLY_TYPE_BATTERY
+ * power_supply_core check this property
+ */
+ val->intval = 0;
+ break;
case POWER_SUPPLY_PROP_STATUS:
rc = smblib_get_prop_batt_status(chg, val);
break;
@@ -1052,6 +1153,9 @@ static int smb2_batt_get_prop(struct power_supply *psy,
case POWER_SUPPLY_PROP_SW_JEITA_ENABLED:
val->intval = chg->sw_jeita_enabled;
break;
+ case POWER_SUPPLY_PROP_TAPER_CONTROL:
+ val->intval = chg->taper_control_enabled;
+ break;
case POWER_SUPPLY_PROP_VOLTAGE_NOW:
rc = smblib_get_prop_batt_voltage_now(chg, val);
break;
@@ -1083,6 +1187,9 @@ static int smb2_batt_get_prop(struct power_supply *psy,
case POWER_SUPPLY_PROP_TECHNOLOGY:
val->intval = POWER_SUPPLY_TECHNOLOGY_LION;
break;
+ case POWER_SUPPLY_PROP_CHARGE_DISABLE:
+ val->intval = get_effective_result(chg->chg_disable_votable);
+ break;
case POWER_SUPPLY_PROP_CHARGE_DONE:
rc = smblib_get_prop_batt_charge_done(chg, val);
break;
@@ -1109,6 +1216,14 @@ static int smb2_batt_get_prop(struct power_supply *psy,
case POWER_SUPPLY_PROP_CHARGE_COUNTER:
rc = smblib_get_prop_batt_charge_counter(chg, val);
break;
+ case POWER_SUPPLY_PROP_INPUT_CURRENT_MAX:
+ rc = smblib_get_prop_input_current_max(chg, val);
+ break;
+#ifndef CONFIG_QPNP_FG_GEN3_LEGACY_CYCLE_COUNT
+ case POWER_SUPPLY_PROP_CYCLE_COUNT:
+ rc = smblib_get_cycle_count(chg, val);
+ break;
+#endif
default:
pr_err("batt power supply prop %d not supported\n", psp);
return -EINVAL;
@@ -1178,10 +1293,17 @@ static int smb2_batt_set_prop(struct power_supply *psy,
chg->sw_jeita_enabled = !!val->intval;
}
break;
+ case POWER_SUPPLY_PROP_TAPER_CONTROL:
+ chg->taper_control_enabled = val->intval;
+ break;
case POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT_MAX:
chg->batt_profile_fcc_ua = val->intval;
vote(chg->fcc_votable, BATT_PROFILE_VOTER, true, val->intval);
break;
+ case POWER_SUPPLY_PROP_CHARGE_DISABLE:
+ vote(chg->chg_disable_votable, USER_VOTER,
+ (bool)val->intval, 0);
+ break;
case POWER_SUPPLY_PROP_SET_SHIP_MODE:
/* Not in ship mode as long as the device is active */
if (!val->intval)
@@ -1204,6 +1326,9 @@ static int smb2_batt_set_prop(struct power_supply *psy,
chg->die_health = val->intval;
power_supply_changed(chg->batt_psy);
break;
+ case POWER_SUPPLY_PROP_INPUT_CURRENT_MAX:
+ rc = smblib_set_prop_input_current_max(chg, val);
+ break;
default:
rc = -EINVAL;
}
@@ -1225,7 +1350,12 @@ static int smb2_batt_prop_is_writeable(struct power_supply *psy,
case POWER_SUPPLY_PROP_INPUT_CURRENT_LIMITED:
case POWER_SUPPLY_PROP_STEP_CHARGING_ENABLED:
case POWER_SUPPLY_PROP_SW_JEITA_ENABLED:
+ case POWER_SUPPLY_PROP_TAPER_CONTROL:
case POWER_SUPPLY_PROP_DIE_HEALTH:
+ case POWER_SUPPLY_PROP_INPUT_CURRENT_MAX:
+ case POWER_SUPPLY_PROP_VOLTAGE_MAX:
+ case POWER_SUPPLY_PROP_CONSTANT_CHARGE_CURRENT_MAX:
+ case POWER_SUPPLY_PROP_CHARGE_DISABLE:
return 1;
default:
break;
@@ -1234,7 +1364,7 @@ static int smb2_batt_prop_is_writeable(struct power_supply *psy,
return 0;
}
-static const struct power_supply_desc batt_psy_desc = {
+static struct power_supply_desc batt_psy_desc = {
.name = "battery",
.type = POWER_SUPPLY_TYPE_BATTERY,
.properties = smb2_batt_props,
@@ -1252,6 +1382,8 @@ static int smb2_init_batt_psy(struct smb2 *chip)
batt_cfg.drv_data = chg;
batt_cfg.of_node = chg->dev->of_node;
+ if (chip->dt.batt_psy_is_bms)
+ batt_psy_desc.type = POWER_SUPPLY_TYPE_BMS;
chg->batt_psy = power_supply_register(chg->dev,
&batt_psy_desc,
&batt_cfg);
@@ -1284,6 +1416,19 @@ static int smb2_init_vbus_regulator(struct smb2 *chip)
if (!chg->vbus_vreg)
return -ENOMEM;
+ if (of_property_read_bool(chg->dev->of_node,
+ "google,boost_to_5_1V")) {
+ rc = smblib_masked_write(chg, CFG_OTG_OUTPUT_REG,
+ OTG_OUT_5_1_V,
+ OTG_OUT_5_1_V);
+ if (rc < 0) {
+ dev_err(chg->dev,
+ "Unable to boost smb2-vbus to 5.1V rc=%d\n",
+ rc);
+ return rc;
+ }
+ }
+
cfg.dev = chg->dev;
cfg.driver_data = chip;
@@ -1611,6 +1756,9 @@ static int smb2_init_hw(struct smb2 *chip)
true, 0);
vote(chg->pd_disallowed_votable_indirect, HVDCP_TIMEOUT_VOTER,
true, 0);
+ if (chg->hvdcp_disable)
+ vote(chg->pd_disallowed_votable_indirect, HVDCP_TIMEOUT_VOTER,
+ false, 0);
/*
* AICL configuration:
@@ -2327,6 +2475,7 @@ static int smb2_probe(struct platform_device *pdev)
chg->mode = PARALLEL_MASTER;
chg->irq_info = smb2_irqs;
chg->die_health = -EINVAL;
+ chg->taper_control_enabled = POWER_SUPPLY_TAPER_CONTROL_MODE_STEPPER;
chg->name = "PMI";
chg->audio_headset_drp_wait_ms = &__audio_headset_drp_wait_ms;
diff --git a/drivers/power/supply/qcom/smb-lib.c b/drivers/power/supply/qcom/smb-lib.c
index addd55e..59c5f5c 100644
--- a/drivers/power/supply/qcom/smb-lib.c
+++ b/drivers/power/supply/qcom/smb-lib.c
@@ -301,6 +301,24 @@ static const struct apsd_result *smblib_get_apsd_result(struct smb_charger *chg)
return result;
}
+int smblib_get_prop_otg_fastroleswap(struct smb_charger *chg,
+ union power_supply_propval *val)
+{
+ int rc = 0;
+ u8 stat = 0;
+
+ rc = smblib_read(chg, OTG_CFG_REG, &stat);
+ if (rc < 0) {
+ smblib_err(chg, "Couldn't read OTG quickstart rc=%d\n",
+ rc);
+ }
+ if (stat & QUICKSTART_OTG_FASTROLESWAP_BIT)
+ val->intval = 1;
+ else
+ val->intval = 0;
+ return rc;
+}
+
/********************
* REGISTER SETTERS *
********************/
@@ -503,6 +521,21 @@ static int smblib_set_usb_pd_allowed_voltage(struct smb_charger *chg,
return rc;
}
+int smblib_set_prop_otg_fastroleswap(struct smb_charger *chg,
+ const union power_supply_propval *val)
+{
+ int rc;
+
+ rc = smblib_masked_write(chg, OTG_CFG_REG,
+ QUICKSTART_OTG_FASTROLESWAP_BIT,
+ val->intval ? QUICKSTART_OTG_FASTROLESWAP_BIT : 0);
+ if (rc < 0) {
+ smblib_err(chg, "Couldn't set OTG quickstart rc=%d\n",
+ rc);
+ }
+ return rc;
+}
+
/********************
* HELPER FUNCTIONS *
********************/
@@ -1117,6 +1150,7 @@ static int __smblib_set_prop_typec_power_role(struct smb_charger *chg,
power_role = UFP_EN_CMD_BIT;
break;
case POWER_SUPPLY_TYPEC_PR_SOURCE:
+ case POWER_SUPPLY_TYPEC_PR_SOURCE_1_5:
power_role = DFP_EN_CMD_BIT;
break;
default:
@@ -1142,6 +1176,13 @@ static int __smblib_set_prop_typec_power_role(struct smb_charger *chg,
}
}
+ rc = smblib_masked_write(chg, TYPE_C_CFG_2_REG,
+ EN_80UA_180UA_CUR_SOURCE_BIT,
+ val->intval == POWER_SUPPLY_TYPEC_PR_SOURCE
+ ? 0 : EN_80UA_180UA_CUR_SOURCE_BIT);
+ if (rc < 0)
+ smblib_err(chg, "Couldnt update EN_ISRC_180UA_BIT rc=%d\n", rc);
+
rc = smblib_masked_write(chg, TYPE_C_INTRPT_ENB_SOFTWARE_CTRL_REG,
TYPEC_POWER_ROLE_CMD_MASK, power_role);
if (rc < 0) {
@@ -1403,7 +1444,7 @@ static int smblib_disable_power_role_switch_callback(struct votable *votable,
if (chg->connector_type == POWER_SUPPLY_CONNECTOR_MICRO_USB) {
rc = smblib_micro_usb_disable_power_role_switch(chg, disable);
} else {
- pval.intval = disable ? POWER_SUPPLY_TYPEC_PR_SINK
+ pval.intval = disable ? POWER_SUPPLY_TYPEC_PR_NONE
: POWER_SUPPLY_TYPEC_PR_DUAL;
rc = __smblib_set_prop_typec_power_role(chg, &pval);
}
@@ -1418,6 +1459,23 @@ static int smblib_disable_power_role_switch_callback(struct votable *votable,
return rc;
}
+static int smblib_disable_prebias_resistor_callback(struct votable *votable,
+ void *data, int disable, const char *client)
+{
+ struct smb_charger *chg = data;
+ int rc = 0;
+
+ rc = smblib_masked_write(chg, DC_ENG_SSUPPLY_CFG2_REG,
+ DISABLE_PREBIAS_RESISTOR,
+ disable ? 0 : DISABLE_PREBIAS_RESISTOR);
+ if (rc < 0) {
+ smblib_err(chg, "Couldn't write prebias resister rc=%d\n", rc);
+ return rc;
+ }
+
+ return rc;
+}
+
/*******************
* VCONN REGULATOR *
* *****************/
@@ -1714,6 +1772,50 @@ int smblib_vbus_regulator_is_enabled(struct regulator_dev *rdev)
return ret;
}
+int smblib_get_prop_input_current_max(struct smb_charger *chg,
+ union power_supply_propval *val)
+{
+ int rc;
+
+ rc = smblib_get_charge_param(chg, &chg->param.usb_icl, &val->intval);
+
+ return rc;
+}
+
+int smblib_set_prop_input_current_max(struct smb_charger *chg,
+ const union power_supply_propval *val)
+{
+ int rc;
+ bool override = true;
+
+ set_sdp_current(chg, 100000);
+ rc = smblib_set_charge_param(chg, &chg->param.usb_icl, val->intval);
+ if (rc < 0) {
+ smblib_err(chg, "Couldn't set HC ICL rc=%d\n", rc);
+ goto err_rc;
+ }
+
+ /* enforce override */
+ rc = smblib_masked_write(chg, USBIN_ICL_OPTIONS_REG,
+ USBIN_MODE_CHG_BIT, override ? USBIN_MODE_CHG_BIT : 0);
+
+ rc = smblib_icl_override(chg, override);
+ if (rc < 0) {
+ smblib_err(chg, "Couldn't set ICL override rc=%d\n", rc);
+ goto err_rc;
+ }
+
+ /* unsuspend after configuring current and override */
+ rc = smblib_set_usb_suspend(chg, false);
+ if (rc < 0) {
+ smblib_err(chg, "Couldn't resume input rc=%d\n", rc);
+ goto err_rc;
+ }
+
+err_rc:
+ return rc;
+}
+
/********************
* BATT PSY GETTERS *
********************/
@@ -2063,6 +2165,21 @@ int smblib_get_prop_batt_charge_counter(struct smb_charger *chg,
return rc;
}
+#ifndef CONFIG_QPNP_FG_GEN3_LEGACY_CYCLE_COUNT
+int smblib_get_cycle_count(struct smb_charger *chg,
+ union power_supply_propval *val)
+{
+ int rc;
+
+ if (!chg->bms_psy)
+ return -EINVAL;
+
+ rc = power_supply_get_property(chg->bms_psy,
+ POWER_SUPPLY_PROP_CYCLE_COUNT, val);
+ return rc;
+}
+#endif
+
/***********************
* BATTERY PSY SETTERS *
***********************/
@@ -2118,6 +2235,8 @@ int smblib_set_prop_batt_status(struct smb_charger *chg,
int smblib_set_prop_system_temp_level(struct smb_charger *chg,
const union power_supply_propval *val)
{
+ union power_supply_propval pval;
+
if (val->intval < 0)
return -EINVAL;
@@ -2129,16 +2248,55 @@ int smblib_set_prop_system_temp_level(struct smb_charger *chg,
chg->system_temp_level = val->intval;
- if (chg->system_temp_level == chg->thermal_levels)
- return vote(chg->chg_disable_votable,
- THERMAL_DAEMON_VOTER, true, 0);
+ if (chg->wlc_thermal_mitigation && !chg->wlc_psy)
+ chg->wlc_psy = power_supply_get_by_name("wireless");
+
+ if (chg->system_temp_level == chg->thermal_levels) {
+ if (chg->thermal_mitigation)
+ vote(chg->chg_disable_votable,
+ THERMAL_DAEMON_VOTER, true, 0);
+ if (chg->wlc_thermal_mitigation) {
+ vote(chg->dc_icl_votable,
+ THERMAL_DAEMON_VOTER, true, 0);
+ if (chg->wlc_psy) {
+ pval.intval = 0;
+ power_supply_set_property(chg->wlc_psy,
+ POWER_SUPPLY_PROP_ONLINE, &pval);
+ }
+ }
+ return 0;
+ }
vote(chg->chg_disable_votable, THERMAL_DAEMON_VOTER, false, 0);
- if (chg->system_temp_level == 0)
- return vote(chg->fcc_votable, THERMAL_DAEMON_VOTER, false, 0);
+ if (chg->system_temp_level == 0) {
+ if (chg->thermal_mitigation)
+ vote(chg->fcc_votable,
+ THERMAL_DAEMON_VOTER, false, 0);
+ if (chg->wlc_thermal_mitigation) {
+ if (chg->wlc_psy) {
+ pval.intval = 1;
+ power_supply_set_property(chg->wlc_psy,
+ POWER_SUPPLY_PROP_ONLINE, &pval);
+ }
+ vote(chg->dc_icl_votable,
+ THERMAL_DAEMON_VOTER, false, 0);
+ }
+ return 0;
+ }
- vote(chg->fcc_votable, THERMAL_DAEMON_VOTER, true,
- chg->thermal_mitigation[chg->system_temp_level]);
+ if (chg->thermal_mitigation)
+ vote(chg->fcc_votable, THERMAL_DAEMON_VOTER, true,
+ chg->thermal_mitigation[chg->system_temp_level]);
+
+ if (chg->wlc_thermal_mitigation) {
+ if (chg->wlc_psy) {
+ pval.intval = 1;
+ power_supply_set_property(chg->wlc_psy,
+ POWER_SUPPLY_PROP_ONLINE, &pval);
+ }
+ vote(chg->dc_icl_votable, THERMAL_DAEMON_VOTER, true,
+ chg->wlc_thermal_mitigation[chg->system_temp_level]);
+ }
return 0;
}
@@ -2523,6 +2681,38 @@ int smblib_get_prop_usb_current_now(struct smb_charger *chg,
return iio_read_channel_processed(chg->iio.usbin_i_chan, &val->intval);
}
+int smblib_get_prop_usb_port_temp(struct smb_charger *chg,
+ union power_supply_propval *val)
+{
+ int rc = 0;
+ int temp = INT_MIN;
+
+ if (!chg->usb_port_tz_name)
+ return -ENODEV;
+
+ /* lazily get the usb port thermal zone */
+ if (!chg->usb_port_tz) {
+ chg->usb_port_tz =
+ thermal_zone_get_zone_by_name(chg->usb_port_tz_name);
+ if (IS_ERR(chg->usb_port_tz)) {
+ rc = PTR_ERR(chg->usb_port_tz);
+ pr_err("Couldn't get USB thermal zone rc=%d\n", rc);
+ return rc;
+ }
+ } else if (IS_ERR(chg->usb_port_tz))
+ return PTR_ERR(chg->usb_port_tz);
+
+ rc = thermal_zone_get_temp(chg->usb_port_tz, &temp);
+ if (rc < 0) {
+ pr_err("Couldn't get temp USB port thermal zone rc=%d\n", rc);
+ return rc;
+ }
+
+ /* power property value needs to be in deciDeg */
+ val->intval = temp / 100;
+ return 0;
+}
+
int smblib_get_prop_charger_temp(struct smb_charger *chg,
union power_supply_propval *val)
{
@@ -2628,7 +2818,7 @@ int smblib_get_prop_typec_power_role(struct smb_charger *chg,
union power_supply_propval *val)
{
int rc = 0;
- u8 ctrl;
+ u8 ctrl, isrc;
rc = smblib_read(chg, TYPE_C_INTRPT_ENB_SOFTWARE_CTRL_REG, &ctrl);
if (rc < 0) {
@@ -2639,6 +2829,12 @@ int smblib_get_prop_typec_power_role(struct smb_charger *chg,
smblib_dbg(chg, PR_REGISTER, "TYPE_C_INTRPT_ENB_SOFTWARE_CTRL = 0x%02x\n",
ctrl);
+ rc = smblib_read(chg, TYPE_C_CFG_2_REG, &isrc);
+ if (rc < 0) {
+ smblib_err(chg, "Couldn't read TYPE_C_CFG_2_REG rc=%d\n", rc);
+ return rc;
+ }
+
if (ctrl & TYPEC_DISABLE_CMD_BIT) {
val->intval = POWER_SUPPLY_TYPEC_PR_NONE;
return rc;
@@ -2649,7 +2845,9 @@ int smblib_get_prop_typec_power_role(struct smb_charger *chg,
val->intval = POWER_SUPPLY_TYPEC_PR_DUAL;
break;
case DFP_EN_CMD_BIT:
- val->intval = POWER_SUPPLY_TYPEC_PR_SOURCE;
+ val->intval = isrc & EN_80UA_180UA_CUR_SOURCE_BIT ?
+ POWER_SUPPLY_TYPEC_PR_SOURCE_1_5 :
+ POWER_SUPPLY_TYPEC_PR_SOURCE;
break;
case UFP_EN_CMD_BIT:
val->intval = POWER_SUPPLY_TYPEC_PR_SINK;
@@ -2721,6 +2919,7 @@ int smblib_get_pe_start(struct smb_charger *chg,
val->intval
= !get_client_vote_locked(chg->pd_disallowed_votable_indirect,
HVDCP_TIMEOUT_VOTER);
+
return 0;
}
@@ -3270,7 +3469,7 @@ int smblib_get_charge_current(struct smb_charger *chg,
bool non_compliant;
u8 stat5;
- if (chg->pd_active) {
+ if (chg->in_explicit_contract) {
*total_current_ua =
get_client_vote_locked(chg->usb_icl_votable, PD_VOTER);
return rc;
@@ -3596,6 +3795,13 @@ void smblib_usb_plugin_locked(struct smb_charger *chg)
if (chg->connector_type == POWER_SUPPLY_CONNECTOR_MICRO_USB)
smblib_micro_usb_plugin(chg, vbus_rising);
+ if (vbus_rising)
+ vote(chg->disable_prebias_resistor, USBIN_PLUGIN_VOTER,
+ true, 0);
+ else
+ vote(chg->disable_prebias_resistor, USBIN_PLUGIN_VOTER,
+ false, 0);
+
power_supply_changed(chg->usb_psy);
smblib_dbg(chg, PR_INTERRUPT, "IRQ: usbin-plugin %s\n",
vbus_rising ? "attached" : "detached");
@@ -3869,18 +4075,27 @@ static void smblib_force_legacy_icl(struct smb_charger *chg, int pst)
break;
case POWER_SUPPLY_TYPE_USB_FLOAT:
/*
- * limit ICL to 100mA, the USB driver will enumerate to check
+ * limit ICL to 500mA, the USB driver will enumerate to check
* if this is a SDP and appropriately set the current
+ * and set rp current for rp non-default case
*/
- vote(chg->usb_icl_votable, LEGACY_UNKNOWN_VOTER, true, 100000);
+ if (chg->typec_mode == POWER_SUPPLY_TYPEC_SOURCE_DEFAULT)
+ vote(chg->usb_icl_votable, LEGACY_UNKNOWN_VOTER,
+ true, 500000);
+ else {
+ typec_mode = smblib_get_prop_typec_mode(chg);
+ rp_ua = get_rp_based_dcp_current(chg, typec_mode);
+ vote(chg->usb_icl_votable, LEGACY_UNKNOWN_VOTER,
+ true, rp_ua);
+ }
break;
case POWER_SUPPLY_TYPE_USB_HVDCP:
case POWER_SUPPLY_TYPE_USB_HVDCP_3:
vote(chg->usb_icl_votable, LEGACY_UNKNOWN_VOTER, true, 3000000);
break;
default:
- smblib_err(chg, "Unknown APSD %d; forcing 500mA\n", pst);
- vote(chg->usb_icl_votable, LEGACY_UNKNOWN_VOTER, true, 500000);
+ smblib_err(chg, "Unknown APSD %d; forcing suspend\n", pst);
+ vote(chg->usb_icl_votable, LEGACY_UNKNOWN_VOTER, true, 0);
break;
}
}
@@ -3946,9 +4161,15 @@ static void smblib_handle_apsd_done(struct smb_charger *chg, bool rising)
false, 0);
break;
case DCP_CHARGER_BIT:
- if (chg->wa_flags & QC_CHARGER_DETECTION_WA_BIT)
+ if ((chg->wa_flags & QC_CHARGER_DETECTION_WA_BIT) &&
+ !chg->hvdcp_disable)
schedule_delayed_work(&chg->hvdcp_detect_work,
msecs_to_jiffies(HVDCP_DET_MS));
+ /* If HVDCP is disabled, enable PD. */
+ if (chg->hvdcp_disable) {
+ vote(chg->pd_disallowed_votable_indirect,
+ HVDCP_TIMEOUT_VOTER, false, 0);
+ }
break;
default:
break;
@@ -4241,7 +4462,7 @@ static void smblib_handle_typec_removal(struct smb_charger *chg)
cancel_delayed_work_sync(&chg->hvdcp_detect_work);
/* reset input current limit voters */
- vote(chg->usb_icl_votable, LEGACY_UNKNOWN_VOTER, true, 100000);
+ vote(chg->usb_icl_votable, LEGACY_UNKNOWN_VOTER, true, 0);
vote(chg->usb_icl_votable, PD_VOTER, false, 0);
vote(chg->usb_icl_votable, USB_PSY_VOTER, false, 0);
vote(chg->usb_icl_votable, DCP_VOTER, false, 0);
@@ -4453,11 +4674,15 @@ static void smblib_handle_typec_cc_state_change(struct smb_charger *chg)
smblib_dbg(chg, PR_MISC, "TypeC %s insertion\n",
smblib_typec_mode_name[chg->typec_mode]);
smblib_handle_typec_insertion(chg);
+ vote(chg->disable_prebias_resistor, CC_DETACHED_VOTER,
+ true, 0);
} else if (chg->typec_present &&
chg->typec_mode == POWER_SUPPLY_TYPEC_NONE) {
chg->typec_present = false;
smblib_dbg(chg, PR_MISC, "TypeC removal\n");
smblib_handle_typec_removal(chg);
+ vote(chg->disable_prebias_resistor, CC_DETACHED_VOTER,
+ false, 0);
}
/* suspend usb if sink */
@@ -5218,6 +5443,16 @@ static int smblib_create_votables(struct smb_charger *chg)
return rc;
}
+ chg->disable_prebias_resistor
+ = create_votable("USBIN_PLUGIN_VOTER",
+ VOTE_SET_ANY,
+ smblib_disable_prebias_resistor_callback,
+ chg);
+ if (IS_ERR(chg->disable_prebias_resistor)) {
+ rc = PTR_ERR(chg->disable_prebias_resistor);
+ return rc;
+ }
+
return rc;
}
@@ -5245,6 +5480,8 @@ static void smblib_destroy_votables(struct smb_charger *chg)
destroy_votable(chg->typec_irq_disable_votable);
if (chg->disable_power_role_switch)
destroy_votable(chg->disable_power_role_switch);
+ if (chg->disable_prebias_resistor)
+ destroy_votable(chg->disable_prebias_resistor);
}
static void smblib_iio_deinit(struct smb_charger *chg)
@@ -5261,6 +5498,24 @@ static void smblib_iio_deinit(struct smb_charger *chg)
iio_channel_release(chg->iio.batt_i_chan);
}
+static void smblib_set_low_batt_threshold(struct smb_charger *chg)
+{
+ u32 val;
+ int rc;
+
+ if (!of_property_read_u32(chg->dev->of_node,
+ "google,low-batt-threshold", &val)) {
+ if (val > 0xf)
+ smblib_err(chg, "invalid google,low-batt-threshold value\n");
+ else {
+ rc = smblib_write(chg, LOW_BATT_THRESHOLD_CFG_REG, val);
+ if (rc < 0)
+ smblib_err(chg, "Couldn't write 0x%02x to LOW_BATT_THRESHOLD_CFG_REG rc=%d\n",
+ val, rc);
+ }
+ }
+}
+
int smblib_init(struct smb_charger *chg)
{
int rc = 0;
@@ -5317,6 +5572,8 @@ int smblib_init(struct smb_charger *chg)
return rc;
}
+ smblib_set_low_batt_threshold(chg);
+
chg->bms_psy = power_supply_get_by_name("bms");
chg->pl.psy = power_supply_get_by_name("parallel");
if (chg->pl.psy) {
diff --git a/drivers/power/supply/qcom/smb-lib.h b/drivers/power/supply/qcom/smb-lib.h
index 1d15b9d9..d92ef26 100644
--- a/drivers/power/supply/qcom/smb-lib.h
+++ b/drivers/power/supply/qcom/smb-lib.h
@@ -15,6 +15,7 @@
#include <linux/types.h>
#include <linux/interrupt.h>
#include <linux/irqreturn.h>
+#include <linux/thermal.h>
#include <linux/regulator/driver.h>
#include <linux/regulator/consumer.h>
#include <linux/extcon.h>
@@ -71,6 +72,8 @@ enum print_reason {
#define MOISTURE_VOTER "MOISTURE_VOTER"
#define HVDCP2_ICL_VOTER "HVDCP2_ICL_VOTER"
#define OV_VOTER "OV_VOTER"
+#define USBIN_PLUGIN_VOTER "USBIN_PLUGIN_VOTER"
+
#define VCONN_MAX_ATTEMPTS 3
#define OTG_MAX_ATTEMPTS 3
@@ -263,6 +266,7 @@ struct smb_charger {
struct power_supply *usb_psy;
struct power_supply *dc_psy;
struct power_supply *bms_psy;
+ struct power_supply *wlc_psy;
struct power_supply_desc usb_psy_desc;
struct power_supply *usb_main_psy;
struct power_supply *usb_port_psy;
@@ -298,6 +302,7 @@ struct smb_charger {
struct votable *usb_irq_enable_votable;
struct votable *typec_irq_disable_votable;
struct votable *disable_power_role_switch;
+ struct votable *disable_prebias_resistor;
/* work */
struct work_struct bms_update_work;
@@ -324,11 +329,13 @@ struct smb_charger {
int system_temp_level;
int thermal_levels;
int *thermal_mitigation;
+ int *wlc_thermal_mitigation;
int dcp_icl_ua;
int fake_capacity;
int fake_batt_status;
bool step_chg_enabled;
bool sw_jeita_enabled;
+ int taper_control_enabled;
bool is_hdc;
bool chg_done;
bool connector_type;
@@ -353,6 +360,8 @@ struct smb_charger {
bool use_extcon;
bool otg_present;
bool is_audio_adapter;
+ bool hvdcp_disable;
+ u8 in_explicit_contract;
/* workaround flag */
u32 wa_flags;
@@ -374,6 +383,10 @@ struct smb_charger {
int usb_icl_delta_ua;
int pulse_cnt;
+ /* USB port temperature */
+ const char *usb_port_tz_name;
+ struct thermal_zone_device *usb_port_tz;
+
int die_health;
};
@@ -486,6 +499,8 @@ int smblib_get_prop_usb_voltage_now(struct smb_charger *chg,
union power_supply_propval *val);
int smblib_get_prop_usb_current_now(struct smb_charger *chg,
union power_supply_propval *val);
+int smblib_get_prop_usb_port_temp(struct smb_charger *chg,
+ union power_supply_propval *val);
int smblib_get_prop_typec_cc_orientation(struct smb_charger *chg,
union power_supply_propval *val);
int smblib_get_prop_typec_power_role(struct smb_charger *chg,
@@ -530,6 +545,10 @@ int smblib_set_prop_ship_mode(struct smb_charger *chg,
const union power_supply_propval *val);
int smblib_set_prop_charge_qnovo_enable(struct smb_charger *chg,
const union power_supply_propval *val);
+int smblib_set_prop_otg_fastroleswap(struct smb_charger *chg,
+ const union power_supply_propval *val);
+int smblib_get_prop_otg_fastroleswap(struct smb_charger *chg,
+ union power_supply_propval *val);
void smblib_suspend_on_debug_battery(struct smb_charger *chg);
int smblib_rerun_apsd_if_required(struct smb_charger *chg);
int smblib_get_prop_fcc_delta(struct smb_charger *chg,
@@ -549,6 +568,16 @@ int smblib_stat_sw_override_cfg(struct smb_charger *chg, bool override);
void smblib_usb_typec_change(struct smb_charger *chg);
int smblib_toggle_stat(struct smb_charger *chg, int reset);
+#ifndef CONFIG_QPNP_FG_GEN3_LEGACY_CYCLE_COUNT
+int smblib_get_cycle_count(struct smb_charger *chg,
+ union power_supply_propval *val);
+#endif
int smblib_init(struct smb_charger *chg);
int smblib_deinit(struct smb_charger *chg);
+
+int smblib_get_prop_input_current_max(struct smb_charger *chg,
+ union power_supply_propval *val);
+int smblib_set_prop_input_current_max(struct smb_charger *chg,
+ const union power_supply_propval *val);
+
#endif /* __SMB2_CHARGER_H */
diff --git a/drivers/power/supply/qcom/smb-reg.h b/drivers/power/supply/qcom/smb-reg.h
index d40d6fd..b6e223f 100644
--- a/drivers/power/supply/qcom/smb-reg.h
+++ b/drivers/power/supply/qcom/smb-reg.h
@@ -804,6 +804,7 @@ enum {
#define DC_ENG_SSUPPLY_CFG2_REG (DCIN_BASE + 0xC1)
#define ENG_SSUPPLY_IVREF_OTG_SS_MASK GENMASK(2, 0)
#define OTG_SS_SLOW 0x3
+#define DISABLE_PREBIAS_RESISTOR BIT(7)
#define DC_ENG_SSUPPLY_CFG3_REG (DCIN_BASE + 0xC2)
#define ENG_SSUPPLY_HI_CAP_BIT BIT(6)
@@ -1020,6 +1021,8 @@ enum {
#define CFG_BUCKBOOST_FREQ_SELECT_BUCK_REG (MISC_BASE + 0xA0)
#define CFG_BUCKBOOST_FREQ_SELECT_BOOST_REG (MISC_BASE + 0xA1)
+#define CFG_OTG_OUTPUT_REG (MISC_BASE + 0xA2)
+#define OTG_OUT_5_1_V BIT(0)
#define TM_IO_DTEST4_SEL (MISC_BASE + 0xE9)
diff --git a/drivers/power/supply/qcom/smb1355-charger.c b/drivers/power/supply/qcom/smb1355-charger.c
index 1d99ccb..fe35f79 100644
--- a/drivers/power/supply/qcom/smb1355-charger.c
+++ b/drivers/power/supply/qcom/smb1355-charger.c
@@ -504,6 +504,7 @@ static int smb1355_parse_dt(struct smb1355 *chip)
static enum power_supply_property smb1355_parallel_props[] = {
POWER_SUPPLY_PROP_CHARGE_TYPE,
+ POWER_SUPPLY_PROP_ONLINE,
POWER_SUPPLY_PROP_CHARGING_ENABLED,
POWER_SUPPLY_PROP_PIN_ENABLED,
POWER_SUPPLY_PROP_INPUT_SUSPEND,
@@ -600,6 +601,9 @@ static int smb1355_parallel_get_prop(struct power_supply *psy,
case POWER_SUPPLY_PROP_CHARGE_TYPE:
rc = smb1355_get_prop_batt_charge_type(chip, val);
break;
+ case POWER_SUPPLY_PROP_ONLINE:
+ val->intval = chip->disabled ? 0 : 1;
+ break;
case POWER_SUPPLY_PROP_CHARGING_ENABLED:
rc = smb1355_read(chip, BATTERY_STATUS_3_REG, &stat);
if (rc >= 0)
diff --git a/drivers/ptp/ptp_chardev.c b/drivers/ptp/ptp_chardev.c
index 58a97d42..a421d6c 100644
--- a/drivers/ptp/ptp_chardev.c
+++ b/drivers/ptp/ptp_chardev.c
@@ -24,6 +24,8 @@
#include <linux/slab.h>
#include <linux/timekeeping.h>
+#include <linux/nospec.h>
+
#include "ptp_private.h"
static int ptp_disable_pinfunc(struct ptp_clock_info *ops,
@@ -89,6 +91,7 @@ int ptp_set_pinfunc(struct ptp_clock *ptp, unsigned int pin,
case PTP_PF_PHYSYNC:
if (chan != 0)
return -EINVAL;
+ break;
default:
return -EINVAL;
}
@@ -247,6 +250,7 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)
err = -EINVAL;
break;
}
+ pin_index = array_index_nospec(pin_index, ops->n_pins);
if (mutex_lock_interruptible(&ptp->pincfg_mux))
return -ERESTARTSYS;
pd = ops->pin_config[pin_index];
@@ -265,6 +269,7 @@ long ptp_ioctl(struct posix_clock *pc, unsigned int cmd, unsigned long arg)
err = -EINVAL;
break;
}
+ pin_index = array_index_nospec(pin_index, ops->n_pins);
if (mutex_lock_interruptible(&ptp->pincfg_mux))
return -ERESTARTSYS;
err = ptp_set_pinfunc(ptp, pin_index, pd.func, pd.chan);
diff --git a/drivers/pwm/pwm-lpss-platform.c b/drivers/pwm/pwm-lpss-platform.c
index 54433fc..e4eaefc 100644
--- a/drivers/pwm/pwm-lpss-platform.c
+++ b/drivers/pwm/pwm-lpss-platform.c
@@ -52,6 +52,10 @@ static int pwm_lpss_remove_platform(struct platform_device *pdev)
return pwm_lpss_remove(lpwm);
}
+static SIMPLE_DEV_PM_OPS(pwm_lpss_platform_pm_ops,
+ pwm_lpss_suspend,
+ pwm_lpss_resume);
+
static const struct acpi_device_id pwm_lpss_acpi_match[] = {
{ "80860F09", (unsigned long)&pwm_lpss_byt_info },
{ "80862288", (unsigned long)&pwm_lpss_bsw_info },
@@ -64,6 +68,7 @@ static struct platform_driver pwm_lpss_driver_platform = {
.driver = {
.name = "pwm-lpss",
.acpi_match_table = pwm_lpss_acpi_match,
+ .pm = &pwm_lpss_platform_pm_ops,
},
.probe = pwm_lpss_probe_platform,
.remove = pwm_lpss_remove_platform,
diff --git a/drivers/pwm/pwm-lpss.c b/drivers/pwm/pwm-lpss.c
index 72c0bce..5208b3f 100644
--- a/drivers/pwm/pwm-lpss.c
+++ b/drivers/pwm/pwm-lpss.c
@@ -31,10 +31,13 @@
/* Size of each PWM register space if multiple */
#define PWM_SIZE 0x400
+#define MAX_PWMS 4
+
struct pwm_lpss_chip {
struct pwm_chip chip;
void __iomem *regs;
const struct pwm_lpss_boardinfo *info;
+ u32 saved_ctrl[MAX_PWMS];
};
/* BayTrail */
@@ -168,6 +171,9 @@ struct pwm_lpss_chip *pwm_lpss_probe(struct device *dev, struct resource *r,
unsigned long c;
int ret;
+ if (WARN_ON(info->npwm > MAX_PWMS))
+ return ERR_PTR(-ENODEV);
+
lpwm = devm_kzalloc(dev, sizeof(*lpwm), GFP_KERNEL);
if (!lpwm)
return ERR_PTR(-ENOMEM);
@@ -203,6 +209,30 @@ int pwm_lpss_remove(struct pwm_lpss_chip *lpwm)
}
EXPORT_SYMBOL_GPL(pwm_lpss_remove);
+int pwm_lpss_suspend(struct device *dev)
+{
+ struct pwm_lpss_chip *lpwm = dev_get_drvdata(dev);
+ int i;
+
+ for (i = 0; i < lpwm->info->npwm; i++)
+ lpwm->saved_ctrl[i] = readl(lpwm->regs + i * PWM_SIZE + PWM);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(pwm_lpss_suspend);
+
+int pwm_lpss_resume(struct device *dev)
+{
+ struct pwm_lpss_chip *lpwm = dev_get_drvdata(dev);
+ int i;
+
+ for (i = 0; i < lpwm->info->npwm; i++)
+ writel(lpwm->saved_ctrl[i], lpwm->regs + i * PWM_SIZE + PWM);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(pwm_lpss_resume);
+
MODULE_DESCRIPTION("PWM driver for Intel LPSS");
MODULE_AUTHOR("Mika Westerberg <mika.westerberg@linux.intel.com>");
MODULE_LICENSE("GPL v2");
diff --git a/drivers/pwm/pwm-lpss.h b/drivers/pwm/pwm-lpss.h
index 04766e0..27d5081 100644
--- a/drivers/pwm/pwm-lpss.h
+++ b/drivers/pwm/pwm-lpss.h
@@ -31,5 +31,7 @@ extern const struct pwm_lpss_boardinfo pwm_lpss_bxt_info;
struct pwm_lpss_chip *pwm_lpss_probe(struct device *dev, struct resource *r,
const struct pwm_lpss_boardinfo *info);
int pwm_lpss_remove(struct pwm_lpss_chip *lpwm);
+int pwm_lpss_suspend(struct device *dev);
+int pwm_lpss_resume(struct device *dev);
#endif /* __PWM_LPSS_H */
diff --git a/drivers/pwm/pwm-qpnp.c b/drivers/pwm/pwm-qpnp.c
index e7267de..959208a 100644
--- a/drivers/pwm/pwm-qpnp.c
+++ b/drivers/pwm/pwm-qpnp.c
@@ -2144,7 +2144,8 @@ static int qpnp_pwm_probe(struct platform_device *pdev)
pwm_chip->regmap = dev_get_regmap(pdev->dev.parent, NULL);
if (!pwm_chip->regmap) {
dev_err(&pdev->dev, "Couldn't get parent's regmap\n");
- return -EINVAL;
+ rc = -EINVAL;
+ goto failed_regmap;
}
spin_lock_init(&pwm_chip->lpg_lock);
@@ -2181,6 +2182,7 @@ static int qpnp_pwm_probe(struct platform_device *pdev)
kfree(pwm_chip->lpg_config.lut_config.duty_pct_list);
failed_config:
dev_set_drvdata(&pdev->dev, NULL);
+failed_regmap:
kfree(pwm_chip);
return rc;
}
diff --git a/drivers/pwm/pwm-tiehrpwm.c b/drivers/pwm/pwm-tiehrpwm.c
index b5c6b063..c0e06f0 100644
--- a/drivers/pwm/pwm-tiehrpwm.c
+++ b/drivers/pwm/pwm-tiehrpwm.c
@@ -382,6 +382,8 @@ static void ehrpwm_pwm_disable(struct pwm_chip *chip, struct pwm_device *pwm)
aqcsfrc_mask = AQCSFRC_CSFA_MASK;
}
+ /* Update shadow register first before modifying active register */
+ ehrpwm_modify(pc->mmio_base, AQCSFRC, aqcsfrc_mask, aqcsfrc_val);
/*
* Changes to immediate action on Action Qualifier. This puts
* Action Qualifier control on PWM output from next TBCLK
diff --git a/drivers/regulator/core.c b/drivers/regulator/core.c
index 1a254999..0d81304 100644
--- a/drivers/regulator/core.c
+++ b/drivers/regulator/core.c
@@ -4458,13 +4458,13 @@ regulator_register(const struct regulator_desc *regulator_desc,
!rdev->desc->fixed_uV)
rdev->is_switch = true;
+ dev_set_drvdata(&rdev->dev, rdev);
ret = device_register(&rdev->dev);
if (ret != 0) {
put_device(&rdev->dev);
goto unset_supplies;
}
- dev_set_drvdata(&rdev->dev, rdev);
rdev_init_debugfs(rdev);
rdev->proxy_consumer = regulator_proxy_consumer_register(dev,
config->of_node);
diff --git a/drivers/regulator/gpio-regulator.c b/drivers/regulator/gpio-regulator.c
index 83e89e5..b73a237 100644
--- a/drivers/regulator/gpio-regulator.c
+++ b/drivers/regulator/gpio-regulator.c
@@ -268,8 +268,7 @@ static int gpio_regulator_probe(struct platform_device *pdev)
drvdata->desc.name = kstrdup(config->supply_name, GFP_KERNEL);
if (drvdata->desc.name == NULL) {
dev_err(&pdev->dev, "Failed to allocate supply name\n");
- ret = -ENOMEM;
- goto err;
+ return -ENOMEM;
}
if (config->nr_gpios != 0) {
@@ -289,7 +288,7 @@ static int gpio_regulator_probe(struct platform_device *pdev)
dev_err(&pdev->dev,
"Could not obtain regulator setting GPIOs: %d\n",
ret);
- goto err_memstate;
+ goto err_memgpio;
}
}
@@ -300,7 +299,7 @@ static int gpio_regulator_probe(struct platform_device *pdev)
if (drvdata->states == NULL) {
dev_err(&pdev->dev, "Failed to allocate state data\n");
ret = -ENOMEM;
- goto err_memgpio;
+ goto err_stategpio;
}
drvdata->nr_states = config->nr_states;
@@ -321,7 +320,7 @@ static int gpio_regulator_probe(struct platform_device *pdev)
default:
dev_err(&pdev->dev, "No regulator type set\n");
ret = -EINVAL;
- goto err_memgpio;
+ goto err_memstate;
}
/* build initial state from gpio init data. */
@@ -358,22 +357,21 @@ static int gpio_regulator_probe(struct platform_device *pdev)
if (IS_ERR(drvdata->dev)) {
ret = PTR_ERR(drvdata->dev);
dev_err(&pdev->dev, "Failed to register regulator: %d\n", ret);
- goto err_stategpio;
+ goto err_memstate;
}
platform_set_drvdata(pdev, drvdata);
return 0;
-err_stategpio:
- gpio_free_array(drvdata->gpios, drvdata->nr_gpios);
err_memstate:
kfree(drvdata->states);
+err_stategpio:
+ gpio_free_array(drvdata->gpios, drvdata->nr_gpios);
err_memgpio:
kfree(drvdata->gpios);
err_name:
kfree(drvdata->desc.name);
-err:
return ret;
}
diff --git a/drivers/regulator/of_regulator.c b/drivers/regulator/of_regulator.c
index 4f613ec..037675b 100644
--- a/drivers/regulator/of_regulator.c
+++ b/drivers/regulator/of_regulator.c
@@ -282,6 +282,7 @@ int of_regulator_match(struct device *dev, struct device_node *node,
dev_err(dev,
"failed to parse DT for regulator %s\n",
child->name);
+ of_node_put(child);
return -EINVAL;
}
match->of_node = of_node_get(child);
diff --git a/drivers/regulator/pfuze100-regulator.c b/drivers/regulator/pfuze100-regulator.c
index cb18b5c..86b3487 100644
--- a/drivers/regulator/pfuze100-regulator.c
+++ b/drivers/regulator/pfuze100-regulator.c
@@ -153,6 +153,7 @@ static struct regulator_ops pfuze100_sw_regulator_ops = {
static struct regulator_ops pfuze100_swb_regulator_ops = {
.enable = regulator_enable_regmap,
.disable = regulator_disable_regmap,
+ .is_enabled = regulator_is_enabled_regmap,
.list_voltage = regulator_list_voltage_table,
.map_voltage = regulator_map_voltage_ascend,
.set_voltage_sel = regulator_set_voltage_sel_regmap,
diff --git a/drivers/regulator/qpnp-labibb-regulator.c b/drivers/regulator/qpnp-labibb-regulator.c
index d72af20..246765c 100644
--- a/drivers/regulator/qpnp-labibb-regulator.c
+++ b/drivers/regulator/qpnp-labibb-regulator.c
@@ -51,6 +51,7 @@
/* LAB register offset definitions */
#define REG_LAB_STATUS1 0x08
+#define REG_LAB_INT_RT_STS 0x10
#define REG_LAB_SWIRE_PGM_CTL 0x40
#define REG_LAB_VOLTAGE 0x41
#define REG_LAB_RING_SUPPRESSION_CTL 0x42
@@ -103,7 +104,6 @@
#define LAB_PD_CTL_STRONG_PULL BIT(0)
#define LAB_PD_CTL_STRENGTH_MASK BIT(0)
#define LAB_PD_CTL_DISABLE_PD BIT(1)
-#define LAB_PD_CTL_EN_MASK BIT(1)
/* REG_LAB_IBB_EN_RDY */
#define LAB_IBB_EN_RDY_EN BIT(7)
@@ -177,6 +177,7 @@
#define REG_IBB_OUTPUT_SLEW_CTL 0x5D
#define REG_IBB_SPARE_CTL 0x60
#define REG_IBB_NLIMIT_DAC 0x61
+#define REG_IBB_SWIRE_WLED_CTL1 0x64
/* IBB registers for PM660A */
#define REG_IBB_DEFAULT_VOLTAGE 0x40
@@ -289,6 +290,10 @@
/* REG_IBB_NLIMIT_DAC */
#define IBB_DEFAULT_NLIMIT_DAC 0x5
+/* REG_IBB_SWIRE_WLED_CTL1 */
+#define AVDD_MIN_MASK GENMASK(7, 4)
+#define AVDD_MIN_SHIFT 4
+
/* REG_IBB_PFM_CTL */
#define IBB_PFM_ENABLE BIT(7)
#define IBB_PFM_PEAK_CURRENT_BIT_SHIFT 1
@@ -570,6 +575,7 @@ struct lab_regulator {
int sc_wait_time_ms;
int vreg_enabled;
+ unsigned int mode;
};
struct ibb_regulator {
@@ -610,6 +616,7 @@ struct qpnp_labibb {
struct mutex bus_mutex;
enum qpnp_labibb_mode mode;
struct work_struct lab_vreg_ok_work;
+ struct work_struct aod_lab_vreg_ok_work;
struct delayed_work sc_err_recovery_work;
struct hrtimer sc_err_check_timer;
int sc_err_count;
@@ -625,6 +632,7 @@ struct qpnp_labibb {
bool notify_lab_vreg_ok_sts;
bool detect_lab_sc;
bool sc_detected;
+ bool aod_mode;
/* Tracks the secure UI mode entry/exit */
bool secure_mode;
u32 swire_2nd_cmd_delay;
@@ -834,6 +842,57 @@ static int qpnp_labibb_sec_masked_write(struct qpnp_labibb *labibb, u16 base,
return rc;
}
+static int qpnp_lab_scp_control(struct qpnp_labibb *labibb, bool enable)
+{
+ int rc;
+ u8 val;
+
+ val = enable ? 0 : LAB_SPARE_DISABLE_SCP_BIT;
+ rc = qpnp_labibb_masked_write(labibb, labibb->lab_base +
+ REG_LAB_SPARE_CTL, LAB_SPARE_DISABLE_SCP_BIT, val);
+ if (rc < 0) {
+ pr_err("qpnp_labibb_write register %x failed rc = %d\n",
+ REG_LAB_SPARE_CTL, rc);
+ return rc;
+ }
+
+ return 0;
+}
+
+static int qpnp_lab_pd_control(struct qpnp_labibb *labibb, bool enable)
+{
+ int rc;
+ u8 val;
+
+ val = enable ? 0 : LAB_PD_CTL_DISABLE_PD;
+ rc = qpnp_labibb_masked_write(labibb, labibb->lab_base + REG_LAB_PD_CTL,
+ LAB_PD_CTL_DISABLE_PD, val);
+ if (rc < 0) {
+ pr_err("write to register %x failed rc = %d\n",
+ REG_LAB_PD_CTL, rc);
+ return rc;
+ }
+
+ return 0;
+}
+
+static int qpnp_ibb_pd_control(struct qpnp_labibb *labibb, bool enable)
+{
+ int rc;
+ u8 val;
+
+ val = enable ? IBB_PD_CTL_EN : 0;
+ rc = qpnp_labibb_masked_write(labibb, labibb->ibb_base +
+ REG_IBB_PD_CTL, IBB_PD_CTL_EN_MASK, val);
+ if (rc < 0) {
+ pr_err("write to register %x failed rc = %d\n",
+ REG_IBB_PD_CTL, rc);
+ return rc;
+ }
+
+ return 0;
+}
+
static int qpnp_ibb_smart_ps_config_v1(struct qpnp_labibb *labibb, bool enable,
int num_swire_trans, int neg_curr_limit)
{
@@ -1494,7 +1553,7 @@ static int qpnp_lab_dt_init(struct qpnp_labibb *labibb,
if (!of_property_read_bool(of_node, "qcom,qpnp-lab-pull-down-enable"))
val |= LAB_PD_CTL_DISABLE_PD;
- mask = LAB_PD_CTL_EN_MASK | LAB_PD_CTL_STRENGTH_MASK;
+ mask = LAB_PD_CTL_DISABLE_PD | LAB_PD_CTL_STRENGTH_MASK;
rc = qpnp_labibb_masked_write(labibb, labibb->lab_base + REG_LAB_PD_CTL,
mask, val);
@@ -2186,6 +2245,75 @@ static int qpnp_labibb_regulator_ttw_mode_exit(struct qpnp_labibb *labibb)
return rc;
}
+static void qpnp_aod_lab_vreg_ok_work(struct work_struct *work)
+{
+ struct qpnp_labibb *labibb = container_of(work, struct qpnp_labibb,
+ aod_lab_vreg_ok_work);
+ int rc, retries = 100;
+ u8 val;
+ ktime_t vreg_en_time;
+ s64 time_ms;
+
+ vreg_en_time = ktime_get();
+
+ while (retries--) {
+ rc = qpnp_labibb_read(labibb, labibb->lab_base +
+ REG_LAB_STATUS1, &val, 1);
+ if (rc < 0) {
+ pr_err("read register %x failed rc = %d\n",
+ REG_LAB_STATUS1, rc);
+ goto out;
+ }
+
+ if (val & LAB_STATUS1_VREG_OK_BIT)
+ break;
+
+ usleep_range(5000, 5010);
+ }
+
+ if (!(val & LAB_STATUS1_VREG_OK_BIT)) {
+ time_ms = ktime_ms_delta(ktime_get(), vreg_en_time);
+ pr_err("LAB_VREG_OK not set (%x) waited for %lld ms!\n", val,
+ time_ms);
+ goto out;
+ }
+
+ rc = qpnp_lab_scp_control(labibb, true);
+ if (rc < 0)
+ goto out;
+
+ rc = qpnp_lab_pd_control(labibb, true);
+ if (rc < 0)
+ goto out;
+
+ retries = 10;
+ while (retries--) {
+ rc = qpnp_labibb_read(labibb, labibb->ibb_base +
+ REG_IBB_STATUS1, &val, 1);
+ if (rc < 0) {
+ pr_err("read register %x failed rc = %d\n",
+ REG_IBB_STATUS1, rc);
+ goto out;
+ }
+
+ if (val & IBB_STATUS1_VREG_OK_BIT)
+ break;
+
+ usleep_range(2000, 2010);
+ }
+
+ if (!(val & IBB_STATUS1_VREG_OK_BIT)) {
+ time_ms = ktime_ms_delta(ktime_get(), vreg_en_time);
+ pr_err("IBB_VREG_OK not set (%x) waited for %lld ms!\n", val,
+ time_ms);
+ goto out;
+ }
+
+ rc = qpnp_ibb_pd_control(labibb, true);
+out:
+ return;
+}
+
static void qpnp_lab_vreg_notifier_work(struct work_struct *work)
{
int rc = 0;
@@ -2525,6 +2653,7 @@ static int qpnp_lab_regulator_disable(struct regulator_dev *rdev)
labibb->lab_vreg.vreg_enabled = 0;
}
+
return 0;
}
@@ -2538,6 +2667,78 @@ static int qpnp_lab_regulator_is_enabled(struct regulator_dev *rdev)
return labibb->lab_vreg.vreg_enabled;
}
+static int qpnp_lab_regulator_set_mode(struct regulator_dev *rdev,
+ unsigned int mode)
+{
+ struct qpnp_labibb *labibb = rdev_get_drvdata(rdev);
+ int rc;
+ u8 val;
+
+ if (!labibb->aod_mode)
+ return 0;
+
+ /* AVDD_MIN voltage = 5.65 + code * 0.15 V */
+ if (mode == REGULATOR_MODE_NORMAL || mode == REGULATOR_MODE_STANDBY)
+ val = 0xD;
+ else if (mode == REGULATOR_MODE_IDLE)
+ val = 0x1;
+ else
+ return -EINVAL;
+
+ val <<= AVDD_MIN_SHIFT;
+ rc = qpnp_labibb_masked_write(labibb, labibb->ibb_base +
+ REG_IBB_SWIRE_WLED_CTL1, AVDD_MIN_MASK, val);
+ if (rc < 0) {
+ pr_err("write to register %x failed rc = %d\n",
+ REG_IBB_SWIRE_WLED_CTL1, rc);
+ return rc;
+ }
+
+ if (mode == REGULATOR_MODE_NORMAL) {
+ schedule_work(&labibb->aod_lab_vreg_ok_work);
+ } else if (mode == REGULATOR_MODE_IDLE) {
+ cancel_work_sync(&labibb->aod_lab_vreg_ok_work);
+ rc = qpnp_ibb_pd_control(labibb, false);
+ if (rc < 0)
+ goto out;
+
+ rc = qpnp_lab_pd_control(labibb, false);
+ if (rc < 0)
+ goto out;
+
+ rc = qpnp_lab_scp_control(labibb, false);
+ if (rc < 0)
+ goto out;
+ } else if (mode == REGULATOR_MODE_STANDBY) {
+ cancel_work_sync(&labibb->aod_lab_vreg_ok_work);
+ rc = qpnp_ibb_pd_control(labibb, true);
+ if (rc < 0)
+ goto out;
+
+ rc = qpnp_lab_pd_control(labibb, true);
+ if (rc < 0)
+ goto out;
+
+ rc = qpnp_lab_scp_control(labibb, true);
+ if (rc < 0)
+ goto out;
+ }
+
+ labibb->lab_vreg.mode = mode;
+out:
+ return rc;
+}
+
+static unsigned int qpnp_lab_regulator_get_mode(struct regulator_dev *rdev)
+{
+ struct qpnp_labibb *labibb = rdev_get_drvdata(rdev);
+
+ if (!labibb->aod_mode)
+ return REGULATOR_MODE_NORMAL;
+
+ return labibb->lab_vreg.mode;
+}
+
static int qpnp_labibb_force_enable(struct qpnp_labibb *labibb)
{
int rc;
@@ -2864,6 +3065,8 @@ static struct regulator_ops qpnp_lab_ops = {
.is_enabled = qpnp_lab_regulator_is_enabled,
.set_voltage = qpnp_lab_regulator_set_voltage,
.get_voltage = qpnp_lab_regulator_get_voltage,
+ .set_mode = qpnp_lab_regulator_set_mode,
+ .get_mode = qpnp_lab_regulator_get_mode,
};
static int register_qpnp_lab_regulator(struct qpnp_labibb *labibb,
@@ -3149,6 +3352,15 @@ static int register_qpnp_lab_regulator(struct qpnp_labibb *labibb,
|= REGULATOR_CHANGE_VOLTAGE |
REGULATOR_CHANGE_STATUS;
+ if (labibb->aod_mode) {
+ init_data->constraints.valid_ops_mask |=
+ REGULATOR_CHANGE_MODE;
+ init_data->constraints.valid_modes_mask |=
+ REGULATOR_MODE_NORMAL | REGULATOR_MODE_IDLE |
+ REGULATOR_MODE_STANDBY;
+ labibb->lab_vreg.mode = REGULATOR_MODE_NORMAL;
+ }
+
labibb->lab_vreg.rdev = regulator_register(rdesc, &cfg);
if (IS_ERR(labibb->lab_vreg.rdev)) {
rc = PTR_ERR(labibb->lab_vreg.rdev);
@@ -4181,6 +4393,9 @@ static int qpnp_labibb_regulator_probe(struct platform_device *pdev)
of_property_read_bool(labibb->dev->of_node,
"qcom,skip-2nd-swire-cmd");
+ labibb->aod_mode = of_property_read_bool(labibb->dev->of_node,
+ "qcom,aod-mode");
+
rc = of_property_read_u32(labibb->dev->of_node,
"qcom,swire-2nd-cmd-delay",
&labibb->swire_2nd_cmd_delay);
@@ -4266,6 +4481,7 @@ static int qpnp_labibb_regulator_probe(struct platform_device *pdev)
}
INIT_WORK(&labibb->lab_vreg_ok_work, qpnp_lab_vreg_notifier_work);
+ INIT_WORK(&labibb->aod_lab_vreg_ok_work, qpnp_aod_lab_vreg_ok_work);
INIT_DELAYED_WORK(&labibb->sc_err_recovery_work,
labibb_sc_err_recovery_work);
hrtimer_init(&labibb->sc_err_check_timer,
@@ -4322,6 +4538,8 @@ static int qpnp_labibb_regulator_remove(struct platform_device *pdev)
regulator_unregister(labibb->ibb_vreg.rdev);
cancel_work_sync(&labibb->lab_vreg_ok_work);
+ if (labibb->aod_mode)
+ cancel_work_sync(&labibb->aod_lab_vreg_ok_work);
}
return 0;
}
diff --git a/drivers/regulator/qpnp-regulator.c b/drivers/regulator/qpnp-regulator.c
index bd70665..c0d04e41 100644
--- a/drivers/regulator/qpnp-regulator.c
+++ b/drivers/regulator/qpnp-regulator.c
@@ -2184,6 +2184,7 @@ static int qpnp_regulator_probe(struct platform_device *pdev)
vreg->regmap = dev_get_regmap(pdev->dev.parent, NULL);
if (!vreg->regmap) {
dev_err(&pdev->dev, "Couldn't get parent's regmap\n");
+ kfree(vreg);
return -EINVAL;
}
diff --git a/drivers/reset/core.c b/drivers/reset/core.c
index b8ae1db..188205a 100644
--- a/drivers/reset/core.c
+++ b/drivers/reset/core.c
@@ -135,11 +135,16 @@ EXPORT_SYMBOL_GPL(devm_reset_controller_register);
* @rstc: reset controller
*
* Calling this on a shared reset controller is an error.
+ *
+ * If rstc is NULL it is an optional reset and the function will just
+ * return 0.
*/
int reset_control_reset(struct reset_control *rstc)
{
- if (WARN_ON(IS_ERR_OR_NULL(rstc)) ||
- WARN_ON(rstc->shared))
+ if (!rstc)
+ return 0;
+
+ if (WARN_ON(IS_ERR(rstc)))
return -EINVAL;
if (rstc->rcdev->ops->reset)
@@ -159,10 +164,16 @@ EXPORT_SYMBOL_GPL(reset_control_reset);
*
* For shared reset controls a driver cannot expect the hw's registers and
* internal state to be reset, but must be prepared for this to happen.
+ *
+ * If rstc is NULL it is an optional reset and the function will just
+ * return 0.
*/
int reset_control_assert(struct reset_control *rstc)
{
- if (WARN_ON(IS_ERR_OR_NULL(rstc)))
+ if (!rstc)
+ return 0;
+
+ if (WARN_ON(IS_ERR(rstc)))
return -EINVAL;
if (!rstc->rcdev->ops->assert)
@@ -185,10 +196,16 @@ EXPORT_SYMBOL_GPL(reset_control_assert);
* @rstc: reset controller
*
* After calling this function, the reset is guaranteed to be deasserted.
+ *
+ * If rstc is NULL it is an optional reset and the function will just
+ * return 0.
*/
int reset_control_deassert(struct reset_control *rstc)
{
- if (WARN_ON(IS_ERR_OR_NULL(rstc)))
+ if (!rstc)
+ return 0;
+
+ if (WARN_ON(IS_ERR(rstc)))
return -EINVAL;
if (!rstc->rcdev->ops->deassert)
@@ -206,12 +223,15 @@ EXPORT_SYMBOL_GPL(reset_control_deassert);
/**
* reset_control_status - returns a negative errno if not supported, a
* positive value if the reset line is asserted, or zero if the reset
- * line is not asserted.
+ * line is not asserted or if the desc is NULL (optional reset).
* @rstc: reset controller
*/
int reset_control_status(struct reset_control *rstc)
{
- if (WARN_ON(IS_ERR_OR_NULL(rstc)))
+ if (!rstc)
+ return 0;
+
+ if (WARN_ON(IS_ERR(rstc)))
return -EINVAL;
if (rstc->rcdev->ops->status)
@@ -221,7 +241,7 @@ int reset_control_status(struct reset_control *rstc)
}
EXPORT_SYMBOL_GPL(reset_control_status);
-static struct reset_control *__reset_control_get(
+static struct reset_control *__reset_control_get_internal(
struct reset_controller_dev *rcdev,
unsigned int index, int shared)
{
@@ -254,7 +274,7 @@ static struct reset_control *__reset_control_get(
return rstc;
}
-static void __reset_control_put(struct reset_control *rstc)
+static void __reset_control_put_internal(struct reset_control *rstc)
{
lockdep_assert_held(&reset_list_mutex);
@@ -268,7 +288,8 @@ static void __reset_control_put(struct reset_control *rstc)
}
struct reset_control *__of_reset_control_get(struct device_node *node,
- const char *id, int index, int shared)
+ const char *id, int index, bool shared,
+ bool optional)
{
struct reset_control *rstc;
struct reset_controller_dev *r, *rcdev;
@@ -282,14 +303,18 @@ struct reset_control *__of_reset_control_get(struct device_node *node,
if (id) {
index = of_property_match_string(node,
"reset-names", id);
+ if (index == -EILSEQ)
+ return ERR_PTR(index);
if (index < 0)
- return ERR_PTR(-ENOENT);
+ return optional ? NULL : ERR_PTR(-ENOENT);
}
ret = of_parse_phandle_with_args(node, "resets", "#reset-cells",
index, &args);
- if (ret)
+ if (ret == -EINVAL)
return ERR_PTR(ret);
+ if (ret)
+ return optional ? NULL : ERR_PTR(ret);
mutex_lock(&reset_list_mutex);
rcdev = NULL;
@@ -318,7 +343,7 @@ struct reset_control *__of_reset_control_get(struct device_node *node,
}
/* reset_list_mutex also protects the rcdev's reset_control list */
- rstc = __reset_control_get(rcdev, rstc_id, shared);
+ rstc = __reset_control_get_internal(rcdev, rstc_id, shared);
mutex_unlock(&reset_list_mutex);
@@ -326,6 +351,17 @@ struct reset_control *__of_reset_control_get(struct device_node *node,
}
EXPORT_SYMBOL_GPL(__of_reset_control_get);
+struct reset_control *__reset_control_get(struct device *dev, const char *id,
+ int index, bool shared, bool optional)
+{
+ if (dev->of_node)
+ return __of_reset_control_get(dev->of_node, id, index, shared,
+ optional);
+
+ return optional ? NULL : ERR_PTR(-EINVAL);
+}
+EXPORT_SYMBOL_GPL(__reset_control_get);
+
/**
* reset_control_put - free the reset controller
* @rstc: reset controller
@@ -333,11 +369,11 @@ EXPORT_SYMBOL_GPL(__of_reset_control_get);
void reset_control_put(struct reset_control *rstc)
{
- if (IS_ERR(rstc))
+ if (IS_ERR_OR_NULL(rstc))
return;
mutex_lock(&reset_list_mutex);
- __reset_control_put(rstc);
+ __reset_control_put_internal(rstc);
mutex_unlock(&reset_list_mutex);
}
EXPORT_SYMBOL_GPL(reset_control_put);
@@ -348,7 +384,8 @@ static void devm_reset_control_release(struct device *dev, void *res)
}
struct reset_control *__devm_reset_control_get(struct device *dev,
- const char *id, int index, int shared)
+ const char *id, int index, bool shared,
+ bool optional)
{
struct reset_control **ptr, *rstc;
@@ -357,8 +394,7 @@ struct reset_control *__devm_reset_control_get(struct device *dev,
if (!ptr)
return ERR_PTR(-ENOMEM);
- rstc = __of_reset_control_get(dev ? dev->of_node : NULL,
- id, index, shared);
+ rstc = __reset_control_get(dev, id, index, shared, optional);
if (!IS_ERR(rstc)) {
*ptr = rstc;
devres_add(dev, ptr);
@@ -374,17 +410,18 @@ EXPORT_SYMBOL_GPL(__devm_reset_control_get);
* device_reset - find reset controller associated with the device
* and perform reset
* @dev: device to be reset by the controller
+ * @optional: whether it is optional to reset the device
*
- * Convenience wrapper for reset_control_get() and reset_control_reset().
+ * Convenience wrapper for __reset_control_get() and reset_control_reset().
* This is useful for the common case of devices with single, dedicated reset
* lines.
*/
-int device_reset(struct device *dev)
+int __device_reset(struct device *dev, bool optional)
{
struct reset_control *rstc;
int ret;
- rstc = reset_control_get(dev, NULL);
+ rstc = __reset_control_get(dev, NULL, 0, 0, optional);
if (IS_ERR(rstc))
return PTR_ERR(rstc);
@@ -394,4 +431,4 @@ int device_reset(struct device *dev)
return ret;
}
-EXPORT_SYMBOL_GPL(device_reset);
+EXPORT_SYMBOL_GPL(__device_reset);
diff --git a/drivers/reset/hisilicon/hi6220_reset.c b/drivers/reset/hisilicon/hi6220_reset.c
index 35ce53e..d5e5229 100644
--- a/drivers/reset/hisilicon/hi6220_reset.c
+++ b/drivers/reset/hisilicon/hi6220_reset.c
@@ -155,3 +155,5 @@ static int __init hi6220_reset_init(void)
}
postcore_initcall(hi6220_reset_init);
+
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/rpmsg/qcom_smd.c b/drivers/rpmsg/qcom_smd.c
index 1d4770c..312cb7f 100644
--- a/drivers/rpmsg/qcom_smd.c
+++ b/drivers/rpmsg/qcom_smd.c
@@ -1006,14 +1006,16 @@ static struct qcom_smd_channel *qcom_smd_create_channel(struct qcom_smd_edge *ed
void *info;
int ret;
- channel = devm_kzalloc(&edge->dev, sizeof(*channel), GFP_KERNEL);
+ channel = kzalloc(sizeof(*channel), GFP_KERNEL);
if (!channel)
return ERR_PTR(-ENOMEM);
channel->edge = edge;
- channel->name = devm_kstrdup(&edge->dev, name, GFP_KERNEL);
- if (!channel->name)
- return ERR_PTR(-ENOMEM);
+ channel->name = kstrdup(name, GFP_KERNEL);
+ if (!channel->name) {
+ ret = -ENOMEM;
+ goto free_channel;
+ }
mutex_init(&channel->tx_lock);
spin_lock_init(&channel->recv_lock);
@@ -1061,8 +1063,9 @@ static struct qcom_smd_channel *qcom_smd_create_channel(struct qcom_smd_edge *ed
return channel;
free_name_and_channel:
- devm_kfree(&edge->dev, channel->name);
- devm_kfree(&edge->dev, channel);
+ kfree(channel->name);
+free_channel:
+ kfree(channel);
return ERR_PTR(ret);
}
@@ -1279,13 +1282,13 @@ static int qcom_smd_parse_edge(struct device *dev,
*/
static void qcom_smd_edge_release(struct device *dev)
{
- struct qcom_smd_channel *channel;
+ struct qcom_smd_channel *channel, *tmp;
struct qcom_smd_edge *edge = to_smd_edge(dev);
- list_for_each_entry(channel, &edge->channels, list) {
- SET_RX_CHANNEL_INFO(channel, state, SMD_CHANNEL_CLOSED);
- SET_RX_CHANNEL_INFO(channel, head, 0);
- SET_RX_CHANNEL_INFO(channel, tail, 0);
+ list_for_each_entry_safe(channel, tmp, &edge->channels, list) {
+ list_del(&channel->list);
+ kfree(channel->name);
+ kfree(channel);
}
kfree(edge);
diff --git a/drivers/rtc/hctosys.c b/drivers/rtc/hctosys.c
index e1cfa06..b9ec4a1 100644
--- a/drivers/rtc/hctosys.c
+++ b/drivers/rtc/hctosys.c
@@ -49,6 +49,13 @@ static int __init rtc_hctosys(void)
tv64.tv_sec = rtc_tm_to_time64(&tm);
+#if BITS_PER_LONG == 32
+ if (tv64.tv_sec > INT_MAX) {
+ err = -ERANGE;
+ goto err_read;
+ }
+#endif
+
err = do_settimeofday64(&tv64);
dev_info(rtc->dev.parent,
diff --git a/drivers/rtc/interface.c b/drivers/rtc/interface.c
index 162afcc..d37960e 100644
--- a/drivers/rtc/interface.c
+++ b/drivers/rtc/interface.c
@@ -359,6 +359,11 @@ int rtc_set_alarm(struct rtc_device *rtc, struct rtc_wkalrm *alarm)
{
int err;
+ if (!rtc->ops)
+ return -ENODEV;
+ else if (!rtc->ops->set_alarm)
+ return -EINVAL;
+
err = rtc_valid_tm(&alarm->time);
if (err != 0)
return err;
diff --git a/drivers/rtc/rtc-bq4802.c b/drivers/rtc/rtc-bq4802.c
index bd170cb..5747a54 100644
--- a/drivers/rtc/rtc-bq4802.c
+++ b/drivers/rtc/rtc-bq4802.c
@@ -164,6 +164,10 @@ static int bq4802_probe(struct platform_device *pdev)
} else if (p->r->flags & IORESOURCE_MEM) {
p->regs = devm_ioremap(&pdev->dev, p->r->start,
resource_size(p->r));
+ if (!p->regs){
+ err = -ENOMEM;
+ goto out;
+ }
p->read = bq4802_read_mem;
p->write = bq4802_write_mem;
} else {
diff --git a/drivers/rtc/rtc-m41t80.c b/drivers/rtc/rtc-m41t80.c
index c4ca6a3..6b6b623 100644
--- a/drivers/rtc/rtc-m41t80.c
+++ b/drivers/rtc/rtc-m41t80.c
@@ -333,7 +333,7 @@ static int m41t80_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
alrm->time.tm_min = bcd2bin(alarmvals[3] & 0x7f);
alrm->time.tm_hour = bcd2bin(alarmvals[2] & 0x3f);
alrm->time.tm_mday = bcd2bin(alarmvals[1] & 0x3f);
- alrm->time.tm_mon = bcd2bin(alarmvals[0] & 0x3f);
+ alrm->time.tm_mon = bcd2bin(alarmvals[0] & 0x3f) - 1;
alrm->enabled = !!(alarmvals[0] & M41T80_ALMON_AFE);
alrm->pending = (flags & M41T80_FLAGS_AF) && alrm->enabled;
diff --git a/drivers/rtc/rtc-omap.c b/drivers/rtc/rtc-omap.c
index 51e5244..bd5ca54 100644
--- a/drivers/rtc/rtc-omap.c
+++ b/drivers/rtc/rtc-omap.c
@@ -817,13 +817,6 @@ static int omap_rtc_probe(struct platform_device *pdev)
goto err;
}
- if (rtc->is_pmic_controller) {
- if (!pm_power_off) {
- omap_rtc_power_off_rtc = rtc;
- pm_power_off = omap_rtc_power_off;
- }
- }
-
/* Support ext_wakeup pinconf */
rtc_pinctrl_desc.name = dev_name(&pdev->dev);
@@ -833,6 +826,13 @@ static int omap_rtc_probe(struct platform_device *pdev)
return PTR_ERR(rtc->pctldev);
}
+ if (rtc->is_pmic_controller) {
+ if (!pm_power_off) {
+ omap_rtc_power_off_rtc = rtc;
+ pm_power_off = omap_rtc_power_off;
+ }
+ }
+
return 0;
err:
diff --git a/drivers/rtc/rtc-opal.c b/drivers/rtc/rtc-opal.c
index aa53fce..180ec1f 100644
--- a/drivers/rtc/rtc-opal.c
+++ b/drivers/rtc/rtc-opal.c
@@ -57,7 +57,7 @@ static void tm_to_opal(struct rtc_time *tm, u32 *y_m_d, u64 *h_m_s_ms)
static int opal_get_rtc_time(struct device *dev, struct rtc_time *tm)
{
- long rc = OPAL_BUSY;
+ s64 rc = OPAL_BUSY;
int retries = 10;
u32 y_m_d;
u64 h_m_s_ms;
@@ -66,13 +66,17 @@ static int opal_get_rtc_time(struct device *dev, struct rtc_time *tm)
while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT) {
rc = opal_rtc_read(&__y_m_d, &__h_m_s_ms);
- if (rc == OPAL_BUSY_EVENT)
+ if (rc == OPAL_BUSY_EVENT) {
+ msleep(OPAL_BUSY_DELAY_MS);
opal_poll_events(NULL);
- else if (retries-- && (rc == OPAL_HARDWARE
- || rc == OPAL_INTERNAL_ERROR))
- msleep(10);
- else if (rc != OPAL_BUSY && rc != OPAL_BUSY_EVENT)
- break;
+ } else if (rc == OPAL_BUSY) {
+ msleep(OPAL_BUSY_DELAY_MS);
+ } else if (rc == OPAL_HARDWARE || rc == OPAL_INTERNAL_ERROR) {
+ if (retries--) {
+ msleep(10); /* Wait 10ms before retry */
+ rc = OPAL_BUSY; /* go around again */
+ }
+ }
}
if (rc != OPAL_SUCCESS)
@@ -87,21 +91,26 @@ static int opal_get_rtc_time(struct device *dev, struct rtc_time *tm)
static int opal_set_rtc_time(struct device *dev, struct rtc_time *tm)
{
- long rc = OPAL_BUSY;
+ s64 rc = OPAL_BUSY;
int retries = 10;
u32 y_m_d = 0;
u64 h_m_s_ms = 0;
tm_to_opal(tm, &y_m_d, &h_m_s_ms);
+
while (rc == OPAL_BUSY || rc == OPAL_BUSY_EVENT) {
rc = opal_rtc_write(y_m_d, h_m_s_ms);
- if (rc == OPAL_BUSY_EVENT)
+ if (rc == OPAL_BUSY_EVENT) {
+ msleep(OPAL_BUSY_DELAY_MS);
opal_poll_events(NULL);
- else if (retries-- && (rc == OPAL_HARDWARE
- || rc == OPAL_INTERNAL_ERROR))
- msleep(10);
- else if (rc != OPAL_BUSY && rc != OPAL_BUSY_EVENT)
- break;
+ } else if (rc == OPAL_BUSY) {
+ msleep(OPAL_BUSY_DELAY_MS);
+ } else if (rc == OPAL_HARDWARE || rc == OPAL_INTERNAL_ERROR) {
+ if (retries--) {
+ msleep(10); /* Wait 10ms before retry */
+ rc = OPAL_BUSY; /* go around again */
+ }
+ }
}
return rc == OPAL_SUCCESS ? 0 : -EIO;
diff --git a/drivers/rtc/rtc-pcf2127.c b/drivers/rtc/rtc-pcf2127.c
index 2bfdf63..8a3667e 100644
--- a/drivers/rtc/rtc-pcf2127.c
+++ b/drivers/rtc/rtc-pcf2127.c
@@ -237,6 +237,9 @@ static int pcf2127_i2c_gather_write(void *context,
memcpy(buf + 1, val, val_size);
ret = i2c_master_send(client, buf, val_size + 1);
+
+ kfree(buf);
+
if (ret != val_size + 1)
return ret < 0 ? ret : -EIO;
diff --git a/drivers/rtc/rtc-snvs.c b/drivers/rtc/rtc-snvs.c
index a753ef9..71eee395 100644
--- a/drivers/rtc/rtc-snvs.c
+++ b/drivers/rtc/rtc-snvs.c
@@ -47,49 +47,83 @@ struct snvs_rtc_data {
struct clk *clk;
};
+/* Read 64 bit timer register, which could be in inconsistent state */
+static u64 rtc_read_lpsrt(struct snvs_rtc_data *data)
+{
+ u32 msb, lsb;
+
+ regmap_read(data->regmap, data->offset + SNVS_LPSRTCMR, &msb);
+ regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &lsb);
+ return (u64)msb << 32 | lsb;
+}
+
+/* Read the secure real time counter, taking care to deal with the cases of the
+ * counter updating while being read.
+ */
static u32 rtc_read_lp_counter(struct snvs_rtc_data *data)
{
u64 read1, read2;
- u32 val;
+ unsigned int timeout = 100;
+ /* As expected, the registers might update between the read of the LSB
+ * reg and the MSB reg. It's also possible that one register might be
+ * in partially modified state as well.
+ */
+ read1 = rtc_read_lpsrt(data);
do {
- regmap_read(data->regmap, data->offset + SNVS_LPSRTCMR, &val);
- read1 = val;
- read1 <<= 32;
- regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &val);
- read1 |= val;
-
- regmap_read(data->regmap, data->offset + SNVS_LPSRTCMR, &val);
- read2 = val;
- read2 <<= 32;
- regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &val);
- read2 |= val;
- } while (read1 != read2);
+ read2 = read1;
+ read1 = rtc_read_lpsrt(data);
+ } while (read1 != read2 && --timeout);
+ if (!timeout)
+ dev_err(&data->rtc->dev, "Timeout trying to get valid LPSRT Counter read\n");
/* Convert 47-bit counter to 32-bit raw second count */
return (u32) (read1 >> CNTR_TO_SECS_SH);
}
-static void rtc_write_sync_lp(struct snvs_rtc_data *data)
+/* Just read the lsb from the counter, dealing with inconsistent state */
+static int rtc_read_lp_counter_lsb(struct snvs_rtc_data *data, u32 *lsb)
{
- u32 count1, count2, count3;
- int i;
+ u32 count1, count2;
+ unsigned int timeout = 100;
- /* Wait for 3 CKIL cycles */
- for (i = 0; i < 3; i++) {
- do {
- regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &count1);
- regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &count2);
- } while (count1 != count2);
-
- /* Now wait until counter value changes */
- do {
- do {
- regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &count2);
- regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &count3);
- } while (count2 != count3);
- } while (count3 == count1);
+ regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &count1);
+ do {
+ count2 = count1;
+ regmap_read(data->regmap, data->offset + SNVS_LPSRTCLR, &count1);
+ } while (count1 != count2 && --timeout);
+ if (!timeout) {
+ dev_err(&data->rtc->dev, "Timeout trying to get valid LPSRT Counter read\n");
+ return -ETIMEDOUT;
}
+
+ *lsb = count1;
+ return 0;
+}
+
+static int rtc_write_sync_lp(struct snvs_rtc_data *data)
+{
+ u32 count1, count2;
+ u32 elapsed;
+ unsigned int timeout = 1000;
+ int ret;
+
+ ret = rtc_read_lp_counter_lsb(data, &count1);
+ if (ret)
+ return ret;
+
+ /* Wait for 3 CKIL cycles, about 61.0-91.5 µs */
+ do {
+ ret = rtc_read_lp_counter_lsb(data, &count2);
+ if (ret)
+ return ret;
+ elapsed = count2 - count1; /* wrap around _is_ handled! */
+ } while (elapsed < 3 && --timeout);
+ if (!timeout) {
+ dev_err(&data->rtc->dev, "Timeout waiting for LPSRT Counter to change\n");
+ return -ETIMEDOUT;
+ }
+ return 0;
}
static int snvs_rtc_enable(struct snvs_rtc_data *data, bool enable)
@@ -132,20 +166,23 @@ static int snvs_rtc_set_time(struct device *dev, struct rtc_time *tm)
{
struct snvs_rtc_data *data = dev_get_drvdata(dev);
unsigned long time;
+ int ret;
rtc_tm_to_time(tm, &time);
/* Disable RTC first */
- snvs_rtc_enable(data, false);
+ ret = snvs_rtc_enable(data, false);
+ if (ret)
+ return ret;
/* Write 32-bit time to 47-bit timer, leaving 15 LSBs blank */
regmap_write(data->regmap, data->offset + SNVS_LPSRTCLR, time << CNTR_TO_SECS_SH);
regmap_write(data->regmap, data->offset + SNVS_LPSRTCMR, time >> (32 - CNTR_TO_SECS_SH));
/* Enable RTC again */
- snvs_rtc_enable(data, true);
+ ret = snvs_rtc_enable(data, true);
- return 0;
+ return ret;
}
static int snvs_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
@@ -170,9 +207,7 @@ static int snvs_rtc_alarm_irq_enable(struct device *dev, unsigned int enable)
(SNVS_LPCR_LPTA_EN | SNVS_LPCR_LPWUI_EN),
enable ? (SNVS_LPCR_LPTA_EN | SNVS_LPCR_LPWUI_EN) : 0);
- rtc_write_sync_lp(data);
-
- return 0;
+ return rtc_write_sync_lp(data);
}
static int snvs_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm)
@@ -180,10 +215,14 @@ static int snvs_rtc_set_alarm(struct device *dev, struct rtc_wkalrm *alrm)
struct snvs_rtc_data *data = dev_get_drvdata(dev);
struct rtc_time *alrm_tm = &alrm->time;
unsigned long time;
+ int ret;
rtc_tm_to_time(alrm_tm, &time);
regmap_update_bits(data->regmap, data->offset + SNVS_LPCR, SNVS_LPCR_LPTA_EN, 0);
+ ret = rtc_write_sync_lp(data);
+ if (ret)
+ return ret;
regmap_write(data->regmap, data->offset + SNVS_LPTAR, time);
/* Clear alarm interrupt status bit */
@@ -287,7 +326,11 @@ static int snvs_rtc_probe(struct platform_device *pdev)
regmap_write(data->regmap, data->offset + SNVS_LPSR, 0xffffffff);
/* Enable RTC */
- snvs_rtc_enable(data, true);
+ ret = snvs_rtc_enable(data, true);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to enable rtc %d\n", ret);
+ goto error_rtc_device_register;
+ }
device_init_wakeup(&pdev->dev, true);
diff --git a/drivers/rtc/rtc-tx4939.c b/drivers/rtc/rtc-tx4939.c
index 560d9a5..a952808 100644
--- a/drivers/rtc/rtc-tx4939.c
+++ b/drivers/rtc/rtc-tx4939.c
@@ -86,7 +86,8 @@ static int tx4939_rtc_read_time(struct device *dev, struct rtc_time *tm)
for (i = 2; i < 6; i++)
buf[i] = __raw_readl(&rtcreg->dat);
spin_unlock_irq(&pdata->lock);
- sec = (buf[5] << 24) | (buf[4] << 16) | (buf[3] << 8) | buf[2];
+ sec = ((unsigned long)buf[5] << 24) | (buf[4] << 16) |
+ (buf[3] << 8) | buf[2];
rtc_time_to_tm(sec, tm);
return rtc_valid_tm(tm);
}
@@ -147,7 +148,8 @@ static int tx4939_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
alrm->enabled = (ctl & TX4939_RTCCTL_ALME) ? 1 : 0;
alrm->pending = (ctl & TX4939_RTCCTL_ALMD) ? 1 : 0;
spin_unlock_irq(&pdata->lock);
- sec = (buf[5] << 24) | (buf[4] << 16) | (buf[3] << 8) | buf[2];
+ sec = ((unsigned long)buf[5] << 24) | (buf[4] << 16) |
+ (buf[3] << 8) | buf[2];
rtc_time_to_tm(sec, &alrm->time);
return rtc_valid_tm(&alrm->time);
}
diff --git a/drivers/s390/block/dasd_alias.c b/drivers/s390/block/dasd_alias.c
index 1e56018..e453d2a 100644
--- a/drivers/s390/block/dasd_alias.c
+++ b/drivers/s390/block/dasd_alias.c
@@ -591,13 +591,22 @@ static int _schedule_lcu_update(struct alias_lcu *lcu,
int dasd_alias_add_device(struct dasd_device *device)
{
struct dasd_eckd_private *private = device->private;
- struct alias_lcu *lcu;
+ __u8 uaddr = private->uid.real_unit_addr;
+ struct alias_lcu *lcu = private->lcu;
unsigned long flags;
int rc;
- lcu = private->lcu;
rc = 0;
spin_lock_irqsave(&lcu->lock, flags);
+ /*
+ * Check if device and lcu type differ. If so, the uac data may be
+ * outdated and needs to be updated.
+ */
+ if (private->uid.type != lcu->uac->unit[uaddr].ua_type) {
+ lcu->flags |= UPDATE_PENDING;
+ DBF_DEV_EVENT(DBF_WARNING, device, "%s",
+ "uid type mismatch - trigger rescan");
+ }
if (!(lcu->flags & UPDATE_PENDING)) {
rc = _add_device_to_lcu(lcu, device, device);
if (rc)
diff --git a/drivers/s390/block/dasd_eckd.c b/drivers/s390/block/dasd_eckd.c
index 0f5bc2f..be17de9 100644
--- a/drivers/s390/block/dasd_eckd.c
+++ b/drivers/s390/block/dasd_eckd.c
@@ -1834,6 +1834,9 @@ static void dasd_eckd_uncheck_device(struct dasd_device *device)
struct dasd_eckd_private *private = device->private;
int i;
+ if (!private)
+ return;
+
dasd_alias_disconnect_device_from_lcu(device);
private->ned = NULL;
private->sneq = NULL;
@@ -2085,8 +2088,11 @@ static int dasd_eckd_basic_to_ready(struct dasd_device *device)
static int dasd_eckd_online_to_ready(struct dasd_device *device)
{
- cancel_work_sync(&device->reload_device);
- cancel_work_sync(&device->kick_validate);
+ if (cancel_work_sync(&device->reload_device))
+ dasd_put_device(device);
+ if (cancel_work_sync(&device->kick_validate))
+ dasd_put_device(device);
+
return 0;
};
diff --git a/drivers/s390/char/Makefile b/drivers/s390/char/Makefile
index 41e28b2..8ac27ef 100644
--- a/drivers/s390/char/Makefile
+++ b/drivers/s390/char/Makefile
@@ -2,6 +2,8 @@
# S/390 character devices
#
+CFLAGS_REMOVE_sclp_early_core.o += $(CC_FLAGS_EXPOLINE)
+
obj-y += ctrlchar.o keyboard.o defkeymap.o sclp.o sclp_rw.o sclp_quiesce.o \
sclp_cmd.o sclp_config.o sclp_cpi_sys.o sclp_ocf.o sclp_ctl.o \
sclp_early.o
diff --git a/drivers/s390/cio/chsc.c b/drivers/s390/cio/chsc.c
index 1167469..67903c9 100644
--- a/drivers/s390/cio/chsc.c
+++ b/drivers/s390/cio/chsc.c
@@ -451,6 +451,7 @@ static void chsc_process_sei_link_incident(struct chsc_sei_nt0_area *sei_area)
static void chsc_process_sei_res_acc(struct chsc_sei_nt0_area *sei_area)
{
+ struct channel_path *chp;
struct chp_link link;
struct chp_id chpid;
int status;
@@ -463,10 +464,17 @@ static void chsc_process_sei_res_acc(struct chsc_sei_nt0_area *sei_area)
chpid.id = sei_area->rsid;
/* allocate a new channel path structure, if needed */
status = chp_get_status(chpid);
- if (status < 0)
- chp_new(chpid);
- else if (!status)
+ if (!status)
return;
+
+ if (status < 0) {
+ chp_new(chpid);
+ } else {
+ chp = chpid_to_chp(chpid);
+ mutex_lock(&chp->lock);
+ chp_update_desc(chp);
+ mutex_unlock(&chp->lock);
+ }
memset(&link, 0, sizeof(struct chp_link));
link.chpid = chpid;
if ((sei_area->vf & 0xc0) != 0) {
diff --git a/drivers/s390/cio/device_fsm.c b/drivers/s390/cio/device_fsm.c
index 8327d47..c46e31e 100644
--- a/drivers/s390/cio/device_fsm.c
+++ b/drivers/s390/cio/device_fsm.c
@@ -822,6 +822,7 @@ ccw_device_online_timeout(struct ccw_device *cdev, enum dev_event dev_event)
ccw_device_set_timeout(cdev, 0);
cdev->private->iretry = 255;
+ cdev->private->async_kill_io_rc = -ETIMEDOUT;
ret = ccw_device_cancel_halt_clear(cdev);
if (ret == -EBUSY) {
ccw_device_set_timeout(cdev, 3*HZ);
@@ -898,7 +899,7 @@ ccw_device_killing_irq(struct ccw_device *cdev, enum dev_event dev_event)
/* OK, i/o is dead now. Call interrupt handler. */
if (cdev->handler)
cdev->handler(cdev, cdev->private->intparm,
- ERR_PTR(-EIO));
+ ERR_PTR(cdev->private->async_kill_io_rc));
}
static void
@@ -915,14 +916,16 @@ ccw_device_killing_timeout(struct ccw_device *cdev, enum dev_event dev_event)
ccw_device_online_verify(cdev, 0);
if (cdev->handler)
cdev->handler(cdev, cdev->private->intparm,
- ERR_PTR(-EIO));
+ ERR_PTR(cdev->private->async_kill_io_rc));
}
void ccw_device_kill_io(struct ccw_device *cdev)
{
int ret;
+ ccw_device_set_timeout(cdev, 0);
cdev->private->iretry = 255;
+ cdev->private->async_kill_io_rc = -EIO;
ret = ccw_device_cancel_halt_clear(cdev);
if (ret == -EBUSY) {
ccw_device_set_timeout(cdev, 3*HZ);
diff --git a/drivers/s390/cio/device_ops.c b/drivers/s390/cio/device_ops.c
index 877d9f6..85b2896 100644
--- a/drivers/s390/cio/device_ops.c
+++ b/drivers/s390/cio/device_ops.c
@@ -158,7 +158,7 @@ int ccw_device_clear(struct ccw_device *cdev, unsigned long intparm)
}
/**
- * ccw_device_start_key() - start a s390 channel program with key
+ * ccw_device_start_timeout_key() - start a s390 channel program with timeout and key
* @cdev: target ccw device
* @cpa: logical start address of channel program
* @intparm: user specific interruption parameter; will be presented back to
@@ -169,10 +169,15 @@ int ccw_device_clear(struct ccw_device *cdev, unsigned long intparm)
* @key: storage key to be used for the I/O
* @flags: additional flags; defines the action to be performed for I/O
* processing.
+ * @expires: timeout value in jiffies
*
* Start a S/390 channel program. When the interrupt arrives, the
* IRQ handler is called, either immediately, delayed (dev-end missing,
* or sense required) or never (no IRQ handler registered).
+ * This function notifies the device driver if the channel program has not
+ * completed during the time specified by @expires. If a timeout occurs, the
+ * channel program is terminated via xsch, hsch or csch, and the device's
+ * interrupt handler will be called with an irb containing ERR_PTR(-%ETIMEDOUT).
* Returns:
* %0, if the operation was successful;
* -%EBUSY, if the device is busy, or status pending;
@@ -181,9 +186,9 @@ int ccw_device_clear(struct ccw_device *cdev, unsigned long intparm)
* Context:
* Interrupts disabled, ccw device lock held
*/
-int ccw_device_start_key(struct ccw_device *cdev, struct ccw1 *cpa,
- unsigned long intparm, __u8 lpm, __u8 key,
- unsigned long flags)
+int ccw_device_start_timeout_key(struct ccw_device *cdev, struct ccw1 *cpa,
+ unsigned long intparm, __u8 lpm, __u8 key,
+ unsigned long flags, int expires)
{
struct subchannel *sch;
int ret;
@@ -223,6 +228,8 @@ int ccw_device_start_key(struct ccw_device *cdev, struct ccw1 *cpa,
switch (ret) {
case 0:
cdev->private->intparm = intparm;
+ if (expires)
+ ccw_device_set_timeout(cdev, expires);
break;
case -EACCES:
case -ENODEV:
@@ -233,7 +240,7 @@ int ccw_device_start_key(struct ccw_device *cdev, struct ccw1 *cpa,
}
/**
- * ccw_device_start_timeout_key() - start a s390 channel program with timeout and key
+ * ccw_device_start_key() - start a s390 channel program with key
* @cdev: target ccw device
* @cpa: logical start address of channel program
* @intparm: user specific interruption parameter; will be presented back to
@@ -244,15 +251,10 @@ int ccw_device_start_key(struct ccw_device *cdev, struct ccw1 *cpa,
* @key: storage key to be used for the I/O
* @flags: additional flags; defines the action to be performed for I/O
* processing.
- * @expires: timeout value in jiffies
*
* Start a S/390 channel program. When the interrupt arrives, the
* IRQ handler is called, either immediately, delayed (dev-end missing,
* or sense required) or never (no IRQ handler registered).
- * This function notifies the device driver if the channel program has not
- * completed during the time specified by @expires. If a timeout occurs, the
- * channel program is terminated via xsch, hsch or csch, and the device's
- * interrupt handler will be called with an irb containing ERR_PTR(-%ETIMEDOUT).
* Returns:
* %0, if the operation was successful;
* -%EBUSY, if the device is busy, or status pending;
@@ -261,19 +263,12 @@ int ccw_device_start_key(struct ccw_device *cdev, struct ccw1 *cpa,
* Context:
* Interrupts disabled, ccw device lock held
*/
-int ccw_device_start_timeout_key(struct ccw_device *cdev, struct ccw1 *cpa,
- unsigned long intparm, __u8 lpm, __u8 key,
- unsigned long flags, int expires)
+int ccw_device_start_key(struct ccw_device *cdev, struct ccw1 *cpa,
+ unsigned long intparm, __u8 lpm, __u8 key,
+ unsigned long flags)
{
- int ret;
-
- if (!cdev)
- return -ENODEV;
- ccw_device_set_timeout(cdev, expires);
- ret = ccw_device_start_key(cdev, cpa, intparm, lpm, key, flags);
- if (ret != 0)
- ccw_device_set_timeout(cdev, 0);
- return ret;
+ return ccw_device_start_timeout_key(cdev, cpa, intparm, lpm, key,
+ flags, 0);
}
/**
@@ -488,18 +483,20 @@ void ccw_device_get_id(struct ccw_device *cdev, struct ccw_dev_id *dev_id)
EXPORT_SYMBOL(ccw_device_get_id);
/**
- * ccw_device_tm_start_key() - perform start function
+ * ccw_device_tm_start_timeout_key() - perform start function
* @cdev: ccw device on which to perform the start function
* @tcw: transport-command word to be started
* @intparm: user defined parameter to be passed to the interrupt handler
* @lpm: mask of paths to use
* @key: storage key to use for storage access
+ * @expires: time span in jiffies after which to abort request
*
* Start the tcw on the given ccw device. Return zero on success, non-zero
* otherwise.
*/
-int ccw_device_tm_start_key(struct ccw_device *cdev, struct tcw *tcw,
- unsigned long intparm, u8 lpm, u8 key)
+int ccw_device_tm_start_timeout_key(struct ccw_device *cdev, struct tcw *tcw,
+ unsigned long intparm, u8 lpm, u8 key,
+ int expires)
{
struct subchannel *sch;
int rc;
@@ -526,37 +523,32 @@ int ccw_device_tm_start_key(struct ccw_device *cdev, struct tcw *tcw,
return -EACCES;
}
rc = cio_tm_start_key(sch, tcw, lpm, key);
- if (rc == 0)
+ if (rc == 0) {
cdev->private->intparm = intparm;
+ if (expires)
+ ccw_device_set_timeout(cdev, expires);
+ }
return rc;
}
-EXPORT_SYMBOL(ccw_device_tm_start_key);
+EXPORT_SYMBOL(ccw_device_tm_start_timeout_key);
/**
- * ccw_device_tm_start_timeout_key() - perform start function
+ * ccw_device_tm_start_key() - perform start function
* @cdev: ccw device on which to perform the start function
* @tcw: transport-command word to be started
* @intparm: user defined parameter to be passed to the interrupt handler
* @lpm: mask of paths to use
* @key: storage key to use for storage access
- * @expires: time span in jiffies after which to abort request
*
* Start the tcw on the given ccw device. Return zero on success, non-zero
* otherwise.
*/
-int ccw_device_tm_start_timeout_key(struct ccw_device *cdev, struct tcw *tcw,
- unsigned long intparm, u8 lpm, u8 key,
- int expires)
+int ccw_device_tm_start_key(struct ccw_device *cdev, struct tcw *tcw,
+ unsigned long intparm, u8 lpm, u8 key)
{
- int ret;
-
- ccw_device_set_timeout(cdev, expires);
- ret = ccw_device_tm_start_key(cdev, tcw, intparm, lpm, key);
- if (ret != 0)
- ccw_device_set_timeout(cdev, 0);
- return ret;
+ return ccw_device_tm_start_timeout_key(cdev, tcw, intparm, lpm, key, 0);
}
-EXPORT_SYMBOL(ccw_device_tm_start_timeout_key);
+EXPORT_SYMBOL(ccw_device_tm_start_key);
/**
* ccw_device_tm_start() - perform start function
diff --git a/drivers/s390/cio/io_sch.h b/drivers/s390/cio/io_sch.h
index 220f491..1d98434 100644
--- a/drivers/s390/cio/io_sch.h
+++ b/drivers/s390/cio/io_sch.h
@@ -154,6 +154,7 @@ struct ccw_device_private {
unsigned long intparm; /* user interruption parameter */
struct qdio_irq *qdio_data;
struct irb irb; /* device status */
+ int async_kill_io_rc;
struct senseid senseid; /* SenseID info */
struct pgid pgid[8]; /* path group IDs per chpid*/
struct ccw1 iccws[2]; /* ccws for SNID/SID/SPGID commands */
diff --git a/drivers/s390/cio/qdio_main.c b/drivers/s390/cio/qdio_main.c
index 66e9bb0..18ab84e 100644
--- a/drivers/s390/cio/qdio_main.c
+++ b/drivers/s390/cio/qdio_main.c
@@ -640,21 +640,20 @@ static inline unsigned long qdio_aob_for_buffer(struct qdio_output_q *q,
unsigned long phys_aob = 0;
if (!q->use_cq)
- goto out;
+ return 0;
if (!q->aobs[bufnr]) {
struct qaob *aob = qdio_allocate_aob();
q->aobs[bufnr] = aob;
}
if (q->aobs[bufnr]) {
- q->sbal_state[bufnr].flags = QDIO_OUTBUF_STATE_FLAG_NONE;
q->sbal_state[bufnr].aob = q->aobs[bufnr];
q->aobs[bufnr]->user1 = (u64) q->sbal_state[bufnr].user;
phys_aob = virt_to_phys(q->aobs[bufnr]);
WARN_ON_ONCE(phys_aob & 0xFF);
}
-out:
+ q->sbal_state[bufnr].flags = 0;
return phys_aob;
}
diff --git a/drivers/s390/cio/qdio_setup.c b/drivers/s390/cio/qdio_setup.c
index 48b3866..3528690 100644
--- a/drivers/s390/cio/qdio_setup.c
+++ b/drivers/s390/cio/qdio_setup.c
@@ -140,7 +140,7 @@ static int __qdio_allocate_qs(struct qdio_q **irq_ptr_qs, int nr_queues)
int i;
for (i = 0; i < nr_queues; i++) {
- q = kmem_cache_alloc(qdio_q_cache, GFP_KERNEL);
+ q = kmem_cache_zalloc(qdio_q_cache, GFP_KERNEL);
if (!q)
return -ENOMEM;
@@ -456,7 +456,6 @@ int qdio_setup_irq(struct qdio_initialize *init_data)
{
struct ciw *ciw;
struct qdio_irq *irq_ptr = init_data->cdev->private->qdio_data;
- int rc;
memset(&irq_ptr->qib, 0, sizeof(irq_ptr->qib));
memset(&irq_ptr->siga_flag, 0, sizeof(irq_ptr->siga_flag));
@@ -493,16 +492,14 @@ int qdio_setup_irq(struct qdio_initialize *init_data)
ciw = ccw_device_get_ciw(init_data->cdev, CIW_TYPE_EQUEUE);
if (!ciw) {
DBF_ERROR("%4x NO EQ", irq_ptr->schid.sch_no);
- rc = -EINVAL;
- goto out_err;
+ return -EINVAL;
}
irq_ptr->equeue = *ciw;
ciw = ccw_device_get_ciw(init_data->cdev, CIW_TYPE_AQUEUE);
if (!ciw) {
DBF_ERROR("%4x NO AQ", irq_ptr->schid.sch_no);
- rc = -EINVAL;
- goto out_err;
+ return -EINVAL;
}
irq_ptr->aqueue = *ciw;
@@ -510,9 +507,6 @@ int qdio_setup_irq(struct qdio_initialize *init_data)
irq_ptr->orig_handler = init_data->cdev->handler;
init_data->cdev->handler = qdio_int_handler;
return 0;
-out_err:
- qdio_release_memory(irq_ptr);
- return rc;
}
void qdio_print_subchannel_info(struct qdio_irq *irq_ptr,
diff --git a/drivers/s390/net/qeth_core_main.c b/drivers/s390/net/qeth_core_main.c
index 283416a..8f77fc0 100644
--- a/drivers/s390/net/qeth_core_main.c
+++ b/drivers/s390/net/qeth_core_main.c
@@ -23,6 +23,7 @@
#include <linux/netdevice.h>
#include <linux/netdev_features.h>
#include <linux/skbuff.h>
+#include <linux/vmalloc.h>
#include <net/iucv/af_iucv.h>
#include <net/dsfield.h>
@@ -3499,13 +3500,14 @@ static void qeth_flush_buffers(struct qeth_qdio_out_q *queue, int index,
qdio_flags = QDIO_FLAG_SYNC_OUTPUT;
if (atomic_read(&queue->set_pci_flags_count))
qdio_flags |= QDIO_FLAG_PCI_OUT;
+ atomic_add(count, &queue->used_buffers);
+
rc = do_QDIO(CARD_DDEV(queue->card), qdio_flags,
queue->queue_no, index, count);
if (queue->card->options.performance_stats)
queue->card->perf_stats.outbound_do_qdio_time +=
qeth_get_micros() -
queue->card->perf_stats.outbound_do_qdio_start_time;
- atomic_add(count, &queue->used_buffers);
if (rc) {
queue->card->stats.tx_errors += count;
/* ignore temporary SIGA errors without busy condition */
@@ -4538,8 +4540,8 @@ static int qeth_snmp_command_cb(struct qeth_card *card,
{
struct qeth_ipa_cmd *cmd;
struct qeth_arp_query_info *qinfo;
- struct qeth_snmp_cmd *snmp;
unsigned char *data;
+ void *snmp_data;
__u16 data_len;
QETH_CARD_TEXT(card, 3, "snpcmdcb");
@@ -4547,7 +4549,6 @@ static int qeth_snmp_command_cb(struct qeth_card *card,
cmd = (struct qeth_ipa_cmd *) sdata;
data = (unsigned char *)((char *)cmd - reply->offset);
qinfo = (struct qeth_arp_query_info *) reply->param;
- snmp = &cmd->data.setadapterparms.data.snmp;
if (cmd->hdr.return_code) {
QETH_CARD_TEXT_(card, 4, "scer1%x", cmd->hdr.return_code);
@@ -4560,10 +4561,15 @@ static int qeth_snmp_command_cb(struct qeth_card *card,
return 0;
}
data_len = *((__u16 *)QETH_IPA_PDU_LEN_PDU1(data));
- if (cmd->data.setadapterparms.hdr.seq_no == 1)
- data_len -= (__u16)((char *)&snmp->data - (char *)cmd);
- else
- data_len -= (__u16)((char *)&snmp->request - (char *)cmd);
+ if (cmd->data.setadapterparms.hdr.seq_no == 1) {
+ snmp_data = &cmd->data.setadapterparms.data.snmp;
+ data_len -= offsetof(struct qeth_ipa_cmd,
+ data.setadapterparms.data.snmp);
+ } else {
+ snmp_data = &cmd->data.setadapterparms.data.snmp.request;
+ data_len -= offsetof(struct qeth_ipa_cmd,
+ data.setadapterparms.data.snmp.request);
+ }
/* check if there is enough room in userspace */
if ((qinfo->udata_len - qinfo->udata_offset) < data_len) {
@@ -4576,16 +4582,9 @@ static int qeth_snmp_command_cb(struct qeth_card *card,
QETH_CARD_TEXT_(card, 4, "sseqn%i",
cmd->data.setadapterparms.hdr.seq_no);
/*copy entries to user buffer*/
- if (cmd->data.setadapterparms.hdr.seq_no == 1) {
- memcpy(qinfo->udata + qinfo->udata_offset,
- (char *)snmp,
- data_len + offsetof(struct qeth_snmp_cmd, data));
- qinfo->udata_offset += offsetof(struct qeth_snmp_cmd, data);
- } else {
- memcpy(qinfo->udata + qinfo->udata_offset,
- (char *)&snmp->request, data_len);
- }
+ memcpy(qinfo->udata + qinfo->udata_offset, snmp_data, data_len);
qinfo->udata_offset += data_len;
+
/* check if all replies received ... */
QETH_CARD_TEXT_(card, 4, "srtot%i",
cmd->data.setadapterparms.hdr.used_total);
@@ -4714,7 +4713,7 @@ int qeth_query_oat_command(struct qeth_card *card, char __user *udata)
priv.buffer_len = oat_data.buffer_len;
priv.response_len = 0;
- priv.buffer = kzalloc(oat_data.buffer_len, GFP_KERNEL);
+ priv.buffer = vzalloc(oat_data.buffer_len);
if (!priv.buffer) {
rc = -ENOMEM;
goto out;
@@ -4755,7 +4754,7 @@ int qeth_query_oat_command(struct qeth_card *card, char __user *udata)
rc = -EFAULT;
out_free:
- kfree(priv.buffer);
+ vfree(priv.buffer);
out:
return rc;
}
diff --git a/drivers/s390/net/qeth_core_sys.c b/drivers/s390/net/qeth_core_sys.c
index db6a285..0a7a6da 100644
--- a/drivers/s390/net/qeth_core_sys.c
+++ b/drivers/s390/net/qeth_core_sys.c
@@ -423,6 +423,7 @@ static ssize_t qeth_dev_layer2_store(struct device *dev,
if (card->discipline) {
card->discipline->remove(card->gdev);
qeth_core_free_discipline(card);
+ card->options.layer2 = -1;
}
rc = qeth_core_load_discipline(card, newdis);
diff --git a/drivers/s390/net/qeth_l2_main.c b/drivers/s390/net/qeth_l2_main.c
index e94e957..58404e6 100644
--- a/drivers/s390/net/qeth_l2_main.c
+++ b/drivers/s390/net/qeth_l2_main.c
@@ -491,7 +491,7 @@ static int qeth_l2_process_inbound_buffer(struct qeth_card *card,
default:
dev_kfree_skb_any(skb);
QETH_CARD_TEXT(card, 3, "inbunkno");
- QETH_DBF_HEX(CTRL, 3, hdr, QETH_DBF_CTRL_LEN);
+ QETH_DBF_HEX(CTRL, 3, hdr, sizeof(*hdr));
continue;
}
work_done++;
diff --git a/drivers/s390/net/qeth_l3_main.c b/drivers/s390/net/qeth_l3_main.c
index 4ca161b..6e6ba1b 100644
--- a/drivers/s390/net/qeth_l3_main.c
+++ b/drivers/s390/net/qeth_l3_main.c
@@ -363,9 +363,6 @@ static void qeth_l3_clear_ip_htable(struct qeth_card *card, int recover)
QETH_CARD_TEXT(card, 4, "clearip");
- if (recover && card->options.sniffer)
- return;
-
spin_lock_bh(&card->ip_lock);
hash_for_each_safe(card->ip_htable, i, tmp, addr, hnode) {
@@ -823,6 +820,8 @@ static int qeth_l3_register_addr_entry(struct qeth_card *card,
int rc = 0;
int cnt = 3;
+ if (card->options.sniffer)
+ return 0;
if (addr->proto == QETH_PROT_IPV4) {
QETH_CARD_TEXT(card, 2, "setaddr4");
@@ -858,6 +857,9 @@ static int qeth_l3_deregister_addr_entry(struct qeth_card *card,
{
int rc = 0;
+ if (card->options.sniffer)
+ return 0;
+
if (addr->proto == QETH_PROT_IPV4) {
QETH_CARD_TEXT(card, 2, "deladdr4");
QETH_CARD_HEX(card, 3, &addr->u.a4.addr, sizeof(int));
@@ -1836,7 +1838,7 @@ static int qeth_l3_process_inbound_buffer(struct qeth_card *card,
default:
dev_kfree_skb_any(skb);
QETH_CARD_TEXT(card, 3, "inbunkno");
- QETH_DBF_HEX(CTRL, 3, hdr, QETH_DBF_CTRL_LEN);
+ QETH_DBF_HEX(CTRL, 3, hdr, sizeof(*hdr));
continue;
}
work_done++;
diff --git a/drivers/s390/scsi/zfcp_aux.c b/drivers/s390/scsi/zfcp_aux.c
index b3f9243..36eb298 100644
--- a/drivers/s390/scsi/zfcp_aux.c
+++ b/drivers/s390/scsi/zfcp_aux.c
@@ -275,16 +275,16 @@ static void zfcp_free_low_mem_buffers(struct zfcp_adapter *adapter)
*/
int zfcp_status_read_refill(struct zfcp_adapter *adapter)
{
- while (atomic_read(&adapter->stat_miss) > 0)
+ while (atomic_add_unless(&adapter->stat_miss, -1, 0))
if (zfcp_fsf_status_read(adapter->qdio)) {
+ atomic_inc(&adapter->stat_miss); /* undo add -1 */
if (atomic_read(&adapter->stat_miss) >=
adapter->stat_read_buf_num) {
zfcp_erp_adapter_reopen(adapter, 0, "axsref1");
return 1;
}
break;
- } else
- atomic_dec(&adapter->stat_miss);
+ }
return 0;
}
diff --git a/drivers/s390/scsi/zfcp_dbf.c b/drivers/s390/scsi/zfcp_dbf.c
index 34367d1..b6caad0 100644
--- a/drivers/s390/scsi/zfcp_dbf.c
+++ b/drivers/s390/scsi/zfcp_dbf.c
@@ -3,7 +3,7 @@
*
* Debug traces for zfcp.
*
- * Copyright IBM Corp. 2002, 2017
+ * Copyright IBM Corp. 2002, 2018
*/
#define KMSG_COMPONENT "zfcp"
@@ -287,6 +287,27 @@ void zfcp_dbf_rec_trig(char *tag, struct zfcp_adapter *adapter,
spin_unlock_irqrestore(&dbf->rec_lock, flags);
}
+/**
+ * zfcp_dbf_rec_trig_lock - trace event related to triggered recovery with lock
+ * @tag: identifier for event
+ * @adapter: adapter on which the erp_action should run
+ * @port: remote port involved in the erp_action
+ * @sdev: scsi device involved in the erp_action
+ * @want: wanted erp_action
+ * @need: required erp_action
+ *
+ * The adapter->erp_lock must not be held.
+ */
+void zfcp_dbf_rec_trig_lock(char *tag, struct zfcp_adapter *adapter,
+ struct zfcp_port *port, struct scsi_device *sdev,
+ u8 want, u8 need)
+{
+ unsigned long flags;
+
+ read_lock_irqsave(&adapter->erp_lock, flags);
+ zfcp_dbf_rec_trig(tag, adapter, port, sdev, want, need);
+ read_unlock_irqrestore(&adapter->erp_lock, flags);
+}
/**
* zfcp_dbf_rec_run_lvl - trace event related to running recovery
@@ -604,6 +625,46 @@ void zfcp_dbf_scsi(char *tag, int level, struct scsi_cmnd *sc,
spin_unlock_irqrestore(&dbf->scsi_lock, flags);
}
+/**
+ * zfcp_dbf_scsi_eh() - Trace event for special cases of scsi_eh callbacks.
+ * @tag: Identifier for event.
+ * @adapter: Pointer to zfcp adapter as context for this event.
+ * @scsi_id: SCSI ID/target to indicate scope of task management function (TMF).
+ * @ret: Return value of calling function.
+ *
+ * This SCSI trace variant does not depend on any of:
+ * scsi_cmnd, zfcp_fsf_req, scsi_device.
+ */
+void zfcp_dbf_scsi_eh(char *tag, struct zfcp_adapter *adapter,
+ unsigned int scsi_id, int ret)
+{
+ struct zfcp_dbf *dbf = adapter->dbf;
+ struct zfcp_dbf_scsi *rec = &dbf->scsi_buf;
+ unsigned long flags;
+ static int const level = 1;
+
+ if (unlikely(!debug_level_enabled(adapter->dbf->scsi, level)))
+ return;
+
+ spin_lock_irqsave(&dbf->scsi_lock, flags);
+ memset(rec, 0, sizeof(*rec));
+
+ memcpy(rec->tag, tag, ZFCP_DBF_TAG_LEN);
+ rec->id = ZFCP_DBF_SCSI_CMND;
+ rec->scsi_result = ret; /* re-use field, int is 4 bytes and fits */
+ rec->scsi_retries = ~0;
+ rec->scsi_allowed = ~0;
+ rec->fcp_rsp_info = ~0;
+ rec->scsi_id = scsi_id;
+ rec->scsi_lun = (u32)ZFCP_DBF_INVALID_LUN;
+ rec->scsi_lun_64_hi = (u32)(ZFCP_DBF_INVALID_LUN >> 32);
+ rec->host_scribble = ~0;
+ memset(rec->scsi_opcode, 0xff, ZFCP_DBF_SCSI_OPCODE);
+
+ debug_event(dbf->scsi, level, rec, sizeof(*rec));
+ spin_unlock_irqrestore(&dbf->scsi_lock, flags);
+}
+
static debug_info_t *zfcp_dbf_reg(const char *name, int size, int rec_size)
{
struct debug_info *d;
diff --git a/drivers/s390/scsi/zfcp_erp.c b/drivers/s390/scsi/zfcp_erp.c
index 3b23d675..2abcd33 100644
--- a/drivers/s390/scsi/zfcp_erp.c
+++ b/drivers/s390/scsi/zfcp_erp.c
@@ -34,11 +34,28 @@ enum zfcp_erp_steps {
ZFCP_ERP_STEP_LUN_OPENING = 0x2000,
};
+/**
+ * enum zfcp_erp_act_type - Type of ERP action object.
+ * @ZFCP_ERP_ACTION_REOPEN_LUN: LUN recovery.
+ * @ZFCP_ERP_ACTION_REOPEN_PORT: Port recovery.
+ * @ZFCP_ERP_ACTION_REOPEN_PORT_FORCED: Forced port recovery.
+ * @ZFCP_ERP_ACTION_REOPEN_ADAPTER: Adapter recovery.
+ * @ZFCP_ERP_ACTION_NONE: Eyecatcher pseudo flag to bitwise or-combine with
+ * either of the first four enum values.
+ * Used to indicate that an ERP action could not be
+ * set up despite a detected need for some recovery.
+ * @ZFCP_ERP_ACTION_FAILED: Eyecatcher pseudo flag to bitwise or-combine with
+ * either of the first four enum values.
+ * Used to indicate that ERP not needed because
+ * the object has ZFCP_STATUS_COMMON_ERP_FAILED.
+ */
enum zfcp_erp_act_type {
ZFCP_ERP_ACTION_REOPEN_LUN = 1,
ZFCP_ERP_ACTION_REOPEN_PORT = 2,
ZFCP_ERP_ACTION_REOPEN_PORT_FORCED = 3,
ZFCP_ERP_ACTION_REOPEN_ADAPTER = 4,
+ ZFCP_ERP_ACTION_NONE = 0xc0,
+ ZFCP_ERP_ACTION_FAILED = 0xe0,
};
enum zfcp_erp_act_state {
@@ -125,6 +142,49 @@ static void zfcp_erp_action_dismiss_adapter(struct zfcp_adapter *adapter)
}
}
+static int zfcp_erp_handle_failed(int want, struct zfcp_adapter *adapter,
+ struct zfcp_port *port,
+ struct scsi_device *sdev)
+{
+ int need = want;
+ struct zfcp_scsi_dev *zsdev;
+
+ switch (want) {
+ case ZFCP_ERP_ACTION_REOPEN_LUN:
+ zsdev = sdev_to_zfcp(sdev);
+ if (atomic_read(&zsdev->status) & ZFCP_STATUS_COMMON_ERP_FAILED)
+ need = 0;
+ break;
+ case ZFCP_ERP_ACTION_REOPEN_PORT_FORCED:
+ if (atomic_read(&port->status) & ZFCP_STATUS_COMMON_ERP_FAILED)
+ need = 0;
+ break;
+ case ZFCP_ERP_ACTION_REOPEN_PORT:
+ if (atomic_read(&port->status) &
+ ZFCP_STATUS_COMMON_ERP_FAILED) {
+ need = 0;
+ /* ensure propagation of failed status to new devices */
+ zfcp_erp_set_port_status(
+ port, ZFCP_STATUS_COMMON_ERP_FAILED);
+ }
+ break;
+ case ZFCP_ERP_ACTION_REOPEN_ADAPTER:
+ if (atomic_read(&adapter->status) &
+ ZFCP_STATUS_COMMON_ERP_FAILED) {
+ need = 0;
+ /* ensure propagation of failed status to new devices */
+ zfcp_erp_set_adapter_status(
+ adapter, ZFCP_STATUS_COMMON_ERP_FAILED);
+ }
+ break;
+ default:
+ need = 0;
+ break;
+ }
+
+ return need;
+}
+
static int zfcp_erp_required_act(int want, struct zfcp_adapter *adapter,
struct zfcp_port *port,
struct scsi_device *sdev)
@@ -248,16 +308,27 @@ static int zfcp_erp_action_enqueue(int want, struct zfcp_adapter *adapter,
int retval = 1, need;
struct zfcp_erp_action *act;
- if (!adapter->erp_thread)
- return -EIO;
+ need = zfcp_erp_handle_failed(want, adapter, port, sdev);
+ if (!need) {
+ need = ZFCP_ERP_ACTION_FAILED; /* marker for trace */
+ goto out;
+ }
+
+ if (!adapter->erp_thread) {
+ need = ZFCP_ERP_ACTION_NONE; /* marker for trace */
+ retval = -EIO;
+ goto out;
+ }
need = zfcp_erp_required_act(want, adapter, port, sdev);
if (!need)
goto out;
act = zfcp_erp_setup_act(need, act_status, adapter, port, sdev);
- if (!act)
+ if (!act) {
+ need |= ZFCP_ERP_ACTION_NONE; /* marker for trace */
goto out;
+ }
atomic_or(ZFCP_STATUS_ADAPTER_ERP_PENDING, &adapter->status);
++adapter->erp_total_count;
list_add_tail(&act->list, &adapter->erp_ready_head);
@@ -268,18 +339,32 @@ static int zfcp_erp_action_enqueue(int want, struct zfcp_adapter *adapter,
return retval;
}
+void zfcp_erp_port_forced_no_port_dbf(char *id, struct zfcp_adapter *adapter,
+ u64 port_name, u32 port_id)
+{
+ unsigned long flags;
+ static /* don't waste stack */ struct zfcp_port tmpport;
+
+ write_lock_irqsave(&adapter->erp_lock, flags);
+ /* Stand-in zfcp port with fields just good enough for
+ * zfcp_dbf_rec_trig() and zfcp_dbf_set_common().
+ * Under lock because tmpport is static.
+ */
+ atomic_set(&tmpport.status, -1); /* unknown */
+ tmpport.wwpn = port_name;
+ tmpport.d_id = port_id;
+ zfcp_dbf_rec_trig(id, adapter, &tmpport, NULL,
+ ZFCP_ERP_ACTION_REOPEN_PORT_FORCED,
+ ZFCP_ERP_ACTION_NONE);
+ write_unlock_irqrestore(&adapter->erp_lock, flags);
+}
+
static int _zfcp_erp_adapter_reopen(struct zfcp_adapter *adapter,
int clear_mask, char *id)
{
zfcp_erp_adapter_block(adapter, clear_mask);
zfcp_scsi_schedule_rports_block(adapter);
- /* ensure propagation of failed status to new devices */
- if (atomic_read(&adapter->status) & ZFCP_STATUS_COMMON_ERP_FAILED) {
- zfcp_erp_set_adapter_status(adapter,
- ZFCP_STATUS_COMMON_ERP_FAILED);
- return -EIO;
- }
return zfcp_erp_action_enqueue(ZFCP_ERP_ACTION_REOPEN_ADAPTER,
adapter, NULL, NULL, id, 0);
}
@@ -298,12 +383,8 @@ void zfcp_erp_adapter_reopen(struct zfcp_adapter *adapter, int clear, char *id)
zfcp_scsi_schedule_rports_block(adapter);
write_lock_irqsave(&adapter->erp_lock, flags);
- if (atomic_read(&adapter->status) & ZFCP_STATUS_COMMON_ERP_FAILED)
- zfcp_erp_set_adapter_status(adapter,
- ZFCP_STATUS_COMMON_ERP_FAILED);
- else
- zfcp_erp_action_enqueue(ZFCP_ERP_ACTION_REOPEN_ADAPTER, adapter,
- NULL, NULL, id, 0);
+ zfcp_erp_action_enqueue(ZFCP_ERP_ACTION_REOPEN_ADAPTER, adapter,
+ NULL, NULL, id, 0);
write_unlock_irqrestore(&adapter->erp_lock, flags);
}
@@ -344,9 +425,6 @@ static void _zfcp_erp_port_forced_reopen(struct zfcp_port *port, int clear,
zfcp_erp_port_block(port, clear);
zfcp_scsi_schedule_rport_block(port);
- if (atomic_read(&port->status) & ZFCP_STATUS_COMMON_ERP_FAILED)
- return;
-
zfcp_erp_action_enqueue(ZFCP_ERP_ACTION_REOPEN_PORT_FORCED,
port->adapter, port, NULL, id, 0);
}
@@ -372,12 +450,6 @@ static int _zfcp_erp_port_reopen(struct zfcp_port *port, int clear, char *id)
zfcp_erp_port_block(port, clear);
zfcp_scsi_schedule_rport_block(port);
- if (atomic_read(&port->status) & ZFCP_STATUS_COMMON_ERP_FAILED) {
- /* ensure propagation of failed status to new devices */
- zfcp_erp_set_port_status(port, ZFCP_STATUS_COMMON_ERP_FAILED);
- return -EIO;
- }
-
return zfcp_erp_action_enqueue(ZFCP_ERP_ACTION_REOPEN_PORT,
port->adapter, port, NULL, id, 0);
}
@@ -417,9 +489,6 @@ static void _zfcp_erp_lun_reopen(struct scsi_device *sdev, int clear, char *id,
zfcp_erp_lun_block(sdev, clear);
- if (atomic_read(&zfcp_sdev->status) & ZFCP_STATUS_COMMON_ERP_FAILED)
- return;
-
zfcp_erp_action_enqueue(ZFCP_ERP_ACTION_REOPEN_LUN, adapter,
zfcp_sdev->port, sdev, id, act_status);
}
diff --git a/drivers/s390/scsi/zfcp_ext.h b/drivers/s390/scsi/zfcp_ext.h
index 21c8c68..b326f05 100644
--- a/drivers/s390/scsi/zfcp_ext.h
+++ b/drivers/s390/scsi/zfcp_ext.h
@@ -3,7 +3,7 @@
*
* External function declarations.
*
- * Copyright IBM Corp. 2002, 2016
+ * Copyright IBM Corp. 2002, 2018
*/
#ifndef ZFCP_EXT_H
@@ -34,6 +34,9 @@ extern int zfcp_dbf_adapter_register(struct zfcp_adapter *);
extern void zfcp_dbf_adapter_unregister(struct zfcp_adapter *);
extern void zfcp_dbf_rec_trig(char *, struct zfcp_adapter *,
struct zfcp_port *, struct scsi_device *, u8, u8);
+extern void zfcp_dbf_rec_trig_lock(char *tag, struct zfcp_adapter *adapter,
+ struct zfcp_port *port,
+ struct scsi_device *sdev, u8 want, u8 need);
extern void zfcp_dbf_rec_run(char *, struct zfcp_erp_action *);
extern void zfcp_dbf_rec_run_lvl(int level, char *tag,
struct zfcp_erp_action *erp);
@@ -49,10 +52,15 @@ extern void zfcp_dbf_san_res(char *, struct zfcp_fsf_req *);
extern void zfcp_dbf_san_in_els(char *, struct zfcp_fsf_req *);
extern void zfcp_dbf_scsi(char *, int, struct scsi_cmnd *,
struct zfcp_fsf_req *);
+extern void zfcp_dbf_scsi_eh(char *tag, struct zfcp_adapter *adapter,
+ unsigned int scsi_id, int ret);
/* zfcp_erp.c */
extern void zfcp_erp_set_adapter_status(struct zfcp_adapter *, u32);
extern void zfcp_erp_clear_adapter_status(struct zfcp_adapter *, u32);
+extern void zfcp_erp_port_forced_no_port_dbf(char *id,
+ struct zfcp_adapter *adapter,
+ u64 port_name, u32 port_id);
extern void zfcp_erp_adapter_reopen(struct zfcp_adapter *, int, char *);
extern void zfcp_erp_adapter_shutdown(struct zfcp_adapter *, int, char *);
extern void zfcp_erp_set_port_status(struct zfcp_port *, u32);
diff --git a/drivers/s390/scsi/zfcp_scsi.c b/drivers/s390/scsi/zfcp_scsi.c
index a9b8104..3afb200 100644
--- a/drivers/s390/scsi/zfcp_scsi.c
+++ b/drivers/s390/scsi/zfcp_scsi.c
@@ -3,7 +3,7 @@
*
* Interface to Linux SCSI midlayer.
*
- * Copyright IBM Corp. 2002, 2017
+ * Copyright IBM Corp. 2002, 2018
*/
#define KMSG_COMPONENT "zfcp"
@@ -180,6 +180,7 @@ static int zfcp_scsi_eh_abort_handler(struct scsi_cmnd *scpnt)
if (abrt_req)
break;
+ zfcp_dbf_scsi_abort("abrt_wt", scpnt, NULL);
zfcp_erp_wait(adapter);
ret = fc_block_scsi_eh(scpnt);
if (ret) {
@@ -276,6 +277,7 @@ static int zfcp_task_mgmt_function(struct scsi_cmnd *scpnt, u8 tm_flags)
if (fsf_req)
break;
+ zfcp_dbf_scsi_devreset("wait", scpnt, tm_flags, NULL);
zfcp_erp_wait(adapter);
ret = fc_block_scsi_eh(scpnt);
if (ret) {
@@ -322,15 +324,16 @@ static int zfcp_scsi_eh_host_reset_handler(struct scsi_cmnd *scpnt)
{
struct zfcp_scsi_dev *zfcp_sdev = sdev_to_zfcp(scpnt->device);
struct zfcp_adapter *adapter = zfcp_sdev->port->adapter;
- int ret;
+ int ret = SUCCESS, fc_ret;
zfcp_erp_adapter_reopen(adapter, 0, "schrh_1");
zfcp_erp_wait(adapter);
- ret = fc_block_scsi_eh(scpnt);
- if (ret)
- return ret;
+ fc_ret = fc_block_scsi_eh(scpnt);
+ if (fc_ret)
+ ret = fc_ret;
- return SUCCESS;
+ zfcp_dbf_scsi_eh("schrh_r", adapter, ~0, ret);
+ return ret;
}
struct scsi_transport_template *zfcp_scsi_transport_template;
@@ -600,6 +603,11 @@ static void zfcp_scsi_terminate_rport_io(struct fc_rport *rport)
if (port) {
zfcp_erp_port_forced_reopen(port, 0, "sctrpi1");
put_device(&port->dev);
+ } else {
+ zfcp_erp_port_forced_no_port_dbf(
+ "sctrpin", adapter,
+ rport->port_name /* zfcp_scsi_rport_register */,
+ rport->port_id /* zfcp_scsi_rport_register */);
}
}
@@ -616,9 +624,9 @@ static void zfcp_scsi_rport_register(struct zfcp_port *port)
ids.port_id = port->d_id;
ids.roles = FC_RPORT_ROLE_FCP_TARGET;
- zfcp_dbf_rec_trig("scpaddy", port->adapter, port, NULL,
- ZFCP_PSEUDO_ERP_ACTION_RPORT_ADD,
- ZFCP_PSEUDO_ERP_ACTION_RPORT_ADD);
+ zfcp_dbf_rec_trig_lock("scpaddy", port->adapter, port, NULL,
+ ZFCP_PSEUDO_ERP_ACTION_RPORT_ADD,
+ ZFCP_PSEUDO_ERP_ACTION_RPORT_ADD);
rport = fc_remote_port_add(port->adapter->scsi_host, 0, &ids);
if (!rport) {
dev_err(&port->adapter->ccw_device->dev,
@@ -640,9 +648,9 @@ static void zfcp_scsi_rport_block(struct zfcp_port *port)
struct fc_rport *rport = port->rport;
if (rport) {
- zfcp_dbf_rec_trig("scpdely", port->adapter, port, NULL,
- ZFCP_PSEUDO_ERP_ACTION_RPORT_DEL,
- ZFCP_PSEUDO_ERP_ACTION_RPORT_DEL);
+ zfcp_dbf_rec_trig_lock("scpdely", port->adapter, port, NULL,
+ ZFCP_PSEUDO_ERP_ACTION_RPORT_DEL,
+ ZFCP_PSEUDO_ERP_ACTION_RPORT_DEL);
fc_remote_port_delete(rport);
port->rport = NULL;
}
diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
index 8688ad4c..3493d44 100644
--- a/drivers/s390/virtio/virtio_ccw.c
+++ b/drivers/s390/virtio/virtio_ccw.c
@@ -59,6 +59,7 @@ struct virtio_ccw_device {
unsigned int revision; /* Transport revision */
wait_queue_head_t wait_q;
spinlock_t lock;
+ struct mutex io_lock; /* Serializes I/O requests */
struct list_head virtqueues;
unsigned long indicators;
unsigned long indicators2;
@@ -307,6 +308,7 @@ static int ccw_io_helper(struct virtio_ccw_device *vcdev,
unsigned long flags;
int flag = intparm & VIRTIO_CCW_INTPARM_MASK;
+ mutex_lock(&vcdev->io_lock);
do {
spin_lock_irqsave(get_ccwdev_lock(vcdev->cdev), flags);
ret = ccw_device_start(vcdev->cdev, ccw, intparm, 0, 0);
@@ -319,7 +321,9 @@ static int ccw_io_helper(struct virtio_ccw_device *vcdev,
cpu_relax();
} while (ret == -EBUSY);
wait_event(vcdev->wait_q, doing_io(vcdev, flag) == 0);
- return ret ? ret : vcdev->err;
+ ret = ret ? ret : vcdev->err;
+ mutex_unlock(&vcdev->io_lock);
+ return ret;
}
static void virtio_ccw_drop_indicator(struct virtio_ccw_device *vcdev,
@@ -837,6 +841,7 @@ static void virtio_ccw_get_config(struct virtio_device *vdev,
int ret;
struct ccw1 *ccw;
void *config_area;
+ unsigned long flags;
ccw = kzalloc(sizeof(*ccw), GFP_DMA | GFP_KERNEL);
if (!ccw)
@@ -855,11 +860,13 @@ static void virtio_ccw_get_config(struct virtio_device *vdev,
if (ret)
goto out_free;
+ spin_lock_irqsave(&vcdev->lock, flags);
memcpy(vcdev->config, config_area, offset + len);
- if (buf)
- memcpy(buf, &vcdev->config[offset], len);
if (vcdev->config_ready < offset + len)
vcdev->config_ready = offset + len;
+ spin_unlock_irqrestore(&vcdev->lock, flags);
+ if (buf)
+ memcpy(buf, config_area + offset, len);
out_free:
kfree(config_area);
@@ -873,6 +880,7 @@ static void virtio_ccw_set_config(struct virtio_device *vdev,
struct virtio_ccw_device *vcdev = to_vc_device(vdev);
struct ccw1 *ccw;
void *config_area;
+ unsigned long flags;
ccw = kzalloc(sizeof(*ccw), GFP_DMA | GFP_KERNEL);
if (!ccw)
@@ -885,9 +893,11 @@ static void virtio_ccw_set_config(struct virtio_device *vdev,
/* Make sure we don't overwrite fields. */
if (vcdev->config_ready < offset)
virtio_ccw_get_config(vdev, 0, NULL, offset);
+ spin_lock_irqsave(&vcdev->lock, flags);
memcpy(&vcdev->config[offset], buf, len);
/* Write the config area to the host. */
memcpy(config_area, vcdev->config, sizeof(vcdev->config));
+ spin_unlock_irqrestore(&vcdev->lock, flags);
ccw->cmd_code = CCW_CMD_WRITE_CONF;
ccw->flags = 0;
ccw->count = offset + len;
@@ -1233,6 +1243,7 @@ static int virtio_ccw_online(struct ccw_device *cdev)
init_waitqueue_head(&vcdev->wait_q);
INIT_LIST_HEAD(&vcdev->virtqueues);
spin_lock_init(&vcdev->lock);
+ mutex_init(&vcdev->io_lock);
spin_lock_irqsave(get_ccwdev_lock(cdev), flags);
dev_set_drvdata(&cdev->dev, vcdev);
diff --git a/drivers/sbus/char/display7seg.c b/drivers/sbus/char/display7seg.c
index 33fbe82..044cffb 100644
--- a/drivers/sbus/char/display7seg.c
+++ b/drivers/sbus/char/display7seg.c
@@ -221,6 +221,7 @@ static int d7s_probe(struct platform_device *op)
dev_set_drvdata(&op->dev, p);
d7s_device = p;
err = 0;
+ of_node_put(opts);
out:
return err;
diff --git a/drivers/sbus/char/envctrl.c b/drivers/sbus/char/envctrl.c
index 5609b60..baa9b32 100644
--- a/drivers/sbus/char/envctrl.c
+++ b/drivers/sbus/char/envctrl.c
@@ -910,8 +910,10 @@ static void envctrl_init_i2c_child(struct device_node *dp,
for (len = 0; len < PCF8584_MAX_CHANNELS; ++len) {
pchild->mon_type[len] = ENVCTRL_NOMON;
}
+ of_node_put(root_node);
return;
}
+ of_node_put(root_node);
}
/* Get the monitor channels. */
diff --git a/drivers/scsi/3w-9xxx.c b/drivers/scsi/3w-9xxx.c
index a56a7b2..b78a2f3 100644
--- a/drivers/scsi/3w-9xxx.c
+++ b/drivers/scsi/3w-9xxx.c
@@ -889,6 +889,11 @@ static int twa_chrdev_open(struct inode *inode, struct file *file)
unsigned int minor_number;
int retval = TW_IOCTL_ERROR_OS_ENODEV;
+ if (!capable(CAP_SYS_ADMIN)) {
+ retval = -EACCES;
+ goto out;
+ }
+
minor_number = iminor(inode);
if (minor_number >= twa_device_extension_count)
goto out;
@@ -2040,6 +2045,7 @@ static int twa_probe(struct pci_dev *pdev, const struct pci_device_id *dev_id)
if (twa_initialize_device_extension(tw_dev)) {
TW_PRINTK(tw_dev->host, TW_DRIVER, 0x25, "Failed to initialize device extension");
+ retval = -ENOMEM;
goto out_free_device_extension;
}
@@ -2062,6 +2068,7 @@ static int twa_probe(struct pci_dev *pdev, const struct pci_device_id *dev_id)
tw_dev->base_addr = ioremap(mem_addr, mem_len);
if (!tw_dev->base_addr) {
TW_PRINTK(tw_dev->host, TW_DRIVER, 0x35, "Failed to ioremap");
+ retval = -ENOMEM;
goto out_release_mem_region;
}
@@ -2069,8 +2076,10 @@ static int twa_probe(struct pci_dev *pdev, const struct pci_device_id *dev_id)
TW_DISABLE_INTERRUPTS(tw_dev);
/* Initialize the card */
- if (twa_reset_sequence(tw_dev, 0))
+ if (twa_reset_sequence(tw_dev, 0)) {
+ retval = -ENOMEM;
goto out_iounmap;
+ }
/* Set host specific parameters */
if ((pdev->device == PCI_DEVICE_ID_3WARE_9650SE) ||
diff --git a/drivers/scsi/3w-sas.c b/drivers/scsi/3w-sas.c
index f837485..f0a5536 100644
--- a/drivers/scsi/3w-sas.c
+++ b/drivers/scsi/3w-sas.c
@@ -1600,6 +1600,7 @@ static int twl_probe(struct pci_dev *pdev, const struct pci_device_id *dev_id)
if (twl_initialize_device_extension(tw_dev)) {
TW_PRINTK(tw_dev->host, TW_DRIVER, 0x1a, "Failed to initialize device extension");
+ retval = -ENOMEM;
goto out_free_device_extension;
}
@@ -1614,6 +1615,7 @@ static int twl_probe(struct pci_dev *pdev, const struct pci_device_id *dev_id)
tw_dev->base_addr = pci_iomap(pdev, 1, 0);
if (!tw_dev->base_addr) {
TW_PRINTK(tw_dev->host, TW_DRIVER, 0x1c, "Failed to ioremap");
+ retval = -ENOMEM;
goto out_release_mem_region;
}
@@ -1623,6 +1625,7 @@ static int twl_probe(struct pci_dev *pdev, const struct pci_device_id *dev_id)
/* Initialize the card */
if (twl_reset_sequence(tw_dev, 0)) {
TW_PRINTK(tw_dev->host, TW_DRIVER, 0x1d, "Controller reset failed during probe");
+ retval = -ENOMEM;
goto out_iounmap;
}
diff --git a/drivers/scsi/3w-xxxx.c b/drivers/scsi/3w-xxxx.c
index 25aba16..0ee0835 100644
--- a/drivers/scsi/3w-xxxx.c
+++ b/drivers/scsi/3w-xxxx.c
@@ -1034,6 +1034,9 @@ static int tw_chrdev_open(struct inode *inode, struct file *file)
dprintk(KERN_WARNING "3w-xxxx: tw_ioctl_open()\n");
+ if (!capable(CAP_SYS_ADMIN))
+ return -EACCES;
+
minor_number = iminor(inode);
if (minor_number >= tw_device_extension_count)
return -ENODEV;
@@ -2278,6 +2281,7 @@ static int tw_probe(struct pci_dev *pdev, const struct pci_device_id *dev_id)
if (tw_initialize_device_extension(tw_dev)) {
printk(KERN_WARNING "3w-xxxx: Failed to initialize device extension.");
+ retval = -ENOMEM;
goto out_free_device_extension;
}
@@ -2292,6 +2296,7 @@ static int tw_probe(struct pci_dev *pdev, const struct pci_device_id *dev_id)
tw_dev->base_addr = pci_resource_start(pdev, 0);
if (!tw_dev->base_addr) {
printk(KERN_WARNING "3w-xxxx: Failed to get io address.");
+ retval = -ENOMEM;
goto out_release_mem_region;
}
diff --git a/drivers/scsi/aacraid/commsup.c b/drivers/scsi/aacraid/commsup.c
index e2962f1..fe670b6 100644
--- a/drivers/scsi/aacraid/commsup.c
+++ b/drivers/scsi/aacraid/commsup.c
@@ -1374,9 +1374,10 @@ static int _aac_reset_adapter(struct aac_dev *aac, int forced)
host = aac->scsi_host_ptr;
scsi_block_requests(host);
aac_adapter_disable_int(aac);
- if (aac->thread->pid != current->pid) {
+ if (aac->thread && aac->thread->pid != current->pid) {
spin_unlock_irq(host->host_lock);
kthread_stop(aac->thread);
+ aac->thread = NULL;
jafo = 1;
}
@@ -1445,6 +1446,7 @@ static int _aac_reset_adapter(struct aac_dev *aac, int forced)
aac->name);
if (IS_ERR(aac->thread)) {
retval = PTR_ERR(aac->thread);
+ aac->thread = NULL;
goto out;
}
}
diff --git a/drivers/scsi/aacraid/linit.c b/drivers/scsi/aacraid/linit.c
index d5b26fa..ad902a6 100644
--- a/drivers/scsi/aacraid/linit.c
+++ b/drivers/scsi/aacraid/linit.c
@@ -1083,6 +1083,7 @@ static void __aac_shutdown(struct aac_dev * aac)
up(&fib->event_wait);
}
kthread_stop(aac->thread);
+ aac->thread = NULL;
}
aac_adapter_disable_int(aac);
cpu = cpumask_first(cpu_online_mask);
@@ -1203,8 +1204,10 @@ static int aac_probe_one(struct pci_dev *pdev, const struct pci_device_id *id)
* Map in the registers from the adapter.
*/
aac->base_size = AAC_MIN_FOOTPRINT_SIZE;
- if ((*aac_drivers[index].init)(aac))
+ if ((*aac_drivers[index].init)(aac)) {
+ error = -ENODEV;
goto out_unmap;
+ }
if (aac->sync_mode) {
if (aac_sync_mode)
diff --git a/drivers/scsi/aacraid/src.c b/drivers/scsi/aacraid/src.c
index 7b178d7..c0592fd 100644
--- a/drivers/scsi/aacraid/src.c
+++ b/drivers/scsi/aacraid/src.c
@@ -445,7 +445,7 @@ static int aac_src_check_health(struct aac_dev *dev)
return -1;
err_blink:
- return (status > 16) & 0xFF;
+ return (status >> 16) & 0xFF;
}
/**
diff --git a/drivers/scsi/aic94xx/aic94xx_init.c b/drivers/scsi/aic94xx/aic94xx_init.c
index 662b232..913ebb6 100644
--- a/drivers/scsi/aic94xx/aic94xx_init.c
+++ b/drivers/scsi/aic94xx/aic94xx_init.c
@@ -1031,8 +1031,10 @@ static int __init aic94xx_init(void)
aic94xx_transport_template =
sas_domain_attach_transport(&aic94xx_transport_functions);
- if (!aic94xx_transport_template)
+ if (!aic94xx_transport_template) {
+ err = -ENOMEM;
goto out_destroy_caches;
+ }
err = pci_register_driver(&aic94xx_pci_driver);
if (err)
diff --git a/drivers/scsi/arm/fas216.c b/drivers/scsi/arm/fas216.c
index 2438879..936e8c7 100644
--- a/drivers/scsi/arm/fas216.c
+++ b/drivers/scsi/arm/fas216.c
@@ -2011,7 +2011,7 @@ static void fas216_rq_sns_done(FAS216_Info *info, struct scsi_cmnd *SCpnt,
* have valid data in the sense buffer that could
* confuse the higher levels.
*/
- memset(SCpnt->sense_buffer, 0, sizeof(SCpnt->sense_buffer));
+ memset(SCpnt->sense_buffer, 0, SCSI_SENSE_BUFFERSIZE);
//printk("scsi%d.%c: sense buffer: ", info->host->host_no, '0' + SCpnt->device->id);
//{ int i; for (i = 0; i < 32; i++) printk("%02x ", SCpnt->sense_buffer[i]); printk("\n"); }
/*
diff --git a/drivers/scsi/bfa/bfa_fcbuild.c b/drivers/scsi/bfa/bfa_fcbuild.c
index b8dadc9..d3b00a4 100644
--- a/drivers/scsi/bfa/bfa_fcbuild.c
+++ b/drivers/scsi/bfa/bfa_fcbuild.c
@@ -1250,8 +1250,8 @@ fc_rspnid_build(struct fchs_s *fchs, void *pyld, u32 s_id, u16 ox_id,
memset(rspnid, 0, sizeof(struct fcgs_rspnid_req_s));
rspnid->dap = s_id;
- rspnid->spn_len = (u8) strlen((char *)name);
- strncpy((char *)rspnid->spn, (char *)name, rspnid->spn_len);
+ strlcpy(rspnid->spn, name, sizeof(rspnid->spn));
+ rspnid->spn_len = (u8) strlen(rspnid->spn);
return sizeof(struct fcgs_rspnid_req_s) + sizeof(struct ct_hdr_s);
}
@@ -1271,8 +1271,8 @@ fc_rsnn_nn_build(struct fchs_s *fchs, void *pyld, u32 s_id,
memset(rsnn_nn, 0, sizeof(struct fcgs_rsnn_nn_req_s));
rsnn_nn->node_name = node_name;
- rsnn_nn->snn_len = (u8) strlen((char *)name);
- strncpy((char *)rsnn_nn->snn, (char *)name, rsnn_nn->snn_len);
+ strlcpy(rsnn_nn->snn, name, sizeof(rsnn_nn->snn));
+ rsnn_nn->snn_len = (u8) strlen(rsnn_nn->snn);
return sizeof(struct fcgs_rsnn_nn_req_s) + sizeof(struct ct_hdr_s);
}
diff --git a/drivers/scsi/bfa/bfa_fcs.c b/drivers/scsi/bfa/bfa_fcs.c
index 1e7e139..f602de0 100644
--- a/drivers/scsi/bfa/bfa_fcs.c
+++ b/drivers/scsi/bfa/bfa_fcs.c
@@ -832,23 +832,23 @@ bfa_fcs_fabric_psymb_init(struct bfa_fcs_fabric_s *fabric)
bfa_ioc_get_adapter_model(&fabric->fcs->bfa->ioc, model);
/* Model name/number */
- strncpy((char *)&port_cfg->sym_name, model,
- BFA_FCS_PORT_SYMBNAME_MODEL_SZ);
- strncat((char *)&port_cfg->sym_name, BFA_FCS_PORT_SYMBNAME_SEPARATOR,
- sizeof(BFA_FCS_PORT_SYMBNAME_SEPARATOR));
+ strlcpy(port_cfg->sym_name.symname, model,
+ BFA_SYMNAME_MAXLEN);
+ strlcat(port_cfg->sym_name.symname, BFA_FCS_PORT_SYMBNAME_SEPARATOR,
+ BFA_SYMNAME_MAXLEN);
/* Driver Version */
- strncat((char *)&port_cfg->sym_name, (char *)driver_info->version,
- BFA_FCS_PORT_SYMBNAME_VERSION_SZ);
- strncat((char *)&port_cfg->sym_name, BFA_FCS_PORT_SYMBNAME_SEPARATOR,
- sizeof(BFA_FCS_PORT_SYMBNAME_SEPARATOR));
+ strlcat(port_cfg->sym_name.symname, driver_info->version,
+ BFA_SYMNAME_MAXLEN);
+ strlcat(port_cfg->sym_name.symname, BFA_FCS_PORT_SYMBNAME_SEPARATOR,
+ BFA_SYMNAME_MAXLEN);
/* Host machine name */
- strncat((char *)&port_cfg->sym_name,
- (char *)driver_info->host_machine_name,
- BFA_FCS_PORT_SYMBNAME_MACHINENAME_SZ);
- strncat((char *)&port_cfg->sym_name, BFA_FCS_PORT_SYMBNAME_SEPARATOR,
- sizeof(BFA_FCS_PORT_SYMBNAME_SEPARATOR));
+ strlcat(port_cfg->sym_name.symname,
+ driver_info->host_machine_name,
+ BFA_SYMNAME_MAXLEN);
+ strlcat(port_cfg->sym_name.symname, BFA_FCS_PORT_SYMBNAME_SEPARATOR,
+ BFA_SYMNAME_MAXLEN);
/*
* Host OS Info :
@@ -856,24 +856,24 @@ bfa_fcs_fabric_psymb_init(struct bfa_fcs_fabric_s *fabric)
* OS name string and instead copy the entire OS info string (64 bytes).
*/
if (driver_info->host_os_patch[0] == '\0') {
- strncat((char *)&port_cfg->sym_name,
- (char *)driver_info->host_os_name,
- BFA_FCS_OS_STR_LEN);
- strncat((char *)&port_cfg->sym_name,
+ strlcat(port_cfg->sym_name.symname,
+ driver_info->host_os_name,
+ BFA_SYMNAME_MAXLEN);
+ strlcat(port_cfg->sym_name.symname,
BFA_FCS_PORT_SYMBNAME_SEPARATOR,
- sizeof(BFA_FCS_PORT_SYMBNAME_SEPARATOR));
+ BFA_SYMNAME_MAXLEN);
} else {
- strncat((char *)&port_cfg->sym_name,
- (char *)driver_info->host_os_name,
- BFA_FCS_PORT_SYMBNAME_OSINFO_SZ);
- strncat((char *)&port_cfg->sym_name,
+ strlcat(port_cfg->sym_name.symname,
+ driver_info->host_os_name,
+ BFA_SYMNAME_MAXLEN);
+ strlcat(port_cfg->sym_name.symname,
BFA_FCS_PORT_SYMBNAME_SEPARATOR,
- sizeof(BFA_FCS_PORT_SYMBNAME_SEPARATOR));
+ BFA_SYMNAME_MAXLEN);
/* Append host OS Patch Info */
- strncat((char *)&port_cfg->sym_name,
- (char *)driver_info->host_os_patch,
- BFA_FCS_PORT_SYMBNAME_OSPATCH_SZ);
+ strlcat(port_cfg->sym_name.symname,
+ driver_info->host_os_patch,
+ BFA_SYMNAME_MAXLEN);
}
/* null terminate */
@@ -893,26 +893,26 @@ bfa_fcs_fabric_nsymb_init(struct bfa_fcs_fabric_s *fabric)
bfa_ioc_get_adapter_model(&fabric->fcs->bfa->ioc, model);
/* Model name/number */
- strncpy((char *)&port_cfg->node_sym_name, model,
- BFA_FCS_PORT_SYMBNAME_MODEL_SZ);
- strncat((char *)&port_cfg->node_sym_name,
+ strlcpy(port_cfg->node_sym_name.symname, model,
+ BFA_SYMNAME_MAXLEN);
+ strlcat(port_cfg->node_sym_name.symname,
BFA_FCS_PORT_SYMBNAME_SEPARATOR,
- sizeof(BFA_FCS_PORT_SYMBNAME_SEPARATOR));
+ BFA_SYMNAME_MAXLEN);
/* Driver Version */
- strncat((char *)&port_cfg->node_sym_name, (char *)driver_info->version,
- BFA_FCS_PORT_SYMBNAME_VERSION_SZ);
- strncat((char *)&port_cfg->node_sym_name,
+ strlcat(port_cfg->node_sym_name.symname, (char *)driver_info->version,
+ BFA_SYMNAME_MAXLEN);
+ strlcat(port_cfg->node_sym_name.symname,
BFA_FCS_PORT_SYMBNAME_SEPARATOR,
- sizeof(BFA_FCS_PORT_SYMBNAME_SEPARATOR));
+ BFA_SYMNAME_MAXLEN);
/* Host machine name */
- strncat((char *)&port_cfg->node_sym_name,
- (char *)driver_info->host_machine_name,
- BFA_FCS_PORT_SYMBNAME_MACHINENAME_SZ);
- strncat((char *)&port_cfg->node_sym_name,
+ strlcat(port_cfg->node_sym_name.symname,
+ driver_info->host_machine_name,
+ BFA_SYMNAME_MAXLEN);
+ strlcat(port_cfg->node_sym_name.symname,
BFA_FCS_PORT_SYMBNAME_SEPARATOR,
- sizeof(BFA_FCS_PORT_SYMBNAME_SEPARATOR));
+ BFA_SYMNAME_MAXLEN);
/* null terminate */
port_cfg->node_sym_name.symname[BFA_SYMNAME_MAXLEN - 1] = 0;
diff --git a/drivers/scsi/bfa/bfa_fcs_lport.c b/drivers/scsi/bfa/bfa_fcs_lport.c
index 4ddda72..eb87949 100644
--- a/drivers/scsi/bfa/bfa_fcs_lport.c
+++ b/drivers/scsi/bfa/bfa_fcs_lport.c
@@ -2631,10 +2631,10 @@ bfa_fcs_fdmi_get_hbaattr(struct bfa_fcs_lport_fdmi_s *fdmi,
bfa_ioc_get_adapter_fw_ver(&port->fcs->bfa->ioc,
hba_attr->fw_version);
- strncpy(hba_attr->driver_version, (char *)driver_info->version,
+ strlcpy(hba_attr->driver_version, (char *)driver_info->version,
sizeof(hba_attr->driver_version));
- strncpy(hba_attr->os_name, driver_info->host_os_name,
+ strlcpy(hba_attr->os_name, driver_info->host_os_name,
sizeof(hba_attr->os_name));
/*
@@ -2642,23 +2642,23 @@ bfa_fcs_fdmi_get_hbaattr(struct bfa_fcs_lport_fdmi_s *fdmi,
* to the os name along with a separator
*/
if (driver_info->host_os_patch[0] != '\0') {
- strncat(hba_attr->os_name, BFA_FCS_PORT_SYMBNAME_SEPARATOR,
- sizeof(BFA_FCS_PORT_SYMBNAME_SEPARATOR));
- strncat(hba_attr->os_name, driver_info->host_os_patch,
- sizeof(driver_info->host_os_patch));
+ strlcat(hba_attr->os_name, BFA_FCS_PORT_SYMBNAME_SEPARATOR,
+ sizeof(hba_attr->os_name));
+ strlcat(hba_attr->os_name, driver_info->host_os_patch,
+ sizeof(hba_attr->os_name));
}
/* Retrieve the max frame size from the port attr */
bfa_fcs_fdmi_get_portattr(fdmi, &fcs_port_attr);
hba_attr->max_ct_pyld = fcs_port_attr.max_frm_size;
- strncpy(hba_attr->node_sym_name.symname,
+ strlcpy(hba_attr->node_sym_name.symname,
port->port_cfg.node_sym_name.symname, BFA_SYMNAME_MAXLEN);
strcpy(hba_attr->vendor_info, "QLogic");
hba_attr->num_ports =
cpu_to_be32(bfa_ioc_get_nports(&port->fcs->bfa->ioc));
hba_attr->fabric_name = port->fabric->lps->pr_nwwn;
- strncpy(hba_attr->bios_ver, hba_attr->option_rom_ver, BFA_VERSION_LEN);
+ strlcpy(hba_attr->bios_ver, hba_attr->option_rom_ver, BFA_VERSION_LEN);
}
@@ -2725,20 +2725,20 @@ bfa_fcs_fdmi_get_portattr(struct bfa_fcs_lport_fdmi_s *fdmi,
/*
* OS device Name
*/
- strncpy(port_attr->os_device_name, (char *)driver_info->os_device_name,
+ strlcpy(port_attr->os_device_name, driver_info->os_device_name,
sizeof(port_attr->os_device_name));
/*
* Host name
*/
- strncpy(port_attr->host_name, (char *)driver_info->host_machine_name,
+ strlcpy(port_attr->host_name, driver_info->host_machine_name,
sizeof(port_attr->host_name));
port_attr->node_name = bfa_fcs_lport_get_nwwn(port);
port_attr->port_name = bfa_fcs_lport_get_pwwn(port);
- strncpy(port_attr->port_sym_name.symname,
- (char *)&bfa_fcs_lport_get_psym_name(port), BFA_SYMNAME_MAXLEN);
+ strlcpy(port_attr->port_sym_name.symname,
+ bfa_fcs_lport_get_psym_name(port).symname, BFA_SYMNAME_MAXLEN);
bfa_fcs_lport_get_attr(port, &lport_attr);
port_attr->port_type = cpu_to_be32(lport_attr.port_type);
port_attr->scos = pport_attr.cos_supported;
@@ -3218,7 +3218,7 @@ bfa_fcs_lport_ms_gmal_response(void *fcsarg, struct bfa_fcxp_s *fcxp,
rsp_str[gmal_entry->len-1] = 0;
/* copy IP Address to fabric */
- strncpy(bfa_fcs_lport_get_fabric_ipaddr(port),
+ strlcpy(bfa_fcs_lport_get_fabric_ipaddr(port),
gmal_entry->ip_addr,
BFA_FCS_FABRIC_IPADDR_SZ);
break;
@@ -4656,21 +4656,13 @@ bfa_fcs_lport_ns_send_rspn_id(void *ns_cbarg, struct bfa_fcxp_s *fcxp_alloced)
* to that of the base port.
*/
- strncpy((char *)psymbl,
- (char *) &
- (bfa_fcs_lport_get_psym_name
+ strlcpy(symbl,
+ (char *)&(bfa_fcs_lport_get_psym_name
(bfa_fcs_get_base_port(port->fcs))),
- strlen((char *) &
- bfa_fcs_lport_get_psym_name(bfa_fcs_get_base_port
- (port->fcs))));
+ sizeof(symbl));
- /* Ensure we have a null terminating string. */
- ((char *)psymbl)[strlen((char *) &
- bfa_fcs_lport_get_psym_name(bfa_fcs_get_base_port
- (port->fcs)))] = 0;
- strncat((char *)psymbl,
- (char *) &(bfa_fcs_lport_get_psym_name(port)),
- strlen((char *) &bfa_fcs_lport_get_psym_name(port)));
+ strlcat(symbl, (char *)&(bfa_fcs_lport_get_psym_name(port)),
+ sizeof(symbl));
} else {
psymbl = (u8 *) &(bfa_fcs_lport_get_psym_name(port));
}
@@ -5162,7 +5154,6 @@ bfa_fcs_lport_ns_util_send_rspn_id(void *cbarg, struct bfa_fcxp_s *fcxp_alloced)
struct fchs_s fchs;
struct bfa_fcxp_s *fcxp;
u8 symbl[256];
- u8 *psymbl = &symbl[0];
int len;
/* Avoid sending RSPN in the following states. */
@@ -5192,22 +5183,17 @@ bfa_fcs_lport_ns_util_send_rspn_id(void *cbarg, struct bfa_fcxp_s *fcxp_alloced)
* For Vports, we append the vport's port symbolic name
* to that of the base port.
*/
- strncpy((char *)psymbl, (char *)&(bfa_fcs_lport_get_psym_name
+ strlcpy(symbl, (char *)&(bfa_fcs_lport_get_psym_name
(bfa_fcs_get_base_port(port->fcs))),
- strlen((char *)&bfa_fcs_lport_get_psym_name(
- bfa_fcs_get_base_port(port->fcs))));
+ sizeof(symbl));
- /* Ensure we have a null terminating string. */
- ((char *)psymbl)[strlen((char *)&bfa_fcs_lport_get_psym_name(
- bfa_fcs_get_base_port(port->fcs)))] = 0;
-
- strncat((char *)psymbl,
+ strlcat(symbl,
(char *)&(bfa_fcs_lport_get_psym_name(port)),
- strlen((char *)&bfa_fcs_lport_get_psym_name(port)));
+ sizeof(symbl));
}
len = fc_rspnid_build(&fchs, bfa_fcxp_get_reqbuf(fcxp),
- bfa_fcs_lport_get_fcid(port), 0, psymbl);
+ bfa_fcs_lport_get_fcid(port), 0, symbl);
bfa_fcxp_send(fcxp, NULL, port->fabric->vf_id, port->lp_tag, BFA_FALSE,
FC_CLASS_3, len, &fchs, NULL, NULL, FC_MAX_PDUSZ, 0);
diff --git a/drivers/scsi/bfa/bfa_ioc.c b/drivers/scsi/bfa/bfa_ioc.c
index a1ada4a..1675041 100644
--- a/drivers/scsi/bfa/bfa_ioc.c
+++ b/drivers/scsi/bfa/bfa_ioc.c
@@ -2803,7 +2803,7 @@ void
bfa_ioc_get_adapter_manufacturer(struct bfa_ioc_s *ioc, char *manufacturer)
{
memset((void *)manufacturer, 0, BFA_ADAPTER_MFG_NAME_LEN);
- strncpy(manufacturer, BFA_MFG_NAME, BFA_ADAPTER_MFG_NAME_LEN);
+ strlcpy(manufacturer, BFA_MFG_NAME, BFA_ADAPTER_MFG_NAME_LEN);
}
void
diff --git a/drivers/scsi/bfa/bfa_svc.c b/drivers/scsi/bfa/bfa_svc.c
index 12de2921..225883d 100644
--- a/drivers/scsi/bfa/bfa_svc.c
+++ b/drivers/scsi/bfa/bfa_svc.c
@@ -366,8 +366,8 @@ bfa_plog_str(struct bfa_plog_s *plog, enum bfa_plog_mid mid,
lp.eid = event;
lp.log_type = BFA_PL_LOG_TYPE_STRING;
lp.misc = misc;
- strncpy(lp.log_entry.string_log, log_str,
- BFA_PL_STRING_LOG_SZ - 1);
+ strlcpy(lp.log_entry.string_log, log_str,
+ BFA_PL_STRING_LOG_SZ);
lp.log_entry.string_log[BFA_PL_STRING_LOG_SZ - 1] = '\0';
bfa_plog_add(plog, &lp);
}
diff --git a/drivers/scsi/bfa/bfad.c b/drivers/scsi/bfa/bfad.c
index e70410b..389f8ef 100644
--- a/drivers/scsi/bfa/bfad.c
+++ b/drivers/scsi/bfa/bfad.c
@@ -983,20 +983,20 @@ bfad_start_ops(struct bfad_s *bfad) {
/* Fill the driver_info info to fcs*/
memset(&driver_info, 0, sizeof(driver_info));
- strncpy(driver_info.version, BFAD_DRIVER_VERSION,
- sizeof(driver_info.version) - 1);
+ strlcpy(driver_info.version, BFAD_DRIVER_VERSION,
+ sizeof(driver_info.version));
if (host_name)
- strncpy(driver_info.host_machine_name, host_name,
- sizeof(driver_info.host_machine_name) - 1);
+ strlcpy(driver_info.host_machine_name, host_name,
+ sizeof(driver_info.host_machine_name));
if (os_name)
- strncpy(driver_info.host_os_name, os_name,
- sizeof(driver_info.host_os_name) - 1);
+ strlcpy(driver_info.host_os_name, os_name,
+ sizeof(driver_info.host_os_name));
if (os_patch)
- strncpy(driver_info.host_os_patch, os_patch,
- sizeof(driver_info.host_os_patch) - 1);
+ strlcpy(driver_info.host_os_patch, os_patch,
+ sizeof(driver_info.host_os_patch));
- strncpy(driver_info.os_device_name, bfad->pci_name,
- sizeof(driver_info.os_device_name) - 1);
+ strlcpy(driver_info.os_device_name, bfad->pci_name,
+ sizeof(driver_info.os_device_name));
/* FCS driver info init */
spin_lock_irqsave(&bfad->bfad_lock, flags);
diff --git a/drivers/scsi/bfa/bfad_attr.c b/drivers/scsi/bfa/bfad_attr.c
index 13db3b7..d0a504a 100644
--- a/drivers/scsi/bfa/bfad_attr.c
+++ b/drivers/scsi/bfa/bfad_attr.c
@@ -843,7 +843,7 @@ bfad_im_symbolic_name_show(struct device *dev, struct device_attribute *attr,
char symname[BFA_SYMNAME_MAXLEN];
bfa_fcs_lport_get_attr(&bfad->bfa_fcs.fabric.bport, &port_attr);
- strncpy(symname, port_attr.port_cfg.sym_name.symname,
+ strlcpy(symname, port_attr.port_cfg.sym_name.symname,
BFA_SYMNAME_MAXLEN);
return snprintf(buf, PAGE_SIZE, "%s\n", symname);
}
diff --git a/drivers/scsi/bfa/bfad_bsg.c b/drivers/scsi/bfa/bfad_bsg.c
index d1ad020..a3bd236 100644
--- a/drivers/scsi/bfa/bfad_bsg.c
+++ b/drivers/scsi/bfa/bfad_bsg.c
@@ -127,7 +127,7 @@ bfad_iocmd_ioc_get_attr(struct bfad_s *bfad, void *cmd)
/* fill in driver attr info */
strcpy(iocmd->ioc_attr.driver_attr.driver, BFAD_DRIVER_NAME);
- strncpy(iocmd->ioc_attr.driver_attr.driver_ver,
+ strlcpy(iocmd->ioc_attr.driver_attr.driver_ver,
BFAD_DRIVER_VERSION, BFA_VERSION_LEN);
strcpy(iocmd->ioc_attr.driver_attr.fw_ver,
iocmd->ioc_attr.adapter_attr.fw_ver);
@@ -315,9 +315,9 @@ bfad_iocmd_port_get_attr(struct bfad_s *bfad, void *cmd)
iocmd->attr.port_type = port_attr.port_type;
iocmd->attr.loopback = port_attr.loopback;
iocmd->attr.authfail = port_attr.authfail;
- strncpy(iocmd->attr.port_symname.symname,
+ strlcpy(iocmd->attr.port_symname.symname,
port_attr.port_cfg.sym_name.symname,
- sizeof(port_attr.port_cfg.sym_name.symname));
+ sizeof(iocmd->attr.port_symname.symname));
iocmd->status = BFA_STATUS_OK;
return 0;
diff --git a/drivers/scsi/bnx2fc/bnx2fc_fcoe.c b/drivers/scsi/bnx2fc/bnx2fc_fcoe.c
index bee7d37..68cc332 100644
--- a/drivers/scsi/bnx2fc/bnx2fc_fcoe.c
+++ b/drivers/scsi/bnx2fc/bnx2fc_fcoe.c
@@ -2331,7 +2331,7 @@ static int _bnx2fc_create(struct net_device *netdev,
if (!interface) {
printk(KERN_ERR PFX "bnx2fc_interface_create failed\n");
rc = -ENOMEM;
- goto ifput_err;
+ goto netdev_err;
}
if (netdev->priv_flags & IFF_802_1Q_VLAN) {
diff --git a/drivers/scsi/bnx2fc/bnx2fc_io.c b/drivers/scsi/bnx2fc/bnx2fc_io.c
index f501095..bd39590 100644
--- a/drivers/scsi/bnx2fc/bnx2fc_io.c
+++ b/drivers/scsi/bnx2fc/bnx2fc_io.c
@@ -1869,6 +1869,7 @@ void bnx2fc_process_scsi_cmd_compl(struct bnx2fc_cmd *io_req,
/* we will not receive ABTS response for this IO */
BNX2FC_IO_DBG(io_req, "Timer context finished processing "
"this scsi cmd\n");
+ return;
}
/* Cancel the timeout_work, as we received IO completion */
diff --git a/drivers/scsi/bnx2i/bnx2i_hwi.c b/drivers/scsi/bnx2i/bnx2i_hwi.c
index 42921db..4ca1050 100644
--- a/drivers/scsi/bnx2i/bnx2i_hwi.c
+++ b/drivers/scsi/bnx2i/bnx2i_hwi.c
@@ -2742,6 +2742,8 @@ int bnx2i_map_ep_dbell_regs(struct bnx2i_endpoint *ep)
BNX2X_DOORBELL_PCI_BAR);
reg_off = (1 << BNX2X_DB_SHIFT) * (cid_num & 0x1FFFF);
ep->qp.ctx_base = ioremap_nocache(reg_base + reg_off, 4);
+ if (!ep->qp.ctx_base)
+ return -ENOMEM;
goto arm_cq;
}
diff --git a/drivers/scsi/csiostor/csio_lnode.c b/drivers/scsi/csiostor/csio_lnode.c
index c00b2ff..be5ee2d 100644
--- a/drivers/scsi/csiostor/csio_lnode.c
+++ b/drivers/scsi/csiostor/csio_lnode.c
@@ -238,14 +238,23 @@ csio_osname(uint8_t *buf, size_t buf_len)
}
static inline void
-csio_append_attrib(uint8_t **ptr, uint16_t type, uint8_t *val, uint16_t len)
+csio_append_attrib(uint8_t **ptr, uint16_t type, void *val, size_t val_len)
{
+ uint16_t len;
struct fc_fdmi_attr_entry *ae = (struct fc_fdmi_attr_entry *)*ptr;
+
+ if (WARN_ON(val_len > U16_MAX))
+ return;
+
+ len = val_len;
+
ae->type = htons(type);
len += 4; /* includes attribute type and length */
len = (len + 3) & ~3; /* should be multiple of 4 bytes */
ae->len = htons(len);
- memcpy(ae->value, val, len);
+ memcpy(ae->value, val, val_len);
+ if (len > val_len)
+ memset(ae->value + val_len, 0, len - val_len);
*ptr += len;
}
@@ -335,7 +344,7 @@ csio_ln_fdmi_rhba_cbfn(struct csio_hw *hw, struct csio_ioreq *fdmi_req)
numattrs++;
val = htonl(FC_PORTSPEED_1GBIT | FC_PORTSPEED_10GBIT);
csio_append_attrib(&pld, FC_FDMI_PORT_ATTR_SUPPORTEDSPEED,
- (uint8_t *)&val,
+ &val,
FC_FDMI_PORT_ATTR_SUPPORTEDSPEED_LEN);
numattrs++;
@@ -346,23 +355,22 @@ csio_ln_fdmi_rhba_cbfn(struct csio_hw *hw, struct csio_ioreq *fdmi_req)
else
val = htonl(CSIO_HBA_PORTSPEED_UNKNOWN);
csio_append_attrib(&pld, FC_FDMI_PORT_ATTR_CURRENTPORTSPEED,
- (uint8_t *)&val,
- FC_FDMI_PORT_ATTR_CURRENTPORTSPEED_LEN);
+ &val, FC_FDMI_PORT_ATTR_CURRENTPORTSPEED_LEN);
numattrs++;
mfs = ln->ln_sparm.csp.sp_bb_data;
csio_append_attrib(&pld, FC_FDMI_PORT_ATTR_MAXFRAMESIZE,
- (uint8_t *)&mfs, FC_FDMI_PORT_ATTR_MAXFRAMESIZE_LEN);
+ &mfs, sizeof(mfs));
numattrs++;
strcpy(buf, "csiostor");
csio_append_attrib(&pld, FC_FDMI_PORT_ATTR_OSDEVICENAME, buf,
- (uint16_t)strlen(buf));
+ strlen(buf));
numattrs++;
if (!csio_hostname(buf, sizeof(buf))) {
csio_append_attrib(&pld, FC_FDMI_PORT_ATTR_HOSTNAME,
- buf, (uint16_t)strlen(buf));
+ buf, strlen(buf));
numattrs++;
}
attrib_blk->numattrs = htonl(numattrs);
@@ -444,33 +452,32 @@ csio_ln_fdmi_dprt_cbfn(struct csio_hw *hw, struct csio_ioreq *fdmi_req)
strcpy(buf, "Chelsio Communications");
csio_append_attrib(&pld, FC_FDMI_HBA_ATTR_MANUFACTURER, buf,
- (uint16_t)strlen(buf));
+ strlen(buf));
numattrs++;
csio_append_attrib(&pld, FC_FDMI_HBA_ATTR_SERIALNUMBER,
- hw->vpd.sn, (uint16_t)sizeof(hw->vpd.sn));
+ hw->vpd.sn, sizeof(hw->vpd.sn));
numattrs++;
csio_append_attrib(&pld, FC_FDMI_HBA_ATTR_MODEL, hw->vpd.id,
- (uint16_t)sizeof(hw->vpd.id));
+ sizeof(hw->vpd.id));
numattrs++;
csio_append_attrib(&pld, FC_FDMI_HBA_ATTR_MODELDESCRIPTION,
- hw->model_desc, (uint16_t)strlen(hw->model_desc));
+ hw->model_desc, strlen(hw->model_desc));
numattrs++;
csio_append_attrib(&pld, FC_FDMI_HBA_ATTR_HARDWAREVERSION,
- hw->hw_ver, (uint16_t)sizeof(hw->hw_ver));
+ hw->hw_ver, sizeof(hw->hw_ver));
numattrs++;
csio_append_attrib(&pld, FC_FDMI_HBA_ATTR_FIRMWAREVERSION,
- hw->fwrev_str, (uint16_t)strlen(hw->fwrev_str));
+ hw->fwrev_str, strlen(hw->fwrev_str));
numattrs++;
if (!csio_osname(buf, sizeof(buf))) {
csio_append_attrib(&pld, FC_FDMI_HBA_ATTR_OSNAMEVERSION,
- buf, (uint16_t)strlen(buf));
+ buf, strlen(buf));
numattrs++;
}
csio_append_attrib(&pld, FC_FDMI_HBA_ATTR_MAXCTPAYLOAD,
- (uint8_t *)&maxpayload,
- FC_FDMI_HBA_ATTR_MAXCTPAYLOAD_LEN);
+ &maxpayload, FC_FDMI_HBA_ATTR_MAXCTPAYLOAD_LEN);
len = (uint32_t)(pld - (uint8_t *)cmd);
numattrs++;
attrib_blk->numattrs = htonl(numattrs);
@@ -1794,6 +1801,8 @@ csio_ln_mgmt_submit_req(struct csio_ioreq *io_req,
struct csio_mgmtm *mgmtm = csio_hw_to_mgmtm(hw);
int rv;
+ BUG_ON(pld_len > pld->len);
+
io_req->io_cbfn = io_cbfn; /* Upper layer callback handler */
io_req->fw_handle = (uintptr_t) (io_req);
io_req->eq_idx = mgmtm->eq_idx;
diff --git a/drivers/scsi/esp_scsi.c b/drivers/scsi/esp_scsi.c
index 71cb05b..60be074 100644
--- a/drivers/scsi/esp_scsi.c
+++ b/drivers/scsi/esp_scsi.c
@@ -1349,6 +1349,7 @@ static int esp_data_bytes_sent(struct esp *esp, struct esp_cmd_entry *ent,
bytes_sent = esp->data_dma_len;
bytes_sent -= ecount;
+ bytes_sent -= esp->send_cmd_residual;
/*
* The am53c974 has a DMA 'pecularity'. The doc states:
diff --git a/drivers/scsi/esp_scsi.h b/drivers/scsi/esp_scsi.h
index 84dcbe4..55be43f 100644
--- a/drivers/scsi/esp_scsi.h
+++ b/drivers/scsi/esp_scsi.h
@@ -540,6 +540,8 @@ struct esp {
void *dma;
int dmarev;
+
+ u32 send_cmd_residual;
};
/* A front-end driver for the ESP chip should do the following in
diff --git a/drivers/scsi/fcoe/fcoe_ctlr.c b/drivers/scsi/fcoe/fcoe_ctlr.c
index dcf3653..cc3994d 100644
--- a/drivers/scsi/fcoe/fcoe_ctlr.c
+++ b/drivers/scsi/fcoe/fcoe_ctlr.c
@@ -755,9 +755,9 @@ int fcoe_ctlr_els_send(struct fcoe_ctlr *fip, struct fc_lport *lport,
case ELS_LOGO:
if (fip->mode == FIP_MODE_VN2VN) {
if (fip->state != FIP_ST_VNMP_UP)
- return -EINVAL;
+ goto drop;
if (ntoh24(fh->fh_d_id) == FC_FID_FLOGI)
- return -EINVAL;
+ goto drop;
} else {
if (fip->state != FIP_ST_ENABLED)
return 0;
diff --git a/drivers/scsi/ibmvscsi/ibmvscsi.c b/drivers/scsi/ibmvscsi/ibmvscsi.c
index d9534ee..e173022 100644
--- a/drivers/scsi/ibmvscsi/ibmvscsi.c
+++ b/drivers/scsi/ibmvscsi/ibmvscsi.c
@@ -93,7 +93,7 @@ static int max_requests = IBMVSCSI_MAX_REQUESTS_DEFAULT;
static int max_events = IBMVSCSI_MAX_REQUESTS_DEFAULT + 2;
static int fast_fail = 1;
static int client_reserve = 1;
-static char partition_name[97] = "UNKNOWN";
+static char partition_name[96] = "UNKNOWN";
static unsigned int partition_number = -1;
static struct scsi_transport_template *ibmvscsi_transport_template;
@@ -259,7 +259,7 @@ static void gather_partition_info(void)
ppartition_name = of_get_property(of_root, "ibm,partition-name", NULL);
if (ppartition_name)
- strncpy(partition_name, ppartition_name,
+ strlcpy(partition_name, ppartition_name,
sizeof(partition_name));
p_number_ptr = of_get_property(of_root, "ibm,partition-no", NULL);
if (p_number_ptr)
diff --git a/drivers/scsi/ibmvscsi_tgt/ibmvscsi_tgt.c b/drivers/scsi/ibmvscsi_tgt/ibmvscsi_tgt.c
index 0414843..5ed2811 100644
--- a/drivers/scsi/ibmvscsi_tgt/ibmvscsi_tgt.c
+++ b/drivers/scsi/ibmvscsi_tgt/ibmvscsi_tgt.c
@@ -3342,11 +3342,10 @@ static int ibmvscsis_probe(struct vio_dev *vdev,
vscsi->dds.window[LOCAL].liobn,
vscsi->dds.window[REMOTE].liobn);
- strcpy(vscsi->eye, "VSCSI ");
- strncat(vscsi->eye, vdev->name, MAX_EYE);
+ snprintf(vscsi->eye, sizeof(vscsi->eye), "VSCSI %s", vdev->name);
vscsi->dds.unit_id = vdev->unit_address;
- strncpy(vscsi->dds.partition_name, partition_name,
+ strscpy(vscsi->dds.partition_name, partition_name,
sizeof(vscsi->dds.partition_name));
vscsi->dds.partition_num = partition_number;
diff --git a/drivers/scsi/libfc/fc_rport.c b/drivers/scsi/libfc/fc_rport.c
index 97aeadd..e3ffd24 100644
--- a/drivers/scsi/libfc/fc_rport.c
+++ b/drivers/scsi/libfc/fc_rport.c
@@ -1935,6 +1935,7 @@ static void fc_rport_recv_logo_req(struct fc_lport *lport, struct fc_frame *fp)
FC_RPORT_DBG(rdata, "Received LOGO request while in state %s\n",
fc_rport_state(rdata));
+ rdata->flags &= ~FC_RP_STARTED;
fc_rport_enter_delete(rdata, RPORT_EV_STOP);
mutex_unlock(&rdata->rp_mutex);
kref_put(&rdata->kref, rdata->local_port->tt.rport_destroy);
diff --git a/drivers/scsi/libiscsi.c b/drivers/scsi/libiscsi.c
index c2b6829..c79743d 100644
--- a/drivers/scsi/libiscsi.c
+++ b/drivers/scsi/libiscsi.c
@@ -283,11 +283,11 @@ static int iscsi_check_tmf_restrictions(struct iscsi_task *task, int opcode)
*/
if (opcode != ISCSI_OP_SCSI_DATA_OUT) {
iscsi_conn_printk(KERN_INFO, conn,
- "task [op %x/%x itt "
+ "task [op %x itt "
"0x%x/0x%x] "
"rejected.\n",
- task->hdr->opcode, opcode,
- task->itt, task->hdr_itt);
+ opcode, task->itt,
+ task->hdr_itt);
return -EACCES;
}
/*
@@ -296,10 +296,10 @@ static int iscsi_check_tmf_restrictions(struct iscsi_task *task, int opcode)
*/
if (conn->session->fast_abort) {
iscsi_conn_printk(KERN_INFO, conn,
- "task [op %x/%x itt "
+ "task [op %x itt "
"0x%x/0x%x] fast abort.\n",
- task->hdr->opcode, opcode,
- task->itt, task->hdr_itt);
+ opcode, task->itt,
+ task->hdr_itt);
return -EACCES;
}
break;
@@ -2414,8 +2414,8 @@ int iscsi_eh_session_reset(struct scsi_cmnd *sc)
failed:
ISCSI_DBG_EH(session,
"failing session reset: Could not log back into "
- "%s, %s [age %d]\n", session->targetname,
- conn->persistent_address, session->age);
+ "%s [age %d]\n", session->targetname,
+ session->age);
spin_unlock_bh(&session->frwd_lock);
mutex_unlock(&session->eh_mutex);
return FAILED;
diff --git a/drivers/scsi/libsas/sas_scsi_host.c b/drivers/scsi/libsas/sas_scsi_host.c
index 519dac4..9a8c2f9 100644
--- a/drivers/scsi/libsas/sas_scsi_host.c
+++ b/drivers/scsi/libsas/sas_scsi_host.c
@@ -222,6 +222,7 @@ int sas_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
static void sas_eh_finish_cmd(struct scsi_cmnd *cmd)
{
struct sas_ha_struct *sas_ha = SHOST_TO_SAS_HA(cmd->device->host);
+ struct domain_device *dev = cmd_to_domain_dev(cmd);
struct sas_task *task = TO_SAS_TASK(cmd);
/* At this point, we only get called following an actual abort
@@ -230,6 +231,14 @@ static void sas_eh_finish_cmd(struct scsi_cmnd *cmd)
*/
sas_end_task(cmd, task);
+ if (dev_is_sata(dev)) {
+ /* defer commands to libata so that libata EH can
+ * handle ata qcs correctly
+ */
+ list_move_tail(&cmd->eh_entry, &sas_ha->eh_ata_q);
+ return;
+ }
+
/* now finish the command and move it on to the error
* handler done list, this also takes it off the
* error handler pending list.
@@ -237,22 +246,6 @@ static void sas_eh_finish_cmd(struct scsi_cmnd *cmd)
scsi_eh_finish_cmd(cmd, &sas_ha->eh_done_q);
}
-static void sas_eh_defer_cmd(struct scsi_cmnd *cmd)
-{
- struct domain_device *dev = cmd_to_domain_dev(cmd);
- struct sas_ha_struct *ha = dev->port->ha;
- struct sas_task *task = TO_SAS_TASK(cmd);
-
- if (!dev_is_sata(dev)) {
- sas_eh_finish_cmd(cmd);
- return;
- }
-
- /* report the timeout to libata */
- sas_end_task(cmd, task);
- list_move_tail(&cmd->eh_entry, &ha->eh_ata_q);
-}
-
static void sas_scsi_clear_queue_lu(struct list_head *error_q, struct scsi_cmnd *my_cmd)
{
struct scsi_cmnd *cmd, *n;
@@ -260,7 +253,7 @@ static void sas_scsi_clear_queue_lu(struct list_head *error_q, struct scsi_cmnd
list_for_each_entry_safe(cmd, n, error_q, eh_entry) {
if (cmd->device->sdev_target == my_cmd->device->sdev_target &&
cmd->device->lun == my_cmd->device->lun)
- sas_eh_defer_cmd(cmd);
+ sas_eh_finish_cmd(cmd);
}
}
@@ -622,12 +615,12 @@ static void sas_eh_handle_sas_errors(struct Scsi_Host *shost, struct list_head *
case TASK_IS_DONE:
SAS_DPRINTK("%s: task 0x%p is done\n", __func__,
task);
- sas_eh_defer_cmd(cmd);
+ sas_eh_finish_cmd(cmd);
continue;
case TASK_IS_ABORTED:
SAS_DPRINTK("%s: task 0x%p is aborted\n",
__func__, task);
- sas_eh_defer_cmd(cmd);
+ sas_eh_finish_cmd(cmd);
continue;
case TASK_IS_AT_LU:
SAS_DPRINTK("task 0x%p is at LU: lu recover\n", task);
@@ -638,7 +631,7 @@ static void sas_eh_handle_sas_errors(struct Scsi_Host *shost, struct list_head *
"recovered\n",
SAS_ADDR(task->dev),
cmd->device->lun);
- sas_eh_defer_cmd(cmd);
+ sas_eh_finish_cmd(cmd);
sas_scsi_clear_queue_lu(work_q, cmd);
goto Again;
}
diff --git a/drivers/scsi/lpfc/lpfc_attr.c b/drivers/scsi/lpfc/lpfc_attr.c
index 4532990..cf15b97 100644
--- a/drivers/scsi/lpfc/lpfc_attr.c
+++ b/drivers/scsi/lpfc/lpfc_attr.c
@@ -635,7 +635,12 @@ lpfc_issue_lip(struct Scsi_Host *shost)
LPFC_MBOXQ_t *pmboxq;
int mbxstatus = MBXERR_ERROR;
+ /*
+ * If the link is offline, disabled or BLOCK_MGMT_IO
+ * it doesn't make any sense to allow issue_lip
+ */
if ((vport->fc_flag & FC_OFFLINE_MODE) ||
+ (phba->hba_flag & LINK_DISABLED) ||
(phba->sli.sli_flag & LPFC_BLOCK_MGMT_IO))
return -EPERM;
diff --git a/drivers/scsi/lpfc/lpfc_hbadisc.c b/drivers/scsi/lpfc/lpfc_hbadisc.c
index 7d2ad63..8173645 100644
--- a/drivers/scsi/lpfc/lpfc_hbadisc.c
+++ b/drivers/scsi/lpfc/lpfc_hbadisc.c
@@ -690,8 +690,9 @@ lpfc_work_done(struct lpfc_hba *phba)
(phba->hba_flag & HBA_SP_QUEUE_EVT)) {
if (pring->flag & LPFC_STOP_IOCB_EVENT) {
pring->flag |= LPFC_DEFERRED_RING_EVENT;
- /* Set the lpfc data pending flag */
- set_bit(LPFC_DATA_READY, &phba->data_flags);
+ /* Preserve legacy behavior. */
+ if (!(phba->hba_flag & HBA_SP_QUEUE_EVT))
+ set_bit(LPFC_DATA_READY, &phba->data_flags);
} else {
if (phba->link_state >= LPFC_LINK_UP) {
pring->flag &= ~LPFC_DEFERRED_RING_EVENT;
diff --git a/drivers/scsi/lpfc/lpfc_sli.c b/drivers/scsi/lpfc/lpfc_sli.c
index 0902ed2..c05fc61 100644
--- a/drivers/scsi/lpfc/lpfc_sli.c
+++ b/drivers/scsi/lpfc/lpfc_sli.c
@@ -116,6 +116,8 @@ lpfc_sli4_wq_put(struct lpfc_queue *q, union lpfc_wqe *wqe)
/* set consumption flag every once in a while */
if (!((q->host_index + 1) % q->entry_repost))
bf_set(wqe_wqec, &wqe->generic.wqe_com, 1);
+ else
+ bf_set(wqe_wqec, &wqe->generic.wqe_com, 0);
if (q->phba->sli3_options & LPFC_SLI4_PHWQ_ENABLED)
bf_set(wqe_wqid, &wqe->generic.wqe_com, q->queue_id);
lpfc_sli_pcimem_bcopy(wqe, temp_wqe, q->entry_size);
@@ -3510,6 +3512,7 @@ lpfc_sli_handle_slow_ring_event_s4(struct lpfc_hba *phba,
struct hbq_dmabuf *dmabuf;
struct lpfc_cq_event *cq_event;
unsigned long iflag;
+ int count = 0;
spin_lock_irqsave(&phba->hbalock, iflag);
phba->hba_flag &= ~HBA_SP_QUEUE_EVT;
@@ -3531,16 +3534,22 @@ lpfc_sli_handle_slow_ring_event_s4(struct lpfc_hba *phba,
if (irspiocbq)
lpfc_sli_sp_handle_rspiocb(phba, pring,
irspiocbq);
+ count++;
break;
case CQE_CODE_RECEIVE:
case CQE_CODE_RECEIVE_V1:
dmabuf = container_of(cq_event, struct hbq_dmabuf,
cq_event);
lpfc_sli4_handle_received_buffer(phba, dmabuf);
+ count++;
break;
default:
break;
}
+
+ /* Limit the number of events to 64 to avoid soft lockups */
+ if (count == 64)
+ break;
}
}
diff --git a/drivers/scsi/mac_esp.c b/drivers/scsi/mac_esp.c
index 26c67c4..1002124 100644
--- a/drivers/scsi/mac_esp.c
+++ b/drivers/scsi/mac_esp.c
@@ -426,6 +426,8 @@ static void mac_esp_send_pio_cmd(struct esp *esp, u32 addr, u32 esp_count,
scsi_esp_cmd(esp, ESP_CMD_TI);
}
}
+
+ esp->send_cmd_residual = esp_count;
}
static int mac_esp_irq_pending(struct esp *esp)
diff --git a/drivers/scsi/megaraid.c b/drivers/scsi/megaraid.c
index 9d05302..19bffe0b 100644
--- a/drivers/scsi/megaraid.c
+++ b/drivers/scsi/megaraid.c
@@ -4197,6 +4197,9 @@ megaraid_probe_one(struct pci_dev *pdev, const struct pci_device_id *id)
int irq, i, j;
int error = -ENODEV;
+ if (hba_count >= MAX_CONTROLLERS)
+ goto out;
+
if (pci_enable_device(pdev))
goto out;
pci_set_master(pdev);
diff --git a/drivers/scsi/megaraid/megaraid_sas_base.c b/drivers/scsi/megaraid/megaraid_sas_base.c
index 35cbd36..5de024a 100644
--- a/drivers/scsi/megaraid/megaraid_sas_base.c
+++ b/drivers/scsi/megaraid/megaraid_sas_base.c
@@ -6193,6 +6193,9 @@ megasas_resume(struct pci_dev *pdev)
goto fail_init_mfi;
}
+ if (megasas_get_ctrl_info(instance) != DCMD_SUCCESS)
+ goto fail_init_mfi;
+
tasklet_init(&instance->isr_tasklet, instance->instancet->tasklet,
(unsigned long)instance);
@@ -6898,6 +6901,9 @@ static int megasas_mgmt_compat_ioctl_fw(struct file *file, unsigned long arg)
get_user(user_sense_off, &cioc->sense_off))
return -EFAULT;
+ if (local_sense_off != user_sense_off)
+ return -EINVAL;
+
if (local_sense_len) {
void __user **sense_ioc_ptr =
(void __user **)((u8 *)((unsigned long)&ioc->frame.raw) + local_sense_off);
diff --git a/drivers/scsi/megaraid/megaraid_sas_fusion.c b/drivers/scsi/megaraid/megaraid_sas_fusion.c
index a156451..f722a0e 100644
--- a/drivers/scsi/megaraid/megaraid_sas_fusion.c
+++ b/drivers/scsi/megaraid/megaraid_sas_fusion.c
@@ -2031,6 +2031,9 @@ megasas_build_syspd_fusion(struct megasas_instance *instance,
pRAID_Context->timeoutValue = cpu_to_le16(os_timeout_value);
pRAID_Context->VirtualDiskTgtId = cpu_to_le16(device_id);
} else {
+ if (os_timeout_value)
+ os_timeout_value++;
+
/* system pd Fast Path */
io_request->Function = MPI2_FUNCTION_SCSI_IO_REQUEST;
timeout_limit = (scmd->device->type == TYPE_DISK) ?
diff --git a/drivers/scsi/mpt3sas/mpt3sas_scsih.c b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
index 44da9d8..caa0045 100644
--- a/drivers/scsi/mpt3sas/mpt3sas_scsih.c
+++ b/drivers/scsi/mpt3sas/mpt3sas_scsih.c
@@ -8853,7 +8853,7 @@ _scsih_probe(struct pci_dev *pdev, const struct pci_device_id *id)
snprintf(ioc->firmware_event_name, sizeof(ioc->firmware_event_name),
"fw_event_%s%d", ioc->driver_name, ioc->id);
ioc->firmware_event_thread = alloc_ordered_workqueue(
- ioc->firmware_event_name, WQ_MEM_RECLAIM);
+ ioc->firmware_event_name, 0);
if (!ioc->firmware_event_thread) {
pr_err(MPT3SAS_FMT "failure at %s:%d/%s()!\n",
ioc->name, __FILE__, __LINE__, __func__);
diff --git a/drivers/scsi/mvsas/mv_94xx.c b/drivers/scsi/mvsas/mv_94xx.c
index 7de5d8d..eb5471b 100644
--- a/drivers/scsi/mvsas/mv_94xx.c
+++ b/drivers/scsi/mvsas/mv_94xx.c
@@ -1080,16 +1080,16 @@ static int mvs_94xx_gpio_write(struct mvs_prv_info *mvs_prv,
void __iomem *regs = mvi->regs_ex - 0x10200;
int drive = (i/3) & (4-1); /* drive number on host */
- u32 block = mr32(MVS_SGPIO_DCTRL +
+ int driveshift = drive * 8; /* bit offset of drive */
+ u32 block = ioread32be(regs + MVS_SGPIO_DCTRL +
MVS_SGPIO_HOST_OFFSET * mvi->id);
-
/*
* if bit is set then create a mask with the first
* bit of the drive set in the mask ...
*/
- u32 bit = (write_data[i/8] & (1 << (i&(8-1)))) ?
- 1<<(24-drive*8) : 0;
+ u32 bit = get_unaligned_be32(write_data) & (1 << i) ?
+ 1 << driveshift : 0;
/*
* ... and then shift it to the right position based
@@ -1098,26 +1098,27 @@ static int mvs_94xx_gpio_write(struct mvs_prv_info *mvs_prv,
switch (i%3) {
case 0: /* activity */
block &= ~((0x7 << MVS_SGPIO_DCTRL_ACT_SHIFT)
- << (24-drive*8));
+ << driveshift);
/* hardwire activity bit to SOF */
block |= LED_BLINKA_SOF << (
MVS_SGPIO_DCTRL_ACT_SHIFT +
- (24-drive*8));
+ driveshift);
break;
case 1: /* id */
block &= ~((0x3 << MVS_SGPIO_DCTRL_LOC_SHIFT)
- << (24-drive*8));
+ << driveshift);
block |= bit << MVS_SGPIO_DCTRL_LOC_SHIFT;
break;
case 2: /* fail */
block &= ~((0x7 << MVS_SGPIO_DCTRL_ERR_SHIFT)
- << (24-drive*8));
+ << driveshift);
block |= bit << MVS_SGPIO_DCTRL_ERR_SHIFT;
break;
}
- mw32(MVS_SGPIO_DCTRL + MVS_SGPIO_HOST_OFFSET * mvi->id,
- block);
+ iowrite32be(block,
+ regs + MVS_SGPIO_DCTRL +
+ MVS_SGPIO_HOST_OFFSET * mvi->id);
}
@@ -1132,7 +1133,7 @@ static int mvs_94xx_gpio_write(struct mvs_prv_info *mvs_prv,
void __iomem *regs = mvi->regs_ex - 0x10200;
mw32(MVS_SGPIO_DCTRL + MVS_SGPIO_HOST_OFFSET * mvi->id,
- be32_to_cpu(((u32 *) write_data)[i]));
+ ((u32 *) write_data)[i]);
}
return reg_count;
}
diff --git a/drivers/scsi/qla2xxx/qla_init.c b/drivers/scsi/qla2xxx/qla_init.c
index 4441a55..2e3a70a 100644
--- a/drivers/scsi/qla2xxx/qla_init.c
+++ b/drivers/scsi/qla2xxx/qla_init.c
@@ -329,11 +329,10 @@ qla2x00_async_tm_cmd(fc_port_t *fcport, uint32_t flags, uint32_t lun,
wait_for_completion(&tm_iocb->u.tmf.comp);
- rval = tm_iocb->u.tmf.comp_status == CS_COMPLETE ?
- QLA_SUCCESS : QLA_FUNCTION_FAILED;
+ rval = tm_iocb->u.tmf.data;
- if ((rval != QLA_SUCCESS) || tm_iocb->u.tmf.data) {
- ql_dbg(ql_dbg_taskm, vha, 0x8030,
+ if (rval != QLA_SUCCESS) {
+ ql_log(ql_log_warn, vha, 0x8030,
"TM IOCB failed (%x).\n", rval);
}
@@ -369,8 +368,8 @@ qla24xx_abort_sp_done(void *data, void *ptr, int res)
srb_t *sp = (srb_t *)ptr;
struct srb_iocb *abt = &sp->u.iocb_cmd;
- del_timer(&sp->u.iocb_cmd.timer);
- complete(&abt->u.abt.comp);
+ if (del_timer(&sp->u.iocb_cmd.timer))
+ complete(&abt->u.abt.comp);
}
static int
@@ -3319,7 +3318,8 @@ qla2x00_iidma_fcport(scsi_qla_host_t *vha, fc_port_t *fcport)
return;
if (fcport->fp_speed == PORT_SPEED_UNKNOWN ||
- fcport->fp_speed > ha->link_data_rate)
+ fcport->fp_speed > ha->link_data_rate ||
+ !ha->flags.gpsc_supported)
return;
rval = qla2x00_set_idma_speed(vha, fcport->loop_id, fcport->fp_speed,
@@ -4894,7 +4894,7 @@ qla2x00_abort_isp(scsi_qla_host_t *vha)
* The next call disables the board
* completely.
*/
- ha->isp_ops->reset_adapter(vha);
+ qla2x00_abort_isp_cleanup(vha);
vha->flags.online = 0;
clear_bit(ISP_ABORT_RETRY,
&vha->dpc_flags);
diff --git a/drivers/scsi/qla2xxx/qla_isr.c b/drivers/scsi/qla2xxx/qla_isr.c
index bddaabb..73c99f2 100644
--- a/drivers/scsi/qla2xxx/qla_isr.c
+++ b/drivers/scsi/qla2xxx/qla_isr.c
@@ -272,7 +272,8 @@ qla2x00_mbx_completion(scsi_qla_host_t *vha, uint16_t mb0)
struct device_reg_2xxx __iomem *reg = &ha->iobase->isp;
/* Read all mbox registers? */
- mboxes = (1 << ha->mbx_count) - 1;
+ WARN_ON_ONCE(ha->mbx_count > 32);
+ mboxes = (1ULL << ha->mbx_count) - 1;
if (!ha->mcp)
ql_dbg(ql_dbg_async, vha, 0x5001, "MBX pointer ERROR.\n");
else
@@ -2516,7 +2517,8 @@ qla24xx_mbx_completion(scsi_qla_host_t *vha, uint16_t mb0)
struct device_reg_24xx __iomem *reg = &ha->iobase->isp24;
/* Read all mbox registers? */
- mboxes = (1 << ha->mbx_count) - 1;
+ WARN_ON_ONCE(ha->mbx_count > 32);
+ mboxes = (1ULL << ha->mbx_count) - 1;
if (!ha->mcp)
ql_dbg(ql_dbg_async, vha, 0x504e, "MBX pointer ERROR.\n");
else
diff --git a/drivers/scsi/qla2xxx/qla_mbx.c b/drivers/scsi/qla2xxx/qla_mbx.c
index a1b01d6..bf29ad4 100644
--- a/drivers/scsi/qla2xxx/qla_mbx.c
+++ b/drivers/scsi/qla2xxx/qla_mbx.c
@@ -3580,10 +3580,7 @@ qla2x00_set_idma_speed(scsi_qla_host_t *vha, uint16_t loop_id,
mcp->mb[0] = MBC_PORT_PARAMS;
mcp->mb[1] = loop_id;
mcp->mb[2] = BIT_0;
- if (IS_CNA_CAPABLE(vha->hw))
- mcp->mb[3] = port_speed & (BIT_5|BIT_4|BIT_3|BIT_2|BIT_1|BIT_0);
- else
- mcp->mb[3] = port_speed & (BIT_2|BIT_1|BIT_0);
+ mcp->mb[3] = port_speed & (BIT_5|BIT_4|BIT_3|BIT_2|BIT_1|BIT_0);
mcp->mb[9] = vha->vp_idx;
mcp->out_mb = MBX_9|MBX_3|MBX_2|MBX_1|MBX_0;
mcp->in_mb = MBX_3|MBX_1|MBX_0;
diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c
index baccd116f..c813c9b 100644
--- a/drivers/scsi/qla2xxx/qla_os.c
+++ b/drivers/scsi/qla2xxx/qla_os.c
@@ -5218,8 +5218,9 @@ qla2x00_do_dpc(void *data)
}
}
- if (test_and_clear_bit(ISP_ABORT_NEEDED,
- &base_vha->dpc_flags)) {
+ if (test_and_clear_bit
+ (ISP_ABORT_NEEDED, &base_vha->dpc_flags) &&
+ !test_bit(UNLOADING, &base_vha->dpc_flags)) {
ql_dbg(ql_dbg_dpc, base_vha, 0x4007,
"ISP abort scheduled.\n");
diff --git a/drivers/scsi/qla2xxx/qla_target.h b/drivers/scsi/qla2xxx/qla_target.h
index 0824a81..07ea4fc 100644
--- a/drivers/scsi/qla2xxx/qla_target.h
+++ b/drivers/scsi/qla2xxx/qla_target.h
@@ -440,8 +440,8 @@ struct atio_from_isp {
static inline int fcpcmd_is_corrupted(struct atio *atio)
{
if (atio->entry_type == ATIO_TYPE7 &&
- (le16_to_cpu(atio->attr_n_length & FCP_CMD_LENGTH_MASK) <
- FCP_CMD_LENGTH_MIN))
+ ((le16_to_cpu(atio->attr_n_length) & FCP_CMD_LENGTH_MASK) <
+ FCP_CMD_LENGTH_MIN))
return 1;
else
return 0;
diff --git a/drivers/scsi/qla4xxx/ql4_def.h b/drivers/scsi/qla4xxx/ql4_def.h
index a7cfc27..ce1d063f 100644
--- a/drivers/scsi/qla4xxx/ql4_def.h
+++ b/drivers/scsi/qla4xxx/ql4_def.h
@@ -168,6 +168,8 @@
#define DEV_DB_NON_PERSISTENT 0
#define DEV_DB_PERSISTENT 1
+#define QL4_ISP_REG_DISCONNECT 0xffffffffU
+
#define COPY_ISID(dst_isid, src_isid) { \
int i, j; \
for (i = 0, j = ISID_SIZE - 1; i < ISID_SIZE;) \
diff --git a/drivers/scsi/qla4xxx/ql4_os.c b/drivers/scsi/qla4xxx/ql4_os.c
index 01c3610..d8c0343 100644
--- a/drivers/scsi/qla4xxx/ql4_os.c
+++ b/drivers/scsi/qla4xxx/ql4_os.c
@@ -262,6 +262,24 @@ static struct iscsi_transport qla4xxx_iscsi_transport = {
static struct scsi_transport_template *qla4xxx_scsi_transport;
+static int qla4xxx_isp_check_reg(struct scsi_qla_host *ha)
+{
+ u32 reg_val = 0;
+ int rval = QLA_SUCCESS;
+
+ if (is_qla8022(ha))
+ reg_val = readl(&ha->qla4_82xx_reg->host_status);
+ else if (is_qla8032(ha) || is_qla8042(ha))
+ reg_val = qla4_8xxx_rd_direct(ha, QLA8XXX_PEG_ALIVE_COUNTER);
+ else
+ reg_val = readw(&ha->reg->ctrl_status);
+
+ if (reg_val == QL4_ISP_REG_DISCONNECT)
+ rval = QLA_ERROR;
+
+ return rval;
+}
+
static int qla4xxx_send_ping(struct Scsi_Host *shost, uint32_t iface_num,
uint32_t iface_type, uint32_t payload_size,
uint32_t pid, struct sockaddr *dst_addr)
@@ -9196,10 +9214,17 @@ static int qla4xxx_eh_abort(struct scsi_cmnd *cmd)
struct srb *srb = NULL;
int ret = SUCCESS;
int wait = 0;
+ int rval;
ql4_printk(KERN_INFO, ha, "scsi%ld:%d:%llu: Abort command issued cmd=%p, cdb=0x%x\n",
ha->host_no, id, lun, cmd, cmd->cmnd[0]);
+ rval = qla4xxx_isp_check_reg(ha);
+ if (rval != QLA_SUCCESS) {
+ ql4_printk(KERN_INFO, ha, "PCI/Register disconnect, exiting.\n");
+ return FAILED;
+ }
+
spin_lock_irqsave(&ha->hardware_lock, flags);
srb = (struct srb *) CMD_SP(cmd);
if (!srb) {
@@ -9251,6 +9276,7 @@ static int qla4xxx_eh_device_reset(struct scsi_cmnd *cmd)
struct scsi_qla_host *ha = to_qla_host(cmd->device->host);
struct ddb_entry *ddb_entry = cmd->device->hostdata;
int ret = FAILED, stat;
+ int rval;
if (!ddb_entry)
return ret;
@@ -9270,6 +9296,12 @@ static int qla4xxx_eh_device_reset(struct scsi_cmnd *cmd)
cmd, jiffies, cmd->request->timeout / HZ,
ha->dpc_flags, cmd->result, cmd->allowed));
+ rval = qla4xxx_isp_check_reg(ha);
+ if (rval != QLA_SUCCESS) {
+ ql4_printk(KERN_INFO, ha, "PCI/Register disconnect, exiting.\n");
+ return FAILED;
+ }
+
/* FIXME: wait for hba to go online */
stat = qla4xxx_reset_lun(ha, ddb_entry, cmd->device->lun);
if (stat != QLA_SUCCESS) {
@@ -9313,6 +9345,7 @@ static int qla4xxx_eh_target_reset(struct scsi_cmnd *cmd)
struct scsi_qla_host *ha = to_qla_host(cmd->device->host);
struct ddb_entry *ddb_entry = cmd->device->hostdata;
int stat, ret;
+ int rval;
if (!ddb_entry)
return FAILED;
@@ -9330,6 +9363,12 @@ static int qla4xxx_eh_target_reset(struct scsi_cmnd *cmd)
ha->host_no, cmd, jiffies, cmd->request->timeout / HZ,
ha->dpc_flags, cmd->result, cmd->allowed));
+ rval = qla4xxx_isp_check_reg(ha);
+ if (rval != QLA_SUCCESS) {
+ ql4_printk(KERN_INFO, ha, "PCI/Register disconnect, exiting.\n");
+ return FAILED;
+ }
+
stat = qla4xxx_reset_target(ha, ddb_entry);
if (stat != QLA_SUCCESS) {
starget_printk(KERN_INFO, scsi_target(cmd->device),
@@ -9384,9 +9423,16 @@ static int qla4xxx_eh_host_reset(struct scsi_cmnd *cmd)
{
int return_status = FAILED;
struct scsi_qla_host *ha;
+ int rval;
ha = to_qla_host(cmd->device->host);
+ rval = qla4xxx_isp_check_reg(ha);
+ if (rval != QLA_SUCCESS) {
+ ql4_printk(KERN_INFO, ha, "PCI/Register disconnect, exiting.\n");
+ return FAILED;
+ }
+
if ((is_qla8032(ha) || is_qla8042(ha)) && ql4xdontresethba)
qla4_83xx_set_idc_dontreset(ha);
diff --git a/drivers/scsi/scsi_devinfo.c b/drivers/scsi/scsi_devinfo.c
index 43d4b30..282ea00 100644
--- a/drivers/scsi/scsi_devinfo.c
+++ b/drivers/scsi/scsi_devinfo.c
@@ -33,7 +33,6 @@ struct scsi_dev_info_list_table {
};
-static const char spaces[] = " "; /* 16 of them */
static unsigned scsi_default_dev_flags;
static LIST_HEAD(scsi_dev_info_list);
static char scsi_dev_flags[256];
@@ -298,20 +297,13 @@ static void scsi_strcpy_devinfo(char *name, char *to, size_t to_length,
size_t from_length;
from_length = strlen(from);
- strncpy(to, from, min(to_length, from_length));
- if (from_length < to_length) {
- if (compatible) {
- /*
- * NUL terminate the string if it is short.
- */
- to[from_length] = '\0';
- } else {
- /*
- * space pad the string if it is short.
- */
- strncpy(&to[from_length], spaces,
- to_length - from_length);
- }
+ /* this zero-pads the destination */
+ strncpy(to, from, to_length);
+ if (from_length < to_length && !compatible) {
+ /*
+ * space pad the string if it is short.
+ */
+ memset(&to[from_length], ' ', to_length - from_length);
}
if (from_length > to_length)
printk(KERN_WARNING "%s: %s string '%s' is too long\n",
diff --git a/drivers/scsi/scsi_dh.c b/drivers/scsi/scsi_dh.c
index a5e30e9..375cede 100644
--- a/drivers/scsi/scsi_dh.c
+++ b/drivers/scsi/scsi_dh.c
@@ -58,7 +58,10 @@ static const struct scsi_dh_blist scsi_dh_blist[] = {
{"IBM", "3526", "rdac", },
{"IBM", "3542", "rdac", },
{"IBM", "3552", "rdac", },
- {"SGI", "TP9", "rdac", },
+ {"SGI", "TP9300", "rdac", },
+ {"SGI", "TP9400", "rdac", },
+ {"SGI", "TP9500", "rdac", },
+ {"SGI", "TP9700", "rdac", },
{"SGI", "IS", "rdac", },
{"STK", "OPENstorage", "rdac", },
{"STK", "FLEXLINE 380", "rdac", },
diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index 3ccc858..6338c4d 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -2046,6 +2046,7 @@ static void __scsi_init_queue(struct Scsi_Host *shost, struct request_queue *q)
if (shost->inlinecrypt_support)
queue_flag_set_unlocked(QUEUE_FLAG_INLINECRYPT, q);
+
/*
* Set a reasonable default alignment: The larger of 32-byte (dword),
* which is a common minimum for HBAs, and the minimum DMA alignment,
diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c
index f14d95e..5294fa3 100644
--- a/drivers/scsi/scsi_sysfs.c
+++ b/drivers/scsi/scsi_sysfs.c
@@ -709,8 +709,24 @@ static ssize_t
sdev_store_delete(struct device *dev, struct device_attribute *attr,
const char *buf, size_t count)
{
- if (device_remove_file_self(dev, attr))
- scsi_remove_device(to_scsi_device(dev));
+ struct kernfs_node *kn;
+
+ kn = sysfs_break_active_protection(&dev->kobj, &attr->attr);
+ WARN_ON_ONCE(!kn);
+ /*
+ * Concurrent writes into the "delete" sysfs attribute may trigger
+ * concurrent calls to device_remove_file() and scsi_remove_device().
+ * device_remove_file() handles concurrent removal calls by
+ * serializing these and by ignoring the second and later removal
+ * attempts. Concurrent calls of scsi_remove_device() are
+ * serialized. The second and later calls of scsi_remove_device() are
+ * ignored because the first call of that function changes the device
+ * state into SDEV_DEL.
+ */
+ device_remove_file(dev, attr);
+ scsi_remove_device(to_scsi_device(dev));
+ if (kn)
+ sysfs_unbreak_active_protection(kn);
return count;
};
static DEVICE_ATTR(delete, S_IWUSR, NULL, sdev_store_delete);
diff --git a/drivers/scsi/scsi_transport_srp.c b/drivers/scsi/scsi_transport_srp.c
index e3cd3ec..c3d1891 100644
--- a/drivers/scsi/scsi_transport_srp.c
+++ b/drivers/scsi/scsi_transport_srp.c
@@ -52,6 +52,8 @@ struct srp_internal {
struct transport_container rport_attr_cont;
};
+static int scsi_is_srp_rport(const struct device *dev);
+
#define to_srp_internal(tmpl) container_of(tmpl, struct srp_internal, t)
#define dev_to_rport(d) container_of(d, struct srp_rport, dev)
@@ -61,9 +63,24 @@ static inline struct Scsi_Host *rport_to_shost(struct srp_rport *r)
return dev_to_shost(r->dev.parent);
}
+static int find_child_rport(struct device *dev, void *data)
+{
+ struct device **child = data;
+
+ if (scsi_is_srp_rport(dev)) {
+ WARN_ON_ONCE(*child);
+ *child = dev;
+ }
+ return 0;
+}
+
static inline struct srp_rport *shost_to_rport(struct Scsi_Host *shost)
{
- return transport_class_to_srp_rport(&shost->shost_gendev);
+ struct device *child = NULL;
+
+ WARN_ON_ONCE(device_for_each_child(&shost->shost_gendev, &child,
+ find_child_rport) < 0);
+ return child ? dev_to_rport(child) : NULL;
}
/**
@@ -637,7 +654,8 @@ static enum blk_eh_timer_return srp_timed_out(struct scsi_cmnd *scmd)
struct srp_rport *rport = shost_to_rport(shost);
pr_debug("timeout for sdev %s\n", dev_name(&sdev->sdev_gendev));
- return rport->fast_io_fail_tmo < 0 && rport->dev_loss_tmo < 0 &&
+ return rport && rport->fast_io_fail_tmo < 0 &&
+ rport->dev_loss_tmo < 0 &&
i->f->reset_timer_if_blocked && scsi_device_blocked(sdev) ?
BLK_EH_RESET_TIMER : BLK_EH_NOT_HANDLED;
}
diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index 4fb494a..1d0351b 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -1164,7 +1164,8 @@ static int sd_init_command(struct scsi_cmnd *cmd)
case REQ_OP_WRITE:
return sd_setup_read_write_cmnd(cmd);
default:
- BUG();
+ WARN_ON_ONCE(1);
+ return BLKPREP_KILL;
}
}
@@ -1860,6 +1861,8 @@ sd_spinup_disk(struct scsi_disk *sdkp)
break; /* standby */
if (sshdr.asc == 4 && sshdr.ascq == 0xc)
break; /* unavailable */
+ if (sshdr.asc == 4 && sshdr.ascq == 0x1b)
+ break; /* sanitize in progress */
/*
* Issue command to spin up drive when not ready
*/
@@ -2319,6 +2322,7 @@ sd_read_write_protect_flag(struct scsi_disk *sdkp, unsigned char *buffer)
int res;
struct scsi_device *sdp = sdkp->device;
struct scsi_mode_data data;
+ int disk_ro = get_disk_ro(sdkp->disk);
set_disk_ro(sdkp->disk, 0);
if (sdp->skip_ms_page_3f) {
@@ -2358,7 +2362,7 @@ sd_read_write_protect_flag(struct scsi_disk *sdkp, unsigned char *buffer)
"Test WP failed, assume Write Enabled\n");
} else {
sdkp->write_prot = ((data.device_specific & 0x80) != 0);
- set_disk_ro(sdkp->disk, sdkp->write_prot);
+ set_disk_ro(sdkp->disk, sdkp->write_prot || disk_ro);
}
}
diff --git a/drivers/scsi/sg.c b/drivers/scsi/sg.c
index 3d9ad4c..edc6362 100644
--- a/drivers/scsi/sg.c
+++ b/drivers/scsi/sg.c
@@ -51,6 +51,7 @@ static int sg_version_num = 30536; /* 2 digits for each component */
#include <linux/atomic.h>
#include <linux/ratelimit.h>
#include <linux/uio.h>
+#include <linux/cred.h> /* for sg_check_file_access() */
#include "scsi.h"
#include <scsi/scsi_dbg.h>
@@ -210,6 +211,33 @@ static void sg_device_destroy(struct kref *kref);
sdev_prefix_printk(prefix, (sdp)->device, \
(sdp)->disk->disk_name, fmt, ##a)
+/*
+ * The SCSI interfaces that use read() and write() as an asynchronous variant of
+ * ioctl(..., SG_IO, ...) are fundamentally unsafe, since there are lots of ways
+ * to trigger read() and write() calls from various contexts with elevated
+ * privileges. This can lead to kernel memory corruption (e.g. if these
+ * interfaces are called through splice()) and privilege escalation inside
+ * userspace (e.g. if a process with access to such a device passes a file
+ * descriptor to a SUID binary as stdin/stdout/stderr).
+ *
+ * This function provides protection for the legacy API by restricting the
+ * calling context.
+ */
+static int sg_check_file_access(struct file *filp, const char *caller)
+{
+ if (filp->f_cred != current_real_cred()) {
+ pr_err_once("%s: process %d (%s) changed security contexts after opening file descriptor, this is not allowed.\n",
+ caller, task_tgid_vnr(current), current->comm);
+ return -EPERM;
+ }
+ if (unlikely(segment_eq(get_fs(), KERNEL_DS))) {
+ pr_err_once("%s: process %d (%s) called from kernel context, this is not allowed.\n",
+ caller, task_tgid_vnr(current), current->comm);
+ return -EACCES;
+ }
+ return 0;
+}
+
static int sg_allow_access(struct file *filp, unsigned char *cmd)
{
struct sg_fd *sfp = filp->private_data;
@@ -394,6 +422,14 @@ sg_read(struct file *filp, char __user *buf, size_t count, loff_t * ppos)
struct sg_header *old_hdr = NULL;
int retval = 0;
+ /*
+ * This could cause a response to be stranded. Close the associated
+ * file descriptor to free up any resources being held.
+ */
+ retval = sg_check_file_access(filp, __func__);
+ if (retval)
+ return retval;
+
if ((!(sfp = (Sg_fd *) filp->private_data)) || (!(sdp = sfp->parentdp)))
return -ENXIO;
SCSI_LOG_TIMEOUT(3, sg_printk(KERN_INFO, sdp,
@@ -581,9 +617,11 @@ sg_write(struct file *filp, const char __user *buf, size_t count, loff_t * ppos)
struct sg_header old_hdr;
sg_io_hdr_t *hp;
unsigned char cmnd[SG_MAX_CDB_SIZE];
+ int retval;
- if (unlikely(segment_eq(get_fs(), KERNEL_DS)))
- return -EINVAL;
+ retval = sg_check_file_access(filp, __func__);
+ if (retval)
+ return retval;
if ((!(sfp = (Sg_fd *) filp->private_data)) || (!(sdp = sfp->parentdp)))
return -ENXIO;
@@ -1897,7 +1935,7 @@ sg_build_indirect(Sg_scatter_hold * schp, Sg_fd * sfp, int buff_size)
num = (rem_sz > scatter_elem_sz_prev) ?
scatter_elem_sz_prev : rem_sz;
- schp->pages[k] = alloc_pages(gfp_mask, order);
+ schp->pages[k] = alloc_pages(gfp_mask | __GFP_ZERO, order);
if (!schp->pages[k])
goto out;
@@ -2151,6 +2189,7 @@ sg_add_sfp(Sg_device * sdp)
write_lock_irqsave(&sdp->sfd_lock, iflags);
if (atomic_read(&sdp->detaching)) {
write_unlock_irqrestore(&sdp->sfd_lock, iflags);
+ kfree(sfp);
return ERR_PTR(-ENODEV);
}
list_add_tail(&sfp->sfd_siblings, &sdp->sfds);
diff --git a/drivers/scsi/sr.c b/drivers/scsi/sr.c
index e635973..cc484cb 100644
--- a/drivers/scsi/sr.c
+++ b/drivers/scsi/sr.c
@@ -520,16 +520,26 @@ static int sr_init_command(struct scsi_cmnd *SCpnt)
static int sr_block_open(struct block_device *bdev, fmode_t mode)
{
struct scsi_cd *cd;
+ struct scsi_device *sdev;
int ret = -ENXIO;
- mutex_lock(&sr_mutex);
cd = scsi_cd_get(bdev->bd_disk);
- if (cd) {
- ret = cdrom_open(&cd->cdi, bdev, mode);
- if (ret)
- scsi_cd_put(cd);
- }
+ if (!cd)
+ goto out;
+
+ sdev = cd->device;
+ scsi_autopm_get_device(sdev);
+ check_disk_change(bdev);
+
+ mutex_lock(&sr_mutex);
+ ret = cdrom_open(&cd->cdi, bdev, mode);
mutex_unlock(&sr_mutex);
+
+ scsi_autopm_put_device(sdev);
+ if (ret)
+ scsi_cd_put(cd);
+
+out:
return ret;
}
@@ -557,6 +567,8 @@ static int sr_block_ioctl(struct block_device *bdev, fmode_t mode, unsigned cmd,
if (ret)
goto out;
+ scsi_autopm_get_device(sdev);
+
/*
* Send SCSI addressing ioctls directly to mid level, send other
* ioctls to cdrom/block level.
@@ -565,15 +577,18 @@ static int sr_block_ioctl(struct block_device *bdev, fmode_t mode, unsigned cmd,
case SCSI_IOCTL_GET_IDLUN:
case SCSI_IOCTL_GET_BUS_NUMBER:
ret = scsi_ioctl(sdev, cmd, argp);
- goto out;
+ goto put;
}
ret = cdrom_ioctl(&cd->cdi, bdev, mode, cmd, arg);
if (ret != -ENOSYS)
- goto out;
+ goto put;
ret = scsi_ioctl(sdev, cmd, argp);
+put:
+ scsi_autopm_put_device(sdev);
+
out:
mutex_unlock(&sr_mutex);
return ret;
@@ -582,18 +597,28 @@ static int sr_block_ioctl(struct block_device *bdev, fmode_t mode, unsigned cmd,
static unsigned int sr_block_check_events(struct gendisk *disk,
unsigned int clearing)
{
- struct scsi_cd *cd = scsi_cd(disk);
+ unsigned int ret = 0;
+ struct scsi_cd *cd;
- if (atomic_read(&cd->device->disk_events_disable_depth))
+ cd = scsi_cd_get(disk);
+ if (!cd)
return 0;
- return cdrom_check_events(&cd->cdi, clearing);
+ if (!atomic_read(&cd->device->disk_events_disable_depth))
+ ret = cdrom_check_events(&cd->cdi, clearing);
+
+ scsi_cd_put(cd);
+ return ret;
}
static int sr_block_revalidate_disk(struct gendisk *disk)
{
- struct scsi_cd *cd = scsi_cd(disk);
struct scsi_sense_hdr sshdr;
+ struct scsi_cd *cd;
+
+ cd = scsi_cd_get(disk);
+ if (!cd)
+ return -ENXIO;
/* if the unit is not ready, nothing more to do */
if (scsi_test_unit_ready(cd->device, SR_TIMEOUT, MAX_RETRIES, &sshdr))
@@ -602,6 +627,7 @@ static int sr_block_revalidate_disk(struct gendisk *disk)
sr_cd_check(&cd->cdi);
get_sectorsize(cd);
out:
+ scsi_cd_put(cd);
return 0;
}
diff --git a/drivers/scsi/sr_ioctl.c b/drivers/scsi/sr_ioctl.c
index 03054c0..3c3e811 100644
--- a/drivers/scsi/sr_ioctl.c
+++ b/drivers/scsi/sr_ioctl.c
@@ -187,30 +187,25 @@ int sr_do_ioctl(Scsi_CD *cd, struct packet_command *cgc)
struct scsi_device *SDev;
struct scsi_sense_hdr sshdr;
int result, err = 0, retries = 0;
- struct request_sense *sense = cgc->sense;
+ unsigned char sense_buffer[SCSI_SENSE_BUFFERSIZE];
SDev = cd->device;
- if (!sense) {
- sense = kmalloc(SCSI_SENSE_BUFFERSIZE, GFP_KERNEL);
- if (!sense) {
- err = -ENOMEM;
- goto out;
- }
- }
-
retry:
if (!scsi_block_when_processing_errors(SDev)) {
err = -ENODEV;
goto out;
}
- memset(sense, 0, sizeof(*sense));
+ memset(sense_buffer, 0, sizeof(sense_buffer));
result = scsi_execute(SDev, cgc->cmd, cgc->data_direction,
- cgc->buffer, cgc->buflen, (char *)sense,
+ cgc->buffer, cgc->buflen, sense_buffer,
cgc->timeout, IOCTL_RETRIES, 0, NULL);
- scsi_normalize_sense((char *)sense, sizeof(*sense), &sshdr);
+ scsi_normalize_sense(sense_buffer, sizeof(sense_buffer), &sshdr);
+
+ if (cgc->sense)
+ memcpy(cgc->sense, sense_buffer, sizeof(*cgc->sense));
/* Minimal error checking. Ignore cases we know about, and report the rest. */
if (driver_byte(result) != 0) {
@@ -261,8 +256,6 @@ int sr_do_ioctl(Scsi_CD *cd, struct packet_command *cgc)
/* Wake up a process waiting for device */
out:
- if (!cgc->sense)
- kfree(sense);
cgc->stat = err;
return err;
}
diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
index 0dd1984..d92b280 100644
--- a/drivers/scsi/storvsc_drv.c
+++ b/drivers/scsi/storvsc_drv.c
@@ -1580,7 +1580,7 @@ static struct scsi_host_template scsi_driver = {
.eh_timed_out = storvsc_eh_timed_out,
.slave_alloc = storvsc_device_alloc,
.slave_configure = storvsc_device_configure,
- .cmd_per_lun = 255,
+ .cmd_per_lun = 2048,
.this_id = -1,
.use_clustering = ENABLE_CLUSTERING,
/* Make sure we dont get a sg segment crosses a page boundary */
diff --git a/drivers/scsi/sym53c8xx_2/sym_hipd.c b/drivers/scsi/sym53c8xx_2/sym_hipd.c
index 6b349e3..c6425e3 100644
--- a/drivers/scsi/sym53c8xx_2/sym_hipd.c
+++ b/drivers/scsi/sym53c8xx_2/sym_hipd.c
@@ -536,7 +536,7 @@ sym_getsync(struct sym_hcb *np, u_char dt, u_char sfac, u_char *divp, u_char *fa
* Look for the greatest clock divisor that allows an
* input speed faster than the period.
*/
- while (div-- > 0)
+ while (--div > 0)
if (kpc >= (div_10M[div] << 2)) break;
/*
diff --git a/drivers/scsi/ufs/Kconfig b/drivers/scsi/ufs/Kconfig
index 1b283b2..d4eb52c 100644
--- a/drivers/scsi/ufs/Kconfig
+++ b/drivers/scsi/ufs/Kconfig
@@ -3,6 +3,8 @@
#
# This code is based on drivers/scsi/ufs/Kconfig
# Copyright (C) 2011-2013 Samsung India Software Operations
+# Copyright (c) 2017-2018 Samsung Electronics Co., Ltd.
+# Copyright (C) 2018, Google, Inc.
#
# Authors:
# Santosh Yaraganavi <santosh.sy@samsung.com>
@@ -126,3 +128,24 @@
Select this if you want above mentioned debug information captured.
If unsure, say N.
+
+config SCSI_UFSHCD_HPB_ACTIVATE
+ int "Activate HPB Host-aware Performance Booster"
+ depends on SCSI_UFSHCD
+ default 1
+ help
+ Activate or deactive UFSHPB v1.3.5 test driver
+
+config SCSI_UFS_IMPAIRED
+ bool "Impaired storage emulation support"
+ depends on SCSI_UFSHCD
+ ---help---
+ This selects UFS driver to support impaired storage emulation.
+ Articificial delay will be added for each UFS command.
+
+config SCSI_UFS_IMPAIRED_FRAGFTL
+ bool "Impaired storage emulation support for fragmented FTL"
+ depends on SCSI_UFSHCD && SCSI_UFS_IMPAIRED
+ ---help---
+ This selects UFS driver to support impaired storage emulation
+ with a latency model from fragmented FTL.
diff --git a/drivers/scsi/ufs/Makefile b/drivers/scsi/ufs/Makefile
index ce98c09..9d92ddb 100644
--- a/drivers/scsi/ufs/Makefile
+++ b/drivers/scsi/ufs/Makefile
@@ -1,8 +1,13 @@
# UFSHCD makefile
obj-$(CONFIG_SCSI_UFS_QCOM) += ufs-qcom.o
obj-$(CONFIG_SCSI_UFS_QCOM_ICE) += ufs-qcom-ice.o
-obj-$(CONFIG_SCSI_UFSHCD) += ufshcd.o ufs_quirks.o
+
+obj-$(CONFIG_SCSI_UFSHCD) += ufshcd-core.o
+ufshcd-core-objs := ufshcd.o ufs_quirks.o ufshpb.o
+
obj-$(CONFIG_SCSI_UFSHCD_PCI) += ufshcd-pci.o
obj-$(CONFIG_SCSI_UFSHCD_PLATFORM) += ufshcd-pltfrm.o
obj-$(CONFIG_SCSI_UFS_TEST) += ufs_test.o
obj-$(CONFIG_DEBUG_FS) += ufs-debugfs.o ufs-qcom-debugfs.o
+
+obj-$(CONFIG_SCSI_UFS_IMPAIRED) += ufs-impaired.o
diff --git a/drivers/scsi/ufs/ufs-debugfs.c b/drivers/scsi/ufs/ufs-debugfs.c
index 1f3967d..9bb781c 100644
--- a/drivers/scsi/ufs/ufs-debugfs.c
+++ b/drivers/scsi/ufs/ufs-debugfs.c
@@ -366,6 +366,7 @@ static const struct file_operations ufsdbg_err_inj_scenario_ops = {
.open = ufsdbg_err_inj_scenario_open,
.read = seq_read,
.write = ufsdbg_err_inj_scenario_write,
+ .release = single_release,
};
static int ufsdbg_err_inj_stats_read(struct seq_file *file, void *data)
@@ -407,6 +408,7 @@ static const struct file_operations ufsdbg_err_inj_stats_ops = {
.open = ufsdbg_err_inj_stats_open,
.read = seq_read,
.write = ufsdbg_err_inj_stats_write,
+ .release = single_release,
};
static void ufsdbg_setup_fault_injection(struct ufs_hba *hba)
@@ -504,8 +506,8 @@ static int ufsdbg_tag_stats_show(struct seq_file *file, void *data)
seq_puts(file, sep);
}
seq_printf(file,
- "\n #\tnum uses\t%s\t #\tAll\tRead\tWrite\tUrg.R\tUrg.W\tFlush\n",
- sep);
+ "\n #\tnum uses\t%s\t #\tAll\tRead\tWrite\tUrg.R\tUrg.W\tFlush"
+ "\tDiscard\n", sep);
/* values */
for (i = 0; i < max_depth; i++) {
@@ -514,7 +516,8 @@ static int ufsdbg_tag_stats_show(struct seq_file *file, void *data)
ufs_stats->tag_stats[i][TS_WRITE] <= 0 &&
ufs_stats->tag_stats[i][TS_URGENT_READ] <= 0 &&
ufs_stats->tag_stats[i][TS_URGENT_WRITE] <= 0 &&
- ufs_stats->tag_stats[i][TS_FLUSH] <= 0)
+ ufs_stats->tag_stats[i][TS_FLUSH] <= 0 &&
+ ufs_stats->tag_stats[i][TS_DISCARD] <= 0)
continue;
is_tag_empty = false;
@@ -528,7 +531,8 @@ static int ufsdbg_tag_stats_show(struct seq_file *file, void *data)
ufs_stats->tag_stats[i][TS_WRITE] +
ufs_stats->tag_stats[i][TS_URGENT_READ] +
ufs_stats->tag_stats[i][TS_URGENT_WRITE] +
- ufs_stats->tag_stats[i][TS_FLUSH]);
+ ufs_stats->tag_stats[i][TS_FLUSH] +
+ ufs_stats->tag_stats[i][TS_DISCARD]);
}
seq_puts(file, "\n");
}
@@ -591,6 +595,7 @@ static const struct file_operations ufsdbg_tag_stats_fops = {
.open = ufsdbg_tag_stats_open,
.read = seq_read,
.write = ufsdbg_tag_stats_write,
+ .release = single_release,
};
static int ufsdbg_query_stats_show(struct seq_file *file, void *data)
@@ -662,6 +667,7 @@ static const struct file_operations ufsdbg_query_stats_fops = {
.open = ufsdbg_query_stats_open,
.read = seq_read,
.write = ufsdbg_query_stats_write,
+ .release = single_release,
};
static int ufsdbg_err_stats_show(struct seq_file *file, void *data)
@@ -766,6 +772,7 @@ static const struct file_operations ufsdbg_err_stats_fops = {
.open = ufsdbg_err_stats_open,
.read = seq_read,
.write = ufsdbg_err_stats_write,
+ .release = single_release,
};
static int ufshcd_init_statistics(struct ufs_hba *hba)
@@ -845,6 +852,7 @@ static int ufsdbg_host_regs_open(struct inode *inode, struct file *file)
static const struct file_operations ufsdbg_host_regs_fops = {
.open = ufsdbg_host_regs_open,
.read = seq_read,
+ .release = single_release,
};
static int ufsdbg_dump_device_desc_show(struct seq_file *file, void *data)
@@ -1008,6 +1016,7 @@ static int ufsdbg_show_hba_open(struct inode *inode, struct file *file)
static const struct file_operations ufsdbg_show_hba_fops = {
.open = ufsdbg_show_hba_open,
.read = seq_read,
+ .release = single_release,
};
static int ufsdbg_dump_device_desc_open(struct inode *inode, struct file *file)
@@ -1019,6 +1028,7 @@ static int ufsdbg_dump_device_desc_open(struct inode *inode, struct file *file)
static const struct file_operations ufsdbg_dump_device_desc = {
.open = ufsdbg_dump_device_desc_open,
.read = seq_read,
+ .release = single_release,
};
static int ufsdbg_power_mode_show(struct seq_file *file, void *data)
@@ -1257,6 +1267,7 @@ static const struct file_operations ufsdbg_power_mode_desc = {
.open = ufsdbg_power_mode_open,
.read = seq_read,
.write = ufsdbg_power_mode_write,
+ .release = single_release,
};
static int ufsdbg_dme_read(void *data, u64 *attr_val, bool peer)
@@ -1402,8 +1413,9 @@ static int ufsdbg_req_stats_show(struct seq_file *file, void *data)
unsigned long flags;
/* Header */
- seq_printf(file, "\t%-10s %-10s %-10s %-10s %-10s %-10s",
- "All", "Write", "Read", "Read(urg)", "Write(urg)", "Flush");
+ seq_printf(file, "\t%-10s %-10s %-10s %-10s %-10s %-10s %-10s",
+ "All", "Read", "Write", "Read(urg)", "Write(urg)", "Flush",
+ "Discard");
spin_lock_irqsave(hba->host->host_lock, flags);
@@ -1436,8 +1448,77 @@ static const struct file_operations ufsdbg_req_stats_desc = {
.open = ufsdbg_req_stats_open,
.read = seq_read,
.write = ufsdbg_req_stats_write,
+ .release = single_release,
};
+static ssize_t ufsdbg_io_stats_write(struct file *filp,
+ const char __user *ubuf, size_t cnt, loff_t *ppos)
+{
+ struct ufs_hba *hba = filp->f_mapping->host->i_private;
+ unsigned long flags;
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ hba->ufs_stats.io_read.max_diff_req_count = 0;
+ hba->ufs_stats.io_read.max_diff_total_bytes = 0;
+ hba->ufs_stats.io_readwrite.max_diff_req_count = 0;
+ hba->ufs_stats.io_readwrite.max_diff_total_bytes = 0;
+ hba->ufs_stats.io_write.max_diff_req_count = 0;
+ hba->ufs_stats.io_write.max_diff_total_bytes = 0;
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+
+ return cnt;
+}
+
+static int ufsdbg_io_stats_show(struct seq_file *file, void *data)
+{
+ struct ufs_hba *hba = (struct ufs_hba *)file->private;
+ unsigned long flags;
+
+ seq_printf(file, "\t\t%-10s %-10s %-10s %-10s %-10s %-10s\n",
+ "ReadCnt", "ReadBytes", "WriteCnt", "WriteBytes", "RWCnt",
+ "RWBytes");
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ seq_printf(file,
+ "Started: \t%-10llu %-10llu %-10llu %-10llu %-10llu %-10llu\n",
+ hba->ufs_stats.io_read.req_count_started,
+ hba->ufs_stats.io_read.total_bytes_started,
+ hba->ufs_stats.io_write.req_count_started,
+ hba->ufs_stats.io_write.total_bytes_started,
+ hba->ufs_stats.io_readwrite.req_count_started,
+ hba->ufs_stats.io_readwrite.total_bytes_started);
+ seq_printf(file,
+ "Completed: \t%-10llu %-10llu %-10llu %-10llu %-10llu %-10llu\n",
+ hba->ufs_stats.io_read.req_count_completed,
+ hba->ufs_stats.io_read.total_bytes_completed,
+ hba->ufs_stats.io_write.req_count_completed,
+ hba->ufs_stats.io_write.total_bytes_completed,
+ hba->ufs_stats.io_readwrite.req_count_completed,
+ hba->ufs_stats.io_readwrite.total_bytes_completed);
+ seq_printf(file,
+ "MaxDiff: \t%-10llu %-10llu %-10llu %-10llu %-10llu %-10llu\n",
+ hba->ufs_stats.io_read.max_diff_req_count,
+ hba->ufs_stats.io_read.max_diff_total_bytes,
+ hba->ufs_stats.io_write.max_diff_req_count,
+ hba->ufs_stats.io_write.max_diff_total_bytes,
+ hba->ufs_stats.io_readwrite.max_diff_req_count,
+ hba->ufs_stats.io_readwrite.max_diff_total_bytes);
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+
+ return 0;
+}
+
+static int ufsdbg_io_stats_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, ufsdbg_io_stats_show, inode->i_private);
+}
+
+static const struct file_operations ufsdbg_io_stats_desc = {
+ .open = ufsdbg_io_stats_open,
+ .read = seq_read,
+ .write = ufsdbg_io_stats_write,
+ .release = single_release,
+};
static int ufsdbg_reset_controller_show(struct seq_file *file, void *data)
{
@@ -1484,6 +1565,7 @@ static const struct file_operations ufsdbg_reset_controller = {
.open = ufsdbg_reset_controller_open,
.read = seq_read,
.write = ufsdbg_reset_controller_write,
+ .release = single_release,
};
static int ufsdbg_clear_err_state(void *data, u64 val)
@@ -1528,14 +1610,17 @@ DEFINE_SIMPLE_ATTRIBUTE(ufsdbg_err_state,
void ufsdbg_add_debugfs(struct ufs_hba *hba)
{
+ char root_name[sizeof("ufshcd00")];
+
if (!hba) {
pr_err("%s: NULL hba, exiting", __func__);
return;
}
- hba->debugfs_files.debugfs_root = debugfs_create_dir(dev_name(hba->dev),
- NULL);
+ snprintf(root_name, ARRAY_SIZE(root_name), "%s%d", UFSHCD,
+ hba->host->host_no);
+ hba->debugfs_files.debugfs_root = debugfs_create_dir(root_name, NULL);
if (IS_ERR(hba->debugfs_files.debugfs_root))
/* Don't complain -- debugfs just isn't enabled */
goto err_no_root;
@@ -1673,6 +1758,17 @@ void ufsdbg_add_debugfs(struct ufs_hba *hba)
goto err;
}
+ hba->debugfs_files.io_stats =
+ debugfs_create_file("io_stats", 0600,
+ hba->debugfs_files.stats_folder, hba,
+ &ufsdbg_io_stats_desc);
+ if (!hba->debugfs_files.io_stats) {
+ dev_err(hba->dev,
+ "%s: failed create io_stats debugfs entry\n",
+ __func__);
+ goto err;
+ }
+
hba->debugfs_files.reset_controller =
debugfs_create_file("reset_controller", S_IRUSR | S_IWUSR,
hba->debugfs_files.debugfs_root, hba,
diff --git a/drivers/scsi/ufs/ufs-impaired.c b/drivers/scsi/ufs/ufs-impaired.c
new file mode 100644
index 0000000..7714dee
--- /dev/null
+++ b/drivers/scsi/ufs/ufs-impaired.c
@@ -0,0 +1,824 @@
+/*
+ * UFS impaired storage emulation.
+ *
+ * Copyright (C) 2018, Google, Inc.
+ *
+ * Authors:
+ * Hyojun Kim <hyojun@google.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * See the COPYING file in the top-level directory or visit
+ * <http://www.gnu.org/licenses/gpl-2.0.html>
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * This program is provided "AS IS" and "WITH ALL FAULTS" and
+ * without warranty of any kind. You are solely responsible for
+ * determining the appropriateness of using and distributing
+ * the program and assume all risks associated with your exercise
+ * of rights with respect to the program, including but not limited
+ * to infringement of third party rights, the risks and costs of
+ * program errors, damage to or loss of data, programs or equipment,
+ * and unavailability or interruption of operations. Under no
+ * circumstances will the contributor of this Program be liable for
+ * any damages of any kind arising from your use or distribution of
+ * this program.
+ *
+ * The Linux Foundation chooses to take subject only to the GPLv2
+ * license terms, and distributes only under these terms.
+ */
+
+#include <linux/random.h>
+#include "ufshcd.h"
+
+#define UFS_IMPAIRED_SLEEP_US (500)
+#define UFS_IMPAIRED_ERROR_US (10)
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+
+/**
+ * Latency distribution model for small (<128KiB) read requests.
+ * - captured from Micron 64GB / DoU App / fragftl90 precondition.
+ */
+static int impaired_latency_small_read[] = {
+ 86, 87, 89, 90, 91, 92, 93, 96,
+ 98, 100, 101, 103, 105, 106, 108, 108,
+ 109, 111, 113, 115, 117, 120, 122, 124,
+ 127, 130, 134, 136, 138, 139, 140, 141,
+ 143, 144, 145, 147, 149, 151, 153, 155,
+ 157, 159, 162, 164, 168, 172, 176, 180,
+ 183, 187, 191, 194, 196, 199, 203, 207,
+ 211, 215, 218, 223, 227, 232, 237, 242,
+ 246, 251, 256, 259, 264, 269, 275, 278,
+ 283, 288, 294, 299, 304, 310, 315, 321,
+ 326, 333, 338, 344, 348, 355, 361, 367,
+ 373, 379, 385, 391, 398, 406, 412, 420,
+ 425, 433, 441, 448, 457, 464, 471, 479,
+ 489, 500, 508, 517, 527, 535, 544, 555,
+ 565, 576, 589, 600, 605, 610, 616, 621,
+ 626, 632, 637, 642, 648, 651, 654, 657,
+ 661, 664, 667, 672, 678, 684, 689, 694,
+ 699, 704, 708, 713, 719, 724, 729, 735,
+ 742, 748, 756, 764, 770, 777, 787, 795,
+ 804, 812, 821, 831, 840, 849, 858, 866,
+ 876, 885, 894, 903, 911, 920, 930, 939,
+ 946, 957, 966, 979, 987, 997, 1006, 1017,
+ 1027, 1036, 1049, 1062, 1073, 1087, 1100, 1110,
+ 1121, 1134, 1145, 1159, 1172, 1188, 1203, 1221,
+ 1239, 1258, 1276, 1288, 1307, 1330, 1353, 1376,
+ 1403, 1427, 1453, 1484, 1509, 1537, 1567, 1596,
+ 1629, 1656, 1704, 1744, 1782, 1847, 1893, 1957,
+ 2011, 2068, 2131, 2204, 2277, 2359, 2443, 2545,
+ 2645, 2730, 2832, 2965, 3064, 3193, 3297, 3436,
+ 3558, 3670, 3829, 4025, 4241, 4482, 4786, 5266,
+ 5709, 6366, 8044, 10499, 11754, 11806, 11850, 11895,
+ 11943, 11978, 12009, 12039, 12092, 12181, 12663, 50713,
+};
+
+/**
+ * Latency distribution model for large (>=128KiB) read requests.
+ * - captured from Micron 64GB / DoU App / fragftl90 precondition.
+ */
+static int impaired_latency_large_read[] = {
+ 324, 364, 380, 382, 384, 385, 387, 387,
+ 388, 389, 390, 390, 390, 391, 391, 391,
+ 391, 391, 392, 393, 393, 393, 393, 393,
+ 393, 394, 394, 395, 395, 395, 395, 395,
+ 396, 396, 396, 396, 396, 396, 396, 397,
+ 398, 398, 398, 398, 398, 398, 398, 398,
+ 399, 399, 399, 399, 399, 399, 400, 401,
+ 401, 401, 402, 403, 403, 404, 404, 405,
+ 406, 409, 410, 411, 412, 414, 415, 417,
+ 418, 420, 422, 423, 426, 428, 431, 437,
+ 441, 444, 449, 453, 460, 463, 468, 471,
+ 474, 476, 480, 482, 484, 487, 490, 492,
+ 495, 499, 503, 508, 512, 516, 520, 524,
+ 529, 536, 540, 546, 552, 557, 565, 572,
+ 580, 595, 601, 614, 622, 631, 640, 645,
+ 654, 668, 676, 684, 697, 707, 718, 731,
+ 748, 760, 775, 794, 802, 815, 821, 836,
+ 845, 866, 877, 889, 900, 907, 911, 918,
+ 927, 943, 958, 979, 987, 997, 1005, 1010,
+ 1021, 1035, 1048, 1059, 1078, 1095, 1105, 1121,
+ 1164, 1180, 1210, 1236, 1259, 1286, 1314, 1364,
+ 1403, 1449, 1481, 1548, 1602, 1615, 1630, 1659,
+ 1686, 1712, 1776, 1817, 1855, 1905, 1994, 2029,
+ 2097, 2164, 2246, 2318, 2382, 2448, 2516, 2547,
+ 2615, 2663, 2733, 2834, 2921, 2987, 3052, 3113,
+ 3224, 3292, 3373, 3494, 3615, 3717, 3805, 3958,
+ 4100, 4225, 4380, 4452, 4557, 4637, 4794, 5033,
+ 5341, 5748, 6100, 6638, 7473, 8374, 11764, 11932,
+ 11999, 12024, 12033, 12037, 12040, 12044, 12047, 12049,
+ 12053, 12056, 12060, 12062, 12064, 12069, 12071, 12079,
+ 12087, 12101, 12128, 12142, 12150, 12158, 12169, 12184,
+ 12200, 12211, 12224, 12275, 12347, 12464, 13961, 50629,
+};
+
+/**
+ * Latency distribution model for small (<128KiB) write requests.
+ * - captured from Samsung 64GB / DoU App / fragftl90 precondition.
+ */
+static int impaired_latency_small_write[] = {
+ 41, 42, 43, 43, 43, 43, 43, 43,
+ 43, 43, 43, 43, 44, 44, 44, 44,
+ 44, 44, 45, 45, 45, 45, 46, 46,
+ 46, 46, 46, 46, 46, 46, 47, 47,
+ 47, 48, 49, 49, 49, 49, 50, 50,
+ 50, 51, 52, 54, 54, 55, 57, 58,
+ 59, 60, 62, 62, 63, 65, 66, 67,
+ 70, 73, 74, 75, 78, 81, 82, 85,
+ 86, 89, 90, 92, 94, 95, 100, 103,
+ 106, 109, 111, 117, 121, 125, 131, 134,
+ 143, 151, 158, 164, 169, 176, 183, 190,
+ 197, 205, 214, 221, 228, 230, 232, 235,
+ 238, 248, 260, 273, 283, 294, 314, 328,
+ 347, 372, 418, 469, 501, 528, 559, 585,
+ 598, 607, 621, 633, 658, 671, 725, 774,
+ 810, 833, 842, 847, 853, 861, 873, 884,
+ 893, 904, 916, 935, 954, 960, 962, 963,
+ 965, 966, 968, 971, 974, 978, 981, 982,
+ 983, 986, 988, 991, 996, 999, 1003, 1008,
+ 1013, 1018, 1025, 1030, 1036, 1043, 1048, 1052,
+ 1057, 1061, 1066, 1072, 1080, 1085, 1095, 1103,
+ 1121, 1154, 1183, 1210, 1245, 1270, 1307, 1340,
+ 1385, 1427, 1447, 1473, 1502, 1530, 1588, 1620,
+ 1671, 1727, 1788, 1822, 1871, 2019, 2107, 2161,
+ 2202, 2267, 2324, 2339, 2347, 2351, 2358, 2363,
+ 2367, 2371, 2377, 2383, 2386, 2390, 2395, 2401,
+ 2411, 2416, 2422, 2455, 2496, 2555, 2626, 2793,
+ 2883, 2956, 3075, 3104, 3136, 3257, 3720, 3990,
+ 4569, 5193, 5462, 5599, 5774, 5965, 6304, 6350,
+ 6367, 6460, 6589, 6648, 7090, 7666, 7921, 8528,
+ 8620, 8758, 8933, 9089, 9310, 10996, 13298, 15989,
+ 27039, 111816, 228143, 242631, 249738, 277515, 344243, 644551,
+};
+
+/**
+ * Latency distribution model for large (>=128KiB) write requests.
+ * - captured from Samsung 64GB / DoU App / fragftl90 precondition.
+ */
+static int impaired_latency_large_write[] = {
+ 931, 1368, 1597, 1717, 1770, 2102, 2190, 2203,
+ 2227, 2321, 2651, 2964, 3044, 3107, 3125, 3187,
+ 3619, 3740, 4298, 4404, 4500, 4518, 4536, 4612,
+ 4680, 5248, 5408, 5803, 6486, 7171, 7603, 8139,
+ 8590, 8934, 9661, 10135, 10815, 11387, 11653, 12698,
+ 13028, 13461, 13918, 14262, 14653, 15023, 16056, 16959,
+ 18488, 19161, 20670, 21717, 22937, 23748, 25017, 26392,
+ 28037, 30083, 31255, 32115, 33473, 33867, 35541, 36389,
+ 36881, 37604, 38095, 38809, 40167, 42621, 45367, 49762,
+ 51501, 55550, 58046, 59042, 59974, 60293, 61093, 63117,
+ 65404, 66943, 67522, 69874, 74325, 76652, 79318, 80906,
+ 82579, 83591, 85002, 86212, 87530, 88704, 90705, 93660,
+ 96113, 100477, 103027, 104373, 106349, 106865, 107940, 109363,
+ 110861, 112585, 114433, 117842, 121719, 124761, 127053, 127735,
+ 128408, 130274, 131996, 133333, 136319, 138339, 140988, 143100,
+ 145622, 148063, 149843, 151187, 153021, 155009, 157893, 160171,
+ 162224, 165607, 168586, 172387, 173645, 174777, 176540, 178189,
+ 179568, 182432, 186736, 190361, 192542, 195341, 196778, 197484,
+ 198562, 200706, 203469, 208503, 210125, 213157, 216006, 218331,
+ 220201, 221135, 222878, 224437, 226826, 231460, 234924, 236136,
+ 238898, 240970, 243045, 245437, 246771, 249320, 255538, 257795,
+ 261502, 266142, 270622, 273351, 274984, 282415, 289584, 295506,
+ 299918, 307177, 310620, 317469, 330186, 339543, 353783, 365386,
+ 384748, 418406, 452769, 487990, 522070, 531204, 539830, 548083,
+ 551891, 554911, 560052, 563793, 564732, 565626, 567059, 571978,
+ 574526, 576305, 578064, 579098, 581912, 582565, 582703, 583912,
+ 585966, 588347, 590459, 592906, 595216, 597190, 599410, 603470,
+ 605402, 606438, 609488, 612438, 615461, 618508, 620230, 620825,
+ 624151, 624814, 625431, 626220, 627428, 628625, 629637, 630157,
+ 631669, 633001, 633431, 634438, 635181, 635682, 637344, 637964,
+ 638930, 639790, 640542, 641104, 641752, 642765, 643495, 644437,
+ 645105, 646311, 648009, 652824, 658469, 680256, 709137, 735646,
+};
+
+#define MODEL_SIZE (sizeof(impaired_latency_large_write) / sizeof(int))
+#define REQSZ_THRESHOLD (128 * 1024)
+
+#endif
+
+/**
+ * Impaired storage sysfs entries
+ */
+struct impaired_attr {
+ struct attribute attr;
+
+ ssize_t (*show)(struct kobject *kobj, struct attribute *attr,
+ char *buf);
+
+ ssize_t (*store)(struct kobject *kobj, struct attribute *attr,
+ char *buf, size_t count);
+};
+
+struct impaired_delay_attr {
+ struct attribute attr;
+
+ ssize_t (*show)(struct kobject *kobj, struct attribute *attr,
+ char *buf);
+
+ ssize_t (*store)(struct kobject *kobj, struct attribute *attr,
+ char *buf, size_t count);
+ enum ufs_impaired_optype optype;
+ enum ufs_impaired_delaytype delaytype;
+ u32 maximum;
+};
+
+struct impaired_stat_attr {
+ struct attribute attr;
+
+ ssize_t (*show)(struct kobject *kobj, struct attribute *attr,
+ char *buf);
+
+ ssize_t (*store)(struct kobject *kobj, struct attribute *attr,
+ char *buf, size_t count);
+ enum ufs_impaired_optype optype;
+ enum ufs_impaired_stattype stattype;
+};
+
+static inline struct ufs_hba *__get_hba_from_kobj(struct kobject *obj)
+{
+ struct device *dev = container_of(obj->parent, struct device, kobj);
+ return dev_get_drvdata(dev);
+}
+
+static ssize_t
+impaired_show_enabled(struct kobject *obj, struct attribute *attr, char *buf)
+{
+ struct ufs_hba *hba = __get_hba_from_kobj(obj);
+ return snprintf(buf, PAGE_SIZE, "%d\n", hba->impaired.enabled);
+}
+
+static ssize_t
+impaired_store_enabled(struct kobject *obj, struct attribute *attr, char *buf,
+ size_t count)
+{
+ struct ufs_hba *hba = __get_hba_from_kobj(obj);
+ unsigned long flags, value;
+
+ if (kstrtol(buf, 0, &value))
+ return -EINVAL;
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ hba->impaired.enabled = (value) ? 1 : 0;
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+
+ return count;
+}
+
+static ssize_t
+impaired_show_delay(struct kobject *obj, struct attribute *_attr, char *buf)
+{
+ struct ufs_hba *hba = __get_hba_from_kobj(obj);
+ struct impaired_delay_attr *attr = (struct impaired_delay_attr *)_attr;
+ u32 value = hba->impaired.io[attr->optype].delay[attr->delaytype];
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+ if (attr->delaytype == UFS_IMPAIRED_DELAYTYPE_MODEL) {
+ return snprintf(buf, PAGE_SIZE,
+ (value) ? "ftlfrag\n" : "none\n");
+ }
+#endif
+ return snprintf(buf, PAGE_SIZE, "%d\n", value);
+}
+
+static ssize_t
+impaired_store_delay(struct kobject *obj, struct attribute *_attr, char *buf,
+ size_t count)
+{
+ struct ufs_hba *hba = __get_hba_from_kobj(obj);
+ struct impaired_delay_attr *attr = (struct impaired_delay_attr *)_attr;
+ unsigned long flags, value;
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+ if (attr->delaytype == UFS_IMPAIRED_DELAYTYPE_MODEL) {
+ if (strncmp(buf, "none", 4) == 0)
+ value = 0;
+ else if (strncmp(buf, "ftlfrag", 7) == 0)
+ value = 1;
+ else
+ return -EINVAL;
+ } else if (kstrtol(buf, 0, &value)) {
+ return -EINVAL;
+ }
+#else
+ if (kstrtol(buf, 0, &value))
+ return -EINVAL;
+#endif
+
+ if (value > attr->maximum)
+ return -EINVAL;
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ hba->impaired.io[attr->optype].delay[attr->delaytype] = (u32)value;
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+
+ return count;
+}
+
+static ssize_t
+impaired_show_stat(struct kobject *obj, struct attribute *_attr, char *buf)
+{
+ struct ufs_hba *hba = __get_hba_from_kobj(obj);
+ struct impaired_stat_attr *attr = (struct impaired_stat_attr *)_attr;
+
+ return snprintf(buf, PAGE_SIZE, "%lld\n",
+ hba->impaired.io[attr->optype].stat[attr->stattype]);
+}
+
+static ssize_t
+impaired_store_stat(struct kobject *obj, struct attribute *_attr, char *buf,
+ size_t count)
+{
+ struct ufs_hba *hba = __get_hba_from_kobj(obj);
+ struct impaired_stat_attr *attr = (struct impaired_stat_attr *)_attr;
+ unsigned long flags;
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ memset(&hba->impaired.io[attr->optype].stat, 0,
+ sizeof(hba->impaired.io[attr->optype].stat));
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+
+ return count;
+}
+
+#define IMPAIRED_ATTR_RW(_name) \
+ static struct impaired_attr impaired_##_name = { \
+ .attr = {.name = __stringify(_name), \
+ .mode = 0644 }, \
+ .show = impaired_show_##_name, \
+ .store = impaired_store_##_name, \
+ }
+
+#define IMPAIRED_DELAY_ATTR(_name, _op, _delay, _maximum) \
+ static struct impaired_delay_attr \
+ impaired_##_name = { \
+ .attr = {.name = __stringify(_name), \
+ .mode = 0644 }, \
+ .show = impaired_show_delay, \
+ .store = impaired_store_delay, \
+ .optype = _op, \
+ .delaytype = _delay, \
+ .maximum = _maximum, \
+ }
+
+#define IMPAIRED_STAT_ATTR(_name, _op, _stat) \
+ static struct impaired_stat_attr \
+ impaired_##_name = { \
+ .attr = {.name = __stringify(_name), \
+ .mode = 0644 }, \
+ .show = impaired_show_stat, \
+ .store = impaired_store_stat, \
+ .optype = _op, \
+ .stattype = _stat, \
+ }
+
+/* sysfs impaired: root */
+IMPAIRED_ATTR_RW(enabled);
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+IMPAIRED_DELAY_ATTR(read_model, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_DELAYTYPE_MODEL, 1);
+#endif
+IMPAIRED_DELAY_ATTR(read_delay_us, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_DELAYTYPE_US, 1000000);
+IMPAIRED_DELAY_ATTR(read_delay_percent, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_DELAYTYPE_PERCENT, 100000);
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+IMPAIRED_DELAY_ATTR(write_model, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_DELAYTYPE_MODEL, 1);
+#endif
+IMPAIRED_DELAY_ATTR(write_delay_us, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_DELAYTYPE_US, 1000000);
+IMPAIRED_DELAY_ATTR(write_delay_percent, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_DELAYTYPE_PERCENT, 100000);
+
+/* sysfs impaired/advanced_control */
+IMPAIRED_DELAY_ATTR(read_max_delayed_us, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_DELAYTYPE_MAX, 2000000);
+IMPAIRED_DELAY_ATTR(read_skip_delay_cnt, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_DELAYTYPE_SKIP, 2000000);
+
+IMPAIRED_DELAY_ATTR(write_max_delayed_us, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_DELAYTYPE_MAX, 2000000);
+IMPAIRED_DELAY_ATTR(write_skip_delay_cnt, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_DELAYTYPE_SKIP, 2000000);
+
+/* sysfs impaired/stat */
+IMPAIRED_STAT_ATTR(total_read_cnt, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_STATTYPE_TOTAL_CNT);
+IMPAIRED_STAT_ATTR(total_read_original_us, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_STATTYPE_TOTAL_ORIGINAL_US);
+IMPAIRED_STAT_ATTR(total_read_delayed_us, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_STATTYPE_TOTAL_DELAYED_US);
+
+IMPAIRED_STAT_ATTR(total_write_cnt, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_STATTYPE_TOTAL_CNT);
+IMPAIRED_STAT_ATTR(total_write_original_us, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_STATTYPE_TOTAL_ORIGINAL_US);
+IMPAIRED_STAT_ATTR(total_write_delayed_us, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_STATTYPE_TOTAL_DELAYED_US);
+
+/* sysfs impaired/delay_stat */
+IMPAIRED_STAT_ATTR(max_read_delay_us, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_STATTYPE_MAX_DELAY);
+IMPAIRED_STAT_ATTR(max_read_delay_error_us, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_STATTYPE_MAX_ERROR);
+IMPAIRED_STAT_ATTR(total_read_delay_error_us, UFS_IMPAIRED_OPTYPE_READ, \
+ UFS_IMPAIRED_STATTYPE_TOTAL_DELAY_ERROR);
+
+IMPAIRED_STAT_ATTR(max_write_delay_us, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_STATTYPE_MAX_DELAY);
+IMPAIRED_STAT_ATTR(max_write_delay_error_us, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_STATTYPE_MAX_ERROR);
+IMPAIRED_STAT_ATTR(total_write_delay_error_us, UFS_IMPAIRED_OPTYPE_WRITE, \
+ UFS_IMPAIRED_STATTYPE_TOTAL_DELAY_ERROR);
+
+static struct attribute *impaired_attrs[] = {
+ /* Basic delay control */
+ &impaired_enabled.attr,
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+ &impaired_read_model.attr,
+#endif
+ &impaired_read_delay_us.attr,
+ &impaired_read_delay_percent.attr,
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+ &impaired_write_model.attr,
+#endif
+ &impaired_write_delay_us.attr,
+ &impaired_write_delay_percent.attr,
+ NULL
+};
+
+const struct attribute_group impaired_group = {
+ .attrs = impaired_attrs,
+};
+
+static struct attribute *impaired_advanced_attrs[] = {
+ /* Advanced delay control */
+ &impaired_read_max_delayed_us.attr,
+ &impaired_read_skip_delay_cnt.attr,
+
+ &impaired_write_max_delayed_us.attr,
+ &impaired_write_skip_delay_cnt.attr,
+ NULL
+};
+
+const struct attribute_group impaired_advanced_group = {
+ .name = "advanced_control",
+ .attrs = impaired_advanced_attrs,
+};
+
+static struct attribute *impaired_stat_attrs[] = {
+ /* Statistics */
+ &impaired_total_read_cnt.attr,
+ &impaired_total_read_original_us.attr,
+ &impaired_total_read_delayed_us.attr,
+
+ &impaired_total_write_cnt.attr,
+ &impaired_total_write_original_us.attr,
+ &impaired_total_write_delayed_us.attr,
+ NULL
+};
+
+const struct attribute_group impaired_stat_group = {
+ .name = "stat",
+ .attrs = impaired_stat_attrs,
+};
+
+static struct attribute *impaired_delay_stat_attrs[] = {
+ /* Delay related statistics */
+ &impaired_max_read_delay_us.attr,
+ &impaired_max_read_delay_error_us.attr,
+ &impaired_total_read_delay_error_us.attr,
+
+ &impaired_max_write_delay_us.attr,
+ &impaired_max_write_delay_error_us.attr,
+ &impaired_total_write_delay_error_us.attr,
+ NULL
+};
+
+const struct attribute_group impaired_delay_stat_group = {
+ .name = "delay_stat",
+ .attrs = impaired_delay_stat_attrs,
+};
+
+const struct attribute_group *impaired_sysfs_groups[] = {
+ &impaired_group,
+ &impaired_advanced_group,
+ &impaired_stat_group,
+ &impaired_delay_stat_group,
+ NULL
+};
+
+void ufs_impaired_init_sysfs(struct ufs_hba *hba)
+{
+ hba->impaired_kobj = kobject_create_and_add("impaired",
+ &hba->dev->kobj);
+ if (!hba->impaired_kobj)
+ dev_err(hba->dev, "Failed to create impaired storage sysfs folder\n");
+
+ if (sysfs_create_groups(hba->impaired_kobj, impaired_sysfs_groups))
+ dev_err(hba->dev, "Failed to create impaired storage sysfs group\n");
+}
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+/**
+ * __pick_target_latency_with_model
+ * - pick a latency for a UFS request by using latency distribution models.
+ * @op: operation type
+ * @req_size: request size in bytes
+ */
+static int __pick_target_latency_with_model(
+ enum ufs_impaired_optype op, u32 req_size)
+{
+ unsigned int idx;
+ int *model;
+
+ /* Pick a model based on request type and size */
+ if (op == UFS_IMPAIRED_OPTYPE_READ)
+ model = (req_size < REQSZ_THRESHOLD) ?
+ impaired_latency_small_read :
+ impaired_latency_large_read;
+ else
+ model = (req_size < REQSZ_THRESHOLD) ?
+ impaired_latency_small_write :
+ impaired_latency_large_write;
+
+ /* Generate random index and pick a target latency */
+ prandom_bytes(&idx, sizeof(idx));
+ return model[idx % MODEL_SIZE];
+}
+#endif
+
+/**
+ * __set_delay_in_lrb - pick a target delay and set lrbp->complete_time and
+ * lrbp->target_compete_time fields.
+ * @hba: per adapter instance
+ * @lrbp: pointer to local reference block to add delay
+ */
+void __set_delay_in_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+{
+ struct scsi_cmnd *cmd = lrbp->cmd;
+ struct request *rq = (cmd) ? cmd->request : NULL;
+ s32 actual_us = ktime_us_delta(lrbp->complete_time_stamp,
+ lrbp->issue_time_stamp);
+ struct ufs_impaired_io *io;
+ enum ufs_impaired_optype op;
+ s32 target_us;
+
+ /* Set default value: no delay */
+ lrbp->complete_delay = 0;
+ lrbp->target_complete_time = lrbp->complete_time_stamp;
+
+ /* Return if delay is not needed */
+ if (!hba->impaired.enabled ||
+ hba->impaired_should_stop ||
+ !rq ||
+ !(rq->cmd_type & REQ_TYPE_FS) ||
+ (rq->cmd_flags & REQ_PREFLUSH) ||
+ req_op(rq) == REQ_OP_FLUSH ||
+ req_op(rq) == REQ_OP_DISCARD ||
+ !rq->bio)
+ return;
+
+ op = (rq_data_dir(rq) == READ) ?
+ UFS_IMPAIRED_OPTYPE_READ : UFS_IMPAIRED_OPTYPE_WRITE;
+ io = &hba->impaired.io[op];
+
+ /* Apply skip logic: delay one request after N skipping */
+ if (io->delay[UFS_IMPAIRED_DELAYTYPE_SKIP] > io->skip_count++)
+ return;
+ io->skip_count = 0;
+
+ /* Set default target latency */
+ target_us = actual_us;
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+ /* If fragmented FTL emulation is on,
+ call __pick_target_latency_with_model() to pick a target latency */
+ if (io->delay[UFS_IMPAIRED_DELAYTYPE_MODEL])
+ target_us = __pick_target_latency_with_model(op,
+ scsi_get_bytes(lrbp->cmd));
+#endif
+
+ /* Apply additional knobs */
+ target_us = target_us * io->delay[UFS_IMPAIRED_DELAYTYPE_PERCENT] / 100;
+ target_us += io->delay[UFS_IMPAIRED_DELAYTYPE_US];
+
+ /* Apply maximum latency */
+ if (io->delay[UFS_IMPAIRED_DELAYTYPE_MAX] &&
+ target_us > io->delay[UFS_IMPAIRED_DELAYTYPE_MAX])
+ target_us = io->delay[UFS_IMPAIRED_DELAYTYPE_MAX];
+
+ /* Target latency cannot be smaller than the actual latency */
+ if (target_us < actual_us)
+ target_us = actual_us;
+
+ /* Update delay related statistics */
+ io->stat[UFS_IMPAIRED_STATTYPE_TOTAL_CNT] += 1;
+ io->stat[UFS_IMPAIRED_STATTYPE_TOTAL_ORIGINAL_US] += actual_us;
+
+ /* Delayed time will be added later when the request is completed */
+ io->stat[UFS_IMPAIRED_STATTYPE_TOTAL_DELAYED_US] += actual_us;
+
+ /* Set delay fields */
+ lrbp->complete_delay = (int)(target_us - actual_us);
+ lrbp->target_complete_time = ktime_add_us(lrbp->complete_time_stamp,
+ lrbp->complete_delay);
+
+ /* Keep track of the maximum delay */
+ if (io->stat[UFS_IMPAIRED_STATTYPE_MAX_DELAY] < lrbp->complete_delay)
+ io->stat[UFS_IMPAIRED_STATTYPE_MAX_DELAY] =
+ lrbp->complete_delay;
+}
+
+
+/**
+ * __queue_lrb - put a lrb in to the delay list
+ * @hba: per adapter instance
+ * @lrbp: pointer to local reference block to be queued
+ */
+void __queue_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+{
+ struct list_head *parent;
+ struct ufshcd_lrb *l;
+
+ /* Find insertion point in the sorted list by target_complete_time */
+ parent = &hba->impaired_list_head;
+ list_for_each_entry(l, &hba->impaired_list_head, list) {
+ if (ktime_compare(l->target_complete_time,
+ lrbp->target_complete_time) > 0)
+ break;
+ parent = &l->list;
+ }
+
+ /* Insert the LRB at the right position */
+ list_add(&lrbp->list, parent);
+
+ /* Mark the LRB is being delayed */
+ __set_bit(lrbp - hba->lrb, &hba->delayed_reqs);
+}
+
+/**
+ * __update_statistics - update delay related statistics
+ * @hba: per adapter instance
+ * @lrbp: pointer to local reference block to update
+ */
+static void __update_statistics(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+{
+ enum ufs_impaired_optype op = (rq_data_dir(lrbp->cmd->request) == READ)
+ ? UFS_IMPAIRED_OPTYPE_READ : UFS_IMPAIRED_OPTYPE_WRITE;
+ s64 delay_error;
+
+ /* Final (actual) completion time with delay */
+ ktime_t actual_completion_time = ktime_get();
+
+ /* Update statistics */
+ hba->impaired.io[op].stat[UFS_IMPAIRED_STATTYPE_TOTAL_DELAYED_US] +=
+ ktime_us_delta(actual_completion_time,
+ lrbp->complete_time_stamp);
+
+ /* Monitor how much more delayed than intended */
+ delay_error = ktime_us_delta(actual_completion_time,
+ lrbp->target_complete_time);
+ hba->impaired.io[op].stat[UFS_IMPAIRED_STATTYPE_TOTAL_DELAY_ERROR] +=
+ delay_error;
+
+ /* Keep track the maximum delay error */
+ if (hba->impaired.io[op].stat[UFS_IMPAIRED_STATTYPE_MAX_ERROR]
+ < delay_error)
+ hba->impaired.io[op].stat[UFS_IMPAIRED_STATTYPE_MAX_ERROR] =
+ delay_error;
+
+
+ /* Update completion time for UFS driver stats */
+ lrbp->complete_time_stamp = actual_completion_time;
+}
+
+/**
+ * __impaired_thread_fn - kernel thread to complete requests
+ * @param: kthread parameter (hba)
+ */
+
+static int __impaired_thread_fn(void *param)
+{
+ struct ufs_hba *hba = param;
+ struct ufshcd_lrb *lrbp;
+ unsigned long flags;
+ s64 delay_us;
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ while (!kthread_should_stop() && !hba->impaired_should_stop) {
+ if (list_empty(&hba->impaired_list_head)) {
+ /* Go to sleep if the list is empty */
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+
+ usleep_range(UFS_IMPAIRED_SLEEP_US,
+ UFS_IMPAIRED_SLEEP_US * 2);
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ continue;
+ }
+
+ /* Get the first entry */
+ lrbp = list_first_entry(&hba->impaired_list_head,
+ struct ufshcd_lrb, list);
+
+ /* Calculate needed delay time at this point */
+ delay_us = ktime_us_delta(lrbp->target_complete_time,
+ ktime_get());
+
+ if (delay_us < UFS_IMPAIRED_ERROR_US) {
+ /* Complete an LRB */
+ list_del(&lrbp->list);
+ __update_statistics(hba, lrbp);
+ ufshcd_complete_lrb(hba, lrbp);
+ wake_up(&hba->dev_cmd.tag_wq);
+ } else {
+ /* Do sleep */
+ if (delay_us > UFS_IMPAIRED_SLEEP_US)
+ delay_us = UFS_IMPAIRED_SLEEP_US;
+
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+ usleep_range(delay_us / 2, delay_us);
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ }
+ }
+
+ /* Complete pending I/Os, if exist */
+ list_for_each_entry(lrbp, &hba->impaired_list_head, list) {
+ __update_statistics(hba, lrbp);
+ ufshcd_complete_lrb(hba, lrbp);
+ }
+
+ /* Empty the queue */
+ INIT_LIST_HEAD(&hba->impaired_list_head);
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+
+ return 0;
+}
+
+/**
+ * ufs_impaired_delay_completion - delay an LRB.
+ * @hba: per adapter instance
+ * @lrbp: pointer to local reference block to be queued
+ */
+void ufs_impaired_delay_completion(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+{
+ /* Pick delay and set in LRB */
+ __set_delay_in_lrb(hba, lrbp);
+
+ /* If delay is not needed, complete and return.
+ Otherwise, put the LRB into the delay queue */
+ if (!lrbp->complete_delay)
+ ufshcd_complete_lrb(hba, lrbp);
+ else {
+ __queue_lrb(hba, lrbp);
+ wake_up_process(hba->impaired_thread);
+ }
+}
+
+/**
+ * ufs_impaired_init - Start impaired storage emulation service.
+ * @hba: per adapter instance
+ */
+void ufs_impaired_init(struct ufs_hba *hba)
+{
+ /* Set default value to the sysfs entries */
+ hba->impaired.io[UFS_IMPAIRED_OPTYPE_READ].delay[
+ UFS_IMPAIRED_DELAYTYPE_PERCENT] = 100;
+ hba->impaired.io[UFS_IMPAIRED_OPTYPE_WRITE].delay[
+ UFS_IMPAIRED_DELAYTYPE_PERCENT] = 100;
+
+ /* Initialize impaired list */
+ INIT_LIST_HEAD(&hba->impaired_list_head);
+
+ /* Create a kernel thread */
+ hba->impaired_thread =
+ kthread_run(__impaired_thread_fn, hba, "impaired");
+}
+
+/**
+ * ufs_impaired_exit - Stop impaired storage emulation service.
+ * @hba: per adapter instance
+ */
+void ufs_impaired_exit(struct ufs_hba *hba)
+{
+ /* Wake up impaired_thread to let suicide */
+ hba->impaired_should_stop = true;
+ wake_up_process(hba->impaired_thread);
+
+ /* Wait until all pending I/Os are completed */
+ while (!list_empty(&hba->impaired_list_head))
+ usleep_range(1000, 1100);
+}
+
diff --git a/drivers/scsi/ufs/ufs-impaired.h b/drivers/scsi/ufs/ufs-impaired.h
new file mode 100644
index 0000000..3d3a6ef
--- /dev/null
+++ b/drivers/scsi/ufs/ufs-impaired.h
@@ -0,0 +1,94 @@
+/*
+ * UFS impaired storage emulation.
+ *
+ * Copyright (C) 2018, Google, Inc.
+ *
+ * Authors:
+ * Hyojun Kim <hyojun@google.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * See the COPYING file in the top-level directory or visit
+ * <http://www.gnu.org/licenses/gpl-2.0.html>
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * This program is provided "AS IS" and "WITH ALL FAULTS" and
+ * without warranty of any kind. You are solely responsible for
+ * determining the appropriateness of using and distributing
+ * the program and assume all risks associated with your exercise
+ * of rights with respect to the program, including but not limited
+ * to infringement of third party rights, the risks and costs of
+ * program errors, damage to or loss of data, programs or equipment,
+ * and unavailability or interruption of operations. Under no
+ * circumstances will the contributor of this Program be liable for
+ * any damages of any kind arising from your use or distribution of
+ * this program.
+ *
+ * The Linux Foundation chooses to take subject only to the GPLv2
+ * license terms, and distributes only under these terms.
+ */
+
+#ifndef _UFS_IMPAIRED_H
+#define _UFS_IMPAIRED_H
+
+enum ufs_impaired_optype {
+ UFS_IMPAIRED_OPTYPE_READ = 0,
+ UFS_IMPAIRED_OPTYPE_WRITE = 1,
+ UFS_IMPAIRED_OPTYPE_COUNT = 2,
+};
+
+enum ufs_impaired_delaytype {
+ UFS_IMPAIRED_DELAYTYPE_PERCENT = 0,
+ UFS_IMPAIRED_DELAYTYPE_US = 1,
+ UFS_IMPAIRED_DELAYTYPE_MAX = 2,
+ UFS_IMPAIRED_DELAYTYPE_SKIP = 3,
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+ UFS_IMPAIRED_DELAYTYPE_MODEL = 4,
+ UFS_IMPAIRED_DELAYTYPE_COUNT = 5,
+#else
+ UFS_IMPAIRED_DELAYTYPE_COUNT = 4,
+#endif
+};
+
+enum ufs_impaired_stattype {
+ UFS_IMPAIRED_STATTYPE_TOTAL_CNT = 0,
+ UFS_IMPAIRED_STATTYPE_TOTAL_ORIGINAL_US = 1,
+ UFS_IMPAIRED_STATTYPE_TOTAL_DELAYED_US = 2,
+ UFS_IMPAIRED_STATTYPE_TOTAL_DELAY_ERROR = 3,
+ UFS_IMPAIRED_STATTYPE_MAX_DELAY = 4,
+ UFS_IMPAIRED_STATTYPE_MAX_ERROR = 5,
+ UFS_IMPAIRED_STATTYPE_COUNT = 6,
+};
+
+struct ufs_impaired_io {
+ /* Delay control */
+ u32 delay[UFS_IMPAIRED_DELAYTYPE_COUNT];
+
+ /* Statistics */
+ s64 stat[UFS_IMPAIRED_STATTYPE_COUNT];
+
+ /* Skip count */
+ u32 skip_count;
+};
+
+struct ufs_impaired_storage {
+ u8 enabled;
+#ifdef CONFIG_SCSI_UFS_IMPAIRED_FRAGFTL
+ u8 emulate_fragftl;
+#endif
+ struct ufs_impaired_io io[UFS_IMPAIRED_OPTYPE_COUNT];
+};
+
+extern void ufs_impaired_init(struct ufs_hba *hba);
+extern void ufs_impaired_exit(struct ufs_hba *hba);
+extern void ufs_impaired_init_sysfs(struct ufs_hba *hba);
+extern void ufs_impaired_delay_completion(struct ufs_hba *hba,
+ struct ufshcd_lrb *lrbp);
+
+#endif /* End of Header */
diff --git a/drivers/scsi/ufs/ufs-qcom-debugfs.c b/drivers/scsi/ufs/ufs-qcom-debugfs.c
index db4ecec..26147fe 100644
--- a/drivers/scsi/ufs/ufs-qcom-debugfs.c
+++ b/drivers/scsi/ufs/ufs-qcom-debugfs.c
@@ -186,6 +186,7 @@ static const struct file_operations ufs_qcom_dbg_testbus_cfg_desc = {
.open = ufs_qcom_dbg_testbus_cfg_open,
.read = seq_read,
.write = ufs_qcom_dbg_testbus_cfg_write,
+ .release = single_release,
};
static int ufs_qcom_dbg_testbus_bus_read(void *data, u64 *attr_val)
@@ -240,6 +241,7 @@ static int ufs_qcom_dbg_dbg_regs_open(struct inode *inode,
static const struct file_operations ufs_qcom_dbg_dbg_regs_desc = {
.open = ufs_qcom_dbg_dbg_regs_open,
.read = seq_read,
+ .release = single_release,
};
static int ufs_qcom_dbg_pm_qos_show(struct seq_file *file, void *data)
@@ -273,6 +275,7 @@ static int ufs_qcom_dbg_pm_qos_open(struct inode *inode,
static const struct file_operations ufs_qcom_dbg_pm_qos_desc = {
.open = ufs_qcom_dbg_pm_qos_open,
.read = seq_read,
+ .release = single_release,
};
void ufs_qcom_dbg_add_debugfs(struct ufs_hba *hba, struct dentry *root)
diff --git a/drivers/scsi/ufs/ufs-qcom-ice.c b/drivers/scsi/ufs/ufs-qcom-ice.c
index d4fe6ee..ef682f3 100644
--- a/drivers/scsi/ufs/ufs-qcom-ice.c
+++ b/drivers/scsi/ufs/ufs-qcom-ice.c
@@ -197,7 +197,6 @@ static void ufs_qcom_ice_cfg_work(struct work_struct *work)
qcom_host->req_pending = NULL;
qcom_host->work_pending = false;
spin_unlock_irqrestore(&qcom_host->ice_work_lock, flags);
-
}
/**
@@ -411,7 +410,6 @@ int ufs_qcom_ice_cfg_start(struct ufs_qcom_host *qcom_host,
__func__);
if (!qcom_host->work_pending) {
-
qcom_host->req_pending = cmd->request;
if (!queue_work(ice_workqueue,
&qcom_host->ice_cfg_work)) {
diff --git a/drivers/scsi/ufs/ufs-qcom.c b/drivers/scsi/ufs/ufs-qcom.c
index 5b23d5c..2c717ab 100644
--- a/drivers/scsi/ufs/ufs-qcom.c
+++ b/drivers/scsi/ufs/ufs-qcom.c
@@ -1,5 +1,5 @@
/*
- * Copyright (c) 2013-2018, Linux Foundation. All rights reserved.
+ * Copyright (c) 2013-2017, Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -1516,7 +1516,6 @@ static void ufs_qcom_set_caps(struct ufs_hba *hba)
if (!host->disable_lpm) {
hba->caps |= UFSHCD_CAP_CLK_GATING;
hba->caps |= UFSHCD_CAP_HIBERN8_WITH_CLK_GATING;
- hba->caps |= UFSHCD_CAP_CLK_SCALING;
}
hba->caps |= UFSHCD_CAP_AUTO_BKOPS_SUSPEND;
diff --git a/drivers/scsi/ufs/ufs.h b/drivers/scsi/ufs/ufs.h
index 0ae51b9..66675a3 100644
--- a/drivers/scsi/ufs/ufs.h
+++ b/drivers/scsi/ufs/ufs.h
@@ -3,6 +3,7 @@
*
* This code is based on drivers/scsi/ufs/ufs.h
* Copyright (C) 2011-2013 Samsung India Software Operations
+ # Copyright (c) 2017-2018 Samsung Electronics Co., Ltd.
*
* Authors:
* Santosh Yaraganavi <santosh.sy@samsung.com>
@@ -143,6 +144,8 @@ enum {
UPIU_QUERY_FUNC_STANDARD_WRITE_REQUEST = 0x81,
};
+#define UFSHCD_DEFAULT_PE_CYCLE 3000
+
enum desc_header_offset {
QUERY_DESC_LENGTH_OFFSET = 0x00,
QUERY_DESC_DESC_TYPE_OFFSET = 0x01,
@@ -155,6 +158,18 @@ enum ufs_desc_def_size {
QUERY_DESC_INTERCONNECT_DEF_SIZE = 0x06,
QUERY_DESC_GEOMETRY_DEF_SIZE = 0x44,
QUERY_DESC_POWER_DEF_SIZE = 0x62,
+ QUERY_DESC_HEALTH_DEF_SIZE = 0x25,
+};
+
+enum ufs_health_offset {
+ UFSHCD_HEALTH_LEN_OFFSET = 0x0,
+ UFSHCD_HEALTH_TYPE_OFFSET = 0x1,
+ UFSHCD_HEALTH_EOL_OFFSET = 0x2,
+ UFSHCD_HEALTH_LIFEA_OFFSET = 0x3,
+ UFSHCD_HEALTH_LIFEB_OFFSET = 0x4,
+ UFSHCD_HEALTH_ERASE_OFFSET = 0x0D,
+ UFSHCD_HEALTH_WRITE_OFFSET = 0x15,
+ UFSHCD_HEALTH_LIFEC_OFFSET = 0x24,
};
/* Unit descriptor parameters offsets in bytes*/
@@ -175,6 +190,9 @@ enum unit_desc_param {
UNIT_DESC_PARAM_PHY_MEM_RSRC_CNT = 0x18,
UNIT_DESC_PARAM_CTX_CAPABILITIES = 0x20,
UNIT_DESC_PARAM_LARGE_UNIT_SIZE_M1 = 0x22,
+ UNIT_DESC_HPB_LU_MAX_ACTIVE_REGIONS = 0x23,
+ UNIT_DESC_HPB_LU_PIN_REGION_START_OFFSET = 0x25,
+ UNIT_DESC_HPB_LU_NUM_PIN_REGIONS = 0x27,
};
/* Device descriptor parameters offsets in bytes*/
@@ -206,7 +224,18 @@ enum device_desc_param {
DEVICE_DESC_PARAM_UD_LEN = 0x1B,
DEVICE_DESC_PARAM_RTT_CAP = 0x1C,
DEVICE_DESC_PARAM_FRQ_RTC = 0x1D,
+ DEVICE_DESC_PARAM_FEAT_SUP = 0x1F,
+ DEVICE_DESC_PARAM_HPB_VER = 0x40,
};
+
+enum geometry_desc_param {
+ GEOMETRY_DESC_SEGMENT_SIZE = 0x0D,
+ GEOMETRY_DESC_HPB_REGION_SIZE = 0x48,
+ GEOMETRY_DESC_HPB_NUMBER_LU = 0x49,
+ GEOMETRY_DESC_HPB_SUBREGION_SIZE = 0x4A,
+ GEOMETRY_DESC_HPB_DEVICE_MAX_ACTIVE_REGIONS = 0x4B,
+};
+
/*
* Logical Unit Write Protect
* 00h: LU not write protected
@@ -516,6 +545,11 @@ struct ufs_dev_info {
/* Device deviations from standard UFS device spec. */
unsigned int quirks;
+
+ unsigned int pre_eol_info;
+ unsigned int lifetime_a;
+ unsigned int lifetime_b;
+ unsigned int lifetime_c;
};
#endif /* End of Header */
diff --git a/drivers/scsi/ufs/ufs_quirks.c b/drivers/scsi/ufs/ufs_quirks.c
index a2b98fb..f79cbd4 100644
--- a/drivers/scsi/ufs/ufs_quirks.c
+++ b/drivers/scsi/ufs/ufs_quirks.c
@@ -84,3 +84,50 @@ void ufs_advertise_fixup_device(struct ufs_hba *hba)
out:
kfree(model);
}
+
+static struct ufs_qdepth_fix ufs_qdepth[] = {
+ UFS_QDEPTH("H28S7Q302BMR", 16),
+ UFS_QDEPTH("H28S8Q302CMR", 16),
+ UFS_QDEPTH("KLUCG2K1EA-B0C1", 16),
+ UFS_QDEPTH("KLUDG4U1EA-B0C1", 16),
+ UFS_QDEPTH("128GB-UFS-MT", 16),
+ UFS_QDEPTH("64GB-UFS-MT", 16),
+ END_FIX
+};
+
+int ufs_fix_qdepth_device(struct ufs_hba *hba, struct scsi_device *sdev)
+{
+ u8 str_desc_buf[QUERY_DESC_MAX_SIZE + 1];
+ char *model;
+ struct ufs_qdepth_fix *f;
+ int err;
+
+ model = kmalloc(MAX_MODEL_LEN + 1, GFP_KERNEL);
+ if (!model)
+ goto out;
+
+ memset(str_desc_buf, 0, QUERY_DESC_MAX_SIZE);
+ err = ufshcd_read_string_desc(hba, hba->dev_info.i_product_name,
+ str_desc_buf, QUERY_DESC_MAX_SIZE, ASCII_STD);
+ if (err)
+ goto out;
+
+ str_desc_buf[QUERY_DESC_MAX_SIZE] = '\0';
+ strlcpy(model, (str_desc_buf + QUERY_DESC_HDR_SIZE),
+ min_t(u8, str_desc_buf[QUERY_DESC_LENGTH_OFFSET],
+ MAX_MODEL_LEN));
+ /* Null terminate the model string */
+ model[MAX_MODEL_LEN] = '\0';
+
+ for (f = ufs_qdepth; f->qdepth; f++) {
+ if (STR_PRFX_EQUAL(f->model, model) &&
+ sdev->queue_depth > f->qdepth) {
+ scsi_change_queue_depth(sdev, f->qdepth);
+ dev_err(hba->dev, "Change qdepth to %u for LU=0x%llx\n",
+ sdev->queue_depth, sdev->lun);
+ }
+ }
+out:
+ kfree(model);
+ return sdev->queue_depth;
+}
diff --git a/drivers/scsi/ufs/ufs_quirks.h b/drivers/scsi/ufs/ufs_quirks.h
index 0fa7d93..54297c3 100644
--- a/drivers/scsi/ufs/ufs_quirks.h
+++ b/drivers/scsi/ufs/ufs_quirks.h
@@ -30,6 +30,17 @@
#define UFS_MODEL_TOSHIBA_32GB "THGLF2G8D4KBADR"
#define UFS_MODEL_TOSHIBA_64GB "THGLF2G9D8KBADG"
+struct ufs_qdepth_fix {
+ char *model;
+ unsigned int qdepth;
+};
+
+#define UFS_QDEPTH(_model, _qdepth) \
+ { \
+ .model = (_model), \
+ .qdepth = (_qdepth), \
+ }
+
/**
* ufs_card_fix - ufs device quirk info
* @card: ufs card details
@@ -141,5 +152,5 @@ struct ufs_card_fix {
struct ufs_hba;
void ufs_advertise_fixup_device(struct ufs_hba *hba);
-
+int ufs_fix_qdepth_device(struct ufs_hba *hba, struct scsi_device *sdev);
#endif /* UFS_QUIRKS_H_ */
diff --git a/drivers/scsi/ufs/ufs_test.c b/drivers/scsi/ufs/ufs_test.c
index 2e3997d..d91236e 100644
--- a/drivers/scsi/ufs/ufs_test.c
+++ b/drivers/scsi/ufs/ufs_test.c
@@ -78,6 +78,7 @@ static const struct file_operations ufs_test_ ## test_name ## _ops = { \
.open = ufs_test_ ## test_name ## _open, \
.read = seq_read, \
.write = ufs_test_ ## test_name ## _write, \
+ .release = single_release, \
};
#define add_test(utd, test_name, upper_case_name) \
diff --git a/drivers/scsi/ufs/ufshcd.c b/drivers/scsi/ufs/ufshcd.c
index d427fb3..c417738 100644
--- a/drivers/scsi/ufs/ufshcd.c
+++ b/drivers/scsi/ufs/ufshcd.c
@@ -4,6 +4,8 @@
* This code is based on drivers/scsi/ufs/ufshcd.c
* Copyright (C) 2011-2013 Samsung India Software Operations
* Copyright (c) 2013-2018, The Linux Foundation. All rights reserved.
+ * Copyright (c) 2017-2018 Samsung Electronics Co., Ltd.
+ * Copyright (C) 2018, Google, Inc.
*
* Authors:
* Santosh Yaraganavi <santosh.sy@samsung.com>
@@ -52,6 +54,69 @@
#define CREATE_TRACE_POINTS
#include <trace/events/ufs.h>
+static void ufshcd_update_slowio_min_us(struct ufs_hba *hba)
+{
+ enum ufshcd_slowio_optype i;
+ u64 us;
+
+ hba->slowio_min_us = hba->slowio[UFSHCD_SLOWIO_READ][UFSHCD_SLOWIO_US];
+ for (i = UFSHCD_SLOWIO_WRITE; i < UFSHCD_SLOWIO_OP_MAX; i++) {
+ us = hba->slowio[i][UFSHCD_SLOWIO_US];
+ if (us < hba->slowio_min_us)
+ hba->slowio_min_us = us;
+ }
+}
+
+static enum ufshcd_slowio_optype ufshcd_get_slowio_optype(u8 opcode)
+{
+ if (opcode == READ_10 || opcode == READ_16)
+ return UFSHCD_SLOWIO_READ;
+ else if (opcode == WRITE_10 || opcode == WRITE_16)
+ return UFSHCD_SLOWIO_WRITE;
+ else if (opcode == UNMAP)
+ return UFSHCD_SLOWIO_UNMAP;
+ else if (opcode == SYNCHRONIZE_CACHE)
+ return UFSHCD_SLOWIO_SYNC;
+ return UFSHCD_SLOWIO_OP_MAX;
+}
+
+static void ufshcd_log_slowio(struct ufs_hba *hba,
+ struct ufshcd_lrb *lrbp, s64 iotime_us)
+{
+ sector_t lba = ULONG_MAX;
+ u32 transfer_len = UINT_MAX;
+ u8 opcode = 0xff;
+ char opcode_str[16];
+ u64 slowio_cnt = 0;
+ enum ufshcd_slowio_optype optype;
+
+ /* For common case */
+ if (likely(iotime_us < hba->slowio_min_us))
+ return;
+
+ if (lrbp->cmd) {
+ opcode = (u8)(*lrbp->cmd->cmnd);
+ optype = ufshcd_get_slowio_optype(opcode);
+ if (optype < UFSHCD_SLOWIO_OP_MAX) {
+ if (iotime_us < hba->slowio[optype][UFSHCD_SLOWIO_US])
+ return;
+ slowio_cnt = ++hba->slowio[optype][UFSHCD_SLOWIO_CNT];
+ }
+ if (is_read_opcode(opcode) || is_write_opcode(opcode) ||
+ is_unmap_opcode(opcode)) {
+ if (lrbp->cmd->request && lrbp->cmd->request->bio) {
+ lba = scsi_get_lba(lrbp->cmd);
+ transfer_len = scsi_get_bytes(lrbp->cmd);
+ }
+ }
+ }
+ snprintf(opcode_str, 16, "%02x: %s", opcode, parse_opcode(opcode));
+ dev_err_ratelimited(hba->dev,
+ "Slow UFS (%lld): time = %lld us, opcode = %16s, lba = %ld, "
+ "len = %u\n",
+ slowio_cnt, iotime_us, opcode_str, lba, transfer_len);
+}
+
#ifdef CONFIG_DEBUG_FS
static int ufshcd_tag_req_type(struct request *rq)
@@ -60,8 +125,11 @@ static int ufshcd_tag_req_type(struct request *rq)
if (!rq || !(rq->cmd_type & REQ_TYPE_FS))
rq_type = TS_NOT_SUPPORTED;
- else if (rq->cmd_flags & REQ_PREFLUSH)
+ else if ((rq->cmd_flags & REQ_PREFLUSH) ||
+ (req_op(rq) == REQ_OP_FLUSH))
rq_type = TS_FLUSH;
+ else if (req_op(rq) == REQ_OP_DISCARD)
+ rq_type = TS_DISCARD;
else if (rq_data_dir(rq) == READ)
rq_type = (rq->cmd_flags & REQ_URGENT) ?
TS_URGENT_READ : TS_READ;
@@ -85,9 +153,6 @@ static void ufshcd_update_tag_stats(struct ufs_hba *hba, int tag)
u64 **tag_stats = hba->ufs_stats.tag_stats;
int rq_type;
- if (!hba->ufs_stats.enabled)
- return;
-
tag_stats[tag][TS_TAG]++;
if (!rq || !(rq->cmd_type & REQ_TYPE_FS))
return;
@@ -107,12 +172,14 @@ static void ufshcd_update_tag_stats_completion(struct ufs_hba *hba,
hba->ufs_stats.q_depth--;
}
-static void update_req_stats(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+static void update_req_stats(struct ufs_hba *hba, struct ufshcd_lrb *lrbp,
+ s64 delta)
{
int rq_type;
struct request *rq = lrbp->cmd ? lrbp->cmd->request : NULL;
- s64 delta = ktime_us_delta(lrbp->complete_time_stamp,
- lrbp->issue_time_stamp);
+
+ /* Log for slow I/O */
+ ufshcd_log_slowio(hba, lrbp, delta);
/* update general request statistics */
if (hba->ufs_stats.req_stats[TS_TAG].count == 0)
@@ -146,6 +213,51 @@ ufshcd_update_query_stats(struct ufs_hba *hba, enum query_opcode opcode, u8 idn)
hba->ufs_stats.query_stats_arr[opcode][idn]++;
}
+static void
+__update_io_stat(struct ufs_hba *hba, struct ufshcd_io_stat *io_stat,
+ u32 transfer_len, int is_start)
+{
+ if (is_start) {
+ u64 diff;
+ io_stat->req_count_started++;
+ io_stat->total_bytes_started += transfer_len;
+ diff = io_stat->req_count_started -
+ io_stat->req_count_completed;
+ if (diff > io_stat->max_diff_req_count) {
+ io_stat->max_diff_req_count = diff;
+ }
+ diff = io_stat->total_bytes_started -
+ io_stat->total_bytes_completed;
+ if (diff > io_stat->max_diff_total_bytes) {
+ io_stat->max_diff_total_bytes = diff;
+ }
+ } else {
+ io_stat->req_count_completed++;
+ io_stat->total_bytes_completed += transfer_len;
+ }
+}
+
+static void
+update_io_stat(struct ufs_hba *hba, int tag, int is_start)
+{
+ struct ufshcd_lrb *lrbp = &hba->lrb[tag];
+ u8 opcode;
+ u32 transfer_len;
+
+ if (!lrbp->cmd)
+ return;
+ opcode = (u8)(*lrbp->cmd->cmnd);
+ if (!is_read_opcode(opcode) && !is_write_opcode(opcode))
+ return;
+
+ transfer_len = scsi_get_bytes(lrbp->cmd);
+
+ __update_io_stat(hba, &hba->ufs_stats.io_readwrite, transfer_len,
+ is_start);
+ __update_io_stat(hba, is_read_opcode(opcode) ? &hba->ufs_stats.io_read:
+ &hba->ufs_stats.io_write, transfer_len, is_start);
+}
+
#else
static inline void ufshcd_update_tag_stats(struct ufs_hba *hba, int tag)
{
@@ -161,8 +273,10 @@ static inline void ufshcd_update_error_stats(struct ufs_hba *hba, int type)
}
static inline
-void update_req_stats(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+void update_req_stats(struct ufs_hba *hba, struct ufshcd_lrb *lrbp, s64 delta)
{
+ /* Log for slow I/O */
+ ufshcd_log_slowio(hba, lrbp, delta);
}
static inline
@@ -170,6 +284,12 @@ void ufshcd_update_query_stats(struct ufs_hba *hba,
enum query_opcode opcode, u8 idn)
{
}
+
+static void
+update_io_stat(struct ufs_hba *hba, int tag, int is_start)
+{
+}
+
#endif
static void ufshcd_update_uic_error_cnt(struct ufs_hba *hba, u32 reg, int type)
@@ -252,6 +372,9 @@ static void ufshcd_update_uic_error_cnt(struct ufs_hba *hba, u32 reg, int type)
#define UFSHCD_DEFAULT_LANES_PER_DIRECTION 2
+/* for manual gc */
+#define UFSHCD_MANUAL_GC_HOLD_HIBERN8 2000 /* 2 seconds */
+
#define ufshcd_toggle_vreg(_dev, _vreg, _on) \
({ \
int _ret; \
@@ -609,7 +732,7 @@ static void ufshcd_cmd_log_init(struct ufs_hba *hba)
static void __ufshcd_cmd_log(struct ufs_hba *hba, char *str, char *cmd_type,
unsigned int tag, u8 cmd_id, u8 idn, u8 lun,
- sector_t lba, int transfer_len)
+ sector_t lba, u32 transfer_len)
{
struct ufshcd_cmd_log_entry *entry;
@@ -628,6 +751,9 @@ static void __ufshcd_cmd_log(struct ufs_hba *hba, char *str, char *cmd_type,
entry->tag = tag;
entry->tstamp = ktime_get();
entry->outstanding_reqs = hba->outstanding_reqs;
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ entry->delayed_reqs = hba->delayed_reqs;
+#endif
entry->seq_num = hba->cmd_log.seq_num;
hba->cmd_log.seq_num++;
hba->cmd_log.pos =
@@ -667,13 +793,23 @@ static void ufshcd_print_cmd_log(struct ufs_hba *hba)
pos = (pos + 1) % UFSHCD_MAX_CMD_LOGGING;
if (ktime_to_us(p->tstamp)) {
- pr_err("%s: %s: seq_no=%u lun=0x%x cmd_id=0x%02x lba=0x%llx txfer_len=%d tag=%u, doorbell=0x%x outstanding=0x%x idn=%d time=%lld us\n",
+#ifndef CONFIG_SCSI_UFS_IMPAIRED
+ pr_err("%s: %s: seq_no=%u lun=0x%x cmd_id=0x%02x lba=0x%llx txfer_len=%u tag=%u, doorbell=0x%x outstanding=0x%x idn=%d time=%lld us\n",
p->cmd_type, p->str, p->seq_num,
p->lun, p->cmd_id, (unsigned long long)p->lba,
p->transfer_len, p->tag, p->doorbell,
p->outstanding_reqs, p->idn,
ktime_to_us(p->tstamp));
- usleep_range(1000, 1100);
+#else
+ pr_err("%s: %s: seq_no=%u lun=0x%x cmd_id=0x%02x lba=0x%llx txfer_len=%u tag=%u, doorbell=0x%x outstanding=0x%x delayed=0x%x idn=%d time=%lld us\n",
+ p->cmd_type, p->str, p->seq_num,
+ p->lun, p->cmd_id, (unsigned long long)p->lba,
+ p->transfer_len, p->tag, p->doorbell,
+ p->outstanding_reqs, p->delayed_reqs, p->idn,
+ ktime_to_us(p->tstamp));
+#endif
+
+ usleep_range(1000, 1100);
}
}
}
@@ -684,7 +820,7 @@ static void ufshcd_cmd_log_init(struct ufs_hba *hba)
static void __ufshcd_cmd_log(struct ufs_hba *hba, char *str, char *cmd_type,
unsigned int tag, u8 cmd_id, u8 idn, u8 lun,
- sector_t lba, int transfer_len)
+ sector_t lba, u32 transfer_len)
{
struct ufshcd_cmd_log_entry entry;
@@ -719,23 +855,23 @@ static inline void ufshcd_cond_add_cmd_trace(struct ufs_hba *hba,
char *cmd_type = NULL;
u8 opcode = 0;
u8 cmd_id = 0, idn = 0;
- sector_t lba = 0;
- int transfer_len = 0;
+ sector_t lba = ULONG_MAX;
+ u32 transfer_len = UINT_MAX;
lrbp = &hba->lrb[tag];
if (lrbp->cmd) { /* data phase exists */
opcode = (u8)(*lrbp->cmd->cmnd);
- if ((opcode == READ_10) || (opcode == WRITE_10)) {
+ if (is_read_opcode(opcode) || is_write_opcode(opcode) ||
+ is_unmap_opcode(opcode)) {
/*
- * Currently we only fully trace read(10) and write(10)
- * commands
+ * Currently we only fully trace read(10), write(10),
+ * read(16), write(16), unmap commands
*/
- if (lrbp->cmd->request && lrbp->cmd->request->bio)
- lba =
- lrbp->cmd->request->bio->bi_iter.bi_sector;
- transfer_len = be32_to_cpu(
- lrbp->ucd_req_ptr->sc.exp_data_transfer_len);
+ if (lrbp->cmd->request && lrbp->cmd->request->bio) {
+ lba = scsi_get_lba(lrbp->cmd);
+ transfer_len = scsi_get_bytes(lrbp->cmd);
+ }
}
}
@@ -820,6 +956,9 @@ static inline void __ufshcd_print_host_regs(struct ufs_hba *hba, bool no_sleep)
dev_err(hba->dev,
"hba->outstanding_reqs = 0x%x, hba->outstanding_tasks = 0x%x",
(u32)hba->outstanding_reqs, (u32)hba->outstanding_tasks);
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ dev_err(hba->dev, "hba->delayed_reqs=0x%x", (u32)hba->delayed_reqs);
+#endif
dev_err(hba->dev,
"last_hibern8_exit_tstamp at %lld us, hibern8_exit_cnt = %d",
ktime_to_us(hba->ufs_stats.last_hibern8_exit_tstamp),
@@ -927,8 +1066,31 @@ static void ufshcd_print_host_state(struct ufs_hba *hba)
return;
dev_err(hba->dev, "UFS Host state=%d\n", hba->ufshcd_state);
+ if (hba->sdev_ufs_device) {
+ dev_err(hba->dev, " vendor = %.8s\n",
+ hba->sdev_ufs_device->vendor);
+ dev_err(hba->dev, " model = %.16s\n",
+ hba->sdev_ufs_device->model);
+ dev_err(hba->dev, " rev = %.4s\n",
+ hba->sdev_ufs_device->rev);
+ dev_err(hba->dev, " nutrs = %d\n",
+ hba->nutrs);
+ dev_err(hba->dev, " queue_depth = %u\n",
+ hba->sdev_ufs_device->queue_depth);
+ }
+ dev_err(hba->dev, " pre_eol_info = 0x%x\n",
+ hba->dev_info.pre_eol_info);
+ dev_err(hba->dev, " LifeTimeA = 0x%x\n",
+ hba->dev_info.lifetime_a);
+ dev_err(hba->dev, " LifeTimeB = 0x%x\n",
+ hba->dev_info.lifetime_b);
+ dev_err(hba->dev, " LifeTimeC = %u\n",
+ hba->dev_info.lifetime_c);
dev_err(hba->dev, "lrb in use=0x%lx, outstanding reqs=0x%lx tasks=0x%lx\n",
hba->lrb_in_use, hba->outstanding_reqs, hba->outstanding_tasks);
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ dev_err(hba->dev, "delayed reqs=0x%lx\n", hba->delayed_reqs);
+#endif
dev_err(hba->dev, "saved_err=0x%x, saved_uic_err=0x%x, saved_ce_err=0x%x\n",
hba->saved_err, hba->saved_uic_err, hba->saved_ce_err);
dev_err(hba->dev, "Device power mode=%d, UIC link state=%d\n",
@@ -1513,6 +1675,8 @@ static void ufshcd_ungate_work(struct work_struct *work)
ufshcd_hba_vreg_set_hpm(hba);
ufshcd_enable_clocks(hba);
+ ufshcd_enable_irq(hba);
+
/* Exit from hibern8 */
if (ufshcd_can_hibern8_during_gating(hba)) {
/* Prevent gating in this path */
@@ -1670,6 +1834,8 @@ static void ufshcd_gate_work(struct work_struct *work)
ufshcd_set_link_hibern8(hba);
}
+ ufshcd_disable_irq(hba);
+
/*
* If auto hibern8 is supported then the link will already
* be in hibern8 state and the ref clock can be gated.
@@ -2419,7 +2585,19 @@ static void ufshcd_clk_scaling_update_busy(struct ufs_hba *hba)
static inline
int ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag)
{
- int ret = 0;
+ if (hba->lrb[task_tag].cmd) {
+ u8 opcode = (u8)(*hba->lrb[task_tag].cmd->cmnd);
+
+ if (opcode == SECURITY_PROTOCOL_OUT && hba->security_in) {
+ hba->security_in--;
+ } else if (opcode == SECURITY_PROTOCOL_IN) {
+ if (hba->security_in) {
+ WARN_ON(1);
+ return -EINVAL;
+ }
+ hba->security_in++;
+ }
+ }
hba->lrb[task_tag].issue_time_stamp = ktime_get();
hba->lrb[task_tag].complete_time_stamp = ktime_set(0, 0);
@@ -2431,7 +2609,8 @@ int ufshcd_send_command(struct ufs_hba *hba, unsigned int task_tag)
ufshcd_cond_add_cmd_trace(hba, task_tag,
hba->lrb[task_tag].cmd ? "scsi_send" : "dev_cmd_send");
ufshcd_update_tag_stats(hba, task_tag);
- return ret;
+ update_io_stat(hba, task_tag, 1);
+ return 0;
}
/**
@@ -2932,9 +3111,15 @@ static int ufshcd_compose_upiu(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
switch (lrbp->command_type) {
case UTP_CMD_TYPE_SCSI:
if (likely(lrbp->cmd)) {
+ if (hba->ufshpb_state == HPB_PRESENT &&
+ hba->issue_ioctl == true)
+ lrbp->lun = 0x7F;
ret = ufshcd_prepare_req_desc_hdr(hba, lrbp,
&upiu_flags, lrbp->cmd->sc_data_direction);
ufshcd_prepare_utp_scsi_cmd_upiu(lrbp, upiu_flags);
+ if (hba->ufshpb_state == HPB_PRESENT &&
+ hba->issue_ioctl == false)
+ ufshpb_prep_fn(hba, lrbp);
} else {
ret = -EINVAL;
}
@@ -3168,15 +3353,6 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
/* Vote PM QoS for the request */
ufshcd_vops_pm_qos_req_start(hba, cmd->request);
- /* IO svc time latency histogram */
- if (hba->latency_hist_enabled &&
- (cmd->request->cmd_type == REQ_TYPE_FS)) {
- cmd->request->lat_hist_io_start = ktime_get();
- cmd->request->lat_hist_enabled = 1;
- } else {
- cmd->request->lat_hist_enabled = 0;
- }
-
WARN_ON(hba->clk_gating.state != CLKS_ON);
lrbp = &hba->lrb[tag];
@@ -3191,6 +3367,10 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
lrbp->command_type = UTP_CMD_TYPE_SCSI;
lrbp->req_abort_skip = false;
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ INIT_LIST_HEAD(&lrbp->list);
+#endif
+
/* form UPIU before issuing the command */
err = ufshcd_compose_upiu(hba, lrbp);
if (err) {
@@ -3246,7 +3426,13 @@ static int ufshcd_queuecommand(struct Scsi_Host *host, struct scsi_cmnd *cmd)
ufshcd_vops_crypto_engine_cfg_end(hba, lrbp, cmd->request);
dev_err(hba->dev, "%s: failed sending command, %d\n",
__func__, err);
- err = DID_ERROR;
+ if (err == -EINVAL) {
+ set_host_byte(cmd, DID_ERROR);
+ if (has_read_lock)
+ ufshcd_put_read_lock(hba);
+ cmd->scsi_done(cmd);
+ return 0;
+ }
goto out;
}
@@ -3804,8 +3990,10 @@ int ufshcd_query_descriptor(struct ufs_hba *hba,
int retries;
for (retries = QUERY_REQ_RETRIES; retries > 0; retries--) {
+ down_read(&hba->query_lock);
err = __ufshcd_query_descriptor(hba, opcode, idn, index,
selector, desc_buf, buf_len);
+ up_read(&hba->query_lock);
if (!err || err == -EINVAL)
break;
}
@@ -3815,47 +4003,6 @@ int ufshcd_query_descriptor(struct ufs_hba *hba,
EXPORT_SYMBOL(ufshcd_query_descriptor);
/**
- * ufshcd_read_desc_length - read the specified descriptor length from header
- * @hba: Pointer to adapter instance
- * @desc_id: descriptor idn value
- * @desc_index: descriptor index
- * @desc_length: pointer to variable to read the length of descriptor
- *
- * Return 0 in case of success, non-zero otherwise
- */
-static int ufshcd_read_desc_length(struct ufs_hba *hba,
- enum desc_idn desc_id,
- int desc_index,
- int *desc_length)
-{
- int ret;
- u8 header[QUERY_DESC_HDR_SIZE];
- int header_len = QUERY_DESC_HDR_SIZE;
-
- if (desc_id >= QUERY_DESC_IDN_MAX)
- return -EINVAL;
-
- ret = ufshcd_query_descriptor(hba, UPIU_QUERY_OPCODE_READ_DESC,
- desc_id, desc_index, 0, header,
- &header_len);
-
- if (ret) {
- dev_err(hba->dev, "%s: Failed to get descriptor header id %d",
- __func__, desc_id);
- return ret;
- } else if (desc_id != header[QUERY_DESC_DESC_TYPE_OFFSET]) {
- dev_warn(hba->dev, "%s: descriptor header id %d and desc_id %d mismatch",
- __func__, header[QUERY_DESC_DESC_TYPE_OFFSET],
- desc_id);
- ret = -EINVAL;
- }
-
- *desc_length = header[QUERY_DESC_LENGTH_OFFSET];
- return ret;
-
-}
-
-/**
* ufshcd_map_desc_id_to_length - map descriptor IDN to its length
* @hba: Pointer to adapter instance
* @desc_id: descriptor idn value
@@ -3885,6 +4032,9 @@ int ufshcd_map_desc_id_to_length(struct ufs_hba *hba,
case QUERY_DESC_IDN_INTERCONNECT:
*desc_len = hba->desc_size.interc_desc;
break;
+ case QUERY_DESC_IDN_HEALTH:
+ *desc_len = hba->desc_size.health_desc;
+ break;
case QUERY_DESC_IDN_STRING:
*desc_len = QUERY_DESC_MAX_SIZE;
break;
@@ -3970,8 +4120,8 @@ static int ufshcd_read_desc_param(struct ufs_hba *hba,
}
/* Check wherher we will not copy more data, than available */
- if (is_kmalloc && param_size > buff_len)
- param_size = buff_len;
+ if (is_kmalloc && param_offset + param_size > buff_len)
+ param_size = buff_len - param_offset;
if (is_kmalloc)
memcpy(param_read_buf, &desc_buf[param_offset], param_size);
@@ -4003,6 +4153,44 @@ int ufshcd_read_device_desc(struct ufs_hba *hba, u8 *buf, u32 size)
}
/**
+ * ufshcd_read_desc_length - read the specified descriptor length from header
+ * @hba: Pointer to adapter instance
+ * @desc_id: descriptor idn value
+ * @desc_index: descriptor index
+ * @desc_length: pointer to variable to read the length of descriptor
+ *
+ * Return 0 in case of success, non-zero otherwise
+ */
+static int ufshcd_read_desc_length(struct ufs_hba *hba,
+ enum desc_idn desc_id,
+ int desc_index,
+ int *desc_length)
+{
+ int ret;
+ u8 header[QUERY_DESC_HDR_SIZE];
+ int header_len = QUERY_DESC_HDR_SIZE;
+
+ if (desc_id >= QUERY_DESC_IDN_MAX)
+ return -EINVAL;
+
+ ret = ufshcd_read_desc_param(hba, desc_id, desc_index,
+ 0, header, header_len);
+ if (ret) {
+ dev_err(hba->dev, "%s: Failed to get descriptor header id %d",
+ __func__, desc_id);
+ return ret;
+ } else if (desc_id != header[QUERY_DESC_DESC_TYPE_OFFSET]) {
+ dev_warn(hba->dev, "%s: descriptor header id %d and desc_id %d mismatch",
+ __func__, header[QUERY_DESC_DESC_TYPE_OFFSET],
+ desc_id);
+ ret = -EINVAL;
+ }
+
+ *desc_length = header[QUERY_DESC_LENGTH_OFFSET];
+ return ret;
+}
+
+/**
* ufshcd_read_string_desc - read string descriptor
* @hba: pointer to adapter instance
* @desc_index: descriptor index
@@ -5332,6 +5520,7 @@ static void ufshcd_set_queue_depth(struct scsi_device *sdev)
int ret = 0;
u8 lun_qdepth;
struct ufs_hba *hba;
+ bool fixable_qdepth = false;
hba = shost_priv(sdev->host);
@@ -5343,17 +5532,23 @@ static void ufshcd_set_queue_depth(struct scsi_device *sdev)
sizeof(lun_qdepth));
/* Some WLUN doesn't support unit descriptor */
- if (ret == -EOPNOTSUPP)
+ if (ret == -EOPNOTSUPP ||
+ (sdev->lun ==
+ ufshcd_upiu_wlun_to_scsi_wlun(UFS_UPIU_RPMB_WLUN))) {
lun_qdepth = 1;
- else if (!lun_qdepth)
+ } else if (!lun_qdepth) {
/* eventually, we can figure out the real queue depth */
lun_qdepth = hba->nutrs;
- else
+ fixable_qdepth = true;
+ } else {
lun_qdepth = min_t(int, lun_qdepth, hba->nutrs);
+ }
- dev_dbg(hba->dev, "%s: activate tcq with queue depth %d\n",
+ dev_err(hba->dev, "%s: activate tcq with queue depth %d\n",
__func__, lun_qdepth);
scsi_change_queue_depth(sdev, lun_qdepth);
+ if (fixable_qdepth)
+ ufs_fix_qdepth_device(hba, sdev);
}
/*
@@ -5421,6 +5616,9 @@ static inline void ufshcd_get_lu_power_on_wp_status(struct ufs_hba *hba,
static int ufshcd_slave_alloc(struct scsi_device *sdev)
{
struct ufs_hba *hba;
+ int buff_len = QUERY_DESC_HEALTH_DEF_SIZE;
+ u8 desc_buf[QUERY_DESC_HEALTH_DEF_SIZE];
+ int err;
hba = shost_priv(sdev->host);
@@ -5433,13 +5631,26 @@ static int ufshcd_slave_alloc(struct scsi_device *sdev)
/* REPORT SUPPORTED OPERATION CODES is not supported */
sdev->no_report_opcodes = 1;
- /* WRITE_SAME command is not supported*/
+ /* WRITE_SAME command is not supported */
sdev->no_write_same = 1;
ufshcd_set_queue_depth(sdev);
ufshcd_get_lu_power_on_wp_status(hba, sdev);
+ err = ufshcd_read_desc(hba, QUERY_DESC_IDN_HEALTH, 0,
+ desc_buf, buff_len);
+ if (!err) {
+ hba->dev_info.pre_eol_info =
+ (u8)desc_buf[UFSHCD_HEALTH_EOL_OFFSET];
+ hba->dev_info.lifetime_a =
+ (u8)desc_buf[UFSHCD_HEALTH_LIFEA_OFFSET];
+ hba->dev_info.lifetime_b =
+ (u8)desc_buf[UFSHCD_HEALTH_LIFEB_OFFSET];
+ hba->dev_info.lifetime_c =
+ (u8)desc_buf[UFSHCD_HEALTH_LIFEC_OFFSET];
+ }
+
return 0;
}
@@ -5456,7 +5667,9 @@ static int ufshcd_change_queue_depth(struct scsi_device *sdev, int depth)
if (depth > hba->nutrs)
depth = hba->nutrs;
- return scsi_change_queue_depth(sdev, depth);
+
+ scsi_change_queue_depth(sdev, depth);
+ return ufs_fix_qdepth_device(hba, sdev);
}
/**
@@ -5479,6 +5692,8 @@ static int ufshcd_slave_configure(struct scsi_device *sdev)
sdev->autosuspend_delay = UFSHCD_AUTO_SUSPEND_DELAY_MS;
sdev->use_rpm_auto = 1;
+ if (sdev->lun < UFS_UPIU_MAX_GENERAL_LUN)
+ hba->sdev_ufs_lu[sdev->lun] = sdev;
return 0;
}
@@ -5634,6 +5849,9 @@ ufshcd_transfer_rsp_status(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
if (schedule_work(&hba->eeh_work))
pm_runtime_get_noresume(hba->dev);
}
+ if (hba->ufshpb_state == HPB_PRESENT &&
+ scsi_status == SAM_STAT_GOOD)
+ ufshpb_rsp_upiu(hba, lrbp);
break;
case UPIU_TRANSACTION_REJECT_UPIU:
/* TODO: handle Reject UPIU Response */
@@ -5757,6 +5975,7 @@ void ufshcd_abort_outstanding_transfer_requests(struct ufs_hba *hba, int result)
u8 index;
struct ufshcd_lrb *lrbp;
struct scsi_cmnd *cmd;
+ s64 delta_us;
if (!hba->outstanding_reqs)
return;
@@ -5773,11 +5992,13 @@ void ufshcd_abort_outstanding_transfer_requests(struct ufs_hba *hba, int result)
/* Clear pending transfer requests */
ufshcd_clear_cmd(hba, index);
ufshcd_outstanding_req_clear(hba, index);
- clear_bit_unlock(index, &hba->lrb_in_use);
lrbp->complete_time_stamp = ktime_get();
- update_req_stats(hba, lrbp);
+ delta_us = ktime_us_delta(lrbp->complete_time_stamp,
+ lrbp->issue_time_stamp);
+ update_req_stats(hba, lrbp, delta_us);
/* Mark completed command as NULL in LRB */
lrbp->cmd = NULL;
+ clear_bit_unlock(index, &hba->lrb_in_use);
ufshcd_release_all(hba);
if (cmd->request) {
/*
@@ -5806,66 +6027,77 @@ void ufshcd_abort_outstanding_transfer_requests(struct ufs_hba *hba, int result)
}
/**
+ * ufshcd_complete_lrb - complete a UFS command
+ * @hba: per adapter instance
+ * @lrbp: pointer to local reference block to complete
+ */
+void ufshcd_complete_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+{
+ struct scsi_cmnd *cmd = lrbp->cmd;
+ int index = lrbp - hba->lrb;
+ s64 delta_us = ktime_us_delta(lrbp->complete_time_stamp,
+ lrbp->issue_time_stamp);
+
+ ufshcd_cond_add_cmd_trace(hba, index, "scsi_cmpl");
+ ufshcd_update_tag_stats_completion(hba, cmd);
+ update_io_stat(hba, index, 0);
+ update_req_stats(hba, lrbp, delta_us);
+
+ /* Update IO svc time latency histogram */
+ if (hba->latency_hist_enabled &&
+ (cmd->request->cmd_type == REQ_TYPE_FS)) {
+ blk_update_latency_hist(
+ (rq_data_dir(cmd->request) ==
+ READ)? &hba->io_lat_read:
+ &hba->io_lat_write, delta_us);
+ }
+
+ lrbp->cmd = NULL;
+
+ /* Do not touch lrbp after scsi done */
+ cmd->scsi_done(cmd);
+ clear_bit_unlock(index, &hba->lrb_in_use);
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ __clear_bit(index, &hba->delayed_reqs);
+#endif
+}
+
+/**
* __ufshcd_transfer_req_compl - handle SCSI and query command completion
* @hba: per adapter instance
* @completed_reqs: requests to complete
*/
static void __ufshcd_transfer_req_compl(struct ufs_hba *hba,
- unsigned long completed_reqs)
+ unsigned long completed_reqs)
{
struct ufshcd_lrb *lrbp;
struct scsi_cmnd *cmd;
- int result;
int index;
- struct request *req;
for_each_set_bit(index, &completed_reqs, hba->nutrs) {
lrbp = &hba->lrb[index];
cmd = lrbp->cmd;
if (cmd) {
- ufshcd_cond_add_cmd_trace(hba, index, "scsi_cmpl");
- ufshcd_update_tag_stats_completion(hba, cmd);
- result = ufshcd_transfer_rsp_status(hba, lrbp);
+ cmd->result = ufshcd_transfer_rsp_status(hba, lrbp);
scsi_dma_unmap(cmd);
- cmd->result = result;
- clear_bit_unlock(index, &hba->lrb_in_use);
- lrbp->complete_time_stamp = ktime_get();
- update_req_stats(hba, lrbp);
- /* Mark completed command as NULL in LRB */
- lrbp->cmd = NULL;
hba->ufs_stats.clk_rel.ctx = XFR_REQ_COMPL;
__ufshcd_release(hba, false);
__ufshcd_hibern8_release(hba, false);
if (cmd->request) {
- /*
- * As we are accessing the "request" structure,
- * this must be called before calling
- * ->scsi_done() callback.
- */
- ufshcd_vops_pm_qos_req_end(hba, cmd->request,
- false);
- ufshcd_vops_crypto_engine_cfg_end(hba,
- lrbp, cmd->request);
+ ufshcd_vops_pm_qos_req_end(
+ hba, cmd->request, false);
+ ufshcd_vops_crypto_engine_cfg_end(
+ hba, lrbp, cmd->request);
}
- req = cmd->request;
- if (req) {
- /* Update IO svc time latency histogram */
- if (req->lat_hist_enabled) {
- ktime_t completion;
- u_int64_t delta_us;
+ lrbp->complete_time_stamp = ktime_get();
- completion = ktime_get();
- delta_us = ktime_us_delta(completion,
- req->lat_hist_io_start);
- blk_update_latency_hist(
- (rq_data_dir(req) == READ) ?
- &hba->io_lat_read :
- &hba->io_lat_write, delta_us);
- }
- }
- /* Do not touch lrbp after scsi done */
- cmd->scsi_done(cmd);
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ ufs_impaired_delay_completion(hba, lrbp);
+#else
+ ufshcd_complete_lrb(hba, lrbp);
+#endif
} else if (lrbp->command_type == UTP_CMD_TYPE_DEV_MANAGE) {
if (hba->dev_cmd.complete) {
ufshcd_cond_add_cmd_trace(hba, index,
@@ -7020,12 +7252,16 @@ static int ufshcd_eh_device_reset_handler(struct scsi_cmnd *cmd)
}
}
spin_lock_irqsave(host->host_lock, flags);
+ if (hba->ufshpb_state == HPB_PRESENT)
+ hba->ufshpb_state = HPB_RESET;
ufshcd_transfer_req_compl(hba);
spin_unlock_irqrestore(host->host_lock, flags);
out:
hba->req_abort_count = 0;
if (!err) {
+ schedule_delayed_work(&hba->ufshpb_init_work,
+ msecs_to_jiffies(10));
err = SUCCESS;
} else {
dev_err(hba->dev, "%s: failed with err %d\n", __func__, err);
@@ -7484,18 +7720,9 @@ static void ufshcd_set_active_icc_lvl(struct ufs_hba *hba)
{
int ret;
int buff_len = hba->desc_size.pwr_desc;
- u8 *desc_buf = NULL;
+ u8 desc_buf[hba->desc_size.pwr_desc];
u32 icc_level;
- if (buff_len) {
- desc_buf = kmalloc(buff_len, GFP_KERNEL);
- if (!desc_buf) {
- dev_err(hba->dev,
- "%s: Failed to allocate desc_buf\n", __func__);
- return;
- }
- }
-
ret = ufshcd_read_power_desc(hba, desc_buf, buff_len);
if (ret) {
dev_err(hba->dev,
@@ -7873,17 +8100,8 @@ static int ufshcd_set_dev_ref_clk(struct ufs_hba *hba)
static int ufs_read_device_desc_data(struct ufs_hba *hba)
{
int err;
- u8 *desc_buf = NULL;
+ u8 desc_buf[hba->desc_size.dev_desc];
- if (hba->desc_size.dev_desc) {
- desc_buf = kmalloc(hba->desc_size.dev_desc, GFP_KERNEL);
- if (!desc_buf) {
- err = -ENOMEM;
- dev_err(hba->dev,
- "%s: Failed to allocate desc_buf\n", __func__);
- return err;
- }
- }
err = ufshcd_read_device_desc(hba, desc_buf, hba->desc_size.dev_desc);
if (err)
return err;
@@ -7935,6 +8153,11 @@ static void ufshcd_init_desc_sizes(struct ufs_hba *hba)
&hba->desc_size.geom_desc);
if (err)
hba->desc_size.geom_desc = QUERY_DESC_GEOMETRY_DEF_SIZE;
+
+ err = ufshcd_read_desc_length(hba, QUERY_DESC_IDN_HEALTH, 0,
+ &hba->desc_size.health_desc);
+ if (err)
+ hba->desc_size.health_desc = QUERY_DESC_HEALTH_DEF_SIZE;
}
static void ufshcd_def_desc_sizes(struct ufs_hba *hba)
@@ -7945,6 +8168,7 @@ static void ufshcd_def_desc_sizes(struct ufs_hba *hba)
hba->desc_size.conf_desc = QUERY_DESC_CONFIGURATION_DEF_SIZE;
hba->desc_size.unit_desc = QUERY_DESC_UNIT_DEF_SIZE;
hba->desc_size.geom_desc = QUERY_DESC_GEOMETRY_DEF_SIZE;
+ hba->desc_size.health_desc = QUERY_DESC_HEALTH_DEF_SIZE;
}
/**
@@ -8066,6 +8290,9 @@ static int ufshcd_probe_hba(struct ufs_hba *hba)
hba->clk_scaling.is_allowed = true;
}
+ schedule_delayed_work(&hba->ufshpb_init_work,
+ msecs_to_jiffies(0));
+
scsi_scan_host(hba->host);
pm_runtime_put_sync(hba->dev);
}
@@ -8239,6 +8466,120 @@ static void ufshcd_async_scan(void *data, async_cookie_t cookie)
ufshcd_extcon_register(hba);
}
+static int ufshcd_query_desc_for_ufshpb(struct ufs_hba *hba, int lun,
+ struct ufs_ioctl_query_data *ioctl_data, void __user *buffer)
+{
+ unsigned char *kernel_buf;
+ int kernel_buf_len;
+ int opcode, selector;
+ int err = 0;
+ int index = 0;
+ int length = 0;
+
+ opcode = UPIU_QUERY_OPCODE_LOW(ioctl_data->opcode);
+ selector = 1;
+
+ if (ioctl_data->idn == QUERY_DESC_IDN_STRING)
+ kernel_buf_len = IOCTL_DEV_CTX_MAX_SIZE;
+ else
+ kernel_buf_len = QUERY_DESC_MAX_SIZE;
+
+ kernel_buf = kzalloc(kernel_buf_len, GFP_KERNEL);
+ if (!kernel_buf) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ switch (opcode) {
+ case UPIU_QUERY_OPCODE_WRITE_DESC:
+ if (kernel_buf_len < ioctl_data->buf_size ||
+ !ioctl_data->buf_size) {
+ err = -EINVAL;
+ goto out_release_mem;
+ }
+ /* support configuration change only */
+ if (ioctl_data->idn != QUERY_DESC_IDN_CONFIGURATION &&
+ ioctl_data->buf_size != UFSHPB_CONFIG_LEN) {
+ err = -ENOTSUPP;
+ goto out_release_mem;
+ }
+ err = copy_from_user(kernel_buf, buffer +
+ sizeof(struct ufs_ioctl_query_data),
+ ioctl_data->buf_size);
+ if (!err)
+ err = ufshpb_control_validation(hba,
+ (struct ufshpb_config_desc *)kernel_buf);
+ if (err)
+ goto out_release_mem;
+ break;
+
+ case UPIU_QUERY_OPCODE_READ_DESC:
+ switch (ioctl_data->idn) {
+ case QUERY_DESC_IDN_UNIT:
+ if (!ufs_is_valid_unit_desc_lun(lun)) {
+ err = -EINVAL;
+ dev_err(hba->dev,
+ "%s: No unit descriptor for lun 0x%x\n",
+ __func__, lun);
+ goto out_release_mem;
+ }
+ index = lun;
+ break;
+ case QUERY_DESC_IDN_STRING:
+ if (!ufs_is_valid_unit_desc_lun(lun)) {
+ err = -EINVAL;
+ dev_err(hba->dev,
+ "No unit descriptor for lun 0x%x\n",
+ lun);
+ goto out_release_mem;
+ }
+ err = ufshpb_issue_req_dev_ctx(hba->ufshpb_lup[lun],
+ kernel_buf,
+ ioctl_data->buf_size);
+ if (err < 0)
+ goto out_release_mem;
+ goto copy_buffer;
+
+ case QUERY_DESC_IDN_DEVICE:
+ case QUERY_DESC_IDN_GEOMETRY:
+ case QUERY_DESC_IDN_CONFIGURAION:
+ break;
+
+ default:
+ err = -EINVAL;
+ dev_err(hba->dev, "invalid idn %d\n", ioctl_data->idn);
+ goto out_release_mem;
+ }
+ break;
+ default:
+ err = -EINVAL;
+ dev_err(hba->dev, "invalid opcode %d\n", opcode);
+ goto out_release_mem;
+ }
+
+ length = ioctl_data->buf_size;
+ err = ufshcd_query_descriptor(hba, opcode, ioctl_data->idn, index,
+ selector, kernel_buf, &length);
+ if (err)
+ goto out_release_mem;
+copy_buffer:
+ if (opcode == UPIU_QUERY_OPCODE_READ_DESC) {
+ err = copy_to_user(buffer, ioctl_data,
+ sizeof(struct ufs_ioctl_query_data));
+ if (err)
+ dev_err(hba->dev, "Failed copying back to user.\n");
+ err = copy_to_user(buffer + sizeof(struct ufs_ioctl_query_data),
+ kernel_buf, ioctl_data->buf_size);
+ if (err)
+ dev_err(hba->dev,
+ "Failed copying back to user : rsp_buffer.\n");
+ }
+out_release_mem:
+ kfree(kernel_buf);
+out:
+ return err;
+}
+
/**
* ufshcd_query_ioctl - perform user read queries
* @hba: per-adapter instance
@@ -8279,6 +8620,14 @@ static int ufshcd_query_ioctl(struct ufs_hba *hba, u8 lun, void __user *buffer)
goto out_release_mem;
}
+ if (UPIU_QUERY_OPCODE_HIGH(ioctl_data->opcode) ==
+ UPIU_QUERY_OPCODE_HIGH_HPB) {
+ err = ufshcd_query_desc_for_ufshpb(hba, lun,
+ ioctl_data, buffer);
+ kfree(ioctl_data);
+ goto out;
+ }
+
/* verify legal parameters & send query */
switch (ioctl_data->opcode) {
case UPIU_QUERY_OPCODE_READ_DESC:
@@ -8331,6 +8680,7 @@ static int ufshcd_query_ioctl(struct ufs_hba *hba, u8 lun, void __user *buffer)
case QUERY_ATTR_IDN_EE_CONTROL:
case QUERY_ATTR_IDN_EE_STATUS:
case QUERY_ATTR_IDN_SECONDS_PASSED:
+ case QUERY_ATTR_IDN_MANUAL_GC_STATUS:
index = 0;
break;
case QUERY_ATTR_IDN_DYN_CAP_NEEDED:
@@ -8383,6 +8733,7 @@ static int ufshcd_query_ioctl(struct ufs_hba *hba, u8 lun, void __user *buffer)
case QUERY_FLAG_IDN_PURGE_ENABLE:
case QUERY_FLAG_IDN_FPHYRESOURCEREMOVAL:
case QUERY_FLAG_IDN_BUSY_RTC:
+ case QUERY_FLAG_IDN_MANUAL_GC_CONT:
break;
default:
goto out_einval;
@@ -9286,8 +9637,8 @@ static int ufshcd_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
enum ufs_pm_level pm_lvl;
enum ufs_dev_pwr_mode req_dev_pwr_mode;
enum uic_link_state req_link_state;
+ bool need_upwrite = false;
- hba->pm_op_in_progress = 1;
if (!ufshcd_is_shutdown_pm(pm_op)) {
pm_lvl = ufshcd_is_runtime_pm(pm_op) ?
hba->rpm_lvl : hba->spm_lvl;
@@ -9298,6 +9649,15 @@ static int ufshcd_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
req_link_state = UIC_LINK_OFF_STATE;
}
+ if (ufshcd_is_runtime_pm(pm_op) &&
+ req_link_state == UIC_LINK_HIBERN8_STATE &&
+ ufshcd_is_auto_hibern8_supported(hba)) {
+ need_upwrite = true;
+ if (!down_write_trylock(&hba->query_lock))
+ return -EBUSY;
+ }
+ hba->pm_op_in_progress = 1;
+
/*
* If we can't transition into any of the low power modes
* just gate the clocks.
@@ -9417,6 +9777,8 @@ static int ufshcd_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op)
ufshcd_release_all(hba);
out:
hba->pm_op_in_progress = 0;
+ if (need_upwrite)
+ up_write(&hba->query_lock);
if (ret)
ufshcd_update_error_stats(hba, UFS_ERR_SUSPEND);
@@ -9727,7 +10089,7 @@ static inline ssize_t ufshcd_pm_lvl_store(struct device *dev,
return count;
}
-static ssize_t ufshcd_rpm_lvl_show(struct device *dev,
+static ssize_t rpm_lvl_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct ufs_hba *hba = dev_get_drvdata(dev);
@@ -9756,24 +10118,13 @@ static ssize_t ufshcd_rpm_lvl_show(struct device *dev,
return curr_len;
}
-static ssize_t ufshcd_rpm_lvl_store(struct device *dev,
+static ssize_t rpm_lvl_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
return ufshcd_pm_lvl_store(dev, attr, buf, count, true);
}
-static void ufshcd_add_rpm_lvl_sysfs_nodes(struct ufs_hba *hba)
-{
- hba->rpm_lvl_attr.show = ufshcd_rpm_lvl_show;
- hba->rpm_lvl_attr.store = ufshcd_rpm_lvl_store;
- sysfs_attr_init(&hba->rpm_lvl_attr.attr);
- hba->rpm_lvl_attr.attr.name = "rpm_lvl";
- hba->rpm_lvl_attr.attr.mode = S_IRUGO | S_IWUSR;
- if (device_create_file(hba->dev, &hba->rpm_lvl_attr))
- dev_err(hba->dev, "Failed to create sysfs for rpm_lvl\n");
-}
-
-static ssize_t ufshcd_spm_lvl_show(struct device *dev,
+static ssize_t spm_lvl_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct ufs_hba *hba = dev_get_drvdata(dev);
@@ -9802,27 +10153,482 @@ static ssize_t ufshcd_spm_lvl_show(struct device *dev,
return curr_len;
}
-static ssize_t ufshcd_spm_lvl_store(struct device *dev,
+static ssize_t spm_lvl_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t count)
{
return ufshcd_pm_lvl_store(dev, attr, buf, count, false);
}
-static void ufshcd_add_spm_lvl_sysfs_nodes(struct ufs_hba *hba)
+/*
+ * Values permitted 0, 1, 2.
+ * 0 -> Disable IO latency histograms (default)
+ * 1 -> Enable IO latency histograms
+ * 2 -> Zero out IO latency histograms
+ */
+static ssize_t
+latency_hist_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
{
- hba->spm_lvl_attr.show = ufshcd_spm_lvl_show;
- hba->spm_lvl_attr.store = ufshcd_spm_lvl_store;
- sysfs_attr_init(&hba->spm_lvl_attr.attr);
- hba->spm_lvl_attr.attr.name = "spm_lvl";
- hba->spm_lvl_attr.attr.mode = S_IRUGO | S_IWUSR;
- if (device_create_file(hba->dev, &hba->spm_lvl_attr))
- dev_err(hba->dev, "Failed to create sysfs for spm_lvl\n");
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ long value;
+
+ if (kstrtol(buf, 0, &value))
+ return -EINVAL;
+ if (value == BLK_IO_LAT_HIST_ZERO) {
+ memset(&hba->io_lat_read, 0, sizeof(hba->io_lat_read));
+ memset(&hba->io_lat_write, 0, sizeof(hba->io_lat_write));
+ } else if (value == BLK_IO_LAT_HIST_ENABLE ||
+ value == BLK_IO_LAT_HIST_DISABLE)
+ hba->latency_hist_enabled = value;
+ return count;
}
+ssize_t
+latency_hist_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ size_t written_bytes;
+
+ written_bytes = blk_latency_hist_show("Read", &hba->io_lat_read,
+ buf, PAGE_SIZE);
+ written_bytes += blk_latency_hist_show("Write", &hba->io_lat_write,
+ buf + written_bytes, PAGE_SIZE - written_bytes);
+
+ return written_bytes;
+}
+
+struct slowio_attr {
+ struct device_attribute attr;
+ enum ufshcd_slowio_optype optype;
+ enum ufshcd_slowio_systype systype;
+};
+
+static ssize_t
+slowio_store(struct device *dev, struct device_attribute *_attr,
+ const char *buf, size_t count)
+{
+ struct slowio_attr *attr = (struct slowio_attr *)_attr;
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ unsigned long flags, value;
+
+ if (kstrtol(buf, 0, &value))
+ return -EINVAL;
+
+ if (attr->systype == UFSHCD_SLOWIO_CNT)
+ value = 0;
+ else if (value < UFSHCD_MIN_SLOWIO_US)
+ return -EINVAL;
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ hba->slowio[attr->optype][attr->systype] = value;
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+
+ if (attr->systype == UFSHCD_SLOWIO_US)
+ ufshcd_update_slowio_min_us(hba);
+ return count;
+}
+
+static ssize_t
+slowio_show(struct device *dev, struct device_attribute *_attr, char *buf)
+{
+ struct slowio_attr *attr = (struct slowio_attr *)_attr;
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ return snprintf(buf, PAGE_SIZE, "%lld\n",
+ hba->slowio[attr->optype][attr->systype]);
+}
+
+#define __SLOWIO_ATTR(_name) \
+ __ATTR(slowio_##_name, 0644, slowio_show, slowio_store)
+
+#define SLOWIO_ATTR_RW(_name, _optype) \
+static struct slowio_attr ufs_slowio_##_name##_us = { \
+ .attr = __SLOWIO_ATTR(_name##_us), \
+ .optype = _optype, \
+ .systype = UFSHCD_SLOWIO_US, \
+}; \
+ \
+static struct slowio_attr ufs_slowio_##_name##_cnt = { \
+ .attr = __SLOWIO_ATTR(_name##_cnt), \
+ .optype = _optype, \
+ .systype = UFSHCD_SLOWIO_CNT, \
+}
+
+SLOWIO_ATTR_RW(read, UFSHCD_SLOWIO_READ);
+SLOWIO_ATTR_RW(write, UFSHCD_SLOWIO_WRITE);
+SLOWIO_ATTR_RW(unmap, UFSHCD_SLOWIO_UNMAP);
+SLOWIO_ATTR_RW(sync, UFSHCD_SLOWIO_SYNC);
+
+static ssize_t
+erasesize_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ u32 erasesize = 0;
+ int ret = -EOPNOTSUPP;
+ int i;
+
+ pm_runtime_get_sync(hba->dev);
+ for (i = 0; i < UFS_UPIU_MAX_GENERAL_LUN; i++) {
+ ret = ufshcd_read_unit_desc_param(hba,
+ i, UNIT_DESC_PARAM_ERASE_BLK_SIZE,
+ (u8 *)&erasesize, sizeof(erasesize));
+ if (ret == -EOPNOTSUPP)
+ continue;
+ /* Got a valid one, and should be same as others. */
+ break;
+ }
+ pm_runtime_mark_last_busy(hba->dev);
+ pm_runtime_put_noidle(hba->dev);
+ if (ret)
+ return 0;
+
+ return snprintf(buf, PAGE_SIZE, "0x%x\n", erasesize);
+}
+
+static ssize_t
+vendor_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ return snprintf(buf, PAGE_SIZE, "%.8s\n", hba->sdev_ufs_device->vendor);
+}
+
+static ssize_t
+model_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ return snprintf(buf, PAGE_SIZE, "%.16s\n", hba->sdev_ufs_device->model);
+}
+
+static ssize_t
+rev_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ return snprintf(buf, PAGE_SIZE, "%.4s\n", hba->sdev_ufs_device->rev);
+}
+
+static ssize_t
+version_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ return snprintf(buf, PAGE_SIZE, "0x%x\n", hba->ufs_version);
+}
+
+enum {
+ MANUAL_GC_OFF = 0,
+ MANUAL_GC_ON,
+ MANUAL_GC_DISABLE,
+ MANUAL_GC_ENABLE,
+ MANUAL_GC_MAX,
+};
+
+static ssize_t
+manual_gc_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ u32 status = MANUAL_GC_OFF;
+
+ if (hba->manual_gc.state == MANUAL_GC_DISABLE)
+ return scnprintf(buf, PAGE_SIZE, "%s", "disabled\n");
+
+ pm_runtime_get_sync(hba->dev);
+
+ down_read(&hba->query_lock);
+ if (hba->manual_gc.hagc_support) {
+ int err = ufshcd_query_attr_retry(hba,
+ UPIU_QUERY_OPCODE_READ_ATTR,
+ QUERY_ATTR_IDN_MANUAL_GC_STATUS, 0, 0, &status);
+
+ hba->manual_gc.hagc_support = err ? false: true;
+ }
+ up_read(&hba->query_lock);
+ pm_runtime_mark_last_busy(hba->dev);
+ pm_runtime_put_noidle(hba->dev);
+
+ if (!hba->manual_gc.hagc_support)
+ return scnprintf(buf, PAGE_SIZE, "%s", "bkops\n");
+
+ return scnprintf(buf, PAGE_SIZE, "%s",
+ status == MANUAL_GC_OFF ? "off\n" : "on\n");
+}
+
+static ssize_t
+manual_gc_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ enum query_opcode opcode;
+ u32 value;
+ int err = 0;
+
+ if (kstrtou32(buf, 0, &value))
+ return -EINVAL;
+
+ if (value >= MANUAL_GC_MAX)
+ return -EINVAL;
+
+ if (value == MANUAL_GC_DISABLE || value == MANUAL_GC_ENABLE) {
+ hba->manual_gc.state = value;
+ return count;
+ }
+ if (hba->manual_gc.state == MANUAL_GC_DISABLE)
+ return count;
+
+ pm_runtime_get_sync(hba->dev);
+
+ if (hba->manual_gc.hagc_support) {
+ opcode = (value == MANUAL_GC_ON) ? UPIU_QUERY_OPCODE_SET_FLAG:
+ UPIU_QUERY_OPCODE_CLEAR_FLAG;
+ err = ufshcd_query_flag_retry(hba, opcode,
+ QUERY_FLAG_IDN_MANUAL_GC_CONT, NULL);
+ hba->manual_gc.hagc_support = err ? false: true;
+ }
+
+ if (!hba->manual_gc.hagc_support) {
+ err = ufshcd_bkops_ctrl(hba, (value == MANUAL_GC_ON) ?
+ BKOPS_STATUS_NON_CRITICAL:
+ BKOPS_STATUS_CRITICAL);
+ if (!hba->auto_bkops_enabled)
+ err = -EAGAIN;
+ }
+
+ if (err || !ufshcd_is_auto_hibern8_supported(hba)) {
+ pm_runtime_mark_last_busy(hba->dev);
+ pm_runtime_put_noidle(hba->dev);
+ return count;
+ } else {
+ /* pm_runtime_put_sync in delay_ms */
+ hrtimer_start(&hba->manual_gc.hrtimer,
+ ms_to_ktime(hba->manual_gc.delay_ms),
+ HRTIMER_MODE_REL);
+ }
+ return count;
+}
+
+static ssize_t
+manual_gc_hold_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+
+ return snprintf(buf, PAGE_SIZE, "%lu\n", hba->manual_gc.delay_ms);
+}
+
+static ssize_t
+manual_gc_hold_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ unsigned long value;
+
+ if (kstrtoul(buf, 0, &value))
+ return -EINVAL;
+
+ hba->manual_gc.delay_ms = value;
+ return count;
+}
+
+static enum hrtimer_restart ufshcd_mgc_hrtimer_handler(struct hrtimer *timer)
+{
+ struct ufs_hba *hba = container_of(timer, struct ufs_hba,
+ manual_gc.hrtimer);
+
+ queue_work(hba->manual_gc.mgc_workq, &hba->manual_gc.hibern8_work);
+ return HRTIMER_NORESTART;
+}
+
+static void ufshcd_mgc_hibern8_work(struct work_struct *work)
+{
+ struct ufs_hba *hba = container_of(work, struct ufs_hba,
+ manual_gc.hibern8_work);
+ pm_runtime_mark_last_busy(hba->dev);
+ pm_runtime_put_noidle(hba->dev);
+ /* bkops will be disabled when power down */
+}
+
+static void ufshcd_init_manual_gc(struct ufs_hba *hba)
+{
+ struct ufs_manual_gc *mgc = &hba->manual_gc;
+ char wq_name[sizeof("ufs_mgc_hibern8_work")];
+
+ if (!ufshcd_is_auto_hibern8_supported(hba)) {
+ hba->manual_gc.state = MANUAL_GC_DISABLE;
+ return;
+ }
+
+ mgc->state = MANUAL_GC_ENABLE;
+ mgc->hagc_support = true;
+ mgc->delay_ms = UFSHCD_MANUAL_GC_HOLD_HIBERN8;
+
+ hrtimer_init(&mgc->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
+ mgc->hrtimer.function = ufshcd_mgc_hrtimer_handler;
+
+ INIT_WORK(&mgc->hibern8_work, ufshcd_mgc_hibern8_work);
+ snprintf(wq_name, ARRAY_SIZE(wq_name), "ufs_mgc_hibern8_work_%d",
+ hba->host->host_no);
+ hba->manual_gc.mgc_workq = create_singlethread_workqueue(wq_name);
+}
+
+static void ufshcd_exit_manual_gc(struct ufs_hba *hba)
+{
+ if (!ufshcd_is_auto_hibern8_supported(hba))
+ return;
+
+ hrtimer_cancel(&hba->manual_gc.hrtimer);
+ cancel_work_sync(&hba->manual_gc.hibern8_work);
+ destroy_workqueue(hba->manual_gc.mgc_workq);
+}
+
+static DEVICE_ATTR_RW(rpm_lvl);
+static DEVICE_ATTR_RW(spm_lvl);
+static DEVICE_ATTR_RW(latency_hist);
+static DEVICE_ATTR_RO(erasesize);
+static DEVICE_ATTR_RO(vendor);
+static DEVICE_ATTR_RO(model);
+static DEVICE_ATTR_RO(rev);
+static DEVICE_ATTR_RO(version);
+static DEVICE_ATTR_RW(manual_gc);
+static DEVICE_ATTR_RW(manual_gc_hold);
+
+static struct attribute *ufshcd_attrs[] = {
+ &dev_attr_rpm_lvl.attr,
+ &dev_attr_spm_lvl.attr,
+ &dev_attr_latency_hist.attr,
+ &dev_attr_erasesize.attr,
+ &dev_attr_vendor.attr,
+ &dev_attr_model.attr,
+ &dev_attr_rev.attr,
+ &dev_attr_version.attr,
+ &dev_attr_manual_gc.attr,
+ &dev_attr_manual_gc_hold.attr,
+ &ufs_slowio_read_us.attr.attr,
+ &ufs_slowio_read_cnt.attr.attr,
+ &ufs_slowio_write_us.attr.attr,
+ &ufs_slowio_write_cnt.attr.attr,
+ &ufs_slowio_unmap_us.attr.attr,
+ &ufs_slowio_unmap_cnt.attr.attr,
+ &ufs_slowio_sync_us.attr.attr,
+ &ufs_slowio_sync_cnt.attr.attr,
+ NULL
+};
+
+static const struct attribute_group ufshcd_attr_group = {
+ .attrs = ufshcd_attrs,
+};
+
+struct health_attr {
+ struct device_attribute attr;
+ int bytes;
+ int len;
+};
+
+static u32 health_get_bytes(u8 *desc_buf, int bytes, int len)
+{
+ u32 value = 0;
+
+ switch (len) {
+ case 1:
+ value = desc_buf[bytes];
+ break;
+ case 2:
+ value = desc_buf[bytes] << 8;
+ value += desc_buf[bytes + 1];
+ break;
+ case 4:
+ value = desc_buf[bytes] << 24;
+ value += desc_buf[bytes + 1] << 16;
+ value += desc_buf[bytes + 2] << 8;
+ value += desc_buf[bytes + 3];
+ break;
+ }
+ return value;
+}
+
+static ssize_t health_attr_show(struct device *dev,
+ struct device_attribute *_attr, char *buf)
+{
+ struct health_attr *attr = (struct health_attr *)_attr;
+ struct ufs_hba *hba = dev_get_drvdata(dev);
+ int buff_len = hba->desc_size.health_desc;
+ u8 desc_buf[hba->desc_size.health_desc];
+ u32 value;
+ int err;
+
+ pm_runtime_get_sync(hba->dev);
+ err = ufshcd_read_desc(hba, QUERY_DESC_IDN_HEALTH, 0,
+ desc_buf, buff_len);
+ pm_runtime_mark_last_busy(hba->dev);
+ pm_runtime_put_noidle(hba->dev);
+ if (err || (attr->bytes + attr->len) > desc_buf[0])
+ return 0;
+
+ value = health_get_bytes(desc_buf, attr->bytes, attr->len);
+
+ if (attr->bytes == UFSHCD_HEALTH_LIFEC_OFFSET) {
+ if (!value) {
+ u32 erase = health_get_bytes(desc_buf,
+ UFSHCD_HEALTH_ERASE_OFFSET, 2);
+ if (erase)
+ value = erase * 100 / UFSHCD_DEFAULT_PE_CYCLE;
+ }
+ return scnprintf(buf, PAGE_SIZE, "%u\n",
+ value > 100 ? 100: value);
+ }
+ return scnprintf(buf, PAGE_SIZE, "0x%02x\n", value);
+}
+
+#define HEALTH_ATTR_RO(_name, _bytes, _len) \
+static struct health_attr ufs_health_##_name = { \
+ .attr = { \
+ .attr = { \
+ .name = __stringify(_name), \
+ .mode = 0444 \
+ }, \
+ .show = health_attr_show, \
+ }, \
+ .bytes = _bytes, \
+ .len = _len, \
+}
+
+HEALTH_ATTR_RO(length, UFSHCD_HEALTH_LEN_OFFSET, 1);
+HEALTH_ATTR_RO(type, UFSHCD_HEALTH_TYPE_OFFSET, 1);
+HEALTH_ATTR_RO(eol, UFSHCD_HEALTH_EOL_OFFSET, 1);
+HEALTH_ATTR_RO(lifetimeA, UFSHCD_HEALTH_LIFEA_OFFSET, 1);
+HEALTH_ATTR_RO(lifetimeB, UFSHCD_HEALTH_LIFEB_OFFSET, 1);
+HEALTH_ATTR_RO(lifetimeC, UFSHCD_HEALTH_LIFEC_OFFSET, 1);
+HEALTH_ATTR_RO(erasecount, UFSHCD_HEALTH_ERASE_OFFSET, 2);
+HEALTH_ATTR_RO(totalwrite, UFSHCD_HEALTH_WRITE_OFFSET, 4);
+
+static struct attribute *ufshcd_health_attrs[] = {
+ &ufs_health_length.attr.attr,
+ &ufs_health_type.attr.attr,
+ &ufs_health_eol.attr.attr,
+ &ufs_health_lifetimeA.attr.attr,
+ &ufs_health_lifetimeB.attr.attr,
+ &ufs_health_lifetimeC.attr.attr,
+ &ufs_health_erasecount.attr.attr,
+ &ufs_health_totalwrite.attr.attr,
+ NULL
+};
+
+static const struct attribute_group ufshcd_health_attr_group = {
+ .name = "health",
+ .attrs = ufshcd_health_attrs,
+};
+
static inline void ufshcd_add_sysfs_nodes(struct ufs_hba *hba)
{
- ufshcd_add_rpm_lvl_sysfs_nodes(hba);
- ufshcd_add_spm_lvl_sysfs_nodes(hba);
+ if (sysfs_create_group(&hba->dev->kobj, &ufshcd_attr_group))
+ dev_err(hba->dev, "Failed to create default sysfs group\n");
+ if (sysfs_create_group(&hba->dev->kobj, &ufshcd_health_attr_group))
+ dev_err(hba->dev, "Failed to create health sysfs group\n");
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ ufs_impaired_init_sysfs(hba);
+#endif
}
static void __ufshcd_shutdown_clkscaling(struct ufs_hba *hba)
@@ -9906,61 +10712,6 @@ int ufshcd_shutdown(struct ufs_hba *hba)
}
EXPORT_SYMBOL(ufshcd_shutdown);
-/*
- * Values permitted 0, 1, 2.
- * 0 -> Disable IO latency histograms (default)
- * 1 -> Enable IO latency histograms
- * 2 -> Zero out IO latency histograms
- */
-static ssize_t
-latency_hist_store(struct device *dev, struct device_attribute *attr,
- const char *buf, size_t count)
-{
- struct ufs_hba *hba = dev_get_drvdata(dev);
- long value;
-
- if (kstrtol(buf, 0, &value))
- return -EINVAL;
- if (value == BLK_IO_LAT_HIST_ZERO) {
- memset(&hba->io_lat_read, 0, sizeof(hba->io_lat_read));
- memset(&hba->io_lat_write, 0, sizeof(hba->io_lat_write));
- } else if (value == BLK_IO_LAT_HIST_ENABLE ||
- value == BLK_IO_LAT_HIST_DISABLE)
- hba->latency_hist_enabled = value;
- return count;
-}
-
-ssize_t
-latency_hist_show(struct device *dev, struct device_attribute *attr,
- char *buf)
-{
- struct ufs_hba *hba = dev_get_drvdata(dev);
- size_t written_bytes;
-
- written_bytes = blk_latency_hist_show("Read", &hba->io_lat_read,
- buf, PAGE_SIZE);
- written_bytes += blk_latency_hist_show("Write", &hba->io_lat_write,
- buf + written_bytes, PAGE_SIZE - written_bytes);
-
- return written_bytes;
-}
-
-static DEVICE_ATTR(latency_hist, S_IRUGO | S_IWUSR,
- latency_hist_show, latency_hist_store);
-
-static void
-ufshcd_init_latency_hist(struct ufs_hba *hba)
-{
- if (device_create_file(hba->dev, &dev_attr_latency_hist))
- dev_err(hba->dev, "Failed to create latency_hist sysfs entry\n");
-}
-
-static void
-ufshcd_exit_latency_hist(struct ufs_hba *hba)
-{
- device_create_file(hba->dev, &dev_attr_latency_hist);
-}
-
/**
* ufshcd_remove - de-allocate SCSI host and host memory space
* data structure memory
@@ -9968,22 +10719,23 @@ ufshcd_exit_latency_hist(struct ufs_hba *hba)
*/
void ufshcd_remove(struct ufs_hba *hba)
{
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ ufs_impaired_exit(hba);
+#endif
+ ufshpb_release(hba, HPB_NEED_INIT);
scsi_remove_host(hba->host);
/* disable interrupts */
ufshcd_disable_intr(hba, hba->intr_mask);
ufshcd_hba_stop(hba, true);
+ ufshcd_exit_manual_gc(hba);
ufshcd_exit_clk_gating(hba);
ufshcd_exit_hibern8_on_idle(hba);
- ufshcd_exit_latency_hist(hba);
if (ufshcd_is_clkscaling_supported(hba)) {
device_remove_file(hba->dev, &hba->clk_scaling.enable_attr);
if (hba->devfreq)
devfreq_remove_device(hba->devfreq);
}
-
- ufshcd_exit_latency_hist(hba);
-
ufshcd_hba_exit(hba);
ufsdbg_remove_debugfs(hba);
}
@@ -10230,9 +10982,9 @@ static int ufshcd_devfreq_scale(struct ufs_hba *hba, bool scale_up)
*/
if (ufshcd_is_auto_hibern8_supported(hba)) {
ret = ufshcd_uic_hibern8_enter(hba);
+ /* link will be bad state so no need to scale_up_gear */
if (ret)
- /* link will be bad state so no need to scale_up_gear */
- return ret;
+ goto clk_scaling_unprepare;
ufshcd_custom_cmd_log(hba, "Hibern8-entered");
}
@@ -10243,9 +10995,9 @@ static int ufshcd_devfreq_scale(struct ufs_hba *hba, bool scale_up)
if (ufshcd_is_auto_hibern8_supported(hba)) {
ret = ufshcd_uic_hibern8_exit(hba);
+ /* link will be bad state so no need to scale_up_gear */
if (ret)
- /* link will be bad state so no need to scale_up_gear */
- return ret;
+ goto clk_scaling_unprepare;
ufshcd_custom_cmd_log(hba, "Hibern8-Exited");
}
@@ -10600,6 +11352,21 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
hba->max_pwr_info.is_valid = false;
+ /* Set default slow io value. */
+ hba->slowio[UFSHCD_SLOWIO_READ][UFSHCD_SLOWIO_US] =
+ UFSHCD_DEFAULT_SLOWIO_READ_US;
+ hba->slowio[UFSHCD_SLOWIO_WRITE][UFSHCD_SLOWIO_US] =
+ UFSHCD_DEFAULT_SLOWIO_WRITE_US;
+ hba->slowio[UFSHCD_SLOWIO_UNMAP][UFSHCD_SLOWIO_US] =
+ UFSHCD_DEFAULT_SLOWIO_UNMAP_US;
+ hba->slowio[UFSHCD_SLOWIO_SYNC][UFSHCD_SLOWIO_US] =
+ UFSHCD_DEFAULT_SLOWIO_SYNC_US;
+ ufshcd_update_slowio_min_us(hba);
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ ufs_impaired_init(hba);
+#endif
+
/* Initailize wait queue for task management */
init_waitqueue_head(&hba->tm_wq);
init_waitqueue_head(&hba->tm_tag_wq);
@@ -10617,12 +11384,14 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
mutex_init(&hba->dev_cmd.lock);
init_rwsem(&hba->lock);
+ init_rwsem(&hba->query_lock);
/* Initialize device management tag acquire wait queue */
init_waitqueue_head(&hba->dev_cmd.tag_wq);
ufshcd_init_clk_gating(hba);
ufshcd_init_hibern8_on_idle(hba);
+ ufshcd_init_manual_gc(hba);
/*
* In order to avoid any spurious interrupt immediately after
@@ -10705,8 +11474,6 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
/* Hold auto suspend until async scan completes */
pm_runtime_get_sync(dev);
- ufshcd_init_latency_hist(hba);
-
/*
* We are assuming that device wasn't put in sleep/power-down
* state exclusively during the boot stage before kernel.
@@ -10715,6 +11482,9 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
*/
ufshcd_set_ufs_dev_active(hba);
+ /* initialize hpb structures */
+ ufshcd_init_hpb(hba);
+
ufshcd_cmd_log_init(hba);
async_schedule(ufshcd_async_scan, hba);
@@ -10728,8 +11498,8 @@ int ufshcd_init(struct ufs_hba *hba, void __iomem *mmio_base, unsigned int irq)
out_remove_scsi_host:
scsi_remove_host(hba->host);
exit_gating:
+ ufshcd_exit_manual_gc(hba);
ufshcd_exit_clk_gating(hba);
- ufshcd_exit_latency_hist(hba);
out_disable:
hba->is_irq_enabled = false;
ufshcd_hba_exit(hba);
diff --git a/drivers/scsi/ufs/ufshcd.h b/drivers/scsi/ufs/ufshcd.h
index 6e329c0..a9ef20e 100644
--- a/drivers/scsi/ufs/ufshcd.h
+++ b/drivers/scsi/ufs/ufshcd.h
@@ -4,6 +4,8 @@
* This code is based on drivers/scsi/ufs/ufshcd.h
* Copyright (C) 2011-2013 Samsung India Software Operations
* Copyright (c) 2013-2018, The Linux Foundation. All rights reserved.
+ * Copyright (c) 2017-2018 Samsung Electronics Co., Ltd.
+ * Copyright (C) 2018, Google, Inc.
*
* Authors:
* Santosh Yaraganavi <santosh.sy@samsung.com>
@@ -44,6 +46,7 @@
#include <linux/interrupt.h>
#include <linux/io.h>
#include <linux/delay.h>
+#include <linux/kthread.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/rwsem.h>
@@ -72,6 +75,10 @@
#include <linux/fault-inject.h>
#include "ufs.h"
#include "ufshci.h"
+#include "ufshpb.h"
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+#include "ufs-impaired.h"
+#endif
#define UFSHCD "ufshcd"
#define UFSHCD_DRIVER_VERSION "0.3"
@@ -195,6 +202,11 @@ struct ufs_pm_lvl_states {
* @issue_time_stamp: time stamp for debug purposes
* @complete_time_stamp: time stamp for statistics
* @req_abort_skip: skip request abort task flag
+ *
+ * impaired storage related
+ * @list_head: LRB list node head
+ * @complete_delay: target delay in us
+ * @target_complete_time: target completion time with delay emulation.
*/
struct ufshcd_lrb {
struct utp_transfer_req_desc *utr_descriptor_ptr;
@@ -220,6 +232,12 @@ struct ufshcd_lrb {
ktime_t complete_time_stamp;
bool req_abort_skip;
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ struct list_head list;
+ int complete_delay;
+ ktime_t target_complete_time;
+#endif
};
/**
@@ -256,6 +274,7 @@ struct ufs_desc_size {
int interc_desc;
int unit_desc;
int conf_desc;
+ int health_desc;
};
/**
@@ -402,6 +421,16 @@ struct ufs_hba_variant {
void (*dbg_register_dump)(struct ufs_hba *hba);
};
+/* for manual gc */
+struct ufs_manual_gc {
+ int state;
+ bool hagc_support;
+ struct hrtimer hrtimer;
+ unsigned long delay_ms;
+ struct work_struct hibern8_work;
+ struct workqueue_struct *mgc_workq;
+};
+
/* clock gating state */
enum clk_gating_state {
CLKS_OFF,
@@ -553,6 +582,7 @@ struct debugfs_files {
struct dentry *dbg_print_en;
struct dentry *req_stats;
struct dentry *query_stats;
+ struct dentry *io_stats;
u32 dme_local_attr_id;
u32 dme_peer_attr_id;
struct dentry *reset_controller;
@@ -576,7 +606,8 @@ enum ts_types {
TS_URGENT_READ = 3,
TS_URGENT_WRITE = 4,
TS_FLUSH = 5,
- TS_NUM_STATS = 6,
+ TS_DISCARD = 6,
+ TS_NUM_STATS = 7,
};
/**
@@ -592,6 +623,24 @@ struct ufshcd_req_stat {
u64 sum;
u64 count;
};
+
+/**
+ * struct ufshcd_io_stat - statistics for I/O amount.
+ * @req_count_started: total number of I/O requests, which were started.
+ * @total_bytes_started: total I/O amount in bytes, which were started.
+ * @req_count_completed: total number of I/O request, which were completed.
+ * @total_bytes_completed: total I/O amount in bytes, which were completed.
+ * @max_diff_req_count: MAX of 'req_count_started - req_count_completed'.
+ * @max_diff_total_bytes: MAX of 'total_bytes_started - total_bytes_completed'.
+ */
+struct ufshcd_io_stat {
+ u64 req_count_started;
+ u64 total_bytes_started;
+ u64 req_count_completed;
+ u64 total_bytes_completed;
+ u64 max_diff_req_count;
+ u64 max_diff_total_bytes;
+};
#endif
enum ufshcd_ctx {
@@ -636,6 +685,9 @@ struct ufs_stats {
int err_stats[UFS_ERR_MAX];
struct ufshcd_req_stat req_stats[TS_NUM_STATS];
int query_stats_arr[UPIU_QUERY_OPCODE_MAX][MAX_QUERY_IDN];
+ struct ufshcd_io_stat io_read;
+ struct ufshcd_io_stat io_write;
+ struct ufshcd_io_stat io_readwrite;
#endif
u32 last_intr_status;
@@ -657,6 +709,41 @@ struct ufs_stats {
u32 dme_err_cnt;
};
+static inline bool is_read_opcode(u8 opcode)
+{
+ return opcode == READ_10 || opcode == READ_16;
+}
+
+static inline bool is_write_opcode(u8 opcode)
+{
+ return opcode == WRITE_10 || opcode == WRITE_16;
+}
+
+static inline bool is_unmap_opcode(u8 opcode)
+{
+ return opcode == UNMAP;
+}
+
+static inline char *parse_opcode(u8 opcode)
+{
+ /* string should be less than 12 byte-long */
+ switch (opcode) {
+ case READ_10:
+ return "READ_10";
+ case READ_16:
+ return "READ_16";
+ case WRITE_10:
+ return "WRITE_10";
+ case WRITE_16:
+ return "WRITE_16";
+ case UNMAP:
+ return "UNMAP";
+ case SYNCHRONIZE_CACHE:
+ return "SYNC_CACHE";
+ }
+ return NULL;
+}
+
/* UFS Host Controller debug print bitmask */
#define UFSHCD_DBG_PRINT_CLK_FREQ_EN UFS_BIT(0)
#define UFSHCD_DBG_PRINT_UIC_ERR_HIST_EN UFS_BIT(1)
@@ -679,10 +766,13 @@ struct ufshcd_cmd_log_entry {
u8 lun;
u8 cmd_id;
sector_t lba;
- int transfer_len;
+ u32 transfer_len;
u8 idn; /* used only for query idn */
u32 doorbell;
u32 outstanding_reqs;
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ u32 delayed_reqs;
+#endif
u32 seq_num;
unsigned int tag;
ktime_t tstamp;
@@ -701,6 +791,26 @@ enum ufshcd_card_state {
UFS_CARD_STATE_OFFLINE = 2,
};
+/* UFS Slow I/O operation types */
+enum ufshcd_slowio_optype {
+ UFSHCD_SLOWIO_READ = 0,
+ UFSHCD_SLOWIO_WRITE = 1,
+ UFSHCD_SLOWIO_UNMAP = 2,
+ UFSHCD_SLOWIO_SYNC = 3,
+ UFSHCD_SLOWIO_OP_MAX = 4,
+};
+
+/* UFS Slow I/O sysfs entry types */
+enum ufshcd_slowio_systype {
+ UFSHCD_SLOWIO_US = 0,
+ UFSHCD_SLOWIO_CNT = 1,
+ UFSHCD_SLOWIO_SYS_MAX = 2,
+};
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+extern void ufshcd_complete_lrb(struct ufs_hba *hba, struct ufshcd_lrb *lrbp);
+#endif
+
/**
* struct ufs_hba - per adapter private structure
* @mmio_base: UFSHCI base register address
@@ -789,8 +899,6 @@ struct ufs_hba {
int rpm_lvl;
/* Desired UFS power management level during system PM */
int spm_lvl;
- struct device_attribute rpm_lvl_attr;
- struct device_attribute spm_lvl_attr;
int pm_op_in_progress;
struct ufshcd_lrb *lrb;
@@ -912,6 +1020,8 @@ struct ufs_hba {
/* Number of requests aborts */
int req_abort_count;
+ u32 security_in;
+
/* Number of lanes available (1 or 2) for Rx/Tx */
u32 lanes_per_direction;
@@ -979,6 +1089,7 @@ struct ufs_hba {
/* sync b/w diff contexts */
struct rw_semaphore lock;
+ struct rw_semaphore query_lock;
unsigned long shutdown_in_prog;
struct reset_control *core_reset;
@@ -995,6 +1106,31 @@ struct ufs_hba {
struct io_latency_state io_lat_write;
struct ufs_desc_size desc_size;
bool restore_needed;
+
+ /* To monitor slow UFS I/O requests. */
+ u64 slowio_min_us;
+ u64 slowio[UFSHCD_SLOWIO_OP_MAX][UFSHCD_SLOWIO_SYS_MAX];
+
+ /* HPB support */
+ u32 ufshpb_feat;
+ int ufshpb_state;
+ int ufshpb_max_regions;
+ struct delayed_work ufshpb_init_work;
+ bool issue_ioctl;
+ struct ufshpb_lu *ufshpb_lup[UFS_UPIU_MAX_GENERAL_LUN];
+ struct scsi_device *sdev_ufs_lu[UFS_UPIU_MAX_GENERAL_LUN];
+ struct work_struct ufshpb_eh_work;
+
+ struct ufs_manual_gc manual_gc;
+
+#ifdef CONFIG_SCSI_UFS_IMPAIRED
+ struct kobject *impaired_kobj;
+ struct ufs_impaired_storage impaired;
+ struct task_struct *impaired_thread;
+ struct list_head impaired_list_head;
+ unsigned long delayed_reqs;
+ bool impaired_should_stop;
+#endif
};
static inline void ufshcd_mark_shutdown_ongoing(struct ufs_hba *hba)
@@ -1468,4 +1604,10 @@ static inline void ufshcd_vops_pm_qos_req_end(struct ufs_hba *hba,
hba->var->pm_qos_vops->req_end(hba, req, lock);
}
+#define UFSHCD_MIN_SLOWIO_US (1000) /* 1 ms */
+#define UFSHCD_DEFAULT_SLOWIO_READ_US (5000000) /* 5 seconds */
+#define UFSHCD_DEFAULT_SLOWIO_WRITE_US (10000000) /* 10 seconds */
+#define UFSHCD_DEFAULT_SLOWIO_UNMAP_US (30000000) /* 30 seconds */
+#define UFSHCD_DEFAULT_SLOWIO_SYNC_US (10000000) /* 10 seconds */
+
#endif /* End of Header */
diff --git a/drivers/scsi/ufs/ufshpb.c b/drivers/scsi/ufs/ufshpb.c
new file mode 100644
index 0000000..1116bfe
--- /dev/null
+++ b/drivers/scsi/ufs/ufshpb.c
@@ -0,0 +1,2820 @@
+/*
+ * Universal Flash Storage Host Performance Booster
+ *
+ * Copyright (C) 2017-2018 Samsung Electronics Co., Ltd.
+ * Copyright (C) 2018, Google, Inc.
+ *
+ * Authors:
+ * Yongmyung Lee <ymhungry.lee@samsung.com>
+ * Jinyoung Choi <j-young.choi@samsung.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * See the COPYING file in the top-level directory or visit
+ * <http://www.gnu.org/licenses/gpl-2.0.html>
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * This program is provided "AS IS" and "WITH ALL FAULTS" and
+ * without warranty of any kind. You are solely responsible for
+ * determining the appropriateness of using and distributing
+ * the program and assume all risks associated with your exercise
+ * of rights with respect to the program, including but not limited
+ * to infringement of third party rights, the risks and costs of
+ * program errors, damage to or loss of data, programs or equipment,
+ * and unavailability or interruption of operations. Under no
+ * circumstances will the contributor of this Program be liable for
+ * any damages of any kind arising from your use or distribution of
+ * this program.
+ *
+ * The Linux Foundation chooses to take subject only to the GPLv2
+ * license terms, and distributes only under these terms.
+ */
+
+#include <linux/slab.h>
+#include <linux/blkdev.h>
+#include <scsi/scsi.h>
+#include <linux/sysfs.h>
+#include <linux/blktrace_api.h>
+
+#include "../../../block/blk.h"
+
+#include "ufs.h"
+#include "ufshcd.h"
+#include "ufshpb.h"
+
+/*
+ * UFSHPB DEBUG
+ */
+#define hpb_dbg(hba, msg, args...) \
+ do { \
+ if (hba) \
+ dev_dbg(hba->dev, msg, ##args); \
+ } while (0)
+
+#define hpb_trace(hpb, args...) \
+ do { \
+ if (hpb->hba->sdev_ufs_lu[hpb->lun] && \
+ hpb->hba->sdev_ufs_lu[hpb->lun]->request_queue) \
+ blk_add_trace_msg( \
+ hpb->hba->sdev_ufs_lu[hpb->lun]->request_queue, \
+ ##args); \
+ } while (0)
+
+/*
+ * debug variables
+ */
+static int alloc_mctx;
+
+/*
+ * define global constants
+ */
+static int sects_per_blk_shift;
+static int bits_per_dword_shift;
+static int bits_per_dword_mask;
+static int bits_per_byte_shift;
+
+static int ufshpb_create_sysfs(struct ufs_hba *hba, struct ufshpb_lu *hpb);
+static void ufshpb_error_handler(struct work_struct *work);
+static void ufshpb_evict_region(struct ufshpb_lu *hpb,
+ struct ufshpb_region *cb);
+
+static inline void ufshpb_get_bit_offset(
+ struct ufshpb_lu *hpb, int subregion_offset,
+ int *dword, int *offset)
+{
+ *dword = subregion_offset >> bits_per_dword_shift;
+ *offset = subregion_offset & bits_per_dword_mask;
+}
+
+/* called with hpb_lock (irq) */
+static bool ufshpb_ppn_dirty_check(struct ufshpb_lu *hpb,
+ struct ufshpb_subregion *cp, int subregion_offset)
+{
+ bool is_dirty;
+ unsigned int bit_dword, bit_offset;
+
+ ufshpb_get_bit_offset(hpb, subregion_offset,
+ &bit_dword, &bit_offset);
+
+ if (!cp->mctx->ppn_dirty)
+ return false;
+
+ is_dirty = cp->mctx->ppn_dirty[bit_dword] &
+ (1 << bit_offset) ? true : false;
+
+ return is_dirty;
+}
+
+static void ufshpb_ppn_prep(struct ufshpb_lu *hpb,
+ struct ufshcd_lrb *lrbp, unsigned long long ppn)
+{
+ unsigned char cmd[16] = { 0 };
+
+ cmd[0] = READ_16;
+ cmd[2] = lrbp->cmd->cmnd[2];
+ cmd[3] = lrbp->cmd->cmnd[3];
+ cmd[4] = lrbp->cmd->cmnd[4];
+ cmd[5] = lrbp->cmd->cmnd[5];
+ cmd[6] = GET_BYTE_7(ppn);
+ cmd[7] = GET_BYTE_6(ppn);
+ cmd[8] = GET_BYTE_5(ppn);
+ cmd[9] = GET_BYTE_4(ppn);
+ cmd[10] = GET_BYTE_3(ppn);
+ cmd[11] = GET_BYTE_2(ppn);
+ cmd[12] = GET_BYTE_1(ppn);
+ cmd[13] = GET_BYTE_0(ppn);
+ cmd[14] = 0x11; // Group Number
+ cmd[15] = 0x01; // Transfer_len = 0x01 (FW defined)
+
+ memcpy(lrbp->cmd->cmnd, cmd, MAX_CDB_SIZE);
+ memcpy(lrbp->ucd_req_ptr->sc.cdb, cmd, MAX_CDB_SIZE);
+}
+
+/* called with hpb_lock (irq) */
+static inline void ufshpb_set_dirty_bits(struct ufshpb_lu *hpb,
+ struct ufshpb_region *cb, struct ufshpb_subregion *cp,
+ int dword, int offset, unsigned int count)
+{
+ const unsigned long mask = ((1UL << count) - 1) & 0xffffffff;
+
+ if (cb->region_state == HPBREGION_INACTIVE)
+ return;
+
+ cp->mctx->ppn_dirty[dword] |= (mask << offset);
+}
+
+static void ufshpb_set_dirty(struct ufshpb_lu *hpb,
+ struct ufshcd_lrb *lrbp, int region,
+ int subregion, int subregion_offset)
+{
+ struct ufshpb_region *cb;
+ struct ufshpb_subregion *cp;
+ int count;
+ int bit_count, bit_dword, bit_offset;
+
+ count = blk_rq_sectors(lrbp->cmd->request) >> sects_per_blk_shift;
+ ufshpb_get_bit_offset(hpb, subregion_offset,
+ &bit_dword, &bit_offset);
+
+ do {
+ bit_count = min(count, BITS_PER_DWORD - bit_offset);
+
+ cb = hpb->region_tbl + region;
+ cp = cb->subregion_tbl + subregion;
+
+ ufshpb_set_dirty_bits(hpb, cb, cp,
+ bit_dword, bit_offset, bit_count);
+
+ bit_offset = 0;
+ bit_dword++;
+
+ if (bit_dword == hpb->dwords_per_subregion) {
+ bit_dword = 0;
+ subregion++;
+
+ if (subregion == hpb->subregions_per_region) {
+ subregion = 0;
+ region++;
+ }
+ }
+
+ count -= bit_count;
+ } while (count);
+}
+
+static inline bool ufshpb_is_read_lrbp(struct ufshcd_lrb *lrbp)
+{
+ if (lrbp->cmd->cmnd[0] == READ_10 || lrbp->cmd->cmnd[0] == READ_16)
+ return true;
+
+ return false;
+}
+
+static inline bool ufshpb_is_encrypted_lrbp(struct ufshcd_lrb *lrbp)
+{
+ return (lrbp->utr_descriptor_ptr->header.dword_0 & UTRD_CRYPTO_ENABLE);
+}
+
+static inline bool ufshpb_is_write_discard_lrbp(struct ufshcd_lrb *lrbp)
+{
+ return lrbp->cmd->cmnd[0] == WRITE_10 ||
+ lrbp->cmd->cmnd[0] == WRITE_16 ||
+ lrbp->cmd->cmnd[0] == UNMAP;
+}
+
+static inline void ufshpb_get_pos_from_lpn(struct ufshpb_lu *hpb,
+ unsigned int lpn, int *region, int *subregion, int *offset)
+{
+ int region_offset;
+
+ *region = lpn >> hpb->entries_per_region_shift;
+ region_offset = lpn & hpb->entries_per_region_mask;
+ *subregion = region_offset >> hpb->entries_per_subregion_shift;
+ *offset = region_offset & hpb->entries_per_subregion_mask;
+}
+
+static unsigned long long ufshpb_get_ppn(struct ufshpb_map_ctx *mctx, int pos)
+{
+ unsigned long long *ppn_table;
+ int index, offset;
+
+ index = pos / HPB_ENTREIS_PER_OS_PAGE;
+ offset = pos % HPB_ENTREIS_PER_OS_PAGE;
+
+ ppn_table = page_address(mctx->m_page[index]);
+ return ppn_table[offset];
+}
+
+void ufshpb_prep_fn(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+{
+ struct ufshpb_lu *hpb;
+ struct ufshpb_region *cb;
+ struct ufshpb_subregion *cp;
+ unsigned int lpn;
+ unsigned long long ppn = 0;
+ int region, subregion, subregion_offset;
+ const struct request *rq = lrbp->cmd->request;
+ unsigned long long rq_pos = blk_rq_pos(rq);
+ unsigned int rq_sectors = blk_rq_sectors(rq);
+
+ /* WKLU could not be HPB-LU */
+ if (lrbp->lun >= UFS_UPIU_MAX_GENERAL_LUN)
+ return;
+
+ hpb = hba->ufshpb_lup[lrbp->lun];
+ if (!hpb || !hpb->lu_hpb_enable) {
+ if (ufshpb_is_read_lrbp(lrbp))
+ goto read_10;
+ return;
+ }
+
+ if (hpb->force_disable) {
+ if (ufshpb_is_read_lrbp(lrbp))
+ goto read_10;
+ return;
+ }
+
+ /*
+ * TODO: check if ICE is not supported or not.
+ *
+ * if (ufshpb_is_read_lrbp(lrbp) && ufshpb_is_encrypted_lrbp(lrbp))
+ * goto read_10;
+ */
+
+ lpn = rq_pos / SECTORS_PER_BLOCK;
+ ufshpb_get_pos_from_lpn(hpb, lpn, ®ion,
+ &subregion, &subregion_offset);
+ cb = hpb->region_tbl + region;
+
+ if (ufshpb_is_write_discard_lrbp(lrbp)) {
+ spin_lock_bh(&hpb->hpb_lock);
+
+ if (cb->region_state == HPBREGION_INACTIVE) {
+ spin_unlock_bh(&hpb->hpb_lock);
+ return;
+ }
+ ufshpb_set_dirty(hpb, lrbp, region, subregion,
+ subregion_offset);
+ spin_unlock_bh(&hpb->hpb_lock);
+ return;
+ }
+
+ if (!ufshpb_is_read_lrbp(lrbp))
+ return;
+
+ if (rq_sectors != SECTORS_PER_BLOCK) {
+ hpb_trace(hpb, "%llu + %u READ_10 many_blocks %d - %d",
+ rq_pos, rq_sectors, region, subregion);
+ return;
+ }
+
+ cp = cb->subregion_tbl + subregion;
+
+ spin_lock_bh(&hpb->hpb_lock);
+ if (cb->region_state == HPBREGION_INACTIVE ||
+ cp->subregion_state != HPBSUBREGION_CLEAN) {
+ if (cb->region_state == HPBREGION_INACTIVE) {
+ atomic64_inc(&hpb->region_miss);
+ hpb_trace(hpb, "%llu + %u READ_10 RG_INACT %d - %d",
+ rq_pos, rq_sectors, region, subregion);
+ } else if (cp->subregion_state == HPBSUBREGION_DIRTY
+ || cp->subregion_state == HPBSUBREGION_ISSUED) {
+ atomic64_inc(&hpb->subregion_miss);
+ hpb_trace(hpb, "%llu + %u READ_10 SRG_D %d - %d",
+ rq_pos, rq_sectors, region, subregion);
+ } else {
+ hpb_trace(hpb, "%llu + %u READ_10 ( %d %d ) %d - %d",
+ rq_pos, rq_sectors,
+ cb->region_state, cp->subregion_state,
+ region, subregion);
+ }
+ spin_unlock_bh(&hpb->hpb_lock);
+ return;
+ }
+
+ if (ufshpb_ppn_dirty_check(hpb, cp, subregion_offset)) {
+ atomic64_inc(&hpb->entry_dirty_miss);
+ hpb_trace(hpb, "%llu + %u READ_10 E_D %d - %d",
+ rq_pos, rq_sectors, region, subregion);
+ spin_unlock_bh(&hpb->hpb_lock);
+ return;
+ }
+
+ ppn = ufshpb_get_ppn(cp->mctx, subregion_offset);
+ spin_unlock_bh(&hpb->hpb_lock);
+
+ ufshpb_ppn_prep(hpb, lrbp, ppn);
+ hpb_trace(hpb, "%llu + %u READ_16 %d - %d",
+ rq_pos, rq_sectors, region, subregion);
+ atomic64_inc(&hpb->hit);
+ return;
+read_10:
+ if (!hpb || !lrbp)
+ return;
+ hpb_trace(hpb, "%llu + %u READ_10", rq_pos, rq_sectors);
+ atomic64_inc(&hpb->miss);
+}
+
+static int ufshpb_clean_dirty_bitmap(
+ struct ufshpb_lu *hpb, struct ufshpb_subregion *cp)
+{
+ struct ufshpb_region *cb;
+
+ cb = hpb->region_tbl + cp->region;
+
+ /* if mctx is null, active block had been evicted out */
+ if (cb->region_state == HPBREGION_INACTIVE || !cp->mctx) {
+ hpb_dbg(hpb->hba, "%d - %d evicted\n",
+ cp->region, cp->subregion);
+ return -EINVAL;
+ }
+
+ memset(cp->mctx->ppn_dirty, 0x00,
+ hpb->entries_per_subregion >> bits_per_byte_shift);
+ return 0;
+}
+
+static void ufshpb_clean_active_subregion(
+ struct ufshpb_lu *hpb, struct ufshpb_subregion *cp)
+{
+ struct ufshpb_region *cb;
+
+ cb = hpb->region_tbl + cp->region;
+
+ /* if mctx is null, active block had been evicted out */
+ if (cb->region_state == HPBREGION_INACTIVE || !cp->mctx) {
+ hpb_dbg(hpb->hba, "%d - %d evicted\n",
+ cp->region, cp->subregion);
+ return;
+ }
+ cp->subregion_state = HPBSUBREGION_CLEAN;
+}
+
+static void ufshpb_error_active_subregion(
+ struct ufshpb_lu *hpb, struct ufshpb_subregion *cp)
+{
+ struct ufshpb_region *cb;
+
+ cb = hpb->region_tbl + cp->region;
+
+ /* if mctx is null, active block had been evicted out */
+ if (cb->region_state == HPBREGION_INACTIVE || !cp->mctx) {
+ dev_err(HPB_DEV(hpb),
+ "%d - %d evicted\n", cp->region, cp->subregion);
+ return;
+ }
+ cp->subregion_state = HPBSUBREGION_DIRTY;
+}
+
+static void ufshpb_map_compl_process(struct ufshpb_lu *hpb,
+ struct ufshpb_map_req *map_req)
+{
+ unsigned long long debug_ppn_0, debug_ppn_65535;
+
+ map_req->RSP_end = ktime_to_ns(ktime_get());
+
+ debug_ppn_0 = ufshpb_get_ppn(map_req->mctx, 0);
+ debug_ppn_65535 = ufshpb_get_ppn(map_req->mctx, 65535);
+
+ hpb_trace(hpb, "Noti: C RB %d - %d",
+ map_req->region, map_req->subregion);
+ hpb_dbg(hpb->hba, "UFSHPB COMPL READ BUFFER %d - %d ( %llx ~ %llx )\n",
+ map_req->region, map_req->subregion,
+ debug_ppn_0, debug_ppn_65535);
+ hpb_dbg(hpb->hba, "start=%llu, issue=%llu, endio=%llu, endio=%llu\n",
+ map_req->RSP_tasklet_enter1 - map_req->RSP_start,
+ map_req->RSP_issue - map_req->RSP_tasklet_enter1,
+ map_req->RSP_endio - map_req->RSP_issue,
+ map_req->RSP_end - map_req->RSP_endio);
+
+ spin_lock(&hpb->hpb_lock);
+ ufshpb_clean_active_subregion(hpb,
+ hpb->region_tbl[map_req->region].subregion_tbl +
+ map_req->subregion);
+ spin_unlock(&hpb->hpb_lock);
+}
+
+/*
+ * Must held rsp_list_lock before enter this function
+ */
+static struct ufshpb_rsp_info *ufshpb_get_req_info(struct ufshpb_lu *hpb)
+{
+ struct ufshpb_rsp_info *rsp_info =
+ list_first_entry_or_null(&hpb->lh_rsp_info_free,
+ struct ufshpb_rsp_info,
+ list_rsp_info);
+ if (!rsp_info) {
+ hpb_dbg(hpb->hba, "there is no rsp_info");
+ return NULL;
+ }
+ list_del(&rsp_info->list_rsp_info);
+ memset(rsp_info, 0x00, sizeof(struct ufshpb_rsp_info));
+
+ INIT_LIST_HEAD(&rsp_info->list_rsp_info);
+
+ return rsp_info;
+}
+
+static void ufshpb_map_req_compl_fn(struct request *req, int error)
+{
+ struct ufshpb_map_req *map_req =
+ (struct ufshpb_map_req *)req->end_io_data;
+ struct ufs_hba *hba;
+ struct ufshpb_lu *hpb;
+ struct scsi_sense_hdr sshdr;
+ struct ufshpb_region *cb;
+ struct ufshpb_rsp_info *rsp_info;
+ unsigned long flags;
+
+ hpb = map_req->hpb;
+ hba = hpb->hba;
+ cb = hpb->region_tbl + map_req->region;
+ map_req->RSP_endio = ktime_to_ns(ktime_get());
+
+ if (hba->ufshpb_state != HPB_PRESENT)
+ goto free_map_req;
+
+ if (!error) {
+ ufshpb_map_compl_process(hpb, map_req);
+ goto free_map_req;
+ }
+
+ dev_err(HPB_DEV(hpb),
+ "error number %d ( %d - %d )\n",
+ error, map_req->region, map_req->subregion);
+ scsi_normalize_sense(map_req->sense,
+ SCSI_SENSE_BUFFERSIZE, &sshdr);
+ dev_err(HPB_DEV(hpb),
+ "code %x sense_key %x asc %x ascq %x\n",
+ sshdr.response_code,
+ sshdr.sense_key, sshdr.asc, sshdr.ascq);
+ dev_err(HPB_DEV(hpb),
+ "byte4 %x byte5 %x byte6 %x additional_len %x\n",
+ sshdr.byte4, sshdr.byte5,
+ sshdr.byte6, sshdr.additional_length);
+ atomic64_inc(&hpb->rb_fail);
+
+ if (sshdr.sense_key == ILLEGAL_REQUEST) {
+ spin_lock(&hpb->hpb_lock);
+ if (cb->region_state == HPBREGION_PINNED) {
+ if (sshdr.asc == 0x06 && sshdr.ascq == 0x01) {
+ dev_err(HPB_DEV(hpb), "retry pinned rb %d - %d",
+ map_req->region,
+ map_req->subregion);
+ INIT_LIST_HEAD(&map_req->list_map_req);
+ list_add_tail(&map_req->list_map_req,
+ &hpb->lh_map_req_retry);
+ spin_unlock(&hpb->hpb_lock);
+
+ schedule_delayed_work(&hpb->ufshpb_retry_work,
+ msecs_to_jiffies(5000));
+ return;
+ }
+
+ hpb_dbg(hpb->hba, "pinned rb %d - %d(dirty)",
+ map_req->region, map_req->subregion);
+ ufshpb_error_active_subregion(hpb,
+ cb->subregion_tbl + map_req->subregion);
+ spin_unlock(&hpb->hpb_lock);
+ } else {
+ spin_unlock(&hpb->hpb_lock);
+
+ spin_lock_irqsave(&hpb->rsp_list_lock, flags);
+ rsp_info = ufshpb_get_req_info(hpb);
+ spin_unlock_irqrestore(&hpb->rsp_list_lock, flags);
+ if (!rsp_info) {
+ dev_warn(hba->dev,
+ "%s:%d No rsp_info\n",
+ __func__, __LINE__);
+ goto free_map_req;
+ }
+
+ rsp_info->type = HPB_RSP_REQ_REGION_UPDATE;
+ rsp_info->RSP_start = ktime_to_ns(ktime_get());
+ rsp_info->active_cnt = 0;
+ rsp_info->inactive_cnt = 1;
+ rsp_info->inactive_list.region[0] = map_req->region;
+ hpb_dbg(hpb->hba,
+ "Non-pinned rb %d is added to rsp_info_list",
+ map_req->region);
+
+ spin_lock_irqsave(&hpb->rsp_list_lock, flags);
+ list_add_tail(&rsp_info->list_rsp_info,
+ &hpb->lh_rsp_info);
+ spin_unlock_irqrestore(&hpb->rsp_list_lock, flags);
+
+ tasklet_schedule(&hpb->ufshpb_tasklet);
+ }
+ }
+free_map_req:
+ spin_lock(&hpb->hpb_lock);
+ INIT_LIST_HEAD(&map_req->list_map_req);
+ list_add_tail(&map_req->list_map_req, &hpb->lh_map_req_free);
+ spin_unlock(&hpb->hpb_lock);
+}
+
+static int ufshpb_execute_req_dev_ctx(struct ufshpb_lu *hpb,
+ unsigned char *cmd, void *buf, int length)
+{
+ unsigned long flags;
+ struct scsi_sense_hdr sshdr;
+ struct scsi_device *sdp;
+ struct ufs_hba *hba = hpb->hba;
+ int ret = 0;
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ sdp = hba->sdev_ufs_lu[hpb->lun];
+ if (sdp) {
+ ret = scsi_device_get(sdp);
+ if (!ret && !scsi_device_online(sdp)) {
+ ret = -ENODEV;
+ scsi_device_put(sdp);
+ } else if (!ret) {
+ hba->issue_ioctl = true;
+ }
+ } else {
+ ret = -ENODEV;
+ }
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+ if (ret)
+ return ret;
+
+ ret = scsi_execute_req_flags(sdp, cmd, DMA_FROM_DEVICE,
+ buf, length, &sshdr,
+ msecs_to_jiffies(30000), 3, NULL, 0);
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ hba->issue_ioctl = false;
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+ scsi_device_put(sdp);
+ return ret;
+}
+
+static inline void ufshpb_set_read_dev_ctx_cmd(unsigned char *cmd, int lba,
+ int length)
+{
+ cmd[0] = READ_10;
+ cmd[1] = 0x02;
+ cmd[2] = GET_BYTE_3(lba);
+ cmd[3] = GET_BYTE_2(lba);
+ cmd[4] = GET_BYTE_1(lba);
+ cmd[5] = GET_BYTE_0(lba);
+ cmd[6] = GET_BYTE_2(length);
+ cmd[7] = GET_BYTE_1(length);
+ cmd[8] = GET_BYTE_0(length);
+}
+
+int ufshpb_issue_req_dev_ctx(struct ufshpb_lu *hpb, unsigned char *buf,
+ int buf_length)
+{
+ unsigned char cmd[10] = { 0 };
+ int cmd_len = buf_length >> OS_PAGE_SHIFT;
+ int ret = 0;
+
+ ufshpb_set_read_dev_ctx_cmd(cmd, 0x48504230, cmd_len);
+
+ ret = ufshpb_execute_req_dev_ctx(hpb, cmd, buf, buf_length);
+ if (ret < 0)
+ hpb_dbg(hpb->hba, "failed with err %d\n", ret);
+ return ret;
+}
+
+static inline void ufshpb_set_read_buf_cmd(unsigned char *cmd,
+ int region, int subregion, int subregion_mem_size)
+{
+ cmd[0] = UFSHPB_READ_BUFFER;
+ cmd[1] = 0x01;
+ cmd[2] = GET_BYTE_1(region);
+ cmd[3] = GET_BYTE_0(region);
+ cmd[4] = GET_BYTE_1(subregion);
+ cmd[5] = GET_BYTE_0(subregion);
+ cmd[6] = GET_BYTE_2(subregion_mem_size);
+ cmd[7] = GET_BYTE_1(subregion_mem_size);
+ cmd[8] = GET_BYTE_0(subregion_mem_size);
+ cmd[9] = 0x00;
+}
+
+static void ufshpb_bio_init(struct bio *bio, struct bio_vec *table,
+ int max_vecs)
+{
+ bio_init(bio);
+
+ bio->bi_io_vec = table;
+ bio->bi_max_vecs = max_vecs;
+}
+
+static int ufshpb_add_bio_page(struct ufshpb_lu *hpb,
+ struct request_queue *q, struct bio *bio, struct bio_vec *bvec,
+ struct ufshpb_map_ctx *mctx)
+{
+ struct page *page = NULL;
+ int i, ret;
+
+ ufshpb_bio_init(bio, bvec, hpb->mpages_per_subregion);
+
+ for (i = 0; i < hpb->mpages_per_subregion; i++) {
+ /* virt_to_page(p + (OS_PAGE_SIZE * i)); */
+ page = mctx->m_page[i];
+ if (!page)
+ return -ENOMEM;
+
+ ret = bio_add_pc_page(q, bio, page, hpb->mpage_bytes, 0);
+ if (ret != hpb->mpage_bytes) {
+ dev_err(HPB_DEV(hpb), "error ret %d\n", ret);
+ return -EINVAL;
+ }
+ }
+ return 0;
+}
+
+static inline void ufshpb_issue_map_req(struct request_queue *q,
+ struct request *req)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(q->queue_lock, flags);
+ list_add(&req->queuelist, &q->queue_head);
+ spin_unlock_irqrestore(q->queue_lock, flags);
+}
+
+static int ufshpb_map_req_issue(struct ufshpb_lu *hpb,
+ struct request_queue *q, struct ufshpb_map_req *map_req)
+{
+ struct request *req;
+ struct bio *bio;
+ unsigned char cmd[16] = { 0 };
+ int ret;
+
+ bio = &map_req->bio;
+ ret = ufshpb_add_bio_page(hpb, q, bio, map_req->bvec, map_req->mctx);
+ if (ret) {
+ hpb_dbg(hpb->hba, "ufshpb_add_bio_page_error %d\n", ret);
+ return ret;
+ }
+
+ ufshpb_set_read_buf_cmd(cmd, map_req->region, map_req->subregion,
+ hpb->subregion_mem_size);
+
+ req = &map_req->req;
+ blk_rq_init(q, req);
+ blk_rq_append_bio(req, bio);
+
+ req->cmd_len = COMMAND_SIZE(cmd[0]);
+ memcpy(req->cmd, cmd, req->cmd_len);
+
+ req->cmd_type = REQ_TYPE_BLOCK_PC;
+ req->cmd_flags = READ | REQ_SOFTBARRIER | REQ_QUIET | REQ_PREEMPT;
+ req->timeout = msecs_to_jiffies(30000);
+ req->end_io = ufshpb_map_req_compl_fn;
+ req->end_io_data = (void *)map_req;
+ req->sense = map_req->sense;
+ req->sense_len = 0;
+
+ hpb_dbg(hpb->hba, "issue map_request: %d - %d\n",
+ map_req->region, map_req->subregion);
+
+ /* this sequence already has spin_lock_irqsave(queue_lock) */
+ ufshpb_issue_map_req(q, req);
+ map_req->RSP_issue = ktime_to_ns(ktime_get());
+
+ atomic64_inc(&hpb->map_req_cnt);
+
+ return 0;
+}
+
+static int ufshpb_set_map_req(struct ufshpb_lu *hpb,
+ int region, int subregion, struct ufshpb_map_ctx *mctx,
+ struct ufshpb_rsp_info *rsp_info)
+{
+ struct ufshpb_map_req *map_req;
+
+ spin_lock(&hpb->hpb_lock);
+ map_req = list_first_entry_or_null(&hpb->lh_map_req_free,
+ struct ufshpb_map_req,
+ list_map_req);
+ if (!map_req) {
+ hpb_dbg(hpb->hba, "There is no map_req\n");
+ spin_unlock(&hpb->hpb_lock);
+ return -ENOMEM;
+ }
+ list_del(&map_req->list_map_req);
+ memset(map_req, 0x00, sizeof(struct ufshpb_map_req));
+
+ spin_unlock(&hpb->hpb_lock);
+
+ map_req->hpb = hpb;
+ map_req->region = region;
+ map_req->subregion = subregion;
+ map_req->mctx = mctx;
+ map_req->lun = hpb->lun;
+ map_req->RSP_start = rsp_info->RSP_start;
+ map_req->RSP_tasklet_enter1 = rsp_info->RSP_tasklet_enter;
+
+ return ufshpb_map_req_issue(hpb,
+ hpb->hba->sdev_ufs_lu[hpb->lun]->request_queue, map_req);
+}
+
+static struct ufshpb_map_ctx *ufshpb_get_map_ctx(struct ufshpb_lu *hpb)
+{
+ struct ufshpb_map_ctx *mctx;
+
+ mctx = list_first_entry_or_null(&hpb->lh_map_ctx,
+ struct ufshpb_map_ctx, list_table);
+ if (mctx) {
+ list_del_init(&mctx->list_table);
+ hpb->debug_free_table--;
+ return mctx;
+ }
+ return ERR_PTR(-ENOMEM);
+}
+
+static inline void ufshpb_add_lru_info(struct victim_select_info *lru_info,
+ struct ufshpb_region *cb)
+{
+ cb->region_state = HPBREGION_ACTIVE;
+ list_add_tail(&cb->list_region, &lru_info->lru);
+ atomic64_inc(&lru_info->active_cnt);
+}
+
+static inline int ufshpb_add_region(struct ufshpb_lu *hpb,
+ struct ufshpb_region *cb)
+{
+ struct victim_select_info *lru_info;
+ int subregion;
+ int err = 0;
+
+ lru_info = &hpb->lru_info;
+
+ hpb_dbg(hpb->hba,
+ "\x1b[44m\x1b[32m E->active region: %d \x1b[0m\n", cb->region);
+ hpb_trace(hpb, "Noti: ACT RG: %d", cb->region);
+
+ for (subregion = 0; subregion < cb->subregion_count; subregion++) {
+ struct ufshpb_subregion *cp;
+
+ cp = cb->subregion_tbl + subregion;
+ cp->mctx = ufshpb_get_map_ctx(hpb);
+ if (IS_ERR(cp->mctx)) {
+ err = PTR_ERR(cp->mctx);
+ goto out;
+ }
+ cp->subregion_state = HPBSUBREGION_DIRTY;
+ }
+ ufshpb_add_lru_info(lru_info, cb);
+
+ atomic64_inc(&hpb->region_add);
+out:
+ if (err)
+ hpb_dbg(hpb->hba,
+ "get mctx failed. err %d subregion %d free_table %d\n",
+ err, subregion, hpb->debug_free_table);
+ return err;
+}
+
+static inline void ufshpb_put_map_ctx(
+ struct ufshpb_lu *hpb, struct ufshpb_map_ctx *mctx)
+{
+ list_add(&mctx->list_table, &hpb->lh_map_ctx);
+ hpb->debug_free_table++;
+}
+
+static inline void ufshpb_purge_active_page(struct ufshpb_lu *hpb,
+ struct ufshpb_subregion *cp, int state)
+{
+ if (state == HPBSUBREGION_UNUSED) {
+ ufshpb_put_map_ctx(hpb, cp->mctx);
+ cp->mctx = NULL;
+ }
+ cp->subregion_state = state;
+}
+
+static inline void ufshpb_cleanup_lru_info(struct victim_select_info *lru_info,
+ struct ufshpb_region *cb)
+{
+ list_del_init(&cb->list_region);
+ cb->region_state = HPBREGION_INACTIVE;
+ cb->hit_count = 0;
+ atomic64_dec(&lru_info->active_cnt);
+}
+
+static inline void ufshpb_evict_region(struct ufshpb_lu *hpb,
+ struct ufshpb_region *cb)
+{
+ struct victim_select_info *lru_info;
+ struct ufshpb_subregion *cp;
+ int subregion;
+
+ lru_info = &hpb->lru_info;
+
+ hpb_dbg(hpb->hba, "\x1b[41m\x1b[33m C->EVICT region: %d \x1b[0m\n",
+ cb->region);
+ hpb_trace(hpb, "Noti: EVIC RG: %d", cb->region);
+
+ ufshpb_cleanup_lru_info(lru_info, cb);
+ atomic64_inc(&hpb->region_evict);
+ for (subregion = 0; subregion < cb->subregion_count; subregion++) {
+ cp = cb->subregion_tbl + subregion;
+
+ ufshpb_purge_active_page(hpb, cp, HPBSUBREGION_UNUSED);
+ }
+}
+
+static void ufshpb_hit_lru_info(struct victim_select_info *lru_info,
+ struct ufshpb_region *cb)
+{
+ switch (lru_info->selection_type) {
+ case LRU:
+ list_move_tail(&cb->list_region, &lru_info->lru);
+ break;
+ case LFU:
+ if (cb->hit_count != 0xffffffff)
+ cb->hit_count++;
+
+ list_move_tail(&cb->list_region, &lru_info->lru);
+ break;
+ default:
+ break;
+ }
+}
+
+static struct ufshpb_region *ufshpb_victim_lru_info(
+ struct victim_select_info *lru_info)
+{
+ struct ufshpb_region *cb;
+ struct ufshpb_region *victim_cb = NULL;
+ u32 hit_count = 0xffffffff;
+
+ switch (lru_info->selection_type) {
+ case LRU:
+ victim_cb = list_first_entry(&lru_info->lru,
+ struct ufshpb_region, list_region);
+ break;
+ case LFU:
+ list_for_each_entry(cb, &lru_info->lru, list_region) {
+ if (hit_count > cb->hit_count) {
+ hit_count = cb->hit_count;
+ victim_cb = cb;
+ }
+ }
+ break;
+ default:
+ break;
+ }
+ return victim_cb;
+}
+
+static int ufshpb_evict_load_region(struct ufshpb_lu *hpb,
+ struct ufshpb_rsp_info *rsp_info)
+{
+ struct ufshpb_region *cb;
+ struct ufshpb_region *victim_cb;
+ struct victim_select_info *lru_info;
+ int region, ret, iter;
+
+ lru_info = &hpb->lru_info;
+ hpb_dbg(hpb->hba, "active_cnt :%lld\n",
+ (long long)atomic64_read(&lru_info->active_cnt));
+
+ for (iter = 0; iter < rsp_info->inactive_cnt; iter++) {
+ region = rsp_info->inactive_list.region[iter];
+ cb = hpb->region_tbl + region;
+
+ if (cb->region_state == HPBREGION_PINNED) {
+ /*
+ * Pinned active-block should not drop-out.
+ * But if so, it would treat error as critical,
+ * and it will run ufshpb_eh_work
+ */
+ dev_warn(hpb->hba->dev,
+ "UFSHPB pinned active-block drop-out error\n");
+ return -ENOMEM;
+ }
+
+ if (list_empty(&cb->list_region))
+ continue;
+
+ spin_lock(&hpb->hpb_lock);
+ ufshpb_evict_region(hpb, cb);
+ spin_unlock(&hpb->hpb_lock);
+ }
+
+ for (iter = 0; iter < rsp_info->active_cnt; iter++) {
+ region = rsp_info->active_list.region[iter];
+ cb = hpb->region_tbl + region;
+
+ /*
+ * if already region is added to lru_list,
+ * just initiate the information of lru.
+ * because the region already has the map ctx.
+ * (!list_empty(&cb->list_region) == region->state=active...)
+ */
+ if (!list_empty(&cb->list_region)) {
+ ufshpb_hit_lru_info(lru_info, cb);
+ continue;
+ }
+
+ spin_lock(&hpb->hpb_lock);
+ if (cb->region_state != HPBREGION_INACTIVE) {
+ spin_unlock(&hpb->hpb_lock);
+ continue;
+ }
+
+ if (atomic64_read(&lru_info->active_cnt) ==
+ lru_info->max_lru_active_cnt) {
+
+ victim_cb = ufshpb_victim_lru_info(lru_info);
+
+ if (!victim_cb) {
+ dev_warn(hpb->hba->dev,
+ "UFSHPB victim_cb is NULL\n");
+ goto unlock_error;
+ }
+
+ hpb_trace(hpb, "Noti: VT RG %d", victim_cb->region);
+ hpb_dbg(hpb->hba, "max lru case. victim : %d\n",
+ victim_cb->region);
+
+ ufshpb_evict_region(hpb, victim_cb);
+ }
+
+ ret = ufshpb_add_region(hpb, cb);
+ if (ret) {
+ dev_warn(hpb->hba->dev,
+ "UFSHPB memory allocation failed\n");
+ goto unlock_error;
+ }
+ spin_unlock(&hpb->hpb_lock);
+ }
+ return 0;
+
+unlock_error:
+ spin_unlock(&hpb->hpb_lock);
+ return -ENOMEM;
+}
+
+static inline struct ufshpb_rsp_field *ufshpb_get_hpb_rsp(
+ struct ufshcd_lrb *lrbp)
+{
+ return (struct ufshpb_rsp_field *)&lrbp->ucd_rsp_ptr->sr.sense_data_len;
+}
+
+static void ufshpb_rsp_map_cmd_req(struct ufshpb_lu *hpb,
+ struct ufshpb_rsp_info *rsp_info)
+{
+ struct ufshpb_region *cb;
+ struct ufshpb_subregion *cp;
+ int region, subregion;
+ int iter;
+ int ret;
+
+ /*
+ * Before Issue read buffer CMD for active active block,
+ * prepare the memory from memory pool.
+ */
+ ret = ufshpb_evict_load_region(hpb, rsp_info);
+ if (ret) {
+ hpb_dbg(hpb->hba, "region evict/load failed. ret %d\n", ret);
+ goto wakeup_ee_worker;
+ }
+
+ for (iter = 0; iter < rsp_info->active_cnt; iter++) {
+ region = rsp_info->active_list.region[iter];
+ subregion = rsp_info->active_list.subregion[iter];
+ cb = hpb->region_tbl + region;
+
+ if (region >= hpb->regions_per_lu ||
+ subregion >= cb->subregion_count) {
+ hpb_dbg(hpb->hba,
+ "ufshpb issue-map %d - %d range error\n",
+ region, subregion);
+ goto wakeup_ee_worker;
+ }
+
+ cp = cb->subregion_tbl + subregion;
+
+ /*
+ * if subregion_state set HPBSUBREGION_ISSUED,
+ * active_page has already been added to list,
+ * so it just ends function.
+ */
+ spin_lock(&hpb->hpb_lock);
+ if (cp->subregion_state == HPBSUBREGION_ISSUED) {
+ spin_unlock(&hpb->hpb_lock);
+ continue;
+ }
+
+ cp->subregion_state = HPBSUBREGION_ISSUED;
+
+ ret = ufshpb_clean_dirty_bitmap(hpb, cp);
+
+ spin_unlock(&hpb->hpb_lock);
+
+ if (ret)
+ continue;
+
+ if (hpb->force_map_req_disable ||
+ !hpb->hba->sdev_ufs_lu[hpb->lun] ||
+ !hpb->hba->sdev_ufs_lu[hpb->lun]->request_queue)
+ return;
+
+ ret = ufshpb_set_map_req(hpb, region, subregion,
+ cp->mctx, rsp_info);
+ if (ret) {
+ hpb_dbg(hpb->hba, "ufshpb_set_map_req error %d\n", ret);
+ goto wakeup_ee_worker;
+ }
+ }
+ return;
+
+wakeup_ee_worker:
+ hpb->hba->ufshpb_state = HPB_FAILED;
+ schedule_work(&hpb->hba->ufshpb_eh_work);
+}
+
+/* routine : isr (ufs) */
+void ufshpb_rsp_upiu(struct ufs_hba *hba, struct ufshcd_lrb *lrbp)
+{
+ struct ufshpb_lu *hpb;
+ struct ufshpb_rsp_field *rsp_field;
+ struct ufshpb_rsp_info *rsp_info;
+ struct ufshpb_region *region;
+ int data_seg_len, num, blk_idx;
+
+ data_seg_len = be32_to_cpu(lrbp->ucd_rsp_ptr->header.dword_2)
+ & MASK_RSP_UPIU_DATA_SEG_LEN;
+ if (!data_seg_len) {
+ bool do_tasklet = false;
+
+ if (lrbp->lun >= UFS_UPIU_MAX_GENERAL_LUN)
+ return;
+
+ hpb = hba->ufshpb_lup[lrbp->lun];
+ if (!hpb)
+ return;
+
+ spin_lock(&hpb->rsp_list_lock);
+ do_tasklet = !list_empty(&hpb->lh_rsp_info);
+ spin_unlock(&hpb->rsp_list_lock);
+
+ if (do_tasklet)
+ tasklet_schedule(&hpb->ufshpb_tasklet);
+ return;
+ }
+
+ rsp_field = ufshpb_get_hpb_rsp(lrbp);
+
+ if ((BE_BYTE(rsp_field->sense_data_len, 0) != DEV_SENSE_SEG_LEN) ||
+ rsp_field->desc_type != DEV_DES_TYPE ||
+ rsp_field->additional_len != DEV_ADDITIONAL_LEN ||
+ rsp_field->hpb_type == HPB_RSP_NONE ||
+ rsp_field->active_region_cnt > MAX_ACTIVE_NUM ||
+ rsp_field->inactive_region_cnt > MAX_INACTIVE_NUM)
+ return;
+
+ if (lrbp->lun >= UFS_UPIU_MAX_GENERAL_LUN) {
+ dev_warn(hba->dev, "lun is not general = %d", lrbp->lun);
+ return;
+ }
+
+ hpb = hba->ufshpb_lup[lrbp->lun];
+ if (!hpb) {
+ dev_warn(hba->dev,
+ "%s:%d UFS-LU%d is not UFSHPB LU\n", __func__,
+ __LINE__, lrbp->lun);
+ return;
+ }
+
+ hpb_dbg(hpb->hba,
+ "HPB-Info Noti: %d LUN: %d Seg-Len %d, Req_type = %d\n",
+ rsp_field->hpb_type, lrbp->lun,
+ be32_to_cpu(lrbp->ucd_rsp_ptr->header.dword_2) &
+ MASK_RSP_UPIU_DATA_SEG_LEN, rsp_field->reserved);
+
+ atomic64_inc(&hpb->rb_noti_cnt);
+
+ if (!hpb->lu_hpb_enable) {
+ dev_warn(hba->dev, "UFSHPB(%s) LU(%d) not HPB-LU\n",
+ __func__, lrbp->lun);
+ return;
+ }
+
+ spin_lock(&hpb->rsp_list_lock);
+ rsp_info = ufshpb_get_req_info(hpb);
+ spin_unlock(&hpb->rsp_list_lock);
+ if (!rsp_info)
+ return;
+
+ switch (rsp_field->hpb_type) {
+ case HPB_RSP_REQ_REGION_UPDATE:
+ WARN_ON(data_seg_len != DEV_DATA_SEG_LEN);
+ rsp_info->type = HPB_RSP_REQ_REGION_UPDATE;
+
+ rsp_info->RSP_start = ktime_to_ns(ktime_get());
+
+ for (num = 0; num < rsp_field->active_region_cnt; num++) {
+ blk_idx = num * PER_ACTIVE_INFO_BYTES;
+ rsp_info->active_list.region[num] =
+ BE_BYTE(rsp_field->hpb_active_field, blk_idx);
+ rsp_info->active_list.subregion[num] =
+ BE_BYTE(rsp_field->hpb_active_field,
+ blk_idx + 2);
+ hpb_dbg(hpb->hba,
+ "active num: %d, #block: %d, page#: %d\n",
+ num + 1,
+ rsp_info->active_list.region[num],
+ rsp_info->active_list.subregion[num]);
+ }
+ rsp_info->active_cnt = num;
+
+ for (num = 0; num < rsp_field->inactive_region_cnt; num++) {
+ blk_idx = num * PER_INACTIVE_INFO_BYTES;
+ rsp_info->inactive_list.region[num] =
+ BE_BYTE(rsp_field->hpb_inactive_field, blk_idx);
+ hpb_dbg(hpb->hba, "inactive num: %d, #block: %d\n",
+ num + 1, rsp_info->inactive_list.region[num]);
+ }
+ rsp_info->inactive_cnt = num;
+
+ hpb_trace(hpb, "Noti: #ACTIVE %d, #INACTIVE %d",
+ rsp_info->active_cnt, rsp_info->inactive_cnt);
+ hpb_dbg(hpb->hba, "active cnt: %d, inactive cnt: %d\n",
+ rsp_info->active_cnt, rsp_info->inactive_cnt);
+ list_for_each_entry(region, &hpb->lru_info.lru, list_region)
+ hpb_dbg(hpb->hba, "active list : %d (cnt: %d)\n",
+ region->region, region->hit_count);
+ hpb_dbg(hpb->hba, "add_list %p -> %p\n",
+ rsp_info, &hpb->lh_rsp_info);
+
+ spin_lock(&hpb->rsp_list_lock);
+ list_add_tail(&rsp_info->list_rsp_info, &hpb->lh_rsp_info);
+ spin_unlock(&hpb->rsp_list_lock);
+
+ tasklet_schedule(&hpb->ufshpb_tasklet);
+ break;
+ default:
+ hpb_dbg(hpb->hba, "hpb_type is not available : %d\n",
+ rsp_field->hpb_type);
+ break;
+ }
+}
+
+static int ufshpb_read_desc(struct ufs_hba *hba,
+ u8 desc_id, u8 desc_index, u8 *desc_buf, u32 size)
+{
+ int err = 0;
+ u8 selector = 1;
+
+ err = ufshcd_query_descriptor(hba, UPIU_QUERY_OPCODE_READ_DESC,
+ desc_id, desc_index, selector, desc_buf, &size);
+ if (!err)
+ dev_dbg(hba->dev, "%s:%d reading Device Desc failed. err = %d\n",
+ __func__, __LINE__, err);
+ return err;
+}
+
+static int ufshpb_read_device_desc(struct ufs_hba *hba, u8 *desc_buf, u32 size)
+{
+ return ufshpb_read_desc(hba, QUERY_DESC_IDN_DEVICE, 0, desc_buf, size);
+}
+
+static int ufshpb_read_geo_desc(struct ufs_hba *hba, u8 *desc_buf, u32 size)
+{
+ return ufshpb_read_desc(hba, QUERY_DESC_IDN_GEOMETRY, 0,
+ desc_buf, size);
+}
+
+static int ufshpb_read_unit_desc(struct ufs_hba *hba, int lun,
+ u8 *desc_buf, u32 size)
+{
+ return ufshpb_read_desc(hba, QUERY_DESC_IDN_UNIT,
+ lun, desc_buf, size);
+}
+
+static int ufshpb_read_config_desc(struct ufs_hba *hba, u8 *desc_buf, u32 size)
+{
+ return ufshpb_read_desc(hba, QUERY_DESC_IDN_CONFIGURATION, 0,
+ desc_buf, size);
+}
+
+static inline void ufshpb_add_subregion_to_req_list(struct ufshpb_lu *hpb,
+ struct ufshpb_subregion *cp)
+{
+ list_add_tail(&cp->list_subregion, &hpb->lh_subregion_req);
+ cp->subregion_state = HPBSUBREGION_ISSUED;
+}
+
+static int ufshpb_execute_req(struct ufshpb_lu *hpb, unsigned char *cmd,
+ struct ufshpb_subregion *cp)
+{
+ unsigned long flags;
+ struct request_queue *q;
+ char sense[SCSI_SENSE_BUFFERSIZE];
+ struct scsi_sense_hdr sshdr;
+ struct scsi_device *sdp;
+ struct ufs_hba *hba = hpb->hba;
+ struct request req;
+ struct bio bio;
+ int ret = 0;
+
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ sdp = hba->sdev_ufs_lu[hpb->lun];
+ if (sdp) {
+ ret = scsi_device_get(sdp);
+ if (!ret && !scsi_device_online(sdp)) {
+ ret = -ENODEV;
+ scsi_device_put(sdp);
+ }
+ } else {
+ ret = -ENODEV;
+ }
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+ if (ret)
+ return ret;
+
+ q = sdp->request_queue;
+ ret = ufshpb_add_bio_page(hpb, q, &bio, hpb->bvec, cp->mctx);
+ if (ret)
+ goto put_out;
+
+ blk_rq_init(q, &req);
+ blk_rq_append_bio(&req, &bio);
+
+ req.cmd_len = COMMAND_SIZE(cmd[0]);
+ memcpy(req.cmd, cmd, req.cmd_len);
+ req.sense = sense;
+ req.sense_len = 0;
+ req.retries = 3;
+ req.timeout = msecs_to_jiffies(10000);
+ req.cmd_type = REQ_TYPE_BLOCK_PC;
+ req.cmd_flags = REQ_QUIET | REQ_PREEMPT;
+
+ blk_execute_rq(q, NULL, &req, 1);
+ if (req.errors) {
+ ret = -EIO;
+ scsi_normalize_sense(req.sense, SCSI_SENSE_BUFFERSIZE, &sshdr);
+ dev_err(HPB_DEV(hpb),
+ "code %x sense_key %x asc %x ascq %x",
+ sshdr.response_code, sshdr.sense_key, sshdr.asc,
+ sshdr.ascq);
+ dev_err(HPB_DEV(hpb),
+ "byte4 %x byte5 %x byte6 %x additional_len %x",
+ sshdr.byte4, sshdr.byte5, sshdr.byte6,
+ sshdr.additional_length);
+ spin_lock_bh(&hpb->hpb_lock);
+ ufshpb_error_active_subregion(hpb, cp);
+ spin_unlock_bh(&hpb->hpb_lock);
+ } else {
+ ret = 0;
+ spin_lock_bh(&hpb->hpb_lock);
+ ufshpb_clean_dirty_bitmap(hpb, cp);
+ spin_unlock_bh(&hpb->hpb_lock);
+ }
+put_out:
+ scsi_device_put(sdp);
+ return ret;
+}
+
+static int ufshpb_issue_map_req_from_list(struct ufshpb_lu *hpb)
+{
+ struct ufshpb_subregion *cp, *next_cp;
+ int ret;
+
+ LIST_HEAD(req_list);
+
+ spin_lock(&hpb->hpb_lock);
+ list_splice_init(&hpb->lh_subregion_req, &req_list);
+ spin_unlock(&hpb->hpb_lock);
+
+ list_for_each_entry_safe(cp, next_cp, &req_list, list_subregion) {
+ unsigned char cmd[10] = { 0 };
+
+ ufshpb_set_read_buf_cmd(cmd, cp->region, cp->subregion,
+ hpb->subregion_mem_size);
+
+ hpb_dbg(hpb->hba, "issue map_request: %d - %d\n",
+ cp->region, cp->subregion);
+
+ ret = ufshpb_execute_req(hpb, cmd, cp);
+ if (ret < 0) {
+ dev_err(HPB_DEV(hpb),
+ "region %d sub %d failed with err %d",
+ cp->region, cp->subregion, ret);
+ continue;
+ }
+
+ spin_lock_bh(&hpb->hpb_lock);
+ ufshpb_clean_active_subregion(hpb, cp);
+ list_del_init(&cp->list_subregion);
+ spin_unlock_bh(&hpb->hpb_lock);
+ }
+
+ return 0;
+}
+
+static void ufshpb_work_handler(struct work_struct *work)
+{
+ struct ufshpb_lu *hpb;
+ int ret;
+
+ hpb = container_of(work, struct ufshpb_lu, ufshpb_work);
+ hpb_dbg(hpb->hba, "worker start\n");
+
+ if (!list_empty(&hpb->lh_subregion_req)) {
+ pm_runtime_get_sync(HPB_DEV(hpb));
+ ret = ufshpb_issue_map_req_from_list(hpb);
+ /*
+ * if its function failed at init time,
+ * ufshpb-device will request map-req,
+ * so it is not critical-error, and just finish work-handler
+ */
+ if (ret)
+ hpb_dbg(hpb->hba, "failed map-issue. ret %d\n", ret);
+ pm_runtime_mark_last_busy(HPB_DEV(hpb));
+ pm_runtime_put_noidle(HPB_DEV(hpb));
+ }
+
+ hpb_dbg(hpb->hba, "worker end\n");
+}
+
+static void ufshpb_retry_work_handler(struct work_struct *work)
+{
+ struct ufshpb_lu *hpb;
+ struct delayed_work *dwork = to_delayed_work(work);
+ struct ufshpb_map_req *map_req;
+ int ret = 0;
+
+ LIST_HEAD(retry_list);
+
+ hpb = container_of(dwork, struct ufshpb_lu, ufshpb_retry_work);
+ hpb_dbg(hpb->hba, "retry worker start");
+
+ spin_lock_bh(&hpb->hpb_lock);
+ list_splice_init(&hpb->lh_map_req_retry, &retry_list);
+ spin_unlock_bh(&hpb->hpb_lock);
+
+ while (1) {
+ map_req = list_first_entry_or_null(&retry_list,
+ struct ufshpb_map_req, list_map_req);
+ if (!map_req) {
+ hpb_dbg(hpb->hba, "There is no map_req");
+ break;
+ }
+ list_del(&map_req->list_map_req);
+
+ map_req->retry_cnt++;
+
+ ret = ufshpb_map_req_issue(hpb,
+ hpb->hba->sdev_ufs_lu[hpb->lun]->request_queue,
+ map_req);
+ if (ret) {
+ hpb_dbg(hpb->hba, "ufshpb_set_map_req error %d", ret);
+ goto wakeup_ee_worker;
+ }
+ }
+ hpb_dbg(hpb->hba, "worker end");
+ return;
+
+wakeup_ee_worker:
+ hpb->hba->ufshpb_state = HPB_FAILED;
+ schedule_work(&hpb->hba->ufshpb_eh_work);
+}
+
+static void ufshpb_tasklet_fn(unsigned long private)
+{
+ struct ufshpb_lu *hpb = (struct ufshpb_lu *)private;
+ struct ufshpb_rsp_info *rsp_info;
+ unsigned long flags;
+
+ while (1) {
+ spin_lock_irqsave(&hpb->rsp_list_lock, flags);
+ rsp_info = list_first_entry_or_null(&hpb->lh_rsp_info,
+ struct ufshpb_rsp_info, list_rsp_info);
+ if (!rsp_info) {
+ spin_unlock_irqrestore(&hpb->rsp_list_lock, flags);
+ break;
+ }
+
+ rsp_info->RSP_tasklet_enter = ktime_to_ns(ktime_get());
+
+ list_del_init(&rsp_info->list_rsp_info);
+ spin_unlock_irqrestore(&hpb->rsp_list_lock, flags);
+
+ switch (rsp_info->type) {
+ case HPB_RSP_REQ_REGION_UPDATE:
+ ufshpb_rsp_map_cmd_req(hpb, rsp_info);
+ break;
+ default:
+ break;
+ }
+
+ spin_lock_irqsave(&hpb->rsp_list_lock, flags);
+ list_add_tail(&rsp_info->list_rsp_info, &hpb->lh_rsp_info_free);
+ spin_unlock_irqrestore(&hpb->rsp_list_lock, flags);
+ }
+}
+
+static void ufshpb_init_constant(void)
+{
+ sects_per_blk_shift = ffs(BLOCK) - ffs(SECTOR);
+ dev_dbg(NULL, "sects_per_blk_shift: %u %u\n",
+ sects_per_blk_shift, ffs(SECTORS_PER_BLOCK) - 1);
+
+ bits_per_dword_shift = ffs(BITS_PER_DWORD) - 1;
+ bits_per_dword_mask = BITS_PER_DWORD - 1;
+ dev_dbg(NULL, "bits_per_dword %u shift %u mask 0x%X\n",
+ BITS_PER_DWORD, bits_per_dword_shift, bits_per_dword_mask);
+
+ bits_per_byte_shift = ffs(BITS_PER_BYTE) - 1;
+ dev_dbg(NULL, "bits_per_byte %u shift %u\n",
+ BITS_PER_BYTE, bits_per_byte_shift);
+}
+
+static inline void ufshpb_req_mempool_remove(struct ufshpb_lu *hpb)
+{
+ if (!hpb)
+ return;
+ kfree(hpb->rsp_info);
+ kfree(hpb->map_req);
+}
+
+static void ufshpb_table_mempool_remove(struct ufshpb_lu *hpb)
+{
+ struct ufshpb_map_ctx *mctx, *next;
+ int i;
+
+ /*
+ * the mctx in the lh_map_ctx has been allocated completely.
+ */
+ list_for_each_entry_safe(mctx, next, &hpb->lh_map_ctx, list_table) {
+ for (i = 0; i < hpb->mpages_per_subregion; i++)
+ __free_page(mctx->m_page[i]);
+
+ vfree(mctx->ppn_dirty);
+ kfree(mctx->m_page);
+ kfree(mctx);
+ alloc_mctx--;
+ }
+}
+
+static int ufshpb_init_pinned_active_block(struct ufshpb_lu *hpb,
+ struct ufshpb_region *cb)
+{
+ struct ufshpb_subregion *cp;
+ int subregion, j;
+ int err = 0;
+
+ for (subregion = 0 ; subregion < cb->subregion_count ; subregion++) {
+ cp = cb->subregion_tbl + subregion;
+
+ cp->mctx = ufshpb_get_map_ctx(hpb);
+ if (IS_ERR(cp->mctx)) {
+ err = PTR_ERR(cp->mctx);
+ goto release;
+ }
+ spin_lock(&hpb->hpb_lock);
+ ufshpb_add_subregion_to_req_list(hpb, cp);
+ spin_unlock(&hpb->hpb_lock);
+ }
+
+ cb->region_state = HPBREGION_PINNED;
+ return 0;
+
+release:
+ for (j = 0 ; j < subregion ; j++) {
+ cp = cb->subregion_tbl + j;
+ ufshpb_put_map_ctx(hpb, cp->mctx);
+ }
+ return err;
+}
+
+static inline bool ufshpb_is_HPBREGION_PINNED(
+ struct ufshpb_lu_desc *lu_desc, int region)
+{
+ if (lu_desc->lu_hpb_pinned_end_offset != -1 &&
+ region >= lu_desc->hpb_pinned_region_startidx &&
+ region <= lu_desc->lu_hpb_pinned_end_offset)
+ return true;
+
+ return false;
+}
+
+static inline void ufshpb_init_jobs(struct ufshpb_lu *hpb)
+{
+ INIT_WORK(&hpb->ufshpb_work, ufshpb_work_handler);
+ INIT_DELAYED_WORK(&hpb->ufshpb_retry_work, ufshpb_retry_work_handler);
+ tasklet_init(&hpb->ufshpb_tasklet, ufshpb_tasklet_fn,
+ (unsigned long)hpb);
+}
+
+static inline void ufshpb_cancel_jobs(struct ufshpb_lu *hpb)
+{
+ cancel_work_sync(&hpb->ufshpb_work);
+ cancel_delayed_work_sync(&hpb->ufshpb_retry_work);
+ tasklet_kill(&hpb->ufshpb_tasklet);
+}
+
+static void ufshpb_init_subregion_tbl(struct ufshpb_lu *hpb,
+ struct ufshpb_region *cb)
+{
+ int subregion;
+
+ for (subregion = 0 ; subregion < cb->subregion_count ; subregion++) {
+ struct ufshpb_subregion *cp = cb->subregion_tbl + subregion;
+
+ cp->region = cb->region;
+ cp->subregion = subregion;
+ cp->subregion_state = HPBSUBREGION_UNUSED;
+ }
+}
+
+static inline int ufshpb_alloc_subregion_tbl(struct ufshpb_lu *hpb,
+ struct ufshpb_region *cb, int subregion_count)
+{
+ cb->subregion_tbl = kzalloc(
+ sizeof(struct ufshpb_subregion) * subregion_count,
+ GFP_KERNEL);
+ if (!cb->subregion_tbl)
+ return -ENOMEM;
+
+ cb->subregion_count = subregion_count;
+ hpb_dbg(hpb->hba,
+ "region %d subregion_count %d active_page_table bytes %lu\n",
+ cb->region, subregion_count,
+ sizeof(struct ufshpb_subregion *) * hpb->subregions_per_region);
+
+ return 0;
+}
+
+static int ufshpb_table_mempool_init(struct ufshpb_lu *hpb,
+ int num_regions, int subregions_per_region,
+ int entry_count, int entry_byte)
+{
+ int i, j;
+ struct ufshpb_map_ctx *mctx = NULL;
+
+ INIT_LIST_HEAD(&hpb->lh_map_ctx);
+
+ for (i = 0 ; i < num_regions * subregions_per_region ; i++) {
+ mctx = kzalloc(sizeof(struct ufshpb_map_ctx), GFP_KERNEL);
+ if (!mctx)
+ goto release_mem;
+
+ mctx->m_page = kzalloc(sizeof(struct page *) *
+ hpb->mpages_per_subregion, GFP_KERNEL);
+ if (!mctx->m_page)
+ goto release_mem;
+
+ mctx->ppn_dirty = vzalloc(entry_count >>
+ bits_per_byte_shift);
+ if (!mctx->ppn_dirty)
+ goto release_mem;
+
+ for (j = 0; j < hpb->mpages_per_subregion; j++) {
+ mctx->m_page[j] = alloc_page(GFP_KERNEL | __GFP_ZERO);
+ if (!mctx->m_page[j])
+ goto release_mem;
+ }
+ hpb_dbg(hpb->hba, "[%d] mctx->m_page %p get_order %d\n", i,
+ mctx->m_page, get_order(hpb->mpages_per_subregion));
+
+ INIT_LIST_HEAD(&mctx->list_table);
+ list_add(&mctx->list_table, &hpb->lh_map_ctx);
+
+ hpb->debug_free_table++;
+ }
+
+ alloc_mctx = num_regions * subregions_per_region;
+ hpb_dbg(hpb->hba, "number of mctx %d %d %d. debug_free_table %d\n",
+ num_regions * subregions_per_region, num_regions,
+ subregions_per_region, hpb->debug_free_table);
+ return 0;
+
+release_mem:
+ /*
+ * mctxs already added in lh_map_ctx will be removed
+ * in the caller function.
+ */
+ if (!mctx)
+ goto out;
+
+ if (mctx->m_page) {
+ for (j = 0; j < hpb->mpages_per_subregion; j++)
+ if (mctx->m_page[i])
+ __free_page(mctx->m_page[j]);
+ kfree(mctx->m_page);
+ }
+ if (mctx->ppn_dirty)
+ vfree(mctx->ppn_dirty);
+ kfree(mctx);
+out:
+ return -ENOMEM;
+}
+
+static int ufshpb_req_mempool_init(struct ufs_hba *hba,
+ struct ufshpb_lu *hpb, int queue_depth)
+{
+ struct ufshpb_rsp_info *rsp_info = NULL;
+ struct ufshpb_map_req *map_req = NULL;
+ int map_req_cnt = 0;
+ int i;
+
+ if (!queue_depth) {
+ queue_depth = hba->nutrs;
+ hpb_dbg(hba,
+ "lu_queue_depth is 0. we use device's queue info.\n");
+ hpb_dbg(hba, "hba->nutrs = %d\n", hba->nutrs);
+ }
+
+ INIT_LIST_HEAD(&hpb->lh_rsp_info_free);
+ INIT_LIST_HEAD(&hpb->lh_map_req_free);
+ INIT_LIST_HEAD(&hpb->lh_map_req_retry);
+
+ hpb->rsp_info = kcalloc(queue_depth, sizeof(struct ufshpb_rsp_info),
+ GFP_KERNEL);
+ if (!hpb->rsp_info)
+ goto release_mem;
+
+ map_req_cnt = max(hpb->subregions_per_lu, queue_depth);
+
+ hpb->map_req = kcalloc(map_req_cnt, sizeof(struct ufshpb_map_req),
+ GFP_KERNEL);
+ if (!hpb->map_req)
+ goto release_mem;
+
+ for (i = 0; i < queue_depth; i++) {
+ rsp_info = hpb->rsp_info + i;
+ INIT_LIST_HEAD(&rsp_info->list_rsp_info);
+ list_add_tail(&rsp_info->list_rsp_info, &hpb->lh_rsp_info_free);
+ }
+
+ for (i = 0; i < map_req_cnt; i++) {
+ map_req = hpb->map_req + i;
+ INIT_LIST_HEAD(&map_req->list_map_req);
+ list_add_tail(&map_req->list_map_req, &hpb->lh_map_req_free);
+ }
+
+ return 0;
+
+release_mem:
+ kfree(hpb->rsp_info);
+ return -ENOMEM;
+}
+
+static void ufshpb_init_lu_constant(struct ufshpb_lu *hpb,
+ struct ufshpb_lu_desc *lu_desc,
+ struct ufshpb_func_desc *func_desc)
+{
+ unsigned long long region_unit_size, region_mem_size;
+ int entries_per_region;
+
+ /* From descriptors */
+ region_unit_size = (unsigned long long)
+ SECTOR * (0x01 << func_desc->hpb_region_size);
+ region_mem_size = region_unit_size / BLOCK * HPB_ENTRY_SIZE;
+
+ hpb->subregion_unit_size = (unsigned long long)
+ SECTOR * (0x01 << func_desc->hpb_subregion_size);
+ hpb->subregion_mem_size = hpb->subregion_unit_size /
+ BLOCK * HPB_ENTRY_SIZE;
+
+ hpb->hpb_ver = func_desc->hpb_ver;
+ hpb->lu_max_active_regions = lu_desc->lu_max_active_hpb_regions;
+ hpb->lru_info.max_lru_active_cnt =
+ lu_desc->lu_max_active_hpb_regions
+ - lu_desc->lu_num_hpb_pinned_regions;
+
+ /* relation : lu <-> region <-> sub region <-> entry */
+ hpb->lu_num_blocks = lu_desc->lu_logblk_cnt;
+ entries_per_region = region_mem_size / HPB_ENTRY_SIZE;
+ hpb->entries_per_subregion = hpb->subregion_mem_size / HPB_ENTRY_SIZE;
+ hpb->subregions_per_region = region_mem_size / hpb->subregion_mem_size;
+
+ /*
+ * 1. regions_per_lu
+ * = (lu_num_blocks * 4096) / region_unit_size
+ * = (lu_num_blocks * HPB_ENTRY_SIZE) / region_mem_size
+ * = lu_num_blocks / (region_mem_size / HPB_ENTRY_SIZE)
+ *
+ * 2. regions_per_lu = lu_num_blocks / subregion_mem_size (is trik...)
+ * if HPB_ENTRY_SIZE != subregions_per_region, it is error.
+ */
+ hpb->regions_per_lu = ((unsigned long long)hpb->lu_num_blocks
+ + (region_mem_size / HPB_ENTRY_SIZE) - 1)
+ / (region_mem_size / HPB_ENTRY_SIZE);
+ hpb->subregions_per_lu = ((unsigned long long)hpb->lu_num_blocks
+ + (hpb->subregion_mem_size / HPB_ENTRY_SIZE) - 1)
+ / (hpb->subregion_mem_size / HPB_ENTRY_SIZE);
+
+ /* mempool info */
+ hpb->mpage_bytes = OS_PAGE_SIZE;
+ hpb->mpages_per_subregion = hpb->subregion_mem_size / hpb->mpage_bytes;
+
+ /* Bitmask Info. */
+ hpb->dwords_per_subregion = hpb->entries_per_subregion / BITS_PER_DWORD;
+ hpb->entries_per_region_shift = ffs(entries_per_region) - 1;
+ hpb->entries_per_region_mask = entries_per_region - 1;
+ hpb->entries_per_subregion_shift = ffs(hpb->entries_per_subregion) - 1;
+ hpb->entries_per_subregion_mask = hpb->entries_per_subregion - 1;
+
+ pr_info("===== From Device Descriptor! =====\n");
+ pr_info("hpb_region_size = %d, hpb_subregion_size = %d\n",
+ func_desc->hpb_region_size,
+ func_desc->hpb_subregion_size);
+ pr_info("===== Constant Values(LU) =====\n");
+ pr_info("region_unit_size = %llu, region_mem_size %llu\n",
+ region_unit_size, region_mem_size);
+ pr_info("subregion_unit_size = %llu, subregion_mem_size %d\n",
+ hpb->subregion_unit_size, hpb->subregion_mem_size);
+ pr_info("lu_num_blks = %d, reg_per_lu = %d, subreg_per_lu = %d\n",
+ hpb->lu_num_blocks, hpb->regions_per_lu,
+ hpb->subregions_per_lu);
+ pr_info("subregions_per_region = %d\n",
+ hpb->subregions_per_region);
+ pr_info("entries_per_region %u shift %u mask 0x%X\n",
+ entries_per_region, hpb->entries_per_region_shift,
+ hpb->entries_per_region_mask);
+ pr_info("entries_per_subregion %u shift %u mask 0x%X\n",
+ hpb->entries_per_subregion,
+ hpb->entries_per_subregion_shift,
+ hpb->entries_per_subregion_mask);
+ pr_info("mpages_per_subregion : %d\n",
+ hpb->mpages_per_subregion);
+ pr_info("===================================\n");
+}
+
+static int ufshpb_lu_hpb_init(struct ufs_hba *hba, struct ufshpb_lu *hpb,
+ struct ufshpb_func_desc *func_desc,
+ struct ufshpb_lu_desc *lu_desc, int lun)
+{
+ struct ufshpb_region *region_table, *cb;
+ struct ufshpb_subregion *cp;
+ int region, subregion;
+ int total_subregion_count, subregion_count;
+ bool do_work_handler;
+ int ret, j;
+
+ hpb->lu_hpb_enable = true;
+
+ ufshpb_init_lu_constant(hpb, lu_desc, func_desc);
+
+ region_table = kzalloc(sizeof(struct ufshpb_region) *
+ hpb->regions_per_lu, GFP_KERNEL);
+ if (!region_table)
+ return -ENOMEM;
+
+ hpb_dbg(hba, "active_block_table bytes: %lu\n",
+ (sizeof(struct ufshpb_region) * hpb->regions_per_lu));
+
+ hpb->region_tbl = region_table;
+
+ spin_lock_init(&hpb->hpb_lock);
+ spin_lock_init(&hpb->rsp_list_lock);
+
+ /* init lru information */
+ INIT_LIST_HEAD(&hpb->lru_info.lru);
+ hpb->lru_info.selection_type = LRU;
+
+ INIT_LIST_HEAD(&hpb->lh_subregion_req);
+ INIT_LIST_HEAD(&hpb->lh_rsp_info);
+ INIT_LIST_HEAD(&hpb->lh_map_ctx);
+
+ ret = ufshpb_table_mempool_init(hpb,
+ lu_desc->lu_max_active_hpb_regions,
+ hpb->subregions_per_region,
+ hpb->entries_per_subregion, HPB_ENTRY_SIZE);
+ if (ret) {
+ dev_err(HPB_DEV(hpb), "ppn table mempool init fail!\n");
+ goto release_mempool;
+ }
+
+ ret = ufshpb_req_mempool_init(hba, hpb, lu_desc->lu_queue_depth);
+ if (ret) {
+ dev_err(HPB_DEV(hpb), "rsp_info_mempool init fail!\n");
+ goto release_mempool;
+ }
+
+ total_subregion_count = hpb->subregions_per_lu;
+
+ ufshpb_init_jobs(hpb);
+
+ hpb_dbg(hba, "total_subregion_count: %d\n", total_subregion_count);
+ for (region = 0, subregion_count = 0,
+ total_subregion_count = hpb->subregions_per_lu;
+ region < hpb->regions_per_lu;
+ region++, total_subregion_count -= subregion_count) {
+ cb = region_table + region;
+ cb->region = region;
+
+ /* init lru region information*/
+ INIT_LIST_HEAD(&cb->list_region);
+ cb->hit_count = 0;
+
+ subregion_count = min(total_subregion_count,
+ hpb->subregions_per_region);
+ hpb_dbg(hba, "total: %d subregion_count: %d\n",
+ total_subregion_count, subregion_count);
+
+ ret = ufshpb_alloc_subregion_tbl(hpb, cb, subregion_count);
+ if (ret)
+ goto release_region_cp;
+ ufshpb_init_subregion_tbl(hpb, cb);
+
+ if (ufshpb_is_HPBREGION_PINNED(lu_desc, region)) {
+ hpb_dbg(hba, "region: %d PINNED %d ~ %d\n",
+ region, lu_desc->hpb_pinned_region_startidx,
+ lu_desc->lu_hpb_pinned_end_offset);
+ ret = ufshpb_init_pinned_active_block(hpb, cb);
+ if (ret)
+ goto release_region_cp;
+ do_work_handler = true;
+ } else {
+ hpb_dbg(hba, "region: %d inactive\n", cb->region);
+ cb->region_state = HPBREGION_INACTIVE;
+ }
+ }
+
+ if (total_subregion_count != 0) {
+ dev_err(HPB_DEV(hpb),
+ "error total_subregion_count: %d\n",
+ total_subregion_count);
+ goto release_region_cp;
+ }
+
+ hpb->hba = hba;
+ hpb->lun = lun;
+
+ if (do_work_handler)
+ schedule_work(&hpb->ufshpb_work);
+
+ /*
+ * even if creating sysfs failed, ufshpb could run normally.
+ * so we don't deal with error handling
+ */
+ ufshpb_create_sysfs(hba, hpb);
+ return 0;
+
+release_region_cp:
+ for (j = 0 ; j < region ; j++) {
+ cb = region_table + j;
+ if (cb->subregion_tbl) {
+ for (subregion = 0; subregion < cb->subregion_count;
+ subregion++) {
+ cp = cb->subregion_tbl + subregion;
+
+ if (cp->mctx)
+ ufshpb_put_map_ctx(hpb, cp->mctx);
+ }
+ kfree(cb->subregion_tbl);
+ }
+ }
+
+release_mempool:
+ ufshpb_table_mempool_remove(hpb);
+ kfree(region_table);
+ return ret;
+}
+
+static int ufshpb_get_hpb_lu_desc(struct ufs_hba *hba,
+ struct ufshpb_lu_desc *lu_desc, int lun)
+{
+ int ret;
+ u8 logical_buf[UFSHPB_QUERY_DESC_UNIT_MAX_SIZE] = { 0 };
+
+ ret = ufshpb_read_unit_desc(hba, lun, logical_buf,
+ UFSHPB_QUERY_DESC_UNIT_MAX_SIZE);
+ if (ret) {
+ dev_err(hba->dev, "read unit desc failed. ret %d\n", ret);
+ return ret;
+ }
+
+ lu_desc->lu_queue_depth = logical_buf[UNIT_DESC_PARAM_LU_Q_DEPTH];
+
+ // 2^log, ex) 0x0C = 4KB
+ lu_desc->lu_logblk_size = logical_buf[UNIT_DESC_PARAM_LOGICAL_BLK_SIZE];
+ lu_desc->lu_logblk_cnt =
+ SHIFT_BYTE_7((u64)
+ logical_buf[UNIT_DESC_PARAM_LOGICAL_BLK_COUNT]) |
+ SHIFT_BYTE_6((u64)
+ logical_buf[UNIT_DESC_PARAM_LOGICAL_BLK_COUNT + 1]) |
+ SHIFT_BYTE_5((u64)
+ logical_buf[UNIT_DESC_PARAM_LOGICAL_BLK_COUNT + 2]) |
+ SHIFT_BYTE_4((u64)
+ logical_buf[UNIT_DESC_PARAM_LOGICAL_BLK_COUNT + 3]) |
+ SHIFT_BYTE_3(
+ logical_buf[UNIT_DESC_PARAM_LOGICAL_BLK_COUNT + 4]) |
+ SHIFT_BYTE_2(
+ logical_buf[UNIT_DESC_PARAM_LOGICAL_BLK_COUNT + 5]) |
+ SHIFT_BYTE_1(
+ logical_buf[UNIT_DESC_PARAM_LOGICAL_BLK_COUNT + 6]) |
+ SHIFT_BYTE_0(
+ logical_buf[UNIT_DESC_PARAM_LOGICAL_BLK_COUNT + 7]);
+
+ if (logical_buf[UNIT_DESC_PARAM_LU_ENABLE] == LU_HPB_ENABLE)
+ lu_desc->lu_hpb_enable = true;
+ else
+ lu_desc->lu_hpb_enable = false;
+
+ lu_desc->lu_max_active_hpb_regions =
+ BE_BYTE(logical_buf, UNIT_DESC_HPB_LU_MAX_ACTIVE_REGIONS);
+ lu_desc->hpb_pinned_region_startidx =
+ BE_BYTE(logical_buf, UNIT_DESC_HPB_LU_PIN_REGION_START_OFFSET);
+ lu_desc->lu_num_hpb_pinned_regions =
+ BE_BYTE(logical_buf, UNIT_DESC_HPB_LU_NUM_PIN_REGIONS);
+
+ if (lu_desc->lu_hpb_enable) {
+ hpb_dbg(hba, "LUN(%d) [0A] bLogicalBlockSize %d\n",
+ lun, lu_desc->lu_logblk_size);
+ hpb_dbg(hba, "LUN(%d) [0B] qLogicalBlockCount %llu\n",
+ lun, lu_desc->lu_logblk_cnt);
+ hpb_dbg(hba, "LUN(%d) [03] bLuEnable %d\n",
+ lun, logical_buf[UNIT_DESC_PARAM_LU_ENABLE]);
+ hpb_dbg(hba, "LUN(%d) [06] bLuQueueDepth %d\n",
+ lun, lu_desc->lu_queue_depth);
+ hpb_dbg(hba, "LUN(%d) [23:24] wLUMaxActiveHPBRegions %d\n",
+ lun, lu_desc->lu_max_active_hpb_regions);
+ hpb_dbg(hba, "LUN(%d) [25:26] wHPBPinnedRegionStartIdx %d\n",
+ lun, lu_desc->hpb_pinned_region_startidx);
+ hpb_dbg(hba, "LUN(%d) [27:28] wNumHPBPinnedRegions %d\n",
+ lun, lu_desc->lu_num_hpb_pinned_regions);
+ }
+
+ if (lu_desc->lu_num_hpb_pinned_regions > 0) {
+ lu_desc->lu_hpb_pinned_end_offset =
+ lu_desc->hpb_pinned_region_startidx +
+ lu_desc->lu_num_hpb_pinned_regions - 1;
+ } else
+ lu_desc->lu_hpb_pinned_end_offset = -1;
+
+ if (lu_desc->lu_hpb_enable)
+ pr_info("UFSHPB: Enable, LU: %d, REG: %d, PIN: %d - %d\n",
+ lun,
+ lu_desc->lu_max_active_hpb_regions,
+ lu_desc->hpb_pinned_region_startidx,
+ lu_desc->lu_num_hpb_pinned_regions);
+ return 0;
+}
+
+static int ufshpb_read_dev_desc_support(struct ufs_hba *hba,
+ struct ufshpb_func_desc *desc)
+{
+ u8 desc_buf[UFSHPB_QUERY_DESC_DEVICE_MAX_SIZE];
+ int err;
+
+ err = ufshpb_read_device_desc(hba, desc_buf,
+ UFSHPB_QUERY_DESC_DEVICE_MAX_SIZE);
+ if (err)
+ return err;
+
+ if (desc_buf[DEVICE_DESC_PARAM_FEAT_SUP] &
+ UFS_FEATURE_SUPPORT_HPB_BIT) {
+ hba->ufshpb_feat |= UFS_FEATURE_SUPPORT_HPB_BIT;
+ pr_info("UFSHPB: FeaturesSupport = support\n");
+ } else {
+ pr_info("UFSHPB: FeaturesSupport = not support\n");
+ return -ENODEV;
+ }
+
+ desc->lu_cnt = desc_buf[DEVICE_DESC_PARAM_NUM_LU];
+ hpb_dbg(hba, "device lu count %d\n", desc->lu_cnt);
+
+ desc->hpb_ver =
+ (u16)SHIFT_BYTE_1(desc_buf[DEVICE_DESC_PARAM_HPB_VER]) |
+ (u16)SHIFT_BYTE_0(desc_buf[DEVICE_DESC_PARAM_HPB_VER + 1]);
+
+ pr_info("UFSHPB: Version = %.2x.%.2x, Driver Version = %.2x.%.2x\n",
+ GET_BYTE_1(desc->hpb_ver),
+ GET_BYTE_0(desc->hpb_ver),
+ GET_BYTE_1(UFSHPB_VER),
+ GET_BYTE_0(UFSHPB_VER));
+
+ if (desc->hpb_ver != UFSHPB_VER)
+ return -ENODEV;
+ return 0;
+}
+
+static int ufshpb_read_geo_desc_support(struct ufs_hba *hba,
+ struct ufshpb_func_desc *desc)
+{
+ int err;
+ u8 geometry_buf[UFSHPB_QUERY_DESC_GEOMETRY_MAX_SIZE];
+
+ err = ufshpb_read_geo_desc(hba, geometry_buf,
+ UFSHPB_QUERY_DESC_GEOMETRY_MAX_SIZE);
+ if (err)
+ return err;
+
+ desc->hpb_region_size = geometry_buf[GEOMETRY_DESC_HPB_REGION_SIZE];
+ desc->hpb_number_lu = geometry_buf[GEOMETRY_DESC_HPB_NUMBER_LU];
+ desc->hpb_subregion_size =
+ geometry_buf[GEOMETRY_DESC_HPB_SUBREGION_SIZE];
+ desc->hpb_device_max_active_regions =
+ BE_BYTE(geometry_buf,
+ GEOMETRY_DESC_HPB_DEVICE_MAX_ACTIVE_REGIONS);
+
+ hpb_dbg(hba, "[48] bHPBRegionSiz %u\n", desc->hpb_region_size);
+ hpb_dbg(hba, "[49] bHPBNumberLU %u\n", desc->hpb_number_lu);
+ hpb_dbg(hba, "[4A] bHPBSubRegionSize %u\n", desc->hpb_subregion_size);
+ hpb_dbg(hba, "[4B:4C] wDeviceMaxActiveHPBRegions %u\n",
+ desc->hpb_device_max_active_regions);
+
+ if (desc->hpb_number_lu == 0) {
+ dev_warn(hba->dev, "UFSHPB: HPB is not supported\n");
+ return -ENODEV;
+ }
+ /* for activation */
+ hba->ufshpb_max_regions = desc->hpb_device_max_active_regions;
+ return 0;
+}
+
+int ufshpb_control_validation(struct ufs_hba *hba,
+ struct ufshpb_config_desc *config)
+{
+ unsigned int num_regions = 0;
+ int lun;
+
+ if (!(hba->ufshpb_feat & UFS_FEATURE_SUPPORT_HPB_BIT))
+ return -ENOTSUPP;
+
+ for (lun = 0 ; lun < UFS_UPIU_MAX_GENERAL_LUN ; lun++) {
+ unsigned char *unit = config->unit[lun];
+
+ if (unit[UFSHPB_CONF_LU_ENABLE] >= LU_SET_MAX)
+ return -EINVAL;
+
+ /* total should not exceed max_active_regions */
+ num_regions += unit[UFSHPB_CONF_ACTIVE_REGIONS] << 8;
+ num_regions += unit[UFSHPB_CONF_ACTIVE_REGIONS + 1];
+ if (num_regions > hba->ufshpb_max_regions)
+ return -EINVAL;
+ }
+ return 0;
+}
+
+static int ufshpb_control_activation(struct ufs_hba *hba, bool activate)
+{
+ struct ufshpb_config_desc config;
+ unsigned char *unit;
+ unsigned short num_regions;
+ unsigned short pinned_start;
+ unsigned short pinned_num;
+ unsigned char enable;
+ u32 length = UFSHPB_CONFIG_LEN;
+ int lun = 0; /* LU:0 for Android */
+ int ret;
+
+ if (activate) {
+ /* pinned system image: 0 ~ 4GB range */
+ enable = LU_HPB_ENABLE;
+ num_regions = hba->ufshpb_max_regions;
+ pinned_start = 0;
+ pinned_num = 2;
+ } else {
+ enable = LU_ENABLE;
+ num_regions = 0;
+ pinned_start = 0;
+ pinned_num = 0;
+ }
+
+ pr_info("UFSHPB: Control - %s, LU: %d, REG: %d, PIN: %d - %d\n",
+ activate ? "Enable" : "Disable",
+ lun, num_regions, pinned_start, pinned_num);
+
+ ret = ufshpb_read_config_desc(hba, (u8 *)&config, length);
+ if (ret)
+ return ret;
+
+ unit = config.unit[lun];
+ unit[UFSHPB_CONF_LU_ENABLE] = enable;
+ unit[UFSHPB_CONF_ACTIVE_REGIONS] = (num_regions >> 8) & 0xff;
+ unit[UFSHPB_CONF_ACTIVE_REGIONS + 1] = num_regions & 0xff;
+ unit[UFSHPB_CONF_PINNED_START] = (pinned_start >> 8) & 0xff;
+ unit[UFSHPB_CONF_PINNED_START + 1] = pinned_start & 0xff;
+ unit[UFSHPB_CONF_PINNED_NUM] = (pinned_num >> 8) & 0xff;
+ unit[UFSHPB_CONF_PINNED_NUM + 1] = pinned_num & 0xff;
+
+ return ufshcd_query_descriptor(hba, UPIU_QUERY_OPCODE_WRITE_DESC,
+ QUERY_DESC_IDN_CONFIGURATION, 0, 1,
+ (u8 *)&config, &length);
+}
+
+static int ufshpb_init(struct ufs_hba *hba)
+{
+ struct ufshpb_func_desc func_desc;
+ int lun, ret;
+ int hpb_dev = 0;
+ int activate = CONFIG_SCSI_UFSHCD_HPB_ACTIVATE;
+ bool need_disable = false;
+
+ pm_runtime_get_sync(hba->dev);
+
+ ret = ufshpb_read_dev_desc_support(hba, &func_desc);
+ if (ret)
+ goto out_state;
+
+ ret = ufshpb_read_geo_desc_support(hba, &func_desc);
+ if (ret)
+ goto out_state;
+
+ ufshpb_init_constant();
+reinit:
+ for (lun = 0 ; lun < UFS_UPIU_MAX_GENERAL_LUN ; lun++) {
+ struct ufshpb_lu_desc lu_desc;
+
+ ret = ufshpb_get_hpb_lu_desc(hba, &lu_desc, lun);
+ if (ret)
+ goto out_state;
+
+ if (lu_desc.lu_hpb_enable == false)
+ continue;
+
+ if (!activate) {
+ need_disable = true;
+ break;
+ }
+ hba->ufshpb_lup[lun] = kzalloc(sizeof(struct ufshpb_lu),
+ GFP_KERNEL);
+ if (!hba->ufshpb_lup[lun])
+ goto out_free_mem;
+
+ ret = ufshpb_lu_hpb_init(hba, hba->ufshpb_lup[lun],
+ &func_desc, &lu_desc, lun);
+ if (ret) {
+ if (ret == -ENODEV)
+ continue;
+ else
+ goto out_free_mem;
+ }
+ hpb_dev++;
+ }
+
+ if (hpb_dev)
+ goto done;
+
+ if (activate) {
+ if (!ufshpb_control_activation(hba, true))
+ goto reinit;
+ goto out_free_mem;
+ }
+ if (need_disable) {
+ ufshpb_control_activation(hba, false);
+ ret = -ENOTSUPP;
+ }
+ goto out_free_mem;
+done:
+ INIT_WORK(&hba->ufshpb_eh_work, ufshpb_error_handler);
+ hba->ufshpb_state = HPB_PRESENT;
+ hba->issue_ioctl = false;
+ pm_runtime_mark_last_busy(hba->dev);
+ pm_runtime_put_noidle(hba->dev);
+ return 0;
+
+out_free_mem:
+ ufshpb_release(hba, HPB_NOT_SUPPORTED);
+out_state:
+ hba->ufshpb_state = HPB_NOT_SUPPORTED;
+ pm_runtime_mark_last_busy(hba->dev);
+ pm_runtime_put_noidle(hba->dev);
+ return ret;
+}
+
+static void ufshpb_map_loading_trigger(struct ufshpb_lu *hpb,
+ bool dirty, bool only_pinned, bool do_work_handler)
+{
+ int region, subregion;
+
+ if (do_work_handler)
+ goto work_out;
+
+ for (region = 0 ; region < hpb->regions_per_lu ; region++) {
+ struct ufshpb_region *cb;
+
+ cb = hpb->region_tbl + region;
+
+ if (cb->region_state != HPBREGION_ACTIVE &&
+ cb->region_state != HPBREGION_PINNED)
+ continue;
+
+ if ((only_pinned && cb->region_state == HPBREGION_PINNED) ||
+ !only_pinned) {
+ spin_lock(&hpb->hpb_lock);
+ for (subregion = 0; subregion < cb->subregion_count;
+ subregion++)
+ ufshpb_add_subregion_to_req_list(hpb,
+ cb->subregion_tbl + subregion);
+ spin_unlock(&hpb->hpb_lock);
+ do_work_handler = true;
+ }
+ if (dirty) {
+ for (subregion = 0; subregion < cb->subregion_count;
+ subregion++)
+ cb->subregion_tbl[subregion].subregion_state =
+ HPBSUBREGION_DIRTY;
+ }
+ }
+work_out:
+ if (do_work_handler)
+ schedule_work(&hpb->ufshpb_work);
+}
+
+static void ufshpb_purge_active_block(struct ufshpb_lu *hpb)
+{
+ int region, subregion;
+ int state;
+ struct ufshpb_region *cb;
+ struct ufshpb_subregion *cp;
+
+ spin_lock(&hpb->hpb_lock);
+ for (region = 0 ; region < hpb->regions_per_lu ; region++) {
+ cb = hpb->region_tbl + region;
+
+ if (cb->region_state == HPBREGION_INACTIVE) {
+ hpb_dbg(hpb->hba, "region %d inactive\n", region);
+ continue;
+ }
+
+ if (cb->region_state == HPBREGION_PINNED) {
+ state = HPBSUBREGION_DIRTY;
+ } else if (cb->region_state == HPBREGION_ACTIVE) {
+ state = HPBSUBREGION_UNUSED;
+ ufshpb_cleanup_lru_info(&hpb->lru_info, cb);
+ } else {
+ hpb_dbg(hpb->hba, "Unsupported state of region\n");
+ continue;
+ }
+
+ hpb_dbg(hpb->hba, "region %d state %d dft %d\n",
+ region, state,
+ hpb->debug_free_table);
+ for (subregion = 0 ; subregion < cb->subregion_count;
+ subregion++) {
+ cp = cb->subregion_tbl + subregion;
+
+ ufshpb_purge_active_page(hpb, cp, state);
+ }
+ hpb_dbg(hpb->hba, "region %d state %d dft %d\n",
+ region, state, hpb->debug_free_table);
+ }
+ spin_unlock(&hpb->hpb_lock);
+}
+
+static void ufshpb_retrieve_rsp_info(struct ufshpb_lu *hpb)
+{
+ struct ufshpb_rsp_info *rsp_info;
+ unsigned long flags;
+
+ while (1) {
+ spin_lock_irqsave(&hpb->rsp_list_lock, flags);
+ rsp_info = list_first_entry_or_null(&hpb->lh_rsp_info,
+ struct ufshpb_rsp_info, list_rsp_info);
+ if (!rsp_info) {
+ spin_unlock_irqrestore(&hpb->rsp_list_lock, flags);
+ break;
+ }
+ list_move_tail(&rsp_info->list_rsp_info,
+ &hpb->lh_rsp_info_free);
+ spin_unlock_irqrestore(&hpb->rsp_list_lock, flags);
+ hpb_dbg(hpb->hba, "add_list %p -> %p",
+ &hpb->lh_rsp_info_free, rsp_info);
+ }
+}
+
+static void ufshpb_probe(struct ufs_hba *hba)
+{
+ struct ufshpb_lu *hpb;
+ int lu;
+
+ for (lu = 0 ; lu < UFS_UPIU_MAX_GENERAL_LUN ; lu++) {
+ hpb = hba->ufshpb_lup[lu];
+
+ if (hpb && hpb->lu_hpb_enable) {
+ ufshpb_cancel_jobs(hpb);
+ ufshpb_retrieve_rsp_info(hpb);
+ ufshpb_purge_active_block(hpb);
+ dev_info(hba->dev, "UFSHPB lun %d reset\n", lu);
+ tasklet_init(&hpb->ufshpb_tasklet,
+ ufshpb_tasklet_fn, (unsigned long)hpb);
+ }
+ }
+ hba->ufshpb_state = HPB_PRESENT;
+}
+
+static void ufshpb_destroy_subregion_tbl(struct ufshpb_lu *hpb,
+ struct ufshpb_region *cb)
+{
+ int subregion;
+
+ for (subregion = 0 ; subregion < cb->subregion_count ; subregion++) {
+ struct ufshpb_subregion *cp;
+
+ cp = cb->subregion_tbl + subregion;
+ cp->subregion_state = HPBSUBREGION_UNUSED;
+ ufshpb_put_map_ctx(hpb, cp->mctx);
+ }
+
+ kfree(cb->subregion_tbl);
+}
+
+static void ufshpb_destroy_region_tbl(struct ufshpb_lu *hpb)
+{
+ int region;
+
+ if (!hpb)
+ return;
+
+ for (region = 0 ; region < hpb->regions_per_lu ; region++) {
+ struct ufshpb_region *cb;
+
+ cb = hpb->region_tbl + region;
+ if (cb->region_state == HPBREGION_PINNED ||
+ cb->region_state == HPBREGION_ACTIVE) {
+ cb->region_state = HPBREGION_INACTIVE;
+
+ ufshpb_destroy_subregion_tbl(hpb, cb);
+ }
+ }
+
+ ufshpb_table_mempool_remove(hpb);
+ kfree(hpb->region_tbl);
+}
+
+void ufshpb_release(struct ufs_hba *hba, int state)
+{
+ int lun;
+
+ hba->ufshpb_state = HPB_FAILED;
+
+ for (lun = 0 ; lun < UFS_UPIU_MAX_GENERAL_LUN ; lun++) {
+ struct ufshpb_lu *hpb = hba->ufshpb_lup[lun];
+
+ if (!hpb)
+ continue;
+
+ hba->ufshpb_lup[lun] = NULL;
+
+ if (!hpb->lu_hpb_enable)
+ continue;
+
+ hpb->lu_hpb_enable = false;
+
+ ufshpb_cancel_jobs(hpb);
+
+ ufshpb_destroy_region_tbl(hpb);
+
+ ufshpb_req_mempool_remove(hpb);
+
+ kobject_uevent(&hpb->kobj, KOBJ_REMOVE);
+ kobject_del(&hpb->kobj); // TODO --count & del?
+
+ kfree(hpb);
+ }
+
+ if (alloc_mctx != 0)
+ dev_warn(hba->dev, "warning: alloc_mctx %d", alloc_mctx);
+
+ hba->ufshpb_state = state;
+}
+
+void ufshpb_init_handler(struct work_struct *work)
+{
+ struct ufs_hba *hba;
+ struct delayed_work *dwork = to_delayed_work(work);
+#if defined(CONFIG_SCSI_SCAN_ASYNC)
+ unsigned long flags;
+#endif
+
+ hba = container_of(dwork, struct ufs_hba, ufshpb_init_work);
+
+ if (hba->ufshpb_state == HPB_NOT_SUPPORTED)
+ return;
+
+#if defined(CONFIG_SCSI_SCAN_ASYNC)
+ mutex_lock(&hba->host->scan_mutex);
+ spin_lock_irqsave(hba->host->host_lock, flags);
+ if (hba->host->async_scan == 1) {
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+ mutex_unlock(&hba->host->scan_mutex);
+ schedule_delayed_work(&hba->ufshpb_init_work,
+ msecs_to_jiffies(100));
+ return;
+ }
+ spin_unlock_irqrestore(hba->host->host_lock, flags);
+ mutex_unlock(&hba->host->scan_mutex);
+#endif
+ if (hba->ufshpb_state == HPB_NEED_INIT) {
+ int err = ufshpb_init(hba);
+
+ if (hba->ufshpb_state == HPB_NOT_SUPPORTED)
+ pr_info("UFSHPB: run without HPB - err=%d\n", err);
+ } else if (hba->ufshpb_state == HPB_RESET) {
+ ufshpb_probe(hba);
+ }
+}
+
+void ufshcd_init_hpb(struct ufs_hba *hba)
+{
+ int lun;
+
+ hba->ufshpb_feat = 0;
+ hba->ufshpb_state = HPB_NEED_INIT;
+ for (lun = 0 ; lun < UFS_UPIU_MAX_GENERAL_LUN ; lun++) {
+ hba->ufshpb_lup[lun] = NULL;
+ hba->sdev_ufs_lu[lun] = NULL;
+ }
+
+ INIT_DELAYED_WORK(&hba->ufshpb_init_work, ufshpb_init_handler);
+}
+
+static void ufshpb_error_handler(struct work_struct *work)
+{
+ struct ufs_hba *hba;
+
+ hba = container_of(work, struct ufs_hba, ufshpb_eh_work);
+
+ dev_warn(hba->dev, "UFSHPB driver runs without UFSHPB\n");
+ dev_warn(hba->dev, "UFSHPB will be removed from the kernel\n");
+
+ ufshpb_release(hba, HPB_FAILED);
+}
+
+static void ufshpb_stat_init(struct ufshpb_lu *hpb)
+{
+ atomic64_set(&hpb->hit, 0);
+ atomic64_set(&hpb->miss, 0);
+ atomic64_set(&hpb->region_miss, 0);
+ atomic64_set(&hpb->subregion_miss, 0);
+ atomic64_set(&hpb->entry_dirty_miss, 0);
+ atomic64_set(&hpb->map_req_cnt, 0);
+ atomic64_set(&hpb->region_evict, 0);
+ atomic64_set(&hpb->region_add, 0);
+ atomic64_set(&hpb->rb_fail, 0);
+}
+
+static ssize_t ufshpb_sysfs_info_lba_store(struct ufshpb_lu *hpb,
+ const char *buf, size_t count)
+{
+ unsigned long long ppn;
+ unsigned long value;
+ unsigned int lpn;
+ int region, subregion, subregion_offset;
+ struct ufshpb_region *cb;
+ struct ufshpb_subregion *cp;
+ int dirty;
+
+ if (kstrtoul(buf, 0, &value)) {
+ dev_err(HPB_DEV(hpb), "kstrtoul error\n");
+ return -EINVAL;
+ }
+
+ if (value > hpb->lu_num_blocks * SECTORS_PER_BLOCK) {
+ dev_err(HPB_DEV(hpb),
+ "value %lu > lu_num_blocks %d error\n",
+ value, hpb->lu_num_blocks);
+ return -EINVAL;
+ }
+ lpn = value / SECTORS_PER_BLOCK;
+
+ ufshpb_get_pos_from_lpn(hpb, lpn, ®ion, &subregion,
+ &subregion_offset);
+
+ cb = hpb->region_tbl + region;
+ cp = cb->subregion_tbl + subregion;
+
+ if (cb->region_state != HPBREGION_INACTIVE) {
+ ppn = ufshpb_get_ppn(cp->mctx, subregion_offset);
+ spin_lock_bh(&hpb->hpb_lock);
+ dirty = ufshpb_ppn_dirty_check(hpb, cp, subregion_offset);
+ spin_unlock_bh(&hpb->hpb_lock);
+ } else {
+ ppn = 0;
+ dirty = -1;
+ }
+
+ dev_info(HPB_DEV(hpb),
+ "sector %lu region %d state %d subregion %d state %d",
+ value, region, cb->region_state, subregion,
+ cp->subregion_state);
+ dev_info(HPB_DEV(hpb),
+ "sector %lu lpn %u ppn %llx dirty %d",
+ value, lpn, ppn, dirty);
+ return count;
+}
+
+static ssize_t ufshpb_sysfs_map_req_show(struct ufshpb_lu *hpb, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%lld %lld\n",
+ (long long)atomic64_read(&hpb->rb_noti_cnt),
+ (long long)atomic64_read(&hpb->map_req_cnt));
+}
+
+static ssize_t ufshpb_sysfs_count_reset_store(struct ufshpb_lu *hpb,
+ const char *buf, size_t count)
+{
+ unsigned long debug;
+
+ if (kstrtoul(buf, 0, &debug))
+ return -EINVAL;
+
+ ufshpb_stat_init(hpb);
+
+ return count;
+}
+
+static ssize_t ufshpb_sysfs_add_evict_show(struct ufshpb_lu *hpb, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%lld %lld\n",
+ (long long)atomic64_read(&hpb->region_add),
+ (long long)atomic64_read(&hpb->region_evict));
+}
+
+static ssize_t ufshpb_sysfs_hit_show(struct ufshpb_lu *hpb, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%lld\n",
+ (long long)atomic64_read(&hpb->hit));
+}
+
+static ssize_t ufshpb_sysfs_miss_show(struct ufshpb_lu *hpb, char *buf)
+{
+ long long region_miss, subregion_miss, entry_dirty_miss, rb_fail;
+
+ region_miss = atomic64_read(&hpb->region_miss);
+ subregion_miss = atomic64_read(&hpb->subregion_miss);
+ entry_dirty_miss = atomic64_read(&hpb->entry_dirty_miss);
+ rb_fail = atomic64_read(&hpb->rb_fail);
+
+ return snprintf(buf, PAGE_SIZE,
+ "%lld %lld %lld %lld %lld\n",
+ region_miss + subregion_miss + entry_dirty_miss,
+ region_miss, subregion_miss, entry_dirty_miss, rb_fail);
+}
+
+static ssize_t ufshpb_sysfs_version_show(struct ufshpb_lu *hpb, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE,
+ "%.2x.%.2x %.2x.%.2x\n",
+ GET_BYTE_1(hpb->hpb_ver), GET_BYTE_0(hpb->hpb_ver),
+ GET_BYTE_1(UFSHPB_DD_VER), GET_BYTE_0(UFSHPB_DD_VER));
+}
+
+static ssize_t ufshpb_sysfs_active_list_show(struct ufshpb_lu *hpb, char *buf)
+{
+ int ret = 0, count = 0;
+ struct ufshpb_region *region;
+
+ list_for_each_entry(region, &hpb->lru_info.lru, list_region) {
+ ret = snprintf(buf + count, PAGE_SIZE - count,
+ "%d %d ",
+ region->region, region->hit_count);
+ count += ret;
+ }
+ ret = snprintf(buf + count, PAGE_SIZE - count, "\n");
+ count += ret;
+
+ return count;
+}
+
+static ssize_t ufshpb_sysfs_active_block_status_show(struct ufshpb_lu *hpb,
+ char *buf)
+{
+ int ret = 0, count = 0, region;
+
+ for (region = 0 ; region < hpb->regions_per_lu ; region++) {
+ ret = snprintf(buf + count, PAGE_SIZE - count,
+ "%d:%d ",
+ region, hpb->region_tbl[region].region_state);
+ count += ret;
+ }
+
+ ret = snprintf(buf + count, PAGE_SIZE - count, "\n");
+ count += ret;
+
+ return count;
+}
+
+static ssize_t ufshpb_sysfs_map_loading_store(struct ufshpb_lu *hpb,
+ const char *buf, size_t count)
+{
+ unsigned long value;
+
+ if (kstrtoul(buf, 0, &value))
+ return -EINVAL;
+
+ if (value > 2)
+ return -EINVAL;
+
+ if (value == 1)
+ ufshpb_map_loading_trigger(hpb, false, false, false);
+ else if (value == 2)
+ ufshpb_map_loading_trigger(hpb, false, false, true);
+
+ return count;
+}
+
+static ssize_t ufshpb_sysfs_map_disable_show(struct ufshpb_lu *hpb, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%d\n", hpb->force_map_req_disable);
+}
+
+static ssize_t ufshpb_sysfs_map_disable_store(struct ufshpb_lu *hpb,
+ const char *buf, size_t count)
+{
+ unsigned long value;
+
+ if (kstrtoul(buf, 0, &value))
+ return -EINVAL;
+
+ if (value)
+ value = 1;
+
+ hpb->force_map_req_disable = value;
+ return count;
+}
+
+static ssize_t ufshpb_sysfs_disable_show(struct ufshpb_lu *hpb, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%d\n", hpb->force_disable);
+}
+
+static ssize_t ufshpb_sysfs_disable_store(struct ufshpb_lu *hpb,
+ const char *buf, size_t count)
+{
+ unsigned long value;
+
+ if (kstrtoul(buf, 0, &value))
+ return -EINVAL;
+
+ if (value)
+ value = 1;
+
+ hpb->force_disable = value;
+ return count;
+}
+
+static int global_region;
+
+static inline bool is_region_active(struct ufshpb_lu *hpb, int region)
+{
+ if (hpb->region_tbl[region].region_state == HPBREGION_ACTIVE ||
+ hpb->region_tbl[region].region_state ==
+ HPBREGION_PINNED)
+ return true;
+
+ return false;
+}
+
+static ssize_t ufshpb_sysfs_active_group_store(struct ufshpb_lu *hpb,
+ const char *buf, size_t count)
+{
+ unsigned long block;
+ int region;
+
+ if (kstrtoul(buf, 0, &block))
+ return -EINVAL;
+
+ region = block >> hpb->entries_per_region_shift;
+ if (region >= hpb->regions_per_lu) {
+ dev_err(HPB_DEV(hpb),
+ "error region %d max %d\n",
+ region, hpb->regions_per_lu);
+ region = hpb->regions_per_lu - 1;
+ }
+
+ global_region = region;
+
+ dev_info(HPB_DEV(hpb),
+ "block %lu region %d active %d",
+ block, region, is_region_active(hpb, region));
+
+ return count;
+}
+
+static ssize_t ufshpb_sysfs_active_group_show(struct ufshpb_lu *hpb, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE,
+ "%d\n", is_region_active(hpb, global_region));
+}
+
+static struct ufshpb_sysfs_entry ufshpb_sysfs_entries[] = {
+ __ATTR(is_active_group, 0644,
+ ufshpb_sysfs_active_group_show,
+ ufshpb_sysfs_active_group_store),
+ __ATTR(read_16_disable, 0644,
+ ufshpb_sysfs_disable_show, ufshpb_sysfs_disable_store),
+ __ATTR(map_cmd_disable, 0644,
+ ufshpb_sysfs_map_disable_show,
+ ufshpb_sysfs_map_disable_store),
+ __ATTR(map_loading, 0200, NULL, ufshpb_sysfs_map_loading_store),
+ __ATTR(active_block_status, 0444,
+ ufshpb_sysfs_active_block_status_show, NULL),
+ __ATTR(HPBVersion, 0444, ufshpb_sysfs_version_show, NULL),
+ __ATTR(hit_count, 0444, ufshpb_sysfs_hit_show, NULL),
+ __ATTR(miss_count, 0444, ufshpb_sysfs_miss_show, NULL),
+ __ATTR(active_list, 0444, ufshpb_sysfs_active_list_show, NULL),
+ __ATTR(add_evict_count, 0444, ufshpb_sysfs_add_evict_show, NULL),
+ __ATTR(count_reset, 0200, NULL, ufshpb_sysfs_count_reset_store),
+ __ATTR(map_req_count, 0444, ufshpb_sysfs_map_req_show, NULL),
+ __ATTR(get_info_from_lba, 0200, NULL, ufshpb_sysfs_info_lba_store),
+ __ATTR_NULL
+};
+
+static ssize_t ufshpb_attr_show(struct kobject *kobj,
+ struct attribute *attr, char *page)
+{
+ struct ufshpb_sysfs_entry *entry;
+ struct ufshpb_lu *hpb;
+ ssize_t error;
+
+ entry = container_of(attr,
+ struct ufshpb_sysfs_entry, attr);
+ hpb = container_of(kobj, struct ufshpb_lu, kobj);
+
+ if (!entry->show)
+ return -EIO;
+
+ mutex_lock(&hpb->sysfs_lock);
+ error = entry->show(hpb, page);
+ mutex_unlock(&hpb->sysfs_lock);
+ return error;
+}
+
+static ssize_t ufshpb_attr_store(struct kobject *kobj,
+ struct attribute *attr,
+ const char *page, size_t length)
+{
+ struct ufshpb_sysfs_entry *entry;
+ struct ufshpb_lu *hpb;
+ ssize_t error;
+
+ entry = container_of(attr, struct ufshpb_sysfs_entry, attr);
+ hpb = container_of(kobj, struct ufshpb_lu, kobj);
+
+ if (!entry->store)
+ return -EIO;
+
+ mutex_lock(&hpb->sysfs_lock);
+ error = entry->store(hpb, page, length);
+ mutex_unlock(&hpb->sysfs_lock);
+ return error;
+}
+
+static const struct sysfs_ops ufshpb_sysfs_ops = {
+ .show = ufshpb_attr_show,
+ .store = ufshpb_attr_store,
+};
+
+static struct kobj_type ufshpb_ktype = {
+ .sysfs_ops = &ufshpb_sysfs_ops,
+ .release = NULL,
+};
+
+static int ufshpb_create_sysfs(struct ufs_hba *hba,
+ struct ufshpb_lu *hpb)
+{
+ struct device *dev = hba->dev;
+ struct ufshpb_sysfs_entry *entry;
+ int err;
+
+ hpb->sysfs_entries = ufshpb_sysfs_entries;
+
+ ufshpb_stat_init(hpb);
+
+ kobject_init(&hpb->kobj, &ufshpb_ktype);
+ mutex_init(&hpb->sysfs_lock);
+
+ err = kobject_add(&hpb->kobj, kobject_get(&dev->kobj),
+ "ufshpb_lu%d", hpb->lun);
+ if (!err) {
+ for (entry = hpb->sysfs_entries;
+ entry->attr.name != NULL ; entry++) {
+ if (sysfs_create_file(&hpb->kobj, &entry->attr))
+ break;
+ }
+ kobject_uevent(&hpb->kobj, KOBJ_ADD);
+ }
+
+ return err;
+}
diff --git a/drivers/scsi/ufs/ufshpb.h b/drivers/scsi/ufs/ufshpb.h
new file mode 100644
index 0000000..7aad09b
--- /dev/null
+++ b/drivers/scsi/ufs/ufshpb.h
@@ -0,0 +1,414 @@
+/*
+ * Universal Flash Storage Host Performance Booster
+ *
+ * Copyright (C) 2017-2018 Samsung Electronics Co., Ltd.
+ * Copyright (C) 2018, Google, Inc.
+ *
+ * Authors:
+ * Yongmyung Lee <ymhungry.lee@samsung.com>
+ * Jinyoung Choi <j-young.choi@samsung.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ * See the COPYING file in the top-level directory or visit
+ * <http://www.gnu.org/licenses/gpl-2.0.html>
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * This program is provided "AS IS" and "WITH ALL FAULTS" and
+ * without warranty of any kind. You are solely responsible for
+ * determining the appropriateness of using and distributing
+ * the program and assume all risks associated with your exercise
+ * of rights with respect to the program, including but not limited
+ * to infringement of third party rights, the risks and costs of
+ * program errors, damage to or loss of data, programs or equipment,
+ * and unavailability or interruption of operations. Under no
+ * circumstances will the contributor of this Program be liable for
+ * any damages of any kind arising from your use or distribution of
+ * this program.
+ *
+ * The Linux Foundation chooses to take subject only to the GPLv2
+ * license terms, and distributes only under these terms.
+ */
+
+#ifndef _UFSHPB_H_
+#define _UFSHPB_H_
+
+#include <linux/spinlock.h>
+#include <linux/circ_buf.h>
+#include <linux/workqueue.h>
+
+/* Version info*/
+#define UFSHPB_VER 0x0103
+#define UFSHPB_DD_VER 0x0135
+
+/* Constant value*/
+#define SECTOR 512
+#define BLOCK 4096
+#define SECTORS_PER_BLOCK (BLOCK / SECTOR)
+#define BITS_PER_DWORD 32
+#define MAX_MAP_REQ 16
+#define MAX_ACTIVE_NUM 2
+#define MAX_INACTIVE_NUM 2
+
+#define HPB_ENTRY_SIZE 0x08
+#define OS_PAGE_SIZE 4096
+#define HPB_ENTREIS_PER_OS_PAGE (OS_PAGE_SIZE / HPB_ENTRY_SIZE)
+#define IOCTL_DEV_CTX_MAX_SIZE OS_PAGE_SIZE
+#define OS_PAGE_SHIFT 12
+
+/* Description */
+#define UFS_FEATURE_SUPPORT_HPB_BIT 0x80
+#define UFSHPB_QUERY_DESC_DEVICE_MAX_SIZE 0x48
+#define UFSHPB_QUERY_DESC_CONFIGURAION_MAX_SIZE 0xD0
+#define UFSHPB_QUERY_DESC_UNIT_MAX_SIZE 0x2C
+#define UFSHPB_QUERY_DESC_GEOMETRY_MAX_SIZE 0x50
+
+/* Configuration for HPB */
+#define UFSHPB_CONF_LU_ENABLE 0x00
+#define UFSHPB_CONF_ACTIVE_REGIONS 0x10
+#define UFSHPB_CONF_PINNED_START 0x12
+#define UFSHPB_CONF_PINNED_NUM 0x14
+
+/* Parameter Macros */
+#define HPB_DEV(h) ((h)->hba->dev)
+#define MAX_BVEC_SIZE 128
+
+/* Use for HPB activate */
+#define UFSHPB_CONFIG_LEN 0xd0
+
+enum ufshpb_lu_set {
+ LU_DISABLE = 0x00,
+ LU_ENABLE = 0x01,
+ LU_HPB_ENABLE = 0x02,
+ LU_SET_MAX,
+};
+
+struct ufshpb_config_desc {
+ unsigned char conf_dev_desc[16];
+ unsigned char unit[UFS_UPIU_MAX_GENERAL_LUN][24];
+};
+
+/* Response UPIU types */
+#define HPB_RSP_NONE 0x00
+#define HPB_RSP_REQ_REGION_UPDATE 0x01
+#define PER_ACTIVE_INFO_BYTES 4
+#define PER_INACTIVE_INFO_BYTES 2
+
+/* Vender defined OPCODE */
+#define UFSHPB_READ_BUFFER 0xF9
+
+#define DEV_DATA_SEG_LEN 0x14
+#define DEV_SENSE_SEG_LEN 0x12
+#define DEV_DES_TYPE 0x80
+#define DEV_ADDITIONAL_LEN 0x11
+
+/* BYTE SHIFT */
+#define ZERO_BYTE_SHIFT 0
+#define ONE_BYTE_SHIFT 8
+#define TWO_BYTE_SHIFT 16
+#define THREE_BYTE_SHIFT 24
+#define FOUR_BYTE_SHIFT 32
+#define FIVE_BYTE_SHIFT 40
+#define SIX_BYTE_SHIFT 48
+#define SEVEN_BYTE_SHIFT 56
+
+#define SHIFT_BYTE_0(num) ((num) << ZERO_BYTE_SHIFT)
+#define SHIFT_BYTE_1(num) ((num) << ONE_BYTE_SHIFT)
+#define SHIFT_BYTE_2(num) ((num) << TWO_BYTE_SHIFT)
+#define SHIFT_BYTE_3(num) ((num) << THREE_BYTE_SHIFT)
+#define SHIFT_BYTE_4(num) ((num) << FOUR_BYTE_SHIFT)
+#define SHIFT_BYTE_5(num) ((num) << FIVE_BYTE_SHIFT)
+#define SHIFT_BYTE_6(num) ((num) << SIX_BYTE_SHIFT)
+#define SHIFT_BYTE_7(num) ((num) << SEVEN_BYTE_SHIFT)
+
+#define GET_BYTE_0(num) (((num) >> ZERO_BYTE_SHIFT) & 0xff)
+#define GET_BYTE_1(num) (((num) >> ONE_BYTE_SHIFT) & 0xff)
+#define GET_BYTE_2(num) (((num) >> TWO_BYTE_SHIFT) & 0xff)
+#define GET_BYTE_3(num) (((num) >> THREE_BYTE_SHIFT) & 0xff)
+#define GET_BYTE_4(num) (((num) >> FOUR_BYTE_SHIFT) & 0xff)
+#define GET_BYTE_5(num) (((num) >> FIVE_BYTE_SHIFT) & 0xff)
+#define GET_BYTE_6(num) (((num) >> SIX_BYTE_SHIFT) & 0xff)
+#define GET_BYTE_7(num) (((num) >> SEVEN_BYTE_SHIFT) & 0xff)
+
+#define BE_BYTE(p, i) \
+ ((u16) SHIFT_BYTE_1(*((u8 *)(p) + (i))) | \
+ (u16) SHIFT_BYTE_0(*((u8 *)(p) + (i + 1))))
+#define REGION_UNIT_SIZE(bit_offset) (0x01 << (bit_offset))
+
+enum UFSHPB_STATE {
+ HPB_PRESENT = 1,
+ HPB_NOT_SUPPORTED = -1,
+ HPB_FAILED = -2,
+ HPB_NEED_INIT = 0,
+ HPB_RESET = -3,
+};
+
+enum HPBREGION_STATE {
+ HPBREGION_INACTIVE,
+ HPBREGION_ACTIVE,
+ HPBREGION_PINNED,
+};
+
+enum HPBSUBREGION_STATE {
+ HPBSUBREGION_UNUSED,
+ HPBSUBREGION_DIRTY,
+ HPBSUBREGION_CLEAN,
+ HPBSUBREGION_ISSUED,
+};
+
+struct ufshpb_func_desc {
+ /*** Device Descriptor ***/
+ /* 06h bNumberLU */
+ int lu_cnt;
+ /* 40h HPB Version */
+ u16 hpb_ver;
+
+ /*** Geometry Descriptor ***/
+ /* 48h bHPBRegionSize (UNIT: 512KB) */
+ u8 hpb_region_size;
+ /* 49h bHPBNumberLU */
+ u8 hpb_number_lu;
+ /* 4Ah bHPBSubRegionSize */
+ u8 hpb_subregion_size;
+ /* 4B:4Ch wDeviceMaxActiveHPBRegions */
+ u16 hpb_device_max_active_regions;
+};
+
+struct ufshpb_lu_desc {
+ /*** Unit Descriptor ****/
+ /* 03h bLUEnable */
+ int lu_enable;
+ /* 06h lu queue depth info*/
+ int lu_queue_depth;
+ /* 0Ah bLogicalBlockSize. default 0x0C = 4KB */
+ int lu_logblk_size;
+ /* 0Bh qLogicalBlockCount. same as the read_capacity ret val. */
+ u64 lu_logblk_cnt;
+
+ /* 23h:24h wLUMaxActiveHPBRegions */
+ u16 lu_max_active_hpb_regions;
+ /* 25h:26h wHPBPinnedRegionStartIdx */
+ u16 hpb_pinned_region_startidx;
+ /* 27h:28h wNumHPBPinnedRegions */
+ u16 lu_num_hpb_pinned_regions;
+
+
+ /* if 03h value is 02h, hpb_enable is set. */
+ bool lu_hpb_enable;
+
+ int lu_hpb_pinned_end_offset;
+};
+
+struct ufshpb_rsp_active_list {
+ u16 region[MAX_ACTIVE_NUM];
+ u16 subregion[MAX_ACTIVE_NUM];
+};
+
+struct ufshpb_rsp_inactive_list {
+ u16 region[MAX_INACTIVE_NUM];
+};
+
+struct ufshpb_rsp_update_entry {
+ unsigned int lpn;
+ unsigned long long ppn;
+};
+
+struct ufshpb_rsp_info {
+ int type;
+ int active_cnt;
+ int inactive_cnt;
+ struct ufshpb_rsp_active_list active_list;
+ struct ufshpb_rsp_inactive_list inactive_list;
+
+ __u64 RSP_start;
+ __u64 RSP_tasklet_enter;
+
+ struct list_head list_rsp_info;
+};
+
+struct ufshpb_rsp_field {
+ u8 sense_data_len[2];
+ u8 desc_type;
+ u8 additional_len;
+ u8 hpb_type;
+ u8 reserved;
+ u8 active_region_cnt;
+ u8 inactive_region_cnt;
+ u8 hpb_active_field[8];
+ u8 hpb_inactive_field[4];
+};
+
+struct ufshpb_map_ctx {
+ struct page **m_page;
+ unsigned int *ppn_dirty;
+
+ struct list_head list_table;
+};
+
+struct ufshpb_subregion {
+ struct ufshpb_map_ctx *mctx;
+ enum HPBSUBREGION_STATE subregion_state;
+ int region;
+ int subregion;
+
+ struct list_head list_subregion;
+};
+
+struct ufshpb_region {
+ struct ufshpb_subregion *subregion_tbl;
+ enum HPBREGION_STATE region_state;
+ int region;
+ int subregion_count;
+
+ /*below information is used by lru*/
+ struct list_head list_region;
+ int hit_count;
+};
+
+struct ufshpb_map_req {
+ struct ufshpb_lu *hpb;
+ struct ufshpb_map_ctx *mctx;
+ struct request req;
+ struct bio bio;
+ struct bio_vec bvec[MAX_BVEC_SIZE];
+ void (*end_io)(struct request *rq, int err);
+ void *end_io_data;
+ int region;
+ int subregion;
+ int lun;
+ int retry_cnt;
+
+ /* for debug : RSP Profiling */
+ __u64 RSP_start; // get the request from device
+ __u64 RSP_tasklet_enter1; // tesklet sched time
+ __u64 RSP_issue; // issue scsi cmd
+ __u64 RSP_endio;
+ __u64 RSP_tasklet_enter2;
+ __u64 RSP_end; // complete the request
+
+ char sense[SCSI_SENSE_BUFFERSIZE];
+
+ struct list_head list_map_req;
+};
+
+enum selection_type {
+ LRU = 1,
+ LFU = 2,
+};
+
+struct victim_select_info {
+ int selection_type;
+ struct list_head lru;
+ int max_lru_active_cnt; // supported hpb #region - pinned #region
+ atomic64_t active_cnt;
+};
+
+struct ufshpb_lu {
+ struct ufshpb_region *region_tbl;
+ struct ufshpb_rsp_info *rsp_info;
+ struct ufshpb_map_req *map_req;
+
+ struct list_head lh_map_ctx;
+ struct list_head lh_subregion_req;
+ struct list_head lh_rsp_info;
+
+ struct list_head lh_rsp_info_free;
+ struct list_head lh_map_req_free;
+ struct list_head lh_map_req_retry;
+ int debug_free_table;
+
+ bool lu_hpb_enable;
+
+ struct work_struct ufshpb_work;
+ struct delayed_work ufshpb_retry_work;
+ struct tasklet_struct ufshpb_tasklet;
+ struct bio_vec bvec[MAX_BVEC_SIZE];
+
+ int subregions_per_lu;
+ int regions_per_lu;
+ int subregion_mem_size;
+
+ /* for selecting victim */
+ struct victim_select_info lru_info;
+
+ int hpb_ver;
+ int lu_max_active_regions;
+
+ int entries_per_subregion;
+ int entries_per_subregion_shift;
+ int entries_per_subregion_mask;
+
+ int entries_per_region_shift;
+ int entries_per_region_mask;
+ int subregions_per_region;
+
+ int dwords_per_subregion;
+ unsigned long long subregion_unit_size;
+
+ int mpage_bytes;
+ int mpages_per_subregion;
+
+ /* for debug constant variables */
+ int lu_num_blocks;
+
+ int lun;
+
+ struct ufs_hba *hba;
+
+ spinlock_t hpb_lock;
+ spinlock_t rsp_list_lock;
+
+ struct kobject kobj;
+ struct mutex sysfs_lock;
+ struct ufshpb_sysfs_entry *sysfs_entries;
+
+ /* for debug */
+ bool force_disable;
+ bool force_map_req_disable;
+ bool read_buf_debug;
+ atomic64_t hit;
+ atomic64_t miss;
+ atomic64_t region_miss;
+ atomic64_t subregion_miss;
+ atomic64_t entry_dirty_miss;
+ atomic64_t rb_noti_cnt;
+ atomic64_t map_req_cnt;
+ atomic64_t region_add;
+ atomic64_t region_evict;
+ atomic64_t rb_fail;
+};
+
+struct ufshpb_sysfs_entry {
+ struct attribute attr;
+ ssize_t (*show)(struct ufshpb_lu *hpb, char *buf);
+ ssize_t (*store)(struct ufshpb_lu *hpb, const char *, size_t);
+};
+
+static inline void *kvzalloc(size_t size, gfp_t flags)
+{
+ void *ret;
+
+ ret = kzalloc(size, flags | __GFP_NOWARN);
+ if (!ret)
+ ret = __vmalloc(size, flags | __GFP_ZERO, PAGE_KERNEL);
+ return ret;
+}
+
+struct ufshcd_lrb;
+
+void ufshcd_init_hpb(struct ufs_hba *hba);
+void ufshpb_init_handler(struct work_struct *work);
+void ufshpb_prep_fn(struct ufs_hba *hba, struct ufshcd_lrb *lrbp);
+void ufshpb_rsp_upiu(struct ufs_hba *hba, struct ufshcd_lrb *lrbp);
+void ufshpb_release(struct ufs_hba *hba, int state);
+int ufshpb_issue_req_dev_ctx(struct ufshpb_lu *hpb, unsigned char *buf,
+ int buf_length);
+int ufshpb_control_validation(struct ufs_hba *hba,
+ struct ufshpb_config_desc *config);
+#endif /* End of Header */
diff --git a/drivers/scsi/vmw_pvscsi.c b/drivers/scsi/vmw_pvscsi.c
index 15ca09c..fcfbe2d 100644
--- a/drivers/scsi/vmw_pvscsi.c
+++ b/drivers/scsi/vmw_pvscsi.c
@@ -564,9 +564,14 @@ static void pvscsi_complete_request(struct pvscsi_adapter *adapter,
(btstat == BTSTAT_SUCCESS ||
btstat == BTSTAT_LINKED_COMMAND_COMPLETED ||
btstat == BTSTAT_LINKED_COMMAND_COMPLETED_WITH_FLAG)) {
- cmd->result = (DID_OK << 16) | sdstat;
- if (sdstat == SAM_STAT_CHECK_CONDITION && cmd->sense_buffer)
- cmd->result |= (DRIVER_SENSE << 24);
+ if (sdstat == SAM_STAT_COMMAND_TERMINATED) {
+ cmd->result = (DID_RESET << 16);
+ } else {
+ cmd->result = (DID_OK << 16) | sdstat;
+ if (sdstat == SAM_STAT_CHECK_CONDITION &&
+ cmd->sense_buffer)
+ cmd->result |= (DRIVER_SENSE << 24);
+ }
} else
switch (btstat) {
case BTSTAT_SUCCESS:
@@ -1228,8 +1233,6 @@ static void pvscsi_shutdown_intr(struct pvscsi_adapter *adapter)
static void pvscsi_release_resources(struct pvscsi_adapter *adapter)
{
- pvscsi_shutdown_intr(adapter);
-
if (adapter->workqueue)
destroy_workqueue(adapter->workqueue);
@@ -1558,6 +1561,7 @@ static int pvscsi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
out_reset_adapter:
ll_adapter_reset(adapter);
out_release_resources:
+ pvscsi_shutdown_intr(adapter);
pvscsi_release_resources(adapter);
scsi_host_put(host);
out_disable_device:
@@ -1566,6 +1570,7 @@ static int pvscsi_probe(struct pci_dev *pdev, const struct pci_device_id *id)
return error;
out_release_resources_and_disable:
+ pvscsi_shutdown_intr(adapter);
pvscsi_release_resources(adapter);
goto out_disable_device;
}
diff --git a/drivers/scsi/xen-scsifront.c b/drivers/scsi/xen-scsifront.c
index 9dc8687..e1b32ed 100644
--- a/drivers/scsi/xen-scsifront.c
+++ b/drivers/scsi/xen-scsifront.c
@@ -676,10 +676,17 @@ static int scsifront_dev_reset_handler(struct scsi_cmnd *sc)
static int scsifront_sdev_configure(struct scsi_device *sdev)
{
struct vscsifrnt_info *info = shost_priv(sdev->host);
+ int err;
- if (info && current == info->curr)
- xenbus_printf(XBT_NIL, info->dev->nodename,
+ if (info && current == info->curr) {
+ err = xenbus_printf(XBT_NIL, info->dev->nodename,
info->dev_state_path, "%d", XenbusStateConnected);
+ if (err) {
+ xenbus_dev_error(info->dev, err,
+ "%s: writing dev_state_path", __func__);
+ return err;
+ }
+ }
return 0;
}
@@ -687,10 +694,15 @@ static int scsifront_sdev_configure(struct scsi_device *sdev)
static void scsifront_sdev_destroy(struct scsi_device *sdev)
{
struct vscsifrnt_info *info = shost_priv(sdev->host);
+ int err;
- if (info && current == info->curr)
- xenbus_printf(XBT_NIL, info->dev->nodename,
+ if (info && current == info->curr) {
+ err = xenbus_printf(XBT_NIL, info->dev->nodename,
info->dev_state_path, "%d", XenbusStateClosed);
+ if (err)
+ xenbus_dev_error(info->dev, err,
+ "%s: writing dev_state_path", __func__);
+ }
}
static struct scsi_host_template scsifront_sht = {
@@ -1025,9 +1037,12 @@ static void scsifront_do_lun_hotplug(struct vscsifrnt_info *info, int op)
if (scsi_add_device(info->host, chn, tgt, lun)) {
dev_err(&dev->dev, "scsi_add_device\n");
- xenbus_printf(XBT_NIL, dev->nodename,
+ err = xenbus_printf(XBT_NIL, dev->nodename,
info->dev_state_path,
"%d", XenbusStateClosed);
+ if (err)
+ xenbus_dev_error(dev, err,
+ "%s: writing dev_state_path", __func__);
}
break;
case VSCSIFRONT_OP_DEL_LUN:
@@ -1041,10 +1056,14 @@ static void scsifront_do_lun_hotplug(struct vscsifrnt_info *info, int op)
}
break;
case VSCSIFRONT_OP_READD_LUN:
- if (device_state == XenbusStateConnected)
- xenbus_printf(XBT_NIL, dev->nodename,
+ if (device_state == XenbusStateConnected) {
+ err = xenbus_printf(XBT_NIL, dev->nodename,
info->dev_state_path,
"%d", XenbusStateConnected);
+ if (err)
+ xenbus_dev_error(dev, err,
+ "%s: writing dev_state_path", __func__);
+ }
break;
default:
break;
diff --git a/drivers/slimbus/slim-msm-ngd.c b/drivers/slimbus/slim-msm-ngd.c
index 26dfd3f..49a4691 100644
--- a/drivers/slimbus/slim-msm-ngd.c
+++ b/drivers/slimbus/slim-msm-ngd.c
@@ -110,7 +110,7 @@ static irqreturn_t ngd_slim_interrupt(int irq, void *d)
else
dev->err = -EIO;
- SLIM_WARN(dev, "NGD interrupt error:0x%x, err:%d\n", stat,
+ SLIM_DBG(dev, "NGD interrupt error:0x%x, err:%d\n", stat,
dev->err);
/* Guarantee that error interrupts are cleared */
mb();
@@ -722,7 +722,7 @@ static int ngd_xfer_msg(struct slim_controller *ctrl, struct slim_msg_txn *txn)
u32 conf, stat, rx_msgq, int_stat, int_en, int_clr;
void __iomem *ngd = dev->base + NGD_BASE(dev->ctrl.nr,
dev->ver);
- SLIM_WARN(dev, "TX failed :MC:0x%x,mt:0x%x, ret:%d, ver:%d\n",
+ SLIM_DBG(dev, "TX failed :MC:0x%x,mt:0x%x, ret:%d, ver:%d\n",
txn_mc, txn_mt, ret, dev->ver);
conf = readl_relaxed(ngd);
stat = readl_relaxed(ngd + NGD_STATUS);
@@ -731,9 +731,9 @@ static int ngd_xfer_msg(struct slim_controller *ctrl, struct slim_msg_txn *txn)
int_en = readl_relaxed(ngd + NGD_INT_EN);
int_clr = readl_relaxed(ngd + NGD_INT_CLR);
- SLIM_WARN(dev, "conf:0x%x,stat:0x%x,rxmsgq:0x%x\n",
+ SLIM_DBG(dev, "conf:0x%x,stat:0x%x,rxmsgq:0x%x\n",
conf, stat, rx_msgq);
- SLIM_ERR(dev, "int_stat:0x%x,int_en:0x%x,int_cll:0x%x\n",
+ SLIM_DBG(dev, "int_stat:0x%x,int_en:0x%x,int_cll:0x%x\n",
int_stat, int_en, int_clr);
}
diff --git a/drivers/slimbus/slim-msm.c b/drivers/slimbus/slim-msm.c
index 24a3ccf..82232d4 100644
--- a/drivers/slimbus/slim-msm.c
+++ b/drivers/slimbus/slim-msm.c
@@ -665,7 +665,7 @@ void msm_slim_tx_msg_return(struct msm_slim_ctrl *dev, int err)
(idx * (SLIM_MSGQ_BUF_LEN >> 2));
/* print the descriptor that resulted in error */
for (i = 0; i < (SLIM_MSGQ_BUF_LEN >> 2); i++)
- SLIM_WARN(dev, "err desc[%d]:0x%x", i, addr[i]);
+ SLIM_DBG(dev, "err desc[%d]:0x%x", i, addr[i]);
}
/* reclaim all packets that were delivered out of order */
if (idx != dev->tx_head)
diff --git a/drivers/slimbus/slimbus.c b/drivers/slimbus/slimbus.c
index 1140b33..d459582 100644
--- a/drivers/slimbus/slimbus.c
+++ b/drivers/slimbus/slimbus.c
@@ -1355,8 +1355,10 @@ int slim_config_mgrports(struct slim_device *sb, u32 *ph, int nports,
for (i = 0; i < nports; i++) {
u8 pn = SLIM_HDL_TO_PORT(ph[i]);
- if (ctrl->ports[pn].state == SLIM_P_CFG)
+ if (ctrl->ports[pn].state == SLIM_P_CFG) {
+ mutex_unlock(&ctrl->sched.m_reconf);
return -EISCONN;
+ }
ctrl->ports[pn].cfg = *cfg;
}
mutex_unlock(&ctrl->sched.m_reconf);
diff --git a/drivers/soc/fsl/qbman/qman.c b/drivers/soc/fsl/qbman/qman.c
index 2caacd9..2cc82ed 100644
--- a/drivers/soc/fsl/qbman/qman.c
+++ b/drivers/soc/fsl/qbman/qman.c
@@ -2713,6 +2713,9 @@ static int qman_alloc_range(struct gen_pool *p, u32 *result, u32 cnt)
{
unsigned long addr;
+ if (!p)
+ return -ENODEV;
+
addr = gen_pool_alloc(p, cnt);
if (!addr)
return -ENOMEM;
diff --git a/drivers/soc/fsl/qe/ucc.c b/drivers/soc/fsl/qe/ucc.c
index c646d87..681f7d4 100644
--- a/drivers/soc/fsl/qe/ucc.c
+++ b/drivers/soc/fsl/qe/ucc.c
@@ -626,7 +626,7 @@ static u32 ucc_get_tdm_sync_shift(enum comm_dir mode, u32 tdm_num)
{
u32 shift;
- shift = (mode == COMM_DIR_RX) ? RX_SYNC_SHIFT_BASE : RX_SYNC_SHIFT_BASE;
+ shift = (mode == COMM_DIR_RX) ? RX_SYNC_SHIFT_BASE : TX_SYNC_SHIFT_BASE;
shift -= tdm_num * 2;
return shift;
diff --git a/drivers/soc/qcom/Kconfig b/drivers/soc/qcom/Kconfig
index 3ecca59..aca30c8 100644
--- a/drivers/soc/qcom/Kconfig
+++ b/drivers/soc/qcom/Kconfig
@@ -887,3 +887,10 @@
This driver is used to set the floor of the min frequency of big cluster
to the user specified value when the cluster is not power collapsed. When
the cluster is power collpsed it resets the value to physical limits.
+
+config MODEMSMEM
+ bool "Modem SMEM driver"
+ default N
+ ---help---
+ Enable to pass device information and other data to shared memory for
+ the modem subsystem
diff --git a/drivers/soc/qcom/Makefile b/drivers/soc/qcom/Makefile
index 0b71121..7dcd0a2 100644
--- a/drivers/soc/qcom/Makefile
+++ b/drivers/soc/qcom/Makefile
@@ -107,3 +107,13 @@
obj-$(CONFIG_MSM_BAM_DMUX) += bam_dmux.o
obj-$(CONFIG_WCNSS_CORE) += wcnss/
obj-$(CONFIG_BIG_CLUSTER_MIN_FREQ_ADJUST) += big_cluster_min_freq_adjust.o
+obj-$(CONFIG_MODEMSMEM) += modemsmem/
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-asm-operand-widths \
+ -Wno-ignored-attributes \
+ -Wno-pointer-bool-conversion \
+ -Wno-enum-conversion \
+ -Wno-strlcpy-strlcat-size \
+ -Wno-gcc-compat \
diff --git a/drivers/soc/qcom/modemsmem/Makefile b/drivers/soc/qcom/modemsmem/Makefile
new file mode 100644
index 0000000..b7c435d
--- /dev/null
+++ b/drivers/soc/qcom/modemsmem/Makefile
@@ -0,0 +1 @@
+obj-$(CONFIG_MODEMSMEM) += modemsmem.o
diff --git a/drivers/soc/qcom/modemsmem/modemsmem.c b/drivers/soc/qcom/modemsmem/modemsmem.c
new file mode 100644
index 0000000..4b055c0
--- /dev/null
+++ b/drivers/soc/qcom/modemsmem/modemsmem.c
@@ -0,0 +1,188 @@
+/*
+ * Copyright (c) 2018 Google Inc.
+ *
+ * @file kernel/drivers/modemsmem/modemsmem.c
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/fs.h>
+#include <linux/of.h>
+#include <linux/ctype.h>
+#include <asm/io.h>
+#include <soc/qcom/smem.h>
+#include "modemsmem.h"
+#include <soc/qcom/socinfo.h>
+
+#define BOOTMODE_LENGTH 20
+#define FACTORY_STR "factory"
+#define FFBM00_STR "ffbm-00"
+#define FFBM01_STR "ffbm-01"
+
+static char bootmode[BOOTMODE_LENGTH];
+
+static int __init get_bootmode(char *str)
+{
+ if (str[0] != '\0')
+ strlcpy(bootmode, str, BOOTMODE_LENGTH);
+
+ return 1;
+}
+__setup("androidboot.mode=", get_bootmode);
+
+static ssize_t modem_smem_show(struct device *d,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct modem_smem_type *modem_smem;
+
+ modem_smem = smem_alloc(SMEM_ID_VENDOR0,
+ sizeof(struct modem_smem_type),
+ 0, SMEM_ANY_HOST_FLAG);
+ if (modem_smem) {
+ return snprintf(buf, PAGE_SIZE,
+ "version:0x%x\n"
+ "major_id:0x%x\n"
+ "minor_id:0x%x\n"
+ "subtype:0x%x\n"
+ "platform:0x%x\n"
+ "ftm:0x%x\n",
+ modem_smem->version,
+ modem_smem->major_id,
+ modem_smem->minor_id,
+ modem_smem->subtype,
+ modem_smem->platform,
+ modem_smem->ftm_magic);
+ } else
+ return snprintf(buf, PAGE_SIZE, "The modem smem is null\n");
+}
+
+static DEVICE_ATTR(modem_smem, 0664, modem_smem_show, NULL);
+static struct attribute *modem_smem_attributes[] = {
+ &dev_attr_modem_smem.attr,
+ NULL
+};
+
+static const struct attribute_group modem_smem_group = {
+ .name = "modemsmem",
+ .attrs = modem_smem_attributes,
+};
+
+static int modem_smem_probe(struct platform_device *pdev)
+{
+ int ret = -1;
+ struct device_node *np = NULL;
+ struct device *dev = NULL;
+ struct modem_smem_type *modem_smem;
+
+ pr_debug("[SMEM] %s: Enter probe\n", __func__);
+
+ if (!pdev) {
+ pr_err("[SMEM] %s: Invalid pdev\n", __func__);
+ return -ENODEV;
+ }
+
+ np = pdev->dev.of_node;
+ if (!np) {
+ pr_err("[SMEM] %s: Invalid DT node\n", __func__);
+ return -EINVAL;
+ }
+
+ dev = &pdev->dev;
+ if (dev == NULL) {
+ pr_err("[SMEM] %s: Invalid dev\n", __func__);
+ return -EINVAL;
+ }
+
+ /* Create sysfs */
+ platform_set_drvdata(pdev, dev);
+ ret = sysfs_create_group(&pdev->dev.kobj, &modem_smem_group);
+ if (ret)
+ dev_err(dev, "Failed to create sysfs\n");
+
+ /* Allocate with SMEM channel */
+ modem_smem = smem_alloc(SMEM_ID_VENDOR0,
+ sizeof(struct modem_smem_type),
+ 0, SMEM_ANY_HOST_FLAG);
+ if (!modem_smem) {
+ dev_err(dev, "smem alloc fail\n");
+ return -ENOMEM;
+ }
+
+ /* Initialize smem with zeros */
+ memset_io(modem_smem, 0, sizeof(*modem_smem));
+
+ /* Setup modem SMEM parameters */
+ modem_smem_set_u32(modem_smem, version, MODEM_SMEM_VERSION);
+
+ modem_smem_set_u32(modem_smem, major_id,
+ (socinfo_get_platform_version() >> 16) & 0xff);
+
+ modem_smem_set_u32(modem_smem, minor_id,
+ socinfo_get_platform_version() & 0x00ff);
+
+ modem_smem_set_u32(modem_smem, platform, socinfo_get_platform_type());
+
+ modem_smem_set_u32(modem_smem, subtype, socinfo_get_platform_subtype());
+
+ if ((!strncmp(bootmode, FACTORY_STR, sizeof(FACTORY_STR))) ||
+ (!strncmp(bootmode, FFBM00_STR, sizeof(FFBM00_STR))) ||
+ (!strncmp(bootmode, FFBM01_STR, sizeof(FFBM01_STR)))) {
+ modem_smem_set_u32(modem_smem, ftm_magic, MODEM_FTM_MAGIC);
+ dev_info(dev, "Set FTM mode due to %s\n", bootmode);
+ }
+
+ dev_dbg(dev, "End probe\n");
+ return 0;
+}
+
+static const struct of_device_id modem_smem_of_match[] = {
+ {.compatible = "modemsmem",}
+};
+MODULE_DEVICE_TABLE(of, modem_smem_of_match);
+
+static struct platform_driver modem_smem_driver = {
+ .probe = modem_smem_probe,
+ .driver = {
+ .name = "modemsmem",
+ .owner = THIS_MODULE,
+ .of_match_table = modem_smem_of_match,
+ }
+};
+
+static int __init modem_smem_init(void)
+{
+ int ret = -1;
+
+ pr_debug("[SMEM] %s: Enter\n", __func__);
+
+ ret = platform_driver_register(&modem_smem_driver);
+ if (ret < 0)
+ pr_err("[SMEM] %s: platform_driver register fail. ret:%d\n",
+ __func__, ret);
+
+ return ret;
+}
+
+static void __exit modem_smem_exit(void)
+{
+ platform_driver_unregister(&modem_smem_driver);
+}
+
+module_init(modem_smem_init);
+module_exit(modem_smem_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Modem SMEM Driver");
diff --git a/drivers/soc/qcom/modemsmem/modemsmem.h b/drivers/soc/qcom/modemsmem/modemsmem.h
new file mode 100644
index 0000000..a78203c
--- /dev/null
+++ b/drivers/soc/qcom/modemsmem/modemsmem.h
@@ -0,0 +1,49 @@
+/*
+ * Copyright (c) 2018 Google Inc.
+ *
+ * @file /kernel/drivers/modemsmem/modemsmem.h
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _MODEM_SMEM_H
+#define _MODEM_SMEM_H
+
+#include <linux/types.h>
+
+/* Modem smem driver version */
+#define MODEM_SMEM_VERSION 0x0
+#define MODEM_FTM_MAGIC 0x6846544D
+
+struct modem_smem_type {
+ uint32_t version;
+ uint32_t reserved_1;
+ uint32_t major_id;
+ uint32_t minor_id;
+ uint32_t subtype;
+ uint32_t platform;
+ uint32_t reserved_2;
+ uint32_t ftm_magic;
+};
+
+#define modem_smem_addr(smem, field) \
+ ({ \
+ volatile void __iomem *__p = (smem); \
+ __p += offsetof(typeof(*(smem)), field); \
+ __p; \
+ })
+
+#define modem_smem_set_u32(p, field, value) \
+ writel_relaxed((value), modem_smem_addr((p), field))
+
+#define modem_smem_copy(p, field, src) \
+ memcpy_toio(modem_smem_addr((p), field), (src), sizeof((p)->field))
+
+#endif /* end of _MODEM_SMEM_H */
diff --git a/drivers/soc/qcom/msm_bus/Makefile b/drivers/soc/qcom/msm_bus/Makefile
index c2ef70c..3dc6387 100644
--- a/drivers/soc/qcom/msm_bus/Makefile
+++ b/drivers/soc/qcom/msm_bus/Makefile
@@ -18,3 +18,7 @@
obj-$(CONFIG_DEBUG_BUS_VOTER) += msm_bus_dbg_voter.o
obj-$(CONFIG_DEBUG_FS) += msm_bus_dbg.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-self-assign \
diff --git a/drivers/soc/qcom/msm_bus/msm_bus_arb_adhoc.c b/drivers/soc/qcom/msm_bus/msm_bus_arb_adhoc.c
index b6104f0..cb215a2 100644
--- a/drivers/soc/qcom/msm_bus/msm_bus_arb_adhoc.c
+++ b/drivers/soc/qcom/msm_bus/msm_bus_arb_adhoc.c
@@ -25,6 +25,8 @@
#define NUM_LNODES 3
#define MAX_STR_CL 50
+#define DEBUG_REC_TRANSACTION 0
+
struct bus_search_type {
struct list_head link;
struct list_head node_list;
@@ -1254,7 +1256,8 @@ static int update_bw_adhoc(struct msm_bus_client_handle *cl, u64 ab, u64 ib)
if (!strcmp(test_cl, cl->name))
log_transaction = true;
- msm_bus_dbg_rec_transaction(cl, ab, ib);
+ if (DEBUG_REC_TRANSACTION)
+ msm_bus_dbg_rec_transaction(cl, ab, ib);
if ((cl->cur_act_ib == ib) && (cl->cur_act_ab == ab)) {
MSM_BUS_DBG("%s:no change in request", cl->name);
@@ -1315,7 +1318,9 @@ static int update_bw_context(struct msm_bus_client_handle *cl, u64 act_ab,
if (!slp_ab && !slp_ib)
cl->active_only = true;
- msm_bus_dbg_rec_transaction(cl, cl->cur_act_ab, cl->cur_dual_ib);
+ if (DEBUG_REC_TRANSACTION)
+ msm_bus_dbg_rec_transaction(cl, cl->cur_act_ab,
+ cl->cur_dual_ib);
ret = update_path(cl->mas_dev, cl->slv, act_ib, act_ab, slp_ib,
slp_ab, cl->cur_act_ab, cl->cur_act_ab,
cl->first_hop, cl->active_only);
diff --git a/drivers/soc/qcom/msm_bus/msm_bus_fabric_rpmh.c b/drivers/soc/qcom/msm_bus/msm_bus_fabric_rpmh.c
index 5a28e98..5d3bd75 100644
--- a/drivers/soc/qcom/msm_bus/msm_bus_fabric_rpmh.c
+++ b/drivers/soc/qcom/msm_bus/msm_bus_fabric_rpmh.c
@@ -78,6 +78,7 @@ ssize_t bw_show(struct device *dev, struct device_attribute *attr,
bus_node->lnode_list[i].lnode_ab[ACTIVE_CTX],
bus_node->lnode_list[i].lnode_ib[DUAL_CTX],
bus_node->lnode_list[i].lnode_ab[DUAL_CTX]);
+#if defined(CONFIG_TRACING) && defined(DEBUG)
trace_printk(
"[%d]:%s:Act_IB %llu Act_AB %llu Slp_IB %llu Slp_AB %llu\n",
i, bus_node->lnode_list[i].cl_name,
@@ -85,6 +86,7 @@ ssize_t bw_show(struct device *dev, struct device_attribute *attr,
bus_node->lnode_list[i].lnode_ab[ACTIVE_CTX],
bus_node->lnode_list[i].lnode_ib[DUAL_CTX],
bus_node->lnode_list[i].lnode_ab[DUAL_CTX]);
+#endif
}
off += scnprintf((buf + off), PAGE_SIZE,
"Max_Act_IB %llu Sum_Act_AB %llu Act_Util_fact %d Act_Vrail_comp %d\n",
@@ -98,6 +100,7 @@ ssize_t bw_show(struct device *dev, struct device_attribute *attr,
bus_node->node_bw[DUAL_CTX].sum_ab,
bus_node->node_bw[DUAL_CTX].util_used,
bus_node->node_bw[DUAL_CTX].vrail_used);
+#if defined(CONFIG_TRACING) && defined(DEBUG)
trace_printk(
"Max_Act_IB %llu Sum_Act_AB %llu Act_Util_fact %d Act_Vrail_comp %d\n",
bus_node->node_bw[ACTIVE_CTX].max_ib,
@@ -110,6 +113,7 @@ ssize_t bw_show(struct device *dev, struct device_attribute *attr,
bus_node->node_bw[DUAL_CTX].sum_ab,
bus_node->node_bw[DUAL_CTX].util_used,
bus_node->node_bw[DUAL_CTX].vrail_used);
+#endif
return off;
}
@@ -469,6 +473,23 @@ static int bcm_clist_add(struct msm_bus_node_device_type *cur_dev)
return ret;
}
+static void tcs_cmd_n_shrink(int *n)
+{
+ int i = 0, j = 0, sum = 0;
+
+ do {
+ if (sum + n[i] > MAX_RPMH_PAYLOAD) {
+ n[j] = sum;
+ sum = 0;
+ j++;
+ }
+ sum += n[i];
+ } while (n[i++]);
+
+ n[j] = sum;
+ n[j+1] = 0;
+}
+
static int bcm_query_list_add(struct msm_bus_node_device_type *cur_dev)
{
int ret = 0;
@@ -610,6 +631,10 @@ int msm_bus_commit_data(struct list_head *clist)
bcm_cnt = tcs_cmd_list_gen(n_active, n_wake, n_sleep, cmdlist_active,
cmdlist_wake, cmdlist_sleep, cur_bcm_clist);
+ tcs_cmd_n_shrink(n_active);
+ tcs_cmd_n_shrink(n_wake);
+ tcs_cmd_n_shrink(n_sleep);
+
ret = rpmh_invalidate(cur_mbox);
if (ret)
MSM_BUS_ERR("%s: Error invalidating mbox: %d\n",
diff --git a/drivers/soc/qcom/msm_smem.c b/drivers/soc/qcom/msm_smem.c
index 959aab9..79ba1e3 100644
--- a/drivers/soc/qcom/msm_smem.c
+++ b/drivers/soc/qcom/msm_smem.c
@@ -924,7 +924,7 @@ unsigned int smem_get_version(unsigned int idx)
int *version_array;
struct smem_shared *smem = smem_ram_base;
- if (idx > 32) {
+ if (idx >= 32) {
pr_err("%s: invalid idx:%d\n", __func__, idx);
return 0;
}
diff --git a/drivers/soc/qcom/msm_smp2p.c b/drivers/soc/qcom/msm_smp2p.c
index 35d836d..1cd4750 100644
--- a/drivers/soc/qcom/msm_smp2p.c
+++ b/drivers/soc/qcom/msm_smp2p.c
@@ -442,7 +442,7 @@ static bool smp2p_ssr_ack_needed(uint32_t rpid)
if (!out_list[rpid].feature_ssr_ack_enabled)
return false;
- ssr_done = SMP2P_GET_RESTART_DONE(in_list[rpid].smem_edge_in->flags);
+ ssr_done = SMP2P_GET_RESTART_DONE(in_list[rpid].smem_edge_in, flags);
if (ssr_done != out_list[rpid].restart_ack)
return true;
@@ -467,7 +467,7 @@ static void smp2p_do_ssr_ack(uint32_t rpid)
SMP2P_INFO("%s: ssr ack pid %d: %d -> %d\n", __func__, rpid,
out_list[rpid].restart_ack, ack);
out_list[rpid].restart_ack = ack;
- SMP2P_SET_RESTART_ACK(out_list[rpid].smem_edge_out->flags, ack);
+ SMP2P_SET_RESTART_ACK(out_list[rpid].smem_edge_out, flags, ack);
smp2p_send_interrupt(rpid);
}
@@ -495,7 +495,7 @@ static void smp2p_negotiation_complete_v1(struct smp2p_out_list_item *out_item)
{
uint32_t features;
- features = SMP2P_GET_FEATURES(out_item->smem_edge_out->feature_version);
+ features = SMP2P_GET_FEATURES(out_item->smem_edge_out, feature_version);
if (features & SMP2P_FEATURE_SSR_ACK)
out_item->feature_ssr_ack_enabled = true;
@@ -574,8 +574,8 @@ static int smp2p_out_create_v1(struct msm_smp2p_out *out_entry)
}
smp2p_h_ptr = p_list->smem_edge_out;
- entries_total = SMP2P_GET_ENT_TOTAL(smp2p_h_ptr->valid_total_ent);
- entries_valid = SMP2P_GET_ENT_VALID(smp2p_h_ptr->valid_total_ent);
+ entries_total = SMP2P_GET_ENT_TOTAL(smp2p_h_ptr, valid_total_ent);
+ entries_valid = SMP2P_GET_ENT_VALID(smp2p_h_ptr, valid_total_ent);
p_list->ops_ptr->find_entry(smp2p_h_ptr, entries_total,
out_entry->name, &state_entry_ptr, &empty_spot);
@@ -604,7 +604,7 @@ static int smp2p_out_create_v1(struct msm_smp2p_out *out_entry)
__func__, out_entry->name,
out_entry->remote_pid,
entries_valid, entries_total);
- SMP2P_SET_ENT_VALID(smp2p_h_ptr->valid_total_ent,
+ SMP2P_SET_ENT_VALID(smp2p_h_ptr, valid_total_ent,
entries_valid);
smp2p_send_interrupt(out_entry->remote_pid);
}
@@ -632,7 +632,7 @@ static int smp2p_out_read_v1(struct msm_smp2p_out *out_entry, uint32_t *data)
return -EINVAL;
smp2p_h_ptr = out_list[out_entry->remote_pid].smem_edge_out;
- remote_pid = SMP2P_GET_REMOTE_PID(smp2p_h_ptr->rem_loc_proc_id);
+ remote_pid = SMP2P_GET_REMOTE_PID(smp2p_h_ptr, rem_loc_proc_id);
if (remote_pid != out_entry->remote_pid)
return -EINVAL;
@@ -666,7 +666,7 @@ static int smp2p_out_write_v1(struct msm_smp2p_out *out_entry, uint32_t data)
return -EINVAL;
smp2p_h_ptr = out_list[out_entry->remote_pid].smem_edge_out;
- remote_pid = SMP2P_GET_REMOTE_PID(smp2p_h_ptr->rem_loc_proc_id);
+ remote_pid = SMP2P_GET_REMOTE_PID(smp2p_h_ptr, rem_loc_proc_id);
if (remote_pid != out_entry->remote_pid)
return -EINVAL;
@@ -705,7 +705,7 @@ static int smp2p_out_modify_v1(struct msm_smp2p_out *out_entry,
return -EINVAL;
smp2p_h_ptr = out_list[out_entry->remote_pid].smem_edge_out;
- remote_pid = SMP2P_GET_REMOTE_PID(smp2p_h_ptr->rem_loc_proc_id);
+ remote_pid = SMP2P_GET_REMOTE_PID(smp2p_h_ptr, rem_loc_proc_id);
if (remote_pid != out_entry->remote_pid)
return -EINVAL;
@@ -756,7 +756,7 @@ static struct smp2p_smem __iomem *smp2p_in_validate_size_v1(int remote_pid,
in_item = &in_list[remote_pid];
item_ptr = (struct smp2p_smem __iomem *)smem_item;
- total_entries = SMP2P_GET_ENT_TOTAL(item_ptr->valid_total_ent);
+ total_entries = SMP2P_GET_ENT_TOTAL(item_ptr, valid_total_ent);
if (total_entries > 0) {
in_item->safe_total_entries = total_entries;
in_item->item_size = size;
@@ -812,7 +812,7 @@ static void smp2p_negotiation_complete_v0(struct smp2p_out_list_item *out_item)
{
SMP2P_ERR("%s: invalid negotiation complete for v0 pid %d\n",
__func__,
- SMP2P_GET_REMOTE_PID(out_item->smem_edge_out->rem_loc_proc_id));
+ SMP2P_GET_REMOTE_PID(out_item->smem_edge_out, rem_loc_proc_id));
}
/**
@@ -990,16 +990,16 @@ void smp2p_init_header(struct smp2p_smem __iomem *header_ptr,
uint32_t features, uint32_t version)
{
header_ptr->magic = SMP2P_MAGIC;
- SMP2P_SET_LOCAL_PID(header_ptr->rem_loc_proc_id, local_pid);
- SMP2P_SET_REMOTE_PID(header_ptr->rem_loc_proc_id, remote_pid);
- SMP2P_SET_FEATURES(header_ptr->feature_version, features);
- SMP2P_SET_ENT_TOTAL(header_ptr->valid_total_ent, SMP2P_MAX_ENTRY);
- SMP2P_SET_ENT_VALID(header_ptr->valid_total_ent, 0);
+ SMP2P_SET_LOCAL_PID(header_ptr, rem_loc_proc_id, local_pid);
+ SMP2P_SET_REMOTE_PID(header_ptr, rem_loc_proc_id, remote_pid);
+ SMP2P_SET_FEATURES(header_ptr, feature_version, features);
+ SMP2P_SET_ENT_TOTAL(header_ptr, valid_total_ent, SMP2P_MAX_ENTRY);
+ SMP2P_SET_ENT_VALID(header_ptr, valid_total_ent, 0);
header_ptr->flags = 0;
/* ensure that all fields are valid before version is written */
wmb();
- SMP2P_SET_VERSION(header_ptr->feature_version, version);
+ SMP2P_SET_VERSION(header_ptr, feature_version, version);
}
/**
@@ -1049,8 +1049,8 @@ static int smp2p_do_negotiation(int remote_pid,
r_version = 0;
if (r_smem_ptr) {
- r_version = SMP2P_GET_VERSION(r_smem_ptr->feature_version);
- r_feature = SMP2P_GET_FEATURES(r_smem_ptr->feature_version);
+ r_version = SMP2P_GET_VERSION(r_smem_ptr, feature_version);
+ r_feature = SMP2P_GET_FEATURES(r_smem_ptr, feature_version);
}
if (r_version == 0) {
@@ -1084,7 +1084,7 @@ static int smp2p_do_negotiation(int remote_pid,
"%s: negotiation failure pid %d: RV %d RF %x\n",
__func__, remote_pid, r_version, r_feature
);
- SMP2P_SET_VERSION(l_smem_ptr->feature_version,
+ SMP2P_SET_VERSION(l_smem_ptr, feature_version,
SMP2P_EDGE_STATE_FAILED);
smp2p_send_interrupt(remote_pid);
out_item->smem_edge_state = SMP2P_EDGE_STATE_FAILED;
diff --git a/drivers/soc/qcom/peripheral-loader.c b/drivers/soc/qcom/peripheral-loader.c
index 8776379..cad6943 100644
--- a/drivers/soc/qcom/peripheral-loader.c
+++ b/drivers/soc/qcom/peripheral-loader.c
@@ -825,6 +825,7 @@ static int pil_load_seg(struct pil_desc *desc, struct pil_seg *seg)
.dev = desc->dev,
};
void *map_data = desc->map_data ? desc->map_data : &map_fw_info;
+ int pil_retry_count = 0;
if (seg->filesz) {
snprintf(fw_name, ARRAY_SIZE(fw_name), "%s.b%02d",
@@ -840,6 +841,15 @@ static int pil_load_seg(struct pil_desc *desc, struct pil_seg *seg)
firmware_buf, seg->filesz);
desc->unmap_fw_mem(firmware_buf, seg->filesz, map_data);
+ while ((ret == -EAGAIN) &&
+ (pil_retry_count < PERIPHERAL_LOADER_MAX_RETRY)) {
+ pil_err(desc, "Failed to locate blob %s, retry: %d\n",
+ fw_name, pil_retry_count++);
+ ret = request_firmware_into_buf(&fw, fw_name, desc->dev,
+ firmware_buf, seg->filesz);
+ desc->unmap_fw_mem(firmware_buf, seg->filesz, map_data);
+ }
+
if (ret) {
pil_err(desc, "Failed to locate blob %s or blob is too big(rc:%d)\n",
fw_name, ret);
@@ -959,6 +969,7 @@ int pil_boot(struct pil_desc *desc)
struct pil_priv *priv = desc->priv;
bool mem_protect = false;
bool hyp_assign = false;
+ int pil_retry_count = 0;
ret = pil_notify_aop(desc, "on");
if (ret < 0) {
@@ -975,6 +986,13 @@ int pil_boot(struct pil_desc *desc)
down_read(&pil_pm_rwsem);
snprintf(fw_name, sizeof(fw_name), "%s.mdt", desc->fw_name);
ret = request_firmware(&fw, fw_name, desc->dev);
+ while ((ret == -EAGAIN) &&
+ (pil_retry_count < PERIPHERAL_LOADER_MAX_RETRY)) {
+ pil_err(desc, "request_firmware:%s failed, retry: %d\n",
+ fw_name, pil_retry_count++);
+ ret = request_firmware(&fw, fw_name, desc->dev);
+ }
+
if (ret) {
pil_err(desc, "Failed to locate %s(rc:%d)\n", fw_name, ret);
goto out;
@@ -1050,6 +1068,18 @@ int pil_boot(struct pil_desc *desc)
if (ret)
pil_err(desc, "Failed to assign to linux, ret- %d\n",
ret);
+ else {
+ /*
+ * In design, modem ssr flag is used to
+ * distinguish between modem 1st boot-up and
+ * restart cases. For modem pil_boot retry
+ * cases, the memory is already assigned back to
+ * linux. There is no need to assgin memory to
+ * linux again. Clear modem ssr flag to prevent
+ * memory re-assignment in pil_boot retry cases.
+ */
+ desc->modem_ssr = false;
+ }
}
ret = pil_assign_mem_to_subsys_and_linux(desc,
priv->region_start,
diff --git a/drivers/soc/qcom/peripheral-loader.h b/drivers/soc/qcom/peripheral-loader.h
index 6ea6b2a..714d201 100644
--- a/drivers/soc/qcom/peripheral-loader.h
+++ b/drivers/soc/qcom/peripheral-loader.h
@@ -16,6 +16,8 @@
#include <linux/mailbox/qmp.h>
#include "minidump_private.h"
+#define PERIPHERAL_LOADER_MAX_RETRY (3)
+
struct device;
struct module;
struct pil_priv;
@@ -106,6 +108,8 @@ struct pil_reset_ops {
int (*shutdown)(struct pil_desc *pil);
};
+extern int pil_boot_retry_count;
+
#ifdef CONFIG_MSM_PIL
extern int pil_desc_init(struct pil_desc *desc);
extern int pil_boot(struct pil_desc *desc);
diff --git a/drivers/soc/qcom/pil-msa.c b/drivers/soc/qcom/pil-msa.c
index 2a46d0b..ebae830 100644
--- a/drivers/soc/qcom/pil-msa.c
+++ b/drivers/soc/qcom/pil-msa.c
@@ -670,12 +670,20 @@ int pil_mss_reset_load_mba(struct pil_desc *pil)
int ret;
const u8 *data;
struct device *dma_dev = md->mba_mem_dev_fixed ?: &md->mba_mem_dev;
+ int pil_retry_count = 0;
trace_pil_func(__func__);
if (drv->mba_dp_virt && md->mba_mem_dev_fixed)
goto mss_reset;
fw_name_p = drv->non_elf_image ? fw_name_legacy : fw_name;
ret = request_firmware(&fw, fw_name_p, pil->dev);
+ while ((ret == -EAGAIN) &&
+ (pil_retry_count < PERIPHERAL_LOADER_MAX_RETRY)) {
+ dev_err(pil->dev,"request_firmware:%s failed, retry: %d\n",
+ fw_name_p, pil_retry_count++);
+ ret = request_firmware(&fw, fw_name_p, pil->dev);
+ }
+
if (ret) {
dev_err(pil->dev, "Failed to locate %s (rc:%d)\n",
fw_name_p, ret);
diff --git a/drivers/soc/qcom/pil-q6v5-mss.c b/drivers/soc/qcom/pil-q6v5-mss.c
index 4ede3b6..a843ed4 100644
--- a/drivers/soc/qcom/pil-q6v5-mss.c
+++ b/drivers/soc/qcom/pil-q6v5-mss.c
@@ -36,16 +36,18 @@
#include "pil-msa.h"
#define PROXY_TIMEOUT_MS 10000
-#define MAX_SSR_REASON_LEN 256U
#define STOP_ACK_TIMEOUT_MS 1000
#define subsys_to_drv(d) container_of(d, struct modem_data, subsys_desc)
-static void log_modem_sfr(void)
+int pil_boot_retry_count;
+
+static void log_modem_sfr(struct modem_data *drv)
{
u32 size;
- char *smem_reason, reason[MAX_SSR_REASON_LEN];
+ char *smem_reason, *reason;
+ reason = drv->subsys_desc.last_crash_reason;
smem_reason = smem_get_entry_no_rlock(SMEM_SSR_REASON_MSS0, &size, 0,
SMEM_ANY_HOST_FLAG);
if (!smem_reason || !size) {
@@ -63,7 +65,7 @@ static void log_modem_sfr(void)
static void restart_modem(struct modem_data *drv)
{
- log_modem_sfr();
+ log_modem_sfr(drv);
drv->ignore_errors = true;
subsystem_restart_dev(drv->subsys);
}
@@ -124,6 +126,7 @@ static int modem_shutdown(const struct subsys_desc *subsys, bool force_stop)
static int modem_powerup(const struct subsys_desc *subsys)
{
struct modem_data *drv = subsys_to_drv(subsys);
+ int ret;
if (subsys->is_not_loadable)
return 0;
@@ -136,7 +139,13 @@ static int modem_powerup(const struct subsys_desc *subsys)
drv->subsys_desc.ramdump_disable = 0;
drv->ignore_errors = false;
drv->q6->desc.fw_name = subsys->fw_name;
- return pil_boot(&drv->q6->desc);
+ pil_boot_retry_count = 0;
+ while ((ret = pil_boot(&drv->q6->desc)) &&
+ pil_boot_retry_count <= PERIPHERAL_LOADER_MAX_RETRY)
+ pr_err("Modem powerup failed ret:%d, retry %d times\n",
+ ret, pil_boot_retry_count++);
+
+ return ret;
}
static void modem_crash_shutdown(const struct subsys_desc *subsys)
diff --git a/drivers/soc/qcom/qmi_interface.c b/drivers/soc/qcom/qmi_interface.c
index 0b952a4..7f29fdb 100644
--- a/drivers/soc/qcom/qmi_interface.c
+++ b/drivers/soc/qcom/qmi_interface.c
@@ -1974,8 +1974,10 @@ int qmi_svc_event_notifier_register(uint32_t service_id,
svc_info_arr = kmalloc_array(num_servers,
sizeof(*svc_info_arr),
GFP_KERNEL);
- if (!svc_info_arr)
- return -ENOMEM;
+ if (!svc_info_arr) {
+ ret = -ENOMEM;
+ goto out;
+ }
num_servers = msm_ipc_router_lookup_server_name(
&svc_name,
svc_info_arr,
@@ -1993,6 +1995,8 @@ int qmi_svc_event_notifier_register(uint32_t service_id,
spin_unlock_irqrestore(&temp->nb_lock, flags);
}
}
+
+out:
mutex_unlock(&temp->svc_addr_list_lock);
return ret;
diff --git a/drivers/soc/qcom/rpmh.c b/drivers/soc/qcom/rpmh.c
index 036990c..c783362 100644
--- a/drivers/soc/qcom/rpmh.c
+++ b/drivers/soc/qcom/rpmh.c
@@ -32,7 +32,7 @@
#define RPMH_MAX_MBOXES 2
#define RPMH_MAX_FAST_RES 32
#define RPMH_MAX_REQ_IN_BATCH 10
-#define RPMH_TIMEOUT msecs_to_jiffies(10000)
+#define RPMH_TIMEOUT msecs_to_jiffies(20000)
#define DEFINE_RPMH_MSG_ONSTACK(rc, s, q, c, name) \
struct rpmh_msg name = { \
diff --git a/drivers/soc/qcom/rpmh_master_stat.c b/drivers/soc/qcom/rpmh_master_stat.c
index d1752c7..16b93a9 100644
--- a/drivers/soc/qcom/rpmh_master_stat.c
+++ b/drivers/soc/qcom/rpmh_master_stat.c
@@ -163,6 +163,10 @@ static ssize_t msm_rpmh_master_stats_show(struct kobject *kobj,
static inline void msm_rpmh_apss_master_stats_update(
struct msm_rpmh_profile_unit *profile_unit)
{
+ if (apss_master_stats.last_entered ==
+ profile_unit[POWER_DOWN_END].value)
+ return;
+
apss_master_stats.counts++;
apss_master_stats.last_entered = profile_unit[POWER_DOWN_END].value;
apss_master_stats.last_exited = profile_unit[POWER_UP_START].value;
diff --git a/drivers/soc/qcom/scm.c b/drivers/soc/qcom/scm.c
index a64ddae..e6ab0d7 100644
--- a/drivers/soc/qcom/scm.c
+++ b/drivers/soc/qcom/scm.c
@@ -194,9 +194,9 @@ static int scm_remap_error(int err)
static u32 smc(u32 cmd_addr)
{
int context_id;
- register u32 r0 asm("r0") = 1;
- register u32 r1 asm("r1") = (uintptr_t)&context_id;
- register u32 r2 asm("r2") = cmd_addr;
+ register u32 r0 asm(R0_STR) = 1;
+ register u32 r1 asm(R1_STR) = (uintptr_t)&context_id;
+ register u32 r2 asm(R2_STR) = cmd_addr;
do {
asm volatile(
__asmeq("%0", R0_STR)
@@ -209,7 +209,7 @@ static u32 smc(u32 cmd_addr)
"smc #0\n"
: "=r" (r0)
: "r" (r0), "r" (r1), "r" (r2)
- : "r3");
+ : R3_STR);
} while (r0 == SCM_INTERRUPTED);
return r0;
@@ -382,13 +382,13 @@ int scm_call_noalloc(u32 svc_id, u32 cmd_id, const void *cmd_buf,
static int __scm_call_armv8_64(u64 x0, u64 x1, u64 x2, u64 x3, u64 x4, u64 x5,
u64 *ret1, u64 *ret2, u64 *ret3)
{
- register u64 r0 asm("r0") = x0;
- register u64 r1 asm("r1") = x1;
- register u64 r2 asm("r2") = x2;
- register u64 r3 asm("r3") = x3;
- register u64 r4 asm("r4") = x4;
- register u64 r5 asm("r5") = x5;
- register u64 r6 asm("r6") = 0;
+ register u64 r0 asm("x0") = x0;
+ register u64 r1 asm("x1") = x1;
+ register u64 r2 asm("x2") = x2;
+ register u64 r3 asm("x3") = x3;
+ register u64 r4 asm("x4") = x4;
+ register u64 r5 asm("x5") = x5;
+ register u64 r6 asm("x6") = 0;
do {
asm volatile(
@@ -431,13 +431,13 @@ static int __scm_call_armv8_64(u64 x0, u64 x1, u64 x2, u64 x3, u64 x4, u64 x5,
static int __scm_call_armv8_32(u32 w0, u32 w1, u32 w2, u32 w3, u32 w4, u32 w5,
u64 *ret1, u64 *ret2, u64 *ret3)
{
- register u32 r0 asm("r0") = w0;
- register u32 r1 asm("r1") = w1;
- register u32 r2 asm("r2") = w2;
- register u32 r3 asm("r3") = w3;
- register u32 r4 asm("r4") = w4;
- register u32 r5 asm("r5") = w5;
- register u32 r6 asm("r6") = 0;
+ register u32 r0 asm("w0") = w0;
+ register u32 r1 asm("w1") = w1;
+ register u32 r2 asm("w2") = w2;
+ register u32 r3 asm("w3") = w3;
+ register u32 r4 asm("w4") = w4;
+ register u32 r5 asm("w5") = w5;
+ register u32 r6 asm("w6") = 0;
do {
asm volatile(
@@ -843,9 +843,9 @@ EXPORT_SYMBOL(scm_call);
s32 scm_call_atomic1(u32 svc, u32 cmd, u32 arg1)
{
int context_id;
- register u32 r0 asm("r0") = SCM_ATOMIC(svc, cmd, 1);
- register u32 r1 asm("r1") = (uintptr_t)&context_id;
- register u32 r2 asm("r2") = arg1;
+ register u32 r0 asm(R0_STR) = SCM_ATOMIC(svc, cmd, 1);
+ register u32 r1 asm(R1_STR) = (uintptr_t)&context_id;
+ register u32 r2 asm(R2_STR) = arg1;
asm volatile(
__asmeq("%0", R0_STR)
@@ -858,7 +858,7 @@ s32 scm_call_atomic1(u32 svc, u32 cmd, u32 arg1)
"smc #0\n"
: "=r" (r0)
: "r" (r0), "r" (r1), "r" (r2)
- : "r3");
+ : R3_STR);
return r0;
}
EXPORT_SYMBOL(scm_call_atomic1);
@@ -876,9 +876,9 @@ EXPORT_SYMBOL(scm_call_atomic1);
s32 scm_call_atomic1_1(u32 svc, u32 cmd, u32 arg1, u32 *ret1)
{
int context_id;
- register u32 r0 asm("r0") = SCM_ATOMIC(svc, cmd, 1);
- register u32 r1 asm("r1") = (uintptr_t)&context_id;
- register u32 r2 asm("r2") = arg1;
+ register u32 r0 asm(R0_STR) = SCM_ATOMIC(svc, cmd, 1);
+ register u32 r1 asm(R1_STR) = (uintptr_t)&context_id;
+ register u32 r2 asm(R2_STR) = arg1;
asm volatile(
__asmeq("%0", R0_STR)
@@ -892,7 +892,7 @@ s32 scm_call_atomic1_1(u32 svc, u32 cmd, u32 arg1, u32 *ret1)
"smc #0\n"
: "=r" (r0), "=r" (r1)
: "r" (r0), "r" (r1), "r" (r2)
- : "r3");
+ : R3_STR);
if (ret1)
*ret1 = r1;
return r0;
@@ -912,10 +912,10 @@ EXPORT_SYMBOL(scm_call_atomic1_1);
s32 scm_call_atomic2(u32 svc, u32 cmd, u32 arg1, u32 arg2)
{
int context_id;
- register u32 r0 asm("r0") = SCM_ATOMIC(svc, cmd, 2);
- register u32 r1 asm("r1") = (uintptr_t)&context_id;
- register u32 r2 asm("r2") = arg1;
- register u32 r3 asm("r3") = arg2;
+ register u32 r0 asm(R0_STR) = SCM_ATOMIC(svc, cmd, 2);
+ register u32 r1 asm(R1_STR) = (uintptr_t)&context_id;
+ register u32 r2 asm(R2_STR) = arg1;
+ register u32 r3 asm(R3_STR) = arg2;
asm volatile(
__asmeq("%0", R0_STR)
@@ -947,11 +947,11 @@ EXPORT_SYMBOL(scm_call_atomic2);
s32 scm_call_atomic3(u32 svc, u32 cmd, u32 arg1, u32 arg2, u32 arg3)
{
int context_id;
- register u32 r0 asm("r0") = SCM_ATOMIC(svc, cmd, 3);
- register u32 r1 asm("r1") = (uintptr_t)&context_id;
- register u32 r2 asm("r2") = arg1;
- register u32 r3 asm("r3") = arg2;
- register u32 r4 asm("r4") = arg3;
+ register u32 r0 asm(R0_STR) = SCM_ATOMIC(svc, cmd, 3);
+ register u32 r1 asm(R1_STR) = (uintptr_t)&context_id;
+ register u32 r2 asm(R2_STR) = arg1;
+ register u32 r3 asm(R3_STR) = arg2;
+ register u32 r4 asm(R4_STR) = arg3;
asm volatile(
__asmeq("%0", R0_STR)
@@ -975,12 +975,12 @@ s32 scm_call_atomic4_3(u32 svc, u32 cmd, u32 arg1, u32 arg2,
{
int ret;
int context_id;
- register u32 r0 asm("r0") = SCM_ATOMIC(svc, cmd, 4);
- register u32 r1 asm("r1") = (uintptr_t)&context_id;
- register u32 r2 asm("r2") = arg1;
- register u32 r3 asm("r3") = arg2;
- register u32 r4 asm("r4") = arg3;
- register u32 r5 asm("r5") = arg4;
+ register u32 r0 asm(R0_STR) = SCM_ATOMIC(svc, cmd, 4);
+ register u32 r1 asm(R1_STR) = (uintptr_t)&context_id;
+ register u32 r2 asm(R2_STR) = arg1;
+ register u32 r3 asm(R3_STR) = arg2;
+ register u32 r4 asm(R4_STR) = arg3;
+ register u32 r5 asm(R5_STR) = arg4;
asm volatile(
__asmeq("%0", R0_STR)
@@ -1026,13 +1026,13 @@ s32 scm_call_atomic5_3(u32 svc, u32 cmd, u32 arg1, u32 arg2,
{
int ret;
int context_id;
- register u32 r0 asm("r0") = SCM_ATOMIC(svc, cmd, 5);
- register u32 r1 asm("r1") = (uintptr_t)&context_id;
- register u32 r2 asm("r2") = arg1;
- register u32 r3 asm("r3") = arg2;
- register u32 r4 asm("r4") = arg3;
- register u32 r5 asm("r5") = arg4;
- register u32 r6 asm("r6") = arg5;
+ register u32 r0 asm(R0_STR) = SCM_ATOMIC(svc, cmd, 5);
+ register u32 r1 asm(R1_STR) = (uintptr_t)&context_id;
+ register u32 r2 asm(R2_STR) = arg1;
+ register u32 r3 asm(R3_STR) = arg2;
+ register u32 r4 asm(R4_STR) = arg3;
+ register u32 r5 asm(R5_STR) = arg4;
+ register u32 r6 asm(R6_STR) = arg5;
asm volatile(
__asmeq("%0", R0_STR)
@@ -1066,8 +1066,8 @@ u32 scm_get_version(void)
{
int context_id;
static u32 version = -1;
- register u32 r0 asm("r0");
- register u32 r1 asm("r1");
+ register u32 r0 asm(R0_STR);
+ register u32 r1 asm(R1_STR);
if (version != -1)
return version;
@@ -1088,7 +1088,7 @@ u32 scm_get_version(void)
"smc #0\n"
: "=r" (r0), "=r" (r1)
: "r" (r0), "r" (r1)
- : "r2", "r3");
+ : R2_STR, R3_STR);
} while (r0 == SCM_INTERRUPTED);
version = r1;
diff --git a/drivers/soc/qcom/service-locator.c b/drivers/soc/qcom/service-locator.c
index 5abc093..7b50300 100644
--- a/drivers/soc/qcom/service-locator.c
+++ b/drivers/soc/qcom/service-locator.c
@@ -24,6 +24,7 @@
#include <linux/device.h>
#include <linux/delay.h>
#include <linux/workqueue.h>
+#include <linux/debugfs.h>
#include <soc/qcom/msm_qmi_interface.h>
#include <soc/qcom/service-locator.h>
@@ -458,3 +459,140 @@ int find_subsys(const char *pd_path, char *subsys)
return 0;
}
EXPORT_SYMBOL(find_subsys);
+
+static struct pd_qmi_client_data test_data;
+
+static int servloc_test_pdr_cb(struct notifier_block *this,
+ unsigned long opcode, void *ptr)
+{
+ int i, rc = 0;
+ char subsys[QMI_SERVREG_LOC_NAME_LENGTH_V01];
+ struct pd_qmi_client_data *return_data;
+
+ return_data = (struct pd_qmi_client_data *)ptr;
+
+ if (opcode) {
+ pr_err("%s: Failed to get process domain!, opcode = %lu\n",
+ __func__, opcode);
+ return -EIO;
+ }
+
+ pr_err("Service Name: %s\tTotal Domains: %d\n",
+ return_data->service_name, return_data->total_domains);
+
+ for (i = 0; i < return_data->total_domains; i++) {
+ pr_err("Instance ID: %d\t ",
+ return_data->domain_list[i].instance_id);
+ pr_err("Domain Name: %s\n",
+ return_data->domain_list[i].name);
+ rc = find_subsys(return_data->domain_list[i].name,
+ subsys);
+ if (rc < 0)
+ pr_err("No valid subsys found for %s!\n",
+ return_data->domain_list[i].name);
+ else
+ pr_err("Subsys: %s\n", subsys);
+ }
+ return 0;
+}
+
+static struct notifier_block pdr_service_nb = {
+ .notifier_call = servloc_test_pdr_cb,
+};
+
+static ssize_t servloc_read(struct file *filp, char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ int rc = 0;
+ char *node_name = filp->private_data;
+
+ if (!strcmp(node_name, "test_servloc_get"))
+ rc = get_service_location(test_data.client_name,
+ test_data.service_name, &pdr_service_nb);
+
+ return rc;
+}
+
+static ssize_t servloc_write(struct file *fp, const char __user *buf,
+ size_t count, loff_t *unused)
+{
+ char *node_name = fp->private_data;
+
+ if (!buf)
+ return -EIO;
+ if (!strcmp(node_name, "service_name")) {
+ snprintf(test_data.service_name, sizeof(test_data.service_name),
+ "%.*s", (int) min((size_t)count - 1,
+ (sizeof(test_data.service_name) - 1)), buf);
+ } else {
+ snprintf(test_data.client_name, sizeof(test_data.client_name),
+ "%.*s", (int) min((size_t)count - 1,
+ (sizeof(test_data.client_name) - 1)), buf);
+ }
+ return count;
+}
+
+static const struct file_operations servloc_fops = {
+ .open = simple_open,
+ .read = servloc_read,
+ .write = servloc_write,
+};
+
+static struct dentry *servloc_base_dir;
+static struct dentry *test_servloc_file;
+
+static int __init servloc_debugfs_init(void)
+{
+ servloc_base_dir = debugfs_create_dir("test_servloc", NULL);
+ return !servloc_base_dir ? -ENOMEM : 0;
+}
+
+static void servloc_debugfs_exit(void)
+{
+ debugfs_remove_recursive(servloc_base_dir);
+}
+
+static int servloc_debugfs_add(void)
+{
+ int rc;
+
+ if (!servloc_base_dir)
+ return -ENOMEM;
+
+ test_servloc_file = debugfs_create_file("client_name",
+ 0644, servloc_base_dir,
+ "client_name", &servloc_fops);
+ rc = !test_servloc_file ? -ENOMEM : 0;
+
+ if (rc == 0) {
+ test_servloc_file = debugfs_create_file("service_name",
+ 0644, servloc_base_dir,
+ "service_name", &servloc_fops);
+ rc = !test_servloc_file ? -ENOMEM : 0;
+ }
+
+ if (rc == 0) {
+ test_servloc_file = debugfs_create_file("test_servloc_get",
+ 0644, servloc_base_dir,
+ "test_servloc_get", &servloc_fops);
+ rc = !test_servloc_file ? -ENOMEM : 0;
+ }
+ return rc;
+}
+
+static int __init service_locator_init(void)
+{
+ pr_debug("service_locator_status = %d\n", locator_status);
+ if (servloc_debugfs_init())
+ pr_err("Could not create test_servloc base directory!");
+ if (servloc_debugfs_add())
+ pr_err("Could not create test_servloc node entries!");
+ return 0;
+}
+
+static void __exit service_locator_exit(void)
+{
+ servloc_debugfs_exit();
+}
+module_init(service_locator_init);
+module_exit(service_locator_exit);
diff --git a/drivers/soc/qcom/service-notifier.c b/drivers/soc/qcom/service-notifier.c
index dc45b20..0f5cbfd 100644
--- a/drivers/soc/qcom/service-notifier.c
+++ b/drivers/soc/qcom/service-notifier.c
@@ -21,6 +21,7 @@
#include <linux/slab.h>
#include <linux/of.h>
#include <linux/err.h>
+#include <linux/debugfs.h>
#include <linux/uaccess.h>
#include <soc/qcom/subsystem_restart.h>
@@ -756,3 +757,179 @@ int service_notif_unregister_notifier(void *service_notif_handle,
&service_notif->service_notif_rcvr_list, nb);
}
EXPORT_SYMBOL(service_notif_unregister_notifier);
+
+struct service_notifier_test_data {
+ char service_path[MAX_STRING_LEN];
+ int instance_id;
+ struct notifier_block nb;
+ void *service_notif_handle;
+};
+
+static struct service_notifier_test_data test_data;
+
+static void print_service_provider_state(int notification, char *type)
+{
+ if (notification == SERVREG_NOTIF_SERVICE_STATE_DOWN_V01)
+ pr_info("%s: Service %s down!\n", type, test_data.service_path);
+ else if (notification == SERVREG_NOTIF_SERVICE_STATE_UP_V01)
+ pr_info("%s: Service %s up!\n", type, test_data.service_path);
+ else if (notification == SERVREG_NOTIF_SERVICE_STATE_UNINIT_V01)
+ pr_info("%s: Service %s state uninit!\n", type,
+ test_data.service_path);
+ else
+ pr_info("%s: Service %s state Unknown 0x%x!\n", type,
+ test_data.service_path, notification);
+}
+
+static int nb_callback(struct notifier_block *nb,
+ unsigned long notification,
+ void *data)
+{
+ print_service_provider_state((int)notification, "Notification:");
+ return 0;
+}
+
+static ssize_t show_service_path(struct seq_file *f, void *unused)
+{
+ if (test_data.service_notif_handle)
+ seq_printf(f, "Service Path: %s\n", test_data.service_path);
+ else
+ seq_puts(f, "No existing notifier\n");
+ return 0;
+}
+
+
+static ssize_t set_service_notifier_register(struct file *fp,
+ const char __user *buf,
+ size_t count, loff_t *ppos)
+{
+ int curr_state = INT_MAX, rc;
+
+ if (!buf)
+ return -EIO;
+ if (test_data.service_notif_handle) {
+ service_notif_unregister_notifier(
+ test_data.service_notif_handle,
+ &test_data.nb);
+ test_data.service_notif_handle = NULL;
+ pr_info("Unregistering existing notifier for %s\n",
+ test_data.service_path);
+ }
+ rc = simple_write_to_buffer(test_data.service_path, MAX_STRING_LEN,
+ ppos, buf, count - 1);
+ if (rc != count - 1) {
+ pr_err("Unable to read data into kernel buffer\n");
+ goto err;
+ }
+ test_data.nb.notifier_call = nb_callback;
+ test_data.service_notif_handle = service_notif_register_notifier(
+ test_data.service_path,
+ test_data.instance_id, &test_data.nb,
+ &curr_state);
+ if (!IS_ERR(test_data.service_notif_handle)) {
+ pr_info("Notifier Registered for service %s\n",
+ test_data.service_path);
+ print_service_provider_state(curr_state, "Initial State");
+ return count;
+ }
+err:
+ test_data.service_notif_handle = NULL;
+ pr_err("Unable to register notifier for %s\n", test_data.service_path);
+ return -EIO;
+}
+
+static int open_service_notifier_register(struct inode *inode, struct file *f)
+{
+ return single_open(f, (void *) show_service_path,
+ inode->i_private);
+}
+
+static const struct file_operations service_notifier_register_fops = {
+ .open = open_service_notifier_register,
+ .read = seq_read,
+ .write = set_service_notifier_register,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+
+static ssize_t show_service_notifier_id(struct seq_file *f, void *unused)
+{
+ seq_printf(f, "Service instance ID: %d\n", test_data.instance_id);
+ return 0;
+}
+
+static ssize_t set_service_notifier_id(struct file *fp,
+ const char __user *buf,
+ size_t count, loff_t *unused)
+{
+ int val, rc;
+ char kbuf[MAX_STRING_LEN];
+
+ if (count > MAX_STRING_LEN) {
+ rc = -EIO;
+ goto err;
+ }
+ rc = copy_from_user(kbuf, buf, count);
+ if (rc != 0) {
+ rc = -EFAULT;
+ goto err;
+ }
+
+ kbuf[count - 1] = '\0';
+ rc = kstrtoint(kbuf, 0, &val);
+ if (rc < 0)
+ goto err;
+
+ test_data.instance_id = val;
+ return count;
+err:
+ pr_err("Invalid input parameters: rc = %d\n", rc);
+ return rc;
+}
+
+static int open_service_notifier_id(struct inode *inode, struct file *f)
+{
+ return single_open(f, (void *) show_service_notifier_id,
+ inode->i_private);
+}
+
+static const struct file_operations service_notifier_id_fops = {
+ .open = open_service_notifier_id,
+ .read = seq_read,
+ .write = set_service_notifier_id,
+ .llseek = seq_lseek,
+ .release = seq_release,
+};
+
+static struct dentry *service_notifier_dir;
+static struct dentry *service_path_file;
+static struct dentry *service_id_file;
+
+static int __init service_notifier_init(void)
+{
+ service_notifier_dir = debugfs_create_dir("service_notifier", NULL);
+ if (service_notifier_dir) {
+ service_path_file = debugfs_create_file("service_path",
+ 0644, service_notifier_dir, NULL,
+ &service_notifier_register_fops);
+ if (!service_path_file)
+ goto err;
+ service_id_file = debugfs_create_file("service_id",
+ 0644, service_notifier_dir, NULL,
+ &service_notifier_id_fops);
+ if (!service_id_file)
+ goto err;
+ }
+ return 0;
+err:
+ debugfs_remove_recursive(service_notifier_dir);
+ return 0;
+}
+
+static void __exit service_notifier_exit(void)
+{
+ debugfs_remove_recursive(service_notifier_dir);
+ test_data.nb.notifier_call = nb_callback;
+}
+module_init(service_notifier_init);
+module_exit(service_notifier_exit);
diff --git a/drivers/soc/qcom/smp2p_debug.c b/drivers/soc/qcom/smp2p_debug.c
index 8d98d07..ffa7d4de 100644
--- a/drivers/soc/qcom/smp2p_debug.c
+++ b/drivers/soc/qcom/smp2p_debug.c
@@ -89,8 +89,8 @@ static int smp2p_item_header1(char *buf, int max, struct smp2p_smem *item_ptr,
i += scnprintf(buf + i, max - i,
"%-14s LPID %d RPID %d",
state_text,
- SMP2P_GET_LOCAL_PID(item_ptr->rem_loc_proc_id),
- SMP2P_GET_REMOTE_PID(item_ptr->rem_loc_proc_id)
+ SMP2P_GET_LOCAL_PID(item_ptr, rem_loc_proc_id),
+ SMP2P_GET_REMOTE_PID(item_ptr, rem_loc_proc_id)
);
return i;
@@ -115,8 +115,8 @@ static int smp2p_item_header2(char *buf, int max, struct smp2p_smem *item_ptr)
i += scnprintf(buf + i, max - i,
"Version: %08x Features: %08x",
- SMP2P_GET_VERSION(item_ptr->feature_version),
- SMP2P_GET_FEATURES(item_ptr->feature_version)
+ SMP2P_GET_VERSION(item_ptr, feature_version),
+ SMP2P_GET_FEATURES(item_ptr, feature_version)
);
return i;
@@ -142,8 +142,8 @@ static int smp2p_item_header3(char *buf, int max, struct smp2p_smem *item_ptr)
i += scnprintf(buf + i, max - i,
"Entries #/Max: %d/%d Flags: %c%c",
- SMP2P_GET_ENT_VALID(item_ptr->valid_total_ent),
- SMP2P_GET_ENT_TOTAL(item_ptr->valid_total_ent),
+ SMP2P_GET_ENT_VALID(item_ptr, valid_total_ent),
+ SMP2P_GET_ENT_TOTAL(item_ptr, valid_total_ent),
item_ptr->flags & SMP2P_FLAGS_RESTART_ACK_MASK ? 'A' : 'a',
item_ptr->flags & SMP2P_FLAGS_RESTART_DONE_MASK ? 'D' : 'd'
);
@@ -230,13 +230,13 @@ static void smp2p_item(struct seq_file *s, int remote_pid)
if (out_ptr) {
out_entries = (struct smp2p_entry_v1 *)((void *)out_ptr +
sizeof(struct smp2p_smem));
- out_valid = SMP2P_GET_ENT_VALID(out_ptr->valid_total_ent);
+ out_valid = SMP2P_GET_ENT_VALID(out_ptr, valid_total_ent);
}
if (in_ptr) {
in_entries = (struct smp2p_entry_v1 *)((void *)in_ptr +
sizeof(struct smp2p_smem));
- in_valid = SMP2P_GET_ENT_VALID(in_ptr->valid_total_ent);
+ in_valid = SMP2P_GET_ENT_VALID(in_ptr, valid_total_ent);
}
for (entry = 0; out_entries || in_entries; ++entry) {
diff --git a/drivers/soc/qcom/smp2p_private.h b/drivers/soc/qcom/smp2p_private.h
index b332f7b..b93659f 100644
--- a/drivers/soc/qcom/smp2p_private.h
+++ b/drivers/soc/qcom/smp2p_private.h
@@ -50,49 +50,63 @@
| (((new_value) << (bit)) & (mask)); \
}
-#define SMP2P_GET_LOCAL_PID(hdr) \
- SMP2P_GET_BITS(hdr, SMP2P_LOCAL_PID_MASK, SMP2P_LOCAL_PID_BIT)
-#define SMP2P_SET_LOCAL_PID(hdr, pid) \
- SMP2P_SET_BITS(hdr, SMP2P_LOCAL_PID_MASK, SMP2P_LOCAL_PID_BIT, pid)
+#define SMP2P_IOMEM_GET_BITS(hdr, val, mask, bit) \
+ ({\
+ const volatile void __iomem *__p = (hdr); \
+ __p += offsetof(typeof(*(hdr)), val); \
+ (readl(__p) & (mask)) >> (bit); \
+ })
+#define SMP2P_IOMEM_SET_BITS(hdr, val, mask, bit, new_value) \
+ ({\
+ volatile void __iomem *__p = (hdr); \
+ __p += offsetof(typeof(*(hdr)), val); \
+ writel((readl(__p) & ~(mask)) \
+ | (((new_value) << (bit)) & (mask)), __p); \
+ })
-#define SMP2P_GET_REMOTE_PID(hdr) \
- SMP2P_GET_BITS(hdr, SMP2P_REMOTE_PID_MASK, SMP2P_REMOTE_PID_BIT)
-#define SMP2P_SET_REMOTE_PID(hdr, pid) \
- SMP2P_SET_BITS(hdr, SMP2P_REMOTE_PID_MASK, SMP2P_REMOTE_PID_BIT, pid)
+#define SMP2P_GET_LOCAL_PID(hdr, val) \
+ SMP2P_IOMEM_GET_BITS(hdr, val, SMP2P_LOCAL_PID_MASK, SMP2P_LOCAL_PID_BIT)
+#define SMP2P_SET_LOCAL_PID(hdr, val, pid) \
+ SMP2P_IOMEM_SET_BITS(hdr, val, SMP2P_LOCAL_PID_MASK, SMP2P_LOCAL_PID_BIT, pid)
-#define SMP2P_GET_VERSION(hdr) \
- SMP2P_GET_BITS(hdr, SMP2P_VERSION_MASK, SMP2P_VERSION_BIT)
-#define SMP2P_SET_VERSION(hdr, version) \
- SMP2P_SET_BITS(hdr, SMP2P_VERSION_MASK, SMP2P_VERSION_BIT, version)
+#define SMP2P_GET_REMOTE_PID(hdr, val) \
+ SMP2P_IOMEM_GET_BITS(hdr, val, SMP2P_REMOTE_PID_MASK, SMP2P_REMOTE_PID_BIT)
+#define SMP2P_SET_REMOTE_PID(hdr, val, pid) \
+ SMP2P_IOMEM_SET_BITS(hdr, val, SMP2P_REMOTE_PID_MASK, SMP2P_REMOTE_PID_BIT, pid)
-#define SMP2P_GET_FEATURES(hdr) \
- SMP2P_GET_BITS(hdr, SMP2P_FEATURE_MASK, SMP2P_FEATURE_BIT)
-#define SMP2P_SET_FEATURES(hdr, features) \
- SMP2P_SET_BITS(hdr, SMP2P_FEATURE_MASK, SMP2P_FEATURE_BIT, features)
+#define SMP2P_GET_VERSION(hdr, val) \
+ SMP2P_IOMEM_GET_BITS(hdr, val, SMP2P_VERSION_MASK, SMP2P_VERSION_BIT)
+#define SMP2P_SET_VERSION(hdr, val, version) \
+ SMP2P_IOMEM_SET_BITS(hdr, val, SMP2P_VERSION_MASK, SMP2P_VERSION_BIT, version)
-#define SMP2P_GET_ENT_TOTAL(hdr) \
- SMP2P_GET_BITS(hdr, SMP2P_ENT_TOTAL_MASK, SMP2P_ENT_TOTAL_BIT)
-#define SMP2P_SET_ENT_TOTAL(hdr, entries) \
- SMP2P_SET_BITS(hdr, SMP2P_ENT_TOTAL_MASK, SMP2P_ENT_TOTAL_BIT, entries)
+#define SMP2P_GET_FEATURES(hdr, val) \
+ SMP2P_IOMEM_GET_BITS(hdr, val, SMP2P_FEATURE_MASK, SMP2P_FEATURE_BIT)
+#define SMP2P_SET_FEATURES(hdr, val, features) \
+ SMP2P_IOMEM_SET_BITS(hdr, val, SMP2P_FEATURE_MASK, SMP2P_FEATURE_BIT, features)
-#define SMP2P_GET_ENT_VALID(hdr) \
- SMP2P_GET_BITS(hdr, SMP2P_ENT_VALID_MASK, SMP2P_ENT_VALID_BIT)
-#define SMP2P_SET_ENT_VALID(hdr, entries) \
- SMP2P_SET_BITS(hdr, SMP2P_ENT_VALID_MASK, SMP2P_ENT_VALID_BIT,\
+#define SMP2P_GET_ENT_TOTAL(hdr, val) \
+ SMP2P_IOMEM_GET_BITS(hdr, val, SMP2P_ENT_TOTAL_MASK, SMP2P_ENT_TOTAL_BIT)
+#define SMP2P_SET_ENT_TOTAL(hdr, val, entries) \
+ SMP2P_IOMEM_SET_BITS(hdr, val, SMP2P_ENT_TOTAL_MASK, SMP2P_ENT_TOTAL_BIT, entries)
+
+#define SMP2P_GET_ENT_VALID(hdr, val) \
+ SMP2P_IOMEM_GET_BITS(hdr, val, SMP2P_ENT_VALID_MASK, SMP2P_ENT_VALID_BIT)
+#define SMP2P_SET_ENT_VALID(hdr, val, entries) \
+ SMP2P_IOMEM_SET_BITS(hdr, val, SMP2P_ENT_VALID_MASK, SMP2P_ENT_VALID_BIT,\
entries)
-#define SMP2P_GET_RESTART_DONE(hdr) \
- SMP2P_GET_BITS(hdr, SMP2P_FLAGS_RESTART_DONE_MASK, \
+#define SMP2P_GET_RESTART_DONE(hdr, val) \
+ SMP2P_IOMEM_GET_BITS(hdr, val, SMP2P_FLAGS_RESTART_DONE_MASK, \
SMP2P_FLAGS_RESTART_DONE_BIT)
-#define SMP2P_SET_RESTART_DONE(hdr, value) \
- SMP2P_SET_BITS(hdr, SMP2P_FLAGS_RESTART_DONE_MASK, \
+#define SMP2P_SET_RESTART_DONE(hdr, val, value) \
+ SMP2P_IOMEM_SET_BITS(hdr, val, SMP2P_FLAGS_RESTART_DONE_MASK, \
SMP2P_FLAGS_RESTART_DONE_BIT, value)
-#define SMP2P_GET_RESTART_ACK(hdr) \
- SMP2P_GET_BITS(hdr, SMP2P_FLAGS_RESTART_ACK_MASK, \
+#define SMP2P_GET_RESTART_ACK(hdr, val) \
+ SMP2P_IOMEM_GET_BITS(hdr, val, SMP2P_FLAGS_RESTART_ACK_MASK, \
SMP2P_FLAGS_RESTART_ACK_BIT)
-#define SMP2P_SET_RESTART_ACK(hdr, value) \
- SMP2P_SET_BITS(hdr, SMP2P_FLAGS_RESTART_ACK_MASK, \
+#define SMP2P_SET_RESTART_ACK(hdr, val, value) \
+ SMP2P_IOMEM_SET_BITS(hdr, val, SMP2P_FLAGS_RESTART_ACK_MASK, \
SMP2P_FLAGS_RESTART_ACK_BIT, value)
/* Loopback Command Macros */
diff --git a/drivers/soc/qcom/socinfo.c b/drivers/soc/qcom/socinfo.c
index e02bf84..366b929 100644
--- a/drivers/soc/qcom/socinfo.c
+++ b/drivers/soc/qcom/socinfo.c
@@ -54,7 +54,7 @@ enum {
HW_PLATFORM_FFA = 2,
HW_PLATFORM_FLUID = 3,
HW_PLATFORM_SVLTE_FFA = 4,
- HW_PLATFORM_SVLTE_SURF = 5,
+ HW_PLATFORM_OEM = 5,
HW_PLATFORM_MTP_MDM = 7,
HW_PLATFORM_MTP = 8,
HW_PLATFORM_LIQUID = 9,
@@ -76,7 +76,7 @@ const char *hw_platform[] = {
[HW_PLATFORM_FFA] = "FFA",
[HW_PLATFORM_FLUID] = "Fluid",
[HW_PLATFORM_SVLTE_FFA] = "SVLTE_FFA",
- [HW_PLATFORM_SVLTE_SURF] = "SLVTE_SURF",
+ [HW_PLATFORM_OEM] = "OEM",
[HW_PLATFORM_MTP_MDM] = "MDM_MTP_NO_DISPLAY",
[HW_PLATFORM_MTP] = "MTP",
[HW_PLATFORM_RCM] = "RCM",
diff --git a/drivers/soc/qcom/spcom.c b/drivers/soc/qcom/spcom.c
index d568014..708be88 100644
--- a/drivers/soc/qcom/spcom.c
+++ b/drivers/soc/qcom/spcom.c
@@ -2012,7 +2012,8 @@ static int spcom_handle_write(struct spcom_channel *ch,
if (!ch && cmd_id != SPCOM_CMD_CREATE_CHANNEL) {
pr_err("channel context is null\n");
- return -EINVAL;
+ ret = -EINVAL;
+ goto exit_err;
}
switch (cmd_id) {
@@ -2036,6 +2037,7 @@ static int spcom_handle_write(struct spcom_channel *ch,
ret = -EINVAL;
}
+exit_err:
mutex_unlock(&spcom_dev->cmd_lock);
return ret;
diff --git a/drivers/soc/qcom/subsys-pil-tz.c b/drivers/soc/qcom/subsys-pil-tz.c
index e221e6b..60386fa 100644
--- a/drivers/soc/qcom/subsys-pil-tz.c
+++ b/drivers/soc/qcom/subsys-pil-tz.c
@@ -36,7 +36,6 @@
#define XO_FREQ 19200000
#define PROXY_TIMEOUT_MS 10000
-#define MAX_SSR_REASON_LEN 256U
#define STOP_ACK_TIMEOUT_MS 1000
#define CRASH_STOP_ACK_TO_MS 200
@@ -839,15 +838,15 @@ static struct pil_reset_ops pil_ops_trusted = {
.deinit_image = pil_deinit_image_trusted,
};
-static void log_failure_reason(const struct pil_tz_data *d)
+static void log_failure_reason(struct pil_tz_data *d)
{
u32 size;
- char *smem_reason, reason[MAX_SSR_REASON_LEN];
+ char *smem_reason, *reason;
const char *name = d->subsys_desc.name;
if (d->smem_id == -1)
return;
-
+ reason = d->subsys_desc.last_crash_reason;
smem_reason = smem_get_entry_no_rlock(d->smem_id, &size, 0,
SMEM_ANY_HOST_FLAG);
if (!smem_reason || !size) {
diff --git a/drivers/soc/qcom/subsystem_restart.c b/drivers/soc/qcom/subsystem_restart.c
index 110cdf7..b336fd1 100644
--- a/drivers/soc/qcom/subsystem_restart.c
+++ b/drivers/soc/qcom/subsystem_restart.c
@@ -22,12 +22,14 @@
#include <linux/io.h>
#include <linux/kthread.h>
#include <linux/time.h>
+#include <linux/rtc.h>
#include <linux/suspend.h>
#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/spinlock.h>
#include <linux/device.h>
#include <linux/idr.h>
+#include <linux/debugfs.h>
#include <linux/interrupt.h>
#include <linux/of_gpio.h>
#include <linux/cdev.h>
@@ -149,6 +151,7 @@ struct restart_log {
* @restart_level: restart level (0 - panic, 1 - related, 2 - independent, etc.)
* @restart_order: order of other devices this devices restarts with
* @crash_count: number of times the device has crashed
+ * @dentry: debugfs directory for this device
* @do_ramdump_on_put: ramdump on subsystem_put() if true
* @err_ready: completion variable to record error ready from subsystem
* @crashed: indicates if subsystem has crashed
@@ -170,6 +173,9 @@ struct subsys_device {
int restart_level;
int crash_count;
struct subsys_soc_restart_order *restart_order;
+#ifdef CONFIG_DEBUG_FS
+ struct dentry *dentry;
+#endif
bool do_ramdump_on_put;
struct cdev char_dev;
dev_t dev_no;
@@ -218,6 +224,19 @@ static ssize_t crash_count_show(struct device *dev,
{
return snprintf(buf, PAGE_SIZE, "%d\n", to_subsys(dev)->crash_count);
}
+static ssize_t crash_reason_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%s\n",
+ to_subsys(dev)->desc->last_crash_reason);
+}
+
+static ssize_t crash_timestamp_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "%s\n",
+ to_subsys(dev)->desc->last_crash_timestamp);
+}
static ssize_t
restart_level_show(struct device *dev, struct device_attribute *attr, char *buf)
@@ -344,17 +363,18 @@ static struct device_attribute subsys_attrs[] = {
__ATTR_RO(name),
__ATTR_RO(state),
__ATTR_RO(crash_count),
+ __ATTR_RO(crash_reason),
+ __ATTR_RO(crash_timestamp),
__ATTR(restart_level, 0644, restart_level_show, restart_level_store),
__ATTR(firmware_name, 0644, firmware_name_show, firmware_name_store),
__ATTR(system_debug, 0644, system_debug_show, system_debug_store),
__ATTR_NULL,
};
-struct bus_type subsys_bus_type = {
+static struct bus_type subsys_bus_type = {
.name = "msm_subsys",
.dev_attrs = subsys_attrs,
};
-EXPORT_SYMBOL(subsys_bus_type);
static DEFINE_IDA(subsys_ida);
@@ -603,7 +623,10 @@ static int wait_for_err_ready(struct subsys_device *subsys)
static int subsystem_shutdown(struct subsys_device *dev, void *data)
{
const char *name = dev->desc->name;
+ char *timestamp = dev->desc->last_crash_timestamp;
int ret;
+ struct timespec ts_rtc;
+ struct rtc_time tm;
pr_info("[%s:%d]: Shutting down %s\n",
current->comm, current->pid, name);
@@ -617,6 +640,13 @@ static int subsystem_shutdown(struct subsys_device *dev, void *data)
return ret;
}
}
+ /* record crash time */
+ getnstimeofday(&ts_rtc);
+ rtc_time_to_tm(ts_rtc.tv_sec - (sys_tz.tz_minuteswest * 60), &tm);
+ snprintf(timestamp, MAX_CRASH_TIMESTAMP_LEN,
+ "%d-%02d-%02d_%02d-%02d-%02d",
+ tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,
+ tm.tm_hour, tm.tm_min, tm.tm_sec);
dev->crash_count++;
subsys_set_state(dev, SUBSYS_OFFLINE);
disable_all_irqs(dev);
@@ -1212,6 +1242,87 @@ void notify_proxy_unvote(struct device *device)
notify_each_subsys_device(&dev, 1, SUBSYS_PROXY_UNVOTE, NULL);
}
+#ifdef CONFIG_DEBUG_FS
+static ssize_t subsys_debugfs_read(struct file *filp, char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ int r;
+ char buf[40];
+ struct subsys_device *subsys = filp->private_data;
+
+ r = snprintf(buf, sizeof(buf), "%d\n", subsys->count);
+ return simple_read_from_buffer(ubuf, cnt, ppos, buf, r);
+}
+
+static ssize_t subsys_debugfs_write(struct file *filp,
+ const char __user *ubuf, size_t cnt, loff_t *ppos)
+{
+ struct subsys_device *subsys = filp->private_data;
+ char buf[10];
+ char *cmp;
+
+ cnt = min(cnt, sizeof(buf) - 1);
+ if (copy_from_user(&buf, ubuf, cnt))
+ return -EFAULT;
+ buf[cnt] = '\0';
+ cmp = strstrip(buf);
+
+ if (!strcmp(cmp, "restart")) {
+ if (subsystem_restart_dev(subsys))
+ return -EIO;
+ } else if (!strcmp(cmp, "get")) {
+ if (subsystem_get(subsys->desc->name))
+ return -EIO;
+ } else if (!strcmp(cmp, "put")) {
+ subsystem_put(subsys);
+ } else {
+ return -EINVAL;
+ }
+
+ return cnt;
+}
+
+static const struct file_operations subsys_debugfs_fops = {
+ .open = simple_open,
+ .read = subsys_debugfs_read,
+ .write = subsys_debugfs_write,
+};
+
+static struct dentry *subsys_base_dir;
+
+static int __init subsys_debugfs_init(void)
+{
+ subsys_base_dir = debugfs_create_dir("msm_subsys", NULL);
+ return !subsys_base_dir ? -ENOMEM : 0;
+}
+
+static void subsys_debugfs_exit(void)
+{
+ debugfs_remove_recursive(subsys_base_dir);
+}
+
+static int subsys_debugfs_add(struct subsys_device *subsys)
+{
+ if (!subsys_base_dir)
+ return -ENOMEM;
+
+ subsys->dentry = debugfs_create_file(subsys->desc->name,
+ 0644, subsys_base_dir,
+ subsys, &subsys_debugfs_fops);
+ return !subsys->dentry ? -ENOMEM : 0;
+}
+
+static void subsys_debugfs_remove(struct subsys_device *subsys)
+{
+ debugfs_remove(subsys->dentry);
+}
+#else
+static int __init subsys_debugfs_init(void) { return 0; };
+static void subsys_debugfs_exit(void) { }
+static int subsys_debugfs_add(struct subsys_device *subsys) { return 0; }
+static void subsys_debugfs_remove(struct subsys_device *subsys) { }
+#endif
+
static int subsys_device_open(struct inode *inode, struct file *file)
{
struct subsys_device *device, *subsys_dev = 0;
@@ -1652,8 +1763,17 @@ struct subsys_device *subsys_register(struct subsys_desc *desc)
mutex_init(&subsys->track.lock);
+ ret = subsys_debugfs_add(subsys);
+ if (ret) {
+ ida_simple_remove(&subsys_ida, subsys->id);
+ wakeup_source_trash(&subsys->ssr_wlock);
+ kfree(subsys);
+ return ERR_PTR(ret);
+ }
+
ret = device_register(&subsys->dev);
if (ret) {
+ subsys_debugfs_remove(subsys);
put_device(&subsys->dev);
return ERR_PTR(ret);
}
@@ -1714,6 +1834,7 @@ struct subsys_device *subsys_register(struct subsys_desc *desc)
if (ofnode)
subsys_remove_restart_order(ofnode);
err_register:
+ subsys_debugfs_remove(subsys);
device_unregister(&subsys->dev);
return ERR_PTR(ret);
}
@@ -1742,6 +1863,7 @@ void subsys_unregister(struct subsys_device *subsys)
WARN_ON(subsys->count);
device_unregister(&subsys->dev);
mutex_unlock(&subsys->track.lock);
+ subsys_debugfs_remove(subsys);
subsys_char_device_remove(subsys);
sysmon_notifier_unregister(subsys->desc);
if (subsys->desc->edge)
@@ -1781,6 +1903,9 @@ static int __init subsys_restart_init(void)
ret = bus_register(&subsys_bus_type);
if (ret)
goto err_bus;
+ ret = subsys_debugfs_init();
+ if (ret)
+ goto err_debugfs;
char_class = class_create(THIS_MODULE, "subsys");
if (IS_ERR(char_class)) {
@@ -1799,6 +1924,8 @@ static int __init subsys_restart_init(void)
err_soc:
class_destroy(char_class);
err_class:
+ subsys_debugfs_exit();
+err_debugfs:
bus_unregister(&subsys_bus_type);
err_bus:
destroy_workqueue(ssr_wq);
diff --git a/drivers/soc/qcom/tracer_pkt_private.h b/drivers/soc/qcom/tracer_pkt_private.h
index fc760e6..79e0bd3 100644
--- a/drivers/soc/qcom/tracer_pkt_private.h
+++ b/drivers/soc/qcom/tracer_pkt_private.h
@@ -15,7 +15,7 @@
#undef TRACE_SYSTEM
#define TRACE_SYSTEM tracer_pkt
#undef TRACE_INCLUDE_PATH
-#define TRACE_INCLUDE_PATH .
+#define TRACE_INCLUDE_PATH ../../drivers/soc/qcom
#undef TRACE_INCLUDE_FILE
#define TRACE_INCLUDE_FILE tracer_pkt_private
diff --git a/drivers/soc/qcom/wcnss_ctrl.c b/drivers/soc/qcom/wcnss_ctrl.c
index 520aedd..78d3dba 100644
--- a/drivers/soc/qcom/wcnss_ctrl.c
+++ b/drivers/soc/qcom/wcnss_ctrl.c
@@ -247,7 +247,7 @@ static int wcnss_download_nv(struct wcnss_ctrl *wcnss, bool *expect_cbc)
/* Increment for next fragment */
req->seq++;
- data += req->hdr.len;
+ data += NV_FRAGMENT_SIZE;
left -= NV_FRAGMENT_SIZE;
} while (left > 0);
diff --git a/drivers/soc/tegra/pmc.c b/drivers/soc/tegra/pmc.c
index 7792ed8..9685f9b8 100644
--- a/drivers/soc/tegra/pmc.c
+++ b/drivers/soc/tegra/pmc.c
@@ -1189,7 +1189,7 @@ static void tegra_pmc_init_tsense_reset(struct tegra_pmc *pmc)
if (!pmc->soc->has_tsense_reset)
return;
- np = of_find_node_by_name(pmc->dev->of_node, "i2c-thermtrip");
+ np = of_get_child_by_name(pmc->dev->of_node, "i2c-thermtrip");
if (!np) {
dev_warn(dev, "i2c-thermtrip node not found, %s.\n", disabled);
return;
diff --git a/drivers/spi/spi-bcm-qspi.c b/drivers/spi/spi-bcm-qspi.c
index 7d629b4..63231760f 100644
--- a/drivers/spi/spi-bcm-qspi.c
+++ b/drivers/spi/spi-bcm-qspi.c
@@ -514,7 +514,7 @@ static int bcm_qspi_bspi_set_mode(struct bcm_qspi *qspi,
static void bcm_qspi_enable_bspi(struct bcm_qspi *qspi)
{
- if (!has_bspi(qspi) || (qspi->bspi_enabled))
+ if (!has_bspi(qspi))
return;
qspi->bspi_enabled = 1;
@@ -529,7 +529,7 @@ static void bcm_qspi_enable_bspi(struct bcm_qspi *qspi)
static void bcm_qspi_disable_bspi(struct bcm_qspi *qspi)
{
- if (!has_bspi(qspi) || (!qspi->bspi_enabled))
+ if (!has_bspi(qspi))
return;
qspi->bspi_enabled = 0;
@@ -543,16 +543,19 @@ static void bcm_qspi_disable_bspi(struct bcm_qspi *qspi)
static void bcm_qspi_chip_select(struct bcm_qspi *qspi, int cs)
{
- u32 data = 0;
+ u32 rd = 0;
+ u32 wr = 0;
- if (qspi->curr_cs == cs)
- return;
if (qspi->base[CHIP_SELECT]) {
- data = bcm_qspi_read(qspi, CHIP_SELECT, 0);
- data = (data & ~0xff) | (1 << cs);
- bcm_qspi_write(qspi, CHIP_SELECT, 0, data);
+ rd = bcm_qspi_read(qspi, CHIP_SELECT, 0);
+ wr = (rd & ~0xff) | (1 << cs);
+ if (rd == wr)
+ return;
+ bcm_qspi_write(qspi, CHIP_SELECT, 0, wr);
usleep_range(10, 20);
}
+
+ dev_dbg(&qspi->pdev->dev, "using cs:%d\n", cs);
qspi->curr_cs = cs;
}
@@ -770,8 +773,13 @@ static int write_to_hw(struct bcm_qspi *qspi, struct spi_device *spi)
dev_dbg(&qspi->pdev->dev, "WR %04x\n", val);
}
mspi_cdram = MSPI_CDRAM_CONT_BIT;
- mspi_cdram |= (~(1 << spi->chip_select) &
- MSPI_CDRAM_PCS);
+
+ if (has_bspi(qspi))
+ mspi_cdram &= ~1;
+ else
+ mspi_cdram |= (~(1 << spi->chip_select) &
+ MSPI_CDRAM_PCS);
+
mspi_cdram |= ((tp.trans->bits_per_word <= 8) ? 0 :
MSPI_CDRAM_BITSE_BIT);
@@ -1212,7 +1220,7 @@ int bcm_qspi_probe(struct platform_device *pdev,
qspi->base[MSPI] = devm_ioremap_resource(dev, res);
if (IS_ERR(qspi->base[MSPI])) {
ret = PTR_ERR(qspi->base[MSPI]);
- goto qspi_probe_err;
+ goto qspi_resource_err;
}
} else {
goto qspi_resource_err;
@@ -1223,7 +1231,7 @@ int bcm_qspi_probe(struct platform_device *pdev,
qspi->base[BSPI] = devm_ioremap_resource(dev, res);
if (IS_ERR(qspi->base[BSPI])) {
ret = PTR_ERR(qspi->base[BSPI]);
- goto qspi_probe_err;
+ goto qspi_resource_err;
}
qspi->bspi_mode = true;
} else {
diff --git a/drivers/spi/spi-bcm2835.c b/drivers/spi/spi-bcm2835.c
index f35cc10..25abf2d 100644
--- a/drivers/spi/spi-bcm2835.c
+++ b/drivers/spi/spi-bcm2835.c
@@ -88,7 +88,7 @@ struct bcm2835_spi {
u8 *rx_buf;
int tx_len;
int rx_len;
- bool dma_pending;
+ unsigned int dma_pending;
};
static inline u32 bcm2835_rd(struct bcm2835_spi *bs, unsigned reg)
@@ -155,8 +155,7 @@ static irqreturn_t bcm2835_spi_interrupt(int irq, void *dev_id)
/* Write as many bytes as possible to FIFO */
bcm2835_wr_fifo(bs);
- /* based on flags decide if we can finish the transfer */
- if (bcm2835_rd(bs, BCM2835_SPI_CS) & BCM2835_SPI_CS_DONE) {
+ if (!bs->rx_len) {
/* Transfer complete - reset SPI HW */
bcm2835_spi_reset_hw(master);
/* wake up the framework */
@@ -233,10 +232,9 @@ static void bcm2835_spi_dma_done(void *data)
* is called the tx-dma must have finished - can't get to this
* situation otherwise...
*/
- dmaengine_terminate_all(master->dma_tx);
-
- /* mark as no longer pending */
- bs->dma_pending = 0;
+ if (cmpxchg(&bs->dma_pending, true, false)) {
+ dmaengine_terminate_all(master->dma_tx);
+ }
/* and mark as completed */;
complete(&master->xfer_completion);
@@ -342,6 +340,7 @@ static int bcm2835_spi_transfer_one_dma(struct spi_master *master,
if (ret) {
/* need to reset on errors */
dmaengine_terminate_all(master->dma_tx);
+ bs->dma_pending = false;
bcm2835_spi_reset_hw(master);
return ret;
}
@@ -617,10 +616,9 @@ static void bcm2835_spi_handle_err(struct spi_master *master,
struct bcm2835_spi *bs = spi_master_get_devdata(master);
/* if an error occurred and we have an active dma, then terminate */
- if (bs->dma_pending) {
+ if (cmpxchg(&bs->dma_pending, true, false)) {
dmaengine_terminate_all(master->dma_tx);
dmaengine_terminate_all(master->dma_rx);
- bs->dma_pending = 0;
}
/* and reset */
bcm2835_spi_reset_hw(master);
diff --git a/drivers/spi/spi-davinci.c b/drivers/spi/spi-davinci.c
index 0d8f43a..1905d20 100644
--- a/drivers/spi/spi-davinci.c
+++ b/drivers/spi/spi-davinci.c
@@ -215,7 +215,7 @@ static void davinci_spi_chipselect(struct spi_device *spi, int value)
pdata = &dspi->pdata;
/* program delay transfers if tx_delay is non zero */
- if (spicfg->wdelay)
+ if (spicfg && spicfg->wdelay)
spidat1 |= SPIDAT1_WDEL;
/*
diff --git a/drivers/spi/spi-fsl-dspi.c b/drivers/spi/spi-fsl-dspi.c
index a67b0ff..db3b6e9 100644
--- a/drivers/spi/spi-fsl-dspi.c
+++ b/drivers/spi/spi-fsl-dspi.c
@@ -715,21 +715,6 @@ static int dspi_probe(struct platform_device *pdev)
return PTR_ERR(dspi->regmap);
}
- dspi_init(dspi);
- dspi->irq = platform_get_irq(pdev, 0);
- if (dspi->irq < 0) {
- dev_err(&pdev->dev, "can't get platform irq\n");
- ret = dspi->irq;
- goto out_master_put;
- }
-
- ret = devm_request_irq(&pdev->dev, dspi->irq, dspi_interrupt, 0,
- pdev->name, dspi);
- if (ret < 0) {
- dev_err(&pdev->dev, "Unable to attach DSPI interrupt\n");
- goto out_master_put;
- }
-
dspi->clk = devm_clk_get(&pdev->dev, "dspi");
if (IS_ERR(dspi->clk)) {
ret = PTR_ERR(dspi->clk);
@@ -740,6 +725,21 @@ static int dspi_probe(struct platform_device *pdev)
if (ret)
goto out_master_put;
+ dspi_init(dspi);
+ dspi->irq = platform_get_irq(pdev, 0);
+ if (dspi->irq < 0) {
+ dev_err(&pdev->dev, "can't get platform irq\n");
+ ret = dspi->irq;
+ goto out_clk_put;
+ }
+
+ ret = devm_request_irq(&pdev->dev, dspi->irq, dspi_interrupt, 0,
+ pdev->name, dspi);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "Unable to attach DSPI interrupt\n");
+ goto out_clk_put;
+ }
+
master->max_speed_hz =
clk_get_rate(dspi->clk) / dspi->devtype_data->max_clock_factor;
diff --git a/drivers/spi/spi-geni-qcom.c b/drivers/spi/spi-geni-qcom.c
index a959864..663c1db 100644
--- a/drivers/spi/spi-geni-qcom.c
+++ b/drivers/spi/spi-geni-qcom.c
@@ -155,6 +155,7 @@ struct spi_geni_master {
int num_xfers;
void *ipc;
bool shared_se;
+ bool dis_autosuspend;
};
static struct spi_master *get_spi_master(struct device *dev)
@@ -333,7 +334,8 @@ static int select_xfer_mode(struct spi_master *spi,
}
static struct msm_gpi_tre *setup_config0_tre(struct spi_transfer *xfer,
- struct spi_geni_master *mas, u16 mode,
+ struct spi_geni_master *mas,
+ struct spi_master *spi, u16 mode,
u32 cs_clk_delay, u32 inter_words_delay)
{
struct msm_gpi_tre *c0_tre = &mas->gsi[mas->num_xfers].config0_tre;
@@ -551,7 +553,7 @@ static int setup_gsi_xfer(struct spi_transfer *xfer,
mas->cur_word_len = xfer->bits_per_word;
mas->cur_speed_hz = xfer->speed_hz;
tx_nent++;
- c0_tre = setup_config0_tre(xfer, mas, spi_slv->mode,
+ c0_tre = setup_config0_tre(xfer, mas, spi, spi_slv->mode,
cs_clk_delay, inter_words_delay);
if (IS_ERR_OR_NULL(c0_tre)) {
dev_err(mas->dev, "%s:Err setting c0tre:%d\n",
@@ -583,11 +585,10 @@ static int setup_gsi_xfer(struct spi_transfer *xfer,
}
cs |= spi_slv->chip_select;
- if (!xfer->cs_change) {
- if (!list_is_last(&xfer->transfer_list,
- &spi->cur_msg->transfers))
- go_flags |= FRAGMENTATION;
- }
+ if (!list_is_last(&xfer->transfer_list, &spi->cur_msg->transfers) ==
+ !xfer->cs_change)
+ go_flags |= FRAGMENTATION;
+
go_tre = setup_go_tre(cmd, cs, rx_len, go_flags, mas);
sg_init_table(xfer_tx_sg, tx_nent);
@@ -752,7 +753,7 @@ static int spi_geni_unprepare_message(struct spi_master *spi_mas,
static int spi_geni_prepare_transfer_hardware(struct spi_master *spi)
{
struct spi_geni_master *mas = spi_master_get_devdata(spi);
- int ret = 0;
+ int ret = 0, count = 0;
u32 max_speed = spi->cur_msg->spi->max_speed_hz;
struct se_geni_rsc *rsc = &mas->spi_rsc;
@@ -780,7 +781,12 @@ static int spi_geni_prepare_transfer_hardware(struct spi_master *spi)
} else {
ret = 0;
}
-
+ if (mas->dis_autosuspend) {
+ count = atomic_read(&mas->dev->power.usage_count);
+ if (count <= 0)
+ GENI_SE_ERR(mas->ipc, false, NULL,
+ "resume usage count mismatch:%d", count);
+ }
if (unlikely(!mas->setup)) {
int proto = get_se_proto(mas->base);
unsigned int major;
@@ -869,7 +875,12 @@ static int spi_geni_prepare_transfer_hardware(struct spi_master *spi)
mas->shared_se =
(geni_read_reg(mas->base, GENI_IF_FIFO_DISABLE_RO) &
FIFO_IF_DISABLE);
+ if (mas->dis_autosuspend)
+ GENI_SE_DBG(mas->ipc, false, mas->dev,
+ "Auto Suspend is disabled\n");
}
+ if (mas->dis_autosuspend)
+ dmaengine_resume(mas->tx);
exit_prepare_transfer_hardware:
return ret;
}
@@ -877,7 +888,7 @@ static int spi_geni_prepare_transfer_hardware(struct spi_master *spi)
static int spi_geni_unprepare_transfer_hardware(struct spi_master *spi)
{
struct spi_geni_master *mas = spi_master_get_devdata(spi);
-
+ int count = 0;
if (mas->shared_se) {
struct se_geni_rsc *rsc;
int ret = 0;
@@ -890,8 +901,17 @@ static int spi_geni_unprepare_transfer_hardware(struct spi_master *spi)
"%s: Error %d pinctrl_select_state\n", __func__, ret);
}
- pm_runtime_mark_last_busy(mas->dev);
- pm_runtime_put_autosuspend(mas->dev);
+ if (mas->dis_autosuspend) {
+ dmaengine_pause(mas->tx);
+ pm_runtime_put_sync(mas->dev);
+ count = atomic_read(&mas->dev->power.usage_count);
+ if (count < 0)
+ GENI_SE_ERR(mas->ipc, false, NULL,
+ "suspend usage count mismatch:%d", count);
+ } else {
+ pm_runtime_mark_last_busy(mas->dev);
+ pm_runtime_put_autosuspend(mas->dev);
+ }
return 0;
}
@@ -949,11 +969,9 @@ static void setup_fifo_xfer(struct spi_transfer *xfer,
trans_len = (xfer->len / bytes_per_word) & TRANS_LEN_MSK;
}
- if (!xfer->cs_change) {
- if (!list_is_last(&xfer->transfer_list,
- &spi->cur_msg->transfers))
- m_param |= FRAGMENTATION;
- }
+ if (!list_is_last(&xfer->transfer_list, &spi->cur_msg->transfers) ==
+ !xfer->cs_change)
+ m_param |= FRAGMENTATION;
mas->cur_xfer = xfer;
if (m_cmd & SPI_TX_ONLY) {
@@ -1329,7 +1347,9 @@ static int spi_geni_probe(struct platform_device *pdev)
rt_pri = of_property_read_bool(pdev->dev.of_node, "qcom,rt");
if (rt_pri)
spi->rt = true;
-
+ geni_mas->dis_autosuspend =
+ of_property_read_bool(pdev->dev.of_node,
+ "qcom,disable-autosuspend");
geni_mas->phys_addr = res->start;
geni_mas->size = resource_size(res);
geni_mas->base = devm_ioremap(&pdev->dev, res->start,
@@ -1369,8 +1389,11 @@ static int spi_geni_probe(struct platform_device *pdev)
init_completion(&geni_mas->tx_cb);
init_completion(&geni_mas->rx_cb);
pm_runtime_set_suspended(&pdev->dev);
- pm_runtime_set_autosuspend_delay(&pdev->dev, SPI_AUTO_SUSPEND_DELAY);
- pm_runtime_use_autosuspend(&pdev->dev);
+ if (!geni_mas->dis_autosuspend) {
+ pm_runtime_set_autosuspend_delay(&pdev->dev,
+ SPI_AUTO_SUSPEND_DELAY);
+ pm_runtime_use_autosuspend(&pdev->dev);
+ }
pm_runtime_enable(&pdev->dev);
ret = spi_register_master(spi);
if (ret) {
diff --git a/drivers/spi/spi-pxa2xx.h b/drivers/spi/spi-pxa2xx.h
index ce31b81..b8e004d 100644
--- a/drivers/spi/spi-pxa2xx.h
+++ b/drivers/spi/spi-pxa2xx.h
@@ -38,7 +38,7 @@ struct driver_data {
/* SSP register addresses */
void __iomem *ioaddr;
- u32 ssdr_physical;
+ phys_addr_t ssdr_physical;
/* SSP masks*/
u32 dma_cr1;
diff --git a/drivers/spi/spi-rspi.c b/drivers/spi/spi-rspi.c
index a816f07..093c9cf 100644
--- a/drivers/spi/spi-rspi.c
+++ b/drivers/spi/spi-rspi.c
@@ -597,11 +597,13 @@ static int rspi_dma_transfer(struct rspi_data *rspi, struct sg_table *tx,
ret = wait_event_interruptible_timeout(rspi->wait,
rspi->dma_callbacked, HZ);
- if (ret > 0 && rspi->dma_callbacked)
+ if (ret > 0 && rspi->dma_callbacked) {
ret = 0;
- else if (!ret) {
- dev_err(&rspi->master->dev, "DMA timeout\n");
- ret = -ETIMEDOUT;
+ } else {
+ if (!ret) {
+ dev_err(&rspi->master->dev, "DMA timeout\n");
+ ret = -ETIMEDOUT;
+ }
if (tx)
dmaengine_terminate_all(rspi->master->dma_tx);
if (rx)
@@ -1313,12 +1315,36 @@ static const struct platform_device_id spi_driver_ids[] = {
MODULE_DEVICE_TABLE(platform, spi_driver_ids);
+#ifdef CONFIG_PM_SLEEP
+static int rspi_suspend(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct rspi_data *rspi = platform_get_drvdata(pdev);
+
+ return spi_master_suspend(rspi->master);
+}
+
+static int rspi_resume(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct rspi_data *rspi = platform_get_drvdata(pdev);
+
+ return spi_master_resume(rspi->master);
+}
+
+static SIMPLE_DEV_PM_OPS(rspi_pm_ops, rspi_suspend, rspi_resume);
+#define DEV_PM_OPS &rspi_pm_ops
+#else
+#define DEV_PM_OPS NULL
+#endif /* CONFIG_PM_SLEEP */
+
static struct platform_driver rspi_driver = {
.probe = rspi_probe,
.remove = rspi_remove,
.id_table = spi_driver_ids,
.driver = {
.name = "renesas_spi",
+ .pm = DEV_PM_OPS,
.of_match_table = of_match_ptr(rspi_of_match),
},
};
diff --git a/drivers/spi/spi-sh-msiof.c b/drivers/spi/spi-sh-msiof.c
index cbf02eb..711ea52 100644
--- a/drivers/spi/spi-sh-msiof.c
+++ b/drivers/spi/spi-sh-msiof.c
@@ -373,7 +373,8 @@ static void sh_msiof_spi_set_mode_regs(struct sh_msiof_spi_priv *p,
static void sh_msiof_reset_str(struct sh_msiof_spi_priv *p)
{
- sh_msiof_write(p, STR, sh_msiof_read(p, STR));
+ sh_msiof_write(p, STR,
+ sh_msiof_read(p, STR) & ~(STR_TDREQ | STR_RDREQ));
}
static void sh_msiof_spi_write_fifo_8(struct sh_msiof_spi_priv *p,
@@ -1275,12 +1276,37 @@ static const struct platform_device_id spi_driver_ids[] = {
};
MODULE_DEVICE_TABLE(platform, spi_driver_ids);
+#ifdef CONFIG_PM_SLEEP
+static int sh_msiof_spi_suspend(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct sh_msiof_spi_priv *p = platform_get_drvdata(pdev);
+
+ return spi_master_suspend(p->master);
+}
+
+static int sh_msiof_spi_resume(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct sh_msiof_spi_priv *p = platform_get_drvdata(pdev);
+
+ return spi_master_resume(p->master);
+}
+
+static SIMPLE_DEV_PM_OPS(sh_msiof_spi_pm_ops, sh_msiof_spi_suspend,
+ sh_msiof_spi_resume);
+#define DEV_PM_OPS &sh_msiof_spi_pm_ops
+#else
+#define DEV_PM_OPS NULL
+#endif /* CONFIG_PM_SLEEP */
+
static struct platform_driver sh_msiof_spi_drv = {
.probe = sh_msiof_spi_probe,
.remove = sh_msiof_spi_remove,
.id_table = spi_driver_ids,
.driver = {
.name = "spi_sh_msiof",
+ .pm = DEV_PM_OPS,
.of_match_table = of_match_ptr(sh_msiof_match),
},
};
diff --git a/drivers/spi/spi-tegra20-slink.c b/drivers/spi/spi-tegra20-slink.c
index 85c91f5..af2880d 100644
--- a/drivers/spi/spi-tegra20-slink.c
+++ b/drivers/spi/spi-tegra20-slink.c
@@ -1063,6 +1063,24 @@ static int tegra_slink_probe(struct platform_device *pdev)
goto exit_free_master;
}
+ /* disabled clock may cause interrupt storm upon request */
+ tspi->clk = devm_clk_get(&pdev->dev, NULL);
+ if (IS_ERR(tspi->clk)) {
+ ret = PTR_ERR(tspi->clk);
+ dev_err(&pdev->dev, "Can not get clock %d\n", ret);
+ goto exit_free_master;
+ }
+ ret = clk_prepare(tspi->clk);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "Clock prepare failed %d\n", ret);
+ goto exit_free_master;
+ }
+ ret = clk_enable(tspi->clk);
+ if (ret < 0) {
+ dev_err(&pdev->dev, "Clock enable failed %d\n", ret);
+ goto exit_free_master;
+ }
+
spi_irq = platform_get_irq(pdev, 0);
tspi->irq = spi_irq;
ret = request_threaded_irq(tspi->irq, tegra_slink_isr,
@@ -1071,14 +1089,7 @@ static int tegra_slink_probe(struct platform_device *pdev)
if (ret < 0) {
dev_err(&pdev->dev, "Failed to register ISR for IRQ %d\n",
tspi->irq);
- goto exit_free_master;
- }
-
- tspi->clk = devm_clk_get(&pdev->dev, NULL);
- if (IS_ERR(tspi->clk)) {
- dev_err(&pdev->dev, "can not get clock\n");
- ret = PTR_ERR(tspi->clk);
- goto exit_free_irq;
+ goto exit_clk_disable;
}
tspi->rst = devm_reset_control_get(&pdev->dev, "spi");
@@ -1138,6 +1149,8 @@ static int tegra_slink_probe(struct platform_device *pdev)
tegra_slink_deinit_dma_param(tspi, true);
exit_free_irq:
free_irq(spi_irq, tspi);
+exit_clk_disable:
+ clk_disable(tspi->clk);
exit_free_master:
spi_master_put(master);
return ret;
@@ -1150,6 +1163,8 @@ static int tegra_slink_remove(struct platform_device *pdev)
free_irq(tspi->irq, tspi);
+ clk_disable(tspi->clk);
+
if (tspi->tx_dma_chan)
tegra_slink_deinit_dma_param(tspi, false);
diff --git a/drivers/spi/spidev.c b/drivers/spi/spidev.c
index 2e05046..38d4cb8 100644
--- a/drivers/spi/spidev.c
+++ b/drivers/spi/spidev.c
@@ -696,6 +696,7 @@ static struct class *spidev_class;
static const struct of_device_id spidev_dt_ids[] = {
{ .compatible = "rohm,dh2228fv" },
{ .compatible = "lineartechnology,ltc2488" },
+ { .compatible = "intel,mnh-spi", },
{},
};
MODULE_DEVICE_TABLE(of, spidev_dt_ids);
diff --git a/drivers/spmi/spmi-pmic-arb.c b/drivers/spmi/spmi-pmic-arb.c
index 6199523..a7c616f 100644
--- a/drivers/spmi/spmi-pmic-arb.c
+++ b/drivers/spmi/spmi-pmic-arb.c
@@ -569,7 +569,7 @@ static void periph_interrupt(struct spmi_pmic_arb *pa, u16 apid)
}
}
-static void pmic_arb_chained_irq(struct irq_desc *desc)
+static bool pmic_arb_chained_irq(struct irq_desc *desc)
{
struct spmi_pmic_arb *pa = irq_desc_get_handler_data(desc);
struct irq_chip *chip = irq_desc_get_chip(desc);
@@ -626,8 +626,8 @@ static void pmic_arb_chained_irq(struct irq_desc *desc)
}
}
}
-
chained_irq_exit(chip, desc);
+ return true;
}
static void qpnpint_irq_ack(struct irq_data *d)
diff --git a/drivers/staging/Kconfig b/drivers/staging/Kconfig
index 58a7b35..541cc33 100644
--- a/drivers/staging/Kconfig
+++ b/drivers/staging/Kconfig
@@ -106,4 +106,10 @@
source "drivers/staging/vc04_services/Kconfig"
+source "drivers/staging/bcm15602/Kconfig"
+
+source "drivers/staging/s2mpb04/Kconfig"
+
+source "drivers/staging/wlan_mac/Kconfig"
+
endif # STAGING
diff --git a/drivers/staging/Makefile b/drivers/staging/Makefile
index 2fa9745..c78fa86 100644
--- a/drivers/staging/Makefile
+++ b/drivers/staging/Makefile
@@ -42,3 +42,6 @@
obj-$(CONFIG_KS7010) += ks7010/
obj-$(CONFIG_GREYBUS) += greybus/
obj-$(CONFIG_BCM2708_VCHIQ) += vc04_services/
+obj-$(CONFIG_BCM15602) += bcm15602/
+obj-$(CONFIG_S2MPB04) += s2mpb04/
+obj-$(CONFIG_GOOGLE_WLAN_MAC) += wlan_mac/
diff --git a/drivers/staging/android/ashmem.c b/drivers/staging/android/ashmem.c
index f4ffac4..5af176b 100644
--- a/drivers/staging/android/ashmem.c
+++ b/drivers/staging/android/ashmem.c
@@ -383,6 +383,12 @@ static int ashmem_mmap(struct file *file, struct vm_area_struct *vma)
goto out;
}
+ /* requested mapping size larger than object size */
+ if (vma->vm_end - vma->vm_start > PAGE_ALIGN(asma->size)) {
+ ret = -EINVAL;
+ goto out;
+ }
+
/* requested protection bits must match our allowed protection mask */
if (unlikely((vma->vm_flags & ~calc_vm_prot_bits(asma->prot_mask, 0)) &
calc_vm_prot_bits(PROT_MASK, 0))) {
diff --git a/drivers/staging/android/ion/Makefile b/drivers/staging/android/ion/Makefile
index f3c8d51..c9f5bde 100644
--- a/drivers/staging/android/ion/Makefile
+++ b/drivers/staging/android/ion/Makefile
@@ -12,3 +12,7 @@
obj-$(CONFIG_ION_HISI) += hisilicon/
obj-$(CONFIG_ION_MSM) += msm/
+
+#TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-enum-conversion \
diff --git a/drivers/staging/android/ion/ion.c b/drivers/staging/android/ion/ion.c
index 787a6424..296e87a 100644
--- a/drivers/staging/android/ion/ion.c
+++ b/drivers/staging/android/ion/ion.c
@@ -47,6 +47,15 @@
#include "ion_priv.h"
#include "compat_ion.h"
+/*
+ * For fragmentation analysis,
+ * Group allocations of order 10 and above together
+ */
+#define ION_DEBUG_MAX_ORDER 10
+#define ION_DEBUG_ROW_HEADER " " \
+ "4K 8K 16K 32K 64K 128K 256K " \
+ "512K 1M 2M 4M >=8M"
+
/**
* struct ion_device - the metadata of the ion device node
* @dev: the actual misc device
@@ -265,6 +274,8 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,
mutex_lock(&dev->buffer_lock);
ion_buffer_add(dev, buffer);
mutex_unlock(&dev->buffer_lock);
+ trace_ion_heap_grow(heap->name, len,
+ atomic_long_read(&heap->total_allocated));
atomic_long_add(len, &heap->total_allocated);
return buffer;
@@ -279,10 +290,15 @@ static struct ion_buffer *ion_buffer_create(struct ion_heap *heap,
void ion_buffer_destroy(struct ion_buffer *buffer)
{
- if (WARN_ON(buffer->kmap_cnt > 0))
+ if (buffer->kmap_cnt > 0) {
+ pr_warn_once("%s: buffer still mapped in the kernel\n",
+ __func__);
buffer->heap->ops->unmap_kernel(buffer->heap, buffer);
+ }
buffer->heap->ops->unmap_dma(buffer->heap, buffer);
+ trace_ion_heap_shrink(buffer->heap->name, buffer->size,
+ atomic_long_read(&buffer->heap->total_allocated));
atomic_long_sub(buffer->size, &buffer->heap->total_allocated);
buffer->heap->ops->free(buffer);
vfree(buffer->pages);
@@ -1710,8 +1726,28 @@ static const struct file_operations ion_fops = {
.compat_ioctl = compat_ion_ioctl,
};
-static size_t ion_debug_heap_total(struct ion_client *client,
- unsigned int id)
+static void ion_debug_update_orders(struct ion_buffer *buffer,
+ size_t *orders)
+{
+ struct scatterlist *sg;
+ struct sg_table *table = buffer->sg_table;
+ int i;
+
+ if (!table)
+ return;
+
+ for_each_sg(table->sgl, sg, table->nents, i) {
+ int order = get_order(sg->length);
+
+ if (order > ION_DEBUG_MAX_ORDER)
+ orders[ION_DEBUG_MAX_ORDER + 1] += sg->length;
+ else
+ orders[order]++;
+ }
+}
+
+static size_t ion_debug_heap_totals(struct ion_client *client,
+ unsigned int id, size_t *orders)
{
size_t size = 0;
struct rb_node *n;
@@ -1721,8 +1757,10 @@ static size_t ion_debug_heap_total(struct ion_client *client,
struct ion_handle *handle = rb_entry(n,
struct ion_handle,
node);
- if (handle->buffer->heap->id == id)
+ if (handle->buffer->heap->id == id) {
size += handle->buffer->size;
+ ion_debug_update_orders(handle->buffer, orders);
+ }
}
mutex_unlock(&client->lock);
return size;
@@ -1840,14 +1878,18 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)
size_t total_size = 0;
size_t total_orphaned_size = 0;
- seq_printf(s, "%16s %16s %16s\n", "client", "pid", "size");
- seq_puts(s, "----------------------------------------------------\n");
+ seq_printf(s, "%16s %16s %16s %16s\n", "client", "pid", "size",
+ "page counts");
+ seq_puts(s, "--------------------------------------------------"
+ ION_DEBUG_ROW_HEADER "\n");
down_read(&dev->lock);
for (n = rb_first(&dev->clients); n; n = rb_next(n)) {
struct ion_client *client = rb_entry(n, struct ion_client,
node);
- size_t size = ion_debug_heap_total(client, heap->id);
+ int i;
+ size_t orders[ION_DEBUG_MAX_ORDER + 2] = {};
+ size_t size = ion_debug_heap_totals(client, heap->id, orders);
if (!size)
continue;
@@ -1855,12 +1897,17 @@ static int ion_debug_heap_show(struct seq_file *s, void *unused)
char task_comm[TASK_COMM_LEN];
get_task_comm(task_comm, client->task);
- seq_printf(s, "%16s %16u %16zu\n", task_comm,
+ seq_printf(s, "%16s %16u %16zu ", task_comm,
client->pid, size);
} else {
- seq_printf(s, "%16s %16u %16zu\n", client->name,
+ seq_printf(s, "%16s %16u %16zu ", client->name,
client->pid, size);
}
+ for (i = 0; i < ION_DEBUG_MAX_ORDER + 1; i++)
+ seq_printf(s, "%8zu ", orders[i]);
+ seq_printf(s, "%8zuM", orders[i] / (1024 * 1024));
+ seq_puts(s, "\n");
+
}
up_read(&dev->lock);
diff --git a/drivers/staging/android/ion/ion_page_pool.c b/drivers/staging/android/ion/ion_page_pool.c
index 38d4175..d182b15 100644
--- a/drivers/staging/android/ion/ion_page_pool.c
+++ b/drivers/staging/android/ion/ion_page_pool.c
@@ -41,6 +41,9 @@ static void *ion_page_pool_alloc_pages(struct ion_page_pool *pool)
goto error_free_pages;
ion_page_pool_alloc_set_cache_policy(pool, page);
+ mod_node_page_state(page_pgdat(page), NR_ION_HEAP,
+ 1 << pool->order);
+ mm_event_count(MM_KERN_ALLOC, 1 << pool->order);
return page;
error_free_pages:
@@ -53,6 +56,8 @@ static void ion_page_pool_free_pages(struct ion_page_pool *pool,
{
ion_page_pool_free_set_cache_policy(pool, page);
__free_pages(page, pool->order);
+ mod_node_page_state(page_pgdat(page), NR_ION_HEAP,
+ -(1 << pool->order));
}
static int ion_page_pool_add(struct ion_page_pool *pool, struct page *page)
@@ -65,6 +70,9 @@ static int ion_page_pool_add(struct ion_page_pool *pool, struct page *page)
list_add_tail(&page->lru, &pool->low_items);
pool->low_count++;
}
+
+ mod_node_page_state(page_pgdat(page), NR_INDIRECTLY_RECLAIMABLE_BYTES,
+ (1 << (PAGE_SHIFT + pool->order)));
mutex_unlock(&pool->mutex);
return 0;
}
@@ -84,6 +92,8 @@ static struct page *ion_page_pool_remove(struct ion_page_pool *pool, bool high)
}
list_del(&page->lru);
+ mod_node_page_state(page_pgdat(page), NR_INDIRECTLY_RECLAIMABLE_BYTES,
+ -(1 << (PAGE_SHIFT + pool->order)));
return page;
}
diff --git a/drivers/staging/android/ion/ion_system_secure_heap.c b/drivers/staging/android/ion/ion_system_secure_heap.c
index cf86ea2..4bd2079 100644
--- a/drivers/staging/android/ion/ion_system_secure_heap.c
+++ b/drivers/staging/android/ion/ion_system_secure_heap.c
@@ -49,10 +49,10 @@ struct prefetch_info {
static bool is_cp_flag_present(unsigned long flags)
{
- return flags && (ION_FLAG_CP_TOUCH ||
- ION_FLAG_CP_BITSTREAM ||
- ION_FLAG_CP_PIXEL ||
- ION_FLAG_CP_NON_PIXEL ||
+ return flags & (ION_FLAG_CP_TOUCH |
+ ION_FLAG_CP_BITSTREAM |
+ ION_FLAG_CP_PIXEL |
+ ION_FLAG_CP_NON_PIXEL |
ION_FLAG_CP_CAMERA);
}
diff --git a/drivers/staging/android/ion/msm/Makefile b/drivers/staging/android/ion/msm/Makefile
index c4c01c4..11fca27 100644
--- a/drivers/staging/android/ion/msm/Makefile
+++ b/drivers/staging/android/ion/msm/Makefile
@@ -2,3 +2,7 @@
ifdef CONFIG_COMPAT
obj-y += compat_msm_ion.o
endif
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-enum-conversion \
diff --git a/drivers/staging/android/ion/msm/msm_ion.c b/drivers/staging/android/ion/msm/msm_ion.c
index 656eb49..6910164 100644
--- a/drivers/staging/android/ion/msm/msm_ion.c
+++ b/drivers/staging/android/ion/msm/msm_ion.c
@@ -38,6 +38,25 @@
#include "compat_msm_ion.h"
#include <soc/qcom/secure_buffer.h>
+#ifdef CONFIG_CFI_CLANG
+static inline void __cfi_dma_flush_area(const void *p, size_t s)
+{
+ __dma_flush_area(p, s);
+}
+static inline void __cfi_dma_inv_area(const void *p, size_t s)
+{
+ __dma_inv_area(p, s);
+}
+static inline void __cfi_dma_clean_area(const void *p, size_t s)
+{
+ __dma_clean_area(p, s);
+}
+
+#define __dma_flush_area __cfi_dma_flush_area
+#define __dma_inv_area __cfi_dma_inv_area
+#define __dma_clean_area __cfi_dma_clean_area
+#endif
+
#define ION_COMPAT_STR "qcom,msm-ion"
static struct ion_device *idev;
@@ -75,6 +94,10 @@ static struct ion_heap_desc ion_heap_meta[] = {
.name = ION_MM_FIRMWARE_HEAP_NAME,
},
{
+ .id = ION_GOOGLE_HEAP_ID,
+ .name = ION_GOOGLE_HEAP_NAME,
+ },
+ {
.id = ION_CP_MFC_HEAP_ID,
.name = ION_MFC_HEAP_NAME,
.permission_type = IPT_TYPE_MFC_SHAREDMEM,
diff --git a/drivers/staging/android/lowmemorykiller.c b/drivers/staging/android/lowmemorykiller.c
index a1602e4..9a9208c 100644
--- a/drivers/staging/android/lowmemorykiller.c
+++ b/drivers/staging/android/lowmemorykiller.c
@@ -101,7 +101,6 @@ static unsigned long lowmem_count(struct shrinker *s,
static atomic_t shift_adj = ATOMIC_INIT(0);
static short adj_max_shift = 353;
-module_param_named(adj_max_shift, adj_max_shift, short, 0644);
/* User knob to enable/disable adaptive lmk feature */
static int enable_adaptive_lmk;
diff --git a/drivers/staging/android/samples/ion/dma_buf_info.c b/drivers/staging/android/samples/ion/dma_buf_info.c
new file mode 100644
index 0000000..18430d0
--- /dev/null
+++ b/drivers/staging/android/samples/ion/dma_buf_info.c
@@ -0,0 +1,296 @@
+/*
+ * Copyright (C) 2018 The Android Open Source Project
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <errno.h>
+#include <error.h>
+#include <fcntl.h>
+#include <linux/dma-buf.h>
+#include <linux/ion.h>
+#include <linux/ion_4.12.h>
+#include <linux/limits.h>
+#include <linux/types.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+struct dmabuf_info {
+ /**
+ * The total number of references held across the system.
+ */
+ unsigned int count;
+ /**
+ * The size of the buffer
+ */
+ size_t size;
+ /**
+ * A NULL-terminated name for the buffer.
+ */
+ char *name;
+};
+
+struct memoryinfo_buffer {
+ /**
+ * The buffer's unique inode.
+ */
+ ino_t ino;
+ /**
+ * The buffer's size, in bytes.
+ */
+ size_t size;
+ /**
+ * The total number of references held across the system.
+ */
+ unsigned int count;
+ /**
+ * Number of elements in \ref refs
+ */
+ size_t refs_count;
+ /**
+ * A NULL-terminated name for the buffer.
+ */
+ char *name;
+};
+
+enum ion_version {
+ ION_VERSION_UNKNOWN,
+ ION_VERSION_MODERN,
+ ION_VERSION_LEGACY
+};
+
+int parse_dmabuf_debug_fs(struct dmabuf_info *info, uint64_t inode_number)
+{
+ FILE *fp;
+ char *line = NULL;
+ size_t len = 0;
+ ssize_t read;
+
+ fp = fopen("/sys/kernel/debug/dma_buf/bufinfo", "r");
+ if (!fp)
+ return -ENOENT;
+
+ while ((read = getline(&line, &len, fp)) != -1) {
+ long count;
+ size_t size;
+ unsigned long ino;
+ // format: size flags mode count exp_name ino name
+ int matched = sscanf(line, "%zu %*x %*x %ld %*s %lu %ms",
+ &size, &count, &ino, &info->name);
+
+ if (matched > 2 && ino == inode_number) {
+ info->count = count;
+ info->size = size;
+ free(line);
+ return 0;
+ }
+ }
+ free(line);
+ return -ENOENT;
+}
+
+int parse_dmabuf_fd(pid_t pid, int fd, struct memoryinfo_buffer *buf)
+{
+ FILE *fp;
+ char *line = NULL;
+ ssize_t read;
+ struct stat s;
+ size_t len = 0;
+ char fdinfo_path[100];
+ int err = fstat(fd, &s);
+
+ if (err < 0)
+ error(1, errno, "fstat(%d) failed", fd);
+
+ buf->ino = s.st_ino;
+ buf->size = s.st_blocks * 512;
+ if (!buf->size)
+ error(1, EINVAL, "unknown size of the fd: %d", fd);
+
+ snprintf(fdinfo_path, 100, "/proc/%d/fdinfo/%d", pid, fd);
+
+ fp = fopen(fdinfo_path, "r");
+ if (!fp)
+ error(1, ENOENT, "cannot open file at: %s", fdinfo_path);
+
+ while ((read = getline(&line, &len, fp)) != -1) {
+ int matched = sscanf(line, "count:\t%u", &buf->count);
+
+ matched = sscanf(line, "name:\t%m[^\n]", &buf->name);
+ }
+ free(line);
+ return 0;
+}
+
+int memoryinfo_print(pid_t pid, int fd)
+{
+ struct dmabuf_info dmabuf;
+ struct memoryinfo_buffer buf;
+ int ret = parse_dmabuf_fd(pid, fd, &buf);
+
+ ret = parse_dmabuf_debug_fs(&dmabuf, buf.ino);
+
+ printf("dma_buf information parsed from fdinfo:\n");
+ printf("inode: %lu, size: %zu, ref_count: %u, name: %s\n\n",
+ buf.ino, buf.size, buf.count, buf.name);
+ printf("dma_buf information parsed from dma_buf debug fs:\n");
+ printf("inode: %lu, size: %zu, ref_count: %u, name: %s\n\n",
+ buf.ino, dmabuf.size, dmabuf.count, dmabuf.name);
+ if (dmabuf.size != buf.size || strcmp(dmabuf.name, buf.name))
+ error(1, EINVAL, "dma_buf information is different!");
+ return 0;
+}
+
+int ion_ioctl(int fd, int req, void *arg)
+{
+ int ret = ioctl(fd, req, arg);
+
+ if (ret < 0)
+ printf("ioctl %x failed with code %d: %s", req, ret,
+ strerror(errno));
+ return ret;
+}
+
+int ion_free(int fd, ion_user_handle_t handle)
+{
+ struct ion_handle_data data = {
+ .handle = handle,
+ };
+ return ion_ioctl(fd, ION_IOC_FREE, &data);
+}
+
+int get_ion_version(int fd)
+{
+ int err = ion_free(fd, (ion_user_handle_t)NULL);
+
+ return (err == -ENOTTY) ? ION_VERSION_MODERN : ION_VERSION_LEGACY;
+}
+
+int ion_alloc(int fd, size_t len, size_t align, unsigned int heap_mask,
+ unsigned int flags, ion_user_handle_t *handle)
+{
+ int ret = 0;
+
+ if (!handle)
+ return -EINVAL;
+
+ struct ion_allocation_data data = {
+ .len = len,
+ .align = align,
+ .heap_id_mask = heap_mask,
+ .flags = flags,
+ };
+
+ ret = ion_ioctl(fd, ION_IOC_ALLOC, &data);
+ if (ret < 0)
+ return ret;
+
+ *handle = data.handle;
+
+ return ret;
+}
+
+int ion_share(int fd, ion_user_handle_t handle, int *share_fd)
+{
+ int ret;
+ struct ion_fd_data data = {
+ .handle = handle,
+ };
+
+ if (!share_fd)
+ return -EINVAL;
+
+ ret = ion_ioctl(fd, ION_IOC_SHARE, &data);
+ if (ret < 0)
+ return ret;
+ if (data.fd < 0)
+ error(1, EINVAL, "share ioctl returned negative fd");
+ *share_fd = data.fd;
+ return ret;
+}
+
+int ion_alloc_fd(int fd, size_t len, size_t align, unsigned int heap_mask,
+ unsigned int flags, int *handle_fd, int version)
+{
+ ion_user_handle_t handle;
+ int ret;
+
+ if (version == ION_VERSION_MODERN) {
+ struct ion_new_allocation_data data = {
+ .len = len,
+ .heap_id_mask = heap_mask,
+ .flags = flags,
+ };
+
+ ret = ion_ioctl(fd, ION_IOC_NEW_ALLOC, &data);
+ if (ret < 0)
+ return ret;
+ *handle_fd = data.fd;
+ } else {
+ ret = ion_alloc(fd, len, align, heap_mask, flags, &handle);
+ if (ret < 0)
+ return ret;
+ ret = ion_share(fd, handle, handle_fd);
+ ion_free(fd, handle);
+ }
+ return ret;
+}
+
+int alloc_dma_bufs(const char *name)
+{
+ int dma_buf_fd, version, err;
+ int ion_fd = open("/dev/ion", O_RDONLY | O_CLOEXEC);
+
+ if (ion_fd < 0)
+ error(1, errno, "open /dev/ion failed");
+ version = get_ion_version(ion_fd);
+ err = ion_alloc_fd(ion_fd, 1024, 0, ION_HEAP_SYSTEM_MASK, 0,
+ &dma_buf_fd, version);
+ if (err < 0)
+ error(1, errno, "allocate ion buffer failed");
+
+ err = ioctl(dma_buf_fd, DMA_BUF_SET_NAME, name);
+ if (err < 0)
+ error(1, errno, "set dma_buf name failed");
+
+ return dma_buf_fd;
+}
+
+int free_dma_bufs(fd)
+{
+ close(fd);
+ return 0;
+}
+
+int main(void)
+{
+ int pid, fd, ret;
+
+ printf("Allocate dma_bufs\n\n");
+
+ fd = alloc_dma_bufs("TEST");
+ if (fd < 0)
+ error(1, errno, "alloc dma_buf failed");
+
+ pid = getpid();
+ printf("print memoryinfo for pid: %d\n\n", pid);
+
+ ret = memoryinfo_print(pid, fd);
+
+ ret = free_dma_bufs(fd);
+ return 0;
+}
diff --git a/drivers/staging/android/uapi/msm_ion.h b/drivers/staging/android/uapi/msm_ion.h
index a6672ff..bcb05af 100644
--- a/drivers/staging/android/uapi/msm_ion.h
+++ b/drivers/staging/android/uapi/msm_ion.h
@@ -47,6 +47,7 @@ enum ion_heap_ids {
ION_AUDIO_HEAP_ID = 28,
ION_MM_FIRMWARE_HEAP_ID = 29,
+ ION_GOOGLE_HEAP_ID = 30,
ION_HEAP_ID_RESERVED = 31 /** Bit reserved for ION_FLAG_SECURE flag */
};
@@ -138,6 +139,7 @@ enum cp_mem_usage {
#define ION_SECURE_CARVEOUT_HEAP_NAME "secure_carveout"
#define ION_WB_HEAP_NAME "wb"
#define ION_MM_FIRMWARE_HEAP_NAME "mm_fw"
+#define ION_GOOGLE_HEAP_NAME "easel_mem"
#define ION_PIL1_HEAP_NAME "pil_1"
#define ION_PIL2_HEAP_NAME "pil_2"
#define ION_QSECOM_HEAP_NAME "qsecom"
diff --git a/drivers/staging/bcm15602/Kconfig b/drivers/staging/bcm15602/Kconfig
new file mode 100644
index 0000000..5a25a83
--- /dev/null
+++ b/drivers/staging/bcm15602/Kconfig
@@ -0,0 +1,7 @@
+config BCM15602
+ tristate "BCM15602 regulator"
+ depends on I2C
+ select REGMAP_I2C
+ help
+ This driver controls the BCM15602 voltage regulator via I2C.
+
diff --git a/drivers/staging/bcm15602/Makefile b/drivers/staging/bcm15602/Makefile
new file mode 100644
index 0000000..5aafdf1
--- /dev/null
+++ b/drivers/staging/bcm15602/Makefile
@@ -0,0 +1,5 @@
+#
+# Makefile for bcm15602-regulator.c
+#
+
+obj-$(CONFIG_BCM15602) += bcm15602-regulator.o bcm15602-regulator-sysfs.o bcm15602-gpio.o bcm15602-thermal.o
diff --git a/drivers/staging/bcm15602/bcm15602-gpio.c b/drivers/staging/bcm15602/bcm15602-gpio.c
new file mode 100644
index 0000000..f607c0a
--- /dev/null
+++ b/drivers/staging/bcm15602/bcm15602-gpio.c
@@ -0,0 +1,149 @@
+/*
+ * Copyright (C) 2016 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/bitops.h>
+#include <linux/gpio/driver.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+
+#include "bcm15602-regulator.h"
+
+#define DRIVER_NAME "bcm15602-gpio"
+
+struct bcm15602_gpio {
+ struct gpio_chip gpio_chip;
+ struct bcm15602_chip *bcm15602;
+ unsigned long int pin_used;
+};
+
+static inline struct bcm15602_gpio *to_bcm15602_gpio
+ (struct gpio_chip *_gpio_chip)
+{
+ return container_of(_gpio_chip, struct bcm15602_gpio, gpio_chip);
+}
+
+static int bcm15602_gpio_request(struct gpio_chip *gpio_chip, unsigned offset)
+{
+ struct bcm15602_gpio *bcm15602_gpio = to_bcm15602_gpio(gpio_chip);
+
+ if (test_and_set_bit(offset, &bcm15602_gpio->pin_used))
+ return -EBUSY;
+ else
+ return 0;
+}
+
+static void bcm15602_gpio_free(struct gpio_chip *gpio_chip, unsigned offset)
+{
+ struct bcm15602_gpio *bcm15602_gpio = to_bcm15602_gpio(gpio_chip);
+
+ clear_bit(offset, &bcm15602_gpio->pin_used);
+}
+
+static int bcm15602_gpio_direction_input(struct gpio_chip *gpio_chip,
+ unsigned offset)
+{
+ /* pins are hard-coded as outputs */
+ return -EIO;
+}
+
+static int bcm15602_gpio_get(struct gpio_chip *gpio_chip, unsigned offset)
+{
+ struct bcm15602_gpio *bcm15602_gpio = to_bcm15602_gpio(gpio_chip);
+ struct bcm15602_chip *bcm15602 = bcm15602_gpio->bcm15602;
+ u8 byte;
+
+ bcm15602_read_byte(bcm15602, BCM15602_REG_SYS_GPIO_IN, &byte);
+ return byte & (1 << offset);
+}
+
+static void bcm15602_gpio_set(struct gpio_chip *gpio_chip, unsigned offset,
+ int value)
+{
+ struct bcm15602_gpio *bcm15602_gpio = to_bcm15602_gpio(gpio_chip);
+ struct bcm15602_chip *bcm15602 = bcm15602_gpio->bcm15602;
+
+ bcm15602_update_bits(bcm15602, BCM15602_REG_SYS_GPIO_OUT_CTRL,
+ (1 << offset), ((value ? 1 : 0) << offset));
+}
+
+static int bcm15602_gpio_direction_output(struct gpio_chip *gpio_chip,
+ unsigned offset, int value)
+{
+ /* pins are already configured as outputs */
+ bcm15602_gpio_set(gpio_chip, offset, value);
+ return 0;
+}
+
+static const struct gpio_chip bcm15602_gpio_chip = {
+ .label = DRIVER_NAME,
+ .owner = THIS_MODULE,
+ .request = bcm15602_gpio_request,
+ .free = bcm15602_gpio_free,
+ .direction_input = bcm15602_gpio_direction_input,
+ .get = bcm15602_gpio_get,
+ .direction_output = bcm15602_gpio_direction_output,
+ .set = bcm15602_gpio_set,
+ .base = -1,
+ .ngpio = BCM15602_NUM_GPIOS,
+ .can_sleep = 1,
+};
+
+static int bcm15602_gpio_probe(struct platform_device *pdev)
+{
+ struct bcm15602_chip *bcm15602 = dev_get_drvdata(pdev->dev.parent);
+ struct bcm15602_gpio *bcm15602_gpio;
+
+ bcm15602_gpio = devm_kzalloc(&pdev->dev, sizeof(*bcm15602_gpio),
+ GFP_KERNEL);
+ if (!bcm15602_gpio)
+ return -ENOMEM;
+
+ bcm15602_gpio->bcm15602 = bcm15602;
+ bcm15602_gpio->gpio_chip = bcm15602_gpio_chip;
+ bcm15602_gpio->gpio_chip.parent = &pdev->dev;
+
+ platform_set_drvdata(pdev, bcm15602_gpio);
+
+ return gpiochip_add(&bcm15602_gpio->gpio_chip);
+}
+
+static int bcm15602_gpio_remove(struct platform_device *pdev)
+{
+ struct bcm15602_gpio *bcm15602_gpio = platform_get_drvdata(pdev);
+
+ gpiochip_remove(&bcm15602_gpio->gpio_chip);
+ return 0;
+}
+
+static const struct of_device_id bcm15602_gpio_of_match[] = {
+ { .compatible = "brcm,bcm15602-gpio", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, bcm15602_gpio_of_match);
+
+static struct platform_driver bcm15602_gpio_driver = {
+ .probe = bcm15602_gpio_probe,
+ .remove = bcm15602_gpio_remove,
+ .driver = {
+ .name = DRIVER_NAME,
+ .of_match_table = bcm15602_gpio_of_match,
+ },
+};
+module_platform_driver(bcm15602_gpio_driver);
+
+MODULE_AUTHOR("Trevor Bunker <trevorbunker@google.com>");
+MODULE_DESCRIPTION("BCM15602 GPIO Driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/staging/bcm15602/bcm15602-regulator-sysfs.c b/drivers/staging/bcm15602/bcm15602-regulator-sysfs.c
new file mode 100644
index 0000000..375e09d
--- /dev/null
+++ b/drivers/staging/bcm15602/bcm15602-regulator-sysfs.c
@@ -0,0 +1,335 @@
+/*
+ * Copyright (C) 2016 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/device.h>
+#include <linux/module.h>
+
+#include "bcm15602-regulator.h"
+
+static ssize_t bcm15602_attr_show_asr_curr
+ (struct device *dev, struct device_attribute *mattr, char *data);
+static ssize_t bcm15602_attr_show_sdsr_curr
+ (struct device *dev, struct device_attribute *mattr, char *data);
+static ssize_t bcm15602_attr_show_sdldo_curr
+ (struct device *dev, struct device_attribute *mattr, char *data);
+static ssize_t bcm15602_attr_show_ioldo_curr
+ (struct device *dev, struct device_attribute *mattr, char *data);
+static ssize_t bcm15602_attr_show_vbat
+ (struct device *dev, struct device_attribute *mattr, char *data);
+static ssize_t bcm15602_attr_show_temperature
+ (struct device *dev, struct device_attribute *mattr, char *data);
+static ssize_t bcm15602_attr_show_total_power
+ (struct device *dev, struct device_attribute *mattr, char *data);
+static ssize_t bcm15602_attr_show_hk_status
+ (struct device *dev, struct device_attribute *mattr, char *data);
+static ssize_t bcm15602_attr_show_asr_vsel
+ (struct device *dev, struct device_attribute *mattr, char *data);
+static ssize_t bcm15602_attr_store_asr_vsel
+ (struct device *dev, struct device_attribute *mattr, const char *data,
+ size_t count);
+static ssize_t bcm15602_attr_show_asr_dual_phase
+ (struct device *dev, struct device_attribute *mattr, char *data);
+static ssize_t bcm15602_attr_store_asr_dual_phase
+ (struct device *dev, struct device_attribute *mattr, const char *data,
+ size_t count);
+
+DEVICE_ATTR(asr_curr, 0440, bcm15602_attr_show_asr_curr, NULL);
+DEVICE_ATTR(sdsr_curr, 0440, bcm15602_attr_show_sdsr_curr, NULL);
+DEVICE_ATTR(sdldo_curr, 0440, bcm15602_attr_show_sdldo_curr, NULL);
+DEVICE_ATTR(ioldo_curr, 0440, bcm15602_attr_show_ioldo_curr, NULL);
+DEVICE_ATTR(vbat, 0440, bcm15602_attr_show_vbat, NULL);
+DEVICE_ATTR(temperature, 0440, bcm15602_attr_show_temperature, NULL);
+DEVICE_ATTR(total_power, 0440, bcm15602_attr_show_total_power, NULL);
+DEVICE_ATTR(hk_status, 0440, bcm15602_attr_show_hk_status, NULL);
+DEVICE_ATTR(asr_vsel, S_IWUSR | S_IRUGO, bcm15602_attr_show_asr_vsel,
+ bcm15602_attr_store_asr_vsel);
+DEVICE_ATTR(asr_dual_phase, S_IWUSR | S_IRUGO,
+ bcm15602_attr_show_asr_dual_phase,
+ bcm15602_attr_store_asr_dual_phase);
+
+static struct attribute *bcm15602_attrs[] = {
+ &dev_attr_asr_curr.attr,
+ &dev_attr_sdsr_curr.attr,
+ &dev_attr_sdldo_curr.attr,
+ &dev_attr_ioldo_curr.attr,
+ &dev_attr_vbat.attr,
+ &dev_attr_temperature.attr,
+ &dev_attr_total_power.attr,
+ &dev_attr_hk_status.attr,
+ &dev_attr_asr_vsel.attr,
+ &dev_attr_asr_dual_phase.attr,
+ NULL
+};
+
+static const struct attribute_group bcm15602_attr_group = {
+ .attrs = bcm15602_attrs,
+};
+
+/* asr_curr is in uA */
+static int bcm15602_get_asr_curr(struct bcm15602_chip *ddata, int *asr_curr)
+{
+ u16 mstr_curr_chan_data, slv_curr_chan_data;
+
+ bcm15602_read_adc_chan(ddata, BCM15602_ADC_ASR_MSTR_CURR,
+ &mstr_curr_chan_data);
+ bcm15602_read_adc_chan(ddata, BCM15602_ADC_ASR_SLV_CURR,
+ &slv_curr_chan_data);
+
+ *asr_curr = (mstr_curr_chan_data * BCM15602_ADC_SCALE_ASR_CURR) +
+ (slv_curr_chan_data * BCM15602_ADC_SCALE_ASR_CURR);
+
+ return 0;
+}
+
+/* sdsr_curr is in uA */
+static int bcm15602_get_sdsr_curr(struct bcm15602_chip *ddata, int *sdsr_curr)
+{
+ u16 chan_data;
+
+ bcm15602_read_adc_chan(ddata, BCM15602_ADC_SDSR_CURR, &chan_data);
+
+ *sdsr_curr = chan_data * BCM15602_ADC_SCALE_SDSR_CURR;
+
+ return 0;
+}
+
+/* sdldo_curr is in uA */
+static int bcm15602_get_sdldo_curr(struct bcm15602_chip *ddata, int *sdldo_curr)
+{
+ u16 chan_data;
+
+ bcm15602_read_adc_chan(ddata, BCM15602_ADC_SDLDO_CURR, &chan_data);
+
+ *sdldo_curr = chan_data * BCM15602_ADC_SCALE_SDLDO_CURR;
+
+ return 0;
+}
+
+/* ioldo_curr is in uA */
+static int bcm15602_get_ioldo_curr(struct bcm15602_chip *ddata, int *ioldo_curr)
+{
+ u16 chan_data;
+
+ bcm15602_read_adc_chan(ddata, BCM15602_ADC_IOLDO_CURR, &chan_data);
+
+ *ioldo_curr = chan_data * BCM15602_ADC_SCALE_IOLDO_CURR;
+
+ return 0;
+}
+
+static ssize_t bcm15602_attr_show_asr_curr(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ int asr_curr;
+
+ bcm15602_get_asr_curr(ddata, &asr_curr);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", asr_curr);
+}
+
+static ssize_t bcm15602_attr_show_sdsr_curr(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ int sdsr_curr;
+
+ bcm15602_get_sdsr_curr(ddata, &sdsr_curr);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", sdsr_curr);
+}
+
+static ssize_t bcm15602_attr_show_sdldo_curr(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ int sdldo_curr;
+
+ bcm15602_get_sdldo_curr(ddata, &sdldo_curr);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", sdldo_curr);
+}
+
+static ssize_t bcm15602_attr_show_ioldo_curr(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ int ioldo_curr;
+
+ bcm15602_get_ioldo_curr(ddata, &ioldo_curr);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", ioldo_curr);
+}
+
+static ssize_t bcm15602_attr_show_vbat(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ u16 chan_data;
+
+ bcm15602_read_adc_chan(ddata, BCM15602_ADC_VBAT, &chan_data);
+
+ return snprintf(data, PAGE_SIZE, "%d\n",
+ chan_data * BCM15602_ADC_SCALE_VBAT);
+}
+
+static ssize_t bcm15602_attr_show_temperature(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ u16 chan_data;
+
+ bcm15602_read_adc_chan(ddata, BCM15602_ADC_PTAT, &chan_data);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", PTAT_CODE_TO_TEMP(chan_data));
+}
+
+/* shows power in mW */
+static ssize_t bcm15602_attr_show_total_power(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ int asr_curr, sdsr_curr, sdldo_curr, ioldo_curr;
+ long int total_power;
+
+ bcm15602_get_asr_curr(ddata, &asr_curr);
+ bcm15602_get_sdsr_curr(ddata, &sdsr_curr);
+ bcm15602_get_sdldo_curr(ddata, &sdldo_curr);
+ bcm15602_get_ioldo_curr(ddata, &ioldo_curr);
+
+ total_power = (((long int)asr_curr) * 900 / 1000) +
+ (((long int)sdsr_curr) * 1100 / 1000) +
+ (((long int)sdldo_curr) * 1800 / 1000) +
+ (((long int)ioldo_curr) * 1800 / 1000);
+
+ return snprintf(data, PAGE_SIZE, "%ld\n", total_power);
+}
+
+static ssize_t bcm15602_attr_show_hk_status(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ u16 status;
+
+ status = ddata->hk_status;
+ ddata->hk_status &= ~status;
+
+ return snprintf(data, PAGE_SIZE, "0x%04x\n", status);
+}
+
+static ssize_t bcm15602_attr_show_asr_vsel(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ u8 reg_data;
+ int voctrl;
+
+ bcm15602_read_byte(ddata, BCM15602_REG_BUCK_ASR_VOCTRL, ®_data);
+ voctrl = reg_data & 0x7F;
+
+ return snprintf(data, PAGE_SIZE, "%d\n", 565 + voctrl * 5);
+}
+
+static ssize_t bcm15602_attr_store_asr_vsel(struct device *dev,
+ struct device_attribute *mattr,
+ const char *data, size_t count)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ int vsel, voctrl;
+
+ if (kstrtoint(data, 0, &vsel)) {
+ dev_err(dev, "%s: Not a valid int\n", __func__);
+ return -EINVAL;
+ }
+
+ if (vsel == 0) {
+ bcm15602_update_bits(ddata, BCM15602_REG_BUCK_ASR_CTRL0, 0x1,
+ 0x0);
+ } else if ((vsel < 565) || (vsel > 1200)) {
+ dev_err(dev, "%s: Not a valid voltage level, must be 0 or 565-1200\n",
+ __func__);
+ return -EINVAL;
+ }
+
+ voctrl = ((vsel - 565) + 4) / 5;
+ bcm15602_write_byte(ddata, BCM15602_REG_BUCK_ASR_VOCTRL, voctrl & 0x7F);
+ bcm15602_update_bits(ddata, BCM15602_REG_BUCK_ASR_CTRL0, 0x1, 0x1);
+
+ return count;
+}
+
+static ssize_t bcm15602_attr_show_asr_dual_phase(
+ struct device *dev, struct device_attribute *mattr, char *data)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ u8 reg_data;
+
+ bcm15602_read_byte(ddata, BCM15602_REG_BUCK_ASR_TSET_CTRL2, ®_data);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", (reg_data & 0x10) >> 4);
+}
+
+static ssize_t bcm15602_attr_store_asr_dual_phase(
+ struct device *dev, struct device_attribute *mattr, const char *data,
+ size_t count)
+{
+ struct bcm15602_chip *ddata = dev_get_drvdata(dev);
+ int val;
+
+ if (kstrtoint(data, 0, &val)) {
+ dev_err(dev, "%s: Not a valid int\n", __func__);
+ return -EINVAL;
+ }
+
+ if ((val != 0) && (val != 1)) {
+ dev_err(dev, "%s: Can only write 0 or 1\n", __func__);
+ return -EINVAL;
+ }
+
+ if (ddata->rev_id >= BCM15602_REV_A1) {
+ dev_warn(dev,
+ "%s: Skipping workaround for this silicon version\n",
+ __func__);
+ return count;
+ }
+
+ bcm15602_update_bits(ddata, BCM15602_REG_BUCK_ASR_TSET_CTRL2, 0x10,
+ (val & 0x1) << 4);
+
+ return count;
+}
+
+void bcm15602_config_sysfs(struct device *dev)
+{
+ int ret;
+
+ ret = sysfs_create_group(&dev->kobj, &bcm15602_attr_group);
+ if (ret < 0) {
+ dev_err(dev, "%s: could not create sysfs attributes (%d)\n",
+ __func__, ret);
+ }
+}
+EXPORT_SYMBOL_GPL(bcm15602_config_sysfs);
+
diff --git a/drivers/staging/bcm15602/bcm15602-regulator.c b/drivers/staging/bcm15602/bcm15602-regulator.c
new file mode 100644
index 0000000..e075d9c
--- /dev/null
+++ b/drivers/staging/bcm15602/bcm15602-regulator.c
@@ -0,0 +1,1403 @@
+/*
+ * Copyright (C) 2016 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+/* #define DEBUG */
+
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/i2c.h>
+#include <linux/interrupt.h>
+#include <linux/mfd/core.h>
+#include <linux/module.h>
+#include <linux/of_gpio.h>
+#include <linux/regmap.h>
+#include <linux/regulator/driver.h>
+#include <linux/regulator/machine.h>
+
+#include "bcm15602-regulator.h"
+
+#define DRIVER_NAME "bcm15602"
+
+/* defines the number of tries to repeat an I2C transaction */
+#define BCM15602_I2C_RETRY_COUNT 10
+
+/* defines the delay in ms for reset completion */
+#define BCM15602_PON_RESET_DELAY 35
+
+/* defines the timeout in jiffies for reset completion */
+#define BCM15602_PON_RESET_TIMEOUT msecs_to_jiffies(15)
+
+/* defines the number of retries for powering on */
+#define BCM15602_PON_RETRY_CNT 4
+
+/* defines the timeout in jiffies for reset completion after shutdown */
+#define BCM15602_SHUTDOWN_RESET_TIMEOUT msecs_to_jiffies(10000)
+
+/* defines the timeout in jiffies for ADC conversion completion */
+#define BCM15602_ADC_CONV_TIMEOUT msecs_to_jiffies(100)
+
+static int bcm15602_chip_init(struct bcm15602_chip *ddata);
+static int bcm15602_regulator_set_voltage(struct regulator_dev *rdev,
+ int min_uV, int max_uV,
+ unsigned *selector);
+static int bcm15602_regulator_get_voltage(struct regulator_dev *rdev);
+static int bcm15602_regulator_enable(struct regulator_dev *rdev);
+static int bcm15602_regulator_disable(struct regulator_dev *rdev);
+static int bcm15602_regulator_is_enabled(struct regulator_dev *rdev);
+static void bcm15602_enable_wdt(struct bcm15602_chip *ddata);
+static void bcm15602_disable_wdt(struct bcm15602_chip *ddata);
+
+static const struct mfd_cell bcm15602_devs[] = {
+ {
+ .name = "bcm15602-gpio",
+ .of_compatible = "brcm,bcm15602-gpio",
+ },
+ {
+ .name = "bcm15602-thermal",
+ .of_compatible = "brcm,bcm15602-thermal",
+ },
+};
+
+/* adc slot configurations */
+static const struct bcm15602_adc_ctrl adc_ctrls[] = {
+ [BCM15602_HK_ASR_MSTR_CURR_OVERI] = {
+ .state = BCM15602_ADC_STATE_ACTIVE,
+ .ovsp = BCM15602_ADC_OVSP_1,
+ .chan_num = BCM15602_ADC_ASR_MSTR_CURR,
+ .threshold = BCM15602_HK_TH_ASR_MSTR_CURR,
+ .ot_ut = BCM15602_ADC_OVER_TH,
+ .deb_en = 0,
+ .th_en = 1,
+ },
+ [BCM15602_HK_ASR_SLV_CURR_OVERI] = {
+ .state = BCM15602_ADC_STATE_ACTIVE,
+ .ovsp = BCM15602_ADC_OVSP_1,
+ .chan_num = BCM15602_ADC_ASR_SLV_CURR,
+ .threshold = BCM15602_HK_TH_ASR_SLV_CURR,
+ .ot_ut = BCM15602_ADC_OVER_TH,
+ .deb_en = 0,
+ .th_en = 1,
+ },
+ [BCM15602_HK_SDSR_CURR_OVERI] = {
+ .state = BCM15602_ADC_STATE_ACTIVE,
+ .ovsp = BCM15602_ADC_OVSP_1,
+ .chan_num = BCM15602_ADC_SDSR_CURR,
+ .threshold = BCM15602_HK_TH_SDSR_CURR,
+ .ot_ut = BCM15602_ADC_OVER_TH,
+ .deb_en = 0,
+ .th_en = 1,
+ },
+ [BCM15602_HK_SDLDO_CURR_OVERI] = {
+ .state = BCM15602_ADC_STATE_ACTIVE,
+ .ovsp = BCM15602_ADC_OVSP_1,
+ .chan_num = BCM15602_ADC_SDLDO_CURR,
+ .threshold = BCM15602_HK_TH_SDLDO_CURR,
+ .ot_ut = BCM15602_ADC_OVER_TH,
+ .deb_en = 0,
+ .th_en = 1,
+ },
+ [BCM15602_HK_IOLDO_CURR_OVERI] = {
+ .state = BCM15602_ADC_STATE_ACTIVE,
+ .ovsp = BCM15602_ADC_OVSP_1,
+ .chan_num = BCM15602_ADC_IOLDO_CURR,
+ .threshold = BCM15602_HK_TH_IOLDO_CURR,
+ .ot_ut = BCM15602_ADC_OVER_TH,
+ .deb_en = 0,
+ .th_en = 1,
+ },
+ [BCM15602_HK_VBAT_UNDERV0] = {
+ .state = BCM15602_ADC_STATE_ACTIVE,
+ .ovsp = BCM15602_ADC_OVSP_1,
+ .chan_num = BCM15602_ADC_VBAT,
+ .threshold = BCM15602_HK_TH_VBAT_UNDERV0,
+ .ot_ut = BCM15602_ADC_UNDER_TH,
+ .deb_en = 0,
+ .th_en = 1,
+ },
+ [BCM15602_HK_VBAT_UNDERV1] = {
+ .state = BCM15602_ADC_STATE_ACTIVE,
+ .ovsp = BCM15602_ADC_OVSP_1,
+ .chan_num = BCM15602_ADC_VBAT,
+ .threshold = BCM15602_HK_TH_VBAT_UNDERV1,
+ .ot_ut = BCM15602_ADC_UNDER_TH,
+ .deb_en = 0,
+ .th_en = 1,
+ },
+ [BCM15602_HK_VBAT_UNDERV2] = {
+ .state = BCM15602_ADC_STATE_ACTIVE,
+ .ovsp = BCM15602_ADC_OVSP_1,
+ .chan_num = BCM15602_ADC_VBAT,
+ .threshold = BCM15602_HK_TH_VBAT_UNDERV2,
+ .ot_ut = BCM15602_ADC_UNDER_TH,
+ .deb_en = 0,
+ .th_en = 1,
+ },
+ [BCM15602_HK_PTAT_OVERT] = {
+ .state = BCM15602_ADC_STATE_ACTIVE,
+ .ovsp = BCM15602_ADC_OVSP_1,
+ .chan_num = BCM15602_ADC_PTAT,
+ .threshold = BCM15602_HK_TH_PTAT_OVERT,
+ .ot_ut = BCM15602_ADC_OVER_TH,
+ .deb_en = 0,
+ .th_en = 1,
+ },
+ [BCM15602_HK_PTAT_UNDERT] = {
+ .state = BCM15602_ADC_STATE_ACTIVE,
+ .ovsp = BCM15602_ADC_OVSP_1,
+ .chan_num = BCM15602_ADC_PTAT,
+ .threshold = BCM15602_HK_TH_PTAT_UNDERT,
+ .ot_ut = BCM15602_ADC_UNDER_TH,
+ .deb_en = 0,
+ .th_en = 1,
+ },
+};
+
+static const struct regmap_config bcm15602_regmap_config = {
+ .reg_bits = 8,
+ .val_bits = 8,
+};
+
+static struct regulator_ops bcm15602_regulator_ops = {
+ .list_voltage = regulator_list_voltage_table,
+ .map_voltage = regulator_map_voltage_ascend,
+ .set_voltage = bcm15602_regulator_set_voltage,
+ .get_voltage = bcm15602_regulator_get_voltage,
+ .enable = bcm15602_regulator_enable,
+ .disable = bcm15602_regulator_disable,
+ .is_enabled = bcm15602_regulator_is_enabled,
+};
+
+/* No support for DVS so just a single voltage level */
+static const unsigned int bcm15602_ldo_vtbl[] = { 1800000 };
+static const unsigned int bcm15602_asr_vtbl[] = { 900000, 1000000 };
+static const unsigned int bcm15602_sdsr_vtbl[] = { 1100000 };
+
+static struct regulator_desc
+ bcm15602_regulator_desc[BCM15602_NUM_REGULATORS] = {
+ [BCM15602_ID_SDLDO] = {
+ .name = BCM15602_REGLTR_NAME_SDLDO,
+ .id = BCM15602_ID_SDLDO,
+ .ops = &bcm15602_regulator_ops,
+ .n_voltages = ARRAY_SIZE(bcm15602_ldo_vtbl),
+ .volt_table = bcm15602_ldo_vtbl,
+ .enable_time = 200,
+ .type = REGULATOR_VOLTAGE,
+ .owner = THIS_MODULE,
+ },
+ [BCM15602_ID_IOLDO] = {
+ .name = BCM15602_REGLTR_NAME_IOLDO,
+ .id = BCM15602_ID_IOLDO,
+ .ops = &bcm15602_regulator_ops,
+ .n_voltages = ARRAY_SIZE(bcm15602_ldo_vtbl),
+ .volt_table = bcm15602_ldo_vtbl,
+ .enable_time = 200,
+ .type = REGULATOR_VOLTAGE,
+ .owner = THIS_MODULE,
+ },
+ [BCM15602_ID_ASR] = {
+ .name = BCM15602_REGLTR_NAME_ASR,
+ .id = BCM15602_ID_ASR,
+ .ops = &bcm15602_regulator_ops,
+ .n_voltages = ARRAY_SIZE(bcm15602_asr_vtbl),
+ .volt_table = bcm15602_asr_vtbl,
+ .enable_time = 200,
+ .type = REGULATOR_VOLTAGE,
+ .owner = THIS_MODULE,
+ },
+ [BCM15602_ID_SDSR] = {
+ .name = BCM15602_REGLTR_NAME_SDSR,
+ .id = BCM15602_ID_SDSR,
+ .ops = &bcm15602_regulator_ops,
+ .n_voltages = ARRAY_SIZE(bcm15602_sdsr_vtbl),
+ .volt_table = bcm15602_sdsr_vtbl,
+ .enable_time = 200,
+ .type = REGULATOR_VOLTAGE,
+ .owner = THIS_MODULE,
+ },
+};
+
+static struct regulator_init_data
+ bcm15602_regulator_init_data[BCM15602_NUM_REGULATORS] = {
+ [BCM15602_ID_SDLDO] = {
+ .constraints = {
+ .name = "bcm15602_sdldo",
+ .valid_ops_mask = REGULATOR_CHANGE_STATUS,
+ .min_uV = 1800000,
+ .max_uV = 1800000,
+ },
+ },
+ [BCM15602_ID_IOLDO] = {
+ .constraints = {
+ .name = "bcm15602_ioldo",
+ .valid_ops_mask = REGULATOR_CHANGE_STATUS,
+ .min_uV = 1800000,
+ .max_uV = 1800000,
+ },
+ },
+ [BCM15602_ID_ASR] = {
+ .constraints = {
+ .name = "bcm15602_asr",
+ .valid_ops_mask = REGULATOR_CHANGE_STATUS |
+ REGULATOR_CHANGE_VOLTAGE,
+ .min_uV = 900000,
+ .max_uV = 1000000,
+ },
+ },
+ [BCM15602_ID_SDSR] = {
+ .constraints = {
+ .name = "bcm15602_sdsr",
+ .valid_ops_mask = REGULATOR_CHANGE_STATUS,
+ .min_uV = 1100000,
+ .max_uV = 1100000,
+ },
+ },
+};
+
+static int bcm15602_toggle_pon(struct bcm15602_chip *ddata)
+{
+ dev_err(ddata->dev, "%s: device is stuck, toggling PON\n", __func__);
+
+ gpio_set_value_cansleep(ddata->pdata->pon_gpio, 0);
+ gpio_set_value_cansleep(ddata->pdata->pon_gpio, 1);
+
+ return 0;
+}
+
+static inline bool bcm15602_is_ready(struct bcm15602_chip *ddata)
+{
+ return (gpio_get_value(ddata->pdata->resetb_gpio) == 1);
+}
+
+int bcm15602_read_byte(struct bcm15602_chip *ddata, u8 addr, u8 *data)
+{
+ int ret;
+ unsigned int val;
+ int retry_cnt = 0;
+
+ if (!bcm15602_is_ready(ddata))
+ return -EBUSY;
+
+ do {
+ ret = regmap_read(ddata->regmap, addr, &val);
+ if (!ret) {
+ *data = (u8)val;
+ return 0;
+ }
+
+ dev_err(ddata->dev,
+ "failed to read addr 0x%.2x (%d), retry %d\n",
+ addr, ret, retry_cnt);
+ } while (++retry_cnt < BCM15602_I2C_RETRY_COUNT);
+
+ dev_err(ddata->dev, "%s: failed with %d retries, power cycling device\n",
+ __func__, BCM15602_I2C_RETRY_COUNT);
+ bcm15602_toggle_pon(ddata);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(bcm15602_read_byte);
+
+int bcm15602_read_bytes(struct bcm15602_chip *ddata, u8 addr, u8 *data,
+ size_t count)
+{
+ int ret;
+ int retry_cnt = 0;
+
+ if (!bcm15602_is_ready(ddata))
+ return -EBUSY;
+
+ do {
+ ret = regmap_bulk_read(ddata->regmap, addr, data, count);
+ if (!ret)
+ return 0;
+
+ dev_err(ddata->dev, "%s: failed to read %zd bytes from addr 0x%.2x (%d), retry %d\n",
+ __func__, count, addr, ret, retry_cnt);
+ } while (++retry_cnt < BCM15602_I2C_RETRY_COUNT);
+
+ dev_err(ddata->dev, "%s: failed with %d retries, power cycling device\n",
+ __func__, BCM15602_I2C_RETRY_COUNT);
+ bcm15602_toggle_pon(ddata);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(bcm15602_read_bytes);
+
+int bcm15602_write_byte(struct bcm15602_chip *ddata, u8 addr, u8 data)
+{
+ int ret;
+ int retry_cnt = 0;
+
+ if (!bcm15602_is_ready(ddata))
+ return -EBUSY;
+
+ do {
+ ret = regmap_write(ddata->regmap, addr, data);
+ if (!ret)
+ return 0;
+
+ dev_err(ddata->dev,
+ "failed to write addr 0x%.2x (%d), retry %d\n",
+ addr, ret, retry_cnt);
+ } while (++retry_cnt < BCM15602_I2C_RETRY_COUNT);
+
+ dev_err(ddata->dev, "%s: failed with %d retries, power cycling device\n",
+ __func__, BCM15602_I2C_RETRY_COUNT);
+ bcm15602_toggle_pon(ddata);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(bcm15602_write_byte);
+
+int bcm15602_write_bytes(struct bcm15602_chip *ddata, u8 addr, u8 *data,
+ size_t count)
+{
+ int ret;
+ int retry_cnt = 0;
+
+ if (!bcm15602_is_ready(ddata))
+ return -EBUSY;
+
+ do {
+ ret = regmap_bulk_write(ddata->regmap, addr, data, count);
+ if (!ret)
+ return 0;
+
+ dev_err(ddata->dev,
+ "failed to write %zd bytes to addr 0x%.2x (%d), retry %d\n",
+ count, addr, ret, retry_cnt);
+ } while (++retry_cnt < BCM15602_I2C_RETRY_COUNT);
+
+ dev_err(ddata->dev, "%s: failed with %d retries, power cycling device\n",
+ __func__, BCM15602_I2C_RETRY_COUNT);
+ bcm15602_toggle_pon(ddata);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(bcm15602_write_bytes);
+
+int bcm15602_update_bits(struct bcm15602_chip *ddata, u8 addr,
+ unsigned int mask, u8 data)
+{
+ int ret;
+ int retry_cnt = 0;
+
+ if (!bcm15602_is_ready(ddata))
+ return -EBUSY;
+
+ do {
+ ret = regmap_update_bits(ddata->regmap, addr, mask, data);
+ if (!ret)
+ return 0;
+
+ dev_err(ddata->dev,
+ "failed to update addr 0x%.2x (%d), retry %d\n",
+ addr, ret, retry_cnt);
+ } while (++retry_cnt < BCM15602_I2C_RETRY_COUNT);
+
+ dev_err(ddata->dev, "%s: failed with %d retries, power cycling device\n",
+ __func__, BCM15602_I2C_RETRY_COUNT);
+ bcm15602_toggle_pon(ddata);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(bcm15602_update_bits);
+
+int bcm15602_read_adc_chan(struct bcm15602_chip *ddata,
+ int chan_num, u16 *chan_data)
+{
+ u8 bytes[2];
+ int ret = 0;
+ unsigned long timeout;
+
+ /* serialize manual adc conversions */
+ if (test_and_set_bit(0, &ddata->adc_conv_busy))
+ return -EAGAIN;
+
+ /* clear adc conversion completion before starting conversion */
+ reinit_completion(&ddata->adc_conv_complete);
+
+ /* enable the ADC clock and write the channel number to trigger
+ * the conversion */
+ bytes[0] = 0x1;
+ bytes[1] = BCM15602_ADC_OVSP_4 | chan_num;
+ ret = bcm15602_write_bytes(ddata, BCM15602_REG_ADC_MAN_CTRL, bytes, 2);
+ if (ret)
+ goto adc_cleanup;
+ dev_dbg(ddata->dev, "%s: started ADC converstion\n", __func__);
+
+ /* wait for completion signaled by interrupt */
+ timeout = wait_for_completion_timeout(&ddata->adc_conv_complete,
+ BCM15602_ADC_CONV_TIMEOUT);
+ if (!timeout) {
+ ret = -ETIMEDOUT;
+ dev_err(ddata->dev, "%s: ADC converstion timeout\n", __func__);
+ goto finish;
+ }
+
+ /* read and format the conversion result */
+ ret = bcm15602_read_bytes(ddata, BCM15602_REG_ADC_MAN_RESULT_L, bytes,
+ 2);
+ if (ret)
+ goto adc_cleanup;
+ *chan_data = ((bytes[1] & 0x7F) << 3) | (bytes[0] & 0x7);
+
+finish:
+ /* disable the ADC clock */
+ bcm15602_write_byte(ddata, BCM15602_REG_ADC_MAN_CTRL, 0x0);
+
+adc_cleanup:
+ /* release adc conversion */
+ clear_bit(0, &ddata->adc_conv_busy);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(bcm15602_read_adc_chan);
+
+int bcm15602_read_hk_slot(struct bcm15602_chip *ddata,
+ int slot_num, u16 *slot_data)
+{
+ u8 reading_mask[2];
+ u8 zeros[2] = {0, 0};
+ u8 byte = 0;
+ int ret = 0;
+
+ /* serialize reads to the hk slots */
+ if (test_and_set_bit(0, &ddata->hk_read_busy))
+ return -EAGAIN;
+
+ /*
+ * set the reading mask so the adc does not update the slot data while
+ * we are performing a read
+ */
+ reading_mask[0] = (1 << slot_num) & 0xFF;
+ reading_mask[1] = ((1 << slot_num) >> 8) & 0xF;
+ ret = bcm15602_write_bytes(ddata, BCM15602_REG_ADC_SLOTDATA_READINGL,
+ reading_mask, 2);
+ if (ret)
+ goto read_hk_cleanup;
+
+ ret = bcm15602_read_byte(ddata, BCM15602_REG_ADC_SLOTDATA0 + slot_num,
+ &byte);
+ if (ret)
+ goto read_hk_cleanup;
+ *slot_data = (byte << 2);
+ ret = bcm15602_read_byte(ddata, BCM15602_REG_ADC_SLOTDATA3_0_LSB +
+ (slot_num / 4), &byte);
+ if (ret)
+ goto read_hk_cleanup;
+ *slot_data |= (byte >> (slot_num % 4)) & 0x3;
+
+ /* unset the reading mask */
+ ret = bcm15602_write_bytes(ddata, BCM15602_REG_ADC_SLOTDATA_READINGL,
+ zeros, 2);
+ if (ret)
+ goto read_hk_cleanup;
+
+read_hk_cleanup:
+ /* release hk read */
+ clear_bit(0, &ddata->hk_read_busy);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(bcm15602_read_hk_slot);
+
+/* reset the watchdog timer */
+static void bcm15602_clear_wdt(struct bcm15602_chip *ddata)
+{
+ struct device *dev = ddata->dev;
+
+ dev_dbg(dev, "clearing watchdog timer alarm\n");
+
+ bcm15602_write_byte(ddata, BCM15602_REG_WDT_WDTCTRL2, 0x1E);
+}
+
+#define NOTIFY(id, event) regulator_notifier_call_chain(ddata->rdevs[id], \
+ event, NULL)
+
+/* read housekeeping adc slot status and notify any polling threads */
+static int bcm15602_handle_hk_int(struct bcm15602_chip *ddata)
+{
+ struct device *dev = ddata->dev;
+ u16 status;
+ u8 bytes[2];
+ int ret;
+
+ /* read interrupt status flags */
+ ret = bcm15602_read_bytes(ddata, BCM15602_REG_ADC_SLOTSTAL, bytes, 2);
+ if (ret)
+ return ret;
+ status = ((bytes[1] & 0x3) << 8) | bytes[0];
+ dev_dbg(dev, "%s: ADC_SLOTSTA: 0x%04x\n", __func__, status);
+
+ /* clear handled interrupts */
+ ret = bcm15602_write_bytes(ddata, BCM15602_REG_ADC_SLOTSTAL, bytes, 2);
+ if (ret)
+ return ret;
+
+ ddata->hk_status = status;
+
+ /* notify the userspace thread that is polling on the status */
+ sysfs_notify(&dev->kobj, NULL, "hk_status");
+
+ return 0;
+}
+
+/* print the device id */
+static void bcm15602_print_id(struct bcm15602_chip *ddata)
+{
+ struct device *dev = ddata->dev;
+ u8 bytes[4];
+ int ret;
+
+ ret = bcm15602_read_bytes(ddata, BCM15602_REG_SYS_PMIC_ID_LSB, bytes, 4);
+
+ if (!ret) {
+ dev_info(dev,
+ "Part: 0x%02x%02x, Rev: %d, Vendor Rev: 0x%02x\n",
+ bytes[1], bytes[0], bytes[3], bytes[2]);
+
+ /* byte 3 is the customer supplied id */
+ ddata->rev_id = bytes[3];
+ } else {
+ dev_err(dev, "%s: Could not read PMIC ID (%d)\n", __func__,
+ ret);
+ }
+}
+
+/* print the power state machine status register */
+static void bcm15602_print_psm_status(struct bcm15602_chip *ddata)
+{
+ struct device *dev = ddata->dev;
+ u8 byte;
+ int ret;
+
+ ret = bcm15602_read_byte(ddata, BCM15602_REG_PSM_PSM_ENV, &byte);
+
+ if (!ret)
+ dev_err(dev, "%s: PSM Status: 0x%02x\n", __func__, byte);
+}
+
+#define BCM15602_PSM_ENV_THERM_SHDWN_BIT 0x20
+#define BCM15602_PSM_ENV_SWCMD_EMSHDWN_BIT 0x10
+#define BCM15602_PSM_ENV_SWCMD_SHDWN_BIT 0x08
+#define BCM15602_PSM_ENV_WDTEXP_SHDWN 0x04
+#define BCM15602_PSM_ENV_TOOHOT_SHDWN 0x02
+#define BCM15602_PSM_ENV_VBATUVB_FALL_SHDWN_BIT 0x01
+
+/* print the shutdown event register */
+static void bcm15602_print_psm_event(struct bcm15602_chip *ddata)
+{
+ struct device *dev = ddata->dev;
+ char buffer[80] = "normal";
+ u8 byte;
+ int ret;
+
+ ret = bcm15602_read_byte(ddata, BCM15602_REG_PSM_PSM_ENV, &byte);
+
+ if (!ret) {
+ if (byte & BCM15602_PSM_ENV_THERM_SHDWN_BIT)
+ strncpy(buffer, "downstream thermal trip", 80);
+ else if (byte & BCM15602_PSM_ENV_SWCMD_EMSHDWN_BIT)
+ strncpy(buffer, "software emergency shutdown", 80);
+ else if (byte & BCM15602_PSM_ENV_SWCMD_SHDWN_BIT)
+ strncpy(buffer, "software shutdown", 80);
+ else if (byte & BCM15602_PSM_ENV_WDTEXP_SHDWN)
+ strncpy(buffer, "watchdog timer expiration", 80);
+ else if (byte & BCM15602_PSM_ENV_TOOHOT_SHDWN)
+ strncpy(buffer, "over temperature", 80);
+ else if (byte & BCM15602_PSM_ENV_VBATUVB_FALL_SHDWN_BIT)
+ strncpy(buffer, "supply under voltage", 80);
+ dev_info(dev, "Last reboot reason: %s\n", buffer);
+ }
+}
+
+/* handle an interrupt flag */
+static int bcm15602_handle_int(struct bcm15602_chip *ddata,
+ unsigned int flag_num)
+{
+ struct device *dev = ddata->dev;
+
+ dev_dbg(dev, "%s: flag %d\n", __func__, flag_num);
+
+ switch (flag_num) {
+ case BCM15602_INT_ASR_OVERI:
+ dev_err(dev, "Detected ASR over-current event\n");
+ NOTIFY(BCM15602_ID_ASR, REGULATOR_EVENT_OVER_CURRENT);
+ break;
+
+ case BCM15602_INT_ASR_SHUTDOWN:
+ dev_err(dev, "Detected ASR shutdown event\n");
+ NOTIFY(BCM15602_ID_ASR, REGULATOR_EVENT_FAIL);
+ break;
+
+ case BCM15602_INT_ASR_UV:
+ dev_err(dev, "Detected ASR under-voltage event\n");
+ NOTIFY(BCM15602_ID_ASR, REGULATOR_EVENT_UNDER_VOLTAGE);
+ break;
+
+ case BCM15602_INT_SDSR_OVERI:
+ dev_err(dev, "Detected SDSR over-current event\n");
+ NOTIFY(BCM15602_ID_SDSR, REGULATOR_EVENT_OVER_CURRENT);
+ break;
+
+ case BCM15602_INT_SDSR_SHUTDOWN:
+ dev_err(dev, "Detected SDSR shutdown event\n");
+ NOTIFY(BCM15602_ID_SDSR, REGULATOR_EVENT_FAIL);
+ break;
+
+ case BCM15602_INT_IOLDO_OVERI:
+ dev_err(dev, "Detected IOLDO over-current event\n");
+ NOTIFY(BCM15602_ID_IOLDO, REGULATOR_EVENT_OVER_CURRENT);
+ break;
+
+ case BCM15602_INT_IOLDO_SHUTDOWN:
+ dev_err(dev, "Detected IOLDO shutdown event\n");
+ NOTIFY(BCM15602_ID_IOLDO, REGULATOR_EVENT_FAIL);
+ break;
+
+ case BCM15602_INT_SDLDO_OVERI:
+ dev_err(dev, "Detected SDLDO over-current event\n");
+ NOTIFY(BCM15602_ID_SDLDO, REGULATOR_EVENT_OVER_CURRENT);
+ break;
+
+ case BCM15602_INT_SDLDO_SHUTDOWN:
+ dev_err(dev, "Detected SDLDO shutdown event\n");
+ NOTIFY(BCM15602_ID_SDLDO, REGULATOR_EVENT_FAIL);
+ break;
+
+ case BCM15602_INT_ADC_HK_INT:
+ bcm15602_handle_hk_int(ddata);
+ break;
+
+ case BCM15602_INT_ADC_CONV_DONE:
+ dev_dbg(dev, "%s: completing adc conversion\n", __func__);
+ complete(&ddata->adc_conv_complete);
+ break;
+
+ case BCM15602_INT_OTP_ECC_FAULT:
+ dev_dbg(dev, "Ignoring interrupt flag %d\n", flag_num);
+ break;
+
+ case BCM15602_INT_WDT_ALARM:
+ bcm15602_clear_wdt(ddata);
+ break;
+
+ case BCM15602_INT_WDT_EXP:
+ case BCM15602_INT_THERM_TRIP:
+ case BCM15602_INT_THERM_TRIP_DONE:
+ if (flag_num == BCM15602_INT_WDT_EXP)
+ dev_err(dev, "Watchdog timer expired\n");
+ else if (flag_num == BCM15602_INT_THERM_TRIP)
+ dev_err(dev, "Received thermal trip event from device\n");
+ else
+ dev_err(dev, "Thermal condition has returned to normal\n");
+
+ /* notify regulator clients of failures */
+ NOTIFY(BCM15602_ID_ASR, REGULATOR_EVENT_FAIL);
+ NOTIFY(BCM15602_ID_SDSR, REGULATOR_EVENT_FAIL);
+ NOTIFY(BCM15602_ID_SDLDO, REGULATOR_EVENT_FAIL);
+ NOTIFY(BCM15602_ID_IOLDO, REGULATOR_EVENT_FAIL);
+ bcm15602_print_psm_status(ddata);
+ break;
+
+ default:
+ dev_err(dev, "%s: Unknown flag %d\n", __func__, flag_num);
+ break;
+ }
+
+ return 0;
+}
+
+/* find pending interrupt flags */
+static int bcm15602_check_int_flags(struct bcm15602_chip *ddata)
+{
+ u8 flags[4], flags_dup[4], flag_mask, flag_clr_mask[4];
+ unsigned int first_bit, flag_num;
+ int ret = 0;
+ int i;
+
+ /* read interrupt status flags */
+ ret = bcm15602_read_bytes(ddata, BCM15602_REG_INT_INTFLAG1, flags, 4);
+ if (ret)
+ return ret;
+ ret = bcm15602_read_bytes(ddata, BCM15602_REG_INT_INTFLAG1, flags_dup,
+ 4);
+ if (ret)
+ return ret;
+
+ dev_dbg(ddata->dev,
+ "%s: [0] = 0x%02x, [1] = 0x%02x, [2] = 0x%02x, [3] = 0x%02x\n",
+ __func__, flags[0], flags[1], flags[2], flags[3]);
+
+ /* iterate through each interrupt */
+ for (i = 0; i < 4; i++) {
+ flag_clr_mask[i] = 0;
+
+ /*
+ * Workaround for b/63872924; bitwise AND the flags to reject
+ * false interrupts
+ */
+ if (flags[i] != flags_dup[i])
+ dev_err(ddata->dev,
+ "%s: mismatch irq for flag %d, 0x%02x != 0x%02x\n",
+ __func__, i, flags[i], flags_dup[i]);
+ flags[i] &= flags_dup[i];
+
+ while (flags[i]) {
+ /* find first set interrupt flag */
+ first_bit = ffs(flags[i]);
+ flag_mask = 1 << (first_bit - 1);
+ flag_num = (i * 8) + (first_bit - 1);
+
+ /* handle interrupt */
+ ret = bcm15602_handle_int(ddata, flag_num);
+
+ flags[i] &= ~flag_mask;
+ flag_clr_mask[i] |= flag_mask;
+ }
+ }
+
+ /* clear handled interrupts */
+ ret = bcm15602_write_bytes(ddata, BCM15602_REG_INT_INTFLAG1,
+ flag_clr_mask, 4);
+
+ return ret;
+}
+
+/* kernel thread for waiting for chip to come out of reset */
+static void bcm15602_reset_work(struct work_struct *data)
+{
+ struct bcm15602_chip *ddata = container_of(data, struct bcm15602_chip,
+ reset_work);
+ unsigned long timeout;
+
+ /* notify regulators of shutdown event */
+ dev_err(ddata->dev,
+ "%s: Notifying regulators of shutdown event\n", __func__);
+ NOTIFY(BCM15602_ID_ASR, REGULATOR_EVENT_FAIL);
+ NOTIFY(BCM15602_ID_SDSR, REGULATOR_EVENT_FAIL);
+ NOTIFY(BCM15602_ID_IOLDO, REGULATOR_EVENT_FAIL);
+ NOTIFY(BCM15602_ID_SDLDO, REGULATOR_EVENT_FAIL);
+
+ dev_err(ddata->dev,
+ "%s: waiting for chip to come out of reset\n", __func__);
+
+ /* wait for chip to come out of reset, signaled by resetb interrupt */
+ timeout = wait_for_completion_timeout(&ddata->reset_complete,
+ BCM15602_SHUTDOWN_RESET_TIMEOUT);
+ if (!timeout)
+ dev_err(ddata->dev,
+ "%s: timeout waiting for device to return from reset\n",
+ __func__);
+ else
+ bcm15602_chip_init(ddata);
+}
+
+/* irq handler for resetb pin */
+static irqreturn_t bcm15602_resetb_irq_handler(int irq, void *cookie)
+{
+ struct bcm15602_chip *ddata = (struct bcm15602_chip *)cookie;
+
+ if (gpio_get_value(ddata->pdata->resetb_gpio)) {
+ dev_dbg(ddata->dev, "%s: completing reset\n", __func__);
+ complete(&ddata->reset_complete);
+ } else {
+ dev_err(ddata->dev, "%s: unexpected device reset\n", __func__);
+ schedule_work(&ddata->reset_work);
+ }
+
+ return IRQ_HANDLED;
+}
+
+/* irq handler for intb pin */
+static irqreturn_t bcm15602_intb_irq_handler(int irq, void *cookie)
+{
+ struct bcm15602_chip *ddata = (struct bcm15602_chip *)cookie;
+ int ret;
+
+ dev_dbg(ddata->dev, "%s: observed irq\n", __func__);
+
+ while (!gpio_get_value(ddata->pdata->intb_gpio)) {
+ ret = bcm15602_check_int_flags(ddata);
+ if (ret)
+ return IRQ_RETVAL(ret);
+ }
+
+ return IRQ_HANDLED;
+}
+
+#define BCM15602_ASR_VOCTRL_VBASE 565000 /* microvolts */
+#define BCM15602_ASR_VOCTRL_VSTEP 5000 /* microvolts */
+
+/* set the current voltage of the regulator in microvolts */
+static int bcm15602_regulator_set_voltage(struct regulator_dev *rdev,
+ int min_uV, int max_uV,
+ unsigned *selector)
+{
+ struct bcm15602_chip *ddata = rdev_get_drvdata(rdev);
+ enum bcm15602_regulator_ids rid = rdev_get_id(rdev);
+ int vsel, vout;
+ int i;
+
+ if (rid != BCM15602_ID_ASR)
+ return -EINVAL;
+
+ for (i = 0; i < ARRAY_SIZE(bcm15602_asr_vtbl); i++) {
+ if ((min_uV <= bcm15602_asr_vtbl[i]) &&
+ (bcm15602_asr_vtbl[i] <= max_uV)) {
+ *selector = i;
+ vout = bcm15602_asr_vtbl[i];
+ break;
+ }
+ }
+
+ if (i == ARRAY_SIZE(bcm15602_asr_vtbl))
+ return -EINVAL;
+
+ vsel = (vout - BCM15602_ASR_VOCTRL_VBASE) / BCM15602_ASR_VOCTRL_VSTEP;
+
+ dev_dbg(ddata->dev, "%s: rid %d, setting vsel to 0x%02x\n", __func__,
+ rid, vsel);
+
+ return bcm15602_write_byte(ddata, BCM15602_REG_BUCK_ASR_VOCTRL,
+ vsel & 0x7F);
+}
+
+/* get the current voltage of the regulator in microvolts */
+static int bcm15602_regulator_get_voltage(struct regulator_dev *rdev)
+{
+ struct bcm15602_chip *ddata = rdev_get_drvdata(rdev);
+ enum bcm15602_regulator_ids rid = rdev_get_id(rdev);
+ u8 reg_data;
+ int vsel, vstep, vbase;
+ int ret;
+
+ switch (rid) {
+ case BCM15602_ID_SDLDO:
+ ret = bcm15602_read_byte(ddata, BCM15602_REG_LDO_SDLDO_VOCTRL,
+ ®_data);
+ if (ret)
+ return ret;
+ vbase = (reg_data & 0x80) ? 1725000 : 1380000;
+ vstep = (reg_data & 0x80) ? 25000 : 10000;
+ vsel = reg_data & 0x3F;
+ break;
+ case BCM15602_ID_IOLDO:
+ ret = bcm15602_read_byte(ddata, BCM15602_REG_LDO_IOLDO_VOCTRL,
+ ®_data);
+ if (ret)
+ return ret;
+ vbase = (reg_data & 0x80) ? 1725000 : 1380000;
+ vstep = (reg_data & 0x80) ? 25000 : 10000;
+ vsel = reg_data & 0x3F;
+ break;
+ case BCM15602_ID_ASR:
+ ret = bcm15602_read_byte(ddata, BCM15602_REG_BUCK_ASR_VOCTRL,
+ ®_data);
+ if (ret)
+ return ret;
+ vbase = 565000;
+ vstep = 5000;
+ vsel = reg_data & 0x7F;
+ break;
+ case BCM15602_ID_SDSR:
+ ret = bcm15602_read_byte(ddata, BCM15602_REG_BUCK_SDSR_CTRL1,
+ ®_data);
+ if (ret)
+ return ret;
+ switch ((reg_data & 0xC0) >> 6) {
+ case 0:
+ vbase = 565000;
+ vstep = 5000;
+ break;
+ case 1:
+ vbase = 678000;
+ vstep = 6000;
+ break;
+ case 2:
+ vbase = 1130000;
+ vstep = 10000;
+ break;
+ case 3:
+ vbase = 1695000;
+ vstep = 15000;
+ break;
+ default: // make compiler happy
+ return -EINVAL;
+ }
+ ret = bcm15602_read_byte(ddata, BCM15602_REG_BUCK_SDSR_VOCTRL,
+ ®_data);
+ if (ret)
+ return ret;
+ vsel = reg_data & 0x7F;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ dev_dbg(ddata->dev, "%s: rid %d, returning voltage %d\n", __func__, rid,
+ vbase + vsel * vstep);
+
+ return vbase + vsel * vstep;
+}
+
+/* enable the regulator */
+static int bcm15602_regulator_enable(struct regulator_dev *rdev)
+{
+ struct bcm15602_chip *ddata = rdev_get_drvdata(rdev);
+ enum bcm15602_regulator_ids rid = rdev_get_id(rdev);
+ int ret;
+
+ switch (rid) {
+ case BCM15602_ID_SDLDO:
+ ret = bcm15602_write_byte(ddata, BCM15602_REG_LDO_SDLDO_ENCTRL,
+ 0x3);
+ break;
+ case BCM15602_ID_IOLDO:
+ ret = bcm15602_write_byte(ddata, BCM15602_REG_LDO_IOLDO_ENCTRL,
+ 0x3);
+ break;
+ case BCM15602_ID_ASR:
+ ret = bcm15602_write_byte(ddata, BCM15602_REG_BUCK_ASR_CTRL0,
+ 0xC5);
+ break;
+ case BCM15602_ID_SDSR:
+ ret = bcm15602_write_byte(ddata, BCM15602_REG_BUCK_SDSR_CTRL0,
+ 0xDB);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (!ret) {
+ set_bit(rid, &ddata->reg_enabled_mask);
+ bcm15602_enable_wdt(ddata);
+ }
+
+ return ret;
+}
+
+/* disable the regulator */
+static int bcm15602_regulator_disable(struct regulator_dev *rdev)
+{
+ struct bcm15602_chip *ddata = rdev_get_drvdata(rdev);
+ enum bcm15602_regulator_ids rid = rdev_get_id(rdev);
+ int ret;
+
+ switch (rid) {
+ case BCM15602_ID_SDLDO:
+ ret = bcm15602_write_byte(ddata, BCM15602_REG_LDO_SDLDO_ENCTRL,
+ 0x2);
+ break;
+ case BCM15602_ID_IOLDO:
+ ret = bcm15602_write_byte(ddata, BCM15602_REG_LDO_IOLDO_ENCTRL,
+ 0x2);
+ break;
+ case BCM15602_ID_ASR:
+ ret = bcm15602_write_byte(ddata, BCM15602_REG_BUCK_ASR_CTRL0,
+ 0xC4);
+ break;
+ case BCM15602_ID_SDSR:
+ ret = bcm15602_write_byte(ddata, BCM15602_REG_BUCK_SDSR_CTRL0,
+ 0xDA);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (!ret) {
+ clear_bit(rid, &ddata->reg_enabled_mask);
+ if (!ddata->reg_enabled_mask)
+ bcm15602_disable_wdt(ddata);
+ }
+
+ return ret;
+}
+
+/* get regulator enable status */
+static int bcm15602_regulator_is_enabled(struct regulator_dev *rdev)
+{
+ struct bcm15602_chip *ddata = rdev_get_drvdata(rdev);
+ enum bcm15602_regulator_ids rid = rdev_get_id(rdev);
+
+ if ((rid >= 0) && (rid < BCM15602_NUM_REGULATORS))
+ return !!(ddata->reg_enabled_mask & (1 << rid));
+ else
+ return -EINVAL;
+}
+
+/* get platform data from the device tree */
+static struct bcm15602_platform_data *bcm15602_get_platform_data_from_dt
+ (struct device *dev)
+{
+ struct device_node *np = dev->of_node;
+ struct bcm15602_platform_data *pdata;
+
+ if (!np)
+ return ERR_PTR(-ENODEV);
+
+ pdata = devm_kzalloc(dev, sizeof(*pdata), GFP_KERNEL);
+ if (!pdata)
+ return ERR_PTR(-ENOMEM);
+
+ pdata->pon_gpio = of_get_named_gpio(np, "bcm,pon-gpio", 0);
+ pdata->resetb_gpio = of_get_named_gpio(np, "bcm,resetb-gpio", 0);
+ pdata->intb_gpio = of_get_named_gpio(np, "bcm,intb-gpio", 0);
+ pdata->resetb_irq = gpio_to_irq(pdata->resetb_gpio);
+ pdata->intb_irq = gpio_to_irq(pdata->intb_gpio);
+
+ return pdata;
+}
+
+/* register a regulator with the kernel regulator framework */
+static int bcm15602_regulator_register(struct bcm15602_chip *ddata)
+{
+ struct device *dev = ddata->dev;
+ struct regulator_config cfg = {
+ .dev = dev,
+ .driver_data = ddata,
+ .regmap = ddata->regmap
+ };
+ struct regulator_dev *rdev;
+ int i;
+
+ for (i = 0; i < BCM15602_NUM_REGULATORS; i++) {
+ cfg.init_data = &bcm15602_regulator_init_data[i];
+ rdev = devm_regulator_register(dev,
+ &bcm15602_regulator_desc[i],
+ &cfg);
+ if (IS_ERR(rdev)) {
+ dev_err(dev,
+ "%s: failed to register regulator %d\n",
+ __func__, i);
+ return PTR_ERR(rdev);
+ }
+
+ *(ddata->rdevs + i) = rdev;
+ }
+
+ return 0;
+}
+
+/* configure the adc and housekeeping slots */
+static void bcm15602_config_adc(struct bcm15602_chip *ddata)
+{
+ u32 slot_mask = 0;
+ int i;
+
+ for (i = 0; i < BCM15602_HK_NUM_SLOTS; i++) {
+ bcm15602_write_byte(ddata,
+ BCM15602_REG_ADC_SLOTCTRL0 + i,
+ adc_ctrls[i].ctrl);
+ bcm15602_write_byte(ddata,
+ BCM15602_REG_ADC_SLOTTHL0 + (i * 2),
+ adc_ctrls[i].th_ctrl & 0xFF);
+ bcm15602_write_byte(ddata,
+ BCM15602_REG_ADC_SLOTTHL0 + (i * 2) + 1,
+ (adc_ctrls[i].th_ctrl >> 8) & 0xFF);
+
+ if (!adc_ctrls[i].th_en)
+ slot_mask |= 1 << i;
+ };
+
+ for (i = 0; i < ((BCM15602_HK_NUM_SLOTS + 7) / 8); i++) {
+ bcm15602_write_byte(ddata,
+ BCM15602_REG_ADC_SLOTMSKL + i,
+ (slot_mask >> (i * 8)) & 0xFF);
+ }
+
+ /* update hk every 10 ms */
+ bcm15602_write_byte(ddata, BCM15602_REG_ADC_HKCTRL, 0x8);
+}
+
+/* enable all of the interrupts */
+static void bcm15602_config_ints(struct bcm15602_chip *ddata)
+{
+ /* clear any pending interrupts */
+ bcm15602_write_byte(ddata, BCM15602_REG_INT_INTFLAG1, 0x1F);
+ bcm15602_write_byte(ddata, BCM15602_REG_INT_INTFLAG2, 0x1B);
+ bcm15602_write_byte(ddata, BCM15602_REG_INT_INTFLAG3, 0x03);
+ bcm15602_write_byte(ddata, BCM15602_REG_INT_INTFLAG4, 0x1F);
+
+ /* enable interrupts we care about */
+ bcm15602_write_byte(ddata, BCM15602_REG_INT_INTEN1, 0x1F);
+ bcm15602_write_byte(ddata, BCM15602_REG_INT_INTEN2, 0x1B);
+ bcm15602_write_byte(ddata, BCM15602_REG_INT_INTEN3, 0x02);
+ bcm15602_write_byte(ddata, BCM15602_REG_INT_INTEN4, 0x1F);
+}
+
+/* enable the watchdog timer and reset the count */
+static void bcm15602_enable_wdt(struct bcm15602_chip *ddata)
+{
+ if (!ddata->wdt_enabled) {
+ bcm15602_write_byte(ddata, BCM15602_REG_WDT_WDTCTRL1, 0x01);
+ bcm15602_write_byte(ddata, BCM15602_REG_WDT_WDTCTRL2, 0x1E);
+ ddata->wdt_enabled = true;
+ }
+}
+
+/* disable the watchdog timer */
+static void bcm15602_disable_wdt(struct bcm15602_chip *ddata)
+{
+ if (ddata->wdt_enabled) {
+ bcm15602_write_byte(ddata, BCM15602_REG_WDT_WDTCTRL1, 0x00);
+ ddata->wdt_enabled = false;
+ }
+}
+
+/* some changes to default configuration based on bringup */
+static int bcm15602_chip_fixup(struct bcm15602_chip *ddata)
+{
+ dev_dbg(ddata->dev, "%s: rev %d\n", __func__, ddata->rev_id);
+
+ if (ddata->rev_id < BCM15602_REV_A1) {
+ /* enable bandgap curvature correction for improved accuracy */
+ bcm15602_update_bits(ddata, BCM15602_REG_ADC_BGCTRL, 0x40,
+ 0x40);
+
+ /* unlock register, then set ASR switching frequency trim */
+ bcm15602_write_byte(ddata, BCM15602_REG_SYS_WRLOCKEY, 0x38);
+ bcm15602_write_byte(ddata, BCM15602_REG_BUCK_ASR_CTRL2, 0x04);
+
+ /* set ASR undervoltage threshold */
+ bcm15602_update_bits(ddata, BCM15602_REG_BUCK_ASR_CTRL5, 0x0C,
+ 0x04);
+
+ /* set ASR comparator input filter select */
+ bcm15602_update_bits(ddata, BCM15602_REG_BUCK_ASR_CTRL11, 0x03,
+ 0x03);
+
+ /* set ASR feedback network R2 adjustment */
+ bcm15602_update_bits(ddata, BCM15602_REG_BUCK_ASR_TSET_CTRL2,
+ 0x03, 0x02);
+
+ /* set ASR rail to 0.9V */
+ bcm15602_write_byte(ddata, BCM15602_REG_BUCK_ASR_VOCTRL, 0x43);
+
+ /* set ASR to single phase */
+ bcm15602_update_bits(ddata, BCM15602_REG_BUCK_ASR_TSET_CTRL2,
+ 0x10, 0x00);
+ } else if (ddata->rev_id == BCM15602_REV_A1) {
+ /* enable bandgap curvature correction for improved accuracy */
+ bcm15602_update_bits(ddata, BCM15602_REG_ADC_BGCTRL, 0x40,
+ 0x40);
+ }
+
+ return 0;
+}
+
+/* initialize the chip */
+static int bcm15602_chip_init(struct bcm15602_chip *ddata)
+{
+ bcm15602_print_id(ddata);
+ bcm15602_print_psm_event(ddata);
+
+ bcm15602_chip_fixup(ddata);
+
+ bcm15602_config_adc(ddata);
+ bcm15602_config_ints(ddata);
+
+ return 0;
+}
+
+static int bcm15602_probe(struct i2c_client *client,
+ const struct i2c_device_id *dev_id)
+{
+ struct device *dev = &client->dev;
+ struct bcm15602_chip *ddata;
+ struct bcm15602_platform_data *pdata;
+ unsigned long timeout;
+ int ret;
+ int i;
+
+ /* allocate memory for chip structure */
+ ddata = devm_kzalloc(dev, sizeof(struct bcm15602_chip),
+ GFP_KERNEL);
+ if (!ddata)
+ return -ENOMEM;
+
+ /* set client data */
+ i2c_set_clientdata(client, ddata);
+
+ /* get platform data */
+ pdata = dev_get_platdata(dev);
+ if (!pdata) {
+ pdata = bcm15602_get_platform_data_from_dt(dev);
+ if (IS_ERR(pdata))
+ return PTR_ERR(pdata);
+ }
+
+ /* initialize chip structure */
+ ddata->dev = dev;
+ ddata->pdata = pdata;
+ dev->platform_data = pdata;
+
+ /* initialize some structures */
+ INIT_WORK(&ddata->reset_work, bcm15602_reset_work);
+
+ /* initialize completions */
+ init_completion(&ddata->reset_complete);
+ init_completion(&ddata->adc_conv_complete);
+
+ /* initialize regmap */
+ ddata->regmap = devm_regmap_init_i2c(client, &bcm15602_regmap_config);
+ if (IS_ERR(ddata->regmap)) {
+ ret = PTR_ERR(ddata->regmap);
+ dev_err(dev,
+ "%s: could not initialize regmap (%d)\n",
+ __func__, ret);
+ return -ENOMEM;
+ }
+
+ /* create sysfs attributes */
+ bcm15602_config_sysfs(dev);
+
+ /* request GPIOs and IRQs */
+ devm_gpio_request_one(dev, pdata->pon_gpio, GPIOF_OUT_INIT_LOW,
+ "BCM15602 PON");
+ devm_gpio_request_one(dev, pdata->resetb_gpio, GPIOF_IN,
+ "BCM15602 RESETB");
+ devm_gpio_request_one(dev, pdata->intb_gpio, GPIOF_IN,
+ "BCM15602 INTB");
+ ret = devm_request_threaded_irq(dev, pdata->resetb_irq, NULL,
+ bcm15602_resetb_irq_handler,
+ IRQF_TRIGGER_FALLING |
+ IRQF_TRIGGER_RISING | IRQF_ONESHOT,
+ "bcm15602-resetb", ddata);
+ ret = devm_request_threaded_irq(dev, pdata->intb_irq, NULL,
+ bcm15602_intb_irq_handler,
+ IRQF_TRIGGER_FALLING | IRQF_ONESHOT,
+ "bcm15602-intb", ddata);
+
+ /* disable the irq while doing initial power on */
+ disable_irq(pdata->resetb_irq);
+
+ for (i = 0; i < BCM15602_PON_RETRY_CNT; i++) {
+ dev_dbg(dev, "%s: powering on bcm15602\n", __func__);
+ gpio_set_value_cansleep(pdata->pon_gpio, 1);
+
+ /* give the chip some time to power on */
+ msleep(BCM15602_PON_RESET_DELAY);
+
+ /* poll on the gpio until it goes high or until we timeout */
+ timeout = jiffies + BCM15602_PON_RESET_TIMEOUT;
+ while (!gpio_get_value(pdata->resetb_gpio) &&
+ time_before(jiffies, timeout)) {
+ usleep_range(100, 105);
+ }
+
+ if (gpio_get_value(pdata->resetb_gpio))
+ break;
+
+ dev_err(dev, "%s: powering on timed out, try (%d/%d)\n",
+ __func__, i + 1, BCM15602_PON_RETRY_CNT);
+
+ gpio_set_value_cansleep(pdata->pon_gpio, 0);
+ usleep_range(100, 105);
+ }
+
+ /* check for failure */
+ if (i == BCM15602_PON_RETRY_CNT) {
+ ret = -ETIMEDOUT;
+ dev_err(dev, "%s: powering on failed\n", __func__);
+ goto error_reset;
+ }
+
+ /* enable the irq after power on */
+ enable_irq(pdata->resetb_irq);
+
+ /* initialize chip */
+ bcm15602_chip_init(ddata);
+
+ /* initialize and register device regulators */
+ ddata->rdevs =
+ devm_kzalloc(dev,
+ BCM15602_NUM_REGULATORS *
+ sizeof(struct regulator_dev *),
+ GFP_KERNEL);
+ if (!ddata->rdevs) {
+ dev_err(dev,
+ "%s: could not initialize rdevs array\n",
+ __func__);
+ return -ENOMEM;
+ }
+ bcm15602_regulator_register(ddata);
+
+ return mfd_add_devices(dev, -1, bcm15602_devs,
+ ARRAY_SIZE(bcm15602_devs),
+ NULL, 0, NULL);
+
+error_reset:
+ return ret;
+}
+
+#ifdef CONFIG_PM
+static int bcm15602_suspend(struct device *dev)
+{
+ struct bcm15602_platform_data *pdata;
+
+ pdata = dev_get_platdata(dev);
+ if (pdata)
+ enable_irq_wake(pdata->intb_irq);
+ return 0;
+}
+
+static int bcm15602_resume(struct device *dev)
+{
+ struct bcm15602_platform_data *pdata;
+
+ pdata = dev_get_platdata(dev);
+ if (pdata)
+ disable_irq_wake(pdata->intb_irq);
+ return 0;
+}
+
+static const struct dev_pm_ops bcm15602_dev_pm_ops = {
+ .suspend = bcm15602_suspend,
+ .resume = bcm15602_resume,
+};
+#endif
+
+static const struct of_device_id bcm15602_dt_ids[] = {
+ { .compatible = "brcm,bcm15602", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, bcm15602_dt_ids);
+
+static const struct i2c_device_id bcm15602_id_table[] = {
+ { .name = DRIVER_NAME, .driver_data = 0 },
+ { },
+};
+MODULE_DEVICE_TABLE(i2c, bcm15602_id_table);
+
+static struct i2c_driver bcm15602_driver = {
+ .driver = {
+ .name = DRIVER_NAME,
+ .owner = THIS_MODULE,
+#ifdef CONFIG_PM
+ .pm = &bcm15602_dev_pm_ops,
+#endif
+ },
+ .probe = bcm15602_probe,
+ .id_table = bcm15602_id_table,
+};
+
+module_i2c_driver(bcm15602_driver);
+
+MODULE_AUTHOR("Trevor Bunker <trevorbunker@google.com>");
+MODULE_DESCRIPTION("BCM15602 Regulator Device Driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/staging/bcm15602/bcm15602-regulator.h b/drivers/staging/bcm15602/bcm15602-regulator.h
new file mode 100644
index 0000000..8fe2e67
--- /dev/null
+++ b/drivers/staging/bcm15602/bcm15602-regulator.h
@@ -0,0 +1,363 @@
+/*
+ * Copyright (C) 2016 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#ifndef BCM15602_REGULATOR_H
+#define BCM15602_REGULATOR_H
+
+/* registers */
+#define BCM15602_REG_SYS_PMIC_ID_LSB 0x00
+#define BCM15602_REG_SYS_PMIC_ID_MSB 0x01
+#define BCM15602_REG_SYS_PMIC_REV 0x02
+#define BCM15602_REG_SYS_PMIC_GID 0x03
+#define BCM15602_REG_SYS_WRLOCKEY 0x04
+#define BCM15602_REG_SYS_RCOTRIM_LSB 0x05
+#define BCM15602_REG_SYS_RCOTRIM_MSB 0x06
+#define BCM15602_REG_SYS_GPIO_CTRL 0x07
+#define BCM15602_REG_SYS_GPIO_OUT_CTRL 0x08
+#define BCM15602_REG_SYS_GPIO_IN 0x09
+#define BCM15602_REG_I2C_CSR_I2CCTRL1 0x10
+#define BCM15602_REG_I2C_CSR_I2CCTRL2 0x11
+#define BCM15602_REG_I2C_CSR_I2CFIFO 0x12
+#define BCM15602_REG_I2C_I2CCTRL3 0x20
+#define BCM15602_REG_I2C_I2C_SLAVE_ADDR 0x21
+#define BCM15602_REG_INT_INTEN1 0x28
+#define BCM15602_REG_INT_INTEN2 0x29
+#define BCM15602_REG_INT_INTEN3 0x2a
+#define BCM15602_REG_INT_INTEN4 0x2b
+#define BCM15602_REG_INT_INTSET1 0x2c
+#define BCM15602_REG_INT_INTSET2 0x2d
+#define BCM15602_REG_INT_INTSET3 0x2e
+#define BCM15602_REG_INT_INTSET4 0x2f
+#define BCM15602_REG_INT_INTFLAG1 0x30
+#define BCM15602_REG_INT_INTFLAG2 0x31
+#define BCM15602_REG_INT_INTFLAG3 0x32
+#define BCM15602_REG_INT_INTFLAG4 0x33
+#define BCM15602_REG_PSM_PSM_CTRL1 0x38
+#define BCM15602_REG_PSM_PSM_CTRL2 0x39
+#define BCM15602_REG_PSM_PSM_CTRL3 0x3a
+#define BCM15602_REG_PSM_PSM_CTRL4 0x3b
+#define BCM15602_REG_PSM_PSM_STS 0x3c
+#define BCM15602_REG_PSM_PSM_ENV 0x3d
+#define BCM15602_REG_PMC_PWR_GRP_CTRL1 0x40
+#define BCM15602_REG_PMC_PWR_GRP_CTRL2 0x41
+#define BCM15602_REG_PMC_PWR_GRP_CONFG 0x42
+#define BCM15602_REG_PMC_PMC_TEST 0x43
+#define BCM15602_REG_BUCK_SDSR_VOCTRL 0x50
+#define BCM15602_REG_BUCK_SDSR_VMAX 0x51
+#define BCM15602_REG_BUCK_SDSR_CTRL0 0x52
+#define BCM15602_REG_BUCK_SDSR_CTRL1 0x53
+#define BCM15602_REG_BUCK_SDSR_CTRL2 0x54
+#define BCM15602_REG_BUCK_SDSR_CTRL3 0x55
+#define BCM15602_REG_BUCK_SDSR_CTRL4 0x56
+#define BCM15602_REG_BUCK_SDSR_CTRL5 0x57
+#define BCM15602_REG_BUCK_SDSR_TSET_CTRL1 0x58
+#define BCM15602_REG_BUCK_ASR_VOCTRL 0x59
+#define BCM15602_REG_BUCK_ASR_VMAX 0x5a
+#define BCM15602_REG_BUCK_ASR_VOTRIM 0x5b
+#define BCM15602_REG_BUCK_ASR_CTRL0 0x5c
+#define BCM15602_REG_BUCK_ASR_CTRL1 0x5d
+#define BCM15602_REG_BUCK_ASR_CTRL2 0x5e
+#define BCM15602_REG_BUCK_ASR_CTRL3 0x5f
+#define BCM15602_REG_BUCK_ASR_CTRL4 0x60
+#define BCM15602_REG_BUCK_ASR_CTRL5 0x61
+#define BCM15602_REG_BUCK_ASR_CTRL6 0x62
+#define BCM15602_REG_BUCK_ASR_CTRL7 0x63
+#define BCM15602_REG_BUCK_ASR_CTRL8 0x64
+#define BCM15602_REG_BUCK_ASR_CTRL9 0x65
+#define BCM15602_REG_BUCK_ASR_CTRL10 0x66
+#define BCM15602_REG_BUCK_ASR_CTRL11 0x67
+#define BCM15602_REG_BUCK_ASR_TSET_CTRL1 0x68
+#define BCM15602_REG_BUCK_ASR_TSET_CTRL2 0x69
+#define BCM15602_REG_BUCK_SDSR_OI_CTRL 0x6a
+#define BCM15602_REG_LDO_IOLDO_VOCTRL 0x70
+#define BCM15602_REG_LDO_IOLDO_VMAX 0x71
+#define BCM15602_REG_LDO_IOLDO_ENCTRL 0x72
+#define BCM15602_REG_LDO_IOLDO_OVRI_CNT1 0x73
+#define BCM15602_REG_LDO_IOLDO_OVRI_CNT2 0x74
+#define BCM15602_REG_LDO_SDLDO_VOCTRL 0x75
+#define BCM15602_REG_LDO_SDLDO_VMAX 0x76
+#define BCM15602_REG_LDO_SDLDO_ENCTRL 0x77
+#define BCM15602_REG_LDO_SDLDO_OVRI_CNT1 0x78
+#define BCM15602_REG_LDO_SDLDO_OVRI_CNT2 0x79
+#define BCM15602_REG_LDO_LDO_TEST_CTRL1 0x7a
+#define BCM15602_REG_ADC_MAN_CTRL 0x80
+#define BCM15602_REG_ADC_MAN_CONV_CHNUM 0x81
+#define BCM15602_REG_ADC_MAN_RESULT_L 0x82
+#define BCM15602_REG_ADC_MAN_RESULT_H 0x83
+#define BCM15602_REG_ADC_DIE_TEMP_TRIM 0x84
+#define BCM15602_REG_ADC_ISNS_RTRIM 0x85
+#define BCM15602_REG_ADC_ADC_OFFSET_TRIM 0x86
+#define BCM15602_REG_ADC_ADC_GAIN_TRIM 0x87
+#define BCM15602_REG_ADC_ISNS_GAIN_TRIM_LSB 0x88
+#define BCM15602_REG_ADC_SDLDO_ISNS_GAIN_TRIMH 0x89
+#define BCM15602_REG_ADC_SDLDO_ISNS_OFFSET_TRIM 0x8a
+#define BCM15602_REG_ADC_HKCTRL 0x8b
+#define BCM15602_REG_ADC_HYST 0x8c
+#define BCM15602_REG_ADC_SLOTCTRL0 0x8d
+#define BCM15602_REG_ADC_SLOTCTRL1 0x8e
+#define BCM15602_REG_ADC_SLOTCTRL2 0x8f
+#define BCM15602_REG_ADC_SLOTCTRL3 0x90
+#define BCM15602_REG_ADC_SLOTCTRL4 0x91
+#define BCM15602_REG_ADC_SLOTCTRL5 0x92
+#define BCM15602_REG_ADC_SLOTCTRL6 0x93
+#define BCM15602_REG_ADC_SLOTCTRL7 0x94
+#define BCM15602_REG_ADC_SLOTCTRL8 0x95
+#define BCM15602_REG_ADC_SLOTCTRL9 0x96
+#define BCM15602_REG_ADC_SLOTCTRL10 0x97
+#define BCM15602_REG_ADC_SLOTCTRL11 0x98
+#define BCM15602_REG_ADC_SLOTTHL0 0x99
+#define BCM15602_REG_ADC_SLOTTHH0 0x9a
+#define BCM15602_REG_ADC_SLOTTHL1 0x9b
+#define BCM15602_REG_ADC_SLOTTHH1 0x9c
+#define BCM15602_REG_ADC_SLOTTHL2 0x9d
+#define BCM15602_REG_ADC_SLOTTHH2 0x9e
+#define BCM15602_REG_ADC_SLOTTHL3 0x9f
+#define BCM15602_REG_ADC_SLOTTHH3 0xa0
+#define BCM15602_REG_ADC_SLOTTHL4 0xa1
+#define BCM15602_REG_ADC_SLOTTHH4 0xa2
+#define BCM15602_REG_ADC_SLOTTHL5 0xa3
+#define BCM15602_REG_ADC_SLOTTHH5 0xa4
+#define BCM15602_REG_ADC_SLOTTHL6 0xa5
+#define BCM15602_REG_ADC_SLOTTHH6 0xa6
+#define BCM15602_REG_ADC_SLOTTHL7 0xa7
+#define BCM15602_REG_ADC_SLOTTHH7 0xa8
+#define BCM15602_REG_ADC_SLOTTHL8 0xa9
+#define BCM15602_REG_ADC_SLOTTHH8 0xaa
+#define BCM15602_REG_ADC_SLOTTHL9 0xab
+#define BCM15602_REG_ADC_SLOTTHH9 0xac
+#define BCM15602_REG_ADC_SLOTDATA0 0xad
+#define BCM15602_REG_ADC_SLOTDATA1 0xae
+#define BCM15602_REG_ADC_SLOTDATA2 0xaf
+#define BCM15602_REG_ADC_SLOTDATA3 0xb0
+#define BCM15602_REG_ADC_SLOTDATA4 0xb1
+#define BCM15602_REG_ADC_SLOTDATA5 0xb2
+#define BCM15602_REG_ADC_SLOTDATA6 0xb3
+#define BCM15602_REG_ADC_SLOTDATA7 0xb4
+#define BCM15602_REG_ADC_SLOTDATA8 0xb5
+#define BCM15602_REG_ADC_SLOTDATA9 0xb6
+#define BCM15602_REG_ADC_SLOTDATA10 0xb7
+#define BCM15602_REG_ADC_SLOTDATA11 0xb8
+#define BCM15602_REG_ADC_SLOTDATA3_0_LSB 0xb9
+#define BCM15602_REG_ADC_SLOTDATA7_4_LSB 0xba
+#define BCM15602_REG_ADC_SLOTDATA11_8_LSB 0xbb
+#define BCM15602_REG_ADC_SLOTSTAL 0xbc
+#define BCM15602_REG_ADC_SLOTSTAH 0xbd
+#define BCM15602_REG_ADC_SLOTMSKL 0xbe
+#define BCM15602_REG_ADC_SLOTMSKH 0xbf
+#define BCM15602_REG_ADC_SLOTDATA_READINGL 0xc0
+#define BCM15602_REG_ADC_SLOTDATA_READINGH 0xc1
+#define BCM15602_REG_ADC_BGCTRL 0xc2
+#define BCM15602_REG_ADC_BG_OFFSET_TRIM 0xc3
+#define BCM15602_REG_ADC_HOTTEMP_THRES 0xc4
+#define BCM15602_REG_ADC_LBG_TC_TRIM 0xc5
+#define BCM15602_REG_ADC_TCTRIM 0xc6
+#define BCM15602_REG_ADC_IOLDO_ISNS_GAIN_TRIMH 0xc8
+#define BCM15602_REG_ADC_IOLDO_ISNS_OFFSET_TRIM 0xc9
+#define BCM15602_REG_ADC_ASR_ISNS_GAIN_TRIMH 0xca
+#define BCM15602_REG_ADC_ASR_ISNS_OFFSET_TRIM 0xcb
+#define BCM15602_REG_ADC_ASR_MSTR_ISNS_GAIN_TRIMH 0xcc
+#define BCM15602_REG_ADC_ASR_MSTR_ISNS_OFFSET_TRIM 0xcd
+#define BCM15602_REG_ADC_SDSR_ISNS_GAIN_TRIMH 0xce
+#define BCM15602_REG_ADC_SDSR_ISNS_OFFSET_TRIM 0xcf
+#define BCM15602_REG_WDT_WDTCTRL1 0xd0
+#define BCM15602_REG_WDT_WDTCTRL2 0xd1
+#define BCM15602_REG_WDT_WDTSTST 0xd2
+
+/* adc logical channel number */
+#define BCM15602_ADC_SDLDO_CURR 0
+#define BCM15602_ADC_IOLDO_CURR 1
+#define BCM15602_ADC_ASR_SLV_CURR 2
+#define BCM15602_ADC_ASR_MSTR_CURR 3
+#define BCM15602_ADC_SDSR_CURR 4
+#define BCM15602_ADC_RESERVED1 5
+#define BCM15602_ADC_PTAT 6
+#define BCM15602_ADC_RESERVED2 7
+#define BCM15602_ADC_VIO 8
+#define BCM15602_ADC_VSRC 9
+#define BCM15602_ADC_VBAT 10
+#define BCM15602_ADC_ANATEST 11
+
+/* full scale of 10-bit adc slots */
+#define BCM15602_ADC_SCALE_ASR_CURR (3512000 / 1023) /* in uA/lsb */
+#define BCM15602_ADC_SCALE_SDSR_CURR (2106000 / 1023) /* in uA/lsb */
+#define BCM15602_ADC_SCALE_SDLDO_CURR (523000 / 1023) /* in uA/lsb */
+#define BCM15602_ADC_SCALE_IOLDO_CURR (523000 / 1023) /* in uA/lsb */
+#define BCM15602_ADC_SCALE_VBAT (4800000 / 1023) /* in uV/lsb */
+#define BCM15602_ADC_SCALE_PTAT (1200000 / 1023) /* in uV/lsb */
+
+/* housekeeping slot numbers */
+#define BCM15602_HK_ASR_MSTR_CURR_OVERI 0
+#define BCM15602_HK_ASR_SLV_CURR_OVERI 1
+#define BCM15602_HK_SDSR_CURR_OVERI 2
+#define BCM15602_HK_SDLDO_CURR_OVERI 3
+#define BCM15602_HK_IOLDO_CURR_OVERI 4
+#define BCM15602_HK_VBAT_UNDERV0 5
+#define BCM15602_HK_VBAT_UNDERV1 6
+#define BCM15602_HK_VBAT_UNDERV2 7
+#define BCM15602_HK_PTAT_OVERT 8
+#define BCM15602_HK_PTAT_UNDERT 9
+#define BCM15602_HK_NUM_SLOTS 10
+
+/* macros for PTAT-to-temperature conversion */
+#define TEMP_TO_PTAT_CODE(X) ((X + 273150) / 563)
+#define PTAT_CODE_TO_TEMP(X) (X * 563 - 273150)
+
+/* threshold for housekeeping slots */
+#define BCM15602_HK_TH_ASR_MSTR_CURR \
+ (2500000 / BCM15602_ADC_SCALE_ASR_CURR)
+#define BCM15602_HK_TH_ASR_SLV_CURR \
+ (2500000 / BCM15602_ADC_SCALE_ASR_CURR)
+#define BCM15602_HK_TH_SDSR_CURR \
+ (1200000 / BCM15602_ADC_SCALE_SDSR_CURR)
+#define BCM15602_HK_TH_SDLDO_CURR \
+ (100000 / BCM15602_ADC_SCALE_SDLDO_CURR)
+#define BCM15602_HK_TH_IOLDO_CURR \
+ (100000 / BCM15602_ADC_SCALE_IOLDO_CURR)
+#define BCM15602_HK_TH_VBAT_UNDERV0 \
+ (3200000 / BCM15602_ADC_SCALE_VBAT)
+#define BCM15602_HK_TH_VBAT_UNDERV1 \
+ (3000000 / BCM15602_ADC_SCALE_VBAT)
+#define BCM15602_HK_TH_VBAT_UNDERV2 \
+ (2800000 / BCM15602_ADC_SCALE_VBAT)
+#define BCM15602_HK_TH_PTAT_OVERT TEMP_TO_PTAT_CODE(65000)
+#define BCM15602_HK_TH_PTAT_UNDERT TEMP_TO_PTAT_CODE(45000)
+
+/* adc config state */
+#define BCM15602_ADC_STATE_OFF 0
+#define BCM15602_ADC_STATE_ACTIVE 1
+
+/* adc config oversampling */
+#define BCM15602_ADC_OVSP_1 0x00
+#define BCM15602_ADC_OVSP_2 0x10
+#define BCM15602_ADC_OVSP_4 0x20
+#define BCM15602_ADC_OVSP_8 0x30
+#define BCM15602_ADC_OVSP_16 0x40
+
+/* adc config over/under threshold */
+#define BCM15602_ADC_UNDER_TH 0
+#define BCM15602_ADC_OVER_TH 1
+
+/* interrupt flags */
+#define BCM15602_INT_ASR_OVERI 0
+#define BCM15602_INT_ASR_SHUTDOWN 1
+#define BCM15602_INT_ASR_UV 2
+#define BCM15602_INT_SDSR_OVERI 3
+#define BCM15602_INT_SDSR_SHUTDOWN 4
+#define BCM15602_INT_IOLDO_OVERI 8
+#define BCM15602_INT_IOLDO_SHUTDOWN 9
+#define BCM15602_INT_SDLDO_OVERI 11
+#define BCM15602_INT_SDLDO_SHUTDOWN 12
+#define BCM15602_INT_ADC_HK_INT 16
+#define BCM15602_INT_ADC_CONV_DONE 17
+#define BCM15602_INT_OTP_ECC_FAULT 24
+#define BCM15602_INT_WDT_EXP 25
+#define BCM15602_INT_WDT_ALARM 26
+#define BCM15602_INT_THERM_TRIP 27
+#define BCM15602_INT_THERM_TRIP_DONE 28
+
+/* regulator names */
+#define BCM15602_REGLTR_NAME_SDLDO "bcm15602_sdldo"
+#define BCM15602_REGLTR_NAME_IOLDO "bcm15602_ioldo"
+#define BCM15602_REGLTR_NAME_ASR "bcm15602_asr"
+#define BCM15602_REGLTR_NAME_SDSR "bcm15602_sdsr"
+
+/* silicon versions */
+#define BCM15602_REV_ES 0
+#define BCM15602_REV_A0 1
+#define BCM15602_REV_A1 2
+
+#define BCM15602_NUM_GPIOS 2
+
+/* regulator id enum */
+enum bcm15602_regulator_ids {
+ BCM15602_ID_SDLDO,
+ BCM15602_ID_IOLDO,
+ BCM15602_ID_ASR,
+ BCM15602_ID_SDSR,
+ BCM15602_ID_ALL,
+ BCM15602_NUM_REGULATORS = BCM15602_ID_ALL,
+};
+
+/* driver data structure */
+struct bcm15602_chip {
+ struct device *dev;
+ struct regmap *regmap;
+ struct regulator_dev **rdevs;
+ struct bcm15602_platform_data *pdata;
+ u8 rev_id;
+ u16 hk_status;
+ bool wdt_enabled;
+
+ /* kernel thread for waiting for reset after shutdown */
+ struct work_struct reset_work;
+
+ /* completion used for initialization */
+ struct completion reset_complete;
+
+ /* bitmask for tracking enabled regulators */
+ unsigned long reg_enabled_mask;
+
+ /* flags used for serialization */
+ unsigned long adc_conv_busy;
+ unsigned long hk_read_busy;
+
+ /* completion used for signaling end of adc conversion */
+ struct completion adc_conv_complete;
+};
+
+/* platform data structure */
+struct bcm15602_platform_data {
+ int pon_gpio;
+ int resetb_gpio;
+ int intb_gpio;
+ unsigned int resetb_irq;
+ unsigned int intb_irq;
+};
+
+/* adc channel configuration */
+struct bcm15602_adc_ctrl {
+ union {
+ struct {
+ u8 chan_num : 4;
+ u8 ovsp : 3;
+ u8 state : 1;
+ };
+ u8 ctrl;
+ };
+ union {
+ struct {
+ u16 threshold : 10;
+ u8 ot_ut : 1;
+ u8 deb_en : 1;
+ u8 th_en : 1;
+ u8 reserved : 3;
+ };
+ u16 th_ctrl;
+ };
+};
+
+int bcm15602_read_byte(struct bcm15602_chip *ddata, u8 addr, u8 *data);
+int bcm15602_write_byte(struct bcm15602_chip *ddata, u8 addr, u8 data);
+int bcm15602_update_bits(struct bcm15602_chip *ddata, u8 addr,
+ unsigned int mask, u8 data);
+int bcm15602_read_adc_chan(struct bcm15602_chip *ddata,
+ int chan_num, u16 *chan_data);
+int bcm15602_read_hk_slot(struct bcm15602_chip *ddata,
+ int slot_num, u16 *slot_data);
+
+void bcm15602_config_sysfs(struct device *dev);
+
+#endif /* BCM15602_REGULATOR_H */
diff --git a/drivers/staging/bcm15602/bcm15602-thermal.c b/drivers/staging/bcm15602/bcm15602-thermal.c
new file mode 100644
index 0000000..22a53b2
--- /dev/null
+++ b/drivers/staging/bcm15602/bcm15602-thermal.c
@@ -0,0 +1,120 @@
+/*
+ * Copyright (C) 2016 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/thermal.h>
+
+#include "bcm15602-regulator.h"
+
+#define DRIVER_NAME "bcm15602-thermal"
+
+#define BCM15602_THERM_NUM_TRIPS 0
+#define BCM15602_THERM_TRIPS_MASK ((1 << BCM15602_THERM_NUM_TRIPS) - 1)
+
+struct bcm15602_thermal {
+ struct bcm15602_chip *bcm15602;
+ struct thermal_zone_device *tz_dev;
+};
+
+static int bcm15602_thermal_get_temp(struct thermal_zone_device *tz_dev,
+ int *temp)
+{
+ struct bcm15602_thermal *bcm15602_thermal = tz_dev->devdata;
+ struct bcm15602_chip *bcm15602 = bcm15602_thermal->bcm15602;
+ u16 chan_data = 0;
+ int ret;
+
+ ret = bcm15602_read_adc_chan(bcm15602, BCM15602_ADC_PTAT, &chan_data);
+ if (ret)
+ return ret;
+
+ *temp = PTAT_CODE_TO_TEMP(chan_data);
+ return 0;
+}
+
+static int bcm15602_thermal_get_mode(struct thermal_zone_device *tz_dev,
+ enum thermal_device_mode *mode)
+{
+ *mode = THERMAL_DEVICE_ENABLED;
+ return 0;
+}
+
+static struct thermal_zone_device_ops bcm15602_thermal_ops = {
+ .get_temp = bcm15602_thermal_get_temp,
+ .get_mode = bcm15602_thermal_get_mode,
+};
+
+static int bcm15602_thermal_probe(struct platform_device *pdev)
+{
+ struct bcm15602_chip *bcm15602 = dev_get_drvdata(pdev->dev.parent);
+ struct bcm15602_thermal *bcm15602_thermal;
+ char tz_name[] = "bcm15602_tz";
+
+ bcm15602_thermal = devm_kzalloc(&pdev->dev,
+ sizeof(struct bcm15602_thermal),
+ GFP_KERNEL);
+ if (!bcm15602_thermal)
+ return -ENOMEM;
+
+ bcm15602_thermal->bcm15602 = bcm15602;
+
+ /* register the thermal zone device */
+ bcm15602_thermal->tz_dev =
+ thermal_zone_device_register(tz_name, BCM15602_THERM_NUM_TRIPS,
+ BCM15602_THERM_TRIPS_MASK,
+ bcm15602_thermal,
+ &bcm15602_thermal_ops,
+ NULL, 0, 0);
+ if (IS_ERR(bcm15602_thermal->tz_dev)) {
+ dev_err(&pdev->dev, "%s: failed to register thermal zone device (%ld)\n",
+ __func__, PTR_ERR(bcm15602_thermal->tz_dev));
+ return -ENODEV;
+ }
+
+ platform_set_drvdata(pdev, bcm15602_thermal);
+
+ return 0;
+}
+
+static int bcm15602_thermal_remove(struct platform_device *pdev)
+{
+ struct bcm15602_thermal *bcm15602_thermal = platform_get_drvdata(pdev);
+
+ if (bcm15602_thermal->tz_dev)
+ thermal_zone_device_unregister(bcm15602_thermal->tz_dev);
+ return 0;
+}
+
+static const struct of_device_id bcm15602_thermal_of_match[] = {
+ { .compatible = "brcm,bcm15602-thermal", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, bcm15602_thermal_of_match);
+
+static struct platform_driver bcm15602_thermal_driver = {
+ .probe = bcm15602_thermal_probe,
+ .remove = bcm15602_thermal_remove,
+ .driver = {
+ .name = DRIVER_NAME,
+ .of_match_table = bcm15602_thermal_of_match,
+ },
+};
+module_platform_driver(bcm15602_thermal_driver);
+
+MODULE_AUTHOR("Trevor Bunker <trevorbunker@google.com>");
+MODULE_DESCRIPTION("BCM15602 Thermal Sensor Driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/staging/comedi/drivers/ni_mio_common.c b/drivers/staging/comedi/drivers/ni_mio_common.c
index 18c5312f..0fa85d5 100644
--- a/drivers/staging/comedi/drivers/ni_mio_common.c
+++ b/drivers/staging/comedi/drivers/ni_mio_common.c
@@ -5407,11 +5407,11 @@ static int ni_E_init(struct comedi_device *dev,
/* Digital I/O (PFI) subdevice */
s = &dev->subdevices[NI_PFI_DIO_SUBDEV];
s->type = COMEDI_SUBD_DIO;
- s->subdev_flags = SDF_READABLE | SDF_WRITABLE | SDF_INTERNAL;
s->maxdata = 1;
if (devpriv->is_m_series) {
s->n_chan = 16;
s->insn_bits = ni_pfi_insn_bits;
+ s->subdev_flags = SDF_READABLE | SDF_WRITABLE | SDF_INTERNAL;
ni_writew(dev, s->state, NI_M_PFI_DO_REG);
for (i = 0; i < NUM_PFI_OUTPUT_SELECT_REGS; ++i) {
@@ -5420,6 +5420,7 @@ static int ni_E_init(struct comedi_device *dev,
}
} else {
s->n_chan = 10;
+ s->subdev_flags = SDF_INTERNAL;
}
s->insn_config = ni_pfi_insn_config;
diff --git a/drivers/staging/comedi/drivers/quatech_daqp_cs.c b/drivers/staging/comedi/drivers/quatech_daqp_cs.c
index 802f51e..1719605 100644
--- a/drivers/staging/comedi/drivers/quatech_daqp_cs.c
+++ b/drivers/staging/comedi/drivers/quatech_daqp_cs.c
@@ -642,7 +642,7 @@ static int daqp_ao_insn_write(struct comedi_device *dev,
/* Make sure D/A update mode is direct update */
outb(0, dev->iobase + DAQP_AUX_REG);
- for (i = 0; i > insn->n; i++) {
+ for (i = 0; i < insn->n; i++) {
unsigned int val = data[i];
int ret;
diff --git a/drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c b/drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c
index eaeb3c5..cb95c3e 100644
--- a/drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c
+++ b/drivers/staging/fsl-mc/bus/irq-gic-v3-its-fsl-mc-msi.c
@@ -75,6 +75,8 @@ int __init its_fsl_mc_msi_init(void)
for (np = of_find_matching_node(NULL, its_device_id); np;
np = of_find_matching_node(np, its_device_id)) {
+ if (!of_device_is_available(np))
+ continue;
if (!of_property_read_bool(np, "msi-controller"))
continue;
diff --git a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
index ea9a0c2..4ff2931 100644
--- a/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
+++ b/drivers/staging/lustre/lnet/klnds/o2iblnd/o2iblnd_cb.c
@@ -1299,11 +1299,6 @@ kiblnd_connect_peer(struct kib_peer *peer)
goto failed2;
}
- LASSERT(cmid->device);
- CDEBUG(D_NET, "%s: connection bound to %s:%pI4h:%s\n",
- libcfs_nid2str(peer->ibp_nid), dev->ibd_ifname,
- &dev->ibd_ifip, cmid->device->name);
-
return;
failed2:
@@ -3005,8 +3000,19 @@ kiblnd_cm_callback(struct rdma_cm_id *cmid, struct rdma_cm_event *event)
} else {
rc = rdma_resolve_route(
cmid, *kiblnd_tunables.kib_timeout * 1000);
- if (!rc)
+ if (!rc) {
+ struct kib_net *net = peer->ibp_ni->ni_data;
+ struct kib_dev *dev = net->ibn_dev;
+
+ CDEBUG(D_NET, "%s: connection bound to "\
+ "%s:%pI4h:%s\n",
+ libcfs_nid2str(peer->ibp_nid),
+ dev->ibd_ifname,
+ &dev->ibd_ifip, cmid->device->name);
+
return 0;
+ }
+
/* Can't initiate route resolution */
CERROR("Can't resolve route for %s: %d\n",
libcfs_nid2str(peer->ibp_nid), rc);
diff --git a/drivers/staging/lustre/lnet/lnet/config.c b/drivers/staging/lustre/lnet/lnet/config.c
index 9e2183f..7a4d5a0 100644
--- a/drivers/staging/lustre/lnet/lnet/config.c
+++ b/drivers/staging/lustre/lnet/lnet/config.c
@@ -354,8 +354,7 @@ lnet_parse_networks(struct list_head *nilist, char *networks)
CERROR("Can't allocate net interface name\n");
goto failed;
}
- strncpy(ni->ni_interfaces[niface], iface,
- strlen(iface));
+ strcpy(ni->ni_interfaces[niface], iface);
niface++;
iface = comma;
} while (iface);
diff --git a/drivers/staging/lustre/lustre/include/obd.h b/drivers/staging/lustre/lustre/include/obd.h
index f6fc4dd..722c33f 100644
--- a/drivers/staging/lustre/lustre/include/obd.h
+++ b/drivers/staging/lustre/lustre/include/obd.h
@@ -253,7 +253,7 @@ struct client_obd {
struct sptlrpc_flavor cl_flvr_mgc; /* fixed flavor of mgc->mgs */
/* the grant values are protected by loi_list_lock below */
- unsigned long cl_dirty_pages; /* all _dirty_ in pahges */
+ unsigned long cl_dirty_pages; /* all _dirty_ in pages */
unsigned long cl_dirty_max_pages; /* allowed w/o rpc */
unsigned long cl_dirty_transit; /* dirty synchronous */
unsigned long cl_avail_grant; /* bytes of credit for ost */
diff --git a/drivers/staging/lustre/lustre/ldlm/ldlm_lock.c b/drivers/staging/lustre/lustre/ldlm/ldlm_lock.c
index d18ab3f..9addcdb 100644
--- a/drivers/staging/lustre/lustre/ldlm/ldlm_lock.c
+++ b/drivers/staging/lustre/lustre/ldlm/ldlm_lock.c
@@ -1489,8 +1489,10 @@ struct ldlm_lock *ldlm_lock_create(struct ldlm_namespace *ns,
return ERR_CAST(res);
lock = ldlm_lock_new(res);
- if (!lock)
+ if (!lock) {
+ ldlm_resource_putref(res);
return ERR_PTR(-ENOMEM);
+ }
lock->l_req_mode = mode;
lock->l_ast_data = data;
@@ -1533,6 +1535,8 @@ struct ldlm_lock *ldlm_lock_create(struct ldlm_namespace *ns,
return ERR_PTR(rc);
}
+
+
/**
* Enqueue (request) a lock.
* On the client this is called from ldlm_cli_enqueue_fini
diff --git a/drivers/staging/lustre/lustre/llite/xattr.c b/drivers/staging/lustre/lustre/llite/xattr.c
index e070adb..57121fd 100644
--- a/drivers/staging/lustre/lustre/llite/xattr.c
+++ b/drivers/staging/lustre/lustre/llite/xattr.c
@@ -103,7 +103,11 @@ ll_xattr_set_common(const struct xattr_handler *handler,
__u64 valid;
int rc;
- if (flags == XATTR_REPLACE) {
+ /* When setxattr() is called with a size of 0 the value is
+ * unconditionally replaced by "". When removexattr() is
+ * called we get a NULL value and XATTR_REPLACE for flags.
+ */
+ if (!value && flags == XATTR_REPLACE) {
ll_stats_ops_tally(ll_i2sbi(inode), LPROC_LL_REMOVEXATTR, 1);
valid = OBD_MD_FLXATTRRM;
} else {
diff --git a/drivers/staging/lustre/lustre/lmv/lmv_obd.c b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
index cd19ce8..514f078 100644
--- a/drivers/staging/lustre/lustre/lmv/lmv_obd.c
+++ b/drivers/staging/lustre/lustre/lmv/lmv_obd.c
@@ -684,7 +684,7 @@ static int lmv_fid2path(struct obd_export *exp, int len, void *karg,
memmove(ptr + strlen(gf->gf_path) + 1, ptr,
strlen(ori_gf->gf_path));
- strncpy(ptr, gf->gf_path, strlen(gf->gf_path));
+ strcpy(ptr, gf->gf_path);
ptr += strlen(gf->gf_path);
*ptr = '/';
}
@@ -2928,7 +2928,7 @@ int lmv_unpack_md(struct obd_export *exp, struct lmv_stripe_md **lsmp,
if (lsm && !lmm) {
int i;
- for (i = 1; i < lsm->lsm_md_stripe_count; i++) {
+ for (i = 0; i < lsm->lsm_md_stripe_count; i++) {
/*
* For migrating inode, the master stripe and master
* object will be the same, so do not need iput, see
diff --git a/drivers/staging/lustre/lustre/osc/osc_cache.c b/drivers/staging/lustre/lustre/osc/osc_cache.c
index 4bbe219..1a8c9f5 100644
--- a/drivers/staging/lustre/lustre/osc/osc_cache.c
+++ b/drivers/staging/lustre/lustre/osc/osc_cache.c
@@ -1542,7 +1542,7 @@ static int osc_enter_cache_try(struct client_obd *cli,
if (rc < 0)
return 0;
- if (cli->cl_dirty_pages <= cli->cl_dirty_max_pages &&
+ if (cli->cl_dirty_pages < cli->cl_dirty_max_pages &&
atomic_long_read(&obd_dirty_pages) + 1 <= obd_max_dirty_pages) {
osc_consume_write_grant(cli, &oap->oap_brw_page);
if (transient) {
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index c16927a..395c7a2 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -11,7 +11,6 @@
* (at your option) any later version.
*/
-#include <asm/cacheflush.h>
#include <linux/clk.h>
#include <linux/mm.h>
#include <linux/pagemap.h>
@@ -24,6 +23,8 @@
#include <media/v4l2-ioctl.h>
#include <media/v4l2-mc.h>
+#include <asm/cacheflush.h>
+
#include "iss_video.h"
#include "iss.h"
diff --git a/drivers/staging/rtl8192u/r8192U_core.c b/drivers/staging/rtl8192u/r8192U_core.c
index 457eeb5..5fe9593 100644
--- a/drivers/staging/rtl8192u/r8192U_core.c
+++ b/drivers/staging/rtl8192u/r8192U_core.c
@@ -1705,6 +1705,8 @@ static short rtl8192_usb_initendpoints(struct net_device *dev)
priv->rx_urb[16] = usb_alloc_urb(0, GFP_KERNEL);
priv->oldaddr = kmalloc(16, GFP_KERNEL);
+ if (!priv->oldaddr)
+ return -ENOMEM;
oldaddr = priv->oldaddr;
align = ((long)oldaddr) & 3;
if (align) {
diff --git a/drivers/staging/rtl8712/mlme_linux.c b/drivers/staging/rtl8712/mlme_linux.c
index af7c4a4..590d43c 100644
--- a/drivers/staging/rtl8712/mlme_linux.c
+++ b/drivers/staging/rtl8712/mlme_linux.c
@@ -158,7 +158,7 @@ void r8712_report_sec_ie(struct _adapter *adapter, u8 authmode, u8 *sec_ie)
p = buff;
p += sprintf(p, "ASSOCINFO(ReqIEs=");
len = sec_ie[1] + 2;
- len = (len < IW_CUSTOM_MAX) ? len : IW_CUSTOM_MAX - 1;
+ len = (len < IW_CUSTOM_MAX) ? len : IW_CUSTOM_MAX;
for (i = 0; i < len; i++)
p += sprintf(p, "%02x", sec_ie[i]);
p += sprintf(p, ")");
diff --git a/drivers/staging/rtl8712/rtl871x_mlme.c b/drivers/staging/rtl8712/rtl871x_mlme.c
index c1feef3..9a5da7f8 100644
--- a/drivers/staging/rtl8712/rtl871x_mlme.c
+++ b/drivers/staging/rtl8712/rtl871x_mlme.c
@@ -1365,7 +1365,7 @@ sint r8712_restruct_sec_ie(struct _adapter *adapter, u8 *in_ie,
u8 *out_ie, uint in_len)
{
u8 authmode = 0, match;
- u8 sec_ie[255], uncst_oui[4], bkup_ie[255];
+ u8 sec_ie[IW_CUSTOM_MAX], uncst_oui[4], bkup_ie[255];
u8 wpa_oui[4] = {0x0, 0x50, 0xf2, 0x01};
uint ielength, cnt, remove_cnt;
int iEntry;
diff --git a/drivers/staging/rts5208/sd.c b/drivers/staging/rts5208/sd.c
index b0bbb36..4e233f3 100644
--- a/drivers/staging/rts5208/sd.c
+++ b/drivers/staging/rts5208/sd.c
@@ -4110,12 +4110,6 @@ int ext_sd_send_cmd_get_rsp(struct rtsx_chip *chip, u8 cmd_idx,
rtsx_trace(chip);
return STATUS_FAIL;
}
-
- } else if (rsp_type == SD_RSP_TYPE_R0) {
- if ((ptr[3] & 0x1E) != 0x03) {
- rtsx_trace(chip);
- return STATUS_FAIL;
- }
}
}
}
@@ -4976,7 +4970,7 @@ int sd_execute_write_data(struct scsi_cmnd *srb, struct rtsx_chip *chip)
goto SD_Execute_Write_Cmd_Failed;
}
- rtsx_write_register(chip, SD_BYTE_CNT_L, 0xFF, 0x00);
+ retval = rtsx_write_register(chip, SD_BYTE_CNT_L, 0xFF, 0x00);
if (retval != STATUS_SUCCESS) {
rtsx_trace(chip);
goto SD_Execute_Write_Cmd_Failed;
diff --git a/drivers/staging/rts5208/xd.c b/drivers/staging/rts5208/xd.c
index 1de02bb..647f6be 100644
--- a/drivers/staging/rts5208/xd.c
+++ b/drivers/staging/rts5208/xd.c
@@ -1247,7 +1247,7 @@ static int xd_copy_page(struct rtsx_chip *chip, u32 old_blk, u32 new_blk,
reg = 0;
rtsx_read_register(chip, XD_CTL, ®);
if (reg & (XD_ECC1_ERROR | XD_ECC2_ERROR)) {
- wait_timeout(100);
+ mdelay(100);
if (detect_card_cd(chip,
XD_CARD) != STATUS_SUCCESS) {
diff --git a/drivers/staging/s2mpb04/Kconfig b/drivers/staging/s2mpb04/Kconfig
new file mode 100644
index 0000000..f732fdf
--- /dev/null
+++ b/drivers/staging/s2mpb04/Kconfig
@@ -0,0 +1,7 @@
+config S2MPB04
+ tristate "S2MPB04 regulator"
+ depends on I2C
+ select REGMAP_I2C
+ help
+ This driver controls the S2MPB04 voltage regulator via I2C.
+
diff --git a/drivers/staging/s2mpb04/Makefile b/drivers/staging/s2mpb04/Makefile
new file mode 100644
index 0000000..9d80a3b
--- /dev/null
+++ b/drivers/staging/s2mpb04/Makefile
@@ -0,0 +1,5 @@
+#
+# Makefile for S2MPB04
+#
+
+obj-$(CONFIG_S2MPB04) += s2mpb04-core.o s2mpb04-regulator.o s2mpb04-sysfs.o s2mpb04-gpio.o s2mpb04-thermal.o
diff --git a/drivers/staging/s2mpb04/s2mpb04-core.c b/drivers/staging/s2mpb04/s2mpb04-core.c
new file mode 100644
index 0000000..3548966
--- /dev/null
+++ b/drivers/staging/s2mpb04/s2mpb04-core.c
@@ -0,0 +1,958 @@
+/*
+ * Copyright (C) 2017 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/delay.h>
+#include <linux/device.h>
+#include <linux/i2c.h>
+#include <linux/interrupt.h>
+#include <linux/mfd/core.h>
+#include <linux/module.h>
+#include <linux/of_gpio.h>
+#include <linux/regmap.h>
+#include <linux/regulator/driver.h>
+#include <linux/regulator/machine.h>
+
+#include "s2mpb04-core.h"
+
+#define DRIVER_NAME "s2mpb04"
+
+/* defines the number of tries to repeat an I2C transaction */
+#define S2MPB04_I2C_RETRY_COUNT 10
+
+/* defines the delay in ms for reset completion */
+#define S2MPB04_PON_RESET_DELAY 2
+
+/* defines the timeout in jiffies for reset completion */
+#define S2MPB04_PON_RESET_TIMEOUT msecs_to_jiffies(15)
+
+/* defines the number of retries for powering on */
+#define S2MPB04_PON_RETRY_CNT 4
+
+/* defines the timeout in jiffies for reset completion after shutdown */
+#define S2MPB04_SHUTDOWN_RESET_TIMEOUT msecs_to_jiffies(10000)
+
+/* defines the timeout in jiffies for ADC conversion completion */
+#define S2MPB04_ADC_CONV_TIMEOUT msecs_to_jiffies(100)
+
+/* defines the number of tries to recover chip */
+#define S2MPB04_RECOVER_RETRY_COUNT 3
+
+static int s2mpb04_pdn_seq_en(struct s2mpb04_core *ddata);
+static int s2mpb04_pdn_seq_dis(struct s2mpb04_core *ddata);
+static int s2mpb04_prepare_pon(struct s2mpb04_core *ddata);
+static int s2mpb04_chip_init(struct s2mpb04_core *ddata);
+static int s2mpb04_core_fixup(struct s2mpb04_core *ddata);
+static void s2mpb04_print_status(struct s2mpb04_core *ddata);
+
+static const struct mfd_cell s2mpb04_devs[] = {
+ {
+ .name = "s2mpb04-regulator",
+ .of_compatible = "samsung,s2mpb04-regulator",
+ },
+ {
+ .name = "s2mpb04-gpio",
+ .of_compatible = "samsung,s2mpb04-gpio",
+ },
+ /*
+ * Disable s2mpb04 thermal zone to reduce overall i2c xfer.
+ * See b/122984225.
+ */
+#if 0
+ {
+ .name = "s2mpb04-thermal",
+ .of_compatible = "samsung,s2mpb04-thermal",
+ },
+#endif
+};
+
+static const struct regmap_config s2mpb04_regmap_config = {
+ .reg_bits = 8,
+ .val_bits = 8,
+};
+
+int s2mpb04_toggle_pon(struct s2mpb04_core *ddata)
+{
+ unsigned long timeout;
+
+ dev_info(ddata->dev, "%s: toggling PON\n", __func__);
+
+ reinit_completion(&ddata->init_complete);
+
+ /* disable hardware sequence when SW is initiating power-down.
+ * hw-driven power-down sequence is required only in case of a
+ * TSD event and has 4ms extra latency.
+ */
+ s2mpb04_pdn_seq_dis(ddata);
+
+ gpio_set_value_cansleep(ddata->pdata->pon_gpio, 0);
+
+ /* force state machine in low power state */
+ s2mpb04_prepare_pon(ddata);
+
+ usleep_range(20, 25);
+ gpio_set_value_cansleep(ddata->pdata->pon_gpio, 1);
+
+ /* wait for chip to come out of reset, signaled by resetb interrupt */
+ timeout = wait_for_completion_timeout(&ddata->init_complete,
+ S2MPB04_SHUTDOWN_RESET_TIMEOUT);
+ if (!timeout)
+ dev_err(ddata->dev,
+ "%s: timeout waiting for device to return from reset\n",
+ __func__);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(s2mpb04_toggle_pon);
+
+/* When chip is hung resetb is stuck high. Function to toggle PON to recover
+ * the chip but don't wait for resetb transition
+ */
+int s2mpb04_toggle_pon_no_resetb_wait(struct s2mpb04_core *ddata)
+{
+ dev_dbg(ddata->dev, "%s: toggling PON\n", __func__);
+
+ gpio_set_value_cansleep(ddata->pdata->pon_gpio, 0);
+
+ /* force state machine in low power state */
+ s2mpb04_prepare_pon(ddata);
+
+ usleep_range(20, 25);
+ gpio_set_value_cansleep(ddata->pdata->pon_gpio, 1);
+
+ /* give the chip some time to power on */
+ msleep(S2MPB04_PON_RESET_DELAY);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(s2mpb04_toggle_pon_no_resetb_wait);
+
+int s2mpb04_dump_regs(struct s2mpb04_core *ddata)
+{
+ u8 reg_data;
+ int i;
+
+ for (i = 0; i <= S2MPB04_REG_SEQ2; i++) {
+ s2mpb04_read_byte(ddata, i, ®_data);
+ dev_info(ddata->dev, "[0x%02x] = 0x%02x\n", i, reg_data);
+ }
+ for (i = S2MPB04_REG_ADC_CTRL; i <= S2MPB04_REG_ADC_DBNC_CTRL2; i++) {
+ s2mpb04_read_byte(ddata, i, ®_data);
+ dev_info(ddata->dev, "[0x%02x] = 0x%02x\n", i, reg_data);
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(s2mpb04_dump_regs);
+
+int s2mpb04_read_byte(struct s2mpb04_core *ddata, u8 addr, u8 *data)
+{
+ int ret;
+ unsigned int val;
+ int retry_cnt = 0;
+
+ do {
+ ret = regmap_read(ddata->regmap, addr, &val);
+ if (!ret) {
+ *data = (u8)val;
+ return 0;
+ }
+
+ dev_err(ddata->dev,
+ "failed to read addr 0x%.2x (%d), retry %d\n",
+ addr, ret, retry_cnt);
+ } while (++retry_cnt < S2MPB04_I2C_RETRY_COUNT);
+
+ dev_err(ddata->dev, "%s: failed with %d retries, power cycling device\n",
+ __func__, S2MPB04_I2C_RETRY_COUNT);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(s2mpb04_read_byte);
+
+int s2mpb04_read_bytes(struct s2mpb04_core *ddata, u8 addr, u8 *data,
+ size_t count)
+{
+ int ret;
+ int retry_cnt = 0;
+
+ do {
+ ret = regmap_bulk_read(ddata->regmap, addr, data, count);
+ if (!ret)
+ return 0;
+
+ dev_err(ddata->dev, "%s: failed to read %zd bytes from addr 0x%.2x (%d), retry %d\n",
+ __func__, count, addr, ret, retry_cnt);
+ } while (++retry_cnt < S2MPB04_I2C_RETRY_COUNT);
+
+ dev_err(ddata->dev, "%s: failed with %d retries, power cycling device\n",
+ __func__, S2MPB04_I2C_RETRY_COUNT);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(s2mpb04_read_bytes);
+
+int s2mpb04_write_byte(struct s2mpb04_core *ddata, u8 addr, u8 data)
+{
+ int ret;
+ int retry_cnt = 0;
+
+ do {
+ ret = regmap_write(ddata->regmap, addr, data);
+ if (!ret)
+ return 0;
+
+ dev_err(ddata->dev,
+ "failed to write addr 0x%.2x (%d), retry %d\n",
+ addr, ret, retry_cnt);
+ } while (++retry_cnt < S2MPB04_I2C_RETRY_COUNT);
+
+ dev_err(ddata->dev, "%s: failed with %d retries, power cycling device\n",
+ __func__, S2MPB04_I2C_RETRY_COUNT);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(s2mpb04_write_byte);
+
+int s2mpb04_write_bytes(struct s2mpb04_core *ddata, u8 addr, u8 *data,
+ size_t count)
+{
+ int ret;
+ int retry_cnt = 0;
+
+ do {
+ ret = regmap_bulk_write(ddata->regmap, addr, data, count);
+ if (!ret)
+ return 0;
+
+ dev_err(ddata->dev,
+ "failed to write %zd bytes to addr 0x%.2x (%d), retry %d\n",
+ count, addr, ret, retry_cnt);
+ } while (++retry_cnt < S2MPB04_I2C_RETRY_COUNT);
+
+ dev_err(ddata->dev, "%s: failed with %d retries, power cycling device\n",
+ __func__, S2MPB04_I2C_RETRY_COUNT);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(s2mpb04_write_bytes);
+
+int s2mpb04_update_bits(struct s2mpb04_core *ddata, u8 addr,
+ unsigned int mask, u8 data)
+{
+ int ret;
+ int retry_cnt = 0;
+
+ do {
+ ret = regmap_update_bits(ddata->regmap, addr, mask, data);
+ if (!ret)
+ return 0;
+
+ dev_err(ddata->dev,
+ "failed to update addr 0x%.2x (%d), retry %d\n",
+ addr, ret, retry_cnt);
+ } while (++retry_cnt < S2MPB04_I2C_RETRY_COUNT);
+
+ dev_err(ddata->dev, "%s: failed with %d retries, power cycling device\n",
+ __func__, S2MPB04_I2C_RETRY_COUNT);
+
+ return -EIO;
+}
+EXPORT_SYMBOL_GPL(s2mpb04_update_bits);
+
+int s2mpb04_read_adc_chan(struct s2mpb04_core *ddata,
+ int chan_num, u8 *chan_data)
+{
+ int ret = 0;
+ unsigned long timeout;
+
+ /* serialize manual adc conversions */
+ if (test_and_set_bit(0, &ddata->adc_conv_busy))
+ return -EAGAIN;
+
+ /* clear adc conversion completion before starting conversion */
+ reinit_completion(&ddata->adc_conv_complete);
+
+ /* write the channel number */
+ ret = s2mpb04_write_byte(ddata, S2MPB04_REG_MUXSEL1, chan_num);
+ if (ret)
+ goto adc_cleanup;
+
+ /* enable the clock at 125 kHz, 1 channel, and 8 samples */
+ ret = s2mpb04_write_byte(ddata, S2MPB04_REG_ADC_CTRL, 0xC2);
+ if (ret)
+ goto adc_cleanup;
+
+ dev_dbg(ddata->dev, "%s: started ADC conversion\n", __func__);
+
+ /* wait for completion signaled by interrupt */
+ timeout = wait_for_completion_timeout(&ddata->adc_conv_complete,
+ S2MPB04_ADC_CONV_TIMEOUT);
+ if (!timeout) {
+ ret = -ETIMEDOUT;
+ dev_err(ddata->dev, "%s: ADC conversion timeout\n", __func__);
+ goto adc_cleanup;
+ }
+
+ /* read and format the conversion result */
+ ret = s2mpb04_read_byte(ddata, S2MPB04_REG_ADC0DATA, chan_data);
+ if (ret)
+ goto adc_cleanup;
+
+ dev_dbg(ddata->dev, "%s: chan_data 0x%02x\n", __func__, *chan_data);
+
+adc_cleanup:
+
+ /* set ADC muxsel to 0 (no monitoring). This is a workaround to prevent
+ * disabling ADC when it's connected to temperature sensor */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_MUXSEL1, 0x0);
+
+ /*
+ * Disable thermal shutdown when disabling the ADC. This is a workaround
+ * for a silicon bug that causes thermal shutdown comparator input to be
+ * shorted to ground for a short duration when the ADC is enabled.
+ */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_TSD_CTRL, 0x20);
+
+ /* disable the ADC clock */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_ADC_CTRL, 0x42);
+
+ /* Re-enable thermal shutdown */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_TSD_CTRL, 0xA0);
+
+ /* release adc conversion */
+ clear_bit(0, &ddata->adc_conv_busy);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(s2mpb04_read_adc_chan);
+
+#define NOTIFY(id, event) s2mpb04_regulator_notify(id, event)
+
+/* print the device id */
+static void s2mpb04_print_id(struct s2mpb04_core *ddata)
+{
+ struct device *dev = ddata->dev;
+ u8 reg_data;
+ int ret;
+
+ ret = s2mpb04_read_byte(ddata, S2MPB04_REG_CHIP_ID, ®_data);
+ if (!ret) {
+ dev_info(dev, "ID: 0x%02X\n", reg_data);
+ ddata->rev_id = reg_data;
+ } else {
+ dev_err(dev, "%s: Could not read PMIC ID (%d)\n", __func__,
+ ret);
+ }
+}
+
+/* print the status register */
+static void s2mpb04_print_status(struct s2mpb04_core *ddata)
+{
+ struct device *dev = ddata->dev;
+ u8 flags[3], status[2];
+ int ret;
+
+ ret = s2mpb04_read_bytes(ddata, S2MPB04_REG_INT1, flags, 3);
+ if (!ret)
+ dev_info(dev, "%s: Flags: 0x%02x, 0x%02x, 0x%02x\n",
+ __func__, flags[0], flags[1], flags[2]);
+
+ ret = s2mpb04_read_bytes(ddata, S2MPB04_REG_STATUS1, status, 2);
+ if (!ret)
+ dev_info(dev, "%s: Status: 0x%02x, 0x%02x\n", __func__,
+ status[0], status[1]);
+}
+
+/* handle an interrupt flag */
+static int s2mpb04_handle_int(struct s2mpb04_core *ddata,
+ unsigned int flag_num)
+{
+ struct device *dev = ddata->dev;
+
+ dev_dbg(dev, "%s: flag %d\n", __func__, flag_num);
+
+ switch (flag_num) {
+ case S2MPB04_INT_PONR:
+ dev_dbg(dev, "%s: Observed PON rising edge\n", __func__);
+ break;
+
+ case S2MPB04_INT_PONF:
+ dev_dbg(dev, "%s: Observed PON falling edge\n", __func__);
+ break;
+
+ case S2MPB04_INT_ADC_DONE:
+ dev_dbg(dev, "%s: completing adc conversion\n", __func__);
+ complete(&ddata->adc_conv_complete);
+ break;
+
+ case S2MPB04_INT_SMPS1_OI:
+ dev_err(dev, "Detected SMPS1 over-current event\n");
+ NOTIFY(S2MPB04_ID_SMPS1,
+ REGULATOR_EVENT_OVER_CURRENT | REGULATOR_EVENT_FAIL);
+ break;
+
+ case S2MPB04_INT_SMPS2_OI:
+ dev_err(dev, "Detected SMPS2 over-current event\n");
+ NOTIFY(S2MPB04_ID_SMPS2,
+ REGULATOR_EVENT_OVER_CURRENT | REGULATOR_EVENT_FAIL);
+ break;
+
+ case S2MPB04_INT_SMPS1_UV:
+ dev_err(dev, "Detected SMPS1 under-voltage event\n");
+ NOTIFY(S2MPB04_ID_SMPS1,
+ REGULATOR_EVENT_UNDER_VOLTAGE | REGULATOR_EVENT_FAIL);
+ break;
+
+ case S2MPB04_INT_LDO1_OI:
+ dev_err(dev, "Detected LDO1 over-current event\n");
+ NOTIFY(S2MPB04_ID_LDO1,
+ REGULATOR_EVENT_OVER_CURRENT | REGULATOR_EVENT_FAIL);
+ break;
+
+ case S2MPB04_INT_LDO2_OI:
+ dev_err(dev, "Detected LDO2 over-current event\n");
+ NOTIFY(S2MPB04_ID_LDO2,
+ REGULATOR_EVENT_OVER_CURRENT | REGULATOR_EVENT_FAIL);
+ break;
+
+ case S2MPB04_INT_TH_TINT:
+ dev_err(dev, "%s: unhandled thermal warm interrupt\n",
+ __func__);
+ break;
+
+ case S2MPB04_INT_TH_TRIPF:
+ dev_err(dev, "%s: therm_trip has returned to normal\n",
+ __func__);
+ break;
+
+ case S2MPB04_INT_WATCHDOG:
+ case S2MPB04_INT_TH_TRIPR:
+ case S2MPB04_INT_TSD:
+ if (flag_num == S2MPB04_INT_WATCHDOG)
+ dev_err(dev, "%s: Watchdog timer expired\n", __func__);
+ else if (flag_num == S2MPB04_INT_TH_TRIPR)
+ dev_err(dev, "%s: therm_trip asserted\n", __func__);
+ else
+ dev_err(dev, "%s: thermal shutdown\n", __func__);
+
+ /* notify regulator clients of failures */
+ NOTIFY(S2MPB04_ID_SMPS1, REGULATOR_EVENT_FAIL);
+ NOTIFY(S2MPB04_ID_SMPS2, REGULATOR_EVENT_FAIL);
+ NOTIFY(S2MPB04_ID_LDO2, REGULATOR_EVENT_FAIL);
+ NOTIFY(S2MPB04_ID_LDO1, REGULATOR_EVENT_FAIL);
+ break;
+
+ default:
+ dev_dbg(dev, "%s: Reserved flag %d\n", __func__, flag_num);
+ break;
+ }
+
+ return 0;
+}
+
+/* find pending interrupt flags */
+static int s2mpb04_check_int_flags(struct s2mpb04_core *ddata)
+{
+ u8 flags[3], flag_mask;
+ unsigned int first_bit, flag_num;
+ int ret = 0;
+ int i;
+
+ /* read interrupt status flags */
+ ret = s2mpb04_read_bytes(ddata, S2MPB04_REG_INT1, flags, 3);
+ if (ret)
+ return ret;
+
+ dev_dbg(ddata->dev,
+ "%s: [0] = 0x%02x, [1] = 0x%02x, [2] = 0x%02x\n",
+ __func__, flags[0], flags[1], flags[2]);
+
+ /* iterate through each interrupt */
+ for (i = 0; i < 3; i++) {
+ while (flags[i]) {
+ /* find first set interrupt flag */
+ first_bit = ffs(flags[i]);
+ flag_mask = 1 << (first_bit - 1);
+ flag_num = (i * 8) + (first_bit - 1);
+
+ /* handle interrupt */
+ ret = s2mpb04_handle_int(ddata, flag_num);
+
+ flags[i] &= ~flag_mask;
+ }
+ }
+
+ return ret;
+}
+
+/* kernel thread to notify regulator fail event and clear any pending
+ * interrupt status
+ */
+static void s2mpb04_reset_work(struct work_struct *data)
+{
+ struct s2mpb04_core *ddata = container_of(data, struct s2mpb04_core,
+ reset_work);
+
+ /* notify regulators of shutdown event */
+ dev_err(ddata->dev,
+ "%s: Notifying regulators of shutdown event\n", __func__);
+ NOTIFY(S2MPB04_ID_SMPS1, REGULATOR_EVENT_FAIL);
+ NOTIFY(S2MPB04_ID_SMPS2, REGULATOR_EVENT_FAIL);
+ NOTIFY(S2MPB04_ID_LDO1, REGULATOR_EVENT_FAIL);
+ NOTIFY(S2MPB04_ID_LDO2, REGULATOR_EVENT_FAIL);
+}
+
+/* kernel thread to detect if PMIC is hung and run the recovery sequence */
+static void s2mpb04_recover_work(struct work_struct *data)
+{
+ struct s2mpb04_core *ddata = container_of(data, struct s2mpb04_core,
+ recover_work);
+ struct s2mpb04_platform_data *pdata = ddata->pdata;
+ struct device *dev = ddata->dev;
+ unsigned long timeout;
+
+ /* disable interrupts during recovery sequence */
+ disable_irq(pdata->resetb_irq);
+ disable_irq(pdata->intb_irq);
+
+ /* when chip is hung resetb is stuck high. Toggle PON to recover chip
+ * but don't wait for resetb transition
+ */
+ s2mpb04_toggle_pon_no_resetb_wait(ddata);
+
+ /* check if recovery sequence worked. resetb should now become low
+ * when PON is low
+ */
+ gpio_set_value_cansleep(pdata->pon_gpio, 0);
+ usleep_range(100, 105);
+
+ /* re-enable interrupts */
+ enable_irq(pdata->resetb_irq);
+ enable_irq(pdata->intb_irq);
+
+ if (gpio_get_value(pdata->resetb_gpio)) {
+ dev_err(dev, "%s: recovery sequence failed", __func__);
+ return;
+ }
+
+ dev_info(dev, "%s: recovery sequence success", __func__);
+
+ gpio_set_value_cansleep(ddata->pdata->pon_gpio, 1);
+
+ /* wait for chip to come out of reset and get initialized,
+ * signaled by resetb interrupt handler
+ */
+ timeout = wait_for_completion_timeout(&ddata->init_complete,
+ S2MPB04_SHUTDOWN_RESET_TIMEOUT);
+ if (!timeout)
+ dev_err(ddata->dev,
+ "%s: timeout waiting for device to return from reset\n",
+ __func__);
+}
+
+/* check if chip is hung */
+static bool s2mpb04_is_chip_hung(struct s2mpb04_core *ddata)
+{
+ int ret;
+ u8 chan_data;
+
+ /* Detect if chip is hung by reading ADC */
+ do {
+ ret = s2mpb04_read_adc_chan(ddata,
+ S2MPB04_ADC_VBAT,
+ &chan_data);
+ } while (ret == -EAGAIN);
+
+ /* chip is hung if ADC read timesout or raw vbat data
+ * is 0x10
+ */
+ return ((ret == -ETIMEDOUT) || (chan_data == 0x10));
+}
+
+/* irq handler for resetb pin.
+ * It may not catch falling edge. But, that will not be an issue as intb
+ * interrupt handler will notify regulator fail event.
+ */
+static irqreturn_t s2mpb04_resetb_irq_handler(int irq, void *cookie)
+{
+ struct s2mpb04_core *ddata = (struct s2mpb04_core *)cookie;
+ struct device *dev = ddata->dev;
+ bool is_chip_hung;
+
+ if (gpio_get_value(ddata->pdata->resetb_gpio)) {
+ dev_dbg(dev, "%s: completing reset\n", __func__);
+
+ /* initialize chip */
+ s2mpb04_chip_init(ddata);
+
+ /* check if chip is hung */
+ is_chip_hung = s2mpb04_is_chip_hung(ddata);
+
+ if (!is_chip_hung) {
+ /* Fixup some chip settings that are different than
+ * reset values.
+ */
+ s2mpb04_core_fixup(ddata);
+
+ complete(&ddata->init_complete);
+
+ /* clear recovery attempt counter */
+ ddata->recover_count = 0;
+ } else if (ddata->recover_count < S2MPB04_RECOVER_RETRY_COUNT) {
+ dev_err(dev,
+ "%s: lockup detected, trying recovery: (%d/%d)",
+ __func__,
+ (int)++ddata->recover_count,
+ S2MPB04_RECOVER_RETRY_COUNT);
+ schedule_work(&ddata->recover_work);
+ } else {
+ dev_err(dev, "%s: max recovery attempts reached: %d",
+ __func__, S2MPB04_RECOVER_RETRY_COUNT);
+ }
+ } else {
+ dev_err(dev, "%s: device reset\n", __func__);
+ reinit_completion(&ddata->init_complete);
+ schedule_work(&ddata->reset_work);
+ }
+
+ return IRQ_HANDLED;
+}
+
+/* irq handler for intb pin */
+static irqreturn_t s2mpb04_intb_irq_handler(int irq, void *cookie)
+{
+ struct s2mpb04_core *ddata = (struct s2mpb04_core *)cookie;
+ int ret;
+
+ dev_dbg(ddata->dev, "%s: observed irq\n", __func__);
+
+ while (!gpio_get_value(ddata->pdata->intb_gpio)) {
+ ret = s2mpb04_check_int_flags(ddata);
+ if (ret)
+ return IRQ_RETVAL(ret);
+ }
+
+ return IRQ_HANDLED;
+}
+
+/* get platform data from the device tree */
+static struct s2mpb04_platform_data *s2mpb04_get_platform_data_from_dt
+ (struct device *dev)
+{
+ struct device_node *np = dev->of_node;
+ struct s2mpb04_platform_data *pdata;
+
+ if (!np)
+ return ERR_PTR(-ENODEV);
+
+ pdata = devm_kzalloc(dev, sizeof(*pdata), GFP_KERNEL);
+ if (!pdata)
+ return ERR_PTR(-ENOMEM);
+
+ pdata->pon_gpio = of_get_named_gpio(np, "samsung,pon-gpio", 0);
+ pdata->resetb_gpio = of_get_named_gpio(np, "samsung,resetb-gpio", 0);
+ pdata->intb_gpio = of_get_named_gpio(np, "samsung,intb-gpio", 0);
+ pdata->resetb_irq = gpio_to_irq(pdata->resetb_gpio);
+ pdata->intb_irq = gpio_to_irq(pdata->intb_gpio);
+
+ return pdata;
+}
+
+/* enable all of the interrupts */
+static void s2mpb04_config_ints(struct s2mpb04_core *ddata)
+{
+ u8 bytes[3];
+
+ /* clear any pending interrupts */
+ s2mpb04_read_bytes(ddata, S2MPB04_REG_INT1, bytes, 3);
+
+ /* unmask all (non-reserved) interrupts */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_INT1M, 0x00);
+ s2mpb04_write_byte(ddata, S2MPB04_REG_INT2M, 0x00);
+ s2mpb04_write_byte(ddata, S2MPB04_REG_INT3M, 0x7F);
+}
+
+/* some changes to default configuration based on bringup */
+static int s2mpb04_core_fixup(struct s2mpb04_core *ddata)
+{
+ dev_dbg(ddata->dev, "%s: rev %d\n", __func__, ddata->rev_id);
+
+ s2mpb04_config_ints(ddata);
+
+ /* set SMPS1 output voltage to 0.9125V */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_BUCK1_OUT, 0x62);
+
+ /* set SMPS2 output voltage to 1.15V */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_BUCK2_OUT, 0x88);
+
+ /* set LDO1 output voltage to 1.87V */
+ /* also modify s2mpb04_regulator_enable to match LDO1 fixup */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_LDO1_CTRL, 0x2F);
+
+ /* set LDO2 output voltage to 1.87V */
+ /* also modify s2mpb04_regulator_enable to match LDO2 fixup */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_LDO2_CTRL, 0x2F);
+
+ /* disable watchdog timer */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_WTSR_CTRL, 0x1D);
+
+ /* Disable temprature compensation. This is a workaround to prevent
+ * disabling ADC when it's connected to temperature sensor */
+ s2mpb04_write_byte(ddata, S2MPB04_REG_ADC_CTRL2, 0x5);
+
+ /* enable hardware power-down sequence */
+ s2mpb04_pdn_seq_en(ddata);
+
+ return 0;
+}
+
+/* Enable hardware power down sequence.
+ * PMIC state machine locks up when BGROK drops low in Standby state.
+ * When hardware power down sequence is enabled, on TSD event,
+ * PMIC state machine will first go to power-down state before
+ * going to low-power state. This will make BGROK to drop low
+ * on power-down -> low-power state transition and will prevent
+ * the lockup.
+ */
+static int s2mpb04_pdn_seq_en(struct s2mpb04_core *ddata)
+{
+ dev_dbg(ddata->dev, "%s: enable power down sequence\n",
+ __func__);
+
+ s2mpb04_write_byte(ddata, S2MPB04_REG_OFF_SEQ_CTRL, 0x80);
+ s2mpb04_write_byte(ddata, S2MPB04_REG_SEQ1, 0x44);
+ s2mpb04_write_byte(ddata, S2MPB04_REG_SEQ2, 0x44);
+
+ return 0;
+}
+
+/* Disable hardware power down sequence */
+static int s2mpb04_pdn_seq_dis(struct s2mpb04_core *ddata)
+{
+ dev_dbg(ddata->dev, "%s: disable power down sequence\n",
+ __func__);
+
+ s2mpb04_write_byte(ddata, S2MPB04_REG_OFF_SEQ_CTRL, 0x0);
+
+ return 0;
+}
+
+/* Force state machine in low power state. This should be state machine's
+ * state when PON=0 and BUV (battery under volatge)=0. But, when LDO OI
+ * issue happens, it's stuck in power-down state.
+ *
+ * In low power state, PON 0 -> 1 toggle will initiate power-up sequence
+ * and reload register values from OTP. So, this function should be called
+ * when PON is set to 0.
+ */
+static int s2mpb04_prepare_pon(struct s2mpb04_core *ddata)
+{
+ dev_info(ddata->dev, "%s: applying workaround for LDO OI issue\n",
+ __func__);
+
+ s2mpb04_write_byte(ddata, S2MPB04_REG_SEQ1, 0x66);
+ s2mpb04_write_byte(ddata, S2MPB04_REG_SEQ2, 0x66);
+
+ return 0;
+}
+
+/* initialize the chip */
+static int s2mpb04_chip_init(struct s2mpb04_core *ddata)
+{
+ s2mpb04_print_id(ddata);
+ s2mpb04_print_status(ddata);
+
+ return 0;
+}
+
+static int s2mpb04_probe(struct i2c_client *client,
+ const struct i2c_device_id *dev_id)
+{
+ struct device *dev = &client->dev;
+ struct s2mpb04_core *ddata;
+ struct s2mpb04_platform_data *pdata;
+ unsigned long timeout;
+ int ret;
+ int i;
+
+ /* allocate memory for chip structure */
+ ddata = devm_kzalloc(dev, sizeof(struct s2mpb04_core),
+ GFP_KERNEL);
+ if (!ddata)
+ return -ENOMEM;
+
+ /* set client data */
+ i2c_set_clientdata(client, ddata);
+ dev_set_drvdata(dev, ddata);
+
+ /* get platform data */
+ pdata = dev_get_platdata(dev);
+ if (!pdata) {
+ pdata = s2mpb04_get_platform_data_from_dt(dev);
+ if (IS_ERR(pdata))
+ return PTR_ERR(pdata);
+ }
+
+ /* initialize chip structure */
+ ddata->dev = dev;
+ ddata->pdata = pdata;
+ dev->platform_data = pdata;
+
+ /* initialize some structures */
+ INIT_WORK(&ddata->reset_work, s2mpb04_reset_work);
+ INIT_WORK(&ddata->recover_work, s2mpb04_recover_work);
+
+ /* initialize completions */
+ init_completion(&ddata->init_complete);
+ init_completion(&ddata->adc_conv_complete);
+
+ /* initialize regmap */
+ ddata->regmap = devm_regmap_init_i2c(client, &s2mpb04_regmap_config);
+ if (IS_ERR(ddata->regmap)) {
+ ret = PTR_ERR(ddata->regmap);
+ dev_err(dev,
+ "%s: could not initialize regmap (%d)\n",
+ __func__, ret);
+ return -ENOMEM;
+ }
+
+ /* request GPIOs and IRQs */
+ devm_gpio_request_one(dev, pdata->pon_gpio, GPIOF_OUT_INIT_LOW,
+ "S2MPB04 PON");
+ devm_gpio_request_one(dev, pdata->resetb_gpio, GPIOF_IN,
+ "S2MPB04 RESETB");
+ devm_gpio_request_one(dev, pdata->intb_gpio, GPIOF_IN,
+ "S2MPB04 INTB");
+ ret = devm_request_threaded_irq(dev, pdata->intb_irq, NULL,
+ s2mpb04_intb_irq_handler,
+ IRQF_TRIGGER_FALLING | IRQF_ONESHOT,
+ "s2mpb04-intb", ddata);
+
+ /* initialize chip */
+ s2mpb04_chip_init(ddata);
+
+ /* force state machine in low power state */
+ s2mpb04_prepare_pon(ddata);
+
+ for (i = 0; i < S2MPB04_PON_RETRY_CNT; i++) {
+ dev_dbg(dev, "%s: powering on s2mpb04\n", __func__);
+ gpio_set_value_cansleep(pdata->pon_gpio, 1);
+
+ /* give the chip some time to power on */
+ msleep(S2MPB04_PON_RESET_DELAY);
+
+ /* poll on the gpio until it goes high or until we timeout */
+ timeout = jiffies + S2MPB04_PON_RESET_TIMEOUT;
+ while (!gpio_get_value(pdata->resetb_gpio) &&
+ time_before(jiffies, timeout)) {
+ usleep_range(100, 105);
+ }
+
+ if (gpio_get_value(pdata->resetb_gpio))
+ break;
+
+ dev_err(dev, "%s: powering on timed out, try (%d/%d)\n",
+ __func__, i + 1, S2MPB04_PON_RETRY_CNT);
+
+ gpio_set_value_cansleep(pdata->pon_gpio, 0);
+ usleep_range(100, 105);
+ }
+
+ /* check for failure */
+ if (i == S2MPB04_PON_RETRY_CNT) {
+ ret = -ETIMEDOUT;
+ dev_err(dev, "%s: powering on failed\n", __func__);
+ goto error_reset;
+ }
+
+ /* create sysfs attributes */
+ s2mpb04_config_sysfs(dev);
+
+ /* Register resetb irq handler after setting PON to avoid IRQ
+ * firing unnecessarily.
+ */
+ ret = devm_request_threaded_irq(dev, pdata->resetb_irq, NULL,
+ s2mpb04_resetb_irq_handler,
+ IRQF_TRIGGER_FALLING |
+ IRQF_TRIGGER_RISING | IRQF_ONESHOT,
+ "s2mpb04-resetb", ddata);
+
+ /* fixup some chip settings that are different than reset values */
+ s2mpb04_core_fixup(ddata);
+
+ return mfd_add_devices(dev, -1, s2mpb04_devs, ARRAY_SIZE(s2mpb04_devs),
+ NULL, 0, NULL);
+
+error_reset:
+ return ret;
+}
+
+#ifdef CONFIG_PM
+static int s2mpb04_suspend(struct device *dev)
+{
+ struct s2mpb04_platform_data *pdata;
+
+ pdata = dev_get_platdata(dev);
+ if (pdata)
+ enable_irq_wake(pdata->intb_irq);
+ return 0;
+}
+
+static int s2mpb04_resume(struct device *dev)
+{
+ struct s2mpb04_platform_data *pdata;
+
+ pdata = dev_get_platdata(dev);
+ if (pdata)
+ disable_irq_wake(pdata->intb_irq);
+ return 0;
+}
+
+static const struct dev_pm_ops s2mpb04_dev_pm_ops = {
+ .suspend = s2mpb04_suspend,
+ .resume = s2mpb04_resume,
+};
+#endif
+
+static const struct of_device_id s2mpb04_dt_ids[] = {
+ { .compatible = "samsung,s2mpb04", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, s2mpb04_dt_ids);
+
+static const struct i2c_device_id s2mpb04_id_table[] = {
+ { .name = DRIVER_NAME, .driver_data = 0 },
+ { },
+};
+MODULE_DEVICE_TABLE(i2c, s2mpb04_id_table);
+
+static struct i2c_driver s2mpb04_driver = {
+ .driver = {
+ .name = DRIVER_NAME,
+ .owner = THIS_MODULE,
+#ifdef CONFIG_PM
+ .pm = &s2mpb04_dev_pm_ops,
+#endif
+ },
+ .probe = s2mpb04_probe,
+ .id_table = s2mpb04_id_table,
+};
+
+module_i2c_driver(s2mpb04_driver);
+
+MODULE_AUTHOR("Trevor Bunker <trevorbunker@google.com>");
+MODULE_DESCRIPTION("S2MPB04 Device Driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/staging/s2mpb04/s2mpb04-core.h b/drivers/staging/s2mpb04/s2mpb04-core.h
new file mode 100644
index 0000000..3df389c
--- /dev/null
+++ b/drivers/staging/s2mpb04/s2mpb04-core.h
@@ -0,0 +1,193 @@
+/*
+ * Copyright (C) 2017 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#ifndef S2MPB04_REGULATOR_H
+#define S2MPB04_REGULATOR_H
+
+/* registers */
+#define S2MPB04_REG_CHIP_ID 0x00
+#define S2MPB04_REG_INT1 0x01
+#define S2MPB04_REG_INT2 0x02
+#define S2MPB04_REG_INT3 0x03
+#define S2MPB04_REG_INT1M 0x04
+#define S2MPB04_REG_INT2M 0x05
+#define S2MPB04_REG_INT3M 0x06
+#define S2MPB04_REG_STATUS1 0x07
+#define S2MPB04_REG_STATUS2 0x08
+#define S2MPB04_REG_TIME_CTRL 0x0E
+#define S2MPB04_REG_BUCK1_CTRL 0x0F
+#define S2MPB04_REG_BUCK1_OUT 0x10
+#define S2MPB04_REG_BUCK2_CTRL 0x11
+#define S2MPB04_REG_BUCK2_OUT 0x12
+#define S2MPB04_REG_BUCK_RAMP 0x13
+#define S2MPB04_REG_BUCK_IRQ_CTRL1 0x15
+#define S2MPB04_REG_BUCK_IRQ_CTRL2 0x16
+#define S2MPB04_REG_LDO1_CTRL 0x17
+#define S2MPB04_REG_LDO2_CTRL 0x18
+#define S2MPB04_REG_LDO_DSCH 0x19
+#define S2MPB04_REG_LDO_CTRL 0x1A
+#define S2MPB04_REG_TSD_CTRL 0x1D
+#define S2MPB04_REG_WTSR_CTRL 0x1E
+#define S2MPB04_REG_GPIO_CTRL 0x1F
+#define S2MPB04_REG_GPIO_A 0x20
+#define S2MPB04_REG_GPIO_Y 0x21
+#define S2MPB04_REG_OFF_SEQ_CTRL 0x22
+#define S2MPB04_REG_SEQ1 0x23
+#define S2MPB04_REG_SEQ2 0x24
+#define S2MPB04_REG_ADC_CTRL 0x80
+#define S2MPB04_REG_ADC_CTRL2 0x81
+#define S2MPB04_REG_ADC_CTRL3 0x82
+#define S2MPB04_REG_ADC0DATA 0x83
+#define S2MPB04_REG_ADC1DATA 0x84
+#define S2MPB04_REG_ADC2DATA 0x85
+#define S2MPB04_REG_ADC3DATA 0x86
+#define S2MPB04_REG_MUXSEL1 0x87
+#define S2MPB04_REG_MUXSEL2 0x88
+#define S2MPB04_REG_ADC_THRES_CTRL1 0x89
+#define S2MPB04_REG_ADC_THRES_CTRL2 0x8A
+#define S2MPB04_REG_ADC_THRES_CTRL3 0x8B
+#define S2MPB04_REG_ADC_THRES_CTRL4 0x8C
+#define S2MPB04_REG_ADC_THRES_CTRL5 0x8D
+#define S2MPB04_REG_ADC_THRES_CTRL6 0x8E
+#define S2MPB04_REG_ADC_THRES_CTRL7 0x8F
+#define S2MPB04_REG_ADC_THRES_CTRL8 0x90
+#define S2MPB04_REG_ADC_DBNC_CTRL1 0x91
+#define S2MPB04_REG_ADC_DBNC_CTRL2 0x92
+
+/* adc inputs */
+#define S2MPB04_ADC_I_SMPS1_PH1 0x2
+#define S2MPB04_ADC_I_SMPS1_PH2 0x3
+#define S2MPB04_ADC_I_SMPS1_SUM 0x4
+#define S2MPB04_ADC_I_SMPS2 0x5
+#define S2MPB04_ADC_V_SMPS1 0x6
+#define S2MPB04_ADC_V_SMPS2 0x7
+#define S2MPB04_ADC_I_LDO1 0x8
+#define S2MPB04_ADC_I_LDO2 0x9
+#define S2MPB04_ADC_V_LDO1 0xA
+#define S2MPB04_ADC_V_LDO2 0xB
+#define S2MPB04_ADC_PTAT 0xC
+#define S2MPB04_ADC_VBAT 0xD
+
+/* full scale of 8-bit adc slots */
+#define S2MPB04_ADC_SCALE_V_SMPS1 4688 /* in uV/lsb */
+#define S2MPB04_ADC_SCALE_V_SMPS2 6250 /* in uV/lsb */
+#define S2MPB04_ADC_SCALE_I_SMPS1 31250 /* in uA/lsb */
+#define S2MPB04_ADC_SCALE_I_SMPS2 15625 /* in uA/lsb */
+#define S2MPB04_ADC_SCALE_V_LDO 9375 /* in uA/lsb */
+#define S2MPB04_ADC_SCALE_I_LDO 586 /* in uA/lsb */
+#define S2MPB04_ADC_SCALE_VBAT 23438 /* in uV/lsb */
+
+/* macros for PTAT-to-temperature conversion */
+#define TEMP_TO_PTAT_CODE(X) ((238260 - (X)) / 1167)
+#define PTAT_CODE_TO_TEMP(X) (238260 - ((X) * 1167))
+
+/* interrupt flags */
+#define S2MPB04_INT_PONR 0
+#define S2MPB04_INT_PONF 1
+#define S2MPB04_INT_ADC_CH0 2
+#define S2MPB04_INT_ADC_CH1 3
+#define S2MPB04_INT_ADC_CH2 4
+#define S2MPB04_INT_ADC_CH3 5
+#define S2MPB04_INT_ADC_DONE 6
+#define S2MPB04_INT_WATCHDOG 7
+#define S2MPB04_INT_SMPS1_OI 8
+#define S2MPB04_INT_SMPS2_OI 9
+#define S2MPB04_INT_SMPS1_UV 10
+#define S2MPB04_INT_LDO1_OI 11
+#define S2MPB04_INT_LDO2_OI 12
+#define S2MPB04_INT_TSD 13
+#define S2MPB04_INT_TH_TRIPR 14
+#define S2MPB04_INT_TH_TRIPF 15
+#define S2MPB04_INT_TH_TINT 23
+
+/* regulator names */
+#define S2MPB04_REGLTR_NAME_SMPS1 "s2mpb04_smps1"
+#define S2MPB04_REGLTR_NAME_SMPS2 "s2mpb04_smps2"
+#define S2MPB04_REGLTR_NAME_LDO1 "s2mpb04_ldo1"
+#define S2MPB04_REGLTR_NAME_LDO2 "s2mpb04_ldo2"
+
+/* silicon versions */
+#define S2MPB04_REV_ES 0
+
+#define S2MPB04_NUM_GPIOS 2
+
+/* regulator id enum */
+enum s2mpb04_regulator_ids {
+ S2MPB04_ID_SMPS1,
+ S2MPB04_ID_SMPS2,
+ S2MPB04_ID_LDO1,
+ S2MPB04_ID_LDO2,
+ S2MPB04_ID_ALL,
+ S2MPB04_NUM_REGULATORS = S2MPB04_ID_ALL,
+};
+
+/* driver data structure */
+struct s2mpb04_core {
+ struct device *dev;
+ struct regmap *regmap;
+ struct s2mpb04_platform_data *pdata;
+ u8 rev_id;
+
+ /* kernel thread to notify regulator fail event and
+ * clear any pending interrupt status
+ */
+ struct work_struct reset_work;
+
+ /* kernel thread to detect if PMIC is hung and run the
+ * recovery sequence
+ */
+ struct work_struct recover_work;
+
+ /* completion used for initialization */
+ struct completion init_complete;
+
+ /* flags used for serialization */
+ unsigned long adc_conv_busy;
+
+ /* completion used for signaling end of adc conversion */
+ struct completion adc_conv_complete;
+
+ /* counter for recovery attempts */
+ unsigned long recover_count;
+};
+
+/* platform data structure */
+struct s2mpb04_platform_data {
+ int pon_gpio;
+ int resetb_gpio;
+ int intb_gpio;
+ unsigned int resetb_irq;
+ unsigned int intb_irq;
+};
+
+int s2mpb04_read_byte(struct s2mpb04_core *ddata, u8 addr, u8 *data);
+int s2mpb04_write_byte(struct s2mpb04_core *ddata, u8 addr, u8 data);
+int s2mpb04_update_bits(struct s2mpb04_core *ddata, u8 addr, unsigned int mask,
+ u8 data);
+int s2mpb04_toggle_pon(struct s2mpb04_core *ddata);
+int s2mpb04_toggle_pon_no_resetb_wait(struct s2mpb04_core *ddata);
+int s2mpb04_dump_regs(struct s2mpb04_core *ddata);
+int s2mpb04_read_adc_chan(struct s2mpb04_core *ddata, int chan_num,
+ u8 *chan_data);
+
+/* s2mpb04-sysfs.c */
+void s2mpb04_config_sysfs(struct device *dev);
+
+/* s2mpb04-regulator.c */
+void s2mpb04_regulator_notify(enum s2mpb04_regulator_ids rid,
+ unsigned long event);
+
+#endif /* S2MPB04_REGULATOR_H */
diff --git a/drivers/staging/s2mpb04/s2mpb04-gpio.c b/drivers/staging/s2mpb04/s2mpb04-gpio.c
new file mode 100644
index 0000000..06c6d66
--- /dev/null
+++ b/drivers/staging/s2mpb04/s2mpb04-gpio.c
@@ -0,0 +1,183 @@
+/*
+ * Copyright (C) 2017 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/bitops.h>
+#include <linux/delay.h>
+#include <linux/gpio/driver.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+
+#include "s2mpb04-core.h"
+
+#define DRIVER_NAME "s2mpb04-gpio"
+
+struct s2mpb04_gpio {
+ struct gpio_chip gpio_chip;
+ struct s2mpb04_core *s2mpb04_core;
+ struct device *dev;
+ unsigned long int pin_used;
+};
+
+static inline struct s2mpb04_gpio *to_s2mpb04_gpio
+ (struct gpio_chip *_gpio_chip)
+{
+ return container_of(_gpio_chip, struct s2mpb04_gpio, gpio_chip);
+}
+
+static int s2mpb04_gpio_request(struct gpio_chip *gpio_chip,
+ unsigned int offset)
+{
+ struct s2mpb04_gpio *s2mpb04_gpio = to_s2mpb04_gpio(gpio_chip);
+
+ if (test_and_set_bit(offset, &s2mpb04_gpio->pin_used))
+ return -EBUSY;
+ else
+ return 0;
+}
+
+static void s2mpb04_gpio_free(struct gpio_chip *gpio_chip,
+ unsigned int offset)
+{
+ struct s2mpb04_gpio *s2mpb04_gpio = to_s2mpb04_gpio(gpio_chip);
+
+ clear_bit(offset, &s2mpb04_gpio->pin_used);
+}
+
+static int s2mpb04_gpio_direction_input(struct gpio_chip *gpio_chip,
+ unsigned int offset)
+{
+ /* pins are hard-coded as outputs */
+ return -EIO;
+}
+
+static int s2mpb04_gpio_get(struct gpio_chip *gpio_chip, unsigned int offset)
+{
+ struct s2mpb04_gpio *s2mpb04_gpio = to_s2mpb04_gpio(gpio_chip);
+ struct s2mpb04_core *s2mpb04_core = s2mpb04_gpio->s2mpb04_core;
+ u8 byte;
+
+ s2mpb04_read_byte(s2mpb04_core, S2MPB04_REG_GPIO_Y, &byte);
+ return byte & (1 << offset);
+}
+
+static void s2mpb04_gpio_set(struct gpio_chip *gpio_chip, unsigned int offset,
+ int value)
+{
+ struct s2mpb04_gpio *s2mpb04_gpio = to_s2mpb04_gpio(gpio_chip);
+ struct s2mpb04_core *s2mpb04_core = s2mpb04_gpio->s2mpb04_core;
+ const void *ptr;
+ bool p1_workaround = false;
+
+ /*
+ * NOTE: B1 P1 has a schematic bug where output is not connected to the
+ * correct rail. If the workaround is enabled, then we drive GPIO1 as
+ * push-pull. Otherwise, and by default, it is an open-drain output.
+ */
+ ptr = of_get_property(s2mpb04_gpio->dev->of_node, "p1_workaround",
+ NULL);
+ if (ptr)
+ p1_workaround = true;
+
+ dev_dbg(s2mpb04_core->dev, "%s: offset %d, value %d, p1_workaround %d\n",
+ __func__, offset, value, p1_workaround);
+
+ /*
+ * GPIO1 is an open-drain output and has to have the output disabled to
+ * allow it to float high when value is not 0.
+ */
+ if (!p1_workaround && (offset == 1) && (value != 0)) {
+ s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_GPIO_CTRL, 0x40);
+ } else {
+ s2mpb04_update_bits(s2mpb04_core, S2MPB04_REG_GPIO_A,
+ (1 << offset), ((value ? 1 : 0) << offset));
+ s2mpb04_update_bits(s2mpb04_core, S2MPB04_REG_GPIO_CTRL,
+ (0x40 << offset), (0x40 << offset));
+ }
+}
+
+static int s2mpb04_gpio_direction_output(struct gpio_chip *gpio_chip,
+ unsigned int offset, int value)
+{
+ /* pins are already configured as outputs */
+ s2mpb04_gpio_set(gpio_chip, offset, value);
+ return 0;
+}
+
+static const struct gpio_chip s2mpb04_gpio_chip = {
+ .label = DRIVER_NAME,
+ .owner = THIS_MODULE,
+ .request = s2mpb04_gpio_request,
+ .free = s2mpb04_gpio_free,
+ .direction_input = s2mpb04_gpio_direction_input,
+ .get = s2mpb04_gpio_get,
+ .direction_output = s2mpb04_gpio_direction_output,
+ .set = s2mpb04_gpio_set,
+ .base = -1,
+ .ngpio = S2MPB04_NUM_GPIOS,
+ .can_sleep = 1,
+};
+
+static int s2mpb04_gpio_probe(struct platform_device *pdev)
+{
+ struct s2mpb04_core *s2mpb04_core = dev_get_drvdata(pdev->dev.parent);
+ struct s2mpb04_gpio *s2mpb04_gpio;
+
+ s2mpb04_gpio = devm_kzalloc(&pdev->dev, sizeof(*s2mpb04_gpio),
+ GFP_KERNEL);
+ if (!s2mpb04_gpio)
+ return -ENOMEM;
+
+ s2mpb04_gpio->s2mpb04_core = s2mpb04_core;
+ s2mpb04_gpio->gpio_chip = s2mpb04_gpio_chip;
+ s2mpb04_gpio->gpio_chip.parent = &pdev->dev;
+ s2mpb04_gpio->dev = &pdev->dev;
+
+ platform_set_drvdata(pdev, s2mpb04_gpio);
+
+ /* enable both GPIOs for output with no-pull */
+ s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_GPIO_A, 0x00);
+ s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_GPIO_CTRL, 0xC0);
+
+ return gpiochip_add(&s2mpb04_gpio->gpio_chip);
+}
+
+static int s2mpb04_gpio_remove(struct platform_device *pdev)
+{
+ struct s2mpb04_gpio *s2mpb04_gpio = platform_get_drvdata(pdev);
+
+ gpiochip_remove(&s2mpb04_gpio->gpio_chip);
+ return 0;
+}
+
+static const struct of_device_id s2mpb04_gpio_of_match[] = {
+ { .compatible = "samsung,s2mpb04-gpio", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, s2mpb04_gpio_of_match);
+
+static struct platform_driver s2mpb04_gpio_driver = {
+ .probe = s2mpb04_gpio_probe,
+ .remove = s2mpb04_gpio_remove,
+ .driver = {
+ .name = DRIVER_NAME,
+ .of_match_table = s2mpb04_gpio_of_match,
+ },
+};
+module_platform_driver(s2mpb04_gpio_driver);
+
+MODULE_AUTHOR("Trevor Bunker <trevorbunker@google.com>");
+MODULE_DESCRIPTION("S2MPB04 GPIO Driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/staging/s2mpb04/s2mpb04-regulator.c b/drivers/staging/s2mpb04/s2mpb04-regulator.c
new file mode 100644
index 0000000..9c34f9a
--- /dev/null
+++ b/drivers/staging/s2mpb04/s2mpb04-regulator.c
@@ -0,0 +1,391 @@
+/*
+ * Copyright (C) 2017 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/bitops.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/regulator/driver.h>
+#include <linux/regulator/machine.h>
+
+#include "s2mpb04-core.h"
+
+#define DRIVER_NAME "s2mpb04-regulator"
+
+struct s2mpb04_regulator {
+ struct device *dev;
+ struct s2mpb04_core *s2mpb04_core;
+ struct regulator_dev **rdevs;
+
+ /* bitmask for tracking enabled regulators */
+ unsigned long reg_enabled_mask;
+};
+
+static struct s2mpb04_regulator *_s2mpb04_regulator;
+
+static int s2mpb04_regulator_get_voltage(struct regulator_dev *rdev);
+static int s2mpb04_regulator_enable(struct regulator_dev *rdev);
+static int s2mpb04_regulator_disable(struct regulator_dev *rdev);
+static int s2mpb04_regulator_is_enabled(struct regulator_dev *rdev);
+
+static struct regulator_ops s2mpb04_regulator_ops = {
+ .list_voltage = regulator_list_voltage_table,
+ .map_voltage = regulator_map_voltage_ascend,
+ .get_voltage = s2mpb04_regulator_get_voltage,
+ .enable = s2mpb04_regulator_enable,
+ .disable = s2mpb04_regulator_disable,
+ .is_enabled = s2mpb04_regulator_is_enabled,
+};
+
+/* No support for DVS so just a single voltage level */
+static const unsigned int s2mpb04_ldo_vtbl[] = { 1800000 };
+static const unsigned int s2mpb04_smps1_vtbl[] = { 900000 };
+static const unsigned int s2mpb04_smps2_vtbl[] = { 1100000 };
+
+static struct regulator_desc
+ s2mpb04_regulator_desc[S2MPB04_NUM_REGULATORS] = {
+ [S2MPB04_ID_SMPS1] = {
+ .name = S2MPB04_REGLTR_NAME_SMPS1,
+ .id = S2MPB04_ID_SMPS1,
+ .ops = &s2mpb04_regulator_ops,
+ .n_voltages = ARRAY_SIZE(s2mpb04_smps1_vtbl),
+ .volt_table = s2mpb04_smps1_vtbl,
+ .enable_time = 200,
+ .type = REGULATOR_VOLTAGE,
+ .owner = THIS_MODULE,
+ },
+ [S2MPB04_ID_SMPS2] = {
+ .name = S2MPB04_REGLTR_NAME_SMPS2,
+ .id = S2MPB04_ID_SMPS2,
+ .ops = &s2mpb04_regulator_ops,
+ .n_voltages = ARRAY_SIZE(s2mpb04_smps2_vtbl),
+ .volt_table = s2mpb04_smps2_vtbl,
+ .enable_time = 200,
+ .type = REGULATOR_VOLTAGE,
+ .owner = THIS_MODULE,
+ },
+ [S2MPB04_ID_LDO1] = {
+ .name = S2MPB04_REGLTR_NAME_LDO1,
+ .id = S2MPB04_ID_LDO1,
+ .ops = &s2mpb04_regulator_ops,
+ .n_voltages = ARRAY_SIZE(s2mpb04_ldo_vtbl),
+ .volt_table = s2mpb04_ldo_vtbl,
+ .enable_time = 200,
+ .type = REGULATOR_VOLTAGE,
+ .owner = THIS_MODULE,
+ },
+ [S2MPB04_ID_LDO2] = {
+ .name = S2MPB04_REGLTR_NAME_LDO2,
+ .id = S2MPB04_ID_LDO2,
+ .ops = &s2mpb04_regulator_ops,
+ .n_voltages = ARRAY_SIZE(s2mpb04_ldo_vtbl),
+ .volt_table = s2mpb04_ldo_vtbl,
+ .enable_time = 200,
+ .type = REGULATOR_VOLTAGE,
+ .owner = THIS_MODULE,
+ },
+};
+
+static struct regulator_init_data
+ s2mpb04_regulator_init_data[S2MPB04_NUM_REGULATORS] = {
+ [S2MPB04_ID_SMPS1] = {
+ .constraints = {
+ .name = "s2mpb04_smps1",
+ .valid_ops_mask = REGULATOR_CHANGE_STATUS,
+ .min_uV = 900000,
+ .max_uV = 900000,
+ },
+ },
+ [S2MPB04_ID_SMPS2] = {
+ .constraints = {
+ .name = "s2mpb04_smps2",
+ .valid_ops_mask = REGULATOR_CHANGE_STATUS,
+ .min_uV = 1100000,
+ .max_uV = 1100000,
+ },
+ },
+ [S2MPB04_ID_LDO1] = {
+ .constraints = {
+ .name = "s2mpb04_ldo1",
+ .valid_ops_mask = REGULATOR_CHANGE_STATUS,
+ .min_uV = 1800000,
+ .max_uV = 1800000,
+ },
+ },
+ [S2MPB04_ID_LDO2] = {
+ .constraints = {
+ .name = "s2mpb04_ldo2",
+ .valid_ops_mask = REGULATOR_CHANGE_STATUS,
+ .min_uV = 1800000,
+ .max_uV = 1800000,
+ },
+ },
+};
+
+/* get the current voltage of the regulator in microvolts */
+static int s2mpb04_regulator_get_voltage(struct regulator_dev *rdev)
+{
+ struct s2mpb04_regulator *s2mpb04_regulator = rdev_get_drvdata(rdev);
+ struct s2mpb04_core *s2mpb04_core = s2mpb04_regulator->s2mpb04_core;
+ enum s2mpb04_regulator_ids rid = rdev_get_id(rdev);
+ u8 reg_data;
+ int vsel, vstep, vbase;
+ int ret;
+
+ dev_dbg(s2mpb04_regulator->dev, "%s: rid %d\n", __func__, rid);
+
+ switch (rid) {
+ case S2MPB04_ID_SMPS1:
+ ret = s2mpb04_read_byte(s2mpb04_core, S2MPB04_REG_BUCK1_OUT,
+ ®_data);
+ if (ret)
+ return ret;
+ vbase = 300000;
+ vstep = 6250;
+ vsel = reg_data;
+ break;
+ case S2MPB04_ID_SMPS2:
+ ret = s2mpb04_read_byte(s2mpb04_core, S2MPB04_REG_BUCK2_OUT,
+ ®_data);
+ vbase = 300000;
+ vstep = 6250;
+ if (ret)
+ return ret;
+ vsel = reg_data;
+ break;
+ case S2MPB04_ID_LDO1:
+ ret = s2mpb04_read_byte(s2mpb04_core, S2MPB04_REG_LDO1_CTRL,
+ ®_data);
+ if (ret)
+ return ret;
+ vbase = 700000;
+ vstep = 25000;
+ vsel = reg_data & 0x3F;
+ break;
+ case S2MPB04_ID_LDO2:
+ ret = s2mpb04_read_byte(s2mpb04_core, S2MPB04_REG_LDO2_CTRL,
+ ®_data);
+ if (ret)
+ return ret;
+ vbase = 700000;
+ vstep = 25000;
+ vsel = reg_data & 0x3F;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ dev_dbg(s2mpb04_regulator->dev, "%s: rid %d, returning voltage %d\n",
+ __func__, rid, vbase + vsel * vstep);
+
+ return vbase + vsel * vstep;
+}
+
+/* enable the regulator */
+static int s2mpb04_regulator_enable(struct regulator_dev *rdev)
+{
+ struct s2mpb04_regulator *s2mpb04_regulator = rdev_get_drvdata(rdev);
+ struct s2mpb04_core *s2mpb04_core = s2mpb04_regulator->s2mpb04_core;
+ enum s2mpb04_regulator_ids rid = rdev_get_id(rdev);
+ int ret;
+
+ dev_dbg(s2mpb04_regulator->dev, "%s: rid %d\n", __func__, rid);
+
+ switch (rid) {
+ case S2MPB04_ID_SMPS1:
+ ret = s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_BUCK1_CTRL,
+ 0xF8);
+ break;
+ case S2MPB04_ID_SMPS2:
+ ret = s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_BUCK2_CTRL,
+ 0xD8);
+ break;
+ case S2MPB04_ID_LDO1:
+ ret = s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_LDO1_CTRL,
+ 0xEF);
+ break;
+ case S2MPB04_ID_LDO2:
+ ret = s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_LDO2_CTRL,
+ 0xEF);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (!ret)
+ set_bit(rid, &s2mpb04_regulator->reg_enabled_mask);
+
+ return ret;
+}
+
+/* disable the regulator */
+static int s2mpb04_regulator_disable(struct regulator_dev *rdev)
+{
+ struct s2mpb04_regulator *s2mpb04_regulator = rdev_get_drvdata(rdev);
+ struct s2mpb04_core *s2mpb04_core = s2mpb04_regulator->s2mpb04_core;
+ enum s2mpb04_regulator_ids rid = rdev_get_id(rdev);
+ int ret;
+
+ dev_dbg(s2mpb04_regulator->dev, "%s: rid %d\n", __func__, rid);
+
+ switch (rid) {
+ case S2MPB04_ID_SMPS1:
+ ret = s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_BUCK1_CTRL,
+ 0x38);
+ break;
+ case S2MPB04_ID_SMPS2:
+ ret = s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_BUCK2_CTRL,
+ 0x18);
+ break;
+ case S2MPB04_ID_LDO1:
+ ret = s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_LDO1_CTRL,
+ 0x2C);
+ break;
+ case S2MPB04_ID_LDO2:
+ ret = s2mpb04_write_byte(s2mpb04_core, S2MPB04_REG_LDO2_CTRL,
+ 0x2C);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (!ret)
+ clear_bit(rid, &s2mpb04_regulator->reg_enabled_mask);
+
+ return ret;
+}
+
+/* get regulator enable status */
+static int s2mpb04_regulator_is_enabled(struct regulator_dev *rdev)
+{
+ struct s2mpb04_regulator *s2mpb04_regulator = rdev_get_drvdata(rdev);
+ enum s2mpb04_regulator_ids rid = rdev_get_id(rdev);
+
+ dev_dbg(s2mpb04_regulator->dev, "%s: rid %d\n", __func__, rid);
+
+ if ((rid >= 0) && (rid < S2MPB04_NUM_REGULATORS))
+ return !!(s2mpb04_regulator->reg_enabled_mask & (1 << rid));
+ else
+ return -EINVAL;
+}
+
+void s2mpb04_regulator_notify(enum s2mpb04_regulator_ids rid,
+ unsigned long event)
+{
+ if (!_s2mpb04_regulator || !_s2mpb04_regulator->rdevs ||
+ !_s2mpb04_regulator->rdevs[rid])
+ return;
+
+ if (!s2mpb04_regulator_is_enabled(_s2mpb04_regulator->rdevs[rid]))
+ return;
+
+ dev_err(_s2mpb04_regulator->dev, "%s: rid %d, event 0x%lx\n", __func__,
+ rid, event);
+
+ regulator_notifier_call_chain(_s2mpb04_regulator->rdevs[rid], event,
+ NULL);
+}
+EXPORT_SYMBOL_GPL(s2mpb04_regulator_notify);
+
+/* register a regulator with the kernel regulator framework */
+static int
+s2mpb04_regulator_register(struct s2mpb04_regulator *s2mpb04_regulator)
+{
+ struct device *dev = s2mpb04_regulator->dev;
+ struct s2mpb04_core *s2mpb04_core = s2mpb04_regulator->s2mpb04_core;
+ struct regulator_config cfg = {
+ .dev = dev,
+ .driver_data = s2mpb04_regulator,
+ .regmap = s2mpb04_core->regmap
+ };
+ struct regulator_dev *rdev;
+ int i;
+
+ for (i = 0; i < S2MPB04_NUM_REGULATORS; i++) {
+ cfg.init_data = &s2mpb04_regulator_init_data[i];
+ rdev = devm_regulator_register(dev,
+ &s2mpb04_regulator_desc[i],
+ &cfg);
+ if (IS_ERR(rdev)) {
+ dev_err(dev,
+ "%s: failed to register regulator %d\n",
+ __func__, i);
+ return PTR_ERR(rdev);
+ }
+
+ *(s2mpb04_regulator->rdevs + i) = rdev;
+ }
+
+ return 0;
+}
+
+static int s2mpb04_regulator_probe(struct platform_device *pdev)
+{
+ struct s2mpb04_core *s2mpb04_core = dev_get_drvdata(pdev->dev.parent);
+ struct s2mpb04_regulator *s2mpb04_regulator;
+ struct device *dev = &pdev->dev;
+
+ s2mpb04_regulator = devm_kzalloc(dev, sizeof(*s2mpb04_regulator),
+ GFP_KERNEL);
+ if (!s2mpb04_regulator)
+ return -ENOMEM;
+
+ s2mpb04_regulator->dev = dev;
+ s2mpb04_regulator->s2mpb04_core = s2mpb04_core;
+ _s2mpb04_regulator = s2mpb04_regulator;
+
+ platform_set_drvdata(pdev, s2mpb04_regulator);
+
+ /* initialize and register device regulators */
+ s2mpb04_regulator->rdevs =
+ devm_kzalloc(dev,
+ S2MPB04_NUM_REGULATORS *
+ sizeof(struct regulator_dev *),
+ GFP_KERNEL);
+ if (!s2mpb04_regulator->rdevs) {
+ dev_err(dev,
+ "%s: could not initialize rdevs array\n",
+ __func__);
+ return -ENOMEM;
+ }
+
+ return s2mpb04_regulator_register(s2mpb04_regulator);
+}
+
+static int s2mpb04_regulator_remove(struct platform_device *pdev)
+{
+ return 0;
+}
+
+static const struct of_device_id s2mpb04_regulator_of_match[] = {
+ { .compatible = "samsung,s2mpb04-regulator", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, s2mpb04_regulator_of_match);
+
+static struct platform_driver s2mpb04_regulator_driver = {
+ .probe = s2mpb04_regulator_probe,
+ .remove = s2mpb04_regulator_remove,
+ .driver = {
+ .name = DRIVER_NAME,
+ .of_match_table = s2mpb04_regulator_of_match,
+ },
+};
+module_platform_driver(s2mpb04_regulator_driver);
+
+MODULE_AUTHOR("Trevor Bunker <trevorbunker@google.com>");
+MODULE_DESCRIPTION("S2MPB04 Regulator Driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/staging/s2mpb04/s2mpb04-sysfs.c b/drivers/staging/s2mpb04/s2mpb04-sysfs.c
new file mode 100644
index 0000000..8105e61
--- /dev/null
+++ b/drivers/staging/s2mpb04/s2mpb04-sysfs.c
@@ -0,0 +1,401 @@
+/*
+ * Copyright (C) 2017 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/device.h>
+#include <linux/module.h>
+
+#include "s2mpb04-core.h"
+
+/* used for register access attributes */
+static u8 _reg_addr;
+
+/* smps1_volt is in uV */
+static int s2mpb04_get_smps1_volt(struct s2mpb04_core *ddata, int *smps1_volt)
+{
+ u8 chan_data;
+
+ s2mpb04_read_adc_chan(ddata, S2MPB04_ADC_V_SMPS1, &chan_data);
+
+ *smps1_volt = chan_data * S2MPB04_ADC_SCALE_V_SMPS1;
+
+ return 0;
+}
+
+/* smps1_curr is in uA */
+static int s2mpb04_get_smps1_curr(struct s2mpb04_core *ddata, int *smps1_curr)
+{
+ u8 chan_data;
+
+ s2mpb04_read_adc_chan(ddata, S2MPB04_ADC_I_SMPS1_SUM, &chan_data);
+
+ *smps1_curr = chan_data * S2MPB04_ADC_SCALE_I_SMPS1;
+
+ return 0;
+}
+
+/* smps2_volt is in uV */
+static int s2mpb04_get_smps2_volt(struct s2mpb04_core *ddata, int *smps2_volt)
+{
+ u8 chan_data;
+
+ s2mpb04_read_adc_chan(ddata, S2MPB04_ADC_V_SMPS2, &chan_data);
+
+ *smps2_volt = chan_data * S2MPB04_ADC_SCALE_V_SMPS2;
+
+ return 0;
+}
+
+/* smps2_curr is in uA */
+static int s2mpb04_get_smps2_curr(struct s2mpb04_core *ddata, int *smps2_curr)
+{
+ u8 chan_data;
+
+ s2mpb04_read_adc_chan(ddata, S2MPB04_ADC_I_SMPS2, &chan_data);
+
+ *smps2_curr = chan_data * S2MPB04_ADC_SCALE_I_SMPS2;
+
+ return 0;
+}
+
+/* ldo1_volt is in uV */
+static int s2mpb04_get_ldo1_volt(struct s2mpb04_core *ddata, int *ldo1_volt)
+{
+ u8 chan_data;
+
+ s2mpb04_read_adc_chan(ddata, S2MPB04_ADC_V_LDO1, &chan_data);
+
+ *ldo1_volt = chan_data * S2MPB04_ADC_SCALE_V_LDO;
+
+ return 0;
+}
+
+/* ldo1_curr is in uA */
+static int s2mpb04_get_ldo1_curr(struct s2mpb04_core *ddata, int *ldo1_curr)
+{
+ u8 chan_data;
+
+ s2mpb04_read_adc_chan(ddata, S2MPB04_ADC_I_LDO1, &chan_data);
+
+ *ldo1_curr = chan_data * S2MPB04_ADC_SCALE_I_LDO;
+
+ return 0;
+}
+
+/* ldo2_volt is in uV */
+static int s2mpb04_get_ldo2_volt(struct s2mpb04_core *ddata, int *ldo2_volt)
+{
+ u8 chan_data;
+
+ s2mpb04_read_adc_chan(ddata, S2MPB04_ADC_V_LDO2, &chan_data);
+
+ *ldo2_volt = chan_data * S2MPB04_ADC_SCALE_V_LDO;
+
+ return 0;
+}
+
+/* ldo2_curr is in uA */
+static int s2mpb04_get_ldo2_curr(struct s2mpb04_core *ddata, int *ldo2_curr)
+{
+ u8 chan_data;
+
+ s2mpb04_read_adc_chan(ddata, S2MPB04_ADC_I_LDO2, &chan_data);
+
+ *ldo2_curr = chan_data * S2MPB04_ADC_SCALE_I_LDO;
+
+ return 0;
+}
+
+static ssize_t smps1_volt_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int smps1_volt;
+
+ s2mpb04_get_smps1_volt(ddata, &smps1_volt);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", smps1_volt);
+}
+DEVICE_ATTR_RO(smps1_volt);
+
+static ssize_t smps1_curr_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int smps1_curr;
+
+ s2mpb04_get_smps1_curr(ddata, &smps1_curr);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", smps1_curr);
+}
+DEVICE_ATTR_RO(smps1_curr);
+
+static ssize_t smps2_volt_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int smps2_volt;
+
+ s2mpb04_get_smps2_volt(ddata, &smps2_volt);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", smps2_volt);
+}
+DEVICE_ATTR_RO(smps2_volt);
+
+static ssize_t smps2_curr_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int smps2_curr;
+
+ s2mpb04_get_smps2_curr(ddata, &smps2_curr);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", smps2_curr);
+}
+DEVICE_ATTR_RO(smps2_curr);
+
+static ssize_t ldo1_volt_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int ldo1_volt;
+
+ s2mpb04_get_ldo1_volt(ddata, &ldo1_volt);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", ldo1_volt);
+}
+DEVICE_ATTR_RO(ldo1_volt);
+
+static ssize_t ldo1_curr_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int ldo1_curr;
+
+ s2mpb04_get_ldo1_curr(ddata, &ldo1_curr);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", ldo1_curr);
+}
+DEVICE_ATTR_RO(ldo1_curr);
+
+static ssize_t ldo2_volt_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int ldo2_volt;
+
+ s2mpb04_get_ldo2_volt(ddata, &ldo2_volt);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", ldo2_volt);
+}
+DEVICE_ATTR_RO(ldo2_volt);
+
+static ssize_t ldo2_curr_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int ldo2_curr;
+
+ s2mpb04_get_ldo2_curr(ddata, &ldo2_curr);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", ldo2_curr);
+}
+DEVICE_ATTR_RO(ldo2_curr);
+
+static ssize_t vbat_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ u8 chan_data;
+
+ s2mpb04_read_adc_chan(ddata, S2MPB04_ADC_VBAT, &chan_data);
+
+ return snprintf(data, PAGE_SIZE, "%d\n",
+ chan_data * S2MPB04_ADC_SCALE_VBAT);
+}
+DEVICE_ATTR_RO(vbat);
+
+static ssize_t temperature_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ u8 chan_data;
+
+ s2mpb04_read_adc_chan(ddata, S2MPB04_ADC_PTAT, &chan_data);
+
+ return snprintf(data, PAGE_SIZE, "%d\n", PTAT_CODE_TO_TEMP(chan_data));
+}
+DEVICE_ATTR_RO(temperature);
+
+/* shows power in mW */
+static ssize_t total_power_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int smps1_curr, smps2_curr, ldo2_curr, ldo1_curr;
+ long int total_power;
+
+ s2mpb04_get_smps1_curr(ddata, &smps1_curr);
+ s2mpb04_get_smps2_curr(ddata, &smps2_curr);
+ s2mpb04_get_ldo2_curr(ddata, &ldo2_curr);
+ s2mpb04_get_ldo1_curr(ddata, &ldo1_curr);
+
+ total_power = (((long int)smps1_curr) * 900 / 1000) +
+ (((long int)smps2_curr) * 1100 / 1000) +
+ (((long int)ldo2_curr) * 1800 / 1000) +
+ (((long int)ldo1_curr) * 1800 / 1000);
+
+ return snprintf(data, PAGE_SIZE, "%ld\n", total_power);
+}
+DEVICE_ATTR_RO(total_power);
+
+static ssize_t dump_regs_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+
+ s2mpb04_dump_regs(ddata);
+
+ return snprintf(data, PAGE_SIZE, "ok\n");
+}
+DEVICE_ATTR_RO(dump_regs);
+
+static ssize_t toggle_pon_show(struct device *dev,
+ struct device_attribute *mattr,
+ char *data)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+
+ s2mpb04_toggle_pon(ddata);
+
+ return snprintf(data, PAGE_SIZE, "ok\n");
+}
+DEVICE_ATTR_RO(toggle_pon);
+
+static ssize_t reg_addr_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ return snprintf(buf, PAGE_SIZE, "0x%02x\n", _reg_addr);
+}
+
+static ssize_t reg_addr_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int val = 0;
+ int ret;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (ret < 0)
+ return ret;
+
+ if ((val < 0) || (val > 0xff))
+ return -EINVAL;
+
+ dev_dbg(ddata->dev, "%s: _reg_addr %d\n", __func__, val);
+
+ _reg_addr = val;
+
+ return count;
+}
+DEVICE_ATTR_RW(reg_addr);
+
+static ssize_t reg_data_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ u8 reg_data;
+
+ s2mpb04_read_byte(ddata, _reg_addr, ®_data);
+
+ return snprintf(buf, PAGE_SIZE, "0x%02x\n", reg_data);
+}
+
+static ssize_t reg_data_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf,
+ size_t count)
+{
+ struct s2mpb04_core *ddata = dev_get_drvdata(dev);
+ int val = 0;
+ int ret;
+
+ ret = kstrtoint(buf, 0, &val);
+ if (ret < 0)
+ return ret;
+
+ if ((val < 0) || (val > 0xff))
+ return -EINVAL;
+
+ dev_dbg(ddata->dev, "%s: reg 0x%02x, data 0x%02x\n", __func__,
+ _reg_addr, val);
+
+ s2mpb04_write_byte(ddata, _reg_addr, val);
+
+ return count;
+}
+DEVICE_ATTR_RW(reg_data);
+
+static struct attribute *s2mpb04_attrs[] = {
+ &dev_attr_smps1_volt.attr,
+ &dev_attr_smps1_curr.attr,
+ &dev_attr_smps2_volt.attr,
+ &dev_attr_smps2_curr.attr,
+ &dev_attr_ldo1_volt.attr,
+ &dev_attr_ldo1_curr.attr,
+ &dev_attr_ldo2_volt.attr,
+ &dev_attr_ldo2_curr.attr,
+ &dev_attr_vbat.attr,
+ &dev_attr_temperature.attr,
+ &dev_attr_total_power.attr,
+ &dev_attr_dump_regs.attr,
+ &dev_attr_toggle_pon.attr,
+ &dev_attr_reg_addr.attr,
+ &dev_attr_reg_data.attr,
+ NULL
+};
+
+static const struct attribute_group s2mpb04_attr_group = {
+ .attrs = s2mpb04_attrs,
+};
+
+void s2mpb04_config_sysfs(struct device *dev)
+{
+ int ret;
+
+ ret = sysfs_create_group(&dev->kobj, &s2mpb04_attr_group);
+ if (ret < 0) {
+ dev_err(dev, "%s: could not create sysfs attributes (%d)\n",
+ __func__, ret);
+ }
+}
+EXPORT_SYMBOL_GPL(s2mpb04_config_sysfs);
+
diff --git a/drivers/staging/s2mpb04/s2mpb04-thermal.c b/drivers/staging/s2mpb04/s2mpb04-thermal.c
new file mode 100644
index 0000000..c864696
--- /dev/null
+++ b/drivers/staging/s2mpb04/s2mpb04-thermal.c
@@ -0,0 +1,120 @@
+/*
+ * Copyright (C) 2017 Google, Inc.
+ *
+ * Author: Trevor Bunker <trevorbunker@google.com>
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/thermal.h>
+
+#include "s2mpb04-core.h"
+
+#define DRIVER_NAME "s2mpb04-thermal"
+
+#define S2MPB04_THERM_NUM_TRIPS 0
+#define S2MPB04_THERM_TRIPS_MASK ((1 << S2MPB04_THERM_NUM_TRIPS) - 1)
+
+struct s2mpb04_thermal {
+ struct s2mpb04_core *s2mpb04;
+ struct thermal_zone_device *tz_dev;
+};
+
+static int s2mpb04_thermal_get_temp(struct thermal_zone_device *tz_dev,
+ int *temp)
+{
+ struct s2mpb04_thermal *s2mpb04_thermal = tz_dev->devdata;
+ struct s2mpb04_core *s2mpb04 = s2mpb04_thermal->s2mpb04;
+ u8 chan_data = 0;
+ int ret;
+
+ ret = s2mpb04_read_adc_chan(s2mpb04, S2MPB04_ADC_PTAT, &chan_data);
+ if (ret)
+ return ret;
+
+ *temp = PTAT_CODE_TO_TEMP(chan_data);
+ return 0;
+}
+
+static int s2mpb04_thermal_get_mode(struct thermal_zone_device *tz_dev,
+ enum thermal_device_mode *mode)
+{
+ *mode = THERMAL_DEVICE_ENABLED;
+ return 0;
+}
+
+static struct thermal_zone_device_ops s2mpb04_thermal_ops = {
+ .get_temp = s2mpb04_thermal_get_temp,
+ .get_mode = s2mpb04_thermal_get_mode,
+};
+
+static int s2mpb04_thermal_probe(struct platform_device *pdev)
+{
+ struct s2mpb04_core *s2mpb04 = dev_get_drvdata(pdev->dev.parent);
+ struct s2mpb04_thermal *s2mpb04_thermal;
+ char tz_name[] = "s2mpb04_tz";
+
+ s2mpb04_thermal = devm_kzalloc(&pdev->dev,
+ sizeof(struct s2mpb04_thermal),
+ GFP_KERNEL);
+ if (!s2mpb04_thermal)
+ return -ENOMEM;
+
+ s2mpb04_thermal->s2mpb04 = s2mpb04;
+
+ /* register the thermal zone device */
+ s2mpb04_thermal->tz_dev =
+ thermal_zone_device_register(tz_name, S2MPB04_THERM_NUM_TRIPS,
+ S2MPB04_THERM_TRIPS_MASK,
+ s2mpb04_thermal,
+ &s2mpb04_thermal_ops,
+ NULL, 0, 0);
+ if (IS_ERR(s2mpb04_thermal->tz_dev)) {
+ dev_err(&pdev->dev, "%s: failed to register thermal zone device (%ld)\n",
+ __func__, PTR_ERR(s2mpb04_thermal->tz_dev));
+ return -ENODEV;
+ }
+
+ platform_set_drvdata(pdev, s2mpb04_thermal);
+
+ return 0;
+}
+
+static int s2mpb04_thermal_remove(struct platform_device *pdev)
+{
+ struct s2mpb04_thermal *s2mpb04_thermal = platform_get_drvdata(pdev);
+
+ if (s2mpb04_thermal->tz_dev)
+ thermal_zone_device_unregister(s2mpb04_thermal->tz_dev);
+ return 0;
+}
+
+static const struct of_device_id s2mpb04_thermal_of_match[] = {
+ { .compatible = "brcm,s2mpb04-thermal", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, s2mpb04_thermal_of_match);
+
+static struct platform_driver s2mpb04_thermal_driver = {
+ .probe = s2mpb04_thermal_probe,
+ .remove = s2mpb04_thermal_remove,
+ .driver = {
+ .name = DRIVER_NAME,
+ .of_match_table = s2mpb04_thermal_of_match,
+ },
+};
+module_platform_driver(s2mpb04_thermal_driver);
+
+MODULE_AUTHOR("Trevor Bunker <trevorbunker@google.com>");
+MODULE_DESCRIPTION("S2MPB04 Thermal Sensor Driver");
+MODULE_LICENSE("GPL");
diff --git a/drivers/staging/speakup/kobjects.c b/drivers/staging/speakup/kobjects.c
index dea018c..1253fe11 100644
--- a/drivers/staging/speakup/kobjects.c
+++ b/drivers/staging/speakup/kobjects.c
@@ -387,7 +387,7 @@ static ssize_t synth_store(struct kobject *kobj, struct kobj_attribute *attr,
len = strlen(buf);
if (len < 2 || len > 9)
return -EINVAL;
- strncpy(new_synth_name, buf, len);
+ memcpy(new_synth_name, buf, len);
if (new_synth_name[len - 1] == '\n')
len--;
new_synth_name[len] = '\0';
@@ -517,7 +517,7 @@ static ssize_t punc_store(struct kobject *kobj, struct kobj_attribute *attr,
return -EINVAL;
}
- strncpy(punc_buf, buf, x);
+ memcpy(punc_buf, buf, x);
while (x && punc_buf[x - 1] == '\n')
x--;
diff --git a/drivers/staging/wilc1000/linux_wlan.c b/drivers/staging/wilc1000/linux_wlan.c
index 07d6e48..2e5e3b3 100644
--- a/drivers/staging/wilc1000/linux_wlan.c
+++ b/drivers/staging/wilc1000/linux_wlan.c
@@ -1260,11 +1260,12 @@ int wilc_netdev_init(struct wilc **wilc, struct device *dev, int io_type,
else
strcpy(ndev->name, "p2p%d");
- vif->idx = wl->vif_num;
vif->wilc = *wilc;
vif->ndev = ndev;
wl->vif[i] = vif;
wl->vif_num = i;
+ vif->idx = wl->vif_num;
+
ndev->netdev_ops = &wilc_netdev_ops;
{
diff --git a/drivers/staging/wilc1000/wilc_sdio.c b/drivers/staging/wilc1000/wilc_sdio.c
index 39b73fb2..63c8701 100644
--- a/drivers/staging/wilc1000/wilc_sdio.c
+++ b/drivers/staging/wilc1000/wilc_sdio.c
@@ -830,6 +830,7 @@ static int sdio_read_int(struct wilc *wilc, u32 *int_status)
if (!g_sdio.irq_gpio) {
int i;
+ cmd.read_write = 0;
cmd.function = 1;
cmd.address = 0x04;
cmd.data = 0;
diff --git a/drivers/staging/wlan_mac/Kconfig b/drivers/staging/wlan_mac/Kconfig
new file mode 100644
index 0000000..d2c4978
--- /dev/null
+++ b/drivers/staging/wlan_mac/Kconfig
@@ -0,0 +1,8 @@
+#
+# google set wlan mac
+#
+config GOOGLE_WLAN_MAC
+ bool "Read MAC from device tree"
+ default n
+ help
+ Say Y here if you want the GOOGLE WLAN MAC features
diff --git a/drivers/staging/wlan_mac/Makefile b/drivers/staging/wlan_mac/Makefile
new file mode 100644
index 0000000..fdee76f
--- /dev/null
+++ b/drivers/staging/wlan_mac/Makefile
@@ -0,0 +1,4 @@
+#
+# google set wlan mac
+#
+obj-$(CONFIG_GOOGLE_WLAN_MAC) += google_wlan_mac.o
diff --git a/drivers/staging/wlan_mac/google_wlan_mac.c b/drivers/staging/wlan_mac/google_wlan_mac.c
new file mode 100644
index 0000000..6b41887
--- /dev/null
+++ b/drivers/staging/wlan_mac/google_wlan_mac.c
@@ -0,0 +1,129 @@
+/*
+ * Copyright (c) 2016, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <asm/setup.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/string.h>
+#include <linux/export.h>
+#include <linux/of.h>
+#include <linux/random.h>
+#include <linux/slab.h>
+#include <linux/uaccess.h>
+#include <net/cnss_utils.h>
+#include <soc/qcom/icnss.h>
+
+#define CDB_PATH "/chosen/cdt/cdb2"
+#define WIFI_MAC_1 "wlan_mac1"
+#define WIFI_MAC_2 "wlan_mac2"
+
+static void set_wifi_mac(void)
+{
+ u8 mac[12] = {0};
+ unsigned int size;
+ unsigned char *mac_addr1 = NULL;
+ unsigned char *mac_addr2 = NULL;
+ struct device_node *node;
+ unsigned int mac1_found = 0;
+ unsigned int mac2_found = 0;
+
+ node = of_find_node_by_path(CDB_PATH);
+ if (!node)
+ pr_err("[WLAN] CDB Node not created under %s", CDB_PATH);
+ else {
+ mac_addr1 = (unsigned char *)
+ of_get_property(node, WIFI_MAC_1, &size);
+
+ mac_addr2 = (unsigned char *)
+ of_get_property(node, WIFI_MAC_2, &size);
+ }
+
+ /* In case Missing Provisioned MAC Addresses, exit with error */
+ if (!mac_addr1 || !mac_addr2) {
+ pr_err("[WLAN] Missing Provisioned MAC addresses\n");
+ return;
+ }
+
+ /* Start decoding MAC Addresses
+ * Note that 2 formats are supported for now
+ * AA:BB:CC:DD:EE:FF (with separating colons) and
+ * AABBCCDDEEFF (without separating colons) */
+ if (sscanf(mac_addr1,
+ "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx",
+ &mac[0], &mac[1], &mac[2], &mac[3], &mac[4],
+ &mac[5]) == 6) {
+ mac1_found = 1;
+ } else if (sscanf(mac_addr1,
+ "%02hhx%02hhx%02hhx%02hhx%02hhx%02hhx",
+ &mac[0], &mac[1], &mac[2], &mac[3], &mac[4],
+ &mac[5]) == 6) {
+ mac1_found = 1;
+ }
+
+ if (sscanf(mac_addr2,
+ "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx",
+ &mac[6], &mac[7], &mac[8], &mac[9], &mac[10],
+ &mac[11]) == 6) {
+ mac2_found = 1;
+ } else if (sscanf(mac_addr2,
+ "%02hhx%02hhx%02hhx%02hhx%02hhx%02hhx",
+ &mac[6], &mac[7], &mac[8], &mac[9], &mac[10],
+ &mac[11]) == 6) {
+ mac2_found = 1;
+ }
+
+ /* Make sure Addresses decoding succeeds */
+ if (!mac1_found || !mac2_found) {
+ pr_err("[WLAN] Invalid format for Provisioned MAC Addresses\n");
+ return;
+ }
+
+ /* Make sure Provisioned MAC Addresses are globally Administered */
+ if ((mac[0] & 2) || (mac[6] & 2)) {
+ pr_err("[WLAN] Invalid Provisioned MAC Addresses\n");
+ return;
+ }
+
+ /* Send provisioned MAC Addresses to the platform driver */
+ if (cnss_utils_set_wlan_mac_address(mac, sizeof(mac)) != 0) {
+ pr_err("[WLAN] set wlan mac addresses failed\n");
+ return;
+ }
+
+ /* Now derive the derived mac addresses
+ * by flipping the locally administred bit */
+ mac[0] = mac[0] | 2;
+ mac[6] = mac[6] | 2;
+
+ if (cnss_utils_set_wlan_derived_mac_address(mac, sizeof(mac)) != 0) {
+ pr_err("[WLAN] set wlan derived mac addresses failed\n");
+ return;
+ }
+
+ return;
+}
+
+static int __init wifi_mac_init(void)
+{
+ set_wifi_mac();
+
+ return 0;
+}
+
+late_initcall(wifi_mac_init);
+
+MODULE_AUTHOR("Google");
+MODULE_DESCRIPTION("Google set wlan mac");
+MODULE_LICENSE("GPL v2");
+
diff --git a/drivers/target/iscsi/cxgbit/cxgbit_cm.c b/drivers/target/iscsi/cxgbit/cxgbit_cm.c
index 2fb1bf1..8652475 100644
--- a/drivers/target/iscsi/cxgbit/cxgbit_cm.c
+++ b/drivers/target/iscsi/cxgbit/cxgbit_cm.c
@@ -631,8 +631,11 @@ static void cxgbit_send_halfclose(struct cxgbit_sock *csk)
static void cxgbit_arp_failure_discard(void *handle, struct sk_buff *skb)
{
+ struct cxgbit_sock *csk = handle;
+
pr_debug("%s cxgbit_device %p\n", __func__, handle);
kfree_skb(skb);
+ cxgbit_put_csk(csk);
}
static void cxgbit_abort_arp_failure(void *handle, struct sk_buff *skb)
@@ -1136,7 +1139,7 @@ cxgbit_pass_accept_rpl(struct cxgbit_sock *csk, struct cpl_pass_accept_req *req)
rpl5->opt0 = cpu_to_be64(opt0);
rpl5->opt2 = cpu_to_be32(opt2);
set_wr_txq(skb, CPL_PRIORITY_SETUP, csk->ctrlq_idx);
- t4_set_arp_err_handler(skb, NULL, cxgbit_arp_failure_discard);
+ t4_set_arp_err_handler(skb, csk, cxgbit_arp_failure_discard);
cxgbit_l2t_send(csk->com.cdev, skb, csk->l2t);
}
diff --git a/drivers/target/iscsi/cxgbit/cxgbit_main.c b/drivers/target/iscsi/cxgbit/cxgbit_main.c
index ad26b93..d55261e 100644
--- a/drivers/target/iscsi/cxgbit/cxgbit_main.c
+++ b/drivers/target/iscsi/cxgbit/cxgbit_main.c
@@ -58,6 +58,7 @@ static void *cxgbit_uld_add(const struct cxgb4_lld_info *lldi)
return ERR_PTR(-ENOMEM);
kref_init(&cdev->kref);
+ spin_lock_init(&cdev->np_lock);
cdev->lldi = *lldi;
diff --git a/drivers/target/iscsi/iscsi_target.c b/drivers/target/iscsi/iscsi_target.c
index 04d2b6e..80205f3 100644
--- a/drivers/target/iscsi/iscsi_target.c
+++ b/drivers/target/iscsi/iscsi_target.c
@@ -1435,7 +1435,8 @@ static void iscsit_do_crypto_hash_buf(
sg_init_table(sg, ARRAY_SIZE(sg));
sg_set_buf(sg, buf, payload_length);
- sg_set_buf(sg + 1, pad_bytes, padding);
+ if (padding)
+ sg_set_buf(sg + 1, pad_bytes, padding);
ahash_request_set_crypt(hash, sg, data_crc, payload_length + padding);
@@ -3949,10 +3950,14 @@ static bool iscsi_target_check_conn_state(struct iscsi_conn *conn)
static void iscsit_get_rx_pdu(struct iscsi_conn *conn)
{
int ret;
- u8 buffer[ISCSI_HDR_LEN], opcode;
+ u8 *buffer, opcode;
u32 checksum = 0, digest = 0;
struct kvec iov;
+ buffer = kcalloc(ISCSI_HDR_LEN, sizeof(*buffer), GFP_KERNEL);
+ if (!buffer)
+ return;
+
while (!kthread_should_stop()) {
/*
* Ensure that both TX and RX per connection kthreads
@@ -3960,7 +3965,6 @@ static void iscsit_get_rx_pdu(struct iscsi_conn *conn)
*/
iscsit_thread_check_cpumask(conn, current, 0);
- memset(buffer, 0, ISCSI_HDR_LEN);
memset(&iov, 0, sizeof(struct kvec));
iov.iov_base = buffer;
@@ -3969,7 +3973,7 @@ static void iscsit_get_rx_pdu(struct iscsi_conn *conn)
ret = rx_data(conn, &iov, 1, ISCSI_HDR_LEN);
if (ret != ISCSI_HDR_LEN) {
iscsit_rx_thread_wait_for_tcp(conn);
- return;
+ break;
}
if (conn->conn_ops->HeaderDigest) {
@@ -3979,7 +3983,7 @@ static void iscsit_get_rx_pdu(struct iscsi_conn *conn)
ret = rx_data(conn, &iov, 1, ISCSI_CRC_LEN);
if (ret != ISCSI_CRC_LEN) {
iscsit_rx_thread_wait_for_tcp(conn);
- return;
+ break;
}
iscsit_do_crypto_hash_buf(conn->conn_rx_hash,
@@ -4003,7 +4007,7 @@ static void iscsit_get_rx_pdu(struct iscsi_conn *conn)
}
if (conn->conn_state == TARG_CONN_STATE_IN_LOGOUT)
- return;
+ break;
opcode = buffer[0] & ISCSI_OPCODE_MASK;
@@ -4014,13 +4018,15 @@ static void iscsit_get_rx_pdu(struct iscsi_conn *conn)
" while in Discovery Session, rejecting.\n", opcode);
iscsit_add_reject(conn, ISCSI_REASON_PROTOCOL_ERROR,
buffer);
- return;
+ break;
}
ret = iscsi_target_rx_opcode(conn, buffer);
if (ret < 0)
- return;
+ break;
}
+
+ kfree(buffer);
}
int iscsi_target_rx_thread(void *arg)
diff --git a/drivers/target/iscsi/iscsi_target_auth.c b/drivers/target/iscsi/iscsi_target_auth.c
index 98f75e5..f0d9730 100644
--- a/drivers/target/iscsi/iscsi_target_auth.c
+++ b/drivers/target/iscsi/iscsi_target_auth.c
@@ -26,27 +26,6 @@
#include "iscsi_target_nego.h"
#include "iscsi_target_auth.h"
-static int chap_string_to_hex(unsigned char *dst, unsigned char *src, int len)
-{
- int j = DIV_ROUND_UP(len, 2), rc;
-
- rc = hex2bin(dst, src, j);
- if (rc < 0)
- pr_debug("CHAP string contains non hex digit symbols\n");
-
- dst[j] = '\0';
- return j;
-}
-
-static void chap_binaryhex_to_asciihex(char *dst, char *src, int src_len)
-{
- int i;
-
- for (i = 0; i < src_len; i++) {
- sprintf(&dst[i*2], "%02x", (int) src[i] & 0xff);
- }
-}
-
static void chap_gen_challenge(
struct iscsi_conn *conn,
int caller,
@@ -59,7 +38,7 @@ static void chap_gen_challenge(
memset(challenge_asciihex, 0, CHAP_CHALLENGE_LENGTH * 2 + 1);
get_random_bytes(chap->challenge, CHAP_CHALLENGE_LENGTH);
- chap_binaryhex_to_asciihex(challenge_asciihex, chap->challenge,
+ bin2hex(challenge_asciihex, chap->challenge,
CHAP_CHALLENGE_LENGTH);
/*
* Set CHAP_C, and copy the generated challenge into c_str.
@@ -240,9 +219,16 @@ static int chap_server_compute_md5(
pr_err("Could not find CHAP_R.\n");
goto out;
}
+ if (strlen(chap_r) != MD5_SIGNATURE_SIZE * 2) {
+ pr_err("Malformed CHAP_R\n");
+ goto out;
+ }
+ if (hex2bin(client_digest, chap_r, MD5_SIGNATURE_SIZE) < 0) {
+ pr_err("Malformed CHAP_R\n");
+ goto out;
+ }
pr_debug("[server] Got CHAP_R=%s\n", chap_r);
- chap_string_to_hex(client_digest, chap_r, strlen(chap_r));
tfm = crypto_alloc_shash("md5", 0, 0);
if (IS_ERR(tfm)) {
@@ -286,7 +272,7 @@ static int chap_server_compute_md5(
goto out;
}
- chap_binaryhex_to_asciihex(response, server_digest, MD5_SIGNATURE_SIZE);
+ bin2hex(response, server_digest, MD5_SIGNATURE_SIZE);
pr_debug("[server] MD5 Server Digest: %s\n", response);
if (memcmp(server_digest, client_digest, MD5_SIGNATURE_SIZE) != 0) {
@@ -341,9 +327,7 @@ static int chap_server_compute_md5(
pr_err("Could not find CHAP_C.\n");
goto out;
}
- pr_debug("[server] Got CHAP_C=%s\n", challenge);
- challenge_len = chap_string_to_hex(challenge_binhex, challenge,
- strlen(challenge));
+ challenge_len = DIV_ROUND_UP(strlen(challenge), 2);
if (!challenge_len) {
pr_err("Unable to convert incoming challenge\n");
goto out;
@@ -352,6 +336,11 @@ static int chap_server_compute_md5(
pr_err("CHAP_C exceeds maximum binary size of 1024 bytes\n");
goto out;
}
+ if (hex2bin(challenge_binhex, challenge, challenge_len) < 0) {
+ pr_err("Malformed CHAP_C\n");
+ goto out;
+ }
+ pr_debug("[server] Got CHAP_C=%s\n", challenge);
/*
* During mutual authentication, the CHAP_C generated by the
* initiator must not match the original CHAP_C generated by
@@ -405,7 +394,7 @@ static int chap_server_compute_md5(
/*
* Convert response from binary hex to ascii hext.
*/
- chap_binaryhex_to_asciihex(response, digest, MD5_SIGNATURE_SIZE);
+ bin2hex(response, digest, MD5_SIGNATURE_SIZE);
*nr_out_len += sprintf(nr_out_ptr + *nr_out_len, "CHAP_R=0x%s",
response);
*nr_out_len += 1;
diff --git a/drivers/target/iscsi/iscsi_target_login.c b/drivers/target/iscsi/iscsi_target_login.c
index 9ccd5da..d2f82aa 100644
--- a/drivers/target/iscsi/iscsi_target_login.c
+++ b/drivers/target/iscsi/iscsi_target_login.c
@@ -333,8 +333,7 @@ static int iscsi_login_zero_tsih_s1(
pr_err("idr_alloc() for sess_idr failed\n");
iscsit_tx_login_rsp(conn, ISCSI_STATUS_CLS_TARGET_ERR,
ISCSI_LOGIN_STATUS_NO_RESOURCES);
- kfree(sess);
- return -ENOMEM;
+ goto free_sess;
}
sess->creation_time = get_jiffies_64();
@@ -350,20 +349,28 @@ static int iscsi_login_zero_tsih_s1(
ISCSI_LOGIN_STATUS_NO_RESOURCES);
pr_err("Unable to allocate memory for"
" struct iscsi_sess_ops.\n");
- kfree(sess);
- return -ENOMEM;
+ goto remove_idr;
}
sess->se_sess = transport_init_session(TARGET_PROT_NORMAL);
if (IS_ERR(sess->se_sess)) {
iscsit_tx_login_rsp(conn, ISCSI_STATUS_CLS_TARGET_ERR,
ISCSI_LOGIN_STATUS_NO_RESOURCES);
- kfree(sess->sess_ops);
- kfree(sess);
- return -ENOMEM;
+ goto free_ops;
}
return 0;
+
+free_ops:
+ kfree(sess->sess_ops);
+remove_idr:
+ spin_lock_bh(&sess_idr_lock);
+ idr_remove(&sess_idr, sess->session_index);
+ spin_unlock_bh(&sess_idr_lock);
+free_sess:
+ kfree(sess);
+ conn->sess = NULL;
+ return -ENOMEM;
}
static int iscsi_login_zero_tsih_s2(
@@ -1152,13 +1159,13 @@ void iscsi_target_login_sess_out(struct iscsi_conn *conn,
ISCSI_LOGIN_STATUS_INIT_ERR);
if (!zero_tsih || !conn->sess)
goto old_sess_out;
- if (conn->sess->se_sess)
- transport_free_session(conn->sess->se_sess);
- if (conn->sess->session_index != 0) {
- spin_lock_bh(&sess_idr_lock);
- idr_remove(&sess_idr, conn->sess->session_index);
- spin_unlock_bh(&sess_idr_lock);
- }
+
+ transport_free_session(conn->sess->se_sess);
+
+ spin_lock_bh(&sess_idr_lock);
+ idr_remove(&sess_idr, conn->sess->session_index);
+ spin_unlock_bh(&sess_idr_lock);
+
kfree(conn->sess->sess_ops);
kfree(conn->sess);
conn->sess = NULL;
diff --git a/drivers/target/iscsi/iscsi_target_tpg.c b/drivers/target/iscsi/iscsi_target_tpg.c
index 63e1dcc..761b065 100644
--- a/drivers/target/iscsi/iscsi_target_tpg.c
+++ b/drivers/target/iscsi/iscsi_target_tpg.c
@@ -637,8 +637,7 @@ int iscsit_ta_authentication(struct iscsi_portal_group *tpg, u32 authentication)
none = strstr(buf1, NONE);
if (none)
goto out;
- strncat(buf1, ",", strlen(","));
- strncat(buf1, NONE, strlen(NONE));
+ strlcat(buf1, "," NONE, sizeof(buf1));
if (iscsi_update_param_value(param, buf1) < 0)
return -EINVAL;
}
diff --git a/drivers/target/target_core_transport.c b/drivers/target/target_core_transport.c
index 6f3eccf..e738b46 100644
--- a/drivers/target/target_core_transport.c
+++ b/drivers/target/target_core_transport.c
@@ -316,6 +316,7 @@ void __transport_register_session(
{
const struct target_core_fabric_ops *tfo = se_tpg->se_tpg_tfo;
unsigned char buf[PR_REG_ISID_LEN];
+ unsigned long flags;
se_sess->se_tpg = se_tpg;
se_sess->fabric_sess_ptr = fabric_sess_ptr;
@@ -352,7 +353,7 @@ void __transport_register_session(
se_sess->sess_bin_isid = get_unaligned_be64(&buf[0]);
}
- spin_lock_irq(&se_nacl->nacl_sess_lock);
+ spin_lock_irqsave(&se_nacl->nacl_sess_lock, flags);
/*
* The se_nacl->nacl_sess pointer will be set to the
* last active I_T Nexus for each struct se_node_acl.
@@ -361,7 +362,7 @@ void __transport_register_session(
list_add_tail(&se_sess->sess_acl_list,
&se_nacl->acl_sess_list);
- spin_unlock_irq(&se_nacl->nacl_sess_lock);
+ spin_unlock_irqrestore(&se_nacl->nacl_sess_lock, flags);
}
list_add_tail(&se_sess->sess_list, &se_tpg->tpg_sess_list);
diff --git a/drivers/tc/tc.c b/drivers/tc/tc.c
index 3be9519..cf3fad2 100644
--- a/drivers/tc/tc.c
+++ b/drivers/tc/tc.c
@@ -2,7 +2,7 @@
* TURBOchannel bus services.
*
* Copyright (c) Harald Koerfgen, 1998
- * Copyright (c) 2001, 2003, 2005, 2006 Maciej W. Rozycki
+ * Copyright (c) 2001, 2003, 2005, 2006, 2018 Maciej W. Rozycki
* Copyright (c) 2005 James Simmons
*
* This file is subject to the terms and conditions of the GNU
@@ -10,6 +10,7 @@
* directory of this archive for more details.
*/
#include <linux/compiler.h>
+#include <linux/dma-mapping.h>
#include <linux/errno.h>
#include <linux/init.h>
#include <linux/ioport.h>
@@ -92,6 +93,11 @@ static void __init tc_bus_add_devices(struct tc_bus *tbus)
tdev->dev.bus = &tc_bus_type;
tdev->slot = slot;
+ /* TURBOchannel has 34-bit DMA addressing (16GiB space). */
+ tdev->dma_mask = DMA_BIT_MASK(34);
+ tdev->dev.dma_mask = &tdev->dma_mask;
+ tdev->dev.coherent_dma_mask = DMA_BIT_MASK(34);
+
for (i = 0; i < 8; i++) {
tdev->firmware[i] =
readb(module + offset + TC_FIRM_VER + 4 * i);
diff --git a/drivers/thermal/Makefile b/drivers/thermal/Makefile
index 1259654..e3fd72f 100644
--- a/drivers/thermal/Makefile
+++ b/drivers/thermal/Makefile
@@ -59,3 +59,9 @@
obj-$(CONFIG_GENERIC_ADC_THERMAL) += thermal-generic-adc.o
obj-$(CONFIG_THERMAL_QPNP_ADC_TM) += qpnp-adc-tm.o
obj-$(CONFIG_THERMAL_TSENS) += msm-tsens.o tsens2xxx.o tsens-dbg.o tsens-mtc.o tsens1xxx.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-enum-conversion \
+ -Wno-pointer-bool-conversion \
+
diff --git a/drivers/thermal/of-thermal.c b/drivers/thermal/of-thermal.c
index c662cd7..fe811d7 100644
--- a/drivers/thermal/of-thermal.c
+++ b/drivers/thermal/of-thermal.c
@@ -396,10 +396,13 @@ static int of_thermal_set_mode(struct thermal_zone_device *tz,
mutex_lock(&tz->lock);
- if (mode == THERMAL_DEVICE_ENABLED)
+ if (mode == THERMAL_DEVICE_ENABLED) {
tz->polling_delay = data->polling_delay;
- else
+ tz->passive_delay = data->passive_delay;
+ } else {
tz->polling_delay = 0;
+ tz->passive_delay = 0;
+ }
mutex_unlock(&tz->lock);
diff --git a/drivers/thermal/qcom/bcl_peripheral.c b/drivers/thermal/qcom/bcl_peripheral.c
index 3dccff5..4efd0b0 100644
--- a/drivers/thermal/qcom/bcl_peripheral.c
+++ b/drivers/thermal/qcom/bcl_peripheral.c
@@ -82,6 +82,7 @@ struct bcl_peripheral_data {
bool irq_enabled;
struct thermal_zone_of_device_ops ops;
struct thermal_zone_device *tz_dev;
+ const char *bat_psy_name;
};
struct bcl_device {
@@ -519,10 +520,12 @@ static int bcl_read_soc(void *data, int *val)
static struct power_supply *batt_psy;
union power_supply_propval ret = {0,};
int err = 0;
+ struct bcl_peripheral_data *perph_data =
+ &bcl_perph->param[BCL_SOC_MONITOR];
*val = 100;
if (!batt_psy)
- batt_psy = power_supply_get_by_name("battery");
+ batt_psy = power_supply_get_by_name(perph_data->bat_psy_name);
if (batt_psy) {
err = power_supply_get_property(batt_psy,
POWER_SUPPLY_PROP_CAPACITY, &ret);
@@ -548,9 +551,10 @@ static void bcl_evaluate_soc(struct work_struct *work)
return;
mutex_lock(&perph_data->state_trans_lock);
- if (!perph_data->irq_enabled)
- goto eval_exit;
- if (battery_percentage > perph_data->trip_temp)
+ if (!perph_data->irq_enabled) {
+ if (battery_percentage <= perph_data->trip_temp)
+ goto eval_exit;
+ } else if (battery_percentage > perph_data->trip_temp)
goto eval_exit;
perph_data->trip_val = battery_percentage;
@@ -566,8 +570,10 @@ static int battery_supply_callback(struct notifier_block *nb,
unsigned long event, void *data)
{
struct power_supply *psy = data;
+ struct bcl_peripheral_data *perph_data =
+ &bcl_perph->param[BCL_SOC_MONITOR];
- if (strcmp(psy->desc->name, "battery"))
+ if (strcmp(psy->desc->name, perph_data->bat_psy_name))
return NOTIFY_OK;
schedule_work(&bcl_perph->soc_eval_work);
@@ -619,6 +625,13 @@ static void bcl_probe_soc(struct platform_device *pdev)
{
int ret = 0;
struct bcl_peripheral_data *soc_data;
+ const char *bat_psy_name = NULL;
+
+ ret = of_property_read_string(pdev->dev.of_node,
+ "google,bat-power-supply",
+ &bat_psy_name);
+ if (ret)
+ bat_psy_name = "battery";
soc_data = &bcl_perph->param[BCL_SOC_MONITOR];
mutex_init(&soc_data->state_trans_lock);
@@ -626,6 +639,10 @@ static void bcl_probe_soc(struct platform_device *pdev)
soc_data->ops.set_trips = bcl_set_soc;
INIT_WORK(&bcl_perph->soc_eval_work, bcl_evaluate_soc);
bcl_perph->psy_nb.notifier_call = battery_supply_callback;
+
+ soc_data->bat_psy_name =
+ devm_kstrdup(&pdev->dev, bat_psy_name, GFP_KERNEL);
+
ret = power_supply_reg_notifier(&bcl_perph->psy_nb);
if (ret < 0) {
pr_err("Unable to register soc notifier. err:%d\n", ret);
diff --git a/drivers/thermal/qcom/msm_lmh_dcvs.c b/drivers/thermal/qcom/msm_lmh_dcvs.c
index 66f67ce..1b00aeb 100644
--- a/drivers/thermal/qcom/msm_lmh_dcvs.c
+++ b/drivers/thermal/qcom/msm_lmh_dcvs.c
@@ -38,6 +38,8 @@
#define CREATE_TRACE_POINTS
#include <trace/events/lmh.h>
+#undef CREATE_TRACE_POINTS
+#include <trace/events/power.h>
#define LIMITS_DCVSH 0x10
#define LIMITS_PROFILE_CHANGE 0x01
@@ -178,6 +180,9 @@ static unsigned long limits_mitigation_notify(struct limits_dcvs_hw *hw)
pr_debug("CPU:%d max limit:%lu\n", cpumask_first(&hw->core_map),
max_limit);
trace_lmh_dcvs_freq(cpumask_first(&hw->core_map), max_limit);
+ trace_clock_set_rate(hw->sensor_name,
+ max_limit,
+ cpumask_first(&hw->core_map));
notify_exit:
hw->hw_freq_limit = max_limit;
@@ -662,6 +667,7 @@ static int limits_dcvs_probe(struct platform_device *pdev)
hw->lmh_freq_attr.attr.name = "lmh_freq_limit";
hw->lmh_freq_attr.show = lmh_freq_limit_show;
hw->lmh_freq_attr.attr.mode = 0444;
+ sysfs_attr_init(&hw->lmh_freq_attr.attr);
device_create_file(&pdev->dev, &hw->lmh_freq_attr);
probe_exit:
diff --git a/drivers/thermal/qpnp-adc-tm.c b/drivers/thermal/qpnp-adc-tm.c
index 5d345cc..f49a63a 100644
--- a/drivers/thermal/qpnp-adc-tm.c
+++ b/drivers/thermal/qpnp-adc-tm.c
@@ -2951,21 +2951,21 @@ int32_t qpnp_adc_tm_disable_chan_meas(struct qpnp_adc_tm_chip *chip,
QPNP_BTM_Mn_HIGH_THR_INT_EN, false);
if (rc < 0) {
pr_err("high thr disable err:%d\n", btm_chan_num);
- return rc;
+ goto fail;
}
rc = qpnp_adc_tm_reg_update(chip, QPNP_BTM_Mn_EN(btm_chan_num),
QPNP_BTM_Mn_LOW_THR_INT_EN, false);
if (rc < 0) {
pr_err("low thr disable err:%d\n", btm_chan_num);
- return rc;
+ goto fail;
}
rc = qpnp_adc_tm_reg_update(chip, QPNP_BTM_Mn_EN(btm_chan_num),
QPNP_BTM_Mn_MEAS_EN, false);
if (rc < 0) {
pr_err("multi measurement disable failed\n");
- return rc;
+ goto fail;
}
}
diff --git a/drivers/thermal/samsung/exynos_tmu.c b/drivers/thermal/samsung/exynos_tmu.c
index ad1186d..c974cb5 100644
--- a/drivers/thermal/samsung/exynos_tmu.c
+++ b/drivers/thermal/samsung/exynos_tmu.c
@@ -185,6 +185,7 @@
* @regulator: pointer to the TMU regulator structure.
* @reg_conf: pointer to structure to register with core thermal.
* @ntrip: number of supported trip points.
+ * @enabled: current status of TMU device
* @tmu_initialize: SoC specific TMU initialization method
* @tmu_control: SoC specific TMU control method
* @tmu_read: SoC specific TMU temperature read method
@@ -205,6 +206,7 @@ struct exynos_tmu_data {
struct regulator *regulator;
struct thermal_zone_device *tzd;
unsigned int ntrip;
+ bool enabled;
int (*tmu_initialize)(struct platform_device *pdev);
void (*tmu_control)(struct platform_device *pdev, bool on);
@@ -398,6 +400,7 @@ static void exynos_tmu_control(struct platform_device *pdev, bool on)
mutex_lock(&data->lock);
clk_enable(data->clk);
data->tmu_control(pdev, on);
+ data->enabled = on;
clk_disable(data->clk);
mutex_unlock(&data->lock);
}
@@ -595,6 +598,7 @@ static int exynos5433_tmu_initialize(struct platform_device *pdev)
threshold_code = temp_to_code(data, temp);
rising_threshold = readl(data->base + rising_reg_offset);
+ rising_threshold &= ~(0xff << j * 8);
rising_threshold |= (threshold_code << j * 8);
writel(rising_threshold, data->base + rising_reg_offset);
@@ -889,19 +893,24 @@ static void exynos7_tmu_control(struct platform_device *pdev, bool on)
static int exynos_get_temp(void *p, int *temp)
{
struct exynos_tmu_data *data = p;
+ int value, ret = 0;
- if (!data || !data->tmu_read)
+ if (!data || !data->tmu_read || !data->enabled)
return -EINVAL;
mutex_lock(&data->lock);
clk_enable(data->clk);
- *temp = code_to_temp(data, data->tmu_read(data)) * MCELSIUS;
+ value = data->tmu_read(data);
+ if (value < 0)
+ ret = value;
+ else
+ *temp = code_to_temp(data, value) * MCELSIUS;
clk_disable(data->clk);
mutex_unlock(&data->lock);
- return 0;
+ return ret;
}
#ifdef CONFIG_THERMAL_EMULATION
diff --git a/drivers/thermal/thermal_core.c b/drivers/thermal/thermal_core.c
index cc3af57..44c2d83 100644
--- a/drivers/thermal/thermal_core.c
+++ b/drivers/thermal/thermal_core.c
@@ -38,6 +38,8 @@
#include <net/netlink.h>
#include <net/genetlink.h>
#include <linux/suspend.h>
+#include <linux/kobject.h>
+#include <../base/base.h>
#define CREATE_TRACE_POINTS
#include <trace/events/thermal.h>
@@ -61,6 +63,8 @@ static LIST_HEAD(thermal_governor_list);
static DEFINE_MUTEX(thermal_list_lock);
static DEFINE_MUTEX(thermal_governor_lock);
+static DEFINE_MUTEX(cdev_softlink_lock);
+static DEFINE_MUTEX(tz_softlink_lock);
static atomic_t in_suspend;
@@ -604,16 +608,20 @@ static void update_temperature(struct thermal_zone_device *tz)
tz->last_temperature, tz->temperature);
}
-static void thermal_zone_device_reset(struct thermal_zone_device *tz)
+static void thermal_zone_device_init(struct thermal_zone_device *tz)
{
struct thermal_instance *pos;
-
tz->temperature = THERMAL_TEMP_INVALID;
- tz->passive = 0;
list_for_each_entry(pos, &tz->thermal_instances, tz_node)
pos->initialized = false;
}
+static void thermal_zone_device_reset(struct thermal_zone_device *tz)
+{
+ tz->passive = 0;
+ thermal_zone_device_init(tz);
+}
+
void thermal_zone_device_update(struct thermal_zone_device *tz,
enum thermal_notify_event event)
{
@@ -631,6 +639,13 @@ void thermal_zone_device_update(struct thermal_zone_device *tz,
thermal_zone_set_trips(tz);
tz->notify_event = event;
+ /*
+ * To prevent cooling_device throttling
+ * when tz->temperature keep initialized status.
+ */
+ if (tz->temperature == THERMAL_TEMP_INVALID ||
+ tz->temperature == THERMAL_TEMP_INVALID_LOW)
+ return;
for (count = 0; count < tz->trips; count++)
handle_thermal_trip(tz, count);
@@ -1775,10 +1790,14 @@ __thermal_cooling_device_register(struct device_node *np,
struct thermal_cooling_device *cdev;
struct thermal_zone_device *pos = NULL;
int result;
+ static struct kobject *cdev_softlink_kobj;
if (type && strlen(type) >= THERMAL_NAME_LENGTH)
return ERR_PTR(-EINVAL);
+ if (!strcmp(type, ""))
+ return ERR_PTR(-EINVAL);
+
if (!ops || !ops->get_max_state || !ops->get_cur_state ||
!ops->set_cur_state)
return ERR_PTR(-EINVAL);
@@ -1812,6 +1831,26 @@ __thermal_cooling_device_register(struct device_node *np,
return ERR_PTR(result);
}
+ mutex_lock(&cdev_softlink_lock);
+ if (cdev_softlink_kobj == NULL) {
+ cdev_softlink_kobj = kobject_create_and_add("cdev-by-name",
+ cdev->device.kobj.parent);
+ result = sysfs_create_link(&cdev->device.class->p->subsys.kobj,
+ cdev_softlink_kobj,
+ "cdev-by-name");
+ if (result) {
+ dev_err(&cdev->device,
+ "Fail to create cdev_map "
+ "soft link in class\n");
+ }
+ }
+ mutex_unlock(&cdev_softlink_lock);
+
+ result = sysfs_create_link(cdev_softlink_kobj,
+ &cdev->device.kobj, cdev->type);
+ if (result)
+ dev_err(&cdev->device, "Fail to create cdev_map soft link\n");
+
/* Add 'this' new cdev to the global cdev list */
mutex_lock(&thermal_list_lock);
list_add(&cdev->node, &thermal_cdev_list);
@@ -2139,10 +2178,14 @@ struct thermal_zone_device *thermal_zone_device_register(const char *type,
int count;
int passive = 0;
struct thermal_governor *governor;
+ static struct kobject *tz_softlink_kobj;
if (type && strlen(type) >= THERMAL_NAME_LENGTH)
return ERR_PTR(-EINVAL);
+ if (!strcmp(type, ""))
+ return ERR_PTR(-EINVAL);
+
if (trips > THERMAL_MAX_TRIPS || trips < 0 || mask >> trips)
return ERR_PTR(-EINVAL);
@@ -2287,6 +2330,26 @@ struct thermal_zone_device *thermal_zone_device_register(const char *type,
if (atomic_cmpxchg(&tz->need_update, 1, 0))
thermal_zone_device_update(tz, THERMAL_EVENT_UNSPECIFIED);
+ /* Create softlink now */
+ mutex_lock(&tz_softlink_lock);
+ if (tz_softlink_kobj == NULL) {
+ tz_softlink_kobj = kobject_create_and_add("tz-by-name",
+ tz->device.kobj.parent);
+ result = sysfs_create_link(&tz->device.class->p->subsys.kobj,
+ tz_softlink_kobj,
+ "tz-by-name");
+ if (result) {
+ dev_err(&tz->device,
+ "Fail to create tz_map soft link in class\n");
+ }
+ }
+ mutex_unlock(&tz_softlink_lock);
+
+ result = sysfs_create_link(tz_softlink_kobj,
+ &tz->device.kobj, tz->type);
+ if (result)
+ dev_err(&tz->device, "Fail to create tz_map soft link\n");
+
return tz;
unregister:
@@ -2591,7 +2654,7 @@ static int thermal_pm_notify(struct notifier_block *nb,
case PM_POST_SUSPEND:
atomic_set(&in_suspend, 0);
list_for_each_entry(tz, &thermal_tz_list, node) {
- thermal_zone_device_reset(tz);
+ thermal_zone_device_init(tz);
thermal_zone_device_update(tz,
THERMAL_EVENT_UNSPECIFIED);
}
diff --git a/drivers/tty/hvc/hvc_opal.c b/drivers/tty/hvc/hvc_opal.c
index 5107993..1fc5d5b 100644
--- a/drivers/tty/hvc/hvc_opal.c
+++ b/drivers/tty/hvc/hvc_opal.c
@@ -332,7 +332,6 @@ static void udbg_init_opal_common(void)
udbg_putc = udbg_opal_putc;
udbg_getc = udbg_opal_getc;
udbg_getc_poll = udbg_opal_getc_poll;
- tb_ticks_per_usec = 0x200; /* Make udelay not suck */
}
void __init hvc_opal_init_early(void)
diff --git a/drivers/tty/n_gsm.c b/drivers/tty/n_gsm.c
index fe22917..9e9016e 100644
--- a/drivers/tty/n_gsm.c
+++ b/drivers/tty/n_gsm.c
@@ -137,6 +137,9 @@ struct gsm_dlci {
struct mutex mutex;
/* Link layer */
+ int mode;
+#define DLCI_MODE_ABM 0 /* Normal Asynchronous Balanced Mode */
+#define DLCI_MODE_ADM 1 /* Asynchronous Disconnected Mode */
spinlock_t lock; /* Protects the internal state */
struct timer_list t1; /* Retransmit timer for SABM and UA */
int retries;
@@ -1380,7 +1383,13 @@ static struct gsm_control *gsm_control_send(struct gsm_mux *gsm,
ctrl->data = data;
ctrl->len = clen;
gsm->pending_cmd = ctrl;
- gsm->cretries = gsm->n2;
+
+ /* If DLCI0 is in ADM mode skip retries, it won't respond */
+ if (gsm->dlci[0]->mode == DLCI_MODE_ADM)
+ gsm->cretries = 1;
+ else
+ gsm->cretries = gsm->n2;
+
mod_timer(&gsm->t2_timer, jiffies + gsm->t2 * HZ / 100);
gsm_control_transmit(gsm, ctrl);
spin_unlock_irqrestore(&gsm->control_lock, flags);
@@ -1488,6 +1497,7 @@ static void gsm_dlci_t1(unsigned long data)
if (debug & 8)
pr_info("DLCI %d opening in ADM mode.\n",
dlci->addr);
+ dlci->mode = DLCI_MODE_ADM;
gsm_dlci_open(dlci);
} else {
gsm_dlci_close(dlci);
@@ -2865,11 +2875,22 @@ static int gsmtty_modem_update(struct gsm_dlci *dlci, u8 brk)
static int gsm_carrier_raised(struct tty_port *port)
{
struct gsm_dlci *dlci = container_of(port, struct gsm_dlci, port);
+ struct gsm_mux *gsm = dlci->gsm;
+
/* Not yet open so no carrier info */
if (dlci->state != DLCI_OPEN)
return 0;
if (debug & 2)
return 1;
+
+ /*
+ * Basic mode with control channel in ADM mode may not respond
+ * to CMD_MSC at all and modem_rx is empty.
+ */
+ if (gsm->encoding == 0 && gsm->dlci[0]->mode == DLCI_MODE_ADM &&
+ !dlci->modem_rx)
+ return 1;
+
return dlci->modem_rx & TIOCM_CD;
}
diff --git a/drivers/tty/n_tty.c b/drivers/tty/n_tty.c
index 1c70541..904fc9c 100644
--- a/drivers/tty/n_tty.c
+++ b/drivers/tty/n_tty.c
@@ -126,6 +126,8 @@ struct n_tty_data {
struct mutex output_lock;
};
+#define MASK(x) ((x) & (N_TTY_BUF_SIZE - 1))
+
static inline size_t read_cnt(struct n_tty_data *ldata)
{
return ldata->read_head - ldata->read_tail;
@@ -143,6 +145,7 @@ static inline unsigned char *read_buf_addr(struct n_tty_data *ldata, size_t i)
static inline unsigned char echo_buf(struct n_tty_data *ldata, size_t i)
{
+ smp_rmb(); /* Matches smp_wmb() in add_echo_byte(). */
return ldata->echo_buf[i & (N_TTY_BUF_SIZE - 1)];
}
@@ -151,17 +154,28 @@ static inline unsigned char *echo_buf_addr(struct n_tty_data *ldata, size_t i)
return &ldata->echo_buf[i & (N_TTY_BUF_SIZE - 1)];
}
+/* If we are not echoing the data, perhaps this is a secret so erase it */
+static void zero_buffer(struct tty_struct *tty, u8 *buffer, int size)
+{
+ bool icanon = !!L_ICANON(tty);
+ bool no_echo = !L_ECHO(tty);
+
+ if (icanon && no_echo)
+ memset(buffer, 0x00, size);
+}
+
static int tty_copy_to_user(struct tty_struct *tty, void __user *to,
size_t tail, size_t n)
{
struct n_tty_data *ldata = tty->disc_data;
size_t size = N_TTY_BUF_SIZE - tail;
- const void *from = read_buf_addr(ldata, tail);
+ void *from = read_buf_addr(ldata, tail);
int uncopied;
if (n > size) {
tty_audit_add_data(tty, from, size);
uncopied = copy_to_user(to, from, size);
+ zero_buffer(tty, from, size - uncopied);
if (uncopied)
return uncopied;
to += size;
@@ -170,7 +184,9 @@ static int tty_copy_to_user(struct tty_struct *tty, void __user *to,
}
tty_audit_add_data(tty, from, n);
- return copy_to_user(to, from, n);
+ uncopied = copy_to_user(to, from, n);
+ zero_buffer(tty, from, n - uncopied);
+ return uncopied;
}
/**
@@ -318,9 +334,7 @@ static inline void put_tty_queue(unsigned char c, struct n_tty_data *ldata)
static void reset_buffer_flags(struct n_tty_data *ldata)
{
ldata->read_head = ldata->canon_head = ldata->read_tail = 0;
- ldata->echo_head = ldata->echo_tail = ldata->echo_commit = 0;
ldata->commit_head = 0;
- ldata->echo_mark = 0;
ldata->line_start = 0;
ldata->erasing = 0;
@@ -619,13 +633,20 @@ static size_t __process_echoes(struct tty_struct *tty)
old_space = space = tty_write_room(tty);
tail = ldata->echo_tail;
- while (ldata->echo_commit != tail) {
+ while (MASK(ldata->echo_commit) != MASK(tail)) {
c = echo_buf(ldata, tail);
if (c == ECHO_OP_START) {
unsigned char op;
int no_space_left = 0;
/*
+ * Since add_echo_byte() is called without holding
+ * output_lock, we might see only portion of multi-byte
+ * operation.
+ */
+ if (MASK(ldata->echo_commit) == MASK(tail + 1))
+ goto not_yet_stored;
+ /*
* If the buffer byte is the start of a multi-byte
* operation, get the next byte, which is either the
* op code or a control character value.
@@ -636,6 +657,8 @@ static size_t __process_echoes(struct tty_struct *tty)
unsigned int num_chars, num_bs;
case ECHO_OP_ERASE_TAB:
+ if (MASK(ldata->echo_commit) == MASK(tail + 2))
+ goto not_yet_stored;
num_chars = echo_buf(ldata, tail + 2);
/*
@@ -730,7 +753,8 @@ static size_t __process_echoes(struct tty_struct *tty)
/* If the echo buffer is nearly full (so that the possibility exists
* of echo overrun before the next commit), then discard enough
* data at the tail to prevent a subsequent overrun */
- while (ldata->echo_commit - tail >= ECHO_DISCARD_WATERMARK) {
+ while (ldata->echo_commit > tail &&
+ ldata->echo_commit - tail >= ECHO_DISCARD_WATERMARK) {
if (echo_buf(ldata, tail) == ECHO_OP_START) {
if (echo_buf(ldata, tail + 1) == ECHO_OP_ERASE_TAB)
tail += 3;
@@ -740,6 +764,7 @@ static size_t __process_echoes(struct tty_struct *tty)
tail++;
}
+ not_yet_stored:
ldata->echo_tail = tail;
return old_space - space;
}
@@ -750,6 +775,7 @@ static void commit_echoes(struct tty_struct *tty)
size_t nr, old, echoed;
size_t head;
+ mutex_lock(&ldata->output_lock);
head = ldata->echo_head;
ldata->echo_mark = head;
old = ldata->echo_commit - ldata->echo_tail;
@@ -758,10 +784,12 @@ static void commit_echoes(struct tty_struct *tty)
* is over the threshold (and try again each time another
* block is accumulated) */
nr = head - ldata->echo_tail;
- if (nr < ECHO_COMMIT_WATERMARK || (nr % ECHO_BLOCK > old % ECHO_BLOCK))
+ if (nr < ECHO_COMMIT_WATERMARK ||
+ (nr % ECHO_BLOCK > old % ECHO_BLOCK)) {
+ mutex_unlock(&ldata->output_lock);
return;
+ }
- mutex_lock(&ldata->output_lock);
ldata->echo_commit = head;
echoed = __process_echoes(tty);
mutex_unlock(&ldata->output_lock);
@@ -812,7 +840,9 @@ static void flush_echoes(struct tty_struct *tty)
static inline void add_echo_byte(unsigned char c, struct n_tty_data *ldata)
{
- *echo_buf_addr(ldata, ldata->echo_head++) = c;
+ *echo_buf_addr(ldata, ldata->echo_head) = c;
+ smp_wmb(); /* Matches smp_rmb() in echo_buf(). */
+ ldata->echo_head++;
}
/**
@@ -980,14 +1010,15 @@ static void eraser(unsigned char c, struct tty_struct *tty)
}
seen_alnums = 0;
- while (ldata->read_head != ldata->canon_head) {
+ while (MASK(ldata->read_head) != MASK(ldata->canon_head)) {
head = ldata->read_head;
/* erase a single possibly multibyte character */
do {
head--;
c = read_buf(ldata, head);
- } while (is_continuation(c, tty) && head != ldata->canon_head);
+ } while (is_continuation(c, tty) &&
+ MASK(head) != MASK(ldata->canon_head));
/* do not partially erase */
if (is_continuation(c, tty))
@@ -1029,7 +1060,7 @@ static void eraser(unsigned char c, struct tty_struct *tty)
* This info is used to go back the correct
* number of columns.
*/
- while (tail != ldata->canon_head) {
+ while (MASK(tail) != MASK(ldata->canon_head)) {
tail--;
c = read_buf(ldata, tail);
if (c == '\t') {
@@ -1304,7 +1335,7 @@ n_tty_receive_char_special(struct tty_struct *tty, unsigned char c)
finish_erasing(ldata);
echo_char(c, tty);
echo_char_raw('\n', ldata);
- while (tail != ldata->read_head) {
+ while (MASK(tail) != MASK(ldata->read_head)) {
echo_char(read_buf(ldata, tail), tty);
tail++;
}
@@ -1880,30 +1911,21 @@ static int n_tty_open(struct tty_struct *tty)
struct n_tty_data *ldata;
/* Currently a malloc failure here can panic */
- ldata = vmalloc(sizeof(*ldata));
+ ldata = vzalloc(sizeof(*ldata));
if (!ldata)
- goto err;
+ return -ENOMEM;
ldata->overrun_time = jiffies;
mutex_init(&ldata->atomic_read_lock);
mutex_init(&ldata->output_lock);
tty->disc_data = ldata;
- reset_buffer_flags(tty->disc_data);
- ldata->column = 0;
- ldata->canon_column = 0;
- ldata->num_overrun = 0;
- ldata->no_room = 0;
- ldata->lnext = 0;
tty->closing = 0;
/* indicate buffer work may resume */
clear_bit(TTY_LDISC_HALTED, &tty->flags);
n_tty_set_termios(tty, NULL);
tty_unthrottle(tty);
-
return 0;
-err:
- return -ENOMEM;
}
static inline int input_available_p(struct tty_struct *tty, int poll)
@@ -1953,11 +1975,12 @@ static int copy_from_read_buf(struct tty_struct *tty,
n = min(head - ldata->read_tail, N_TTY_BUF_SIZE - tail);
n = min(*nr, n);
if (n) {
- const unsigned char *from = read_buf_addr(ldata, tail);
+ unsigned char *from = read_buf_addr(ldata, tail);
retval = copy_to_user(*b, from, n);
n -= retval;
is_eof = n == 1 && *from == EOF_CHAR(tty);
tty_audit_add_data(tty, from, n);
+ zero_buffer(tty, from, n);
smp_store_release(&ldata->read_tail, ldata->read_tail + n);
/* Turn single EOF into zero-length read */
if (L_EXTPROC(tty) && ldata->icanon && is_eof &&
@@ -2413,7 +2436,7 @@ static unsigned long inq_canon(struct n_tty_data *ldata)
tail = ldata->read_tail;
nr = head - tail;
/* Skip EOF-chars.. */
- while (head != tail) {
+ while (MASK(head) != MASK(tail)) {
if (test_bit(tail & (N_TTY_BUF_SIZE - 1), ldata->read_flags) &&
read_buf(ldata, tail) == __DISABLED_CHAR)
nr--;
diff --git a/drivers/tty/pty.c b/drivers/tty/pty.c
index 2b90738..171130a 100644
--- a/drivers/tty/pty.c
+++ b/drivers/tty/pty.c
@@ -106,16 +106,19 @@ static void pty_unthrottle(struct tty_struct *tty)
static int pty_write(struct tty_struct *tty, const unsigned char *buf, int c)
{
struct tty_struct *to = tty->link;
+ unsigned long flags;
if (tty->stopped)
return 0;
if (c > 0) {
+ spin_lock_irqsave(&to->port->lock, flags);
/* Stuff the data into the input queue of the other end */
c = tty_insert_flip_string(to->port, buf, c);
/* And shovel */
if (c)
tty_flip_buffer_push(to->port);
+ spin_unlock_irqrestore(&to->port->lock, flags);
}
return c;
}
diff --git a/drivers/tty/rocket.c b/drivers/tty/rocket.c
index b0cc47c..e8e8973 100644
--- a/drivers/tty/rocket.c
+++ b/drivers/tty/rocket.c
@@ -1913,7 +1913,7 @@ static __init int register_PCI(int i, struct pci_dev *dev)
ByteIO_t UPCIRingInd = 0;
if (!dev || !pci_match_id(rocket_pci_ids, dev) ||
- pci_enable_device(dev))
+ pci_enable_device(dev) || i >= NUM_BOARDS)
return 0;
rcktpt_io_addr[i] = pci_resource_start(dev, 0);
diff --git a/drivers/tty/serial/8250/8250_dw.c b/drivers/tty/serial/8250/8250_dw.c
index 3eb01a71..3177264 100644
--- a/drivers/tty/serial/8250/8250_dw.c
+++ b/drivers/tty/serial/8250/8250_dw.c
@@ -235,7 +235,7 @@ static void dw8250_set_termios(struct uart_port *p, struct ktermios *termios,
unsigned int rate;
int ret;
- if (IS_ERR(d->clk) || !old)
+ if (IS_ERR(d->clk))
goto out;
clk_disable_unprepare(d->clk);
@@ -626,6 +626,7 @@ static const struct acpi_device_id dw8250_acpi_match[] = {
{ "APMC0D08", 0},
{ "AMD0020", 0 },
{ "AMDI0020", 0 },
+ { "BRCM2032", 0 },
{ "HISI0031", 0 },
{ },
};
diff --git a/drivers/tty/serial/8250/8250_mtk.c b/drivers/tty/serial/8250/8250_mtk.c
index ce0cc47..91db9ca1 100644
--- a/drivers/tty/serial/8250/8250_mtk.c
+++ b/drivers/tty/serial/8250/8250_mtk.c
@@ -225,17 +225,17 @@ static int mtk8250_probe(struct platform_device *pdev)
platform_set_drvdata(pdev, data);
- pm_runtime_enable(&pdev->dev);
- if (!pm_runtime_enabled(&pdev->dev)) {
- err = mtk8250_runtime_resume(&pdev->dev);
- if (err)
- return err;
- }
+ err = mtk8250_runtime_resume(&pdev->dev);
+ if (err)
+ return err;
data->line = serial8250_register_8250_port(&uart);
if (data->line < 0)
return data->line;
+ pm_runtime_set_active(&pdev->dev);
+ pm_runtime_enable(&pdev->dev);
+
return 0;
}
@@ -246,13 +246,11 @@ static int mtk8250_remove(struct platform_device *pdev)
pm_runtime_get_sync(&pdev->dev);
serial8250_unregister_port(data->line);
+ mtk8250_runtime_suspend(&pdev->dev);
pm_runtime_disable(&pdev->dev);
pm_runtime_put_noidle(&pdev->dev);
- if (!pm_runtime_status_suspended(&pdev->dev))
- mtk8250_runtime_suspend(&pdev->dev);
-
return 0;
}
diff --git a/drivers/tty/serial/8250/8250_omap.c b/drivers/tty/serial/8250/8250_omap.c
index e8b34f1..a3adf21 100644
--- a/drivers/tty/serial/8250/8250_omap.c
+++ b/drivers/tty/serial/8250/8250_omap.c
@@ -1078,13 +1078,14 @@ static int omap8250_no_handle_irq(struct uart_port *port)
return 0;
}
+static const u8 omap4_habit = UART_ERRATA_CLOCK_DISABLE;
static const u8 am3352_habit = OMAP_DMA_TX_KICK | UART_ERRATA_CLOCK_DISABLE;
static const u8 dra742_habit = UART_ERRATA_CLOCK_DISABLE;
static const struct of_device_id omap8250_dt_ids[] = {
{ .compatible = "ti,omap2-uart" },
{ .compatible = "ti,omap3-uart" },
- { .compatible = "ti,omap4-uart" },
+ { .compatible = "ti,omap4-uart", .data = &omap4_habit, },
{ .compatible = "ti,am3352-uart", .data = &am3352_habit, },
{ .compatible = "ti,am4372-uart", .data = &am3352_habit, },
{ .compatible = "ti,dra742-uart", .data = &dra742_habit, },
@@ -1326,6 +1327,19 @@ static int omap8250_soft_reset(struct device *dev)
int sysc;
int syss;
+ /*
+ * At least on omap4, unused uarts may not idle after reset without
+ * a basic scr dma configuration even with no dma in use. The
+ * module clkctrl status bits will be 1 instead of 3 blocking idle
+ * for the whole clockdomain. The softreset below will clear scr,
+ * and we restore it on resume so this is safe to do on all SoCs
+ * needing omap8250_soft_reset() quirk. Do it in two writes as
+ * recommended in the comment for omap8250_update_scr().
+ */
+ serial_out(up, UART_OMAP_SCR, OMAP_UART_SCR_DMAMODE_1);
+ serial_out(up, UART_OMAP_SCR,
+ OMAP_UART_SCR_DMAMODE_1 | OMAP_UART_SCR_DMAMODE_CTL);
+
sysc = serial_in(up, UART_OMAP_SYSC);
/* softreset the UART */
diff --git a/drivers/tty/serial/8250/8250_port.c b/drivers/tty/serial/8250/8250_port.c
index f6e4373..5b54439 100644
--- a/drivers/tty/serial/8250/8250_port.c
+++ b/drivers/tty/serial/8250/8250_port.c
@@ -83,8 +83,7 @@ static const struct serial8250_config uart_config[] = {
.name = "16550A",
.fifo_size = 16,
.tx_loadsz = 16,
- .fcr = UART_FCR_ENABLE_FIFO | UART_FCR_R_TRIG_10 |
- UART_FCR_CLEAR_RCVR | UART_FCR_CLEAR_XMIT,
+ .fcr = UART_FCR_ENABLE_FIFO | UART_FCR_R_TRIG_10,
.rxtrig_bytes = {1, 4, 8, 14},
.flags = UART_CAP_FIFO,
},
@@ -1815,7 +1814,8 @@ int serial8250_handle_irq(struct uart_port *port, unsigned int iir)
status = serial_port_in(port, UART_LSR);
- if (status & (UART_LSR_DR | UART_LSR_BI)) {
+ if (status & (UART_LSR_DR | UART_LSR_BI) &&
+ iir & UART_IIR_RDI) {
if (!up->dma || handle_rx_dma(up, iir))
status = serial8250_rx_chars(up, status);
}
diff --git a/drivers/tty/serial/8250/serial_cs.c b/drivers/tty/serial/8250/serial_cs.c
index 933c268..8106353 100644
--- a/drivers/tty/serial/8250/serial_cs.c
+++ b/drivers/tty/serial/8250/serial_cs.c
@@ -637,8 +637,10 @@ static int serial_config(struct pcmcia_device *link)
(link->has_func_id) &&
(link->socket->pcmcia_pfc == 0) &&
((link->func_id == CISTPL_FUNCID_MULTI) ||
- (link->func_id == CISTPL_FUNCID_SERIAL)))
- pcmcia_loop_config(link, serial_check_for_multi, info);
+ (link->func_id == CISTPL_FUNCID_SERIAL))) {
+ if (pcmcia_loop_config(link, serial_check_for_multi, info))
+ goto failed;
+ }
/*
* Apply any multi-port quirk.
diff --git a/drivers/tty/serial/amba-pl011.c b/drivers/tty/serial/amba-pl011.c
index b42d7f1..41b0dd6 100644
--- a/drivers/tty/serial/amba-pl011.c
+++ b/drivers/tty/serial/amba-pl011.c
@@ -1726,10 +1726,26 @@ static int pl011_allocate_irq(struct uart_amba_port *uap)
*/
static void pl011_enable_interrupts(struct uart_amba_port *uap)
{
+ unsigned int i;
+
spin_lock_irq(&uap->port.lock);
/* Clear out any spuriously appearing RX interrupts */
pl011_write(UART011_RTIS | UART011_RXIS, uap, REG_ICR);
+
+ /*
+ * RXIS is asserted only when the RX FIFO transitions from below
+ * to above the trigger threshold. If the RX FIFO is already
+ * full to the threshold this can't happen and RXIS will now be
+ * stuck off. Drain the RX FIFO explicitly to fix this:
+ */
+ for (i = 0; i < uap->fifosize * 2; ++i) {
+ if (pl011_read(uap, REG_FR) & UART01x_FR_RXFE)
+ break;
+
+ pl011_read(uap, REG_DR);
+ }
+
uap->im = UART011_RTIM;
if (!pl011_dma_rx_running(uap))
uap->im |= UART011_RXIM;
@@ -2320,12 +2336,67 @@ static int __init pl011_console_setup(struct console *co, char *options)
return uart_set_options(&uap->port, co, baud, parity, bits, flow);
}
+/**
+ * pl011_console_match - non-standard console matching
+ * @co: registering console
+ * @name: name from console command line
+ * @idx: index from console command line
+ * @options: ptr to option string from console command line
+ *
+ * Only attempts to match console command lines of the form:
+ * console=pl011,mmio|mmio32,<addr>[,<options>]
+ * console=pl011,0x<addr>[,<options>]
+ * This form is used to register an initial earlycon boot console and
+ * replace it with the amba_console at pl011 driver init.
+ *
+ * Performs console setup for a match (as required by interface)
+ * If no <options> are specified, then assume the h/w is already setup.
+ *
+ * Returns 0 if console matches; otherwise non-zero to use default matching
+ */
+static int __init pl011_console_match(struct console *co, char *name, int idx,
+ char *options)
+{
+ unsigned char iotype;
+ resource_size_t addr;
+ int i;
+
+ if (strcmp(name, "pl011") != 0)
+ return -ENODEV;
+
+ if (uart_parse_earlycon(options, &iotype, &addr, &options))
+ return -ENODEV;
+
+ if (iotype != UPIO_MEM && iotype != UPIO_MEM32)
+ return -ENODEV;
+
+ /* try to match the port specified on the command line */
+ for (i = 0; i < ARRAY_SIZE(amba_ports); i++) {
+ struct uart_port *port;
+
+ if (!amba_ports[i])
+ continue;
+
+ port = &amba_ports[i]->port;
+
+ if (port->mapbase != addr)
+ continue;
+
+ co->index = i;
+ port->cons = co;
+ return pl011_console_setup(co, options);
+ }
+
+ return -ENODEV;
+}
+
static struct uart_driver amba_reg;
static struct console amba_console = {
.name = "ttyAMA",
.write = pl011_console_write,
.device = uart_console_device,
.setup = pl011_console_setup,
+ .match = pl011_console_match,
.flags = CON_PRINTBUFFER,
.index = -1,
.data = &amba_reg,
diff --git a/drivers/tty/serial/arc_uart.c b/drivers/tty/serial/arc_uart.c
index 5ac06fc..fec48deb 100644
--- a/drivers/tty/serial/arc_uart.c
+++ b/drivers/tty/serial/arc_uart.c
@@ -596,6 +596,11 @@ static int arc_serial_probe(struct platform_device *pdev)
if (dev_id < 0)
dev_id = 0;
+ if (dev_id >= ARRAY_SIZE(arc_uart_ports)) {
+ dev_err(&pdev->dev, "serial%d out of range\n", dev_id);
+ return -EINVAL;
+ }
+
uart = &arc_uart_ports[dev_id];
port = &uart->port;
diff --git a/drivers/tty/serial/atmel_serial.c b/drivers/tty/serial/atmel_serial.c
index addb287..5a341b1 100644
--- a/drivers/tty/serial/atmel_serial.c
+++ b/drivers/tty/serial/atmel_serial.c
@@ -1803,7 +1803,6 @@ static int atmel_startup(struct uart_port *port)
{
struct platform_device *pdev = to_platform_device(port->dev);
struct atmel_uart_port *atmel_port = to_atmel_uart_port(port);
- struct tty_struct *tty = port->state->port.tty;
int retval;
/*
@@ -1818,8 +1817,8 @@ static int atmel_startup(struct uart_port *port)
* Allocate the IRQ
*/
retval = request_irq(port->irq, atmel_interrupt,
- IRQF_SHARED | IRQF_COND_SUSPEND,
- tty ? tty->name : "atmel_serial", port);
+ IRQF_SHARED | IRQF_COND_SUSPEND,
+ dev_name(&pdev->dev), port);
if (retval) {
dev_err(port->dev, "atmel_startup - Can't get irq\n");
return retval;
diff --git a/drivers/tty/serial/cpm_uart/cpm_uart_core.c b/drivers/tty/serial/cpm_uart/cpm_uart_core.c
index d3e3d42..0040c29f 100644
--- a/drivers/tty/serial/cpm_uart/cpm_uart_core.c
+++ b/drivers/tty/serial/cpm_uart/cpm_uart_core.c
@@ -1068,8 +1068,8 @@ static int poll_wait_key(char *obuf, struct uart_cpm_port *pinfo)
/* Get the address of the host memory buffer.
*/
bdp = pinfo->rx_cur;
- while (bdp->cbd_sc & BD_SC_EMPTY)
- ;
+ if (bdp->cbd_sc & BD_SC_EMPTY)
+ return NO_POLL_CHAR;
/* If the buffer address is in the CPM DPRAM, don't
* convert it.
@@ -1104,7 +1104,11 @@ static int cpm_get_poll_char(struct uart_port *port)
poll_chars = 0;
}
if (poll_chars <= 0) {
- poll_chars = poll_wait_key(poll_buf, pinfo);
+ int ret = poll_wait_key(poll_buf, pinfo);
+
+ if (ret == NO_POLL_CHAR)
+ return ret;
+ poll_chars = ret;
pollp = poll_buf;
}
poll_chars--;
diff --git a/drivers/tty/serial/earlycon.c b/drivers/tty/serial/earlycon.c
index 3b31fd8..b9a4625 100644
--- a/drivers/tty/serial/earlycon.c
+++ b/drivers/tty/serial/earlycon.c
@@ -172,7 +172,7 @@ static int __init register_earlycon(char *buf, const struct earlycon_id *match)
*/
int __init setup_earlycon(char *buf)
{
- const struct earlycon_id *match;
+ const struct earlycon_id **p_match;
if (!buf || !buf[0])
return -EINVAL;
@@ -180,7 +180,9 @@ int __init setup_earlycon(char *buf)
if (early_con.flags & CON_ENABLED)
return -EALREADY;
- for (match = __earlycon_table; match < __earlycon_table_end; match++) {
+ for (p_match = __earlycon_table; p_match < __earlycon_table_end;
+ p_match++) {
+ const struct earlycon_id *match = *p_match;
size_t len = strlen(match->name);
if (strncmp(buf, match->name, len))
diff --git a/drivers/tty/serial/fsl_lpuart.c b/drivers/tty/serial/fsl_lpuart.c
index 76103f2..e2ec049 100644
--- a/drivers/tty/serial/fsl_lpuart.c
+++ b/drivers/tty/serial/fsl_lpuart.c
@@ -833,7 +833,8 @@ static inline int lpuart_start_rx_dma(struct lpuart_port *sport)
struct circ_buf *ring = &sport->rx_ring;
int ret, nent;
int bits, baud;
- struct tty_struct *tty = tty_port_tty_get(&sport->port.state->port);
+ struct tty_port *port = &sport->port.state->port;
+ struct tty_struct *tty = port->tty;
struct ktermios *termios = &tty->termios;
baud = tty_get_baud_rate(tty);
@@ -1902,6 +1903,10 @@ static int lpuart_probe(struct platform_device *pdev)
dev_err(&pdev->dev, "failed to get alias id, errno %d\n", ret);
return ret;
}
+ if (ret >= ARRAY_SIZE(lpuart_ports)) {
+ dev_err(&pdev->dev, "serial%d out of range\n", ret);
+ return -EINVAL;
+ }
sport->port.line = ret;
sport->lpuart32 = of_device_is_compatible(np, "fsl,ls1021a-lpuart");
diff --git a/drivers/tty/serial/imx.c b/drivers/tty/serial/imx.c
index f575a33..0d82be1 100644
--- a/drivers/tty/serial/imx.c
+++ b/drivers/tty/serial/imx.c
@@ -2080,6 +2080,12 @@ static int serial_imx_probe(struct platform_device *pdev)
else if (ret < 0)
return ret;
+ if (sport->port.line >= ARRAY_SIZE(imx_ports)) {
+ dev_err(&pdev->dev, "serial%d out of range\n",
+ sport->port.line);
+ return -EINVAL;
+ }
+
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
base = devm_ioremap_resource(&pdev->dev, res);
if (IS_ERR(base))
@@ -2145,7 +2151,9 @@ static int serial_imx_probe(struct platform_device *pdev)
* and DCD (when they are outputs) or enables the respective
* irqs. So set this bit early, i.e. before requesting irqs.
*/
- writel(UFCR_DCEDTE, sport->port.membase + UFCR);
+ reg = readl(sport->port.membase + UFCR);
+ if (!(reg & UFCR_DCEDTE))
+ writel(reg | UFCR_DCEDTE, sport->port.membase + UFCR);
/*
* Disable UCR3_RI and UCR3_DCD irqs. They are also not
@@ -2156,7 +2164,15 @@ static int serial_imx_probe(struct platform_device *pdev)
sport->port.membase + UCR3);
} else {
- writel(0, sport->port.membase + UFCR);
+ unsigned long ucr3 = UCR3_DSR;
+
+ reg = readl(sport->port.membase + UFCR);
+ if (reg & UFCR_DCEDTE)
+ writel(reg & ~UFCR_DCEDTE, sport->port.membase + UFCR);
+
+ if (!is_imx1_uart(sport))
+ ucr3 |= IMX21_UCR3_RXDMUXSEL | UCR3_ADNIMP;
+ writel(ucr3, sport->port.membase + UCR3);
}
clk_disable_unprepare(sport->clk_ipg);
@@ -2181,6 +2197,14 @@ static int serial_imx_probe(struct platform_device *pdev)
ret);
return ret;
}
+
+ ret = devm_request_irq(&pdev->dev, rtsirq, imx_rtsint, 0,
+ dev_name(&pdev->dev), sport);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to request rts irq: %d\n",
+ ret);
+ return ret;
+ }
} else {
ret = devm_request_irq(&pdev->dev, rxirq, imx_int, 0,
dev_name(&pdev->dev), sport);
diff --git a/drivers/tty/serial/kgdboc.c b/drivers/tty/serial/kgdboc.c
index 5532c44..6c0c6c5d 100644
--- a/drivers/tty/serial/kgdboc.c
+++ b/drivers/tty/serial/kgdboc.c
@@ -131,19 +131,6 @@ static void kgdboc_unregister_kbd(void)
#define kgdboc_restore_input()
#endif /* ! CONFIG_KDB_KEYBOARD */
-static int kgdboc_option_setup(char *opt)
-{
- if (strlen(opt) >= MAX_CONFIG_LEN) {
- printk(KERN_ERR "kgdboc: config string too long\n");
- return -ENOSPC;
- }
- strcpy(config, opt);
-
- return 0;
-}
-
-__setup("kgdboc=", kgdboc_option_setup);
-
static void cleanup_kgdboc(void)
{
if (kgdb_unregister_nmi_console())
@@ -157,15 +144,13 @@ static int configure_kgdboc(void)
{
struct tty_driver *p;
int tty_line = 0;
- int err;
+ int err = -ENODEV;
char *cptr = config;
struct console *cons;
- err = kgdboc_option_setup(config);
- if (err || !strlen(config) || isspace(config[0]))
+ if (!strlen(config) || isspace(config[0]))
goto noconfig;
- err = -ENODEV;
kgdboc_io_ops.is_console = 0;
kgdb_tty_driver = NULL;
@@ -248,7 +233,7 @@ static void kgdboc_put_char(u8 chr)
static int param_set_kgdboc_var(const char *kmessage,
const struct kernel_param *kp)
{
- int len = strlen(kmessage);
+ size_t len = strlen(kmessage);
if (len >= MAX_CONFIG_LEN) {
printk(KERN_ERR "kgdboc: config string too long\n");
@@ -270,7 +255,7 @@ static int param_set_kgdboc_var(const char *kmessage,
strcpy(config, kmessage);
/* Chop out \n char as a result of echo */
- if (config[len - 1] == '\n')
+ if (len && config[len - 1] == '\n')
config[len - 1] = '\0';
if (configured == 1)
@@ -314,6 +299,25 @@ static struct kgdb_io kgdboc_io_ops = {
};
#ifdef CONFIG_KGDB_SERIAL_CONSOLE
+static int kgdboc_option_setup(char *opt)
+{
+ if (!opt) {
+ pr_err("config string not provided\n");
+ return -EINVAL;
+ }
+
+ if (strlen(opt) >= MAX_CONFIG_LEN) {
+ pr_err("config string too long\n");
+ return -ENOSPC;
+ }
+ strcpy(config, opt);
+
+ return 0;
+}
+
+__setup("kgdboc=", kgdboc_option_setup);
+
+
/* This is only available if kgdboc is a built in for early debugging */
static int __init kgdboc_early_init(char *opt)
{
diff --git a/drivers/tty/serial/msm_geni_serial.c b/drivers/tty/serial/msm_geni_serial.c
index 9025707..782780e 100644
--- a/drivers/tty/serial/msm_geni_serial.c
+++ b/drivers/tty/serial/msm_geni_serial.c
@@ -107,7 +107,7 @@
#define STALE_TIMEOUT (16)
#define DEFAULT_BITS_PER_CHAR (10)
#define GENI_UART_NR_PORTS (15)
-#define GENI_UART_CONS_PORTS (1)
+#define GENI_UART_CONS_PORTS (2)
#define DEF_FIFO_DEPTH_WORDS (16)
#define DEF_TX_WM (2)
#define DEF_FIFO_WIDTH_BITS (32)
@@ -195,7 +195,7 @@ static atomic_t uart_line_id = ATOMIC_INIT(0);
#define GET_DEV_PORT(uport) \
container_of(uport, struct msm_geni_serial_port, uport)
-static struct msm_geni_serial_port msm_geni_console_port;
+static struct msm_geni_serial_port msm_geni_console_port[GENI_UART_CONS_PORTS];
static struct msm_geni_serial_port msm_geni_serial_ports[GENI_UART_NR_PORTS];
static void msm_geni_serial_config_port(struct uart_port *uport, int cfg_flags)
@@ -465,7 +465,7 @@ static struct msm_geni_serial_port *get_port_from_line(int line,
if (is_console) {
if ((line < 0) || (line >= GENI_UART_CONS_PORTS))
port = ERR_PTR(-ENXIO);
- port = &msm_geni_console_port;
+ port = &msm_geni_console_port[line];
} else {
if ((line < 0) || (line >= GENI_UART_NR_PORTS))
return ERR_PTR(-ENXIO);
@@ -707,11 +707,11 @@ static void
__msm_geni_serial_console_write(struct uart_port *uport, const char *s,
unsigned int count)
{
- int new_line = 0;
- int i;
- int bytes_to_send = count;
- int fifo_depth = DEF_FIFO_DEPTH_WORDS;
- int tx_wm = DEF_TX_WM;
+ unsigned int new_line = 0;
+ unsigned int i;
+ unsigned int bytes_to_send = count;
+ unsigned int fifo_depth = DEF_FIFO_DEPTH_WORDS;
+ unsigned int tx_wm = DEF_TX_WM;
for (i = 0; i < count; i++) {
if (s[i] == '\n')
@@ -732,11 +732,10 @@ __msm_geni_serial_console_write(struct uart_port *uport, const char *s,
* command. Unfortunately the current data being written is
* lost.
*/
- while (!msm_geni_serial_poll_bit(uport, SE_GENI_M_IRQ_STATUS,
+ if (!msm_geni_serial_poll_bit(uport, SE_GENI_M_IRQ_STATUS,
M_TX_FIFO_WATERMARK_EN, true))
break;
- chars_to_write = min((unsigned int)(count - i),
- avail_fifo_bytes);
+ chars_to_write = min((count - i), avail_fifo_bytes);
if ((chars_to_write << 1) > avail_fifo_bytes)
chars_to_write = (avail_fifo_bytes >> 1);
uart_console_write(uport, (s + i), chars_to_write,
@@ -751,7 +750,7 @@ __msm_geni_serial_console_write(struct uart_port *uport, const char *s,
}
static void msm_geni_serial_console_write(struct console *co, const char *s,
- unsigned int count)
+ unsigned count)
{
struct uart_port *uport;
struct msm_geni_serial_port *port;
@@ -1242,56 +1241,30 @@ static int msm_geni_serial_handle_tx(struct uart_port *uport)
unsigned int xmit_size;
unsigned int fifo_width_bytes =
(uart_console(uport) ? 1 : (msm_port->tx_fifo_width >> 3));
- unsigned int geni_m_irq_en;
int temp_tail = 0;
xmit_size = uart_circ_chars_pending(xmit);
tx_fifo_status = geni_read_reg_nolog(uport->membase,
SE_GENI_TX_FIFO_STATUS);
/* Both FIFO and framework buffer are drained */
- if ((xmit_size == msm_port->xmit_size) && !tx_fifo_status) {
- /*
- * This will balance out the power vote put in during start_tx
- * allowing the device to suspend.
- */
- if (!uart_console(uport)) {
- IPC_LOG_MSG(msm_port->ipc_log_misc,
- "%s.Power Off.\n", __func__);
- msm_geni_serial_power_off(uport);
- }
- msm_port->xmit_size = 0;
- uart_circ_clear(xmit);
+ if (!xmit_size && !tx_fifo_status) {
msm_geni_serial_stop_tx(uport);
goto exit_handle_tx;
}
- xmit_size -= msm_port->xmit_size;
-
- if (!uart_console(uport)) {
- geni_m_irq_en = geni_read_reg_nolog(uport->membase,
- SE_GENI_M_IRQ_EN);
- geni_m_irq_en &= ~(M_TX_FIFO_WATERMARK_EN);
- geni_write_reg_nolog(0, uport->membase,
- SE_GENI_TX_WATERMARK_REG);
- geni_write_reg_nolog(geni_m_irq_en, uport->membase,
- SE_GENI_M_IRQ_EN);
- }
avail_fifo_bytes = (msm_port->tx_fifo_depth - msm_port->tx_wm) *
fifo_width_bytes;
- temp_tail = (xmit->tail + msm_port->xmit_size) & (UART_XMIT_SIZE - 1);
+ temp_tail = xmit->tail & (UART_XMIT_SIZE - 1);
+
if (xmit_size > (UART_XMIT_SIZE - temp_tail))
xmit_size = (UART_XMIT_SIZE - temp_tail);
if (xmit_size > avail_fifo_bytes)
xmit_size = avail_fifo_bytes;
-
if (!xmit_size)
goto exit_handle_tx;
msm_geni_serial_setup_tx(uport, xmit_size);
-
bytes_remaining = xmit_size;
- dump_ipc(msm_port->ipc_log_tx, "Tx", (char *)&xmit->buf[temp_tail], 0,
- xmit_size);
while (i < xmit_size) {
unsigned int tx_bytes;
unsigned int buf = 0;
@@ -1304,17 +1277,18 @@ static int msm_geni_serial_handle_tx(struct uart_port *uport)
buf |= (xmit->buf[temp_tail + c] << (c * 8));
geni_write_reg_nolog(buf, uport->membase, SE_GENI_TX_FIFOn);
i += tx_bytes;
- temp_tail = (temp_tail + tx_bytes) & (UART_XMIT_SIZE - 1);
- uport->icount.tx += tx_bytes;
bytes_remaining -= tx_bytes;
+ uport->icount.tx += tx_bytes;
+ temp_tail += tx_bytes;
/* Ensure FIFO write goes through */
wmb();
}
+ xmit->tail = temp_tail & (UART_XMIT_SIZE - 1);
if (uart_console(uport))
msm_geni_serial_poll_cancel_tx(uport);
- msm_port->xmit_size += xmit_size;
exit_handle_tx:
- uart_write_wakeup(uport);
+ if (uart_circ_chars_pending(xmit) < WAKEUP_CHARS)
+ uart_write_wakeup(uport);
return ret;
}
@@ -2094,7 +2068,7 @@ static int __init msm_geni_console_setup(struct console *co, char *options)
static void
msm_geni_serial_early_console_write(struct console *con, const char *s,
- unsigned int n)
+ unsigned n)
{
struct earlycon_device *dev = con->data;
@@ -2740,10 +2714,10 @@ static int __init msm_geni_serial_init(void)
}
for (i = 0; i < GENI_UART_CONS_PORTS; i++) {
- msm_geni_console_port.uport.iotype = UPIO_MEM;
- msm_geni_console_port.uport.ops = &msm_geni_console_pops;
- msm_geni_console_port.uport.flags = UPF_BOOT_AUTOCONF;
- msm_geni_console_port.uport.line = i;
+ msm_geni_console_port[i].uport.iotype = UPIO_MEM;
+ msm_geni_console_port[i].uport.ops = &msm_geni_console_pops;
+ msm_geni_console_port[i].uport.flags = UPF_BOOT_AUTOCONF;
+ msm_geni_console_port[i].uport.line = i;
}
ret = console_register(&msm_geni_console_driver);
diff --git a/drivers/tty/serial/msm_serial_hs.c b/drivers/tty/serial/msm_serial_hs.c
index a2522c6..d099b32 100644
--- a/drivers/tty/serial/msm_serial_hs.c
+++ b/drivers/tty/serial/msm_serial_hs.c
@@ -61,6 +61,7 @@
#include <linux/ipc_logging.h>
#include <asm/irq.h>
#include <linux/kthread.h>
+#include <uapi/linux/sched.h>
#include <linux/msm-sps.h>
#include <linux/platform_data/msm_serial_hs.h>
@@ -2789,6 +2790,7 @@ static int uartdm_init_port(struct uart_port *uport)
struct msm_hs_port *msm_uport = UARTDM_TO_MSM(uport);
struct msm_hs_tx *tx = &msm_uport->tx;
struct msm_hs_rx *rx = &msm_uport->rx;
+ struct sched_param param = { .sched_priority = 1 };
init_waitqueue_head(&rx->wait);
init_waitqueue_head(&tx->wait);
@@ -2803,6 +2805,8 @@ static int uartdm_init_port(struct uart_port *uport)
MSM_HS_ERR("%s(): error creating task", __func__);
goto exit_lh_init;
}
+ sched_setscheduler(rx->task, SCHED_FIFO, ¶m);
+
kthread_init_work(&rx->kwork, msm_serial_hs_rx_work);
kthread_init_worker(&tx->kworker);
@@ -2812,6 +2816,7 @@ static int uartdm_init_port(struct uart_port *uport)
MSM_HS_ERR("%s(): error creating task", __func__);
goto exit_lh_init;
}
+ sched_setscheduler(tx->task, SCHED_FIFO, ¶m);
kthread_init_work(&tx->kwork, msm_serial_hs_tx_work);
@@ -3326,6 +3331,7 @@ static void msm_serial_hs_rt_init(struct uart_port *uport)
msm_uport->pm_state = MSM_HS_PM_SUSPENDED;
mutex_unlock(&msm_uport->mtx);
pm_runtime_enable(uport->dev);
+ tty_port_set_policy(&uport->state->port, SCHED_FIFO, 1);
}
static int msm_hs_runtime_suspend(struct device *dev)
diff --git a/drivers/tty/serial/mvebu-uart.c b/drivers/tty/serial/mvebu-uart.c
index 45b57c2..401c983 100644
--- a/drivers/tty/serial/mvebu-uart.c
+++ b/drivers/tty/serial/mvebu-uart.c
@@ -327,8 +327,10 @@ static void mvebu_uart_set_termios(struct uart_port *port,
if ((termios->c_cflag & CREAD) == 0)
port->ignore_status_mask |= STAT_RX_RDY | STAT_BRK_ERR;
- if (old)
+ if (old) {
tty_termios_copy_hw(termios, old);
+ termios->c_cflag |= CS8;
+ }
baud = uart_get_baud_rate(port, termios, old, 0, 460800);
uart_update_timeout(port, termios->c_cflag, baud);
diff --git a/drivers/tty/serial/mxs-auart.c b/drivers/tty/serial/mxs-auart.c
index 07390f8..1d9d778 100644
--- a/drivers/tty/serial/mxs-auart.c
+++ b/drivers/tty/serial/mxs-auart.c
@@ -1664,6 +1664,10 @@ static int mxs_auart_probe(struct platform_device *pdev)
s->port.line = pdev->id < 0 ? 0 : pdev->id;
else if (ret < 0)
return ret;
+ if (s->port.line >= ARRAY_SIZE(auart_port)) {
+ dev_err(&pdev->dev, "serial%d out of range\n", s->port.line);
+ return -EINVAL;
+ }
if (of_id) {
pdev->id_entry = of_id->data;
diff --git a/drivers/tty/serial/samsung.c b/drivers/tty/serial/samsung.c
index d65f92b..5609305 100644
--- a/drivers/tty/serial/samsung.c
+++ b/drivers/tty/serial/samsung.c
@@ -866,15 +866,12 @@ static int s3c24xx_serial_request_dma(struct s3c24xx_uart_port *p)
dma->rx_conf.direction = DMA_DEV_TO_MEM;
dma->rx_conf.src_addr_width = DMA_SLAVE_BUSWIDTH_1_BYTE;
dma->rx_conf.src_addr = p->port.mapbase + S3C2410_URXH;
- dma->rx_conf.src_maxburst = 16;
+ dma->rx_conf.src_maxburst = 1;
dma->tx_conf.direction = DMA_MEM_TO_DEV;
dma->tx_conf.dst_addr_width = DMA_SLAVE_BUSWIDTH_1_BYTE;
dma->tx_conf.dst_addr = p->port.mapbase + S3C2410_UTXH;
- if (dma_get_cache_alignment() >= 16)
- dma->tx_conf.dst_maxburst = 16;
- else
- dma->tx_conf.dst_maxburst = 1;
+ dma->tx_conf.dst_maxburst = 1;
dma_cap_zero(mask);
dma_cap_set(DMA_SLAVE, mask);
@@ -1813,6 +1810,10 @@ static int s3c24xx_serial_probe(struct platform_device *pdev)
dbg("s3c24xx_serial_probe(%p) %d\n", pdev, index);
+ if (index >= ARRAY_SIZE(s3c24xx_serial_ports)) {
+ dev_err(&pdev->dev, "serial%d out of range\n", index);
+ return -EINVAL;
+ }
ourport = &s3c24xx_serial_ports[index];
ourport->drv_data = s3c24xx_get_driver_data(pdev);
diff --git a/drivers/tty/serial/sc16is7xx.c b/drivers/tty/serial/sc16is7xx.c
index 7933954..ea6b62c 100644
--- a/drivers/tty/serial/sc16is7xx.c
+++ b/drivers/tty/serial/sc16is7xx.c
@@ -661,7 +661,7 @@ static void sc16is7xx_handle_tx(struct uart_port *port)
uart_write_wakeup(port);
}
-static void sc16is7xx_port_irq(struct sc16is7xx_port *s, int portno)
+static bool sc16is7xx_port_irq(struct sc16is7xx_port *s, int portno)
{
struct uart_port *port = &s->p[portno].port;
@@ -670,7 +670,7 @@ static void sc16is7xx_port_irq(struct sc16is7xx_port *s, int portno)
iir = sc16is7xx_port_read(port, SC16IS7XX_IIR_REG);
if (iir & SC16IS7XX_IIR_NO_INT_BIT)
- break;
+ return false;
iir &= SC16IS7XX_IIR_ID_MASK;
@@ -692,16 +692,23 @@ static void sc16is7xx_port_irq(struct sc16is7xx_port *s, int portno)
port->line, iir);
break;
}
- } while (1);
+ } while (0);
+ return true;
}
static void sc16is7xx_ist(struct kthread_work *ws)
{
struct sc16is7xx_port *s = to_sc16is7xx_port(ws, irq_work);
- int i;
- for (i = 0; i < s->devtype->nr_uart; ++i)
- sc16is7xx_port_irq(s, i);
+ while (1) {
+ bool keep_polling = false;
+ int i;
+
+ for (i = 0; i < s->devtype->nr_uart; ++i)
+ keep_polling |= sc16is7xx_port_irq(s, i);
+ if (!keep_polling)
+ break;
+ }
}
static irqreturn_t sc16is7xx_irq(int irq, void *dev_id)
diff --git a/drivers/tty/serial/sccnxp.c b/drivers/tty/serial/sccnxp.c
index cdd2f94..b9c7a90 100644
--- a/drivers/tty/serial/sccnxp.c
+++ b/drivers/tty/serial/sccnxp.c
@@ -889,7 +889,16 @@ static int sccnxp_probe(struct platform_device *pdev)
goto err_out;
uartclk = 0;
} else {
- clk_prepare_enable(clk);
+ ret = clk_prepare_enable(clk);
+ if (ret)
+ goto err_out;
+
+ ret = devm_add_action_or_reset(&pdev->dev,
+ (void(*)(void *))clk_disable_unprepare,
+ clk);
+ if (ret)
+ goto err_out;
+
uartclk = clk_get_rate(clk);
}
@@ -988,7 +997,7 @@ static int sccnxp_probe(struct platform_device *pdev)
uart_unregister_driver(&s->uart);
err_out:
if (!IS_ERR(s->regulator))
- return regulator_disable(s->regulator);
+ regulator_disable(s->regulator);
return ret;
}
diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
index 839cee4..17c2ee2a 100644
--- a/drivers/tty/serial/serial_core.c
+++ b/drivers/tty/serial/serial_core.c
@@ -175,6 +175,7 @@ static int uart_port_startup(struct tty_struct *tty, struct uart_state *state,
{
struct uart_port *uport = uart_port_check(state);
unsigned long page;
+ unsigned long flags = 0;
int retval = 0;
if (uport->type == PORT_UNKNOWN)
@@ -189,15 +190,18 @@ static int uart_port_startup(struct tty_struct *tty, struct uart_state *state,
* Initialise and allocate the transmit and temporary
* buffer.
*/
- if (!state->xmit.buf) {
- /* This is protected by the per port mutex */
- page = get_zeroed_page(GFP_KERNEL);
- if (!page)
- return -ENOMEM;
+ page = get_zeroed_page(GFP_KERNEL);
+ if (!page)
+ return -ENOMEM;
+ uart_port_lock(state, flags);
+ if (!state->xmit.buf) {
state->xmit.buf = (unsigned char *) page;
uart_circ_clear(&state->xmit);
+ } else {
+ free_page(page);
}
+ uart_port_unlock(uport, flags);
retval = uport->ops->startup(uport);
if (retval == 0) {
@@ -256,6 +260,7 @@ static void uart_shutdown(struct tty_struct *tty, struct uart_state *state)
{
struct uart_port *uport = uart_port_check(state);
struct tty_port *port = &state->port;
+ unsigned long flags = 0;
/*
* Set the TTY IO error marker
@@ -288,10 +293,12 @@ static void uart_shutdown(struct tty_struct *tty, struct uart_state *state)
/*
* Free the transmit buffer page.
*/
+ uart_port_lock(state, flags);
if (state->xmit.buf) {
free_page((unsigned long)state->xmit.buf);
state->xmit.buf = NULL;
}
+ uart_port_unlock(uport, flags);
}
/**
diff --git a/drivers/tty/serial/sh-sci.c b/drivers/tty/serial/sh-sci.c
index 107f0d1..6ff53b6 100644
--- a/drivers/tty/serial/sh-sci.c
+++ b/drivers/tty/serial/sh-sci.c
@@ -2626,8 +2626,8 @@ static int sci_init_clocks(struct sci_port *sci_port, struct device *dev)
dev_dbg(dev, "failed to get %s (%ld)\n", clk_names[i],
PTR_ERR(clk));
else
- dev_dbg(dev, "clk %s is %pC rate %pCr\n", clk_names[i],
- clk, clk);
+ dev_dbg(dev, "clk %s is %pC rate %lu\n", clk_names[i],
+ clk, clk_get_rate(clk));
sci_port->clks[i] = IS_ERR(clk) ? NULL : clk;
}
return 0;
@@ -2807,16 +2807,15 @@ static void serial_console_write(struct console *co, const char *s,
unsigned long flags;
int locked = 1;
- local_irq_save(flags);
#if defined(SUPPORT_SYSRQ)
if (port->sysrq)
locked = 0;
else
#endif
if (oops_in_progress)
- locked = spin_trylock(&port->lock);
+ locked = spin_trylock_irqsave(&port->lock, flags);
else
- spin_lock(&port->lock);
+ spin_lock_irqsave(&port->lock, flags);
/* first save SCSCR then disable interrupts, keep clock source */
ctrl = serial_port_in(port, SCSCR);
@@ -2835,8 +2834,7 @@ static void serial_console_write(struct console *co, const char *s,
serial_port_out(port, SCSCR, ctrl);
if (locked)
- spin_unlock(&port->lock);
- local_irq_restore(flags);
+ spin_unlock_irqrestore(&port->lock, flags);
}
static int serial_console_setup(struct console *co, char *options)
diff --git a/drivers/tty/serial/suncore.c b/drivers/tty/serial/suncore.c
index 127472b..209f314 100644
--- a/drivers/tty/serial/suncore.c
+++ b/drivers/tty/serial/suncore.c
@@ -111,6 +111,7 @@ void sunserial_console_termios(struct console *con, struct device_node *uart_dp)
mode = of_get_property(dp, mode_prop, NULL);
if (!mode)
mode = "9600,8,n,1,-";
+ of_node_put(dp);
}
cflag = CREAD | HUPCL | CLOCAL;
diff --git a/drivers/tty/serial/sunsu.c b/drivers/tty/serial/sunsu.c
index 9ad98ea..26ae830 100644
--- a/drivers/tty/serial/sunsu.c
+++ b/drivers/tty/serial/sunsu.c
@@ -1393,22 +1393,43 @@ static inline struct console *SUNSU_CONSOLE(void)
static enum su_type su_get_type(struct device_node *dp)
{
struct device_node *ap = of_find_node_by_path("/aliases");
+ enum su_type rc = SU_PORT_PORT;
if (ap) {
const char *keyb = of_get_property(ap, "keyboard", NULL);
const char *ms = of_get_property(ap, "mouse", NULL);
+ struct device_node *match;
if (keyb) {
- if (dp == of_find_node_by_path(keyb))
- return SU_PORT_KBD;
+ match = of_find_node_by_path(keyb);
+
+ /*
+ * The pointer is used as an identifier not
+ * as a pointer, we can drop the refcount on
+ * the of__node immediately after getting it.
+ */
+ of_node_put(match);
+
+ if (dp == match) {
+ rc = SU_PORT_KBD;
+ goto out;
+ }
}
if (ms) {
- if (dp == of_find_node_by_path(ms))
- return SU_PORT_MS;
+ match = of_find_node_by_path(ms);
+
+ of_node_put(match);
+
+ if (dp == match) {
+ rc = SU_PORT_MS;
+ goto out;
+ }
}
}
- return SU_PORT_PORT;
+out:
+ of_node_put(ap);
+ return rc;
}
static int su_probe(struct platform_device *op)
diff --git a/drivers/tty/serial/xilinx_uartps.c b/drivers/tty/serial/xilinx_uartps.c
index dd4c02f..fcf2e51 100644
--- a/drivers/tty/serial/xilinx_uartps.c
+++ b/drivers/tty/serial/xilinx_uartps.c
@@ -128,7 +128,7 @@ MODULE_PARM_DESC(rx_timeout, "Rx timeout, 1-255");
#define CDNS_UART_IXR_RXTRIG 0x00000001 /* RX FIFO trigger interrupt */
#define CDNS_UART_IXR_RXFULL 0x00000004 /* RX FIFO full interrupt. */
#define CDNS_UART_IXR_RXEMPTY 0x00000002 /* RX FIFO empty interrupt. */
-#define CDNS_UART_IXR_MASK 0x00001FFF /* Valid bit mask */
+#define CDNS_UART_IXR_RXMASK 0x000021e7 /* Valid RX bit mask */
/*
* Do not enable parity error interrupt for the following
@@ -362,7 +362,7 @@ static irqreturn_t cdns_uart_isr(int irq, void *dev_id)
cdns_uart_handle_tx(dev_id);
isrstatus &= ~CDNS_UART_IXR_TXEMPTY;
}
- if (isrstatus & CDNS_UART_IXR_MASK)
+ if (isrstatus & CDNS_UART_IXR_RXMASK)
cdns_uart_handle_rx(dev_id, isrstatus);
spin_unlock(&port->lock);
@@ -1106,7 +1106,7 @@ static struct uart_port *cdns_uart_get_port(int id)
struct uart_port *port;
/* Try the given port id if failed use default method */
- if (cdns_uart_port[id].mapbase != 0) {
+ if (id < CDNS_UART_NR_PORTS && cdns_uart_port[id].mapbase != 0) {
/* Find the next unused port */
for (id = 0; id < CDNS_UART_NR_PORTS; id++)
if (cdns_uart_port[id].mapbase == 0)
diff --git a/drivers/tty/tty_buffer.c b/drivers/tty/tty_buffer.c
index e99f1c5..41b9a7c 100644
--- a/drivers/tty/tty_buffer.c
+++ b/drivers/tty/tty_buffer.c
@@ -458,6 +458,8 @@ int tty_ldisc_receive_buf(struct tty_ldisc *ld, unsigned char *p,
if (count && ld->ops->receive_buf)
ld->ops->receive_buf(ld->tty, p, f, count);
}
+ if (count > 0)
+ memset(p, 0, count);
return count;
}
EXPORT_SYMBOL_GPL(tty_ldisc_receive_buf);
diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c
index 4ee0a9d..e6c4321 100644
--- a/drivers/tty/tty_io.c
+++ b/drivers/tty/tty_io.c
@@ -354,7 +354,7 @@ struct tty_driver *tty_find_polling_driver(char *name, int *line)
mutex_lock(&tty_mutex);
/* Search through the tty devices to look for a match */
list_for_each_entry(p, &tty_drivers, tty_drivers) {
- if (strncmp(name, p->name, len) != 0)
+ if (!len || strncmp(name, p->name, len) != 0)
continue;
stp = str;
if (*stp == ',')
@@ -1475,6 +1475,7 @@ static void tty_driver_remove_tty(struct tty_driver *driver, struct tty_struct *
static int tty_reopen(struct tty_struct *tty)
{
struct tty_driver *driver = tty->driver;
+ int retval;
if (driver->type == TTY_DRIVER_TYPE_PTY &&
driver->subtype == PTY_TYPE_MASTER)
@@ -1488,10 +1489,14 @@ static int tty_reopen(struct tty_struct *tty)
tty->count++;
- if (!tty->ldisc)
- return tty_ldisc_reinit(tty, tty->termios.c_line);
+ if (tty->ldisc)
+ return 0;
- return 0;
+ retval = tty_ldisc_reinit(tty, tty->termios.c_line);
+ if (retval)
+ tty->count--;
+
+ return retval;
}
/**
@@ -3170,7 +3175,10 @@ struct tty_struct *alloc_tty_struct(struct tty_driver *driver, int idx)
kref_init(&tty->kref);
tty->magic = TTY_MAGIC;
- tty_ldisc_init(tty);
+ if (tty_ldisc_init(tty)) {
+ kfree(tty);
+ return NULL;
+ }
tty->session = NULL;
tty->pgrp = NULL;
mutex_init(&tty->legacy_mutex);
diff --git a/drivers/tty/tty_ioctl.c b/drivers/tty/tty_ioctl.c
index bf36ac9..11bb9a5 100644
--- a/drivers/tty/tty_ioctl.c
+++ b/drivers/tty/tty_ioctl.c
@@ -325,7 +325,7 @@ speed_t tty_termios_baud_rate(struct ktermios *termios)
else
cbaud += 15;
}
- return baud_table[cbaud];
+ return cbaud >= n_baud_table ? 0 : baud_table[cbaud];
}
EXPORT_SYMBOL(tty_termios_baud_rate);
@@ -361,7 +361,7 @@ speed_t tty_termios_input_baud_rate(struct ktermios *termios)
else
cbaud += 15;
}
- return baud_table[cbaud];
+ return cbaud >= n_baud_table ? 0 : baud_table[cbaud];
#else
return tty_termios_baud_rate(termios);
#endif
diff --git a/drivers/tty/tty_ldisc.c b/drivers/tty/tty_ldisc.c
index 3a9e2a2..4ab518d 100644
--- a/drivers/tty/tty_ldisc.c
+++ b/drivers/tty/tty_ldisc.c
@@ -175,12 +175,11 @@ static struct tty_ldisc *tty_ldisc_get(struct tty_struct *tty, int disc)
return ERR_CAST(ldops);
}
- ld = kmalloc(sizeof(struct tty_ldisc), GFP_KERNEL);
- if (ld == NULL) {
- put_ldops(ldops);
- return ERR_PTR(-ENOMEM);
- }
-
+ /*
+ * There is no way to handle allocation failure of only 16 bytes.
+ * Let's simplify error handling and save more memory.
+ */
+ ld = kmalloc(sizeof(struct tty_ldisc), GFP_KERNEL | __GFP_NOFAIL);
ld->ops = ldops;
ld->tty = tty;
@@ -753,12 +752,13 @@ void tty_ldisc_release(struct tty_struct *tty)
* the tty structure is not completely set up when this call is made.
*/
-void tty_ldisc_init(struct tty_struct *tty)
+int tty_ldisc_init(struct tty_struct *tty)
{
struct tty_ldisc *ld = tty_ldisc_get(tty, N_TTY);
if (IS_ERR(ld))
- panic("n_tty: init_tty");
+ return PTR_ERR(ld);
tty->ldisc = ld;
+ return 0;
}
/**
diff --git a/drivers/tty/tty_port.c b/drivers/tty/tty_port.c
index c3f9d93..7f2f20b 100644
--- a/drivers/tty/tty_port.c
+++ b/drivers/tty/tty_port.c
@@ -531,7 +531,8 @@ void tty_port_close(struct tty_port *port, struct tty_struct *tty,
if (tty_port_close_start(port, tty, filp) == 0)
return;
tty_port_shutdown(port, tty);
- set_bit(TTY_IO_ERROR, &tty->flags);
+ if (!port->console)
+ set_bit(TTY_IO_ERROR, &tty->flags);
tty_port_close_end(port, tty);
tty_port_tty_set(port, NULL);
}
diff --git a/drivers/tty/vt/vt.c b/drivers/tty/vt/vt.c
index 9e1ac58..9d3e413 100644
--- a/drivers/tty/vt/vt.c
+++ b/drivers/tty/vt/vt.c
@@ -785,7 +785,7 @@ int vc_allocate(unsigned int currcons) /* return 0 on success */
if (!*vc->vc_uni_pagedir_loc)
con_set_default_unimap(vc);
- vc->vc_screenbuf = kmalloc(vc->vc_screenbuf_size, GFP_KERNEL);
+ vc->vc_screenbuf = kzalloc(vc->vc_screenbuf_size, GFP_KERNEL);
if (!vc->vc_screenbuf)
goto err_free;
@@ -872,7 +872,7 @@ static int vc_do_resize(struct tty_struct *tty, struct vc_data *vc,
if (new_screen_size > (4 << 20))
return -EINVAL;
- newscreen = kmalloc(new_screen_size, GFP_USER);
+ newscreen = kzalloc(new_screen_size, GFP_USER);
if (!newscreen)
return -ENOMEM;
diff --git a/drivers/tty/vt/vt_ioctl.c b/drivers/tty/vt/vt_ioctl.c
index f62c598..638eb9b 100644
--- a/drivers/tty/vt/vt_ioctl.c
+++ b/drivers/tty/vt/vt_ioctl.c
@@ -31,6 +31,8 @@
#include <asm/io.h>
#include <asm/uaccess.h>
+#include <linux/nospec.h>
+
#include <linux/kbd_kern.h>
#include <linux/vt_kern.h>
#include <linux/kbd_diacr.h>
@@ -703,6 +705,8 @@ int vt_ioctl(struct tty_struct *tty,
if (vsa.console == 0 || vsa.console > MAX_NR_CONSOLES)
ret = -ENXIO;
else {
+ vsa.console = array_index_nospec(vsa.console,
+ MAX_NR_CONSOLES + 1);
vsa.console--;
console_lock();
ret = vc_allocate(vsa.console);
diff --git a/drivers/uio/uio.c b/drivers/uio/uio.c
index 208bc52..e6b2071 100644
--- a/drivers/uio/uio.c
+++ b/drivers/uio/uio.c
@@ -249,6 +249,8 @@ static struct class uio_class = {
.dev_groups = uio_groups,
};
+bool uio_class_registered;
+
/*
* device functions
*/
@@ -780,6 +782,9 @@ static int init_uio_class(void)
printk(KERN_ERR "class_register failed for uio\n");
goto err_class_register;
}
+
+ uio_class_registered = true;
+
return 0;
err_class_register:
@@ -790,6 +795,7 @@ static int init_uio_class(void)
static void release_uio_class(void)
{
+ uio_class_registered = false;
class_unregister(&uio_class);
uio_major_cleanup();
}
@@ -809,6 +815,9 @@ int __uio_register_device(struct module *owner,
struct uio_device *idev;
int ret = 0;
+ if (!uio_class_registered)
+ return -EPROBE_DEFER;
+
if (!parent || !info || !info->name || !info->version)
return -EINVAL;
@@ -854,8 +863,10 @@ int __uio_register_device(struct module *owner,
*/
ret = request_irq(info->irq, uio_interrupt,
info->irq_flags, info->name, idev);
- if (ret)
+ if (ret) {
+ info->uio_dev = NULL;
goto err_request_irq;
+ }
}
return 0;
diff --git a/drivers/usb/Kconfig b/drivers/usb/Kconfig
index e0ed5d8..d0644bf 100644
--- a/drivers/usb/Kconfig
+++ b/drivers/usb/Kconfig
@@ -160,6 +160,8 @@
source "drivers/usb/gadget/Kconfig"
+source "drivers/usb/typec/Kconfig"
+
config USB_LED_TRIG
bool "USB LED Triggers"
depends on LEDS_CLASS && LEDS_TRIGGERS
diff --git a/drivers/usb/Makefile b/drivers/usb/Makefile
index 7a11362..e2d07c0 100644
--- a/drivers/usb/Makefile
+++ b/drivers/usb/Makefile
@@ -63,3 +63,5 @@
obj-$(CONFIG_USBIP_CORE) += usbip/
obj-$(CONFIG_USB_PD) += pd/
+
+obj-$(CONFIG_TYPEC) += typec/
diff --git a/drivers/usb/chipidea/otg.h b/drivers/usb/chipidea/otg.h
index 9ecb598..a5557c7 100644
--- a/drivers/usb/chipidea/otg.h
+++ b/drivers/usb/chipidea/otg.h
@@ -20,7 +20,8 @@ void ci_handle_vbus_change(struct ci_hdrc *ci);
static inline void ci_otg_queue_work(struct ci_hdrc *ci)
{
disable_irq_nosync(ci->irq);
- queue_work(ci->wq, &ci->work);
+ if (queue_work(ci->wq, &ci->work) == false)
+ enable_irq(ci->irq);
}
#endif /* __DRIVERS_USB_CHIPIDEA_OTG_H */
diff --git a/drivers/usb/class/cdc-acm.c b/drivers/usb/class/cdc-acm.c
index 34d23cc..6c0bb38 100644
--- a/drivers/usb/class/cdc-acm.c
+++ b/drivers/usb/class/cdc-acm.c
@@ -174,6 +174,7 @@ static int acm_wb_alloc(struct acm *acm)
wb = &acm->wb[wbn];
if (!wb->use) {
wb->use = 1;
+ wb->len = 0;
return wbn;
}
wbn = (wbn + 1) % ACM_NW;
@@ -332,17 +333,17 @@ static void acm_ctrl_irq(struct urb *urb)
if (difference & ACM_CTRL_DSR)
acm->iocount.dsr++;
- if (difference & ACM_CTRL_BRK)
- acm->iocount.brk++;
- if (difference & ACM_CTRL_RI)
- acm->iocount.rng++;
if (difference & ACM_CTRL_DCD)
acm->iocount.dcd++;
- if (difference & ACM_CTRL_FRAMING)
+ if (newctrl & ACM_CTRL_BRK)
+ acm->iocount.brk++;
+ if (newctrl & ACM_CTRL_RI)
+ acm->iocount.rng++;
+ if (newctrl & ACM_CTRL_FRAMING)
acm->iocount.frame++;
- if (difference & ACM_CTRL_PARITY)
+ if (newctrl & ACM_CTRL_PARITY)
acm->iocount.parity++;
- if (difference & ACM_CTRL_OVERRUN)
+ if (newctrl & ACM_CTRL_OVERRUN)
acm->iocount.overrun++;
spin_unlock(&acm->read_lock);
@@ -501,6 +502,13 @@ static int acm_tty_install(struct tty_driver *driver, struct tty_struct *tty)
if (retval)
goto error_init_termios;
+ /*
+ * Suppress initial echoing for some devices which might send data
+ * immediately after acm driver has been installed.
+ */
+ if (acm->quirks & DISABLE_ECHO)
+ tty->termios.c_lflag &= ~ECHO;
+
tty->driver_data = acm;
return 0;
@@ -704,20 +712,9 @@ static int acm_tty_write(struct tty_struct *tty,
}
if (acm->susp_count) {
- if (acm->putbuffer) {
- /* now to preserve order */
- usb_anchor_urb(acm->putbuffer->urb, &acm->delayed);
- acm->putbuffer = NULL;
- }
usb_anchor_urb(wb->urb, &acm->delayed);
spin_unlock_irqrestore(&acm->write_lock, flags);
return count;
- } else {
- if (acm->putbuffer) {
- /* at this point there is no good way to handle errors */
- acm_start_wb(acm, acm->putbuffer);
- acm->putbuffer = NULL;
- }
}
stat = acm_start_wb(acm, wb);
@@ -728,64 +725,6 @@ static int acm_tty_write(struct tty_struct *tty,
return count;
}
-static void acm_tty_flush_chars(struct tty_struct *tty)
-{
- struct acm *acm = tty->driver_data;
- struct acm_wb *cur = acm->putbuffer;
- int err;
- unsigned long flags;
-
- if (!cur) /* nothing to do */
- return;
-
- acm->putbuffer = NULL;
- err = usb_autopm_get_interface_async(acm->control);
- spin_lock_irqsave(&acm->write_lock, flags);
- if (err < 0) {
- cur->use = 0;
- acm->putbuffer = cur;
- goto out;
- }
-
- if (acm->susp_count)
- usb_anchor_urb(cur->urb, &acm->delayed);
- else
- acm_start_wb(acm, cur);
-out:
- spin_unlock_irqrestore(&acm->write_lock, flags);
- return;
-}
-
-static int acm_tty_put_char(struct tty_struct *tty, unsigned char ch)
-{
- struct acm *acm = tty->driver_data;
- struct acm_wb *cur;
- int wbn;
- unsigned long flags;
-
-overflow:
- cur = acm->putbuffer;
- if (!cur) {
- spin_lock_irqsave(&acm->write_lock, flags);
- wbn = acm_wb_alloc(acm);
- if (wbn >= 0) {
- cur = &acm->wb[wbn];
- acm->putbuffer = cur;
- }
- spin_unlock_irqrestore(&acm->write_lock, flags);
- if (!cur)
- return 0;
- }
-
- if (cur->len == acm->writesize) {
- acm_tty_flush_chars(tty);
- goto overflow;
- }
-
- cur->buf[cur->len++] = ch;
- return 1;
-}
-
static int acm_tty_write_room(struct tty_struct *tty)
{
struct acm *acm = tty->driver_data;
@@ -1688,6 +1627,9 @@ static const struct usb_device_id acm_ids[] = {
{ USB_DEVICE(0x0e8d, 0x0003), /* FIREFLY, MediaTek Inc; andrey.arapov@gmail.com */
.driver_info = NO_UNION_NORMAL, /* has no union descriptor */
},
+ { USB_DEVICE(0x0e8d, 0x2000), /* MediaTek Inc Preloader */
+ .driver_info = DISABLE_ECHO, /* DISABLE ECHO in termios flag */
+ },
{ USB_DEVICE(0x0e8d, 0x3329), /* MediaTek Inc GPS */
.driver_info = NO_UNION_NORMAL, /* has no union descriptor */
},
@@ -1709,6 +1651,9 @@ static const struct usb_device_id acm_ids[] = {
{ USB_DEVICE(0x11ca, 0x0201), /* VeriFone Mx870 Gadget Serial */
.driver_info = SINGLE_RX_URB,
},
+ { USB_DEVICE(0x1965, 0x0018), /* Uniden UBC125XLT */
+ .driver_info = NO_UNION_NORMAL, /* has no union descriptor */
+ },
{ USB_DEVICE(0x22b8, 0x7000), /* Motorola Q Phone */
.driver_info = NO_UNION_NORMAL, /* has no union descriptor */
},
@@ -1724,6 +1669,9 @@ static const struct usb_device_id acm_ids[] = {
{ USB_DEVICE(0x0572, 0x1328), /* Shiro / Aztech USB MODEM UM-3100 */
.driver_info = NO_UNION_NORMAL, /* has no union descriptor */
},
+ { USB_DEVICE(0x0572, 0x1349), /* Hiro (Conexant) USB MODEM H50228 */
+ .driver_info = NO_UNION_NORMAL, /* has no union descriptor */
+ },
{ USB_DEVICE(0x20df, 0x0001), /* Simtec Electronics Entropy Key */
.driver_info = QUIRK_CONTROL_LINE_STATE, },
{ USB_DEVICE(0x2184, 0x001c) }, /* GW Instek AFG-2225 */
@@ -1779,6 +1727,9 @@ static const struct usb_device_id acm_ids[] = {
{ USB_DEVICE(0x09d8, 0x0320), /* Elatec GmbH TWN3 */
.driver_info = NO_UNION_NORMAL, /* has misplaced union descriptor */
},
+ { USB_DEVICE(0x0ca6, 0xa050), /* Castles VEGA3000 */
+ .driver_info = NO_UNION_NORMAL, /* reports zero length descriptor */
+ },
{ USB_DEVICE(0x2912, 0x0001), /* ATOL FPrint */
.driver_info = CLEAR_HALT_CONDITIONS,
@@ -1931,8 +1882,6 @@ static const struct tty_operations acm_ops = {
.cleanup = acm_tty_cleanup,
.hangup = acm_tty_hangup,
.write = acm_tty_write,
- .put_char = acm_tty_put_char,
- .flush_chars = acm_tty_flush_chars,
.write_room = acm_tty_write_room,
.ioctl = acm_tty_ioctl,
.throttle = acm_tty_throttle,
diff --git a/drivers/usb/class/cdc-acm.h b/drivers/usb/class/cdc-acm.h
index 1f1eabf..1ad9ff9 100644
--- a/drivers/usb/class/cdc-acm.h
+++ b/drivers/usb/class/cdc-acm.h
@@ -94,7 +94,6 @@ struct acm {
unsigned long read_urbs_free;
struct urb *read_urbs[ACM_NR];
struct acm_rb read_buffers[ACM_NR];
- struct acm_wb *putbuffer; /* for acm_tty_put_char() */
int rx_buflimit;
spinlock_t read_lock;
int write_used; /* number of non-empty write buffers */
@@ -135,3 +134,4 @@ struct acm {
#define QUIRK_CONTROL_LINE_STATE BIT(6)
#define CLEAR_HALT_CONDITIONS BIT(7)
#define SEND_ZERO_PACKET BIT(8)
+#define DISABLE_ECHO BIT(9)
diff --git a/drivers/usb/core/config.c b/drivers/usb/core/config.c
index 7cc04ff..f794741 100644
--- a/drivers/usb/core/config.c
+++ b/drivers/usb/core/config.c
@@ -186,7 +186,9 @@ static const unsigned short full_speed_maxpacket_maxes[4] = {
static const unsigned short high_speed_maxpacket_maxes[4] = {
[USB_ENDPOINT_XFER_CONTROL] = 64,
[USB_ENDPOINT_XFER_ISOC] = 1024,
- [USB_ENDPOINT_XFER_BULK] = 512,
+
+ /* Bulk should be 512, but some devices use 1024: we will warn below */
+ [USB_ENDPOINT_XFER_BULK] = 1024,
[USB_ENDPOINT_XFER_INT] = 1024,
};
static const unsigned short super_speed_maxpacket_maxes[4] = {
diff --git a/drivers/usb/core/devio.c b/drivers/usb/core/devio.c
index 893ebae5..52fc208 100644
--- a/drivers/usb/core/devio.c
+++ b/drivers/usb/core/devio.c
@@ -1450,10 +1450,13 @@ static int proc_do_submiturb(struct usb_dev_state *ps, struct usbdevfs_urb *uurb
struct async *as = NULL;
struct usb_ctrlrequest *dr = NULL;
unsigned int u, totlen, isofrmlen;
- int i, ret, is_in, num_sgs = 0, ifnum = -1;
+ int i, ret, num_sgs = 0, ifnum = -1;
int number_of_packets = 0;
unsigned int stream_id = 0;
void *buf;
+ bool is_in;
+ bool allow_short = false;
+ bool allow_zero = false;
unsigned long mask = USBDEVFS_URB_SHORT_NOT_OK |
USBDEVFS_URB_BULK_CONTINUATION |
USBDEVFS_URB_NO_FSBR |
@@ -1516,6 +1519,8 @@ static int proc_do_submiturb(struct usb_dev_state *ps, struct usbdevfs_urb *uurb
is_in = 0;
uurb->endpoint &= ~USB_DIR_IN;
}
+ if (is_in)
+ allow_short = true;
snoop(&ps->dev->dev, "control urb: bRequestType=%02x "
"bRequest=%02x wValue=%04x "
"wIndex=%04x wLength=%04x\n",
@@ -1527,6 +1532,10 @@ static int proc_do_submiturb(struct usb_dev_state *ps, struct usbdevfs_urb *uurb
break;
case USBDEVFS_URB_TYPE_BULK:
+ if (!is_in)
+ allow_zero = true;
+ else
+ allow_short = true;
switch (usb_endpoint_type(&ep->desc)) {
case USB_ENDPOINT_XFER_CONTROL:
case USB_ENDPOINT_XFER_ISOC:
@@ -1547,6 +1556,10 @@ static int proc_do_submiturb(struct usb_dev_state *ps, struct usbdevfs_urb *uurb
if (!usb_endpoint_xfer_int(&ep->desc))
return -EINVAL;
interrupt_urb:
+ if (!is_in)
+ allow_zero = true;
+ else
+ allow_short = true;
break;
case USBDEVFS_URB_TYPE_ISO:
@@ -1691,16 +1704,21 @@ static int proc_do_submiturb(struct usb_dev_state *ps, struct usbdevfs_urb *uurb
u = (is_in ? URB_DIR_IN : URB_DIR_OUT);
if (uurb->flags & USBDEVFS_URB_ISO_ASAP)
u |= URB_ISO_ASAP;
- if (uurb->flags & USBDEVFS_URB_SHORT_NOT_OK && is_in)
+ if (allow_short && uurb->flags & USBDEVFS_URB_SHORT_NOT_OK)
u |= URB_SHORT_NOT_OK;
if (uurb->flags & USBDEVFS_URB_NO_FSBR)
u |= URB_NO_FSBR;
- if (uurb->flags & USBDEVFS_URB_ZERO_PACKET)
+ if (allow_zero && uurb->flags & USBDEVFS_URB_ZERO_PACKET)
u |= URB_ZERO_PACKET;
if (uurb->flags & USBDEVFS_URB_NO_INTERRUPT)
u |= URB_NO_INTERRUPT;
as->urb->transfer_flags = u;
+ if (!allow_short && uurb->flags & USBDEVFS_URB_SHORT_NOT_OK)
+ dev_warn(&ps->dev->dev, "Requested nonsensical USBDEVFS_URB_SHORT_NOT_OK.\n");
+ if (!allow_zero && uurb->flags & USBDEVFS_URB_ZERO_PACKET)
+ dev_warn(&ps->dev->dev, "Requested nonsensical USBDEVFS_URB_ZERO_PACKET.\n");
+
as->urb->transfer_buffer_length = uurb->buffer_length;
as->urb->setup_packet = (unsigned char *)dr;
dr = NULL;
diff --git a/drivers/usb/core/driver.c b/drivers/usb/core/driver.c
index 5532246..4ec1317 100644
--- a/drivers/usb/core/driver.c
+++ b/drivers/usb/core/driver.c
@@ -509,7 +509,6 @@ int usb_driver_claim_interface(struct usb_driver *driver,
struct device *dev;
struct usb_device *udev;
int retval = 0;
- int lpm_disable_error = -ENODEV;
if (!iface)
return -ENODEV;
@@ -530,16 +529,6 @@ int usb_driver_claim_interface(struct usb_driver *driver,
iface->condition = USB_INTERFACE_BOUND;
- /* See the comment about disabling LPM in usb_probe_interface(). */
- if (driver->disable_hub_initiated_lpm) {
- lpm_disable_error = usb_unlocked_disable_lpm(udev);
- if (lpm_disable_error) {
- dev_err(&iface->dev, "%s Failed to disable LPM for driver %s\n.",
- __func__, driver->name);
- return -ENOMEM;
- }
- }
-
/* Claimed interfaces are initially inactive (suspended) and
* runtime-PM-enabled, but only if the driver has autosuspend
* support. Otherwise they are marked active, to prevent the
@@ -558,9 +547,20 @@ int usb_driver_claim_interface(struct usb_driver *driver,
if (device_is_registered(dev))
retval = device_bind_driver(dev);
- /* Attempt to re-enable USB3 LPM, if the disable was successful. */
- if (!lpm_disable_error)
- usb_unlocked_enable_lpm(udev);
+ if (retval) {
+ dev->driver = NULL;
+ usb_set_intfdata(iface, NULL);
+ iface->needs_remote_wakeup = 0;
+ iface->condition = USB_INTERFACE_UNBOUND;
+
+ /*
+ * Unbound interfaces are always runtime-PM-disabled
+ * and runtime-PM-suspended
+ */
+ if (driver->supports_autosuspend)
+ pm_runtime_disable(dev);
+ pm_runtime_set_suspended(dev);
+ }
return retval;
}
@@ -1924,4 +1924,5 @@ struct bus_type usb_bus_type = {
.name = "usb",
.match = usb_device_match,
.uevent = usb_uevent,
+ .need_parent_lock = true,
};
diff --git a/drivers/usb/core/hcd-pci.c b/drivers/usb/core/hcd-pci.c
index 7859d73..7af23b2 100644
--- a/drivers/usb/core/hcd-pci.c
+++ b/drivers/usb/core/hcd-pci.c
@@ -528,8 +528,6 @@ static int resume_common(struct device *dev, int event)
event == PM_EVENT_RESTORE);
if (retval) {
dev_err(dev, "PCI post-resume error %d!\n", retval);
- if (hcd->shared_hcd)
- usb_hc_died(hcd->shared_hcd);
usb_hc_died(hcd);
}
}
diff --git a/drivers/usb/core/hcd.c b/drivers/usb/core/hcd.c
index e4b39a7..6a4ea98 100644
--- a/drivers/usb/core/hcd.c
+++ b/drivers/usb/core/hcd.c
@@ -2430,6 +2430,7 @@ void usb_hcd_resume_root_hub (struct usb_hcd *hcd)
spin_lock_irqsave (&hcd_root_hub_lock, flags);
if (hcd->rh_registered) {
+ pm_wakeup_event(&hcd->self.root_hub->dev, 0);
set_bit(HCD_FLAG_WAKEUP_PENDING, &hcd->flags);
queue_work(pm_wq, &hcd->wakeup_work);
}
diff --git a/drivers/usb/core/hub.c b/drivers/usb/core/hub.c
index a9117ee..fdcea607 100644
--- a/drivers/usb/core/hub.c
+++ b/drivers/usb/core/hub.c
@@ -659,12 +659,17 @@ void usb_wakeup_notification(struct usb_device *hdev,
unsigned int portnum)
{
struct usb_hub *hub;
+ struct usb_port *port_dev;
if (!hdev)
return;
hub = usb_hub_to_struct_hub(hdev);
if (hub) {
+ port_dev = hub->ports[portnum - 1];
+ if (port_dev && port_dev->child)
+ pm_wakeup_event(&port_dev->child->dev, 0);
+
set_bit(portnum, hub->wakeup_bits);
kick_hub_wq(hub);
}
@@ -1145,10 +1150,14 @@ static void hub_activate(struct usb_hub *hub, enum hub_activation_type type)
if (!udev || udev->state == USB_STATE_NOTATTACHED) {
/* Tell hub_wq to disconnect the device or
- * check for a new connection
+ * check for a new connection or over current condition.
+ * Based on USB2.0 Spec Section 11.12.5,
+ * C_PORT_OVER_CURRENT could be set while
+ * PORT_OVER_CURRENT is not. So check for any of them.
*/
if (udev || (portstatus & USB_PORT_STAT_CONNECTION) ||
- (portstatus & USB_PORT_STAT_OVERCURRENT))
+ (portstatus & USB_PORT_STAT_OVERCURRENT) ||
+ (portchange & USB_PORT_STAT_C_OVERCURRENT))
set_bit(port1, hub->change_bits);
} else if (portstatus & USB_PORT_STAT_ENABLE) {
@@ -1681,47 +1690,6 @@ static int hub_probe(struct usb_interface *intf, const struct usb_device_id *id)
hdev = interface_to_usbdev(intf);
/*
- * Set default autosuspend delay as 0 to speedup bus suspend,
- * based on the below considerations:
- *
- * - Unlike other drivers, the hub driver does not rely on the
- * autosuspend delay to provide enough time to handle a wakeup
- * event, and the submitted status URB is just to check future
- * change on hub downstream ports, so it is safe to do it.
- *
- * - The patch might cause one or more auto supend/resume for
- * below very rare devices when they are plugged into hub
- * first time:
- *
- * devices having trouble initializing, and disconnect
- * themselves from the bus and then reconnect a second
- * or so later
- *
- * devices just for downloading firmware, and disconnects
- * themselves after completing it
- *
- * For these quite rare devices, their drivers may change the
- * autosuspend delay of their parent hub in the probe() to one
- * appropriate value to avoid the subtle problem if someone
- * does care it.
- *
- * - The patch may cause one or more auto suspend/resume on
- * hub during running 'lsusb', but it is probably too
- * infrequent to worry about.
- *
- * - Change autosuspend delay of hub can avoid unnecessary auto
- * suspend timer for hub, also may decrease power consumption
- * of USB bus.
- *
- * - If user has indicated to prevent autosuspend by passing
- * usbcore.autosuspend = -1 then keep autosuspend disabled.
- */
-#ifdef CONFIG_PM
- if (hdev->dev.power.autosuspend_delay >= 0)
- pm_runtime_set_autosuspend_delay(&hdev->dev, 0);
-#endif
-
- /*
* Hubs have proper suspend/resume support, except for root hubs
* where the controller driver doesn't have bus_suspend and
* bus_resume methods.
@@ -2227,7 +2195,7 @@ static int usb_enumerate_device_otg(struct usb_device *udev)
/* descriptor may appear anywhere in config */
err = __usb_get_extra_descriptor(udev->rawdescriptors[0],
le16_to_cpu(udev->config[0].desc.wTotalLength),
- USB_DT_OTG, (void **) &desc);
+ USB_DT_OTG, (void **) &desc, sizeof(*desc));
if (err || !(desc->bmAttributes & USB_OTG_HNP))
return 0;
@@ -2811,7 +2779,9 @@ static int hub_port_reset(struct usb_hub *hub, int port1,
USB_PORT_FEAT_C_BH_PORT_RESET);
usb_clear_port_feature(hub->hdev, port1,
USB_PORT_FEAT_C_PORT_LINK_STATE);
- usb_clear_port_feature(hub->hdev, port1,
+
+ if (udev)
+ usb_clear_port_feature(hub->hdev, port1,
USB_PORT_FEAT_C_CONNECTION);
/*
@@ -3365,6 +3335,10 @@ static int wait_for_connected(struct usb_device *udev,
while (delay_ms < 2000) {
if (status || *portstatus & USB_PORT_STAT_CONNECTION)
break;
+ if (!port_is_power_on(hub, *portstatus)) {
+ status = -ENODEV;
+ break;
+ }
msleep(20);
delay_ms += 20;
status = hub_port_status(hub, *port1, portstatus, portchange);
@@ -3428,8 +3402,11 @@ int usb_port_resume(struct usb_device *udev, pm_message_t msg)
/* Skip the initial Clear-Suspend step for a remote wakeup */
status = hub_port_status(hub, port1, &portstatus, &portchange);
- if (status == 0 && !port_is_suspended(hub, portstatus))
+ if (status == 0 && !port_is_suspended(hub, portstatus)) {
+ if (portchange & USB_PORT_STAT_C_SUSPEND)
+ pm_wakeup_event(&udev->dev, 0);
goto SuspendCleared;
+ }
/* see 7.1.7.7; affects power usage, but not budgeting */
if (hub_is_superspeed(hub->hdev))
@@ -4514,7 +4491,9 @@ hub_port_init(struct usb_hub *hub, struct usb_device *udev, int port1,
* reset. But only on the first attempt,
* lest we get into a time out/reset loop
*/
- if (r == 0 || (r == -ETIMEDOUT && retries == 0))
+ if (r == 0 || (r == -ETIMEDOUT &&
+ retries == 0 &&
+ udev->speed > USB_SPEED_FULL))
break;
}
udev->descriptor.bMaxPacketSize0 =
diff --git a/drivers/usb/core/message.c b/drivers/usb/core/message.c
index 9016a9b..255fecc 100644
--- a/drivers/usb/core/message.c
+++ b/drivers/usb/core/message.c
@@ -1279,6 +1279,11 @@ void usb_enable_interface(struct usb_device *dev,
* is submitted that needs that bandwidth. Some other operating systems
* allocate bandwidth early, when a configuration is chosen.
*
+ * xHCI reserves bandwidth and configures the alternate setting in
+ * usb_hcd_alloc_bandwidth(). If it fails the original interface altsetting
+ * may be disabled. Drivers cannot rely on any particular alternate
+ * setting being in effect after a failure.
+ *
* This call is synchronous, and may not be used in an interrupt context.
* Also, drivers must not change altsettings while urbs are scheduled for
* endpoints in that interface; all such urbs must first be completed
@@ -1314,6 +1319,12 @@ int usb_set_interface(struct usb_device *dev, int interface, int alternate)
alternate);
return -EINVAL;
}
+ /*
+ * usb3 hosts configure the interface in usb_hcd_alloc_bandwidth,
+ * including freeing dropped endpoint ring buffers.
+ * Make sure the interface endpoints are flushed before that
+ */
+ usb_disable_interface(dev, iface, false);
/* Make sure we have enough bandwidth for this alternate interface.
* Remove the current alt setting and add the new alt setting.
diff --git a/drivers/usb/core/quirks.c b/drivers/usb/core/quirks.c
index 4f1c6f8..cf378b1 100644
--- a/drivers/usb/core/quirks.c
+++ b/drivers/usb/core/quirks.c
@@ -37,6 +37,10 @@ static const struct usb_device_id usb_quirk_list[] = {
/* CBM - Flash disk */
{ USB_DEVICE(0x0204, 0x6025), .driver_info = USB_QUIRK_RESET_RESUME },
+ /* WORLDE Controller KS49 or Prodipe MIDI 49C USB controller */
+ { USB_DEVICE(0x0218, 0x0201), .driver_info =
+ USB_QUIRK_CONFIG_INTF_STRINGS },
+
/* WORLDE easy key (easykey.25) MIDI controller */
{ USB_DEVICE(0x0218, 0x0401), .driver_info =
USB_QUIRK_CONFIG_INTF_STRINGS },
@@ -45,6 +49,9 @@ static const struct usb_device_id usb_quirk_list[] = {
{ USB_DEVICE(0x03f0, 0x0701), .driver_info =
USB_QUIRK_STRING_FETCH_255 },
+ /* HP v222w 16GB Mini USB Drive */
+ { USB_DEVICE(0x03f0, 0x3f40), .driver_info = USB_QUIRK_DELAY_INIT },
+
/* Creative SB Audigy 2 NX */
{ USB_DEVICE(0x041e, 0x3020), .driver_info = USB_QUIRK_RESET_RESUME },
@@ -57,6 +64,9 @@ static const struct usb_device_id usb_quirk_list[] = {
/* Microsoft LifeCam-VX700 v2.0 */
{ USB_DEVICE(0x045e, 0x0770), .driver_info = USB_QUIRK_RESET_RESUME },
+ /* Cherry Stream G230 2.0 (G85-231) and 3.0 (G85-232) */
+ { USB_DEVICE(0x046a, 0x0023), .driver_info = USB_QUIRK_RESET_RESUME },
+
/* Logitech HD Pro Webcams C920, C920-C, C925e and C930e */
{ USB_DEVICE(0x046d, 0x082d), .driver_info = USB_QUIRK_DELAY_INIT },
{ USB_DEVICE(0x046d, 0x0841), .driver_info = USB_QUIRK_DELAY_INIT },
@@ -178,6 +188,10 @@ static const struct usb_device_id usb_quirk_list[] = {
/* Midiman M-Audio Keystation 88es */
{ USB_DEVICE(0x0763, 0x0192), .driver_info = USB_QUIRK_RESET_RESUME },
+ /* SanDisk Ultra Fit and Ultra Flair */
+ { USB_DEVICE(0x0781, 0x5583), .driver_info = USB_QUIRK_NO_LPM },
+ { USB_DEVICE(0x0781, 0x5591), .driver_info = USB_QUIRK_NO_LPM },
+
/* M-Systems Flash Disk Pioneers */
{ USB_DEVICE(0x08ec, 0x1000), .driver_info = USB_QUIRK_RESET_RESUME },
@@ -228,10 +242,17 @@ static const struct usb_device_id usb_quirk_list[] = {
/* Corsair K70 RGB */
{ USB_DEVICE(0x1b1c, 0x1b13), .driver_info = USB_QUIRK_DELAY_INIT },
+ /* Corsair Strafe */
+ { USB_DEVICE(0x1b1c, 0x1b15), .driver_info = USB_QUIRK_DELAY_INIT |
+ USB_QUIRK_DELAY_CTRL_MSG },
+
/* Corsair Strafe RGB */
{ USB_DEVICE(0x1b1c, 0x1b20), .driver_info = USB_QUIRK_DELAY_INIT |
USB_QUIRK_DELAY_CTRL_MSG },
+ /* Corsair K70 LUX RGB */
+ { USB_DEVICE(0x1b1c, 0x1b33), .driver_info = USB_QUIRK_DELAY_INIT },
+
/* Corsair K70 LUX */
{ USB_DEVICE(0x1b1c, 0x1b36), .driver_info = USB_QUIRK_DELAY_INIT },
@@ -252,6 +273,14 @@ static const struct usb_device_id usb_quirk_list[] = {
{ USB_DEVICE(0x2040, 0x7200), .driver_info =
USB_QUIRK_CONFIG_INTF_STRINGS },
+ /* Raydium Touchscreen */
+ { USB_DEVICE(0x2386, 0x3114), .driver_info = USB_QUIRK_NO_LPM },
+
+ { USB_DEVICE(0x2386, 0x3119), .driver_info = USB_QUIRK_NO_LPM },
+
+ /* DJI CineSSD */
+ { USB_DEVICE(0x2ca3, 0x0031), .driver_info = USB_QUIRK_NO_LPM },
+
/* INTEL VALUE SSD */
{ USB_DEVICE(0x8086, 0xf1a5), .driver_info = USB_QUIRK_RESET_RESUME },
diff --git a/drivers/usb/core/usb.c b/drivers/usb/core/usb.c
index bb2a4fe..793c849 100644
--- a/drivers/usb/core/usb.c
+++ b/drivers/usb/core/usb.c
@@ -91,6 +91,8 @@ struct usb_host_interface *usb_find_alt_setting(
struct usb_interface_cache *intf_cache = NULL;
int i;
+ if (!config)
+ return NULL;
for (i = 0; i < config->desc.bNumInterfaces; i++) {
if (config->intf_cache[i]->altsetting[0].desc.bInterfaceNumber
== iface_num) {
@@ -747,14 +749,14 @@ EXPORT_SYMBOL(usb_stop_endpoint);
*/
int __usb_get_extra_descriptor(char *buffer, unsigned size,
- unsigned char type, void **ptr)
+ unsigned char type, void **ptr, size_t minsize)
{
struct usb_descriptor_header *header;
while (size >= sizeof(struct usb_descriptor_header)) {
header = (struct usb_descriptor_header *)buffer;
- if (header->bLength < 2) {
+ if (header->bLength < 2 || header->bLength > size) {
printk(KERN_ERR
"%s: bogus descriptor, type %d length %d\n",
usbcore_name,
@@ -763,7 +765,7 @@ int __usb_get_extra_descriptor(char *buffer, unsigned size,
return -1;
}
- if (header->bDescriptorType == type) {
+ if (header->bDescriptorType == type && header->bLength >= minsize) {
*ptr = header;
return 0;
}
diff --git a/drivers/usb/dwc2/core.h b/drivers/usb/dwc2/core.h
index 2a21a04..0f45a2f 100644
--- a/drivers/usb/dwc2/core.h
+++ b/drivers/usb/dwc2/core.h
@@ -209,7 +209,7 @@ struct dwc2_hsotg_ep {
unsigned char dir_in;
unsigned char index;
unsigned char mc;
- unsigned char interval;
+ u16 interval;
unsigned int halted:1;
unsigned int periodic:1;
diff --git a/drivers/usb/dwc2/gadget.c b/drivers/usb/dwc2/gadget.c
index cfdd5c3..3ae27b6 100644
--- a/drivers/usb/dwc2/gadget.c
+++ b/drivers/usb/dwc2/gadget.c
@@ -2642,12 +2642,6 @@ void dwc2_hsotg_core_init_disconnected(struct dwc2_hsotg *hsotg,
dwc2_writel(dwc2_hsotg_ep0_mps(hsotg->eps_out[0]->ep.maxpacket) |
DXEPCTL_USBACTEP, hsotg->regs + DIEPCTL0);
- dwc2_hsotg_enqueue_setup(hsotg);
-
- dev_dbg(hsotg->dev, "EP0: DIEPCTL0=0x%08x, DOEPCTL0=0x%08x\n",
- dwc2_readl(hsotg->regs + DIEPCTL0),
- dwc2_readl(hsotg->regs + DOEPCTL0));
-
/* clear global NAKs */
val = DCTL_CGOUTNAK | DCTL_CGNPINNAK;
if (!is_usb_reset)
@@ -2658,6 +2652,12 @@ void dwc2_hsotg_core_init_disconnected(struct dwc2_hsotg *hsotg,
mdelay(3);
hsotg->lx_state = DWC2_L0;
+
+ dwc2_hsotg_enqueue_setup(hsotg);
+
+ dev_dbg(hsotg->dev, "EP0: DIEPCTL0=0x%08x, DOEPCTL0=0x%08x\n",
+ dwc2_readl(hsotg->regs + DIEPCTL0),
+ dwc2_readl(hsotg->regs + DOEPCTL0));
}
static void dwc2_hsotg_core_disconnect(struct dwc2_hsotg *hsotg)
@@ -3948,9 +3948,11 @@ int dwc2_gadget_init(struct dwc2_hsotg *hsotg, int irq)
}
ret = usb_add_gadget_udc(dev, &hsotg->gadget);
- if (ret)
+ if (ret) {
+ dwc2_hsotg_ep_free_request(&hsotg->eps_out[0]->ep,
+ hsotg->ctrl_req);
return ret;
-
+ }
dwc2_hsotg_dump(hsotg);
return 0;
@@ -3963,6 +3965,7 @@ int dwc2_gadget_init(struct dwc2_hsotg *hsotg, int irq)
int dwc2_hsotg_remove(struct dwc2_hsotg *hsotg)
{
usb_del_gadget_udc(&hsotg->gadget);
+ dwc2_hsotg_ep_free_request(&hsotg->eps_out[0]->ep, hsotg->ctrl_req);
return 0;
}
diff --git a/drivers/usb/dwc2/hcd.c b/drivers/usb/dwc2/hcd.c
index 919a321..984d6aa 100644
--- a/drivers/usb/dwc2/hcd.c
+++ b/drivers/usb/dwc2/hcd.c
@@ -2268,10 +2268,22 @@ static int dwc2_core_init(struct dwc2_hsotg *hsotg, bool initial_setup)
*/
static void dwc2_core_host_init(struct dwc2_hsotg *hsotg)
{
- u32 hcfg, hfir, otgctl;
+ u32 hcfg, hfir, otgctl, usbcfg;
dev_dbg(hsotg->dev, "%s(%p)\n", __func__, hsotg);
+ /* Set HS/FS Timeout Calibration to 7 (max available value).
+ * The number of PHY clocks that the application programs in
+ * this field is added to the high/full speed interpacket timeout
+ * duration in the core to account for any additional delays
+ * introduced by the PHY. This can be required, because the delay
+ * introduced by the PHY in generating the linestate condition
+ * can vary from one PHY to another.
+ */
+ usbcfg = dwc2_readl(hsotg->regs + GUSBCFG);
+ usbcfg |= GUSBCFG_TOUTCAL(7);
+ dwc2_writel(usbcfg, hsotg->regs + GUSBCFG);
+
/* Restart the Phy Clock */
dwc2_writel(0, hsotg->regs + PCGCTL);
@@ -2532,34 +2544,29 @@ static void dwc2_hc_init_xfer(struct dwc2_hsotg *hsotg,
#define DWC2_USB_DMA_ALIGN 4
-struct dma_aligned_buffer {
- void *kmalloc_ptr;
- void *old_xfer_buffer;
- u8 data[0];
-};
-
static void dwc2_free_dma_aligned_buffer(struct urb *urb)
{
- struct dma_aligned_buffer *temp;
+ void *stored_xfer_buffer;
if (!(urb->transfer_flags & URB_ALIGNED_TEMP_BUFFER))
return;
- temp = container_of(urb->transfer_buffer,
- struct dma_aligned_buffer, data);
+ /* Restore urb->transfer_buffer from the end of the allocated area */
+ memcpy(&stored_xfer_buffer, urb->transfer_buffer +
+ urb->transfer_buffer_length, sizeof(urb->transfer_buffer));
if (usb_urb_dir_in(urb))
- memcpy(temp->old_xfer_buffer, temp->data,
+ memcpy(stored_xfer_buffer, urb->transfer_buffer,
urb->transfer_buffer_length);
- urb->transfer_buffer = temp->old_xfer_buffer;
- kfree(temp->kmalloc_ptr);
+ kfree(urb->transfer_buffer);
+ urb->transfer_buffer = stored_xfer_buffer;
urb->transfer_flags &= ~URB_ALIGNED_TEMP_BUFFER;
}
static int dwc2_alloc_dma_aligned_buffer(struct urb *urb, gfp_t mem_flags)
{
- struct dma_aligned_buffer *temp, *kmalloc_ptr;
+ void *kmalloc_ptr;
size_t kmalloc_size;
if (urb->num_sgs || urb->sg ||
@@ -2567,22 +2574,29 @@ static int dwc2_alloc_dma_aligned_buffer(struct urb *urb, gfp_t mem_flags)
!((uintptr_t)urb->transfer_buffer & (DWC2_USB_DMA_ALIGN - 1)))
return 0;
- /* Allocate a buffer with enough padding for alignment */
+ /*
+ * Allocate a buffer with enough padding for original transfer_buffer
+ * pointer. This allocation is guaranteed to be aligned properly for
+ * DMA
+ */
kmalloc_size = urb->transfer_buffer_length +
- sizeof(struct dma_aligned_buffer) + DWC2_USB_DMA_ALIGN - 1;
+ sizeof(urb->transfer_buffer);
kmalloc_ptr = kmalloc(kmalloc_size, mem_flags);
if (!kmalloc_ptr)
return -ENOMEM;
- /* Position our struct dma_aligned_buffer such that data is aligned */
- temp = PTR_ALIGN(kmalloc_ptr + 1, DWC2_USB_DMA_ALIGN) - 1;
- temp->kmalloc_ptr = kmalloc_ptr;
- temp->old_xfer_buffer = urb->transfer_buffer;
+ /*
+ * Position value of original urb->transfer_buffer pointer to the end
+ * of allocation for later referencing
+ */
+ memcpy(kmalloc_ptr + urb->transfer_buffer_length,
+ &urb->transfer_buffer, sizeof(urb->transfer_buffer));
+
if (usb_urb_dir_out(urb))
- memcpy(temp->data, urb->transfer_buffer,
+ memcpy(kmalloc_ptr, urb->transfer_buffer,
urb->transfer_buffer_length);
- urb->transfer_buffer = temp->data;
+ urb->transfer_buffer = kmalloc_ptr;
urb->transfer_flags |= URB_ALIGNED_TEMP_BUFFER;
diff --git a/drivers/usb/dwc2/hcd_intr.c b/drivers/usb/dwc2/hcd_intr.c
index 906f223..8066fa9 100644
--- a/drivers/usb/dwc2/hcd_intr.c
+++ b/drivers/usb/dwc2/hcd_intr.c
@@ -922,9 +922,8 @@ static int dwc2_xfercomp_isoc_split_in(struct dwc2_hsotg *hsotg,
frame_desc = &qtd->urb->iso_descs[qtd->isoc_frame_index];
len = dwc2_get_actual_xfer_length(hsotg, chan, chnum, qtd,
DWC2_HC_XFER_COMPLETE, NULL);
- if (!len) {
+ if (!len && !qtd->isoc_split_offset) {
qtd->complete_split = 0;
- qtd->isoc_split_offset = 0;
return 0;
}
diff --git a/drivers/usb/dwc2/hcd_queue.c b/drivers/usb/dwc2/hcd_queue.c
index 1375435..9669184 100644
--- a/drivers/usb/dwc2/hcd_queue.c
+++ b/drivers/usb/dwc2/hcd_queue.c
@@ -479,7 +479,7 @@ static unsigned long *dwc2_get_ls_map(struct dwc2_hsotg *hsotg,
/* Get the map and adjust if this is a multi_tt hub */
map = qh->dwc_tt->periodic_bitmaps;
if (qh->dwc_tt->usb_tt->multi)
- map += DWC2_ELEMENTS_PER_LS_BITMAP * qh->ttport;
+ map += DWC2_ELEMENTS_PER_LS_BITMAP * (qh->ttport - 1);
return map;
}
diff --git a/drivers/usb/dwc3/core.h b/drivers/usb/dwc3/core.h
index 09e353b..dc8b4e3 100644
--- a/drivers/usb/dwc3/core.h
+++ b/drivers/usb/dwc3/core.h
@@ -269,6 +269,8 @@
#define DWC3_GUSB3PIPECTL_ELASTIC_BUF_MODE (1 << 0)
/* Global TX Fifo Size Register */
+#define DWC31_GTXFIFOSIZ_TXFRAMNUM BIT(15) /* DWC_usb31 only */
+#define DWC31_GTXFIFOSIZ_TXFDEF(n) ((n) & 0x7fff) /* DWC_usb31 only */
#define DWC3_GTXFIFOSIZ_TXFDEF(n) ((n) & 0xffff)
#define DWC3_GTXFIFOSIZ_TXFSTADDR(n) ((n) & 0xffff0000)
diff --git a/drivers/usb/dwc3/dwc3-msm.c b/drivers/usb/dwc3/dwc3-msm.c
index f592e6b..a311cbd 100644
--- a/drivers/usb/dwc3/dwc3-msm.c
+++ b/drivers/usb/dwc3/dwc3-msm.c
@@ -54,6 +54,7 @@
#include "dbm.h"
#include "debug.h"
#include "xhci.h"
+#include "../pd/usbpd.h"
#define SDP_CONNETION_CHECK_TIME 10000 /* in ms */
@@ -273,6 +274,7 @@ struct dwc3_msm {
struct mutex suspend_resume_mutex;
enum usb_device_speed override_usb_speed;
+ u32 auto_vbus_src_sel_threshold;
};
#define USB_HSPHY_3P3_VOL_MIN 3050000 /* uV */
@@ -1689,7 +1691,7 @@ static int msm_dwc3_usbdev_notify(struct notifier_block *self,
}
mdwc->hc_died = true;
- schedule_delayed_work(&mdwc->sm_work, 0);
+ queue_delayed_work(system_freezable_wq, &mdwc->sm_work, 0);
return 0;
}
@@ -2638,7 +2640,7 @@ static void dwc3_ext_event_notify(struct dwc3_msm *mdwc)
clear_bit(B_SUSPEND, &mdwc->inputs);
}
- schedule_delayed_work(&mdwc->sm_work, 0);
+ queue_delayed_work(system_freezable_wq, &mdwc->sm_work, 0);
}
static void dwc3_resume_work(struct work_struct *w)
@@ -3302,11 +3304,34 @@ static ssize_t usb_compliance_mode_store(struct device *dev,
static DEVICE_ATTR_RW(usb_compliance_mode);
+static ssize_t auto_vbus_src_sel_threshold_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct dwc3_msm *mdwc = dev_get_drvdata(dev);
+ return snprintf(buf, PAGE_SIZE, "%u\n",
+ mdwc->auto_vbus_src_sel_threshold);
+}
+
+static ssize_t auto_vbus_src_sel_threshold_store(struct device *dev,
+ struct device_attribute *attr, const char *buf, size_t count)
+{
+ int ret = 0;
+ struct dwc3_msm *mdwc = dev_get_drvdata(dev);
+
+
+ ret = kstrtou32(buf, 10, &mdwc->auto_vbus_src_sel_threshold);
+
+ if (ret)
+ return ret;
+
+ return count;
+}
+static DEVICE_ATTR_RW(auto_vbus_src_sel_threshold);
+
static ssize_t xhci_link_compliance_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct dwc3_msm *mdwc = dev_get_drvdata(dev);
-
if (mdwc->xhci_ss_compliance_enable)
return snprintf(buf, PAGE_SIZE, "y\n");
else
@@ -3331,6 +3356,9 @@ static ssize_t xhci_link_compliance_store(struct device *dev,
static DEVICE_ATTR_RW(xhci_link_compliance);
+
+
+
static int dwc3_msm_probe(struct platform_device *pdev)
{
struct device_node *node = pdev->dev.of_node, *dwc3_node;
@@ -3380,6 +3408,14 @@ static int dwc3_msm_probe(struct platform_device *pdev)
mdwc->charging_disabled = of_property_read_bool(node,
"qcom,charging-disabled");
+ ret = of_property_read_u32(node, "google,switch-vbus",
+ &mdwc->auto_vbus_src_sel_threshold);
+ if (ret) {
+ dev_dbg(&pdev->dev,
+ "setting auto_vbus_src_sel_threshold to zero.\n");
+ mdwc->auto_vbus_src_sel_threshold = 0;
+ }
+
ret = of_property_read_u32(node, "qcom,lpm-to-suspend-delay-ms",
&mdwc->lpm_to_suspend_delay);
if (ret) {
@@ -3679,7 +3715,7 @@ static int dwc3_msm_probe(struct platform_device *pdev)
dwc3_msm_id_notifier(&mdwc->id_nb, true, mdwc->extcon_id);
else if (!pval.intval) {
/* USB cable is not connected */
- schedule_delayed_work(&mdwc->sm_work, 0);
+ queue_delayed_work(system_freezable_wq, &mdwc->sm_work, 0);
} else {
if (pval.intval > 0)
dev_info(mdwc->dev, "charger detection in progress\n");
@@ -3688,6 +3724,7 @@ static int dwc3_msm_probe(struct platform_device *pdev)
device_create_file(&pdev->dev, &dev_attr_mode);
device_create_file(&pdev->dev, &dev_attr_speed);
device_create_file(&pdev->dev, &dev_attr_usb_compliance_mode);
+ device_create_file(&pdev->dev, &dev_attr_auto_vbus_src_sel_threshold);
device_create_file(&pdev->dev, &dev_attr_xhci_link_compliance);
host_mode = usb_get_dr_mode(&mdwc->dwc3->dev) == USB_DR_MODE_HOST;
@@ -3727,6 +3764,9 @@ static int dwc3_msm_remove(struct platform_device *pdev)
int ret_pm;
device_remove_file(&pdev->dev, &dev_attr_mode);
+ device_remove_file(&pdev->dev, &dev_attr_speed);
+ device_remove_file(&pdev->dev, &dev_attr_usb_compliance_mode);
+ device_remove_file(&pdev->dev, &dev_attr_auto_vbus_src_sel_threshold);
device_remove_file(&pdev->dev, &dev_attr_xhci_link_compliance);
if (mdwc->usb_psy)
power_supply_put(mdwc->usb_psy);
@@ -3803,6 +3843,50 @@ static int dwc3_msm_remove(struct platform_device *pdev)
return 0;
}
+static bool dwc3_is_root_hub_direct_child(struct usb_device *udev,
+ struct dwc3 *dwc)
+{
+ return udev->parent &&
+ !udev->parent->parent &&
+ udev->dev.parent->parent == &dwc->xhci->dev;
+}
+
+static bool dwc3_use_external_vbus_booster(struct usb_device *udev,
+ struct dwc3 *dwc,
+ struct dwc3_msm *mdwc)
+{
+ unsigned max_power;
+
+ if (!udev->actconfig)
+ return false;
+
+ if (udev->speed >= USB_SPEED_SUPER)
+ max_power = udev->actconfig->desc.bMaxPower * 8;
+ else
+ max_power = udev->actconfig->desc.bMaxPower * 2;
+
+ if (dwc3_is_root_hub_direct_child(udev, dwc) &&
+ udev->descriptor.bDeviceClass != USB_CLASS_HUB &&
+ max_power < mdwc->auto_vbus_src_sel_threshold)
+ return true;
+
+ return false;
+}
+
+static BLOCKING_NOTIFIER_HEAD(ext_vbus_notifier_list);
+
+void ext_vbus_register_notify(struct notifier_block *nb)
+{
+ blocking_notifier_chain_register(&ext_vbus_notifier_list, nb);
+}
+EXPORT_SYMBOL_GPL(ext_vbus_register_notify);
+
+void ext_vbus_unregister_notify(struct notifier_block *nb)
+{
+ blocking_notifier_chain_unregister(&ext_vbus_notifier_list, nb);
+}
+EXPORT_SYMBOL_GPL(ext_vbus_unregister_notify);
+
static int dwc3_msm_host_notifier(struct notifier_block *nb,
unsigned long event, void *ptr)
{
@@ -3826,8 +3910,7 @@ static int dwc3_msm_host_notifier(struct notifier_block *nb,
* i.e. dwc -> xhci -> root_hub -> udev
* root_hub's udev->parent==NULL, so traverse struct device hierarchy
*/
- if (udev->parent && !udev->parent->parent &&
- udev->dev.parent->parent == &dwc->xhci->dev) {
+ if (dwc3_is_root_hub_direct_child(udev, dwc)) {
if (event == USB_DEVICE_ADD && udev->actconfig) {
if (!dwc3_msm_is_ss_rhport_connected(mdwc)) {
/*
@@ -3868,6 +3951,16 @@ static int dwc3_msm_host_notifier(struct notifier_block *nb,
}
}
+ if (mdwc->auto_vbus_src_sel_threshold) {
+ if (event == USB_DEVICE_ADD &&
+ dwc3_use_external_vbus_booster(udev, dwc, mdwc))
+ blocking_notifier_call_chain(&ext_vbus_notifier_list,
+ EXT_VBUS_ON, NULL);
+ else
+ blocking_notifier_call_chain(&ext_vbus_notifier_list,
+ EXT_VBUS_OFF, NULL);
+ }
+
return NOTIFY_DONE;
}
@@ -4452,7 +4545,7 @@ static void dwc3_otg_sm_work(struct work_struct *w)
}
if (work)
- schedule_delayed_work(&mdwc->sm_work, delay);
+ queue_delayed_work(system_freezable_wq, &mdwc->sm_work, delay);
ret:
return;
diff --git a/drivers/usb/dwc3/dwc3-of-simple.c b/drivers/usb/dwc3/dwc3-of-simple.c
index a3e2200..58526932 100644
--- a/drivers/usb/dwc3/dwc3-of-simple.c
+++ b/drivers/usb/dwc3/dwc3-of-simple.c
@@ -132,8 +132,9 @@ static int dwc3_of_simple_remove(struct platform_device *pdev)
of_platform_depopulate(dev);
- pm_runtime_put_sync(dev);
pm_runtime_disable(dev);
+ pm_runtime_put_noidle(dev);
+ pm_runtime_set_suspended(dev);
return 0;
}
diff --git a/drivers/usb/dwc3/dwc3-omap.c b/drivers/usb/dwc3/dwc3-omap.c
index 35b6351..8e69150 100644
--- a/drivers/usb/dwc3/dwc3-omap.c
+++ b/drivers/usb/dwc3/dwc3-omap.c
@@ -512,15 +512,6 @@ static int dwc3_omap_probe(struct platform_device *pdev)
/* check the DMA Status */
reg = dwc3_omap_readl(omap->base, USBOTGSS_SYSCONFIG);
- irq_set_status_flags(omap->irq, IRQ_NOAUTOEN);
- ret = devm_request_threaded_irq(dev, omap->irq, dwc3_omap_interrupt,
- dwc3_omap_interrupt_thread, IRQF_SHARED,
- "dwc3-omap", omap);
- if (ret) {
- dev_err(dev, "failed to request IRQ #%d --> %d\n",
- omap->irq, ret);
- goto err1;
- }
ret = dwc3_omap_extcon_register(omap);
if (ret < 0)
@@ -532,8 +523,15 @@ static int dwc3_omap_probe(struct platform_device *pdev)
goto err2;
}
+ ret = devm_request_threaded_irq(dev, omap->irq, dwc3_omap_interrupt,
+ dwc3_omap_interrupt_thread, IRQF_SHARED,
+ "dwc3-omap", omap);
+ if (ret) {
+ dev_err(dev, "failed to request IRQ #%d --> %d\n",
+ omap->irq, ret);
+ goto err1;
+ }
dwc3_omap_enable_irqs(omap);
- enable_irq(omap->irq);
return 0;
err2:
@@ -598,9 +596,25 @@ static int dwc3_omap_resume(struct device *dev)
return 0;
}
+static void dwc3_omap_complete(struct device *dev)
+{
+ struct dwc3_omap *omap = dev_get_drvdata(dev);
+
+ if (extcon_get_state(omap->edev, EXTCON_USB))
+ dwc3_omap_set_mailbox(omap, OMAP_DWC3_VBUS_VALID);
+ else
+ dwc3_omap_set_mailbox(omap, OMAP_DWC3_VBUS_OFF);
+
+ if (extcon_get_state(omap->edev, EXTCON_USB_HOST))
+ dwc3_omap_set_mailbox(omap, OMAP_DWC3_ID_GROUND);
+ else
+ dwc3_omap_set_mailbox(omap, OMAP_DWC3_ID_FLOAT);
+}
+
static const struct dev_pm_ops dwc3_omap_dev_pm_ops = {
SET_SYSTEM_SLEEP_PM_OPS(dwc3_omap_suspend, dwc3_omap_resume)
+ .complete = dwc3_omap_complete,
};
#define DEV_PM_OPS (&dwc3_omap_dev_pm_ops)
diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index 804f5df..2cf2d5f 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -2670,9 +2670,9 @@ static int dwc3_cleanup_done_reqs(struct dwc3 *dwc, struct dwc3_ep *dep,
const struct dwc3_event_depevt *event, int status)
{
struct dwc3_request *req, *n;
- struct dwc3_trb *trb;
+ struct dwc3_trb *trb = NULL;
bool ioc = false;
- int ret;
+ int ret = -EINVAL;
list_for_each_entry_safe(req, n, &dep->started_list, list) {
unsigned length;
diff --git a/drivers/usb/gadget/Makefile b/drivers/usb/gadget/Makefile
index 2bde573..b9e7c2a 100644
--- a/drivers/usb/gadget/Makefile
+++ b/drivers/usb/gadget/Makefile
@@ -4,6 +4,7 @@
subdir-ccflags-$(CONFIG_USB_GADGET_DEBUG) := -DDEBUG
subdir-ccflags-$(CONFIG_USB_GADGET_VERBOSE) += -DVERBOSE_DEBUG
ccflags-y += -I$(srctree)/drivers/usb/gadget/udc
+ccflags-y += -I$(srctree)/drivers/usb/gadget/function
obj-$(CONFIG_USB_LIBCOMPOSITE) += libcomposite.o
libcomposite-y := usbstring.o config.o epautoconf.o
diff --git a/drivers/usb/gadget/composite.c b/drivers/usb/gadget/composite.c
index 0f555ee..a3ce228 100644
--- a/drivers/usb/gadget/composite.c
+++ b/drivers/usb/gadget/composite.c
@@ -1567,7 +1567,7 @@ static int count_ext_compat(struct usb_configuration *c)
return res;
}
-static void fill_ext_compat(struct usb_configuration *c, u8 *buf)
+static int fill_ext_compat(struct usb_configuration *c, u8 *buf)
{
int i, count;
@@ -1594,10 +1594,12 @@ static void fill_ext_compat(struct usb_configuration *c, u8 *buf)
buf += 23;
}
count += 24;
- if (count >= 4096)
- return;
+ if (count + 24 >= USB_COMP_EP0_OS_DESC_BUFSIZ)
+ return count;
}
}
+
+ return count;
}
static int count_ext_prop(struct usb_configuration *c, int interface)
@@ -1642,25 +1644,20 @@ static int fill_ext_prop(struct usb_configuration *c, int interface, u8 *buf)
struct usb_os_desc *d;
struct usb_os_desc_ext_prop *ext_prop;
int j, count, n, ret;
- u8 *start = buf;
f = c->interface[interface];
+ count = 10; /* header length */
for (j = 0; j < f->os_desc_n; ++j) {
if (interface != f->os_desc_table[j].if_id)
continue;
d = f->os_desc_table[j].os_desc;
if (d)
list_for_each_entry(ext_prop, &d->ext_prop, entry) {
- /* 4kB minus header length */
- n = buf - start;
- if (n >= 4086)
- return 0;
-
- count = ext_prop->data_len +
+ n = ext_prop->data_len +
ext_prop->name_len + 14;
- if (count > 4086 - n)
- return -EINVAL;
- usb_ext_prop_put_size(buf, count);
+ if (count + n >= USB_COMP_EP0_OS_DESC_BUFSIZ)
+ return count;
+ usb_ext_prop_put_size(buf, n);
usb_ext_prop_put_type(buf, ext_prop->type);
ret = usb_ext_prop_put_name(buf, ext_prop->name,
ext_prop->name_len);
@@ -1686,11 +1683,12 @@ static int fill_ext_prop(struct usb_configuration *c, int interface, u8 *buf)
default:
return -EINVAL;
}
- buf += count;
+ buf += n;
+ count += n;
}
}
- return 0;
+ return count;
}
/*
@@ -1871,6 +1869,8 @@ composite_setup(struct usb_gadget *gadget, const struct usb_ctrlrequest *ctrl)
*/
if (w_value && !f->get_alt)
break;
+
+ spin_lock(&cdev->lock);
value = f->set_alt(f, w_index, w_value);
if (value == USB_GADGET_DELAYED_STATUS) {
DBG(cdev,
@@ -1880,6 +1880,7 @@ composite_setup(struct usb_gadget *gadget, const struct usb_ctrlrequest *ctrl)
DBG(cdev, "delayed_status count %d\n",
cdev->delayed_status);
}
+ spin_unlock(&cdev->lock);
break;
case USB_REQ_GET_INTERFACE:
if (ctrl->bRequestType != (USB_DIR_IN|USB_RECIP_INTERFACE))
@@ -1984,6 +1985,7 @@ composite_setup(struct usb_gadget *gadget, const struct usb_ctrlrequest *ctrl)
req->complete = composite_setup_complete;
buf = req->buf;
os_desc_cfg = cdev->os_desc_config;
+ w_length = min_t(u16, w_length, USB_COMP_EP0_OS_DESC_BUFSIZ);
memset(buf, 0, w_length);
buf[5] = 0x01;
switch (ctrl->bRequestType & USB_RECIP_MASK) {
@@ -2007,8 +2009,8 @@ composite_setup(struct usb_gadget *gadget, const struct usb_ctrlrequest *ctrl)
count += 16; /* header */
put_unaligned_le32(count, buf);
buf += 16;
- fill_ext_compat(os_desc_cfg, buf);
- value = w_length;
+ value = fill_ext_compat(os_desc_cfg, buf);
+ value = min_t(u16, w_length, value);
}
break;
case USB_RECIP_INTERFACE:
@@ -2037,8 +2039,7 @@ composite_setup(struct usb_gadget *gadget, const struct usb_ctrlrequest *ctrl)
interface, buf);
if (value < 0)
return value;
-
- value = w_length;
+ value = min_t(u16, w_length, value);
}
break;
}
@@ -2340,8 +2341,8 @@ int composite_os_desc_req_prepare(struct usb_composite_dev *cdev,
goto end;
}
- /* OS feature descriptor length <= 4kB */
- cdev->os_desc_req->buf = kmalloc(4096, GFP_KERNEL);
+ cdev->os_desc_req->buf = kmalloc(USB_COMP_EP0_OS_DESC_BUFSIZ,
+ GFP_KERNEL);
if (!cdev->os_desc_req->buf) {
ret = -ENOMEM;
usb_ep_free_request(ep0, cdev->os_desc_req);
diff --git a/drivers/usb/gadget/configfs.c b/drivers/usb/gadget/configfs.c
index f779fdc30..f2e2c0d 100644
--- a/drivers/usb/gadget/configfs.c
+++ b/drivers/usb/gadget/configfs.c
@@ -15,7 +15,7 @@
#include <linux/usb/ch9.h>
#ifdef CONFIG_USB_F_NCM
-#include <function/u_ncm.h>
+#include "u_ncm.h"
#endif
#ifdef CONFIG_USB_CONFIGFS_F_ACC
diff --git a/drivers/usb/gadget/function/f_fs.c b/drivers/usb/gadget/function/f_fs.c
index 9974332..d47d354 100644
--- a/drivers/usb/gadget/function/f_fs.c
+++ b/drivers/usb/gadget/function/f_fs.c
@@ -3734,7 +3734,7 @@ static int ffs_func_setup(struct usb_function *f,
ffs_log("exit");
- return 0;
+ return creq->wLength == 0 ? USB_GADGET_DELAYED_STATUS : 0;
}
static bool ffs_func_req_match(struct usb_function *f,
diff --git a/drivers/usb/gadget/function/f_gsi.c b/drivers/usb/gadget/function/f_gsi.c
index 5787039..b80555d 100644
--- a/drivers/usb/gadget/function/f_gsi.c
+++ b/drivers/usb/gadget/function/f_gsi.c
@@ -323,7 +323,7 @@ static int ipa_connect_channels(struct gsi_data_port *d_port)
(gsi->prot_id != USB_PROT_DIAG_IPA) ? IPA_CLIENT_USB_CONS :
IPA_CLIENT_USB_DPL_CONS;
in_params->ipa_ep_cfg.mode.mode = IPA_BASIC;
- in_params->teth_prot = gsi->prot_id;
+ in_params->teth_prot = (enum ipa_usb_teth_prot)gsi->prot_id;
in_params->gevntcount_low_addr =
gsi_channel_info.gevntcount_low_addr;
in_params->gevntcount_hi_addr =
@@ -364,7 +364,7 @@ static int ipa_connect_channels(struct gsi_data_port *d_port)
log_event_dbg("%s: USB GSI OUT OPS Completed", __func__);
out_params->client = IPA_CLIENT_USB_PROD;
out_params->ipa_ep_cfg.mode.mode = IPA_BASIC;
- out_params->teth_prot = gsi->prot_id;
+ out_params->teth_prot = (enum ipa_usb_teth_prot)gsi->prot_id;
out_params->gevntcount_low_addr =
gsi_channel_info.gevntcount_low_addr;
out_params->gevntcount_hi_addr =
@@ -406,7 +406,7 @@ static int ipa_connect_channels(struct gsi_data_port *d_port)
conn_params->usb_to_ipa_xferrscidx_valid =
(gsi->prot_id != USB_PROT_DIAG_IPA) ? true : false;
conn_params->ipa_to_usb_xferrscidx_valid = true;
- conn_params->teth_prot = gsi->prot_id;
+ conn_params->teth_prot = (enum ipa_usb_teth_prot)gsi->prot_id;
conn_params->teth_prot_params.max_xfer_size_bytes_to_dev = 23700;
conn_params->teth_prot_params.max_xfer_size_bytes_to_dev
= d_port->out_aggr_size;
@@ -527,7 +527,7 @@ static void ipa_disconnect_work_handler(struct gsi_data_port *d_port)
log_event_dbg("%s: Calling xdci_disconnect", __func__);
ret = ipa_usb_xdci_disconnect(gsi->d_port.out_channel_handle,
- gsi->d_port.in_channel_handle, gsi->prot_id);
+ gsi->d_port.in_channel_handle, (enum ipa_usb_teth_prot)gsi->prot_id);
if (ret)
log_event_err("%s: IPA disconnect failed %d",
__func__, ret);
@@ -570,7 +570,7 @@ static int ipa_suspend_work_handler(struct gsi_data_port *d_port)
log_event_dbg("%s: Calling xdci_suspend", __func__);
ret = ipa_usb_xdci_suspend(gsi->d_port.out_channel_handle,
- gsi->d_port.in_channel_handle, gsi->prot_id,
+ gsi->d_port.in_channel_handle, (enum ipa_usb_teth_prot)gsi->prot_id,
usb_gsi_remote_wakeup_allowed(f));
if (!ret) {
d_port->sm_state = STATE_SUSPENDED;
@@ -604,7 +604,7 @@ static void ipa_resume_work_handler(struct gsi_data_port *d_port)
ret = ipa_usb_xdci_resume(gsi->d_port.out_channel_handle,
gsi->d_port.in_channel_handle,
- gsi->prot_id);
+ (enum ipa_usb_teth_prot)gsi->prot_id);
if (ret)
log_event_dbg("%s: xdci_resume ret %d", __func__, ret);
@@ -2913,7 +2913,7 @@ static int gsi_bind(struct usb_configuration *c, struct usb_function *f)
}
gsi->d_port.ipa_usb_notify_cb = ipa_usb_notify_cb;
- status = ipa_usb_init_teth_prot(gsi->prot_id,
+ status = ipa_usb_init_teth_prot((enum ipa_usb_teth_prot)gsi->prot_id,
&gsi->d_port.ipa_init_params, gsi->d_port.ipa_usb_notify_cb,
gsi);
if (status) {
@@ -2962,7 +2962,7 @@ static void gsi_unbind(struct usb_configuration *c, struct usb_function *f)
* with ipa driver shall not fail due to unexpected state.
*/
drain_workqueue(gsi->d_port.ipa_usb_wq);
- ipa_usb_deinit_teth_prot(gsi->prot_id);
+ ipa_usb_deinit_teth_prot((enum ipa_usb_teth_prot)gsi->prot_id);
skip_ipa_dinit:
if (gsi->prot_id == USB_PROT_RNDIS_IPA) {
diff --git a/drivers/usb/gadget/function/f_mass_storage.c b/drivers/usb/gadget/function/f_mass_storage.c
index 675741b..e2c5eed 100644
--- a/drivers/usb/gadget/function/f_mass_storage.c
+++ b/drivers/usb/gadget/function/f_mass_storage.c
@@ -220,6 +220,8 @@
#include <linux/usb/gadget.h>
#include <linux/usb/composite.h>
+#include <linux/nospec.h>
+
#include "configfs.h"
@@ -3313,6 +3315,7 @@ static struct config_group *fsg_lun_make(struct config_group *group,
fsg_opts = to_fsg_opts(&group->cg_item);
if (num >= FSG_MAX_LUNS)
return ERR_PTR(-ERANGE);
+ num = array_index_nospec(num, FSG_MAX_LUNS);
mutex_lock(&fsg_opts->lock);
if (fsg_opts->refcnt || fsg_opts->common->luns[num]) {
diff --git a/drivers/usb/gadget/function/f_uac2.c b/drivers/usb/gadget/function/f_uac2.c
index c330814..326c907 100644
--- a/drivers/usb/gadget/function/f_uac2.c
+++ b/drivers/usb/gadget/function/f_uac2.c
@@ -1042,14 +1042,14 @@ static struct usb_descriptor_header *ss_playback_audio_desc[] = {
};
struct cntrl_cur_lay3 {
- __u32 dCUR;
+ __le32 dCUR;
};
struct cntrl_range_lay3 {
- __u16 wNumSubRanges;
- __u32 dMIN;
- __u32 dMAX;
- __u32 dRES;
+ __le16 wNumSubRanges;
+ __le32 dMIN;
+ __le32 dMAX;
+ __le32 dRES;
} __packed;
static inline void
@@ -1156,8 +1156,9 @@ afunc_bind(struct usb_configuration *cfg, struct usb_function *fn)
dev_err(dev, "%s:%d Error!\n", __func__, __LINE__);
return ret;
}
- std_ac_if_desc.bInterfaceNumber = ret;
iad_desc.bFirstInterface = ret;
+
+ std_ac_if_desc.bInterfaceNumber = ret;
agdev->ac_intf = ret;
agdev->ac_alt = 0;
@@ -1419,9 +1420,9 @@ in_rq_cur(struct usb_function *fn, const struct usb_ctrlrequest *cr)
memset(&c, 0, sizeof(struct cntrl_cur_lay3));
if (entity_id == USB_IN_CLK_ID)
- c.dCUR = p_srate;
+ c.dCUR = cpu_to_le32(p_srate);
else if (entity_id == USB_OUT_CLK_ID)
- c.dCUR = c_srate;
+ c.dCUR = cpu_to_le32(c_srate);
value = min_t(unsigned, w_length, sizeof c);
memcpy(req->buf, &c, value);
@@ -1459,15 +1460,15 @@ in_rq_range(struct usb_function *fn, const struct usb_ctrlrequest *cr)
if (control_selector == UAC2_CS_CONTROL_SAM_FREQ) {
if (entity_id == USB_IN_CLK_ID)
- r.dMIN = p_srate;
+ r.dMIN = cpu_to_le32(p_srate);
else if (entity_id == USB_OUT_CLK_ID)
- r.dMIN = c_srate;
+ r.dMIN = cpu_to_le32(c_srate);
else
return -EOPNOTSUPP;
r.dMAX = r.dMIN;
r.dRES = 0;
- r.wNumSubRanges = 1;
+ r.wNumSubRanges = cpu_to_le16(1);
value = min_t(unsigned, w_length, sizeof r);
memcpy(req->buf, &r, value);
diff --git a/drivers/usb/gadget/function/u_serial.c b/drivers/usb/gadget/function/u_serial.c
index 2ec28c8..24dd626 100644
--- a/drivers/usb/gadget/function/u_serial.c
+++ b/drivers/usb/gadget/function/u_serial.c
@@ -538,7 +538,7 @@ static void gs_rx_push(unsigned long _port)
}
/* push data to (open) tty */
- if (req->actual) {
+ if (req->actual && tty) {
char *packet = req->buf;
unsigned size = req->actual;
unsigned n;
diff --git a/drivers/usb/gadget/udc/atmel_usba_udc.c b/drivers/usb/gadget/udc/atmel_usba_udc.c
index ad84029..9705bcd 100644
--- a/drivers/usb/gadget/udc/atmel_usba_udc.c
+++ b/drivers/usb/gadget/udc/atmel_usba_udc.c
@@ -1923,6 +1923,8 @@ static struct usba_ep * atmel_udc_of_init(struct platform_device *pdev,
udc->errata = match->data;
udc->pmc = syscon_regmap_lookup_by_compatible("atmel,at91sam9g45-pmc");
if (IS_ERR(udc->pmc))
+ udc->pmc = syscon_regmap_lookup_by_compatible("atmel,at91sam9rl-pmc");
+ if (IS_ERR(udc->pmc))
udc->pmc = syscon_regmap_lookup_by_compatible("atmel,at91sam9x5-pmc");
if (udc->errata && IS_ERR(udc->pmc))
return ERR_CAST(udc->pmc);
diff --git a/drivers/usb/gadget/udc/dummy_hcd.c b/drivers/usb/gadget/udc/dummy_hcd.c
index ff4d6ca..ab89fa3 100644
--- a/drivers/usb/gadget/udc/dummy_hcd.c
+++ b/drivers/usb/gadget/udc/dummy_hcd.c
@@ -379,11 +379,10 @@ static void set_link_state_by_speed(struct dummy_hcd *dum_hcd)
USB_PORT_STAT_CONNECTION) == 0)
dum_hcd->port_status |=
(USB_PORT_STAT_C_CONNECTION << 16);
- if ((dum_hcd->port_status &
- USB_PORT_STAT_ENABLE) == 1 &&
- (dum_hcd->port_status &
- USB_SS_PORT_LS_U0) == 1 &&
- dum_hcd->rh_state != DUMMY_RH_SUSPENDED)
+ if ((dum_hcd->port_status & USB_PORT_STAT_ENABLE) &&
+ (dum_hcd->port_status &
+ USB_PORT_STAT_LINK_STATE) == USB_SS_PORT_LS_U0 &&
+ dum_hcd->rh_state != DUMMY_RH_SUSPENDED)
dum_hcd->active = 1;
}
} else {
diff --git a/drivers/usb/gadget/udc/fotg210-udc.c b/drivers/usb/gadget/udc/fotg210-udc.c
index 6ba122c..95df2b3 100644
--- a/drivers/usb/gadget/udc/fotg210-udc.c
+++ b/drivers/usb/gadget/udc/fotg210-udc.c
@@ -1066,12 +1066,15 @@ static struct usb_gadget_ops fotg210_gadget_ops = {
static int fotg210_udc_remove(struct platform_device *pdev)
{
struct fotg210_udc *fotg210 = platform_get_drvdata(pdev);
+ int i;
usb_del_gadget_udc(&fotg210->gadget);
iounmap(fotg210->reg);
free_irq(platform_get_irq(pdev, 0), fotg210);
fotg210_ep_free_request(&fotg210->ep[0]->ep, fotg210->ep0_req);
+ for (i = 0; i < FOTG210_MAX_NUM_EP; i++)
+ kfree(fotg210->ep[i]);
kfree(fotg210);
return 0;
@@ -1102,7 +1105,7 @@ static int fotg210_udc_probe(struct platform_device *pdev)
/* initialize udc */
fotg210 = kzalloc(sizeof(struct fotg210_udc), GFP_KERNEL);
if (fotg210 == NULL)
- goto err_alloc;
+ goto err;
for (i = 0; i < FOTG210_MAX_NUM_EP; i++) {
_ep[i] = kzalloc(sizeof(struct fotg210_ep), GFP_KERNEL);
@@ -1114,7 +1117,7 @@ static int fotg210_udc_probe(struct platform_device *pdev)
fotg210->reg = ioremap(res->start, resource_size(res));
if (fotg210->reg == NULL) {
pr_err("ioremap error.\n");
- goto err_map;
+ goto err_alloc;
}
spin_lock_init(&fotg210->lock);
@@ -1162,7 +1165,7 @@ static int fotg210_udc_probe(struct platform_device *pdev)
fotg210->ep0_req = fotg210_ep_alloc_request(&fotg210->ep[0]->ep,
GFP_KERNEL);
if (fotg210->ep0_req == NULL)
- goto err_req;
+ goto err_map;
fotg210_init(fotg210);
@@ -1190,12 +1193,14 @@ static int fotg210_udc_probe(struct platform_device *pdev)
fotg210_ep_free_request(&fotg210->ep[0]->ep, fotg210->ep0_req);
err_map:
- if (fotg210->reg)
- iounmap(fotg210->reg);
+ iounmap(fotg210->reg);
err_alloc:
+ for (i = 0; i < FOTG210_MAX_NUM_EP; i++)
+ kfree(fotg210->ep[i]);
kfree(fotg210);
+err:
return ret;
}
diff --git a/drivers/usb/gadget/udc/fsl_udc_core.c b/drivers/usb/gadget/udc/fsl_udc_core.c
index aac0ce8..8991a40 100644
--- a/drivers/usb/gadget/udc/fsl_udc_core.c
+++ b/drivers/usb/gadget/udc/fsl_udc_core.c
@@ -1310,7 +1310,7 @@ static void udc_reset_ep_queue(struct fsl_udc *udc, u8 pipe)
{
struct fsl_ep *ep = get_ep_by_pipe(udc, pipe);
- if (ep->name)
+ if (ep->ep.name)
nuke(ep, -ESHUTDOWN);
}
@@ -1698,7 +1698,7 @@ static void dtd_complete_irq(struct fsl_udc *udc)
curr_ep = get_ep_by_pipe(udc, i);
/* If the ep is configured */
- if (curr_ep->name == NULL) {
+ if (!curr_ep->ep.name) {
WARNING("Invalid EP?");
continue;
}
diff --git a/drivers/usb/gadget/udc/goku_udc.h b/drivers/usb/gadget/udc/goku_udc.h
index 86d2ada..64eb0f2 100644
--- a/drivers/usb/gadget/udc/goku_udc.h
+++ b/drivers/usb/gadget/udc/goku_udc.h
@@ -28,7 +28,7 @@ struct goku_udc_regs {
# define INT_EP1DATASET 0x00040
# define INT_EP2DATASET 0x00080
# define INT_EP3DATASET 0x00100
-#define INT_EPnNAK(n) (0x00100 < (n)) /* 0 < n < 4 */
+#define INT_EPnNAK(n) (0x00100 << (n)) /* 0 < n < 4 */
# define INT_EP1NAK 0x00200
# define INT_EP2NAK 0x00400
# define INT_EP3NAK 0x00800
diff --git a/drivers/usb/gadget/udc/net2280.c b/drivers/usb/gadget/udc/net2280.c
index d133252..7a8c366 100644
--- a/drivers/usb/gadget/udc/net2280.c
+++ b/drivers/usb/gadget/udc/net2280.c
@@ -1549,11 +1549,14 @@ static int net2280_pullup(struct usb_gadget *_gadget, int is_on)
writel(tmp | BIT(USB_DETECT_ENABLE), &dev->usb->usbctl);
} else {
writel(tmp & ~BIT(USB_DETECT_ENABLE), &dev->usb->usbctl);
- stop_activity(dev, dev->driver);
+ stop_activity(dev, NULL);
}
spin_unlock_irqrestore(&dev->lock, flags);
+ if (!is_on && dev->driver)
+ dev->driver->disconnect(&dev->gadget);
+
return 0;
}
@@ -2470,8 +2473,11 @@ static void stop_activity(struct net2280 *dev, struct usb_gadget_driver *driver)
nuke(&dev->ep[i]);
/* report disconnect; the driver is already quiesced */
- if (driver)
+ if (driver) {
+ spin_unlock(&dev->lock);
driver->disconnect(&dev->gadget);
+ spin_lock(&dev->lock);
+ }
usb_reinit(dev);
}
@@ -3345,6 +3351,8 @@ static void handle_stat0_irqs(struct net2280 *dev, u32 stat)
BIT(PCI_RETRY_ABORT_INTERRUPT))
static void handle_stat1_irqs(struct net2280 *dev, u32 stat)
+__releases(dev->lock)
+__acquires(dev->lock)
{
struct net2280_ep *ep;
u32 tmp, num, mask, scratch;
@@ -3385,12 +3393,14 @@ static void handle_stat1_irqs(struct net2280 *dev, u32 stat)
if (disconnect || reset) {
stop_activity(dev, dev->driver);
ep0_start(dev);
+ spin_unlock(&dev->lock);
if (reset)
usb_gadget_udc_reset
(&dev->gadget, dev->driver);
else
(dev->driver->disconnect)
(&dev->gadget);
+ spin_lock(&dev->lock);
return;
}
}
@@ -3409,6 +3419,7 @@ static void handle_stat1_irqs(struct net2280 *dev, u32 stat)
tmp = BIT(SUSPEND_REQUEST_CHANGE_INTERRUPT);
if (stat & tmp) {
writel(tmp, &dev->regs->irqstat1);
+ spin_unlock(&dev->lock);
if (stat & BIT(SUSPEND_REQUEST_INTERRUPT)) {
if (dev->driver->suspend)
dev->driver->suspend(&dev->gadget);
@@ -3419,6 +3430,7 @@ static void handle_stat1_irqs(struct net2280 *dev, u32 stat)
dev->driver->resume(&dev->gadget);
/* at high speed, note erratum 0133 */
}
+ spin_lock(&dev->lock);
stat &= ~tmp;
}
diff --git a/drivers/usb/gadget/udc/omap_udc.c b/drivers/usb/gadget/udc/omap_udc.c
index a8709f9..9eed494 100644
--- a/drivers/usb/gadget/udc/omap_udc.c
+++ b/drivers/usb/gadget/udc/omap_udc.c
@@ -2037,6 +2037,7 @@ static inline int machine_without_vbus_sense(void)
{
return machine_is_omap_innovator()
|| machine_is_omap_osk()
+ || machine_is_omap_palmte()
|| machine_is_sx1()
/* No known omap7xx boards with vbus sense */
|| cpu_is_omap7xx();
@@ -2045,7 +2046,7 @@ static inline int machine_without_vbus_sense(void)
static int omap_udc_start(struct usb_gadget *g,
struct usb_gadget_driver *driver)
{
- int status = -ENODEV;
+ int status;
struct omap_ep *ep;
unsigned long flags;
@@ -2083,6 +2084,7 @@ static int omap_udc_start(struct usb_gadget *g,
goto done;
}
} else {
+ status = 0;
if (can_pullup(udc))
pullup_enable(udc);
else
@@ -2612,9 +2614,22 @@ omap_ep_setup(char *name, u8 addr, u8 type,
static void omap_udc_release(struct device *dev)
{
- complete(udc->done);
+ pullup_disable(udc);
+ if (!IS_ERR_OR_NULL(udc->transceiver)) {
+ usb_put_phy(udc->transceiver);
+ udc->transceiver = NULL;
+ }
+ omap_writew(0, UDC_SYSCON1);
+ remove_proc_file();
+ if (udc->dc_clk) {
+ if (udc->clk_requested)
+ omap_udc_enable_clock(0);
+ clk_put(udc->hhc_clk);
+ clk_put(udc->dc_clk);
+ }
+ if (udc->done)
+ complete(udc->done);
kfree(udc);
- udc = NULL;
}
static int
@@ -2886,8 +2901,8 @@ static int omap_udc_probe(struct platform_device *pdev)
udc->clr_halt = UDC_RESET_EP;
/* USB general purpose IRQ: ep0, state changes, dma, etc */
- status = request_irq(pdev->resource[1].start, omap_udc_irq,
- 0, driver_name, udc);
+ status = devm_request_irq(&pdev->dev, pdev->resource[1].start,
+ omap_udc_irq, 0, driver_name, udc);
if (status != 0) {
ERR("can't get irq %d, err %d\n",
(int) pdev->resource[1].start, status);
@@ -2895,20 +2910,20 @@ static int omap_udc_probe(struct platform_device *pdev)
}
/* USB "non-iso" IRQ (PIO for all but ep0) */
- status = request_irq(pdev->resource[2].start, omap_udc_pio_irq,
- 0, "omap_udc pio", udc);
+ status = devm_request_irq(&pdev->dev, pdev->resource[2].start,
+ omap_udc_pio_irq, 0, "omap_udc pio", udc);
if (status != 0) {
ERR("can't get irq %d, err %d\n",
(int) pdev->resource[2].start, status);
- goto cleanup2;
+ goto cleanup1;
}
#ifdef USE_ISO
- status = request_irq(pdev->resource[3].start, omap_udc_iso_irq,
- 0, "omap_udc iso", udc);
+ status = devm_request_irq(&pdev->dev, pdev->resource[3].start,
+ omap_udc_iso_irq, 0, "omap_udc iso", udc);
if (status != 0) {
ERR("can't get irq %d, err %d\n",
(int) pdev->resource[3].start, status);
- goto cleanup3;
+ goto cleanup1;
}
#endif
if (cpu_is_omap16xx() || cpu_is_omap7xx()) {
@@ -2919,23 +2934,8 @@ static int omap_udc_probe(struct platform_device *pdev)
}
create_proc_file();
- status = usb_add_gadget_udc_release(&pdev->dev, &udc->gadget,
- omap_udc_release);
- if (status)
- goto cleanup4;
-
- return 0;
-
-cleanup4:
- remove_proc_file();
-
-#ifdef USE_ISO
-cleanup3:
- free_irq(pdev->resource[2].start, udc);
-#endif
-
-cleanup2:
- free_irq(pdev->resource[1].start, udc);
+ return usb_add_gadget_udc_release(&pdev->dev, &udc->gadget,
+ omap_udc_release);
cleanup1:
kfree(udc);
@@ -2962,42 +2962,15 @@ static int omap_udc_remove(struct platform_device *pdev)
{
DECLARE_COMPLETION_ONSTACK(done);
- if (!udc)
- return -ENODEV;
-
- usb_del_gadget_udc(&udc->gadget);
- if (udc->driver)
- return -EBUSY;
-
udc->done = &done;
- pullup_disable(udc);
- if (!IS_ERR_OR_NULL(udc->transceiver)) {
- usb_put_phy(udc->transceiver);
- udc->transceiver = NULL;
- }
- omap_writew(0, UDC_SYSCON1);
+ usb_del_gadget_udc(&udc->gadget);
- remove_proc_file();
-
-#ifdef USE_ISO
- free_irq(pdev->resource[3].start, udc);
-#endif
- free_irq(pdev->resource[2].start, udc);
- free_irq(pdev->resource[1].start, udc);
-
- if (udc->dc_clk) {
- if (udc->clk_requested)
- omap_udc_enable_clock(0);
- clk_put(udc->hhc_clk);
- clk_put(udc->dc_clk);
- }
+ wait_for_completion(&done);
release_mem_region(pdev->resource[0].start,
pdev->resource[0].end - pdev->resource[0].start + 1);
- wait_for_completion(&done);
-
return 0;
}
diff --git a/drivers/usb/gadget/udc/r8a66597-udc.c b/drivers/usb/gadget/udc/r8a66597-udc.c
index f2c8862..230e324 100644
--- a/drivers/usb/gadget/udc/r8a66597-udc.c
+++ b/drivers/usb/gadget/udc/r8a66597-udc.c
@@ -835,11 +835,11 @@ static void init_controller(struct r8a66597 *r8a66597)
r8a66597_bset(r8a66597, XCKE, SYSCFG0);
- msleep(3);
+ mdelay(3);
r8a66597_bset(r8a66597, PLLC, SYSCFG0);
- msleep(1);
+ mdelay(1);
r8a66597_bset(r8a66597, SCKE, SYSCFG0);
@@ -1193,7 +1193,7 @@ __acquires(r8a66597->lock)
r8a66597->ep0_req->length = 2;
/* AV: what happens if we get called again before that gets through? */
spin_unlock(&r8a66597->lock);
- r8a66597_queue(r8a66597->gadget.ep0, r8a66597->ep0_req, GFP_KERNEL);
+ r8a66597_queue(r8a66597->gadget.ep0, r8a66597->ep0_req, GFP_ATOMIC);
spin_lock(&r8a66597->lock);
}
diff --git a/drivers/usb/gadget/udc/renesas_usb3.c b/drivers/usb/gadget/udc/renesas_usb3.c
index 2197a50..924c08a 100644
--- a/drivers/usb/gadget/udc/renesas_usb3.c
+++ b/drivers/usb/gadget/udc/renesas_usb3.c
@@ -521,6 +521,13 @@ static void usb3_disconnect(struct renesas_usb3 *usb3)
usb3_usb2_pullup(usb3, 0);
usb3_clear_bit(usb3, USB30_CON_B3_CONNECT, USB3_USB30_CON);
usb3_reset_epc(usb3);
+ usb3_disable_irq_1(usb3, USB_INT_1_B2_RSUM | USB_INT_1_B3_PLLWKUP |
+ USB_INT_1_B3_LUPSUCS | USB_INT_1_B3_DISABLE |
+ USB_INT_1_SPEED | USB_INT_1_B3_WRMRST |
+ USB_INT_1_B3_HOTRST | USB_INT_1_B2_SPND |
+ USB_INT_1_B2_L1SPND | USB_INT_1_B2_USBRST);
+ usb3_clear_bit(usb3, USB_COM_CON_SPD_MODE, USB3_USB_COM_CON);
+ usb3_init_epc_registers(usb3);
if (usb3->driver)
usb3->driver->disconnect(&usb3->gadget);
@@ -621,12 +628,15 @@ static void usb3_irq_epc_int_1_speed(struct renesas_usb3 *usb3)
switch (speed) {
case USB_STA_SPEED_SS:
usb3->gadget.speed = USB_SPEED_SUPER;
+ usb3->gadget.ep0->maxpacket = USB3_EP0_SS_MAX_PACKET_SIZE;
break;
case USB_STA_SPEED_HS:
usb3->gadget.speed = USB_SPEED_HIGH;
+ usb3->gadget.ep0->maxpacket = USB3_EP0_HSFS_MAX_PACKET_SIZE;
break;
case USB_STA_SPEED_FS:
usb3->gadget.speed = USB_SPEED_FULL;
+ usb3->gadget.ep0->maxpacket = USB3_EP0_HSFS_MAX_PACKET_SIZE;
break;
default:
usb3->gadget.speed = USB_SPEED_UNKNOWN;
@@ -1851,7 +1861,7 @@ static int renesas_usb3_init_ep(struct renesas_usb3 *usb3, struct device *dev,
/* for control pipe */
usb3->gadget.ep0 = &usb3_ep->ep;
usb_ep_set_maxpacket_limit(&usb3_ep->ep,
- USB3_EP0_HSFS_MAX_PACKET_SIZE);
+ USB3_EP0_SS_MAX_PACKET_SIZE);
usb3_ep->ep.caps.type_control = true;
usb3_ep->ep.caps.dir_in = true;
usb3_ep->ep.caps.dir_out = true;
diff --git a/drivers/usb/host/hwa-hc.c b/drivers/usb/host/hwa-hc.c
index 1db0626..97750f1 100644
--- a/drivers/usb/host/hwa-hc.c
+++ b/drivers/usb/host/hwa-hc.c
@@ -654,7 +654,7 @@ static int hwahc_security_create(struct hwahc *hwahc)
top = itr + itr_size;
result = __usb_get_extra_descriptor(usb_dev->rawdescriptors[index],
le16_to_cpu(usb_dev->actconfig->desc.wTotalLength),
- USB_DT_SECURITY, (void **) &secd);
+ USB_DT_SECURITY, (void **) &secd, sizeof(*secd));
if (result == -1) {
dev_warn(dev, "BUG? WUSB host has no security descriptors\n");
return 0;
diff --git a/drivers/usb/host/ohci-hcd.c b/drivers/usb/host/ohci-hcd.c
index 2c39571..216ddee 100644
--- a/drivers/usb/host/ohci-hcd.c
+++ b/drivers/usb/host/ohci-hcd.c
@@ -446,7 +446,8 @@ static int ohci_init (struct ohci_hcd *ohci)
struct usb_hcd *hcd = ohci_to_hcd(ohci);
/* Accept arbitrarily long scatter-gather lists */
- hcd->self.sg_tablesize = ~0;
+ if (!(hcd->driver->flags & HCD_LOCAL_MEM))
+ hcd->self.sg_tablesize = ~0;
if (distrust_firmware)
ohci->flags |= OHCI_QUIRK_HUB_POWER;
diff --git a/drivers/usb/host/r8a66597-hcd.c b/drivers/usb/host/r8a66597-hcd.c
index 7bf78be..72c3ed7 100644
--- a/drivers/usb/host/r8a66597-hcd.c
+++ b/drivers/usb/host/r8a66597-hcd.c
@@ -1990,6 +1990,8 @@ static int r8a66597_urb_dequeue(struct usb_hcd *hcd, struct urb *urb,
static void r8a66597_endpoint_disable(struct usb_hcd *hcd,
struct usb_host_endpoint *hep)
+__acquires(r8a66597->lock)
+__releases(r8a66597->lock)
{
struct r8a66597 *r8a66597 = hcd_to_r8a66597(hcd);
struct r8a66597_pipe *pipe = (struct r8a66597_pipe *)hep->hcpriv;
@@ -2002,13 +2004,14 @@ static void r8a66597_endpoint_disable(struct usb_hcd *hcd,
return;
pipenum = pipe->info.pipenum;
+ spin_lock_irqsave(&r8a66597->lock, flags);
if (pipenum == 0) {
kfree(hep->hcpriv);
hep->hcpriv = NULL;
+ spin_unlock_irqrestore(&r8a66597->lock, flags);
return;
}
- spin_lock_irqsave(&r8a66597->lock, flags);
pipe_stop(r8a66597, pipe);
pipe_irq_disable(r8a66597, pipenum);
disable_irq_empty(r8a66597, pipenum);
diff --git a/drivers/usb/host/u132-hcd.c b/drivers/usb/host/u132-hcd.c
index 43d5293..4361897 100644
--- a/drivers/usb/host/u132-hcd.c
+++ b/drivers/usb/host/u132-hcd.c
@@ -2559,7 +2559,7 @@ static int u132_get_frame(struct usb_hcd *hcd)
} else {
int frame = 0;
dev_err(&u132->platform_dev->dev, "TODO: u132_get_frame\n");
- msleep(100);
+ mdelay(100);
return frame;
}
}
diff --git a/drivers/usb/host/xhci-hub.c b/drivers/usb/host/xhci-hub.c
index 643e087..8679875 100644
--- a/drivers/usb/host/xhci-hub.c
+++ b/drivers/usb/host/xhci-hub.c
@@ -772,7 +772,7 @@ static u32 xhci_get_port_status(struct usb_hcd *hcd,
status |= USB_PORT_STAT_SUSPEND;
}
if ((raw_port_status & PORT_PLS_MASK) == XDEV_RESUME &&
- !DEV_SUPERSPEED_ANY(raw_port_status)) {
+ !DEV_SUPERSPEED_ANY(raw_port_status) && hcd->speed < HCD_USB3) {
if ((raw_port_status & PORT_RESET) ||
!(raw_port_status & PORT_PE))
return 0xffffffff;
@@ -818,7 +818,7 @@ static u32 xhci_get_port_status(struct usb_hcd *hcd,
time_left = wait_for_completion_timeout(
&bus_state->rexit_done[wIndex],
msecs_to_jiffies(
- XHCI_MAX_REXIT_TIMEOUT));
+ XHCI_MAX_REXIT_TIMEOUT_MS));
spin_lock_irqsave(&xhci->lock, flags);
if (time_left) {
@@ -832,7 +832,7 @@ static u32 xhci_get_port_status(struct usb_hcd *hcd,
} else {
int port_status = readl(port_array[wIndex]);
xhci_warn(xhci, "Port resume took longer than %i msec, port status = 0x%x\n",
- XHCI_MAX_REXIT_TIMEOUT,
+ XHCI_MAX_REXIT_TIMEOUT_MS,
port_status);
status |= USB_PORT_STAT_SUSPEND;
clear_bit(wIndex, &bus_state->rexit_ports);
@@ -1257,16 +1257,17 @@ int xhci_hub_control(struct usb_hcd *hcd, u16 typeReq, u16 wValue,
break;
}
- /* Software should not attempt to set
- * port link state above '3' (U3) and the port
- * must be enabled.
- */
- if ((temp & PORT_PE) == 0 ||
- (link_state > USB_SS_PORT_LS_U3)) {
- xhci_warn(xhci, "Cannot set link state.\n");
+ /* Port must be enabled */
+ if (!(temp & PORT_PE)) {
+ retval = -ENODEV;
+ break;
+ }
+ /* Can't set port link state above '3' (U3) */
+ if (link_state > USB_SS_PORT_LS_U3) {
+ xhci_warn(xhci, "Cannot set port %d link state %d\n",
+ wIndex, link_state);
goto error;
}
-
if (link_state == USB_SS_PORT_LS_U3) {
slot_id = xhci_find_slot_id_by_port(hcd, xhci,
wIndex + 1);
@@ -1536,13 +1537,16 @@ int xhci_bus_suspend(struct usb_hcd *hcd)
__le32 __iomem **port_array;
struct xhci_bus_state *bus_state;
unsigned long flags;
+ u32 portsc_buf[USB_MAXCHILDREN];
+ bool wake_enabled;
max_ports = xhci_get_ports(hcd, &port_array);
bus_state = &xhci->bus_state[hcd_index(hcd)];
+ wake_enabled = hcd->self.root_hub->do_remote_wakeup;
spin_lock_irqsave(&xhci->lock, flags);
- if (hcd->self.root_hub->do_remote_wakeup) {
+ if (wake_enabled) {
if (bus_state->resuming_ports || /* USB2 */
bus_state->port_remote_wakeup) { /* USB3 */
spin_unlock_irqrestore(&xhci->lock, flags);
@@ -1550,26 +1554,37 @@ int xhci_bus_suspend(struct usb_hcd *hcd)
return -EBUSY;
}
}
-
- port_index = max_ports;
+ /*
+ * Prepare ports for suspend, but don't write anything before all ports
+ * are checked and we know bus suspend can proceed
+ */
bus_state->bus_suspended = 0;
+ port_index = max_ports;
while (port_index--) {
- /* suspend the port if the port is not suspended */
u32 t1, t2;
- int slot_id;
t1 = readl(port_array[port_index]);
t2 = xhci_port_state_to_neutral(t1);
+ portsc_buf[port_index] = 0;
- if ((t1 & PORT_PE) && !(t1 & PORT_PLS_MASK)) {
- xhci_dbg(xhci, "port %d not suspended\n", port_index);
- slot_id = xhci_find_slot_id_by_port(hcd, xhci,
- port_index + 1);
- if (slot_id) {
+ /* Bail out if a USB3 port has a new device in link training */
+ if ((hcd->speed >= HCD_USB3) &&
+ (t1 & PORT_PLS_MASK) == XDEV_POLLING) {
+ bus_state->bus_suspended = 0;
+ spin_unlock_irqrestore(&xhci->lock, flags);
+ xhci_dbg(xhci, "Bus suspend bailout, port in polling\n");
+ return -EBUSY;
+ }
+
+ /* suspend ports in U0, or bail out for new connect changes */
+ if ((t1 & PORT_PE) && (t1 & PORT_PLS_MASK) == XDEV_U0) {
+ if ((t1 & PORT_CSC) && wake_enabled) {
+ bus_state->bus_suspended = 0;
spin_unlock_irqrestore(&xhci->lock, flags);
- xhci_stop_device(xhci, slot_id, 1);
- spin_lock_irqsave(&xhci->lock, flags);
+ xhci_dbg(xhci, "Bus suspend bailout, port connect change\n");
+ return -EBUSY;
}
+ xhci_dbg(xhci, "port %d not suspended\n", port_index);
t2 &= ~PORT_PLS_MASK;
t2 |= PORT_LINK_STROBE | XDEV_U3;
set_bit(port_index, &bus_state->bus_suspended);
@@ -1578,7 +1593,7 @@ int xhci_bus_suspend(struct usb_hcd *hcd)
* including the USB 3.0 roothub, but only if CONFIG_PM
* is enabled, so also enable remote wake here.
*/
- if (hcd->self.root_hub->do_remote_wakeup) {
+ if (wake_enabled) {
if (t1 & PORT_CONNECT) {
t2 |= PORT_WKOC_E | PORT_WKDISC_E;
t2 &= ~PORT_WKCONN_E;
@@ -1591,7 +1606,26 @@ int xhci_bus_suspend(struct usb_hcd *hcd)
t1 = xhci_port_state_to_neutral(t1);
if (t1 != t2)
- writel(t2, port_array[port_index]);
+ portsc_buf[port_index] = t2;
+ }
+
+ /* write port settings, stopping and suspending ports if needed */
+ port_index = max_ports;
+ while (port_index--) {
+ if (!portsc_buf[port_index])
+ continue;
+ if (test_bit(port_index, &bus_state->bus_suspended)) {
+ int slot_id;
+
+ slot_id = xhci_find_slot_id_by_port(hcd, xhci,
+ port_index + 1);
+ if (slot_id) {
+ spin_unlock_irqrestore(&xhci->lock, flags);
+ xhci_stop_device(xhci, slot_id, 1);
+ spin_lock_irqsave(&xhci->lock, flags);
+ }
+ }
+ writel(portsc_buf[port_index], port_array[port_index]);
}
hcd->state = HC_STATE_SUSPENDED;
bus_state->next_statechange = jiffies + msecs_to_jiffies(10);
diff --git a/drivers/usb/host/xhci-mem.c b/drivers/usb/host/xhci-mem.c
index 588e053..6b71b45 100644
--- a/drivers/usb/host/xhci-mem.c
+++ b/drivers/usb/host/xhci-mem.c
@@ -650,7 +650,7 @@ struct xhci_ring *xhci_stream_id_to_ring(
if (!ep->stream_info)
return NULL;
- if (stream_id > ep->stream_info->num_streams)
+ if (stream_id >= ep->stream_info->num_streams)
return NULL;
return ep->stream_info->stream_rings[stream_id];
}
@@ -975,6 +975,8 @@ void xhci_free_virt_device(struct xhci_hcd *xhci, int slot_id)
if (dev->out_ctx)
xhci_free_container_ctx(xhci, dev->out_ctx);
+ if (dev->udev && dev->udev->slot_id)
+ dev->udev->slot_id = 0;
kfree(xhci->devs[slot_id]);
xhci->devs[slot_id] = NULL;
}
diff --git a/drivers/usb/host/xhci-mtk.c b/drivers/usb/host/xhci-mtk.c
index ce9e457..c108758 100644
--- a/drivers/usb/host/xhci-mtk.c
+++ b/drivers/usb/host/xhci-mtk.c
@@ -735,10 +735,10 @@ static int __maybe_unused xhci_mtk_resume(struct device *dev)
xhci_mtk_host_enable(mtk);
xhci_dbg(xhci, "%s: restart port polling\n", __func__);
- set_bit(HCD_FLAG_POLL_RH, &hcd->flags);
- usb_hcd_poll_rh_status(hcd);
set_bit(HCD_FLAG_POLL_RH, &xhci->shared_hcd->flags);
usb_hcd_poll_rh_status(xhci->shared_hcd);
+ set_bit(HCD_FLAG_POLL_RH, &hcd->flags);
+ usb_hcd_poll_rh_status(hcd);
return 0;
}
diff --git a/drivers/usb/host/xhci-pci.c b/drivers/usb/host/xhci-pci.c
index f6782a3..b514055 100644
--- a/drivers/usb/host/xhci-pci.c
+++ b/drivers/usb/host/xhci-pci.c
@@ -179,6 +179,8 @@ static void xhci_pci_quirks(struct device *dev, struct xhci_hcd *xhci)
}
if (pdev->vendor == PCI_VENDOR_ID_INTEL &&
(pdev->device == PCI_DEVICE_ID_INTEL_CHERRYVIEW_XHCI ||
+ pdev->device == PCI_DEVICE_ID_INTEL_SUNRISEPOINT_LP_XHCI ||
+ pdev->device == PCI_DEVICE_ID_INTEL_SUNRISEPOINT_H_XHCI ||
pdev->device == PCI_DEVICE_ID_INTEL_APL_XHCI ||
pdev->device == PCI_DEVICE_ID_INTEL_DNV_XHCI))
xhci->quirks |= XHCI_MISSING_CAS;
diff --git a/drivers/usb/host/xhci-ring.c b/drivers/usb/host/xhci-ring.c
index f7c753f..32cb995 100644
--- a/drivers/usb/host/xhci-ring.c
+++ b/drivers/usb/host/xhci-ring.c
@@ -1666,7 +1666,7 @@ static void handle_port_status(struct xhci_hcd *xhci,
* RExit to a disconnect state). If so, let the the driver know it's
* out of the RExit state.
*/
- if (!DEV_SUPERSPEED_ANY(temp) &&
+ if (!DEV_SUPERSPEED_ANY(temp) && hcd->speed < HCD_USB3 &&
test_and_clear_bit(faked_port_index,
&bus_state->rexit_ports)) {
complete(&bus_state->rexit_done[faked_port_index]);
diff --git a/drivers/usb/host/xhci-tegra.c b/drivers/usb/host/xhci-tegra.c
index a59fafb..97d57a9 100644
--- a/drivers/usb/host/xhci-tegra.c
+++ b/drivers/usb/host/xhci-tegra.c
@@ -482,7 +482,7 @@ static void tegra_xusb_mbox_handle(struct tegra_xusb *tegra,
unsigned long mask;
unsigned int port;
bool idle, enable;
- int err;
+ int err = 0;
memset(&rsp, 0, sizeof(rsp));
diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
index 9cba037..06fa995e 100644
--- a/drivers/usb/host/xhci.c
+++ b/drivers/usb/host/xhci.c
@@ -934,6 +934,41 @@ static void xhci_disable_port_wake_on_bits(struct xhci_hcd *xhci)
spin_unlock_irqrestore(&xhci->lock, flags);
}
+static bool xhci_pending_portevent(struct xhci_hcd *xhci)
+{
+ __le32 __iomem **port_array;
+ int port_index;
+ u32 status;
+ u32 portsc;
+
+ status = readl(&xhci->op_regs->status);
+ if (status & STS_EINT)
+ return true;
+ /*
+ * Checking STS_EINT is not enough as there is a lag between a change
+ * bit being set and the Port Status Change Event that it generated
+ * being written to the Event Ring. See note in xhci 1.1 section 4.19.2.
+ */
+
+ port_index = xhci->num_usb2_ports;
+ port_array = xhci->usb2_ports;
+ while (port_index--) {
+ portsc = readl(port_array[port_index]);
+ if (portsc & PORT_CHANGE_MASK ||
+ (portsc & PORT_PLS_MASK) == XDEV_RESUME)
+ return true;
+ }
+ port_index = xhci->num_usb3_ports;
+ port_array = xhci->usb3_ports;
+ while (port_index--) {
+ portsc = readl(port_array[port_index]);
+ if (portsc & PORT_CHANGE_MASK ||
+ (portsc & PORT_PLS_MASK) == XDEV_RESUME)
+ return true;
+ }
+ return false;
+}
+
/*
* Stop HC (not bus-specific)
*
@@ -1031,7 +1066,7 @@ EXPORT_SYMBOL_GPL(xhci_suspend);
*/
int xhci_resume(struct xhci_hcd *xhci, bool hibernated)
{
- u32 command, temp = 0, status;
+ u32 command, temp = 0;
struct usb_hcd *hcd = xhci_to_hcd(xhci);
struct usb_hcd *secondary_hcd;
int retval = 0;
@@ -1065,8 +1100,13 @@ int xhci_resume(struct xhci_hcd *xhci, bool hibernated)
command = readl(&xhci->op_regs->command);
command |= CMD_CRS;
writel(command, &xhci->op_regs->command);
+ /*
+ * Some controllers take up to 55+ ms to complete the controller
+ * restore so setting the timeout to 100ms. Xhci specification
+ * doesn't mention any timeout value.
+ */
if (xhci_handshake(&xhci->op_regs->status,
- STS_RESTORE, 0, 10 * 1000)) {
+ STS_RESTORE, 0, 100 * 1000)) {
xhci_warn(xhci, "WARN: xHC restore state timeout\n");
spin_unlock_irq(&xhci->lock);
return -ETIMEDOUT;
@@ -1153,8 +1193,7 @@ int xhci_resume(struct xhci_hcd *xhci, bool hibernated)
done:
if (retval == 0) {
/* Resume root hubs only when have pending events. */
- status = readl(&xhci->op_regs->status);
- if (status & STS_EINT) {
+ if (xhci_pending_portevent(xhci)) {
usb_hcd_resume_root_hub(xhci->shared_hcd);
usb_hcd_resume_root_hub(hcd);
}
@@ -3662,6 +3701,9 @@ void xhci_free_dev(struct usb_hcd *hcd, struct usb_device *udev)
}
spin_lock_irqsave(&xhci->lock, flags);
+
+ virt_dev->udev = NULL;
+
/* Don't disable the slot if the host controller is dead. */
state = readl(&xhci->op_regs->status);
if (state == 0xffffffff || (xhci->xhc_state & XHCI_STATE_DYING) ||
@@ -4401,6 +4443,14 @@ static u16 xhci_calculate_u1_timeout(struct xhci_hcd *xhci,
{
unsigned long long timeout_ns;
+ /* Prevent U1 if service interval is shorter than U1 exit latency */
+ if (usb_endpoint_xfer_int(desc) || usb_endpoint_xfer_isoc(desc)) {
+ if (xhci_service_interval_to_ns(desc) <= udev->u1_params.mel) {
+ dev_dbg(&udev->dev, "Disable U1, ESIT shorter than exit latency\n");
+ return USB3_LPM_DISABLED;
+ }
+ }
+
if (xhci->quirks & XHCI_INTEL_HOST)
timeout_ns = xhci_calculate_intel_u1_timeout(udev, desc);
else
@@ -4457,6 +4507,14 @@ static u16 xhci_calculate_u2_timeout(struct xhci_hcd *xhci,
{
unsigned long long timeout_ns;
+ /* Prevent U2 if service interval is shorter than U2 exit latency */
+ if (usb_endpoint_xfer_int(desc) || usb_endpoint_xfer_isoc(desc)) {
+ if (xhci_service_interval_to_ns(desc) <= udev->u2_params.mel) {
+ dev_dbg(&udev->dev, "Disable U2, ESIT shorter than exit latency\n");
+ return USB3_LPM_DISABLED;
+ }
+ }
+
if (xhci->quirks & XHCI_INTEL_HOST)
timeout_ns = xhci_calculate_intel_u2_timeout(udev, desc);
else
diff --git a/drivers/usb/host/xhci.h b/drivers/usb/host/xhci.h
index a5153ca..93043ed 100644
--- a/drivers/usb/host/xhci.h
+++ b/drivers/usb/host/xhci.h
@@ -385,6 +385,10 @@ struct xhci_op_regs {
#define PORT_PLC (1 << 22)
/* port configure error change - port failed to configure its link partner */
#define PORT_CEC (1 << 23)
+#define PORT_CHANGE_MASK (PORT_CSC | PORT_PEC | PORT_WRC | PORT_OCC | \
+ PORT_RC | PORT_PLC | PORT_CEC)
+
+
/* Cold Attach Status - xHC can set this bit to report device attached during
* Sx state. Warm port reset should be perfomed to clear this bit and move port
* to connected state.
@@ -1505,7 +1509,7 @@ struct xhci_bus_state {
* It can take up to 20 ms to transition from RExit to U0 on the
* Intel Lynx Point LP xHCI host.
*/
-#define XHCI_MAX_REXIT_TIMEOUT (20 * 1000)
+#define XHCI_MAX_REXIT_TIMEOUT_MS 20
static inline unsigned int hcd_index(struct usb_hcd *hcd)
{
diff --git a/drivers/usb/misc/appledisplay.c b/drivers/usb/misc/appledisplay.c
index da5ff40..b8092bc 100644
--- a/drivers/usb/misc/appledisplay.c
+++ b/drivers/usb/misc/appledisplay.c
@@ -63,6 +63,8 @@ static const struct usb_device_id appledisplay_table[] = {
{ APPLEDISPLAY_DEVICE(0x9219) },
{ APPLEDISPLAY_DEVICE(0x921c) },
{ APPLEDISPLAY_DEVICE(0x921d) },
+ { APPLEDISPLAY_DEVICE(0x9222) },
+ { APPLEDISPLAY_DEVICE(0x9226) },
{ APPLEDISPLAY_DEVICE(0x9236) },
/* Terminating entry */
diff --git a/drivers/usb/misc/uss720.c b/drivers/usb/misc/uss720.c
index 9ff6652..e77465a 100644
--- a/drivers/usb/misc/uss720.c
+++ b/drivers/usb/misc/uss720.c
@@ -385,7 +385,7 @@ static unsigned char parport_uss720_frob_control(struct parport *pp, unsigned ch
mask &= 0x0f;
val &= 0x0f;
d = (priv->reg[1] & (~mask)) ^ val;
- if (set_1284_register(pp, 2, d, GFP_KERNEL))
+ if (set_1284_register(pp, 2, d, GFP_ATOMIC))
return 0;
priv->reg[1] = d;
return d & 0xf;
@@ -395,7 +395,7 @@ static unsigned char parport_uss720_read_status(struct parport *pp)
{
unsigned char ret;
- if (get_1284_register(pp, 1, &ret, GFP_KERNEL))
+ if (get_1284_register(pp, 1, &ret, GFP_ATOMIC))
return 0;
return ret & 0xf8;
}
diff --git a/drivers/usb/misc/yurex.c b/drivers/usb/misc/yurex.c
index 54e53ac..1e67234 100644
--- a/drivers/usb/misc/yurex.c
+++ b/drivers/usb/misc/yurex.c
@@ -406,8 +406,7 @@ static ssize_t yurex_read(struct file *file, char __user *buffer, size_t count,
loff_t *ppos)
{
struct usb_yurex *dev;
- int retval = 0;
- int bytes_read = 0;
+ int len = 0;
char in_buffer[20];
unsigned long flags;
@@ -415,26 +414,19 @@ static ssize_t yurex_read(struct file *file, char __user *buffer, size_t count,
mutex_lock(&dev->io_mutex);
if (!dev->interface) { /* already disconnected */
- retval = -ENODEV;
- goto exit;
+ mutex_unlock(&dev->io_mutex);
+ return -ENODEV;
}
spin_lock_irqsave(&dev->lock, flags);
- bytes_read = snprintf(in_buffer, 20, "%lld\n", dev->bbu);
+ len = snprintf(in_buffer, 20, "%lld\n", dev->bbu);
spin_unlock_irqrestore(&dev->lock, flags);
-
- if (*ppos < bytes_read) {
- if (copy_to_user(buffer, in_buffer + *ppos, bytes_read - *ppos))
- retval = -EFAULT;
- else {
- retval = bytes_read - *ppos;
- *ppos += bytes_read;
- }
- }
-
-exit:
mutex_unlock(&dev->io_mutex);
- return retval;
+
+ if (WARN_ON_ONCE(len >= sizeof(in_buffer)))
+ return -EIO;
+
+ return simple_read_from_buffer(buffer, count, ppos, in_buffer, len);
}
static ssize_t yurex_write(struct file *file, const char __user *user_buffer,
@@ -442,13 +434,13 @@ static ssize_t yurex_write(struct file *file, const char __user *user_buffer,
{
struct usb_yurex *dev;
int i, set = 0, retval = 0;
- char buffer[16];
+ char buffer[16 + 1];
char *data = buffer;
unsigned long long c, c2 = 0;
signed long timeout = 0;
DEFINE_WAIT(wait);
- count = min(sizeof(buffer), count);
+ count = min(sizeof(buffer) - 1, count);
dev = file->private_data;
/* verify that we actually have some data to write */
@@ -467,6 +459,7 @@ static ssize_t yurex_write(struct file *file, const char __user *user_buffer,
retval = -EFAULT;
goto error;
}
+ buffer[count] = 0;
memset(dev->cntl_buffer, CMD_PADDING, YUREX_BUF_SIZE);
switch (buffer[0]) {
diff --git a/drivers/usb/musb/musb_core.c b/drivers/usb/musb/musb_core.c
index 2d9a806..579aa9a 100644
--- a/drivers/usb/musb/musb_core.c
+++ b/drivers/usb/musb/musb_core.c
@@ -1774,6 +1774,7 @@ musb_vbus_show(struct device *dev, struct device_attribute *attr, char *buf)
int vbus;
u8 devctl;
+ pm_runtime_get_sync(dev);
spin_lock_irqsave(&musb->lock, flags);
val = musb->a_wait_bcon;
vbus = musb_platform_get_vbus_status(musb);
@@ -1787,6 +1788,7 @@ musb_vbus_show(struct device *dev, struct device_attribute *attr, char *buf)
vbus = 0;
}
spin_unlock_irqrestore(&musb->lock, flags);
+ pm_runtime_put_sync(dev);
return sprintf(buf, "Vbus %s, timeout %lu msec\n",
vbus ? "on" : "off", val);
@@ -2483,10 +2485,11 @@ static int musb_remove(struct platform_device *pdev)
musb_generic_disable(musb);
spin_unlock_irqrestore(&musb->lock, flags);
musb_writeb(musb->mregs, MUSB_DEVCTL, 0);
+ musb_platform_exit(musb);
+
pm_runtime_dont_use_autosuspend(musb->controller);
pm_runtime_put_sync(musb->controller);
pm_runtime_disable(musb->controller);
- musb_platform_exit(musb);
musb_phy_callback = NULL;
if (musb->dma_controller)
musb_dma_controller_destroy(musb->dma_controller);
@@ -2710,7 +2713,8 @@ static int musb_resume(struct device *dev)
if ((devctl & mask) != (musb->context.devctl & mask))
musb->port1_status = 0;
- musb_start(musb);
+ musb_enable_interrupts(musb);
+ musb_platform_enable(musb);
spin_lock_irqsave(&musb->lock, flags);
error = musb_run_resume_work(musb);
diff --git a/drivers/usb/musb/musb_gadget.c b/drivers/usb/musb/musb_gadget.c
index a55173c..f1219f6 100644
--- a/drivers/usb/musb/musb_gadget.c
+++ b/drivers/usb/musb/musb_gadget.c
@@ -442,7 +442,6 @@ void musb_g_tx(struct musb *musb, u8 epnum)
req = next_request(musb_ep);
request = &req->request;
- trace_musb_req_tx(req);
csr = musb_readw(epio, MUSB_TXCSR);
musb_dbg(musb, "<== %s, txcsr %04x", musb_ep->end_point.name, csr);
@@ -481,6 +480,8 @@ void musb_g_tx(struct musb *musb, u8 epnum)
u8 is_dma = 0;
bool short_packet = false;
+ trace_musb_req_tx(req);
+
if (dma && (csr & MUSB_TXCSR_DMAENAB)) {
is_dma = 1;
csr |= MUSB_TXCSR_P_WZC_BITS;
diff --git a/drivers/usb/musb/musb_host.c b/drivers/usb/musb/musb_host.c
index 4303389..19b5f08 100644
--- a/drivers/usb/musb/musb_host.c
+++ b/drivers/usb/musb/musb_host.c
@@ -1023,7 +1023,9 @@ static void musb_bulk_nak_timeout(struct musb *musb, struct musb_hw_ep *ep,
/* set tx_reinit and schedule the next qh */
ep->tx_reinit = 1;
}
- musb_start_urb(musb, is_in, next_qh);
+
+ if (next_qh)
+ musb_start_urb(musb, is_in, next_qh);
}
}
@@ -2552,8 +2554,11 @@ static int musb_bus_suspend(struct usb_hcd *hcd)
{
struct musb *musb = hcd_to_musb(hcd);
u8 devctl;
+ int ret;
- musb_port_suspend(musb, true);
+ ret = musb_port_suspend(musb, true);
+ if (ret)
+ return ret;
if (!is_host_active(musb))
return 0;
diff --git a/drivers/usb/musb/musb_host.h b/drivers/usb/musb/musb_host.h
index 7bbf01bf..54d02ed 100644
--- a/drivers/usb/musb/musb_host.h
+++ b/drivers/usb/musb/musb_host.h
@@ -92,7 +92,7 @@ extern void musb_host_rx(struct musb *, u8);
extern void musb_root_disconnect(struct musb *musb);
extern void musb_host_resume_root_hub(struct musb *musb);
extern void musb_host_poke_root_hub(struct musb *musb);
-extern void musb_port_suspend(struct musb *musb, bool do_suspend);
+extern int musb_port_suspend(struct musb *musb, bool do_suspend);
extern void musb_port_reset(struct musb *musb, bool do_reset);
extern void musb_host_finish_resume(struct work_struct *work);
#else
@@ -124,7 +124,10 @@ static inline void musb_root_disconnect(struct musb *musb) {}
static inline void musb_host_resume_root_hub(struct musb *musb) {}
static inline void musb_host_poll_rh_status(struct musb *musb) {}
static inline void musb_host_poke_root_hub(struct musb *musb) {}
-static inline void musb_port_suspend(struct musb *musb, bool do_suspend) {}
+static inline int musb_port_suspend(struct musb *musb, bool do_suspend)
+{
+ return 0;
+}
static inline void musb_port_reset(struct musb *musb, bool do_reset) {}
static inline void musb_host_finish_resume(struct work_struct *work) {}
#endif
diff --git a/drivers/usb/musb/musb_virthub.c b/drivers/usb/musb/musb_virthub.c
index 61b5f1c..71678a4 100644
--- a/drivers/usb/musb/musb_virthub.c
+++ b/drivers/usb/musb/musb_virthub.c
@@ -73,14 +73,14 @@ void musb_host_finish_resume(struct work_struct *work)
spin_unlock_irqrestore(&musb->lock, flags);
}
-void musb_port_suspend(struct musb *musb, bool do_suspend)
+int musb_port_suspend(struct musb *musb, bool do_suspend)
{
struct usb_otg *otg = musb->xceiv->otg;
u8 power;
void __iomem *mbase = musb->mregs;
if (!is_host_active(musb))
- return;
+ return 0;
/* NOTE: this doesn't necessarily put PHY into low power mode,
* turning off its clock; that's a function of PHY integration and
@@ -91,16 +91,20 @@ void musb_port_suspend(struct musb *musb, bool do_suspend)
if (do_suspend) {
int retries = 10000;
- power &= ~MUSB_POWER_RESUME;
- power |= MUSB_POWER_SUSPENDM;
- musb_writeb(mbase, MUSB_POWER, power);
+ if (power & MUSB_POWER_RESUME)
+ return -EBUSY;
- /* Needed for OPT A tests */
- power = musb_readb(mbase, MUSB_POWER);
- while (power & MUSB_POWER_SUSPENDM) {
+ if (!(power & MUSB_POWER_SUSPENDM)) {
+ power |= MUSB_POWER_SUSPENDM;
+ musb_writeb(mbase, MUSB_POWER, power);
+
+ /* Needed for OPT A tests */
power = musb_readb(mbase, MUSB_POWER);
- if (retries-- < 1)
- break;
+ while (power & MUSB_POWER_SUSPENDM) {
+ power = musb_readb(mbase, MUSB_POWER);
+ if (retries-- < 1)
+ break;
+ }
}
musb_dbg(musb, "Root port suspended, power %02x", power);
@@ -137,6 +141,7 @@ void musb_port_suspend(struct musb *musb, bool do_suspend)
schedule_delayed_work(&musb->finish_resume_work,
msecs_to_jiffies(USB_RESUME_TIMEOUT));
}
+ return 0;
}
void musb_port_reset(struct musb *musb, bool do_reset)
diff --git a/drivers/usb/pd/Kconfig b/drivers/usb/pd/Kconfig
index f3facc6..7c9caf6 100644
--- a/drivers/usb/pd/Kconfig
+++ b/drivers/usb/pd/Kconfig
@@ -18,6 +18,15 @@
PD PHY driver in order to transmit and receive PD
messages on its behalf.
+config USB_PD_ENGINE
+ tristate "USB Power Delivery Engine based on Type-C Port Manager"
+ depends on EXTCON
+ select TYPEC_TCPM
+ select USB_PD
+ help
+ Say Y here to enable Type-C Port Manager as PD protocol
+ and policy engine.
+
config QPNP_USB_PDPHY
tristate "QPNP USB Power Delivery PHY"
depends on SPMI
diff --git a/drivers/usb/pd/Makefile b/drivers/usb/pd/Makefile
index f487070..c28e8e9 100644
--- a/drivers/usb/pd/Makefile
+++ b/drivers/usb/pd/Makefile
@@ -3,4 +3,5 @@
#
obj-$(CONFIG_USB_PD_POLICY) += policy_engine.o
+obj-$(CONFIG_USB_PD_ENGINE) += pd_engine.o
obj-$(CONFIG_QPNP_USB_PDPHY) += qpnp-pdphy.o
diff --git a/drivers/usb/pd/pd_engine.c b/drivers/usb/pd/pd_engine.c
new file mode 100644
index 0000000..fd9c7e5
--- /dev/null
+++ b/drivers/usb/pd/pd_engine.c
@@ -0,0 +1,2358 @@
+/*
+ * Copyright 2016-2017 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/debugfs.h>
+#include <linux/device.h>
+#include <linux/delay.h>
+#include <linux/extcon.h>
+#include <linux/jiffies.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/pmic-voter.h>
+#include <linux/power_supply.h>
+#include <linux/property.h>
+#include <linux/regulator/consumer.h>
+#include <linux/rtc.h>
+#include <linux/seq_file.h>
+#include <linux/slab.h>
+#include <linux/types.h>
+#include <linux/usb/typec.h>
+#include <linux/usb/pd.h>
+#include <linux/usb/tcpm.h>
+#include <linux/vmalloc.h>
+#include <linux/workqueue.h>
+#include "usbpd.h"
+
+#define LOG_BUFFER_ENTRIES 1024
+#define LOG_BUFFER_ENTRY_SIZE 256
+
+#define EXT_VBUS_WORK_DELAY_MS 5000
+#define EXT_VBUS_OVERLAP_MS 5
+
+#define OTG_ICL_VOTER "OTG_ICL_VOTER"
+#define OTG_DISABLE_APSD_VOTER "OTG_DISABLE_APSD_VOTER"
+#define DISABLE_CC_VOTER "DISABLE_CC_VOTER"
+
+struct usbpd {
+ struct device dev;
+
+ struct tcpm_port *tcpm_port;
+ struct tcpc_dev tcpc_dev;
+
+ struct pd_phy_params pdphy_params;
+ bool pdphy_open;
+
+ struct extcon_dev *extcon;
+
+ struct power_supply *usb_psy;
+ struct power_supply *wireless_psy;
+ struct notifier_block psy_nb;
+
+ struct regulator *vbus;
+ struct regulator *vconn;
+ struct regulator *ext_vbus;
+ bool vbus_output;
+ bool external_vbus;
+ bool external_vbus_update;
+ bool vconn_output;
+
+ bool vbus_present;
+ enum power_supply_type psy_type;
+ enum typec_cc_status cc1;
+ enum typec_cc_status cc2;
+ bool is_cable_flipped;
+
+ bool pending_update_usb_data;
+ bool extcon_usb;
+ bool extcon_usb_host;
+ bool extcon_usb_cc;
+
+ struct mutex lock; /* struct usbpd access lock */
+ struct workqueue_struct *wq;
+
+ enum typec_role cur_pwr_role;
+ enum typec_data_role cur_data_role;
+
+ /* Flag gets set when hard_reset is being performed */
+ bool in_hard_reset;
+
+ /* Flag gets set when a pd_capable partner is attached */
+ bool pd_capable;
+
+ /* debugfs logging */
+ struct dentry *dentry;
+ spinlock_t logbuffer_lock; /* log buffer access lock */
+ int logbuffer_head;
+ int logbuffer_tail;
+ u8 *logbuffer;
+ bool in_pr_swap;
+
+ /* Ext vbus management */
+ struct delayed_work ext_vbus_work;
+ struct notifier_block ext_vbus_nb;
+ struct votable *usb_icl_votable;
+ struct votable *apsd_disable_votable;
+
+ /* Acutual regulator status */
+ bool smb2_vbus_reg;
+ bool ext_vbus_reg;
+
+ bool suspend_since_last_logged;
+ bool first_suspend;
+ /* Indicates whether the device has to honor usb suspend power limits*/
+ bool suspend_supported;
+ bool usb_comm_capable;
+
+ bool apsd_done;
+ bool wlc_supported;
+ bool wireless_online;
+ bool in_explicit_contract;
+
+ /* alternate source capabilities */
+ struct work_struct update_pdo_work;
+ bool default_src_cap;
+
+ struct votable *disable_pr_switch;
+};
+
+/*
+ * Logging
+ */
+
+static void __pd_engine_log(struct usbpd *pd, const char *tmpbuffer,
+ bool record_utc) {
+ u64 ts_nsec = local_clock();
+ unsigned long rem_nsec = do_div(ts_nsec, 1000000000);
+
+ if (record_utc) {
+ struct timespec ts;
+ struct rtc_time tm;
+
+ getnstimeofday(&ts);
+ rtc_time_to_tm(ts.tv_sec, &tm);
+ scnprintf(pd->logbuffer + (pd->logbuffer_head *
+ LOG_BUFFER_ENTRY_SIZE),
+ LOG_BUFFER_ENTRY_SIZE,
+ "[%5lu.%06lu] %d-%02d-%02d %02d:%02d:%02d.%09lu UTC",
+ (unsigned long)ts_nsec, rem_nsec / 1000,
+ tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday,
+ tm.tm_hour, tm.tm_min, tm.tm_sec, ts.tv_nsec);
+ } else {
+ scnprintf(pd->logbuffer + (pd->logbuffer_head *
+ LOG_BUFFER_ENTRY_SIZE),
+ LOG_BUFFER_ENTRY_SIZE, "[%5lu.%06lu] %s",
+ (unsigned long)ts_nsec, rem_nsec / 1000,
+ tmpbuffer);
+ }
+
+ pd->logbuffer_head = (pd->logbuffer_head + 1) % LOG_BUFFER_ENTRIES;
+ if (pd->logbuffer_head == pd->logbuffer_tail) {
+ pd->logbuffer_tail = (pd->logbuffer_tail + 1)
+ % LOG_BUFFER_ENTRIES;
+ }
+}
+
+static void _pd_engine_log(struct usbpd *pd, const char *fmt, va_list args,
+ bool force_print_utc)
+{
+ char tmpbuffer[LOG_BUFFER_ENTRY_SIZE];
+ unsigned long flags;
+
+ if (!force_print_utc)
+ vsnprintf(tmpbuffer, sizeof(tmpbuffer), fmt, args);
+
+ spin_lock_irqsave(&pd->logbuffer_lock, flags);
+ if (pd->logbuffer_head < 0 ||
+ pd->logbuffer_head >= LOG_BUFFER_ENTRIES) {
+ dev_warn(&pd->dev,
+ "Bad log buffer index %d\n", pd->logbuffer_head);
+ goto abort;
+ }
+
+ if ((pd->logbuffer_head == pd->logbuffer_tail) ||
+ (pd->logbuffer_head == LOG_BUFFER_ENTRIES - 1))
+ __pd_engine_log(pd, tmpbuffer, true);
+
+ if (pd->suspend_since_last_logged) {
+ __pd_engine_log(pd, tmpbuffer, true);
+ pd->suspend_since_last_logged = false;
+ }
+
+ if (!force_print_utc)
+ __pd_engine_log(pd, tmpbuffer, false);
+
+abort:
+ spin_unlock_irqrestore(&pd->logbuffer_lock, flags);
+}
+
+static void pd_engine_log(struct usbpd *pd, const char *fmt, ...)
+{
+ va_list args;
+
+ va_start(args, fmt);
+ _pd_engine_log(pd, fmt, args, false);
+ va_end(args);
+}
+
+static int pd_engine_seq_show(struct seq_file *s, void *v)
+{
+ struct usbpd *pd = (struct usbpd *)s->private;
+ int tail;
+
+ spin_lock(&pd->logbuffer_lock);
+ tail = pd->logbuffer_tail;
+ while (tail != pd->logbuffer_head) {
+ seq_printf(s, "%s\n", pd->logbuffer +
+ (tail * LOG_BUFFER_ENTRY_SIZE));
+ tail = (tail + 1) % LOG_BUFFER_ENTRIES;
+ }
+ if (!seq_has_overflowed(s))
+ pd->logbuffer_tail = tail;
+ spin_unlock(&pd->logbuffer_lock);
+
+ return 0;
+}
+
+static int pd_engine_debug_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, pd_engine_seq_show, inode->i_private);
+}
+
+static const struct file_operations pd_engine_debug_operations = {
+ .open = pd_engine_debug_open,
+ .llseek = seq_lseek,
+ .read = seq_read,
+ .release = single_release,
+};
+
+static struct dentry *rootdir;
+
+static int pd_engine_debugfs_init(struct usbpd *pd)
+{
+ spin_lock_init(&pd->logbuffer_lock);
+ /* /sys/kernel/debug/pd_engine/usbpdX */
+ if (!rootdir) {
+ rootdir = debugfs_create_dir("pd_engine", NULL);
+ if (!rootdir)
+ return -ENOMEM;
+ }
+
+ pd->dentry = debugfs_create_file(dev_name(&pd->dev),
+ S_IFREG | S_IRUGO, rootdir,
+ pd, &pd_engine_debug_operations);
+
+ return 0;
+}
+
+static void pd_engine_debugfs_exit(struct usbpd *pd)
+{
+ debugfs_remove(pd->dentry);
+}
+
+static const char * const get_typec_mode_name(
+ enum power_supply_typec_mode typec_mode)
+{
+ switch (typec_mode) {
+ case POWER_SUPPLY_TYPEC_NONE:
+ return "NONE";
+ case POWER_SUPPLY_TYPEC_SOURCE_DEFAULT:
+ return "SOURCE_DEFAULT";
+ case POWER_SUPPLY_TYPEC_SOURCE_MEDIUM:
+ return "SOURCE_MEDIUM";
+ case POWER_SUPPLY_TYPEC_SOURCE_HIGH:
+ return "SOURCE_HIGH";
+ case POWER_SUPPLY_TYPEC_NON_COMPLIANT:
+ return "NON_COMPLIANT";
+ case POWER_SUPPLY_TYPEC_SINK:
+ return "SINK";
+ case POWER_SUPPLY_TYPEC_SINK_POWERED_CABLE:
+ return "SINK_POWERED_CABLE";
+ case POWER_SUPPLY_TYPEC_SINK_DEBUG_ACCESSORY:
+ return "SINK_DEBUG_ACCESSORY";
+ case POWER_SUPPLY_TYPEC_SINK_AUDIO_ADAPTER:
+ return "SINK_AUDIO_ADAPTER";
+ case POWER_SUPPLY_TYPEC_POWERED_CABLE_ONLY:
+ return "POWERED_CABLE_ONLY";
+ default:
+ return "UNDEFINED";
+ }
+}
+
+static const char * const get_psy_type_name(enum power_supply_type psy_type)
+{
+ switch (psy_type) {
+ case POWER_SUPPLY_TYPE_UNKNOWN:
+ return "UNKNOWN";
+ case POWER_SUPPLY_TYPE_USB:
+ return "SDP";
+ case POWER_SUPPLY_TYPE_USB_CDP:
+ return "CDP";
+ case POWER_SUPPLY_TYPE_USB_DCP:
+ return "DCP";
+ case POWER_SUPPLY_TYPE_USB_HVDCP:
+ return "HVDCP2";
+ case POWER_SUPPLY_TYPE_USB_HVDCP_3:
+ return "HVDCP3";
+ case POWER_SUPPLY_TYPE_USB_PD:
+ return "PD";
+ default:
+ return "UNDEFINED";
+ }
+}
+
+/* must align with smblib's integer representation of cc orientation. */
+enum typec_cc_orientation {
+ TYPEC_CC_ORIENTATION_NONE = 0,
+ TYPEC_CC_ORIENTATION_CC1 = 1,
+ TYPEC_CC_ORIENTATION_CC2 = 2,
+};
+
+static const char * const get_typec_cc_orientation_name(
+ enum typec_cc_orientation typec_cc_orientation)
+{
+ switch (typec_cc_orientation) {
+ case TYPEC_CC_ORIENTATION_NONE:
+ return "NONE";
+ case TYPEC_CC_ORIENTATION_CC1:
+ return "CC1";
+ case TYPEC_CC_ORIENTATION_CC2:
+ return "CC2";
+ default:
+ return "UNDEFINED";
+ }
+}
+
+static const char * const get_typec_cc_status_name(
+ enum typec_cc_status cc_status)
+{
+ switch (cc_status) {
+ case TYPEC_CC_OPEN:
+ return "OPEN";
+ case TYPEC_CC_RA:
+ return "Ra";
+ case TYPEC_CC_RD:
+ return "Rd";
+ case TYPEC_CC_RP_DEF:
+ return "Rd-def";
+ case TYPEC_CC_RP_1_5:
+ return "Rd-1.5";
+ case TYPEC_CC_RP_3_0:
+ return "Rd-3.0";
+ default:
+ return "UNDEFINED";
+ }
+}
+
+/*
+ * parses the type-c mode to cc pin status in the orientation_none special case.
+ */
+static void parse_cc_status_none_orientation(
+ enum power_supply_typec_mode typec_mode,
+ enum typec_cc_status *cc1,
+ enum typec_cc_status *cc2)
+{
+ switch (typec_mode) {
+ case POWER_SUPPLY_TYPEC_SINK_DEBUG_ACCESSORY:
+ *cc1 = *cc2 = TYPEC_CC_RD;
+ break;
+ case POWER_SUPPLY_TYPEC_SINK_AUDIO_ADAPTER:
+ *cc1 = *cc2 = TYPEC_CC_RA;
+ break;
+ default:
+ *cc1 = *cc2 = TYPEC_CC_OPEN;
+ break;
+ }
+}
+
+/*
+ * parses the type-c mode to the status of the active and inactive cc pin.
+ */
+static void parse_cc_status_active_inactive(
+ enum power_supply_typec_mode typec_mode,
+ enum typec_cc_status *cc_active,
+ enum typec_cc_status *cc_inactive)
+{
+ switch (typec_mode) {
+ case POWER_SUPPLY_TYPEC_NONE:
+ *cc_active = TYPEC_CC_OPEN;
+ *cc_inactive = TYPEC_CC_OPEN;
+ break;
+ case POWER_SUPPLY_TYPEC_SOURCE_DEFAULT:
+ *cc_active = TYPEC_CC_RP_DEF;
+ *cc_inactive = TYPEC_CC_OPEN;
+ break;
+ case POWER_SUPPLY_TYPEC_SOURCE_MEDIUM:
+ *cc_active = TYPEC_CC_RP_1_5;
+ *cc_inactive = TYPEC_CC_OPEN;
+ break;
+ case POWER_SUPPLY_TYPEC_SOURCE_HIGH:
+ *cc_active = TYPEC_CC_RP_3_0;
+ *cc_inactive = TYPEC_CC_OPEN;
+ break;
+ case POWER_SUPPLY_TYPEC_NON_COMPLIANT:
+ *cc_active = TYPEC_CC_OPEN;
+ *cc_inactive = TYPEC_CC_OPEN;
+ break;
+ case POWER_SUPPLY_TYPEC_SINK:
+ *cc_active = TYPEC_CC_RD;
+ *cc_inactive = TYPEC_CC_OPEN;
+ break;
+ case POWER_SUPPLY_TYPEC_SINK_POWERED_CABLE:
+ *cc_active = TYPEC_CC_RD;
+ *cc_inactive = TYPEC_CC_RA;
+ break;
+ case POWER_SUPPLY_TYPEC_SINK_DEBUG_ACCESSORY:
+ *cc_active = TYPEC_CC_RD;
+ *cc_inactive = TYPEC_CC_RD;
+ break;
+ case POWER_SUPPLY_TYPEC_SINK_AUDIO_ADAPTER:
+ *cc_active = TYPEC_CC_RA;
+ *cc_inactive = TYPEC_CC_RA;
+ break;
+ case POWER_SUPPLY_TYPEC_POWERED_CABLE_ONLY:
+ *cc_active = TYPEC_CC_RA;
+ *cc_inactive = TYPEC_CC_OPEN;
+ break;
+ default:
+ *cc_active = TYPEC_CC_OPEN;
+ *cc_inactive = TYPEC_CC_OPEN;
+ break;
+ }
+}
+
+static void parse_cc_status(enum power_supply_typec_mode typec_mode,
+ enum typec_cc_orientation typec_cc_orientation,
+ enum typec_cc_status *cc1,
+ enum typec_cc_status *cc2)
+{
+ enum typec_cc_status cc_active, cc_inactive;
+
+ /* handle orientation_none special cases */
+ if (typec_cc_orientation == TYPEC_CC_ORIENTATION_NONE) {
+ parse_cc_status_none_orientation(typec_mode, cc1, cc2);
+ return;
+ }
+
+ parse_cc_status_active_inactive(typec_mode, &cc_active, &cc_inactive);
+
+ /* assign cc1, cc2 status based on orientation */
+ if (typec_cc_orientation == TYPEC_CC_ORIENTATION_CC1) {
+ *cc1 = cc_active;
+ *cc2 = cc_inactive;
+ } else if (typec_cc_orientation == TYPEC_CC_ORIENTATION_CC2) {
+ *cc1 = cc_inactive;
+ *cc2 = cc_active;
+ } else {
+ *cc1 = *cc2 = TYPEC_CC_OPEN;
+ }
+}
+
+static inline bool psy_support_usb_data(enum power_supply_type psy_type)
+{
+ return (psy_type == POWER_SUPPLY_TYPE_USB) ||
+ (psy_type == POWER_SUPPLY_TYPE_USB_CDP);
+}
+
+static int update_extcon_prop(struct usbpd *pd, int extcon_type)
+{
+ int ret;
+ union extcon_property_value val;
+
+ val.intval = pd->extcon_usb_cc ? 1 : 0;
+ ret = extcon_set_property(pd->extcon, extcon_type,
+ EXTCON_PROP_USB_TYPEC_POLARITY,
+ val);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "unable to set extcon usb polarity prop [%s], ret=%d",
+ pd->extcon_usb_cc ? "Y" : "N", ret);
+ return ret;
+ }
+
+ /*
+ * When wirless charging is on and attempting to start host mode,
+ * degrade to HS as internal pmic cannot be on.
+ */
+ if (pd->wireless_online && extcon_type == EXTCON_USB_HOST)
+ val.intval = 0;
+ else
+ val.intval = 1;
+
+ ret = extcon_set_property(pd->extcon, extcon_type,
+ EXTCON_PROP_USB_SS,
+ val);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to set extcon usb ss prop, ret=%d",
+ ret);
+ return ret;
+ }
+
+ return ret;
+}
+
+static int update_usb_data_role(struct usbpd *pd)
+{
+ int ret;
+ bool extcon_usb;
+
+ /* Turn on USB data (device role) only when a valid power supply that
+ * supports USB data connection is connected, i.e. SDP or CDP
+ *
+ * update_usb_data_role() is called when APSD is done, thus here
+ * pd->psy_type is a valid detection result.
+ */
+ extcon_usb = pd->extcon_usb &&
+ (psy_support_usb_data(pd->psy_type) ||
+ pd->usb_comm_capable);
+
+ /*
+ * EXTCON_USB_HOST and EXTCON_USB is mutually exlusive.
+ * Therefore while responding to commands such as DR_SWAP,
+ * update the cable state of the one being turned off before
+ * turning on the other one.
+ */
+ if (!pd->extcon_usb_host) {
+ ret = extcon_set_state_sync(pd->extcon, EXTCON_USB_HOST,
+ pd->extcon_usb_host);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "unable to turn off extcon usb host, ret=%d",
+ ret);
+ return ret;
+ }
+ }
+
+ if (!extcon_usb) {
+ ret = extcon_set_state_sync(pd->extcon, EXTCON_USB,
+ extcon_usb);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "unable to turn off extcon usb device, ret=%d",
+ ret);
+ return ret;
+ }
+ }
+
+ /*
+ * Shouldnt be turning on extcon_usb and extcon_usb_host
+ * at the same time.
+ */
+ BUG_ON(extcon_usb && pd->extcon_usb_host);
+
+ if (extcon_usb || pd->extcon_usb_host) {
+ ret = update_extcon_prop(pd, extcon_usb ?
+ EXTCON_USB : EXTCON_USB_HOST);
+ if (ret < 0)
+ return ret;
+
+ ret = extcon_set_state_sync(pd->extcon, extcon_usb ?
+ EXTCON_USB : EXTCON_USB_HOST,
+ 1);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "unable to turn on extcon [%s], ret=%d",
+ extcon_usb ? "device" : "host", ret);
+ return ret;
+ }
+ }
+
+ pd_engine_log(pd,
+ "usb extcon: cc [%s], host [%s], usb [%s]",
+ pd->extcon_usb_cc ? "Y" : "N",
+ pd->extcon_usb_host ? "Y" : "N",
+ extcon_usb ? "Y" : "N");
+
+ return ret;
+}
+
+struct psy_changed_event {
+ struct work_struct work;
+ struct usbpd *pd;
+};
+
+static int pd_regulator_update(struct usbpd *pd, bool external_reg, bool on)
+{
+ int ret = 0;
+
+ if (on) {
+ ret = regulator_enable(external_reg ? pd->ext_vbus
+ : pd->vbus);
+ if (ret) {
+ pd_engine_log(pd,
+ "pd_regulator_update: unable to turn on %s vbus ret = %d"
+ , external_reg ? "external" : "pmic"
+ , ret);
+ return ret;
+ } else {
+ pd_engine_log(pd,
+ "pd_regulator_update: turned on %s vbus ret = %d"
+ , external_reg ? "external" : "pmic"
+ , ret);
+ }
+ } else {
+ ret = regulator_disable(external_reg ? pd->ext_vbus
+ : pd->vbus);
+ if (ret) {
+ pd_engine_log(pd,
+ "pd_regulator_update: unable to turn off %s vbus ret = %d"
+ , external_reg ? "external" : "pmic"
+ , ret);
+ return ret;
+ } else {
+ pd_engine_log(pd,
+ "pd_regulator_update: turned off %s vbus ret = %d"
+ , external_reg ? "external" : "pmic"
+ , ret);
+ }
+ }
+
+ if (external_reg)
+ pd->ext_vbus_reg = on;
+ else
+ pd->smb2_vbus_reg = on;
+
+ return ret;
+}
+
+static int update_vbus_locked(struct usbpd *pd, bool vbus_output)
+{
+ int ret = 0;
+
+ if (pd->vbus_output == vbus_output)
+ return ret;
+
+ pd->external_vbus = pd->wireless_online ? true : false;
+
+ pd_engine_log(pd,
+ "update_vbus_locked: vbus_output: %c wireless_online: %c"
+ , vbus_output ? 'Y' : 'N', pd->wireless_online ?
+ 'Y' : 'N');
+
+ schedule_work(&pd->update_pdo_work);
+
+ if (vbus_output) {
+ ret = pd_regulator_update(pd, pd->external_vbus, true);
+ if (ret)
+ return ret;
+ } else {
+ if (pd->ext_vbus_reg) {
+ ret = pd_regulator_update(pd, true, false);
+ if (ret)
+ return ret;
+ }
+
+ if (pd->smb2_vbus_reg) {
+ ret = pd_regulator_update(pd, false, false);
+ if (ret)
+ return ret;
+ }
+ }
+ pd->vbus_output = vbus_output;
+ /*
+ * No interrupt will be trigerred for the vbus change.
+ * Manually inform tcpm to check the vbus change.
+ */
+ tcpm_vbus_change(pd->tcpm_port);
+ return ret;
+}
+
+void update_external_vbus(struct work_struct *work)
+{
+ int ret;
+ union power_supply_propval val = {0};
+ struct usbpd *pd = container_of(to_delayed_work(work), struct usbpd,
+ ext_vbus_work);
+
+ mutex_lock(&pd->lock);
+ /* Exit when the old value is same as the new value or
+ * update the value and exit when vbus is not on.
+ */
+ if (pd->external_vbus == pd->external_vbus_update || !pd->vbus_output) {
+ pd_engine_log(pd,
+ "skipping update value changed: %c vbus_output: %c"
+ , pd->external_vbus == pd->external_vbus_update ?
+ 'Y' : 'N', pd->vbus_output ? 'Y' : 'N');
+ goto exit;
+ }
+
+ if (pd->wireless_online) {
+ pd_engine_log(pd, "skipping update as wireless charger online");
+ goto err;
+ }
+
+ val.intval = 1;
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_OTG_FASTROLESWAP,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to set FASTROLESWAP, ret=%d",
+ ret);
+ goto err;
+ }
+
+ /*
+ * Turn on the other regulator before turning off the other one.
+ */
+ ret = pd_regulator_update(pd, pd->external_vbus_update, true);
+ if (ret)
+ goto err;
+
+ /* Regulator overlap according to hardware recommendation */
+ msleep(EXT_VBUS_OVERLAP_MS);
+ ret = pd_regulator_update(pd, !pd->external_vbus_update, false);
+ if (ret)
+ goto err;
+
+ val.intval = 0;
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_OTG_FASTROLESWAP,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to clear FASTROLESWAP, ret=%d",
+ ret);
+ }
+
+ pd_engine_log(pd, "EXT_VBUS_%s", pd->external_vbus_update ? "ON" :
+ "OFF");
+exit:
+ pd->external_vbus = pd->external_vbus_update;
+ schedule_work(&pd->update_pdo_work);
+err:
+ pm_relax(&pd->dev);
+ mutex_unlock(&pd->lock);
+}
+
+#define PDO_FIXED_FLAGS \
+ (PDO_FIXED_DUAL_ROLE | PDO_FIXED_DATA_SWAP | PDO_FIXED_USB_COMM)
+
+static void update_src_caps(struct work_struct *work)
+{
+ struct usbpd *pd = container_of(work, struct usbpd, update_pdo_work);
+ struct tcpc_config *config = pd->tcpc_dev.config;
+ u32 pdo[1];
+
+ if (pd->wireless_online && pd->default_src_cap) {
+ pd_engine_log(pd, "alternative src_cap");
+ pdo[0] = PDO_FIXED(5000, 500, PDO_FIXED_FLAGS);
+ tcpm_update_source_capabilities(pd->tcpm_port, pdo, 1);
+ pd->default_src_cap = false;
+ } else if (!pd->wireless_online &&
+ !pd->external_vbus &&
+ !pd->default_src_cap) {
+ pd_engine_log(pd, "default src_cap");
+ tcpm_update_source_capabilities(pd->tcpm_port,
+ config->src_pdo,
+ config->nr_src_pdo);
+ pd->default_src_cap = true;
+ }
+}
+
+static void psy_changed_handler(struct work_struct *work)
+{
+ struct psy_changed_event *event = container_of(work,
+ struct psy_changed_event,
+ work);
+ struct usbpd *pd = event->pd;
+ bool vbus_present;
+ enum typec_cc_status cc1;
+ enum typec_cc_status cc2;
+
+ enum power_supply_type psy_type;
+ enum power_supply_typec_mode typec_mode;
+ enum typec_cc_orientation typec_cc_orientation;
+
+ bool pe_start;
+
+ union power_supply_propval val;
+ bool wireless_online = false;
+ int ret = 0;
+
+ pm_wakeup_event(&pd->dev, PD_ACTIVITY_TIMEOUT_MS);
+ ret = power_supply_get_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_TYPEC_MODE, &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "Unable to read TYPEC_MODE, ret=%d",
+ ret);
+ return;
+ }
+ typec_mode = val.intval;
+
+ ret = power_supply_get_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_PE_START, &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "Unable to read PE_START, ret=%d",
+ ret);
+ return;
+ }
+ pe_start = !!val.intval;
+
+ ret = power_supply_get_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_REAL_TYPE, &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "Unable to read TYPE, ret=%d", ret);
+ return;
+ }
+ psy_type = val.intval;
+ pd->apsd_done = !!psy_type;
+
+ ret = power_supply_get_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_PRESENT, &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "Unable to read ONLINE, ret=%d",
+ ret);
+ return;
+ }
+ vbus_present = val.intval;
+
+ ret = power_supply_get_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_TYPEC_CC_ORIENTATION,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "Unable to read TYPEC_CC_ORIENTATION, ret=%d",
+ ret);
+ return;
+ }
+ typec_cc_orientation = val.intval;
+
+ if (pd->wlc_supported) {
+ ret = power_supply_get_property(pd->wireless_psy,
+ POWER_SUPPLY_PROP_ONLINE,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "Unable to read wireless online property, ret=%d",
+ ret);
+ return;
+ }
+ wireless_online = val.intval ? true : false;
+ }
+
+ parse_cc_status(typec_mode, typec_cc_orientation, &cc1, &cc2);
+
+ pd_engine_log(pd,
+ "type [%s], pe_start [%s], vbus_present [%s], mode [%s], orientation [%s], cc1 [%s], cc2 [%s] ext_vbus [%s] wireless_online [%s]",
+ get_psy_type_name(psy_type),
+ pe_start ? "Y" : "N",
+ vbus_present ? "Y" : "N",
+ get_typec_mode_name(typec_mode),
+ get_typec_cc_orientation_name(typec_cc_orientation),
+ get_typec_cc_status_name(cc1),
+ get_typec_cc_status_name(cc2),
+ pd->external_vbus_update ? "Y" : "N",
+ wireless_online ? "Y" : "N");
+
+ mutex_lock(&pd->lock);
+ if (wireless_online != pd->wireless_online) {
+ pd->wireless_online = wireless_online;
+ pd_engine_log(pd, "pd->vbus_output: %c pd->external_vbus: %c",
+ pd->vbus_output ? 'Y' : 'N',
+ pd->external_vbus ? 'Y' : 'N');
+
+ /* Setting different src_cap depending on wireless status */
+ schedule_work(&pd->update_pdo_work);
+
+ if (pd->vbus_output && !pd->external_vbus) {
+ /* Turn off internal vbus */
+ if (pd_regulator_update(pd, false, false))
+ goto unlock;
+ pd->vbus_output = false;
+
+ /* Force disconnect */
+ tcpm_port_reset(pd->tcpm_port);
+
+ goto unlock;
+ }
+ }
+
+ /* Dont proceed as pmi might still be evaluating connections */
+ if (!pe_start && (typec_mode != POWER_SUPPLY_TYPEC_NONE)) {
+ pd_engine_log(pd, "Skipping update as PE_START not set yet");
+ goto unlock;
+ }
+
+ pd->psy_type = psy_type;
+ pd->is_cable_flipped = typec_cc_orientation == TYPEC_CC_ORIENTATION_CC2;
+ pd->extcon_usb_cc = pd->is_cable_flipped;
+
+ if (vbus_present != pd->vbus_present) {
+ pd_engine_log(pd, "vbus present: %s -> %s",
+ pd->vbus_present ? "True" : "False",
+ vbus_present ? "True" : "False");
+ pd->vbus_present = vbus_present;
+ tcpm_vbus_change(pd->tcpm_port);
+ }
+
+ if (cc1 != pd->cc1 || cc2 != pd->cc2) {
+ pd_engine_log(pd, "cc1: %s -> %s, cc2: %s -> %s",
+ get_typec_cc_status_name(pd->cc1),
+ get_typec_cc_status_name(cc1),
+ get_typec_cc_status_name(pd->cc2),
+ get_typec_cc_status_name(cc2));
+ pd->cc1 = cc1;
+ pd->cc2 = cc2;
+ tcpm_cc_change(pd->tcpm_port);
+ }
+
+ if (pd->apsd_done && pd->pending_update_usb_data) {
+ pd_engine_log(pd,
+ "APSD is done now, update pending usb data role");
+ ret = update_usb_data_role(pd);
+ if (ret < 0)
+ goto unlock;
+ pd->pending_update_usb_data = false;
+ }
+unlock:
+ kfree(event);
+ mutex_unlock(&pd->lock);
+}
+
+static int psy_changed(struct notifier_block *nb, unsigned long evt, void *ptr)
+{
+ struct usbpd *pd;
+ struct psy_changed_event *event;
+
+ pd = container_of(nb, struct usbpd, psy_nb);
+ if (ptr == pd->wireless_psy)
+ pd_engine_log(pd, "wireless supply changed");
+
+ if (!((ptr == pd->usb_psy || ptr == pd->wireless_psy)
+ && evt == PSY_EVENT_PROP_CHANGED))
+ return 0;
+
+ event = kzalloc(sizeof(*event), GFP_ATOMIC);
+ if (!event)
+ return -ENOMEM;
+
+ INIT_WORK(&event->work, psy_changed_handler);
+ event->pd = pd;
+ queue_work(pd->wq, &event->work);
+
+ return 0;
+}
+
+static int tcpm_init(struct tcpc_dev *dev)
+{
+ return 0;
+}
+
+static int tcpm_get_vbus(struct tcpc_dev *dev)
+{
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ bool vbus_on;
+ int ret;
+
+ mutex_lock(&pd->lock);
+
+ vbus_on = pd->vbus_present || pd->vbus_output;
+ pd_engine_log(pd, "tcpm_get_vbus: %s",
+ vbus_on ? "True" : "False");
+ ret = vbus_on ? 1 : 0;
+
+ mutex_unlock(&pd->lock);
+ return ret;
+}
+
+static int tcpm_set_cc(struct tcpc_dev *dev, enum typec_cc_status cc)
+{
+ union power_supply_propval val = {0};
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+
+ mutex_lock(&pd->lock);
+
+ switch (cc) {
+ case TYPEC_CC_OPEN:
+ val.intval = POWER_SUPPLY_TYPEC_PR_NONE;
+ break;
+ case TYPEC_CC_RD:
+ val.intval = POWER_SUPPLY_TYPEC_PR_SINK;
+ break;
+ case TYPEC_CC_RP_DEF:
+ val.intval = POWER_SUPPLY_TYPEC_PR_SOURCE;
+ break;
+ case TYPEC_CC_RP_1_5:
+ val.intval = POWER_SUPPLY_TYPEC_PR_SOURCE_1_5;
+ break;
+ default:
+ pd_engine_log(pd, "tcpm_set_cc: invalid cc %s",
+ get_typec_cc_status_name(cc));
+ ret = -EINVAL;
+ goto unlock;
+ }
+
+ if (cc == TYPEC_CC_OPEN) {
+ /* Set CC to open and block later changes */
+ ret = vote(pd->disable_pr_switch, DISABLE_CC_VOTER, true, 0);
+ if (ret < 0) {
+ pd_engine_log(pd, "vote disable_pr_switch fail, ret %d",
+ ret);
+ }
+ } else {
+ /* Make CC be configurable */
+ ret = vote(pd->disable_pr_switch, DISABLE_CC_VOTER, false, 0);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "unvote disable_pr_switch fail, ret %d",
+ ret);
+ }
+ }
+
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_TYPEC_POWER_ROLE,
+ &val);
+ if (ret < 0)
+ pd_engine_log(pd, "unable to set POWER_ROLE, ret=%d",
+ ret);
+ else
+ pd_engine_log(pd, "set POWER_ROLE to %d",
+ cc);
+
+unlock:
+ mutex_unlock(&pd->lock);
+ return ret;
+}
+
+static int tcpm_get_cc(struct tcpc_dev *dev, enum typec_cc_status *cc1,
+ enum typec_cc_status *cc2)
+{
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+
+ mutex_lock(&pd->lock);
+
+ pd_engine_log(pd, "tcpm_get_cc: cc1=%s, cc2=%s",
+ get_typec_cc_status_name(pd->cc1),
+ get_typec_cc_status_name(pd->cc2));
+ *cc1 = pd->cc1;
+ *cc2 = pd->cc2;
+
+ mutex_unlock(&pd->lock);
+ return 0;
+}
+
+static int tcpm_set_polarity(struct tcpc_dev *dev,
+ enum typec_cc_polarity polarity)
+{
+ /* Hardware handles polarity, no op here. */
+ return 0;
+}
+
+static int tcpm_set_vconn(struct tcpc_dev *dev, bool on)
+{
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+
+ mutex_lock(&pd->lock);
+
+ if (on != pd->vconn_output) {
+ if (on)
+ ret = regulator_enable(pd->vconn);
+ else
+ ret = regulator_disable(pd->vconn);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to turn %s vconn, ret=%d",
+ on ? "on" : "off", ret);
+ goto unlock;
+ }
+ pd->vconn_output = on;
+ }
+
+ pd_engine_log(pd, "set vconn: %s", on ? "on" : "off");
+
+unlock:
+ mutex_unlock(&pd->lock);
+ return ret;
+}
+
+static int tcpm_set_vbus(struct tcpc_dev *dev, bool on, bool charge)
+{
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+ int usb_icl;
+ bool work_flushed, apsd_disabled;
+
+ work_flushed = flush_delayed_work(&pd->ext_vbus_work);
+ pd_engine_log(pd, "Flushed ext vbus delayed work: %s", work_flushed ?
+ "yes" : "no");
+ mutex_lock(&pd->lock);
+
+ if (on) {
+ /* disable charging */
+ ret = vote(pd->usb_icl_votable, OTG_ICL_VOTER, true, 0);
+ if (ret < 0) {
+ pd_engine_log(pd, "vote usb_icl 0 fail, ret %d", ret);
+ goto unlock;
+ }
+ /* disable APSD */
+ ret = vote(pd->apsd_disable_votable,
+ OTG_DISABLE_APSD_VOTER, true, 0);
+ if (ret < 0) {
+ pd_engine_log(pd, "vote apsd_disable fail, ret %d",
+ ret);
+ goto vote_icl_default;
+ }
+ } else {
+ /* enable APSD */
+ ret = vote(pd->apsd_disable_votable,
+ OTG_DISABLE_APSD_VOTER, false, 0);
+ if (ret < 0) {
+ pd_engine_log(pd, "unvote apsd_disable fail, ret %d",
+ ret);
+ goto vote_icl_default;
+ }
+ /* enable charging */
+ ret = vote(pd->usb_icl_votable, OTG_ICL_VOTER, false, 0);
+ if (ret < 0) {
+ pd_engine_log(pd, "unvote usb_icl fail, ret %d", ret);
+ goto unlock;
+ }
+ }
+
+ apsd_disabled = get_effective_result(pd->apsd_disable_votable);
+ usb_icl = get_effective_result(pd->usb_icl_votable);
+ if (on && !apsd_disabled) {
+ pd_engine_log(pd, "apsd not disabled, icl = %d", usb_icl);
+ goto vote_apsd_default;
+ }
+
+ ret = update_vbus_locked(pd, on);
+
+ if (ret)
+ goto vote_apsd_default;
+
+ pd_engine_log(pd, "set vbus: %s, apsd: %s, icl: %d",
+ on ? "on" : "off",
+ apsd_disabled ? "disabled" : "enabled",
+ usb_icl);
+ goto unlock;
+
+vote_apsd_default:
+ /* enable APSD by default */
+ ret = vote(pd->apsd_disable_votable,
+ OTG_DISABLE_APSD_VOTER, false, 0);
+ if (ret < 0)
+ pd_engine_log(pd, "unvote apsd_disable fail, ret %d", ret);
+vote_icl_default:
+ /* enable charging by default */
+ ret = vote(pd->usb_icl_votable, OTG_ICL_VOTER, false, 0);
+ if (ret < 0)
+ pd_engine_log(pd, "unvote usb_icl fail, ret %d", ret);
+unlock:
+ mutex_unlock(&pd->lock);
+ return ret;
+}
+
+static int tcpm_set_current_limit(struct tcpc_dev *dev, u32 max_ma, u32 min_mv,
+ u32 max_mv)
+{
+ union power_supply_propval val = {0};
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+
+ val.intval = max_mv * 1000;
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_PD_VOLTAGE_MAX,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to set max voltage to %d, ret=%d",
+ max_mv, ret);
+ return ret;
+ }
+
+ val.intval = min_mv * 1000;
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_PD_VOLTAGE_MIN,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to set min voltage to %d, ret=%d",
+ min_mv, ret);
+ return ret;
+ }
+
+ /*
+ * smb-lib manages the current limits when pd capable partner is
+ * not attached.
+ */
+ if (pd->pd_capable) {
+ val.intval = max_ma * 1000;
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_PD_CURRENT_MAX,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to set pd current max to %d, ret=%d",
+ max_ma, ret);
+ return ret;
+ }
+ }
+
+ pd_engine_log(pd, "max_ma := %d, min_mv := %d, max_mv := %d, pd_capable := %c",
+ max_ma, min_mv, max_mv, pd->pd_capable ? 'Y' : 'N');
+ return ret;
+}
+
+static int tcpm_set_pd_rx(struct tcpc_dev *dev, bool on)
+{
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+
+ if (pd->pdphy_open == on) {
+ pd_engine_log(pd, "pd_phy already %s",
+ on ? "open" : "closed");
+ return 0;
+ }
+
+ if (on) {
+ ret = pd_phy_open(&pd->pdphy_params);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to %s pd_phy, ret=%d",
+ on ? "open" : "close", ret);
+ return ret;
+ }
+ } else {
+ /* pd_phy_close() has no return value. */
+ pd_phy_close();
+ }
+
+ pd->pdphy_open = on;
+ pd_engine_log(pd, "%s pd_phy", on ? "open" : "close");
+
+ return 0;
+}
+
+static int get_data_len(__le16 header)
+{
+ int ret;
+
+ ret = pd_header_cnt(__le16_to_cpu(header));
+ ret *= 4;
+
+ return ret;
+}
+
+static const char * const get_tcpm_transmit_type_name(
+ enum tcpm_transmit_type type)
+{
+ switch (type) {
+ case TCPC_TX_SOP:
+ return "SOP";
+ case TCPC_TX_SOP_PRIME:
+ return "SOP_PRIME";
+ case TCPC_TX_SOP_PRIME_PRIME:
+ return "SOP_PRIME_PRIME";
+ case TCPC_TX_SOP_DEBUG_PRIME:
+ return "SOP_DEBUG_PRIME";
+ case TCPC_TX_SOP_DEBUG_PRIME_PRIME:
+ return "SOP_DEBUG_PRIME_PRIME";
+ case TCPC_TX_HARD_RESET:
+ return "HARD_RESET";
+ case TCPC_TX_CABLE_RESET:
+ return "CABLE_RESET";
+ case TCPC_TX_BIST_MODE_2:
+ return "BIST_MODE_2";
+ default:
+ return "UNDEFINED";
+ }
+}
+
+struct pd_transmit_work {
+ struct work_struct work;
+ struct usbpd *pd;
+ enum tcpm_transmit_type type;
+ struct pd_message msg;
+};
+
+static const char * const get_tcpm_transmit_status_name(
+ enum tcpm_transmit_status status)
+{
+ switch (status) {
+ case TCPC_TX_SUCCESS:
+ return "SUCCESS";
+ case TCPC_TX_DISCARDED:
+ return "DISCARDED";
+ case TCPC_TX_FAILED:
+ return "FAILED";
+ default:
+ return "UNDEFINED";
+ }
+}
+
+static void pd_transmit_handler(struct work_struct *work)
+{
+ struct pd_transmit_work *pd_tx_work = container_of(work,
+ struct pd_transmit_work,
+ work);
+ struct usbpd *pd = pd_tx_work->pd;
+ enum tcpm_transmit_type type = pd_tx_work->type;
+ struct pd_message *msg = &pd_tx_work->msg;
+ enum tcpm_transmit_status status;
+ bool signal;
+ int ret = 0;
+
+ switch (type) {
+ case TCPC_TX_HARD_RESET:
+ ret = pd_phy_signal(HARD_RESET_SIG);
+ signal = true;
+ break;
+ case TCPC_TX_CABLE_RESET:
+ ret = pd_phy_signal(CABLE_RESET_SIG);
+ signal = true;
+ break;
+ case TCPC_TX_SOP:
+ ret = pd_phy_write(msg->header, (u8 *)msg->payload,
+ get_data_len(msg->header),
+ SOP_MSG);
+ signal = false;
+ break;
+ case TCPC_TX_SOP_PRIME:
+ ret = pd_phy_write(msg->header, (u8 *)msg->payload,
+ get_data_len(msg->header),
+ SOPI_MSG);
+ signal = false;
+ break;
+ case TCPC_TX_SOP_PRIME_PRIME:
+ ret = pd_phy_write(msg->header, (u8 *)msg->payload,
+ get_data_len(msg->header),
+ SOPII_MSG);
+ signal = false;
+ break;
+ default:
+ pd_engine_log(pd, "unknown pd tx type");
+ kfree(pd_tx_work);
+ return;
+ }
+ /*
+ * pd_phy_write()/pd_phy_signal() return value for hw irq event:
+ * - tx succeeded hw irq: 0 for signal or actual tx len for write
+ * - tx discarded hw irq: -EBUSY
+ * - tx failed hw irq: -EFAULT
+ */
+ if (ret >= 0)
+ status = TCPC_TX_SUCCESS;
+ else if (ret == -EBUSY)
+ status = TCPC_TX_DISCARDED;
+ else
+ status = TCPC_TX_FAILED;
+
+ tcpm_pd_transmit_complete(pd->tcpm_port, status);
+
+ if (signal)
+ pd_engine_log(pd,
+ "pd tx type [%s], pdphy ret [%d], status [%s]",
+ get_tcpm_transmit_type_name(type),
+ ret, get_tcpm_transmit_status_name(status));
+ else
+ pd_engine_log(pd,
+ "pd tx header [%#x], type [%s], pdphy ret [%d], status [%s]",
+ msg->header, get_tcpm_transmit_type_name(type),
+ ret, get_tcpm_transmit_status_name(status));
+
+ kfree(pd_tx_work);
+}
+
+static int tcpm_pd_transmit(struct tcpc_dev *dev, enum tcpm_transmit_type type,
+ const struct pd_message *msg)
+{
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ struct pd_transmit_work *pd_tx_work;
+
+ switch (type) {
+ case TCPC_TX_HARD_RESET:
+ case TCPC_TX_CABLE_RESET:
+ case TCPC_TX_SOP:
+ case TCPC_TX_SOP_PRIME:
+ case TCPC_TX_SOP_PRIME_PRIME:
+ break;
+ default:
+ pd_engine_log(pd, "unsupported pd type: %s",
+ get_tcpm_transmit_type_name(type));
+ return -EINVAL;
+ }
+
+ pd_tx_work = kzalloc(sizeof(*pd_tx_work), GFP_ATOMIC);
+ if (!pd_tx_work)
+ return -ENOMEM;
+ INIT_WORK(&pd_tx_work->work, pd_transmit_handler);
+ pd_tx_work->pd = pd;
+ pd_tx_work->type = type;
+ if (msg)
+ memcpy(&pd_tx_work->msg, msg, sizeof(*msg));
+ queue_work(pd->wq, &pd_tx_work->work);
+
+ if (msg)
+ pd_engine_log(pd, "queue pd tx header [%#x], type [%s]",
+ msg->header, get_tcpm_transmit_type_name(type));
+ else
+ pd_engine_log(pd, "queue pd tx type [%s]",
+ get_tcpm_transmit_type_name(type));
+ return 0;
+}
+
+static int tcpm_start_drp_toggling(struct tcpc_dev *dev,
+ enum typec_cc_status cc)
+{
+ /*
+ * Ignore the typec_cc_status for now. As current no
+ * src_pdo is configured, the default is Rp_def.
+ */
+ union power_supply_propval val = {0};
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret;
+
+ mutex_lock(&pd->lock);
+
+ /* Make CC be configurable */
+ ret = vote(pd->disable_pr_switch, DISABLE_CC_VOTER, false, 0);
+ if (ret < 0)
+ pd_engine_log(pd, "unvote disable_pr_switch fail, ret %d", ret);
+
+ val.intval = POWER_SUPPLY_TYPEC_PR_DUAL;
+
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_TYPEC_POWER_ROLE,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to set POWER_ROLE to PR_DUAL, ret=%d",
+ ret);
+ goto unlock;
+ }
+
+ pd->cc1 = TYPEC_CC_OPEN;
+ pd->cc2 = TYPEC_CC_OPEN;
+
+ pd_engine_log(pd, "start toggling");
+
+ /* Force a recheck of the CC status since cc lines are
+ * assumed to be open.
+ */
+ psy_changed(&pd->psy_nb, PSY_EVENT_PROP_CHANGED, pd->usb_psy);
+
+unlock:
+ mutex_unlock(&pd->lock);
+ return ret;
+}
+
+static int tcpm_set_in_pr_swap(struct tcpc_dev *dev, bool pr_swap)
+{
+ union power_supply_propval val = {0};
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+
+ mutex_lock(&pd->lock);
+ if (pd->in_pr_swap != pr_swap) {
+ if (!pr_swap) {
+ val.intval = POWER_SUPPLY_TYPEC_PR_DUAL;
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_TYPEC_POWER_ROLE,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "unable to set POWER_ROLE to PR_DUAL, ret=%d",
+ ret);
+ goto unlock;
+ }
+ /* Required for the PMIC to recover */
+ pd_engine_log(pd, "sleeping for 20mec");
+ msleep(20);
+ psy_changed(&pd->psy_nb, PSY_EVENT_PROP_CHANGED,
+ pd->usb_psy);
+ }
+ val.intval = pr_swap ? 1 : 0;
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_PR_SWAP,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "unable to set PR_SWAP to %d, ret=%d",
+ pr_swap ? 1 : 0, ret);
+ goto unlock;
+ }
+
+ pd->in_pr_swap = pr_swap;
+ pd_engine_log(pd, "PR_SWAP = %d", pr_swap ? 1 : 0);
+ }
+unlock:
+ mutex_unlock(&pd->lock);
+ return ret;
+}
+
+static int tcpm_in_pd_contract(struct tcpc_dev *dev,
+ bool status)
+{
+ union power_supply_propval val = {0};
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+
+ mutex_lock(&pd->lock);
+
+ if (status == pd->in_explicit_contract)
+ goto unlock;
+
+ /* Attempt once */
+ val.intval = status ? 1 : 0;
+ pd_engine_log(pd, "pd contract %d", status ? 1 : 0);
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_PD_IN_EXPLICIT_CONTRACT,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "unable to set pd contract to %d, ret=%d",
+ status ? 1 : 0, ret);
+ } else {
+ pd->in_explicit_contract = status;
+ }
+
+unlock:
+ mutex_unlock(&pd->lock);
+ return ret;
+}
+
+static int tcpm_set_suspend_supported(struct tcpc_dev *dev,
+ bool suspend_supported)
+{
+ union power_supply_propval val = {0};
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+
+ mutex_lock(&pd->lock);
+
+ if (suspend_supported == pd->suspend_supported)
+ goto unlock;
+
+ /* Attempt once */
+ pd->suspend_supported = suspend_supported;
+ val.intval = suspend_supported ? 1 : 0;
+ pd_engine_log(pd, "usb suspend %d", suspend_supported ? 1 : 0);
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_PD_USB_SUSPEND_SUPPORTED,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "unable to set suspend flag to %d, ret=%d",
+ suspend_supported ? 1 : 0, ret);
+ }
+
+unlock:
+ mutex_unlock(&pd->lock);
+ return ret;
+}
+
+enum power_role get_pdphy_power_role(enum typec_role role)
+{
+ switch (role) {
+ case TYPEC_SINK:
+ return PR_SINK;
+ case TYPEC_SOURCE:
+ return PR_SRC;
+ default:
+ return PR_NONE;
+ }
+}
+
+static const char * const get_typec_role_name(enum typec_role role)
+{
+ switch (role) {
+ case TYPEC_SINK:
+ return "SINK";
+ case TYPEC_SOURCE:
+ return "SOURCE";
+ default:
+ return "UNDEFINED";
+ }
+}
+
+static const char * const get_pdphy_pr_name(enum power_role pdphy_pr)
+{
+ switch (pdphy_pr) {
+ case PR_SINK:
+ return "PR_SINK";
+ case PR_SRC:
+ return "PR_SRC";
+ case PR_NONE:
+ return "PR_NONE";
+ default:
+ return "UNDEFINED";
+ }
+}
+
+enum data_role get_pdphy_data_role(enum typec_data_role data)
+{
+ switch (data) {
+ case TYPEC_DEVICE:
+ return DR_UFP;
+ case TYPEC_HOST:
+ return DR_DFP;
+ default:
+ return DR_NONE;
+ }
+}
+
+static const char * const get_typec_data_role_name(enum typec_data_role data)
+{
+ switch (data) {
+ case TYPEC_DEVICE:
+ return "DEVICE";
+ case TYPEC_HOST:
+ return "HOST";
+ default:
+ return "UNDEFINED";
+ }
+}
+
+static const char * const get_pdphy_dr_name(enum data_role pdphy_dr)
+{
+ switch (pdphy_dr) {
+ case DR_UFP:
+ return "DR_UFP";
+ case DR_DFP:
+ return "DR_DFP";
+ case DR_NONE:
+ return "DR_NONE";
+ default:
+ return "UNDEFINED";
+ }
+}
+
+static int set_pd_header(struct usbpd *pd, enum typec_role role,
+ enum typec_data_role data)
+{
+ enum power_role pdphy_pr;
+ enum data_role pdphy_dr;
+ int ret = 0;
+
+ pdphy_pr = get_pdphy_power_role(role);
+ pdphy_dr = get_pdphy_data_role(data);
+ ret = pd_phy_update_roles(pdphy_dr, pdphy_pr);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to set pd_phy_header: %s, %s, ret=%d",
+ get_pdphy_pr_name(pdphy_pr),
+ get_pdphy_dr_name(pdphy_dr),
+ ret);
+ return ret;
+ }
+
+ pd_engine_log(pd, "set pd_phy_header: %s, %s",
+ get_pdphy_pr_name(pdphy_pr),
+ get_pdphy_dr_name(pdphy_dr));
+
+ return 0;
+}
+
+#define EXTCON_USB_SUPER_SPEED true
+static int set_usb_data_role(struct usbpd *pd, bool attached,
+ enum typec_role role,
+ enum typec_data_role data,
+ bool usb_comm_capable)
+{
+ int ret = 0;
+
+ pd->extcon_usb_cc = pd->is_cable_flipped;
+
+ if (!attached) {
+ pd->extcon_usb = false;
+ pd->extcon_usb_host = false;
+ } else if (data == TYPEC_HOST) {
+ pd->extcon_usb = false;
+ pd->extcon_usb_host = true;
+ } else {
+ pd->extcon_usb = true;
+ pd->extcon_usb_host = false;
+ }
+
+ pd->usb_comm_capable = usb_comm_capable;
+
+ pd_engine_log(pd,
+ "set usb_data_role: power [%s], data [%s], apsd_done [%s], attached [%s], comm [%s]",
+ get_typec_role_name(role),
+ get_typec_data_role_name(data),
+ pd->apsd_done ? "Y" : "N",
+ attached ? "Y" : "N",
+ usb_comm_capable ? "Y" : "N");
+
+ if (attached && role == TYPEC_SINK &&
+ !(pd->apsd_done || usb_comm_capable)) {
+ /* wait for APSD done */
+ pd_engine_log(pd,
+ "APSD is not done, delay update usb data role");
+ pd->pending_update_usb_data = true;
+ } else {
+ ret = update_usb_data_role(pd);
+ if (ret < 0)
+ return ret;
+ pd->pending_update_usb_data = false;
+ }
+
+ return ret;
+}
+
+static int tcpm_set_roles(struct tcpc_dev *dev, bool attached,
+ enum typec_role role, enum typec_data_role data,
+ bool usb_comm_capable)
+{
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+
+ mutex_lock(&pd->lock);
+
+ ret = set_pd_header(pd, role, data);
+ if (ret < 0)
+ goto unlock;
+
+ ret = set_usb_data_role(pd, attached, role, data, usb_comm_capable);
+
+ if (!ret) {
+ pd->cur_pwr_role = role;
+ pd->cur_data_role = data;
+ }
+unlock:
+ mutex_unlock(&pd->lock);
+ return ret;
+}
+
+static void pd_phy_signal_rx(struct usbpd *pd, enum pd_sig_type type)
+{
+ switch (type) {
+ case HARD_RESET_SIG:
+ tcpm_pd_hard_reset(pd->tcpm_port);
+ pd_engine_log(pd, "received pd hard reset");
+ break;
+ default:
+ pd_engine_log(pd, "received unsupported signal: %d",
+ type);
+ return;
+ }
+}
+
+static void pd_phy_message_rx(struct usbpd *pd, enum pd_sop_type sop,
+ u8 *buf, size_t len)
+{
+ struct pd_message msg;
+
+ if (len < 2) {
+ pd_engine_log(pd, "invalid message received, len=%ld",
+ len);
+ return;
+ }
+
+ if (sop != SOP_MSG) {
+ pd_engine_log(pd, "invalid msg type (%d) received; only SOP supported\n",
+ sop);
+ return;
+ }
+
+ msg.header = cpu_to_le16(*((u16 *)buf));
+ buf += sizeof(u16);
+ len -= sizeof(u16);
+ if (get_data_len(msg.header) != len) {
+ pd_engine_log(pd, "header len %d != len %ld",
+ get_data_len(msg.header), len);
+ return;
+ }
+ if (len > PD_MAX_PAYLOAD * 4) {
+ pd_engine_log(pd, "len %ld > PD_MAX_PAYLOAD", len);
+ return;
+ }
+ memcpy(msg.payload, buf, len);
+ pd_engine_log(pd, "pd rx header [%#x]", msg.header);
+ tcpm_pd_receive(pd->tcpm_port, &msg);
+}
+
+static void pd_phy_shutdown(struct usbpd *pd)
+{
+ int rc = 0;
+
+ flush_delayed_work(&pd->ext_vbus_work);
+ mutex_lock(&pd->lock);
+ if (regulator_is_enabled(pd->vbus)) {
+ rc = regulator_disable(pd->vbus);
+ if (rc < 0) {
+ pr_err("unable to disable vbus\n");
+ } else {
+ pd->vbus_output = false;
+ pd->smb2_vbus_reg = false;
+ }
+ }
+ if (pd->ext_vbus_reg) {
+ rc = regulator_disable(pd->ext_vbus);
+ if (rc < 0) {
+ pr_err("unable to disable vbus\n");
+ } else {
+ pd->vbus_output = false;
+ pd->ext_vbus_reg = false;
+ }
+ }
+ if (regulator_is_enabled(pd->vconn)) {
+ rc = regulator_disable(pd->vconn);
+ if (rc < 0)
+ pr_err("unable to disable vconn\n");
+ else
+ pd->vconn_output = false;
+ }
+ mutex_unlock(&pd->lock);
+
+ pd_engine_log(pd, "pd phy shutdown");
+}
+
+static void pd_phy_suspend(struct usbpd *pd)
+{
+ if (pd->first_suspend) {
+ /* Set to print UTC right away*/
+ pd->suspend_since_last_logged = true;
+ pd_engine_log(pd, "first suspend");
+ pd->first_suspend = false;
+ }
+
+ /* UTC gets printed for the next pd_engine_log */
+ pd->suspend_since_last_logged = true;
+}
+
+static void log_rtc(struct tcpc_dev *dev)
+{
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+
+ if (pd->suspend_since_last_logged) {
+ va_list args;
+
+ _pd_engine_log(pd, NULL, args, true);
+ }
+}
+
+static void set_pd_capable(struct tcpc_dev *dev, bool capable)
+{
+ union power_supply_propval val = {0};
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+
+ val.intval = capable ? 1 : 0;
+
+ if (pd->pd_capable == capable)
+ return;
+
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_PD_ACTIVE,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd, "unable to set pd capable to %s, ret=%d",
+ capable ? "true" : "false", ret);
+ }
+
+ pd->pd_capable = capable;
+}
+
+static void set_in_hard_reset(struct tcpc_dev *dev, bool status)
+{
+ union power_supply_propval val = {0};
+ struct usbpd *pd = container_of(dev, struct usbpd, tcpc_dev);
+ int ret = 0;
+
+ mutex_lock(&pd->lock);
+
+ if (status == pd->in_hard_reset)
+ goto unlock;
+
+ val.intval = status ? 1 : 0;
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_PD_IN_HARD_RESET,
+ &val);
+ if (ret < 0) {
+ pd_engine_log(pd,
+ "unable to set hard reset status to %s, ret=%d",
+ status ? "true" : "false", ret);
+ goto unlock;
+ }
+
+ switch (pd->cur_pwr_role) {
+ case TYPEC_SINK:
+ val.intval = POWER_SUPPLY_TYPEC_PR_SINK;
+ break;
+ case TYPEC_SOURCE:
+ val.intval = POWER_SUPPLY_TYPEC_PR_SOURCE;
+ break;
+ default:
+ goto unlock;
+ }
+ ret = power_supply_set_property(pd->usb_psy,
+ POWER_SUPPLY_PROP_TYPEC_POWER_ROLE,
+ &val);
+ if (ret < 0)
+ pd_engine_log(pd, "unable to set POWER_ROLE, ret=%d",
+ ret);
+
+ pd->in_hard_reset = status;
+
+unlock:
+ mutex_unlock(&pd->lock);
+}
+
+enum pdo_role {
+ SNK_PDO,
+ SRC_PDO,
+};
+
+static const char * const pdo_prop_name[] = {
+ [SNK_PDO] = "goog,snk-pdo",
+ [SRC_PDO] = "goog,src-pdo",
+};
+
+static u32 *parse_pdo(struct usbpd *pd, enum pdo_role role,
+ unsigned int *nr_pdo)
+{
+ struct device *dev = &pd->dev;
+ u32 *dt_array;
+ u32 *pdo;
+ int i, count, rc;
+
+ count = device_property_read_u32_array(dev->parent, pdo_prop_name[role],
+ NULL, 0);
+ if (count > 0) {
+ if (count % 4)
+ return ERR_PTR(-EINVAL);
+
+ *nr_pdo = count / 4;
+ dt_array = devm_kcalloc(dev, count, sizeof(*dt_array),
+ GFP_KERNEL);
+ if (!dt_array)
+ return ERR_PTR(-ENOMEM);
+
+ rc = device_property_read_u32_array(dev->parent,
+ pdo_prop_name[role],
+ dt_array, count);
+ if (rc)
+ return ERR_PTR(rc);
+
+ pdo = devm_kcalloc(dev, *nr_pdo, sizeof(*pdo), GFP_KERNEL);
+ if (!pdo)
+ return ERR_PTR(-ENOMEM);
+
+ for (i = 0; i < *nr_pdo; i++) {
+ switch (dt_array[i * 4]) {
+ case PDO_TYPE_FIXED:
+ pdo[i] = PDO_FIXED(dt_array[i * 4 + 1],
+ dt_array[i * 4 + 2],
+ PDO_FIXED_FLAGS);
+ break;
+ case PDO_TYPE_BATT:
+ pdo[i] = PDO_BATT(dt_array[i * 4 + 1],
+ dt_array[i * 4 + 2],
+ dt_array[i * 4 + 3]);
+ break;
+ case PDO_TYPE_VAR:
+ pdo[i] = PDO_VAR(dt_array[i * 4 + 1],
+ dt_array[i * 4 + 2],
+ dt_array[i * 4 + 3]);
+ break;
+ /*case PDO_TYPE_AUG:*/
+ default:
+ return ERR_PTR(-EINVAL);
+ }
+ }
+ return pdo;
+ }
+
+ return ERR_PTR(-EINVAL);
+}
+
+static int init_tcpc_config(struct tcpc_dev *pd_tcpc_dev)
+{
+ struct usbpd *pd = container_of(pd_tcpc_dev, struct usbpd, tcpc_dev);
+ struct device *dev = &pd->dev;
+ struct tcpc_config *config;
+ int ret;
+
+ pd_tcpc_dev->config = devm_kzalloc(dev, sizeof(*config), GFP_KERNEL);
+ if (!pd_tcpc_dev->config)
+ return -ENOMEM;
+
+ config = pd_tcpc_dev->config;
+
+ ret = device_property_read_u32(dev->parent, "goog,port-type",
+ &config->type);
+ if (ret < 0)
+ return ret;
+
+ switch (config->type) {
+ case TYPEC_PORT_UFP:
+ config->snk_pdo = parse_pdo(pd, SNK_PDO, &config->nr_snk_pdo);
+ if (IS_ERR(config->snk_pdo))
+ return PTR_ERR(config->snk_pdo);
+ break;
+ case TYPEC_PORT_DFP:
+ config->src_pdo = parse_pdo(pd, SRC_PDO, &config->nr_src_pdo);
+ if (IS_ERR(config->src_pdo))
+ return PTR_ERR(config->src_pdo);
+ break;
+ case TYPEC_PORT_DRP:
+ config->snk_pdo = parse_pdo(pd, SNK_PDO, &config->nr_snk_pdo);
+ if (IS_ERR(config->snk_pdo))
+ return PTR_ERR(config->snk_pdo);
+ config->src_pdo = parse_pdo(pd, SRC_PDO, &config->nr_src_pdo);
+ if (IS_ERR(config->src_pdo))
+ return PTR_ERR(config->src_pdo);
+
+ ret = device_property_read_u32(dev->parent, "goog,default-role",
+ &config->default_role);
+ if (ret < 0)
+ return ret;
+
+ config->try_role_hw = device_property_read_bool(
+ dev->parent,
+ "goog,try-role-hw");
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (config->type == TYPEC_PORT_UFP || config->type == TYPEC_PORT_DRP) {
+ ret = device_property_read_u32(dev->parent, "goog,max-snk-mv",
+ &config->max_snk_mv);
+ ret = device_property_read_u32(dev->parent, "goog,max-snk-ma",
+ &config->max_snk_ma);
+ ret = device_property_read_u32(dev->parent, "goog,max-snk-mw",
+ &config->max_snk_mw);
+ ret = device_property_read_u32(dev->parent, "goog,op-snk-mw",
+ &config->operating_snk_mw);
+ if (ret < 0)
+ return ret;
+ }
+
+ /* TODO: parse alt mode from DT */
+ config->alt_modes = NULL;
+ config->self_powered = true;
+
+ return 0;
+}
+
+static int init_tcpc_dev(struct tcpc_dev *pd_tcpc_dev)
+{
+ int ret;
+
+ ret = init_tcpc_config(pd_tcpc_dev);
+ if (ret < 0)
+ return ret;
+ pd_tcpc_dev->init = tcpm_init;
+ pd_tcpc_dev->get_vbus = tcpm_get_vbus;
+ pd_tcpc_dev->set_cc = tcpm_set_cc;
+ pd_tcpc_dev->get_cc = tcpm_get_cc;
+ pd_tcpc_dev->set_polarity = tcpm_set_polarity;
+ pd_tcpc_dev->set_vconn = tcpm_set_vconn;
+ pd_tcpc_dev->set_vbus = tcpm_set_vbus;
+ pd_tcpc_dev->set_current_limit = tcpm_set_current_limit;
+ pd_tcpc_dev->set_pd_rx = tcpm_set_pd_rx;
+ pd_tcpc_dev->set_roles = tcpm_set_roles;
+ pd_tcpc_dev->try_role = NULL;
+ pd_tcpc_dev->pd_transmit = tcpm_pd_transmit;
+ pd_tcpc_dev->start_drp_toggling = tcpm_start_drp_toggling;
+ pd_tcpc_dev->set_in_pr_swap = tcpm_set_in_pr_swap;
+ pd_tcpc_dev->set_pd_capable = set_pd_capable;
+ pd_tcpc_dev->set_in_hard_reset = set_in_hard_reset;
+ pd_tcpc_dev->log_rtc = log_rtc;
+ pd_tcpc_dev->set_suspend_supported = tcpm_set_suspend_supported;
+ pd_tcpc_dev->in_pd_contract = tcpm_in_pd_contract;
+ pd_tcpc_dev->mux = NULL;
+ return 0;
+}
+
+static void init_pd_phy_params(struct pd_phy_params *pdphy_params)
+{
+ pdphy_params->signal_cb = pd_phy_signal_rx;
+ pdphy_params->msg_rx_cb = pd_phy_message_rx;
+ pdphy_params->shutdown_cb = pd_phy_shutdown;
+ pdphy_params->suspend_cb = pd_phy_suspend;
+ pdphy_params->data_role = DR_UFP;
+ pdphy_params->power_role = PR_SINK;
+ pdphy_params->frame_filter_val = FRAME_FILTER_EN_SOP |
+ FRAME_FILTER_EN_HARD_RESET;
+}
+
+static int update_ext_vbus(struct notifier_block *self, unsigned long action,
+ void *dev)
+{
+ struct usbpd *pd = container_of(self, struct usbpd, ext_vbus_nb);
+ bool turn_on_ext_vbus = (action == EXT_VBUS_ON) ? true : false;
+ bool work_queued, work_cancelled;
+
+ work_cancelled = cancel_delayed_work_sync(&pd->ext_vbus_work);
+ pd_engine_log(pd, "ext_vbus_work_cancelled: %s", work_cancelled ? "yes"
+ : "no");
+ if (work_cancelled)
+ pm_relax(&pd->dev);
+
+ mutex_lock(&pd->lock);
+ pd->external_vbus_update = turn_on_ext_vbus;
+ work_queued = queue_delayed_work(pd->wq, &pd->ext_vbus_work,
+ turn_on_ext_vbus ?
+ msecs_to_jiffies(EXT_VBUS_WORK_DELAY_MS)
+ : 0);
+ if (!work_queued)
+ pd_engine_log(pd, "error: queueing ext_vbus_work failed");
+ else {
+ pm_stay_awake(&pd->dev);
+ pd_engine_log(pd, "queued work EXT_VBUS_%s",
+ (action == EXT_VBUS_ON) ?
+ "ON" : "OFF");
+ }
+ mutex_unlock(&pd->lock);
+ return NOTIFY_OK;
+}
+
+static const unsigned int usbpd_extcon_cable[] = {
+ EXTCON_USB,
+ EXTCON_USB_HOST,
+ EXTCON_NONE,
+};
+
+/* EXTCON_USB and EXTCON_USB_HOST are mutually exclusive */
+static const u32 usbpd_extcon_exclusive[] = {0x3, 0};
+
+static int num_pd_instances;
+/**
+ * usbpd_create - Create a new instance of USB PD protocol/policy engine
+ * @parent - parent device to associate with
+ *
+ * This creates a new usbpd class device which manages the state of a
+ * USB PD-capable port. The parent device that is passed in should be
+ * associated with the physical device port, e.g. a PD PHY.
+ *
+ * Derived from policy_engine.c.
+ *
+ * Return: struct usbpd pointer, or an ERR_PTR value
+ */
+struct usbpd *usbpd_create(struct device *parent)
+{
+ int ret;
+ struct usbpd *pd;
+
+ pd = kzalloc(sizeof(*pd), GFP_KERNEL);
+ if (!pd)
+ return ERR_PTR(-ENOMEM);
+
+ pd->logbuffer =
+ (u8 *) vzalloc(LOG_BUFFER_ENTRIES * LOG_BUFFER_ENTRY_SIZE);
+
+ if (!pd->logbuffer)
+ return ERR_PTR(-ENOMEM);
+
+ mutex_init(&pd->lock);
+
+ device_initialize(&pd->dev);
+ pd->dev.parent = parent;
+ dev_set_drvdata(&pd->dev, pd);
+
+ ret = dev_set_name(&pd->dev, "usbpd%d", num_pd_instances++);
+ if (ret < 0)
+ goto free_pd;
+
+ ret = device_add(&pd->dev);
+ if (ret < 0)
+ goto free_pd;
+
+ pd->wlc_supported = device_property_read_bool(parent,
+ "goog,wlc-supported");
+
+ ret = pd_engine_debugfs_init(pd);
+ if (ret < 0)
+ goto del_pd;
+
+ pd->first_suspend = true;
+
+ device_init_wakeup(&pd->dev, true);
+
+ pd->vbus = devm_regulator_get(parent, "vbus");
+ if (IS_ERR(pd->vbus)) {
+ ret = PTR_ERR(pd->vbus);
+ goto exit_debugfs;
+ }
+
+ pd->vconn = devm_regulator_get(parent, "vconn");
+ if (IS_ERR(pd->vconn)) {
+ ret = PTR_ERR(pd->vconn);
+ goto exit_debugfs;
+ }
+
+ pd->ext_vbus = devm_regulator_get(parent, "ext-vbus");
+ if (IS_ERR(pd->ext_vbus)) {
+ ret = PTR_ERR(pd->ext_vbus);
+ goto exit_debugfs;
+ }
+
+ pd->extcon = devm_extcon_dev_allocate(parent, usbpd_extcon_cable);
+ if (IS_ERR(pd->extcon)) {
+ pd_engine_log(pd, "failed to allocate extcon device");
+ ret = PTR_ERR(pd->extcon);
+ goto exit_debugfs;
+ }
+
+ pd->extcon->mutually_exclusive = usbpd_extcon_exclusive;
+ ret = devm_extcon_dev_register(parent, pd->extcon);
+ if (ret < 0) {
+ pd_engine_log(pd, "failed to register extcon device");
+ goto exit_debugfs;
+ }
+
+ /* Support reporting polarity and speed via properties */
+ extcon_set_property_capability(pd->extcon, EXTCON_USB,
+ EXTCON_PROP_USB_TYPEC_POLARITY);
+ extcon_set_property_capability(pd->extcon, EXTCON_USB,
+ EXTCON_PROP_USB_SS);
+ extcon_set_property_capability(pd->extcon, EXTCON_USB_HOST,
+ EXTCON_PROP_USB_TYPEC_POLARITY);
+ extcon_set_property_capability(pd->extcon, EXTCON_USB_HOST,
+ EXTCON_PROP_USB_SS);
+
+ pd->wq = create_singlethread_workqueue(dev_name(&pd->dev));
+ if (!pd->wq) {
+ ret = -ENOMEM;
+ goto exit_debugfs;
+ }
+
+ pd->default_src_cap = true;
+
+ INIT_DELAYED_WORK(&pd->ext_vbus_work, update_external_vbus);
+ INIT_WORK(&pd->update_pdo_work, update_src_caps);
+
+ pd->usb_psy = power_supply_get_by_name("usb");
+ if (!pd->usb_psy) {
+ pd_engine_log(pd,
+ "Could not get USB power_supply, deferring probe");
+ ret = -EPROBE_DEFER;
+ goto del_wq;
+ }
+
+ if (pd->wlc_supported) {
+ pd->wireless_psy = power_supply_get_by_name("wireless");
+ if (!pd->wireless_psy) {
+ pd_engine_log(pd,
+ "Could not get wireless power_supply, deferring probe");
+ ret = -EPROBE_DEFER;
+ goto put_psy_usb;
+ }
+ }
+
+ pd->usb_icl_votable = find_votable("USB_ICL");
+ if (pd->usb_icl_votable == NULL) {
+ pd_engine_log(pd,
+ "Couldn't find USB_ICL votable, deferring probe");
+ ret = -EPROBE_DEFER;
+ goto put_psy_wireless;
+ }
+
+ pd->apsd_disable_votable = find_votable("APSD_DISABLE");
+ if (pd->apsd_disable_votable == NULL) {
+ pd_engine_log(pd,
+ "Couldn't find APSD_DISABLE votable, deferring probe");
+ ret = -EPROBE_DEFER;
+ goto put_psy_wireless;
+ }
+
+ pd->disable_pr_switch = find_votable("DISABLE_POWER_ROLE_SWITCH");
+ if (pd->disable_pr_switch == NULL) {
+ pd_engine_log(pd,
+ "Couldn't find DISABLE_POWER_ROLE_SWITCH votable, deferring");
+ ret = -EPROBE_DEFER;
+ goto put_psy_wireless;
+ }
+
+ /* initialize votable */
+ vote(pd->usb_icl_votable, OTG_ICL_VOTER, false, 0);
+ vote(pd->apsd_disable_votable, OTG_DISABLE_APSD_VOTER, false, 0);
+ vote(pd->disable_pr_switch, DISABLE_CC_VOTER, false, 0);
+
+ pd->ext_vbus_nb.notifier_call = update_ext_vbus;
+ ext_vbus_register_notify(&pd->ext_vbus_nb);
+
+ /*
+ * TCPM callbacks may access pd->usb_psy. Therefore, tcpm_register_port
+ * must be called after pd->usb_psy is initialized.
+ */
+ ret = init_tcpc_dev(&pd->tcpc_dev);
+ if (ret < 0)
+ goto put_psy_wireless;
+ pd->tcpm_port = tcpm_register_port(&pd->dev, &pd->tcpc_dev);
+ if (IS_ERR(pd->tcpm_port)) {
+ ret = PTR_ERR(pd->tcpm_port);
+ goto put_psy_wireless;
+ }
+
+ psy_changed(&pd->psy_nb, PSY_EVENT_PROP_CHANGED, pd->usb_psy);
+
+ pd->psy_nb.notifier_call = psy_changed;
+ ret = power_supply_reg_notifier(&pd->psy_nb);
+ if (ret < 0)
+ goto unreg_tcpm;
+
+ init_pd_phy_params(&pd->pdphy_params);
+ pd_phy_assign_pm_callbacks(&pd->pdphy_params);
+
+ pd->suspend_supported = true;
+
+ return pd;
+
+unreg_tcpm:
+ tcpm_unregister_port(pd->tcpm_port);
+put_psy_wireless:
+ ext_vbus_unregister_notify(&pd->ext_vbus_nb);
+ if (pd->wlc_supported)
+ power_supply_put(pd->wireless_psy);
+put_psy_usb:
+ power_supply_put(pd->usb_psy);
+del_wq:
+ destroy_workqueue(pd->wq);
+exit_debugfs:
+ pd_engine_debugfs_exit(pd);
+del_pd:
+ device_del(&pd->dev);
+free_pd:
+ num_pd_instances--;
+ vfree(pd->logbuffer);
+ kfree(pd);
+ return ERR_PTR(ret);
+}
+EXPORT_SYMBOL(usbpd_create);
+
+/**
+ * usbpd_destroy - Removes and frees a usbpd instance
+ * @pd: the instance to destroy
+ *
+ * Derived from policy_engine.c.
+ */
+void usbpd_destroy(struct usbpd *pd)
+{
+ if (!pd)
+ return;
+ power_supply_unreg_notifier(&pd->psy_nb);
+ tcpm_unregister_port(pd->tcpm_port);
+ ext_vbus_unregister_notify(&pd->ext_vbus_nb);
+ if (pd->wlc_supported)
+ power_supply_put(pd->wireless_psy);
+ power_supply_put(pd->usb_psy);
+ destroy_workqueue(pd->wq);
+ pd_engine_debugfs_exit(pd);
+ device_del(&pd->dev);
+ num_pd_instances--;
+ vfree(pd->logbuffer);
+ kfree(pd);
+}
+EXPORT_SYMBOL(usbpd_destroy);
+
+static int __init pd_engine_init(void)
+{
+ return 0;
+}
+module_init(pd_engine_init);
+
+static void __exit pd_engine_exit(void) {}
+module_exit(pd_engine_exit);
+
+MODULE_DESCRIPTION("USB PD Engine based on Type-C Port Manager");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/usb/pd/qpnp-pdphy.c b/drivers/usb/pd/qpnp-pdphy.c
index 2997976..ac4eabc62 100644
--- a/drivers/usb/pd/qpnp-pdphy.c
+++ b/drivers/usb/pd/qpnp-pdphy.c
@@ -104,6 +104,8 @@ struct usb_pdphy {
void (*msg_rx_cb)(struct usbpd *pd, enum pd_sop_type sop,
u8 *buf, size_t len);
void (*shutdown_cb)(struct usbpd *pd);
+ void (*suspend_cb)(struct usbpd *pd);
+ void (*resume_cb)(struct usbpd *pd);
/* write waitq */
wait_queue_head_t tx_waitq;
@@ -338,6 +340,17 @@ int pd_phy_update_roles(enum data_role dr, enum power_role pr)
}
EXPORT_SYMBOL(pd_phy_update_roles);
+int pd_phy_assign_pm_callbacks(struct pd_phy_params *params)
+{
+ struct usb_pdphy *pdphy = __pdphy;
+
+ pdphy->suspend_cb = params->suspend_cb;
+ pdphy->resume_cb = params->resume_cb;
+
+ return 0;
+}
+EXPORT_SYMBOL(pd_phy_assign_pm_callbacks);
+
int pd_phy_open(struct pd_phy_params *params)
{
int ret;
@@ -582,6 +595,7 @@ static irqreturn_t pdphy_msg_tx_irq(int irq, void *data)
{
struct usb_pdphy *pdphy = data;
+ pm_wakeup_event(pdphy->dev, PD_ACTIVITY_TIMEOUT_MS);
/* TX already aborted by received signal */
if (pdphy->tx_status != -EINPROGRESS)
return IRQ_HANDLED;
@@ -609,6 +623,7 @@ static irqreturn_t pdphy_msg_rx_discarded_irq(int irq, void *data)
{
struct usb_pdphy *pdphy = data;
+ pm_wakeup_event(pdphy->dev, PD_ACTIVITY_TIMEOUT_MS);
pdphy->msg_rx_discarded_cnt++;
return IRQ_HANDLED;
@@ -620,6 +635,7 @@ static irqreturn_t pdphy_sig_rx_irq_thread(int irq, void *data)
int ret;
struct usb_pdphy *pdphy = data;
+ pm_wakeup_event(pdphy->dev, PD_ACTIVITY_TIMEOUT_MS);
pdphy->sig_rx_cnt++;
ret = pdphy_reg_read(pdphy, &rx_status, USB_PDPHY_RX_STATUS, 1);
@@ -651,6 +667,7 @@ static irqreturn_t pdphy_sig_tx_irq_thread(int irq, void *data)
{
struct usb_pdphy *pdphy = data;
+ pm_wakeup_event(pdphy->dev, PD_ACTIVITY_TIMEOUT_MS);
/* in case of exit from BIST Carrier Mode 2, clear BIST_MODE */
pdphy_reg_write(pdphy, USB_PDPHY_BIST_MODE, 0);
@@ -682,6 +699,7 @@ static irqreturn_t pdphy_msg_rx_irq(int irq, void *data)
int ret;
struct usb_pdphy *pdphy = data;
+ pm_wakeup_event(pdphy->dev, PD_ACTIVITY_TIMEOUT_MS);
pdphy->msg_rx_cnt++;
ret = pdphy_reg_read(pdphy, &size, USB_PDPHY_RX_SIZE, 1);
@@ -889,6 +907,32 @@ static void pdphy_shutdown(struct platform_device *pdev)
pdphy->shutdown_cb(pdphy->usbpd);
}
+#ifdef CONFIG_PM_SLEEP
+static int pdphy_pm_resume(struct device *dev)
+{
+ struct usb_pdphy *pdphy = dev_get_drvdata(dev);
+
+ if (pdphy->resume_cb)
+ pdphy->resume_cb(pdphy->usbpd);
+
+ return 0;
+}
+
+static int pdphy_pm_suspend(struct device *dev)
+{
+ struct usb_pdphy *pdphy = dev_get_drvdata(dev);
+
+ if (pdphy->suspend_cb)
+ pdphy->suspend_cb(pdphy->usbpd);
+
+ return 0;
+}
+
+static const struct dev_pm_ops pdphy_dev_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(pdphy_pm_suspend, pdphy_pm_resume)
+};
+#endif
+
static const struct of_device_id pdphy_match_table[] = {
{
.compatible = "qcom,qpnp-pdphy",
@@ -901,6 +945,9 @@ static struct platform_driver pdphy_driver = {
.driver = {
.name = "qpnp-pdphy",
.of_match_table = pdphy_match_table,
+#ifdef CONFIG_PM_SLEEP
+ .pm = &pdphy_dev_pm_ops,
+#endif
},
.probe = pdphy_probe,
.remove = pdphy_remove,
diff --git a/drivers/usb/pd/usbpd.h b/drivers/usb/pd/usbpd.h
index 9b6053e..8548a93 100644
--- a/drivers/usb/pd/usbpd.h
+++ b/drivers/usb/pd/usbpd.h
@@ -14,10 +14,13 @@
#define _USBPD_H
#include <linux/device.h>
+#include <linux/notifier.h>
+
+#define PD_ACTIVITY_TIMEOUT_MS 10000
struct usbpd;
-#if IS_ENABLED(CONFIG_USB_PD_POLICY)
+#if IS_ENABLED(CONFIG_USB_PD_POLICY) || IS_ENABLED(CONFIG_USB_PD_ENGINE)
struct usbpd *usbpd_create(struct device *parent);
void usbpd_destroy(struct usbpd *pd);
#else
@@ -58,20 +61,33 @@ enum pd_spec_rev {
/* enable msg and signal to be received by phy */
#define FRAME_FILTER_EN_SOP BIT(0)
+#define FRAME_FILTER_EN_SOPI BIT(1)
+#define FRAME_FILTER_EN_SOPII BIT(2)
+#define FRAME_FILTER_EN_SOPI_DEBUG BIT(3)
+#define FRAME_FILTER_EN_SOPII_DEBUG BIT(4)
#define FRAME_FILTER_EN_HARD_RESET BIT(5)
+#define FRAME_FILTER_EN_CABLE_RESET BIT(6)
struct pd_phy_params {
void (*signal_cb)(struct usbpd *pd, enum pd_sig_type sig);
void (*msg_rx_cb)(struct usbpd *pd, enum pd_sop_type sop,
u8 *buf, size_t len);
void (*shutdown_cb)(struct usbpd *pd);
+ void (*suspend_cb)(struct usbpd *pd);
+ void (*resume_cb)(struct usbpd *pd);
enum data_role data_role;
enum power_role power_role;
u8 frame_filter_val;
};
+#define EXT_VBUS_ON 1
+#define EXT_VBUS_OFF 0
+extern void ext_vbus_register_notify(struct notifier_block *nb);
+extern void ext_vbus_unregister_notify(struct notifier_block *nb);
+
#if IS_ENABLED(CONFIG_QPNP_USB_PDPHY)
int pd_phy_open(struct pd_phy_params *params);
+int pd_phy_assign_pm_callbacks(struct pd_phy_params *params);
int pd_phy_signal(enum pd_sig_type sig);
int pd_phy_write(u16 hdr, const u8 *data, size_t data_len,
enum pd_sop_type sop);
diff --git a/drivers/usb/phy/phy-fsl-usb.c b/drivers/usb/phy/phy-fsl-usb.c
index 94eb292..85d031c 100644
--- a/drivers/usb/phy/phy-fsl-usb.c
+++ b/drivers/usb/phy/phy-fsl-usb.c
@@ -879,6 +879,7 @@ int usb_otg_start(struct platform_device *pdev)
if (pdata->init && pdata->init(pdev) != 0)
return -EINVAL;
+#ifdef CONFIG_PPC32
if (pdata->big_endian_mmio) {
_fsl_readl = _fsl_readl_be;
_fsl_writel = _fsl_writel_be;
@@ -886,6 +887,7 @@ int usb_otg_start(struct platform_device *pdev)
_fsl_readl = _fsl_readl_le;
_fsl_writel = _fsl_writel_le;
}
+#endif
/* request irq */
p_otg->irq = platform_get_irq(pdev, 0);
@@ -976,7 +978,7 @@ int usb_otg_start(struct platform_device *pdev)
/*
* state file in sysfs
*/
-static int show_fsl_usb2_otg_state(struct device *dev,
+static ssize_t show_fsl_usb2_otg_state(struct device *dev,
struct device_attribute *attr, char *buf)
{
struct otg_fsm *fsm = &fsl_otg_dev->fsm;
diff --git a/drivers/usb/phy/phy-msm-qusb-v2.c b/drivers/usb/phy/phy-msm-qusb-v2.c
index e28173b..2f17f67 100644
--- a/drivers/usb/phy/phy-msm-qusb-v2.c
+++ b/drivers/usb/phy/phy-msm-qusb-v2.c
@@ -67,6 +67,7 @@
#define LINESTATE_DM BIT(1)
#define BIAS_CTRL_2_OVERRIDE_VAL 0x28
+#define BIAS_CTRL_2_POR_VAL 0x20
#define SQ_CTRL1_CHIRP_DISABLE 0x20
#define SQ_CTRL2_CHIRP_DISABLE 0x80
@@ -77,6 +78,26 @@
/* PERIPH_SS_PHY_REFGEN_NORTH_BG_CTRL register bits */
#define BANDGAP_BYPASS BIT(0)
+unsigned int phy_tune1;
+module_param(phy_tune1, uint, 0644);
+MODULE_PARM_DESC(phy_tune1, "QUSB PHY v2 TUNE1");
+
+unsigned int phy_tune2;
+module_param(phy_tune2, uint, 0644);
+MODULE_PARM_DESC(phy_tune2, "QUSB PHY v2 TUNE2");
+
+unsigned int phy_tune3;
+module_param(phy_tune3, uint, 0644);
+MODULE_PARM_DESC(phy_tune3, "QUSB PHY v2 TUNE3");
+
+unsigned int phy_tune4;
+module_param(phy_tune4, uint, 0644);
+MODULE_PARM_DESC(phy_tune4, "QUSB PHY v2 TUNE4");
+
+unsigned int phy_tune5;
+module_param(phy_tune5, uint, 0644);
+MODULE_PARM_DESC(phy_tune5, "QUSB PHY v2 TUNE5");
+
enum qusb_phy_reg {
PORT_TUNE1,
PLL_COMMON_STATUS_ONE,
@@ -84,6 +105,7 @@ enum qusb_phy_reg {
INTR_CTRL,
PLL_CORE_INPUT_OVERRIDE,
TEST1,
+ BIAS_CTRL_1,
BIAS_CTRL_2,
SQ_CTRL1,
SQ_CTRL2,
@@ -150,9 +172,15 @@ struct qusb_phy {
struct dentry *root;
u8 tune[5];
+ u8 bias_ctrl[2];
+
struct hrtimer timer;
int soc_min_rev;
bool host_chirp_erratum;
+ bool skip_efuse_reg;
+
+ unsigned int override_bias_ctrl2;
+ u8 default_bias_ctrl2;
};
#ifdef CONFIG_NVMEM
@@ -453,6 +481,28 @@ static void qusb_phy_write_seq(void __iomem *base, u32 *seq, int cnt,
}
}
+static void qusb_phy_check_bias_ctrl2_override(struct qusb_phy *qphy, u32 *seq,
+ int cnt)
+{
+ int i;
+ bool override = false;
+
+ for (i = 0; i < cnt; i = i + 2) {
+ if(seq[i + 1] == qphy->phy_reg[BIAS_CTRL_2]) {
+ pr_info("using PLL BIAS CTRL2 override 0x%02x\n",
+ seq[i]);
+ override = true;
+ break;
+ }
+ }
+
+ if(override && seq[i] < BIAS_CTRL_2_OVERRIDE_VAL) {
+ qphy->override_bias_ctrl2 = BIAS_CTRL_2_OVERRIDE_VAL - seq[i];
+ pr_info("override BIAS_CTRL_2 has offest of %d\n",
+ qphy->override_bias_ctrl2);
+ }
+}
+
static void qusb_phy_reset(struct qusb_phy *qphy)
{
int ret;
@@ -469,6 +519,43 @@ static void qusb_phy_reset(struct qusb_phy *qphy)
__func__);
}
+static void qusb_phy_set_pll_bias_ctrl2(struct qusb_phy *qphy)
+{
+ if (qphy->refgen_north_bg_reg){
+ if (readl_relaxed(qphy->refgen_north_bg_reg) & BANDGAP_BYPASS) {
+ writel_relaxed(BIAS_CTRL_2_OVERRIDE_VAL -
+ qphy->override_bias_ctrl2,
+ qphy->base +
+ qphy->phy_reg[BIAS_CTRL_2]);
+ } else {
+ u8 ctrl2_val;
+ if (qphy->override_bias_ctrl2 < 3)
+ ctrl2_val = qphy->override_bias_ctrl2;
+ else if (qphy->override_bias_ctrl2 >= 3 &&
+ qphy->override_bias_ctrl2 < 10)
+ ctrl2_val = qphy->override_bias_ctrl2 - 1;
+ else if (qphy->override_bias_ctrl2 >= 10 &&
+ qphy->override_bias_ctrl2 < 17)
+ ctrl2_val = qphy->override_bias_ctrl2 - 2;
+ else if (qphy->override_bias_ctrl2 >= 17 &&
+ qphy->override_bias_ctrl2 < 23)
+ ctrl2_val = qphy->override_bias_ctrl2 - 3;
+ else if (qphy->override_bias_ctrl2 >= 23 &&
+ qphy->override_bias_ctrl2 < 30)
+ ctrl2_val = qphy->override_bias_ctrl2 - 4;
+ else
+ ctrl2_val = qphy->override_bias_ctrl2 - 5;
+
+ pr_debug("%s(): programming BIAS CTRL2 as:%x\n",
+ __func__,
+ qphy->default_bias_ctrl2 - ctrl2_val);
+ writel_relaxed(qphy->default_bias_ctrl2 - ctrl2_val,
+ qphy->base + qphy->phy_reg[BIAS_CTRL_2]);
+ }
+ }
+
+}
+
static void qusb_phy_host_init(struct usb_phy *phy)
{
u8 reg;
@@ -489,11 +576,7 @@ static void qusb_phy_host_init(struct usb_phy *phy)
qphy->base + qphy->phy_reg[DEBUG_CTRL1]);
}
- if (qphy->refgen_north_bg_reg)
- if (readl_relaxed(qphy->refgen_north_bg_reg) & BANDGAP_BYPASS)
- writel_relaxed(BIAS_CTRL_2_OVERRIDE_VAL,
- qphy->base + qphy->phy_reg[BIAS_CTRL_2]);
-
+ qusb_phy_set_pll_bias_ctrl2(qphy);
/* Ensure above write is completed before turning ON ref clk */
wmb();
@@ -562,7 +645,7 @@ static int qusb_phy_init(struct usb_phy *phy)
if (qphy->qusb_phy_init_seq)
qusb_phy_write_seq(qphy->base, qphy->qusb_phy_init_seq,
qphy->init_seq_len, 0);
- if (qphy->efuse_reg) {
+ if (!qphy->skip_efuse_reg && qphy->efuse_reg) {
if (!qphy->tune_val)
qusb_phy_get_tune1_param(qphy);
@@ -572,6 +655,8 @@ static int qusb_phy_init(struct usb_phy *phy)
qphy->base + qphy->phy_reg[PORT_TUNE1]);
}
+ qusb_phy_set_pll_bias_ctrl2(qphy);
+
/* if debugfs based tunex params are set, use that value. */
for (p_index = 0; p_index < 5; p_index++) {
if (qphy->tune[p_index])
@@ -580,10 +665,12 @@ static int qusb_phy_init(struct usb_phy *phy)
(4 * p_index));
}
- if (qphy->refgen_north_bg_reg)
- if (readl_relaxed(qphy->refgen_north_bg_reg) & BANDGAP_BYPASS)
- writel_relaxed(BIAS_CTRL_2_OVERRIDE_VAL,
- qphy->base + qphy->phy_reg[BIAS_CTRL_2]);
+ for (p_index = 0; p_index < 2; p_index++) {
+ if (qphy->bias_ctrl[p_index])
+ writel_relaxed(qphy->bias_ctrl[p_index],
+ qphy->base + qphy->phy_reg[BIAS_CTRL_1] +
+ (4 * p_index));
+ }
/* ensure above writes are completed before re-enabling PHY */
wmb();
@@ -923,11 +1010,75 @@ static int qusb_phy_regulator_init(struct qusb_phy *qphy)
return 0;
}
+#define PHY_REG_SET(reg, index) \
+static int phy_##reg##_##index##_set(void *data, u64 val) \
+{ \
+ struct qusb_phy *qusb = (struct qusb_phy *)data; \
+ \
+ qusb->reg[index - 1] = val; \
+ return 0; \
+}
+
+PHY_REG_SET(tune, 1);
+PHY_REG_SET(tune, 2);
+PHY_REG_SET(tune, 3);
+PHY_REG_SET(tune, 4);
+PHY_REG_SET(tune, 5);
+PHY_REG_SET(bias_ctrl, 1);
+PHY_REG_SET(bias_ctrl, 2);
+
+#define PHY_REG_GET(reg, index, offset) \
+static int phy_##reg##_##index##_get(void *data, u64 *val) \
+{ \
+ struct qusb_phy *qusb = (struct qusb_phy *)data; \
+ \
+ *val = readl_relaxed( \
+ qusb->base + qusb->phy_reg[offset] + \
+ (4 * (index - 1))); \
+ return 0; \
+}
+
+PHY_REG_GET(tune, 1, PORT_TUNE1);
+PHY_REG_GET(tune, 2, PORT_TUNE1);
+PHY_REG_GET(tune, 3, PORT_TUNE1);
+PHY_REG_GET(tune, 4, PORT_TUNE1);
+PHY_REG_GET(tune, 5, PORT_TUNE1);
+PHY_REG_GET(bias_ctrl, 1, BIAS_CTRL_1);
+PHY_REG_GET(bias_ctrl, 2, BIAS_CTRL_1);
+
+DEFINE_SIMPLE_ATTRIBUTE(tune_1_fops, phy_tune_1_get, phy_tune_1_set,
+ "0x%llx\n");
+DEFINE_SIMPLE_ATTRIBUTE(tune_2_fops, phy_tune_2_get, phy_tune_2_set,
+ "0x%llx\n");
+DEFINE_SIMPLE_ATTRIBUTE(tune_3_fops, phy_tune_3_get, phy_tune_3_set,
+ "0x%llx\n");
+DEFINE_SIMPLE_ATTRIBUTE(tune_4_fops, phy_tune_4_get, phy_tune_4_set,
+ "0x%llx\n");
+DEFINE_SIMPLE_ATTRIBUTE(tune_5_fops, phy_tune_5_get, phy_tune_5_set,
+ "0x%llx\n");
+DEFINE_SIMPLE_ATTRIBUTE(bias_ctrl_1_fops, phy_bias_ctrl_1_get,
+ phy_bias_ctrl_1_set, "0x%llx\n");
+DEFINE_SIMPLE_ATTRIBUTE(bias_ctrl_2_fops, phy_bias_ctrl_2_get,
+ phy_bias_ctrl_2_set, "0x%llx\n");
+
+static struct reg_node {
+ char *name;
+ const struct file_operations *fops;
+} regs[] = {
+ { .name = "tune1", .fops = &tune_1_fops},
+ { .name = "tune2", .fops = &tune_2_fops},
+ { .name = "tune3", .fops = &tune_3_fops},
+ { .name = "tune4", .fops = &tune_4_fops},
+ { .name = "tune5", .fops = &tune_5_fops},
+ { .name = "bias_ctrl_1", .fops = &bias_ctrl_1_fops},
+ { .name = "bias_ctrl_2", .fops = &bias_ctrl_2_fops},
+};
+
static int qusb_phy_create_debugfs(struct qusb_phy *qphy)
{
struct dentry *file;
int ret = 0, i;
- char name[6];
+ char name[15];
qphy->root = debugfs_create_dir(dev_name(qphy->phy.dev), NULL);
if (IS_ERR_OR_NULL(qphy->root)) {
@@ -938,19 +1089,21 @@ static int qusb_phy_create_debugfs(struct qusb_phy *qphy)
goto create_err;
}
- for (i = 0; i < 5; i++) {
- snprintf(name, sizeof(name), "tune%d", (i + 1));
- file = debugfs_create_x8(name, 0644, qphy->root,
- &qphy->tune[i]);
+ for (i = 0; i < ARRAY_SIZE(regs); i++) {
+ file = debugfs_create_file(regs[i].name,
+ 0644,
+ qphy->root,
+ qphy,
+ regs[i].fops);
+
if (IS_ERR_OR_NULL(file)) {
dev_err(qphy->phy.dev,
"can't create debugfs entry for %s\n", name);
debugfs_remove_recursive(qphy->root);
- ret = ENOMEM;
+ ret = -ENOMEM;
goto create_err;
}
}
-
create_err:
return ret;
}
@@ -1012,6 +1165,8 @@ static int qusb_phy_probe(struct platform_device *pdev)
qphy->refgen_north_bg_reg = devm_ioremap(dev, res->start,
resource_size(res));
+ qphy->skip_efuse_reg = of_property_read_bool(dev->of_node,
+ "google,skip_efuse_reg");
/* ref_clk_src is needed irrespective of SE_CLK or DIFF_CLK usage */
qphy->ref_clk_src = devm_clk_get(dev, "ref_clk_src");
if (IS_ERR(qphy->ref_clk_src)) {
@@ -1164,12 +1319,21 @@ static int qusb_phy_probe(struct platform_device *pdev)
"qcom,qusb-phy-init-seq",
qphy->qusb_phy_init_seq,
qphy->init_seq_len);
+
+ qusb_phy_check_bias_ctrl2_override(qphy,
+ qphy->qusb_phy_init_seq,
+ qphy->init_seq_len);
} else {
dev_err(dev,
"error allocating memory for phy_init_seq\n");
}
}
+ of_property_read_u8(dev->of_node, "qcom,qusb-bias-ctrl2-default",
+ &qphy->default_bias_ctrl2);
+ if (!qphy->default_bias_ctrl2)
+ qphy->default_bias_ctrl2 = BIAS_CTRL_2_POR_VAL;
+
qphy->host_init_seq_len = of_property_count_elems_of_size(dev->of_node,
"qcom,qusb-phy-host-init-seq",
sizeof(*qphy->qusb_phy_host_init_seq));
diff --git a/drivers/usb/renesas_usbhs/mod_gadget.c b/drivers/usb/renesas_usbhs/mod_gadget.c
index 93fba90..5984fb1 100644
--- a/drivers/usb/renesas_usbhs/mod_gadget.c
+++ b/drivers/usb/renesas_usbhs/mod_gadget.c
@@ -639,14 +639,11 @@ static int usbhsg_ep_disable(struct usb_ep *ep)
struct usbhsg_uep *uep = usbhsg_ep_to_uep(ep);
struct usbhs_pipe *pipe;
unsigned long flags;
- int ret = 0;
spin_lock_irqsave(&uep->lock, flags);
pipe = usbhsg_uep_to_pipe(uep);
- if (!pipe) {
- ret = -EINVAL;
+ if (!pipe)
goto out;
- }
usbhsg_pipe_disable(uep);
usbhs_pipe_free(pipe);
@@ -1085,7 +1082,6 @@ int usbhs_mod_gadget_probe(struct usbhs_priv *priv)
ret = -ENOMEM;
goto usbhs_mod_gadget_probe_err_gpriv;
}
- spin_lock_init(&uep->lock);
gpriv->transceiver = usb_get_phy(USB_PHY_TYPE_UNDEFINED);
dev_info(dev, "%stransceiver found\n",
@@ -1135,6 +1131,7 @@ int usbhs_mod_gadget_probe(struct usbhs_priv *priv)
uep->ep.name = uep->ep_name;
uep->ep.ops = &usbhsg_ep_ops;
INIT_LIST_HEAD(&uep->ep.ep_list);
+ spin_lock_init(&uep->lock);
/* init DCP */
if (usbhsg_is_dcp(uep)) {
diff --git a/drivers/usb/serial/Kconfig b/drivers/usb/serial/Kconfig
index 584ae8c..77c3ebe 100644
--- a/drivers/usb/serial/Kconfig
+++ b/drivers/usb/serial/Kconfig
@@ -62,6 +62,7 @@
- Fundamental Software dongle.
- Google USB serial devices
- HP4x calculators
+ - Libtransistor USB console
- a number of Motorola phones
- Motorola Tetra devices
- Novatel Wireless GPS receivers
diff --git a/drivers/usb/serial/ch341.c b/drivers/usb/serial/ch341.c
index e98590a..9a2c0c7 100644
--- a/drivers/usb/serial/ch341.c
+++ b/drivers/usb/serial/ch341.c
@@ -118,7 +118,7 @@ static int ch341_control_in(struct usb_device *dev,
r = usb_control_msg(dev, usb_rcvctrlpipe(dev, 0), request,
USB_TYPE_VENDOR | USB_RECIP_DEVICE | USB_DIR_IN,
value, index, buf, bufsize, DEFAULT_TIMEOUT);
- if (r < bufsize) {
+ if (r < (int)bufsize) {
if (r >= 0) {
dev_err(&dev->dev,
"short control message received (%d < %u)\n",
diff --git a/drivers/usb/serial/cp210x.c b/drivers/usb/serial/cp210x.c
index cab80ac..c2b1200 100644
--- a/drivers/usb/serial/cp210x.c
+++ b/drivers/usb/serial/cp210x.c
@@ -33,7 +33,7 @@ static int cp210x_open(struct tty_struct *tty, struct usb_serial_port *);
static void cp210x_close(struct usb_serial_port *);
static void cp210x_get_termios(struct tty_struct *, struct usb_serial_port *);
static void cp210x_get_termios_port(struct usb_serial_port *port,
- unsigned int *cflagp, unsigned int *baudp);
+ tcflag_t *cflagp, unsigned int *baudp);
static void cp210x_change_speed(struct tty_struct *, struct usb_serial_port *,
struct ktermios *);
static void cp210x_set_termios(struct tty_struct *, struct usb_serial_port *,
@@ -92,6 +92,9 @@ static const struct usb_device_id id_table[] = {
{ USB_DEVICE(0x10C4, 0x8156) }, /* B&G H3000 link cable */
{ USB_DEVICE(0x10C4, 0x815E) }, /* Helicomm IP-Link 1220-DVM */
{ USB_DEVICE(0x10C4, 0x815F) }, /* Timewave HamLinkUSB */
+ { USB_DEVICE(0x10C4, 0x817C) }, /* CESINEL MEDCAL N Power Quality Monitor */
+ { USB_DEVICE(0x10C4, 0x817D) }, /* CESINEL MEDCAL NT Power Quality Monitor */
+ { USB_DEVICE(0x10C4, 0x817E) }, /* CESINEL MEDCAL S Power Quality Monitor */
{ USB_DEVICE(0x10C4, 0x818B) }, /* AVIT Research USB to TTL */
{ USB_DEVICE(0x10C4, 0x819F) }, /* MJS USB Toslink Switcher */
{ USB_DEVICE(0x10C4, 0x81A6) }, /* ThinkOptics WavIt */
@@ -109,6 +112,9 @@ static const struct usb_device_id id_table[] = {
{ USB_DEVICE(0x10C4, 0x826B) }, /* Cygnal Integrated Products, Inc., Fasttrax GPS demonstration module */
{ USB_DEVICE(0x10C4, 0x8281) }, /* Nanotec Plug & Drive */
{ USB_DEVICE(0x10C4, 0x8293) }, /* Telegesis ETRX2USB */
+ { USB_DEVICE(0x10C4, 0x82EF) }, /* CESINEL FALCO 6105 AC Power Supply */
+ { USB_DEVICE(0x10C4, 0x82F1) }, /* CESINEL MEDCAL EFD Earth Fault Detector */
+ { USB_DEVICE(0x10C4, 0x82F2) }, /* CESINEL MEDCAL ST Network Analyzer */
{ USB_DEVICE(0x10C4, 0x82F4) }, /* Starizona MicroTouch */
{ USB_DEVICE(0x10C4, 0x82F9) }, /* Procyon AVS */
{ USB_DEVICE(0x10C4, 0x8341) }, /* Siemens MC35PU GPRS Modem */
@@ -121,7 +127,9 @@ static const struct usb_device_id id_table[] = {
{ USB_DEVICE(0x10C4, 0x8470) }, /* Juniper Networks BX Series System Console */
{ USB_DEVICE(0x10C4, 0x8477) }, /* Balluff RFID */
{ USB_DEVICE(0x10C4, 0x84B6) }, /* Starizona Hyperion */
+ { USB_DEVICE(0x10C4, 0x851E) }, /* CESINEL MEDCAL PT Network Analyzer */
{ USB_DEVICE(0x10C4, 0x85A7) }, /* LifeScan OneTouch Verio IQ */
+ { USB_DEVICE(0x10C4, 0x85B8) }, /* CESINEL ReCon T Energy Logger */
{ USB_DEVICE(0x10C4, 0x85EA) }, /* AC-Services IBUS-IF */
{ USB_DEVICE(0x10C4, 0x85EB) }, /* AC-Services CIS-IBUS */
{ USB_DEVICE(0x10C4, 0x85F8) }, /* Virtenio Preon32 */
@@ -131,17 +139,24 @@ static const struct usb_device_id id_table[] = {
{ USB_DEVICE(0x10C4, 0x8857) }, /* CEL EM357 ZigBee USB Stick */
{ USB_DEVICE(0x10C4, 0x88A4) }, /* MMB Networks ZigBee USB Device */
{ USB_DEVICE(0x10C4, 0x88A5) }, /* Planet Innovation Ingeni ZigBee USB Device */
+ { USB_DEVICE(0x10C4, 0x88FB) }, /* CESINEL MEDCAL STII Network Analyzer */
+ { USB_DEVICE(0x10C4, 0x8938) }, /* CESINEL MEDCAL S II Network Analyzer */
{ USB_DEVICE(0x10C4, 0x8946) }, /* Ketra N1 Wireless Interface */
{ USB_DEVICE(0x10C4, 0x8962) }, /* Brim Brothers charging dock */
{ USB_DEVICE(0x10C4, 0x8977) }, /* CEL MeshWorks DevKit Device */
{ USB_DEVICE(0x10C4, 0x8998) }, /* KCF Technologies PRN */
+ { USB_DEVICE(0x10C4, 0x89A4) }, /* CESINEL FTBC Flexible Thyristor Bridge Controller */
+ { USB_DEVICE(0x10C4, 0x89FB) }, /* Qivicon ZigBee USB Radio Stick */
{ USB_DEVICE(0x10C4, 0x8A2A) }, /* HubZ dual ZigBee and Z-Wave dongle */
{ USB_DEVICE(0x10C4, 0x8A5E) }, /* CEL EM3588 ZigBee USB Stick Long Range */
{ USB_DEVICE(0x10C4, 0x8B34) }, /* Qivicon ZigBee USB Radio Stick */
{ USB_DEVICE(0x10C4, 0xEA60) }, /* Silicon Labs factory default */
{ USB_DEVICE(0x10C4, 0xEA61) }, /* Silicon Labs factory default */
+ { USB_DEVICE(0x10C4, 0xEA63) }, /* Silicon Labs Windows Update (CP2101-4/CP2102N) */
{ USB_DEVICE(0x10C4, 0xEA70) }, /* Silicon Labs factory default */
{ USB_DEVICE(0x10C4, 0xEA71) }, /* Infinity GPS-MIC-1 Radio Monophone */
+ { USB_DEVICE(0x10C4, 0xEA7A) }, /* Silicon Labs Windows Update (CP2105) */
+ { USB_DEVICE(0x10C4, 0xEA7B) }, /* Silicon Labs Windows Update (CP2108) */
{ USB_DEVICE(0x10C4, 0xF001) }, /* Elan Digital Systems USBscope50 */
{ USB_DEVICE(0x10C4, 0xF002) }, /* Elan Digital Systems USBwave12 */
{ USB_DEVICE(0x10C4, 0xF003) }, /* Elan Digital Systems USBpulse100 */
@@ -211,6 +226,7 @@ static const struct usb_device_id id_table[] = {
{ USB_DEVICE(0x3195, 0xF190) }, /* Link Instruments MSO-19 */
{ USB_DEVICE(0x3195, 0xF280) }, /* Link Instruments MSO-28 */
{ USB_DEVICE(0x3195, 0xF281) }, /* Link Instruments MSO-28 */
+ { USB_DEVICE(0x3923, 0x7A0B) }, /* National Instruments USB Serial Console */
{ USB_DEVICE(0x413C, 0x9500) }, /* DW700 GPS USB interface */
{ } /* Terminating Entry */
};
@@ -727,7 +743,7 @@ static void cp210x_get_termios(struct tty_struct *tty,
&tty->termios.c_cflag, &baud);
tty_encode_baud_rate(tty, baud, baud);
} else {
- unsigned int cflag;
+ tcflag_t cflag;
cflag = 0;
cp210x_get_termios_port(port, &cflag, &baud);
}
@@ -738,10 +754,10 @@ static void cp210x_get_termios(struct tty_struct *tty,
* This is the heart of cp210x_get_termios which always uses a &usb_serial_port.
*/
static void cp210x_get_termios_port(struct usb_serial_port *port,
- unsigned int *cflagp, unsigned int *baudp)
+ tcflag_t *cflagp, unsigned int *baudp)
{
struct device *dev = &port->dev;
- unsigned int cflag;
+ tcflag_t cflag;
struct cp210x_flow_ctl flow_ctl;
u32 baud;
u16 bits;
diff --git a/drivers/usb/serial/ftdi_sio.c b/drivers/usb/serial/ftdi_sio.c
index 71cbc68..2e2f7363 100644
--- a/drivers/usb/serial/ftdi_sio.c
+++ b/drivers/usb/serial/ftdi_sio.c
@@ -1911,7 +1911,8 @@ static int ftdi_8u2232c_probe(struct usb_serial *serial)
return ftdi_jtag_probe(serial);
if (udev->product &&
- (!strcmp(udev->product, "BeagleBone/XDS100V2") ||
+ (!strcmp(udev->product, "Arrow USB Blaster") ||
+ !strcmp(udev->product, "BeagleBone/XDS100V2") ||
!strcmp(udev->product, "SNAP Connect E10")))
return ftdi_jtag_probe(serial);
diff --git a/drivers/usb/serial/io_ti.h b/drivers/usb/serial/io_ti.h
index 1bd67b2..bc9ff5e 100644
--- a/drivers/usb/serial/io_ti.h
+++ b/drivers/usb/serial/io_ti.h
@@ -178,7 +178,7 @@ struct ump_interrupt {
} __attribute__((packed));
-#define TIUMP_GET_PORT_FROM_CODE(c) (((c) >> 4) - 3)
+#define TIUMP_GET_PORT_FROM_CODE(c) (((c) >> 6) & 0x01)
#define TIUMP_GET_FUNC_FROM_CODE(c) ((c) & 0x0f)
#define TIUMP_INTERRUPT_CODE_LSR 0x03
#define TIUMP_INTERRUPT_CODE_MSR 0x04
diff --git a/drivers/usb/serial/keyspan_pda.c b/drivers/usb/serial/keyspan_pda.c
index d2dab2a..d17f787 100644
--- a/drivers/usb/serial/keyspan_pda.c
+++ b/drivers/usb/serial/keyspan_pda.c
@@ -373,8 +373,10 @@ static int keyspan_pda_get_modem_info(struct usb_serial *serial,
3, /* get pins */
USB_TYPE_VENDOR|USB_RECIP_INTERFACE|USB_DIR_IN,
0, 0, data, 1, 2000);
- if (rc >= 0)
+ if (rc == 1)
*value = *data;
+ else if (rc >= 0)
+ rc = -EIO;
kfree(data);
return rc;
diff --git a/drivers/usb/serial/kobil_sct.c b/drivers/usb/serial/kobil_sct.c
index 813035f..7d252678 100644
--- a/drivers/usb/serial/kobil_sct.c
+++ b/drivers/usb/serial/kobil_sct.c
@@ -408,12 +408,20 @@ static int kobil_tiocmget(struct tty_struct *tty)
transfer_buffer_length,
KOBIL_TIMEOUT);
- dev_dbg(&port->dev, "%s - Send get_status_line_state URB returns: %i. Statusline: %02x\n",
- __func__, result, transfer_buffer[0]);
+ dev_dbg(&port->dev, "Send get_status_line_state URB returns: %i\n",
+ result);
+ if (result < 1) {
+ if (result >= 0)
+ result = -EIO;
+ goto out_free;
+ }
+
+ dev_dbg(&port->dev, "Statusline: %02x\n", transfer_buffer[0]);
result = 0;
if ((transfer_buffer[0] & SUSBCR_GSL_DSR) != 0)
result = TIOCM_DSR;
+out_free:
kfree(transfer_buffer);
return result;
}
diff --git a/drivers/usb/serial/mos7840.c b/drivers/usb/serial/mos7840.c
index 6baacf6..03d63ba 100644
--- a/drivers/usb/serial/mos7840.c
+++ b/drivers/usb/serial/mos7840.c
@@ -482,6 +482,9 @@ static void mos7840_control_callback(struct urb *urb)
}
dev_dbg(dev, "%s urb buffer size is %d\n", __func__, urb->actual_length);
+ if (urb->actual_length < 1)
+ goto out;
+
dev_dbg(dev, "%s mos7840_port->MsrLsr is %d port %d\n", __func__,
mos7840_port->MsrLsr, mos7840_port->port_num);
data = urb->transfer_buffer;
diff --git a/drivers/usb/serial/option.c b/drivers/usb/serial/option.c
index 1799aa0..7bc2c9f 100644
--- a/drivers/usb/serial/option.c
+++ b/drivers/usb/serial/option.c
@@ -199,6 +199,8 @@ static void option_instat_callback(struct urb *urb);
#define DELL_PRODUCT_5800_V2_MINICARD_VZW 0x8196 /* Novatel E362 */
#define DELL_PRODUCT_5804_MINICARD_ATT 0x819b /* Novatel E371 */
+#define DELL_PRODUCT_5821E 0x81d7
+
#define KYOCERA_VENDOR_ID 0x0c88
#define KYOCERA_PRODUCT_KPC650 0x17da
#define KYOCERA_PRODUCT_KPC680 0x180a
@@ -236,6 +238,8 @@ static void option_instat_callback(struct urb *urb);
/* These Quectel products use Qualcomm's vendor ID */
#define QUECTEL_PRODUCT_UC20 0x9003
#define QUECTEL_PRODUCT_UC15 0x9090
+/* These u-blox products use Qualcomm's vendor ID */
+#define UBLOX_PRODUCT_R410M 0x90b2
/* These Yuga products use Qualcomm's vendor ID */
#define YUGA_PRODUCT_CLM920_NC5 0x9625
@@ -244,6 +248,7 @@ static void option_instat_callback(struct urb *urb);
#define QUECTEL_PRODUCT_EC21 0x0121
#define QUECTEL_PRODUCT_EC25 0x0125
#define QUECTEL_PRODUCT_BG96 0x0296
+#define QUECTEL_PRODUCT_EP06 0x0306
#define CMOTECH_VENDOR_ID 0x16d8
#define CMOTECH_PRODUCT_6001 0x6001
@@ -550,147 +555,15 @@ static void option_instat_callback(struct urb *urb);
#define WETELECOM_PRODUCT_6802 0x6802
#define WETELECOM_PRODUCT_WMD300 0x6803
-struct option_blacklist_info {
- /* bitmask of interface numbers blacklisted for send_setup */
- const unsigned long sendsetup;
- /* bitmask of interface numbers that are reserved */
- const unsigned long reserved;
-};
-static const struct option_blacklist_info four_g_w14_blacklist = {
- .sendsetup = BIT(0) | BIT(1),
-};
+/* Device flags */
-static const struct option_blacklist_info four_g_w100_blacklist = {
- .sendsetup = BIT(1) | BIT(2),
- .reserved = BIT(3),
-};
+/* Interface does not support modem-control requests */
+#define NCTRL(ifnum) ((BIT(ifnum) & 0xff) << 8)
-static const struct option_blacklist_info alcatel_x200_blacklist = {
- .sendsetup = BIT(0) | BIT(1),
- .reserved = BIT(4),
-};
+/* Interface is reserved */
+#define RSVD(ifnum) ((BIT(ifnum) & 0xff) << 0)
-static const struct option_blacklist_info zte_0037_blacklist = {
- .sendsetup = BIT(0) | BIT(1),
-};
-
-static const struct option_blacklist_info zte_k3765_z_blacklist = {
- .sendsetup = BIT(0) | BIT(1) | BIT(2),
- .reserved = BIT(4),
-};
-
-static const struct option_blacklist_info zte_ad3812_z_blacklist = {
- .sendsetup = BIT(0) | BIT(1) | BIT(2),
-};
-
-static const struct option_blacklist_info zte_mc2718_z_blacklist = {
- .sendsetup = BIT(1) | BIT(2) | BIT(3) | BIT(4),
-};
-
-static const struct option_blacklist_info zte_mc2716_z_blacklist = {
- .sendsetup = BIT(1) | BIT(2) | BIT(3),
-};
-
-static const struct option_blacklist_info zte_me3620_mbim_blacklist = {
- .reserved = BIT(2) | BIT(3) | BIT(4),
-};
-
-static const struct option_blacklist_info zte_me3620_xl_blacklist = {
- .reserved = BIT(3) | BIT(4) | BIT(5),
-};
-
-static const struct option_blacklist_info zte_zm8620_x_blacklist = {
- .reserved = BIT(3) | BIT(4) | BIT(5),
-};
-
-static const struct option_blacklist_info huawei_cdc12_blacklist = {
- .reserved = BIT(1) | BIT(2),
-};
-
-static const struct option_blacklist_info net_intf0_blacklist = {
- .reserved = BIT(0),
-};
-
-static const struct option_blacklist_info net_intf1_blacklist = {
- .reserved = BIT(1),
-};
-
-static const struct option_blacklist_info net_intf2_blacklist = {
- .reserved = BIT(2),
-};
-
-static const struct option_blacklist_info net_intf3_blacklist = {
- .reserved = BIT(3),
-};
-
-static const struct option_blacklist_info net_intf4_blacklist = {
- .reserved = BIT(4),
-};
-
-static const struct option_blacklist_info net_intf5_blacklist = {
- .reserved = BIT(5),
-};
-
-static const struct option_blacklist_info net_intf6_blacklist = {
- .reserved = BIT(6),
-};
-
-static const struct option_blacklist_info zte_mf626_blacklist = {
- .sendsetup = BIT(0) | BIT(1),
- .reserved = BIT(4),
-};
-
-static const struct option_blacklist_info zte_1255_blacklist = {
- .reserved = BIT(3) | BIT(4),
-};
-
-static const struct option_blacklist_info simcom_sim7100e_blacklist = {
- .reserved = BIT(5) | BIT(6),
-};
-
-static const struct option_blacklist_info telit_me910_blacklist = {
- .sendsetup = BIT(0),
- .reserved = BIT(1) | BIT(3),
-};
-
-static const struct option_blacklist_info telit_me910_dual_modem_blacklist = {
- .sendsetup = BIT(0),
- .reserved = BIT(3),
-};
-
-static const struct option_blacklist_info telit_le910_blacklist = {
- .sendsetup = BIT(0),
- .reserved = BIT(1) | BIT(2),
-};
-
-static const struct option_blacklist_info telit_le920_blacklist = {
- .sendsetup = BIT(0),
- .reserved = BIT(1) | BIT(5),
-};
-
-static const struct option_blacklist_info telit_le920a4_blacklist_1 = {
- .sendsetup = BIT(0),
- .reserved = BIT(1),
-};
-
-static const struct option_blacklist_info telit_le922_blacklist_usbcfg0 = {
- .sendsetup = BIT(2),
- .reserved = BIT(0) | BIT(1) | BIT(3),
-};
-
-static const struct option_blacklist_info telit_le922_blacklist_usbcfg3 = {
- .sendsetup = BIT(0),
- .reserved = BIT(1) | BIT(2) | BIT(3),
-};
-
-static const struct option_blacklist_info cinterion_rmnet2_blacklist = {
- .reserved = BIT(4) | BIT(5),
-};
-
-static const struct option_blacklist_info yuga_clm920_nc5_blacklist = {
- .reserved = BIT(1) | BIT(4),
-};
static const struct usb_device_id option_ids[] = {
{ USB_DEVICE(OPTION_VENDOR_ID, OPTION_PRODUCT_COLT) },
@@ -724,26 +597,26 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE(QUANTA_VENDOR_ID, QUANTA_PRODUCT_GKE) },
{ USB_DEVICE(QUANTA_VENDOR_ID, QUANTA_PRODUCT_GLE) },
{ USB_DEVICE(QUANTA_VENDOR_ID, 0xea42),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, 0x1c05, USB_CLASS_COMM, 0x02, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, 0x1c1f, USB_CLASS_COMM, 0x02, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, 0x1c23, USB_CLASS_COMM, 0x02, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, HUAWEI_PRODUCT_E173, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t) &net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, HUAWEI_PRODUCT_E173S6, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t) &net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, HUAWEI_PRODUCT_E1750, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t) &net_intf2_blacklist },
+ .driver_info = RSVD(2) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, 0x1441, USB_CLASS_COMM, 0x02, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, 0x1442, USB_CLASS_COMM, 0x02, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, HUAWEI_PRODUCT_K4505, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t) &huawei_cdc12_blacklist },
+ .driver_info = RSVD(1) | RSVD(2) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, HUAWEI_PRODUCT_K3765, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t) &huawei_cdc12_blacklist },
+ .driver_info = RSVD(1) | RSVD(2) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, 0x14ac, 0xff, 0xff, 0xff), /* Huawei E1820 */
- .driver_info = (kernel_ulong_t) &net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, HUAWEI_PRODUCT_K4605, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t) &huawei_cdc12_blacklist },
+ .driver_info = RSVD(1) | RSVD(2) },
{ USB_VENDOR_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, 0xff, 0xff, 0xff) },
{ USB_VENDOR_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, 0xff, 0x01, 0x01) },
{ USB_VENDOR_AND_INTERFACE_INFO(HUAWEI_VENDOR_ID, 0xff, 0x01, 0x02) },
@@ -1162,6 +1035,8 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(DELL_VENDOR_ID, DELL_PRODUCT_5800_MINICARD_VZW, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(DELL_VENDOR_ID, DELL_PRODUCT_5800_V2_MINICARD_VZW, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(DELL_VENDOR_ID, DELL_PRODUCT_5804_MINICARD_ATT, 0xff, 0xff, 0xff) },
+ { USB_DEVICE(DELL_VENDOR_ID, DELL_PRODUCT_5821E),
+ .driver_info = RSVD(0) | RSVD(1) | RSVD(6) },
{ USB_DEVICE(ANYDATA_VENDOR_ID, ANYDATA_PRODUCT_ADU_E100A) }, /* ADU-E100, ADU-310 */
{ USB_DEVICE(ANYDATA_VENDOR_ID, ANYDATA_PRODUCT_ADU_500A) },
{ USB_DEVICE(ANYDATA_VENDOR_ID, ANYDATA_PRODUCT_ADU_620UW) },
@@ -1188,65 +1063,70 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE(KYOCERA_VENDOR_ID, KYOCERA_PRODUCT_KPC680) },
{ USB_DEVICE(QUALCOMM_VENDOR_ID, 0x6000)}, /* ZTE AC8700 */
{ USB_DEVICE_AND_INTERFACE_INFO(QUALCOMM_VENDOR_ID, 0x6001, 0xff, 0xff, 0xff), /* 4G LTE usb-modem U901 */
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE(QUALCOMM_VENDOR_ID, 0x6613)}, /* Onda H600/ZTE MF330 */
{ USB_DEVICE(QUALCOMM_VENDOR_ID, 0x0023)}, /* ONYX 3G device */
{ USB_DEVICE(QUALCOMM_VENDOR_ID, 0x9000)}, /* SIMCom SIM5218 */
/* Quectel products using Qualcomm vendor ID */
{ USB_DEVICE(QUALCOMM_VENDOR_ID, QUECTEL_PRODUCT_UC15)},
{ USB_DEVICE(QUALCOMM_VENDOR_ID, QUECTEL_PRODUCT_UC20),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
/* Yuga products use Qualcomm vendor ID */
{ USB_DEVICE(QUALCOMM_VENDOR_ID, YUGA_PRODUCT_CLM920_NC5),
- .driver_info = (kernel_ulong_t)&yuga_clm920_nc5_blacklist },
+ .driver_info = RSVD(1) | RSVD(4) },
+ /* u-blox products using Qualcomm vendor ID */
+ { USB_DEVICE(QUALCOMM_VENDOR_ID, UBLOX_PRODUCT_R410M),
+ .driver_info = RSVD(1) | RSVD(3) },
/* Quectel products using Quectel vendor ID */
{ USB_DEVICE(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EC21),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EC25),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_BG96),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
+ { USB_DEVICE(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EP06),
+ .driver_info = RSVD(4) | RSVD(5) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_6001) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CMU_300) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_6003),
- .driver_info = (kernel_ulong_t)&net_intf0_blacklist },
+ .driver_info = RSVD(0) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_6004) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_6005) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CGU_628A) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CHE_628S),
- .driver_info = (kernel_ulong_t)&net_intf0_blacklist },
+ .driver_info = RSVD(0) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CMU_301),
- .driver_info = (kernel_ulong_t)&net_intf0_blacklist },
+ .driver_info = RSVD(0) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CHU_628),
- .driver_info = (kernel_ulong_t)&net_intf0_blacklist },
+ .driver_info = RSVD(0) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CHU_628S) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CDU_680) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CDU_685A) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CHU_720S),
- .driver_info = (kernel_ulong_t)&net_intf0_blacklist },
+ .driver_info = RSVD(0) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_7002),
- .driver_info = (kernel_ulong_t)&net_intf0_blacklist },
+ .driver_info = RSVD(0) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CHU_629K),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_7004),
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_7005) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CGU_629),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CHU_629S),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_CHU_720I),
- .driver_info = (kernel_ulong_t)&net_intf0_blacklist },
+ .driver_info = RSVD(0) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_7212),
- .driver_info = (kernel_ulong_t)&net_intf0_blacklist },
+ .driver_info = RSVD(0) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_7213),
- .driver_info = (kernel_ulong_t)&net_intf0_blacklist },
+ .driver_info = RSVD(0) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_7251),
- .driver_info = (kernel_ulong_t)&net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_7252),
- .driver_info = (kernel_ulong_t)&net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE(CMOTECH_VENDOR_ID, CMOTECH_PRODUCT_7253),
- .driver_info = (kernel_ulong_t)&net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_UC864E) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_UC864G) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_CC864_DUAL) },
@@ -1254,38 +1134,42 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_DE910_DUAL) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_UE910_V2) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE922_USBCFG0),
- .driver_info = (kernel_ulong_t)&telit_le922_blacklist_usbcfg0 },
+ .driver_info = RSVD(0) | RSVD(1) | NCTRL(2) | RSVD(3) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE922_USBCFG1),
- .driver_info = (kernel_ulong_t)&telit_le910_blacklist },
+ .driver_info = NCTRL(0) | RSVD(1) | RSVD(2) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE922_USBCFG2),
- .driver_info = (kernel_ulong_t)&telit_le922_blacklist_usbcfg3 },
+ .driver_info = NCTRL(0) | RSVD(1) | RSVD(2) | RSVD(3) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE922_USBCFG3),
- .driver_info = (kernel_ulong_t)&telit_le922_blacklist_usbcfg3 },
+ .driver_info = NCTRL(0) | RSVD(1) | RSVD(2) | RSVD(3) },
{ USB_DEVICE_INTERFACE_CLASS(TELIT_VENDOR_ID, TELIT_PRODUCT_LE922_USBCFG5, 0xff),
- .driver_info = (kernel_ulong_t)&telit_le922_blacklist_usbcfg0 },
+ .driver_info = RSVD(0) | RSVD(1) | NCTRL(2) | RSVD(3) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_ME910),
- .driver_info = (kernel_ulong_t)&telit_me910_blacklist },
+ .driver_info = NCTRL(0) | RSVD(1) | RSVD(3) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_ME910_DUAL_MODEM),
- .driver_info = (kernel_ulong_t)&telit_me910_dual_modem_blacklist },
+ .driver_info = NCTRL(0) | RSVD(3) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE910),
- .driver_info = (kernel_ulong_t)&telit_le910_blacklist },
+ .driver_info = NCTRL(0) | RSVD(1) | RSVD(2) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE910_USBCFG4),
- .driver_info = (kernel_ulong_t)&telit_le922_blacklist_usbcfg3 },
+ .driver_info = NCTRL(0) | RSVD(1) | RSVD(2) | RSVD(3) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE920),
- .driver_info = (kernel_ulong_t)&telit_le920_blacklist },
+ .driver_info = NCTRL(0) | RSVD(1) | RSVD(5) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE920A4_1207) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE920A4_1208),
- .driver_info = (kernel_ulong_t)&telit_le920a4_blacklist_1 },
+ .driver_info = NCTRL(0) | RSVD(1) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE920A4_1211),
- .driver_info = (kernel_ulong_t)&telit_le922_blacklist_usbcfg3 },
+ .driver_info = NCTRL(0) | RSVD(1) | RSVD(2) | RSVD(3) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE920A4_1212),
- .driver_info = (kernel_ulong_t)&telit_le920a4_blacklist_1 },
+ .driver_info = NCTRL(0) | RSVD(1) },
{ USB_DEVICE_INTERFACE_CLASS(TELIT_VENDOR_ID, TELIT_PRODUCT_LE920A4_1213, 0xff) },
{ USB_DEVICE(TELIT_VENDOR_ID, TELIT_PRODUCT_LE920A4_1214),
- .driver_info = (kernel_ulong_t)&telit_le922_blacklist_usbcfg3 },
+ .driver_info = NCTRL(0) | RSVD(1) | RSVD(2) | RSVD(3) },
+ { USB_DEVICE(TELIT_VENDOR_ID, 0x1900), /* Telit LN940 (QMI) */
+ .driver_info = NCTRL(0) | RSVD(1) },
+ { USB_DEVICE_INTERFACE_CLASS(TELIT_VENDOR_ID, 0x1901, 0xff), /* Telit LN940 (MBIM) */
+ .driver_info = NCTRL(0) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, ZTE_PRODUCT_MF622, 0xff, 0xff, 0xff) }, /* ZTE WCDMA products */
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0002, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0003, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0004, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0005, 0xff, 0xff, 0xff) },
@@ -1301,58 +1185,58 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0010, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0011, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0012, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0013, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, ZTE_PRODUCT_MF628, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0016, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0017, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0018, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0019, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0020, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0021, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0022, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0023, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0024, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0025, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0028, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0029, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0030, 0xff, 0xff, 0xff) },
- { USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, ZTE_PRODUCT_MF626, 0xff,
- 0xff, 0xff), .driver_info = (kernel_ulong_t)&zte_mf626_blacklist },
+ { USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, ZTE_PRODUCT_MF626, 0xff, 0xff, 0xff),
+ .driver_info = NCTRL(0) | NCTRL(1) | RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0032, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0033, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0034, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0037, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&zte_0037_blacklist },
+ .driver_info = NCTRL(0) | NCTRL(1) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0038, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0039, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0040, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0042, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0043, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0044, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0048, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0049, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0050, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0051, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0052, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0054, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0055, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0056, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0057, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0058, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0061, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0062, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0063, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0064, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0065, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0066, 0xff, 0xff, 0xff) },
@@ -1377,26 +1261,26 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0096, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0097, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0104, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0105, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0106, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0108, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0113, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0117, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0118, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0121, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0122, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0123, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0124, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0125, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf6_blacklist },
+ .driver_info = RSVD(6) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0126, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0128, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0135, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0136, 0xff, 0xff, 0xff) },
@@ -1412,50 +1296,51 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0155, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0156, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0157, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0158, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0159, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0161, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0162, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0164, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0165, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0167, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0189, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0191, 0xff, 0xff, 0xff), /* ZTE EuFi890 */
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0196, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0197, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0199, 0xff, 0xff, 0xff), /* ZTE MF820S */
- .driver_info = (kernel_ulong_t)&net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0200, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0201, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0254, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0257, 0xff, 0xff, 0xff), /* ZTE MF821 */
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0265, 0xff, 0xff, 0xff), /* ONDA MT8205 */
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0284, 0xff, 0xff, 0xff), /* ZTE MF880 */
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0317, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0326, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0330, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0395, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0412, 0xff, 0xff, 0xff), /* Telewell TW-LTE 4G */
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0414, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0417, 0xff, 0xff, 0xff) },
+ { USB_DEVICE_INTERFACE_CLASS(ZTE_VENDOR_ID, 0x0602, 0xff) }, /* GosunCn ZTE WeLink ME3630 (MBIM mode) */
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1008, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1010, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1012, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1018, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1021, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf2_blacklist },
+ .driver_info = RSVD(2) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1057, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1058, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1059, 0xff, 0xff, 0xff) },
@@ -1572,23 +1457,23 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1170, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1244, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1245, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1246, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1247, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1248, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1249, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1250, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1251, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1252, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1253, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1254, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1255, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&zte_1255_blacklist },
+ .driver_info = RSVD(3) | RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1256, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1257, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1258, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1259, 0xff, 0xff, 0xff) },
@@ -1603,7 +1488,7 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1268, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1269, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1270, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1271, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1272, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1273, 0xff, 0xff, 0xff) },
@@ -1639,17 +1524,18 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1303, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1333, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1401, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf2_blacklist },
+ .driver_info = RSVD(2) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1402, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf2_blacklist },
+ .driver_info = RSVD(2) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1424, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf2_blacklist },
+ .driver_info = RSVD(2) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1425, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf2_blacklist },
+ .driver_info = RSVD(2) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1426, 0xff, 0xff, 0xff), /* ZTE MF91 */
- .driver_info = (kernel_ulong_t)&net_intf2_blacklist },
+ .driver_info = RSVD(2) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1428, 0xff, 0xff, 0xff), /* Telewell TW-LTE 4G v2 */
- .driver_info = (kernel_ulong_t)&net_intf2_blacklist },
+ .driver_info = RSVD(2) },
+ { USB_DEVICE_INTERFACE_CLASS(ZTE_VENDOR_ID, 0x1476, 0xff) }, /* GosunCn ZTE WeLink ME3630 (ECM/NCM mode) */
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1533, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1534, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1535, 0xff, 0xff, 0xff) },
@@ -1667,8 +1553,8 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1596, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1598, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x1600, 0xff, 0xff, 0xff) },
- { USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x2002, 0xff,
- 0xff, 0xff), .driver_info = (kernel_ulong_t)&zte_k3765_z_blacklist },
+ { USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x2002, 0xff, 0xff, 0xff),
+ .driver_info = NCTRL(0) | NCTRL(1) | NCTRL(2) | RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x2003, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0014, 0xff, 0xff, 0xff) }, /* ZTE CDMA products */
@@ -1679,20 +1565,20 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0073, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0094, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0130, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf1_blacklist },
+ .driver_info = RSVD(1) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0133, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0141, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0147, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0152, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0168, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0170, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0176, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0x0178, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0xff42, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0xff43, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0xff44, 0xff, 0xff, 0xff) },
@@ -1844,19 +1730,19 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, ZTE_PRODUCT_AC2726, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, ZTE_PRODUCT_AC8710T, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, ZTE_PRODUCT_MC2718, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&zte_mc2718_z_blacklist },
+ .driver_info = NCTRL(1) | NCTRL(2) | NCTRL(3) | NCTRL(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, ZTE_PRODUCT_AD3812, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&zte_ad3812_z_blacklist },
+ .driver_info = NCTRL(0) | NCTRL(1) | NCTRL(2) },
{ USB_DEVICE_AND_INTERFACE_INFO(ZTE_VENDOR_ID, ZTE_PRODUCT_MC2716, 0xff, 0xff, 0xff),
- .driver_info = (kernel_ulong_t)&zte_mc2716_z_blacklist },
+ .driver_info = NCTRL(1) | NCTRL(2) | NCTRL(3) },
{ USB_DEVICE(ZTE_VENDOR_ID, ZTE_PRODUCT_ME3620_L),
- .driver_info = (kernel_ulong_t)&zte_me3620_xl_blacklist },
+ .driver_info = RSVD(3) | RSVD(4) | RSVD(5) },
{ USB_DEVICE(ZTE_VENDOR_ID, ZTE_PRODUCT_ME3620_MBIM),
- .driver_info = (kernel_ulong_t)&zte_me3620_mbim_blacklist },
+ .driver_info = RSVD(2) | RSVD(3) | RSVD(4) },
{ USB_DEVICE(ZTE_VENDOR_ID, ZTE_PRODUCT_ME3620_X),
- .driver_info = (kernel_ulong_t)&zte_me3620_xl_blacklist },
+ .driver_info = RSVD(3) | RSVD(4) | RSVD(5) },
{ USB_DEVICE(ZTE_VENDOR_ID, ZTE_PRODUCT_ZM8620_X),
- .driver_info = (kernel_ulong_t)&zte_zm8620_x_blacklist },
+ .driver_info = RSVD(3) | RSVD(4) | RSVD(5) },
{ USB_VENDOR_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0xff, 0x02, 0x01) },
{ USB_VENDOR_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0xff, 0x02, 0x05) },
{ USB_VENDOR_AND_INTERFACE_INFO(ZTE_VENDOR_ID, 0xff, 0x86, 0x10) },
@@ -1876,37 +1762,35 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE(ALINK_VENDOR_ID, ALINK_PRODUCT_PH300) },
{ USB_DEVICE_AND_INTERFACE_INFO(ALINK_VENDOR_ID, ALINK_PRODUCT_3GU, 0xff, 0xff, 0xff) },
{ USB_DEVICE(ALINK_VENDOR_ID, SIMCOM_PRODUCT_SIM7100E),
- .driver_info = (kernel_ulong_t)&simcom_sim7100e_blacklist },
+ .driver_info = RSVD(5) | RSVD(6) },
+ { USB_DEVICE_INTERFACE_CLASS(0x1e0e, 0x9003, 0xff) }, /* Simcom SIM7500/SIM7600 MBIM mode */
{ USB_DEVICE(ALCATEL_VENDOR_ID, ALCATEL_PRODUCT_X060S_X200),
- .driver_info = (kernel_ulong_t)&alcatel_x200_blacklist
- },
+ .driver_info = NCTRL(0) | NCTRL(1) | RSVD(4) },
{ USB_DEVICE(ALCATEL_VENDOR_ID, ALCATEL_PRODUCT_X220_X500D),
- .driver_info = (kernel_ulong_t)&net_intf6_blacklist },
+ .driver_info = RSVD(6) },
{ USB_DEVICE(ALCATEL_VENDOR_ID, 0x0052),
- .driver_info = (kernel_ulong_t)&net_intf6_blacklist },
+ .driver_info = RSVD(6) },
{ USB_DEVICE(ALCATEL_VENDOR_ID, 0x00b6),
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE(ALCATEL_VENDOR_ID, 0x00b7),
- .driver_info = (kernel_ulong_t)&net_intf5_blacklist },
+ .driver_info = RSVD(5) },
{ USB_DEVICE(ALCATEL_VENDOR_ID, ALCATEL_PRODUCT_L100V),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(ALCATEL_VENDOR_ID, ALCATEL_PRODUCT_L800MA),
- .driver_info = (kernel_ulong_t)&net_intf2_blacklist },
+ .driver_info = RSVD(2) },
{ USB_DEVICE(AIRPLUS_VENDOR_ID, AIRPLUS_PRODUCT_MCD650) },
{ USB_DEVICE(TLAYTECH_VENDOR_ID, TLAYTECH_PRODUCT_TEU800) },
{ USB_DEVICE(LONGCHEER_VENDOR_ID, FOUR_G_SYSTEMS_PRODUCT_W14),
- .driver_info = (kernel_ulong_t)&four_g_w14_blacklist
- },
+ .driver_info = NCTRL(0) | NCTRL(1) },
{ USB_DEVICE(LONGCHEER_VENDOR_ID, FOUR_G_SYSTEMS_PRODUCT_W100),
- .driver_info = (kernel_ulong_t)&four_g_w100_blacklist
- },
+ .driver_info = NCTRL(1) | NCTRL(2) | RSVD(3) },
{USB_DEVICE(LONGCHEER_VENDOR_ID, FUJISOFT_PRODUCT_FS040U),
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist},
+ .driver_info = RSVD(3)},
{ USB_DEVICE_INTERFACE_CLASS(LONGCHEER_VENDOR_ID, SPEEDUP_PRODUCT_SU9800, 0xff) },
{ USB_DEVICE_INTERFACE_CLASS(LONGCHEER_VENDOR_ID, 0x9801, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf3_blacklist },
+ .driver_info = RSVD(3) },
{ USB_DEVICE_INTERFACE_CLASS(LONGCHEER_VENDOR_ID, 0x9803, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(LONGCHEER_VENDOR_ID, ZOOM_PRODUCT_4597) },
{ USB_DEVICE(LONGCHEER_VENDOR_ID, IBALL_3_5G_CONNECT) },
{ USB_DEVICE(HAIER_VENDOR_ID, HAIER_PRODUCT_CE100) },
@@ -1932,14 +1816,14 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE(CINTERION_VENDOR_ID, CINTERION_PRODUCT_EU3_E) },
{ USB_DEVICE(CINTERION_VENDOR_ID, CINTERION_PRODUCT_EU3_P) },
{ USB_DEVICE(CINTERION_VENDOR_ID, CINTERION_PRODUCT_PH8),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_INTERFACE_CLASS(CINTERION_VENDOR_ID, CINTERION_PRODUCT_AHXX, 0xff) },
{ USB_DEVICE(CINTERION_VENDOR_ID, CINTERION_PRODUCT_PLXX),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_INTERFACE_CLASS(CINTERION_VENDOR_ID, CINTERION_PRODUCT_PH8_2RMNET, 0xff),
- .driver_info = (kernel_ulong_t)&cinterion_rmnet2_blacklist },
+ .driver_info = RSVD(4) | RSVD(5) },
{ USB_DEVICE_INTERFACE_CLASS(CINTERION_VENDOR_ID, CINTERION_PRODUCT_PH8_AUDIO, 0xff),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_INTERFACE_CLASS(CINTERION_VENDOR_ID, CINTERION_PRODUCT_AHXX_2RMNET, 0xff) },
{ USB_DEVICE_INTERFACE_CLASS(CINTERION_VENDOR_ID, CINTERION_PRODUCT_AHXX_AUDIO, 0xff) },
{ USB_DEVICE(CINTERION_VENDOR_ID, CINTERION_PRODUCT_HC28_MDM) },
@@ -1949,20 +1833,20 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE(SIEMENS_VENDOR_ID, CINTERION_PRODUCT_HC28_MDM) }, /* HC28 enumerates with Siemens or Cinterion VID depending on FW revision */
{ USB_DEVICE(SIEMENS_VENDOR_ID, CINTERION_PRODUCT_HC28_MDMNET) },
{ USB_DEVICE(OLIVETTI_VENDOR_ID, OLIVETTI_PRODUCT_OLICARD100),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(OLIVETTI_VENDOR_ID, OLIVETTI_PRODUCT_OLICARD120),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(OLIVETTI_VENDOR_ID, OLIVETTI_PRODUCT_OLICARD140),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(OLIVETTI_VENDOR_ID, OLIVETTI_PRODUCT_OLICARD145) },
{ USB_DEVICE(OLIVETTI_VENDOR_ID, OLIVETTI_PRODUCT_OLICARD155),
- .driver_info = (kernel_ulong_t)&net_intf6_blacklist },
+ .driver_info = RSVD(6) },
{ USB_DEVICE(OLIVETTI_VENDOR_ID, OLIVETTI_PRODUCT_OLICARD200),
- .driver_info = (kernel_ulong_t)&net_intf6_blacklist },
+ .driver_info = RSVD(6) },
{ USB_DEVICE(OLIVETTI_VENDOR_ID, OLIVETTI_PRODUCT_OLICARD160),
- .driver_info = (kernel_ulong_t)&net_intf6_blacklist },
+ .driver_info = RSVD(6) },
{ USB_DEVICE(OLIVETTI_VENDOR_ID, OLIVETTI_PRODUCT_OLICARD500),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(CELOT_VENDOR_ID, CELOT_PRODUCT_CT680M) }, /* CT-650 CDMA 450 1xEVDO modem */
{ USB_DEVICE_AND_INTERFACE_INFO(SAMSUNG_VENDOR_ID, SAMSUNG_PRODUCT_GT_B3730, USB_CLASS_CDC_DATA, 0x00, 0x00) }, /* Samsung GT-B3730 LTE USB modem.*/
{ USB_DEVICE(YUGA_VENDOR_ID, YUGA_PRODUCT_CEM600) },
@@ -2039,9 +1923,9 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE(PETATEL_VENDOR_ID, PETATEL_PRODUCT_NP10T_600E) },
{ USB_DEVICE_AND_INTERFACE_INFO(TPLINK_VENDOR_ID, TPLINK_PRODUCT_LTE, 0xff, 0x00, 0x00) }, /* TP-Link LTE Module */
{ USB_DEVICE(TPLINK_VENDOR_ID, TPLINK_PRODUCT_MA180),
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(TPLINK_VENDOR_ID, 0x9000), /* TP-Link MA260 */
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE(CHANGHONG_VENDOR_ID, CHANGHONG_PRODUCT_CH690) },
{ USB_DEVICE_AND_INTERFACE_INFO(0x2001, 0x7d01, 0xff, 0x02, 0x01) }, /* D-Link DWM-156 (variant) */
{ USB_DEVICE_AND_INTERFACE_INFO(0x2001, 0x7d01, 0xff, 0x00, 0x00) }, /* D-Link DWM-156 (variant) */
@@ -2052,9 +1936,9 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_INTERFACE_CLASS(0x2001, 0x7d04, 0xff) }, /* D-Link DWM-158 */
{ USB_DEVICE_INTERFACE_CLASS(0x2001, 0x7d0e, 0xff) }, /* D-Link DWM-157 C1 */
{ USB_DEVICE_INTERFACE_CLASS(0x2001, 0x7e19, 0xff), /* D-Link DWM-221 B1 */
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_INTERFACE_CLASS(0x2001, 0x7e35, 0xff), /* D-Link DWM-222 */
- .driver_info = (kernel_ulong_t)&net_intf4_blacklist },
+ .driver_info = RSVD(4) },
{ USB_DEVICE_AND_INTERFACE_INFO(0x07d1, 0x3e01, 0xff, 0xff, 0xff) }, /* D-Link DWM-152/C1 */
{ USB_DEVICE_AND_INTERFACE_INFO(0x07d1, 0x3e02, 0xff, 0xff, 0xff) }, /* D-Link DWM-156/C1 */
{ USB_DEVICE_AND_INTERFACE_INFO(0x07d1, 0x7e11, 0xff, 0xff, 0xff) }, /* D-Link DWM-156/A3 */
@@ -2064,7 +1948,18 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(WETELECOM_VENDOR_ID, WETELECOM_PRODUCT_WMD200, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(WETELECOM_VENDOR_ID, WETELECOM_PRODUCT_6802, 0xff, 0xff, 0xff) },
{ USB_DEVICE_AND_INTERFACE_INFO(WETELECOM_VENDOR_ID, WETELECOM_PRODUCT_WMD300, 0xff, 0xff, 0xff) },
- { USB_DEVICE_AND_INTERFACE_INFO(0x03f0, 0x421d, 0xff, 0xff, 0xff) }, /* HP lt2523 (Novatel E371) */
+ { USB_DEVICE_AND_INTERFACE_INFO(0x03f0, 0x421d, 0xff, 0xff, 0xff) }, /* HP lt2523 (Novatel E371) */
+ { USB_DEVICE_AND_INTERFACE_INFO(0x03f0, 0xa31d, 0xff, 0x06, 0x10) }, /* HP lt4132 (Huawei ME906s-158) */
+ { USB_DEVICE_AND_INTERFACE_INFO(0x03f0, 0xa31d, 0xff, 0x06, 0x12) },
+ { USB_DEVICE_AND_INTERFACE_INFO(0x03f0, 0xa31d, 0xff, 0x06, 0x13) },
+ { USB_DEVICE_AND_INTERFACE_INFO(0x03f0, 0xa31d, 0xff, 0x06, 0x14) },
+ { USB_DEVICE_AND_INTERFACE_INFO(0x03f0, 0xa31d, 0xff, 0x06, 0x1b) },
+ { USB_DEVICE(0x1508, 0x1001), /* Fibocom NL668 */
+ .driver_info = RSVD(4) | RSVD(5) | RSVD(6) },
+ { USB_DEVICE(0x2cb7, 0x0104), /* Fibocom NL678 series */
+ .driver_info = RSVD(4) | RSVD(5) },
+ { USB_DEVICE_INTERFACE_CLASS(0x2cb7, 0x0105, 0xff), /* Fibocom NL678 series */
+ .driver_info = RSVD(6) },
{ } /* Terminating entry */
};
MODULE_DEVICE_TABLE(usb, option_ids);
@@ -2114,7 +2009,7 @@ static int option_probe(struct usb_serial *serial,
struct usb_interface_descriptor *iface_desc =
&serial->interface->cur_altsetting->desc;
struct usb_device_descriptor *dev_desc = &serial->dev->descriptor;
- const struct option_blacklist_info *blacklist;
+ unsigned long device_flags = id->driver_info;
/* Never bind to the CD-Rom emulation interface */
if (iface_desc->bInterfaceClass == 0x08)
@@ -2125,9 +2020,7 @@ static int option_probe(struct usb_serial *serial,
* the same class/subclass/protocol as the serial interfaces. Look at
* the Windows driver .INF files for reserved interface numbers.
*/
- blacklist = (void *)id->driver_info;
- if (blacklist && test_bit(iface_desc->bInterfaceNumber,
- &blacklist->reserved))
+ if (device_flags & RSVD(iface_desc->bInterfaceNumber))
return -ENODEV;
/*
* Don't bind network interface on Samsung GT-B3730, it is handled by
@@ -2138,8 +2031,8 @@ static int option_probe(struct usb_serial *serial,
iface_desc->bInterfaceClass != USB_CLASS_CDC_DATA)
return -ENODEV;
- /* Store the blacklist info so we can use it during attach. */
- usb_set_serial_data(serial, (void *)blacklist);
+ /* Store the device flags so we can use them during attach. */
+ usb_set_serial_data(serial, (void *)device_flags);
return 0;
}
@@ -2147,22 +2040,21 @@ static int option_probe(struct usb_serial *serial,
static int option_attach(struct usb_serial *serial)
{
struct usb_interface_descriptor *iface_desc;
- const struct option_blacklist_info *blacklist;
struct usb_wwan_intf_private *data;
+ unsigned long device_flags;
data = kzalloc(sizeof(struct usb_wwan_intf_private), GFP_KERNEL);
if (!data)
return -ENOMEM;
- /* Retrieve blacklist info stored at probe. */
- blacklist = usb_get_serial_data(serial);
+ /* Retrieve device flags stored at probe. */
+ device_flags = (unsigned long)usb_get_serial_data(serial);
iface_desc = &serial->interface->cur_altsetting->desc;
- if (!blacklist || !test_bit(iface_desc->bInterfaceNumber,
- &blacklist->sendsetup)) {
+ if (!(device_flags & NCTRL(iface_desc->bInterfaceNumber)))
data->use_send_setup = 1;
- }
+
spin_lock_init(&data->susp_lock);
usb_set_serial_data(serial, data);
diff --git a/drivers/usb/serial/pl2303.c b/drivers/usb/serial/pl2303.c
index 3da25ad..4966768 100644
--- a/drivers/usb/serial/pl2303.c
+++ b/drivers/usb/serial/pl2303.c
@@ -86,9 +86,14 @@ static const struct usb_device_id id_table[] = {
{ USB_DEVICE(YCCABLE_VENDOR_ID, YCCABLE_PRODUCT_ID) },
{ USB_DEVICE(SUPERIAL_VENDOR_ID, SUPERIAL_PRODUCT_ID) },
{ USB_DEVICE(HP_VENDOR_ID, HP_LD220_PRODUCT_ID) },
+ { USB_DEVICE(HP_VENDOR_ID, HP_LD220TA_PRODUCT_ID) },
{ USB_DEVICE(HP_VENDOR_ID, HP_LD960_PRODUCT_ID) },
+ { USB_DEVICE(HP_VENDOR_ID, HP_LD960TA_PRODUCT_ID) },
{ USB_DEVICE(HP_VENDOR_ID, HP_LCM220_PRODUCT_ID) },
{ USB_DEVICE(HP_VENDOR_ID, HP_LCM960_PRODUCT_ID) },
+ { USB_DEVICE(HP_VENDOR_ID, HP_LM920_PRODUCT_ID) },
+ { USB_DEVICE(HP_VENDOR_ID, HP_LM940_PRODUCT_ID) },
+ { USB_DEVICE(HP_VENDOR_ID, HP_TD620_PRODUCT_ID) },
{ USB_DEVICE(CRESSI_VENDOR_ID, CRESSI_EDY_PRODUCT_ID) },
{ USB_DEVICE(ZEAGLE_VENDOR_ID, ZEAGLE_N2ITION3_PRODUCT_ID) },
{ USB_DEVICE(SONY_VENDOR_ID, SONY_QN3USB_PRODUCT_ID) },
diff --git a/drivers/usb/serial/pl2303.h b/drivers/usb/serial/pl2303.h
index 1232890..a84f095 100644
--- a/drivers/usb/serial/pl2303.h
+++ b/drivers/usb/serial/pl2303.h
@@ -123,10 +123,15 @@
/* Hewlett-Packard POS Pole Displays */
#define HP_VENDOR_ID 0x03f0
+#define HP_LM920_PRODUCT_ID 0x026b
+#define HP_TD620_PRODUCT_ID 0x0956
#define HP_LD960_PRODUCT_ID 0x0b39
#define HP_LCM220_PRODUCT_ID 0x3139
#define HP_LCM960_PRODUCT_ID 0x3239
#define HP_LD220_PRODUCT_ID 0x3524
+#define HP_LD220TA_PRODUCT_ID 0x4349
+#define HP_LD960TA_PRODUCT_ID 0x4439
+#define HP_LM940_PRODUCT_ID 0x5039
/* Cressi Edy (diving computer) PC interface */
#define CRESSI_VENDOR_ID 0x04b8
diff --git a/drivers/usb/serial/sierra.c b/drivers/usb/serial/sierra.c
index e1994e2..fbc7b29 100644
--- a/drivers/usb/serial/sierra.c
+++ b/drivers/usb/serial/sierra.c
@@ -790,9 +790,9 @@ static void sierra_close(struct usb_serial_port *port)
kfree(urb->transfer_buffer);
usb_free_urb(urb);
usb_autopm_put_interface_async(serial->interface);
- spin_lock(&portdata->lock);
+ spin_lock_irq(&portdata->lock);
portdata->outstanding_urbs--;
- spin_unlock(&portdata->lock);
+ spin_unlock_irq(&portdata->lock);
}
sierra_stop_rx_urbs(port);
diff --git a/drivers/usb/serial/ti_usb_3410_5052.c b/drivers/usb/serial/ti_usb_3410_5052.c
index 6bcb874b..836cb93 100644
--- a/drivers/usb/serial/ti_usb_3410_5052.c
+++ b/drivers/usb/serial/ti_usb_3410_5052.c
@@ -1129,7 +1129,7 @@ static void ti_break(struct tty_struct *tty, int break_state)
static int ti_get_port_from_code(unsigned char code)
{
- return (code >> 4) - 3;
+ return (code >> 6) & 0x01;
}
static int ti_get_func_from_code(unsigned char code)
diff --git a/drivers/usb/serial/usb-serial-simple.c b/drivers/usb/serial/usb-serial-simple.c
index 6aa7ff2..6d6acf2 100644
--- a/drivers/usb/serial/usb-serial-simple.c
+++ b/drivers/usb/serial/usb-serial-simple.c
@@ -66,6 +66,11 @@ DEVICE(flashloader, FLASHLOADER_IDS);
0x01) }
DEVICE(google, GOOGLE_IDS);
+/* Libtransistor USB console */
+#define LIBTRANSISTOR_IDS() \
+ { USB_DEVICE(0x1209, 0x8b00) }
+DEVICE(libtransistor, LIBTRANSISTOR_IDS);
+
/* ViVOpay USB Serial Driver */
#define VIVOPAY_IDS() \
{ USB_DEVICE(0x1d5f, 0x1004) } /* ViVOpay 8800 */
@@ -82,7 +87,8 @@ DEVICE(moto_modem, MOTO_IDS);
/* Motorola Tetra driver */
#define MOTOROLA_TETRA_IDS() \
- { USB_DEVICE(0x0cad, 0x9011) } /* Motorola Solutions TETRA PEI */
+ { USB_DEVICE(0x0cad, 0x9011) }, /* Motorola Solutions TETRA PEI */ \
+ { USB_DEVICE(0x0cad, 0x9012) } /* MTP6550 */
DEVICE(motorola_tetra, MOTOROLA_TETRA_IDS);
/* Novatel Wireless GPS driver */
@@ -113,6 +119,7 @@ static struct usb_serial_driver * const serial_drivers[] = {
&funsoft_device,
&flashloader_device,
&google_device,
+ &libtransistor_device,
&vivopay_device,
&moto_modem_device,
&motorola_tetra_device,
@@ -129,6 +136,7 @@ static const struct usb_device_id id_table[] = {
FUNSOFT_IDS(),
FLASHLOADER_IDS(),
GOOGLE_IDS(),
+ LIBTRANSISTOR_IDS(),
VIVOPAY_IDS(),
MOTO_IDS(),
MOTOROLA_TETRA_IDS(),
diff --git a/drivers/usb/serial/visor.c b/drivers/usb/serial/visor.c
index 337a0be8..dbc3801 100644
--- a/drivers/usb/serial/visor.c
+++ b/drivers/usb/serial/visor.c
@@ -338,47 +338,48 @@ static int palm_os_3_probe(struct usb_serial *serial,
goto exit;
}
- if (retval == sizeof(*connection_info)) {
- connection_info = (struct visor_connection_info *)
- transfer_buffer;
-
- num_ports = le16_to_cpu(connection_info->num_ports);
- for (i = 0; i < num_ports; ++i) {
- switch (
- connection_info->connections[i].port_function_id) {
- case VISOR_FUNCTION_GENERIC:
- string = "Generic";
- break;
- case VISOR_FUNCTION_DEBUGGER:
- string = "Debugger";
- break;
- case VISOR_FUNCTION_HOTSYNC:
- string = "HotSync";
- break;
- case VISOR_FUNCTION_CONSOLE:
- string = "Console";
- break;
- case VISOR_FUNCTION_REMOTE_FILE_SYS:
- string = "Remote File System";
- break;
- default:
- string = "unknown";
- break;
- }
- dev_info(dev, "%s: port %d, is for %s use\n",
- serial->type->description,
- connection_info->connections[i].port, string);
- }
+ if (retval != sizeof(*connection_info)) {
+ dev_err(dev, "Invalid connection information received from device\n");
+ retval = -ENODEV;
+ goto exit;
}
- /*
- * Handle devices that report invalid stuff here.
- */
+
+ connection_info = (struct visor_connection_info *)transfer_buffer;
+
+ num_ports = le16_to_cpu(connection_info->num_ports);
+
+ /* Handle devices that report invalid stuff here. */
if (num_ports == 0 || num_ports > 2) {
dev_warn(dev, "%s: No valid connect info available\n",
serial->type->description);
num_ports = 2;
}
+ for (i = 0; i < num_ports; ++i) {
+ switch (connection_info->connections[i].port_function_id) {
+ case VISOR_FUNCTION_GENERIC:
+ string = "Generic";
+ break;
+ case VISOR_FUNCTION_DEBUGGER:
+ string = "Debugger";
+ break;
+ case VISOR_FUNCTION_HOTSYNC:
+ string = "HotSync";
+ break;
+ case VISOR_FUNCTION_CONSOLE:
+ string = "Console";
+ break;
+ case VISOR_FUNCTION_REMOTE_FILE_SYS:
+ string = "Remote File System";
+ break;
+ default:
+ string = "unknown";
+ break;
+ }
+ dev_info(dev, "%s: port %d, is for %s use\n",
+ serial->type->description,
+ connection_info->connections[i].port, string);
+ }
dev_info(dev, "%s: Number of ports: %d\n", serial->type->description,
num_ports);
diff --git a/drivers/usb/storage/scsiglue.c b/drivers/usb/storage/scsiglue.c
index 8cd2926..344ec86 100644
--- a/drivers/usb/storage/scsiglue.c
+++ b/drivers/usb/storage/scsiglue.c
@@ -392,6 +392,15 @@ static int queuecommand_lck(struct scsi_cmnd *srb,
return 0;
}
+ if ((us->fflags & US_FL_NO_ATA_1X) &&
+ (srb->cmnd[0] == ATA_12 || srb->cmnd[0] == ATA_16)) {
+ memcpy(srb->sense_buffer, usb_stor_sense_invalidCDB,
+ sizeof(usb_stor_sense_invalidCDB));
+ srb->result = SAM_STAT_CHECK_CONDITION;
+ done(srb);
+ return 0;
+ }
+
/* enqueue the command and wake up the control thread */
srb->scsi_done = done;
us->srb = srb;
diff --git a/drivers/usb/storage/uas.c b/drivers/usb/storage/uas.c
index a96dcc6..64af889 100644
--- a/drivers/usb/storage/uas.c
+++ b/drivers/usb/storage/uas.c
@@ -836,6 +836,33 @@ static int uas_slave_configure(struct scsi_device *sdev)
if (devinfo->flags & US_FL_BROKEN_FUA)
sdev->broken_fua = 1;
+ /* UAS also needs to support FL_ALWAYS_SYNC */
+ if (devinfo->flags & US_FL_ALWAYS_SYNC) {
+ sdev->skip_ms_page_3f = 1;
+ sdev->skip_ms_page_8 = 1;
+ sdev->wce_default_on = 1;
+ }
+
+ /*
+ * Some disks return the total number of blocks in response
+ * to READ CAPACITY rather than the highest block number.
+ * If this device makes that mistake, tell the sd driver.
+ */
+ if (devinfo->flags & US_FL_FIX_CAPACITY)
+ sdev->fix_capacity = 1;
+
+ /*
+ * Some devices don't like MODE SENSE with page=0x3f,
+ * which is the command used for checking if a device
+ * is write-protected. Now that we tell the sd driver
+ * to do a 192-byte transfer with this command the
+ * majority of devices work fine, but a few still can't
+ * handle it. The sd driver will simply assume those
+ * devices are write-enabled.
+ */
+ if (devinfo->flags & US_FL_NO_WP_DETECT)
+ sdev->skip_ms_page_3f = 1;
+
scsi_change_queue_depth(sdev, devinfo->qdepth - 2);
return 0;
}
diff --git a/drivers/usb/storage/unusual_devs.h b/drivers/usb/storage/unusual_devs.h
index ca3a5d4..0a86b3f 100644
--- a/drivers/usb/storage/unusual_devs.h
+++ b/drivers/usb/storage/unusual_devs.h
@@ -2307,6 +2307,13 @@ UNUSUAL_DEV( 0x2735, 0x100b, 0x0000, 0x9999,
USB_SC_DEVICE, USB_PR_DEVICE, NULL,
US_FL_GO_SLOW ),
+/* Reported-by: Tim Anderson <tsa@biglakesoftware.com> */
+UNUSUAL_DEV( 0x2ca3, 0x0031, 0x0000, 0x9999,
+ "DJI",
+ "CineSSD",
+ USB_SC_DEVICE, USB_PR_DEVICE, NULL,
+ US_FL_NO_ATA_1X),
+
/*
* Reported by Frederic Marchal <frederic.marchal@wowcompany.com>
* Mio Moov 330
@@ -2340,6 +2347,15 @@ UNUSUAL_DEV( 0x4146, 0xba01, 0x0100, 0x0100,
"Micro Mini 1GB",
USB_SC_DEVICE, USB_PR_DEVICE, NULL, US_FL_NOT_LOCKABLE ),
+/* "G-DRIVE" external HDD hangs on write without these.
+ * Patch submitted by Alexander Kappner <agk@godking.net>
+ */
+UNUSUAL_DEV(0x4971, 0x8024, 0x0000, 0x9999,
+ "SimpleTech",
+ "External HDD",
+ USB_SC_DEVICE, USB_PR_DEVICE, NULL,
+ US_FL_ALWAYS_SYNC),
+
/*
* Nick Bowler <nbowler@elliptictech.com>
* SCSI stack spams (otherwise harmless) error messages.
diff --git a/drivers/usb/storage/unusual_realtek.h b/drivers/usb/storage/unusual_realtek.h
index 8fe624a..7ca7794 100644
--- a/drivers/usb/storage/unusual_realtek.h
+++ b/drivers/usb/storage/unusual_realtek.h
@@ -39,4 +39,14 @@ UNUSUAL_DEV(0x0bda, 0x0159, 0x0000, 0x9999,
"USB Card Reader",
USB_SC_DEVICE, USB_PR_DEVICE, init_realtek_cr, 0),
+UNUSUAL_DEV(0x0bda, 0x0177, 0x0000, 0x9999,
+ "Realtek",
+ "USB Card Reader",
+ USB_SC_DEVICE, USB_PR_DEVICE, init_realtek_cr, 0),
+
+UNUSUAL_DEV(0x0bda, 0x0184, 0x0000, 0x9999,
+ "Realtek",
+ "USB Card Reader",
+ USB_SC_DEVICE, USB_PR_DEVICE, init_realtek_cr, 0),
+
#endif /* defined(CONFIG_USB_STORAGE_REALTEK) || ... */
diff --git a/drivers/usb/storage/unusual_uas.h b/drivers/usb/storage/unusual_uas.h
index 719ec68..f15aa47 100644
--- a/drivers/usb/storage/unusual_uas.h
+++ b/drivers/usb/storage/unusual_uas.h
@@ -183,3 +183,12 @@ UNUSUAL_DEV(0x4971, 0x8017, 0x0000, 0x9999,
"External HDD",
USB_SC_DEVICE, USB_PR_DEVICE, NULL,
US_FL_NO_REPORT_OPCODES),
+
+/* "G-DRIVE" external HDD hangs on write without these.
+ * Patch submitted by Alexander Kappner <agk@godking.net>
+ */
+UNUSUAL_DEV(0x4971, 0x8024, 0x0000, 0x9999,
+ "SimpleTech",
+ "External HDD",
+ USB_SC_DEVICE, USB_PR_DEVICE, NULL,
+ US_FL_ALWAYS_SYNC),
diff --git a/drivers/usb/typec/Kconfig b/drivers/usb/typec/Kconfig
new file mode 100644
index 0000000..ad85d0f
--- /dev/null
+++ b/drivers/usb/typec/Kconfig
@@ -0,0 +1,15 @@
+
+menu "USB Power Delivery and Type-C drivers"
+
+config TYPEC
+ tristate
+
+config TYPEC_TCPM
+ tristate "USB Type-C Port Controller Manager"
+ depends on USB
+ select TYPEC
+ help
+ The Type-C Port Controller Manager provides a USB PD and USB Type-C
+ state machine for use with Type-C Port Controllers.
+
+endmenu
diff --git a/drivers/usb/typec/Makefile b/drivers/usb/typec/Makefile
new file mode 100644
index 0000000..075c2a4
--- /dev/null
+++ b/drivers/usb/typec/Makefile
@@ -0,0 +1,2 @@
+obj-$(CONFIG_TYPEC) += typec.o
+obj-$(CONFIG_TYPEC_TCPM) += tcpm.o
diff --git a/drivers/usb/typec/tcpm.c b/drivers/usb/typec/tcpm.c
new file mode 100644
index 0000000..4336e9f
--- /dev/null
+++ b/drivers/usb/typec/tcpm.c
@@ -0,0 +1,3973 @@
+/*
+ * Copyright 2015-2017 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * USB Power Delivery protocol stack.
+ */
+
+#include <linux/completion.h>
+#include <linux/debugfs.h>
+#include <linux/device.h>
+#include <linux/jiffies.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/proc_fs.h>
+#include <linux/seq_file.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/usb/pd.h>
+#include <linux/usb/pd_bdo.h>
+#include <linux/usb/pd_vdo.h>
+#include <linux/usb/tcpm.h>
+#include <linux/usb/typec.h>
+#include <linux/workqueue.h>
+
+#define FOREACH_STATE(S) \
+ S(INVALID_STATE), \
+ S(DRP_TOGGLING), \
+ S(SRC_UNATTACHED), \
+ S(SRC_ATTACH_WAIT), \
+ S(SRC_ATTACHED), \
+ S(SRC_STARTUP), \
+ S(SRC_SEND_CAPABILITIES), \
+ S(SRC_NEGOTIATE_CAPABILITIES), \
+ S(SRC_TRANSITION_SUPPLY), \
+ S(SRC_READY), \
+ S(SRC_WAIT_NEW_CAPABILITIES), \
+ \
+ S(SNK_UNATTACHED), \
+ S(SNK_ATTACH_WAIT), \
+ S(SNK_DEBOUNCED), \
+ S(SNK_ATTACHED), \
+ S(SNK_STARTUP), \
+ S(SNK_DISCOVERY), \
+ S(SNK_DISCOVERY_DEBOUNCE), \
+ S(SNK_DISCOVERY_DEBOUNCE_DONE), \
+ S(SNK_WAIT_CAPABILITIES), \
+ S(SNK_NEGOTIATE_CAPABILITIES), \
+ S(SNK_TRANSITION_SINK), \
+ S(SNK_TRANSITION_SINK_VBUS), \
+ S(SNK_READY), \
+ \
+ S(ACC_UNATTACHED), \
+ S(DEBUG_ACC_ATTACHED), \
+ S(AUDIO_ACC_ATTACHED), \
+ S(AUDIO_ACC_DEBOUNCE), \
+ \
+ S(HARD_RESET_SEND), \
+ S(HARD_RESET_START), \
+ S(SRC_HARD_RESET_VBUS_OFF), \
+ S(SRC_HARD_RESET_VBUS_ON), \
+ S(SNK_HARD_RESET_SINK_OFF), \
+ S(SNK_HARD_RESET_WAIT_VBUS), \
+ S(SNK_HARD_RESET_SINK_ON), \
+ \
+ S(SOFT_RESET), \
+ S(SOFT_RESET_SEND), \
+ \
+ S(DR_SWAP_ACCEPT), \
+ S(DR_SWAP_SEND), \
+ S(DR_SWAP_SEND_TIMEOUT), \
+ S(DR_SWAP_CANCEL), \
+ S(DR_SWAP_CHANGE_DR), \
+ \
+ S(PR_SWAP_ACCEPT), \
+ S(PR_SWAP_SEND), \
+ S(PR_SWAP_SEND_TIMEOUT), \
+ S(PR_SWAP_CANCEL), \
+ S(PR_SWAP_START), \
+ S(PR_SWAP_SRC_SNK_TRANSITION_OFF), \
+ S(PR_SWAP_SRC_SNK_SOURCE_OFF), \
+ S(PR_SWAP_SRC_SNK_SOURCE_OFF_CC_DEBOUNCED), \
+ S(PR_SWAP_SRC_SNK_SINK_ON), \
+ S(PR_SWAP_SNK_SRC_SINK_OFF), \
+ S(PR_SWAP_SNK_SRC_SOURCE_ON), \
+ S(PR_SWAP_SNK_SRC_SOURCE_ON_VBUS_RAMPED_UP), \
+ \
+ S(VCONN_SWAP_ACCEPT), \
+ S(VCONN_SWAP_SEND), \
+ S(VCONN_SWAP_SEND_TIMEOUT), \
+ S(VCONN_SWAP_CANCEL), \
+ S(VCONN_SWAP_START), \
+ S(VCONN_SWAP_WAIT_FOR_VCONN), \
+ S(VCONN_SWAP_TURN_ON_VCONN), \
+ S(VCONN_SWAP_TURN_OFF_VCONN), \
+ \
+ S(SNK_TRY), \
+ S(SNK_TRY_WAIT), \
+ S(SNK_TRY_WAIT_DEBOUNCE), \
+ S(SNK_TRY_WAIT_DEBOUNCE_CHECK_VBUS), \
+ S(SRC_TRYWAIT), \
+ S(SRC_TRYWAIT_DEBOUNCE), \
+ S(SRC_TRYWAIT_UNATTACHED), \
+ \
+ S(SRC_TRY), \
+ S(SRC_TRY_WAIT), \
+ S(SRC_TRY_DEBOUNCE), \
+ S(SNK_TRYWAIT), \
+ S(SNK_TRYWAIT_DEBOUNCE), \
+ S(SNK_TRYWAIT_VBUS), \
+ S(BIST_RX), \
+ \
+ S(ERROR_RECOVERY), \
+ S(PORT_RESET), \
+ S(PORT_RESET_WAIT_OFF)
+
+#define GENERATE_ENUM(e) e
+#define GENERATE_STRING(s) #s
+
+enum tcpm_state {
+ FOREACH_STATE(GENERATE_ENUM)
+};
+
+static const char * const tcpm_states[] = {
+ FOREACH_STATE(GENERATE_STRING)
+};
+
+enum vdm_states {
+ VDM_STATE_ERR_BUSY = -3,
+ VDM_STATE_ERR_SEND = -2,
+ VDM_STATE_ERR_TMOUT = -1,
+ VDM_STATE_DONE = 0,
+ /* Anything >0 represents an active state */
+ VDM_STATE_READY = 1,
+ VDM_STATE_BUSY = 2,
+ VDM_STATE_WAIT_RSP_BUSY = 3,
+};
+
+enum pd_msg_request {
+ PD_MSG_NONE = 0,
+ PD_MSG_CTRL_REJECT,
+ PD_MSG_CTRL_WAIT,
+ PD_MSG_DATA_SINK_CAP,
+ PD_MSG_DATA_SOURCE_CAP,
+};
+
+/* Events from low level driver */
+
+#define TCPM_CC_EVENT BIT(0)
+#define TCPM_VBUS_EVENT BIT(1)
+#define TCPM_RESET_EVENT BIT(2)
+#define TCPM_PORT_RESET_EVENT BIT(3)
+
+#define LOG_BUFFER_ENTRIES 1024
+#define LOG_BUFFER_ENTRY_SIZE 128
+
+/* Alternate mode support */
+
+#define SVID_DISCOVERY_MAX 16
+
+struct pd_mode_data {
+ int svid_index; /* current SVID index */
+ int nsvids;
+ u16 svids[SVID_DISCOVERY_MAX];
+ int altmodes; /* number of alternate modes */
+ struct typec_altmode_desc altmode_desc[SVID_DISCOVERY_MAX];
+};
+
+struct tcpm_port {
+ struct device *dev;
+
+ struct mutex lock; /* tcpm state machine lock */
+ struct workqueue_struct *wq;
+
+ struct typec_capability typec_caps;
+ struct typec_port *typec_port;
+
+ struct tcpc_dev *tcpc;
+
+ enum typec_role vconn_role;
+ enum typec_role pwr_role;
+ enum typec_data_role data_role;
+ enum typec_pwr_opmode pwr_opmode;
+
+ struct usb_pd_identity partner_ident;
+ struct typec_partner_desc partner_desc;
+ struct typec_partner *partner;
+
+ enum typec_cc_status cc_req;
+
+ enum typec_cc_status cc1;
+ enum typec_cc_status cc2;
+ enum typec_cc_polarity polarity;
+
+ bool attached;
+ bool connected;
+ enum typec_port_type port_type;
+ bool vbus_present;
+ bool vbus_never_low;
+ bool vbus_source;
+ bool vbus_charge;
+
+ bool send_discover;
+ bool op_vsafe5v;
+
+ int try_role;
+ int try_snk_count;
+ int try_src_count;
+
+ enum pd_msg_request queued_message;
+
+ enum tcpm_state enter_state;
+ enum tcpm_state prev_state;
+ enum tcpm_state state;
+ enum tcpm_state delayed_state;
+ unsigned long delayed_runtime;
+ unsigned long delay_ms;
+
+ spinlock_t pd_event_lock;
+ u32 pd_events;
+
+ struct work_struct event_work;
+ struct delayed_work state_machine;
+ struct delayed_work vdm_state_machine;
+ bool state_machine_running;
+
+ struct completion tx_complete;
+ enum tcpm_transmit_status tx_status;
+
+ struct mutex swap_lock; /* swap command lock */
+ bool swap_pending;
+ bool non_pd_role_swap;
+ struct completion swap_complete;
+ int swap_status;
+
+ unsigned int message_id;
+ unsigned int caps_count;
+ unsigned int hard_reset_count;
+ bool pd_capable;
+ bool explicit_contract;
+ bool usb_comm_capable;
+ unsigned int rx_msgid;
+
+ /* Partner capabilities/requests */
+ u32 sink_request;
+ u32 source_caps[PDO_MAX_OBJECTS];
+ unsigned int nr_source_caps;
+ u32 sink_caps[PDO_MAX_OBJECTS];
+ unsigned int nr_sink_caps;
+
+ /* Local capabilities */
+ u32 src_pdo[PDO_MAX_OBJECTS];
+ unsigned int nr_src_pdo;
+ u32 snk_pdo[PDO_MAX_OBJECTS];
+ unsigned int nr_snk_pdo;
+ unsigned int nr_fixed; /* number of fixed sink PDOs */
+ unsigned int nr_var; /* number of variable sink PDOs */
+ unsigned int nr_batt; /* number of battery sink PDOs */
+ u32 snk_vdo[VDO_MAX_OBJECTS];
+ unsigned int nr_snk_vdo;
+
+ unsigned int max_snk_mv;
+ unsigned int max_snk_ma;
+ unsigned int max_snk_mw;
+ unsigned int operating_snk_mw;
+
+ /* Requested current / voltage */
+ u32 current_limit;
+ u32 supply_min_mv;
+ u32 supply_max_mv;
+ u32 requested_min_mv;
+ u32 requested_max_mv;
+
+ u32 bist_request;
+
+ /* PD state for Vendor Defined Messages */
+ enum vdm_states vdm_state;
+ u32 vdm_retries;
+ /* next Vendor Defined Message to send */
+ u32 vdo_data[VDO_MAX_SIZE];
+ u8 vdo_count;
+ /* VDO to retry if UFP responder replied busy */
+ u32 vdo_retry;
+
+ /* Alternate mode data */
+
+ struct pd_mode_data mode_data;
+ struct typec_altmode *partner_altmode[SVID_DISCOVERY_MAX];
+ struct typec_altmode *port_altmode[SVID_DISCOVERY_MAX];
+
+ /* Deadline in jiffies to exit src_try_wait state */
+ unsigned long max_wait;
+
+ /* port belongs to a self powered device */
+ bool self_powered;
+
+#ifdef CONFIG_DEBUG_FS
+ struct dentry *dentry;
+ struct mutex logbuffer_lock; /* log buffer access lock */
+ int logbuffer_head;
+ int logbuffer_tail;
+ u8 *logbuffer[LOG_BUFFER_ENTRIES];
+#endif
+};
+
+struct pd_rx_event {
+ struct work_struct work;
+ struct tcpm_port *port;
+ struct pd_message msg;
+};
+
+#define tcpm_cc_is_sink(cc) \
+ ((cc) == TYPEC_CC_RP_DEF || (cc) == TYPEC_CC_RP_1_5 || \
+ (cc) == TYPEC_CC_RP_3_0)
+
+#define tcpm_port_is_sink(port) \
+ ((tcpm_cc_is_sink((port)->cc1) && !tcpm_cc_is_sink((port)->cc2)) || \
+ (tcpm_cc_is_sink((port)->cc2) && !tcpm_cc_is_sink((port)->cc1)))
+
+#define tcpm_cc_is_source(cc) ((cc) == TYPEC_CC_RD)
+#define tcpm_cc_is_audio(cc) ((cc) == TYPEC_CC_RA)
+#define tcpm_cc_is_open(cc) ((cc) == TYPEC_CC_OPEN)
+
+#define tcpm_port_is_source(port) \
+ ((tcpm_cc_is_source((port)->cc1) && \
+ !tcpm_cc_is_source((port)->cc2)) || \
+ (tcpm_cc_is_source((port)->cc2) && \
+ !tcpm_cc_is_source((port)->cc1)))
+
+#define tcpm_port_is_debug(port) \
+ (tcpm_cc_is_source((port)->cc1) && tcpm_cc_is_source((port)->cc2))
+
+#define tcpm_port_is_audio(port) \
+ (tcpm_cc_is_audio((port)->cc1) && tcpm_cc_is_audio((port)->cc2))
+
+#define tcpm_port_is_audio_detached(port) \
+ ((tcpm_cc_is_audio((port)->cc1) && tcpm_cc_is_open((port)->cc2)) || \
+ (tcpm_cc_is_audio((port)->cc2) && tcpm_cc_is_open((port)->cc1)))
+
+#define tcpm_try_snk(port) \
+ ((port)->try_snk_count == 0 && (port)->try_role == TYPEC_SINK && \
+ (port)->port_type == TYPEC_PORT_DRP)
+
+#define tcpm_try_src(port) \
+ ((port)->try_src_count == 0 && (port)->try_role == TYPEC_SOURCE && \
+ (port)->port_type == TYPEC_PORT_DRP)
+
+static enum tcpm_state tcpm_default_state(struct tcpm_port *port)
+{
+ if (port->port_type == TYPEC_PORT_DRP) {
+ if (port->try_role == TYPEC_SINK)
+ return SNK_UNATTACHED;
+ else if (port->try_role == TYPEC_SOURCE)
+ return SRC_UNATTACHED;
+ else if (port->tcpc->config->default_role == TYPEC_SINK)
+ return SNK_UNATTACHED;
+ /* Fall through to return SRC_UNATTACHED */
+ } else if (port->port_type == TYPEC_PORT_UFP) {
+ return SNK_UNATTACHED;
+ }
+ return SRC_UNATTACHED;
+}
+
+static inline
+struct tcpm_port *typec_cap_to_tcpm(const struct typec_capability *cap)
+{
+ return container_of(cap, struct tcpm_port, typec_caps);
+}
+
+static bool tcpm_port_is_disconnected(struct tcpm_port *port)
+{
+ return (!port->attached && port->cc1 == TYPEC_CC_OPEN &&
+ port->cc2 == TYPEC_CC_OPEN) ||
+ (port->attached && ((port->polarity == TYPEC_POLARITY_CC1 &&
+ port->cc1 == TYPEC_CC_OPEN) ||
+ (port->polarity == TYPEC_POLARITY_CC2 &&
+ port->cc2 == TYPEC_CC_OPEN)));
+}
+
+/*
+ * Logging
+ */
+
+#ifdef CONFIG_DEBUG_FS
+
+static bool tcpm_log_full(struct tcpm_port *port)
+{
+ return port->logbuffer_tail ==
+ (port->logbuffer_head + 1) % LOG_BUFFER_ENTRIES;
+}
+
+__printf(2, 0)
+static void _tcpm_log(struct tcpm_port *port, const char *fmt, va_list args)
+{
+ char tmpbuffer[LOG_BUFFER_ENTRY_SIZE];
+ u64 ts_nsec = local_clock();
+ unsigned long rem_nsec;
+
+ if (!port->logbuffer[port->logbuffer_head]) {
+ port->logbuffer[port->logbuffer_head] =
+ kzalloc(LOG_BUFFER_ENTRY_SIZE, GFP_KERNEL);
+ if (!port->logbuffer[port->logbuffer_head])
+ return;
+ }
+
+ vsnprintf(tmpbuffer, sizeof(tmpbuffer), fmt, args);
+
+ mutex_lock(&port->logbuffer_lock);
+
+ if (tcpm_log_full(port)) {
+ port->logbuffer_head = max(port->logbuffer_head - 1, 0);
+ strcpy(tmpbuffer, "overflow");
+ }
+
+ if (port->logbuffer_head < 0 ||
+ port->logbuffer_head >= LOG_BUFFER_ENTRIES) {
+ dev_warn(port->dev,
+ "Bad log buffer index %d\n", port->logbuffer_head);
+ goto abort;
+ }
+
+ if (!port->logbuffer[port->logbuffer_head]) {
+ dev_warn(port->dev,
+ "Log buffer index %d is NULL\n", port->logbuffer_head);
+ goto abort;
+ }
+
+ rem_nsec = do_div(ts_nsec, 1000000000);
+ scnprintf(port->logbuffer[port->logbuffer_head],
+ LOG_BUFFER_ENTRY_SIZE, "[%5lu.%06lu] %s",
+ (unsigned long)ts_nsec, rem_nsec / 1000,
+ tmpbuffer);
+ port->logbuffer_head = (port->logbuffer_head + 1) % LOG_BUFFER_ENTRIES;
+
+abort:
+ mutex_unlock(&port->logbuffer_lock);
+}
+
+__printf(2, 3)
+static void tcpm_log(struct tcpm_port *port, const char *fmt, ...)
+{
+ va_list args;
+
+ /* Do not log while disconnected and unattached */
+ if (tcpm_port_is_disconnected(port) &&
+ (port->state == SRC_UNATTACHED || port->state == SNK_UNATTACHED ||
+ port->state == DRP_TOGGLING))
+ return;
+
+ if (port->tcpc->log_rtc)
+ port->tcpc->log_rtc(port->tcpc);
+
+ va_start(args, fmt);
+ _tcpm_log(port, fmt, args);
+ va_end(args);
+}
+
+__printf(2, 3)
+static void tcpm_log_force(struct tcpm_port *port, const char *fmt, ...)
+{
+ va_list args;
+
+ va_start(args, fmt);
+ _tcpm_log(port, fmt, args);
+ va_end(args);
+}
+
+static void tcpm_log_source_caps(struct tcpm_port *port)
+{
+ int i;
+
+ for (i = 0; i < port->nr_source_caps; i++) {
+ u32 pdo = port->source_caps[i];
+ enum pd_pdo_type type = pdo_type(pdo);
+ char msg[64];
+
+ switch (type) {
+ case PDO_TYPE_FIXED:
+ scnprintf(msg, sizeof(msg),
+ "%u mV, %u mA [%s%s%s%s%s%s]",
+ pdo_fixed_voltage(pdo),
+ pdo_max_current(pdo),
+ (pdo & PDO_FIXED_DUAL_ROLE) ?
+ "R" : "",
+ (pdo & PDO_FIXED_SUSPEND) ?
+ "S" : "",
+ (pdo & PDO_FIXED_HIGHER_CAP) ?
+ "H" : "",
+ (pdo & PDO_FIXED_USB_COMM) ?
+ "U" : "",
+ (pdo & PDO_FIXED_DATA_SWAP) ?
+ "D" : "",
+ (pdo & PDO_FIXED_EXTPOWER) ?
+ "E" : "");
+ break;
+ case PDO_TYPE_VAR:
+ scnprintf(msg, sizeof(msg),
+ "%u-%u mV, %u mA",
+ pdo_min_voltage(pdo),
+ pdo_max_voltage(pdo),
+ pdo_max_current(pdo));
+ break;
+ case PDO_TYPE_BATT:
+ scnprintf(msg, sizeof(msg),
+ "%u-%u mV, %u mW",
+ pdo_min_voltage(pdo),
+ pdo_max_voltage(pdo),
+ pdo_max_power(pdo));
+ break;
+ default:
+ strcpy(msg, "undefined");
+ break;
+ }
+ tcpm_log(port, " PDO %d: type %d, %s",
+ i, type, msg);
+ }
+}
+
+static int tcpm_seq_show(struct seq_file *s, void *v)
+{
+ struct tcpm_port *port = (struct tcpm_port *)s->private;
+ int tail;
+
+ mutex_lock(&port->logbuffer_lock);
+ tail = port->logbuffer_tail;
+ while (tail != port->logbuffer_head) {
+ seq_printf(s, "%s\n", port->logbuffer[tail]);
+ tail = (tail + 1) % LOG_BUFFER_ENTRIES;
+ }
+ if (!seq_has_overflowed(s))
+ port->logbuffer_tail = tail;
+ mutex_unlock(&port->logbuffer_lock);
+
+ return 0;
+}
+
+static int tcpm_debug_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, tcpm_seq_show, inode->i_private);
+}
+
+static const struct file_operations tcpm_debug_operations = {
+ .open = tcpm_debug_open,
+ .llseek = seq_lseek,
+ .read = seq_read,
+ .release = single_release,
+};
+
+static struct dentry *rootdir;
+
+static int tcpm_debugfs_init(struct tcpm_port *port)
+{
+ mutex_init(&port->logbuffer_lock);
+ /* /sys/kernel/debug/tcpm/usbcX */
+ if (!rootdir) {
+ rootdir = debugfs_create_dir("tcpm", NULL);
+ if (!rootdir)
+ return -ENOMEM;
+ }
+
+ port->dentry = debugfs_create_file(dev_name(port->dev),
+ S_IFREG | 0444, rootdir,
+ port, &tcpm_debug_operations);
+
+ return 0;
+}
+
+static void tcpm_debugfs_exit(struct tcpm_port *port)
+{
+ debugfs_remove(port->dentry);
+}
+
+#else
+
+__printf(2, 3)
+static void tcpm_log(const struct tcpm_port *port, const char *fmt, ...) { }
+__printf(2, 3)
+static void tcpm_log_force(struct tcpm_port *port, const char *fmt, ...) { }
+static void tcpm_log_source_caps(struct tcpm_port *port) { }
+static int tcpm_debugfs_init(const struct tcpm_port *port) { return 0; }
+static void tcpm_debugfs_exit(const struct tcpm_port *port) { }
+
+#endif
+
+static int tcpm_pd_transmit(struct tcpm_port *port,
+ enum tcpm_transmit_type type,
+ const struct pd_message *msg)
+{
+ unsigned long timeout;
+ int ret;
+
+ if (msg)
+ tcpm_log(port, "PD TX, header: %#x", le16_to_cpu(msg->header));
+ else
+ tcpm_log(port, "PD TX, type: %#x", type);
+
+ reinit_completion(&port->tx_complete);
+ ret = port->tcpc->pd_transmit(port->tcpc, type, msg);
+ if (ret < 0)
+ return ret;
+
+ mutex_unlock(&port->lock);
+ timeout = wait_for_completion_timeout(&port->tx_complete,
+ msecs_to_jiffies(PD_T_TCPC_TX_TIMEOUT));
+ mutex_lock(&port->lock);
+ if (!timeout)
+ return -ETIMEDOUT;
+
+ switch (port->tx_status) {
+ case TCPC_TX_SUCCESS:
+ port->message_id = (port->message_id + 1) & PD_HEADER_ID_MASK;
+ return 0;
+ case TCPC_TX_DISCARDED:
+ return -EAGAIN;
+ case TCPC_TX_FAILED:
+ default:
+ return -EIO;
+ }
+}
+
+void tcpm_pd_transmit_complete(struct tcpm_port *port,
+ enum tcpm_transmit_status status)
+{
+ tcpm_log(port, "PD TX complete, status: %u", status);
+ port->tx_status = status;
+ complete(&port->tx_complete);
+}
+EXPORT_SYMBOL_GPL(tcpm_pd_transmit_complete);
+
+static int tcpm_mux_set(struct tcpm_port *port, enum tcpc_mux_mode mode,
+ enum tcpc_usb_switch config)
+{
+ int ret = 0;
+
+ tcpm_log(port, "Requesting mux mode %d, config %d, polarity %d",
+ mode, config, port->polarity);
+
+ if (port->tcpc->mux)
+ ret = port->tcpc->mux->set(port->tcpc->mux, mode, config,
+ port->polarity);
+
+ return ret;
+}
+
+static int tcpm_set_polarity(struct tcpm_port *port,
+ enum typec_cc_polarity polarity)
+{
+ int ret;
+
+ tcpm_log(port, "polarity %d", polarity);
+
+ ret = port->tcpc->set_polarity(port->tcpc, polarity);
+ if (ret < 0)
+ return ret;
+
+ port->polarity = polarity;
+
+ return 0;
+}
+
+static int tcpm_set_vconn(struct tcpm_port *port, bool enable)
+{
+ int ret;
+
+ tcpm_log(port, "vconn:=%d", enable);
+
+ ret = port->tcpc->set_vconn(port->tcpc, enable);
+ if (!ret) {
+ port->vconn_role = enable ? TYPEC_SOURCE : TYPEC_SINK;
+ typec_set_vconn_role(port->typec_port, port->vconn_role);
+ }
+
+ return ret;
+}
+
+static u32 tcpm_get_current_limit(struct tcpm_port *port)
+{
+ enum typec_cc_status cc;
+ u32 limit;
+
+ cc = port->polarity ? port->cc2 : port->cc1;
+ switch (cc) {
+ case TYPEC_CC_RP_1_5:
+ limit = 1500;
+ break;
+ case TYPEC_CC_RP_3_0:
+ limit = 3000;
+ break;
+ case TYPEC_CC_RP_DEF:
+ default:
+ if (port->tcpc->get_current_limit)
+ limit = port->tcpc->get_current_limit(port->tcpc);
+ else
+ limit = 0;
+ break;
+ }
+
+ return limit;
+}
+
+static int tcpm_set_current_limit(struct tcpm_port *port, u32 max_ma,
+ u32 min_mv, u32 max_mv)
+{
+ int ret = -EOPNOTSUPP;
+
+ tcpm_log(port, "Setting voltage/current limit max_mv %u mV min_mv %u mV %u mA"
+ , max_mv, min_mv, max_ma);
+
+ if (port->tcpc->set_current_limit)
+ ret = port->tcpc->set_current_limit(port->tcpc, max_ma, min_mv,
+ max_mv);
+
+ if (!ret) {
+ port->supply_min_mv = min_mv;
+ port->supply_max_mv = max_mv;
+ }
+
+ return ret;
+}
+
+static void tcpm_set_pd_capable(struct tcpm_port *port, bool capable)
+{
+ tcpm_log(port, "Setting pd capable %s", capable ? "true" : "false");
+
+ if (port->tcpc->set_pd_capable)
+ port->tcpc->set_pd_capable(port->tcpc, capable);
+
+ port->pd_capable = capable;
+}
+
+static void tcpm_port_in_hard_reset(struct tcpm_port *port, bool status)
+{
+ tcpm_log(port, "Setting hard reset %s", status ? "true" : "false");
+
+ if (port->tcpc->set_in_hard_reset)
+ port->tcpc->set_in_hard_reset(port->tcpc, status);
+}
+
+
+/*
+ * Determine RP value to set based on maximum current supported
+ * by a port if configured as source.
+ * Returns CC value to report to link partner.
+ */
+static enum typec_cc_status tcpm_rp_cc(struct tcpm_port *port)
+{
+ const u32 *src_pdo = port->src_pdo;
+ int nr_pdo = port->nr_src_pdo;
+ int i;
+
+ /*
+ * Search for first entry with matching voltage.
+ * It should report the maximum supported current.
+ */
+ for (i = 0; i < nr_pdo; i++) {
+ const u32 pdo = src_pdo[i];
+
+ if (pdo_type(pdo) == PDO_TYPE_FIXED &&
+ pdo_fixed_voltage(pdo) == 5000) {
+ unsigned int curr = pdo_max_current(pdo);
+
+ if (curr >= 3000)
+ return TYPEC_CC_RP_3_0;
+ else if (curr >= 1500)
+ return TYPEC_CC_RP_1_5;
+ return TYPEC_CC_RP_DEF;
+ }
+ }
+
+ return TYPEC_CC_RP_DEF;
+}
+
+static int tcpm_set_attached_state(struct tcpm_port *port, bool attached)
+{
+ return port->tcpc->set_roles(port->tcpc, attached, port->pwr_role,
+ port->data_role, port->usb_comm_capable);
+}
+
+static int tcpm_set_roles(struct tcpm_port *port, bool attached,
+ enum typec_role role, enum typec_data_role data)
+{
+ int ret;
+
+ if (data == TYPEC_HOST)
+ ret = tcpm_mux_set(port, TYPEC_MUX_USB,
+ TCPC_USB_SWITCH_CONNECT);
+ else
+ ret = tcpm_mux_set(port, TYPEC_MUX_NONE,
+ TCPC_USB_SWITCH_DISCONNECT);
+ if (ret < 0)
+ return ret;
+
+ ret = port->tcpc->set_roles(port->tcpc, attached, role, data,
+ port->usb_comm_capable);
+ if (ret < 0)
+ return ret;
+
+ port->pwr_role = role;
+ port->data_role = data;
+ typec_set_data_role(port->typec_port, data);
+ typec_set_pwr_role(port->typec_port, role);
+
+ return 0;
+}
+
+static int tcpm_set_pwr_role(struct tcpm_port *port, enum typec_role role)
+{
+ int ret;
+
+ ret = port->tcpc->set_roles(port->tcpc, true, role,
+ port->data_role, port->usb_comm_capable);
+ if (ret < 0)
+ return ret;
+
+ port->pwr_role = role;
+ typec_set_pwr_role(port->typec_port, role);
+
+ return 0;
+}
+
+static int tcpm_pd_send_source_caps(struct tcpm_port *port)
+{
+ struct pd_message msg;
+ int i;
+
+ memset(&msg, 0, sizeof(msg));
+ if (!port->nr_src_pdo) {
+ /* No source capabilities defined, sink only */
+ msg.header = PD_HEADER_LE(PD_CTRL_REJECT,
+ port->pwr_role,
+ port->data_role,
+ port->message_id, 0);
+ } else {
+ msg.header = PD_HEADER_LE(PD_DATA_SOURCE_CAP,
+ port->pwr_role,
+ port->data_role,
+ port->message_id,
+ port->nr_src_pdo);
+ }
+ for (i = 0; i < port->nr_src_pdo; i++)
+ msg.payload[i] = cpu_to_le32(port->src_pdo[i]);
+
+ return tcpm_pd_transmit(port, TCPC_TX_SOP, &msg);
+}
+
+static int tcpm_pd_send_sink_caps(struct tcpm_port *port)
+{
+ struct pd_message msg;
+ int i;
+
+ memset(&msg, 0, sizeof(msg));
+ if (!port->nr_snk_pdo) {
+ /* No sink capabilities defined, source only */
+ msg.header = PD_HEADER_LE(PD_CTRL_REJECT,
+ port->pwr_role,
+ port->data_role,
+ port->message_id, 0);
+ } else {
+ msg.header = PD_HEADER_LE(PD_DATA_SINK_CAP,
+ port->pwr_role,
+ port->data_role,
+ port->message_id,
+ port->nr_snk_pdo);
+ }
+ for (i = 0; i < port->nr_snk_pdo; i++)
+ msg.payload[i] = cpu_to_le32(port->snk_pdo[i]);
+
+ return tcpm_pd_transmit(port, TCPC_TX_SOP, &msg);
+}
+
+static void tcpm_set_state(struct tcpm_port *port, enum tcpm_state state,
+ unsigned int delay_ms)
+{
+ if (delay_ms) {
+ tcpm_log(port, "pending state change %s -> %s @ %u ms",
+ tcpm_states[port->state], tcpm_states[state],
+ delay_ms);
+ port->delayed_state = state;
+ mod_delayed_work(port->wq, &port->state_machine,
+ msecs_to_jiffies(delay_ms));
+ port->delayed_runtime = jiffies + msecs_to_jiffies(delay_ms);
+ port->delay_ms = delay_ms;
+ } else {
+ tcpm_log(port, "state change %s -> %s",
+ tcpm_states[port->state], tcpm_states[state]);
+ port->delayed_state = INVALID_STATE;
+ port->prev_state = port->state;
+ port->state = state;
+ /*
+ * Don't re-queue the state machine work item if we're currently
+ * in the state machine and we're immediately changing states.
+ * tcpm_state_machine_work() will continue running the state
+ * machine.
+ */
+ if (!port->state_machine_running)
+ mod_delayed_work(port->wq, &port->state_machine, 0);
+ }
+}
+
+static void tcpm_set_state_cond(struct tcpm_port *port, enum tcpm_state state,
+ unsigned int delay_ms)
+{
+ if (port->enter_state == port->state)
+ tcpm_set_state(port, state, delay_ms);
+ else
+ tcpm_log(port,
+ "skipped %sstate change %s -> %s [%u ms], context state %s",
+ delay_ms ? "delayed " : "",
+ tcpm_states[port->state], tcpm_states[state],
+ delay_ms, tcpm_states[port->enter_state]);
+}
+
+static void tcpm_queue_message(struct tcpm_port *port,
+ enum pd_msg_request message)
+{
+ port->queued_message = message;
+ mod_delayed_work(port->wq, &port->state_machine, 0);
+}
+
+/*
+ * VDM/VDO handling functions
+ */
+static void tcpm_queue_vdm(struct tcpm_port *port, const u32 header,
+ const u32 *data, int cnt)
+{
+ port->vdo_count = cnt + 1;
+ port->vdo_data[0] = header;
+ memcpy(&port->vdo_data[1], data, sizeof(u32) * cnt);
+ /* Set ready, vdm state machine will actually send */
+ port->vdm_retries = 0;
+ port->vdm_state = VDM_STATE_READY;
+}
+
+static void svdm_consume_identity(struct tcpm_port *port, const __le32 *payload,
+ int cnt)
+{
+ u32 vdo = le32_to_cpu(payload[VDO_INDEX_IDH]);
+ u32 product = le32_to_cpu(payload[VDO_INDEX_PRODUCT]);
+
+ memset(&port->mode_data, 0, sizeof(port->mode_data));
+
+ port->partner_ident.id_header = vdo;
+ port->partner_ident.cert_stat = le32_to_cpu(payload[VDO_INDEX_CSTAT]);
+ port->partner_ident.product = product;
+
+ typec_partner_set_identity(port->partner);
+
+ tcpm_log(port, "Identity: %04x:%04x.%04x",
+ PD_IDH_VID(vdo),
+ PD_PRODUCT_PID(product), product & 0xffff);
+}
+
+static bool svdm_consume_svids(struct tcpm_port *port, const __le32 *payload,
+ int cnt)
+{
+ struct pd_mode_data *pmdata = &port->mode_data;
+ int i;
+
+ for (i = 1; i < cnt; i++) {
+ u32 p = le32_to_cpu(payload[i]);
+ u16 svid;
+
+ svid = (p >> 16) & 0xffff;
+ if (!svid)
+ return false;
+
+ if (pmdata->nsvids >= SVID_DISCOVERY_MAX)
+ goto abort;
+
+ pmdata->svids[pmdata->nsvids++] = svid;
+ tcpm_log(port, "SVID %d: 0x%x", pmdata->nsvids, svid);
+
+ svid = p & 0xffff;
+ if (!svid)
+ return false;
+
+ if (pmdata->nsvids >= SVID_DISCOVERY_MAX)
+ goto abort;
+
+ pmdata->svids[pmdata->nsvids++] = svid;
+ tcpm_log(port, "SVID %d: 0x%x", pmdata->nsvids, svid);
+ }
+ return true;
+abort:
+ tcpm_log(port, "SVID_DISCOVERY_MAX(%d) too low!", SVID_DISCOVERY_MAX);
+ return false;
+}
+
+static void svdm_consume_modes(struct tcpm_port *port, const __le32 *payload,
+ int cnt)
+{
+ struct pd_mode_data *pmdata = &port->mode_data;
+ struct typec_altmode_desc *paltmode;
+ struct typec_mode_desc *pmode;
+ int i;
+
+ if (pmdata->altmodes >= ARRAY_SIZE(port->partner_altmode)) {
+ /* Already logged in svdm_consume_svids() */
+ return;
+ }
+
+ paltmode = &pmdata->altmode_desc[pmdata->altmodes];
+ memset(paltmode, 0, sizeof(*paltmode));
+
+ paltmode->svid = pmdata->svids[pmdata->svid_index];
+
+ tcpm_log(port, " Alternate mode %d: SVID 0x%04x",
+ pmdata->altmodes, paltmode->svid);
+
+ for (i = 1; i < cnt && paltmode->n_modes < ALTMODE_MAX_MODES; i++) {
+ pmode = &paltmode->modes[paltmode->n_modes];
+ memset(pmode, 0, sizeof(*pmode));
+ pmode->vdo = le32_to_cpu(payload[i]);
+ pmode->index = i - 1;
+ paltmode->n_modes++;
+ tcpm_log(port, " VDO %d: 0x%08x",
+ pmode->index, pmode->vdo);
+ }
+ port->partner_altmode[pmdata->altmodes] =
+ typec_partner_register_altmode(port->partner, paltmode);
+ if (!port->partner_altmode[pmdata->altmodes]) {
+ tcpm_log(port,
+ "Failed to register alternate modes for SVID 0x%04x",
+ paltmode->svid);
+ return;
+ }
+ pmdata->altmodes++;
+}
+
+#define supports_modal(port) PD_IDH_MODAL_SUPP((port)->partner_ident.id_header)
+
+static int tcpm_pd_svdm(struct tcpm_port *port, const __le32 *payload, int cnt,
+ u32 *response)
+{
+ u32 p0 = le32_to_cpu(payload[0]);
+ int cmd_type = PD_VDO_CMDT(p0);
+ int cmd = PD_VDO_CMD(p0);
+ struct pd_mode_data *modep;
+ int rlen = 0;
+ u16 svid;
+ int i;
+
+ tcpm_log(port, "Rx VDM cmd 0x%x type %d cmd %d len %d",
+ p0, cmd_type, cmd, cnt);
+
+ modep = &port->mode_data;
+
+ switch (cmd_type) {
+ case CMDT_INIT:
+ switch (cmd) {
+ case CMD_DISCOVER_IDENT:
+ /* 6.4.4.3.1: Only respond as UFP (device) */
+ if (port->data_role == TYPEC_DEVICE &&
+ port->nr_snk_vdo) {
+ for (i = 0; i < port->nr_snk_vdo; i++)
+ response[i + 1] = port->snk_vdo[i];
+ rlen = port->nr_snk_vdo + 1;
+ }
+ break;
+ case CMD_DISCOVER_SVID:
+ break;
+ case CMD_DISCOVER_MODES:
+ break;
+ case CMD_ENTER_MODE:
+ break;
+ case CMD_EXIT_MODE:
+ break;
+ case CMD_ATTENTION:
+ break;
+ default:
+ break;
+ }
+ if (rlen >= 1) {
+ response[0] = p0 | VDO_CMDT(CMDT_RSP_ACK);
+ } else if (rlen == 0) {
+ response[0] = p0 | VDO_CMDT(CMDT_RSP_NAK);
+ rlen = 1;
+ } else {
+ response[0] = p0 | VDO_CMDT(CMDT_RSP_BUSY);
+ rlen = 1;
+ }
+ break;
+ case CMDT_RSP_ACK:
+ /* silently drop message if we are not connected */
+ if (!port->partner)
+ break;
+
+ switch (cmd) {
+ case CMD_DISCOVER_IDENT:
+ /* 6.4.4.3.1 */
+ svdm_consume_identity(port, payload, cnt);
+ response[0] = VDO(USB_SID_PD, 1, CMD_DISCOVER_SVID);
+ rlen = 1;
+ break;
+ case CMD_DISCOVER_SVID:
+ /* 6.4.4.3.2 */
+ if (svdm_consume_svids(port, payload, cnt)) {
+ response[0] = VDO(USB_SID_PD, 1,
+ CMD_DISCOVER_SVID);
+ rlen = 1;
+ } else if (modep->nsvids && supports_modal(port)) {
+ response[0] = VDO(modep->svids[0], 1,
+ CMD_DISCOVER_MODES);
+ rlen = 1;
+ }
+ break;
+ case CMD_DISCOVER_MODES:
+ /* 6.4.4.3.3 */
+ svdm_consume_modes(port, payload, cnt);
+ modep->svid_index++;
+ if (modep->svid_index < modep->nsvids) {
+ svid = modep->svids[modep->svid_index];
+ response[0] = VDO(svid, 1, CMD_DISCOVER_MODES);
+ rlen = 1;
+ } else {
+ /* enter alternate mode if/when implemented */
+ }
+ break;
+ case CMD_ENTER_MODE:
+ break;
+ default:
+ break;
+ }
+ break;
+ default:
+ break;
+ }
+
+ return rlen;
+}
+
+static void tcpm_handle_vdm_request(struct tcpm_port *port,
+ const __le32 *payload, int cnt)
+{
+ int rlen = 0;
+ u32 response[8] = { };
+ u32 p0 = le32_to_cpu(payload[0]);
+
+ if (port->vdm_state == VDM_STATE_BUSY) {
+ /* If UFP responded busy retry after timeout */
+ if (PD_VDO_CMDT(p0) == CMDT_RSP_BUSY) {
+ port->vdm_state = VDM_STATE_WAIT_RSP_BUSY;
+ port->vdo_retry = (p0 & ~VDO_CMDT_MASK) |
+ CMDT_INIT;
+ mod_delayed_work(port->wq, &port->vdm_state_machine,
+ msecs_to_jiffies(PD_T_VDM_BUSY));
+ return;
+ }
+ port->vdm_state = VDM_STATE_DONE;
+ }
+
+ if (PD_VDO_SVDM(p0))
+ rlen = tcpm_pd_svdm(port, payload, cnt, response);
+
+ if (rlen > 0) {
+ tcpm_queue_vdm(port, response[0], &response[1], rlen - 1);
+ mod_delayed_work(port->wq, &port->vdm_state_machine, 0);
+ }
+}
+
+static void tcpm_send_vdm(struct tcpm_port *port, u32 vid, int cmd,
+ const u32 *data, int count)
+{
+ u32 header;
+
+ if (WARN_ON(count > VDO_MAX_SIZE - 1))
+ count = VDO_MAX_SIZE - 1;
+
+ /* set VDM header with VID & CMD */
+ header = VDO(vid, ((vid & USB_SID_PD) == USB_SID_PD) ?
+ 1 : (PD_VDO_CMD(cmd) <= CMD_ATTENTION), cmd);
+ tcpm_queue_vdm(port, header, data, count);
+
+ mod_delayed_work(port->wq, &port->vdm_state_machine, 0);
+}
+
+static unsigned int vdm_ready_timeout(u32 vdm_hdr)
+{
+ unsigned int timeout;
+ int cmd = PD_VDO_CMD(vdm_hdr);
+
+ /* its not a structured VDM command */
+ if (!PD_VDO_SVDM(vdm_hdr))
+ return PD_T_VDM_UNSTRUCTURED;
+
+ switch (PD_VDO_CMDT(vdm_hdr)) {
+ case CMDT_INIT:
+ if (cmd == CMD_ENTER_MODE || cmd == CMD_EXIT_MODE)
+ timeout = PD_T_VDM_WAIT_MODE_E;
+ else
+ timeout = PD_T_VDM_SNDR_RSP;
+ break;
+ default:
+ if (cmd == CMD_ENTER_MODE || cmd == CMD_EXIT_MODE)
+ timeout = PD_T_VDM_E_MODE;
+ else
+ timeout = PD_T_VDM_RCVR_RSP;
+ break;
+ }
+ return timeout;
+}
+
+static void vdm_run_state_machine(struct tcpm_port *port)
+{
+ struct pd_message msg;
+ int i, res;
+
+ switch (port->vdm_state) {
+ case VDM_STATE_READY:
+ /* Only transmit VDM if attached */
+ if (!port->attached) {
+ port->vdm_state = VDM_STATE_ERR_BUSY;
+ break;
+ }
+
+ /*
+ * if there's traffic or we're not in PDO ready state don't send
+ * a VDM.
+ */
+ if (port->state != SRC_READY && port->state != SNK_READY)
+ break;
+
+ /* Prepare and send VDM */
+ memset(&msg, 0, sizeof(msg));
+ msg.header = PD_HEADER_LE(PD_DATA_VENDOR_DEF,
+ port->pwr_role,
+ port->data_role,
+ port->message_id, port->vdo_count);
+ for (i = 0; i < port->vdo_count; i++)
+ msg.payload[i] = cpu_to_le32(port->vdo_data[i]);
+ res = tcpm_pd_transmit(port, TCPC_TX_SOP, &msg);
+ if (res < 0) {
+ port->vdm_state = VDM_STATE_ERR_SEND;
+ } else {
+ unsigned long timeout;
+
+ port->vdm_retries = 0;
+ port->vdm_state = VDM_STATE_BUSY;
+ timeout = vdm_ready_timeout(port->vdo_data[0]);
+ mod_delayed_work(port->wq, &port->vdm_state_machine,
+ timeout);
+ }
+ break;
+ case VDM_STATE_WAIT_RSP_BUSY:
+ port->vdo_data[0] = port->vdo_retry;
+ port->vdo_count = 1;
+ port->vdm_state = VDM_STATE_READY;
+ break;
+ case VDM_STATE_BUSY:
+ port->vdm_state = VDM_STATE_ERR_TMOUT;
+ break;
+ case VDM_STATE_ERR_SEND:
+ /*
+ * A partner which does not support USB PD will not reply,
+ * so this is not a fatal error. At the same time, some
+ * devices may not return GoodCRC under some circumstances,
+ * so we need to retry.
+ */
+ if (port->vdm_retries < 3) {
+ tcpm_log(port, "VDM Tx error, retry");
+ port->vdm_retries++;
+ port->vdm_state = VDM_STATE_READY;
+ }
+ break;
+ default:
+ break;
+ }
+}
+
+static void vdm_state_machine_work(struct work_struct *work)
+{
+ struct tcpm_port *port = container_of(work, struct tcpm_port,
+ vdm_state_machine.work);
+ enum vdm_states prev_state;
+
+ mutex_lock(&port->lock);
+
+ /*
+ * Continue running as long as the port is not busy and there was
+ * a state change.
+ */
+ do {
+ prev_state = port->vdm_state;
+ vdm_run_state_machine(port);
+ } while (port->vdm_state != prev_state &&
+ port->vdm_state != VDM_STATE_BUSY);
+
+ mutex_unlock(&port->lock);
+}
+
+enum pdo_err {
+ PDO_NO_ERR,
+ PDO_ERR_NO_VSAFE5V,
+ PDO_ERR_VSAFE5V_NOT_FIRST,
+ PDO_ERR_PDO_TYPE_NOT_IN_ORDER,
+ PDO_ERR_FIXED_NOT_SORTED,
+ PDO_ERR_VARIABLE_BATT_NOT_SORTED,
+ PDO_ERR_DUPE_PDO,
+};
+
+static const char * const pdo_err_msg[] = {
+ [PDO_ERR_NO_VSAFE5V] =
+ " err: source/sink caps should atleast have vSafe5V",
+ [PDO_ERR_VSAFE5V_NOT_FIRST] =
+ " err: vSafe5V Fixed Supply Object Shall always be the first object",
+ [PDO_ERR_PDO_TYPE_NOT_IN_ORDER] =
+ " err: PDOs should be in the following order: Fixed; Battery; Variable",
+ [PDO_ERR_FIXED_NOT_SORTED] =
+ " err: Fixed supply pdos should be in increasing order of their fixed voltage",
+ [PDO_ERR_VARIABLE_BATT_NOT_SORTED] =
+ " err: Variable/Battery supply pdos should be in increasing order of their minimum voltage",
+ [PDO_ERR_DUPE_PDO] =
+ " err: Variable/Batt supply pdos cannot have same min/max voltage",
+};
+
+static enum pdo_err tcpm_caps_err(struct tcpm_port *port, const u32 *pdo,
+ unsigned int nr_pdo)
+{
+ unsigned int i;
+
+ /* Should at least contain vSafe5v */
+ if (nr_pdo < 1)
+ return PDO_ERR_NO_VSAFE5V;
+
+ /* The vSafe5V Fixed Supply Object Shall always be the first object */
+ if (pdo_type(pdo[0]) != PDO_TYPE_FIXED ||
+ pdo_fixed_voltage(pdo[0]) != VSAFE5V)
+ return PDO_ERR_VSAFE5V_NOT_FIRST;
+
+ for (i = 1; i < nr_pdo; i++) {
+ if (pdo_type(pdo[i]) < pdo_type(pdo[i - 1])) {
+ return PDO_ERR_PDO_TYPE_NOT_IN_ORDER;
+ } else if (pdo_type(pdo[i]) == pdo_type(pdo[i - 1])) {
+ enum pd_pdo_type type = pdo_type(pdo[i]);
+
+ switch (type) {
+ /*
+ * The remaining Fixed Supply Objects, if
+ * present, shall be sent in voltage order;
+ * lowest to highest.
+ */
+ case PDO_TYPE_FIXED:
+ if (pdo_fixed_voltage(pdo[i]) <=
+ pdo_fixed_voltage(pdo[i - 1]))
+ return PDO_ERR_FIXED_NOT_SORTED;
+ break;
+ /*
+ * The Battery Supply Objects and Variable
+ * supply, if present shall be sent in Minimum
+ * Voltage order; lowest to highest.
+ */
+ case PDO_TYPE_VAR:
+ case PDO_TYPE_BATT:
+ if (pdo_min_voltage(pdo[i]) <
+ pdo_min_voltage(pdo[i - 1]))
+ return PDO_ERR_VARIABLE_BATT_NOT_SORTED;
+ else if ((pdo_min_voltage(pdo[i]) ==
+ pdo_min_voltage(pdo[i - 1])) &&
+ (pdo_max_voltage(pdo[i]) ==
+ pdo_min_voltage(pdo[i - 1])))
+ return PDO_ERR_DUPE_PDO;
+ break;
+ default:
+ tcpm_log_force(port, " Unknown pdo type");
+ }
+ }
+ }
+
+ return PDO_NO_ERR;
+}
+
+static int tcpm_validate_caps(struct tcpm_port *port, const u32 *pdo,
+ unsigned int nr_pdo)
+{
+ enum pdo_err err_index = tcpm_caps_err(port, pdo, nr_pdo);
+
+ if (err_index != PDO_NO_ERR) {
+ tcpm_log_force(port, " %s", pdo_err_msg[err_index]);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+/*
+ * PD (data, control) command handling functions
+ */
+static void tcpm_pd_data_request(struct tcpm_port *port,
+ const struct pd_message *msg)
+{
+ enum pd_data_msg_type type = pd_header_type_le(msg->header);
+ unsigned int cnt = pd_header_cnt_le(msg->header);
+ unsigned int i;
+
+ switch (type) {
+ case PD_DATA_SOURCE_CAP:
+ if (port->pwr_role != TYPEC_SINK)
+ break;
+
+ for (i = 0; i < cnt; i++)
+ port->source_caps[i] = le32_to_cpu(msg->payload[i]);
+
+ port->nr_source_caps = cnt;
+
+ tcpm_log_source_caps(port);
+
+ tcpm_validate_caps(port, port->source_caps,
+ port->nr_source_caps);
+
+ /*
+ * This message may be received even if VBUS is not
+ * present. This is quite unexpected; see USB PD
+ * specification, sections 8.3.3.6.3.1 and 8.3.3.6.3.2.
+ * However, at the same time, we must be ready to
+ * receive this message and respond to it 15ms after
+ * receiving PS_RDY during power swap operations, no matter
+ * if VBUS is available or not (USB PD specification,
+ * section 6.5.9.2).
+ * So we need to accept the message either way,
+ * but be prepared to keep waiting for VBUS after it was
+ * handled.
+ */
+ tcpm_set_state(port, SNK_NEGOTIATE_CAPABILITIES, 0);
+ break;
+ case PD_DATA_REQUEST:
+ if (port->pwr_role != TYPEC_SOURCE ||
+ cnt != 1) {
+ tcpm_queue_message(port, PD_MSG_CTRL_REJECT);
+ break;
+ }
+ port->sink_request = le32_to_cpu(msg->payload[0]);
+ tcpm_set_state(port, SRC_NEGOTIATE_CAPABILITIES, 0);
+ break;
+ case PD_DATA_SINK_CAP:
+ /* We don't do anything with this at the moment... */
+ for (i = 0; i < cnt; i++)
+ port->sink_caps[i] = le32_to_cpu(msg->payload[i]);
+ port->nr_sink_caps = cnt;
+ break;
+ case PD_DATA_VENDOR_DEF:
+ tcpm_handle_vdm_request(port, msg->payload, cnt);
+ break;
+ case PD_DATA_BIST:
+ if (port->state == SRC_READY || port->state == SNK_READY) {
+ port->bist_request = le32_to_cpu(msg->payload[0]);
+ tcpm_set_state(port, BIST_RX, 0);
+ }
+ break;
+ default:
+ tcpm_log(port, "Unhandled data message type %#x", type);
+ break;
+ }
+}
+
+static void tcpm_pd_ctrl_request(struct tcpm_port *port,
+ const struct pd_message *msg)
+{
+ enum pd_ctrl_msg_type type = pd_header_type_le(msg->header);
+ enum tcpm_state next_state;
+
+ switch (type) {
+ case PD_CTRL_GOOD_CRC:
+ case PD_CTRL_PING:
+ break;
+ case PD_CTRL_GET_SOURCE_CAP:
+ switch (port->state) {
+ case SRC_READY:
+ case SNK_READY:
+ tcpm_queue_message(port, PD_MSG_DATA_SOURCE_CAP);
+ break;
+ default:
+ tcpm_queue_message(port, PD_MSG_CTRL_REJECT);
+ break;
+ }
+ break;
+ case PD_CTRL_GET_SINK_CAP:
+ switch (port->state) {
+ case SRC_READY:
+ case SNK_READY:
+ tcpm_queue_message(port, PD_MSG_DATA_SINK_CAP);
+ break;
+ default:
+ tcpm_queue_message(port, PD_MSG_CTRL_REJECT);
+ break;
+ }
+ break;
+ case PD_CTRL_GOTO_MIN:
+ break;
+ case PD_CTRL_PS_RDY:
+ switch (port->state) {
+ case SNK_TRANSITION_SINK:
+ tcpm_log(port, "in PR_SWAP := false");
+ port->tcpc->set_in_pr_swap(port->tcpc, false);
+ if (port->vbus_present) {
+ tcpm_set_current_limit(port,
+ port->current_limit,
+ port->requested_min_mv,
+ port->requested_max_mv)
+ ;
+ port->explicit_contract = true;
+ tcpm_set_state(port, SNK_READY, 0);
+ } else {
+ /*
+ * Seen after power swap. Keep waiting for VBUS
+ * in a transitional state.
+ */
+ tcpm_set_state(port,
+ SNK_TRANSITION_SINK_VBUS, 0);
+ }
+ break;
+ case PR_SWAP_SRC_SNK_SOURCE_OFF_CC_DEBOUNCED:
+ tcpm_set_state(port, PR_SWAP_SRC_SNK_SINK_ON, 0);
+ break;
+ case PR_SWAP_SNK_SRC_SINK_OFF:
+ tcpm_set_state(port, PR_SWAP_SNK_SRC_SOURCE_ON, 0);
+ break;
+ case VCONN_SWAP_WAIT_FOR_VCONN:
+ tcpm_set_state(port, VCONN_SWAP_TURN_OFF_VCONN, 0);
+ break;
+ default:
+ break;
+ }
+ break;
+ case PD_CTRL_REJECT:
+ case PD_CTRL_WAIT:
+ switch (port->state) {
+ case SNK_NEGOTIATE_CAPABILITIES:
+ /* USB PD specification, Figure 8-43 */
+ if (port->explicit_contract)
+ next_state = SNK_READY;
+ else
+ next_state = SNK_WAIT_CAPABILITIES;
+ tcpm_set_state(port, next_state, 0);
+ break;
+ case DR_SWAP_SEND:
+ port->swap_status = (type == PD_CTRL_WAIT ?
+ -EAGAIN : -EOPNOTSUPP);
+ tcpm_set_state(port, DR_SWAP_CANCEL, 0);
+ break;
+ case PR_SWAP_SEND:
+ port->swap_status = (type == PD_CTRL_WAIT ?
+ -EAGAIN : -EOPNOTSUPP);
+ tcpm_set_state(port, PR_SWAP_CANCEL, 0);
+ break;
+ case VCONN_SWAP_SEND:
+ port->swap_status = (type == PD_CTRL_WAIT ?
+ -EAGAIN : -EOPNOTSUPP);
+ tcpm_set_state(port, VCONN_SWAP_CANCEL, 0);
+ break;
+ default:
+ break;
+ }
+ break;
+ case PD_CTRL_ACCEPT:
+ switch (port->state) {
+ case SNK_NEGOTIATE_CAPABILITIES:
+ tcpm_set_state(port, SNK_TRANSITION_SINK, 0);
+ break;
+ case SOFT_RESET_SEND:
+ port->message_id = 0;
+ port->rx_msgid = -1;
+ if (port->pwr_role == TYPEC_SOURCE)
+ next_state = SRC_SEND_CAPABILITIES;
+ else
+ next_state = SNK_WAIT_CAPABILITIES;
+ tcpm_set_state(port, next_state, 0);
+ break;
+ case DR_SWAP_SEND:
+ tcpm_set_state(port, DR_SWAP_CHANGE_DR, 0);
+ break;
+ case PR_SWAP_SEND:
+ tcpm_set_state(port, PR_SWAP_START, 0);
+ break;
+ case VCONN_SWAP_SEND:
+ tcpm_set_state(port, VCONN_SWAP_START, 0);
+ break;
+ default:
+ break;
+ }
+ break;
+ case PD_CTRL_SOFT_RESET:
+ tcpm_set_state(port, SOFT_RESET, 0);
+ break;
+ case PD_CTRL_DR_SWAP:
+ if (port->port_type != TYPEC_PORT_DRP) {
+ tcpm_queue_message(port, PD_MSG_CTRL_REJECT);
+ break;
+ }
+ /*
+ * XXX
+ * 6.3.9: If an alternate mode is active, a request to swap
+ * alternate modes shall trigger a port reset.
+ */
+ switch (port->state) {
+ case SRC_READY:
+ case SNK_READY:
+ tcpm_set_state(port, DR_SWAP_ACCEPT, 0);
+ break;
+ default:
+ tcpm_queue_message(port, PD_MSG_CTRL_WAIT);
+ break;
+ }
+ break;
+ case PD_CTRL_PR_SWAP:
+ if (port->port_type != TYPEC_PORT_DRP) {
+ tcpm_queue_message(port, PD_MSG_CTRL_REJECT);
+ break;
+ }
+ switch (port->state) {
+ case SRC_READY:
+ case SNK_READY:
+ tcpm_set_state(port, PR_SWAP_ACCEPT, 0);
+ break;
+ default:
+ tcpm_queue_message(port, PD_MSG_CTRL_WAIT);
+ break;
+ }
+ break;
+ case PD_CTRL_VCONN_SWAP:
+ switch (port->state) {
+ case SRC_READY:
+ case SNK_READY:
+ tcpm_set_state(port, VCONN_SWAP_ACCEPT, 0);
+ break;
+ default:
+ tcpm_queue_message(port, PD_MSG_CTRL_WAIT);
+ break;
+ }
+ break;
+ default:
+ tcpm_log(port, "Unhandled ctrl message type %#x", type);
+ break;
+ }
+}
+
+static void tcpm_pd_rx_handler(struct work_struct *work)
+{
+ struct pd_rx_event *event = container_of(work,
+ struct pd_rx_event, work);
+ const struct pd_message *msg = &event->msg;
+ unsigned int cnt = pd_header_cnt_le(msg->header);
+ struct tcpm_port *port = event->port;
+
+ mutex_lock(&port->lock);
+
+ tcpm_log(port, "PD RX, header: %#x [%d]", le16_to_cpu(msg->header),
+ port->attached);
+
+ if (port->attached) {
+ enum pd_ctrl_msg_type type = pd_header_type_le(msg->header);
+ unsigned int msgid = pd_header_msgid_le(msg->header);
+
+ /*
+ * USB PD standard, 6.6.1.2:
+ * "... if MessageID value in a received Message is the
+ * same as the stored value, the receiver shall return a
+ * GoodCRC Message with that MessageID value and drop
+ * the Message (this is a retry of an already received
+ * Message). Note: this shall not apply to the Soft_Reset
+ * Message which always has a MessageID value of zero."
+ */
+ if (msgid == port->rx_msgid && type != PD_CTRL_SOFT_RESET)
+ goto done;
+ port->rx_msgid = msgid;
+
+ /*
+ * If both ends believe to be DFP/host, we have a data role
+ * mismatch.
+ */
+ if (!!(le16_to_cpu(msg->header) & PD_HEADER_DATA_ROLE) ==
+ (port->data_role == TYPEC_HOST)) {
+ tcpm_log(port,
+ "Data role mismatch, initiating error recovery");
+ tcpm_set_state(port, ERROR_RECOVERY, 0);
+ } else {
+ if (cnt)
+ tcpm_pd_data_request(port, msg);
+ else
+ tcpm_pd_ctrl_request(port, msg);
+ }
+ }
+
+done:
+ mutex_unlock(&port->lock);
+ kfree(event);
+}
+
+void tcpm_pd_receive(struct tcpm_port *port, const struct pd_message *msg)
+{
+ struct pd_rx_event *event;
+
+ event = kzalloc(sizeof(*event), GFP_ATOMIC);
+ if (!event)
+ return;
+
+ INIT_WORK(&event->work, tcpm_pd_rx_handler);
+ event->port = port;
+ memcpy(&event->msg, msg, sizeof(*msg));
+ queue_work(port->wq, &event->work);
+}
+EXPORT_SYMBOL_GPL(tcpm_pd_receive);
+
+static int tcpm_pd_send_control(struct tcpm_port *port,
+ enum pd_ctrl_msg_type type)
+{
+ struct pd_message msg;
+
+ memset(&msg, 0, sizeof(msg));
+ msg.header = PD_HEADER_LE(type, port->pwr_role,
+ port->data_role,
+ port->message_id, 0);
+
+ return tcpm_pd_transmit(port, TCPC_TX_SOP, &msg);
+}
+
+/*
+ * Send queued message without affecting state.
+ * Return true if state machine should go back to sleep,
+ * false otherwise.
+ */
+static bool tcpm_send_queued_message(struct tcpm_port *port)
+{
+ enum pd_msg_request queued_message;
+
+ do {
+ queued_message = port->queued_message;
+ port->queued_message = PD_MSG_NONE;
+
+ switch (queued_message) {
+ case PD_MSG_CTRL_WAIT:
+ tcpm_pd_send_control(port, PD_CTRL_WAIT);
+ break;
+ case PD_MSG_CTRL_REJECT:
+ tcpm_pd_send_control(port, PD_CTRL_REJECT);
+ break;
+ case PD_MSG_DATA_SINK_CAP:
+ tcpm_pd_send_sink_caps(port);
+ break;
+ case PD_MSG_DATA_SOURCE_CAP:
+ tcpm_pd_send_source_caps(port);
+ break;
+ default:
+ break;
+ }
+ } while (port->queued_message != PD_MSG_NONE);
+
+ if (port->delayed_state != INVALID_STATE) {
+ if (time_is_after_jiffies(port->delayed_runtime)) {
+ mod_delayed_work(port->wq, &port->state_machine,
+ port->delayed_runtime - jiffies);
+ return true;
+ }
+ port->delayed_state = INVALID_STATE;
+ }
+ return false;
+}
+
+static int tcpm_pd_check_request(struct tcpm_port *port)
+{
+ u32 pdo, rdo = port->sink_request;
+ unsigned int max, op, pdo_max, index;
+ enum pd_pdo_type type;
+
+ index = rdo_index(rdo);
+ if (!index || index > port->nr_src_pdo)
+ return -EINVAL;
+
+ pdo = port->src_pdo[index - 1];
+ type = pdo_type(pdo);
+ switch (type) {
+ case PDO_TYPE_FIXED:
+ case PDO_TYPE_VAR:
+ max = rdo_max_current(rdo);
+ op = rdo_op_current(rdo);
+ pdo_max = pdo_max_current(pdo);
+
+ if (op > pdo_max)
+ return -EINVAL;
+ if (max > pdo_max && !(rdo & RDO_CAP_MISMATCH))
+ return -EINVAL;
+
+ if (type == PDO_TYPE_FIXED)
+ tcpm_log(port,
+ "Requested %u mV, %u mA for %u / %u mA",
+ pdo_fixed_voltage(pdo), pdo_max, op, max);
+ else
+ tcpm_log(port,
+ "Requested %u -> %u mV, %u mA for %u / %u mA",
+ pdo_min_voltage(pdo), pdo_max_voltage(pdo),
+ pdo_max, op, max);
+ break;
+ case PDO_TYPE_BATT:
+ max = rdo_max_power(rdo);
+ op = rdo_op_power(rdo);
+ pdo_max = pdo_max_power(pdo);
+
+ if (op > pdo_max)
+ return -EINVAL;
+ if (max > pdo_max && !(rdo & RDO_CAP_MISMATCH))
+ return -EINVAL;
+ tcpm_log(port,
+ "Requested %u -> %u mV, %u mW for %u / %u mW",
+ pdo_min_voltage(pdo), pdo_max_voltage(pdo),
+ pdo_max, op, max);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ port->op_vsafe5v = index == 1;
+
+ return 0;
+}
+
+#define min_power(x, y) min(pdo_max_power(x), pdo_max_power(y))
+#define min_current(x, y) min(pdo_max_current(x), pdo_max_current(y))
+
+static int tcpm_pd_select_pdo(struct tcpm_port *port, int *sink_pdo,
+ int *src_pdo)
+{
+ unsigned int i, j, max_mw = 0, max_mv = 0, mw = 0, mv = 0, ma = 0;
+ int ret = -EINVAL;
+
+ /*
+ * Select the source PDO providing the most power which has a
+ * matchig sink cap.
+ */
+ for (i = 0; i < port->nr_source_caps; i++) {
+ u32 pdo = port->source_caps[i];
+ enum pd_pdo_type type = pdo_type(pdo);
+
+ if (type == PDO_TYPE_FIXED) {
+ for (j = 0; j < port->nr_fixed; j++) {
+ if (pdo_fixed_voltage(pdo) ==
+ pdo_fixed_voltage(port->snk_pdo[j])) {
+ ma = min_current(pdo, port->snk_pdo[j]);
+ mv = pdo_fixed_voltage(pdo);
+ mw = ma * mv / 1000;
+ if (mw > max_mw ||
+ (mw == max_mw && mv > max_mv)) {
+ ret = 0;
+ *src_pdo = i;
+ *sink_pdo = j;
+ max_mw = mw;
+ max_mv = mv;
+ }
+ /* There could only be one fixed pdo
+ * at a specific voltage level.
+ * So breaking here.
+ */
+ break;
+ }
+ }
+ } else if (type == PDO_TYPE_BATT) {
+ for (j = port->nr_fixed;
+ j < port->nr_fixed +
+ port->nr_batt;
+ j++) {
+ if (pdo_min_voltage(pdo) >=
+ pdo_min_voltage(port->snk_pdo[j]) &&
+ pdo_max_voltage(pdo) <=
+ pdo_max_voltage(port->snk_pdo[j])) {
+ mw = min_power(pdo, port->snk_pdo[j]);
+ mv = pdo_min_voltage(pdo);
+ if (mw > max_mw ||
+ (mw == max_mw && mv > max_mv)) {
+ ret = 0;
+ *src_pdo = i;
+ *sink_pdo = j;
+ max_mw = mw;
+ max_mv = mv;
+ }
+ }
+ }
+ } else if (type == PDO_TYPE_VAR) {
+ for (j = port->nr_fixed +
+ port->nr_batt;
+ j < port->nr_fixed +
+ port->nr_batt +
+ port->nr_var;
+ j++) {
+ if (pdo_min_voltage(pdo) >=
+ pdo_min_voltage(port->snk_pdo[j]) &&
+ pdo_max_voltage(pdo) <=
+ pdo_max_voltage(port->snk_pdo[j])) {
+ ma = min_current(pdo, port->snk_pdo[j]);
+ mv = pdo_min_voltage(pdo);
+ mw = ma * mv / 1000;
+ if (mw > max_mw ||
+ (mw == max_mw && mv > max_mv)) {
+ ret = 0;
+ *src_pdo = i;
+ *sink_pdo = j;
+ max_mw = mw;
+ max_mv = mv;
+ }
+ }
+ }
+ }
+ }
+
+ return ret;
+}
+
+static int tcpm_pd_build_request(struct tcpm_port *port, u32 *rdo)
+{
+ unsigned int mv, ma, mw, flags;
+ unsigned int max_ma, max_mw;
+ enum pd_pdo_type type;
+ int src_pdo_index, snk_pdo_index;
+ u32 pdo, matching_snk_pdo;
+
+ if (tcpm_pd_select_pdo(port, &snk_pdo_index, &src_pdo_index) < 0)
+ return -EINVAL;
+
+ pdo = port->source_caps[src_pdo_index];
+ matching_snk_pdo = port->snk_pdo[snk_pdo_index];
+ type = pdo_type(pdo);
+
+ if (type == PDO_TYPE_FIXED)
+ mv = pdo_fixed_voltage(pdo);
+ else
+ mv = pdo_min_voltage(pdo);
+
+ /* Select maximum available current within the sink pdo's limit */
+ if (type == PDO_TYPE_BATT) {
+ mw = min_power(pdo, matching_snk_pdo);
+ ma = 1000 * mw / mv;
+ } else {
+ ma = min_current(pdo, matching_snk_pdo);
+ mw = ma * mv / 1000;
+ }
+
+ flags = RDO_USB_COMM | RDO_NO_SUSPEND;
+
+ /* Set mismatch bit if offered power is less than operating power */
+ max_ma = ma;
+ max_mw = mw;
+ if (mw < port->operating_snk_mw) {
+ flags |= RDO_CAP_MISMATCH;
+ if (type == PDO_TYPE_BATT &&
+ (pdo_max_power(matching_snk_pdo) > pdo_max_power(pdo)))
+ max_mw = pdo_max_power(matching_snk_pdo);
+ else if (pdo_max_current(matching_snk_pdo) >
+ pdo_max_current(pdo))
+ max_ma = pdo_max_current(matching_snk_pdo);
+ }
+
+ tcpm_log(port, "cc=%d cc1=%d cc2=%d vbus=%d vconn=%s polarity=%d",
+ port->cc_req, port->cc1, port->cc2, port->vbus_source,
+ port->vconn_role == TYPEC_SOURCE ? "source" : "sink",
+ port->polarity);
+
+ if (type == PDO_TYPE_BATT) {
+ *rdo = RDO_BATT(src_pdo_index + 1, mw, max_mw, flags);
+
+ tcpm_log(port, "Requesting PDO %d: %u mV, %u mW%s",
+ src_pdo_index, mv, mw,
+ flags & RDO_CAP_MISMATCH ? " [mismatch]" : "");
+ } else {
+ *rdo = RDO_FIXED(src_pdo_index + 1, ma, max_ma, flags);
+
+ tcpm_log(port, "Requesting PDO %d: %u mV, %u mA%s",
+ src_pdo_index, mv, ma,
+ flags & RDO_CAP_MISMATCH ? " [mismatch]" : "");
+ }
+
+ port->current_limit = ma;
+ port->requested_max_mv = mv;
+ port->requested_min_mv = mv;
+
+ return 0;
+}
+
+static int tcpm_pd_send_request(struct tcpm_port *port)
+{
+ struct pd_message msg;
+ int ret;
+ u32 rdo;
+
+ ret = tcpm_pd_build_request(port, &rdo);
+ if (ret < 0)
+ return ret;
+
+ memset(&msg, 0, sizeof(msg));
+ msg.header = PD_HEADER_LE(PD_DATA_REQUEST,
+ port->pwr_role,
+ port->data_role,
+ port->message_id, 1);
+ msg.payload[0] = cpu_to_le32(rdo);
+
+ return tcpm_pd_transmit(port, TCPC_TX_SOP, &msg);
+}
+
+static int tcpm_set_vbus(struct tcpm_port *port, bool enable)
+{
+ int ret;
+
+ if (enable && port->vbus_charge)
+ return -EINVAL;
+
+ tcpm_log(port, "vbus:=%d charge=%d", enable, port->vbus_charge);
+
+ ret = port->tcpc->set_vbus(port->tcpc, enable, port->vbus_charge);
+ if (ret < 0)
+ return ret;
+
+ port->vbus_source = enable;
+ return 0;
+}
+
+static int tcpm_set_charge(struct tcpm_port *port, bool charge)
+{
+ int ret;
+
+ if (charge && port->vbus_source)
+ return -EINVAL;
+
+ if (charge != port->vbus_charge) {
+ tcpm_log(port, "vbus=%d charge:=%d", port->vbus_source, charge);
+ ret = port->tcpc->set_vbus(port->tcpc, port->vbus_source,
+ charge);
+ if (ret < 0)
+ return ret;
+ }
+ port->vbus_charge = charge;
+ return 0;
+}
+
+static bool tcpm_start_drp_toggling(struct tcpm_port *port)
+{
+ int ret;
+
+ if (port->tcpc->start_drp_toggling &&
+ port->port_type == TYPEC_PORT_DRP) {
+ tcpm_log_force(port, "Start DRP toggling");
+ ret = port->tcpc->start_drp_toggling(port->tcpc,
+ tcpm_rp_cc(port));
+ if (!ret)
+ return true;
+ }
+
+ return false;
+}
+
+static void tcpm_set_cc(struct tcpm_port *port, enum typec_cc_status cc)
+{
+ tcpm_log(port, "cc:=%d", cc);
+ port->cc_req = cc;
+ port->tcpc->set_cc(port->tcpc, cc);
+}
+
+static int tcpm_init_vbus(struct tcpm_port *port)
+{
+ int ret;
+
+ ret = port->tcpc->set_vbus(port->tcpc, false, false);
+ port->vbus_source = false;
+ port->vbus_charge = false;
+ return ret;
+}
+
+static int tcpm_init_vconn(struct tcpm_port *port)
+{
+ int ret;
+
+ ret = port->tcpc->set_vconn(port->tcpc, false);
+ port->vconn_role = TYPEC_SINK;
+ return ret;
+}
+
+static void tcpm_typec_connect(struct tcpm_port *port)
+{
+ if (!port->connected) {
+ /* Make sure we don't report stale identity information */
+ memset(&port->partner_ident, 0, sizeof(port->partner_ident));
+ port->partner_desc.usb_pd = port->pd_capable;
+ if (tcpm_port_is_debug(port))
+ port->partner_desc.accessory = TYPEC_ACCESSORY_DEBUG;
+ else if (tcpm_port_is_audio(port))
+ port->partner_desc.accessory = TYPEC_ACCESSORY_AUDIO;
+ else
+ port->partner_desc.accessory = TYPEC_ACCESSORY_NONE;
+ port->partner = typec_register_partner(port->typec_port,
+ &port->partner_desc);
+ port->connected = true;
+ }
+}
+
+static int tcpm_src_attach(struct tcpm_port *port)
+{
+ enum typec_cc_polarity polarity =
+ port->cc2 == TYPEC_CC_RD ? TYPEC_POLARITY_CC2
+ : TYPEC_POLARITY_CC1;
+ int ret;
+
+ if (port->attached)
+ return 0;
+
+ ret = tcpm_set_polarity(port, polarity);
+ if (ret < 0)
+ return ret;
+
+ ret = tcpm_set_roles(port, true, TYPEC_SOURCE, TYPEC_HOST);
+ if (ret < 0)
+ return ret;
+
+ tcpm_set_cc(port, tcpm_rp_cc(port));
+
+ ret = port->tcpc->set_pd_rx(port->tcpc, true);
+ if (ret < 0)
+ goto out_disable_mux;
+
+ /*
+ * USB Type-C specification, version 1.2,
+ * chapter 4.5.2.2.8.1 (Attached.SRC Requirements)
+ * Enable VCONN only if the non-RD port is set to RA.
+ */
+ if ((polarity == TYPEC_POLARITY_CC1 && port->cc2 == TYPEC_CC_RA) ||
+ (polarity == TYPEC_POLARITY_CC2 && port->cc1 == TYPEC_CC_RA)) {
+ ret = tcpm_set_vconn(port, true);
+ if (ret < 0)
+ goto out_disable_pd;
+ }
+
+ ret = tcpm_set_vbus(port, true);
+ if (ret < 0)
+ goto out_disable_vconn;
+
+ tcpm_set_pd_capable(port, false);
+
+ port->partner = NULL;
+
+ port->attached = true;
+ port->send_discover = true;
+
+ return 0;
+
+out_disable_vconn:
+ tcpm_set_vconn(port, false);
+out_disable_pd:
+ port->tcpc->set_pd_rx(port->tcpc, false);
+out_disable_mux:
+ tcpm_mux_set(port, TYPEC_MUX_NONE, TCPC_USB_SWITCH_DISCONNECT);
+ return ret;
+}
+
+static void tcpm_typec_disconnect(struct tcpm_port *port)
+{
+ if (port->connected) {
+ typec_unregister_partner(port->partner);
+ port->partner = NULL;
+ port->connected = false;
+ }
+}
+
+static void tcpm_unregister_altmodes(struct tcpm_port *port)
+{
+ struct pd_mode_data *modep = &port->mode_data;
+ int i;
+
+ for (i = 0; i < modep->altmodes; i++) {
+ typec_unregister_altmode(port->partner_altmode[i]);
+ port->partner_altmode[i] = NULL;
+ }
+
+ memset(modep, 0, sizeof(*modep));
+}
+
+static void update_pd_contract_status(struct tcpm_port *port,
+ bool in_explicit_contract)
+{
+ if (port->tcpc->in_pd_contract)
+ port->tcpc->in_pd_contract(port->tcpc, in_explicit_contract);
+}
+
+
+static void tcpm_reset_port(struct tcpm_port *port)
+{
+ tcpm_unregister_altmodes(port);
+ tcpm_typec_disconnect(port);
+ port->attached = false;
+ tcpm_set_pd_capable(port, false);
+ update_pd_contract_status(port, false);
+ port->usb_comm_capable = false;
+
+ /*
+ * First Rx ID should be 0; set this to a sentinel of -1 so that
+ * we can check tcpm_pd_rx_handler() if we had seen it before.
+ */
+ port->rx_msgid = -1;
+
+ port->tcpc->set_pd_rx(port->tcpc, false);
+ tcpm_init_vbus(port); /* also disables charging */
+ tcpm_init_vconn(port);
+ tcpm_set_current_limit(port, 0, 0, 0);
+ tcpm_set_polarity(port, TYPEC_POLARITY_CC1);
+ tcpm_set_attached_state(port, false);
+ port->try_src_count = 0;
+ port->try_snk_count = 0;
+}
+
+static void tcpm_detach(struct tcpm_port *port)
+{
+ if (!port->attached)
+ return;
+
+ if (tcpm_port_is_disconnected(port))
+ port->hard_reset_count = 0;
+
+ tcpm_reset_port(port);
+}
+
+static void tcpm_src_detach(struct tcpm_port *port)
+{
+ tcpm_detach(port);
+}
+
+static int tcpm_snk_attach(struct tcpm_port *port)
+{
+ int ret;
+
+ if (port->attached)
+ return 0;
+
+ ret = tcpm_set_polarity(port, port->cc2 != TYPEC_CC_OPEN ?
+ TYPEC_POLARITY_CC2 : TYPEC_POLARITY_CC1);
+ if (ret < 0)
+ return ret;
+
+ ret = tcpm_set_roles(port, true, TYPEC_SINK, TYPEC_DEVICE);
+ if (ret < 0)
+ return ret;
+
+ tcpm_set_pd_capable(port, false);
+
+ port->partner = NULL;
+
+ port->attached = true;
+ port->send_discover = true;
+
+ return 0;
+}
+
+static void tcpm_snk_detach(struct tcpm_port *port)
+{
+ /* Conservatively reset back to suspend should be honored */
+ if (port->tcpc->set_suspend_supported)
+ port->tcpc->set_suspend_supported(port->tcpc, true);
+ tcpm_detach(port);
+
+ /* XXX: (Dis)connect SuperSpeed mux? */
+}
+
+static int tcpm_acc_attach(struct tcpm_port *port)
+{
+ int ret;
+
+ if (port->attached)
+ return 0;
+
+ ret = tcpm_set_roles(port, true, TYPEC_SOURCE, TYPEC_HOST);
+ if (ret < 0)
+ return ret;
+
+ port->partner = NULL;
+
+ tcpm_typec_connect(port);
+
+ port->attached = true;
+
+ return 0;
+}
+
+static void tcpm_acc_detach(struct tcpm_port *port)
+{
+ tcpm_detach(port);
+}
+
+static inline enum tcpm_state hard_reset_state(struct tcpm_port *port)
+{
+ if (port->hard_reset_count < PD_N_HARD_RESET_COUNT)
+ return HARD_RESET_SEND;
+ if (port->pd_capable)
+ return ERROR_RECOVERY;
+ if (port->pwr_role == TYPEC_SOURCE)
+ return SRC_UNATTACHED;
+ if (port->state == SNK_WAIT_CAPABILITIES)
+ return SNK_READY;
+ return SNK_UNATTACHED;
+}
+
+static inline enum tcpm_state ready_state(struct tcpm_port *port)
+{
+ if (port->pwr_role == TYPEC_SOURCE)
+ return SRC_READY;
+ else
+ return SNK_READY;
+}
+
+static inline enum tcpm_state unattached_state(struct tcpm_port *port)
+{
+ if (port->port_type == TYPEC_PORT_DRP) {
+ if (port->pwr_role == TYPEC_SOURCE)
+ return SRC_UNATTACHED;
+ else
+ return SNK_UNATTACHED;
+ } else if (port->port_type == TYPEC_PORT_DFP) {
+ return SRC_UNATTACHED;
+ }
+
+ return SNK_UNATTACHED;
+}
+
+static void tcpm_check_send_discover(struct tcpm_port *port)
+{
+ if (port->data_role == TYPEC_HOST && port->send_discover &&
+ port->pd_capable) {
+ tcpm_send_vdm(port, USB_SID_PD, CMD_DISCOVER_IDENT, NULL, 0);
+ port->send_discover = false;
+ }
+}
+
+static void tcpm_swap_complete(struct tcpm_port *port, int result)
+{
+ if (port->swap_pending) {
+ port->swap_status = result;
+ port->swap_pending = false;
+ port->non_pd_role_swap = false;
+ complete(&port->swap_complete);
+ }
+}
+
+static enum typec_pwr_opmode tcpm_get_pwr_opmode(enum typec_cc_status cc)
+{
+ switch (cc) {
+ case TYPEC_CC_RP_1_5:
+ return TYPEC_PWR_MODE_1_5A;
+ case TYPEC_CC_RP_3_0:
+ return TYPEC_PWR_MODE_3_0A;
+ case TYPEC_CC_RP_DEF:
+ default:
+ return TYPEC_PWR_MODE_USB;
+ }
+}
+
+static void run_state_machine(struct tcpm_port *port)
+{
+ int ret;
+ enum typec_pwr_opmode opmode;
+ unsigned int msecs;
+ u32 min_mv, max_mv, max_ma;
+
+ port->enter_state = port->state;
+ switch (port->state) {
+ case DRP_TOGGLING:
+ tcpm_log(port, "in PR_SWAP := false");
+ port->tcpc->set_in_pr_swap(port->tcpc, false);
+ break;
+ /* SRC states */
+ case SRC_UNATTACHED:
+ tcpm_port_in_hard_reset(port, false);
+ if (!port->non_pd_role_swap)
+ tcpm_swap_complete(port, -ENOTCONN);
+ tcpm_src_detach(port);
+ if (tcpm_start_drp_toggling(port)) {
+ tcpm_set_state(port, DRP_TOGGLING, 0);
+ break;
+ }
+ tcpm_set_cc(port, tcpm_rp_cc(port));
+ if (port->port_type == TYPEC_PORT_DRP)
+ tcpm_set_state(port, SNK_UNATTACHED, PD_T_DRP_SNK);
+ break;
+ case SRC_ATTACH_WAIT:
+ if (tcpm_port_is_debug(port))
+ tcpm_set_state(port, DEBUG_ACC_ATTACHED,
+ PD_T_CC_DEBOUNCE);
+ else if (tcpm_port_is_audio(port))
+ tcpm_set_state(port, AUDIO_ACC_ATTACHED,
+ PD_T_CC_DEBOUNCE);
+ else if (tcpm_port_is_source(port))
+ tcpm_set_state(port,
+ tcpm_try_snk(port) ? SNK_TRY
+ : SRC_ATTACHED,
+ PD_T_CC_DEBOUNCE);
+ break;
+
+ case SNK_TRY:
+ port->try_snk_count++;
+ /*
+ * Requirements:
+ * - Do not drive vconn or vbus
+ * - Terminate CC pins (both) to Rd
+ * Action:
+ * - Wait for tDRPTry (PD_T_DRP_TRY).
+ * Until then, ignore any state changes.
+ */
+ tcpm_set_cc(port, TYPEC_CC_RD);
+ tcpm_set_state(port, SNK_TRY_WAIT, PD_T_DRP_TRY);
+ break;
+ case SNK_TRY_WAIT:
+ if (tcpm_port_is_sink(port)) {
+ tcpm_set_state(port, SNK_TRY_WAIT_DEBOUNCE, 0);
+ } else {
+ tcpm_set_state(port, SRC_TRYWAIT, 0);
+ port->max_wait = 0;
+ }
+ break;
+ case SNK_TRY_WAIT_DEBOUNCE:
+ tcpm_set_state(port, SNK_TRY_WAIT_DEBOUNCE_CHECK_VBUS,
+ PD_T_PD_DEBOUNCE);
+ break;
+ case SNK_TRY_WAIT_DEBOUNCE_CHECK_VBUS:
+ if (port->vbus_present && tcpm_port_is_sink(port)) {
+ tcpm_set_state(port, SNK_ATTACHED, 0);
+ } else {
+ tcpm_set_state(port, SRC_TRYWAIT, 0);
+ port->max_wait = 0;
+ }
+ break;
+ case SRC_TRYWAIT:
+ tcpm_set_cc(port, tcpm_rp_cc(port));
+ if (port->max_wait == 0) {
+ port->max_wait = jiffies +
+ msecs_to_jiffies(PD_T_DRP_TRY);
+ tcpm_set_state(port, SRC_TRYWAIT_UNATTACHED,
+ PD_T_DRP_TRY);
+ } else {
+ if (time_is_after_jiffies(port->max_wait))
+ tcpm_set_state(port, SRC_TRYWAIT_UNATTACHED,
+ jiffies_to_msecs(port->max_wait -
+ jiffies));
+ else
+ tcpm_set_state(port, SNK_UNATTACHED, 0);
+ }
+ break;
+ case SRC_TRYWAIT_DEBOUNCE:
+ tcpm_set_state(port, SRC_ATTACHED, PD_T_CC_DEBOUNCE);
+ break;
+ case SRC_TRYWAIT_UNATTACHED:
+ tcpm_set_state(port, SNK_UNATTACHED, 0);
+ break;
+
+ case SRC_ATTACHED:
+ ret = tcpm_src_attach(port);
+ tcpm_set_state(port, SRC_UNATTACHED,
+ ret < 0 ? 0 : PD_T_PS_SOURCE_ON);
+ break;
+ case SRC_STARTUP:
+ opmode = tcpm_get_pwr_opmode(tcpm_rp_cc(port));
+ typec_set_pwr_opmode(port->typec_port, opmode);
+ port->pwr_opmode = TYPEC_PWR_MODE_USB;
+ port->caps_count = 0;
+ port->message_id = 0;
+ port->rx_msgid = -1;
+ port->explicit_contract = false;
+ tcpm_set_state(port, SRC_SEND_CAPABILITIES, 0);
+ break;
+ case SRC_SEND_CAPABILITIES:
+ port->caps_count++;
+ if (port->caps_count > PD_N_CAPS_COUNT) {
+ tcpm_set_state(port, SRC_READY, 0);
+ break;
+ }
+ ret = tcpm_pd_send_source_caps(port);
+ if (ret < 0) {
+ tcpm_set_state(port, SRC_SEND_CAPABILITIES,
+ PD_T_SEND_SOURCE_CAP);
+ } else {
+ /*
+ * Per standard, we should clear the reset counter here.
+ * However, that can result in state machine hang-ups.
+ * Reset it only in READY state to improve stability.
+ */
+ /* port->hard_reset_count = 0; */
+ port->caps_count = 0;
+ tcpm_set_pd_capable(port, true);
+ update_pd_contract_status(port, false);
+ tcpm_set_state_cond(port, hard_reset_state(port),
+ PD_T_SEND_SOURCE_CAP);
+ }
+ break;
+ case SRC_NEGOTIATE_CAPABILITIES:
+ ret = tcpm_pd_check_request(port);
+ if (ret < 0) {
+ tcpm_pd_send_control(port, PD_CTRL_REJECT);
+ if (!port->explicit_contract) {
+ tcpm_set_state(port,
+ SRC_WAIT_NEW_CAPABILITIES, 0);
+ } else {
+ tcpm_set_state(port, SRC_READY, 0);
+ }
+ } else {
+ tcpm_pd_send_control(port, PD_CTRL_ACCEPT);
+ port->usb_comm_capable = port->sink_request &
+ RDO_USB_COMM;
+ /* Notify TCPC of usb_comm_capable. */
+ tcpm_set_attached_state(port, true);
+ tcpm_set_state(port, SRC_TRANSITION_SUPPLY,
+ PD_T_SRC_TRANSITION);
+ }
+ break;
+ case SRC_TRANSITION_SUPPLY:
+ /* XXX: regulator_set_voltage(vbus, ...) */
+ tcpm_pd_send_control(port, PD_CTRL_PS_RDY);
+ port->explicit_contract = true;
+ typec_set_pwr_opmode(port->typec_port, TYPEC_PWR_MODE_PD);
+ port->pwr_opmode = TYPEC_PWR_MODE_PD;
+ tcpm_set_state_cond(port, SRC_READY, 0);
+ break;
+ case SRC_READY:
+ tcpm_log(port, "in PR_SWAP := false");
+ port->tcpc->set_in_pr_swap(port->tcpc, false);
+#if 1
+ port->hard_reset_count = 0;
+#endif
+ port->try_src_count = 0;
+
+ tcpm_swap_complete(port, 0);
+ tcpm_typec_connect(port);
+ if (port->explicit_contract && tcpm_rp_cc(port)
+ == TYPEC_CC_RP_DEF)
+ tcpm_set_cc(port, TYPEC_CC_RP_1_5);
+ tcpm_check_send_discover(port);
+ update_pd_contract_status(port, port->pd_capable
+ ? true : false);
+
+ /*
+ * 6.3.5
+ * Sending ping messages is not necessary if
+ * - the source operates at vSafe5V
+ * or
+ * - The system is not operating in PD mode
+ * or
+ * - Both partners are connected using a Type-C connector
+ *
+ * There is no actual need to send PD messages since the local
+ * port type-c and the spec does not clearly say whether PD is
+ * possible when type-c is connected to Type-A/B
+ */
+ break;
+ case SRC_WAIT_NEW_CAPABILITIES:
+ /* Nothing to do... */
+ break;
+
+ /* SNK states */
+ case SNK_UNATTACHED:
+ tcpm_port_in_hard_reset(port, false);
+ if (!port->non_pd_role_swap)
+ tcpm_swap_complete(port, -ENOTCONN);
+ tcpm_snk_detach(port);
+ if (tcpm_start_drp_toggling(port)) {
+ tcpm_set_state(port, DRP_TOGGLING, 0);
+ break;
+ }
+ tcpm_set_cc(port, TYPEC_CC_RD);
+ if (port->port_type == TYPEC_PORT_DRP)
+ tcpm_set_state(port, SRC_UNATTACHED, PD_T_DRP_SRC);
+ break;
+ case SNK_ATTACH_WAIT:
+ if ((port->cc1 == TYPEC_CC_OPEN &&
+ port->cc2 != TYPEC_CC_OPEN) ||
+ (port->cc1 != TYPEC_CC_OPEN &&
+ port->cc2 == TYPEC_CC_OPEN))
+ tcpm_set_state(port, SNK_DEBOUNCED,
+ PD_T_CC_DEBOUNCE);
+ else if (tcpm_port_is_disconnected(port))
+ tcpm_set_state(port, SNK_UNATTACHED,
+ PD_T_PD_DEBOUNCE);
+ break;
+ case SNK_DEBOUNCED:
+ if (tcpm_port_is_disconnected(port))
+ tcpm_set_state(port, SNK_UNATTACHED,
+ PD_T_PD_DEBOUNCE);
+ else if (port->vbus_present)
+ tcpm_set_state(port,
+ tcpm_try_src(port) ? SRC_TRY
+ : SNK_ATTACHED,
+ 0);
+ else
+ /* Wait for VBUS, but not forever */
+ tcpm_set_state(port, PORT_RESET, PD_T_PS_SOURCE_ON);
+ break;
+
+ case SRC_TRY:
+ port->try_src_count++;
+ tcpm_set_cc(port, tcpm_rp_cc(port));
+ port->max_wait = 0;
+ tcpm_set_state(port, SRC_TRY_WAIT, 0);
+ break;
+ case SRC_TRY_WAIT:
+ if (port->max_wait == 0) {
+ port->max_wait = jiffies +
+ msecs_to_jiffies(PD_T_DRP_TRY);
+ msecs = PD_T_DRP_TRY;
+ } else {
+ if (time_is_after_jiffies(port->max_wait))
+ msecs = jiffies_to_msecs(port->max_wait -
+ jiffies);
+ else
+ msecs = 0;
+ }
+ tcpm_set_state(port, SNK_TRYWAIT, msecs);
+ break;
+ case SRC_TRY_DEBOUNCE:
+ tcpm_set_state(port, SRC_ATTACHED, PD_T_PD_DEBOUNCE);
+ break;
+ case SNK_TRYWAIT:
+ tcpm_set_cc(port, TYPEC_CC_RD);
+ tcpm_set_state(port, SNK_TRYWAIT_VBUS, PD_T_CC_DEBOUNCE);
+ break;
+ case SNK_TRYWAIT_VBUS:
+ /*
+ * TCPM stays in this state indefinitely until VBUS
+ * is detected as long as Rp is not detected for
+ * more than a time period of tPDDebounce.
+ */
+ if (port->vbus_present && tcpm_port_is_sink(port)) {
+ tcpm_set_state(port, SNK_ATTACHED, 0);
+ break;
+ }
+ if (!tcpm_port_is_sink(port))
+ tcpm_set_state(port, SNK_TRYWAIT_DEBOUNCE, 0);
+ break;
+ case SNK_TRYWAIT_DEBOUNCE:
+ tcpm_set_state(port, SNK_UNATTACHED, PD_T_PD_DEBOUNCE);
+ break;
+ case SNK_ATTACHED:
+ ret = tcpm_snk_attach(port);
+ if (ret < 0)
+ tcpm_set_state(port, SNK_UNATTACHED, 0);
+ else
+ tcpm_set_state(port, SNK_STARTUP, 0);
+ break;
+ case SNK_STARTUP:
+ tcpm_port_in_hard_reset(port, false);
+ opmode = tcpm_get_pwr_opmode(port->polarity ?
+ port->cc2 : port->cc1);
+ typec_set_pwr_opmode(port->typec_port, opmode);
+ port->pwr_opmode = TYPEC_PWR_MODE_USB;
+ port->message_id = 0;
+ port->rx_msgid = -1;
+ port->explicit_contract = false;
+ tcpm_set_state(port, SNK_DISCOVERY, 0);
+ break;
+ case SNK_DISCOVERY:
+ if (port->vbus_present) {
+ tcpm_set_current_limit(port,
+ tcpm_get_current_limit(port),
+ 5000, 5000);
+ tcpm_set_charge(port, true);
+ tcpm_set_state(port, SNK_WAIT_CAPABILITIES, 0);
+ break;
+ }
+ /*
+ * For DRP, timeouts differ. Also, handling is supposed to be
+ * different and much more complex (dead battery detection;
+ * see USB power delivery specification, section 8.3.3.6.1.5.1).
+ */
+ tcpm_set_state(port, hard_reset_state(port),
+ port->port_type == TYPEC_PORT_DRP ?
+ PD_T_DB_DETECT : PD_T_NO_RESPONSE);
+ break;
+ case SNK_DISCOVERY_DEBOUNCE:
+ tcpm_set_state(port, SNK_DISCOVERY_DEBOUNCE_DONE,
+ PD_T_CC_DEBOUNCE);
+ break;
+ case SNK_DISCOVERY_DEBOUNCE_DONE:
+ if (!tcpm_port_is_disconnected(port) &&
+ tcpm_port_is_sink(port) &&
+ time_is_after_jiffies(port->delayed_runtime)) {
+ tcpm_set_state(port, SNK_DISCOVERY,
+ port->delayed_runtime - jiffies);
+ break;
+ }
+ tcpm_set_state(port, unattached_state(port), 0);
+ break;
+ case SNK_WAIT_CAPABILITIES:
+ ret = port->tcpc->set_pd_rx(port->tcpc, true);
+ if (ret < 0) {
+ tcpm_set_state(port, SNK_READY, 0);
+ break;
+ }
+ /*
+ * If VBUS has never been low, and we time out waiting
+ * for source cap, try a soft reset first, in case we
+ * were already in a stable contract before this boot.
+ * Do this only once.
+ */
+ if (port->vbus_never_low) {
+ port->vbus_never_low = false;
+ tcpm_set_state(port, SOFT_RESET_SEND,
+ PD_T_SINK_WAIT_CAP);
+ } else {
+ tcpm_set_state(port, hard_reset_state(port),
+ PD_T_SINK_WAIT_CAP);
+ }
+ break;
+ case SNK_NEGOTIATE_CAPABILITIES:
+ tcpm_set_pd_capable(port, true);
+ update_pd_contract_status(port, false);
+
+ port->usb_comm_capable = port->source_caps[0] &
+ PDO_FIXED_USB_COMM;
+ /* Notify TCPC of usb_comm_capable. */
+ tcpm_set_attached_state(port, true);
+ port->hard_reset_count = 0;
+ ret = tcpm_pd_send_request(port);
+ if (ret < 0) {
+ /* Let the Source send capabilities again. */
+ tcpm_set_state(port, SNK_WAIT_CAPABILITIES, 0);
+ } else {
+ tcpm_set_state_cond(port, hard_reset_state(port),
+ PD_T_SENDER_RESPONSE);
+ }
+ if (port->tcpc->set_suspend_supported) {
+ port->tcpc->set_suspend_supported(port->tcpc,
+ port->source_caps[0] &
+ PDO_FIXED_SUSPEND);
+ }
+ break;
+ case SNK_TRANSITION_SINK:
+ min_mv = min(port->supply_min_mv, port->requested_min_mv);
+ max_mv = max(port->supply_max_mv, port->requested_max_mv);
+ max_ma = (PD_P_SNKSTDBY * 1000) / max_mv;
+ tcpm_set_current_limit(port, max_ma, min_mv, max_mv);
+ case SNK_TRANSITION_SINK_VBUS:
+ tcpm_set_state(port, hard_reset_state(port),
+ PD_T_PS_TRANSITION);
+ break;
+ case SNK_READY:
+ port->try_snk_count = 0;
+ if (port->explicit_contract) {
+ typec_set_pwr_opmode(port->typec_port,
+ TYPEC_PWR_MODE_PD);
+ port->pwr_opmode = TYPEC_PWR_MODE_PD;
+ }
+
+ tcpm_swap_complete(port, 0);
+ tcpm_typec_connect(port);
+ tcpm_check_send_discover(port);
+ update_pd_contract_status(port, port->pd_capable
+ ? true : false);
+ break;
+
+ /* Accessory states */
+ case ACC_UNATTACHED:
+ tcpm_acc_detach(port);
+ tcpm_set_state(port, SRC_UNATTACHED, 0);
+ break;
+ case DEBUG_ACC_ATTACHED:
+ case AUDIO_ACC_ATTACHED:
+ ret = tcpm_acc_attach(port);
+ if (ret < 0)
+ tcpm_set_state(port, ACC_UNATTACHED, 0);
+ break;
+ case AUDIO_ACC_DEBOUNCE:
+ tcpm_set_state(port, ACC_UNATTACHED, PD_T_CC_DEBOUNCE);
+ break;
+
+ /* Hard_Reset states */
+ case HARD_RESET_SEND:
+ tcpm_pd_transmit(port, TCPC_TX_HARD_RESET, NULL);
+ tcpm_set_state(port, HARD_RESET_START, 0);
+ break;
+ case HARD_RESET_START:
+ tcpm_port_in_hard_reset(port, true);
+ update_pd_contract_status(port, false);
+ port->hard_reset_count++;
+ port->tcpc->set_pd_rx(port->tcpc, false);
+ tcpm_unregister_altmodes(port);
+ port->send_discover = true;
+ port->usb_comm_capable = false;
+ if (port->pwr_role == TYPEC_SOURCE)
+ tcpm_set_state(port, SRC_HARD_RESET_VBUS_OFF,
+ PD_T_PS_HARD_RESET);
+ else
+ tcpm_set_state(port, SNK_HARD_RESET_SINK_OFF, 0);
+ break;
+ case SRC_HARD_RESET_VBUS_OFF:
+ tcpm_set_vconn(port, false);
+ tcpm_set_vbus(port, false);
+ tcpm_set_roles(port, port->self_powered, TYPEC_SOURCE,
+ TYPEC_HOST);
+ tcpm_set_state(port, SRC_HARD_RESET_VBUS_ON, PD_T_SRC_RECOVER);
+ break;
+ case SRC_HARD_RESET_VBUS_ON:
+ tcpm_set_vconn(port, true);
+ tcpm_set_vbus(port, true);
+ port->tcpc->set_pd_rx(port->tcpc, true);
+ tcpm_set_attached_state(port, true);
+ tcpm_set_state(port, SRC_UNATTACHED, PD_T_PS_SOURCE_ON);
+ break;
+ case SNK_HARD_RESET_SINK_OFF:
+ tcpm_set_vconn(port, false);
+ if (port->pd_capable)
+ tcpm_set_charge(port, false);
+ tcpm_set_roles(port, port->self_powered, TYPEC_SINK,
+ TYPEC_DEVICE);
+ /*
+ * VBUS may or may not toggle, depending on the adapter.
+ * If it doesn't toggle, transition to SNK_HARD_RESET_SINK_ON
+ * directly after timeout.
+ */
+ tcpm_set_state(port, SNK_HARD_RESET_SINK_ON, PD_T_SAFE_0V);
+ break;
+ case SNK_HARD_RESET_WAIT_VBUS:
+ /* Assume we're disconnected if VBUS doesn't come back. */
+ tcpm_set_state(port, SNK_UNATTACHED,
+ PD_T_SRC_RECOVER_MAX + PD_T_SRC_TURN_ON);
+ break;
+ case SNK_HARD_RESET_SINK_ON:
+ /* Note: There is no guarantee that VBUS is on in this state */
+ /*
+ * XXX:
+ * The specification suggests that dual mode ports in sink
+ * mode should transition to state PE_SRC_Transition_to_default.
+ * See USB power delivery specification chapter 8.3.3.6.1.3.
+ * This would mean to to
+ * - turn off VCONN, reset power supply
+ * - request hardware reset
+ * - turn on VCONN
+ * - Transition to state PE_Src_Startup
+ * SNK only ports shall transition to state Snk_Startup
+ * (see chapter 8.3.3.3.8).
+ * Similar, dual-mode ports in source mode should transition
+ * to PE_SNK_Transition_to_default.
+ */
+ if (port->pd_capable) {
+ tcpm_set_current_limit(port,
+ tcpm_get_current_limit(port),
+ 5000, 5000);
+ tcpm_set_charge(port, true);
+ }
+ tcpm_set_attached_state(port, true);
+ tcpm_set_state(port, SNK_STARTUP, 0);
+ break;
+
+ /* Soft_Reset states */
+ case SOFT_RESET:
+ port->message_id = 0;
+ port->rx_msgid = -1;
+ tcpm_pd_send_control(port, PD_CTRL_ACCEPT);
+ if (port->pwr_role == TYPEC_SOURCE)
+ tcpm_set_state(port, SRC_SEND_CAPABILITIES, 0);
+ else
+ tcpm_set_state(port, SNK_WAIT_CAPABILITIES, 0);
+ break;
+ case SOFT_RESET_SEND:
+ port->message_id = 0;
+ port->rx_msgid = -1;
+ if (tcpm_pd_send_control(port, PD_CTRL_SOFT_RESET))
+ tcpm_set_state_cond(port, hard_reset_state(port), 0);
+ else
+ tcpm_set_state_cond(port, hard_reset_state(port),
+ PD_T_SENDER_RESPONSE);
+ break;
+
+ /* DR_Swap states */
+ case DR_SWAP_SEND:
+ tcpm_pd_send_control(port, PD_CTRL_DR_SWAP);
+ tcpm_set_state_cond(port, DR_SWAP_SEND_TIMEOUT,
+ PD_T_SENDER_RESPONSE);
+ break;
+ case DR_SWAP_ACCEPT:
+ tcpm_pd_send_control(port, PD_CTRL_ACCEPT);
+ tcpm_set_state_cond(port, DR_SWAP_CHANGE_DR, 0);
+ break;
+ case DR_SWAP_SEND_TIMEOUT:
+ tcpm_swap_complete(port, -ETIMEDOUT);
+ tcpm_set_state(port, ready_state(port), 0);
+ break;
+ case DR_SWAP_CHANGE_DR:
+ if (port->data_role == TYPEC_HOST) {
+ tcpm_unregister_altmodes(port);
+ tcpm_set_roles(port, true, port->pwr_role,
+ TYPEC_DEVICE);
+ } else {
+ tcpm_set_roles(port, true, port->pwr_role,
+ TYPEC_HOST);
+ port->send_discover = true;
+ }
+ tcpm_set_state(port, ready_state(port), 0);
+ break;
+
+ /* PR_Swap states */
+ case PR_SWAP_ACCEPT:
+ if (port->pwr_role == TYPEC_SOURCE)
+ tcpm_set_cc(port, tcpm_rp_cc(port));
+ else
+ tcpm_set_cc(port, TYPEC_CC_RD);
+ tcpm_log(port, "in PR_SWAP := true");
+ port->tcpc->set_in_pr_swap(port->tcpc, true);
+ tcpm_pd_send_control(port, PD_CTRL_ACCEPT);
+ tcpm_set_state(port, PR_SWAP_START, 0);
+ break;
+ case PR_SWAP_SEND:
+ if (port->pwr_role == TYPEC_SOURCE)
+ tcpm_set_cc(port, tcpm_rp_cc(port));
+ else
+ tcpm_set_cc(port, TYPEC_CC_RD);
+ tcpm_log(port, "in PR_SWAP := true");
+ port->tcpc->set_in_pr_swap(port->tcpc, true);
+ tcpm_pd_send_control(port, PD_CTRL_PR_SWAP);
+ tcpm_set_state_cond(port, PR_SWAP_SEND_TIMEOUT,
+ PD_T_SENDER_RESPONSE);
+ break;
+ case PR_SWAP_SEND_TIMEOUT:
+ tcpm_log(port, "in PR_SWAP := false");
+ port->tcpc->set_in_pr_swap(port->tcpc, false);
+ tcpm_swap_complete(port, -ETIMEDOUT);
+ tcpm_set_state(port, ready_state(port), 0);
+ break;
+ case PR_SWAP_START:
+ tcpm_log(port, "in PR_SWAP := true");
+ if (port->pwr_role == TYPEC_SOURCE)
+ tcpm_set_state(port, PR_SWAP_SRC_SNK_TRANSITION_OFF,
+ PD_T_SRC_TRANSITION);
+ else
+ tcpm_set_state(port, PR_SWAP_SNK_SRC_SINK_OFF, 0);
+ break;
+ case PR_SWAP_SRC_SNK_TRANSITION_OFF:
+ tcpm_set_vbus(port, false);
+ port->explicit_contract = false;
+ /* allow time for Vbus discharge, must be < tSrcSwapStdby */
+ tcpm_set_state(port, PR_SWAP_SRC_SNK_SOURCE_OFF,
+ PD_T_SRCSWAPSTDBY - PD_T_CC_DEBOUNCE);
+ break;
+ case PR_SWAP_SRC_SNK_SOURCE_OFF:
+ tcpm_set_cc(port, TYPEC_CC_RD);
+ /* allow CC debounce */
+ tcpm_set_state(port, PR_SWAP_SRC_SNK_SOURCE_OFF_CC_DEBOUNCED,
+ PD_T_CC_DEBOUNCE);
+ break;
+ case PR_SWAP_SRC_SNK_SOURCE_OFF_CC_DEBOUNCED:
+ /*
+ * USB-PD standard, 6.2.1.4, Port Power Role:
+ * "During the Power Role Swap Sequence, for the initial Source
+ * Port, the Port Power Role field shall be set to Sink in the
+ * PS_RDY Message indicating that the initial Source’s power
+ * supply is turned off"
+ */
+ tcpm_set_pwr_role(port, TYPEC_SINK);
+ if (tcpm_pd_send_control(port, PD_CTRL_PS_RDY)) {
+ tcpm_set_state(port, ERROR_RECOVERY, 0);
+ break;
+ }
+ tcpm_set_state_cond(port, SNK_UNATTACHED, PD_T_PS_SOURCE_ON);
+ break;
+ case PR_SWAP_SRC_SNK_SINK_ON:
+ tcpm_set_state(port, SNK_STARTUP, 0);
+ break;
+ case PR_SWAP_SNK_SRC_SINK_OFF:
+ tcpm_set_charge(port, false);
+ tcpm_set_state(port, hard_reset_state(port),
+ PD_T_PS_SOURCE_OFF);
+ break;
+ case PR_SWAP_SNK_SRC_SOURCE_ON:
+ tcpm_set_cc(port, tcpm_rp_cc(port));
+ tcpm_set_vbus(port, true);
+ /*
+ * allow time VBUS ramp-up, must be < tNewSrc
+ * Also, this window overlaps with CC debounce as well.
+ * So, Wait for the max of two which is PD_T_NEWSRC
+ */
+ tcpm_set_state(port, PR_SWAP_SNK_SRC_SOURCE_ON_VBUS_RAMPED_UP,
+ PD_T_NEWSRC);
+ break;
+ case PR_SWAP_SNK_SRC_SOURCE_ON_VBUS_RAMPED_UP:
+ /*
+ * USB PD standard, 6.2.1.4:
+ * "Subsequent Messages initiated by the Policy Engine,
+ * such as the PS_RDY Message sent to indicate that Vbus
+ * is ready, will have the Port Power Role field set to
+ * Source."
+ */
+ tcpm_set_pwr_role(port, TYPEC_SOURCE);
+ tcpm_pd_send_control(port, PD_CTRL_PS_RDY);
+ tcpm_set_state(port, SRC_STARTUP, PD_T_CC_DEBOUNCE);
+ break;
+ case VCONN_SWAP_ACCEPT:
+ tcpm_pd_send_control(port, PD_CTRL_ACCEPT);
+ tcpm_set_state(port, VCONN_SWAP_START, 0);
+ break;
+ case VCONN_SWAP_SEND:
+ tcpm_pd_send_control(port, PD_CTRL_VCONN_SWAP);
+ tcpm_set_state(port, VCONN_SWAP_SEND_TIMEOUT,
+ PD_T_SENDER_RESPONSE);
+ break;
+ case VCONN_SWAP_SEND_TIMEOUT:
+ tcpm_swap_complete(port, -ETIMEDOUT);
+ tcpm_set_state(port, ready_state(port), 0);
+ break;
+ case VCONN_SWAP_START:
+ if (port->vconn_role == TYPEC_SOURCE)
+ tcpm_set_state(port, VCONN_SWAP_WAIT_FOR_VCONN, 0);
+ else
+ tcpm_set_state(port, VCONN_SWAP_TURN_ON_VCONN, 0);
+ break;
+ case VCONN_SWAP_WAIT_FOR_VCONN:
+ tcpm_set_state(port, hard_reset_state(port),
+ PD_T_VCONN_SOURCE_ON);
+ break;
+ case VCONN_SWAP_TURN_ON_VCONN:
+ tcpm_set_vconn(port, true);
+ tcpm_pd_send_control(port, PD_CTRL_PS_RDY);
+ tcpm_set_state(port, ready_state(port), 0);
+ break;
+ case VCONN_SWAP_TURN_OFF_VCONN:
+ tcpm_set_vconn(port, false);
+ tcpm_set_state(port, ready_state(port), 0);
+ break;
+
+ case PR_SWAP_CANCEL:
+ tcpm_log(port, "in PR_SWAP := false");
+ port->tcpc->set_in_pr_swap(port->tcpc, false);
+ case DR_SWAP_CANCEL:
+ case VCONN_SWAP_CANCEL:
+ tcpm_swap_complete(port, port->swap_status);
+ if (port->pwr_role == TYPEC_SOURCE)
+ tcpm_set_state(port, SRC_READY, 0);
+ else
+ tcpm_set_state(port, SNK_READY, 0);
+ break;
+
+ case BIST_RX:
+ switch (BDO_MODE_MASK(port->bist_request)) {
+ case BDO_MODE_CARRIER2:
+ tcpm_pd_transmit(port, TCPC_TX_BIST_MODE_2, NULL);
+ break;
+ default:
+ break;
+ }
+ /* Always switch to unattached state */
+ tcpm_set_state(port, unattached_state(port), 0);
+ break;
+ case ERROR_RECOVERY:
+ tcpm_swap_complete(port, -EPROTO);
+ tcpm_set_state(port, PORT_RESET, 0);
+ break;
+ case PORT_RESET:
+ tcpm_reset_port(port);
+ tcpm_set_cc(port, TYPEC_CC_OPEN);
+ tcpm_set_state(port, PORT_RESET_WAIT_OFF,
+ PD_T_ERROR_RECOVERY);
+ break;
+ case PORT_RESET_WAIT_OFF:
+ tcpm_set_state(port,
+ tcpm_default_state(port),
+ port->vbus_present ? PD_T_PS_SOURCE_OFF : 0);
+ break;
+ default:
+ WARN(1, "Unexpected port state %d\n", port->state);
+ break;
+ }
+}
+
+static void tcpm_state_machine_work(struct work_struct *work)
+{
+ struct tcpm_port *port = container_of(work, struct tcpm_port,
+ state_machine.work);
+ enum tcpm_state prev_state;
+
+ mutex_lock(&port->lock);
+ port->state_machine_running = true;
+
+ if (port->queued_message && tcpm_send_queued_message(port))
+ goto done;
+
+ /* If we were queued due to a delayed state change, update it now */
+ if (port->delayed_state) {
+ tcpm_log(port, "state change %s -> %s [delayed %ld ms]",
+ tcpm_states[port->state],
+ tcpm_states[port->delayed_state], port->delay_ms);
+ port->prev_state = port->state;
+ port->state = port->delayed_state;
+ port->delayed_state = INVALID_STATE;
+ }
+
+ /*
+ * Continue running as long as we have (non-delayed) state changes
+ * to make.
+ */
+ do {
+ prev_state = port->state;
+ run_state_machine(port);
+ if (port->queued_message)
+ tcpm_send_queued_message(port);
+ } while (port->state != prev_state && !port->delayed_state);
+
+done:
+ port->state_machine_running = false;
+ mutex_unlock(&port->lock);
+}
+
+static void _tcpm_cc_change(struct tcpm_port *port, enum typec_cc_status cc1,
+ enum typec_cc_status cc2)
+{
+ enum typec_cc_status old_cc1, old_cc2;
+ enum tcpm_state new_state;
+
+ old_cc1 = port->cc1;
+ old_cc2 = port->cc2;
+ port->cc1 = cc1;
+ port->cc2 = cc2;
+
+ tcpm_log_force(port,
+ "CC1: %u -> %u, CC2: %u -> %u [state %s, polarity %d, %s]",
+ old_cc1, cc1, old_cc2, cc2, tcpm_states[port->state],
+ port->polarity,
+ tcpm_port_is_disconnected(port) ? "disconnected"
+ : "connected");
+
+ switch (port->state) {
+ case DRP_TOGGLING:
+ if (tcpm_port_is_debug(port) || tcpm_port_is_audio(port) ||
+ tcpm_port_is_source(port))
+ tcpm_set_state(port, SRC_ATTACH_WAIT, 0);
+ else if (tcpm_port_is_sink(port))
+ tcpm_set_state(port, SNK_ATTACH_WAIT, 0);
+ break;
+ case SRC_UNATTACHED:
+ case ACC_UNATTACHED:
+ if (tcpm_port_is_debug(port) || tcpm_port_is_audio(port) ||
+ tcpm_port_is_source(port))
+ tcpm_set_state(port, SRC_ATTACH_WAIT, 0);
+ break;
+ case SRC_ATTACH_WAIT:
+ if (tcpm_port_is_disconnected(port) ||
+ tcpm_port_is_audio_detached(port))
+ tcpm_set_state(port, SRC_UNATTACHED, 0);
+ else if (cc1 != old_cc1 || cc2 != old_cc2)
+ tcpm_set_state(port, SRC_ATTACH_WAIT, 0);
+ break;
+ case SRC_ATTACHED:
+ case SRC_SEND_CAPABILITIES:
+ case SRC_READY:
+ if (tcpm_port_is_disconnected(port) ||
+ !tcpm_port_is_source(port))
+ tcpm_set_state(port, SRC_UNATTACHED, 0);
+ break;
+ case SNK_UNATTACHED:
+ if (tcpm_port_is_sink(port))
+ tcpm_set_state(port, SNK_ATTACH_WAIT, 0);
+ break;
+ case SNK_ATTACH_WAIT:
+ if ((port->cc1 == TYPEC_CC_OPEN &&
+ port->cc2 != TYPEC_CC_OPEN) ||
+ (port->cc1 != TYPEC_CC_OPEN &&
+ port->cc2 == TYPEC_CC_OPEN))
+ new_state = SNK_DEBOUNCED;
+ else if (tcpm_port_is_disconnected(port))
+ new_state = SNK_UNATTACHED;
+ else
+ break;
+ if (new_state != port->delayed_state)
+ tcpm_set_state(port, SNK_ATTACH_WAIT, 0);
+ break;
+ case SNK_DEBOUNCED:
+ if (tcpm_port_is_disconnected(port))
+ new_state = SNK_UNATTACHED;
+ else if (port->vbus_present)
+ new_state = tcpm_try_src(port) ? SRC_TRY : SNK_ATTACHED;
+ else
+ new_state = SNK_UNATTACHED;
+ if (new_state != port->delayed_state)
+ tcpm_set_state(port, SNK_DEBOUNCED, 0);
+ break;
+ case SNK_READY:
+ if (tcpm_port_is_disconnected(port))
+ tcpm_set_state(port, unattached_state(port), 0);
+ else if (!port->pd_capable &&
+ (cc1 != old_cc1 || cc2 != old_cc2))
+ tcpm_set_current_limit(port,
+ tcpm_get_current_limit(port),
+ 5000, 5000);
+ break;
+
+ case AUDIO_ACC_ATTACHED:
+ if (cc1 == TYPEC_CC_OPEN || cc2 == TYPEC_CC_OPEN)
+ tcpm_set_state(port, AUDIO_ACC_DEBOUNCE, 0);
+ break;
+ case AUDIO_ACC_DEBOUNCE:
+ if (tcpm_port_is_audio(port))
+ tcpm_set_state(port, AUDIO_ACC_ATTACHED, 0);
+ break;
+
+ case DEBUG_ACC_ATTACHED:
+ if (cc1 == TYPEC_CC_OPEN || cc2 == TYPEC_CC_OPEN)
+ tcpm_set_state(port, ACC_UNATTACHED, 0);
+ break;
+
+ case SNK_TRY:
+ /* Do nothing, waiting for timeout */
+ break;
+
+ case SNK_DISCOVERY:
+ /* CC line is unstable, wait for debounce */
+ if (tcpm_port_is_disconnected(port))
+ tcpm_set_state(port, SNK_DISCOVERY_DEBOUNCE, 0);
+ break;
+ case SNK_DISCOVERY_DEBOUNCE:
+ break;
+
+ case SRC_TRYWAIT:
+ /* Hand over to state machine if needed */
+ if (!port->vbus_present && tcpm_port_is_source(port))
+ tcpm_set_state(port, SRC_TRYWAIT_DEBOUNCE, 0);
+ break;
+ case SRC_TRYWAIT_DEBOUNCE:
+ if (port->vbus_present || !tcpm_port_is_source(port))
+ tcpm_set_state(port, SRC_TRYWAIT, 0);
+ break;
+ case SNK_TRY_WAIT_DEBOUNCE:
+ if (!tcpm_port_is_sink(port)) {
+ port->max_wait = 0;
+ tcpm_set_state(port, SRC_TRYWAIT, 0);
+ }
+ break;
+ case SRC_TRY_WAIT:
+ if (tcpm_port_is_source(port))
+ tcpm_set_state(port, SRC_TRY_DEBOUNCE, 0);
+ break;
+ case SRC_TRY_DEBOUNCE:
+ tcpm_set_state(port, SRC_TRY_WAIT, 0);
+ break;
+ case SNK_TRYWAIT_DEBOUNCE:
+ if (tcpm_port_is_sink(port))
+ tcpm_set_state(port, SNK_TRYWAIT_VBUS, 0);
+ break;
+ case SNK_TRYWAIT_VBUS:
+ if (!tcpm_port_is_sink(port))
+ tcpm_set_state(port, SNK_TRYWAIT_DEBOUNCE, 0);
+ break;
+ case SNK_TRYWAIT:
+ /* Do nothing, waiting for tCCDebounce */
+ break;
+ case PR_SWAP_SNK_SRC_SINK_OFF:
+ case PR_SWAP_SRC_SNK_TRANSITION_OFF:
+ case PR_SWAP_SRC_SNK_SOURCE_OFF:
+ case PR_SWAP_SRC_SNK_SOURCE_OFF_CC_DEBOUNCED:
+ case PR_SWAP_SNK_SRC_SOURCE_ON:
+ case PR_SWAP_SNK_SRC_SOURCE_ON_VBUS_RAMPED_UP:
+ /*
+ * CC state change is expected in PR_SWAP
+ * Ignore it.
+ */
+ break;
+
+ case PORT_RESET:
+ case PORT_RESET_WAIT_OFF:
+ /*
+ * State set back to default mode once the timer completes.
+ * Ignore CC changes here.
+ */
+ break;
+
+ default:
+ if (tcpm_port_is_disconnected(port))
+ tcpm_set_state(port, unattached_state(port), 0);
+ break;
+ }
+}
+
+static void _tcpm_pd_vbus_on(struct tcpm_port *port)
+{
+ tcpm_log_force(port, "VBUS on");
+ port->vbus_present = true;
+ switch (port->state) {
+ case SNK_TRANSITION_SINK_VBUS:
+ port->explicit_contract = true;
+ tcpm_set_state(port, SNK_READY, 0);
+ break;
+ case SNK_DISCOVERY:
+ tcpm_set_state(port, SNK_DISCOVERY, 0);
+ break;
+
+ case SNK_DEBOUNCED:
+ tcpm_set_state(port, tcpm_try_src(port) ? SRC_TRY
+ : SNK_ATTACHED,
+ 0);
+ break;
+ case SNK_HARD_RESET_WAIT_VBUS:
+ tcpm_set_state(port, SNK_HARD_RESET_SINK_ON, 0);
+ break;
+ case SRC_ATTACHED:
+ tcpm_set_state(port, SRC_STARTUP, 0);
+ break;
+ case SRC_HARD_RESET_VBUS_ON:
+ tcpm_set_state(port, SRC_STARTUP, 0);
+ break;
+
+ case SNK_TRY:
+ /* Do nothing, waiting for timeout */
+ break;
+ case SRC_TRYWAIT:
+ /* Do nothing, Waiting for Rd to be detected */
+ break;
+ case SRC_TRYWAIT_DEBOUNCE:
+ tcpm_set_state(port, SRC_TRYWAIT, 0);
+ break;
+ case SNK_TRY_WAIT_DEBOUNCE:
+ /* Do nothing, waiting for PD_DEBOUNCE to do be done */
+ break;
+ case SNK_TRYWAIT:
+ /* Do nothing, waiting for tCCDebounce */
+ break;
+ case SNK_TRYWAIT_VBUS:
+ if (tcpm_port_is_sink(port))
+ tcpm_set_state(port, SNK_ATTACHED, 0);
+ break;
+ case SNK_TRYWAIT_DEBOUNCE:
+ /* Do nothing, waiting for Rp */
+ break;
+ case SRC_TRY_WAIT:
+ case SRC_TRY_DEBOUNCE:
+ /* Do nothing, waiting for sink detection */
+ break;
+
+ case PORT_RESET:
+ case PORT_RESET_WAIT_OFF:
+ /*
+ * State set back to default mode once the timer completes.
+ * Ignore vbus changes here.
+ */
+ break;
+
+ default:
+ break;
+ }
+}
+
+static void _tcpm_pd_vbus_off(struct tcpm_port *port)
+{
+ tcpm_log_force(port, "VBUS off");
+ port->vbus_present = false;
+ port->vbus_never_low = false;
+ switch (port->state) {
+ case SNK_HARD_RESET_SINK_OFF:
+ tcpm_set_state(port, SNK_HARD_RESET_WAIT_VBUS, 0);
+ break;
+ case SRC_HARD_RESET_VBUS_OFF:
+ /*
+ * Mandatory tSrcRecovery time. So do nothing.
+ */
+ break;
+ case HARD_RESET_SEND:
+ break;
+
+ case SNK_TRY:
+ /* Do nothing, waiting for timeout */
+ break;
+ case SRC_TRYWAIT:
+ /* Hand over to state machine if needed */
+ if (tcpm_port_is_source(port))
+ tcpm_set_state(port, SRC_TRYWAIT_DEBOUNCE, 0);
+ break;
+ case SNK_TRY_WAIT_DEBOUNCE:
+ /* Do nothing, waiting for PD_DEBOUNCE to do be done */
+ break;
+ case SNK_TRYWAIT:
+ case SNK_TRYWAIT_VBUS:
+ case SNK_TRYWAIT_DEBOUNCE:
+ break;
+ case SNK_ATTACH_WAIT:
+ tcpm_set_state(port, SNK_UNATTACHED, 0);
+ break;
+
+ case SNK_NEGOTIATE_CAPABILITIES:
+ break;
+
+ case PR_SWAP_SRC_SNK_TRANSITION_OFF:
+ tcpm_set_state(port, PR_SWAP_SRC_SNK_SOURCE_OFF, 0);
+ break;
+
+ case PR_SWAP_SNK_SRC_SINK_OFF:
+ /* Do nothing, expected */
+ break;
+
+ case PORT_RESET_WAIT_OFF:
+ tcpm_set_state(port, tcpm_default_state(port), 0);
+ break;
+
+ case SRC_TRY_WAIT:
+ case SRC_TRY_DEBOUNCE:
+ /* Do nothing, waiting for sink detection */
+ break;
+
+ case PORT_RESET:
+ /*
+ * State set back to default mode once the timer completes.
+ * Ignore vbus changes here.
+ */
+ break;
+
+ default:
+ if (port->pwr_role == TYPEC_SINK &&
+ port->attached)
+ tcpm_set_state(port, SNK_UNATTACHED, 0);
+ break;
+ }
+}
+
+static void _tcpm_pd_hard_reset(struct tcpm_port *port)
+{
+ tcpm_log_force(port, "Received hard reset");
+ /*
+ * If we keep receiving hard reset requests, executing the hard reset
+ * must have failed. Revert to error recovery if that happens.
+ */
+ tcpm_set_state(port,
+ port->hard_reset_count < PD_N_HARD_RESET_COUNT ?
+ HARD_RESET_START : ERROR_RECOVERY,
+ 0);
+}
+
+static void tcpm_pd_event_handler(struct work_struct *work)
+{
+ struct tcpm_port *port = container_of(work, struct tcpm_port,
+ event_work);
+ u32 events;
+
+ mutex_lock(&port->lock);
+
+ spin_lock(&port->pd_event_lock);
+ while (port->pd_events) {
+ events = port->pd_events;
+ port->pd_events = 0;
+ spin_unlock(&port->pd_event_lock);
+ if (events & TCPM_RESET_EVENT)
+ _tcpm_pd_hard_reset(port);
+ if (events & TCPM_VBUS_EVENT) {
+ bool vbus;
+
+ vbus = port->tcpc->get_vbus(port->tcpc);
+ if (vbus)
+ _tcpm_pd_vbus_on(port);
+ else
+ _tcpm_pd_vbus_off(port);
+ }
+ if (events & TCPM_CC_EVENT) {
+ enum typec_cc_status cc1, cc2;
+
+ if (port->tcpc->get_cc(port->tcpc, &cc1, &cc2) == 0)
+ _tcpm_cc_change(port, cc1, cc2);
+ }
+ if (events & TCPM_PORT_RESET_EVENT) {
+ tcpm_set_state(port, PORT_RESET, 0);
+ }
+ spin_lock(&port->pd_event_lock);
+ }
+ spin_unlock(&port->pd_event_lock);
+ mutex_unlock(&port->lock);
+}
+
+void tcpm_port_reset(struct tcpm_port *port)
+{
+ spin_lock(&port->pd_event_lock);
+ port->pd_events = TCPM_PORT_RESET_EVENT;
+ spin_unlock(&port->pd_event_lock);
+ queue_work(port->wq, &port->event_work);
+}
+EXPORT_SYMBOL_GPL(tcpm_port_reset);
+
+void tcpm_cc_change(struct tcpm_port *port)
+{
+ spin_lock(&port->pd_event_lock);
+ port->pd_events |= TCPM_CC_EVENT;
+ spin_unlock(&port->pd_event_lock);
+ queue_work(port->wq, &port->event_work);
+}
+EXPORT_SYMBOL_GPL(tcpm_cc_change);
+
+void tcpm_vbus_change(struct tcpm_port *port)
+{
+ spin_lock(&port->pd_event_lock);
+ port->pd_events |= TCPM_VBUS_EVENT;
+ spin_unlock(&port->pd_event_lock);
+ queue_work(port->wq, &port->event_work);
+}
+EXPORT_SYMBOL_GPL(tcpm_vbus_change);
+
+void tcpm_pd_hard_reset(struct tcpm_port *port)
+{
+ spin_lock(&port->pd_event_lock);
+ port->pd_events = TCPM_RESET_EVENT;
+ spin_unlock(&port->pd_event_lock);
+ queue_work(port->wq, &port->event_work);
+}
+EXPORT_SYMBOL_GPL(tcpm_pd_hard_reset);
+
+static int tcpm_dr_set(const struct typec_capability *cap,
+ enum typec_data_role data)
+{
+ struct tcpm_port *port = typec_cap_to_tcpm(cap);
+ int ret;
+
+ mutex_lock(&port->swap_lock);
+ mutex_lock(&port->lock);
+
+ if (port->port_type != TYPEC_PORT_DRP) {
+ ret = -EINVAL;
+ goto port_unlock;
+ }
+ if (port->state != SRC_READY && port->state != SNK_READY) {
+ ret = -EAGAIN;
+ goto port_unlock;
+ }
+
+ if (port->data_role == data) {
+ ret = 0;
+ goto port_unlock;
+ }
+
+ /*
+ * XXX
+ * 6.3.9: If an alternate mode is active, a request to swap
+ * alternate modes shall trigger a port reset.
+ * Reject data role swap request in this case.
+ */
+
+ if (!port->pd_capable) {
+ /*
+ * If the partner is not PD capable, reset the port to
+ * trigger a role change. This can only work if a preferred
+ * role is configured, and if it matches the requested role.
+ */
+ if (port->try_role == TYPEC_NO_PREFERRED_ROLE ||
+ port->try_role == port->pwr_role) {
+ ret = -EINVAL;
+ goto port_unlock;
+ }
+ port->non_pd_role_swap = true;
+ tcpm_set_state(port, PORT_RESET, 0);
+ } else {
+ tcpm_set_state(port, DR_SWAP_SEND, 0);
+ }
+
+ port->swap_status = 0;
+ port->swap_pending = true;
+ reinit_completion(&port->swap_complete);
+ mutex_unlock(&port->lock);
+
+ if (!wait_for_completion_timeout(&port->swap_complete,
+ msecs_to_jiffies(PD_ROLE_SWAP_TIMEOUT)))
+ ret = -ETIMEDOUT;
+ else
+ ret = port->swap_status;
+
+ port->non_pd_role_swap = false;
+ goto swap_unlock;
+
+port_unlock:
+ mutex_unlock(&port->lock);
+swap_unlock:
+ mutex_unlock(&port->swap_lock);
+ return ret;
+}
+
+static int tcpm_pr_set(const struct typec_capability *cap,
+ enum typec_role role)
+{
+ struct tcpm_port *port = typec_cap_to_tcpm(cap);
+ int ret;
+
+ mutex_lock(&port->swap_lock);
+ mutex_lock(&port->lock);
+
+ if (port->port_type != TYPEC_PORT_DRP) {
+ ret = -EINVAL;
+ goto port_unlock;
+ }
+ if (port->state != SRC_READY && port->state != SNK_READY) {
+ ret = -EAGAIN;
+ goto port_unlock;
+ }
+
+ if (role == port->pwr_role) {
+ ret = 0;
+ goto port_unlock;
+ }
+
+ port->swap_status = 0;
+ port->swap_pending = true;
+ reinit_completion(&port->swap_complete);
+ tcpm_set_state(port, PR_SWAP_SEND, 0);
+ mutex_unlock(&port->lock);
+
+ if (!wait_for_completion_timeout(&port->swap_complete,
+ msecs_to_jiffies(PD_ROLE_SWAP_TIMEOUT)))
+ ret = -ETIMEDOUT;
+ else
+ ret = port->swap_status;
+
+ goto swap_unlock;
+
+port_unlock:
+ mutex_unlock(&port->lock);
+swap_unlock:
+ mutex_unlock(&port->swap_lock);
+ return ret;
+}
+
+static int tcpm_vconn_set(const struct typec_capability *cap,
+ enum typec_role role)
+{
+ struct tcpm_port *port = typec_cap_to_tcpm(cap);
+ int ret;
+
+ mutex_lock(&port->swap_lock);
+ mutex_lock(&port->lock);
+
+ if (port->state != SRC_READY && port->state != SNK_READY) {
+ ret = -EAGAIN;
+ goto port_unlock;
+ }
+
+ if (role == port->vconn_role) {
+ ret = 0;
+ goto port_unlock;
+ }
+
+ port->swap_status = 0;
+ port->swap_pending = true;
+ reinit_completion(&port->swap_complete);
+ tcpm_set_state(port, VCONN_SWAP_SEND, 0);
+ mutex_unlock(&port->lock);
+
+ if (!wait_for_completion_timeout(&port->swap_complete,
+ msecs_to_jiffies(PD_ROLE_SWAP_TIMEOUT)))
+ ret = -ETIMEDOUT;
+ else
+ ret = port->swap_status;
+
+ goto swap_unlock;
+
+port_unlock:
+ mutex_unlock(&port->lock);
+swap_unlock:
+ mutex_unlock(&port->swap_lock);
+ return ret;
+}
+
+static int tcpm_try_role(const struct typec_capability *cap, int role)
+{
+ struct tcpm_port *port = typec_cap_to_tcpm(cap);
+ struct tcpc_dev *tcpc = port->tcpc;
+ int ret = 0;
+
+ mutex_lock(&port->lock);
+ if (tcpc->try_role)
+ ret = tcpc->try_role(tcpc, role);
+ if (!ret && !tcpc->config->try_role_hw)
+ port->try_role = role;
+ port->try_src_count = 0;
+ port->try_snk_count = 0;
+ mutex_unlock(&port->lock);
+
+ return ret;
+}
+
+static void tcpm_init(struct tcpm_port *port)
+{
+ enum typec_cc_status cc1, cc2;
+
+ port->tcpc->init(port->tcpc);
+
+ tcpm_reset_port(port);
+
+ /*
+ * XXX
+ * Should possibly wait for VBUS to settle if it was enabled locally
+ * since tcpm_reset_port() will disable VBUS.
+ */
+ port->vbus_present = port->tcpc->get_vbus(port->tcpc);
+ if (port->vbus_present)
+ port->vbus_never_low = true;
+
+ tcpm_set_state(port, tcpm_default_state(port), 0);
+
+ if (port->tcpc->get_cc(port->tcpc, &cc1, &cc2) == 0)
+ _tcpm_cc_change(port, cc1, cc2);
+
+ /*
+ * Some adapters need a clean slate at startup, and won't recover
+ * otherwise. So do not try to be fancy and force a clean disconnect.
+ */
+ tcpm_set_state(port, PORT_RESET, 0);
+}
+
+static int tcpm_port_type_set(const struct typec_capability *cap,
+ enum typec_port_type type)
+{
+ struct tcpm_port *port = typec_cap_to_tcpm(cap);
+
+ mutex_lock(&port->lock);
+ if (type == port->port_type)
+ goto port_unlock;
+
+ port->port_type = type;
+
+ if (!port->connected) {
+ tcpm_set_state(port, PORT_RESET, 0);
+ } else if (type == TYPEC_PORT_UFP) {
+ if (!(port->pwr_role == TYPEC_SINK &&
+ port->data_role == TYPEC_DEVICE))
+ tcpm_set_state(port, PORT_RESET, 0);
+ } else if (type == TYPEC_PORT_DFP) {
+ if (!(port->pwr_role == TYPEC_SOURCE &&
+ port->data_role == TYPEC_HOST))
+ tcpm_set_state(port, PORT_RESET, 0);
+ }
+
+port_unlock:
+ mutex_unlock(&port->lock);
+ return 0;
+}
+
+void tcpm_tcpc_reset(struct tcpm_port *port)
+{
+ mutex_lock(&port->lock);
+ /* XXX: Maintain PD connection if possible? */
+ tcpm_init(port);
+ mutex_unlock(&port->lock);
+}
+EXPORT_SYMBOL_GPL(tcpm_tcpc_reset);
+
+static int tcpm_copy_pdos(u32 *dest_pdo, const u32 *src_pdo,
+ unsigned int nr_pdo)
+{
+ unsigned int i;
+
+ if (nr_pdo > PDO_MAX_OBJECTS)
+ nr_pdo = PDO_MAX_OBJECTS;
+
+ for (i = 0; i < nr_pdo; i++)
+ dest_pdo[i] = src_pdo[i];
+
+ return nr_pdo;
+}
+
+static int tcpm_copy_vdos(u32 *dest_vdo, const u32 *src_vdo,
+ unsigned int nr_vdo)
+{
+ unsigned int i;
+
+ if (nr_vdo > VDO_MAX_OBJECTS)
+ nr_vdo = VDO_MAX_OBJECTS;
+
+ for (i = 0; i < nr_vdo; i++)
+ dest_vdo[i] = src_vdo[i];
+
+ return nr_vdo;
+}
+
+int tcpm_update_source_capabilities(struct tcpm_port *port, const u32 *pdo,
+ unsigned int nr_pdo)
+{
+ if (tcpm_validate_caps(port, pdo, nr_pdo))
+ return -EINVAL;
+
+ mutex_lock(&port->lock);
+ port->nr_src_pdo = tcpm_copy_pdos(port->src_pdo, pdo, nr_pdo);
+ switch (port->state) {
+ case SRC_UNATTACHED:
+ case SRC_ATTACH_WAIT:
+ case SRC_TRYWAIT:
+ tcpm_set_cc(port, tcpm_rp_cc(port));
+ break;
+ case SRC_SEND_CAPABILITIES:
+ case SRC_NEGOTIATE_CAPABILITIES:
+ case SRC_READY:
+ case SRC_WAIT_NEW_CAPABILITIES:
+ tcpm_set_cc(port, tcpm_rp_cc(port));
+ tcpm_set_state(port, SRC_SEND_CAPABILITIES, 0);
+ break;
+ default:
+ break;
+ }
+ mutex_unlock(&port->lock);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(tcpm_update_source_capabilities);
+
+int tcpm_update_sink_capabilities(struct tcpm_port *port, const u32 *pdo,
+ unsigned int nr_pdo,
+ unsigned int max_snk_mv,
+ unsigned int max_snk_ma,
+ unsigned int max_snk_mw,
+ unsigned int operating_snk_mw)
+{
+ if (tcpm_validate_caps(port, pdo, nr_pdo))
+ return -EINVAL;
+
+ mutex_lock(&port->lock);
+ port->nr_snk_pdo = tcpm_copy_pdos(port->snk_pdo, pdo, nr_pdo);
+ port->max_snk_mv = max_snk_mv;
+ port->max_snk_ma = max_snk_ma;
+ port->max_snk_mw = max_snk_mw;
+ port->operating_snk_mw = operating_snk_mw;
+
+ switch (port->state) {
+ case SNK_NEGOTIATE_CAPABILITIES:
+ case SNK_READY:
+ case SNK_TRANSITION_SINK:
+ case SNK_TRANSITION_SINK_VBUS:
+ tcpm_log(port, "in PR_SWAP := false");
+ port->tcpc->set_in_pr_swap(port->tcpc, false);
+ tcpm_set_state(port, SNK_NEGOTIATE_CAPABILITIES, 0);
+ break;
+ default:
+ break;
+ }
+ mutex_unlock(&port->lock);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(tcpm_update_sink_capabilities);
+
+static int nr_type_pdos(const u32 *pdo, unsigned int nr_pdo,
+ enum pd_pdo_type type)
+{
+ int count = 0;
+ int i;
+
+ for (i = 0; i < nr_pdo; i++) {
+ if (pdo_type(pdo[i]) == type)
+ count++;
+ }
+ return count;
+}
+
+struct tcpm_port *tcpm_register_port(struct device *dev, struct tcpc_dev *tcpc)
+{
+ struct tcpm_port *port;
+ int i, err;
+
+ if (!dev || !tcpc || !tcpc->config ||
+ !tcpc->get_vbus || !tcpc->set_cc || !tcpc->get_cc ||
+ !tcpc->set_polarity || !tcpc->set_vconn || !tcpc->set_vbus ||
+ !tcpc->set_pd_rx || !tcpc->set_roles || !tcpc->pd_transmit)
+ return ERR_PTR(-EINVAL);
+
+ port = devm_kzalloc(dev, sizeof(*port), GFP_KERNEL);
+ if (!port)
+ return ERR_PTR(-ENOMEM);
+
+ port->dev = dev;
+ port->tcpc = tcpc;
+
+ mutex_init(&port->lock);
+ mutex_init(&port->swap_lock);
+
+ port->wq = create_singlethread_workqueue(dev_name(dev));
+ if (!port->wq)
+ return ERR_PTR(-ENOMEM);
+ INIT_DELAYED_WORK(&port->state_machine, tcpm_state_machine_work);
+ INIT_DELAYED_WORK(&port->vdm_state_machine, vdm_state_machine_work);
+ INIT_WORK(&port->event_work, tcpm_pd_event_handler);
+
+ spin_lock_init(&port->pd_event_lock);
+
+ init_completion(&port->tx_complete);
+ init_completion(&port->swap_complete);
+ tcpm_debugfs_init(port);
+
+ if (tcpm_validate_caps(port, tcpc->config->src_pdo,
+ tcpc->config->nr_src_pdo) ||
+ tcpm_validate_caps(port, tcpc->config->snk_pdo,
+ tcpc->config->nr_snk_pdo)) {
+ err = -EINVAL;
+ goto out_destroy_wq;
+ }
+ port->nr_src_pdo = tcpm_copy_pdos(port->src_pdo, tcpc->config->src_pdo,
+ tcpc->config->nr_src_pdo);
+ port->nr_snk_pdo = tcpm_copy_pdos(port->snk_pdo, tcpc->config->snk_pdo,
+ tcpc->config->nr_snk_pdo);
+ port->nr_fixed = nr_type_pdos(port->snk_pdo,
+ port->nr_snk_pdo,
+ PDO_TYPE_FIXED);
+ port->nr_var = nr_type_pdos(port->snk_pdo,
+ port->nr_snk_pdo,
+ PDO_TYPE_VAR);
+ port->nr_batt = nr_type_pdos(port->snk_pdo,
+ port->nr_snk_pdo,
+ PDO_TYPE_BATT);
+ port->nr_snk_vdo = tcpm_copy_vdos(port->snk_vdo, tcpc->config->snk_vdo,
+ tcpc->config->nr_snk_vdo);
+
+ port->max_snk_mv = tcpc->config->max_snk_mv;
+ port->max_snk_ma = tcpc->config->max_snk_ma;
+ port->max_snk_mw = tcpc->config->max_snk_mw;
+ port->operating_snk_mw = tcpc->config->operating_snk_mw;
+ if (!tcpc->config->try_role_hw)
+ port->try_role = tcpc->config->default_role;
+ else
+ port->try_role = TYPEC_NO_PREFERRED_ROLE;
+
+ port->typec_caps.prefer_role = tcpc->config->default_role;
+ port->typec_caps.type = tcpc->config->type;
+ port->typec_caps.revision = 0x0120; /* Type-C spec release 1.2 */
+ port->typec_caps.pd_revision = 0x0200; /* USB-PD spec release 2.0 */
+ port->typec_caps.dr_set = tcpm_dr_set;
+ port->typec_caps.pr_set = tcpm_pr_set;
+ port->typec_caps.vconn_set = tcpm_vconn_set;
+ port->typec_caps.try_role = tcpm_try_role;
+ port->typec_caps.port_type_set = tcpm_port_type_set;
+ port->self_powered = tcpc->config->self_powered;
+
+ port->partner_desc.identity = &port->partner_ident;
+ port->port_type = tcpc->config->type;
+
+ port->typec_port = typec_register_port(port->dev, &port->typec_caps);
+ if (!port->typec_port) {
+ err = -ENOMEM;
+ goto out_destroy_wq;
+ }
+
+ if (tcpc->config->alt_modes) {
+ const struct typec_altmode_desc *paltmode = tcpc->config->alt_modes;
+
+ i = 0;
+ while (paltmode->svid && i < ARRAY_SIZE(port->port_altmode)) {
+ port->port_altmode[i] =
+ typec_port_register_altmode(port->typec_port,
+ paltmode);
+ if (!port->port_altmode[i]) {
+ tcpm_log(port,
+ "%s: failed to register port alternate mode 0x%x",
+ dev_name(dev), paltmode->svid);
+ break;
+ }
+ i++;
+ paltmode++;
+ }
+ }
+
+ mutex_lock(&port->lock);
+ tcpm_init(port);
+ mutex_unlock(&port->lock);
+
+ tcpm_log(port, "%s: registered", dev_name(dev));
+ return port;
+
+out_destroy_wq:
+ destroy_workqueue(port->wq);
+ return ERR_PTR(err);
+}
+EXPORT_SYMBOL_GPL(tcpm_register_port);
+
+void tcpm_unregister_port(struct tcpm_port *port)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(port->port_altmode); i++)
+ typec_unregister_altmode(port->port_altmode[i]);
+ typec_unregister_port(port->typec_port);
+ tcpm_debugfs_exit(port);
+ destroy_workqueue(port->wq);
+}
+EXPORT_SYMBOL_GPL(tcpm_unregister_port);
+
+MODULE_AUTHOR("Guenter Roeck <groeck@chromium.org>");
+MODULE_DESCRIPTION("USB Type-C Port Manager");
+MODULE_LICENSE("GPL");
diff --git a/drivers/usb/typec/typec.c b/drivers/usb/typec/typec.c
new file mode 100644
index 0000000..24e355b
--- /dev/null
+++ b/drivers/usb/typec/typec.c
@@ -0,0 +1,1368 @@
+/*
+ * USB Type-C Connector Class
+ *
+ * Copyright (C) 2017, Intel Corporation
+ * Author: Heikki Krogerus <heikki.krogerus@linux.intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/device.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/slab.h>
+#include <linux/usb/typec.h>
+
+struct typec_mode {
+ int index;
+ u32 vdo;
+ char *desc;
+ enum typec_port_type roles;
+
+ struct typec_altmode *alt_mode;
+
+ unsigned int active:1;
+
+ char group_name[6];
+ struct attribute_group group;
+ struct attribute *attrs[5];
+ struct device_attribute vdo_attr;
+ struct device_attribute desc_attr;
+ struct device_attribute active_attr;
+ struct device_attribute roles_attr;
+};
+
+struct typec_altmode {
+ struct device dev;
+ u16 svid;
+ int n_modes;
+ struct typec_mode modes[ALTMODE_MAX_MODES];
+ const struct attribute_group *mode_groups[ALTMODE_MAX_MODES];
+};
+
+struct typec_plug {
+ struct device dev;
+ enum typec_plug_index index;
+};
+
+struct typec_cable {
+ struct device dev;
+ enum typec_plug_type type;
+ struct usb_pd_identity *identity;
+ unsigned int active:1;
+};
+
+struct typec_partner {
+ struct device dev;
+ unsigned int usb_pd:1;
+ struct usb_pd_identity *identity;
+ enum typec_accessory accessory;
+};
+
+struct typec_port {
+ unsigned int id;
+ struct device dev;
+
+ int prefer_role;
+ enum typec_data_role data_role;
+ enum typec_role pwr_role;
+ enum typec_role vconn_role;
+ enum typec_pwr_opmode pwr_opmode;
+ enum typec_port_type port_type;
+ struct mutex port_type_lock;
+
+ const struct typec_capability *cap;
+};
+
+#define to_typec_port(_dev_) container_of(_dev_, struct typec_port, dev)
+#define to_typec_plug(_dev_) container_of(_dev_, struct typec_plug, dev)
+#define to_typec_cable(_dev_) container_of(_dev_, struct typec_cable, dev)
+#define to_typec_partner(_dev_) container_of(_dev_, struct typec_partner, dev)
+#define to_altmode(_dev_) container_of(_dev_, struct typec_altmode, dev)
+
+static const struct device_type typec_partner_dev_type;
+static const struct device_type typec_cable_dev_type;
+static const struct device_type typec_plug_dev_type;
+static const struct device_type typec_port_dev_type;
+
+#define is_typec_partner(_dev_) (_dev_->type == &typec_partner_dev_type)
+#define is_typec_cable(_dev_) (_dev_->type == &typec_cable_dev_type)
+#define is_typec_plug(_dev_) (_dev_->type == &typec_plug_dev_type)
+#define is_typec_port(_dev_) (_dev_->type == &typec_port_dev_type)
+
+static DEFINE_IDA(typec_index_ida);
+static struct class *typec_class;
+
+/* Common attributes */
+
+static const char * const typec_accessory_modes[] = {
+ [TYPEC_ACCESSORY_NONE] = "none",
+ [TYPEC_ACCESSORY_AUDIO] = "analog_audio",
+ [TYPEC_ACCESSORY_DEBUG] = "debug",
+};
+
+static struct usb_pd_identity *get_pd_identity(struct device *dev)
+{
+ if (is_typec_partner(dev)) {
+ struct typec_partner *partner = to_typec_partner(dev);
+
+ return partner->identity;
+ } else if (is_typec_cable(dev)) {
+ struct typec_cable *cable = to_typec_cable(dev);
+
+ return cable->identity;
+ }
+ return NULL;
+}
+
+static ssize_t id_header_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct usb_pd_identity *id = get_pd_identity(dev);
+
+ return sprintf(buf, "0x%08x\n", id->id_header);
+}
+static DEVICE_ATTR_RO(id_header);
+
+static ssize_t cert_stat_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct usb_pd_identity *id = get_pd_identity(dev);
+
+ return sprintf(buf, "0x%08x\n", id->cert_stat);
+}
+static DEVICE_ATTR_RO(cert_stat);
+
+static ssize_t product_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct usb_pd_identity *id = get_pd_identity(dev);
+
+ return sprintf(buf, "0x%08x\n", id->product);
+}
+static DEVICE_ATTR_RO(product);
+
+static struct attribute *usb_pd_id_attrs[] = {
+ &dev_attr_id_header.attr,
+ &dev_attr_cert_stat.attr,
+ &dev_attr_product.attr,
+ NULL
+};
+
+static const struct attribute_group usb_pd_id_group = {
+ .name = "identity",
+ .attrs = usb_pd_id_attrs,
+};
+
+static const struct attribute_group *usb_pd_id_groups[] = {
+ &usb_pd_id_group,
+ NULL,
+};
+
+static void typec_report_identity(struct device *dev)
+{
+ sysfs_notify(&dev->kobj, "identity", "id_header");
+ sysfs_notify(&dev->kobj, "identity", "cert_stat");
+ sysfs_notify(&dev->kobj, "identity", "product");
+}
+
+/* ------------------------------------------------------------------------- */
+/* Alternate Modes */
+
+/**
+ * typec_altmode_update_active - Report Enter/Exit mode
+ * @alt: Handle to the alternate mode
+ * @mode: Mode index
+ * @active: True when the mode has been entered
+ *
+ * If a partner or cable plug executes Enter/Exit Mode command successfully, the
+ * drivers use this routine to report the updated state of the mode.
+ */
+void typec_altmode_update_active(struct typec_altmode *alt, int mode,
+ bool active)
+{
+ struct typec_mode *m = &alt->modes[mode];
+ char dir[6];
+
+ if (m->active == active)
+ return;
+
+ m->active = active;
+ snprintf(dir, sizeof(dir), "mode%d", mode);
+ sysfs_notify(&alt->dev.kobj, dir, "active");
+ kobject_uevent(&alt->dev.kobj, KOBJ_CHANGE);
+}
+EXPORT_SYMBOL_GPL(typec_altmode_update_active);
+
+/**
+ * typec_altmode2port - Alternate Mode to USB Type-C port
+ * @alt: The Alternate Mode
+ *
+ * Returns handle to the port that a cable plug or partner with @alt is
+ * connected to.
+ */
+struct typec_port *typec_altmode2port(struct typec_altmode *alt)
+{
+ if (is_typec_plug(alt->dev.parent))
+ return to_typec_port(alt->dev.parent->parent->parent);
+ if (is_typec_partner(alt->dev.parent))
+ return to_typec_port(alt->dev.parent->parent);
+ if (is_typec_port(alt->dev.parent))
+ return to_typec_port(alt->dev.parent);
+
+ return NULL;
+}
+EXPORT_SYMBOL_GPL(typec_altmode2port);
+
+static ssize_t
+typec_altmode_vdo_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_mode *mode = container_of(attr, struct typec_mode,
+ vdo_attr);
+
+ return sprintf(buf, "0x%08x\n", mode->vdo);
+}
+
+static ssize_t
+typec_altmode_desc_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_mode *mode = container_of(attr, struct typec_mode,
+ desc_attr);
+
+ return sprintf(buf, "%s\n", mode->desc ? mode->desc : "");
+}
+
+static ssize_t
+typec_altmode_active_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_mode *mode = container_of(attr, struct typec_mode,
+ active_attr);
+
+ return sprintf(buf, "%s\n", mode->active ? "yes" : "no");
+}
+
+static ssize_t
+typec_altmode_active_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t size)
+{
+ struct typec_mode *mode = container_of(attr, struct typec_mode,
+ active_attr);
+ struct typec_port *port = typec_altmode2port(mode->alt_mode);
+ bool activate;
+ int ret;
+
+ if (!port->cap->activate_mode)
+ return -EOPNOTSUPP;
+
+ ret = kstrtobool(buf, &activate);
+ if (ret)
+ return ret;
+
+ ret = port->cap->activate_mode(port->cap, mode->index, activate);
+ if (ret)
+ return ret;
+
+ return size;
+}
+
+static ssize_t
+typec_altmode_roles_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_mode *mode = container_of(attr, struct typec_mode,
+ roles_attr);
+ ssize_t ret;
+
+ switch (mode->roles) {
+ case TYPEC_PORT_DFP:
+ ret = sprintf(buf, "source\n");
+ break;
+ case TYPEC_PORT_UFP:
+ ret = sprintf(buf, "sink\n");
+ break;
+ case TYPEC_PORT_DRP:
+ default:
+ ret = sprintf(buf, "source sink\n");
+ break;
+ }
+ return ret;
+}
+
+static void typec_init_modes(struct typec_altmode *alt,
+ const struct typec_mode_desc *desc, bool is_port)
+{
+ int i;
+
+ for (i = 0; i < alt->n_modes; i++, desc++) {
+ struct typec_mode *mode = &alt->modes[i];
+
+ /* Not considering the human readable description critical */
+ mode->desc = kstrdup(desc->desc, GFP_KERNEL);
+ if (desc->desc && !mode->desc)
+ dev_err(&alt->dev, "failed to copy mode%d desc\n", i);
+
+ mode->alt_mode = alt;
+ mode->vdo = desc->vdo;
+ mode->roles = desc->roles;
+ mode->index = desc->index;
+ sprintf(mode->group_name, "mode%d", desc->index);
+
+ sysfs_attr_init(&mode->vdo_attr.attr);
+ mode->vdo_attr.attr.name = "vdo";
+ mode->vdo_attr.attr.mode = 0444;
+ mode->vdo_attr.show = typec_altmode_vdo_show;
+
+ sysfs_attr_init(&mode->desc_attr.attr);
+ mode->desc_attr.attr.name = "description";
+ mode->desc_attr.attr.mode = 0444;
+ mode->desc_attr.show = typec_altmode_desc_show;
+
+ sysfs_attr_init(&mode->active_attr.attr);
+ mode->active_attr.attr.name = "active";
+ mode->active_attr.attr.mode = 0644;
+ mode->active_attr.show = typec_altmode_active_show;
+ mode->active_attr.store = typec_altmode_active_store;
+
+ mode->attrs[0] = &mode->vdo_attr.attr;
+ mode->attrs[1] = &mode->desc_attr.attr;
+ mode->attrs[2] = &mode->active_attr.attr;
+
+ /* With ports, list the roles that the mode is supported with */
+ if (is_port) {
+ sysfs_attr_init(&mode->roles_attr.attr);
+ mode->roles_attr.attr.name = "supported_roles";
+ mode->roles_attr.attr.mode = 0444;
+ mode->roles_attr.show = typec_altmode_roles_show;
+
+ mode->attrs[3] = &mode->roles_attr.attr;
+ }
+
+ mode->group.attrs = mode->attrs;
+ mode->group.name = mode->group_name;
+
+ alt->mode_groups[i] = &mode->group;
+ }
+}
+
+static ssize_t svid_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_altmode *alt = to_altmode(dev);
+
+ return sprintf(buf, "%04x\n", alt->svid);
+}
+static DEVICE_ATTR_RO(svid);
+
+static struct attribute *typec_altmode_attrs[] = {
+ &dev_attr_svid.attr,
+ NULL
+};
+ATTRIBUTE_GROUPS(typec_altmode);
+
+static void typec_altmode_release(struct device *dev)
+{
+ struct typec_altmode *alt = to_altmode(dev);
+ int i;
+
+ for (i = 0; i < alt->n_modes; i++)
+ kfree(alt->modes[i].desc);
+ kfree(alt);
+}
+
+static const struct device_type typec_altmode_dev_type = {
+ .name = "typec_alternate_mode",
+ .groups = typec_altmode_groups,
+ .release = typec_altmode_release,
+};
+
+static struct typec_altmode *
+typec_register_altmode(struct device *parent,
+ const struct typec_altmode_desc *desc)
+{
+ struct typec_altmode *alt;
+ int ret;
+
+ alt = kzalloc(sizeof(*alt), GFP_KERNEL);
+ if (!alt)
+ return NULL;
+
+ alt->svid = desc->svid;
+ alt->n_modes = desc->n_modes;
+ typec_init_modes(alt, desc->modes, is_typec_port(parent));
+
+ alt->dev.parent = parent;
+ alt->dev.groups = alt->mode_groups;
+ alt->dev.type = &typec_altmode_dev_type;
+ dev_set_name(&alt->dev, "svid-%04x", alt->svid);
+
+ ret = device_register(&alt->dev);
+ if (ret) {
+ dev_err(parent, "failed to register alternate mode (%d)\n",
+ ret);
+ put_device(&alt->dev);
+ return NULL;
+ }
+
+ return alt;
+}
+
+/**
+ * typec_unregister_altmode - Unregister Alternate Mode
+ * @alt: The alternate mode to be unregistered
+ *
+ * Unregister device created with typec_partner_register_altmode(),
+ * typec_plug_register_altmode() or typec_port_register_altmode().
+ */
+void typec_unregister_altmode(struct typec_altmode *alt)
+{
+ if (alt)
+ device_unregister(&alt->dev);
+}
+EXPORT_SYMBOL_GPL(typec_unregister_altmode);
+
+/* ------------------------------------------------------------------------- */
+/* Type-C Partners */
+
+static ssize_t accessory_mode_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_partner *p = to_typec_partner(dev);
+
+ return sprintf(buf, "%s\n", typec_accessory_modes[p->accessory]);
+}
+static DEVICE_ATTR_RO(accessory_mode);
+
+static ssize_t supports_usb_power_delivery_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_partner *p = to_typec_partner(dev);
+
+ return sprintf(buf, "%s\n", p->usb_pd ? "yes" : "no");
+}
+static DEVICE_ATTR_RO(supports_usb_power_delivery);
+
+static struct attribute *typec_partner_attrs[] = {
+ &dev_attr_accessory_mode.attr,
+ &dev_attr_supports_usb_power_delivery.attr,
+ NULL
+};
+ATTRIBUTE_GROUPS(typec_partner);
+
+static void typec_partner_release(struct device *dev)
+{
+ struct typec_partner *partner = to_typec_partner(dev);
+
+ kfree(partner);
+}
+
+static const struct device_type typec_partner_dev_type = {
+ .name = "typec_partner",
+ .groups = typec_partner_groups,
+ .release = typec_partner_release,
+};
+
+/**
+ * typec_partner_set_identity - Report result from Discover Identity command
+ * @partner: The partner updated identity values
+ *
+ * This routine is used to report that the result of Discover Identity USB power
+ * delivery command has become available.
+ */
+int typec_partner_set_identity(struct typec_partner *partner)
+{
+ if (!partner->identity)
+ return -EINVAL;
+
+ typec_report_identity(&partner->dev);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(typec_partner_set_identity);
+
+/**
+ * typec_partner_register_altmode - Register USB Type-C Partner Alternate Mode
+ * @partner: USB Type-C Partner that supports the alternate mode
+ * @desc: Description of the alternate mode
+ *
+ * This routine is used to register each alternate mode individually that
+ * @partner has listed in response to Discover SVIDs command. The modes for a
+ * SVID listed in response to Discover Modes command need to be listed in an
+ * array in @desc.
+ *
+ * Returns handle to the alternate mode on success or NULL on failure.
+ */
+struct typec_altmode *
+typec_partner_register_altmode(struct typec_partner *partner,
+ const struct typec_altmode_desc *desc)
+{
+ return typec_register_altmode(&partner->dev, desc);
+}
+EXPORT_SYMBOL_GPL(typec_partner_register_altmode);
+
+/**
+ * typec_register_partner - Register a USB Type-C Partner
+ * @port: The USB Type-C Port the partner is connected to
+ * @desc: Description of the partner
+ *
+ * Registers a device for USB Type-C Partner described in @desc.
+ *
+ * Returns handle to the partner on success or NULL on failure.
+ */
+struct typec_partner *typec_register_partner(struct typec_port *port,
+ struct typec_partner_desc *desc)
+{
+ struct typec_partner *partner;
+ int ret;
+
+ partner = kzalloc(sizeof(*partner), GFP_KERNEL);
+ if (!partner)
+ return NULL;
+
+ partner->usb_pd = desc->usb_pd;
+ partner->accessory = desc->accessory;
+
+ if (desc->identity) {
+ /*
+ * Creating directory for the identity only if the driver is
+ * able to provide data to it.
+ */
+ partner->dev.groups = usb_pd_id_groups;
+ partner->identity = desc->identity;
+ }
+
+ partner->dev.class = typec_class;
+ partner->dev.parent = &port->dev;
+ partner->dev.type = &typec_partner_dev_type;
+ dev_set_name(&partner->dev, "%s-partner", dev_name(&port->dev));
+
+ ret = device_register(&partner->dev);
+ if (ret) {
+ dev_err(&port->dev, "failed to register partner (%d)\n", ret);
+ put_device(&partner->dev);
+ return NULL;
+ }
+
+ return partner;
+}
+EXPORT_SYMBOL_GPL(typec_register_partner);
+
+/**
+ * typec_unregister_partner - Unregister a USB Type-C Partner
+ * @partner: The partner to be unregistered
+ *
+ * Unregister device created with typec_register_partner().
+ */
+void typec_unregister_partner(struct typec_partner *partner)
+{
+ if (partner)
+ device_unregister(&partner->dev);
+}
+EXPORT_SYMBOL_GPL(typec_unregister_partner);
+
+/* ------------------------------------------------------------------------- */
+/* Type-C Cable Plugs */
+
+static void typec_plug_release(struct device *dev)
+{
+ struct typec_plug *plug = to_typec_plug(dev);
+
+ kfree(plug);
+}
+
+static const struct device_type typec_plug_dev_type = {
+ .name = "typec_plug",
+ .release = typec_plug_release,
+};
+
+/**
+ * typec_plug_register_altmode - Register USB Type-C Cable Plug Alternate Mode
+ * @plug: USB Type-C Cable Plug that supports the alternate mode
+ * @desc: Description of the alternate mode
+ *
+ * This routine is used to register each alternate mode individually that @plug
+ * has listed in response to Discover SVIDs command. The modes for a SVID that
+ * the plug lists in response to Discover Modes command need to be listed in an
+ * array in @desc.
+ *
+ * Returns handle to the alternate mode on success or NULL on failure.
+ */
+struct typec_altmode *
+typec_plug_register_altmode(struct typec_plug *plug,
+ const struct typec_altmode_desc *desc)
+{
+ return typec_register_altmode(&plug->dev, desc);
+}
+EXPORT_SYMBOL_GPL(typec_plug_register_altmode);
+
+/**
+ * typec_register_plug - Register a USB Type-C Cable Plug
+ * @cable: USB Type-C Cable with the plug
+ * @desc: Description of the cable plug
+ *
+ * Registers a device for USB Type-C Cable Plug described in @desc. A USB Type-C
+ * Cable Plug represents a plug with electronics in it that can response to USB
+ * Power Delivery SOP Prime or SOP Double Prime packages.
+ *
+ * Returns handle to the cable plug on success or NULL on failure.
+ */
+struct typec_plug *typec_register_plug(struct typec_cable *cable,
+ struct typec_plug_desc *desc)
+{
+ struct typec_plug *plug;
+ char name[8];
+ int ret;
+
+ plug = kzalloc(sizeof(*plug), GFP_KERNEL);
+ if (!plug)
+ return NULL;
+
+ sprintf(name, "plug%d", desc->index);
+
+ plug->index = desc->index;
+ plug->dev.class = typec_class;
+ plug->dev.parent = &cable->dev;
+ plug->dev.type = &typec_plug_dev_type;
+ dev_set_name(&plug->dev, "%s-%s", dev_name(cable->dev.parent), name);
+
+ ret = device_register(&plug->dev);
+ if (ret) {
+ dev_err(&cable->dev, "failed to register plug (%d)\n", ret);
+ put_device(&plug->dev);
+ return NULL;
+ }
+
+ return plug;
+}
+EXPORT_SYMBOL_GPL(typec_register_plug);
+
+/**
+ * typec_unregister_plug - Unregister a USB Type-C Cable Plug
+ * @plug: The cable plug to be unregistered
+ *
+ * Unregister device created with typec_register_plug().
+ */
+void typec_unregister_plug(struct typec_plug *plug)
+{
+ if (plug)
+ device_unregister(&plug->dev);
+}
+EXPORT_SYMBOL_GPL(typec_unregister_plug);
+
+/* Type-C Cables */
+
+static ssize_t
+type_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ struct typec_cable *cable = to_typec_cable(dev);
+
+ return sprintf(buf, "%s\n", cable->active ? "active" : "passive");
+}
+static DEVICE_ATTR_RO(type);
+
+static const char * const typec_plug_types[] = {
+ [USB_PLUG_NONE] = "unknown",
+ [USB_PLUG_TYPE_A] = "type-a",
+ [USB_PLUG_TYPE_B] = "type-b",
+ [USB_PLUG_TYPE_C] = "type-c",
+ [USB_PLUG_CAPTIVE] = "captive",
+};
+
+static ssize_t plug_type_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct typec_cable *cable = to_typec_cable(dev);
+
+ return sprintf(buf, "%s\n", typec_plug_types[cable->type]);
+}
+static DEVICE_ATTR_RO(plug_type);
+
+static struct attribute *typec_cable_attrs[] = {
+ &dev_attr_type.attr,
+ &dev_attr_plug_type.attr,
+ NULL
+};
+ATTRIBUTE_GROUPS(typec_cable);
+
+static void typec_cable_release(struct device *dev)
+{
+ struct typec_cable *cable = to_typec_cable(dev);
+
+ kfree(cable);
+}
+
+static const struct device_type typec_cable_dev_type = {
+ .name = "typec_cable",
+ .groups = typec_cable_groups,
+ .release = typec_cable_release,
+};
+
+/**
+ * typec_cable_set_identity - Report result from Discover Identity command
+ * @cable: The cable updated identity values
+ *
+ * This routine is used to report that the result of Discover Identity USB power
+ * delivery command has become available.
+ */
+int typec_cable_set_identity(struct typec_cable *cable)
+{
+ if (!cable->identity)
+ return -EINVAL;
+
+ typec_report_identity(&cable->dev);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(typec_cable_set_identity);
+
+/**
+ * typec_register_cable - Register a USB Type-C Cable
+ * @port: The USB Type-C Port the cable is connected to
+ * @desc: Description of the cable
+ *
+ * Registers a device for USB Type-C Cable described in @desc. The cable will be
+ * parent for the optional cable plug devises.
+ *
+ * Returns handle to the cable on success or NULL on failure.
+ */
+struct typec_cable *typec_register_cable(struct typec_port *port,
+ struct typec_cable_desc *desc)
+{
+ struct typec_cable *cable;
+ int ret;
+
+ cable = kzalloc(sizeof(*cable), GFP_KERNEL);
+ if (!cable)
+ return NULL;
+
+ cable->type = desc->type;
+ cable->active = desc->active;
+
+ if (desc->identity) {
+ /*
+ * Creating directory for the identity only if the driver is
+ * able to provide data to it.
+ */
+ cable->dev.groups = usb_pd_id_groups;
+ cable->identity = desc->identity;
+ }
+
+ cable->dev.class = typec_class;
+ cable->dev.parent = &port->dev;
+ cable->dev.type = &typec_cable_dev_type;
+ dev_set_name(&cable->dev, "%s-cable", dev_name(&port->dev));
+
+ ret = device_register(&cable->dev);
+ if (ret) {
+ dev_err(&port->dev, "failed to register cable (%d)\n", ret);
+ put_device(&cable->dev);
+ return NULL;
+ }
+
+ return cable;
+}
+EXPORT_SYMBOL_GPL(typec_register_cable);
+
+/**
+ * typec_unregister_cable - Unregister a USB Type-C Cable
+ * @cable: The cable to be unregistered
+ *
+ * Unregister device created with typec_register_cable().
+ */
+void typec_unregister_cable(struct typec_cable *cable)
+{
+ if (cable)
+ device_unregister(&cable->dev);
+}
+EXPORT_SYMBOL_GPL(typec_unregister_cable);
+
+/* ------------------------------------------------------------------------- */
+/* USB Type-C ports */
+
+static const char * const typec_roles[] = {
+ [TYPEC_SINK] = "sink",
+ [TYPEC_SOURCE] = "source",
+};
+
+static const char * const typec_data_roles[] = {
+ [TYPEC_DEVICE] = "device",
+ [TYPEC_HOST] = "host",
+};
+
+static const char * const typec_port_types[] = {
+ [TYPEC_PORT_DFP] = "source",
+ [TYPEC_PORT_UFP] = "sink",
+ [TYPEC_PORT_DRP] = "dual",
+};
+
+static const char * const typec_port_types_drp[] = {
+ [TYPEC_PORT_DFP] = "dual [source] sink",
+ [TYPEC_PORT_UFP] = "dual source [sink]",
+ [TYPEC_PORT_DRP] = "[dual] source sink",
+};
+
+static ssize_t
+preferred_role_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t size)
+{
+ struct typec_port *port = to_typec_port(dev);
+ int role;
+ int ret;
+
+ if (port->cap->type != TYPEC_PORT_DRP) {
+ dev_dbg(dev, "Preferred role only supported with DRP ports\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (!port->cap->try_role) {
+ dev_dbg(dev, "Setting preferred role not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ role = sysfs_match_string(typec_roles, buf);
+ if (role < 0) {
+ if (sysfs_streq(buf, "none"))
+ role = TYPEC_NO_PREFERRED_ROLE;
+ else
+ return -EINVAL;
+ }
+
+ ret = port->cap->try_role(port->cap, role);
+ if (ret)
+ return ret;
+
+ port->prefer_role = role;
+ return size;
+}
+
+static ssize_t
+preferred_role_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_port *port = to_typec_port(dev);
+
+ if (port->cap->type != TYPEC_PORT_DRP)
+ return 0;
+
+ if (port->prefer_role < 0)
+ return 0;
+
+ return sprintf(buf, "%s\n", typec_roles[port->prefer_role]);
+}
+static DEVICE_ATTR_RW(preferred_role);
+
+static ssize_t data_role_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t size)
+{
+ struct typec_port *port = to_typec_port(dev);
+ int ret;
+
+ if (!port->cap->dr_set) {
+ dev_dbg(dev, "data role swapping not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ ret = sysfs_match_string(typec_data_roles, buf);
+ if (ret < 0)
+ return ret;
+
+ mutex_lock(&port->port_type_lock);
+ if (port->port_type != TYPEC_PORT_DRP) {
+ dev_dbg(dev, "port type fixed at \"%s\"",
+ typec_port_types[port->port_type]);
+ ret = -EOPNOTSUPP;
+ goto unlock_and_ret;
+ }
+
+ ret = port->cap->dr_set(port->cap, ret);
+ if (ret)
+ goto unlock_and_ret;
+
+ ret = size;
+unlock_and_ret:
+ mutex_unlock(&port->port_type_lock);
+ return ret;
+}
+
+static ssize_t data_role_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct typec_port *port = to_typec_port(dev);
+
+ if (port->cap->type == TYPEC_PORT_DRP)
+ return sprintf(buf, "%s\n", port->data_role == TYPEC_HOST ?
+ "[host] device" : "host [device]");
+
+ return sprintf(buf, "[%s]\n", typec_data_roles[port->data_role]);
+}
+static DEVICE_ATTR_RW(data_role);
+
+static ssize_t power_role_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t size)
+{
+ struct typec_port *port = to_typec_port(dev);
+ int ret;
+
+ if (!port->cap->pd_revision) {
+ dev_dbg(dev, "USB Power Delivery not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (!port->cap->pr_set) {
+ dev_dbg(dev, "power role swapping not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (port->pwr_opmode != TYPEC_PWR_MODE_PD) {
+ dev_dbg(dev, "partner unable to swap power role\n");
+ return -EIO;
+ }
+
+ ret = sysfs_match_string(typec_roles, buf);
+ if (ret < 0)
+ return ret;
+
+ mutex_lock(&port->port_type_lock);
+ if (port->port_type != TYPEC_PORT_DRP) {
+ dev_dbg(dev, "port type fixed at \"%s\"",
+ typec_port_types[port->port_type]);
+ ret = -EOPNOTSUPP;
+ goto unlock_and_ret;
+ }
+
+ ret = port->cap->pr_set(port->cap, ret);
+ if (ret)
+ goto unlock_and_ret;
+
+ ret = size;
+unlock_and_ret:
+ mutex_unlock(&port->port_type_lock);
+ return ret;
+}
+
+static ssize_t power_role_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct typec_port *port = to_typec_port(dev);
+
+ if (port->cap->type == TYPEC_PORT_DRP)
+ return sprintf(buf, "%s\n", port->pwr_role == TYPEC_SOURCE ?
+ "[source] sink" : "source [sink]");
+
+ return sprintf(buf, "[%s]\n", typec_roles[port->pwr_role]);
+}
+static DEVICE_ATTR_RW(power_role);
+
+static ssize_t
+port_type_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t size)
+{
+ struct typec_port *port = to_typec_port(dev);
+ int ret;
+ enum typec_port_type type;
+
+ if (!port->cap->port_type_set || port->cap->type != TYPEC_PORT_DRP) {
+ dev_dbg(dev, "changing port type not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ ret = sysfs_match_string(typec_port_types, buf);
+ if (ret < 0)
+ return ret;
+
+ type = ret;
+ mutex_lock(&port->port_type_lock);
+
+ if (port->port_type == type) {
+ ret = size;
+ goto unlock_and_ret;
+ }
+
+ ret = port->cap->port_type_set(port->cap, type);
+ if (ret)
+ goto unlock_and_ret;
+
+ port->port_type = type;
+ ret = size;
+
+unlock_and_ret:
+ mutex_unlock(&port->port_type_lock);
+ return ret;
+}
+
+static ssize_t
+port_type_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_port *port = to_typec_port(dev);
+
+ if (port->cap->type == TYPEC_PORT_DRP)
+ return sprintf(buf, "%s\n",
+ typec_port_types_drp[port->port_type]);
+
+ return sprintf(buf, "[%s]\n", typec_port_types[port->cap->type]);
+}
+static DEVICE_ATTR_RW(port_type);
+
+static const char * const typec_pwr_opmodes[] = {
+ [TYPEC_PWR_MODE_USB] = "default",
+ [TYPEC_PWR_MODE_1_5A] = "1.5A",
+ [TYPEC_PWR_MODE_3_0A] = "3.0A",
+ [TYPEC_PWR_MODE_PD] = "usb_power_delivery",
+};
+
+static ssize_t power_operation_mode_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_port *port = to_typec_port(dev);
+
+ return sprintf(buf, "%s\n", typec_pwr_opmodes[port->pwr_opmode]);
+}
+static DEVICE_ATTR_RO(power_operation_mode);
+
+static ssize_t vconn_source_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t size)
+{
+ struct typec_port *port = to_typec_port(dev);
+ bool source;
+ int ret;
+
+ if (!port->cap->pd_revision) {
+ dev_dbg(dev, "VCONN swap depends on USB Power Delivery\n");
+ return -EOPNOTSUPP;
+ }
+
+ if (!port->cap->vconn_set) {
+ dev_dbg(dev, "VCONN swapping not supported\n");
+ return -EOPNOTSUPP;
+ }
+
+ ret = kstrtobool(buf, &source);
+ if (ret)
+ return ret;
+
+ ret = port->cap->vconn_set(port->cap, (enum typec_role)source);
+ if (ret)
+ return ret;
+
+ return size;
+}
+
+static ssize_t vconn_source_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct typec_port *port = to_typec_port(dev);
+
+ return sprintf(buf, "%s\n",
+ port->vconn_role == TYPEC_SOURCE ? "yes" : "no");
+}
+static DEVICE_ATTR_RW(vconn_source);
+
+static ssize_t supported_accessory_modes_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_port *port = to_typec_port(dev);
+ ssize_t ret = 0;
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(port->cap->accessory); i++) {
+ if (port->cap->accessory[i])
+ ret += sprintf(buf + ret, "%s ",
+ typec_accessory_modes[port->cap->accessory[i]]);
+ }
+
+ if (!ret)
+ return sprintf(buf, "none\n");
+
+ buf[ret - 1] = '\n';
+
+ return ret;
+}
+static DEVICE_ATTR_RO(supported_accessory_modes);
+
+static ssize_t usb_typec_revision_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_port *port = to_typec_port(dev);
+ u16 rev = port->cap->revision;
+
+ return sprintf(buf, "%d.%d\n", (rev >> 8) & 0xff, (rev >> 4) & 0xf);
+}
+static DEVICE_ATTR_RO(usb_typec_revision);
+
+static ssize_t usb_power_delivery_revision_show(struct device *dev,
+ struct device_attribute *attr,
+ char *buf)
+{
+ struct typec_port *p = to_typec_port(dev);
+
+ return sprintf(buf, "%d\n", (p->cap->pd_revision >> 8) & 0xff);
+}
+static DEVICE_ATTR_RO(usb_power_delivery_revision);
+
+static struct attribute *typec_attrs[] = {
+ &dev_attr_data_role.attr,
+ &dev_attr_power_operation_mode.attr,
+ &dev_attr_power_role.attr,
+ &dev_attr_preferred_role.attr,
+ &dev_attr_supported_accessory_modes.attr,
+ &dev_attr_usb_power_delivery_revision.attr,
+ &dev_attr_usb_typec_revision.attr,
+ &dev_attr_vconn_source.attr,
+ &dev_attr_port_type.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(typec);
+
+static int typec_uevent(struct device *dev, struct kobj_uevent_env *env)
+{
+ int ret;
+
+ ret = add_uevent_var(env, "TYPEC_PORT=%s", dev_name(dev));
+ if (ret)
+ dev_err(dev, "failed to add uevent TYPEC_PORT\n");
+
+ return ret;
+}
+
+static void typec_release(struct device *dev)
+{
+ struct typec_port *port = to_typec_port(dev);
+
+ ida_simple_remove(&typec_index_ida, port->id);
+ kfree(port);
+}
+
+static const struct device_type typec_port_dev_type = {
+ .name = "typec_port",
+ .groups = typec_groups,
+ .uevent = typec_uevent,
+ .release = typec_release,
+};
+
+/* --------------------------------------- */
+/* Driver callbacks to report role updates */
+
+/**
+ * typec_set_data_role - Report data role change
+ * @port: The USB Type-C Port where the role was changed
+ * @role: The new data role
+ *
+ * This routine is used by the port drivers to report data role changes.
+ */
+void typec_set_data_role(struct typec_port *port, enum typec_data_role role)
+{
+ if (port->data_role == role)
+ return;
+
+ port->data_role = role;
+ sysfs_notify(&port->dev.kobj, NULL, "data_role");
+ kobject_uevent(&port->dev.kobj, KOBJ_CHANGE);
+}
+EXPORT_SYMBOL_GPL(typec_set_data_role);
+
+/**
+ * typec_set_pwr_role - Report power role change
+ * @port: The USB Type-C Port where the role was changed
+ * @role: The new data role
+ *
+ * This routine is used by the port drivers to report power role changes.
+ */
+void typec_set_pwr_role(struct typec_port *port, enum typec_role role)
+{
+ if (port->pwr_role == role)
+ return;
+
+ port->pwr_role = role;
+ sysfs_notify(&port->dev.kobj, NULL, "power_role");
+ kobject_uevent(&port->dev.kobj, KOBJ_CHANGE);
+}
+EXPORT_SYMBOL_GPL(typec_set_pwr_role);
+
+/**
+ * typec_set_pwr_role - Report VCONN source change
+ * @port: The USB Type-C Port which VCONN role changed
+ * @role: Source when @port is sourcing VCONN, or Sink when it's not
+ *
+ * This routine is used by the port drivers to report if the VCONN source is
+ * changes.
+ */
+void typec_set_vconn_role(struct typec_port *port, enum typec_role role)
+{
+ if (port->vconn_role == role)
+ return;
+
+ port->vconn_role = role;
+ sysfs_notify(&port->dev.kobj, NULL, "vconn_source");
+ kobject_uevent(&port->dev.kobj, KOBJ_CHANGE);
+}
+EXPORT_SYMBOL_GPL(typec_set_vconn_role);
+
+static int partner_match(struct device *dev, void *data)
+{
+ return is_typec_partner(dev);
+}
+
+/**
+ * typec_set_pwr_opmode - Report changed power operation mode
+ * @port: The USB Type-C Port where the mode was changed
+ * @opmode: New power operation mode
+ *
+ * This routine is used by the port drivers to report changed power operation
+ * mode in @port. The modes are USB (default), 1.5A, 3.0A as defined in USB
+ * Type-C specification, and "USB Power Delivery" when the power levels are
+ * negotiated with methods defined in USB Power Delivery specification.
+ */
+void typec_set_pwr_opmode(struct typec_port *port,
+ enum typec_pwr_opmode opmode)
+{
+ struct device *partner_dev;
+
+ if (port->pwr_opmode == opmode)
+ return;
+
+ port->pwr_opmode = opmode;
+ sysfs_notify(&port->dev.kobj, NULL, "power_operation_mode");
+ kobject_uevent(&port->dev.kobj, KOBJ_CHANGE);
+
+ partner_dev = device_find_child(&port->dev, NULL, partner_match);
+ if (partner_dev) {
+ struct typec_partner *partner = to_typec_partner(partner_dev);
+
+ if (opmode == TYPEC_PWR_MODE_PD && !partner->usb_pd) {
+ partner->usb_pd = 1;
+ sysfs_notify(&partner_dev->kobj, NULL,
+ "supports_usb_power_delivery");
+ }
+ put_device(partner_dev);
+ }
+}
+EXPORT_SYMBOL_GPL(typec_set_pwr_opmode);
+
+/* --------------------------------------- */
+
+/**
+ * typec_port_register_altmode - Register USB Type-C Port Alternate Mode
+ * @port: USB Type-C Port that supports the alternate mode
+ * @desc: Description of the alternate mode
+ *
+ * This routine is used to register an alternate mode that @port is capable of
+ * supporting.
+ *
+ * Returns handle to the alternate mode on success or NULL on failure.
+ */
+struct typec_altmode *
+typec_port_register_altmode(struct typec_port *port,
+ const struct typec_altmode_desc *desc)
+{
+ return typec_register_altmode(&port->dev, desc);
+}
+EXPORT_SYMBOL_GPL(typec_port_register_altmode);
+
+/**
+ * typec_register_port - Register a USB Type-C Port
+ * @parent: Parent device
+ * @cap: Description of the port
+ *
+ * Registers a device for USB Type-C Port described in @cap.
+ *
+ * Returns handle to the port on success or NULL on failure.
+ */
+struct typec_port *typec_register_port(struct device *parent,
+ const struct typec_capability *cap)
+{
+ struct typec_port *port;
+ int role;
+ int ret;
+ int id;
+
+ port = kzalloc(sizeof(*port), GFP_KERNEL);
+ if (!port)
+ return NULL;
+
+ id = ida_simple_get(&typec_index_ida, 0, 0, GFP_KERNEL);
+ if (id < 0) {
+ kfree(port);
+ return NULL;
+ }
+
+ if (cap->type == TYPEC_PORT_DFP)
+ role = TYPEC_SOURCE;
+ else if (cap->type == TYPEC_PORT_UFP)
+ role = TYPEC_SINK;
+ else
+ role = cap->prefer_role;
+
+ if (role == TYPEC_SOURCE) {
+ port->data_role = TYPEC_HOST;
+ port->pwr_role = TYPEC_SOURCE;
+ port->vconn_role = TYPEC_SOURCE;
+ } else {
+ port->data_role = TYPEC_DEVICE;
+ port->pwr_role = TYPEC_SINK;
+ port->vconn_role = TYPEC_SINK;
+ }
+
+ port->id = id;
+ port->cap = cap;
+ port->port_type = cap->type;
+ mutex_init(&port->port_type_lock);
+ port->prefer_role = cap->prefer_role;
+
+ port->dev.class = typec_class;
+ port->dev.parent = parent;
+ port->dev.fwnode = cap->fwnode;
+ port->dev.type = &typec_port_dev_type;
+ dev_set_name(&port->dev, "port%d", id);
+
+ ret = device_register(&port->dev);
+ if (ret) {
+ dev_err(parent, "failed to register port (%d)\n", ret);
+ put_device(&port->dev);
+ return NULL;
+ }
+
+ return port;
+}
+EXPORT_SYMBOL_GPL(typec_register_port);
+
+/**
+ * typec_unregister_port - Unregister a USB Type-C Port
+ * @port: The port to be unregistered
+ *
+ * Unregister device created with typec_register_port().
+ */
+void typec_unregister_port(struct typec_port *port)
+{
+ if (port)
+ device_unregister(&port->dev);
+}
+EXPORT_SYMBOL_GPL(typec_unregister_port);
+
+static int __init typec_init(void)
+{
+ typec_class = class_create(THIS_MODULE, "typec");
+ return PTR_ERR_OR_ZERO(typec_class);
+}
+subsys_initcall(typec_init);
+
+static void __exit typec_exit(void)
+{
+ class_destroy(typec_class);
+ ida_destroy(&typec_index_ida);
+}
+module_exit(typec_exit);
+
+MODULE_AUTHOR("Heikki Krogerus <heikki.krogerus@linux.intel.com>");
+MODULE_LICENSE("GPL v2");
+MODULE_DESCRIPTION("USB Type-C Connector Class");
diff --git a/drivers/usb/usbip/stub.h b/drivers/usb/usbip/stub.h
index 910f027..84c0599 100644
--- a/drivers/usb/usbip/stub.h
+++ b/drivers/usb/usbip/stub.h
@@ -87,6 +87,7 @@ struct bus_id_priv {
struct stub_device *sdev;
struct usb_device *udev;
char shutdown_busid;
+ spinlock_t busid_lock;
};
/* stub_priv is allocated from stub_priv_cache */
@@ -97,6 +98,7 @@ extern struct usb_device_driver stub_driver;
/* stub_main.c */
struct bus_id_priv *get_busid_priv(const char *busid);
+void put_busid_priv(struct bus_id_priv *bid);
int del_match_busid(char *busid);
void stub_device_cleanup_urbs(struct stub_device *sdev);
diff --git a/drivers/usb/usbip/stub_dev.c b/drivers/usb/usbip/stub_dev.c
index 3550224..8e629b6 100644
--- a/drivers/usb/usbip/stub_dev.c
+++ b/drivers/usb/usbip/stub_dev.c
@@ -314,9 +314,9 @@ static int stub_probe(struct usb_device *udev)
struct stub_device *sdev = NULL;
const char *udev_busid = dev_name(&udev->dev);
struct bus_id_priv *busid_priv;
- int rc;
+ int rc = 0;
- dev_dbg(&udev->dev, "Enter\n");
+ dev_dbg(&udev->dev, "Enter probe\n");
/* check we should claim or not by busid_table */
busid_priv = get_busid_priv(udev_busid);
@@ -331,13 +331,15 @@ static int stub_probe(struct usb_device *udev)
* other matched drivers by the driver core.
* See driver_probe_device() in driver/base/dd.c
*/
- return -ENODEV;
+ rc = -ENODEV;
+ goto call_put_busid_priv;
}
if (udev->descriptor.bDeviceClass == USB_CLASS_HUB) {
dev_dbg(&udev->dev, "%s is a usb hub device... skip!\n",
udev_busid);
- return -ENODEV;
+ rc = -ENODEV;
+ goto call_put_busid_priv;
}
if (!strcmp(udev->bus->bus_name, "vhci_hcd")) {
@@ -345,13 +347,16 @@ static int stub_probe(struct usb_device *udev)
"%s is attached on vhci_hcd... skip!\n",
udev_busid);
- return -ENODEV;
+ rc = -ENODEV;
+ goto call_put_busid_priv;
}
/* ok, this is my device */
sdev = stub_device_alloc(udev);
- if (!sdev)
- return -ENOMEM;
+ if (!sdev) {
+ rc = -ENOMEM;
+ goto call_put_busid_priv;
+ }
dev_info(&udev->dev,
"usbip-host: register new device (bus %u dev %u)\n",
@@ -383,7 +388,9 @@ static int stub_probe(struct usb_device *udev)
}
busid_priv->status = STUB_BUSID_ALLOC;
- return 0;
+ rc = 0;
+ goto call_put_busid_priv;
+
err_files:
usb_hub_release_port(udev->parent, udev->portnum,
(struct usb_dev_state *) udev);
@@ -393,6 +400,9 @@ static int stub_probe(struct usb_device *udev)
busid_priv->sdev = NULL;
stub_device_free(sdev);
+
+call_put_busid_priv:
+ put_busid_priv(busid_priv);
return rc;
}
@@ -418,7 +428,7 @@ static void stub_disconnect(struct usb_device *udev)
struct bus_id_priv *busid_priv;
int rc;
- dev_dbg(&udev->dev, "Enter\n");
+ dev_dbg(&udev->dev, "Enter disconnect\n");
busid_priv = get_busid_priv(udev_busid);
if (!busid_priv) {
@@ -431,7 +441,7 @@ static void stub_disconnect(struct usb_device *udev)
/* get stub_device */
if (!sdev) {
dev_err(&udev->dev, "could not get device");
- return;
+ goto call_put_busid_priv;
}
dev_set_drvdata(&udev->dev, NULL);
@@ -446,12 +456,12 @@ static void stub_disconnect(struct usb_device *udev)
(struct usb_dev_state *) udev);
if (rc) {
dev_dbg(&udev->dev, "unable to release port\n");
- return;
+ goto call_put_busid_priv;
}
/* If usb reset is called from event handler */
if (usbip_in_eh(current))
- return;
+ goto call_put_busid_priv;
/* shutdown the current connection */
shutdown_busid(busid_priv);
@@ -462,12 +472,11 @@ static void stub_disconnect(struct usb_device *udev)
busid_priv->sdev = NULL;
stub_device_free(sdev);
- if (busid_priv->status == STUB_BUSID_ALLOC) {
+ if (busid_priv->status == STUB_BUSID_ALLOC)
busid_priv->status = STUB_BUSID_ADDED;
- } else {
- busid_priv->status = STUB_BUSID_OTHER;
- del_match_busid((char *)udev_busid);
- }
+
+call_put_busid_priv:
+ put_busid_priv(busid_priv);
}
#ifdef CONFIG_PM
diff --git a/drivers/usb/usbip/stub_main.c b/drivers/usb/usbip/stub_main.c
index 325b4c0..fa90496 100644
--- a/drivers/usb/usbip/stub_main.c
+++ b/drivers/usb/usbip/stub_main.c
@@ -28,6 +28,7 @@
#define DRIVER_DESC "USB/IP Host Driver"
struct kmem_cache *stub_priv_cache;
+
/*
* busid_tables defines matching busids that usbip can grab. A user can change
* dynamically what device is locally used and what device is exported to a
@@ -39,6 +40,8 @@ static spinlock_t busid_table_lock;
static void init_busid_table(void)
{
+ int i;
+
/*
* This also sets the bus_table[i].status to
* STUB_BUSID_OTHER, which is 0.
@@ -46,6 +49,9 @@ static void init_busid_table(void)
memset(busid_table, 0, sizeof(busid_table));
spin_lock_init(&busid_table_lock);
+
+ for (i = 0; i < MAX_BUSID; i++)
+ spin_lock_init(&busid_table[i].busid_lock);
}
/*
@@ -57,15 +63,20 @@ static int get_busid_idx(const char *busid)
int i;
int idx = -1;
- for (i = 0; i < MAX_BUSID; i++)
+ for (i = 0; i < MAX_BUSID; i++) {
+ spin_lock(&busid_table[i].busid_lock);
if (busid_table[i].name[0])
if (!strncmp(busid_table[i].name, busid, BUSID_SIZE)) {
idx = i;
+ spin_unlock(&busid_table[i].busid_lock);
break;
}
+ spin_unlock(&busid_table[i].busid_lock);
+ }
return idx;
}
+/* Returns holding busid_lock. Should call put_busid_priv() to unlock */
struct bus_id_priv *get_busid_priv(const char *busid)
{
int idx;
@@ -73,13 +84,22 @@ struct bus_id_priv *get_busid_priv(const char *busid)
spin_lock(&busid_table_lock);
idx = get_busid_idx(busid);
- if (idx >= 0)
+ if (idx >= 0) {
bid = &(busid_table[idx]);
+ /* get busid_lock before returning */
+ spin_lock(&bid->busid_lock);
+ }
spin_unlock(&busid_table_lock);
return bid;
}
+void put_busid_priv(struct bus_id_priv *bid)
+{
+ if (bid)
+ spin_unlock(&bid->busid_lock);
+}
+
static int add_match_busid(char *busid)
{
int i;
@@ -92,15 +112,19 @@ static int add_match_busid(char *busid)
goto out;
}
- for (i = 0; i < MAX_BUSID; i++)
+ for (i = 0; i < MAX_BUSID; i++) {
+ spin_lock(&busid_table[i].busid_lock);
if (!busid_table[i].name[0]) {
strlcpy(busid_table[i].name, busid, BUSID_SIZE);
if ((busid_table[i].status != STUB_BUSID_ALLOC) &&
(busid_table[i].status != STUB_BUSID_REMOV))
busid_table[i].status = STUB_BUSID_ADDED;
ret = 0;
+ spin_unlock(&busid_table[i].busid_lock);
break;
}
+ spin_unlock(&busid_table[i].busid_lock);
+ }
out:
spin_unlock(&busid_table_lock);
@@ -121,6 +145,8 @@ int del_match_busid(char *busid)
/* found */
ret = 0;
+ spin_lock(&busid_table[idx].busid_lock);
+
if (busid_table[idx].status == STUB_BUSID_OTHER)
memset(busid_table[idx].name, 0, BUSID_SIZE);
@@ -128,6 +154,7 @@ int del_match_busid(char *busid)
(busid_table[idx].status != STUB_BUSID_ADDED))
busid_table[idx].status = STUB_BUSID_REMOV;
+ spin_unlock(&busid_table[idx].busid_lock);
out:
spin_unlock(&busid_table_lock);
@@ -140,9 +167,12 @@ static ssize_t show_match_busid(struct device_driver *drv, char *buf)
char *out = buf;
spin_lock(&busid_table_lock);
- for (i = 0; i < MAX_BUSID; i++)
+ for (i = 0; i < MAX_BUSID; i++) {
+ spin_lock(&busid_table[i].busid_lock);
if (busid_table[i].name[0])
out += sprintf(out, "%s ", busid_table[i].name);
+ spin_unlock(&busid_table[i].busid_lock);
+ }
spin_unlock(&busid_table_lock);
out += sprintf(out, "\n");
@@ -184,6 +214,51 @@ static ssize_t store_match_busid(struct device_driver *dev, const char *buf,
static DRIVER_ATTR(match_busid, S_IRUSR | S_IWUSR, show_match_busid,
store_match_busid);
+static int do_rebind(char *busid, struct bus_id_priv *busid_priv)
+{
+ int ret;
+
+ /* device_attach() callers should hold parent lock for USB */
+ if (busid_priv->udev->dev.parent)
+ device_lock(busid_priv->udev->dev.parent);
+ ret = device_attach(&busid_priv->udev->dev);
+ if (busid_priv->udev->dev.parent)
+ device_unlock(busid_priv->udev->dev.parent);
+ if (ret < 0) {
+ dev_err(&busid_priv->udev->dev, "rebind failed\n");
+ return ret;
+ }
+ return 0;
+}
+
+static void stub_device_rebind(void)
+{
+#if IS_MODULE(CONFIG_USBIP_HOST)
+ struct bus_id_priv *busid_priv;
+ int i;
+
+ /* update status to STUB_BUSID_OTHER so probe ignores the device */
+ spin_lock(&busid_table_lock);
+ for (i = 0; i < MAX_BUSID; i++) {
+ if (busid_table[i].name[0] &&
+ busid_table[i].shutdown_busid) {
+ busid_priv = &(busid_table[i]);
+ busid_priv->status = STUB_BUSID_OTHER;
+ }
+ }
+ spin_unlock(&busid_table_lock);
+
+ /* now run rebind - no need to hold locks. driver files are removed */
+ for (i = 0; i < MAX_BUSID; i++) {
+ if (busid_table[i].name[0] &&
+ busid_table[i].shutdown_busid) {
+ busid_priv = &(busid_table[i]);
+ do_rebind(busid_table[i].name, busid_priv);
+ }
+ }
+#endif
+}
+
static ssize_t rebind_store(struct device_driver *dev, const char *buf,
size_t count)
{
@@ -201,11 +276,17 @@ static ssize_t rebind_store(struct device_driver *dev, const char *buf,
if (!bid)
return -ENODEV;
- ret = device_attach(&bid->udev->dev);
- if (ret < 0) {
- dev_err(&bid->udev->dev, "rebind failed\n");
+ /* mark the device for deletion so probe ignores it during rescan */
+ bid->status = STUB_BUSID_OTHER;
+ /* release the busid lock */
+ put_busid_priv(bid);
+
+ ret = do_rebind((char *) buf, bid);
+ if (ret < 0)
return ret;
- }
+
+ /* delete device from busid_table */
+ del_match_busid((char *) buf);
return count;
}
@@ -328,6 +409,9 @@ static void __exit usbip_host_exit(void)
*/
usb_deregister_device_driver(&stub_driver);
+ /* initiate scan to attach devices */
+ stub_device_rebind();
+
kmem_cache_destroy(stub_priv_cache);
}
diff --git a/drivers/usb/usbip/usbip_common.h b/drivers/usb/usbip/usbip_common.h
index f0b955f..109e65b 100644
--- a/drivers/usb/usbip/usbip_common.h
+++ b/drivers/usb/usbip/usbip_common.h
@@ -258,7 +258,7 @@ enum usbip_side {
#define VUDC_EVENT_ERROR_USB (USBIP_EH_SHUTDOWN | USBIP_EH_UNUSABLE)
#define VUDC_EVENT_ERROR_MALLOC (USBIP_EH_SHUTDOWN | USBIP_EH_UNUSABLE)
-#define VDEV_EVENT_REMOVED (USBIP_EH_SHUTDOWN | USBIP_EH_BYE)
+#define VDEV_EVENT_REMOVED (USBIP_EH_SHUTDOWN | USBIP_EH_RESET | USBIP_EH_BYE)
#define VDEV_EVENT_DOWN (USBIP_EH_SHUTDOWN | USBIP_EH_RESET)
#define VDEV_EVENT_ERROR_TCP (USBIP_EH_SHUTDOWN | USBIP_EH_RESET)
#define VDEV_EVENT_ERROR_MALLOC (USBIP_EH_SHUTDOWN | USBIP_EH_UNUSABLE)
diff --git a/drivers/usb/usbip/usbip_event.c b/drivers/usb/usbip/usbip_event.c
index f163566..f8f7f38 100644
--- a/drivers/usb/usbip/usbip_event.c
+++ b/drivers/usb/usbip/usbip_event.c
@@ -105,10 +105,6 @@ static void event_handler(struct work_struct *work)
unset_event(ud, USBIP_EH_UNUSABLE);
}
- /* Stop the error handler. */
- if (ud->event & USBIP_EH_BYE)
- usbip_dbg_eh("removed %p\n", ud);
-
wake_up(&ud->eh_waitq);
}
}
diff --git a/drivers/usb/usbip/vhci_sysfs.c b/drivers/usb/usbip/vhci_sysfs.c
index c287ccc..e8a008d 100644
--- a/drivers/usb/usbip/vhci_sysfs.c
+++ b/drivers/usb/usbip/vhci_sysfs.c
@@ -24,6 +24,9 @@
#include <linux/platform_device.h>
#include <linux/slab.h>
+/* Hardening for Spectre-v1 */
+#include <linux/nospec.h>
+
#include "usbip_common.h"
#include "vhci.h"
@@ -181,16 +184,20 @@ static int vhci_port_disconnect(struct vhci_hcd *vhci, __u32 rhport)
return 0;
}
-static int valid_port(__u32 pdev_nr, __u32 rhport)
+static int valid_port(__u32 *pdev_nr, __u32 *rhport)
{
- if (pdev_nr >= vhci_num_controllers) {
- pr_err("pdev %u\n", pdev_nr);
+ if (*pdev_nr >= vhci_num_controllers) {
+ pr_err("pdev %u\n", *pdev_nr);
return 0;
}
- if (rhport >= VHCI_HC_PORTS) {
- pr_err("rhport %u\n", rhport);
+ *pdev_nr = array_index_nospec(*pdev_nr, vhci_num_controllers);
+
+ if (*rhport >= VHCI_HC_PORTS) {
+ pr_err("rhport %u\n", *rhport);
return 0;
}
+ *rhport = array_index_nospec(*rhport, VHCI_HC_PORTS);
+
return 1;
}
@@ -207,7 +214,7 @@ static ssize_t store_detach(struct device *dev, struct device_attribute *attr,
pdev_nr = port_to_pdev_nr(port);
rhport = port_to_rhport(port);
- if (!valid_port(pdev_nr, rhport))
+ if (!valid_port(&pdev_nr, &rhport))
return -EINVAL;
hcd = platform_get_drvdata(*(vhci_pdevs + pdev_nr));
@@ -226,7 +233,8 @@ static ssize_t store_detach(struct device *dev, struct device_attribute *attr,
}
static DEVICE_ATTR(detach, S_IWUSR, NULL, store_detach);
-static int valid_args(__u32 pdev_nr, __u32 rhport, enum usb_device_speed speed)
+static int valid_args(__u32 *pdev_nr, __u32 *rhport,
+ enum usb_device_speed speed)
{
if (!valid_port(pdev_nr, rhport)) {
return 0;
@@ -288,7 +296,7 @@ static ssize_t store_attach(struct device *dev, struct device_attribute *attr,
sockfd, devid, speed);
/* check received parameters */
- if (!valid_args(pdev_nr, rhport, speed))
+ if (!valid_args(&pdev_nr, &rhport, speed))
return -EINVAL;
hcd = platform_get_drvdata(*(vhci_pdevs + pdev_nr));
diff --git a/drivers/usb/usbip/vudc_main.c b/drivers/usb/usbip/vudc_main.c
index 9e65571..916e2ee 100644
--- a/drivers/usb/usbip/vudc_main.c
+++ b/drivers/usb/usbip/vudc_main.c
@@ -85,6 +85,10 @@ static int __init init(void)
cleanup:
list_for_each_entry_safe(udc_dev, udc_dev2, &vudc_devices, dev_entry) {
list_del(&udc_dev->dev_entry);
+ /*
+ * Just do platform_device_del() here, put_vudc_device()
+ * calls the platform_device_put()
+ */
platform_device_del(udc_dev->pdev);
put_vudc_device(udc_dev);
}
@@ -101,7 +105,11 @@ static void __exit cleanup(void)
list_for_each_entry_safe(udc_dev, udc_dev2, &vudc_devices, dev_entry) {
list_del(&udc_dev->dev_entry);
- platform_device_unregister(udc_dev->pdev);
+ /*
+ * Just do platform_device_del() here, put_vudc_device()
+ * calls the platform_device_put()
+ */
+ platform_device_del(udc_dev->pdev);
put_vudc_device(udc_dev);
}
platform_driver_unregister(&vudc_driver);
diff --git a/drivers/usb/wusbcore/security.c b/drivers/usb/wusbcore/security.c
index 8c9421b..6bf86ca 100644
--- a/drivers/usb/wusbcore/security.c
+++ b/drivers/usb/wusbcore/security.c
@@ -230,7 +230,7 @@ int wusb_dev_sec_add(struct wusbhc *wusbhc,
result = usb_get_descriptor(usb_dev, USB_DT_SECURITY,
0, secd, sizeof(*secd));
- if (result < sizeof(*secd)) {
+ if (result < (int)sizeof(*secd)) {
dev_err(dev, "Can't read security descriptor or "
"not enough data: %d\n", result);
goto out;
diff --git a/drivers/uwb/hwa-rc.c b/drivers/uwb/hwa-rc.c
index 9a53912..5d3ba74 100644
--- a/drivers/uwb/hwa-rc.c
+++ b/drivers/uwb/hwa-rc.c
@@ -873,6 +873,7 @@ static int hwarc_probe(struct usb_interface *iface,
error_rc_add:
usb_put_intf(iface);
usb_put_dev(hwarc->usb_dev);
+ kfree(hwarc);
error_alloc:
uwb_rc_put(uwb_rc);
error_rc_alloc:
diff --git a/drivers/vfio/pci/vfio_pci.c b/drivers/vfio/pci/vfio_pci.c
index 43559be..7338e43 100644
--- a/drivers/vfio/pci/vfio_pci.c
+++ b/drivers/vfio/pci/vfio_pci.c
@@ -28,6 +28,7 @@
#include <linux/uaccess.h>
#include <linux/vfio.h>
#include <linux/vgaarb.h>
+#include <linux/nospec.h>
#include "vfio_pci_private.h"
@@ -755,6 +756,9 @@ static long vfio_pci_ioctl(void *device_data,
if (info.index >=
VFIO_PCI_NUM_REGIONS + vdev->num_regions)
return -EINVAL;
+ info.index = array_index_nospec(info.index,
+ VFIO_PCI_NUM_REGIONS +
+ vdev->num_regions);
i = info.index - VFIO_PCI_NUM_REGIONS;
diff --git a/drivers/vfio/platform/vfio_platform_common.c b/drivers/vfio/platform/vfio_platform_common.c
index d781428..d143d08 100644
--- a/drivers/vfio/platform/vfio_platform_common.c
+++ b/drivers/vfio/platform/vfio_platform_common.c
@@ -696,18 +696,23 @@ int vfio_platform_probe_common(struct vfio_platform_device *vdev,
group = vfio_iommu_group_get(dev);
if (!group) {
pr_err("VFIO: No IOMMU group for device %s\n", vdev->name);
- return -EINVAL;
+ ret = -EINVAL;
+ goto put_reset;
}
ret = vfio_add_group_dev(dev, &vfio_platform_ops, vdev);
- if (ret) {
- vfio_iommu_group_put(group, dev);
- return ret;
- }
+ if (ret)
+ goto put_iommu;
mutex_init(&vdev->igate);
return 0;
+
+put_iommu:
+ vfio_iommu_group_put(group, dev);
+put_reset:
+ vfio_platform_put_reset(vdev);
+ return ret;
}
EXPORT_SYMBOL_GPL(vfio_platform_probe_common);
diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c
index 487586e..353c93b 100644
--- a/drivers/vhost/net.c
+++ b/drivers/vhost/net.c
@@ -1052,7 +1052,8 @@ static long vhost_net_set_backend(struct vhost_net *n, unsigned index, int fd)
if (ubufs)
vhost_net_ubuf_put_wait_and_free(ubufs);
err_ubufs:
- sockfd_put(sock);
+ if (sock)
+ sockfd_put(sock);
err_vq:
mutex_unlock(&vq->mutex);
err:
diff --git a/drivers/vhost/scsi.c b/drivers/vhost/scsi.c
index 9e36632..17f9ad6 100644
--- a/drivers/vhost/scsi.c
+++ b/drivers/vhost/scsi.c
@@ -999,7 +999,8 @@ vhost_scsi_handle_vq(struct vhost_scsi *vs, struct vhost_virtqueue *vq)
prot_bytes = vhost32_to_cpu(vq, v_req_pi.pi_bytesin);
}
/*
- * Set prot_iter to data_iter, and advance past any
+ * Set prot_iter to data_iter and truncate it to
+ * prot_bytes, and advance data_iter past any
* preceeding prot_bytes that may be present.
*
* Also fix up the exp_data_len to reflect only the
@@ -1008,6 +1009,7 @@ vhost_scsi_handle_vq(struct vhost_scsi *vs, struct vhost_virtqueue *vq)
if (prot_bytes) {
exp_data_len -= prot_bytes;
prot_iter = data_iter;
+ iov_iter_truncate(&prot_iter, prot_bytes);
iov_iter_advance(&data_iter, prot_bytes);
}
tag = vhost64_to_cpu(vq, v_req_pi.tag);
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index fce49eb..53b1b3c 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -28,6 +28,7 @@
#include <linux/module.h>
#include <linux/sort.h>
#include <linux/interval_tree_generic.h>
+#include <linux/nospec.h>
#include "vhost.h"
@@ -905,7 +906,7 @@ static void vhost_iotlb_notify_vq(struct vhost_dev *d,
list_for_each_entry_safe(node, n, &d->pending_list, node) {
struct vhost_iotlb_msg *vq_msg = &node->msg.iotlb;
if (msg->iova <= vq_msg->iova &&
- msg->iova + msg->size - 1 > vq_msg->iova &&
+ msg->iova + msg->size - 1 >= vq_msg->iova &&
vq_msg->type == VHOST_IOTLB_MISS) {
vhost_poll_queue(&node->vq->poll);
list_del(&node->node);
@@ -938,6 +939,7 @@ int vhost_process_iotlb_msg(struct vhost_dev *dev,
{
int ret = 0;
+ mutex_lock(&dev->mutex);
vhost_dev_lock_vqs(dev);
switch (msg->type) {
case VHOST_IOTLB_UPDATE:
@@ -967,6 +969,8 @@ int vhost_process_iotlb_msg(struct vhost_dev *dev,
}
vhost_dev_unlock_vqs(dev);
+ mutex_unlock(&dev->mutex);
+
return ret;
}
ssize_t vhost_chr_write_iter(struct vhost_dev *dev,
@@ -1286,6 +1290,7 @@ long vhost_vring_ioctl(struct vhost_dev *d, int ioctl, void __user *argp)
if (idx >= d->nvqs)
return -ENOBUFS;
+ idx = array_index_nospec(idx, d->nvqs);
vq = d->vqs[idx];
mutex_lock(&vq->mutex);
@@ -2140,6 +2145,8 @@ int vhost_add_used_n(struct vhost_virtqueue *vq, struct vring_used_elem *heads,
return -EFAULT;
}
if (unlikely(vq->log_used)) {
+ /* Make sure used idx is seen before log. */
+ smp_wmb();
/* Log used index update. */
log_write(vq->log_base,
vq->log_addr + offsetof(struct vring_used, idx),
@@ -2292,6 +2299,9 @@ struct vhost_msg_node *vhost_new_msg(struct vhost_virtqueue *vq, int type)
struct vhost_msg_node *node = kmalloc(sizeof *node, GFP_KERNEL);
if (!node)
return NULL;
+
+ /* Make sure all padding within the structure is initialized. */
+ memset(&node->msg, 0, sizeof node->msg);
node->vq = vq;
node->msg.type = type;
return node;
diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c
index 0ec970c..72e914d 100644
--- a/drivers/vhost/vsock.c
+++ b/drivers/vhost/vsock.c
@@ -15,6 +15,7 @@
#include <net/sock.h>
#include <linux/virtio_vsock.h>
#include <linux/vhost.h>
+#include <linux/hashtable.h>
#include <net/af_vsock.h>
#include "vhost.h"
@@ -27,14 +28,14 @@ enum {
/* Used to track all the vhost_vsock instances on the system. */
static DEFINE_SPINLOCK(vhost_vsock_lock);
-static LIST_HEAD(vhost_vsock_list);
+static DEFINE_READ_MOSTLY_HASHTABLE(vhost_vsock_hash, 8);
struct vhost_vsock {
struct vhost_dev dev;
struct vhost_virtqueue vqs[2];
- /* Link to global vhost_vsock_list, protected by vhost_vsock_lock */
- struct list_head list;
+ /* Link to global vhost_vsock_hash, writes use vhost_vsock_lock */
+ struct hlist_node hash;
struct vhost_work send_pkt_work;
spinlock_t send_pkt_list_lock;
@@ -50,12 +51,14 @@ static u32 vhost_transport_get_local_cid(void)
return VHOST_VSOCK_DEFAULT_HOST_CID;
}
+/* Callers that dereference the return value must hold vhost_vsock_lock or the
+ * RCU read lock.
+ */
static struct vhost_vsock *vhost_vsock_get(u32 guest_cid)
{
struct vhost_vsock *vsock;
- spin_lock_bh(&vhost_vsock_lock);
- list_for_each_entry(vsock, &vhost_vsock_list, list) {
+ hash_for_each_possible_rcu(vhost_vsock_hash, vsock, hash, guest_cid) {
u32 other_cid = vsock->guest_cid;
/* Skip instances that have no CID yet */
@@ -63,11 +66,9 @@ static struct vhost_vsock *vhost_vsock_get(u32 guest_cid)
continue;
if (other_cid == guest_cid) {
- spin_unlock_bh(&vhost_vsock_lock);
return vsock;
}
}
- spin_unlock_bh(&vhost_vsock_lock);
return NULL;
}
@@ -198,9 +199,12 @@ vhost_transport_send_pkt(struct virtio_vsock_pkt *pkt)
struct vhost_virtqueue *vq;
int len = pkt->len;
+ rcu_read_lock();
+
/* Find the vhost_vsock according to guest context id */
vsock = vhost_vsock_get(le64_to_cpu(pkt->hdr.dst_cid));
if (!vsock) {
+ rcu_read_unlock();
virtio_transport_free_pkt(pkt);
return -ENODEV;
}
@@ -215,6 +219,8 @@ vhost_transport_send_pkt(struct virtio_vsock_pkt *pkt)
spin_unlock_bh(&vsock->send_pkt_list_lock);
vhost_work_queue(&vsock->dev, &vsock->send_pkt_work);
+
+ rcu_read_unlock();
return len;
}
@@ -224,12 +230,15 @@ vhost_transport_cancel_pkt(struct vsock_sock *vsk)
struct vhost_vsock *vsock;
struct virtio_vsock_pkt *pkt, *n;
int cnt = 0;
+ int ret = -ENODEV;
LIST_HEAD(freeme);
+ rcu_read_lock();
+
/* Find the vhost_vsock according to guest context id */
vsock = vhost_vsock_get(vsk->remote_addr.svm_cid);
if (!vsock)
- return -ENODEV;
+ goto out;
spin_lock_bh(&vsock->send_pkt_list_lock);
list_for_each_entry_safe(pkt, n, &vsock->send_pkt_list, list) {
@@ -255,7 +264,10 @@ vhost_transport_cancel_pkt(struct vsock_sock *vsk)
vhost_poll_queue(&tx_vq->poll);
}
- return 0;
+ ret = 0;
+out:
+ rcu_read_unlock();
+ return ret;
}
static struct virtio_vsock_pkt *
@@ -508,6 +520,8 @@ static int vhost_vsock_dev_open(struct inode *inode, struct file *file)
goto out;
}
+ vsock->guest_cid = 0; /* no CID assigned yet */
+
atomic_set(&vsock->queued_replies, 0);
vqs[VSOCK_VQ_TX] = &vsock->vqs[VSOCK_VQ_TX];
@@ -521,10 +535,6 @@ static int vhost_vsock_dev_open(struct inode *inode, struct file *file)
spin_lock_init(&vsock->send_pkt_list_lock);
INIT_LIST_HEAD(&vsock->send_pkt_list);
vhost_work_init(&vsock->send_pkt_work, vhost_transport_send_pkt_work);
-
- spin_lock_bh(&vhost_vsock_lock);
- list_add_tail(&vsock->list, &vhost_vsock_list);
- spin_unlock_bh(&vhost_vsock_lock);
return 0;
out:
@@ -551,13 +561,21 @@ static void vhost_vsock_reset_orphans(struct sock *sk)
* executing.
*/
- if (!vhost_vsock_get(vsk->remote_addr.svm_cid)) {
- sock_set_flag(sk, SOCK_DONE);
- vsk->peer_shutdown = SHUTDOWN_MASK;
- sk->sk_state = SS_UNCONNECTED;
- sk->sk_err = ECONNRESET;
- sk->sk_error_report(sk);
- }
+ /* If the peer is still valid, no need to reset connection */
+ if (vhost_vsock_get(vsk->remote_addr.svm_cid))
+ return;
+
+ /* If the close timeout is pending, let it expire. This avoids races
+ * with the timeout callback.
+ */
+ if (vsk->close_work_scheduled)
+ return;
+
+ sock_set_flag(sk, SOCK_DONE);
+ vsk->peer_shutdown = SHUTDOWN_MASK;
+ sk->sk_state = SS_UNCONNECTED;
+ sk->sk_err = ECONNRESET;
+ sk->sk_error_report(sk);
}
static int vhost_vsock_dev_release(struct inode *inode, struct file *file)
@@ -565,9 +583,13 @@ static int vhost_vsock_dev_release(struct inode *inode, struct file *file)
struct vhost_vsock *vsock = file->private_data;
spin_lock_bh(&vhost_vsock_lock);
- list_del(&vsock->list);
+ if (vsock->guest_cid)
+ hash_del_rcu(&vsock->hash);
spin_unlock_bh(&vhost_vsock_lock);
+ /* Wait for other CPUs to finish using vsock */
+ synchronize_rcu();
+
/* Iterating over all connections for all CIDs to find orphans is
* inefficient. Room for improvement here. */
vsock_for_each_connected_socket(vhost_vsock_reset_orphans);
@@ -607,12 +629,18 @@ static int vhost_vsock_set_cid(struct vhost_vsock *vsock, u64 guest_cid)
return -EINVAL;
/* Refuse if CID is already in use */
- other = vhost_vsock_get(guest_cid);
- if (other && other != vsock)
- return -EADDRINUSE;
-
spin_lock_bh(&vhost_vsock_lock);
+ other = vhost_vsock_get(guest_cid);
+ if (other && other != vsock) {
+ spin_unlock_bh(&vhost_vsock_lock);
+ return -EADDRINUSE;
+ }
+
+ if (vsock->guest_cid)
+ hash_del_rcu(&vsock->hash);
+
vsock->guest_cid = guest_cid;
+ hash_add_rcu(vhost_vsock_hash, &vsock->hash, guest_cid);
spin_unlock_bh(&vhost_vsock_lock);
return 0;
diff --git a/drivers/video/backlight/as3711_bl.c b/drivers/video/backlight/as3711_bl.c
index 734a915..e55304d 100644
--- a/drivers/video/backlight/as3711_bl.c
+++ b/drivers/video/backlight/as3711_bl.c
@@ -262,10 +262,10 @@ static int as3711_bl_register(struct platform_device *pdev,
static int as3711_backlight_parse_dt(struct device *dev)
{
struct as3711_bl_pdata *pdata = dev_get_platdata(dev);
- struct device_node *bl =
- of_find_node_by_name(dev->parent->of_node, "backlight"), *fb;
+ struct device_node *bl, *fb;
int ret;
+ bl = of_get_child_by_name(dev->parent->of_node, "backlight");
if (!bl) {
dev_dbg(dev, "backlight node not found\n");
return -ENODEV;
@@ -279,7 +279,7 @@ static int as3711_backlight_parse_dt(struct device *dev)
if (pdata->su1_max_uA <= 0)
ret = -EINVAL;
if (ret < 0)
- return ret;
+ goto err_put_bl;
}
fb = of_parse_phandle(bl, "su2-dev", 0);
@@ -292,7 +292,7 @@ static int as3711_backlight_parse_dt(struct device *dev)
if (pdata->su2_max_uA <= 0)
ret = -EINVAL;
if (ret < 0)
- return ret;
+ goto err_put_bl;
if (of_find_property(bl, "su2-feedback-voltage", NULL)) {
pdata->su2_feedback = AS3711_SU2_VOLTAGE;
@@ -314,8 +314,10 @@ static int as3711_backlight_parse_dt(struct device *dev)
pdata->su2_feedback = AS3711_SU2_CURR_AUTO;
count++;
}
- if (count != 1)
- return -EINVAL;
+ if (count != 1) {
+ ret = -EINVAL;
+ goto err_put_bl;
+ }
count = 0;
if (of_find_property(bl, "su2-fbprot-lx-sd4", NULL)) {
@@ -334,8 +336,10 @@ static int as3711_backlight_parse_dt(struct device *dev)
pdata->su2_fbprot = AS3711_SU2_GPIO4;
count++;
}
- if (count != 1)
- return -EINVAL;
+ if (count != 1) {
+ ret = -EINVAL;
+ goto err_put_bl;
+ }
count = 0;
if (of_find_property(bl, "su2-auto-curr1", NULL)) {
@@ -355,11 +359,20 @@ static int as3711_backlight_parse_dt(struct device *dev)
* At least one su2-auto-curr* must be specified iff
* AS3711_SU2_CURR_AUTO is used
*/
- if (!count ^ (pdata->su2_feedback != AS3711_SU2_CURR_AUTO))
- return -EINVAL;
+ if (!count ^ (pdata->su2_feedback != AS3711_SU2_CURR_AUTO)) {
+ ret = -EINVAL;
+ goto err_put_bl;
+ }
}
+ of_node_put(bl);
+
return 0;
+
+err_put_bl:
+ of_node_put(bl);
+
+ return ret;
}
static int as3711_backlight_probe(struct platform_device *pdev)
diff --git a/drivers/video/backlight/max8925_bl.c b/drivers/video/backlight/max8925_bl.c
index 7b738d6..f3aa608 100644
--- a/drivers/video/backlight/max8925_bl.c
+++ b/drivers/video/backlight/max8925_bl.c
@@ -116,7 +116,7 @@ static void max8925_backlight_dt_init(struct platform_device *pdev)
if (!pdata)
return;
- np = of_find_node_by_name(nproot, "backlight");
+ np = of_get_child_by_name(nproot, "backlight");
if (!np) {
dev_err(&pdev->dev, "failed to find backlight node\n");
return;
@@ -125,6 +125,8 @@ static void max8925_backlight_dt_init(struct platform_device *pdev)
if (!of_property_read_u32(np, "maxim,max8925-dual-string", &val))
pdata->dual_string = val;
+ of_node_put(np);
+
pdev->dev.platform_data = pdata;
}
diff --git a/drivers/video/backlight/tps65217_bl.c b/drivers/video/backlight/tps65217_bl.c
index fd524ad..f45d0c9 100644
--- a/drivers/video/backlight/tps65217_bl.c
+++ b/drivers/video/backlight/tps65217_bl.c
@@ -184,11 +184,11 @@ static struct tps65217_bl_pdata *
tps65217_bl_parse_dt(struct platform_device *pdev)
{
struct tps65217 *tps = dev_get_drvdata(pdev->dev.parent);
- struct device_node *node = of_node_get(tps->dev->of_node);
+ struct device_node *node;
struct tps65217_bl_pdata *pdata, *err;
u32 val;
- node = of_find_node_by_name(node, "backlight");
+ node = of_get_child_by_name(tps->dev->of_node, "backlight");
if (!node)
return ERR_PTR(-ENODEV);
diff --git a/drivers/video/fbdev/aty/atyfb.h b/drivers/video/fbdev/aty/atyfb.h
index 63c4842..46e0e8b 100644
--- a/drivers/video/fbdev/aty/atyfb.h
+++ b/drivers/video/fbdev/aty/atyfb.h
@@ -332,6 +332,8 @@ extern const struct aty_pll_ops aty_pll_ct; /* Integrated */
extern void aty_set_pll_ct(const struct fb_info *info, const union aty_pll *pll);
extern u8 aty_ld_pll_ct(int offset, const struct atyfb_par *par);
+extern const u8 aty_postdividers[8];
+
/*
* Hardware cursor support
@@ -358,7 +360,6 @@ static inline void wait_for_idle(struct atyfb_par *par)
extern void aty_reset_engine(const struct atyfb_par *par);
extern void aty_init_engine(struct atyfb_par *par, struct fb_info *info);
-extern u8 aty_ld_pll_ct(int offset, const struct atyfb_par *par);
void atyfb_copyarea(struct fb_info *info, const struct fb_copyarea *area);
void atyfb_fillrect(struct fb_info *info, const struct fb_fillrect *rect);
diff --git a/drivers/video/fbdev/aty/atyfb_base.c b/drivers/video/fbdev/aty/atyfb_base.c
index 81367cf..da748c3 100644
--- a/drivers/video/fbdev/aty/atyfb_base.c
+++ b/drivers/video/fbdev/aty/atyfb_base.c
@@ -3093,17 +3093,18 @@ static int atyfb_setup_sparc(struct pci_dev *pdev, struct fb_info *info,
/*
* PLL Reference Divider M:
*/
- M = pll_regs[2];
+ M = pll_regs[PLL_REF_DIV];
/*
* PLL Feedback Divider N (Dependent on CLOCK_CNTL):
*/
- N = pll_regs[7 + (clock_cntl & 3)];
+ N = pll_regs[VCLK0_FB_DIV + (clock_cntl & 3)];
/*
* PLL Post Divider P (Dependent on CLOCK_CNTL):
*/
- P = 1 << (pll_regs[6] >> ((clock_cntl & 3) << 1));
+ P = aty_postdividers[((pll_regs[VCLK_POST_DIV] >> ((clock_cntl & 3) << 1)) & 3) |
+ ((pll_regs[PLL_EXT_CNTL] >> (2 + (clock_cntl & 3))) & 4)];
/*
* PLL Divider Q:
diff --git a/drivers/video/fbdev/aty/mach64_accel.c b/drivers/video/fbdev/aty/mach64_accel.c
index 182bd680..e9dfe0e 100644
--- a/drivers/video/fbdev/aty/mach64_accel.c
+++ b/drivers/video/fbdev/aty/mach64_accel.c
@@ -126,7 +126,7 @@ void aty_init_engine(struct atyfb_par *par, struct fb_info *info)
/* set host attributes */
wait_for_fifo(13, par);
- aty_st_le32(HOST_CNTL, 0, par);
+ aty_st_le32(HOST_CNTL, HOST_BYTE_ALIGN, par);
/* set pattern attributes */
aty_st_le32(PAT_REG0, 0, par);
@@ -232,7 +232,8 @@ void atyfb_copyarea(struct fb_info *info, const struct fb_copyarea *area)
rotation = rotation24bpp(dx, direction);
}
- wait_for_fifo(4, par);
+ wait_for_fifo(5, par);
+ aty_st_le32(DP_PIX_WIDTH, par->crtc.dp_pix_width, par);
aty_st_le32(DP_SRC, FRGD_SRC_BLIT, par);
aty_st_le32(SRC_Y_X, (sx << 16) | sy, par);
aty_st_le32(SRC_HEIGHT1_WIDTH1, (width << 16) | area->height, par);
@@ -268,7 +269,8 @@ void atyfb_fillrect(struct fb_info *info, const struct fb_fillrect *rect)
rotation = rotation24bpp(dx, DST_X_LEFT_TO_RIGHT);
}
- wait_for_fifo(3, par);
+ wait_for_fifo(4, par);
+ aty_st_le32(DP_PIX_WIDTH, par->crtc.dp_pix_width, par);
aty_st_le32(DP_FRGD_CLR, color, par);
aty_st_le32(DP_SRC,
BKGD_SRC_BKGD_CLR | FRGD_SRC_FRGD_CLR | MONO_SRC_ONE,
@@ -283,7 +285,7 @@ void atyfb_imageblit(struct fb_info *info, const struct fb_image *image)
{
struct atyfb_par *par = (struct atyfb_par *) info->par;
u32 src_bytes, dx = image->dx, dy = image->dy, width = image->width;
- u32 pix_width_save, pix_width, host_cntl, rotation = 0, src, mix;
+ u32 pix_width, rotation = 0, src, mix;
if (par->asleep)
return;
@@ -295,8 +297,7 @@ void atyfb_imageblit(struct fb_info *info, const struct fb_image *image)
return;
}
- pix_width = pix_width_save = aty_ld_le32(DP_PIX_WIDTH, par);
- host_cntl = aty_ld_le32(HOST_CNTL, par) | HOST_BYTE_ALIGN;
+ pix_width = par->crtc.dp_pix_width;
switch (image->depth) {
case 1:
@@ -344,7 +345,7 @@ void atyfb_imageblit(struct fb_info *info, const struct fb_image *image)
* since Rage 3D IIc we have DP_HOST_TRIPLE_EN bit
* this hwaccelerated triple has an issue with not aligned data
*/
- if (M64_HAS(HW_TRIPLE) && image->width % 8 == 0)
+ if (image->depth == 1 && M64_HAS(HW_TRIPLE) && image->width % 8 == 0)
pix_width |= DP_HOST_TRIPLE_EN;
}
@@ -369,19 +370,18 @@ void atyfb_imageblit(struct fb_info *info, const struct fb_image *image)
mix = FRGD_MIX_D_XOR_S | BKGD_MIX_D;
}
- wait_for_fifo(6, par);
- aty_st_le32(DP_WRITE_MASK, 0xFFFFFFFF, par);
+ wait_for_fifo(5, par);
aty_st_le32(DP_PIX_WIDTH, pix_width, par);
aty_st_le32(DP_MIX, mix, par);
aty_st_le32(DP_SRC, src, par);
- aty_st_le32(HOST_CNTL, host_cntl, par);
+ aty_st_le32(HOST_CNTL, HOST_BYTE_ALIGN, par);
aty_st_le32(DST_CNTL, DST_Y_TOP_TO_BOTTOM | DST_X_LEFT_TO_RIGHT | rotation, par);
draw_rect(dx, dy, width, image->height, par);
src_bytes = (((image->width * image->depth) + 7) / 8) * image->height;
/* manual triple each pixel */
- if (info->var.bits_per_pixel == 24 && !(pix_width & DP_HOST_TRIPLE_EN)) {
+ if (image->depth == 1 && info->var.bits_per_pixel == 24 && !(pix_width & DP_HOST_TRIPLE_EN)) {
int inbit, outbit, mult24, byte_id_in_dword, width;
u8 *pbitmapin = (u8*)image->data, *pbitmapout;
u32 hostdword;
@@ -414,7 +414,7 @@ void atyfb_imageblit(struct fb_info *info, const struct fb_image *image)
}
}
wait_for_fifo(1, par);
- aty_st_le32(HOST_DATA0, hostdword, par);
+ aty_st_le32(HOST_DATA0, le32_to_cpu(hostdword), par);
}
} else {
u32 *pbitmap, dwords = (src_bytes + 3) / 4;
@@ -423,8 +423,4 @@ void atyfb_imageblit(struct fb_info *info, const struct fb_image *image)
aty_st_le32(HOST_DATA0, get_unaligned_le32(pbitmap), par);
}
}
-
- /* restore pix_width */
- wait_for_fifo(1, par);
- aty_st_le32(DP_PIX_WIDTH, pix_width_save, par);
}
diff --git a/drivers/video/fbdev/aty/mach64_ct.c b/drivers/video/fbdev/aty/mach64_ct.c
index 51f29d6..af54256 100644
--- a/drivers/video/fbdev/aty/mach64_ct.c
+++ b/drivers/video/fbdev/aty/mach64_ct.c
@@ -114,7 +114,7 @@ static void aty_st_pll_ct(int offset, u8 val, const struct atyfb_par *par)
*/
#define Maximum_DSP_PRECISION 7
-static u8 postdividers[] = {1,2,4,8,3};
+const u8 aty_postdividers[8] = {1,2,4,8,3,5,6,12};
static int aty_dsp_gt(const struct fb_info *info, u32 bpp, struct pll_ct *pll)
{
@@ -221,7 +221,7 @@ static int aty_valid_pll_ct(const struct fb_info *info, u32 vclk_per, struct pll
pll->vclk_post_div += (q < 64*8);
pll->vclk_post_div += (q < 32*8);
}
- pll->vclk_post_div_real = postdividers[pll->vclk_post_div];
+ pll->vclk_post_div_real = aty_postdividers[pll->vclk_post_div];
// pll->vclk_post_div <<= 6;
pll->vclk_fb_div = q * pll->vclk_post_div_real / 8;
pllvclk = (1000000 * 2 * pll->vclk_fb_div) /
@@ -512,7 +512,7 @@ static int aty_init_pll_ct(const struct fb_info *info, union aty_pll *pll)
u8 mclk_fb_div, pll_ext_cntl;
pll->ct.pll_ref_div = aty_ld_pll_ct(PLL_REF_DIV, par);
pll_ext_cntl = aty_ld_pll_ct(PLL_EXT_CNTL, par);
- pll->ct.xclk_post_div_real = postdividers[pll_ext_cntl & 0x07];
+ pll->ct.xclk_post_div_real = aty_postdividers[pll_ext_cntl & 0x07];
mclk_fb_div = aty_ld_pll_ct(MCLK_FB_DIV, par);
if (pll_ext_cntl & PLL_MFB_TIMES_4_2B)
mclk_fb_div <<= 1;
@@ -534,7 +534,7 @@ static int aty_init_pll_ct(const struct fb_info *info, union aty_pll *pll)
xpost_div += (q < 64*8);
xpost_div += (q < 32*8);
}
- pll->ct.xclk_post_div_real = postdividers[xpost_div];
+ pll->ct.xclk_post_div_real = aty_postdividers[xpost_div];
pll->ct.mclk_fb_div = q * pll->ct.xclk_post_div_real / 8;
#ifdef CONFIG_PPC
@@ -583,7 +583,7 @@ static int aty_init_pll_ct(const struct fb_info *info, union aty_pll *pll)
mpost_div += (q < 64*8);
mpost_div += (q < 32*8);
}
- sclk_post_div_real = postdividers[mpost_div];
+ sclk_post_div_real = aty_postdividers[mpost_div];
pll->ct.sclk_fb_div = q * sclk_post_div_real / 8;
pll->ct.spll_cntl2 = mpost_div << 4;
#ifdef DEBUG
diff --git a/drivers/video/fbdev/core/fbmem.c b/drivers/video/fbdev/core/fbmem.c
index 2ef33d4..87bc1e9 100644
--- a/drivers/video/fbdev/core/fbmem.c
+++ b/drivers/video/fbdev/core/fbmem.c
@@ -1706,12 +1706,12 @@ static int do_register_framebuffer(struct fb_info *fb_info)
return 0;
}
-static int do_unregister_framebuffer(struct fb_info *fb_info)
+static int unbind_console(struct fb_info *fb_info)
{
struct fb_event event;
- int i, ret = 0;
+ int ret;
+ int i = fb_info->node;
- i = fb_info->node;
if (i < 0 || i >= FB_MAX || registered_fb[i] != fb_info)
return -EINVAL;
@@ -1726,17 +1726,29 @@ static int do_unregister_framebuffer(struct fb_info *fb_info)
unlock_fb_info(fb_info);
console_unlock();
+ return ret;
+}
+
+static int __unlink_framebuffer(struct fb_info *fb_info);
+
+static int do_unregister_framebuffer(struct fb_info *fb_info)
+{
+ struct fb_event event;
+ int ret;
+
+ ret = unbind_console(fb_info);
+
if (ret)
return -EINVAL;
pm_vt_switch_unregister(fb_info->dev);
- unlink_framebuffer(fb_info);
+ __unlink_framebuffer(fb_info);
if (fb_info->pixmap.addr &&
(fb_info->pixmap.flags & FB_PIXMAP_DEFAULT))
kfree(fb_info->pixmap.addr);
fb_destroy_modelist(&fb_info->modelist);
- registered_fb[i] = NULL;
+ registered_fb[fb_info->node] = NULL;
num_registered_fb--;
fb_cleanup_device(fb_info);
event.info = fb_info;
@@ -1749,7 +1761,7 @@ static int do_unregister_framebuffer(struct fb_info *fb_info)
return 0;
}
-int unlink_framebuffer(struct fb_info *fb_info)
+static int __unlink_framebuffer(struct fb_info *fb_info)
{
int i;
@@ -1761,6 +1773,20 @@ int unlink_framebuffer(struct fb_info *fb_info)
device_destroy(fb_class, MKDEV(FB_MAJOR, i));
fb_info->dev = NULL;
}
+
+ return 0;
+}
+
+int unlink_framebuffer(struct fb_info *fb_info)
+{
+ int ret;
+
+ ret = __unlink_framebuffer(fb_info);
+ if (ret)
+ return ret;
+
+ unbind_console(fb_info);
+
return 0;
}
EXPORT_SYMBOL(unlink_framebuffer);
diff --git a/drivers/video/fbdev/core/modedb.c b/drivers/video/fbdev/core/modedb.c
index 2510fa72..de119f1 100644
--- a/drivers/video/fbdev/core/modedb.c
+++ b/drivers/video/fbdev/core/modedb.c
@@ -644,7 +644,7 @@ static int fb_try_mode(struct fb_var_screeninfo *var, struct fb_info *info,
*
* Valid mode specifiers for @mode_option:
*
- * <xres>x<yres>[M][R][-<bpp>][@<refresh>][i][m] or
+ * <xres>x<yres>[M][R][-<bpp>][@<refresh>][i][p][m] or
* <name>[-<bpp>][@<refresh>]
*
* with <xres>, <yres>, <bpp> and <refresh> decimal numbers and
@@ -653,10 +653,10 @@ static int fb_try_mode(struct fb_var_screeninfo *var, struct fb_info *info,
* If 'M' is present after yres (and before refresh/bpp if present),
* the function will compute the timings using VESA(tm) Coordinated
* Video Timings (CVT). If 'R' is present after 'M', will compute with
- * reduced blanking (for flatpanels). If 'i' is present, compute
- * interlaced mode. If 'm' is present, add margins equal to 1.8%
- * of xres rounded down to 8 pixels, and 1.8% of yres. The char
- * 'i' and 'm' must be after 'M' and 'R'. Example:
+ * reduced blanking (for flatpanels). If 'i' or 'p' are present, compute
+ * interlaced or progressive mode. If 'm' is present, add margins equal
+ * to 1.8% of xres rounded down to 8 pixels, and 1.8% of yres. The chars
+ * 'i', 'p' and 'm' must be after 'M' and 'R'. Example:
*
* 1024x768MR-8@60m - Reduced blank with margins at 60Hz.
*
@@ -697,7 +697,8 @@ int fb_find_mode(struct fb_var_screeninfo *var,
unsigned int namelen = strlen(name);
int res_specified = 0, bpp_specified = 0, refresh_specified = 0;
unsigned int xres = 0, yres = 0, bpp = default_bpp, refresh = 0;
- int yres_specified = 0, cvt = 0, rb = 0, interlace = 0;
+ int yres_specified = 0, cvt = 0, rb = 0;
+ int interlace_specified = 0, interlace = 0;
int margins = 0;
u32 best, diff, tdiff;
@@ -748,9 +749,17 @@ int fb_find_mode(struct fb_var_screeninfo *var,
if (!cvt)
margins = 1;
break;
+ case 'p':
+ if (!cvt) {
+ interlace = 0;
+ interlace_specified = 1;
+ }
+ break;
case 'i':
- if (!cvt)
+ if (!cvt) {
interlace = 1;
+ interlace_specified = 1;
+ }
break;
default:
goto done;
@@ -819,11 +828,21 @@ int fb_find_mode(struct fb_var_screeninfo *var,
if ((name_matches(db[i], name, namelen) ||
(res_specified && res_matches(db[i], xres, yres))) &&
!fb_try_mode(var, info, &db[i], bpp)) {
- if (refresh_specified && db[i].refresh == refresh)
- return 1;
+ const int db_interlace = (db[i].vmode &
+ FB_VMODE_INTERLACED ? 1 : 0);
+ int score = abs(db[i].refresh - refresh);
- if (abs(db[i].refresh - refresh) < diff) {
- diff = abs(db[i].refresh - refresh);
+ if (interlace_specified)
+ score += abs(db_interlace - interlace);
+
+ if (!interlace_specified ||
+ db_interlace == interlace)
+ if (refresh_specified &&
+ db[i].refresh == refresh)
+ return 1;
+
+ if (score < diff) {
+ diff = score;
best = i;
}
}
diff --git a/drivers/video/fbdev/efifb.c b/drivers/video/fbdev/efifb.c
index 6f2e729..f4b6d06 100644
--- a/drivers/video/fbdev/efifb.c
+++ b/drivers/video/fbdev/efifb.c
@@ -375,7 +375,7 @@ static void efifb_fixup_resources(struct pci_dev *dev)
if (!base)
return;
- for (i = 0; i < PCI_STD_RESOURCE_END; i++) {
+ for (i = 0; i <= PCI_STD_RESOURCE_END; i++) {
struct resource *res = &dev->resource[i];
if (!(res->flags & IORESOURCE_MEM))
diff --git a/drivers/video/fbdev/goldfishfb.c b/drivers/video/fbdev/goldfishfb.c
index 1e56b50..8c93ad1d 100644
--- a/drivers/video/fbdev/goldfishfb.c
+++ b/drivers/video/fbdev/goldfishfb.c
@@ -302,6 +302,7 @@ static int goldfish_fb_remove(struct platform_device *pdev)
dma_free_coherent(&pdev->dev, framesize, (void *)fb->fb.screen_base,
fb->fb.fix.smem_start);
iounmap(fb->reg_base);
+ kfree(fb);
return 0;
}
diff --git a/drivers/video/fbdev/omap/omapfb_main.c b/drivers/video/fbdev/omap/omapfb_main.c
index 6429f33..77c97c6 100644
--- a/drivers/video/fbdev/omap/omapfb_main.c
+++ b/drivers/video/fbdev/omap/omapfb_main.c
@@ -956,7 +956,7 @@ int omapfb_register_client(struct omapfb_notifier_block *omapfb_nb,
{
int r;
- if ((unsigned)omapfb_nb->plane_idx > OMAPFB_PLANE_NUM)
+ if ((unsigned)omapfb_nb->plane_idx >= OMAPFB_PLANE_NUM)
return -EINVAL;
if (!notifier_inited) {
diff --git a/drivers/video/fbdev/omap2/omapfb/omapfb-ioctl.c b/drivers/video/fbdev/omap2/omapfb/omapfb-ioctl.c
index ef69273..a3edb20 100644
--- a/drivers/video/fbdev/omap2/omapfb/omapfb-ioctl.c
+++ b/drivers/video/fbdev/omap2/omapfb/omapfb-ioctl.c
@@ -496,6 +496,9 @@ static int omapfb_memory_read(struct fb_info *fbi,
if (!access_ok(VERIFY_WRITE, mr->buffer, mr->buffer_size))
return -EFAULT;
+ if (mr->w > 4096 || mr->h > 4096)
+ return -EINVAL;
+
if (mr->w * mr->h * 3 > mr->buffer_size)
return -EINVAL;
@@ -509,7 +512,7 @@ static int omapfb_memory_read(struct fb_info *fbi,
mr->x, mr->y, mr->w, mr->h);
if (r > 0) {
- if (copy_to_user(mr->buffer, buf, mr->buffer_size))
+ if (copy_to_user(mr->buffer, buf, r))
r = -EFAULT;
}
diff --git a/drivers/video/fbdev/pxa168fb.c b/drivers/video/fbdev/pxa168fb.c
index def3a50..d059d04 100644
--- a/drivers/video/fbdev/pxa168fb.c
+++ b/drivers/video/fbdev/pxa168fb.c
@@ -712,7 +712,7 @@ static int pxa168fb_probe(struct platform_device *pdev)
/*
* enable controller clock
*/
- clk_enable(fbi->clk);
+ clk_prepare_enable(fbi->clk);
pxa168fb_set_par(info);
@@ -767,7 +767,7 @@ static int pxa168fb_probe(struct platform_device *pdev)
failed_free_cmap:
fb_dealloc_cmap(&info->cmap);
failed_free_clk:
- clk_disable(fbi->clk);
+ clk_disable_unprepare(fbi->clk);
failed_free_fbmem:
dma_free_coherent(fbi->dev, info->fix.smem_len,
info->screen_base, fbi->fb_start_dma);
@@ -807,7 +807,7 @@ static int pxa168fb_remove(struct platform_device *pdev)
dma_free_wc(fbi->dev, PAGE_ALIGN(info->fix.smem_len),
info->screen_base, info->fix.smem_start);
- clk_disable(fbi->clk);
+ clk_disable_unprepare(fbi->clk);
framebuffer_release(info);
diff --git a/drivers/video/fbdev/pxafb.c b/drivers/video/fbdev/pxafb.c
index ef73f14..8503310 100644
--- a/drivers/video/fbdev/pxafb.c
+++ b/drivers/video/fbdev/pxafb.c
@@ -2128,8 +2128,8 @@ static int of_get_pxafb_display(struct device *dev, struct device_node *disp,
return -EINVAL;
ret = -ENOMEM;
- info->modes = kmalloc_array(timings->num_timings,
- sizeof(info->modes[0]), GFP_KERNEL);
+ info->modes = kcalloc(timings->num_timings, sizeof(info->modes[0]),
+ GFP_KERNEL);
if (!info->modes)
goto out;
info->num_modes = timings->num_timings;
diff --git a/drivers/video/fbdev/sbuslib.c b/drivers/video/fbdev/sbuslib.c
index a350209..31c301d 100644
--- a/drivers/video/fbdev/sbuslib.c
+++ b/drivers/video/fbdev/sbuslib.c
@@ -121,7 +121,7 @@ int sbusfb_ioctl_helper(unsigned long cmd, unsigned long arg,
unsigned char __user *ured;
unsigned char __user *ugreen;
unsigned char __user *ublue;
- int index, count, i;
+ unsigned int index, count, i;
if (get_user(index, &c->index) ||
__get_user(count, &c->count) ||
@@ -160,7 +160,7 @@ int sbusfb_ioctl_helper(unsigned long cmd, unsigned long arg,
unsigned char __user *ugreen;
unsigned char __user *ublue;
struct fb_cmap *cmap = &info->cmap;
- int index, count, i;
+ unsigned int index, count, i;
u8 red, green, blue;
if (get_user(index, &c->index) ||
diff --git a/drivers/video/fbdev/uvesafb.c b/drivers/video/fbdev/uvesafb.c
index 98af9e0..9fe0d0b 100644
--- a/drivers/video/fbdev/uvesafb.c
+++ b/drivers/video/fbdev/uvesafb.c
@@ -1059,7 +1059,8 @@ static int uvesafb_setcmap(struct fb_cmap *cmap, struct fb_info *info)
info->cmap.len || cmap->start < info->cmap.start)
return -EINVAL;
- entries = kmalloc(sizeof(*entries) * cmap->len, GFP_KERNEL);
+ entries = kmalloc_array(cmap->len, sizeof(*entries),
+ GFP_KERNEL);
if (!entries)
return -ENOMEM;
diff --git a/drivers/video/fbdev/via/viafbdev.c b/drivers/video/fbdev/via/viafbdev.c
index badee04..71b5dca 100644
--- a/drivers/video/fbdev/via/viafbdev.c
+++ b/drivers/video/fbdev/via/viafbdev.c
@@ -19,6 +19,7 @@
* 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
*/
+#include <linux/compiler.h>
#include <linux/module.h>
#include <linux/seq_file.h>
#include <linux/slab.h>
@@ -1468,7 +1469,7 @@ static const struct file_operations viafb_vt1636_proc_fops = {
#endif /* CONFIG_FB_VIA_DIRECT_PROCFS */
-static int viafb_sup_odev_proc_show(struct seq_file *m, void *v)
+static int __maybe_unused viafb_sup_odev_proc_show(struct seq_file *m, void *v)
{
via_odev_to_seq(m, supported_odev_map[
viaparinfo->shared->chip_info.gfx_chip_name]);
diff --git a/drivers/virtio/virtio_balloon.c b/drivers/virtio/virtio_balloon.c
index a7c08cc..30076956 100644
--- a/drivers/virtio/virtio_balloon.c
+++ b/drivers/virtio/virtio_balloon.c
@@ -493,7 +493,9 @@ static int virtballoon_migratepage(struct balloon_dev_info *vb_dev_info,
tell_host(vb, vb->inflate_vq);
/* balloon's page migration 2nd step -- deflate "page" */
+ spin_lock_irqsave(&vb_dev_info->pages_lock, flags);
balloon_page_delete(page);
+ spin_unlock_irqrestore(&vb_dev_info->pages_lock, flags);
vb->num_pfns = VIRTIO_BALLOON_PAGES_PER_PAGE;
set_page_pfns(vb, vb->pfns, page);
tell_host(vb, vb->deflate_vq);
diff --git a/drivers/virtio/virtio_pci_legacy.c b/drivers/virtio/virtio_pci_legacy.c
index 6d9e517..fbc4761 100644
--- a/drivers/virtio/virtio_pci_legacy.c
+++ b/drivers/virtio/virtio_pci_legacy.c
@@ -121,6 +121,7 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev,
struct virtqueue *vq;
u16 num;
int err;
+ u64 q_pfn;
/* Select the queue we're interested in */
iowrite16(index, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_SEL);
@@ -139,9 +140,17 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev,
if (!vq)
return ERR_PTR(-ENOMEM);
+ q_pfn = virtqueue_get_desc_addr(vq) >> VIRTIO_PCI_QUEUE_ADDR_SHIFT;
+ if (q_pfn >> 32) {
+ dev_err(&vp_dev->pci_dev->dev,
+ "platform bug: legacy virtio-mmio must not be used with RAM above 0x%llxGB\n",
+ 0x1ULL << (32 + PAGE_SHIFT - 30));
+ err = -E2BIG;
+ goto out_del_vq;
+ }
+
/* activate the queue */
- iowrite32(virtqueue_get_desc_addr(vq) >> VIRTIO_PCI_QUEUE_ADDR_SHIFT,
- vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);
+ iowrite32(q_pfn, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);
vq->priv = (void __force *)vp_dev->ioaddr + VIRTIO_PCI_QUEUE_NOTIFY;
@@ -158,6 +167,7 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev,
out_deactivate:
iowrite32(0, vp_dev->ioaddr + VIRTIO_PCI_QUEUE_PFN);
+out_del_vq:
vring_del_virtqueue(vq);
return ERR_PTR(err);
}
diff --git a/drivers/w1/masters/mxc_w1.c b/drivers/w1/masters/mxc_w1.c
index a462175..dacb591 100644
--- a/drivers/w1/masters/mxc_w1.c
+++ b/drivers/w1/masters/mxc_w1.c
@@ -113,6 +113,10 @@ static int mxc_w1_probe(struct platform_device *pdev)
if (IS_ERR(mdev->clk))
return PTR_ERR(mdev->clk);
+ err = clk_prepare_enable(mdev->clk);
+ if (err)
+ return err;
+
clkrate = clk_get_rate(mdev->clk);
if (clkrate < 10000000)
dev_warn(&pdev->dev,
@@ -126,12 +130,10 @@ static int mxc_w1_probe(struct platform_device *pdev)
res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
mdev->regs = devm_ioremap_resource(&pdev->dev, res);
- if (IS_ERR(mdev->regs))
- return PTR_ERR(mdev->regs);
-
- err = clk_prepare_enable(mdev->clk);
- if (err)
- return err;
+ if (IS_ERR(mdev->regs)) {
+ err = PTR_ERR(mdev->regs);
+ goto out_disable_clk;
+ }
/* Software reset 1-Wire module */
writeb(MXC_W1_RESET_RST, mdev->regs + MXC_W1_RESET);
@@ -147,8 +149,12 @@ static int mxc_w1_probe(struct platform_device *pdev)
err = w1_add_master_device(&mdev->bus_master);
if (err)
- clk_disable_unprepare(mdev->clk);
+ goto out_disable_clk;
+ return 0;
+
+out_disable_clk:
+ clk_disable_unprepare(mdev->clk);
return err;
}
diff --git a/drivers/w1/masters/omap_hdq.c b/drivers/w1/masters/omap_hdq.c
index bb09de6..86637fe 100644
--- a/drivers/w1/masters/omap_hdq.c
+++ b/drivers/w1/masters/omap_hdq.c
@@ -784,6 +784,8 @@ static int omap_hdq_remove(struct platform_device *pdev)
/* remove module dependency */
pm_runtime_disable(&pdev->dev);
+ w1_remove_master_device(&omap_w1_master);
+
return 0;
}
diff --git a/drivers/w1/w1.c b/drivers/w1/w1.c
index ab0931e..aa458f2 100644
--- a/drivers/w1/w1.c
+++ b/drivers/w1/w1.c
@@ -741,7 +741,7 @@ int w1_attach_slave_device(struct w1_master *dev, struct w1_reg_num *rn)
/* slave modules need to be loaded in a context with unlocked mutex */
mutex_unlock(&dev->mutex);
- request_module("w1-family-0x%02x", rn->family);
+ request_module("w1-family-0x%02X", rn->family);
mutex_lock(&dev->mutex);
spin_lock(&w1_flock);
diff --git a/drivers/watchdog/f71808e_wdt.c b/drivers/watchdog/f71808e_wdt.c
index e682bf0..88cd2a5 100644
--- a/drivers/watchdog/f71808e_wdt.c
+++ b/drivers/watchdog/f71808e_wdt.c
@@ -566,7 +566,8 @@ static ssize_t watchdog_write(struct file *file, const char __user *buf,
char c;
if (get_user(c, buf + i))
return -EFAULT;
- expect_close = (c == 'V');
+ if (c == 'V')
+ expect_close = true;
}
/* Properly order writes across fork()ed processes */
diff --git a/drivers/watchdog/sbsa_gwdt.c b/drivers/watchdog/sbsa_gwdt.c
index ce0c38b..37523f1 100644
--- a/drivers/watchdog/sbsa_gwdt.c
+++ b/drivers/watchdog/sbsa_gwdt.c
@@ -50,6 +50,7 @@
*/
#include <linux/io.h>
+#include <linux/io-64-nonatomic-lo-hi.h>
#include <linux/interrupt.h>
#include <linux/module.h>
#include <linux/moduleparam.h>
@@ -159,7 +160,7 @@ static unsigned int sbsa_gwdt_get_timeleft(struct watchdog_device *wdd)
!(readl(gwdt->control_base + SBSA_GWDT_WCS) & SBSA_GWDT_WCS_WS0))
timeleft += readl(gwdt->control_base + SBSA_GWDT_WOR);
- timeleft += readq(gwdt->control_base + SBSA_GWDT_WCV) -
+ timeleft += lo_hi_readq(gwdt->control_base + SBSA_GWDT_WCV) -
arch_counter_get_cntvct();
do_div(timeleft, gwdt->clk);
diff --git a/drivers/watchdog/sp5100_tco.h b/drivers/watchdog/sp5100_tco.h
index 2b28c00..dfe20b8 100644
--- a/drivers/watchdog/sp5100_tco.h
+++ b/drivers/watchdog/sp5100_tco.h
@@ -54,7 +54,7 @@
#define SB800_PM_WATCHDOG_CONFIG 0x4C
#define SB800_PCI_WATCHDOG_DECODE_EN (1 << 0)
-#define SB800_PM_WATCHDOG_DISABLE (1 << 2)
+#define SB800_PM_WATCHDOG_DISABLE (1 << 1)
#define SB800_PM_WATCHDOG_SECOND_RES (3 << 0)
#define SB800_ACPI_MMIO_DECODE_EN (1 << 0)
#define SB800_ACPI_MMIO_SEL (1 << 1)
diff --git a/drivers/xen/cpu_hotplug.c b/drivers/xen/cpu_hotplug.c
index 5676aef..f4e59c4 100644
--- a/drivers/xen/cpu_hotplug.c
+++ b/drivers/xen/cpu_hotplug.c
@@ -18,15 +18,16 @@ static void enable_hotplug_cpu(int cpu)
static void disable_hotplug_cpu(int cpu)
{
- if (cpu_online(cpu)) {
- lock_device_hotplug();
+ if (!cpu_is_hotpluggable(cpu))
+ return;
+ lock_device_hotplug();
+ if (cpu_online(cpu))
device_offline(get_cpu_device(cpu));
- unlock_device_hotplug();
- }
- if (cpu_present(cpu))
+ if (!cpu_online(cpu) && cpu_present(cpu)) {
xen_arch_unregister_cpu(cpu);
-
- set_cpu_present(cpu, false);
+ set_cpu_present(cpu, false);
+ }
+ unlock_device_hotplug();
}
static int vcpu_online(unsigned int cpu)
diff --git a/drivers/xen/events/events_base.c b/drivers/xen/events/events_base.c
index d5dbdb9..4b0cc9d 100644
--- a/drivers/xen/events/events_base.c
+++ b/drivers/xen/events/events_base.c
@@ -139,7 +139,7 @@ static int set_evtchn_to_irq(unsigned evtchn, unsigned irq)
clear_evtchn_to_irq_row(row);
}
- evtchn_to_irq[EVTCHN_ROW(evtchn)][EVTCHN_COL(evtchn)] = irq;
+ evtchn_to_irq[row][col] = irq;
return 0;
}
@@ -637,8 +637,6 @@ static void __unbind_from_irq(unsigned int irq)
xen_irq_info_cleanup(info);
}
- BUG_ON(info_for_irq(irq)->type == IRQT_UNBOUND);
-
xen_free_irq(irq);
}
@@ -764,8 +762,8 @@ int xen_bind_pirq_msi_to_irq(struct pci_dev *dev, struct msi_desc *msidesc,
mutex_unlock(&irq_mapping_update_lock);
return irq;
error_irq:
- for (; i >= 0; i--)
- __unbind_from_irq(irq + i);
+ while (nvec--)
+ __unbind_from_irq(irq + nvec);
mutex_unlock(&irq_mapping_update_lock);
return ret;
}
diff --git a/drivers/xen/grant-table.c b/drivers/xen/grant-table.c
index bb36b1e..775d419 100644
--- a/drivers/xen/grant-table.c
+++ b/drivers/xen/grant-table.c
@@ -327,7 +327,7 @@ static void gnttab_handle_deferred(unsigned long unused)
if (entry->page) {
pr_debug("freeing g.e. %#x (pfn %#lx)\n",
entry->ref, page_to_pfn(entry->page));
- __free_page(entry->page);
+ put_page(entry->page);
} else
pr_info("freeing g.e. %#x\n", entry->ref);
kfree(entry);
@@ -383,7 +383,7 @@ void gnttab_end_foreign_access(grant_ref_t ref, int readonly,
if (gnttab_end_foreign_access_ref(ref, readonly)) {
put_free_entry(ref);
if (page != 0)
- free_page(page);
+ put_page(virt_to_page(page));
} else
gnttab_add_deferred(ref, readonly,
page ? virt_to_page(page) : NULL);
diff --git a/drivers/xen/manage.c b/drivers/xen/manage.c
index 9122ba2..abd49bc 100644
--- a/drivers/xen/manage.c
+++ b/drivers/xen/manage.c
@@ -282,17 +282,26 @@ static void sysrq_handler(struct xenbus_watch *watch, const char **vec,
/*
* The Xenstore watch fires directly after registering it and
* after a suspend/resume cycle. So ENOENT is no error but
- * might happen in those cases.
+ * might happen in those cases. ERANGE is observed when we get
+ * an empty value (''), this happens when we acknowledge the
+ * request by writing '\0' below.
*/
- if (err != -ENOENT)
+ if (err != -ENOENT && err != -ERANGE)
pr_err("Error %d reading sysrq code in control/sysrq\n",
err);
xenbus_transaction_end(xbt, 1);
return;
}
- if (sysrq_key != '\0')
- xenbus_printf(xbt, "control", "sysrq", "%c", '\0');
+ if (sysrq_key != '\0') {
+ err = xenbus_printf(xbt, "control", "sysrq", "%c", '\0');
+ if (err) {
+ pr_err("%s: Error %d writing sysrq in control/sysrq\n",
+ __func__, err);
+ xenbus_transaction_end(xbt, 1);
+ return;
+ }
+ }
err = xenbus_transaction_end(xbt, 0);
if (err == -EAGAIN)
@@ -344,7 +353,12 @@ static int setup_shutdown_watcher(void)
continue;
snprintf(node, FEATURE_PATH_SIZE, "feature-%s",
shutdown_handlers[idx].command);
- xenbus_printf(XBT_NIL, "control", node, "%u", 1);
+ err = xenbus_printf(XBT_NIL, "control", node, "%u", 1);
+ if (err) {
+ pr_err("%s: Error %d writing %s\n", __func__,
+ err, node);
+ return err;
+ }
}
return 0;
diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index b68ced5..5d04b36 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -310,6 +310,9 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t size,
*/
flags &= ~(__GFP_DMA | __GFP_HIGHMEM);
+ /* Convert the size to actually allocated. */
+ size = 1UL << (order + XEN_PAGE_SHIFT);
+
/* On ARM this function returns an ioremap'ped virtual address for
* which virt_to_phys doesn't return the corresponding physical
* address. In fact on ARM virt_to_phys only works for kernel direct
@@ -359,7 +362,10 @@ xen_swiotlb_free_coherent(struct device *hwdev, size_t size, void *vaddr,
* physical address */
phys = xen_bus_to_phys(dev_addr);
- if (((dev_addr + size - 1 > dma_mask)) ||
+ /* Convert the size to actually allocated. */
+ size = 1UL << (order + XEN_PAGE_SHIFT);
+
+ if (((dev_addr + size - 1 <= dma_mask)) ||
range_straddles_page_boundary(phys, size))
xen_destroy_contiguous_region(phys, order);
diff --git a/drivers/xen/xen-acpi-processor.c b/drivers/xen/xen-acpi-processor.c
index 4b85746..7ff9d25 100644
--- a/drivers/xen/xen-acpi-processor.c
+++ b/drivers/xen/xen-acpi-processor.c
@@ -362,9 +362,9 @@ read_acpi_id(acpi_handle handle, u32 lvl, void *context, void **rv)
}
/* There are more ACPI Processor objects than in x2APIC or MADT.
* This can happen with incorrect ACPI SSDT declerations. */
- if (acpi_id > nr_acpi_bits) {
- pr_debug("We only have %u, trying to set %u\n",
- nr_acpi_bits, acpi_id);
+ if (acpi_id >= nr_acpi_bits) {
+ pr_debug("max acpi id %u, trying to set %u\n",
+ nr_acpi_bits - 1, acpi_id);
return AE_OK;
}
/* OK, There is a ACPI Processor object */
diff --git a/drivers/xen/xen-scsiback.c b/drivers/xen/xen-scsiback.c
index 980f328..992cb8f 100644
--- a/drivers/xen/xen-scsiback.c
+++ b/drivers/xen/xen-scsiback.c
@@ -1014,6 +1014,7 @@ static void scsiback_do_add_lun(struct vscsibk_info *info, const char *state,
{
struct v2p_entry *entry;
unsigned long flags;
+ int err;
if (try) {
spin_lock_irqsave(&info->v2p_lock, flags);
@@ -1029,8 +1030,11 @@ static void scsiback_do_add_lun(struct vscsibk_info *info, const char *state,
scsiback_del_translation_entry(info, vir);
}
} else if (!try) {
- xenbus_printf(XBT_NIL, info->dev->nodename, state,
+ err = xenbus_printf(XBT_NIL, info->dev->nodename, state,
"%d", XenbusStateClosed);
+ if (err)
+ xenbus_dev_error(info->dev, err,
+ "%s: writing %s", __func__, state);
}
}
@@ -1069,8 +1073,11 @@ static void scsiback_do_1lun_hotplug(struct vscsibk_info *info, int op,
snprintf(str, sizeof(str), "vscsi-devs/%s/p-dev", ent);
val = xenbus_read(XBT_NIL, dev->nodename, str, NULL);
if (IS_ERR(val)) {
- xenbus_printf(XBT_NIL, dev->nodename, state,
+ err = xenbus_printf(XBT_NIL, dev->nodename, state,
"%d", XenbusStateClosed);
+ if (err)
+ xenbus_dev_error(info->dev, err,
+ "%s: writing %s", __func__, state);
return;
}
strlcpy(phy, val, VSCSI_NAMELEN);
@@ -1081,8 +1088,11 @@ static void scsiback_do_1lun_hotplug(struct vscsibk_info *info, int op,
err = xenbus_scanf(XBT_NIL, dev->nodename, str, "%u:%u:%u:%u",
&vir.hst, &vir.chn, &vir.tgt, &vir.lun);
if (XENBUS_EXIST_ERR(err)) {
- xenbus_printf(XBT_NIL, dev->nodename, state,
+ err = xenbus_printf(XBT_NIL, dev->nodename, state,
"%d", XenbusStateClosed);
+ if (err)
+ xenbus_dev_error(info->dev, err,
+ "%s: writing %s", __func__, state);
return;
}
diff --git a/drivers/xen/xenbus/xenbus_probe.c b/drivers/xen/xenbus/xenbus_probe.c
index 33a31cf..c2d4476 100644
--- a/drivers/xen/xenbus/xenbus_probe.c
+++ b/drivers/xen/xenbus/xenbus_probe.c
@@ -470,8 +470,11 @@ int xenbus_probe_node(struct xen_bus_type *bus,
/* Register with generic device framework. */
err = device_register(&xendev->dev);
- if (err)
+ if (err) {
+ put_device(&xendev->dev);
+ xendev = NULL;
goto fail;
+ }
return 0;
fail:
diff --git a/drivers/xen/xlate_mmu.c b/drivers/xen/xlate_mmu.c
index 23f1387..e7df65d 100644
--- a/drivers/xen/xlate_mmu.c
+++ b/drivers/xen/xlate_mmu.c
@@ -36,6 +36,7 @@
#include <asm/xen/hypervisor.h>
#include <xen/xen.h>
+#include <xen/xen-ops.h>
#include <xen/page.h>
#include <xen/interface/xen.h>
#include <xen/interface/memory.h>
diff --git a/drivers/zorro/zorro.c b/drivers/zorro/zorro.c
index d295d98..8ec7938 100644
--- a/drivers/zorro/zorro.c
+++ b/drivers/zorro/zorro.c
@@ -16,6 +16,7 @@
#include <linux/bitops.h>
#include <linux/string.h>
#include <linux/platform_device.h>
+#include <linux/dma-mapping.h>
#include <linux/slab.h>
#include <asm/byteorder.h>
@@ -185,6 +186,17 @@ static int __init amiga_zorro_probe(struct platform_device *pdev)
z->dev.parent = &bus->dev;
z->dev.bus = &zorro_bus_type;
z->dev.id = i;
+ switch (z->rom.er_Type & ERT_TYPEMASK) {
+ case ERT_ZORROIII:
+ z->dev.coherent_dma_mask = DMA_BIT_MASK(32);
+ break;
+
+ case ERT_ZORROII:
+ default:
+ z->dev.coherent_dma_mask = DMA_BIT_MASK(24);
+ break;
+ }
+ z->dev.dma_mask = &z->dev.coherent_dma_mask;
}
/* ... then register them */
diff --git a/fs/9p/vfs_dir.c b/fs/9p/vfs_dir.c
index b0405d6..48db9a9 100644
--- a/fs/9p/vfs_dir.c
+++ b/fs/9p/vfs_dir.c
@@ -76,15 +76,6 @@ static inline int dt_type(struct p9_wstat *mistat)
return rettype;
}
-static void p9stat_init(struct p9_wstat *stbuf)
-{
- stbuf->name = NULL;
- stbuf->uid = NULL;
- stbuf->gid = NULL;
- stbuf->muid = NULL;
- stbuf->extension = NULL;
-}
-
/**
* v9fs_alloc_rdir_buf - Allocate buffer used for read and readdir
* @filp: opened file structure
@@ -145,12 +136,10 @@ static int v9fs_dir_readdir(struct file *file, struct dir_context *ctx)
rdir->tail = n;
}
while (rdir->head < rdir->tail) {
- p9stat_init(&st);
err = p9stat_read(fid->clnt, rdir->buf + rdir->head,
rdir->tail - rdir->head, &st);
if (err) {
p9_debug(P9_DEBUG_VFS, "returned %d\n", err);
- p9stat_free(&st);
return -EIO;
}
reclen = st.size+2;
diff --git a/fs/9p/vfs_file.c b/fs/9p/vfs_file.c
index d7b78d5..398a3ed 100644
--- a/fs/9p/vfs_file.c
+++ b/fs/9p/vfs_file.c
@@ -204,6 +204,14 @@ static int v9fs_file_do_lock(struct file *filp, int cmd, struct file_lock *fl)
break;
if (schedule_timeout_interruptible(P9_LOCK_TIMEOUT) != 0)
break;
+ /*
+ * p9_client_lock_dotl overwrites flock.client_id with the
+ * server message, free and reuse the client name
+ */
+ if (flock.client_id != fid->clnt->name) {
+ kfree(flock.client_id);
+ flock.client_id = fid->clnt->name;
+ }
}
/* map 9p status to VFS status */
@@ -235,6 +243,8 @@ static int v9fs_file_do_lock(struct file *filp, int cmd, struct file_lock *fl)
locks_lock_file_wait(filp, fl);
fl->fl_type = fl_type;
}
+ if (flock.client_id != fid->clnt->name)
+ kfree(flock.client_id);
out:
return res;
}
@@ -269,7 +279,7 @@ static int v9fs_file_getlock(struct file *filp, struct file_lock *fl)
res = p9_client_getlock_dotl(fid, &glock);
if (res < 0)
- return res;
+ goto out;
/* map 9p lock type to os lock type */
switch (glock.type) {
case P9_LOCK_TYPE_RDLCK:
@@ -290,7 +300,9 @@ static int v9fs_file_getlock(struct file *filp, struct file_lock *fl)
fl->fl_end = glock.start + glock.length - 1;
fl->fl_pid = glock.proc_id;
}
- kfree(glock.client_id);
+out:
+ if (glock.client_id != fid->clnt->name)
+ kfree(glock.client_id);
return res;
}
diff --git a/fs/9p/xattr.c b/fs/9p/xattr.c
index f329eee..352abc3 100644
--- a/fs/9p/xattr.c
+++ b/fs/9p/xattr.c
@@ -105,7 +105,7 @@ int v9fs_fid_xattr_set(struct p9_fid *fid, const char *name,
{
struct kvec kvec = {.iov_base = (void *)value, .iov_len = value_len};
struct iov_iter from;
- int retval;
+ int retval, err;
iov_iter_kvec(&from, WRITE | ITER_KVEC, &kvec, 1, value_len);
@@ -126,7 +126,9 @@ int v9fs_fid_xattr_set(struct p9_fid *fid, const char *name,
retval);
else
p9_client_write(fid, 0, &from, &retval);
- p9_client_clunk(fid);
+ err = p9_client_clunk(fid);
+ if (!retval && err)
+ retval = err;
return retval;
}
diff --git a/fs/Kconfig b/fs/Kconfig
index 20a8d95..90276c9 100644
--- a/fs/Kconfig
+++ b/fs/Kconfig
@@ -97,6 +97,8 @@
source "fs/crypto/Kconfig"
+source "fs/verity/Kconfig"
+
source "fs/notify/Kconfig"
source "fs/quota/Kconfig"
diff --git a/fs/Makefile b/fs/Makefile
index f207d43..27c4539 100644
--- a/fs/Makefile
+++ b/fs/Makefile
@@ -31,6 +31,7 @@
obj-$(CONFIG_AIO) += aio.o
obj-$(CONFIG_FS_DAX) += dax.o
obj-$(CONFIG_FS_ENCRYPTION) += crypto/
+obj-$(CONFIG_FS_VERITY) += verity/
obj-$(CONFIG_FILE_LOCKING) += locks.o
obj-$(CONFIG_COMPAT) += compat.o compat_ioctl.o
obj-$(CONFIG_BINFMT_AOUT) += binfmt_aout.o
diff --git a/fs/affs/namei.c b/fs/affs/namei.c
index 29186d2..2d4d495 100644
--- a/fs/affs/namei.c
+++ b/fs/affs/namei.c
@@ -224,9 +224,10 @@ affs_lookup(struct inode *dir, struct dentry *dentry, unsigned int flags)
affs_lock_dir(dir);
bh = affs_find_entry(dir, dentry);
- affs_unlock_dir(dir);
- if (IS_ERR(bh))
+ if (IS_ERR(bh)) {
+ affs_unlock_dir(dir);
return ERR_CAST(bh);
+ }
if (bh) {
u32 ino = bh->b_blocknr;
@@ -240,10 +241,13 @@ affs_lookup(struct inode *dir, struct dentry *dentry, unsigned int flags)
}
affs_brelse(bh);
inode = affs_iget(sb, ino);
- if (IS_ERR(inode))
+ if (IS_ERR(inode)) {
+ affs_unlock_dir(dir);
return ERR_CAST(inode);
+ }
}
d_add(dentry, inode);
+ affs_unlock_dir(dir);
return NULL;
}
diff --git a/fs/aio.c b/fs/aio.c
index 0606f03..c3fc802 100644
--- a/fs/aio.c
+++ b/fs/aio.c
@@ -40,6 +40,7 @@
#include <linux/ramfs.h>
#include <linux/percpu-refcount.h>
#include <linux/mount.h>
+#include <linux/nospec.h>
#include <asm/kmap_types.h>
#include <asm/uaccess.h>
@@ -636,9 +637,8 @@ static void free_ioctx_users(struct percpu_ref *ref)
while (!list_empty(&ctx->active_reqs)) {
req = list_first_entry(&ctx->active_reqs,
struct aio_kiocb, ki_list);
-
- list_del_init(&req->ki_list);
kiocb_cancel(req);
+ list_del_init(&req->ki_list);
}
spin_unlock_irq(&ctx->ctx_lock);
@@ -1072,10 +1072,11 @@ static struct kioctx *lookup_ioctx(unsigned long ctx_id)
if (!table || id >= table->nr)
goto out;
+ id = array_index_nospec(id, table->nr);
ctx = rcu_dereference(table->table[id]);
if (ctx && ctx->user_id == ctx_id) {
- percpu_ref_get(&ctx->users);
- ret = ctx;
+ if (percpu_ref_tryget_live(&ctx->users))
+ ret = ctx;
}
out:
rcu_read_unlock();
diff --git a/fs/autofs4/autofs_i.h b/fs/autofs4/autofs_i.h
index a1fba42..42f8633 100644
--- a/fs/autofs4/autofs_i.h
+++ b/fs/autofs4/autofs_i.h
@@ -14,6 +14,7 @@
#include <linux/mutex.h>
#include <linux/spinlock.h>
#include <linux/list.h>
+#include <linux/magic.h>
/* This is the range of ioctl() numbers we claim as ours */
#define AUTOFS_IOC_FIRST AUTOFS_IOC_READY
@@ -123,7 +124,8 @@ struct autofs_sb_info {
static inline struct autofs_sb_info *autofs4_sbi(struct super_block *sb)
{
- return (struct autofs_sb_info *)(sb->s_fs_info);
+ return sb->s_magic != AUTOFS_SUPER_MAGIC ?
+ NULL : (struct autofs_sb_info *)(sb->s_fs_info);
}
static inline struct autofs_info *autofs4_dentry_ino(struct dentry *dentry)
diff --git a/fs/autofs4/inode.c b/fs/autofs4/inode.c
index 438b5bf..ce0c6ea 100644
--- a/fs/autofs4/inode.c
+++ b/fs/autofs4/inode.c
@@ -14,7 +14,6 @@
#include <linux/pagemap.h>
#include <linux/parser.h>
#include <linux/bitops.h>
-#include <linux/magic.h>
#include "autofs_i.h"
#include <linux/module.h>
diff --git a/fs/bfs/inode.c b/fs/bfs/inode.c
index 1e5c896..0acb83e 100644
--- a/fs/bfs/inode.c
+++ b/fs/bfs/inode.c
@@ -350,7 +350,8 @@ static int bfs_fill_super(struct super_block *s, void *data, int silent)
s->s_magic = BFS_MAGIC;
- if (le32_to_cpu(bfs_sb->s_start) > le32_to_cpu(bfs_sb->s_end)) {
+ if (le32_to_cpu(bfs_sb->s_start) > le32_to_cpu(bfs_sb->s_end) ||
+ le32_to_cpu(bfs_sb->s_start) < BFS_BSIZE) {
printf("Superblock is corrupted\n");
goto out1;
}
@@ -359,9 +360,11 @@ static int bfs_fill_super(struct super_block *s, void *data, int silent)
sizeof(struct bfs_inode)
+ BFS_ROOT_INO - 1;
imap_len = (info->si_lasti / 8) + 1;
- info->si_imap = kzalloc(imap_len, GFP_KERNEL);
- if (!info->si_imap)
+ info->si_imap = kzalloc(imap_len, GFP_KERNEL | __GFP_NOWARN);
+ if (!info->si_imap) {
+ printf("Cannot allocate %u bytes\n", imap_len);
goto out1;
+ }
for (i = 0; i < BFS_ROOT_INO; i++)
set_bit(i, info->si_imap);
diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c
index 1fdf4e5..e7e25a8 100644
--- a/fs/binfmt_elf.c
+++ b/fs/binfmt_elf.c
@@ -1217,9 +1217,8 @@ static int load_elf_library(struct file *file)
goto out_free_ph;
}
- len = ELF_PAGESTART(eppnt->p_filesz + eppnt->p_vaddr +
- ELF_MIN_ALIGN - 1);
- bss = eppnt->p_memsz + eppnt->p_vaddr;
+ len = ELF_PAGEALIGN(eppnt->p_filesz + eppnt->p_vaddr);
+ bss = ELF_PAGEALIGN(eppnt->p_memsz + eppnt->p_vaddr);
if (bss > len) {
error = vm_brk(len, bss - len);
if (error)
@@ -1707,7 +1706,7 @@ static int fill_thread_core_info(struct elf_thread_core_info *t,
const struct user_regset *regset = &view->regsets[i];
do_thread_regset_writeback(t->task, regset);
if (regset->core_note_type && regset->get &&
- (!regset->active || regset->active(t->task, regset))) {
+ (!regset->active || regset->active(t->task, regset) > 0)) {
int ret;
size_t size = regset->n * regset->size;
void *data = kmalloc(size, GFP_KERNEL);
diff --git a/fs/binfmt_misc.c b/fs/binfmt_misc.c
index 9b4688a..f842261 100644
--- a/fs/binfmt_misc.c
+++ b/fs/binfmt_misc.c
@@ -384,8 +384,13 @@ static Node *create_entry(const char __user *buffer, size_t count)
s = strchr(p, del);
if (!s)
goto einval;
- *s++ = '\0';
- e->offset = simple_strtoul(p, &p, 10);
+ *s = '\0';
+ if (p != s) {
+ int r = kstrtoint(p, 10, &e->offset);
+ if (r != 0 || e->offset < 0)
+ goto einval;
+ }
+ p = s;
if (*p++)
goto einval;
pr_debug("register: offset: %#x\n", e->offset);
@@ -425,7 +430,8 @@ static Node *create_entry(const char __user *buffer, size_t count)
if (e->mask &&
string_unescape_inplace(e->mask, UNESCAPE_HEX) != e->size)
goto einval;
- if (e->size + e->offset > BINPRM_BUF_SIZE)
+ if (e->size > BINPRM_BUF_SIZE ||
+ BINPRM_BUF_SIZE - e->size < e->offset)
goto einval;
pr_debug("register: magic/mask length: %i\n", e->size);
if (USE_DEBUG) {
diff --git a/fs/btrfs/Makefile b/fs/btrfs/Makefile
index 128ce17..076ccfb 100644
--- a/fs/btrfs/Makefile
+++ b/fs/btrfs/Makefile
@@ -9,7 +9,7 @@
export.o tree-log.o free-space-cache.o zlib.o lzo.o \
compression.o delayed-ref.o relocation.o delayed-inode.o scrub.o \
reada.o backref.o ulist.o qgroup.o send.o dev-replace.o raid56.o \
- uuid-tree.o props.o hash.o free-space-tree.o
+ uuid-tree.o props.o hash.o free-space-tree.o tree-checker.o
btrfs-$(CONFIG_BTRFS_FS_POSIX_ACL) += acl.o
btrfs-$(CONFIG_BTRFS_FS_CHECK_INTEGRITY) += check-integrity.o
diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
index f6ba165..c94d339 100644
--- a/fs/btrfs/ctree.c
+++ b/fs/btrfs/ctree.c
@@ -2486,10 +2486,8 @@ read_block_for_search(struct btrfs_trans_handle *trans,
if (p->reada != READA_NONE)
reada_for_search(root, p, level, slot, key->objectid);
- btrfs_release_path(p);
-
ret = -EAGAIN;
- tmp = read_tree_block(root, blocknr, 0);
+ tmp = read_tree_block(root, blocknr, gen);
if (!IS_ERR(tmp)) {
/*
* If the read above didn't mark this buffer up to date,
@@ -2503,6 +2501,8 @@ read_block_for_search(struct btrfs_trans_handle *trans,
} else {
ret = PTR_ERR(tmp);
}
+
+ btrfs_release_path(p);
return ret;
}
@@ -2760,6 +2760,8 @@ int btrfs_search_slot(struct btrfs_trans_handle *trans, struct btrfs_root
* contention with the cow code
*/
if (cow) {
+ bool last_level = (level == (BTRFS_MAX_LEVEL - 1));
+
/*
* if we don't really need to cow this block
* then we don't want to set the path blocking,
@@ -2784,9 +2786,13 @@ int btrfs_search_slot(struct btrfs_trans_handle *trans, struct btrfs_root
}
btrfs_set_path_blocking(p);
- err = btrfs_cow_block(trans, root, b,
- p->nodes[level + 1],
- p->slots[level + 1], &b);
+ if (last_level)
+ err = btrfs_cow_block(trans, root, b, NULL, 0,
+ &b);
+ else
+ err = btrfs_cow_block(trans, root, b,
+ p->nodes[level + 1],
+ p->slots[level + 1], &b);
if (err) {
ret = err;
goto done;
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 86245b88..a423c36 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -1415,7 +1415,7 @@ do { \
#define BTRFS_INODE_ROOT_ITEM_INIT (1 << 31)
struct btrfs_map_token {
- struct extent_buffer *eb;
+ const struct extent_buffer *eb;
char *kaddr;
unsigned long offset;
};
@@ -1449,18 +1449,19 @@ static inline void btrfs_init_map_token (struct btrfs_map_token *token)
sizeof(((type *)0)->member)))
#define DECLARE_BTRFS_SETGET_BITS(bits) \
-u##bits btrfs_get_token_##bits(struct extent_buffer *eb, void *ptr, \
- unsigned long off, \
- struct btrfs_map_token *token); \
-void btrfs_set_token_##bits(struct extent_buffer *eb, void *ptr, \
+u##bits btrfs_get_token_##bits(const struct extent_buffer *eb, \
+ const void *ptr, unsigned long off, \
+ struct btrfs_map_token *token); \
+void btrfs_set_token_##bits(struct extent_buffer *eb, const void *ptr, \
unsigned long off, u##bits val, \
struct btrfs_map_token *token); \
-static inline u##bits btrfs_get_##bits(struct extent_buffer *eb, void *ptr, \
+static inline u##bits btrfs_get_##bits(const struct extent_buffer *eb, \
+ const void *ptr, \
unsigned long off) \
{ \
return btrfs_get_token_##bits(eb, ptr, off, NULL); \
} \
-static inline void btrfs_set_##bits(struct extent_buffer *eb, void *ptr, \
+static inline void btrfs_set_##bits(struct extent_buffer *eb, void *ptr,\
unsigned long off, u##bits val) \
{ \
btrfs_set_token_##bits(eb, ptr, off, val, NULL); \
@@ -1472,7 +1473,8 @@ DECLARE_BTRFS_SETGET_BITS(32)
DECLARE_BTRFS_SETGET_BITS(64)
#define BTRFS_SETGET_FUNCS(name, type, member, bits) \
-static inline u##bits btrfs_##name(struct extent_buffer *eb, type *s) \
+static inline u##bits btrfs_##name(const struct extent_buffer *eb, \
+ const type *s) \
{ \
BUILD_BUG_ON(sizeof(u##bits) != sizeof(((type *)0))->member); \
return btrfs_get_##bits(eb, s, offsetof(type, member)); \
@@ -1483,7 +1485,8 @@ static inline void btrfs_set_##name(struct extent_buffer *eb, type *s, \
BUILD_BUG_ON(sizeof(u##bits) != sizeof(((type *)0))->member); \
btrfs_set_##bits(eb, s, offsetof(type, member), val); \
} \
-static inline u##bits btrfs_token_##name(struct extent_buffer *eb, type *s, \
+static inline u##bits btrfs_token_##name(const struct extent_buffer *eb,\
+ const type *s, \
struct btrfs_map_token *token) \
{ \
BUILD_BUG_ON(sizeof(u##bits) != sizeof(((type *)0))->member); \
@@ -1498,9 +1501,9 @@ static inline void btrfs_set_token_##name(struct extent_buffer *eb, \
}
#define BTRFS_SETGET_HEADER_FUNCS(name, type, member, bits) \
-static inline u##bits btrfs_##name(struct extent_buffer *eb) \
+static inline u##bits btrfs_##name(const struct extent_buffer *eb) \
{ \
- type *p = page_address(eb->pages[0]); \
+ const type *p = page_address(eb->pages[0]); \
u##bits res = le##bits##_to_cpu(p->member); \
return res; \
} \
@@ -1512,7 +1515,7 @@ static inline void btrfs_set_##name(struct extent_buffer *eb, \
}
#define BTRFS_SETGET_STACK_FUNCS(name, type, member, bits) \
-static inline u##bits btrfs_##name(type *s) \
+static inline u##bits btrfs_##name(const type *s) \
{ \
return le##bits##_to_cpu(s->member); \
} \
@@ -1818,7 +1821,7 @@ static inline unsigned long btrfs_node_key_ptr_offset(int nr)
sizeof(struct btrfs_key_ptr) * nr;
}
-void btrfs_node_key(struct extent_buffer *eb,
+void btrfs_node_key(const struct extent_buffer *eb,
struct btrfs_disk_key *disk_key, int nr);
static inline void btrfs_set_node_key(struct extent_buffer *eb,
@@ -1847,28 +1850,28 @@ static inline struct btrfs_item *btrfs_item_nr(int nr)
return (struct btrfs_item *)btrfs_item_nr_offset(nr);
}
-static inline u32 btrfs_item_end(struct extent_buffer *eb,
+static inline u32 btrfs_item_end(const struct extent_buffer *eb,
struct btrfs_item *item)
{
return btrfs_item_offset(eb, item) + btrfs_item_size(eb, item);
}
-static inline u32 btrfs_item_end_nr(struct extent_buffer *eb, int nr)
+static inline u32 btrfs_item_end_nr(const struct extent_buffer *eb, int nr)
{
return btrfs_item_end(eb, btrfs_item_nr(nr));
}
-static inline u32 btrfs_item_offset_nr(struct extent_buffer *eb, int nr)
+static inline u32 btrfs_item_offset_nr(const struct extent_buffer *eb, int nr)
{
return btrfs_item_offset(eb, btrfs_item_nr(nr));
}
-static inline u32 btrfs_item_size_nr(struct extent_buffer *eb, int nr)
+static inline u32 btrfs_item_size_nr(const struct extent_buffer *eb, int nr)
{
return btrfs_item_size(eb, btrfs_item_nr(nr));
}
-static inline void btrfs_item_key(struct extent_buffer *eb,
+static inline void btrfs_item_key(const struct extent_buffer *eb,
struct btrfs_disk_key *disk_key, int nr)
{
struct btrfs_item *item = btrfs_item_nr(nr);
@@ -1904,8 +1907,8 @@ BTRFS_SETGET_STACK_FUNCS(stack_dir_name_len, struct btrfs_dir_item,
BTRFS_SETGET_STACK_FUNCS(stack_dir_transid, struct btrfs_dir_item,
transid, 64);
-static inline void btrfs_dir_item_key(struct extent_buffer *eb,
- struct btrfs_dir_item *item,
+static inline void btrfs_dir_item_key(const struct extent_buffer *eb,
+ const struct btrfs_dir_item *item,
struct btrfs_disk_key *key)
{
read_eb_member(eb, item, struct btrfs_dir_item, location, key);
@@ -1913,7 +1916,7 @@ static inline void btrfs_dir_item_key(struct extent_buffer *eb,
static inline void btrfs_set_dir_item_key(struct extent_buffer *eb,
struct btrfs_dir_item *item,
- struct btrfs_disk_key *key)
+ const struct btrfs_disk_key *key)
{
write_eb_member(eb, item, struct btrfs_dir_item, location, key);
}
@@ -1925,8 +1928,8 @@ BTRFS_SETGET_FUNCS(free_space_bitmaps, struct btrfs_free_space_header,
BTRFS_SETGET_FUNCS(free_space_generation, struct btrfs_free_space_header,
generation, 64);
-static inline void btrfs_free_space_key(struct extent_buffer *eb,
- struct btrfs_free_space_header *h,
+static inline void btrfs_free_space_key(const struct extent_buffer *eb,
+ const struct btrfs_free_space_header *h,
struct btrfs_disk_key *key)
{
read_eb_member(eb, h, struct btrfs_free_space_header, location, key);
@@ -1934,7 +1937,7 @@ static inline void btrfs_free_space_key(struct extent_buffer *eb,
static inline void btrfs_set_free_space_key(struct extent_buffer *eb,
struct btrfs_free_space_header *h,
- struct btrfs_disk_key *key)
+ const struct btrfs_disk_key *key)
{
write_eb_member(eb, h, struct btrfs_free_space_header, location, key);
}
@@ -1961,25 +1964,25 @@ static inline void btrfs_cpu_key_to_disk(struct btrfs_disk_key *disk,
disk->objectid = cpu_to_le64(cpu->objectid);
}
-static inline void btrfs_node_key_to_cpu(struct extent_buffer *eb,
- struct btrfs_key *key, int nr)
+static inline void btrfs_node_key_to_cpu(const struct extent_buffer *eb,
+ struct btrfs_key *key, int nr)
{
struct btrfs_disk_key disk_key;
btrfs_node_key(eb, &disk_key, nr);
btrfs_disk_key_to_cpu(key, &disk_key);
}
-static inline void btrfs_item_key_to_cpu(struct extent_buffer *eb,
- struct btrfs_key *key, int nr)
+static inline void btrfs_item_key_to_cpu(const struct extent_buffer *eb,
+ struct btrfs_key *key, int nr)
{
struct btrfs_disk_key disk_key;
btrfs_item_key(eb, &disk_key, nr);
btrfs_disk_key_to_cpu(key, &disk_key);
}
-static inline void btrfs_dir_item_key_to_cpu(struct extent_buffer *eb,
- struct btrfs_dir_item *item,
- struct btrfs_key *key)
+static inline void btrfs_dir_item_key_to_cpu(const struct extent_buffer *eb,
+ const struct btrfs_dir_item *item,
+ struct btrfs_key *key)
{
struct btrfs_disk_key disk_key;
btrfs_dir_item_key(eb, item, &disk_key);
@@ -2012,7 +2015,7 @@ BTRFS_SETGET_STACK_FUNCS(stack_header_nritems, struct btrfs_header,
nritems, 32);
BTRFS_SETGET_STACK_FUNCS(stack_header_bytenr, struct btrfs_header, bytenr, 64);
-static inline int btrfs_header_flag(struct extent_buffer *eb, u64 flag)
+static inline int btrfs_header_flag(const struct extent_buffer *eb, u64 flag)
{
return (btrfs_header_flags(eb) & flag) == flag;
}
@@ -2031,7 +2034,7 @@ static inline int btrfs_clear_header_flag(struct extent_buffer *eb, u64 flag)
return (flags & flag) == flag;
}
-static inline int btrfs_header_backref_rev(struct extent_buffer *eb)
+static inline int btrfs_header_backref_rev(const struct extent_buffer *eb)
{
u64 flags = btrfs_header_flags(eb);
return flags >> BTRFS_BACKREF_REV_SHIFT;
@@ -2051,12 +2054,12 @@ static inline unsigned long btrfs_header_fsid(void)
return offsetof(struct btrfs_header, fsid);
}
-static inline unsigned long btrfs_header_chunk_tree_uuid(struct extent_buffer *eb)
+static inline unsigned long btrfs_header_chunk_tree_uuid(const struct extent_buffer *eb)
{
return offsetof(struct btrfs_header, chunk_tree_uuid);
}
-static inline int btrfs_is_leaf(struct extent_buffer *eb)
+static inline int btrfs_is_leaf(const struct extent_buffer *eb)
{
return btrfs_header_level(eb) == 0;
}
@@ -2090,12 +2093,12 @@ BTRFS_SETGET_STACK_FUNCS(root_stransid, struct btrfs_root_item,
BTRFS_SETGET_STACK_FUNCS(root_rtransid, struct btrfs_root_item,
rtransid, 64);
-static inline bool btrfs_root_readonly(struct btrfs_root *root)
+static inline bool btrfs_root_readonly(const struct btrfs_root *root)
{
return (root->root_item.flags & cpu_to_le64(BTRFS_ROOT_SUBVOL_RDONLY)) != 0;
}
-static inline bool btrfs_root_dead(struct btrfs_root *root)
+static inline bool btrfs_root_dead(const struct btrfs_root *root)
{
return (root->root_item.flags & cpu_to_le64(BTRFS_ROOT_SUBVOL_DEAD)) != 0;
}
@@ -2152,51 +2155,51 @@ BTRFS_SETGET_STACK_FUNCS(backup_num_devices, struct btrfs_root_backup,
/* struct btrfs_balance_item */
BTRFS_SETGET_FUNCS(balance_flags, struct btrfs_balance_item, flags, 64);
-static inline void btrfs_balance_data(struct extent_buffer *eb,
- struct btrfs_balance_item *bi,
+static inline void btrfs_balance_data(const struct extent_buffer *eb,
+ const struct btrfs_balance_item *bi,
struct btrfs_disk_balance_args *ba)
{
read_eb_member(eb, bi, struct btrfs_balance_item, data, ba);
}
static inline void btrfs_set_balance_data(struct extent_buffer *eb,
- struct btrfs_balance_item *bi,
- struct btrfs_disk_balance_args *ba)
+ struct btrfs_balance_item *bi,
+ const struct btrfs_disk_balance_args *ba)
{
write_eb_member(eb, bi, struct btrfs_balance_item, data, ba);
}
-static inline void btrfs_balance_meta(struct extent_buffer *eb,
- struct btrfs_balance_item *bi,
+static inline void btrfs_balance_meta(const struct extent_buffer *eb,
+ const struct btrfs_balance_item *bi,
struct btrfs_disk_balance_args *ba)
{
read_eb_member(eb, bi, struct btrfs_balance_item, meta, ba);
}
static inline void btrfs_set_balance_meta(struct extent_buffer *eb,
- struct btrfs_balance_item *bi,
- struct btrfs_disk_balance_args *ba)
+ struct btrfs_balance_item *bi,
+ const struct btrfs_disk_balance_args *ba)
{
write_eb_member(eb, bi, struct btrfs_balance_item, meta, ba);
}
-static inline void btrfs_balance_sys(struct extent_buffer *eb,
- struct btrfs_balance_item *bi,
+static inline void btrfs_balance_sys(const struct extent_buffer *eb,
+ const struct btrfs_balance_item *bi,
struct btrfs_disk_balance_args *ba)
{
read_eb_member(eb, bi, struct btrfs_balance_item, sys, ba);
}
static inline void btrfs_set_balance_sys(struct extent_buffer *eb,
- struct btrfs_balance_item *bi,
- struct btrfs_disk_balance_args *ba)
+ struct btrfs_balance_item *bi,
+ const struct btrfs_disk_balance_args *ba)
{
write_eb_member(eb, bi, struct btrfs_balance_item, sys, ba);
}
static inline void
btrfs_disk_balance_args_to_cpu(struct btrfs_balance_args *cpu,
- struct btrfs_disk_balance_args *disk)
+ const struct btrfs_disk_balance_args *disk)
{
memset(cpu, 0, sizeof(*cpu));
@@ -2216,7 +2219,7 @@ btrfs_disk_balance_args_to_cpu(struct btrfs_balance_args *cpu,
static inline void
btrfs_cpu_balance_args_to_disk(struct btrfs_disk_balance_args *disk,
- struct btrfs_balance_args *cpu)
+ const struct btrfs_balance_args *cpu)
{
memset(disk, 0, sizeof(*disk));
@@ -2284,7 +2287,7 @@ BTRFS_SETGET_STACK_FUNCS(super_magic, struct btrfs_super_block, magic, 64);
BTRFS_SETGET_STACK_FUNCS(super_uuid_tree_generation, struct btrfs_super_block,
uuid_tree_generation, 64);
-static inline int btrfs_super_csum_size(struct btrfs_super_block *s)
+static inline int btrfs_super_csum_size(const struct btrfs_super_block *s)
{
u16 t = btrfs_super_csum_type(s);
/*
@@ -2303,8 +2306,8 @@ static inline unsigned long btrfs_leaf_data(struct extent_buffer *l)
* this returns the address of the start of the last item,
* which is the stop of the leaf data stack
*/
-static inline unsigned int leaf_data_end(struct btrfs_root *root,
- struct extent_buffer *leaf)
+static inline unsigned int leaf_data_end(const struct btrfs_root *root,
+ const struct extent_buffer *leaf)
{
u32 nr = btrfs_header_nritems(leaf);
@@ -2329,7 +2332,7 @@ BTRFS_SETGET_STACK_FUNCS(stack_file_extent_compression,
struct btrfs_file_extent_item, compression, 8);
static inline unsigned long
-btrfs_file_extent_inline_start(struct btrfs_file_extent_item *e)
+btrfs_file_extent_inline_start(const struct btrfs_file_extent_item *e)
{
return (unsigned long)e + BTRFS_FILE_EXTENT_INLINE_DATA_START;
}
@@ -2363,8 +2366,9 @@ BTRFS_SETGET_FUNCS(file_extent_other_encoding, struct btrfs_file_extent_item,
* size of any extent headers. If a file is compressed on disk, this is
* the compressed size
*/
-static inline u32 btrfs_file_extent_inline_item_len(struct extent_buffer *eb,
- struct btrfs_item *e)
+static inline u32 btrfs_file_extent_inline_item_len(
+ const struct extent_buffer *eb,
+ struct btrfs_item *e)
{
return btrfs_item_size(eb, e) - BTRFS_FILE_EXTENT_INLINE_DATA_START;
}
@@ -2372,9 +2376,9 @@ static inline u32 btrfs_file_extent_inline_item_len(struct extent_buffer *eb,
/* this returns the number of file bytes represented by the inline item.
* If an item is compressed, this is the uncompressed size
*/
-static inline u32 btrfs_file_extent_inline_len(struct extent_buffer *eb,
- int slot,
- struct btrfs_file_extent_item *fi)
+static inline u32 btrfs_file_extent_inline_len(const struct extent_buffer *eb,
+ int slot,
+ const struct btrfs_file_extent_item *fi)
{
struct btrfs_map_token token;
@@ -2396,8 +2400,8 @@ static inline u32 btrfs_file_extent_inline_len(struct extent_buffer *eb,
/* btrfs_dev_stats_item */
-static inline u64 btrfs_dev_stats_value(struct extent_buffer *eb,
- struct btrfs_dev_stats_item *ptr,
+static inline u64 btrfs_dev_stats_value(const struct extent_buffer *eb,
+ const struct btrfs_dev_stats_item *ptr,
int index)
{
u64 val;
diff --git a/fs/btrfs/dev-replace.c b/fs/btrfs/dev-replace.c
index 05169ef..b450adf 100644
--- a/fs/btrfs/dev-replace.c
+++ b/fs/btrfs/dev-replace.c
@@ -586,6 +586,12 @@ static int btrfs_dev_replace_finishing(struct btrfs_fs_info *fs_info,
btrfs_rm_dev_replace_unblocked(fs_info);
/*
+ * Increment dev_stats_ccnt so that btrfs_run_dev_stats() will
+ * update on-disk dev stats value during commit transaction
+ */
+ atomic_inc(&tgt_device->dev_stats_ccnt);
+
+ /*
* this is again a consistent state where no dev_replace procedure
* is running, the target device is part of the filesystem, the
* source device is not part of the filesystem anymore and its 1st
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index c66054c..e3349a6 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -50,6 +50,7 @@
#include "sysfs.h"
#include "qgroup.h"
#include "compression.h"
+#include "tree-checker.h"
#ifdef CONFIG_X86
#include <asm/cpufeature.h>
@@ -59,7 +60,8 @@
BTRFS_HEADER_FLAG_RELOC |\
BTRFS_SUPER_FLAG_ERROR |\
BTRFS_SUPER_FLAG_SEEDING |\
- BTRFS_SUPER_FLAG_METADUMP)
+ BTRFS_SUPER_FLAG_METADUMP |\
+ BTRFS_SUPER_FLAG_METADUMP_V2)
static const struct extent_io_ops btree_extent_io_ops;
static void end_workqueue_fn(struct btrfs_work *work);
@@ -451,9 +453,9 @@ static int btree_read_extent_buffer_pages(struct btrfs_root *root,
int mirror_num = 0;
int failed_mirror = 0;
- clear_bit(EXTENT_BUFFER_CORRUPT, &eb->bflags);
io_tree = &BTRFS_I(root->fs_info->btree_inode)->io_tree;
while (1) {
+ clear_bit(EXTENT_BUFFER_CORRUPT, &eb->bflags);
ret = read_extent_buffer_pages(io_tree, eb, WAIT_COMPLETE,
btree_get_extent, mirror_num);
if (!ret) {
@@ -464,14 +466,6 @@ static int btree_read_extent_buffer_pages(struct btrfs_root *root,
ret = -EIO;
}
- /*
- * This buffer's crc is fine, but its contents are corrupted, so
- * there is no reason to read the other copies, they won't be
- * any less wrong.
- */
- if (test_bit(EXTENT_BUFFER_CORRUPT, &eb->bflags))
- break;
-
num_copies = btrfs_num_copies(root->fs_info,
eb->start, eb->len);
if (num_copies == 1)
@@ -545,145 +539,6 @@ static int check_tree_block_fsid(struct btrfs_fs_info *fs_info,
return ret;
}
-#define CORRUPT(reason, eb, root, slot) \
- btrfs_crit(root->fs_info, "corrupt %s, %s: block=%llu," \
- " root=%llu, slot=%d", \
- btrfs_header_level(eb) == 0 ? "leaf" : "node",\
- reason, btrfs_header_bytenr(eb), root->objectid, slot)
-
-static noinline int check_leaf(struct btrfs_root *root,
- struct extent_buffer *leaf)
-{
- struct btrfs_key key;
- struct btrfs_key leaf_key;
- u32 nritems = btrfs_header_nritems(leaf);
- int slot;
-
- /*
- * Extent buffers from a relocation tree have a owner field that
- * corresponds to the subvolume tree they are based on. So just from an
- * extent buffer alone we can not find out what is the id of the
- * corresponding subvolume tree, so we can not figure out if the extent
- * buffer corresponds to the root of the relocation tree or not. So skip
- * this check for relocation trees.
- */
- if (nritems == 0 && !btrfs_header_flag(leaf, BTRFS_HEADER_FLAG_RELOC)) {
- struct btrfs_root *check_root;
-
- key.objectid = btrfs_header_owner(leaf);
- key.type = BTRFS_ROOT_ITEM_KEY;
- key.offset = (u64)-1;
-
- check_root = btrfs_get_fs_root(root->fs_info, &key, false);
- /*
- * The only reason we also check NULL here is that during
- * open_ctree() some roots has not yet been set up.
- */
- if (!IS_ERR_OR_NULL(check_root)) {
- struct extent_buffer *eb;
-
- eb = btrfs_root_node(check_root);
- /* if leaf is the root, then it's fine */
- if (leaf != eb) {
- CORRUPT("non-root leaf's nritems is 0",
- leaf, check_root, 0);
- free_extent_buffer(eb);
- return -EIO;
- }
- free_extent_buffer(eb);
- }
- return 0;
- }
-
- if (nritems == 0)
- return 0;
-
- /* Check the 0 item */
- if (btrfs_item_offset_nr(leaf, 0) + btrfs_item_size_nr(leaf, 0) !=
- BTRFS_LEAF_DATA_SIZE(root)) {
- CORRUPT("invalid item offset size pair", leaf, root, 0);
- return -EIO;
- }
-
- /*
- * Check to make sure each items keys are in the correct order and their
- * offsets make sense. We only have to loop through nritems-1 because
- * we check the current slot against the next slot, which verifies the
- * next slot's offset+size makes sense and that the current's slot
- * offset is correct.
- */
- for (slot = 0; slot < nritems - 1; slot++) {
- btrfs_item_key_to_cpu(leaf, &leaf_key, slot);
- btrfs_item_key_to_cpu(leaf, &key, slot + 1);
-
- /* Make sure the keys are in the right order */
- if (btrfs_comp_cpu_keys(&leaf_key, &key) >= 0) {
- CORRUPT("bad key order", leaf, root, slot);
- return -EIO;
- }
-
- /*
- * Make sure the offset and ends are right, remember that the
- * item data starts at the end of the leaf and grows towards the
- * front.
- */
- if (btrfs_item_offset_nr(leaf, slot) !=
- btrfs_item_end_nr(leaf, slot + 1)) {
- CORRUPT("slot offset bad", leaf, root, slot);
- return -EIO;
- }
-
- /*
- * Check to make sure that we don't point outside of the leaf,
- * just in case all the items are consistent to each other, but
- * all point outside of the leaf.
- */
- if (btrfs_item_end_nr(leaf, slot) >
- BTRFS_LEAF_DATA_SIZE(root)) {
- CORRUPT("slot end outside of leaf", leaf, root, slot);
- return -EIO;
- }
- }
-
- return 0;
-}
-
-static int check_node(struct btrfs_root *root, struct extent_buffer *node)
-{
- unsigned long nr = btrfs_header_nritems(node);
- struct btrfs_key key, next_key;
- int slot;
- u64 bytenr;
- int ret = 0;
-
- if (nr == 0 || nr > BTRFS_NODEPTRS_PER_BLOCK(root)) {
- btrfs_crit(root->fs_info,
- "corrupt node: block %llu root %llu nritems %lu",
- node->start, root->objectid, nr);
- return -EIO;
- }
-
- for (slot = 0; slot < nr - 1; slot++) {
- bytenr = btrfs_node_blockptr(node, slot);
- btrfs_node_key_to_cpu(node, &key, slot);
- btrfs_node_key_to_cpu(node, &next_key, slot + 1);
-
- if (!bytenr) {
- CORRUPT("invalid item slot", node, root, slot);
- ret = -EIO;
- goto out;
- }
-
- if (btrfs_comp_cpu_keys(&key, &next_key) >= 0) {
- CORRUPT("bad key order", node, root, slot);
- ret = -EIO;
- goto out;
- }
- }
-out:
- return ret;
-}
-
static int btree_readpage_end_io_hook(struct btrfs_io_bio *io_bio,
u64 phy_offset, struct page *page,
u64 start, u64 end, int mirror)
@@ -749,12 +604,12 @@ static int btree_readpage_end_io_hook(struct btrfs_io_bio *io_bio,
* that we don't try and read the other copies of this block, just
* return -EIO.
*/
- if (found_level == 0 && check_leaf(root, eb)) {
+ if (found_level == 0 && btrfs_check_leaf_full(root, eb)) {
set_bit(EXTENT_BUFFER_CORRUPT, &eb->bflags);
ret = -EIO;
}
- if (found_level > 0 && check_node(root, eb))
+ if (found_level > 0 && btrfs_check_node(root, eb))
ret = -EIO;
if (!ret)
@@ -1095,8 +950,9 @@ static int btree_writepages(struct address_space *mapping,
fs_info = BTRFS_I(mapping->host)->root->fs_info;
/* this is a bit racy, but that's ok */
- ret = percpu_counter_compare(&fs_info->dirty_metadata_bytes,
- BTRFS_DIRTY_METADATA_THRESH);
+ ret = __percpu_counter_compare(&fs_info->dirty_metadata_bytes,
+ BTRFS_DIRTY_METADATA_THRESH,
+ fs_info->dirty_metadata_batch);
if (ret < 0)
return 0;
}
@@ -1281,7 +1137,7 @@ static struct btrfs_subvolume_writers *btrfs_alloc_subvolume_writers(void)
if (!writers)
return ERR_PTR(-ENOMEM);
- ret = percpu_counter_init(&writers->counter, 0, GFP_KERNEL);
+ ret = percpu_counter_init(&writers->counter, 0, GFP_NOFS);
if (ret < 0) {
kfree(writers);
return ERR_PTR(ret);
@@ -4084,7 +3940,13 @@ void btrfs_mark_buffer_dirty(struct extent_buffer *buf)
buf->len,
root->fs_info->dirty_metadata_batch);
#ifdef CONFIG_BTRFS_FS_CHECK_INTEGRITY
- if (btrfs_header_level(buf) == 0 && check_leaf(root, buf)) {
+ /*
+ * Since btrfs_mark_buffer_dirty() can be called with item pointer set
+ * but item data not updated.
+ * So here we should only check item pointers, not item data.
+ */
+ if (btrfs_header_level(buf) == 0 &&
+ btrfs_check_leaf_relaxed(root, buf)) {
btrfs_print_leaf(root, buf);
ASSERT(0);
}
@@ -4106,8 +3968,9 @@ static void __btrfs_btree_balance_dirty(struct btrfs_root *root,
if (flush_delayed)
btrfs_balance_delayed_items(root);
- ret = percpu_counter_compare(&root->fs_info->dirty_metadata_bytes,
- BTRFS_DIRTY_METADATA_THRESH);
+ ret = __percpu_counter_compare(&root->fs_info->dirty_metadata_bytes,
+ BTRFS_DIRTY_METADATA_THRESH,
+ root->fs_info->dirty_metadata_batch);
if (ret > 0) {
balance_dirty_pages_ratelimited(
root->fs_info->btree_inode->i_mapping);
@@ -4142,9 +4005,11 @@ static int btrfs_check_super_valid(struct btrfs_fs_info *fs_info,
btrfs_err(fs_info, "no valid FS found");
ret = -EINVAL;
}
- if (btrfs_super_flags(sb) & ~BTRFS_SUPER_FLAG_SUPP)
- btrfs_warn(fs_info, "unrecognized super flag: %llu",
+ if (btrfs_super_flags(sb) & ~BTRFS_SUPER_FLAG_SUPP) {
+ btrfs_err(fs_info, "unrecognized or unsupported super flag: %llu",
btrfs_super_flags(sb) & ~BTRFS_SUPER_FLAG_SUPP);
+ ret = -EINVAL;
+ }
if (btrfs_super_root_level(sb) >= BTRFS_MAX_LEVEL) {
btrfs_err(fs_info, "tree_root level too big: %d >= %d",
btrfs_super_root_level(sb), BTRFS_MAX_LEVEL);
@@ -4486,6 +4351,7 @@ static int btrfs_destroy_marked_extents(struct btrfs_root *root,
static int btrfs_destroy_pinned_extent(struct btrfs_root *root,
struct extent_io_tree *pinned_extents)
{
+ struct btrfs_fs_info *fs_info = root->fs_info;
struct extent_io_tree *unpin;
u64 start;
u64 end;
@@ -4495,21 +4361,31 @@ static int btrfs_destroy_pinned_extent(struct btrfs_root *root,
unpin = pinned_extents;
again:
while (1) {
+ /*
+ * The btrfs_finish_extent_commit() may get the same range as
+ * ours between find_first_extent_bit and clear_extent_dirty.
+ * Hence, hold the unused_bg_unpin_mutex to avoid double unpin
+ * the same extent range.
+ */
+ mutex_lock(&fs_info->unused_bg_unpin_mutex);
ret = find_first_extent_bit(unpin, 0, &start, &end,
EXTENT_DIRTY, NULL);
- if (ret)
+ if (ret) {
+ mutex_unlock(&fs_info->unused_bg_unpin_mutex);
break;
+ }
clear_extent_dirty(unpin, start, end);
btrfs_error_unpin_extent_range(root, start, end);
+ mutex_unlock(&fs_info->unused_bg_unpin_mutex);
cond_resched();
}
if (loop) {
- if (unpin == &root->fs_info->freed_extents[0])
- unpin = &root->fs_info->freed_extents[1];
+ if (unpin == &fs_info->freed_extents[0])
+ unpin = &fs_info->freed_extents[1];
else
- unpin = &root->fs_info->freed_extents[0];
+ unpin = &fs_info->freed_extents[0];
loop = false;
goto again;
}
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index a29730c..7938c48 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -4527,6 +4527,7 @@ static int do_chunk_alloc(struct btrfs_trans_handle *trans,
if (wait_for_alloc) {
mutex_unlock(&fs_info->chunk_mutex);
wait_for_alloc = 0;
+ cond_resched();
goto again;
}
@@ -8262,6 +8263,19 @@ btrfs_init_new_buffer(struct btrfs_trans_handle *trans, struct btrfs_root *root,
if (IS_ERR(buf))
return buf;
+ /*
+ * Extra safety check in case the extent tree is corrupted and extent
+ * allocator chooses to use a tree block which is already used and
+ * locked.
+ */
+ if (buf->lock_owner == current->pid) {
+ btrfs_err_rl(root->fs_info,
+"tree block %llu owner %llu already locked by pid=%d, extent tree corruption detected",
+ buf->start, btrfs_header_owner(buf), current->pid);
+ free_extent_buffer(buf);
+ return ERR_PTR(-EUCLEAN);
+ }
+
btrfs_set_header_generation(buf, trans->transid);
btrfs_set_buffer_lockdep_class(root->root_key.objectid, buf, level);
btrfs_tree_lock(buf);
@@ -9099,15 +9113,14 @@ static noinline int walk_up_proc(struct btrfs_trans_handle *trans,
if (eb == root->node) {
if (wc->flags[level] & BTRFS_BLOCK_FLAG_FULL_BACKREF)
parent = eb->start;
- else
- BUG_ON(root->root_key.objectid !=
- btrfs_header_owner(eb));
+ else if (root->root_key.objectid != btrfs_header_owner(eb))
+ goto owner_mismatch;
} else {
if (wc->flags[level + 1] & BTRFS_BLOCK_FLAG_FULL_BACKREF)
parent = path->nodes[level + 1]->start;
- else
- BUG_ON(root->root_key.objectid !=
- btrfs_header_owner(path->nodes[level + 1]));
+ else if (root->root_key.objectid !=
+ btrfs_header_owner(path->nodes[level + 1]))
+ goto owner_mismatch;
}
btrfs_free_tree_block(trans, root, eb, parent, wc->refs[level] == 1);
@@ -9115,6 +9128,11 @@ static noinline int walk_up_proc(struct btrfs_trans_handle *trans,
wc->refs[level] = 0;
wc->flags[level] = 0;
return 0;
+
+owner_mismatch:
+ btrfs_err_rl(root->fs_info, "unexpected tree owner, have %llu expect %llu",
+ btrfs_header_owner(eb), root->root_key.objectid);
+ return -EUCLEAN;
}
static noinline int walk_down_tree(struct btrfs_trans_handle *trans,
@@ -9168,6 +9186,8 @@ static noinline int walk_up_tree(struct btrfs_trans_handle *trans,
ret = walk_up_proc(trans, root, path, wc);
if (ret > 0)
return 0;
+ if (ret < 0)
+ return ret;
if (path->locks[level]) {
btrfs_tree_unlock_rw(path->nodes[level],
@@ -9876,6 +9896,8 @@ static int find_first_block_group(struct btrfs_root *root,
int ret = 0;
struct btrfs_key found_key;
struct extent_buffer *leaf;
+ struct btrfs_block_group_item bg;
+ u64 flags;
int slot;
ret = btrfs_search_slot(NULL, root, key, path, 0, 0);
@@ -9910,8 +9932,32 @@ static int find_first_block_group(struct btrfs_root *root,
"logical %llu len %llu found bg but no related chunk",
found_key.objectid, found_key.offset);
ret = -ENOENT;
+ } else if (em->start != found_key.objectid ||
+ em->len != found_key.offset) {
+ btrfs_err(root->fs_info,
+ "block group %llu len %llu mismatch with chunk %llu len %llu",
+ found_key.objectid, found_key.offset,
+ em->start, em->len);
+ ret = -EUCLEAN;
} else {
- ret = 0;
+ read_extent_buffer(leaf, &bg,
+ btrfs_item_ptr_offset(leaf, slot),
+ sizeof(bg));
+ flags = btrfs_block_group_flags(&bg) &
+ BTRFS_BLOCK_GROUP_TYPE_MASK;
+
+ if (flags != (em->map_lookup->type &
+ BTRFS_BLOCK_GROUP_TYPE_MASK)) {
+ btrfs_err(root->fs_info,
+"block group %llu len %llu type flags 0x%llx mismatch with chunk type flags 0x%llx",
+ found_key.objectid,
+ found_key.offset, flags,
+ (BTRFS_BLOCK_GROUP_TYPE_MASK &
+ em->map_lookup->type));
+ ret = -EUCLEAN;
+ } else {
+ ret = 0;
+ }
}
free_extent_map(em);
goto out;
@@ -9932,6 +9978,7 @@ void btrfs_put_block_group_cache(struct btrfs_fs_info *info)
block_group = btrfs_lookup_first_block_group(info, last);
while (block_group) {
+ wait_block_group_cache_done(block_group);
spin_lock(&block_group->lock);
if (block_group->iref)
break;
@@ -10138,6 +10185,62 @@ btrfs_create_block_group_cache(struct btrfs_root *root, u64 start, u64 size)
return cache;
}
+
+/*
+ * Iterate all chunks and verify that each of them has the corresponding block
+ * group
+ */
+static int check_chunk_block_group_mappings(struct btrfs_fs_info *fs_info)
+{
+ struct btrfs_mapping_tree *map_tree = &fs_info->mapping_tree;
+ struct extent_map *em;
+ struct btrfs_block_group_cache *bg;
+ u64 start = 0;
+ int ret = 0;
+
+ while (1) {
+ read_lock(&map_tree->map_tree.lock);
+ /*
+ * lookup_extent_mapping will return the first extent map
+ * intersecting the range, so setting @len to 1 is enough to
+ * get the first chunk.
+ */
+ em = lookup_extent_mapping(&map_tree->map_tree, start, 1);
+ read_unlock(&map_tree->map_tree.lock);
+ if (!em)
+ break;
+
+ bg = btrfs_lookup_block_group(fs_info, em->start);
+ if (!bg) {
+ btrfs_err(fs_info,
+ "chunk start=%llu len=%llu doesn't have corresponding block group",
+ em->start, em->len);
+ ret = -EUCLEAN;
+ free_extent_map(em);
+ break;
+ }
+ if (bg->key.objectid != em->start ||
+ bg->key.offset != em->len ||
+ (bg->flags & BTRFS_BLOCK_GROUP_TYPE_MASK) !=
+ (em->map_lookup->type & BTRFS_BLOCK_GROUP_TYPE_MASK)) {
+ btrfs_err(fs_info,
+"chunk start=%llu len=%llu flags=0x%llx doesn't match block group start=%llu len=%llu flags=0x%llx",
+ em->start, em->len,
+ em->map_lookup->type & BTRFS_BLOCK_GROUP_TYPE_MASK,
+ bg->key.objectid, bg->key.offset,
+ bg->flags & BTRFS_BLOCK_GROUP_TYPE_MASK);
+ ret = -EUCLEAN;
+ free_extent_map(em);
+ btrfs_put_block_group(bg);
+ break;
+ }
+ start = em->start + em->len;
+ free_extent_map(em);
+ btrfs_put_block_group(bg);
+ }
+ return ret;
+}
+
int btrfs_read_block_groups(struct btrfs_root *root)
{
struct btrfs_path *path;
@@ -10322,7 +10425,7 @@ int btrfs_read_block_groups(struct btrfs_root *root)
}
init_global_block_rsv(info);
- ret = 0;
+ ret = check_chunk_block_group_mappings(info);
error:
btrfs_free_path(path);
return ret;
@@ -10331,7 +10434,7 @@ int btrfs_read_block_groups(struct btrfs_root *root)
void btrfs_create_pending_block_groups(struct btrfs_trans_handle *trans,
struct btrfs_root *root)
{
- struct btrfs_block_group_cache *block_group, *tmp;
+ struct btrfs_block_group_cache *block_group;
struct btrfs_root *extent_root = root->fs_info->extent_root;
struct btrfs_block_group_item item;
struct btrfs_key key;
@@ -10339,7 +10442,10 @@ void btrfs_create_pending_block_groups(struct btrfs_trans_handle *trans,
bool can_flush_pending_bgs = trans->can_flush_pending_bgs;
trans->can_flush_pending_bgs = false;
- list_for_each_entry_safe(block_group, tmp, &trans->new_bgs, bg_list) {
+ while (!list_empty(&trans->new_bgs)) {
+ block_group = list_first_entry(&trans->new_bgs,
+ struct btrfs_block_group_cache,
+ bg_list);
if (ret)
goto next;
@@ -10852,7 +10958,7 @@ void btrfs_delete_unused_bgs(struct btrfs_fs_info *fs_info)
/* Don't want to race with allocators so take the groups_sem */
down_write(&space_info->groups_sem);
spin_lock(&block_group->lock);
- if (block_group->reserved ||
+ if (block_group->reserved || block_group->pinned ||
btrfs_block_group_used(&block_group->item) ||
block_group->ro ||
list_is_singular(&block_group->list)) {
@@ -11051,6 +11157,10 @@ static int btrfs_trim_free_extents(struct btrfs_device *device,
*trimmed = 0;
+ /* Discard not supported = nothing to do. */
+ if (!blk_queue_discard(bdev_get_queue(device->bdev)))
+ return 0;
+
/* Not writeable = nothing to do. */
if (!device->writeable)
return 0;
@@ -11112,6 +11222,15 @@ static int btrfs_trim_free_extents(struct btrfs_device *device,
return ret;
}
+/*
+ * Trim the whole filesystem by:
+ * 1) trimming the free space in each block group
+ * 2) trimming the unallocated space on each device
+ *
+ * This will also continue trimming even if a block group or device encounters
+ * an error. The return value will be the last error, or 0 if nothing bad
+ * happens.
+ */
int btrfs_trim_fs(struct btrfs_root *root, struct fstrim_range *range)
{
struct btrfs_fs_info *fs_info = root->fs_info;
@@ -11122,18 +11241,14 @@ int btrfs_trim_fs(struct btrfs_root *root, struct fstrim_range *range)
u64 start;
u64 end;
u64 trimmed = 0;
- u64 total_bytes = btrfs_super_total_bytes(fs_info->super_copy);
+ u64 bg_failed = 0;
+ u64 dev_failed = 0;
+ int bg_ret = 0;
+ int dev_ret = 0;
int ret = 0;
- /*
- * try to trim all FS space, our block group may start from non-zero.
- */
- if (range->len == total_bytes)
- cache = btrfs_lookup_first_block_group(fs_info, range->start);
- else
- cache = btrfs_lookup_block_group(fs_info, range->start);
-
- while (cache) {
+ cache = btrfs_lookup_first_block_group(fs_info, range->start);
+ for (; cache; cache = next_block_group(fs_info->tree_root, cache)) {
if (cache->key.objectid >= (range->start + range->len)) {
btrfs_put_block_group(cache);
break;
@@ -11147,13 +11262,15 @@ int btrfs_trim_fs(struct btrfs_root *root, struct fstrim_range *range)
if (!block_group_cache_done(cache)) {
ret = cache_block_group(cache, 0);
if (ret) {
- btrfs_put_block_group(cache);
- break;
+ bg_failed++;
+ bg_ret = ret;
+ continue;
}
ret = wait_block_group_cache_done(cache);
if (ret) {
- btrfs_put_block_group(cache);
- break;
+ bg_failed++;
+ bg_ret = ret;
+ continue;
}
}
ret = btrfs_trim_block_group(cache,
@@ -11164,28 +11281,40 @@ int btrfs_trim_fs(struct btrfs_root *root, struct fstrim_range *range)
trimmed += group_trimmed;
if (ret) {
- btrfs_put_block_group(cache);
- break;
+ bg_failed++;
+ bg_ret = ret;
+ continue;
}
}
-
- cache = next_block_group(fs_info->tree_root, cache);
}
- mutex_lock(&root->fs_info->fs_devices->device_list_mutex);
- devices = &root->fs_info->fs_devices->alloc_list;
- list_for_each_entry(device, devices, dev_alloc_list) {
+ if (bg_failed)
+ btrfs_warn(fs_info,
+ "failed to trim %llu block group(s), last error %d",
+ bg_failed, bg_ret);
+ mutex_lock(&fs_info->fs_devices->device_list_mutex);
+ devices = &fs_info->fs_devices->devices;
+ list_for_each_entry(device, devices, dev_list) {
ret = btrfs_trim_free_extents(device, range->minlen,
&group_trimmed);
- if (ret)
+ if (ret) {
+ dev_failed++;
+ dev_ret = ret;
break;
+ }
trimmed += group_trimmed;
}
mutex_unlock(&root->fs_info->fs_devices->device_list_mutex);
+ if (dev_failed)
+ btrfs_warn(fs_info,
+ "failed to trim %llu device(s), last error %d",
+ dev_failed, dev_ret);
range->len = trimmed;
- return ret;
+ if (bg_ret)
+ return bg_ret;
+ return dev_ret;
}
/*
diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index 03ac3ab..1a45df7 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -3830,8 +3830,8 @@ int btree_write_cache_pages(struct address_space *mapping,
if (wbc->sync_mode == WB_SYNC_ALL)
tag_pages_for_writeback(mapping, index, end);
while (!done && !nr_to_write_done && (index <= end) &&
- (nr_pages = pagevec_lookup_tag(&pvec, mapping, &index, tag,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1))) {
+ (nr_pages = pagevec_lookup_range_tag(&pvec, mapping, &index, end,
+ tag))) {
unsigned i;
scanned = 1;
@@ -3841,11 +3841,6 @@ int btree_write_cache_pages(struct address_space *mapping,
if (!PagePrivate(page))
continue;
- if (!wbc->range_cyclic && page->index > end) {
- done = 1;
- break;
- }
-
spin_lock(&mapping->private_lock);
if (!PagePrivate(page)) {
spin_unlock(&mapping->private_lock);
@@ -3978,8 +3973,8 @@ static int extent_write_cache_pages(struct extent_io_tree *tree,
tag_pages_for_writeback(mapping, index, end);
done_index = index;
while (!done && !nr_to_write_done && (index <= end) &&
- (nr_pages = pagevec_lookup_tag(&pvec, mapping, &index, tag,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1))) {
+ (nr_pages = pagevec_lookup_range_tag(&pvec, mapping,
+ &index, end, tag))) {
unsigned i;
scanned = 1;
@@ -4004,12 +3999,6 @@ static int extent_write_cache_pages(struct extent_io_tree *tree,
continue;
}
- if (!wbc->range_cyclic && page->index > end) {
- done = 1;
- unlock_page(page);
- continue;
- }
-
if (wbc->sync_mode != WB_SYNC_NONE) {
if (PageWriteback(page))
flush_fn(data);
@@ -4298,6 +4287,7 @@ int try_release_extent_mapping(struct extent_map_tree *map,
struct extent_map *em;
u64 start = page_offset(page);
u64 end = start + PAGE_SIZE - 1;
+ struct btrfs_inode *btrfs_inode = BTRFS_I(page->mapping->host);
if (gfpflags_allow_blocking(mask) &&
page->mapping->host->i_size > SZ_16M) {
@@ -4320,6 +4310,8 @@ int try_release_extent_mapping(struct extent_map_tree *map,
extent_map_end(em) - 1,
EXTENT_LOCKED | EXTENT_WRITEBACK,
0, NULL)) {
+ set_bit(BTRFS_INODE_NEEDS_FULL_SYNC,
+ &btrfs_inode->runtime_flags);
remove_extent_mapping(map, em);
/* once for the rb tree */
free_extent_map(em);
@@ -4374,6 +4366,123 @@ static struct extent_map *get_extent_skip_holes(struct inode *inode,
return NULL;
}
+/*
+ * To cache previous fiemap extent
+ *
+ * Will be used for merging fiemap extent
+ */
+struct fiemap_cache {
+ u64 offset;
+ u64 phys;
+ u64 len;
+ u32 flags;
+ bool cached;
+};
+
+/*
+ * Helper to submit fiemap extent.
+ *
+ * Will try to merge current fiemap extent specified by @offset, @phys,
+ * @len and @flags with cached one.
+ * And only when we fails to merge, cached one will be submitted as
+ * fiemap extent.
+ *
+ * Return value is the same as fiemap_fill_next_extent().
+ */
+static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo,
+ struct fiemap_cache *cache,
+ u64 offset, u64 phys, u64 len, u32 flags)
+{
+ int ret = 0;
+
+ if (!cache->cached)
+ goto assign;
+
+ /*
+ * Sanity check, extent_fiemap() should have ensured that new
+ * fiemap extent won't overlap with cahced one.
+ * Not recoverable.
+ *
+ * NOTE: Physical address can overlap, due to compression
+ */
+ if (cache->offset + cache->len > offset) {
+ WARN_ON(1);
+ return -EINVAL;
+ }
+
+ /*
+ * Only merges fiemap extents if
+ * 1) Their logical addresses are continuous
+ *
+ * 2) Their physical addresses are continuous
+ * So truly compressed (physical size smaller than logical size)
+ * extents won't get merged with each other
+ *
+ * 3) Share same flags except FIEMAP_EXTENT_LAST
+ * So regular extent won't get merged with prealloc extent
+ */
+ if (cache->offset + cache->len == offset &&
+ cache->phys + cache->len == phys &&
+ (cache->flags & ~FIEMAP_EXTENT_LAST) ==
+ (flags & ~FIEMAP_EXTENT_LAST)) {
+ cache->len += len;
+ cache->flags |= flags;
+ goto try_submit_last;
+ }
+
+ /* Not mergeable, need to submit cached one */
+ ret = fiemap_fill_next_extent(fieinfo, cache->offset, cache->phys,
+ cache->len, cache->flags);
+ cache->cached = false;
+ if (ret)
+ return ret;
+assign:
+ cache->cached = true;
+ cache->offset = offset;
+ cache->phys = phys;
+ cache->len = len;
+ cache->flags = flags;
+try_submit_last:
+ if (cache->flags & FIEMAP_EXTENT_LAST) {
+ ret = fiemap_fill_next_extent(fieinfo, cache->offset,
+ cache->phys, cache->len, cache->flags);
+ cache->cached = false;
+ }
+ return ret;
+}
+
+/*
+ * Sanity check for fiemap cache
+ *
+ * All fiemap cache should be submitted by emit_fiemap_extent()
+ * Iteration should be terminated either by last fiemap extent or
+ * fieinfo->fi_extents_max.
+ * So no cached fiemap should exist.
+ */
+static int check_fiemap_cache(struct btrfs_fs_info *fs_info,
+ struct fiemap_extent_info *fieinfo,
+ struct fiemap_cache *cache)
+{
+ int ret;
+
+ if (!cache->cached)
+ return 0;
+
+ /* Small and recoverbale problem, only to info developer */
+#ifdef CONFIG_BTRFS_DEBUG
+ WARN_ON(1);
+#endif
+ btrfs_warn(fs_info,
+ "unhandled fiemap cache detected: offset=%llu phys=%llu len=%llu flags=0x%x",
+ cache->offset, cache->phys, cache->len, cache->flags);
+ ret = fiemap_fill_next_extent(fieinfo, cache->offset, cache->phys,
+ cache->len, cache->flags);
+ cache->cached = false;
+ if (ret > 0)
+ ret = 0;
+ return ret;
+}
+
int extent_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
__u64 start, __u64 len, get_extent_t *get_extent)
{
@@ -4391,6 +4500,7 @@ int extent_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
struct extent_state *cached_state = NULL;
struct btrfs_path *path;
struct btrfs_root *root = BTRFS_I(inode)->root;
+ struct fiemap_cache cache = { 0 };
int end = 0;
u64 em_start = 0;
u64 em_len = 0;
@@ -4570,8 +4680,8 @@ int extent_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
flags |= FIEMAP_EXTENT_LAST;
end = 1;
}
- ret = fiemap_fill_next_extent(fieinfo, em_start, disko,
- em_len, flags);
+ ret = emit_fiemap_extent(fieinfo, &cache, em_start, disko,
+ em_len, flags);
if (ret) {
if (ret == 1)
ret = 0;
@@ -4579,6 +4689,8 @@ int extent_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
}
}
out_free:
+ if (!ret)
+ ret = check_fiemap_cache(root->fs_info, fieinfo, &cache);
free_extent_map(em);
out:
btrfs_free_path(path);
@@ -5319,9 +5431,8 @@ int read_extent_buffer_pages(struct extent_io_tree *tree,
return ret;
}
-void read_extent_buffer(struct extent_buffer *eb, void *dstv,
- unsigned long start,
- unsigned long len)
+void read_extent_buffer(const struct extent_buffer *eb, void *dstv,
+ unsigned long start, unsigned long len)
{
size_t cur;
size_t offset;
@@ -5350,9 +5461,9 @@ void read_extent_buffer(struct extent_buffer *eb, void *dstv,
}
}
-int read_extent_buffer_to_user(struct extent_buffer *eb, void __user *dstv,
- unsigned long start,
- unsigned long len)
+int read_extent_buffer_to_user(const struct extent_buffer *eb,
+ void __user *dstv,
+ unsigned long start, unsigned long len)
{
size_t cur;
size_t offset;
@@ -5392,10 +5503,10 @@ int read_extent_buffer_to_user(struct extent_buffer *eb, void __user *dstv,
* return 1 if the item spans two pages.
* return -EINVAL otherwise.
*/
-int map_private_extent_buffer(struct extent_buffer *eb, unsigned long start,
- unsigned long min_len, char **map,
- unsigned long *map_start,
- unsigned long *map_len)
+int map_private_extent_buffer(const struct extent_buffer *eb,
+ unsigned long start, unsigned long min_len,
+ char **map, unsigned long *map_start,
+ unsigned long *map_len)
{
size_t offset = start & (PAGE_SIZE - 1);
char *kaddr;
@@ -5429,9 +5540,8 @@ int map_private_extent_buffer(struct extent_buffer *eb, unsigned long start,
return 0;
}
-int memcmp_extent_buffer(struct extent_buffer *eb, const void *ptrv,
- unsigned long start,
- unsigned long len)
+int memcmp_extent_buffer(const struct extent_buffer *eb, const void *ptrv,
+ unsigned long start, unsigned long len)
{
size_t cur;
size_t offset;
diff --git a/fs/btrfs/extent_io.h b/fs/btrfs/extent_io.h
index ab31d14..9ecdc95 100644
--- a/fs/btrfs/extent_io.h
+++ b/fs/btrfs/extent_io.h
@@ -396,14 +396,13 @@ static inline void extent_buffer_get(struct extent_buffer *eb)
atomic_inc(&eb->refs);
}
-int memcmp_extent_buffer(struct extent_buffer *eb, const void *ptrv,
- unsigned long start,
- unsigned long len);
-void read_extent_buffer(struct extent_buffer *eb, void *dst,
+int memcmp_extent_buffer(const struct extent_buffer *eb, const void *ptrv,
+ unsigned long start, unsigned long len);
+void read_extent_buffer(const struct extent_buffer *eb, void *dst,
unsigned long start,
unsigned long len);
-int read_extent_buffer_to_user(struct extent_buffer *eb, void __user *dst,
- unsigned long start,
+int read_extent_buffer_to_user(const struct extent_buffer *eb,
+ void __user *dst, unsigned long start,
unsigned long len);
void write_extent_buffer(struct extent_buffer *eb, const void *src,
unsigned long start, unsigned long len);
@@ -428,10 +427,10 @@ void set_extent_buffer_uptodate(struct extent_buffer *eb);
void clear_extent_buffer_uptodate(struct extent_buffer *eb);
int extent_buffer_uptodate(struct extent_buffer *eb);
int extent_buffer_under_io(struct extent_buffer *eb);
-int map_private_extent_buffer(struct extent_buffer *eb, unsigned long offset,
- unsigned long min_len, char **map,
- unsigned long *map_start,
- unsigned long *map_len);
+int map_private_extent_buffer(const struct extent_buffer *eb,
+ unsigned long offset, unsigned long min_len,
+ char **map, unsigned long *map_start,
+ unsigned long *map_len);
void extent_range_clear_dirty_for_io(struct inode *inode, u64 start, u64 end);
void extent_range_redirty_for_io(struct inode *inode, u64 start, u64 end);
void extent_clear_unlock_delalloc(struct inode *inode, u64 start, u64 end,
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index c95ff09..4375448 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -1912,10 +1912,19 @@ int btrfs_release_file(struct inode *inode, struct file *filp)
static int start_ordered_ops(struct inode *inode, loff_t start, loff_t end)
{
int ret;
+ struct blk_plug plug;
+ /*
+ * This is only called in fsync, which would do synchronous writes, so
+ * a plug can merge adjacent IOs as much as possible. Esp. in case of
+ * multiple disks using raid profile, a large IO can be split to
+ * several segments of stripe length (currently 64K).
+ */
+ blk_start_plug(&plug);
atomic_inc(&BTRFS_I(inode)->sync_writers);
ret = btrfs_fdatawrite_range(inode, start, end);
atomic_dec(&BTRFS_I(inode)->sync_writers);
+ blk_finish_plug(&plug);
return ret;
}
diff --git a/fs/btrfs/free-space-cache.c b/fs/btrfs/free-space-cache.c
index c56253a..69a3c11 100644
--- a/fs/btrfs/free-space-cache.c
+++ b/fs/btrfs/free-space-cache.c
@@ -1693,6 +1693,8 @@ static inline void __bitmap_clear_bits(struct btrfs_free_space_ctl *ctl,
bitmap_clear(info->bitmap, start, count);
info->bytes -= bytes;
+ if (info->max_extent_size > ctl->unit)
+ info->max_extent_size = 0;
}
static void bitmap_clear_bits(struct btrfs_free_space_ctl *ctl,
@@ -1776,6 +1778,13 @@ static int search_bitmap(struct btrfs_free_space_ctl *ctl,
return -1;
}
+static inline u64 get_max_extent_size(struct btrfs_free_space *entry)
+{
+ if (entry->bitmap)
+ return entry->max_extent_size;
+ return entry->bytes;
+}
+
/* Cache the size of the max extent in bytes */
static struct btrfs_free_space *
find_free_space(struct btrfs_free_space_ctl *ctl, u64 *offset, u64 *bytes,
@@ -1797,8 +1806,8 @@ find_free_space(struct btrfs_free_space_ctl *ctl, u64 *offset, u64 *bytes,
for (node = &entry->offset_index; node; node = rb_next(node)) {
entry = rb_entry(node, struct btrfs_free_space, offset_index);
if (entry->bytes < *bytes) {
- if (entry->bytes > *max_extent_size)
- *max_extent_size = entry->bytes;
+ *max_extent_size = max(get_max_extent_size(entry),
+ *max_extent_size);
continue;
}
@@ -1816,8 +1825,8 @@ find_free_space(struct btrfs_free_space_ctl *ctl, u64 *offset, u64 *bytes,
}
if (entry->bytes < *bytes + align_off) {
- if (entry->bytes > *max_extent_size)
- *max_extent_size = entry->bytes;
+ *max_extent_size = max(get_max_extent_size(entry),
+ *max_extent_size);
continue;
}
@@ -1829,8 +1838,10 @@ find_free_space(struct btrfs_free_space_ctl *ctl, u64 *offset, u64 *bytes,
*offset = tmp;
*bytes = size;
return entry;
- } else if (size > *max_extent_size) {
- *max_extent_size = size;
+ } else {
+ *max_extent_size =
+ max(get_max_extent_size(entry),
+ *max_extent_size);
}
continue;
}
@@ -2453,6 +2464,7 @@ void btrfs_dump_free_space(struct btrfs_block_group_cache *block_group,
struct rb_node *n;
int count = 0;
+ spin_lock(&ctl->tree_lock);
for (n = rb_first(&ctl->free_space_offset); n; n = rb_next(n)) {
info = rb_entry(n, struct btrfs_free_space, offset_index);
if (info->bytes >= bytes && !block_group->ro)
@@ -2462,6 +2474,7 @@ void btrfs_dump_free_space(struct btrfs_block_group_cache *block_group,
info->offset, info->bytes,
(info->bitmap) ? "yes" : "no");
}
+ spin_unlock(&ctl->tree_lock);
btrfs_info(block_group->fs_info, "block group has cluster?: %s",
list_empty(&block_group->cluster_list) ? "no" : "yes");
btrfs_info(block_group->fs_info,
@@ -2689,8 +2702,8 @@ static u64 btrfs_alloc_from_bitmap(struct btrfs_block_group_cache *block_group,
err = search_bitmap(ctl, entry, &search_start, &search_bytes, true);
if (err) {
- if (search_bytes > *max_extent_size)
- *max_extent_size = search_bytes;
+ *max_extent_size = max(get_max_extent_size(entry),
+ *max_extent_size);
return 0;
}
@@ -2727,8 +2740,9 @@ u64 btrfs_alloc_from_cluster(struct btrfs_block_group_cache *block_group,
entry = rb_entry(node, struct btrfs_free_space, offset_index);
while (1) {
- if (entry->bytes < bytes && entry->bytes > *max_extent_size)
- *max_extent_size = entry->bytes;
+ if (entry->bytes < bytes)
+ *max_extent_size = max(get_max_extent_size(entry),
+ *max_extent_size);
if (entry->bytes < bytes ||
(!entry->bitmap && entry->offset < min_start)) {
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index ffd5831..1b1a9e3 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -489,6 +489,7 @@ static noinline void compress_file_range(struct inode *inode,
pages = kcalloc(nr_pages, sizeof(struct page *), GFP_NOFS);
if (!pages) {
/* just bail out to the uncompressed code */
+ nr_pages = 0;
goto cont;
}
@@ -1230,6 +1231,8 @@ static noinline int csum_exist_in_range(struct btrfs_root *root,
list_del(&sums->list);
kfree(sums);
}
+ if (ret < 0)
+ return ret;
return 1;
}
@@ -1381,10 +1384,23 @@ static noinline int run_delalloc_nocow(struct inode *inode,
goto out_check;
if (btrfs_extent_readonly(root, disk_bytenr))
goto out_check;
- if (btrfs_cross_ref_exist(trans, root, ino,
+ ret = btrfs_cross_ref_exist(trans, root, ino,
found_key.offset -
- extent_offset, disk_bytenr))
+ extent_offset, disk_bytenr);
+ if (ret) {
+ /*
+ * ret could be -EIO if the above fails to read
+ * metadata.
+ */
+ if (ret < 0) {
+ if (cow_start != (u64)-1)
+ cur_offset = cow_start;
+ goto error;
+ }
+
+ WARN_ON_ONCE(nolock);
goto out_check;
+ }
disk_bytenr += extent_offset;
disk_bytenr += cur_offset - found_key.offset;
num_bytes = min(end + 1, extent_end) - cur_offset;
@@ -1402,8 +1418,20 @@ static noinline int run_delalloc_nocow(struct inode *inode,
* this ensure that csum for a given extent are
* either valid or do not exist.
*/
- if (csum_exist_in_range(root, disk_bytenr, num_bytes))
+ ret = csum_exist_in_range(root, disk_bytenr, num_bytes);
+ if (ret) {
+ /*
+ * ret could be -EIO if the above fails to read
+ * metadata.
+ */
+ if (ret < 0) {
+ if (cow_start != (u64)-1)
+ cur_offset = cow_start;
+ goto error;
+ }
+ WARN_ON_ONCE(nolock);
goto out_check;
+ }
if (!btrfs_inc_nocow_writers(root->fs_info,
disk_bytenr))
goto out_check;
@@ -1520,12 +1548,11 @@ static noinline int run_delalloc_nocow(struct inode *inode,
}
btrfs_release_path(path);
- if (cur_offset <= end && cow_start == (u64)-1) {
+ if (cur_offset <= end && cow_start == (u64)-1)
cow_start = cur_offset;
- cur_offset = end;
- }
if (cow_start != (u64)-1) {
+ cur_offset = end;
ret = cow_file_range(inode, locked_page, cow_start, end, end,
page_started, nr_written, 1, NULL);
if (ret)
@@ -2939,7 +2966,7 @@ static int btrfs_finish_ordered_io(struct btrfs_ordered_extent *ordered_extent)
ret = test_range_bit(io_tree, ordered_extent->file_offset,
ordered_extent->file_offset + ordered_extent->len - 1,
- EXTENT_DEFRAG, 1, cached_state);
+ EXTENT_DEFRAG, 0, cached_state);
if (ret) {
u64 last_snapshot = btrfs_root_last_snapshot(&root->root_item);
if (0 && last_snapshot >= BTRFS_I(inode)->generation)
@@ -6491,8 +6518,7 @@ static int btrfs_mknod(struct inode *dir, struct dentry *dentry,
goto out_unlock_inode;
} else {
btrfs_update_inode(trans, root, inode);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
}
out_unlock:
@@ -6567,8 +6593,7 @@ static int btrfs_create(struct inode *dir, struct dentry *dentry,
goto out_unlock_inode;
BTRFS_I(inode)->io_tree.ops = &btrfs_extent_io_ops;
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
out_unlock:
btrfs_end_transaction(trans, root);
@@ -6711,12 +6736,7 @@ static int btrfs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode)
if (err)
goto out_fail_inode;
- d_instantiate(dentry, inode);
- /*
- * mkdir is special. We're unlocking after we call d_instantiate
- * to avoid a race with nfsd calling d_instantiate.
- */
- unlock_new_inode(inode);
+ d_instantiate_new(dentry, inode);
drop_on_err = 0;
out_fail:
@@ -9568,6 +9588,7 @@ static int btrfs_rename_exchange(struct inode *old_dir,
u64 new_idx = 0;
u64 root_objectid;
int ret;
+ int ret2;
bool root_log_pinned = false;
bool dest_log_pinned = false;
@@ -9758,7 +9779,8 @@ static int btrfs_rename_exchange(struct inode *old_dir,
dest_log_pinned = false;
}
}
- ret = btrfs_end_transaction(trans, root);
+ ret2 = btrfs_end_transaction(trans, root);
+ ret = ret ? ret : ret2;
out_notrans:
if (new_ino == BTRFS_FIRST_FREE_OBJECTID)
up_read(&dest->fs_info->subvol_sem);
@@ -10354,8 +10376,7 @@ static int btrfs_symlink(struct inode *dir, struct dentry *dentry,
goto out_unlock_inode;
}
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
out_unlock:
btrfs_end_transaction(trans, root);
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index d3dd631..242584a 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -380,7 +380,6 @@ static noinline int btrfs_ioctl_fitrim(struct file *file, void __user *arg)
struct fstrim_range range;
u64 minlen = ULLONG_MAX;
u64 num_devices = 0;
- u64 total_bytes = btrfs_super_total_bytes(fs_info->super_copy);
int ret;
if (!capable(CAP_SYS_ADMIN))
@@ -404,11 +403,15 @@ static noinline int btrfs_ioctl_fitrim(struct file *file, void __user *arg)
return -EOPNOTSUPP;
if (copy_from_user(&range, arg, sizeof(range)))
return -EFAULT;
- if (range.start > total_bytes ||
- range.len < fs_info->sb->s_blocksize)
+
+ /*
+ * NOTE: Don't truncate the range using super->total_bytes. Bytenr of
+ * block group is in the logical address space, which can be any
+ * sectorsize aligned bytenr in the range [0, U64_MAX].
+ */
+ if (range.len < fs_info->sb->s_blocksize)
return -EINVAL;
- range.len = min(range.len, total_bytes - range.start);
range.minlen = max(range.minlen, minlen);
ret = btrfs_trim_fs(fs_info->tree_root, &range);
if (ret < 0)
@@ -2708,8 +2711,10 @@ static long btrfs_ioctl_rm_dev_v2(struct file *file, void __user *arg)
}
/* Check for compatibility reject unknown flags */
- if (vol_args->flags & ~BTRFS_VOL_ARG_V2_FLAGS_SUPPORTED)
- return -EOPNOTSUPP;
+ if (vol_args->flags & ~BTRFS_VOL_ARG_V2_FLAGS_SUPPORTED) {
+ ret = -EOPNOTSUPP;
+ goto out;
+ }
if (atomic_xchg(&root->fs_info->mutually_exclusive_operation_running,
1)) {
@@ -3887,11 +3892,6 @@ static noinline int btrfs_clone_files(struct file *file, struct file *file_src,
src->i_sb != inode->i_sb)
return -EXDEV;
- /* don't make the dst file partly checksummed */
- if ((BTRFS_I(src)->flags & BTRFS_INODE_NODATASUM) !=
- (BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM))
- return -EINVAL;
-
if (S_ISDIR(src->i_mode) || S_ISDIR(inode->i_mode))
return -EISDIR;
@@ -3901,15 +3901,30 @@ static noinline int btrfs_clone_files(struct file *file, struct file *file_src,
inode_lock(src);
}
+ /* don't make the dst file partly checksummed */
+ if ((BTRFS_I(src)->flags & BTRFS_INODE_NODATASUM) !=
+ (BTRFS_I(inode)->flags & BTRFS_INODE_NODATASUM)) {
+ ret = -EINVAL;
+ goto out_unlock;
+ }
+
/* determine range to clone */
ret = -EINVAL;
if (off + len > src->i_size || off + len < off)
goto out_unlock;
if (len == 0)
olen = len = src->i_size - off;
- /* if we extend to eof, continue to block boundary */
- if (off + len == src->i_size)
+ /*
+ * If we extend to eof, continue to block boundary if and only if the
+ * destination end offset matches the destination file's size, otherwise
+ * we would be corrupting data by placing the eof block into the middle
+ * of a file.
+ */
+ if (off + len == src->i_size) {
+ if (!IS_ALIGNED(len, bs) && destoff + len < inode->i_size)
+ goto out_unlock;
len = ALIGN(src->i_size, bs) - off;
+ }
if (len == 0) {
ret = 0;
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index dfd9986..f252330 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -2237,6 +2237,21 @@ void assert_qgroups_uptodate(struct btrfs_trans_handle *trans)
}
/*
+ * Check if the leaf is the last leaf. Which means all node pointers
+ * are at their last position.
+ */
+static bool is_last_leaf(struct btrfs_path *path)
+{
+ int i;
+
+ for (i = 1; i < BTRFS_MAX_LEVEL && path->nodes[i]; i++) {
+ if (path->slots[i] != btrfs_header_nritems(path->nodes[i]) - 1)
+ return false;
+ }
+ return true;
+}
+
+/*
* returns < 0 on error, 0 when more leafs are to be scanned.
* returns 1 when done.
*/
@@ -2249,6 +2264,7 @@ qgroup_rescan_leaf(struct btrfs_fs_info *fs_info, struct btrfs_path *path,
struct ulist *roots = NULL;
struct seq_list tree_mod_seq_elem = SEQ_LIST_INIT(tree_mod_seq_elem);
u64 num_bytes;
+ bool done;
int slot;
int ret;
@@ -2277,6 +2293,7 @@ qgroup_rescan_leaf(struct btrfs_fs_info *fs_info, struct btrfs_path *path,
mutex_unlock(&fs_info->qgroup_rescan_lock);
return ret;
}
+ done = is_last_leaf(path);
btrfs_item_key_to_cpu(path->nodes[0], &found,
btrfs_header_nritems(path->nodes[0]) - 1);
@@ -2323,6 +2340,8 @@ qgroup_rescan_leaf(struct btrfs_fs_info *fs_info, struct btrfs_path *path,
}
btrfs_put_tree_mod_seq(fs_info, &tree_mod_seq_elem);
+ if (done && !ret)
+ ret = 1;
return ret;
}
@@ -2479,6 +2498,7 @@ qgroup_rescan_zero_tracking(struct btrfs_fs_info *fs_info)
qgroup->rfer_cmpr = 0;
qgroup->excl = 0;
qgroup->excl_cmpr = 0;
+ qgroup_dirty(fs_info, qgroup);
}
spin_unlock(&fs_info->qgroup_lock);
}
diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c
index d016d4a..af6a776 100644
--- a/fs/btrfs/raid56.c
+++ b/fs/btrfs/raid56.c
@@ -2161,11 +2161,21 @@ int raid56_parity_recover(struct btrfs_root *root, struct bio *bio,
}
/*
- * reconstruct from the q stripe if they are
- * asking for mirror 3
+ * Loop retry:
+ * for 'mirror == 2', reconstruct from all other stripes.
+ * for 'mirror_num > 2', select a stripe to fail on every retry.
*/
- if (mirror_num == 3)
- rbio->failb = rbio->real_stripes - 2;
+ if (mirror_num > 2) {
+ /*
+ * 'mirror == 3' is to fail the p stripe and
+ * reconstruct from the q stripe. 'mirror > 3' is to
+ * fail a data stripe and reconstruct from p+q stripe.
+ */
+ rbio->failb = rbio->real_stripes - (mirror_num - 1);
+ ASSERT(rbio->failb > 0);
+ if (rbio->failb <= rbio->faila)
+ rbio->failb--;
+ }
ret = lock_stripe_add(rbio);
diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c
index 04c61bc..b0c3a6a 100644
--- a/fs/btrfs/relocation.c
+++ b/fs/btrfs/relocation.c
@@ -1325,18 +1325,19 @@ static void __del_reloc_root(struct btrfs_root *root)
struct mapping_node *node = NULL;
struct reloc_control *rc = root->fs_info->reloc_ctl;
- spin_lock(&rc->reloc_root_tree.lock);
- rb_node = tree_search(&rc->reloc_root_tree.rb_root,
- root->node->start);
- if (rb_node) {
- node = rb_entry(rb_node, struct mapping_node, rb_node);
- rb_erase(&node->rb_node, &rc->reloc_root_tree.rb_root);
+ if (rc && root->node) {
+ spin_lock(&rc->reloc_root_tree.lock);
+ rb_node = tree_search(&rc->reloc_root_tree.rb_root,
+ root->node->start);
+ if (rb_node) {
+ node = rb_entry(rb_node, struct mapping_node, rb_node);
+ rb_erase(&node->rb_node, &rc->reloc_root_tree.rb_root);
+ }
+ spin_unlock(&rc->reloc_root_tree.lock);
+ if (!node)
+ return;
+ BUG_ON((struct btrfs_root *)node->data != root);
}
- spin_unlock(&rc->reloc_root_tree.lock);
-
- if (!node)
- return;
- BUG_ON((struct btrfs_root *)node->data != root);
spin_lock(&root->fs_info->trans_lock);
list_del_init(&root->root_list);
diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c
index fffb9ab..16c0585 100644
--- a/fs/btrfs/scrub.c
+++ b/fs/btrfs/scrub.c
@@ -2519,7 +2519,7 @@ static int scrub_extent(struct scrub_ctx *sctx, u64 logical, u64 len,
have_csum = scrub_find_csum(sctx, logical, csum);
if (have_csum == 0)
++sctx->stat.no_csum;
- if (sctx->is_dev_replace && !have_csum) {
+ if (0 && sctx->is_dev_replace && !have_csum) {
ret = copy_nocow_pages(sctx, logical, l,
mirror_num,
physical_for_dev_replace);
diff --git a/fs/btrfs/send.c b/fs/btrfs/send.c
index d040afc..a45f26a 100644
--- a/fs/btrfs/send.c
+++ b/fs/btrfs/send.c
@@ -3349,7 +3349,8 @@ static void free_pending_move(struct send_ctx *sctx, struct pending_dir_move *m)
kfree(m);
}
-static void tail_append_pending_moves(struct pending_dir_move *moves,
+static void tail_append_pending_moves(struct send_ctx *sctx,
+ struct pending_dir_move *moves,
struct list_head *stack)
{
if (list_empty(&moves->list)) {
@@ -3360,6 +3361,10 @@ static void tail_append_pending_moves(struct pending_dir_move *moves,
list_add_tail(&moves->list, stack);
list_splice_tail(&list, stack);
}
+ if (!RB_EMPTY_NODE(&moves->node)) {
+ rb_erase(&moves->node, &sctx->pending_dir_moves);
+ RB_CLEAR_NODE(&moves->node);
+ }
}
static int apply_children_dir_moves(struct send_ctx *sctx)
@@ -3374,7 +3379,7 @@ static int apply_children_dir_moves(struct send_ctx *sctx)
return 0;
INIT_LIST_HEAD(&stack);
- tail_append_pending_moves(pm, &stack);
+ tail_append_pending_moves(sctx, pm, &stack);
while (!list_empty(&stack)) {
pm = list_first_entry(&stack, struct pending_dir_move, list);
@@ -3385,7 +3390,7 @@ static int apply_children_dir_moves(struct send_ctx *sctx)
goto out;
pm = get_pending_dir_moves(sctx, parent_ino);
if (pm)
- tail_append_pending_moves(pm, &stack);
+ tail_append_pending_moves(sctx, pm, &stack);
}
return 0;
@@ -4822,6 +4827,9 @@ static int send_hole(struct send_ctx *sctx, u64 end)
u64 len;
int ret = 0;
+ if (sctx->flags & BTRFS_SEND_FLAG_NO_FILE_DATA)
+ return send_update_extent(sctx, offset, end - offset);
+
p = fs_path_alloc();
if (!p)
return -ENOMEM;
@@ -5162,15 +5170,12 @@ static int is_extent_unchanged(struct send_ctx *sctx,
goto out;
}
- right_disknr = btrfs_file_extent_disk_bytenr(eb, ei);
if (right_type == BTRFS_FILE_EXTENT_INLINE) {
right_len = btrfs_file_extent_inline_len(eb, slot, ei);
right_len = PAGE_ALIGN(right_len);
} else {
right_len = btrfs_file_extent_num_bytes(eb, ei);
}
- right_offset = btrfs_file_extent_offset(eb, ei);
- right_gen = btrfs_file_extent_generation(eb, ei);
/*
* Are we at extent 8? If yes, we know the extent is changed.
@@ -5195,6 +5200,10 @@ static int is_extent_unchanged(struct send_ctx *sctx,
goto out;
}
+ right_disknr = btrfs_file_extent_disk_bytenr(eb, ei);
+ right_offset = btrfs_file_extent_offset(eb, ei);
+ right_gen = btrfs_file_extent_generation(eb, ei);
+
left_offset_fixed = left_offset;
if (key.offset < ekey->offset) {
/* Fix the right offset for 2a and 7. */
diff --git a/fs/btrfs/struct-funcs.c b/fs/btrfs/struct-funcs.c
index 875c757..5e2b92d 100644
--- a/fs/btrfs/struct-funcs.c
+++ b/fs/btrfs/struct-funcs.c
@@ -50,8 +50,8 @@ static inline void put_unaligned_le8(u8 val, void *p)
*/
#define DEFINE_BTRFS_SETGET_BITS(bits) \
-u##bits btrfs_get_token_##bits(struct extent_buffer *eb, void *ptr, \
- unsigned long off, \
+u##bits btrfs_get_token_##bits(const struct extent_buffer *eb, \
+ const void *ptr, unsigned long off, \
struct btrfs_map_token *token) \
{ \
unsigned long part_offset = (unsigned long)ptr; \
@@ -90,7 +90,8 @@ u##bits btrfs_get_token_##bits(struct extent_buffer *eb, void *ptr, \
return res; \
} \
void btrfs_set_token_##bits(struct extent_buffer *eb, \
- void *ptr, unsigned long off, u##bits val, \
+ const void *ptr, unsigned long off, \
+ u##bits val, \
struct btrfs_map_token *token) \
{ \
unsigned long part_offset = (unsigned long)ptr; \
@@ -133,7 +134,7 @@ DEFINE_BTRFS_SETGET_BITS(16)
DEFINE_BTRFS_SETGET_BITS(32)
DEFINE_BTRFS_SETGET_BITS(64)
-void btrfs_node_key(struct extent_buffer *eb,
+void btrfs_node_key(const struct extent_buffer *eb,
struct btrfs_disk_key *disk_key, int nr)
{
unsigned long ptr = btrfs_node_key_ptr_offset(nr);
diff --git a/fs/btrfs/super.c b/fs/btrfs/super.c
index f6e1119..a7b69de 100644
--- a/fs/btrfs/super.c
+++ b/fs/btrfs/super.c
@@ -2226,6 +2226,7 @@ static long btrfs_control_ioctl(struct file *file, unsigned int cmd,
vol = memdup_user((void __user *)arg, sizeof(*vol));
if (IS_ERR(vol))
return PTR_ERR(vol);
+ vol->name[BTRFS_PATH_NAME_MAX] = '\0';
switch (cmd) {
case BTRFS_IOC_SCAN_DEV:
diff --git a/fs/btrfs/tests/qgroup-tests.c b/fs/btrfs/tests/qgroup-tests.c
index ca7cb5e..9c66666 100644
--- a/fs/btrfs/tests/qgroup-tests.c
+++ b/fs/btrfs/tests/qgroup-tests.c
@@ -63,7 +63,7 @@ static int insert_normal_tree_ref(struct btrfs_root *root, u64 bytenr,
btrfs_set_extent_generation(leaf, item, 1);
btrfs_set_extent_flags(leaf, item, BTRFS_EXTENT_FLAG_TREE_BLOCK);
block_info = (struct btrfs_tree_block_info *)(item + 1);
- btrfs_set_tree_block_level(leaf, block_info, 1);
+ btrfs_set_tree_block_level(leaf, block_info, 0);
iref = (struct btrfs_extent_inline_ref *)(block_info + 1);
if (parent > 0) {
btrfs_set_extent_inline_ref_type(leaf, iref,
diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index 9517de0..fd6c746 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -1924,6 +1924,9 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
return ret;
}
+ btrfs_trans_release_metadata(trans, root);
+ trans->block_rsv = NULL;
+
/* make a pass through all the delayed refs we have so far
* any runnings procs may add more while we are here
*/
@@ -1933,9 +1936,6 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans,
return ret;
}
- btrfs_trans_release_metadata(trans, root);
- trans->block_rsv = NULL;
-
cur_trans = trans->transaction;
/*
diff --git a/fs/btrfs/tree-checker.c b/fs/btrfs/tree-checker.c
new file mode 100644
index 0000000..7b69ba7
--- /dev/null
+++ b/fs/btrfs/tree-checker.c
@@ -0,0 +1,649 @@
+/*
+ * Copyright (C) Qu Wenruo 2017. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program.
+ */
+
+/*
+ * The module is used to catch unexpected/corrupted tree block data.
+ * Such behavior can be caused either by a fuzzed image or bugs.
+ *
+ * The objective is to do leaf/node validation checks when tree block is read
+ * from disk, and check *every* possible member, so other code won't
+ * need to checking them again.
+ *
+ * Due to the potential and unwanted damage, every checker needs to be
+ * carefully reviewed otherwise so it does not prevent mount of valid images.
+ */
+
+#include "ctree.h"
+#include "tree-checker.h"
+#include "disk-io.h"
+#include "compression.h"
+#include "hash.h"
+#include "volumes.h"
+
+#define CORRUPT(reason, eb, root, slot) \
+ btrfs_crit(root->fs_info, \
+ "corrupt %s, %s: block=%llu, root=%llu, slot=%d", \
+ btrfs_header_level(eb) == 0 ? "leaf" : "node", \
+ reason, btrfs_header_bytenr(eb), root->objectid, slot)
+
+/*
+ * Error message should follow the following format:
+ * corrupt <type>: <identifier>, <reason>[, <bad_value>]
+ *
+ * @type: leaf or node
+ * @identifier: the necessary info to locate the leaf/node.
+ * It's recommened to decode key.objecitd/offset if it's
+ * meaningful.
+ * @reason: describe the error
+ * @bad_value: optional, it's recommened to output bad value and its
+ * expected value (range).
+ *
+ * Since comma is used to separate the components, only space is allowed
+ * inside each component.
+ */
+
+/*
+ * Append generic "corrupt leaf/node root=%llu block=%llu slot=%d: " to @fmt.
+ * Allows callers to customize the output.
+ */
+__printf(4, 5)
+static void generic_err(const struct btrfs_root *root,
+ const struct extent_buffer *eb, int slot,
+ const char *fmt, ...)
+{
+ struct va_format vaf;
+ va_list args;
+
+ va_start(args, fmt);
+
+ vaf.fmt = fmt;
+ vaf.va = &args;
+
+ btrfs_crit(root->fs_info,
+ "corrupt %s: root=%llu block=%llu slot=%d, %pV",
+ btrfs_header_level(eb) == 0 ? "leaf" : "node",
+ root->objectid, btrfs_header_bytenr(eb), slot, &vaf);
+ va_end(args);
+}
+
+static int check_extent_data_item(struct btrfs_root *root,
+ struct extent_buffer *leaf,
+ struct btrfs_key *key, int slot)
+{
+ struct btrfs_file_extent_item *fi;
+ u32 sectorsize = root->sectorsize;
+ u32 item_size = btrfs_item_size_nr(leaf, slot);
+
+ if (!IS_ALIGNED(key->offset, sectorsize)) {
+ CORRUPT("unaligned key offset for file extent",
+ leaf, root, slot);
+ return -EUCLEAN;
+ }
+
+ fi = btrfs_item_ptr(leaf, slot, struct btrfs_file_extent_item);
+
+ if (btrfs_file_extent_type(leaf, fi) > BTRFS_FILE_EXTENT_TYPES) {
+ CORRUPT("invalid file extent type", leaf, root, slot);
+ return -EUCLEAN;
+ }
+
+ /*
+ * Support for new compression/encrption must introduce incompat flag,
+ * and must be caught in open_ctree().
+ */
+ if (btrfs_file_extent_compression(leaf, fi) > BTRFS_COMPRESS_TYPES) {
+ CORRUPT("invalid file extent compression", leaf, root, slot);
+ return -EUCLEAN;
+ }
+ if (btrfs_file_extent_encryption(leaf, fi)) {
+ CORRUPT("invalid file extent encryption", leaf, root, slot);
+ return -EUCLEAN;
+ }
+ if (btrfs_file_extent_type(leaf, fi) == BTRFS_FILE_EXTENT_INLINE) {
+ /* Inline extent must have 0 as key offset */
+ if (key->offset) {
+ CORRUPT("inline extent has non-zero key offset",
+ leaf, root, slot);
+ return -EUCLEAN;
+ }
+
+ /* Compressed inline extent has no on-disk size, skip it */
+ if (btrfs_file_extent_compression(leaf, fi) !=
+ BTRFS_COMPRESS_NONE)
+ return 0;
+
+ /* Uncompressed inline extent size must match item size */
+ if (item_size != BTRFS_FILE_EXTENT_INLINE_DATA_START +
+ btrfs_file_extent_ram_bytes(leaf, fi)) {
+ CORRUPT("plaintext inline extent has invalid size",
+ leaf, root, slot);
+ return -EUCLEAN;
+ }
+ return 0;
+ }
+
+ /* Regular or preallocated extent has fixed item size */
+ if (item_size != sizeof(*fi)) {
+ CORRUPT(
+ "regluar or preallocated extent data item size is invalid",
+ leaf, root, slot);
+ return -EUCLEAN;
+ }
+ if (!IS_ALIGNED(btrfs_file_extent_ram_bytes(leaf, fi), sectorsize) ||
+ !IS_ALIGNED(btrfs_file_extent_disk_bytenr(leaf, fi), sectorsize) ||
+ !IS_ALIGNED(btrfs_file_extent_disk_num_bytes(leaf, fi), sectorsize) ||
+ !IS_ALIGNED(btrfs_file_extent_offset(leaf, fi), sectorsize) ||
+ !IS_ALIGNED(btrfs_file_extent_num_bytes(leaf, fi), sectorsize)) {
+ CORRUPT(
+ "regular or preallocated extent data item has unaligned value",
+ leaf, root, slot);
+ return -EUCLEAN;
+ }
+
+ return 0;
+}
+
+static int check_csum_item(struct btrfs_root *root, struct extent_buffer *leaf,
+ struct btrfs_key *key, int slot)
+{
+ u32 sectorsize = root->sectorsize;
+ u32 csumsize = btrfs_super_csum_size(root->fs_info->super_copy);
+
+ if (key->objectid != BTRFS_EXTENT_CSUM_OBJECTID) {
+ CORRUPT("invalid objectid for csum item", leaf, root, slot);
+ return -EUCLEAN;
+ }
+ if (!IS_ALIGNED(key->offset, sectorsize)) {
+ CORRUPT("unaligned key offset for csum item", leaf, root, slot);
+ return -EUCLEAN;
+ }
+ if (!IS_ALIGNED(btrfs_item_size_nr(leaf, slot), csumsize)) {
+ CORRUPT("unaligned csum item size", leaf, root, slot);
+ return -EUCLEAN;
+ }
+ return 0;
+}
+
+/*
+ * Customized reported for dir_item, only important new info is key->objectid,
+ * which represents inode number
+ */
+__printf(4, 5)
+static void dir_item_err(const struct btrfs_root *root,
+ const struct extent_buffer *eb, int slot,
+ const char *fmt, ...)
+{
+ struct btrfs_key key;
+ struct va_format vaf;
+ va_list args;
+
+ btrfs_item_key_to_cpu(eb, &key, slot);
+ va_start(args, fmt);
+
+ vaf.fmt = fmt;
+ vaf.va = &args;
+
+ btrfs_crit(root->fs_info,
+ "corrupt %s: root=%llu block=%llu slot=%d ino=%llu, %pV",
+ btrfs_header_level(eb) == 0 ? "leaf" : "node", root->objectid,
+ btrfs_header_bytenr(eb), slot, key.objectid, &vaf);
+ va_end(args);
+}
+
+static int check_dir_item(struct btrfs_root *root,
+ struct extent_buffer *leaf,
+ struct btrfs_key *key, int slot)
+{
+ struct btrfs_dir_item *di;
+ u32 item_size = btrfs_item_size_nr(leaf, slot);
+ u32 cur = 0;
+
+ di = btrfs_item_ptr(leaf, slot, struct btrfs_dir_item);
+ while (cur < item_size) {
+ u32 name_len;
+ u32 data_len;
+ u32 max_name_len;
+ u32 total_size;
+ u32 name_hash;
+ u8 dir_type;
+
+ /* header itself should not cross item boundary */
+ if (cur + sizeof(*di) > item_size) {
+ dir_item_err(root, leaf, slot,
+ "dir item header crosses item boundary, have %zu boundary %u",
+ cur + sizeof(*di), item_size);
+ return -EUCLEAN;
+ }
+
+ /* dir type check */
+ dir_type = btrfs_dir_type(leaf, di);
+ if (dir_type >= BTRFS_FT_MAX) {
+ dir_item_err(root, leaf, slot,
+ "invalid dir item type, have %u expect [0, %u)",
+ dir_type, BTRFS_FT_MAX);
+ return -EUCLEAN;
+ }
+
+ if (key->type == BTRFS_XATTR_ITEM_KEY &&
+ dir_type != BTRFS_FT_XATTR) {
+ dir_item_err(root, leaf, slot,
+ "invalid dir item type for XATTR key, have %u expect %u",
+ dir_type, BTRFS_FT_XATTR);
+ return -EUCLEAN;
+ }
+ if (dir_type == BTRFS_FT_XATTR &&
+ key->type != BTRFS_XATTR_ITEM_KEY) {
+ dir_item_err(root, leaf, slot,
+ "xattr dir type found for non-XATTR key");
+ return -EUCLEAN;
+ }
+ if (dir_type == BTRFS_FT_XATTR)
+ max_name_len = XATTR_NAME_MAX;
+ else
+ max_name_len = BTRFS_NAME_LEN;
+
+ /* Name/data length check */
+ name_len = btrfs_dir_name_len(leaf, di);
+ data_len = btrfs_dir_data_len(leaf, di);
+ if (name_len > max_name_len) {
+ dir_item_err(root, leaf, slot,
+ "dir item name len too long, have %u max %u",
+ name_len, max_name_len);
+ return -EUCLEAN;
+ }
+ if (name_len + data_len > BTRFS_MAX_XATTR_SIZE(root)) {
+ dir_item_err(root, leaf, slot,
+ "dir item name and data len too long, have %u max %u",
+ name_len + data_len,
+ BTRFS_MAX_XATTR_SIZE(root));
+ return -EUCLEAN;
+ }
+
+ if (data_len && dir_type != BTRFS_FT_XATTR) {
+ dir_item_err(root, leaf, slot,
+ "dir item with invalid data len, have %u expect 0",
+ data_len);
+ return -EUCLEAN;
+ }
+
+ total_size = sizeof(*di) + name_len + data_len;
+
+ /* header and name/data should not cross item boundary */
+ if (cur + total_size > item_size) {
+ dir_item_err(root, leaf, slot,
+ "dir item data crosses item boundary, have %u boundary %u",
+ cur + total_size, item_size);
+ return -EUCLEAN;
+ }
+
+ /*
+ * Special check for XATTR/DIR_ITEM, as key->offset is name
+ * hash, should match its name
+ */
+ if (key->type == BTRFS_DIR_ITEM_KEY ||
+ key->type == BTRFS_XATTR_ITEM_KEY) {
+ char namebuf[max(BTRFS_NAME_LEN, XATTR_NAME_MAX)];
+
+ read_extent_buffer(leaf, namebuf,
+ (unsigned long)(di + 1), name_len);
+ name_hash = btrfs_name_hash(namebuf, name_len);
+ if (key->offset != name_hash) {
+ dir_item_err(root, leaf, slot,
+ "name hash mismatch with key, have 0x%016x expect 0x%016llx",
+ name_hash, key->offset);
+ return -EUCLEAN;
+ }
+ }
+ cur += total_size;
+ di = (struct btrfs_dir_item *)((void *)di + total_size);
+ }
+ return 0;
+}
+
+__printf(4, 5)
+__cold
+static void block_group_err(const struct btrfs_fs_info *fs_info,
+ const struct extent_buffer *eb, int slot,
+ const char *fmt, ...)
+{
+ struct btrfs_key key;
+ struct va_format vaf;
+ va_list args;
+
+ btrfs_item_key_to_cpu(eb, &key, slot);
+ va_start(args, fmt);
+
+ vaf.fmt = fmt;
+ vaf.va = &args;
+
+ btrfs_crit(fs_info,
+ "corrupt %s: root=%llu block=%llu slot=%d bg_start=%llu bg_len=%llu, %pV",
+ btrfs_header_level(eb) == 0 ? "leaf" : "node",
+ btrfs_header_owner(eb), btrfs_header_bytenr(eb), slot,
+ key.objectid, key.offset, &vaf);
+ va_end(args);
+}
+
+static int check_block_group_item(struct btrfs_fs_info *fs_info,
+ struct extent_buffer *leaf,
+ struct btrfs_key *key, int slot)
+{
+ struct btrfs_block_group_item bgi;
+ u32 item_size = btrfs_item_size_nr(leaf, slot);
+ u64 flags;
+ u64 type;
+
+ /*
+ * Here we don't really care about alignment since extent allocator can
+ * handle it. We care more about the size, as if one block group is
+ * larger than maximum size, it's must be some obvious corruption.
+ */
+ if (key->offset > BTRFS_MAX_DATA_CHUNK_SIZE || key->offset == 0) {
+ block_group_err(fs_info, leaf, slot,
+ "invalid block group size, have %llu expect (0, %llu]",
+ key->offset, BTRFS_MAX_DATA_CHUNK_SIZE);
+ return -EUCLEAN;
+ }
+
+ if (item_size != sizeof(bgi)) {
+ block_group_err(fs_info, leaf, slot,
+ "invalid item size, have %u expect %zu",
+ item_size, sizeof(bgi));
+ return -EUCLEAN;
+ }
+
+ read_extent_buffer(leaf, &bgi, btrfs_item_ptr_offset(leaf, slot),
+ sizeof(bgi));
+ if (btrfs_block_group_chunk_objectid(&bgi) !=
+ BTRFS_FIRST_CHUNK_TREE_OBJECTID) {
+ block_group_err(fs_info, leaf, slot,
+ "invalid block group chunk objectid, have %llu expect %llu",
+ btrfs_block_group_chunk_objectid(&bgi),
+ BTRFS_FIRST_CHUNK_TREE_OBJECTID);
+ return -EUCLEAN;
+ }
+
+ if (btrfs_block_group_used(&bgi) > key->offset) {
+ block_group_err(fs_info, leaf, slot,
+ "invalid block group used, have %llu expect [0, %llu)",
+ btrfs_block_group_used(&bgi), key->offset);
+ return -EUCLEAN;
+ }
+
+ flags = btrfs_block_group_flags(&bgi);
+ if (hweight64(flags & BTRFS_BLOCK_GROUP_PROFILE_MASK) > 1) {
+ block_group_err(fs_info, leaf, slot,
+"invalid profile flags, have 0x%llx (%lu bits set) expect no more than 1 bit set",
+ flags & BTRFS_BLOCK_GROUP_PROFILE_MASK,
+ hweight64(flags & BTRFS_BLOCK_GROUP_PROFILE_MASK));
+ return -EUCLEAN;
+ }
+
+ type = flags & BTRFS_BLOCK_GROUP_TYPE_MASK;
+ if (type != BTRFS_BLOCK_GROUP_DATA &&
+ type != BTRFS_BLOCK_GROUP_METADATA &&
+ type != BTRFS_BLOCK_GROUP_SYSTEM &&
+ type != (BTRFS_BLOCK_GROUP_METADATA |
+ BTRFS_BLOCK_GROUP_DATA)) {
+ block_group_err(fs_info, leaf, slot,
+"invalid type, have 0x%llx (%lu bits set) expect either 0x%llx, 0x%llx, 0x%llx or 0x%llx",
+ type, hweight64(type),
+ BTRFS_BLOCK_GROUP_DATA, BTRFS_BLOCK_GROUP_METADATA,
+ BTRFS_BLOCK_GROUP_SYSTEM,
+ BTRFS_BLOCK_GROUP_METADATA | BTRFS_BLOCK_GROUP_DATA);
+ return -EUCLEAN;
+ }
+ return 0;
+}
+
+/*
+ * Common point to switch the item-specific validation.
+ */
+static int check_leaf_item(struct btrfs_root *root,
+ struct extent_buffer *leaf,
+ struct btrfs_key *key, int slot)
+{
+ int ret = 0;
+
+ switch (key->type) {
+ case BTRFS_EXTENT_DATA_KEY:
+ ret = check_extent_data_item(root, leaf, key, slot);
+ break;
+ case BTRFS_EXTENT_CSUM_KEY:
+ ret = check_csum_item(root, leaf, key, slot);
+ break;
+ case BTRFS_DIR_ITEM_KEY:
+ case BTRFS_DIR_INDEX_KEY:
+ case BTRFS_XATTR_ITEM_KEY:
+ ret = check_dir_item(root, leaf, key, slot);
+ break;
+ case BTRFS_BLOCK_GROUP_ITEM_KEY:
+ ret = check_block_group_item(root->fs_info, leaf, key, slot);
+ break;
+ }
+ return ret;
+}
+
+static int check_leaf(struct btrfs_root *root, struct extent_buffer *leaf,
+ bool check_item_data)
+{
+ struct btrfs_fs_info *fs_info = root->fs_info;
+ /* No valid key type is 0, so all key should be larger than this key */
+ struct btrfs_key prev_key = {0, 0, 0};
+ struct btrfs_key key;
+ u32 nritems = btrfs_header_nritems(leaf);
+ int slot;
+
+ if (btrfs_header_level(leaf) != 0) {
+ generic_err(root, leaf, 0,
+ "invalid level for leaf, have %d expect 0",
+ btrfs_header_level(leaf));
+ return -EUCLEAN;
+ }
+
+ /*
+ * Extent buffers from a relocation tree have a owner field that
+ * corresponds to the subvolume tree they are based on. So just from an
+ * extent buffer alone we can not find out what is the id of the
+ * corresponding subvolume tree, so we can not figure out if the extent
+ * buffer corresponds to the root of the relocation tree or not. So
+ * skip this check for relocation trees.
+ */
+ if (nritems == 0 && !btrfs_header_flag(leaf, BTRFS_HEADER_FLAG_RELOC)) {
+ u64 owner = btrfs_header_owner(leaf);
+ struct btrfs_root *check_root;
+
+ /* These trees must never be empty */
+ if (owner == BTRFS_ROOT_TREE_OBJECTID ||
+ owner == BTRFS_CHUNK_TREE_OBJECTID ||
+ owner == BTRFS_EXTENT_TREE_OBJECTID ||
+ owner == BTRFS_DEV_TREE_OBJECTID ||
+ owner == BTRFS_FS_TREE_OBJECTID ||
+ owner == BTRFS_DATA_RELOC_TREE_OBJECTID) {
+ generic_err(root, leaf, 0,
+ "invalid root, root %llu must never be empty",
+ owner);
+ return -EUCLEAN;
+ }
+ key.objectid = owner;
+ key.type = BTRFS_ROOT_ITEM_KEY;
+ key.offset = (u64)-1;
+
+ check_root = btrfs_get_fs_root(fs_info, &key, false);
+ /*
+ * The only reason we also check NULL here is that during
+ * open_ctree() some roots has not yet been set up.
+ */
+ if (!IS_ERR_OR_NULL(check_root)) {
+ struct extent_buffer *eb;
+
+ eb = btrfs_root_node(check_root);
+ /* if leaf is the root, then it's fine */
+ if (leaf != eb) {
+ CORRUPT("non-root leaf's nritems is 0",
+ leaf, check_root, 0);
+ free_extent_buffer(eb);
+ return -EUCLEAN;
+ }
+ free_extent_buffer(eb);
+ }
+ return 0;
+ }
+
+ if (nritems == 0)
+ return 0;
+
+ /*
+ * Check the following things to make sure this is a good leaf, and
+ * leaf users won't need to bother with similar sanity checks:
+ *
+ * 1) key ordering
+ * 2) item offset and size
+ * No overlap, no hole, all inside the leaf.
+ * 3) item content
+ * If possible, do comprehensive sanity check.
+ * NOTE: All checks must only rely on the item data itself.
+ */
+ for (slot = 0; slot < nritems; slot++) {
+ u32 item_end_expected;
+ int ret;
+
+ btrfs_item_key_to_cpu(leaf, &key, slot);
+
+ /* Make sure the keys are in the right order */
+ if (btrfs_comp_cpu_keys(&prev_key, &key) >= 0) {
+ CORRUPT("bad key order", leaf, root, slot);
+ return -EUCLEAN;
+ }
+
+ /*
+ * Make sure the offset and ends are right, remember that the
+ * item data starts at the end of the leaf and grows towards the
+ * front.
+ */
+ if (slot == 0)
+ item_end_expected = BTRFS_LEAF_DATA_SIZE(root);
+ else
+ item_end_expected = btrfs_item_offset_nr(leaf,
+ slot - 1);
+ if (btrfs_item_end_nr(leaf, slot) != item_end_expected) {
+ CORRUPT("slot offset bad", leaf, root, slot);
+ return -EUCLEAN;
+ }
+
+ /*
+ * Check to make sure that we don't point outside of the leaf,
+ * just in case all the items are consistent to each other, but
+ * all point outside of the leaf.
+ */
+ if (btrfs_item_end_nr(leaf, slot) >
+ BTRFS_LEAF_DATA_SIZE(root)) {
+ CORRUPT("slot end outside of leaf", leaf, root, slot);
+ return -EUCLEAN;
+ }
+
+ /* Also check if the item pointer overlaps with btrfs item. */
+ if (btrfs_item_nr_offset(slot) + sizeof(struct btrfs_item) >
+ btrfs_item_ptr_offset(leaf, slot)) {
+ CORRUPT("slot overlap with its data", leaf, root, slot);
+ return -EUCLEAN;
+ }
+
+ if (check_item_data) {
+ /*
+ * Check if the item size and content meet other
+ * criteria
+ */
+ ret = check_leaf_item(root, leaf, &key, slot);
+ if (ret < 0)
+ return ret;
+ }
+
+ prev_key.objectid = key.objectid;
+ prev_key.type = key.type;
+ prev_key.offset = key.offset;
+ }
+
+ return 0;
+}
+
+int btrfs_check_leaf_full(struct btrfs_root *root, struct extent_buffer *leaf)
+{
+ return check_leaf(root, leaf, true);
+}
+
+int btrfs_check_leaf_relaxed(struct btrfs_root *root,
+ struct extent_buffer *leaf)
+{
+ return check_leaf(root, leaf, false);
+}
+
+int btrfs_check_node(struct btrfs_root *root, struct extent_buffer *node)
+{
+ unsigned long nr = btrfs_header_nritems(node);
+ struct btrfs_key key, next_key;
+ int slot;
+ int level = btrfs_header_level(node);
+ u64 bytenr;
+ int ret = 0;
+
+ if (level <= 0 || level >= BTRFS_MAX_LEVEL) {
+ generic_err(root, node, 0,
+ "invalid level for node, have %d expect [1, %d]",
+ level, BTRFS_MAX_LEVEL - 1);
+ return -EUCLEAN;
+ }
+ if (nr == 0 || nr > BTRFS_NODEPTRS_PER_BLOCK(root)) {
+ btrfs_crit(root->fs_info,
+"corrupt node: root=%llu block=%llu, nritems too %s, have %lu expect range [1,%u]",
+ root->objectid, node->start,
+ nr == 0 ? "small" : "large", nr,
+ BTRFS_NODEPTRS_PER_BLOCK(root));
+ return -EUCLEAN;
+ }
+
+ for (slot = 0; slot < nr - 1; slot++) {
+ bytenr = btrfs_node_blockptr(node, slot);
+ btrfs_node_key_to_cpu(node, &key, slot);
+ btrfs_node_key_to_cpu(node, &next_key, slot + 1);
+
+ if (!bytenr) {
+ generic_err(root, node, slot,
+ "invalid NULL node pointer");
+ ret = -EUCLEAN;
+ goto out;
+ }
+ if (!IS_ALIGNED(bytenr, root->sectorsize)) {
+ generic_err(root, node, slot,
+ "unaligned pointer, have %llu should be aligned to %u",
+ bytenr, root->sectorsize);
+ ret = -EUCLEAN;
+ goto out;
+ }
+
+ if (btrfs_comp_cpu_keys(&key, &next_key) >= 0) {
+ generic_err(root, node, slot,
+ "bad key order, current (%llu %u %llu) next (%llu %u %llu)",
+ key.objectid, key.type, key.offset,
+ next_key.objectid, next_key.type,
+ next_key.offset);
+ ret = -EUCLEAN;
+ goto out;
+ }
+ }
+out:
+ return ret;
+}
diff --git a/fs/btrfs/tree-checker.h b/fs/btrfs/tree-checker.h
new file mode 100644
index 0000000..3d53e8d
--- /dev/null
+++ b/fs/btrfs/tree-checker.h
@@ -0,0 +1,38 @@
+/*
+ * Copyright (C) Qu Wenruo 2017. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public
+ * License v2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public
+ * License along with this program.
+ */
+
+#ifndef __BTRFS_TREE_CHECKER__
+#define __BTRFS_TREE_CHECKER__
+
+#include "ctree.h"
+#include "extent_io.h"
+
+/*
+ * Comprehensive leaf checker.
+ * Will check not only the item pointers, but also every possible member
+ * in item data.
+ */
+int btrfs_check_leaf_full(struct btrfs_root *root, struct extent_buffer *leaf);
+
+/*
+ * Less strict leaf checker.
+ * Will only check item pointers, not reading item data.
+ */
+int btrfs_check_leaf_relaxed(struct btrfs_root *root,
+ struct extent_buffer *leaf);
+int btrfs_check_node(struct btrfs_root *root, struct extent_buffer *node);
+
+#endif
diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index 5240173..47d11a3 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -2241,8 +2241,10 @@ static noinline int replay_dir_deletes(struct btrfs_trans_handle *trans,
nritems = btrfs_header_nritems(path->nodes[0]);
if (path->slots[0] >= nritems) {
ret = btrfs_next_leaf(root, path);
- if (ret)
+ if (ret == 1)
break;
+ else if (ret < 0)
+ goto out;
}
btrfs_item_key_to_cpu(path->nodes[0], &found_key,
path->slots[0]);
@@ -2977,8 +2979,11 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
mutex_unlock(&log_root_tree->log_mutex);
/*
- * The barrier before waitqueue_active is implied by mutex_unlock
+ * The barrier before waitqueue_active is needed so all the updates
+ * above are seen by the woken threads. It might not be necessary, but
+ * proving that seems to be hard.
*/
+ smp_mb();
if (waitqueue_active(&log_root_tree->log_commit_wait[index2]))
wake_up(&log_root_tree->log_commit_wait[index2]);
out:
@@ -2989,8 +2994,11 @@ int btrfs_sync_log(struct btrfs_trans_handle *trans,
mutex_unlock(&root->log_mutex);
/*
- * The barrier before waitqueue_active is implied by mutex_unlock
+ * The barrier before waitqueue_active is needed so all the updates
+ * above are seen by the woken threads. It might not be necessary, but
+ * proving that seems to be hard.
*/
+ smp_mb();
if (waitqueue_active(&root->log_commit_wait[index1]))
wake_up(&root->log_commit_wait[index1]);
return ret;
@@ -3008,9 +3016,12 @@ static void free_log_tree(struct btrfs_trans_handle *trans,
};
ret = walk_log_tree(trans, log, &wc);
- /* I don't think this can happen but just in case */
- if (ret)
- btrfs_abort_transaction(trans, ret);
+ if (ret) {
+ if (trans)
+ btrfs_abort_transaction(trans, ret);
+ else
+ btrfs_handle_fs_error(log->fs_info, ret, NULL);
+ }
while (1) {
ret = find_first_extent_bit(&log->dirty_log_pages,
@@ -3397,8 +3408,11 @@ static noinline int log_dir_items(struct btrfs_trans_handle *trans,
* from this directory and from this transaction
*/
ret = btrfs_next_leaf(root, path);
- if (ret == 1) {
- last_offset = (u64)-1;
+ if (ret) {
+ if (ret == 1)
+ last_offset = (u64)-1;
+ else
+ err = ret;
goto done;
}
btrfs_item_key_to_cpu(path->nodes[0], &tmp, path->slots[0]);
@@ -3849,6 +3863,7 @@ static noinline int copy_items(struct btrfs_trans_handle *trans,
ASSERT(ret == 0);
src = src_path->nodes[0];
i = 0;
+ need_find_last_extent = true;
}
btrfs_item_key_to_cpu(src, &key, i);
@@ -4614,6 +4629,7 @@ static int btrfs_log_inode(struct btrfs_trans_handle *trans,
struct extent_map_tree *em_tree = &BTRFS_I(inode)->extent_tree;
u64 logged_isize = 0;
bool need_log_inode_item = true;
+ bool xattrs_logged = false;
path = btrfs_alloc_path();
if (!path)
@@ -4918,6 +4934,7 @@ static int btrfs_log_inode(struct btrfs_trans_handle *trans,
err = btrfs_log_all_xattrs(trans, root, inode, path, dst_path);
if (err)
goto out_unlock;
+ xattrs_logged = true;
if (max_key.type >= BTRFS_EXTENT_DATA_KEY && !fast_search) {
btrfs_release_path(path);
btrfs_release_path(dst_path);
@@ -4930,6 +4947,11 @@ static int btrfs_log_inode(struct btrfs_trans_handle *trans,
btrfs_release_path(dst_path);
if (need_log_inode_item) {
err = log_inode_item(trans, log, dst_path, inode);
+ if (!err && !xattrs_logged) {
+ err = btrfs_log_all_xattrs(trans, root, inode, path,
+ dst_path);
+ btrfs_release_path(path);
+ }
if (err)
goto out_unlock;
}
@@ -5351,9 +5373,33 @@ static int btrfs_log_all_parents(struct btrfs_trans_handle *trans,
dir_inode = btrfs_iget(root->fs_info->sb, &inode_key,
root, NULL);
- /* If parent inode was deleted, skip it. */
- if (IS_ERR(dir_inode))
- continue;
+ /*
+ * If the parent inode was deleted, return an error to
+ * fallback to a transaction commit. This is to prevent
+ * getting an inode that was moved from one parent A to
+ * a parent B, got its former parent A deleted and then
+ * it got fsync'ed, from existing at both parents after
+ * a log replay (and the old parent still existing).
+ * Example:
+ *
+ * mkdir /mnt/A
+ * mkdir /mnt/B
+ * touch /mnt/B/bar
+ * sync
+ * mv /mnt/B/bar /mnt/A/bar
+ * mv -T /mnt/A /mnt/B
+ * fsync /mnt/B/bar
+ * <power fail>
+ *
+ * If we ignore the old parent B which got deleted,
+ * after a log replay we would have file bar linked
+ * at both parents and the old parent B would still
+ * exist.
+ */
+ if (IS_ERR(dir_inode)) {
+ ret = PTR_ERR(dir_inode);
+ goto out;
+ }
if (ctx)
ctx->log_new_dentries = false;
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index 39b917b7..1fe447e 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -3966,6 +3966,15 @@ int btrfs_resume_balance_async(struct btrfs_fs_info *fs_info)
return 0;
}
+ /*
+ * A ro->rw remount sequence should continue with the paused balance
+ * regardless of who pauses it, system or the user as of now, so set
+ * the resume flag.
+ */
+ spin_lock(&fs_info->balance_lock);
+ fs_info->balance_ctl->flags |= BTRFS_BALANCE_RESUME;
+ spin_unlock(&fs_info->balance_lock);
+
tsk = kthread_run(balance_kthread, fs_info, "btrfs-balance");
return PTR_ERR_OR_ZERO(tsk);
}
@@ -4647,7 +4656,7 @@ static int __btrfs_alloc_chunk(struct btrfs_trans_handle *trans,
if (type & BTRFS_BLOCK_GROUP_DATA) {
max_stripe_size = SZ_1G;
- max_chunk_size = 10 * max_stripe_size;
+ max_chunk_size = BTRFS_MAX_DATA_CHUNK_SIZE;
if (!devs_max)
devs_max = BTRFS_MAX_DEVS(info->chunk_root);
} else if (type & BTRFS_BLOCK_GROUP_METADATA) {
@@ -5177,7 +5186,14 @@ int btrfs_num_copies(struct btrfs_fs_info *fs_info, u64 logical, u64 len)
else if (map->type & BTRFS_BLOCK_GROUP_RAID5)
ret = 2;
else if (map->type & BTRFS_BLOCK_GROUP_RAID6)
- ret = 3;
+ /*
+ * There could be two corrupted data stripes, we need
+ * to loop retry in order to rebuild the correct data.
+ *
+ * Fail a stripe at a time on every retry except the
+ * stripe under reconstruction.
+ */
+ ret = map->num_stripes;
else
ret = 1;
free_extent_map(em);
@@ -6354,6 +6370,8 @@ static int btrfs_check_chunk_valid(struct btrfs_root *root,
u16 num_stripes;
u16 sub_stripes;
u64 type;
+ u64 features;
+ bool mixed = false;
length = btrfs_chunk_length(leaf, chunk);
stripe_len = btrfs_chunk_stripe_len(leaf, chunk);
@@ -6394,6 +6412,32 @@ static int btrfs_check_chunk_valid(struct btrfs_root *root,
btrfs_chunk_type(leaf, chunk));
return -EIO;
}
+
+ if ((type & BTRFS_BLOCK_GROUP_TYPE_MASK) == 0) {
+ btrfs_err(root->fs_info, "missing chunk type flag: 0x%llx", type);
+ return -EIO;
+ }
+
+ if ((type & BTRFS_BLOCK_GROUP_SYSTEM) &&
+ (type & (BTRFS_BLOCK_GROUP_METADATA | BTRFS_BLOCK_GROUP_DATA))) {
+ btrfs_err(root->fs_info,
+ "system chunk with data or metadata type: 0x%llx", type);
+ return -EIO;
+ }
+
+ features = btrfs_super_incompat_flags(root->fs_info->super_copy);
+ if (features & BTRFS_FEATURE_INCOMPAT_MIXED_GROUPS)
+ mixed = true;
+
+ if (!mixed) {
+ if ((type & BTRFS_BLOCK_GROUP_METADATA) &&
+ (type & BTRFS_BLOCK_GROUP_DATA)) {
+ btrfs_err(root->fs_info,
+ "mixed chunk type in non-mixed mode: 0x%llx", type);
+ return -EIO;
+ }
+ }
+
if ((type & BTRFS_BLOCK_GROUP_RAID10 && sub_stripes != 2) ||
(type & BTRFS_BLOCK_GROUP_RAID1 && num_stripes < 1) ||
(type & BTRFS_BLOCK_GROUP_RAID5 && num_stripes < 2) ||
diff --git a/fs/btrfs/volumes.h b/fs/btrfs/volumes.h
index 09ed29c..9c09aa2 100644
--- a/fs/btrfs/volumes.h
+++ b/fs/btrfs/volumes.h
@@ -24,6 +24,8 @@
#include <linux/btrfs.h>
#include "async-thread.h"
+#define BTRFS_MAX_DATA_CHUNK_SIZE (10ULL * SZ_1G)
+
extern struct mutex uuid_mutex;
#define BTRFS_STRIPE_LEN SZ_64K
diff --git a/fs/cachefiles/namei.c b/fs/cachefiles/namei.c
index 41df8a2..3fb95e3 100644
--- a/fs/cachefiles/namei.c
+++ b/fs/cachefiles/namei.c
@@ -195,7 +195,6 @@ static int cachefiles_mark_object_active(struct cachefiles_cache *cache,
pr_err("\n");
pr_err("Error: Unexpected object collision\n");
cachefiles_printk_object(object, xobject);
- BUG();
}
atomic_inc(&xobject->usage);
write_unlock(&cache->active_lock);
@@ -341,7 +340,7 @@ static int cachefiles_bury_object(struct cachefiles_cache *cache,
trap = lock_rename(cache->graveyard, dir);
/* do some checks before getting the grave dentry */
- if (rep->d_parent != dir) {
+ if (rep->d_parent != dir || IS_DEADDIR(d_inode(rep))) {
/* the entry was probably culled when we dropped the parent dir
* lock */
unlock_rename(cache->graveyard, dir);
diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c
index afbdc41..799b59d 100644
--- a/fs/cachefiles/rdwr.c
+++ b/fs/cachefiles/rdwr.c
@@ -27,6 +27,7 @@ static int cachefiles_read_waiter(wait_queue_t *wait, unsigned mode,
struct cachefiles_one_read *monitor =
container_of(wait, struct cachefiles_one_read, monitor);
struct cachefiles_object *object;
+ struct fscache_retrieval *op = monitor->op;
struct wait_bit_key *key = _key;
struct page *page = wait->private;
@@ -51,16 +52,22 @@ static int cachefiles_read_waiter(wait_queue_t *wait, unsigned mode,
list_del(&wait->task_list);
/* move onto the action list and queue for FS-Cache thread pool */
- ASSERT(monitor->op);
+ ASSERT(op);
- object = container_of(monitor->op->op.object,
- struct cachefiles_object, fscache);
+ /* We need to temporarily bump the usage count as we don't own a ref
+ * here otherwise cachefiles_read_copier() may free the op between the
+ * monitor being enqueued on the op->to_do list and the op getting
+ * enqueued on the work queue.
+ */
+ fscache_get_retrieval(op);
+ object = container_of(op->op.object, struct cachefiles_object, fscache);
spin_lock(&object->work_lock);
- list_add_tail(&monitor->op_link, &monitor->op->to_do);
+ list_add_tail(&monitor->op_link, &op->to_do);
spin_unlock(&object->work_lock);
- fscache_enqueue_retrieval(monitor->op);
+ fscache_enqueue_retrieval(op);
+ fscache_put_retrieval(op);
return 0;
}
@@ -530,7 +537,10 @@ static int cachefiles_read_backing_file(struct cachefiles_object *object,
netpage->index, cachefiles_gfp);
if (ret < 0) {
if (ret == -EEXIST) {
+ put_page(backpage);
+ backpage = NULL;
put_page(netpage);
+ netpage = NULL;
fscache_retrieval_complete(op, 1);
continue;
}
@@ -603,7 +613,10 @@ static int cachefiles_read_backing_file(struct cachefiles_object *object,
netpage->index, cachefiles_gfp);
if (ret < 0) {
if (ret == -EEXIST) {
+ put_page(backpage);
+ backpage = NULL;
put_page(netpage);
+ netpage = NULL;
fscache_retrieval_complete(op, 1);
continue;
}
@@ -956,11 +969,8 @@ int cachefiles_write_page(struct fscache_storage *op, struct page *page)
void cachefiles_uncache_page(struct fscache_object *_object, struct page *page)
{
struct cachefiles_object *object;
- struct cachefiles_cache *cache;
object = container_of(_object, struct cachefiles_object, fscache);
- cache = container_of(object->fscache.cache,
- struct cachefiles_cache, cache);
_enter("%p,{%lu}", object, page->index);
diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c
index 7b79a54..546d643 100644
--- a/fs/ceph/addr.c
+++ b/fs/ceph/addr.c
@@ -838,21 +838,15 @@ static int ceph_writepages_start(struct address_space *mapping,
struct page **pages = NULL, **data_pages;
mempool_t *pool = NULL; /* Becomes non-null if mempool used */
struct page *page;
- int want;
u64 offset = 0, len = 0;
max_pages = max_pages_ever;
get_more_pages:
first = -1;
- want = min(end - index,
- min((pgoff_t)PAGEVEC_SIZE,
- max_pages - (pgoff_t)locked_pages) - 1)
- + 1;
- pvec_pages = pagevec_lookup_tag(&pvec, mapping, &index,
- PAGECACHE_TAG_DIRTY,
- want);
- dout("pagevec_lookup_tag got %d\n", pvec_pages);
+ pvec_pages = pagevec_lookup_range_tag(&pvec, mapping, &index,
+ end, PAGECACHE_TAG_DIRTY);
+ dout("pagevec_lookup_range_tag got %d\n", pvec_pages);
if (!pvec_pages && !locked_pages)
break;
for (i = 0; i < pvec_pages && locked_pages < max_pages; i++) {
@@ -870,12 +864,6 @@ static int ceph_writepages_start(struct address_space *mapping,
unlock_page(page);
break;
}
- if (!wbc->range_cyclic && page->index > end) {
- dout("end of range %p\n", page);
- done = 1;
- unlock_page(page);
- break;
- }
if (strip_unit_end && (page->index > strip_unit_end)) {
dout("end of strip unit %p\n", page);
unlock_page(page);
diff --git a/fs/ceph/caps.c b/fs/ceph/caps.c
index 3e1c136..f916cd7b 100644
--- a/fs/ceph/caps.c
+++ b/fs/ceph/caps.c
@@ -3343,7 +3343,6 @@ static void handle_cap_export(struct inode *inode, struct ceph_mds_caps *ex,
tcap->cap_id = t_cap_id;
tcap->seq = t_seq - 1;
tcap->issue_seq = t_seq - 1;
- tcap->mseq = t_mseq;
tcap->issued |= issued;
tcap->implemented |= issued;
if (cap == ci->i_auth_cap)
diff --git a/fs/ceph/inode.c b/fs/ceph/inode.c
index 4a6df2c..30d9d9e 100644
--- a/fs/ceph/inode.c
+++ b/fs/ceph/inode.c
@@ -1077,7 +1077,12 @@ static struct dentry *splice_dentry(struct dentry *dn, struct inode *in)
if (IS_ERR(realdn)) {
pr_err("splice_dentry error %ld %p inode %p ino %llx.%llx\n",
PTR_ERR(realdn), dn, in, ceph_vinop(in));
- dn = realdn; /* note realdn contains the error */
+ dn = realdn;
+ /*
+ * Caller should release 'dn' in the case of error.
+ * If 'req->r_dentry' is passed to this function,
+ * caller should leave 'req->r_dentry' untouched.
+ */
goto out;
} else if (realdn) {
dout("dn %p (%d) spliced with %p (%d) "
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 3d2639c..6cbd0d8 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -3983,14 +3983,24 @@ static struct ceph_auth_handshake *get_authorizer(struct ceph_connection *con,
return auth;
}
-
-static int verify_authorizer_reply(struct ceph_connection *con, int len)
+static int add_authorizer_challenge(struct ceph_connection *con,
+ void *challenge_buf, int challenge_buf_len)
{
struct ceph_mds_session *s = con->private;
struct ceph_mds_client *mdsc = s->s_mdsc;
struct ceph_auth_client *ac = mdsc->fsc->client->monc.auth;
- return ceph_auth_verify_authorizer_reply(ac, s->s_auth.authorizer, len);
+ return ceph_auth_add_authorizer_challenge(ac, s->s_auth.authorizer,
+ challenge_buf, challenge_buf_len);
+}
+
+static int verify_authorizer_reply(struct ceph_connection *con)
+{
+ struct ceph_mds_session *s = con->private;
+ struct ceph_mds_client *mdsc = s->s_mdsc;
+ struct ceph_auth_client *ac = mdsc->fsc->client->monc.auth;
+
+ return ceph_auth_verify_authorizer_reply(ac, s->s_auth.authorizer);
}
static int invalidate_authorizer(struct ceph_connection *con)
@@ -4046,6 +4056,7 @@ static const struct ceph_connection_operations mds_con_ops = {
.put = con_put,
.dispatch = dispatch,
.get_authorizer = get_authorizer,
+ .add_authorizer_challenge = add_authorizer_challenge,
.verify_authorizer_reply = verify_authorizer_reply,
.invalidate_authorizer = invalidate_authorizer,
.peer_reset = peer_reset,
diff --git a/fs/ceph/super.c b/fs/ceph/super.c
index b382e59..2a89030 100644
--- a/fs/ceph/super.c
+++ b/fs/ceph/super.c
@@ -816,7 +816,6 @@ static struct dentry *ceph_real_mount(struct ceph_fs_client *fsc)
int err;
unsigned long started = jiffies; /* note the start time */
struct dentry *root;
- int first = 0; /* first vfsmount for this super_block */
dout("mount start %p\n", fsc);
mutex_lock(&fsc->client->mount_mutex);
@@ -834,17 +833,17 @@ static struct dentry *ceph_real_mount(struct ceph_fs_client *fsc)
path = fsc->mount_options->server_path + 1;
dout("mount opening path %s\n", path);
}
+
+ err = ceph_fs_debugfs_init(fsc);
+ if (err < 0)
+ goto out;
+
root = open_root_dentry(fsc, path, started);
if (IS_ERR(root)) {
err = PTR_ERR(root);
goto out;
}
fsc->sb->s_root = dget(root);
- first = 1;
-
- err = ceph_fs_debugfs_init(fsc);
- if (err < 0)
- goto fail;
} else {
root = dget(fsc->sb->s_root);
}
@@ -854,11 +853,6 @@ static struct dentry *ceph_real_mount(struct ceph_fs_client *fsc)
mutex_unlock(&fsc->client->mount_mutex);
return root;
-fail:
- if (first) {
- dput(fsc->sb->s_root);
- fsc->sb->s_root = NULL;
- }
out:
mutex_unlock(&fsc->client->mount_mutex);
return ERR_PTR(err);
diff --git a/fs/cifs/Kconfig b/fs/cifs/Kconfig
index e7b478b..8bef27b 100644
--- a/fs/cifs/Kconfig
+++ b/fs/cifs/Kconfig
@@ -111,7 +111,7 @@
config CIFS_POSIX
bool "CIFS POSIX Extensions"
- depends on CIFS_XATTR
+ depends on CIFS && CIFS_ALLOW_INSECURE_LEGACY && CIFS_XATTR
help
Enabling this option will cause the cifs client to attempt to
negotiate a newer dialect with servers, such as Samba 3.0.5
diff --git a/fs/cifs/cifs_debug.c b/fs/cifs/cifs_debug.c
index 3d03e48..98fc77d 100644
--- a/fs/cifs/cifs_debug.c
+++ b/fs/cifs/cifs_debug.c
@@ -123,25 +123,41 @@ static int cifs_debug_data_proc_show(struct seq_file *m, void *v)
seq_printf(m, "CIFS Version %s\n", CIFS_VERSION);
seq_printf(m, "Features:");
#ifdef CONFIG_CIFS_DFS_UPCALL
- seq_printf(m, " dfs");
+ seq_printf(m, " DFS");
#endif
#ifdef CONFIG_CIFS_FSCACHE
- seq_printf(m, " fscache");
+ seq_printf(m, ",FSCACHE");
+#endif
+#ifdef CONFIG_CIFS_SMB_DIRECT
+ seq_printf(m, ",SMB_DIRECT");
+#endif
+#ifdef CONFIG_CIFS_STATS2
+ seq_printf(m, ",STATS2");
+#elif defined(CONFIG_CIFS_STATS)
+ seq_printf(m, ",STATS");
+#endif
+#ifdef CONFIG_CIFS_DEBUG2
+ seq_printf(m, ",DEBUG2");
+#elif defined(CONFIG_CIFS_DEBUG)
+ seq_printf(m, ",DEBUG");
+#endif
+#ifdef CONFIG_CIFS_ALLOW_INSECURE_LEGACY
+ seq_printf(m, ",ALLOW_INSECURE_LEGACY");
#endif
#ifdef CONFIG_CIFS_WEAK_PW_HASH
- seq_printf(m, " lanman");
+ seq_printf(m, ",WEAK_PW_HASH");
#endif
#ifdef CONFIG_CIFS_POSIX
- seq_printf(m, " posix");
+ seq_printf(m, ",CIFS_POSIX");
#endif
#ifdef CONFIG_CIFS_UPCALL
- seq_printf(m, " spnego");
+ seq_printf(m, ",UPCALL(SPNEGO)");
#endif
#ifdef CONFIG_CIFS_XATTR
- seq_printf(m, " xattr");
+ seq_printf(m, ",XATTR");
#endif
#ifdef CONFIG_CIFS_ACL
- seq_printf(m, " acl");
+ seq_printf(m, ",ACL");
#endif
seq_putc(m, '\n');
seq_printf(m, "Active VFS Requests: %d\n", GlobalTotalActiveXid);
@@ -268,6 +284,13 @@ static ssize_t cifs_stats_proc_write(struct file *file,
atomic_set(&totBufAllocCount, 0);
atomic_set(&totSmBufAllocCount, 0);
#endif /* CONFIG_CIFS_STATS2 */
+ atomic_set(&tcpSesReconnectCount, 0);
+ atomic_set(&tconInfoReconnectCount, 0);
+
+ spin_lock(&GlobalMid_Lock);
+ GlobalMaxActiveXid = 0;
+ GlobalCurrentXid = 0;
+ spin_unlock(&GlobalMid_Lock);
spin_lock(&cifs_tcp_ses_lock);
list_for_each(tmp1, &cifs_tcp_ses_list) {
server = list_entry(tmp1, struct TCP_Server_Info,
@@ -280,6 +303,10 @@ static ssize_t cifs_stats_proc_write(struct file *file,
struct cifs_tcon,
tcon_list);
atomic_set(&tcon->num_smbs_sent, 0);
+ spin_lock(&tcon->stat_lock);
+ tcon->bytes_read = 0;
+ tcon->bytes_written = 0;
+ spin_unlock(&tcon->stat_lock);
if (server->ops->clear_stats)
server->ops->clear_stats(tcon);
}
diff --git a/fs/cifs/cifs_spnego.c b/fs/cifs/cifs_spnego.c
index b611fc2..7f01c6e 100644
--- a/fs/cifs/cifs_spnego.c
+++ b/fs/cifs/cifs_spnego.c
@@ -147,8 +147,10 @@ cifs_get_spnego_key(struct cifs_ses *sesInfo)
sprintf(dp, ";sec=krb5");
else if (server->sec_mskerberos)
sprintf(dp, ";sec=mskrb5");
- else
- goto out;
+ else {
+ cifs_dbg(VFS, "unknown or missing server auth type, use krb5\n");
+ sprintf(dp, ";sec=krb5");
+ }
dp = description + strlen(description);
sprintf(dp, ";uid=0x%x",
diff --git a/fs/cifs/cifs_unicode.c b/fs/cifs/cifs_unicode.c
index a0b3e7d..211ac47 100644
--- a/fs/cifs/cifs_unicode.c
+++ b/fs/cifs/cifs_unicode.c
@@ -101,9 +101,6 @@ convert_sfm_char(const __u16 src_char, char *target)
case SFM_LESSTHAN:
*target = '<';
break;
- case SFM_SLASH:
- *target = '\\';
- break;
case SFM_SPACE:
*target = ' ';
break;
diff --git a/fs/cifs/cifsfs.c b/fs/cifs/cifsfs.c
index 87658f6..be84d49 100644
--- a/fs/cifs/cifsfs.c
+++ b/fs/cifs/cifsfs.c
@@ -927,8 +927,8 @@ static int cifs_clone_file_range(struct file *src_file, loff_t off,
struct inode *src_inode = file_inode(src_file);
struct inode *target_inode = file_inode(dst_file);
struct cifsFileInfo *smb_file_src = src_file->private_data;
- struct cifsFileInfo *smb_file_target = dst_file->private_data;
- struct cifs_tcon *target_tcon = tlink_tcon(smb_file_target->tlink);
+ struct cifsFileInfo *smb_file_target;
+ struct cifs_tcon *target_tcon;
unsigned int xid;
int rc;
@@ -942,6 +942,9 @@ static int cifs_clone_file_range(struct file *src_file, loff_t off,
goto out;
}
+ smb_file_target = dst_file->private_data;
+ target_tcon = tlink_tcon(smb_file_target->tlink);
+
/*
* Note: cifs case is easier than btrfs since server responsible for
* checks for proper open modes and file type and if it wants
diff --git a/fs/cifs/cifssmb.c b/fs/cifs/cifssmb.c
index cc420d6..741b83c 100644
--- a/fs/cifs/cifssmb.c
+++ b/fs/cifs/cifssmb.c
@@ -150,8 +150,14 @@ cifs_reconnect_tcon(struct cifs_tcon *tcon, int smb_command)
* greater than cifs socket timeout which is 7 seconds
*/
while (server->tcpStatus == CifsNeedReconnect) {
- wait_event_interruptible_timeout(server->response_q,
- (server->tcpStatus != CifsNeedReconnect), 10 * HZ);
+ rc = wait_event_interruptible_timeout(server->response_q,
+ (server->tcpStatus != CifsNeedReconnect),
+ 10 * HZ);
+ if (rc < 0) {
+ cifs_dbg(FYI, "%s: aborting reconnect due to a received"
+ " signal by the process\n", __func__);
+ return -ERESTARTSYS;
+ }
/* are we still trying to reconnect? */
if (server->tcpStatus != CifsNeedReconnect)
@@ -571,10 +577,15 @@ CIFSSMBNegotiate(const unsigned int xid, struct cifs_ses *ses)
}
count = 0;
+ /*
+ * We know that all the name entries in the protocols array
+ * are short (< 16 bytes anyway) and are NUL terminated.
+ */
for (i = 0; i < CIFS_NUM_PROT; i++) {
- strncpy(pSMB->DialectsArray+count, protocols[i].name, 16);
- count += strlen(protocols[i].name) + 1;
- /* null at end of source and target buffers anyway */
+ size_t len = strlen(protocols[i].name) + 1;
+
+ memcpy(pSMB->DialectsArray+count, protocols[i].name, len);
+ count += len;
}
inc_rfc1001_len(pSMB, count);
pSMB->ByteCount = cpu_to_le16(count);
@@ -6413,9 +6424,7 @@ CIFSSMBSetEA(const unsigned int xid, struct cifs_tcon *tcon,
pSMB->InformationLevel =
cpu_to_le16(SMB_SET_FILE_EA);
- parm_data =
- (struct fealist *) (((char *) &pSMB->hdr.Protocol) +
- offset);
+ parm_data = (void *)pSMB + offsetof(struct smb_hdr, Protocol) + offset;
pSMB->ParameterOffset = cpu_to_le16(param_offset);
pSMB->DataOffset = cpu_to_le16(offset);
pSMB->SetupCount = 1;
diff --git a/fs/cifs/dir.c b/fs/cifs/dir.c
index d9cbda2..e98e24e 100644
--- a/fs/cifs/dir.c
+++ b/fs/cifs/dir.c
@@ -163,7 +163,7 @@ build_path_from_dentry(struct dentry *direntry)
cifs_dbg(FYI, "using cifs_sb prepath <%s>\n", cifs_sb->prepath);
memcpy(full_path+dfsplen+1, cifs_sb->prepath, pplen-1);
- full_path[dfsplen] = '\\';
+ full_path[dfsplen] = dirsep;
for (i = 0; i < pplen-1; i++)
if (full_path[dfsplen+1+i] == '/')
full_path[dfsplen+1+i] = CIFS_DIR_SEP(cifs_sb);
@@ -673,6 +673,9 @@ int cifs_mknod(struct inode *inode, struct dentry *direntry, umode_t mode,
goto mknod_out;
}
+ if (!S_ISCHR(mode) && !S_ISBLK(mode))
+ goto mknod_out;
+
if (!(cifs_sb->mnt_cifs_flags & CIFS_MOUNT_UNX_EMUL))
goto mknod_out;
@@ -681,10 +684,8 @@ int cifs_mknod(struct inode *inode, struct dentry *direntry, umode_t mode,
buf = kmalloc(sizeof(FILE_ALL_INFO), GFP_KERNEL);
if (buf == NULL) {
- kfree(full_path);
rc = -ENOMEM;
- free_xid(xid);
- return rc;
+ goto mknod_out;
}
if (backup_cred(cifs_sb))
@@ -731,7 +732,7 @@ int cifs_mknod(struct inode *inode, struct dentry *direntry, umode_t mode,
pdev->minor = cpu_to_le64(MINOR(device_number));
rc = tcon->ses->server->ops->sync_write(xid, &fid, &io_parms,
&bytes_written, iov, 1);
- } /* else if (S_ISFIFO) */
+ }
tcon->ses->server->ops->close(xid, tcon, &fid);
d_drop(direntry);
diff --git a/fs/cifs/inode.c b/fs/cifs/inode.c
index 24c19eb..57c938f 100644
--- a/fs/cifs/inode.c
+++ b/fs/cifs/inode.c
@@ -704,7 +704,7 @@ cifs_get_file_info(struct file *filp)
/* Simple function to return a 64 bit hash of string. Rarely called */
static __u64 simple_hashstr(const char *str)
{
- const __u64 hash_mult = 1125899906842597L; /* a big enough prime */
+ const __u64 hash_mult = 1125899906842597ULL; /* a big enough prime */
__u64 hash = 0;
while (*str)
@@ -768,7 +768,15 @@ cifs_get_inode_info(struct inode **inode, const char *full_path,
} else if (rc == -EREMOTE) {
cifs_create_dfs_fattr(&fattr, sb);
rc = 0;
- } else if (rc == -EACCES && backup_cred(cifs_sb)) {
+ } else if ((rc == -EACCES) && backup_cred(cifs_sb) &&
+ (strcmp(server->vals->version_string, SMB1_VERSION_STRING)
+ == 0)) {
+ /*
+ * For SMB2 and later the backup intent flag is already
+ * sent if needed on open and there is no path based
+ * FindFirst operation to use to retry with
+ */
+
srchinf = kzalloc(sizeof(struct cifs_search_info),
GFP_KERNEL);
if (srchinf == NULL) {
@@ -1116,6 +1124,8 @@ cifs_set_file_info(struct inode *inode, struct iattr *attrs, unsigned int xid,
if (!server->ops->set_file_info)
return -ENOSYS;
+ info_buf.Pad = 0;
+
if (attrs->ia_valid & ATTR_ATIME) {
set_time = true;
info_buf.LastAccessTime =
diff --git a/fs/cifs/link.c b/fs/cifs/link.c
index d031af8..38d26cb 100644
--- a/fs/cifs/link.c
+++ b/fs/cifs/link.c
@@ -419,7 +419,7 @@ smb3_query_mf_symlink(unsigned int xid, struct cifs_tcon *tcon,
struct cifs_io_parms io_parms;
int buf_type = CIFS_NO_BUFFER;
__le16 *utf16_path;
- __u8 oplock = SMB2_OPLOCK_LEVEL_II;
+ __u8 oplock = SMB2_OPLOCK_LEVEL_NONE;
struct smb2_file_all_info *pfile_info = NULL;
oparms.tcon = tcon;
@@ -481,7 +481,7 @@ smb3_create_mf_symlink(unsigned int xid, struct cifs_tcon *tcon,
struct cifs_io_parms io_parms;
int create_options = CREATE_NOT_DIR;
__le16 *utf16_path;
- __u8 oplock = SMB2_OPLOCK_LEVEL_EXCLUSIVE;
+ __u8 oplock = SMB2_OPLOCK_LEVEL_NONE;
struct kvec iov[2];
if (backup_cred(cifs_sb))
diff --git a/fs/cifs/misc.c b/fs/cifs/misc.c
index 323d8e3..50559a8 100644
--- a/fs/cifs/misc.c
+++ b/fs/cifs/misc.c
@@ -406,9 +406,17 @@ is_valid_oplock_break(char *buffer, struct TCP_Server_Info *srv)
(struct smb_com_transaction_change_notify_rsp *)buf;
struct file_notify_information *pnotify;
__u32 data_offset = 0;
+ size_t len = srv->total_read - sizeof(pSMBr->hdr.smb_buf_length);
+
if (get_bcc(buf) > sizeof(struct file_notify_information)) {
data_offset = le32_to_cpu(pSMBr->DataOffset);
+ if (data_offset >
+ len - sizeof(struct file_notify_information)) {
+ cifs_dbg(FYI, "invalid data_offset %u\n",
+ data_offset);
+ return true;
+ }
pnotify = (struct file_notify_information *)
((char *)&pSMBr->hdr.Protocol + data_offset);
cifs_dbg(FYI, "dnotify on %s Action: 0x%x\n",
diff --git a/fs/cifs/readdir.c b/fs/cifs/readdir.c
index a27fc879..ef24b45 100644
--- a/fs/cifs/readdir.c
+++ b/fs/cifs/readdir.c
@@ -376,8 +376,15 @@ static char *nxt_dir_entry(char *old_entry, char *end_of_smb, int level)
new_entry = old_entry + sizeof(FIND_FILE_STANDARD_INFO) +
pfData->FileNameLength;
- } else
- new_entry = old_entry + le32_to_cpu(pDirInfo->NextEntryOffset);
+ } else {
+ u32 next_offset = le32_to_cpu(pDirInfo->NextEntryOffset);
+
+ if (old_entry + next_offset < old_entry) {
+ cifs_dbg(VFS, "invalid offset %u\n", next_offset);
+ return NULL;
+ }
+ new_entry = old_entry + next_offset;
+ }
cifs_dbg(FYI, "new entry %p old entry %p\n", new_entry, old_entry);
/* validate that new_entry is not past end of SMB */
if (new_entry >= end_of_smb) {
diff --git a/fs/cifs/sess.c b/fs/cifs/sess.c
index c3db2a8..bb20807 100644
--- a/fs/cifs/sess.c
+++ b/fs/cifs/sess.c
@@ -398,6 +398,12 @@ int build_ntlmssp_auth_blob(unsigned char **pbuffer,
goto setup_ntlmv2_ret;
}
*pbuffer = kmalloc(size_of_ntlmssp_blob(ses), GFP_KERNEL);
+ if (!*pbuffer) {
+ rc = -ENOMEM;
+ cifs_dbg(VFS, "Error %d during NTLMSSP allocation\n", rc);
+ *buflen = 0;
+ goto setup_ntlmv2_ret;
+ }
sec_blob = (AUTHENTICATE_MESSAGE *)*pbuffer;
memcpy(sec_blob->Signature, NTLMSSP_SIGNATURE, 8);
diff --git a/fs/cifs/smb2inode.c b/fs/cifs/smb2inode.c
index 1238cd3..0267d8c 100644
--- a/fs/cifs/smb2inode.c
+++ b/fs/cifs/smb2inode.c
@@ -267,7 +267,7 @@ smb2_set_file_info(struct inode *inode, const char *full_path,
int rc;
if ((buf->CreationTime == 0) && (buf->LastAccessTime == 0) &&
- (buf->LastWriteTime == 0) && (buf->ChangeTime) &&
+ (buf->LastWriteTime == 0) && (buf->ChangeTime == 0) &&
(buf->Attributes == 0))
return 0; /* would be a no op, no sense sending this */
diff --git a/fs/cifs/smb2maperror.c b/fs/cifs/smb2maperror.c
index 8257a5a..98c25b9 100644
--- a/fs/cifs/smb2maperror.c
+++ b/fs/cifs/smb2maperror.c
@@ -377,8 +377,8 @@ static const struct status_to_posix_error smb2_error_map_table[] = {
{STATUS_NONEXISTENT_EA_ENTRY, -EIO, "STATUS_NONEXISTENT_EA_ENTRY"},
{STATUS_NO_EAS_ON_FILE, -ENODATA, "STATUS_NO_EAS_ON_FILE"},
{STATUS_EA_CORRUPT_ERROR, -EIO, "STATUS_EA_CORRUPT_ERROR"},
- {STATUS_FILE_LOCK_CONFLICT, -EIO, "STATUS_FILE_LOCK_CONFLICT"},
- {STATUS_LOCK_NOT_GRANTED, -EIO, "STATUS_LOCK_NOT_GRANTED"},
+ {STATUS_FILE_LOCK_CONFLICT, -EACCES, "STATUS_FILE_LOCK_CONFLICT"},
+ {STATUS_LOCK_NOT_GRANTED, -EACCES, "STATUS_LOCK_NOT_GRANTED"},
{STATUS_DELETE_PENDING, -ENOENT, "STATUS_DELETE_PENDING"},
{STATUS_CTL_FILE_NOT_SUPPORTED, -ENOSYS,
"STATUS_CTL_FILE_NOT_SUPPORTED"},
diff --git a/fs/cifs/smb2misc.c b/fs/cifs/smb2misc.c
index 967dfe6..e96a74d 100644
--- a/fs/cifs/smb2misc.c
+++ b/fs/cifs/smb2misc.c
@@ -209,6 +209,13 @@ smb2_check_message(char *buf, unsigned int length, struct TCP_Server_Info *srvr)
return 0;
/*
+ * Some windows servers (win2016) will pad also the final
+ * PDU in a compound to 8 bytes.
+ */
+ if (((clc_len + 7) & ~7) == len)
+ return 0;
+
+ /*
* MacOS server pads after SMB2.1 write response with 3 bytes
* of junk. Other servers match RFC1001 len to actual
* SMB2/SMB3 frame length (header + smb2 response specific data)
diff --git a/fs/cifs/smb2ops.c b/fs/cifs/smb2ops.c
index 812e488..08c1c86 100644
--- a/fs/cifs/smb2ops.c
+++ b/fs/cifs/smb2ops.c
@@ -894,6 +894,13 @@ smb3_set_integrity(const unsigned int xid, struct cifs_tcon *tcon,
}
+/* GMT Token is @GMT-YYYY.MM.DD-HH.MM.SS Unicode which is 48 bytes + null */
+#define GMT_TOKEN_SIZE 50
+
+/*
+ * Input buffer contains (empty) struct smb_snapshot array with size filled in
+ * For output see struct SRV_SNAPSHOT_ARRAY in MS-SMB2 section 2.2.32.2
+ */
static int
smb3_enum_snapshots(const unsigned int xid, struct cifs_tcon *tcon,
struct cifsFileInfo *cfile, void __user *ioc_buf)
@@ -922,14 +929,27 @@ smb3_enum_snapshots(const unsigned int xid, struct cifs_tcon *tcon,
kfree(retbuf);
return rc;
}
- if (snapshot_in.snapshot_array_size < sizeof(struct smb_snapshot_array)) {
- rc = -ERANGE;
- kfree(retbuf);
- return rc;
- }
- if (ret_data_len > snapshot_in.snapshot_array_size)
- ret_data_len = snapshot_in.snapshot_array_size;
+ /*
+ * Check for min size, ie not large enough to fit even one GMT
+ * token (snapshot). On the first ioctl some users may pass in
+ * smaller size (or zero) to simply get the size of the array
+ * so the user space caller can allocate sufficient memory
+ * and retry the ioctl again with larger array size sufficient
+ * to hold all of the snapshot GMT tokens on the second try.
+ */
+ if (snapshot_in.snapshot_array_size < GMT_TOKEN_SIZE)
+ ret_data_len = sizeof(struct smb_snapshot_array);
+
+ /*
+ * We return struct SRV_SNAPSHOT_ARRAY, followed by
+ * the snapshot array (of 50 byte GMT tokens) each
+ * representing an available previous version of the data
+ */
+ if (ret_data_len > (snapshot_in.snapshot_array_size +
+ sizeof(struct smb_snapshot_array)))
+ ret_data_len = snapshot_in.snapshot_array_size +
+ sizeof(struct smb_snapshot_array);
if (copy_to_user(ioc_buf, retbuf, ret_data_len))
rc = -EFAULT;
@@ -969,7 +989,7 @@ smb2_query_dir_first(const unsigned int xid, struct cifs_tcon *tcon,
}
srch_inf->entries_in_buffer = 0;
- srch_inf->index_of_last_entry = 0;
+ srch_inf->index_of_last_entry = 2;
rc = SMB2_query_directory(xid, tcon, fid->persistent_fid,
fid->volatile_fid, 0, srch_inf);
diff --git a/fs/cifs/smb2pdu.c b/fs/cifs/smb2pdu.c
index 44b7ccb..50251a8 100644
--- a/fs/cifs/smb2pdu.c
+++ b/fs/cifs/smb2pdu.c
@@ -155,7 +155,7 @@ smb2_hdr_assemble(struct smb2_hdr *hdr, __le16 smb2_cmd /* command */ ,
static int
smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon)
{
- int rc = 0;
+ int rc;
struct nls_table *nls_codepage;
struct cifs_ses *ses;
struct TCP_Server_Info *server;
@@ -166,10 +166,10 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon)
* for those three - in the calling routine.
*/
if (tcon == NULL)
- return rc;
+ return 0;
if (smb2_command == SMB2_TREE_CONNECT)
- return rc;
+ return 0;
if (tcon->tidStatus == CifsExiting) {
/*
@@ -212,8 +212,14 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon)
return -EAGAIN;
}
- wait_event_interruptible_timeout(server->response_q,
- (server->tcpStatus != CifsNeedReconnect), 10 * HZ);
+ rc = wait_event_interruptible_timeout(server->response_q,
+ (server->tcpStatus != CifsNeedReconnect),
+ 10 * HZ);
+ if (rc < 0) {
+ cifs_dbg(FYI, "%s: aborting reconnect due to a received"
+ " signal by the process\n", __func__);
+ return -ERESTARTSYS;
+ }
/* are we still trying to reconnect? */
if (server->tcpStatus != CifsNeedReconnect)
@@ -231,7 +237,7 @@ smb2_reconnect(__le16 smb2_command, struct cifs_tcon *tcon)
}
if (!tcon->ses->need_reconnect && !tcon->need_reconnect)
- return rc;
+ return 0;
nls_codepage = load_nls_default();
@@ -314,7 +320,7 @@ small_smb2_init(__le16 smb2_command, struct cifs_tcon *tcon,
smb2_hdr_assemble((struct smb2_hdr *) *request_buf, smb2_command, tcon);
if (tcon != NULL) {
-#ifdef CONFIG_CIFS_STATS2
+#ifdef CONFIG_CIFS_STATS
uint16_t com_code = le16_to_cpu(smb2_command);
cifs_stats_inc(&tcon->stats.smb2_stats.smb2_com_sent[com_code]);
#endif
@@ -1004,6 +1010,7 @@ SMB2_sess_setup(const unsigned int xid, struct cifs_ses *ses,
sess_data->ses = ses;
sess_data->buf0_type = CIFS_NO_BUFFER;
sess_data->nls_cp = (struct nls_table *) nls_cp;
+ sess_data->previous_session = ses->Suid;
while (sess_data->func)
sess_data->func(sess_data);
@@ -2558,33 +2565,38 @@ num_entries(char *bufstart, char *end_of_buf, char **lastentry, size_t size)
int len;
unsigned int entrycount = 0;
unsigned int next_offset = 0;
- FILE_DIRECTORY_INFO *entryptr;
+ char *entryptr;
+ FILE_DIRECTORY_INFO *dir_info;
if (bufstart == NULL)
return 0;
- entryptr = (FILE_DIRECTORY_INFO *)bufstart;
+ entryptr = bufstart;
while (1) {
- entryptr = (FILE_DIRECTORY_INFO *)
- ((char *)entryptr + next_offset);
-
- if ((char *)entryptr + size > end_of_buf) {
+ if (entryptr + next_offset < entryptr ||
+ entryptr + next_offset > end_of_buf ||
+ entryptr + next_offset + size > end_of_buf) {
cifs_dbg(VFS, "malformed search entry would overflow\n");
break;
}
- len = le32_to_cpu(entryptr->FileNameLength);
- if ((char *)entryptr + len + size > end_of_buf) {
+ entryptr = entryptr + next_offset;
+ dir_info = (FILE_DIRECTORY_INFO *)entryptr;
+
+ len = le32_to_cpu(dir_info->FileNameLength);
+ if (entryptr + len < entryptr ||
+ entryptr + len > end_of_buf ||
+ entryptr + len + size > end_of_buf) {
cifs_dbg(VFS, "directory entry name would overflow frame end of buf %p\n",
end_of_buf);
break;
}
- *lastentry = (char *)entryptr;
+ *lastentry = entryptr;
entrycount++;
- next_offset = le32_to_cpu(entryptr->NextEntryOffset);
+ next_offset = le32_to_cpu(dir_info->NextEntryOffset);
if (!next_offset)
break;
}
diff --git a/fs/configfs/dir.c b/fs/configfs/dir.c
index 56fb261..d2a1a79 100644
--- a/fs/configfs/dir.c
+++ b/fs/configfs/dir.c
@@ -1777,6 +1777,16 @@ void configfs_unregister_group(struct config_group *group)
struct dentry *dentry = group->cg_item.ci_dentry;
struct dentry *parent = group->cg_item.ci_parent->ci_dentry;
+ mutex_lock(&subsys->su_mutex);
+ if (!group->cg_item.ci_parent->ci_group) {
+ /*
+ * The parent has already been unlinked and detached
+ * due to a rmdir.
+ */
+ goto unlink_group;
+ }
+ mutex_unlock(&subsys->su_mutex);
+
inode_lock_nested(d_inode(parent), I_MUTEX_PARENT);
spin_lock(&configfs_dirent_lock);
configfs_detach_prep(dentry, NULL);
@@ -1791,6 +1801,7 @@ void configfs_unregister_group(struct config_group *group)
dput(dentry);
mutex_lock(&subsys->su_mutex);
+unlink_group:
unlink_group(group);
mutex_unlock(&subsys->su_mutex);
}
diff --git a/fs/configfs/symlink.c b/fs/configfs/symlink.c
index 314b4ed..fea6db1 100644
--- a/fs/configfs/symlink.c
+++ b/fs/configfs/symlink.c
@@ -64,7 +64,7 @@ static void fill_item_path(struct config_item * item, char * buffer, int length)
/* back up enough to print this bus id with '/' */
length -= cur;
- strncpy(buffer + length,config_item_name(p),cur);
+ memcpy(buffer + length, config_item_name(p), cur);
*(buffer + --length) = '/';
}
}
diff --git a/fs/cramfs/inode.c b/fs/cramfs/inode.c
index 7919967..011c6f5 100644
--- a/fs/cramfs/inode.c
+++ b/fs/cramfs/inode.c
@@ -186,7 +186,8 @@ static void *cramfs_read(struct super_block *sb, unsigned int offset, unsigned i
continue;
blk_offset = (blocknr - buffer_blocknr[i]) << PAGE_SHIFT;
blk_offset += offset;
- if (blk_offset + len > BUFFER_SIZE)
+ if (blk_offset > BUFFER_SIZE ||
+ blk_offset + len > BUFFER_SIZE)
continue;
return read_buffers[i] + blk_offset;
}
diff --git a/fs/crypto/Makefile b/fs/crypto/Makefile
index b0ca0e6..559e0bd 100644
--- a/fs/crypto/Makefile
+++ b/fs/crypto/Makefile
@@ -1,8 +1,5 @@
obj-$(CONFIG_FS_ENCRYPTION) += fscrypto.o
-ccflags-y += -Ifs/ext4
-ccflags-y += -Ifs/f2fs
-
fscrypto-y := crypto.o fname.o hooks.o keyinfo.o policy.o
fscrypto-$(CONFIG_BLOCK) += bio.o
fscrypto-$(CONFIG_PFK) += fscrypt_ice.o
diff --git a/fs/crypto/bio.c b/fs/crypto/bio.c
index c629e97..d7b4c48 100644
--- a/fs/crypto/bio.c
+++ b/fs/crypto/bio.c
@@ -32,18 +32,14 @@ static void __fscrypt_decrypt_bio(struct bio *bio, bool done)
bio_for_each_segment_all(bv, bio, i) {
struct page *page = bv->bv_page;
+ int ret = fscrypt_decrypt_page(page->mapping->host, page,
+ PAGE_SIZE, 0, page->index);
- if (fscrypt_using_hardware_encryption(page->mapping->host)) {
+ if (ret) {
+ WARN_ON_ONCE(1);
+ SetPageError(page);
+ } else if (done) {
SetPageUptodate(page);
- } else {
- int ret = fscrypt_decrypt_page(page->mapping->host,
- page, PAGE_SIZE, 0, page->index);
- if (ret) {
- WARN_ON_ONCE(1);
- SetPageError(page);
- } else if (done) {
- SetPageUptodate(page);
- }
}
if (done)
unlock_page(page);
diff --git a/fs/crypto/crypto.c b/fs/crypto/crypto.c
index 0758d32..4dc788e 100644
--- a/fs/crypto/crypto.c
+++ b/fs/crypto/crypto.c
@@ -133,15 +133,25 @@ struct fscrypt_ctx *fscrypt_get_ctx(const struct inode *inode, gfp_t gfp_flags)
}
EXPORT_SYMBOL(fscrypt_get_ctx);
+void fscrypt_generate_iv(union fscrypt_iv *iv, u64 lblk_num,
+ const struct fscrypt_info *ci)
+{
+ memset(iv, 0, ci->ci_mode->ivsize);
+ iv->lblk_num = cpu_to_le64(lblk_num);
+
+ if (ci->ci_flags & FS_POLICY_FLAG_DIRECT_KEY)
+ memcpy(iv->nonce, ci->ci_nonce, FS_KEY_DERIVATION_NONCE_SIZE);
+
+ if (ci->ci_essiv_tfm != NULL)
+ crypto_cipher_encrypt_one(ci->ci_essiv_tfm, iv->raw, iv->raw);
+}
+
int fscrypt_do_page_crypto(const struct inode *inode, fscrypt_direction_t rw,
u64 lblk_num, struct page *src_page,
struct page *dest_page, unsigned int len,
unsigned int offs, gfp_t gfp_flags)
{
- struct {
- __le64 index;
- u8 padding[FS_IV_SIZE - sizeof(__le64)];
- } iv;
+ union fscrypt_iv iv;
struct skcipher_request *req = NULL;
DECLARE_CRYPTO_WAIT(wait);
struct scatterlist dst, src;
@@ -151,23 +161,11 @@ int fscrypt_do_page_crypto(const struct inode *inode, fscrypt_direction_t rw,
BUG_ON(len == 0);
- BUILD_BUG_ON(sizeof(iv) != FS_IV_SIZE);
- BUILD_BUG_ON(AES_BLOCK_SIZE != FS_IV_SIZE);
- iv.index = cpu_to_le64(lblk_num);
- memset(iv.padding, 0, sizeof(iv.padding));
-
- if (ci->ci_essiv_tfm != NULL) {
- crypto_cipher_encrypt_one(ci->ci_essiv_tfm, (u8 *)&iv,
- (u8 *)&iv);
- }
+ fscrypt_generate_iv(&iv, lblk_num, ci);
req = skcipher_request_alloc(tfm, gfp_flags);
- if (!req) {
- printk_ratelimited(KERN_ERR
- "%s: crypto_request_alloc() failed\n",
- __func__);
+ if (!req)
return -ENOMEM;
- }
skcipher_request_set_callback(
req, CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP,
@@ -184,9 +182,10 @@ int fscrypt_do_page_crypto(const struct inode *inode, fscrypt_direction_t rw,
res = crypto_wait_req(crypto_skcipher_encrypt(req), &wait);
skcipher_request_free(req);
if (res) {
- printk_ratelimited(KERN_ERR
- "%s: crypto_skcipher_encrypt() returned %d\n",
- __func__, res);
+ fscrypt_err(inode->i_sb,
+ "%scryption failed for inode %lu, block %llu: %d",
+ (rw == FS_DECRYPT ? "de" : "en"),
+ inode->i_ino, lblk_num, res);
return res;
}
return 0;
@@ -332,7 +331,6 @@ static int fscrypt_d_revalidate(struct dentry *dentry, unsigned int flags)
return 0;
}
- /* this should eventually be an flag in d_flags */
spin_lock(&dentry->d_lock);
cached_with_key = dentry->d_flags & DCACHE_ENCRYPTED_WITH_KEY;
spin_unlock(&dentry->d_lock);
@@ -359,7 +357,6 @@ static int fscrypt_d_revalidate(struct dentry *dentry, unsigned int flags)
const struct dentry_operations fscrypt_d_ops = {
.d_revalidate = fscrypt_d_revalidate,
};
-EXPORT_SYMBOL(fscrypt_d_ops);
void fscrypt_restore_control_page(struct page *page)
{
@@ -428,13 +425,43 @@ int fscrypt_initialize(unsigned int cop_flags)
return res;
}
+void fscrypt_msg(struct super_block *sb, const char *level,
+ const char *fmt, ...)
+{
+ static DEFINE_RATELIMIT_STATE(rs, DEFAULT_RATELIMIT_INTERVAL,
+ DEFAULT_RATELIMIT_BURST);
+ struct va_format vaf;
+ va_list args;
+
+ if (!__ratelimit(&rs))
+ return;
+
+ va_start(args, fmt);
+ vaf.fmt = fmt;
+ vaf.va = &args;
+ if (sb)
+ printk("%sfscrypt (%s): %pV\n", level, sb->s_id, &vaf);
+ else
+ printk("%sfscrypt: %pV\n", level, &vaf);
+ va_end(args);
+}
+
/**
* fscrypt_init() - Set up for fs encryption.
*/
static int __init fscrypt_init(void)
{
+ /*
+ * Use an unbound workqueue to allow bios to be decrypted in parallel
+ * even when they happen to complete on the same CPU. This sacrifices
+ * locality, but it's worthwhile since decryption is CPU-intensive.
+ *
+ * Also use a high-priority workqueue to prioritize decryption work,
+ * which blocks reads from completing, over regular application tasks.
+ */
fscrypt_read_workqueue = alloc_workqueue("fscrypt_read_queue",
- WQ_HIGHPRI, 0);
+ WQ_UNBOUND | WQ_HIGHPRI,
+ num_online_cpus());
if (!fscrypt_read_workqueue)
goto fail;
diff --git a/fs/crypto/fname.c b/fs/crypto/fname.c
index b18fa32..217d421 100644
--- a/fs/crypto/fname.c
+++ b/fs/crypto/fname.c
@@ -39,10 +39,11 @@ int fname_encrypt(struct inode *inode, const struct qstr *iname,
{
struct skcipher_request *req = NULL;
DECLARE_CRYPTO_WAIT(wait);
- struct crypto_skcipher *tfm = inode->i_crypt_info->ci_ctfm;
- int res = 0;
- char iv[FS_CRYPTO_BLOCK_SIZE];
+ struct fscrypt_info *ci = inode->i_crypt_info;
+ struct crypto_skcipher *tfm = ci->ci_ctfm;
+ union fscrypt_iv iv;
struct scatterlist sg;
+ int res;
/*
* Copy the filename to the output buffer for encrypting in-place and
@@ -54,27 +55,25 @@ int fname_encrypt(struct inode *inode, const struct qstr *iname,
memset(out + iname->len, 0, olen - iname->len);
/* Initialize the IV */
- memset(iv, 0, FS_CRYPTO_BLOCK_SIZE);
+ fscrypt_generate_iv(&iv, 0, ci);
/* Set up the encryption request */
req = skcipher_request_alloc(tfm, GFP_NOFS);
- if (!req) {
- printk_ratelimited(KERN_ERR
- "%s: skcipher_request_alloc() failed\n", __func__);
+ if (!req)
return -ENOMEM;
- }
skcipher_request_set_callback(req,
CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP,
crypto_req_done, &wait);
sg_init_one(&sg, out, olen);
- skcipher_request_set_crypt(req, &sg, &sg, olen, iv);
+ skcipher_request_set_crypt(req, &sg, &sg, olen, &iv);
/* Do the encryption */
res = crypto_wait_req(crypto_skcipher_encrypt(req), &wait);
skcipher_request_free(req);
if (res < 0) {
- printk_ratelimited(KERN_ERR
- "%s: Error (error code %d)\n", __func__, res);
+ fscrypt_err(inode->i_sb,
+ "Filename encryption failed for inode %lu: %d",
+ inode->i_ino, res);
return res;
}
@@ -97,37 +96,30 @@ static int fname_decrypt(struct inode *inode,
struct scatterlist src_sg, dst_sg;
struct fscrypt_info *ci = inode->i_crypt_info;
struct crypto_skcipher *tfm = ci->ci_ctfm;
- int res = 0;
- char iv[FS_CRYPTO_BLOCK_SIZE];
- unsigned lim;
-
- lim = inode->i_sb->s_cop->max_namelen(inode);
- if (iname->len <= 0 || iname->len > lim)
- return -EIO;
+ union fscrypt_iv iv;
+ int res;
/* Allocate request */
req = skcipher_request_alloc(tfm, GFP_NOFS);
- if (!req) {
- printk_ratelimited(KERN_ERR
- "%s: crypto_request_alloc() failed\n", __func__);
+ if (!req)
return -ENOMEM;
- }
skcipher_request_set_callback(req,
CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP,
crypto_req_done, &wait);
/* Initialize IV */
- memset(iv, 0, FS_CRYPTO_BLOCK_SIZE);
+ fscrypt_generate_iv(&iv, 0, ci);
/* Create decryption request */
sg_init_one(&src_sg, iname->name, iname->len);
sg_init_one(&dst_sg, oname->name, oname->len);
- skcipher_request_set_crypt(req, &src_sg, &dst_sg, iname->len, iv);
+ skcipher_request_set_crypt(req, &src_sg, &dst_sg, iname->len, &iv);
res = crypto_wait_req(crypto_skcipher_decrypt(req), &wait);
skcipher_request_free(req);
if (res < 0) {
- printk_ratelimited(KERN_ERR
- "%s: Error (error code %d)\n", __func__, res);
+ fscrypt_err(inode->i_sb,
+ "Filename decryption failed for inode %lu: %d",
+ inode->i_ino, res);
return res;
}
@@ -340,12 +332,12 @@ int fscrypt_setup_filename(struct inode *dir, const struct qstr *iname,
return 0;
}
ret = fscrypt_get_encryption_info(dir);
- if (ret && ret != -EOPNOTSUPP)
+ if (ret)
return ret;
if (dir->i_crypt_info) {
if (!fscrypt_fname_encrypted_size(dir, iname->len,
- dir->i_sb->s_cop->max_namelen(dir),
+ dir->i_sb->s_cop->max_namelen,
&fname->crypto_buf.len))
return -ENAMETOOLONG;
fname->crypto_buf.name = kmalloc(fname->crypto_buf.len,
diff --git a/fs/crypto/fscrypt_ice.c b/fs/crypto/fscrypt_ice.c
index 62dae83..70935eb 100644
--- a/fs/crypto/fscrypt_ice.c
+++ b/fs/crypto/fscrypt_ice.c
@@ -16,8 +16,7 @@ int fscrypt_using_hardware_encryption(const struct inode *inode)
{
struct fscrypt_info *ci = inode->i_crypt_info;
- return S_ISREG(inode->i_mode) && ci &&
- ci->ci_data_mode == FS_ENCRYPTION_MODE_PRIVATE;
+ return ci && is_private_mode(ci->ci_mode);
}
EXPORT_SYMBOL(fscrypt_using_hardware_encryption);
@@ -85,9 +84,7 @@ bool fscrypt_is_ice_encryption_info_equal(const struct inode *inode1,
if (inode1 == inode2)
return true;
- /* both do not belong to ice, so we don't care, they are equal
- *for us
- */
+ /* both do not belong to ice, so we don't care, they are equal for us */
if (!fscrypt_should_be_processed_by_ice(inode1) &&
!fscrypt_should_be_processed_by_ice(inode2))
return true;
@@ -115,7 +112,6 @@ bool fscrypt_is_ice_encryption_info_equal(const struct inode *inode1,
(memcmp(salt1, salt2,
fscrypt_get_ice_encryption_salt_size(inode1)) == 0))
return true;
-
return false;
}
@@ -126,16 +122,29 @@ void fscrypt_set_ice_dun(const struct inode *inode, struct bio *bio, u64 dun)
}
EXPORT_SYMBOL(fscrypt_set_ice_dun);
+void fscrypt_set_ice_skip(struct bio *bio, int bi_crypt_skip)
+{
+#ifdef CONFIG_DM_DEFAULT_KEY
+ bio->bi_crypt_skip = bi_crypt_skip;
+#endif
+}
+EXPORT_SYMBOL(fscrypt_set_ice_skip);
+
/*
* This function will be used for filesystem when deciding to merge bios.
* Basic assumption is, if inline_encryption is set, single bio has to
* guarantee consecutive LBAs as well as ino|pg->index.
*/
-bool fscrypt_mergeable_bio(struct bio *bio, u64 dun, bool bio_encrypted)
+bool fscrypt_mergeable_bio(struct bio *bio, u64 dun, bool bio_encrypted,
+ int bi_crypt_skip)
{
if (!bio)
return true;
+#ifdef CONFIG_DM_DEFAULT_KEY
+ if (bi_crypt_skip != bio->bi_crypt_skip)
+ return false;
+#endif
/* if both of them are not encrypted, no further check is needed */
if (!bio_dun(bio) && !bio_encrypted)
return true;
diff --git a/fs/crypto/fscrypt_ice.h b/fs/crypto/fscrypt_ice.h
index d448eae..5b82185 100644
--- a/fs/crypto/fscrypt_ice.h
+++ b/fs/crypto/fscrypt_ice.h
@@ -17,12 +17,13 @@
#include "fscrypt_private.h"
#if IS_ENABLED(CONFIG_FS_ENCRYPTION)
-static inline bool fscrypt_should_be_processed_by_ice(const struct inode *inode)
+static inline int fscrypt_should_be_processed_by_ice(const struct inode *inode)
{
if (!inode->i_sb->s_cop)
return 0;
+
if (!IS_ENCRYPTED((struct inode *)inode))
- return 0;
+ return 0;
return fscrypt_using_hardware_encryption(inode);
}
@@ -52,7 +53,7 @@ static inline size_t fscrypt_get_ice_encryption_salt_size(
return FS_AES_256_XTS_KEY_SIZE / 2;
}
#else
-static inline bool fscrypt_should_be_processed_by_ice(const struct inode *inode)
+static inline int fscrypt_should_be_processed_by_ice(const struct inode *inode)
{
return 0;
}
diff --git a/fs/crypto/fscrypt_private.h b/fs/crypto/fscrypt_private.h
index 73bf64f..8a77c73 100644
--- a/fs/crypto/fscrypt_private.h
+++ b/fs/crypto/fscrypt_private.h
@@ -11,24 +11,14 @@
#ifndef _FSCRYPT_PRIVATE_H
#define _FSCRYPT_PRIVATE_H
-#ifndef __FS_HAS_ENCRYPTION
#define __FS_HAS_ENCRYPTION 1
-#endif
#include <linux/fscrypt.h>
#include <crypto/hash.h>
#include <linux/pfk.h>
/* Encryption parameters */
-#define FS_IV_SIZE 16
-#define FS_AES_128_ECB_KEY_SIZE 16
-#define FS_AES_128_CBC_KEY_SIZE 16
-#define FS_AES_128_CTS_KEY_SIZE 16
-#define FS_AES_256_GCM_KEY_SIZE 32
-#define FS_AES_256_CBC_KEY_SIZE 32
-#define FS_AES_256_CTS_KEY_SIZE 32
#define FS_AES_256_XTS_KEY_SIZE 64
-
-#define FS_KEY_DERIVATION_NONCE_SIZE 16
+#define FS_KEY_DERIVATION_NONCE_SIZE 16
/**
* Encryption context for inode
@@ -61,24 +51,45 @@ struct fscrypt_symlink_data {
char encrypted_path[1];
} __packed;
-enum ci_mode_info {
- CI_NONE_MODE = 0,
- CI_DATA_MODE,
- CI_FNAME_MODE,
-};
-
/*
- * A pointer to this structure is stored in the file system's in-core
- * representation of an inode.
+ * fscrypt_info - the "encryption key" for an inode
+ *
+ * When an encrypted file's key is made available, an instance of this struct is
+ * allocated and stored in ->i_crypt_info. Once created, it remains until the
+ * inode is evicted.
*/
struct fscrypt_info {
- u8 ci_mode;
+
+ /* The actual crypto transform used for encryption and decryption */
+ struct crypto_skcipher *ci_ctfm;
+
+ /*
+ * Cipher for ESSIV IV generation. Only set for CBC contents
+ * encryption, otherwise is NULL.
+ */
+ struct crypto_cipher *ci_essiv_tfm;
+
+ /*
+ * Encryption mode used for this inode. It corresponds to either
+ * ci_data_mode or ci_filename_mode, depending on the inode type.
+ */
+ struct fscrypt_mode *ci_mode;
+
+ /*
+ * If non-NULL, then this inode uses a master key directly rather than a
+ * derived key, and ci_ctfm will equal ci_master_key->mk_ctfm.
+ * Otherwise, this inode uses a derived key.
+ */
+ struct fscrypt_master_key *ci_master_key;
+
+ /* fields from the fscrypt_context */
u8 ci_data_mode;
u8 ci_filename_mode;
u8 ci_flags;
- struct crypto_skcipher *ci_ctfm;
- struct crypto_cipher *ci_essiv_tfm;
- u8 ci_master_key[FS_KEY_DESCRIPTOR_SIZE];
+ u8 ci_master_key_descriptor[FS_KEY_DESCRIPTOR_SIZE];
+ u8 ci_nonce[FS_KEY_DERIVATION_NONCE_SIZE];
+
+ /* Raw key, only for inline encryption w/ FS_ENCRYPTION_MODE_PRIVATE */
u8 ci_raw_key[FS_MAX_KEY_SIZE];
};
@@ -101,19 +112,17 @@ static inline bool fscrypt_valid_enc_modes(u32 contents_mode,
filenames_mode == FS_ENCRYPTION_MODE_AES_256_CTS)
return true;
+ if (contents_mode == FS_ENCRYPTION_MODE_ADIANTUM &&
+ filenames_mode == FS_ENCRYPTION_MODE_ADIANTUM)
+ return true;
+
if (contents_mode == FS_ENCRYPTION_MODE_PRIVATE &&
- filenames_mode == FS_ENCRYPTION_MODE_AES_256_CTS)
+ filenames_mode == FS_ENCRYPTION_MODE_AES_256_CTS)
return true;
return false;
}
-static inline bool is_private_data_mode(struct fscrypt_info *ci)
-{
- return ci->ci_mode == CI_DATA_MODE &&
- ci->ci_data_mode == FS_ENCRYPTION_MODE_PRIVATE;
-}
-
/* crypto.c */
extern struct kmem_cache *fscrypt_info_cachep;
extern int fscrypt_initialize(unsigned int cop_flags);
@@ -125,6 +134,31 @@ extern int fscrypt_do_page_crypto(const struct inode *inode,
gfp_t gfp_flags);
extern struct page *fscrypt_alloc_bounce_page(struct fscrypt_ctx *ctx,
gfp_t gfp_flags);
+extern const struct dentry_operations fscrypt_d_ops;
+
+extern void __printf(3, 4) __cold
+fscrypt_msg(struct super_block *sb, const char *level, const char *fmt, ...);
+
+#define fscrypt_warn(sb, fmt, ...) \
+ fscrypt_msg(sb, KERN_WARNING, fmt, ##__VA_ARGS__)
+#define fscrypt_err(sb, fmt, ...) \
+ fscrypt_msg(sb, KERN_ERR, fmt, ##__VA_ARGS__)
+
+#define FSCRYPT_MAX_IV_SIZE 32
+
+union fscrypt_iv {
+ struct {
+ /* logical block number within the file */
+ __le64 lblk_num;
+
+ /* per-file nonce; only set in DIRECT_KEY mode */
+ u8 nonce[FS_KEY_DERIVATION_NONCE_SIZE];
+ };
+ u8 raw[FSCRYPT_MAX_IV_SIZE];
+};
+
+void fscrypt_generate_iv(union fscrypt_iv *iv, u64 lblk_num,
+ const struct fscrypt_info *ci);
/* fname.c */
extern int fname_encrypt(struct inode *inode, const struct qstr *iname,
@@ -134,6 +168,22 @@ extern bool fscrypt_fname_encrypted_size(const struct inode *inode,
u32 *encrypted_len_ret);
/* keyinfo.c */
+
+struct fscrypt_mode {
+ const char *friendly_name;
+ const char *cipher_str;
+ int keysize;
+ int ivsize;
+ bool logged_impl_name;
+ bool needs_essiv;
+};
+
+static inline bool is_private_mode(const struct fscrypt_mode *mode)
+{
+ /* Using inline encryption with ICE, rather than the crypto API? */
+ return mode->cipher_str == NULL;
+}
+
extern void __exit fscrypt_essiv_cleanup(void);
#endif /* _FSCRYPT_PRIVATE_H */
diff --git a/fs/crypto/hooks.c b/fs/crypto/hooks.c
index bec0649..926e5df 100644
--- a/fs/crypto/hooks.c
+++ b/fs/crypto/hooks.c
@@ -39,8 +39,9 @@ int fscrypt_file_open(struct inode *inode, struct file *filp)
dir = dget_parent(file_dentry(filp));
if (IS_ENCRYPTED(d_inode(dir)) &&
!fscrypt_has_permitted_context(d_inode(dir), inode)) {
- pr_warn_ratelimited("fscrypt: inconsistent encryption contexts: %lu/%lu",
- d_inode(dir)->i_ino, inode->i_ino);
+ fscrypt_warn(inode->i_sb,
+ "inconsistent encryption contexts: %lu/%lu",
+ d_inode(dir)->i_ino, inode->i_ino);
err = -EPERM;
}
dput(dir);
diff --git a/fs/crypto/keyinfo.c b/fs/crypto/keyinfo.c
index b824f84..2425d83 100644
--- a/fs/crypto/keyinfo.c
+++ b/fs/crypto/keyinfo.c
@@ -9,9 +9,11 @@
*/
#include <keys/user-type.h>
+#include <linux/hashtable.h>
#include <linux/scatterlist.h>
#include <linux/ratelimit.h>
#include <crypto/aes.h>
+#include <crypto/algapi.h>
#include <crypto/sha.h>
#include <crypto/skcipher.h>
#include "fscrypt_private.h"
@@ -19,17 +21,20 @@
static struct crypto_shash *essiv_hash_tfm;
-/**
- * derive_key_aes() - Derive a key using AES-128-ECB
- * @deriving_key: Encryption key used for derivation.
- * @source_key: Source key to which to apply derivation.
- * @derived_raw_key: Derived raw key.
+/* Table of keys referenced by FS_POLICY_FLAG_DIRECT_KEY policies */
+static DEFINE_HASHTABLE(fscrypt_master_keys, 6); /* 6 bits = 64 buckets */
+static DEFINE_SPINLOCK(fscrypt_master_keys_lock);
+
+/*
+ * Key derivation function. This generates the derived key by encrypting the
+ * master key with AES-128-ECB using the inode's nonce as the AES key.
*
- * Return: Zero on success; non-zero otherwise.
+ * The master key must be at least as long as the derived key. If the master
+ * key is longer, then only the first 'derived_keysize' bytes are used.
*/
-static int derive_key_aes(u8 deriving_key[FS_AES_128_ECB_KEY_SIZE],
- const struct fscrypt_key *source_key,
- u8 derived_raw_key[FS_MAX_KEY_SIZE])
+static int derive_key_aes(const u8 *master_key,
+ const struct fscrypt_context *ctx,
+ u8 *derived_key, unsigned int derived_keysize)
{
int res = 0;
struct skcipher_request *req = NULL;
@@ -51,14 +56,13 @@ static int derive_key_aes(u8 deriving_key[FS_AES_128_ECB_KEY_SIZE],
skcipher_request_set_callback(req,
CRYPTO_TFM_REQ_MAY_BACKLOG | CRYPTO_TFM_REQ_MAY_SLEEP,
crypto_req_done, &wait);
- res = crypto_skcipher_setkey(tfm, deriving_key,
- FS_AES_128_ECB_KEY_SIZE);
+ res = crypto_skcipher_setkey(tfm, ctx->nonce, sizeof(ctx->nonce));
if (res < 0)
goto out;
- sg_init_one(&src_sg, source_key->raw, source_key->size);
- sg_init_one(&dst_sg, derived_raw_key, source_key->size);
- skcipher_request_set_crypt(req, &src_sg, &dst_sg, source_key->size,
+ sg_init_one(&src_sg, master_key, derived_keysize);
+ sg_init_one(&dst_sg, derived_key, derived_keysize);
+ skcipher_request_set_crypt(req, &src_sg, &dst_sg, derived_keysize,
NULL);
res = crypto_wait_req(crypto_skcipher_encrypt(req), &wait);
out:
@@ -67,134 +71,323 @@ static int derive_key_aes(u8 deriving_key[FS_AES_128_ECB_KEY_SIZE],
return res;
}
-static int validate_user_key(struct fscrypt_info *crypt_info,
- struct fscrypt_context *ctx,
- const char *prefix, int min_keysize)
+/*
+ * Search the current task's subscribed keyrings for a "logon" key with
+ * description prefix:descriptor, and if found acquire a read lock on it and
+ * return a pointer to its validated payload in *payload_ret.
+ */
+static struct key *
+find_and_lock_process_key(const char *prefix,
+ const u8 descriptor[FS_KEY_DESCRIPTOR_SIZE],
+ unsigned int min_keysize,
+ const struct fscrypt_key **payload_ret)
{
char *description;
- struct key *keyring_key;
- struct fscrypt_key *master_key;
+ struct key *key;
const struct user_key_payload *ukp;
- int res;
+ const struct fscrypt_key *payload;
description = kasprintf(GFP_NOFS, "%s%*phN", prefix,
- FS_KEY_DESCRIPTOR_SIZE,
- ctx->master_key_descriptor);
+ FS_KEY_DESCRIPTOR_SIZE, descriptor);
if (!description)
- return -ENOMEM;
+ return ERR_PTR(-ENOMEM);
- keyring_key = request_key(&key_type_logon, description, NULL);
+ key = request_key(&key_type_logon, description, NULL);
kfree(description);
- if (IS_ERR(keyring_key))
- return PTR_ERR(keyring_key);
- down_read(&keyring_key->sem);
+ if (IS_ERR(key))
+ return key;
- if (keyring_key->type != &key_type_logon) {
- printk_once(KERN_WARNING
- "%s: key type must be logon\n", __func__);
- res = -ENOKEY;
- goto out;
- }
- ukp = user_key_payload_locked(keyring_key);
- if (!ukp) {
- /* key was revoked before we acquired its semaphore */
- res = -EKEYREVOKED;
- goto out;
- }
- if (ukp->datalen != sizeof(struct fscrypt_key)) {
- res = -EINVAL;
- goto out;
- }
- master_key = (struct fscrypt_key *)ukp->data;
- BUILD_BUG_ON(FS_AES_128_ECB_KEY_SIZE != FS_KEY_DERIVATION_NONCE_SIZE);
+ down_read(&key->sem);
+ ukp = user_key_payload_locked(key);
- if (master_key->size < min_keysize || master_key->size > FS_MAX_KEY_SIZE
- || master_key->size % AES_BLOCK_SIZE != 0) {
- printk_once(KERN_WARNING
- "%s: key size incorrect: %d\n",
- __func__, master_key->size);
- res = -ENOKEY;
- goto out;
- }
- res = derive_key_aes(ctx->nonce, master_key, crypt_info->ci_raw_key);
- /* If we don't need to derive, we still want to do everything
- * up until now to validate the key. It's cleaner to fail now
- * than to fail in block I/O.
- if (!is_private_data_mode(crypt_info)) {
- res = derive_key_aes(ctx->nonce, master_key,
- crypt_info->ci_raw_key);
- } else {
- * Inline encryption: no key derivation required because IVs are
- * assigned based on iv_sector.
+ if (!ukp) /* was the key revoked before we acquired its semaphore? */
+ goto invalid;
- BUILD_BUG_ON(sizeof(crypt_info->ci_raw_key) !=
- sizeof(master_key->raw));
- memcpy(crypt_info->ci_raw_key,
- master_key->raw, sizeof(crypt_info->ci_raw_key));
- res = 0;
+ payload = (const struct fscrypt_key *)ukp->data;
+
+ if (ukp->datalen != sizeof(struct fscrypt_key) ||
+ payload->size < 1 || payload->size > FS_MAX_KEY_SIZE) {
+ fscrypt_warn(NULL,
+ "key with description '%s' has invalid payload",
+ key->description);
+ goto invalid;
}
- */
-out:
- up_read(&keyring_key->sem);
- key_put(keyring_key);
- return res;
+
+ if (payload->size < min_keysize) {
+ fscrypt_warn(NULL,
+ "key with description '%s' is too short (got %u bytes, need %u+ bytes)",
+ key->description, payload->size, min_keysize);
+ goto invalid;
+ }
+
+ *payload_ret = payload;
+ return key;
+
+invalid:
+ up_read(&key->sem);
+ key_put(key);
+ return ERR_PTR(-ENOKEY);
}
-static const struct {
- const char *cipher_str;
- int keysize;
-} available_modes[] = {
- [FS_ENCRYPTION_MODE_AES_256_XTS] = { "xts(aes)",
- FS_AES_256_XTS_KEY_SIZE },
- [FS_ENCRYPTION_MODE_AES_256_CTS] = { "cts(cbc(aes))",
- FS_AES_256_CTS_KEY_SIZE },
- [FS_ENCRYPTION_MODE_AES_128_CBC] = { "cbc(aes)",
- FS_AES_128_CBC_KEY_SIZE },
- [FS_ENCRYPTION_MODE_AES_128_CTS] = { "cts(cbc(aes))",
- FS_AES_128_CTS_KEY_SIZE },
- [FS_ENCRYPTION_MODE_PRIVATE] = { "bugon",
- FS_AES_256_XTS_KEY_SIZE },
+static struct fscrypt_mode available_modes[] = {
+ [FS_ENCRYPTION_MODE_AES_256_XTS] = {
+ .friendly_name = "AES-256-XTS",
+ .cipher_str = "xts(aes)",
+ .keysize = 64,
+ .ivsize = 16,
+ },
+ [FS_ENCRYPTION_MODE_AES_256_CTS] = {
+ .friendly_name = "AES-256-CTS-CBC",
+ .cipher_str = "cts(cbc(aes))",
+ .keysize = 32,
+ .ivsize = 16,
+ },
+ [FS_ENCRYPTION_MODE_AES_128_CBC] = {
+ .friendly_name = "AES-128-CBC",
+ .cipher_str = "cbc(aes)",
+ .keysize = 16,
+ .ivsize = 16,
+ .needs_essiv = true,
+ },
+ [FS_ENCRYPTION_MODE_AES_128_CTS] = {
+ .friendly_name = "AES-128-CTS-CBC",
+ .cipher_str = "cts(cbc(aes))",
+ .keysize = 16,
+ .ivsize = 16,
+ },
+ [FS_ENCRYPTION_MODE_ADIANTUM] = {
+ .friendly_name = "Adiantum",
+ .cipher_str = "adiantum(xchacha12,aes)",
+ .keysize = 32,
+ .ivsize = 32,
+ },
+ [FS_ENCRYPTION_MODE_PRIVATE] = {
+ .friendly_name = "Inline encryption (AES-256-XTS)",
+ .cipher_str = NULL,
+ .keysize = 64,
+ .ivsize = 16,
+ },
};
-static int determine_cipher_type(struct fscrypt_info *ci, struct inode *inode,
- const char **cipher_str_ret, int *keysize_ret, int *fname)
+static struct fscrypt_mode *
+select_encryption_mode(const struct fscrypt_info *ci, const struct inode *inode)
{
- u32 mode;
-
if (!fscrypt_valid_enc_modes(ci->ci_data_mode, ci->ci_filename_mode)) {
- pr_warn_ratelimited("fscrypt: inode %lu uses unsupported encryption modes (contents mode %d, filenames mode %d)\n",
- inode->i_ino,
- ci->ci_data_mode, ci->ci_filename_mode);
- return -EINVAL;
+ fscrypt_warn(inode->i_sb,
+ "inode %lu uses unsupported encryption modes (contents mode %d, filenames mode %d)",
+ inode->i_ino, ci->ci_data_mode,
+ ci->ci_filename_mode);
+ return ERR_PTR(-EINVAL);
}
- if (S_ISREG(inode->i_mode)) {
- ci->ci_mode = CI_DATA_MODE;
- mode = ci->ci_data_mode;
- } else if (S_ISDIR(inode->i_mode) || S_ISLNK(inode->i_mode)) {
- ci->ci_mode = CI_FNAME_MODE;
- mode = ci->ci_filename_mode;
- *fname = 1;
- } else {
- WARN_ONCE(1, "fscrypt: filesystem tried to load encryption info for inode %lu, which is not encryptable (file type %d)\n",
- inode->i_ino, (inode->i_mode & S_IFMT));
- return -EINVAL;
- }
+ if (S_ISREG(inode->i_mode))
+ return &available_modes[ci->ci_data_mode];
- *cipher_str_ret = available_modes[mode].cipher_str;
- *keysize_ret = available_modes[mode].keysize;
- return 0;
+ if (S_ISDIR(inode->i_mode) || S_ISLNK(inode->i_mode))
+ return &available_modes[ci->ci_filename_mode];
+
+ WARN_ONCE(1, "fscrypt: filesystem tried to load encryption info for inode %lu, which is not encryptable (file type %d)\n",
+ inode->i_ino, (inode->i_mode & S_IFMT));
+ return ERR_PTR(-EINVAL);
}
-static void put_crypt_info(struct fscrypt_info *ci)
+/* Find the master key, then derive the inode's actual encryption key */
+static int find_and_derive_key(const struct inode *inode,
+ const struct fscrypt_context *ctx,
+ u8 *derived_key, const struct fscrypt_mode *mode)
{
- if (!ci)
- return;
+ struct key *key;
+ const struct fscrypt_key *payload;
+ int err;
- crypto_free_skcipher(ci->ci_ctfm);
- crypto_free_cipher(ci->ci_essiv_tfm);
- memset(ci, 0, sizeof(*ci)); /* sanitizes ->ci_raw_key */
- kmem_cache_free(fscrypt_info_cachep, ci);
+ key = find_and_lock_process_key(FS_KEY_DESC_PREFIX,
+ ctx->master_key_descriptor,
+ mode->keysize, &payload);
+ if (key == ERR_PTR(-ENOKEY) && inode->i_sb->s_cop->key_prefix) {
+ key = find_and_lock_process_key(inode->i_sb->s_cop->key_prefix,
+ ctx->master_key_descriptor,
+ mode->keysize, &payload);
+ }
+ if (IS_ERR(key))
+ return PTR_ERR(key);
+
+ if (is_private_mode(mode)) {
+ /*
+ * Inline encryption: no key derivation required because IVs are
+ * assigned based on iv_sector.
+ */
+ memcpy(derived_key, payload->raw, mode->keysize);
+ err = 0;
+ } else if (ctx->flags & FS_POLICY_FLAG_DIRECT_KEY) {
+ if (mode->ivsize < offsetofend(union fscrypt_iv, nonce)) {
+ fscrypt_warn(inode->i_sb,
+ "direct key mode not allowed with %s",
+ mode->friendly_name);
+ err = -EINVAL;
+ } else if (ctx->contents_encryption_mode !=
+ ctx->filenames_encryption_mode) {
+ fscrypt_warn(inode->i_sb,
+ "direct key mode not allowed with different contents and filenames modes");
+ err = -EINVAL;
+ } else {
+ memcpy(derived_key, payload->raw, mode->keysize);
+ err = 0;
+ }
+ } else {
+ err = derive_key_aes(payload->raw, ctx, derived_key,
+ mode->keysize);
+ }
+ up_read(&key->sem);
+ key_put(key);
+ return err;
+}
+
+/* Allocate and key a symmetric cipher object for the given encryption mode */
+static struct crypto_skcipher *
+allocate_skcipher_for_mode(struct fscrypt_mode *mode, const u8 *raw_key,
+ const struct inode *inode)
+{
+ struct crypto_skcipher *tfm;
+ int err;
+
+ tfm = crypto_alloc_skcipher(mode->cipher_str, 0, 0);
+ if (IS_ERR(tfm)) {
+ fscrypt_warn(inode->i_sb,
+ "error allocating '%s' transform for inode %lu: %ld",
+ mode->cipher_str, inode->i_ino, PTR_ERR(tfm));
+ return tfm;
+ }
+ if (unlikely(!mode->logged_impl_name)) {
+ /*
+ * fscrypt performance can vary greatly depending on which
+ * crypto algorithm implementation is used. Help people debug
+ * performance problems by logging the ->cra_driver_name the
+ * first time a mode is used. Note that multiple threads can
+ * race here, but it doesn't really matter.
+ */
+ mode->logged_impl_name = true;
+ pr_info("fscrypt: %s using implementation \"%s\"\n",
+ mode->friendly_name,
+ crypto_skcipher_alg(tfm)->base.cra_driver_name);
+ }
+ crypto_skcipher_set_flags(tfm, CRYPTO_TFM_REQ_WEAK_KEY);
+ err = crypto_skcipher_setkey(tfm, raw_key, mode->keysize);
+ if (err)
+ goto err_free_tfm;
+
+ return tfm;
+
+err_free_tfm:
+ crypto_free_skcipher(tfm);
+ return ERR_PTR(err);
+}
+
+/* Master key referenced by FS_POLICY_FLAG_DIRECT_KEY policy */
+struct fscrypt_master_key {
+ struct hlist_node mk_node;
+ atomic_t mk_refcount;
+ const struct fscrypt_mode *mk_mode;
+ struct crypto_skcipher *mk_ctfm;
+ u8 mk_descriptor[FS_KEY_DESCRIPTOR_SIZE];
+ u8 mk_raw[FS_MAX_KEY_SIZE];
+};
+
+static void free_master_key(struct fscrypt_master_key *mk)
+{
+ if (mk) {
+ crypto_free_skcipher(mk->mk_ctfm);
+ kzfree(mk);
+ }
+}
+
+static void put_master_key(struct fscrypt_master_key *mk)
+{
+ if (!atomic_dec_and_lock(&mk->mk_refcount, &fscrypt_master_keys_lock))
+ return;
+ hash_del(&mk->mk_node);
+ spin_unlock(&fscrypt_master_keys_lock);
+
+ free_master_key(mk);
+}
+
+/*
+ * Find/insert the given master key into the fscrypt_master_keys table. If
+ * found, it is returned with elevated refcount, and 'to_insert' is freed if
+ * non-NULL. If not found, 'to_insert' is inserted and returned if it's
+ * non-NULL; otherwise NULL is returned.
+ */
+static struct fscrypt_master_key *
+find_or_insert_master_key(struct fscrypt_master_key *to_insert,
+ const u8 *raw_key, const struct fscrypt_mode *mode,
+ const struct fscrypt_info *ci)
+{
+ unsigned long hash_key;
+ struct fscrypt_master_key *mk;
+
+ /*
+ * Careful: to avoid potentially leaking secret key bytes via timing
+ * information, we must key the hash table by descriptor rather than by
+ * raw key, and use crypto_memneq() when comparing raw keys.
+ */
+
+ BUILD_BUG_ON(sizeof(hash_key) > FS_KEY_DESCRIPTOR_SIZE);
+ memcpy(&hash_key, ci->ci_master_key_descriptor, sizeof(hash_key));
+
+ spin_lock(&fscrypt_master_keys_lock);
+ hash_for_each_possible(fscrypt_master_keys, mk, mk_node, hash_key) {
+ if (memcmp(ci->ci_master_key_descriptor, mk->mk_descriptor,
+ FS_KEY_DESCRIPTOR_SIZE) != 0)
+ continue;
+ if (mode != mk->mk_mode)
+ continue;
+ if (crypto_memneq(raw_key, mk->mk_raw, mode->keysize))
+ continue;
+ /* using existing tfm with same (descriptor, mode, raw_key) */
+ atomic_inc(&mk->mk_refcount);
+ spin_unlock(&fscrypt_master_keys_lock);
+ free_master_key(to_insert);
+ return mk;
+ }
+ if (to_insert)
+ hash_add(fscrypt_master_keys, &to_insert->mk_node, hash_key);
+ spin_unlock(&fscrypt_master_keys_lock);
+ return to_insert;
+}
+
+/* Prepare to encrypt directly using the master key in the given mode */
+static struct fscrypt_master_key *
+fscrypt_get_master_key(const struct fscrypt_info *ci, struct fscrypt_mode *mode,
+ const u8 *raw_key, const struct inode *inode)
+{
+ struct fscrypt_master_key *mk;
+ int err;
+
+ /* Is there already a tfm for this key? */
+ mk = find_or_insert_master_key(NULL, raw_key, mode, ci);
+ if (mk)
+ return mk;
+
+ /* Nope, allocate one. */
+ mk = kzalloc(sizeof(*mk), GFP_NOFS);
+ if (!mk)
+ return ERR_PTR(-ENOMEM);
+ atomic_set(&mk->mk_refcount, 1);
+ mk->mk_mode = mode;
+ mk->mk_ctfm = allocate_skcipher_for_mode(mode, raw_key, inode);
+ if (IS_ERR(mk->mk_ctfm)) {
+ err = PTR_ERR(mk->mk_ctfm);
+ mk->mk_ctfm = NULL;
+ goto err_free_mk;
+ }
+ memcpy(mk->mk_descriptor, ci->ci_master_key_descriptor,
+ FS_KEY_DESCRIPTOR_SIZE);
+ memcpy(mk->mk_raw, raw_key, mode->keysize);
+
+ return find_or_insert_master_key(mk, raw_key, mode, ci);
+
+err_free_mk:
+ free_master_key(mk);
+ return ERR_PTR(err);
}
static int derive_essiv_salt(const u8 *key, int keysize, u8 *salt)
@@ -207,8 +400,9 @@ static int derive_essiv_salt(const u8 *key, int keysize, u8 *salt)
tfm = crypto_alloc_shash("sha256", 0, 0);
if (IS_ERR(tfm)) {
- pr_warn_ratelimited("fscrypt: error allocating SHA-256 transform: %ld\n",
- PTR_ERR(tfm));
+ fscrypt_warn(NULL,
+ "error allocating SHA-256 transform: %ld",
+ PTR_ERR(tfm));
return PTR_ERR(tfm);
}
prev_tfm = cmpxchg(&essiv_hash_tfm, NULL, tfm);
@@ -263,21 +457,71 @@ void __exit fscrypt_essiv_cleanup(void)
crypto_free_shash(essiv_hash_tfm);
}
-static int fscrypt_data_encryption_mode(struct inode *inode)
+/*
+ * Given the encryption mode and key (normally the derived key, but for
+ * FS_POLICY_FLAG_DIRECT_KEY mode it's the master key), set up the inode's
+ * symmetric cipher transform object(s).
+ */
+static int setup_crypto_transform(struct fscrypt_info *ci,
+ struct fscrypt_mode *mode,
+ const u8 *raw_key, const struct inode *inode)
{
- return fscrypt_is_ice_capable(inode->i_sb) ?
- FS_ENCRYPTION_MODE_PRIVATE : FS_ENCRYPTION_MODE_AES_256_XTS;
+ struct fscrypt_master_key *mk;
+ struct crypto_skcipher *ctfm;
+ int err;
+
+ if (ci->ci_flags & FS_POLICY_FLAG_DIRECT_KEY) {
+ mk = fscrypt_get_master_key(ci, mode, raw_key, inode);
+ if (IS_ERR(mk))
+ return PTR_ERR(mk);
+ ctfm = mk->mk_ctfm;
+ } else {
+ mk = NULL;
+ ctfm = allocate_skcipher_for_mode(mode, raw_key, inode);
+ if (IS_ERR(ctfm))
+ return PTR_ERR(ctfm);
+ }
+ ci->ci_master_key = mk;
+ ci->ci_ctfm = ctfm;
+
+ if (mode->needs_essiv) {
+ /* ESSIV implies 16-byte IVs which implies !DIRECT_KEY */
+ WARN_ON(mode->ivsize != AES_BLOCK_SIZE);
+ WARN_ON(ci->ci_flags & FS_POLICY_FLAG_DIRECT_KEY);
+
+ err = init_essiv_generator(ci, raw_key, mode->keysize);
+ if (err) {
+ fscrypt_warn(inode->i_sb,
+ "error initializing ESSIV generator for inode %lu: %d",
+ inode->i_ino, err);
+ return err;
+ }
+ }
+ return 0;
+}
+
+static void put_crypt_info(struct fscrypt_info *ci)
+{
+ if (!ci)
+ return;
+
+ if (ci->ci_master_key) {
+ put_master_key(ci->ci_master_key);
+ } else {
+ crypto_free_skcipher(ci->ci_ctfm);
+ crypto_free_cipher(ci->ci_essiv_tfm);
+ }
+ memset(ci->ci_raw_key, 0, FS_MAX_KEY_SIZE);
+ kmem_cache_free(fscrypt_info_cachep, ci);
}
int fscrypt_get_encryption_info(struct inode *inode)
{
struct fscrypt_info *crypt_info;
struct fscrypt_context ctx;
- struct crypto_skcipher *ctfm;
- const char *cipher_str;
- int keysize;
+ struct fscrypt_mode *mode;
+ u8 *raw_key = NULL;
int res;
- int fname = 0;
if (inode->i_crypt_info)
return 0;
@@ -294,8 +538,7 @@ int fscrypt_get_encryption_info(struct inode *inode)
/* Fake up a context for an unencrypted directory */
memset(&ctx, 0, sizeof(ctx));
ctx.format = FS_ENCRYPTION_CONTEXT_FORMAT_V1;
- ctx.contents_encryption_mode =
- fscrypt_data_encryption_mode(inode);
+ ctx.contents_encryption_mode = FS_ENCRYPTION_MODE_AES_256_XTS;
ctx.filenames_encryption_mode = FS_ENCRYPTION_MODE_AES_256_CTS;
memset(ctx.master_key_descriptor, 0x42, FS_KEY_DESCRIPTOR_SIZE);
} else if (res != sizeof(ctx)) {
@@ -308,94 +551,61 @@ int fscrypt_get_encryption_info(struct inode *inode)
if (ctx.flags & ~FS_POLICY_FLAGS_VALID)
return -EINVAL;
- crypt_info = kmem_cache_alloc(fscrypt_info_cachep, GFP_NOFS);
+ crypt_info = kmem_cache_zalloc(fscrypt_info_cachep, GFP_NOFS);
if (!crypt_info)
return -ENOMEM;
crypt_info->ci_flags = ctx.flags;
crypt_info->ci_data_mode = ctx.contents_encryption_mode;
crypt_info->ci_filename_mode = ctx.filenames_encryption_mode;
- crypt_info->ci_ctfm = NULL;
- crypt_info->ci_essiv_tfm = NULL;
- memcpy(crypt_info->ci_master_key, ctx.master_key_descriptor,
- sizeof(crypt_info->ci_master_key));
+ memcpy(crypt_info->ci_master_key_descriptor, ctx.master_key_descriptor,
+ FS_KEY_DESCRIPTOR_SIZE);
+ memcpy(crypt_info->ci_nonce, ctx.nonce, FS_KEY_DERIVATION_NONCE_SIZE);
- res = determine_cipher_type(crypt_info, inode, &cipher_str, &keysize,
- &fname);
+ mode = select_encryption_mode(crypt_info, inode);
+ if (IS_ERR(mode)) {
+ res = PTR_ERR(mode);
+ goto out;
+ }
+ WARN_ON(mode->ivsize > FSCRYPT_MAX_IV_SIZE);
+ crypt_info->ci_mode = mode;
+
+ /*
+ * This cannot be a stack buffer because it may be passed to the
+ * scatterlist crypto API as part of key derivation.
+ */
+ res = -ENOMEM;
+ raw_key = kmalloc(mode->keysize, GFP_NOFS);
+ if (!raw_key)
+ goto out;
+
+ res = find_and_derive_key(inode, &ctx, raw_key, mode);
if (res)
goto out;
- /*
- * This cannot be a stack buffer because it is passed to the scatterlist
- * crypto API as part of key derivation.
- */
- res = -ENOMEM;
-
- res = validate_user_key(crypt_info, &ctx, FS_KEY_DESC_PREFIX,
- keysize);
- if (res && inode->i_sb->s_cop->key_prefix) {
- int res2 = validate_user_key(crypt_info, &ctx,
- inode->i_sb->s_cop->key_prefix,
- keysize);
- if (res2) {
- if (res2 == -ENOKEY)
- res = -ENOKEY;
- goto out;
- }
- res = 0;
- } else if (res) {
- goto out;
- }
-
- if (is_private_data_mode(crypt_info)) {
+ if (is_private_mode(crypt_info->ci_mode)) {
if (!fscrypt_is_ice_capable(inode->i_sb)) {
- pr_warn("%s: ICE support not available\n",
- __func__);
+ fscrypt_warn(inode->i_sb, "ICE support not available");
res = -EINVAL;
goto out;
}
/* Let's encrypt/decrypt by ICE */
- goto do_ice;
+ memcpy(crypt_info->ci_raw_key, raw_key, mode->keysize);
+ goto done;
}
-
- ctfm = crypto_alloc_skcipher(cipher_str, 0, 0);
- if (!ctfm || IS_ERR(ctfm)) {
- res = ctfm ? PTR_ERR(ctfm) : -ENOMEM;
- pr_debug("%s: error %d (inode %lu) allocating crypto tfm\n",
- __func__, res, inode->i_ino);
- goto out;
- }
- crypt_info->ci_ctfm = ctfm;
- crypto_skcipher_clear_flags(ctfm, ~0);
- crypto_skcipher_set_flags(ctfm, CRYPTO_TFM_REQ_WEAK_KEY);
- /*
- * if the provided key is longer than keysize, we use the first
- * keysize bytes of the derived key only
- */
- res = crypto_skcipher_setkey(ctfm, crypt_info->ci_raw_key, keysize);
+ res = setup_crypto_transform(crypt_info, mode, raw_key, inode);
if (res)
goto out;
- if (S_ISREG(inode->i_mode) &&
- crypt_info->ci_data_mode == FS_ENCRYPTION_MODE_AES_128_CBC) {
- res = init_essiv_generator(crypt_info, crypt_info->ci_raw_key,
- keysize);
- if (res) {
- pr_debug("%s: error %d (inode %lu) allocating essiv tfm\n",
- __func__, res, inode->i_ino);
- goto out;
- }
- }
- memzero_explicit(crypt_info->ci_raw_key,
- sizeof(crypt_info->ci_raw_key));
-do_ice:
+done:
if (cmpxchg(&inode->i_crypt_info, NULL, crypt_info) == NULL)
crypt_info = NULL;
out:
if (res == -ENOKEY)
res = 0;
put_crypt_info(crypt_info);
+ kzfree(raw_key);
return res;
}
EXPORT_SYMBOL(fscrypt_get_encryption_info);
diff --git a/fs/crypto/policy.c b/fs/crypto/policy.c
index 2f2c53f..6a9c476 100644
--- a/fs/crypto/policy.c
+++ b/fs/crypto/policy.c
@@ -198,7 +198,8 @@ int fscrypt_has_permitted_context(struct inode *parent, struct inode *child)
child_ci = child->i_crypt_info;
if (parent_ci && child_ci) {
- return memcmp(parent_ci->ci_master_key, child_ci->ci_master_key,
+ return memcmp(parent_ci->ci_master_key_descriptor,
+ child_ci->ci_master_key_descriptor,
FS_KEY_DESCRIPTOR_SIZE) == 0 &&
(parent_ci->ci_data_mode == child_ci->ci_data_mode) &&
(parent_ci->ci_filename_mode ==
@@ -253,7 +254,7 @@ int fscrypt_inherit_context(struct inode *parent, struct inode *child,
ctx.contents_encryption_mode = ci->ci_data_mode;
ctx.filenames_encryption_mode = ci->ci_filename_mode;
ctx.flags = ci->ci_flags;
- memcpy(ctx.master_key_descriptor, ci->ci_master_key,
+ memcpy(ctx.master_key_descriptor, ci->ci_master_key_descriptor,
FS_KEY_DESCRIPTOR_SIZE);
get_random_bytes(ctx.nonce, FS_KEY_DERIVATION_NONCE_SIZE);
res = parent->i_sb->s_cop->set_context(child, &ctx,
diff --git a/fs/dcache.c b/fs/dcache.c
index 885f74e..183e0ae 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -286,7 +286,8 @@ void take_dentry_name_snapshot(struct name_snapshot *name, struct dentry *dentry
spin_unlock(&dentry->d_lock);
name->name = p->name;
} else {
- memcpy(name->inline_name, dentry->d_iname, DNAME_INLINE_LEN);
+ memcpy(name->inline_name, dentry->d_iname,
+ dentry->d_name.len + 1);
spin_unlock(&dentry->d_lock);
name->name = name->inline_name;
}
@@ -352,14 +353,11 @@ static void dentry_unlink_inode(struct dentry * dentry)
__releases(dentry->d_inode->i_lock)
{
struct inode *inode = dentry->d_inode;
- bool hashed = !d_unhashed(dentry);
- if (hashed)
- raw_write_seqcount_begin(&dentry->d_seq);
+ raw_write_seqcount_begin(&dentry->d_seq);
__d_clear_type_and_inode(dentry);
hlist_del_init(&dentry->d_u.d_alias);
- if (hashed)
- raw_write_seqcount_end(&dentry->d_seq);
+ raw_write_seqcount_end(&dentry->d_seq);
spin_unlock(&dentry->d_lock);
spin_unlock(&inode->i_lock);
if (!inode->i_nlink)
@@ -1859,6 +1857,28 @@ void d_instantiate(struct dentry *entry, struct inode * inode)
}
EXPORT_SYMBOL(d_instantiate);
+/*
+ * This should be equivalent to d_instantiate() + unlock_new_inode(),
+ * with lockdep-related part of unlock_new_inode() done before
+ * anything else. Use that instead of open-coding d_instantiate()/
+ * unlock_new_inode() combinations.
+ */
+void d_instantiate_new(struct dentry *entry, struct inode *inode)
+{
+ BUG_ON(!hlist_unhashed(&entry->d_u.d_alias));
+ BUG_ON(!inode);
+ lockdep_annotate_inode_mutex_key(inode);
+ security_d_instantiate(entry, inode);
+ spin_lock(&inode->i_lock);
+ __d_instantiate(entry, inode);
+ WARN_ON(!(inode->i_state & I_NEW));
+ inode->i_state &= ~I_NEW;
+ smp_mb();
+ wake_up_bit(&inode->i_state, __I_NEW);
+ spin_unlock(&inode->i_lock);
+}
+EXPORT_SYMBOL(d_instantiate_new);
+
/**
* d_instantiate_no_diralias - instantiate a non-aliased dentry
* @entry: dentry to complete
@@ -1892,10 +1912,12 @@ struct dentry *d_make_root(struct inode *root_inode)
if (root_inode) {
res = __d_alloc(root_inode->i_sb, NULL);
- if (res)
+ if (res) {
+ res->d_flags |= DCACHE_RCUACCESS;
d_instantiate(res, root_inode);
- else
+ } else {
iput(root_inode);
+ }
}
return res;
}
@@ -2452,7 +2474,7 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
retry:
rcu_read_lock();
- seq = smp_load_acquire(&parent->d_inode->i_dir_seq) & ~1;
+ seq = smp_load_acquire(&parent->d_inode->i_dir_seq);
r_seq = read_seqbegin(&rename_lock);
dentry = __d_lookup_rcu(parent, name, &d_seq);
if (unlikely(dentry)) {
@@ -2473,8 +2495,14 @@ struct dentry *d_alloc_parallel(struct dentry *parent,
rcu_read_unlock();
goto retry;
}
+
+ if (unlikely(seq & 1)) {
+ rcu_read_unlock();
+ goto retry;
+ }
+
hlist_bl_lock(b);
- if (unlikely(parent->d_inode->i_dir_seq != seq)) {
+ if (unlikely(READ_ONCE(parent->d_inode->i_dir_seq) != seq)) {
hlist_bl_unlock(b);
rcu_read_unlock();
goto retry;
diff --git a/fs/direct-io.c b/fs/direct-io.c
index 7f5d2e3..e3d0820 100644
--- a/fs/direct-io.c
+++ b/fs/direct-io.c
@@ -280,8 +280,8 @@ static ssize_t dio_complete(struct dio *dio, ssize_t ret, bool is_async)
*/
dio->iocb->ki_pos += transferred;
- if (dio->op == REQ_OP_WRITE)
- ret = generic_write_sync(dio->iocb, transferred);
+ if (ret > 0 && dio->op == REQ_OP_WRITE)
+ ret = generic_write_sync(dio->iocb, ret);
dio->iocb->ki_complete(dio->iocb, ret, 0);
}
@@ -433,12 +433,12 @@ static inline void dio_bio_submit(struct dio *dio, struct dio_submit *sdio)
#ifdef CONFIG_PFK
bio->bi_dio_inode = dio->inode;
-/* iv sector for security/pfe/pfk_fscrypt.c and f2fs in fs/f2fs/f2fs.h.*/
-#define PG_DUN_NEW(i,p) \
- (((((u64)(i)->i_ino) & 0xffffffff) << 32) | ((p) & 0xffffffff))
+/* iv sector for security/pfe/pfk_fscrypt.c and f2fs in fs/f2fs/f2fs.h */
+#define PG_DUN(i,p) \
+ ((((i)->i_ino & 0xffffffff) << 32) | ((p) & 0xffffffff))
if (is_inode_filesystem_type(dio->inode, "f2fs"))
- fscrypt_set_ice_dun(dio->inode, bio, PG_DUN_NEW(dio->inode,
+ fscrypt_set_ice_dun(dio->inode, bio, PG_DUN(dio->inode,
(sdio->logical_offset_in_bio >> PAGE_SHIFT)));
#endif
dio->bio_bdev = bio->bi_bdev;
@@ -468,6 +468,7 @@ struct inode *dio_bio_get_inode(struct bio *bio)
return inode;
}
EXPORT_SYMBOL(dio_bio_get_inode);
+
/*
* Release any resources in case of a failure
*/
diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
index 35502d4..3a7f401 100644
--- a/fs/dlm/lock.c
+++ b/fs/dlm/lock.c
@@ -1210,6 +1210,7 @@ static int create_lkb(struct dlm_ls *ls, struct dlm_lkb **lkb_ret)
if (rv < 0) {
log_error(ls, "create_lkb idr error %d", rv);
+ dlm_free_lkb(lkb);
return rv;
}
@@ -4177,6 +4178,7 @@ static int receive_convert(struct dlm_ls *ls, struct dlm_message *ms)
(unsigned long long)lkb->lkb_recover_seq,
ms->m_header.h_nodeid, ms->m_lkid);
error = -ENOENT;
+ dlm_put_lkb(lkb);
goto fail;
}
@@ -4230,6 +4232,7 @@ static int receive_unlock(struct dlm_ls *ls, struct dlm_message *ms)
lkb->lkb_id, lkb->lkb_remid,
ms->m_header.h_nodeid, ms->m_lkid);
error = -ENOENT;
+ dlm_put_lkb(lkb);
goto fail;
}
@@ -5792,20 +5795,20 @@ int dlm_user_request(struct dlm_ls *ls, struct dlm_user_args *ua,
goto out;
}
}
-
- /* After ua is attached to lkb it will be freed by dlm_free_lkb().
- When DLM_IFL_USER is set, the dlm knows that this is a userspace
- lock and that lkb_astparam is the dlm_user_args structure. */
-
error = set_lock_args(mode, &ua->lksb, flags, namelen, timeout_cs,
fake_astfn, ua, fake_bastfn, &args);
- lkb->lkb_flags |= DLM_IFL_USER;
-
if (error) {
+ kfree(ua->lksb.sb_lvbptr);
+ ua->lksb.sb_lvbptr = NULL;
+ kfree(ua);
__put_lkb(ls, lkb);
goto out;
}
+ /* After ua is attached to lkb it will be freed by dlm_free_lkb().
+ When DLM_IFL_USER is set, the dlm knows that this is a userspace
+ lock and that lkb_astparam is the dlm_user_args structure. */
+ lkb->lkb_flags |= DLM_IFL_USER;
error = request_lock(ls, lkb, name, namelen, &args);
switch (error) {
diff --git a/fs/dlm/lockspace.c b/fs/dlm/lockspace.c
index f3e7278..30e4e01 100644
--- a/fs/dlm/lockspace.c
+++ b/fs/dlm/lockspace.c
@@ -673,11 +673,11 @@ static int new_lockspace(const char *name, const char *cluster,
kfree(ls->ls_recover_buf);
out_lkbidr:
idr_destroy(&ls->ls_lkbidr);
+ out_rsbtbl:
for (i = 0; i < DLM_REMOVE_NAMES_MAX; i++) {
if (ls->ls_remove_names[i])
kfree(ls->ls_remove_names[i]);
}
- out_rsbtbl:
vfree(ls->ls_rsbtbl);
out_lsfree:
if (do_unreg)
diff --git a/fs/drop_caches.c b/fs/drop_caches.c
index d72d52b..ddf319b 100644
--- a/fs/drop_caches.c
+++ b/fs/drop_caches.c
@@ -13,7 +13,7 @@
/* A global variable is a bit ugly, but it keeps the code simple */
int sysctl_drop_caches;
-static void drop_pagecache_sb(struct super_block *sb, void *unused)
+void drop_pagecache_sb(struct super_block *sb, void *unused)
{
struct inode *inode, *toput_inode = NULL;
diff --git a/fs/ecryptfs/Makefile b/fs/ecryptfs/Makefile
index 49678a6..c29cdd2 100644
--- a/fs/ecryptfs/Makefile
+++ b/fs/ecryptfs/Makefile
@@ -4,7 +4,7 @@
obj-$(CONFIG_ECRYPT_FS) += ecryptfs.o
-ecryptfs-y := dentry.o file.o inode.o main.o super.o mmap.o read_write.o \
+ecryptfs-y := dentry.o file.o inode.o main.o super.o mmap.o read_write.o events.o \
crypto.o keystore.o kthread.o debug.o
ecryptfs-$(CONFIG_ECRYPT_FS_MESSAGING) += messaging.o miscdev.o
diff --git a/fs/ecryptfs/crypto.c b/fs/ecryptfs/crypto.c
index e5e29f8..4ce6c18 100644
--- a/fs/ecryptfs/crypto.c
+++ b/fs/ecryptfs/crypto.c
@@ -36,6 +36,7 @@
#include <linux/scatterlist.h>
#include <linux/slab.h>
#include <asm/unaligned.h>
+#include <linux/ecryptfs.h>
#include "ecryptfs_kernel.h"
#define DECRYPT 0
@@ -343,9 +344,9 @@ static int crypt_scatterlist(struct ecryptfs_crypt_stat *crypt_stat,
|| !(crypt_stat->flags & ECRYPTFS_STRUCT_INITIALIZED));
if (unlikely(ecryptfs_verbosity > 0)) {
ecryptfs_printk(KERN_DEBUG, "Key size [%zd]; key:\n",
- crypt_stat->key_size);
+ ecryptfs_get_key_size_to_enc_data(crypt_stat));
ecryptfs_dump_hex(crypt_stat->key,
- crypt_stat->key_size);
+ ecryptfs_get_key_size_to_enc_data(crypt_stat));
}
init_completion(&ecr.completion);
@@ -364,7 +365,7 @@ static int crypt_scatterlist(struct ecryptfs_crypt_stat *crypt_stat,
/* Consider doing this once, when the file is opened */
if (!(crypt_stat->flags & ECRYPTFS_KEY_SET)) {
rc = crypto_skcipher_setkey(crypt_stat->tfm, crypt_stat->key,
- crypt_stat->key_size);
+ ecryptfs_get_key_size_to_enc_data(crypt_stat));
if (rc) {
ecryptfs_printk(KERN_ERR,
"Error setting key; rc = [%d]\n",
@@ -459,6 +460,30 @@ static int crypt_extent(struct ecryptfs_crypt_stat *crypt_stat,
return rc;
}
+static void init_ecryption_parameters(bool *hw_crypt, bool *cipher_supported,
+ struct ecryptfs_crypt_stat *crypt_stat)
+{
+ if (!hw_crypt || !cipher_supported)
+ return;
+
+ *cipher_supported = false;
+ *hw_crypt = false;
+
+ if (get_events() && get_events()->is_cipher_supported_cb) {
+ *cipher_supported =
+ get_events()->is_cipher_supported_cb(crypt_stat);
+ if (*cipher_supported) {
+
+ /**
+ * we should apply external algorythm
+ * assume that is_hw_crypt() cbck is supplied
+ */
+ if (get_events()->is_hw_crypt_cb)
+ *hw_crypt = get_events()->is_hw_crypt_cb();
+ }
+ }
+}
+
/**
* ecryptfs_encrypt_page
* @page: Page mapped from the eCryptfs inode for the file; contains
@@ -484,11 +509,18 @@ int ecryptfs_encrypt_page(struct page *page)
loff_t extent_offset;
loff_t lower_offset;
int rc = 0;
+ bool is_hw_crypt;
+ bool is_cipher_supported;
+
ecryptfs_inode = page->mapping->host;
crypt_stat =
&(ecryptfs_inode_to_private(ecryptfs_inode)->crypt_stat);
BUG_ON(!(crypt_stat->flags & ECRYPTFS_ENCRYPTED));
+
+ init_ecryption_parameters(&is_hw_crypt,
+ &is_cipher_supported, crypt_stat);
+
enc_extent_page = alloc_page(GFP_USER);
if (!enc_extent_page) {
rc = -ENOMEM;
@@ -496,24 +528,51 @@ int ecryptfs_encrypt_page(struct page *page)
"encrypted extent\n");
goto out;
}
+ if (is_hw_crypt) {
+ /* no need for encryption */
+ } else {
+ for (extent_offset = 0;
+ extent_offset <
+ (PAGE_SIZE / crypt_stat->extent_size);
+ extent_offset++) {
- for (extent_offset = 0;
- extent_offset < (PAGE_SIZE / crypt_stat->extent_size);
- extent_offset++) {
- rc = crypt_extent(crypt_stat, enc_extent_page, page,
- extent_offset, ENCRYPT);
- if (rc) {
- printk(KERN_ERR "%s: Error encrypting extent; "
- "rc = [%d]\n", __func__, rc);
- goto out;
+ if (is_cipher_supported) {
+ if (!get_events()->encrypt_cb) {
+ rc = -EPERM;
+ goto out;
+ }
+ rc = get_events()->encrypt_cb(page,
+ enc_extent_page,
+ ecryptfs_inode_to_lower(
+ ecryptfs_inode),
+ extent_offset);
+ } else {
+ rc = crypt_extent(crypt_stat,
+ enc_extent_page, page,
+ extent_offset, ENCRYPT);
+ }
+ if (rc) {
+ ecryptfs_printk(KERN_ERR,
+ "%s: Error encrypting; rc = [%d]\n",
+ __func__, rc);
+ goto out;
+ }
}
}
lower_offset = lower_offset_for_page(crypt_stat, page);
- enc_extent_virt = kmap(enc_extent_page);
+ if (is_hw_crypt)
+ enc_extent_virt = kmap(page);
+ else
+ enc_extent_virt = kmap(enc_extent_page);
+
rc = ecryptfs_write_lower(ecryptfs_inode, enc_extent_virt, lower_offset,
PAGE_SIZE);
- kunmap(enc_extent_page);
+ if (!is_hw_crypt)
+ kunmap(enc_extent_page);
+ else
+ kunmap(page);
+
if (rc < 0) {
ecryptfs_printk(KERN_ERR,
"Error attempting to write lower page; rc = [%d]\n",
@@ -552,6 +611,8 @@ int ecryptfs_decrypt_page(struct page *page)
unsigned long extent_offset;
loff_t lower_offset;
int rc = 0;
+ bool is_cipher_supported;
+ bool is_hw_crypt;
ecryptfs_inode = page->mapping->host;
crypt_stat =
@@ -570,13 +631,33 @@ int ecryptfs_decrypt_page(struct page *page)
goto out;
}
+ init_ecryption_parameters(&is_hw_crypt,
+ &is_cipher_supported, crypt_stat);
+
+ if (is_hw_crypt) {
+ rc = 0;
+ return rc;
+ }
+
for (extent_offset = 0;
extent_offset < (PAGE_SIZE / crypt_stat->extent_size);
extent_offset++) {
- rc = crypt_extent(crypt_stat, page, page,
+ if (is_cipher_supported) {
+ if (!get_events()->decrypt_cb) {
+ rc = -EPERM;
+ goto out;
+ }
+
+ rc = get_events()->decrypt_cb(page, page,
+ ecryptfs_inode_to_lower(ecryptfs_inode),
+ extent_offset);
+
+ } else
+ rc = crypt_extent(crypt_stat, page, page,
extent_offset, DECRYPT);
+
if (rc) {
- printk(KERN_ERR "%s: Error encrypting extent; "
+ ecryptfs_printk(KERN_ERR, "%s: Error decrypting extent;"
"rc = [%d]\n", __func__, rc);
goto out;
}
@@ -605,7 +686,7 @@ int ecryptfs_init_crypt_ctx(struct ecryptfs_crypt_stat *crypt_stat)
"Initializing cipher [%s]; strlen = [%d]; "
"key_size_bits = [%zd]\n",
crypt_stat->cipher, (int)strlen(crypt_stat->cipher),
- crypt_stat->key_size << 3);
+ ecryptfs_get_key_size_to_enc_data(crypt_stat) << 3);
mutex_lock(&crypt_stat->cs_tfm_mutex);
if (crypt_stat->tfm) {
rc = 0;
@@ -687,7 +768,7 @@ int ecryptfs_compute_root_iv(struct ecryptfs_crypt_stat *crypt_stat)
goto out;
}
rc = ecryptfs_calculate_md5(dst, crypt_stat, crypt_stat->key,
- crypt_stat->key_size);
+ ecryptfs_get_key_size_to_enc_data(crypt_stat));
if (rc) {
ecryptfs_printk(KERN_WARNING, "Error attempting to compute "
"MD5 while generating root IV\n");
@@ -714,6 +795,31 @@ static void ecryptfs_generate_new_key(struct ecryptfs_crypt_stat *crypt_stat)
}
}
+static int ecryptfs_generate_new_salt(struct ecryptfs_crypt_stat *crypt_stat)
+{
+ size_t salt_size = 0;
+
+ salt_size = ecryptfs_get_salt_size_for_cipher(crypt_stat);
+
+ if (0 == salt_size)
+ return 0;
+
+ if (!ecryptfs_check_space_for_salt(crypt_stat->key_size, salt_size)) {
+ ecryptfs_printk(KERN_WARNING, "not enough space for salt\n");
+ crypt_stat->flags |= ECRYPTFS_SECURITY_WARNING;
+ return -EINVAL;
+ }
+
+ get_random_bytes(crypt_stat->key + crypt_stat->key_size, salt_size);
+ if (unlikely(ecryptfs_verbosity > 0)) {
+ ecryptfs_printk(KERN_DEBUG, "Generated new session salt:\n");
+ ecryptfs_dump_hex(crypt_stat->key + crypt_stat->key_size,
+ salt_size);
+ }
+
+ return 0;
+}
+
/**
* ecryptfs_copy_mount_wide_flags_to_inode_flags
* @crypt_stat: The inode's cryptographic context
@@ -816,7 +922,6 @@ int ecryptfs_new_file_context(struct inode *ecryptfs_inode)
struct ecryptfs_mount_crypt_stat *mount_crypt_stat =
&ecryptfs_superblock_to_private(
ecryptfs_inode->i_sb)->mount_crypt_stat;
- int cipher_name_len;
int rc = 0;
ecryptfs_set_default_crypt_stat_vals(crypt_stat, mount_crypt_stat);
@@ -830,15 +935,19 @@ int ecryptfs_new_file_context(struct inode *ecryptfs_inode)
"to the inode key sigs; rc = [%d]\n", rc);
goto out;
}
- cipher_name_len =
- strlen(mount_crypt_stat->global_default_cipher_name);
- memcpy(crypt_stat->cipher,
+ strlcpy(crypt_stat->cipher,
mount_crypt_stat->global_default_cipher_name,
- cipher_name_len);
- crypt_stat->cipher[cipher_name_len] = '\0';
+ sizeof(crypt_stat->cipher));
+
+ strlcpy(crypt_stat->cipher_mode,
+ mount_crypt_stat->global_default_cipher_mode,
+ sizeof(crypt_stat->cipher_mode));
+
crypt_stat->key_size =
mount_crypt_stat->global_default_cipher_key_size;
ecryptfs_generate_new_key(crypt_stat);
+ ecryptfs_generate_new_salt(crypt_stat);
+
rc = ecryptfs_init_crypt_ctx(crypt_stat);
if (rc)
ecryptfs_printk(KERN_ERR, "Error initializing cryptographic "
@@ -964,7 +1073,8 @@ ecryptfs_cipher_code_str_map[] = {
{"twofish", RFC2440_CIPHER_TWOFISH},
{"cast6", RFC2440_CIPHER_CAST_6},
{"aes", RFC2440_CIPHER_AES_192},
- {"aes", RFC2440_CIPHER_AES_256}
+ {"aes", RFC2440_CIPHER_AES_256},
+ {"aes_xts", RFC2440_CIPHER_AES_XTS_256}
};
/**
@@ -992,6 +1102,11 @@ u8 ecryptfs_code_for_cipher_string(char *cipher_name, size_t key_bytes)
case 32:
code = RFC2440_CIPHER_AES_256;
}
+ } else if (strcmp(cipher_name, "aes_xts") == 0) {
+ switch (key_bytes) {
+ case 32:
+ code = RFC2440_CIPHER_AES_XTS_256;
+ }
} else {
for (i = 0; i < ARRAY_SIZE(ecryptfs_cipher_code_str_map); i++)
if (strcmp(cipher_name, map[i].cipher_str) == 0) {
@@ -1031,9 +1146,24 @@ int ecryptfs_read_and_validate_header_region(struct inode *inode)
u8 file_size[ECRYPTFS_SIZE_AND_MARKER_BYTES];
u8 *marker = file_size + ECRYPTFS_FILE_SIZE_BYTES;
int rc;
+ unsigned int ra_pages_org;
+ struct file *lower_file = NULL;
+
+ if (!inode)
+ return -EIO;
+ lower_file = ecryptfs_inode_to_private(inode)->lower_file;
+ if (!lower_file)
+ return -EIO;
+
+ /*disable read a head mechanism for a while */
+ ra_pages_org = lower_file->f_ra.ra_pages;
+ lower_file->f_ra.ra_pages = 0;
rc = ecryptfs_read_lower(file_size, 0, ECRYPTFS_SIZE_AND_MARKER_BYTES,
inode);
+ lower_file->f_ra.ra_pages = ra_pages_org;
+ /* restore read a head mechanism */
+
if (rc < ECRYPTFS_SIZE_AND_MARKER_BYTES)
return rc >= 0 ? -EINVAL : rc;
rc = ecryptfs_validate_marker(marker);
@@ -1427,6 +1557,11 @@ int ecryptfs_read_metadata(struct dentry *ecryptfs_dentry)
struct ecryptfs_mount_crypt_stat *mount_crypt_stat =
&ecryptfs_superblock_to_private(
ecryptfs_dentry->d_sb)->mount_crypt_stat;
+ unsigned int ra_pages_org;
+ struct file *lower_file =
+ ecryptfs_inode_to_private(ecryptfs_inode)->lower_file;
+ if (!lower_file)
+ return -EIO;
ecryptfs_copy_mount_wide_flags_to_inode_flags(crypt_stat,
mount_crypt_stat);
@@ -1438,8 +1573,14 @@ int ecryptfs_read_metadata(struct dentry *ecryptfs_dentry)
__func__);
goto out;
}
+ /*disable read a head mechanism */
+ ra_pages_org = lower_file->f_ra.ra_pages;
+ lower_file->f_ra.ra_pages = 0;
+
rc = ecryptfs_read_lower(page_virt, 0, crypt_stat->extent_size,
ecryptfs_inode);
+ lower_file->f_ra.ra_pages = ra_pages_org; /* restore it back */
+
if (rc >= 0)
rc = ecryptfs_read_headers_virt(page_virt, crypt_stat,
ecryptfs_dentry,
diff --git a/fs/ecryptfs/debug.c b/fs/ecryptfs/debug.c
index 3d2bdf5..7886cd5 100644
--- a/fs/ecryptfs/debug.c
+++ b/fs/ecryptfs/debug.c
@@ -119,3 +119,28 @@ void ecryptfs_dump_hex(char *data, int bytes)
printk("\n");
}
+void ecryptfs_dump_salt_hex(char *data, int key_size,
+ const struct ecryptfs_crypt_stat *crypt_stat)
+{
+ size_t salt_size = ecryptfs_get_salt_size_for_cipher(crypt_stat);
+
+ if (0 == salt_size)
+ return;
+
+ if (!ecryptfs_check_space_for_salt(key_size, salt_size))
+ return;
+
+ ecryptfs_printk(KERN_DEBUG, "Decrypted session salt key:\n");
+ ecryptfs_dump_hex(data + key_size, salt_size);
+}
+
+void ecryptfs_dump_cipher(struct ecryptfs_crypt_stat *stat)
+{
+ if (!stat)
+ return;
+
+ ecryptfs_printk(KERN_DEBUG,
+ "ecryptfs cipher is %s\n", stat->cipher);
+ ecryptfs_printk(KERN_DEBUG, "ecryptfs cipher mode is %s\n",
+ stat->cipher_mode);
+}
diff --git a/fs/ecryptfs/ecryptfs_kernel.h b/fs/ecryptfs/ecryptfs_kernel.h
index d4d8ad1..872e081 100644
--- a/fs/ecryptfs/ecryptfs_kernel.h
+++ b/fs/ecryptfs/ecryptfs_kernel.h
@@ -253,6 +253,7 @@ struct ecryptfs_crypt_stat {
struct mutex keysig_list_mutex;
struct mutex cs_tfm_mutex;
struct mutex cs_mutex;
+ unsigned char cipher_mode[ECRYPTFS_MAX_CIPHER_NAME_SIZE + 1];
};
/* inode private data. */
@@ -353,6 +354,8 @@ struct ecryptfs_mount_crypt_stat {
unsigned char global_default_fn_cipher_name[
ECRYPTFS_MAX_CIPHER_NAME_SIZE + 1];
char global_default_fnek_sig[ECRYPTFS_SIG_SIZE_HEX + 1];
+ unsigned char global_default_cipher_mode[ECRYPTFS_MAX_CIPHER_NAME_SIZE
+ + 1];
};
/* superblock private data. */
@@ -535,6 +538,53 @@ ecryptfs_dentry_to_lower_path(struct dentry *dentry)
return &((struct ecryptfs_dentry_info *)dentry->d_fsdata)->lower_path;
}
+/**
+ * Given a cipher and mode strings, the function
+ * concatenates them to create a new string of
+ * <cipher>_<mode> format.
+ */
+static inline unsigned char *ecryptfs_get_full_cipher(
+ unsigned char *cipher, unsigned char *mode,
+ unsigned char *final, size_t final_size)
+{
+ memset(final, 0, final_size);
+
+ if (strlen(mode) > 0) {
+ snprintf(final, final_size, "%s_%s", cipher, mode);
+ return final;
+ }
+
+ return cipher;
+}
+
+/**
+ * Given a <cipher>[_<mode>] formatted string, the function
+ * extracts cipher string and/or mode string.
+ * Note: the passed cipher and/or mode strings will be null-terminated.
+ */
+static inline void ecryptfs_parse_full_cipher(
+ char *s, char *cipher, char *mode)
+{
+ char input[2*ECRYPTFS_MAX_CIPHER_NAME_SIZE+1+1];
+ /* +1 for '_'; +1 for '\0' */
+ char *p;
+ char *input_p = input;
+
+ if (s == NULL || cipher == NULL)
+ return;
+
+ memset(input, 0, sizeof(input));
+ strlcpy(input, s, sizeof(input));
+
+ p = strsep(&input_p, "_");
+ strlcpy(cipher, p, ECRYPTFS_MAX_CIPHER_NAME_SIZE + 1);
+
+
+ /* check if mode is specified */
+ if (input_p != NULL && mode != NULL)
+ strlcpy(mode, input_p, ECRYPTFS_MAX_CIPHER_NAME_SIZE + 1);
+}
+
#define ecryptfs_printk(type, fmt, arg...) \
__ecryptfs_printk(type "%s: " fmt, __func__, ## arg);
__printf(1, 2)
@@ -582,6 +632,10 @@ int ecryptfs_encrypt_and_encode_filename(
const char *name, size_t name_size);
struct dentry *ecryptfs_lower_dentry(struct dentry *this_dentry);
void ecryptfs_dump_hex(char *data, int bytes);
+void ecryptfs_dump_salt_hex(char *data, int key_size,
+ const struct ecryptfs_crypt_stat *crypt_stat);
+extern void ecryptfs_dump_cipher(struct ecryptfs_crypt_stat *stat);
+
int virt_to_scatterlist(const void *addr, int size, struct scatterlist *sg,
int sg_size);
int ecryptfs_compute_root_iv(struct ecryptfs_crypt_stat *crypt_stat);
@@ -727,4 +781,33 @@ int ecryptfs_derive_iv(char *iv, struct ecryptfs_crypt_stat *crypt_stat,
extern const struct xattr_handler *ecryptfs_xattr_handlers[];
+void clean_inode_pages(struct address_space *mapping,
+ pgoff_t start, pgoff_t end);
+
+void ecryptfs_drop_pagecache_sb(struct super_block *sb, void *unused);
+
+void ecryptfs_free_events(void);
+
+void ecryptfs_freepage(struct page *page);
+
+struct ecryptfs_events *get_events(void);
+
+size_t ecryptfs_get_salt_size_for_cipher(
+ const struct ecryptfs_crypt_stat *crypt_stat);
+
+size_t ecryptfs_get_salt_size_for_cipher_mount(
+ const struct ecryptfs_mount_crypt_stat *mount_crypt_stat);
+
+size_t ecryptfs_get_key_size_to_enc_data(
+ const struct ecryptfs_crypt_stat *crypt_stat);
+
+size_t ecryptfs_get_key_size_to_store_key(
+ const struct ecryptfs_crypt_stat *crypt_stat);
+
+size_t ecryptfs_get_key_size_to_restore_key(size_t stored_key_size,
+ const struct ecryptfs_crypt_stat *crypt_stat);
+
+bool ecryptfs_check_space_for_salt(const size_t key_size,
+ const size_t salt_size);
+
#endif /* #ifndef ECRYPTFS_KERNEL_H */
diff --git a/fs/ecryptfs/events.c b/fs/ecryptfs/events.c
new file mode 100644
index 0000000..8531198
--- /dev/null
+++ b/fs/ecryptfs/events.c
@@ -0,0 +1,393 @@
+/**
+ * eCryptfs: Linux filesystem encryption layer
+ * Copyright (c) 2015-2016, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/string.h>
+#include <linux/ecryptfs.h>
+#include <linux/mutex.h>
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/pagemap.h>
+#include <linux/random.h>
+#include "ecryptfs_kernel.h"
+
+static DEFINE_MUTEX(events_mutex);
+struct ecryptfs_events *events_ptr;
+static int handle;
+
+void ecryptfs_free_events(void)
+{
+ mutex_lock(&events_mutex);
+ if (events_ptr != NULL) {
+ kfree(events_ptr);
+ events_ptr = NULL;
+ }
+
+ mutex_unlock(&events_mutex);
+}
+
+/**
+ * Register to ecryptfs events, by passing callback
+ * functions to be called upon events occurrence.
+ * The function returns a handle to be passed
+ * to unregister function.
+ */
+int ecryptfs_register_to_events(const struct ecryptfs_events *ops)
+{
+ int ret_value = 0;
+
+ if (!ops)
+ return -EINVAL;
+
+ mutex_lock(&events_mutex);
+
+ if (events_ptr != NULL) {
+ ecryptfs_printk(KERN_ERR,
+ "already registered!\n");
+ ret_value = -EPERM;
+ goto out;
+ }
+ events_ptr =
+ kzalloc(sizeof(struct ecryptfs_events), GFP_KERNEL);
+
+ if (!events_ptr) {
+ ecryptfs_printk(KERN_ERR, "malloc failure\n");
+ ret_value = -ENOMEM;
+ goto out;
+ }
+ /* copy the callbacks */
+ events_ptr->open_cb = ops->open_cb;
+ events_ptr->release_cb = ops->release_cb;
+ events_ptr->encrypt_cb = ops->encrypt_cb;
+ events_ptr->decrypt_cb = ops->decrypt_cb;
+ events_ptr->is_cipher_supported_cb =
+ ops->is_cipher_supported_cb;
+ events_ptr->is_hw_crypt_cb = ops->is_hw_crypt_cb;
+ events_ptr->get_salt_key_size_cb = ops->get_salt_key_size_cb;
+
+ get_random_bytes(&handle, sizeof(handle));
+ ret_value = handle;
+
+out:
+ mutex_unlock(&events_mutex);
+ return ret_value;
+}
+
+/**
+ * Unregister from ecryptfs events.
+ */
+int ecryptfs_unregister_from_events(int user_handle)
+{
+ int ret_value = 0;
+
+ mutex_lock(&events_mutex);
+
+ if (!events_ptr) {
+ ret_value = -EINVAL;
+ goto out;
+ }
+ if (user_handle != handle) {
+ ret_value = ECRYPTFS_INVALID_EVENTS_HANDLE;
+ goto out;
+ }
+
+ kfree(events_ptr);
+ events_ptr = NULL;
+
+out:
+ mutex_unlock(&events_mutex);
+ return ret_value;
+}
+
+/**
+ * This function decides whether the passed file offset
+ * belongs to ecryptfs metadata or not.
+ * The caller must pass ecryptfs data, which was received in one
+ * of the callback invocations.
+ */
+bool ecryptfs_is_page_in_metadata(const void *data, pgoff_t offset)
+{
+
+ struct ecryptfs_crypt_stat *stat = NULL;
+ bool ret = true;
+
+ if (!data) {
+ ecryptfs_printk(KERN_ERR, "ecryptfs_is_page_in_metadata: invalid data parameter\n");
+ ret = false;
+ goto end;
+ }
+ stat = (struct ecryptfs_crypt_stat *)data;
+
+ if (stat->flags & ECRYPTFS_METADATA_IN_XATTR) {
+ ret = false;
+ goto end;
+ }
+
+ if (offset >= (stat->metadata_size/PAGE_SIZE)) {
+ ret = false;
+ goto end;
+ }
+end:
+ return ret;
+}
+
+/**
+ * Given two ecryptfs data, the function
+ * decides whether they are equal.
+ */
+inline bool ecryptfs_is_data_equal(const void *data1, const void *data2)
+{
+ /* pointer comparison*/
+ return data1 == data2;
+}
+
+/**
+ * Given ecryptfs data, the function
+ * returns appropriate key size.
+ */
+size_t ecryptfs_get_key_size(const void *data)
+{
+
+ struct ecryptfs_crypt_stat *stat = NULL;
+
+ if (!data)
+ return 0;
+
+ stat = (struct ecryptfs_crypt_stat *)data;
+ return stat->key_size;
+}
+
+/**
+ * Given ecryptfs data, the function
+ * returns appropriate salt size.
+ *
+ * !!! crypt_stat cipher name and mode must be initialized
+ */
+size_t ecryptfs_get_salt_size(const void *data)
+{
+ if (!data) {
+ ecryptfs_printk(KERN_ERR,
+ "ecryptfs_get_salt_size: invalid data parameter\n");
+ return 0;
+ }
+
+ return ecryptfs_get_salt_size_for_cipher(data);
+
+}
+
+/**
+ * Given ecryptfs data and cipher string, the function
+ * returns true if provided cipher and the one in ecryptfs match.
+ */
+bool ecryptfs_cipher_match(const void *data,
+ const unsigned char *cipher, size_t cipher_size)
+{
+ unsigned char final[2*ECRYPTFS_MAX_CIPHER_NAME_SIZE+1];
+ const unsigned char *ecryptfs_cipher = NULL;
+ struct ecryptfs_crypt_stat *stat = NULL;
+
+ if (!data || !cipher) {
+ ecryptfs_printk(KERN_ERR,
+ "ecryptfs_get_cipher: invalid data parameter\n");
+ return false;
+ }
+
+ if (!cipher_size || cipher_size > sizeof(final)) {
+ ecryptfs_printk(KERN_ERR,
+ "ecryptfs_get_cipher: cipher_size\n");
+ return false;
+ }
+
+ stat = (struct ecryptfs_crypt_stat *)data;
+ ecryptfs_cipher = ecryptfs_get_full_cipher(stat->cipher,
+ stat->cipher_mode,
+ final, sizeof(final));
+
+ if (!ecryptfs_cipher) {
+ ecryptfs_printk(KERN_ERR,
+ "ecryptfs_get_cipher: internal error while parsing cipher\n");
+ return false;
+ }
+
+ if (strcmp(ecryptfs_cipher, cipher)) {
+ if (ecryptfs_verbosity > 0)
+ ecryptfs_dump_cipher(stat);
+
+ return false;
+ }
+
+ return true;
+}
+
+/**
+ * Given ecryptfs data, the function
+ * returns file encryption key.
+ */
+const unsigned char *ecryptfs_get_key(const void *data)
+{
+
+ struct ecryptfs_crypt_stat *stat = NULL;
+
+ if (!data) {
+ ecryptfs_printk(KERN_ERR,
+ "ecryptfs_get_key: invalid data parameter\n");
+ return NULL;
+ }
+ stat = (struct ecryptfs_crypt_stat *)data;
+ return stat->key;
+}
+
+/**
+ * Given ecryptfs data, the function
+ * returns file encryption salt.
+ */
+const unsigned char *ecryptfs_get_salt(const void *data)
+{
+ struct ecryptfs_crypt_stat *stat = NULL;
+
+ if (!data) {
+ ecryptfs_printk(KERN_ERR,
+ "ecryptfs_get_salt: invalid data parameter\n");
+ return NULL;
+ }
+ stat = (struct ecryptfs_crypt_stat *)data;
+ return stat->key + ecryptfs_get_salt_size(data);
+}
+
+/**
+ * Returns ecryptfs events pointer
+ */
+inline struct ecryptfs_events *get_events(void)
+{
+ return events_ptr;
+}
+
+/**
+ * If external crypto module requires salt in addition to key,
+ * we store it as part of key array (if there is enough space)
+ * Checks whether a salt key can fit into array allocated for
+ * regular key
+ */
+bool ecryptfs_check_space_for_salt(const size_t key_size,
+ const size_t salt_size)
+{
+ if ((salt_size + key_size) > ECRYPTFS_MAX_KEY_BYTES)
+ return false;
+
+ return true;
+}
+
+/*
+ * If there is salt that is used by external crypto module, it is stored
+ * in the same array where regular key is. Salt is going to be used by
+ * external crypto module only, so for all internal crypto operations salt
+ * should be ignored.
+ *
+ * Get key size in cases where it is going to be used for data encryption
+ * or for all other general purposes
+ */
+size_t ecryptfs_get_key_size_to_enc_data(
+ const struct ecryptfs_crypt_stat *crypt_stat)
+{
+ if (!crypt_stat)
+ return 0;
+
+ return crypt_stat->key_size;
+}
+
+/*
+ * If there is salt that is used by external crypto module, it is stored
+ * in the same array where regular key is. Salt is going to be used by
+ * external crypto module only, but we still need to save and restore it
+ * (in encrypted form) as part of ecryptfs header along with the regular
+ * key.
+ *
+ * Get key size in cases where it is going to be stored persistently
+ *
+ * !!! crypt_stat cipher name and mode must be initialized
+ */
+size_t ecryptfs_get_key_size_to_store_key(
+ const struct ecryptfs_crypt_stat *crypt_stat)
+{
+ size_t salt_size = 0;
+
+ if (!crypt_stat)
+ return 0;
+
+ salt_size = ecryptfs_get_salt_size(crypt_stat);
+
+ if (!ecryptfs_check_space_for_salt(crypt_stat->key_size, salt_size)) {
+ ecryptfs_printk(KERN_WARNING,
+ "ecryptfs_get_key_size_to_store_key: not enough space for salt\n");
+ return crypt_stat->key_size;
+ }
+
+ return crypt_stat->key_size + salt_size;
+}
+
+/*
+ * If there is salt that is used by external crypto module, it is stored
+ * in the same array where regular key is. Salt is going to be used by
+ * external crypto module only, but we still need to save and restore it
+ * (in encrypted form) as part of ecryptfs header along with the regular
+ * key.
+ *
+ * Get key size in cases where it is going to be restored from storage
+ *
+ * !!! crypt_stat cipher name and mode must be initialized
+ */
+size_t ecryptfs_get_key_size_to_restore_key(size_t stored_key_size,
+ const struct ecryptfs_crypt_stat *crypt_stat)
+{
+ size_t salt_size = 0;
+
+ if (!crypt_stat)
+ return 0;
+
+ salt_size = ecryptfs_get_salt_size_for_cipher(crypt_stat);
+
+ if (salt_size >= stored_key_size) {
+ ecryptfs_printk(KERN_WARNING,
+ "ecryptfs_get_key_size_to_restore_key: salt %zu >= stred size %zu\n",
+ salt_size, stored_key_size);
+
+ return stored_key_size;
+ }
+
+ return stored_key_size - salt_size;
+}
+
+/**
+ * Given crypt_stat, the function returns appropriate salt size.
+ */
+size_t ecryptfs_get_salt_size_for_cipher(
+ const struct ecryptfs_crypt_stat *crypt_stat)
+{
+ if (!get_events() || !(get_events()->get_salt_key_size_cb))
+ return 0;
+
+ return get_events()->get_salt_key_size_cb(crypt_stat);
+}
+
+/**
+ * Given mount_crypt_stat, the function returns appropriate salt size.
+ */
+size_t ecryptfs_get_salt_size_for_cipher_mount(
+ const struct ecryptfs_mount_crypt_stat *crypt_stat)
+{
+ if (!get_events() || !(get_events()->get_salt_key_size_cb))
+ return 0;
+
+ return get_events()->get_salt_key_size_cb(crypt_stat);
+}
+
diff --git a/fs/ecryptfs/file.c b/fs/ecryptfs/file.c
index ca4e837..12ce795 100644
--- a/fs/ecryptfs/file.c
+++ b/fs/ecryptfs/file.c
@@ -31,6 +31,7 @@
#include <linux/security.h>
#include <linux/compat.h>
#include <linux/fs_stack.h>
+#include <linux/ecryptfs.h>
#include "ecryptfs_kernel.h"
/**
@@ -196,6 +197,9 @@ static int ecryptfs_open(struct inode *inode, struct file *file)
int rc = 0;
struct ecryptfs_crypt_stat *crypt_stat = NULL;
struct dentry *ecryptfs_dentry = file->f_path.dentry;
+ int ret;
+
+
/* Private value of ecryptfs_dentry allocated in
* ecryptfs_lookup() */
struct ecryptfs_file_info *file_info;
@@ -235,12 +239,31 @@ static int ecryptfs_open(struct inode *inode, struct file *file)
}
ecryptfs_set_file_lower(
file, ecryptfs_inode_to_private(inode)->lower_file);
+
rc = read_or_initialize_metadata(ecryptfs_dentry);
if (rc)
goto out_put;
ecryptfs_printk(KERN_DEBUG, "inode w/ addr = [0x%p], i_ino = "
"[0x%.16lx] size: [0x%.16llx]\n", inode, inode->i_ino,
(unsigned long long)i_size_read(inode));
+
+ if (get_events() && get_events()->open_cb) {
+
+ ret = vfs_fsync(file, false);
+
+ if (ret)
+ ecryptfs_printk(KERN_ERR,
+ "failed to sync file ret = %d.\n", ret);
+
+ get_events()->open_cb(ecryptfs_inode_to_lower(inode),
+ crypt_stat);
+
+ if (crypt_stat->flags & ECRYPTFS_METADATA_IN_XATTR) {
+ truncate_inode_pages(inode->i_mapping, 0);
+ truncate_inode_pages(
+ ecryptfs_inode_to_lower(inode)->i_mapping, 0);
+ }
+ }
goto out;
out_put:
ecryptfs_put_lower_file(inode);
@@ -253,7 +276,7 @@ static int ecryptfs_open(struct inode *inode, struct file *file)
/**
* ecryptfs_dir_open
- * @inode: inode specifying file to open
+ * @inode: inode speciying file to open
* @file: Structure to return filled in
*
* Opens the file specified by inode.
@@ -307,6 +330,7 @@ static int ecryptfs_release(struct inode *inode, struct file *file)
ecryptfs_put_lower_file(inode);
kmem_cache_free(ecryptfs_file_info_cache,
ecryptfs_file_to_private(file));
+
return 0;
}
diff --git a/fs/ecryptfs/inode.c b/fs/ecryptfs/inode.c
index cf390dc..060b417 100644
--- a/fs/ecryptfs/inode.c
+++ b/fs/ecryptfs/inode.c
@@ -260,12 +260,15 @@ int ecryptfs_initialize_file(struct dentry *ecryptfs_dentry,
*
* Returns zero on success; non-zero on error condition
*/
+
+
static int
ecryptfs_create(struct inode *directory_inode, struct dentry *ecryptfs_dentry,
umode_t mode, bool excl)
{
struct inode *ecryptfs_inode;
int rc;
+ struct ecryptfs_crypt_stat *crypt_stat;
ecryptfs_inode = ecryptfs_do_create(directory_inode, ecryptfs_dentry,
mode);
@@ -275,6 +278,7 @@ ecryptfs_create(struct inode *directory_inode, struct dentry *ecryptfs_dentry,
rc = PTR_ERR(ecryptfs_inode);
goto out;
}
+
/* At this point, a file exists on "disk"; we need to make sure
* that this on disk file is prepared to be an ecryptfs file */
rc = ecryptfs_initialize_file(ecryptfs_dentry, ecryptfs_inode);
@@ -284,8 +288,14 @@ ecryptfs_create(struct inode *directory_inode, struct dentry *ecryptfs_dentry,
iget_failed(ecryptfs_inode);
goto out;
}
- unlock_new_inode(ecryptfs_inode);
- d_instantiate(ecryptfs_dentry, ecryptfs_inode);
+
+ crypt_stat = &ecryptfs_inode_to_private(ecryptfs_inode)->crypt_stat;
+ if (get_events() && get_events()->open_cb)
+ get_events()->open_cb(
+ ecryptfs_inode_to_lower(ecryptfs_inode),
+ crypt_stat);
+
+ d_instantiate_new(ecryptfs_dentry, ecryptfs_inode);
out:
return rc;
}
diff --git a/fs/ecryptfs/keystore.c b/fs/ecryptfs/keystore.c
index fa218cd..1f6fffa 100644
--- a/fs/ecryptfs/keystore.c
+++ b/fs/ecryptfs/keystore.c
@@ -316,7 +316,8 @@ write_tag_66_packet(char *signature, u8 cipher_code,
* | File Encryption Key Size | 1 or 2 bytes |
* | File Encryption Key | arbitrary |
*/
- data_len = (5 + ECRYPTFS_SIG_SIZE_HEX + crypt_stat->key_size);
+ data_len = (5 + ECRYPTFS_SIG_SIZE_HEX +
+ ecryptfs_get_key_size_to_store_key(crypt_stat));
*packet = kmalloc(data_len, GFP_KERNEL);
message = *packet;
if (!message) {
@@ -336,8 +337,9 @@ write_tag_66_packet(char *signature, u8 cipher_code,
memcpy(&message[i], signature, ECRYPTFS_SIG_SIZE_HEX);
i += ECRYPTFS_SIG_SIZE_HEX;
/* The encrypted key includes 1 byte cipher code and 2 byte checksum */
- rc = ecryptfs_write_packet_length(&message[i], crypt_stat->key_size + 3,
- &packet_size_len);
+ rc = ecryptfs_write_packet_length(&message[i],
+ ecryptfs_get_key_size_to_store_key(crypt_stat) + 3,
+ &packet_size_len);
if (rc) {
ecryptfs_printk(KERN_ERR, "Error generating tag 66 packet "
"header; cannot generate packet length\n");
@@ -345,9 +347,10 @@ write_tag_66_packet(char *signature, u8 cipher_code,
}
i += packet_size_len;
message[i++] = cipher_code;
- memcpy(&message[i], crypt_stat->key, crypt_stat->key_size);
- i += crypt_stat->key_size;
- for (j = 0; j < crypt_stat->key_size; j++)
+ memcpy(&message[i], crypt_stat->key,
+ ecryptfs_get_key_size_to_store_key(crypt_stat));
+ i += ecryptfs_get_key_size_to_store_key(crypt_stat);
+ for (j = 0; j < ecryptfs_get_key_size_to_store_key(crypt_stat); j++)
checksum += crypt_stat->key[j];
message[i++] = (checksum / 256) % 256;
message[i++] = (checksum % 256);
@@ -920,6 +923,7 @@ ecryptfs_parse_tag_70_packet(char **filename, size_t *filename_size,
struct ecryptfs_parse_tag_70_packet_silly_stack *s;
struct key *auth_tok_key = NULL;
int rc = 0;
+ char full_cipher[ECRYPTFS_MAX_CIPHER_NAME_SIZE];
(*packet_size) = 0;
(*filename_size) = 0;
@@ -977,12 +981,13 @@ ecryptfs_parse_tag_70_packet(char **filename, size_t *filename_size,
s->fnek_sig_hex[ECRYPTFS_SIG_SIZE_HEX] = '\0';
(*packet_size) += ECRYPTFS_SIG_SIZE;
s->cipher_code = data[(*packet_size)++];
- rc = ecryptfs_cipher_code_to_string(s->cipher_string, s->cipher_code);
+ rc = ecryptfs_cipher_code_to_string(full_cipher, s->cipher_code);
if (rc) {
printk(KERN_WARNING "%s: Cipher code [%d] is invalid\n",
__func__, s->cipher_code);
goto out;
}
+ ecryptfs_parse_full_cipher(full_cipher, s->cipher_string, 0);
rc = ecryptfs_find_auth_tok_for_sig(&auth_tok_key,
&s->auth_tok, mount_crypt_stat,
s->fnek_sig_hex);
@@ -1163,6 +1168,7 @@ decrypt_pki_encrypted_session_key(struct ecryptfs_auth_tok *auth_tok,
char *payload = NULL;
size_t payload_len = 0;
int rc;
+ char full_cipher[ECRYPTFS_MAX_CIPHER_NAME_SIZE];
rc = ecryptfs_get_auth_tok_sig(&auth_tok_sig, auth_tok);
if (rc) {
@@ -1196,21 +1202,31 @@ decrypt_pki_encrypted_session_key(struct ecryptfs_auth_tok *auth_tok,
rc);
goto out;
}
- auth_tok->session_key.flags |= ECRYPTFS_CONTAINS_DECRYPTED_KEY;
- memcpy(crypt_stat->key, auth_tok->session_key.decrypted_key,
- auth_tok->session_key.decrypted_key_size);
- crypt_stat->key_size = auth_tok->session_key.decrypted_key_size;
- rc = ecryptfs_cipher_code_to_string(crypt_stat->cipher, cipher_code);
+
+ rc = ecryptfs_cipher_code_to_string(full_cipher, cipher_code);
if (rc) {
ecryptfs_printk(KERN_ERR, "Cipher code [%d] is invalid\n",
cipher_code)
- goto out;
+ goto out;
}
+
+ auth_tok->session_key.flags |= ECRYPTFS_CONTAINS_DECRYPTED_KEY;
+ memcpy(crypt_stat->key, auth_tok->session_key.decrypted_key,
+ auth_tok->session_key.decrypted_key_size);
+ crypt_stat->key_size = ecryptfs_get_key_size_to_restore_key(
+ auth_tok->session_key.decrypted_key_size, crypt_stat);
+
+ ecryptfs_parse_full_cipher(full_cipher,
+ crypt_stat->cipher, crypt_stat->cipher_mode);
+
crypt_stat->flags |= ECRYPTFS_KEY_VALID;
if (ecryptfs_verbosity > 0) {
ecryptfs_printk(KERN_DEBUG, "Decrypted session key:\n");
ecryptfs_dump_hex(crypt_stat->key,
crypt_stat->key_size);
+
+ ecryptfs_dump_salt_hex(crypt_stat->key, crypt_stat->key_size,
+ crypt_stat);
}
out:
kfree(msg);
@@ -1392,6 +1408,7 @@ parse_tag_3_packet(struct ecryptfs_crypt_stat *crypt_stat,
struct ecryptfs_auth_tok_list_item *auth_tok_list_item;
size_t length_size;
int rc = 0;
+ char full_cipher[ECRYPTFS_MAX_CIPHER_NAME_SIZE];
(*packet_size) = 0;
(*new_auth_tok) = NULL;
@@ -1465,10 +1482,13 @@ parse_tag_3_packet(struct ecryptfs_crypt_stat *crypt_stat,
rc = -EINVAL;
goto out_free;
}
- rc = ecryptfs_cipher_code_to_string(crypt_stat->cipher,
+ rc = ecryptfs_cipher_code_to_string(full_cipher,
(u16)data[(*packet_size)]);
if (rc)
goto out_free;
+ ecryptfs_parse_full_cipher(full_cipher,
+ crypt_stat->cipher, crypt_stat->cipher_mode);
+
/* A little extra work to differentiate among the AES key
* sizes; see RFC2440 */
switch(data[(*packet_size)++]) {
@@ -1477,7 +1497,10 @@ parse_tag_3_packet(struct ecryptfs_crypt_stat *crypt_stat,
break;
default:
crypt_stat->key_size =
- (*new_auth_tok)->session_key.encrypted_key_size;
+ ecryptfs_get_key_size_to_restore_key(
+ (*new_auth_tok)->session_key.encrypted_key_size,
+ crypt_stat);
+
}
rc = ecryptfs_init_crypt_ctx(crypt_stat);
if (rc)
@@ -1736,7 +1759,7 @@ decrypt_passphrase_encrypted_session_key(struct ecryptfs_auth_tok *auth_tok,
NULL, NULL);
rc = crypto_skcipher_setkey(
tfm, auth_tok->token.password.session_key_encryption_key,
- crypt_stat->key_size);
+ auth_tok->token.password.session_key_encryption_key_bytes);
if (unlikely(rc < 0)) {
mutex_unlock(tfm_mutex);
printk(KERN_ERR "Error setting key for crypto context\n");
@@ -1761,6 +1784,8 @@ decrypt_passphrase_encrypted_session_key(struct ecryptfs_auth_tok *auth_tok,
crypt_stat->key_size);
ecryptfs_dump_hex(crypt_stat->key,
crypt_stat->key_size);
+ ecryptfs_dump_salt_hex(crypt_stat->key, crypt_stat->key_size,
+ crypt_stat);
}
out:
skcipher_request_free(req);
@@ -1998,12 +2023,17 @@ pki_encrypt_session_key(struct key *auth_tok_key,
size_t payload_len = 0;
struct ecryptfs_message *msg;
int rc;
+ unsigned char final[2*ECRYPTFS_MAX_CIPHER_NAME_SIZE+1];
rc = write_tag_66_packet(auth_tok->token.private_key.signature,
- ecryptfs_code_for_cipher_string(
- crypt_stat->cipher,
- crypt_stat->key_size),
- crypt_stat, &payload, &payload_len);
+ ecryptfs_code_for_cipher_string(
+ ecryptfs_get_full_cipher(
+ crypt_stat->cipher,
+ crypt_stat->cipher_mode,
+ final, sizeof(final)),
+ ecryptfs_get_key_size_to_enc_data(
+ crypt_stat)),
+ crypt_stat, &payload, &payload_len);
up_write(&(auth_tok_key->sem));
key_put(auth_tok_key);
if (rc) {
@@ -2061,7 +2091,7 @@ write_tag_1_packet(char *dest, size_t *remaining_bytes,
ecryptfs_from_hex(key_rec->sig, auth_tok->token.private_key.signature,
ECRYPTFS_SIG_SIZE);
encrypted_session_key_valid = 0;
- for (i = 0; i < crypt_stat->key_size; i++)
+ for (i = 0; i < ecryptfs_get_key_size_to_store_key(crypt_stat); i++)
encrypted_session_key_valid |=
auth_tok->session_key.encrypted_key[i];
if (encrypted_session_key_valid) {
@@ -2215,6 +2245,7 @@ write_tag_3_packet(char *dest, size_t *remaining_bytes,
u8 cipher_code;
size_t packet_size_length;
size_t max_packet_size;
+ unsigned char final[2*ECRYPTFS_MAX_CIPHER_NAME_SIZE+1];
struct ecryptfs_mount_crypt_stat *mount_crypt_stat =
crypt_stat->mount_crypt_stat;
struct crypto_skcipher *tfm;
@@ -2244,13 +2275,14 @@ write_tag_3_packet(char *dest, size_t *remaining_bytes,
mount_crypt_stat->global_default_cipher_key_size;
if (auth_tok->session_key.encrypted_key_size == 0)
auth_tok->session_key.encrypted_key_size =
- crypt_stat->key_size;
+ ecryptfs_get_key_size_to_store_key(crypt_stat);
if (crypt_stat->key_size == 24
&& strcmp("aes", crypt_stat->cipher) == 0) {
memset((crypt_stat->key + 24), 0, 8);
auth_tok->session_key.encrypted_key_size = 32;
} else
- auth_tok->session_key.encrypted_key_size = crypt_stat->key_size;
+ auth_tok->session_key.encrypted_key_size =
+ ecryptfs_get_key_size_to_store_key(crypt_stat);
key_rec->enc_key_size =
auth_tok->session_key.encrypted_key_size;
encrypted_session_key_valid = 0;
@@ -2274,8 +2306,8 @@ write_tag_3_packet(char *dest, size_t *remaining_bytes,
auth_tok->token.password.
session_key_encryption_key_bytes);
memcpy(session_key_encryption_key,
- auth_tok->token.password.session_key_encryption_key,
- crypt_stat->key_size);
+ auth_tok->token.password.session_key_encryption_key,
+ auth_tok->token.password.session_key_encryption_key_bytes);
ecryptfs_printk(KERN_DEBUG,
"Cached session key encryption key:\n");
if (ecryptfs_verbosity > 0)
@@ -2308,7 +2340,7 @@ write_tag_3_packet(char *dest, size_t *remaining_bytes,
}
mutex_lock(tfm_mutex);
rc = crypto_skcipher_setkey(tfm, session_key_encryption_key,
- crypt_stat->key_size);
+ auth_tok->token.password.session_key_encryption_key_bytes);
if (rc < 0) {
mutex_unlock(tfm_mutex);
ecryptfs_printk(KERN_ERR, "Error setting key for crypto "
@@ -2332,6 +2364,8 @@ write_tag_3_packet(char *dest, size_t *remaining_bytes,
rc = 0;
ecryptfs_printk(KERN_DEBUG, "Encrypting [%zd] bytes of the key\n",
crypt_stat->key_size);
+ ecryptfs_printk(KERN_DEBUG, "Encrypting [%zd] bytes of the salt key\n",
+ ecryptfs_get_salt_size_for_cipher(crypt_stat));
skcipher_request_set_crypt(req, src_sg, dst_sg,
(*key_rec).enc_key_size, NULL);
rc = crypto_skcipher_encrypt(req);
@@ -2382,8 +2416,10 @@ write_tag_3_packet(char *dest, size_t *remaining_bytes,
dest[(*packet_size)++] = 0x04; /* version 4 */
/* TODO: Break from RFC2440 so that arbitrary ciphers can be
* specified with strings */
- cipher_code = ecryptfs_code_for_cipher_string(crypt_stat->cipher,
- crypt_stat->key_size);
+ cipher_code = ecryptfs_code_for_cipher_string(
+ ecryptfs_get_full_cipher(crypt_stat->cipher,
+ crypt_stat->cipher_mode, final, sizeof(final)),
+ crypt_stat->key_size);
if (cipher_code == 0) {
ecryptfs_printk(KERN_WARNING, "Unable to generate code for "
"cipher [%s]\n", crypt_stat->cipher);
diff --git a/fs/ecryptfs/kthread.c b/fs/ecryptfs/kthread.c
index 866bb18..e818f5a 100644
--- a/fs/ecryptfs/kthread.c
+++ b/fs/ecryptfs/kthread.c
@@ -25,6 +25,7 @@
#include <linux/slab.h>
#include <linux/wait.h>
#include <linux/mount.h>
+#include <linux/file.h>
#include "ecryptfs_kernel.h"
struct ecryptfs_open_req {
@@ -147,7 +148,7 @@ int ecryptfs_privileged_open(struct file **lower_file,
flags |= IS_RDONLY(d_inode(lower_dentry)) ? O_RDONLY : O_RDWR;
(*lower_file) = dentry_open(&req.path, flags, cred);
if (!IS_ERR(*lower_file))
- goto out;
+ goto have_file;
if ((flags & O_ACCMODE) == O_RDONLY) {
rc = PTR_ERR((*lower_file));
goto out;
@@ -165,8 +166,16 @@ int ecryptfs_privileged_open(struct file **lower_file,
mutex_unlock(&ecryptfs_kthread_ctl.mux);
wake_up(&ecryptfs_kthread_ctl.wait);
wait_for_completion(&req.done);
- if (IS_ERR(*lower_file))
+ if (IS_ERR(*lower_file)) {
rc = PTR_ERR(*lower_file);
+ goto out;
+ }
+have_file:
+ if ((*lower_file)->f_op->mmap == NULL) {
+ fput(*lower_file);
+ *lower_file = NULL;
+ rc = -EMEDIUMTYPE;
+ }
out:
return rc;
}
diff --git a/fs/ecryptfs/main.c b/fs/ecryptfs/main.c
index 151872d..5df0d73 100644
--- a/fs/ecryptfs/main.c
+++ b/fs/ecryptfs/main.c
@@ -155,16 +155,42 @@ int ecryptfs_get_lower_file(struct dentry *dentry, struct inode *inode)
void ecryptfs_put_lower_file(struct inode *inode)
{
+ int ret = 0;
struct ecryptfs_inode_info *inode_info;
+ bool clear_cache_needed = false;
inode_info = ecryptfs_inode_to_private(inode);
if (atomic_dec_and_mutex_lock(&inode_info->lower_file_count,
&inode_info->lower_file_mutex)) {
+
+ if (get_events() && get_events()->is_hw_crypt_cb &&
+ get_events()->is_hw_crypt_cb())
+ clear_cache_needed = true;
+
+ if (clear_cache_needed) {
+ ret = vfs_fsync(inode_info->lower_file, false);
+
+ if (ret)
+ pr_err("failed to sync file ret = %d.\n", ret);
+ }
+
filemap_write_and_wait(inode->i_mapping);
fput(inode_info->lower_file);
inode_info->lower_file = NULL;
mutex_unlock(&inode_info->lower_file_mutex);
+
+ if (clear_cache_needed) {
+ truncate_inode_pages_fill_zero(inode->i_mapping, 0);
+ truncate_inode_pages_fill_zero(
+ ecryptfs_inode_to_lower(inode)->i_mapping, 0);
+ }
+
+ if (get_events() && get_events()->release_cb)
+ get_events()->release_cb(
+ ecryptfs_inode_to_lower(inode));
}
+
+
}
enum { ecryptfs_opt_sig, ecryptfs_opt_ecryptfs_sig,
@@ -279,6 +305,7 @@ static int ecryptfs_parse_options(struct ecryptfs_sb_info *sbi, char *options,
char *cipher_key_bytes_src;
char *fn_cipher_key_bytes_src;
u8 cipher_code;
+ unsigned char final[2*ECRYPTFS_MAX_CIPHER_NAME_SIZE+1];
*check_ruid = 0;
@@ -308,12 +335,14 @@ static int ecryptfs_parse_options(struct ecryptfs_sb_info *sbi, char *options,
case ecryptfs_opt_ecryptfs_cipher:
cipher_name_src = args[0].from;
cipher_name_dst =
- mount_crypt_stat->
- global_default_cipher_name;
- strncpy(cipher_name_dst, cipher_name_src,
- ECRYPTFS_MAX_CIPHER_NAME_SIZE);
- cipher_name_dst[ECRYPTFS_MAX_CIPHER_NAME_SIZE] = '\0';
+ mount_crypt_stat->global_default_cipher_name;
+
+ ecryptfs_parse_full_cipher(cipher_name_src,
+ mount_crypt_stat->global_default_cipher_name,
+ mount_crypt_stat->global_default_cipher_mode);
+
cipher_name_set = 1;
+
break;
case ecryptfs_opt_ecryptfs_key_bytes:
cipher_key_bytes_src = args[0].from;
@@ -410,24 +439,35 @@ static int ecryptfs_parse_options(struct ecryptfs_sb_info *sbi, char *options,
strcpy(mount_crypt_stat->global_default_cipher_name,
ECRYPTFS_DEFAULT_CIPHER);
}
+
if ((mount_crypt_stat->flags & ECRYPTFS_GLOBAL_ENCRYPT_FILENAMES)
&& !fn_cipher_name_set)
strcpy(mount_crypt_stat->global_default_fn_cipher_name,
mount_crypt_stat->global_default_cipher_name);
+
if (!cipher_key_bytes_set)
mount_crypt_stat->global_default_cipher_key_size = 0;
+
if ((mount_crypt_stat->flags & ECRYPTFS_GLOBAL_ENCRYPT_FILENAMES)
&& !fn_cipher_key_bytes_set)
mount_crypt_stat->global_default_fn_cipher_key_bytes =
mount_crypt_stat->global_default_cipher_key_size;
cipher_code = ecryptfs_code_for_cipher_string(
- mount_crypt_stat->global_default_cipher_name,
+ ecryptfs_get_full_cipher(
+ mount_crypt_stat->global_default_cipher_name,
+ mount_crypt_stat->global_default_cipher_mode,
+ final, sizeof(final)),
mount_crypt_stat->global_default_cipher_key_size);
if (!cipher_code) {
- ecryptfs_printk(KERN_ERR,
- "eCryptfs doesn't support cipher: %s",
- mount_crypt_stat->global_default_cipher_name);
+ ecryptfs_printk(
+ KERN_ERR,
+ "eCryptfs doesn't support cipher: %s and key size %zu",
+ ecryptfs_get_full_cipher(
+ mount_crypt_stat->global_default_cipher_name,
+ mount_crypt_stat->global_default_cipher_mode,
+ final, sizeof(final)),
+ mount_crypt_stat->global_default_cipher_key_size);
rc = -EINVAL;
goto out;
}
@@ -487,6 +527,7 @@ static struct file_system_type ecryptfs_fs_type;
* @dev_name: The path to mount over
* @raw_data: The options passed into the kernel
*/
+
static struct dentry *ecryptfs_mount(struct file_system_type *fs_type, int flags,
const char *dev_name, void *raw_data)
{
@@ -557,6 +598,11 @@ static struct dentry *ecryptfs_mount(struct file_system_type *fs_type, int flags
ecryptfs_set_superblock_lower(s, path.dentry->d_sb);
+
+ if (get_events() && get_events()->is_hw_crypt_cb &&
+ get_events()->is_hw_crypt_cb())
+ drop_pagecache_sb(ecryptfs_superblock_to_lower(s), 0);
+
/**
* Set the POSIX ACL flag based on whether they're enabled in the lower
* mount.
@@ -896,6 +942,7 @@ static void __exit ecryptfs_exit(void)
do_sysfs_unregistration();
unregister_filesystem(&ecryptfs_fs_type);
ecryptfs_free_kmem_caches();
+ ecryptfs_free_events();
}
MODULE_AUTHOR("Michael A. Halcrow <mhalcrow@us.ibm.com>");
diff --git a/fs/ecryptfs/mmap.c b/fs/ecryptfs/mmap.c
index 1f0c471..1b42bd8 100644
--- a/fs/ecryptfs/mmap.c
+++ b/fs/ecryptfs/mmap.c
@@ -552,10 +552,16 @@ static sector_t ecryptfs_bmap(struct address_space *mapping, sector_t block)
return rc;
}
+void ecryptfs_freepage(struct page *page)
+{
+ zero_user(page, 0, PAGE_SIZE);
+}
+
const struct address_space_operations ecryptfs_aops = {
.writepage = ecryptfs_writepage,
.readpage = ecryptfs_readpage,
.write_begin = ecryptfs_write_begin,
.write_end = ecryptfs_write_end,
.bmap = ecryptfs_bmap,
+ .freepage = ecryptfs_freepage,
};
diff --git a/fs/ecryptfs/super.c b/fs/ecryptfs/super.c
index 85411ce..de1fd8e 100644
--- a/fs/ecryptfs/super.c
+++ b/fs/ecryptfs/super.c
@@ -71,6 +71,9 @@ static void ecryptfs_i_callback(struct rcu_head *head)
{
struct inode *inode = container_of(head, struct inode, i_rcu);
struct ecryptfs_inode_info *inode_info;
+ if (inode == NULL)
+ return;
+
inode_info = ecryptfs_inode_to_private(inode);
kmem_cache_free(ecryptfs_inode_info_cache, inode_info);
@@ -90,9 +93,12 @@ static void ecryptfs_destroy_inode(struct inode *inode)
struct ecryptfs_inode_info *inode_info;
inode_info = ecryptfs_inode_to_private(inode);
+
BUG_ON(inode_info->lower_file);
+
ecryptfs_destroy_crypt_stat(&inode_info->crypt_stat);
call_rcu(&inode->i_rcu, ecryptfs_i_callback);
+
}
/**
@@ -151,6 +157,9 @@ static int ecryptfs_show_options(struct seq_file *m, struct dentry *root)
struct ecryptfs_mount_crypt_stat *mount_crypt_stat =
&ecryptfs_superblock_to_private(sb)->mount_crypt_stat;
struct ecryptfs_global_auth_tok *walker;
+ unsigned char final[2*ECRYPTFS_MAX_CIPHER_NAME_SIZE+1];
+
+ memset(final, 0, sizeof(final));
mutex_lock(&mount_crypt_stat->global_auth_tok_list_mutex);
list_for_each_entry(walker,
@@ -164,7 +173,10 @@ static int ecryptfs_show_options(struct seq_file *m, struct dentry *root)
mutex_unlock(&mount_crypt_stat->global_auth_tok_list_mutex);
seq_printf(m, ",ecryptfs_cipher=%s",
- mount_crypt_stat->global_default_cipher_name);
+ ecryptfs_get_full_cipher(
+ mount_crypt_stat->global_default_cipher_name,
+ mount_crypt_stat->global_default_cipher_mode,
+ final, sizeof(final)));
if (mount_crypt_stat->global_default_cipher_key_size)
seq_printf(m, ",ecryptfs_key_bytes=%zd",
diff --git a/fs/eventpoll.c b/fs/eventpoll.c
index 5b96ba7..f509702 100644
--- a/fs/eventpoll.c
+++ b/fs/eventpoll.c
@@ -1265,15 +1265,23 @@ static int ep_create_wakeup_source(struct epitem *epi)
{
const char *name;
struct wakeup_source *ws;
+ char task_comm_buf[TASK_COMM_LEN];
+ char buf[64];
+
+ get_task_comm(task_comm_buf, current);
if (!epi->ep->ws) {
- epi->ep->ws = wakeup_source_register("eventpoll");
+ snprintf(buf, sizeof(buf), "epoll_%.*s_epollfd",
+ (int)sizeof(task_comm_buf), task_comm_buf);
+ epi->ep->ws = wakeup_source_register(buf);
if (!epi->ep->ws)
return -ENOMEM;
}
name = epi->ffd.file->f_path.dentry->d_name.name;
- ws = wakeup_source_register(name);
+ snprintf(buf, sizeof(buf), "epoll_%.*s_file:%s",
+ (int)sizeof(task_comm_buf), task_comm_buf, name);
+ ws = wakeup_source_register(buf);
if (!ws)
return -ENOMEM;
diff --git a/fs/exec.c b/fs/exec.c
index 3e2de29..df13a19 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1228,15 +1228,14 @@ static int de_thread(struct task_struct *tsk)
return -EAGAIN;
}
-char *get_task_comm(char *buf, struct task_struct *tsk)
+char *__get_task_comm(char *buf, size_t buf_size, struct task_struct *tsk)
{
- /* buf must be at least sizeof(tsk->comm) in size */
task_lock(tsk);
- strncpy(buf, tsk->comm, sizeof(tsk->comm));
+ strncpy(buf, tsk->comm, buf_size);
task_unlock(tsk);
return buf;
}
-EXPORT_SYMBOL_GPL(get_task_comm);
+EXPORT_SYMBOL_GPL(__get_task_comm);
/*
* These functions flushes out all traces of the currently running executable
diff --git a/fs/exofs/super.c b/fs/exofs/super.c
index 1076a42..0c48138 100644
--- a/fs/exofs/super.c
+++ b/fs/exofs/super.c
@@ -100,6 +100,7 @@ static int parse_options(char *options, struct exofs_mountopt *opts)
token = match_token(p, tokens, args);
switch (token) {
case Opt_name:
+ kfree(opts->dev_name);
opts->dev_name = match_strdup(&args[0]);
if (unlikely(!opts->dev_name)) {
EXOFS_ERR("Error allocating dev_name");
@@ -868,8 +869,10 @@ static struct dentry *exofs_mount(struct file_system_type *type,
int ret;
ret = parse_options(data, &opts);
- if (ret)
+ if (ret) {
+ kfree(opts.dev_name);
return ERR_PTR(ret);
+ }
if (!opts.dev_name)
opts.dev_name = dev_name;
diff --git a/fs/exportfs/expfs.c b/fs/exportfs/expfs.c
index a4b531b..7a7bba7 100644
--- a/fs/exportfs/expfs.c
+++ b/fs/exportfs/expfs.c
@@ -76,7 +76,7 @@ static bool dentry_connected(struct dentry *dentry)
struct dentry *parent = dget_parent(dentry);
dput(dentry);
- if (IS_ROOT(dentry)) {
+ if (dentry == parent) {
dput(parent);
return false;
}
diff --git a/fs/ext2/inode.c b/fs/ext2/inode.c
index 41b8b44..85449a6 100644
--- a/fs/ext2/inode.c
+++ b/fs/ext2/inode.c
@@ -1258,21 +1258,11 @@ static void __ext2_truncate_blocks(struct inode *inode, loff_t offset)
static void ext2_truncate_blocks(struct inode *inode, loff_t offset)
{
- /*
- * XXX: it seems like a bug here that we don't allow
- * IS_APPEND inode to have blocks-past-i_size trimmed off.
- * review and fix this.
- *
- * Also would be nice to be able to handle IO errors and such,
- * but that's probably too much to ask.
- */
if (!(S_ISREG(inode->i_mode) || S_ISDIR(inode->i_mode) ||
S_ISLNK(inode->i_mode)))
return;
if (ext2_inode_is_fast_symlink(inode))
return;
- if (IS_APPEND(inode) || IS_IMMUTABLE(inode))
- return;
dax_sem_down_write(EXT2_I(inode));
__ext2_truncate_blocks(inode, offset);
diff --git a/fs/ext2/namei.c b/fs/ext2/namei.c
index 814e405..c8efc5e 100644
--- a/fs/ext2/namei.c
+++ b/fs/ext2/namei.c
@@ -40,8 +40,7 @@ static inline int ext2_add_nondir(struct dentry *dentry, struct inode *inode)
{
int err = ext2_add_link(dentry, inode);
if (!err) {
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
return 0;
}
inode_dec_link_count(inode);
@@ -268,8 +267,7 @@ static int ext2_mkdir(struct inode * dir, struct dentry * dentry, umode_t mode)
if (err)
goto out_fail;
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
out:
return err;
diff --git a/fs/ext2/xattr.c b/fs/ext2/xattr.c
index fbdb8f1..babef30 100644
--- a/fs/ext2/xattr.c
+++ b/fs/ext2/xattr.c
@@ -609,9 +609,9 @@ bad_block: ext2_error(sb, "ext2_xattr_set",
}
cleanup:
- brelse(bh);
if (!(bh && header == HDR(bh)))
kfree(header);
+ brelse(bh);
up_write(&EXT2_I(inode)->xattr_sem);
return error;
diff --git a/fs/ext4/Kconfig b/fs/ext4/Kconfig
index e9232a0..e38039f 100644
--- a/fs/ext4/Kconfig
+++ b/fs/ext4/Kconfig
@@ -109,16 +109,10 @@
decrypted pages in the page cache.
config EXT4_FS_ENCRYPTION
- bool "Ext4 FS Encryption"
- default n
+ bool
+ default y
depends on EXT4_ENCRYPTION
-config EXT4_FS_ICE_ENCRYPTION
- bool "Ext4 Encryption with ICE support"
- default n
- depends on EXT4_FS_ENCRYPTION
- depends on PFK
-
config EXT4_DEBUG
bool "EXT4 debugging support"
depends on EXT4_FS
diff --git a/fs/ext4/Makefile b/fs/ext4/Makefile
index b9dfa0d..354103f 100644
--- a/fs/ext4/Makefile
+++ b/fs/ext4/Makefile
@@ -1,7 +1,6 @@
#
# Makefile for the linux ext4-filesystem routines.
#
-ccflags-y += -Ifs/crypto
obj-$(CONFIG_EXT4_FS) += ext4.o
diff --git a/fs/ext4/balloc.c b/fs/ext4/balloc.c
index 176b4b2..2455fe1 100644
--- a/fs/ext4/balloc.c
+++ b/fs/ext4/balloc.c
@@ -183,7 +183,6 @@ static int ext4_init_block_bitmap(struct super_block *sb,
unsigned int bit, bit_max;
struct ext4_sb_info *sbi = EXT4_SB(sb);
ext4_fsblk_t start, tmp;
- int flex_bg = 0;
struct ext4_group_info *grp;
J_ASSERT_BH(bh, buffer_locked(bh));
@@ -216,22 +215,19 @@ static int ext4_init_block_bitmap(struct super_block *sb,
start = ext4_group_first_block_no(sb, block_group);
- if (ext4_has_feature_flex_bg(sb))
- flex_bg = 1;
-
/* Set bits for block and inode bitmaps, and inode table */
tmp = ext4_block_bitmap(sb, gdp);
- if (!flex_bg || ext4_block_in_group(sb, tmp, block_group))
+ if (ext4_block_in_group(sb, tmp, block_group))
ext4_set_bit(EXT4_B2C(sbi, tmp - start), bh->b_data);
tmp = ext4_inode_bitmap(sb, gdp);
- if (!flex_bg || ext4_block_in_group(sb, tmp, block_group))
+ if (ext4_block_in_group(sb, tmp, block_group))
ext4_set_bit(EXT4_B2C(sbi, tmp - start), bh->b_data);
tmp = ext4_inode_table(sb, gdp);
for (; tmp < ext4_inode_table(sb, gdp) +
sbi->s_itb_per_group; tmp++) {
- if (!flex_bg || ext4_block_in_group(sb, tmp, block_group))
+ if (ext4_block_in_group(sb, tmp, block_group))
ext4_set_bit(EXT4_B2C(sbi, tmp - start), bh->b_data);
}
@@ -320,6 +316,7 @@ static ext4_fsblk_t ext4_valid_block_bitmap(struct super_block *sb,
struct ext4_sb_info *sbi = EXT4_SB(sb);
ext4_grpblk_t offset;
ext4_grpblk_t next_zero_bit;
+ ext4_grpblk_t max_bit = EXT4_CLUSTERS_PER_GROUP(sb);
ext4_fsblk_t blk;
ext4_fsblk_t group_first_block;
@@ -337,20 +334,25 @@ static ext4_fsblk_t ext4_valid_block_bitmap(struct super_block *sb,
/* check whether block bitmap block number is set */
blk = ext4_block_bitmap(sb, desc);
offset = blk - group_first_block;
- if (!ext4_test_bit(EXT4_B2C(sbi, offset), bh->b_data))
+ if (offset < 0 || EXT4_B2C(sbi, offset) >= max_bit ||
+ !ext4_test_bit(EXT4_B2C(sbi, offset), bh->b_data))
/* bad block bitmap */
return blk;
/* check whether the inode bitmap block number is set */
blk = ext4_inode_bitmap(sb, desc);
offset = blk - group_first_block;
- if (!ext4_test_bit(EXT4_B2C(sbi, offset), bh->b_data))
+ if (offset < 0 || EXT4_B2C(sbi, offset) >= max_bit ||
+ !ext4_test_bit(EXT4_B2C(sbi, offset), bh->b_data))
/* bad block bitmap */
return blk;
/* check whether the inode table block number is set */
blk = ext4_inode_table(sb, desc);
offset = blk - group_first_block;
+ if (offset < 0 || EXT4_B2C(sbi, offset) >= max_bit ||
+ EXT4_B2C(sbi, offset + sbi->s_itb_per_group) >= max_bit)
+ return blk;
next_zero_bit = ext4_find_next_zero_bit(bh->b_data,
EXT4_B2C(sbi, offset + EXT4_SB(sb)->s_itb_per_group),
EXT4_B2C(sbi, offset));
@@ -376,6 +378,8 @@ static int ext4_validate_block_bitmap(struct super_block *sb,
return -EFSCORRUPTED;
ext4_lock_group(sb, block_group);
+ if (buffer_verified(bh))
+ goto verified;
if (unlikely(!ext4_block_bitmap_csum_verify(sb, block_group,
desc, bh))) {
ext4_unlock_group(sb, block_group);
@@ -398,6 +402,7 @@ static int ext4_validate_block_bitmap(struct super_block *sb,
return -EFSCORRUPTED;
}
set_buffer_verified(bh);
+verified:
ext4_unlock_group(sb, block_group);
return 0;
}
@@ -416,6 +421,7 @@ struct buffer_head *
ext4_read_block_bitmap_nowait(struct super_block *sb, ext4_group_t block_group)
{
struct ext4_group_desc *desc;
+ struct ext4_sb_info *sbi = EXT4_SB(sb);
struct buffer_head *bh;
ext4_fsblk_t bitmap_blk;
int err;
@@ -424,6 +430,12 @@ ext4_read_block_bitmap_nowait(struct super_block *sb, ext4_group_t block_group)
if (!desc)
return ERR_PTR(-EFSCORRUPTED);
bitmap_blk = ext4_block_bitmap(sb, desc);
+ if ((bitmap_blk <= le32_to_cpu(sbi->s_es->s_first_data_block)) ||
+ (bitmap_blk >= ext4_blocks_count(sbi->s_es))) {
+ ext4_error(sb, "Invalid block bitmap block %llu in "
+ "block_group %u", bitmap_blk, block_group);
+ return ERR_PTR(-EFSCORRUPTED);
+ }
bh = sb_getblk(sb, bitmap_blk);
if (unlikely(!bh)) {
ext4_error(sb, "Cannot get buffer for block bitmap - "
@@ -441,7 +453,16 @@ ext4_read_block_bitmap_nowait(struct super_block *sb, ext4_group_t block_group)
goto verify;
}
ext4_lock_group(sb, block_group);
- if (desc->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT)) {
+ if (ext4_has_group_desc_csum(sb) &&
+ (desc->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT))) {
+ if (block_group == 0) {
+ ext4_unlock_group(sb, block_group);
+ unlock_buffer(bh);
+ ext4_error(sb, "Block bitmap for bg 0 marked "
+ "uninitialized");
+ err = -EFSCORRUPTED;
+ goto out;
+ }
err = ext4_init_block_bitmap(sb, bh, block_group, desc);
set_bitmap_uptodate(bh);
set_buffer_uptodate(bh);
diff --git a/fs/ext4/dir.c b/fs/ext4/dir.c
index e8b3650..e16bc4c 100644
--- a/fs/ext4/dir.c
+++ b/fs/ext4/dir.c
@@ -74,7 +74,7 @@ int __ext4_check_dir_entry(const char *function, unsigned int line,
else if (unlikely(rlen < EXT4_DIR_REC_LEN(de->name_len)))
error_msg = "rec_len is too small for name_len";
else if (unlikely(((char *) de - buf) + rlen > size))
- error_msg = "directory entry across range";
+ error_msg = "directory entry overrun";
else if (unlikely(le32_to_cpu(de->inode) >
le32_to_cpu(EXT4_SB(dir->i_sb)->s_es->s_inodes_count)))
error_msg = "inode out of bounds";
@@ -83,18 +83,16 @@ int __ext4_check_dir_entry(const char *function, unsigned int line,
if (filp)
ext4_error_file(filp, function, line, bh->b_blocknr,
- "bad entry in directory: %s - offset=%u(%u), "
- "inode=%u, rec_len=%d, name_len=%d",
- error_msg, (unsigned) (offset % size),
- offset, le32_to_cpu(de->inode),
- rlen, de->name_len);
+ "bad entry in directory: %s - offset=%u, "
+ "inode=%u, rec_len=%d, name_len=%d, size=%d",
+ error_msg, offset, le32_to_cpu(de->inode),
+ rlen, de->name_len, size);
else
ext4_error_inode(dir, function, line, bh->b_blocknr,
- "bad entry in directory: %s - offset=%u(%u), "
- "inode=%u, rec_len=%d, name_len=%d",
- error_msg, (unsigned) (offset % size),
- offset, le32_to_cpu(de->inode),
- rlen, de->name_len);
+ "bad entry in directory: %s - offset=%u, "
+ "inode=%u, rec_len=%d, name_len=%d, size=%d",
+ error_msg, offset, le32_to_cpu(de->inode),
+ rlen, de->name_len, size);
return 1;
}
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index c175391..5f3d2e2 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1533,11 +1533,6 @@ static inline struct timespec ext4_current_time(struct inode *inode)
static inline int ext4_valid_inum(struct super_block *sb, unsigned long ino)
{
return ino == EXT4_ROOT_INO ||
- ino == EXT4_USR_QUOTA_INO ||
- ino == EXT4_GRP_QUOTA_INO ||
- ino == EXT4_BOOT_LOADER_INO ||
- ino == EXT4_JOURNAL_INO ||
- ino == EXT4_RESIZE_INO ||
(ino >= EXT4_FIRST_INO(sb) &&
ino <= le32_to_cpu(EXT4_SB(sb)->s_es->s_inodes_count));
}
@@ -3007,9 +3002,6 @@ extern struct buffer_head *ext4_get_first_inline_block(struct inode *inode,
extern int ext4_inline_data_fiemap(struct inode *inode,
struct fiemap_extent_info *fieinfo,
int *has_inline, __u64 start, __u64 len);
-extern int ext4_try_to_evict_inline_data(handle_t *handle,
- struct inode *inode,
- int needed);
extern void ext4_inline_data_truncate(struct inode *inode, int *has_inline);
extern int ext4_convert_inline_data(struct inode *inode);
diff --git a/fs/ext4/ext4_extents.h b/fs/ext4/ext4_extents.h
index 8ecf84b..a284fb2 100644
--- a/fs/ext4/ext4_extents.h
+++ b/fs/ext4/ext4_extents.h
@@ -103,6 +103,7 @@ struct ext4_extent_header {
};
#define EXT4_EXT_MAGIC cpu_to_le16(0xf30a)
+#define EXT4_MAX_EXTENT_DEPTH 5
#define EXT4_EXTENT_TAIL_OFFSET(hdr) \
(sizeof(struct ext4_extent_header) + \
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index a8573fa..054c385 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -881,6 +881,12 @@ ext4_find_extent(struct inode *inode, ext4_lblk_t block,
eh = ext_inode_hdr(inode);
depth = ext_depth(inode);
+ if (depth < 0 || depth > EXT4_MAX_EXTENT_DEPTH) {
+ EXT4_ERROR_INODE(inode, "inode has invalid extent depth: %d",
+ depth);
+ ret = -EFSCORRUPTED;
+ goto err;
+ }
if (path) {
ext4_ext_drop_refs(path);
@@ -5358,8 +5364,9 @@ ext4_ext_shift_extents(struct inode *inode, handle_t *handle,
stop = le32_to_cpu(extent->ee_block);
/*
- * In case of left shift, Don't start shifting extents until we make
- * sure the hole is big enough to accommodate the shift.
+ * For left shifts, make sure the hole on the left is big enough to
+ * accommodate the shift. For right shifts, make sure the last extent
+ * won't be shifted beyond EXT_MAX_BLOCKS.
*/
if (SHIFT == SHIFT_LEFT) {
path = ext4_find_extent(inode, start - 1, &path,
@@ -5379,9 +5386,14 @@ ext4_ext_shift_extents(struct inode *inode, handle_t *handle,
if ((start == ex_start && shift > ex_start) ||
(shift > start - ex_end)) {
- ext4_ext_drop_refs(path);
- kfree(path);
- return -EINVAL;
+ ret = -EINVAL;
+ goto out;
+ }
+ } else {
+ if (shift > EXT_MAX_BLOCKS -
+ (stop + ext4_ext_get_actual_len(extent))) {
+ ret = -EINVAL;
+ goto out;
}
}
diff --git a/fs/ext4/ialloc.c b/fs/ext4/ialloc.c
index ef76b83..004c088 100644
--- a/fs/ext4/ialloc.c
+++ b/fs/ext4/ialloc.c
@@ -88,6 +88,8 @@ static int ext4_validate_inode_bitmap(struct super_block *sb,
return -EFSCORRUPTED;
ext4_lock_group(sb, block_group);
+ if (buffer_verified(bh))
+ goto verified;
blk = ext4_inode_bitmap(sb, desc);
if (!ext4_inode_bitmap_csum_verify(sb, block_group, desc, bh,
EXT4_INODES_PER_GROUP(sb) / 8)) {
@@ -105,6 +107,7 @@ static int ext4_validate_inode_bitmap(struct super_block *sb,
return -EFSBADCRC;
}
set_buffer_verified(bh);
+verified:
ext4_unlock_group(sb, block_group);
return 0;
}
@@ -119,6 +122,7 @@ static struct buffer_head *
ext4_read_inode_bitmap(struct super_block *sb, ext4_group_t block_group)
{
struct ext4_group_desc *desc;
+ struct ext4_sb_info *sbi = EXT4_SB(sb);
struct buffer_head *bh = NULL;
ext4_fsblk_t bitmap_blk;
int err;
@@ -128,6 +132,12 @@ ext4_read_inode_bitmap(struct super_block *sb, ext4_group_t block_group)
return ERR_PTR(-EFSCORRUPTED);
bitmap_blk = ext4_inode_bitmap(sb, desc);
+ if ((bitmap_blk <= le32_to_cpu(sbi->s_es->s_first_data_block)) ||
+ (bitmap_blk >= ext4_blocks_count(sbi->s_es))) {
+ ext4_error(sb, "Invalid inode bitmap blk %llu in "
+ "block_group %u", bitmap_blk, block_group);
+ return ERR_PTR(-EFSCORRUPTED);
+ }
bh = sb_getblk(sb, bitmap_blk);
if (unlikely(!bh)) {
ext4_error(sb, "Cannot read inode bitmap - "
@@ -145,7 +155,16 @@ ext4_read_inode_bitmap(struct super_block *sb, ext4_group_t block_group)
}
ext4_lock_group(sb, block_group);
- if (desc->bg_flags & cpu_to_le16(EXT4_BG_INODE_UNINIT)) {
+ if (ext4_has_group_desc_csum(sb) &&
+ (desc->bg_flags & cpu_to_le16(EXT4_BG_INODE_UNINIT))) {
+ if (block_group == 0) {
+ ext4_unlock_group(sb, block_group);
+ unlock_buffer(bh);
+ ext4_error(sb, "Inode bitmap for bg 0 marked "
+ "uninitialized");
+ err = -EFSCORRUPTED;
+ goto out;
+ }
memset(bh->b_data, 0, (EXT4_INODES_PER_GROUP(sb) + 7) / 8);
ext4_mark_bitmap_end(EXT4_INODES_PER_GROUP(sb),
sb->s_blocksize * 8, bh->b_data);
@@ -919,7 +938,8 @@ struct inode *__ext4_new_inode(handle_t *handle, struct inode *dir,
/* recheck and clear flag under lock if we still need to */
ext4_lock_group(sb, group);
- if (gdp->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT)) {
+ if (ext4_has_group_desc_csum(sb) &&
+ (gdp->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT))) {
gdp->bg_flags &= cpu_to_le16(~EXT4_BG_BLOCK_UNINIT);
ext4_free_group_clusters_set(sb, gdp,
ext4_free_clusters_after_init(sb, group, gdp));
@@ -1300,7 +1320,10 @@ int ext4_init_inode_table(struct super_block *sb, ext4_group_t group,
ext4_itable_unused_count(sb, gdp)),
sbi->s_inodes_per_block);
- if ((used_blks < 0) || (used_blks > sbi->s_itb_per_group)) {
+ if ((used_blks < 0) || (used_blks > sbi->s_itb_per_group) ||
+ ((group == 0) && ((EXT4_INODES_PER_GROUP(sb) -
+ ext4_itable_unused_count(sb, gdp)) <
+ EXT4_FIRST_INO(sb)))) {
ext4_error(sb, "Something is wrong with group %u: "
"used itable blocks: %d; "
"itable unused count: %u",
diff --git a/fs/ext4/indirect.c b/fs/ext4/indirect.c
index bc15c2c..58229c1 100644
--- a/fs/ext4/indirect.c
+++ b/fs/ext4/indirect.c
@@ -560,10 +560,16 @@ int ext4_ind_map_blocks(handle_t *handle, struct inode *inode,
unsigned epb = inode->i_sb->s_blocksize / sizeof(u32);
int i;
- /* Count number blocks in a subtree under 'partial' */
- count = 1;
- for (i = 0; partial + i != chain + depth - 1; i++)
- count *= epb;
+ /*
+ * Count number blocks in a subtree under 'partial'. At each
+ * level we count number of complete empty subtrees beyond
+ * current offset and then descend into the subtree only
+ * partially beyond current offset.
+ */
+ count = 0;
+ for (i = partial - chain + 1; i < depth; i++)
+ count = count * epb + (epb - offsets[i] - 1);
+ count++;
/* Fill in size of a hole we found */
map->m_pblk = 0;
map->m_len = min_t(unsigned int, map->m_len, count);
diff --git a/fs/ext4/inline.c b/fs/ext4/inline.c
index 4d78b93..f4eb7e8 100644
--- a/fs/ext4/inline.c
+++ b/fs/ext4/inline.c
@@ -435,6 +435,7 @@ static int ext4_destroy_inline_data_nolock(handle_t *handle,
memset((void *)ext4_raw_inode(&is.iloc)->i_block,
0, EXT4_MIN_INLINE_DATA_SIZE);
+ memset(ei->i_data, 0, EXT4_MIN_INLINE_DATA_SIZE);
if (ext4_has_feature_extents(inode->i_sb)) {
if (S_ISDIR(inode->i_mode) ||
@@ -692,6 +693,10 @@ int ext4_try_to_write_inline_data(struct address_space *mapping,
goto convert;
}
+ ret = ext4_journal_get_write_access(handle, iloc.bh);
+ if (ret)
+ goto out;
+
flags |= AOP_FLAG_NOFS;
page = grab_cache_page_write_begin(mapping, 0, flags);
@@ -711,8 +716,11 @@ int ext4_try_to_write_inline_data(struct address_space *mapping,
if (!PageUptodate(page)) {
ret = ext4_read_inline_page(inode, page);
- if (ret < 0)
+ if (ret < 0) {
+ unlock_page(page);
+ put_page(page);
goto out_up_read;
+ }
}
ret = 1;
@@ -720,7 +728,7 @@ int ext4_try_to_write_inline_data(struct address_space *mapping,
out_up_read:
up_read(&EXT4_I(inode)->xattr_sem);
out:
- if (handle)
+ if (handle && (ret != 1))
ext4_journal_stop(handle);
brelse(iloc.bh);
return ret;
@@ -762,6 +770,7 @@ int ext4_write_inline_data_end(struct inode *inode, loff_t pos, unsigned len,
ext4_write_unlock_xattr(inode, &no_expand);
brelse(iloc.bh);
+ mark_inode_dirty(inode);
out:
return copied;
}
@@ -868,7 +877,7 @@ int ext4_da_write_inline_data_begin(struct address_space *mapping,
handle_t *handle;
struct page *page;
struct ext4_iloc iloc;
- int retries;
+ int retries = 0;
ret = ext4_get_inode_loc(inode, &iloc);
if (ret)
@@ -897,18 +906,17 @@ int ext4_da_write_inline_data_begin(struct address_space *mapping,
flags |= AOP_FLAG_NOFS;
if (ret == -ENOSPC) {
+ ext4_journal_stop(handle);
ret = ext4_da_convert_inline_data_to_extent(mapping,
inode,
flags,
fsdata);
- ext4_journal_stop(handle);
if (ret == -ENOSPC &&
ext4_should_retry_alloc(inode->i_sb, &retries))
goto retry_journal;
goto out;
}
-
page = grab_cache_page_write_begin(mapping, 0, flags);
if (!page) {
ret = -ENOMEM;
@@ -926,6 +934,9 @@ int ext4_da_write_inline_data_begin(struct address_space *mapping,
if (ret < 0)
goto out_release_page;
}
+ ret = ext4_journal_get_write_access(handle, iloc.bh);
+ if (ret)
+ goto out_release_page;
up_read(&EXT4_I(inode)->xattr_sem);
*pagep = page;
@@ -946,7 +957,6 @@ int ext4_da_write_inline_data_end(struct inode *inode, loff_t pos,
unsigned len, unsigned copied,
struct page *page)
{
- int i_size_changed = 0;
int ret;
ret = ext4_write_inline_data_end(inode, pos, len, copied, page);
@@ -964,10 +974,8 @@ int ext4_da_write_inline_data_end(struct inode *inode, loff_t pos,
* But it's important to update i_size while still holding page lock:
* page writeout could otherwise come in and zero beyond i_size.
*/
- if (pos+copied > inode->i_size) {
+ if (pos+copied > inode->i_size)
i_size_write(inode, pos+copied);
- i_size_changed = 1;
- }
unlock_page(page);
put_page(page);
@@ -977,8 +985,7 @@ int ext4_da_write_inline_data_end(struct inode *inode, loff_t pos,
* ordering of page lock and transaction start for journaling
* filesystems.
*/
- if (i_size_changed)
- mark_inode_dirty(inode);
+ mark_inode_dirty(inode);
return copied;
}
@@ -1764,6 +1771,7 @@ bool empty_inline_dir(struct inode *dir, int *has_inline_data)
{
int err, inline_size;
struct ext4_iloc iloc;
+ size_t inline_len;
void *inline_pos;
unsigned int offset;
struct ext4_dir_entry_2 *de;
@@ -1791,8 +1799,9 @@ bool empty_inline_dir(struct inode *dir, int *has_inline_data)
goto out;
}
+ inline_len = ext4_get_inline_size(dir);
offset = EXT4_INLINE_DOTDOT_SIZE;
- while (offset < dir->i_size) {
+ while (offset < inline_len) {
de = ext4_get_inline_entry(dir, &iloc, offset,
&inline_pos, &inline_size);
if (ext4_check_dir_entry(dir, NULL, de,
@@ -1873,42 +1882,6 @@ int ext4_inline_data_fiemap(struct inode *inode,
return (error < 0 ? error : 0);
}
-/*
- * Called during xattr set, and if we can sparse space 'needed',
- * just create the extent tree evict the data to the outer block.
- *
- * We use jbd2 instead of page cache to move data to the 1st block
- * so that the whole transaction can be committed as a whole and
- * the data isn't lost because of the delayed page cache write.
- */
-int ext4_try_to_evict_inline_data(handle_t *handle,
- struct inode *inode,
- int needed)
-{
- int error;
- struct ext4_xattr_entry *entry;
- struct ext4_inode *raw_inode;
- struct ext4_iloc iloc;
-
- error = ext4_get_inode_loc(inode, &iloc);
- if (error)
- return error;
-
- raw_inode = ext4_raw_inode(&iloc);
- entry = (struct ext4_xattr_entry *)((void *)raw_inode +
- EXT4_I(inode)->i_inline_off);
- if (EXT4_XATTR_LEN(entry->e_name_len) +
- EXT4_XATTR_SIZE(le32_to_cpu(entry->e_value_size)) < needed) {
- error = -ENOSPC;
- goto out;
- }
-
- error = ext4_convert_inline_data_nolock(handle, inode, &iloc);
-out:
- brelse(iloc.bh);
- return error;
-}
-
void ext4_inline_data_truncate(struct inode *inode, int *has_inline)
{
handle_t *handle;
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index ecee29a..fc96c01 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -42,7 +42,6 @@
#include "xattr.h"
#include "acl.h"
#include "truncate.h"
-//#include "ext4_ice.h"
#include <trace/events/ext4.h>
#include <trace/events/android_fs.h>
@@ -379,9 +378,9 @@ static int __check_block_validity(struct inode *inode, const char *func,
if (!ext4_data_block_valid(EXT4_SB(inode->i_sb), map->m_pblk,
map->m_len)) {
ext4_error_inode(inode, func, line, map->m_pblk,
- "lblock %lu mapped to illegal pblock "
+ "lblock %lu mapped to illegal pblock %llu "
"(length %d)", (unsigned long) map->m_lblk,
- map->m_len);
+ map->m_pblk, map->m_len);
return -EFSCORRUPTED;
}
return 0;
@@ -1332,10 +1331,11 @@ static int ext4_write_end(struct file *file,
loff_t old_size = inode->i_size;
int ret = 0, ret2;
int i_size_changed = 0;
+ int inline_data = ext4_has_inline_data(inode);
trace_android_fs_datawrite_end(inode, pos, len);
trace_ext4_write_end(inode, pos, len, copied);
- if (ext4_has_inline_data(inode)) {
+ if (inline_data) {
ret = ext4_write_inline_data_end(inode, pos, len,
copied, page);
if (ret < 0) {
@@ -1363,7 +1363,7 @@ static int ext4_write_end(struct file *file,
* ordering of page lock and transaction start for journaling
* filesystems.
*/
- if (i_size_changed)
+ if (i_size_changed || inline_data)
ext4_mark_inode_dirty(handle, inode);
if (pos + len > inode->i_size && ext4_can_truncate(inode))
@@ -1437,6 +1437,7 @@ static int ext4_journalled_write_end(struct file *file,
int partial = 0;
unsigned from, to;
int size_changed = 0;
+ int inline_data = ext4_has_inline_data(inode);
trace_android_fs_datawrite_end(inode, pos, len);
trace_ext4_journalled_write_end(inode, pos, len, copied);
@@ -1445,7 +1446,7 @@ static int ext4_journalled_write_end(struct file *file,
BUG_ON(!ext4_handle_valid(handle));
- if (ext4_has_inline_data(inode)) {
+ if (inline_data) {
ret = ext4_write_inline_data_end(inode, pos, len,
copied, page);
if (ret < 0) {
@@ -1476,7 +1477,7 @@ static int ext4_journalled_write_end(struct file *file,
if (old_size < pos)
pagecache_isize_extended(inode, old_size, pos);
- if (size_changed) {
+ if (size_changed || inline_data) {
ret2 = ext4_mark_inode_dirty(handle, inode);
if (!ret)
ret = ret2;
@@ -1974,11 +1975,7 @@ static int __ext4_journalled_writepage(struct page *page,
}
if (inline_data) {
- BUFFER_TRACE(inode_bh, "get write access");
- ret = ext4_journal_get_write_access(handle, inode_bh);
-
- err = ext4_handle_dirty_metadata(handle, inode, inode_bh);
-
+ ret = ext4_mark_inode_dirty(handle, inode);
} else {
ret = ext4_walk_page_buffers(handle, page_bufs, 0, len, NULL,
do_journal_get_write_access);
@@ -2571,8 +2568,8 @@ static int mpage_prepare_extent_to_map(struct mpage_da_data *mpd)
mpd->map.m_len = 0;
mpd->next_page = index;
while (index <= end) {
- nr_pages = pagevec_lookup_tag(&pvec, mapping, &index, tag,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
+ nr_pages = pagevec_lookup_range_tag(&pvec, mapping, &index, end,
+ tag);
if (nr_pages == 0)
goto out;
@@ -2580,16 +2577,6 @@ static int mpage_prepare_extent_to_map(struct mpage_da_data *mpd)
struct page *page = pvec.pages[i];
/*
- * At this point, the page may be truncated or
- * invalidated (changing page->mapping to NULL), or
- * even swizzled back from swapper_space to tmpfs file
- * mapping. However, page->index will not change
- * because we have a reference on the page.
- */
- if (page->index > end)
- goto out;
-
- /*
* Accumulated enough dirty pages? This doesn't apply
* to WB_SYNC_ALL mode. For integrity sync we have to
* keep going because someone may be concurrently
@@ -3511,9 +3498,9 @@ static ssize_t ext4_direct_IO_write(struct kiocb *iocb, struct iov_iter *iter)
get_block_func = ext4_dio_get_block_unwritten_async;
dio_flags = DIO_LOCKING;
}
-#if defined(CONFIG_EXT4_FS_ENCRYPTION)
- WARN_ON(ext4_encrypted_inode(inode) && S_ISREG(inode->i_mode)
- && !fscrypt_using_hardware_encryption(inode));
+#ifdef CONFIG_EXT4_FS_ENCRYPTION
+ BUG_ON(ext4_encrypted_inode(inode) && S_ISREG(inode->i_mode) &&
+ !fscrypt_using_hardware_encryption(inode));
#endif
if (IS_DAX(inode)) {
ret = dax_do_io(iocb, inode, iter, get_block_func,
@@ -3634,9 +3621,9 @@ static ssize_t ext4_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
ssize_t ret;
int rw = iov_iter_rw(iter);
-#if defined(CONFIG_EXT4_FS_ENCRYPTION)
- if (ext4_encrypted_inode(inode) && S_ISREG(inode->i_mode)
- && !fscrypt_using_hardware_encryption(inode))
+#ifdef CONFIG_EXT4_FS_ENCRYPTION
+ if (ext4_encrypted_inode(inode) && S_ISREG(inode->i_mode) &&
+ !fscrypt_using_hardware_encryption(inode))
return 0;
#endif
@@ -4101,28 +4088,28 @@ int ext4_punch_hole(struct inode *inode, loff_t offset, loff_t length)
EXT4_BLOCK_SIZE_BITS(sb);
stop_block = (offset + length) >> EXT4_BLOCK_SIZE_BITS(sb);
- /* If there are no blocks to remove, return now */
- if (first_block >= stop_block)
- goto out_stop;
+ /* If there are blocks to remove, do it */
+ if (stop_block > first_block) {
- down_write(&EXT4_I(inode)->i_data_sem);
- ext4_discard_preallocations(inode);
+ down_write(&EXT4_I(inode)->i_data_sem);
+ ext4_discard_preallocations(inode);
- ret = ext4_es_remove_extent(inode, first_block,
- stop_block - first_block);
- if (ret) {
+ ret = ext4_es_remove_extent(inode, first_block,
+ stop_block - first_block);
+ if (ret) {
+ up_write(&EXT4_I(inode)->i_data_sem);
+ goto out_stop;
+ }
+
+ if (ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS))
+ ret = ext4_ext_remove_space(inode, first_block,
+ stop_block - 1);
+ else
+ ret = ext4_ind_remove_space(handle, inode, first_block,
+ stop_block);
+
up_write(&EXT4_I(inode)->i_data_sem);
- goto out_stop;
}
-
- if (ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS))
- ret = ext4_ext_remove_space(inode, first_block,
- stop_block - 1);
- else
- ret = ext4_ind_remove_space(handle, inode, first_block,
- stop_block);
-
- up_write(&EXT4_I(inode)->i_data_sem);
if (IS_SYNC(inode))
ext4_handle_sync(handle);
@@ -4311,7 +4298,8 @@ static int __ext4_get_inode_loc(struct inode *inode,
int inodes_per_block, inode_offset;
iloc->bh = NULL;
- if (!ext4_valid_inum(sb, inode->i_ino))
+ if (inode->i_ino < EXT4_ROOT_INO ||
+ inode->i_ino > le32_to_cpu(EXT4_SB(sb)->s_es->s_inodes_count))
return -EFSCORRUPTED;
iloc->block_group = (inode->i_ino - 1) / EXT4_INODES_PER_GROUP(sb);
diff --git a/fs/ext4/ioctl.c b/fs/ext4/ioctl.c
index 3585e26..4bc35e6 100644
--- a/fs/ext4/ioctl.c
+++ b/fs/ext4/ioctl.c
@@ -347,7 +347,9 @@ static int ext4_ioctl_setproject(struct file *filp, __u32 projid)
}
brelse(iloc.bh);
- dquot_initialize(inode);
+ err = dquot_initialize(inode);
+ if (err)
+ return err;
handle = ext4_journal_start(inode, EXT4_HT_QUOTA,
EXT4_QUOTA_INIT_BLOCKS(sb) +
@@ -774,8 +776,8 @@ long ext4_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
return ext4_ext_precache(inode);
case EXT4_IOC_SET_ENCRYPTION_POLICY:
-// if (!ext4_has_feature_encrypt(sb))
-// return -EOPNOTSUPP;
+ if (!ext4_has_feature_encrypt(sb))
+ return -EOPNOTSUPP;
return fscrypt_ioctl_set_policy(filp, (const void __user *)arg);
case EXT4_IOC_GET_ENCRYPTION_PWSALT: {
diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index 4beca06..3d6f73e 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -26,6 +26,7 @@
#include <linux/log2.h>
#include <linux/module.h>
#include <linux/slab.h>
+#include <linux/nospec.h>
#include <linux/backing-dev.h>
#include <trace/events/ext4.h>
@@ -2144,7 +2145,8 @@ ext4_mb_regular_allocator(struct ext4_allocation_context *ac)
* This should tell if fe_len is exactly power of 2
*/
if ((ac->ac_g_ex.fe_len & (~(1 << (i - 1)))) == 0)
- ac->ac_2order = i - 1;
+ ac->ac_2order = array_index_nospec(i - 1,
+ sb->s_blocksize_bits + 2);
}
/* if stream allocation is enabled, use global goal */
@@ -2444,7 +2446,8 @@ int ext4_mb_add_groupinfo(struct super_block *sb, ext4_group_t group,
* initialize bb_free to be able to skip
* empty groups without initialization
*/
- if (desc->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT)) {
+ if (ext4_has_group_desc_csum(sb) &&
+ (desc->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT))) {
meta_group_info[i]->bb_free =
ext4_free_clusters_after_init(sb, group, desc);
} else {
@@ -2970,7 +2973,8 @@ ext4_mb_mark_diskspace_used(struct ext4_allocation_context *ac,
#endif
ext4_set_bits(bitmap_bh->b_data, ac->ac_b_ex.fe_start,
ac->ac_b_ex.fe_len);
- if (gdp->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT)) {
+ if (ext4_has_group_desc_csum(sb) &&
+ (gdp->bg_flags & cpu_to_le16(EXT4_BG_BLOCK_UNINIT))) {
gdp->bg_flags &= cpu_to_le16(~EXT4_BG_BLOCK_UNINIT);
ext4_free_group_clusters_set(sb, gdp,
ext4_free_clusters_after_init(sb,
diff --git a/fs/ext4/mmp.c b/fs/ext4/mmp.c
index d89754e..c2e830a 100644
--- a/fs/ext4/mmp.c
+++ b/fs/ext4/mmp.c
@@ -48,7 +48,6 @@ static int write_mmp_block(struct super_block *sb, struct buffer_head *bh)
*/
sb_start_write(sb);
ext4_mmp_csum_set(sb, mmp);
- mark_buffer_dirty(bh);
lock_buffer(bh);
bh->b_end_io = end_buffer_write_sync;
get_bh(bh);
diff --git a/fs/ext4/move_extent.c b/fs/ext4/move_extent.c
index 578f8c3..c4434bd 100644
--- a/fs/ext4/move_extent.c
+++ b/fs/ext4/move_extent.c
@@ -526,9 +526,13 @@ mext_check_arguments(struct inode *orig_inode,
orig_inode->i_ino, donor_inode->i_ino);
return -EINVAL;
}
- if (orig_eof < orig_start + *len - 1)
+ if (orig_eof <= orig_start)
+ *len = 0;
+ else if (orig_eof < orig_start + *len - 1)
*len = orig_eof - orig_start;
- if (donor_eof < donor_start + *len - 1)
+ if (donor_eof <= donor_start)
+ *len = 0;
+ else if (donor_eof < donor_start + *len - 1)
*len = donor_eof - donor_start;
if (!*len) {
ext4_debug("ext4 move extent: len should not be 0 "
diff --git a/fs/ext4/namei.c b/fs/ext4/namei.c
index e3183e8..b7b58b1 100644
--- a/fs/ext4/namei.c
+++ b/fs/ext4/namei.c
@@ -124,6 +124,7 @@ static struct buffer_head *__ext4_read_dirblock(struct inode *inode,
if (!is_dx_block && type == INDEX) {
ext4_error_inode(inode, func, line, block,
"directory leaf block found instead of index block");
+ brelse(bh);
return ERR_PTR(-EFSCORRUPTED);
}
if (!ext4_has_metadata_csum(inode->i_sb) ||
@@ -1385,6 +1386,7 @@ static struct buffer_head * ext4_find_entry (struct inode *dir,
goto cleanup_and_exit;
dxtrace(printk(KERN_DEBUG "ext4_find_entry: dx failed, "
"falling back\n"));
+ ret = NULL;
}
nblocks = dir->i_size >> EXT4_BLOCK_SIZE_BITS(sb);
if (!nblocks) {
@@ -1545,24 +1547,14 @@ static struct dentry *ext4_lookup(struct inode *dir, struct dentry *dentry, unsi
struct inode *inode;
struct ext4_dir_entry_2 *de;
struct buffer_head *bh;
+ int err;
- if (ext4_encrypted_inode(dir)) {
- int res = fscrypt_get_encryption_info(dir);
+ err = fscrypt_prepare_lookup(dir, dentry, flags);
+ if (err)
+ return ERR_PTR(err);
- /*
- * DCACHE_ENCRYPTED_WITH_KEY is set if the dentry is
- * created while the directory was encrypted and we
- * have access to the key.
- */
- if (fscrypt_has_encryption_key(dir))
- fscrypt_set_encrypted_dentry(dentry);
- fscrypt_set_d_op(dentry);
- if (res && res != -ENOKEY)
- return ERR_PTR(res);
- }
-
- if (dentry->d_name.len > EXT4_NAME_LEN)
- return ERR_PTR(-ENAMETOOLONG);
+ if (dentry->d_name.len > EXT4_NAME_LEN)
+ return ERR_PTR(-ENAMETOOLONG);
bh = ext4_find_entry(dir, &dentry->d_name, &de, NULL);
if (IS_ERR(bh))
@@ -2387,8 +2379,7 @@ static int ext4_add_nondir(handle_t *handle,
int err = ext4_add_entry(handle, dentry, inode);
if (!err) {
ext4_mark_inode_dirty(handle, inode);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
return 0;
}
drop_nlink(inode);
@@ -2627,8 +2618,7 @@ static int ext4_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode)
err = ext4_mark_inode_dirty(handle, dir);
if (err)
goto out_clear_inode;
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
if (IS_DIRSYNC(dir))
ext4_handle_sync(handle);
@@ -2788,7 +2778,9 @@ int ext4_orphan_add(handle_t *handle, struct inode *inode)
list_del_init(&EXT4_I(inode)->i_orphan);
mutex_unlock(&sbi->s_orphan_lock);
}
- }
+ } else
+ brelse(iloc.bh);
+
jbd_debug(4, "superblock will point to %lu\n", inode->i_ino);
jbd_debug(4, "orphan inode %lu will point to %d\n",
inode->i_ino, NEXT_ORPHAN(inode));
@@ -3445,6 +3437,12 @@ static int ext4_rename(struct inode *old_dir, struct dentry *old_dentry,
int credits;
u8 old_file_type;
+ if (new.inode && new.inode->i_nlink == 0) {
+ EXT4_ERROR_INODE(new.inode,
+ "target of rename is already freed");
+ return -EFSCORRUPTED;
+ }
+
if ((ext4_test_inode_flag(new_dir, EXT4_INODE_PROJINHERIT)) &&
(!projid_eq(EXT4_I(new_dir)->i_projid,
EXT4_I(old_dentry->d_inode)->i_projid)))
diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c
index 8d4ec1a..012b96f 100644
--- a/fs/ext4/page-io.c
+++ b/fs/ext4/page-io.c
@@ -28,7 +28,6 @@
#include "ext4_jbd2.h"
#include "xattr.h"
#include "acl.h"
-//#include "ext4_ice.h"
static struct kmem_cache *io_end_cachep;
@@ -470,9 +469,9 @@ int ext4_bio_write_page(struct ext4_io_submit *io,
gfp_t gfp_flags = GFP_NOFS;
retry_encrypt:
- if (!fscrypt_using_hardware_encryption(inode))
- data_page = fscrypt_encrypt_page(inode, page, PAGE_SIZE, 0,
- page->index, gfp_flags);
+ if (!fscrypt_using_hardware_encryption(inode))
+ data_page = fscrypt_encrypt_page(inode, page, PAGE_SIZE,
+ 0, page->index, gfp_flags);
if (IS_ERR(data_page)) {
ret = PTR_ERR(data_page);
if (ret == -ENOMEM && wbc->sync_mode == WB_SYNC_ALL) {
diff --git a/fs/ext4/resize.c b/fs/ext4/resize.c
index 95bf466..58e6b8a 100644
--- a/fs/ext4/resize.c
+++ b/fs/ext4/resize.c
@@ -18,6 +18,7 @@
int ext4_resize_begin(struct super_block *sb)
{
+ struct ext4_sb_info *sbi = EXT4_SB(sb);
int ret = 0;
if (!capable(CAP_SYS_RESOURCE))
@@ -28,7 +29,7 @@ int ext4_resize_begin(struct super_block *sb)
* because the user tools have no way of handling this. Probably a
* bad time to do it anyways.
*/
- if (EXT4_SB(sb)->s_sbh->b_blocknr !=
+ if (EXT4_B2C(sbi, sbi->s_sbh->b_blocknr) !=
le32_to_cpu(EXT4_SB(sb)->s_es->s_first_data_block)) {
ext4_warning(sb, "won't resize using backup superblock at %llu",
(unsigned long long)EXT4_SB(sb)->s_sbh->b_blocknr);
@@ -441,16 +442,18 @@ static int set_flexbg_block_bitmap(struct super_block *sb, handle_t *handle,
BUFFER_TRACE(bh, "get_write_access");
err = ext4_journal_get_write_access(handle, bh);
- if (err)
+ if (err) {
+ brelse(bh);
return err;
+ }
ext4_debug("mark block bitmap %#04llx (+%llu/%u)\n", block,
block - start, count2);
ext4_set_bits(bh->b_data, block - start, count2);
err = ext4_handle_dirty_metadata(handle, NULL, bh);
+ brelse(bh);
if (unlikely(err))
return err;
- brelse(bh);
}
return 0;
@@ -587,7 +590,6 @@ static int setup_new_flex_group_blocks(struct super_block *sb,
bh = bclean(handle, sb, block);
if (IS_ERR(bh)) {
err = PTR_ERR(bh);
- bh = NULL;
goto out;
}
overhead = ext4_group_overhead_blocks(sb, group);
@@ -599,9 +601,9 @@ static int setup_new_flex_group_blocks(struct super_block *sb,
ext4_mark_bitmap_end(group_data[i].blocks_count,
sb->s_blocksize * 8, bh->b_data);
err = ext4_handle_dirty_metadata(handle, NULL, bh);
+ brelse(bh);
if (err)
goto out;
- brelse(bh);
handle_ib:
if (bg_flags[i] & EXT4_BG_INODE_UNINIT)
@@ -616,18 +618,16 @@ static int setup_new_flex_group_blocks(struct super_block *sb,
bh = bclean(handle, sb, block);
if (IS_ERR(bh)) {
err = PTR_ERR(bh);
- bh = NULL;
goto out;
}
ext4_mark_bitmap_end(EXT4_INODES_PER_GROUP(sb),
sb->s_blocksize * 8, bh->b_data);
err = ext4_handle_dirty_metadata(handle, NULL, bh);
+ brelse(bh);
if (err)
goto out;
- brelse(bh);
}
- bh = NULL;
/* Mark group tables in block bitmap */
for (j = 0; j < GROUP_TABLE_COUNT; j++) {
@@ -658,7 +658,6 @@ static int setup_new_flex_group_blocks(struct super_block *sb,
}
out:
- brelse(bh);
err2 = ext4_journal_stop(handle);
if (err2 && !err)
err = err2;
@@ -845,6 +844,7 @@ static int add_new_gdb(handle_t *handle, struct inode *inode,
err = ext4_handle_dirty_metadata(handle, NULL, gdb_bh);
if (unlikely(err)) {
ext4_std_error(sb, err);
+ iloc.bh = NULL;
goto exit_inode;
}
brelse(dind);
@@ -896,6 +896,7 @@ static int add_new_gdb_meta_bg(struct super_block *sb,
sizeof(struct buffer_head *),
GFP_NOFS);
if (!n_group_desc) {
+ brelse(gdb_bh);
err = -ENOMEM;
ext4_warning(sb, "not enough memory for %lu groups",
gdb_num + 1);
@@ -911,8 +912,6 @@ static int add_new_gdb_meta_bg(struct super_block *sb,
kvfree(o_group_desc);
BUFFER_TRACE(gdb_bh, "get_write_access");
err = ext4_journal_get_write_access(handle, gdb_bh);
- if (unlikely(err))
- brelse(gdb_bh);
return err;
}
@@ -1094,8 +1093,10 @@ static void update_backups(struct super_block *sb, sector_t blk_off, char *data,
backup_block, backup_block -
ext4_group_first_block_no(sb, group));
BUFFER_TRACE(bh, "get_write_access");
- if ((err = ext4_journal_get_write_access(handle, bh)))
+ if ((err = ext4_journal_get_write_access(handle, bh))) {
+ brelse(bh);
break;
+ }
lock_buffer(bh);
memcpy(bh->b_data, data, size);
if (rest)
@@ -1599,7 +1600,7 @@ int ext4_group_add(struct super_block *sb, struct ext4_new_group_data *input)
}
if (reserved_gdb || gdb_off == 0) {
- if (ext4_has_feature_resize_inode(sb) ||
+ if (!ext4_has_feature_resize_inode(sb) ||
!le16_to_cpu(es->s_reserved_gdt_blocks)) {
ext4_warning(sb,
"No reserved GDT blocks, can't resize");
@@ -1903,7 +1904,7 @@ int ext4_resize_fs(struct super_block *sb, ext4_fsblk_t n_blocks_count)
return 0;
n_group = ext4_get_group_number(sb, n_blocks_count - 1);
- if (n_group > (0xFFFFFFFFUL / EXT4_INODES_PER_GROUP(sb))) {
+ if (n_group >= (0xFFFFFFFFUL / EXT4_INODES_PER_GROUP(sb))) {
ext4_warning(sb, "resize would cause inodes_count overflow");
return -EINVAL;
}
@@ -1954,6 +1955,26 @@ int ext4_resize_fs(struct super_block *sb, ext4_fsblk_t n_blocks_count)
}
}
+ /*
+ * Make sure the last group has enough space so that it's
+ * guaranteed to have enough space for all metadata blocks
+ * that it might need to hold. (We might not need to store
+ * the inode table blocks in the last block group, but there
+ * will be cases where this might be needed.)
+ */
+ if ((ext4_group_first_block_no(sb, n_group) +
+ ext4_group_overhead_blocks(sb, n_group) + 2 +
+ sbi->s_itb_per_group + sbi->s_cluster_ratio) >= n_blocks_count) {
+ n_blocks_count = ext4_group_first_block_no(sb, n_group);
+ n_group--;
+ n_blocks_count_retry = 0;
+ if (resize_inode) {
+ iput(resize_inode);
+ resize_inode = NULL;
+ }
+ goto retry;
+ }
+
/* extend the last group */
if (n_group == o_group)
add = n_blocks_count - o_blocks_count;
@@ -1970,7 +1991,7 @@ int ext4_resize_fs(struct super_block *sb, ext4_fsblk_t n_blocks_count)
err = ext4_alloc_flex_bg_array(sb, n_group + 1);
if (err)
- return err;
+ goto out;
err = ext4_mb_alloc_groupinfo(sb, n_group + 1);
if (err)
@@ -2006,6 +2027,10 @@ int ext4_resize_fs(struct super_block *sb, ext4_fsblk_t n_blocks_count)
n_blocks_count_retry = 0;
free_flex_gd(flex_gd);
flex_gd = NULL;
+ if (resize_inode) {
+ iput(resize_inode);
+ resize_inode = NULL;
+ }
goto retry;
}
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index e95b6e1..a22c165 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -66,6 +66,7 @@ static void ext4_mark_recovery_complete(struct super_block *sb,
static void ext4_clear_journal_err(struct super_block *sb,
struct ext4_super_block *es);
static int ext4_sync_fs(struct super_block *sb, int wait);
+static void ext4_umount_end(struct super_block *sb, int flags);
static int ext4_remount(struct super_block *sb, int *flags, char *data);
static int ext4_statfs(struct dentry *dentry, struct kstatfs *buf);
static int ext4_unfreeze(struct super_block *sb);
@@ -1074,6 +1075,16 @@ static struct dentry *ext4_fh_to_parent(struct super_block *sb, struct fid *fid,
ext4_nfs_get_inode);
}
+static int ext4_nfs_commit_metadata(struct inode *inode)
+{
+ struct writeback_control wbc = {
+ .sync_mode = WB_SYNC_ALL
+ };
+
+ trace_ext4_nfs_commit_metadata(inode);
+ return ext4_write_inode(inode, &wbc);
+}
+
/*
* Try to release metadata pages (indirect blocks, directories) which are
* mapped via the block device. Since these pages could have journal heads
@@ -1174,19 +1185,13 @@ static bool ext4_dummy_context(struct inode *inode)
return DUMMY_ENCRYPTION_ENABLED(EXT4_SB(inode->i_sb));
}
-static unsigned ext4_max_namelen(struct inode *inode)
-{
- return S_ISLNK(inode->i_mode) ? inode->i_sb->s_blocksize :
- EXT4_NAME_LEN;
-}
-
static const struct fscrypt_operations ext4_cryptops = {
.key_prefix = "ext4:",
.get_context = ext4_get_context,
.set_context = ext4_set_context,
.dummy_context = ext4_dummy_context,
.empty_dir = ext4_empty_dir,
- .max_namelen = ext4_max_namelen,
+ .max_namelen = EXT4_NAME_LEN,
};
#endif
@@ -1254,6 +1259,7 @@ static const struct super_operations ext4_sops = {
.freeze_fs = ext4_freeze,
.unfreeze_fs = ext4_unfreeze,
.statfs = ext4_statfs,
+ .umount_end = ext4_umount_end,
.remount_fs = ext4_remount,
.show_options = ext4_show_options,
#ifdef CONFIG_QUOTA
@@ -1268,6 +1274,7 @@ static const struct export_operations ext4_export_ops = {
.fh_to_dentry = ext4_fh_to_dentry,
.fh_to_parent = ext4_fh_to_parent,
.get_parent = ext4_get_parent,
+ .commit_metadata = ext4_nfs_commit_metadata,
};
enum {
@@ -2025,6 +2032,8 @@ static int _ext4_show_options(struct seq_file *seq, struct super_block *sb,
SEQ_OPTS_PRINT("max_dir_size_kb=%u", sbi->s_max_dir_size_kb);
if (test_opt(sb, DATA_ERR_ABORT))
SEQ_OPTS_PUTS("data_err=abort");
+ if (DUMMY_ENCRYPTION_ENABLED(sbi))
+ SEQ_OPTS_PUTS("test_dummy_encryption");
ext4_show_quota_options(seq, sb);
return 0;
@@ -2088,7 +2097,7 @@ static int ext4_setup_super(struct super_block *sb, struct ext4_super_block *es,
if (sbi->s_journal)
ext4_set_feature_journal_needs_recovery(sb);
- ext4_commit_super(sb, 1);
+ res = ext4_commit_super(sb, 1);
done:
if (test_opt(sb, DEBUG))
printk(KERN_INFO "[EXT4 FS bs=%lu, gc=%u, "
@@ -2241,6 +2250,7 @@ static int ext4_check_descriptors(struct super_block *sb,
struct ext4_sb_info *sbi = EXT4_SB(sb);
ext4_fsblk_t first_block = le32_to_cpu(sbi->s_es->s_first_data_block);
ext4_fsblk_t last_block;
+ ext4_fsblk_t last_bg_block = sb_block + ext4_bg_num_gdb(sb, 0);
ext4_fsblk_t block_bitmap;
ext4_fsblk_t inode_bitmap;
ext4_fsblk_t inode_table;
@@ -2273,6 +2283,14 @@ static int ext4_check_descriptors(struct super_block *sb,
if (!(sb->s_flags & MS_RDONLY))
return 0;
}
+ if (block_bitmap >= sb_block + 1 &&
+ block_bitmap <= last_bg_block) {
+ ext4_msg(sb, KERN_ERR, "ext4_check_descriptors: "
+ "Block bitmap for group %u overlaps "
+ "block group descriptors", i);
+ if (!(sb->s_flags & MS_RDONLY))
+ return 0;
+ }
if (block_bitmap < first_block || block_bitmap > last_block) {
ext4_msg(sb, KERN_ERR, "ext4_check_descriptors: "
"Block bitmap for group %u not in group "
@@ -2287,6 +2305,14 @@ static int ext4_check_descriptors(struct super_block *sb,
if (!(sb->s_flags & MS_RDONLY))
return 0;
}
+ if (inode_bitmap >= sb_block + 1 &&
+ inode_bitmap <= last_bg_block) {
+ ext4_msg(sb, KERN_ERR, "ext4_check_descriptors: "
+ "Inode bitmap for group %u overlaps "
+ "block group descriptors", i);
+ if (!(sb->s_flags & MS_RDONLY))
+ return 0;
+ }
if (inode_bitmap < first_block || inode_bitmap > last_block) {
ext4_msg(sb, KERN_ERR, "ext4_check_descriptors: "
"Inode bitmap for group %u not in group "
@@ -2301,6 +2327,14 @@ static int ext4_check_descriptors(struct super_block *sb,
if (!(sb->s_flags & MS_RDONLY))
return 0;
}
+ if (inode_table >= sb_block + 1 &&
+ inode_table <= last_bg_block) {
+ ext4_msg(sb, KERN_ERR, "ext4_check_descriptors: "
+ "Inode table for group %u overlaps "
+ "block group descriptors", i);
+ if (!(sb->s_flags & MS_RDONLY))
+ return 0;
+ }
if (inode_table < first_block ||
inode_table + sbi->s_itb_per_group - 1 > last_block) {
ext4_msg(sb, KERN_ERR, "ext4_check_descriptors: "
@@ -3008,13 +3042,22 @@ static ext4_group_t ext4_has_uninit_itable(struct super_block *sb)
ext4_group_t group, ngroups = EXT4_SB(sb)->s_groups_count;
struct ext4_group_desc *gdp = NULL;
+ if (!ext4_has_group_desc_csum(sb))
+ return ngroups;
+
for (group = 0; group < ngroups; group++) {
gdp = ext4_get_group_desc(sb, group, NULL);
if (!gdp)
continue;
- if (!(gdp->bg_flags & cpu_to_le16(EXT4_BG_INODE_ZEROED)))
+ if (gdp->bg_flags & cpu_to_le16(EXT4_BG_INODE_ZEROED))
+ continue;
+ if (group != 0)
break;
+ ext4_error(sb, "Inode table for bg 0 marked as "
+ "needing zeroing");
+ if (sb->s_flags & MS_RDONLY)
+ return ngroups;
}
return group;
@@ -3632,6 +3675,13 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
le32_to_cpu(es->s_log_block_size));
goto failed_mount;
}
+ if (le32_to_cpu(es->s_log_cluster_size) >
+ (EXT4_MAX_CLUSTER_LOG_SIZE - EXT4_MIN_BLOCK_LOG_SIZE)) {
+ ext4_msg(sb, KERN_ERR,
+ "Invalid log cluster size: %u",
+ le32_to_cpu(es->s_log_cluster_size));
+ goto failed_mount;
+ }
if (le16_to_cpu(sbi->s_es->s_reserved_gdt_blocks) > (blocksize / 4)) {
ext4_msg(sb, KERN_ERR,
@@ -3689,6 +3739,11 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
} else {
sbi->s_inode_size = le16_to_cpu(es->s_inode_size);
sbi->s_first_ino = le32_to_cpu(es->s_first_ino);
+ if (sbi->s_first_ino < EXT4_GOOD_OLD_FIRST_INO) {
+ ext4_msg(sb, KERN_ERR, "invalid first ino: %u",
+ sbi->s_first_ino);
+ goto failed_mount;
+ }
if ((sbi->s_inode_size < EXT4_GOOD_OLD_INODE_SIZE) ||
(!is_power_of_2(sbi->s_inode_size)) ||
(sbi->s_inode_size > blocksize)) {
@@ -3765,13 +3820,6 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
"block size (%d)", clustersize, blocksize);
goto failed_mount;
}
- if (le32_to_cpu(es->s_log_cluster_size) >
- (EXT4_MAX_CLUSTER_LOG_SIZE - EXT4_MIN_BLOCK_LOG_SIZE)) {
- ext4_msg(sb, KERN_ERR,
- "Invalid log cluster size: %u",
- le32_to_cpu(es->s_log_cluster_size));
- goto failed_mount;
- }
sbi->s_cluster_bits = le32_to_cpu(es->s_log_cluster_size) -
le32_to_cpu(es->s_log_block_size);
sbi->s_clusters_per_group =
@@ -3792,10 +3840,10 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
}
} else {
if (clustersize != blocksize) {
- ext4_warning(sb, "fragment/cluster size (%d) != "
- "block size (%d)", clustersize,
- blocksize);
- clustersize = blocksize;
+ ext4_msg(sb, KERN_ERR,
+ "fragment/cluster size (%d) != "
+ "block size (%d)", clustersize, blocksize);
+ goto failed_mount;
}
if (sbi->s_blocks_per_group > blocksize * 8) {
ext4_msg(sb, KERN_ERR,
@@ -3849,6 +3897,13 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
ext4_blocks_count(es));
goto failed_mount;
}
+ if ((es->s_first_data_block == 0) && (es->s_log_block_size == 0) &&
+ (sbi->s_cluster_ratio == 1)) {
+ ext4_msg(sb, KERN_WARNING, "bad geometry: first data "
+ "block is 0 with a 1k block and cluster size");
+ goto failed_mount;
+ }
+
blocks_count = (ext4_blocks_count(es) -
le32_to_cpu(es->s_first_data_block) +
EXT4_BLOCKS_PER_GROUP(sb) - 1);
@@ -3865,6 +3920,14 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
sbi->s_groups_count = blocks_count;
sbi->s_blockfile_groups = min_t(ext4_group_t, sbi->s_groups_count,
(EXT4_MAX_BLOCK_FILE_PHYS / EXT4_BLOCKS_PER_GROUP(sb)));
+ if (((u64)sbi->s_groups_count * sbi->s_inodes_per_group) !=
+ le32_to_cpu(es->s_inodes_count)) {
+ ext4_msg(sb, KERN_ERR, "inodes count not valid: %u vs %llu",
+ le32_to_cpu(es->s_inodes_count),
+ ((u64)sbi->s_groups_count * sbi->s_inodes_per_group));
+ ret = -EINVAL;
+ goto failed_mount;
+ }
db_count = (sbi->s_groups_count + EXT4_DESC_PER_BLOCK(sb) - 1) /
EXT4_DESC_PER_BLOCK(sb);
if (ext4_has_feature_meta_bg(sb)) {
@@ -3897,13 +3960,13 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
goto failed_mount2;
}
}
+ sbi->s_gdb_count = db_count;
if (!ext4_check_descriptors(sb, logical_sb_block, &first_not_zeroed)) {
ext4_msg(sb, KERN_ERR, "group descriptors corrupted!");
ret = -EFSCORRUPTED;
goto failed_mount2;
}
- sbi->s_gdb_count = db_count;
get_random_bytes(&sbi->s_next_generation, sizeof(u32));
spin_lock_init(&sbi->s_next_gen_lock);
@@ -4151,11 +4214,13 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
block = ext4_count_free_clusters(sb);
ext4_free_blocks_count_set(sbi->s_es,
EXT4_C2B(sbi, block));
+ ext4_superblock_csum_set(sb);
err = percpu_counter_init(&sbi->s_freeclusters_counter, block,
GFP_KERNEL);
if (!err) {
unsigned long freei = ext4_count_free_inodes(sb);
sbi->s_es->s_free_inodes_count = cpu_to_le32(freei);
+ ext4_superblock_csum_set(sb);
err = percpu_counter_init(&sbi->s_freeinodes_counter, freei,
GFP_KERNEL);
}
@@ -4260,6 +4325,7 @@ static int ext4_fill_super(struct super_block *sb, void *data, int silent)
percpu_counter_destroy(&sbi->s_freeinodes_counter);
percpu_counter_destroy(&sbi->s_dirs_counter);
percpu_counter_destroy(&sbi->s_dirtyclusters_counter);
+ percpu_free_rwsem(&sbi->s_journal_flag_rwsem);
failed_mount5:
ext4_ext_release(sb);
ext4_release_system_zone(sb);
@@ -4582,6 +4648,14 @@ static int ext4_commit_super(struct super_block *sb, int sync)
if (!sbh || block_device_ejected(sb))
return error;
+
+ /*
+ * The superblock bh should be mapped, but it might not be if the
+ * device was hot-removed. Not much we can do but fail the I/O.
+ */
+ if (!buffer_mapped(sbh))
+ return error;
+
/*
* If the file system is mounted read-only, don't update the
* superblock write time. This avoids updating the superblock
@@ -4633,11 +4707,10 @@ static int ext4_commit_super(struct super_block *sb, int sync)
unlock_buffer(sbh);
error = __sync_dirty_buffer(sbh,
test_opt(sb, BARRIER) ? WRITE_FUA : WRITE_SYNC);
- if (error)
+ if (error && !buffer_write_io_error(sbh))
return error;
- error = buffer_write_io_error(sbh);
- if (error) {
+ if (buffer_write_io_error(sbh)) {
ext4_msg(sb, KERN_ERR, "I/O error while writing "
"superblock");
clear_buffer_write_io_error(sbh);
@@ -4847,6 +4920,25 @@ struct ext4_mount_options {
#endif
};
+static void ext4_umount_end(struct super_block *sb, int flags)
+{
+ /*
+ * this is called at the end of umount(2). If there is an unclosed
+ * namespace, ext4 won't do put_super() which triggers fsck in the
+ * next boot.
+ */
+ if ((flags & MNT_FORCE) || atomic_read(&sb->s_active) > 1) {
+ ext4_msg(sb, KERN_ERR,
+ "errors=remount-ro for active namespaces on umount %x",
+ flags);
+ clear_opt(sb, ERRORS_PANIC);
+ set_opt(sb, ERRORS_RO);
+ /* to write the latest s_kbytes_written */
+ if (!(sb->s_flags & MS_RDONLY))
+ ext4_commit_super(sb, 1);
+ }
+}
+
static int ext4_remount(struct super_block *sb, int *flags, char *data)
{
struct ext4_super_block *es;
@@ -5023,8 +5115,11 @@ static int ext4_remount(struct super_block *sb, int *flags, char *data)
if (sbi->s_journal)
ext4_clear_journal_err(sb, es);
sbi->s_mount_state = le16_to_cpu(es->s_state);
- if (!ext4_setup_super(sb, es, 0))
- sb->s_flags &= ~MS_RDONLY;
+ err = ext4_setup_super(sb, es, 0);
+ if (err)
+ goto restore_opts;
+
+ sb->s_flags &= ~MS_RDONLY;
if (ext4_has_feature_mmp(sb))
if (ext4_multi_mount_protect(sb,
le64_to_cpu(es->s_mmp_block))) {
@@ -5048,8 +5143,10 @@ static int ext4_remount(struct super_block *sb, int *flags, char *data)
}
ext4_setup_system_zone(sb);
- if (sbi->s_journal == NULL && !(old_sb_flags & MS_RDONLY))
- ext4_commit_super(sb, 1);
+ if (sbi->s_journal == NULL && !(old_sb_flags & MS_RDONLY)) {
+ err = ext4_commit_super(sb, 1);
+ goto restore_opts;
+ }
#ifdef CONFIG_QUOTA
/* Release old quota file names */
@@ -5371,9 +5468,9 @@ static int ext4_quota_enable(struct super_block *sb, int type, int format_id,
qf_inode->i_flags |= S_NOQUOTA;
lockdep_set_quota_inode(qf_inode, I_DATA_SEM_QUOTA);
err = dquot_enable(qf_inode, type, format_id, flags);
- iput(qf_inode);
if (err)
lockdep_set_quota_inode(qf_inode, I_DATA_SEM_NORMAL);
+ iput(qf_inode);
return err;
}
diff --git a/fs/ext4/sysfs.c b/fs/ext4/sysfs.c
index 5dc655e..54942d6 100644
--- a/fs/ext4/sysfs.c
+++ b/fs/ext4/sysfs.c
@@ -277,8 +277,12 @@ static ssize_t ext4_attr_show(struct kobject *kobj,
case attr_pointer_ui:
if (!ptr)
return 0;
- return snprintf(buf, PAGE_SIZE, "%u\n",
- *((unsigned int *) ptr));
+ if (a->attr_ptr == ptr_ext4_super_block_offset)
+ return snprintf(buf, PAGE_SIZE, "%u\n",
+ le32_to_cpup(ptr));
+ else
+ return snprintf(buf, PAGE_SIZE, "%u\n",
+ *((unsigned int *) ptr));
case attr_pointer_atomic:
if (!ptr)
return 0;
@@ -311,7 +315,10 @@ static ssize_t ext4_attr_store(struct kobject *kobj,
ret = kstrtoul(skip_spaces(buf), 0, &t);
if (ret)
return ret;
- *((unsigned int *) ptr) = t;
+ if (a->attr_ptr == ptr_ext4_super_block_offset)
+ *((__le32 *) ptr) = cpu_to_le32(t);
+ else
+ *((unsigned int *) ptr) = t;
return len;
case attr_inode_readahead:
return inode_readahead_blks_store(a, sbi, buf, len);
diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c
index 3fadfab..ec9beaa6 100644
--- a/fs/ext4/xattr.c
+++ b/fs/ext4/xattr.c
@@ -184,6 +184,8 @@ ext4_xattr_check_names(struct ext4_xattr_entry *entry, void *end,
struct ext4_xattr_entry *next = EXT4_XATTR_NEXT(e);
if ((void *)next >= end)
return -EFSCORRUPTED;
+ if (strnlen(e->e_name, e->e_name_len) != e->e_name_len)
+ return -EFSCORRUPTED;
e = next;
}
@@ -207,12 +209,12 @@ ext4_xattr_check_block(struct inode *inode, struct buffer_head *bh)
{
int error;
- if (buffer_verified(bh))
- return 0;
-
if (BHDR(bh)->h_magic != cpu_to_le32(EXT4_XATTR_MAGIC) ||
BHDR(bh)->h_blocks != cpu_to_le32(1))
return -EFSCORRUPTED;
+ if (buffer_verified(bh))
+ return 0;
+
if (!ext4_xattr_block_csum_verify(inode, bh))
return -EFSBADCRC;
error = ext4_xattr_check_names(BFIRST(bh), bh->b_data + bh->b_size,
@@ -643,14 +645,20 @@ static size_t ext4_xattr_free_space(struct ext4_xattr_entry *last,
}
static int
-ext4_xattr_set_entry(struct ext4_xattr_info *i, struct ext4_xattr_search *s)
+ext4_xattr_set_entry(struct ext4_xattr_info *i, struct ext4_xattr_search *s,
+ struct inode *inode)
{
- struct ext4_xattr_entry *last;
+ struct ext4_xattr_entry *last, *next;
size_t free, min_offs = s->end - s->base, name_len = strlen(i->name);
/* Compute min_offs and last. */
last = s->first;
- for (; !IS_LAST_ENTRY(last); last = EXT4_XATTR_NEXT(last)) {
+ for (; !IS_LAST_ENTRY(last); last = next) {
+ next = EXT4_XATTR_NEXT(last);
+ if ((void *)next >= s->end) {
+ EXT4_ERROR_INODE(inode, "corrupted xattr entries");
+ return -EFSCORRUPTED;
+ }
if (last->e_value_size) {
size_t offs = le16_to_cpu(last->e_value_offs);
if (offs < min_offs)
@@ -832,7 +840,7 @@ ext4_xattr_block_set(handle_t *handle, struct inode *inode,
mb_cache_entry_delete_block(ext4_mb_cache, hash,
bs->bh->b_blocknr);
ea_bdebug(bs->bh, "modifying in-place");
- error = ext4_xattr_set_entry(i, s);
+ error = ext4_xattr_set_entry(i, s, inode);
if (!error) {
if (!IS_LAST_ENTRY(s->first))
ext4_xattr_rehash(header(s->base),
@@ -879,7 +887,7 @@ ext4_xattr_block_set(handle_t *handle, struct inode *inode,
s->end = s->base + sb->s_blocksize;
}
- error = ext4_xattr_set_entry(i, s);
+ error = ext4_xattr_set_entry(i, s, inode);
if (error == -EFSCORRUPTED)
goto bad_block;
if (error)
@@ -1077,23 +1085,9 @@ int ext4_xattr_ibody_inline_set(handle_t *handle, struct inode *inode,
if (EXT4_I(inode)->i_extra_isize == 0)
return -ENOSPC;
- error = ext4_xattr_set_entry(i, s);
- if (error) {
- if (error == -ENOSPC &&
- ext4_has_inline_data(inode)) {
- error = ext4_try_to_evict_inline_data(handle, inode,
- EXT4_XATTR_LEN(strlen(i->name) +
- EXT4_XATTR_SIZE(i->value_len)));
- if (error)
- return error;
- error = ext4_xattr_ibody_find(inode, i, is);
- if (error)
- return error;
- error = ext4_xattr_set_entry(i, s);
- }
- if (error)
- return error;
- }
+ error = ext4_xattr_set_entry(i, s, inode);
+ if (error)
+ return error;
header = IHDR(inode, ext4_raw_inode(&is->iloc));
if (!IS_LAST_ENTRY(s->first)) {
header->h_magic = cpu_to_le32(EXT4_XATTR_MAGIC);
@@ -1115,7 +1109,7 @@ static int ext4_xattr_ibody_set(handle_t *handle, struct inode *inode,
if (EXT4_I(inode)->i_extra_isize == 0)
return -ENOSPC;
- error = ext4_xattr_set_entry(i, s);
+ error = ext4_xattr_set_entry(i, s, inode);
if (error)
return error;
header = IHDR(inode, ext4_raw_inode(&is->iloc));
@@ -1227,6 +1221,8 @@ ext4_xattr_set_handle(handle_t *handle, struct inode *inode, int name_index,
error = ext4_xattr_block_set(handle, inode, &i, &bs);
} else if (error == -ENOSPC) {
if (EXT4_I(inode)->i_file_acl && !bs.s.base) {
+ brelse(bs.bh);
+ bs.bh = NULL;
error = ext4_xattr_block_find(inode, &i, &bs);
if (error)
goto cleanup;
@@ -1397,6 +1393,8 @@ static int ext4_xattr_move_to_block(handle_t *handle, struct inode *inode,
kfree(buffer);
if (is)
brelse(is->iloc.bh);
+ if (bs)
+ brelse(bs->bh);
kfree(is);
kfree(bs);
@@ -1424,6 +1422,11 @@ static int ext4_xattr_make_inode_space(handle_t *handle, struct inode *inode,
last = IFIRST(header);
/* Find the entry best suited to be pushed into EA block */
for (; !IS_LAST_ENTRY(last); last = EXT4_XATTR_NEXT(last)) {
+ /* never move system.data out of the inode */
+ if ((last->e_name_len == 4) &&
+ (last->e_name_index == EXT4_XATTR_INDEX_SYSTEM) &&
+ !memcmp(last->e_name, "data", 4))
+ continue;
total_size =
EXT4_XATTR_SIZE(le32_to_cpu(last->e_value_size)) +
EXT4_XATTR_LEN(last->e_name_len);
@@ -1496,7 +1499,7 @@ int ext4_expand_extra_isize_ea(struct inode *inode, int new_extra_isize,
base = IFIRST(header);
end = (void *)raw_inode + EXT4_SB(inode->i_sb)->s_inode_size;
min_offs = end - base;
- total_ino = sizeof(struct ext4_xattr_ibody_header);
+ total_ino = sizeof(struct ext4_xattr_ibody_header) + sizeof(u32);
error = xattr_check_inode(inode, header, end);
if (error)
diff --git a/fs/f2fs/Kconfig b/fs/f2fs/Kconfig
index 378c221..35a9838 100644
--- a/fs/f2fs/Kconfig
+++ b/fs/f2fs/Kconfig
@@ -85,6 +85,16 @@
efficient since it avoids caching the encrypted and
decrypted pages in the page cache.
+config F2FS_FS_VERITY
+ bool "F2FS Verity"
+ depends on F2FS_FS
+ select FS_VERITY
+ help
+ Enable file-based authentication of f2fs files. This
+ efficiently validates the randomly-accessed content of files
+ using an authenticated dictionary structure hidden at the
+ end of immutable files.
+
config F2FS_IO_TRACE
bool "F2FS IO tracer"
depends on F2FS_FS
diff --git a/fs/f2fs/acl.c b/fs/f2fs/acl.c
index 2bb7c9f..b32efb5 100644
--- a/fs/f2fs/acl.c
+++ b/fs/f2fs/acl.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/acl.c
*
@@ -7,10 +8,6 @@
* Portions of this code from linux/fs/ext2/acl.c
*
* Copyright (C) 2001-2003 Andreas Gruenbacher, <agruen@suse.de>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/f2fs_fs.h>
#include "f2fs.h"
@@ -53,6 +50,9 @@ static struct posix_acl *f2fs_acl_from_disk(const char *value, size_t size)
struct f2fs_acl_entry *entry = (struct f2fs_acl_entry *)(hdr + 1);
const char *end = value + size;
+ if (size < sizeof(struct f2fs_acl_header))
+ return ERR_PTR(-EINVAL);
+
if (hdr->a_version != cpu_to_le32(F2FS_ACL_VERSION))
return ERR_PTR(-EINVAL);
@@ -160,7 +160,7 @@ static void *f2fs_acl_to_disk(struct f2fs_sb_info *sbi,
return (void *)f2fs_acl;
fail:
- kfree(f2fs_acl);
+ kvfree(f2fs_acl);
return ERR_PTR(-EINVAL);
}
@@ -190,7 +190,7 @@ static struct posix_acl *__f2fs_get_acl(struct inode *inode, int type,
acl = NULL;
else
acl = ERR_PTR(retval);
- kfree(value);
+ kvfree(value);
return acl;
}
@@ -240,7 +240,7 @@ static int __f2fs_set_acl(struct inode *inode, int type,
error = f2fs_setxattr(inode, name_index, "", value, size, ipage, 0);
- kfree(value);
+ kvfree(value);
if (!error)
set_cached_acl(inode, type, acl);
@@ -352,12 +352,14 @@ static int f2fs_acl_create(struct inode *dir, umode_t *mode,
return PTR_ERR(p);
clone = f2fs_acl_clone(p, GFP_NOFS);
- if (!clone)
- goto no_mem;
+ if (!clone) {
+ ret = -ENOMEM;
+ goto release_acl;
+ }
ret = f2fs_acl_create_masq(clone, mode);
if (ret < 0)
- goto no_mem_clone;
+ goto release_clone;
if (ret == 0)
posix_acl_release(clone);
@@ -371,11 +373,11 @@ static int f2fs_acl_create(struct inode *dir, umode_t *mode,
return 0;
-no_mem_clone:
+release_clone:
posix_acl_release(clone);
-no_mem:
+release_acl:
posix_acl_release(p);
- return -ENOMEM;
+ return ret;
}
int f2fs_init_acl(struct inode *inode, struct inode *dir, struct page *ipage,
@@ -394,12 +396,16 @@ int f2fs_init_acl(struct inode *inode, struct inode *dir, struct page *ipage,
error = __f2fs_set_acl(inode, ACL_TYPE_DEFAULT, default_acl,
ipage);
posix_acl_release(default_acl);
+ } else {
+ inode->i_default_acl = NULL;
}
if (acl) {
if (!error)
error = __f2fs_set_acl(inode, ACL_TYPE_ACCESS, acl,
ipage);
posix_acl_release(acl);
+ } else {
+ inode->i_acl = NULL;
}
return error;
diff --git a/fs/f2fs/acl.h b/fs/f2fs/acl.h
index 2c68518..b96823c 100644
--- a/fs/f2fs/acl.h
+++ b/fs/f2fs/acl.h
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/acl.h
*
@@ -7,10 +8,6 @@
* Portions of this code from linux/fs/ext2/acl.h
*
* Copyright (C) 2001-2003 Andreas Gruenbacher, <agruen@suse.de>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#ifndef __F2FS_ACL_H__
#define __F2FS_ACL_H__
diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
index 91b2d00..d27d3c4 100644
--- a/fs/f2fs/checkpoint.c
+++ b/fs/f2fs/checkpoint.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/checkpoint.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/bio.h>
@@ -24,10 +21,11 @@
#include <trace/events/f2fs.h>
static struct kmem_cache *ino_entry_slab;
-struct kmem_cache *inode_entry_slab;
+struct kmem_cache *f2fs_inode_entry_slab;
void f2fs_stop_checkpoint(struct f2fs_sb_info *sbi, bool end_io)
{
+ f2fs_build_fault_attr(sbi, 0, 0);
set_ckpt_flags(sbi, CP_ERROR_FLAG);
if (!end_io)
f2fs_flush_merged_writes(sbi);
@@ -36,7 +34,7 @@ void f2fs_stop_checkpoint(struct f2fs_sb_info *sbi, bool end_io)
/*
* We guarantee no failure on the returned page.
*/
-struct page *grab_meta_page(struct f2fs_sb_info *sbi, pgoff_t index)
+struct page *f2fs_grab_meta_page(struct f2fs_sb_info *sbi, pgoff_t index)
{
struct address_space *mapping = META_MAPPING(sbi);
struct page *page = NULL;
@@ -46,7 +44,7 @@ struct page *grab_meta_page(struct f2fs_sb_info *sbi, pgoff_t index)
cond_resched();
goto repeat;
}
- f2fs_wait_on_page_writeback(page, META, true);
+ f2fs_wait_on_page_writeback(page, META, true, true);
if (!PageUptodate(page))
SetPageUptodate(page);
return page;
@@ -70,6 +68,7 @@ static struct page *__get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index,
.encrypted_page = NULL,
.is_meta = is_meta,
};
+ int err;
if (unlikely(!is_meta))
fio.op_flags &= ~REQ_META;
@@ -84,9 +83,10 @@ static struct page *__get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index,
fio.page = page;
- if (f2fs_submit_page_bio(&fio)) {
+ err = f2fs_submit_page_bio(&fio);
+ if (err) {
f2fs_put_page(page, 1);
- goto repeat;
+ return ERR_PTR(err);
}
lock_page(page);
@@ -95,29 +95,43 @@ static struct page *__get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index,
goto repeat;
}
- /*
- * if there is any IO error when accessing device, make our filesystem
- * readonly and make sure do not write checkpoint with non-uptodate
- * meta page.
- */
- if (unlikely(!PageUptodate(page)))
- f2fs_stop_checkpoint(sbi, false);
+ if (unlikely(!PageUptodate(page))) {
+ f2fs_put_page(page, 1);
+ return ERR_PTR(-EIO);
+ }
out:
return page;
}
-struct page *get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index)
+struct page *f2fs_get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index)
{
return __get_meta_page(sbi, index, true);
}
+struct page *f2fs_get_meta_page_nofail(struct f2fs_sb_info *sbi, pgoff_t index)
+{
+ struct page *page;
+ int count = 0;
+
+retry:
+ page = __get_meta_page(sbi, index, true);
+ if (IS_ERR(page)) {
+ if (PTR_ERR(page) == -EIO &&
+ ++count <= DEFAULT_RETRY_IO_COUNT)
+ goto retry;
+ f2fs_stop_checkpoint(sbi, false);
+ }
+ return page;
+}
+
/* for POR only */
-struct page *get_tmp_page(struct f2fs_sb_info *sbi, pgoff_t index)
+struct page *f2fs_get_tmp_page(struct f2fs_sb_info *sbi, pgoff_t index)
{
return __get_meta_page(sbi, index, false);
}
-bool is_valid_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr, int type)
+bool f2fs_is_valid_blkaddr(struct f2fs_sb_info *sbi,
+ block_t blkaddr, int type)
{
switch (type) {
case META_NAT:
@@ -137,8 +151,20 @@ bool is_valid_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr, int type)
return false;
break;
case META_POR:
+ case DATA_GENERIC:
if (unlikely(blkaddr >= MAX_BLKADDR(sbi) ||
- blkaddr < MAIN_BLKADDR(sbi)))
+ blkaddr < MAIN_BLKADDR(sbi))) {
+ if (type == DATA_GENERIC) {
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "access invalid blkaddr:%u", blkaddr);
+ WARN_ON(1);
+ }
+ return false;
+ }
+ break;
+ case META_GENERIC:
+ if (unlikely(blkaddr < SEG0_BLKADDR(sbi) ||
+ blkaddr >= MAIN_BLKADDR(sbi)))
return false;
break;
default:
@@ -151,7 +177,7 @@ bool is_valid_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr, int type)
/*
* Readahead CP/NAT/SIT/SSA pages
*/
-int ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages,
+int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages,
int type, bool sync)
{
struct page *page;
@@ -173,7 +199,7 @@ int ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages,
blk_start_plug(&plug);
for (; nrpages-- > 0; blkno++) {
- if (!is_valid_blkaddr(sbi, blkno, type))
+ if (!f2fs_is_valid_blkaddr(sbi, blkno, type))
goto out;
switch (type) {
@@ -217,7 +243,7 @@ int ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages,
return blkno - start;
}
-void ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index)
+void f2fs_ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index)
{
struct page *page;
bool readahead = false;
@@ -228,7 +254,7 @@ void ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index)
f2fs_put_page(page, 0);
if (readahead)
- ra_meta_pages(sbi, index, BIO_MAX_PAGES, META_POR, true);
+ f2fs_ra_meta_pages(sbi, index, BIO_MAX_PAGES, META_POR, true);
}
static int __f2fs_write_meta_page(struct page *page,
@@ -239,22 +265,18 @@ static int __f2fs_write_meta_page(struct page *page,
trace_f2fs_writepage(page, META);
- if (unlikely(f2fs_cp_error(sbi))) {
- dec_page_count(sbi, F2FS_DIRTY_META);
- unlock_page(page);
- return 0;
- }
+ if (unlikely(f2fs_cp_error(sbi)))
+ goto redirty_out;
if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
goto redirty_out;
if (wbc->for_reclaim && page->index < GET_SUM_BLOCK(sbi, 0))
goto redirty_out;
- write_meta_page(sbi, page, io_type);
+ f2fs_do_write_meta_page(sbi, page, io_type);
dec_page_count(sbi, F2FS_DIRTY_META);
if (wbc->for_reclaim)
- f2fs_submit_merged_write_cond(sbi, page->mapping->host,
- 0, page->index, META);
+ f2fs_submit_merged_write_cond(sbi, NULL, page, 0, META);
unlock_page(page);
@@ -294,7 +316,7 @@ static int f2fs_write_meta_pages(struct address_space *mapping,
trace_f2fs_writepages(mapping->host, wbc, META);
diff = nr_pages_to_write(sbi, META, wbc);
- written = sync_meta_pages(sbi, META, wbc->nr_to_write, FS_META_IO);
+ written = f2fs_sync_meta_pages(sbi, META, wbc->nr_to_write, FS_META_IO);
mutex_unlock(&sbi->cp_mutex);
wbc->nr_to_write = max((long)0, wbc->nr_to_write - written - diff);
return 0;
@@ -305,13 +327,14 @@ static int f2fs_write_meta_pages(struct address_space *mapping,
return 0;
}
-long sync_meta_pages(struct f2fs_sb_info *sbi, enum page_type type,
+long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, enum page_type type,
long nr_to_write, enum iostat_type io_type)
{
struct address_space *mapping = META_MAPPING(sbi);
- pgoff_t index = 0, end = ULONG_MAX, prev = ULONG_MAX;
+ pgoff_t index = 0, prev = ULONG_MAX;
struct pagevec pvec;
long nwritten = 0;
+ int nr_pages;
struct writeback_control wbc = {
.for_reclaim = 0,
};
@@ -321,13 +344,9 @@ long sync_meta_pages(struct f2fs_sb_info *sbi, enum page_type type,
blk_start_plug(&plug);
- while (index <= end) {
- int i, nr_pages;
- nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
- PAGECACHE_TAG_DIRTY,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
- if (unlikely(nr_pages == 0))
- break;
+ while ((nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
+ PAGECACHE_TAG_DIRTY))) {
+ int i;
for (i = 0; i < nr_pages; i++) {
struct page *page = pvec.pages[i];
@@ -351,9 +370,8 @@ long sync_meta_pages(struct f2fs_sb_info *sbi, enum page_type type,
goto continue_unlock;
}
- f2fs_wait_on_page_writeback(page, META, true);
+ f2fs_wait_on_page_writeback(page, META, true, true);
- BUG_ON(PageWriteback(page));
if (!clear_page_dirty_for_io(page))
goto continue_unlock;
@@ -458,20 +476,20 @@ static void __remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type)
spin_unlock(&im->ino_lock);
}
-void add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type)
+void f2fs_add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type)
{
/* add new dirty ino entry into list */
__add_ino_entry(sbi, ino, 0, type);
}
-void remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type)
+void f2fs_remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type)
{
/* remove dirty ino entry from list */
__remove_ino_entry(sbi, ino, type);
}
/* mode should be APPEND_INO or UPDATE_INO */
-bool exist_written_data(struct f2fs_sb_info *sbi, nid_t ino, int mode)
+bool f2fs_exist_written_data(struct f2fs_sb_info *sbi, nid_t ino, int mode)
{
struct inode_management *im = &sbi->im[mode];
struct ino_entry *e;
@@ -482,7 +500,7 @@ bool exist_written_data(struct f2fs_sb_info *sbi, nid_t ino, int mode)
return e ? true : false;
}
-void release_ino_entry(struct f2fs_sb_info *sbi, bool all)
+void f2fs_release_ino_entry(struct f2fs_sb_info *sbi, bool all)
{
struct ino_entry *e, *tmp;
int i;
@@ -501,13 +519,13 @@ void release_ino_entry(struct f2fs_sb_info *sbi, bool all)
}
}
-void set_dirty_device(struct f2fs_sb_info *sbi, nid_t ino,
+void f2fs_set_dirty_device(struct f2fs_sb_info *sbi, nid_t ino,
unsigned int devidx, int type)
{
__add_ino_entry(sbi, ino, devidx, type);
}
-bool is_dirty_device(struct f2fs_sb_info *sbi, nid_t ino,
+bool f2fs_is_dirty_device(struct f2fs_sb_info *sbi, nid_t ino,
unsigned int devidx, int type)
{
struct inode_management *im = &sbi->im[type];
@@ -522,20 +540,19 @@ bool is_dirty_device(struct f2fs_sb_info *sbi, nid_t ino,
return is_dirty;
}
-int acquire_orphan_inode(struct f2fs_sb_info *sbi)
+int f2fs_acquire_orphan_inode(struct f2fs_sb_info *sbi)
{
struct inode_management *im = &sbi->im[ORPHAN_INO];
int err = 0;
spin_lock(&im->ino_lock);
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(sbi, FAULT_ORPHAN)) {
spin_unlock(&im->ino_lock);
f2fs_show_injection_info(FAULT_ORPHAN);
return -ENOSPC;
}
-#endif
+
if (unlikely(im->ino_num >= sbi->max_orphans))
err = -ENOSPC;
else
@@ -545,7 +562,7 @@ int acquire_orphan_inode(struct f2fs_sb_info *sbi)
return err;
}
-void release_orphan_inode(struct f2fs_sb_info *sbi)
+void f2fs_release_orphan_inode(struct f2fs_sb_info *sbi)
{
struct inode_management *im = &sbi->im[ORPHAN_INO];
@@ -555,14 +572,14 @@ void release_orphan_inode(struct f2fs_sb_info *sbi)
spin_unlock(&im->ino_lock);
}
-void add_orphan_inode(struct inode *inode)
+void f2fs_add_orphan_inode(struct inode *inode)
{
/* add new orphan ino entry into list */
__add_ino_entry(F2FS_I_SB(inode), inode->i_ino, 0, ORPHAN_INO);
- update_inode_page(inode);
+ f2fs_update_inode_page(inode);
}
-void remove_orphan_inode(struct f2fs_sb_info *sbi, nid_t ino)
+void f2fs_remove_orphan_inode(struct f2fs_sb_info *sbi, nid_t ino)
{
/* remove orphan entry from orphan list */
__remove_ino_entry(sbi, ino, ORPHAN_INO);
@@ -572,12 +589,7 @@ static int recover_orphan_inode(struct f2fs_sb_info *sbi, nid_t ino)
{
struct inode *inode;
struct node_info ni;
- int err = acquire_orphan_inode(sbi);
-
- if (err)
- goto err_out;
-
- __add_ino_entry(sbi, ino, 0, ORPHAN_INO);
+ int err;
inode = f2fs_iget_retry(sbi->sb, ino);
if (IS_ERR(inode)) {
@@ -590,23 +602,25 @@ static int recover_orphan_inode(struct f2fs_sb_info *sbi, nid_t ino)
}
err = dquot_initialize(inode);
- if (err)
+ if (err) {
+ iput(inode);
goto err_out;
+ }
- dquot_initialize(inode);
clear_nlink(inode);
/* truncate all the data during iput */
iput(inode);
- get_node_info(sbi, ino, &ni);
+ err = f2fs_get_node_info(sbi, ino, &ni);
+ if (err)
+ goto err_out;
/* ENOMEM was fully retried in f2fs_evict_inode. */
if (ni.blk_addr != NULL_ADDR) {
err = -EIO;
goto err_out;
}
- __remove_ino_entry(sbi, ino, ORPHAN_INO);
return 0;
err_out:
@@ -617,7 +631,7 @@ static int recover_orphan_inode(struct f2fs_sb_info *sbi, nid_t ino)
return err;
}
-int recover_orphan_inodes(struct f2fs_sb_info *sbi)
+int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi)
{
block_t start_blk, orphan_blocks, i, j;
unsigned int s_flags = sbi->sb->s_flags;
@@ -638,19 +652,28 @@ int recover_orphan_inodes(struct f2fs_sb_info *sbi)
/* Needed for iput() to work correctly and not trash data */
sbi->sb->s_flags |= MS_ACTIVE;
- /* Turn on quotas so that they are updated correctly */
+ /*
+ * Turn on quotas which were not enabled for read-only mounts if
+ * filesystem has quota feature, so that they are updated correctly.
+ */
quota_enabled = f2fs_enable_quota_files(sbi, s_flags & MS_RDONLY);
#endif
start_blk = __start_cp_addr(sbi) + 1 + __cp_payload(sbi);
orphan_blocks = __start_sum_addr(sbi) - 1 - __cp_payload(sbi);
- ra_meta_pages(sbi, start_blk, orphan_blocks, META_CP, true);
+ f2fs_ra_meta_pages(sbi, start_blk, orphan_blocks, META_CP, true);
for (i = 0; i < orphan_blocks; i++) {
- struct page *page = get_meta_page(sbi, start_blk + i);
+ struct page *page;
struct f2fs_orphan_block *orphan_blk;
+ page = f2fs_get_meta_page(sbi, start_blk + i);
+ if (IS_ERR(page)) {
+ err = PTR_ERR(page);
+ goto out;
+ }
+
orphan_blk = (struct f2fs_orphan_block *)page_address(page);
for (j = 0; j < le32_to_cpu(orphan_blk->entry_count); j++) {
nid_t ino = le32_to_cpu(orphan_blk->ino[j]);
@@ -665,6 +688,8 @@ int recover_orphan_inodes(struct f2fs_sb_info *sbi)
/* clear Orphan Flag */
clear_ckpt_flags(sbi, CP_ORPHAN_PRESENT_FLAG);
out:
+ set_sbi_flag(sbi, SBI_IS_RECOVERED);
+
#ifdef CONFIG_QUOTA
/* Turn quotas off */
if (quota_enabled)
@@ -698,7 +723,7 @@ static void write_orphan_inodes(struct f2fs_sb_info *sbi, block_t start_blk)
/* loop for each orphan inode entry and write them in Jornal block */
list_for_each_entry(orphan, head, list) {
if (!page) {
- page = grab_meta_page(sbi, start_blk++);
+ page = f2fs_grab_meta_page(sbi, start_blk++);
orphan_blk =
(struct f2fs_orphan_block *)page_address(page);
memset(orphan_blk, 0, sizeof(*orphan_blk));
@@ -740,11 +765,15 @@ static int get_checkpoint_version(struct f2fs_sb_info *sbi, block_t cp_addr,
size_t crc_offset = 0;
__u32 crc = 0;
- *cp_page = get_meta_page(sbi, cp_addr);
+ *cp_page = f2fs_get_meta_page(sbi, cp_addr);
+ if (IS_ERR(*cp_page))
+ return PTR_ERR(*cp_page);
+
*cp_block = (struct f2fs_checkpoint *)page_address(*cp_page);
crc_offset = le32_to_cpu((*cp_block)->checksum_offset);
if (crc_offset > (blk_size - sizeof(__le32))) {
+ f2fs_put_page(*cp_page, 1);
f2fs_msg(sbi->sb, KERN_WARNING,
"invalid crc_offset: %zu", crc_offset);
return -EINVAL;
@@ -752,6 +781,7 @@ static int get_checkpoint_version(struct f2fs_sb_info *sbi, block_t cp_addr,
crc = cur_cp_crc(*cp_block);
if (!f2fs_crc_valid(sbi, crc, *cp_block, crc_offset)) {
+ f2fs_put_page(*cp_page, 1);
f2fs_msg(sbi->sb, KERN_WARNING, "invalid crc value");
return -EINVAL;
}
@@ -771,14 +801,22 @@ static struct page *validate_checkpoint(struct f2fs_sb_info *sbi,
err = get_checkpoint_version(sbi, cp_addr, &cp_block,
&cp_page_1, version);
if (err)
- goto invalid_cp1;
+ return NULL;
+
+ if (le32_to_cpu(cp_block->cp_pack_total_block_count) >
+ sbi->blocks_per_seg) {
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "invalid cp_pack_total_block_count:%u",
+ le32_to_cpu(cp_block->cp_pack_total_block_count));
+ goto invalid_cp;
+ }
pre_version = *version;
cp_addr += le32_to_cpu(cp_block->cp_pack_total_block_count) - 1;
err = get_checkpoint_version(sbi, cp_addr, &cp_block,
&cp_page_2, version);
if (err)
- goto invalid_cp2;
+ goto invalid_cp;
cur_version = *version;
if (cur_version == pre_version) {
@@ -786,14 +824,13 @@ static struct page *validate_checkpoint(struct f2fs_sb_info *sbi,
f2fs_put_page(cp_page_2, 1);
return cp_page_1;
}
-invalid_cp2:
f2fs_put_page(cp_page_2, 1);
-invalid_cp1:
+invalid_cp:
f2fs_put_page(cp_page_1, 1);
return NULL;
}
-int get_valid_checkpoint(struct f2fs_sb_info *sbi)
+int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi)
{
struct f2fs_checkpoint *cp_block;
struct f2fs_super_block *fsb = sbi->raw_super;
@@ -805,7 +842,8 @@ int get_valid_checkpoint(struct f2fs_sb_info *sbi)
block_t cp_blk_no;
int i;
- sbi->ckpt = f2fs_kzalloc(sbi, cp_blks * blk_size, GFP_KERNEL);
+ sbi->ckpt = f2fs_kzalloc(sbi, array_size(blk_size, cp_blks),
+ GFP_KERNEL);
if (!sbi->ckpt)
return -ENOMEM;
/*
@@ -836,15 +874,15 @@ int get_valid_checkpoint(struct f2fs_sb_info *sbi)
cp_block = (struct f2fs_checkpoint *)page_address(cur_page);
memcpy(sbi->ckpt, cp_block, blk_size);
- /* Sanity checking of checkpoint */
- if (sanity_check_ckpt(sbi))
- goto free_fail_no_cp;
-
if (cur_page == cp1)
sbi->cur_cp_pack = 1;
else
sbi->cur_cp_pack = 2;
+ /* Sanity checking of checkpoint */
+ if (f2fs_sanity_check_ckpt(sbi))
+ goto free_fail_no_cp;
+
if (cp_blks <= 1)
goto done;
@@ -856,7 +894,9 @@ int get_valid_checkpoint(struct f2fs_sb_info *sbi)
void *sit_bitmap_ptr;
unsigned char *ckpt = (unsigned char *)sbi->ckpt;
- cur_page = get_meta_page(sbi, cp_blk_no + i);
+ cur_page = f2fs_get_meta_page(sbi, cp_blk_no + i);
+ if (IS_ERR(cur_page))
+ goto free_fail_no_cp;
sit_bitmap_ptr = page_address(cur_page);
memcpy(ckpt + i * blk_size, sit_bitmap_ptr, blk_size);
f2fs_put_page(cur_page, 1);
@@ -870,7 +910,7 @@ int get_valid_checkpoint(struct f2fs_sb_info *sbi)
f2fs_put_page(cp1, 1);
f2fs_put_page(cp2, 1);
fail_no_cp:
- kfree(sbi->ckpt);
+ kvfree(sbi->ckpt);
return -EINVAL;
}
@@ -901,7 +941,7 @@ static void __remove_dirty_inode(struct inode *inode, enum inode_type type)
stat_dec_dirty_inode(F2FS_I_SB(inode), type);
}
-void update_dirty_page(struct inode *inode, struct page *page)
+void f2fs_update_dirty_page(struct inode *inode, struct page *page)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
enum inode_type type = S_ISDIR(inode->i_mode) ? DIR_INODE : FILE_INODE;
@@ -920,7 +960,7 @@ void update_dirty_page(struct inode *inode, struct page *page)
f2fs_trace_pid(page);
}
-void remove_dirty_inode(struct inode *inode)
+void f2fs_remove_dirty_inode(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
enum inode_type type = S_ISDIR(inode->i_mode) ? DIR_INODE : FILE_INODE;
@@ -937,7 +977,7 @@ void remove_dirty_inode(struct inode *inode)
spin_unlock(&sbi->inode_lock[type]);
}
-int sync_dirty_inodes(struct f2fs_sb_info *sbi, enum inode_type type)
+int f2fs_sync_dirty_inodes(struct f2fs_sb_info *sbi, enum inode_type type)
{
struct list_head *head;
struct inode *inode;
@@ -978,12 +1018,10 @@ int sync_dirty_inodes(struct f2fs_sb_info *sbi, enum inode_type type)
iput(inode);
/* We need to give cpu to another writers. */
- if (ino == cur_ino) {
- congestion_wait(BLK_RW_ASYNC, HZ/50);
+ if (ino == cur_ino)
cond_resched();
- } else {
+ else
ino = cur_ino;
- }
} else {
/*
* We should submit bio, since it exists several
@@ -1020,7 +1058,7 @@ int f2fs_sync_inode_meta(struct f2fs_sb_info *sbi)
/* it's on eviction */
if (is_inode_flag_set(inode, FI_DIRTY_INODE))
- update_inode_page(inode);
+ f2fs_update_inode_page(inode);
iput(inode);
}
}
@@ -1040,6 +1078,21 @@ static void __prepare_cp_block(struct f2fs_sb_info *sbi)
ckpt->next_free_nid = cpu_to_le32(last_nid);
}
+static bool __need_flush_quota(struct f2fs_sb_info *sbi)
+{
+ if (!is_journalled_quota(sbi))
+ return false;
+ if (is_sbi_flag_set(sbi, SBI_QUOTA_SKIP_FLUSH))
+ return false;
+ if (is_sbi_flag_set(sbi, SBI_QUOTA_NEED_REPAIR))
+ return false;
+ if (is_sbi_flag_set(sbi, SBI_QUOTA_NEED_FLUSH))
+ return true;
+ if (get_pages(sbi, F2FS_DIRTY_QDATA))
+ return true;
+ return false;
+}
+
/*
* Freeze all the FS-operations for checkpoint.
*/
@@ -1051,20 +1104,44 @@ static int block_operations(struct f2fs_sb_info *sbi)
.for_reclaim = 0,
};
struct blk_plug plug;
- int err = 0;
+ int err = 0, cnt = 0;
blk_start_plug(&plug);
-retry_flush_dents:
+retry_flush_quotas:
+ if (__need_flush_quota(sbi)) {
+ int locked;
+
+ if (++cnt > DEFAULT_RETRY_QUOTA_FLUSH_COUNT) {
+ set_sbi_flag(sbi, SBI_QUOTA_SKIP_FLUSH);
+ f2fs_lock_all(sbi);
+ goto retry_flush_dents;
+ }
+ clear_sbi_flag(sbi, SBI_QUOTA_NEED_FLUSH);
+
+ /* only failed during mount/umount/freeze/quotactl */
+ locked = down_read_trylock(&sbi->sb->s_umount);
+ f2fs_quota_sync(sbi->sb, -1);
+ if (locked)
+ up_read(&sbi->sb->s_umount);
+ }
+
f2fs_lock_all(sbi);
+ if (__need_flush_quota(sbi)) {
+ f2fs_unlock_all(sbi);
+ cond_resched();
+ goto retry_flush_quotas;
+ }
+
+retry_flush_dents:
/* write all the dirty dentry pages */
if (get_pages(sbi, F2FS_DIRTY_DENTS)) {
f2fs_unlock_all(sbi);
- err = sync_dirty_inodes(sbi, DIR_INODE);
+ err = f2fs_sync_dirty_inodes(sbi, DIR_INODE);
if (err)
goto out;
cond_resched();
- goto retry_flush_dents;
+ goto retry_flush_quotas;
}
/*
@@ -1073,6 +1150,12 @@ static int block_operations(struct f2fs_sb_info *sbi)
*/
down_write(&sbi->node_change);
+ if (__need_flush_quota(sbi)) {
+ up_write(&sbi->node_change);
+ f2fs_unlock_all(sbi);
+ goto retry_flush_quotas;
+ }
+
if (get_pages(sbi, F2FS_DIRTY_IMETA)) {
up_write(&sbi->node_change);
f2fs_unlock_all(sbi);
@@ -1080,7 +1163,7 @@ static int block_operations(struct f2fs_sb_info *sbi)
if (err)
goto out;
cond_resched();
- goto retry_flush_dents;
+ goto retry_flush_quotas;
}
retry_flush_nodes:
@@ -1088,7 +1171,9 @@ static int block_operations(struct f2fs_sb_info *sbi)
if (get_pages(sbi, F2FS_DIRTY_NODES)) {
up_write(&sbi->node_write);
- err = sync_node_pages(sbi, &wbc, false, FS_CP_NODE_IO);
+ atomic_inc(&sbi->wb_sync_req[NODE]);
+ err = f2fs_sync_node_pages(sbi, &wbc, false, FS_CP_NODE_IO);
+ atomic_dec(&sbi->wb_sync_req[NODE]);
if (err) {
up_write(&sbi->node_change);
f2fs_unlock_all(sbi);
@@ -1115,7 +1200,7 @@ static void unblock_operations(struct f2fs_sb_info *sbi)
f2fs_unlock_all(sbi);
}
-static void wait_on_all_pages_writeback(struct f2fs_sb_info *sbi)
+void f2fs_wait_on_all_pages_writeback(struct f2fs_sb_info *sbi)
{
DEFINE_WAIT(wait);
@@ -1125,6 +1210,9 @@ static void wait_on_all_pages_writeback(struct f2fs_sb_info *sbi)
if (!get_pages(sbi, F2FS_WB_CP_DATA))
break;
+ if (unlikely(f2fs_cp_error(sbi)))
+ break;
+
io_schedule_timeout(5*HZ);
}
finish_wait(&sbi->cp_wait, &wait);
@@ -1166,6 +1254,19 @@ static void update_ckpt_flags(struct f2fs_sb_info *sbi, struct cp_control *cpc)
if (is_sbi_flag_set(sbi, SBI_NEED_FSCK))
__set_ckpt_flags(ckpt, CP_FSCK_FLAG);
+ if (is_sbi_flag_set(sbi, SBI_CP_DISABLED))
+ __set_ckpt_flags(ckpt, CP_DISABLED_FLAG);
+ else
+ __clear_ckpt_flags(ckpt, CP_DISABLED_FLAG);
+
+ if (is_sbi_flag_set(sbi, SBI_QUOTA_SKIP_FLUSH))
+ __set_ckpt_flags(ckpt, CP_QUOTA_NEED_FSCK_FLAG);
+ else
+ __clear_ckpt_flags(ckpt, CP_QUOTA_NEED_FSCK_FLAG);
+
+ if (is_sbi_flag_set(sbi, SBI_QUOTA_NEED_REPAIR))
+ __set_ckpt_flags(ckpt, CP_QUOTA_NEED_FSCK_FLAG);
+
/* set this flag to activate crc|cp_ver for recovery */
__set_ckpt_flags(ckpt, CP_CRC_RECOVERY_FLAG);
__clear_ckpt_flags(ckpt, CP_NOCRC_RECOVERY_FLAG);
@@ -1182,24 +1283,28 @@ static void commit_checkpoint(struct f2fs_sb_info *sbi,
/*
* pagevec_lookup_tag and lock_page again will take
- * some extra time. Therefore, update_meta_pages and
- * sync_meta_pages are combined in this function.
+ * some extra time. Therefore, f2fs_update_meta_pages and
+ * f2fs_sync_meta_pages are combined in this function.
*/
- struct page *page = grab_meta_page(sbi, blk_addr);
+ struct page *page = f2fs_grab_meta_page(sbi, blk_addr);
int err;
- memcpy(page_address(page), src, PAGE_SIZE);
- set_page_dirty(page);
+ f2fs_wait_on_page_writeback(page, META, true, true);
- f2fs_wait_on_page_writeback(page, META, true);
- f2fs_bug_on(sbi, PageWriteback(page));
+ memcpy(page_address(page), src, PAGE_SIZE);
+
+ set_page_dirty(page);
if (unlikely(!clear_page_dirty_for_io(page)))
f2fs_bug_on(sbi, 1);
/* writeout cp pack 2 page */
err = __f2fs_write_meta_page(page, &wbc, FS_CP_META_IO);
- f2fs_bug_on(sbi, err);
+ if (unlikely(err && f2fs_cp_error(sbi))) {
+ f2fs_put_page(page, 1);
+ return;
+ }
+ f2fs_bug_on(sbi, err);
f2fs_put_page(page, 0);
/* submit checkpoint (with barrier if NOBARRIER is not set) */
@@ -1222,17 +1327,15 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
int err;
/* Flush all the NAT/SIT pages */
- while (get_pages(sbi, F2FS_DIRTY_META)) {
- sync_meta_pages(sbi, META, LONG_MAX, FS_CP_META_IO);
- if (unlikely(f2fs_cp_error(sbi)))
- return -EIO;
- }
+ f2fs_sync_meta_pages(sbi, META, LONG_MAX, FS_CP_META_IO);
+ f2fs_bug_on(sbi, get_pages(sbi, F2FS_DIRTY_META) &&
+ !f2fs_cp_error(sbi));
/*
* modify checkpoint
* version number is already updated
*/
- ckpt->elapsed_time = cpu_to_le64(get_mtime(sbi));
+ ckpt->elapsed_time = cpu_to_le64(get_mtime(sbi, true));
ckpt->free_segment_count = cpu_to_le32(free_segments(sbi));
for (i = 0; i < NR_CURSEG_NODE_TYPE; i++) {
ckpt->cur_node_segno[i] =
@@ -1252,7 +1355,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
}
/* 2 cp + n data seg summary + orphan inode blocks */
- data_sum_blocks = npages_for_summary_flush(sbi, false);
+ data_sum_blocks = f2fs_npages_for_summary_flush(sbi, false);
spin_lock_irqsave(&sbi->cp_lock, flags);
if (data_sum_blocks < NR_CURSEG_DATA_TYPE)
__set_ckpt_flags(ckpt, CP_COMPACT_SUM_FLAG);
@@ -1297,22 +1400,15 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
blk = start_blk + sbi->blocks_per_seg - nm_i->nat_bits_blocks;
for (i = 0; i < nm_i->nat_bits_blocks; i++)
- update_meta_page(sbi, nm_i->nat_bits +
+ f2fs_update_meta_page(sbi, nm_i->nat_bits +
(i << F2FS_BLKSIZE_BITS), blk + i);
-
- /* Flush all the NAT BITS pages */
- while (get_pages(sbi, F2FS_DIRTY_META)) {
- sync_meta_pages(sbi, META, LONG_MAX, FS_CP_META_IO);
- if (unlikely(f2fs_cp_error(sbi)))
- return -EIO;
- }
}
/* write out checkpoint buffer at block 0 */
- update_meta_page(sbi, ckpt, start_blk++);
+ f2fs_update_meta_page(sbi, ckpt, start_blk++);
for (i = 1; i < 1 + cp_payload_blks; i++)
- update_meta_page(sbi, (char *)ckpt + i * F2FS_BLKSIZE,
+ f2fs_update_meta_page(sbi, (char *)ckpt + i * F2FS_BLKSIZE,
start_blk++);
if (orphan_num) {
@@ -1320,7 +1416,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
start_blk += orphan_blocks;
}
- write_data_summaries(sbi, start_blk);
+ f2fs_write_data_summaries(sbi, start_blk);
start_blk += data_sum_blocks;
/* Record write statistics in the hot node summary */
@@ -1331,7 +1427,7 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
seg_i->journal->info.kbytes_written = cpu_to_le64(kbytes_written);
if (__remain_node_summaries(cpc->reason)) {
- write_node_summaries(sbi, start_blk);
+ f2fs_write_node_summaries(sbi, start_blk);
start_blk += NR_CURSEG_NODE_TYPE;
}
@@ -1340,13 +1436,12 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
percpu_counter_set(&sbi->alloc_valid_block_count, 0);
/* Here, we have one bio having CP pack except cp pack 2 page */
- sync_meta_pages(sbi, META, LONG_MAX, FS_CP_META_IO);
+ f2fs_sync_meta_pages(sbi, META, LONG_MAX, FS_CP_META_IO);
+ f2fs_bug_on(sbi, get_pages(sbi, F2FS_DIRTY_META) &&
+ !f2fs_cp_error(sbi));
/* wait for previous submitted meta pages writeback */
- wait_on_all_pages_writeback(sbi);
-
- if (unlikely(f2fs_cp_error(sbi)))
- return -EIO;
+ f2fs_wait_on_all_pages_writeback(sbi);
/* flush all device cache */
err = f2fs_flush_device_cache(sbi);
@@ -1355,15 +1450,24 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
/* barrier and flush checkpoint cp pack 2 page if it can */
commit_checkpoint(sbi, ckpt, start_blk);
- wait_on_all_pages_writeback(sbi);
+ f2fs_wait_on_all_pages_writeback(sbi);
- release_ino_entry(sbi, false);
+ /*
+ * invalidate intermediate page cache borrowed from meta inode
+ * which are used for migration of encrypted inode's blocks.
+ */
+ if (f2fs_sb_has_encrypt(sbi))
+ invalidate_mapping_pages(META_MAPPING(sbi),
+ MAIN_BLKADDR(sbi), MAX_BLKADDR(sbi) - 1);
- if (unlikely(f2fs_cp_error(sbi)))
- return -EIO;
+ f2fs_release_ino_entry(sbi, false);
+
+ f2fs_reset_fsync_node_info(sbi);
clear_sbi_flag(sbi, SBI_IS_DIRTY);
clear_sbi_flag(sbi, SBI_NEED_CP);
+ clear_sbi_flag(sbi, SBI_QUOTA_SKIP_FLUSH);
+ sbi->unusable_block_count = 0;
__set_cp_next_pack(sbi);
/*
@@ -1376,18 +1480,24 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
f2fs_bug_on(sbi, get_pages(sbi, F2FS_DIRTY_DENTS));
- return 0;
+ return unlikely(f2fs_cp_error(sbi)) ? -EIO : 0;
}
/*
* We guarantee that this checkpoint procedure will not fail.
*/
-int write_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
+int f2fs_write_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
{
struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi);
unsigned long long ckpt_ver;
int err = 0;
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) {
+ if (cpc->reason != CP_PAUSE)
+ return 0;
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "Start checkpoint disabled!");
+ }
mutex_lock(&sbi->cp_mutex);
if (!is_sbi_flag_set(sbi, SBI_IS_DIRTY) &&
@@ -1415,7 +1525,7 @@ int write_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
/* this is the case of multiple fstrims without any changes */
if (cpc->reason & CP_DISCARD) {
- if (!exist_trim_candidates(sbi, cpc)) {
+ if (!f2fs_exist_trim_candidates(sbi, cpc)) {
unblock_operations(sbi);
goto out;
}
@@ -1423,8 +1533,8 @@ int write_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
if (NM_I(sbi)->dirty_nat_cnt == 0 &&
SIT_I(sbi)->dirty_sentries == 0 &&
prefree_segments(sbi) == 0) {
- flush_sit_entries(sbi, cpc);
- clear_prefree_segments(sbi, cpc);
+ f2fs_flush_sit_entries(sbi, cpc);
+ f2fs_clear_prefree_segments(sbi, cpc);
unblock_operations(sbi);
goto out;
}
@@ -1439,16 +1549,19 @@ int write_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
ckpt->checkpoint_ver = cpu_to_le64(++ckpt_ver);
/* write cached NAT/SIT entries to NAT/SIT area */
- flush_nat_entries(sbi, cpc);
- flush_sit_entries(sbi, cpc);
+ err = f2fs_flush_nat_entries(sbi, cpc);
+ if (err)
+ goto stop;
+
+ f2fs_flush_sit_entries(sbi, cpc);
/* unlock all the fs_lock[] in do_checkpoint() */
err = do_checkpoint(sbi, cpc);
if (err)
- release_discard_addrs(sbi);
+ f2fs_release_discard_addrs(sbi);
else
- clear_prefree_segments(sbi, cpc);
-
+ f2fs_clear_prefree_segments(sbi, cpc);
+stop:
unblock_operations(sbi);
stat_inc_cp_count(sbi->stat_info);
@@ -1464,7 +1577,7 @@ int write_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
return err;
}
-void init_ino_entry_info(struct f2fs_sb_info *sbi)
+void f2fs_init_ino_entry_info(struct f2fs_sb_info *sbi)
{
int i;
@@ -1482,23 +1595,23 @@ void init_ino_entry_info(struct f2fs_sb_info *sbi)
F2FS_ORPHANS_PER_BLOCK;
}
-int __init create_checkpoint_caches(void)
+int __init f2fs_create_checkpoint_caches(void)
{
ino_entry_slab = f2fs_kmem_cache_create("f2fs_ino_entry",
sizeof(struct ino_entry));
if (!ino_entry_slab)
return -ENOMEM;
- inode_entry_slab = f2fs_kmem_cache_create("f2fs_inode_entry",
+ f2fs_inode_entry_slab = f2fs_kmem_cache_create("f2fs_inode_entry",
sizeof(struct inode_entry));
- if (!inode_entry_slab) {
+ if (!f2fs_inode_entry_slab) {
kmem_cache_destroy(ino_entry_slab);
return -ENOMEM;
}
return 0;
}
-void destroy_checkpoint_caches(void)
+void f2fs_destroy_checkpoint_caches(void)
{
kmem_cache_destroy(ino_entry_slab);
- kmem_cache_destroy(inode_entry_slab);
+ kmem_cache_destroy(f2fs_inode_entry_slab);
}
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index ea33d19..0004a54 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/data.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
@@ -48,15 +45,35 @@ static bool __is_cp_guaranteed(struct page *page)
if (inode->i_ino == F2FS_META_INO(sbi) ||
inode->i_ino == F2FS_NODE_INO(sbi) ||
S_ISDIR(inode->i_mode) ||
+ (S_ISREG(inode->i_mode) &&
+ (f2fs_is_atomic_file(inode) || IS_NOQUOTA(inode))) ||
is_cold_data(page))
return true;
return false;
}
+static enum count_type __read_io_type(struct page *page)
+{
+ struct address_space *mapping = page->mapping;
+
+ if (mapping) {
+ struct inode *inode = mapping->host;
+ struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+
+ if (inode->i_ino == F2FS_META_INO(sbi))
+ return F2FS_RD_META;
+
+ if (inode->i_ino == F2FS_NODE_INO(sbi))
+ return F2FS_RD_NODE;
+ }
+ return F2FS_RD_DATA;
+}
+
/* postprocessing steps for read bios */
enum bio_post_read_step {
STEP_INITIAL = 0,
STEP_DECRYPT,
+ STEP_VERITY,
};
struct bio_post_read_ctx {
@@ -78,10 +95,12 @@ static void __read_end_io(struct bio *bio)
/* PG_error was set if any post_read step failed */
if (bio->bi_error || PageError(page)) {
ClearPageUptodate(page);
- SetPageError(page);
+ /* will re-read again later */
+ ClearPageError(page);
} else {
SetPageUptodate(page);
}
+ dec_page_count(F2FS_P_SB(page), __read_io_type(page));
unlock_page(page);
}
if (bio->bi_private)
@@ -101,8 +120,23 @@ static void decrypt_work(struct work_struct *work)
bio_post_read_processing(ctx);
}
+static void verity_work(struct work_struct *work)
+{
+ struct bio_post_read_ctx *ctx =
+ container_of(work, struct bio_post_read_ctx, work);
+
+ fsverity_verify_bio(ctx->bio);
+
+ bio_post_read_processing(ctx);
+}
+
static void bio_post_read_processing(struct bio_post_read_ctx *ctx)
{
+ /*
+ * We use different work queues for decryption and for verity because
+ * verity may require reading metadata pages that need decryption, and
+ * we shouldn't recurse to the same workqueue.
+ */
switch (++ctx->cur_step) {
case STEP_DECRYPT:
if (ctx->enabled_steps & (1 << STEP_DECRYPT)) {
@@ -112,6 +146,14 @@ static void bio_post_read_processing(struct bio_post_read_ctx *ctx)
}
ctx->cur_step++;
/* fall-through */
+ case STEP_VERITY:
+ if (ctx->enabled_steps & (1 << STEP_VERITY)) {
+ INIT_WORK(&ctx->work, verity_work);
+ fsverity_enqueue_verify_work(&ctx->work);
+ return;
+ }
+ ctx->cur_step++;
+ /* fall-through */
default:
__read_end_io(ctx->bio);
}
@@ -124,12 +166,10 @@ static bool f2fs_bio_post_read_required(struct bio *bio)
static void f2fs_read_end_io(struct bio *bio)
{
-#ifdef CONFIG_F2FS_FAULT_INJECTION
- if (time_to_inject(F2FS_P_SB(bio->bi_io_vec->bv_page), FAULT_IO)) {
- f2fs_show_injection_info(FAULT_IO);
+ if (time_to_inject(F2FS_P_SB(bio->bi_io_vec->bv_page), FAULT_READ_IO)) {
+ f2fs_show_injection_info(FAULT_READ_IO);
bio->bi_error = -EIO;
}
-#endif
if (f2fs_bio_post_read_required(bio)) {
struct bio_post_read_ctx *ctx = bio->bi_private;
@@ -148,6 +188,11 @@ static void f2fs_write_end_io(struct bio *bio)
struct bio_vec *bvec;
int i;
+ if (time_to_inject(sbi, FAULT_WRITE_IO)) {
+ f2fs_show_injection_info(FAULT_WRITE_IO);
+ bio->bi_error = -EIO;
+ }
+
bio_for_each_segment_all(bvec, bio, i) {
struct page *page = bvec->bv_page;
enum count_type type = WB_DATA_TYPE(page);
@@ -175,6 +220,8 @@ static void f2fs_write_end_io(struct bio *bio)
page->index != nid_of_node(page));
dec_page_count(sbi, type);
+ if (f2fs_in_warm_node_list(sbi, page))
+ f2fs_del_fsync_node_entry(sbi, page);
clear_cold_data(page);
end_page_writeback(page);
}
@@ -244,7 +291,7 @@ static struct bio *__bio_alloc(struct f2fs_sb_info *sbi, block_t blk_addr,
} else {
bio->bi_end_io = f2fs_write_end_io;
bio->bi_private = sbi;
- bio->bi_write_hint = io_type_to_rw_hint(sbi, type, temp);
+ bio->bi_write_hint = f2fs_io_type_to_rw_hint(sbi, type, temp);
}
if (wbc)
wbc_init_bio(wbc, bio);
@@ -261,7 +308,7 @@ static inline void __submit_bio(struct f2fs_sb_info *sbi,
if (type != DATA && type != NODE)
goto submit_io;
- if (f2fs_sb_has_blkzoned(sbi->sb) && current->plug)
+ if (test_opt(sbi, LFS) && current->plug)
blk_finish_plug(current->plug);
start = bio->bi_iter.bi_size >> F2FS_BLKSIZE_BITS;
@@ -316,8 +363,8 @@ static void __submit_merged_bio(struct f2fs_bio_info *io)
io->bio = NULL;
}
-static bool __has_merged_page(struct f2fs_bio_info *io,
- struct inode *inode, nid_t ino, pgoff_t idx)
+static bool __has_merged_page(struct f2fs_bio_info *io, struct inode *inode,
+ struct page *page, nid_t ino)
{
struct bio_vec *bvec;
struct page *target;
@@ -326,7 +373,7 @@ static bool __has_merged_page(struct f2fs_bio_info *io,
if (!io->bio)
return false;
- if (!inode && !ino)
+ if (!inode && !page && !ino)
return true;
bio_for_each_segment_all(bvec, io->bio, i) {
@@ -336,11 +383,10 @@ static bool __has_merged_page(struct f2fs_bio_info *io,
else
target = fscrypt_control_page(bvec->bv_page);
- if (idx != target->index)
- continue;
-
if (inode && inode == target->mapping->host)
return true;
+ if (page && page == target)
+ return true;
if (ino && ino == ino_of_node(target))
return true;
}
@@ -348,28 +394,6 @@ static bool __has_merged_page(struct f2fs_bio_info *io,
return false;
}
-static bool has_merged_page(struct f2fs_sb_info *sbi, struct inode *inode,
- nid_t ino, pgoff_t idx, enum page_type type)
-{
- enum page_type btype = PAGE_TYPE_OF_BIO(type);
- enum temp_type temp;
- struct f2fs_bio_info *io;
- bool ret = false;
-
- for (temp = HOT; temp < NR_TEMP_TYPE; temp++) {
- io = sbi->write_io[btype] + temp;
-
- down_read(&io->io_rwsem);
- ret = __has_merged_page(io, inode, ino, idx);
- up_read(&io->io_rwsem);
-
- /* TODO: use HOT temp only for meta pages now. */
- if (ret || btype == META)
- break;
- }
- return ret;
-}
-
static void __f2fs_submit_merged_write(struct f2fs_sb_info *sbi,
enum page_type type, enum temp_type temp)
{
@@ -391,17 +415,23 @@ static void __f2fs_submit_merged_write(struct f2fs_sb_info *sbi,
}
static void __submit_merged_write_cond(struct f2fs_sb_info *sbi,
- struct inode *inode, nid_t ino, pgoff_t idx,
- enum page_type type, bool force)
+ struct inode *inode, struct page *page,
+ nid_t ino, enum page_type type, bool force)
{
enum temp_type temp;
-
- if (!force && !has_merged_page(sbi, inode, ino, idx, type))
- return;
+ bool ret = true;
for (temp = HOT; temp < NR_TEMP_TYPE; temp++) {
+ if (!force) {
+ enum page_type btype = PAGE_TYPE_OF_BIO(type);
+ struct f2fs_bio_info *io = sbi->write_io[btype] + temp;
- __f2fs_submit_merged_write(sbi, type, temp);
+ down_read(&io->io_rwsem);
+ ret = __has_merged_page(io, inode, page, ino);
+ up_read(&io->io_rwsem);
+ }
+ if (ret)
+ __f2fs_submit_merged_write(sbi, type, temp);
/* TODO: use HOT temp only for meta pages now. */
if (type >= META)
@@ -415,10 +445,10 @@ void f2fs_submit_merged_write(struct f2fs_sb_info *sbi, enum page_type type)
}
void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi,
- struct inode *inode, nid_t ino, pgoff_t idx,
- enum page_type type)
+ struct inode *inode, struct page *page,
+ nid_t ino, enum page_type type)
{
- __submit_merged_write_cond(sbi, inode, ino, idx, type, false);
+ __submit_merged_write_cond(sbi, inode, page, ino, type, false);
}
void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi)
@@ -439,7 +469,10 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio)
fio->encrypted_page : fio->page;
struct inode *inode = fio->page->mapping->host;
- verify_block_addr(fio, fio->new_blkaddr);
+ if (!f2fs_is_valid_blkaddr(fio->sbi, fio->new_blkaddr,
+ __is_meta_io(fio) ? META_GENERIC : DATA_GENERIC))
+ return -EFAULT;
+
trace_f2fs_submit_page_bio(page, fio);
f2fs_trace_ios(fio, 0);
@@ -448,22 +481,27 @@ int f2fs_submit_page_bio(struct f2fs_io_info *fio)
1, is_read_io(fio->op), fio->type, fio->temp);
if (f2fs_may_encrypt_bio(inode, fio))
- fscrypt_set_ice_dun(inode, bio, PG_DUN(inode, fio->page));
+ fscrypt_set_ice_dun(inode, bio, PG_DUN(inode, fio->page));
+ fscrypt_set_ice_skip(bio, fio->encrypted_page ? 1 : 0);
if (bio_add_page(bio, page, PAGE_SIZE, 0) < PAGE_SIZE) {
bio_put(bio);
return -EFAULT;
}
+
+ if (fio->io_wbc && !is_read_io(fio->op))
+ wbc_account_io(fio->io_wbc, page, PAGE_SIZE);
+
bio_set_op_attrs(bio, fio->op, fio->op_flags);
- __submit_bio(fio->sbi, bio, fio->type);
+ inc_page_count(fio->sbi, is_read_io(fio->op) ?
+ __read_io_type(page): WB_DATA_TYPE(fio->page));
- if (!is_read_io(fio->op))
- inc_page_count(fio->sbi, WB_DATA_TYPE(fio->page));
+ __submit_bio(fio->sbi, bio, fio->type);
return 0;
}
-int f2fs_submit_page_write(struct f2fs_io_info *fio)
+void f2fs_submit_page_write(struct f2fs_io_info *fio)
{
struct f2fs_sb_info *sbi = fio->sbi;
enum page_type btype = PAGE_TYPE_OF_BIO(fio->type);
@@ -471,8 +509,8 @@ int f2fs_submit_page_write(struct f2fs_io_info *fio)
struct page *bio_page;
struct inode *inode;
bool bio_encrypted;
+ int bi_crypt_skip;
u64 dun;
- int err = 0;
f2fs_bug_on(sbi, is_read_io(fio->op));
@@ -482,7 +520,7 @@ int f2fs_submit_page_write(struct f2fs_io_info *fio)
spin_lock(&io->io_lock);
if (list_empty(&io->io_list)) {
spin_unlock(&io->io_lock);
- goto out_fail;
+ goto out;
}
fio = list_first_entry(&io->io_list,
struct f2fs_io_info, list);
@@ -490,13 +528,14 @@ int f2fs_submit_page_write(struct f2fs_io_info *fio)
spin_unlock(&io->io_lock);
}
- if (fio->old_blkaddr != NEW_ADDR)
+ if (__is_valid_data_blkaddr(fio->old_blkaddr))
verify_block_addr(fio, fio->old_blkaddr);
verify_block_addr(fio, fio->new_blkaddr);
bio_page = fio->encrypted_page ? fio->encrypted_page : fio->page;
inode = fio->page->mapping->host;
dun = PG_DUN(inode, fio->page);
+ bi_crypt_skip = fio->encrypted_page ? 1 : 0;
bio_encrypted = f2fs_may_encrypt_bio(inode, fio);
/* set submitted = true as a return value */
@@ -510,23 +549,23 @@ int f2fs_submit_page_write(struct f2fs_io_info *fio)
__submit_merged_bio(io);
/* ICE support */
- if (!fscrypt_mergeable_bio(io->bio, dun, bio_encrypted))
+ if (!fscrypt_mergeable_bio(io->bio, dun, bio_encrypted, bi_crypt_skip))
__submit_merged_bio(io);
alloc_new:
if (io->bio == NULL) {
if ((fio->type == DATA || fio->type == NODE) &&
fio->new_blkaddr & F2FS_IO_SIZE_MASK(sbi)) {
- err = -EAGAIN;
dec_page_count(sbi, WB_DATA_TYPE(bio_page));
- goto out_fail;
+ fio->retry = true;
+ goto skip;
}
io->bio = __bio_alloc(sbi, fio->new_blkaddr, fio->io_wbc,
BIO_MAX_PAGES, false,
fio->type, fio->temp);
if (bio_encrypted)
fscrypt_set_ice_dun(inode, io->bio, dun);
-
+ fscrypt_set_ice_skip(io->bio, bi_crypt_skip);
io->fio = *fio;
}
@@ -542,32 +581,39 @@ int f2fs_submit_page_write(struct f2fs_io_info *fio)
f2fs_trace_ios(fio, 0);
trace_f2fs_submit_page_write(fio->page, fio);
-
+skip:
if (fio->in_list)
goto next;
-out_fail:
+out:
+ if (is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN) ||
+ f2fs_is_checkpoint_ready(sbi))
+ __submit_merged_bio(io);
up_write(&io->io_rwsem);
- return err;
}
static struct bio *f2fs_grab_read_bio(struct inode *inode, block_t blkaddr,
- unsigned nr_pages)
+ unsigned nr_pages, unsigned op_flag, int flags)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct bio *bio;
struct bio_post_read_ctx *ctx;
unsigned int post_read_steps = 0;
+ if (!f2fs_is_valid_blkaddr(sbi, blkaddr, DATA_GENERIC))
+ return ERR_PTR(-EFAULT);
+
bio = f2fs_bio_alloc(sbi, min_t(int, nr_pages, BIO_MAX_PAGES), false);
if (!bio)
return ERR_PTR(-ENOMEM);
f2fs_target_device(sbi, blkaddr, bio);
bio->bi_end_io = f2fs_read_end_io;
- bio_set_op_attrs(bio, REQ_OP_READ, 0);
+ bio_set_op_attrs(bio, REQ_OP_READ, op_flag);
- if (f2fs_encrypted_file(inode) &&
- !fscrypt_using_hardware_encryption(inode))
+ if (f2fs_encrypted_file(inode) &&
+ !fscrypt_using_hardware_encryption(inode))
post_read_steps |= 1 << STEP_DECRYPT;
+ if (f2fs_verity_file(inode) && !(flags & F2FS_GETPAGE_SKIP_VERITY))
+ post_read_steps |= 1 << STEP_VERITY;
if (post_read_steps) {
ctx = mempool_alloc(bio_post_read_ctx_pool, GFP_NOFS);
if (!ctx) {
@@ -577,9 +623,6 @@ static struct bio *f2fs_grab_read_bio(struct inode *inode, block_t blkaddr,
ctx->bio = bio;
ctx->enabled_steps = post_read_steps;
bio->bi_private = ctx;
-
- /* wait the page to be moved by cleaning */
- f2fs_wait_on_block_writeback(sbi, blkaddr);
}
return bio;
@@ -587,9 +630,9 @@ static struct bio *f2fs_grab_read_bio(struct inode *inode, block_t blkaddr,
/* This can handle encryption stuffs */
static int f2fs_submit_page_read(struct inode *inode, struct page *page,
- block_t blkaddr)
+ block_t blkaddr, int flags)
{
- struct bio *bio = f2fs_grab_read_bio(inode, blkaddr, 1);
+ struct bio *bio = f2fs_grab_read_bio(inode, blkaddr, 1, 0, flags);
if (IS_ERR(bio))
return PTR_ERR(bio);
@@ -597,10 +640,15 @@ static int f2fs_submit_page_read(struct inode *inode, struct page *page,
if (f2fs_may_encrypt_bio(inode, NULL))
fscrypt_set_ice_dun(inode, bio, PG_DUN(inode, page));
+ /* wait for GCed page writeback via META_MAPPING */
+ f2fs_wait_on_block_writeback(inode, blkaddr);
+
if (bio_add_page(bio, page, PAGE_SIZE, 0) < PAGE_SIZE) {
bio_put(bio);
return -EFAULT;
}
+ ClearPageError(page);
+ inc_page_count(F2FS_I_SB(inode), F2FS_RD_DATA);
__submit_bio(F2FS_I_SB(inode), bio, DATA);
return 0;
}
@@ -625,9 +673,9 @@ static void __set_data_blkaddr(struct dnode_of_data *dn)
* ->node_page
* update block addresses in the node page
*/
-void set_data_blkaddr(struct dnode_of_data *dn)
+void f2fs_set_data_blkaddr(struct dnode_of_data *dn)
{
- f2fs_wait_on_page_writeback(dn->node_page, NODE, true);
+ f2fs_wait_on_page_writeback(dn->node_page, NODE, true, true);
__set_data_blkaddr(dn);
if (set_page_dirty(dn->node_page))
dn->node_changed = true;
@@ -636,12 +684,12 @@ void set_data_blkaddr(struct dnode_of_data *dn)
void f2fs_update_data_blkaddr(struct dnode_of_data *dn, block_t blkaddr)
{
dn->data_blkaddr = blkaddr;
- set_data_blkaddr(dn);
+ f2fs_set_data_blkaddr(dn);
f2fs_update_extent_cache(dn);
}
/* dn->ofs_in_node will be returned with up-to-date last block pointer */
-int reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count)
+int f2fs_reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode);
int err;
@@ -657,7 +705,7 @@ int reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count)
trace_f2fs_reserve_new_blocks(dn->inode, dn->nid,
dn->ofs_in_node, count);
- f2fs_wait_on_page_writeback(dn->node_page, NODE, true);
+ f2fs_wait_on_page_writeback(dn->node_page, NODE, true, true);
for (; count > 0; dn->ofs_in_node++) {
block_t blkaddr = datablock_addr(dn->inode,
@@ -675,12 +723,12 @@ int reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count)
}
/* Should keep dn->ofs_in_node unchanged */
-int reserve_new_block(struct dnode_of_data *dn)
+int f2fs_reserve_new_block(struct dnode_of_data *dn)
{
unsigned int ofs_in_node = dn->ofs_in_node;
int ret;
- ret = reserve_new_blocks(dn, 1);
+ ret = f2fs_reserve_new_blocks(dn, 1);
dn->ofs_in_node = ofs_in_node;
return ret;
}
@@ -690,12 +738,12 @@ int f2fs_reserve_block(struct dnode_of_data *dn, pgoff_t index)
bool need_put = dn->inode_page ? false : true;
int err;
- err = get_dnode_of_data(dn, index, ALLOC_NODE);
+ err = f2fs_get_dnode_of_data(dn, index, ALLOC_NODE);
if (err)
return err;
if (dn->data_blkaddr == NULL_ADDR)
- err = reserve_new_block(dn);
+ err = f2fs_reserve_new_block(dn);
if (err || need_put)
f2fs_put_dnode(dn);
return err;
@@ -714,8 +762,8 @@ int f2fs_get_block(struct dnode_of_data *dn, pgoff_t index)
return f2fs_reserve_block(dn, index);
}
-struct page *get_read_data_page(struct inode *inode, pgoff_t index,
- int op_flags, bool for_write)
+struct page *f2fs_get_read_data_page(struct inode *inode, pgoff_t index,
+ int op_flags, int flags)
{
struct address_space *mapping = inode->i_mapping;
struct dnode_of_data dn;
@@ -723,7 +771,8 @@ struct page *get_read_data_page(struct inode *inode, pgoff_t index,
struct extent_info ei = {0,0,0};
int err;
- page = f2fs_grab_cache_page(mapping, index, for_write);
+ page = f2fs_grab_cache_page(mapping, index,
+ (flags & F2FS_GETPAGE_FOR_WRITE));
if (!page)
return ERR_PTR(-ENOMEM);
@@ -733,7 +782,7 @@ struct page *get_read_data_page(struct inode *inode, pgoff_t index,
}
set_new_dnode(&dn, inode, NULL, NULL, 0);
- err = get_dnode_of_data(&dn, index, LOOKUP_NODE);
+ err = f2fs_get_dnode_of_data(&dn, index, LOOKUP_NODE);
if (err)
goto put_err;
f2fs_put_dnode(&dn);
@@ -752,7 +801,8 @@ struct page *get_read_data_page(struct inode *inode, pgoff_t index,
* A new dentry page is allocated but not able to be written, since its
* new inode page couldn't be allocated due to -ENOSPC.
* In such the case, its blkaddr can be remained as NEW_ADDR.
- * see, f2fs_add_link -> get_new_data_page -> init_inode_metadata.
+ * see, f2fs_add_link -> f2fs_get_new_data_page ->
+ * f2fs_init_inode_metadata.
*/
if (dn.data_blkaddr == NEW_ADDR) {
zero_user_segment(page, 0, PAGE_SIZE);
@@ -762,7 +812,7 @@ struct page *get_read_data_page(struct inode *inode, pgoff_t index,
return page;
}
- err = f2fs_submit_page_read(inode, page, dn.data_blkaddr);
+ err = f2fs_submit_page_read(inode, page, dn.data_blkaddr, flags);
if (err)
goto put_err;
return page;
@@ -772,7 +822,7 @@ struct page *get_read_data_page(struct inode *inode, pgoff_t index,
return ERR_PTR(err);
}
-struct page *find_data_page(struct inode *inode, pgoff_t index)
+struct page *f2fs_find_data_page(struct inode *inode, pgoff_t index, int flags)
{
struct address_space *mapping = inode->i_mapping;
struct page *page;
@@ -782,7 +832,7 @@ struct page *find_data_page(struct inode *inode, pgoff_t index)
return page;
f2fs_put_page(page, 0);
- page = get_read_data_page(inode, index, 0, false);
+ page = f2fs_get_read_data_page(inode, index, 0, flags);
if (IS_ERR(page))
return page;
@@ -802,13 +852,14 @@ struct page *find_data_page(struct inode *inode, pgoff_t index)
* Because, the callers, functions in dir.c and GC, should be able to know
* whether this page exists or not.
*/
-struct page *get_lock_data_page(struct inode *inode, pgoff_t index,
+struct page *f2fs_get_lock_data_page(struct inode *inode, pgoff_t index,
bool for_write)
{
struct address_space *mapping = inode->i_mapping;
struct page *page;
+ int flags = (for_write ? F2FS_GETPAGE_FOR_WRITE : 0);
repeat:
- page = get_read_data_page(inode, index, 0, for_write);
+ page = f2fs_get_read_data_page(inode, index, 0, flags);
if (IS_ERR(page))
return page;
@@ -834,7 +885,7 @@ struct page *get_lock_data_page(struct inode *inode, pgoff_t index,
* Note that, ipage is set only by make_empty_dir, and if any error occur,
* ipage should be released by this function.
*/
-struct page *get_new_data_page(struct inode *inode,
+struct page *f2fs_get_new_data_page(struct inode *inode,
struct page *ipage, pgoff_t index, bool new_i_size)
{
struct address_space *mapping = inode->i_mapping;
@@ -873,7 +924,7 @@ struct page *get_new_data_page(struct inode *inode,
/* if ipage exists, blkaddr should be NEW_ADDR */
f2fs_bug_on(F2FS_I_SB(inode), ipage);
- page = get_lock_data_page(inode, index, true);
+ page = f2fs_get_lock_data_page(inode, index, true);
if (IS_ERR(page))
return page;
}
@@ -889,35 +940,39 @@ static int __allocate_data_block(struct dnode_of_data *dn, int seg_type)
struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode);
struct f2fs_summary sum;
struct node_info ni;
- pgoff_t fofs;
+ block_t old_blkaddr;
blkcnt_t count = 1;
int err;
if (unlikely(is_inode_flag_set(dn->inode, FI_NO_ALLOC)))
return -EPERM;
+ err = f2fs_get_node_info(sbi, dn->nid, &ni);
+ if (err)
+ return err;
+
dn->data_blkaddr = datablock_addr(dn->inode,
dn->node_page, dn->ofs_in_node);
- if (dn->data_blkaddr == NEW_ADDR)
+ if (dn->data_blkaddr != NULL_ADDR)
goto alloc;
if (unlikely((err = inc_valid_block_count(sbi, dn->inode, &count))))
return err;
alloc:
- get_node_info(sbi, dn->nid, &ni);
set_summary(&sum, dn->nid, dn->ofs_in_node, ni.version);
-
- allocate_data_block(sbi, NULL, dn->data_blkaddr, &dn->data_blkaddr,
+ old_blkaddr = dn->data_blkaddr;
+ f2fs_allocate_data_block(sbi, NULL, old_blkaddr, &dn->data_blkaddr,
&sum, seg_type, NULL, false);
- set_data_blkaddr(dn);
+ if (GET_SEGNO(sbi, old_blkaddr) != NULL_SEGNO)
+ invalidate_mapping_pages(META_MAPPING(sbi),
+ old_blkaddr, old_blkaddr);
+ f2fs_set_data_blkaddr(dn);
- /* update i_size */
- fofs = start_bidx_of_node(ofs_of_node(dn->node_page), dn->inode) +
- dn->ofs_in_node;
- if (i_size_read(dn->inode) < ((loff_t)(fofs + 1) << PAGE_SHIFT))
- f2fs_i_size_write(dn->inode,
- ((loff_t)(fofs + 1) << PAGE_SHIFT));
+ /*
+ * i_size will be updated by direct_IO. Otherwise, we'll get stale
+ * data from unwritten block via dio_read.
+ */
return 0;
}
@@ -936,6 +991,9 @@ int f2fs_preallocate_blocks(struct kiocb *iocb, struct iov_iter *from)
return err;
}
+ if (direct_io && allow_outplace_dio(inode, iocb, from))
+ return 0;
+
if (is_inode_flag_set(inode, FI_NO_PREALLOC))
return 0;
@@ -949,10 +1007,11 @@ int f2fs_preallocate_blocks(struct kiocb *iocb, struct iov_iter *from)
map.m_next_pgofs = NULL;
map.m_next_extent = NULL;
map.m_seg_type = NO_CHECK_TYPE;
+ map.m_may_create = true;
if (direct_io) {
- map.m_seg_type = rw_hint_to_seg_type(iocb->ki_hint);
- flag = f2fs_force_buffered_io(inode, WRITE) ?
+ map.m_seg_type = f2fs_rw_hint_to_seg_type(iocb->ki_hint);
+ flag = f2fs_force_buffered_io(inode, iocb, from) ?
F2FS_GET_BLOCK_PRE_AIO :
F2FS_GET_BLOCK_PRE_DIO;
goto map_blocks;
@@ -977,7 +1036,7 @@ int f2fs_preallocate_blocks(struct kiocb *iocb, struct iov_iter *from)
return err;
}
-static inline void __do_map_lock(struct f2fs_sb_info *sbi, int flag, bool lock)
+void __do_map_lock(struct f2fs_sb_info *sbi, int flag, bool lock)
{
if (flag == F2FS_GET_BLOCK_PRE_AIO) {
if (lock)
@@ -1007,7 +1066,7 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map,
unsigned int maxblocks = map->m_len;
struct dnode_of_data dn;
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
- int mode = create ? ALLOC_NODE : LOOKUP_NODE;
+ int mode = map->m_may_create ? ALLOC_NODE : LOOKUP_NODE;
pgoff_t pgofs, end_offset, end;
int err = 0, ofs = 1;
unsigned int ofs_in_node, last_ofs_in_node;
@@ -1027,21 +1086,30 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map,
end = pgofs + maxblocks;
if (!create && f2fs_lookup_extent_cache(inode, pgofs, &ei)) {
+ if (test_opt(sbi, LFS) && flag == F2FS_GET_BLOCK_DIO &&
+ map->m_may_create)
+ goto next_dnode;
+
map->m_pblk = ei.blk + pgofs - ei.fofs;
map->m_len = min((pgoff_t)maxblocks, ei.fofs + ei.len - pgofs);
map->m_flags = F2FS_MAP_MAPPED;
if (map->m_next_extent)
*map->m_next_extent = pgofs + map->m_len;
+
+ /* for hardware encryption, but to avoid potential issue in future */
+ if (flag == F2FS_GET_BLOCK_DIO)
+ f2fs_wait_on_block_writeback_range(inode,
+ map->m_pblk, map->m_len);
goto out;
}
next_dnode:
- if (create)
+ if (map->m_may_create)
__do_map_lock(sbi, flag, true);
/* When reading holes, we need its node page */
set_new_dnode(&dn, inode, NULL, NULL, 0);
- err = get_dnode_of_data(&dn, pgofs, mode);
+ err = f2fs_get_dnode_of_data(&dn, pgofs, mode);
if (err) {
if (flag == F2FS_GET_BLOCK_BMAP)
map->m_pblk = 0;
@@ -1049,10 +1117,10 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map,
err = 0;
if (map->m_next_pgofs)
*map->m_next_pgofs =
- get_next_page_offset(&dn, pgofs);
+ f2fs_get_next_page_offset(&dn, pgofs);
if (map->m_next_extent)
*map->m_next_extent =
- get_next_page_offset(&dn, pgofs);
+ f2fs_get_next_page_offset(&dn, pgofs);
}
goto unlock_out;
}
@@ -1065,7 +1133,23 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map,
next_block:
blkaddr = datablock_addr(dn.inode, dn.node_page, dn.ofs_in_node);
- if (blkaddr == NEW_ADDR || blkaddr == NULL_ADDR) {
+ if (__is_valid_data_blkaddr(blkaddr) &&
+ !f2fs_is_valid_blkaddr(sbi, blkaddr, DATA_GENERIC)) {
+ err = -EFAULT;
+ goto sync_out;
+ }
+
+ if (is_valid_data_blkaddr(sbi, blkaddr)) {
+ /* use out-place-update for driect IO under LFS mode */
+ if (test_opt(sbi, LFS) && flag == F2FS_GET_BLOCK_DIO &&
+ map->m_may_create) {
+ err = __allocate_data_block(&dn, map->m_seg_type);
+ if (!err) {
+ blkaddr = dn.data_blkaddr;
+ set_inode_flag(inode, FI_APPEND_WRITE);
+ }
+ }
+ } else {
if (create) {
if (unlikely(f2fs_cp_error(sbi))) {
err = -EIO;
@@ -1077,6 +1161,8 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map,
last_ofs_in_node = dn.ofs_in_node;
}
} else {
+ WARN_ON(flag != F2FS_GET_BLOCK_PRE_DIO &&
+ flag != F2FS_GET_BLOCK_DIO);
err = __allocate_data_block(&dn,
map->m_seg_type);
if (!err)
@@ -1138,7 +1224,7 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map,
(pgofs == end || dn.ofs_in_node == end_offset)) {
dn.ofs_in_node = ofs_in_node;
- err = reserve_new_blocks(&dn, prealloc);
+ err = f2fs_reserve_new_blocks(&dn, prealloc);
if (err)
goto sync_out;
@@ -1167,13 +1253,19 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map,
f2fs_put_dnode(&dn);
- if (create) {
+ if (map->m_may_create) {
__do_map_lock(sbi, flag, false);
f2fs_balance_fs(sbi, dn.node_changed);
}
goto next_dnode;
sync_out:
+
+ /* for hardware encryption, but to avoid potential issue in future */
+ if (flag == F2FS_GET_BLOCK_DIO && map->m_flags & F2FS_MAP_MAPPED)
+ f2fs_wait_on_block_writeback_range(inode,
+ map->m_pblk, map->m_len);
+
if (flag == F2FS_GET_BLOCK_PRECACHE) {
if (map->m_flags & F2FS_MAP_MAPPED) {
unsigned int ofs = start_pgofs - map->m_lblk;
@@ -1187,7 +1279,7 @@ int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map,
}
f2fs_put_dnode(&dn);
unlock_out:
- if (create) {
+ if (map->m_may_create) {
__do_map_lock(sbi, flag, false);
f2fs_balance_fs(sbi, dn.node_changed);
}
@@ -1209,6 +1301,7 @@ bool f2fs_overwrite_io(struct inode *inode, loff_t pos, size_t len)
map.m_next_pgofs = NULL;
map.m_next_extent = NULL;
map.m_seg_type = NO_CHECK_TYPE;
+ map.m_may_create = false;
last_lblk = F2FS_BLK_ALIGN(pos + len);
while (map.m_lblk < last_lblk) {
@@ -1223,7 +1316,7 @@ bool f2fs_overwrite_io(struct inode *inode, loff_t pos, size_t len)
static int __get_data_block(struct inode *inode, sector_t iblock,
struct buffer_head *bh, int create, int flag,
- pgoff_t *next_pgofs, int seg_type)
+ pgoff_t *next_pgofs, int seg_type, bool may_write)
{
struct f2fs_map_blocks map;
int err;
@@ -1233,6 +1326,7 @@ static int __get_data_block(struct inode *inode, sector_t iblock,
map.m_next_pgofs = next_pgofs;
map.m_next_extent = NULL;
map.m_seg_type = seg_type;
+ map.m_may_create = may_write;
err = f2fs_map_blocks(inode, &map, create, flag);
if (!err) {
@@ -1249,16 +1343,25 @@ static int get_data_block(struct inode *inode, sector_t iblock,
{
return __get_data_block(inode, iblock, bh_result, create,
flag, next_pgofs,
- NO_CHECK_TYPE);
+ NO_CHECK_TYPE, create);
+}
+
+static int get_data_block_dio_write(struct inode *inode, sector_t iblock,
+ struct buffer_head *bh_result, int create)
+{
+ return __get_data_block(inode, iblock, bh_result, create,
+ F2FS_GET_BLOCK_DIO, NULL,
+ f2fs_rw_hint_to_seg_type(inode->i_write_hint),
+ true);
}
static int get_data_block_dio(struct inode *inode, sector_t iblock,
struct buffer_head *bh_result, int create)
{
return __get_data_block(inode, iblock, bh_result, create,
- F2FS_GET_BLOCK_DEFAULT, NULL,
- rw_hint_to_seg_type(
- inode->i_write_hint));
+ F2FS_GET_BLOCK_DIO, NULL,
+ f2fs_rw_hint_to_seg_type(inode->i_write_hint),
+ false);
}
static int get_data_block_bmap(struct inode *inode, sector_t iblock,
@@ -1270,7 +1373,7 @@ static int get_data_block_bmap(struct inode *inode, sector_t iblock,
return __get_data_block(inode, iblock, bh_result, create,
F2FS_GET_BLOCK_BMAP, NULL,
- NO_CHECK_TYPE);
+ NO_CHECK_TYPE, create);
}
static inline sector_t logical_to_blk(struct inode *inode, loff_t offset)
@@ -1302,7 +1405,11 @@ static int f2fs_xattr_fiemap(struct inode *inode,
if (!page)
return -ENOMEM;
- get_node_info(sbi, inode->i_ino, &ni);
+ err = f2fs_get_node_info(sbi, inode->i_ino, &ni);
+ if (err) {
+ f2fs_put_page(page, 1);
+ return err;
+ }
phys = (__u64)blk_to_logical(inode, ni.blk_addr);
offset = offsetof(struct f2fs_inode, i_addr) +
@@ -1329,7 +1436,11 @@ static int f2fs_xattr_fiemap(struct inode *inode,
if (!page)
return -ENOMEM;
- get_node_info(sbi, xnid, &ni);
+ err = f2fs_get_node_info(sbi, xnid, &ni);
+ if (err) {
+ f2fs_put_page(page, 1);
+ return err;
+ }
phys = (__u64)blk_to_logical(inode, ni.blk_addr);
len = inode->i_sb->s_blocksize;
@@ -1441,10 +1552,15 @@ int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
/*
* This function was originally taken from fs/mpage.c, and customized for f2fs.
* Major change was from block_size == page_size in f2fs by default.
+ *
+ * Note that the aops->readpages() function is ONLY used for read-ahead. If
+ * this function ever deviates from doing just read-ahead, it should either
+ * use ->readpage() or do the necessary surgery to decouple ->readpages()
+ * from read-ahead.
*/
static int f2fs_mpage_readpages(struct address_space *mapping,
struct list_head *pages, struct page *page,
- unsigned nr_pages)
+ unsigned nr_pages, bool is_readahead)
{
struct bio *bio = NULL;
sector_t last_block_in_bio = 0;
@@ -1466,6 +1582,7 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
map.m_next_pgofs = NULL;
map.m_next_extent = NULL;
map.m_seg_type = NO_CHECK_TYPE;
+ map.m_may_create = false;
for (; nr_pages; nr_pages--) {
if (pages) {
@@ -1481,8 +1598,8 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
block_in_file = (sector_t)page->index;
last_block = block_in_file + nr_pages;
- last_block_in_file = (i_size_read(inode) + blocksize - 1) >>
- blkbits;
+ last_block_in_file = (fsverity_full_isize(inode) +
+ blocksize - 1) >> blkbits;
if (last_block > last_block_in_file)
last_block = last_block_in_file;
@@ -1517,8 +1634,15 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
SetPageUptodate(page);
goto confused;
}
+
+ if (!f2fs_is_valid_blkaddr(F2FS_I_SB(inode), block_nr,
+ DATA_GENERIC))
+ goto set_error_page;
} else {
zero_user_segment(page, 0, PAGE_SIZE);
+ if (f2fs_verity_file(inode) &&
+ !fsverity_verify_page(page))
+ goto set_error_page;
if (!PageUptodate(page))
SetPageUptodate(page);
unlock_page(page);
@@ -1538,13 +1662,14 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
dun = PG_DUN(inode, page);
bio_encrypted = f2fs_may_encrypt_bio(inode, NULL);
- if (!fscrypt_mergeable_bio(bio, dun, bio_encrypted)) {
+ if (!fscrypt_mergeable_bio(bio, dun, bio_encrypted, 0)) {
__submit_bio(F2FS_I_SB(inode), bio, DATA);
bio = NULL;
}
if (bio == NULL) {
- bio = f2fs_grab_read_bio(inode, block_nr, nr_pages);
+ bio = f2fs_grab_read_bio(inode, block_nr, nr_pages,
+ is_readahead ? REQ_RAHEAD : 0, 0);
if (IS_ERR(bio)) {
bio = NULL;
goto set_error_page;
@@ -1552,9 +1677,18 @@ static int f2fs_mpage_readpages(struct address_space *mapping,
if (bio_encrypted)
fscrypt_set_ice_dun(inode, bio, dun);
}
+
+ /*
+ * If the page is under writeback, we need to wait for
+ * its completion to see the correct decrypted data.
+ */
+ f2fs_wait_on_block_writeback(inode, block_nr);
+
if (bio_add_page(bio, page, blocksize, 0) < blocksize)
goto submit_and_realloc;
+ inc_page_count(F2FS_I_SB(inode), F2FS_RD_DATA);
+ ClearPageError(page);
last_block_in_bio = block_nr;
goto next_page;
set_error_page:
@@ -1589,7 +1723,7 @@ static int f2fs_read_data_page(struct file *file, struct page *page)
if (f2fs_has_inline_data(inode))
ret = f2fs_read_inline_data(inode, page);
if (ret == -EAGAIN)
- ret = f2fs_mpage_readpages(page->mapping, NULL, page, 1);
+ ret = f2fs_mpage_readpages(page->mapping, NULL, page, 1, false);
return ret;
}
@@ -1606,19 +1740,20 @@ static int f2fs_read_data_pages(struct file *file,
if (f2fs_has_inline_data(inode))
return 0;
- return f2fs_mpage_readpages(mapping, pages, NULL, nr_pages);
+ return f2fs_mpage_readpages(mapping, pages, NULL, nr_pages, true);
}
static int encrypt_one_page(struct f2fs_io_info *fio)
{
struct inode *inode = fio->page->mapping->host;
+ struct page *mpage;
gfp_t gfp_flags = GFP_NOFS;
if (!f2fs_encrypted_file(inode))
return 0;
/* wait for GCed page writeback via META_MAPPING */
- f2fs_wait_on_block_writeback(fio->sbi, fio->old_blkaddr);
+ f2fs_wait_on_block_writeback(inode, fio->old_blkaddr);
retry_encrypt:
if (fscrypt_using_hardware_encryption(inode))
@@ -1626,17 +1761,25 @@ static int encrypt_one_page(struct f2fs_io_info *fio)
fio->encrypted_page = fscrypt_encrypt_page(inode, fio->page,
PAGE_SIZE, 0, fio->page->index, gfp_flags);
- if (!IS_ERR(fio->encrypted_page))
- return 0;
-
- /* flush pending IOs and wait for a while in the ENOMEM case */
- if (PTR_ERR(fio->encrypted_page) == -ENOMEM) {
- f2fs_flush_merged_writes(fio->sbi);
- congestion_wait(BLK_RW_ASYNC, HZ/50);
- gfp_flags |= __GFP_NOFAIL;
- goto retry_encrypt;
+ if (IS_ERR(fio->encrypted_page)) {
+ /* flush pending IOs and wait for a while in the ENOMEM case */
+ if (PTR_ERR(fio->encrypted_page) == -ENOMEM) {
+ f2fs_flush_merged_writes(fio->sbi);
+ congestion_wait(BLK_RW_ASYNC, HZ/50);
+ gfp_flags |= __GFP_NOFAIL;
+ goto retry_encrypt;
+ }
+ return PTR_ERR(fio->encrypted_page);
}
- return PTR_ERR(fio->encrypted_page);
+
+ mpage = find_lock_page(META_MAPPING(fio->sbi), fio->old_blkaddr);
+ if (mpage) {
+ if (PageUptodate(mpage))
+ memcpy(page_address(mpage),
+ page_address(fio->encrypted_page), PAGE_SIZE);
+ f2fs_put_page(mpage, 1);
+ }
+ return 0;
}
static inline bool check_inplace_update_policy(struct inode *inode,
@@ -1647,12 +1790,12 @@ static inline bool check_inplace_update_policy(struct inode *inode,
if (policy & (0x1 << F2FS_IPU_FORCE))
return true;
- if (policy & (0x1 << F2FS_IPU_SSR) && need_SSR(sbi))
+ if (policy & (0x1 << F2FS_IPU_SSR) && f2fs_need_SSR(sbi))
return true;
if (policy & (0x1 << F2FS_IPU_UTIL) &&
utilization(sbi) > SM_I(sbi)->min_ipu_util)
return true;
- if (policy & (0x1 << F2FS_IPU_SSR_UTIL) && need_SSR(sbi) &&
+ if (policy & (0x1 << F2FS_IPU_SSR_UTIL) && f2fs_need_SSR(sbi) &&
utilization(sbi) > SM_I(sbi)->min_ipu_util)
return true;
@@ -1670,10 +1813,14 @@ static inline bool check_inplace_update_policy(struct inode *inode,
is_inode_flag_set(inode, FI_NEED_IPU))
return true;
+ if (unlikely(fio && is_sbi_flag_set(sbi, SBI_CP_DISABLED) &&
+ !f2fs_is_checkpointed_data(sbi, fio->old_blkaddr)))
+ return true;
+
return false;
}
-bool should_update_inplace(struct inode *inode, struct f2fs_io_info *fio)
+bool f2fs_should_update_inplace(struct inode *inode, struct f2fs_io_info *fio)
{
if (f2fs_is_pinned_file(inode))
return true;
@@ -1685,7 +1832,7 @@ bool should_update_inplace(struct inode *inode, struct f2fs_io_info *fio)
return check_inplace_update_policy(inode, fio);
}
-bool should_update_outplace(struct inode *inode, struct f2fs_io_info *fio)
+bool f2fs_should_update_outplace(struct inode *inode, struct f2fs_io_info *fio)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
@@ -1693,6 +1840,8 @@ bool should_update_outplace(struct inode *inode, struct f2fs_io_info *fio)
return true;
if (S_ISDIR(inode->i_mode))
return true;
+ if (IS_NOQUOTA(inode))
+ return true;
if (f2fs_is_atomic_file(inode))
return true;
if (fio) {
@@ -1700,6 +1849,9 @@ bool should_update_outplace(struct inode *inode, struct f2fs_io_info *fio)
return true;
if (IS_ATOMIC_WRITTEN_PAGE(fio->page))
return true;
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED) &&
+ f2fs_is_checkpointed_data(sbi, fio->old_blkaddr)))
+ return true;
}
return false;
}
@@ -1708,27 +1860,19 @@ static inline bool need_inplace_update(struct f2fs_io_info *fio)
{
struct inode *inode = fio->page->mapping->host;
- if (should_update_outplace(inode, fio))
+ if (f2fs_should_update_outplace(inode, fio))
return false;
- return should_update_inplace(inode, fio);
+ return f2fs_should_update_inplace(inode, fio);
}
-static inline bool valid_ipu_blkaddr(struct f2fs_io_info *fio)
-{
- if (fio->old_blkaddr == NEW_ADDR)
- return false;
- if (fio->old_blkaddr == NULL_ADDR)
- return false;
- return true;
-}
-
-int do_write_data_page(struct f2fs_io_info *fio)
+int f2fs_do_write_data_page(struct f2fs_io_info *fio)
{
struct page *page = fio->page;
struct inode *inode = page->mapping->host;
struct dnode_of_data dn;
struct extent_info ei = {0,0,0};
+ struct node_info ni;
bool ipu_force = false;
int err = 0;
@@ -1737,18 +1881,20 @@ int do_write_data_page(struct f2fs_io_info *fio)
f2fs_lookup_extent_cache(inode, page->index, &ei)) {
fio->old_blkaddr = ei.blk + page->index - ei.fofs;
- if (valid_ipu_blkaddr(fio)) {
- ipu_force = true;
- fio->need_lock = LOCK_DONE;
- goto got_it;
- }
+ if (!f2fs_is_valid_blkaddr(fio->sbi, fio->old_blkaddr,
+ DATA_GENERIC))
+ return -EFAULT;
+
+ ipu_force = true;
+ fio->need_lock = LOCK_DONE;
+ goto got_it;
}
/* Deadlock due to between page->lock and f2fs_lock_op */
if (fio->need_lock == LOCK_REQ && !f2fs_trylock_op(fio->sbi))
return -EAGAIN;
- err = get_dnode_of_data(&dn, page->index, LOOKUP_NODE);
+ err = f2fs_get_dnode_of_data(&dn, page->index, LOOKUP_NODE);
if (err)
goto out;
@@ -1757,14 +1903,22 @@ int do_write_data_page(struct f2fs_io_info *fio)
/* This page is already truncated */
if (fio->old_blkaddr == NULL_ADDR) {
ClearPageUptodate(page);
+ clear_cold_data(page);
goto out_writepage;
}
got_it:
+ if (__is_valid_data_blkaddr(fio->old_blkaddr) &&
+ !f2fs_is_valid_blkaddr(fio->sbi, fio->old_blkaddr,
+ DATA_GENERIC)) {
+ err = -EFAULT;
+ goto out_writepage;
+ }
/*
* If current allocation needs SSR,
* it had better in-place writes for updated data.
*/
- if (ipu_force || (valid_ipu_blkaddr(fio) && need_inplace_update(fio))) {
+ if (ipu_force || (is_valid_data_blkaddr(fio->sbi, fio->old_blkaddr) &&
+ need_inplace_update(fio))) {
err = encrypt_one_page(fio);
if (err)
goto out_writepage;
@@ -1774,7 +1928,9 @@ int do_write_data_page(struct f2fs_io_info *fio)
f2fs_put_dnode(&dn);
if (fio->need_lock == LOCK_REQ)
f2fs_unlock_op(fio->sbi);
- err = rewrite_data_page(fio);
+ err = f2fs_inplace_write_data(fio);
+ if (err && PageWriteback(page))
+ end_page_writeback(page);
trace_f2fs_do_write_data_page(fio->page, IPU);
set_inode_flag(inode, FI_UPDATE_WRITE);
return err;
@@ -1788,6 +1944,12 @@ int do_write_data_page(struct f2fs_io_info *fio)
fio->need_lock = LOCK_REQ;
}
+ err = f2fs_get_node_info(fio->sbi, dn.nid, &ni);
+ if (err)
+ goto out_writepage;
+
+ fio->version = ni.version;
+
err = encrypt_one_page(fio);
if (err)
goto out_writepage;
@@ -1796,7 +1958,7 @@ int do_write_data_page(struct f2fs_io_info *fio)
ClearPageError(page);
/* LFS mode write path */
- write_data_page(&dn, fio);
+ f2fs_outplace_write_data(&dn, fio);
trace_f2fs_do_write_data_page(page, OPU);
set_inode_flag(inode, FI_APPEND_WRITE);
if (page->index == 0)
@@ -1842,6 +2004,12 @@ static int __write_data_page(struct page *page, bool *submitted,
/* we should bypass data pages to proceed the kworkder jobs */
if (unlikely(f2fs_cp_error(sbi))) {
mapping_set_error(page->mapping, -EIO);
+ /*
+ * don't drop any dirty dentry pages for keeping lastest
+ * directory structure.
+ */
+ if (S_ISDIR(inode->i_mode))
+ goto redirty_out;
goto out;
}
@@ -1866,13 +2034,13 @@ static int __write_data_page(struct page *page, bool *submitted,
/* we should not write 0'th page having journal header */
if (f2fs_is_volatile_file(inode) && (!page->index ||
(!wbc->for_reclaim &&
- available_free_memory(sbi, BASE_CHECK))))
+ f2fs_available_free_memory(sbi, BASE_CHECK))))
goto redirty_out;
/* Dentry blocks are controlled by checkpoint */
if (S_ISDIR(inode->i_mode)) {
fio.need_lock = LOCK_DONE;
- err = do_write_data_page(&fio);
+ err = f2fs_do_write_data_page(&fio);
goto done;
}
@@ -1891,10 +2059,10 @@ static int __write_data_page(struct page *page, bool *submitted,
}
if (err == -EAGAIN) {
- err = do_write_data_page(&fio);
+ err = f2fs_do_write_data_page(&fio);
if (err == -EAGAIN) {
fio.need_lock = LOCK_REQ;
- err = do_write_data_page(&fio);
+ err = f2fs_do_write_data_page(&fio);
}
}
@@ -1913,18 +2081,20 @@ static int __write_data_page(struct page *page, bool *submitted,
out:
inode_dec_dirty_pages(inode);
- if (err)
+ if (err) {
ClearPageUptodate(page);
+ clear_cold_data(page);
+ }
if (wbc->for_reclaim) {
- f2fs_submit_merged_write_cond(sbi, inode, 0, page->index, DATA);
+ f2fs_submit_merged_write_cond(sbi, NULL, page, 0, DATA);
clear_inode_flag(inode, FI_HOT_DATA);
- remove_dirty_inode(inode);
+ f2fs_remove_dirty_inode(inode);
submitted = NULL;
}
unlock_page(page);
- if (!S_ISDIR(inode->i_mode))
+ if (!S_ISDIR(inode->i_mode) && !IS_NOQUOTA(inode))
f2fs_balance_fs(sbi, need_balance_fs);
if (unlikely(f2fs_cp_error(sbi))) {
@@ -1939,7 +2109,13 @@ static int __write_data_page(struct page *page, bool *submitted,
redirty_out:
redirty_page_for_writepage(wbc, page);
- if (!err)
+ /*
+ * pageout() in MM traslates EAGAIN, so calls handle_write_error()
+ * -> mapping_set_error() -> set_bit(AS_EIO, ...).
+ * file_write_and_wait_range() will see EIO error, which is critical
+ * to return value of fsync() followed by atomic_write failure to user.
+ */
+ if (!err || wbc->for_reclaim)
return AOP_WRITEPAGE_ACTIVATE;
unlock_page(page);
return err;
@@ -1963,15 +2139,16 @@ static int f2fs_write_cache_pages(struct address_space *mapping,
int ret = 0;
int done = 0;
struct pagevec pvec;
+ struct f2fs_sb_info *sbi = F2FS_M_SB(mapping);
int nr_pages;
pgoff_t uninitialized_var(writeback_index);
pgoff_t index;
pgoff_t end; /* Inclusive */
pgoff_t done_index;
- pgoff_t last_idx = ULONG_MAX;
int cycled;
int range_whole = 0;
int tag;
+ int nwritten = 0;
pagevec_init(&pvec, 0);
@@ -2007,8 +2184,8 @@ static int f2fs_write_cache_pages(struct address_space *mapping,
while (!done && (index <= end)) {
int i;
- nr_pages = pagevec_lookup_tag(&pvec, mapping, &index, tag,
- min(end - index, (pgoff_t)PAGEVEC_SIZE - 1) + 1);
+ nr_pages = pagevec_lookup_range_tag(&pvec, mapping, &index, end,
+ tag);
if (nr_pages == 0)
break;
@@ -2016,7 +2193,9 @@ static int f2fs_write_cache_pages(struct address_space *mapping,
struct page *page = pvec.pages[i];
bool submitted = false;
- if (page->index > end) {
+ /* give a priority to WB_SYNC threads */
+ if (atomic_read(&sbi->wb_sync_req[DATA]) &&
+ wbc->sync_mode == WB_SYNC_NONE) {
done = 1;
break;
}
@@ -2039,12 +2218,11 @@ static int f2fs_write_cache_pages(struct address_space *mapping,
if (PageWriteback(page)) {
if (wbc->sync_mode != WB_SYNC_NONE)
f2fs_wait_on_page_writeback(page,
- DATA, true);
+ DATA, true, true);
else
goto continue_unlock;
}
- BUG_ON(PageWriteback(page));
if (!clear_page_dirty_for_io(page))
goto continue_unlock;
@@ -2067,17 +2245,22 @@ static int f2fs_write_cache_pages(struct address_space *mapping,
goto retry_write;
}
continue;
+ } else {
+ f2fs_msg(F2FS_M_SB(mapping)->sb,
+ KERN_WARNING,
+ "__write_data_page failed %d "
+ "mode=%d type=%d index=%lu",
+ ret, wbc->sync_mode,
+ io_type, page->index);
}
done_index = page->index + 1;
done = 1;
break;
} else if (submitted) {
- last_idx = page->index;
+ nwritten++;
}
- /* give a priority to WB_SYNC threads */
- if ((atomic_read(&F2FS_M_SB(mapping)->wb_sync_req) ||
- --wbc->nr_to_write <= 0) &&
+ if (--wbc->nr_to_write <= 0 &&
wbc->sync_mode == WB_SYNC_NONE) {
done = 1;
break;
@@ -2096,9 +2279,9 @@ static int f2fs_write_cache_pages(struct address_space *mapping,
if (wbc->range_cyclic || (range_whole && wbc->nr_to_write > 0))
mapping->writeback_index = done_index;
- if (last_idx != ULONG_MAX)
+ if (nwritten)
f2fs_submit_merged_write_cond(F2FS_M_SB(mapping), mapping->host,
- 0, last_idx, DATA);
+ NULL, 0, DATA);
return ret;
}
@@ -2108,6 +2291,8 @@ static inline bool __should_serialize_io(struct inode *inode,
{
if (!S_ISREG(inode->i_mode))
return false;
+ if (IS_NOQUOTA(inode))
+ return false;
if (wbc->sync_mode != WB_SYNC_ALL)
return true;
if (get_dirty_pages(inode) >= SM_I(F2FS_I_SB(inode))->min_seq_blocks)
@@ -2115,7 +2300,7 @@ static inline bool __should_serialize_io(struct inode *inode,
return false;
}
-int __f2fs_write_data_pages(struct address_space *mapping,
+static int __f2fs_write_data_pages(struct address_space *mapping,
struct writeback_control *wbc,
enum iostat_type io_type)
{
@@ -2137,9 +2322,10 @@ int __f2fs_write_data_pages(struct address_space *mapping,
if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
goto skip_write;
- if (S_ISDIR(inode->i_mode) && wbc->sync_mode == WB_SYNC_NONE &&
+ if ((S_ISDIR(inode->i_mode) || IS_NOQUOTA(inode)) &&
+ wbc->sync_mode == WB_SYNC_NONE &&
get_dirty_pages(inode) < nr_pages_to_skip(sbi, DATA) &&
- available_free_memory(sbi, DIRTY_DENTS))
+ f2fs_available_free_memory(sbi, DIRTY_DENTS))
goto skip_write;
/* skip writing during file defragment */
@@ -2150,8 +2336,8 @@ int __f2fs_write_data_pages(struct address_space *mapping,
/* to avoid spliting IOs due to mixed WB_SYNC_ALL and WB_SYNC_NONE */
if (wbc->sync_mode == WB_SYNC_ALL)
- atomic_inc(&sbi->wb_sync_req);
- else if (atomic_read(&sbi->wb_sync_req))
+ atomic_inc(&sbi->wb_sync_req[DATA]);
+ else if (atomic_read(&sbi->wb_sync_req[DATA]))
goto skip_write;
if (__should_serialize_io(inode, wbc)) {
@@ -2167,13 +2353,13 @@ int __f2fs_write_data_pages(struct address_space *mapping,
mutex_unlock(&sbi->writepages);
if (wbc->sync_mode == WB_SYNC_ALL)
- atomic_dec(&sbi->wb_sync_req);
+ atomic_dec(&sbi->wb_sync_req[DATA]);
/*
* if some pages were truncated, we cannot guarantee its mapping->host
* to detect pending bios.
*/
- remove_dirty_inode(inode);
+ f2fs_remove_dirty_inode(inode);
return ret;
skip_write:
@@ -2198,10 +2384,14 @@ static void f2fs_write_failed(struct address_space *mapping, loff_t to)
loff_t i_size = i_size_read(inode);
if (to > i_size) {
+ down_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
down_write(&F2FS_I(inode)->i_mmap_sem);
+
truncate_pagecache(inode, i_size);
- truncate_blocks(inode, i_size, true);
+ f2fs_truncate_blocks(inode, i_size, true, true);
+
up_write(&F2FS_I(inode)->i_mmap_sem);
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
}
}
@@ -2216,6 +2406,7 @@ static int prepare_write_begin(struct f2fs_sb_info *sbi,
bool locked = false;
struct extent_info ei = {0,0,0};
int err = 0;
+ int flag;
/*
* we already allocated all the blocks, so we don't need to get
@@ -2225,14 +2416,20 @@ static int prepare_write_begin(struct f2fs_sb_info *sbi,
!is_inode_flag_set(inode, FI_NO_PREALLOC))
return 0;
+ /* f2fs_lock_op avoids race between write CP and convert_inline_page */
+ if (f2fs_has_inline_data(inode) && pos + len > MAX_INLINE_DATA(inode))
+ flag = F2FS_GET_BLOCK_DEFAULT;
+ else
+ flag = F2FS_GET_BLOCK_PRE_AIO;
+
if (f2fs_has_inline_data(inode) ||
(pos & PAGE_MASK) >= i_size_read(inode)) {
- __do_map_lock(sbi, F2FS_GET_BLOCK_PRE_AIO, true);
+ __do_map_lock(sbi, flag, true);
locked = true;
}
restart:
/* check inline_data */
- ipage = get_node_page(sbi, inode->i_ino);
+ ipage = f2fs_get_node_page(sbi, inode->i_ino);
if (IS_ERR(ipage)) {
err = PTR_ERR(ipage);
goto unlock_out;
@@ -2242,7 +2439,7 @@ static int prepare_write_begin(struct f2fs_sb_info *sbi,
if (f2fs_has_inline_data(inode)) {
if (pos + len <= MAX_INLINE_DATA(inode)) {
- read_inline_data(page, ipage);
+ f2fs_do_read_inline_data(page, ipage);
set_inode_flag(inode, FI_DATA_EXIST);
if (inode->i_nlink)
set_inline_node(ipage);
@@ -2260,11 +2457,12 @@ static int prepare_write_begin(struct f2fs_sb_info *sbi,
dn.data_blkaddr = ei.blk + index - ei.fofs;
} else {
/* hole case */
- err = get_dnode_of_data(&dn, index, LOOKUP_NODE);
+ err = f2fs_get_dnode_of_data(&dn, index, LOOKUP_NODE);
if (err || dn.data_blkaddr == NULL_ADDR) {
f2fs_put_dnode(&dn);
__do_map_lock(sbi, F2FS_GET_BLOCK_PRE_AIO,
true);
+ WARN_ON(flag != F2FS_GET_BLOCK_PRE_AIO);
locked = true;
goto restart;
}
@@ -2278,7 +2476,7 @@ static int prepare_write_begin(struct f2fs_sb_info *sbi,
f2fs_put_dnode(&dn);
unlock_out:
if (locked)
- __do_map_lock(sbi, F2FS_GET_BLOCK_PRE_AIO, false);
+ __do_map_lock(sbi, flag, false);
return err;
}
@@ -2306,8 +2504,13 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping,
}
trace_f2fs_write_begin(inode, pos, len, flags);
- if (f2fs_is_atomic_file(inode) &&
- !available_free_memory(sbi, INMEM_PAGES)) {
+ err = f2fs_is_checkpoint_ready(sbi);
+ if (err)
+ goto fail;
+
+ if ((f2fs_is_atomic_file(inode) &&
+ !f2fs_available_free_memory(sbi, INMEM_PAGES)) ||
+ is_inode_flag_set(inode, FI_ATOMIC_REVOKE_REQUEST)) {
err = -ENOMEM;
drop_atomic = true;
goto fail;
@@ -2342,7 +2545,8 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping,
if (err)
goto fail;
- if (need_balance && has_not_enough_free_secs(sbi, 0, 0)) {
+ if (need_balance && !IS_NOQUOTA(inode) &&
+ has_not_enough_free_secs(sbi, 0, 0)) {
unlock_page(page);
f2fs_balance_fs(sbi, true);
lock_page(page);
@@ -2353,11 +2557,7 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping,
}
}
- f2fs_wait_on_page_writeback(page, DATA, false);
-
- /* wait for GCed page writeback via META_MAPPING */
- if (f2fs_post_read_required(inode))
- f2fs_wait_on_block_writeback(sbi, blkaddr);
+ f2fs_wait_on_page_writeback(page, DATA, false, true);
if (len == PAGE_SIZE || PageUptodate(page))
return 0;
@@ -2371,7 +2571,7 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping,
zero_user_segment(page, 0, PAGE_SIZE);
SetPageUptodate(page);
} else {
- err = f2fs_submit_page_read(inode, page, blkaddr);
+ err = f2fs_submit_page_read(inode, page, blkaddr, 0);
if (err)
goto fail;
@@ -2391,7 +2591,7 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping,
f2fs_put_page(page, 1);
f2fs_write_failed(mapping, pos + len);
if (drop_atomic)
- drop_inmem_pages_all(sbi);
+ f2fs_drop_inmem_pages_all(sbi, false);
return err;
}
@@ -2432,36 +2632,93 @@ static int f2fs_write_end(struct file *file,
static int check_direct_IO(struct inode *inode, struct iov_iter *iter,
loff_t offset)
{
- unsigned blocksize_mask = inode->i_sb->s_blocksize - 1;
+ unsigned i_blkbits = READ_ONCE(inode->i_blkbits);
+ unsigned blkbits = i_blkbits;
+ unsigned blocksize_mask = (1 << blkbits) - 1;
+ unsigned long align = offset | iov_iter_alignment(iter);
+ struct block_device *bdev = inode->i_sb->s_bdev;
- if (offset & blocksize_mask)
- return -EINVAL;
-
- if (iov_iter_alignment(iter) & blocksize_mask)
- return -EINVAL;
-
+ if (align & blocksize_mask) {
+ if (bdev)
+ blkbits = blksize_bits(bdev_logical_block_size(bdev));
+ blocksize_mask = (1 << blkbits) - 1;
+ if (align & blocksize_mask)
+ return -EINVAL;
+ return 1;
+ }
return 0;
}
+static void f2fs_dio_end_io(struct bio *bio)
+{
+ struct f2fs_private_dio *dio = bio->bi_private;
+
+ dec_page_count(F2FS_I_SB(dio->inode),
+ dio->write ? F2FS_DIO_WRITE : F2FS_DIO_READ);
+
+ bio->bi_private = dio->orig_private;
+ bio->bi_end_io = dio->orig_end_io;
+
+ kvfree(dio);
+
+ bio_endio(bio);
+}
+
+static void f2fs_dio_submit_bio(struct bio *bio, struct inode *inode,
+ loff_t file_offset)
+{
+ struct f2fs_private_dio *dio;
+ bool write = (bio_op(bio) == REQ_OP_WRITE);
+ int err;
+
+ dio = f2fs_kzalloc(F2FS_I_SB(inode),
+ sizeof(struct f2fs_private_dio), GFP_NOFS);
+ if (!dio) {
+ err = -ENOMEM;
+ goto out;
+ }
+
+ dio->inode = inode;
+ dio->orig_end_io = bio->bi_end_io;
+ dio->orig_private = bio->bi_private;
+ dio->write = write;
+
+ bio->bi_end_io = f2fs_dio_end_io;
+ bio->bi_private = dio;
+
+ inc_page_count(F2FS_I_SB(inode),
+ write ? F2FS_DIO_WRITE : F2FS_DIO_READ);
+
+ submit_bio(bio);
+ return;
+out:
+ bio->bi_error = -EIO;
+ bio_endio(bio);
+}
+
static ssize_t f2fs_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
{
struct address_space *mapping = iocb->ki_filp->f_mapping;
struct inode *inode = mapping->host;
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+ struct f2fs_inode_info *fi = F2FS_I(inode);
size_t count = iov_iter_count(iter);
loff_t offset = iocb->ki_pos;
int rw = iov_iter_rw(iter);
int err;
enum rw_hint hint = iocb->ki_hint;
int whint_mode = F2FS_OPTION(sbi).whint_mode;
+ bool do_opu;
err = check_direct_IO(inode, iter, offset);
if (err)
- return err;
+ return err < 0 ? err : 0;
- if (f2fs_force_buffered_io(inode, rw))
+ if (f2fs_force_buffered_io(inode, iocb, iter))
return 0;
+ do_opu = allow_outplace_dio(inode, iocb, iter);
+
trace_f2fs_direct_IO_enter(inode, offset, count, rw);
if (trace_android_fs_dataread_start_enabled() &&
@@ -2489,17 +2746,33 @@ static ssize_t f2fs_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
if (rw == WRITE && whint_mode == WHINT_MODE_OFF)
iocb->ki_hint = WRITE_LIFE_NOT_SET;
- if (!down_read_trylock(&F2FS_I(inode)->dio_rwsem[rw])) {
- if (iocb->ki_flags & IOCB_NOWAIT) {
+ if (iocb->ki_flags & IOCB_NOWAIT) {
+ if (!down_read_trylock(&fi->i_gc_rwsem[rw])) {
iocb->ki_hint = hint;
err = -EAGAIN;
goto out;
}
- down_read(&F2FS_I(inode)->dio_rwsem[rw]);
+ if (do_opu && !down_read_trylock(&fi->i_gc_rwsem[READ])) {
+ up_read(&fi->i_gc_rwsem[rw]);
+ iocb->ki_hint = hint;
+ err = -EAGAIN;
+ goto out;
+ }
+ } else {
+ down_read(&fi->i_gc_rwsem[rw]);
+ if (do_opu)
+ down_read(&fi->i_gc_rwsem[READ]);
}
- err = blockdev_direct_IO(iocb, inode, iter, get_data_block_dio);
- up_read(&F2FS_I(inode)->dio_rwsem[rw]);
+ err = __blockdev_direct_IO(iocb, inode, inode->i_sb->s_bdev,
+ iter, rw == WRITE ? get_data_block_dio_write :
+ get_data_block_dio, NULL, f2fs_dio_submit_bio,
+ DIO_LOCKING | DIO_SKIP_HOLES);
+
+ if (do_opu)
+ up_read(&fi->i_gc_rwsem[READ]);
+
+ up_read(&fi->i_gc_rwsem[rw]);
if (rw == WRITE) {
if (whint_mode == WHINT_MODE_OFF)
@@ -2507,7 +2780,8 @@ static ssize_t f2fs_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
if (err > 0) {
f2fs_update_iostat(F2FS_I_SB(inode), APP_DIRECT_IO,
err);
- set_inode_flag(inode, FI_UPDATE_WRITE);
+ if (!do_opu)
+ set_inode_flag(inode, FI_UPDATE_WRITE);
} else if (err < 0) {
f2fs_write_failed(mapping, offset + count);
}
@@ -2542,13 +2816,15 @@ void f2fs_invalidate_page(struct page *page, unsigned int offset,
dec_page_count(sbi, F2FS_DIRTY_NODES);
} else {
inode_dec_dirty_pages(inode);
- remove_dirty_inode(inode);
+ f2fs_remove_dirty_inode(inode);
}
}
+ clear_cold_data(page);
+
/* This is atomic written page, keep Private */
if (IS_ATOMIC_WRITTEN_PAGE(page))
- return drop_inmem_page(inode, page);
+ return f2fs_drop_inmem_page(inode, page);
set_page_private(page, 0);
ClearPagePrivate(page);
@@ -2564,6 +2840,7 @@ int f2fs_release_page(struct page *page, gfp_t wait)
if (IS_ATOMIC_WRITTEN_PAGE(page))
return 0;
+ clear_cold_data(page);
set_page_private(page, 0);
ClearPagePrivate(page);
return 1;
@@ -2581,7 +2858,7 @@ static int f2fs_set_data_page_dirty(struct page *page)
if (f2fs_is_atomic_file(inode) && !f2fs_is_commit_atomic_write(inode)) {
if (!IS_ATOMIC_WRITTEN_PAGE(page)) {
- register_inmem_page(inode, page);
+ f2fs_register_inmem_page(inode, page);
return 1;
}
/*
@@ -2593,7 +2870,7 @@ static int f2fs_set_data_page_dirty(struct page *page)
if (!PageDirty(page)) {
__set_page_dirty_nobuffers(page);
- update_dirty_page(inode, page);
+ f2fs_update_dirty_page(inode, page);
return 1;
}
return 0;
@@ -2686,6 +2963,17 @@ const struct address_space_operations f2fs_dblock_aops = {
#endif
};
+void f2fs_clear_radix_tree_dirty_tag(struct page *page)
+{
+ struct address_space *mapping = page_mapping(page);
+ unsigned long flags;
+
+ spin_lock_irqsave(&mapping->tree_lock, flags);
+ radix_tree_tag_clear(&mapping->page_tree, page_index(page),
+ PAGECACHE_TAG_DIRTY);
+ spin_unlock_irqrestore(&mapping->tree_lock, flags);
+}
+
int __init f2fs_init_post_read_processing(void)
{
bio_post_read_ctx_cache = KMEM_CACHE(bio_post_read_ctx, 0);
diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c
index a66107b..f05b37e 100644
--- a/fs/f2fs/debug.c
+++ b/fs/f2fs/debug.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* f2fs debugging statistics
*
@@ -5,10 +6,6 @@
* http://www.samsung.com/
* Copyright (c) 2012 Linux Foundation
* Copyright (c) 2012 Greg Kroah-Hartman <gregkh@linuxfoundation.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
@@ -56,13 +53,18 @@ static void update_general_status(struct f2fs_sb_info *sbi)
si->vw_cnt = atomic_read(&sbi->vw_cnt);
si->max_aw_cnt = atomic_read(&sbi->max_aw_cnt);
si->max_vw_cnt = atomic_read(&sbi->max_vw_cnt);
+ si->nr_dio_read = get_pages(sbi, F2FS_DIO_READ);
+ si->nr_dio_write = get_pages(sbi, F2FS_DIO_WRITE);
si->nr_wb_cp_data = get_pages(sbi, F2FS_WB_CP_DATA);
si->nr_wb_data = get_pages(sbi, F2FS_WB_DATA);
+ si->nr_rd_data = get_pages(sbi, F2FS_RD_DATA);
+ si->nr_rd_node = get_pages(sbi, F2FS_RD_NODE);
+ si->nr_rd_meta = get_pages(sbi, F2FS_RD_META);
if (SM_I(sbi) && SM_I(sbi)->fcc_info) {
si->nr_flushed =
atomic_read(&SM_I(sbi)->fcc_info->issued_flush);
si->nr_flushing =
- atomic_read(&SM_I(sbi)->fcc_info->issing_flush);
+ atomic_read(&SM_I(sbi)->fcc_info->queued_flush);
si->flush_list_empty =
llist_empty(&SM_I(sbi)->fcc_info->issue_list);
}
@@ -70,7 +72,7 @@ static void update_general_status(struct f2fs_sb_info *sbi)
si->nr_discarded =
atomic_read(&SM_I(sbi)->dcc_info->issued_discard);
si->nr_discarding =
- atomic_read(&SM_I(sbi)->dcc_info->issing_discard);
+ atomic_read(&SM_I(sbi)->dcc_info->queued_discard);
si->nr_discard_cmd =
atomic_read(&SM_I(sbi)->dcc_info->discard_cmd_cnt);
si->undiscard_blks = SM_I(sbi)->dcc_info->undiscard_blks;
@@ -94,8 +96,10 @@ static void update_general_status(struct f2fs_sb_info *sbi)
si->free_secs = free_sections(sbi);
si->prefree_count = prefree_segments(sbi);
si->dirty_count = dirty_segments(sbi);
- si->node_pages = NODE_MAPPING(sbi)->nrpages;
- si->meta_pages = META_MAPPING(sbi)->nrpages;
+ if (sbi->node_inode)
+ si->node_pages = NODE_MAPPING(sbi)->nrpages;
+ if (sbi->meta_inode)
+ si->meta_pages = META_MAPPING(sbi)->nrpages;
si->nats = NM_I(sbi)->nat_cnt;
si->dirty_nats = NM_I(sbi)->dirty_nat_cnt;
si->sits = MAIN_SEGS(sbi);
@@ -104,6 +108,10 @@ static void update_general_status(struct f2fs_sb_info *sbi)
si->avail_nids = NM_I(sbi)->available_nids;
si->alloc_nids = NM_I(sbi)->nid_cnt[PREALLOC_NID];
si->bg_gc = sbi->bg_gc;
+ si->io_skip_bggc = sbi->io_skip_bggc;
+ si->other_skip_bggc = sbi->other_skip_bggc;
+ si->skipped_atomic_files[BG_GC] = sbi->skipped_atomic_files[BG_GC];
+ si->skipped_atomic_files[FG_GC] = sbi->skipped_atomic_files[FG_GC];
si->util_free = (int)(free_user_blocks(sbi) >> sbi->log_blocks_per_seg)
* 100 / (int)(sbi->user_block_count >> sbi->log_blocks_per_seg)
/ 2;
@@ -119,6 +127,9 @@ static void update_general_status(struct f2fs_sb_info *sbi)
si->curzone[i] = GET_ZONE_FROM_SEC(sbi, si->cursec[i]);
}
+ for (i = META_CP; i < META_MAX; i++)
+ si->meta_count[i] = atomic_read(&sbi->meta_count[i]);
+
for (i = 0; i < 2; i++) {
si->segment_count[i] = sbi->segment_count[i];
si->block_count[i] = sbi->block_count[i];
@@ -166,7 +177,6 @@ static void update_sit_info(struct f2fs_sb_info *sbi)
static void update_mem_info(struct f2fs_sb_info *sbi)
{
struct f2fs_stat_info *si = F2FS_STAT(sbi);
- unsigned npages;
int i;
if (si->base_mem)
@@ -188,10 +198,9 @@ static void update_mem_info(struct f2fs_sb_info *sbi)
si->base_mem += MAIN_SEGS(sbi) * sizeof(struct seg_entry);
si->base_mem += f2fs_bitmap_size(MAIN_SEGS(sbi));
si->base_mem += 2 * SIT_VBLOCK_MAP_SIZE * MAIN_SEGS(sbi);
- if (f2fs_discard_en(sbi))
- si->base_mem += SIT_VBLOCK_MAP_SIZE * MAIN_SEGS(sbi);
+ si->base_mem += SIT_VBLOCK_MAP_SIZE * MAIN_SEGS(sbi);
si->base_mem += SIT_VBLOCK_MAP_SIZE;
- if (sbi->segs_per_sec > 1)
+ if (__is_large_section(sbi))
si->base_mem += MAIN_SECS(sbi) * sizeof(struct sec_entry);
si->base_mem += __bitmap_size(sbi, SIT_BITMAP);
@@ -213,7 +222,8 @@ static void update_mem_info(struct f2fs_sb_info *sbi)
si->base_mem += sizeof(struct f2fs_nm_info);
si->base_mem += __bitmap_size(sbi, NAT_BITMAP);
si->base_mem += (NM_I(sbi)->nat_bits_blocks << F2FS_BLKSIZE_BITS);
- si->base_mem += NM_I(sbi)->nat_blocks * NAT_ENTRY_BITMAP_SIZE;
+ si->base_mem += NM_I(sbi)->nat_blocks *
+ f2fs_bitmap_size(NAT_ENTRY_PER_BLOCK);
si->base_mem += NM_I(sbi)->nat_blocks / 8;
si->base_mem += NM_I(sbi)->nat_blocks * sizeof(unsigned short);
@@ -249,10 +259,14 @@ static void update_mem_info(struct f2fs_sb_info *sbi)
sizeof(struct extent_node);
si->page_mem = 0;
- npages = NODE_MAPPING(sbi)->nrpages;
- si->page_mem += (unsigned long long)npages << PAGE_SHIFT;
- npages = META_MAPPING(sbi)->nrpages;
- si->page_mem += (unsigned long long)npages << PAGE_SHIFT;
+ if (sbi->node_inode) {
+ unsigned npages = NODE_MAPPING(sbi)->nrpages;
+ si->page_mem += (unsigned long long)npages << PAGE_SHIFT;
+ }
+ if (sbi->meta_inode) {
+ unsigned npages = META_MAPPING(sbi)->nrpages;
+ si->page_mem += (unsigned long long)npages << PAGE_SHIFT;
+ }
}
static int stat_show(struct seq_file *s, void *v)
@@ -268,7 +282,8 @@ static int stat_show(struct seq_file *s, void *v)
seq_printf(s, "\n=====[ partition info(%pg). #%d, %s, CP: %s]=====\n",
si->sbi->sb->s_bdev, i++,
f2fs_readonly(si->sbi->sb) ? "RO": "RW",
- f2fs_cp_error(si->sbi) ? "Error": "Good");
+ is_set_ckpt_flags(si->sbi, CP_DISABLED_FLAG) ?
+ "Disabled": (f2fs_cp_error(si->sbi) ? "Error": "Good"));
seq_printf(s, "[SB: 1] [CP: 2] [SIT: %d] [NAT: %d] ",
si->sit_area_segs, si->nat_area_segs);
seq_printf(s, "[SSA: %d] [MAIN: %d",
@@ -330,6 +345,13 @@ static int stat_show(struct seq_file *s, void *v)
si->prefree_count, si->free_segs, si->free_secs);
seq_printf(s, "CP calls: %d (BG: %d)\n",
si->cp_count, si->bg_cp_count);
+ seq_printf(s, " - cp blocks : %u\n", si->meta_count[META_CP]);
+ seq_printf(s, " - sit blocks : %u\n",
+ si->meta_count[META_SIT]);
+ seq_printf(s, " - nat blocks : %u\n",
+ si->meta_count[META_NAT]);
+ seq_printf(s, " - ssa blocks : %u\n",
+ si->meta_count[META_SSA]);
seq_printf(s, "GC calls: %d (BG: %d)\n",
si->call_count, si->bg_gc);
seq_printf(s, " - data segments : %d (%d)\n",
@@ -342,6 +364,12 @@ static int stat_show(struct seq_file *s, void *v)
si->bg_data_blks);
seq_printf(s, " - node blocks : %d (%d)\n", si->node_blks,
si->bg_node_blks);
+ seq_printf(s, "Skipped : atomic write %llu (%llu)\n",
+ si->skipped_atomic_files[BG_GC] +
+ si->skipped_atomic_files[FG_GC],
+ si->skipped_atomic_files[BG_GC]);
+ seq_printf(s, "BG skip : IO: %u, Other: %u\n",
+ si->io_skip_bggc, si->other_skip_bggc);
seq_puts(s, "\nExtent Cache:\n");
seq_printf(s, " - Hit Count: L1-1:%llu L1-2:%llu L2:%llu\n",
si->hit_largest, si->hit_cached,
@@ -353,7 +381,11 @@ static int stat_show(struct seq_file *s, void *v)
seq_printf(s, " - Inner Struct Count: tree: %d(%d), node: %d\n",
si->ext_tree, si->zombie_tree, si->ext_node);
seq_puts(s, "\nBalancing F2FS Async:\n");
- seq_printf(s, " - IO (CP: %4d, Data: %4d, Flush: (%4d %4d %4d), "
+ seq_printf(s, " - DIO (R: %4d, W: %4d)\n",
+ si->nr_dio_read, si->nr_dio_write);
+ seq_printf(s, " - IO_R (Data: %4d, Node: %4d, Meta: %4d\n",
+ si->nr_rd_data, si->nr_rd_node, si->nr_rd_meta);
+ seq_printf(s, " - IO_W (CP: %4d, Data: %4d, Flush: (%4d %4d %4d), "
"Discard: (%4d %4d)) cmd: %4d undiscard:%4u\n",
si->nr_wb_cp_data, si->nr_wb_data,
si->nr_flushing, si->nr_flushed,
@@ -438,6 +470,7 @@ int f2fs_build_stats(struct f2fs_sb_info *sbi)
{
struct f2fs_super_block *raw_super = F2FS_RAW_SUPER(sbi);
struct f2fs_stat_info *si;
+ int i;
si = f2fs_kzalloc(sbi, sizeof(struct f2fs_stat_info), GFP_KERNEL);
if (!si)
@@ -463,6 +496,8 @@ int f2fs_build_stats(struct f2fs_sb_info *sbi)
atomic_set(&sbi->inline_inode, 0);
atomic_set(&sbi->inline_dir, 0);
atomic_set(&sbi->inplace_count, 0);
+ for (i = META_CP; i < META_MAX; i++)
+ atomic_set(&sbi->meta_count[i], 0);
atomic_set(&sbi->aw_cnt, 0);
atomic_set(&sbi->vw_cnt, 0);
@@ -484,7 +519,7 @@ void f2fs_destroy_stats(struct f2fs_sb_info *sbi)
list_del(&si->stat_list);
mutex_unlock(&f2fs_stat_mutex);
- kfree(si);
+ kvfree(si);
}
int __init f2fs_create_root_stats(void)
diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c
index f9a1e18..61d14c1 100644
--- a/fs/f2fs/dir.c
+++ b/fs/f2fs/dir.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/dir.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
@@ -60,12 +57,12 @@ static unsigned char f2fs_type_by_mode[S_IFMT >> S_SHIFT] = {
[S_IFLNK >> S_SHIFT] = F2FS_FT_SYMLINK,
};
-void set_de_type(struct f2fs_dir_entry *de, umode_t mode)
+static void set_de_type(struct f2fs_dir_entry *de, umode_t mode)
{
de->file_type = f2fs_type_by_mode[(mode & S_IFMT) >> S_SHIFT];
}
-unsigned char get_de_type(struct f2fs_dir_entry *de)
+unsigned char f2fs_get_de_type(struct f2fs_dir_entry *de)
{
if (de->file_type < F2FS_FT_MAX)
return f2fs_filetype_table[de->file_type];
@@ -97,14 +94,14 @@ static struct f2fs_dir_entry *find_in_block(struct page *dentry_page,
dentry_blk = (struct f2fs_dentry_block *)page_address(dentry_page);
make_dentry_ptr_block(NULL, &d, dentry_blk);
- de = find_target_dentry(fname, namehash, max_slots, &d);
+ de = f2fs_find_target_dentry(fname, namehash, max_slots, &d);
if (de)
*res_page = dentry_page;
return de;
}
-struct f2fs_dir_entry *find_target_dentry(struct fscrypt_name *fname,
+struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
f2fs_hash_t namehash, int *max_slots,
struct f2fs_dentry_ptr *d)
{
@@ -171,7 +168,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
for (; bidx < end_block; bidx++) {
/* no need to allocate new dentry pages to all the indices */
- dentry_page = find_data_page(dir, bidx);
+ dentry_page = f2fs_find_data_page(dir, bidx, 0);
if (IS_ERR(dentry_page)) {
if (PTR_ERR(dentry_page) == -ENOENT) {
room = true;
@@ -210,7 +207,7 @@ struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir,
if (f2fs_has_inline_dentry(dir)) {
*res_page = NULL;
- de = find_in_inline_dir(dir, fname, res_page);
+ de = f2fs_find_in_inline_dir(dir, fname, res_page);
goto out;
}
@@ -296,7 +293,7 @@ void f2fs_set_link(struct inode *dir, struct f2fs_dir_entry *de,
{
enum page_type type = f2fs_has_inline_dentry(dir) ? NODE : DATA;
lock_page(page);
- f2fs_wait_on_page_writeback(page, type, true);
+ f2fs_wait_on_page_writeback(page, type, true, true);
de->ino = cpu_to_le32(inode->i_ino);
set_de_type(de, inode->i_mode);
set_page_dirty(page);
@@ -310,7 +307,7 @@ static void init_dent_inode(const struct qstr *name, struct page *ipage)
{
struct f2fs_inode *ri;
- f2fs_wait_on_page_writeback(ipage, NODE, true);
+ f2fs_wait_on_page_writeback(ipage, NODE, true, true);
/* copy name info. to this inode page */
ri = F2FS_INODE(ipage);
@@ -319,7 +316,7 @@ static void init_dent_inode(const struct qstr *name, struct page *ipage)
set_page_dirty(ipage);
}
-void do_make_empty_dir(struct inode *inode, struct inode *parent,
+void f2fs_do_make_empty_dir(struct inode *inode, struct inode *parent,
struct f2fs_dentry_ptr *d)
{
struct qstr dot = QSTR_INIT(".", 1);
@@ -340,23 +337,23 @@ static int make_empty_dir(struct inode *inode,
struct f2fs_dentry_ptr d;
if (f2fs_has_inline_dentry(inode))
- return make_empty_inline_dir(inode, parent, page);
+ return f2fs_make_empty_inline_dir(inode, parent, page);
- dentry_page = get_new_data_page(inode, page, 0, true);
+ dentry_page = f2fs_get_new_data_page(inode, page, 0, true);
if (IS_ERR(dentry_page))
return PTR_ERR(dentry_page);
dentry_blk = page_address(dentry_page);
make_dentry_ptr_block(NULL, &d, dentry_blk);
- do_make_empty_dir(inode, parent, &d);
+ f2fs_do_make_empty_dir(inode, parent, &d);
set_page_dirty(dentry_page);
f2fs_put_page(dentry_page, 1);
return 0;
}
-struct page *init_inode_metadata(struct inode *inode, struct inode *dir,
+struct page *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir,
const struct qstr *new_name, const struct qstr *orig_name,
struct page *dpage)
{
@@ -365,7 +362,7 @@ struct page *init_inode_metadata(struct inode *inode, struct inode *dir,
int err;
if (is_inode_flag_set(inode, FI_NEW_INODE)) {
- page = new_inode_page(inode);
+ page = f2fs_new_inode_page(inode);
if (IS_ERR(page))
return page;
@@ -395,7 +392,7 @@ struct page *init_inode_metadata(struct inode *inode, struct inode *dir,
goto put_error;
}
} else {
- page = get_node_page(F2FS_I_SB(dir), inode->i_ino);
+ page = f2fs_get_node_page(F2FS_I_SB(dir), inode->i_ino);
if (IS_ERR(page))
return page;
}
@@ -418,19 +415,19 @@ struct page *init_inode_metadata(struct inode *inode, struct inode *dir,
* we should remove this inode from orphan list.
*/
if (inode->i_nlink == 0)
- remove_orphan_inode(F2FS_I_SB(dir), inode->i_ino);
+ f2fs_remove_orphan_inode(F2FS_I_SB(dir), inode->i_ino);
f2fs_i_links_write(inode, true);
}
return page;
put_error:
clear_nlink(inode);
- update_inode(inode, page);
+ f2fs_update_inode(inode, page);
f2fs_put_page(page, 1);
return ERR_PTR(err);
}
-void update_parent_metadata(struct inode *dir, struct inode *inode,
+void f2fs_update_parent_metadata(struct inode *dir, struct inode *inode,
unsigned int current_depth)
{
if (inode && is_inode_flag_set(inode, FI_NEW_INODE)) {
@@ -448,7 +445,7 @@ void update_parent_metadata(struct inode *dir, struct inode *inode,
clear_inode_flag(inode, FI_INC_LINK);
}
-int room_for_filename(const void *bitmap, int slots, int max_slots)
+int f2fs_room_for_filename(const void *bitmap, int slots, int max_slots)
{
int bit_start = 0;
int zero_start, zero_end;
@@ -517,12 +514,11 @@ int f2fs_add_regular_entry(struct inode *dir, const struct qstr *new_name,
}
start:
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(F2FS_I_SB(dir), FAULT_DIR_DEPTH)) {
f2fs_show_injection_info(FAULT_DIR_DEPTH);
return -ENOSPC;
}
-#endif
+
if (unlikely(current_depth == MAX_DIR_HASH_DEPTH))
return -ENOSPC;
@@ -537,12 +533,12 @@ int f2fs_add_regular_entry(struct inode *dir, const struct qstr *new_name,
(le32_to_cpu(dentry_hash) % nbucket));
for (block = bidx; block <= (bidx + nblock - 1); block++) {
- dentry_page = get_new_data_page(dir, NULL, block, true);
+ dentry_page = f2fs_get_new_data_page(dir, NULL, block, true);
if (IS_ERR(dentry_page))
return PTR_ERR(dentry_page);
dentry_blk = page_address(dentry_page);
- bit_pos = room_for_filename(&dentry_blk->dentry_bitmap,
+ bit_pos = f2fs_room_for_filename(&dentry_blk->dentry_bitmap,
slots, NR_DENTRY_IN_BLOCK);
if (bit_pos < NR_DENTRY_IN_BLOCK)
goto add_dentry;
@@ -554,11 +550,11 @@ int f2fs_add_regular_entry(struct inode *dir, const struct qstr *new_name,
++level;
goto start;
add_dentry:
- f2fs_wait_on_page_writeback(dentry_page, DATA, true);
+ f2fs_wait_on_page_writeback(dentry_page, DATA, true, true);
if (inode) {
down_write(&F2FS_I(inode)->i_sem);
- page = init_inode_metadata(inode, dir, new_name,
+ page = f2fs_init_inode_metadata(inode, dir, new_name,
orig_name, NULL);
if (IS_ERR(page)) {
err = PTR_ERR(page);
@@ -576,7 +572,7 @@ int f2fs_add_regular_entry(struct inode *dir, const struct qstr *new_name,
f2fs_put_page(page, 1);
}
- update_parent_metadata(dir, inode, current_depth);
+ f2fs_update_parent_metadata(dir, inode, current_depth);
fail:
if (inode)
up_write(&F2FS_I(inode)->i_sem);
@@ -586,7 +582,7 @@ int f2fs_add_regular_entry(struct inode *dir, const struct qstr *new_name,
return err;
}
-int __f2fs_do_add_link(struct inode *dir, struct fscrypt_name *fname,
+int f2fs_add_dentry(struct inode *dir, struct fscrypt_name *fname,
struct inode *inode, nid_t ino, umode_t mode)
{
struct qstr new_name;
@@ -610,7 +606,7 @@ int __f2fs_do_add_link(struct inode *dir, struct fscrypt_name *fname,
* Caller should grab and release a rwsem by calling f2fs_lock_op() and
* f2fs_unlock_op().
*/
-int __f2fs_add_link(struct inode *dir, const struct qstr *name,
+int f2fs_do_add_link(struct inode *dir, const struct qstr *name,
struct inode *inode, nid_t ino, umode_t mode)
{
struct fscrypt_name fname;
@@ -639,7 +635,7 @@ int __f2fs_add_link(struct inode *dir, const struct qstr *name,
} else if (IS_ERR(page)) {
err = PTR_ERR(page);
} else {
- err = __f2fs_do_add_link(dir, &fname, inode, ino, mode);
+ err = f2fs_add_dentry(dir, &fname, inode, ino, mode);
}
fscrypt_free_filename(&fname);
return err;
@@ -651,7 +647,7 @@ int f2fs_do_tmpfile(struct inode *inode, struct inode *dir)
int err = 0;
down_write(&F2FS_I(inode)->i_sem);
- page = init_inode_metadata(inode, dir, NULL, NULL, NULL);
+ page = f2fs_init_inode_metadata(inode, dir, NULL, NULL, NULL);
if (IS_ERR(page)) {
err = PTR_ERR(page);
goto fail;
@@ -659,9 +655,9 @@ int f2fs_do_tmpfile(struct inode *inode, struct inode *dir)
f2fs_put_page(page, 1);
clear_inode_flag(inode, FI_NEW_INODE);
+ f2fs_update_time(F2FS_I_SB(inode), REQ_TIME);
fail:
up_write(&F2FS_I(inode)->i_sem);
- f2fs_update_time(F2FS_I_SB(inode), REQ_TIME);
return err;
}
@@ -683,9 +679,9 @@ void f2fs_drop_nlink(struct inode *dir, struct inode *inode)
up_write(&F2FS_I(inode)->i_sem);
if (inode->i_nlink == 0)
- add_orphan_inode(inode);
+ f2fs_add_orphan_inode(inode);
else
- release_orphan_inode(sbi);
+ f2fs_release_orphan_inode(sbi);
}
/*
@@ -698,20 +694,18 @@ void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct page *page,
struct f2fs_dentry_block *dentry_blk;
unsigned int bit_pos;
int slots = GET_DENTRY_SLOTS(le16_to_cpu(dentry->name_len));
- struct address_space *mapping = page_mapping(page);
- unsigned long flags;
int i;
f2fs_update_time(F2FS_I_SB(dir), REQ_TIME);
if (F2FS_OPTION(F2FS_I_SB(dir)).fsync_mode == FSYNC_MODE_STRICT)
- add_ino_entry(F2FS_I_SB(dir), dir->i_ino, TRANS_DIR_INO);
+ f2fs_add_ino_entry(F2FS_I_SB(dir), dir->i_ino, TRANS_DIR_INO);
if (f2fs_has_inline_dentry(dir))
return f2fs_delete_inline_entry(dentry, page, dir, inode);
lock_page(page);
- f2fs_wait_on_page_writeback(page, DATA, true);
+ f2fs_wait_on_page_writeback(page, DATA, true, true);
dentry_blk = page_address(page);
bit_pos = dentry - dentry_blk->dentry;
@@ -731,17 +725,14 @@ void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct page *page,
f2fs_drop_nlink(dir, inode);
if (bit_pos == NR_DENTRY_IN_BLOCK &&
- !truncate_hole(dir, page->index, page->index + 1)) {
- spin_lock_irqsave(&mapping->tree_lock, flags);
- radix_tree_tag_clear(&mapping->page_tree, page_index(page),
- PAGECACHE_TAG_DIRTY);
- spin_unlock_irqrestore(&mapping->tree_lock, flags);
-
+ !f2fs_truncate_hole(dir, page->index, page->index + 1)) {
+ f2fs_clear_radix_tree_dirty_tag(page);
clear_page_dirty_for_io(page);
ClearPagePrivate(page);
ClearPageUptodate(page);
+ clear_cold_data(page);
inode_dec_dirty_pages(dir);
- remove_dirty_inode(dir);
+ f2fs_remove_dirty_inode(dir);
}
f2fs_put_page(page, 1);
}
@@ -758,7 +749,7 @@ bool f2fs_empty_dir(struct inode *dir)
return f2fs_empty_inline_dir(dir);
for (bidx = 0; bidx < nblock; bidx++) {
- dentry_page = get_lock_data_page(dir, bidx, false);
+ dentry_page = f2fs_get_lock_data_page(dir, bidx, false);
if (IS_ERR(dentry_page)) {
if (PTR_ERR(dentry_page) == -ENOENT)
continue;
@@ -791,9 +782,15 @@ int f2fs_fill_dentries(struct dir_context *ctx, struct f2fs_dentry_ptr *d,
struct f2fs_dir_entry *de = NULL;
struct fscrypt_str de_name = FSTR_INIT(NULL, 0);
struct f2fs_sb_info *sbi = F2FS_I_SB(d->inode);
+ struct blk_plug plug;
+ bool readdir_ra = sbi->readdir_ra == 1;
+ int err = 0;
bit_pos = ((unsigned long)ctx->pos % d->max);
+ if (readdir_ra)
+ blk_start_plug(&plug);
+
while (bit_pos < d->max) {
bit_pos = find_next_bit_le(d->bitmap, d->max, bit_pos);
if (bit_pos >= d->max)
@@ -806,36 +803,50 @@ int f2fs_fill_dentries(struct dir_context *ctx, struct f2fs_dentry_ptr *d,
continue;
}
- d_type = get_de_type(de);
+ d_type = f2fs_get_de_type(de);
de_name.name = d->filename[bit_pos];
de_name.len = le16_to_cpu(de->name_len);
+ /* check memory boundary before moving forward */
+ bit_pos += GET_DENTRY_SLOTS(le16_to_cpu(de->name_len));
+ if (unlikely(bit_pos > d->max)) {
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "%s: corrupted namelen=%d, run fsck to fix.",
+ __func__, le16_to_cpu(de->name_len));
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ err = -EINVAL;
+ goto out;
+ }
+
if (f2fs_encrypted_inode(d->inode)) {
int save_len = fstr->len;
- int err;
err = fscrypt_fname_disk_to_usr(d->inode,
(u32)de->hash_code, 0,
&de_name, fstr);
if (err)
- return err;
+ goto out;
de_name = *fstr;
fstr->len = save_len;
}
if (!dir_emit(ctx, de_name.name, de_name.len,
- le32_to_cpu(de->ino), d_type))
- return 1;
+ le32_to_cpu(de->ino), d_type)) {
+ err = 1;
+ goto out;
+ }
- if (sbi->readdir_ra == 1)
- ra_node_page(sbi, le32_to_cpu(de->ino));
+ if (readdir_ra)
+ f2fs_ra_node_page(sbi, le32_to_cpu(de->ino));
- bit_pos += GET_DENTRY_SLOTS(le16_to_cpu(de->name_len));
ctx->pos = start_pos + bit_pos;
}
- return 0;
+out:
+ if (readdir_ra)
+ blk_finish_plug(&plug);
+ return err;
}
static int f2fs_readdir(struct file *file, struct dir_context *ctx)
@@ -880,7 +891,7 @@ static int f2fs_readdir(struct file *file, struct dir_context *ctx)
page_cache_sync_readahead(inode->i_mapping, ra, file, n,
min(npages - n, (pgoff_t)MAX_DIR_RA_PAGES));
- dentry_page = get_lock_data_page(inode, n, false);
+ dentry_page = f2fs_get_lock_data_page(inode, n, false);
if (IS_ERR(dentry_page)) {
err = PTR_ERR(dentry_page);
if (err == -ENOENT) {
diff --git a/fs/f2fs/extent_cache.c b/fs/f2fs/extent_cache.c
index d5a861b..904ad7b 100644
--- a/fs/f2fs/extent_cache.c
+++ b/fs/f2fs/extent_cache.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* f2fs extent cache support
*
@@ -5,10 +6,6 @@
* Copyright (c) 2015 Samsung Electronics
* Authors: Jaegeuk Kim <jaegeuk@kernel.org>
* Chao Yu <chao2.yu@samsung.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
@@ -49,7 +46,7 @@ static struct rb_entry *__lookup_rb_tree_slow(struct rb_root *root,
return NULL;
}
-struct rb_entry *__lookup_rb_tree(struct rb_root *root,
+struct rb_entry *f2fs_lookup_rb_tree(struct rb_root *root,
struct rb_entry *cached_re, unsigned int ofs)
{
struct rb_entry *re;
@@ -61,7 +58,7 @@ struct rb_entry *__lookup_rb_tree(struct rb_root *root,
return re;
}
-struct rb_node **__lookup_rb_tree_for_insert(struct f2fs_sb_info *sbi,
+struct rb_node **f2fs_lookup_rb_tree_for_insert(struct f2fs_sb_info *sbi,
struct rb_root *root, struct rb_node **parent,
unsigned int ofs)
{
@@ -92,7 +89,7 @@ struct rb_node **__lookup_rb_tree_for_insert(struct f2fs_sb_info *sbi,
* in order to simpfy the insertion after.
* tree must stay unchanged between lookup and insertion.
*/
-struct rb_entry *__lookup_rb_tree_ret(struct rb_root *root,
+struct rb_entry *f2fs_lookup_rb_tree_ret(struct rb_root *root,
struct rb_entry *cached_re,
unsigned int ofs,
struct rb_entry **prev_entry,
@@ -159,7 +156,7 @@ struct rb_entry *__lookup_rb_tree_ret(struct rb_root *root,
return re;
}
-bool __check_rb_tree_consistence(struct f2fs_sb_info *sbi,
+bool f2fs_check_rb_tree_consistence(struct f2fs_sb_info *sbi,
struct rb_root *root)
{
#ifdef CONFIG_F2FS_CHECK_FS
@@ -308,14 +305,13 @@ static unsigned int __free_extent_tree(struct f2fs_sb_info *sbi,
return count - atomic_read(&et->node_cnt);
}
-static void __drop_largest_extent(struct inode *inode,
+static void __drop_largest_extent(struct extent_tree *et,
pgoff_t fofs, unsigned int len)
{
- struct extent_info *largest = &F2FS_I(inode)->extent_tree->largest;
-
- if (fofs < largest->fofs + largest->len && fofs + len > largest->fofs) {
- largest->len = 0;
- f2fs_mark_inode_dirty_sync(inode, true);
+ if (fofs < et->largest.fofs + et->largest.len &&
+ fofs + len > et->largest.fofs) {
+ et->largest.len = 0;
+ et->largest_updated = true;
}
}
@@ -390,7 +386,7 @@ static bool f2fs_lookup_extent_tree(struct inode *inode, pgoff_t pgofs,
goto out;
}
- en = (struct extent_node *)__lookup_rb_tree(&et->root,
+ en = (struct extent_node *)f2fs_lookup_rb_tree(&et->root,
(struct rb_entry *)et->cached_en, pgofs);
if (!en)
goto out;
@@ -416,12 +412,11 @@ static bool f2fs_lookup_extent_tree(struct inode *inode, pgoff_t pgofs,
return ret;
}
-static struct extent_node *__try_merge_extent_node(struct inode *inode,
+static struct extent_node *__try_merge_extent_node(struct f2fs_sb_info *sbi,
struct extent_tree *et, struct extent_info *ei,
struct extent_node *prev_ex,
struct extent_node *next_ex)
{
- struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct extent_node *en = NULL;
if (prev_ex && __is_back_mergeable(ei, &prev_ex->ei)) {
@@ -443,7 +438,7 @@ static struct extent_node *__try_merge_extent_node(struct inode *inode,
if (!en)
return NULL;
- __try_update_largest_extent(inode, et, en);
+ __try_update_largest_extent(et, en);
spin_lock(&sbi->extent_lock);
if (!list_empty(&en->list)) {
@@ -454,12 +449,11 @@ static struct extent_node *__try_merge_extent_node(struct inode *inode,
return en;
}
-static struct extent_node *__insert_extent_tree(struct inode *inode,
+static struct extent_node *__insert_extent_tree(struct f2fs_sb_info *sbi,
struct extent_tree *et, struct extent_info *ei,
struct rb_node **insert_p,
struct rb_node *insert_parent)
{
- struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct rb_node **p;
struct rb_node *parent = NULL;
struct extent_node *en = NULL;
@@ -470,13 +464,13 @@ static struct extent_node *__insert_extent_tree(struct inode *inode,
goto do_insert;
}
- p = __lookup_rb_tree_for_insert(sbi, &et->root, &parent, ei->fofs);
+ p = f2fs_lookup_rb_tree_for_insert(sbi, &et->root, &parent, ei->fofs);
do_insert:
en = __attach_extent_node(sbi, et, ei, parent, p);
if (!en)
return NULL;
- __try_update_largest_extent(inode, et, en);
+ __try_update_largest_extent(et, en);
/* update in global extent list */
spin_lock(&sbi->extent_lock);
@@ -497,6 +491,7 @@ static void f2fs_update_extent_tree_range(struct inode *inode,
struct rb_node **insert_p = NULL, *insert_parent = NULL;
unsigned int end = fofs + len;
unsigned int pos = (unsigned int)fofs;
+ bool updated = false;
if (!et)
return;
@@ -517,10 +512,10 @@ static void f2fs_update_extent_tree_range(struct inode *inode,
* drop largest extent before lookup, in case it's already
* been shrunk from extent tree
*/
- __drop_largest_extent(inode, fofs, len);
+ __drop_largest_extent(et, fofs, len);
/* 1. lookup first extent node in range [fofs, fofs + len - 1] */
- en = (struct extent_node *)__lookup_rb_tree_ret(&et->root,
+ en = (struct extent_node *)f2fs_lookup_rb_tree_ret(&et->root,
(struct rb_entry *)et->cached_en, fofs,
(struct rb_entry **)&prev_en,
(struct rb_entry **)&next_en,
@@ -550,7 +545,7 @@ static void f2fs_update_extent_tree_range(struct inode *inode,
set_extent_info(&ei, end,
end - dei.fofs + dei.blk,
org_end - end);
- en1 = __insert_extent_tree(inode, et, &ei,
+ en1 = __insert_extent_tree(sbi, et, &ei,
NULL, NULL);
next_en = en1;
} else {
@@ -570,7 +565,7 @@ static void f2fs_update_extent_tree_range(struct inode *inode,
}
if (parts)
- __try_update_largest_extent(inode, et, en);
+ __try_update_largest_extent(et, en);
else
__release_extent_node(sbi, et, en);
@@ -590,15 +585,16 @@ static void f2fs_update_extent_tree_range(struct inode *inode,
if (blkaddr) {
set_extent_info(&ei, fofs, blkaddr, len);
- if (!__try_merge_extent_node(inode, et, &ei, prev_en, next_en))
- __insert_extent_tree(inode, et, &ei,
+ if (!__try_merge_extent_node(sbi, et, &ei, prev_en, next_en))
+ __insert_extent_tree(sbi, et, &ei,
insert_p, insert_parent);
/* give up extent_cache, if split and small updates happen */
if (dei.len >= 1 &&
prev.len < F2FS_MIN_EXTENT_LEN &&
et->largest.len < F2FS_MIN_EXTENT_LEN) {
- __drop_largest_extent(inode, 0, UINT_MAX);
+ et->largest.len = 0;
+ et->largest_updated = true;
set_inode_flag(inode, FI_NO_EXTENT);
}
}
@@ -606,7 +602,15 @@ static void f2fs_update_extent_tree_range(struct inode *inode,
if (is_inode_flag_set(inode, FI_NO_EXTENT))
__free_extent_tree(sbi, et);
+ if (et->largest_updated) {
+ et->largest_updated = false;
+ updated = true;
+ }
+
write_unlock(&et->lock);
+
+ if (updated)
+ f2fs_mark_inode_dirty_sync(inode, true);
}
unsigned int f2fs_shrink_extent_tree(struct f2fs_sb_info *sbi, int nr_shrink)
@@ -705,6 +709,7 @@ void f2fs_drop_extent_tree(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct extent_tree *et = F2FS_I(inode)->extent_tree;
+ bool updated = false;
if (!f2fs_may_extent_tree(inode))
return;
@@ -713,8 +718,13 @@ void f2fs_drop_extent_tree(struct inode *inode)
write_lock(&et->lock);
__free_extent_tree(sbi, et);
- __drop_largest_extent(inode, 0, UINT_MAX);
+ if (et->largest.len) {
+ et->largest.len = 0;
+ updated = true;
+ }
write_unlock(&et->lock);
+ if (updated)
+ f2fs_mark_inode_dirty_sync(inode, true);
}
void f2fs_destroy_extent_tree(struct inode *inode)
@@ -773,7 +783,7 @@ void f2fs_update_extent_cache(struct dnode_of_data *dn)
else
blkaddr = dn->data_blkaddr;
- fofs = start_bidx_of_node(ofs_of_node(dn->node_page), dn->inode) +
+ fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_page), dn->inode) +
dn->ofs_in_node;
f2fs_update_extent_tree_range(dn->inode, fofs, blkaddr, 1);
}
@@ -788,7 +798,7 @@ void f2fs_update_extent_cache_range(struct dnode_of_data *dn,
f2fs_update_extent_tree_range(dn->inode, fofs, blkaddr, len);
}
-void init_extent_cache_info(struct f2fs_sb_info *sbi)
+void f2fs_init_extent_cache_info(struct f2fs_sb_info *sbi)
{
INIT_RADIX_TREE(&sbi->extent_tree_root, GFP_NOIO);
mutex_init(&sbi->extent_tree_lock);
@@ -800,7 +810,7 @@ void init_extent_cache_info(struct f2fs_sb_info *sbi)
atomic_set(&sbi->total_ext_node, 0);
}
-int __init create_extent_cache(void)
+int __init f2fs_create_extent_cache(void)
{
extent_tree_slab = f2fs_kmem_cache_create("f2fs_extent_tree",
sizeof(struct extent_tree));
@@ -815,7 +825,7 @@ int __init create_extent_cache(void)
return 0;
}
-void destroy_extent_cache(void)
+void f2fs_destroy_extent_cache(void)
{
kmem_cache_destroy(extent_node_slab);
kmem_cache_destroy(extent_tree_slab);
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index 3f9a211..f67f038 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -1,16 +1,14 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/f2fs.h
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#ifndef _LINUX_F2FS_H
#define _LINUX_F2FS_H
+#include <linux/uio.h>
#include <linux/types.h>
#include <linux/page-flags.h>
#include <linux/buffer_head.h>
@@ -26,10 +24,14 @@
#include <linux/blkdev.h>
#include <linux/quotaops.h>
#include <crypto/hash.h>
+#include <linux/overflow.h>
#define __FS_HAS_ENCRYPTION IS_ENABLED(CONFIG_F2FS_FS_ENCRYPTION)
#include <linux/fscrypt.h>
+#define __FS_HAS_VERITY IS_ENABLED(CONFIG_F2FS_FS_VERITY)
+#include <linux/fsverity.h>
+
#ifdef CONFIG_F2FS_CHECK_FS
#define f2fs_bug_on(sbi, condition) BUG_ON(condition)
#else
@@ -42,7 +44,6 @@
} while (0)
#endif
-#ifdef CONFIG_F2FS_FAULT_INJECTION
enum {
FAULT_KMALLOC,
FAULT_KVMALLOC,
@@ -55,18 +56,23 @@ enum {
FAULT_DIR_DEPTH,
FAULT_EVICT_INODE,
FAULT_TRUNCATE,
- FAULT_IO,
+ FAULT_READ_IO,
FAULT_CHECKPOINT,
+ FAULT_DISCARD,
+ FAULT_WRITE_IO,
FAULT_MAX,
};
+#ifdef CONFIG_F2FS_FAULT_INJECTION
+#define F2FS_ALL_FAULT_TYPE ((1 << FAULT_MAX) - 1)
+
struct f2fs_fault_info {
atomic_t inject_ops;
unsigned int inject_rate;
unsigned int inject_type;
};
-extern char *fault_name[FAULT_MAX];
+extern const char *f2fs_fault_name[FAULT_MAX];
#define IS_FAULT_SET(fi, type) ((fi)->inject_type & (1 << (type)))
#endif
@@ -98,6 +104,7 @@ extern char *fault_name[FAULT_MAX];
#define F2FS_MOUNT_QUOTA 0x00400000
#define F2FS_MOUNT_INLINE_XATTR_SIZE 0x00800000
#define F2FS_MOUNT_RESERVE_ROOT 0x01000000
+#define F2FS_MOUNT_DISABLE_CHECKPOINT 0x02000000
#define F2FS_OPTION(sbi) ((sbi)->mount_opt)
#define clear_opt(sbi, option) (F2FS_OPTION(sbi).opt &= ~F2FS_MOUNT_##option)
@@ -147,14 +154,16 @@ struct f2fs_mount_info {
#define F2FS_FEATURE_QUOTA_INO 0x0080
#define F2FS_FEATURE_INODE_CRTIME 0x0100
#define F2FS_FEATURE_LOST_FOUND 0x0200
-#define F2FS_FEATURE_VERITY 0x0400 /* reserved */
+#define F2FS_FEATURE_VERITY 0x0400
+#define F2FS_FEATURE_SB_CHKSUM 0x0800
-#define F2FS_HAS_FEATURE(sb, mask) \
- ((F2FS_SB(sb)->raw_super->feature & cpu_to_le32(mask)) != 0)
-#define F2FS_SET_FEATURE(sb, mask) \
- (F2FS_SB(sb)->raw_super->feature |= cpu_to_le32(mask))
-#define F2FS_CLEAR_FEATURE(sb, mask) \
- (F2FS_SB(sb)->raw_super->feature &= ~cpu_to_le32(mask))
+#define __F2FS_HAS_FEATURE(raw_super, mask) \
+ ((raw_super->feature & cpu_to_le32(mask)) != 0)
+#define F2FS_HAS_FEATURE(sbi, mask) __F2FS_HAS_FEATURE(sbi->raw_super, mask)
+#define F2FS_SET_FEATURE(sbi, mask) \
+ (sbi->raw_super->feature |= cpu_to_le32(mask))
+#define F2FS_CLEAR_FEATURE(sbi, mask) \
+ (sbi->raw_super->feature &= ~cpu_to_le32(mask))
/*
* Default values for user and/or group using reserved blocks
@@ -176,16 +185,17 @@ enum {
#define CP_RECOVERY 0x00000008
#define CP_DISCARD 0x00000010
#define CP_TRIMMED 0x00000020
+#define CP_PAUSE 0x00000040
#define MAX_DISCARD_BLOCKS(sbi) BLKS_PER_SEC(sbi)
#define DEF_MAX_DISCARD_REQUEST 8 /* issue 8 discards per round */
-#define DEF_MAX_DISCARD_LEN 512 /* Max. 2MB per discard */
#define DEF_MIN_DISCARD_ISSUE_TIME 50 /* 50 ms, if exists */
#define DEF_MID_DISCARD_ISSUE_TIME 500 /* 500 ms, if device busy */
#define DEF_MAX_DISCARD_ISSUE_TIME 60000 /* 60 s, if no candidates */
#define DEF_DISCARD_URGENT_UTIL 80 /* do more discard over 80% */
#define DEF_CP_INTERVAL 60 /* 60 secs */
#define DEF_IDLE_INTERVAL 5 /* 5 secs */
+#define DEF_DISABLE_INTERVAL 5 /* 5 secs */
struct cp_control {
int reason;
@@ -195,14 +205,17 @@ struct cp_control {
};
/*
- * For CP/NAT/SIT/SSA readahead
+ * indicate meta/data type
*/
enum {
META_CP,
META_NAT,
META_SIT,
META_SSA,
+ META_MAX,
META_POR,
+ DATA_GENERIC,
+ META_GENERIC,
};
/* for the list of ino */
@@ -227,6 +240,12 @@ struct inode_entry {
struct inode *inode; /* vfs inode pointer */
};
+struct fsync_node_entry {
+ struct list_head list; /* list head */
+ struct page *page; /* warm node page pointer */
+ unsigned int seq_id; /* sequence id */
+};
+
/* for the bitmap indicate blocks to be discarded */
struct discard_entry {
struct list_head list; /* list head */
@@ -243,9 +262,10 @@ struct discard_entry {
(MAX_PLIST_NUM - 1) : (blk_num - 1))
enum {
- D_PREP,
- D_SUBMIT,
- D_DONE,
+ D_PREP, /* initial */
+ D_PARTIAL, /* partially submitted */
+ D_SUBMIT, /* all submitted */
+ D_DONE, /* finished */
};
struct discard_info {
@@ -270,7 +290,10 @@ struct discard_cmd {
struct block_device *bdev; /* bdev */
unsigned short ref; /* reference count */
unsigned char state; /* state */
+ unsigned char queued; /* queued discard */
int error; /* bio error */
+ spinlock_t lock; /* for state/bio_ref updating */
+ unsigned short bio_ref; /* bio reference count */
};
enum {
@@ -290,6 +313,7 @@ struct discard_policy {
unsigned int io_aware_gran; /* minimum granularity discard not be aware of I/O */
bool io_aware; /* issue discard in idle time */
bool sync; /* submit discard with REQ_SYNC flag */
+ bool ordered; /* issue discard by lba order */
unsigned int granularity; /* discard granularity */
};
@@ -306,10 +330,12 @@ struct discard_cmd_control {
unsigned int max_discards; /* max. discards to be issued */
unsigned int discard_granularity; /* discard granularity */
unsigned int undiscard_blks; /* # of undiscard blocks */
+ unsigned int next_pos; /* next discard position */
atomic_t issued_discard; /* # of issued discard */
- atomic_t issing_discard; /* # of issing discard */
+ atomic_t queued_discard; /* # of queued discard */
atomic_t discard_cmd_cnt; /* # of cached cmd count */
struct rb_root root; /* root of discard rb-tree */
+ bool rbtree_check; /* config for consistence check */
};
/* for the list of fsync inodes, used only during recovery */
@@ -396,6 +422,7 @@ static inline bool __has_cursum_space(struct f2fs_journal *journal,
#define F2FS_GOING_DOWN_METASYNC 0x1 /* going down with metadata */
#define F2FS_GOING_DOWN_NOSYNC 0x2 /* going down */
#define F2FS_GOING_DOWN_METAFLUSH 0x3 /* going down with meta flush */
+#define F2FS_GOING_DOWN_NEED_FSCK 0x4 /* going down to trigger fsck */
#if defined(__KERNEL__) && defined(CONFIG_COMPAT)
/*
@@ -506,13 +533,15 @@ enum {
*/
};
+#define DEFAULT_RETRY_IO_COUNT 8 /* maximum retry read IO count */
+
+/* maximum retry quota flush count */
+#define DEFAULT_RETRY_QUOTA_FLUSH_COUNT 8
+
#define F2FS_LINK_MAX 0xffffffff /* maximum link count per file */
#define MAX_DIR_RA_PAGES 4 /* maximum ra pages of dir */
-/* vector size for gang look-up from extent cache that consists of radix tree */
-#define EXT_TREE_VEC_SIZE 64
-
/* for in-memory extent cache entry */
#define F2FS_MIN_EXTENT_LEN 64 /* minimum extent length */
@@ -532,16 +561,8 @@ struct extent_info {
};
struct extent_node {
- struct rb_node rb_node;
- union {
- struct {
- unsigned int fofs;
- unsigned int len;
- u32 blk;
- };
- struct extent_info ei; /* extent info */
-
- };
+ struct rb_node rb_node; /* rb node located in rb-tree */
+ struct extent_info ei; /* extent info */
struct list_head list; /* node in global extent list of sbi */
struct extent_tree *et; /* extent tree pointer */
};
@@ -554,6 +575,7 @@ struct extent_tree {
struct list_head list; /* to be used by sbi->zombie_list */
rwlock_t lock; /* protect extent info rb-tree */
atomic_t node_cnt; /* # of extent node in rb-tree*/
+ bool largest_updated; /* largest extent updated */
};
/*
@@ -575,6 +597,7 @@ struct f2fs_map_blocks {
pgoff_t *m_next_pgofs; /* point next possible non-hole pgofs */
pgoff_t *m_next_extent; /* point to next possible extent */
int m_seg_type;
+ bool m_may_create; /* indicate it is from write path */
};
/* for flag in get_data_block */
@@ -582,6 +605,7 @@ enum {
F2FS_GET_BLOCK_DEFAULT,
F2FS_GET_BLOCK_FIEMAP,
F2FS_GET_BLOCK_BMAP,
+ F2FS_GET_BLOCK_DIO,
F2FS_GET_BLOCK_PRE_DIO,
F2FS_GET_BLOCK_PRE_AIO,
F2FS_GET_BLOCK_PRECACHE,
@@ -596,7 +620,9 @@ enum {
#define FADVISE_ENC_NAME_BIT 0x08
#define FADVISE_KEEP_SIZE_BIT 0x10
#define FADVISE_HOT_BIT 0x20
-#define FADVISE_VERITY_BIT 0x40 /* reserved */
+#define FADVISE_VERITY_BIT 0x40
+
+#define FADVISE_MODIFIABLE_BITS (FADVISE_COLD_BIT | FADVISE_HOT_BIT)
#define file_is_cold(inode) is_file(inode, FADVISE_COLD_BIT)
#define file_wrong_pino(inode) is_file(inode, FADVISE_LOST_PINO_BIT)
@@ -614,18 +640,25 @@ enum {
#define file_is_hot(inode) is_file(inode, FADVISE_HOT_BIT)
#define file_set_hot(inode) set_file(inode, FADVISE_HOT_BIT)
#define file_clear_hot(inode) clear_file(inode, FADVISE_HOT_BIT)
+#define file_is_verity(inode) is_file(inode, FADVISE_VERITY_BIT)
+#define file_set_verity(inode) set_file(inode, FADVISE_VERITY_BIT)
#define DEF_DIR_LEVEL 0
+enum {
+ GC_FAILURE_PIN,
+ GC_FAILURE_ATOMIC,
+ MAX_GC_FAILURE
+};
+
struct f2fs_inode_info {
struct inode vfs_inode; /* serve a vfs inode */
unsigned long i_flags; /* keep an inode flags for ioctl */
unsigned char i_advise; /* use to give file attribute hints */
unsigned char i_dir_level; /* use for dentry level for large dir */
- union {
- unsigned int i_current_depth; /* only for directory depth */
- unsigned short i_gc_failures; /* only for regular file */
- };
+ unsigned int i_current_depth; /* only for directory depth */
+ /* for gc failure statistic */
+ unsigned int i_gc_failures[MAX_GC_FAILURE];
unsigned int i_pino; /* parent inode number */
umode_t i_acl_mode; /* keep file acl mode temporarily */
@@ -653,15 +686,17 @@ struct f2fs_inode_info {
struct task_struct *inmem_task; /* store inmemory task */
struct mutex inmem_lock; /* lock for inmemory pages */
struct extent_tree *extent_tree; /* cached extent_tree entry */
- struct rw_semaphore dio_rwsem[2];/* avoid racing between dio and gc */
+
+ /* avoid racing between foreground op and gc */
+ struct rw_semaphore i_gc_rwsem[2];
struct rw_semaphore i_mmap_sem;
struct rw_semaphore i_xattr_sem; /* avoid racing between reading and changing EAs */
int i_extra_isize; /* size of extra space located in i_addr */
kprojid_t i_projid; /* id for project quota */
int i_inline_xattr_size; /* inline xattr size */
- struct timespec i_crtime; /* inode creation time */
- struct timespec i_disk_time[4]; /* inode disk times */
+ struct timespec64 i_crtime; /* inode creation time */
+ struct timespec64 i_disk_time[4];/* inode disk times */
};
static inline void get_extent_info(struct extent_info *ext,
@@ -689,22 +724,22 @@ static inline void set_extent_info(struct extent_info *ei, unsigned int fofs,
}
static inline bool __is_discard_mergeable(struct discard_info *back,
- struct discard_info *front)
+ struct discard_info *front, unsigned int max_len)
{
return (back->lstart + back->len == front->lstart) &&
- (back->len + front->len < DEF_MAX_DISCARD_LEN);
+ (back->len + front->len <= max_len);
}
static inline bool __is_discard_back_mergeable(struct discard_info *cur,
- struct discard_info *back)
+ struct discard_info *back, unsigned int max_len)
{
- return __is_discard_mergeable(back, cur);
+ return __is_discard_mergeable(back, cur, max_len);
}
static inline bool __is_discard_front_mergeable(struct discard_info *cur,
- struct discard_info *front)
+ struct discard_info *front, unsigned int max_len)
{
- return __is_discard_mergeable(cur, front);
+ return __is_discard_mergeable(cur, front, max_len);
}
static inline bool __is_extent_mergeable(struct extent_info *back,
@@ -727,12 +762,12 @@ static inline bool __is_front_mergeable(struct extent_info *cur,
}
extern void f2fs_mark_inode_dirty_sync(struct inode *inode, bool sync);
-static inline void __try_update_largest_extent(struct inode *inode,
- struct extent_tree *et, struct extent_node *en)
+static inline void __try_update_largest_extent(struct extent_tree *et,
+ struct extent_node *en)
{
if (en->ei.len > et->largest.len) {
et->largest = en->ei;
- f2fs_mark_inode_dirty_sync(inode, true);
+ et->largest_updated = true;
}
}
@@ -759,6 +794,7 @@ struct f2fs_nm_info {
struct radix_tree_root nat_set_root;/* root of the nat set cache */
struct rw_semaphore nat_tree_lock; /* protect nat_tree_lock */
struct list_head nat_entries; /* cached nat entry list (clean) */
+ spinlock_t nat_list_lock; /* protect clean nat entry list */
unsigned int nat_cnt; /* the # of cached nat entries */
unsigned int dirty_nat_cnt; /* total num of nat entries in set */
unsigned int nat_blocks; /* # of nat blocks */
@@ -852,7 +888,7 @@ struct flush_cmd_control {
struct task_struct *f2fs_issue_flush; /* flush thread */
wait_queue_head_t flush_wait_queue; /* waiting queue for wake-up */
atomic_t issued_flush; /* # of issued flushes */
- atomic_t issing_flush; /* # of issing flushes */
+ atomic_t queued_flush; /* # of queued flushes */
struct llist_head issue_list; /* list for command issue */
struct llist_node *dispatch_list; /* list for command dispatch */
};
@@ -916,6 +952,11 @@ enum count_type {
F2FS_DIRTY_IMETA,
F2FS_WB_CP_DATA,
F2FS_WB_DATA,
+ F2FS_RD_DATA,
+ F2FS_RD_NODE,
+ F2FS_RD_META,
+ F2FS_DIO_WRITE,
+ F2FS_DIO_READ,
NR_COUNT_TYPE,
};
@@ -1004,8 +1045,10 @@ struct f2fs_io_info {
int need_lock; /* indicate we need to lock cp_rwsem */
bool in_list; /* indicate fio is in io_list */
bool is_meta; /* indicate borrow meta inode mapping or not */
+ bool retry; /* need to reallocate block address */
enum iostat_type io_type; /* io type */
struct writeback_control *io_wbc; /* writeback control */
+ unsigned char version; /* version of the node */
};
#define is_read_io(rw) ((rw) == READ)
@@ -1019,6 +1062,10 @@ struct f2fs_bio_info {
struct list_head io_list; /* track fios */
};
+/* iv sector for security/pfe/pfk_fscrypt.c and f2fs */
+#define PG_DUN(i,p) \
+ ((((i)->i_ino & 0xffffffff) << 32) | ((p)->index & 0xffffffff))
+
#define FDEV(i) (sbi->devs[i])
#define RDEV(i) (raw_super->devs[i])
struct f2fs_dev_info {
@@ -1057,15 +1104,31 @@ enum {
SBI_POR_DOING, /* recovery is doing or not */
SBI_NEED_SB_WRITE, /* need to recover superblock */
SBI_NEED_CP, /* need to checkpoint */
+ SBI_IS_SHUTDOWN, /* shutdown by ioctl */
+ SBI_IS_RECOVERED, /* recovered orphan/data */
+ SBI_CP_DISABLED, /* CP was disabled last mount */
+ SBI_QUOTA_NEED_FLUSH, /* need to flush quota info in CP */
+ SBI_QUOTA_SKIP_FLUSH, /* skip flushing quota in current CP */
+ SBI_QUOTA_NEED_REPAIR, /* quota file may be corrupted */
};
enum {
CP_TIME,
REQ_TIME,
+ DISCARD_TIME,
+ GC_TIME,
+ DISABLE_TIME,
MAX_TIME,
};
enum {
+ GC_NORMAL,
+ GC_IDLE_CB,
+ GC_IDLE_GREEDY,
+ GC_URGENT,
+};
+
+enum {
WHINT_MODE_OFF, /* not pass down write hints */
WHINT_MODE_USER, /* try to pass down hints given by users */
WHINT_MODE_FS, /* pass down hints with F2FS policy */
@@ -1112,8 +1175,8 @@ struct f2fs_sb_info {
/* for bio operations */
struct f2fs_bio_info *write_io[NR_PAGE_TYPE]; /* for write bios */
- struct mutex wio_mutex[NR_PAGE_TYPE - 1][NR_TEMP_TYPE];
- /* bio ordering for NODE/DATA */
+ /* keep migration IO order for LFS mode */
+ struct rw_semaphore io_order_lock;
mempool_t *write_io_dummy; /* Dummy pages */
/* for checkpoint */
@@ -1131,6 +1194,11 @@ struct f2fs_sb_info {
struct inode_management im[MAX_INO_ENTRY]; /* manage inode cache */
+ spinlock_t fsync_node_lock; /* for node entry lock */
+ struct list_head fsync_node_list; /* node list head */
+ unsigned int fsync_seg_id; /* sequence id */
+ unsigned int fsync_node_num; /* number of node entries */
+
/* for orphan inode, use 0'th array */
unsigned int max_orphans; /* max orphan inodes */
@@ -1164,7 +1232,6 @@ struct f2fs_sb_info {
unsigned int total_valid_node_count; /* valid node block count */
loff_t max_file_blocks; /* max block index of file */
int dir_level; /* directory level */
- unsigned int trigger_ssr_threshold; /* threshold to trigger ssr */
int readdir_ra; /* readahead inode in readdir */
block_t user_block_count; /* # of user blocks */
@@ -1174,6 +1241,9 @@ struct f2fs_sb_info {
block_t reserved_blocks; /* configurable reserved blocks */
block_t current_reserved_blocks; /* current reserved blocks */
+ /* Additional tracking for no checkpoint mode */
+ block_t unusable_block_count; /* # of blocks saved by last cp */
+
unsigned int nquota_files; /* # of quota sysfile */
u32 s_next_generation; /* for NFS support */
@@ -1184,7 +1254,7 @@ struct f2fs_sb_info {
struct percpu_counter alloc_valid_block_count;
/* writeback control */
- atomic_t wb_sync_req; /* count # of WB_SYNC threads */
+ atomic_t wb_sync_req[META]; /* count # of WB_SYNC threads */
/* valid inode count */
struct percpu_counter total_valid_inode_count;
@@ -1195,15 +1265,19 @@ struct f2fs_sb_info {
struct mutex gc_mutex; /* mutex for GC */
struct f2fs_gc_kthread *gc_thread; /* GC thread */
unsigned int cur_victim_sec; /* current victim section num */
-
- /* threshold for converting bg victims for fg */
- u64 fggc_threshold;
+ unsigned int gc_mode; /* current GC state */
+ unsigned int next_victim_seg[2]; /* next segment in victim section */
+ /* for skip statistic */
+ unsigned long long skipped_atomic_files[2]; /* FG_GC and BG_GC */
+ unsigned long long skipped_gc_rwsem; /* FG_GC only */
/* threshold for gc trials on pinned files */
u64 gc_pin_file_threshold;
/* maximum # of trials to find a victim segment for SSR and GC */
unsigned int max_victim_search;
+ /* migration granularity of garbage collection, unit: segment */
+ unsigned int migration_granularity;
/*
* for stat information.
@@ -1211,6 +1285,7 @@ struct f2fs_sb_info {
*/
#ifdef CONFIG_F2FS_STAT_FS
struct f2fs_stat_info *stat_info; /* FS status information */
+ atomic_t meta_count[META_MAX]; /* # of meta blocks */
unsigned int segment_count[2]; /* # of allocated segments */
unsigned int block_count[2]; /* # of allocated blocks */
atomic_t inplace_count; /* # of inplace update */
@@ -1226,6 +1301,8 @@ struct f2fs_sb_info {
atomic_t max_aw_cnt; /* max # of atomic writes */
atomic_t max_vw_cnt; /* max # of volatile writes */
int bg_gc; /* background gc calls */
+ unsigned int io_skip_bggc; /* skip background gc for in-flight IO */
+ unsigned int other_skip_bggc; /* skip background gc for other reasons */
unsigned int ndirty_inode[NR_INODE_TYPE]; /* # of dirty inodes */
#endif
spinlock_t stat_lock; /* lock for stat operations */
@@ -1259,10 +1336,17 @@ struct f2fs_sb_info {
__u32 s_chksum_seed;
};
+struct f2fs_private_dio {
+ struct inode *inode;
+ void *orig_private;
+ bio_end_io_t *orig_end_io;
+ bool write;
+};
+
#ifdef CONFIG_F2FS_FAULT_INJECTION
-#define f2fs_show_injection_info(type) \
- printk("%sF2FS-fs : inject %s in %s of %pF\n", \
- KERN_INFO, fault_name[type], \
+#define f2fs_show_injection_info(type) \
+ printk_ratelimited("%sF2FS-fs : inject %s in %s of %pF\n", \
+ KERN_INFO, f2fs_fault_name[type], \
__func__, __builtin_return_address(0))
static inline bool time_to_inject(struct f2fs_sb_info *sbi, int type)
{
@@ -1281,6 +1365,12 @@ static inline bool time_to_inject(struct f2fs_sb_info *sbi, int type)
}
return false;
}
+#else
+#define f2fs_show_injection_info(type) do { } while (0)
+static inline bool time_to_inject(struct f2fs_sb_info *sbi, int type)
+{
+ return false;
+}
#endif
/* For write statistics. Suppose sector size is 512 bytes,
@@ -1292,7 +1382,15 @@ static inline bool time_to_inject(struct f2fs_sb_info *sbi, int type)
static inline void f2fs_update_time(struct f2fs_sb_info *sbi, int type)
{
- sbi->last_time[type] = jiffies;
+ unsigned long now = jiffies;
+
+ sbi->last_time[type] = now;
+
+ /* DISCARD_TIME and GC_TIME are based on REQ_TIME */
+ if (type == REQ_TIME) {
+ sbi->last_time[DISCARD_TIME] = now;
+ sbi->last_time[GC_TIME] = now;
+ }
}
static inline bool f2fs_time_over(struct f2fs_sb_info *sbi, int type)
@@ -1302,16 +1400,18 @@ static inline bool f2fs_time_over(struct f2fs_sb_info *sbi, int type)
return time_after(jiffies, sbi->last_time[type] + interval);
}
-static inline bool is_idle(struct f2fs_sb_info *sbi)
+static inline unsigned int f2fs_time_to_wait(struct f2fs_sb_info *sbi,
+ int type)
{
- struct block_device *bdev = sbi->sb->s_bdev;
- struct request_queue *q = bdev_get_queue(bdev);
- struct request_list *rl = &q->root_rl;
+ unsigned long interval = sbi->interval_time[type] * HZ;
+ unsigned int wait_ms = 0;
+ long delta;
- if (rl->count[BLK_RW_SYNC] || rl->count[BLK_RW_ASYNC])
- return 0;
+ delta = (sbi->last_time[type] + interval) - jiffies;
+ if (delta > 0)
+ wait_ms = jiffies_to_msecs(delta);
- return f2fs_time_over(sbi, REQ_TIME);
+ return wait_ms;
}
/*
@@ -1521,12 +1621,16 @@ static inline void disable_nat_bits(struct f2fs_sb_info *sbi, bool lock)
{
unsigned long flags;
- set_sbi_flag(sbi, SBI_NEED_FSCK);
+ /*
+ * In order to re-enable nat_bits we need to call fsck.f2fs by
+ * set_sbi_flag(sbi, SBI_NEED_FSCK). But it may give huge cost,
+ * so let's rely on regular fsck or unclean shutdown.
+ */
if (lock)
spin_lock_irqsave(&sbi->cp_lock, flags);
__clear_ckpt_flags(F2FS_CKPT(sbi), CP_NAT_BITS_FLAG);
- kfree(NM_I(sbi)->nat_bits);
+ kvfree(NM_I(sbi)->nat_bits);
NM_I(sbi)->nat_bits = NULL;
if (lock)
spin_unlock_irqrestore(&sbi->cp_lock, flags);
@@ -1588,18 +1692,6 @@ static inline bool __exist_node_summaries(struct f2fs_sb_info *sbi)
}
/*
- * Check whether the given nid is within node id range.
- */
-static inline int check_nid_range(struct f2fs_sb_info *sbi, nid_t nid)
-{
- if (unlikely(nid < F2FS_ROOT_INO(sbi)))
- return -EINVAL;
- if (unlikely(nid >= NM_I(sbi)->max_nid))
- return -EINVAL;
- return 0;
-}
-
-/*
* Check whether the inode has blocks or not
*/
static inline int F2FS_HAS_BLOCKS(struct inode *inode)
@@ -1645,13 +1737,12 @@ static inline int inc_valid_block_count(struct f2fs_sb_info *sbi,
if (ret)
return ret;
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(sbi, FAULT_BLOCK)) {
f2fs_show_injection_info(FAULT_BLOCK);
release = *count;
goto enospc;
}
-#endif
+
/*
* let's increase this in prior to actual block count change in order
* for f2fs_sync_file to avoid data races when deciding checkpoint.
@@ -1665,7 +1756,8 @@ static inline int inc_valid_block_count(struct f2fs_sb_info *sbi,
if (!__allow_reserved_blocks(sbi, inode, true))
avail_user_block_count -= F2FS_OPTION(sbi).root_reserved_blocks;
-
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
+ avail_user_block_count -= sbi->unusable_block_count;
if (unlikely(sbi->total_valid_block_count > avail_user_block_count)) {
diff = sbi->total_valid_block_count - avail_user_block_count;
if (diff > *count)
@@ -1675,18 +1767,20 @@ static inline int inc_valid_block_count(struct f2fs_sb_info *sbi,
sbi->total_valid_block_count -= diff;
if (!*count) {
spin_unlock(&sbi->stat_lock);
- percpu_counter_sub(&sbi->alloc_valid_block_count, diff);
goto enospc;
}
}
spin_unlock(&sbi->stat_lock);
- if (unlikely(release))
+ if (unlikely(release)) {
+ percpu_counter_sub(&sbi->alloc_valid_block_count, release);
dquot_release_reservation_block(inode, release);
+ }
f2fs_i_blocks_write(inode, *count, true, true);
return 0;
enospc:
+ percpu_counter_sub(&sbi->alloc_valid_block_count, release);
dquot_release_reservation_block(inode, release);
return -ENOSPC;
}
@@ -1714,7 +1808,9 @@ static inline void inc_page_count(struct f2fs_sb_info *sbi, int count_type)
atomic_inc(&sbi->nr_pages[count_type]);
if (count_type == F2FS_DIRTY_DATA || count_type == F2FS_INMEM_PAGES ||
- count_type == F2FS_WB_CP_DATA || count_type == F2FS_WB_DATA)
+ count_type == F2FS_WB_CP_DATA || count_type == F2FS_WB_DATA ||
+ count_type == F2FS_RD_DATA || count_type == F2FS_RD_NODE ||
+ count_type == F2FS_RD_META)
return;
set_sbi_flag(sbi, SBI_IS_DIRTY);
@@ -1850,20 +1946,24 @@ static inline int inc_valid_node_count(struct f2fs_sb_info *sbi,
{
block_t valid_block_count;
unsigned int valid_node_count;
- bool quota = inode && !is_inode;
+ int err;
- if (quota) {
- int ret = dquot_reserve_block(inode, 1);
- if (ret)
- return ret;
+ if (is_inode) {
+ if (inode) {
+ err = dquot_alloc_inode(inode);
+ if (err)
+ return err;
+ }
+ } else {
+ err = dquot_reserve_block(inode, 1);
+ if (err)
+ return err;
}
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(sbi, FAULT_BLOCK)) {
f2fs_show_injection_info(FAULT_BLOCK);
goto enospc;
}
-#endif
spin_lock(&sbi->stat_lock);
@@ -1872,6 +1972,8 @@ static inline int inc_valid_node_count(struct f2fs_sb_info *sbi,
if (!__allow_reserved_blocks(sbi, inode, false))
valid_block_count += F2FS_OPTION(sbi).root_reserved_blocks;
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
+ valid_block_count += sbi->unusable_block_count;
if (unlikely(valid_block_count > sbi->user_block_count)) {
spin_unlock(&sbi->stat_lock);
@@ -1899,8 +2001,12 @@ static inline int inc_valid_node_count(struct f2fs_sb_info *sbi,
return 0;
enospc:
- if (quota)
+ if (is_inode) {
+ if (inode)
+ dquot_free_inode(inode);
+ } else {
dquot_release_reservation_block(inode, 1);
+ }
return -ENOSPC;
}
@@ -1921,7 +2027,9 @@ static inline void dec_valid_node_count(struct f2fs_sb_info *sbi,
spin_unlock(&sbi->stat_lock);
- if (!is_inode)
+ if (is_inode)
+ dquot_free_inode(inode);
+ else
f2fs_i_blocks_write(inode, 1, false, true);
}
@@ -1948,17 +2056,23 @@ static inline s64 valid_inode_count(struct f2fs_sb_info *sbi)
static inline struct page *f2fs_grab_cache_page(struct address_space *mapping,
pgoff_t index, bool for_write)
{
-#ifdef CONFIG_F2FS_FAULT_INJECTION
- struct page *page = find_lock_page(mapping, index);
+ struct page *page;
- if (page)
- return page;
+ if (IS_ENABLED(CONFIG_F2FS_FAULT_INJECTION)) {
+ if (!for_write)
+ page = find_get_page_flags(mapping, index,
+ FGP_LOCK | FGP_ACCESSED);
+ else
+ page = find_lock_page(mapping, index);
+ if (page)
+ return page;
- if (time_to_inject(F2FS_M_SB(mapping), FAULT_PAGE_ALLOC)) {
- f2fs_show_injection_info(FAULT_PAGE_ALLOC);
- return NULL;
+ if (time_to_inject(F2FS_M_SB(mapping), FAULT_PAGE_ALLOC)) {
+ f2fs_show_injection_info(FAULT_PAGE_ALLOC);
+ return NULL;
+ }
}
-#endif
+
if (!for_write)
return grab_cache_page(mapping, index);
return grab_cache_page_write_begin(mapping, index, AOP_FLAG_NOFS);
@@ -1968,12 +2082,11 @@ static inline struct page *f2fs_pagecache_get_page(
struct address_space *mapping, pgoff_t index,
int fgp_flags, gfp_t gfp_mask)
{
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(F2FS_M_SB(mapping), FAULT_PAGE_GET)) {
f2fs_show_injection_info(FAULT_PAGE_GET);
return NULL;
}
-#endif
+
return pagecache_get_page(mapping, index, fgp_flags, gfp_mask);
}
@@ -2038,15 +2151,27 @@ static inline struct bio *f2fs_bio_alloc(struct f2fs_sb_info *sbi,
bio = bio_alloc(GFP_NOIO | __GFP_NOFAIL, npages);
return bio;
}
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(sbi, FAULT_ALLOC_BIO)) {
f2fs_show_injection_info(FAULT_ALLOC_BIO);
return NULL;
}
-#endif
+
return bio_alloc(GFP_KERNEL, npages);
}
+static inline bool is_idle(struct f2fs_sb_info *sbi, int type)
+{
+ if (get_pages(sbi, F2FS_RD_DATA) || get_pages(sbi, F2FS_RD_NODE) ||
+ get_pages(sbi, F2FS_RD_META) || get_pages(sbi, F2FS_WB_DATA) ||
+ get_pages(sbi, F2FS_WB_CP_DATA) ||
+ get_pages(sbi, F2FS_DIO_READ) ||
+ get_pages(sbi, F2FS_DIO_WRITE) ||
+ atomic_read(&SM_I(sbi)->dcc_info->queued_discard) ||
+ atomic_read(&SM_I(sbi)->fcc_info->queued_flush))
+ return false;
+ return f2fs_time_over(sbi, type);
+}
+
static inline void f2fs_radix_tree_insert(struct radix_tree_root *root,
unsigned long index, void *item)
{
@@ -2157,9 +2282,60 @@ static inline void f2fs_change_bit(unsigned int nr, char *addr)
*addr ^= mask;
}
-#define F2FS_REG_FLMASK (~(FS_DIRSYNC_FL | FS_TOPDIR_FL))
-#define F2FS_OTHER_FLMASK (FS_NODUMP_FL | FS_NOATIME_FL)
-#define F2FS_FL_INHERITED (FS_PROJINHERIT_FL)
+/*
+ * Inode flags
+ */
+#define F2FS_SECRM_FL 0x00000001 /* Secure deletion */
+#define F2FS_UNRM_FL 0x00000002 /* Undelete */
+#define F2FS_COMPR_FL 0x00000004 /* Compress file */
+#define F2FS_SYNC_FL 0x00000008 /* Synchronous updates */
+#define F2FS_IMMUTABLE_FL 0x00000010 /* Immutable file */
+#define F2FS_APPEND_FL 0x00000020 /* writes to file may only append */
+#define F2FS_NODUMP_FL 0x00000040 /* do not dump file */
+#define F2FS_NOATIME_FL 0x00000080 /* do not update atime */
+/* Reserved for compression usage... */
+#define F2FS_DIRTY_FL 0x00000100
+#define F2FS_COMPRBLK_FL 0x00000200 /* One or more compressed clusters */
+#define F2FS_NOCOMPR_FL 0x00000400 /* Don't compress */
+#define F2FS_ENCRYPT_FL 0x00000800 /* encrypted file */
+/* End compression flags --- maybe not all used */
+#define F2FS_INDEX_FL 0x00001000 /* hash-indexed directory */
+#define F2FS_IMAGIC_FL 0x00002000 /* AFS directory */
+#define F2FS_JOURNAL_DATA_FL 0x00004000 /* file data should be journaled */
+#define F2FS_NOTAIL_FL 0x00008000 /* file tail should not be merged */
+#define F2FS_DIRSYNC_FL 0x00010000 /* dirsync behaviour (directories only) */
+#define F2FS_TOPDIR_FL 0x00020000 /* Top of directory hierarchies*/
+#define F2FS_HUGE_FILE_FL 0x00040000 /* Set to each huge file */
+#define F2FS_EXTENTS_FL 0x00080000 /* Inode uses extents */
+#define F2FS_EA_INODE_FL 0x00200000 /* Inode used for large EA */
+#define F2FS_EOFBLOCKS_FL 0x00400000 /* Blocks allocated beyond EOF */
+#define F2FS_INLINE_DATA_FL 0x10000000 /* Inode has inline data. */
+#define F2FS_PROJINHERIT_FL 0x20000000 /* Create with parents projid */
+#define F2FS_RESERVED_FL 0x80000000 /* reserved for ext4 lib */
+
+#define F2FS_FL_USER_VISIBLE 0x304BDFFF /* User visible flags */
+#define F2FS_FL_USER_MODIFIABLE 0x204BC0FF /* User modifiable flags */
+
+/* Flags we can manipulate with through F2FS_IOC_FSSETXATTR */
+#define F2FS_FL_XFLAG_VISIBLE (F2FS_SYNC_FL | \
+ F2FS_IMMUTABLE_FL | \
+ F2FS_APPEND_FL | \
+ F2FS_NODUMP_FL | \
+ F2FS_NOATIME_FL | \
+ F2FS_PROJINHERIT_FL)
+
+/* Flags that should be inherited by new inodes from their parent. */
+#define F2FS_FL_INHERITED (F2FS_SECRM_FL | F2FS_UNRM_FL | F2FS_COMPR_FL |\
+ F2FS_SYNC_FL | F2FS_NODUMP_FL | F2FS_NOATIME_FL |\
+ F2FS_NOCOMPR_FL | F2FS_JOURNAL_DATA_FL |\
+ F2FS_NOTAIL_FL | F2FS_DIRSYNC_FL |\
+ F2FS_PROJINHERIT_FL)
+
+/* Flags that are appropriate for regular files (all but dir-specific ones). */
+#define F2FS_REG_FLMASK (~(F2FS_DIRSYNC_FL | F2FS_TOPDIR_FL))
+
+/* Flags that are appropriate for non-directories/regular files. */
+#define F2FS_OTHER_FLMASK (F2FS_NODUMP_FL | F2FS_NOATIME_FL)
static inline __u32 f2fs_mask_flags(umode_t mode, __u32 flags)
{
@@ -2202,6 +2378,7 @@ enum {
FI_EXTRA_ATTR, /* indicate file has extra attribute */
FI_PROJ_INHERIT, /* indicate file inherits projectid */
FI_PIN_FILE, /* indicate file should not be gced */
+ FI_ATOMIC_REVOKE_REQUEST, /* request to drop atomic data */
};
static inline void __mark_inode_dirty_flag(struct inode *inode,
@@ -2214,6 +2391,7 @@ static inline void __mark_inode_dirty_flag(struct inode *inode,
case FI_NEW_INODE:
if (set)
return;
+ /* fall through */
case FI_DATA_EXIST:
case FI_INLINE_DOTS:
case FI_PIN_FILE:
@@ -2300,7 +2478,7 @@ static inline void f2fs_i_depth_write(struct inode *inode, unsigned int depth)
static inline void f2fs_i_gc_failures_write(struct inode *inode,
unsigned int count)
{
- F2FS_I(inode)->i_gc_failures = count;
+ F2FS_I(inode)->i_gc_failures[GC_FAILURE_PIN] = count;
f2fs_mark_inode_dirty_sync(inode, true);
}
@@ -2476,13 +2654,13 @@ static inline bool f2fs_skip_inode_update(struct inode *inode, int dsync)
i_size_read(inode) & ~PAGE_MASK)
return false;
- if (!timespec_equal(F2FS_I(inode)->i_disk_time, &inode->i_atime))
+ if (!timespec64_equal(F2FS_I(inode)->i_disk_time, &inode->i_atime))
return false;
- if (!timespec_equal(F2FS_I(inode)->i_disk_time + 1, &inode->i_ctime))
+ if (!timespec64_equal(F2FS_I(inode)->i_disk_time + 1, &inode->i_ctime))
return false;
- if (!timespec_equal(F2FS_I(inode)->i_disk_time + 2, &inode->i_mtime))
+ if (!timespec64_equal(F2FS_I(inode)->i_disk_time + 2, &inode->i_mtime))
return false;
- if (!timespec_equal(F2FS_I(inode)->i_disk_time + 3,
+ if (!timespec64_equal(F2FS_I(inode)->i_disk_time + 3,
&F2FS_I(inode)->i_crtime))
return false;
@@ -2516,23 +2694,20 @@ static inline bool is_dot_dotdot(const struct qstr *str)
static inline bool f2fs_may_extent_tree(struct inode *inode)
{
- if (!test_opt(F2FS_I_SB(inode), EXTENT_CACHE) ||
+ struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+
+ if (!test_opt(sbi, EXTENT_CACHE) ||
is_inode_flag_set(inode, FI_NO_EXTENT))
return false;
- return S_ISREG(inode->i_mode);
-}
+ /*
+ * for recovered files during mount do not create extents
+ * if shrinker is not registered.
+ */
+ if (list_empty(&sbi->s_list))
+ return false;
-static inline void *f2fs_kmalloc(struct f2fs_sb_info *sbi,
- size_t size, gfp_t flags)
-{
-#ifdef CONFIG_F2FS_FAULT_INJECTION
- if (time_to_inject(sbi, FAULT_KMALLOC)) {
- f2fs_show_injection_info(FAULT_KMALLOC);
- return NULL;
- }
-#endif
- return kmalloc(size, flags);
+ return S_ISREG(inode->i_mode);
}
static inline void *kvmalloc(size_t size, gfp_t flags)
@@ -2545,6 +2720,23 @@ static inline void *kvmalloc(size_t size, gfp_t flags)
return ret;
}
+static inline void *f2fs_kmalloc(struct f2fs_sb_info *sbi,
+ size_t size, gfp_t flags)
+{
+ void *ret;
+
+ if (time_to_inject(sbi, FAULT_KMALLOC)) {
+ f2fs_show_injection_info(FAULT_KMALLOC);
+ return NULL;
+ }
+
+ ret = kmalloc(size, flags);
+ if (ret)
+ return ret;
+
+ return kvmalloc(size, flags);
+}
+
static inline void *kvzalloc(size_t size, gfp_t flags)
{
void *ret;
@@ -2574,12 +2766,11 @@ static inline void *f2fs_kzalloc(struct f2fs_sb_info *sbi,
static inline void *f2fs_kvmalloc(struct f2fs_sb_info *sbi,
size_t size, gfp_t flags)
{
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(sbi, FAULT_KVMALLOC)) {
f2fs_show_injection_info(FAULT_KVMALLOC);
return NULL;
}
-#endif
+
return kvmalloc(size, flags);
}
@@ -2599,7 +2790,7 @@ static inline int get_inline_xattr_addrs(struct inode *inode)
return F2FS_I(inode)->i_inline_xattr_size;
}
-#define get_inode_mode(i) \
+#define f2fs_get_inode_mode(i) \
((is_inode_flag_set(i, FI_ACL_MODE)) ? \
(F2FS_I(i)->i_acl_mode) : ((i)->i_mode))
@@ -2638,18 +2829,54 @@ static inline void f2fs_update_iostat(struct f2fs_sb_info *sbi,
spin_unlock(&sbi->iostat_lock);
}
+#define __is_large_section(sbi) ((sbi)->segs_per_sec > 1)
+
+#define __is_meta_io(fio) (PAGE_TYPE_OF_BIO(fio->type) == META && \
+ (!is_read_io(fio->op) || fio->is_meta))
+
+bool f2fs_is_valid_blkaddr(struct f2fs_sb_info *sbi,
+ block_t blkaddr, int type);
+void f2fs_msg(struct super_block *sb, const char *level, const char *fmt, ...);
+static inline void verify_blkaddr(struct f2fs_sb_info *sbi,
+ block_t blkaddr, int type)
+{
+ if (!f2fs_is_valid_blkaddr(sbi, blkaddr, type)) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "invalid blkaddr: %u, type: %d, run fsck to fix.",
+ blkaddr, type);
+ f2fs_bug_on(sbi, 1);
+ }
+}
+
+static inline bool __is_valid_data_blkaddr(block_t blkaddr)
+{
+ if (blkaddr == NEW_ADDR || blkaddr == NULL_ADDR)
+ return false;
+ return true;
+}
+
+static inline bool is_valid_data_blkaddr(struct f2fs_sb_info *sbi,
+ block_t blkaddr)
+{
+ if (!__is_valid_data_blkaddr(blkaddr))
+ return false;
+ verify_blkaddr(sbi, blkaddr, DATA_GENERIC);
+ return true;
+}
+
/*
* file.c
*/
int f2fs_sync_file(struct file *file, loff_t start, loff_t end, int datasync);
-void truncate_data_blocks(struct dnode_of_data *dn);
-int truncate_blocks(struct inode *inode, u64 from, bool lock);
+void f2fs_truncate_data_blocks(struct dnode_of_data *dn);
+int f2fs_truncate_blocks(struct inode *inode, u64 from, bool lock,
+ bool buf_write);
int f2fs_truncate(struct inode *inode);
int f2fs_getattr(struct vfsmount *mnt, struct dentry *dentry,
struct kstat *stat);
int f2fs_setattr(struct dentry *dentry, struct iattr *attr);
-int truncate_hole(struct inode *inode, pgoff_t pg_start, pgoff_t pg_end);
-void truncate_data_blocks_range(struct dnode_of_data *dn, int count);
+int f2fs_truncate_hole(struct inode *inode, pgoff_t pg_start, pgoff_t pg_end);
+void f2fs_truncate_data_blocks_range(struct dnode_of_data *dn, int count);
int f2fs_precache_extents(struct inode *inode);
long f2fs_ioctl(struct file *filp, unsigned int cmd, unsigned long arg);
long f2fs_compat_ioctl(struct file *file, unsigned int cmd, unsigned long arg);
@@ -2663,38 +2890,37 @@ bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct page *page);
void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct page *page);
struct inode *f2fs_iget(struct super_block *sb, unsigned long ino);
struct inode *f2fs_iget_retry(struct super_block *sb, unsigned long ino);
-int try_to_free_nats(struct f2fs_sb_info *sbi, int nr_shrink);
-void update_inode(struct inode *inode, struct page *node_page);
-void update_inode_page(struct inode *inode);
+int f2fs_try_to_free_nats(struct f2fs_sb_info *sbi, int nr_shrink);
+void f2fs_update_inode(struct inode *inode, struct page *node_page);
+void f2fs_update_inode_page(struct inode *inode);
int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc);
void f2fs_evict_inode(struct inode *inode);
-void handle_failed_inode(struct inode *inode);
+void f2fs_handle_failed_inode(struct inode *inode);
/*
* namei.c
*/
-int update_extension_list(struct f2fs_sb_info *sbi, const char *name,
+int f2fs_update_extension_list(struct f2fs_sb_info *sbi, const char *name,
bool hot, bool set);
struct dentry *f2fs_get_parent(struct dentry *child);
/*
* dir.c
*/
-void set_de_type(struct f2fs_dir_entry *de, umode_t mode);
-unsigned char get_de_type(struct f2fs_dir_entry *de);
-struct f2fs_dir_entry *find_target_dentry(struct fscrypt_name *fname,
+unsigned char f2fs_get_de_type(struct f2fs_dir_entry *de);
+struct f2fs_dir_entry *f2fs_find_target_dentry(struct fscrypt_name *fname,
f2fs_hash_t namehash, int *max_slots,
struct f2fs_dentry_ptr *d);
int f2fs_fill_dentries(struct dir_context *ctx, struct f2fs_dentry_ptr *d,
unsigned int start_pos, struct fscrypt_str *fstr);
-void do_make_empty_dir(struct inode *inode, struct inode *parent,
+void f2fs_do_make_empty_dir(struct inode *inode, struct inode *parent,
struct f2fs_dentry_ptr *d);
-struct page *init_inode_metadata(struct inode *inode, struct inode *dir,
+struct page *f2fs_init_inode_metadata(struct inode *inode, struct inode *dir,
const struct qstr *new_name,
const struct qstr *orig_name, struct page *dpage);
-void update_parent_metadata(struct inode *dir, struct inode *inode,
+void f2fs_update_parent_metadata(struct inode *dir, struct inode *inode,
unsigned int current_depth);
-int room_for_filename(const void *bitmap, int slots, int max_slots);
+int f2fs_room_for_filename(const void *bitmap, int slots, int max_slots);
void f2fs_drop_nlink(struct inode *dir, struct inode *inode);
struct f2fs_dir_entry *__f2fs_find_entry(struct inode *dir,
struct fscrypt_name *fname, struct page **res_page);
@@ -2711,9 +2937,9 @@ void f2fs_update_dentry(nid_t ino, umode_t mode, struct f2fs_dentry_ptr *d,
int f2fs_add_regular_entry(struct inode *dir, const struct qstr *new_name,
const struct qstr *orig_name,
struct inode *inode, nid_t ino, umode_t mode);
-int __f2fs_do_add_link(struct inode *dir, struct fscrypt_name *fname,
+int f2fs_add_dentry(struct inode *dir, struct fscrypt_name *fname,
struct inode *inode, nid_t ino, umode_t mode);
-int __f2fs_add_link(struct inode *dir, const struct qstr *name,
+int f2fs_do_add_link(struct inode *dir, const struct qstr *name,
struct inode *inode, nid_t ino, umode_t mode);
void f2fs_delete_entry(struct f2fs_dir_entry *dentry, struct page *page,
struct inode *dir, struct inode *inode);
@@ -2722,7 +2948,7 @@ bool f2fs_empty_dir(struct inode *dir);
static inline int f2fs_add_link(struct dentry *dentry, struct inode *inode)
{
- return __f2fs_add_link(d_inode(dentry->d_parent), &dentry->d_name,
+ return f2fs_do_add_link(d_inode(dentry->d_parent), &dentry->d_name,
inode, inode->i_ino, inode->i_mode);
}
@@ -2732,12 +2958,13 @@ static inline int f2fs_add_link(struct dentry *dentry, struct inode *inode)
int f2fs_inode_dirtied(struct inode *inode, bool sync);
void f2fs_inode_synced(struct inode *inode);
int f2fs_enable_quota_files(struct f2fs_sb_info *sbi, bool rdonly);
+int f2fs_quota_sync(struct super_block *sb, int type);
void f2fs_quota_off_umount(struct super_block *sb);
int f2fs_commit_super(struct f2fs_sb_info *sbi, bool recover);
int f2fs_sync_fs(struct super_block *sb, int sync);
extern __printf(3, 4)
void f2fs_msg(struct super_block *sb, const char *level, const char *fmt, ...);
-int sanity_check_ckpt(struct f2fs_sb_info *sbi);
+int f2fs_sanity_check_ckpt(struct f2fs_sb_info *sbi);
/*
* hash.c
@@ -2751,178 +2978,198 @@ f2fs_hash_t f2fs_dentry_hash(const struct qstr *name_info,
struct dnode_of_data;
struct node_info;
-bool available_free_memory(struct f2fs_sb_info *sbi, int type);
-int need_dentry_mark(struct f2fs_sb_info *sbi, nid_t nid);
-bool is_checkpointed_node(struct f2fs_sb_info *sbi, nid_t nid);
-bool need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino);
-void get_node_info(struct f2fs_sb_info *sbi, nid_t nid, struct node_info *ni);
-pgoff_t get_next_page_offset(struct dnode_of_data *dn, pgoff_t pgofs);
-int get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode);
-int truncate_inode_blocks(struct inode *inode, pgoff_t from);
-int truncate_xattr_node(struct inode *inode);
-int wait_on_node_pages_writeback(struct f2fs_sb_info *sbi, nid_t ino);
-int remove_inode_page(struct inode *inode);
-struct page *new_inode_page(struct inode *inode);
-struct page *new_node_page(struct dnode_of_data *dn, unsigned int ofs);
-void ra_node_page(struct f2fs_sb_info *sbi, nid_t nid);
-struct page *get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid);
-struct page *get_node_page_ra(struct page *parent, int start);
-void move_node_page(struct page *node_page, int gc_type);
-int fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
- struct writeback_control *wbc, bool atomic);
-int sync_node_pages(struct f2fs_sb_info *sbi, struct writeback_control *wbc,
+int f2fs_check_nid_range(struct f2fs_sb_info *sbi, nid_t nid);
+bool f2fs_available_free_memory(struct f2fs_sb_info *sbi, int type);
+bool f2fs_in_warm_node_list(struct f2fs_sb_info *sbi, struct page *page);
+void f2fs_init_fsync_node_info(struct f2fs_sb_info *sbi);
+void f2fs_del_fsync_node_entry(struct f2fs_sb_info *sbi, struct page *page);
+void f2fs_reset_fsync_node_info(struct f2fs_sb_info *sbi);
+int f2fs_need_dentry_mark(struct f2fs_sb_info *sbi, nid_t nid);
+bool f2fs_is_checkpointed_node(struct f2fs_sb_info *sbi, nid_t nid);
+bool f2fs_need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino);
+int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid,
+ struct node_info *ni);
+pgoff_t f2fs_get_next_page_offset(struct dnode_of_data *dn, pgoff_t pgofs);
+int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode);
+int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from);
+int f2fs_truncate_xattr_node(struct inode *inode);
+int f2fs_wait_on_node_pages_writeback(struct f2fs_sb_info *sbi,
+ unsigned int seq_id);
+int f2fs_remove_inode_page(struct inode *inode);
+struct page *f2fs_new_inode_page(struct inode *inode);
+struct page *f2fs_new_node_page(struct dnode_of_data *dn, unsigned int ofs);
+void f2fs_ra_node_page(struct f2fs_sb_info *sbi, nid_t nid);
+struct page *f2fs_get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid);
+struct page *f2fs_get_node_page_ra(struct page *parent, int start);
+int f2fs_move_node_page(struct page *node_page, int gc_type);
+int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
+ struct writeback_control *wbc, bool atomic,
+ unsigned int *seq_id);
+int f2fs_sync_node_pages(struct f2fs_sb_info *sbi,
+ struct writeback_control *wbc,
bool do_balance, enum iostat_type io_type);
-void build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount);
-bool alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid);
-void alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid);
-void alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid);
-int try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink);
-void recover_inline_xattr(struct inode *inode, struct page *page);
-int recover_xattr_data(struct inode *inode, struct page *page);
-int recover_inode_page(struct f2fs_sb_info *sbi, struct page *page);
-void restore_node_summary(struct f2fs_sb_info *sbi,
+int f2fs_build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount);
+bool f2fs_alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid);
+void f2fs_alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid);
+void f2fs_alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid);
+int f2fs_try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink);
+void f2fs_recover_inline_xattr(struct inode *inode, struct page *page);
+int f2fs_recover_xattr_data(struct inode *inode, struct page *page);
+int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct page *page);
+int f2fs_restore_node_summary(struct f2fs_sb_info *sbi,
unsigned int segno, struct f2fs_summary_block *sum);
-void flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc);
-int build_node_manager(struct f2fs_sb_info *sbi);
-void destroy_node_manager(struct f2fs_sb_info *sbi);
-int __init create_node_manager_caches(void);
-void destroy_node_manager_caches(void);
+int f2fs_flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc);
+int f2fs_build_node_manager(struct f2fs_sb_info *sbi);
+void f2fs_destroy_node_manager(struct f2fs_sb_info *sbi);
+int __init f2fs_create_node_manager_caches(void);
+void f2fs_destroy_node_manager_caches(void);
/*
* segment.c
*/
-bool need_SSR(struct f2fs_sb_info *sbi);
-void register_inmem_page(struct inode *inode, struct page *page);
-void drop_inmem_pages_all(struct f2fs_sb_info *sbi);
-void drop_inmem_pages(struct inode *inode);
-void drop_inmem_page(struct inode *inode, struct page *page);
-int commit_inmem_pages(struct inode *inode);
+bool f2fs_need_SSR(struct f2fs_sb_info *sbi);
+void f2fs_register_inmem_page(struct inode *inode, struct page *page);
+void f2fs_drop_inmem_pages_all(struct f2fs_sb_info *sbi, bool gc_failure);
+void f2fs_drop_inmem_pages(struct inode *inode);
+void f2fs_drop_inmem_page(struct inode *inode, struct page *page);
+int f2fs_commit_inmem_pages(struct inode *inode);
void f2fs_balance_fs(struct f2fs_sb_info *sbi, bool need);
void f2fs_balance_fs_bg(struct f2fs_sb_info *sbi);
int f2fs_issue_flush(struct f2fs_sb_info *sbi, nid_t ino);
-int create_flush_cmd_control(struct f2fs_sb_info *sbi);
+int f2fs_create_flush_cmd_control(struct f2fs_sb_info *sbi);
int f2fs_flush_device_cache(struct f2fs_sb_info *sbi);
-void destroy_flush_cmd_control(struct f2fs_sb_info *sbi, bool free);
-void invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr);
-bool is_checkpointed_data(struct f2fs_sb_info *sbi, block_t blkaddr);
-void drop_discard_cmd(struct f2fs_sb_info *sbi);
-void stop_discard_thread(struct f2fs_sb_info *sbi);
+void f2fs_destroy_flush_cmd_control(struct f2fs_sb_info *sbi, bool free);
+void f2fs_invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr);
+bool f2fs_is_checkpointed_data(struct f2fs_sb_info *sbi, block_t blkaddr);
+void f2fs_drop_discard_cmd(struct f2fs_sb_info *sbi);
+void f2fs_stop_discard_thread(struct f2fs_sb_info *sbi);
bool f2fs_wait_discard_bios(struct f2fs_sb_info *sbi);
-void clear_prefree_segments(struct f2fs_sb_info *sbi, struct cp_control *cpc);
-void release_discard_addrs(struct f2fs_sb_info *sbi);
-int npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra);
-void allocate_new_segments(struct f2fs_sb_info *sbi);
+void f2fs_clear_prefree_segments(struct f2fs_sb_info *sbi,
+ struct cp_control *cpc);
+void f2fs_dirty_to_prefree(struct f2fs_sb_info *sbi);
+int f2fs_disable_cp_again(struct f2fs_sb_info *sbi);
+void f2fs_release_discard_addrs(struct f2fs_sb_info *sbi);
+int f2fs_npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra);
+void f2fs_allocate_new_segments(struct f2fs_sb_info *sbi);
int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range);
-bool exist_trim_candidates(struct f2fs_sb_info *sbi, struct cp_control *cpc);
-struct page *get_sum_page(struct f2fs_sb_info *sbi, unsigned int segno);
-void update_meta_page(struct f2fs_sb_info *sbi, void *src, block_t blk_addr);
-void write_meta_page(struct f2fs_sb_info *sbi, struct page *page,
+bool f2fs_exist_trim_candidates(struct f2fs_sb_info *sbi,
+ struct cp_control *cpc);
+struct page *f2fs_get_sum_page(struct f2fs_sb_info *sbi, unsigned int segno);
+void f2fs_update_meta_page(struct f2fs_sb_info *sbi, void *src,
+ block_t blk_addr);
+void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct page *page,
enum iostat_type io_type);
-void write_node_page(unsigned int nid, struct f2fs_io_info *fio);
-void write_data_page(struct dnode_of_data *dn, struct f2fs_io_info *fio);
-int rewrite_data_page(struct f2fs_io_info *fio);
-void __f2fs_replace_block(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
+void f2fs_do_write_node_page(unsigned int nid, struct f2fs_io_info *fio);
+void f2fs_outplace_write_data(struct dnode_of_data *dn,
+ struct f2fs_io_info *fio);
+int f2fs_inplace_write_data(struct f2fs_io_info *fio);
+void f2fs_do_replace_block(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
block_t old_blkaddr, block_t new_blkaddr,
bool recover_curseg, bool recover_newaddr);
void f2fs_replace_block(struct f2fs_sb_info *sbi, struct dnode_of_data *dn,
block_t old_addr, block_t new_addr,
unsigned char version, bool recover_curseg,
bool recover_newaddr);
-void allocate_data_block(struct f2fs_sb_info *sbi, struct page *page,
+void f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct page *page,
block_t old_blkaddr, block_t *new_blkaddr,
struct f2fs_summary *sum, int type,
struct f2fs_io_info *fio, bool add_list);
void f2fs_wait_on_page_writeback(struct page *page,
- enum page_type type, bool ordered);
-void f2fs_wait_on_block_writeback(struct f2fs_sb_info *sbi, block_t blkaddr);
-void write_data_summaries(struct f2fs_sb_info *sbi, block_t start_blk);
-void write_node_summaries(struct f2fs_sb_info *sbi, block_t start_blk);
-int lookup_journal_in_cursum(struct f2fs_journal *journal, int type,
+ enum page_type type, bool ordered, bool locked);
+void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr);
+void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr,
+ block_t len);
+void f2fs_write_data_summaries(struct f2fs_sb_info *sbi, block_t start_blk);
+void f2fs_write_node_summaries(struct f2fs_sb_info *sbi, block_t start_blk);
+int f2fs_lookup_journal_in_cursum(struct f2fs_journal *journal, int type,
unsigned int val, int alloc);
-void flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc);
-int build_segment_manager(struct f2fs_sb_info *sbi);
-void destroy_segment_manager(struct f2fs_sb_info *sbi);
-int __init create_segment_manager_caches(void);
-void destroy_segment_manager_caches(void);
-int rw_hint_to_seg_type(enum rw_hint hint);
-enum rw_hint io_type_to_rw_hint(struct f2fs_sb_info *sbi, enum page_type type,
- enum temp_type temp);
+void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc);
+int f2fs_build_segment_manager(struct f2fs_sb_info *sbi);
+void f2fs_destroy_segment_manager(struct f2fs_sb_info *sbi);
+int __init f2fs_create_segment_manager_caches(void);
+void f2fs_destroy_segment_manager_caches(void);
+int f2fs_rw_hint_to_seg_type(enum rw_hint hint);
+enum rw_hint f2fs_io_type_to_rw_hint(struct f2fs_sb_info *sbi,
+ enum page_type type, enum temp_type temp);
/*
* checkpoint.c
*/
void f2fs_stop_checkpoint(struct f2fs_sb_info *sbi, bool end_io);
-struct page *grab_meta_page(struct f2fs_sb_info *sbi, pgoff_t index);
-struct page *get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index);
-struct page *get_tmp_page(struct f2fs_sb_info *sbi, pgoff_t index);
-bool is_valid_blkaddr(struct f2fs_sb_info *sbi, block_t blkaddr, int type);
-int ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages,
+struct page *f2fs_grab_meta_page(struct f2fs_sb_info *sbi, pgoff_t index);
+struct page *f2fs_get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index);
+struct page *f2fs_get_meta_page_nofail(struct f2fs_sb_info *sbi, pgoff_t index);
+struct page *f2fs_get_tmp_page(struct f2fs_sb_info *sbi, pgoff_t index);
+bool f2fs_is_valid_blkaddr(struct f2fs_sb_info *sbi,
+ block_t blkaddr, int type);
+int f2fs_ra_meta_pages(struct f2fs_sb_info *sbi, block_t start, int nrpages,
int type, bool sync);
-void ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index);
-long sync_meta_pages(struct f2fs_sb_info *sbi, enum page_type type,
+void f2fs_ra_meta_pages_cond(struct f2fs_sb_info *sbi, pgoff_t index);
+long f2fs_sync_meta_pages(struct f2fs_sb_info *sbi, enum page_type type,
long nr_to_write, enum iostat_type io_type);
-void add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type);
-void remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type);
-void release_ino_entry(struct f2fs_sb_info *sbi, bool all);
-bool exist_written_data(struct f2fs_sb_info *sbi, nid_t ino, int mode);
-void set_dirty_device(struct f2fs_sb_info *sbi, nid_t ino,
+void f2fs_add_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type);
+void f2fs_remove_ino_entry(struct f2fs_sb_info *sbi, nid_t ino, int type);
+void f2fs_release_ino_entry(struct f2fs_sb_info *sbi, bool all);
+bool f2fs_exist_written_data(struct f2fs_sb_info *sbi, nid_t ino, int mode);
+void f2fs_set_dirty_device(struct f2fs_sb_info *sbi, nid_t ino,
unsigned int devidx, int type);
-bool is_dirty_device(struct f2fs_sb_info *sbi, nid_t ino,
+bool f2fs_is_dirty_device(struct f2fs_sb_info *sbi, nid_t ino,
unsigned int devidx, int type);
int f2fs_sync_inode_meta(struct f2fs_sb_info *sbi);
-int acquire_orphan_inode(struct f2fs_sb_info *sbi);
-void release_orphan_inode(struct f2fs_sb_info *sbi);
-void add_orphan_inode(struct inode *inode);
-void remove_orphan_inode(struct f2fs_sb_info *sbi, nid_t ino);
-int recover_orphan_inodes(struct f2fs_sb_info *sbi);
-int get_valid_checkpoint(struct f2fs_sb_info *sbi);
-void update_dirty_page(struct inode *inode, struct page *page);
-void remove_dirty_inode(struct inode *inode);
-int sync_dirty_inodes(struct f2fs_sb_info *sbi, enum inode_type type);
-int write_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc);
-void init_ino_entry_info(struct f2fs_sb_info *sbi);
-int __init create_checkpoint_caches(void);
-void destroy_checkpoint_caches(void);
+int f2fs_acquire_orphan_inode(struct f2fs_sb_info *sbi);
+void f2fs_release_orphan_inode(struct f2fs_sb_info *sbi);
+void f2fs_add_orphan_inode(struct inode *inode);
+void f2fs_remove_orphan_inode(struct f2fs_sb_info *sbi, nid_t ino);
+int f2fs_recover_orphan_inodes(struct f2fs_sb_info *sbi);
+int f2fs_get_valid_checkpoint(struct f2fs_sb_info *sbi);
+void f2fs_update_dirty_page(struct inode *inode, struct page *page);
+void f2fs_remove_dirty_inode(struct inode *inode);
+int f2fs_sync_dirty_inodes(struct f2fs_sb_info *sbi, enum inode_type type);
+void f2fs_wait_on_all_pages_writeback(struct f2fs_sb_info *sbi);
+int f2fs_write_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc);
+void f2fs_init_ino_entry_info(struct f2fs_sb_info *sbi);
+int __init f2fs_create_checkpoint_caches(void);
+void f2fs_destroy_checkpoint_caches(void);
/*
* data.c
*/
+#define F2FS_GETPAGE_FOR_WRITE 0x01
+#define F2FS_GETPAGE_SKIP_VERITY 0x02
int f2fs_init_post_read_processing(void);
void f2fs_destroy_post_read_processing(void);
void f2fs_submit_merged_write(struct f2fs_sb_info *sbi, enum page_type type);
void f2fs_submit_merged_write_cond(struct f2fs_sb_info *sbi,
- struct inode *inode, nid_t ino, pgoff_t idx,
- enum page_type type);
+ struct inode *inode, struct page *page,
+ nid_t ino, enum page_type type);
void f2fs_flush_merged_writes(struct f2fs_sb_info *sbi);
int f2fs_submit_page_bio(struct f2fs_io_info *fio);
-int f2fs_submit_page_write(struct f2fs_io_info *fio);
+void f2fs_submit_page_write(struct f2fs_io_info *fio);
struct block_device *f2fs_target_device(struct f2fs_sb_info *sbi,
block_t blk_addr, struct bio *bio);
int f2fs_target_device_index(struct f2fs_sb_info *sbi, block_t blkaddr);
-void set_data_blkaddr(struct dnode_of_data *dn);
+void f2fs_set_data_blkaddr(struct dnode_of_data *dn);
void f2fs_update_data_blkaddr(struct dnode_of_data *dn, block_t blkaddr);
-int reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count);
-int reserve_new_block(struct dnode_of_data *dn);
+int f2fs_reserve_new_blocks(struct dnode_of_data *dn, blkcnt_t count);
+int f2fs_reserve_new_block(struct dnode_of_data *dn);
int f2fs_get_block(struct dnode_of_data *dn, pgoff_t index);
int f2fs_preallocate_blocks(struct kiocb *iocb, struct iov_iter *from);
int f2fs_reserve_block(struct dnode_of_data *dn, pgoff_t index);
-struct page *get_read_data_page(struct inode *inode, pgoff_t index,
- int op_flags, bool for_write);
-struct page *find_data_page(struct inode *inode, pgoff_t index);
-struct page *get_lock_data_page(struct inode *inode, pgoff_t index,
+struct page *f2fs_get_read_data_page(struct inode *inode, pgoff_t index,
+ int op_flags, int flags);
+struct page *f2fs_find_data_page(struct inode *inode, pgoff_t index, int flags);
+struct page *f2fs_get_lock_data_page(struct inode *inode, pgoff_t index,
bool for_write);
-struct page *get_new_data_page(struct inode *inode,
+struct page *f2fs_get_new_data_page(struct inode *inode,
struct page *ipage, pgoff_t index, bool new_i_size);
-int do_write_data_page(struct f2fs_io_info *fio);
+int f2fs_do_write_data_page(struct f2fs_io_info *fio);
+void __do_map_lock(struct f2fs_sb_info *sbi, int flag, bool lock);
int f2fs_map_blocks(struct inode *inode, struct f2fs_map_blocks *map,
int create, int flag);
int f2fs_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
u64 start, u64 len);
-bool should_update_inplace(struct inode *inode, struct f2fs_io_info *fio);
-bool should_update_outplace(struct inode *inode, struct f2fs_io_info *fio);
-int __f2fs_write_data_pages(struct address_space *mapping,
- struct writeback_control *wbc,
- enum iostat_type io_type);
+bool f2fs_should_update_inplace(struct inode *inode, struct f2fs_io_info *fio);
+bool f2fs_should_update_outplace(struct inode *inode, struct f2fs_io_info *fio);
void f2fs_invalidate_page(struct page *page, unsigned int offset,
unsigned int length);
int f2fs_release_page(struct page *page, gfp_t wait);
@@ -2931,22 +3178,23 @@ int f2fs_migrate_page(struct address_space *mapping, struct page *newpage,
struct page *page, enum migrate_mode mode);
#endif
bool f2fs_overwrite_io(struct inode *inode, loff_t pos, size_t len);
+void f2fs_clear_radix_tree_dirty_tag(struct page *page);
/*
* gc.c
*/
-int start_gc_thread(struct f2fs_sb_info *sbi);
-void stop_gc_thread(struct f2fs_sb_info *sbi);
-block_t start_bidx_of_node(unsigned int node_ofs, struct inode *inode);
+int f2fs_start_gc_thread(struct f2fs_sb_info *sbi);
+void f2fs_stop_gc_thread(struct f2fs_sb_info *sbi);
+block_t f2fs_start_bidx_of_node(unsigned int node_ofs, struct inode *inode);
int f2fs_gc(struct f2fs_sb_info *sbi, bool sync, bool background,
unsigned int segno);
-void build_gc_manager(struct f2fs_sb_info *sbi);
+void f2fs_build_gc_manager(struct f2fs_sb_info *sbi);
/*
* recovery.c
*/
-int recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only);
-bool space_for_roll_forward(struct f2fs_sb_info *sbi);
+int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only);
+bool f2fs_space_for_roll_forward(struct f2fs_sb_info *sbi);
/*
* debug.c
@@ -2968,6 +3216,9 @@ struct f2fs_stat_info {
int free_nids, avail_nids, alloc_nids;
int total_count, utilization;
int bg_gc, nr_wb_cp_data, nr_wb_data;
+ int nr_rd_data, nr_rd_node, nr_rd_meta;
+ int nr_dio_read, nr_dio_write;
+ unsigned int io_skip_bggc, other_skip_bggc;
int nr_flushing, nr_flushed, flush_list_empty;
int nr_discarding, nr_discarded;
int nr_discard_cmd;
@@ -2984,10 +3235,12 @@ struct f2fs_stat_info {
int bg_node_segs, bg_data_segs;
int tot_blks, data_blks, node_blks;
int bg_data_blks, bg_node_blks;
+ unsigned long long skipped_atomic_files[2];
int curseg[NR_CURSEG_TYPE];
int cursec[NR_CURSEG_TYPE];
int curzone[NR_CURSEG_TYPE];
+ unsigned int meta_count[META_MAX];
unsigned int segment_count[2];
unsigned int block_count[2];
unsigned int inplace_count;
@@ -3003,6 +3256,8 @@ static inline struct f2fs_stat_info *F2FS_STAT(struct f2fs_sb_info *sbi)
#define stat_inc_bg_cp_count(si) ((si)->bg_cp_count++)
#define stat_inc_call_count(si) ((si)->call_count++)
#define stat_inc_bggc_count(sbi) ((sbi)->bg_gc++)
+#define stat_io_skip_bggc_count(sbi) ((sbi)->io_skip_bggc++)
+#define stat_other_skip_bggc_count(sbi) ((sbi)->other_skip_bggc++)
#define stat_inc_dirty_inode(sbi, type) ((sbi)->ndirty_inode[type]++)
#define stat_dec_dirty_inode(sbi, type) ((sbi)->ndirty_inode[type]--)
#define stat_inc_total_hit(sbi) (atomic64_inc(&(sbi)->total_hit_ext))
@@ -3039,6 +3294,17 @@ static inline struct f2fs_stat_info *F2FS_STAT(struct f2fs_sb_info *sbi)
if (f2fs_has_inline_dentry(inode)) \
(atomic_dec(&F2FS_I_SB(inode)->inline_dir)); \
} while (0)
+#define stat_inc_meta_count(sbi, blkaddr) \
+ do { \
+ if (blkaddr < SIT_I(sbi)->sit_base_addr) \
+ atomic_inc(&(sbi)->meta_count[META_CP]); \
+ else if (blkaddr < NM_I(sbi)->nat_blkaddr) \
+ atomic_inc(&(sbi)->meta_count[META_SIT]); \
+ else if (blkaddr < SM_I(sbi)->ssa_blkaddr) \
+ atomic_inc(&(sbi)->meta_count[META_NAT]); \
+ else if (blkaddr < SM_I(sbi)->main_blkaddr) \
+ atomic_inc(&(sbi)->meta_count[META_SSA]); \
+ } while (0)
#define stat_inc_seg_type(sbi, curseg) \
((sbi)->segment_count[(curseg)->alloc_type]++)
#define stat_inc_block_count(sbi, curseg) \
@@ -3108,6 +3374,8 @@ void f2fs_destroy_root_stats(void);
#define stat_inc_bg_cp_count(si) do { } while (0)
#define stat_inc_call_count(si) do { } while (0)
#define stat_inc_bggc_count(si) do { } while (0)
+#define stat_io_skip_bggc_count(sbi) do { } while (0)
+#define stat_other_skip_bggc_count(sbi) do { } while (0)
#define stat_inc_dirty_inode(sbi, type) do { } while (0)
#define stat_dec_dirty_inode(sbi, type) do { } while (0)
#define stat_inc_total_hit(sb) do { } while (0)
@@ -3126,6 +3394,7 @@ void f2fs_destroy_root_stats(void);
#define stat_inc_volatile_write(inode) do { } while (0)
#define stat_dec_volatile_write(inode) do { } while (0)
#define stat_update_max_volatile_write(inode) do { } while (0)
+#define stat_inc_meta_count(sbi, blkaddr) do { } while (0)
#define stat_inc_seg_type(sbi, curseg) do { } while (0)
#define stat_inc_block_count(sbi, curseg) do { } while (0)
#define stat_inc_inplace_blocks(sbi) do { } while (0)
@@ -3150,29 +3419,31 @@ extern const struct inode_operations f2fs_dir_inode_operations;
extern const struct inode_operations f2fs_symlink_inode_operations;
extern const struct inode_operations f2fs_encrypted_symlink_inode_operations;
extern const struct inode_operations f2fs_special_inode_operations;
-extern struct kmem_cache *inode_entry_slab;
+extern struct kmem_cache *f2fs_inode_entry_slab;
/*
* inline.c
*/
bool f2fs_may_inline_data(struct inode *inode);
bool f2fs_may_inline_dentry(struct inode *inode);
-void read_inline_data(struct page *page, struct page *ipage);
-void truncate_inline_inode(struct inode *inode, struct page *ipage, u64 from);
+void f2fs_do_read_inline_data(struct page *page, struct page *ipage);
+void f2fs_truncate_inline_inode(struct inode *inode,
+ struct page *ipage, u64 from);
int f2fs_read_inline_data(struct inode *inode, struct page *page);
int f2fs_convert_inline_page(struct dnode_of_data *dn, struct page *page);
int f2fs_convert_inline_inode(struct inode *inode);
int f2fs_write_inline_data(struct inode *inode, struct page *page);
-bool recover_inline_data(struct inode *inode, struct page *npage);
-struct f2fs_dir_entry *find_in_inline_dir(struct inode *dir,
+bool f2fs_recover_inline_data(struct inode *inode, struct page *npage);
+struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir,
struct fscrypt_name *fname, struct page **res_page);
-int make_empty_inline_dir(struct inode *inode, struct inode *parent,
+int f2fs_make_empty_inline_dir(struct inode *inode, struct inode *parent,
struct page *ipage);
int f2fs_add_inline_entry(struct inode *dir, const struct qstr *new_name,
const struct qstr *orig_name,
struct inode *inode, nid_t ino, umode_t mode);
-void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, struct page *page,
- struct inode *dir, struct inode *inode);
+void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry,
+ struct page *page, struct inode *dir,
+ struct inode *inode);
bool f2fs_empty_inline_dir(struct inode *dir);
int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx,
struct fscrypt_str *fstr);
@@ -3193,17 +3464,17 @@ void f2fs_leave_shrinker(struct f2fs_sb_info *sbi);
/*
* extent_cache.c
*/
-struct rb_entry *__lookup_rb_tree(struct rb_root *root,
+struct rb_entry *f2fs_lookup_rb_tree(struct rb_root *root,
struct rb_entry *cached_re, unsigned int ofs);
-struct rb_node **__lookup_rb_tree_for_insert(struct f2fs_sb_info *sbi,
+struct rb_node **f2fs_lookup_rb_tree_for_insert(struct f2fs_sb_info *sbi,
struct rb_root *root, struct rb_node **parent,
unsigned int ofs);
-struct rb_entry *__lookup_rb_tree_ret(struct rb_root *root,
+struct rb_entry *f2fs_lookup_rb_tree_ret(struct rb_root *root,
struct rb_entry *cached_re, unsigned int ofs,
struct rb_entry **prev_entry, struct rb_entry **next_entry,
struct rb_node ***insert_p, struct rb_node **insert_parent,
bool force);
-bool __check_rb_tree_consistence(struct f2fs_sb_info *sbi,
+bool f2fs_check_rb_tree_consistence(struct f2fs_sb_info *sbi,
struct rb_root *root);
unsigned int f2fs_shrink_extent_tree(struct f2fs_sb_info *sbi, int nr_shrink);
bool f2fs_init_extent_tree(struct inode *inode, struct f2fs_extent *i_ext);
@@ -3215,9 +3486,9 @@ bool f2fs_lookup_extent_cache(struct inode *inode, pgoff_t pgofs,
void f2fs_update_extent_cache(struct dnode_of_data *dn);
void f2fs_update_extent_cache_range(struct dnode_of_data *dn,
pgoff_t fofs, block_t blkaddr, unsigned int len);
-void init_extent_cache_info(struct f2fs_sb_info *sbi);
-int __init create_extent_cache(void);
-void destroy_extent_cache(void);
+void f2fs_init_extent_cache_info(struct f2fs_sb_info *sbi);
+int __init f2fs_create_extent_cache(void);
+void f2fs_destroy_extent_cache(void);
/*
* sysfs.c
@@ -3244,23 +3515,28 @@ static inline void f2fs_set_encrypted_inode(struct inode *inode)
{
#ifdef CONFIG_F2FS_FS_ENCRYPTION
file_set_encrypt(inode);
- inode->i_flags |= S_ENCRYPTED;
+ f2fs_set_inode_flags(inode);
#endif
}
+static inline bool f2fs_verity_file(struct inode *inode)
+{
+ return file_is_verity(inode);
+}
+
/*
* Returns true if the reads of the inode's data need to undergo some
* postprocessing step, like decryption or authenticity verification.
*/
static inline bool f2fs_post_read_required(struct inode *inode)
{
- return f2fs_encrypted_file(inode);
+ return f2fs_encrypted_file(inode) || f2fs_verity_file(inode);
}
#define F2FS_FEATURE_FUNCS(name, flagname) \
-static inline int f2fs_sb_has_##name(struct super_block *sb) \
+static inline int f2fs_sb_has_##name(struct f2fs_sb_info *sbi) \
{ \
- return F2FS_HAS_FEATURE(sb, F2FS_FEATURE_##flagname); \
+ return F2FS_HAS_FEATURE(sbi, F2FS_FEATURE_##flagname); \
}
F2FS_FEATURE_FUNCS(encrypt, ENCRYPT);
@@ -3272,6 +3548,8 @@ F2FS_FEATURE_FUNCS(flexible_inline_xattr, FLEXIBLE_INLINE_XATTR);
F2FS_FEATURE_FUNCS(quota_ino, QUOTA_INO);
F2FS_FEATURE_FUNCS(inode_crtime, INODE_CRTIME);
F2FS_FEATURE_FUNCS(lost_found, LOST_FOUND);
+F2FS_FEATURE_FUNCS(verity, VERITY);
+F2FS_FEATURE_FUNCS(sb_chksum, SB_CHKSUM);
#ifdef CONFIG_BLK_DEV_ZONED
static inline int get_blkz_type(struct f2fs_sb_info *sbi,
@@ -3287,11 +3565,20 @@ static inline int get_blkz_type(struct f2fs_sb_info *sbi,
}
#endif
-static inline bool f2fs_discard_en(struct f2fs_sb_info *sbi)
+static inline bool f2fs_hw_should_discard(struct f2fs_sb_info *sbi)
{
- struct request_queue *q = bdev_get_queue(sbi->sb->s_bdev);
+ return f2fs_sb_has_blkzoned(sbi);
+}
- return blk_queue_discard(q) || f2fs_sb_has_blkzoned(sbi->sb);
+static inline bool f2fs_hw_support_discard(struct f2fs_sb_info *sbi)
+{
+ return blk_queue_discard(bdev_get_queue(sbi->sb->s_bdev));
+}
+
+static inline bool f2fs_realtime_discard_enable(struct f2fs_sb_info *sbi)
+{
+ return (test_opt(sbi, DISCARD) && f2fs_hw_support_discard(sbi)) ||
+ f2fs_hw_should_discard(sbi);
}
static inline void set_opt_mode(struct f2fs_sb_info *sbi, unsigned int mt)
@@ -3316,18 +3603,10 @@ static inline bool f2fs_may_encrypt(struct inode *inode)
return (S_ISREG(mode) || S_ISDIR(mode) || S_ISLNK(mode));
#else
- return 0;
+ return false;
#endif
}
-static inline bool f2fs_force_buffered_io(struct inode *inode, int rw)
-{
- return ((f2fs_post_read_required(inode) &&
- !fscrypt_using_hardware_encryption(inode)) ||
- (rw == WRITE && test_opt(F2FS_I_SB(inode), LFS)) ||
- F2FS_I_SB(inode)->s_ndevs);
-}
-
static inline bool f2fs_may_encrypt_bio(struct inode *inode,
struct f2fs_io_info *fio)
{
@@ -3338,4 +3617,72 @@ static inline bool f2fs_may_encrypt_bio(struct inode *inode,
fscrypt_using_hardware_encryption(inode));
}
+static inline int block_unaligned_IO(struct inode *inode,
+ struct kiocb *iocb, struct iov_iter *iter)
+{
+ unsigned int i_blkbits = READ_ONCE(inode->i_blkbits);
+ unsigned int blocksize_mask = (1 << i_blkbits) - 1;
+ loff_t offset = iocb->ki_pos;
+ unsigned long align = offset | iov_iter_alignment(iter);
+
+ return align & blocksize_mask;
+}
+
+static inline int allow_outplace_dio(struct inode *inode,
+ struct kiocb *iocb, struct iov_iter *iter)
+{
+ struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+ int rw = iov_iter_rw(iter);
+
+ return (test_opt(sbi, LFS) && (rw == WRITE) &&
+ !block_unaligned_IO(inode, iocb, iter));
+}
+
+static inline bool f2fs_force_buffered_io(struct inode *inode,
+ struct kiocb *iocb, struct iov_iter *iter)
+{
+ struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+ int rw = iov_iter_rw(iter);
+
+ if ((f2fs_encrypted_file(inode) &&
+ !fscrypt_using_hardware_encryption(inode)) ||
+ f2fs_verity_file(inode))
+ return true;
+ if (sbi->s_ndevs)
+ return true;
+ /*
+ * for blkzoned device, fallback direct IO to buffered IO, so
+ * all IOs can be serialized by log-structured write.
+ */
+ if (f2fs_sb_has_blkzoned(sbi))
+ return true;
+ if (test_opt(sbi, LFS) && (rw == WRITE) &&
+ block_unaligned_IO(inode, iocb, iter))
+ return true;
+ if (is_sbi_flag_set(F2FS_I_SB(inode), SBI_CP_DISABLED))
+ return true;
+
+ return false;
+}
+
+#ifdef CONFIG_F2FS_FAULT_INJECTION
+extern void f2fs_build_fault_attr(struct f2fs_sb_info *sbi, unsigned int rate,
+ unsigned int type);
+#else
+#define f2fs_build_fault_attr(sbi, rate, type) do { } while (0)
#endif
+
+#endif
+
+static inline bool is_journalled_quota(struct f2fs_sb_info *sbi)
+{
+#ifdef CONFIG_QUOTA
+ if (f2fs_sb_has_quota_ino(sbi))
+ return true;
+ if (F2FS_OPTION(sbi).s_qf_names[USRQUOTA] ||
+ F2FS_OPTION(sbi).s_qf_names[GRPQUOTA] ||
+ F2FS_OPTION(sbi).s_qf_names[PRJQUOTA])
+ return true;
+#endif
+ return false;
+}
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index 44a2e32..84c35b7 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/file.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
@@ -52,7 +49,7 @@ static int f2fs_vm_page_mkwrite(struct vm_area_struct *vma,
struct page *page = vmf->page;
struct inode *inode = file_inode(vma->vm_file);
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
- struct dnode_of_data dn;
+ struct dnode_of_data dn = { .node_changed = false };
int err;
if (unlikely(f2fs_cp_error(sbi))) {
@@ -64,19 +61,6 @@ static int f2fs_vm_page_mkwrite(struct vm_area_struct *vma,
f2fs_bug_on(sbi, f2fs_has_inline_data(inode));
- /* block allocation */
- f2fs_lock_op(sbi);
- set_new_dnode(&dn, inode, NULL, NULL, 0);
- err = f2fs_reserve_block(&dn, page->index);
- if (err) {
- f2fs_unlock_op(sbi);
- goto out;
- }
- f2fs_put_dnode(&dn);
- f2fs_unlock_op(sbi);
-
- f2fs_balance_fs(sbi, dn.node_changed);
-
file_update_time(vma->vm_file);
down_read(&F2FS_I(inode)->i_mmap_sem);
lock_page(page);
@@ -88,16 +72,34 @@ static int f2fs_vm_page_mkwrite(struct vm_area_struct *vma,
goto out_sem;
}
+ /* block allocation */
+ __do_map_lock(sbi, F2FS_GET_BLOCK_PRE_AIO, true);
+ set_new_dnode(&dn, inode, NULL, NULL, 0);
+ err = f2fs_get_block(&dn, page->index);
+ f2fs_put_dnode(&dn);
+ __do_map_lock(sbi, F2FS_GET_BLOCK_PRE_AIO, false);
+ if (err) {
+ unlock_page(page);
+ goto out_sem;
+ }
+
+ /* fill the page */
+ f2fs_wait_on_page_writeback(page, DATA, false, true);
+
+ /* wait for GCed page writeback via META_MAPPING */
+ f2fs_wait_on_block_writeback(inode, dn.data_blkaddr);
+
/*
* check to see if the page is mapped already (no holes)
*/
if (PageMappedToDisk(page))
- goto mapped;
+ goto out_sem;
/* page is wholly or partially inside EOF */
if (((loff_t)(page->index + 1) << PAGE_SHIFT) >
i_size_read(inode)) {
- unsigned offset;
+ loff_t offset;
+
offset = i_size_read(inode) & ~PAGE_MASK;
zero_user_segment(page, offset, PAGE_SIZE);
}
@@ -106,21 +108,15 @@ static int f2fs_vm_page_mkwrite(struct vm_area_struct *vma,
SetPageUptodate(page);
f2fs_update_iostat(sbi, APP_MAPPED_IO, F2FS_BLKSIZE);
+ f2fs_update_time(sbi, REQ_TIME);
trace_f2fs_vm_page_mkwrite(page, DATA);
-mapped:
- /* fill the page */
- f2fs_wait_on_page_writeback(page, DATA, false);
-
- /* wait for GCed page writeback via META_MAPPING */
- if (f2fs_post_read_required(inode))
- f2fs_wait_on_block_writeback(sbi, dn.data_blkaddr);
-
out_sem:
up_read(&F2FS_I(inode)->i_mmap_sem);
-out:
+
+ f2fs_balance_fs(sbi, dn.node_changed);
+
sb_end_pagefault(inode->i_sb);
- f2fs_update_time(sbi, REQ_TIME);
err:
return block_page_mkwrite_return(err);
}
@@ -159,17 +155,18 @@ static inline enum cp_reason_type need_do_checkpoint(struct inode *inode)
cp_reason = CP_SB_NEED_CP;
else if (file_wrong_pino(inode))
cp_reason = CP_WRONG_PINO;
- else if (!space_for_roll_forward(sbi))
+ else if (!f2fs_space_for_roll_forward(sbi))
cp_reason = CP_NO_SPC_ROLL;
- else if (!is_checkpointed_node(sbi, F2FS_I(inode)->i_pino))
+ else if (!f2fs_is_checkpointed_node(sbi, F2FS_I(inode)->i_pino))
cp_reason = CP_NODE_NEED_CP;
else if (test_opt(sbi, FASTBOOT))
cp_reason = CP_FASTBOOT_MODE;
else if (F2FS_OPTION(sbi).active_logs == 2)
cp_reason = CP_SPEC_LOG_NUM;
else if (F2FS_OPTION(sbi).fsync_mode == FSYNC_MODE_STRICT &&
- need_dentry_mark(sbi, inode->i_ino) &&
- exist_written_data(sbi, F2FS_I(inode)->i_pino, TRANS_DIR_INO))
+ f2fs_need_dentry_mark(sbi, inode->i_ino) &&
+ f2fs_exist_written_data(sbi, F2FS_I(inode)->i_pino,
+ TRANS_DIR_INO))
cp_reason = CP_RECOVER_DIR;
return cp_reason;
@@ -180,7 +177,7 @@ static bool need_inode_page_update(struct f2fs_sb_info *sbi, nid_t ino)
struct page *i = find_get_page(NODE_MAPPING(sbi), ino);
bool ret = false;
/* But we need to avoid that there are some inode updates */
- if ((i && PageDirty(i)) || need_inode_block_update(sbi, ino))
+ if ((i && PageDirty(i)) || f2fs_need_inode_block_update(sbi, ino))
ret = true;
f2fs_put_page(i, 0);
return ret;
@@ -213,12 +210,17 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
.nr_to_write = LONG_MAX,
.for_reclaim = 0,
};
+ unsigned int seq_id = 0;
- if (unlikely(f2fs_readonly(inode->i_sb)))
+ if (unlikely(f2fs_readonly(inode->i_sb) ||
+ is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
return 0;
trace_f2fs_sync_file_enter(inode);
+ if (S_ISDIR(inode->i_mode))
+ goto go_write;
+
/* if fdatasync is triggered, let's do in-place-update */
if (datasync || get_dirty_pages(inode) <= SM_I(sbi)->min_fsync_blocks)
set_inode_flag(inode, FI_NEED_IPU);
@@ -226,6 +228,9 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
clear_inode_flag(inode, FI_NEED_IPU);
if (ret) {
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "filemap_write failed %d dsync=%d atomic=%d",
+ ret, datasync, atomic);
trace_f2fs_sync_file_exit(inode, cp_reason, datasync, ret);
return ret;
}
@@ -240,14 +245,14 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
* if there is no written data, don't waste time to write recovery info.
*/
if (!is_inode_flag_set(inode, FI_APPEND_WRITE) &&
- !exist_written_data(sbi, ino, APPEND_INO)) {
+ !f2fs_exist_written_data(sbi, ino, APPEND_INO)) {
/* it may call write_inode just prior to fsync */
if (need_inode_page_update(sbi, ino))
goto go_write;
if (is_inode_flag_set(inode, FI_UPDATE_WRITE) ||
- exist_written_data(sbi, ino, UPDATE_INO))
+ f2fs_exist_written_data(sbi, ino, UPDATE_INO))
goto flush_out;
goto out;
}
@@ -263,6 +268,10 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
if (cp_reason) {
/* all the dirty node pages should be flushed for POR */
ret = f2fs_sync_fs(inode->i_sb, 1);
+ if (ret)
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "f2fs_sync_fs failed %d dsync=%d atomic=%d",
+ ret, datasync, atomic);
/*
* We've secured consistency through sync_fs. Following pino
@@ -274,9 +283,15 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
goto out;
}
sync_nodes:
- ret = fsync_node_pages(sbi, inode, &wbc, atomic);
- if (ret)
+ atomic_inc(&sbi->wb_sync_req[NODE]);
+ ret = f2fs_fsync_node_pages(sbi, inode, &wbc, atomic, &seq_id);
+ atomic_dec(&sbi->wb_sync_req[NODE]);
+ if (ret) {
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "fsync_node_pages failed %d dsync=%d atomic=%d",
+ ret, datasync, atomic);
goto out;
+ }
/* if cp_error was enabled, we should avoid infinite loop */
if (unlikely(f2fs_cp_error(sbi))) {
@@ -284,7 +299,7 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
goto out;
}
- if (need_inode_block_update(sbi, ino)) {
+ if (f2fs_need_inode_block_update(sbi, ino)) {
f2fs_mark_inode_dirty_sync(inode, true);
f2fs_write_inode(inode, NULL);
goto sync_nodes;
@@ -299,21 +314,26 @@ static int f2fs_do_sync_file(struct file *file, loff_t start, loff_t end,
* given fsync mark.
*/
if (!atomic) {
- ret = wait_on_node_pages_writeback(sbi, ino);
+ ret = f2fs_wait_on_node_pages_writeback(sbi, seq_id);
if (ret)
goto out;
}
/* once recovery info is written, don't need to tack this */
- remove_ino_entry(sbi, ino, APPEND_INO);
+ f2fs_remove_ino_entry(sbi, ino, APPEND_INO);
clear_inode_flag(inode, FI_APPEND_WRITE);
flush_out:
- if (!atomic && F2FS_OPTION(sbi).fsync_mode != FSYNC_MODE_NOBARRIER)
+ if (!atomic && F2FS_OPTION(sbi).fsync_mode != FSYNC_MODE_NOBARRIER) {
ret = f2fs_issue_flush(sbi, inode->i_ino);
+ if (ret)
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "f2fs_issue_flush failed %d dsync=%d atomic=%d",
+ ret, datasync, atomic);
+ }
if (!ret) {
- remove_ino_entry(sbi, ino, UPDATE_INO);
+ f2fs_remove_ino_entry(sbi, ino, UPDATE_INO);
clear_inode_flag(inode, FI_UPDATE_WRITE);
- remove_ino_entry(sbi, ino, FLUSH_INO);
+ f2fs_remove_ino_entry(sbi, ino, FLUSH_INO);
}
f2fs_update_time(sbi, REQ_TIME);
out:
@@ -332,28 +352,29 @@ int f2fs_sync_file(struct file *file, loff_t start, loff_t end, int datasync)
static pgoff_t __get_first_dirty_index(struct address_space *mapping,
pgoff_t pgofs, int whence)
{
- struct pagevec pvec;
+ struct page *page;
int nr_pages;
if (whence != SEEK_DATA)
return 0;
/* find first dirty page index */
- pagevec_init(&pvec, 0);
- nr_pages = pagevec_lookup_tag(&pvec, mapping, &pgofs,
- PAGECACHE_TAG_DIRTY, 1);
- pgofs = nr_pages ? pvec.pages[0]->index : ULONG_MAX;
- pagevec_release(&pvec);
+ nr_pages = find_get_pages_tag(mapping, &pgofs, PAGECACHE_TAG_DIRTY,
+ 1, &page);
+ if (!nr_pages)
+ return ULONG_MAX;
+ pgofs = page->index;
+ put_page(page);
return pgofs;
}
-static bool __found_offset(block_t blkaddr, pgoff_t dirty, pgoff_t pgofs,
- int whence)
+static bool __found_offset(struct f2fs_sb_info *sbi, block_t blkaddr,
+ pgoff_t dirty, pgoff_t pgofs, int whence)
{
switch (whence) {
case SEEK_DATA:
if ((blkaddr == NEW_ADDR && dirty == pgofs) ||
- (blkaddr != NEW_ADDR && blkaddr != NULL_ADDR))
+ is_valid_data_blkaddr(sbi, blkaddr))
return true;
break;
case SEEK_HOLE:
@@ -393,13 +414,13 @@ static loff_t f2fs_seek_block(struct file *file, loff_t offset, int whence)
for (; data_ofs < isize; data_ofs = (loff_t)pgofs << PAGE_SHIFT) {
set_new_dnode(&dn, inode, NULL, NULL, 0);
- err = get_dnode_of_data(&dn, pgofs, LOOKUP_NODE);
+ err = f2fs_get_dnode_of_data(&dn, pgofs, LOOKUP_NODE);
if (err && err != -ENOENT) {
goto fail;
} else if (err == -ENOENT) {
/* direct node does not exists */
if (whence == SEEK_DATA) {
- pgofs = get_next_page_offset(&dn, pgofs);
+ pgofs = f2fs_get_next_page_offset(&dn, pgofs);
continue;
} else {
goto found;
@@ -413,10 +434,19 @@ static loff_t f2fs_seek_block(struct file *file, loff_t offset, int whence)
dn.ofs_in_node++, pgofs++,
data_ofs = (loff_t)pgofs << PAGE_SHIFT) {
block_t blkaddr;
+
blkaddr = datablock_addr(dn.inode,
dn.node_page, dn.ofs_in_node);
- if (__found_offset(blkaddr, dirty, pgofs, whence)) {
+ if (__is_valid_data_blkaddr(blkaddr) &&
+ !f2fs_is_valid_blkaddr(F2FS_I_SB(inode),
+ blkaddr, DATA_GENERIC)) {
+ f2fs_put_dnode(&dn);
+ goto fail;
+ }
+
+ if (__found_offset(F2FS_I_SB(inode), blkaddr, dirty,
+ pgofs, whence)) {
f2fs_put_dnode(&dn);
goto found;
}
@@ -482,12 +512,18 @@ static int f2fs_file_open(struct inode *inode, struct file *filp)
if (err)
return err;
+ if (f2fs_verity_file(inode)) {
+ err = fsverity_file_open(inode, filp);
+ if (err)
+ return err;
+ }
+
filp->f_mode |= FMODE_NOWAIT;
return dquot_file_open(inode, filp);
}
-void truncate_data_blocks_range(struct dnode_of_data *dn, int count)
+void f2fs_truncate_data_blocks_range(struct dnode_of_data *dn, int count)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode);
struct f2fs_node *raw_node;
@@ -503,12 +539,18 @@ void truncate_data_blocks_range(struct dnode_of_data *dn, int count)
for (; count > 0; count--, addr++, dn->ofs_in_node++) {
block_t blkaddr = le32_to_cpu(*addr);
+
if (blkaddr == NULL_ADDR)
continue;
dn->data_blkaddr = NULL_ADDR;
- set_data_blkaddr(dn);
- invalidate_blocks(sbi, blkaddr);
+ f2fs_set_data_blkaddr(dn);
+
+ if (__is_valid_data_blkaddr(blkaddr) &&
+ !f2fs_is_valid_blkaddr(sbi, blkaddr, DATA_GENERIC))
+ continue;
+
+ f2fs_invalidate_blocks(sbi, blkaddr);
if (dn->ofs_in_node == 0 && IS_INODE(dn->node_page))
clear_inode_flag(dn->inode, FI_FIRST_BLOCK_WRITTEN);
nr_free++;
@@ -520,7 +562,7 @@ void truncate_data_blocks_range(struct dnode_of_data *dn, int count)
* once we invalidate valid blkaddr in range [ofs, ofs + count],
* we will invalidate all blkaddr in the whole range.
*/
- fofs = start_bidx_of_node(ofs_of_node(dn->node_page),
+ fofs = f2fs_start_bidx_of_node(ofs_of_node(dn->node_page),
dn->inode) + ofs;
f2fs_update_extent_cache_range(dn, fofs, 0, len);
dec_valid_block_count(sbi, dn->inode, nr_free);
@@ -532,15 +574,15 @@ void truncate_data_blocks_range(struct dnode_of_data *dn, int count)
dn->ofs_in_node, nr_free);
}
-void truncate_data_blocks(struct dnode_of_data *dn)
+void f2fs_truncate_data_blocks(struct dnode_of_data *dn)
{
- truncate_data_blocks_range(dn, ADDRS_PER_BLOCK);
+ f2fs_truncate_data_blocks_range(dn, ADDRS_PER_BLOCK);
}
static int truncate_partial_data_page(struct inode *inode, u64 from,
bool cache_only)
{
- unsigned offset = from & (PAGE_SIZE - 1);
+ loff_t offset = from & (PAGE_SIZE - 1);
pgoff_t index = from >> PAGE_SHIFT;
struct address_space *mapping = inode->i_mapping;
struct page *page;
@@ -556,11 +598,11 @@ static int truncate_partial_data_page(struct inode *inode, u64 from,
return 0;
}
- page = get_lock_data_page(inode, index, true);
+ page = f2fs_get_lock_data_page(inode, index, true);
if (IS_ERR(page))
return PTR_ERR(page) == -ENOENT ? 0 : PTR_ERR(page);
truncate_out:
- f2fs_wait_on_page_writeback(page, DATA, true);
+ f2fs_wait_on_page_writeback(page, DATA, true, true);
zero_user(page, offset, PAGE_SIZE - offset);
/* An encrypted inode should have a key and truncate the last page. */
@@ -571,7 +613,8 @@ static int truncate_partial_data_page(struct inode *inode, u64 from,
return 0;
}
-int truncate_blocks(struct inode *inode, u64 from, bool lock)
+int f2fs_truncate_blocks(struct inode *inode, u64 from, bool lock,
+ bool buf_write)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct dnode_of_data dn;
@@ -579,6 +622,7 @@ int truncate_blocks(struct inode *inode, u64 from, bool lock)
int count = 0, err = 0;
struct page *ipage;
bool truncate_page = false;
+ int flag = buf_write ? F2FS_GET_BLOCK_PRE_AIO : F2FS_GET_BLOCK_PRE_DIO;
trace_f2fs_truncate_blocks_enter(inode, from);
@@ -588,23 +632,23 @@ int truncate_blocks(struct inode *inode, u64 from, bool lock)
goto free_partial;
if (lock)
- f2fs_lock_op(sbi);
+ __do_map_lock(sbi, flag, true);
- ipage = get_node_page(sbi, inode->i_ino);
+ ipage = f2fs_get_node_page(sbi, inode->i_ino);
if (IS_ERR(ipage)) {
err = PTR_ERR(ipage);
goto out;
}
if (f2fs_has_inline_data(inode)) {
- truncate_inline_inode(inode, ipage, from);
+ f2fs_truncate_inline_inode(inode, ipage, from);
f2fs_put_page(ipage, 1);
truncate_page = true;
goto out;
}
set_new_dnode(&dn, inode, ipage, NULL, 0);
- err = get_dnode_of_data(&dn, free_from, LOOKUP_NODE_RA);
+ err = f2fs_get_dnode_of_data(&dn, free_from, LOOKUP_NODE_RA);
if (err) {
if (err == -ENOENT)
goto free_next;
@@ -617,16 +661,16 @@ int truncate_blocks(struct inode *inode, u64 from, bool lock)
f2fs_bug_on(sbi, count < 0);
if (dn.ofs_in_node || IS_INODE(dn.node_page)) {
- truncate_data_blocks_range(&dn, count);
+ f2fs_truncate_data_blocks_range(&dn, count);
free_from += count;
}
f2fs_put_dnode(&dn);
free_next:
- err = truncate_inode_blocks(inode, free_from);
+ err = f2fs_truncate_inode_blocks(inode, free_from);
out:
if (lock)
- f2fs_unlock_op(sbi);
+ __do_map_lock(sbi, flag, false);
free_partial:
/* lastly zero out the first data page */
if (!err)
@@ -649,12 +693,11 @@ int f2fs_truncate(struct inode *inode)
trace_f2fs_truncate(inode);
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(F2FS_I_SB(inode), FAULT_TRUNCATE)) {
f2fs_show_injection_info(FAULT_TRUNCATE);
return -EIO;
}
-#endif
+
/* we should check inline_data size */
if (!f2fs_may_inline_data(inode)) {
err = f2fs_convert_inline_inode(inode);
@@ -662,7 +705,7 @@ int f2fs_truncate(struct inode *inode)
return err;
}
- err = truncate_blocks(inode, i_size_read(inode), true);
+ err = f2fs_truncate_blocks(inode, i_size_read(inode), true, false);
if (err)
return err;
@@ -681,23 +724,23 @@ int f2fs_getattr(struct vfsmount *mnt,
unsigned int flags;
if (f2fs_has_extra_attr(inode) &&
- f2fs_sb_has_inode_crtime(inode->i_sb) &&
+ f2fs_sb_has_inode_crtime(F2FS_I_SB(inode)) &&
F2FS_FITS_IN_INODE(ri, fi->i_extra_isize, i_crtime)) {
stat->result_mask |= STATX_BTIME;
stat->btime.tv_sec = fi->i_crtime.tv_sec;
stat->btime.tv_nsec = fi->i_crtime.tv_nsec;
}
- flags = fi->i_flags & (FS_FL_USER_VISIBLE | FS_PROJINHERIT_FL);
- if (flags & FS_APPEND_FL)
+ flags = fi->i_flags & F2FS_FL_USER_VISIBLE;
+ if (flags & F2FS_APPEND_FL)
stat->attributes |= STATX_ATTR_APPEND;
- if (flags & FS_COMPR_FL)
+ if (flags & F2FS_COMPR_FL)
stat->attributes |= STATX_ATTR_COMPRESSED;
if (f2fs_encrypted_inode(inode))
stat->attributes |= STATX_ATTR_ENCRYPTED;
- if (flags & FS_IMMUTABLE_FL)
+ if (flags & F2FS_IMMUTABLE_FL)
stat->attributes |= STATX_ATTR_IMMUTABLE;
- if (flags & FS_NODUMP_FL)
+ if (flags & F2FS_NODUMP_FL)
stat->attributes |= STATX_ATTR_NODUMP;
stat->attributes_mask |= (STATX_ATTR_APPEND |
@@ -706,6 +749,23 @@ int f2fs_getattr(struct vfsmount *mnt,
STATX_ATTR_IMMUTABLE |
STATX_ATTR_NODUMP);
#endif
+
+ if (f2fs_verity_file(inode)) {
+ /*
+ * For fs-verity we need to override i_size with the original
+ * data i_size. This requires I/O to the file which with
+ * fscrypt requires that the key be set up. But, if the key is
+ * unavailable just continue on without the i_size override.
+ */
+ int err = fscrypt_require_key(inode);
+
+ if (err != -ENOKEY) {
+ err = fsverity_prepare_getattr(inode);
+ if (err)
+ return err;
+ }
+ }
+
generic_fillattr(inode, stat);
/* we need to show initial sectors used for inline_data/dentries */
@@ -763,6 +823,12 @@ int f2fs_setattr(struct dentry *dentry, struct iattr *attr)
if (err)
return err;
+ if (f2fs_verity_file(inode)) {
+ err = fsverity_prepare_setattr(dentry, attr);
+ if (err)
+ return err;
+ }
+
if (is_quota_modification(inode, attr)) {
err = dquot_initialize(inode);
if (err)
@@ -772,28 +838,47 @@ int f2fs_setattr(struct dentry *dentry, struct iattr *attr)
!uid_eq(attr->ia_uid, inode->i_uid)) ||
(attr->ia_valid & ATTR_GID &&
!gid_eq(attr->ia_gid, inode->i_gid))) {
+ f2fs_lock_op(F2FS_I_SB(inode));
err = dquot_transfer(inode, attr);
- if (err)
+ if (err) {
+ set_sbi_flag(F2FS_I_SB(inode),
+ SBI_QUOTA_NEED_REPAIR);
+ f2fs_unlock_op(F2FS_I_SB(inode));
return err;
+ }
+ /*
+ * update uid/gid under lock_op(), so that dquot and inode can
+ * be updated atomically.
+ */
+ if (attr->ia_valid & ATTR_UID)
+ inode->i_uid = attr->ia_uid;
+ if (attr->ia_valid & ATTR_GID)
+ inode->i_gid = attr->ia_gid;
+ f2fs_mark_inode_dirty_sync(inode, true);
+ f2fs_unlock_op(F2FS_I_SB(inode));
}
if (attr->ia_valid & ATTR_SIZE) {
- if (attr->ia_size <= i_size_read(inode)) {
- down_write(&F2FS_I(inode)->i_mmap_sem);
- truncate_setsize(inode, attr->ia_size);
- err = f2fs_truncate(inode);
- up_write(&F2FS_I(inode)->i_mmap_sem);
- if (err)
- return err;
- } else {
- /*
- * do not trim all blocks after i_size if target size is
- * larger than i_size.
- */
- down_write(&F2FS_I(inode)->i_mmap_sem);
- truncate_setsize(inode, attr->ia_size);
- up_write(&F2FS_I(inode)->i_mmap_sem);
+ bool to_smaller = (attr->ia_size <= i_size_read(inode));
+ down_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
+ down_write(&F2FS_I(inode)->i_mmap_sem);
+
+ truncate_setsize(inode, attr->ia_size);
+
+ if (to_smaller)
+ err = f2fs_truncate(inode);
+ /*
+ * do not trim all blocks after i_size if target size is
+ * larger than i_size.
+ */
+ up_write(&F2FS_I(inode)->i_mmap_sem);
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
+
+ if (err)
+ return err;
+
+ if (!to_smaller) {
/* should convert inline inode here */
if (!f2fs_may_inline_data(inode)) {
err = f2fs_convert_inline_inode(inode);
@@ -813,7 +898,7 @@ int f2fs_setattr(struct dentry *dentry, struct iattr *attr)
__setattr_copy(inode, attr);
if (attr->ia_valid & ATTR_MODE) {
- err = posix_acl_chmod(inode, get_inode_mode(inode));
+ err = posix_acl_chmod(inode, f2fs_get_inode_mode(inode));
if (err || is_inode_flag_set(inode, FI_ACL_MODE)) {
inode->i_mode = F2FS_I(inode)->i_acl_mode;
clear_inode_flag(inode, FI_ACL_MODE);
@@ -852,20 +937,20 @@ static int fill_zero(struct inode *inode, pgoff_t index,
f2fs_balance_fs(sbi, true);
f2fs_lock_op(sbi);
- page = get_new_data_page(inode, NULL, index, false);
+ page = f2fs_get_new_data_page(inode, NULL, index, false);
f2fs_unlock_op(sbi);
if (IS_ERR(page))
return PTR_ERR(page);
- f2fs_wait_on_page_writeback(page, DATA, true);
+ f2fs_wait_on_page_writeback(page, DATA, true, true);
zero_user(page, start, len);
set_page_dirty(page);
f2fs_put_page(page, 1);
return 0;
}
-int truncate_hole(struct inode *inode, pgoff_t pg_start, pgoff_t pg_end)
+int f2fs_truncate_hole(struct inode *inode, pgoff_t pg_start, pgoff_t pg_end)
{
int err;
@@ -874,10 +959,11 @@ int truncate_hole(struct inode *inode, pgoff_t pg_start, pgoff_t pg_end)
pgoff_t end_offset, count;
set_new_dnode(&dn, inode, NULL, NULL, 0);
- err = get_dnode_of_data(&dn, pg_start, LOOKUP_NODE);
+ err = f2fs_get_dnode_of_data(&dn, pg_start, LOOKUP_NODE);
if (err) {
if (err == -ENOENT) {
- pg_start = get_next_page_offset(&dn, pg_start);
+ pg_start = f2fs_get_next_page_offset(&dn,
+ pg_start);
continue;
}
return err;
@@ -888,7 +974,7 @@ int truncate_hole(struct inode *inode, pgoff_t pg_start, pgoff_t pg_end)
f2fs_bug_on(F2FS_I_SB(inode), count == 0 || count > end_offset);
- truncate_data_blocks_range(&dn, count);
+ f2fs_truncate_data_blocks_range(&dn, count);
f2fs_put_dnode(&dn);
pg_start += count;
@@ -939,14 +1025,19 @@ static int punch_hole(struct inode *inode, loff_t offset, loff_t len)
blk_start = (loff_t)pg_start << PAGE_SHIFT;
blk_end = (loff_t)pg_end << PAGE_SHIFT;
+
+ down_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
down_write(&F2FS_I(inode)->i_mmap_sem);
+
truncate_inode_pages_range(mapping, blk_start,
blk_end - 1);
f2fs_lock_op(sbi);
- ret = truncate_hole(inode, pg_start, pg_end);
+ ret = f2fs_truncate_hole(inode, pg_start, pg_end);
f2fs_unlock_op(sbi);
+
up_write(&F2FS_I(inode)->i_mmap_sem);
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
}
}
@@ -962,7 +1053,7 @@ static int __read_out_blkaddrs(struct inode *inode, block_t *blkaddr,
next_dnode:
set_new_dnode(&dn, inode, NULL, NULL, 0);
- ret = get_dnode_of_data(&dn, off, LOOKUP_NODE_RA);
+ ret = f2fs_get_dnode_of_data(&dn, off, LOOKUP_NODE_RA);
if (ret && ret != -ENOENT) {
return ret;
} else if (ret == -ENOENT) {
@@ -979,7 +1070,7 @@ static int __read_out_blkaddrs(struct inode *inode, block_t *blkaddr,
for (i = 0; i < done; i++, blkaddr++, do_replace++, dn.ofs_in_node++) {
*blkaddr = datablock_addr(dn.inode,
dn.node_page, dn.ofs_in_node);
- if (!is_checkpointed_data(sbi, *blkaddr)) {
+ if (!f2fs_is_checkpointed_data(sbi, *blkaddr)) {
if (test_opt(sbi, LFS)) {
f2fs_put_dnode(&dn);
@@ -1012,10 +1103,10 @@ static int __roll_back_blkaddrs(struct inode *inode, block_t *blkaddr,
continue;
set_new_dnode(&dn, inode, NULL, NULL, 0);
- ret = get_dnode_of_data(&dn, off + i, LOOKUP_NODE_RA);
+ ret = f2fs_get_dnode_of_data(&dn, off + i, LOOKUP_NODE_RA);
if (ret) {
dec_valid_block_count(sbi, inode, 1);
- invalidate_blocks(sbi, *blkaddr);
+ f2fs_invalidate_blocks(sbi, *blkaddr);
} else {
f2fs_update_data_blkaddr(&dn, *blkaddr);
}
@@ -1045,18 +1136,23 @@ static int __clone_blkaddrs(struct inode *src_inode, struct inode *dst_inode,
pgoff_t ilen;
set_new_dnode(&dn, dst_inode, NULL, NULL, 0);
- ret = get_dnode_of_data(&dn, dst + i, ALLOC_NODE);
+ ret = f2fs_get_dnode_of_data(&dn, dst + i, ALLOC_NODE);
if (ret)
return ret;
- get_node_info(sbi, dn.nid, &ni);
+ ret = f2fs_get_node_info(sbi, dn.nid, &ni);
+ if (ret) {
+ f2fs_put_dnode(&dn);
+ return ret;
+ }
+
ilen = min((pgoff_t)
ADDRS_PER_PAGE(dn.node_page, dst_inode) -
dn.ofs_in_node, len - i);
do {
dn.data_blkaddr = datablock_addr(dn.inode,
dn.node_page, dn.ofs_in_node);
- truncate_data_blocks_range(&dn, 1);
+ f2fs_truncate_data_blocks_range(&dn, 1);
if (do_replace[i]) {
f2fs_i_blocks_write(src_inode,
@@ -1079,10 +1175,11 @@ static int __clone_blkaddrs(struct inode *src_inode, struct inode *dst_inode,
} else {
struct page *psrc, *pdst;
- psrc = get_lock_data_page(src_inode, src + i, true);
+ psrc = f2fs_get_lock_data_page(src_inode,
+ src + i, true);
if (IS_ERR(psrc))
return PTR_ERR(psrc);
- pdst = get_new_data_page(dst_inode, NULL, dst + i,
+ pdst = f2fs_get_new_data_page(dst_inode, NULL, dst + i,
true);
if (IS_ERR(pdst)) {
f2fs_put_page(psrc, 1);
@@ -1093,7 +1190,8 @@ static int __clone_blkaddrs(struct inode *src_inode, struct inode *dst_inode,
f2fs_put_page(pdst, 1);
f2fs_put_page(psrc, 1);
- ret = truncate_hole(src_inode, src + i, src + i + 1);
+ ret = f2fs_truncate_hole(src_inode,
+ src + i, src + i + 1);
if (ret)
return ret;
i++;
@@ -1115,12 +1213,14 @@ static int __exchange_data_block(struct inode *src_inode,
olen = min((pgoff_t)4 * ADDRS_PER_BLOCK, len);
src_blkaddr = f2fs_kvzalloc(F2FS_I_SB(src_inode),
- sizeof(block_t) * olen, GFP_KERNEL);
+ array_size(olen, sizeof(block_t)),
+ GFP_KERNEL);
if (!src_blkaddr)
return -ENOMEM;
do_replace = f2fs_kvzalloc(F2FS_I_SB(src_inode),
- sizeof(int) * olen, GFP_KERNEL);
+ array_size(olen, sizeof(int)),
+ GFP_KERNEL);
if (!do_replace) {
kvfree(src_blkaddr);
return -ENOMEM;
@@ -1146,31 +1246,39 @@ static int __exchange_data_block(struct inode *src_inode,
return 0;
roll_back:
- __roll_back_blkaddrs(src_inode, src_blkaddr, do_replace, src, len);
+ __roll_back_blkaddrs(src_inode, src_blkaddr, do_replace, src, olen);
kvfree(src_blkaddr);
kvfree(do_replace);
return ret;
}
-static int f2fs_do_collapse(struct inode *inode, pgoff_t start, pgoff_t end)
+static int f2fs_do_collapse(struct inode *inode, loff_t offset, loff_t len)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
pgoff_t nrpages = (i_size_read(inode) + PAGE_SIZE - 1) / PAGE_SIZE;
+ pgoff_t start = offset >> PAGE_SHIFT;
+ pgoff_t end = (offset + len) >> PAGE_SHIFT;
int ret;
f2fs_balance_fs(sbi, true);
+
+ /* avoid gc operation during block exchange */
+ down_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
+ down_write(&F2FS_I(inode)->i_mmap_sem);
+
f2fs_lock_op(sbi);
-
f2fs_drop_extent_tree(inode);
-
+ truncate_pagecache(inode, offset);
ret = __exchange_data_block(inode, inode, end, start, nrpages - end, true);
f2fs_unlock_op(sbi);
+
+ up_write(&F2FS_I(inode)->i_mmap_sem);
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
return ret;
}
static int f2fs_collapse_range(struct inode *inode, loff_t offset, loff_t len)
{
- pgoff_t pg_start, pg_end;
loff_t new_size;
int ret;
@@ -1185,37 +1293,27 @@ static int f2fs_collapse_range(struct inode *inode, loff_t offset, loff_t len)
if (ret)
return ret;
- pg_start = offset >> PAGE_SHIFT;
- pg_end = (offset + len) >> PAGE_SHIFT;
-
- /* avoid gc operation during block exchange */
- down_write(&F2FS_I(inode)->dio_rwsem[WRITE]);
-
- down_write(&F2FS_I(inode)->i_mmap_sem);
/* write out all dirty pages from offset */
ret = filemap_write_and_wait_range(inode->i_mapping, offset, LLONG_MAX);
if (ret)
- goto out_unlock;
+ return ret;
- truncate_pagecache(inode, offset);
-
- ret = f2fs_do_collapse(inode, pg_start, pg_end);
+ ret = f2fs_do_collapse(inode, offset, len);
if (ret)
- goto out_unlock;
+ return ret;
/* write out all moved pages, if possible */
+ down_write(&F2FS_I(inode)->i_mmap_sem);
filemap_write_and_wait_range(inode->i_mapping, offset, LLONG_MAX);
truncate_pagecache(inode, offset);
new_size = i_size_read(inode) - len;
truncate_pagecache(inode, new_size);
- ret = truncate_blocks(inode, new_size, true);
+ ret = f2fs_truncate_blocks(inode, new_size, true, false);
+ up_write(&F2FS_I(inode)->i_mmap_sem);
if (!ret)
f2fs_i_size_write(inode, new_size);
-out_unlock:
- up_write(&F2FS_I(inode)->i_mmap_sem);
- up_write(&F2FS_I(inode)->dio_rwsem[WRITE]);
return ret;
}
@@ -1235,7 +1333,7 @@ static int f2fs_do_zero_range(struct dnode_of_data *dn, pgoff_t start,
}
dn->ofs_in_node = ofs_in_node;
- ret = reserve_new_blocks(dn, count);
+ ret = f2fs_reserve_new_blocks(dn, count);
if (ret)
return ret;
@@ -1244,7 +1342,7 @@ static int f2fs_do_zero_range(struct dnode_of_data *dn, pgoff_t start,
dn->data_blkaddr = datablock_addr(dn->inode,
dn->node_page, dn->ofs_in_node);
/*
- * reserve_new_blocks will not guarantee entire block
+ * f2fs_reserve_new_blocks will not guarantee entire block
* allocation.
*/
if (dn->data_blkaddr == NULL_ADDR) {
@@ -1252,9 +1350,9 @@ static int f2fs_do_zero_range(struct dnode_of_data *dn, pgoff_t start,
break;
}
if (dn->data_blkaddr != NEW_ADDR) {
- invalidate_blocks(sbi, dn->data_blkaddr);
+ f2fs_invalidate_blocks(sbi, dn->data_blkaddr);
dn->data_blkaddr = NEW_ADDR;
- set_data_blkaddr(dn);
+ f2fs_set_data_blkaddr(dn);
}
}
@@ -1281,12 +1379,9 @@ static int f2fs_zero_range(struct inode *inode, loff_t offset, loff_t len,
if (ret)
return ret;
- down_write(&F2FS_I(inode)->i_mmap_sem);
ret = filemap_write_and_wait_range(mapping, offset, offset + len - 1);
if (ret)
- goto out_sem;
-
- truncate_pagecache_range(inode, offset, offset + len - 1);
+ return ret;
pg_start = ((unsigned long long) offset) >> PAGE_SHIFT;
pg_end = ((unsigned long long) offset + len) >> PAGE_SHIFT;
@@ -1298,7 +1393,7 @@ static int f2fs_zero_range(struct inode *inode, loff_t offset, loff_t len,
ret = fill_zero(inode, pg_start, off_start,
off_end - off_start);
if (ret)
- goto out_sem;
+ return ret;
new_size = max_t(loff_t, new_size, offset + len);
} else {
@@ -1306,7 +1401,7 @@ static int f2fs_zero_range(struct inode *inode, loff_t offset, loff_t len,
ret = fill_zero(inode, pg_start++, off_start,
PAGE_SIZE - off_start);
if (ret)
- goto out_sem;
+ return ret;
new_size = max_t(loff_t, new_size,
(loff_t)pg_start << PAGE_SHIFT);
@@ -1317,12 +1412,21 @@ static int f2fs_zero_range(struct inode *inode, loff_t offset, loff_t len,
unsigned int end_offset;
pgoff_t end;
+ down_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
+ down_write(&F2FS_I(inode)->i_mmap_sem);
+
+ truncate_pagecache_range(inode,
+ (loff_t)index << PAGE_SHIFT,
+ ((loff_t)pg_end << PAGE_SHIFT) - 1);
+
f2fs_lock_op(sbi);
set_new_dnode(&dn, inode, NULL, NULL, 0);
- ret = get_dnode_of_data(&dn, index, ALLOC_NODE);
+ ret = f2fs_get_dnode_of_data(&dn, index, ALLOC_NODE);
if (ret) {
f2fs_unlock_op(sbi);
+ up_write(&F2FS_I(inode)->i_mmap_sem);
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
goto out;
}
@@ -1331,7 +1435,10 @@ static int f2fs_zero_range(struct inode *inode, loff_t offset, loff_t len,
ret = f2fs_do_zero_range(&dn, index, end);
f2fs_put_dnode(&dn);
+
f2fs_unlock_op(sbi);
+ up_write(&F2FS_I(inode)->i_mmap_sem);
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
f2fs_balance_fs(sbi, dn.node_changed);
@@ -1359,9 +1466,6 @@ static int f2fs_zero_range(struct inode *inode, loff_t offset, loff_t len,
else
f2fs_i_size_write(inode, new_size);
}
-out_sem:
- up_write(&F2FS_I(inode)->i_mmap_sem);
-
return ret;
}
@@ -1390,26 +1494,27 @@ static int f2fs_insert_range(struct inode *inode, loff_t offset, loff_t len)
f2fs_balance_fs(sbi, true);
- /* avoid gc operation during block exchange */
- down_write(&F2FS_I(inode)->dio_rwsem[WRITE]);
-
down_write(&F2FS_I(inode)->i_mmap_sem);
- ret = truncate_blocks(inode, i_size_read(inode), true);
+ ret = f2fs_truncate_blocks(inode, i_size_read(inode), true, false);
+ up_write(&F2FS_I(inode)->i_mmap_sem);
if (ret)
- goto out;
+ return ret;
/* write out all dirty pages from offset */
ret = filemap_write_and_wait_range(inode->i_mapping, offset, LLONG_MAX);
if (ret)
- goto out;
-
- truncate_pagecache(inode, offset);
+ return ret;
pg_start = offset >> PAGE_SHIFT;
pg_end = (offset + len) >> PAGE_SHIFT;
delta = pg_end - pg_start;
idx = (i_size_read(inode) + PAGE_SIZE - 1) / PAGE_SIZE;
+ /* avoid gc operation during block exchange */
+ down_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
+ down_write(&F2FS_I(inode)->i_mmap_sem);
+ truncate_pagecache(inode, offset);
+
while (!ret && idx > pg_start) {
nr = idx - pg_start;
if (nr > delta)
@@ -1423,16 +1528,17 @@ static int f2fs_insert_range(struct inode *inode, loff_t offset, loff_t len)
idx + delta, nr, false);
f2fs_unlock_op(sbi);
}
+ up_write(&F2FS_I(inode)->i_mmap_sem);
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
/* write out all moved pages, if possible */
+ down_write(&F2FS_I(inode)->i_mmap_sem);
filemap_write_and_wait_range(inode->i_mapping, offset, LLONG_MAX);
truncate_pagecache(inode, offset);
+ up_write(&F2FS_I(inode)->i_mmap_sem);
if (!ret)
f2fs_i_size_write(inode, new_size);
-out:
- up_write(&F2FS_I(inode)->i_mmap_sem);
- up_write(&F2FS_I(inode)->dio_rwsem[WRITE]);
return ret;
}
@@ -1441,7 +1547,8 @@ static int expand_inode_data(struct inode *inode, loff_t offset,
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_map_blocks map = { .m_next_pgofs = NULL,
- .m_next_extent = NULL, .m_seg_type = NO_CHECK_TYPE };
+ .m_next_extent = NULL, .m_seg_type = NO_CHECK_TYPE,
+ .m_may_create = true };
pgoff_t pg_end;
loff_t new_size = i_size_read(inode);
loff_t off_end;
@@ -1475,7 +1582,7 @@ static int expand_inode_data(struct inode *inode, loff_t offset,
last_off = map.m_lblk + map.m_len - 1;
/* update new size to the failed position */
- new_size = (last_off == pg_end) ? offset + len:
+ new_size = (last_off == pg_end) ? offset + len :
(loff_t)(last_off + 1) << PAGE_SHIFT;
} else {
new_size = ((loff_t)pg_end << PAGE_SHIFT) + off_end;
@@ -1555,13 +1662,13 @@ static int f2fs_release_file(struct inode *inode, struct file *filp)
/* some remained atomic pages should discarded */
if (f2fs_is_atomic_file(inode))
- drop_inmem_pages(inode);
+ f2fs_drop_inmem_pages(inode);
if (f2fs_is_volatile_file(inode)) {
- clear_inode_flag(inode, FI_VOLATILE_FILE);
- stat_dec_volatile_write(inode);
set_inode_flag(inode, FI_DROP_CACHE);
filemap_fdatawrite(inode->i_mapping);
clear_inode_flag(inode, FI_DROP_CACHE);
+ clear_inode_flag(inode, FI_VOLATILE_FILE);
+ stat_dec_volatile_write(inode);
}
return 0;
}
@@ -1578,7 +1685,7 @@ static int f2fs_file_flush(struct file *file, fl_owner_t id)
*/
if (f2fs_is_atomic_file(inode) &&
F2FS_I(inode)->inmem_task == current)
- drop_inmem_pages(inode);
+ f2fs_drop_inmem_pages(inode);
return 0;
}
@@ -1586,7 +1693,15 @@ static int f2fs_ioc_getflags(struct file *filp, unsigned long arg)
{
struct inode *inode = file_inode(filp);
struct f2fs_inode_info *fi = F2FS_I(inode);
- unsigned int flags = fi->i_flags & FS_FL_USER_VISIBLE;
+ unsigned int flags = fi->i_flags;
+
+ if (f2fs_encrypted_inode(inode))
+ flags |= F2FS_ENCRYPT_FL;
+ if (f2fs_has_inline_data(inode) || f2fs_has_inline_dentry(inode))
+ flags |= F2FS_INLINE_DATA_FL;
+
+ flags &= F2FS_FL_USER_VISIBLE;
+
return put_user(flags, (int __user *)arg);
}
@@ -1620,15 +1735,15 @@ static int f2fs_ioc_setflags(struct file *filp, unsigned long arg)
oldflags = fi->i_flags;
- if ((flags ^ oldflags) & (FS_APPEND_FL | FS_IMMUTABLE_FL)) {
+ if ((flags ^ oldflags) & (F2FS_APPEND_FL | F2FS_IMMUTABLE_FL)) {
if (!capable(CAP_LINUX_IMMUTABLE)) {
ret = -EPERM;
goto unlock_out;
}
}
- flags = flags & FS_FL_USER_MODIFIABLE;
- flags |= oldflags & ~FS_FL_USER_MODIFIABLE;
+ flags = flags & (F2FS_FL_USER_MODIFIABLE);
+ flags |= oldflags & ~(F2FS_FL_USER_MODIFIABLE);
fi->i_flags = flags;
inode->i_ctime = current_time(inode);
@@ -1664,31 +1779,37 @@ static int f2fs_ioc_start_atomic_write(struct file *filp)
inode_lock(inode);
- if (f2fs_is_atomic_file(inode))
+ if (f2fs_is_atomic_file(inode)) {
+ if (is_inode_flag_set(inode, FI_ATOMIC_REVOKE_REQUEST))
+ ret = -EINVAL;
goto out;
+ }
ret = f2fs_convert_inline_inode(inode);
if (ret)
goto out;
- set_inode_flag(inode, FI_ATOMIC_FILE);
- set_inode_flag(inode, FI_HOT_DATA);
- f2fs_update_time(F2FS_I_SB(inode), REQ_TIME);
+ down_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
- if (!get_dirty_pages(inode))
- goto inc_stat;
-
- f2fs_msg(F2FS_I_SB(inode)->sb, KERN_WARNING,
+ /*
+ * Should wait end_io to count F2FS_WB_CP_DATA correctly by
+ * f2fs_is_atomic_file.
+ */
+ if (get_dirty_pages(inode))
+ f2fs_msg(F2FS_I_SB(inode)->sb, KERN_WARNING,
"Unexpected flush for atomic writes: ino=%lu, npages=%u",
inode->i_ino, get_dirty_pages(inode));
ret = filemap_write_and_wait_range(inode->i_mapping, 0, LLONG_MAX);
if (ret) {
- clear_inode_flag(inode, FI_ATOMIC_FILE);
- clear_inode_flag(inode, FI_HOT_DATA);
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
goto out;
}
-inc_stat:
+ set_inode_flag(inode, FI_ATOMIC_FILE);
+ clear_inode_flag(inode, FI_ATOMIC_REVOKE_REQUEST);
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
+
+ f2fs_update_time(F2FS_I_SB(inode), REQ_TIME);
F2FS_I(inode)->inmem_task = current;
stat_inc_atomic_write(inode);
stat_update_max_atomic_write(inode);
@@ -1710,29 +1831,34 @@ static int f2fs_ioc_commit_atomic_write(struct file *filp)
if (ret)
return ret;
+ f2fs_balance_fs(F2FS_I_SB(inode), true);
+
inode_lock(inode);
- down_write(&F2FS_I(inode)->dio_rwsem[WRITE]);
-
- if (f2fs_is_volatile_file(inode))
+ if (f2fs_is_volatile_file(inode)) {
+ ret = -EINVAL;
goto err_out;
+ }
if (f2fs_is_atomic_file(inode)) {
- ret = commit_inmem_pages(inode);
+ ret = f2fs_commit_inmem_pages(inode);
if (ret)
goto err_out;
ret = f2fs_do_sync_file(filp, 0, LLONG_MAX, 0, true);
if (!ret) {
clear_inode_flag(inode, FI_ATOMIC_FILE);
- clear_inode_flag(inode, FI_HOT_DATA);
+ F2FS_I(inode)->i_gc_failures[GC_FAILURE_ATOMIC] = 0;
stat_dec_atomic_write(inode);
}
} else {
ret = f2fs_do_sync_file(filp, 0, LLONG_MAX, 1, false);
}
err_out:
- up_write(&F2FS_I(inode)->dio_rwsem[WRITE]);
+ if (is_inode_flag_set(inode, FI_ATOMIC_REVOKE_REQUEST)) {
+ clear_inode_flag(inode, FI_ATOMIC_REVOKE_REQUEST);
+ ret = -EINVAL;
+ }
inode_unlock(inode);
mnt_drop_write_file(filp);
return ret;
@@ -1817,13 +1943,15 @@ static int f2fs_ioc_abort_volatile_write(struct file *filp)
inode_lock(inode);
if (f2fs_is_atomic_file(inode))
- drop_inmem_pages(inode);
+ f2fs_drop_inmem_pages(inode);
if (f2fs_is_volatile_file(inode)) {
clear_inode_flag(inode, FI_VOLATILE_FILE);
stat_dec_volatile_write(inode);
ret = f2fs_do_sync_file(filp, 0, LLONG_MAX, 0, true);
}
+ clear_inode_flag(inode, FI_ATOMIC_REVOKE_REQUEST);
+
inode_unlock(inode);
mnt_drop_write_file(filp);
@@ -1837,7 +1965,7 @@ static int f2fs_ioc_shutdown(struct file *filp, unsigned long arg)
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct super_block *sb = sbi->sb;
__u32 in;
- int ret;
+ int ret = 0;
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
@@ -1845,9 +1973,11 @@ static int f2fs_ioc_shutdown(struct file *filp, unsigned long arg)
if (get_user(in, (__u32 __user *)arg))
return -EFAULT;
- ret = mnt_want_write_file(filp);
- if (ret)
- return ret;
+ if (in != F2FS_GOING_DOWN_FULLSYNC) {
+ ret = mnt_want_write_file(filp);
+ if (ret)
+ return ret;
+ }
switch (in) {
case F2FS_GOING_DOWN_FULLSYNC:
@@ -1858,6 +1988,7 @@ static int f2fs_ioc_shutdown(struct file *filp, unsigned long arg)
}
if (sb) {
f2fs_stop_checkpoint(sbi, false);
+ set_sbi_flag(sbi, SBI_IS_SHUTDOWN);
thaw_bdev(sb->s_bdev, sb);
}
break;
@@ -1867,28 +1998,39 @@ static int f2fs_ioc_shutdown(struct file *filp, unsigned long arg)
if (ret)
goto out;
f2fs_stop_checkpoint(sbi, false);
+ set_sbi_flag(sbi, SBI_IS_SHUTDOWN);
break;
case F2FS_GOING_DOWN_NOSYNC:
f2fs_stop_checkpoint(sbi, false);
+ set_sbi_flag(sbi, SBI_IS_SHUTDOWN);
break;
case F2FS_GOING_DOWN_METAFLUSH:
- sync_meta_pages(sbi, META, LONG_MAX, FS_META_IO);
+ f2fs_sync_meta_pages(sbi, META, LONG_MAX, FS_META_IO);
f2fs_stop_checkpoint(sbi, false);
+ set_sbi_flag(sbi, SBI_IS_SHUTDOWN);
+ break;
+ case F2FS_GOING_DOWN_NEED_FSCK:
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ /* do checkpoint only */
+ ret = f2fs_sync_fs(sb, 1);
+ if (ret)
+ goto out;
break;
default:
ret = -EINVAL;
goto out;
}
- stop_gc_thread(sbi);
- stop_discard_thread(sbi);
+ f2fs_stop_gc_thread(sbi);
+ f2fs_stop_discard_thread(sbi);
- drop_discard_cmd(sbi);
+ f2fs_drop_discard_cmd(sbi);
clear_opt(sbi, DISCARD);
f2fs_update_time(sbi, REQ_TIME);
out:
- mnt_drop_write_file(filp);
+ if (in != F2FS_GOING_DOWN_FULLSYNC)
+ mnt_drop_write_file(filp);
return ret;
}
@@ -1903,7 +2045,7 @@ static int f2fs_ioc_fitrim(struct file *filp, unsigned long arg)
if (!capable(CAP_SYS_ADMIN))
return -EPERM;
- if (!blk_queue_discard(q))
+ if (!f2fs_hw_support_discard(F2FS_SB(sb)))
return -EOPNOTSUPP;
if (copy_from_user(&range, (struct fstrim_range __user *)arg,
@@ -1942,7 +2084,7 @@ static int f2fs_ioc_set_encryption_policy(struct file *filp, unsigned long arg)
{
struct inode *inode = file_inode(filp);
- if (!f2fs_sb_has_encrypt(inode->i_sb))
+ if (!f2fs_sb_has_encrypt(F2FS_I_SB(inode)))
return -EOPNOTSUPP;
f2fs_update_time(F2FS_I_SB(inode), REQ_TIME);
@@ -1952,7 +2094,7 @@ static int f2fs_ioc_set_encryption_policy(struct file *filp, unsigned long arg)
static int f2fs_ioc_get_encryption_policy(struct file *filp, unsigned long arg)
{
- if (!f2fs_sb_has_encrypt(file_inode(filp)->i_sb))
+ if (!f2fs_sb_has_encrypt(F2FS_I_SB(file_inode(filp))))
return -EOPNOTSUPP;
return fscrypt_ioctl_get_policy(filp, (void __user *)arg);
}
@@ -1963,7 +2105,7 @@ static int f2fs_ioc_get_encryption_pwsalt(struct file *filp, unsigned long arg)
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
int err;
- if (!f2fs_sb_has_encrypt(inode->i_sb))
+ if (!f2fs_sb_has_encrypt(sbi))
return -EOPNOTSUPP;
err = mnt_want_write_file(filp);
@@ -2047,15 +2189,15 @@ static int f2fs_ioc_gc_range(struct file *filp, unsigned long arg)
if (f2fs_readonly(sbi->sb))
return -EROFS;
+ end = range.start + range.len;
+ if (range.start < MAIN_BLKADDR(sbi) || end >= MAX_BLKADDR(sbi)) {
+ return -EINVAL;
+ }
+
ret = mnt_want_write_file(filp);
if (ret)
return ret;
- end = range.start + range.len;
- if (range.start < MAIN_BLKADDR(sbi) || end >= MAX_BLKADDR(sbi)) {
- ret = -EINVAL;
- goto out;
- }
do_more:
if (!range.sync) {
if (!mutex_trylock(&sbi->gc_mutex)) {
@@ -2067,7 +2209,7 @@ static int f2fs_ioc_gc_range(struct file *filp, unsigned long arg)
}
ret = f2fs_gc(sbi, range.sync, true, GET_SEGNO(sbi, range.start));
- range.start += sbi->blocks_per_seg;
+ range.start += BLKS_PER_SEC(sbi);
if (range.start <= end)
goto do_more;
out:
@@ -2087,6 +2229,12 @@ static int f2fs_ioc_write_checkpoint(struct file *filp, unsigned long arg)
if (f2fs_readonly(sbi->sb))
return -EROFS;
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) {
+ f2fs_msg(sbi->sb, KERN_INFO,
+ "Skipping Checkpoint. Checkpoints currently disabled.");
+ return -EINVAL;
+ }
+
ret = mnt_want_write_file(filp);
if (ret)
return ret;
@@ -2103,8 +2251,9 @@ static int f2fs_defragment_range(struct f2fs_sb_info *sbi,
{
struct inode *inode = file_inode(filp);
struct f2fs_map_blocks map = { .m_next_extent = NULL,
- .m_seg_type = NO_CHECK_TYPE };
- struct extent_info ei = {0,0,0};
+ .m_seg_type = NO_CHECK_TYPE ,
+ .m_may_create = false };
+ struct extent_info ei = {0, 0, 0};
pgoff_t pg_start, pg_end, next_pgofs;
unsigned int blk_per_seg = sbi->blocks_per_seg;
unsigned int total = 0, sec_num;
@@ -2113,7 +2262,7 @@ static int f2fs_defragment_range(struct f2fs_sb_info *sbi,
int err;
/* if in-place-update policy is enabled, don't waste time here */
- if (should_update_inplace(inode, NULL))
+ if (f2fs_should_update_inplace(inode, NULL))
return -EINVAL;
pg_start = range->start >> PAGE_SHIFT;
@@ -2208,7 +2357,7 @@ static int f2fs_defragment_range(struct f2fs_sb_info *sbi,
while (idx < map.m_lblk + map.m_len && cnt < blk_per_seg) {
struct page *page;
- page = get_lock_data_page(inode, idx, true);
+ page = f2fs_get_lock_data_page(inode, idx, true);
if (IS_ERR(page)) {
err = PTR_ERR(page);
goto clear_out;
@@ -2319,15 +2468,10 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in,
}
inode_lock(src);
- down_write(&F2FS_I(src)->dio_rwsem[WRITE]);
if (src != dst) {
ret = -EBUSY;
if (!inode_trylock(dst))
goto out;
- if (!down_write_trylock(&F2FS_I(dst)->dio_rwsem[WRITE])) {
- inode_unlock(dst);
- goto out;
- }
}
ret = -EINVAL;
@@ -2372,6 +2516,14 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in,
goto out_unlock;
f2fs_balance_fs(sbi, true);
+
+ down_write(&F2FS_I(src)->i_gc_rwsem[WRITE]);
+ if (src != dst) {
+ ret = -EBUSY;
+ if (!down_write_trylock(&F2FS_I(dst)->i_gc_rwsem[WRITE]))
+ goto out_src;
+ }
+
f2fs_lock_op(sbi);
ret = __exchange_data_block(src, dst, pos_in >> F2FS_BLKSIZE_BITS,
pos_out >> F2FS_BLKSIZE_BITS,
@@ -2384,13 +2536,15 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in,
f2fs_i_size_write(dst, dst_osize);
}
f2fs_unlock_op(sbi);
+
+ if (src != dst)
+ up_write(&F2FS_I(dst)->i_gc_rwsem[WRITE]);
+out_src:
+ up_write(&F2FS_I(src)->i_gc_rwsem[WRITE]);
out_unlock:
- if (src != dst) {
- up_write(&F2FS_I(dst)->dio_rwsem[WRITE]);
+ if (src != dst)
inode_unlock(dst);
- }
out:
- up_write(&F2FS_I(src)->dio_rwsem[WRITE]);
inode_unlock(src);
return ret;
}
@@ -2453,12 +2607,15 @@ static int f2fs_ioc_flush_device(struct file *filp, unsigned long arg)
if (f2fs_readonly(sbi->sb))
return -EROFS;
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
+ return -EINVAL;
+
if (copy_from_user(&range, (struct f2fs_flush_device __user *)arg,
sizeof(range)))
return -EFAULT;
if (sbi->s_ndevs <= 1 || sbi->s_ndevs - 1 <= range.dev_num ||
- sbi->segs_per_sec != 1) {
+ __is_large_section(sbi)) {
f2fs_msg(sbi->sb, KERN_WARNING,
"Can't flush %u in %d for segs_per_sec %u != 1\n",
range.dev_num, sbi->s_ndevs,
@@ -2517,12 +2674,14 @@ int f2fs_pin_file_control(struct inode *inode, bool inc)
/* Use i_gc_failures for normal file as a risk signal. */
if (inc)
- f2fs_i_gc_failures_write(inode, fi->i_gc_failures + 1);
+ f2fs_i_gc_failures_write(inode,
+ fi->i_gc_failures[GC_FAILURE_PIN] + 1);
- if (fi->i_gc_failures > sbi->gc_pin_file_threshold) {
+ if (fi->i_gc_failures[GC_FAILURE_PIN] > sbi->gc_pin_file_threshold) {
f2fs_msg(sbi->sb, KERN_WARNING,
"%s: Enable GC = ino %lx after %x GC trials\n",
- __func__, inode->i_ino, fi->i_gc_failures);
+ __func__, inode->i_ino,
+ fi->i_gc_failures[GC_FAILURE_PIN]);
clear_inode_flag(inode, FI_PIN_FILE);
return -EAGAIN;
}
@@ -2553,14 +2712,14 @@ static int f2fs_ioc_set_pin_file(struct file *filp, unsigned long arg)
inode_lock(inode);
- if (should_update_outplace(inode, NULL)) {
+ if (f2fs_should_update_outplace(inode, NULL)) {
ret = -EINVAL;
goto out;
}
if (!pin) {
clear_inode_flag(inode, FI_PIN_FILE);
- F2FS_I(inode)->i_gc_failures = 1;
+ f2fs_i_gc_failures_write(inode, 0);
goto done;
}
@@ -2573,7 +2732,7 @@ static int f2fs_ioc_set_pin_file(struct file *filp, unsigned long arg)
goto out;
set_inode_flag(inode, FI_PIN_FILE);
- ret = F2FS_I(inode)->i_gc_failures;
+ ret = F2FS_I(inode)->i_gc_failures[GC_FAILURE_PIN];
done:
f2fs_update_time(F2FS_I_SB(inode), REQ_TIME);
out:
@@ -2588,7 +2747,7 @@ static int f2fs_ioc_get_pin_file(struct file *filp, unsigned long arg)
__u32 pin = 0;
if (is_inode_flag_set(inode, FI_PIN_FILE))
- pin = F2FS_I(inode)->i_gc_failures;
+ pin = F2FS_I(inode)->i_gc_failures[GC_FAILURE_PIN];
return put_user(pin, (u32 __user *)arg);
}
@@ -2607,14 +2766,15 @@ int f2fs_precache_extents(struct inode *inode)
map.m_next_pgofs = NULL;
map.m_next_extent = &m_next_extent;
map.m_seg_type = NO_CHECK_TYPE;
+ map.m_may_create = false;
end = F2FS_I_SB(inode)->max_file_blocks;
while (map.m_lblk < end) {
map.m_len = end - map.m_lblk;
- down_write(&fi->dio_rwsem[WRITE]);
+ down_write(&fi->i_gc_rwsem[WRITE]);
err = f2fs_map_blocks(inode, &map, 0, F2FS_GET_BLOCK_PRECACHE);
- up_write(&fi->dio_rwsem[WRITE]);
+ up_write(&fi->i_gc_rwsem[WRITE]);
if (err)
return err;
@@ -2629,6 +2789,27 @@ static int f2fs_ioc_precache_extents(struct file *filp, unsigned long arg)
return f2fs_precache_extents(file_inode(filp));
}
+static int f2fs_ioc_enable_verity(struct file *filp, unsigned long arg)
+{
+ struct inode *inode = file_inode(filp);
+
+ f2fs_update_time(F2FS_I_SB(inode), REQ_TIME);
+
+ if (!f2fs_sb_has_verity(F2FS_I_SB(inode))) {
+ f2fs_msg(inode->i_sb, KERN_WARNING,
+ "Can't enable fs-verity on inode %lu: the fs-verity feature is disabled on this filesystem.\n",
+ inode->i_ino);
+ return -EOPNOTSUPP;
+ }
+
+ return fsverity_ioctl_enable(filp, (const void __user *)arg);
+}
+
+static int f2fs_ioc_set_verity_measurement(struct file *filp, unsigned long arg)
+{
+ return fsverity_ioctl_set_measurement(filp, (const void __user *)arg);
+}
+
long f2fs_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
{
if (unlikely(f2fs_cp_error(F2FS_I_SB(file_inode(filp)))))
@@ -2681,6 +2862,10 @@ long f2fs_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
return f2fs_ioc_set_pin_file(filp, arg);
case F2FS_IOC_PRECACHE_EXTENTS:
return f2fs_ioc_precache_extents(filp, arg);
+ case FS_IOC_ENABLE_VERITY:
+ return f2fs_ioc_enable_verity(filp, arg);
+ case FS_IOC_SET_VERITY_MEASUREMENT:
+ return f2fs_ioc_set_verity_measurement(filp, arg);
default:
return -ENOTTY;
}
@@ -2690,7 +2875,6 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
{
struct file *file = iocb->ki_filp;
struct inode *inode = file_inode(file);
- struct blk_plug plug;
ssize_t ret;
if (unlikely(f2fs_cp_error(F2FS_I_SB(inode))))
@@ -2719,7 +2903,10 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
if (!f2fs_overwrite_io(inode, iocb->ki_pos,
iov_iter_count(from)) ||
f2fs_has_inline_data(inode) ||
- f2fs_force_buffered_io(inode, WRITE)) {
+ f2fs_force_buffered_io(inode,
+ iocb, from)) {
+ clear_inode_flag(inode,
+ FI_NO_PREALLOC);
inode_unlock(inode);
return -EAGAIN;
}
@@ -2735,9 +2922,7 @@ static ssize_t f2fs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
return err;
}
}
- blk_start_plug(&plug);
ret = __generic_file_write_iter(iocb, from);
- blk_finish_plug(&plug);
clear_inode_flag(inode, FI_NO_PREALLOC);
/* if we couldn't write data, we should deallocate blocks. */
@@ -2786,6 +2971,8 @@ long f2fs_compat_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
case F2FS_IOC_GET_PIN_FILE:
case F2FS_IOC_SET_PIN_FILE:
case F2FS_IOC_PRECACHE_EXTENTS:
+ case FS_IOC_ENABLE_VERITY:
+ case FS_IOC_SET_VERITY_MEASUREMENT:
break;
default:
return -ENOIOCTLCMD;
diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c
index 604d3fc..c2114ff 100644
--- a/fs/f2fs/gc.c
+++ b/fs/f2fs/gc.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/gc.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/module.h>
@@ -43,25 +40,28 @@ static int gc_thread_func(void *data)
if (gc_th->gc_wake)
gc_th->gc_wake = 0;
- if (try_to_freeze())
+ if (try_to_freeze()) {
+ stat_other_skip_bggc_count(sbi);
continue;
+ }
if (kthread_should_stop())
break;
if (sbi->sb->s_writers.frozen >= SB_FREEZE_WRITE) {
increase_sleep_time(gc_th, &wait_ms);
+ stat_other_skip_bggc_count(sbi);
continue;
}
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(sbi, FAULT_CHECKPOINT)) {
f2fs_show_injection_info(FAULT_CHECKPOINT);
f2fs_stop_checkpoint(sbi, false);
}
-#endif
- if (!sb_start_write_trylock(sbi->sb))
+ if (!sb_start_write_trylock(sbi->sb)) {
+ stat_other_skip_bggc_count(sbi);
continue;
+ }
/*
* [GC triggering condition]
@@ -76,18 +76,21 @@ static int gc_thread_func(void *data)
* invalidated soon after by user update or deletion.
* So, I'd like to wait some time to collect dirty segments.
*/
- if (gc_th->gc_urgent) {
+ if (sbi->gc_mode == GC_URGENT) {
wait_ms = gc_th->urgent_sleep_time;
mutex_lock(&sbi->gc_mutex);
goto do_gc;
}
- if (!mutex_trylock(&sbi->gc_mutex))
+ if (!mutex_trylock(&sbi->gc_mutex)) {
+ stat_other_skip_bggc_count(sbi);
goto next;
+ }
- if (!is_idle(sbi)) {
+ if (!is_idle(sbi, GC_TIME)) {
increase_sleep_time(gc_th, &wait_ms);
mutex_unlock(&sbi->gc_mutex);
+ stat_io_skip_bggc_count(sbi);
goto next;
}
@@ -114,7 +117,7 @@ static int gc_thread_func(void *data)
return 0;
}
-int start_gc_thread(struct f2fs_sb_info *sbi)
+int f2fs_start_gc_thread(struct f2fs_sb_info *sbi)
{
struct f2fs_gc_kthread *gc_th;
dev_t dev = sbi->sb->s_bdev->bd_dev;
@@ -131,8 +134,6 @@ int start_gc_thread(struct f2fs_sb_info *sbi)
gc_th->max_sleep_time = DEF_GC_THREAD_MAX_SLEEP_TIME;
gc_th->no_gc_sleep_time = DEF_GC_THREAD_NOGC_SLEEP_TIME;
- gc_th->gc_idle = 0;
- gc_th->gc_urgent = 0;
gc_th->gc_wake= 0;
sbi->gc_thread = gc_th;
@@ -141,38 +142,36 @@ int start_gc_thread(struct f2fs_sb_info *sbi)
"f2fs_gc-%u:%u", MAJOR(dev), MINOR(dev));
if (IS_ERR(gc_th->f2fs_gc_task)) {
err = PTR_ERR(gc_th->f2fs_gc_task);
- kfree(gc_th);
+ kvfree(gc_th);
sbi->gc_thread = NULL;
}
out:
return err;
}
-void stop_gc_thread(struct f2fs_sb_info *sbi)
+void f2fs_stop_gc_thread(struct f2fs_sb_info *sbi)
{
struct f2fs_gc_kthread *gc_th = sbi->gc_thread;
if (!gc_th)
return;
kthread_stop(gc_th->f2fs_gc_task);
- kfree(gc_th);
+ kvfree(gc_th);
sbi->gc_thread = NULL;
}
-static int select_gc_type(struct f2fs_gc_kthread *gc_th, int gc_type)
+static int select_gc_type(struct f2fs_sb_info *sbi, int gc_type)
{
int gc_mode = (gc_type == BG_GC) ? GC_CB : GC_GREEDY;
- if (!gc_th)
- return gc_mode;
-
- if (gc_th->gc_idle) {
- if (gc_th->gc_idle == 1)
- gc_mode = GC_CB;
- else if (gc_th->gc_idle == 2)
- gc_mode = GC_GREEDY;
- }
- if (gc_th->gc_urgent)
+ switch (sbi->gc_mode) {
+ case GC_IDLE_CB:
+ gc_mode = GC_CB;
+ break;
+ case GC_IDLE_GREEDY:
+ case GC_URGENT:
gc_mode = GC_GREEDY;
+ break;
+ }
return gc_mode;
}
@@ -187,7 +186,7 @@ static void select_policy(struct f2fs_sb_info *sbi, int gc_type,
p->max_search = dirty_i->nr_dirty[type];
p->ofs_unit = 1;
} else {
- p->gc_mode = select_gc_type(sbi->gc_thread, gc_type);
+ p->gc_mode = select_gc_type(sbi, gc_type);
p->dirty_segmap = dirty_i->dirty_segmap[DIRTY];
p->max_search = dirty_i->nr_dirty[DIRTY];
p->ofs_unit = sbi->segs_per_sec;
@@ -195,7 +194,7 @@ static void select_policy(struct f2fs_sb_info *sbi, int gc_type,
/* we need to check every dirty segments in the FG_GC case */
if (gc_type != FG_GC &&
- (sbi->gc_thread && !sbi->gc_thread->gc_urgent) &&
+ (sbi->gc_mode != GC_URGENT) &&
p->max_search > sbi->max_victim_search)
p->max_search = sbi->max_victim_search;
@@ -234,10 +233,6 @@ static unsigned int check_bg_victims(struct f2fs_sb_info *sbi)
for_each_set_bit(secno, dirty_i->victim_secmap, MAIN_SECS(sbi)) {
if (sec_usage_check(sbi, secno))
continue;
-
- if (no_fggc_candidate(sbi, secno))
- continue;
-
clear_bit(secno, dirty_i->victim_secmap);
return GET_SEG_FROM_SEC(sbi, secno);
}
@@ -328,8 +323,7 @@ static int get_victim_by_default(struct f2fs_sb_info *sbi,
p.min_cost = get_max_cost(sbi, &p);
if (*result != NULL_SEGNO) {
- if (IS_DATASEG(get_seg_entry(sbi, *result)->type) &&
- get_valid_blocks(sbi, *result, false) &&
+ if (get_valid_blocks(sbi, *result, false) &&
!sec_usage_check(sbi, GET_SEC_FROM_SEG(sbi, *result)))
p.min_segno = *result;
goto out;
@@ -338,6 +332,22 @@ static int get_victim_by_default(struct f2fs_sb_info *sbi,
if (p.max_search == 0)
goto out;
+ if (__is_large_section(sbi) && p.alloc_mode == LFS) {
+ if (sbi->next_victim_seg[BG_GC] != NULL_SEGNO) {
+ p.min_segno = sbi->next_victim_seg[BG_GC];
+ *result = p.min_segno;
+ sbi->next_victim_seg[BG_GC] = NULL_SEGNO;
+ goto got_result;
+ }
+ if (gc_type == FG_GC &&
+ sbi->next_victim_seg[FG_GC] != NULL_SEGNO) {
+ p.min_segno = sbi->next_victim_seg[FG_GC];
+ *result = p.min_segno;
+ sbi->next_victim_seg[FG_GC] = NULL_SEGNO;
+ goto got_result;
+ }
+ }
+
last_victim = sm->last_victim[p.gc_mode];
if (p.alloc_mode == LFS && gc_type == FG_GC) {
p.min_segno = check_bg_victims(sbi);
@@ -375,10 +385,11 @@ static int get_victim_by_default(struct f2fs_sb_info *sbi,
if (sec_usage_check(sbi, secno))
goto next;
- if (gc_type == BG_GC && test_bit(secno, dirty_i->victim_secmap))
+ /* Don't touch checkpointed data */
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED) &&
+ get_ckpt_valid_blocks(sbi, segno)))
goto next;
- if (gc_type == FG_GC && p.alloc_mode == LFS &&
- no_fggc_candidate(sbi, secno))
+ if (gc_type == BG_GC && test_bit(secno, dirty_i->victim_secmap))
goto next;
cost = get_gc_cost(sbi, segno, &p);
@@ -399,6 +410,8 @@ static int get_victim_by_default(struct f2fs_sb_info *sbi,
}
if (p.min_segno != NULL_SEGNO) {
got_it:
+ *result = (p.min_segno / p.ofs_unit) * p.ofs_unit;
+got_result:
if (p.alloc_mode == LFS) {
secno = GET_SEC_FROM_SEG(sbi, p.min_segno);
if (gc_type == FG_GC)
@@ -406,13 +419,13 @@ static int get_victim_by_default(struct f2fs_sb_info *sbi,
else
set_bit(secno, dirty_i->victim_secmap);
}
- *result = (p.min_segno / p.ofs_unit) * p.ofs_unit;
+ }
+out:
+ if (p.min_segno != NULL_SEGNO)
trace_f2fs_get_victim(sbi->sb, type, gc_type, &p,
sbi->cur_victim_sec,
prefree_segments(sbi), free_segments(sbi));
- }
-out:
mutex_unlock(&dirty_i->seglist_lock);
return (p.min_segno == NULL_SEGNO) ? 0 : 1;
@@ -440,7 +453,7 @@ static void add_gc_inode(struct gc_inode_list *gc_list, struct inode *inode)
iput(inode);
return;
}
- new_ie = f2fs_kmem_cache_alloc(inode_entry_slab, GFP_NOFS);
+ new_ie = f2fs_kmem_cache_alloc(f2fs_inode_entry_slab, GFP_NOFS);
new_ie->inode = inode;
f2fs_radix_tree_insert(&gc_list->iroot, inode->i_ino, new_ie);
@@ -454,7 +467,7 @@ static void put_gc_inode(struct gc_inode_list *gc_list)
radix_tree_delete(&gc_list->iroot, ie->inode->i_ino);
iput(ie->inode);
list_del(&ie->list);
- kmem_cache_free(inode_entry_slab, ie);
+ kmem_cache_free(f2fs_inode_entry_slab, ie);
}
}
@@ -477,65 +490,81 @@ static int check_valid_map(struct f2fs_sb_info *sbi,
* On validity, copy that node with cold status, otherwise (invalid node)
* ignore that.
*/
-static void gc_node_segment(struct f2fs_sb_info *sbi,
+static int gc_node_segment(struct f2fs_sb_info *sbi,
struct f2fs_summary *sum, unsigned int segno, int gc_type)
{
struct f2fs_summary *entry;
block_t start_addr;
int off;
int phase = 0;
+ bool fggc = (gc_type == FG_GC);
+ int submitted = 0;
start_addr = START_BLOCK(sbi, segno);
next_step:
entry = sum;
+ if (fggc && phase == 2)
+ atomic_inc(&sbi->wb_sync_req[NODE]);
+
for (off = 0; off < sbi->blocks_per_seg; off++, entry++) {
nid_t nid = le32_to_cpu(entry->nid);
struct page *node_page;
struct node_info ni;
+ int err;
/* stop BG_GC if there is not enough free sections. */
if (gc_type == BG_GC && has_not_enough_free_secs(sbi, 0, 0))
- return;
+ return submitted;
if (check_valid_map(sbi, segno, off) == 0)
continue;
if (phase == 0) {
- ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1,
+ f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1,
META_NAT, true);
continue;
}
if (phase == 1) {
- ra_node_page(sbi, nid);
+ f2fs_ra_node_page(sbi, nid);
continue;
}
/* phase == 2 */
- node_page = get_node_page(sbi, nid);
+ node_page = f2fs_get_node_page(sbi, nid);
if (IS_ERR(node_page))
continue;
- /* block may become invalid during get_node_page */
+ /* block may become invalid during f2fs_get_node_page */
if (check_valid_map(sbi, segno, off) == 0) {
f2fs_put_page(node_page, 1);
continue;
}
- get_node_info(sbi, nid, &ni);
+ if (f2fs_get_node_info(sbi, nid, &ni)) {
+ f2fs_put_page(node_page, 1);
+ continue;
+ }
+
if (ni.blk_addr != start_addr + off) {
f2fs_put_page(node_page, 1);
continue;
}
- move_node_page(node_page, gc_type);
+ err = f2fs_move_node_page(node_page, gc_type);
+ if (!err && gc_type == FG_GC)
+ submitted++;
stat_inc_node_blk_count(sbi, 1, gc_type);
}
if (++phase < 3)
goto next_step;
+
+ if (fggc)
+ atomic_dec(&sbi->wb_sync_req[NODE]);
+ return submitted;
}
/*
@@ -545,7 +574,7 @@ static void gc_node_segment(struct f2fs_sb_info *sbi,
* as indirect or double indirect node blocks, are given, it must be a caller's
* bug.
*/
-block_t start_bidx_of_node(unsigned int node_ofs, struct inode *inode)
+block_t f2fs_start_bidx_of_node(unsigned int node_ofs, struct inode *inode)
{
unsigned int indirect_blks = 2 * NIDS_PER_BLOCK + 4;
unsigned int bidx;
@@ -576,11 +605,14 @@ static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
nid = le32_to_cpu(sum->nid);
ofs_in_node = le16_to_cpu(sum->ofs_in_node);
- node_page = get_node_page(sbi, nid);
+ node_page = f2fs_get_node_page(sbi, nid);
if (IS_ERR(node_page))
return false;
- get_node_info(sbi, nid, dni);
+ if (f2fs_get_node_info(sbi, nid, dni)) {
+ f2fs_put_page(node_page, 1);
+ return false;
+ }
if (sum->version != dni->version) {
f2fs_msg(sbi->sb, KERN_WARNING,
@@ -598,12 +630,86 @@ static bool is_alive(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
return true;
}
+static int ra_data_block(struct inode *inode, pgoff_t index)
+{
+ struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+ struct address_space *mapping = inode->i_mapping;
+ struct dnode_of_data dn;
+ struct page *page;
+ struct extent_info ei = {0, 0, 0};
+ struct f2fs_io_info fio = {
+ .sbi = sbi,
+ .ino = inode->i_ino,
+ .type = DATA,
+ .temp = COLD,
+ .op = REQ_OP_READ,
+ .op_flags = 0,
+ .encrypted_page = NULL,
+ .in_list = false,
+ .retry = false,
+ };
+ int err;
+
+ page = f2fs_grab_cache_page(mapping, index, true);
+ if (!page)
+ return -ENOMEM;
+
+ if (f2fs_lookup_extent_cache(inode, index, &ei)) {
+ dn.data_blkaddr = ei.blk + index - ei.fofs;
+ goto got_it;
+ }
+
+ set_new_dnode(&dn, inode, NULL, NULL, 0);
+ err = f2fs_get_dnode_of_data(&dn, index, LOOKUP_NODE);
+ if (err)
+ goto put_page;
+ f2fs_put_dnode(&dn);
+
+ if (unlikely(!f2fs_is_valid_blkaddr(sbi, dn.data_blkaddr,
+ DATA_GENERIC))) {
+ err = -EFAULT;
+ goto put_page;
+ }
+got_it:
+ /* read page */
+ fio.page = page;
+ fio.new_blkaddr = fio.old_blkaddr = dn.data_blkaddr;
+
+ /*
+ * don't cache encrypted data into meta inode until previous dirty
+ * data were writebacked to avoid racing between GC and flush.
+ */
+ f2fs_wait_on_page_writeback(page, DATA, true, true);
+
+ f2fs_wait_on_block_writeback(inode, dn.data_blkaddr);
+
+ fio.encrypted_page = f2fs_pagecache_get_page(META_MAPPING(sbi),
+ dn.data_blkaddr,
+ FGP_LOCK | FGP_CREAT, GFP_NOFS);
+ if (!fio.encrypted_page) {
+ err = -ENOMEM;
+ goto put_page;
+ }
+
+ err = f2fs_submit_page_bio(&fio);
+ if (err)
+ goto put_encrypted_page;
+ f2fs_put_page(fio.encrypted_page, 0);
+ f2fs_put_page(page, 1);
+ return 0;
+put_encrypted_page:
+ f2fs_put_page(fio.encrypted_page, 1);
+put_page:
+ f2fs_put_page(page, 1);
+ return err;
+}
+
/*
* Move data block via META_MAPPING while keeping locked data page.
* This can be used to move blocks, aka LBAs, directly on disk.
*/
-static void move_data_block(struct inode *inode, block_t bidx,
- unsigned int segno, int off)
+static int move_data_block(struct inode *inode, block_t bidx,
+ int gc_type, unsigned int segno, int off)
{
struct f2fs_io_info fio = {
.sbi = F2FS_I_SB(inode),
@@ -614,37 +720,47 @@ static void move_data_block(struct inode *inode, block_t bidx,
.op_flags = 0,
.encrypted_page = NULL,
.in_list = false,
+ .retry = false,
};
struct dnode_of_data dn;
struct f2fs_summary sum;
struct node_info ni;
- struct page *page;
+ struct page *page, *mpage;
block_t newaddr;
- int err;
+ int err = 0;
+ bool lfs_mode = test_opt(fio.sbi, LFS);
/* do not read out */
page = f2fs_grab_cache_page(inode->i_mapping, bidx, false);
if (!page)
- return;
+ return -ENOMEM;
- if (!check_valid_map(F2FS_I_SB(inode), segno, off))
+ if (!check_valid_map(F2FS_I_SB(inode), segno, off)) {
+ err = -ENOENT;
goto out;
+ }
- if (f2fs_is_atomic_file(inode))
+ if (f2fs_is_atomic_file(inode)) {
+ F2FS_I(inode)->i_gc_failures[GC_FAILURE_ATOMIC]++;
+ F2FS_I_SB(inode)->skipped_atomic_files[gc_type]++;
+ err = -EAGAIN;
goto out;
+ }
if (f2fs_is_pinned_file(inode)) {
f2fs_pin_file_control(inode, true);
+ err = -EAGAIN;
goto out;
}
set_new_dnode(&dn, inode, NULL, NULL, 0);
- err = get_dnode_of_data(&dn, bidx, LOOKUP_NODE);
+ err = f2fs_get_dnode_of_data(&dn, bidx, LOOKUP_NODE);
if (err)
goto out;
if (unlikely(dn.data_blkaddr == NULL_ADDR)) {
ClearPageUptodate(page);
+ err = -ENOENT;
goto put_out;
}
@@ -652,16 +768,27 @@ static void move_data_block(struct inode *inode, block_t bidx,
* don't cache encrypted data into meta inode until previous dirty
* data were writebacked to avoid racing between GC and flush.
*/
- f2fs_wait_on_page_writeback(page, DATA, true);
+ f2fs_wait_on_page_writeback(page, DATA, true, true);
- get_node_info(fio.sbi, dn.nid, &ni);
+ f2fs_wait_on_block_writeback(inode, dn.data_blkaddr);
+
+ err = f2fs_get_node_info(fio.sbi, dn.nid, &ni);
+ if (err)
+ goto put_out;
+
set_summary(&sum, dn.nid, dn.ofs_in_node, ni.version);
/* read page */
fio.page = page;
fio.new_blkaddr = fio.old_blkaddr = dn.data_blkaddr;
- allocate_data_block(fio.sbi, NULL, fio.old_blkaddr, &newaddr,
+ /* wait writeback before reading out */
+ f2fs_wait_on_block_writeback(inode, fio.old_blkaddr);
+
+ if (lfs_mode)
+ down_write(&fio.sbi->io_order_lock);
+
+ f2fs_allocate_data_block(fio.sbi, NULL, fio.old_blkaddr, &newaddr,
&sum, CURSEG_COLD_DATA, NULL, false);
fio.encrypted_page = f2fs_pagecache_get_page(META_MAPPING(fio.sbi),
@@ -671,6 +798,23 @@ static void move_data_block(struct inode *inode, block_t bidx,
goto recover_block;
}
+ mpage = f2fs_pagecache_get_page(META_MAPPING(fio.sbi),
+ fio.old_blkaddr, FGP_LOCK, GFP_NOFS);
+ if (mpage) {
+ bool updated = false;
+
+ if (PageUptodate(mpage)) {
+ memcpy(page_address(fio.encrypted_page),
+ page_address(mpage), PAGE_SIZE);
+ updated = true;
+ }
+ f2fs_put_page(mpage, 1);
+ invalidate_mapping_pages(META_MAPPING(fio.sbi),
+ fio.old_blkaddr, fio.old_blkaddr);
+ if (updated)
+ goto write_page;
+ }
+
err = f2fs_submit_page_bio(&fio);
if (err)
goto put_page_out;
@@ -687,21 +831,24 @@ static void move_data_block(struct inode *inode, block_t bidx,
goto put_page_out;
}
+write_page:
+ f2fs_wait_on_page_writeback(fio.encrypted_page, DATA, true, true);
set_page_dirty(fio.encrypted_page);
- f2fs_wait_on_page_writeback(fio.encrypted_page, DATA, true);
if (clear_page_dirty_for_io(fio.encrypted_page))
dec_page_count(fio.sbi, F2FS_DIRTY_META);
set_page_writeback(fio.encrypted_page);
+ ClearPageError(page);
/* allocate block address */
- f2fs_wait_on_page_writeback(dn.node_page, NODE, true);
+ f2fs_wait_on_page_writeback(dn.node_page, NODE, true, true);
fio.op = REQ_OP_WRITE;
fio.op_flags = REQ_SYNC;
fio.new_blkaddr = newaddr;
- err = f2fs_submit_page_write(&fio);
- if (err) {
+ f2fs_submit_page_write(&fio);
+ if (fio.retry) {
+ err = -EAGAIN;
if (PageWriteback(fio.encrypted_page))
end_page_writeback(fio.encrypted_page);
goto put_page_out;
@@ -716,38 +863,51 @@ static void move_data_block(struct inode *inode, block_t bidx,
put_page_out:
f2fs_put_page(fio.encrypted_page, 1);
recover_block:
+ if (lfs_mode)
+ up_write(&fio.sbi->io_order_lock);
if (err)
- __f2fs_replace_block(fio.sbi, &sum, newaddr, fio.old_blkaddr,
+ f2fs_do_replace_block(fio.sbi, &sum, newaddr, fio.old_blkaddr,
true, true);
put_out:
f2fs_put_dnode(&dn);
out:
f2fs_put_page(page, 1);
+ return err;
}
-static void move_data_page(struct inode *inode, block_t bidx, int gc_type,
+static int move_data_page(struct inode *inode, block_t bidx, int gc_type,
unsigned int segno, int off)
{
struct page *page;
+ int err = 0;
- page = get_lock_data_page(inode, bidx, true);
+ page = f2fs_get_lock_data_page(inode, bidx, true);
if (IS_ERR(page))
- return;
+ return PTR_ERR(page);
- if (!check_valid_map(F2FS_I_SB(inode), segno, off))
+ if (!check_valid_map(F2FS_I_SB(inode), segno, off)) {
+ err = -ENOENT;
goto out;
+ }
- if (f2fs_is_atomic_file(inode))
+ if (f2fs_is_atomic_file(inode)) {
+ F2FS_I(inode)->i_gc_failures[GC_FAILURE_ATOMIC]++;
+ F2FS_I_SB(inode)->skipped_atomic_files[gc_type]++;
+ err = -EAGAIN;
goto out;
+ }
if (f2fs_is_pinned_file(inode)) {
if (gc_type == FG_GC)
f2fs_pin_file_control(inode, true);
+ err = -EAGAIN;
goto out;
}
if (gc_type == BG_GC) {
- if (PageWriteback(page))
+ if (PageWriteback(page)) {
+ err = -EAGAIN;
goto out;
+ }
set_page_dirty(page);
set_cold_data(page);
} else {
@@ -765,26 +925,32 @@ static void move_data_page(struct inode *inode, block_t bidx, int gc_type,
.io_type = FS_GC_DATA_IO,
};
bool is_dirty = PageDirty(page);
- int err;
retry:
+ f2fs_wait_on_page_writeback(page, DATA, true, true);
+
set_page_dirty(page);
- f2fs_wait_on_page_writeback(page, DATA, true);
if (clear_page_dirty_for_io(page)) {
inode_dec_dirty_pages(inode);
- remove_dirty_inode(inode);
+ f2fs_remove_dirty_inode(inode);
}
set_cold_data(page);
- err = do_write_data_page(&fio);
- if (err == -ENOMEM && is_dirty) {
- congestion_wait(BLK_RW_ASYNC, HZ/50);
- goto retry;
+ err = f2fs_do_write_data_page(&fio);
+ if (err) {
+ clear_cold_data(page);
+ if (err == -ENOMEM) {
+ congestion_wait(BLK_RW_ASYNC, HZ/50);
+ goto retry;
+ }
+ if (is_dirty)
+ set_page_dirty(page);
}
}
out:
f2fs_put_page(page, 1);
+ return err;
}
/*
@@ -794,7 +960,7 @@ static void move_data_page(struct inode *inode, block_t bidx, int gc_type,
* If the parent node is not valid or the data block address is different,
* the victim data block is ignored.
*/
-static void gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
+static int gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
struct gc_inode_list *gc_list, unsigned int segno, int gc_type)
{
struct super_block *sb = sbi->sb;
@@ -802,6 +968,7 @@ static void gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
block_t start_addr;
int off;
int phase = 0;
+ int submitted = 0;
start_addr = START_BLOCK(sbi, segno);
@@ -818,19 +985,19 @@ static void gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
/* stop BG_GC if there is not enough free sections. */
if (gc_type == BG_GC && has_not_enough_free_secs(sbi, 0, 0))
- return;
+ return submitted;
if (check_valid_map(sbi, segno, off) == 0)
continue;
if (phase == 0) {
- ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1,
+ f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), 1,
META_NAT, true);
continue;
}
if (phase == 1) {
- ra_node_page(sbi, nid);
+ f2fs_ra_node_page(sbi, nid);
continue;
}
@@ -839,7 +1006,7 @@ static void gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
continue;
if (phase == 2) {
- ra_node_page(sbi, dni.ino);
+ f2fs_ra_node_page(sbi, dni.ino);
continue;
}
@@ -850,23 +1017,32 @@ static void gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
if (IS_ERR(inode) || is_bad_inode(inode))
continue;
- /* if inode uses special I/O path, let's go phase 3 */
+ if (!down_write_trylock(
+ &F2FS_I(inode)->i_gc_rwsem[WRITE])) {
+ iput(inode);
+ sbi->skipped_gc_rwsem++;
+ continue;
+ }
+
+ start_bidx = f2fs_start_bidx_of_node(nofs, inode) +
+ ofs_in_node;
+
if (f2fs_post_read_required(inode)) {
+ int err = ra_data_block(inode, start_bidx);
+
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
+ if (err) {
+ iput(inode);
+ continue;
+ }
add_gc_inode(gc_list, inode);
continue;
}
- if (!down_write_trylock(
- &F2FS_I(inode)->dio_rwsem[WRITE])) {
- iput(inode);
- continue;
- }
-
- start_bidx = start_bidx_of_node(nofs, inode);
- data_page = get_read_data_page(inode,
- start_bidx + ofs_in_node, REQ_RAHEAD,
- true);
- up_write(&F2FS_I(inode)->dio_rwsem[WRITE]);
+ data_page = f2fs_get_read_data_page(inode,
+ start_bidx, REQ_RAHEAD,
+ F2FS_GETPAGE_FOR_WRITE);
+ up_write(&F2FS_I(inode)->i_gc_rwsem[WRITE]);
if (IS_ERR(data_page)) {
iput(inode);
continue;
@@ -882,13 +1058,15 @@ static void gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
if (inode) {
struct f2fs_inode_info *fi = F2FS_I(inode);
bool locked = false;
+ int err;
if (S_ISREG(inode->i_mode)) {
- if (!down_write_trylock(&fi->dio_rwsem[READ]))
+ if (!down_write_trylock(&fi->i_gc_rwsem[READ]))
continue;
if (!down_write_trylock(
- &fi->dio_rwsem[WRITE])) {
- up_write(&fi->dio_rwsem[READ]);
+ &fi->i_gc_rwsem[WRITE])) {
+ sbi->skipped_gc_rwsem++;
+ up_write(&fi->i_gc_rwsem[READ]);
continue;
}
locked = true;
@@ -897,17 +1075,22 @@ static void gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
inode_dio_wait(inode);
}
- start_bidx = start_bidx_of_node(nofs, inode)
+ start_bidx = f2fs_start_bidx_of_node(nofs, inode)
+ ofs_in_node;
if (f2fs_post_read_required(inode))
- move_data_block(inode, start_bidx, segno, off);
+ err = move_data_block(inode, start_bidx,
+ gc_type, segno, off);
else
- move_data_page(inode, start_bidx, gc_type,
+ err = move_data_page(inode, start_bidx, gc_type,
segno, off);
+ if (!err && (gc_type == FG_GC ||
+ f2fs_post_read_required(inode)))
+ submitted++;
+
if (locked) {
- up_write(&fi->dio_rwsem[WRITE]);
- up_write(&fi->dio_rwsem[READ]);
+ up_write(&fi->i_gc_rwsem[WRITE]);
+ up_write(&fi->i_gc_rwsem[READ]);
}
stat_inc_data_blk_count(sbi, 1, gc_type);
@@ -916,6 +1099,8 @@ static void gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
if (++phase < 5)
goto next_step;
+
+ return submitted;
}
static int __get_victim(struct f2fs_sb_info *sbi, unsigned int *victim,
@@ -940,18 +1125,34 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi,
struct blk_plug plug;
unsigned int segno = start_segno;
unsigned int end_segno = start_segno + sbi->segs_per_sec;
- int seg_freed = 0;
+ int seg_freed = 0, migrated = 0;
unsigned char type = IS_DATASEG(get_seg_entry(sbi, segno)->type) ?
SUM_TYPE_DATA : SUM_TYPE_NODE;
+ int submitted = 0;
+
+ if (__is_large_section(sbi))
+ end_segno = rounddown(end_segno, sbi->segs_per_sec);
/* readahead multi ssa blocks those have contiguous address */
- if (sbi->segs_per_sec > 1)
- ra_meta_pages(sbi, GET_SUM_BLOCK(sbi, segno),
- sbi->segs_per_sec, META_SSA, true);
+ if (__is_large_section(sbi))
+ f2fs_ra_meta_pages(sbi, GET_SUM_BLOCK(sbi, segno),
+ end_segno - segno, META_SSA, true);
/* reference all summary page */
while (segno < end_segno) {
- sum_page = get_sum_page(sbi, segno++);
+ sum_page = f2fs_get_sum_page(sbi, segno++);
+ if (IS_ERR(sum_page)) {
+ int err = PTR_ERR(sum_page);
+
+ end_segno = segno - 1;
+ for (segno = start_segno; segno < end_segno; segno++) {
+ sum_page = find_get_page(META_MAPPING(sbi),
+ GET_SUM_BLOCK(sbi, segno));
+ f2fs_put_page(sum_page, 0);
+ f2fs_put_page(sum_page, 0);
+ }
+ return err;
+ }
unlock_page(sum_page);
}
@@ -964,13 +1165,22 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi,
GET_SUM_BLOCK(sbi, segno));
f2fs_put_page(sum_page, 0);
- if (get_valid_blocks(sbi, segno, false) == 0 ||
- !PageUptodate(sum_page) ||
- unlikely(f2fs_cp_error(sbi)))
- goto next;
+ if (get_valid_blocks(sbi, segno, false) == 0)
+ goto freed;
+ if (__is_large_section(sbi) &&
+ migrated >= sbi->migration_granularity)
+ goto skip;
+ if (!PageUptodate(sum_page) || unlikely(f2fs_cp_error(sbi)))
+ goto skip;
sum = page_address(sum_page);
- f2fs_bug_on(sbi, type != GET_SUM_TYPE((&sum->footer)));
+ if (type != GET_SUM_TYPE((&sum->footer))) {
+ f2fs_msg(sbi->sb, KERN_ERR, "Inconsistent segment (%u) "
+ "type [%d, %d] in SSA and SIT",
+ segno, type, GET_SUM_TYPE((&sum->footer)));
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ goto skip;
+ }
/*
* this is to avoid deadlock:
@@ -980,21 +1190,27 @@ static int do_garbage_collect(struct f2fs_sb_info *sbi,
* - lock_page(sum_page)
*/
if (type == SUM_TYPE_NODE)
- gc_node_segment(sbi, sum->entries, segno, gc_type);
- else
- gc_data_segment(sbi, sum->entries, gc_list, segno,
+ submitted += gc_node_segment(sbi, sum->entries, segno,
gc_type);
+ else
+ submitted += gc_data_segment(sbi, sum->entries, gc_list,
+ segno, gc_type);
stat_inc_seg_count(sbi, type, gc_type);
+freed:
if (gc_type == FG_GC &&
get_valid_blocks(sbi, segno, false) == 0)
seg_freed++;
-next:
+ migrated++;
+
+ if (__is_large_section(sbi) && segno + 1 < end_segno)
+ sbi->next_victim_seg[gc_type] = segno + 1;
+skip:
f2fs_put_page(sum_page, 0);
}
- if (gc_type == FG_GC)
+ if (submitted)
f2fs_submit_merged_write(sbi,
(type == SUM_TYPE_NODE) ? NODE : DATA);
@@ -1017,6 +1233,9 @@ int f2fs_gc(struct f2fs_sb_info *sbi, bool sync,
.ilist = LIST_HEAD_INIT(gc_list.ilist),
.iroot = RADIX_TREE_INIT(GFP_NOFS),
};
+ unsigned long long last_skipped = sbi->skipped_atomic_files[FG_GC];
+ unsigned long long first_skipped;
+ unsigned int skipped_round = 0, round = 0;
trace_f2fs_gc_begin(sbi->sb, sync, background,
get_pages(sbi, F2FS_DIRTY_NODES),
@@ -1028,6 +1247,8 @@ int f2fs_gc(struct f2fs_sb_info *sbi, bool sync,
prefree_segments(sbi));
cpc.reason = __get_cp_reason(sbi);
+ sbi->skipped_gc_rwsem = 0;
+ first_skipped = last_skipped;
gc_more:
if (unlikely(!(sbi->sb->s_flags & MS_ACTIVE))) {
ret = -EINVAL;
@@ -1044,8 +1265,9 @@ int f2fs_gc(struct f2fs_sb_info *sbi, bool sync,
* threshold, we can make them free by checkpoint. Then, we
* secure free segments which doesn't need fggc any more.
*/
- if (prefree_segments(sbi)) {
- ret = write_checkpoint(sbi, &cpc);
+ if (prefree_segments(sbi) &&
+ !is_sbi_flag_set(sbi, SBI_CP_DISABLED)) {
+ ret = f2fs_write_checkpoint(sbi, &cpc);
if (ret)
goto stop;
}
@@ -1068,17 +1290,36 @@ int f2fs_gc(struct f2fs_sb_info *sbi, bool sync,
sec_freed++;
total_freed += seg_freed;
+ if (gc_type == FG_GC) {
+ if (sbi->skipped_atomic_files[FG_GC] > last_skipped ||
+ sbi->skipped_gc_rwsem)
+ skipped_round++;
+ last_skipped = sbi->skipped_atomic_files[FG_GC];
+ round++;
+ }
+
if (gc_type == FG_GC)
sbi->cur_victim_sec = NULL_SEGNO;
- if (!sync) {
- if (has_not_enough_free_secs(sbi, sec_freed, 0)) {
+ if (sync)
+ goto stop;
+
+ if (has_not_enough_free_secs(sbi, sec_freed, 0)) {
+ if (skipped_round <= MAX_SKIP_GC_COUNT ||
+ skipped_round * 2 < round) {
segno = NULL_SEGNO;
goto gc_more;
}
- if (gc_type == FG_GC)
- ret = write_checkpoint(sbi, &cpc);
+ if (first_skipped < last_skipped &&
+ (last_skipped - first_skipped) >
+ sbi->skipped_gc_rwsem) {
+ f2fs_drop_inmem_pages_all(sbi, true);
+ segno = NULL_SEGNO;
+ goto gc_more;
+ }
+ if (gc_type == FG_GC && !is_sbi_flag_set(sbi, SBI_CP_DISABLED))
+ ret = f2fs_write_checkpoint(sbi, &cpc);
}
stop:
SIT_I(sbi)->last_victim[ALLOC_NEXT] = 0;
@@ -1097,28 +1338,19 @@ int f2fs_gc(struct f2fs_sb_info *sbi, bool sync,
put_gc_inode(&gc_list);
- if (sync)
+ if (sync && !ret)
ret = sec_freed ? 0 : -EAGAIN;
return ret;
}
-void build_gc_manager(struct f2fs_sb_info *sbi)
+void f2fs_build_gc_manager(struct f2fs_sb_info *sbi)
{
- u64 main_count, resv_count, ovp_count;
-
DIRTY_I(sbi)->v_ops = &default_v_ops;
- /* threshold of # of valid blocks in a section for victims of FG_GC */
- main_count = SM_I(sbi)->main_segments << sbi->log_blocks_per_seg;
- resv_count = SM_I(sbi)->reserved_segments << sbi->log_blocks_per_seg;
- ovp_count = SM_I(sbi)->ovp_segments << sbi->log_blocks_per_seg;
-
- sbi->fggc_threshold = div64_u64((main_count - ovp_count) *
- BLKS_PER_SEC(sbi), (main_count - resv_count));
sbi->gc_pin_file_threshold = DEF_GC_FAILED_PINNED_FILES;
/* give warm/cold data area from slower device */
- if (sbi->s_ndevs && sbi->segs_per_sec == 1)
+ if (sbi->s_ndevs && !__is_large_section(sbi))
SIT_I(sbi)->last_victim[ALLOC_NEXT] =
GET_SEGNO(sbi, FDEV(0).end_blk) + 1;
}
diff --git a/fs/f2fs/gc.h b/fs/f2fs/gc.h
index b0045d4..bbac9d3 100644
--- a/fs/f2fs/gc.h
+++ b/fs/f2fs/gc.h
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/gc.h
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#define GC_THREAD_MIN_WB_PAGES 1 /*
* a threshold to determine
@@ -36,8 +33,6 @@ struct f2fs_gc_kthread {
unsigned int no_gc_sleep_time;
/* for changing gc mode */
- unsigned int gc_idle;
- unsigned int gc_urgent;
unsigned int gc_wake;
};
diff --git a/fs/f2fs/hash.c b/fs/f2fs/hash.c
index eb2e031..cc82f14 100644
--- a/fs/f2fs/hash.c
+++ b/fs/f2fs/hash.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/hash.c
*
@@ -7,10 +8,6 @@
* Portions of this code from linux/fs/ext3/hash.c
*
* Copyright (C) 2002 by Theodore Ts'o
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/types.h>
#include <linux/fs.h>
diff --git a/fs/f2fs/inline.c b/fs/f2fs/inline.c
index 26832e7..b8676a2 100644
--- a/fs/f2fs/inline.c
+++ b/fs/f2fs/inline.c
@@ -1,11 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/inline.c
* Copyright (c) 2013, Intel Corporation
* Authors: Huajun Li <huajun.li@intel.com>
* Haicheng Li <haicheng.li@intel.com>
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
@@ -43,7 +41,7 @@ bool f2fs_may_inline_dentry(struct inode *inode)
return true;
}
-void read_inline_data(struct page *page, struct page *ipage)
+void f2fs_do_read_inline_data(struct page *page, struct page *ipage)
{
struct inode *inode = page->mapping->host;
void *src_addr, *dst_addr;
@@ -65,7 +63,8 @@ void read_inline_data(struct page *page, struct page *ipage)
SetPageUptodate(page);
}
-void truncate_inline_inode(struct inode *inode, struct page *ipage, u64 from)
+void f2fs_truncate_inline_inode(struct inode *inode,
+ struct page *ipage, u64 from)
{
void *addr;
@@ -74,7 +73,7 @@ void truncate_inline_inode(struct inode *inode, struct page *ipage, u64 from)
addr = inline_data_addr(inode, ipage);
- f2fs_wait_on_page_writeback(ipage, NODE, true);
+ f2fs_wait_on_page_writeback(ipage, NODE, true, true);
memset(addr + from, 0, MAX_INLINE_DATA(inode) - from);
set_page_dirty(ipage);
@@ -97,7 +96,7 @@ int f2fs_read_inline_data(struct inode *inode, struct page *page)
path, current->comm);
}
- ipage = get_node_page(F2FS_I_SB(inode), inode->i_ino);
+ ipage = f2fs_get_node_page(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ipage)) {
trace_android_fs_dataread_end(inode, page_offset(page),
PAGE_SIZE);
@@ -115,7 +114,7 @@ int f2fs_read_inline_data(struct inode *inode, struct page *page)
if (page->index)
zero_user_segment(page, 0, PAGE_SIZE);
else
- read_inline_data(page, ipage);
+ f2fs_do_read_inline_data(page, ipage);
if (!PageUptodate(page))
SetPageUptodate(page);
@@ -138,6 +137,7 @@ int f2fs_convert_inline_page(struct dnode_of_data *dn, struct page *page)
.encrypted_page = NULL,
.io_type = FS_DATA_IO,
};
+ struct node_info ni;
int dirty, err;
if (!f2fs_exist_data(dn->inode))
@@ -147,9 +147,27 @@ int f2fs_convert_inline_page(struct dnode_of_data *dn, struct page *page)
if (err)
return err;
+ err = f2fs_get_node_info(fio.sbi, dn->nid, &ni);
+ if (err) {
+ f2fs_put_dnode(dn);
+ return err;
+ }
+
+ fio.version = ni.version;
+
+ if (unlikely(dn->data_blkaddr != NEW_ADDR)) {
+ f2fs_put_dnode(dn);
+ set_sbi_flag(fio.sbi, SBI_NEED_FSCK);
+ f2fs_msg(fio.sbi->sb, KERN_WARNING,
+ "%s: corrupted inline inode ino=%lx, i_addr[0]:0x%x, "
+ "run fsck to fix.",
+ __func__, dn->inode->i_ino, dn->data_blkaddr);
+ return -EINVAL;
+ }
+
f2fs_bug_on(F2FS_P_SB(page), PageWriteback(page));
- read_inline_data(page, dn->inode_page);
+ f2fs_do_read_inline_data(page, dn->inode_page);
set_page_dirty(page);
/* clear dirty state */
@@ -157,20 +175,21 @@ int f2fs_convert_inline_page(struct dnode_of_data *dn, struct page *page)
/* write data page to try to make data consistent */
set_page_writeback(page);
+ ClearPageError(page);
fio.old_blkaddr = dn->data_blkaddr;
set_inode_flag(dn->inode, FI_HOT_DATA);
- write_data_page(dn, &fio);
- f2fs_wait_on_page_writeback(page, DATA, true);
+ f2fs_outplace_write_data(dn, &fio);
+ f2fs_wait_on_page_writeback(page, DATA, true, true);
if (dirty) {
inode_dec_dirty_pages(dn->inode);
- remove_dirty_inode(dn->inode);
+ f2fs_remove_dirty_inode(dn->inode);
}
/* this converted inline_data should be recovered. */
set_inode_flag(dn->inode, FI_APPEND_WRITE);
/* clear inline data and flag after data writeback */
- truncate_inline_inode(dn->inode, dn->inode_page, 0);
+ f2fs_truncate_inline_inode(dn->inode, dn->inode_page, 0);
clear_inline_node(dn->inode_page);
clear_out:
stat_dec_inline_inode(dn->inode);
@@ -195,7 +214,7 @@ int f2fs_convert_inline_inode(struct inode *inode)
f2fs_lock_op(sbi);
- ipage = get_node_page(sbi, inode->i_ino);
+ ipage = f2fs_get_node_page(sbi, inode->i_ino);
if (IS_ERR(ipage)) {
err = PTR_ERR(ipage);
goto out;
@@ -221,12 +240,10 @@ int f2fs_write_inline_data(struct inode *inode, struct page *page)
{
void *src_addr, *dst_addr;
struct dnode_of_data dn;
- struct address_space *mapping = page_mapping(page);
- unsigned long flags;
int err;
set_new_dnode(&dn, inode, NULL, NULL, 0);
- err = get_dnode_of_data(&dn, 0, LOOKUP_NODE);
+ err = f2fs_get_dnode_of_data(&dn, 0, LOOKUP_NODE);
if (err)
return err;
@@ -237,17 +254,14 @@ int f2fs_write_inline_data(struct inode *inode, struct page *page)
f2fs_bug_on(F2FS_I_SB(inode), page->index);
- f2fs_wait_on_page_writeback(dn.inode_page, NODE, true);
+ f2fs_wait_on_page_writeback(dn.inode_page, NODE, true, true);
src_addr = kmap_atomic(page);
dst_addr = inline_data_addr(inode, dn.inode_page);
memcpy(dst_addr, src_addr, MAX_INLINE_DATA(inode));
kunmap_atomic(src_addr);
set_page_dirty(dn.inode_page);
- spin_lock_irqsave(&mapping->tree_lock, flags);
- radix_tree_tag_clear(&mapping->page_tree, page_index(page),
- PAGECACHE_TAG_DIRTY);
- spin_unlock_irqrestore(&mapping->tree_lock, flags);
+ f2fs_clear_radix_tree_dirty_tag(page);
set_inode_flag(inode, FI_APPEND_WRITE);
set_inode_flag(inode, FI_DATA_EXIST);
@@ -257,7 +271,7 @@ int f2fs_write_inline_data(struct inode *inode, struct page *page)
return 0;
}
-bool recover_inline_data(struct inode *inode, struct page *npage)
+bool f2fs_recover_inline_data(struct inode *inode, struct page *npage)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_inode *ri = NULL;
@@ -278,10 +292,10 @@ bool recover_inline_data(struct inode *inode, struct page *npage)
if (f2fs_has_inline_data(inode) &&
ri && (ri->i_inline & F2FS_INLINE_DATA)) {
process_inline:
- ipage = get_node_page(sbi, inode->i_ino);
+ ipage = f2fs_get_node_page(sbi, inode->i_ino);
f2fs_bug_on(sbi, IS_ERR(ipage));
- f2fs_wait_on_page_writeback(ipage, NODE, true);
+ f2fs_wait_on_page_writeback(ipage, NODE, true, true);
src_addr = inline_data_addr(inode, npage);
dst_addr = inline_data_addr(inode, ipage);
@@ -296,20 +310,20 @@ bool recover_inline_data(struct inode *inode, struct page *npage)
}
if (f2fs_has_inline_data(inode)) {
- ipage = get_node_page(sbi, inode->i_ino);
+ ipage = f2fs_get_node_page(sbi, inode->i_ino);
f2fs_bug_on(sbi, IS_ERR(ipage));
- truncate_inline_inode(inode, ipage, 0);
+ f2fs_truncate_inline_inode(inode, ipage, 0);
clear_inode_flag(inode, FI_INLINE_DATA);
f2fs_put_page(ipage, 1);
} else if (ri && (ri->i_inline & F2FS_INLINE_DATA)) {
- if (truncate_blocks(inode, 0, false))
+ if (f2fs_truncate_blocks(inode, 0, false, false))
return false;
goto process_inline;
}
return false;
}
-struct f2fs_dir_entry *find_in_inline_dir(struct inode *dir,
+struct f2fs_dir_entry *f2fs_find_in_inline_dir(struct inode *dir,
struct fscrypt_name *fname, struct page **res_page)
{
struct f2fs_sb_info *sbi = F2FS_SB(dir->i_sb);
@@ -320,7 +334,7 @@ struct f2fs_dir_entry *find_in_inline_dir(struct inode *dir,
void *inline_dentry;
f2fs_hash_t namehash;
- ipage = get_node_page(sbi, dir->i_ino);
+ ipage = f2fs_get_node_page(sbi, dir->i_ino);
if (IS_ERR(ipage)) {
*res_page = ipage;
return NULL;
@@ -331,7 +345,7 @@ struct f2fs_dir_entry *find_in_inline_dir(struct inode *dir,
inline_dentry = inline_data_addr(dir, ipage);
make_dentry_ptr_inline(dir, &d, inline_dentry);
- de = find_target_dentry(fname, namehash, NULL, &d);
+ de = f2fs_find_target_dentry(fname, namehash, NULL, &d);
unlock_page(ipage);
if (de)
*res_page = ipage;
@@ -341,7 +355,7 @@ struct f2fs_dir_entry *find_in_inline_dir(struct inode *dir,
return de;
}
-int make_empty_inline_dir(struct inode *inode, struct inode *parent,
+int f2fs_make_empty_inline_dir(struct inode *inode, struct inode *parent,
struct page *ipage)
{
struct f2fs_dentry_ptr d;
@@ -350,7 +364,7 @@ int make_empty_inline_dir(struct inode *inode, struct inode *parent,
inline_dentry = inline_data_addr(inode, ipage);
make_dentry_ptr_inline(inode, &d, inline_dentry);
- do_make_empty_dir(inode, parent, &d);
+ f2fs_do_make_empty_dir(inode, parent, &d);
set_page_dirty(ipage);
@@ -384,8 +398,18 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct page *ipage,
if (err)
goto out;
- f2fs_wait_on_page_writeback(page, DATA, true);
- zero_user_segment(page, MAX_INLINE_DATA(dir), PAGE_SIZE);
+ if (unlikely(dn.data_blkaddr != NEW_ADDR)) {
+ f2fs_put_dnode(&dn);
+ set_sbi_flag(F2FS_P_SB(page), SBI_NEED_FSCK);
+ f2fs_msg(F2FS_P_SB(page)->sb, KERN_WARNING,
+ "%s: corrupted inline inode ino=%lx, i_addr[0]:0x%x, "
+ "run fsck to fix.",
+ __func__, dir->i_ino, dn.data_blkaddr);
+ err = -EINVAL;
+ goto out;
+ }
+
+ f2fs_wait_on_page_writeback(page, DATA, true, true);
dentry_blk = page_address(page);
@@ -409,7 +433,7 @@ static int f2fs_move_inline_dirents(struct inode *dir, struct page *ipage,
set_page_dirty(page);
/* clear inline dir and flag after data writeback */
- truncate_inline_inode(dir, ipage, 0);
+ f2fs_truncate_inline_inode(dir, ipage, 0);
stat_dec_inline_dir(dir);
clear_inode_flag(dir, FI_INLINE_DENTRY);
@@ -452,7 +476,7 @@ static int f2fs_add_inline_entries(struct inode *dir, void *inline_dentry)
new_name.len = le16_to_cpu(de->name_len);
ino = le32_to_cpu(de->ino);
- fake_mode = get_de_type(de) << S_SHIFT;
+ fake_mode = f2fs_get_de_type(de) << S_SHIFT;
err = f2fs_add_regular_entry(dir, &new_name, NULL, NULL,
ino, fake_mode);
@@ -464,8 +488,8 @@ static int f2fs_add_inline_entries(struct inode *dir, void *inline_dentry)
return 0;
punch_dentry_pages:
truncate_inode_pages(&dir->i_data, 0);
- truncate_blocks(dir, 0, false);
- remove_dirty_inode(dir);
+ f2fs_truncate_blocks(dir, 0, false, false);
+ f2fs_remove_dirty_inode(dir);
return err;
}
@@ -483,7 +507,7 @@ static int f2fs_move_rehashed_dirents(struct inode *dir, struct page *ipage,
}
memcpy(backup_dentry, inline_dentry, MAX_INLINE_DATA(dir));
- truncate_inline_inode(dir, ipage, 0);
+ f2fs_truncate_inline_inode(dir, ipage, 0);
unlock_page(ipage);
@@ -495,17 +519,18 @@ static int f2fs_move_rehashed_dirents(struct inode *dir, struct page *ipage,
stat_dec_inline_dir(dir);
clear_inode_flag(dir, FI_INLINE_DENTRY);
- kfree(backup_dentry);
+ kvfree(backup_dentry);
return 0;
recover:
lock_page(ipage);
+ f2fs_wait_on_page_writeback(ipage, NODE, true, true);
memcpy(inline_dentry, backup_dentry, MAX_INLINE_DATA(dir));
f2fs_i_depth_write(dir, 0);
f2fs_i_size_write(dir, MAX_INLINE_DATA(dir));
set_page_dirty(ipage);
f2fs_put_page(ipage, 1);
- kfree(backup_dentry);
+ kvfree(backup_dentry);
return err;
}
@@ -532,14 +557,14 @@ int f2fs_add_inline_entry(struct inode *dir, const struct qstr *new_name,
struct page *page = NULL;
int err = 0;
- ipage = get_node_page(sbi, dir->i_ino);
+ ipage = f2fs_get_node_page(sbi, dir->i_ino);
if (IS_ERR(ipage))
return PTR_ERR(ipage);
inline_dentry = inline_data_addr(dir, ipage);
make_dentry_ptr_inline(dir, &d, inline_dentry);
- bit_pos = room_for_filename(d.bitmap, slots, d.max);
+ bit_pos = f2fs_room_for_filename(d.bitmap, slots, d.max);
if (bit_pos >= d.max) {
err = f2fs_convert_inline_dir(dir, ipage, inline_dentry);
if (err)
@@ -550,7 +575,7 @@ int f2fs_add_inline_entry(struct inode *dir, const struct qstr *new_name,
if (inode) {
down_write(&F2FS_I(inode)->i_sem);
- page = init_inode_metadata(inode, dir, new_name,
+ page = f2fs_init_inode_metadata(inode, dir, new_name,
orig_name, ipage);
if (IS_ERR(page)) {
err = PTR_ERR(page);
@@ -558,7 +583,7 @@ int f2fs_add_inline_entry(struct inode *dir, const struct qstr *new_name,
}
}
- f2fs_wait_on_page_writeback(ipage, NODE, true);
+ f2fs_wait_on_page_writeback(ipage, NODE, true, true);
name_hash = f2fs_dentry_hash(new_name, NULL);
f2fs_update_dentry(ino, mode, &d, new_name, name_hash, bit_pos);
@@ -571,7 +596,7 @@ int f2fs_add_inline_entry(struct inode *dir, const struct qstr *new_name,
f2fs_put_page(page, 1);
}
- update_parent_metadata(dir, inode, 0);
+ f2fs_update_parent_metadata(dir, inode, 0);
fail:
if (inode)
up_write(&F2FS_I(inode)->i_sem);
@@ -590,7 +615,7 @@ void f2fs_delete_inline_entry(struct f2fs_dir_entry *dentry, struct page *page,
int i;
lock_page(page);
- f2fs_wait_on_page_writeback(page, NODE, true);
+ f2fs_wait_on_page_writeback(page, NODE, true, true);
inline_dentry = inline_data_addr(dir, page);
make_dentry_ptr_inline(dir, &d, inline_dentry);
@@ -617,7 +642,7 @@ bool f2fs_empty_inline_dir(struct inode *dir)
void *inline_dentry;
struct f2fs_dentry_ptr d;
- ipage = get_node_page(sbi, dir->i_ino);
+ ipage = f2fs_get_node_page(sbi, dir->i_ino);
if (IS_ERR(ipage))
return false;
@@ -648,7 +673,7 @@ int f2fs_read_inline_dir(struct file *file, struct dir_context *ctx,
if (ctx->pos == d.max)
return 0;
- ipage = get_node_page(F2FS_I_SB(inode), inode->i_ino);
+ ipage = f2fs_get_node_page(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ipage))
return PTR_ERR(ipage);
@@ -674,7 +699,7 @@ int f2fs_inline_data_fiemap(struct inode *inode,
struct page *ipage;
int err = 0;
- ipage = get_node_page(F2FS_I_SB(inode), inode->i_ino);
+ ipage = f2fs_get_node_page(F2FS_I_SB(inode), inode->i_ino);
if (IS_ERR(ipage))
return PTR_ERR(ipage);
@@ -690,7 +715,10 @@ int f2fs_inline_data_fiemap(struct inode *inode,
ilen = start + len;
ilen -= start;
- get_node_info(F2FS_I_SB(inode), inode->i_ino, &ni);
+ err = f2fs_get_node_info(F2FS_I_SB(inode), inode->i_ino, &ni);
+ if (err)
+ goto out;
+
byteaddr = (__u64)ni.blk_addr << inode->i_sb->s_blocksize_bits;
byteaddr += (char *)inline_data_addr(inode, ipage) -
(char *)F2FS_INODE(ipage);
diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c
index e0d9e8f..9a43e26 100644
--- a/fs/f2fs/inode.c
+++ b/fs/f2fs/inode.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/inode.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
@@ -36,15 +33,15 @@ void f2fs_set_inode_flags(struct inode *inode)
unsigned int flags = F2FS_I(inode)->i_flags;
unsigned int new_fl = 0;
- if (flags & FS_SYNC_FL)
+ if (flags & F2FS_SYNC_FL)
new_fl |= S_SYNC;
- if (flags & FS_APPEND_FL)
+ if (flags & F2FS_APPEND_FL)
new_fl |= S_APPEND;
- if (flags & FS_IMMUTABLE_FL)
+ if (flags & F2FS_IMMUTABLE_FL)
new_fl |= S_IMMUTABLE;
- if (flags & FS_NOATIME_FL)
+ if (flags & F2FS_NOATIME_FL)
new_fl |= S_NOATIME;
- if (flags & FS_DIRSYNC_FL)
+ if (flags & F2FS_DIRSYNC_FL)
new_fl |= S_DIRSYNC;
if (f2fs_encrypted_inode(inode))
new_fl |= S_ENCRYPTED;
@@ -68,13 +65,16 @@ static void __get_inode_rdev(struct inode *inode, struct f2fs_inode *ri)
}
}
-static bool __written_first_block(struct f2fs_inode *ri)
+static int __written_first_block(struct f2fs_sb_info *sbi,
+ struct f2fs_inode *ri)
{
block_t addr = le32_to_cpu(ri->i_addr[offset_in_addr(ri)]);
- if (addr != NEW_ADDR && addr != NULL_ADDR)
- return true;
- return false;
+ if (!__is_valid_data_blkaddr(addr))
+ return 1;
+ if (!f2fs_is_valid_blkaddr(sbi, addr, DATA_GENERIC))
+ return -EFAULT;
+ return 0;
}
static void __set_inode_rdev(struct inode *inode, struct f2fs_inode *ri)
@@ -103,7 +103,7 @@ static void __recover_inline_status(struct inode *inode, struct page *ipage)
while (start < end) {
if (*start++) {
- f2fs_wait_on_page_writeback(ipage, NODE, true);
+ f2fs_wait_on_page_writeback(ipage, NODE, true, true);
set_inode_flag(inode, FI_DATA_EXIST);
set_raw_inline(inode, F2FS_INODE(ipage));
@@ -117,15 +117,15 @@ static void __recover_inline_status(struct inode *inode, struct page *ipage)
static bool f2fs_enable_inode_chksum(struct f2fs_sb_info *sbi, struct page *page)
{
struct f2fs_inode *ri = &F2FS_NODE(page)->i;
- int extra_isize = le32_to_cpu(ri->i_extra_isize);
- if (!f2fs_sb_has_inode_chksum(sbi->sb))
+ if (!f2fs_sb_has_inode_chksum(sbi))
return false;
- if (!RAW_IS_INODE(F2FS_NODE(page)) || !(ri->i_inline & F2FS_EXTRA_ATTR))
+ if (!IS_INODE(page) || !(ri->i_inline & F2FS_EXTRA_ATTR))
return false;
- if (!F2FS_FITS_IN_INODE(ri, extra_isize, i_inode_checksum))
+ if (!F2FS_FITS_IN_INODE(ri, le16_to_cpu(ri->i_extra_isize),
+ i_inode_checksum))
return false;
return true;
@@ -159,8 +159,15 @@ bool f2fs_inode_chksum_verify(struct f2fs_sb_info *sbi, struct page *page)
struct f2fs_inode *ri;
__u32 provided, calculated;
+ if (unlikely(is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN)))
+ return true;
+
+#ifdef CONFIG_F2FS_CHECK_FS
+ if (!f2fs_enable_inode_chksum(sbi, page))
+#else
if (!f2fs_enable_inode_chksum(sbi, page) ||
PageDirty(page) || PageWriteback(page))
+#endif
return true;
ri = &F2FS_NODE(page)->i;
@@ -185,6 +192,101 @@ void f2fs_inode_chksum_set(struct f2fs_sb_info *sbi, struct page *page)
ri->i_inode_checksum = cpu_to_le32(f2fs_inode_chksum(sbi, page));
}
+static bool sanity_check_inode(struct inode *inode, struct page *node_page)
+{
+ struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+ struct f2fs_inode_info *fi = F2FS_I(inode);
+ unsigned long long iblocks;
+
+ iblocks = le64_to_cpu(F2FS_INODE(node_page)->i_blocks);
+ if (!iblocks) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "%s: corrupted inode i_blocks i_ino=%lx iblocks=%llu, "
+ "run fsck to fix.",
+ __func__, inode->i_ino, iblocks);
+ return false;
+ }
+
+ if (ino_of_node(node_page) != nid_of_node(node_page)) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "%s: corrupted inode footer i_ino=%lx, ino,nid: "
+ "[%u, %u] run fsck to fix.",
+ __func__, inode->i_ino,
+ ino_of_node(node_page), nid_of_node(node_page));
+ return false;
+ }
+
+ if (f2fs_sb_has_flexible_inline_xattr(sbi)
+ && !f2fs_has_extra_attr(inode)) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "%s: corrupted inode ino=%lx, run fsck to fix.",
+ __func__, inode->i_ino);
+ return false;
+ }
+
+ if (f2fs_has_extra_attr(inode) &&
+ !f2fs_sb_has_extra_attr(sbi)) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "%s: inode (ino=%lx) is with extra_attr, "
+ "but extra_attr feature is off",
+ __func__, inode->i_ino);
+ return false;
+ }
+
+ if (fi->i_extra_isize > F2FS_TOTAL_EXTRA_ATTR_SIZE ||
+ fi->i_extra_isize % sizeof(__le32)) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "%s: inode (ino=%lx) has corrupted i_extra_isize: %d, "
+ "max: %zu",
+ __func__, inode->i_ino, fi->i_extra_isize,
+ F2FS_TOTAL_EXTRA_ATTR_SIZE);
+ return false;
+ }
+
+ if (F2FS_I(inode)->extent_tree) {
+ struct extent_info *ei = &F2FS_I(inode)->extent_tree->largest;
+
+ if (ei->len &&
+ (!f2fs_is_valid_blkaddr(sbi, ei->blk, DATA_GENERIC) ||
+ !f2fs_is_valid_blkaddr(sbi, ei->blk + ei->len - 1,
+ DATA_GENERIC))) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "%s: inode (ino=%lx) extent info [%u, %u, %u] "
+ "is incorrect, run fsck to fix",
+ __func__, inode->i_ino,
+ ei->blk, ei->fofs, ei->len);
+ return false;
+ }
+ }
+
+ if (f2fs_has_inline_data(inode) &&
+ (!S_ISREG(inode->i_mode) && !S_ISLNK(inode->i_mode))) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "%s: inode (ino=%lx, mode=%u) should not have "
+ "inline_data, run fsck to fix",
+ __func__, inode->i_ino, inode->i_mode);
+ return false;
+ }
+
+ if (f2fs_has_inline_dentry(inode) && !S_ISDIR(inode->i_mode)) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "%s: inode (ino=%lx, mode=%u) should not have "
+ "inline_dentry, run fsck to fix",
+ __func__, inode->i_ino, inode->i_mode);
+ return false;
+ }
+
+ return true;
+}
+
static int do_read_inode(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
@@ -192,16 +294,13 @@ static int do_read_inode(struct inode *inode)
struct page *node_page;
struct f2fs_inode *ri;
projid_t i_projid;
+ int err;
/* Check if ino is within scope */
- if (check_nid_range(sbi, inode->i_ino)) {
- f2fs_msg(inode->i_sb, KERN_ERR, "bad inode number: %lu",
- (unsigned long) inode->i_ino);
- WARN_ON(1);
+ if (f2fs_check_nid_range(sbi, inode->i_ino))
return -EINVAL;
- }
- node_page = get_node_page(sbi, inode->i_ino);
+ node_page = f2fs_get_node_page(sbi, inode->i_ino);
if (IS_ERR(node_page))
return PTR_ERR(node_page);
@@ -221,8 +320,11 @@ static int do_read_inode(struct inode *inode)
inode->i_ctime.tv_nsec = le32_to_cpu(ri->i_ctime_nsec);
inode->i_mtime.tv_nsec = le32_to_cpu(ri->i_mtime_nsec);
inode->i_generation = le32_to_cpu(ri->i_generation);
-
- fi->i_current_depth = le32_to_cpu(ri->i_current_depth);
+ if (S_ISDIR(inode->i_mode))
+ fi->i_current_depth = le32_to_cpu(ri->i_current_depth);
+ else if (S_ISREG(inode->i_mode))
+ fi->i_gc_failures[GC_FAILURE_PIN] =
+ le16_to_cpu(ri->i_gc_failures);
fi->i_xattr_nid = le32_to_cpu(ri->i_xattr_nid);
fi->i_flags = le32_to_cpu(ri->i_flags);
fi->flags = 0;
@@ -238,8 +340,7 @@ static int do_read_inode(struct inode *inode)
fi->i_extra_isize = f2fs_has_extra_attr(inode) ?
le16_to_cpu(ri->i_extra_isize) : 0;
- if (f2fs_sb_has_flexible_inline_xattr(sbi->sb)) {
- f2fs_bug_on(sbi, !f2fs_has_extra_attr(inode));
+ if (f2fs_sb_has_flexible_inline_xattr(sbi)) {
fi->i_inline_xattr_size = le16_to_cpu(ri->i_inline_xattr_size);
} else if (f2fs_has_inline_xattr(inode) ||
f2fs_has_inline_dentry(inode)) {
@@ -255,30 +356,48 @@ static int do_read_inode(struct inode *inode)
fi->i_inline_xattr_size = 0;
}
+ if (!sanity_check_inode(inode, node_page)) {
+ f2fs_put_page(node_page, 1);
+ return -EINVAL;
+ }
+
/* check data exist */
if (f2fs_has_inline_data(inode) && !f2fs_exist_data(inode))
__recover_inline_status(inode, node_page);
+ /* try to recover cold bit for non-dir inode */
+ if (!S_ISDIR(inode->i_mode) && !is_cold_node(node_page)) {
+ set_cold_node(node_page, false);
+ set_page_dirty(node_page);
+ }
+
/* get rdev by using inline_info */
__get_inode_rdev(inode, ri);
- if (__written_first_block(ri))
- set_inode_flag(inode, FI_FIRST_BLOCK_WRITTEN);
+ if (S_ISREG(inode->i_mode)) {
+ err = __written_first_block(sbi, ri);
+ if (err < 0) {
+ f2fs_put_page(node_page, 1);
+ return err;
+ }
+ if (!err)
+ set_inode_flag(inode, FI_FIRST_BLOCK_WRITTEN);
+ }
- if (!need_inode_block_update(sbi, inode->i_ino))
+ if (!f2fs_need_inode_block_update(sbi, inode->i_ino))
fi->last_disk_size = inode->i_size;
- if (fi->i_flags & FS_PROJINHERIT_FL)
+ if (fi->i_flags & F2FS_PROJINHERIT_FL)
set_inode_flag(inode, FI_PROJ_INHERIT);
- if (f2fs_has_extra_attr(inode) && f2fs_sb_has_project_quota(sbi->sb) &&
+ if (f2fs_has_extra_attr(inode) && f2fs_sb_has_project_quota(sbi) &&
F2FS_FITS_IN_INODE(ri, fi->i_extra_isize, i_projid))
i_projid = (projid_t)le32_to_cpu(ri->i_projid);
else
i_projid = F2FS_DEF_PROJID;
fi->i_projid = make_kprojid(&init_user_ns, i_projid);
- if (f2fs_has_extra_attr(inode) && f2fs_sb_has_inode_crtime(sbi->sb) &&
+ if (f2fs_has_extra_attr(inode) && f2fs_sb_has_inode_crtime(sbi) &&
F2FS_FITS_IN_INODE(ri, fi->i_extra_isize, i_crtime)) {
fi->i_crtime.tv_sec = le64_to_cpu(ri->i_crtime);
fi->i_crtime.tv_nsec = le32_to_cpu(ri->i_crtime_nsec);
@@ -320,10 +439,10 @@ struct inode *f2fs_iget(struct super_block *sb, unsigned long ino)
make_now:
if (ino == F2FS_NODE_INO(sbi)) {
inode->i_mapping->a_ops = &f2fs_node_aops;
- mapping_set_gfp_mask(inode->i_mapping, GFP_F2FS_ZERO);
+ mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS);
} else if (ino == F2FS_META_INO(sbi)) {
inode->i_mapping->a_ops = &f2fs_meta_aops;
- mapping_set_gfp_mask(inode->i_mapping, GFP_F2FS_ZERO);
+ mapping_set_gfp_mask(inode->i_mapping, GFP_NOFS);
} else if (S_ISREG(inode->i_mode)) {
inode->i_op = &f2fs_file_inode_operations;
inode->i_fop = &f2fs_file_operations;
@@ -373,12 +492,12 @@ struct inode *f2fs_iget_retry(struct super_block *sb, unsigned long ino)
return inode;
}
-void update_inode(struct inode *inode, struct page *node_page)
+void f2fs_update_inode(struct inode *inode, struct page *node_page)
{
struct f2fs_inode *ri;
struct extent_tree *et = F2FS_I(inode)->extent_tree;
- f2fs_wait_on_page_writeback(node_page, NODE, true);
+ f2fs_wait_on_page_writeback(node_page, NODE, true, true);
set_page_dirty(node_page);
f2fs_inode_synced(inode);
@@ -390,7 +509,7 @@ void update_inode(struct inode *inode, struct page *node_page)
ri->i_uid = cpu_to_le32(i_uid_read(inode));
ri->i_gid = cpu_to_le32(i_gid_read(inode));
ri->i_links = cpu_to_le32(inode->i_nlink);
- ri->i_size = cpu_to_le64(i_size_read(inode));
+ ri->i_size = cpu_to_le64(fsverity_full_isize(inode));
ri->i_blocks = cpu_to_le64(SECTOR_TO_BLOCK(inode->i_blocks) + 1);
if (et) {
@@ -408,7 +527,12 @@ void update_inode(struct inode *inode, struct page *node_page)
ri->i_atime_nsec = cpu_to_le32(inode->i_atime.tv_nsec);
ri->i_ctime_nsec = cpu_to_le32(inode->i_ctime.tv_nsec);
ri->i_mtime_nsec = cpu_to_le32(inode->i_mtime.tv_nsec);
- ri->i_current_depth = cpu_to_le32(F2FS_I(inode)->i_current_depth);
+ if (S_ISDIR(inode->i_mode))
+ ri->i_current_depth =
+ cpu_to_le32(F2FS_I(inode)->i_current_depth);
+ else if (S_ISREG(inode->i_mode))
+ ri->i_gc_failures =
+ cpu_to_le16(F2FS_I(inode)->i_gc_failures[GC_FAILURE_PIN]);
ri->i_xattr_nid = cpu_to_le32(F2FS_I(inode)->i_xattr_nid);
ri->i_flags = cpu_to_le32(F2FS_I(inode)->i_flags);
ri->i_pino = cpu_to_le32(F2FS_I(inode)->i_pino);
@@ -418,11 +542,11 @@ void update_inode(struct inode *inode, struct page *node_page)
if (f2fs_has_extra_attr(inode)) {
ri->i_extra_isize = cpu_to_le16(F2FS_I(inode)->i_extra_isize);
- if (f2fs_sb_has_flexible_inline_xattr(F2FS_I_SB(inode)->sb))
+ if (f2fs_sb_has_flexible_inline_xattr(F2FS_I_SB(inode)))
ri->i_inline_xattr_size =
cpu_to_le16(F2FS_I(inode)->i_inline_xattr_size);
- if (f2fs_sb_has_project_quota(F2FS_I_SB(inode)->sb) &&
+ if (f2fs_sb_has_project_quota(F2FS_I_SB(inode)) &&
F2FS_FITS_IN_INODE(ri, F2FS_I(inode)->i_extra_isize,
i_projid)) {
projid_t i_projid;
@@ -432,7 +556,7 @@ void update_inode(struct inode *inode, struct page *node_page)
ri->i_projid = cpu_to_le32(i_projid);
}
- if (f2fs_sb_has_inode_crtime(F2FS_I_SB(inode)->sb) &&
+ if (f2fs_sb_has_inode_crtime(F2FS_I_SB(inode)) &&
F2FS_FITS_IN_INODE(ri, F2FS_I(inode)->i_extra_isize,
i_crtime)) {
ri->i_crtime =
@@ -452,14 +576,18 @@ void update_inode(struct inode *inode, struct page *node_page)
F2FS_I(inode)->i_disk_time[1] = inode->i_ctime;
F2FS_I(inode)->i_disk_time[2] = inode->i_mtime;
F2FS_I(inode)->i_disk_time[3] = F2FS_I(inode)->i_crtime;
+
+#ifdef CONFIG_F2FS_CHECK_FS
+ f2fs_inode_chksum_set(F2FS_I_SB(inode), node_page);
+#endif
}
-void update_inode_page(struct inode *inode)
+void f2fs_update_inode_page(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct page *node_page;
retry:
- node_page = get_node_page(sbi, inode->i_ino);
+ node_page = f2fs_get_node_page(sbi, inode->i_ino);
if (IS_ERR(node_page)) {
int err = PTR_ERR(node_page);
if (err == -ENOMEM) {
@@ -470,7 +598,7 @@ void update_inode_page(struct inode *inode)
}
return;
}
- update_inode(inode, node_page);
+ f2fs_update_inode(inode, node_page);
f2fs_put_page(node_page, 1);
}
@@ -485,11 +613,14 @@ int f2fs_write_inode(struct inode *inode, struct writeback_control *wbc)
if (!is_inode_flag_set(inode, FI_DIRTY_INODE))
return 0;
+ if (f2fs_is_checkpoint_ready(sbi))
+ return -ENOSPC;
+
/*
* We need to balance fs here to prevent from producing dirty node pages
* during the urgent cleaning time when runing out of free sections.
*/
- update_inode_page(inode);
+ f2fs_update_inode_page(inode);
if (wbc && wbc->nr_to_write)
f2fs_balance_fs(sbi, true);
return 0;
@@ -506,7 +637,7 @@ void f2fs_evict_inode(struct inode *inode)
/* some remained atomic pages should discarded */
if (f2fs_is_atomic_file(inode))
- drop_inmem_pages(inode);
+ f2fs_drop_inmem_pages(inode);
trace_f2fs_evict_inode(inode);
truncate_inode_pages_final(&inode->i_data);
@@ -516,18 +647,22 @@ void f2fs_evict_inode(struct inode *inode)
goto out_clear;
f2fs_bug_on(sbi, get_dirty_pages(inode));
- remove_dirty_inode(inode);
+ f2fs_remove_dirty_inode(inode);
f2fs_destroy_extent_tree(inode);
if (inode->i_nlink || is_bad_inode(inode))
goto no_delete;
- dquot_initialize(inode);
+ err = dquot_initialize(inode);
+ if (err) {
+ err = 0;
+ set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
+ }
- remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
- remove_ino_entry(sbi, inode->i_ino, UPDATE_INO);
- remove_ino_entry(sbi, inode->i_ino, FLUSH_INO);
+ f2fs_remove_ino_entry(sbi, inode->i_ino, APPEND_INO);
+ f2fs_remove_ino_entry(sbi, inode->i_ino, UPDATE_INO);
+ f2fs_remove_ino_entry(sbi, inode->i_ino, FLUSH_INO);
sb_start_intwrite(inode->i_sb);
set_inode_flag(inode, FI_NO_ALLOC);
@@ -536,15 +671,14 @@ void f2fs_evict_inode(struct inode *inode)
if (F2FS_HAS_BLOCKS(inode))
err = f2fs_truncate(inode);
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(sbi, FAULT_EVICT_INODE)) {
f2fs_show_injection_info(FAULT_EVICT_INODE);
err = -EIO;
}
-#endif
+
if (!err) {
f2fs_lock_op(sbi);
- err = remove_inode_page(inode);
+ err = f2fs_remove_inode_page(inode);
f2fs_unlock_op(sbi);
if (err == -ENOENT)
err = 0;
@@ -556,9 +690,10 @@ void f2fs_evict_inode(struct inode *inode)
goto retry;
}
- if (err)
- update_inode_page(inode);
- dquot_free_inode(inode);
+ if (err) {
+ f2fs_update_inode_page(inode);
+ set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
+ }
sb_end_intwrite(inode->i_sb);
no_delete:
dquot_drop(inode);
@@ -567,7 +702,8 @@ void f2fs_evict_inode(struct inode *inode)
stat_dec_inline_dir(inode);
stat_dec_inline_inode(inode);
- if (likely(!is_set_ckpt_flags(sbi, CP_ERROR_FLAG)))
+ if (likely(!is_set_ckpt_flags(sbi, CP_ERROR_FLAG) &&
+ !is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
f2fs_bug_on(sbi, is_inode_flag_set(inode, FI_DIRTY_INODE));
else
f2fs_inode_synced(inode);
@@ -580,27 +716,32 @@ void f2fs_evict_inode(struct inode *inode)
invalidate_mapping_pages(NODE_MAPPING(sbi), xnid, xnid);
if (inode->i_nlink) {
if (is_inode_flag_set(inode, FI_APPEND_WRITE))
- add_ino_entry(sbi, inode->i_ino, APPEND_INO);
+ f2fs_add_ino_entry(sbi, inode->i_ino, APPEND_INO);
if (is_inode_flag_set(inode, FI_UPDATE_WRITE))
- add_ino_entry(sbi, inode->i_ino, UPDATE_INO);
+ f2fs_add_ino_entry(sbi, inode->i_ino, UPDATE_INO);
}
if (is_inode_flag_set(inode, FI_FREE_NID)) {
- alloc_nid_failed(sbi, inode->i_ino);
+ f2fs_alloc_nid_failed(sbi, inode->i_ino);
clear_inode_flag(inode, FI_FREE_NID);
} else {
- f2fs_bug_on(sbi, err &&
- !exist_written_data(sbi, inode->i_ino, ORPHAN_INO));
+ /*
+ * If xattr nid is corrupted, we can reach out error condition,
+ * err & !f2fs_exist_written_data(sbi, inode->i_ino, ORPHAN_INO)).
+ * In that case, f2fs_check_nid_range() is enough to give a clue.
+ */
}
out_clear:
fscrypt_put_encryption_info(inode);
+ fsverity_cleanup_inode(inode);
clear_inode(inode);
}
/* caller should call f2fs_lock_op() */
-void handle_failed_inode(struct inode *inode)
+void f2fs_handle_failed_inode(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct node_info ni;
+ int err;
/*
* clear nlink of inode in order to release resource of inode
@@ -612,7 +753,7 @@ void handle_failed_inode(struct inode *inode)
* we must call this to avoid inode being remained as dirty, resulting
* in a panic when flushing dirty inodes in gdirty_list.
*/
- update_inode_page(inode);
+ f2fs_update_inode_page(inode);
f2fs_inode_synced(inode);
/* don't make bad inode, since it becomes a regular file. */
@@ -623,22 +764,29 @@ void handle_failed_inode(struct inode *inode)
* so we can prevent losing this orphan when encoutering checkpoint
* and following suddenly power-off.
*/
- get_node_info(sbi, inode->i_ino, &ni);
+ err = f2fs_get_node_info(sbi, inode->i_ino, &ni);
+ if (err) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "May loss orphan inode, run fsck to fix.");
+ goto out;
+ }
if (ni.blk_addr != NULL_ADDR) {
- int err = acquire_orphan_inode(sbi);
+ err = f2fs_acquire_orphan_inode(sbi);
if (err) {
set_sbi_flag(sbi, SBI_NEED_FSCK);
f2fs_msg(sbi->sb, KERN_WARNING,
"Too many orphan inodes, run fsck to fix.");
} else {
- add_orphan_inode(inode);
+ f2fs_add_orphan_inode(inode);
}
- alloc_nid_done(sbi, inode->i_ino);
+ f2fs_alloc_nid_done(sbi, inode->i_ino);
} else {
set_inode_flag(inode, FI_FREE_NID);
}
+out:
f2fs_unlock_op(sbi);
/* iput will drop the inode object */
diff --git a/fs/f2fs/namei.c b/fs/f2fs/namei.c
index 392d1ed..e5ecc70 100644
--- a/fs/f2fs/namei.c
+++ b/fs/f2fs/namei.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/namei.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
@@ -19,6 +16,7 @@
#include "f2fs.h"
#include "node.h"
+#include "segment.h"
#include "xattr.h"
#include "acl.h"
#include <trace/events/f2fs.h>
@@ -37,7 +35,7 @@ static struct inode *f2fs_new_inode(struct inode *dir, umode_t mode)
return ERR_PTR(-ENOMEM);
f2fs_lock_op(sbi);
- if (!alloc_nid(sbi, &ino)) {
+ if (!f2fs_alloc_nid(sbi, &ino)) {
f2fs_unlock_op(sbi);
err = -ENOSPC;
goto fail;
@@ -50,18 +48,21 @@ static struct inode *f2fs_new_inode(struct inode *dir, umode_t mode)
inode->i_ino = ino;
inode->i_blocks = 0;
- inode->i_mtime = inode->i_atime = inode->i_ctime =
- F2FS_I(inode)->i_crtime = current_time(inode);
+ inode->i_mtime = inode->i_atime = inode->i_ctime = current_time(inode);
+ F2FS_I(inode)->i_crtime = inode->i_mtime;
inode->i_generation = sbi->s_next_generation++;
+ if (S_ISDIR(inode->i_mode))
+ F2FS_I(inode)->i_current_depth = 1;
+
err = insert_inode_locked(inode);
if (err) {
err = -EINVAL;
goto fail;
}
- if (f2fs_sb_has_project_quota(sbi->sb) &&
- (F2FS_I(dir)->i_flags & FS_PROJINHERIT_FL))
+ if (f2fs_sb_has_project_quota(sbi) &&
+ (F2FS_I(dir)->i_flags & F2FS_PROJINHERIT_FL))
F2FS_I(inode)->i_projid = F2FS_I(dir)->i_projid;
else
F2FS_I(inode)->i_projid = make_kprojid(&init_user_ns,
@@ -71,10 +72,6 @@ static struct inode *f2fs_new_inode(struct inode *dir, umode_t mode)
if (err)
goto fail_drop;
- err = dquot_alloc_inode(inode);
- if (err)
- goto fail_drop;
-
set_inode_flag(inode, FI_NEW_INODE);
/* If the directory encrypted, then we should encrypt the inode. */
@@ -82,7 +79,7 @@ static struct inode *f2fs_new_inode(struct inode *dir, umode_t mode)
f2fs_may_encrypt(inode))
f2fs_set_encrypted_inode(inode);
- if (f2fs_sb_has_extra_attr(sbi->sb)) {
+ if (f2fs_sb_has_extra_attr(sbi)) {
set_inode_flag(inode, FI_EXTRA_ATTR);
F2FS_I(inode)->i_extra_isize = F2FS_TOTAL_EXTRA_ATTR_SIZE;
}
@@ -95,7 +92,7 @@ static struct inode *f2fs_new_inode(struct inode *dir, umode_t mode)
if (f2fs_may_inline_dentry(inode))
set_inode_flag(inode, FI_INLINE_DENTRY);
- if (f2fs_sb_has_flexible_inline_xattr(sbi->sb)) {
+ if (f2fs_sb_has_flexible_inline_xattr(sbi)) {
f2fs_bug_on(sbi, !f2fs_has_extra_attr(inode));
if (f2fs_has_inline_xattr(inode))
xattr_size = F2FS_OPTION(sbi).inline_xattr_size;
@@ -116,11 +113,13 @@ static struct inode *f2fs_new_inode(struct inode *dir, umode_t mode)
f2fs_mask_flags(mode, F2FS_I(dir)->i_flags & F2FS_FL_INHERITED);
if (S_ISDIR(inode->i_mode))
- F2FS_I(inode)->i_flags |= FS_INDEX_FL;
+ F2FS_I(inode)->i_flags |= F2FS_INDEX_FL;
- if (F2FS_I(inode)->i_flags & FS_PROJINHERIT_FL)
+ if (F2FS_I(inode)->i_flags & F2FS_PROJINHERIT_FL)
set_inode_flag(inode, FI_PROJ_INHERIT);
+ f2fs_set_inode_flags(inode);
+
trace_f2fs_new_inode(inode, 0);
return inode;
@@ -181,19 +180,22 @@ static inline void set_file_temperature(struct f2fs_sb_info *sbi, struct inode *
hot_count = sbi->raw_super->hot_ext_count;
for (i = 0; i < cold_count + hot_count; i++) {
- if (!is_extension_exist(name, extlist[i]))
- continue;
- if (i < cold_count)
- file_set_cold(inode);
- else
- file_set_hot(inode);
- break;
+ if (is_extension_exist(name, extlist[i]))
+ break;
}
up_read(&sbi->sb_lock);
+
+ if (i == cold_count + hot_count)
+ return;
+
+ if (i < cold_count)
+ file_set_cold(inode);
+ else
+ file_set_hot(inode);
}
-int update_extension_list(struct f2fs_sb_info *sbi, const char *name,
+int f2fs_update_extension_list(struct f2fs_sb_info *sbi, const char *name,
bool hot, bool set)
{
__u8 (*extlist)[F2FS_EXTENSION_LEN] = sbi->raw_super->extension_list;
@@ -243,7 +245,7 @@ int update_extension_list(struct f2fs_sb_info *sbi, const char *name,
return -EINVAL;
if (hot) {
- strncpy(extlist[count], name, strlen(name));
+ memcpy(extlist[count], name, strlen(name));
sbi->raw_super->hot_ext_count = hot_count + 1;
} else {
char buf[F2FS_MAX_EXTENSION][F2FS_EXTENSION_LEN];
@@ -251,7 +253,7 @@ int update_extension_list(struct f2fs_sb_info *sbi, const char *name,
memcpy(buf, &extlist[cold_count],
F2FS_EXTENSION_LEN * hot_count);
memset(extlist[cold_count], 0, F2FS_EXTENSION_LEN);
- strncpy(extlist[cold_count], name, strlen(name));
+ memcpy(extlist[cold_count], name, strlen(name));
memcpy(&extlist[cold_count + 1], buf,
F2FS_EXTENSION_LEN * hot_count);
sbi->raw_super->extension_count = cpu_to_le32(cold_count + 1);
@@ -269,6 +271,9 @@ static int f2fs_create(struct inode *dir, struct dentry *dentry, umode_t mode,
if (unlikely(f2fs_cp_error(sbi)))
return -EIO;
+ err = f2fs_is_checkpoint_ready(sbi);
+ if (err)
+ return err;
err = dquot_initialize(dir);
if (err)
@@ -292,10 +297,9 @@ static int f2fs_create(struct inode *dir, struct dentry *dentry, umode_t mode,
goto out;
f2fs_unlock_op(sbi);
- alloc_nid_done(sbi, ino);
+ f2fs_alloc_nid_done(sbi, ino);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
if (IS_DIRSYNC(dir))
f2fs_sync_fs(sbi->sb, 1);
@@ -303,7 +307,7 @@ static int f2fs_create(struct inode *dir, struct dentry *dentry, umode_t mode,
f2fs_balance_fs(sbi, true);
return 0;
out:
- handle_failed_inode(inode);
+ f2fs_handle_failed_inode(inode);
return err;
}
@@ -316,6 +320,9 @@ static int f2fs_link(struct dentry *old_dentry, struct inode *dir,
if (unlikely(f2fs_cp_error(sbi)))
return -EIO;
+ err = f2fs_is_checkpoint_ready(sbi);
+ if (err)
+ return err;
err = fscrypt_prepare_link(old_dentry, dir, dentry);
if (err)
@@ -398,7 +405,7 @@ static int __recover_dot_dentries(struct inode *dir, nid_t pino)
err = PTR_ERR(page);
goto out;
} else {
- err = __f2fs_add_link(dir, &dot, NULL, dir->i_ino, S_IFDIR);
+ err = f2fs_do_add_link(dir, &dot, NULL, dir->i_ino, S_IFDIR);
if (err)
goto out;
}
@@ -409,7 +416,7 @@ static int __recover_dot_dentries(struct inode *dir, nid_t pino)
else if (IS_ERR(page))
err = PTR_ERR(page);
else
- err = __f2fs_add_link(dir, &dotdot, NULL, pino, S_IFDIR);
+ err = f2fs_do_add_link(dir, &dotdot, NULL, pino, S_IFDIR);
out:
if (!err)
clear_inode_flag(dir, FI_INLINE_DOTS);
@@ -521,7 +528,7 @@ static int f2fs_unlink(struct inode *dir, struct dentry *dentry)
f2fs_balance_fs(sbi, true);
f2fs_lock_op(sbi);
- err = acquire_orphan_inode(sbi);
+ err = f2fs_acquire_orphan_inode(sbi);
if (err) {
f2fs_unlock_op(sbi);
f2fs_put_page(page, 0);
@@ -562,6 +569,9 @@ static int f2fs_symlink(struct inode *dir, struct dentry *dentry,
if (unlikely(f2fs_cp_error(sbi)))
return -EIO;
+ err = f2fs_is_checkpoint_ready(sbi);
+ if (err)
+ return err;
err = fscrypt_prepare_symlink(dir, symname, len, dir->i_sb->s_blocksize,
&disk_link);
@@ -586,9 +596,9 @@ static int f2fs_symlink(struct inode *dir, struct dentry *dentry,
f2fs_lock_op(sbi);
err = f2fs_add_link(dentry, inode);
if (err)
- goto out_handle_failed_inode;
+ goto out_f2fs_handle_failed_inode;
f2fs_unlock_op(sbi);
- alloc_nid_done(sbi, inode->i_ino);
+ f2fs_alloc_nid_done(sbi, inode->i_ino);
err = fscrypt_encrypt_symlink(inode, symname, len, &disk_link);
if (err)
@@ -597,8 +607,7 @@ static int f2fs_symlink(struct inode *dir, struct dentry *dentry,
err = page_symlink(inode, disk_link.name, disk_link.len);
err_out:
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
/*
* Let's flush symlink data in order to avoid broken symlink as much as
@@ -622,11 +631,11 @@ static int f2fs_symlink(struct inode *dir, struct dentry *dentry,
f2fs_balance_fs(sbi, true);
goto out_free_encrypted_link;
-out_handle_failed_inode:
- handle_failed_inode(inode);
+out_f2fs_handle_failed_inode:
+ f2fs_handle_failed_inode(inode);
out_free_encrypted_link:
if (disk_link.name != (unsigned char *)symname)
- kfree(disk_link.name);
+ kvfree(disk_link.name);
return err;
}
@@ -659,10 +668,9 @@ static int f2fs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode)
goto out_fail;
f2fs_unlock_op(sbi);
- alloc_nid_done(sbi, inode->i_ino);
+ f2fs_alloc_nid_done(sbi, inode->i_ino);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
if (IS_DIRSYNC(dir))
f2fs_sync_fs(sbi->sb, 1);
@@ -672,7 +680,7 @@ static int f2fs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode)
out_fail:
clear_inode_flag(inode, FI_INC_LINK);
- handle_failed_inode(inode);
+ f2fs_handle_failed_inode(inode);
return err;
}
@@ -693,6 +701,9 @@ static int f2fs_mknod(struct inode *dir, struct dentry *dentry,
if (unlikely(f2fs_cp_error(sbi)))
return -EIO;
+ err = f2fs_is_checkpoint_ready(sbi);
+ if (err)
+ return err;
err = dquot_initialize(dir);
if (err)
@@ -711,10 +722,9 @@ static int f2fs_mknod(struct inode *dir, struct dentry *dentry,
goto out;
f2fs_unlock_op(sbi);
- alloc_nid_done(sbi, inode->i_ino);
+ f2fs_alloc_nid_done(sbi, inode->i_ino);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
if (IS_DIRSYNC(dir))
f2fs_sync_fs(sbi->sb, 1);
@@ -722,7 +732,7 @@ static int f2fs_mknod(struct inode *dir, struct dentry *dentry,
f2fs_balance_fs(sbi, true);
return 0;
out:
- handle_failed_inode(inode);
+ f2fs_handle_failed_inode(inode);
return err;
}
@@ -751,7 +761,7 @@ static int __f2fs_tmpfile(struct inode *dir, struct dentry *dentry,
}
f2fs_lock_op(sbi);
- err = acquire_orphan_inode(sbi);
+ err = f2fs_acquire_orphan_inode(sbi);
if (err)
goto out;
@@ -763,8 +773,8 @@ static int __f2fs_tmpfile(struct inode *dir, struct dentry *dentry,
* add this non-linked tmpfile to orphan list, in this way we could
* remove all unused data of tmpfile after abnormal power-off.
*/
- add_orphan_inode(inode);
- alloc_nid_done(sbi, inode->i_ino);
+ f2fs_add_orphan_inode(inode);
+ f2fs_alloc_nid_done(sbi, inode->i_ino);
if (whiteout) {
f2fs_i_links_write(inode, false);
@@ -780,9 +790,9 @@ static int __f2fs_tmpfile(struct inode *dir, struct dentry *dentry,
return 0;
release_out:
- release_orphan_inode(sbi);
+ f2fs_release_orphan_inode(sbi);
out:
- handle_failed_inode(inode);
+ f2fs_handle_failed_inode(inode);
return err;
}
@@ -824,10 +834,13 @@ static int f2fs_rename(struct inode *old_dir, struct dentry *old_dentry,
struct f2fs_dir_entry *old_entry;
struct f2fs_dir_entry *new_entry;
bool is_old_inline = f2fs_has_inline_dentry(old_dir);
- int err = -ENOENT;
+ int err;
if (unlikely(f2fs_cp_error(sbi)))
return -EIO;
+ err = f2fs_is_checkpoint_ready(sbi);
+ if (err)
+ return err;
if (is_inode_flag_set(new_dir, FI_PROJ_INHERIT) &&
(!projid_eq(F2FS_I(new_dir)->i_projid,
@@ -848,6 +861,7 @@ static int f2fs_rename(struct inode *old_dir, struct dentry *old_dentry,
goto out;
}
+ err = -ENOENT;
old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_page);
if (!old_entry) {
if (IS_ERR(old_page))
@@ -889,7 +903,7 @@ static int f2fs_rename(struct inode *old_dir, struct dentry *old_dentry,
f2fs_lock_op(sbi);
- err = acquire_orphan_inode(sbi);
+ err = f2fs_acquire_orphan_inode(sbi);
if (err)
goto put_out_dir;
@@ -903,9 +917,9 @@ static int f2fs_rename(struct inode *old_dir, struct dentry *old_dentry,
up_write(&F2FS_I(new_inode)->i_sem);
if (!new_inode->i_nlink)
- add_orphan_inode(new_inode);
+ f2fs_add_orphan_inode(new_inode);
else
- release_orphan_inode(sbi);
+ f2fs_release_orphan_inode(sbi);
} else {
f2fs_balance_fs(sbi, true);
@@ -973,13 +987,19 @@ static int f2fs_rename(struct inode *old_dir, struct dentry *old_dentry,
f2fs_put_page(old_dir_page, 0);
f2fs_i_links_write(old_dir, false);
}
- if (F2FS_OPTION(sbi).fsync_mode == FSYNC_MODE_STRICT)
- add_ino_entry(sbi, new_dir->i_ino, TRANS_DIR_INO);
+ if (F2FS_OPTION(sbi).fsync_mode == FSYNC_MODE_STRICT) {
+ f2fs_add_ino_entry(sbi, new_dir->i_ino, TRANS_DIR_INO);
+ if (S_ISDIR(old_inode->i_mode))
+ f2fs_add_ino_entry(sbi, old_inode->i_ino,
+ TRANS_DIR_INO);
+ }
f2fs_unlock_op(sbi);
if (IS_DIRSYNC(old_dir) || IS_DIRSYNC(new_dir))
f2fs_sync_fs(sbi->sb, 1);
+
+ f2fs_update_time(sbi, REQ_TIME);
return 0;
put_out_dir:
@@ -1009,10 +1029,13 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
struct f2fs_dir_entry *old_dir_entry = NULL, *new_dir_entry = NULL;
struct f2fs_dir_entry *old_entry, *new_entry;
int old_nlink = 0, new_nlink = 0;
- int err = -ENOENT;
+ int err;
if (unlikely(f2fs_cp_error(sbi)))
return -EIO;
+ err = f2fs_is_checkpoint_ready(sbi);
+ if (err)
+ return err;
if ((is_inode_flag_set(new_dir, FI_PROJ_INHERIT) &&
!projid_eq(F2FS_I(new_dir)->i_projid,
@@ -1030,6 +1053,7 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
if (err)
goto out;
+ err = -ENOENT;
old_entry = f2fs_find_entry(old_dir, &old_dentry->d_name, &old_page);
if (!old_entry) {
if (IS_ERR(old_page))
@@ -1125,14 +1149,16 @@ static int f2fs_cross_rename(struct inode *old_dir, struct dentry *old_dentry,
f2fs_mark_inode_dirty_sync(new_dir, false);
if (F2FS_OPTION(sbi).fsync_mode == FSYNC_MODE_STRICT) {
- add_ino_entry(sbi, old_dir->i_ino, TRANS_DIR_INO);
- add_ino_entry(sbi, new_dir->i_ino, TRANS_DIR_INO);
+ f2fs_add_ino_entry(sbi, old_dir->i_ino, TRANS_DIR_INO);
+ f2fs_add_ino_entry(sbi, new_dir->i_ino, TRANS_DIR_INO);
}
f2fs_unlock_op(sbi);
if (IS_DIRSYNC(old_dir) || IS_DIRSYNC(new_dir))
f2fs_sync_fs(sbi->sb, 1);
+
+ f2fs_update_time(sbi, REQ_TIME);
return 0;
out_new_dir:
if (new_dir_entry) {
diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
index ccf410a..46fdaeb 100644
--- a/fs/f2fs/node.c
+++ b/fs/f2fs/node.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/node.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
@@ -23,13 +20,29 @@
#include "trace.h"
#include <trace/events/f2fs.h>
-#define on_build_free_nids(nmi) mutex_is_locked(&(nm_i)->build_lock)
+#define on_f2fs_build_free_nids(nmi) mutex_is_locked(&(nm_i)->build_lock)
static struct kmem_cache *nat_entry_slab;
static struct kmem_cache *free_nid_slab;
static struct kmem_cache *nat_entry_set_slab;
+static struct kmem_cache *fsync_node_entry_slab;
-bool available_free_memory(struct f2fs_sb_info *sbi, int type)
+/*
+ * Check whether the given nid is within node id range.
+ */
+int f2fs_check_nid_range(struct f2fs_sb_info *sbi, nid_t nid)
+{
+ if (unlikely(nid < F2FS_ROOT_INO(sbi) || nid >= NM_I(sbi)->max_nid)) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "%s: out-of-range nid=%x, run fsck to fix.",
+ __func__, nid);
+ return -EINVAL;
+ }
+ return 0;
+}
+
+bool f2fs_available_free_memory(struct f2fs_sb_info *sbi, int type)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct sysinfo val;
@@ -87,44 +100,35 @@ bool available_free_memory(struct f2fs_sb_info *sbi, int type)
static void clear_node_page_dirty(struct page *page)
{
- struct address_space *mapping = page->mapping;
- unsigned int long flags;
-
if (PageDirty(page)) {
- spin_lock_irqsave(&mapping->tree_lock, flags);
- radix_tree_tag_clear(&mapping->page_tree,
- page_index(page),
- PAGECACHE_TAG_DIRTY);
- spin_unlock_irqrestore(&mapping->tree_lock, flags);
-
+ f2fs_clear_radix_tree_dirty_tag(page);
clear_page_dirty_for_io(page);
- dec_page_count(F2FS_M_SB(mapping), F2FS_DIRTY_NODES);
+ dec_page_count(F2FS_P_SB(page), F2FS_DIRTY_NODES);
}
ClearPageUptodate(page);
}
static struct page *get_current_nat_page(struct f2fs_sb_info *sbi, nid_t nid)
{
- pgoff_t index = current_nat_addr(sbi, nid);
- return get_meta_page(sbi, index);
+ return f2fs_get_meta_page_nofail(sbi, current_nat_addr(sbi, nid));
}
static struct page *get_next_nat_page(struct f2fs_sb_info *sbi, nid_t nid)
{
struct page *src_page;
struct page *dst_page;
- pgoff_t src_off;
pgoff_t dst_off;
void *src_addr;
void *dst_addr;
struct f2fs_nm_info *nm_i = NM_I(sbi);
- src_off = current_nat_addr(sbi, nid);
- dst_off = next_nat_addr(sbi, src_off);
+ dst_off = next_nat_addr(sbi, current_nat_addr(sbi, nid));
/* get current nat block page with lock */
- src_page = get_meta_page(sbi, src_off);
- dst_page = grab_meta_page(sbi, dst_off);
+ src_page = get_current_nat_page(sbi, nid);
+ if (IS_ERR(src_page))
+ return src_page;
+ dst_page = f2fs_grab_meta_page(sbi, dst_off);
f2fs_bug_on(sbi, PageDirty(src_page));
src_addr = page_address(src_page);
@@ -169,14 +173,30 @@ static struct nat_entry *__init_nat_entry(struct f2fs_nm_info *nm_i,
if (raw_ne)
node_info_from_raw_nat(&ne->ni, raw_ne);
+
+ spin_lock(&nm_i->nat_list_lock);
list_add_tail(&ne->list, &nm_i->nat_entries);
+ spin_unlock(&nm_i->nat_list_lock);
+
nm_i->nat_cnt++;
return ne;
}
static struct nat_entry *__lookup_nat_cache(struct f2fs_nm_info *nm_i, nid_t n)
{
- return radix_tree_lookup(&nm_i->nat_root, n);
+ struct nat_entry *ne;
+
+ ne = radix_tree_lookup(&nm_i->nat_root, n);
+
+ /* for recent accessed nat entry, move it to tail of lru list */
+ if (ne && !get_nat_flag(ne, IS_DIRTY)) {
+ spin_lock(&nm_i->nat_list_lock);
+ if (!list_empty(&ne->list))
+ list_move_tail(&ne->list, &nm_i->nat_entries);
+ spin_unlock(&nm_i->nat_list_lock);
+ }
+
+ return ne;
}
static unsigned int __gang_lookup_nat_cache(struct f2fs_nm_info *nm_i,
@@ -187,7 +207,6 @@ static unsigned int __gang_lookup_nat_cache(struct f2fs_nm_info *nm_i,
static void __del_from_nat_cache(struct f2fs_nm_info *nm_i, struct nat_entry *e)
{
- list_del(&e->list);
radix_tree_delete(&nm_i->nat_root, nat_get_nid(e));
nm_i->nat_cnt--;
__free_nat_entry(e);
@@ -238,16 +257,21 @@ static void __set_nat_cache_dirty(struct f2fs_nm_info *nm_i,
nm_i->dirty_nat_cnt++;
set_nat_flag(ne, IS_DIRTY, true);
refresh_list:
+ spin_lock(&nm_i->nat_list_lock);
if (new_ne)
list_del_init(&ne->list);
else
list_move_tail(&ne->list, &head->entry_list);
+ spin_unlock(&nm_i->nat_list_lock);
}
static void __clear_nat_cache_dirty(struct f2fs_nm_info *nm_i,
struct nat_entry_set *set, struct nat_entry *ne)
{
+ spin_lock(&nm_i->nat_list_lock);
list_move_tail(&ne->list, &nm_i->nat_entries);
+ spin_unlock(&nm_i->nat_list_lock);
+
set_nat_flag(ne, IS_DIRTY, false);
set->entry_cnt--;
nm_i->dirty_nat_cnt--;
@@ -260,7 +284,73 @@ static unsigned int __gang_lookup_nat_set(struct f2fs_nm_info *nm_i,
start, nr);
}
-int need_dentry_mark(struct f2fs_sb_info *sbi, nid_t nid)
+bool f2fs_in_warm_node_list(struct f2fs_sb_info *sbi, struct page *page)
+{
+ return NODE_MAPPING(sbi) == page->mapping &&
+ IS_DNODE(page) && is_cold_node(page);
+}
+
+void f2fs_init_fsync_node_info(struct f2fs_sb_info *sbi)
+{
+ spin_lock_init(&sbi->fsync_node_lock);
+ INIT_LIST_HEAD(&sbi->fsync_node_list);
+ sbi->fsync_seg_id = 0;
+ sbi->fsync_node_num = 0;
+}
+
+static unsigned int f2fs_add_fsync_node_entry(struct f2fs_sb_info *sbi,
+ struct page *page)
+{
+ struct fsync_node_entry *fn;
+ unsigned long flags;
+ unsigned int seq_id;
+
+ fn = f2fs_kmem_cache_alloc(fsync_node_entry_slab, GFP_NOFS);
+
+ get_page(page);
+ fn->page = page;
+ INIT_LIST_HEAD(&fn->list);
+
+ spin_lock_irqsave(&sbi->fsync_node_lock, flags);
+ list_add_tail(&fn->list, &sbi->fsync_node_list);
+ fn->seq_id = sbi->fsync_seg_id++;
+ seq_id = fn->seq_id;
+ sbi->fsync_node_num++;
+ spin_unlock_irqrestore(&sbi->fsync_node_lock, flags);
+
+ return seq_id;
+}
+
+void f2fs_del_fsync_node_entry(struct f2fs_sb_info *sbi, struct page *page)
+{
+ struct fsync_node_entry *fn;
+ unsigned long flags;
+
+ spin_lock_irqsave(&sbi->fsync_node_lock, flags);
+ list_for_each_entry(fn, &sbi->fsync_node_list, list) {
+ if (fn->page == page) {
+ list_del(&fn->list);
+ sbi->fsync_node_num--;
+ spin_unlock_irqrestore(&sbi->fsync_node_lock, flags);
+ kmem_cache_free(fsync_node_entry_slab, fn);
+ put_page(page);
+ return;
+ }
+ }
+ spin_unlock_irqrestore(&sbi->fsync_node_lock, flags);
+ f2fs_bug_on(sbi, 1);
+}
+
+void f2fs_reset_fsync_node_info(struct f2fs_sb_info *sbi)
+{
+ unsigned long flags;
+
+ spin_lock_irqsave(&sbi->fsync_node_lock, flags);
+ sbi->fsync_seg_id = 0;
+ spin_unlock_irqrestore(&sbi->fsync_node_lock, flags);
+}
+
+int f2fs_need_dentry_mark(struct f2fs_sb_info *sbi, nid_t nid)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct nat_entry *e;
@@ -277,7 +367,7 @@ int need_dentry_mark(struct f2fs_sb_info *sbi, nid_t nid)
return need;
}
-bool is_checkpointed_node(struct f2fs_sb_info *sbi, nid_t nid)
+bool f2fs_is_checkpointed_node(struct f2fs_sb_info *sbi, nid_t nid)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct nat_entry *e;
@@ -291,7 +381,7 @@ bool is_checkpointed_node(struct f2fs_sb_info *sbi, nid_t nid)
return is_cp;
}
-bool need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino)
+bool f2fs_need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct nat_entry *e;
@@ -364,8 +454,7 @@ static void set_node_addr(struct f2fs_sb_info *sbi, struct node_info *ni,
new_blkaddr == NULL_ADDR);
f2fs_bug_on(sbi, nat_get_blkaddr(e) == NEW_ADDR &&
new_blkaddr == NEW_ADDR);
- f2fs_bug_on(sbi, nat_get_blkaddr(e) != NEW_ADDR &&
- nat_get_blkaddr(e) != NULL_ADDR &&
+ f2fs_bug_on(sbi, is_valid_data_blkaddr(sbi, nat_get_blkaddr(e)) &&
new_blkaddr == NEW_ADDR);
/* increment version no as node is removed */
@@ -376,7 +465,7 @@ static void set_node_addr(struct f2fs_sb_info *sbi, struct node_info *ni,
/* change address */
nat_set_blkaddr(e, new_blkaddr);
- if (new_blkaddr == NEW_ADDR || new_blkaddr == NULL_ADDR)
+ if (!is_valid_data_blkaddr(sbi, new_blkaddr))
set_nat_flag(e, IS_CHECKPOINTED, false);
__set_nat_cache_dirty(nm_i, e);
@@ -391,7 +480,7 @@ static void set_node_addr(struct f2fs_sb_info *sbi, struct node_info *ni,
up_write(&nm_i->nat_tree_lock);
}
-int try_to_free_nats(struct f2fs_sb_info *sbi, int nr_shrink)
+int f2fs_try_to_free_nats(struct f2fs_sb_info *sbi, int nr_shrink)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
int nr = nr_shrink;
@@ -399,13 +488,25 @@ int try_to_free_nats(struct f2fs_sb_info *sbi, int nr_shrink)
if (!down_write_trylock(&nm_i->nat_tree_lock))
return 0;
- while (nr_shrink && !list_empty(&nm_i->nat_entries)) {
+ spin_lock(&nm_i->nat_list_lock);
+ while (nr_shrink) {
struct nat_entry *ne;
+
+ if (list_empty(&nm_i->nat_entries))
+ break;
+
ne = list_first_entry(&nm_i->nat_entries,
struct nat_entry, list);
+ list_del(&ne->list);
+ spin_unlock(&nm_i->nat_list_lock);
+
__del_from_nat_cache(nm_i, ne);
nr_shrink--;
+
+ spin_lock(&nm_i->nat_list_lock);
}
+ spin_unlock(&nm_i->nat_list_lock);
+
up_write(&nm_i->nat_tree_lock);
return nr - nr_shrink;
}
@@ -413,7 +514,8 @@ int try_to_free_nats(struct f2fs_sb_info *sbi, int nr_shrink)
/*
* This function always returns success
*/
-void get_node_info(struct f2fs_sb_info *sbi, nid_t nid, struct node_info *ni)
+int f2fs_get_node_info(struct f2fs_sb_info *sbi, nid_t nid,
+ struct node_info *ni)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct curseg_info *curseg = CURSEG_I(sbi, CURSEG_HOT_DATA);
@@ -436,14 +538,14 @@ void get_node_info(struct f2fs_sb_info *sbi, nid_t nid, struct node_info *ni)
ni->blk_addr = nat_get_blkaddr(e);
ni->version = nat_get_version(e);
up_read(&nm_i->nat_tree_lock);
- return;
+ return 0;
}
memset(&ne, 0, sizeof(struct f2fs_nat_entry));
/* Check current segment summary */
down_read(&curseg->journal_rwsem);
- i = lookup_journal_in_cursum(journal, NAT_JOURNAL, nid, 0);
+ i = f2fs_lookup_journal_in_cursum(journal, NAT_JOURNAL, nid, 0);
if (i >= 0) {
ne = nat_in_journal(journal, i);
node_info_from_raw_nat(ni, &ne);
@@ -458,7 +560,10 @@ void get_node_info(struct f2fs_sb_info *sbi, nid_t nid, struct node_info *ni)
index = current_nat_addr(sbi, nid);
up_read(&nm_i->nat_tree_lock);
- page = get_meta_page(sbi, index);
+ page = f2fs_get_meta_page(sbi, index);
+ if (IS_ERR(page))
+ return PTR_ERR(page);
+
nat_blk = (struct f2fs_nat_block *)page_address(page);
ne = nat_blk->entries[nid - start_nid];
node_info_from_raw_nat(ni, &ne);
@@ -466,12 +571,13 @@ void get_node_info(struct f2fs_sb_info *sbi, nid_t nid, struct node_info *ni)
cache:
/* cache nat entry */
cache_nat_entry(sbi, nid, &ne);
+ return 0;
}
/*
* readahead MAX_RA_NODE number of node pages.
*/
-static void ra_node_pages(struct page *parent, int start, int n)
+static void f2fs_ra_node_pages(struct page *parent, int start, int n)
{
struct f2fs_sb_info *sbi = F2FS_P_SB(parent);
struct blk_plug plug;
@@ -485,13 +591,13 @@ static void ra_node_pages(struct page *parent, int start, int n)
end = min(end, NIDS_PER_BLOCK);
for (i = start; i < end; i++) {
nid = get_nid(parent, i, false);
- ra_node_page(sbi, nid);
+ f2fs_ra_node_page(sbi, nid);
}
blk_finish_plug(&plug);
}
-pgoff_t get_next_page_offset(struct dnode_of_data *dn, pgoff_t pgofs)
+pgoff_t f2fs_get_next_page_offset(struct dnode_of_data *dn, pgoff_t pgofs)
{
const long direct_index = ADDRS_PER_INODE(dn->inode);
const long direct_blks = ADDRS_PER_BLOCK;
@@ -606,7 +712,7 @@ static int get_node_path(struct inode *inode, long block,
* f2fs_unlock_op() only if ro is not set RDONLY_NODE.
* In the case of RDONLY_NODE, we don't need to care about mutex.
*/
-int get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode)
+int f2fs_get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode);
struct page *npage[4];
@@ -625,7 +731,7 @@ int get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode)
npage[0] = dn->inode_page;
if (!npage[0]) {
- npage[0] = get_node_page(sbi, nids[0]);
+ npage[0] = f2fs_get_node_page(sbi, nids[0]);
if (IS_ERR(npage[0]))
return PTR_ERR(npage[0]);
}
@@ -649,24 +755,24 @@ int get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode)
if (!nids[i] && mode == ALLOC_NODE) {
/* alloc new node */
- if (!alloc_nid(sbi, &(nids[i]))) {
+ if (!f2fs_alloc_nid(sbi, &(nids[i]))) {
err = -ENOSPC;
goto release_pages;
}
dn->nid = nids[i];
- npage[i] = new_node_page(dn, noffset[i]);
+ npage[i] = f2fs_new_node_page(dn, noffset[i]);
if (IS_ERR(npage[i])) {
- alloc_nid_failed(sbi, nids[i]);
+ f2fs_alloc_nid_failed(sbi, nids[i]);
err = PTR_ERR(npage[i]);
goto release_pages;
}
set_nid(parent, offset[i - 1], nids[i], i == 1);
- alloc_nid_done(sbi, nids[i]);
+ f2fs_alloc_nid_done(sbi, nids[i]);
done = true;
} else if (mode == LOOKUP_NODE_RA && i == level && level > 1) {
- npage[i] = get_node_page_ra(parent, offset[i - 1]);
+ npage[i] = f2fs_get_node_page_ra(parent, offset[i - 1]);
if (IS_ERR(npage[i])) {
err = PTR_ERR(npage[i]);
goto release_pages;
@@ -681,7 +787,7 @@ int get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode)
}
if (!done) {
- npage[i] = get_node_page(sbi, nids[i]);
+ npage[i] = f2fs_get_node_page(sbi, nids[i]);
if (IS_ERR(npage[i])) {
err = PTR_ERR(npage[i]);
f2fs_put_page(npage[0], 0);
@@ -715,20 +821,24 @@ int get_dnode_of_data(struct dnode_of_data *dn, pgoff_t index, int mode)
return err;
}
-static void truncate_node(struct dnode_of_data *dn)
+static int truncate_node(struct dnode_of_data *dn)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode);
struct node_info ni;
+ int err;
+ pgoff_t index;
- get_node_info(sbi, dn->nid, &ni);
+ err = f2fs_get_node_info(sbi, dn->nid, &ni);
+ if (err)
+ return err;
/* Deallocate node address */
- invalidate_blocks(sbi, ni.blk_addr);
+ f2fs_invalidate_blocks(sbi, ni.blk_addr);
dec_valid_node_count(sbi, dn->inode, dn->nid == dn->inode->i_ino);
set_node_addr(sbi, &ni, NULL_ADDR, false);
if (dn->nid == dn->inode->i_ino) {
- remove_orphan_inode(sbi, dn->nid);
+ f2fs_remove_orphan_inode(sbi, dn->nid);
dec_valid_inode_count(sbi);
f2fs_inode_synced(dn->inode);
}
@@ -736,24 +846,28 @@ static void truncate_node(struct dnode_of_data *dn)
clear_node_page_dirty(dn->node_page);
set_sbi_flag(sbi, SBI_IS_DIRTY);
+ index = dn->node_page->index;
f2fs_put_page(dn->node_page, 1);
invalidate_mapping_pages(NODE_MAPPING(sbi),
- dn->node_page->index, dn->node_page->index);
+ index, index);
dn->node_page = NULL;
trace_f2fs_truncate_node(dn->inode, dn->nid, ni.blk_addr);
+
+ return 0;
}
static int truncate_dnode(struct dnode_of_data *dn)
{
struct page *page;
+ int err;
if (dn->nid == 0)
return 1;
/* get direct node */
- page = get_node_page(F2FS_I_SB(dn->inode), dn->nid);
+ page = f2fs_get_node_page(F2FS_I_SB(dn->inode), dn->nid);
if (IS_ERR(page) && PTR_ERR(page) == -ENOENT)
return 1;
else if (IS_ERR(page))
@@ -762,8 +876,11 @@ static int truncate_dnode(struct dnode_of_data *dn)
/* Make dnode_of_data for parameter */
dn->node_page = page;
dn->ofs_in_node = 0;
- truncate_data_blocks(dn);
- truncate_node(dn);
+ f2fs_truncate_data_blocks(dn);
+ err = truncate_node(dn);
+ if (err)
+ return err;
+
return 1;
}
@@ -783,13 +900,13 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs,
trace_f2fs_truncate_nodes_enter(dn->inode, dn->nid, dn->data_blkaddr);
- page = get_node_page(F2FS_I_SB(dn->inode), dn->nid);
+ page = f2fs_get_node_page(F2FS_I_SB(dn->inode), dn->nid);
if (IS_ERR(page)) {
trace_f2fs_truncate_nodes_exit(dn->inode, PTR_ERR(page));
return PTR_ERR(page);
}
- ra_node_pages(page, ofs, NIDS_PER_BLOCK);
+ f2fs_ra_node_pages(page, ofs, NIDS_PER_BLOCK);
rn = F2FS_NODE(page);
if (depth < 3) {
@@ -828,7 +945,9 @@ static int truncate_nodes(struct dnode_of_data *dn, unsigned int nofs,
if (!ofs) {
/* remove current indirect node */
dn->node_page = page;
- truncate_node(dn);
+ ret = truncate_node(dn);
+ if (ret)
+ goto out_err;
freed++;
} else {
f2fs_put_page(page, 1);
@@ -859,7 +978,7 @@ static int truncate_partial_nodes(struct dnode_of_data *dn,
/* get indirect nodes in the path */
for (i = 0; i < idx + 1; i++) {
/* reference count'll be increased */
- pages[i] = get_node_page(F2FS_I_SB(dn->inode), nid[i]);
+ pages[i] = f2fs_get_node_page(F2FS_I_SB(dn->inode), nid[i]);
if (IS_ERR(pages[i])) {
err = PTR_ERR(pages[i]);
idx = i - 1;
@@ -868,7 +987,7 @@ static int truncate_partial_nodes(struct dnode_of_data *dn,
nid[i + 1] = get_nid(pages[i], offset[i + 1], false);
}
- ra_node_pages(pages[idx], offset[idx + 1], NIDS_PER_BLOCK);
+ f2fs_ra_node_pages(pages[idx], offset[idx + 1], NIDS_PER_BLOCK);
/* free direct nodes linked to a partial indirect node */
for (i = offset[idx + 1]; i < NIDS_PER_BLOCK; i++) {
@@ -886,7 +1005,9 @@ static int truncate_partial_nodes(struct dnode_of_data *dn,
if (offset[idx + 1] == 0) {
dn->node_page = pages[idx];
dn->nid = nid[idx];
- truncate_node(dn);
+ err = truncate_node(dn);
+ if (err)
+ goto fail;
} else {
f2fs_put_page(pages[idx], 1);
}
@@ -905,7 +1026,7 @@ static int truncate_partial_nodes(struct dnode_of_data *dn,
/*
* All the block addresses of data and nodes should be nullified.
*/
-int truncate_inode_blocks(struct inode *inode, pgoff_t from)
+int f2fs_truncate_inode_blocks(struct inode *inode, pgoff_t from)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
int err = 0, cont = 1;
@@ -921,7 +1042,7 @@ int truncate_inode_blocks(struct inode *inode, pgoff_t from)
if (level < 0)
return level;
- page = get_node_page(sbi, inode->i_ino);
+ page = f2fs_get_node_page(sbi, inode->i_ino);
if (IS_ERR(page)) {
trace_f2fs_truncate_inode_blocks_exit(inode, PTR_ERR(page));
return PTR_ERR(page);
@@ -985,7 +1106,7 @@ int truncate_inode_blocks(struct inode *inode, pgoff_t from)
ri->i_nid[offset[0] - NODE_DIR1_BLOCK]) {
lock_page(page);
BUG_ON(page->mapping != NODE_MAPPING(sbi));
- f2fs_wait_on_page_writeback(page, NODE, true);
+ f2fs_wait_on_page_writeback(page, NODE, true, true);
ri->i_nid[offset[0] - NODE_DIR1_BLOCK] = 0;
set_page_dirty(page);
unlock_page(page);
@@ -1001,24 +1122,30 @@ int truncate_inode_blocks(struct inode *inode, pgoff_t from)
}
/* caller must lock inode page */
-int truncate_xattr_node(struct inode *inode)
+int f2fs_truncate_xattr_node(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
nid_t nid = F2FS_I(inode)->i_xattr_nid;
struct dnode_of_data dn;
struct page *npage;
+ int err;
if (!nid)
return 0;
- npage = get_node_page(sbi, nid);
+ npage = f2fs_get_node_page(sbi, nid);
if (IS_ERR(npage))
return PTR_ERR(npage);
+ set_new_dnode(&dn, inode, NULL, npage, nid);
+ err = truncate_node(&dn);
+ if (err) {
+ f2fs_put_page(npage, 1);
+ return err;
+ }
+
f2fs_i_xnid_write(inode, 0);
- set_new_dnode(&dn, inode, NULL, npage, nid);
- truncate_node(&dn);
return 0;
}
@@ -1026,17 +1153,17 @@ int truncate_xattr_node(struct inode *inode)
* Caller should grab and release a rwsem by calling f2fs_lock_op() and
* f2fs_unlock_op().
*/
-int remove_inode_page(struct inode *inode)
+int f2fs_remove_inode_page(struct inode *inode)
{
struct dnode_of_data dn;
int err;
set_new_dnode(&dn, inode, NULL, NULL, inode->i_ino);
- err = get_dnode_of_data(&dn, 0, LOOKUP_NODE);
+ err = f2fs_get_dnode_of_data(&dn, 0, LOOKUP_NODE);
if (err)
return err;
- err = truncate_xattr_node(inode);
+ err = f2fs_truncate_xattr_node(inode);
if (err) {
f2fs_put_dnode(&dn);
return err;
@@ -1045,18 +1172,26 @@ int remove_inode_page(struct inode *inode)
/* remove potential inline_data blocks */
if (S_ISREG(inode->i_mode) || S_ISDIR(inode->i_mode) ||
S_ISLNK(inode->i_mode))
- truncate_data_blocks_range(&dn, 1);
+ f2fs_truncate_data_blocks_range(&dn, 1);
/* 0 is possible, after f2fs_new_inode() has failed */
+ if (unlikely(f2fs_cp_error(F2FS_I_SB(inode)))) {
+ f2fs_put_dnode(&dn);
+ return -EIO;
+ }
f2fs_bug_on(F2FS_I_SB(inode),
inode->i_blocks != 0 && inode->i_blocks != 8);
/* will put inode & node pages */
- truncate_node(&dn);
+ err = truncate_node(&dn);
+ if (err) {
+ f2fs_put_dnode(&dn);
+ return err;
+ }
return 0;
}
-struct page *new_inode_page(struct inode *inode)
+struct page *f2fs_new_inode_page(struct inode *inode)
{
struct dnode_of_data dn;
@@ -1064,10 +1199,10 @@ struct page *new_inode_page(struct inode *inode)
set_new_dnode(&dn, inode, NULL, NULL, inode->i_ino);
/* caller should f2fs_put_page(page, 1); */
- return new_node_page(&dn, 0);
+ return f2fs_new_node_page(&dn, 0);
}
-struct page *new_node_page(struct dnode_of_data *dn, unsigned int ofs)
+struct page *f2fs_new_node_page(struct dnode_of_data *dn, unsigned int ofs)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode);
struct node_info new_ni;
@@ -1085,7 +1220,11 @@ struct page *new_node_page(struct dnode_of_data *dn, unsigned int ofs)
goto fail;
#ifdef CONFIG_F2FS_CHECK_FS
- get_node_info(sbi, dn->nid, &new_ni);
+ err = f2fs_get_node_info(sbi, dn->nid, &new_ni);
+ if (err) {
+ dec_valid_node_count(sbi, dn->inode, !ofs);
+ goto fail;
+ }
f2fs_bug_on(sbi, new_ni.blk_addr != NULL_ADDR);
#endif
new_ni.nid = dn->nid;
@@ -1095,7 +1234,7 @@ struct page *new_node_page(struct dnode_of_data *dn, unsigned int ofs)
new_ni.version = 0;
set_node_addr(sbi, &new_ni, NEW_ADDR, false);
- f2fs_wait_on_page_writeback(page, NODE, true);
+ f2fs_wait_on_page_writeback(page, NODE, true, true);
fill_node_footer(page, dn->nid, dn->inode->i_ino, ofs, true);
set_cold_node(page, S_ISDIR(dn->inode->i_mode));
if (!PageUptodate(page))
@@ -1133,13 +1272,21 @@ static int read_node_page(struct page *page, int op_flags)
.page = page,
.encrypted_page = NULL,
};
+ int err;
- if (PageUptodate(page))
+ if (PageUptodate(page)) {
+#ifdef CONFIG_F2FS_CHECK_FS
+ f2fs_bug_on(sbi, !f2fs_inode_chksum_verify(sbi, page));
+#endif
return LOCKED_PAGE;
+ }
- get_node_info(sbi, page->index, &ni);
+ err = f2fs_get_node_info(sbi, page->index, &ni);
+ if (err)
+ return err;
- if (unlikely(ni.blk_addr == NULL_ADDR)) {
+ if (unlikely(ni.blk_addr == NULL_ADDR) ||
+ is_sbi_flag_set(sbi, SBI_IS_SHUTDOWN)) {
ClearPageUptodate(page);
return -ENOENT;
}
@@ -1151,14 +1298,15 @@ static int read_node_page(struct page *page, int op_flags)
/*
* Readahead a node page
*/
-void ra_node_page(struct f2fs_sb_info *sbi, nid_t nid)
+void f2fs_ra_node_page(struct f2fs_sb_info *sbi, nid_t nid)
{
struct page *apage;
int err;
if (!nid)
return;
- f2fs_bug_on(sbi, check_nid_range(sbi, nid));
+ if (f2fs_check_nid_range(sbi, nid))
+ return;
rcu_read_lock();
apage = radix_tree_lookup(&NODE_MAPPING(sbi)->page_tree, nid);
@@ -1182,7 +1330,8 @@ static struct page *__get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid,
if (!nid)
return ERR_PTR(-ENOENT);
- f2fs_bug_on(sbi, check_nid_range(sbi, nid));
+ if (f2fs_check_nid_range(sbi, nid))
+ return ERR_PTR(-EINVAL);
repeat:
page = f2fs_grab_cache_page(NODE_MAPPING(sbi), nid, false);
if (!page)
@@ -1198,7 +1347,7 @@ static struct page *__get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid,
}
if (parent)
- ra_node_pages(parent, start + 1, MAX_RA_NODE);
+ f2fs_ra_node_pages(parent, start + 1, MAX_RA_NODE);
lock_page(page);
@@ -1232,12 +1381,12 @@ static struct page *__get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid,
return page;
}
-struct page *get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid)
+struct page *f2fs_get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid)
{
return __get_node_page(sbi, nid, NULL, 0);
}
-struct page *get_node_page_ra(struct page *parent, int start)
+struct page *f2fs_get_node_page_ra(struct page *parent, int start)
{
struct f2fs_sb_info *sbi = F2FS_P_SB(parent);
nid_t nid = get_nid(parent, start, false);
@@ -1272,7 +1421,7 @@ static void flush_inline_data(struct f2fs_sb_info *sbi, nid_t ino)
ret = f2fs_write_inline_data(inode, page);
inode_dec_dirty_pages(inode);
- remove_dirty_inode(inode);
+ f2fs_remove_dirty_inode(inode);
if (ret)
set_page_dirty(page);
page_out:
@@ -1283,21 +1432,17 @@ static void flush_inline_data(struct f2fs_sb_info *sbi, nid_t ino)
static struct page *last_fsync_dnode(struct f2fs_sb_info *sbi, nid_t ino)
{
- pgoff_t index, end;
+ pgoff_t index;
struct pagevec pvec;
struct page *last_page = NULL;
+ int nr_pages;
pagevec_init(&pvec, 0);
index = 0;
- end = ULONG_MAX;
- while (index <= end) {
- int i, nr_pages;
- nr_pages = pagevec_lookup_tag(&pvec, NODE_MAPPING(sbi), &index,
- PAGECACHE_TAG_DIRTY,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
- if (nr_pages == 0)
- break;
+ while ((nr_pages = pagevec_lookup_tag(&pvec, NODE_MAPPING(sbi), &index,
+ PAGECACHE_TAG_DIRTY))) {
+ int i;
for (i = 0; i < nr_pages; i++) {
struct page *page = pvec.pages[i];
@@ -1343,7 +1488,7 @@ static struct page *last_fsync_dnode(struct f2fs_sb_info *sbi, nid_t ino)
static int __write_node_page(struct page *page, bool atomic, bool *submitted,
struct writeback_control *wbc, bool do_balance,
- enum iostat_type io_type)
+ enum iostat_type io_type, unsigned int *seq_id)
{
struct f2fs_sb_info *sbi = F2FS_P_SB(page);
nid_t nid;
@@ -1360,22 +1505,27 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted,
.io_type = io_type,
.io_wbc = wbc,
};
+ unsigned int seq;
trace_f2fs_writepage(page, NODE);
- if (unlikely(f2fs_cp_error(sbi))) {
- dec_page_count(sbi, F2FS_DIRTY_NODES);
- unlock_page(page);
- return 0;
- }
+ if (unlikely(f2fs_cp_error(sbi)))
+ goto redirty_out;
if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
goto redirty_out;
+ if (wbc->sync_mode == WB_SYNC_NONE &&
+ IS_DNODE(page) && is_cold_node(page))
+ goto redirty_out;
+
/* get old block addr of this node page */
nid = nid_of_node(page);
f2fs_bug_on(sbi, page->index != nid);
+ if (f2fs_get_node_info(sbi, nid, &ni))
+ goto redirty_out;
+
if (wbc->for_reclaim) {
if (!down_read_trylock(&sbi->node_write))
goto redirty_out;
@@ -1383,8 +1533,6 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted,
down_read(&sbi->node_write);
}
- get_node_info(sbi, nid, &ni);
-
/* This page is already truncated */
if (unlikely(ni.blk_addr == NULL_ADDR)) {
ClearPageUptodate(page);
@@ -1394,19 +1542,32 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted,
return 0;
}
+ if (__is_valid_data_blkaddr(ni.blk_addr) &&
+ !f2fs_is_valid_blkaddr(sbi, ni.blk_addr, DATA_GENERIC)) {
+ up_read(&sbi->node_write);
+ goto redirty_out;
+ }
+
if (atomic && !test_opt(sbi, NOBARRIER))
fio.op_flags |= REQ_PREFLUSH | REQ_FUA;
set_page_writeback(page);
+ ClearPageError(page);
+
+ if (f2fs_in_warm_node_list(sbi, page)) {
+ seq = f2fs_add_fsync_node_entry(sbi, page);
+ if (seq_id)
+ *seq_id = seq;
+ }
+
fio.old_blkaddr = ni.blk_addr;
- write_node_page(nid, &fio);
+ f2fs_do_write_node_page(nid, &fio);
set_node_addr(sbi, &ni, fio.new_blkaddr, is_fsync_dnode(page));
dec_page_count(sbi, F2FS_DIRTY_NODES);
up_read(&sbi->node_write);
if (wbc->for_reclaim) {
- f2fs_submit_merged_write_cond(sbi, page->mapping->host, 0,
- page->index, NODE);
+ f2fs_submit_merged_write_cond(sbi, NULL, page, 0, NODE);
submitted = NULL;
}
@@ -1428,8 +1589,10 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted,
return AOP_WRITEPAGE_ACTIVATE;
}
-void move_node_page(struct page *node_page, int gc_type)
+int f2fs_move_node_page(struct page *node_page, int gc_type)
{
+ int err = 0;
+
if (gc_type == FG_GC) {
struct writeback_control wbc = {
.sync_mode = WB_SYNC_ALL,
@@ -1437,16 +1600,20 @@ void move_node_page(struct page *node_page, int gc_type)
.for_reclaim = 0,
};
- set_page_dirty(node_page);
- f2fs_wait_on_page_writeback(node_page, NODE, true);
+ f2fs_wait_on_page_writeback(node_page, NODE, true, true);
- f2fs_bug_on(F2FS_P_SB(node_page), PageWriteback(node_page));
- if (!clear_page_dirty_for_io(node_page))
+ set_page_dirty(node_page);
+
+ if (!clear_page_dirty_for_io(node_page)) {
+ err = -EAGAIN;
goto out_page;
+ }
if (__write_node_page(node_page, false, NULL,
- &wbc, false, FS_GC_NODE_IO))
+ &wbc, false, FS_GC_NODE_IO, NULL)) {
+ err = -EAGAIN;
unlock_page(node_page);
+ }
goto release_page;
} else {
/* set page dirty and write it */
@@ -1457,24 +1624,28 @@ void move_node_page(struct page *node_page, int gc_type)
unlock_page(node_page);
release_page:
f2fs_put_page(node_page, 0);
+ return err;
}
static int f2fs_write_node_page(struct page *page,
struct writeback_control *wbc)
{
- return __write_node_page(page, false, NULL, wbc, false, FS_NODE_IO);
+ return __write_node_page(page, false, NULL, wbc, false,
+ FS_NODE_IO, NULL);
}
-int fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
- struct writeback_control *wbc, bool atomic)
+int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
+ struct writeback_control *wbc, bool atomic,
+ unsigned int *seq_id)
{
- pgoff_t index, end;
- pgoff_t last_idx = ULONG_MAX;
+ pgoff_t index;
struct pagevec pvec;
int ret = 0;
struct page *last_page = NULL;
bool marked = false;
nid_t ino = inode->i_ino;
+ int nr_pages;
+ int nwritten = 0;
if (atomic) {
last_page = last_fsync_dnode(sbi, ino);
@@ -1484,15 +1655,10 @@ int fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
retry:
pagevec_init(&pvec, 0);
index = 0;
- end = ULONG_MAX;
- while (index <= end) {
- int i, nr_pages;
- nr_pages = pagevec_lookup_tag(&pvec, NODE_MAPPING(sbi), &index,
- PAGECACHE_TAG_DIRTY,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
- if (nr_pages == 0)
- break;
+ while ((nr_pages = pagevec_lookup_tag(&pvec, NODE_MAPPING(sbi), &index,
+ PAGECACHE_TAG_DIRTY))) {
+ int i;
for (i = 0; i < nr_pages; i++) {
struct page *page = pvec.pages[i];
@@ -1525,8 +1691,7 @@ int fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
goto continue_unlock;
}
- f2fs_wait_on_page_writeback(page, NODE, true);
- BUG_ON(PageWriteback(page));
+ f2fs_wait_on_page_writeback(page, NODE, true, true);
set_fsync_mark(page, 0);
set_dentry_mark(page, 0);
@@ -1536,9 +1701,9 @@ int fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
if (IS_INODE(page)) {
if (is_inode_flag_set(inode,
FI_DIRTY_INODE))
- update_inode(inode, page);
+ f2fs_update_inode(inode, page);
set_dentry_mark(page,
- need_dentry_mark(sbi, ino));
+ f2fs_need_dentry_mark(sbi, ino));
}
/* may be written by other thread */
if (!PageDirty(page))
@@ -1551,13 +1716,13 @@ int fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
ret = __write_node_page(page, atomic &&
page == last_page,
&submitted, wbc, true,
- FS_NODE_IO);
+ FS_NODE_IO, seq_id);
if (ret) {
unlock_page(page);
f2fs_put_page(last_page, 0);
break;
} else if (submitted) {
- last_idx = page->index;
+ nwritten++;
}
if (page == last_page) {
@@ -1577,44 +1742,48 @@ int fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
"Retry to write fsync mark: ino=%u, idx=%lx",
ino, last_page->index);
lock_page(last_page);
- f2fs_wait_on_page_writeback(last_page, NODE, true);
+ f2fs_wait_on_page_writeback(last_page, NODE, true, true);
set_page_dirty(last_page);
unlock_page(last_page);
goto retry;
}
out:
- if (last_idx != ULONG_MAX)
- f2fs_submit_merged_write_cond(sbi, NULL, ino, last_idx, NODE);
+ if (nwritten)
+ f2fs_submit_merged_write_cond(sbi, NULL, NULL, ino, NODE);
return ret ? -EIO: 0;
}
-int sync_node_pages(struct f2fs_sb_info *sbi, struct writeback_control *wbc,
+int f2fs_sync_node_pages(struct f2fs_sb_info *sbi,
+ struct writeback_control *wbc,
bool do_balance, enum iostat_type io_type)
{
- pgoff_t index, end;
+ pgoff_t index;
struct pagevec pvec;
int step = 0;
int nwritten = 0;
int ret = 0;
+ int nr_pages, done = 0;
pagevec_init(&pvec, 0);
next_step:
index = 0;
- end = ULONG_MAX;
- while (index <= end) {
- int i, nr_pages;
- nr_pages = pagevec_lookup_tag(&pvec, NODE_MAPPING(sbi), &index,
- PAGECACHE_TAG_DIRTY,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
- if (nr_pages == 0)
- break;
+ while (!done && (nr_pages = pagevec_lookup_tag(&pvec,
+ NODE_MAPPING(sbi), &index, PAGECACHE_TAG_DIRTY))) {
+ int i;
for (i = 0; i < nr_pages; i++) {
struct page *page = pvec.pages[i];
bool submitted = false;
+ /* give a priority to WB_SYNC threads */
+ if (atomic_read(&sbi->wb_sync_req[NODE]) &&
+ wbc->sync_mode == WB_SYNC_NONE) {
+ done = 1;
+ break;
+ }
+
/*
* flushing sequence with step:
* 0. indirect nodes
@@ -1630,7 +1799,9 @@ int sync_node_pages(struct f2fs_sb_info *sbi, struct writeback_control *wbc,
!is_cold_node(page)))
continue;
lock_node:
- if (!trylock_page(page))
+ if (wbc->sync_mode == WB_SYNC_ALL)
+ lock_page(page);
+ else if (!trylock_page(page))
continue;
if (unlikely(page->mapping != NODE_MAPPING(sbi))) {
@@ -1652,9 +1823,8 @@ int sync_node_pages(struct f2fs_sb_info *sbi, struct writeback_control *wbc,
goto lock_node;
}
- f2fs_wait_on_page_writeback(page, NODE, true);
+ f2fs_wait_on_page_writeback(page, NODE, true, true);
- BUG_ON(PageWriteback(page));
if (!clear_page_dirty_for_io(page))
goto continue_unlock;
@@ -1662,7 +1832,7 @@ int sync_node_pages(struct f2fs_sb_info *sbi, struct writeback_control *wbc,
set_dentry_mark(page, 0);
ret = __write_node_page(page, false, &submitted,
- wbc, do_balance, io_type);
+ wbc, do_balance, io_type, NULL);
if (ret)
unlock_page(page);
else if (submitted)
@@ -1681,10 +1851,12 @@ int sync_node_pages(struct f2fs_sb_info *sbi, struct writeback_control *wbc,
}
if (step < 2) {
+ if (wbc->sync_mode == WB_SYNC_NONE && step == 1)
+ goto out;
step++;
goto next_step;
}
-
+out:
if (nwritten)
f2fs_submit_merged_write(sbi, NODE);
@@ -1693,42 +1865,46 @@ int sync_node_pages(struct f2fs_sb_info *sbi, struct writeback_control *wbc,
return ret;
}
-int wait_on_node_pages_writeback(struct f2fs_sb_info *sbi, nid_t ino)
+int f2fs_wait_on_node_pages_writeback(struct f2fs_sb_info *sbi,
+ unsigned int seq_id)
{
- pgoff_t index = 0, end = ULONG_MAX;
- struct pagevec pvec;
+ struct fsync_node_entry *fn;
+ struct page *page;
+ struct list_head *head = &sbi->fsync_node_list;
+ unsigned long flags;
+ unsigned int cur_seq_id = 0;
int ret2, ret = 0;
- pagevec_init(&pvec, 0);
-
- while (index <= end) {
- int i, nr_pages;
- nr_pages = pagevec_lookup_tag(&pvec, NODE_MAPPING(sbi), &index,
- PAGECACHE_TAG_WRITEBACK,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
- if (nr_pages == 0)
+ while (seq_id && cur_seq_id < seq_id) {
+ spin_lock_irqsave(&sbi->fsync_node_lock, flags);
+ if (list_empty(head)) {
+ spin_unlock_irqrestore(&sbi->fsync_node_lock, flags);
break;
-
- for (i = 0; i < nr_pages; i++) {
- struct page *page = pvec.pages[i];
-
- /* until radix tree lookup accepts end_index */
- if (unlikely(page->index > end))
- continue;
-
- if (ino && ino_of_node(page) == ino) {
- f2fs_wait_on_page_writeback(page, NODE, true);
- if (TestClearPageError(page))
- ret = -EIO;
- }
}
- pagevec_release(&pvec);
- cond_resched();
+ fn = list_first_entry(head, struct fsync_node_entry, list);
+ if (fn->seq_id > seq_id) {
+ spin_unlock_irqrestore(&sbi->fsync_node_lock, flags);
+ break;
+ }
+ cur_seq_id = fn->seq_id;
+ page = fn->page;
+ get_page(page);
+ spin_unlock_irqrestore(&sbi->fsync_node_lock, flags);
+
+ f2fs_wait_on_page_writeback(page, NODE, true, false);
+ if (TestClearPageError(page))
+ ret = -EIO;
+
+ put_page(page);
+
+ if (ret)
+ break;
}
ret2 = filemap_check_errors(NODE_MAPPING(sbi));
if (!ret)
ret = ret2;
+
return ret;
}
@@ -1749,14 +1925,21 @@ static int f2fs_write_node_pages(struct address_space *mapping,
if (get_pages(sbi, F2FS_DIRTY_NODES) < nr_pages_to_skip(sbi, NODE))
goto skip_write;
+ if (wbc->sync_mode == WB_SYNC_ALL)
+ atomic_inc(&sbi->wb_sync_req[NODE]);
+ else if (atomic_read(&sbi->wb_sync_req[NODE]))
+ goto skip_write;
+
trace_f2fs_writepages(mapping->host, wbc, NODE);
diff = nr_pages_to_write(sbi, NODE, wbc);
- wbc->sync_mode = WB_SYNC_NONE;
blk_start_plug(&plug);
- sync_node_pages(sbi, wbc, true, FS_NODE_IO);
+ f2fs_sync_node_pages(sbi, wbc, true, FS_NODE_IO);
blk_finish_plug(&plug);
wbc->nr_to_write = max((long)0, wbc->nr_to_write - diff);
+
+ if (wbc->sync_mode == WB_SYNC_ALL)
+ atomic_dec(&sbi->wb_sync_req[NODE]);
return 0;
skip_write:
@@ -1771,6 +1954,10 @@ static int f2fs_set_node_page_dirty(struct page *page)
if (!PageUptodate(page))
SetPageUptodate(page);
+#ifdef CONFIG_F2FS_CHECK_FS
+ if (IS_INODE(page))
+ f2fs_inode_chksum_set(F2FS_P_SB(page), page);
+#endif
if (!PageDirty(page)) {
__set_page_dirty_nobuffers(page);
inc_page_count(F2FS_P_SB(page), F2FS_DIRTY_NODES);
@@ -1902,20 +2089,20 @@ static bool add_free_nid(struct f2fs_sb_info *sbi,
* Thread A Thread B
* - f2fs_create
* - f2fs_new_inode
- * - alloc_nid
+ * - f2fs_alloc_nid
* - __insert_nid_to_list(PREALLOC_NID)
* - f2fs_balance_fs_bg
- * - build_free_nids
- * - __build_free_nids
+ * - f2fs_build_free_nids
+ * - __f2fs_build_free_nids
* - scan_nat_page
* - add_free_nid
* - __lookup_nat_cache
* - f2fs_add_link
- * - init_inode_metadata
- * - new_inode_page
- * - new_node_page
+ * - f2fs_init_inode_metadata
+ * - f2fs_new_inode_page
+ * - f2fs_new_node_page
* - set_node_addr
- * - alloc_nid_done
+ * - f2fs_alloc_nid_done
* - __remove_nid_from_list(PREALLOC_NID)
* - __insert_nid_to_list(FREE_NID)
*/
@@ -1965,7 +2152,7 @@ static void remove_free_nid(struct f2fs_sb_info *sbi, nid_t nid)
kmem_cache_free(free_nid_slab, i);
}
-static void scan_nat_page(struct f2fs_sb_info *sbi,
+static int scan_nat_page(struct f2fs_sb_info *sbi,
struct page *nat_page, nid_t start_nid)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
@@ -1983,7 +2170,10 @@ static void scan_nat_page(struct f2fs_sb_info *sbi,
break;
blk_addr = le32_to_cpu(nat_blk->entries[i].block_addr);
- f2fs_bug_on(sbi, blk_addr == NEW_ADDR);
+
+ if (blk_addr == NEW_ADDR)
+ return -EINVAL;
+
if (blk_addr == NULL_ADDR) {
add_free_nid(sbi, start_nid, true, true);
} else {
@@ -1992,6 +2182,8 @@ static void scan_nat_page(struct f2fs_sb_info *sbi,
spin_unlock(&NM_I(sbi)->nid_list_lock);
}
}
+
+ return 0;
}
static void scan_curseg_cache(struct f2fs_sb_info *sbi)
@@ -2047,10 +2239,11 @@ static void scan_free_nid_bits(struct f2fs_sb_info *sbi)
up_read(&nm_i->nat_tree_lock);
}
-static void __build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount)
+static int __f2fs_build_free_nids(struct f2fs_sb_info *sbi,
+ bool sync, bool mount)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
- int i = 0;
+ int i = 0, ret;
nid_t nid = nm_i->next_scan_nid;
if (unlikely(nid >= nm_i->max_nid))
@@ -2058,21 +2251,21 @@ static void __build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount)
/* Enough entries */
if (nm_i->nid_cnt[FREE_NID] >= NAT_ENTRY_PER_BLOCK)
- return;
+ return 0;
- if (!sync && !available_free_memory(sbi, FREE_NIDS))
- return;
+ if (!sync && !f2fs_available_free_memory(sbi, FREE_NIDS))
+ return 0;
if (!mount) {
/* try to find free nids in free_nid_bitmap */
scan_free_nid_bits(sbi);
if (nm_i->nid_cnt[FREE_NID] >= NAT_ENTRY_PER_BLOCK)
- return;
+ return 0;
}
/* readahead nat pages to be scanned */
- ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), FREE_NID_PAGES,
+ f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nid), FREE_NID_PAGES,
META_NAT, true);
down_read(&nm_i->nat_tree_lock);
@@ -2082,8 +2275,20 @@ static void __build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount)
nm_i->nat_block_bitmap)) {
struct page *page = get_current_nat_page(sbi, nid);
- scan_nat_page(sbi, page, nid);
- f2fs_put_page(page, 1);
+ if (IS_ERR(page)) {
+ ret = PTR_ERR(page);
+ } else {
+ ret = scan_nat_page(sbi, page, nid);
+ f2fs_put_page(page, 1);
+ }
+
+ if (ret) {
+ up_read(&nm_i->nat_tree_lock);
+ f2fs_bug_on(sbi, !mount);
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "NAT is corrupt, run fsck to fix it");
+ return ret;
+ }
}
nid += (NAT_ENTRY_PER_BLOCK - (nid % NAT_ENTRY_PER_BLOCK));
@@ -2102,15 +2307,21 @@ static void __build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount)
up_read(&nm_i->nat_tree_lock);
- ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nm_i->next_scan_nid),
+ f2fs_ra_meta_pages(sbi, NAT_BLOCK_OFFSET(nm_i->next_scan_nid),
nm_i->ra_nid_pages, META_NAT, false);
+
+ return 0;
}
-void build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount)
+int f2fs_build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount)
{
+ int ret;
+
mutex_lock(&NM_I(sbi)->build_lock);
- __build_free_nids(sbi, sync, mount);
+ ret = __f2fs_build_free_nids(sbi, sync, mount);
mutex_unlock(&NM_I(sbi)->build_lock);
+
+ return ret;
}
/*
@@ -2118,17 +2329,16 @@ void build_free_nids(struct f2fs_sb_info *sbi, bool sync, bool mount)
* from second parameter of this function.
* The returned nid could be used ino as well as nid when inode is created.
*/
-bool alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid)
+bool f2fs_alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct free_nid *i = NULL;
retry:
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(sbi, FAULT_ALLOC_NID)) {
f2fs_show_injection_info(FAULT_ALLOC_NID);
return false;
}
-#endif
+
spin_lock(&nm_i->nid_list_lock);
if (unlikely(nm_i->available_nids == 0)) {
@@ -2136,8 +2346,8 @@ bool alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid)
return false;
}
- /* We should not use stale free nids created by build_free_nids */
- if (nm_i->nid_cnt[FREE_NID] && !on_build_free_nids(nm_i)) {
+ /* We should not use stale free nids created by f2fs_build_free_nids */
+ if (nm_i->nid_cnt[FREE_NID] && !on_f2fs_build_free_nids(nm_i)) {
f2fs_bug_on(sbi, list_empty(&nm_i->free_nid_list));
i = list_first_entry(&nm_i->free_nid_list,
struct free_nid, list);
@@ -2154,14 +2364,15 @@ bool alloc_nid(struct f2fs_sb_info *sbi, nid_t *nid)
spin_unlock(&nm_i->nid_list_lock);
/* Let's scan nat pages and its caches to get free nids */
- build_free_nids(sbi, true, false);
- goto retry;
+ if (!f2fs_build_free_nids(sbi, true, false))
+ goto retry;
+ return false;
}
/*
- * alloc_nid() should be called prior to this function.
+ * f2fs_alloc_nid() should be called prior to this function.
*/
-void alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid)
+void f2fs_alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct free_nid *i;
@@ -2176,9 +2387,9 @@ void alloc_nid_done(struct f2fs_sb_info *sbi, nid_t nid)
}
/*
- * alloc_nid() should be called prior to this function.
+ * f2fs_alloc_nid() should be called prior to this function.
*/
-void alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid)
+void f2fs_alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct free_nid *i;
@@ -2191,7 +2402,7 @@ void alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid)
i = __lookup_free_nid_list(nm_i, nid);
f2fs_bug_on(sbi, !i);
- if (!available_free_memory(sbi, FREE_NIDS)) {
+ if (!f2fs_available_free_memory(sbi, FREE_NIDS)) {
__remove_free_nid(sbi, i, PREALLOC_NID);
need_free = true;
} else {
@@ -2208,7 +2419,7 @@ void alloc_nid_failed(struct f2fs_sb_info *sbi, nid_t nid)
kmem_cache_free(free_nid_slab, i);
}
-int try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink)
+int f2fs_try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct free_nid *i, *next;
@@ -2236,14 +2447,14 @@ int try_to_free_nids(struct f2fs_sb_info *sbi, int nr_shrink)
return nr - nr_shrink;
}
-void recover_inline_xattr(struct inode *inode, struct page *page)
+void f2fs_recover_inline_xattr(struct inode *inode, struct page *page)
{
void *src_addr, *dst_addr;
size_t inline_size;
struct page *ipage;
struct f2fs_inode *ri;
- ipage = get_node_page(F2FS_I_SB(inode), inode->i_ino);
+ ipage = f2fs_get_node_page(F2FS_I_SB(inode), inode->i_ino);
f2fs_bug_on(F2FS_I_SB(inode), IS_ERR(ipage));
ri = F2FS_INODE(page);
@@ -2258,14 +2469,14 @@ void recover_inline_xattr(struct inode *inode, struct page *page)
src_addr = inline_xattr_addr(inode, page);
inline_size = inline_xattr_size(inode);
- f2fs_wait_on_page_writeback(ipage, NODE, true);
+ f2fs_wait_on_page_writeback(ipage, NODE, true, true);
memcpy(dst_addr, src_addr, inline_size);
update_inode:
- update_inode(inode, ipage);
+ f2fs_update_inode(inode, ipage);
f2fs_put_page(ipage, 1);
}
-int recover_xattr_data(struct inode *inode, struct page *page)
+int f2fs_recover_xattr_data(struct inode *inode, struct page *page)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
nid_t prev_xnid = F2FS_I(inode)->i_xattr_nid;
@@ -2273,30 +2484,34 @@ int recover_xattr_data(struct inode *inode, struct page *page)
struct dnode_of_data dn;
struct node_info ni;
struct page *xpage;
+ int err;
if (!prev_xnid)
goto recover_xnid;
/* 1: invalidate the previous xattr nid */
- get_node_info(sbi, prev_xnid, &ni);
- invalidate_blocks(sbi, ni.blk_addr);
+ err = f2fs_get_node_info(sbi, prev_xnid, &ni);
+ if (err)
+ return err;
+
+ f2fs_invalidate_blocks(sbi, ni.blk_addr);
dec_valid_node_count(sbi, inode, false);
set_node_addr(sbi, &ni, NULL_ADDR, false);
recover_xnid:
/* 2: update xattr nid in inode */
- if (!alloc_nid(sbi, &new_xnid))
+ if (!f2fs_alloc_nid(sbi, &new_xnid))
return -ENOSPC;
set_new_dnode(&dn, inode, NULL, NULL, new_xnid);
- xpage = new_node_page(&dn, XATTR_NODE_OFFSET);
+ xpage = f2fs_new_node_page(&dn, XATTR_NODE_OFFSET);
if (IS_ERR(xpage)) {
- alloc_nid_failed(sbi, new_xnid);
+ f2fs_alloc_nid_failed(sbi, new_xnid);
return PTR_ERR(xpage);
}
- alloc_nid_done(sbi, new_xnid);
- update_inode_page(inode);
+ f2fs_alloc_nid_done(sbi, new_xnid);
+ f2fs_update_inode_page(inode);
/* 3: update and set xattr node page dirty */
memcpy(F2FS_NODE(xpage), F2FS_NODE(page), VALID_XATTR_BLOCK_SIZE);
@@ -2307,14 +2522,17 @@ int recover_xattr_data(struct inode *inode, struct page *page)
return 0;
}
-int recover_inode_page(struct f2fs_sb_info *sbi, struct page *page)
+int f2fs_recover_inode_page(struct f2fs_sb_info *sbi, struct page *page)
{
struct f2fs_inode *src, *dst;
nid_t ino = ino_of_node(page);
struct node_info old_ni, new_ni;
struct page *ipage;
+ int err;
- get_node_info(sbi, ino, &old_ni);
+ err = f2fs_get_node_info(sbi, ino, &old_ni);
+ if (err)
+ return err;
if (unlikely(old_ni.blk_addr != NULL_ADDR))
return -EINVAL;
@@ -2331,7 +2549,7 @@ int recover_inode_page(struct f2fs_sb_info *sbi, struct page *page)
if (!PageUptodate(ipage))
SetPageUptodate(ipage);
fill_node_footer(ipage, ino, ino, 0, true);
- set_cold_node(page, false);
+ set_cold_node(ipage, false);
src = F2FS_INODE(page);
dst = F2FS_INODE(ipage);
@@ -2345,15 +2563,22 @@ int recover_inode_page(struct f2fs_sb_info *sbi, struct page *page)
if (dst->i_inline & F2FS_EXTRA_ATTR) {
dst->i_extra_isize = src->i_extra_isize;
- if (f2fs_sb_has_flexible_inline_xattr(sbi->sb) &&
+ if (f2fs_sb_has_flexible_inline_xattr(sbi) &&
F2FS_FITS_IN_INODE(src, le16_to_cpu(src->i_extra_isize),
i_inline_xattr_size))
dst->i_inline_xattr_size = src->i_inline_xattr_size;
- if (f2fs_sb_has_project_quota(sbi->sb) &&
+ if (f2fs_sb_has_project_quota(sbi) &&
F2FS_FITS_IN_INODE(src, le16_to_cpu(src->i_extra_isize),
i_projid))
dst->i_projid = src->i_projid;
+
+ if (f2fs_sb_has_inode_crtime(sbi) &&
+ F2FS_FITS_IN_INODE(src, le16_to_cpu(src->i_extra_isize),
+ i_crtime_nsec)) {
+ dst->i_crtime = src->i_crtime;
+ dst->i_crtime_nsec = src->i_crtime_nsec;
+ }
}
new_ni = old_ni;
@@ -2368,7 +2593,7 @@ int recover_inode_page(struct f2fs_sb_info *sbi, struct page *page)
return 0;
}
-void restore_node_summary(struct f2fs_sb_info *sbi,
+int f2fs_restore_node_summary(struct f2fs_sb_info *sbi,
unsigned int segno, struct f2fs_summary_block *sum)
{
struct f2fs_node *rn;
@@ -2385,10 +2610,13 @@ void restore_node_summary(struct f2fs_sb_info *sbi,
nrpages = min(last_offset - i, BIO_MAX_PAGES);
/* readahead node pages */
- ra_meta_pages(sbi, addr, nrpages, META_POR, true);
+ f2fs_ra_meta_pages(sbi, addr, nrpages, META_POR, true);
for (idx = addr; idx < addr + nrpages; idx++) {
- struct page *page = get_tmp_page(sbi, idx);
+ struct page *page = f2fs_get_tmp_page(sbi, idx);
+
+ if (IS_ERR(page))
+ return PTR_ERR(page);
rn = F2FS_NODE(page);
sum_entry->nid = rn->footer.nid;
@@ -2401,6 +2629,7 @@ void restore_node_summary(struct f2fs_sb_info *sbi,
invalidate_mapping_pages(META_MAPPING(sbi), addr,
addr + nrpages);
}
+ return 0;
}
static void remove_nats_in_journal(struct f2fs_sb_info *sbi)
@@ -2493,7 +2722,7 @@ static void __update_nat_bits(struct f2fs_sb_info *sbi, nid_t start_nid,
__clear_bit_le(nat_index, nm_i->full_nat_bits);
}
-static void __flush_nat_entry_set(struct f2fs_sb_info *sbi,
+static int __flush_nat_entry_set(struct f2fs_sb_info *sbi,
struct nat_entry_set *set, struct cp_control *cpc)
{
struct curseg_info *curseg = CURSEG_I(sbi, CURSEG_HOT_DATA);
@@ -2517,6 +2746,9 @@ static void __flush_nat_entry_set(struct f2fs_sb_info *sbi,
down_write(&curseg->journal_rwsem);
} else {
page = get_next_nat_page(sbi, start_nid);
+ if (IS_ERR(page))
+ return PTR_ERR(page);
+
nat_blk = page_address(page);
f2fs_bug_on(sbi, !nat_blk);
}
@@ -2530,7 +2762,7 @@ static void __flush_nat_entry_set(struct f2fs_sb_info *sbi,
f2fs_bug_on(sbi, nat_get_blkaddr(ne) == NEW_ADDR);
if (to_journal) {
- offset = lookup_journal_in_cursum(journal,
+ offset = f2fs_lookup_journal_in_cursum(journal,
NAT_JOURNAL, nid, 1);
f2fs_bug_on(sbi, offset < 0);
raw_ne = &nat_in_journal(journal, offset);
@@ -2562,12 +2794,13 @@ static void __flush_nat_entry_set(struct f2fs_sb_info *sbi,
radix_tree_delete(&NM_I(sbi)->nat_set_root, set->set);
kmem_cache_free(nat_entry_set_slab, set);
}
+ return 0;
}
/*
* This function is called during the checkpointing process.
*/
-void flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
+int f2fs_flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct curseg_info *curseg = CURSEG_I(sbi, CURSEG_HOT_DATA);
@@ -2577,9 +2810,17 @@ void flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
unsigned int found;
nid_t set_idx = 0;
LIST_HEAD(sets);
+ int err = 0;
+
+ /* during unmount, let's flush nat_bits before checking dirty_nat_cnt */
+ if (enabled_nat_bits(sbi, cpc)) {
+ down_write(&nm_i->nat_tree_lock);
+ remove_nats_in_journal(sbi);
+ up_write(&nm_i->nat_tree_lock);
+ }
if (!nm_i->dirty_nat_cnt)
- return;
+ return 0;
down_write(&nm_i->nat_tree_lock);
@@ -2602,11 +2843,16 @@ void flush_nat_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
}
/* flush dirty nats in nat entry set */
- list_for_each_entry_safe(set, tmp, &sets, set_list)
- __flush_nat_entry_set(sbi, set, cpc);
+ list_for_each_entry_safe(set, tmp, &sets, set_list) {
+ err = __flush_nat_entry_set(sbi, set, cpc);
+ if (err)
+ break;
+ }
up_write(&nm_i->nat_tree_lock);
/* Allow dirty nats by node block allocation in write_begin */
+
+ return err;
}
static int __get_nat_bitmaps(struct f2fs_sb_info *sbi)
@@ -2630,7 +2876,11 @@ static int __get_nat_bitmaps(struct f2fs_sb_info *sbi)
nat_bits_addr = __start_cp_addr(sbi) + sbi->blocks_per_seg -
nm_i->nat_bits_blocks;
for (i = 0; i < nm_i->nat_bits_blocks; i++) {
- struct page *page = get_meta_page(sbi, nat_bits_addr++);
+ struct page *page;
+
+ page = f2fs_get_meta_page(sbi, nat_bits_addr++);
+ if (IS_ERR(page))
+ return PTR_ERR(page);
memcpy(nm_i->nat_bits + (i << F2FS_BLKSIZE_BITS),
page_address(page), F2FS_BLKSIZE);
@@ -2714,6 +2964,7 @@ static int init_node_manager(struct f2fs_sb_info *sbi)
INIT_RADIX_TREE(&nm_i->nat_root, GFP_NOIO);
INIT_RADIX_TREE(&nm_i->nat_set_root, GFP_NOIO);
INIT_LIST_HEAD(&nm_i->nat_entries);
+ spin_lock_init(&nm_i->nat_list_lock);
mutex_init(&nm_i->build_lock);
spin_lock_init(&nm_i->nid_list_lock);
@@ -2749,15 +3000,17 @@ static int init_free_nid_cache(struct f2fs_sb_info *sbi)
struct f2fs_nm_info *nm_i = NM_I(sbi);
int i;
- nm_i->free_nid_bitmap = f2fs_kzalloc(sbi, nm_i->nat_blocks *
- sizeof(unsigned char *), GFP_KERNEL);
+ nm_i->free_nid_bitmap =
+ f2fs_kzalloc(sbi, array_size(sizeof(unsigned char *),
+ nm_i->nat_blocks),
+ GFP_KERNEL);
if (!nm_i->free_nid_bitmap)
return -ENOMEM;
for (i = 0; i < nm_i->nat_blocks; i++) {
nm_i->free_nid_bitmap[i] = f2fs_kvzalloc(sbi,
- NAT_ENTRY_BITMAP_SIZE_ALIGNED, GFP_KERNEL);
- if (!nm_i->free_nid_bitmap)
+ f2fs_bitmap_size(NAT_ENTRY_PER_BLOCK), GFP_KERNEL);
+ if (!nm_i->free_nid_bitmap[i])
return -ENOMEM;
}
@@ -2766,14 +3019,16 @@ static int init_free_nid_cache(struct f2fs_sb_info *sbi)
if (!nm_i->nat_block_bitmap)
return -ENOMEM;
- nm_i->free_nid_count = f2fs_kvzalloc(sbi, nm_i->nat_blocks *
- sizeof(unsigned short), GFP_KERNEL);
+ nm_i->free_nid_count =
+ f2fs_kvzalloc(sbi, array_size(sizeof(unsigned short),
+ nm_i->nat_blocks),
+ GFP_KERNEL);
if (!nm_i->free_nid_count)
return -ENOMEM;
return 0;
}
-int build_node_manager(struct f2fs_sb_info *sbi)
+int f2fs_build_node_manager(struct f2fs_sb_info *sbi)
{
int err;
@@ -2793,11 +3048,10 @@ int build_node_manager(struct f2fs_sb_info *sbi)
/* load free nid status from nat_bits table */
load_free_nid_bitmap(sbi);
- build_free_nids(sbi, true, true);
- return 0;
+ return f2fs_build_free_nids(sbi, true, true);
}
-void destroy_node_manager(struct f2fs_sb_info *sbi)
+void f2fs_destroy_node_manager(struct f2fs_sb_info *sbi)
{
struct f2fs_nm_info *nm_i = NM_I(sbi);
struct free_nid *i, *next_i;
@@ -2829,8 +3083,13 @@ void destroy_node_manager(struct f2fs_sb_info *sbi)
unsigned idx;
nid = nat_get_nid(natvec[found - 1]) + 1;
- for (idx = 0; idx < found; idx++)
+ for (idx = 0; idx < found; idx++) {
+ spin_lock(&nm_i->nat_list_lock);
+ list_del(&natvec[idx]->list);
+ spin_unlock(&nm_i->nat_list_lock);
+
__del_from_nat_cache(nm_i, natvec[idx]);
+ }
}
f2fs_bug_on(sbi, nm_i->nat_cnt);
@@ -2856,20 +3115,20 @@ void destroy_node_manager(struct f2fs_sb_info *sbi)
for (i = 0; i < nm_i->nat_blocks; i++)
kvfree(nm_i->free_nid_bitmap[i]);
- kfree(nm_i->free_nid_bitmap);
+ kvfree(nm_i->free_nid_bitmap);
}
kvfree(nm_i->free_nid_count);
- kfree(nm_i->nat_bitmap);
- kfree(nm_i->nat_bits);
+ kvfree(nm_i->nat_bitmap);
+ kvfree(nm_i->nat_bits);
#ifdef CONFIG_F2FS_CHECK_FS
- kfree(nm_i->nat_bitmap_mir);
+ kvfree(nm_i->nat_bitmap_mir);
#endif
sbi->nm_info = NULL;
- kfree(nm_i);
+ kvfree(nm_i);
}
-int __init create_node_manager_caches(void)
+int __init f2fs_create_node_manager_caches(void)
{
nat_entry_slab = f2fs_kmem_cache_create("nat_entry",
sizeof(struct nat_entry));
@@ -2885,8 +3144,15 @@ int __init create_node_manager_caches(void)
sizeof(struct nat_entry_set));
if (!nat_entry_set_slab)
goto destroy_free_nid;
+
+ fsync_node_entry_slab = f2fs_kmem_cache_create("fsync_node_entry",
+ sizeof(struct fsync_node_entry));
+ if (!fsync_node_entry_slab)
+ goto destroy_nat_entry_set;
return 0;
+destroy_nat_entry_set:
+ kmem_cache_destroy(nat_entry_set_slab);
destroy_free_nid:
kmem_cache_destroy(free_nid_slab);
destroy_nat_entry:
@@ -2895,8 +3161,9 @@ int __init create_node_manager_caches(void)
return -ENOMEM;
}
-void destroy_node_manager_caches(void)
+void f2fs_destroy_node_manager_caches(void)
{
+ kmem_cache_destroy(fsync_node_entry_slab);
kmem_cache_destroy(nat_entry_set_slab);
kmem_cache_destroy(free_nid_slab);
kmem_cache_destroy(nat_entry_slab);
diff --git a/fs/f2fs/node.h b/fs/f2fs/node.h
index b95e49e..e05af5d 100644
--- a/fs/f2fs/node.h
+++ b/fs/f2fs/node.h
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/node.h
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
/* start node id of a node block dedicated to the given node id */
#define START_NID(nid) (((nid) / NAT_ENTRY_PER_BLOCK) * NAT_ENTRY_PER_BLOCK)
@@ -135,6 +132,11 @@ static inline bool excess_cached_nats(struct f2fs_sb_info *sbi)
return NM_I(sbi)->nat_cnt >= DEF_NAT_CACHE_THRESHOLD;
}
+static inline bool excess_dirty_nodes(struct f2fs_sb_info *sbi)
+{
+ return get_pages(sbi, F2FS_DIRTY_NODES) >= sbi->blocks_per_seg * 8;
+}
+
enum mem_type {
FREE_NIDS, /* indicates the free nid list */
NAT_ENTRIES, /* indicates the cached nat entry */
@@ -359,7 +361,7 @@ static inline int set_nid(struct page *p, int off, nid_t nid, bool i)
{
struct f2fs_node *rn = F2FS_NODE(p);
- f2fs_wait_on_page_writeback(p, NODE, true);
+ f2fs_wait_on_page_writeback(p, NODE, true, true);
if (i)
rn->i.i_nid[off - NODE_DIR1_BLOCK] = cpu_to_le32(nid);
@@ -444,6 +446,10 @@ static inline void set_mark(struct page *page, int mark, int type)
else
flag &= ~(0x1 << type);
rn->footer.flag = cpu_to_le32(flag);
+
+#ifdef CONFIG_F2FS_CHECK_FS
+ f2fs_inode_chksum_set(F2FS_P_SB(page), page);
+#endif
}
#define set_dentry_mark(page, mark) set_mark(page, mark, DENT_BIT_SHIFT)
#define set_fsync_mark(page, mark) set_mark(page, mark, FSYNC_BIT_SHIFT)
diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c
index 4ddc226..5a99911d 100644
--- a/fs/f2fs/recovery.c
+++ b/fs/f2fs/recovery.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/recovery.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
@@ -47,7 +44,7 @@
static struct kmem_cache *fsync_entry_slab;
-bool space_for_roll_forward(struct f2fs_sb_info *sbi)
+bool f2fs_space_for_roll_forward(struct f2fs_sb_info *sbi)
{
s64 nalloc = percpu_counter_sum_positive(&sbi->alloc_valid_block_count);
@@ -99,8 +96,12 @@ static struct fsync_inode_entry *add_fsync_inode(struct f2fs_sb_info *sbi,
return ERR_PTR(err);
}
-static void del_fsync_inode(struct fsync_inode_entry *entry)
+static void del_fsync_inode(struct fsync_inode_entry *entry, int drop)
{
+ if (drop) {
+ /* inode should not be recovered, drop it */
+ f2fs_inode_synced(entry->inode);
+ }
iput(entry->inode);
list_del(&entry->list);
kmem_cache_free(fsync_entry_slab, entry);
@@ -162,7 +163,7 @@ static int recover_dentry(struct inode *inode, struct page *ipage,
goto out_put;
}
- err = acquire_orphan_inode(F2FS_I_SB(inode));
+ err = f2fs_acquire_orphan_inode(F2FS_I_SB(inode));
if (err) {
iput(einode);
goto out_put;
@@ -173,7 +174,7 @@ static int recover_dentry(struct inode *inode, struct page *ipage,
} else if (IS_ERR(page)) {
err = PTR_ERR(page);
} else {
- err = __f2fs_do_add_link(dir, &fname, inode,
+ err = f2fs_add_dentry(dir, &fname, inode,
inode->i_ino, inode->i_mode);
}
if (err == -ENOMEM)
@@ -194,6 +195,33 @@ static int recover_dentry(struct inode *inode, struct page *ipage,
return err;
}
+static int recover_quota_data(struct inode *inode, struct page *page)
+{
+ struct f2fs_inode *raw = F2FS_INODE(page);
+ struct iattr attr;
+ uid_t i_uid = le32_to_cpu(raw->i_uid);
+ gid_t i_gid = le32_to_cpu(raw->i_gid);
+ int err;
+
+ memset(&attr, 0, sizeof(attr));
+
+ attr.ia_uid = make_kuid(inode->i_sb->s_user_ns, i_uid);
+ attr.ia_gid = make_kgid(inode->i_sb->s_user_ns, i_gid);
+
+ if (!uid_eq(attr.ia_uid, inode->i_uid))
+ attr.ia_valid |= ATTR_UID;
+ if (!gid_eq(attr.ia_gid, inode->i_gid))
+ attr.ia_valid |= ATTR_GID;
+
+ if (!attr.ia_valid)
+ return 0;
+
+ err = dquot_transfer(inode, &attr);
+ if (err)
+ set_sbi_flag(F2FS_I_SB(inode), SBI_QUOTA_NEED_REPAIR);
+ return err;
+}
+
static void recover_inline_flags(struct inode *inode, struct f2fs_inode *ri)
{
if (ri->i_inline & F2FS_PIN_FILE)
@@ -204,16 +232,35 @@ static void recover_inline_flags(struct inode *inode, struct f2fs_inode *ri)
set_inode_flag(inode, FI_DATA_EXIST);
else
clear_inode_flag(inode, FI_DATA_EXIST);
- if (!(ri->i_inline & F2FS_INLINE_DOTS))
- clear_inode_flag(inode, FI_INLINE_DOTS);
}
-static void recover_inode(struct inode *inode, struct page *page)
+static int recover_inode(struct inode *inode, struct page *page)
{
struct f2fs_inode *raw = F2FS_INODE(page);
char *name;
+ int err;
inode->i_mode = le16_to_cpu(raw->i_mode);
+
+ err = recover_quota_data(inode, page);
+ if (err)
+ return err;
+
+ i_uid_write(inode, le32_to_cpu(raw->i_uid));
+ i_gid_write(inode, le32_to_cpu(raw->i_gid));
+
+ if (raw->i_inline & F2FS_EXTRA_ATTR) {
+ if (f2fs_sb_has_project_quota(F2FS_I_SB(inode)) &&
+ F2FS_FITS_IN_INODE(raw, le16_to_cpu(raw->i_extra_isize),
+ i_projid)) {
+ projid_t i_projid;
+
+ i_projid = (projid_t)le32_to_cpu(raw->i_projid);
+ F2FS_I(inode)->i_projid =
+ make_kprojid(&init_user_ns, i_projid);
+ }
+ }
+
f2fs_i_size_write(inode, le64_to_cpu(raw->i_size));
inode->i_atime.tv_sec = le64_to_cpu(raw->i_atime);
inode->i_ctime.tv_sec = le64_to_cpu(raw->i_ctime);
@@ -223,9 +270,15 @@ static void recover_inode(struct inode *inode, struct page *page)
inode->i_mtime.tv_nsec = le32_to_cpu(raw->i_mtime_nsec);
F2FS_I(inode)->i_advise = raw->i_advise;
+ F2FS_I(inode)->i_flags = le32_to_cpu(raw->i_flags);
+ f2fs_set_inode_flags(inode);
+ F2FS_I(inode)->i_gc_failures[GC_FAILURE_PIN] =
+ le16_to_cpu(raw->i_gc_failures);
recover_inline_flags(inode, raw);
+ f2fs_mark_inode_dirty_sync(inode, true);
+
if (file_enc_name(inode))
name = "<encrypted>";
else
@@ -234,6 +287,7 @@ static void recover_inode(struct inode *inode, struct page *page)
f2fs_msg(inode->i_sb, KERN_NOTICE,
"recover_inode: ino = %x, name = %s, inline = %x",
ino_of_node(page), name, raw->i_inline);
+ return 0;
}
static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head,
@@ -243,8 +297,8 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head,
struct page *page = NULL;
block_t blkaddr;
unsigned int loop_cnt = 0;
- unsigned int free_blocks = sbi->user_block_count -
- valid_user_blocks(sbi);
+ unsigned int free_blocks = MAIN_SEGS(sbi) * sbi->blocks_per_seg -
+ valid_user_blocks(sbi);
int err = 0;
/* get node pages in the current segment */
@@ -254,10 +308,14 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head,
while (1) {
struct fsync_inode_entry *entry;
- if (!is_valid_blkaddr(sbi, blkaddr, META_POR))
+ if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR))
return 0;
- page = get_tmp_page(sbi, blkaddr);
+ page = f2fs_get_tmp_page(sbi, blkaddr);
+ if (IS_ERR(page)) {
+ err = PTR_ERR(page);
+ break;
+ }
if (!is_recoverable_dnode(page))
break;
@@ -271,7 +329,7 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head,
if (!check_only &&
IS_INODE(page) && is_dent_dnode(page)) {
- err = recover_inode_page(sbi, page);
+ err = f2fs_recover_inode_page(sbi, page);
if (err)
break;
quota_inode = true;
@@ -312,18 +370,18 @@ static int find_fsync_dnodes(struct f2fs_sb_info *sbi, struct list_head *head,
blkaddr = next_blkaddr_of_node(page);
f2fs_put_page(page, 1);
- ra_meta_pages_cond(sbi, blkaddr);
+ f2fs_ra_meta_pages_cond(sbi, blkaddr);
}
f2fs_put_page(page, 1);
return err;
}
-static void destroy_fsync_dnodes(struct list_head *head)
+static void destroy_fsync_dnodes(struct list_head *head, int drop)
{
struct fsync_inode_entry *entry, *tmp;
list_for_each_entry_safe(entry, tmp, head, list)
- del_fsync_inode(entry);
+ del_fsync_inode(entry, drop);
}
static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
@@ -355,7 +413,9 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
}
}
- sum_page = get_sum_page(sbi, segno);
+ sum_page = f2fs_get_sum_page(sbi, segno);
+ if (IS_ERR(sum_page))
+ return PTR_ERR(sum_page);
sum_node = (struct f2fs_summary_block *)page_address(sum_page);
sum = sum_node->entries[blkoff];
f2fs_put_page(sum_page, 1);
@@ -375,7 +435,7 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
}
/* Get the node page */
- node_page = get_node_page(sbi, nid);
+ node_page = f2fs_get_node_page(sbi, nid);
if (IS_ERR(node_page))
return PTR_ERR(node_page);
@@ -400,7 +460,8 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
inode = dn->inode;
}
- bidx = start_bidx_of_node(offset, inode) + le16_to_cpu(sum.ofs_in_node);
+ bidx = f2fs_start_bidx_of_node(offset, inode) +
+ le16_to_cpu(sum.ofs_in_node);
/*
* if inode page is locked, unlock temporarily, but its reference
@@ -410,11 +471,11 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
unlock_page(dn->inode_page);
set_new_dnode(&tdn, inode, NULL, NULL, 0);
- if (get_dnode_of_data(&tdn, bidx, LOOKUP_NODE))
+ if (f2fs_get_dnode_of_data(&tdn, bidx, LOOKUP_NODE))
goto out;
if (tdn.data_blkaddr == blkaddr)
- truncate_data_blocks_range(&tdn, 1);
+ f2fs_truncate_data_blocks_range(&tdn, 1);
f2fs_put_dnode(&tdn);
out:
@@ -427,7 +488,7 @@ static int check_index_in_prev_nodes(struct f2fs_sb_info *sbi,
truncate_out:
if (datablock_addr(tdn.inode, tdn.node_page,
tdn.ofs_in_node) == blkaddr)
- truncate_data_blocks_range(&tdn, 1);
+ f2fs_truncate_data_blocks_range(&tdn, 1);
if (dn->inode->i_ino == nid && !dn->inode_page_locked)
unlock_page(dn->inode_page);
return 0;
@@ -443,25 +504,25 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
/* step 1: recover xattr */
if (IS_INODE(page)) {
- recover_inline_xattr(inode, page);
+ f2fs_recover_inline_xattr(inode, page);
} else if (f2fs_has_xattr_block(ofs_of_node(page))) {
- err = recover_xattr_data(inode, page);
+ err = f2fs_recover_xattr_data(inode, page);
if (!err)
recovered++;
goto out;
}
/* step 2: recover inline data */
- if (recover_inline_data(inode, page))
+ if (f2fs_recover_inline_data(inode, page))
goto out;
/* step 3: recover data indices */
- start = start_bidx_of_node(ofs_of_node(page), inode);
+ start = f2fs_start_bidx_of_node(ofs_of_node(page), inode);
end = start + ADDRS_PER_PAGE(page, inode);
set_new_dnode(&dn, inode, NULL, NULL, 0);
retry_dn:
- err = get_dnode_of_data(&dn, start, ALLOC_NODE);
+ err = f2fs_get_dnode_of_data(&dn, start, ALLOC_NODE);
if (err) {
if (err == -ENOMEM) {
congestion_wait(BLK_RW_ASYNC, HZ/50);
@@ -470,9 +531,12 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
goto out;
}
- f2fs_wait_on_page_writeback(dn.node_page, NODE, true);
+ f2fs_wait_on_page_writeback(dn.node_page, NODE, true, true);
- get_node_info(sbi, dn.nid, &ni);
+ err = f2fs_get_node_info(sbi, dn.nid, &ni);
+ if (err)
+ goto err;
+
f2fs_bug_on(sbi, ni.ino != ino_of_node(page));
f2fs_bug_on(sbi, ofs_of_node(dn.node_page) != ofs_of_node(page));
@@ -488,7 +552,7 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
/* dest is invalid, just invalidate src block */
if (dest == NULL_ADDR) {
- truncate_data_blocks_range(&dn, 1);
+ f2fs_truncate_data_blocks_range(&dn, 1);
continue;
}
@@ -502,20 +566,19 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
* and then reserve one new block in dnode page.
*/
if (dest == NEW_ADDR) {
- truncate_data_blocks_range(&dn, 1);
- reserve_new_block(&dn);
+ f2fs_truncate_data_blocks_range(&dn, 1);
+ f2fs_reserve_new_block(&dn);
continue;
}
/* dest is valid block, try to recover from src to dest */
- if (is_valid_blkaddr(sbi, dest, META_POR)) {
+ if (f2fs_is_valid_blkaddr(sbi, dest, META_POR)) {
if (src == NULL_ADDR) {
- err = reserve_new_block(&dn);
-#ifdef CONFIG_F2FS_FAULT_INJECTION
- while (err)
- err = reserve_new_block(&dn);
-#endif
+ err = f2fs_reserve_new_block(&dn);
+ while (err &&
+ IS_ENABLED(CONFIG_F2FS_FAULT_INJECTION))
+ err = f2fs_reserve_new_block(&dn);
/* We should not get -ENOSPC */
f2fs_bug_on(sbi, err);
if (err)
@@ -555,7 +618,7 @@ static int do_recover_data(struct f2fs_sb_info *sbi, struct inode *inode,
}
static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list,
- struct list_head *dir_list)
+ struct list_head *tmp_inode_list, struct list_head *dir_list)
{
struct curseg_info *curseg;
struct page *page = NULL;
@@ -569,12 +632,16 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list,
while (1) {
struct fsync_inode_entry *entry;
- if (!is_valid_blkaddr(sbi, blkaddr, META_POR))
+ if (!f2fs_is_valid_blkaddr(sbi, blkaddr, META_POR))
break;
- ra_meta_pages_cond(sbi, blkaddr);
+ f2fs_ra_meta_pages_cond(sbi, blkaddr);
- page = get_tmp_page(sbi, blkaddr);
+ page = f2fs_get_tmp_page(sbi, blkaddr);
+ if (IS_ERR(page)) {
+ err = PTR_ERR(page);
+ break;
+ }
if (!is_recoverable_dnode(page)) {
f2fs_put_page(page, 1);
@@ -589,8 +656,11 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list,
* In this case, we can lose the latest inode(x).
* So, call recover_inode for the inode update.
*/
- if (IS_INODE(page))
- recover_inode(entry->inode, page);
+ if (IS_INODE(page)) {
+ err = recover_inode(entry->inode, page);
+ if (err)
+ break;
+ }
if (entry->last_dentry == blkaddr) {
err = recover_dentry(entry->inode, page, dir_list);
if (err) {
@@ -605,20 +675,20 @@ static int recover_data(struct f2fs_sb_info *sbi, struct list_head *inode_list,
}
if (entry->blkaddr == blkaddr)
- del_fsync_inode(entry);
+ list_move_tail(&entry->list, tmp_inode_list);
next:
/* check next segment */
blkaddr = next_blkaddr_of_node(page);
f2fs_put_page(page, 1);
}
if (!err)
- allocate_new_segments(sbi);
+ f2fs_allocate_new_segments(sbi);
return err;
}
-int recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only)
+int f2fs_recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only)
{
- struct list_head inode_list;
+ struct list_head inode_list, tmp_inode_list;
struct list_head dir_list;
int err;
int ret = 0;
@@ -629,7 +699,8 @@ int recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only)
#endif
if (s_flags & MS_RDONLY) {
- f2fs_msg(sbi->sb, KERN_INFO, "orphan cleanup on readonly fs");
+ f2fs_msg(sbi->sb, KERN_INFO,
+ "recover fsync data on readonly fs");
sbi->sb->s_flags &= ~MS_RDONLY;
}
@@ -648,6 +719,7 @@ int recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only)
}
INIT_LIST_HEAD(&inode_list);
+ INIT_LIST_HEAD(&tmp_inode_list);
INIT_LIST_HEAD(&dir_list);
/* prevent checkpoint */
@@ -666,11 +738,16 @@ int recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only)
need_writecp = true;
/* step #2: recover data */
- err = recover_data(sbi, &inode_list, &dir_list);
+ err = recover_data(sbi, &inode_list, &tmp_inode_list, &dir_list);
if (!err)
f2fs_bug_on(sbi, !list_empty(&inode_list));
+ else {
+ /* restore s_flags to let iput() trash data */
+ sbi->sb->s_flags = s_flags;
+ }
skip:
- destroy_fsync_dnodes(&inode_list);
+ destroy_fsync_dnodes(&inode_list, err);
+ destroy_fsync_dnodes(&tmp_inode_list, err);
/* truncate meta pages to be used by the recovery */
truncate_inode_pages_range(META_MAPPING(sbi),
@@ -679,19 +756,23 @@ int recover_fsync_data(struct f2fs_sb_info *sbi, bool check_only)
if (err) {
truncate_inode_pages_final(NODE_MAPPING(sbi));
truncate_inode_pages_final(META_MAPPING(sbi));
+ } else {
+ clear_sbi_flag(sbi, SBI_POR_DOING);
}
-
- clear_sbi_flag(sbi, SBI_POR_DOING);
mutex_unlock(&sbi->cp_mutex);
/* let's drop all the directory inodes for clean checkpoint */
- destroy_fsync_dnodes(&dir_list);
+ destroy_fsync_dnodes(&dir_list, err);
- if (!err && need_writecp) {
- struct cp_control cpc = {
- .reason = CP_RECOVERY,
- };
- err = write_checkpoint(sbi, &cpc);
+ if (need_writecp) {
+ set_sbi_flag(sbi, SBI_IS_RECOVERED);
+
+ if (!err) {
+ struct cp_control cpc = {
+ .reason = CP_RECOVERY,
+ };
+ err = f2fs_write_checkpoint(sbi, &cpc);
+ }
}
kmem_cache_destroy(fsync_entry_slab);
diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
index 041b9d7..eaf7be5 100644
--- a/fs/f2fs/segment.c
+++ b/fs/f2fs/segment.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/segment.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
@@ -169,7 +166,7 @@ static unsigned long __find_rev_next_zero_bit(const unsigned long *addr,
return result - size + __reverse_ffz(tmp);
}
-bool need_SSR(struct f2fs_sb_info *sbi)
+bool f2fs_need_SSR(struct f2fs_sb_info *sbi)
{
int node_secs = get_blocktype_secs(sbi, F2FS_DIRTY_NODES);
int dent_secs = get_blocktype_secs(sbi, F2FS_DIRTY_DENTS);
@@ -177,14 +174,16 @@ bool need_SSR(struct f2fs_sb_info *sbi)
if (test_opt(sbi, LFS))
return false;
- if (sbi->gc_thread && sbi->gc_thread->gc_urgent)
+ if (sbi->gc_mode == GC_URGENT)
+ return true;
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
return true;
return free_sections(sbi) <= (node_secs + 2 * dent_secs + imeta_secs +
SM_I(sbi)->min_ssr_sections + reserved_sections(sbi));
}
-void register_inmem_page(struct inode *inode, struct page *page)
+void f2fs_register_inmem_page(struct inode *inode, struct page *page)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_inode_info *fi = F2FS_I(inode);
@@ -230,6 +229,8 @@ static int __revoke_inmem_pages(struct inode *inode,
lock_page(page);
+ f2fs_wait_on_page_writeback(page, DATA, true, true);
+
if (recover) {
struct dnode_of_data dn;
struct node_info ni;
@@ -237,7 +238,8 @@ static int __revoke_inmem_pages(struct inode *inode,
trace_f2fs_commit_inmem_page(page, INMEM_REVOKE);
retry:
set_new_dnode(&dn, inode, NULL, NULL, 0);
- err = get_dnode_of_data(&dn, page->index, LOOKUP_NODE);
+ err = f2fs_get_dnode_of_data(&dn, page->index,
+ LOOKUP_NODE);
if (err) {
if (err == -ENOMEM) {
congestion_wait(BLK_RW_ASYNC, HZ/50);
@@ -247,9 +249,15 @@ static int __revoke_inmem_pages(struct inode *inode,
err = -EAGAIN;
goto next;
}
- get_node_info(sbi, dn.nid, &ni);
+
+ err = f2fs_get_node_info(sbi, dn.nid, &ni);
+ if (err) {
+ f2fs_put_dnode(&dn);
+ return err;
+ }
+
if (cur->old_addr == NEW_ADDR) {
- invalidate_blocks(sbi, dn.data_blkaddr);
+ f2fs_invalidate_blocks(sbi, dn.data_blkaddr);
f2fs_update_data_blkaddr(&dn, NEW_ADDR);
} else
f2fs_replace_block(sbi, &dn, dn.data_blkaddr,
@@ -258,8 +266,10 @@ static int __revoke_inmem_pages(struct inode *inode,
}
next:
/* we don't need to invalidate this in the sccessful status */
- if (drop || recover)
+ if (drop || recover) {
ClearPageUptodate(page);
+ clear_cold_data(page);
+ }
set_page_private(page, 0);
ClearPagePrivate(page);
f2fs_put_page(page, 1);
@@ -271,7 +281,7 @@ static int __revoke_inmem_pages(struct inode *inode,
return err;
}
-void drop_inmem_pages_all(struct f2fs_sb_info *sbi)
+void f2fs_drop_inmem_pages_all(struct f2fs_sb_info *sbi, bool gc_failure)
{
struct list_head *head = &sbi->inode_list[ATOMIC_FILE];
struct inode *inode;
@@ -287,15 +297,23 @@ void drop_inmem_pages_all(struct f2fs_sb_info *sbi)
spin_unlock(&sbi->inode_lock[ATOMIC_FILE]);
if (inode) {
- drop_inmem_pages(inode);
+ if (gc_failure) {
+ if (fi->i_gc_failures[GC_FAILURE_ATOMIC])
+ goto drop;
+ goto skip;
+ }
+drop:
+ set_inode_flag(inode, FI_ATOMIC_REVOKE_REQUEST);
+ f2fs_drop_inmem_pages(inode);
iput(inode);
}
+skip:
congestion_wait(BLK_RW_ASYNC, HZ/50);
cond_resched();
goto next;
}
-void drop_inmem_pages(struct inode *inode)
+void f2fs_drop_inmem_pages(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_inode_info *fi = F2FS_I(inode);
@@ -309,11 +327,11 @@ void drop_inmem_pages(struct inode *inode)
mutex_unlock(&fi->inmem_lock);
clear_inode_flag(inode, FI_ATOMIC_FILE);
- clear_inode_flag(inode, FI_HOT_DATA);
+ fi->i_gc_failures[GC_FAILURE_ATOMIC] = 0;
stat_dec_atomic_write(inode);
}
-void drop_inmem_page(struct inode *inode, struct page *page)
+void f2fs_drop_inmem_page(struct inode *inode, struct page *page)
{
struct f2fs_inode_info *fi = F2FS_I(inode);
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
@@ -328,7 +346,7 @@ void drop_inmem_page(struct inode *inode, struct page *page)
break;
}
- f2fs_bug_on(sbi, !cur || cur->page != page);
+ f2fs_bug_on(sbi, list_empty(head) || cur->page != page);
list_del(&cur->list);
mutex_unlock(&fi->inmem_lock);
@@ -343,8 +361,7 @@ void drop_inmem_page(struct inode *inode, struct page *page)
trace_f2fs_commit_inmem_page(page, INMEM_INVALIDATE);
}
-static int __commit_inmem_pages(struct inode *inode,
- struct list_head *revoke_list)
+static int __f2fs_commit_inmem_pages(struct inode *inode)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct f2fs_inode_info *fi = F2FS_I(inode);
@@ -357,9 +374,12 @@ static int __commit_inmem_pages(struct inode *inode,
.op_flags = REQ_SYNC | REQ_PRIO,
.io_type = FS_DATA_IO,
};
- pgoff_t last_idx = ULONG_MAX;
+ struct list_head revoke_list;
+ bool submit_bio = false;
int err = 0;
+ INIT_LIST_HEAD(&revoke_list);
+
list_for_each_entry_safe(cur, tmp, &fi->inmem_pages, list) {
struct page *page = cur->page;
@@ -367,18 +387,19 @@ static int __commit_inmem_pages(struct inode *inode,
if (page->mapping == inode->i_mapping) {
trace_f2fs_commit_inmem_page(page, INMEM);
+ f2fs_wait_on_page_writeback(page, DATA, true, true);
+
set_page_dirty(page);
- f2fs_wait_on_page_writeback(page, DATA, true);
if (clear_page_dirty_for_io(page)) {
inode_dec_dirty_pages(inode);
- remove_dirty_inode(inode);
+ f2fs_remove_dirty_inode(inode);
}
retry:
fio.page = page;
fio.old_blkaddr = NULL_ADDR;
fio.encrypted_page = NULL;
fio.need_lock = LOCK_DONE;
- err = do_write_data_page(&fio);
+ err = f2fs_do_write_data_page(&fio);
if (err) {
if (err == -ENOMEM) {
congestion_wait(BLK_RW_ASYNC, HZ/50);
@@ -390,38 +411,16 @@ static int __commit_inmem_pages(struct inode *inode,
}
/* record old blkaddr for revoking */
cur->old_addr = fio.old_blkaddr;
- last_idx = page->index;
+ submit_bio = true;
}
unlock_page(page);
- list_move_tail(&cur->list, revoke_list);
+ list_move_tail(&cur->list, &revoke_list);
}
- if (last_idx != ULONG_MAX)
- f2fs_submit_merged_write_cond(sbi, inode, 0, last_idx, DATA);
+ if (submit_bio)
+ f2fs_submit_merged_write_cond(sbi, inode, NULL, 0, DATA);
- if (!err)
- __revoke_inmem_pages(inode, revoke_list, false, false);
-
- return err;
-}
-
-int commit_inmem_pages(struct inode *inode)
-{
- struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
- struct f2fs_inode_info *fi = F2FS_I(inode);
- struct list_head revoke_list;
- int err;
-
- INIT_LIST_HEAD(&revoke_list);
- f2fs_balance_fs(sbi, true);
- f2fs_lock_op(sbi);
-
- set_inode_flag(inode, FI_ATOMIC_COMMIT);
-
- mutex_lock(&fi->inmem_lock);
- err = __commit_inmem_pages(inode, &revoke_list);
if (err) {
- int ret;
/*
* try to revoke all committed pages, but still we could fail
* due to no memory or other reason, if that happened, EAGAIN
@@ -430,13 +429,33 @@ int commit_inmem_pages(struct inode *inode)
* recovery or rewrite & commit last transaction. For other
* error number, revoking was done by filesystem itself.
*/
- ret = __revoke_inmem_pages(inode, &revoke_list, false, true);
- if (ret)
- err = ret;
+ err = __revoke_inmem_pages(inode, &revoke_list, false, true);
/* drop all uncommitted pages */
__revoke_inmem_pages(inode, &fi->inmem_pages, true, false);
+ } else {
+ __revoke_inmem_pages(inode, &revoke_list, false, false);
}
+
+ return err;
+}
+
+int f2fs_commit_inmem_pages(struct inode *inode)
+{
+ struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+ struct f2fs_inode_info *fi = F2FS_I(inode);
+ int err;
+
+ f2fs_balance_fs(sbi, true);
+
+ down_write(&fi->i_gc_rwsem[WRITE]);
+
+ f2fs_lock_op(sbi);
+ set_inode_flag(inode, FI_ATOMIC_COMMIT);
+
+ mutex_lock(&fi->inmem_lock);
+ err = __f2fs_commit_inmem_pages(inode);
+
spin_lock(&sbi->inode_lock[ATOMIC_FILE]);
if (!list_empty(&fi->inmem_ilist))
list_del_init(&fi->inmem_ilist);
@@ -446,6 +465,8 @@ int commit_inmem_pages(struct inode *inode)
clear_inode_flag(inode, FI_ATOMIC_COMMIT);
f2fs_unlock_op(sbi);
+ up_write(&fi->i_gc_rwsem[WRITE]);
+
return err;
}
@@ -455,17 +476,18 @@ int commit_inmem_pages(struct inode *inode)
*/
void f2fs_balance_fs(struct f2fs_sb_info *sbi, bool need)
{
-#ifdef CONFIG_F2FS_FAULT_INJECTION
if (time_to_inject(sbi, FAULT_CHECKPOINT)) {
f2fs_show_injection_info(FAULT_CHECKPOINT);
f2fs_stop_checkpoint(sbi, false);
}
-#endif
/* balance_fs_bg is able to be pending */
if (need && excess_cached_nats(sbi))
f2fs_balance_fs_bg(sbi);
+ if (f2fs_is_checkpoint_ready(sbi))
+ return;
+
/*
* We should do GC or end up with checkpoint, if there are so many dirty
* dir/node pages without enough free segments.
@@ -478,33 +500,38 @@ void f2fs_balance_fs(struct f2fs_sb_info *sbi, bool need)
void f2fs_balance_fs_bg(struct f2fs_sb_info *sbi)
{
+ if (unlikely(is_sbi_flag_set(sbi, SBI_POR_DOING)))
+ return;
+
/* try to shrink extent cache when there is no enough memory */
- if (!available_free_memory(sbi, EXTENT_CACHE))
+ if (!f2fs_available_free_memory(sbi, EXTENT_CACHE))
f2fs_shrink_extent_tree(sbi, EXTENT_CACHE_SHRINK_NUMBER);
/* check the # of cached NAT entries */
- if (!available_free_memory(sbi, NAT_ENTRIES))
- try_to_free_nats(sbi, NAT_ENTRY_PER_BLOCK);
+ if (!f2fs_available_free_memory(sbi, NAT_ENTRIES))
+ f2fs_try_to_free_nats(sbi, NAT_ENTRY_PER_BLOCK);
- if (!available_free_memory(sbi, FREE_NIDS))
- try_to_free_nids(sbi, MAX_FREE_NIDS);
+ if (!f2fs_available_free_memory(sbi, FREE_NIDS))
+ f2fs_try_to_free_nids(sbi, MAX_FREE_NIDS);
else
- build_free_nids(sbi, false, false);
+ f2fs_build_free_nids(sbi, false, false);
- if (!is_idle(sbi) && !excess_dirty_nats(sbi))
+ if (!is_idle(sbi, REQ_TIME) &&
+ (!excess_dirty_nats(sbi) && !excess_dirty_nodes(sbi)))
return;
/* checkpoint is the only way to shrink partial cached entries */
- if (!available_free_memory(sbi, NAT_ENTRIES) ||
- !available_free_memory(sbi, INO_ENTRIES) ||
+ if (!f2fs_available_free_memory(sbi, NAT_ENTRIES) ||
+ !f2fs_available_free_memory(sbi, INO_ENTRIES) ||
excess_prefree_segs(sbi) ||
excess_dirty_nats(sbi) ||
+ excess_dirty_nodes(sbi) ||
f2fs_time_over(sbi, CP_TIME)) {
if (test_opt(sbi, DATA_FLUSH)) {
struct blk_plug plug;
blk_start_plug(&plug);
- sync_dirty_inodes(sbi, FILE_INODE);
+ f2fs_sync_dirty_inodes(sbi, FILE_INODE);
blk_finish_plug(&plug);
}
f2fs_sync_fs(sbi->sb, true);
@@ -537,7 +564,7 @@ static int submit_flush_wait(struct f2fs_sb_info *sbi, nid_t ino)
return __submit_flush_wait(sbi, sbi->sb->s_bdev);
for (i = 0; i < sbi->s_ndevs; i++) {
- if (!is_dirty_device(sbi, ino, i, FLUSH_INO))
+ if (!f2fs_is_dirty_device(sbi, ino, i, FLUSH_INO))
continue;
ret = __submit_flush_wait(sbi, FDEV(i).bdev);
if (ret)
@@ -594,14 +621,16 @@ int f2fs_issue_flush(struct f2fs_sb_info *sbi, nid_t ino)
return 0;
if (!test_opt(sbi, FLUSH_MERGE)) {
+ atomic_inc(&fcc->queued_flush);
ret = submit_flush_wait(sbi, ino);
+ atomic_dec(&fcc->queued_flush);
atomic_inc(&fcc->issued_flush);
return ret;
}
- if (atomic_inc_return(&fcc->issing_flush) == 1 || sbi->s_ndevs > 1) {
+ if (atomic_inc_return(&fcc->queued_flush) == 1 || sbi->s_ndevs > 1) {
ret = submit_flush_wait(sbi, ino);
- atomic_dec(&fcc->issing_flush);
+ atomic_dec(&fcc->queued_flush);
atomic_inc(&fcc->issued_flush);
return ret;
@@ -620,14 +649,14 @@ int f2fs_issue_flush(struct f2fs_sb_info *sbi, nid_t ino)
if (fcc->f2fs_issue_flush) {
wait_for_completion(&cmd.wait);
- atomic_dec(&fcc->issing_flush);
+ atomic_dec(&fcc->queued_flush);
} else {
struct llist_node *list;
list = llist_del_all(&fcc->issue_list);
if (!list) {
wait_for_completion(&cmd.wait);
- atomic_dec(&fcc->issing_flush);
+ atomic_dec(&fcc->queued_flush);
} else {
struct flush_cmd *tmp, *next;
@@ -636,7 +665,7 @@ int f2fs_issue_flush(struct f2fs_sb_info *sbi, nid_t ino)
llist_for_each_entry_safe(tmp, next, list, llnode) {
if (tmp == &cmd) {
cmd.ret = ret;
- atomic_dec(&fcc->issing_flush);
+ atomic_dec(&fcc->queued_flush);
continue;
}
tmp->ret = ret;
@@ -648,7 +677,7 @@ int f2fs_issue_flush(struct f2fs_sb_info *sbi, nid_t ino)
return cmd.ret;
}
-int create_flush_cmd_control(struct f2fs_sb_info *sbi)
+int f2fs_create_flush_cmd_control(struct f2fs_sb_info *sbi)
{
dev_t dev = sbi->sb->s_bdev->bd_dev;
struct flush_cmd_control *fcc;
@@ -665,7 +694,7 @@ int create_flush_cmd_control(struct f2fs_sb_info *sbi)
if (!fcc)
return -ENOMEM;
atomic_set(&fcc->issued_flush, 0);
- atomic_set(&fcc->issing_flush, 0);
+ atomic_set(&fcc->queued_flush, 0);
init_waitqueue_head(&fcc->flush_wait_queue);
init_llist_head(&fcc->issue_list);
SM_I(sbi)->fcc_info = fcc;
@@ -677,7 +706,7 @@ int create_flush_cmd_control(struct f2fs_sb_info *sbi)
"f2fs_flush-%u:%u", MAJOR(dev), MINOR(dev));
if (IS_ERR(fcc->f2fs_issue_flush)) {
err = PTR_ERR(fcc->f2fs_issue_flush);
- kfree(fcc);
+ kvfree(fcc);
SM_I(sbi)->fcc_info = NULL;
return err;
}
@@ -685,7 +714,7 @@ int create_flush_cmd_control(struct f2fs_sb_info *sbi)
return err;
}
-void destroy_flush_cmd_control(struct f2fs_sb_info *sbi, bool free)
+void f2fs_destroy_flush_cmd_control(struct f2fs_sb_info *sbi, bool free)
{
struct flush_cmd_control *fcc = SM_I(sbi)->fcc_info;
@@ -696,7 +725,7 @@ void destroy_flush_cmd_control(struct f2fs_sb_info *sbi, bool free)
kthread_stop(flush_thread);
}
if (free) {
- kfree(fcc);
+ kvfree(fcc);
SM_I(sbi)->fcc_info = NULL;
}
}
@@ -777,7 +806,7 @@ static void __remove_dirty_segment(struct f2fs_sb_info *sbi, unsigned int segno,
static void locate_dirty_segment(struct f2fs_sb_info *sbi, unsigned int segno)
{
struct dirty_seglist_info *dirty_i = DIRTY_I(sbi);
- unsigned short valid_blocks;
+ unsigned short valid_blocks, ckpt_valid_blocks;
if (segno == NULL_SEGNO || IS_CURSEG(sbi, segno))
return;
@@ -785,8 +814,10 @@ static void locate_dirty_segment(struct f2fs_sb_info *sbi, unsigned int segno)
mutex_lock(&dirty_i->seglist_lock);
valid_blocks = get_valid_blocks(sbi, segno, false);
+ ckpt_valid_blocks = get_ckpt_valid_blocks(sbi, segno);
- if (valid_blocks == 0) {
+ if (valid_blocks == 0 && (!is_sbi_flag_set(sbi, SBI_CP_DISABLED) ||
+ ckpt_valid_blocks == sbi->blocks_per_seg)) {
__locate_dirty_segment(sbi, segno, PRE);
__remove_dirty_segment(sbi, segno, DIRTY);
} else if (valid_blocks < sbi->blocks_per_seg) {
@@ -799,6 +830,66 @@ static void locate_dirty_segment(struct f2fs_sb_info *sbi, unsigned int segno)
mutex_unlock(&dirty_i->seglist_lock);
}
+/* This moves currently empty dirty blocks to prefree. Must hold seglist_lock */
+void f2fs_dirty_to_prefree(struct f2fs_sb_info *sbi)
+{
+ struct dirty_seglist_info *dirty_i = DIRTY_I(sbi);
+ unsigned int segno;
+
+ mutex_lock(&dirty_i->seglist_lock);
+ for_each_set_bit(segno, dirty_i->dirty_segmap[DIRTY], MAIN_SEGS(sbi)) {
+ if (get_valid_blocks(sbi, segno, false))
+ continue;
+ if (IS_CURSEG(sbi, segno))
+ continue;
+ __locate_dirty_segment(sbi, segno, PRE);
+ __remove_dirty_segment(sbi, segno, DIRTY);
+ }
+ mutex_unlock(&dirty_i->seglist_lock);
+}
+
+int f2fs_disable_cp_again(struct f2fs_sb_info *sbi)
+{
+ struct dirty_seglist_info *dirty_i = DIRTY_I(sbi);
+ block_t ovp = overprovision_segments(sbi) << sbi->log_blocks_per_seg;
+ block_t holes[2] = {0, 0}; /* DATA and NODE */
+ struct seg_entry *se;
+ unsigned int segno;
+
+ mutex_lock(&dirty_i->seglist_lock);
+ for_each_set_bit(segno, dirty_i->dirty_segmap[DIRTY], MAIN_SEGS(sbi)) {
+ se = get_seg_entry(sbi, segno);
+ if (IS_NODESEG(se->type))
+ holes[NODE] += sbi->blocks_per_seg - se->valid_blocks;
+ else
+ holes[DATA] += sbi->blocks_per_seg - se->valid_blocks;
+ }
+ mutex_unlock(&dirty_i->seglist_lock);
+
+ if (holes[DATA] > ovp || holes[NODE] > ovp)
+ return -EAGAIN;
+ return 0;
+}
+
+/* This is only used by SBI_CP_DISABLED */
+static unsigned int get_free_segment(struct f2fs_sb_info *sbi)
+{
+ struct dirty_seglist_info *dirty_i = DIRTY_I(sbi);
+ unsigned int segno = 0;
+
+ mutex_lock(&dirty_i->seglist_lock);
+ for_each_set_bit(segno, dirty_i->dirty_segmap[DIRTY], MAIN_SEGS(sbi)) {
+ if (get_valid_blocks(sbi, segno, false))
+ continue;
+ if (get_ckpt_valid_blocks(sbi, segno))
+ continue;
+ mutex_unlock(&dirty_i->seglist_lock);
+ return segno;
+ }
+ mutex_unlock(&dirty_i->seglist_lock);
+ return NULL_SEGNO;
+}
+
static struct discard_cmd *__create_discard_cmd(struct f2fs_sb_info *sbi,
struct block_device *bdev, block_t lstart,
block_t start, block_t len)
@@ -819,9 +910,12 @@ static struct discard_cmd *__create_discard_cmd(struct f2fs_sb_info *sbi,
dc->len = len;
dc->ref = 0;
dc->state = D_PREP;
+ dc->queued = 0;
dc->error = 0;
init_completion(&dc->wait);
list_add_tail(&dc->list, pend_list);
+ spin_lock_init(&dc->lock);
+ dc->bio_ref = 0;
atomic_inc(&dcc->discard_cmd_cnt);
dcc->undiscard_blks += len;
@@ -848,7 +942,7 @@ static void __detach_discard_cmd(struct discard_cmd_control *dcc,
struct discard_cmd *dc)
{
if (dc->state == D_DONE)
- atomic_dec(&dcc->issing_discard);
+ atomic_sub(dc->queued, &dcc->queued_discard);
list_del(&dc->list);
rb_erase(&dc->rb_node, &dcc->root);
@@ -863,28 +957,43 @@ static void __remove_discard_cmd(struct f2fs_sb_info *sbi,
struct discard_cmd *dc)
{
struct discard_cmd_control *dcc = SM_I(sbi)->dcc_info;
+ unsigned long flags;
trace_f2fs_remove_discard(dc->bdev, dc->start, dc->len);
+ spin_lock_irqsave(&dc->lock, flags);
+ if (dc->bio_ref) {
+ spin_unlock_irqrestore(&dc->lock, flags);
+ return;
+ }
+ spin_unlock_irqrestore(&dc->lock, flags);
+
f2fs_bug_on(sbi, dc->ref);
if (dc->error == -EOPNOTSUPP)
dc->error = 0;
if (dc->error)
- f2fs_msg(sbi->sb, KERN_INFO,
- "Issue discard(%u, %u, %u) failed, ret: %d",
- dc->lstart, dc->start, dc->len, dc->error);
+ printk_ratelimited(
+ "%sF2FS-fs: Issue discard(%u, %u, %u) failed, ret: %d",
+ KERN_INFO, dc->lstart, dc->start, dc->len, dc->error);
__detach_discard_cmd(dcc, dc);
}
static void f2fs_submit_discard_endio(struct bio *bio)
{
struct discard_cmd *dc = (struct discard_cmd *)bio->bi_private;
+ unsigned long flags;
dc->error = bio->bi_error;
- dc->state = D_DONE;
- complete_all(&dc->wait);
+
+ spin_lock_irqsave(&dc->lock, flags);
+ dc->bio_ref--;
+ if (!dc->bio_ref && dc->state == D_SUBMIT) {
+ dc->state = D_DONE;
+ complete_all(&dc->wait);
+ }
+ spin_unlock_irqrestore(&dc->lock, flags);
bio_put(bio);
}
@@ -922,6 +1031,7 @@ static void __init_discard_policy(struct f2fs_sb_info *sbi,
/* common policy */
dpolicy->type = discard_type;
dpolicy->sync = true;
+ dpolicy->ordered = false;
dpolicy->granularity = granularity;
dpolicy->max_requests = DEF_MAX_DISCARD_REQUEST;
@@ -933,6 +1043,7 @@ static void __init_discard_policy(struct f2fs_sb_info *sbi,
dpolicy->max_interval = DEF_MAX_DISCARD_ISSUE_TIME;
dpolicy->io_aware = true;
dpolicy->sync = false;
+ dpolicy->ordered = true;
if (utilization(sbi) > DEF_DISCARD_URGENT_UTIL) {
dpolicy->granularity = 1;
dpolicy->max_interval = DEF_MIN_DISCARD_ISSUE_TIME;
@@ -945,49 +1056,120 @@ static void __init_discard_policy(struct f2fs_sb_info *sbi,
} else if (discard_type == DPOLICY_FSTRIM) {
dpolicy->io_aware = false;
} else if (discard_type == DPOLICY_UMOUNT) {
+ dpolicy->max_requests = UINT_MAX;
dpolicy->io_aware = false;
}
}
-
+static void __update_discard_tree_range(struct f2fs_sb_info *sbi,
+ struct block_device *bdev, block_t lstart,
+ block_t start, block_t len);
/* this function is copied from blkdev_issue_discard from block/blk-lib.c */
-static void __submit_discard_cmd(struct f2fs_sb_info *sbi,
+static int __submit_discard_cmd(struct f2fs_sb_info *sbi,
struct discard_policy *dpolicy,
- struct discard_cmd *dc)
+ struct discard_cmd *dc,
+ unsigned int *issued)
{
+ struct block_device *bdev = dc->bdev;
+ struct request_queue *q = bdev_get_queue(bdev);
+ unsigned int max_discard_blocks =
+ SECTOR_TO_BLOCK(q->limits.max_discard_sectors);
struct discard_cmd_control *dcc = SM_I(sbi)->dcc_info;
struct list_head *wait_list = (dpolicy->type == DPOLICY_FSTRIM) ?
&(dcc->fstrim_list) : &(dcc->wait_list);
- struct bio *bio = NULL;
int flag = dpolicy->sync ? REQ_SYNC : 0;
+ block_t lstart, start, len, total_len;
+ int err = 0;
if (dc->state != D_PREP)
- return;
+ return 0;
- trace_f2fs_issue_discard(dc->bdev, dc->start, dc->len);
+ if (is_sbi_flag_set(sbi, SBI_NEED_FSCK))
+ return 0;
- dc->error = __blkdev_issue_discard(dc->bdev,
- SECTOR_FROM_BLOCK(dc->start),
- SECTOR_FROM_BLOCK(dc->len),
- GFP_NOFS, 0, &bio);
- if (!dc->error) {
- /* should keep before submission to avoid D_DONE right away */
- dc->state = D_SUBMIT;
- atomic_inc(&dcc->issued_discard);
- atomic_inc(&dcc->issing_discard);
- if (bio) {
- bio->bi_private = dc;
- bio->bi_end_io = f2fs_submit_discard_endio;
- bio->bi_opf |= flag;
- submit_bio(bio);
- list_move_tail(&dc->list, wait_list);
- __check_sit_bitmap(sbi, dc->start, dc->start + dc->len);
+ trace_f2fs_issue_discard(bdev, dc->start, dc->len);
- f2fs_update_iostat(sbi, FS_DISCARD, 1);
+ lstart = dc->lstart;
+ start = dc->start;
+ len = dc->len;
+ total_len = len;
+
+ dc->len = 0;
+
+ while (total_len && *issued < dpolicy->max_requests && !err) {
+ struct bio *bio = NULL;
+ unsigned long flags;
+ bool last = true;
+
+ if (len > max_discard_blocks) {
+ len = max_discard_blocks;
+ last = false;
}
- } else {
- __remove_discard_cmd(sbi, dc);
+
+ (*issued)++;
+ if (*issued == dpolicy->max_requests)
+ last = true;
+
+ dc->len += len;
+
+ if (time_to_inject(sbi, FAULT_DISCARD)) {
+ f2fs_show_injection_info(FAULT_DISCARD);
+ err = -EIO;
+ goto submit;
+ }
+ err = __blkdev_issue_discard(bdev,
+ SECTOR_FROM_BLOCK(start),
+ SECTOR_FROM_BLOCK(len),
+ GFP_NOFS, 0, &bio);
+submit:
+ if (err) {
+ spin_lock_irqsave(&dc->lock, flags);
+ if (dc->state == D_PARTIAL)
+ dc->state = D_SUBMIT;
+ spin_unlock_irqrestore(&dc->lock, flags);
+
+ break;
+ }
+
+ f2fs_bug_on(sbi, !bio);
+
+ /*
+ * should keep before submission to avoid D_DONE
+ * right away
+ */
+ spin_lock_irqsave(&dc->lock, flags);
+ if (last)
+ dc->state = D_SUBMIT;
+ else
+ dc->state = D_PARTIAL;
+ dc->bio_ref++;
+ spin_unlock_irqrestore(&dc->lock, flags);
+
+ atomic_inc(&dcc->queued_discard);
+ dc->queued++;
+ list_move_tail(&dc->list, wait_list);
+
+ /* sanity check on discard range */
+ __check_sit_bitmap(sbi, lstart, lstart + len);
+
+ bio->bi_private = dc;
+ bio->bi_end_io = f2fs_submit_discard_endio;
+ bio->bi_opf |= flag;
+ submit_bio(bio);
+
+ atomic_inc(&dcc->issued_discard);
+
+ f2fs_update_iostat(sbi, FS_DISCARD, 1);
+
+ lstart += len;
+ start += len;
+ total_len -= len;
+ len = total_len;
}
+
+ if (!err && len)
+ __update_discard_tree_range(sbi, bdev, lstart, start, len);
+ return err;
}
static struct discard_cmd *__insert_discard_tree(struct f2fs_sb_info *sbi,
@@ -1007,7 +1189,7 @@ static struct discard_cmd *__insert_discard_tree(struct f2fs_sb_info *sbi,
goto do_insert;
}
- p = __lookup_rb_tree_for_insert(sbi, &dcc->root, &parent, lstart);
+ p = f2fs_lookup_rb_tree_for_insert(sbi, &dcc->root, &parent, lstart);
do_insert:
dc = __attach_discard_cmd(sbi, bdev, lstart, start, len, parent, p);
if (!dc)
@@ -1068,11 +1250,12 @@ static void __update_discard_tree_range(struct f2fs_sb_info *sbi,
struct discard_cmd *dc;
struct discard_info di = {0};
struct rb_node **insert_p = NULL, *insert_parent = NULL;
+ struct request_queue *q = bdev_get_queue(bdev);
+ unsigned int max_discard_blocks =
+ SECTOR_TO_BLOCK(q->limits.max_discard_sectors);
block_t end = lstart + len;
- mutex_lock(&dcc->cmd_lock);
-
- dc = (struct discard_cmd *)__lookup_rb_tree_ret(&dcc->root,
+ dc = (struct discard_cmd *)f2fs_lookup_rb_tree_ret(&dcc->root,
NULL, lstart,
(struct rb_entry **)&prev_dc,
(struct rb_entry **)&next_dc,
@@ -1111,7 +1294,8 @@ static void __update_discard_tree_range(struct f2fs_sb_info *sbi,
if (prev_dc && prev_dc->state == D_PREP &&
prev_dc->bdev == bdev &&
- __is_discard_back_mergeable(&di, &prev_dc->di)) {
+ __is_discard_back_mergeable(&di, &prev_dc->di,
+ max_discard_blocks)) {
prev_dc->di.len += di.len;
dcc->undiscard_blks += di.len;
__relocate_discard_cmd(dcc, prev_dc);
@@ -1122,7 +1306,8 @@ static void __update_discard_tree_range(struct f2fs_sb_info *sbi,
if (next_dc && next_dc->state == D_PREP &&
next_dc->bdev == bdev &&
- __is_discard_front_mergeable(&di, &next_dc->di)) {
+ __is_discard_front_mergeable(&di, &next_dc->di,
+ max_discard_blocks)) {
next_dc->di.lstart = di.lstart;
next_dc->di.len += di.len;
next_dc->di.start = di.start;
@@ -1145,8 +1330,6 @@ static void __update_discard_tree_range(struct f2fs_sb_info *sbi,
node = rb_next(&prev_dc->rb_node);
next_dc = rb_entry_safe(node, struct discard_cmd, rb_node);
}
-
- mutex_unlock(&dcc->cmd_lock);
}
static int __queue_discard_cmd(struct f2fs_sb_info *sbi,
@@ -1161,10 +1344,72 @@ static int __queue_discard_cmd(struct f2fs_sb_info *sbi,
blkstart -= FDEV(devi).start_blk;
}
+ mutex_lock(&SM_I(sbi)->dcc_info->cmd_lock);
__update_discard_tree_range(sbi, bdev, lblkstart, blkstart, blklen);
+ mutex_unlock(&SM_I(sbi)->dcc_info->cmd_lock);
return 0;
}
+static unsigned int __issue_discard_cmd_orderly(struct f2fs_sb_info *sbi,
+ struct discard_policy *dpolicy)
+{
+ struct discard_cmd_control *dcc = SM_I(sbi)->dcc_info;
+ struct discard_cmd *prev_dc = NULL, *next_dc = NULL;
+ struct rb_node **insert_p = NULL, *insert_parent = NULL;
+ struct discard_cmd *dc;
+ struct blk_plug plug;
+ unsigned int pos = dcc->next_pos;
+ unsigned int issued = 0;
+ bool io_interrupted = false;
+
+ mutex_lock(&dcc->cmd_lock);
+ dc = (struct discard_cmd *)f2fs_lookup_rb_tree_ret(&dcc->root,
+ NULL, pos,
+ (struct rb_entry **)&prev_dc,
+ (struct rb_entry **)&next_dc,
+ &insert_p, &insert_parent, true);
+ if (!dc)
+ dc = next_dc;
+
+ blk_start_plug(&plug);
+
+ while (dc) {
+ struct rb_node *node;
+ int err = 0;
+
+ if (dc->state != D_PREP)
+ goto next;
+
+ if (dpolicy->io_aware && !is_idle(sbi, DISCARD_TIME)) {
+ io_interrupted = true;
+ break;
+ }
+
+ dcc->next_pos = dc->lstart + dc->len;
+ err = __submit_discard_cmd(sbi, dpolicy, dc, &issued);
+
+ if (issued >= dpolicy->max_requests)
+ break;
+next:
+ node = rb_next(&dc->rb_node);
+ if (err)
+ __remove_discard_cmd(sbi, dc);
+ dc = rb_entry_safe(node, struct discard_cmd, rb_node);
+ }
+
+ blk_finish_plug(&plug);
+
+ if (!dc)
+ dcc->next_pos = 0;
+
+ mutex_unlock(&dcc->cmd_lock);
+
+ if (!issued && io_interrupted)
+ issued = -1;
+
+ return issued;
+}
+
static int __issue_discard_cmd(struct f2fs_sb_info *sbi,
struct discard_policy *dpolicy)
{
@@ -1172,39 +1417,44 @@ static int __issue_discard_cmd(struct f2fs_sb_info *sbi,
struct list_head *pend_list;
struct discard_cmd *dc, *tmp;
struct blk_plug plug;
- int i, iter = 0, issued = 0;
+ int i, issued = 0;
bool io_interrupted = false;
for (i = MAX_PLIST_NUM - 1; i >= 0; i--) {
if (i + 1 < dpolicy->granularity)
break;
+
+ if (i < DEFAULT_DISCARD_GRANULARITY && dpolicy->ordered)
+ return __issue_discard_cmd_orderly(sbi, dpolicy);
+
pend_list = &dcc->pend_list[i];
mutex_lock(&dcc->cmd_lock);
if (list_empty(pend_list))
goto next;
- f2fs_bug_on(sbi, !__check_rb_tree_consistence(sbi, &dcc->root));
+ if (unlikely(dcc->rbtree_check))
+ f2fs_bug_on(sbi, !f2fs_check_rb_tree_consistence(sbi,
+ &dcc->root));
blk_start_plug(&plug);
list_for_each_entry_safe(dc, tmp, pend_list, list) {
f2fs_bug_on(sbi, dc->state != D_PREP);
if (dpolicy->io_aware && i < dpolicy->io_aware_gran &&
- !is_idle(sbi)) {
+ !is_idle(sbi, DISCARD_TIME)) {
io_interrupted = true;
- goto skip;
+ break;
}
- __submit_discard_cmd(sbi, dpolicy, dc);
- issued++;
-skip:
- if (++iter >= dpolicy->max_requests)
+ __submit_discard_cmd(sbi, dpolicy, dc, &issued);
+
+ if (issued >= dpolicy->max_requests)
break;
}
blk_finish_plug(&plug);
next:
mutex_unlock(&dcc->cmd_lock);
- if (iter >= dpolicy->max_requests)
+ if (issued >= dpolicy->max_requests || io_interrupted)
break;
}
@@ -1236,7 +1486,7 @@ static bool __drop_discard_cmd(struct f2fs_sb_info *sbi)
return dropped;
}
-void drop_discard_cmd(struct f2fs_sb_info *sbi)
+void f2fs_drop_discard_cmd(struct f2fs_sb_info *sbi)
{
__drop_discard_cmd(sbi);
}
@@ -1302,21 +1552,22 @@ static unsigned int __wait_discard_cmd_range(struct f2fs_sb_info *sbi,
return trimmed;
}
-static void __wait_all_discard_cmd(struct f2fs_sb_info *sbi,
+static unsigned int __wait_all_discard_cmd(struct f2fs_sb_info *sbi,
struct discard_policy *dpolicy)
{
struct discard_policy dp;
+ unsigned int discard_blks;
- if (dpolicy) {
- __wait_discard_cmd_range(sbi, dpolicy, 0, UINT_MAX);
- return;
- }
+ if (dpolicy)
+ return __wait_discard_cmd_range(sbi, dpolicy, 0, UINT_MAX);
/* wait all */
__init_discard_policy(sbi, &dp, DPOLICY_FSTRIM, 1);
- __wait_discard_cmd_range(sbi, &dp, 0, UINT_MAX);
+ discard_blks = __wait_discard_cmd_range(sbi, &dp, 0, UINT_MAX);
__init_discard_policy(sbi, &dp, DPOLICY_UMOUNT, 1);
- __wait_discard_cmd_range(sbi, &dp, 0, UINT_MAX);
+ discard_blks += __wait_discard_cmd_range(sbi, &dp, 0, UINT_MAX);
+
+ return discard_blks;
}
/* This should be covered by global mutex, &sit_i->sentry_lock */
@@ -1327,7 +1578,8 @@ static void f2fs_wait_discard_bio(struct f2fs_sb_info *sbi, block_t blkaddr)
bool need_wait = false;
mutex_lock(&dcc->cmd_lock);
- dc = (struct discard_cmd *)__lookup_rb_tree(&dcc->root, NULL, blkaddr);
+ dc = (struct discard_cmd *)f2fs_lookup_rb_tree(&dcc->root,
+ NULL, blkaddr);
if (dc) {
if (dc->state == D_PREP) {
__punch_discard_cmd(sbi, dc, blkaddr);
@@ -1342,7 +1594,7 @@ static void f2fs_wait_discard_bio(struct f2fs_sb_info *sbi, block_t blkaddr)
__wait_one_discard_bio(sbi, dc);
}
-void stop_discard_thread(struct f2fs_sb_info *sbi)
+void f2fs_stop_discard_thread(struct f2fs_sb_info *sbi)
{
struct discard_cmd_control *dcc = SM_I(sbi)->dcc_info;
@@ -1368,6 +1620,8 @@ bool f2fs_wait_discard_bios(struct f2fs_sb_info *sbi)
/* just to make sure there is no pending discard commands */
__wait_all_discard_cmd(sbi, NULL);
+
+ f2fs_bug_on(sbi, atomic_read(&dcc->discard_cmd_cnt));
return dropped;
}
@@ -1379,8 +1633,6 @@ static int issue_discard_thread(void *data)
struct discard_policy dpolicy;
unsigned int wait_ms = DEF_MIN_DISCARD_ISSUE_TIME;
int issued;
- unsigned long interval = sbi->interval_time[REQ_TIME] * HZ;
- long delta;
set_freezable();
@@ -1392,17 +1644,26 @@ static int issue_discard_thread(void *data)
kthread_should_stop() || freezing(current) ||
dcc->discard_wake,
msecs_to_jiffies(wait_ms));
+
+ if (dcc->discard_wake)
+ dcc->discard_wake = 0;
+
+ /* clean up pending candidates before going to sleep */
+ if (atomic_read(&dcc->queued_discard))
+ __wait_all_discard_cmd(sbi, NULL);
+
if (try_to_freeze())
continue;
if (f2fs_readonly(sbi->sb))
continue;
if (kthread_should_stop())
return 0;
+ if (is_sbi_flag_set(sbi, SBI_NEED_FSCK)) {
+ wait_ms = dpolicy.max_interval;
+ continue;
+ }
- if (dcc->discard_wake)
- dcc->discard_wake = 0;
-
- if (sbi->gc_thread && sbi->gc_thread->gc_urgent)
+ if (sbi->gc_mode == GC_URGENT)
__init_discard_policy(sbi, &dpolicy, DPOLICY_FORCE, 1);
sb_start_intwrite(sbi->sb);
@@ -1412,10 +1673,8 @@ static int issue_discard_thread(void *data)
__wait_all_discard_cmd(sbi, &dpolicy);
wait_ms = dpolicy.min_interval;
} else if (issued == -1){
- delta = (sbi->last_time[REQ_TIME] + interval) - jiffies;
- if (delta > 0)
- wait_ms = jiffies_to_msecs(delta);
- else
+ wait_ms = f2fs_time_to_wait(sbi, DISCARD_TIME);
+ if (!wait_ms)
wait_ms = dpolicy.mid_interval;
} else {
wait_ms = dpolicy.max_interval;
@@ -1478,7 +1737,7 @@ static int __issue_discard_async(struct f2fs_sb_info *sbi,
struct block_device *bdev, block_t blkstart, block_t blklen)
{
#ifdef CONFIG_BLK_DEV_ZONED
- if (f2fs_sb_has_blkzoned(sbi->sb) &&
+ if (f2fs_sb_has_blkzoned(sbi) &&
bdev_zoned_model(bdev) != BLK_ZONED_NONE)
return __f2fs_issue_discard_zone(sbi, bdev, blkstart, blklen);
#endif
@@ -1541,11 +1800,11 @@ static bool add_discard_addrs(struct f2fs_sb_info *sbi, struct cp_control *cpc,
struct list_head *head = &SM_I(sbi)->dcc_info->entry_list;
int i;
- if (se->valid_blocks == max_blocks || !f2fs_discard_en(sbi))
+ if (se->valid_blocks == max_blocks || !f2fs_hw_support_discard(sbi))
return false;
if (!force) {
- if (!test_opt(sbi, DISCARD) || !se->valid_blocks ||
+ if (!f2fs_realtime_discard_enable(sbi) || !se->valid_blocks ||
SM_I(sbi)->dcc_info->nr_discards >=
SM_I(sbi)->dcc_info->max_discards)
return false;
@@ -1585,20 +1844,24 @@ static bool add_discard_addrs(struct f2fs_sb_info *sbi, struct cp_control *cpc,
return false;
}
-void release_discard_addrs(struct f2fs_sb_info *sbi)
+static void release_discard_addr(struct discard_entry *entry)
+{
+ list_del(&entry->list);
+ kmem_cache_free(discard_entry_slab, entry);
+}
+
+void f2fs_release_discard_addrs(struct f2fs_sb_info *sbi)
{
struct list_head *head = &(SM_I(sbi)->dcc_info->entry_list);
struct discard_entry *entry, *this;
/* drop caches */
- list_for_each_entry_safe(entry, this, head, list) {
- list_del(&entry->list);
- kmem_cache_free(discard_entry_slab, entry);
- }
+ list_for_each_entry_safe(entry, this, head, list)
+ release_discard_addr(entry);
}
/*
- * Should call clear_prefree_segments after checkpoint is done.
+ * Should call f2fs_clear_prefree_segments after checkpoint is done.
*/
static void set_prefree_as_free_segments(struct f2fs_sb_info *sbi)
{
@@ -1611,7 +1874,8 @@ static void set_prefree_as_free_segments(struct f2fs_sb_info *sbi)
mutex_unlock(&dirty_i->seglist_lock);
}
-void clear_prefree_segments(struct f2fs_sb_info *sbi, struct cp_control *cpc)
+void f2fs_clear_prefree_segments(struct f2fs_sb_info *sbi,
+ struct cp_control *cpc)
{
struct discard_cmd_control *dcc = SM_I(sbi)->dcc_info;
struct list_head *head = &dcc->entry_list;
@@ -1621,30 +1885,39 @@ void clear_prefree_segments(struct f2fs_sb_info *sbi, struct cp_control *cpc)
unsigned int start = 0, end = -1;
unsigned int secno, start_segno;
bool force = (cpc->reason & CP_DISCARD);
+ bool need_align = test_opt(sbi, LFS) && __is_large_section(sbi);
mutex_lock(&dirty_i->seglist_lock);
while (1) {
int i;
+
+ if (need_align && end != -1)
+ end--;
start = find_next_bit(prefree_map, MAIN_SEGS(sbi), end + 1);
if (start >= MAIN_SEGS(sbi))
break;
end = find_next_zero_bit(prefree_map, MAIN_SEGS(sbi),
start + 1);
- for (i = start; i < end; i++)
- clear_bit(i, prefree_map);
+ if (need_align) {
+ start = rounddown(start, sbi->segs_per_sec);
+ end = roundup(end, sbi->segs_per_sec);
+ }
- dirty_i->nr_dirty[PRE] -= end - start;
+ for (i = start; i < end; i++) {
+ if (test_and_clear_bit(i, prefree_map))
+ dirty_i->nr_dirty[PRE]--;
+ }
- if (!test_opt(sbi, DISCARD))
+ if (!f2fs_realtime_discard_enable(sbi))
continue;
if (force && start >= cpc->trim_start &&
(end - 1) <= cpc->trim_end)
continue;
- if (!test_opt(sbi, LFS) || sbi->segs_per_sec == 1) {
+ if (!test_opt(sbi, LFS) || !__is_large_section(sbi)) {
f2fs_issue_discard(sbi, START_BLOCK(sbi, start),
(end - start) << sbi->log_blocks_per_seg);
continue;
@@ -1676,7 +1949,7 @@ void clear_prefree_segments(struct f2fs_sb_info *sbi, struct cp_control *cpc)
sbi->blocks_per_seg, cur_pos);
len = next_pos - cur_pos;
- if (f2fs_sb_has_blkzoned(sbi->sb) ||
+ if (f2fs_sb_has_blkzoned(sbi) ||
(force && len < cpc->trim_minlen))
goto skip;
@@ -1694,9 +1967,8 @@ void clear_prefree_segments(struct f2fs_sb_info *sbi, struct cp_control *cpc)
if (cur_pos < sbi->blocks_per_seg)
goto find_next;
- list_del(&entry->list);
+ release_discard_addr(entry);
dcc->nr_discards -= total_len;
- kmem_cache_free(discard_entry_slab, entry);
}
wake_up_discard_thread(sbi, false);
@@ -1725,12 +1997,14 @@ static int create_discard_cmd_control(struct f2fs_sb_info *sbi)
INIT_LIST_HEAD(&dcc->fstrim_list);
mutex_init(&dcc->cmd_lock);
atomic_set(&dcc->issued_discard, 0);
- atomic_set(&dcc->issing_discard, 0);
+ atomic_set(&dcc->queued_discard, 0);
atomic_set(&dcc->discard_cmd_cnt, 0);
dcc->nr_discards = 0;
dcc->max_discards = MAIN_SEGS(sbi) << sbi->log_blocks_per_seg;
dcc->undiscard_blks = 0;
+ dcc->next_pos = 0;
dcc->root = RB_ROOT;
+ dcc->rbtree_check = false;
init_waitqueue_head(&dcc->discard_wait_queue);
SM_I(sbi)->dcc_info = dcc;
@@ -1739,7 +2013,7 @@ static int create_discard_cmd_control(struct f2fs_sb_info *sbi)
"f2fs_discard-%u:%u", MAJOR(dev), MINOR(dev));
if (IS_ERR(dcc->f2fs_issue_discard)) {
err = PTR_ERR(dcc->f2fs_issue_discard);
- kfree(dcc);
+ kvfree(dcc);
SM_I(sbi)->dcc_info = NULL;
return err;
}
@@ -1754,9 +2028,9 @@ static void destroy_discard_cmd_control(struct f2fs_sb_info *sbi)
if (!dcc)
return;
- stop_discard_thread(sbi);
+ f2fs_stop_discard_thread(sbi);
- kfree(dcc);
+ kvfree(dcc);
SM_I(sbi)->dcc_info = NULL;
}
@@ -1801,8 +2075,9 @@ static void update_sit_entry(struct f2fs_sb_info *sbi, block_t blkaddr, int del)
(new_vblocks > sbi->blocks_per_seg)));
se->valid_blocks = new_vblocks;
- se->mtime = get_mtime(sbi);
- SIT_I(sbi)->max_mtime = se->mtime;
+ se->mtime = get_mtime(sbi, false);
+ if (se->mtime > SIT_I(sbi)->max_mtime)
+ SIT_I(sbi)->max_mtime = se->mtime;
/* Update valid block bitmap */
if (del > 0) {
@@ -1825,12 +2100,12 @@ static void update_sit_entry(struct f2fs_sb_info *sbi, block_t blkaddr, int del)
del = 0;
}
- if (f2fs_discard_en(sbi) &&
- !f2fs_test_and_set_bit(offset, se->discard_map))
+ if (!f2fs_test_and_set_bit(offset, se->discard_map))
sbi->discard_blks--;
/* don't overwrite by SSR to keep node chain */
- if (IS_NODESEG(se->type)) {
+ if (IS_NODESEG(se->type) &&
+ !is_sbi_flag_set(sbi, SBI_CP_DISABLED)) {
if (!f2fs_test_and_set_bit(offset, se->ckpt_valid_map))
se->ckpt_valid_blocks++;
}
@@ -1852,10 +2127,18 @@ static void update_sit_entry(struct f2fs_sb_info *sbi, block_t blkaddr, int del)
f2fs_bug_on(sbi, 1);
se->valid_blocks++;
del = 0;
+ } else if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) {
+ /*
+ * If checkpoints are off, we must not reuse data that
+ * was used in the previous checkpoint. If it was used
+ * before, we must track that to know how much space we
+ * really have.
+ */
+ if (f2fs_test_bit(offset, se->ckpt_valid_map))
+ sbi->unusable_block_count++;
}
- if (f2fs_discard_en(sbi) &&
- f2fs_test_and_clear_bit(offset, se->discard_map))
+ if (f2fs_test_and_clear_bit(offset, se->discard_map))
sbi->discard_blks++;
}
if (!f2fs_test_bit(offset, se->ckpt_valid_map))
@@ -1866,11 +2149,11 @@ static void update_sit_entry(struct f2fs_sb_info *sbi, block_t blkaddr, int del)
/* update total number of valid blocks to be written in ckpt area */
SIT_I(sbi)->written_valid_blocks += del;
- if (sbi->segs_per_sec > 1)
+ if (__is_large_section(sbi))
get_sec_entry(sbi, segno)->valid_blocks += del;
}
-void invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr)
+void f2fs_invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr)
{
unsigned int segno = GET_SEGNO(sbi, addr);
struct sit_info *sit_i = SIT_I(sbi);
@@ -1879,6 +2162,8 @@ void invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr)
if (addr == NEW_ADDR)
return;
+ invalidate_mapping_pages(META_MAPPING(sbi), addr, addr);
+
/* add it into sit main buffer */
down_write(&sit_i->sentry_lock);
@@ -1890,14 +2175,14 @@ void invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr)
up_write(&sit_i->sentry_lock);
}
-bool is_checkpointed_data(struct f2fs_sb_info *sbi, block_t blkaddr)
+bool f2fs_is_checkpointed_data(struct f2fs_sb_info *sbi, block_t blkaddr)
{
struct sit_info *sit_i = SIT_I(sbi);
unsigned int segno, offset;
struct seg_entry *se;
bool is_cp = false;
- if (blkaddr == NEW_ADDR || blkaddr == NULL_ADDR)
+ if (!is_valid_data_blkaddr(sbi, blkaddr))
return true;
down_read(&sit_i->sentry_lock);
@@ -1929,7 +2214,7 @@ static void __add_sum_entry(struct f2fs_sb_info *sbi, int type,
/*
* Calculate the number of current summary pages for writing
*/
-int npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra)
+int f2fs_npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra)
{
int valid_sum_count = 0;
int i, sum_in_page;
@@ -1959,14 +2244,15 @@ int npages_for_summary_flush(struct f2fs_sb_info *sbi, bool for_ra)
/*
* Caller should put this summary page
*/
-struct page *get_sum_page(struct f2fs_sb_info *sbi, unsigned int segno)
+struct page *f2fs_get_sum_page(struct f2fs_sb_info *sbi, unsigned int segno)
{
- return get_meta_page(sbi, GET_SUM_BLOCK(sbi, segno));
+ return f2fs_get_meta_page_nofail(sbi, GET_SUM_BLOCK(sbi, segno));
}
-void update_meta_page(struct f2fs_sb_info *sbi, void *src, block_t blk_addr)
+void f2fs_update_meta_page(struct f2fs_sb_info *sbi,
+ void *src, block_t blk_addr)
{
- struct page *page = grab_meta_page(sbi, blk_addr);
+ struct page *page = f2fs_grab_meta_page(sbi, blk_addr);
memcpy(page_address(page), src, PAGE_SIZE);
set_page_dirty(page);
@@ -1976,18 +2262,19 @@ void update_meta_page(struct f2fs_sb_info *sbi, void *src, block_t blk_addr)
static void write_sum_page(struct f2fs_sb_info *sbi,
struct f2fs_summary_block *sum_blk, block_t blk_addr)
{
- update_meta_page(sbi, (void *)sum_blk, blk_addr);
+ f2fs_update_meta_page(sbi, (void *)sum_blk, blk_addr);
}
static void write_current_sum_page(struct f2fs_sb_info *sbi,
int type, block_t blk_addr)
{
struct curseg_info *curseg = CURSEG_I(sbi, type);
- struct page *page = grab_meta_page(sbi, blk_addr);
+ struct page *page = f2fs_grab_meta_page(sbi, blk_addr);
struct f2fs_summary_block *src = curseg->sum_blk;
struct f2fs_summary_block *dst;
dst = (struct f2fs_summary_block *)page_address(page);
+ memset(dst, 0, PAGE_SIZE);
mutex_lock(&curseg->curseg_mutex);
@@ -2128,9 +2415,12 @@ static void reset_curseg(struct f2fs_sb_info *sbi, int type, int modified)
static unsigned int __get_next_segno(struct f2fs_sb_info *sbi, int type)
{
/* if segs_per_sec is large than 1, we need to keep original policy. */
- if (sbi->segs_per_sec != 1)
+ if (__is_large_section(sbi))
return CURSEG_I(sbi, type)->segno;
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
+ return 0;
+
if (test_opt(sbi, NOHEAP) &&
(type == CURSEG_HOT_DATA || IS_NODESEG(type)))
return 0;
@@ -2227,7 +2517,8 @@ static void change_curseg(struct f2fs_sb_info *sbi, int type)
curseg->alloc_type = SSR;
__next_free_blkoff(sbi, curseg, 0);
- sum_page = get_sum_page(sbi, new_segno);
+ sum_page = f2fs_get_sum_page(sbi, new_segno);
+ f2fs_bug_on(sbi, IS_ERR(sum_page));
sum_node = (struct f2fs_summary_block *)page_address(sum_page);
memcpy(curseg->sum_blk, sum_node, SUM_ENTRY_SIZE);
f2fs_put_page(sum_page, 1);
@@ -2241,7 +2532,7 @@ static int get_ssr_segment(struct f2fs_sb_info *sbi, int type)
int i, cnt;
bool reversed = false;
- /* need_SSR() already forces to do this */
+ /* f2fs_need_SSR() already forces to do this */
if (v_ops->get_victim(sbi, &segno, BG_GC, type, SSR)) {
curseg->next_segno = segno;
return 1;
@@ -2274,6 +2565,15 @@ static int get_ssr_segment(struct f2fs_sb_info *sbi, int type)
return 1;
}
}
+
+ /* find valid_blocks=0 in dirty list */
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED))) {
+ segno = get_free_segment(sbi);
+ if (segno != NULL_SEGNO) {
+ curseg->next_segno = segno;
+ return 1;
+ }
+ }
return 0;
}
@@ -2291,9 +2591,10 @@ static void allocate_segment_by_default(struct f2fs_sb_info *sbi,
else if (!is_set_ckpt_flags(sbi, CP_CRC_RECOVERY_FLAG) &&
type == CURSEG_WARM_NODE)
new_curseg(sbi, type, false);
- else if (curseg->alloc_type == LFS && is_next_segment_free(sbi, type))
+ else if (curseg->alloc_type == LFS && is_next_segment_free(sbi, type) &&
+ likely(!is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
new_curseg(sbi, type, false);
- else if (need_SSR(sbi) && get_ssr_segment(sbi, type))
+ else if (f2fs_need_SSR(sbi) && get_ssr_segment(sbi, type))
change_curseg(sbi, type);
else
new_curseg(sbi, type, false);
@@ -2301,7 +2602,7 @@ static void allocate_segment_by_default(struct f2fs_sb_info *sbi,
stat_inc_seg_type(sbi, curseg);
}
-void allocate_new_segments(struct f2fs_sb_info *sbi)
+void f2fs_allocate_new_segments(struct f2fs_sb_info *sbi)
{
struct curseg_info *curseg;
unsigned int old_segno;
@@ -2323,7 +2624,8 @@ static const struct segment_allocation default_salloc_ops = {
.allocate_segment = allocate_segment_by_default,
};
-bool exist_trim_candidates(struct f2fs_sb_info *sbi, struct cp_control *cpc)
+bool f2fs_exist_trim_candidates(struct f2fs_sb_info *sbi,
+ struct cp_control *cpc)
{
__u64 trim_start = cpc->trim_start;
bool has_candidate = false;
@@ -2341,7 +2643,7 @@ bool exist_trim_candidates(struct f2fs_sb_info *sbi, struct cp_control *cpc)
return has_candidate;
}
-static void __issue_discard_cmd_range(struct f2fs_sb_info *sbi,
+static unsigned int __issue_discard_cmd_range(struct f2fs_sb_info *sbi,
struct discard_policy *dpolicy,
unsigned int start, unsigned int end)
{
@@ -2351,14 +2653,17 @@ static void __issue_discard_cmd_range(struct f2fs_sb_info *sbi,
struct discard_cmd *dc;
struct blk_plug plug;
int issued;
+ unsigned int trimmed = 0;
next:
issued = 0;
mutex_lock(&dcc->cmd_lock);
- f2fs_bug_on(sbi, !__check_rb_tree_consistence(sbi, &dcc->root));
+ if (unlikely(dcc->rbtree_check))
+ f2fs_bug_on(sbi, !f2fs_check_rb_tree_consistence(sbi,
+ &dcc->root));
- dc = (struct discard_cmd *)__lookup_rb_tree_ret(&dcc->root,
+ dc = (struct discard_cmd *)f2fs_lookup_rb_tree_ret(&dcc->root,
NULL, start,
(struct rb_entry **)&prev_dc,
(struct rb_entry **)&next_dc,
@@ -2370,6 +2675,7 @@ static void __issue_discard_cmd_range(struct f2fs_sb_info *sbi,
while (dc && dc->lstart <= end) {
struct rb_node *node;
+ int err = 0;
if (dc->len < dpolicy->granularity)
goto skip;
@@ -2379,19 +2685,24 @@ static void __issue_discard_cmd_range(struct f2fs_sb_info *sbi,
goto skip;
}
- __submit_discard_cmd(sbi, dpolicy, dc);
+ err = __submit_discard_cmd(sbi, dpolicy, dc, &issued);
- if (++issued >= dpolicy->max_requests) {
+ if (issued >= dpolicy->max_requests) {
start = dc->lstart + dc->len;
+ if (err)
+ __remove_discard_cmd(sbi, dc);
+
blk_finish_plug(&plug);
mutex_unlock(&dcc->cmd_lock);
- __wait_all_discard_cmd(sbi, NULL);
+ trimmed += __wait_all_discard_cmd(sbi, NULL);
congestion_wait(BLK_RW_ASYNC, HZ/50);
goto next;
}
skip:
node = rb_next(&dc->rb_node);
+ if (err)
+ __remove_discard_cmd(sbi, dc);
dc = rb_entry_safe(node, struct discard_cmd, rb_node);
if (fatal_signal_pending(current))
@@ -2400,6 +2711,8 @@ static void __issue_discard_cmd_range(struct f2fs_sb_info *sbi,
blk_finish_plug(&plug);
mutex_unlock(&dcc->cmd_lock);
+
+ return trimmed;
}
int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range)
@@ -2412,23 +2725,28 @@ int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range)
struct discard_policy dpolicy;
unsigned long long trimmed = 0;
int err = 0;
+ bool need_align = test_opt(sbi, LFS) && __is_large_section(sbi);
if (start >= MAX_BLKADDR(sbi) || range->len < sbi->blocksize)
return -EINVAL;
- if (end <= MAIN_BLKADDR(sbi))
+ if (end < MAIN_BLKADDR(sbi))
goto out;
if (is_sbi_flag_set(sbi, SBI_NEED_FSCK)) {
f2fs_msg(sbi->sb, KERN_WARNING,
"Found FS corruption, run fsck to fix.");
- goto out;
+ return -EIO;
}
/* start/end segment number in main_area */
start_segno = (start <= MAIN_BLKADDR(sbi)) ? 0 : GET_SEGNO(sbi, start);
end_segno = (end >= MAX_BLKADDR(sbi)) ? MAIN_SEGS(sbi) - 1 :
GET_SEGNO(sbi, end);
+ if (need_align) {
+ start_segno = rounddown(start_segno, sbi->segs_per_sec);
+ end_segno = roundup(end_segno + 1, sbi->segs_per_sec) - 1;
+ }
cpc.reason = CP_DISCARD;
cpc.trim_minlen = max_t(__u64, 1, F2FS_BYTES_TO_BLK(range->minlen));
@@ -2439,20 +2757,32 @@ int f2fs_trim_fs(struct f2fs_sb_info *sbi, struct fstrim_range *range)
goto out;
mutex_lock(&sbi->gc_mutex);
- err = write_checkpoint(sbi, &cpc);
+ err = f2fs_write_checkpoint(sbi, &cpc);
mutex_unlock(&sbi->gc_mutex);
if (err)
goto out;
+ /*
+ * We filed discard candidates, but actually we don't need to wait for
+ * all of them, since they'll be issued in idle time along with runtime
+ * discard option. User configuration looks like using runtime discard
+ * or periodic fstrim instead of it.
+ */
+ if (f2fs_realtime_discard_enable(sbi))
+ goto out;
+
start_block = START_BLOCK(sbi, start_segno);
end_block = START_BLOCK(sbi, end_segno + 1);
__init_discard_policy(sbi, &dpolicy, DPOLICY_FSTRIM, cpc.trim_minlen);
- __issue_discard_cmd_range(sbi, &dpolicy, start_block, end_block);
- trimmed = __wait_discard_cmd_range(sbi, &dpolicy,
+ trimmed = __issue_discard_cmd_range(sbi, &dpolicy,
start_block, end_block);
- range->len = F2FS_BLK_TO_BYTES(trimmed);
+
+ trimmed += __wait_discard_cmd_range(sbi, &dpolicy,
+ start_block, end_block);
out:
+ if (!err)
+ range->len = F2FS_BLK_TO_BYTES(trimmed);
return err;
}
@@ -2464,7 +2794,7 @@ static bool __has_curseg_space(struct f2fs_sb_info *sbi, int type)
return false;
}
-int rw_hint_to_seg_type(enum rw_hint hint)
+int f2fs_rw_hint_to_seg_type(enum rw_hint hint)
{
switch (hint) {
case WRITE_LIFE_SHORT:
@@ -2537,7 +2867,7 @@ int rw_hint_to_seg_type(enum rw_hint hint)
* WRITE_LIFE_LONG " WRITE_LIFE_LONG
*/
-enum rw_hint io_type_to_rw_hint(struct f2fs_sb_info *sbi,
+enum rw_hint f2fs_io_type_to_rw_hint(struct f2fs_sb_info *sbi,
enum page_type type, enum temp_type temp)
{
if (F2FS_OPTION(sbi).whint_mode == WHINT_MODE_USER) {
@@ -2604,9 +2934,11 @@ static int __get_segment_type_6(struct f2fs_io_info *fio)
if (is_cold_data(fio->page) || file_is_cold(inode))
return CURSEG_COLD_DATA;
if (file_is_hot(inode) ||
- is_inode_flag_set(inode, FI_HOT_DATA))
+ is_inode_flag_set(inode, FI_HOT_DATA) ||
+ f2fs_is_atomic_file(inode) ||
+ f2fs_is_volatile_file(inode))
return CURSEG_HOT_DATA;
- /* rw_hint_to_seg_type(inode->i_write_hint); */
+ /* f2fs_rw_hint_to_seg_type(inode->i_write_hint); */
return CURSEG_WARM_DATA;
} else {
if (IS_DNODE(fio->page))
@@ -2643,7 +2975,7 @@ static int __get_segment_type(struct f2fs_io_info *fio)
return type;
}
-void allocate_data_block(struct f2fs_sb_info *sbi, struct page *page,
+void f2fs_allocate_data_block(struct f2fs_sb_info *sbi, struct page *page,
block_t old_blkaddr, block_t *new_blkaddr,
struct f2fs_summary *sum, int type,
struct f2fs_io_info *fio, bool add_list)
@@ -2703,6 +3035,7 @@ void allocate_data_block(struct f2fs_sb_info *sbi, struct page *page,
INIT_LIST_HEAD(&fio->list);
fio->in_list = true;
+ fio->retry = false;
io = sbi->write_io[fio->type] + fio->temp;
spin_lock(&io->io_lock);
list_add_tail(&fio->list, &io->io_list);
@@ -2725,7 +3058,7 @@ static void update_device_state(struct f2fs_io_info *fio)
devidx = f2fs_target_device_index(sbi, fio->new_blkaddr);
/* update device state for fsync */
- set_dirty_device(sbi, fio->ino, devidx, FLUSH_INO);
+ f2fs_set_dirty_device(sbi, fio->ino, devidx, FLUSH_INO);
/* update device state for checkpoint */
if (!f2fs_test_bit(devidx, (char *)&sbi->dirty_device)) {
@@ -2738,23 +3071,31 @@ static void update_device_state(struct f2fs_io_info *fio)
static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio)
{
int type = __get_segment_type(fio);
- int err;
+ bool keep_order = (test_opt(fio->sbi, LFS) && type == CURSEG_COLD_DATA);
+ if (keep_order)
+ down_read(&fio->sbi->io_order_lock);
reallocate:
- allocate_data_block(fio->sbi, fio->page, fio->old_blkaddr,
+ f2fs_allocate_data_block(fio->sbi, fio->page, fio->old_blkaddr,
&fio->new_blkaddr, sum, type, fio, true);
+ if (GET_SEGNO(fio->sbi, fio->old_blkaddr) != NULL_SEGNO)
+ invalidate_mapping_pages(META_MAPPING(fio->sbi),
+ fio->old_blkaddr, fio->old_blkaddr);
/* writeout dirty page into bdev */
- err = f2fs_submit_page_write(fio);
- if (err == -EAGAIN) {
+ f2fs_submit_page_write(fio);
+ if (fio->retry) {
fio->old_blkaddr = fio->new_blkaddr;
goto reallocate;
- } else if (!err) {
- update_device_state(fio);
}
+
+ update_device_state(fio);
+
+ if (keep_order)
+ up_read(&fio->sbi->io_order_lock);
}
-void write_meta_page(struct f2fs_sb_info *sbi, struct page *page,
+void f2fs_do_write_meta_page(struct f2fs_sb_info *sbi, struct page *page,
enum iostat_type io_type)
{
struct f2fs_io_info fio = {
@@ -2774,12 +3115,14 @@ void write_meta_page(struct f2fs_sb_info *sbi, struct page *page,
fio.op_flags &= ~REQ_META;
set_page_writeback(page);
+ ClearPageError(page);
f2fs_submit_page_write(&fio);
+ stat_inc_meta_count(sbi, page->index);
f2fs_update_iostat(sbi, io_type, F2FS_BLKSIZE);
}
-void write_node_page(unsigned int nid, struct f2fs_io_info *fio)
+void f2fs_do_write_node_page(unsigned int nid, struct f2fs_io_info *fio)
{
struct f2fs_summary sum;
@@ -2789,22 +3132,21 @@ void write_node_page(unsigned int nid, struct f2fs_io_info *fio)
f2fs_update_iostat(fio->sbi, fio->io_type, F2FS_BLKSIZE);
}
-void write_data_page(struct dnode_of_data *dn, struct f2fs_io_info *fio)
+void f2fs_outplace_write_data(struct dnode_of_data *dn,
+ struct f2fs_io_info *fio)
{
struct f2fs_sb_info *sbi = fio->sbi;
struct f2fs_summary sum;
- struct node_info ni;
f2fs_bug_on(sbi, dn->data_blkaddr == NULL_ADDR);
- get_node_info(sbi, dn->nid, &ni);
- set_summary(&sum, dn->nid, dn->ofs_in_node, ni.version);
+ set_summary(&sum, dn->nid, dn->ofs_in_node, fio->version);
do_write_page(&sum, fio);
f2fs_update_data_blkaddr(dn, fio->new_blkaddr);
f2fs_update_iostat(sbi, fio->io_type, F2FS_BLKSIZE);
}
-int rewrite_data_page(struct f2fs_io_info *fio)
+int f2fs_inplace_write_data(struct f2fs_io_info *fio)
{
int err;
struct f2fs_sb_info *sbi = fio->sbi;
@@ -2839,7 +3181,7 @@ static inline int __f2fs_get_curseg(struct f2fs_sb_info *sbi,
return i;
}
-void __f2fs_replace_block(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
+void f2fs_do_replace_block(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
block_t old_blkaddr, block_t new_blkaddr,
bool recover_curseg, bool recover_newaddr)
{
@@ -2894,8 +3236,11 @@ void __f2fs_replace_block(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
if (!recover_curseg || recover_newaddr)
update_sit_entry(sbi, new_blkaddr, 1);
- if (GET_SEGNO(sbi, old_blkaddr) != NULL_SEGNO)
+ if (GET_SEGNO(sbi, old_blkaddr) != NULL_SEGNO) {
+ invalidate_mapping_pages(META_MAPPING(sbi),
+ old_blkaddr, old_blkaddr);
update_sit_entry(sbi, old_blkaddr, -1);
+ }
locate_dirty_segment(sbi, GET_SEGNO(sbi, old_blkaddr));
locate_dirty_segment(sbi, GET_SEGNO(sbi, new_blkaddr));
@@ -2924,42 +3269,56 @@ void f2fs_replace_block(struct f2fs_sb_info *sbi, struct dnode_of_data *dn,
set_summary(&sum, dn->nid, dn->ofs_in_node, version);
- __f2fs_replace_block(sbi, &sum, old_addr, new_addr,
+ f2fs_do_replace_block(sbi, &sum, old_addr, new_addr,
recover_curseg, recover_newaddr);
f2fs_update_data_blkaddr(dn, new_addr);
}
void f2fs_wait_on_page_writeback(struct page *page,
- enum page_type type, bool ordered)
+ enum page_type type, bool ordered, bool locked)
{
if (PageWriteback(page)) {
struct f2fs_sb_info *sbi = F2FS_P_SB(page);
- f2fs_submit_merged_write_cond(sbi, page->mapping->host,
- 0, page->index, type);
- if (ordered)
+ f2fs_submit_merged_write_cond(sbi, NULL, page, 0, type);
+ if (ordered) {
wait_on_page_writeback(page);
- else
+ f2fs_bug_on(sbi, locked && PageWriteback(page));
+ } else {
wait_for_stable_page(page);
+ }
}
}
-void f2fs_wait_on_block_writeback(struct f2fs_sb_info *sbi, block_t blkaddr)
+void f2fs_wait_on_block_writeback(struct inode *inode, block_t blkaddr)
{
+ struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct page *cpage;
- if (blkaddr == NEW_ADDR || blkaddr == NULL_ADDR)
+ if (!f2fs_post_read_required(inode))
+ return;
+
+ if (!is_valid_data_blkaddr(sbi, blkaddr))
return;
cpage = find_lock_page(META_MAPPING(sbi), blkaddr);
if (cpage) {
- f2fs_wait_on_page_writeback(cpage, DATA, true);
+ f2fs_wait_on_page_writeback(cpage, DATA, true, true);
f2fs_put_page(cpage, 1);
}
}
-static void read_compacted_summaries(struct f2fs_sb_info *sbi)
+void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr,
+ block_t len)
+{
+ block_t i;
+
+ for (i = 0; i < len; i++)
+ f2fs_wait_on_block_writeback(inode, blkaddr + i);
+}
+
+static int read_compacted_summaries(struct f2fs_sb_info *sbi)
{
struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi);
struct curseg_info *seg_i;
@@ -2970,7 +3329,9 @@ static void read_compacted_summaries(struct f2fs_sb_info *sbi)
start = start_sum_block(sbi);
- page = get_meta_page(sbi, start++);
+ page = f2fs_get_meta_page(sbi, start++);
+ if (IS_ERR(page))
+ return PTR_ERR(page);
kaddr = (unsigned char *)page_address(page);
/* Step 1: restore nat cache */
@@ -3010,12 +3371,15 @@ static void read_compacted_summaries(struct f2fs_sb_info *sbi)
f2fs_put_page(page, 1);
page = NULL;
- page = get_meta_page(sbi, start++);
+ page = f2fs_get_meta_page(sbi, start++);
+ if (IS_ERR(page))
+ return PTR_ERR(page);
kaddr = (unsigned char *)page_address(page);
offset = 0;
}
}
f2fs_put_page(page, 1);
+ return 0;
}
static int read_normal_summaries(struct f2fs_sb_info *sbi, int type)
@@ -3027,6 +3391,7 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type)
unsigned short blk_off;
unsigned int segno = 0;
block_t blk_addr = 0;
+ int err = 0;
/* get segment number and block addr */
if (IS_DATASEG(type)) {
@@ -3049,7 +3414,9 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type)
blk_addr = GET_SUM_BLOCK(sbi, segno);
}
- new = get_meta_page(sbi, blk_addr);
+ new = f2fs_get_meta_page(sbi, blk_addr);
+ if (IS_ERR(new))
+ return PTR_ERR(new);
sum = (struct f2fs_summary_block *)page_address(new);
if (IS_NODESEG(type)) {
@@ -3061,7 +3428,9 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type)
ns->ofs_in_node = 0;
}
} else {
- restore_node_summary(sbi, segno, sum);
+ err = f2fs_restore_node_summary(sbi, segno, sum);
+ if (err)
+ goto out;
}
}
@@ -3081,8 +3450,9 @@ static int read_normal_summaries(struct f2fs_sb_info *sbi, int type)
curseg->alloc_type = ckpt->alloc_type[type];
curseg->next_blkoff = blk_off;
mutex_unlock(&curseg->curseg_mutex);
+out:
f2fs_put_page(new, 1);
- return 0;
+ return err;
}
static int restore_curseg_summaries(struct f2fs_sb_info *sbi)
@@ -3093,19 +3463,21 @@ static int restore_curseg_summaries(struct f2fs_sb_info *sbi)
int err;
if (is_set_ckpt_flags(sbi, CP_COMPACT_SUM_FLAG)) {
- int npages = npages_for_summary_flush(sbi, true);
+ int npages = f2fs_npages_for_summary_flush(sbi, true);
if (npages >= 2)
- ra_meta_pages(sbi, start_sum_block(sbi), npages,
+ f2fs_ra_meta_pages(sbi, start_sum_block(sbi), npages,
META_CP, true);
/* restore for compacted data summary */
- read_compacted_summaries(sbi);
+ err = read_compacted_summaries(sbi);
+ if (err)
+ return err;
type = CURSEG_HOT_NODE;
}
if (__exist_node_summaries(sbi))
- ra_meta_pages(sbi, sum_blk_addr(sbi, NR_CURSEG_TYPE, type),
+ f2fs_ra_meta_pages(sbi, sum_blk_addr(sbi, NR_CURSEG_TYPE, type),
NR_CURSEG_TYPE - type, META_CP, true);
for (; type <= CURSEG_COLD_NODE; type++) {
@@ -3131,8 +3503,9 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr)
int written_size = 0;
int i, j;
- page = grab_meta_page(sbi, blkaddr++);
+ page = f2fs_grab_meta_page(sbi, blkaddr++);
kaddr = (unsigned char *)page_address(page);
+ memset(kaddr, 0, PAGE_SIZE);
/* Step 1: write nat cache */
seg_i = CURSEG_I(sbi, CURSEG_HOT_DATA);
@@ -3155,8 +3528,9 @@ static void write_compacted_summaries(struct f2fs_sb_info *sbi, block_t blkaddr)
for (j = 0; j < blkoff; j++) {
if (!page) {
- page = grab_meta_page(sbi, blkaddr++);
+ page = f2fs_grab_meta_page(sbi, blkaddr++);
kaddr = (unsigned char *)page_address(page);
+ memset(kaddr, 0, PAGE_SIZE);
written_size = 0;
}
summary = (struct f2fs_summary *)(kaddr + written_size);
@@ -3191,7 +3565,7 @@ static void write_normal_summaries(struct f2fs_sb_info *sbi,
write_current_sum_page(sbi, i, blkaddr + (i - type));
}
-void write_data_summaries(struct f2fs_sb_info *sbi, block_t start_blk)
+void f2fs_write_data_summaries(struct f2fs_sb_info *sbi, block_t start_blk)
{
if (is_set_ckpt_flags(sbi, CP_COMPACT_SUM_FLAG))
write_compacted_summaries(sbi, start_blk);
@@ -3199,12 +3573,12 @@ void write_data_summaries(struct f2fs_sb_info *sbi, block_t start_blk)
write_normal_summaries(sbi, start_blk, CURSEG_HOT_DATA);
}
-void write_node_summaries(struct f2fs_sb_info *sbi, block_t start_blk)
+void f2fs_write_node_summaries(struct f2fs_sb_info *sbi, block_t start_blk)
{
write_normal_summaries(sbi, start_blk, CURSEG_HOT_NODE);
}
-int lookup_journal_in_cursum(struct f2fs_journal *journal, int type,
+int f2fs_lookup_journal_in_cursum(struct f2fs_journal *journal, int type,
unsigned int val, int alloc)
{
int i;
@@ -3229,7 +3603,7 @@ int lookup_journal_in_cursum(struct f2fs_journal *journal, int type,
static struct page *get_current_sit_page(struct f2fs_sb_info *sbi,
unsigned int segno)
{
- return get_meta_page(sbi, current_sit_addr(sbi, segno));
+ return f2fs_get_meta_page_nofail(sbi, current_sit_addr(sbi, segno));
}
static struct page *get_next_sit_page(struct f2fs_sb_info *sbi,
@@ -3242,7 +3616,7 @@ static struct page *get_next_sit_page(struct f2fs_sb_info *sbi,
src_off = current_sit_addr(sbi, start);
dst_off = next_sit_addr(sbi, src_off);
- page = grab_meta_page(sbi, dst_off);
+ page = f2fs_grab_meta_page(sbi, dst_off);
seg_info_to_sit_page(sbi, page, start);
set_page_dirty(page);
@@ -3338,7 +3712,7 @@ static void remove_sits_in_journal(struct f2fs_sb_info *sbi)
* CP calls this function, which flushes SIT entries including sit_journal,
* and moves prefree segs to free segs.
*/
-void flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
+void f2fs_flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
{
struct sit_info *sit_i = SIT_I(sbi);
unsigned long *bitmap = sit_i->dirty_sentries_bitmap;
@@ -3397,6 +3771,11 @@ void flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
int offset, sit_offset;
se = get_seg_entry(sbi, segno);
+#ifdef CONFIG_F2FS_CHECK_FS
+ if (memcmp(se->cur_valid_map, se->cur_valid_map_mir,
+ SIT_VBLOCK_MAP_SIZE))
+ f2fs_bug_on(sbi, 1);
+#endif
/* add discard candidates */
if (!(cpc->reason & CP_DISCARD)) {
@@ -3405,17 +3784,21 @@ void flush_sit_entries(struct f2fs_sb_info *sbi, struct cp_control *cpc)
}
if (to_journal) {
- offset = lookup_journal_in_cursum(journal,
+ offset = f2fs_lookup_journal_in_cursum(journal,
SIT_JOURNAL, segno, 1);
f2fs_bug_on(sbi, offset < 0);
segno_in_journal(journal, offset) =
cpu_to_le32(segno);
seg_info_to_raw_sit(se,
&sit_in_journal(journal, offset));
+ check_block_count(sbi, segno,
+ &sit_in_journal(journal, offset));
} else {
sit_offset = SIT_ENTRY_OFFSET(sit_i, segno);
seg_info_to_raw_sit(se,
&raw_sit->entries[sit_offset]);
+ check_block_count(sbi, segno,
+ &raw_sit->entries[sit_offset]);
}
__clear_bit(segno, bitmap);
@@ -3463,8 +3846,10 @@ static int build_sit_info(struct f2fs_sb_info *sbi)
SM_I(sbi)->sit_info = sit_i;
- sit_i->sentries = f2fs_kvzalloc(sbi, MAIN_SEGS(sbi) *
- sizeof(struct seg_entry), GFP_KERNEL);
+ sit_i->sentries =
+ f2fs_kvzalloc(sbi, array_size(sizeof(struct seg_entry),
+ MAIN_SEGS(sbi)),
+ GFP_KERNEL);
if (!sit_i->sentries)
return -ENOMEM;
@@ -3490,22 +3875,22 @@ static int build_sit_info(struct f2fs_sb_info *sbi)
return -ENOMEM;
#endif
- if (f2fs_discard_en(sbi)) {
- sit_i->sentries[start].discard_map
- = f2fs_kzalloc(sbi, SIT_VBLOCK_MAP_SIZE,
- GFP_KERNEL);
- if (!sit_i->sentries[start].discard_map)
- return -ENOMEM;
- }
+ sit_i->sentries[start].discard_map
+ = f2fs_kzalloc(sbi, SIT_VBLOCK_MAP_SIZE,
+ GFP_KERNEL);
+ if (!sit_i->sentries[start].discard_map)
+ return -ENOMEM;
}
sit_i->tmp_map = f2fs_kzalloc(sbi, SIT_VBLOCK_MAP_SIZE, GFP_KERNEL);
if (!sit_i->tmp_map)
return -ENOMEM;
- if (sbi->segs_per_sec > 1) {
- sit_i->sec_entries = f2fs_kvzalloc(sbi, MAIN_SECS(sbi) *
- sizeof(struct sec_entry), GFP_KERNEL);
+ if (__is_large_section(sbi)) {
+ sit_i->sec_entries =
+ f2fs_kvzalloc(sbi, array_size(sizeof(struct sec_entry),
+ MAIN_SECS(sbi)),
+ GFP_KERNEL);
if (!sit_i->sec_entries)
return -ENOMEM;
}
@@ -3581,7 +3966,8 @@ static int build_curseg(struct f2fs_sb_info *sbi)
struct curseg_info *array;
int i;
- array = f2fs_kzalloc(sbi, sizeof(*array) * NR_CURSEG_TYPE, GFP_KERNEL);
+ array = f2fs_kzalloc(sbi, array_size(NR_CURSEG_TYPE, sizeof(*array)),
+ GFP_KERNEL);
if (!array)
return -ENOMEM;
@@ -3614,9 +4000,10 @@ static int build_sit_entries(struct f2fs_sb_info *sbi)
unsigned int i, start, end;
unsigned int readed, start_blk = 0;
int err = 0;
+ block_t total_node_blocks = 0;
do {
- readed = ra_meta_pages(sbi, start_blk, BIO_MAX_PAGES,
+ readed = f2fs_ra_meta_pages(sbi, start_blk, BIO_MAX_PAGES,
META_SIT, true);
start = start_blk * sit_i->sents_per_block;
@@ -3628,6 +4015,8 @@ static int build_sit_entries(struct f2fs_sb_info *sbi)
se = &sit_i->sentries[start];
page = get_current_sit_page(sbi, start);
+ if (IS_ERR(page))
+ return PTR_ERR(page);
sit_blk = (struct f2fs_sit_block *)page_address(page);
sit = sit_blk->entries[SIT_ENTRY_OFFSET(sit_i, start)];
f2fs_put_page(page, 1);
@@ -3636,23 +4025,23 @@ static int build_sit_entries(struct f2fs_sb_info *sbi)
if (err)
return err;
seg_info_from_raw_sit(se, &sit);
+ if (IS_NODESEG(se->type))
+ total_node_blocks += se->valid_blocks;
/* build discard map only one time */
- if (f2fs_discard_en(sbi)) {
- if (is_set_ckpt_flags(sbi, CP_TRIMMED_FLAG)) {
- memset(se->discard_map, 0xff,
- SIT_VBLOCK_MAP_SIZE);
- } else {
- memcpy(se->discard_map,
- se->cur_valid_map,
- SIT_VBLOCK_MAP_SIZE);
- sbi->discard_blks +=
- sbi->blocks_per_seg -
- se->valid_blocks;
- }
+ if (is_set_ckpt_flags(sbi, CP_TRIMMED_FLAG)) {
+ memset(se->discard_map, 0xff,
+ SIT_VBLOCK_MAP_SIZE);
+ } else {
+ memcpy(se->discard_map,
+ se->cur_valid_map,
+ SIT_VBLOCK_MAP_SIZE);
+ sbi->discard_blks +=
+ sbi->blocks_per_seg -
+ se->valid_blocks;
}
- if (sbi->segs_per_sec > 1)
+ if (__is_large_section(sbi))
get_sec_entry(sbi, start)->valid_blocks +=
se->valid_blocks;
}
@@ -3664,33 +4053,55 @@ static int build_sit_entries(struct f2fs_sb_info *sbi)
unsigned int old_valid_blocks;
start = le32_to_cpu(segno_in_journal(journal, i));
+ if (start >= MAIN_SEGS(sbi)) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "Wrong journal entry on segno %u",
+ start);
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ err = -EINVAL;
+ break;
+ }
+
se = &sit_i->sentries[start];
sit = sit_in_journal(journal, i);
old_valid_blocks = se->valid_blocks;
+ if (IS_NODESEG(se->type))
+ total_node_blocks -= old_valid_blocks;
err = check_block_count(sbi, start, &sit);
if (err)
break;
seg_info_from_raw_sit(se, &sit);
+ if (IS_NODESEG(se->type))
+ total_node_blocks += se->valid_blocks;
- if (f2fs_discard_en(sbi)) {
- if (is_set_ckpt_flags(sbi, CP_TRIMMED_FLAG)) {
- memset(se->discard_map, 0xff,
- SIT_VBLOCK_MAP_SIZE);
- } else {
- memcpy(se->discard_map, se->cur_valid_map,
- SIT_VBLOCK_MAP_SIZE);
- sbi->discard_blks += old_valid_blocks -
- se->valid_blocks;
- }
+ if (is_set_ckpt_flags(sbi, CP_TRIMMED_FLAG)) {
+ memset(se->discard_map, 0xff, SIT_VBLOCK_MAP_SIZE);
+ } else {
+ memcpy(se->discard_map, se->cur_valid_map,
+ SIT_VBLOCK_MAP_SIZE);
+ sbi->discard_blks += old_valid_blocks;
+ sbi->discard_blks -= se->valid_blocks;
}
- if (sbi->segs_per_sec > 1)
+ if (__is_large_section(sbi)) {
get_sec_entry(sbi, start)->valid_blocks +=
- se->valid_blocks - old_valid_blocks;
+ se->valid_blocks;
+ get_sec_entry(sbi, start)->valid_blocks -=
+ old_valid_blocks;
+ }
}
up_read(&curseg->journal_rwsem);
+
+ if (!err && total_node_blocks != valid_node_count(sbi)) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "SIT is corrupted node# %u vs %u",
+ total_node_blocks, valid_node_count(sbi));
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ err = -EINVAL;
+ }
+
return err;
}
@@ -3789,7 +4200,7 @@ static void init_min_max_mtime(struct f2fs_sb_info *sbi)
down_write(&sit_i->sentry_lock);
- sit_i->min_mtime = LLONG_MAX;
+ sit_i->min_mtime = ULLONG_MAX;
for (segno = 0; segno < MAIN_SEGS(sbi); segno += sbi->segs_per_sec) {
unsigned int i;
@@ -3803,11 +4214,11 @@ static void init_min_max_mtime(struct f2fs_sb_info *sbi)
if (sit_i->min_mtime > mtime)
sit_i->min_mtime = mtime;
}
- sit_i->max_mtime = get_mtime(sbi);
+ sit_i->max_mtime = get_mtime(sbi, false);
up_write(&sit_i->sentry_lock);
}
-int build_segment_manager(struct f2fs_sb_info *sbi)
+int f2fs_build_segment_manager(struct f2fs_sb_info *sbi)
{
struct f2fs_super_block *raw_super = F2FS_RAW_SUPER(sbi);
struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi);
@@ -3845,7 +4256,7 @@ int build_segment_manager(struct f2fs_sb_info *sbi)
init_rwsem(&sm_info->curseg_lock);
if (!f2fs_readonly(sbi->sb)) {
- err = create_flush_cmd_control(sbi);
+ err = f2fs_create_flush_cmd_control(sbi);
if (err)
return err;
}
@@ -3909,7 +4320,7 @@ static void destroy_dirty_segmap(struct f2fs_sb_info *sbi)
destroy_victim_secmap(sbi);
SM_I(sbi)->dirty_info = NULL;
- kfree(dirty_i);
+ kvfree(dirty_i);
}
static void destroy_curseg(struct f2fs_sb_info *sbi)
@@ -3921,10 +4332,10 @@ static void destroy_curseg(struct f2fs_sb_info *sbi)
return;
SM_I(sbi)->curseg_array = NULL;
for (i = 0; i < NR_CURSEG_TYPE; i++) {
- kfree(array[i].sum_blk);
- kfree(array[i].journal);
+ kvfree(array[i].sum_blk);
+ kvfree(array[i].journal);
}
- kfree(array);
+ kvfree(array);
}
static void destroy_free_segmap(struct f2fs_sb_info *sbi)
@@ -3935,7 +4346,7 @@ static void destroy_free_segmap(struct f2fs_sb_info *sbi)
SM_I(sbi)->free_info = NULL;
kvfree(free_i->free_segmap);
kvfree(free_i->free_secmap);
- kfree(free_i);
+ kvfree(free_i);
}
static void destroy_sit_info(struct f2fs_sb_info *sbi)
@@ -3948,45 +4359,45 @@ static void destroy_sit_info(struct f2fs_sb_info *sbi)
if (sit_i->sentries) {
for (start = 0; start < MAIN_SEGS(sbi); start++) {
- kfree(sit_i->sentries[start].cur_valid_map);
+ kvfree(sit_i->sentries[start].cur_valid_map);
#ifdef CONFIG_F2FS_CHECK_FS
- kfree(sit_i->sentries[start].cur_valid_map_mir);
+ kvfree(sit_i->sentries[start].cur_valid_map_mir);
#endif
- kfree(sit_i->sentries[start].ckpt_valid_map);
- kfree(sit_i->sentries[start].discard_map);
+ kvfree(sit_i->sentries[start].ckpt_valid_map);
+ kvfree(sit_i->sentries[start].discard_map);
}
}
- kfree(sit_i->tmp_map);
+ kvfree(sit_i->tmp_map);
kvfree(sit_i->sentries);
kvfree(sit_i->sec_entries);
kvfree(sit_i->dirty_sentries_bitmap);
SM_I(sbi)->sit_info = NULL;
- kfree(sit_i->sit_bitmap);
+ kvfree(sit_i->sit_bitmap);
#ifdef CONFIG_F2FS_CHECK_FS
- kfree(sit_i->sit_bitmap_mir);
+ kvfree(sit_i->sit_bitmap_mir);
#endif
- kfree(sit_i);
+ kvfree(sit_i);
}
-void destroy_segment_manager(struct f2fs_sb_info *sbi)
+void f2fs_destroy_segment_manager(struct f2fs_sb_info *sbi)
{
struct f2fs_sm_info *sm_info = SM_I(sbi);
if (!sm_info)
return;
- destroy_flush_cmd_control(sbi, true);
+ f2fs_destroy_flush_cmd_control(sbi, true);
destroy_discard_cmd_control(sbi);
destroy_dirty_segmap(sbi);
destroy_curseg(sbi);
destroy_free_segmap(sbi);
destroy_sit_info(sbi);
sbi->sm_info = NULL;
- kfree(sm_info);
+ kvfree(sm_info);
}
-int __init create_segment_manager_caches(void)
+int __init f2fs_create_segment_manager_caches(void)
{
discard_entry_slab = f2fs_kmem_cache_create("discard_entry",
sizeof(struct discard_entry));
@@ -4019,7 +4430,7 @@ int __init create_segment_manager_caches(void)
return -ENOMEM;
}
-void destroy_segment_manager_caches(void)
+void f2fs_destroy_segment_manager_caches(void)
{
kmem_cache_destroy(sit_entry_set_slab);
kmem_cache_destroy(discard_cmd_slab);
diff --git a/fs/f2fs/segment.h b/fs/f2fs/segment.h
index 3325d07..a77f76f 100644
--- a/fs/f2fs/segment.h
+++ b/fs/f2fs/segment.h
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/segment.h
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/blkdev.h>
#include <linux/backing-dev.h>
@@ -85,7 +82,7 @@
(GET_SEGOFF_FROM_SEG0(sbi, blk_addr) & ((sbi)->blocks_per_seg - 1))
#define GET_SEGNO(sbi, blk_addr) \
- ((((blk_addr) == NULL_ADDR) || ((blk_addr) == NEW_ADDR)) ? \
+ ((!is_valid_data_blkaddr(sbi, blk_addr)) ? \
NULL_SEGNO : GET_L2R_SEGNO(FREE_I(sbi), \
GET_SEGNO_FROM_SEG0(sbi, blk_addr)))
#define BLKS_PER_SEC(sbi) \
@@ -215,6 +212,8 @@ struct segment_allocation {
#define IS_DUMMY_WRITTEN_PAGE(page) \
(page_private(page) == (unsigned long)DUMMY_WRITTEN_PAGE)
+#define MAX_SKIP_GC_COUNT 16
+
struct inmem_pages {
struct list_head list;
struct page *page;
@@ -334,12 +333,18 @@ static inline unsigned int get_valid_blocks(struct f2fs_sb_info *sbi,
* In order to get # of valid blocks in a section instantly from many
* segments, f2fs manages two counting structures separately.
*/
- if (use_section && sbi->segs_per_sec > 1)
+ if (use_section && __is_large_section(sbi))
return get_sec_entry(sbi, segno)->valid_blocks;
else
return get_seg_entry(sbi, segno)->valid_blocks;
}
+static inline unsigned int get_ckpt_valid_blocks(struct f2fs_sb_info *sbi,
+ unsigned int segno)
+{
+ return get_seg_entry(sbi, segno)->ckpt_valid_blocks;
+}
+
static inline void seg_info_from_raw_sit(struct seg_entry *se,
struct f2fs_sit_entry *rs)
{
@@ -375,6 +380,7 @@ static inline void seg_info_to_sit_page(struct f2fs_sb_info *sbi,
int i;
raw_sit = (struct f2fs_sit_block *)page_address(page);
+ memset(raw_sit, 0, PAGE_SIZE);
for (i = 0; i < end - start; i++) {
rs = &raw_sit->entries[i];
se = get_seg_entry(sbi, start + i);
@@ -445,6 +451,8 @@ static inline void __set_test_and_free(struct f2fs_sb_info *sbi,
if (test_and_clear_bit(segno, free_i->free_segmap)) {
free_i->free_segments++;
+ if (IS_CURSEC(sbi, secno))
+ goto skip_free;
next = find_next_bit(free_i->free_segmap,
start_segno + sbi->segs_per_sec, start_segno);
if (next >= start_segno + sbi->segs_per_sec) {
@@ -452,6 +460,7 @@ static inline void __set_test_and_free(struct f2fs_sb_info *sbi,
free_i->free_sections++;
}
}
+skip_free:
spin_unlock(&free_i->segmap_lock);
}
@@ -573,6 +582,15 @@ static inline bool has_not_enough_free_secs(struct f2fs_sb_info *sbi,
reserved_sections(sbi) + needed);
}
+static inline int f2fs_is_checkpoint_ready(struct f2fs_sb_info *sbi)
+{
+ if (likely(!is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
+ return 0;
+ if (likely(!has_not_enough_free_secs(sbi, 0, 0)))
+ return 0;
+ return -ENOSPC;
+}
+
static inline bool excess_prefree_segs(struct f2fs_sb_info *sbi)
{
return prefree_segments(sbi) > SM_I(sbi)->rec_prefree_segments;
@@ -642,13 +660,10 @@ static inline void verify_block_addr(struct f2fs_io_info *fio, block_t blk_addr)
{
struct f2fs_sb_info *sbi = fio->sbi;
- if (PAGE_TYPE_OF_BIO(fio->type) == META &&
- (!is_read_io(fio->op) || fio->is_meta))
- BUG_ON(blk_addr < SEG0_BLKADDR(sbi) ||
- blk_addr >= MAIN_BLKADDR(sbi));
+ if (__is_meta_io(fio))
+ verify_blkaddr(sbi, blk_addr, META_GENERIC);
else
- BUG_ON(blk_addr < MAIN_BLKADDR(sbi) ||
- blk_addr >= MAX_BLKADDR(sbi));
+ verify_blkaddr(sbi, blk_addr, DATA_GENERIC);
}
/*
@@ -742,12 +757,23 @@ static inline void set_to_next_sit(struct sit_info *sit_i, unsigned int start)
#endif
}
-static inline unsigned long long get_mtime(struct f2fs_sb_info *sbi)
+static inline unsigned long long get_mtime(struct f2fs_sb_info *sbi,
+ bool base_time)
{
struct sit_info *sit_i = SIT_I(sbi);
- time64_t now = ktime_get_real_seconds();
+ time64_t diff, now = ktime_get_real_seconds();
- return sit_i->elapsed_time + now - sit_i->mounted_time;
+ if (now >= sit_i->mounted_time)
+ return sit_i->elapsed_time + now - sit_i->mounted_time;
+
+ /* system time is set to the past */
+ if (!base_time) {
+ diff = sit_i->mounted_time - now;
+ if (sit_i->elapsed_time >= diff)
+ return sit_i->elapsed_time - diff;
+ return 0;
+ }
+ return sit_i->elapsed_time;
}
static inline void set_summary(struct f2fs_summary *sum, nid_t nid,
@@ -771,15 +797,6 @@ static inline block_t sum_blk_addr(struct f2fs_sb_info *sbi, int base, int type)
- (base + 1) + type;
}
-static inline bool no_fggc_candidate(struct f2fs_sb_info *sbi,
- unsigned int secno)
-{
- if (get_valid_blocks(sbi, GET_SEG_FROM_SEC(sbi, secno), true) >
- sbi->fggc_threshold)
- return true;
- return false;
-}
-
static inline bool sec_usage_check(struct f2fs_sb_info *sbi, unsigned int secno)
{
if (IS_CURSEC(sbi, secno) || (sbi->cur_victim_sec == secno))
diff --git a/fs/f2fs/shrinker.c b/fs/f2fs/shrinker.c
index 0b5664a..a467aca 100644
--- a/fs/f2fs/shrinker.c
+++ b/fs/f2fs/shrinker.c
@@ -1,13 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* f2fs shrinker support
* the basic infra was copied from fs/ubifs/shrinker.c
*
* Copyright (c) 2015 Motorola Mobility
* Copyright (c) 2015 Jaegeuk Kim <jaegeuk@kernel.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
@@ -109,11 +106,11 @@ unsigned long f2fs_shrink_scan(struct shrinker *shrink,
/* shrink clean nat cache entries */
if (freed < nr)
- freed += try_to_free_nats(sbi, nr - freed);
+ freed += f2fs_try_to_free_nats(sbi, nr - freed);
/* shrink free nids cache entries */
if (freed < nr)
- freed += try_to_free_nids(sbi, nr - freed);
+ freed += f2fs_try_to_free_nids(sbi, nr - freed);
spin_lock(&f2fs_list_lock);
p = p->next;
@@ -138,6 +135,6 @@ void f2fs_leave_shrinker(struct f2fs_sb_info *sbi)
f2fs_shrink_extent_tree(sbi, __count_extent_cache(sbi));
spin_lock(&f2fs_list_lock);
- list_del(&sbi->s_list);
+ list_del_init(&sbi->s_list);
spin_unlock(&f2fs_list_lock);
}
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index ee2ebbf..f81a655 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/super.c
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/module.h>
#include <linux/init.h>
@@ -41,7 +38,7 @@ static struct kmem_cache *f2fs_inode_cachep;
#ifdef CONFIG_F2FS_FAULT_INJECTION
-char *fault_name[FAULT_MAX] = {
+const char *f2fs_fault_name[FAULT_MAX] = {
[FAULT_KMALLOC] = "kmalloc",
[FAULT_KVMALLOC] = "kvmalloc",
[FAULT_PAGE_ALLOC] = "page alloc",
@@ -53,22 +50,27 @@ char *fault_name[FAULT_MAX] = {
[FAULT_DIR_DEPTH] = "too big dir depth",
[FAULT_EVICT_INODE] = "evict_inode fail",
[FAULT_TRUNCATE] = "truncate fail",
- [FAULT_IO] = "IO error",
+ [FAULT_READ_IO] = "read IO error",
[FAULT_CHECKPOINT] = "checkpoint error",
+ [FAULT_DISCARD] = "discard error",
+ [FAULT_WRITE_IO] = "write IO error",
};
-static void f2fs_build_fault_attr(struct f2fs_sb_info *sbi,
- unsigned int rate)
+void f2fs_build_fault_attr(struct f2fs_sb_info *sbi, unsigned int rate,
+ unsigned int type)
{
struct f2fs_fault_info *ffi = &F2FS_OPTION(sbi).fault_info;
if (rate) {
atomic_set(&ffi->inject_ops, 0);
ffi->inject_rate = rate;
- ffi->inject_type = (1 << FAULT_MAX) - 1;
- } else {
- memset(ffi, 0, sizeof(struct f2fs_fault_info));
}
+
+ if (type)
+ ffi->inject_type = type;
+
+ if (!rate && !type)
+ memset(ffi, 0, sizeof(struct f2fs_fault_info));
}
#endif
@@ -113,6 +115,7 @@ enum {
Opt_mode,
Opt_io_size_bits,
Opt_fault_injection,
+ Opt_fault_type,
Opt_lazytime,
Opt_nolazytime,
Opt_quota,
@@ -133,6 +136,7 @@ enum {
Opt_alloc,
Opt_fsync,
Opt_test_dummy_encryption,
+ Opt_checkpoint,
Opt_err,
};
@@ -170,6 +174,7 @@ static match_table_t f2fs_tokens = {
{Opt_mode, "mode=%s"},
{Opt_io_size_bits, "io_bits=%u"},
{Opt_fault_injection, "fault_injection=%u"},
+ {Opt_fault_type, "fault_type=%u"},
{Opt_lazytime, "lazytime"},
{Opt_nolazytime, "nolazytime"},
{Opt_quota, "quota"},
@@ -190,6 +195,7 @@ static match_table_t f2fs_tokens = {
{Opt_alloc, "alloc_mode=%s"},
{Opt_fsync, "fsync_mode=%s"},
{Opt_test_dummy_encryption, "test_dummy_encryption"},
+ {Opt_checkpoint, "checkpoint=%s"},
{Opt_err, NULL},
};
@@ -201,7 +207,7 @@ void f2fs_msg(struct super_block *sb, const char *level, const char *fmt, ...)
va_start(args, fmt);
vaf.fmt = fmt;
vaf.va = &args;
- printk_ratelimited("%sF2FS-fs (%s): %pV\n", level, sb->s_id, &vaf);
+ printk("%sF2FS-fs (%s): %pV\n", level, sb->s_id, &vaf);
va_end(args);
}
@@ -253,7 +259,7 @@ static int f2fs_set_qf_name(struct super_block *sb, int qtype,
"quota options when quota turned on");
return -EINVAL;
}
- if (f2fs_sb_has_quota_ino(sb)) {
+ if (f2fs_sb_has_quota_ino(sbi)) {
f2fs_msg(sb, KERN_INFO,
"QUOTA feature is enabled, so ignore qf_name");
return 0;
@@ -283,7 +289,7 @@ static int f2fs_set_qf_name(struct super_block *sb, int qtype,
set_opt(sbi, QUOTA);
return 0;
errout:
- kfree(qname);
+ kvfree(qname);
return ret;
}
@@ -296,7 +302,7 @@ static int f2fs_clear_qf_name(struct super_block *sb, int qtype)
" when quota turned on");
return -EINVAL;
}
- kfree(F2FS_OPTION(sbi).s_qf_names[qtype]);
+ kvfree(F2FS_OPTION(sbi).s_qf_names[qtype]);
F2FS_OPTION(sbi).s_qf_names[qtype] = NULL;
return 0;
}
@@ -308,7 +314,7 @@ static int f2fs_check_quota_options(struct f2fs_sb_info *sbi)
* 'grpquota' mount options are allowed even without quota feature
* to support legacy quotas in quota files.
*/
- if (test_opt(sbi, PRJQUOTA) && !f2fs_sb_has_project_quota(sbi->sb)) {
+ if (test_opt(sbi, PRJQUOTA) && !f2fs_sb_has_project_quota(sbi)) {
f2fs_msg(sbi->sb, KERN_ERR, "Project quota feature not enabled. "
"Cannot enable project quota enforcement.");
return -1;
@@ -342,17 +348,11 @@ static int f2fs_check_quota_options(struct f2fs_sb_info *sbi)
}
}
- if (f2fs_sb_has_quota_ino(sbi->sb) && F2FS_OPTION(sbi).s_jquota_fmt) {
+ if (f2fs_sb_has_quota_ino(sbi) && F2FS_OPTION(sbi).s_jquota_fmt) {
f2fs_msg(sbi->sb, KERN_INFO,
"QUOTA feature is enabled, so ignore jquota_fmt");
F2FS_OPTION(sbi).s_jquota_fmt = 0;
}
- if (f2fs_sb_has_quota_ino(sbi->sb) && f2fs_readonly(sbi->sb)) {
- f2fs_msg(sbi->sb, KERN_INFO,
- "Filesystem with quota feature cannot be mounted RDWR "
- "without CONFIG_QUOTA");
- return -1;
- }
return 0;
}
#endif
@@ -360,7 +360,6 @@ static int f2fs_check_quota_options(struct f2fs_sb_info *sbi)
static int parse_options(struct super_block *sb, char *options)
{
struct f2fs_sb_info *sbi = F2FS_SB(sb);
- struct request_queue *q;
substring_t args[MAX_OPT_ARGS];
char *p, *name;
int arg = 0;
@@ -400,10 +399,10 @@ static int parse_options(struct super_block *sb, char *options)
set_opt(sbi, BG_GC);
set_opt(sbi, FORCE_FG_GC);
} else {
- kfree(name);
+ kvfree(name);
return -EINVAL;
}
- kfree(name);
+ kvfree(name);
break;
case Opt_disable_roll_forward:
set_opt(sbi, DISABLE_ROLL_FORWARD);
@@ -415,17 +414,10 @@ static int parse_options(struct super_block *sb, char *options)
return -EINVAL;
break;
case Opt_discard:
- q = bdev_get_queue(sb->s_bdev);
- if (blk_queue_discard(q)) {
- set_opt(sbi, DISCARD);
- } else if (!f2fs_sb_has_blkzoned(sb)) {
- f2fs_msg(sb, KERN_WARNING,
- "mounting with \"discard\" option, but "
- "the device does not support discard");
- }
+ set_opt(sbi, DISCARD);
break;
case Opt_nodiscard:
- if (f2fs_sb_has_blkzoned(sb)) {
+ if (f2fs_sb_has_blkzoned(sbi)) {
f2fs_msg(sb, KERN_WARNING,
"discard is required for zoned block devices");
return -EINVAL;
@@ -574,11 +566,11 @@ static int parse_options(struct super_block *sb, char *options)
return -ENOMEM;
if (strlen(name) == 8 &&
!strncmp(name, "adaptive", 8)) {
- if (f2fs_sb_has_blkzoned(sb)) {
+ if (f2fs_sb_has_blkzoned(sbi)) {
f2fs_msg(sb, KERN_WARNING,
"adaptive mode is not allowed with "
"zoned block device feature");
- kfree(name);
+ kvfree(name);
return -EINVAL;
}
set_opt_mode(sbi, F2FS_MOUNT_ADAPTIVE);
@@ -586,10 +578,10 @@ static int parse_options(struct super_block *sb, char *options)
!strncmp(name, "lfs", 3)) {
set_opt_mode(sbi, F2FS_MOUNT_LFS);
} else {
- kfree(name);
+ kvfree(name);
return -EINVAL;
}
- kfree(name);
+ kvfree(name);
break;
case Opt_io_size_bits:
if (args->from && match_int(args, &arg))
@@ -602,17 +594,31 @@ static int parse_options(struct super_block *sb, char *options)
}
F2FS_OPTION(sbi).write_io_size_bits = arg;
break;
+#ifdef CONFIG_F2FS_FAULT_INJECTION
case Opt_fault_injection:
if (args->from && match_int(args, &arg))
return -EINVAL;
-#ifdef CONFIG_F2FS_FAULT_INJECTION
- f2fs_build_fault_attr(sbi, arg);
+ f2fs_build_fault_attr(sbi, arg, F2FS_ALL_FAULT_TYPE);
set_opt(sbi, FAULT_INJECTION);
-#else
- f2fs_msg(sb, KERN_INFO,
- "FAULT_INJECTION was not selected");
-#endif
break;
+
+ case Opt_fault_type:
+ if (args->from && match_int(args, &arg))
+ return -EINVAL;
+ f2fs_build_fault_attr(sbi, 0, arg);
+ set_opt(sbi, FAULT_INJECTION);
+ break;
+#else
+ case Opt_fault_injection:
+ f2fs_msg(sb, KERN_INFO,
+ "fault_injection options not supported");
+ break;
+
+ case Opt_fault_type:
+ f2fs_msg(sb, KERN_INFO,
+ "fault_type options not supported");
+ break;
+#endif
case Opt_lazytime:
sb->s_flags |= MS_LAZYTIME;
break;
@@ -708,10 +714,10 @@ static int parse_options(struct super_block *sb, char *options)
!strncmp(name, "fs-based", 8)) {
F2FS_OPTION(sbi).whint_mode = WHINT_MODE_FS;
} else {
- kfree(name);
+ kvfree(name);
return -EINVAL;
}
- kfree(name);
+ kvfree(name);
break;
case Opt_alloc:
name = match_strdup(&args[0]);
@@ -725,10 +731,10 @@ static int parse_options(struct super_block *sb, char *options)
!strncmp(name, "reuse", 5)) {
F2FS_OPTION(sbi).alloc_mode = ALLOC_MODE_REUSE;
} else {
- kfree(name);
+ kvfree(name);
return -EINVAL;
}
- kfree(name);
+ kvfree(name);
break;
case Opt_fsync:
name = match_strdup(&args[0]);
@@ -745,14 +751,14 @@ static int parse_options(struct super_block *sb, char *options)
F2FS_OPTION(sbi).fsync_mode =
FSYNC_MODE_NOBARRIER;
} else {
- kfree(name);
+ kvfree(name);
return -EINVAL;
}
- kfree(name);
+ kvfree(name);
break;
case Opt_test_dummy_encryption:
#ifdef CONFIG_F2FS_FS_ENCRYPTION
- if (!f2fs_sb_has_encrypt(sb)) {
+ if (!f2fs_sb_has_encrypt(sbi)) {
f2fs_msg(sb, KERN_ERR, "Encrypt feature is off");
return -EINVAL;
}
@@ -765,6 +771,23 @@ static int parse_options(struct super_block *sb, char *options)
"Test dummy encryption mount option ignored");
#endif
break;
+ case Opt_checkpoint:
+ name = match_strdup(&args[0]);
+ if (!name)
+ return -ENOMEM;
+
+ if (strlen(name) == 6 &&
+ !strncmp(name, "enable", 6)) {
+ clear_opt(sbi, DISABLE_CHECKPOINT);
+ } else if (strlen(name) == 7 &&
+ !strncmp(name, "disable", 7)) {
+ set_opt(sbi, DISABLE_CHECKPOINT);
+ } else {
+ kvfree(name);
+ return -EINVAL;
+ }
+ kvfree(name);
+ break;
default:
f2fs_msg(sb, KERN_ERR,
"Unrecognized mount option \"%s\" or missing value",
@@ -775,6 +798,19 @@ static int parse_options(struct super_block *sb, char *options)
#ifdef CONFIG_QUOTA
if (f2fs_check_quota_options(sbi))
return -EINVAL;
+#else
+ if (f2fs_sb_has_quota_ino(sbi) && !f2fs_readonly(sbi->sb)) {
+ f2fs_msg(sbi->sb, KERN_INFO,
+ "Filesystem with quota feature cannot be mounted RDWR "
+ "without CONFIG_QUOTA");
+ return -EINVAL;
+ }
+ if (f2fs_sb_has_project_quota(sbi) && !f2fs_readonly(sbi->sb)) {
+ f2fs_msg(sb, KERN_ERR,
+ "Filesystem with project quota feature cannot be "
+ "mounted RDWR without CONFIG_QUOTA");
+ return -EINVAL;
+ }
#endif
if (F2FS_IO_SIZE_BITS(sbi) && !test_opt(sbi, LFS)) {
@@ -785,8 +821,8 @@ static int parse_options(struct super_block *sb, char *options)
}
if (test_opt(sbi, INLINE_XATTR_SIZE)) {
- if (!f2fs_sb_has_extra_attr(sb) ||
- !f2fs_sb_has_flexible_inline_xattr(sb)) {
+ if (!f2fs_sb_has_extra_attr(sbi) ||
+ !f2fs_sb_has_flexible_inline_xattr(sbi)) {
f2fs_msg(sb, KERN_ERR,
"extra_attr or flexible_inline_xattr "
"feature is off");
@@ -810,6 +846,12 @@ static int parse_options(struct super_block *sb, char *options)
}
}
+ if (test_opt(sbi, DISABLE_CHECKPOINT) && test_opt(sbi, LFS)) {
+ f2fs_msg(sb, KERN_ERR,
+ "LFS not compatible with checkpoint=disable\n");
+ return -EINVAL;
+ }
+
/* Not pass down write hints if the number of active logs is lesser
* than NR_CURSEG_TYPE.
*/
@@ -830,15 +872,14 @@ static struct inode *f2fs_alloc_inode(struct super_block *sb)
/* Initialize f2fs-specific inode info */
atomic_set(&fi->dirty_pages, 0);
- fi->i_current_depth = 1;
init_rwsem(&fi->i_sem);
INIT_LIST_HEAD(&fi->dirty_list);
INIT_LIST_HEAD(&fi->gdirty_list);
INIT_LIST_HEAD(&fi->inmem_ilist);
INIT_LIST_HEAD(&fi->inmem_pages);
mutex_init(&fi->inmem_lock);
- init_rwsem(&fi->dio_rwsem[READ]);
- init_rwsem(&fi->dio_rwsem[WRITE]);
+ init_rwsem(&fi->i_gc_rwsem[READ]);
+ init_rwsem(&fi->i_gc_rwsem[WRITE]);
init_rwsem(&fi->i_mmap_sem);
init_rwsem(&fi->i_xattr_sem);
@@ -866,7 +907,7 @@ static int f2fs_drop_inode(struct inode *inode)
/* some remained atomic pages should discarded */
if (f2fs_is_atomic_file(inode))
- drop_inmem_pages(inode);
+ f2fs_drop_inmem_pages(inode);
/* should remain fi->extent_tree for writepage */
f2fs_destroy_extent_node(inode);
@@ -976,10 +1017,37 @@ static void destroy_device_list(struct f2fs_sb_info *sbi)
for (i = 0; i < sbi->s_ndevs; i++) {
blkdev_put(FDEV(i).bdev, FMODE_EXCL);
#ifdef CONFIG_BLK_DEV_ZONED
- kfree(FDEV(i).blkz_type);
+ kvfree(FDEV(i).blkz_type);
#endif
}
- kfree(sbi->devs);
+ kvfree(sbi->devs);
+}
+
+static void f2fs_umount_end(struct super_block *sb, int flags)
+{
+ /*
+ * this is called at the end of umount(2). If there is an unclosed
+ * namespace, f2fs won't do put_super() which triggers fsck in the
+ * next boot.
+ */
+ if ((flags & MNT_FORCE) || atomic_read(&sb->s_active) > 1) {
+ /* to write the latest kbytes_written */
+ if (!(sb->s_flags & MS_RDONLY)) {
+ struct cp_control cpc = {
+ .reason = CP_UMOUNT,
+ };
+ int locked, err;
+
+ /* only failed during mount/umount/freeze/quotactl */
+ locked = down_read_trylock(&sb->s_umount);
+ err = f2fs_quota_sync(sb, -1);
+ if (locked)
+ up_read(&sb->s_umount);
+ if (err)
+ return;
+ f2fs_write_checkpoint(F2FS_SB(sb), &cpc);
+ }
+ }
}
static void f2fs_put_super(struct super_block *sb)
@@ -998,32 +1066,30 @@ static void f2fs_put_super(struct super_block *sb)
* But, the previous checkpoint was not done by umount, it needs to do
* clean checkpoint again.
*/
- if (is_sbi_flag_set(sbi, SBI_IS_DIRTY) ||
- !is_set_ckpt_flags(sbi, CP_UMOUNT_FLAG)) {
+ if ((is_sbi_flag_set(sbi, SBI_IS_DIRTY) ||
+ !is_set_ckpt_flags(sbi, CP_UMOUNT_FLAG))) {
struct cp_control cpc = {
.reason = CP_UMOUNT,
};
- write_checkpoint(sbi, &cpc);
+ f2fs_write_checkpoint(sbi, &cpc);
}
/* be sure to wait for any on-going discard commands */
dropped = f2fs_wait_discard_bios(sbi);
- if (f2fs_discard_en(sbi) && !sbi->discard_blks && !dropped) {
+ if ((f2fs_hw_support_discard(sbi) || f2fs_hw_should_discard(sbi)) &&
+ !sbi->discard_blks && !dropped) {
struct cp_control cpc = {
.reason = CP_UMOUNT | CP_TRIMMED,
};
- write_checkpoint(sbi, &cpc);
+ f2fs_write_checkpoint(sbi, &cpc);
}
- /* write_checkpoint can update stat informaion */
- f2fs_destroy_stats(sbi);
-
/*
* normally superblock is clean, so we need to release this.
* In addition, EIO will skip do checkpoint, we need this as well.
*/
- release_ino_entry(sbi, true);
+ f2fs_release_ino_entry(sbi, true);
f2fs_leave_shrinker(sbi);
mutex_unlock(&sbi->umount_mutex);
@@ -1031,32 +1097,45 @@ static void f2fs_put_super(struct super_block *sb)
/* our cp_error case, we can wait for any writeback page */
f2fs_flush_merged_writes(sbi);
+ f2fs_wait_on_all_pages_writeback(sbi);
+
+ f2fs_bug_on(sbi, sbi->fsync_node_num);
+
iput(sbi->node_inode);
+ sbi->node_inode = NULL;
+
iput(sbi->meta_inode);
+ sbi->meta_inode = NULL;
+
+ /*
+ * iput() can update stat information, if f2fs_write_checkpoint()
+ * above failed with error.
+ */
+ f2fs_destroy_stats(sbi);
/* destroy f2fs internal modules */
- destroy_node_manager(sbi);
- destroy_segment_manager(sbi);
+ f2fs_destroy_node_manager(sbi);
+ f2fs_destroy_segment_manager(sbi);
- kfree(sbi->ckpt);
+ kvfree(sbi->ckpt);
f2fs_unregister_sysfs(sbi);
sb->s_fs_info = NULL;
if (sbi->s_chksum_driver)
crypto_free_shash(sbi->s_chksum_driver);
- kfree(sbi->raw_super);
+ kvfree(sbi->raw_super);
destroy_device_list(sbi);
mempool_destroy(sbi->write_io_dummy);
#ifdef CONFIG_QUOTA
for (i = 0; i < MAXQUOTAS; i++)
- kfree(F2FS_OPTION(sbi).s_qf_names[i]);
+ kvfree(F2FS_OPTION(sbi).s_qf_names[i]);
#endif
destroy_percpu_info(sbi);
for (i = 0; i < NR_PAGE_TYPE; i++)
- kfree(sbi->write_io[i]);
- kfree(sbi);
+ kvfree(sbi->write_io[i]);
+ kvfree(sbi);
}
int f2fs_sync_fs(struct super_block *sb, int sync)
@@ -1066,6 +1145,8 @@ int f2fs_sync_fs(struct super_block *sb, int sync)
if (unlikely(f2fs_cp_error(sbi)))
return 0;
+ if (unlikely(is_sbi_flag_set(sbi, SBI_CP_DISABLED)))
+ return 0;
trace_f2fs_sync_fs(sb, sync);
@@ -1078,7 +1159,7 @@ int f2fs_sync_fs(struct super_block *sb, int sync)
cpc.reason = __get_cp_reason(sbi);
mutex_lock(&sbi->gc_mutex);
- err = write_checkpoint(sbi, &cpc);
+ err = f2fs_write_checkpoint(sbi, &cpc);
mutex_unlock(&sbi->gc_mutex);
}
f2fs_trace_ios(NULL, 1);
@@ -1165,6 +1246,11 @@ static int f2fs_statfs(struct dentry *dentry, struct kstatfs *buf)
buf->f_blocks = total_count - start_count;
buf->f_bfree = user_block_count - valid_user_blocks(sbi) -
sbi->current_reserved_blocks;
+ if (unlikely(buf->f_bfree <= sbi->unusable_block_count))
+ buf->f_bfree = 0;
+ else
+ buf->f_bfree -= sbi->unusable_block_count;
+
if (buf->f_bfree > F2FS_OPTION(sbi).root_reserved_blocks)
buf->f_bavail = buf->f_bfree -
F2FS_OPTION(sbi).root_reserved_blocks;
@@ -1309,11 +1395,15 @@ static int f2fs_show_options(struct seq_file *seq, struct dentry *root)
from_kgid_munged(&init_user_ns,
F2FS_OPTION(sbi).s_resgid));
if (F2FS_IO_SIZE_BITS(sbi))
- seq_printf(seq, ",io_size=%uKB", F2FS_IO_SIZE_KB(sbi));
+ seq_printf(seq, ",io_bits=%u",
+ F2FS_OPTION(sbi).write_io_size_bits);
#ifdef CONFIG_F2FS_FAULT_INJECTION
- if (test_opt(sbi, FAULT_INJECTION))
+ if (test_opt(sbi, FAULT_INJECTION)) {
seq_printf(seq, ",fault_injection=%u",
F2FS_OPTION(sbi).fault_info.inject_rate);
+ seq_printf(seq, ",fault_type=%u",
+ F2FS_OPTION(sbi).fault_info.inject_type);
+ }
#endif
#ifdef CONFIG_QUOTA
if (test_opt(sbi, QUOTA))
@@ -1340,6 +1430,9 @@ static int f2fs_show_options(struct seq_file *seq, struct dentry *root)
else if (F2FS_OPTION(sbi).alloc_mode == ALLOC_MODE_REUSE)
seq_printf(seq, ",alloc_mode=%s", "reuse");
+ if (test_opt(sbi, DISABLE_CHECKPOINT))
+ seq_puts(seq, ",checkpoint=disable");
+
if (F2FS_OPTION(sbi).fsync_mode == FSYNC_MODE_POSIX)
seq_printf(seq, ",fsync_mode=%s", "posix");
else if (F2FS_OPTION(sbi).fsync_mode == FSYNC_MODE_STRICT)
@@ -1358,7 +1451,8 @@ static void default_options(struct f2fs_sb_info *sbi)
F2FS_OPTION(sbi).alloc_mode = ALLOC_MODE_DEFAULT;
F2FS_OPTION(sbi).fsync_mode = FSYNC_MODE_POSIX;
F2FS_OPTION(sbi).test_dummy_encryption = false;
- sbi->readdir_ra = 1;
+ F2FS_OPTION(sbi).s_resuid = make_kuid(&init_user_ns, F2FS_DEF_RESUID);
+ F2FS_OPTION(sbi).s_resgid = make_kgid(&init_user_ns, F2FS_DEF_RESGID);
set_opt(sbi, BG_GC);
set_opt(sbi, INLINE_XATTR);
@@ -1367,13 +1461,13 @@ static void default_options(struct f2fs_sb_info *sbi)
set_opt(sbi, EXTENT_CACHE);
set_opt(sbi, NOHEAP);
sbi->sb->s_flags |= MS_LAZYTIME;
+ clear_opt(sbi, DISABLE_CHECKPOINT);
set_opt(sbi, FLUSH_MERGE);
- if (f2fs_sb_has_blkzoned(sbi->sb)) {
+ set_opt(sbi, DISCARD);
+ if (f2fs_sb_has_blkzoned(sbi))
set_opt_mode(sbi, F2FS_MOUNT_LFS);
- set_opt(sbi, DISCARD);
- } else {
+ else
set_opt_mode(sbi, F2FS_MOUNT_ADAPTIVE);
- }
#ifdef CONFIG_F2FS_FS_XATTR
set_opt(sbi, XATTR_USER);
@@ -1382,14 +1476,60 @@ static void default_options(struct f2fs_sb_info *sbi)
set_opt(sbi, POSIX_ACL);
#endif
-#ifdef CONFIG_F2FS_FAULT_INJECTION
- f2fs_build_fault_attr(sbi, 0);
-#endif
+ f2fs_build_fault_attr(sbi, 0, 0);
}
#ifdef CONFIG_QUOTA
static int f2fs_enable_quotas(struct super_block *sb);
#endif
+
+static int f2fs_disable_checkpoint(struct f2fs_sb_info *sbi)
+{
+ struct cp_control cpc;
+ int err;
+
+ sbi->sb->s_flags |= MS_ACTIVE;
+
+ f2fs_update_time(sbi, DISABLE_TIME);
+
+ while (!f2fs_time_over(sbi, DISABLE_TIME)) {
+ mutex_lock(&sbi->gc_mutex);
+ err = f2fs_gc(sbi, true, false, NULL_SEGNO);
+ if (err == -ENODATA)
+ break;
+ if (err && err != -EAGAIN)
+ return err;
+ }
+
+ err = sync_filesystem(sbi->sb);
+ if (err)
+ return err;
+
+ if (f2fs_disable_cp_again(sbi))
+ return -EAGAIN;
+
+ mutex_lock(&sbi->gc_mutex);
+ cpc.reason = CP_PAUSE;
+ set_sbi_flag(sbi, SBI_CP_DISABLED);
+ f2fs_write_checkpoint(sbi, &cpc);
+
+ sbi->unusable_block_count = 0;
+ mutex_unlock(&sbi->gc_mutex);
+ return 0;
+}
+
+static void f2fs_enable_checkpoint(struct f2fs_sb_info *sbi)
+{
+ mutex_lock(&sbi->gc_mutex);
+ f2fs_dirty_to_prefree(sbi);
+
+ clear_sbi_flag(sbi, SBI_CP_DISABLED);
+ set_sbi_flag(sbi, SBI_IS_DIRTY);
+ mutex_unlock(&sbi->gc_mutex);
+
+ f2fs_sync_fs(sbi->sb, 1);
+}
+
static int f2fs_remount(struct super_block *sb, int *flags, char *data)
{
struct f2fs_sb_info *sbi = F2FS_SB(sb);
@@ -1399,6 +1539,8 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
bool need_restart_gc = false;
bool need_stop_gc = false;
bool no_extent_cache = !test_opt(sbi, EXTENT_CACHE);
+ bool disable_checkpoint = test_opt(sbi, DISABLE_CHECKPOINT);
+ bool checkpoint_changed;
#ifdef CONFIG_QUOTA
int i, j;
#endif
@@ -1419,7 +1561,7 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
GFP_KERNEL);
if (!org_mount_opt.s_qf_names[i]) {
for (j = 0; j < i; j++)
- kfree(org_mount_opt.s_qf_names[j]);
+ kvfree(org_mount_opt.s_qf_names[j]);
return -ENOMEM;
}
} else {
@@ -1443,6 +1585,8 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
err = parse_options(sb, data);
if (err)
goto restore_opts;
+ checkpoint_changed =
+ disable_checkpoint != test_opt(sbi, DISABLE_CHECKPOINT);
/*
* Previous and new state of filesystem is RO,
@@ -1461,7 +1605,7 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
sb->s_flags &= ~MS_RDONLY;
if (sb_any_quota_suspended(sb)) {
dquot_resume(sb, -1);
- } else if (f2fs_sb_has_quota_ino(sb)) {
+ } else if (f2fs_sb_has_quota_ino(sbi)) {
err = f2fs_enable_quotas(sb);
if (err)
goto restore_opts;
@@ -1476,6 +1620,13 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
goto restore_opts;
}
+ if ((*flags & MS_RDONLY) && test_opt(sbi, DISABLE_CHECKPOINT)) {
+ err = -EINVAL;
+ f2fs_msg(sbi->sb, KERN_WARNING,
+ "disabling checkpoint not compatible with read-only");
+ goto restore_opts;
+ }
+
/*
* We stop the GC thread if FS is mounted as RO
* or if background_gc = off is passed in mount
@@ -1483,11 +1634,11 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
*/
if ((*flags & MS_RDONLY) || !test_opt(sbi, BG_GC)) {
if (sbi->gc_thread) {
- stop_gc_thread(sbi);
+ f2fs_stop_gc_thread(sbi);
need_restart_gc = true;
}
} else if (!sbi->gc_thread) {
- err = start_gc_thread(sbi);
+ err = f2fs_start_gc_thread(sbi);
if (err)
goto restore_opts;
need_stop_gc = true;
@@ -1504,15 +1655,25 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
clear_sbi_flag(sbi, SBI_IS_CLOSE);
}
+ if (checkpoint_changed) {
+ if (test_opt(sbi, DISABLE_CHECKPOINT)) {
+ err = f2fs_disable_checkpoint(sbi);
+ if (err)
+ goto restore_gc;
+ } else {
+ f2fs_enable_checkpoint(sbi);
+ }
+ }
+
/*
* We stop issue flush thread if FS is mounted as RO
* or if flush_merge is not passed in mount option.
*/
if ((*flags & MS_RDONLY) || !test_opt(sbi, FLUSH_MERGE)) {
clear_opt(sbi, FLUSH_MERGE);
- destroy_flush_cmd_control(sbi, false);
+ f2fs_destroy_flush_cmd_control(sbi, false);
} else {
- err = create_flush_cmd_control(sbi);
+ err = f2fs_create_flush_cmd_control(sbi);
if (err)
goto restore_gc;
}
@@ -1520,27 +1681,28 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
#ifdef CONFIG_QUOTA
/* Release old quota file names */
for (i = 0; i < MAXQUOTAS; i++)
- kfree(org_mount_opt.s_qf_names[i]);
+ kvfree(org_mount_opt.s_qf_names[i]);
#endif
/* Update the POSIXACL Flag */
sb->s_flags = (sb->s_flags & ~MS_POSIXACL) |
(test_opt(sbi, POSIX_ACL) ? MS_POSIXACL : 0);
limit_reserve_root(sbi);
+ *flags = (*flags & ~MS_LAZYTIME) | (sb->s_flags & MS_LAZYTIME);
return 0;
restore_gc:
if (need_restart_gc) {
- if (start_gc_thread(sbi))
+ if (f2fs_start_gc_thread(sbi))
f2fs_msg(sbi->sb, KERN_WARNING,
"background gc thread has stopped");
} else if (need_stop_gc) {
- stop_gc_thread(sbi);
+ f2fs_stop_gc_thread(sbi);
}
restore_opts:
#ifdef CONFIG_QUOTA
F2FS_OPTION(sbi).s_jquota_fmt = org_mount_opt.s_jquota_fmt;
for (i = 0; i < MAXQUOTAS; i++) {
- kfree(F2FS_OPTION(sbi).s_qf_names[i]);
+ kvfree(F2FS_OPTION(sbi).s_qf_names[i]);
F2FS_OPTION(sbi).s_qf_names[i] = org_mount_opt.s_qf_names[i];
}
#endif
@@ -1579,6 +1741,7 @@ static ssize_t f2fs_quota_read(struct super_block *sb, int type, char *data,
congestion_wait(BLK_RW_ASYNC, HZ/50);
goto repeat;
}
+ set_sbi_flag(F2FS_SB(sb), SBI_QUOTA_NEED_REPAIR);
return PTR_ERR(page);
}
@@ -1590,6 +1753,7 @@ static ssize_t f2fs_quota_read(struct super_block *sb, int type, char *data,
}
if (unlikely(!PageUptodate(page))) {
f2fs_put_page(page, 1);
+ set_sbi_flag(F2FS_SB(sb), SBI_QUOTA_NEED_REPAIR);
return -EIO;
}
@@ -1631,6 +1795,7 @@ static ssize_t f2fs_quota_write(struct super_block *sb, int type,
congestion_wait(BLK_RW_ASYNC, HZ/50);
goto retry;
}
+ set_sbi_flag(F2FS_SB(sb), SBI_QUOTA_NEED_REPAIR);
break;
}
@@ -1667,6 +1832,12 @@ static qsize_t *f2fs_get_reserved_space(struct inode *inode)
static int f2fs_quota_on_mount(struct f2fs_sb_info *sbi, int type)
{
+ if (is_set_ckpt_flags(sbi, CP_QUOTA_NEED_FSCK_FLAG)) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "quota sysfile may be corrupted, skip loading it");
+ return 0;
+ }
+
return dquot_quota_on_mount(sbi->sb, F2FS_OPTION(sbi).s_qf_names[type],
F2FS_OPTION(sbi).s_jquota_fmt, type);
}
@@ -1676,7 +1847,7 @@ int f2fs_enable_quota_files(struct f2fs_sb_info *sbi, bool rdonly)
int enabled = 0;
int i, err;
- if (f2fs_sb_has_quota_ino(sbi->sb) && rdonly) {
+ if (f2fs_sb_has_quota_ino(sbi) && rdonly) {
err = f2fs_enable_quotas(sbi->sb);
if (err) {
f2fs_msg(sbi->sb, KERN_ERR,
@@ -1707,7 +1878,7 @@ static int f2fs_quota_enable(struct super_block *sb, int type, int format_id,
unsigned long qf_inum;
int err;
- BUG_ON(!f2fs_sb_has_quota_ino(sb));
+ BUG_ON(!f2fs_sb_has_quota_ino(F2FS_SB(sb)));
qf_inum = f2fs_qf_ino(sb, type);
if (!qf_inum)
@@ -1737,7 +1908,14 @@ static int f2fs_enable_quotas(struct super_block *sb)
test_opt(F2FS_SB(sb), PRJQUOTA),
};
+ if (is_set_ckpt_flags(F2FS_SB(sb), CP_QUOTA_NEED_FSCK_FLAG)) {
+ f2fs_msg(sb, KERN_ERR,
+ "quota file may be corrupted, skip loading it");
+ return 0;
+ }
+
sb_dqopt(sb)->flags |= DQUOT_QUOTA_SYS_FILE;
+
for (type = 0; type < MAXQUOTAS; type++) {
qf_inum = f2fs_qf_ino(sb, type);
if (qf_inum) {
@@ -1751,6 +1929,8 @@ static int f2fs_enable_quotas(struct super_block *sb)
"fsck to fix.", type, err);
for (type--; type >= 0; type--)
dquot_quota_off(sb, type);
+ set_sbi_flag(F2FS_SB(sb),
+ SBI_QUOTA_NEED_REPAIR);
return err;
}
}
@@ -1758,35 +1938,51 @@ static int f2fs_enable_quotas(struct super_block *sb)
return 0;
}
-static int f2fs_quota_sync(struct super_block *sb, int type)
+int f2fs_quota_sync(struct super_block *sb, int type)
{
+ struct f2fs_sb_info *sbi = F2FS_SB(sb);
struct quota_info *dqopt = sb_dqopt(sb);
int cnt;
int ret;
ret = dquot_writeback_dquots(sb, type);
if (ret)
- return ret;
+ goto out;
/*
* Now when everything is written we can discard the pagecache so
* that userspace sees the changes.
*/
for (cnt = 0; cnt < MAXQUOTAS; cnt++) {
+ struct address_space *mapping;
+
if (type != -1 && cnt != type)
continue;
if (!sb_has_quota_active(sb, cnt))
continue;
- ret = filemap_write_and_wait(dqopt->files[cnt]->i_mapping);
+ mapping = dqopt->files[cnt]->i_mapping;
+
+ ret = filemap_fdatawrite(mapping);
if (ret)
- return ret;
+ goto out;
+
+ /* if we are using journalled quota */
+ if (is_journalled_quota(sbi))
+ continue;
+
+ ret = filemap_fdatawait(mapping);
+ if (ret)
+ set_sbi_flag(F2FS_SB(sb), SBI_QUOTA_NEED_REPAIR);
inode_lock(dqopt->files[cnt]);
truncate_inode_pages(&dqopt->files[cnt]->i_data, 0);
inode_unlock(dqopt->files[cnt]);
}
- return 0;
+out:
+ if (ret)
+ set_sbi_flag(F2FS_SB(sb), SBI_QUOTA_NEED_REPAIR);
+ return ret;
}
static int f2fs_quota_on(struct super_block *sb, int type, int format_id,
@@ -1806,9 +2002,8 @@ static int f2fs_quota_on(struct super_block *sb, int type, int format_id,
inode = d_inode(path->dentry);
inode_lock(inode);
- F2FS_I(inode)->i_flags |= FS_NOATIME_FL | FS_IMMUTABLE_FL;
- inode_set_flags(inode, S_NOATIME | S_IMMUTABLE,
- S_NOATIME | S_IMMUTABLE);
+ F2FS_I(inode)->i_flags |= F2FS_NOATIME_FL | F2FS_IMMUTABLE_FL;
+ f2fs_set_inode_flags(inode);
inode_unlock(inode);
f2fs_mark_inode_dirty_sync(inode, false);
@@ -1823,15 +2018,17 @@ static int f2fs_quota_off(struct super_block *sb, int type)
if (!inode || !igrab(inode))
return dquot_quota_off(sb, type);
- f2fs_quota_sync(sb, type);
+ err = f2fs_quota_sync(sb, type);
+ if (err)
+ goto out_put;
err = dquot_quota_off(sb, type);
- if (err || f2fs_sb_has_quota_ino(sb))
+ if (err || f2fs_sb_has_quota_ino(F2FS_SB(sb)))
goto out_put;
inode_lock(inode);
- F2FS_I(inode)->i_flags &= ~(FS_NOATIME_FL | FS_IMMUTABLE_FL);
- inode_set_flags(inode, 0, S_NOATIME | S_IMMUTABLE);
+ F2FS_I(inode)->i_flags &= ~(F2FS_NOATIME_FL | F2FS_IMMUTABLE_FL);
+ f2fs_set_inode_flags(inode);
inode_unlock(inode);
f2fs_mark_inode_dirty_sync(inode, false);
out_put:
@@ -1842,9 +2039,88 @@ static int f2fs_quota_off(struct super_block *sb, int type)
void f2fs_quota_off_umount(struct super_block *sb)
{
int type;
+ int err;
- for (type = 0; type < MAXQUOTAS; type++)
- f2fs_quota_off(sb, type);
+ for (type = 0; type < MAXQUOTAS; type++) {
+ err = f2fs_quota_off(sb, type);
+ if (err) {
+ int ret = dquot_quota_off(sb, type);
+
+ f2fs_msg(sb, KERN_ERR,
+ "Fail to turn off disk quota "
+ "(type: %d, err: %d, ret:%d), Please "
+ "run fsck to fix it.", type, err, ret);
+ set_sbi_flag(F2FS_SB(sb), SBI_QUOTA_NEED_REPAIR);
+ }
+ }
+}
+
+static void f2fs_truncate_quota_inode_pages(struct super_block *sb)
+{
+ struct quota_info *dqopt = sb_dqopt(sb);
+ int type;
+
+ for (type = 0; type < MAXQUOTAS; type++) {
+ if (!dqopt->files[type])
+ continue;
+ f2fs_inode_synced(dqopt->files[type]);
+ }
+}
+
+static int f2fs_dquot_commit(struct dquot *dquot)
+{
+ int ret;
+
+ ret = dquot_commit(dquot);
+ if (ret < 0)
+ set_sbi_flag(F2FS_SB(dquot->dq_sb), SBI_QUOTA_NEED_REPAIR);
+ return ret;
+}
+
+static int f2fs_dquot_acquire(struct dquot *dquot)
+{
+ int ret;
+
+ ret = dquot_acquire(dquot);
+ if (ret < 0)
+ set_sbi_flag(F2FS_SB(dquot->dq_sb), SBI_QUOTA_NEED_REPAIR);
+
+ return ret;
+}
+
+static int f2fs_dquot_release(struct dquot *dquot)
+{
+ int ret;
+
+ ret = dquot_release(dquot);
+ if (ret < 0)
+ set_sbi_flag(F2FS_SB(dquot->dq_sb), SBI_QUOTA_NEED_REPAIR);
+ return ret;
+}
+
+static int f2fs_dquot_mark_dquot_dirty(struct dquot *dquot)
+{
+ struct super_block *sb = dquot->dq_sb;
+ struct f2fs_sb_info *sbi = F2FS_SB(sb);
+ int ret;
+
+ ret = dquot_mark_dquot_dirty(dquot);
+
+ /* if we are using journalled quota */
+ if (is_journalled_quota(sbi))
+ set_sbi_flag(sbi, SBI_QUOTA_NEED_FLUSH);
+
+ return ret;
+}
+
+static int f2fs_dquot_commit_info(struct super_block *sb, int type)
+{
+ int ret;
+
+ ret = dquot_commit_info(sb, type);
+ if (ret < 0)
+ set_sbi_flag(F2FS_SB(sb), SBI_QUOTA_NEED_REPAIR);
+ return ret;
}
static int f2fs_get_projid(struct inode *inode, kprojid_t *projid)
@@ -1855,11 +2131,11 @@ static int f2fs_get_projid(struct inode *inode, kprojid_t *projid)
static const struct dquot_operations f2fs_quota_operations = {
.get_reserved_space = f2fs_get_reserved_space,
- .write_dquot = dquot_commit,
- .acquire_dquot = dquot_acquire,
- .release_dquot = dquot_release,
- .mark_dirty = dquot_mark_dquot_dirty,
- .write_info = dquot_commit_info,
+ .write_dquot = f2fs_dquot_commit,
+ .acquire_dquot = f2fs_dquot_acquire,
+ .release_dquot = f2fs_dquot_release,
+ .mark_dirty = f2fs_dquot_mark_dquot_dirty,
+ .write_info = f2fs_dquot_commit_info,
.alloc_dquot = dquot_alloc,
.destroy_dquot = dquot_destroy,
.get_projid = f2fs_get_projid,
@@ -1877,6 +2153,11 @@ static const struct quotactl_ops f2fs_quotactl_ops = {
.get_nextdqblk = dquot_get_next_dqblk,
};
#else
+int f2fs_quota_sync(struct super_block *sb, int type)
+{
+ return 0;
+}
+
void f2fs_quota_off_umount(struct super_block *sb)
{
}
@@ -1896,6 +2177,7 @@ static const struct super_operations f2fs_sops = {
#endif
.evict_inode = f2fs_evict_inode,
.put_super = f2fs_put_super,
+ .umount_end = f2fs_umount_end,
.sync_fs = f2fs_sync_fs,
.freeze_fs = f2fs_freeze,
.unfreeze_fs = f2fs_unfreeze,
@@ -1922,7 +2204,7 @@ static int f2fs_set_context(struct inode *inode, const void *ctx, size_t len,
* if LOST_FOUND feature is enabled.
*
*/
- if (f2fs_sb_has_lost_found(sbi->sb) &&
+ if (f2fs_sb_has_lost_found(sbi) &&
inode->i_ino == F2FS_ROOT_INO(sbi))
return -EPERM;
@@ -1936,29 +2218,57 @@ static bool f2fs_dummy_context(struct inode *inode)
return DUMMY_ENCRYPTION_ENABLED(F2FS_I_SB(inode));
}
-static unsigned f2fs_max_namelen(struct inode *inode)
-{
- return S_ISLNK(inode->i_mode) ?
- inode->i_sb->s_blocksize : F2FS_NAME_LEN;
-}
-
static const struct fscrypt_operations f2fs_cryptops = {
.key_prefix = "f2fs:",
.get_context = f2fs_get_context,
.set_context = f2fs_set_context,
.dummy_context = f2fs_dummy_context,
.empty_dir = f2fs_empty_dir,
- .max_namelen = f2fs_max_namelen,
+ .max_namelen = F2FS_NAME_LEN,
};
#endif
+#ifdef CONFIG_F2FS_FS_VERITY
+static bool f2fs_is_verity(struct inode *inode)
+{
+ return f2fs_verity_file(inode);
+}
+
+static int f2fs_set_verity(struct inode *inode)
+{
+ int err;
+
+ err = f2fs_convert_inline_inode(inode);
+ if (err)
+ return err;
+
+ file_set_verity(inode);
+ f2fs_mark_inode_dirty_sync(inode, true);
+ return 0;
+}
+
+static struct page *f2fs_read_metadata_page(struct inode *inode, pgoff_t index)
+{
+ if (WARN_ON(f2fs_has_inline_data(inode)))
+ return ERR_PTR(-EINVAL);
+
+ return f2fs_find_data_page(inode, index, F2FS_GETPAGE_SKIP_VERITY);
+}
+
+static const struct fsverity_operations f2fs_verityops = {
+ .is_verity = f2fs_is_verity,
+ .set_verity = f2fs_set_verity,
+ .read_metadata_page = f2fs_read_metadata_page,
+};
+#endif /* CONFIG_F2FS_FS_VERITY */
+
static struct inode *f2fs_nfs_get_inode(struct super_block *sb,
u64 ino, u32 generation)
{
struct f2fs_sb_info *sbi = F2FS_SB(sb);
struct inode *inode;
- if (check_nid_range(sbi, ino))
+ if (f2fs_check_nid_range(sbi, ino))
return ERR_PTR(-ESTALE);
/*
@@ -2141,10 +2451,32 @@ static inline bool sanity_check_area_boundary(struct f2fs_sb_info *sbi,
static int sanity_check_raw_super(struct f2fs_sb_info *sbi,
struct buffer_head *bh)
{
+ block_t segment_count, segs_per_sec, secs_per_zone;
+ block_t total_sections, blocks_per_seg;
struct f2fs_super_block *raw_super = (struct f2fs_super_block *)
(bh->b_data + F2FS_SUPER_OFFSET);
struct super_block *sb = sbi->sb;
unsigned int blocksize;
+ size_t crc_offset = 0;
+ __u32 crc = 0;
+
+ /* Check checksum_offset and crc in superblock */
+ if (__F2FS_HAS_FEATURE(raw_super, F2FS_FEATURE_SB_CHKSUM)) {
+ crc_offset = le32_to_cpu(raw_super->checksum_offset);
+ if (crc_offset !=
+ offsetof(struct f2fs_super_block, crc)) {
+ f2fs_msg(sb, KERN_INFO,
+ "Invalid SB checksum offset: %zu",
+ crc_offset);
+ return 1;
+ }
+ crc = le32_to_cpu(raw_super->crc);
+ if (!f2fs_crc_valid(sbi, crc, raw_super, crc_offset)) {
+ f2fs_msg(sb, KERN_INFO,
+ "Invalid SB checksum value: %u", crc);
+ return 1;
+ }
+ }
if (F2FS_SUPER_MAGIC != le32_to_cpu(raw_super->magic)) {
f2fs_msg(sb, KERN_INFO,
@@ -2197,6 +2529,72 @@ static int sanity_check_raw_super(struct f2fs_sb_info *sbi,
return 1;
}
+ segment_count = le32_to_cpu(raw_super->segment_count);
+ segs_per_sec = le32_to_cpu(raw_super->segs_per_sec);
+ secs_per_zone = le32_to_cpu(raw_super->secs_per_zone);
+ total_sections = le32_to_cpu(raw_super->section_count);
+
+ /* blocks_per_seg should be 512, given the above check */
+ blocks_per_seg = 1 << le32_to_cpu(raw_super->log_blocks_per_seg);
+
+ if (segment_count > F2FS_MAX_SEGMENT ||
+ segment_count < F2FS_MIN_SEGMENTS) {
+ f2fs_msg(sb, KERN_INFO,
+ "Invalid segment count (%u)",
+ segment_count);
+ return 1;
+ }
+
+ if (total_sections > segment_count ||
+ total_sections < F2FS_MIN_SEGMENTS ||
+ segs_per_sec > segment_count || !segs_per_sec) {
+ f2fs_msg(sb, KERN_INFO,
+ "Invalid segment/section count (%u, %u x %u)",
+ segment_count, total_sections, segs_per_sec);
+ return 1;
+ }
+
+ if ((segment_count / segs_per_sec) < total_sections) {
+ f2fs_msg(sb, KERN_INFO,
+ "Small segment_count (%u < %u * %u)",
+ segment_count, segs_per_sec, total_sections);
+ return 1;
+ }
+
+ if (segment_count > (le64_to_cpu(raw_super->block_count) >> 9)) {
+ f2fs_msg(sb, KERN_INFO,
+ "Wrong segment_count / block_count (%u > %llu)",
+ segment_count, le64_to_cpu(raw_super->block_count));
+ return 1;
+ }
+
+ if (secs_per_zone > total_sections || !secs_per_zone) {
+ f2fs_msg(sb, KERN_INFO,
+ "Wrong secs_per_zone / total_sections (%u, %u)",
+ secs_per_zone, total_sections);
+ return 1;
+ }
+ if (le32_to_cpu(raw_super->extension_count) > F2FS_MAX_EXTENSION ||
+ raw_super->hot_ext_count > F2FS_MAX_EXTENSION ||
+ (le32_to_cpu(raw_super->extension_count) +
+ raw_super->hot_ext_count) > F2FS_MAX_EXTENSION) {
+ f2fs_msg(sb, KERN_INFO,
+ "Corrupted extension count (%u + %u > %u)",
+ le32_to_cpu(raw_super->extension_count),
+ raw_super->hot_ext_count,
+ F2FS_MAX_EXTENSION);
+ return 1;
+ }
+
+ if (le32_to_cpu(raw_super->cp_payload) >
+ (blocks_per_seg - F2FS_CP_PACKS)) {
+ f2fs_msg(sb, KERN_INFO,
+ "Insane cp_payload (%u > %u)",
+ le32_to_cpu(raw_super->cp_payload),
+ blocks_per_seg - F2FS_CP_PACKS);
+ return 1;
+ }
+
/* check reserved ino info */
if (le32_to_cpu(raw_super->node_ino) != 1 ||
le32_to_cpu(raw_super->meta_ino) != 2 ||
@@ -2209,13 +2607,6 @@ static int sanity_check_raw_super(struct f2fs_sb_info *sbi,
return 1;
}
- if (le32_to_cpu(raw_super->segment_count) > F2FS_MAX_SEGMENT) {
- f2fs_msg(sb, KERN_INFO,
- "Invalid segment count (%u)",
- le32_to_cpu(raw_super->segment_count));
- return 1;
- }
-
/* check CP/SIT/NAT/SSA/MAIN_AREA area boundary */
if (sanity_check_area_boundary(sbi, bh))
return 1;
@@ -2223,19 +2614,27 @@ static int sanity_check_raw_super(struct f2fs_sb_info *sbi,
return 0;
}
-int sanity_check_ckpt(struct f2fs_sb_info *sbi)
+int f2fs_sanity_check_ckpt(struct f2fs_sb_info *sbi)
{
unsigned int total, fsmeta;
struct f2fs_super_block *raw_super = F2FS_RAW_SUPER(sbi);
struct f2fs_checkpoint *ckpt = F2FS_CKPT(sbi);
unsigned int ovp_segments, reserved_segments;
unsigned int main_segs, blocks_per_seg;
- int i;
+ unsigned int sit_segs, nat_segs;
+ unsigned int sit_bitmap_size, nat_bitmap_size;
+ unsigned int log_blocks_per_seg;
+ unsigned int segment_count_main;
+ unsigned int cp_pack_start_sum, cp_payload;
+ block_t user_block_count;
+ int i, j;
total = le32_to_cpu(raw_super->segment_count);
fsmeta = le32_to_cpu(raw_super->segment_count_ckpt);
- fsmeta += le32_to_cpu(raw_super->segment_count_sit);
- fsmeta += le32_to_cpu(raw_super->segment_count_nat);
+ sit_segs = le32_to_cpu(raw_super->segment_count_sit);
+ fsmeta += sit_segs;
+ nat_segs = le32_to_cpu(raw_super->segment_count_nat);
+ fsmeta += nat_segs;
fsmeta += le32_to_cpu(ckpt->rsvd_segment_count);
fsmeta += le32_to_cpu(raw_super->segment_count_ssa);
@@ -2252,6 +2651,16 @@ int sanity_check_ckpt(struct f2fs_sb_info *sbi)
return 1;
}
+ user_block_count = le64_to_cpu(ckpt->user_block_count);
+ segment_count_main = le32_to_cpu(raw_super->segment_count_main);
+ log_blocks_per_seg = le32_to_cpu(raw_super->log_blocks_per_seg);
+ if (!user_block_count || user_block_count >=
+ segment_count_main << log_blocks_per_seg) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "Wrong user_block_count: %u", user_block_count);
+ return 1;
+ }
+
main_segs = le32_to_cpu(raw_super->segment_count_main);
blocks_per_seg = sbi->blocks_per_seg;
@@ -2259,11 +2668,65 @@ int sanity_check_ckpt(struct f2fs_sb_info *sbi)
if (le32_to_cpu(ckpt->cur_node_segno[i]) >= main_segs ||
le16_to_cpu(ckpt->cur_node_blkoff[i]) >= blocks_per_seg)
return 1;
+ for (j = i + 1; j < NR_CURSEG_NODE_TYPE; j++) {
+ if (le32_to_cpu(ckpt->cur_node_segno[i]) ==
+ le32_to_cpu(ckpt->cur_node_segno[j])) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "Node segment (%u, %u) has the same "
+ "segno: %u", i, j,
+ le32_to_cpu(ckpt->cur_node_segno[i]));
+ return 1;
+ }
+ }
}
for (i = 0; i < NR_CURSEG_DATA_TYPE; i++) {
if (le32_to_cpu(ckpt->cur_data_segno[i]) >= main_segs ||
le16_to_cpu(ckpt->cur_data_blkoff[i]) >= blocks_per_seg)
return 1;
+ for (j = i + 1; j < NR_CURSEG_DATA_TYPE; j++) {
+ if (le32_to_cpu(ckpt->cur_data_segno[i]) ==
+ le32_to_cpu(ckpt->cur_data_segno[j])) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "Data segment (%u, %u) has the same "
+ "segno: %u", i, j,
+ le32_to_cpu(ckpt->cur_data_segno[i]));
+ return 1;
+ }
+ }
+ }
+ for (i = 0; i < NR_CURSEG_NODE_TYPE; i++) {
+ for (j = i; j < NR_CURSEG_DATA_TYPE; j++) {
+ if (le32_to_cpu(ckpt->cur_node_segno[i]) ==
+ le32_to_cpu(ckpt->cur_data_segno[j])) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "Data segment (%u) and Data segment (%u)"
+ " has the same segno: %u", i, j,
+ le32_to_cpu(ckpt->cur_node_segno[i]));
+ return 1;
+ }
+ }
+ }
+
+ sit_bitmap_size = le32_to_cpu(ckpt->sit_ver_bitmap_bytesize);
+ nat_bitmap_size = le32_to_cpu(ckpt->nat_ver_bitmap_bytesize);
+
+ if (sit_bitmap_size != ((sit_segs / 2) << log_blocks_per_seg) / 8 ||
+ nat_bitmap_size != ((nat_segs / 2) << log_blocks_per_seg) / 8) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "Wrong bitmap size: sit: %u, nat:%u",
+ sit_bitmap_size, nat_bitmap_size);
+ return 1;
+ }
+
+ cp_pack_start_sum = __start_sum_addr(sbi);
+ cp_payload = __cp_payload(sbi);
+ if (cp_pack_start_sum < cp_payload + 1 ||
+ cp_pack_start_sum > blocks_per_seg - 1 -
+ NR_CURSEG_TYPE) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "Wrong cp_pack_start_sum: %u",
+ cp_pack_start_sum);
+ return 1;
}
if (unlikely(f2fs_cp_error(sbi))) {
@@ -2276,7 +2739,7 @@ int sanity_check_ckpt(struct f2fs_sb_info *sbi)
static void init_sb_info(struct f2fs_sb_info *sbi)
{
struct f2fs_super_block *raw_super = sbi->raw_super;
- int i, j;
+ int i;
sbi->log_sectors_per_block =
le32_to_cpu(raw_super->log_sectors_per_block);
@@ -2294,23 +2757,28 @@ static void init_sb_info(struct f2fs_sb_info *sbi)
sbi->node_ino_num = le32_to_cpu(raw_super->node_ino);
sbi->meta_ino_num = le32_to_cpu(raw_super->meta_ino);
sbi->cur_victim_sec = NULL_SECNO;
+ sbi->next_victim_seg[BG_GC] = NULL_SEGNO;
+ sbi->next_victim_seg[FG_GC] = NULL_SEGNO;
sbi->max_victim_search = DEF_MAX_VICTIM_SEARCH;
+ sbi->migration_granularity = sbi->segs_per_sec;
sbi->dir_level = DEF_DIR_LEVEL;
sbi->interval_time[CP_TIME] = DEF_CP_INTERVAL;
sbi->interval_time[REQ_TIME] = DEF_IDLE_INTERVAL;
+ sbi->interval_time[DISCARD_TIME] = DEF_IDLE_INTERVAL;
+ sbi->interval_time[GC_TIME] = DEF_IDLE_INTERVAL;
+ sbi->interval_time[DISABLE_TIME] = DEF_DISABLE_INTERVAL;
clear_sbi_flag(sbi, SBI_NEED_FSCK);
for (i = 0; i < NR_COUNT_TYPE; i++)
atomic_set(&sbi->nr_pages[i], 0);
- atomic_set(&sbi->wb_sync_req, 0);
+ for (i = 0; i < META; i++)
+ atomic_set(&sbi->wb_sync_req[i], 0);
INIT_LIST_HEAD(&sbi->s_list);
mutex_init(&sbi->umount_mutex);
- for (i = 0; i < NR_PAGE_TYPE - 1; i++)
- for (j = HOT; j < NR_TEMP_TYPE; j++)
- mutex_init(&sbi->wio_mutex[i][j]);
+ init_rwsem(&sbi->io_order_lock);
spin_lock_init(&sbi->cp_lock);
sbi->dirty_device = 0;
@@ -2327,8 +2795,12 @@ static int init_percpu_info(struct f2fs_sb_info *sbi)
if (err)
return err;
- return percpu_counter_init(&sbi->total_valid_inode_count, 0,
+ err = percpu_counter_init(&sbi->total_valid_inode_count, 0,
GFP_KERNEL);
+ if (err)
+ percpu_counter_destroy(&sbi->alloc_valid_block_count);
+
+ return err;
}
#ifdef CONFIG_BLK_DEV_ZONED
@@ -2342,7 +2814,7 @@ static int init_blkz_info(struct f2fs_sb_info *sbi, int devi)
unsigned int n = 0;
int err = -EIO;
- if (!f2fs_sb_has_blkzoned(sbi->sb))
+ if (!f2fs_sb_has_blkzoned(sbi))
return 0;
if (sbi->blocks_per_blkz && sbi->blocks_per_blkz !=
@@ -2365,8 +2837,10 @@ static int init_blkz_info(struct f2fs_sb_info *sbi, int devi)
#define F2FS_REPORT_NR_ZONES 4096
- zones = f2fs_kzalloc(sbi, sizeof(struct blk_zone) *
- F2FS_REPORT_NR_ZONES, GFP_KERNEL);
+ zones = f2fs_kzalloc(sbi,
+ array_size(F2FS_REPORT_NR_ZONES,
+ sizeof(struct blk_zone)),
+ GFP_KERNEL);
if (!zones)
return -ENOMEM;
@@ -2391,7 +2865,7 @@ static int init_blkz_info(struct f2fs_sb_info *sbi, int devi)
}
}
- kfree(zones);
+ kvfree(zones);
return err;
}
@@ -2451,7 +2925,7 @@ static int read_raw_super_block(struct f2fs_sb_info *sbi,
/* No valid superblock */
if (!*raw_super)
- kfree(super);
+ kvfree(super);
else
err = 0;
@@ -2461,6 +2935,7 @@ static int read_raw_super_block(struct f2fs_sb_info *sbi,
int f2fs_commit_super(struct f2fs_sb_info *sbi, bool recover)
{
struct buffer_head *bh;
+ __u32 crc = 0;
int err;
if ((recover && f2fs_readonly(sbi->sb)) ||
@@ -2469,6 +2944,13 @@ int f2fs_commit_super(struct f2fs_sb_info *sbi, bool recover)
return -EROFS;
}
+ /* we should update superblock crc here */
+ if (!recover && f2fs_sb_has_sb_chksum(sbi)) {
+ crc = f2fs_crc32(sbi, F2FS_RAW_SUPER(sbi),
+ offsetof(struct f2fs_super_block, crc));
+ F2FS_RAW_SUPER(sbi)->crc = cpu_to_le32(crc);
+ }
+
/* write back-up superblock first */
bh = sb_bread(sbi->sb, sbi->valid_super_block ? 0 : 1);
if (!bh)
@@ -2510,8 +2992,10 @@ static int f2fs_scan_devices(struct f2fs_sb_info *sbi)
* Initialize multiple devices information, or single
* zoned block device information.
*/
- sbi->devs = f2fs_kzalloc(sbi, sizeof(struct f2fs_dev_info) *
- max_devices, GFP_KERNEL);
+ sbi->devs = f2fs_kzalloc(sbi,
+ array_size(max_devices,
+ sizeof(struct f2fs_dev_info)),
+ GFP_KERNEL);
if (!sbi->devs)
return -ENOMEM;
@@ -2553,7 +3037,7 @@ static int f2fs_scan_devices(struct f2fs_sb_info *sbi)
#ifdef CONFIG_BLK_DEV_ZONED
if (bdev_zoned_model(FDEV(i).bdev) == BLK_ZONED_HM &&
- !f2fs_sb_has_blkzoned(sbi->sb)) {
+ !f2fs_sb_has_blkzoned(sbi)) {
f2fs_msg(sbi->sb, KERN_ERR,
"Zoned block device feature not enabled\n");
return -EINVAL;
@@ -2597,6 +3081,8 @@ static void f2fs_tuning_parameters(struct f2fs_sb_info *sbi)
sm_i->dcc_info->discard_granularity = 1;
sm_i->ipu_policy = 1 << F2FS_IPU_FORCE;
}
+
+ sbi->readdir_ra = 1;
}
static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
@@ -2646,11 +3132,8 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
sb->s_fs_info = sbi;
sbi->raw_super = raw_super;
- F2FS_OPTION(sbi).s_resuid = make_kuid(&init_user_ns, F2FS_DEF_RESUID);
- F2FS_OPTION(sbi).s_resgid = make_kgid(&init_user_ns, F2FS_DEF_RESGID);
-
/* precompute checksum seed for metadata */
- if (f2fs_sb_has_inode_chksum(sb))
+ if (f2fs_sb_has_inode_chksum(sbi))
sbi->s_chksum_seed = f2fs_chksum(sbi, ~0, raw_super->uuid,
sizeof(raw_super->uuid));
@@ -2660,7 +3143,7 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
* devices, but mandatory for host-managed zoned block devices.
*/
#ifndef CONFIG_BLK_DEV_ZONED
- if (f2fs_sb_has_blkzoned(sb)) {
+ if (f2fs_sb_has_blkzoned(sbi)) {
f2fs_msg(sb, KERN_ERR,
"Zoned block device support is not enabled\n");
err = -EOPNOTSUPP;
@@ -2687,13 +3170,13 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
#ifdef CONFIG_QUOTA
sb->dq_op = &f2fs_quota_operations;
- if (f2fs_sb_has_quota_ino(sb))
+ if (f2fs_sb_has_quota_ino(sbi))
sb->s_qcop = &dquot_quotactl_sysfile_ops;
else
sb->s_qcop = &f2fs_quotactl_ops;
sb->s_quota_types = QTYPE_MASK_USR | QTYPE_MASK_GRP | QTYPE_MASK_PRJ;
- if (f2fs_sb_has_quota_ino(sbi->sb)) {
+ if (f2fs_sb_has_quota_ino(sbi)) {
for (i = 0; i < MAXQUOTAS; i++) {
if (f2fs_qf_ino(sbi->sb, i))
sbi->nquota_files++;
@@ -2705,6 +3188,9 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
#ifdef CONFIG_F2FS_FS_ENCRYPTION
sb->s_cop = &f2fs_cryptops;
#endif
+#ifdef CONFIG_F2FS_FS_VERITY
+ sb->s_vop = &f2fs_verityops;
+#endif
sb->s_xattr = f2fs_xattr_handlers;
sb->s_export_op = &f2fs_export_ops;
sb->s_magic = F2FS_SUPER_MAGIC;
@@ -2734,12 +3220,14 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
int n = (i == META) ? 1: NR_TEMP_TYPE;
int j;
- sbi->write_io[i] = f2fs_kmalloc(sbi,
- n * sizeof(struct f2fs_bio_info),
- GFP_KERNEL);
+ sbi->write_io[i] =
+ f2fs_kmalloc(sbi,
+ array_size(n,
+ sizeof(struct f2fs_bio_info)),
+ GFP_KERNEL);
if (!sbi->write_io[i]) {
err = -ENOMEM;
- goto free_options;
+ goto free_bio_info;
}
for (j = HOT; j < n; j++) {
@@ -2776,12 +3264,15 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
goto free_io_dummy;
}
- err = get_valid_checkpoint(sbi);
+ err = f2fs_get_valid_checkpoint(sbi);
if (err) {
f2fs_msg(sb, KERN_ERR, "Failed to get valid F2FS checkpoint");
goto free_meta_inode;
}
+ if (__is_set_ckpt_flags(F2FS_CKPT(sbi), CP_QUOTA_NEED_FSCK_FLAG))
+ set_sbi_flag(sbi, SBI_QUOTA_NEED_REPAIR);
+
/* Initialize device list */
err = f2fs_scan_devices(sbi);
if (err) {
@@ -2806,18 +3297,20 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
spin_lock_init(&sbi->inode_lock[i]);
}
- init_extent_cache_info(sbi);
+ f2fs_init_extent_cache_info(sbi);
- init_ino_entry_info(sbi);
+ f2fs_init_ino_entry_info(sbi);
+
+ f2fs_init_fsync_node_info(sbi);
/* setup f2fs internal modules */
- err = build_segment_manager(sbi);
+ err = f2fs_build_segment_manager(sbi);
if (err) {
f2fs_msg(sb, KERN_ERR,
"Failed to initialize F2FS segment manager");
goto free_sm;
}
- err = build_node_manager(sbi);
+ err = f2fs_build_node_manager(sbi);
if (err) {
f2fs_msg(sb, KERN_ERR,
"Failed to initialize F2FS node manager");
@@ -2835,28 +3328,29 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
sbi->kbytes_written =
le64_to_cpu(seg_i->journal->info.kbytes_written);
- build_gc_manager(sbi);
+ f2fs_build_gc_manager(sbi);
+
+ err = f2fs_build_stats(sbi);
+ if (err)
+ goto free_nm;
/* get an inode for node space */
sbi->node_inode = f2fs_iget(sb, F2FS_NODE_INO(sbi));
if (IS_ERR(sbi->node_inode)) {
f2fs_msg(sb, KERN_ERR, "Failed to read node inode");
err = PTR_ERR(sbi->node_inode);
- goto free_nm;
+ goto free_stats;
}
- err = f2fs_build_stats(sbi);
- if (err)
- goto free_node_inode;
-
/* read root inode and dentry */
root = f2fs_iget(sb, F2FS_ROOT_INO(sbi));
if (IS_ERR(root)) {
f2fs_msg(sb, KERN_ERR, "Failed to read root inode");
err = PTR_ERR(root);
- goto free_stats;
+ goto free_node_inode;
}
- if (!S_ISDIR(root->i_mode) || !root->i_blocks || !root->i_size) {
+ if (!S_ISDIR(root->i_mode) || !root->i_blocks ||
+ !root->i_size || !root->i_nlink) {
iput(root);
err = -EINVAL;
goto free_node_inode;
@@ -2873,24 +3367,22 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
goto free_root_inode;
#ifdef CONFIG_QUOTA
- /*
- * Turn on quotas which were not enabled for read-only mounts if
- * filesystem has quota feature, so that they are updated correctly.
- */
- if (f2fs_sb_has_quota_ino(sb) && !f2fs_readonly(sb)) {
+ /* Enable quota usage during mount */
+ if (f2fs_sb_has_quota_ino(sbi) && !f2fs_readonly(sb)) {
err = f2fs_enable_quotas(sb);
- if (err) {
+ if (err)
f2fs_msg(sb, KERN_ERR,
"Cannot turn on quotas: error %d", err);
- goto free_sysfs;
- }
}
#endif
/* if there are nt orphan nodes free them */
- err = recover_orphan_inodes(sbi);
+ err = f2fs_recover_orphan_inodes(sbi);
if (err)
goto free_meta;
+ if (unlikely(is_set_ckpt_flags(sbi, CP_DISABLED_FLAG)))
+ goto skip_recovery;
+
/* recover fsynced data */
if (!test_opt(sbi, DISABLE_ROLL_FORWARD)) {
/*
@@ -2909,7 +3401,7 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
if (!retry)
goto skip_recovery;
- err = recover_fsync_data(sbi, false);
+ err = f2fs_recover_fsync_data(sbi, false);
if (err < 0) {
need_fsck = true;
f2fs_msg(sb, KERN_ERR,
@@ -2917,7 +3409,7 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
goto free_meta;
}
} else {
- err = recover_fsync_data(sbi, true);
+ err = f2fs_recover_fsync_data(sbi, true);
if (!f2fs_readonly(sb) && err > 0) {
err = -EINVAL;
@@ -2927,20 +3419,28 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
}
}
skip_recovery:
- /* recover_fsync_data() cleared this already */
+ /* f2fs_recover_fsync_data() cleared this already */
clear_sbi_flag(sbi, SBI_POR_DOING);
+ if (test_opt(sbi, DISABLE_CHECKPOINT)) {
+ err = f2fs_disable_checkpoint(sbi);
+ if (err)
+ goto free_meta;
+ } else if (is_set_ckpt_flags(sbi, CP_DISABLED_FLAG)) {
+ f2fs_enable_checkpoint(sbi);
+ }
+
/*
* If filesystem is not mounted as read-only then
* do start the gc_thread.
*/
if (test_opt(sbi, BG_GC) && !f2fs_readonly(sb)) {
/* After POR, we can run background GC thread.*/
- err = start_gc_thread(sbi);
+ err = f2fs_start_gc_thread(sbi);
if (err)
goto free_meta;
}
- kfree(options);
+ kvfree(options);
/* recover broken superblock */
if (recovery) {
@@ -2962,59 +3462,58 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
free_meta:
#ifdef CONFIG_QUOTA
- if (f2fs_sb_has_quota_ino(sb) && !f2fs_readonly(sb))
+ f2fs_truncate_quota_inode_pages(sb);
+ if (f2fs_sb_has_quota_ino(sbi) && !f2fs_readonly(sb))
f2fs_quota_off_umount(sbi->sb);
#endif
- f2fs_sync_inode_meta(sbi);
/*
- * Some dirty meta pages can be produced by recover_orphan_inodes()
+ * Some dirty meta pages can be produced by f2fs_recover_orphan_inodes()
* failed by EIO. Then, iput(node_inode) can trigger balance_fs_bg()
- * followed by write_checkpoint() through f2fs_write_node_pages(), which
- * falls into an infinite loop in sync_meta_pages().
+ * followed by f2fs_write_checkpoint() through f2fs_write_node_pages(), which
+ * falls into an infinite loop in f2fs_sync_meta_pages().
*/
truncate_inode_pages_final(META_MAPPING(sbi));
-#ifdef CONFIG_QUOTA
-free_sysfs:
-#endif
f2fs_unregister_sysfs(sbi);
free_root_inode:
dput(sb->s_root);
sb->s_root = NULL;
-free_stats:
- f2fs_destroy_stats(sbi);
free_node_inode:
- release_ino_entry(sbi, true);
+ f2fs_release_ino_entry(sbi, true);
truncate_inode_pages_final(NODE_MAPPING(sbi));
iput(sbi->node_inode);
+ sbi->node_inode = NULL;
+free_stats:
+ f2fs_destroy_stats(sbi);
free_nm:
- destroy_node_manager(sbi);
+ f2fs_destroy_node_manager(sbi);
free_sm:
- destroy_segment_manager(sbi);
+ f2fs_destroy_segment_manager(sbi);
free_devices:
destroy_device_list(sbi);
- kfree(sbi->ckpt);
+ kvfree(sbi->ckpt);
free_meta_inode:
make_bad_inode(sbi->meta_inode);
iput(sbi->meta_inode);
+ sbi->meta_inode = NULL;
free_io_dummy:
mempool_destroy(sbi->write_io_dummy);
free_percpu:
destroy_percpu_info(sbi);
free_bio_info:
for (i = 0; i < NR_PAGE_TYPE; i++)
- kfree(sbi->write_io[i]);
+ kvfree(sbi->write_io[i]);
free_options:
#ifdef CONFIG_QUOTA
for (i = 0; i < MAXQUOTAS; i++)
- kfree(F2FS_OPTION(sbi).s_qf_names[i]);
+ kvfree(F2FS_OPTION(sbi).s_qf_names[i]);
#endif
- kfree(options);
+ kvfree(options);
free_sb_buf:
- kfree(raw_super);
+ kvfree(raw_super);
free_sbi:
if (sbi->s_chksum_driver)
crypto_free_shash(sbi->s_chksum_driver);
- kfree(sbi);
+ kvfree(sbi);
/* give only one another chance */
if (retry) {
@@ -3034,9 +3533,22 @@ static struct dentry *f2fs_mount(struct file_system_type *fs_type, int flags,
static void kill_f2fs_super(struct super_block *sb)
{
if (sb->s_root) {
- set_sbi_flag(F2FS_SB(sb), SBI_IS_CLOSE);
- stop_gc_thread(F2FS_SB(sb));
- stop_discard_thread(F2FS_SB(sb));
+ struct f2fs_sb_info *sbi = F2FS_SB(sb);
+
+ set_sbi_flag(sbi, SBI_IS_CLOSE);
+ f2fs_stop_gc_thread(sbi);
+ f2fs_stop_discard_thread(sbi);
+
+ if (is_sbi_flag_set(sbi, SBI_IS_DIRTY) ||
+ !is_set_ckpt_flags(sbi, CP_UMOUNT_FLAG)) {
+ struct cp_control cpc = {
+ .reason = CP_UMOUNT,
+ };
+ f2fs_write_checkpoint(sbi, &cpc);
+ }
+
+ if (is_sbi_flag_set(sbi, SBI_IS_RECOVERED) && f2fs_readonly(sb))
+ sb->s_flags &= ~MS_RDONLY;
}
kill_block_super(sb);
}
@@ -3074,21 +3586,27 @@ static int __init init_f2fs_fs(void)
{
int err;
+ if (PAGE_SIZE != F2FS_BLKSIZE) {
+ printk("F2FS not supported on PAGE_SIZE(%lu) != %d\n",
+ PAGE_SIZE, F2FS_BLKSIZE);
+ return -EINVAL;
+ }
+
f2fs_build_trace_ios();
err = init_inodecache();
if (err)
goto fail;
- err = create_node_manager_caches();
+ err = f2fs_create_node_manager_caches();
if (err)
goto free_inodecache;
- err = create_segment_manager_caches();
+ err = f2fs_create_segment_manager_caches();
if (err)
goto free_node_manager_caches;
- err = create_checkpoint_caches();
+ err = f2fs_create_checkpoint_caches();
if (err)
goto free_segment_manager_caches;
- err = create_extent_cache();
+ err = f2fs_create_extent_cache();
if (err)
goto free_checkpoint_caches;
err = f2fs_init_sysfs();
@@ -3117,13 +3635,13 @@ static int __init init_f2fs_fs(void)
free_sysfs:
f2fs_exit_sysfs();
free_extent_cache:
- destroy_extent_cache();
+ f2fs_destroy_extent_cache();
free_checkpoint_caches:
- destroy_checkpoint_caches();
+ f2fs_destroy_checkpoint_caches();
free_segment_manager_caches:
- destroy_segment_manager_caches();
+ f2fs_destroy_segment_manager_caches();
free_node_manager_caches:
- destroy_node_manager_caches();
+ f2fs_destroy_node_manager_caches();
free_inodecache:
destroy_inodecache();
fail:
@@ -3137,10 +3655,10 @@ static void __exit exit_f2fs_fs(void)
unregister_filesystem(&f2fs_fs_type);
unregister_shrinker(&f2fs_shrinker_info);
f2fs_exit_sysfs();
- destroy_extent_cache();
- destroy_checkpoint_caches();
- destroy_segment_manager_caches();
- destroy_node_manager_caches();
+ f2fs_destroy_extent_cache();
+ f2fs_destroy_checkpoint_caches();
+ f2fs_destroy_segment_manager_caches();
+ f2fs_destroy_node_manager_caches();
destroy_inodecache();
f2fs_destroy_trace_ios();
}
diff --git a/fs/f2fs/sysfs.c b/fs/f2fs/sysfs.c
index 26cb855..bf9e529 100644
--- a/fs/f2fs/sysfs.c
+++ b/fs/f2fs/sysfs.c
@@ -1,14 +1,12 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* f2fs sysfs interface
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
* Copyright (c) 2017 Chao Yu <chao@kernel.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
+#include <linux/compiler.h>
#include <linux/proc_fs.h>
#include <linux/f2fs_fs.h>
#include <linux/seq_file.h>
@@ -92,33 +90,39 @@ static ssize_t features_show(struct f2fs_attr *a,
if (!sb->s_bdev->bd_part)
return snprintf(buf, PAGE_SIZE, "0\n");
- if (f2fs_sb_has_encrypt(sb))
+ if (f2fs_sb_has_encrypt(sbi))
len += snprintf(buf, PAGE_SIZE - len, "%s",
"encryption");
- if (f2fs_sb_has_blkzoned(sb))
+ if (f2fs_sb_has_blkzoned(sbi))
len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
len ? ", " : "", "blkzoned");
- if (f2fs_sb_has_extra_attr(sb))
+ if (f2fs_sb_has_extra_attr(sbi))
len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
len ? ", " : "", "extra_attr");
- if (f2fs_sb_has_project_quota(sb))
+ if (f2fs_sb_has_project_quota(sbi))
len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
len ? ", " : "", "projquota");
- if (f2fs_sb_has_inode_chksum(sb))
+ if (f2fs_sb_has_inode_chksum(sbi))
len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
len ? ", " : "", "inode_checksum");
- if (f2fs_sb_has_flexible_inline_xattr(sb))
+ if (f2fs_sb_has_flexible_inline_xattr(sbi))
len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
len ? ", " : "", "flexible_inline_xattr");
- if (f2fs_sb_has_quota_ino(sb))
+ if (f2fs_sb_has_quota_ino(sbi))
len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
len ? ", " : "", "quota_ino");
- if (f2fs_sb_has_inode_crtime(sb))
+ if (f2fs_sb_has_inode_crtime(sbi))
len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
len ? ", " : "", "inode_crtime");
- if (f2fs_sb_has_lost_found(sb))
+ if (f2fs_sb_has_lost_found(sbi))
len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
len ? ", " : "", "lost_found");
+ if (f2fs_sb_has_verity(sbi))
+ len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
+ len ? ", " : "", "verity");
+ if (f2fs_sb_has_sb_chksum(sbi))
+ len += snprintf(buf + len, PAGE_SIZE - len, "%s%s",
+ len ? ", " : "", "sb_checksum");
len += snprintf(buf + len, PAGE_SIZE - len, "\n");
return len;
}
@@ -147,13 +151,13 @@ static ssize_t f2fs_sbi_show(struct f2fs_attr *a,
int len = 0, i;
len += snprintf(buf + len, PAGE_SIZE - len,
- "cold file extenstion:\n");
+ "cold file extension:\n");
for (i = 0; i < cold_count; i++)
len += snprintf(buf + len, PAGE_SIZE - len, "%s\n",
extlist[i]);
len += snprintf(buf + len, PAGE_SIZE - len,
- "hot file extenstion:\n");
+ "hot file extension:\n");
for (i = cold_count; i < cold_count + hot_count; i++)
len += snprintf(buf + len, PAGE_SIZE - len, "%s\n",
extlist[i]);
@@ -165,7 +169,7 @@ static ssize_t f2fs_sbi_show(struct f2fs_attr *a,
return snprintf(buf, PAGE_SIZE, "%u\n", *ui);
}
-static ssize_t f2fs_sbi_store(struct f2fs_attr *a,
+static ssize_t __sbi_store(struct f2fs_attr *a,
struct f2fs_sb_info *sbi,
const char *buf, size_t count)
{
@@ -201,13 +205,13 @@ static ssize_t f2fs_sbi_store(struct f2fs_attr *a,
down_write(&sbi->sb_lock);
- ret = update_extension_list(sbi, name, hot, set);
+ ret = f2fs_update_extension_list(sbi, name, hot, set);
if (ret)
goto out;
ret = f2fs_commit_super(sbi, false);
if (ret)
- update_extension_list(sbi, name, hot, !set);
+ f2fs_update_extension_list(sbi, name, hot, !set);
out:
up_write(&sbi->sb_lock);
return ret ? ret : count;
@@ -245,22 +249,64 @@ static ssize_t f2fs_sbi_store(struct f2fs_attr *a,
return count;
}
+ if (!strcmp(a->attr.name, "migration_granularity")) {
+ if (t == 0 || t > sbi->segs_per_sec)
+ return -EINVAL;
+ }
+
if (!strcmp(a->attr.name, "trim_sections"))
return -EINVAL;
+ if (!strcmp(a->attr.name, "gc_urgent")) {
+ if (t >= 1) {
+ sbi->gc_mode = GC_URGENT;
+ if (sbi->gc_thread) {
+ sbi->gc_thread->gc_wake = 1;
+ wake_up_interruptible_all(
+ &sbi->gc_thread->gc_wait_queue_head);
+ wake_up_discard_thread(sbi, true);
+ }
+ } else {
+ sbi->gc_mode = GC_NORMAL;
+ }
+ return count;
+ }
+ if (!strcmp(a->attr.name, "gc_idle")) {
+ if (t == GC_IDLE_CB)
+ sbi->gc_mode = GC_IDLE_CB;
+ else if (t == GC_IDLE_GREEDY)
+ sbi->gc_mode = GC_IDLE_GREEDY;
+ else
+ sbi->gc_mode = GC_NORMAL;
+ return count;
+ }
+
*ui = t;
if (!strcmp(a->attr.name, "iostat_enable") && *ui == 0)
f2fs_reset_iostat(sbi);
- if (!strcmp(a->attr.name, "gc_urgent") && t == 1 && sbi->gc_thread) {
- sbi->gc_thread->gc_wake = 1;
- wake_up_interruptible_all(&sbi->gc_thread->gc_wait_queue_head);
- wake_up_discard_thread(sbi, true);
- }
-
return count;
}
+static ssize_t f2fs_sbi_store(struct f2fs_attr *a,
+ struct f2fs_sb_info *sbi,
+ const char *buf, size_t count)
+{
+ ssize_t ret;
+ bool gc_entry = (!strcmp(a->attr.name, "gc_urgent") ||
+ a->struct_type == GC_THREAD);
+
+ if (gc_entry) {
+ if (!down_read_trylock(&sbi->sb->s_umount))
+ return -EAGAIN;
+ }
+ ret = __sbi_store(a, sbi, buf, count);
+ if (gc_entry)
+ up_read(&sbi->sb->s_umount);
+
+ return ret;
+}
+
static ssize_t f2fs_attr_show(struct kobject *kobj,
struct attribute *attr, char *buf)
{
@@ -299,6 +345,8 @@ enum feat_id {
FEAT_QUOTA_INO,
FEAT_INODE_CRTIME,
FEAT_LOST_FOUND,
+ FEAT_VERITY,
+ FEAT_SB_CHECKSUM,
};
static ssize_t f2fs_feature_show(struct f2fs_attr *a,
@@ -315,6 +363,8 @@ static ssize_t f2fs_feature_show(struct f2fs_attr *a,
case FEAT_QUOTA_INO:
case FEAT_INODE_CRTIME:
case FEAT_LOST_FOUND:
+ case FEAT_VERITY:
+ case FEAT_SB_CHECKSUM:
return snprintf(buf, PAGE_SIZE, "supported\n");
}
return 0;
@@ -349,8 +399,8 @@ F2FS_RW_ATTR(GC_THREAD, f2fs_gc_kthread, gc_urgent_sleep_time,
F2FS_RW_ATTR(GC_THREAD, f2fs_gc_kthread, gc_min_sleep_time, min_sleep_time);
F2FS_RW_ATTR(GC_THREAD, f2fs_gc_kthread, gc_max_sleep_time, max_sleep_time);
F2FS_RW_ATTR(GC_THREAD, f2fs_gc_kthread, gc_no_gc_sleep_time, no_gc_sleep_time);
-F2FS_RW_ATTR(GC_THREAD, f2fs_gc_kthread, gc_idle, gc_idle);
-F2FS_RW_ATTR(GC_THREAD, f2fs_gc_kthread, gc_urgent, gc_urgent);
+F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, gc_idle, gc_mode);
+F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, gc_urgent, gc_mode);
F2FS_RW_ATTR(SM_INFO, f2fs_sm_info, reclaim_segments, rec_prefree_segments);
F2FS_RW_ATTR(DCC_INFO, discard_cmd_control, max_small_discards, max_discards);
F2FS_RW_ATTR(DCC_INFO, discard_cmd_control, discard_granularity, discard_granularity);
@@ -366,9 +416,13 @@ F2FS_RW_ATTR(NM_INFO, f2fs_nm_info, ram_thresh, ram_thresh);
F2FS_RW_ATTR(NM_INFO, f2fs_nm_info, ra_nid_pages, ra_nid_pages);
F2FS_RW_ATTR(NM_INFO, f2fs_nm_info, dirty_nats_ratio, dirty_nats_ratio);
F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, max_victim_search, max_victim_search);
+F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, migration_granularity, migration_granularity);
F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, dir_level, dir_level);
F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, cp_interval, interval_time[CP_TIME]);
F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, idle_interval, interval_time[REQ_TIME]);
+F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, discard_idle_interval,
+ interval_time[DISCARD_TIME]);
+F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, gc_idle_interval, interval_time[GC_TIME]);
F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, iostat_enable, iostat_enable);
F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, readdir_ra, readdir_ra);
F2FS_RW_ATTR(F2FS_SBI, f2fs_sb_info, gc_pin_file_thresh, gc_pin_file_threshold);
@@ -396,6 +450,10 @@ F2FS_FEATURE_RO_ATTR(flexible_inline_xattr, FEAT_FLEXIBLE_INLINE_XATTR);
F2FS_FEATURE_RO_ATTR(quota_ino, FEAT_QUOTA_INO);
F2FS_FEATURE_RO_ATTR(inode_crtime, FEAT_INODE_CRTIME);
F2FS_FEATURE_RO_ATTR(lost_found, FEAT_LOST_FOUND);
+#ifdef CONFIG_F2FS_FS_VERITY
+F2FS_FEATURE_RO_ATTR(verity, FEAT_VERITY);
+#endif
+F2FS_FEATURE_RO_ATTR(sb_checksum, FEAT_SB_CHECKSUM);
#define ATTR_LIST(name) (&f2fs_attr_##name.attr)
static struct attribute *f2fs_attrs[] = {
@@ -416,12 +474,15 @@ static struct attribute *f2fs_attrs[] = {
ATTR_LIST(min_hot_blocks),
ATTR_LIST(min_ssr_sections),
ATTR_LIST(max_victim_search),
+ ATTR_LIST(migration_granularity),
ATTR_LIST(dir_level),
ATTR_LIST(ram_thresh),
ATTR_LIST(ra_nid_pages),
ATTR_LIST(dirty_nats_ratio),
ATTR_LIST(cp_interval),
ATTR_LIST(idle_interval),
+ ATTR_LIST(discard_idle_interval),
+ ATTR_LIST(gc_idle_interval),
ATTR_LIST(iostat_enable),
ATTR_LIST(readdir_ra),
ATTR_LIST(gc_pin_file_thresh),
@@ -453,6 +514,10 @@ static struct attribute *f2fs_feat_attrs[] = {
ATTR_LIST(quota_ino),
ATTR_LIST(inode_crtime),
ATTR_LIST(lost_found),
+#ifdef CONFIG_F2FS_FS_VERITY
+ ATTR_LIST(verity),
+#endif
+ ATTR_LIST(sb_checksum),
NULL,
};
@@ -484,7 +549,8 @@ static struct kobject f2fs_feat = {
.kset = &f2fs_kset,
};
-static int segment_info_seq_show(struct seq_file *seq, void *offset)
+static int __maybe_unused segment_info_seq_show(struct seq_file *seq,
+ void *offset)
{
struct super_block *sb = seq->private;
struct f2fs_sb_info *sbi = F2FS_SB(sb);
@@ -511,7 +577,8 @@ static int segment_info_seq_show(struct seq_file *seq, void *offset)
return 0;
}
-static int segment_bits_seq_show(struct seq_file *seq, void *offset)
+static int __maybe_unused segment_bits_seq_show(struct seq_file *seq,
+ void *offset)
{
struct super_block *sb = seq->private;
struct f2fs_sb_info *sbi = F2FS_SB(sb);
@@ -535,7 +602,8 @@ static int segment_bits_seq_show(struct seq_file *seq, void *offset)
return 0;
}
-static int iostat_info_seq_show(struct seq_file *seq, void *offset)
+static int __maybe_unused iostat_info_seq_show(struct seq_file *seq,
+ void *offset)
{
struct super_block *sb = seq->private;
struct f2fs_sb_info *sbi = F2FS_SB(sb);
@@ -577,6 +645,28 @@ static int iostat_info_seq_show(struct seq_file *seq, void *offset)
return 0;
}
+static int __maybe_unused victim_bits_seq_show(struct seq_file *seq,
+ void *offset)
+{
+ struct super_block *sb = seq->private;
+ struct f2fs_sb_info *sbi = F2FS_SB(sb);
+ struct dirty_seglist_info *dirty_i = DIRTY_I(sbi);
+ int i;
+
+ seq_puts(seq, "format: victim_secmap bitmaps\n");
+
+ for (i = 0; i < MAIN_SECS(sbi); i++) {
+ if ((i % 10) == 0)
+ seq_printf(seq, "%-10d", i);
+ seq_printf(seq, "%d", test_bit(i, dirty_i->victim_secmap) ? 1 : 0);
+ if ((i % 10) == 9 || i == (MAIN_SECS(sbi) - 1))
+ seq_putc(seq, '\n');
+ else
+ seq_putc(seq, ' ');
+ }
+ return 0;
+}
+
#define F2FS_PROC_FILE_DEF(_name) \
static int _name##_open_fs(struct inode *inode, struct file *file) \
{ \
@@ -593,6 +683,7 @@ static const struct file_operations f2fs_seq_##_name##_fops = { \
F2FS_PROC_FILE_DEF(segment_info);
F2FS_PROC_FILE_DEF(segment_bits);
F2FS_PROC_FILE_DEF(iostat_info);
+F2FS_PROC_FILE_DEF(victim_bits);
int __init f2fs_init_sysfs(void)
{
@@ -643,6 +734,8 @@ int f2fs_register_sysfs(struct f2fs_sb_info *sbi)
&f2fs_seq_segment_bits_fops, sb);
proc_create_data("iostat_info", S_IRUGO, sbi->s_proc,
&f2fs_seq_iostat_info_fops, sb);
+ proc_create_data("victim_bits", S_IRUGO, sbi->s_proc,
+ &f2fs_seq_victim_bits_fops, sb);
}
return 0;
}
@@ -653,6 +746,7 @@ void f2fs_unregister_sysfs(struct f2fs_sb_info *sbi)
remove_proc_entry("iostat_info", sbi->s_proc);
remove_proc_entry("segment_info", sbi->s_proc);
remove_proc_entry("segment_bits", sbi->s_proc);
+ remove_proc_entry("victim_bits", sbi->s_proc);
remove_proc_entry(sbi->sb->s_id, f2fs_proc_root);
}
kobject_del(&sbi->s_kobj);
diff --git a/fs/f2fs/trace.c b/fs/f2fs/trace.c
index a1fcd00..ce2a5eb 100644
--- a/fs/f2fs/trace.c
+++ b/fs/f2fs/trace.c
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* f2fs IO tracer
*
* Copyright (c) 2014 Motorola Mobility
* Copyright (c) 2014 Jaegeuk Kim <jaegeuk@kernel.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/fs.h>
#include <linux/f2fs_fs.h>
diff --git a/fs/f2fs/trace.h b/fs/f2fs/trace.h
index 67db24a..e8075fc 100644
--- a/fs/f2fs/trace.h
+++ b/fs/f2fs/trace.h
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* f2fs IO tracer
*
* Copyright (c) 2014 Motorola Mobility
* Copyright (c) 2014 Jaegeuk Kim <jaegeuk@kernel.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#ifndef __F2FS_TRACE_H__
#define __F2FS_TRACE_H__
diff --git a/fs/f2fs/xattr.c b/fs/f2fs/xattr.c
index ae2dfa7..18d5ffb 100644
--- a/fs/f2fs/xattr.c
+++ b/fs/f2fs/xattr.c
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/xattr.c
*
@@ -13,10 +14,6 @@
* suggestion of Luka Renko <luka.renko@hermes.si>.
* xattr consolidation Copyright (c) 2004 James Morris <jmorris@redhat.com>,
* Red Hat Inc.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#include <linux/rwsem.h>
#include <linux/f2fs_fs.h>
@@ -37,9 +34,6 @@ static int f2fs_xattr_generic_get(const struct xattr_handler *handler,
return -EOPNOTSUPP;
break;
case F2FS_XATTR_INDEX_TRUSTED:
- if (!capable(CAP_SYS_ADMIN))
- return -EPERM;
- break;
case F2FS_XATTR_INDEX_SECURITY:
break;
default:
@@ -62,9 +56,6 @@ static int f2fs_xattr_generic_set(const struct xattr_handler *handler,
return -EOPNOTSUPP;
break;
case F2FS_XATTR_INDEX_TRUSTED:
- if (!capable(CAP_SYS_ADMIN))
- return -EPERM;
- break;
case F2FS_XATTR_INDEX_SECURITY:
break;
default:
@@ -100,12 +91,22 @@ static int f2fs_xattr_advise_set(const struct xattr_handler *handler,
const char *name, const void *value,
size_t size, int flags)
{
+ unsigned char old_advise = F2FS_I(inode)->i_advise;
+ unsigned char new_advise;
+
if (!inode_owner_or_capable(inode))
return -EPERM;
if (value == NULL)
return -EINVAL;
- F2FS_I(inode)->i_advise |= *(char *)value;
+ new_advise = *(char *)value;
+ if (new_advise & ~FADVISE_MODIFIABLE_BITS)
+ return -EINVAL;
+
+ new_advise = new_advise & FADVISE_MODIFIABLE_BITS;
+ new_advise |= old_advise & ~FADVISE_MODIFIABLE_BITS;
+
+ F2FS_I(inode)->i_advise = new_advise;
f2fs_mark_inode_dirty_sync(inode, true);
return 0;
}
@@ -252,7 +253,7 @@ static int read_inline_xattr(struct inode *inode, struct page *ipage,
if (ipage) {
inline_addr = inline_xattr_addr(inode, ipage);
} else {
- page = get_node_page(sbi, inode->i_ino);
+ page = f2fs_get_node_page(sbi, inode->i_ino);
if (IS_ERR(page))
return PTR_ERR(page);
@@ -273,7 +274,7 @@ static int read_xattr_block(struct inode *inode, void *txattr_addr)
void *xattr_addr;
/* The inode already has an extended attribute block. */
- xpage = get_node_page(sbi, xnid);
+ xpage = f2fs_get_node_page(sbi, xnid);
if (IS_ERR(xpage))
return PTR_ERR(xpage);
@@ -287,7 +288,7 @@ static int read_xattr_block(struct inode *inode, void *txattr_addr)
static int lookup_all_xattrs(struct inode *inode, struct page *ipage,
unsigned int index, unsigned int len,
const char *name, struct f2fs_xattr_entry **xe,
- void **base_addr)
+ void **base_addr, int *base_size)
{
void *cur_addr, *txattr_addr, *last_addr = NULL;
nid_t xnid = F2FS_I(inode)->i_xattr_nid;
@@ -298,8 +299,8 @@ static int lookup_all_xattrs(struct inode *inode, struct page *ipage,
if (!size && !inline_size)
return -ENODATA;
- txattr_addr = f2fs_kzalloc(F2FS_I_SB(inode),
- inline_size + size + XATTR_PADDING_SIZE, GFP_NOFS);
+ *base_size = inline_size + size + XATTR_PADDING_SIZE;
+ txattr_addr = f2fs_kzalloc(F2FS_I_SB(inode), *base_size, GFP_NOFS);
if (!txattr_addr)
return -ENOMEM;
@@ -311,8 +312,10 @@ static int lookup_all_xattrs(struct inode *inode, struct page *ipage,
*xe = __find_inline_xattr(inode, txattr_addr, &last_addr,
index, len, name);
- if (*xe)
+ if (*xe) {
+ *base_size = inline_size;
goto check;
+ }
}
/* read from xattr node block */
@@ -397,7 +400,7 @@ static inline int write_all_xattrs(struct inode *inode, __u32 hsize,
int err = 0;
if (hsize > inline_size && !F2FS_I(inode)->i_xattr_nid)
- if (!alloc_nid(sbi, &new_nid))
+ if (!f2fs_alloc_nid(sbi, &new_nid))
return -ENOSPC;
/* write to inline xattr */
@@ -405,20 +408,20 @@ static inline int write_all_xattrs(struct inode *inode, __u32 hsize,
if (ipage) {
inline_addr = inline_xattr_addr(inode, ipage);
} else {
- in_page = get_node_page(sbi, inode->i_ino);
+ in_page = f2fs_get_node_page(sbi, inode->i_ino);
if (IS_ERR(in_page)) {
- alloc_nid_failed(sbi, new_nid);
+ f2fs_alloc_nid_failed(sbi, new_nid);
return PTR_ERR(in_page);
}
inline_addr = inline_xattr_addr(inode, in_page);
}
f2fs_wait_on_page_writeback(ipage ? ipage : in_page,
- NODE, true);
+ NODE, true, true);
/* no need to use xattr node block */
if (hsize <= inline_size) {
- err = truncate_xattr_node(inode);
- alloc_nid_failed(sbi, new_nid);
+ err = f2fs_truncate_xattr_node(inode);
+ f2fs_alloc_nid_failed(sbi, new_nid);
if (err) {
f2fs_put_page(in_page, 1);
return err;
@@ -431,24 +434,24 @@ static inline int write_all_xattrs(struct inode *inode, __u32 hsize,
/* write to xattr node block */
if (F2FS_I(inode)->i_xattr_nid) {
- xpage = get_node_page(sbi, F2FS_I(inode)->i_xattr_nid);
+ xpage = f2fs_get_node_page(sbi, F2FS_I(inode)->i_xattr_nid);
if (IS_ERR(xpage)) {
err = PTR_ERR(xpage);
- alloc_nid_failed(sbi, new_nid);
+ f2fs_alloc_nid_failed(sbi, new_nid);
goto in_page_out;
}
f2fs_bug_on(sbi, new_nid);
- f2fs_wait_on_page_writeback(xpage, NODE, true);
+ f2fs_wait_on_page_writeback(xpage, NODE, true, true);
} else {
struct dnode_of_data dn;
set_new_dnode(&dn, inode, NULL, NULL, new_nid);
- xpage = new_node_page(&dn, XATTR_NODE_OFFSET);
+ xpage = f2fs_new_node_page(&dn, XATTR_NODE_OFFSET);
if (IS_ERR(xpage)) {
err = PTR_ERR(xpage);
- alloc_nid_failed(sbi, new_nid);
+ f2fs_alloc_nid_failed(sbi, new_nid);
goto in_page_out;
}
- alloc_nid_done(sbi, new_nid);
+ f2fs_alloc_nid_done(sbi, new_nid);
}
xattr_addr = page_address(xpage);
@@ -473,6 +476,7 @@ int f2fs_getxattr(struct inode *inode, int index, const char *name,
int error = 0;
unsigned int size, len;
void *base_addr = NULL;
+ int base_size;
if (name == NULL)
return -EINVAL;
@@ -483,7 +487,7 @@ int f2fs_getxattr(struct inode *inode, int index, const char *name,
down_read(&F2FS_I(inode)->i_xattr_sem);
error = lookup_all_xattrs(inode, ipage, index, len, name,
- &entry, &base_addr);
+ &entry, &base_addr, &base_size);
up_read(&F2FS_I(inode)->i_xattr_sem);
if (error)
return error;
@@ -497,6 +501,11 @@ int f2fs_getxattr(struct inode *inode, int index, const char *name,
if (buffer) {
char *pval = entry->e_name + entry->e_name_len;
+
+ if (base_size - (pval - (char *)base_addr) < size) {
+ error = -ERANGE;
+ goto out;
+ }
memcpy(buffer, pval, size);
}
error = size;
@@ -693,7 +702,7 @@ int f2fs_setxattr(struct inode *inode, int index, const char *name,
if (err)
return err;
- /* this case is only from init_inode_metadata */
+ /* this case is only from f2fs_init_inode_metadata */
if (ipage)
return __f2fs_setxattr(inode, index, name, value,
size, ipage, flags);
diff --git a/fs/f2fs/xattr.h b/fs/f2fs/xattr.h
index dbcd1d1..67db134 100644
--- a/fs/f2fs/xattr.h
+++ b/fs/f2fs/xattr.h
@@ -1,3 +1,4 @@
+// SPDX-License-Identifier: GPL-2.0
/*
* fs/f2fs/xattr.h
*
@@ -9,10 +10,6 @@
* On-disk format of extended attributes for the ext2 filesystem.
*
* (C) 2001 Andreas Gruenbacher, <a.gruenbacher@computer.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#ifndef __F2FS_XATTR_H__
#define __F2FS_XATTR_H__
diff --git a/fs/fat/cache.c b/fs/fat/cache.c
index 5d38492..f04b189 100644
--- a/fs/fat/cache.c
+++ b/fs/fat/cache.c
@@ -224,7 +224,8 @@ static inline void cache_init(struct fat_cache_id *cid, int fclus, int dclus)
int fat_get_cluster(struct inode *inode, int cluster, int *fclus, int *dclus)
{
struct super_block *sb = inode->i_sb;
- const int limit = sb->s_maxbytes >> MSDOS_SB(sb)->cluster_bits;
+ struct msdos_sb_info *sbi = MSDOS_SB(sb);
+ const int limit = sb->s_maxbytes >> sbi->cluster_bits;
struct fat_entry fatent;
struct fat_cache_id cid;
int nr;
@@ -233,6 +234,12 @@ int fat_get_cluster(struct inode *inode, int cluster, int *fclus, int *dclus)
*fclus = 0;
*dclus = MSDOS_I(inode)->i_start;
+ if (!fat_valid_entry(sbi, *dclus)) {
+ fat_fs_error_ratelimit(sb,
+ "%s: invalid start cluster (i_pos %lld, start %08x)",
+ __func__, MSDOS_I(inode)->i_pos, *dclus);
+ return -EIO;
+ }
if (cluster == 0)
return 0;
@@ -249,9 +256,8 @@ int fat_get_cluster(struct inode *inode, int cluster, int *fclus, int *dclus)
/* prevent the infinite loop of cluster chain */
if (*fclus > limit) {
fat_fs_error_ratelimit(sb,
- "%s: detected the cluster chain loop"
- " (i_pos %lld)", __func__,
- MSDOS_I(inode)->i_pos);
+ "%s: detected the cluster chain loop (i_pos %lld)",
+ __func__, MSDOS_I(inode)->i_pos);
nr = -EIO;
goto out;
}
@@ -261,9 +267,8 @@ int fat_get_cluster(struct inode *inode, int cluster, int *fclus, int *dclus)
goto out;
else if (nr == FAT_ENT_FREE) {
fat_fs_error_ratelimit(sb,
- "%s: invalid cluster chain (i_pos %lld)",
- __func__,
- MSDOS_I(inode)->i_pos);
+ "%s: invalid cluster chain (i_pos %lld)",
+ __func__, MSDOS_I(inode)->i_pos);
nr = -EIO;
goto out;
} else if (nr == FAT_ENT_EOF) {
diff --git a/fs/fat/fat.h b/fs/fat/fat.h
index e6b764a..437affe 100644
--- a/fs/fat/fat.h
+++ b/fs/fat/fat.h
@@ -347,6 +347,11 @@ static inline void fatent_brelse(struct fat_entry *fatent)
fatent->fat_inode = NULL;
}
+static inline bool fat_valid_entry(struct msdos_sb_info *sbi, int entry)
+{
+ return FAT_START_ENT <= entry && entry < sbi->max_cluster;
+}
+
extern void fat_ent_access_init(struct super_block *sb);
extern int fat_ent_read(struct inode *inode, struct fat_entry *fatent,
int entry);
diff --git a/fs/fat/fatent.c b/fs/fat/fatent.c
index 57b0902..18121c7 100644
--- a/fs/fat/fatent.c
+++ b/fs/fat/fatent.c
@@ -23,7 +23,7 @@ static void fat12_ent_blocknr(struct super_block *sb, int entry,
{
struct msdos_sb_info *sbi = MSDOS_SB(sb);
int bytes = entry + (entry >> 1);
- WARN_ON(entry < FAT_START_ENT || sbi->max_cluster <= entry);
+ WARN_ON(!fat_valid_entry(sbi, entry));
*offset = bytes & (sb->s_blocksize - 1);
*blocknr = sbi->fat_start + (bytes >> sb->s_blocksize_bits);
}
@@ -33,7 +33,7 @@ static void fat_ent_blocknr(struct super_block *sb, int entry,
{
struct msdos_sb_info *sbi = MSDOS_SB(sb);
int bytes = (entry << sbi->fatent_shift);
- WARN_ON(entry < FAT_START_ENT || sbi->max_cluster <= entry);
+ WARN_ON(!fat_valid_entry(sbi, entry));
*offset = bytes & (sb->s_blocksize - 1);
*blocknr = sbi->fat_start + (bytes >> sb->s_blocksize_bits);
}
@@ -354,7 +354,7 @@ int fat_ent_read(struct inode *inode, struct fat_entry *fatent, int entry)
int err, offset;
sector_t blocknr;
- if (entry < FAT_START_ENT || sbi->max_cluster <= entry) {
+ if (!fat_valid_entry(sbi, entry)) {
fatent_brelse(fatent);
fat_fs_error(sb, "invalid access to FAT (entry 0x%08x)", entry);
return -EIO;
@@ -682,6 +682,7 @@ int fat_count_free_clusters(struct super_block *sb)
if (ops->ent_get(&fatent) == FAT_ENT_FREE)
free++;
} while (fat_ent_next(sbi, &fatent));
+ cond_resched();
}
sbi->free_clusters = free;
sbi->free_clus_valid = 1;
diff --git a/fs/fat/inode.c b/fs/fat/inode.c
index 0b6ba8c..eea5bc2 100644
--- a/fs/fat/inode.c
+++ b/fs/fat/inode.c
@@ -696,13 +696,21 @@ static void fat_set_state(struct super_block *sb,
brelse(bh);
}
+static void fat_reset_iocharset(struct fat_mount_options *opts)
+{
+ if (opts->iocharset != fat_default_iocharset) {
+ /* Note: opts->iocharset can be NULL here */
+ kfree(opts->iocharset);
+ opts->iocharset = fat_default_iocharset;
+ }
+}
+
static void delayed_free(struct rcu_head *p)
{
struct msdos_sb_info *sbi = container_of(p, struct msdos_sb_info, rcu);
unload_nls(sbi->nls_disk);
unload_nls(sbi->nls_io);
- if (sbi->options.iocharset != fat_default_iocharset)
- kfree(sbi->options.iocharset);
+ fat_reset_iocharset(&sbi->options);
kfree(sbi);
}
@@ -1118,7 +1126,7 @@ static int parse_options(struct super_block *sb, char *options, int is_vfat,
opts->fs_fmask = opts->fs_dmask = current_umask();
opts->allow_utime = -1;
opts->codepage = fat_default_codepage;
- opts->iocharset = fat_default_iocharset;
+ fat_reset_iocharset(opts);
if (is_vfat) {
opts->shortname = VFAT_SFN_DISPLAY_WINNT|VFAT_SFN_CREATE_WIN95;
opts->rodir = 0;
@@ -1275,8 +1283,7 @@ static int parse_options(struct super_block *sb, char *options, int is_vfat,
/* vfat specific */
case Opt_charset:
- if (opts->iocharset != fat_default_iocharset)
- kfree(opts->iocharset);
+ fat_reset_iocharset(opts);
iocharset = match_strdup(&args[0]);
if (!iocharset)
return -ENOMEM;
@@ -1867,8 +1874,7 @@ int fat_fill_super(struct super_block *sb, void *data, int silent, int isvfat,
iput(fat_inode);
unload_nls(sbi->nls_io);
unload_nls(sbi->nls_disk);
- if (sbi->options.iocharset != fat_default_iocharset)
- kfree(sbi->options.iocharset);
+ fat_reset_iocharset(&sbi->options);
sb->s_fs_info = NULL;
kfree(sbi);
return error;
diff --git a/fs/file_table.c b/fs/file_table.c
index ad17e05..5a274d5 100644
--- a/fs/file_table.c
+++ b/fs/file_table.c
@@ -261,6 +261,12 @@ void flush_delayed_fput(void)
static DECLARE_DELAYED_WORK(delayed_fput_work, delayed_fput);
+void flush_delayed_fput_wait(void)
+{
+ delayed_fput(NULL);
+ flush_delayed_work(&delayed_fput_work);
+}
+
void fput(struct file *file)
{
if (atomic_long_dec_and_test(&file->f_count)) {
diff --git a/fs/fs-writeback.c b/fs/fs-writeback.c
index 5af226f..17ad41d 100644
--- a/fs/fs-writeback.c
+++ b/fs/fs-writeback.c
@@ -1942,7 +1942,7 @@ void wb_workfn(struct work_struct *work)
}
if (!list_empty(&wb->work_list))
- mod_delayed_work(bdi_wq, &wb->dwork, 0);
+ wb_wakeup(wb);
else if (wb_has_dirty_io(wb) && dirty_writeback_interval)
wb_wakeup_delayed(wb);
diff --git a/fs/fscache/object.c b/fs/fscache/object.c
index 7a182c8..ab1d7f3 100644
--- a/fs/fscache/object.c
+++ b/fs/fscache/object.c
@@ -715,6 +715,9 @@ static const struct fscache_state *fscache_drop_object(struct fscache_object *ob
if (awaken)
wake_up_bit(&cookie->flags, FSCACHE_COOKIE_INVALIDATING);
+ if (test_and_clear_bit(FSCACHE_COOKIE_LOOKING_UP, &cookie->flags))
+ wake_up_bit(&cookie->flags, FSCACHE_COOKIE_LOOKING_UP);
+
/* Prevent a race with our last child, which has to signal EV_CLEARED
* before dropping our spinlock.
diff --git a/fs/fscache/operation.c b/fs/fscache/operation.c
index de67745..77946d6 100644
--- a/fs/fscache/operation.c
+++ b/fs/fscache/operation.c
@@ -66,7 +66,8 @@ void fscache_enqueue_operation(struct fscache_operation *op)
ASSERT(op->processor != NULL);
ASSERT(fscache_object_is_available(op->object));
ASSERTCMP(atomic_read(&op->usage), >, 0);
- ASSERTCMP(op->state, ==, FSCACHE_OP_ST_IN_PROGRESS);
+ ASSERTIFCMP(op->state != FSCACHE_OP_ST_IN_PROGRESS,
+ op->state, ==, FSCACHE_OP_ST_CANCELLED);
fscache_stat(&fscache_n_op_enqueue);
switch (op->flags & FSCACHE_OP_TYPE) {
@@ -481,7 +482,8 @@ void fscache_put_operation(struct fscache_operation *op)
struct fscache_cache *cache;
_enter("{OBJ%x OP%x,%d}",
- op->object->debug_id, op->debug_id, atomic_read(&op->usage));
+ op->object ? op->object->debug_id : 0,
+ op->debug_id, atomic_read(&op->usage));
ASSERTCMP(atomic_read(&op->usage), >, 0);
diff --git a/fs/fscache/page.c b/fs/fscache/page.c
index c8c4f79..8a7923a 100644
--- a/fs/fscache/page.c
+++ b/fs/fscache/page.c
@@ -776,6 +776,7 @@ static void fscache_write_op(struct fscache_operation *_op)
_enter("{OP%x,%d}", op->op.debug_id, atomic_read(&op->op.usage));
+again:
spin_lock(&object->lock);
cookie = object->cookie;
@@ -816,10 +817,6 @@ static void fscache_write_op(struct fscache_operation *_op)
goto superseded;
page = results[0];
_debug("gang %d [%lx]", n, page->index);
- if (page->index >= op->store_limit) {
- fscache_stat(&fscache_n_store_pages_over_limit);
- goto superseded;
- }
radix_tree_tag_set(&cookie->stores, page->index,
FSCACHE_COOKIE_STORING_TAG);
@@ -829,6 +826,9 @@ static void fscache_write_op(struct fscache_operation *_op)
spin_unlock(&cookie->stores_lock);
spin_unlock(&object->lock);
+ if (page->index >= op->store_limit)
+ goto discard_page;
+
fscache_stat(&fscache_n_store_pages);
fscache_stat(&fscache_n_cop_write_page);
ret = object->cache->ops->write_page(op, page);
@@ -844,6 +844,11 @@ static void fscache_write_op(struct fscache_operation *_op)
_leave("");
return;
+discard_page:
+ fscache_stat(&fscache_n_store_pages_over_limit);
+ fscache_end_page_write(object, page);
+ goto again;
+
superseded:
/* this writer is going away and there aren't any more things to
* write */
diff --git a/fs/fuse/control.c b/fs/fuse/control.c
index 6e22748..e25c40c 100644
--- a/fs/fuse/control.c
+++ b/fs/fuse/control.c
@@ -211,10 +211,11 @@ static struct dentry *fuse_ctl_add_dentry(struct dentry *parent,
if (!dentry)
return NULL;
- fc->ctl_dentry[fc->ctl_ndents++] = dentry;
inode = new_inode(fuse_control_sb);
- if (!inode)
+ if (!inode) {
+ dput(dentry);
return NULL;
+ }
inode->i_ino = get_next_ino();
inode->i_mode = mode;
@@ -228,6 +229,9 @@ static struct dentry *fuse_ctl_add_dentry(struct dentry *parent,
set_nlink(inode, nlink);
inode->i_private = fc;
d_add(dentry, inode);
+
+ fc->ctl_dentry[fc->ctl_ndents++] = dentry;
+
return dentry;
}
@@ -284,7 +288,10 @@ void fuse_ctl_remove_conn(struct fuse_conn *fc)
for (i = fc->ctl_ndents - 1; i >= 0; i--) {
struct dentry *dentry = fc->ctl_dentry[i];
d_inode(dentry)->i_private = NULL;
- d_drop(dentry);
+ if (!i) {
+ /* Get rid of submounts: */
+ d_invalidate(dentry);
+ }
dput(dentry);
}
drop_nlink(d_inode(fuse_control_sb->s_root));
diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index a0b0683..3b094e3 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -133,6 +133,16 @@ static bool fuse_block_alloc(struct fuse_conn *fc, bool for_background)
return !fc->initialized || (for_background && fc->blocked);
}
+static void fuse_drop_waiting(struct fuse_conn *fc)
+{
+ if (fc->connected) {
+ atomic_dec(&fc->num_waiting);
+ } else if (atomic_dec_and_test(&fc->num_waiting)) {
+ /* wake up aborters */
+ wake_up_all(&fc->blocked_waitq);
+ }
+}
+
static struct fuse_req *__fuse_get_req(struct fuse_conn *fc, unsigned npages,
bool for_background)
{
@@ -173,7 +183,7 @@ static struct fuse_req *__fuse_get_req(struct fuse_conn *fc, unsigned npages,
return req;
out:
- atomic_dec(&fc->num_waiting);
+ fuse_drop_waiting(fc);
return ERR_PTR(err);
}
@@ -280,7 +290,7 @@ void fuse_put_request(struct fuse_conn *fc, struct fuse_req *req)
if (test_bit(FR_WAITING, &req->flags)) {
__clear_bit(FR_WAITING, &req->flags);
- atomic_dec(&fc->num_waiting);
+ fuse_drop_waiting(fc);
}
if (req->stolen_file)
@@ -366,7 +376,7 @@ static void request_end(struct fuse_conn *fc, struct fuse_req *req)
struct fuse_iqueue *fiq = &fc->iq;
if (test_and_set_bit(FR_FINISHED, &req->flags))
- return;
+ goto put_request;
spin_lock(&fiq->waitq.lock);
list_del_init(&req->intr_entry);
@@ -376,12 +386,19 @@ static void request_end(struct fuse_conn *fc, struct fuse_req *req)
if (test_bit(FR_BACKGROUND, &req->flags)) {
spin_lock(&fc->lock);
clear_bit(FR_BACKGROUND, &req->flags);
- if (fc->num_background == fc->max_background)
+ if (fc->num_background == fc->max_background) {
fc->blocked = 0;
-
- /* Wake up next waiter, if any */
- if (!fc->blocked && waitqueue_active(&fc->blocked_waitq))
wake_up(&fc->blocked_waitq);
+ } else if (!fc->blocked) {
+ /*
+ * Wake up next waiter, if any. It's okay to use
+ * waitqueue_active(), as we've already synced up
+ * fc->blocked with waiters with the wake_up() call
+ * above.
+ */
+ if (waitqueue_active(&fc->blocked_waitq))
+ wake_up(&fc->blocked_waitq);
+ }
if (fc->num_background == fc->congestion_threshold &&
fc->connected && fc->bdi_initialized) {
@@ -396,6 +413,7 @@ static void request_end(struct fuse_conn *fc, struct fuse_req *req)
wake_up(&req->waitq);
if (req->end)
req->end(fc, req);
+put_request:
fuse_put_request(fc, req);
}
@@ -1302,12 +1320,14 @@ static ssize_t fuse_dev_do_read(struct fuse_dev *fud, struct file *file,
goto out_end;
}
list_move_tail(&req->list, &fpq->processing);
- spin_unlock(&fpq->lock);
+ __fuse_get_request(req);
set_bit(FR_SENT, &req->flags);
+ spin_unlock(&fpq->lock);
/* matches barrier in request_wait_answer() */
smp_mb__after_atomic();
if (test_bit(FR_INTERRUPTED, &req->flags))
queue_interrupt(fiq, req);
+ fuse_put_request(fc, req);
return reqsize;
@@ -1705,8 +1725,10 @@ static int fuse_retrieve(struct fuse_conn *fc, struct inode *inode,
req->in.args[1].size = total_len;
err = fuse_request_send_notify_reply(fc, req, outarg->notify_unique);
- if (err)
+ if (err) {
fuse_retrieve_end(fc, req);
+ fuse_put_request(fc, req);
+ }
return err;
}
@@ -1865,16 +1887,20 @@ static ssize_t fuse_dev_do_write(struct fuse_dev *fud,
/* Is it an interrupt reply? */
if (req->intr_unique == oh.unique) {
+ __fuse_get_request(req);
spin_unlock(&fpq->lock);
err = -EINVAL;
- if (nbytes != sizeof(struct fuse_out_header))
+ if (nbytes != sizeof(struct fuse_out_header)) {
+ fuse_put_request(fc, req);
goto err_finish;
+ }
if (oh.error == -ENOSYS)
fc->no_interrupt = 1;
else if (oh.error == -EAGAIN)
queue_interrupt(&fc->iq, req);
+ fuse_put_request(fc, req);
fuse_copy_finish(cs);
return nbytes;
@@ -1954,11 +1980,14 @@ static ssize_t fuse_dev_splice_write(struct pipe_inode_info *pipe,
if (!fud)
return -EPERM;
- bufs = kmalloc(pipe->buffers * sizeof(struct pipe_buffer), GFP_KERNEL);
- if (!bufs)
- return -ENOMEM;
-
pipe_lock(pipe);
+
+ bufs = kmalloc(pipe->buffers * sizeof(struct pipe_buffer), GFP_KERNEL);
+ if (!bufs) {
+ pipe_unlock(pipe);
+ return -ENOMEM;
+ }
+
nbuf = 0;
rem = 0;
for (idx = 0; idx < pipe->nrbufs && rem < len; idx++)
@@ -2113,6 +2142,7 @@ void fuse_abort_conn(struct fuse_conn *fc)
set_bit(FR_ABORTED, &req->flags);
if (!test_bit(FR_LOCKED, &req->flags)) {
set_bit(FR_PRIVATE, &req->flags);
+ __fuse_get_request(req);
list_move(&req->list, &to_end1);
}
spin_unlock(&req->waitq.lock);
@@ -2139,7 +2169,6 @@ void fuse_abort_conn(struct fuse_conn *fc)
while (!list_empty(&to_end1)) {
req = list_first_entry(&to_end1, struct fuse_req, list);
- __fuse_get_request(req);
list_del_init(&req->list);
request_end(fc, req);
}
@@ -2150,6 +2179,11 @@ void fuse_abort_conn(struct fuse_conn *fc)
}
EXPORT_SYMBOL_GPL(fuse_abort_conn);
+void fuse_wait_aborted(struct fuse_conn *fc)
+{
+ wait_event(fc->blocked_waitq, atomic_read(&fc->num_waiting) == 0);
+}
+
int fuse_dev_release(struct inode *inode, struct file *file)
{
struct fuse_dev *fud = fuse_get_dev(file);
@@ -2157,9 +2191,15 @@ int fuse_dev_release(struct inode *inode, struct file *file)
if (fud) {
struct fuse_conn *fc = fud->fc;
struct fuse_pqueue *fpq = &fud->pq;
+ LIST_HEAD(to_end);
+ spin_lock(&fpq->lock);
WARN_ON(!list_empty(&fpq->io));
- end_requests(fc, &fpq->processing);
+ list_splice_init(&fpq->processing, &to_end);
+ spin_unlock(&fpq->lock);
+
+ end_requests(fc, &to_end);
+
/* Are we the last open device? */
if (atomic_dec_and_test(&fc->dev_count)) {
WARN_ON(fc->iq.fasync != NULL);
diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c
index 1693308..e43f1c6 100644
--- a/fs/fuse/dir.c
+++ b/fs/fuse/dir.c
@@ -401,11 +401,12 @@ static struct dentry *fuse_lookup(struct inode *dir, struct dentry *entry,
struct inode *inode;
struct dentry *newent;
bool outarg_valid = true;
+ bool locked;
- fuse_lock_inode(dir);
+ locked = fuse_lock_inode(dir);
err = fuse_lookup_name(dir->i_sb, get_node_id(dir), &entry->d_name,
&outarg, &inode);
- fuse_unlock_inode(dir);
+ fuse_unlock_inode(dir, locked);
if (err == -ENOENT) {
outarg_valid = false;
err = 0;
@@ -1385,6 +1386,7 @@ static int fuse_readdir(struct file *file, struct dir_context *ctx)
struct fuse_conn *fc = get_fuse_conn(inode);
struct fuse_req *req;
u64 attr_version = 0;
+ bool locked;
if (is_bad_inode(inode))
return -EIO;
@@ -1412,9 +1414,9 @@ static int fuse_readdir(struct file *file, struct dir_context *ctx)
fuse_read_fill(req, file, ctx->pos, PAGE_SIZE,
FUSE_READDIR);
}
- fuse_lock_inode(inode);
+ locked = fuse_lock_inode(inode);
fuse_request_send(fc, req);
- fuse_unlock_inode(inode);
+ fuse_unlock_inode(inode, locked);
nbytes = req->out.args[0].size;
err = req->out.h.error;
fuse_put_request(fc, req);
@@ -1682,8 +1684,19 @@ int fuse_do_setattr(struct dentry *dentry, struct iattr *attr,
return err;
if (attr->ia_valid & ATTR_OPEN) {
- if (fc->atomic_o_trunc)
+ /* This is coming from open(..., ... | O_TRUNC); */
+ WARN_ON(!(attr->ia_valid & ATTR_SIZE));
+ WARN_ON(attr->ia_size != 0);
+ if (fc->atomic_o_trunc) {
+ /*
+ * No need to send request to userspace, since actual
+ * truncation has already been done by OPEN. But still
+ * need to truncate page cache.
+ */
+ i_size_write(inode, 0);
+ truncate_pagecache(inode, 0);
return 0;
+ }
file = NULL;
}
diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index 7adf871..a474c66 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -886,6 +886,7 @@ static int fuse_readpages_fill(struct file *_data, struct page *page)
}
if (WARN_ON(req->num_pages >= req->max_pages)) {
+ unlock_page(page);
fuse_put_request(fc, req);
return -EIO;
}
@@ -2936,10 +2937,12 @@ fuse_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
}
if (io->async) {
+ bool blocking = io->blocking;
+
fuse_aio_complete(io, ret < 0 ? ret : 0, -1);
/* we have a non-extending, async request, so return */
- if (!io->blocking)
+ if (!blocking)
return -EIOCBQUEUED;
wait_for_completion(&wait);
diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h
index cc2c82c..2c1e88c 100644
--- a/fs/fuse/fuse_i.h
+++ b/fs/fuse/fuse_i.h
@@ -868,6 +868,7 @@ void fuse_request_send_background_locked(struct fuse_conn *fc,
/* Abort all requests */
void fuse_abort_conn(struct fuse_conn *fc);
+void fuse_wait_aborted(struct fuse_conn *fc);
/**
* Invalidate inode attributes
@@ -981,8 +982,8 @@ int fuse_do_setattr(struct dentry *dentry, struct iattr *attr,
void fuse_set_initialized(struct fuse_conn *fc);
-void fuse_unlock_inode(struct inode *inode);
-void fuse_lock_inode(struct inode *inode);
+void fuse_unlock_inode(struct inode *inode, bool locked);
+bool fuse_lock_inode(struct inode *inode);
int fuse_setxattr(struct inode *inode, const char *name, const void *value,
size_t size, int flags);
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index 8aa98b1..20d30eb 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -356,15 +356,21 @@ int fuse_reverse_inval_inode(struct super_block *sb, u64 nodeid,
return 0;
}
-void fuse_lock_inode(struct inode *inode)
+bool fuse_lock_inode(struct inode *inode)
{
- if (!get_fuse_conn(inode)->parallel_dirops)
+ bool locked = false;
+
+ if (!get_fuse_conn(inode)->parallel_dirops) {
mutex_lock(&get_fuse_inode(inode)->mutex);
+ locked = true;
+ }
+
+ return locked;
}
-void fuse_unlock_inode(struct inode *inode)
+void fuse_unlock_inode(struct inode *inode, bool locked)
{
- if (!get_fuse_conn(inode)->parallel_dirops)
+ if (locked)
mutex_unlock(&get_fuse_inode(inode)->mutex);
}
@@ -396,9 +402,6 @@ static void fuse_put_super(struct super_block *sb)
{
struct fuse_conn *fc = get_fuse_conn_super(sb);
- fuse_send_destroy(fc);
-
- fuse_abort_conn(fc);
mutex_lock(&fuse_mutex);
list_del(&fc->entry);
fuse_ctl_remove_conn(fc);
@@ -1190,6 +1193,7 @@ static int fuse_fill_super(struct super_block *sb, void *data, int silent)
err_put_conn:
fuse_bdi_destroy(fc);
fuse_conn_put(fc);
+ sb->s_fs_info = NULL;
err_fput:
fput(file);
err:
@@ -1203,16 +1207,25 @@ static struct dentry *fuse_mount(struct file_system_type *fs_type,
return mount_nodev(fs_type, flags, raw_data, fuse_fill_super);
}
-static void fuse_kill_sb_anon(struct super_block *sb)
+static void fuse_sb_destroy(struct super_block *sb)
{
struct fuse_conn *fc = get_fuse_conn_super(sb);
if (fc) {
+ fuse_send_destroy(fc);
+
+ fuse_abort_conn(fc);
+ fuse_wait_aborted(fc);
+
down_write(&fc->killsb);
fc->sb = NULL;
up_write(&fc->killsb);
}
+}
+static void fuse_kill_sb_anon(struct super_block *sb)
+{
+ fuse_sb_destroy(sb);
kill_anon_super(sb);
}
@@ -1235,14 +1248,7 @@ static struct dentry *fuse_mount_blk(struct file_system_type *fs_type,
static void fuse_kill_sb_blk(struct super_block *sb)
{
- struct fuse_conn *fc = get_fuse_conn_super(sb);
-
- if (fc) {
- down_write(&fc->killsb);
- fc->sb = NULL;
- up_write(&fc->killsb);
- }
-
+ fuse_sb_destroy(sb);
kill_block_super(sb);
}
diff --git a/fs/gfs2/aops.c b/fs/gfs2/aops.c
index c6c507c..0283ee0 100644
--- a/fs/gfs2/aops.c
+++ b/fs/gfs2/aops.c
@@ -268,22 +268,6 @@ static int gfs2_write_jdata_pagevec(struct address_space *mapping,
for(i = 0; i < nr_pages; i++) {
struct page *page = pvec->pages[i];
- /*
- * At this point, the page may be truncated or
- * invalidated (changing page->mapping to NULL), or
- * even swizzled back from swapper_space to tmpfs file
- * mapping. However, page->index will not change
- * because we have a reference on the page.
- */
- if (page->index > end) {
- /*
- * can't be range_cyclic (1st pass) because
- * end == -1 in that case.
- */
- ret = 1;
- break;
- }
-
*done_index = page->index;
lock_page(page);
@@ -401,8 +385,8 @@ static int gfs2_write_cache_jdata(struct address_space *mapping,
tag_pages_for_writeback(mapping, index, end);
done_index = index;
while (!done && (index <= end)) {
- nr_pages = pagevec_lookup_tag(&pvec, mapping, &index, tag,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
+ nr_pages = pagevec_lookup_range_tag(&pvec, mapping, &index, end,
+ tag);
if (nr_pages == 0)
break;
diff --git a/fs/gfs2/bmap.c b/fs/gfs2/bmap.c
index fc5da4c..39af17b 100644
--- a/fs/gfs2/bmap.c
+++ b/fs/gfs2/bmap.c
@@ -1472,7 +1472,7 @@ int gfs2_write_alloc_required(struct gfs2_inode *ip, u64 offset,
end_of_file = (i_size_read(&ip->i_inode) + sdp->sd_sb.sb_bsize - 1) >> shift;
lblock = offset >> shift;
lblock_stop = (offset + len + sdp->sd_sb.sb_bsize - 1) >> shift;
- if (lblock_stop > end_of_file)
+ if (lblock_stop > end_of_file && ip != GFS2_I(sdp->sd_rindex))
return 1;
size = (lblock_stop - lblock) << shift;
diff --git a/fs/gfs2/file.c b/fs/gfs2/file.c
index 39c382f..ff93e96 100644
--- a/fs/gfs2/file.c
+++ b/fs/gfs2/file.c
@@ -801,7 +801,7 @@ static long __gfs2_fallocate(struct file *file, int mode, loff_t offset, loff_t
struct gfs2_inode *ip = GFS2_I(inode);
struct gfs2_alloc_parms ap = { .aflags = 0, };
unsigned int data_blocks = 0, ind_blocks = 0, rblocks;
- loff_t bytes, max_bytes, max_blks = UINT_MAX;
+ loff_t bytes, max_bytes, max_blks;
int error;
const loff_t pos = offset;
const loff_t count = len;
@@ -853,7 +853,8 @@ static long __gfs2_fallocate(struct file *file, int mode, loff_t offset, loff_t
return error;
/* ap.allowed tells us how many blocks quota will allow
* us to write. Check if this reduces max_blks */
- if (ap.allowed && ap.allowed < max_blks)
+ max_blks = UINT_MAX;
+ if (ap.allowed)
max_blks = ap.allowed;
error = gfs2_inplace_reserve(ip, &ap);
diff --git a/fs/gfs2/inode.c b/fs/gfs2/inode.c
index fe3f849..bd6202b 100644
--- a/fs/gfs2/inode.c
+++ b/fs/gfs2/inode.c
@@ -740,17 +740,19 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
the gfs2 structures. */
if (default_acl) {
error = __gfs2_set_acl(inode, default_acl, ACL_TYPE_DEFAULT);
+ if (error)
+ goto fail_gunlock3;
posix_acl_release(default_acl);
+ default_acl = NULL;
}
if (acl) {
- if (!error)
- error = __gfs2_set_acl(inode, acl, ACL_TYPE_ACCESS);
+ error = __gfs2_set_acl(inode, acl, ACL_TYPE_ACCESS);
+ if (error)
+ goto fail_gunlock3;
posix_acl_release(acl);
+ acl = NULL;
}
- if (error)
- goto fail_gunlock3;
-
error = security_inode_init_security(&ip->i_inode, &dip->i_inode, name,
&gfs2_initxattrs, NULL);
if (error)
@@ -783,10 +785,8 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry,
gfs2_glock_put(ip->i_gl);
gfs2_rsqa_delete(ip, NULL);
fail_free_acls:
- if (default_acl)
- posix_acl_release(default_acl);
- if (acl)
- posix_acl_release(acl);
+ posix_acl_release(default_acl);
+ posix_acl_release(acl);
fail_gunlock:
gfs2_dir_no_add(&da);
gfs2_glock_dq_uninit(ghs);
diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c
index 77f1e25..ebea15c 100644
--- a/fs/gfs2/ops_fstype.c
+++ b/fs/gfs2/ops_fstype.c
@@ -71,13 +71,13 @@ static struct gfs2_sbd *init_sbd(struct super_block *sb)
if (!sdp)
return NULL;
- sb->s_fs_info = sdp;
sdp->sd_vfs = sb;
sdp->sd_lkstats = alloc_percpu(struct gfs2_pcpu_lkstats);
if (!sdp->sd_lkstats) {
kfree(sdp);
return NULL;
}
+ sb->s_fs_info = sdp;
set_bit(SDF_NOJOURNALID, &sdp->sd_flags);
gfs2_tune_init(&sdp->sd_tune);
@@ -1355,6 +1355,9 @@ static struct dentry *gfs2_mount_meta(struct file_system_type *fs_type,
struct path path;
int error;
+ if (!dev_name || !*dev_name)
+ return ERR_PTR(-EINVAL);
+
error = kern_path(dev_name, LOOKUP_FOLLOW, &path);
if (error) {
pr_warn("path_lookup on %s returned error %d\n",
diff --git a/fs/gfs2/quota.h b/fs/gfs2/quota.h
index 5e47c93..836f294 100644
--- a/fs/gfs2/quota.h
+++ b/fs/gfs2/quota.h
@@ -45,6 +45,8 @@ static inline int gfs2_quota_lock_check(struct gfs2_inode *ip,
{
struct gfs2_sbd *sdp = GFS2_SB(&ip->i_inode);
int ret;
+
+ ap->allowed = UINT_MAX; /* Assume we are permitted a whole lot */
if (sdp->sd_args.ar_quota == GFS2_QUOTA_OFF)
return 0;
ret = gfs2_quota_lock(ip, NO_UID_QUOTA_CHANGE, NO_GID_QUOTA_CHANGE);
diff --git a/fs/gfs2/rgrp.c b/fs/gfs2/rgrp.c
index 86ccc015..05f1ec7 100644
--- a/fs/gfs2/rgrp.c
+++ b/fs/gfs2/rgrp.c
@@ -715,6 +715,7 @@ void gfs2_clear_rgrpd(struct gfs2_sbd *sdp)
spin_lock(&gl->gl_lockref.lock);
gl->gl_object = NULL;
spin_unlock(&gl->gl_lockref.lock);
+ gfs2_rgrp_brelse(rgd);
gfs2_glock_add_to_lru(gl);
gfs2_glock_put(gl);
}
@@ -1125,7 +1126,7 @@ static u32 count_unlinked(struct gfs2_rgrpd *rgd)
* @rgd: the struct gfs2_rgrpd describing the RG to read in
*
* Read in all of a Resource Group's header and bitmap blocks.
- * Caller must eventually call gfs2_rgrp_relse() to free the bitmaps.
+ * Caller must eventually call gfs2_rgrp_brelse() to free the bitmaps.
*
* Returns: errno
*/
@@ -1675,7 +1676,8 @@ static int gfs2_rbm_find(struct gfs2_rbm *rbm, u8 state, u32 *minext,
while(1) {
bi = rbm_bi(rbm);
- if (test_bit(GBF_FULL, &bi->bi_flags) &&
+ if ((ip == NULL || !gfs2_rs_active(&ip->i_res)) &&
+ test_bit(GBF_FULL, &bi->bi_flags) &&
(state == GFS2_BLKST_FREE))
goto next_bitmap;
@@ -1703,9 +1705,9 @@ static int gfs2_rbm_find(struct gfs2_rbm *rbm, u8 state, u32 *minext,
goto next_iter;
}
if (ret == -E2BIG) {
+ n += rbm->bii - initial_bii;
rbm->bii = 0;
rbm->offset = 0;
- n += (rbm->bii - initial_bii);
goto res_covered_end_of_rgrp;
}
return ret;
diff --git a/fs/hfs/brec.c b/fs/hfs/brec.c
index 6fc766df..2e71367 100644
--- a/fs/hfs/brec.c
+++ b/fs/hfs/brec.c
@@ -74,9 +74,10 @@ int hfs_brec_insert(struct hfs_find_data *fd, void *entry, int entry_len)
if (!fd->bnode) {
if (!tree->root)
hfs_btree_inc_height(tree);
- fd->bnode = hfs_bnode_find(tree, tree->leaf_head);
- if (IS_ERR(fd->bnode))
- return PTR_ERR(fd->bnode);
+ node = hfs_bnode_find(tree, tree->leaf_head);
+ if (IS_ERR(node))
+ return PTR_ERR(node);
+ fd->bnode = node;
fd->record = -1;
}
new_node = NULL;
@@ -423,6 +424,10 @@ static int hfs_brec_update_parent(struct hfs_find_data *fd)
if (new_node) {
__be32 cnid;
+ if (!new_node->parent) {
+ hfs_btree_inc_height(tree);
+ new_node->parent = tree->root;
+ }
fd->bnode = hfs_bnode_find(tree, new_node->parent);
/* create index key and entry */
hfs_bnode_read_key(new_node, fd->search_key, 14);
diff --git a/fs/hfs/btree.c b/fs/hfs/btree.c
index 37cdd95..320f437 100644
--- a/fs/hfs/btree.c
+++ b/fs/hfs/btree.c
@@ -328,13 +328,14 @@ void hfs_bmap_free(struct hfs_bnode *node)
nidx -= len * 8;
i = node->next;
- hfs_bnode_put(node);
if (!i) {
/* panic */;
pr_crit("unable to free bnode %u. bmap not found!\n",
node->this);
+ hfs_bnode_put(node);
return;
}
+ hfs_bnode_put(node);
node = hfs_bnode_find(tree, i);
if (IS_ERR(node))
return;
diff --git a/fs/hfsplus/brec.c b/fs/hfsplus/brec.c
index 754fdf8..1002a0c 100644
--- a/fs/hfsplus/brec.c
+++ b/fs/hfsplus/brec.c
@@ -427,6 +427,10 @@ static int hfs_brec_update_parent(struct hfs_find_data *fd)
if (new_node) {
__be32 cnid;
+ if (!new_node->parent) {
+ hfs_btree_inc_height(tree);
+ new_node->parent = tree->root;
+ }
fd->bnode = hfs_bnode_find(tree, new_node->parent);
/* create index key and entry */
hfs_bnode_read_key(new_node, fd->search_key, 14);
diff --git a/fs/hfsplus/btree.c b/fs/hfsplus/btree.c
index d9d1a36..8d22564 100644
--- a/fs/hfsplus/btree.c
+++ b/fs/hfsplus/btree.c
@@ -453,14 +453,15 @@ void hfs_bmap_free(struct hfs_bnode *node)
nidx -= len * 8;
i = node->next;
- hfs_bnode_put(node);
if (!i) {
/* panic */;
pr_crit("unable to free bnode %u. "
"bmap not found!\n",
node->this);
+ hfs_bnode_put(node);
return;
}
+ hfs_bnode_put(node);
node = hfs_bnode_find(tree, i);
if (IS_ERR(node))
return;
diff --git a/fs/hfsplus/dir.c b/fs/hfsplus/dir.c
index 31d5e3f..193d5411 100644
--- a/fs/hfsplus/dir.c
+++ b/fs/hfsplus/dir.c
@@ -77,13 +77,13 @@ static struct dentry *hfsplus_lookup(struct inode *dir, struct dentry *dentry,
cpu_to_be32(HFSP_HARDLINK_TYPE) &&
entry.file.user_info.fdCreator ==
cpu_to_be32(HFSP_HFSPLUS_CREATOR) &&
+ HFSPLUS_SB(sb)->hidden_dir &&
(entry.file.create_date ==
HFSPLUS_I(HFSPLUS_SB(sb)->hidden_dir)->
create_date ||
entry.file.create_date ==
HFSPLUS_I(d_inode(sb->s_root))->
- create_date) &&
- HFSPLUS_SB(sb)->hidden_dir) {
+ create_date)) {
struct qstr str;
char name[32];
diff --git a/fs/hfsplus/super.c b/fs/hfsplus/super.c
index 11854dd..7fb976e 100644
--- a/fs/hfsplus/super.c
+++ b/fs/hfsplus/super.c
@@ -524,8 +524,10 @@ static int hfsplus_fill_super(struct super_block *sb, void *data, int silent)
goto out_put_root;
if (!hfs_brec_read(&fd, &entry, sizeof(entry))) {
hfs_find_exit(&fd);
- if (entry.type != cpu_to_be16(HFSPLUS_FOLDER))
+ if (entry.type != cpu_to_be16(HFSPLUS_FOLDER)) {
+ err = -EINVAL;
goto out_put_root;
+ }
inode = hfsplus_iget(sb, be32_to_cpu(entry.folder.id));
if (IS_ERR(inode)) {
err = PTR_ERR(inode);
@@ -588,6 +590,7 @@ static int hfsplus_fill_super(struct super_block *sb, void *data, int silent)
return 0;
out_put_hidden_dir:
+ cancel_delayed_work_sync(&sbi->sync_work);
iput(sbi->hidden_dir);
out_put_root:
dput(sb->s_root);
diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c
index 2c2f182..f53c139 100644
--- a/fs/hugetlbfs/inode.c
+++ b/fs/hugetlbfs/inode.c
@@ -118,6 +118,16 @@ static void huge_pagevec_release(struct pagevec *pvec)
pagevec_reinit(pvec);
}
+/*
+ * Mask used when checking the page offset value passed in via system
+ * calls. This value will be converted to a loff_t which is signed.
+ * Therefore, we want to check the upper PAGE_SHIFT + 1 bits of the
+ * value. The extra bit (- 1 in the shift value) is to take the sign
+ * bit into account.
+ */
+#define PGOFF_LOFFT_MAX \
+ (((1UL << (PAGE_SHIFT + 1)) - 1) << (BITS_PER_LONG - (PAGE_SHIFT + 1)))
+
static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
{
struct inode *inode = file_inode(file);
@@ -136,17 +146,31 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
vma->vm_flags |= VM_HUGETLB | VM_DONTEXPAND;
vma->vm_ops = &hugetlb_vm_ops;
+ /*
+ * page based offset in vm_pgoff could be sufficiently large to
+ * overflow a loff_t when converted to byte offset. This can
+ * only happen on architectures where sizeof(loff_t) ==
+ * sizeof(unsigned long). So, only check in those instances.
+ */
+ if (sizeof(unsigned long) == sizeof(loff_t)) {
+ if (vma->vm_pgoff & PGOFF_LOFFT_MAX)
+ return -EINVAL;
+ }
+
+ /* must be huge page aligned */
if (vma->vm_pgoff & (~huge_page_mask(h) >> PAGE_SHIFT))
return -EINVAL;
vma_len = (loff_t)(vma->vm_end - vma->vm_start);
+ len = vma_len + ((loff_t)vma->vm_pgoff << PAGE_SHIFT);
+ /* check for overflow */
+ if (len < vma_len)
+ return -EINVAL;
inode_lock(inode);
file_accessed(file);
ret = -ENOMEM;
- len = vma_len + ((loff_t)vma->vm_pgoff << PAGE_SHIFT);
-
if (hugetlb_reserve_pages(inode,
vma->vm_pgoff >> huge_page_order(h),
len >> huge_page_shift(h), vma,
@@ -155,7 +179,7 @@ static int hugetlbfs_file_mmap(struct file *file, struct vm_area_struct *vma)
ret = 0;
if (vma->vm_flags & VM_WRITE && inode->i_size < len)
- inode->i_size = len;
+ i_size_write(inode, len);
out:
inode_unlock(inode);
diff --git a/fs/inode.c b/fs/inode.c
index 3844c31..ef82a62 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -281,8 +281,17 @@ void drop_nlink(struct inode *inode)
{
WARN_ON(inode->i_nlink == 0);
inode->__i_nlink--;
- if (!inode->i_nlink)
+ if (!inode->i_nlink) {
atomic_long_inc(&inode->i_sb->s_remove_count);
+#if IS_ENABLED(CONFIG_FS_VERITY)
+ if (!list_empty(&inode->i_fsverity_list)) {
+ spin_lock(&inode->i_sb->s_inode_fsveritylist_lock);
+ list_del_init(&inode->i_fsverity_list);
+ spin_unlock(&inode->i_sb->s_inode_fsveritylist_lock);
+ iput(inode);
+ }
+#endif
+ }
}
EXPORT_SYMBOL(drop_nlink);
@@ -369,6 +378,9 @@ void inode_init_once(struct inode *inode)
INIT_LIST_HEAD(&inode->i_io_list);
INIT_LIST_HEAD(&inode->i_wb_list);
INIT_LIST_HEAD(&inode->i_lru);
+#if IS_ENABLED(CONFIG_FS_VERITY)
+ INIT_LIST_HEAD(&inode->i_fsverity_list);
+#endif
address_space_init_once(&inode->i_data);
i_size_ordered_init(inode);
#ifdef CONFIG_FSNOTIFY
@@ -2003,8 +2015,14 @@ void inode_init_owner(struct inode *inode, const struct inode *dir,
inode->i_uid = current_fsuid();
if (dir && dir->i_mode & S_ISGID) {
inode->i_gid = dir->i_gid;
+
+ /* Directories are special, and always inherit S_ISGID */
if (S_ISDIR(mode))
mode |= S_ISGID;
+ else if ((mode & (S_ISGID | S_IXGRP)) == (S_ISGID | S_IXGRP) &&
+ !in_group_p(inode->i_gid) &&
+ !capable_wrt_inode_uidgid(dir, CAP_FSETID))
+ mode &= ~S_ISGID;
} else
inode->i_gid = current_fsgid();
inode->i_mode = mode;
diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c
index 684996c..4d5a5a4 100644
--- a/fs/jbd2/checkpoint.c
+++ b/fs/jbd2/checkpoint.c
@@ -254,8 +254,8 @@ int jbd2_log_do_checkpoint(journal_t *journal)
bh = jh2bh(jh);
if (buffer_locked(bh)) {
- spin_unlock(&journal->j_list_lock);
get_bh(bh);
+ spin_unlock(&journal->j_list_lock);
wait_on_buffer(bh);
/* the journal_head may have gone by now */
BUFFER_TRACE(bh, "brelse");
@@ -336,8 +336,8 @@ int jbd2_log_do_checkpoint(journal_t *journal)
jh = transaction->t_checkpoint_io_list;
bh = jh2bh(jh);
if (buffer_locked(bh)) {
- spin_unlock(&journal->j_list_lock);
get_bh(bh);
+ spin_unlock(&journal->j_list_lock);
wait_on_buffer(bh);
/* the journal_head may have gone by now */
BUFFER_TRACE(bh, "brelse");
diff --git a/fs/jbd2/transaction.c b/fs/jbd2/transaction.c
index 4e5c610..b320c1b 100644
--- a/fs/jbd2/transaction.c
+++ b/fs/jbd2/transaction.c
@@ -528,6 +528,7 @@ int jbd2_journal_start_reserved(handle_t *handle, unsigned int type,
*/
ret = start_this_handle(journal, handle, GFP_NOFS);
if (ret < 0) {
+ handle->h_journal = journal;
jbd2_journal_free_reserved(handle);
return ret;
}
@@ -1352,6 +1353,13 @@ int jbd2_journal_dirty_metadata(handle_t *handle, struct buffer_head *bh)
if (jh->b_transaction == transaction &&
jh->b_jlist != BJ_Metadata) {
jbd_lock_bh_state(bh);
+ if (jh->b_transaction == transaction &&
+ jh->b_jlist != BJ_Metadata)
+ pr_err("JBD2: assertion failure: h_type=%u "
+ "h_line_no=%u block_no=%llu jlist=%u\n",
+ handle->h_type, handle->h_line_no,
+ (unsigned long long) bh->b_blocknr,
+ jh->b_jlist);
J_ASSERT_JH(jh, jh->b_transaction != transaction ||
jh->b_jlist == BJ_Metadata);
jbd_unlock_bh_state(bh);
@@ -1371,11 +1379,11 @@ int jbd2_journal_dirty_metadata(handle_t *handle, struct buffer_head *bh)
* of the transaction. This needs to be done
* once a transaction -bzzz
*/
- jh->b_modified = 1;
if (handle->h_buffer_credits <= 0) {
ret = -ENOSPC;
goto out_unlock_bh;
}
+ jh->b_modified = 1;
handle->h_buffer_credits--;
}
diff --git a/fs/jffs2/dir.c b/fs/jffs2/dir.c
index 0a754f3..e5a6deb 100644
--- a/fs/jffs2/dir.c
+++ b/fs/jffs2/dir.c
@@ -209,8 +209,7 @@ static int jffs2_create(struct inode *dir_i, struct dentry *dentry,
__func__, inode->i_ino, inode->i_mode, inode->i_nlink,
f->inocache->pino_nlink, inode->i_mapping->nrpages);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
return 0;
fail:
@@ -430,8 +429,7 @@ static int jffs2_symlink (struct inode *dir_i, struct dentry *dentry, const char
mutex_unlock(&dir_f->sem);
jffs2_complete_reservation(c);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
return 0;
fail:
@@ -575,8 +573,7 @@ static int jffs2_mkdir (struct inode *dir_i, struct dentry *dentry, umode_t mode
mutex_unlock(&dir_f->sem);
jffs2_complete_reservation(c);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
return 0;
fail:
@@ -747,8 +744,7 @@ static int jffs2_mknod (struct inode *dir_i, struct dentry *dentry, umode_t mode
mutex_unlock(&dir_f->sem);
jffs2_complete_reservation(c);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
return 0;
fail:
diff --git a/fs/jffs2/fs.c b/fs/jffs2/fs.c
index 567653f..c9c47d0 100644
--- a/fs/jffs2/fs.c
+++ b/fs/jffs2/fs.c
@@ -361,7 +361,6 @@ struct inode *jffs2_iget(struct super_block *sb, unsigned long ino)
ret = -EIO;
error:
mutex_unlock(&f->sem);
- jffs2_do_clear_inode(c, f);
iget_failed(inode);
return ERR_PTR(ret);
}
diff --git a/fs/jffs2/super.c b/fs/jffs2/super.c
index 59c019a..79b0de8 100644
--- a/fs/jffs2/super.c
+++ b/fs/jffs2/super.c
@@ -285,10 +285,8 @@ static int jffs2_fill_super(struct super_block *sb, void *data, int silent)
sb->s_fs_info = c;
ret = jffs2_parse_options(c, data);
- if (ret) {
- kfree(c);
+ if (ret)
return -EINVAL;
- }
/* Initialize JFFS2 superblock locks, the further initialization will
* be done later */
diff --git a/fs/jfs/namei.c b/fs/jfs/namei.c
index b41596d..56c3fcb 100644
--- a/fs/jfs/namei.c
+++ b/fs/jfs/namei.c
@@ -178,8 +178,7 @@ static int jfs_create(struct inode *dip, struct dentry *dentry, umode_t mode,
unlock_new_inode(ip);
iput(ip);
} else {
- unlock_new_inode(ip);
- d_instantiate(dentry, ip);
+ d_instantiate_new(dentry, ip);
}
out2:
@@ -313,8 +312,7 @@ static int jfs_mkdir(struct inode *dip, struct dentry *dentry, umode_t mode)
unlock_new_inode(ip);
iput(ip);
} else {
- unlock_new_inode(ip);
- d_instantiate(dentry, ip);
+ d_instantiate_new(dentry, ip);
}
out2:
@@ -1059,8 +1057,7 @@ static int jfs_symlink(struct inode *dip, struct dentry *dentry,
unlock_new_inode(ip);
iput(ip);
} else {
- unlock_new_inode(ip);
- d_instantiate(dentry, ip);
+ d_instantiate_new(dentry, ip);
}
out2:
@@ -1447,8 +1444,7 @@ static int jfs_mknod(struct inode *dir, struct dentry *dentry,
unlock_new_inode(ip);
iput(ip);
} else {
- unlock_new_inode(ip);
- d_instantiate(dentry, ip);
+ d_instantiate_new(dentry, ip);
}
out1:
diff --git a/fs/jfs/xattr.c b/fs/jfs/xattr.c
index c60f3d3..a679798 100644
--- a/fs/jfs/xattr.c
+++ b/fs/jfs/xattr.c
@@ -491,15 +491,17 @@ static int ea_get(struct inode *inode, struct ea_buffer *ea_buf, int min_size)
if (size > PSIZE) {
/*
* To keep the rest of the code simple. Allocate a
- * contiguous buffer to work with
+ * contiguous buffer to work with. Make the buffer large
+ * enough to make use of the whole extent.
*/
- ea_buf->xattr = kmalloc(size, GFP_KERNEL);
+ ea_buf->max_size = (size + sb->s_blocksize - 1) &
+ ~(sb->s_blocksize - 1);
+
+ ea_buf->xattr = kmalloc(ea_buf->max_size, GFP_KERNEL);
if (ea_buf->xattr == NULL)
return -ENOMEM;
ea_buf->flag = EA_MALLOC;
- ea_buf->max_size = (size + sb->s_blocksize - 1) &
- ~(sb->s_blocksize - 1);
if (ea_size == 0)
return 0;
diff --git a/fs/kernfs/symlink.c b/fs/kernfs/symlink.c
index 9b43ca0..80317b0 100644
--- a/fs/kernfs/symlink.c
+++ b/fs/kernfs/symlink.c
@@ -88,7 +88,7 @@ static int kernfs_get_target_path(struct kernfs_node *parent,
int slen = strlen(kn->name);
len -= slen;
- strncpy(s + len, kn->name, slen);
+ memcpy(s + len, kn->name, slen);
if (len)
s[--len] = '/';
diff --git a/fs/lockd/host.c b/fs/lockd/host.c
index d716c99..c7eb47f2 100644
--- a/fs/lockd/host.c
+++ b/fs/lockd/host.c
@@ -340,7 +340,7 @@ struct nlm_host *nlmsvc_lookup_host(const struct svc_rqst *rqstp,
};
struct lockd_net *ln = net_generic(net, lockd_net_id);
- dprintk("lockd: %s(host='%*s', vers=%u, proto=%s)\n", __func__,
+ dprintk("lockd: %s(host='%.*s', vers=%u, proto=%s)\n", __func__,
(int)hostname_len, hostname, rqstp->rq_vers,
(rqstp->rq_prot == IPPROTO_UDP ? "udp" : "tcp"));
diff --git a/fs/lockd/svc.c b/fs/lockd/svc.c
index 4d51259..d484c63 100644
--- a/fs/lockd/svc.c
+++ b/fs/lockd/svc.c
@@ -274,6 +274,8 @@ static void lockd_down_net(struct svc_serv *serv, struct net *net)
if (ln->nlmsvc_users) {
if (--ln->nlmsvc_users == 0) {
nlm_shutdown_hosts_net(net);
+ cancel_delayed_work_sync(&ln->grace_period_end);
+ locks_end_grace(&ln->lockd_manager);
svc_shutdown_net(serv, net);
dprintk("lockd_down_net: per-net data destroyed; net=%p\n", net);
}
diff --git a/fs/namei.c b/fs/namei.c
index c138ab1..d8f8584 100644
--- a/fs/namei.c
+++ b/fs/namei.c
@@ -907,6 +907,8 @@ static inline void put_link(struct nameidata *nd)
int sysctl_protected_symlinks __read_mostly = 0;
int sysctl_protected_hardlinks __read_mostly = 0;
+int sysctl_protected_fifos __read_mostly;
+int sysctl_protected_regular __read_mostly;
/**
* may_follow_link - Check symlink following for unsafe situations
@@ -1020,6 +1022,45 @@ static int may_linkat(struct path *link)
return -EPERM;
}
+/**
+ * may_create_in_sticky - Check whether an O_CREAT open in a sticky directory
+ * should be allowed, or not, on files that already
+ * exist.
+ * @dir: the sticky parent directory
+ * @inode: the inode of the file to open
+ *
+ * Block an O_CREAT open of a FIFO (or a regular file) when:
+ * - sysctl_protected_fifos (or sysctl_protected_regular) is enabled
+ * - the file already exists
+ * - we are in a sticky directory
+ * - we don't own the file
+ * - the owner of the directory doesn't own the file
+ * - the directory is world writable
+ * If the sysctl_protected_fifos (or sysctl_protected_regular) is set to 2
+ * the directory doesn't have to be world writable: being group writable will
+ * be enough.
+ *
+ * Returns 0 if the open is allowed, -ve on error.
+ */
+static int may_create_in_sticky(struct dentry * const dir,
+ struct inode * const inode)
+{
+ if ((!sysctl_protected_fifos && S_ISFIFO(inode->i_mode)) ||
+ (!sysctl_protected_regular && S_ISREG(inode->i_mode)) ||
+ likely(!(dir->d_inode->i_mode & S_ISVTX)) ||
+ uid_eq(inode->i_uid, dir->d_inode->i_uid) ||
+ uid_eq(current_fsuid(), inode->i_uid))
+ return 0;
+
+ if (likely(dir->d_inode->i_mode & 0002) ||
+ (dir->d_inode->i_mode & 0020 &&
+ ((sysctl_protected_fifos >= 2 && S_ISFIFO(inode->i_mode)) ||
+ (sysctl_protected_regular >= 2 && S_ISREG(inode->i_mode))))) {
+ return -EACCES;
+ }
+ return 0;
+}
+
static __always_inline
const char *get_link(struct nameidata *nd)
{
@@ -3392,9 +3433,15 @@ static int do_last(struct nameidata *nd,
if (error)
return error;
audit_inode(nd->name, nd->path.dentry, 0);
- error = -EISDIR;
- if ((open_flag & O_CREAT) && d_is_dir(nd->path.dentry))
- goto out;
+ if (open_flag & O_CREAT) {
+ error = -EISDIR;
+ if (d_is_dir(nd->path.dentry))
+ goto out;
+ error = may_create_in_sticky(dir,
+ d_backing_inode(nd->path.dentry));
+ if (unlikely(error))
+ goto out;
+ }
error = -ENOTDIR;
if ((nd->flags & LOOKUP_DIRECTORY) && !d_can_lookup(nd->path.dentry))
goto out;
diff --git a/fs/namespace.c b/fs/namespace.c
index 4628d08c..57ca4da 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -20,6 +20,7 @@
#include <linux/fs_struct.h> /* get_fs_root et.al. */
#include <linux/fsnotify.h> /* fsnotify_vfsmount_delete */
#include <linux/uaccess.h>
+#include <linux/file.h>
#include <linux/proc_ns.h>
#include <linux/magic.h>
#include <linux/bootmem.h>
@@ -605,12 +606,21 @@ int __legitimize_mnt(struct vfsmount *bastard, unsigned seq)
return 0;
mnt = real_mount(bastard);
mnt_add_count(mnt, 1);
+ smp_mb(); // see mntput_no_expire()
if (likely(!read_seqretry(&mount_lock, seq)))
return 0;
if (bastard->mnt_flags & MNT_SYNC_UMOUNT) {
mnt_add_count(mnt, -1);
return 1;
}
+ lock_mount_hash();
+ if (unlikely(bastard->mnt_flags & MNT_DOOMED)) {
+ mnt_add_count(mnt, -1);
+ unlock_mount_hash();
+ return 1;
+ }
+ unlock_mount_hash();
+ /* caller will mntput() */
return -1;
}
@@ -1154,15 +1164,36 @@ static void delayed_mntput(struct work_struct *unused)
}
static DECLARE_DELAYED_WORK(delayed_mntput_work, delayed_mntput);
+void flush_delayed_mntput_wait(void)
+{
+ delayed_mntput(NULL);
+ flush_delayed_work(&delayed_mntput_work);
+}
+
static void mntput_no_expire(struct mount *mnt)
{
rcu_read_lock();
- mnt_add_count(mnt, -1);
- if (likely(mnt->mnt_ns)) { /* shouldn't be the last one */
+ if (likely(READ_ONCE(mnt->mnt_ns))) {
+ /*
+ * Since we don't do lock_mount_hash() here,
+ * ->mnt_ns can change under us. However, if it's
+ * non-NULL, then there's a reference that won't
+ * be dropped until after an RCU delay done after
+ * turning ->mnt_ns NULL. So if we observe it
+ * non-NULL under rcu_read_lock(), the reference
+ * we are dropping is not the final one.
+ */
+ mnt_add_count(mnt, -1);
rcu_read_unlock();
return;
}
lock_mount_hash();
+ /*
+ * make sure that if __legitimize_mnt() has not seen us grab
+ * mount_lock, we'll see their refcount increment here.
+ */
+ smp_mb();
+ mnt_add_count(mnt, -1);
if (mnt_get_count(mnt)) {
rcu_read_unlock();
unlock_mount_hash();
@@ -1593,8 +1624,13 @@ static int do_umount(struct mount *mnt, int flags)
namespace_lock();
lock_mount_hash();
- event++;
+ /* Recheck MNT_LOCKED with the locks held */
+ retval = -EINVAL;
+ if (mnt->mnt.mnt_flags & MNT_LOCKED)
+ goto out;
+
+ event++;
if (flags & MNT_DETACH) {
if (!list_empty(&mnt->mnt_list))
umount_tree(mnt, UMOUNT_PROPAGATE);
@@ -1608,6 +1644,7 @@ static int do_umount(struct mount *mnt, int flags)
retval = 0;
}
}
+out:
unlock_mount_hash();
namespace_unlock();
return retval;
@@ -1679,6 +1716,7 @@ SYSCALL_DEFINE2(umount, char __user *, name, int, flags)
struct mount *mnt;
int retval;
int lookup_flags = 0;
+ bool user_request = !(current->flags & PF_KTHREAD);
if (flags & ~(MNT_FORCE | MNT_DETACH | MNT_EXPIRE | UMOUNT_NOFOLLOW))
return -EINVAL;
@@ -1698,17 +1736,41 @@ SYSCALL_DEFINE2(umount, char __user *, name, int, flags)
goto dput_and_out;
if (!check_mnt(mnt))
goto dput_and_out;
- if (mnt->mnt.mnt_flags & MNT_LOCKED)
+ if (mnt->mnt.mnt_flags & MNT_LOCKED) /* Check optimistically */
goto dput_and_out;
retval = -EPERM;
if (flags & MNT_FORCE && !capable(CAP_SYS_ADMIN))
goto dput_and_out;
+ /* flush delayed_fput to put mnt_count */
+ if (user_request)
+ flush_delayed_fput_wait();
+
retval = do_umount(mnt, flags);
dput_and_out:
/* we mustn't call path_put() as that would clear mnt_expiry_mark */
dput(path.dentry);
mntput_no_expire(mnt);
+
+ if (!user_request)
+ goto out;
+
+ if (!retval) {
+ /*
+ * If the last delayed_fput() is called during do_umount()
+ * and makes mnt_count zero, we need to guarantee to register
+ * delayed_mntput by waiting for delayed_fput work again.
+ */
+ flush_delayed_fput_wait();
+
+ /* flush delayed_mntput_work to put sb->s_active */
+ flush_delayed_mntput_wait();
+ }
+ if (!retval || (flags & MNT_FORCE)) {
+ /* filesystem needs to handle unclosed namespaces */
+ if (mnt->mnt.mnt_sb->s_op->umount_end)
+ mnt->mnt.mnt_sb->s_op->umount_end(mnt->mnt.mnt_sb, flags);
+ }
out:
return retval;
}
@@ -1776,8 +1838,14 @@ struct mount *copy_tree(struct mount *mnt, struct dentry *dentry,
for (s = r; s; s = next_mnt(s, r)) {
if (!(flag & CL_COPY_UNBINDABLE) &&
IS_MNT_UNBINDABLE(s)) {
- s = skip_mnt_tree(s);
- continue;
+ if (s->mnt.mnt_flags & MNT_LOCKED) {
+ /* Both unbindable and locked. */
+ q = ERR_PTR(-EPERM);
+ goto out;
+ } else {
+ s = skip_mnt_tree(s);
+ continue;
+ }
}
if (!(flag & CL_COPY_MNT_NS_FILE) &&
is_mnt_ns_file(s->mnt.mnt_root)) {
@@ -1830,7 +1898,7 @@ void drop_collected_mounts(struct vfsmount *mnt)
{
namespace_lock();
lock_mount_hash();
- umount_tree(real_mount(mnt), UMOUNT_SYNC);
+ umount_tree(real_mount(mnt), 0);
unlock_mount_hash();
namespace_unlock();
}
diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c
index a69ef4e..d6e4191 100644
--- a/fs/nfs/blocklayout/dev.c
+++ b/fs/nfs/blocklayout/dev.c
@@ -203,7 +203,7 @@ static bool bl_map_stripe(struct pnfs_block_dev *dev, u64 offset,
chunk = div_u64(offset, dev->chunk_size);
div_u64_rem(chunk, dev->nr_children, &chunk_idx);
- if (chunk_idx > dev->nr_children) {
+ if (chunk_idx >= dev->nr_children) {
dprintk("%s: invalid chunk idx %d (%lld/%lld)\n",
__func__, chunk_idx, offset, dev->chunk_size);
/* error, should not happen */
diff --git a/fs/nfs/callback_proc.c b/fs/nfs/callback_proc.c
index e9aa235e..9d75374 100644
--- a/fs/nfs/callback_proc.c
+++ b/fs/nfs/callback_proc.c
@@ -175,9 +175,9 @@ static u32 pnfs_check_callback_stateid(struct pnfs_layout_hdr *lo,
{
u32 oldseq, newseq;
- /* Is the stateid still not initialised? */
+ /* Is the stateid not initialised? */
if (!pnfs_layout_is_valid(lo))
- return NFS4ERR_DELAY;
+ return NFS4ERR_NOMATCHING_LAYOUT;
/* Mismatched stateid? */
if (!nfs4_stateid_match_other(&lo->plh_stateid, new))
@@ -402,11 +402,8 @@ validate_seqid(const struct nfs4_slot_table *tbl, const struct nfs4_slot *slot,
return htonl(NFS4ERR_SEQ_FALSE_RETRY);
}
- /* Wraparound */
- if (unlikely(slot->seq_nr == 0xFFFFFFFFU)) {
- if (args->csa_sequenceid == 1)
- return htonl(NFS4_OK);
- } else if (likely(args->csa_sequenceid == slot->seq_nr + 1))
+ /* Note: wraparound relies on seq_nr being of type u32 */
+ if (likely(args->csa_sequenceid == slot->seq_nr + 1))
return htonl(NFS4_OK);
/* Misordered request */
diff --git a/fs/nfs/callback_xdr.c b/fs/nfs/callback_xdr.c
index eb094c6..67903ee 100644
--- a/fs/nfs/callback_xdr.c
+++ b/fs/nfs/callback_xdr.c
@@ -968,16 +968,21 @@ static __be32 nfs4_callback_compound(struct svc_rqst *rqstp, void *argp, void *r
if (hdr_arg.minorversion == 0) {
cps.clp = nfs4_find_client_ident(SVC_NET(rqstp), hdr_arg.cb_ident);
- if (!cps.clp || !check_gss_callback_principal(cps.clp, rqstp))
+ if (!cps.clp || !check_gss_callback_principal(cps.clp, rqstp)) {
+ if (cps.clp)
+ nfs_put_client(cps.clp);
goto out_invalidcred;
+ }
}
cps.minorversion = hdr_arg.minorversion;
hdr_res.taglen = hdr_arg.taglen;
hdr_res.tag = hdr_arg.tag;
- if (encode_compound_hdr_res(&xdr_out, &hdr_res) != 0)
+ if (encode_compound_hdr_res(&xdr_out, &hdr_res) != 0) {
+ if (cps.clp)
+ nfs_put_client(cps.clp);
return rpc_system_err;
-
+ }
while (status == 0 && nops != hdr_arg.nops) {
status = process_op(nops, rqstp, &xdr_in,
argp, &xdr_out, resp, &cps);
diff --git a/fs/nfs/direct.c b/fs/nfs/direct.c
index 1ab9112..53f0012 100644
--- a/fs/nfs/direct.c
+++ b/fs/nfs/direct.c
@@ -98,8 +98,11 @@ struct nfs_direct_req {
struct pnfs_ds_commit_info ds_cinfo; /* Storage for cinfo */
struct work_struct work;
int flags;
+ /* for write */
#define NFS_ODIRECT_DO_COMMIT (1) /* an unstable reply was received */
#define NFS_ODIRECT_RESCHED_WRITES (2) /* write verification failed */
+ /* for read */
+#define NFS_ODIRECT_SHOULD_DIRTY (3) /* dirty user-space page after read */
struct nfs_writeverf verf; /* unstable write verifier */
};
@@ -422,7 +425,8 @@ static void nfs_direct_read_completion(struct nfs_pgio_header *hdr)
struct nfs_page *req = nfs_list_entry(hdr->pages.next);
struct page *page = req->wb_page;
- if (!PageCompound(page) && bytes < hdr->good_bytes)
+ if (!PageCompound(page) && bytes < hdr->good_bytes &&
+ (dreq->flags == NFS_ODIRECT_SHOULD_DIRTY))
set_page_dirty(page);
bytes += req->wb_bytes;
nfs_list_remove_request(req);
@@ -597,6 +601,9 @@ ssize_t nfs_file_direct_read(struct kiocb *iocb, struct iov_iter *iter)
if (!is_sync_kiocb(iocb))
dreq->iocb = iocb;
+ if (iter_is_iovec(iter))
+ dreq->flags = NFS_ODIRECT_SHOULD_DIRTY;
+
nfs_start_io_direct(inode);
NFS_I(inode)->read_io += count;
diff --git a/fs/nfs/nfs4client.c b/fs/nfs/nfs4client.c
index f6b0848..43f42cc 100644
--- a/fs/nfs/nfs4client.c
+++ b/fs/nfs/nfs4client.c
@@ -988,10 +988,10 @@ EXPORT_SYMBOL_GPL(nfs4_set_ds_client);
/*
* Session has been established, and the client marked ready.
- * Set the mount rsize and wsize with negotiated fore channel
- * attributes which will be bound checked in nfs_server_set_fsinfo.
+ * Limit the mount rsize, wsize and dtsize using negotiated fore
+ * channel attributes.
*/
-static void nfs4_session_set_rwsize(struct nfs_server *server)
+static void nfs4_session_limit_rwsize(struct nfs_server *server)
{
#ifdef CONFIG_NFS_V4_1
struct nfs4_session *sess;
@@ -1004,9 +1004,11 @@ static void nfs4_session_set_rwsize(struct nfs_server *server)
server_resp_sz = sess->fc_attrs.max_resp_sz - nfs41_maxread_overhead;
server_rqst_sz = sess->fc_attrs.max_rqst_sz - nfs41_maxwrite_overhead;
- if (!server->rsize || server->rsize > server_resp_sz)
+ if (server->dtsize > server_resp_sz)
+ server->dtsize = server_resp_sz;
+ if (server->rsize > server_resp_sz)
server->rsize = server_resp_sz;
- if (!server->wsize || server->wsize > server_rqst_sz)
+ if (server->wsize > server_rqst_sz)
server->wsize = server_rqst_sz;
#endif /* CONFIG_NFS_V4_1 */
}
@@ -1053,12 +1055,12 @@ static int nfs4_server_common_setup(struct nfs_server *server,
(unsigned long long) server->fsid.minor);
nfs_display_fhandle(mntfh, "Pseudo-fs root FH");
- nfs4_session_set_rwsize(server);
-
error = nfs_probe_fsinfo(server, mntfh, fattr);
if (error < 0)
goto out;
+ nfs4_session_limit_rwsize(server);
+
if (server->namelen == 0 || server->namelen > NFS4_MAXNAMLEN)
server->namelen = NFS4_MAXNAMLEN;
diff --git a/fs/nfs/nfs4idmap.c b/fs/nfs/nfs4idmap.c
index eaac878..b5f02f1 100644
--- a/fs/nfs/nfs4idmap.c
+++ b/fs/nfs/nfs4idmap.c
@@ -343,7 +343,7 @@ static ssize_t nfs_idmap_lookup_name(__u32 id, const char *type, char *buf,
int id_len;
ssize_t ret;
- id_len = snprintf(id_str, sizeof(id_str), "%u", id);
+ id_len = nfs_map_numeric_to_string(id, id_str, sizeof(id_str));
ret = nfs_idmap_get_key(id_str, id_len, type, buf, buflen, idmap);
if (ret < 0)
return -EINVAL;
@@ -626,7 +626,8 @@ static int nfs_idmap_read_and_verify_message(struct idmap_msg *im,
if (strcmp(upcall->im_name, im->im_name) != 0)
break;
/* Note: here we store the NUL terminator too */
- len = sprintf(id_str, "%d", im->im_id) + 1;
+ len = 1 + nfs_map_numeric_to_string(im->im_id, id_str,
+ sizeof(id_str));
ret = nfs_idmap_instantiate(key, authkey, id_str, len);
break;
case IDMAP_CONV_IDTONAME:
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 1b1b616..eb55ab6 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -541,8 +541,15 @@ nfs4_async_handle_exception(struct rpc_task *task, struct nfs_server *server,
ret = -EIO;
return ret;
out_retry:
- if (ret == 0)
+ if (ret == 0) {
exception->retry = 1;
+ /*
+ * For NFS4ERR_MOVED, the client transport will need to
+ * be recomputed after migration recovery has completed.
+ */
+ if (errorcode == -NFS4ERR_MOVED)
+ rpc_task_release_transport(task);
+ }
return ret;
}
@@ -1934,7 +1941,7 @@ static int nfs4_open_reclaim(struct nfs4_state_owner *sp, struct nfs4_state *sta
return ret;
}
-static int nfs4_handle_delegation_recall_error(struct nfs_server *server, struct nfs4_state *state, const nfs4_stateid *stateid, int err)
+static int nfs4_handle_delegation_recall_error(struct nfs_server *server, struct nfs4_state *state, const nfs4_stateid *stateid, struct file_lock *fl, int err)
{
switch (err) {
default:
@@ -1981,7 +1988,11 @@ static int nfs4_handle_delegation_recall_error(struct nfs_server *server, struct
return -EAGAIN;
case -ENOMEM:
case -NFS4ERR_DENIED:
- /* kill_proc(fl->fl_pid, SIGLOST, 1); */
+ if (fl) {
+ struct nfs4_lock_state *lsp = fl->fl_u.nfs4_fl.owner;
+ if (lsp)
+ set_bit(NFS_LOCK_LOST, &lsp->ls_flags);
+ }
return 0;
}
return err;
@@ -2017,7 +2028,7 @@ int nfs4_open_delegation_recall(struct nfs_open_context *ctx,
err = nfs4_open_recover_helper(opendata, FMODE_READ);
}
nfs4_opendata_put(opendata);
- return nfs4_handle_delegation_recall_error(server, state, stateid, err);
+ return nfs4_handle_delegation_recall_error(server, state, stateid, NULL, err);
}
static void nfs4_open_confirm_prepare(struct rpc_task *task, void *calldata)
@@ -2528,14 +2539,18 @@ static void nfs41_check_delegation_stateid(struct nfs4_state *state)
}
nfs4_stateid_copy(&stateid, &delegation->stateid);
- if (test_bit(NFS_DELEGATION_REVOKED, &delegation->flags) ||
- !test_and_clear_bit(NFS_DELEGATION_TEST_EXPIRED,
- &delegation->flags)) {
+ if (test_bit(NFS_DELEGATION_REVOKED, &delegation->flags)) {
rcu_read_unlock();
nfs_finish_clear_delegation_stateid(state, &stateid);
return;
}
+ if (!test_and_clear_bit(NFS_DELEGATION_TEST_EXPIRED,
+ &delegation->flags)) {
+ rcu_read_unlock();
+ return;
+ }
+
cred = get_rpccred(delegation->cred);
rcu_read_unlock();
status = nfs41_test_and_free_expired_stateid(server, &stateid, cred);
@@ -2697,7 +2712,7 @@ static int _nfs4_open_and_get_state(struct nfs4_opendata *opendata,
if (ret != 0)
goto out;
- state = nfs4_opendata_to_nfs4_state(opendata);
+ state = _nfs4_opendata_to_nfs4_state(opendata);
ret = PTR_ERR(state);
if (IS_ERR(state))
goto out;
@@ -2733,6 +2748,7 @@ static int _nfs4_open_and_get_state(struct nfs4_opendata *opendata,
nfs4_schedule_stateid_recovery(server, state);
}
out:
+ nfs4_sequence_free_slot(&opendata->o_res.seq_res);
return ret;
}
@@ -6499,7 +6515,7 @@ int nfs4_lock_delegation_recall(struct file_lock *fl, struct nfs4_state *state,
if (err != 0)
return err;
err = _nfs4_do_setlk(state, F_SETLK, fl, NFS_LOCK_NEW);
- return nfs4_handle_delegation_recall_error(server, state, stateid, err);
+ return nfs4_handle_delegation_recall_error(server, state, stateid, fl, err);
}
struct nfs_release_lockowner_data {
diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c
index 0bb0e62..857af95 100644
--- a/fs/nfs/nfs4state.c
+++ b/fs/nfs/nfs4state.c
@@ -1336,6 +1336,8 @@ int nfs4_schedule_stateid_recovery(const struct nfs_server *server, struct nfs4_
if (!nfs4_state_mark_reclaim_nograce(clp, state))
return -EBADF;
+ nfs_inode_find_delegation_state_and_recover(state->inode,
+ &state->stateid);
dprintk("%s: scheduling stateid recovery for server %s\n", __func__,
clp->cl_hostname);
nfs4_schedule_state_manager(clp);
@@ -1429,6 +1431,7 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
struct inode *inode = state->inode;
struct nfs_inode *nfsi = NFS_I(inode);
struct file_lock *fl;
+ struct nfs4_lock_state *lsp;
int status = 0;
struct file_lock_context *flctx = inode->i_flctx;
struct list_head *list;
@@ -1469,7 +1472,9 @@ static int nfs4_reclaim_locks(struct nfs4_state *state, const struct nfs4_state_
case -NFS4ERR_DENIED:
case -NFS4ERR_RECLAIM_BAD:
case -NFS4ERR_RECLAIM_CONFLICT:
- /* kill_proc(fl->fl_pid, SIGLOST, 1); */
+ lsp = fl->fl_u.nfs4_fl.owner;
+ if (lsp)
+ set_bit(NFS_LOCK_LOST, &lsp->ls_flags);
status = 0;
}
spin_lock(&flctx->flc_lock);
diff --git a/fs/nfs/nfs4sysctl.c b/fs/nfs/nfs4sysctl.c
index 8693d77..76241aa 100644
--- a/fs/nfs/nfs4sysctl.c
+++ b/fs/nfs/nfs4sysctl.c
@@ -31,7 +31,7 @@ static struct ctl_table nfs4_cb_sysctls[] = {
.data = &nfs_idmap_cache_timeout,
.maxlen = sizeof(int),
.mode = 0644,
- .proc_handler = proc_dointvec_jiffies,
+ .proc_handler = proc_dointvec,
},
{ }
};
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index eef0caf..66985a6 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -1016,6 +1016,9 @@ nfsd4_verify_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
{
__be32 status;
+ if (!cstate->save_fh.fh_dentry)
+ return nfserr_nofilehandle;
+
status = nfs4_preprocess_stateid_op(rqstp, cstate, &cstate->save_fh,
src_stateid, RD_STATE, src, NULL);
if (status) {
@@ -1725,6 +1728,7 @@ nfsd4_proc_compound(struct svc_rqst *rqstp,
if (status) {
op = &args->ops[0];
op->status = status;
+ resp->opcnt = 1;
goto encode_op;
}
diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c
index 2c4f7a2..b16a6c0 100644
--- a/fs/nfsd/nfs4xdr.c
+++ b/fs/nfsd/nfs4xdr.c
@@ -1536,6 +1536,8 @@ nfsd4_decode_getdeviceinfo(struct nfsd4_compoundargs *argp,
gdev->gd_maxcount = be32_to_cpup(p++);
num = be32_to_cpup(p++);
if (num) {
+ if (num > 1000)
+ goto xdr_error;
READ_BUF(4 * num);
gdev->gd_notify_types = be32_to_cpup(p++);
for (i = 1; i < num; i++) {
@@ -3638,7 +3640,8 @@ nfsd4_encode_readdir(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4
nfserr = nfserr_resource;
goto err_no_verf;
}
- maxcount = min_t(u32, readdir->rd_maxcount, INT_MAX);
+ maxcount = svc_max_payload(resp->rqstp);
+ maxcount = min_t(u32, readdir->rd_maxcount, maxcount);
/*
* Note the rfc defines rd_maxcount as the size of the
* READDIR4resok structure, which includes the verifier above
@@ -3652,7 +3655,7 @@ nfsd4_encode_readdir(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4
/* RFC 3530 14.2.24 allows us to ignore dircount when it's 0: */
if (!readdir->rd_dircount)
- readdir->rd_dircount = INT_MAX;
+ readdir->rd_dircount = svc_max_payload(resp->rqstp);
readdir->xdr = xdr;
readdir->rd_maxcount = maxcount;
diff --git a/fs/nilfs2/btree.c b/fs/nilfs2/btree.c
index 2e315f9..ac1ec8f 100644
--- a/fs/nilfs2/btree.c
+++ b/fs/nilfs2/btree.c
@@ -2158,8 +2158,8 @@ static void nilfs_btree_lookup_dirty_buffers(struct nilfs_bmap *btree,
pagevec_init(&pvec, 0);
- while (pagevec_lookup_tag(&pvec, btcache, &index, PAGECACHE_TAG_DIRTY,
- PAGEVEC_SIZE)) {
+ while (pagevec_lookup_tag(&pvec, btcache, &index,
+ PAGECACHE_TAG_DIRTY)) {
for (i = 0; i < pagevec_count(&pvec); i++) {
bh = head = page_buffers(pvec.pages[i]);
do {
diff --git a/fs/nilfs2/namei.c b/fs/nilfs2/namei.c
index 2b71c60..163131809 100644
--- a/fs/nilfs2/namei.c
+++ b/fs/nilfs2/namei.c
@@ -46,8 +46,7 @@ static inline int nilfs_add_nondir(struct dentry *dentry, struct inode *inode)
int err = nilfs_add_link(dentry, inode);
if (!err) {
- d_instantiate(dentry, inode);
- unlock_new_inode(inode);
+ d_instantiate_new(dentry, inode);
return 0;
}
inode_dec_link_count(inode);
@@ -243,8 +242,7 @@ static int nilfs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode)
goto out_fail;
nilfs_mark_inode_dirty(inode);
- d_instantiate(dentry, inode);
- unlock_new_inode(inode);
+ d_instantiate_new(dentry, inode);
out:
if (!err)
err = nilfs_transaction_commit(dir->i_sb);
diff --git a/fs/nilfs2/page.c b/fs/nilfs2/page.c
index f11a3ad..454ee52 100644
--- a/fs/nilfs2/page.c
+++ b/fs/nilfs2/page.c
@@ -257,8 +257,7 @@ int nilfs_copy_dirty_pages(struct address_space *dmap,
pagevec_init(&pvec, 0);
repeat:
- if (!pagevec_lookup_tag(&pvec, smap, &index, PAGECACHE_TAG_DIRTY,
- PAGEVEC_SIZE))
+ if (!pagevec_lookup_tag(&pvec, smap, &index, PAGECACHE_TAG_DIRTY))
return 0;
for (i = 0; i < pagevec_count(&pvec); i++) {
@@ -377,8 +376,8 @@ void nilfs_clear_dirty_pages(struct address_space *mapping, bool silent)
pagevec_init(&pvec, 0);
- while (pagevec_lookup_tag(&pvec, mapping, &index, PAGECACHE_TAG_DIRTY,
- PAGEVEC_SIZE)) {
+ while (pagevec_lookup_tag(&pvec, mapping, &index,
+ PAGECACHE_TAG_DIRTY)) {
for (i = 0; i < pagevec_count(&pvec); i++) {
struct page *page = pvec.pages[i];
diff --git a/fs/nilfs2/segment.c b/fs/nilfs2/segment.c
index 36362d4..b0eb58c 100644
--- a/fs/nilfs2/segment.c
+++ b/fs/nilfs2/segment.c
@@ -709,18 +709,14 @@ static size_t nilfs_lookup_dirty_data_buffers(struct inode *inode,
pagevec_init(&pvec, 0);
repeat:
if (unlikely(index > last) ||
- !pagevec_lookup_tag(&pvec, mapping, &index, PAGECACHE_TAG_DIRTY,
- min_t(pgoff_t, last - index,
- PAGEVEC_SIZE - 1) + 1))
+ !pagevec_lookup_range_tag(&pvec, mapping, &index, last,
+ PAGECACHE_TAG_DIRTY))
return ndirties;
for (i = 0; i < pagevec_count(&pvec); i++) {
struct buffer_head *bh, *head;
struct page *page = pvec.pages[i];
- if (unlikely(page->index > last))
- break;
-
lock_page(page);
if (!page_has_buffers(page))
create_empty_buffers(page, i_blocksize(inode), 0);
@@ -757,8 +753,8 @@ static void nilfs_lookup_dirty_node_buffers(struct inode *inode,
pagevec_init(&pvec, 0);
- while (pagevec_lookup_tag(&pvec, mapping, &index, PAGECACHE_TAG_DIRTY,
- PAGEVEC_SIZE)) {
+ while (pagevec_lookup_tag(&pvec, mapping, &index,
+ PAGECACHE_TAG_DIRTY)) {
for (i = 0; i < pagevec_count(&pvec); i++) {
bh = head = page_buffers(pvec.pages[i]);
do {
diff --git a/fs/ocfs2/acl.c b/fs/ocfs2/acl.c
index bed1fcb..ee8dbba 100644
--- a/fs/ocfs2/acl.c
+++ b/fs/ocfs2/acl.c
@@ -314,7 +314,9 @@ struct posix_acl *ocfs2_iop_get_acl(struct inode *inode, int type)
return ERR_PTR(ret);
}
+ down_read(&OCFS2_I(inode)->ip_xattr_sem);
acl = ocfs2_get_acl_nolock(inode, type, di_bh);
+ up_read(&OCFS2_I(inode)->ip_xattr_sem);
ocfs2_inode_unlock(inode, 0);
brelse(di_bh);
@@ -333,7 +335,9 @@ int ocfs2_acl_chmod(struct inode *inode, struct buffer_head *bh)
if (!(osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL))
return 0;
+ down_read(&OCFS2_I(inode)->ip_xattr_sem);
acl = ocfs2_get_acl_nolock(inode, ACL_TYPE_ACCESS, bh);
+ up_read(&OCFS2_I(inode)->ip_xattr_sem);
if (IS_ERR(acl) || !acl)
return PTR_ERR(acl);
ret = __posix_acl_chmod(&acl, GFP_KERNEL, inode->i_mode);
@@ -364,8 +368,10 @@ int ocfs2_init_acl(handle_t *handle,
if (!S_ISLNK(inode->i_mode)) {
if (osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL) {
+ down_read(&OCFS2_I(dir)->ip_xattr_sem);
acl = ocfs2_get_acl_nolock(dir, ACL_TYPE_DEFAULT,
dir_bh);
+ up_read(&OCFS2_I(dir)->ip_xattr_sem);
if (IS_ERR(acl))
return PTR_ERR(acl);
}
diff --git a/fs/ocfs2/aops.c b/fs/ocfs2/aops.c
index f2961b1..c26d046 100644
--- a/fs/ocfs2/aops.c
+++ b/fs/ocfs2/aops.c
@@ -134,6 +134,19 @@ static int ocfs2_symlink_get_block(struct inode *inode, sector_t iblock,
return err;
}
+static int ocfs2_lock_get_block(struct inode *inode, sector_t iblock,
+ struct buffer_head *bh_result, int create)
+{
+ int ret = 0;
+ struct ocfs2_inode_info *oi = OCFS2_I(inode);
+
+ down_read(&oi->ip_alloc_sem);
+ ret = ocfs2_get_block(inode, iblock, bh_result, create);
+ up_read(&oi->ip_alloc_sem);
+
+ return ret;
+}
+
int ocfs2_get_block(struct inode *inode, sector_t iblock,
struct buffer_head *bh_result, int create)
{
@@ -2120,7 +2133,7 @@ static void ocfs2_dio_free_write_ctx(struct inode *inode,
* called like this: dio->get_blocks(dio->inode, fs_startblk,
* fs_count, map_bh, dio->rw == WRITE);
*/
-static int ocfs2_dio_get_block(struct inode *inode, sector_t iblock,
+static int ocfs2_dio_wr_get_block(struct inode *inode, sector_t iblock,
struct buffer_head *bh_result, int create)
{
struct ocfs2_super *osb = OCFS2_SB(inode->i_sb);
@@ -2146,12 +2159,9 @@ static int ocfs2_dio_get_block(struct inode *inode, sector_t iblock,
* while file size will be changed.
*/
if (pos + total_len <= i_size_read(inode)) {
- down_read(&oi->ip_alloc_sem);
+
/* This is the fast path for re-write. */
- ret = ocfs2_get_block(inode, iblock, bh_result, create);
-
- up_read(&oi->ip_alloc_sem);
-
+ ret = ocfs2_lock_get_block(inode, iblock, bh_result, create);
if (buffer_mapped(bh_result) &&
!buffer_new(bh_result) &&
ret == 0)
@@ -2416,9 +2426,9 @@ static ssize_t ocfs2_direct_IO(struct kiocb *iocb, struct iov_iter *iter)
return 0;
if (iov_iter_rw(iter) == READ)
- get_block = ocfs2_get_block;
+ get_block = ocfs2_lock_get_block;
else
- get_block = ocfs2_dio_get_block;
+ get_block = ocfs2_dio_wr_get_block;
return __blockdev_direct_IO(iocb, inode, inode->i_sb->s_bdev,
iter, get_block,
diff --git a/fs/ocfs2/buffer_head_io.c b/fs/ocfs2/buffer_head_io.c
index 8f040f8..25c8b32 100644
--- a/fs/ocfs2/buffer_head_io.c
+++ b/fs/ocfs2/buffer_head_io.c
@@ -341,6 +341,7 @@ int ocfs2_read_blocks(struct ocfs2_caching_info *ci, u64 block, int nr,
* for this bh as it's not marked locally
* uptodate. */
status = -EIO;
+ clear_buffer_needs_validate(bh);
put_bh(bh);
bhs[i] = NULL;
continue;
diff --git a/fs/ocfs2/cluster/nodemanager.c b/fs/ocfs2/cluster/nodemanager.c
index b17d180..c204ac9b 100644
--- a/fs/ocfs2/cluster/nodemanager.c
+++ b/fs/ocfs2/cluster/nodemanager.c
@@ -40,6 +40,9 @@ char *o2nm_fence_method_desc[O2NM_FENCE_METHODS] = {
"panic", /* O2NM_FENCE_PANIC */
};
+static inline void o2nm_lock_subsystem(void);
+static inline void o2nm_unlock_subsystem(void);
+
struct o2nm_node *o2nm_get_node_by_num(u8 node_num)
{
struct o2nm_node *node = NULL;
@@ -181,7 +184,10 @@ static struct o2nm_cluster *to_o2nm_cluster_from_node(struct o2nm_node *node)
{
/* through the first node_set .parent
* mycluster/nodes/mynode == o2nm_cluster->o2nm_node_group->o2nm_node */
- return to_o2nm_cluster(node->nd_item.ci_parent->ci_parent);
+ if (node->nd_item.ci_parent)
+ return to_o2nm_cluster(node->nd_item.ci_parent->ci_parent);
+ else
+ return NULL;
}
enum {
@@ -194,7 +200,7 @@ static ssize_t o2nm_node_num_store(struct config_item *item, const char *page,
size_t count)
{
struct o2nm_node *node = to_o2nm_node(item);
- struct o2nm_cluster *cluster = to_o2nm_cluster_from_node(node);
+ struct o2nm_cluster *cluster;
unsigned long tmp;
char *p = (char *)page;
int ret = 0;
@@ -214,6 +220,13 @@ static ssize_t o2nm_node_num_store(struct config_item *item, const char *page,
!test_bit(O2NM_NODE_ATTR_PORT, &node->nd_set_attributes))
return -EINVAL; /* XXX */
+ o2nm_lock_subsystem();
+ cluster = to_o2nm_cluster_from_node(node);
+ if (!cluster) {
+ o2nm_unlock_subsystem();
+ return -EINVAL;
+ }
+
write_lock(&cluster->cl_nodes_lock);
if (cluster->cl_nodes[tmp])
ret = -EEXIST;
@@ -226,6 +239,8 @@ static ssize_t o2nm_node_num_store(struct config_item *item, const char *page,
set_bit(tmp, cluster->cl_nodes_bitmap);
}
write_unlock(&cluster->cl_nodes_lock);
+ o2nm_unlock_subsystem();
+
if (ret)
return ret;
@@ -269,7 +284,7 @@ static ssize_t o2nm_node_ipv4_address_store(struct config_item *item,
size_t count)
{
struct o2nm_node *node = to_o2nm_node(item);
- struct o2nm_cluster *cluster = to_o2nm_cluster_from_node(node);
+ struct o2nm_cluster *cluster;
int ret, i;
struct rb_node **p, *parent;
unsigned int octets[4];
@@ -286,6 +301,13 @@ static ssize_t o2nm_node_ipv4_address_store(struct config_item *item,
be32_add_cpu(&ipv4_addr, octets[i] << (i * 8));
}
+ o2nm_lock_subsystem();
+ cluster = to_o2nm_cluster_from_node(node);
+ if (!cluster) {
+ o2nm_unlock_subsystem();
+ return -EINVAL;
+ }
+
ret = 0;
write_lock(&cluster->cl_nodes_lock);
if (o2nm_node_ip_tree_lookup(cluster, ipv4_addr, &p, &parent))
@@ -298,6 +320,8 @@ static ssize_t o2nm_node_ipv4_address_store(struct config_item *item,
rb_insert_color(&node->nd_ip_node, &cluster->cl_node_ip_tree);
}
write_unlock(&cluster->cl_nodes_lock);
+ o2nm_unlock_subsystem();
+
if (ret)
return ret;
@@ -315,7 +339,7 @@ static ssize_t o2nm_node_local_store(struct config_item *item, const char *page,
size_t count)
{
struct o2nm_node *node = to_o2nm_node(item);
- struct o2nm_cluster *cluster = to_o2nm_cluster_from_node(node);
+ struct o2nm_cluster *cluster;
unsigned long tmp;
char *p = (char *)page;
ssize_t ret;
@@ -333,17 +357,26 @@ static ssize_t o2nm_node_local_store(struct config_item *item, const char *page,
!test_bit(O2NM_NODE_ATTR_PORT, &node->nd_set_attributes))
return -EINVAL; /* XXX */
+ o2nm_lock_subsystem();
+ cluster = to_o2nm_cluster_from_node(node);
+ if (!cluster) {
+ ret = -EINVAL;
+ goto out;
+ }
+
/* the only failure case is trying to set a new local node
* when a different one is already set */
if (tmp && tmp == cluster->cl_has_local &&
- cluster->cl_local_node != node->nd_num)
- return -EBUSY;
+ cluster->cl_local_node != node->nd_num) {
+ ret = -EBUSY;
+ goto out;
+ }
/* bring up the rx thread if we're setting the new local node. */
if (tmp && !cluster->cl_has_local) {
ret = o2net_start_listening(node);
if (ret)
- return ret;
+ goto out;
}
if (!tmp && cluster->cl_has_local &&
@@ -358,7 +391,11 @@ static ssize_t o2nm_node_local_store(struct config_item *item, const char *page,
cluster->cl_local_node = node->nd_num;
}
- return count;
+ ret = count;
+
+out:
+ o2nm_unlock_subsystem();
+ return ret;
}
CONFIGFS_ATTR(o2nm_node_, num);
@@ -738,6 +775,16 @@ static struct o2nm_cluster_group o2nm_cluster_group = {
},
};
+static inline void o2nm_lock_subsystem(void)
+{
+ mutex_lock(&o2nm_cluster_group.cs_subsys.su_mutex);
+}
+
+static inline void o2nm_unlock_subsystem(void)
+{
+ mutex_unlock(&o2nm_cluster_group.cs_subsys.su_mutex);
+}
+
int o2nm_depend_item(struct config_item *item)
{
return configfs_depend_item(&o2nm_cluster_group.cs_subsys, item);
diff --git a/fs/ocfs2/dir.c b/fs/ocfs2/dir.c
index 3ecb9f3..20e6104 100644
--- a/fs/ocfs2/dir.c
+++ b/fs/ocfs2/dir.c
@@ -1896,8 +1896,7 @@ static int ocfs2_dir_foreach_blk_el(struct inode *inode,
/* On error, skip the f_pos to the
next block. */
ctx->pos = (ctx->pos | (sb->s_blocksize - 1)) + 1;
- brelse(bh);
- continue;
+ break;
}
if (le64_to_cpu(de->inode)) {
unsigned char d_type = DT_UNKNOWN;
diff --git a/fs/ocfs2/dlm/dlmdomain.c b/fs/ocfs2/dlm/dlmdomain.c
index 733e4e7..73be0c6 100644
--- a/fs/ocfs2/dlm/dlmdomain.c
+++ b/fs/ocfs2/dlm/dlmdomain.c
@@ -675,20 +675,6 @@ static void dlm_leave_domain(struct dlm_ctxt *dlm)
spin_unlock(&dlm->spinlock);
}
-int dlm_shutting_down(struct dlm_ctxt *dlm)
-{
- int ret = 0;
-
- spin_lock(&dlm_domain_lock);
-
- if (dlm->dlm_state == DLM_CTXT_IN_SHUTDOWN)
- ret = 1;
-
- spin_unlock(&dlm_domain_lock);
-
- return ret;
-}
-
void dlm_unregister_domain(struct dlm_ctxt *dlm)
{
int leave = 0;
diff --git a/fs/ocfs2/dlm/dlmdomain.h b/fs/ocfs2/dlm/dlmdomain.h
index fd6122a..8a92814 100644
--- a/fs/ocfs2/dlm/dlmdomain.h
+++ b/fs/ocfs2/dlm/dlmdomain.h
@@ -28,7 +28,30 @@
extern spinlock_t dlm_domain_lock;
extern struct list_head dlm_domains;
-int dlm_shutting_down(struct dlm_ctxt *dlm);
+static inline int dlm_joined(struct dlm_ctxt *dlm)
+{
+ int ret = 0;
+
+ spin_lock(&dlm_domain_lock);
+ if (dlm->dlm_state == DLM_CTXT_JOINED)
+ ret = 1;
+ spin_unlock(&dlm_domain_lock);
+
+ return ret;
+}
+
+static inline int dlm_shutting_down(struct dlm_ctxt *dlm)
+{
+ int ret = 0;
+
+ spin_lock(&dlm_domain_lock);
+ if (dlm->dlm_state == DLM_CTXT_IN_SHUTDOWN)
+ ret = 1;
+ spin_unlock(&dlm_domain_lock);
+
+ return ret;
+}
+
void dlm_fire_domain_eviction_callbacks(struct dlm_ctxt *dlm,
int node_num);
diff --git a/fs/ocfs2/dlm/dlmmaster.c b/fs/ocfs2/dlm/dlmmaster.c
index 3f828a1..0cc30a5 100644
--- a/fs/ocfs2/dlm/dlmmaster.c
+++ b/fs/ocfs2/dlm/dlmmaster.c
@@ -589,9 +589,9 @@ static void dlm_init_lockres(struct dlm_ctxt *dlm,
res->last_used = 0;
- spin_lock(&dlm->spinlock);
+ spin_lock(&dlm->track_lock);
list_add_tail(&res->tracking, &dlm->tracking_list);
- spin_unlock(&dlm->spinlock);
+ spin_unlock(&dlm->track_lock);
memset(res->lvb, 0, DLM_LVB_LEN);
memset(res->refmap, 0, sizeof(res->refmap));
diff --git a/fs/ocfs2/dlm/dlmrecovery.c b/fs/ocfs2/dlm/dlmrecovery.c
index eef32482..844dc8d 100644
--- a/fs/ocfs2/dlm/dlmrecovery.c
+++ b/fs/ocfs2/dlm/dlmrecovery.c
@@ -1378,6 +1378,15 @@ int dlm_mig_lockres_handler(struct o2net_msg *msg, u32 len, void *data,
if (!dlm_grab(dlm))
return -EINVAL;
+ if (!dlm_joined(dlm)) {
+ mlog(ML_ERROR, "Domain %s not joined! "
+ "lockres %.*s, master %u\n",
+ dlm->name, mres->lockname_len,
+ mres->lockname, mres->master);
+ dlm_put(dlm);
+ return -EINVAL;
+ }
+
BUG_ON(!(mres->flags & (DLM_MRES_RECOVERY|DLM_MRES_MIGRATION)));
real_master = mres->master;
diff --git a/fs/ocfs2/dlmglue.c b/fs/ocfs2/dlmglue.c
index 785fcc2..5729d55 100644
--- a/fs/ocfs2/dlmglue.c
+++ b/fs/ocfs2/dlmglue.c
@@ -2599,6 +2599,10 @@ void ocfs2_inode_unlock_tracker(struct inode *inode,
struct ocfs2_lock_res *lockres;
lockres = &OCFS2_I(inode)->ip_inode_lockres;
+ /* had_lock means that the currect process already takes the cluster
+ * lock previously. If had_lock is 1, we have nothing to do here, and
+ * it will get unlocked where we got the lock.
+ */
if (!had_lock) {
ocfs2_remove_holder(lockres, oh);
ocfs2_inode_unlock(inode, ex);
diff --git a/fs/ocfs2/export.c b/fs/ocfs2/export.c
index 827fc98..3494e220 100644
--- a/fs/ocfs2/export.c
+++ b/fs/ocfs2/export.c
@@ -125,10 +125,10 @@ static struct dentry *ocfs2_get_dentry(struct super_block *sb,
check_gen:
if (handle->ih_generation != inode->i_generation) {
- iput(inode);
trace_ocfs2_get_dentry_generation((unsigned long long)blkno,
handle->ih_generation,
inode->i_generation);
+ iput(inode);
result = ERR_PTR(-ESTALE);
goto bail;
}
diff --git a/fs/ocfs2/journal.c b/fs/ocfs2/journal.c
index a244f14..fa947d3 100644
--- a/fs/ocfs2/journal.c
+++ b/fs/ocfs2/journal.c
@@ -666,23 +666,24 @@ static int __ocfs2_journal_access(handle_t *handle,
/* we can safely remove this assertion after testing. */
if (!buffer_uptodate(bh)) {
mlog(ML_ERROR, "giving me a buffer that's not uptodate!\n");
- mlog(ML_ERROR, "b_blocknr=%llu\n",
- (unsigned long long)bh->b_blocknr);
+ mlog(ML_ERROR, "b_blocknr=%llu, b_state=0x%lx\n",
+ (unsigned long long)bh->b_blocknr, bh->b_state);
lock_buffer(bh);
/*
- * A previous attempt to write this buffer head failed.
- * Nothing we can do but to retry the write and hope for
- * the best.
+ * A previous transaction with a couple of buffer heads fail
+ * to checkpoint, so all the bhs are marked as BH_Write_EIO.
+ * For current transaction, the bh is just among those error
+ * bhs which previous transaction handle. We can't just clear
+ * its BH_Write_EIO and reuse directly, since other bhs are
+ * not written to disk yet and that will cause metadata
+ * inconsistency. So we should set fs read-only to avoid
+ * further damage.
*/
if (buffer_write_io_error(bh) && !buffer_uptodate(bh)) {
- clear_buffer_write_io_error(bh);
- set_buffer_uptodate(bh);
- }
-
- if (!buffer_uptodate(bh)) {
unlock_buffer(bh);
- return -EIO;
+ return ocfs2_error(osb->sb, "A previous attempt to "
+ "write this buffer head failed\n");
}
unlock_buffer(bh);
}
diff --git a/fs/ocfs2/move_extents.c b/fs/ocfs2/move_extents.c
index 4e8f32eb..c179afd 100644
--- a/fs/ocfs2/move_extents.c
+++ b/fs/ocfs2/move_extents.c
@@ -156,18 +156,14 @@ static int __ocfs2_move_extent(handle_t *handle,
}
/*
- * lock allocators, and reserving appropriate number of bits for
- * meta blocks and data clusters.
- *
- * in some cases, we don't need to reserve clusters, just let data_ac
- * be NULL.
+ * lock allocator, and reserve appropriate number of bits for
+ * meta blocks.
*/
-static int ocfs2_lock_allocators_move_extents(struct inode *inode,
+static int ocfs2_lock_meta_allocator_move_extents(struct inode *inode,
struct ocfs2_extent_tree *et,
u32 clusters_to_move,
u32 extents_to_split,
struct ocfs2_alloc_context **meta_ac,
- struct ocfs2_alloc_context **data_ac,
int extra_blocks,
int *credits)
{
@@ -192,13 +188,6 @@ static int ocfs2_lock_allocators_move_extents(struct inode *inode,
goto out;
}
- if (data_ac) {
- ret = ocfs2_reserve_clusters(osb, clusters_to_move, data_ac);
- if (ret) {
- mlog_errno(ret);
- goto out;
- }
- }
*credits += ocfs2_calc_extend_credits(osb->sb, et->et_root_el);
@@ -260,10 +249,10 @@ static int ocfs2_defrag_extent(struct ocfs2_move_extents_context *context,
}
}
- ret = ocfs2_lock_allocators_move_extents(inode, &context->et, *len, 1,
- &context->meta_ac,
- &context->data_ac,
- extra_blocks, &credits);
+ ret = ocfs2_lock_meta_allocator_move_extents(inode, &context->et,
+ *len, 1,
+ &context->meta_ac,
+ extra_blocks, &credits);
if (ret) {
mlog_errno(ret);
goto out;
@@ -286,6 +275,21 @@ static int ocfs2_defrag_extent(struct ocfs2_move_extents_context *context,
}
}
+ /*
+ * Make sure ocfs2_reserve_cluster is called after
+ * __ocfs2_flush_truncate_log, otherwise, dead lock may happen.
+ *
+ * If ocfs2_reserve_cluster is called
+ * before __ocfs2_flush_truncate_log, dead lock on global bitmap
+ * may happen.
+ *
+ */
+ ret = ocfs2_reserve_clusters(osb, *len, &context->data_ac);
+ if (ret) {
+ mlog_errno(ret);
+ goto out_unlock_mutex;
+ }
+
handle = ocfs2_start_trans(osb, credits);
if (IS_ERR(handle)) {
ret = PTR_ERR(handle);
@@ -606,9 +610,10 @@ static int ocfs2_move_extent(struct ocfs2_move_extents_context *context,
}
}
- ret = ocfs2_lock_allocators_move_extents(inode, &context->et, len, 1,
- &context->meta_ac,
- NULL, extra_blocks, &credits);
+ ret = ocfs2_lock_meta_allocator_move_extents(inode, &context->et,
+ len, 1,
+ &context->meta_ac,
+ extra_blocks, &credits);
if (ret) {
mlog_errno(ret);
goto out;
diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c
index f56fe39..64dfbe5 100644
--- a/fs/ocfs2/super.c
+++ b/fs/ocfs2/super.c
@@ -473,9 +473,8 @@ static int ocfs2_init_global_system_inodes(struct ocfs2_super *osb)
new = ocfs2_get_system_file_inode(osb, i, osb->slot_num);
if (!new) {
ocfs2_release_system_inodes(osb);
- status = -EINVAL;
+ status = ocfs2_is_soft_readonly(osb) ? -EROFS : -EINVAL;
mlog_errno(status);
- /* FIXME: Should ERROR_RO_FS */
mlog(ML_ERROR, "Unable to load system inode %d, "
"possibly corrupt fs?", i);
goto bail;
@@ -504,7 +503,7 @@ static int ocfs2_init_local_system_inodes(struct ocfs2_super *osb)
new = ocfs2_get_system_file_inode(osb, i, osb->slot_num);
if (!new) {
ocfs2_release_system_inodes(osb);
- status = -EINVAL;
+ status = ocfs2_is_soft_readonly(osb) ? -EROFS : -EINVAL;
mlog(ML_ERROR, "status=%d, sysfile=%d, slot=%d\n",
status, i, osb->slot_num);
goto bail;
diff --git a/fs/ocfs2/xattr.c b/fs/ocfs2/xattr.c
index cb157a3..0193276 100644
--- a/fs/ocfs2/xattr.c
+++ b/fs/ocfs2/xattr.c
@@ -638,9 +638,11 @@ int ocfs2_calc_xattr_init(struct inode *dir,
si->value_len);
if (osb->s_mount_opt & OCFS2_MOUNT_POSIX_ACL) {
+ down_read(&OCFS2_I(dir)->ip_xattr_sem);
acl_len = ocfs2_xattr_get_nolock(dir, dir_bh,
OCFS2_XATTR_INDEX_POSIX_ACL_DEFAULT,
"", NULL, 0);
+ up_read(&OCFS2_I(dir)->ip_xattr_sem);
if (acl_len > 0) {
a_size = ocfs2_xattr_entry_real_size(0, acl_len);
if (S_ISDIR(mode))
@@ -1328,20 +1330,21 @@ static int ocfs2_xattr_get(struct inode *inode,
void *buffer,
size_t buffer_size)
{
- int ret;
+ int ret, had_lock;
struct buffer_head *di_bh = NULL;
+ struct ocfs2_lock_holder oh;
- ret = ocfs2_inode_lock(inode, &di_bh, 0);
- if (ret < 0) {
- mlog_errno(ret);
- return ret;
+ had_lock = ocfs2_inode_lock_tracker(inode, &di_bh, 0, &oh);
+ if (had_lock < 0) {
+ mlog_errno(had_lock);
+ return had_lock;
}
down_read(&OCFS2_I(inode)->ip_xattr_sem);
ret = ocfs2_xattr_get_nolock(inode, di_bh, name_index,
name, buffer, buffer_size);
up_read(&OCFS2_I(inode)->ip_xattr_sem);
- ocfs2_inode_unlock(inode, 0);
+ ocfs2_inode_unlock_tracker(inode, 0, &oh, had_lock);
brelse(di_bh);
@@ -3537,11 +3540,12 @@ int ocfs2_xattr_set(struct inode *inode,
{
struct buffer_head *di_bh = NULL;
struct ocfs2_dinode *di;
- int ret, credits, ref_meta = 0, ref_credits = 0;
+ int ret, credits, had_lock, ref_meta = 0, ref_credits = 0;
struct ocfs2_super *osb = OCFS2_SB(inode->i_sb);
struct inode *tl_inode = osb->osb_tl_inode;
struct ocfs2_xattr_set_ctxt ctxt = { NULL, NULL, NULL, };
struct ocfs2_refcount_tree *ref_tree = NULL;
+ struct ocfs2_lock_holder oh;
struct ocfs2_xattr_info xi = {
.xi_name_index = name_index,
@@ -3572,8 +3576,9 @@ int ocfs2_xattr_set(struct inode *inode,
return -ENOMEM;
}
- ret = ocfs2_inode_lock(inode, &di_bh, 1);
- if (ret < 0) {
+ had_lock = ocfs2_inode_lock_tracker(inode, &di_bh, 1, &oh);
+ if (had_lock < 0) {
+ ret = had_lock;
mlog_errno(ret);
goto cleanup_nolock;
}
@@ -3670,7 +3675,7 @@ int ocfs2_xattr_set(struct inode *inode,
if (ret)
mlog_errno(ret);
}
- ocfs2_inode_unlock(inode, 1);
+ ocfs2_inode_unlock_tracker(inode, 1, &oh, had_lock);
cleanup_nolock:
brelse(di_bh);
brelse(xbs.xattr_bh);
diff --git a/fs/orangefs/namei.c b/fs/orangefs/namei.c
index 7c31593..5fe4586 100644
--- a/fs/orangefs/namei.c
+++ b/fs/orangefs/namei.c
@@ -70,8 +70,7 @@ static int orangefs_create(struct inode *dir,
get_khandle_from_ino(inode),
dentry);
- d_instantiate(dentry, inode);
- unlock_new_inode(inode);
+ d_instantiate_new(dentry, inode);
orangefs_set_timeout(dentry);
ORANGEFS_I(inode)->getattr_time = jiffies - 1;
@@ -313,13 +312,19 @@ static int orangefs_symlink(struct inode *dir,
ret = PTR_ERR(inode);
goto out;
}
+ /*
+ * This is necessary because orangefs_inode_getattr will not
+ * re-read symlink size as it is impossible for it to change.
+ * Invalidating the cache does not help. orangefs_new_inode
+ * does not set the correct size (it does not know symname).
+ */
+ inode->i_size = strlen(symname);
gossip_debug(GOSSIP_NAME_DEBUG,
"Assigned symlink inode new number of %pU\n",
get_khandle_from_ino(inode));
- d_instantiate(dentry, inode);
- unlock_new_inode(inode);
+ d_instantiate_new(dentry, inode);
orangefs_set_timeout(dentry);
ORANGEFS_I(inode)->getattr_time = jiffies - 1;
@@ -382,8 +387,7 @@ static int orangefs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode
"Assigned dir inode new number of %pU\n",
get_khandle_from_ino(inode));
- d_instantiate(dentry, inode);
- unlock_new_inode(inode);
+ d_instantiate_new(dentry, inode);
orangefs_set_timeout(dentry);
ORANGEFS_I(inode)->getattr_time = jiffies - 1;
diff --git a/fs/orangefs/xattr.c b/fs/orangefs/xattr.c
index 237c9c0..a34b25b 100644
--- a/fs/orangefs/xattr.c
+++ b/fs/orangefs/xattr.c
@@ -76,7 +76,7 @@ ssize_t orangefs_inode_getxattr(struct inode *inode, const char *name,
if (S_ISLNK(inode->i_mode))
return -EOPNOTSUPP;
- if (strlen(name) > ORANGEFS_MAX_XATTR_NAMELEN)
+ if (strlen(name) >= ORANGEFS_MAX_XATTR_NAMELEN)
return -EINVAL;
fsuid = from_kuid(&init_user_ns, current_fsuid());
@@ -169,7 +169,7 @@ static int orangefs_inode_removexattr(struct inode *inode, const char *name,
struct orangefs_kernel_op_s *new_op = NULL;
int ret = -ENOMEM;
- if (strlen(name) > ORANGEFS_MAX_XATTR_NAMELEN)
+ if (strlen(name) >= ORANGEFS_MAX_XATTR_NAMELEN)
return -EINVAL;
down_write(&orangefs_inode->xattr_sem);
@@ -233,7 +233,7 @@ int orangefs_inode_setxattr(struct inode *inode, const char *name,
if (size > ORANGEFS_MAX_XATTR_VALUELEN)
return -EINVAL;
- if (strlen(name) > ORANGEFS_MAX_XATTR_NAMELEN)
+ if (strlen(name) >= ORANGEFS_MAX_XATTR_NAMELEN)
return -EINVAL;
internal_flag = convert_to_internal_xattr_flags(flags);
diff --git a/fs/overlayfs/Kconfig b/fs/overlayfs/Kconfig
index 3435581..887a7c9 100644
--- a/fs/overlayfs/Kconfig
+++ b/fs/overlayfs/Kconfig
@@ -8,3 +8,23 @@
merged with the 'upper' object.
For more information see Documentation/filesystems/overlayfs.txt
+
+config OVERLAY_FS_OVERRIDE_CREDS
+ bool "Overlay filesystem override credentials"
+ depends on OVERLAY_FS
+ default y
+ help
+ If set, all access to the upper, lower and work directories is the
+ recorded mounter's MAC and DAC credentials. The incoming accesses
+ are checked against the caller's credentials.
+
+ If the principles of least privilege are applied, the mounter's
+ credentials might not overlap the credentials of the caller's when
+ accessing the overlayfs filesystem. The mount option
+ "override_creds=off" drops the mounter's credential check, so that
+ all subsequent operations, after mount, on the filesystem will only
+ be the caller's credentials. This option sets the default for the
+ module option override_creds, and thus the default for all mounts
+ that do not specify this option.
+
+ For more information see Documentation/filesystems/overlayfs.txt
diff --git a/fs/overlayfs/copy_up.c b/fs/overlayfs/copy_up.c
index 36795ee..0b71cb5 100644
--- a/fs/overlayfs/copy_up.c
+++ b/fs/overlayfs/copy_up.c
@@ -421,7 +421,7 @@ int ovl_copy_up(struct dentry *dentry)
dput(parent);
dput(next);
}
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
return err;
}
diff --git a/fs/overlayfs/dir.c b/fs/overlayfs/dir.c
index 8546384..57764da 100644
--- a/fs/overlayfs/dir.c
+++ b/fs/overlayfs/dir.c
@@ -148,7 +148,7 @@ static int ovl_dir_getattr(struct vfsmount *mnt, struct dentry *dentry,
type = ovl_path_real(dentry, &realpath);
old_cred = ovl_override_creds(dentry->d_sb);
err = vfs_getattr(&realpath, stat);
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
if (err)
return err;
@@ -497,7 +497,8 @@ static int ovl_create_or_link(struct dentry *dentry, struct inode *inode,
override_cred->fsgid = inode->i_gid;
if (!hardlink) {
err = security_dentry_create_files_as(dentry,
- stat->mode, &dentry->d_name, old_cred,
+ stat->mode, &dentry->d_name,
+ old_cred ? old_cred : current_cred(),
override_cred);
if (err) {
put_cred(override_cred);
@@ -515,7 +516,7 @@ static int ovl_create_or_link(struct dentry *dentry, struct inode *inode,
link, hardlink);
}
out_revert_creds:
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
if (!err) {
struct inode *realinode = d_inode(ovl_dentry_upper(dentry));
@@ -757,7 +758,7 @@ static int ovl_do_remove(struct dentry *dentry, bool is_dir)
err = ovl_remove_upper(dentry, is_dir);
else
err = ovl_remove_and_whiteout(dentry, is_dir);
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
if (!err) {
if (is_dir)
clear_nlink(dentry->d_inode);
@@ -998,7 +999,7 @@ static int ovl_rename2(struct inode *olddir, struct dentry *old,
out_unlock:
unlock_rename(new_upperdir, old_upperdir);
out_revert_creds:
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
out_drop_write:
ovl_drop_write(old);
out:
diff --git a/fs/overlayfs/inode.c b/fs/overlayfs/inode.c
index 16f6db8..12c8e5e 100644
--- a/fs/overlayfs/inode.c
+++ b/fs/overlayfs/inode.c
@@ -34,7 +34,7 @@ static int ovl_copy_up_truncate(struct dentry *dentry)
stat.size = 0;
err = ovl_copy_up_one(parent, dentry, &lowerpath, &stat);
}
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
out_dput_parent:
dput(parent);
@@ -91,7 +91,7 @@ int ovl_setattr(struct dentry *dentry, struct iattr *attr)
inode_lock(upperdentry->d_inode);
old_cred = ovl_override_creds(dentry->d_sb);
err = notify_change(upperdentry, attr, NULL);
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
if (!err)
ovl_copyattr(upperdentry->d_inode, dentry->d_inode);
inode_unlock(upperdentry->d_inode);
@@ -115,7 +115,7 @@ static int ovl_getattr(struct vfsmount *mnt, struct dentry *dentry,
ovl_path_real(dentry, &realpath);
old_cred = ovl_override_creds(dentry->d_sb);
err = vfs_getattr(&realpath, stat);
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
return err;
}
@@ -147,7 +147,7 @@ int ovl_permission(struct inode *inode, int mask)
mask |= MAY_READ;
}
err = inode_permission(realinode, mask);
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
return err;
}
@@ -164,7 +164,7 @@ static const char *ovl_get_link(struct dentry *dentry,
old_cred = ovl_override_creds(dentry->d_sb);
p = vfs_get_link(ovl_dentry_real(dentry), done);
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
return p;
}
@@ -206,7 +206,7 @@ int ovl_xattr_set(struct dentry *dentry, const char *name, const void *value,
WARN_ON(flags != XATTR_REPLACE);
err = vfs_removexattr(realpath.dentry, name);
}
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
out_drop_write:
ovl_drop_write(dentry);
@@ -223,7 +223,7 @@ int ovl_xattr_get(struct dentry *dentry, const char *name,
old_cred = ovl_override_creds(dentry->d_sb);
res = vfs_getxattr(realdentry, name, value, size);
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
return res;
}
@@ -247,7 +247,7 @@ ssize_t ovl_listxattr(struct dentry *dentry, char *list, size_t size)
old_cred = ovl_override_creds(dentry->d_sb);
res = vfs_listxattr(realdentry, list, size);
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
if (res <= 0 || size == 0)
return res;
@@ -282,7 +282,7 @@ struct posix_acl *ovl_get_acl(struct inode *inode, int type)
old_cred = ovl_override_creds(inode->i_sb);
acl = get_acl(realinode, type);
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
return acl;
}
diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h
index e218e74..83f22da 100644
--- a/fs/overlayfs/overlayfs.h
+++ b/fs/overlayfs/overlayfs.h
@@ -164,6 +164,7 @@ bool ovl_dentry_is_opaque(struct dentry *dentry);
void ovl_dentry_set_opaque(struct dentry *dentry, bool opaque);
bool ovl_is_whiteout(struct dentry *dentry);
const struct cred *ovl_override_creds(struct super_block *sb);
+void ovl_revert_creds(const struct cred *oldcred);
void ovl_dentry_update(struct dentry *dentry, struct dentry *upperdentry);
void ovl_inode_update(struct inode *inode, struct inode *upperinode);
struct dentry *ovl_lookup(struct inode *dir, struct dentry *dentry,
diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c
index a1be6ba..4fcbfd0 100644
--- a/fs/overlayfs/readdir.c
+++ b/fs/overlayfs/readdir.c
@@ -223,7 +223,7 @@ static int ovl_check_whiteouts(struct dentry *dir, struct ovl_readdir_data *rdd)
}
inode_unlock(dir->d_inode);
}
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
return err;
}
diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c
index e7c8ac4..94ec5b4 100644
--- a/fs/overlayfs/super.c
+++ b/fs/overlayfs/super.c
@@ -31,6 +31,7 @@ struct ovl_config {
char *upperdir;
char *workdir;
bool default_permissions;
+ bool override_creds;
};
/* private information held for overlayfs's superblock */
@@ -266,9 +267,17 @@ const struct cred *ovl_override_creds(struct super_block *sb)
{
struct ovl_fs *ofs = sb->s_fs_info;
+ if (!ofs->config.override_creds)
+ return NULL;
return override_creds(ofs->creator_cred);
}
+void ovl_revert_creds(const struct cred *old_cred)
+{
+ if (old_cred)
+ revert_creds(old_cred);
+}
+
static bool ovl_is_opaquedir(struct dentry *dentry)
{
int res;
@@ -284,6 +293,12 @@ static bool ovl_is_opaquedir(struct dentry *dentry)
return false;
}
+static bool __read_mostly ovl_default_override_creds =
+ IS_ENABLED(CONFIG_OVERLAY_FS_OVERRIDE_CREDS);
+module_param_named(override_creds, ovl_default_override_creds, bool, 0644);
+MODULE_PARM_DESC(ovl_default_override_creds,
+ "Use mounter's credentials for accesses");
+
static void ovl_dentry_release(struct dentry *dentry)
{
struct ovl_entry *oe = dentry->d_fsdata;
@@ -584,7 +599,7 @@ struct dentry *ovl_lookup(struct inode *dir, struct dentry *dentry,
ovl_copyattr(realdentry->d_inode, inode);
}
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
oe->opaque = upperopaque;
oe->__upperdentry = upperdentry;
memcpy(oe->lowerstack, stack, sizeof(struct path) * ctr);
@@ -603,7 +618,7 @@ struct dentry *ovl_lookup(struct inode *dir, struct dentry *dentry,
out_put_upper:
dput(upperdentry);
out:
- revert_creds(old_cred);
+ ovl_revert_creds(old_cred);
return ERR_PTR(err);
}
@@ -626,7 +641,8 @@ static void ovl_put_super(struct super_block *sb)
kfree(ufs->config.lowerdir);
kfree(ufs->config.upperdir);
kfree(ufs->config.workdir);
- put_cred(ufs->creator_cred);
+ if (ufs->creator_cred)
+ put_cred(ufs->creator_cred);
kfree(ufs);
}
@@ -674,6 +690,8 @@ static int ovl_show_options(struct seq_file *m, struct dentry *dentry)
}
if (ufs->config.default_permissions)
seq_puts(m, ",default_permissions");
+ seq_show_option(m, "override_creds",
+ ufs->config.override_creds ? "on" : "off");
return 0;
}
@@ -700,6 +718,8 @@ enum {
OPT_UPPERDIR,
OPT_WORKDIR,
OPT_DEFAULT_PERMISSIONS,
+ OPT_OVERRIDE_CREDS_ON,
+ OPT_OVERRIDE_CREDS_OFF,
OPT_ERR,
};
@@ -708,6 +728,8 @@ static const match_table_t ovl_tokens = {
{OPT_UPPERDIR, "upperdir=%s"},
{OPT_WORKDIR, "workdir=%s"},
{OPT_DEFAULT_PERMISSIONS, "default_permissions"},
+ {OPT_OVERRIDE_CREDS_ON, "override_creds=on"},
+ {OPT_OVERRIDE_CREDS_OFF, "override_creds=off"},
{OPT_ERR, NULL}
};
@@ -738,6 +760,7 @@ static int ovl_parse_opt(char *opt, struct ovl_config *config)
{
char *p;
+ config->override_creds = ovl_default_override_creds;
while ((p = ovl_next_opt(&opt)) != NULL) {
int token;
substring_t args[MAX_OPT_ARGS];
@@ -772,6 +795,14 @@ static int ovl_parse_opt(char *opt, struct ovl_config *config)
config->default_permissions = true;
break;
+ case OPT_OVERRIDE_CREDS_ON:
+ config->override_creds = true;
+ break;
+
+ case OPT_OVERRIDE_CREDS_OFF:
+ config->override_creds = false;
+ break;
+
default:
pr_err("overlayfs: unrecognized mount option \"%s\" or missing value\n", p);
return -EINVAL;
diff --git a/fs/proc/array.c b/fs/proc/array.c
index 794b52a..94f83e7 100644
--- a/fs/proc/array.c
+++ b/fs/proc/array.c
@@ -80,6 +80,7 @@
#include <linux/delayacct.h>
#include <linux/seq_file.h>
#include <linux/pid_namespace.h>
+#include <linux/prctl.h>
#include <linux/ptrace.h>
#include <linux/tracehook.h>
#include <linux/string_helpers.h>
@@ -345,8 +346,32 @@ static inline void task_seccomp(struct seq_file *m, struct task_struct *p)
{
#ifdef CONFIG_SECCOMP
seq_put_decimal_ull(m, "Seccomp:\t", p->seccomp.mode);
- seq_putc(m, '\n');
#endif
+ seq_printf(m, "\nSpeculation_Store_Bypass:\t");
+ switch (arch_prctl_spec_ctrl_get(p, PR_SPEC_STORE_BYPASS)) {
+ case -EINVAL:
+ seq_printf(m, "unknown");
+ break;
+ case PR_SPEC_NOT_AFFECTED:
+ seq_printf(m, "not vulnerable");
+ break;
+ case PR_SPEC_PRCTL | PR_SPEC_FORCE_DISABLE:
+ seq_printf(m, "thread force mitigated");
+ break;
+ case PR_SPEC_PRCTL | PR_SPEC_DISABLE:
+ seq_printf(m, "thread mitigated");
+ break;
+ case PR_SPEC_PRCTL | PR_SPEC_ENABLE:
+ seq_printf(m, "thread vulnerable");
+ break;
+ case PR_SPEC_DISABLE:
+ seq_printf(m, "globally mitigated");
+ break;
+ default:
+ seq_printf(m, "vulnerable");
+ break;
+ }
+ seq_putc(m, '\n');
}
static inline void task_context_switch_counts(struct seq_file *m,
diff --git a/fs/proc/base.c b/fs/proc/base.c
index abe157a5..549839f 100644
--- a/fs/proc/base.c
+++ b/fs/proc/base.c
@@ -95,6 +95,8 @@
#include "internal.h"
#include "fd.h"
+#include "../../lib/kstrtox.h"
+
/* NOTE:
* Implementing inode permission operations in /proc is almost
* certainly an error. Permission checks need to happen during
@@ -253,7 +255,7 @@ static ssize_t proc_pid_cmdline_read(struct file *file, char __user *buf,
* Inherently racy -- command line shares address space
* with code and data.
*/
- rv = access_remote_vm(mm, arg_end - 1, &c, 1, 0);
+ rv = access_remote_vm(mm, arg_end - 1, &c, 1, FOLL_ANON);
if (rv <= 0)
goto out_free_page;
@@ -271,7 +273,7 @@ static ssize_t proc_pid_cmdline_read(struct file *file, char __user *buf,
int nr_read;
_count = min3(count, len, PAGE_SIZE);
- nr_read = access_remote_vm(mm, p, page, _count, 0);
+ nr_read = access_remote_vm(mm, p, page, _count, FOLL_ANON);
if (nr_read < 0)
rv = nr_read;
if (nr_read <= 0)
@@ -306,7 +308,7 @@ static ssize_t proc_pid_cmdline_read(struct file *file, char __user *buf,
bool final;
_count = min3(count, len, PAGE_SIZE);
- nr_read = access_remote_vm(mm, p, page, _count, 0);
+ nr_read = access_remote_vm(mm, p, page, _count, FOLL_ANON);
if (nr_read < 0)
rv = nr_read;
if (nr_read <= 0)
@@ -355,7 +357,7 @@ static ssize_t proc_pid_cmdline_read(struct file *file, char __user *buf,
bool final;
_count = min3(count, len, PAGE_SIZE);
- nr_read = access_remote_vm(mm, p, page, _count, 0);
+ nr_read = access_remote_vm(mm, p, page, _count, FOLL_ANON);
if (nr_read < 0)
rv = nr_read;
if (nr_read <= 0)
@@ -453,6 +455,20 @@ static int proc_pid_stack(struct seq_file *m, struct pid_namespace *ns,
int err;
int i;
+ /*
+ * The ability to racily run the kernel stack unwinder on a running task
+ * and then observe the unwinder output is scary; while it is useful for
+ * debugging kernel issues, it can also allow an attacker to leak kernel
+ * stack contents.
+ * Doing this in a manner that is at least safe from races would require
+ * some work to ensure that the remote task can not be scheduled; and
+ * even then, this would still expose the unwinder as local attack
+ * surface.
+ * Therefore, this interface is restricted to root.
+ */
+ if (!file_ns_capable(m->file, &init_user_ns, CAP_SYS_ADMIN))
+ return -EACCES;
+
entries = kmalloc(MAX_STACK_TRACE_DEPTH * sizeof(*entries), GFP_KERNEL);
if (!entries)
return -ENOMEM;
@@ -971,7 +987,7 @@ static ssize_t environ_read(struct file *file, char __user *buf,
max_len = min_t(size_t, PAGE_SIZE, count);
this_len = min(max_len, this_len);
- retval = access_remote_vm(mm, (env_start + src), page, this_len, 0);
+ retval = access_remote_vm(mm, (env_start + src), page, this_len, FOLL_ANON);
if (retval <= 0) {
ret = retval;
@@ -2063,8 +2079,33 @@ bool proc_fill_cache(struct file *file, struct dir_context *ctx,
static int dname_to_vma_addr(struct dentry *dentry,
unsigned long *start, unsigned long *end)
{
- if (sscanf(dentry->d_name.name, "%lx-%lx", start, end) != 2)
+ const char *str = dentry->d_name.name;
+ unsigned long long sval, eval;
+ unsigned int len;
+
+ len = _parse_integer(str, 16, &sval);
+ if (len & KSTRTOX_OVERFLOW)
return -EINVAL;
+ if (sval != (unsigned long)sval)
+ return -EINVAL;
+ str += len;
+
+ if (*str != '-')
+ return -EINVAL;
+ str++;
+
+ len = _parse_integer(str, 16, &eval);
+ if (len & KSTRTOX_OVERFLOW)
+ return -EINVAL;
+ if (eval != (unsigned long)eval)
+ return -EINVAL;
+ str += len;
+
+ if (*str != '\0')
+ return -EINVAL;
+
+ *start = sval;
+ *end = eval;
return 0;
}
@@ -3138,11 +3179,12 @@ static const struct pid_entry tgid_base_stuff[] = {
REG("mountinfo", S_IRUGO, proc_mountinfo_operations),
REG("mountstats", S_IRUSR, proc_mountstats_operations),
#ifdef CONFIG_PROCESS_RECLAIM
- REG("reclaim", S_IWUSR, proc_reclaim_operations),
+ REG("reclaim", S_IWUGO, proc_reclaim_operations),
#endif
#ifdef CONFIG_PROC_PAGE_MONITOR
REG("clear_refs", S_IWUSR, proc_clear_refs_operations),
REG("smaps", S_IRUGO, proc_pid_smaps_operations),
+ REG("smaps_rollup", S_IRUGO, proc_pid_smaps_rollup_operations),
REG("pagemap", S_IRUSR, proc_pagemap_operations),
#endif
#ifdef CONFIG_SECURITY
@@ -3201,6 +3243,8 @@ static const struct pid_entry tgid_base_stuff[] = {
REG("timerslack_ns", S_IRUGO|S_IWUGO, proc_pid_set_timerslack_ns_operations),
#ifdef CONFIG_CPU_FREQ_TIMES
ONE("time_in_state", 0444, proc_time_in_state_show),
+ ONE("concurrent_active_time", 0444, proc_concurrent_active_time_show),
+ ONE("concurrent_policy_time", 0444, proc_concurrent_policy_time_show),
#endif
};
@@ -3541,6 +3585,7 @@ static const struct pid_entry tid_base_stuff[] = {
#ifdef CONFIG_PROC_PAGE_MONITOR
REG("clear_refs", S_IWUSR, proc_clear_refs_operations),
REG("smaps", S_IRUGO, proc_tid_smaps_operations),
+ REG("smaps_rollup", S_IRUGO, proc_pid_smaps_rollup_operations),
REG("pagemap", S_IRUSR, proc_pagemap_operations),
#endif
#ifdef CONFIG_SECURITY
@@ -3592,6 +3637,8 @@ static const struct pid_entry tid_base_stuff[] = {
#endif
#ifdef CONFIG_CPU_FREQ_TIMES
ONE("time_in_state", 0444, proc_time_in_state_show),
+ ONE("concurrent_active_time", 0444, proc_concurrent_active_time_show),
+ ONE("concurrent_policy_time", 0444, proc_concurrent_policy_time_show),
#endif
};
diff --git a/fs/proc/inode.c b/fs/proc/inode.c
index e69ebe6..4298a39 100644
--- a/fs/proc/inode.c
+++ b/fs/proc/inode.c
@@ -43,10 +43,11 @@ static void proc_evict_inode(struct inode *inode)
de = PDE(inode);
if (de)
pde_put(de);
+
head = PROC_I(inode)->sysctl;
if (head) {
RCU_INIT_POINTER(PROC_I(inode)->sysctl, NULL);
- sysctl_head_put(head);
+ proc_sys_evict_inode(inode, head);
}
}
@@ -456,17 +457,12 @@ struct inode *proc_get_inode(struct super_block *sb, struct proc_dir_entry *de)
return inode;
}
-int proc_fill_super(struct super_block *s, void *data, int silent)
+int proc_fill_super(struct super_block *s)
{
- struct pid_namespace *ns = get_pid_ns(s->s_fs_info);
struct inode *root_inode;
int ret;
- if (!proc_parse_options(data, ns))
- return -EINVAL;
-
- /* User space would break if executables or devices appear on proc */
- s->s_iflags |= SB_I_USERNS_VISIBLE | SB_I_NOEXEC | SB_I_NODEV;
+ s->s_iflags |= SB_I_USERNS_VISIBLE | SB_I_NODEV;
s->s_flags |= MS_NODIRATIME | MS_NOSUID | MS_NOEXEC;
s->s_blocksize = 1024;
s->s_blocksize_bits = 10;
diff --git a/fs/proc/internal.h b/fs/proc/internal.h
index d8105cd..18e6170 100644
--- a/fs/proc/internal.h
+++ b/fs/proc/internal.h
@@ -65,6 +65,7 @@ struct proc_inode {
struct proc_dir_entry *pde;
struct ctl_table_header *sysctl;
struct ctl_table *sysctl_entry;
+ struct hlist_node sysctl_inodes;
const struct proc_ns_operations *ns_ops;
struct inode vfs_inode;
};
@@ -213,7 +214,7 @@ extern const struct file_operations proc_reclaim_operations;
extern void proc_init_inodecache(void);
extern struct inode *proc_get_inode(struct super_block *, struct proc_dir_entry *);
-extern int proc_fill_super(struct super_block *, void *data, int flags);
+extern int proc_fill_super(struct super_block *);
extern void proc_entry_rundown(struct proc_dir_entry *);
/*
@@ -250,10 +251,12 @@ extern void proc_thread_self_init(void);
*/
#ifdef CONFIG_PROC_SYSCTL
extern int proc_sys_init(void);
-extern void sysctl_head_put(struct ctl_table_header *);
+extern void proc_sys_evict_inode(struct inode *inode,
+ struct ctl_table_header *head);
#else
static inline void proc_sys_init(void) { }
-static inline void sysctl_head_put(struct ctl_table_header *head) { }
+static inline void proc_sys_evict_inode(struct inode *inode,
+ struct ctl_table_header *head) { }
#endif
/*
@@ -278,7 +281,6 @@ static inline void proc_tty_init(void) {}
* root.c
*/
extern struct proc_dir_entry proc_root;
-extern int proc_parse_options(char *options, struct pid_namespace *pid);
extern void proc_self_init(void);
extern int proc_remount(struct super_block *, int *, char *);
@@ -286,10 +288,12 @@ extern int proc_remount(struct super_block *, int *, char *);
/*
* task_[no]mmu.c
*/
+struct mem_size_stats;
struct proc_maps_private {
struct inode *inode;
struct task_struct *task;
struct mm_struct *mm;
+ struct mem_size_stats *rollup;
#ifdef CONFIG_MMU
struct vm_area_struct *tail_vma;
#endif
@@ -305,6 +309,7 @@ extern const struct file_operations proc_tid_maps_operations;
extern const struct file_operations proc_pid_numa_maps_operations;
extern const struct file_operations proc_tid_numa_maps_operations;
extern const struct file_operations proc_pid_smaps_operations;
+extern const struct file_operations proc_pid_smaps_rollup_operations;
extern const struct file_operations proc_tid_smaps_operations;
extern const struct file_operations proc_clear_refs_operations;
extern const struct file_operations proc_pagemap_operations;
diff --git a/fs/proc/kcore.c b/fs/proc/kcore.c
index df7e0798..7ed961c 100644
--- a/fs/proc/kcore.c
+++ b/fs/proc/kcore.c
@@ -505,6 +505,10 @@ read_kcore(struct file *file, char __user *buffer, size_t buflen, loff_t *fpos)
/* we have to zero-fill user buffer even if no read */
if (copy_to_user(buffer, buf, tsz))
return -EFAULT;
+ } else if (m->type == KCORE_USER) {
+ /* User page is handled prior to normal kernel page: */
+ if (copy_to_user(buffer, (char *)start, tsz))
+ return -EFAULT;
} else {
if (kern_addr_valid(start)) {
/*
diff --git a/fs/proc/loadavg.c b/fs/proc/loadavg.c
index aec66e6..e4b986e 100644
--- a/fs/proc/loadavg.c
+++ b/fs/proc/loadavg.c
@@ -3,13 +3,11 @@
#include <linux/pid_namespace.h>
#include <linux/proc_fs.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/seq_file.h>
#include <linux/seqlock.h>
#include <linux/time.h>
-#define LOAD_INT(x) ((x) >> FSHIFT)
-#define LOAD_FRAC(x) LOAD_INT(((x) & (FIXED_1-1)) * 100)
-
static int loadavg_proc_show(struct seq_file *m, void *v)
{
unsigned long avnrun[3];
diff --git a/fs/proc/meminfo.c b/fs/proc/meminfo.c
index 8a42849..df80e9a 100644
--- a/fs/proc/meminfo.c
+++ b/fs/proc/meminfo.c
@@ -115,6 +115,10 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
global_page_state(NR_SLAB_UNRECLAIMABLE));
seq_printf(m, "KernelStack: %8lu kB\n",
global_page_state(NR_KERNEL_STACK_KB));
+#ifdef CONFIG_SHADOW_CALL_STACK
+ seq_printf(m, "ShadowCallStack:%8lu kB\n",
+ global_page_state(NR_KERNEL_SCS_BYTES) / 1024);
+#endif
show_val_kb(m, "PageTables: ",
global_page_state(NR_PAGETABLE));
#ifdef CONFIG_QUICKLIST
@@ -125,6 +129,11 @@ static int meminfo_proc_show(struct seq_file *m, void *v)
global_node_page_state(NR_UNSTABLE_NFS));
show_val_kb(m, "Bounce: ",
global_page_state(NR_BOUNCE));
+ show_val_kb(m, "ION_heap: ",
+ global_node_page_state(NR_ION_HEAP));
+ show_val_kb(m, "ION_page_pool: ",
+ global_node_page_state(NR_INDIRECTLY_RECLAIMABLE_BYTES)
+ / 4096);
show_val_kb(m, "WritebackTmp: ",
global_node_page_state(NR_WRITEBACK_TEMP));
show_val_kb(m, "CommitLimit: ", vm_commit_limit());
diff --git a/fs/proc/proc_sysctl.c b/fs/proc/proc_sysctl.c
index d4e37ac..1999e85 100644
--- a/fs/proc/proc_sysctl.c
+++ b/fs/proc/proc_sysctl.c
@@ -190,6 +190,7 @@ static void init_header(struct ctl_table_header *head,
head->set = set;
head->parent = NULL;
head->node = node;
+ INIT_HLIST_HEAD(&head->inodes);
if (node) {
struct ctl_table *entry;
for (entry = table; entry->procname; entry++, node++)
@@ -259,6 +260,44 @@ static void unuse_table(struct ctl_table_header *p)
complete(p->unregistering);
}
+static void proc_sys_prune_dcache(struct ctl_table_header *head)
+{
+ struct inode *inode;
+ struct proc_inode *ei;
+ struct hlist_node *node;
+ struct super_block *sb;
+
+ rcu_read_lock();
+ for (;;) {
+ node = hlist_first_rcu(&head->inodes);
+ if (!node)
+ break;
+ ei = hlist_entry(node, struct proc_inode, sysctl_inodes);
+ spin_lock(&sysctl_lock);
+ hlist_del_init_rcu(&ei->sysctl_inodes);
+ spin_unlock(&sysctl_lock);
+
+ inode = &ei->vfs_inode;
+ sb = inode->i_sb;
+ if (!atomic_inc_not_zero(&sb->s_active))
+ continue;
+ inode = igrab(inode);
+ rcu_read_unlock();
+ if (unlikely(!inode)) {
+ deactivate_super(sb);
+ rcu_read_lock();
+ continue;
+ }
+
+ d_prune_aliases(inode);
+ iput(inode);
+ deactivate_super(sb);
+
+ rcu_read_lock();
+ }
+ rcu_read_unlock();
+}
+
/* called under sysctl_lock, will reacquire if has to wait */
static void start_unregistering(struct ctl_table_header *p)
{
@@ -272,33 +311,24 @@ static void start_unregistering(struct ctl_table_header *p)
p->unregistering = &wait;
spin_unlock(&sysctl_lock);
wait_for_completion(&wait);
- spin_lock(&sysctl_lock);
} else {
/* anything non-NULL; we'll never dereference it */
p->unregistering = ERR_PTR(-EINVAL);
+ spin_unlock(&sysctl_lock);
}
/*
+ * Prune dentries for unregistered sysctls: namespaced sysctls
+ * can have duplicate names and contaminate dcache very badly.
+ */
+ proc_sys_prune_dcache(p);
+ /*
* do not remove from the list until nobody holds it; walking the
* list in do_sysctl() relies on that.
*/
+ spin_lock(&sysctl_lock);
erase_header(p);
}
-static void sysctl_head_get(struct ctl_table_header *head)
-{
- spin_lock(&sysctl_lock);
- head->count++;
- spin_unlock(&sysctl_lock);
-}
-
-void sysctl_head_put(struct ctl_table_header *head)
-{
- spin_lock(&sysctl_lock);
- if (!--head->count)
- kfree_rcu(head, rcu);
- spin_unlock(&sysctl_lock);
-}
-
static struct ctl_table_header *sysctl_head_grab(struct ctl_table_header *head)
{
BUG_ON(!head);
@@ -436,14 +466,23 @@ static struct inode *proc_sys_make_inode(struct super_block *sb,
inode = new_inode(sb);
if (!inode)
- goto out;
+ return ERR_PTR(-ENOMEM);
inode->i_ino = get_next_ino();
- sysctl_head_get(head);
ei = PROC_I(inode);
+
+ spin_lock(&sysctl_lock);
+ if (unlikely(head->unregistering)) {
+ spin_unlock(&sysctl_lock);
+ iput(inode);
+ return ERR_PTR(-ENOENT);
+ }
ei->sysctl = head;
ei->sysctl_entry = table;
+ hlist_add_head_rcu(&ei->sysctl_inodes, &head->inodes);
+ head->count++;
+ spin_unlock(&sysctl_lock);
inode->i_mtime = inode->i_atime = inode->i_ctime = current_time(inode);
inode->i_mode = table->mode;
@@ -462,10 +501,18 @@ static struct inode *proc_sys_make_inode(struct super_block *sb,
if (root->set_ownership)
root->set_ownership(head, table, &inode->i_uid, &inode->i_gid);
-out:
return inode;
}
+void proc_sys_evict_inode(struct inode *inode, struct ctl_table_header *head)
+{
+ spin_lock(&sysctl_lock);
+ hlist_del_init_rcu(&PROC_I(inode)->sysctl_inodes);
+ if (!--head->count)
+ kfree_rcu(head, rcu);
+ spin_unlock(&sysctl_lock);
+}
+
static struct ctl_table_header *grab_header(struct inode *inode)
{
struct ctl_table_header *head = PROC_I(inode)->sysctl;
@@ -502,10 +549,11 @@ static struct dentry *proc_sys_lookup(struct inode *dir, struct dentry *dentry,
goto out;
}
- err = ERR_PTR(-ENOMEM);
inode = proc_sys_make_inode(dir->i_sb, h ? h : head, p);
- if (!inode)
+ if (IS_ERR(inode)) {
+ err = ERR_CAST(inode);
goto out;
+ }
err = NULL;
d_set_d_op(dentry, &proc_sys_dentry_operations);
@@ -638,7 +686,7 @@ static bool proc_sys_fill_cache(struct file *file,
return false;
if (d_in_lookup(child)) {
inode = proc_sys_make_inode(dir->d_sb, head, table);
- if (!inode) {
+ if (IS_ERR(inode)) {
d_lookup_done(child);
dput(child);
return false;
@@ -660,7 +708,10 @@ static bool proc_sys_link_fill_cache(struct file *file,
struct ctl_table *table)
{
bool ret = true;
+
head = sysctl_head_grab(head);
+ if (IS_ERR(head))
+ return false;
if (S_ISLNK(table->mode)) {
/* It is not an error if we can not follow the link ignore it */
diff --git a/fs/proc/root.c b/fs/proc/root.c
index c2f5014..1d68fcd 100644
--- a/fs/proc/root.c
+++ b/fs/proc/root.c
@@ -23,6 +23,21 @@
#include "internal.h"
+static int proc_test_super(struct super_block *sb, void *data)
+{
+ return sb->s_fs_info == data;
+}
+
+static int proc_set_super(struct super_block *sb, void *data)
+{
+ int err = set_anon_super(sb, NULL);
+ if (!err) {
+ struct pid_namespace *ns = (struct pid_namespace *)data;
+ sb->s_fs_info = get_pid_ns(ns);
+ }
+ return err;
+}
+
enum {
Opt_gid, Opt_hidepid, Opt_err,
};
@@ -33,7 +48,7 @@ static const match_table_t tokens = {
{Opt_err, NULL},
};
-int proc_parse_options(char *options, struct pid_namespace *pid)
+static int proc_parse_options(char *options, struct pid_namespace *pid)
{
char *p;
substring_t args[MAX_OPT_ARGS];
@@ -85,16 +100,45 @@ int proc_remount(struct super_block *sb, int *flags, char *data)
static struct dentry *proc_mount(struct file_system_type *fs_type,
int flags, const char *dev_name, void *data)
{
+ int err;
+ struct super_block *sb;
struct pid_namespace *ns;
+ char *options;
if (flags & MS_KERNMOUNT) {
- ns = data;
- data = NULL;
+ ns = (struct pid_namespace *)data;
+ options = NULL;
} else {
ns = task_active_pid_ns(current);
+ options = data;
+
+ /* Does the mounter have privilege over the pid namespace? */
+ if (!ns_capable(ns->user_ns, CAP_SYS_ADMIN))
+ return ERR_PTR(-EPERM);
}
- return mount_ns(fs_type, flags, data, ns, ns->user_ns, proc_fill_super);
+ sb = sget(fs_type, proc_test_super, proc_set_super, flags, ns);
+ if (IS_ERR(sb))
+ return ERR_CAST(sb);
+
+ if (!proc_parse_options(options, ns)) {
+ deactivate_locked_super(sb);
+ return ERR_PTR(-EINVAL);
+ }
+
+ if (!sb->s_root) {
+ err = proc_fill_super(sb);
+ if (err) {
+ deactivate_locked_super(sb);
+ return ERR_PTR(err);
+ }
+
+ sb->s_flags |= MS_ACTIVE;
+ /* User space would break if executables appear on proc */
+ sb->s_iflags |= SB_I_NOEXEC;
+ }
+
+ return dget(sb->s_root);
}
static void proc_kill_sb(struct super_block *sb)
diff --git a/fs/proc/stat.c b/fs/proc/stat.c
index d700c42..44475a4 100644
--- a/fs/proc/stat.c
+++ b/fs/proc/stat.c
@@ -21,23 +21,23 @@
#ifdef arch_idle_time
-static cputime64_t get_idle_time(int cpu)
+static u64 get_idle_time(int cpu)
{
- cputime64_t idle;
+ u64 idle;
idle = kcpustat_cpu(cpu).cpustat[CPUTIME_IDLE];
if (cpu_online(cpu) && !nr_iowait_cpu(cpu))
- idle += arch_idle_time(cpu);
+ idle += cputime_to_nsecs(arch_idle_time(cpu));
return idle;
}
-static cputime64_t get_iowait_time(int cpu)
+static u64 get_iowait_time(int cpu)
{
- cputime64_t iowait;
+ u64 iowait;
iowait = kcpustat_cpu(cpu).cpustat[CPUTIME_IOWAIT];
if (cpu_online(cpu) && nr_iowait_cpu(cpu))
- iowait += arch_idle_time(cpu);
+ iowait += cputime_to_nsecs(arch_idle_time(cpu));
return iowait;
}
@@ -45,32 +45,32 @@ static cputime64_t get_iowait_time(int cpu)
static u64 get_idle_time(int cpu)
{
- u64 idle, idle_time = -1ULL;
+ u64 idle, idle_usecs = -1ULL;
if (cpu_online(cpu))
- idle_time = get_cpu_idle_time_us(cpu, NULL);
+ idle_usecs = get_cpu_idle_time_us(cpu, NULL);
- if (idle_time == -1ULL)
+ if (idle_usecs == -1ULL)
/* !NO_HZ or cpu offline so we can rely on cpustat.idle */
idle = kcpustat_cpu(cpu).cpustat[CPUTIME_IDLE];
else
- idle = usecs_to_cputime64(idle_time);
+ idle = idle_usecs * NSEC_PER_USEC;
return idle;
}
static u64 get_iowait_time(int cpu)
{
- u64 iowait, iowait_time = -1ULL;
+ u64 iowait, iowait_usecs = -1ULL;
if (cpu_online(cpu))
- iowait_time = get_cpu_iowait_time_us(cpu, NULL);
+ iowait_usecs = get_cpu_iowait_time_us(cpu, NULL);
- if (iowait_time == -1ULL)
+ if (iowait_usecs == -1ULL)
/* !NO_HZ or cpu offline so we can rely on cpustat.iowait */
iowait = kcpustat_cpu(cpu).cpustat[CPUTIME_IOWAIT];
else
- iowait = usecs_to_cputime64(iowait_time);
+ iowait = iowait_usecs * NSEC_PER_USEC;
return iowait;
}
@@ -115,16 +115,16 @@ static int show_stat(struct seq_file *p, void *v)
}
sum += arch_irq_stat();
- seq_put_decimal_ull(p, "cpu ", cputime64_to_clock_t(user));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(nice));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(system));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(idle));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(iowait));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(irq));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(softirq));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(steal));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(guest));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(guest_nice));
+ seq_put_decimal_ull(p, "cpu ", nsec_to_clock_t(user));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(nice));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(system));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(idle));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(iowait));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(irq));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(softirq));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(steal));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(guest));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(guest_nice));
seq_putc(p, '\n');
for_each_online_cpu(i) {
@@ -140,16 +140,16 @@ static int show_stat(struct seq_file *p, void *v)
guest = kcpustat_cpu(i).cpustat[CPUTIME_GUEST];
guest_nice = kcpustat_cpu(i).cpustat[CPUTIME_GUEST_NICE];
seq_printf(p, "cpu%d", i);
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(user));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(nice));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(system));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(idle));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(iowait));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(irq));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(softirq));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(steal));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(guest));
- seq_put_decimal_ull(p, " ", cputime64_to_clock_t(guest_nice));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(user));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(nice));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(system));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(idle));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(iowait));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(irq));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(softirq));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(steal));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(guest));
+ seq_put_decimal_ull(p, " ", nsec_to_clock_t(guest_nice));
seq_putc(p, '\n');
}
seq_put_decimal_ull(p, "intr ", (unsigned long long)sum);
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index c585e7e..c07e4e6 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -16,7 +16,6 @@
#include <linux/page_idle.h>
#include <linux/shmem_fs.h>
#include <linux/mm_inline.h>
-#include <linux/ctype.h>
#include <asm/elf.h>
#include <asm/uaccess.h>
@@ -303,6 +302,7 @@ static int proc_map_release(struct inode *inode, struct file *file)
if (priv->mm)
mmdrop(priv->mm);
+ kfree(priv->rollup);
return seq_release_private(inode, file);
}
@@ -329,6 +329,23 @@ static int is_stack(struct proc_maps_private *priv,
vma->vm_end >= vma->vm_mm->start_stack;
}
+static void show_vma_header_prefix(struct seq_file *m,
+ unsigned long start, unsigned long end,
+ vm_flags_t flags, unsigned long long pgoff,
+ dev_t dev, unsigned long ino)
+{
+ seq_setwidth(m, 25 + sizeof(void *) * 6 - 1);
+ seq_printf(m, "%08lx-%08lx %c%c%c%c %08llx %02x:%02x %lu ",
+ start,
+ end,
+ flags & VM_READ ? 'r' : '-',
+ flags & VM_WRITE ? 'w' : '-',
+ flags & VM_EXEC ? 'x' : '-',
+ flags & VM_MAYSHARE ? 's' : 'p',
+ pgoff,
+ MAJOR(dev), MINOR(dev), ino);
+}
+
static void
show_map_vma(struct seq_file *m, struct vm_area_struct *vma, int is_pid)
{
@@ -352,17 +369,7 @@ show_map_vma(struct seq_file *m, struct vm_area_struct *vma, int is_pid)
/* We don't show the stack guard page in /proc/maps */
start = vma->vm_start;
end = vma->vm_end;
-
- seq_setwidth(m, 25 + sizeof(void *) * 6 - 1);
- seq_printf(m, "%08lx-%08lx %c%c%c%c %08llx %02x:%02x %lu ",
- start,
- end,
- flags & VM_READ ? 'r' : '-',
- flags & VM_WRITE ? 'w' : '-',
- flags & VM_EXEC ? 'x' : '-',
- flags & VM_MAYSHARE ? 's' : 'p',
- pgoff,
- MAJOR(dev), MINOR(dev), ino);
+ show_vma_header_prefix(m, start, end, flags, pgoff, dev, ino);
/*
* Print the dentry name for named mappings, and a
@@ -488,6 +495,7 @@ const struct file_operations proc_tid_maps_operations = {
#ifdef CONFIG_PROC_PAGE_MONITOR
struct mem_size_stats {
+ bool first;
unsigned long resident;
unsigned long shared_clean;
unsigned long shared_dirty;
@@ -500,7 +508,9 @@ struct mem_size_stats {
unsigned long swap;
unsigned long shared_hugetlb;
unsigned long private_hugetlb;
+ unsigned long first_vma_start;
u64 pss;
+ u64 pss_locked;
u64 swap_pss;
bool check_shmem_swap;
};
@@ -773,20 +783,40 @@ void __weak arch_show_smap(struct seq_file *m, struct vm_area_struct *vma)
static int show_smap(struct seq_file *m, void *v, int is_pid)
{
+ struct proc_maps_private *priv = m->private;
struct vm_area_struct *vma = v;
- struct mem_size_stats mss;
+ struct mem_size_stats mss_stack;
+ struct mem_size_stats *mss;
struct mm_walk smaps_walk = {
.pmd_entry = smaps_pte_range,
#ifdef CONFIG_HUGETLB_PAGE
.hugetlb_entry = smaps_hugetlb_range,
#endif
.mm = vma->vm_mm,
- .private = &mss,
};
+ int ret = 0;
+ bool rollup_mode;
+ bool last_vma;
- memset(&mss, 0, sizeof mss);
+ if (priv->rollup) {
+ rollup_mode = true;
+ mss = priv->rollup;
+ if (mss->first) {
+ mss->first_vma_start = vma->vm_start;
+ mss->first = false;
+ }
+ last_vma = !m_next_vma(priv, vma);
+ } else {
+ rollup_mode = false;
+ memset(&mss_stack, 0, sizeof(mss_stack));
+ mss = &mss_stack;
+ }
+
+ smaps_walk.private = mss;
#ifdef CONFIG_SHMEM
+ /* In case of smaps_rollup, reset the value from previous vma */
+ mss->check_shmem_swap = false;
if (vma->vm_file && shmem_mapping(vma->vm_file->f_mapping)) {
/*
* For shared or readonly shmem mappings we know that all
@@ -802,9 +832,9 @@ static int show_smap(struct seq_file *m, void *v, int is_pid)
if (!shmem_swapped || (vma->vm_flags & VM_SHARED) ||
!(vma->vm_flags & VM_WRITE)) {
- mss.swap = shmem_swapped;
+ mss->swap += shmem_swapped;
} else {
- mss.check_shmem_swap = true;
+ mss->check_shmem_swap = true;
smaps_walk.pte_hole = smaps_pte_hole;
}
}
@@ -812,58 +842,76 @@ static int show_smap(struct seq_file *m, void *v, int is_pid)
/* mmap_sem is held in m_start */
walk_page_vma(vma, &smaps_walk);
+ if (vma->vm_flags & VM_LOCKED)
+ mss->pss_locked += mss->pss;
- show_map_vma(m, vma, is_pid);
+ if (!rollup_mode) {
+ show_map_vma(m, vma, is_pid);
+ } else if (last_vma) {
+ show_vma_header_prefix(
+ m, mss->first_vma_start, vma->vm_end, 0, 0, 0, 0);
+ seq_pad(m, ' ');
+ seq_puts(m, "[rollup]\n");
+ } else {
+ ret = SEQ_SKIP;
+ }
- if (vma_get_anon_name(vma)) {
+ if (!rollup_mode && vma_get_anon_name(vma)) {
seq_puts(m, "Name: ");
seq_print_vma_name(m, vma);
seq_putc(m, '\n');
}
- seq_printf(m,
- "Size: %8lu kB\n"
- "Rss: %8lu kB\n"
- "Pss: %8lu kB\n"
- "Shared_Clean: %8lu kB\n"
- "Shared_Dirty: %8lu kB\n"
- "Private_Clean: %8lu kB\n"
- "Private_Dirty: %8lu kB\n"
- "Referenced: %8lu kB\n"
- "Anonymous: %8lu kB\n"
- "AnonHugePages: %8lu kB\n"
- "ShmemPmdMapped: %8lu kB\n"
- "Shared_Hugetlb: %8lu kB\n"
- "Private_Hugetlb: %7lu kB\n"
- "Swap: %8lu kB\n"
- "SwapPss: %8lu kB\n"
- "KernelPageSize: %8lu kB\n"
- "MMUPageSize: %8lu kB\n"
- "Locked: %8lu kB\n",
- (vma->vm_end - vma->vm_start) >> 10,
- mss.resident >> 10,
- (unsigned long)(mss.pss >> (10 + PSS_SHIFT)),
- mss.shared_clean >> 10,
- mss.shared_dirty >> 10,
- mss.private_clean >> 10,
- mss.private_dirty >> 10,
- mss.referenced >> 10,
- mss.anonymous >> 10,
- mss.anonymous_thp >> 10,
- mss.shmem_thp >> 10,
- mss.shared_hugetlb >> 10,
- mss.private_hugetlb >> 10,
- mss.swap >> 10,
- (unsigned long)(mss.swap_pss >> (10 + PSS_SHIFT)),
- vma_kernel_pagesize(vma) >> 10,
- vma_mmu_pagesize(vma) >> 10,
- (vma->vm_flags & VM_LOCKED) ?
- (unsigned long)(mss.pss >> (10 + PSS_SHIFT)) : 0);
+ if (!rollup_mode)
+ seq_printf(m,
+ "Size: %8lu kB\n"
+ "KernelPageSize: %8lu kB\n"
+ "MMUPageSize: %8lu kB\n",
+ (vma->vm_end - vma->vm_start) >> 10,
+ vma_kernel_pagesize(vma) >> 10,
+ vma_mmu_pagesize(vma) >> 10);
- arch_show_smap(m, vma);
- show_smap_vma_flags(m, vma);
+ if (!rollup_mode || last_vma)
+ seq_printf(m,
+ "Rss: %8lu kB\n"
+ "Pss: %8lu kB\n"
+ "Shared_Clean: %8lu kB\n"
+ "Shared_Dirty: %8lu kB\n"
+ "Private_Clean: %8lu kB\n"
+ "Private_Dirty: %8lu kB\n"
+ "Referenced: %8lu kB\n"
+ "Anonymous: %8lu kB\n"
+ "AnonHugePages: %8lu kB\n"
+ "ShmemPmdMapped: %8lu kB\n"
+ "Shared_Hugetlb: %8lu kB\n"
+ "Private_Hugetlb: %7lu kB\n"
+ "Swap: %8lu kB\n"
+ "SwapPss: %8lu kB\n"
+ "Locked: %8lu kB\n",
+ mss->resident >> 10,
+ (unsigned long)(mss->pss >> (10 + PSS_SHIFT)),
+ mss->shared_clean >> 10,
+ mss->shared_dirty >> 10,
+ mss->private_clean >> 10,
+ mss->private_dirty >> 10,
+ mss->referenced >> 10,
+ mss->anonymous >> 10,
+ mss->anonymous_thp >> 10,
+ mss->shmem_thp >> 10,
+ mss->shared_hugetlb >> 10,
+ mss->private_hugetlb >> 10,
+ mss->swap >> 10,
+ (unsigned long)(mss->swap_pss >> (10 + PSS_SHIFT)),
+ (unsigned long)(mss->pss_locked >> (10 + PSS_SHIFT)));
+
+
+ if (!rollup_mode) {
+ arch_show_smap(m, vma);
+ show_smap_vma_flags(m, vma);
+ }
+
m_cache_vma(m, vma);
- return 0;
+ return ret;
}
static int show_pid_smap(struct seq_file *m, void *v)
@@ -895,6 +943,25 @@ static int pid_smaps_open(struct inode *inode, struct file *file)
return do_maps_open(inode, file, &proc_pid_smaps_op);
}
+static int pid_smaps_rollup_open(struct inode *inode, struct file *file)
+{
+ struct seq_file *seq;
+ struct proc_maps_private *priv;
+ int ret = do_maps_open(inode, file, &proc_pid_smaps_op);
+
+ if (ret < 0)
+ return ret;
+ seq = file->private_data;
+ priv = seq->private;
+ priv->rollup = kzalloc(sizeof(*priv->rollup), GFP_KERNEL);
+ if (!priv->rollup) {
+ proc_map_release(inode, file);
+ return -ENOMEM;
+ }
+ priv->rollup->first = true;
+ return 0;
+}
+
static int tid_smaps_open(struct inode *inode, struct file *file)
{
return do_maps_open(inode, file, &proc_tid_smaps_op);
@@ -907,6 +974,13 @@ const struct file_operations proc_pid_smaps_operations = {
.release = proc_map_release,
};
+const struct file_operations proc_pid_smaps_rollup_operations = {
+ .open = pid_smaps_rollup_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = proc_map_release,
+};
+
const struct file_operations proc_tid_smaps_operations = {
.open = tid_smaps_open,
.read = seq_read,
@@ -1529,25 +1603,24 @@ const struct file_operations proc_pagemap_operations = {
#endif /* CONFIG_PROC_PAGE_MONITOR */
#ifdef CONFIG_PROCESS_RECLAIM
-static int reclaim_pte_range(pmd_t *pmd, unsigned long addr,
+enum reclaim_type {
+ RECLAIM_FILE,
+ RECLAIM_ANON,
+ RECLAIM_ALL,
+};
+
+static int reclaim_pmd_range(pmd_t *pmd, unsigned long addr,
unsigned long end, struct mm_walk *walk)
{
- struct reclaim_param *rp = walk->private;
- struct vm_area_struct *vma = rp->vma;
- pte_t *pte, ptent;
+ pte_t *orig_pte, *pte, ptent;
spinlock_t *ptl;
- struct page *page;
LIST_HEAD(page_list);
- int isolated;
- int reclaimed;
+ struct page *page;
+ int isolated = 0;
+ struct vm_area_struct *vma = walk->vma;
- split_huge_pmd(vma, addr, pmd);
- if (pmd_trans_unstable(pmd) || !rp->nr_to_reclaim)
- return 0;
-cont:
- isolated = 0;
- pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
- for (; addr != end; pte++, addr += PAGE_SIZE) {
+ orig_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
+ for (pte = orig_pte; addr < end; pte++, addr += PAGE_SIZE) {
ptent = *pte;
if (!pte_present(ptent))
continue;
@@ -1555,100 +1628,58 @@ static int reclaim_pte_range(pmd_t *pmd, unsigned long addr,
page = vm_normal_page(vma, addr, ptent);
if (!page)
continue;
+ /*
+ * XXX: we don't handle compound page at this moment but
+ * it should revisit for THP page before upstream.
+ */
+ if (PageCompound(page)) {
+ unsigned int order = compound_order(page);
+ unsigned int nr_pages = (1 << order) - 1;
+
+ addr += (nr_pages * PAGE_SIZE);
+ pte += nr_pages;
+ continue;
+ }
+
+ if (!PageLRU(page))
+ continue;
+
+ if (page_mapcount(page) > 1)
+ continue;
if (isolate_lru_page(page))
continue;
- list_add(&page->lru, &page_list);
- inc_node_page_state(page, NR_ISOLATED_ANON +
- page_is_file_cache(page));
isolated++;
- rp->nr_scanned++;
- if ((isolated >= SWAP_CLUSTER_MAX) || !rp->nr_to_reclaim)
- break;
+ list_add(&page->lru, &page_list);
+ if (isolated >= SWAP_CLUSTER_MAX) {
+ pte_unmap_unlock(orig_pte, ptl);
+ reclaim_pages(&page_list);
+ isolated = 0;
+ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl);
+ orig_pte = pte;
+ }
}
- pte_unmap_unlock(pte - 1, ptl);
- reclaimed = reclaim_pages_from_list(&page_list, vma);
- rp->nr_reclaimed += reclaimed;
- rp->nr_to_reclaim -= reclaimed;
- if (rp->nr_to_reclaim < 0)
- rp->nr_to_reclaim = 0;
- if (rp->nr_to_reclaim && (addr != end))
- goto cont;
+ pte_unmap_unlock(orig_pte, ptl);
+ reclaim_pages(&page_list);
cond_resched();
return 0;
}
-enum reclaim_type {
- RECLAIM_FILE,
- RECLAIM_ANON,
- RECLAIM_ALL,
- RECLAIM_RANGE,
-};
-
-struct reclaim_param reclaim_task_anon(struct task_struct *task,
- int nr_to_reclaim)
-{
- struct mm_struct *mm;
- struct vm_area_struct *vma;
- struct mm_walk reclaim_walk = {};
- struct reclaim_param rp;
-
- rp.nr_reclaimed = 0;
- rp.nr_scanned = 0;
- get_task_struct(task);
- mm = get_task_mm(task);
- if (!mm)
- goto out;
-
- reclaim_walk.mm = mm;
- reclaim_walk.pmd_entry = reclaim_pte_range;
-
- rp.nr_to_reclaim = nr_to_reclaim;
- reclaim_walk.private = &rp;
-
- down_read(&mm->mmap_sem);
- for (vma = mm->mmap; vma; vma = vma->vm_next) {
- if (is_vm_hugetlb_page(vma))
- continue;
-
- if (vma->vm_file)
- continue;
-
- if (vma->vm_flags & VM_LOCKED)
- continue;
-
- if (!rp.nr_to_reclaim)
- break;
-
- rp.vma = vma;
- walk_page_range(vma->vm_start, vma->vm_end,
- &reclaim_walk);
- }
-
- flush_tlb_mm(mm);
- up_read(&mm->mmap_sem);
- mmput(mm);
-out:
- put_task_struct(task);
- return rp;
-}
-
static ssize_t reclaim_write(struct file *file, const char __user *buf,
size_t count, loff_t *ppos)
{
struct task_struct *task;
- char buffer[200];
+ char buffer[PROC_NUMBUF];
struct mm_struct *mm;
struct vm_area_struct *vma;
enum reclaim_type type;
char *type_buf;
- struct mm_walk reclaim_walk = {};
- unsigned long start = 0;
- unsigned long end = 0;
- struct reclaim_param rp;
+
+ if (!capable(CAP_SYS_NICE))
+ return -EPERM;
memset(buffer, 0, sizeof(buffer));
if (count > sizeof(buffer) - 1)
@@ -1664,97 +1695,43 @@ static ssize_t reclaim_write(struct file *file, const char __user *buf,
type = RECLAIM_ANON;
else if (!strcmp(type_buf, "all"))
type = RECLAIM_ALL;
- else if (isdigit(*type_buf))
- type = RECLAIM_RANGE;
else
- goto out_err;
-
- if (type == RECLAIM_RANGE) {
- char *token;
- unsigned long long len, len_in, tmp;
- token = strsep(&type_buf, " ");
- if (!token)
- goto out_err;
- tmp = memparse(token, &token);
- if (tmp & ~PAGE_MASK || tmp > ULONG_MAX)
- goto out_err;
- start = tmp;
-
- token = strsep(&type_buf, " ");
- if (!token)
- goto out_err;
- len_in = memparse(token, &token);
- len = (len_in + ~PAGE_MASK) & PAGE_MASK;
- if (len > ULONG_MAX)
- goto out_err;
- /*
- * Check to see whether len was rounded up from small -ve
- * to zero.
- */
- if (len_in && !len)
- goto out_err;
-
- end = start + len;
- if (end < start)
- goto out_err;
- }
+ return -EINVAL;
task = get_proc_task(file->f_path.dentry->d_inode);
if (!task)
return -ESRCH;
mm = get_task_mm(task);
- if (!mm)
- goto out;
+ if (mm) {
+ struct mm_walk reclaim_walk = {
+ .pmd_entry = reclaim_pmd_range,
+ .mm = mm,
+ };
- reclaim_walk.mm = mm;
- reclaim_walk.pmd_entry = reclaim_pte_range;
-
- rp.nr_to_reclaim = INT_MAX;
- rp.nr_reclaimed = 0;
- reclaim_walk.private = &rp;
-
- down_read(&mm->mmap_sem);
- if (type == RECLAIM_RANGE) {
- vma = find_vma(mm, start);
- while (vma) {
- if (vma->vm_start > end)
- break;
- if (is_vm_hugetlb_page(vma))
- continue;
-
- rp.vma = vma;
- walk_page_range(max(vma->vm_start, start),
- min(vma->vm_end, end),
- &reclaim_walk);
- vma = vma->vm_next;
- }
- } else {
+ down_read(&mm->mmap_sem);
for (vma = mm->mmap; vma; vma = vma->vm_next) {
if (is_vm_hugetlb_page(vma))
continue;
- if (type == RECLAIM_ANON && vma->vm_file)
+ if (vma->vm_flags & VM_LOCKED)
continue;
- if (type == RECLAIM_FILE && !vma->vm_file)
+ if (type == RECLAIM_ANON && !vma_is_anonymous(vma))
+ continue;
+ if (type == RECLAIM_FILE && vma_is_anonymous(vma))
continue;
- rp.vma = vma;
walk_page_range(vma->vm_start, vma->vm_end,
- &reclaim_walk);
+ &reclaim_walk);
}
+ flush_tlb_mm(mm);
+ up_read(&mm->mmap_sem);
+ mmput(mm);
}
-
- flush_tlb_mm(mm);
- up_read(&mm->mmap_sem);
- mmput(mm);
-out:
put_task_struct(task);
- return count;
-out_err:
- return -EINVAL;
+ return count;
}
const struct file_operations proc_reclaim_operations = {
diff --git a/fs/proc/uptime.c b/fs/proc/uptime.c
index 33de567..7981c4f 100644
--- a/fs/proc/uptime.c
+++ b/fs/proc/uptime.c
@@ -5,23 +5,20 @@
#include <linux/seq_file.h>
#include <linux/time.h>
#include <linux/kernel_stat.h>
-#include <linux/cputime.h>
static int uptime_proc_show(struct seq_file *m, void *v)
{
struct timespec uptime;
struct timespec idle;
- u64 idletime;
u64 nsec;
u32 rem;
int i;
- idletime = 0;
+ nsec = 0;
for_each_possible_cpu(i)
- idletime += (__force u64) kcpustat_cpu(i).cpustat[CPUTIME_IDLE];
+ nsec += (__force u64) kcpustat_cpu(i).cpustat[CPUTIME_IDLE];
get_monotonic_boottime(&uptime);
- nsec = cputime64_to_jiffies64(idletime) * TICK_NSEC;
idle.tv_sec = div_u64_rem(nsec, NSEC_PER_SEC, &rem);
idle.tv_nsec = rem;
seq_printf(m, "%lu.%02lu %lu.%02lu\n",
diff --git a/fs/pstore/Kconfig b/fs/pstore/Kconfig
index be40813..f4fdb82 100644
--- a/fs/pstore/Kconfig
+++ b/fs/pstore/Kconfig
@@ -87,3 +87,9 @@
"ramoops.ko".
For more information, see Documentation/ramoops.txt.
+
+config BLDR_DEBUG_LOG
+ bool "Append bootloader log to ramoops"
+ depends on PSTORE_CONSOLE
+ help
+ Say Y here if you want to have bootloader log in ram console
diff --git a/fs/pstore/Makefile b/fs/pstore/Makefile
index b8803cc..ed7d912 100644
--- a/fs/pstore/Makefile
+++ b/fs/pstore/Makefile
@@ -11,3 +11,5 @@
ramoops-objs += ram.o ram_core.o
obj-$(CONFIG_PSTORE_RAM) += ramoops.o
+
+obj-$(CONFIG_BLDR_DEBUG_LOG) += bldr_log.o
diff --git a/fs/pstore/bldr_log.c b/fs/pstore/bldr_log.c
new file mode 100644
index 0000000..8d31041
--- /dev/null
+++ b/fs/pstore/bldr_log.c
@@ -0,0 +1,263 @@
+/* Copyright 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/module.h>
+#include <linux/fs.h>
+#include <linux/fsnotify.h>
+#include <linux/io.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/of_platform.h>
+#include <linux/of_fdt.h>
+#include <linux/bldr_debug_tools.h>
+#include <linux/uaccess.h>
+
+
+#define RAMLOG_COMPATIBLE_NAME "qcom,bldr_log"
+#define RAMLOG_LAST_RSE_NAME "bl_old_log"
+#define RAMLOG_CUR_RSE_NAME "bl_log"
+
+/* Header structure defined in LK */
+struct bldr_log_header {
+ __u32 magic;
+ __u32 offset;
+ __u32 rotate_flag;
+};
+
+static char *bl_last_log_buf, *bl_cur_log_buf;
+static unsigned long bl_last_log_buf_size, bl_cur_log_buf_size;
+
+static int bldr_log_check_header(struct bldr_log_header *header,
+ unsigned long bldr_log_size)
+{
+ /* Check magic first */
+ if (header->magic == BOOT_DEBUG_MAGIC) {
+ /* Check offset range */
+ if (header->offset >= sizeof(struct bldr_log_header) &&
+ header->offset < sizeof(struct bldr_log_header) +
+ bldr_log_size)
+ return 0;
+ }
+ return -EINVAL;
+}
+
+static void bldr_log_parser(const char *bldr_log, char *bldr_log_buf,
+ unsigned long bldr_log_size, unsigned long *bldr_log_buf_size)
+{
+ struct bldr_log_header *header = (struct bldr_log_header *)bldr_log;
+
+ if (bldr_log_check_header(header, bldr_log_size)) {
+ pr_warn("bldr_log_parser: bldr_log invalid in %p\n", bldr_log);
+ return;
+ }
+
+ if (header->rotate_flag) {
+ char *bldr_log_buf_ptr = bldr_log_buf;
+ unsigned long bldr_log_bottom_size, bldr_log_top_size;
+
+ /* Bottom half part */
+ bldr_log_bottom_size = bldr_log_size - header->offset;
+ memcpy(bldr_log_buf_ptr, bldr_log + header->offset,
+ bldr_log_bottom_size);
+ bldr_log_buf_ptr += bldr_log_bottom_size;
+
+ /* Top half part */
+ bldr_log_top_size = header->offset - sizeof(*header);
+ memcpy(bldr_log_buf_ptr, bldr_log + sizeof(*header),
+ bldr_log_top_size);
+
+ *bldr_log_buf_size = bldr_log_bottom_size + bldr_log_top_size;
+ } else {
+ *bldr_log_buf_size = header->offset - sizeof(*header);
+
+ memcpy(bldr_log_buf, bldr_log + sizeof(*header),
+ *bldr_log_buf_size);
+ }
+
+ pr_debug("bldr_log_parser: size %ld\n", *bldr_log_buf_size);
+}
+
+ssize_t bldr_last_log_read_once(char __user *userbuf, ssize_t klog_size)
+{
+ ssize_t len = 0;
+
+ len = (klog_size > bl_last_log_buf_size) ?
+ (ssize_t)bl_last_log_buf_size : 0;
+
+ if (len > 0) {
+ if (copy_to_user(userbuf, bl_last_log_buf, len)) {
+ pr_warn("bldr_last_log_read_once, copy_to_user failed\n");
+ return -EFAULT;
+ }
+ }
+ return len;
+}
+
+ssize_t bldr_log_read_once(char __user *userbuf, ssize_t klog_size)
+{
+ ssize_t len = 0;
+
+ len = (klog_size > bl_cur_log_buf_size) ?
+ (ssize_t)bl_cur_log_buf_size : 0;
+
+ if (len > 0) {
+ if (copy_to_user(userbuf, bl_cur_log_buf, len)) {
+ pr_warn("bldr_log_read_once, copy_to_user failed\n");
+ return -EFAULT;
+ }
+ }
+ return len;
+}
+
+/**
+ * Read last bootloader logs, kernel logs, current bootloader logs in order.
+ *
+ * Handle reads that overlap different regions so the file appears like one
+ * contiguous file to the reader.
+ */
+ssize_t bldr_log_read(const void *lastk_buf, ssize_t lastk_size,
+ char __user *userbuf, size_t count, loff_t *ppos)
+{
+ loff_t pos;
+ ssize_t total_len = 0;
+ ssize_t len;
+ int i;
+
+ struct {
+ const char *buf;
+ const ssize_t size;
+ } log_regions[] = {
+ { .buf = bl_last_log_buf, .size = bl_last_log_buf_size },
+ { .buf = lastk_buf, .size = lastk_size },
+ { .buf = bl_cur_log_buf, .size = bl_cur_log_buf_size },
+ };
+
+ pos = *ppos;
+ if (pos < 0)
+ return -EINVAL;
+
+ if (!count)
+ return 0;
+
+ for (i = 0; i < ARRAY_SIZE(log_regions); ++i) {
+ if (pos < log_regions[i].size && log_regions[i].buf != NULL) {
+ len = simple_read_from_buffer(userbuf, count, &pos,
+ log_regions[i].buf, log_regions[i].size);
+ if (len < 0)
+ return len;
+ count -= len;
+ userbuf += len;
+ total_len += len;
+ }
+ pos -= log_regions[i].size;
+ if (pos < 0)
+ break;
+ }
+
+ *ppos += total_len;
+ return total_len;
+}
+
+ssize_t bldr_log_total_size(void)
+{
+ return bl_last_log_buf_size + bl_cur_log_buf_size;
+}
+
+int bldr_log_setup(phys_addr_t bldr_phy_addr, size_t bldr_log_size,
+ bool is_last_bldr)
+{
+ char *bldr_base;
+ int ret = 0;
+
+ if (!bldr_log_size) {
+ ret = EINVAL;
+ goto _out;
+ }
+
+ bldr_base = ioremap(bldr_phy_addr, bldr_log_size);
+ if (!bldr_base) {
+ pr_warn("%s: failed to map last bootloader log buffer\n",
+ __func__);
+ ret = -ENOMEM;
+ goto _out;
+ }
+
+ if (is_last_bldr) {
+ bl_last_log_buf = kmalloc(bldr_log_size, GFP_KERNEL);
+ if (!bl_last_log_buf) {
+ pr_warn("%s: failed to alloc last bootloader log buffer\n",
+ __func__);
+ goto _unmap;
+ } else {
+ pr_debug("bootloader_log: allocate for last bootloader log, size: %zu\n",
+ bldr_log_size);
+ bldr_log_parser(bldr_base, bl_last_log_buf,
+ bldr_log_size, &bl_last_log_buf_size);
+ }
+ } else {
+ bl_cur_log_buf = kmalloc(bldr_log_size, GFP_KERNEL);
+ if (!bl_cur_log_buf) {
+ pr_warn("%s: failed to alloc last bootloader log buffer\n",
+ __func__);
+ goto _unmap;
+ } else {
+ pr_debug("bootloader_log: allocate buffer for bootloader log, size: %zu\n",
+ bldr_log_size);
+ bldr_log_parser(bldr_base, bl_cur_log_buf,
+ bldr_log_size, &bl_cur_log_buf_size);
+ }
+ }
+
+
+_unmap:
+ iounmap(bldr_base);
+_out:
+ return ret;
+}
+
+int bldr_log_init(void)
+{
+ struct device_node *np;
+ struct resource temp_res;
+ int num_reg = 0;
+
+ np = of_find_compatible_node(NULL, NULL, RAMLOG_COMPATIBLE_NAME);
+
+ if (np) {
+ while (of_address_to_resource(np, num_reg, &temp_res) == 0) {
+ if (!strcmp(temp_res.name, RAMLOG_LAST_RSE_NAME))
+ bldr_log_setup(temp_res.start,
+ resource_size(&temp_res), true);
+ else if (!strcmp(temp_res.name, RAMLOG_CUR_RSE_NAME))
+ bldr_log_setup(temp_res.start,
+ resource_size(&temp_res), false);
+ else
+ pr_warn("%s: unknown bldr resource %s\n",
+ __func__, temp_res.name);
+
+ num_reg++;
+ }
+ if (!num_reg)
+ pr_warn("%s: can't find address resource\n", __func__);
+ } else {
+ pr_warn("%s: can't find compatible '%s'\n",
+ __func__, RAMLOG_COMPATIBLE_NAME);
+ }
+
+ return num_reg;
+}
+
+void bldr_log_release(void)
+{
+ kfree(bl_last_log_buf);
+ kfree(bl_cur_log_buf);
+}
diff --git a/fs/pstore/inode.c b/fs/pstore/inode.c
index 1781dc50..ffe8d5a 100644
--- a/fs/pstore/inode.c
+++ b/fs/pstore/inode.c
@@ -37,6 +37,8 @@
#include <linux/spinlock.h>
#include <linux/uaccess.h>
#include <linux/syslog.h>
+#include <linux/bldr_debug_tools.h>
+#include <linux/vmalloc.h>
#include "internal.h"
@@ -121,27 +123,19 @@ static const struct seq_operations pstore_ftrace_seq_ops = {
.show = pstore_ftrace_seq_show,
};
-static int pstore_check_syslog_permissions(struct pstore_private *ps)
-{
- switch (ps->type) {
- case PSTORE_TYPE_DMESG:
- case PSTORE_TYPE_CONSOLE:
- return check_syslog_permissions(SYSLOG_ACTION_READ_ALL,
- SYSLOG_FROM_READER);
- default:
- return 0;
- }
-}
-
static ssize_t pstore_file_read(struct file *file, char __user *userbuf,
size_t count, loff_t *ppos)
{
struct seq_file *sf = file->private_data;
struct pstore_private *ps = sf->private;
+ ssize_t len = 0;
if (ps->type == PSTORE_TYPE_FTRACE)
return seq_read(file, userbuf, count, ppos);
- return simple_read_from_buffer(userbuf, count, ppos, ps->data, ps->size);
+ if (ps->type == PSTORE_TYPE_CONSOLE)
+ len = bldr_log_read(ps->data, ps->size, userbuf, count, ppos);
+ return (len > 0) ? len : simple_read_from_buffer(userbuf,
+ count, ppos, ps->data, ps->size);
}
static int pstore_file_open(struct inode *inode, struct file *file)
@@ -151,10 +145,6 @@ static int pstore_file_open(struct inode *inode, struct file *file)
int err;
const struct seq_operations *sops = NULL;
- err = pstore_check_syslog_permissions(ps);
- if (err)
- return err;
-
if (ps->type == PSTORE_TYPE_FTRACE)
sops = &pstore_ftrace_seq_ops;
@@ -191,11 +181,6 @@ static const struct file_operations pstore_file_operations = {
static int pstore_unlink(struct inode *dir, struct dentry *dentry)
{
struct pstore_private *p = d_inode(dentry)->i_private;
- int err;
-
- err = pstore_check_syslog_permissions(p);
- if (err)
- return err;
if (p->psi->erase)
p->psi->erase(p->type, p->id, p->count,
@@ -216,7 +201,7 @@ static void pstore_evict_inode(struct inode *inode)
spin_lock_irqsave(&allpstore_lock, flags);
list_del(&p->list);
spin_unlock_irqrestore(&allpstore_lock, flags);
- kfree(p);
+ vfree(p);
}
}
@@ -328,7 +313,7 @@ int pstore_mkfile(enum pstore_type_id type, char *psname, u64 id, int count,
goto fail;
inode->i_mode = S_IFREG | 0444;
inode->i_fop = &pstore_file_operations;
- private = kmalloc(sizeof *private + size, GFP_KERNEL);
+ private = vmalloc(sizeof *private + size);
if (!private)
goto fail_alloc;
private->type = type;
@@ -385,6 +370,9 @@ int pstore_mkfile(enum pstore_type_id type, char *psname, u64 id, int count,
memcpy(private->data, data, size);
inode->i_size = private->size = size;
+ if (type == PSTORE_TYPE_CONSOLE)
+ inode->i_size += bldr_log_total_size();
+
inode->i_private = private;
if (time.tv_sec)
@@ -402,7 +390,7 @@ int pstore_mkfile(enum pstore_type_id type, char *psname, u64 id, int count,
fail_lockedalloc:
inode_unlock(d_inode(root));
- kfree(private);
+ vfree(private);
fail_alloc:
iput(inode);
@@ -429,7 +417,7 @@ static int pstore_fill_super(struct super_block *sb, void *data, int silent)
inode = pstore_get_inode(sb);
if (inode) {
- inode->i_mode = S_IFDIR | 0755;
+ inode->i_mode = S_IFDIR | 0750;
inode->i_op = &pstore_dir_inode_operations;
inode->i_fop = &simple_dir_operations;
inc_nlink(inode);
diff --git a/fs/pstore/platform.c b/fs/pstore/platform.c
index 43033a3..3198781 100644
--- a/fs/pstore/platform.c
+++ b/fs/pstore/platform.c
@@ -44,6 +44,7 @@
#include <linux/hardirq.h>
#include <linux/jiffies.h>
#include <linux/workqueue.h>
+#include <linux/vmalloc.h>
#include "internal.h"
@@ -584,8 +585,8 @@ static void pstore_console_write(struct console *con, const char *s, unsigned c)
} else {
spin_lock_irqsave(&psinfo->buf_lock, flags);
}
- memcpy(psinfo->buf, s, c);
- psinfo->write(PSTORE_TYPE_CONSOLE, 0, &id, 0, 0, 0, c, psinfo);
+ psinfo->write_buf(PSTORE_TYPE_CONSOLE, 0, &id, 0,
+ s, 0, c, psinfo);
spin_unlock_irqrestore(&psinfo->buf_lock, flags);
s += c;
c = e - s;
@@ -787,7 +788,7 @@ void pstore_get_records(int quiet)
if (ecc_notice_size)
memcpy(big_oops_buf + unzipped_len,
buf + size, ecc_notice_size);
- kfree(buf);
+ vfree(buf);
buf = big_oops_buf;
size = unzipped_len;
compressed = false;
@@ -802,7 +803,7 @@ void pstore_get_records(int quiet)
time, psi);
if (unzipped_len < 0) {
/* Free buffer other than big oops */
- kfree(buf);
+ vfree(buf);
buf = NULL;
} else
unzipped_len = -1;
diff --git a/fs/pstore/ram.c b/fs/pstore/ram.c
index 8e151fb..5a0b602 100644
--- a/fs/pstore/ram.c
+++ b/fs/pstore/ram.c
@@ -36,6 +36,9 @@
#include <linux/pstore_ram.h>
#include <linux/of.h>
#include <linux/of_address.h>
+#include <linux/bldr_debug_tools.h>
+#include <linux/scatterlist.h>
+#include <crypto/aead.h>
#define RAMOOPS_KERNMSG_HDR "===="
#define MIN_MEM_SIZE 4096UL
@@ -84,11 +87,16 @@ MODULE_PARM_DESC(ramoops_ecc,
"ECC buffer size in bytes (1 is a special value, means 16 "
"bytes ECC)");
+#define AES_KEY_MAX_LEN (256 / 8)
+#define AES_KEY_IV_LEN 12
+#define AES_KEY_TAG_LEN 16
+
struct ramoops_context {
struct persistent_ram_zone **przs;
struct persistent_ram_zone *cprz;
struct persistent_ram_zone *fprz;
struct persistent_ram_zone *mprz;
+ phys_addr_t alt_phys_addr;
phys_addr_t phys_addr;
unsigned long size;
unsigned int memtype;
@@ -106,6 +114,15 @@ struct ramoops_context {
unsigned int ftrace_read_cnt;
unsigned int pmsg_read_cnt;
struct pstore_info pstore;
+ bool use_alt;
+ bool need_update;
+ int decrypt_state;
+ int aes_key_len;
+ unsigned char aes_key[AES_KEY_MAX_LEN];
+ unsigned char aes_key_iv[AES_KEY_IV_LEN];
+ unsigned char aes_key_tag[AES_KEY_TAG_LEN];
+ struct class *ramoops_class;
+ struct device *ramoops_device;
};
static struct platform_device *dummy;
@@ -122,11 +139,19 @@ static int ramoops_pstore_open(struct pstore_info *psi)
return 0;
}
+static int ramoops_pstore_close(struct pstore_info *psi)
+{
+ struct ramoops_context *cxt = psi->data;
+
+ cxt->need_update = 0;
+ return 0;
+}
+
static struct persistent_ram_zone *
ramoops_get_next_prz(struct persistent_ram_zone *przs[], uint *c, uint max,
u64 *id,
enum pstore_type_id *typep, enum pstore_type_id type,
- bool update)
+ bool update, bool use_alt)
{
struct persistent_ram_zone *prz;
int i = (*c)++;
@@ -140,7 +165,7 @@ ramoops_get_next_prz(struct persistent_ram_zone *przs[], uint *c, uint max,
/* Update old/shadowed buffer. */
if (update)
- persistent_ram_save_old(prz);
+ persistent_ram_save_old(prz, use_alt);
if (!persistent_ram_old_size(prz))
return NULL;
@@ -180,6 +205,127 @@ static bool prz_ok(struct persistent_ram_zone *prz)
persistent_ram_ecc_string(prz, NULL, 0));
}
+static int ramoops_decrypt_alt(struct ramoops_context *cxt)
+{
+ struct crypto_aead *tfm;
+ struct sg_table sgt;
+ struct page **pages;
+ unsigned int nr_pages;
+ unsigned int i;
+ int ret = 0;
+ struct aead_request *aead_req;
+ void *va;
+ phys_addr_t start;
+ size_t size;
+ size_t buffer_size;
+ void *buf;
+
+ if (!cxt->alt_phys_addr)
+ return -EINVAL;
+
+ start = cxt->alt_phys_addr;
+ size = cxt->size;
+ if (!request_mem_region(start, size, "persistent_ram")) {
+ pr_err("request mem region (0x%llx@0x%llx) failed\n",
+ (unsigned long long)size, (unsigned long long)start);
+ return -ENOMEM;
+ }
+
+ if (cxt->memtype)
+ va = ioremap(start, size);
+ else
+ va = ioremap_wc(start, size);
+ if (va == NULL) {
+ pr_err("ioremap mem region (0x%llx@0x%llx) failed\n",
+ (unsigned long long)size, (unsigned long long)start);
+ release_mem_region(start, size);
+ ret = -ENOMEM;
+ goto out_iomap;
+ }
+
+ tfm = crypto_alloc_aead("gcm(aes)", 0, CRYPTO_ALG_ASYNC);
+ if (IS_ERR(tfm)) {
+ ret = PTR_ERR(tfm);
+ pr_err("Failed to alloc aes, %d\n", ret);
+ goto out_crypto_alloc;
+ }
+
+ ret = crypto_aead_setkey(tfm, cxt->aes_key, cxt->aes_key_len);
+ if (ret) {
+ pr_err("Failed to set key, %d\n", ret);
+ goto out_set;
+ }
+
+ ret = crypto_aead_setauthsize(tfm, 16);
+ if (ret) {
+ pr_err("Failed to set auth size, %d\n", ret);
+ goto out_set;
+ }
+
+ aead_req = aead_request_alloc(tfm, GFP_KERNEL);
+ if (!aead_req) {
+ ret = -ENOMEM;
+ pr_err("Failed to allocate aead request\n");
+ goto out_set;
+ }
+
+ buffer_size = size + AES_KEY_TAG_LEN;
+ buf = vmalloc(buffer_size);
+ if (!buf) {
+ ret = -ENOMEM;
+ pr_err("Failed to allocate bounce\n");
+ goto out_no_buf;
+ }
+
+ memcpy_fromio(buf, va, size);
+ memcpy(buf + size, cxt->aes_key_tag, AES_KEY_TAG_LEN);
+
+ nr_pages = DIV_ROUND_UP(buffer_size, PAGE_SIZE);
+ pages = kmalloc_array(nr_pages, sizeof(*pages), GFP_KERNEL);
+ if (!pages) {
+ ret = -ENOMEM;
+ pr_err("Failed to allocate page list\n");
+ goto out_no_page_list;
+ }
+
+ for (i = 0; i < nr_pages; ++i)
+ pages[i] = vmalloc_to_page(buf + (i * PAGE_SIZE));
+
+ ret = sg_alloc_table_from_pages(&sgt, pages, nr_pages, 0,
+ buffer_size, GFP_KERNEL);
+
+ if (ret) {
+ pr_err("Failed to allocate sg_table\n");
+ goto out_no_sg_table;
+ }
+
+ aead_request_set_crypt(aead_req, sgt.sgl, sgt.sgl, buffer_size,
+ cxt->aes_key_iv);
+ aead_request_set_ad(aead_req, 0);
+ ret = crypto_aead_decrypt(aead_req);
+ if (ret) {
+ pr_err("Failed to decrypt %d\n", ret);
+ goto out_decrypt;
+ }
+ memcpy_toio(va, buf, size);
+
+out_decrypt:
+ sg_free_table(&sgt);
+out_no_sg_table:
+ kfree(pages);
+out_no_page_list:
+ vfree(buf);
+out_no_buf:
+ aead_request_free(aead_req);
+out_set:
+ crypto_free_aead(tfm);
+out_crypto_alloc:
+ iounmap(va);
+out_iomap:
+ release_mem_region(start, size);
+ return ret;
+}
+
static ssize_t ramoops_pstore_read(u64 *id, enum pstore_type_id *type,
int *count, struct timespec *time,
char **buf, bool *compressed,
@@ -190,6 +336,8 @@ static ssize_t ramoops_pstore_read(u64 *id, enum pstore_type_id *type,
struct ramoops_context *cxt = psi->data;
struct persistent_ram_zone *prz = NULL;
int header_length = 0;
+ bool use_alt = cxt->use_alt;
+ bool update = cxt->need_update;
/* Ramoops headers provide time stamps for PSTORE_TYPE_DMESG, but
* PSTORE_TYPE_CONSOLE and PSTORE_TYPE_FTRACE don't currently have
@@ -203,7 +351,7 @@ static ssize_t ramoops_pstore_read(u64 *id, enum pstore_type_id *type,
while (cxt->dump_read_cnt < cxt->max_dump_cnt && !prz) {
prz = ramoops_get_next_prz(cxt->przs, &cxt->dump_read_cnt,
cxt->max_dump_cnt, id, type,
- PSTORE_TYPE_DMESG, 1);
+ PSTORE_TYPE_DMESG, 1, use_alt);
if (!prz_ok(prz))
continue;
header_length = ramoops_read_kmsg_hdr(persistent_ram_old(prz),
@@ -211,20 +359,23 @@ static ssize_t ramoops_pstore_read(u64 *id, enum pstore_type_id *type,
/* Clear and skip this DMESG record if it has no valid header */
if (!header_length) {
persistent_ram_free_old(prz);
- persistent_ram_zap(prz);
+ persistent_ram_zap(prz, use_alt);
prz = NULL;
}
}
if (!prz_ok(prz))
prz = ramoops_get_next_prz(&cxt->cprz, &cxt->console_read_cnt,
- 1, id, type, PSTORE_TYPE_CONSOLE, 0);
+ 1, id, type, PSTORE_TYPE_CONSOLE,
+ update, use_alt);
if (!prz_ok(prz))
prz = ramoops_get_next_prz(&cxt->fprz, &cxt->ftrace_read_cnt,
- 1, id, type, PSTORE_TYPE_FTRACE, 0);
+ 1, id, type, PSTORE_TYPE_FTRACE,
+ update, use_alt);
if (!prz_ok(prz))
prz = ramoops_get_next_prz(&cxt->mprz, &cxt->pmsg_read_cnt,
- 1, id, type, PSTORE_TYPE_PMSG, 0);
+ 1, id, type, PSTORE_TYPE_PMSG,
+ update, use_alt);
if (!prz_ok(prz))
return 0;
@@ -233,7 +384,7 @@ static ssize_t ramoops_pstore_read(u64 *id, enum pstore_type_id *type,
/* ECC correction notice */
*ecc_notice_size = persistent_ram_ecc_string(prz, NULL, 0);
- *buf = kmalloc(size + *ecc_notice_size + 1, GFP_KERNEL);
+ *buf = vmalloc(size + *ecc_notice_size + 1);
if (*buf == NULL)
return -ENOMEM;
@@ -375,7 +526,7 @@ static int ramoops_pstore_erase(enum pstore_type_id type, u64 id, int count,
}
persistent_ram_free_old(prz);
- persistent_ram_zap(prz);
+ persistent_ram_zap(prz, 0);
return 0;
}
@@ -386,12 +537,102 @@ static struct ramoops_context oops_cxt = {
.name = "ramoops",
.open = ramoops_pstore_open,
.read = ramoops_pstore_read,
+ .close = ramoops_pstore_close,
.write_buf = ramoops_pstore_write_buf,
.write_buf_user = ramoops_pstore_write_buf_user,
.erase = ramoops_pstore_erase,
},
};
+
+static ssize_t use_alt_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ramoops_context *cxt = &oops_cxt;
+ return snprintf(buf, PAGE_SIZE, "%d\n", cxt->use_alt);
+}
+
+static ssize_t use_alt_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct ramoops_context *cxt = &oops_cxt;
+ int use_alt = 0;
+
+ if (kstrtoint(buf, 10, &use_alt) < 0)
+ return -EINVAL;
+ if (!cxt->use_alt && use_alt && !cxt->decrypt_state) {
+ int ret = ramoops_decrypt_alt(cxt);
+ if (ret < 0) {
+ cxt->decrypt_state = ret;
+ return ret;
+ }
+ cxt->decrypt_state = 1;
+ }
+ cxt->use_alt = use_alt;
+ cxt->need_update = 1;
+ return count;
+}
+static DEVICE_ATTR_RW(use_alt);
+
+static ssize_t decrypt_state_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct ramoops_context *cxt = &oops_cxt;
+ return snprintf(buf, PAGE_SIZE, "%d\n", cxt->decrypt_state);
+}
+static DEVICE_ATTR_RO(decrypt_state);
+
+static ssize_t aes_key_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct ramoops_context *cxt = &oops_cxt;
+
+ if (count > AES_KEY_MAX_LEN)
+ count = AES_KEY_MAX_LEN;
+ if ((count != 16) && (count != 32))
+ return -EINVAL;
+ cxt->aes_key_len = count;
+ memcpy(cxt->aes_key, buf, count);
+ return count;
+}
+static DEVICE_ATTR_WO(aes_key);
+
+static ssize_t aes_key_iv_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct ramoops_context *cxt = &oops_cxt;
+
+ if (count > AES_KEY_IV_LEN)
+ count = AES_KEY_IV_LEN;
+ memcpy(cxt->aes_key_iv, buf, count);
+ return count;
+}
+static DEVICE_ATTR_WO(aes_key_iv);
+
+static ssize_t aes_key_tag_store(struct device *dev,
+ struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct ramoops_context *cxt = &oops_cxt;
+
+ if (count > AES_KEY_TAG_LEN)
+ count = AES_KEY_TAG_LEN;
+ memcpy(cxt->aes_key_tag, buf, count);
+ return count;
+}
+static DEVICE_ATTR_WO(aes_key_tag);
+
+static struct attribute *ramoops_attrs[] = {
+ &dev_attr_use_alt.attr,
+ &dev_attr_decrypt_state.attr,
+ &dev_attr_aes_key.attr,
+ &dev_attr_aes_key_iv.attr,
+ &dev_attr_aes_key_tag.attr,
+ NULL,
+};
+ATTRIBUTE_GROUPS(ramoops);
+
static void ramoops_free_przs(struct ramoops_context *cxt)
{
int i;
@@ -407,7 +648,8 @@ static void ramoops_free_przs(struct ramoops_context *cxt)
}
static int ramoops_init_przs(struct device *dev, struct ramoops_context *cxt,
- phys_addr_t *paddr, size_t dump_mem_sz)
+ phys_addr_t *paddr, phys_addr_t *alt_paddr,
+ size_t dump_mem_sz)
{
int err = -ENOMEM;
int i;
@@ -432,7 +674,8 @@ static int ramoops_init_przs(struct device *dev, struct ramoops_context *cxt,
}
for (i = 0; i < cxt->max_dump_cnt; i++) {
- cxt->przs[i] = persistent_ram_new(*paddr, cxt->record_size, 0,
+ cxt->przs[i] = persistent_ram_new(*paddr, *alt_paddr,
+ cxt->record_size, 0,
&cxt->ecc_info,
cxt->memtype, 0);
if (IS_ERR(cxt->przs[i])) {
@@ -447,6 +690,8 @@ static int ramoops_init_przs(struct device *dev, struct ramoops_context *cxt,
goto fail_prz;
}
*paddr += cxt->record_size;
+ if (*alt_paddr)
+ *alt_paddr += cxt->record_size;
}
return 0;
@@ -459,7 +704,8 @@ static int ramoops_init_przs(struct device *dev, struct ramoops_context *cxt,
static int ramoops_init_prz(struct device *dev, struct ramoops_context *cxt,
struct persistent_ram_zone **prz,
- phys_addr_t *paddr, size_t sz, u32 sig)
+ phys_addr_t *paddr, phys_addr_t *alt_paddr,
+ size_t sz, u32 sig)
{
if (!sz)
return 0;
@@ -471,7 +717,7 @@ static int ramoops_init_prz(struct device *dev, struct ramoops_context *cxt,
return -ENOMEM;
}
- *prz = persistent_ram_new(*paddr, sz, sig, &cxt->ecc_info,
+ *prz = persistent_ram_new(*paddr, *alt_paddr, sz, sig, &cxt->ecc_info,
cxt->memtype, 0);
if (IS_ERR(*prz)) {
int err = PTR_ERR(*prz);
@@ -481,9 +727,11 @@ static int ramoops_init_prz(struct device *dev, struct ramoops_context *cxt,
return err;
}
- persistent_ram_zap(*prz);
+ persistent_ram_zap(*prz, 0);
*paddr += sz;
+ if (*alt_paddr)
+ *alt_paddr += sz;
return 0;
}
@@ -514,22 +762,57 @@ static int ramoops_parse_dt(struct platform_device *pdev,
struct ramoops_platform_data *pdata)
{
struct device_node *of_node = pdev->dev.of_node;
- struct resource *res;
+ struct device_node *mem_region;
+ struct resource res;
+ size_t alt_mem_size;
+ unsigned int alt_mem_type;
u32 value;
int ret;
dev_dbg(&pdev->dev, "using Device Tree\n");
- res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- if (!res) {
- dev_err(&pdev->dev,
- "failed to locate DT /reserved-memory resource\n");
- return -EINVAL;
+ mem_region = of_parse_phandle(of_node, "memory-region", 0);
+ if (!mem_region) {
+ dev_err(&pdev->dev, "no memory-region phandle\n");
+ return -ENODEV;
}
- pdata->mem_size = resource_size(res);
- pdata->mem_address = res->start;
+ ret = of_address_to_resource(mem_region, 0, &res);
+ of_node_put(mem_region);
+ if (ret) {
+ dev_err(&pdev->dev,
+ "failed to translate memory-region to resource: %d\n",
+ ret);
+ return ret;
+ }
+
+ pdata->mem_size = resource_size(&res);
+ pdata->mem_address = res.start;
pdata->mem_type = of_property_read_bool(of_node, "unbuffered");
+
+ mem_region = of_parse_phandle(of_node, "alt-memory-region", 0);
+ if (mem_region) {
+ ret = of_address_to_resource(mem_region, 0, &res);
+ of_node_put(mem_region);
+ if (ret) {
+ dev_err(&pdev->dev,
+ "failed alt-memory-region to resource: %d\n",
+ ret);
+ return ret;
+ }
+
+ pdata->alt_mem_address = res.start;
+ alt_mem_size = resource_size(&res);
+ alt_mem_type = of_property_read_bool(of_node, "unbuffered");
+ /* Alt memory region should be same size/type as orig. region */
+ if (alt_mem_size != pdata->mem_size ||
+ alt_mem_type != pdata->mem_type) {
+ dev_err(&pdev->dev,
+ "alt region not the same size/type as main.\n");
+ return -EINVAL;
+ }
+ }
+
pdata->dump_oops = !of_property_read_bool(of_node, "no-dump-oops");
#define parse_size(name, field) { \
@@ -563,6 +846,7 @@ static int ramoops_probe(struct platform_device *pdev)
struct ramoops_context *cxt = &oops_cxt;
size_t dump_mem_sz;
phys_addr_t paddr;
+ phys_addr_t alt_paddr;
int err = -EINVAL;
if (dev_of_node(dev) && !pdata) {
@@ -602,6 +886,7 @@ static int ramoops_probe(struct platform_device *pdev)
cxt->size = pdata->mem_size;
cxt->phys_addr = pdata->mem_address;
cxt->memtype = pdata->mem_type;
+ cxt->alt_phys_addr = pdata->alt_mem_address;
cxt->record_size = pdata->record_size;
cxt->console_size = pdata->console_size;
cxt->ftrace_size = pdata->ftrace_size;
@@ -610,24 +895,26 @@ static int ramoops_probe(struct platform_device *pdev)
cxt->ecc_info = pdata->ecc_info;
paddr = cxt->phys_addr;
+ alt_paddr = cxt->alt_phys_addr;
dump_mem_sz = cxt->size - cxt->console_size - cxt->ftrace_size
- cxt->pmsg_size;
- err = ramoops_init_przs(dev, cxt, &paddr, dump_mem_sz);
+ err = ramoops_init_przs(dev, cxt, &paddr, &alt_paddr, dump_mem_sz);
if (err)
goto fail_out;
- err = ramoops_init_prz(dev, cxt, &cxt->cprz, &paddr,
+ err = ramoops_init_prz(dev, cxt, &cxt->cprz, &paddr, &alt_paddr,
cxt->console_size, 0);
if (err)
goto fail_init_cprz;
- err = ramoops_init_prz(dev, cxt, &cxt->fprz, &paddr, cxt->ftrace_size,
- LINUX_VERSION_CODE);
+ err = ramoops_init_prz(dev, cxt, &cxt->fprz, &paddr, &alt_paddr,
+ cxt->ftrace_size, LINUX_VERSION_CODE);
if (err)
goto fail_init_fprz;
- err = ramoops_init_prz(dev, cxt, &cxt->mprz, &paddr, cxt->pmsg_size, 0);
+ err = ramoops_init_prz(dev, cxt, &cxt->mprz, &paddr, &alt_paddr,
+ cxt->pmsg_size, 0);
if (err)
goto fail_init_mprz;
@@ -675,12 +962,33 @@ static int ramoops_probe(struct platform_device *pdev)
ramoops_pmsg_size = pdata->pmsg_size;
ramoops_ftrace_size = pdata->ftrace_size;
+ cxt->ramoops_class = class_create(THIS_MODULE, "ramoops");
+ if (IS_ERR(cxt->ramoops_class)) {
+ pr_err("device class file already in use\n");
+ err = PTR_ERR(cxt->ramoops_class);
+ goto fail_buf;
+ }
+ cxt->ramoops_class->dev_groups = ramoops_groups;
+
+ cxt->ramoops_device = device_create(cxt->ramoops_class, NULL, 0,
+ NULL, "pstore");
+ if (IS_ERR(cxt->ramoops_device)) {
+ pr_err("failed to create device\n");
+ err = PTR_ERR(cxt->ramoops_device);
+ goto fail_device;
+ }
+
pr_info("attached 0x%lx@0x%llx, ecc: %d/%d\n",
cxt->size, (unsigned long long)cxt->phys_addr,
cxt->ecc_info.ecc_size, cxt->ecc_info.block_size);
+ if (cxt->console_size)
+ bldr_log_init();
+
return 0;
+fail_device:
+ class_destroy(cxt->ramoops_class);
fail_buf:
kfree(cxt->pstore.buf);
fail_clear:
@@ -700,6 +1008,8 @@ static int ramoops_remove(struct platform_device *pdev)
{
struct ramoops_context *cxt = &oops_cxt;
+ device_unregister(cxt->ramoops_device);
+ class_destroy(cxt->ramoops_class);
pstore_unregister(&cxt->pstore);
kfree(cxt->pstore.buf);
diff --git a/fs/pstore/ram_core.c b/fs/pstore/ram_core.c
index e11672a..7561826 100644
--- a/fs/pstore/ram_core.c
+++ b/fs/pstore/ram_core.c
@@ -36,16 +36,18 @@ struct persistent_ram_buffer {
uint8_t data[0];
};
+static int persistent_ram_buffer_map_alt(struct persistent_ram_zone *prz);
+
#define PERSISTENT_RAM_SIG (0x43474244) /* DBGC */
-static inline size_t buffer_size(struct persistent_ram_zone *prz)
+static inline size_t buffer_size(struct persistent_ram_buffer *buffer)
{
- return atomic_read(&prz->buffer->size);
+ return atomic_read(&buffer->size);
}
-static inline size_t buffer_start(struct persistent_ram_zone *prz)
+static inline size_t buffer_start(struct persistent_ram_buffer *buffer)
{
- return atomic_read(&prz->buffer->start);
+ return atomic_read(&buffer->start);
}
/* increase and wrap the start pointer, returning the old value */
@@ -156,9 +158,11 @@ static void persistent_ram_update_header_ecc(struct persistent_ram_zone *prz)
prz->par_header);
}
-static void persistent_ram_ecc_old(struct persistent_ram_zone *prz)
+static void persistent_ram_ecc_old(struct persistent_ram_zone *prz,
+ bool use_alt)
{
- struct persistent_ram_buffer *buffer = prz->buffer;
+ struct persistent_ram_buffer *buffer =
+ use_alt ? prz->alt_buffer : prz->buffer;
uint8_t *block;
uint8_t *par;
@@ -166,8 +170,8 @@ static void persistent_ram_ecc_old(struct persistent_ram_zone *prz)
return;
block = buffer->data;
- par = prz->par_buffer;
- while (block < buffer->data + buffer_size(prz)) {
+ par = buffer->data + prz->buffer_size;
+ while (block < buffer->data + buffer_size(buffer)) {
int numerr;
int size = prz->ecc_info.block_size;
if (block + size > buffer->data + prz->buffer_size)
@@ -280,18 +284,31 @@ static int notrace persistent_ram_update_user(struct persistent_ram_zone *prz,
return ret;
}
-void persistent_ram_save_old(struct persistent_ram_zone *prz)
+void persistent_ram_save_old(struct persistent_ram_zone *prz, bool use_alt)
{
- struct persistent_ram_buffer *buffer = prz->buffer;
- size_t size = buffer_size(prz);
- size_t start = buffer_start(prz);
+ struct persistent_ram_buffer *buffer;
+ size_t size;
+ size_t start;
- if (!size)
+ if (!use_alt)
+ buffer = prz->buffer;
+ else {
+ if (!prz->alt_buffer)
+ if (persistent_ram_buffer_map_alt(prz))
+ return;
+ buffer = prz->alt_buffer;
+ }
+ size = buffer_size(buffer);
+ start = buffer_start(buffer);
+ if (!size || size < start)
return;
+ if (prz->old_log && prz->old_log_alloc_size < size)
+ persistent_ram_free_old(prz);
if (!prz->old_log) {
- persistent_ram_ecc_old(prz);
- prz->old_log = kmalloc(size, GFP_KERNEL);
+ persistent_ram_ecc_old(prz, use_alt);
+ prz->old_log = vmalloc(size);
+ prz->old_log_alloc_size = size;
}
if (!prz->old_log) {
pr_err("failed to allocate buffer\n");
@@ -377,16 +394,26 @@ void *persistent_ram_old(struct persistent_ram_zone *prz)
void persistent_ram_free_old(struct persistent_ram_zone *prz)
{
- kfree(prz->old_log);
+ vfree(prz->old_log);
prz->old_log = NULL;
prz->old_log_size = 0;
+ prz->old_log_alloc_size = 0;
}
-void persistent_ram_zap(struct persistent_ram_zone *prz)
+void persistent_ram_zap(struct persistent_ram_zone *prz, bool use_alt)
{
- atomic_set(&prz->buffer->start, 0);
- atomic_set(&prz->buffer->size, 0);
- persistent_ram_update_header_ecc(prz);
+ struct persistent_ram_buffer *buffer =
+ use_alt ? prz->alt_buffer : prz->buffer;
+
+ atomic_set(&buffer->start, 0);
+ atomic_set(&buffer->size, 0);
+
+ /*
+ * The alternate buffer is read-only, we don't need to
+ * update the ecc
+ */
+ if (!use_alt)
+ persistent_ram_update_header_ecc(prz);
}
static void *persistent_ram_vmap(phys_addr_t start, size_t size,
@@ -421,7 +448,12 @@ static void *persistent_ram_vmap(phys_addr_t start, size_t size,
vaddr = vmap(pages, page_count, VM_MAP, prot);
kfree(pages);
- return vaddr;
+ /*
+ * Since vmap() uses page granularity, we must add the offset
+ * into the page here, to get the byte granularity address
+ * into the mapping to represent the actual "start" location.
+ */
+ return vaddr + offset_in_page(start);
}
static void *persistent_ram_iomap(phys_addr_t start, size_t size,
@@ -440,36 +472,73 @@ static void *persistent_ram_iomap(phys_addr_t start, size_t size,
else
va = ioremap_wc(start, size);
+ /*
+ * Since request_mem_region() and ioremap() are byte-granularity
+ * there is no need handle anything special like we do when the
+ * vmap() case in persistent_ram_vmap() above.
+ */
return va;
}
-static int persistent_ram_buffer_map(phys_addr_t start, phys_addr_t size,
- struct persistent_ram_zone *prz, int memtype)
+static int _persistent_ram_buffer_map(phys_addr_t start, phys_addr_t size,
+ int memtype, void **vaddr)
{
- prz->paddr = start;
- prz->size = size;
-
if (pfn_valid(start >> PAGE_SHIFT))
- prz->vaddr = persistent_ram_vmap(start, size, memtype);
+ *vaddr = persistent_ram_vmap(start, size, memtype);
else
- prz->vaddr = persistent_ram_iomap(start, size, memtype);
+ *vaddr = persistent_ram_iomap(start, size, memtype);
- if (!prz->vaddr) {
+ if (!*vaddr) {
pr_err("%s: Failed to map 0x%llx pages at 0x%llx\n", __func__,
(unsigned long long)size, (unsigned long long)start);
return -ENOMEM;
}
- prz->buffer = prz->vaddr + offset_in_page(start);
+ return 0;
+}
+
+static int persistent_ram_buffer_map(phys_addr_t start, phys_addr_t size,
+ struct persistent_ram_zone *prz,
+ int memtype)
+{
+ int ret;
+
+ ret = _persistent_ram_buffer_map(start, size, memtype, &prz->vaddr);
+ if (ret)
+ return ret;
+
+ prz->paddr = start;
+ prz->size = size;
+ prz->memtype = memtype;
+ prz->buffer = prz->vaddr;
prz->buffer_size = size - sizeof(struct persistent_ram_buffer);
return 0;
}
+static int persistent_ram_buffer_map_alt(struct persistent_ram_zone *prz)
+{
+ phys_addr_t alt_start = prz->alt_paddr;
+ int ret;
+
+ if (!alt_start)
+ return -ENOMEM;
+
+ ret = _persistent_ram_buffer_map(alt_start, prz->size, prz->memtype,
+ &prz->alt_vaddr);
+ if (ret)
+ return ret;
+
+ prz->alt_buffer = prz->alt_vaddr;
+
+ return 0;
+}
+
static int persistent_ram_post_init(struct persistent_ram_zone *prz, u32 sig,
struct persistent_ram_ecc_info *ecc_info)
{
int ret;
+ struct persistent_ram_buffer *buffer = prz->buffer;
ret = persistent_ram_init_ecc(prz, ecc_info);
if (ret)
@@ -478,24 +547,24 @@ static int persistent_ram_post_init(struct persistent_ram_zone *prz, u32 sig,
sig ^= PERSISTENT_RAM_SIG;
if (prz->buffer->sig == sig) {
- if (buffer_size(prz) > prz->buffer_size ||
- buffer_start(prz) > buffer_size(prz))
+ if (buffer_size(buffer) > prz->buffer_size ||
+ buffer_start(buffer) > buffer_size(buffer))
pr_info("found existing invalid buffer, size %zu, start %zu\n",
- buffer_size(prz), buffer_start(prz));
+ buffer_size(buffer), buffer_start(buffer));
else {
pr_debug("found existing buffer, size %zu, start %zu\n",
- buffer_size(prz), buffer_start(prz));
- persistent_ram_save_old(prz);
+ buffer_size(buffer), buffer_start(buffer));
+ persistent_ram_save_old(prz, 0);
return 0;
}
} else {
pr_debug("no valid data in buffer (sig = 0x%08x)\n",
- prz->buffer->sig);
+ buffer->sig);
}
/* Rewind missing or invalid memory area. */
prz->buffer->sig = sig;
- persistent_ram_zap(prz);
+ persistent_ram_zap(prz, 0);
return 0;
}
@@ -507,18 +576,31 @@ void persistent_ram_free(struct persistent_ram_zone *prz)
if (prz->vaddr) {
if (pfn_valid(prz->paddr >> PAGE_SHIFT)) {
- vunmap(prz->vaddr);
+ /* We must vunmap() at page-granularity. */
+ vunmap(prz->vaddr - offset_in_page(prz->paddr));
} else {
iounmap(prz->vaddr);
release_mem_region(prz->paddr, prz->size);
}
prz->vaddr = NULL;
}
+
+ if (prz->alt_vaddr) {
+ if (pfn_valid(prz->alt_paddr >> PAGE_SHIFT)) {
+ /* We must vunmap() at page-granularity. */
+ vunmap(prz->alt_vaddr - offset_in_page(prz->alt_paddr));
+ } else {
+ iounmap(prz->alt_vaddr);
+ release_mem_region(prz->alt_paddr, prz->size);
+ }
+ prz->alt_vaddr = NULL;
+ }
persistent_ram_free_old(prz);
kfree(prz);
}
-struct persistent_ram_zone *persistent_ram_new(phys_addr_t start, size_t size,
+struct persistent_ram_zone *persistent_ram_new(phys_addr_t start,
+ phys_addr_t alt_start, size_t size,
u32 sig, struct persistent_ram_ecc_info *ecc_info,
unsigned int memtype, u32 flags)
{
@@ -538,6 +620,7 @@ struct persistent_ram_zone *persistent_ram_new(phys_addr_t start, size_t size,
ret = persistent_ram_buffer_map(start, size, prz, memtype);
if (ret)
goto err;
+ prz->alt_paddr = alt_start;
ret = persistent_ram_post_init(prz, sig, ecc_info);
if (ret)
diff --git a/fs/quota/quota.c b/fs/quota/quota.c
index 2d44542..a2329f7 100644
--- a/fs/quota/quota.c
+++ b/fs/quota/quota.c
@@ -17,6 +17,7 @@
#include <linux/quotaops.h>
#include <linux/types.h>
#include <linux/writeback.h>
+#include <linux/nospec.h>
static int check_quotactl_permission(struct super_block *sb, int type, int cmd,
qid_t id)
@@ -706,6 +707,7 @@ static int do_quotactl(struct super_block *sb, int type, int cmd, qid_t id,
if (type >= (XQM_COMMAND(cmd) ? XQM_MAXQUOTAS : MAXQUOTAS))
return -EINVAL;
+ type = array_index_nospec(type, MAXQUOTAS);
/*
* Quota not supported on this fs? Check this before s_quota_types
* since they needn't be set if quota is not supported at all.
diff --git a/fs/reiserfs/namei.c b/fs/reiserfs/namei.c
index e6a2b40..1ec728c 100644
--- a/fs/reiserfs/namei.c
+++ b/fs/reiserfs/namei.c
@@ -687,8 +687,7 @@ static int reiserfs_create(struct inode *dir, struct dentry *dentry, umode_t mod
reiserfs_update_inode_transaction(inode);
reiserfs_update_inode_transaction(dir);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
retval = journal_end(&th);
out_failed:
@@ -771,8 +770,7 @@ static int reiserfs_mknod(struct inode *dir, struct dentry *dentry, umode_t mode
goto out_failed;
}
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
retval = journal_end(&th);
out_failed:
@@ -871,8 +869,7 @@ static int reiserfs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode
/* the above add_entry did not update dir's stat data */
reiserfs_update_sd(&th, dir);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
retval = journal_end(&th);
out_failed:
reiserfs_write_unlock(dir->i_sb);
@@ -1187,8 +1184,7 @@ static int reiserfs_symlink(struct inode *parent_dir,
goto out_failed;
}
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
retval = journal_end(&th);
out_failed:
reiserfs_write_unlock(parent_dir->i_sb);
diff --git a/fs/reiserfs/prints.c b/fs/reiserfs/prints.c
index 4f3f928..92470e5 100644
--- a/fs/reiserfs/prints.c
+++ b/fs/reiserfs/prints.c
@@ -76,83 +76,99 @@ static char *le_type(struct reiserfs_key *key)
}
/* %k */
-static void sprintf_le_key(char *buf, struct reiserfs_key *key)
+static int scnprintf_le_key(char *buf, size_t size, struct reiserfs_key *key)
{
if (key)
- sprintf(buf, "[%d %d %s %s]", le32_to_cpu(key->k_dir_id),
- le32_to_cpu(key->k_objectid), le_offset(key),
- le_type(key));
+ return scnprintf(buf, size, "[%d %d %s %s]",
+ le32_to_cpu(key->k_dir_id),
+ le32_to_cpu(key->k_objectid), le_offset(key),
+ le_type(key));
else
- sprintf(buf, "[NULL]");
+ return scnprintf(buf, size, "[NULL]");
}
/* %K */
-static void sprintf_cpu_key(char *buf, struct cpu_key *key)
+static int scnprintf_cpu_key(char *buf, size_t size, struct cpu_key *key)
{
if (key)
- sprintf(buf, "[%d %d %s %s]", key->on_disk_key.k_dir_id,
- key->on_disk_key.k_objectid, reiserfs_cpu_offset(key),
- cpu_type(key));
+ return scnprintf(buf, size, "[%d %d %s %s]",
+ key->on_disk_key.k_dir_id,
+ key->on_disk_key.k_objectid,
+ reiserfs_cpu_offset(key), cpu_type(key));
else
- sprintf(buf, "[NULL]");
+ return scnprintf(buf, size, "[NULL]");
}
-static void sprintf_de_head(char *buf, struct reiserfs_de_head *deh)
+static int scnprintf_de_head(char *buf, size_t size,
+ struct reiserfs_de_head *deh)
{
if (deh)
- sprintf(buf,
- "[offset=%d dir_id=%d objectid=%d location=%d state=%04x]",
- deh_offset(deh), deh_dir_id(deh), deh_objectid(deh),
- deh_location(deh), deh_state(deh));
+ return scnprintf(buf, size,
+ "[offset=%d dir_id=%d objectid=%d location=%d state=%04x]",
+ deh_offset(deh), deh_dir_id(deh),
+ deh_objectid(deh), deh_location(deh),
+ deh_state(deh));
else
- sprintf(buf, "[NULL]");
+ return scnprintf(buf, size, "[NULL]");
}
-static void sprintf_item_head(char *buf, struct item_head *ih)
+static int scnprintf_item_head(char *buf, size_t size, struct item_head *ih)
{
if (ih) {
- strcpy(buf,
- (ih_version(ih) == KEY_FORMAT_3_6) ? "*3.6* " : "*3.5*");
- sprintf_le_key(buf + strlen(buf), &(ih->ih_key));
- sprintf(buf + strlen(buf), ", item_len %d, item_location %d, "
- "free_space(entry_count) %d",
- ih_item_len(ih), ih_location(ih), ih_free_space(ih));
+ char *p = buf;
+ char * const end = buf + size;
+
+ p += scnprintf(p, end - p, "%s",
+ (ih_version(ih) == KEY_FORMAT_3_6) ?
+ "*3.6* " : "*3.5*");
+
+ p += scnprintf_le_key(p, end - p, &ih->ih_key);
+
+ p += scnprintf(p, end - p,
+ ", item_len %d, item_location %d, free_space(entry_count) %d",
+ ih_item_len(ih), ih_location(ih),
+ ih_free_space(ih));
+ return p - buf;
} else
- sprintf(buf, "[NULL]");
+ return scnprintf(buf, size, "[NULL]");
}
-static void sprintf_direntry(char *buf, struct reiserfs_dir_entry *de)
+static int scnprintf_direntry(char *buf, size_t size,
+ struct reiserfs_dir_entry *de)
{
char name[20];
memcpy(name, de->de_name, de->de_namelen > 19 ? 19 : de->de_namelen);
name[de->de_namelen > 19 ? 19 : de->de_namelen] = 0;
- sprintf(buf, "\"%s\"==>[%d %d]", name, de->de_dir_id, de->de_objectid);
+ return scnprintf(buf, size, "\"%s\"==>[%d %d]",
+ name, de->de_dir_id, de->de_objectid);
}
-static void sprintf_block_head(char *buf, struct buffer_head *bh)
+static int scnprintf_block_head(char *buf, size_t size, struct buffer_head *bh)
{
- sprintf(buf, "level=%d, nr_items=%d, free_space=%d rdkey ",
- B_LEVEL(bh), B_NR_ITEMS(bh), B_FREE_SPACE(bh));
+ return scnprintf(buf, size,
+ "level=%d, nr_items=%d, free_space=%d rdkey ",
+ B_LEVEL(bh), B_NR_ITEMS(bh), B_FREE_SPACE(bh));
}
-static void sprintf_buffer_head(char *buf, struct buffer_head *bh)
+static int scnprintf_buffer_head(char *buf, size_t size, struct buffer_head *bh)
{
- sprintf(buf,
- "dev %pg, size %zd, blocknr %llu, count %d, state 0x%lx, page %p, (%s, %s, %s)",
- bh->b_bdev, bh->b_size,
- (unsigned long long)bh->b_blocknr, atomic_read(&(bh->b_count)),
- bh->b_state, bh->b_page,
- buffer_uptodate(bh) ? "UPTODATE" : "!UPTODATE",
- buffer_dirty(bh) ? "DIRTY" : "CLEAN",
- buffer_locked(bh) ? "LOCKED" : "UNLOCKED");
+ return scnprintf(buf, size,
+ "dev %pg, size %zd, blocknr %llu, count %d, state 0x%lx, page %p, (%s, %s, %s)",
+ bh->b_bdev, bh->b_size,
+ (unsigned long long)bh->b_blocknr,
+ atomic_read(&(bh->b_count)),
+ bh->b_state, bh->b_page,
+ buffer_uptodate(bh) ? "UPTODATE" : "!UPTODATE",
+ buffer_dirty(bh) ? "DIRTY" : "CLEAN",
+ buffer_locked(bh) ? "LOCKED" : "UNLOCKED");
}
-static void sprintf_disk_child(char *buf, struct disk_child *dc)
+static int scnprintf_disk_child(char *buf, size_t size, struct disk_child *dc)
{
- sprintf(buf, "[dc_number=%d, dc_size=%u]", dc_block_number(dc),
- dc_size(dc));
+ return scnprintf(buf, size, "[dc_number=%d, dc_size=%u]",
+ dc_block_number(dc), dc_size(dc));
}
static char *is_there_reiserfs_struct(char *fmt, int *what)
@@ -189,55 +205,60 @@ static void prepare_error_buf(const char *fmt, va_list args)
char *fmt1 = fmt_buf;
char *k;
char *p = error_buf;
+ char * const end = &error_buf[sizeof(error_buf)];
int what;
spin_lock(&error_lock);
- strcpy(fmt1, fmt);
+ if (WARN_ON(strscpy(fmt_buf, fmt, sizeof(fmt_buf)) < 0)) {
+ strscpy(error_buf, "format string too long", end - error_buf);
+ goto out_unlock;
+ }
while ((k = is_there_reiserfs_struct(fmt1, &what)) != NULL) {
*k = 0;
- p += vsprintf(p, fmt1, args);
+ p += vscnprintf(p, end - p, fmt1, args);
switch (what) {
case 'k':
- sprintf_le_key(p, va_arg(args, struct reiserfs_key *));
+ p += scnprintf_le_key(p, end - p,
+ va_arg(args, struct reiserfs_key *));
break;
case 'K':
- sprintf_cpu_key(p, va_arg(args, struct cpu_key *));
+ p += scnprintf_cpu_key(p, end - p,
+ va_arg(args, struct cpu_key *));
break;
case 'h':
- sprintf_item_head(p, va_arg(args, struct item_head *));
+ p += scnprintf_item_head(p, end - p,
+ va_arg(args, struct item_head *));
break;
case 't':
- sprintf_direntry(p,
- va_arg(args,
- struct reiserfs_dir_entry *));
+ p += scnprintf_direntry(p, end - p,
+ va_arg(args, struct reiserfs_dir_entry *));
break;
case 'y':
- sprintf_disk_child(p,
- va_arg(args, struct disk_child *));
+ p += scnprintf_disk_child(p, end - p,
+ va_arg(args, struct disk_child *));
break;
case 'z':
- sprintf_block_head(p,
- va_arg(args, struct buffer_head *));
+ p += scnprintf_block_head(p, end - p,
+ va_arg(args, struct buffer_head *));
break;
case 'b':
- sprintf_buffer_head(p,
- va_arg(args, struct buffer_head *));
+ p += scnprintf_buffer_head(p, end - p,
+ va_arg(args, struct buffer_head *));
break;
case 'a':
- sprintf_de_head(p,
- va_arg(args,
- struct reiserfs_de_head *));
+ p += scnprintf_de_head(p, end - p,
+ va_arg(args, struct reiserfs_de_head *));
break;
}
- p += strlen(p);
fmt1 = k + 2;
}
- vsprintf(p, fmt1, args);
+ p += vscnprintf(p, end - p, fmt1, args);
+out_unlock:
spin_unlock(&error_lock);
}
diff --git a/fs/reiserfs/reiserfs.h b/fs/reiserfs/reiserfs.h
index 6ca0047..d920a64 100644
--- a/fs/reiserfs/reiserfs.h
+++ b/fs/reiserfs/reiserfs.h
@@ -270,7 +270,7 @@ struct reiserfs_journal_list {
struct mutex j_commit_mutex;
unsigned int j_trans_id;
- time_t j_timestamp;
+ time64_t j_timestamp; /* write-only but useful for crash dump analysis */
struct reiserfs_list_bitmap *j_list_bitmap;
struct buffer_head *j_commit_bh; /* commit buffer head */
struct reiserfs_journal_cnode *j_realblock;
diff --git a/fs/reiserfs/xattr.c b/fs/reiserfs/xattr.c
index e87aa21..9e313fc 100644
--- a/fs/reiserfs/xattr.c
+++ b/fs/reiserfs/xattr.c
@@ -184,6 +184,7 @@ struct reiserfs_dentry_buf {
struct dir_context ctx;
struct dentry *xadir;
int count;
+ int err;
struct dentry *dentries[8];
};
@@ -206,6 +207,7 @@ fill_with_dentries(struct dir_context *ctx, const char *name, int namelen,
dentry = lookup_one_len(name, dbuf->xadir, namelen);
if (IS_ERR(dentry)) {
+ dbuf->err = PTR_ERR(dentry);
return PTR_ERR(dentry);
} else if (d_really_is_negative(dentry)) {
/* A directory entry exists, but no file? */
@@ -214,6 +216,7 @@ fill_with_dentries(struct dir_context *ctx, const char *name, int namelen,
"not found for file %pd.\n",
dentry, dbuf->xadir);
dput(dentry);
+ dbuf->err = -EIO;
return -EIO;
}
@@ -261,6 +264,10 @@ static int reiserfs_for_each_xattr(struct inode *inode,
err = reiserfs_readdir_inode(d_inode(dir), &buf.ctx);
if (err)
break;
+ if (buf.err) {
+ err = buf.err;
+ break;
+ }
if (!buf.count)
break;
for (i = 0; !err && i < buf.count && buf.dentries[i]; i++) {
@@ -791,8 +798,10 @@ static int listxattr_filler(struct dir_context *ctx, const char *name,
return 0;
size = namelen + 1;
if (b->buf) {
- if (size > b->size)
+ if (b->pos + size > b->size) {
+ b->pos = -ERANGE;
return -ERANGE;
+ }
memcpy(b->buf + b->pos, name, namelen);
b->buf[b->pos + namelen] = 0;
}
diff --git a/fs/sdcardfs/file.c b/fs/sdcardfs/file.c
index 1461254..271c4c4 100644
--- a/fs/sdcardfs/file.c
+++ b/fs/sdcardfs/file.c
@@ -118,7 +118,11 @@ static long sdcardfs_unlocked_ioctl(struct file *file, unsigned int cmd,
goto out;
/* save current_cred and override it */
- OVERRIDE_CRED(sbi, saved_cred, SDCARDFS_I(file_inode(file)));
+ saved_cred = override_fsids(sbi, SDCARDFS_I(file_inode(file))->data);
+ if (!saved_cred) {
+ err = -ENOMEM;
+ goto out;
+ }
if (lower_file->f_op->unlocked_ioctl)
err = lower_file->f_op->unlocked_ioctl(lower_file, cmd, arg);
@@ -127,7 +131,7 @@ static long sdcardfs_unlocked_ioctl(struct file *file, unsigned int cmd,
if (!err)
sdcardfs_copy_and_fix_attrs(file_inode(file),
file_inode(lower_file));
- REVERT_CRED(saved_cred);
+ revert_fsids(saved_cred);
out:
return err;
}
@@ -149,12 +153,16 @@ static long sdcardfs_compat_ioctl(struct file *file, unsigned int cmd,
goto out;
/* save current_cred and override it */
- OVERRIDE_CRED(sbi, saved_cred, SDCARDFS_I(file_inode(file)));
+ saved_cred = override_fsids(sbi, SDCARDFS_I(file_inode(file))->data);
+ if (!saved_cred) {
+ err = -ENOMEM;
+ goto out;
+ }
if (lower_file->f_op->compat_ioctl)
err = lower_file->f_op->compat_ioctl(lower_file, cmd, arg);
- REVERT_CRED(saved_cred);
+ revert_fsids(saved_cred);
out:
return err;
}
@@ -241,7 +249,11 @@ static int sdcardfs_open(struct inode *inode, struct file *file)
}
/* save current_cred and override it */
- OVERRIDE_CRED(sbi, saved_cred, SDCARDFS_I(inode));
+ saved_cred = override_fsids(sbi, SDCARDFS_I(inode)->data);
+ if (!saved_cred) {
+ err = -ENOMEM;
+ goto out_err;
+ }
file->private_data =
kzalloc(sizeof(struct sdcardfs_file_info), GFP_KERNEL);
@@ -271,7 +283,7 @@ static int sdcardfs_open(struct inode *inode, struct file *file)
sdcardfs_copy_and_fix_attrs(inode, sdcardfs_lower_inode(inode));
out_revert_cred:
- REVERT_CRED(saved_cred);
+ revert_fsids(saved_cred);
out_err:
dput(parent);
return err;
diff --git a/fs/sdcardfs/inode.c b/fs/sdcardfs/inode.c
index fafc09c4..de34561 100644
--- a/fs/sdcardfs/inode.c
+++ b/fs/sdcardfs/inode.c
@@ -22,7 +22,6 @@
#include <linux/fs_struct.h>
#include <linux/ratelimit.h>
-/* Do not directly use this function. Use OVERRIDE_CRED() instead. */
const struct cred *override_fsids(struct sdcardfs_sb_info *sbi,
struct sdcardfs_inode_data *data)
{
@@ -50,7 +49,6 @@ const struct cred *override_fsids(struct sdcardfs_sb_info *sbi,
return old_cred;
}
-/* Do not directly use this function, use REVERT_CRED() instead. */
void revert_fsids(const struct cred *old_cred)
{
const struct cred *cur_cred;
@@ -78,7 +76,10 @@ static int sdcardfs_create(struct inode *dir, struct dentry *dentry,
}
/* save current_cred and override it */
- OVERRIDE_CRED(SDCARDFS_SB(dir->i_sb), saved_cred, SDCARDFS_I(dir));
+ saved_cred = override_fsids(SDCARDFS_SB(dir->i_sb),
+ SDCARDFS_I(dir)->data);
+ if (!saved_cred)
+ return -ENOMEM;
sdcardfs_get_lower_path(dentry, &lower_path);
lower_dentry = lower_path.dentry;
@@ -95,8 +96,11 @@ static int sdcardfs_create(struct inode *dir, struct dentry *dentry,
err = -ENOMEM;
goto out_unlock;
}
+ copied_fs->umask = 0;
+ task_lock(current);
current->fs = copied_fs;
- current->fs->umask = 0;
+ task_unlock(current);
+
err = vfs_create2(lower_dentry_mnt, d_inode(lower_parent_dentry), lower_dentry, mode, want_excl);
if (err)
goto out;
@@ -110,58 +114,18 @@ static int sdcardfs_create(struct inode *dir, struct dentry *dentry,
fixup_lower_ownership(dentry, dentry->d_name.name);
out:
+ task_lock(current);
current->fs = saved_fs;
+ task_unlock(current);
free_fs_struct(copied_fs);
out_unlock:
unlock_dir(lower_parent_dentry);
sdcardfs_put_lower_path(dentry, &lower_path);
- REVERT_CRED(saved_cred);
+ revert_fsids(saved_cred);
out_eacces:
return err;
}
-#if 0
-static int sdcardfs_link(struct dentry *old_dentry, struct inode *dir,
- struct dentry *new_dentry)
-{
- struct dentry *lower_old_dentry;
- struct dentry *lower_new_dentry;
- struct dentry *lower_dir_dentry;
- u64 file_size_save;
- int err;
- struct path lower_old_path, lower_new_path;
-
- OVERRIDE_CRED(SDCARDFS_SB(dir->i_sb));
-
- file_size_save = i_size_read(d_inode(old_dentry));
- sdcardfs_get_lower_path(old_dentry, &lower_old_path);
- sdcardfs_get_lower_path(new_dentry, &lower_new_path);
- lower_old_dentry = lower_old_path.dentry;
- lower_new_dentry = lower_new_path.dentry;
- lower_dir_dentry = lock_parent(lower_new_dentry);
-
- err = vfs_link(lower_old_dentry, d_inode(lower_dir_dentry),
- lower_new_dentry, NULL);
- if (err || !d_inode(lower_new_dentry))
- goto out;
-
- err = sdcardfs_interpose(new_dentry, dir->i_sb, &lower_new_path);
- if (err)
- goto out;
- fsstack_copy_attr_times(dir, d_inode(lower_new_dentry));
- fsstack_copy_inode_size(dir, d_inode(lower_new_dentry));
- set_nlink(d_inode(old_dentry),
- sdcardfs_lower_inode(d_inode(old_dentry))->i_nlink);
- i_size_write(d_inode(new_dentry), file_size_save);
-out:
- unlock_dir(lower_dir_dentry);
- sdcardfs_put_lower_path(old_dentry, &lower_old_path);
- sdcardfs_put_lower_path(new_dentry, &lower_new_path);
- REVERT_CRED();
- return err;
-}
-#endif
-
static int sdcardfs_unlink(struct inode *dir, struct dentry *dentry)
{
int err;
@@ -178,7 +142,10 @@ static int sdcardfs_unlink(struct inode *dir, struct dentry *dentry)
}
/* save current_cred and override it */
- OVERRIDE_CRED(SDCARDFS_SB(dir->i_sb), saved_cred, SDCARDFS_I(dir));
+ saved_cred = override_fsids(SDCARDFS_SB(dir->i_sb),
+ SDCARDFS_I(dir)->data);
+ if (!saved_cred)
+ return -ENOMEM;
sdcardfs_get_lower_path(dentry, &lower_path);
lower_dentry = lower_path.dentry;
@@ -209,43 +176,11 @@ static int sdcardfs_unlink(struct inode *dir, struct dentry *dentry)
unlock_dir(lower_dir_dentry);
dput(lower_dentry);
sdcardfs_put_lower_path(dentry, &lower_path);
- REVERT_CRED(saved_cred);
+ revert_fsids(saved_cred);
out_eacces:
return err;
}
-#if 0
-static int sdcardfs_symlink(struct inode *dir, struct dentry *dentry,
- const char *symname)
-{
- int err;
- struct dentry *lower_dentry;
- struct dentry *lower_parent_dentry = NULL;
- struct path lower_path;
-
- OVERRIDE_CRED(SDCARDFS_SB(dir->i_sb));
-
- sdcardfs_get_lower_path(dentry, &lower_path);
- lower_dentry = lower_path.dentry;
- lower_parent_dentry = lock_parent(lower_dentry);
-
- err = vfs_symlink(d_inode(lower_parent_dentry), lower_dentry, symname);
- if (err)
- goto out;
- err = sdcardfs_interpose(dentry, dir->i_sb, &lower_path);
- if (err)
- goto out;
- fsstack_copy_attr_times(dir, sdcardfs_lower_inode(dir));
- fsstack_copy_inode_size(dir, d_inode(lower_parent_dentry));
-
-out:
- unlock_dir(lower_parent_dentry);
- sdcardfs_put_lower_path(dentry, &lower_path);
- REVERT_CRED();
- return err;
-}
-#endif
-
static int touch(char *abs_path, mode_t mode)
{
struct file *filp = filp_open(abs_path, O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW, mode);
@@ -286,7 +221,10 @@ static int sdcardfs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode
}
/* save current_cred and override it */
- OVERRIDE_CRED(SDCARDFS_SB(dir->i_sb), saved_cred, SDCARDFS_I(dir));
+ saved_cred = override_fsids(SDCARDFS_SB(dir->i_sb),
+ SDCARDFS_I(dir)->data);
+ if (!saved_cred)
+ return -ENOMEM;
/* check disk space */
if (!check_min_free_space(dentry, 0, 1)) {
@@ -312,8 +250,11 @@ static int sdcardfs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode
unlock_dir(lower_parent_dentry);
goto out_unlock;
}
+ copied_fs->umask = 0;
+ task_lock(current);
current->fs = copied_fs;
- current->fs->umask = 0;
+ task_unlock(current);
+
err = vfs_mkdir2(lower_mnt, d_inode(lower_parent_dentry), lower_dentry, mode);
if (err) {
@@ -362,23 +303,34 @@ static int sdcardfs_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode
if (make_nomedia_in_obb ||
((pd->perm == PERM_ANDROID)
&& (qstr_case_eq(&dentry->d_name, &q_data)))) {
- REVERT_CRED(saved_cred);
- OVERRIDE_CRED(SDCARDFS_SB(dir->i_sb), saved_cred, SDCARDFS_I(d_inode(dentry)));
+ revert_fsids(saved_cred);
+ saved_cred = override_fsids(sbi,
+ SDCARDFS_I(d_inode(dentry))->data);
+ if (!saved_cred) {
+ pr_err("sdcardfs: failed to set up .nomedia in %s: %d\n",
+ lower_path.dentry->d_name.name,
+ -ENOMEM);
+ goto out;
+ }
set_fs_pwd(current->fs, &lower_path);
touch_err = touch(".nomedia", 0664);
if (touch_err) {
pr_err("sdcardfs: failed to create .nomedia in %s: %d\n",
- lower_path.dentry->d_name.name, touch_err);
+ lower_path.dentry->d_name.name,
+ touch_err);
goto out;
}
}
out:
+ task_lock(current);
current->fs = saved_fs;
+ task_unlock(current);
+
free_fs_struct(copied_fs);
out_unlock:
sdcardfs_put_lower_path(dentry, &lower_path);
out_revert:
- REVERT_CRED(saved_cred);
+ revert_fsids(saved_cred);
out_eacces:
return err;
}
@@ -398,7 +350,10 @@ static int sdcardfs_rmdir(struct inode *dir, struct dentry *dentry)
}
/* save current_cred and override it */
- OVERRIDE_CRED(SDCARDFS_SB(dir->i_sb), saved_cred, SDCARDFS_I(dir));
+ saved_cred = override_fsids(SDCARDFS_SB(dir->i_sb),
+ SDCARDFS_I(dir)->data);
+ if (!saved_cred)
+ return -ENOMEM;
/* sdcardfs_get_real_lower(): in case of remove an user's obb dentry
* the dentry on the original path should be deleted.
@@ -423,44 +378,11 @@ static int sdcardfs_rmdir(struct inode *dir, struct dentry *dentry)
out:
unlock_dir(lower_dir_dentry);
sdcardfs_put_real_lower(dentry, &lower_path);
- REVERT_CRED(saved_cred);
+ revert_fsids(saved_cred);
out_eacces:
return err;
}
-#if 0
-static int sdcardfs_mknod(struct inode *dir, struct dentry *dentry, umode_t mode,
- dev_t dev)
-{
- int err;
- struct dentry *lower_dentry;
- struct dentry *lower_parent_dentry = NULL;
- struct path lower_path;
-
- OVERRIDE_CRED(SDCARDFS_SB(dir->i_sb));
-
- sdcardfs_get_lower_path(dentry, &lower_path);
- lower_dentry = lower_path.dentry;
- lower_parent_dentry = lock_parent(lower_dentry);
-
- err = vfs_mknod(d_inode(lower_parent_dentry), lower_dentry, mode, dev);
- if (err)
- goto out;
-
- err = sdcardfs_interpose(dentry, dir->i_sb, &lower_path);
- if (err)
- goto out;
- fsstack_copy_attr_times(dir, sdcardfs_lower_inode(dir));
- fsstack_copy_inode_size(dir, d_inode(lower_parent_dentry));
-
-out:
- unlock_dir(lower_parent_dentry);
- sdcardfs_put_lower_path(dentry, &lower_path);
- REVERT_CRED();
- return err;
-}
-#endif
-
/*
* The locking rules in sdcardfs_rename are complex. We could use a simpler
* superblock-level name-space lock for renames and copy-ups.
@@ -489,7 +411,10 @@ static int sdcardfs_rename(struct inode *old_dir, struct dentry *old_dentry,
}
/* save current_cred and override it */
- OVERRIDE_CRED(SDCARDFS_SB(old_dir->i_sb), saved_cred, SDCARDFS_I(new_dir));
+ saved_cred = override_fsids(SDCARDFS_SB(old_dir->i_sb),
+ SDCARDFS_I(new_dir)->data);
+ if (!saved_cred)
+ return -ENOMEM;
sdcardfs_get_real_lower(old_dentry, &lower_old_path);
sdcardfs_get_lower_path(new_dentry, &lower_new_path);
@@ -536,7 +461,7 @@ static int sdcardfs_rename(struct inode *old_dir, struct dentry *old_dentry,
dput(lower_new_dir_dentry);
sdcardfs_put_real_lower(old_dentry, &lower_old_path);
sdcardfs_put_lower_path(new_dentry, &lower_new_path);
- REVERT_CRED(saved_cred);
+ revert_fsids(saved_cred);
out_eacces:
return err;
}
@@ -655,33 +580,7 @@ static int sdcardfs_permission(struct vfsmount *mnt, struct inode *inode, int ma
if (IS_POSIXACL(inode))
pr_warn("%s: This may be undefined behavior...\n", __func__);
err = generic_permission(&tmp, mask);
- /* XXX
- * Original sdcardfs code calls inode_permission(lower_inode,.. )
- * for checking inode permission. But doing such things here seems
- * duplicated work, because the functions called after this func,
- * such as vfs_create, vfs_unlink, vfs_rename, and etc,
- * does exactly same thing, i.e., they calls inode_permission().
- * So we just let they do the things.
- * If there are any security hole, just uncomment following if block.
- */
-#if 0
- if (!err) {
- /*
- * Permission check on lower_inode(=EXT4).
- * we check it with AID_MEDIA_RW permission
- */
- struct inode *lower_inode;
-
- OVERRIDE_CRED(SDCARDFS_SB(inode->sb));
-
- lower_inode = sdcardfs_lower_inode(inode);
- err = inode_permission(lower_inode, mask);
-
- REVERT_CRED();
- }
-#endif
return err;
-
}
static int sdcardfs_setattr_wrn(struct dentry *dentry, struct iattr *ia)
@@ -759,7 +658,10 @@ static int sdcardfs_setattr(struct vfsmount *mnt, struct dentry *dentry, struct
goto out_err;
/* save current_cred and override it */
- OVERRIDE_CRED(SDCARDFS_SB(dentry->d_sb), saved_cred, SDCARDFS_I(inode));
+ saved_cred = override_fsids(SDCARDFS_SB(dentry->d_sb),
+ SDCARDFS_I(inode)->data);
+ if (!saved_cred)
+ return -ENOMEM;
sdcardfs_get_lower_path(dentry, &lower_path);
lower_dentry = lower_path.dentry;
@@ -818,7 +720,7 @@ static int sdcardfs_setattr(struct vfsmount *mnt, struct dentry *dentry, struct
out:
sdcardfs_put_lower_path(dentry, &lower_path);
- REVERT_CRED(saved_cred);
+ revert_fsids(saved_cred);
out_err:
return err;
}
@@ -901,13 +803,6 @@ const struct inode_operations sdcardfs_dir_iops = {
.setattr = sdcardfs_setattr_wrn,
.setattr2 = sdcardfs_setattr,
.getattr = sdcardfs_getattr,
- /* XXX Following operations are implemented,
- * but FUSE(sdcard) or FAT does not support them
- * These methods are *NOT* perfectly tested.
- .symlink = sdcardfs_symlink,
- .link = sdcardfs_link,
- .mknod = sdcardfs_mknod,
- */
};
const struct inode_operations sdcardfs_main_iops = {
diff --git a/fs/sdcardfs/lookup.c b/fs/sdcardfs/lookup.c
index 843fcd2..a6e6615 100644
--- a/fs/sdcardfs/lookup.c
+++ b/fs/sdcardfs/lookup.c
@@ -18,6 +18,8 @@
* General Public License.
*/
+#define __FS_HAS_ENCRYPTION IS_ENABLED(CONFIG_FS_ENCRYPTION)
+#include <linux/fscrypt.h>
#include "sdcardfs.h"
#include "linux/delay.h"
@@ -276,6 +278,7 @@ static struct dentry *__sdcardfs_lookup(struct dentry *dentry,
lower_dir_dentry = lower_parent_path->dentry;
lower_dir_mnt = lower_parent_path->mnt;
+retry_lookup:
/* Use vfs_path_lookup to check if the dentry exists or not */
err = vfs_path_lookup(lower_dir_dentry, lower_dir_mnt, name->name, 0,
&lower_path);
@@ -379,8 +382,14 @@ static struct dentry *__sdcardfs_lookup(struct dentry *dentry,
* dentry then. Don't confuse the lower filesystem by forcing
* one on it now...
*/
- err = -ENOENT;
- goto out;
+ struct inode *lower_dir = d_inode(lower_dir_dentry);
+
+ if (IS_ENCRYPTED(lower_dir) &&
+ !fscrypt_has_encryption_key(lower_dir)) {
+ err = -ENOENT;
+ goto out;
+ }
+ goto retry_lookup;
}
lower_path.dentry = lower_dentry;
@@ -428,7 +437,12 @@ struct dentry *sdcardfs_lookup(struct inode *dir, struct dentry *dentry,
}
/* save current_cred and override it */
- OVERRIDE_CRED_PTR(SDCARDFS_SB(dir->i_sb), saved_cred, SDCARDFS_I(dir));
+ saved_cred = override_fsids(SDCARDFS_SB(dir->i_sb),
+ SDCARDFS_I(dir)->data);
+ if (!saved_cred) {
+ ret = ERR_PTR(-ENOMEM);
+ goto out_err;
+ }
sdcardfs_get_lower_path(parent, &lower_parent_path);
@@ -459,7 +473,7 @@ struct dentry *sdcardfs_lookup(struct inode *dir, struct dentry *dentry,
out:
sdcardfs_put_lower_path(parent, &lower_parent_path);
- REVERT_CRED(saved_cred);
+ revert_fsids(saved_cred);
out_err:
dput(parent);
return ret;
diff --git a/fs/sdcardfs/sdcardfs.h b/fs/sdcardfs/sdcardfs.h
index 826afb5..ec2290a 100644
--- a/fs/sdcardfs/sdcardfs.h
+++ b/fs/sdcardfs/sdcardfs.h
@@ -88,31 +88,6 @@
(x)->i_mode = ((x)->i_mode & S_IFMT) | 0775;\
} while (0)
-/* OVERRIDE_CRED() and REVERT_CRED()
- * OVERRIDE_CRED()
- * backup original task->cred
- * and modifies task->cred->fsuid/fsgid to specified value.
- * REVERT_CRED()
- * restore original task->cred->fsuid/fsgid.
- * These two macro should be used in pair, and OVERRIDE_CRED() should be
- * placed at the beginning of a function, right after variable declaration.
- */
-#define OVERRIDE_CRED(sdcardfs_sbi, saved_cred, info) \
- do { \
- saved_cred = override_fsids(sdcardfs_sbi, info->data); \
- if (!saved_cred) \
- return -ENOMEM; \
- } while (0)
-
-#define OVERRIDE_CRED_PTR(sdcardfs_sbi, saved_cred, info) \
- do { \
- saved_cred = override_fsids(sdcardfs_sbi, info->data); \
- if (!saved_cred) \
- return ERR_PTR(-ENOMEM); \
- } while (0)
-
-#define REVERT_CRED(saved_cred) revert_fsids(saved_cred)
-
/* Android 5.0 support */
/* Permission mode for a specific node. Controls how file permissions
diff --git a/fs/splice.c b/fs/splice.c
index 8dd79ec..14b98c1 100644
--- a/fs/splice.c
+++ b/fs/splice.c
@@ -710,7 +710,7 @@ iter_file_splice_write(struct pipe_inode_info *pipe, struct file *out,
int nbufs = pipe->buffers;
struct bio_vec *array = kcalloc(nbufs, sizeof(struct bio_vec),
GFP_KERNEL);
- ssize_t ret;
+ ssize_t ret = -EINVAL;
if (unlikely(!array))
return -ENOMEM;
diff --git a/fs/squashfs/cache.c b/fs/squashfs/cache.c
index 05e4244..9d9d4aa 100644
--- a/fs/squashfs/cache.c
+++ b/fs/squashfs/cache.c
@@ -340,6 +340,9 @@ int squashfs_read_metadata(struct super_block *sb, void *buffer,
TRACE("Entered squashfs_read_metadata [%llx:%x]\n", *block, *offset);
+ if (unlikely(length < 0))
+ return -EIO;
+
while (length) {
entry = squashfs_cache_get(sb, msblk->block_cache, *block, 0);
if (entry->error) {
diff --git a/fs/squashfs/file.c b/fs/squashfs/file.c
index bb2e77e..cd3c5c8 100644
--- a/fs/squashfs/file.c
+++ b/fs/squashfs/file.c
@@ -195,7 +195,11 @@ static long long read_indexes(struct super_block *sb, int n,
}
for (i = 0; i < blocks; i++) {
- int size = le32_to_cpu(blist[i]);
+ int size = squashfs_block_size(blist[i]);
+ if (size < 0) {
+ err = size;
+ goto failure;
+ }
block += SQUASHFS_COMPRESSED_SIZE_BLOCK(size);
}
n -= blocks;
@@ -368,7 +372,7 @@ static int read_blocklist(struct inode *inode, int index, u64 *block)
sizeof(size));
if (res < 0)
return res;
- return le32_to_cpu(size);
+ return squashfs_block_size(size);
}
/* Copy data into page cache */
diff --git a/fs/squashfs/fragment.c b/fs/squashfs/fragment.c
index 0ed6edb..0681fea 100644
--- a/fs/squashfs/fragment.c
+++ b/fs/squashfs/fragment.c
@@ -49,11 +49,16 @@ int squashfs_frag_lookup(struct super_block *sb, unsigned int fragment,
u64 *fragment_block)
{
struct squashfs_sb_info *msblk = sb->s_fs_info;
- int block = SQUASHFS_FRAGMENT_INDEX(fragment);
- int offset = SQUASHFS_FRAGMENT_INDEX_OFFSET(fragment);
- u64 start_block = le64_to_cpu(msblk->fragment_index[block]);
+ int block, offset, size;
struct squashfs_fragment_entry fragment_entry;
- int size;
+ u64 start_block;
+
+ if (fragment >= msblk->fragments)
+ return -EIO;
+ block = SQUASHFS_FRAGMENT_INDEX(fragment);
+ offset = SQUASHFS_FRAGMENT_INDEX_OFFSET(fragment);
+
+ start_block = le64_to_cpu(msblk->fragment_index[block]);
size = squashfs_read_metadata(sb, &fragment_entry, &start_block,
&offset, sizeof(fragment_entry));
@@ -61,9 +66,7 @@ int squashfs_frag_lookup(struct super_block *sb, unsigned int fragment,
return size;
*fragment_block = le64_to_cpu(fragment_entry.start_block);
- size = le32_to_cpu(fragment_entry.size);
-
- return size;
+ return squashfs_block_size(fragment_entry.size);
}
diff --git a/fs/squashfs/squashfs_fs.h b/fs/squashfs/squashfs_fs.h
index 506f4ba..e6648636 100644
--- a/fs/squashfs/squashfs_fs.h
+++ b/fs/squashfs/squashfs_fs.h
@@ -129,6 +129,12 @@
#define SQUASHFS_COMPRESSED_BLOCK(B) (!((B) & SQUASHFS_COMPRESSED_BIT_BLOCK))
+static inline int squashfs_block_size(__le32 raw)
+{
+ u32 size = le32_to_cpu(raw);
+ return (size >> 25) ? -EIO : size;
+}
+
/*
* Inode number ops. Inodes consist of a compressed block number, and an
* uncompressed offset within that block
diff --git a/fs/squashfs/squashfs_fs_sb.h b/fs/squashfs/squashfs_fs_sb.h
index 8a6995d..3b767ce 100644
--- a/fs/squashfs/squashfs_fs_sb.h
+++ b/fs/squashfs/squashfs_fs_sb.h
@@ -75,6 +75,7 @@ struct squashfs_sb_info {
unsigned short block_log;
long long bytes_used;
unsigned int inodes;
+ unsigned int fragments;
int xattr_ids;
};
#endif
diff --git a/fs/squashfs/super.c b/fs/squashfs/super.c
index e2a0a73..445ce58 100644
--- a/fs/squashfs/super.c
+++ b/fs/squashfs/super.c
@@ -175,6 +175,7 @@ static int squashfs_fill_super(struct super_block *sb, void *data, int silent)
msblk->inode_table = le64_to_cpu(sblk->inode_table_start);
msblk->directory_table = le64_to_cpu(sblk->directory_table_start);
msblk->inodes = le32_to_cpu(sblk->inodes);
+ msblk->fragments = le32_to_cpu(sblk->fragments);
flags = le16_to_cpu(sblk->flags);
TRACE("Found valid superblock on %pg\n", sb->s_bdev);
@@ -185,7 +186,7 @@ static int squashfs_fill_super(struct super_block *sb, void *data, int silent)
TRACE("Filesystem size %lld bytes\n", msblk->bytes_used);
TRACE("Block size %d\n", msblk->block_size);
TRACE("Number of inodes %d\n", msblk->inodes);
- TRACE("Number of fragments %d\n", le32_to_cpu(sblk->fragments));
+ TRACE("Number of fragments %d\n", msblk->fragments);
TRACE("Number of ids %d\n", le16_to_cpu(sblk->no_ids));
TRACE("sblk->inode_table_start %llx\n", msblk->inode_table);
TRACE("sblk->directory_table_start %llx\n", msblk->directory_table);
@@ -272,7 +273,7 @@ static int squashfs_fill_super(struct super_block *sb, void *data, int silent)
sb->s_export_op = &squashfs_export_ops;
handle_fragments:
- fragments = le32_to_cpu(sblk->fragments);
+ fragments = msblk->fragments;
if (fragments == 0)
goto check_directory_table;
diff --git a/fs/super.c b/fs/super.c
index ce404a2..5861cc3 100644
--- a/fs/super.c
+++ b/fs/super.c
@@ -215,7 +215,11 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags,
spin_lock_init(&s->s_inode_list_lock);
INIT_LIST_HEAD(&s->s_inodes_wb);
spin_lock_init(&s->s_inode_wblist_lock);
-
+#if IS_ENABLED(CONFIG_FS_VERITY)
+ /* TODO(mhalcrow): Not for upstream */
+ INIT_LIST_HEAD(&s->s_inodes_fsverity);
+ spin_lock_init(&s->s_inode_fsveritylist_lock);
+#endif
if (list_lru_init_memcg(&s->s_dentry_lru))
goto fail;
if (list_lru_init_memcg(&s->s_inode_lru))
@@ -400,6 +404,29 @@ bool trylock_super(struct super_block *sb)
return false;
}
+/* TODO(mhalcrow): Not for upstream (fsverity list on the sb) */
+#if IS_ENABLED(CONFIG_FS_VERITY)
+static void fsverity_unmount_inodes(struct super_block *sb)
+{
+ struct inode *inode, *tmp;
+
+ /*
+ * No need to take the spinlock; the filesystem is going away and can't
+ * have any open files via which the list can be added to. And iput()
+ * can sleep, so it can't be called while holding a spinlock.
+ */
+ list_for_each_entry_safe(inode, tmp, &sb->s_inodes_fsverity,
+ i_fsverity_list) {
+ list_del_init(&inode->i_fsverity_list);
+ iput(inode);
+ }
+}
+#else
+static inline void fsverity_unmount_inodes(struct super_block *sb)
+{
+}
+#endif /* !CONFIG_FS_VERITY */
+
/**
* generic_shutdown_super - common helper for ->kill_sb()
* @sb: superblock to kill
@@ -425,6 +452,7 @@ void generic_shutdown_super(struct super_block *sb)
fsnotify_unmount_inodes(sb);
cgroup_writeback_umount();
+ fsverity_unmount_inodes(sb);
evict_inodes(sb);
diff --git a/fs/sysfs/file.c b/fs/sysfs/file.c
index 39c75a86..666986b 100644
--- a/fs/sysfs/file.c
+++ b/fs/sysfs/file.c
@@ -408,6 +408,50 @@ int sysfs_chmod_file(struct kobject *kobj, const struct attribute *attr,
EXPORT_SYMBOL_GPL(sysfs_chmod_file);
/**
+ * sysfs_break_active_protection - break "active" protection
+ * @kobj: The kernel object @attr is associated with.
+ * @attr: The attribute to break the "active" protection for.
+ *
+ * With sysfs, just like kernfs, deletion of an attribute is postponed until
+ * all active .show() and .store() callbacks have finished unless this function
+ * is called. Hence this function is useful in methods that implement self
+ * deletion.
+ */
+struct kernfs_node *sysfs_break_active_protection(struct kobject *kobj,
+ const struct attribute *attr)
+{
+ struct kernfs_node *kn;
+
+ kobject_get(kobj);
+ kn = kernfs_find_and_get(kobj->sd, attr->name);
+ if (kn)
+ kernfs_break_active_protection(kn);
+ return kn;
+}
+EXPORT_SYMBOL_GPL(sysfs_break_active_protection);
+
+/**
+ * sysfs_unbreak_active_protection - restore "active" protection
+ * @kn: Pointer returned by sysfs_break_active_protection().
+ *
+ * Undo the effects of sysfs_break_active_protection(). Since this function
+ * calls kernfs_put() on the kernfs node that corresponds to the 'attr'
+ * argument passed to sysfs_break_active_protection() that attribute may have
+ * been removed between the sysfs_break_active_protection() and
+ * sysfs_unbreak_active_protection() calls, it is not safe to access @kn after
+ * this function has returned.
+ */
+void sysfs_unbreak_active_protection(struct kernfs_node *kn)
+{
+ struct kobject *kobj = kn->parent->priv;
+
+ kernfs_unbreak_active_protection(kn);
+ kernfs_put(kn);
+ kobject_put(kobj);
+}
+EXPORT_SYMBOL_GPL(sysfs_unbreak_active_protection);
+
+/**
* sysfs_remove_file_ns - remove an object attribute with a custom ns tag
* @kobj: object we're acting for
* @attr: attribute descriptor
diff --git a/fs/sysv/inode.c b/fs/sysv/inode.c
index d62c423..7b391b4 100644
--- a/fs/sysv/inode.c
+++ b/fs/sysv/inode.c
@@ -275,7 +275,7 @@ static int __sysv_write_inode(struct inode *inode, int wait)
}
}
brelse(bh);
- return 0;
+ return err;
}
int sysv_write_inode(struct inode *inode, struct writeback_control *wbc)
diff --git a/fs/timerfd.c b/fs/timerfd.c
index ab8dd15..147b723 100644
--- a/fs/timerfd.c
+++ b/fs/timerfd.c
@@ -44,6 +44,8 @@ struct timerfd_ctx {
bool might_cancel;
};
+static atomic_t instance_count = ATOMIC_INIT(0);
+
static LIST_HEAD(cancel_list);
static DEFINE_SPINLOCK(cancel_lock);
@@ -387,6 +389,9 @@ SYSCALL_DEFINE2(timerfd_create, int, clockid, int, flags)
{
int ufd;
struct timerfd_ctx *ctx;
+ char task_comm_buf[TASK_COMM_LEN];
+ char file_name_buf[32];
+ int instance;
/* Check the TFD_* constants for consistency. */
BUILD_BUG_ON(TFD_CLOEXEC != O_CLOEXEC);
@@ -423,7 +428,12 @@ SYSCALL_DEFINE2(timerfd_create, int, clockid, int, flags)
ctx->moffs = ktime_mono_to_real((ktime_t){ .tv64 = 0 });
- ufd = anon_inode_getfd("[timerfd]", &timerfd_fops, ctx,
+ instance = atomic_inc_return(&instance_count);
+ get_task_comm(task_comm_buf, current);
+ snprintf(file_name_buf, sizeof(file_name_buf), "[timerfd%d_%.*s]",
+ instance, (int)sizeof(task_comm_buf), task_comm_buf);
+
+ ufd = anon_inode_getfd(file_name_buf, &timerfd_fops, ctx,
O_RDWR | (flags & TFD_SHARED_FCNTL_FLAGS));
if (ufd < 0)
kfree(ctx);
diff --git a/fs/ubifs/journal.c b/fs/ubifs/journal.c
index 7d764e3..f8ce849 100644
--- a/fs/ubifs/journal.c
+++ b/fs/ubifs/journal.c
@@ -661,6 +661,11 @@ int ubifs_jnl_update(struct ubifs_info *c, const struct inode *dir,
spin_lock(&ui->ui_lock);
ui->synced_i_size = ui->ui_size;
spin_unlock(&ui->ui_lock);
+ if (xent) {
+ spin_lock(&host_ui->ui_lock);
+ host_ui->synced_i_size = host_ui->ui_size;
+ spin_unlock(&host_ui->ui_lock);
+ }
mark_inode_clean(c, ui);
mark_inode_clean(c, host_ui);
return 0;
@@ -1344,7 +1349,16 @@ int ubifs_jnl_truncate(struct ubifs_info *c, const struct inode *inode,
else if (err)
goto out_free;
else {
- if (le32_to_cpu(dn->size) <= dlen)
+ int dn_len = le32_to_cpu(dn->size);
+
+ if (dn_len <= 0 || dn_len > UBIFS_BLOCK_SIZE) {
+ ubifs_err(c, "bad data node (block %u, inode %lu)",
+ blk, inode->i_ino);
+ ubifs_dump_node(c, dn);
+ goto out_free;
+ }
+
+ if (dn_len <= dlen)
dlen = 0; /* Nothing to do */
else {
int compr_type = le16_to_cpu(dn->compr_type);
diff --git a/fs/ubifs/lprops.c b/fs/ubifs/lprops.c
index 6c3a1ab..780a436 100644
--- a/fs/ubifs/lprops.c
+++ b/fs/ubifs/lprops.c
@@ -1091,10 +1091,6 @@ static int scan_check_cb(struct ubifs_info *c,
}
}
- buf = __vmalloc(c->leb_size, GFP_NOFS, PAGE_KERNEL);
- if (!buf)
- return -ENOMEM;
-
/*
* After an unclean unmount, empty and freeable LEBs
* may contain garbage - do not scan them.
@@ -1113,6 +1109,10 @@ static int scan_check_cb(struct ubifs_info *c,
return LPT_SCAN_CONTINUE;
}
+ buf = __vmalloc(c->leb_size, GFP_NOFS, PAGE_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
sleb = ubifs_scan(c, lnum, 0, buf, 0);
if (IS_ERR(sleb)) {
ret = PTR_ERR(sleb);
diff --git a/fs/ubifs/replay.c b/fs/ubifs/replay.c
index fb0f44c..de7799a 100644
--- a/fs/ubifs/replay.c
+++ b/fs/ubifs/replay.c
@@ -210,6 +210,38 @@ static int trun_remove_range(struct ubifs_info *c, struct replay_entry *r)
}
/**
+ * inode_still_linked - check whether inode in question will be re-linked.
+ * @c: UBIFS file-system description object
+ * @rino: replay entry to test
+ *
+ * O_TMPFILE files can be re-linked, this means link count goes from 0 to 1.
+ * This case needs special care, otherwise all references to the inode will
+ * be removed upon the first replay entry of an inode with link count 0
+ * is found.
+ */
+static bool inode_still_linked(struct ubifs_info *c, struct replay_entry *rino)
+{
+ struct replay_entry *r;
+
+ ubifs_assert(rino->deletion);
+ ubifs_assert(key_type(c, &rino->key) == UBIFS_INO_KEY);
+
+ /*
+ * Find the most recent entry for the inode behind @rino and check
+ * whether it is a deletion.
+ */
+ list_for_each_entry_reverse(r, &c->replay_list, list) {
+ ubifs_assert(r->sqnum >= rino->sqnum);
+ if (key_inum(c, &r->key) == key_inum(c, &rino->key))
+ return r->deletion == 0;
+
+ }
+
+ ubifs_assert(0);
+ return false;
+}
+
+/**
* apply_replay_entry - apply a replay entry to the TNC.
* @c: UBIFS file-system description object
* @r: replay entry to apply
@@ -239,6 +271,11 @@ static int apply_replay_entry(struct ubifs_info *c, struct replay_entry *r)
{
ino_t inum = key_inum(c, &r->key);
+ if (inode_still_linked(c, r)) {
+ err = 0;
+ break;
+ }
+
err = ubifs_tnc_remove_ino(c, inum);
break;
}
diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c
index 03dda1c..727a9e3 100644
--- a/fs/ubifs/super.c
+++ b/fs/ubifs/super.c
@@ -1918,6 +1918,9 @@ static struct ubi_volume_desc *open_ubi(const char *name, int mode)
int dev, vol;
char *endptr;
+ if (!name || !*name)
+ return ERR_PTR(-EINVAL);
+
/* First, try to open using the device node path method */
ubi = ubi_open_volume_path(name, mode);
if (!IS_ERR(ubi))
diff --git a/fs/udf/directory.c b/fs/udf/directory.c
index 988d5352..48ef184 100644
--- a/fs/udf/directory.c
+++ b/fs/udf/directory.c
@@ -150,6 +150,9 @@ struct fileIdentDesc *udf_fileident_read(struct inode *dir, loff_t *nf_pos,
sizeof(struct fileIdentDesc));
}
}
+ /* Got last entry outside of dir size - fs is corrupted! */
+ if (*nf_pos > dir->i_size)
+ return NULL;
return fi;
}
diff --git a/fs/udf/namei.c b/fs/udf/namei.c
index 2d65e28..348b922 100644
--- a/fs/udf/namei.c
+++ b/fs/udf/namei.c
@@ -621,8 +621,7 @@ static int udf_add_nondir(struct dentry *dentry, struct inode *inode)
if (fibh.sbh != fibh.ebh)
brelse(fibh.ebh);
brelse(fibh.sbh);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
return 0;
}
@@ -732,8 +731,7 @@ static int udf_mkdir(struct inode *dir, struct dentry *dentry, umode_t mode)
inc_nlink(dir);
dir->i_ctime = dir->i_mtime = current_time(dir);
mark_inode_dirty(dir);
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
if (fibh.sbh != fibh.ebh)
brelse(fibh.ebh);
brelse(fibh.sbh);
diff --git a/fs/udf/super.c b/fs/udf/super.c
index 4b1f6d5..03369a8 100644
--- a/fs/udf/super.c
+++ b/fs/udf/super.c
@@ -929,16 +929,20 @@ static int udf_load_pvoldesc(struct super_block *sb, sector_t block)
}
ret = udf_dstrCS0toUTF8(outstr, 31, pvoldesc->volIdent, 32);
- if (ret < 0)
- goto out_bh;
-
- strncpy(UDF_SB(sb)->s_volume_ident, outstr, ret);
+ if (ret < 0) {
+ strcpy(UDF_SB(sb)->s_volume_ident, "InvalidName");
+ pr_warn("incorrect volume identification, setting to "
+ "'InvalidName'\n");
+ } else {
+ strncpy(UDF_SB(sb)->s_volume_ident, outstr, ret);
+ }
udf_debug("volIdent[] = '%s'\n", UDF_SB(sb)->s_volume_ident);
ret = udf_dstrCS0toUTF8(outstr, 127, pvoldesc->volSetIdent, 128);
- if (ret < 0)
+ if (ret < 0) {
+ ret = 0;
goto out_bh;
-
+ }
outstr[ret] = 0;
udf_debug("volSetIdent[] = '%s'\n", outstr);
@@ -2094,8 +2098,9 @@ static int udf_fill_super(struct super_block *sb, void *options, int silent)
bool lvid_open = false;
uopt.flags = (1 << UDF_FLAG_USE_AD_IN_ICB) | (1 << UDF_FLAG_STRICT);
- uopt.uid = INVALID_UID;
- uopt.gid = INVALID_GID;
+ /* By default we'll use overflow[ug]id when UDF inode [ug]id == -1 */
+ uopt.uid = make_kuid(current_user_ns(), overflowuid);
+ uopt.gid = make_kgid(current_user_ns(), overflowgid);
uopt.umask = 0;
uopt.fmode = UDF_INVALID_MODE;
uopt.dmode = UDF_INVALID_MODE;
diff --git a/fs/udf/unicode.c b/fs/udf/unicode.c
index 3a3be23..61a1738 100644
--- a/fs/udf/unicode.c
+++ b/fs/udf/unicode.c
@@ -341,6 +341,11 @@ static int udf_name_to_CS0(uint8_t *ocu, int ocu_max_len,
return u_len;
}
+/*
+ * Convert CS0 dstring to output charset. Warning: This function may truncate
+ * input string if it is too long as it is used for informational strings only
+ * and it is better to truncate the string than to refuse mounting a media.
+ */
int udf_dstrCS0toUTF8(uint8_t *utf_o, int o_len,
const uint8_t *ocu_i, int i_len)
{
@@ -349,9 +354,12 @@ int udf_dstrCS0toUTF8(uint8_t *utf_o, int o_len,
if (i_len > 0) {
s_len = ocu_i[i_len - 1];
if (s_len >= i_len) {
- pr_err("incorrect dstring lengths (%d/%d)\n",
- s_len, i_len);
- return -EINVAL;
+ pr_warn("incorrect dstring lengths (%d/%d),"
+ " truncating\n", s_len, i_len);
+ s_len = i_len - 1;
+ /* 2-byte encoding? Need to round properly... */
+ if (ocu_i[0] == 16)
+ s_len -= (s_len - 1) & 2;
}
}
diff --git a/fs/ufs/inode.c b/fs/ufs/inode.c
index a2760a2..0f22c036 100644
--- a/fs/ufs/inode.c
+++ b/fs/ufs/inode.c
@@ -846,6 +846,7 @@ void ufs_evict_inode(struct inode * inode)
inode->i_size = 0;
if (inode->i_blocks)
ufs_truncate_blocks(inode);
+ ufs_update_inode(inode, inode_needs_sync(inode));
}
invalidate_inode_buffers(inode);
diff --git a/fs/ufs/namei.c b/fs/ufs/namei.c
index 8eca4ed..2109c07 100644
--- a/fs/ufs/namei.c
+++ b/fs/ufs/namei.c
@@ -38,8 +38,7 @@ static inline int ufs_add_nondir(struct dentry *dentry, struct inode *inode)
{
int err = ufs_add_link(dentry, inode);
if (!err) {
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
return 0;
}
inode_dec_link_count(inode);
@@ -192,8 +191,7 @@ static int ufs_mkdir(struct inode * dir, struct dentry * dentry, umode_t mode)
if (err)
goto out_fail;
- unlock_new_inode(inode);
- d_instantiate(dentry, inode);
+ d_instantiate_new(dentry, inode);
return 0;
out_fail:
diff --git a/fs/verity/Kconfig b/fs/verity/Kconfig
new file mode 100644
index 0000000..8614367
--- /dev/null
+++ b/fs/verity/Kconfig
@@ -0,0 +1,31 @@
+config FS_VERITY
+ tristate "FS Verity (File-based authentication)"
+ depends on BLOCK
+ select CRYPTO
+ select CRYPTO_CRC32
+ select CRYPTO_SHA256
+ select CRYPTO_HASH
+ help
+ This option enables file-based verity feature. The
+ filesystem checks data blocks' authenticity when reading
+ them from underlying disk, similar to how dm-verity does
+ this when reading from a block device.
+
+ If unsure, say N.
+
+config FS_VERITY_DEBUG
+ bool "FS Verity debugging"
+ depends on FS_VERITY
+ help
+ Enable debugging messages related to fs-verity.
+
+ Say N unless you are an fs-verity developer.
+
+config FS_VERITY_INTEGRITY_ONLY
+ bool "FS Verity integrity-only mode (EXPERIMENTAL)"
+ depends on FS_VERITY
+ help
+ Enable experimental integrity-only mode for fs-verity.
+ The on-disk format of such files is not stable.
+
+ Say N unless you are an fs-verity developer.
diff --git a/fs/verity/Makefile b/fs/verity/Makefile
new file mode 100644
index 0000000..ad8113e
--- /dev/null
+++ b/fs/verity/Makefile
@@ -0,0 +1,3 @@
+obj-$(CONFIG_FS_VERITY) += fsverity.o
+
+fsverity-y := verity.o
diff --git a/fs/verity/fsverity_private.h b/fs/verity/fsverity_private.h
new file mode 100644
index 0000000..1492c85
--- /dev/null
+++ b/fs/verity/fsverity_private.h
@@ -0,0 +1,153 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * fs-verity: read-only file-based integrity/authenticity
+ *
+ * Copyright (C) 2018, Google, Inc.
+ *
+ * Written by Jaegeuk Kim, Michael Halcrow, and Eric Biggers.
+ */
+
+#ifndef _FSVERITY_PRIVATE_H
+#define _FSVERITY_PRIVATE_H
+
+#ifdef CONFIG_FS_VERITY_DEBUG
+#define DEBUG
+#endif
+
+#define pr_fmt(fmt) "fs-verity: " fmt
+
+#include <crypto/sha.h>
+#define __FS_HAS_VERITY 1
+#include <linux/fsverity.h>
+
+#define FS_VERITY_MAGIC "TrueBrew"
+#define FS_VERITY_MAJOR 1
+#define FS_VERITY_MINOR 0
+#define FS_VERITY_SALT_SIZE 8
+
+struct fsverity_footer {
+ u8 magic[8]; /* must be FS_VERITY_MAGIC */
+ u8 major_version; /* must be FS_VERITY_MAJOR */
+ u8 minor_version; /* must be FS_VERITY_MINOR */
+ u8 log_blocksize; /* log2(data-bytes-per-hash), e.g. 12 for 4KB */
+ u8 log_arity; /* log2(leaves-per-node), e.g. 7 for SHA-256 */
+ __le16 meta_algorithm; /* hash algorithm for tree blocks */
+ __le16 data_algorithm; /* hash algorithm for data blocks */
+ __le32 flags; /* flags */
+ __le32 reserved1; /* must be 0 */
+ __le64 size; /* size of the original, unpadded data */
+ u8 authenticated_ext_count; /* number of authenticated extensions */
+ u8 unauthenticated_ext_count; /* number of unauthenticated extensions */
+ u8 salt[FS_VERITY_SALT_SIZE]; /* used to salt the hashes */
+ u8 reserved2[22]; /* must be 0 */
+ /* This structure is 64 bytes long */
+} __packed;
+
+#define FS_VERITY_FLAG_INTEGRITY_ONLY 0x00000001
+
+/* extension types */
+#define FS_VERITY_EXT_ELIDE 1
+#define FS_VERITY_EXT_PATCH 2
+#define FS_VERITY_EXT_SALT 3
+#define FS_VERITY_EXT_PKCS7_SIGNATURE 4
+
+/* Header of each variable-length metadata item following the footer */
+struct fsverity_extension {
+ /*
+ * Length of this extension in bytes, including this footer. Must be
+ * rounded up to an 8-byte boundary when advancing to the next
+ * extension.
+ */
+ __le32 length;
+ __le16 type; /* Type of this extension (see codes above) */
+ __le16 reserved; /* Reserved, must be 0 */
+} __packed;
+
+/* On-disk format */
+struct fsverity_extension_elide {
+ __le64 offset;
+ __le64 length;
+} __packed;
+
+/* In-kernel struct */
+struct fsverity_elision {
+ struct list_head link;
+ pgoff_t index;
+ pgoff_t nr_pages;
+};
+
+/* On-disk format */
+struct fsverity_extension_patch {
+ __le64 offset;
+ u8 databytes[];
+} __packed;
+
+#define FS_VERITY_MAX_PATCH_SIZE 255
+
+/* In-kernel struct */
+struct fsverity_patch {
+ struct list_head link;
+ pgoff_t index;
+ unsigned int offset;
+ unsigned int length;
+ u8 patch[];
+};
+
+/*
+ * Up to 64 levels are theoretically possible with a very small block size, but
+ * we'd like to limit stack usage, and in practice this is plenty. E.g., with
+ * SHA-256 and 4K blocks, a file with size UINT64_MAX bytes needs just 8 levels.
+ */
+#define FS_VERITY_MAX_LEVELS 16
+
+#define FS_VERITY_MAX_DIGEST_SIZE SHA256_DIGEST_SIZE
+
+struct fsverity_hash_alg;
+
+/* Mode of an fs-verity file */
+enum fsverity_mode {
+ /* Root of trust not provided yet, reads will succeed with warnings */
+ FS_VERITY_MODE_NEED_AUTHENTICATION,
+
+ /* File contents don't match root of trust, reads will fail */
+ FS_VERITY_MODE_AUTHENTICATION_FAILED,
+
+ /* File contents match root of trust, reads will succeed */
+ FS_VERITY_MODE_AUTHENTICATED,
+
+ /* Integrity-only mode with no root of trust, reads will succeed */
+ FS_VERITY_MODE_INTEGRITY_ONLY,
+};
+
+
+/**
+ * fsverity_info - cached fs-verity metadata for an inode
+ *
+ * When a fs-verity file is first opened, an instance of this struct is
+ * allocated and stored in ->i_verity_info. It caches various values from the
+ * fs-verity metadata, such as the tree topology and the root hash, which are
+ * needed to efficiently verify data read from the file. Once created, it
+ * remains until the inode is evicted.
+ */
+struct fsverity_info {
+ const struct fsverity_hash_alg *hash_alg; /* hash algorithm */
+ u8 block_bits; /* log2(block size) */
+ u8 log_arity; /* log2(hashes per hash block) */
+ u8 depth; /* depth of the Merkle tree */
+ enum fsverity_mode mode; /* current mode of the file */
+ u8 salt[FS_VERITY_SALT_SIZE]; /* used to salt the hashes */
+ u64 data_i_size; /* original file size */
+ u64 elided_i_size; /* original file size after elisions */
+ u64 full_i_size; /* full file size including metadata */
+ u8 root_hash[FS_VERITY_MAX_DIGEST_SIZE]; /* Merkle tree root hash */
+ u8 measurement[FS_VERITY_MAX_DIGEST_SIZE]; /* file measurement */
+
+ /* Starting blocks for each tree level. 'depth-1' is the root level. */
+ u64 hash_lvl_region_idx[FS_VERITY_MAX_LEVELS];
+
+ /* Optional elide and patch extensions, sorted by increasing offset */
+ struct list_head elisions;
+ struct list_head patches;
+};
+
+#endif /* _FSVERITY_PRIVATE_H */
diff --git a/fs/verity/verity.c b/fs/verity/verity.c
new file mode 100644
index 0000000..1c96703
--- /dev/null
+++ b/fs/verity/verity.c
@@ -0,0 +1,1614 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * fs-verity: read-only file-based integrity/authenticity
+ *
+ * Copyright (C) 2018, Google, Inc.
+ *
+ * Written by Jaegeuk Kim, Michael Halcrow, and Eric Biggers.
+ */
+
+#include "fsverity_private.h"
+
+#include <asm/unaligned.h>
+#include <crypto/hash.h>
+#include <linux/bio.h>
+#include <linux/list.h>
+#include <linux/list_sort.h>
+#include <linux/module.h>
+#include <linux/mount.h>
+#include <linux/printk.h>
+#include <linux/ratelimit.h>
+#include <linux/slab.h>
+#include <linux/uaccess.h>
+#include <linux/workqueue.h>
+
+/* Allocated at module initialization time */
+static struct workqueue_struct *fsverity_read_workqueue;
+static struct kmem_cache *fsverity_info_cachep;
+
+/* List of supported hash algorithms */
+static struct fsverity_hash_alg {
+ struct crypto_shash *tfm; /* allocated on demand */
+ const char *name;
+ unsigned int digest_size;
+} fsverity_hash_algs[] = {
+ [FS_VERITY_ALG_SHA256] = {
+ .name = "sha256",
+ .digest_size = 32,
+ },
+ [FS_VERITY_ALG_CRC32] = {
+ .name = "crc32",
+ .digest_size = 4,
+ },
+};
+
+/*
+ * Translate the given fs-verity hash algorithm number into a struct describing
+ * the algorithm, and ensure it has a hash transform ready to go. The hash
+ * transforms are allocated on-demand firstly to not waste resources when they
+ * aren't needed, and secondly because the fs-verity module may be loaded
+ * earlier than the needed crypto modules.
+ */
+static const struct fsverity_hash_alg *get_hash_algorithm(unsigned int num)
+{
+ struct fsverity_hash_alg *alg;
+ struct crypto_shash *tfm;
+ int err;
+
+ if (num >= ARRAY_SIZE(fsverity_hash_algs) ||
+ !fsverity_hash_algs[num].name) {
+ pr_warn("Unknown hash algorithm: %u\n", num);
+ return ERR_PTR(-EINVAL);
+ }
+ alg = &fsverity_hash_algs[num];
+retry:
+ /* pairs with cmpxchg_release() below */
+ tfm = smp_load_acquire(&alg->tfm);
+ if (tfm)
+ return alg;
+
+ tfm = crypto_alloc_shash(alg->name, 0, 0);
+ if (IS_ERR(tfm)) {
+ if (PTR_ERR(tfm) == -ENOENT)
+ pr_warn("Algorithm %u (%s) is unavailable\n",
+ num, alg->name);
+ else
+ pr_warn("Error allocating algorithm %u (%s): %ld\n",
+ num, alg->name, PTR_ERR(tfm));
+ return ERR_CAST(tfm);
+ }
+
+ err = -EINVAL;
+ if (WARN_ON(alg->digest_size != crypto_shash_digestsize(tfm)))
+ goto err_free_tfm;
+
+ pr_debug("Allocated '%s' transform; implementation '%s'\n",
+ alg->name, crypto_shash_alg(tfm)->base.cra_driver_name);
+
+ /* Set the initial value if required by the algorithm */
+ if (num == FS_VERITY_ALG_CRC32) {
+ __le32 initial_value = cpu_to_le32(0xFFFFFFFF);
+
+ err = crypto_shash_setkey(tfm, (u8 *)&initial_value, 4);
+ if (err) {
+ pr_warn("Error setting CRC-32 initial value: %d\n",
+ err);
+ goto err_free_tfm;
+ }
+ }
+
+ /* pairs with smp_load_acquire() above */
+ if (cmpxchg_release(&alg->tfm, NULL, tfm) != NULL) {
+ crypto_free_shash(tfm);
+ goto retry;
+ }
+
+ return alg;
+
+err_free_tfm:
+ crypto_free_shash(tfm);
+ return ERR_PTR(err);
+}
+
+static int finalize_hash(struct fsverity_info *vi, struct shash_desc *desc,
+ u8 *out)
+{
+ int err;
+
+ err = crypto_shash_final(desc, out);
+ if (err)
+ return err;
+
+ /* For CRC-32, invert at the end and convert to big-endian format */
+ if (vi->hash_alg == &fsverity_hash_algs[FS_VERITY_ALG_CRC32])
+ put_unaligned_be32(~get_unaligned_le32(out), out);
+ return 0;
+}
+
+static struct fsverity_info *get_fsverity_info(const struct inode *inode)
+{
+ /* pairs with cmpxchg_release() in set_fsverity_info() */
+ return smp_load_acquire(&inode->i_verity_info);
+}
+
+static bool set_fsverity_info(struct inode *inode, struct fsverity_info *vi)
+{
+ /* pairs with smp_load_acquire() in get_fsverity_info() */
+ if (cmpxchg_release(&inode->i_verity_info, NULL, vi) != NULL)
+ return false;
+
+ /*
+ * Set the in-memory i_size to the data i_size. But it should *not* be
+ * written to disk, since then it would no longer be possible to find
+ * the fs-verity footer.
+ */
+ i_size_write(inode, vi->data_i_size);
+ return true;
+}
+
+static void dump_fsverity_footer(const struct fsverity_footer *ftr)
+{
+ pr_debug("magic = %.*s\n", (int)sizeof(ftr->magic), ftr->magic);
+ pr_debug("major_version = %u\n", ftr->major_version);
+ pr_debug("minor_version = %u\n", ftr->minor_version);
+ pr_debug("log_blocksize = %u\n", ftr->log_blocksize);
+ pr_debug("log_arity = %u\n", ftr->log_arity);
+ pr_debug("meta_algorithm = %u\n", le16_to_cpu(ftr->meta_algorithm));
+ pr_debug("data_algorithm = %u\n", le16_to_cpu(ftr->data_algorithm));
+ pr_debug("flags = %#x\n", le32_to_cpu(ftr->flags));
+ pr_debug("size = %llu\n", le64_to_cpu(ftr->size));
+ pr_debug("authenticated_ext_count = %u\n",
+ ftr->authenticated_ext_count);
+ pr_debug("unauthenticated_ext_count = %u\n",
+ ftr->unauthenticated_ext_count);
+ pr_debug("salt = %*phN\n", (int)sizeof(ftr->salt), ftr->salt);
+}
+
+static int parse_elide_extension(struct fsverity_info *vi,
+ const void *_ext, size_t extra_len)
+{
+ const struct fsverity_extension_elide *ext = _ext;
+ u64 offset = le64_to_cpu(ext->offset);
+ u64 length = le64_to_cpu(ext->length);
+ struct fsverity_elision *elision;
+
+ pr_debug("Found elide extension: offset=%llu, length=%llu\n",
+ offset, length);
+
+ if ((offset | length) & ~PAGE_MASK) {
+ pr_warn("Misaligned elision (offset=%llu, length=%llu)\n",
+ offset, length);
+ return -EINVAL;
+ }
+
+ if (length < 1) {
+ pr_warn("Empty elision\n");
+ return -EINVAL;
+ }
+
+ if (offset >= vi->data_i_size || length > vi->data_i_size - offset) {
+ pr_warn("Elision offset and/or length too large (offset=%llu, length=%llu)\n",
+ offset, length);
+ return -EINVAL;
+ }
+
+ if (length >= vi->elided_i_size) {
+ pr_warn("Entire file is elided!\n");
+ return -EINVAL;
+ }
+
+ elision = kmalloc(sizeof(*elision), GFP_NOFS);
+ if (!elision)
+ return -ENOMEM;
+
+ elision->index = offset >> PAGE_SHIFT;
+ elision->nr_pages = length >> PAGE_SHIFT;
+ list_add_tail(&elision->link, &vi->elisions);
+ vi->elided_i_size -= length;
+ return 0;
+}
+
+static int cmp_elisions(void *priv, struct list_head *_a, struct list_head *_b)
+{
+ const struct fsverity_elision *a, *b;
+
+ a = list_entry(_a, struct fsverity_elision, link);
+ b = list_entry(_b, struct fsverity_elision, link);
+ if (a->index > b->index)
+ return 1;
+ if (a->index < b->index)
+ return -1;
+ return 0;
+}
+
+/*
+ * Sort the elisions (if any) in order of increasing offset, then verify they
+ * don't overlap.
+ */
+static int sort_and_check_elisions(struct fsverity_info *vi)
+{
+ const struct fsverity_elision *elision;
+ pgoff_t next_unelided_index = 0;
+
+ list_sort(NULL, &vi->elisions, cmp_elisions);
+
+ list_for_each_entry(elision, &vi->elisions, link) {
+ if (elision->index < next_unelided_index) {
+ pr_warn("Elisions overlap\n");
+ return -EINVAL;
+ }
+ next_unelided_index = elision->index + elision->nr_pages;
+ }
+ return 0;
+}
+
+static int parse_patch_extension(struct fsverity_info *vi,
+ const void *_ext, size_t extra_len)
+{
+ const struct fsverity_extension_patch *ext = _ext;
+ u64 offset = le64_to_cpu(ext->offset);
+ struct fsverity_patch *patch;
+
+ pr_debug("Found patch extension: offset=%llu, length=%zu\n",
+ offset, extra_len);
+
+ if (extra_len < 1) {
+ pr_warn("Patch is empty\n");
+ return -EINVAL;
+ }
+
+ if (extra_len > FS_VERITY_MAX_PATCH_SIZE) {
+ pr_warn("Patch is too long (got %zu, limit is %d)\n",
+ extra_len, FS_VERITY_MAX_PATCH_SIZE);
+ return -EINVAL;
+ }
+
+ if (offset >= vi->data_i_size || extra_len > vi->data_i_size - offset) {
+ pr_warn("Patch offset is too large (%llu)\n", offset);
+ return -EINVAL;
+ }
+
+ pr_debug("databytes=%*phN\n", (int)extra_len, ext->databytes);
+
+ patch = kmalloc(sizeof(*patch) + extra_len, GFP_NOFS);
+ if (!patch)
+ return -ENOMEM;
+ patch->index = offset >> PAGE_SHIFT;
+ patch->offset = offset & ~PAGE_MASK;
+ patch->length = extra_len;
+ memcpy(patch->patch, ext->databytes, extra_len);
+ list_add_tail(&patch->link, &vi->patches);
+ return 0;
+}
+
+static inline u64 patch_begin_byte(const struct fsverity_patch *patch)
+{
+ return ((u64)patch->index << PAGE_SHIFT) + patch->offset;
+}
+
+static inline u64 patch_end_byte(const struct fsverity_patch *patch)
+{
+ return patch_begin_byte(patch) + patch->length;
+}
+
+static int cmp_patches(void *priv, struct list_head *_a, struct list_head *_b)
+{
+ const struct fsverity_patch *a, *b;
+
+ a = list_entry(_a, struct fsverity_patch, link);
+ b = list_entry(_b, struct fsverity_patch, link);
+ if (a->index > b->index)
+ return 1;
+ if (a->index < b->index)
+ return -1;
+ return 0;
+}
+
+/*
+ * Sort the patches (if any) in order of increasing offset, then verify they
+ * don't overlap and that no page has multiple patches.
+ */
+static int sort_and_check_patches(struct fsverity_info *vi)
+{
+ const struct fsverity_patch *patch;
+ u64 next_unpatched_byte = 0;
+ pgoff_t prev_patched_index = 0;
+
+ list_sort(NULL, &vi->patches, cmp_patches);
+
+ list_for_each_entry(patch, &vi->patches, link) {
+ u64 begin = patch_begin_byte(patch);
+ u64 end = patch_end_byte(patch);
+
+ if (begin < next_unpatched_byte) {
+ pr_warn("Patches overlap\n");
+ return -EINVAL;
+ }
+ if (next_unpatched_byte != 0 &&
+ patch->index <= prev_patched_index) {
+ pr_warn("Multiple patches per page\n");
+ return -EINVAL;
+ }
+ next_unpatched_byte = end;
+ prev_patched_index = (end - 1) >> PAGE_SHIFT;
+ }
+ return 0;
+}
+
+static const struct extension_type {
+ int (*parse)(struct fsverity_info *vi, const void *_ext,
+ size_t extra_len);
+ size_t base_len;
+ bool unauthenticated;
+} extension_types[] = {
+ [FS_VERITY_EXT_ELIDE] = {
+ .parse = parse_elide_extension,
+ .base_len = sizeof(struct fsverity_extension_elide),
+ },
+ [FS_VERITY_EXT_PATCH] = {
+ .parse = parse_patch_extension,
+ .base_len = sizeof(struct fsverity_extension_patch),
+ },
+};
+
+/*
+ * Parse the extension items, if any, following the fixed-size portion of the
+ * fs-verity footer. The fsverity_info is updated accordingly.
+ *
+ * Return: On success, the size of the authenticated portion of the footer (the
+ * fixed-size portion plus the authenticated extensions).
+ * Otherwise, a -errno value.
+ */
+static int parse_extensions(struct fsverity_info *vi,
+ const struct fsverity_footer *ftr, size_t limit)
+{
+ const struct fsverity_extension *ext_hdr = (const void *)(ftr + 1);
+ const void * const end = (const void *)ftr + limit;
+ const void *auth_end = ext_hdr;
+ int i;
+ int err;
+ int num_extensions = ftr->authenticated_ext_count +
+ ftr->unauthenticated_ext_count;
+
+ for (i = 0; i < num_extensions; i++) {
+ const struct extension_type *type;
+ u32 len, rounded_len;
+ u16 type_code;
+ bool unauthenticated = (i >= ftr->authenticated_ext_count);
+
+ if (end - (const void *)ext_hdr < sizeof(*ext_hdr)) {
+ pr_warn("Extension list overflows buffer\n");
+ return -EINVAL;
+ }
+ type_code = le16_to_cpu(ext_hdr->type);
+ if (type_code >= ARRAY_SIZE(extension_types) ||
+ !extension_types[type_code].parse) {
+ pr_warn("Unknown extension type: %u\n", type_code);
+ return -EINVAL;
+ }
+ type = &extension_types[type_code];
+ if (unauthenticated != type->unauthenticated) {
+ pr_warn("Extension type %u must be %sauthenticated\n",
+ type_code, type->unauthenticated ? "un" : "");
+ return -EINVAL;
+ }
+ if (ext_hdr->reserved) {
+ pr_warn("Reserved bits set in extension header\n");
+ return -EINVAL;
+ }
+ len = le32_to_cpu(ext_hdr->length);
+ if (len < sizeof(*ext_hdr)) {
+ pr_warn("Invalid length in extension header\n");
+ return -EINVAL;
+ }
+ rounded_len = round_up(len, 8);
+ if (rounded_len == 0 ||
+ rounded_len > end - (const void *)ext_hdr) {
+ pr_warn("Extension item overflows buffer\n");
+ return -EINVAL;
+ }
+ if (len < sizeof(*ext_hdr) + type->base_len) {
+ pr_warn("Extension length too small for type\n");
+ return -EINVAL;
+ }
+ err = type->parse(vi, ext_hdr + 1,
+ len - sizeof(*ext_hdr) - type->base_len);
+ if (err)
+ return err;
+ ext_hdr = (const void *)ext_hdr + rounded_len;
+ if (!unauthenticated)
+ auth_end = ext_hdr;
+ }
+
+ err = sort_and_check_elisions(vi);
+ if (err)
+ return err;
+
+ err = sort_and_check_patches(vi);
+ if (err)
+ return err;
+
+ return auth_end - (const void *)ftr;
+}
+/*
+ * Parse an fs-verity footer, loading information into the fsverity_info.
+ *
+ * Return: On success, the size of the authenticated portion of the footer (the
+ * fixed-size portion plus the authenticated extensions).
+ * Otherwise, a -errno value.
+ */
+static int parse_footer(struct fsverity_info *vi,
+ const struct fsverity_footer *ftr, size_t limit)
+{
+ unsigned int alg_num;
+ u32 flags;
+ int ftr_auth_len;
+
+ /* magic */
+ if (memcmp(ftr->magic, FS_VERITY_MAGIC, sizeof(ftr->magic))) {
+ pr_warn("fs-verity footer not found\n");
+ return -EINVAL;
+ }
+
+ /* major_version */
+ if (ftr->major_version != FS_VERITY_MAJOR) {
+ pr_warn("Unsupported major version (%u)\n", ftr->major_version);
+ return -EINVAL;
+ }
+
+ /* minor_version */
+ if (ftr->minor_version != FS_VERITY_MINOR) {
+ pr_warn("Unsupported minor version (%u)\n", ftr->minor_version);
+ return -EINVAL;
+ }
+
+ /* data_algorithm and meta_algorithm */
+ alg_num = le16_to_cpu(ftr->data_algorithm);
+ if (alg_num != le16_to_cpu(ftr->meta_algorithm)) {
+ pr_warn("Unimplemented case: data (%u) and metadata (%u) hash algorithms differ\n",
+ alg_num, le16_to_cpu(ftr->meta_algorithm));
+ return -EINVAL;
+ }
+ vi->hash_alg = get_hash_algorithm(alg_num);
+ if (IS_ERR(vi->hash_alg))
+ return PTR_ERR(vi->hash_alg);
+
+ /* log_blocksize */
+ if (ftr->log_blocksize != PAGE_SHIFT) {
+ pr_warn("Unsupported log_blocksize (%u). Need block_size == PAGE_SIZE.\n",
+ ftr->log_blocksize);
+ return -EINVAL;
+ }
+ vi->block_bits = ftr->log_blocksize;
+
+ /* log_arity */
+ vi->log_arity = ftr->log_arity;
+ if (vi->log_arity !=
+ vi->block_bits - ilog2(vi->hash_alg->digest_size)) {
+ pr_warn("Unsupported log_arity (%u)\n", vi->log_arity);
+ return -EINVAL;
+ }
+
+ /* flags */
+ flags = le32_to_cpu(ftr->flags);
+ if (flags & ~FS_VERITY_FLAG_INTEGRITY_ONLY) {
+ pr_warn("Unsupported flags (%#x)\n", flags);
+ return -EINVAL;
+ }
+
+ if (flags & FS_VERITY_FLAG_INTEGRITY_ONLY) {
+#ifdef CONFIG_FS_VERITY_INTEGRITY_ONLY
+ pr_warn("Using experimental integrity-only mode!\n");
+ vi->mode = FS_VERITY_MODE_INTEGRITY_ONLY;
+#else
+ pr_warn("Integrity-only mode not supported\n");
+ return -EINVAL;
+#endif
+ }
+
+ /* reserved fields */
+ if (ftr->reserved1 ||
+ memchr_inv(ftr->reserved2, 0, sizeof(ftr->reserved2))) {
+ pr_warn("Reserved bits set in footer\n");
+ return -EINVAL;
+ }
+
+ /* size */
+ vi->data_i_size = le64_to_cpu(ftr->size);
+ if (vi->data_i_size == 0) {
+ pr_warn("Original file size is 0; this is not supported\n");
+ return -EINVAL;
+ }
+ if (vi->data_i_size > vi->full_i_size) {
+ pr_warn("Original file size is too large (%llu)\n",
+ vi->data_i_size);
+ return -EINVAL;
+ }
+ vi->elided_i_size = vi->data_i_size;
+
+ /* salt */
+ memcpy(vi->salt, ftr->salt, FS_VERITY_SALT_SIZE);
+
+ /* extensions */
+ ftr_auth_len = parse_extensions(vi, ftr, limit);
+ if (ftr_auth_len < 0)
+ pr_warn("Invalid or unsupported extensions list\n");
+ return ftr_auth_len;
+}
+
+/*
+ * Calculate the depth of the Merkle tree, then create a map from level to the
+ * block offset at which that level's hash blocks start. Level 'depth - 1' is
+ * the root and is stored first in the file, in the first block following the
+ * original data. Level 0 is the level directly "above" the data blocks and is
+ * stored last in the file, just before the fs-verity footer.
+ */
+static int compute_tree_depth_and_offsets(struct fsverity_info *vi)
+{
+ unsigned int hashes_per_block = 1 << vi->log_arity;
+ u64 blocks = (vi->elided_i_size + (1 << vi->block_bits) - 1) >>
+ vi->block_bits;
+ u64 offset = (vi->data_i_size + (1 << vi->block_bits) - 1) >>
+ vi->block_bits;
+ int depth = 0;
+ int i;
+
+ while (blocks > 1) {
+ if (depth >= FS_VERITY_MAX_LEVELS) {
+ pr_warn("Too many tree levels (max is %d)\n",
+ FS_VERITY_MAX_LEVELS);
+ return -EINVAL;
+ }
+ blocks = (blocks + hashes_per_block - 1) >> vi->log_arity;
+ vi->hash_lvl_region_idx[depth++] = blocks;
+ }
+ vi->depth = depth;
+ pr_debug("depth = %d\n", depth);
+
+ for (i = depth - 1; i >= 0; i--) {
+ u64 next_count = vi->hash_lvl_region_idx[i];
+
+ vi->hash_lvl_region_idx[i] = offset;
+ pr_debug("Level %d is [%llu..%llu] (%llu blocks)\n",
+ i, offset, offset + next_count - 1, next_count);
+ offset += next_count;
+ }
+ return 0;
+}
+
+static pgoff_t first_unelided_page(struct fsverity_info *vi)
+{
+ pgoff_t index = 0;
+ const struct fsverity_elision *elision;
+
+ list_for_each_entry(elision, &vi->elisions, link) {
+ if (index != elision->index)
+ break;
+ index += elision->nr_pages;
+ }
+ return index;
+}
+
+/*
+ * Compute the hash of the root of the Merkle tree (or of the lone data block
+ * for files <= blocksize in length) and store it in vi->root_hash.
+ */
+static int compute_root_hash(struct inode *inode, struct fsverity_info *vi)
+{
+ SHASH_DESC_ON_STACK(desc, vi->hash_alg->tfm);
+ struct page *root_page;
+ pgoff_t root_idx;
+ int err;
+
+ desc->tfm = vi->hash_alg->tfm;
+ desc->flags = 0;
+
+ if (vi->depth)
+ root_idx = vi->hash_lvl_region_idx[vi->depth - 1];
+ else
+ root_idx = first_unelided_page(vi);
+
+ root_page = inode->i_sb->s_vop->read_metadata_page(inode, root_idx);
+ if (IS_ERR(root_page)) {
+ pr_warn("Error reading root block: %ld\n", PTR_ERR(root_page));
+ return PTR_ERR(root_page);
+ }
+
+ err = crypto_shash_init(desc);
+ if (!err)
+ err = crypto_shash_update(desc, vi->salt, FS_VERITY_SALT_SIZE);
+ if (!err) {
+ /* atomic, since we aren't using CRYPTO_TFM_REQ_MAY_SLEEP */
+ void *root = kmap_atomic(root_page);
+
+ err = crypto_shash_update(desc, root, PAGE_SIZE);
+ kunmap_atomic(root);
+ }
+ if (!err)
+ err = finalize_hash(vi, desc, vi->root_hash);
+
+ if (err)
+ pr_warn("Error computing Merkle tree root hash: %d\n", err);
+ else
+ pr_debug("Computed Merkle tree root hash: %*phN\n",
+ vi->hash_alg->digest_size, vi->root_hash);
+ put_page(root_page);
+
+ if (vi->depth == 0) {
+ /*
+ * The data page must not be left in the page cache, otherwise
+ * it would appear to have been verified.
+ */
+ truncate_inode_pages_range(inode->i_mapping,
+ (loff_t)root_idx << PAGE_SHIFT,
+ (((loff_t)root_idx + 1) << PAGE_SHIFT) - 1);
+ }
+
+ return err;
+}
+
+/* Compute the file's measurement and store it in vi->measurement */
+static int compute_measurement(struct fsverity_info *vi,
+ const struct fsverity_footer *ftr,
+ int ftr_auth_len)
+{
+ SHASH_DESC_ON_STACK(desc, vi->hash_alg->tfm);
+ int err;
+
+ desc->tfm = vi->hash_alg->tfm;
+ desc->flags = 0;
+
+ err = crypto_shash_init(desc);
+ if (!err)
+ err = crypto_shash_update(desc, (const u8 *)ftr, ftr_auth_len);
+ if (!err)
+ err = crypto_shash_update(desc, vi->root_hash,
+ vi->hash_alg->digest_size);
+ if (!err)
+ err = finalize_hash(vi, desc, vi->measurement);
+
+ if (err)
+ pr_warn("Error computing fs-verity measurement: %d\n", err);
+ else
+ pr_debug("Computed measurement: %*phN (used ftr_auth_len %d)\n",
+ vi->hash_alg->digest_size, vi->measurement,
+ ftr_auth_len);
+ return err;
+}
+
+static struct fsverity_info *alloc_fsverity_info(void)
+{
+ struct fsverity_info *vi;
+
+ vi = kmem_cache_zalloc(fsverity_info_cachep, GFP_NOFS);
+ if (!vi)
+ return NULL;
+ vi->mode = FS_VERITY_MODE_NEED_AUTHENTICATION;
+ INIT_LIST_HEAD(&vi->elisions);
+ INIT_LIST_HEAD(&vi->patches);
+ return vi;
+}
+
+static void free_fsverity_info(struct fsverity_info *vi)
+{
+ if (!vi)
+ return;
+ kmem_cache_free(fsverity_info_cachep, vi);
+}
+
+/*
+ * Read the file's fs-verity footer and create an fsverity_info for it.
+ *
+ * TODO(mhalcrow): This logic only works with full-size Merkle
+ * trees that include all padding and/or when footer/extent
+ * content fits in one page.
+ */
+static struct fsverity_info *create_fsverity_info(struct inode *inode)
+{
+ struct fsverity_info *vi;
+ loff_t full_isize = i_size_read(inode);
+ unsigned int last_validsize = ((full_isize - 1) & ~PAGE_MASK) + 1;
+ pgoff_t ftr_pgoff = (full_isize - 1) >> PAGE_SHIFT;
+ struct page *ftr_page = NULL;
+ const void *ftr_virt;
+ unsigned int ftr_reverse_offset_loc;
+ u32 ftr_reverse_offset;
+ unsigned int ftr_loc;
+ const struct fsverity_footer *ftr;
+ int ftr_auth_len;
+ int err;
+
+ pr_debug("full_isize = %lld\n", full_isize);
+ pr_debug("last_validsize=%u\n", last_validsize);
+ pr_debug("ftr_pgoff=%lu\n", ftr_pgoff);
+
+ if (full_isize <= 0) {
+ pr_warn("File is empty!\n");
+ return ERR_PTR(-EINVAL);
+ }
+
+ if (last_validsize < sizeof(*ftr) + sizeof(__le32)) {
+ pr_warn("Unsupported alignment for fs-verity metadata -- not enough data in last page (only %u bytes)\n",
+ last_validsize);
+ return ERR_PTR(-EINVAL);
+ }
+
+ vi = alloc_fsverity_info();
+ if (!vi)
+ return ERR_PTR(-ENOMEM);
+
+ vi->full_i_size = full_isize;
+
+ ftr_page = inode->i_sb->s_vop->read_metadata_page(inode, ftr_pgoff);
+ if (IS_ERR(ftr_page)) {
+ err = PTR_ERR(ftr_page);
+ pr_warn("Error reading footer page: %d\n", err);
+ ftr_page = NULL;
+ goto out;
+ }
+ ftr_virt = kmap(ftr_page);
+
+ ftr_reverse_offset_loc = last_validsize - sizeof(__le32);
+ pr_debug("ftr_reverse_offset_loc=%u\n", ftr_reverse_offset_loc);
+ ftr_reverse_offset = get_unaligned_le32(ftr_virt +
+ ftr_reverse_offset_loc);
+ pr_debug("ftr_reverse_offset=%u\n", ftr_reverse_offset);
+ if (ftr_reverse_offset < sizeof(*ftr) + sizeof(__le32)) {
+ pr_warn("Invalid fs-verity metadata (unexpected ftr_reverse_offset: %u)\n",
+ ftr_reverse_offset);
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (ftr_reverse_offset > last_validsize) {
+ pr_warn("Unimplemented case - fs-verity footer crosses pages (last_validsize=%u, ftr_reverse_offset=%u)",
+ last_validsize, ftr_reverse_offset);
+ err = -EINVAL;
+ goto out;
+ }
+ ftr_loc = last_validsize - ftr_reverse_offset;
+ if (ftr_loc & 7) {
+ pr_warn("fs-verity footer is misaligned. last_validsize=%u, ftr_reverse_offset=%u\n",
+ last_validsize, ftr_reverse_offset);
+ err = -EINVAL;
+ goto out;
+ }
+ ftr = ftr_virt + ftr_loc;
+ dump_fsverity_footer(ftr);
+ ftr_auth_len = parse_footer(vi, ftr, ftr_reverse_offset_loc - ftr_loc);
+ if (ftr_auth_len < 0) {
+ err = ftr_auth_len;
+ goto out;
+ }
+
+ err = compute_tree_depth_and_offsets(vi);
+ if (err)
+ goto out;
+
+ err = compute_root_hash(inode, vi);
+ if (err)
+ goto out;
+ err = compute_measurement(vi, ftr, ftr_auth_len);
+out:
+ if (ftr_page) {
+ kunmap(ftr_page);
+ put_page(ftr_page);
+ }
+ if (err) {
+ free_fsverity_info(vi);
+ vi = ERR_PTR(err);
+ }
+ return vi;
+}
+
+/* Ensure the inode has an ->i_verity_info */
+static int setup_fsverity_info(struct inode *inode)
+{
+ struct fsverity_info *vi = get_fsverity_info(inode);
+
+ if (vi)
+ return 0;
+
+ vi = create_fsverity_info(inode);
+ if (IS_ERR(vi))
+ return PTR_ERR(vi);
+
+ if (!set_fsverity_info(inode, vi))
+ free_fsverity_info(vi);
+ return 0;
+}
+
+/**
+ * fsverity_file_open - prepare to open an fs-verity file
+ * @inode: the inode being opened
+ * @filp: the struct file being set up
+ *
+ * When opening an fs-verity file, deny the open if it is for writing.
+ * Otherwise, set up the inode's ->i_verity_info (if not already done) by
+ * parsing the fs-verity metadata at the end of the file.
+ *
+ * When combined with fscrypt, this must be called after fscrypt_file_open().
+ * Otherwise, we won't have the key set up to decrypt the fs-verity metadata.
+ *
+ * TODO: currently userspace has to provide the expected measurement using
+ * FS_IOC_SET_VERITY_MEASUREMENT, but in the future here we will validate the
+ * computed measurement against a PKCS#7 message embedded in the fs-verity
+ * metadata which contains the signed expected measurement.
+ *
+ * Return: 0 on success, -errno on failure
+ */
+int fsverity_file_open(struct inode *inode, struct file *filp)
+{
+ if (filp->f_mode & FMODE_WRITE) {
+ pr_debug("Denying opening fs-verity file (ino %lu) for write\n",
+ inode->i_ino);
+ return -EPERM;
+ }
+
+ pr_debug("Opening fs-verity file (ino %lu)\n", inode->i_ino);
+
+ return setup_fsverity_info(inode);
+}
+EXPORT_SYMBOL_GPL(fsverity_file_open);
+
+/**
+ * fsverity_prepare_setattr - prepare to change an fs-verity inode's attributes
+ * @dentry: dentry through which the inode is being changed
+ * @attr: attributes to change
+ *
+ * fs-verity files are immutable, so deny truncates. This isn't covered by the
+ * open-time check because sys_truncate() takes a path, not a file descriptor.
+ *
+ * Return: 0 on success, -errno on failure
+ */
+int fsverity_prepare_setattr(struct dentry *dentry, struct iattr *attr)
+{
+ if (attr->ia_valid & ATTR_SIZE) {
+ pr_debug("Denying truncate of fs-verity file (ino %lu)\n",
+ d_inode(dentry)->i_ino);
+ return -EPERM;
+ }
+ return 0;
+}
+EXPORT_SYMBOL_GPL(fsverity_prepare_setattr);
+
+/**
+ * fsverity_prepare_getattr - prepare to get an fsverity inode's attributes
+ * @inode: the inode for which the attributes are being retrieved
+ *
+ * To make st_size exclude the fs-verity metadata even before the file has been
+ * opened for the first time, we need to grab the original data size from the
+ * fs-verity footer. Currently, to implement this we just set up the
+ * ->i_verity_info, like in the ->open() hook.
+ *
+ * However, when combined with fscrypt, on an encrypted file this must only be
+ * called if the encryption key has been set up!
+ *
+ * Return: 0 on success, -errno on failure
+ */
+int fsverity_prepare_getattr(struct inode *inode)
+{
+ return setup_fsverity_info(inode);
+}
+EXPORT_SYMBOL_GPL(fsverity_prepare_getattr);
+
+/**
+ * fsverity_cleanup_inode - free the inode's verity info, if present
+ *
+ * Filesystems must call this on inode eviction to free ->i_verity_info.
+ */
+void fsverity_cleanup_inode(struct inode *inode)
+{
+ free_fsverity_info(inode->i_verity_info);
+ inode->i_verity_info = NULL;
+}
+EXPORT_SYMBOL_GPL(fsverity_cleanup_inode);
+
+/**
+ * hash_at_level() - compute the location of the block's hash at the given level
+ *
+ * @vi: (in) the file's verity info
+ * @dindex: (in) the index of the data block being verified
+ * @level: (in) the level of hash we want
+ * @hindex: (out) the index of the hash block containing the wanted hash
+ * @hoffset: (out) the byte offset to the wanted hash within the hash block
+ */
+static void hash_at_level(const struct fsverity_info *vi, pgoff_t dindex,
+ unsigned int level, pgoff_t *hindex,
+ unsigned int *hoffset)
+{
+ pgoff_t hoffset_in_lvl;
+
+ /*
+ * Compute the offset of the hash within the level's region, in hashes.
+ * For example, with 4096-byte blocks and 32-byte hashes, there are
+ * 4096/32 = 128 = 2^7 hashes per hash block, i.e. log_arity = 7. Then,
+ * if the data block index is 65668 and we want the level 1 hash, it is
+ * located at 65668 >> 7 = 513 hashes into the level 1 region.
+ */
+ hoffset_in_lvl = dindex >> (level * vi->log_arity);
+
+ /*
+ * Compute the index of the hash block containing the wanted hash.
+ * Continuing the above example, the block would be at index 513 >> 7 =
+ * 4 within the level 1 region. To this we'd add the index at which the
+ * level 1 region starts.
+ */
+ *hindex = vi->hash_lvl_region_idx[level] +
+ (hoffset_in_lvl >> vi->log_arity);
+
+ /*
+ * Finally, compute the index of the hash within the block rather than
+ * the region, and multiply by the hash size to turn it into a byte
+ * offset. Continuing the above example, the hash would be at byte
+ * offset (513 & ((1 << 7) - 1)) * 32 = 32 within the block.
+ */
+ *hoffset = (hoffset_in_lvl & ((1 << vi->log_arity) - 1)) *
+ vi->hash_alg->digest_size;
+}
+
+/* Extract a hash from a hash page */
+static void extract_hash(struct page *hpage, unsigned int hoffset,
+ unsigned int hsize, u8 *out)
+{
+ void *virt = kmap_atomic(hpage);
+
+ memcpy(out, virt + hoffset, hsize);
+ kunmap_atomic(virt);
+}
+
+static int hash_page(struct fsverity_info *vi, struct page *page,
+ const struct fsverity_patch *patch, u8 *out)
+{
+ SHASH_DESC_ON_STACK(desc, vi->hash_alg->tfm);
+ void *virt;
+ int err;
+
+ desc->tfm = vi->hash_alg->tfm;
+ desc->flags = 0;
+
+ err = crypto_shash_init(desc);
+ if (err)
+ return err;
+
+ err = crypto_shash_update(desc, vi->salt, FS_VERITY_SALT_SIZE);
+ if (err)
+ return err;
+
+ virt = kmap_atomic(page);
+ if (patch) {
+ unsigned int patch_offset = patch->offset;
+ unsigned int patch_length = patch->length;
+ unsigned int patch_skip = 0;
+
+ if (patch->index != page->index) {
+ /* Patch started on a prior page */
+ BUG_ON(patch->index > page->index);
+ patch_skip = (PAGE_SIZE - patch_offset) +
+ ((page->index - patch->index - 1) <<
+ PAGE_SHIFT);
+ patch_offset = 0;
+ patch_length -= patch_skip;
+ }
+ patch_length = min_t(unsigned int, patch_length,
+ PAGE_SIZE - patch_offset);
+
+ err = crypto_shash_update(desc, virt, patch_offset);
+ if (!err)
+ err = crypto_shash_update(desc,
+ patch->patch + patch_skip,
+ patch_length);
+ if (!err)
+ err = crypto_shash_update(desc,
+ virt + patch_offset + patch_length,
+ PAGE_SIZE - patch_offset - patch_length);
+ } else {
+ /* Normal case: no patch, just hash the page */
+ err = crypto_shash_update(desc, virt, PAGE_SIZE);
+ }
+ kunmap_atomic(virt);
+ if (err)
+ return err;
+
+ return finalize_hash(vi, desc, out);
+}
+
+/*
+ * Find the patch, if any, that needs to be applied to the page at the specified
+ * index when verifying.
+ */
+static const struct fsverity_patch *find_patch(struct fsverity_info *vi,
+ pgoff_t index)
+{
+ const struct fsverity_patch *patch;
+
+ list_for_each_entry(patch, &vi->patches, link) {
+ if (index < patch->index)
+ break; /* list is sorted, so can stop here */
+ if (index <= (patch_end_byte(patch) - 1) >> PAGE_SHIFT)
+ return patch;
+ }
+ return NULL;
+}
+
+/*
+ * Determine whether the given page index is elided (bypasses verification). If
+ * so, return true. Else, return false and adjust the page index to account for
+ * any previous elisions.
+ */
+static bool page_elided(struct fsverity_info *vi, pgoff_t *index_p)
+{
+ const struct fsverity_elision *elision;
+ pgoff_t orig_idx = *index_p;
+ pgoff_t elided_idx = *index_p;
+
+ list_for_each_entry(elision, &vi->elisions, link) {
+ if (orig_idx < elision->index)
+ break; /* list is sorted, so can stop here */
+ if (orig_idx < elision->index + elision->nr_pages)
+ return true;
+ elided_idx -= elision->nr_pages;
+ }
+ *index_p = elided_idx;
+ return false;
+}
+
+static int compare_hashes(const u8 *want_hash, const u8 *real_hash,
+ int digest_size, struct inode *inode, pgoff_t index,
+ int level)
+{
+ if (memcmp(want_hash, real_hash, digest_size) == 0)
+ return 0;
+
+ pr_warn_ratelimited("VERIFICATION FAILURE! ino=%lu, index=%lu, level=%d, want_hash=%*phN, real_hash=%*phN\n",
+ inode->i_ino, index, level,
+ digest_size, want_hash, digest_size, real_hash);
+ return -EBADMSG;
+}
+
+/**
+ * fsverity_verify_page - verify a data page
+ *
+ * Verify the integrity and/or authenticity of a page that has just been read
+ * from the file. The page is assumed to be a pagecache page.
+ *
+ * Return: true if the page is valid, else false.
+ */
+bool fsverity_verify_page(struct page *data_page)
+{
+ struct address_space *mapping = data_page->mapping;
+ struct inode *inode = mapping->host;
+ struct fsverity_info *vi = get_fsverity_info(inode);
+ pgoff_t index = data_page->index;
+ int level = 0;
+ u8 _want_hash[FS_VERITY_MAX_DIGEST_SIZE];
+ u8 *want_hash = NULL;
+ u8 real_hash[FS_VERITY_MAX_DIGEST_SIZE];
+ struct page *hpages[FS_VERITY_MAX_LEVELS];
+ unsigned int hoffsets[FS_VERITY_MAX_LEVELS];
+ const struct fsverity_patch *patch;
+ int err;
+
+ /*
+ * It shouldn't be possible to get here without ->i_verity_info set,
+ * since it's set on ->open().
+ */
+ if (WARN_ON_ONCE(!vi))
+ return false;
+
+ /* The page must not be unlocked until verification has completed. */
+ if (WARN_ON_ONCE(!PageLocked(data_page)))
+ return false;
+
+ /*
+ * Reads are forbidden if the measurement being enforced doesn't match
+ * the expected one. Otherwise reads are allowed, but we warn if they
+ * are unauthenticated, i.e. if FS_IOC_SET_VERITY_MEASUREMENT hasn't
+ * been called yet. (Some users need to use unauthenticated reads to
+ * find a signature stored in the file. Allowing these doesn't actually
+ * decrease security, since an attacker could just replace the file with
+ * a non-verity one anyway.)
+ */
+ switch (vi->mode) {
+ case FS_VERITY_MODE_NEED_AUTHENTICATION:
+ pr_warn_ratelimited("Unauthenticated read; ino=%lu, index=%lu\n",
+ inode->i_ino, index);
+ break;
+ case FS_VERITY_MODE_AUTHENTICATION_FAILED:
+ pr_warn("Root authentication failed, failing read; inode=%lu, index=%lu\n",
+ inode->i_ino, index);
+ return false;
+ case FS_VERITY_MODE_AUTHENTICATED:
+ case FS_VERITY_MODE_INTEGRITY_ONLY:
+ break;
+ default:
+ WARN_ON_ONCE(1);
+ return false;
+ }
+
+ /*
+ * Since ->i_size is overridden with ->data_i_size, and fs-verity avoids
+ * recursing into itself when reading hash pages, we shouldn't normally
+ * get here with a page beyond ->data_i_size. But, it can happen if a
+ * read is issued at or beyond EOF since the VFS doesn't check i_size
+ * before calling ->readpage(). Thus, just skip verification if the
+ * page is beyond ->data_i_size.
+ */
+ if (index >= (vi->data_i_size + PAGE_SIZE - 1) >> PAGE_SHIFT) {
+ pr_debug("Page %lu is in metadata region\n", index);
+ return true;
+ }
+
+ patch = find_patch(vi, index);
+ if (patch)
+ pr_debug("Selected patch: index=%lu, offset=%u, length=%u for page index %lu\n",
+ patch->index, patch->offset, patch->length, index);
+
+ if (page_elided(vi, &index)) {
+ pr_debug("Page %lu is elided, not verifying!\n", index);
+ return true;
+ }
+ if (index != data_page->index)
+ pr_debug_ratelimited("Adjusted index because of elisions: %lu => %lu\n",
+ data_page->index, index);
+
+ pr_debug_ratelimited("Verifying data page %lu...\n", index);
+
+ /*
+ * Starting at the leaves, ascend the tree saving hash pages along the
+ * way until we find a verified hash page, indicated by PageChecked; or
+ * until we reach the root.
+ */
+ for (level = 0; level < vi->depth; level++) {
+ pgoff_t hindex;
+ unsigned int hoffset;
+ struct page *hpage;
+
+ hash_at_level(vi, index, level, &hindex, &hoffset);
+
+ pr_debug_ratelimited("Level %d: hindex=%lu, hoffset=%u\n",
+ level, hindex, hoffset);
+
+ hpage = inode->i_sb->s_vop->read_metadata_page(inode, hindex);
+ if (IS_ERR(hpage)) {
+ err = PTR_ERR(hpage);
+ goto out;
+ }
+
+ if (PageChecked(hpage)) {
+ want_hash = _want_hash;
+ extract_hash(hpage, hoffset, vi->hash_alg->digest_size,
+ want_hash);
+ put_page(hpage);
+ pr_debug_ratelimited("Hash page already checked, want hash %*phN\n",
+ vi->hash_alg->digest_size,
+ want_hash);
+ break;
+ }
+ pr_debug_ratelimited("Hash page not yet checked\n");
+ hpages[level] = hpage;
+ hoffsets[level] = hoffset;
+ }
+
+ if (!want_hash) {
+ want_hash = vi->root_hash;
+ pr_debug("Want root hash: %*phN\n", vi->hash_alg->digest_size,
+ want_hash);
+ }
+
+ /* Descend the tree verifying hash pages */
+ for (; level > 0; level--) {
+ struct page *hpage = hpages[level - 1];
+ unsigned int hoffset = hoffsets[level - 1];
+
+ err = hash_page(vi, hpage, NULL, real_hash);
+ if (err)
+ goto out;
+ err = compare_hashes(want_hash, real_hash,
+ vi->hash_alg->digest_size,
+ inode, index, level - 1);
+ if (err)
+ goto out;
+ SetPageChecked(hpage);
+ want_hash = _want_hash;
+ extract_hash(hpage, hoffset, vi->hash_alg->digest_size,
+ want_hash);
+ put_page(hpage);
+ pr_debug("Verified hash page at level %d, now want hash %*phN\n",
+ level - 1, vi->hash_alg->digest_size, want_hash);
+ }
+
+ /* Finally, verify the data page */
+ err = hash_page(vi, data_page, patch, real_hash);
+ if (err)
+ goto out;
+ err = compare_hashes(want_hash, real_hash, vi->hash_alg->digest_size,
+ inode, index, -1);
+out:
+ for (; level > 0; level--)
+ put_page(hpages[level - 1]);
+ if (err) {
+ pr_warn_ratelimited("Error verifying page; ino=%lu, index=%lu (err=%d)\n",
+ inode->i_ino, data_page->index, err);
+ return false;
+ }
+ return true;
+}
+EXPORT_SYMBOL_GPL(fsverity_verify_page);
+
+/**
+ * fsverity_verify_bio - verify a 'read' bio that has just completed
+ *
+ * Verify the integrity and/or authenticity of a set of pages that have just
+ * been read from the file. The pages are assumed to be pagecache pages. Pages
+ * that fail verification are set to the Error state. Verification is skipped
+ * for pages already in the Error state, e.g. due to fscrypt decryption failure.
+ */
+void fsverity_verify_bio(struct bio *bio)
+{
+ struct bio_vec *bv;
+ int i;
+
+ bio_for_each_segment_all(bv, bio, i) {
+ struct page *page = bv->bv_page;
+
+ if (!PageError(page) && !fsverity_verify_page(page))
+ SetPageError(page);
+ }
+}
+EXPORT_SYMBOL_GPL(fsverity_verify_bio);
+
+/**
+ * fsverity_enqueue_verify_work - enqueue work on the fs-verity workqueue
+ *
+ * Enqueue verification work for asynchronous processing.
+ */
+void fsverity_enqueue_verify_work(struct work_struct *work)
+{
+ queue_work(fsverity_read_workqueue, work);
+}
+EXPORT_SYMBOL_GPL(fsverity_enqueue_verify_work);
+
+/**
+ * fsverity_full_isize - get the full (on-disk) file size
+ *
+ * If the inode has had its in-memory ->i_size overridden for fs-verity (to
+ * exclude the metadata at the end of the file), then return the full i_size
+ * which is stored on-disk. Otherwise, just return the in-memory ->i_size.
+ *
+ * Return: the full (on-disk) file size
+ */
+loff_t fsverity_full_isize(struct inode *inode)
+{
+ struct fsverity_info *vi = get_fsverity_info(inode);
+
+ if (vi)
+ return vi->full_i_size;
+
+ return i_size_read(inode);
+}
+EXPORT_SYMBOL_GPL(fsverity_full_isize);
+
+/**
+ * fsverity_ioctl_set_measurement - set a verity file's expected measurement
+ *
+ * The FS_IOC_SET_VERITY_MEASUREMENT ioctl informs the kernel which
+ * "measurement" (hash of the Merkle tree root hash and the fsverity footer) is
+ * expected for an fs-verity file. If the actual measurement matches the
+ * expected one, then the ioctl succeeds and further reads from the file are
+ * allowed without warnings. Additionally, the inode is pinned in memory so
+ * that the "authenticated" status doesn't get lost on eviction. Otherwise,
+ * reads from the file are forbidden and the ioctl fails.
+ *
+ * TODO(mhalcrow): In the future, we're going to want to have a signature
+ * attached to the files that we validate whenever we load the authenticated
+ * data structure root into memory.
+ *
+ * Return: 0 on success, -errno on failure
+ */
+int fsverity_ioctl_set_measurement(struct file *filp, const void __user *_uarg)
+{
+ const struct fsverity_measurement __user *uarg = _uarg;
+ struct inode *inode = file_inode(filp);
+ struct super_block *sb = inode->i_sb;
+ struct fsverity_measurement arg;
+ u8 digest[FS_VERITY_MAX_DIGEST_SIZE];
+ const struct fsverity_hash_alg *hash_alg;
+ struct fsverity_info *vi;
+ int err;
+
+ if (!capable(CAP_SYS_ADMIN)) {
+ pr_debug("Process '%s' is not authorized to provide fs-verity measurements\n",
+ current->comm);
+ err = -EACCES;
+ goto out;
+ }
+
+ if (copy_from_user(&arg, uarg, sizeof(arg))) {
+ err = -EFAULT;
+ goto out;
+ }
+
+ if (arg.reserved1 ||
+ memchr_inv(arg.reserved2, 0, sizeof(arg.reserved2))) {
+ pr_debug("Reserved fields are set in fsverity_measurement\n");
+ err = -EINVAL;
+ goto out;
+ }
+
+ hash_alg = get_hash_algorithm(arg.digest_algorithm);
+ if (IS_ERR(hash_alg)) {
+ err = PTR_ERR(hash_alg);
+ goto out;
+ }
+
+ if (arg.digest_size != hash_alg->digest_size) {
+ pr_debug("Wrong digest_size in fsverity_measurement: wanted %u for algorithm '%s', but got %u\n",
+ hash_alg->digest_size, hash_alg->name,
+ arg.digest_size);
+ err = -EINVAL;
+ goto out;
+ }
+
+ /* Redundant, but just in case */
+ if (WARN_ON(hash_alg->digest_size > sizeof(digest))) {
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (copy_from_user(digest, uarg->digest, hash_alg->digest_size)) {
+ err = -EFAULT;
+ goto out;
+ }
+
+ vi = get_fsverity_info(inode);
+ if (!vi) {
+ pr_debug("File does not have fs-verity enabled\n");
+ err = -EINVAL;
+ goto out;
+ }
+
+ inode_lock(inode);
+
+ if (hash_alg != vi->hash_alg) {
+ pr_debug("Hash algorithm mismatch: provided '%s', but file uses '%s'\n",
+ hash_alg->name, vi->hash_alg->name);
+ err = -EBADMSG;
+ vi->mode = FS_VERITY_MODE_AUTHENTICATION_FAILED;
+ truncate_inode_pages(inode->i_mapping, 0);
+ goto out_unlock;
+ }
+
+ if (memcmp(digest, vi->measurement, hash_alg->digest_size)) {
+ if (vi->mode == FS_VERITY_MODE_AUTHENTICATED)
+ pr_warn("Inconsistent measurements were provided! %*phN and %*phN\n",
+ hash_alg->digest_size, digest,
+ hash_alg->digest_size, vi->measurement);
+ else
+ pr_warn("File corrupted! Wanted measurement: %*phN, real measurement: %*phN\n",
+ hash_alg->digest_size, digest,
+ hash_alg->digest_size, vi->measurement);
+ vi->mode = FS_VERITY_MODE_AUTHENTICATION_FAILED;
+ truncate_inode_pages(inode->i_mapping, 0);
+ err = -EBADMSG;
+ goto out_unlock;
+ }
+
+ if (vi->mode == FS_VERITY_MODE_AUTHENTICATED) {
+ pr_debug("Measurement already being enforced: %*phN\n",
+ hash_alg->digest_size, vi->measurement);
+ } else {
+ pr_debug("Measurement validated: %*phN\n",
+ hash_alg->digest_size, vi->measurement);
+ vi->mode = FS_VERITY_MODE_AUTHENTICATED;
+
+ /*
+ * No need to truncate pages here. Any pages that were read
+ * without authentication still had their integrity verified up
+ * to ->root_hash, so now we know they are authentic as we've
+ * just authenticated ->root_hash.
+ */
+
+ /* Pin the inode so that the measurement doesn't go away */
+ ihold(inode);
+ spin_lock(&sb->s_inode_fsveritylist_lock);
+ list_add(&inode->i_fsverity_list, &sb->s_inodes_fsverity);
+ spin_unlock(&sb->s_inode_fsveritylist_lock);
+ }
+ err = 0;
+out_unlock:
+ inode_unlock(inode);
+out:
+ if (err)
+ pr_debug("FS_IOC_SET_VERITY_MEASUREMENT failed on inode %lu (err %d)\n",
+ inode->i_ino, err);
+ else
+ pr_debug("FS_IOC_SET_VERITY_MEASUREMENT succeeded on inode %lu\n",
+ inode->i_ino);
+ return err;
+}
+EXPORT_SYMBOL_GPL(fsverity_ioctl_set_measurement);
+
+/**
+ * fsverity_ioctl_enable - enable fs-verity on a file
+ *
+ * Enable the fs-verity bit on a file. Userspace must have already appended the
+ * fs-verity metadata to the file.
+ *
+ * Enabling fs-verity makes the file contents immutable, and the filesystem
+ * doesn't allow disabling it (other than by replacing the file).
+ *
+ * To avoid races with the file contents being modified, no processes must have
+ * the file open for writing. This includes the caller!
+ *
+ * Return: 0 on success, -errno on failure
+ */
+int fsverity_ioctl_enable(struct file *filp, const void __user *arg)
+{
+ struct inode *inode = file_inode(filp);
+ struct fsverity_info *vi;
+ int err;
+
+ if (!capable(CAP_SYS_ADMIN)) {
+ pr_debug("Process '%s' is not authorized to enable fs-verity\n",
+ current->comm);
+ err = -EACCES;
+ goto out;
+ }
+
+ if (arg) {
+ pr_debug("FS_IOC_ENABLE_VERITY doesn't take an argument\n");
+ err = -EINVAL;
+ goto out;
+ }
+
+ if (S_ISDIR(inode->i_mode)) {
+ pr_debug("Inode is a directory\n");
+ err = -EISDIR;
+ goto out;
+ }
+
+ if (!S_ISREG(inode->i_mode)) {
+ pr_debug("Inode is not a regular file\n");
+ err = -EINVAL;
+ goto out;
+ }
+
+ err = mnt_want_write_file(filp);
+ if (err)
+ goto out;
+
+ /*
+ * Temporarily lock out writers via writable file descriptors or
+ * truncate(). This should stabilize the contents of the file as well
+ * as its size. Note that at the end of this ioctl we will unlock
+ * writers, but at that point the fs-verity bit will be set (if the
+ * ioctl succeeded), preventing future writers.
+ */
+ err = deny_write_access(filp);
+ if (err) {
+ pr_debug("File is open for writing!\n");
+ goto out_drop_write;
+ }
+
+ /*
+ * fsync so that the fs-verity bit can't be persisted to disk prior to
+ * the data, causing verification errors after a crash.
+ */
+ err = vfs_fsync(filp, 1);
+ if (err) {
+ pr_debug("I/O error occurred during fsync\n");
+ goto out_allow_write;
+ }
+
+ /* Serialize concurrent use of this ioctl on the same inode */
+ inode_lock(inode);
+
+ if (inode->i_sb->s_vop->is_verity(inode)) {
+ pr_debug("Fs-verity is already enabled on this file\n");
+ err = -EEXIST;
+ goto out_unlock;
+ }
+
+ /* Validate the fs-verity footer */
+ vi = create_fsverity_info(inode);
+ if (IS_ERR(vi)) {
+ pr_debug("create_fsverity_info() failed\n");
+ err = PTR_ERR(vi);
+ goto out_unlock;
+ }
+
+ /* Set the fs-verity bit */
+ err = inode->i_sb->s_vop->set_verity(inode);
+ if (err) {
+ pr_debug("Filesystem ->set_verity() method failed\n");
+ goto out_free_vi;
+ }
+
+ /* Invalidate all cached pages, forcing re-verification */
+ truncate_inode_pages(inode->i_mapping, 0);
+
+ /* Set ->i_verity_info */
+ if (set_fsverity_info(inode, vi))
+ vi = NULL;
+ err = 0;
+out_free_vi:
+ free_fsverity_info(vi);
+out_unlock:
+ inode_unlock(inode);
+out_allow_write:
+ allow_write_access(filp);
+out_drop_write:
+ mnt_drop_write_file(filp);
+out:
+ if (err)
+ pr_debug("Failed to enable fs-verity on inode %lu (err=%d)\n",
+ inode->i_ino, err);
+ else
+ pr_debug("Successfully enabled fs-verity on inode %lu\n",
+ inode->i_ino);
+ return err;
+}
+EXPORT_SYMBOL_GPL(fsverity_ioctl_enable);
+
+static int __init fsverity_module_init(void)
+{
+ int err;
+ int i;
+
+ /*
+ * Use an unbound workqueue to better distribute verification work among
+ * multiple CPUs.
+ *
+ * Use a high-priority workqueue to prioritize verification work, which
+ * blocks reads from completing, over regular application tasks.
+ */
+ err = -ENOMEM;
+ fsverity_read_workqueue = alloc_workqueue("fsverity_read_queue",
+ WQ_CPU_INTENSIVE |
+ WQ_HIGHPRI | WQ_UNBOUND,
+ num_online_cpus());
+ if (!fsverity_read_workqueue)
+ goto error;
+
+ err = -ENOMEM;
+ fsverity_info_cachep = KMEM_CACHE(fsverity_info, SLAB_RECLAIM_ACCOUNT);
+ if (!fsverity_info_cachep)
+ goto error_free_workqueue;
+
+ /*
+ * Sanity check the digest sizes (could be a build-time check, but
+ * they're in an array)
+ */
+ for (i = 0; i < ARRAY_SIZE(fsverity_hash_algs); i++) {
+ struct fsverity_hash_alg *alg = &fsverity_hash_algs[i];
+
+ if (!alg->name)
+ continue;
+ BUG_ON(alg->digest_size > FS_VERITY_MAX_DIGEST_SIZE);
+ BUG_ON(!is_power_of_2(alg->digest_size));
+ }
+
+ pr_debug("Initialized fs-verity\n");
+ return 0;
+
+error_free_workqueue:
+ destroy_workqueue(fsverity_read_workqueue);
+error:
+ return err;
+}
+
+static void __exit fsverity_module_exit(void)
+{
+ int i;
+
+ destroy_workqueue(fsverity_read_workqueue);
+ kmem_cache_destroy(fsverity_info_cachep);
+ for (i = 0; i < ARRAY_SIZE(fsverity_hash_algs); i++)
+ crypto_free_shash(fsverity_hash_algs[i].tfm);
+}
+
+module_init(fsverity_module_init)
+module_exit(fsverity_module_exit);
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("fs-verity: read-only file-based integrity/authenticity");
diff --git a/fs/xattr.c b/fs/xattr.c
index 1b00bab..1c91835 100644
--- a/fs/xattr.c
+++ b/fs/xattr.c
@@ -540,7 +540,7 @@ getxattr(struct dentry *d, const char __user *name, void __user *value,
if (error > 0) {
if ((strcmp(kname, XATTR_NAME_POSIX_ACL_ACCESS) == 0) ||
(strcmp(kname, XATTR_NAME_POSIX_ACL_DEFAULT) == 0))
- posix_acl_fix_xattr_to_user(kvalue, size);
+ posix_acl_fix_xattr_to_user(kvalue, error);
if (size && copy_to_user(value, kvalue, error))
error = -EFAULT;
} else if (error == -ERANGE && size >= XATTR_SIZE_MAX) {
@@ -953,17 +953,19 @@ ssize_t simple_xattr_list(struct inode *inode, struct simple_xattrs *xattrs,
int err = 0;
#ifdef CONFIG_FS_POSIX_ACL
- if (inode->i_acl) {
- err = xattr_list_one(&buffer, &remaining_size,
- XATTR_NAME_POSIX_ACL_ACCESS);
- if (err)
- return err;
- }
- if (inode->i_default_acl) {
- err = xattr_list_one(&buffer, &remaining_size,
- XATTR_NAME_POSIX_ACL_DEFAULT);
- if (err)
- return err;
+ if (IS_POSIXACL(inode)) {
+ if (inode->i_acl) {
+ err = xattr_list_one(&buffer, &remaining_size,
+ XATTR_NAME_POSIX_ACL_ACCESS);
+ if (err)
+ return err;
+ }
+ if (inode->i_default_acl) {
+ err = xattr_list_one(&buffer, &remaining_size,
+ XATTR_NAME_POSIX_ACL_DEFAULT);
+ if (err)
+ return err;
+ }
}
#endif
diff --git a/fs/xfs/libxfs/xfs_alloc.c b/fs/xfs/libxfs/xfs_alloc.c
index c3702cd..e567551 100644
--- a/fs/xfs/libxfs/xfs_alloc.c
+++ b/fs/xfs/libxfs/xfs_alloc.c
@@ -2035,6 +2035,93 @@ xfs_alloc_space_available(
}
/*
+ * Check the agfl fields of the agf for inconsistency or corruption. The purpose
+ * is to detect an agfl header padding mismatch between current and early v5
+ * kernels. This problem manifests as a 1-slot size difference between the
+ * on-disk flcount and the active [first, last] range of a wrapped agfl. This
+ * may also catch variants of agfl count corruption unrelated to padding. Either
+ * way, we'll reset the agfl and warn the user.
+ *
+ * Return true if a reset is required before the agfl can be used, false
+ * otherwise.
+ */
+static bool
+xfs_agfl_needs_reset(
+ struct xfs_mount *mp,
+ struct xfs_agf *agf)
+{
+ uint32_t f = be32_to_cpu(agf->agf_flfirst);
+ uint32_t l = be32_to_cpu(agf->agf_fllast);
+ uint32_t c = be32_to_cpu(agf->agf_flcount);
+ int agfl_size = XFS_AGFL_SIZE(mp);
+ int active;
+
+ /* no agfl header on v4 supers */
+ if (!xfs_sb_version_hascrc(&mp->m_sb))
+ return false;
+
+ /*
+ * The agf read verifier catches severe corruption of these fields.
+ * Repeat some sanity checks to cover a packed -> unpacked mismatch if
+ * the verifier allows it.
+ */
+ if (f >= agfl_size || l >= agfl_size)
+ return true;
+ if (c > agfl_size)
+ return true;
+
+ /*
+ * Check consistency between the on-disk count and the active range. An
+ * agfl padding mismatch manifests as an inconsistent flcount.
+ */
+ if (c && l >= f)
+ active = l - f + 1;
+ else if (c)
+ active = agfl_size - f + l + 1;
+ else
+ active = 0;
+
+ return active != c;
+}
+
+/*
+ * Reset the agfl to an empty state. Ignore/drop any existing blocks since the
+ * agfl content cannot be trusted. Warn the user that a repair is required to
+ * recover leaked blocks.
+ *
+ * The purpose of this mechanism is to handle filesystems affected by the agfl
+ * header padding mismatch problem. A reset keeps the filesystem online with a
+ * relatively minor free space accounting inconsistency rather than suffer the
+ * inevitable crash from use of an invalid agfl block.
+ */
+static void
+xfs_agfl_reset(
+ struct xfs_trans *tp,
+ struct xfs_buf *agbp,
+ struct xfs_perag *pag)
+{
+ struct xfs_mount *mp = tp->t_mountp;
+ struct xfs_agf *agf = XFS_BUF_TO_AGF(agbp);
+
+ ASSERT(pag->pagf_agflreset);
+ trace_xfs_agfl_reset(mp, agf, 0, _RET_IP_);
+
+ xfs_warn(mp,
+ "WARNING: Reset corrupted AGFL on AG %u. %d blocks leaked. "
+ "Please unmount and run xfs_repair.",
+ pag->pag_agno, pag->pagf_flcount);
+
+ agf->agf_flfirst = 0;
+ agf->agf_fllast = cpu_to_be32(XFS_AGFL_SIZE(mp) - 1);
+ agf->agf_flcount = 0;
+ xfs_alloc_log_agf(tp, agbp, XFS_AGF_FLFIRST | XFS_AGF_FLLAST |
+ XFS_AGF_FLCOUNT);
+
+ pag->pagf_flcount = 0;
+ pag->pagf_agflreset = false;
+}
+
+/*
* Decide whether to use this allocation group for this allocation.
* If so, fix up the btree freelist's size.
*/
@@ -2095,6 +2182,10 @@ xfs_alloc_fix_freelist(
}
}
+ /* reset a padding mismatched agfl before final free space check */
+ if (pag->pagf_agflreset)
+ xfs_agfl_reset(tp, agbp, pag);
+
/* If there isn't enough total space or single-extent, reject it. */
need = xfs_alloc_min_freelist(mp, pag);
if (!xfs_alloc_space_available(args, need, flags))
@@ -2251,6 +2342,7 @@ xfs_alloc_get_freelist(
agf->agf_flfirst = 0;
pag = xfs_perag_get(mp, be32_to_cpu(agf->agf_seqno));
+ ASSERT(!pag->pagf_agflreset);
be32_add_cpu(&agf->agf_flcount, -1);
xfs_trans_agflist_delta(tp, -1);
pag->pagf_flcount--;
@@ -2362,6 +2454,7 @@ xfs_alloc_put_freelist(
agf->agf_fllast = 0;
pag = xfs_perag_get(mp, be32_to_cpu(agf->agf_seqno));
+ ASSERT(!pag->pagf_agflreset);
be32_add_cpu(&agf->agf_flcount, 1);
xfs_trans_agflist_delta(tp, 1);
pag->pagf_flcount++;
@@ -2568,6 +2661,7 @@ xfs_alloc_read_agf(
pag->pagb_count = 0;
pag->pagb_tree = RB_ROOT;
pag->pagf_init = 1;
+ pag->pagf_agflreset = xfs_agfl_needs_reset(mp, agf);
}
#ifdef DEBUG
else if (!XFS_FORCED_SHUTDOWN(mp)) {
diff --git a/fs/xfs/libxfs/xfs_attr.c b/fs/xfs/libxfs/xfs_attr.c
index 6622d46..9687208 100644
--- a/fs/xfs/libxfs/xfs_attr.c
+++ b/fs/xfs/libxfs/xfs_attr.c
@@ -487,7 +487,14 @@ xfs_attr_shortform_addname(xfs_da_args_t *args)
if (args->flags & ATTR_CREATE)
return retval;
retval = xfs_attr_shortform_remove(args);
- ASSERT(retval == 0);
+ if (retval)
+ return retval;
+ /*
+ * Since we have removed the old attr, clear ATTR_REPLACE so
+ * that the leaf format add routine won't trip over the attr
+ * not being around.
+ */
+ args->flags &= ~ATTR_REPLACE;
}
if (args->namelen >= XFS_ATTR_SF_ENTSIZE_MAX ||
diff --git a/fs/xfs/libxfs/xfs_trans_resv.c b/fs/xfs/libxfs/xfs_trans_resv.c
index b456cca..c0ecdec 100644
--- a/fs/xfs/libxfs/xfs_trans_resv.c
+++ b/fs/xfs/libxfs/xfs_trans_resv.c
@@ -232,8 +232,6 @@ xfs_calc_write_reservation(
* the super block to reflect the freed blocks: sector size
* worst case split in allocation btrees per extent assuming 4 extents:
* 4 exts * 2 trees * (2 * max depth - 1) * block size
- * the inode btree: max depth * blocksize
- * the allocation btrees: 2 trees * (max depth - 1) * block size
*/
STATIC uint
xfs_calc_itruncate_reservation(
@@ -245,12 +243,7 @@ xfs_calc_itruncate_reservation(
XFS_FSB_TO_B(mp, 1))),
(xfs_calc_buf_res(9, mp->m_sb.sb_sectsize) +
xfs_calc_buf_res(xfs_allocfree_log_count(mp, 4),
- XFS_FSB_TO_B(mp, 1)) +
- xfs_calc_buf_res(5, 0) +
- xfs_calc_buf_res(xfs_allocfree_log_count(mp, 1),
- XFS_FSB_TO_B(mp, 1)) +
- xfs_calc_buf_res(2 + mp->m_ialloc_blks +
- mp->m_in_maxlevels, 0)));
+ XFS_FSB_TO_B(mp, 1))));
}
/*
diff --git a/fs/xfs/xfs_discard.c b/fs/xfs/xfs_discard.c
index 4ff499a..b2ab123 100644
--- a/fs/xfs/xfs_discard.c
+++ b/fs/xfs/xfs_discard.c
@@ -50,6 +50,13 @@ xfs_trim_extents(
pag = xfs_perag_get(mp, agno);
+ /*
+ * Force out the log. This means any transactions that might have freed
+ * space before we take the AGF buffer lock are now on disk, and the
+ * volatile disk cache is flushed.
+ */
+ xfs_log_force(mp, XFS_LOG_SYNC);
+
error = xfs_alloc_read_agf(mp, NULL, agno, 0, &agbp);
if (error || !agbp)
goto out_put_perag;
@@ -57,13 +64,6 @@ xfs_trim_extents(
cur = xfs_allocbt_init_cursor(mp, NULL, agbp, agno, XFS_BTNUM_CNT);
/*
- * Force out the log. This means any transactions that might have freed
- * space before we took the AGF buffer lock are now on disk, and the
- * volatile disk cache is flushed.
- */
- xfs_log_force(mp, XFS_LOG_SYNC);
-
- /*
* Look up the longest btree in the AGF and start with it.
*/
error = xfs_alloc_lookup_ge(cur, 0,
diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c
index 362c6b4..1410835 100644
--- a/fs/xfs/xfs_file.c
+++ b/fs/xfs/xfs_file.c
@@ -846,22 +846,26 @@ xfs_file_fallocate(
if (error)
goto out_unlock;
} else if (mode & FALLOC_FL_INSERT_RANGE) {
- unsigned int blksize_mask = i_blocksize(inode) - 1;
+ unsigned int blksize_mask = i_blocksize(inode) - 1;
+ loff_t isize = i_size_read(inode);
- new_size = i_size_read(inode) + len;
if (offset & blksize_mask || len & blksize_mask) {
error = -EINVAL;
goto out_unlock;
}
- /* check the new inode size does not wrap through zero */
- if (new_size > inode->i_sb->s_maxbytes) {
+ /*
+ * New inode size must not exceed ->s_maxbytes, accounting for
+ * possible signed overflow.
+ */
+ if (inode->i_sb->s_maxbytes - isize < len) {
error = -EFBIG;
goto out_unlock;
}
+ new_size = isize + len;
/* Offset should be less than i_size */
- if (offset >= i_size_read(inode)) {
+ if (offset >= isize) {
error = -EINVAL;
goto out_unlock;
}
diff --git a/fs/xfs/xfs_mount.h b/fs/xfs/xfs_mount.h
index 5415f90..7cb099e 100644
--- a/fs/xfs/xfs_mount.h
+++ b/fs/xfs/xfs_mount.h
@@ -368,6 +368,7 @@ typedef struct xfs_perag {
char pagi_inodeok; /* The agi is ok for inodes */
__uint8_t pagf_levels[XFS_BTNUM_AGF];
/* # of levels in bno & cnt btree */
+ bool pagf_agflreset; /* agfl requires reset before use */
__uint32_t pagf_flcount; /* count of blocks in freelist */
xfs_extlen_t pagf_freeblks; /* total free blocks */
xfs_extlen_t pagf_longest; /* longest free space */
diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h
index bdf69e1..42a7c0d 100644
--- a/fs/xfs/xfs_trace.h
+++ b/fs/xfs/xfs_trace.h
@@ -1516,7 +1516,7 @@ TRACE_EVENT(xfs_trans_commit_lsn,
__entry->lsn)
);
-TRACE_EVENT(xfs_agf,
+DECLARE_EVENT_CLASS(xfs_agf_class,
TP_PROTO(struct xfs_mount *mp, struct xfs_agf *agf, int flags,
unsigned long caller_ip),
TP_ARGS(mp, agf, flags, caller_ip),
@@ -1572,6 +1572,13 @@ TRACE_EVENT(xfs_agf,
__entry->longest,
(void *)__entry->caller_ip)
);
+#define DEFINE_AGF_EVENT(name) \
+DEFINE_EVENT(xfs_agf_class, name, \
+ TP_PROTO(struct xfs_mount *mp, struct xfs_agf *agf, int flags, \
+ unsigned long caller_ip), \
+ TP_ARGS(mp, agf, flags, caller_ip))
+DEFINE_AGF_EVENT(xfs_agf);
+DEFINE_AGF_EVENT(xfs_agfl_reset);
TRACE_EVENT(xfs_free_extent,
TP_PROTO(struct xfs_mount *mp, xfs_agnumber_t agno, xfs_agblock_t agbno,
diff --git a/include/asm-generic/futex.h b/include/asm-generic/futex.h
index bf2d34c..f0d8b1c 100644
--- a/include/asm-generic/futex.h
+++ b/include/asm-generic/futex.h
@@ -13,7 +13,7 @@
*/
/**
- * futex_atomic_op_inuser() - Atomic arithmetic operation with constant
+ * arch_futex_atomic_op_inuser() - Atomic arithmetic operation with constant
* argument and comparison of the previous
* futex value with another constant.
*
@@ -25,18 +25,11 @@
* <0 - On error
*/
static inline int
-futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
+arch_futex_atomic_op_inuser(int op, u32 oparg, int *oval, u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval, ret;
u32 tmp;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
preempt_disable();
pagefault_disable();
@@ -74,17 +67,9 @@ futex_atomic_op_inuser(int encoded_op, u32 __user *uaddr)
pagefault_enable();
preempt_enable();
- if (ret == 0) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (ret == 0)
+ *oval = oldval;
+
return ret;
}
@@ -126,18 +111,9 @@ futex_atomic_cmpxchg_inatomic(u32 *uval, u32 __user *uaddr,
#else
static inline int
-futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
+arch_futex_atomic_op_inuser(int op, u32 oparg, int *oval, u32 __user *uaddr)
{
- int op = (encoded_op >> 28) & 7;
- int cmp = (encoded_op >> 24) & 15;
- int oparg = (encoded_op << 8) >> 20;
- int cmparg = (encoded_op << 20) >> 20;
int oldval = 0, ret;
- if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28))
- oparg = 1 << oparg;
-
- if (! access_ok (VERIFY_WRITE, uaddr, sizeof(u32)))
- return -EFAULT;
pagefault_disable();
@@ -153,17 +129,9 @@ futex_atomic_op_inuser (int encoded_op, u32 __user *uaddr)
pagefault_enable();
- if (!ret) {
- switch (cmp) {
- case FUTEX_OP_CMP_EQ: ret = (oldval == cmparg); break;
- case FUTEX_OP_CMP_NE: ret = (oldval != cmparg); break;
- case FUTEX_OP_CMP_LT: ret = (oldval < cmparg); break;
- case FUTEX_OP_CMP_GE: ret = (oldval >= cmparg); break;
- case FUTEX_OP_CMP_LE: ret = (oldval <= cmparg); break;
- case FUTEX_OP_CMP_GT: ret = (oldval > cmparg); break;
- default: ret = -ENOSYS;
- }
- }
+ if (!ret)
+ *oval = oldval;
+
return ret;
}
diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h
index f6ea0f3..0a4c2d4 100644
--- a/include/asm-generic/pgtable.h
+++ b/include/asm-generic/pgtable.h
@@ -234,6 +234,21 @@ extern void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp,
extern pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp);
#endif
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+/*
+ * This is an implementation of pmdp_establish() that is only suitable for an
+ * architecture that doesn't have hardware dirty/accessed bits. In this case we
+ * can't race with CPU which sets these bits and non-atomic aproach is fine.
+ */
+static inline pmd_t generic_pmdp_establish(struct vm_area_struct *vma,
+ unsigned long address, pmd_t *pmdp, pmd_t pmd)
+{
+ pmd_t old_pmd = *pmdp;
+ set_pmd_at(vma->vm_mm, address, pmdp, pmd);
+ return old_pmd;
+}
+#endif
+
#ifndef __HAVE_ARCH_PMDP_INVALIDATE
extern void pmdp_invalidate(struct vm_area_struct *vma, unsigned long address,
pmd_t *pmdp);
@@ -764,8 +779,8 @@ int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot);
int pmd_set_huge(pmd_t *pmd, phys_addr_t addr, pgprot_t prot);
int pud_clear_huge(pud_t *pud);
int pmd_clear_huge(pmd_t *pmd);
-int pud_free_pmd_page(pud_t *pud);
-int pmd_free_pte_page(pmd_t *pmd);
+int pud_free_pmd_page(pud_t *pud, unsigned long addr);
+int pmd_free_pte_page(pmd_t *pmd, unsigned long addr);
#else /* !CONFIG_HAVE_ARCH_HUGE_VMAP */
static inline int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot)
{
@@ -783,11 +798,11 @@ static inline int pmd_clear_huge(pmd_t *pmd)
{
return 0;
}
-static inline int pud_free_pmd_page(pud_t *pud)
+static inline int pud_free_pmd_page(pud_t *pud, unsigned long addr)
{
return 0;
}
-static inline int pmd_free_pte_page(pmd_t *pmd)
+static inline int pmd_free_pte_page(pmd_t *pmd, unsigned long addr)
{
return 0;
}
@@ -813,6 +828,19 @@ static inline int pmd_free_pte_page(pmd_t *pmd)
struct file;
int phys_mem_access_prot_allowed(struct file *file, unsigned long pfn,
unsigned long size, pgprot_t *vma_prot);
+
+#ifndef __HAVE_ARCH_PFN_MODIFY_ALLOWED
+static inline bool pfn_modify_allowed(unsigned long pfn, pgprot_t prot)
+{
+ return true;
+}
+
+static inline bool arch_has_pfn_modify_check(void)
+{
+ return false;
+}
+#endif /* !_HAVE_ARCH_PFN_MODIFY_ALLOWED */
+
#endif /* !__ASSEMBLY__ */
#ifndef io_remap_pfn_range
diff --git a/include/asm-generic/qspinlock_types.h b/include/asm-generic/qspinlock_types.h
index 034acd0..d10f1e7 100644
--- a/include/asm-generic/qspinlock_types.h
+++ b/include/asm-generic/qspinlock_types.h
@@ -29,13 +29,41 @@
#endif
typedef struct qspinlock {
- atomic_t val;
+ union {
+ atomic_t val;
+
+ /*
+ * By using the whole 2nd least significant byte for the
+ * pending bit, we can allow better optimization of the lock
+ * acquisition for the pending bit holder.
+ */
+#ifdef __LITTLE_ENDIAN
+ struct {
+ u8 locked;
+ u8 pending;
+ };
+ struct {
+ u16 locked_pending;
+ u16 tail;
+ };
+#else
+ struct {
+ u16 tail;
+ u16 locked_pending;
+ };
+ struct {
+ u8 reserved[2];
+ u8 pending;
+ u8 locked;
+ };
+#endif
+ };
} arch_spinlock_t;
/*
* Initializier
*/
-#define __ARCH_SPIN_LOCK_UNLOCKED { ATOMIC_INIT(0) }
+#define __ARCH_SPIN_LOCK_UNLOCKED { { .val = ATOMIC_INIT(0) } }
/*
* Bitfields in the atomic value:
diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h
index 8236dbd..3c3519b 100644
--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -172,7 +172,7 @@
#endif
#ifdef CONFIG_SERIAL_EARLYCON
-#define EARLYCON_TABLE() STRUCT_ALIGN(); \
+#define EARLYCON_TABLE() . = ALIGN(8); \
VMLINUX_SYMBOL(__earlycon_table) = .; \
*(__earlycon_table) \
VMLINUX_SYMBOL(__earlycon_table_end) = .;
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
new file mode 100644
index 0000000..e328b52
--- /dev/null
+++ b/include/crypto/acompress.h
@@ -0,0 +1,269 @@
+/*
+ * Asynchronous Compression operations
+ *
+ * Copyright (c) 2016, Intel Corporation
+ * Authors: Weigang Li <weigang.li@intel.com>
+ * Giovanni Cabiddu <giovanni.cabiddu@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ */
+#ifndef _CRYPTO_ACOMP_H
+#define _CRYPTO_ACOMP_H
+#include <linux/crypto.h>
+
+#define CRYPTO_ACOMP_ALLOC_OUTPUT 0x00000001
+
+/**
+ * struct acomp_req - asynchronous (de)compression request
+ *
+ * @base: Common attributes for asynchronous crypto requests
+ * @src: Source Data
+ * @dst: Destination data
+ * @slen: Size of the input buffer
+ * @dlen: Size of the output buffer and number of bytes produced
+ * @flags: Internal flags
+ * @__ctx: Start of private context data
+ */
+struct acomp_req {
+ struct crypto_async_request base;
+ struct scatterlist *src;
+ struct scatterlist *dst;
+ unsigned int slen;
+ unsigned int dlen;
+ u32 flags;
+ void *__ctx[] CRYPTO_MINALIGN_ATTR;
+};
+
+/**
+ * struct crypto_acomp - user-instantiated objects which encapsulate
+ * algorithms and core processing logic
+ *
+ * @compress: Function performs a compress operation
+ * @decompress: Function performs a de-compress operation
+ * @dst_free: Frees destination buffer if allocated inside the
+ * algorithm
+ * @reqsize: Context size for (de)compression requests
+ * @base: Common crypto API algorithm data structure
+ */
+struct crypto_acomp {
+ int (*compress)(struct acomp_req *req);
+ int (*decompress)(struct acomp_req *req);
+ void (*dst_free)(struct scatterlist *dst);
+ unsigned int reqsize;
+ struct crypto_tfm base;
+};
+
+/**
+ * struct acomp_alg - asynchronous compression algorithm
+ *
+ * @compress: Function performs a compress operation
+ * @decompress: Function performs a de-compress operation
+ * @dst_free: Frees destination buffer if allocated inside the algorithm
+ * @init: Initialize the cryptographic transformation object.
+ * This function is used to initialize the cryptographic
+ * transformation object. This function is called only once at
+ * the instantiation time, right after the transformation context
+ * was allocated. In case the cryptographic hardware has some
+ * special requirements which need to be handled by software, this
+ * function shall check for the precise requirement of the
+ * transformation and put any software fallbacks in place.
+ * @exit: Deinitialize the cryptographic transformation object. This is a
+ * counterpart to @init, used to remove various changes set in
+ * @init.
+ *
+ * @reqsize: Context size for (de)compression requests
+ * @base: Common crypto API algorithm data structure
+ */
+struct acomp_alg {
+ int (*compress)(struct acomp_req *req);
+ int (*decompress)(struct acomp_req *req);
+ void (*dst_free)(struct scatterlist *dst);
+ int (*init)(struct crypto_acomp *tfm);
+ void (*exit)(struct crypto_acomp *tfm);
+ unsigned int reqsize;
+ struct crypto_alg base;
+};
+
+/**
+ * DOC: Asynchronous Compression API
+ *
+ * The Asynchronous Compression API is used with the algorithms of type
+ * CRYPTO_ALG_TYPE_ACOMPRESS (listed as type "acomp" in /proc/crypto)
+ */
+
+/**
+ * crypto_alloc_acomp() -- allocate ACOMPRESS tfm handle
+ * @alg_name: is the cra_name / name or cra_driver_name / driver name of the
+ * compression algorithm e.g. "deflate"
+ * @type: specifies the type of the algorithm
+ * @mask: specifies the mask for the algorithm
+ *
+ * Allocate a handle for a compression algorithm. The returned struct
+ * crypto_acomp is the handle that is required for any subsequent
+ * API invocation for the compression operations.
+ *
+ * Return: allocated handle in case of success; IS_ERR() is true in case
+ * of an error, PTR_ERR() returns the error code.
+ */
+struct crypto_acomp *crypto_alloc_acomp(const char *alg_name, u32 type,
+ u32 mask);
+
+static inline struct crypto_tfm *crypto_acomp_tfm(struct crypto_acomp *tfm)
+{
+ return &tfm->base;
+}
+
+static inline struct acomp_alg *__crypto_acomp_alg(struct crypto_alg *alg)
+{
+ return container_of(alg, struct acomp_alg, base);
+}
+
+static inline struct crypto_acomp *__crypto_acomp_tfm(struct crypto_tfm *tfm)
+{
+ return container_of(tfm, struct crypto_acomp, base);
+}
+
+static inline struct acomp_alg *crypto_acomp_alg(struct crypto_acomp *tfm)
+{
+ return __crypto_acomp_alg(crypto_acomp_tfm(tfm)->__crt_alg);
+}
+
+static inline unsigned int crypto_acomp_reqsize(struct crypto_acomp *tfm)
+{
+ return tfm->reqsize;
+}
+
+static inline void acomp_request_set_tfm(struct acomp_req *req,
+ struct crypto_acomp *tfm)
+{
+ req->base.tfm = crypto_acomp_tfm(tfm);
+}
+
+static inline struct crypto_acomp *crypto_acomp_reqtfm(struct acomp_req *req)
+{
+ return __crypto_acomp_tfm(req->base.tfm);
+}
+
+/**
+ * crypto_free_acomp() -- free ACOMPRESS tfm handle
+ *
+ * @tfm: ACOMPRESS tfm handle allocated with crypto_alloc_acomp()
+ */
+static inline void crypto_free_acomp(struct crypto_acomp *tfm)
+{
+ crypto_destroy_tfm(tfm, crypto_acomp_tfm(tfm));
+}
+
+static inline int crypto_has_acomp(const char *alg_name, u32 type, u32 mask)
+{
+ type &= ~CRYPTO_ALG_TYPE_MASK;
+ type |= CRYPTO_ALG_TYPE_ACOMPRESS;
+ mask |= CRYPTO_ALG_TYPE_MASK;
+
+ return crypto_has_alg(alg_name, type, mask);
+}
+
+/**
+ * acomp_request_alloc() -- allocates asynchronous (de)compression request
+ *
+ * @tfm: ACOMPRESS tfm handle allocated with crypto_alloc_acomp()
+ *
+ * Return: allocated handle in case of success or NULL in case of an error
+ */
+struct acomp_req *acomp_request_alloc(struct crypto_acomp *tfm);
+
+/**
+ * acomp_request_free() -- zeroize and free asynchronous (de)compression
+ * request as well as the output buffer if allocated
+ * inside the algorithm
+ *
+ * @req: request to free
+ */
+void acomp_request_free(struct acomp_req *req);
+
+/**
+ * acomp_request_set_callback() -- Sets an asynchronous callback
+ *
+ * Callback will be called when an asynchronous operation on a given
+ * request is finished.
+ *
+ * @req: request that the callback will be set for
+ * @flgs: specify for instance if the operation may backlog
+ * @cmlp: callback which will be called
+ * @data: private data used by the caller
+ */
+static inline void acomp_request_set_callback(struct acomp_req *req,
+ u32 flgs,
+ crypto_completion_t cmpl,
+ void *data)
+{
+ req->base.complete = cmpl;
+ req->base.data = data;
+ req->base.flags = flgs;
+}
+
+/**
+ * acomp_request_set_params() -- Sets request parameters
+ *
+ * Sets parameters required by an acomp operation
+ *
+ * @req: asynchronous compress request
+ * @src: pointer to input buffer scatterlist
+ * @dst: pointer to output buffer scatterlist. If this is NULL, the
+ * acomp layer will allocate the output memory
+ * @slen: size of the input buffer
+ * @dlen: size of the output buffer. If dst is NULL, this can be used by
+ * the user to specify the maximum amount of memory to allocate
+ */
+static inline void acomp_request_set_params(struct acomp_req *req,
+ struct scatterlist *src,
+ struct scatterlist *dst,
+ unsigned int slen,
+ unsigned int dlen)
+{
+ req->src = src;
+ req->dst = dst;
+ req->slen = slen;
+ req->dlen = dlen;
+
+ if (!req->dst)
+ req->flags |= CRYPTO_ACOMP_ALLOC_OUTPUT;
+}
+
+/**
+ * crypto_acomp_compress() -- Invoke asynchronous compress operation
+ *
+ * Function invokes the asynchronous compress operation
+ *
+ * @req: asynchronous compress request
+ *
+ * Return: zero on success; error code in case of error
+ */
+static inline int crypto_acomp_compress(struct acomp_req *req)
+{
+ struct crypto_acomp *tfm = crypto_acomp_reqtfm(req);
+
+ return tfm->compress(req);
+}
+
+/**
+ * crypto_acomp_decompress() -- Invoke asynchronous decompress operation
+ *
+ * Function invokes the asynchronous decompress operation
+ *
+ * @req: asynchronous compress request
+ *
+ * Return: zero on success; error code in case of error
+ */
+static inline int crypto_acomp_decompress(struct acomp_req *req)
+{
+ struct crypto_acomp *tfm = crypto_acomp_reqtfm(req);
+
+ return tfm->decompress(req);
+}
+
+#endif
diff --git a/include/crypto/ice.h b/include/crypto/ice.h
index 0b8f048..9b1ec4f 100644
--- a/include/crypto/ice.h
+++ b/include/crypto/ice.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2014-2018, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2014-2017, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
new file mode 100644
index 0000000..1de2b5a
--- /dev/null
+++ b/include/crypto/internal/acompress.h
@@ -0,0 +1,81 @@
+/*
+ * Asynchronous Compression operations
+ *
+ * Copyright (c) 2016, Intel Corporation
+ * Authors: Weigang Li <weigang.li@intel.com>
+ * Giovanni Cabiddu <giovanni.cabiddu@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ */
+#ifndef _CRYPTO_ACOMP_INT_H
+#define _CRYPTO_ACOMP_INT_H
+#include <crypto/acompress.h>
+
+/*
+ * Transform internal helpers.
+ */
+static inline void *acomp_request_ctx(struct acomp_req *req)
+{
+ return req->__ctx;
+}
+
+static inline void *acomp_tfm_ctx(struct crypto_acomp *tfm)
+{
+ return tfm->base.__crt_ctx;
+}
+
+static inline void acomp_request_complete(struct acomp_req *req,
+ int err)
+{
+ req->base.complete(&req->base, err);
+}
+
+static inline const char *acomp_alg_name(struct crypto_acomp *tfm)
+{
+ return crypto_acomp_tfm(tfm)->__crt_alg->cra_name;
+}
+
+static inline struct acomp_req *__acomp_request_alloc(struct crypto_acomp *tfm)
+{
+ struct acomp_req *req;
+
+ req = kzalloc(sizeof(*req) + crypto_acomp_reqsize(tfm), GFP_KERNEL);
+ if (likely(req))
+ acomp_request_set_tfm(req, tfm);
+ return req;
+}
+
+static inline void __acomp_request_free(struct acomp_req *req)
+{
+ kzfree(req);
+}
+
+/**
+ * crypto_register_acomp() -- Register asynchronous compression algorithm
+ *
+ * Function registers an implementation of an asynchronous
+ * compression algorithm
+ *
+ * @alg: algorithm definition
+ *
+ * Return: zero on success; error code in case of error
+ */
+int crypto_register_acomp(struct acomp_alg *alg);
+
+/**
+ * crypto_unregister_acomp() -- Unregister asynchronous compression algorithm
+ *
+ * Function unregisters an implementation of an asynchronous
+ * compression algorithm
+ *
+ * @alg: algorithm definition
+ *
+ * Return: zero on success; error code in case of error
+ */
+int crypto_unregister_acomp(struct acomp_alg *alg);
+
+#endif
diff --git a/include/crypto/internal/scompress.h b/include/crypto/internal/scompress.h
new file mode 100644
index 0000000..3fda3c5
--- /dev/null
+++ b/include/crypto/internal/scompress.h
@@ -0,0 +1,136 @@
+/*
+ * Synchronous Compression operations
+ *
+ * Copyright 2015 LG Electronics Inc.
+ * Copyright (c) 2016, Intel Corporation
+ * Author: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the Free
+ * Software Foundation; either version 2 of the License, or (at your option)
+ * any later version.
+ *
+ */
+#ifndef _CRYPTO_SCOMP_INT_H
+#define _CRYPTO_SCOMP_INT_H
+#include <linux/crypto.h>
+
+#define SCOMP_SCRATCH_SIZE 131072
+
+struct crypto_scomp {
+ struct crypto_tfm base;
+};
+
+/**
+ * struct scomp_alg - synchronous compression algorithm
+ *
+ * @alloc_ctx: Function allocates algorithm specific context
+ * @free_ctx: Function frees context allocated with alloc_ctx
+ * @compress: Function performs a compress operation
+ * @decompress: Function performs a de-compress operation
+ * @init: Initialize the cryptographic transformation object.
+ * This function is used to initialize the cryptographic
+ * transformation object. This function is called only once at
+ * the instantiation time, right after the transformation context
+ * was allocated. In case the cryptographic hardware has some
+ * special requirements which need to be handled by software, this
+ * function shall check for the precise requirement of the
+ * transformation and put any software fallbacks in place.
+ * @exit: Deinitialize the cryptographic transformation object. This is a
+ * counterpart to @init, used to remove various changes set in
+ * @init.
+ * @base: Common crypto API algorithm data structure
+ */
+struct scomp_alg {
+ void *(*alloc_ctx)(struct crypto_scomp *tfm);
+ void (*free_ctx)(struct crypto_scomp *tfm, void *ctx);
+ int (*compress)(struct crypto_scomp *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen,
+ void *ctx);
+ int (*decompress)(struct crypto_scomp *tfm, const u8 *src,
+ unsigned int slen, u8 *dst, unsigned int *dlen,
+ void *ctx);
+ struct crypto_alg base;
+};
+
+static inline struct scomp_alg *__crypto_scomp_alg(struct crypto_alg *alg)
+{
+ return container_of(alg, struct scomp_alg, base);
+}
+
+static inline struct crypto_scomp *__crypto_scomp_tfm(struct crypto_tfm *tfm)
+{
+ return container_of(tfm, struct crypto_scomp, base);
+}
+
+static inline struct crypto_tfm *crypto_scomp_tfm(struct crypto_scomp *tfm)
+{
+ return &tfm->base;
+}
+
+static inline void crypto_free_scomp(struct crypto_scomp *tfm)
+{
+ crypto_destroy_tfm(tfm, crypto_scomp_tfm(tfm));
+}
+
+static inline struct scomp_alg *crypto_scomp_alg(struct crypto_scomp *tfm)
+{
+ return __crypto_scomp_alg(crypto_scomp_tfm(tfm)->__crt_alg);
+}
+
+static inline void *crypto_scomp_alloc_ctx(struct crypto_scomp *tfm)
+{
+ return crypto_scomp_alg(tfm)->alloc_ctx(tfm);
+}
+
+static inline void crypto_scomp_free_ctx(struct crypto_scomp *tfm,
+ void *ctx)
+{
+ return crypto_scomp_alg(tfm)->free_ctx(tfm, ctx);
+}
+
+static inline int crypto_scomp_compress(struct crypto_scomp *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen, void *ctx)
+{
+ return crypto_scomp_alg(tfm)->compress(tfm, src, slen, dst, dlen, ctx);
+}
+
+static inline int crypto_scomp_decompress(struct crypto_scomp *tfm,
+ const u8 *src, unsigned int slen,
+ u8 *dst, unsigned int *dlen,
+ void *ctx)
+{
+ return crypto_scomp_alg(tfm)->decompress(tfm, src, slen, dst, dlen,
+ ctx);
+}
+
+int crypto_init_scomp_ops_async(struct crypto_tfm *tfm);
+struct acomp_req *crypto_acomp_scomp_alloc_ctx(struct acomp_req *req);
+void crypto_acomp_scomp_free_ctx(struct acomp_req *req);
+
+/**
+ * crypto_register_scomp() -- Register synchronous compression algorithm
+ *
+ * Function registers an implementation of a synchronous
+ * compression algorithm
+ *
+ * @alg: algorithm definition
+ *
+ * Return: zero on success; error code in case of error
+ */
+int crypto_register_scomp(struct scomp_alg *alg);
+
+/**
+ * crypto_unregister_scomp() -- Unregister synchronous compression algorithm
+ *
+ * Function unregisters an implementation of a synchronous
+ * compression algorithm
+ *
+ * @alg: algorithm definition
+ *
+ * Return: zero on success; error code in case of error
+ */
+int crypto_unregister_scomp(struct scomp_alg *alg);
+
+#endif
diff --git a/include/crypto/vmac.h b/include/crypto/vmac.h
deleted file mode 100644
index 6b700c7..0000000
--- a/include/crypto/vmac.h
+++ /dev/null
@@ -1,63 +0,0 @@
-/*
- * Modified to interface to the Linux kernel
- * Copyright (c) 2009, Intel Corporation.
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms and conditions of the GNU General Public License,
- * version 2, as published by the Free Software Foundation.
- *
- * This program is distributed in the hope it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
- * more details.
- *
- * You should have received a copy of the GNU General Public License along with
- * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
- * Place - Suite 330, Boston, MA 02111-1307 USA.
- */
-
-#ifndef __CRYPTO_VMAC_H
-#define __CRYPTO_VMAC_H
-
-/* --------------------------------------------------------------------------
- * VMAC and VHASH Implementation by Ted Krovetz (tdk@acm.org) and Wei Dai.
- * This implementation is herby placed in the public domain.
- * The authors offers no warranty. Use at your own risk.
- * Please send bug reports to the authors.
- * Last modified: 17 APR 08, 1700 PDT
- * ----------------------------------------------------------------------- */
-
-/*
- * User definable settings.
- */
-#define VMAC_TAG_LEN 64
-#define VMAC_KEY_SIZE 128/* Must be 128, 192 or 256 */
-#define VMAC_KEY_LEN (VMAC_KEY_SIZE/8)
-#define VMAC_NHBYTES 128/* Must 2^i for any 3 < i < 13 Standard = 128*/
-
-/*
- * This implementation uses u32 and u64 as names for unsigned 32-
- * and 64-bit integer types. These are defined in C99 stdint.h. The
- * following may need adaptation if you are not running a C99 or
- * Microsoft C environment.
- */
-struct vmac_ctx {
- u64 nhkey[(VMAC_NHBYTES/8)+2*(VMAC_TAG_LEN/64-1)];
- u64 polykey[2*VMAC_TAG_LEN/64];
- u64 l3key[2*VMAC_TAG_LEN/64];
- u64 polytmp[2*VMAC_TAG_LEN/64];
- u64 cached_nonce[2];
- u64 cached_aes[2];
- int first_block_processed;
-};
-
-typedef u64 vmac_t;
-
-struct vmac_ctx_t {
- struct crypto_cipher *child;
- struct vmac_ctx __vmac_ctx;
- u8 partial[VMAC_NHBYTES]; /* partial block */
- int partial_size; /* size of the partial block */
-};
-
-#endif /* __CRYPTO_VMAC_H */
diff --git a/include/drm/drm_dp_helper.h b/include/drm/drm_dp_helper.h
index 50810be..1daeacb 100644
--- a/include/drm/drm_dp_helper.h
+++ b/include/drm/drm_dp_helper.h
@@ -346,6 +346,7 @@
# define DP_PSR_FRAME_CAPTURE (1 << 3)
# define DP_PSR_SELECTIVE_UPDATE (1 << 4)
# define DP_PSR_IRQ_HPD_WITH_CRC_ERRORS (1 << 5)
+# define DP_PSR_ENABLE_PSR2 (1 << 6) /* eDP 1.4a */
#define DP_ADAPTER_CTRL 0x1a0
# define DP_ADAPTER_CTRL_FORCE_LOAD_SENSE (1 << 0)
diff --git a/include/drm/drm_mipi_dsi.h b/include/drm/drm_mipi_dsi.h
index 328f232..b208412 100644
--- a/include/drm/drm_mipi_dsi.h
+++ b/include/drm/drm_mipi_dsi.h
@@ -283,9 +283,9 @@ int mipi_dsi_dcs_set_tear_on(struct mipi_dsi_device *dsi,
int mipi_dsi_dcs_set_pixel_format(struct mipi_dsi_device *dsi, u8 format);
int mipi_dsi_dcs_set_tear_scanline(struct mipi_dsi_device *dsi, u16 scanline);
int mipi_dsi_dcs_set_display_brightness(struct mipi_dsi_device *dsi,
- u16 brightness);
+ u16 brightness, size_t num_params);
int mipi_dsi_dcs_get_display_brightness(struct mipi_dsi_device *dsi,
- u16 *brightness);
+ u16 *brightness, size_t num_params);
/**
* struct mipi_dsi_driver - DSI driver
diff --git a/include/dt-bindings/usb/typec.h b/include/dt-bindings/usb/typec.h
new file mode 100644
index 0000000..82ea851
--- /dev/null
+++ b/include/dt-bindings/usb/typec.h
@@ -0,0 +1,23 @@
+/*
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ */
+
+#ifndef _DT_BINDINGS_USB_TYPEC
+#define _DT_BINDINGS_USB_TYPEC
+
+#define TYPEC_PORT_DFP 0x0
+#define TYPEC_PORT_UFP 0x1
+#define TYPEC_PORT_DRP 0x2
+
+#define PDO_TYPE_FIXED 0x0
+#define PDO_TYPE_BATT 0x1
+#define PDO_TYPE_VAR 0x2
+
+#define TYPEC_SINK 0x0
+#define TYPEC_SOURCE 0x1
+
+#endif /* _DT_BINDINGS_USB_TYPEC */
diff --git a/include/kvm/arm_psci.h b/include/kvm/arm_psci.h
index e518e4e..4b15481 100644
--- a/include/kvm/arm_psci.h
+++ b/include/kvm/arm_psci.h
@@ -37,10 +37,15 @@ static inline int kvm_psci_version(struct kvm_vcpu *vcpu, struct kvm *kvm)
* Our PSCI implementation stays the same across versions from
* v0.2 onward, only adding the few mandatory functions (such
* as FEATURES with 1.0) that are required by newer
- * revisions. It is thus safe to return the latest.
+ * revisions. It is thus safe to return the latest, unless
+ * userspace has instructed us otherwise.
*/
- if (test_bit(KVM_ARM_VCPU_PSCI_0_2, vcpu->arch.features))
+ if (test_bit(KVM_ARM_VCPU_PSCI_0_2, vcpu->arch.features)) {
+ if (vcpu->kvm->arch.psci_version)
+ return vcpu->kvm->arch.psci_version;
+
return KVM_ARM_PSCI_LATEST;
+ }
return KVM_ARM_PSCI_0_1;
}
@@ -48,4 +53,11 @@ static inline int kvm_psci_version(struct kvm_vcpu *vcpu, struct kvm *kvm)
int kvm_hvc_call_handler(struct kvm_vcpu *vcpu);
+struct kvm_one_reg;
+
+int kvm_arm_get_fw_num_regs(struct kvm_vcpu *vcpu);
+int kvm_arm_copy_fw_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices);
+int kvm_arm_get_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg);
+int kvm_arm_set_fw_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg);
+
#endif /* __KVM_ARM_PSCI_H__ */
diff --git a/include/linux/arm-smccc.h b/include/linux/arm-smccc.h
index 43690f5..6ddecfb 100644
--- a/include/linux/arm-smccc.h
+++ b/include/linux/arm-smccc.h
@@ -80,6 +80,11 @@
ARM_SMCCC_SMC_32, \
0, 0x8000)
+#define ARM_SMCCC_ARCH_WORKAROUND_2 \
+ ARM_SMCCC_CALL_VAL(ARM_SMCCC_FAST_CALL, \
+ ARM_SMCCC_SMC_32, \
+ 0, 0x7fff)
+
#ifndef __ASSEMBLY__
#include <linux/linkage.h>
@@ -196,47 +201,57 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,
#define __declare_arg_0(a0, res) \
struct arm_smccc_res *___res = res; \
- register u32 r0 SMCCC_REG(0) = a0; \
- register unsigned long r1 SMCCC_REG(1); \
- register unsigned long r2 SMCCC_REG(2); \
- register unsigned long r3 SMCCC_REG(3)
+ register unsigned long r0 asm("r0") = (u32)a0; \
+ register unsigned long r1 asm("r1"); \
+ register unsigned long r2 asm("r2"); \
+ register unsigned long r3 asm("r3")
#define __declare_arg_1(a0, a1, res) \
+ typeof(a1) __a1 = a1; \
struct arm_smccc_res *___res = res; \
- register u32 r0 SMCCC_REG(0) = a0; \
- register typeof(a1) r1 SMCCC_REG(1) = a1; \
- register unsigned long r2 SMCCC_REG(2); \
- register unsigned long r3 SMCCC_REG(3)
+ register unsigned long r0 asm("r0") = (u32)a0; \
+ register unsigned long r1 asm("r1") = __a1; \
+ register unsigned long r2 asm("r2"); \
+ register unsigned long r3 asm("r3")
#define __declare_arg_2(a0, a1, a2, res) \
+ typeof(a1) __a1 = a1; \
+ typeof(a2) __a2 = a2; \
struct arm_smccc_res *___res = res; \
- register u32 r0 SMCCC_REG(0) = a0; \
- register typeof(a1) r1 SMCCC_REG(1) = a1; \
- register typeof(a2) r2 SMCCC_REG(2) = a2; \
- register unsigned long r3 SMCCC_REG(3)
+ register unsigned long r0 asm("r0") = (u32)a0; \
+ register unsigned long r1 asm("r1") = __a1; \
+ register unsigned long r2 asm("r2") = __a2; \
+ register unsigned long r3 asm("r3")
#define __declare_arg_3(a0, a1, a2, a3, res) \
+ typeof(a1) __a1 = a1; \
+ typeof(a2) __a2 = a2; \
+ typeof(a3) __a3 = a3; \
struct arm_smccc_res *___res = res; \
- register u32 r0 SMCCC_REG(0) = a0; \
- register typeof(a1) r1 SMCCC_REG(1) = a1; \
- register typeof(a2) r2 SMCCC_REG(2) = a2; \
- register typeof(a3) r3 SMCCC_REG(3) = a3
+ register unsigned long r0 asm("r0") = (u32)a0; \
+ register unsigned long r1 asm("r1") = __a1; \
+ register unsigned long r2 asm("r2") = __a2; \
+ register unsigned long r3 asm("r3") = __a3
#define __declare_arg_4(a0, a1, a2, a3, a4, res) \
+ typeof(a4) __a4 = a4; \
__declare_arg_3(a0, a1, a2, a3, res); \
- register typeof(a4) r4 SMCCC_REG(4) = a4
+ register unsigned long r4 asm("r4") = __a4
#define __declare_arg_5(a0, a1, a2, a3, a4, a5, res) \
+ typeof(a5) __a5 = a5; \
__declare_arg_4(a0, a1, a2, a3, a4, res); \
- register typeof(a5) r5 SMCCC_REG(5) = a5
+ register unsigned long r5 asm("r5") = __a5
#define __declare_arg_6(a0, a1, a2, a3, a4, a5, a6, res) \
+ typeof(a6) __a6 = a6; \
__declare_arg_5(a0, a1, a2, a3, a4, a5, res); \
- register typeof(a6) r6 SMCCC_REG(6) = a6
+ register unsigned long r6 asm("r6") = __a6
#define __declare_arg_7(a0, a1, a2, a3, a4, a5, a6, a7, res) \
+ typeof(a7) __a7 = a7; \
__declare_arg_6(a0, a1, a2, a3, a4, a5, a6, res); \
- register typeof(a7) r7 SMCCC_REG(7) = a7
+ register unsigned long r7 asm("r7") = __a7
#define ___declare_args(count, ...) __declare_arg_ ## count(__VA_ARGS__)
#define __declare_args(count, ...) ___declare_args(count, __VA_ARGS__)
@@ -293,5 +308,10 @@ asmlinkage void __arm_smccc_hvc(unsigned long a0, unsigned long a1,
*/
#define arm_smccc_1_1_hvc(...) __arm_smccc_1_1(SMCCC_HVC_INST, __VA_ARGS__)
+/* Return codes defined in ARM DEN 0070A */
+#define SMCCC_RET_SUCCESS 0
+#define SMCCC_RET_NOT_SUPPORTED -1
+#define SMCCC_RET_NOT_REQUIRED -2
+
#endif /*__ASSEMBLY__*/
#endif /*__LINUX_ARM_SMCCC_H*/
diff --git a/include/linux/backing-dev-defs.h b/include/linux/backing-dev-defs.h
index 002f87c..4f15630e 100644
--- a/include/linux/backing-dev-defs.h
+++ b/include/linux/backing-dev-defs.h
@@ -140,6 +140,7 @@ struct bdi_writeback {
struct backing_dev_info {
struct list_head bdi_list;
unsigned long ra_pages; /* max readahead in PAGE_SIZE units */
+ unsigned long io_pages; /* max allowed IO size */
unsigned int capabilities; /* Device capabilities */
congested_fn *congested_fn; /* Function pointer if device is md/dm */
void *congested_data; /* Pointer to aux data for congested func */
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 0885c9f..af12869 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -61,6 +61,7 @@
((bio)->bi_iter.bi_size != bio_iovec(bio).bv_len)
#define bio_sectors(bio) ((bio)->bi_iter.bi_size >> 9)
#define bio_end_sector(bio) ((bio)->bi_iter.bi_sector + bio_sectors((bio)))
+
#define bio_dun(bio) ((bio)->bi_iter.bi_dun)
#define bio_duns(bio) (bio_sectors(bio) >> 3) /* 4KB unit */
#define bio_end_dun(bio) (bio_dun(bio) + bio_duns(bio))
@@ -171,12 +172,10 @@ static inline void bio_advance_iter(struct bio *bio, struct bvec_iter *iter,
unsigned bytes)
{
iter->bi_sector += bytes >> 9;
-
#ifdef CONFIG_PFK
if (iter->bi_dun)
iter->bi_dun += bytes >> 12;
#endif
-
if (bio_no_advance_iter(bio))
iter->bi_size -= bytes;
else
diff --git a/include/linux/bldr_debug_tools.h b/include/linux/bldr_debug_tools.h
new file mode 100644
index 0000000..9da15c1
--- /dev/null
+++ b/include/linux/bldr_debug_tools.h
@@ -0,0 +1,40 @@
+/* Copyright 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __BLDR_DEBUG_TOOLS_H__
+#define __BLDR_DEBUG_TOOLS_H__
+
+#include <linux/types.h>
+
+#if defined(CONFIG_BLDR_DEBUG_LOG)
+#define BOOT_DEBUG_MAGIC 0xAACCBBDD
+
+ssize_t bldr_log_read_once(char __user *userbuf, ssize_t klog_size);
+ssize_t bldr_last_log_read_once(char __user *userbuf, ssize_t klog_size);
+ssize_t bldr_log_read(const void *lastk_buf, ssize_t lastk_size,
+ char __user *userbuf, size_t count, loff_t *ppos);
+ssize_t bldr_log_total_size(void);
+int bldr_log_init(void);
+void bldr_log_release(void);
+#else
+static inline ssize_t bldr_log_read_once(char __user *userbuf,
+ ssize_t klog_size) { return 0; }
+static inline ssize_t bldr_last_log_read_once(char __user *userbuf,
+ ssize_t klog_size) { return 0; }
+static inline ssize_t bldr_log_read(const void *lastk_buf, ssize_t lastk_size,
+ char __user *userbuf, size_t count, loff_t *ppos) { return 0; }
+static inline ssize_t bldr_log_total_size(void) { return 0; }
+static inline int bldr_log_init(void) { return 0; }
+static inline void bldr_log_release(void) { return; }
+#endif
+
+#endif /* __BLDR_DEBUG_TOOLS_H__ */
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index 42ad6b5..9d66e7f 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -70,15 +70,17 @@ struct bio {
#ifdef CONFIG_PFK
/* Encryption key to use (NULL if none) */
const struct blk_encryption_key *bi_crypt_key;
-
- /*
- * When using dircet-io (O_DIRECT), we can't get the inode from a bio
- * by walking bio->bi_io_vec->bv_page->mapping->host
- * since the page is anon.
- */
- struct inode *bi_dio_inode;
+#ifdef CONFIG_DM_DEFAULT_KEY
+ int bi_crypt_skip;
#endif
+ /*
+ * When using dircet-io (O_DIRECT), we can't get the inode from a bio
+ * by walking bio->bi_io_vec->bv_page->mapping->host
+ * since the page is anon.
+ */
+ struct inode *bi_dio_inode;
+#endif
unsigned short bi_vcnt; /* how many bio_vec's */
/*
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index d3052b4..a5815e7 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -916,8 +916,8 @@ static inline unsigned int blk_max_size_offset(struct request_queue *q,
if (!q->limits.chunk_sectors)
return q->limits.max_sectors;
- return q->limits.chunk_sectors -
- (offset & (q->limits.chunk_sectors - 1));
+ return min(q->limits.max_sectors, (unsigned int)(q->limits.chunk_sectors -
+ (offset & (q->limits.chunk_sectors - 1))));
}
static inline unsigned int blk_rq_get_max_sectors(struct request *rq,
@@ -1769,7 +1769,7 @@ blk_update_latency_hist(struct io_latency_state *s, u_int64_t delta_us)
s->latency_sum += delta_us;
}
-ssize_t blk_latency_hist_show(char* name, struct io_latency_state *s,
+ssize_t blk_latency_hist_show(char *name, struct io_latency_state *s,
char *buf, int buf_size);
#else /* CONFIG_BLOCK */
diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h
index 070fc49..5031def 100644
--- a/include/linux/bpf_verifier.h
+++ b/include/linux/bpf_verifier.h
@@ -71,6 +71,7 @@ struct bpf_insn_aux_data {
enum bpf_reg_type ptr_type; /* pointer type for load/store insns */
struct bpf_map *map_ptr; /* pointer for call insn into lookup_elem */
};
+ int sanitize_stack_off; /* stack slot to be cleared */
bool seen; /* this insn was processed by the verifier */
};
diff --git a/include/linux/bug.h b/include/linux/bug.h
index 6f3da08..8da93b9 100644
--- a/include/linux/bug.h
+++ b/include/linux/bug.h
@@ -3,6 +3,7 @@
#include <asm/bug.h>
#include <linux/compiler.h>
+#include <linux/build_bug.h>
enum bug_trap_type {
BUG_TRAP_TYPE_NONE = 0,
@@ -13,80 +14,9 @@ enum bug_trap_type {
struct pt_regs;
#ifdef __CHECKER__
-#define __BUILD_BUG_ON_NOT_POWER_OF_2(n) (0)
-#define BUILD_BUG_ON_NOT_POWER_OF_2(n) (0)
-#define BUILD_BUG_ON_ZERO(e) (0)
-#define BUILD_BUG_ON_NULL(e) ((void*)0)
-#define BUILD_BUG_ON_INVALID(e) (0)
-#define BUILD_BUG_ON_MSG(cond, msg) (0)
-#define BUILD_BUG_ON(condition) (0)
-#define BUILD_BUG() (0)
#define MAYBE_BUILD_BUG_ON(cond) (0)
#else /* __CHECKER__ */
-/* Force a compilation error if a constant expression is not a power of 2 */
-#define __BUILD_BUG_ON_NOT_POWER_OF_2(n) \
- BUILD_BUG_ON(((n) & ((n) - 1)) != 0)
-#define BUILD_BUG_ON_NOT_POWER_OF_2(n) \
- BUILD_BUG_ON((n) == 0 || (((n) & ((n) - 1)) != 0))
-
-/* Force a compilation error if condition is true, but also produce a
- result (of value 0 and type size_t), so the expression can be used
- e.g. in a structure initializer (or where-ever else comma expressions
- aren't permitted). */
-#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
-#define BUILD_BUG_ON_NULL(e) ((void *)sizeof(struct { int:-!!(e); }))
-
-/*
- * BUILD_BUG_ON_INVALID() permits the compiler to check the validity of the
- * expression but avoids the generation of any code, even if that expression
- * has side-effects.
- */
-#define BUILD_BUG_ON_INVALID(e) ((void)(sizeof((__force long)(e))))
-
-/**
- * BUILD_BUG_ON_MSG - break compile if a condition is true & emit supplied
- * error message.
- * @condition: the condition which the compiler should know is false.
- *
- * See BUILD_BUG_ON for description.
- */
-#define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
-
-/**
- * BUILD_BUG_ON - break compile if a condition is true.
- * @condition: the condition which the compiler should know is false.
- *
- * If you have some code which relies on certain constants being equal, or
- * some other compile-time-evaluated condition, you should use BUILD_BUG_ON to
- * detect if someone changes it.
- *
- * The implementation uses gcc's reluctance to create a negative array, but gcc
- * (as of 4.4) only emits that error for obvious cases (e.g. not arguments to
- * inline functions). Luckily, in 4.3 they added the "error" function
- * attribute just for this type of case. Thus, we use a negative sized array
- * (should always create an error on gcc versions older than 4.4) and then call
- * an undefined function with the error attribute (should always create an
- * error on gcc 4.3 and later). If for some reason, neither creates a
- * compile-time error, we'll still have a link-time error, which is harder to
- * track down.
- */
-#ifndef __OPTIMIZE__
-#define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)]))
-#else
-#define BUILD_BUG_ON(condition) \
- BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
-#endif
-
-/**
- * BUILD_BUG - break compile if used.
- *
- * If you have some code that you expect the compiler to eliminate at
- * build time, you should use BUILD_BUG to detect if it is
- * unexpectedly used.
- */
-#define BUILD_BUG() BUILD_BUG_ON_MSG(1, "BUILD_BUG failed")
-
#define MAYBE_BUILD_BUG_ON(cond) \
do { \
if (__builtin_constant_p((cond))) \
diff --git a/include/linux/build_bug.h b/include/linux/build_bug.h
new file mode 100644
index 0000000..b7d22d6
--- /dev/null
+++ b/include/linux/build_bug.h
@@ -0,0 +1,84 @@
+#ifndef _LINUX_BUILD_BUG_H
+#define _LINUX_BUILD_BUG_H
+
+#include <linux/compiler.h>
+
+#ifdef __CHECKER__
+#define __BUILD_BUG_ON_NOT_POWER_OF_2(n) (0)
+#define BUILD_BUG_ON_NOT_POWER_OF_2(n) (0)
+#define BUILD_BUG_ON_ZERO(e) (0)
+#define BUILD_BUG_ON_NULL(e) ((void *)0)
+#define BUILD_BUG_ON_INVALID(e) (0)
+#define BUILD_BUG_ON_MSG(cond, msg) (0)
+#define BUILD_BUG_ON(condition) (0)
+#define BUILD_BUG() (0)
+#else /* __CHECKER__ */
+
+/* Force a compilation error if a constant expression is not a power of 2 */
+#define __BUILD_BUG_ON_NOT_POWER_OF_2(n) \
+ BUILD_BUG_ON(((n) & ((n) - 1)) != 0)
+#define BUILD_BUG_ON_NOT_POWER_OF_2(n) \
+ BUILD_BUG_ON((n) == 0 || (((n) & ((n) - 1)) != 0))
+
+/*
+ * Force a compilation error if condition is true, but also produce a
+ * result (of value 0 and type size_t), so the expression can be used
+ * e.g. in a structure initializer (or where-ever else comma expressions
+ * aren't permitted).
+ */
+#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:(-!!(e)); }))
+#define BUILD_BUG_ON_NULL(e) ((void *)sizeof(struct { int:(-!!(e)); }))
+
+/*
+ * BUILD_BUG_ON_INVALID() permits the compiler to check the validity of the
+ * expression but avoids the generation of any code, even if that expression
+ * has side-effects.
+ */
+#define BUILD_BUG_ON_INVALID(e) ((void)(sizeof((__force long)(e))))
+
+/**
+ * BUILD_BUG_ON_MSG - break compile if a condition is true & emit supplied
+ * error message.
+ * @condition: the condition which the compiler should know is false.
+ *
+ * See BUILD_BUG_ON for description.
+ */
+#define BUILD_BUG_ON_MSG(cond, msg) compiletime_assert(!(cond), msg)
+
+/**
+ * BUILD_BUG_ON - break compile if a condition is true.
+ * @condition: the condition which the compiler should know is false.
+ *
+ * If you have some code which relies on certain constants being equal, or
+ * some other compile-time-evaluated condition, you should use BUILD_BUG_ON to
+ * detect if someone changes it.
+ *
+ * The implementation uses gcc's reluctance to create a negative array, but gcc
+ * (as of 4.4) only emits that error for obvious cases (e.g. not arguments to
+ * inline functions). Luckily, in 4.3 they added the "error" function
+ * attribute just for this type of case. Thus, we use a negative sized array
+ * (should always create an error on gcc versions older than 4.4) and then call
+ * an undefined function with the error attribute (should always create an
+ * error on gcc 4.3 and later). If for some reason, neither creates a
+ * compile-time error, we'll still have a link-time error, which is harder to
+ * track down.
+ */
+#ifndef __OPTIMIZE__
+#define BUILD_BUG_ON(condition) ((void)sizeof(char[1 - 2*!!(condition)]))
+#else
+#define BUILD_BUG_ON(condition) \
+ BUILD_BUG_ON_MSG(condition, "BUILD_BUG_ON failed: " #condition)
+#endif
+
+/**
+ * BUILD_BUG - break compile if used.
+ *
+ * If you have some code that you expect the compiler to eliminate at
+ * build time, you should use BUILD_BUG to detect if it is
+ * unexpectedly used.
+ */
+#define BUILD_BUG() BUILD_BUG_ON_MSG(1, "BUILD_BUG failed")
+
+#endif /* __CHECKER__ */
+
+#endif /* _LINUX_BUILD_BUG_H */
diff --git a/include/linux/can/dev.h b/include/linux/can/dev.h
index 5f52709..f7178f4 100644
--- a/include/linux/can/dev.h
+++ b/include/linux/can/dev.h
@@ -154,6 +154,7 @@ void can_change_state(struct net_device *dev, struct can_frame *cf,
void can_put_echo_skb(struct sk_buff *skb, struct net_device *dev,
unsigned int idx);
+struct sk_buff *__can_get_echo_skb(struct net_device *dev, unsigned int idx, u8 *len_ptr);
unsigned int can_get_echo_skb(struct net_device *dev, unsigned int idx);
void can_free_echo_skb(struct net_device *dev, unsigned int idx);
diff --git a/include/linux/ceph/auth.h b/include/linux/ceph/auth.h
index 374bb1c..035f26a 100644
--- a/include/linux/ceph/auth.h
+++ b/include/linux/ceph/auth.h
@@ -63,8 +63,12 @@ struct ceph_auth_client_ops {
/* ensure that an existing authorizer is up to date */
int (*update_authorizer)(struct ceph_auth_client *ac, int peer_type,
struct ceph_auth_handshake *auth);
+ int (*add_authorizer_challenge)(struct ceph_auth_client *ac,
+ struct ceph_authorizer *a,
+ void *challenge_buf,
+ int challenge_buf_len);
int (*verify_authorizer_reply)(struct ceph_auth_client *ac,
- struct ceph_authorizer *a, size_t len);
+ struct ceph_authorizer *a);
void (*invalidate_authorizer)(struct ceph_auth_client *ac,
int peer_type);
@@ -117,9 +121,12 @@ void ceph_auth_destroy_authorizer(struct ceph_authorizer *a);
extern int ceph_auth_update_authorizer(struct ceph_auth_client *ac,
int peer_type,
struct ceph_auth_handshake *a);
+int ceph_auth_add_authorizer_challenge(struct ceph_auth_client *ac,
+ struct ceph_authorizer *a,
+ void *challenge_buf,
+ int challenge_buf_len);
extern int ceph_auth_verify_authorizer_reply(struct ceph_auth_client *ac,
- struct ceph_authorizer *a,
- size_t len);
+ struct ceph_authorizer *a);
extern void ceph_auth_invalidate_authorizer(struct ceph_auth_client *ac,
int peer_type);
diff --git a/include/linux/ceph/ceph_features.h b/include/linux/ceph/ceph_features.h
index ae2f668..cf765db 100644
--- a/include/linux/ceph/ceph_features.h
+++ b/include/linux/ceph/ceph_features.h
@@ -76,6 +76,7 @@
// duplicated since it was introduced at the same time as CEPH_FEATURE_CRUSH_TUNABLES5
#define CEPH_FEATURE_NEW_OSDOPREPLY_ENCODING (1ULL<<58) /* New, v7 encoding */
#define CEPH_FEATURE_FS_FILE_LAYOUT_V2 (1ULL<<58) /* file_layout_t */
+#define CEPH_FEATURE_CEPHX_V2 (1ULL<<61) // *do not share this bit*
/*
* The introduction of CEPH_FEATURE_OSD_SNAPMAPPER caused the feature
@@ -124,7 +125,8 @@ static inline u64 ceph_sanitize_features(u64 features)
CEPH_FEATURE_MSGR_KEEPALIVE2 | \
CEPH_FEATURE_CRUSH_V4 | \
CEPH_FEATURE_CRUSH_TUNABLES5 | \
- CEPH_FEATURE_NEW_OSDOPREPLY_ENCODING)
+ CEPH_FEATURE_NEW_OSDOPREPLY_ENCODING | \
+ CEPH_FEATURE_CEPHX_V2)
#define CEPH_FEATURES_REQUIRED_DEFAULT \
(CEPH_FEATURE_NOSRCADDR | \
diff --git a/include/linux/ceph/libceph.h b/include/linux/ceph/libceph.h
index 1816c5e..a8a5748 100644
--- a/include/linux/ceph/libceph.h
+++ b/include/linux/ceph/libceph.h
@@ -77,7 +77,13 @@ struct ceph_options {
#define CEPH_MSG_MAX_FRONT_LEN (16*1024*1024)
#define CEPH_MSG_MAX_MIDDLE_LEN (16*1024*1024)
-#define CEPH_MSG_MAX_DATA_LEN (16*1024*1024)
+
+/*
+ * Handle the largest possible rbd object in one message.
+ * There is no limit on the size of cephfs objects, but it has to obey
+ * rsize and wsize mount options anyway.
+ */
+#define CEPH_MSG_MAX_DATA_LEN (32*1024*1024)
#define CEPH_AUTH_NAME_DEFAULT "guest"
diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h
index 8dbd787..5e1c9c8 100644
--- a/include/linux/ceph/messenger.h
+++ b/include/linux/ceph/messenger.h
@@ -30,7 +30,10 @@ struct ceph_connection_operations {
struct ceph_auth_handshake *(*get_authorizer) (
struct ceph_connection *con,
int *proto, int force_new);
- int (*verify_authorizer_reply) (struct ceph_connection *con, int len);
+ int (*add_authorizer_challenge)(struct ceph_connection *con,
+ void *challenge_buf,
+ int challenge_buf_len);
+ int (*verify_authorizer_reply) (struct ceph_connection *con);
int (*invalidate_authorizer)(struct ceph_connection *con);
/* there was some error on the socket (disconnect, whatever) */
@@ -200,9 +203,8 @@ struct ceph_connection {
attempt for this connection, client */
u32 peer_global_seq; /* peer's global seq for this connection */
+ struct ceph_auth_handshake *auth;
int auth_retry; /* true if we need a newer authorizer */
- void *auth_reply_buf; /* where to put the authorizer reply */
- int auth_reply_buf_len;
struct mutex mutex;
diff --git a/include/linux/ceph/msgr.h b/include/linux/ceph/msgr.h
index 0fe2656..063f9d7 100644
--- a/include/linux/ceph/msgr.h
+++ b/include/linux/ceph/msgr.h
@@ -90,7 +90,7 @@ struct ceph_entity_inst {
#define CEPH_MSGR_TAG_SEQ 13 /* 64-bit int follows with seen seq number */
#define CEPH_MSGR_TAG_KEEPALIVE2 14 /* keepalive2 byte + ceph_timespec */
#define CEPH_MSGR_TAG_KEEPALIVE2_ACK 15 /* keepalive2 reply */
-
+#define CEPH_MSGR_TAG_CHALLENGE_AUTHORIZER 16 /* cephx v2 doing server challenge */
/*
* connection negotiation
diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index 34d3f27..648432d 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -17,6 +17,7 @@
#include <linux/percpu-rwsem.h>
#include <linux/workqueue.h>
#include <linux/bpf-cgroup.h>
+#include <linux/psi_types.h>
#ifdef CONFIG_CGROUPS
@@ -302,6 +303,9 @@ struct cgroup {
/* used to schedule release agent */
struct work_struct release_agent_work;
+ /* used to track pressure stalls */
+ struct psi_group psi;
+
/* used to store eBPF programs */
struct cgroup_bpf bpf;
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index 3b242a3..acbaa20 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -388,6 +388,16 @@ static inline void css_put_many(struct cgroup_subsys_state *css, unsigned int n)
percpu_ref_put_many(&css->refcnt, n);
}
+static inline void cgroup_get(struct cgroup *cgrp)
+{
+ css_get(&cgrp->self);
+}
+
+static inline bool cgroup_tryget(struct cgroup *cgrp)
+{
+ return css_tryget(&cgrp->self);
+}
+
static inline void cgroup_put(struct cgroup *cgrp)
{
css_put(&cgrp->self);
@@ -500,6 +510,20 @@ static inline struct cgroup *task_cgroup(struct task_struct *task,
return task_css(task, subsys_id)->cgroup;
}
+static inline struct cgroup *task_dfl_cgroup(struct task_struct *task)
+{
+ return task_css_set(task)->dfl_cgrp;
+}
+
+static inline struct cgroup *cgroup_parent(struct cgroup *cgrp)
+{
+ struct cgroup_subsys_state *parent_css = cgrp->self.parent;
+
+ if (parent_css)
+ return container_of(parent_css, struct cgroup, self);
+ return NULL;
+}
+
/**
* cgroup_is_descendant - test ancestry
* @cgrp: the cgroup to be tested
@@ -590,6 +614,11 @@ static inline void pr_cont_cgroup_path(struct cgroup *cgrp)
pr_cont_kernfs_path(cgrp->kn);
}
+static inline struct psi_group *cgroup_psi(struct cgroup *cgrp)
+{
+ return &cgrp->psi;
+}
+
/*
* Default Android check for whether the current process is allowed to move a
* task across cgroups, either because CAP_SYS_NICE is set or because the uid
@@ -641,6 +670,16 @@ static inline int cgroup_init(void) { return 0; }
static inline void cgroup_init_kthreadd(void) {}
static inline void cgroup_kthread_ready(void) {}
+static inline struct cgroup *cgroup_parent(struct cgroup *cgrp)
+{
+ return NULL;
+}
+
+static inline struct psi_group *cgroup_psi(struct cgroup *cgrp)
+{
+ return NULL;
+}
+
static inline bool task_under_cgroup_hierarchy(struct task_struct *task,
struct cgroup *ancestor)
{
diff --git a/include/linux/compat.h b/include/linux/compat.h
index d8535a4..fab35da 100644
--- a/include/linux/compat.h
+++ b/include/linux/compat.h
@@ -67,6 +67,9 @@ typedef struct compat_sigaltstack {
compat_size_t ss_size;
} compat_stack_t;
#endif
+#ifndef COMPAT_MINSIGSTKSZ
+#define COMPAT_MINSIGSTKSZ MINSIGSTKSZ
+#endif
#define compat_jiffies_to_clock_t(x) \
(((unsigned long)(x) * COMPAT_USER_HZ) / HZ)
diff --git a/include/linux/compiler-clang.h b/include/linux/compiler-clang.h
index e4f142c..058dd5d 100644
--- a/include/linux/compiler-clang.h
+++ b/include/linux/compiler-clang.h
@@ -16,6 +16,14 @@
*/
#define __UNIQUE_ID(prefix) __PASTE(__PASTE(__UNIQUE_ID_, prefix), __COUNTER__)
+/*
+ * GCC does not warn about unused static inline functions for
+ * -Wunused-function. This turns out to avoid the need for complex #ifdef
+ * directives. Suppress the warning in clang as well.
+ */
+#undef inline
+#define inline inline __attribute__((unused)) notrace
+
#ifdef CONFIG_LTO_CLANG
#ifdef CONFIG_FTRACE_MCOUNT_RECORD
#define __norecordmcount \
@@ -25,6 +33,8 @@
#define __nocfi __attribute__((no_sanitize("cfi")))
#endif
+#define __noscs __attribute__((no_sanitize("shadow-call-stack")))
+
/* all clang versions usable with the kernel support KASAN ABI version 5 */
#define KASAN_ABI_VERSION 5
@@ -33,6 +43,13 @@
#define __SANITIZE_ADDRESS__
#endif
+#undef __no_sanitize_address
+#if __has_feature(address_sanitizer)
+#define __no_sanitize_address __attribute__((no_sanitize("kernel-address")))
+#else
+#define __no_sanitize_address
+#endif
+
/* Clang doesn't have a way to turn it off per-function, yet. */
#ifdef __noretpoline
#undef __noretpoline
diff --git a/include/linux/compiler-gcc.h b/include/linux/compiler-gcc.h
index 27101bb..8e82e33 100644
--- a/include/linux/compiler-gcc.h
+++ b/include/linux/compiler-gcc.h
@@ -65,25 +65,40 @@
#endif
/*
+ * Feature detection for gnu_inline (gnu89 extern inline semantics). Either
+ * __GNUC_STDC_INLINE__ is defined (not using gnu89 extern inline semantics,
+ * and we opt in to the gnu89 semantics), or __GNUC_STDC_INLINE__ is not
+ * defined so the gnu89 semantics are the default.
+ */
+#ifdef __GNUC_STDC_INLINE__
+# define __gnu_inline __attribute__((gnu_inline))
+#else
+# define __gnu_inline
+#endif
+
+/*
* Force always-inline if the user requests it so via the .config,
* or if gcc is too old.
* GCC does not warn about unused static inline functions for
* -Wunused-function. This turns out to avoid the need for complex #ifdef
* directives. Suppress the warning in clang as well by using "unused"
* function attribute, which is redundant but not harmful for gcc.
+ * Prefer gnu_inline, so that extern inline functions do not emit an
+ * externally visible function. This makes extern inline behave as per gnu89
+ * semantics rather than c99. This prevents multiple symbol definition errors
+ * of extern inline functions at link time.
+ * A lot of inline functions can cause havoc with function tracing.
*/
#if !defined(CONFIG_ARCH_SUPPORTS_OPTIMIZED_INLINING) || \
!defined(CONFIG_OPTIMIZE_INLINING) || (__GNUC__ < 4)
-#define inline inline __attribute__((always_inline,unused)) notrace
-#define __inline__ __inline__ __attribute__((always_inline,unused)) notrace
-#define __inline __inline __attribute__((always_inline,unused)) notrace
+#define inline \
+ inline __attribute__((always_inline, unused)) notrace __gnu_inline
#else
-/* A lot of inline functions can cause havoc with function tracing */
-#define inline inline __attribute__((unused)) notrace
-#define __inline__ __inline__ __attribute__((unused)) notrace
-#define __inline __inline __attribute__((unused)) notrace
+#define inline inline __attribute__((unused)) notrace __gnu_inline
#endif
+#define __inline__ inline
+#define __inline inline
#define __always_inline inline __attribute__((always_inline))
#define noinline __attribute__((noinline))
@@ -207,6 +222,17 @@
#endif
#endif
+#ifdef CONFIG_STACK_VALIDATION
+#define annotate_unreachable() ({ \
+ asm("1:\t\n" \
+ ".pushsection .discard.unreachable\t\n" \
+ ".long 1b\t\n" \
+ ".popsection\t\n"); \
+})
+#else
+#define annotate_unreachable()
+#endif
+
/*
* Mark a position in code as unreachable. This can be used to
* suppress control flow warnings after asm blocks that transfer
@@ -216,7 +242,8 @@
* this in the preprocessor, but we can live with this because they're
* unreleased. Really, we need to have autoconf for the kernel.
*/
-#define unreachable() __builtin_unreachable()
+#define unreachable() \
+ do { annotate_unreachable(); __builtin_unreachable(); } while (0)
/* Mark a function definition as prohibited from being cloned. */
#define __noclone __attribute__((__noclone__, __optimize__("no-tracer")))
diff --git a/include/linux/compiler.h b/include/linux/compiler.h
index 81bcdcaa..8b1b837 100644
--- a/include/linux/compiler.h
+++ b/include/linux/compiler.h
@@ -113,7 +113,7 @@ void ftrace_likely_update(struct ftrace_branch_data *f, int val, int expect);
#define unlikely_notrace(x) __builtin_expect(!!(x), 0)
#define __branch_check__(x, expect) ({ \
- int ______r; \
+ long ______r; \
static struct ftrace_branch_data \
__attribute__((__aligned__(4))) \
__attribute__((section("_ftrace_annotated_branch"))) \
@@ -459,6 +459,10 @@ static __always_inline void __write_once_size(volatile void *p, void *res, int s
#define __nocfi
#endif
+#ifndef __noscs
+#define __noscs
+#endif
+
/*
* Assume alignment of return value.
*/
diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index 1f7e4ec..fdf5be4 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -29,7 +29,7 @@ struct cpu {
};
extern void boot_cpu_init(void);
-extern void boot_cpu_state_init(void);
+extern void boot_cpu_hotplug_init(void);
extern int register_cpu(struct cpu *cpu, int num);
extern struct device *get_cpu_device(unsigned cpu);
@@ -50,6 +50,10 @@ extern ssize_t cpu_show_spectre_v1(struct device *dev,
struct device_attribute *attr, char *buf);
extern ssize_t cpu_show_spectre_v2(struct device *dev,
struct device_attribute *attr, char *buf);
+extern ssize_t cpu_show_spec_store_bypass(struct device *dev,
+ struct device_attribute *attr, char *buf);
+extern ssize_t cpu_show_l1tf(struct device *dev,
+ struct device_attribute *attr, char *buf);
extern __printf(4, 5)
struct device *cpu_device_create(struct device *parent, void *drvdata,
@@ -186,7 +190,6 @@ extern struct bus_type cpu_subsys;
extern void cpu_hotplug_begin(void);
extern void cpu_hotplug_done(void);
extern void get_online_cpus(void);
-extern void cpu_hotplug_mutex_held(void);
extern void put_online_cpus(void);
extern void cpu_hotplug_disable(void);
extern void cpu_hotplug_enable(void);
@@ -209,7 +212,6 @@ static inline void cpu_hotplug_done(void) {}
#define cpu_hotplug_enable() do { } while (0)
#define hotcpu_notifier(fn, pri) do { (void)(fn); } while (0)
#define __hotcpu_notifier(fn, pri) do { (void)(fn); } while (0)
-#define cpu_hotplug_mutex_held() do { } while (0)
/* These aren't inline functions due to a GCC bug. */
#define register_hotcpu_notifier(nb) ({ (void)(nb); 0; })
#define __register_hotcpu_notifier(nb) ({ (void)(nb); 0; })
@@ -255,6 +257,25 @@ void cpuhp_report_idle_dead(void);
static inline void cpuhp_report_idle_dead(void) { }
#endif /* #ifdef CONFIG_HOTPLUG_CPU */
+enum cpuhp_smt_control {
+ CPU_SMT_ENABLED,
+ CPU_SMT_DISABLED,
+ CPU_SMT_FORCE_DISABLED,
+ CPU_SMT_NOT_SUPPORTED,
+};
+
+#if defined(CONFIG_SMP) && defined(CONFIG_HOTPLUG_SMT)
+extern enum cpuhp_smt_control cpu_smt_control;
+extern void cpu_smt_disable(bool force);
+extern void cpu_smt_check_topology_early(void);
+extern void cpu_smt_check_topology(void);
+#else
+# define cpu_smt_control (CPU_SMT_ENABLED)
+static inline void cpu_smt_disable(bool force) { }
+static inline void cpu_smt_check_topology_early(void) { }
+static inline void cpu_smt_check_topology(void) { }
+#endif
+
#define IDLE_START 1
#define IDLE_END 2
diff --git a/include/linux/cpufreq_times.h b/include/linux/cpufreq_times.h
index 3fb3875..0d87b80 100644
--- a/include/linux/cpufreq_times.h
+++ b/include/linux/cpufreq_times.h
@@ -22,9 +22,14 @@
#ifdef CONFIG_CPU_FREQ_TIMES
void cpufreq_task_times_init(struct task_struct *p);
+void cpufreq_task_times_alloc(struct task_struct *p);
void cpufreq_task_times_exit(struct task_struct *p);
int proc_time_in_state_show(struct seq_file *m, struct pid_namespace *ns,
struct pid *pid, struct task_struct *p);
+int proc_concurrent_active_time_show(struct seq_file *m,
+ struct pid_namespace *ns, struct pid *pid, struct task_struct *p);
+int proc_concurrent_policy_time_show(struct seq_file *m,
+ struct pid_namespace *ns, struct pid *pid, struct task_struct *p);
void cpufreq_acct_update_power(struct task_struct *p, cputime_t cputime);
void cpufreq_times_create_policy(struct cpufreq_policy *policy);
void cpufreq_times_record_transition(struct cpufreq_freqs *freq);
diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h
index bd96bb2..df17901 100644
--- a/include/linux/cpumask.h
+++ b/include/linux/cpumask.h
@@ -179,6 +179,8 @@ static inline unsigned int cpumask_local_spread(unsigned int i, int node)
for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask)
#define for_each_cpu_not(cpu, mask) \
for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask)
+#define for_each_cpu_wrap(cpu, mask, start) \
+ for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask, (void)(start))
#define for_each_cpu_and(cpu, mask, and) \
for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask, (void)and)
#else
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 6666ea0..0fb7487 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -51,6 +51,8 @@
#define CRYPTO_ALG_TYPE_SKCIPHER 0x00000005
#define CRYPTO_ALG_TYPE_GIVCIPHER 0x00000006
#define CRYPTO_ALG_TYPE_KPP 0x00000008
+#define CRYPTO_ALG_TYPE_ACOMPRESS 0x0000000a
+#define CRYPTO_ALG_TYPE_SCOMPRESS 0x0000000b
#define CRYPTO_ALG_TYPE_RNG 0x0000000c
#define CRYPTO_ALG_TYPE_AKCIPHER 0x0000000d
#define CRYPTO_ALG_TYPE_DIGEST 0x0000000e
@@ -61,6 +63,7 @@
#define CRYPTO_ALG_TYPE_HASH_MASK 0x0000000e
#define CRYPTO_ALG_TYPE_AHASH_MASK 0x0000000e
#define CRYPTO_ALG_TYPE_BLKCIPHER_MASK 0x0000000c
+#define CRYPTO_ALG_TYPE_ACOMPRESS_MASK 0x0000000e
#define CRYPTO_ALG_LARVAL 0x00000010
#define CRYPTO_ALG_DEAD 0x00000020
@@ -110,6 +113,11 @@
#define CRYPTO_ALG_OPTIONAL_KEY 0x00004000
/*
+ * Don't trigger module loading
+ */
+#define CRYPTO_NOLOAD 0x00008000
+
+/*
* Transform masks and values (for crt_flags).
*/
#define CRYPTO_TFM_NEED_KEY 0x00000001
diff --git a/include/linux/dcache.h b/include/linux/dcache.h
index 3d4a198..f658307 100644
--- a/include/linux/dcache.h
+++ b/include/linux/dcache.h
@@ -220,6 +220,7 @@ extern seqlock_t rename_lock;
* These are the low-level FS interfaces to the dcache..
*/
extern void d_instantiate(struct dentry *, struct inode *);
+extern void d_instantiate_new(struct dentry *, struct inode *);
extern struct dentry * d_instantiate_unique(struct dentry *, struct inode *);
extern int d_instantiate_no_diralias(struct dentry *, struct inode *);
extern void __d_drop(struct dentry *dentry);
@@ -592,8 +593,8 @@ static inline struct inode *d_real_inode(const struct dentry *dentry)
}
struct name_snapshot {
- const char *name;
- char inline_name[DNAME_INLINE_LEN];
+ const unsigned char *name;
+ unsigned char inline_name[DNAME_INLINE_LEN];
};
void take_dentry_name_snapshot(struct name_snapshot *, struct dentry *);
void release_dentry_name_snapshot(struct name_snapshot *);
diff --git a/include/linux/delayacct.h b/include/linux/delayacct.h
index 6cee17c..84c2f682 100644
--- a/include/linux/delayacct.h
+++ b/include/linux/delayacct.h
@@ -29,7 +29,48 @@
#define DELAYACCT_PF_BLKIO 0x00000002 /* I am waiting on IO */
#ifdef CONFIG_TASK_DELAY_ACCT
+struct task_delay_info {
+ spinlock_t lock;
+ unsigned int flags; /* Private per-task flags */
+ /* For each stat XXX, add following, aligned appropriately
+ *
+ * struct timespec XXX_start, XXX_end;
+ * u64 XXX_delay;
+ * u32 XXX_count;
+ *
+ * Atomicity of updates to XXX_delay, XXX_count protected by
+ * single lock above (split into XXX_lock if contention is an issue).
+ */
+
+ /*
+ * XXX_count is incremented on every XXX operation, the delay
+ * associated with the operation is added to XXX_delay.
+ * XXX_delay contains the accumulated delay time in nanoseconds.
+ */
+ u64 blkio_start; /* Shared by blkio, swapin */
+ u64 blkio_delay; /* wait for sync block io completion */
+ u64 swapin_delay; /* wait for swapin block io completion */
+ u32 blkio_count; /* total count of the number of sync block */
+ /* io operations performed */
+ u32 swapin_count; /* total count of the number of swapin block */
+ /* io operations performed */
+
+ u64 freepages_start;
+ u64 freepages_delay; /* wait for memory reclaim */
+
+ u64 thrashing_start;
+ u64 thrashing_delay; /* wait for thrashing page */
+
+ u32 freepages_count; /* total count of memory reclaim */
+ u32 thrashing_count; /* total count of thrash waits */
+};
+#endif
+
+#include <linux/sched.h>
+#include <linux/slab.h>
+
+#ifdef CONFIG_TASK_DELAY_ACCT
extern int delayacct_on; /* Delay accounting turned on/off */
extern struct kmem_cache *delayacct_cache;
extern void delayacct_init(void);
@@ -41,6 +82,8 @@ extern int __delayacct_add_tsk(struct taskstats *, struct task_struct *);
extern __u64 __delayacct_blkio_ticks(struct task_struct *);
extern void __delayacct_freepages_start(void);
extern void __delayacct_freepages_end(void);
+extern void __delayacct_thrashing_start(void);
+extern void __delayacct_thrashing_end(void);
static inline int delayacct_is_task_waiting_on_io(struct task_struct *p)
{
@@ -121,6 +164,18 @@ static inline void delayacct_freepages_end(void)
__delayacct_freepages_end();
}
+static inline void delayacct_thrashing_start(void)
+{
+ if (current->delays)
+ __delayacct_thrashing_start();
+}
+
+static inline void delayacct_thrashing_end(void)
+{
+ if (current->delays)
+ __delayacct_thrashing_end();
+}
+
#else
static inline void delayacct_set_flag(int flag)
{}
@@ -147,6 +202,10 @@ static inline void delayacct_freepages_start(void)
{}
static inline void delayacct_freepages_end(void)
{}
+static inline void delayacct_thrashing_start(void)
+{}
+static inline void delayacct_thrashing_end(void)
+{}
#endif /* CONFIG_TASK_DELAY_ACCT */
diff --git a/include/linux/device.h b/include/linux/device.h
index f43db28..0ea6ed1 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -98,6 +98,8 @@ extern void bus_remove_file(struct bus_type *, struct bus_attribute *);
* @p: The private data of the driver core, only the driver core can
* touch this.
* @lock_key: Lock class key for use by the lock validator
+ * @need_parent_lock: When probing or removing a device on this bus, the
+ * device core should lock the device's parent.
*
* A bus is a channel between the processor and one or more devices. For the
* purposes of the device model, all devices are connected via a bus, even if
@@ -135,6 +137,7 @@ struct bus_type {
struct subsys_private *p;
struct lock_class_key lock_key;
+ bool need_parent_lock;
};
extern int __must_check bus_register(struct bus_type *bus);
diff --git a/include/linux/dma-buf.h b/include/linux/dma-buf.h
index e0b0741..428877d 100644
--- a/include/linux/dma-buf.h
+++ b/include/linux/dma-buf.h
@@ -112,10 +112,12 @@ struct dma_buf_ops {
* @file: file pointer used for sharing buffers across, and for refcounting.
* @attachments: list of dma_buf_attachment that denotes all devices attached.
* @ops: dma_buf_ops associated with this buffer object.
- * @lock: used internally to serialize list manipulation, attach/detach and vmap/unmap
+ * @lock: used internally to serialize list manipulation, attach/detach and
+ * vmap/unmap, and accesses to name
* @vmapping_counter: used internally to refcnt the vmaps
* @vmap_ptr: the current vmap ptr if vmapping_counter > 0
* @exp_name: name of the exporter; useful for debugging.
+ * @name: userspace-provided name; useful for accounting and debugging.
* @owner: pointer to exporter module; used for refcounting when exporter is a
* kernel module.
* @list_node: node for dma_buf accounting and debugging.
@@ -134,6 +136,7 @@ struct dma_buf {
unsigned vmapping_counter;
void *vmap_ptr;
const char *exp_name;
+ const char *name;
struct module *owner;
struct list_head list_node;
void *priv;
diff --git a/include/linux/dma-iommu.h b/include/linux/dma-iommu.h
index 8dce6fd..f17e1f2 100644
--- a/include/linux/dma-iommu.h
+++ b/include/linux/dma-iommu.h
@@ -17,6 +17,7 @@
#define __DMA_IOMMU_H
#ifdef __KERNEL__
+#include <linux/types.h>
#include <asm/errno.h>
#ifdef CONFIG_IOMMU_DMA
diff --git a/include/linux/ecryptfs.h b/include/linux/ecryptfs.h
index 8d5ab99..33d35ea 100644
--- a/include/linux/ecryptfs.h
+++ b/include/linux/ecryptfs.h
@@ -1,6 +1,9 @@
#ifndef _LINUX_ECRYPTFS_H
#define _LINUX_ECRYPTFS_H
+struct inode;
+struct page;
+
/* Version verification for shared data structures w/ userspace */
#define ECRYPTFS_VERSION_MAJOR 0x00
#define ECRYPTFS_VERSION_MINOR 0x04
@@ -41,6 +44,7 @@
#define RFC2440_CIPHER_AES_256 0x09
#define RFC2440_CIPHER_TWOFISH 0x0a
#define RFC2440_CIPHER_CAST_6 0x0b
+#define RFC2440_CIPHER_AES_XTS_256 0x0c
#define RFC2440_CIPHER_RSA 0x01
@@ -102,4 +106,49 @@ struct ecryptfs_auth_tok {
} token;
} __attribute__ ((packed));
+#define ECRYPTFS_INVALID_EVENTS_HANDLE -1
+
+/**
+ * ecryptfs_events struct represents a partial interface
+ * towards ecryptfs module. If registered to ecryptfs events,
+ * one can receive push notifications.
+ * A first callback received from ecryptfs will probably be
+ * about file opening (open_cb),
+ * in which ecryptfs passes its ecryptfs_data for future usage.
+ * This data represents a file and must be passed in every query functions
+ * such as ecryptfs_get_key_size(), ecryptfs_get_cipher() etc.
+ */
+struct ecryptfs_events {
+ bool (*is_cipher_supported_cb)(const void *ecrytpfs_data);
+ void (*open_cb)(struct inode *inode, void *ecrytpfs_data);
+ void (*release_cb)(struct inode *inode);
+ int (*encrypt_cb)(struct page *in_page, struct page *out_page,
+ struct inode *inode, unsigned long extent_offset);
+ int (*decrypt_cb)(struct page *in_page, struct page *out_page,
+ struct inode *inode, unsigned long extent_offset);
+ bool (*is_hw_crypt_cb)(void);
+ size_t (*get_salt_key_size_cb)(const void *ecrytpfs_data);
+};
+
+
+int ecryptfs_register_to_events(const struct ecryptfs_events *ops);
+
+int ecryptfs_unregister_from_events(int user_handle);
+
+const unsigned char *ecryptfs_get_key(const void *ecrytpfs_data);
+
+size_t ecryptfs_get_key_size(const void *ecrytpfs_data);
+
+const unsigned char *ecryptfs_get_salt(const void *ecrytpfs_data);
+
+size_t ecryptfs_get_salt_size(const void *ecrytpfs_data);
+
+bool ecryptfs_cipher_match(const void *ecrytpfs_data,
+ const unsigned char *cipher, size_t cipher_size);
+
+bool ecryptfs_is_page_in_metadata(const void *ecrytpfs_data, pgoff_t offset);
+
+bool ecryptfs_is_data_equal(const void *ecrytpfs_data1,
+ const void *ecrytpfs_data2);
+
#endif /* _LINUX_ECRYPTFS_H */
diff --git a/include/linux/efi.h b/include/linux/efi.h
index 5e204a5..2877ccb 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -380,8 +380,8 @@ typedef struct {
u32 attributes;
u32 get_bar_attributes;
u32 set_bar_attributes;
- uint64_t romsize;
- void *romimage;
+ u64 romsize;
+ u32 romimage;
} efi_pci_io_protocol_32;
typedef struct {
@@ -400,8 +400,8 @@ typedef struct {
u64 attributes;
u64 get_bar_attributes;
u64 set_bar_attributes;
- uint64_t romsize;
- void *romimage;
+ u64 romsize;
+ u64 romimage;
} efi_pci_io_protocol_64;
typedef struct {
diff --git a/include/linux/elevator.h b/include/linux/elevator.h
index e7f358d..eaa58c0 100644
--- a/include/linux/elevator.h
+++ b/include/linux/elevator.h
@@ -102,7 +102,7 @@ struct elevator_type
struct module *elevator_owner;
/* managed by elevator core */
- char icq_cache_name[ELV_NAME_MAX + 5]; /* elvname + "_io_cq" */
+ char icq_cache_name[ELV_NAME_MAX + 6]; /* elvname + "_io_cq" */
struct list_head list;
};
diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h
index aa5db8b..d771104 100644
--- a/include/linux/f2fs_fs.h
+++ b/include/linux/f2fs_fs.h
@@ -1,12 +1,9 @@
+// SPDX-License-Identifier: GPL-2.0
/**
* include/linux/f2fs_fs.h
*
* Copyright (c) 2012 Samsung Electronics Co., Ltd.
* http://www.samsung.com/
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
*/
#ifndef _LINUX_F2FS_FS_H
#define _LINUX_F2FS_FS_H
@@ -112,12 +109,15 @@ struct f2fs_super_block {
struct f2fs_device devs[MAX_DEVICES]; /* device list */
__le32 qf_ino[F2FS_MAX_QUOTAS]; /* quota inode numbers */
__u8 hot_ext_count; /* # of hot file extension */
- __u8 reserved[314]; /* valid reserved region */
+ __u8 reserved[310]; /* valid reserved region */
+ __le32 crc; /* checksum of superblock */
} __packed;
/*
* For checkpoint
*/
+#define CP_DISABLED_FLAG 0x00001000
+#define CP_QUOTA_NEED_FSCK_FLAG 0x00000800
#define CP_LARGE_NAT_BITMAP_FLAG 0x00000400
#define CP_NOCRC_RECOVERY_FLAG 0x00000200
#define CP_TRIMMED_FLAG 0x00000100
@@ -304,11 +304,6 @@ struct f2fs_node {
* For NAT entries
*/
#define NAT_ENTRY_PER_BLOCK (PAGE_SIZE / sizeof(struct f2fs_nat_entry))
-#define NAT_ENTRY_BITMAP_SIZE ((NAT_ENTRY_PER_BLOCK + 7) / 8)
-#define NAT_ENTRY_BITMAP_SIZE_ALIGNED \
- ((NAT_ENTRY_BITMAP_SIZE + BITS_PER_LONG - 1) / \
- BITS_PER_LONG * BITS_PER_LONG)
-
struct f2fs_nat_entry {
__u8 version; /* latest version of cached nat entry */
diff --git a/include/linux/file.h b/include/linux/file.h
index 7444f5fe..9cf3ecf 100644
--- a/include/linux/file.h
+++ b/include/linux/file.h
@@ -84,6 +84,7 @@ extern void put_unused_fd(unsigned int fd);
extern void fd_install(unsigned int fd, struct file *file);
extern void flush_delayed_fput(void);
+extern void flush_delayed_fput_wait(void);
extern void __fput_sync(struct file *);
#endif /* __LINUX_FILE_H */
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 1c4da43..4f4c1c0 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -57,6 +57,8 @@ struct workqueue_struct;
struct iov_iter;
struct fscrypt_info;
struct fscrypt_operations;
+struct fsverity_info;
+struct fsverity_operations;
extern void __init inode_init(void);
extern void __init inode_init_early(void);
@@ -70,6 +72,8 @@ extern struct inodes_stat_t inodes_stat;
extern int leases_enable, lease_break_time;
extern int sysctl_protected_symlinks;
extern int sysctl_protected_hardlinks;
+extern int sysctl_protected_fifos;
+extern int sysctl_protected_regular;
struct buffer_head;
typedef int (get_block_t)(struct inode *inode, sector_t iblock,
@@ -723,6 +727,11 @@ struct inode {
struct fscrypt_info *i_crypt_info;
#endif
+#if IS_ENABLED(CONFIG_FS_VERITY)
+ struct fsverity_info *i_verity_info;
+ /* TODO(mhalcrow): Not for upstream */
+ struct list_head i_fsverity_list; /* Not for upstream */
+#endif
void *i_private; /* fs or device private pointer */
};
@@ -1385,6 +1394,10 @@ struct super_block {
const struct fscrypt_operations *s_cop;
+#if IS_ENABLED(CONFIG_FS_VERITY)
+ const struct fsverity_operations *s_vop;
+#endif
+
struct hlist_bl_head s_anon; /* anonymous dentries for (nfs) exporting */
struct list_head s_mounts; /* list of mounts; _not_ for fs use */
struct block_device *s_bdev;
@@ -1472,6 +1485,12 @@ struct super_block {
spinlock_t s_inode_wblist_lock;
struct list_head s_inodes_wb; /* writeback inodes */
+#if IS_ENABLED(CONFIG_FS_VERITY)
+ /* TODO(mhalcrow): Not for upstream */
+ /* fs-verity inodes being pinned in memory */
+ spinlock_t s_inode_fsveritylist_lock;
+ struct list_head s_inodes_fsverity;
+#endif
};
/* Helper functions so that in most cases filesystems will
@@ -1838,6 +1857,7 @@ struct super_operations {
void *(*clone_mnt_data) (void *);
void (*copy_mnt_data) (void *, void *);
void (*umount_begin) (struct super_block *);
+ void (*umount_end) (struct super_block *, int);
int (*show_options)(struct seq_file *, struct dentry *);
int (*show_options2)(struct vfsmount *,struct seq_file *, struct dentry *);
diff --git a/include/linux/fscrypt.h b/include/linux/fscrypt.h
index 1e03614..1c993ae 100644
--- a/include/linux/fscrypt.h
+++ b/include/linux/fscrypt.h
@@ -18,11 +18,6 @@
#define FS_CRYPTO_BLOCK_SIZE 16
struct fscrypt_ctx;
-
-/* iv sector for security/pfe/pfk_fscrypt.c and f2fs */
-#define PG_DUN(i, p) \
- (((((u64)(i)->i_ino) & 0xffffffff) << 32) | ((p)->index & 0xffffffff))
-
struct fscrypt_info;
struct fscrypt_str {
@@ -256,8 +251,10 @@ static inline int fscrypt_encrypt_symlink(struct inode *inode,
#ifdef CONFIG_PFK
extern int fscrypt_using_hardware_encryption(const struct inode *inode);
extern void fscrypt_set_ice_dun(const struct inode *inode,
- struct bio *bio, u64 dun);
-extern bool fscrypt_mergeable_bio(struct bio *bio, u64 dun, bool bio_encrypted);
+ struct bio *bio, u64 dun);
+extern void fscrypt_set_ice_skip(struct bio *bio, int bi_crypt_skip);
+extern bool fscrypt_mergeable_bio(struct bio *bio, u64 dun, bool bio_encrypted,
+ int bi_crypt_skip);
#else
static inline int fscrypt_using_hardware_encryption(const struct inode *inode)
{
@@ -265,13 +262,18 @@ static inline int fscrypt_using_hardware_encryption(const struct inode *inode)
}
static inline void fscrypt_set_ice_dun(const struct inode *inode,
- struct bio *bio, u64 dun)
+ struct bio *bio, u64 dun)
+{
+ return;
+}
+
+static inline void fscrypt_set_ice_skip(struct bio *bio, int bi_crypt_skip)
{
return;
}
static inline bool fscrypt_mergeable_bio(struct bio *bio,
- u64 dun, bool bio_encrypted)
+ u64 dun, bool bio_encrypted, int bi_crypt_skip)
{
return true;
}
diff --git a/include/linux/fscrypt_notsupp.h b/include/linux/fscrypt_notsupp.h
index 172d418..5017122 100644
--- a/include/linux/fscrypt_notsupp.h
+++ b/include/linux/fscrypt_notsupp.h
@@ -67,16 +67,6 @@ static inline void fscrypt_restore_control_page(struct page *page)
return;
}
-static inline void fscrypt_set_d_op(struct dentry *dentry)
-{
- return;
-}
-
-static inline void fscrypt_set_encrypted_dentry(struct dentry *dentry)
-{
- return;
-}
-
/* policy.c */
static inline int fscrypt_ioctl_set_policy(struct file *filp,
const void __user *arg)
diff --git a/include/linux/fscrypt_supp.h b/include/linux/fscrypt_supp.h
index 1ed79ee..99b6c52 100644
--- a/include/linux/fscrypt_supp.h
+++ b/include/linux/fscrypt_supp.h
@@ -28,7 +28,7 @@ struct fscrypt_operations {
int (*set_context)(struct inode *, const void *, size_t, void *);
bool (*dummy_context)(struct inode *);
bool (*empty_dir)(struct inode *);
- unsigned (*max_namelen)(struct inode *);
+ unsigned int max_namelen;
};
struct fscrypt_ctx {
@@ -74,20 +74,6 @@ static inline struct page *fscrypt_control_page(struct page *page)
extern void fscrypt_restore_control_page(struct page *);
-extern const struct dentry_operations fscrypt_d_ops;
-
-static inline void fscrypt_set_d_op(struct dentry *dentry)
-{
- d_set_d_op(dentry, &fscrypt_d_ops);
-}
-
-static inline void fscrypt_set_encrypted_dentry(struct dentry *dentry)
-{
- spin_lock(&dentry->d_lock);
- dentry->d_flags |= DCACHE_ENCRYPTED_WITH_KEY;
- spin_unlock(&dentry->d_lock);
-}
-
/* policy.c */
extern int fscrypt_ioctl_set_policy(struct file *, const void __user *);
extern int fscrypt_ioctl_get_policy(struct file *, void __user *);
diff --git a/include/linux/fsl/guts.h b/include/linux/fsl/guts.h
index 649e917..270eef7 100644
--- a/include/linux/fsl/guts.h
+++ b/include/linux/fsl/guts.h
@@ -16,6 +16,7 @@
#define __FSL_GUTS_H__
#include <linux/types.h>
+#include <linux/io.h>
/**
* Global Utility Registers.
diff --git a/include/linux/fsverity.h b/include/linux/fsverity.h
new file mode 100644
index 0000000..7a34b4c
--- /dev/null
+++ b/include/linux/fsverity.h
@@ -0,0 +1,92 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * fs-verity: read-only file-based integrity/authenticity
+ *
+ * Copyright (C) 2018, Google, Inc.
+ *
+ * Written by Jaegeuk Kim, Michael Halcrow, and Eric Biggers.
+ */
+
+#ifndef _LINUX_FSVERITY_H
+#define _LINUX_FSVERITY_H
+
+#include <linux/fs.h>
+
+/*
+ * fs-verity operations for filesystems
+ */
+struct fsverity_operations {
+ bool (*is_verity)(struct inode *inode);
+ int (*set_verity)(struct inode *inode);
+ struct page *(*read_metadata_page)(struct inode *inode, pgoff_t index);
+};
+
+#if __FS_HAS_VERITY
+extern int fsverity_file_open(struct inode *inode, struct file *filp);
+extern int fsverity_prepare_setattr(struct dentry *dentry, struct iattr *attr);
+extern int fsverity_prepare_getattr(struct inode *inode);
+extern void fsverity_cleanup_inode(struct inode *inode);
+extern bool fsverity_verify_page(struct page *page);
+extern void fsverity_verify_bio(struct bio *bio);
+extern void fsverity_enqueue_verify_work(struct work_struct *work);
+extern loff_t fsverity_full_isize(struct inode *inode);
+extern int fsverity_ioctl_set_measurement(struct file *filp,
+ const void __user *arg);
+extern int fsverity_ioctl_enable(struct file *filp, const void __user *arg);
+#else
+static inline int fsverity_file_open(struct inode *inode, struct file *filp)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline int fsverity_prepare_setattr(struct dentry *dentry,
+ struct iattr *attr)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline int fsverity_prepare_getattr(struct inode *inode)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline void fsverity_cleanup_inode(struct inode *inode)
+{
+}
+
+static inline bool fsverity_verify_page(struct page *page)
+{
+ WARN_ON(1);
+ return false;
+}
+
+static inline void fsverity_verify_bio(struct bio *bio)
+{
+ WARN_ON(1);
+}
+
+static inline void fsverity_enqueue_verify_work(struct work_struct *work)
+{
+ WARN_ON(1);
+}
+
+static inline loff_t fsverity_full_isize(struct inode *inode)
+{
+ return i_size_read(inode);
+}
+
+static inline int fsverity_ioctl_set_measurement(struct file *filp,
+ const void __user *arg)
+{
+ return -EOPNOTSUPP;
+}
+
+static inline int fsverity_ioctl_enable(struct file *filp,
+ const void __user *arg)
+{
+ return -EOPNOTSUPP;
+}
+
+#endif /* !__FS_HAS_VERITY */
+
+#endif /* _LINUX_FSVERITY_H */
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index f4c0d36..ab7938a 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -244,8 +244,16 @@ static inline int ftrace_function_local_disabled(struct ftrace_ops *ops)
return *this_cpu_ptr(ops->disabled);
}
+#ifdef CONFIG_CFI_CLANG
+/* Use a C stub with the correct type for CFI */
+static inline void ftrace_stub(unsigned long a0, unsigned long a1,
+ struct ftrace_ops *op, struct pt_regs *regs)
+{
+}
+#else
extern void ftrace_stub(unsigned long a0, unsigned long a1,
struct ftrace_ops *op, struct pt_regs *regs);
+#endif
#else /* !CONFIG_FUNCTION_TRACER */
/*
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index e35e6de..9b9f65d 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -22,7 +22,7 @@ extern int mincore_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
unsigned char *vec);
extern bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
unsigned long new_addr, unsigned long old_end,
- pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush);
+ pmd_t *old_pmd, pmd_t *new_pmd);
extern int change_huge_pmd(struct vm_area_struct *vma, pmd_t *pmd,
unsigned long addr, pgprot_t newprot,
int prot_numa);
diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h
index 48c76d6..b699d59 100644
--- a/include/linux/hugetlb.h
+++ b/include/linux/hugetlb.h
@@ -109,6 +109,8 @@ pte_t *huge_pte_alloc(struct mm_struct *mm,
unsigned long addr, unsigned long sz);
pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr);
int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep);
+void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
+ unsigned long *start, unsigned long *end);
struct page *follow_huge_addr(struct mm_struct *mm, unsigned long address,
int write);
struct page *follow_huge_pmd(struct mm_struct *mm, unsigned long address,
@@ -131,6 +133,18 @@ static inline unsigned long hugetlb_total_pages(void)
return 0;
}
+static inline int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr,
+ pte_t *ptep)
+{
+ return 0;
+}
+
+static inline void adjust_range_if_pmd_sharing_possible(
+ struct vm_area_struct *vma,
+ unsigned long *start, unsigned long *end)
+{
+}
+
#define follow_hugetlb_page(m,v,p,vs,a,b,i,w) ({ BUG(); 0; })
#define follow_huge_addr(mm, addr, write) ERR_PTR(-EINVAL)
#define copy_hugetlb_page_range(src, dst, vma) ({ BUG(); 0; })
diff --git a/include/linux/i2c/fts.h b/include/linux/i2c/fts.h
new file mode 100644
index 0000000..e13aa8d
--- /dev/null
+++ b/include/linux/i2c/fts.h
@@ -0,0 +1,89 @@
+#ifndef _LINUX_FTS_I2C_H_
+#define _LINUX_FTS_I2C_H_
+
+extern struct fts_callbacks *fts_charger_callbacks;
+struct fts_callbacks {
+ void (*inform_charger) (struct fts_callbacks *, int);
+};
+
+#ifdef FTS_SUPPORT_NOISE_PARAM
+#define MAX_NOISE_PARAM 5
+struct fts_noise_param {
+ unsigned short pAddr[MAX_NOISE_PARAM];
+ unsigned short pData[MAX_NOISE_PARAM];
+};
+#endif
+
+#ifdef FTS_SUPPORT_TOUCH_KEY
+/* TSP Key Feature*/
+#define KEY_PRESS 1
+#define KEY_RELEASE 0
+#define TOUCH_KEY_NULL 0
+
+/* support 2 touch keys */
+#define TOUCH_KEY_RECENT 0x01
+#define TOUCH_KEY_BACK 0x02
+
+struct fts_touchkey {
+ unsigned int value;
+ unsigned int keycode;
+ char *name;
+};
+#endif
+
+struct fts_i2c_platform_data {
+ bool factory_flatform;
+ bool recovery_mode;
+ bool support_hover;
+ bool support_mshover;
+ int max_x;
+ int max_y;
+ int max_width;
+ int grip_area;
+ int SenseChannelLength;
+ int ForceChannelLength;
+ unsigned char panel_revision; /* to identify panel info */
+
+ const char *firmware_name;
+ const char *project_name;
+ const char *model_name;
+ const char *regulator_dvdd;
+ const char *regulator_avdd;
+
+ struct pinctrl *pinctrl;
+ struct pinctrl_state *pins_default;
+ struct pinctrl_state *pins_sleep;
+
+ int (*power)(void *data, bool on);
+ void (*register_cb)(void *);
+ void (*enable_sync)(bool on);
+ unsigned char (*get_ddi_type)(void); /* to indentify ddi type */
+
+ unsigned tspid;
+ unsigned tspid2;
+ unsigned gpio;
+ int vdd_gpio;
+ int vio_gpio;
+ int irq_type;
+
+#ifdef FTS_SUPPORT_TOUCH_KEY
+ bool support_mskey;
+ unsigned int num_touchkey;
+ struct fts_touchkey *touchkey;
+ const char *regulator_tk_led;
+ int (*led_power) (void *, bool);
+#endif
+ unsigned gpio_scl;
+ unsigned gpio_sda;
+ int switch_gpio;
+};
+
+// #define FTS_SUPPORT_TA_MODE // DE version don't need.
+
+extern unsigned int lcdtype;
+
+void fts_charger_infom(bool en);
+#ifdef CONFIG_TRUSTONIC_TRUSTED_UI
+extern void trustedui_mode_on(void);
+#endif
+#endif
diff --git a/include/linux/i8253.h b/include/linux/i8253.h
index e6bb36a..8336b2f 100644
--- a/include/linux/i8253.h
+++ b/include/linux/i8253.h
@@ -21,6 +21,7 @@
#define PIT_LATCH ((PIT_TICK_RATE + HZ/2) / HZ)
extern raw_spinlock_t i8253_lock;
+extern bool i8253_clear_counter_on_shutdown;
extern struct clock_event_device i8253_clockevent;
extern void clockevent_i8253_init(bool oneshot);
diff --git a/include/linux/if_vlan.h b/include/linux/if_vlan.h
index 8feecd5..7e39719 100644
--- a/include/linux/if_vlan.h
+++ b/include/linux/if_vlan.h
@@ -600,7 +600,7 @@ static inline bool skb_vlan_tagged(const struct sk_buff *skb)
* Returns true if the skb is tagged with multiple vlan headers, regardless
* of whether it is hardware accelerated or not.
*/
-static inline bool skb_vlan_tagged_multi(const struct sk_buff *skb)
+static inline bool skb_vlan_tagged_multi(struct sk_buff *skb)
{
__be16 protocol = skb->protocol;
@@ -610,6 +610,9 @@ static inline bool skb_vlan_tagged_multi(const struct sk_buff *skb)
if (likely(!eth_type_vlan(protocol)))
return false;
+ if (unlikely(!pskb_may_pull(skb, VLAN_ETH_HLEN)))
+ return false;
+
veh = (struct vlan_ethhdr *)skb->data;
protocol = veh->h_vlan_encapsulated_proto;
}
@@ -627,7 +630,7 @@ static inline bool skb_vlan_tagged_multi(const struct sk_buff *skb)
*
* Returns features without unsafe ones if the skb has multiple tags.
*/
-static inline netdev_features_t vlan_features_check(const struct sk_buff *skb,
+static inline netdev_features_t vlan_features_check(struct sk_buff *skb,
netdev_features_t features)
{
if (skb_vlan_tagged_multi(skb)) {
diff --git a/include/linux/iio/buffer-dma.h b/include/linux/iio/buffer-dma.h
index 767467d..67c7537 100644
--- a/include/linux/iio/buffer-dma.h
+++ b/include/linux/iio/buffer-dma.h
@@ -141,7 +141,7 @@ int iio_dma_buffer_read(struct iio_buffer *buffer, size_t n,
char __user *user_buffer);
size_t iio_dma_buffer_data_available(struct iio_buffer *buffer);
int iio_dma_buffer_set_bytes_per_datum(struct iio_buffer *buffer, size_t bpd);
-int iio_dma_buffer_set_length(struct iio_buffer *buffer, int length);
+int iio_dma_buffer_set_length(struct iio_buffer *buffer, unsigned int length);
int iio_dma_buffer_request_update(struct iio_buffer *buffer);
int iio_dma_buffer_init(struct iio_dma_buffer_queue *queue,
diff --git a/include/linux/iio/buffer.h b/include/linux/iio/buffer.h
index 70a5164..821965c 100644
--- a/include/linux/iio/buffer.h
+++ b/include/linux/iio/buffer.h
@@ -61,7 +61,7 @@ struct iio_buffer_access_funcs {
int (*request_update)(struct iio_buffer *buffer);
int (*set_bytes_per_datum)(struct iio_buffer *buffer, size_t bpd);
- int (*set_length)(struct iio_buffer *buffer, int length);
+ int (*set_length)(struct iio_buffer *buffer, unsigned int length);
int (*enable)(struct iio_buffer *buffer, struct iio_dev *indio_dev);
int (*disable)(struct iio_buffer *buffer, struct iio_dev *indio_dev);
@@ -96,8 +96,8 @@ struct iio_buffer_access_funcs {
* @watermark: [INTERN] number of datums to wait for poll/read.
*/
struct iio_buffer {
- int length;
- int bytes_per_datum;
+ unsigned int length;
+ size_t bytes_per_datum;
struct attribute_group *scan_el_attrs;
long *scan_mask;
bool scan_timestamp;
diff --git a/include/linux/input.h b/include/linux/input.h
index a65e3b2..c90ebef 100644
--- a/include/linux/input.h
+++ b/include/linux/input.h
@@ -175,7 +175,9 @@ struct input_dev {
struct mutex mutex;
unsigned int users;
+ unsigned int users_private;
bool going_away;
+ bool disabled;
struct device dev;
diff --git a/include/linux/input/heatmap.h b/include/linux/input/heatmap.h
new file mode 100644
index 0000000..07e01e3
--- /dev/null
+++ b/include/linux/input/heatmap.h
@@ -0,0 +1,47 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#include <media/v4l2-device.h>
+#include <media/v4l2-ioctl.h>
+#include <media/videobuf2-v4l2.h>
+
+typedef int16_t strength_t;
+
+struct v4l2_heatmap {
+ struct device *parent_dev;
+ /* Can be NULL. Used to get the input device name */
+ struct input_dev *input_dev;
+ struct v4l2_device device;
+ struct v4l2_pix_format format;
+ struct video_device vdev;
+ struct vb2_queue queue;
+ struct mutex lock;
+ unsigned int input_index;
+
+ size_t width;
+ size_t height;
+
+ struct v4l2_fract timeperframe;
+
+ /* Used to protect access to the buffer queue */
+ spinlock_t heatmap_lock;
+ /* guarded by heatmap_lock */
+ struct list_head heatmap_buffer_list;
+
+ /*
+ * Function read_frame must be provided by the driver
+ * It should return true on successful heatmap read
+ * and false on failure
+ */
+ bool (*read_frame)(struct v4l2_heatmap *v4l2, strength_t *data);
+};
+
+int heatmap_probe(struct v4l2_heatmap *v4l2);
+void heatmap_remove(struct v4l2_heatmap *v4l2);
+/**
+ * Read the heatmap and populate an available buffer with data.
+ * The timestamp provided to this function will be used as the frame time.
+ * Designed to be called from interrupt context.
+ * This function should be called from the driver. Internally, it will call
+ * read_frame(..) provided by the driver to read the actual data.
+ */
+void heatmap_read(struct v4l2_heatmap *v4l2, uint64_t timestamp);
\ No newline at end of file
diff --git a/include/linux/input/qpnp-power-on.h b/include/linux/input/qpnp-power-on.h
index 5944f0f..ce3f0c7 100644
--- a/include/linux/input/qpnp-power-on.h
+++ b/include/linux/input/qpnp-power-on.h
@@ -59,9 +59,13 @@ enum pon_restart_reason {
PON_RESTART_REASON_DMVERITY_CORRUPTED = 0x04,
PON_RESTART_REASON_DMVERITY_ENFORCE = 0x05,
PON_RESTART_REASON_KEYS_CLEAR = 0x06,
+ PON_RESTART_REASON_PANIC = 0x07,
+ PON_RESTART_REASON_WDT = 0x08,
+ PON_RESTART_REASON_REBOOT_LONGKEY = 0x09,
/* 32 ~ 63 for OEMs/ODMs secific features */
PON_RESTART_REASON_OEM_MIN = 0x20,
+ PON_RESTART_REASON_OEM_PACKOUT = 0x21,
PON_RESTART_REASON_OEM_MAX = 0x3f,
};
diff --git a/include/linux/input/sec_cmd.h b/include/linux/input/sec_cmd.h
new file mode 100644
index 0000000..3887329
--- /dev/null
+++ b/include/linux/input/sec_cmd.h
@@ -0,0 +1,86 @@
+
+#ifndef _SEC_CMD_H_
+#define _SEC_CMD_H_
+
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+#include <linux/device.h>
+#include <linux/workqueue.h>
+#include <linux/stat.h>
+#include <linux/err.h>
+#include <linux/input.h>
+#ifdef CONFIG_SEC_SYSFS
+#include <linux/sec_sysfs.h>
+#endif
+
+#ifndef CONFIG_SEC_FACTORY
+#include <linux/kfifo.h>
+#endif
+
+#ifndef CONFIG_SEC_SYSFS
+extern struct class *sec_class;
+#endif
+
+#define SEC_CLASS_DEVT_TSP 10
+#define SEC_CLASS_DEVT_TKEY 11
+#define SEC_CLASS_DEVT_WACOM 12
+
+#define SEC_CLASS_DEV_NAME_TSP "tsp"
+#define SEC_CLASS_DEV_NAME_TKEY "sec_touchkey"
+#define SEC_CLASS_DEV_NAME_WACOM "sec_epen"
+
+#define SEC_CMD(name, func) .cmd_name = name, .cmd_func = func
+
+#define SEC_CMD_BUF_SIZE (4096 - 1)
+#define SEC_CMD_STR_LEN 256
+#define SEC_CMD_RESULT_STR_LEN (4096 - 1)
+#define SEC_CMD_PARAM_NUM 8
+
+struct sec_cmd {
+ struct list_head list;
+ const char *cmd_name;
+ void (*cmd_func)(void *device_data);
+};
+
+enum SEC_CMD_STATUS {
+ SEC_CMD_STATUS_WAITING = 0,
+ SEC_CMD_STATUS_RUNNING, // = 1
+ SEC_CMD_STATUS_OK, // = 2
+ SEC_CMD_STATUS_FAIL, // = 3
+ SEC_CMD_STATUS_NOT_APPLICABLE, // = 4
+};
+
+#ifdef USE_SEC_CMD_QUEUE
+#define SEC_CMD_MAX_QUEUE 10
+
+struct command {
+ char cmd[SEC_CMD_STR_LEN];
+};
+#endif
+
+struct sec_cmd_data {
+ struct device *fac_dev;
+ struct list_head cmd_list_head;
+ u8 cmd_state;
+ char cmd[SEC_CMD_STR_LEN];
+ int cmd_param[SEC_CMD_PARAM_NUM];
+ char cmd_result[SEC_CMD_RESULT_STR_LEN];
+ int cmd_buffer_size;
+ atomic_t cmd_is_running;
+#ifdef USE_SEC_CMD_QUEUE
+ struct kfifo cmd_queue;
+ struct mutex fifo_lock;
+#endif
+};
+
+extern void sec_cmd_set_cmd_exit(struct sec_cmd_data *data);
+extern void sec_cmd_set_default_result(struct sec_cmd_data *data);
+extern void sec_cmd_set_cmd_result(struct sec_cmd_data *data, char *buff, int len);
+extern int sec_cmd_init(struct sec_cmd_data *data,
+ struct sec_cmd *cmds, int len, int devt);
+extern void sec_cmd_exit(struct sec_cmd_data *data, int devt);
+
+#endif /* _SEC_CMD_H_ */
+
+
diff --git a/include/linux/integrity.h b/include/linux/integrity.h
index c2d6082..858d3f4 100644
--- a/include/linux/integrity.h
+++ b/include/linux/integrity.h
@@ -14,6 +14,7 @@
enum integrity_status {
INTEGRITY_PASS = 0,
+ INTEGRITY_PASS_IMMUTABLE,
INTEGRITY_FAIL,
INTEGRITY_NOLABEL,
INTEGRITY_NOXATTRS,
diff --git a/include/linux/intel-iommu.h b/include/linux/intel-iommu.h
index 23e129e..e353f66 100644
--- a/include/linux/intel-iommu.h
+++ b/include/linux/intel-iommu.h
@@ -125,6 +125,7 @@ static inline void dmar_writeq(void __iomem *addr, u64 val)
* Extended Capability Register
*/
+#define ecap_dit(e) ((e >> 41) & 0x1)
#define ecap_pasid(e) ((e >> 40) & 0x1)
#define ecap_pss(e) ((e >> 35) & 0x1f)
#define ecap_eafs(e) ((e >> 34) & 0x1)
@@ -294,6 +295,7 @@ enum {
#define QI_DEV_IOTLB_SID(sid) ((u64)((sid) & 0xffff) << 32)
#define QI_DEV_IOTLB_QDEP(qdep) (((qdep) & 0x1f) << 16)
#define QI_DEV_IOTLB_ADDR(addr) ((u64)(addr) & VTD_PAGE_MASK)
+#define QI_DEV_IOTLB_PFSID(pfsid) (((u64)(pfsid & 0xf) << 12) | ((u64)(pfsid & 0xfff) << 52))
#define QI_DEV_IOTLB_SIZE 1
#define QI_DEV_IOTLB_MAX_INVS 32
@@ -318,6 +320,7 @@ enum {
#define QI_DEV_EIOTLB_PASID(p) (((u64)p) << 32)
#define QI_DEV_EIOTLB_SID(sid) ((u64)((sid) & 0xffff) << 16)
#define QI_DEV_EIOTLB_QDEP(qd) ((u64)((qd) & 0x1f) << 4)
+#define QI_DEV_EIOTLB_PFSID(pfsid) (((u64)(pfsid & 0xf) << 12) | ((u64)(pfsid & 0xfff) << 52))
#define QI_DEV_EIOTLB_MAX_INVS 32
#define QI_PGRP_IDX(idx) (((u64)(idx)) << 55)
@@ -463,9 +466,8 @@ extern void qi_flush_context(struct intel_iommu *iommu, u16 did, u16 sid,
u8 fm, u64 type);
extern void qi_flush_iotlb(struct intel_iommu *iommu, u16 did, u64 addr,
unsigned int size_order, u64 type);
-extern void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 qdep,
- u64 addr, unsigned mask);
-
+extern void qi_flush_dev_iotlb(struct intel_iommu *iommu, u16 sid, u16 pfsid,
+ u16 qdep, u64 addr, unsigned mask);
extern int qi_submit_sync(struct qi_desc *desc, struct intel_iommu *iommu);
extern int dmar_ir_support(void);
diff --git a/include/linux/ipc_logging.h b/include/linux/ipc_logging.h
index 5bc8b6a..5bf61af 100644
--- a/include/linux/ipc_logging.h
+++ b/include/linux/ipc_logging.h
@@ -13,6 +13,7 @@
#ifndef _IPC_LOGGING_H
#define _IPC_LOGGING_H
+#include <linux/errno.h>
#include <linux/types.h>
#define MAX_MSG_SIZE 255
diff --git a/include/linux/irq.h b/include/linux/irq.h
index 39e3254..3df7028 100644
--- a/include/linux/irq.h
+++ b/include/linux/irq.h
@@ -503,16 +503,16 @@ static inline int irq_set_parent(int irq, int parent_irq)
* Built-in IRQ handlers for various IRQ types,
* callable via desc->handle_irq()
*/
-extern void handle_level_irq(struct irq_desc *desc);
-extern void handle_fasteoi_irq(struct irq_desc *desc);
-extern void handle_edge_irq(struct irq_desc *desc);
-extern void handle_edge_eoi_irq(struct irq_desc *desc);
-extern void handle_simple_irq(struct irq_desc *desc);
-extern void handle_untracked_irq(struct irq_desc *desc);
-extern void handle_percpu_irq(struct irq_desc *desc);
-extern void handle_percpu_devid_irq(struct irq_desc *desc);
-extern void handle_bad_irq(struct irq_desc *desc);
-extern void handle_nested_irq(unsigned int irq);
+extern bool handle_level_irq(struct irq_desc *desc);
+extern bool handle_fasteoi_irq(struct irq_desc *desc);
+extern bool handle_edge_irq(struct irq_desc *desc);
+extern bool handle_edge_eoi_irq(struct irq_desc *desc);
+extern bool handle_simple_irq(struct irq_desc *desc);
+extern bool handle_untracked_irq(struct irq_desc *desc);
+extern bool handle_percpu_irq(struct irq_desc *desc);
+extern bool handle_percpu_devid_irq(struct irq_desc *desc);
+extern bool handle_bad_irq(struct irq_desc *desc);
+extern bool handle_nested_irq(unsigned int irq);
extern int irq_chip_compose_msi_msg(struct irq_data *data, struct msi_msg *msg);
extern int irq_chip_pm_get(struct irq_data *data);
diff --git a/include/linux/irqdesc.h b/include/linux/irqdesc.h
index c9be579..abad323 100644
--- a/include/linux/irqdesc.h
+++ b/include/linux/irqdesc.h
@@ -145,9 +145,9 @@ static inline struct msi_desc *irq_desc_get_msi_desc(struct irq_desc *desc)
* Architectures call this to let the generic IRQ layer
* handle an interrupt.
*/
-static inline void generic_handle_irq_desc(struct irq_desc *desc)
+static inline bool generic_handle_irq_desc(struct irq_desc *desc)
{
- desc->handle_irq(desc);
+ return desc->handle_irq(desc);
}
int generic_handle_irq(unsigned int irq);
diff --git a/include/linux/irqhandler.h b/include/linux/irqhandler.h
index 661bed0..b31ab4b 100644
--- a/include/linux/irqhandler.h
+++ b/include/linux/irqhandler.h
@@ -8,7 +8,7 @@
struct irq_desc;
struct irq_data;
-typedef void (*irq_flow_handler_t)(struct irq_desc *desc);
+typedef bool (*irq_flow_handler_t)(struct irq_desc *desc);
typedef void (*irq_preflow_handler_t)(struct irq_data *data);
#endif
diff --git a/include/linux/jiffies.h b/include/linux/jiffies.h
index c2a0f00..734377a 100644
--- a/include/linux/jiffies.h
+++ b/include/linux/jiffies.h
@@ -292,6 +292,8 @@ static inline u64 jiffies_to_nsecs(const unsigned long j)
return (u64)jiffies_to_usecs(j) * NSEC_PER_USEC;
}
+extern u64 jiffies64_to_nsecs(u64 j);
+
extern unsigned long __msecs_to_jiffies(const unsigned int m);
#if HZ <= MSEC_PER_SEC && !(MSEC_PER_SEC % HZ)
/*
diff --git a/include/linux/kcore.h b/include/linux/kcore.h
index d927622..3ffade4 100644
--- a/include/linux/kcore.h
+++ b/include/linux/kcore.h
@@ -9,6 +9,7 @@ enum kcore_type {
KCORE_VMALLOC,
KCORE_RAM,
KCORE_VMEMMAP,
+ KCORE_USER,
KCORE_OTHER,
};
diff --git a/include/linux/kernel.h b/include/linux/kernel.h
index 61054f1..e823d71 100644
--- a/include/linux/kernel.h
+++ b/include/linux/kernel.h
@@ -11,6 +11,7 @@
#include <linux/log2.h>
#include <linux/typecheck.h>
#include <linux/printk.h>
+#include <linux/build_bug.h>
#include <asm/byteorder.h>
#include <uapi/linux/kernel.h>
@@ -833,9 +834,12 @@ static inline void ftrace_dump(enum ftrace_dump_mode oops_dump_mode) { }
* @member: the name of the member within the struct.
*
*/
-#define container_of(ptr, type, member) ({ \
- const typeof( ((type *)0)->member ) *__mptr = (ptr); \
- (type *)( (char *)__mptr - offsetof(type,member) );})
+#define container_of(ptr, type, member) ({ \
+ void *__mptr = (void *)(ptr); \
+ BUILD_BUG_ON_MSG(!__same_type(*(ptr), ((type *)0)->member) && \
+ !__same_type(*(ptr), void), \
+ "pointer type mismatch in container_of()"); \
+ ((type *)(__mptr - offsetof(type, member))); })
/* Rebuild everything on CONFIG_FTRACE_MCOUNT_RECORD */
#ifdef CONFIG_FTRACE_MCOUNT_RECORD
diff --git a/include/linux/keydebug-func.h b/include/linux/keydebug-func.h
new file mode 100644
index 0000000..e84b844
--- /dev/null
+++ b/include/linux/keydebug-func.h
@@ -0,0 +1,27 @@
+/*
+ * include/linux/keydebug-func.h - function and structure for debug
+ * functions used by keydebug driver
+ *
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#ifndef _LINUX_KEYDEBUG_FUNC_H
+#define _LINUX_KEYDEBUG_FUNC_H
+
+void kernel_top_monitor(void);
+void kernel_top_init(void);
+void kernel_top_exit(void);
+
+void keydebug_showallcpus(void);
+
+#endif /* _LINUX_KEYDEBUG_FUNC_H */
diff --git a/include/linux/keydebug.h b/include/linux/keydebug.h
new file mode 100644
index 0000000..2a3ff95
--- /dev/null
+++ b/include/linux/keydebug.h
@@ -0,0 +1,31 @@
+/*
+ * include/linux/keydebug.h - platform data structure for keydebug driver
+ *
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#ifndef _LINUX_KEYDEBUG_H
+#define _LINUX_KEYDEBUG_H
+
+#define KEYDEBUG_NAME "keydebug"
+
+struct keydebug_platform_data {
+ uint32_t key_down_delay;
+ uint32_t dbg_fn_delay;
+ uint32_t *keys_down; /* 0 terminated */
+ struct platform_device *pdev_child;
+ struct delayed_work delayed_work;
+ bool keydebug_requested;
+};
+
+#endif /* _LINUX_KEYDEBUG_H */
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 8c58db2..eb55374 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1070,7 +1070,6 @@ static inline void kvm_irq_routing_update(struct kvm *kvm)
{
}
#endif
-void kvm_arch_irq_routing_update(struct kvm *kvm);
static inline int kvm_ioeventfd(struct kvm *kvm, struct kvm_ioeventfd *args)
{
@@ -1079,6 +1078,8 @@ static inline int kvm_ioeventfd(struct kvm *kvm, struct kvm_ioeventfd *args)
#endif /* CONFIG_HAVE_KVM_EVENTFD */
+void kvm_arch_irq_routing_update(struct kvm *kvm);
+
static inline void kvm_make_request(int req, struct kvm_vcpu *vcpu)
{
/*
diff --git a/include/linux/leds.h b/include/linux/leds.h
index 66a1f4e..2a83ab8 100644
--- a/include/linux/leds.h
+++ b/include/linux/leds.h
@@ -53,6 +53,7 @@ struct led_classdev {
#define LED_HW_PLUGGABLE (1 << 25)
#define LED_PANIC_INDICATOR (1 << 26)
#define LED_KEEP_TRIGGER (1 << 27)
+#define LED_BRIGHTNESS_FAST (1 << 28)
/* Set LED brightness level
* Must not sleep. Use brightness_set_blocking for drivers
diff --git a/include/linux/libata.h b/include/linux/libata.h
index 616eef4..df58b01 100644
--- a/include/linux/libata.h
+++ b/include/linux/libata.h
@@ -208,6 +208,7 @@ enum {
ATA_FLAG_SLAVE_POSS = (1 << 0), /* host supports slave dev */
/* (doesn't imply presence) */
ATA_FLAG_SATA = (1 << 1),
+ ATA_FLAG_NO_LPM = (1 << 2), /* host not happy with LPM */
ATA_FLAG_NO_LOG_PAGE = (1 << 5), /* do not issue log page read */
ATA_FLAG_NO_ATAPI = (1 << 6), /* No ATAPI support */
ATA_FLAG_PIO_DMA = (1 << 7), /* PIO cmds via DMA */
diff --git a/include/linux/lz4.h b/include/linux/lz4.h
index 6b784c5..1b0f8ca 100644
--- a/include/linux/lz4.h
+++ b/include/linux/lz4.h
@@ -1,87 +1,717 @@
-#ifndef __LZ4_H__
-#define __LZ4_H__
-/*
- * LZ4 Kernel Interface
+/* LZ4 Kernel Interface
*
* Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
+ * Copyright (C) 2016, Sven Schmidt <4sschmid@informatik.uni-hamburg.de>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
+ *
+ * This file is based on the original header file
+ * for LZ4 - Fast LZ compression algorithm.
+ *
+ * LZ4 - Fast LZ compression algorithm
+ * Copyright (C) 2011-2016, Yann Collet.
+ * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following disclaimer
+ * in the documentation and/or other materials provided with the
+ * distribution.
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ * You can contact the author at :
+ * - LZ4 homepage : http://www.lz4.org
+ * - LZ4 source repository : https://github.com/lz4/lz4
*/
-#define LZ4_MEM_COMPRESS (16384)
-#define LZ4HC_MEM_COMPRESS (262144 + (2 * sizeof(unsigned char *)))
+
+#ifndef __LZ4_H__
+#define __LZ4_H__
+
+#include <linux/types.h>
+#include <linux/string.h> /* memset, memcpy */
+
+/*-************************************************************************
+ * CONSTANTS
+ **************************************************************************/
+/*
+ * LZ4_MEMORY_USAGE :
+ * Memory usage formula : N->2^N Bytes
+ * (examples : 10 -> 1KB; 12 -> 4KB ; 16 -> 64KB; 20 -> 1MB; etc.)
+ * Increasing memory usage improves compression ratio
+ * Reduced memory usage can improve speed, due to cache effect
+ * Default value is 14, for 16KB, which nicely fits into Intel x86 L1 cache
+ */
+#define LZ4_MEMORY_USAGE 14
+
+#define LZ4_MAX_INPUT_SIZE 0x7E000000 /* 2 113 929 216 bytes */
+#define LZ4_COMPRESSBOUND(isize) (\
+ (unsigned int)(isize) > (unsigned int)LZ4_MAX_INPUT_SIZE \
+ ? 0 \
+ : (isize) + ((isize)/255) + 16)
+
+#define LZ4_ACCELERATION_DEFAULT 1
+#define LZ4_HASHLOG (LZ4_MEMORY_USAGE-2)
+#define LZ4_HASHTABLESIZE (1 << LZ4_MEMORY_USAGE)
+#define LZ4_HASH_SIZE_U32 (1 << LZ4_HASHLOG)
+
+#define LZ4HC_MIN_CLEVEL 3
+#define LZ4HC_DEFAULT_CLEVEL 9
+#define LZ4HC_MAX_CLEVEL 16
+
+#define LZ4HC_DICTIONARY_LOGSIZE 16
+#define LZ4HC_MAXD (1<<LZ4HC_DICTIONARY_LOGSIZE)
+#define LZ4HC_MAXD_MASK (LZ4HC_MAXD - 1)
+#define LZ4HC_HASH_LOG (LZ4HC_DICTIONARY_LOGSIZE - 1)
+#define LZ4HC_HASHTABLESIZE (1 << LZ4HC_HASH_LOG)
+#define LZ4HC_HASH_MASK (LZ4HC_HASHTABLESIZE - 1)
+
+/*-************************************************************************
+ * STREAMING CONSTANTS AND STRUCTURES
+ **************************************************************************/
+#define LZ4_STREAMSIZE_U64 ((1 << (LZ4_MEMORY_USAGE - 3)) + 4)
+#define LZ4_STREAMSIZE (LZ4_STREAMSIZE_U64 * sizeof(unsigned long long))
+
+#define LZ4_STREAMHCSIZE 262192
+#define LZ4_STREAMHCSIZE_SIZET (262192 / sizeof(size_t))
+
+#define LZ4_STREAMDECODESIZE_U64 4
+#define LZ4_STREAMDECODESIZE (LZ4_STREAMDECODESIZE_U64 * \
+ sizeof(unsigned long long))
/*
- * lz4_compressbound()
- * Provides the maximum size that LZ4 may output in a "worst case" scenario
- * (input data not compressible)
+ * LZ4_stream_t - information structure to track an LZ4 stream.
*/
-static inline size_t lz4_compressbound(size_t isize)
+typedef struct {
+ uint32_t hashTable[LZ4_HASH_SIZE_U32];
+ uint32_t currentOffset;
+ uint32_t initCheck;
+ const uint8_t *dictionary;
+ uint8_t *bufferStart;
+ uint32_t dictSize;
+} LZ4_stream_t_internal;
+typedef union {
+ unsigned long long table[LZ4_STREAMSIZE_U64];
+ LZ4_stream_t_internal internal_donotuse;
+} LZ4_stream_t;
+
+/*
+ * LZ4_streamHC_t - information structure to track an LZ4HC stream.
+ */
+typedef struct {
+ unsigned int hashTable[LZ4HC_HASHTABLESIZE];
+ unsigned short chainTable[LZ4HC_MAXD];
+ /* next block to continue on current prefix */
+ const unsigned char *end;
+ /* All index relative to this position */
+ const unsigned char *base;
+ /* alternate base for extDict */
+ const unsigned char *dictBase;
+ /* below that point, need extDict */
+ unsigned int dictLimit;
+ /* below that point, no more dict */
+ unsigned int lowLimit;
+ /* index from which to continue dict update */
+ unsigned int nextToUpdate;
+ unsigned int compressionLevel;
+} LZ4HC_CCtx_internal;
+typedef union {
+ size_t table[LZ4_STREAMHCSIZE_SIZET];
+ LZ4HC_CCtx_internal internal_donotuse;
+} LZ4_streamHC_t;
+
+/*
+ * LZ4_streamDecode_t - information structure to track an
+ * LZ4 stream during decompression.
+ *
+ * init this structure using LZ4_setStreamDecode (or memset()) before first use
+ */
+typedef struct {
+ const uint8_t *externalDict;
+ size_t extDictSize;
+ const uint8_t *prefixEnd;
+ size_t prefixSize;
+} LZ4_streamDecode_t_internal;
+typedef union {
+ unsigned long long table[LZ4_STREAMDECODESIZE_U64];
+ LZ4_streamDecode_t_internal internal_donotuse;
+} LZ4_streamDecode_t;
+
+/*-************************************************************************
+ * SIZE OF STATE
+ **************************************************************************/
+#define LZ4_MEM_COMPRESS LZ4_STREAMSIZE
+#define LZ4HC_MEM_COMPRESS LZ4_STREAMHCSIZE
+
+/*-************************************************************************
+ * Compression Functions
+ **************************************************************************/
+
+/**
+ * LZ4_compressBound() - Max. output size in worst case szenarios
+ * @isize: Size of the input data
+ *
+ * Return: Max. size LZ4 may output in a "worst case" szenario
+ * (data not compressible)
+ */
+static inline int LZ4_compressBound(size_t isize)
{
- return isize + (isize / 255) + 16;
+ return LZ4_COMPRESSBOUND(isize);
}
-/*
- * lz4_compress()
- * src : source address of the original data
- * src_len : size of the original data
- * dst : output buffer address of the compressed data
- * This requires 'dst' of size LZ4_COMPRESSBOUND.
- * dst_len : is the output size, which is returned after compress done
- * workmem : address of the working memory.
- * This requires 'workmem' of size LZ4_MEM_COMPRESS.
- * return : Success if return 0
- * Error if return (< 0)
- * note : Destination buffer and workmem must be already allocated with
- * the defined size.
+/**
+ * lz4_compressbound() - For backwards compatibility; see LZ4_compressBound
+ * @isize: Size of the input data
+ *
+ * Return: Max. size LZ4 may output in a "worst case" szenario
+ * (data not compressible)
*/
-int lz4_compress(const unsigned char *src, size_t src_len,
- unsigned char *dst, size_t *dst_len, void *wrkmem);
+static inline int lz4_compressbound(size_t isize)
+{
+ return LZ4_COMPRESSBOUND(isize);
+}
- /*
- * lz4hc_compress()
- * src : source address of the original data
- * src_len : size of the original data
- * dst : output buffer address of the compressed data
- * This requires 'dst' of size LZ4_COMPRESSBOUND.
- * dst_len : is the output size, which is returned after compress done
- * workmem : address of the working memory.
- * This requires 'workmem' of size LZ4HC_MEM_COMPRESS.
- * return : Success if return 0
- * Error if return (< 0)
- * note : Destination buffer and workmem must be already allocated with
- * the defined size.
- */
-int lz4hc_compress(const unsigned char *src, size_t src_len,
- unsigned char *dst, size_t *dst_len, void *wrkmem);
-
-/*
- * lz4_decompress()
- * src : source address of the compressed data
- * src_len : is the input size, whcih is returned after decompress done
- * dest : output buffer address of the decompressed data
- * actual_dest_len: is the size of uncompressed data, supposing it's known
- * return : Success if return 0
- * Error if return (< 0)
- * note : Destination buffer must be already allocated.
- * slightly faster than lz4_decompress_unknownoutputsize()
+/**
+ * LZ4_compress_default() - Compress data from source to dest
+ * @source: source address of the original data
+ * @dest: output buffer address of the compressed data
+ * @inputSize: size of the input data. Max supported value is LZ4_MAX_INPUT_SIZE
+ * @maxOutputSize: full or partial size of buffer 'dest'
+ * which must be already allocated
+ * @wrkmem: address of the working memory.
+ * This requires 'workmem' of LZ4_MEM_COMPRESS.
+ *
+ * Compresses 'sourceSize' bytes from buffer 'source'
+ * into already allocated 'dest' buffer of size 'maxOutputSize'.
+ * Compression is guaranteed to succeed if
+ * 'maxOutputSize' >= LZ4_compressBound(inputSize).
+ * It also runs faster, so it's a recommended setting.
+ * If the function cannot compress 'source' into a more limited 'dest' budget,
+ * compression stops *immediately*, and the function result is zero.
+ * As a consequence, 'dest' content is not valid.
+ *
+ * Return: Number of bytes written into buffer 'dest'
+ * (necessarily <= maxOutputSize) or 0 if compression fails
*/
-int lz4_decompress(const unsigned char *src, size_t *src_len,
- unsigned char *dest, size_t actual_dest_len);
+int LZ4_compress_default(const char *source, char *dest, int inputSize,
+ int maxOutputSize, void *wrkmem);
+
+/**
+ * LZ4_compress_fast() - As LZ4_compress_default providing an acceleration param
+ * @source: source address of the original data
+ * @dest: output buffer address of the compressed data
+ * @inputSize: size of the input data. Max supported value is LZ4_MAX_INPUT_SIZE
+ * @maxOutputSize: full or partial size of buffer 'dest'
+ * which must be already allocated
+ * @acceleration: acceleration factor
+ * @wrkmem: address of the working memory.
+ * This requires 'workmem' of LZ4_MEM_COMPRESS.
+ *
+ * Same as LZ4_compress_default(), but allows to select an "acceleration"
+ * factor. The larger the acceleration value, the faster the algorithm,
+ * but also the lesser the compression. It's a trade-off. It can be fine tuned,
+ * with each successive value providing roughly +~3% to speed.
+ * An acceleration value of "1" is the same as regular LZ4_compress_default()
+ * Values <= 0 will be replaced by LZ4_ACCELERATION_DEFAULT, which is 1.
+ *
+ * Return: Number of bytes written into buffer 'dest'
+ * (necessarily <= maxOutputSize) or 0 if compression fails
+ */
+int LZ4_compress_fast(const char *source, char *dest, int inputSize,
+ int maxOutputSize, int acceleration, void *wrkmem);
+
+/**
+ * LZ4_compress_destSize() - Compress as much data as possible
+ * from source to dest
+ * @source: source address of the original data
+ * @dest: output buffer address of the compressed data
+ * @sourceSizePtr: will be modified to indicate how many bytes where read
+ * from 'source' to fill 'dest'. New value is necessarily <= old value.
+ * @targetDestSize: Size of buffer 'dest' which must be already allocated
+ * @wrkmem: address of the working memory.
+ * This requires 'workmem' of LZ4_MEM_COMPRESS.
+ *
+ * Reverse the logic, by compressing as much data as possible
+ * from 'source' buffer into already allocated buffer 'dest'
+ * of size 'targetDestSize'.
+ * This function either compresses the entire 'source' content into 'dest'
+ * if it's large enough, or fill 'dest' buffer completely with as much data as
+ * possible from 'source'.
+ *
+ * Return: Number of bytes written into 'dest' (necessarily <= targetDestSize)
+ * or 0 if compression fails
+ */
+int LZ4_compress_destSize(const char *source, char *dest, int *sourceSizePtr,
+ int targetDestSize, void *wrkmem);
/*
- * lz4_decompress_unknownoutputsize()
- * src : source address of the compressed data
- * src_len : is the input size, therefore the compressed size
- * dest : output buffer address of the decompressed data
- * dest_len: is the max size of the destination buffer, which is
- * returned with actual size of decompressed data after
- * decompress done
- * return : Success if return 0
- * Error if return (< 0)
- * note : Destination buffer must be already allocated.
+ * lz4_compress() - For backward compatibility, see LZ4_compress_default
+ * @src: source address of the original data
+ * @src_len: size of the original data
+ * @dst: output buffer address of the compressed data. This requires 'dst'
+ * of size LZ4_COMPRESSBOUND
+ * @dst_len: is the output size, which is returned after compress done
+ * @workmem: address of the working memory.
+ *
+ * Return: Success if return 0, Error if return < 0
+ */
+int lz4_compress(const unsigned char *src, size_t src_len, unsigned char *dst,
+ size_t *dst_len, void *wrkmem);
+
+/*-************************************************************************
+ * Decompression Functions
+ **************************************************************************/
+
+/**
+ * LZ4_decompress_fast() - Decompresses data from 'source' into 'dest'
+ * @source: source address of the compressed data
+ * @dest: output buffer address of the uncompressed data
+ * which must be already allocated with 'originalSize' bytes
+ * @originalSize: is the original and therefore uncompressed size
+ *
+ * Decompresses data from 'source' into 'dest'.
+ * This function fully respect memory boundaries for properly formed
+ * compressed data.
+ * It is a bit faster than LZ4_decompress_safe().
+ * However, it does not provide any protection against intentionally
+ * modified data stream (malicious input).
+ * Use this function in trusted environment only
+ * (data to decode comes from a trusted source).
+ *
+ * Return: number of bytes read from the source buffer
+ * or a negative result if decompression fails.
+ */
+int LZ4_decompress_fast(const char *source, char *dest, int originalSize);
+
+/**
+ * LZ4_decompress_safe() - Decompression protected against buffer overflow
+ * @source: source address of the compressed data
+ * @dest: output buffer address of the uncompressed data
+ * which must be already allocated
+ * @compressedSize: is the precise full size of the compressed block
+ * @maxDecompressedSize: is the size of 'dest' buffer
+ *
+ * Decompresses data fom 'source' into 'dest'.
+ * If the source stream is detected malformed, the function will
+ * stop decoding and return a negative result.
+ * This function is protected against buffer overflow exploits,
+ * including malicious data packets. It never writes outside output buffer,
+ * nor reads outside input buffer.
+ *
+ * Return: number of bytes decompressed into destination buffer
+ * (necessarily <= maxDecompressedSize)
+ * or a negative result in case of error
+ */
+int LZ4_decompress_safe(const char *source, char *dest, int compressedSize,
+ int maxDecompressedSize);
+
+/**
+ * LZ4_decompress_safe_partial() - Decompress a block of size 'compressedSize'
+ * at position 'source' into buffer 'dest'
+ * @source: source address of the compressed data
+ * @dest: output buffer address of the decompressed data which must be
+ * already allocated
+ * @compressedSize: is the precise full size of the compressed block.
+ * @targetOutputSize: the decompression operation will try
+ * to stop as soon as 'targetOutputSize' has been reached
+ * @maxDecompressedSize: is the size of destination buffer
+ *
+ * This function decompresses a compressed block of size 'compressedSize'
+ * at position 'source' into destination buffer 'dest'
+ * of size 'maxDecompressedSize'.
+ * The function tries to stop decompressing operation as soon as
+ * 'targetOutputSize' has been reached, reducing decompression time.
+ * This function never writes outside of output buffer,
+ * and never reads outside of input buffer.
+ * It is therefore protected against malicious data packets.
+ *
+ * Return: the number of bytes decoded in the destination buffer
+ * (necessarily <= maxDecompressedSize)
+ * or a negative result in case of error
+ *
+ */
+int LZ4_decompress_safe_partial(const char *source, char *dest,
+ int compressedSize, int targetOutputSize, int maxDecompressedSize);
+
+/*
+ * lz4_decompress_unknownoutputsize() - For backwards compatibility,
+ * see LZ4_decompress_safe
+ * @src: source address of the compressed data
+ * @src_len: is the input size, therefore the compressed size
+ * @dest: output buffer address of the decompressed data
+ * which must be already allocated
+ * @dest_len: is the max size of the destination buffer, which is
+ * returned with actual size of decompressed data after decompress done
+ *
+ * Return: Success if return 0, Error if return (< 0)
*/
int lz4_decompress_unknownoutputsize(const unsigned char *src, size_t src_len,
- unsigned char *dest, size_t *dest_len);
+ unsigned char *dest, size_t *dest_len);
+
+/**
+ * lz4_decompress() - For backwards cocmpatibility, see LZ4_decompress_fast
+ * @src: source address of the compressed data
+ * @src_len: is the input size, which is returned after decompress done
+ * @dest: output buffer address of the decompressed data,
+ * which must be already allocated
+ * @actual_dest_len: is the size of uncompressed data, supposing it's known
+ *
+ * Return: Success if return 0, Error if return (< 0)
+ */
+int lz4_decompress(const unsigned char *src, size_t *src_len,
+ unsigned char *dest, size_t actual_dest_len);
+
+/*-************************************************************************
+ * LZ4 HC Compression
+ **************************************************************************/
+
+/**
+ * LZ4_compress_HC() - Compress data from `src` into `dst`, using HC algorithm
+ * @src: source address of the original data
+ * @dst: output buffer address of the compressed data
+ * @srcSize: size of the input data. Max supported value is LZ4_MAX_INPUT_SIZE
+ * @dstCapacity: full or partial size of buffer 'dst',
+ * which must be already allocated
+ * @compressionLevel: Recommended values are between 4 and 9, although any
+ * value between 1 and LZ4HC_MAX_CLEVEL will work.
+ * Values >LZ4HC_MAX_CLEVEL behave the same as 16.
+ * @wrkmem: address of the working memory.
+ * This requires 'wrkmem' of size LZ4HC_MEM_COMPRESS.
+ *
+ * Compress data from 'src' into 'dst', using the more powerful
+ * but slower "HC" algorithm. Compression is guaranteed to succeed if
+ * `dstCapacity >= LZ4_compressBound(srcSize)
+ *
+ * Return : the number of bytes written into 'dst' or 0 if compression fails.
+ */
+int LZ4_compress_HC(const char *src, char *dst, int srcSize, int dstCapacity,
+ int compressionLevel, void *wrkmem);
+
+/**
+ * lz4hc_compress() - For backwards compatibility, see LZ4_compress_HC
+ * @src: source address of the original data
+ * @src_len: size of the original data
+ * @dst: output buffer address of the compressed data. This requires 'dst'
+ * of size LZ4_COMPRESSBOUND.
+ * @dst_len: is the output size, which is returned after compress done
+ * @wrkmem: address of the working memory.
+ * This requires 'workmem' of size LZ4HC_MEM_COMPRESS.
+ *
+ * Return : Success if return 0, Error if return (< 0)
+ */
+int lz4hc_compress(const unsigned char *src, size_t src_len, unsigned char *dst,
+ size_t *dst_len, void *wrkmem);
+
+/**
+ * LZ4_resetStreamHC() - Init an allocated 'LZ4_streamHC_t' structure
+ * @streamHCPtr: pointer to the 'LZ4_streamHC_t' structure
+ * @compressionLevel: Recommended values are between 4 and 9, although any
+ * value between 1 and LZ4HC_MAX_CLEVEL will work.
+ * Values >LZ4HC_MAX_CLEVEL behave the same as 16.
+ *
+ * An LZ4_streamHC_t structure can be allocated once
+ * and re-used multiple times.
+ * Use this function to init an allocated `LZ4_streamHC_t` structure
+ * and start a new compression.
+ */
+void LZ4_resetStreamHC(LZ4_streamHC_t *streamHCPtr, int compressionLevel);
+
+/**
+ * LZ4_loadDictHC() - Load a static dictionary into LZ4_streamHC
+ * @streamHCPtr: pointer to the LZ4HC_stream_t
+ * @dictionary: dictionary to load
+ * @dictSize: size of dictionary
+ *
+ * Use this function to load a static dictionary into LZ4HC_stream.
+ * Any previous data will be forgotten, only 'dictionary'
+ * will remain in memory.
+ * Loading a size of 0 is allowed.
+ *
+ * Return : dictionary size, in bytes (necessarily <= 64 KB)
+ */
+int LZ4_loadDictHC(LZ4_streamHC_t *streamHCPtr, const char *dictionary,
+ int dictSize);
+
+/**
+ * LZ4_compress_HC_continue() - Compress 'src' using data from previously
+ * compressed blocks as a dictionary using the HC algorithm
+ * @streamHCPtr: Pointer to the previous 'LZ4_streamHC_t' structure
+ * @src: source address of the original data
+ * @dst: output buffer address of the compressed data,
+ * which must be already allocated
+ * @srcSize: size of the input data. Max supported value is LZ4_MAX_INPUT_SIZE
+ * @maxDstSize: full or partial size of buffer 'dest'
+ * which must be already allocated
+ *
+ * These functions compress data in successive blocks of any size, using
+ * previous blocks as dictionary. One key assumption is that previous
+ * blocks (up to 64 KB) remain read-accessible while
+ * compressing next blocks. There is an exception for ring buffers,
+ * which can be smaller than 64 KB.
+ * Ring buffers scenario is automatically detected and handled by
+ * LZ4_compress_HC_continue().
+ * Before starting compression, state must be properly initialized,
+ * using LZ4_resetStreamHC().
+ * A first "fictional block" can then be designated as
+ * initial dictionary, using LZ4_loadDictHC() (Optional).
+ * Then, use LZ4_compress_HC_continue()
+ * to compress each successive block. Previous memory blocks
+ * (including initial dictionary when present) must remain accessible
+ * and unmodified during compression.
+ * 'dst' buffer should be sized to handle worst case scenarios, using
+ * LZ4_compressBound(), to ensure operation success.
+ * If, for any reason, previous data blocks can't be preserved unmodified
+ * in memory during next compression block,
+ * you must save it to a safer memory space, using LZ4_saveDictHC().
+ * Return value of LZ4_saveDictHC() is the size of dictionary
+ * effectively saved into 'safeBuffer'.
+ *
+ * Return: Number of bytes written into buffer 'dst' or 0 if compression fails
+ */
+int LZ4_compress_HC_continue(LZ4_streamHC_t *streamHCPtr, const char *src,
+ char *dst, int srcSize, int maxDstSize);
+
+/**
+ * LZ4_saveDictHC() - Save static dictionary from LZ4HC_stream
+ * @streamHCPtr: pointer to the 'LZ4HC_stream_t' structure
+ * @safeBuffer: buffer to save dictionary to, must be already allocated
+ * @maxDictSize: size of 'safeBuffer'
+ *
+ * If previously compressed data block is not guaranteed
+ * to remain available at its memory location,
+ * save it into a safer place (char *safeBuffer).
+ * Note : you don't need to call LZ4_loadDictHC() afterwards,
+ * dictionary is immediately usable, you can therefore call
+ * LZ4_compress_HC_continue().
+ *
+ * Return : saved dictionary size in bytes (necessarily <= maxDictSize),
+ * or 0 if error.
+ */
+int LZ4_saveDictHC(LZ4_streamHC_t *streamHCPtr, char *safeBuffer,
+ int maxDictSize);
+
+/*-*********************************************
+ * Streaming Compression Functions
+ ***********************************************/
+
+/**
+ * LZ4_resetStream() - Init an allocated 'LZ4_stream_t' structure
+ * @LZ4_stream: pointer to the 'LZ4_stream_t' structure
+ *
+ * An LZ4_stream_t structure can be allocated once
+ * and re-used multiple times.
+ * Use this function to init an allocated `LZ4_stream_t` structure
+ * and start a new compression.
+ */
+void LZ4_resetStream(LZ4_stream_t *LZ4_stream);
+
+/**
+ * LZ4_loadDict() - Load a static dictionary into LZ4_stream
+ * @streamPtr: pointer to the LZ4_stream_t
+ * @dictionary: dictionary to load
+ * @dictSize: size of dictionary
+ *
+ * Use this function to load a static dictionary into LZ4_stream.
+ * Any previous data will be forgotten, only 'dictionary'
+ * will remain in memory.
+ * Loading a size of 0 is allowed.
+ *
+ * Return : dictionary size, in bytes (necessarily <= 64 KB)
+ */
+int LZ4_loadDict(LZ4_stream_t *streamPtr, const char *dictionary,
+ int dictSize);
+
+/**
+ * LZ4_saveDict() - Save static dictionary from LZ4_stream
+ * @streamPtr: pointer to the 'LZ4_stream_t' structure
+ * @safeBuffer: buffer to save dictionary to, must be already allocated
+ * @dictSize: size of 'safeBuffer'
+ *
+ * If previously compressed data block is not guaranteed
+ * to remain available at its memory location,
+ * save it into a safer place (char *safeBuffer).
+ * Note : you don't need to call LZ4_loadDict() afterwards,
+ * dictionary is immediately usable, you can therefore call
+ * LZ4_compress_fast_continue().
+ *
+ * Return : saved dictionary size in bytes (necessarily <= dictSize),
+ * or 0 if error.
+ */
+int LZ4_saveDict(LZ4_stream_t *streamPtr, char *safeBuffer, int dictSize);
+
+/**
+ * LZ4_compress_fast_continue() - Compress 'src' using data from previously
+ * compressed blocks as a dictionary
+ * @streamPtr: Pointer to the previous 'LZ4_stream_t' structure
+ * @src: source address of the original data
+ * @dst: output buffer address of the compressed data,
+ * which must be already allocated
+ * @srcSize: size of the input data. Max supported value is LZ4_MAX_INPUT_SIZE
+ * @maxDstSize: full or partial size of buffer 'dest'
+ * which must be already allocated
+ * @acceleration: acceleration factor
+ *
+ * Compress buffer content 'src', using data from previously compressed blocks
+ * as dictionary to improve compression ratio.
+ * Important : Previous data blocks are assumed to still
+ * be present and unmodified !
+ * If maxDstSize >= LZ4_compressBound(srcSize),
+ * compression is guaranteed to succeed, and runs faster.
+ *
+ * Return: Number of bytes written into buffer 'dst' or 0 if compression fails
+ */
+int LZ4_compress_fast_continue(LZ4_stream_t *streamPtr, const char *src,
+ char *dst, int srcSize, int maxDstSize, int acceleration);
+
+/**
+ * LZ4_setStreamDecode() - Instruct where to find dictionary
+ * @LZ4_streamDecode: the 'LZ4_streamDecode_t' structure
+ * @dictionary: dictionary to use
+ * @dictSize: size of dictionary
+ *
+ * Use this function to instruct where to find the dictionary.
+ * Setting a size of 0 is allowed (same effect as reset).
+ *
+ * Return: 1 if OK, 0 if error
+ */
+int LZ4_setStreamDecode(LZ4_streamDecode_t *LZ4_streamDecode,
+ const char *dictionary, int dictSize);
+
+/**
+ * LZ4_decompress_fast_continue() - Decompress blocks in streaming mode
+ * @LZ4_streamDecode: the 'LZ4_streamDecode_t' structure
+ * @source: source address of the compressed data
+ * @dest: output buffer address of the uncompressed data
+ * which must be already allocated
+ * @compressedSize: is the precise full size of the compressed block
+ * @maxDecompressedSize: is the size of 'dest' buffer
+ *
+ * These decoding function allows decompression of multiple blocks
+ * in "streaming" mode.
+ * Previously decoded blocks *must* remain available at the memory position
+ * where they were decoded (up to 64 KB)
+ * In the case of a ring buffers, decoding buffer must be either :
+ * - Exactly same size as encoding buffer, with same update rule
+ * (block boundaries at same positions) In which case,
+ * the decoding & encoding ring buffer can have any size,
+ * including very small ones ( < 64 KB).
+ * - Larger than encoding buffer, by a minimum of maxBlockSize more bytes.
+ * maxBlockSize is implementation dependent.
+ * It's the maximum size you intend to compress into a single block.
+ * In which case, encoding and decoding buffers do not need
+ * to be synchronized, and encoding ring buffer can have any size,
+ * including small ones ( < 64 KB).
+ * - _At least_ 64 KB + 8 bytes + maxBlockSize.
+ * In which case, encoding and decoding buffers do not need to be
+ * synchronized, and encoding ring buffer can have any size,
+ * including larger than decoding buffer. W
+ * Whenever these conditions are not possible, save the last 64KB of decoded
+ * data into a safe buffer, and indicate where it is saved
+ * using LZ4_setStreamDecode()
+ *
+ * Return: number of bytes decompressed into destination buffer
+ * (necessarily <= maxDecompressedSize)
+ * or a negative result in case of error
+ */
+int LZ4_decompress_safe_continue(LZ4_streamDecode_t *LZ4_streamDecode,
+ const char *source, char *dest, int compressedSize,
+ int maxDecompressedSize);
+
+/**
+ * LZ4_decompress_fast_continue() - Decompress blocks in streaming mode
+ * @LZ4_streamDecode: the 'LZ4_streamDecode_t' structure
+ * @source: source address of the compressed data
+ * @dest: output buffer address of the uncompressed data
+ * which must be already allocated with 'originalSize' bytes
+ * @originalSize: is the original and therefore uncompressed size
+ *
+ * These decoding function allows decompression of multiple blocks
+ * in "streaming" mode.
+ * Previously decoded blocks *must* remain available at the memory position
+ * where they were decoded (up to 64 KB)
+ * In the case of a ring buffers, decoding buffer must be either :
+ * - Exactly same size as encoding buffer, with same update rule
+ * (block boundaries at same positions) In which case,
+ * the decoding & encoding ring buffer can have any size,
+ * including very small ones ( < 64 KB).
+ * - Larger than encoding buffer, by a minimum of maxBlockSize more bytes.
+ * maxBlockSize is implementation dependent.
+ * It's the maximum size you intend to compress into a single block.
+ * In which case, encoding and decoding buffers do not need
+ * to be synchronized, and encoding ring buffer can have any size,
+ * including small ones ( < 64 KB).
+ * - _At least_ 64 KB + 8 bytes + maxBlockSize.
+ * In which case, encoding and decoding buffers do not need to be
+ * synchronized, and encoding ring buffer can have any size,
+ * including larger than decoding buffer. W
+ * Whenever these conditions are not possible, save the last 64KB of decoded
+ * data into a safe buffer, and indicate where it is saved
+ * using LZ4_setStreamDecode()
+ *
+ * Return: number of bytes decompressed into destination buffer
+ * (necessarily <= maxDecompressedSize)
+ * or a negative result in case of error
+ */
+int LZ4_decompress_fast_continue(LZ4_streamDecode_t *LZ4_streamDecode,
+ const char *source, char *dest, int originalSize);
+
+/**
+ * LZ4_decompress_safe_usingDict() - Same as LZ4_setStreamDecode()
+ * followed by LZ4_decompress_safe_continue()
+ * @source: source address of the compressed data
+ * @dest: output buffer address of the uncompressed data
+ * which must be already allocated
+ * @compressedSize: is the precise full size of the compressed block
+ * @maxDecompressedSize: is the size of 'dest' buffer
+ * @dictStart: pointer to the start of the dictionary in memory
+ * @dictSize: size of dictionary
+ *
+ * These decoding function works the same as
+ * a combination of LZ4_setStreamDecode() followed by
+ * LZ4_decompress_safe_continue()
+ * It is stand-alone, and don'tn eed a LZ4_streamDecode_t structure.
+ *
+ * Return: number of bytes decompressed into destination buffer
+ * (necessarily <= maxDecompressedSize)
+ * or a negative result in case of error
+ */
+int LZ4_decompress_safe_usingDict(const char *source, char *dest,
+ int compressedSize, int maxDecompressedSize, const char *dictStart,
+ int dictSize);
+
+/**
+ * LZ4_decompress_fast_usingDict() - Same as LZ4_setStreamDecode()
+ * followed by LZ4_decompress_fast_continue()
+ * @source: source address of the compressed data
+ * @dest: output buffer address of the uncompressed data
+ * which must be already allocated with 'originalSize' bytes
+ * @originalSize: is the original and therefore uncompressed size
+ * @dictStart: pointer to the start of the dictionary in memory
+ * @dictSize: size of dictionary
+ *
+ * These decoding function works the same as
+ * a combination of LZ4_setStreamDecode() followed by
+ * LZ4_decompress_safe_continue()
+ * It is stand-alone, and don'tn eed a LZ4_streamDecode_t structure.
+ *
+ * Return: number of bytes decompressed into destination buffer
+ * (necessarily <= maxDecompressedSize)
+ * or a negative result in case of error
+ */
+int LZ4_decompress_fast_usingDict(const char *source, char *dest,
+ int originalSize, const char *dictStart, int dictSize);
+
#endif
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 7751d72..859fd20 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -786,7 +786,7 @@ void mlx5_unmap_free_uar(struct mlx5_core_dev *mdev, struct mlx5_uar *uar);
void mlx5_health_cleanup(struct mlx5_core_dev *dev);
int mlx5_health_init(struct mlx5_core_dev *dev);
void mlx5_start_health_poll(struct mlx5_core_dev *dev);
-void mlx5_stop_health_poll(struct mlx5_core_dev *dev);
+void mlx5_stop_health_poll(struct mlx5_core_dev *dev, bool disable_health);
void mlx5_drain_health_wq(struct mlx5_core_dev *dev);
void mlx5_drain_health_recovery(struct mlx5_core_dev *dev);
int mlx5_buf_alloc_node(struct mlx5_core_dev *dev, int size,
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 270f032..b3a41aa 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -31,6 +31,7 @@ struct file_ra_state;
struct user_struct;
struct writeback_control;
struct bdi_writeback;
+struct super_block;
#ifndef CONFIG_NEED_MULTIPLE_NODES /* Don't use mapnrs, do it properly */
extern unsigned long max_mapnr;
@@ -1110,6 +1111,7 @@ static inline void clear_page_pfmemalloc(struct page *page)
#define VM_FAULT_HWPOISON 0x0010 /* Hit poisoned small page */
#define VM_FAULT_HWPOISON_LARGE 0x0020 /* Hit poisoned large page. Index encoded in upper bits */
#define VM_FAULT_SIGSEGV 0x0040
+#define VM_FAULT_SWAP 0x0080
#define VM_FAULT_NOPAGE 0x0100 /* ->fault installed the pte, not return page */
#define VM_FAULT_LOCKED 0x0200 /* ->fault locked the returned page */
@@ -1414,19 +1416,27 @@ static inline unsigned long get_mm_counter(struct mm_struct *mm, int member)
return (unsigned long)val;
}
+void mm_trace_rss_stat(int member, long count, long value);
+
static inline void add_mm_counter(struct mm_struct *mm, int member, long value)
{
- atomic_long_add(value, &mm->rss_stat.count[member]);
+ long count = atomic_long_add_return(value, &mm->rss_stat.count[member]);
+
+ mm_trace_rss_stat(member, count, value);
}
static inline void inc_mm_counter(struct mm_struct *mm, int member)
{
- atomic_long_inc(&mm->rss_stat.count[member]);
+ long count = atomic_long_inc_return(&mm->rss_stat.count[member]);
+
+ mm_trace_rss_stat(member, count, 1);
}
static inline void dec_mm_counter(struct mm_struct *mm, int member)
{
- atomic_long_dec(&mm->rss_stat.count[member]);
+ long count = atomic_long_dec_return(&mm->rss_stat.count[member]);
+
+ mm_trace_rss_stat(member, count, -1);
}
/* Optimized variant when page is already known not to be PageAnon */
@@ -1759,6 +1769,8 @@ static inline spinlock_t *pmd_lock(struct mm_struct *mm, pmd_t *pmd)
return ptl;
}
+extern void __init pagecache_init(void);
+
extern void free_area_init(unsigned long * zones_size);
extern void free_area_init_node(int nid, unsigned long * zones_size,
unsigned long zone_start_pfn, unsigned long *zholes_size);
@@ -2083,8 +2095,11 @@ vm_unmapped_area(struct vm_unmapped_area_info *info)
/* truncate.c */
extern void truncate_inode_pages(struct address_space *, loff_t);
+extern void truncate_inode_pages_fill_zero(struct address_space *, loff_t);
extern void truncate_inode_pages_range(struct address_space *,
loff_t lstart, loff_t lend);
+extern void truncate_inode_pages_range_fill_zero(struct address_space *,
+ loff_t lstart, loff_t lend);
extern void truncate_inode_pages_final(struct address_space *);
/* generic vm_area_ops exported for stackable file systems */
@@ -2098,7 +2113,7 @@ int write_one_page(struct page *page, int wait);
void task_dirty_inc(struct task_struct *tsk);
/* readahead.c */
-#define VM_MAX_READAHEAD 512 /* kbytes */
+#define VM_MAX_READAHEAD 128 /* kbytes */
#define VM_MIN_READAHEAD 16 /* kbytes (includes current page) */
int force_page_cache_readahead(struct address_space *mapping, struct file *filp,
@@ -2187,6 +2202,12 @@ static inline struct vm_area_struct *find_exact_vma(struct mm_struct *mm,
return vma;
}
+static inline bool range_in_vma(struct vm_area_struct *vma,
+ unsigned long start, unsigned long end)
+{
+ return (vma && vma->vm_start <= start && end <= vma->vm_end);
+}
+
#ifdef CONFIG_MMU
pgprot_t vm_get_page_prot(unsigned long vm_flags);
void vma_set_page_prot(struct vm_area_struct *vma);
@@ -2246,6 +2267,7 @@ static inline struct page *follow_page(struct vm_area_struct *vma,
#define FOLL_MLOCK 0x1000 /* lock present pages */
#define FOLL_REMOTE 0x2000 /* we are working on non-current tsk/mm */
#define FOLL_COW 0x4000 /* internal GUP flag */
+#define FOLL_ANON 0x8000 /* don't do file mappings */
typedef int (*pte_fn_t)(pte_t *pte, pgtable_t token, unsigned long addr,
void *data);
@@ -2323,6 +2345,8 @@ int drop_caches_sysctl_handler(struct ctl_table *, int,
void drop_slab(void);
void drop_slab_node(int nid);
+void drop_pagecache_sb(struct super_block *sb, void *unused);
+
#ifndef CONFIG_MMU
#define randomize_va_space 0
#else
@@ -2464,19 +2488,5 @@ void __init setup_nr_node_ids(void);
static inline void setup_nr_node_ids(void) {}
#endif
-#ifdef CONFIG_PROCESS_RECLAIM
-struct reclaim_param {
- struct vm_area_struct *vma;
- /* Number of pages scanned */
- int nr_scanned;
- /* max pages to reclaim */
- int nr_to_reclaim;
- /* pages reclaimed */
- int nr_reclaimed;
-};
-extern struct reclaim_param reclaim_task_anon(struct task_struct *task,
- int nr_to_reclaim);
-#endif
-
#endif /* __KERNEL__ */
#endif /* _LINUX_MM_H */
diff --git a/include/linux/mm_event.h b/include/linux/mm_event.h
new file mode 100644
index 0000000..52e6e24
--- /dev/null
+++ b/include/linux/mm_event.h
@@ -0,0 +1,55 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_MM_EVENT_H
+#define _LINUX_MM_EVENT_H
+
+/*
+ * These enums are exposed to userspace via the ftrace trace_pipe_raw endpoint
+ * and are an ABI. Userspace tools depend on them.
+ */
+enum mm_event_type {
+ MM_MIN_FAULT = 0,
+ MM_MAJ_FAULT = 1,
+ MM_READ_IO = 2,
+ MM_COMPACTION = 3,
+ MM_RECLAIM = 4,
+ MM_SWP_FAULT = 5,
+ MM_KERN_ALLOC = 6,
+ MM_TYPE_NUM = 7,
+};
+
+struct mm_event_task {
+ unsigned int count;
+ unsigned int max_lat;
+ u64 accm_lat;
+} __attribute__ ((packed));
+
+struct mm_event_vmstat {
+ unsigned long free;
+ unsigned long file;
+ unsigned long anon;
+ unsigned long ion;
+ unsigned long slab;
+ unsigned long ws_refault;
+ unsigned long ws_activate;
+ unsigned long mapped;
+ unsigned long pgin;
+ unsigned long pgout;
+ unsigned long swpin;
+ unsigned long swpout;
+ unsigned long reclaim_steal;
+ unsigned long reclaim_scan;
+ unsigned long compact_scan;
+};
+
+#ifdef CONFIG_MM_EVENT_STAT
+void mm_event_task_init(struct task_struct *tsk);
+void mm_event_start(ktime_t *time);
+void mm_event_end(enum mm_event_type event, ktime_t start);
+void mm_event_count(enum mm_event_type event, int count);
+#else
+static inline void mm_event_task_init(struct task_struct *tsk) {}
+static inline void mm_event_start(ktime_t *time) {}
+static inline void mm_event_end(enum mm_event_type event, ktime_t start) {}
+static inline void mm_event_count(enum mm_event_type event, int count) {}
+#endif /* _LINUX_MM_EVENT_H */
+#endif
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index 5942478..35daed7 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -403,7 +403,7 @@ struct kioctx_table;
struct mm_struct {
struct vm_area_struct *mmap; /* list of VMAs */
struct rb_root mm_rb;
- u32 vmacache_seqnum; /* per-thread vmacache */
+ u64 vmacache_seqnum; /* per-thread vmacache */
#ifdef CONFIG_MMU
unsigned long (*get_unmapped_area) (struct file *filp,
unsigned long addr, unsigned long len,
diff --git a/include/linux/mmc/sdio_ids.h b/include/linux/mmc/sdio_ids.h
index d43ef96..3e4d4f4 100644
--- a/include/linux/mmc/sdio_ids.h
+++ b/include/linux/mmc/sdio_ids.h
@@ -34,6 +34,7 @@
#define SDIO_DEVICE_ID_BROADCOM_4335_4339 0x4335
#define SDIO_DEVICE_ID_BROADCOM_4339 0x4339
#define SDIO_DEVICE_ID_BROADCOM_43362 0xa962
+#define SDIO_DEVICE_ID_BROADCOM_43364 0xa9a4
#define SDIO_DEVICE_ID_BROADCOM_43430 0xa9a6
#define SDIO_DEVICE_ID_BROADCOM_4345 0x4345
#define SDIO_DEVICE_ID_BROADCOM_4354 0x4354
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 815d0f41..45c79a6 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -133,6 +133,9 @@ enum zone_stat_item {
NR_SLAB_UNRECLAIMABLE,
NR_PAGETABLE, /* used for pagetables */
NR_KERNEL_STACK_KB, /* measured in KiB */
+#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK)
+ NR_KERNEL_SCS_BYTES, /* measured in bytes */
+#endif
NR_KAISERTABLE,
NR_BOUNCE,
/* Second 128 byte cacheline */
@@ -161,6 +164,7 @@ enum node_stat_item {
NR_ISOLATED_FILE, /* Temporary isolated pages from file lru */
WORKINGSET_REFAULT,
WORKINGSET_ACTIVATE,
+ WORKINGSET_RESTORE,
WORKINGSET_NODERECLAIM,
NR_ANON_MAPPED, /* Mapped anonymous pages */
NR_FILE_MAPPED, /* pagecache pages mapped into pagetables.
@@ -178,6 +182,23 @@ enum node_stat_item {
NR_VMSCAN_IMMEDIATE, /* Prioritise for reclaim when writeback ends */
NR_DIRTIED, /* page dirtyings since bootup */
NR_WRITTEN, /* page writings since bootup */
+ /*
+ * Currently, NR_INDIRECTLY_RECLAIMABLE_BYTES covers only memory in ion
+ * system heap pool(If user free memory allocated from system heap via
+ * pool, it could stay this ION heap pool until memory pressure happens
+ * or reallocated so that it consumes the memory) so we could regard it
+ * as the amount of ION heap system heap pool.
+ * Upstream is going to change it with new stuff so don't bother to
+ * follow it at this moment. https://lkml.org/lkml/2018/7/18/505
+ */
+ NR_INDIRECTLY_RECLAIMABLE_BYTES, /* measured in bytes */
+ /*
+ * NR_ION_HEAP covers every allocation from ION heap system heap.
+ * It means it already includes NR_INDIRECTLY_RECLAIMABLE_BYTES.
+ * Therefore, the amount of ION heap application using at this moment
+ * is NR_ION_HEAP - (NR_INDIRECTLY_RECLAIMABLE_BYTES / PAGE_SIZE).
+ */
+ NR_ION_HEAP,
NR_VM_NODE_STAT_ITEMS
};
diff --git a/include/linux/mod_devicetable.h b/include/linux/mod_devicetable.h
index 579fd0a..a408dea 100644
--- a/include/linux/mod_devicetable.h
+++ b/include/linux/mod_devicetable.h
@@ -291,7 +291,7 @@ struct pcmcia_device_id {
#define INPUT_DEVICE_ID_LED_MAX 0x0f
#define INPUT_DEVICE_ID_SND_MAX 0x07
#define INPUT_DEVICE_ID_FF_MAX 0x7f
-#define INPUT_DEVICE_ID_SW_MAX 0x20
+#define INPUT_DEVICE_ID_SW_MAX 0x1f
#define INPUT_DEVICE_ID_MATCH_BUS 1
#define INPUT_DEVICE_ID_MATCH_VENDOR 2
diff --git a/include/linux/module.h b/include/linux/module.h
index 9d6fd1d..1b57954 100644
--- a/include/linux/module.h
+++ b/include/linux/module.h
@@ -211,7 +211,7 @@ extern void cleanup_module(void);
#ifdef MODULE
/* Creates an alias so file2alias.c can find device table. */
#define MODULE_DEVICE_TABLE(type, name) \
-extern const typeof(name) __mod_##type##__##name##_device_table \
+extern typeof(name) __mod_##type##__##name##_device_table \
__attribute__ ((unused, alias(__stringify(name))))
#else /* !MODULE */
#define MODULE_DEVICE_TABLE(type, name)
diff --git a/include/linux/msi.h b/include/linux/msi.h
index 0db320b..debc8aa 100644
--- a/include/linux/msi.h
+++ b/include/linux/msi.h
@@ -108,6 +108,8 @@ struct msi_desc {
list_first_entry(dev_to_msi_list((dev)), struct msi_desc, list)
#define for_each_msi_entry(desc, dev) \
list_for_each_entry((desc), dev_to_msi_list((dev)), list)
+#define for_each_msi_entry_safe(desc, tmp, dev) \
+ list_for_each_entry_safe((desc), (tmp), dev_to_msi_list((dev)), list)
#ifdef CONFIG_PCI_MSI
#define first_pci_msi_entry(pdev) first_msi_entry(&(pdev)->dev)
diff --git a/include/linux/msm_drm_notify.h b/include/linux/msm_drm_notify.h
index 924ba852..141925d 100644
--- a/include/linux/msm_drm_notify.h
+++ b/include/linux/msm_drm_notify.h
@@ -25,6 +25,8 @@ enum {
MSM_DRM_BLANK_UNBLANK,
/* panel: power off */
MSM_DRM_BLANK_POWERDOWN,
+ /* panel: low power mode */
+ MSM_DRM_BLANK_LP,
};
enum msm_drm_display_id {
diff --git a/include/linux/msm_pcie.h b/include/linux/msm_pcie.h
index b9527d3..eba2558 100644
--- a/include/linux/msm_pcie.h
+++ b/include/linux/msm_pcie.h
@@ -21,6 +21,7 @@ enum msm_pcie_config {
MSM_PCIE_CONFIG_NO_CFG_RESTORE = 0x1,
MSM_PCIE_CONFIG_LINKDOWN = 0x2,
MSM_PCIE_CONFIG_NO_RECOVERY = 0x4,
+ MSM_PCIE_CONFIG_NO_CFG_FREE = 0x8,
};
enum msm_pcie_pm_opt {
@@ -157,6 +158,7 @@ int msm_pcie_shadow_control(struct pci_dev *dev, bool enable);
int msm_pcie_debug_info(struct pci_dev *dev, u32 option, u32 base,
u32 offset, u32 mask, u32 value);
+int msm_pcie_set_reset(u32 rc_idx, bool enable);
#else /* !CONFIG_PCI_MSM */
static inline int msm_pcie_pm_control(enum msm_pcie_pm_opt pm_opt, u32 busnr,
void *user, void *data, u32 options)
diff --git a/include/linux/mtd/flashchip.h b/include/linux/mtd/flashchip.h
index b63fa45..3529683 100644
--- a/include/linux/mtd/flashchip.h
+++ b/include/linux/mtd/flashchip.h
@@ -85,6 +85,7 @@ struct flchip {
unsigned int write_suspended:1;
unsigned int erase_suspended:1;
unsigned long in_progress_block_addr;
+ unsigned long in_progress_block_mask;
struct mutex mutex;
wait_queue_head_t wq; /* Wait on here when we're waiting for the chip
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index e1b845a..153ed93 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -2335,6 +2335,13 @@ struct netdev_notifier_info {
struct net_device *dev;
};
+struct netdev_notifier_info_ext {
+ struct netdev_notifier_info info; /* must be first */
+ union {
+ u32 mtu;
+ } ext;
+};
+
struct netdev_notifier_change_info {
struct netdev_notifier_info info; /* must be first */
unsigned int flags_changed;
diff --git a/include/linux/netfilter/ipset/ip_set_comment.h b/include/linux/netfilter/ipset/ip_set_comment.h
index 8d02485..9f34204 100644
--- a/include/linux/netfilter/ipset/ip_set_comment.h
+++ b/include/linux/netfilter/ipset/ip_set_comment.h
@@ -41,11 +41,11 @@ ip_set_init_comment(struct ip_set_comment *comment,
rcu_assign_pointer(comment->c, c);
}
-/* Used only when dumping a set, protected by rcu_read_lock_bh() */
+/* Used only when dumping a set, protected by rcu_read_lock() */
static inline int
ip_set_put_comment(struct sk_buff *skb, struct ip_set_comment *comment)
{
- struct ip_set_comment_rcu *c = rcu_dereference_bh(comment->c);
+ struct ip_set_comment_rcu *c = rcu_dereference(comment->c);
if (!c)
return 0;
diff --git a/include/linux/netfilter/ipset/ip_set_timeout.h b/include/linux/netfilter/ipset/ip_set_timeout.h
index 1d6a935c..8793f5a 100644
--- a/include/linux/netfilter/ipset/ip_set_timeout.h
+++ b/include/linux/netfilter/ipset/ip_set_timeout.h
@@ -65,8 +65,14 @@ ip_set_timeout_set(unsigned long *timeout, u32 value)
static inline u32
ip_set_timeout_get(unsigned long *timeout)
{
- return *timeout == IPSET_ELEM_PERMANENT ? 0 :
- jiffies_to_msecs(*timeout - jiffies)/MSEC_PER_SEC;
+ u32 t;
+
+ if (*timeout == IPSET_ELEM_PERMANENT)
+ return 0;
+
+ t = jiffies_to_msecs(*timeout - jiffies)/MSEC_PER_SEC;
+ /* Zero value in userspace means no timeout */
+ return t == 0 ? 1 : t;
}
#endif /* __KERNEL__ */
diff --git a/include/linux/netfilter_bridge/ebtables.h b/include/linux/netfilter_bridge/ebtables.h
index 984b211..ea8a977 100644
--- a/include/linux/netfilter_bridge/ebtables.h
+++ b/include/linux/netfilter_bridge/ebtables.h
@@ -123,4 +123,9 @@ extern unsigned int ebt_do_table(struct sk_buff *skb,
/* True if the target is not a standard target */
#define INVALID_TARGET (info->target < -NUM_STANDARD_TARGETS || info->target >= 0)
+static inline bool ebt_invalid_target(int target)
+{
+ return (target < -NUM_STANDARD_TARGETS || target >= 0);
+}
+
#endif
diff --git a/include/linux/nospec.h b/include/linux/nospec.h
index e791ebc..0c5ef54 100644
--- a/include/linux/nospec.h
+++ b/include/linux/nospec.h
@@ -7,6 +7,8 @@
#define _LINUX_NOSPEC_H
#include <asm/barrier.h>
+struct task_struct;
+
/**
* array_index_mask_nospec() - generate a ~0 mask when index < size, 0 otherwise
* @index: array element index
@@ -55,4 +57,12 @@ static inline unsigned long array_index_mask_nospec(unsigned long index,
\
(typeof(_i)) (_i & _mask); \
})
+
+/* Speculation control prctl */
+int arch_prctl_spec_ctrl_get(struct task_struct *task, unsigned long which);
+int arch_prctl_spec_ctrl_set(struct task_struct *task, unsigned long which,
+ unsigned long ctrl);
+/* Speculation control for seccomp enforced mitigation */
+void arch_seccomp_spec_mitigate(struct task_struct *task);
+
#endif /* _LINUX_NOSPEC_H */
diff --git a/include/linux/of.h b/include/linux/of.h
index 3dcf853..67081b1 100644
--- a/include/linux/of.h
+++ b/include/linux/of.h
@@ -277,6 +277,8 @@ extern struct device_node *of_get_next_child(const struct device_node *node,
extern struct device_node *of_get_next_available_child(
const struct device_node *node, struct device_node *prev);
+extern struct device_node *of_get_compatible_child(const struct device_node *parent,
+ const char *compatible);
extern struct device_node *of_get_child_by_name(const struct device_node *node,
const char *name);
@@ -608,6 +610,12 @@ static inline bool of_have_populated_dt(void)
return false;
}
+static inline struct device_node *of_get_compatible_child(const struct device_node *parent,
+ const char *compatible)
+{
+ return NULL;
+}
+
static inline struct device_node *of_get_child_by_name(
const struct device_node *node,
const char *name)
diff --git a/include/linux/overflow.h b/include/linux/overflow.h
new file mode 100644
index 0000000..8712ff7
--- /dev/null
+++ b/include/linux/overflow.h
@@ -0,0 +1,278 @@
+/* SPDX-License-Identifier: GPL-2.0 OR MIT */
+#ifndef __LINUX_OVERFLOW_H
+#define __LINUX_OVERFLOW_H
+
+#include <linux/compiler.h>
+
+/*
+ * In the fallback code below, we need to compute the minimum and
+ * maximum values representable in a given type. These macros may also
+ * be useful elsewhere, so we provide them outside the
+ * COMPILER_HAS_GENERIC_BUILTIN_OVERFLOW block.
+ *
+ * It would seem more obvious to do something like
+ *
+ * #define type_min(T) (T)(is_signed_type(T) ? (T)1 << (8*sizeof(T)-1) : 0)
+ * #define type_max(T) (T)(is_signed_type(T) ? ((T)1 << (8*sizeof(T)-1)) - 1 : ~(T)0)
+ *
+ * Unfortunately, the middle expressions, strictly speaking, have
+ * undefined behaviour, and at least some versions of gcc warn about
+ * the type_max expression (but not if -fsanitize=undefined is in
+ * effect; in that case, the warning is deferred to runtime...).
+ *
+ * The slightly excessive casting in type_min is to make sure the
+ * macros also produce sensible values for the exotic type _Bool. [The
+ * overflow checkers only almost work for _Bool, but that's
+ * a-feature-not-a-bug, since people shouldn't be doing arithmetic on
+ * _Bools. Besides, the gcc builtins don't allow _Bool* as third
+ * argument.]
+ *
+ * Idea stolen from
+ * https://mail-index.netbsd.org/tech-misc/2007/02/05/0000.html -
+ * credit to Christian Biere.
+ */
+#define is_signed_type(type) (((type)(-1)) < (type)1)
+#define __type_half_max(type) ((type)1 << (8*sizeof(type) - 1 - is_signed_type(type)))
+#define type_max(T) ((T)((__type_half_max(T) - 1) + __type_half_max(T)))
+#define type_min(T) ((T)((T)-type_max(T)-(T)1))
+
+
+#ifdef COMPILER_HAS_GENERIC_BUILTIN_OVERFLOW
+/*
+ * For simplicity and code hygiene, the fallback code below insists on
+ * a, b and *d having the same type (similar to the min() and max()
+ * macros), whereas gcc's type-generic overflow checkers accept
+ * different types. Hence we don't just make check_add_overflow an
+ * alias for __builtin_add_overflow, but add type checks similar to
+ * below.
+ */
+#define check_add_overflow(a, b, d) ({ \
+ typeof(a) __a = (a); \
+ typeof(b) __b = (b); \
+ typeof(d) __d = (d); \
+ (void) (&__a == &__b); \
+ (void) (&__a == __d); \
+ __builtin_add_overflow(__a, __b, __d); \
+})
+
+#define check_sub_overflow(a, b, d) ({ \
+ typeof(a) __a = (a); \
+ typeof(b) __b = (b); \
+ typeof(d) __d = (d); \
+ (void) (&__a == &__b); \
+ (void) (&__a == __d); \
+ __builtin_sub_overflow(__a, __b, __d); \
+})
+
+#define check_mul_overflow(a, b, d) ({ \
+ typeof(a) __a = (a); \
+ typeof(b) __b = (b); \
+ typeof(d) __d = (d); \
+ (void) (&__a == &__b); \
+ (void) (&__a == __d); \
+ __builtin_mul_overflow(__a, __b, __d); \
+})
+
+#else
+
+
+/* Checking for unsigned overflow is relatively easy without causing UB. */
+#define __unsigned_add_overflow(a, b, d) ({ \
+ typeof(a) __a = (a); \
+ typeof(b) __b = (b); \
+ typeof(d) __d = (d); \
+ (void) (&__a == &__b); \
+ (void) (&__a == __d); \
+ *__d = __a + __b; \
+ *__d < __a; \
+})
+#define __unsigned_sub_overflow(a, b, d) ({ \
+ typeof(a) __a = (a); \
+ typeof(b) __b = (b); \
+ typeof(d) __d = (d); \
+ (void) (&__a == &__b); \
+ (void) (&__a == __d); \
+ *__d = __a - __b; \
+ __a < __b; \
+})
+/*
+ * If one of a or b is a compile-time constant, this avoids a division.
+ */
+#define __unsigned_mul_overflow(a, b, d) ({ \
+ typeof(a) __a = (a); \
+ typeof(b) __b = (b); \
+ typeof(d) __d = (d); \
+ (void) (&__a == &__b); \
+ (void) (&__a == __d); \
+ *__d = __a * __b; \
+ __builtin_constant_p(__b) ? \
+ __b > 0 && __a > type_max(typeof(__a)) / __b : \
+ __a > 0 && __b > type_max(typeof(__b)) / __a; \
+})
+
+/*
+ * For signed types, detecting overflow is much harder, especially if
+ * we want to avoid UB. But the interface of these macros is such that
+ * we must provide a result in *d, and in fact we must produce the
+ * result promised by gcc's builtins, which is simply the possibly
+ * wrapped-around value. Fortunately, we can just formally do the
+ * operations in the widest relevant unsigned type (u64) and then
+ * truncate the result - gcc is smart enough to generate the same code
+ * with and without the (u64) casts.
+ */
+
+/*
+ * Adding two signed integers can overflow only if they have the same
+ * sign, and overflow has happened iff the result has the opposite
+ * sign.
+ */
+#define __signed_add_overflow(a, b, d) ({ \
+ typeof(a) __a = (a); \
+ typeof(b) __b = (b); \
+ typeof(d) __d = (d); \
+ (void) (&__a == &__b); \
+ (void) (&__a == __d); \
+ *__d = (u64)__a + (u64)__b; \
+ (((~(__a ^ __b)) & (*__d ^ __a)) \
+ & type_min(typeof(__a))) != 0; \
+})
+
+/*
+ * Subtraction is similar, except that overflow can now happen only
+ * when the signs are opposite. In this case, overflow has happened if
+ * the result has the opposite sign of a.
+ */
+#define __signed_sub_overflow(a, b, d) ({ \
+ typeof(a) __a = (a); \
+ typeof(b) __b = (b); \
+ typeof(d) __d = (d); \
+ (void) (&__a == &__b); \
+ (void) (&__a == __d); \
+ *__d = (u64)__a - (u64)__b; \
+ ((((__a ^ __b)) & (*__d ^ __a)) \
+ & type_min(typeof(__a))) != 0; \
+})
+
+/*
+ * Signed multiplication is rather hard. gcc always follows C99, so
+ * division is truncated towards 0. This means that we can write the
+ * overflow check like this:
+ *
+ * (a > 0 && (b > MAX/a || b < MIN/a)) ||
+ * (a < -1 && (b > MIN/a || b < MAX/a) ||
+ * (a == -1 && b == MIN)
+ *
+ * The redundant casts of -1 are to silence an annoying -Wtype-limits
+ * (included in -Wextra) warning: When the type is u8 or u16, the
+ * __b_c_e in check_mul_overflow obviously selects
+ * __unsigned_mul_overflow, but unfortunately gcc still parses this
+ * code and warns about the limited range of __b.
+ */
+
+#define __signed_mul_overflow(a, b, d) ({ \
+ typeof(a) __a = (a); \
+ typeof(b) __b = (b); \
+ typeof(d) __d = (d); \
+ typeof(a) __tmax = type_max(typeof(a)); \
+ typeof(a) __tmin = type_min(typeof(a)); \
+ (void) (&__a == &__b); \
+ (void) (&__a == __d); \
+ *__d = (u64)__a * (u64)__b; \
+ (__b > 0 && (__a > __tmax/__b || __a < __tmin/__b)) || \
+ (__b < (typeof(__b))-1 && (__a > __tmin/__b || __a < __tmax/__b)) || \
+ (__b == (typeof(__b))-1 && __a == __tmin); \
+})
+
+
+#define check_add_overflow(a, b, d) \
+ __builtin_choose_expr(is_signed_type(typeof(a)), \
+ __signed_add_overflow(a, b, d), \
+ __unsigned_add_overflow(a, b, d))
+
+#define check_sub_overflow(a, b, d) \
+ __builtin_choose_expr(is_signed_type(typeof(a)), \
+ __signed_sub_overflow(a, b, d), \
+ __unsigned_sub_overflow(a, b, d))
+
+#define check_mul_overflow(a, b, d) \
+ __builtin_choose_expr(is_signed_type(typeof(a)), \
+ __signed_mul_overflow(a, b, d), \
+ __unsigned_mul_overflow(a, b, d))
+
+
+#endif /* COMPILER_HAS_GENERIC_BUILTIN_OVERFLOW */
+
+/**
+ * array_size() - Calculate size of 2-dimensional array.
+ *
+ * @a: dimension one
+ * @b: dimension two
+ *
+ * Calculates size of 2-dimensional array: @a * @b.
+ *
+ * Returns: number of bytes needed to represent the array or SIZE_MAX on
+ * overflow.
+ */
+static inline __must_check size_t array_size(size_t a, size_t b)
+{
+ size_t bytes;
+
+ if (check_mul_overflow(a, b, &bytes))
+ return SIZE_MAX;
+
+ return bytes;
+}
+
+/**
+ * array3_size() - Calculate size of 3-dimensional array.
+ *
+ * @a: dimension one
+ * @b: dimension two
+ * @c: dimension three
+ *
+ * Calculates size of 3-dimensional array: @a * @b * @c.
+ *
+ * Returns: number of bytes needed to represent the array or SIZE_MAX on
+ * overflow.
+ */
+static inline __must_check size_t array3_size(size_t a, size_t b, size_t c)
+{
+ size_t bytes;
+
+ if (check_mul_overflow(a, b, &bytes))
+ return SIZE_MAX;
+ if (check_mul_overflow(bytes, c, &bytes))
+ return SIZE_MAX;
+
+ return bytes;
+}
+
+static inline __must_check size_t __ab_c_size(size_t n, size_t size, size_t c)
+{
+ size_t bytes;
+
+ if (check_mul_overflow(n, size, &bytes))
+ return SIZE_MAX;
+ if (check_add_overflow(bytes, c, &bytes))
+ return SIZE_MAX;
+
+ return bytes;
+}
+
+/**
+ * struct_size() - Calculate size of structure with trailing array.
+ * @p: Pointer to the structure.
+ * @member: Name of the array member.
+ * @n: Number of elements in the array.
+ *
+ * Calculates size of memory needed for structure @p followed by an
+ * array of @n @member elements.
+ *
+ * Return: number of bytes needed or SIZE_MAX on overflow.
+ */
+#define struct_size(p, member, n) \
+ __ab_c_size(n, \
+ sizeof(*(p)->member) + __must_be_array((p)->member),\
+ sizeof(*(p)))
+
+#endif /* __LINUX_OVERFLOW_H */
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index 74e4dda..4077d60 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -73,12 +73,14 @@
*/
enum pageflags {
PG_locked, /* Page is locked. Don't touch. */
- PG_error,
+ PG_waiters, /* Page has waiters, check its waitqueue */
PG_referenced,
PG_uptodate,
PG_dirty,
PG_lru,
PG_active,
+ PG_workingset,
+ PG_error,
PG_slab,
PG_owner_priv_1, /* Owner use. If pagecache, fs may use*/
PG_arch_1,
@@ -167,6 +169,9 @@ static __always_inline int PageCompound(struct page *page)
* for compound page all operations related to the page flag applied to
* head page.
*
+ * PF_ONLY_HEAD:
+ * for compound page, callers only ever operate on the head page.
+ *
* PF_NO_TAIL:
* modifications of the page flag must be done on small or head pages,
* checks can be done on tail pages too.
@@ -176,6 +181,9 @@ static __always_inline int PageCompound(struct page *page)
*/
#define PF_ANY(page, enforce) page
#define PF_HEAD(page, enforce) compound_head(page)
+#define PF_ONLY_HEAD(page, enforce) ({ \
+ VM_BUG_ON_PGFLAGS(PageTail(page), page); \
+ page;})
#define PF_NO_TAIL(page, enforce) ({ \
VM_BUG_ON_PGFLAGS(enforce && PageTail(page), page); \
compound_head(page);})
@@ -253,6 +261,7 @@ static inline int TestClearPage##uname(struct page *page) { return 0; }
TESTSETFLAG_FALSE(uname) TESTCLEARFLAG_FALSE(uname)
__PAGEFLAG(Locked, locked, PF_NO_TAIL)
+PAGEFLAG(Waiters, waiters, PF_ONLY_HEAD) __CLEARPAGEFLAG(Waiters, waiters, PF_ONLY_HEAD)
PAGEFLAG(Error, error, PF_NO_COMPOUND) TESTCLEARFLAG(Error, error, PF_NO_COMPOUND)
PAGEFLAG(Referenced, referenced, PF_HEAD)
TESTCLEARFLAG(Referenced, referenced, PF_HEAD)
@@ -262,6 +271,8 @@ PAGEFLAG(Dirty, dirty, PF_HEAD) TESTSCFLAG(Dirty, dirty, PF_HEAD)
PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD)
PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD)
TESTCLEARFLAG(Active, active, PF_HEAD)
+PAGEFLAG(Workingset, workingset, PF_HEAD)
+ TESTCLEARFLAG(Workingset, workingset, PF_HEAD)
__PAGEFLAG(Slab, slab, PF_NO_TAIL)
__PAGEFLAG(SlobFree, slob_free, PF_NO_TAIL)
PAGEFLAG(Checked, checked, PF_NO_COMPOUND) /* Used by some filesystems */
@@ -735,6 +746,7 @@ static inline int page_has_private(struct page *page)
#undef PF_ANY
#undef PF_HEAD
+#undef PF_ONLY_HEAD
#undef PF_NO_TAIL
#undef PF_NO_COMPOUND
#endif /* !__GENERATING_BOUNDS_H */
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index 791c547..dfa6182 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -238,6 +238,7 @@ pgoff_t page_cache_prev_hole(struct address_space *mapping,
#define FGP_WRITE 0x00000008
#define FGP_NOFS 0x00000010
#define FGP_NOWAIT 0x00000020
+#define FGP_FOR_MMAP 0x00000040
struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
int fgp_flags, gfp_t cache_gfp_mask);
@@ -341,8 +342,16 @@ unsigned find_get_pages(struct address_space *mapping, pgoff_t start,
unsigned int nr_pages, struct page **pages);
unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t start,
unsigned int nr_pages, struct page **pages);
-unsigned find_get_pages_tag(struct address_space *mapping, pgoff_t *index,
- int tag, unsigned int nr_pages, struct page **pages);
+unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index,
+ pgoff_t end, int tag, unsigned int nr_pages,
+ struct page **pages);
+static inline unsigned find_get_pages_tag(struct address_space *mapping,
+ pgoff_t *index, int tag, unsigned int nr_pages,
+ struct page **pages)
+{
+ return find_get_pages_range_tag(mapping, index, (pgoff_t)-1, tag,
+ nr_pages, pages);
+}
unsigned find_get_entries_tag(struct address_space *mapping, pgoff_t start,
int tag, unsigned int nr_entries,
struct page **entries, pgoff_t *indices);
@@ -486,22 +495,14 @@ static inline int lock_page_or_retry(struct page *page, struct mm_struct *mm,
* and for filesystems which need to wait on PG_private.
*/
extern void wait_on_page_bit(struct page *page, int bit_nr);
-
extern int wait_on_page_bit_killable(struct page *page, int bit_nr);
-extern int wait_on_page_bit_killable_timeout(struct page *page,
- int bit_nr, unsigned long timeout);
+extern void wake_up_page_bit(struct page *page, int bit_nr);
-static inline int wait_on_page_locked_killable(struct page *page)
-{
- if (!PageLocked(page))
- return 0;
- return wait_on_page_bit_killable(compound_head(page), PG_locked);
-}
-
-extern wait_queue_head_t *page_waitqueue(struct page *page);
static inline void wake_up_page(struct page *page, int bit)
{
- __wake_up_bit(page_waitqueue(page), &page->flags, bit);
+ if (!PageWaiters(page))
+ return;
+ wake_up_page_bit(page, bit);
}
/*
@@ -517,6 +518,13 @@ static inline void wait_on_page_locked(struct page *page)
wait_on_page_bit(compound_head(page), PG_locked);
}
+static inline int wait_on_page_locked_killable(struct page *page)
+{
+ if (!PageLocked(page))
+ return 0;
+ return wait_on_page_bit_killable(compound_head(page), PG_locked);
+}
+
/*
* Wait for a page to complete writeback
*/
diff --git a/include/linux/pagevec.h b/include/linux/pagevec.h
index b45d391..cead441 100644
--- a/include/linux/pagevec.h
+++ b/include/linux/pagevec.h
@@ -29,9 +29,17 @@ unsigned pagevec_lookup_entries(struct pagevec *pvec,
void pagevec_remove_exceptionals(struct pagevec *pvec);
unsigned pagevec_lookup(struct pagevec *pvec, struct address_space *mapping,
pgoff_t start, unsigned nr_pages);
-unsigned pagevec_lookup_tag(struct pagevec *pvec,
- struct address_space *mapping, pgoff_t *index, int tag,
- unsigned nr_pages);
+unsigned pagevec_lookup_range_tag(struct pagevec *pvec,
+ struct address_space *mapping, pgoff_t *index, pgoff_t end,
+ int tag);
+unsigned pagevec_lookup_range_nr_tag(struct pagevec *pvec,
+ struct address_space *mapping, pgoff_t *index, pgoff_t end,
+ int tag, unsigned max_pages);
+static inline unsigned pagevec_lookup_tag(struct pagevec *pvec,
+ struct address_space *mapping, pgoff_t *index, int tag)
+{
+ return pagevec_lookup_range_tag(pvec, mapping, index, (pgoff_t)-1, tag);
+}
static inline void pagevec_init(struct pagevec *pvec, int cold)
{
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 3652290..534cb43 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -1190,6 +1190,8 @@ int pci_register_io_range(phys_addr_t addr, resource_size_t size);
unsigned long pci_address_to_pio(phys_addr_t addr);
phys_addr_t pci_pio_to_address(unsigned long pio);
int pci_remap_iospace(const struct resource *res, phys_addr_t phys_addr);
+int devm_pci_remap_iospace(struct device *dev, const struct resource *res,
+ phys_addr_t phys_addr);
void pci_unmap_iospace(struct resource *res);
static inline pci_bus_addr_t pci_bus_address(struct pci_dev *pdev, int bar)
@@ -2149,4 +2151,16 @@ static inline bool pci_ari_enabled(struct pci_bus *bus)
/* provide the legacy pci_dma_* API */
#include <linux/pci-dma-compat.h>
+#define pci_printk(level, pdev, fmt, arg...) \
+ dev_printk(level, &(pdev)->dev, fmt, ##arg)
+
+#define pci_emerg(pdev, fmt, arg...) dev_emerg(&(pdev)->dev, fmt, ##arg)
+#define pci_alert(pdev, fmt, arg...) dev_alert(&(pdev)->dev, fmt, ##arg)
+#define pci_crit(pdev, fmt, arg...) dev_crit(&(pdev)->dev, fmt, ##arg)
+#define pci_err(pdev, fmt, arg...) dev_err(&(pdev)->dev, fmt, ##arg)
+#define pci_warn(pdev, fmt, arg...) dev_warn(&(pdev)->dev, fmt, ##arg)
+#define pci_notice(pdev, fmt, arg...) dev_notice(&(pdev)->dev, fmt, ##arg)
+#define pci_info(pdev, fmt, arg...) dev_info(&(pdev)->dev, fmt, ##arg)
+#define pci_dbg(pdev, fmt, arg...) dev_dbg(&(pdev)->dev, fmt, ##arg)
+
#endif /* LINUX_PCI_H */
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index 4308204..bba5604 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -3054,4 +3054,6 @@
#define PCI_VENDOR_ID_OCZ 0x1b85
+#define PCI_VENDOR_ID_NCUBE 0x10ff
+
#endif /* _LINUX_PCI_IDS_H */
diff --git a/include/linux/pfk.h b/include/linux/pfk.h
index d7405ea..de215c9 100644
--- a/include/linux/pfk.h
+++ b/include/linux/pfk.h
@@ -17,8 +17,6 @@
struct ice_crypto_setting;
-#ifdef CONFIG_PFK
-
/*
* Default key for inline encryption.
*
@@ -32,13 +30,13 @@ struct blk_encryption_key {
u8 raw[BLK_ENCRYPTION_KEY_SIZE_AES_256_XTS];
};
+#ifdef CONFIG_PFK
int pfk_load_key_start(const struct bio *bio,
struct ice_crypto_setting *ice_setting, bool *is_pfe, bool);
int pfk_load_key_end(const struct bio *bio, bool *is_pfe);
int pfk_remove_key(const unsigned char *key, size_t key_size);
bool pfk_allow_merge_bio(const struct bio *bio1, const struct bio *bio2);
void pfk_clear_on_reset(void);
-
#else
static inline int pfk_load_key_start(const struct bio *bio,
struct ice_crypto_setting *ice_setting, bool *is_pfe, bool async)
diff --git a/include/linux/pfn_t.h b/include/linux/pfn_t.h
index a3d90b9..4078745 100644
--- a/include/linux/pfn_t.h
+++ b/include/linux/pfn_t.h
@@ -9,7 +9,7 @@
* PFN_DEV - pfn is not covered by system memmap by default
* PFN_MAP - pfn has a dynamic page mapping established by a device driver
*/
-#define PFN_FLAGS_MASK (((u64) ~PAGE_MASK) << (BITS_PER_LONG_LONG - PAGE_SHIFT))
+#define PFN_FLAGS_MASK (((u64) (~PAGE_MASK)) << (BITS_PER_LONG_LONG - PAGE_SHIFT))
#define PFN_SG_CHAIN (1ULL << (BITS_PER_LONG_LONG - 1))
#define PFN_SG_LAST (1ULL << (BITS_PER_LONG_LONG - 2))
#define PFN_DEV (1ULL << (BITS_PER_LONG_LONG - 3))
diff --git a/include/linux/pinctrl/pinctrl.h b/include/linux/pinctrl/pinctrl.h
index a42e57d..a800832 100644
--- a/include/linux/pinctrl/pinctrl.h
+++ b/include/linux/pinctrl/pinctrl.h
@@ -34,15 +34,18 @@ struct device_node;
* @number: unique pin number from the global pin number space
* @name: a name for this pin
* @drv_data: driver-defined per-pin data. pinctrl core does not touch this
+ * @no_read: the registers of this pin can't be read by userspace process
*/
struct pinctrl_pin_desc {
unsigned number;
const char *name;
void *drv_data;
+ bool no_read;
};
/* Convenience macro to define a single named or anonymous pin descriptor */
#define PINCTRL_PIN(a, b) { .number = a, .name = b }
+#define PINCTRL_PIN_NR(a, b) { .number = a, .name = b, .no_read = true }
#define PINCTRL_PIN_ANON(a) { .number = a }
/**
diff --git a/include/linux/platform_data/cs40l2x.h b/include/linux/platform_data/cs40l2x.h
new file mode 100644
index 0000000..881a379
--- /dev/null
+++ b/include/linux/platform_data/cs40l2x.h
@@ -0,0 +1,32 @@
+/*
+ * linux/platform_data/cs40l2x.h -- Platform data for
+ * CS40L20/CS40L25/CS40L25A/CS40L25B
+ *
+ * Copyright 2018 Cirrus Logic, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef __CS40L2X_H
+#define __CS40L2X_H
+
+struct cs40l2x_platform_data {
+ unsigned int boost_ind;
+ unsigned int boost_cap;
+ unsigned int boost_ipk;
+ bool refclk_gpio2;
+ unsigned int f0_default;
+ unsigned int f0_min;
+ unsigned int f0_max;
+ unsigned int redc_default;
+ unsigned int redc_min;
+ unsigned int redc_max;
+ unsigned int gpio1_rise_index;
+ unsigned int gpio1_fall_index;
+ unsigned int gpio1_fall_timeout;
+ unsigned int gpio1_mode;
+};
+
+#endif /* __CS40L2X_H */
diff --git a/include/linux/platform_data/ina2xx.h b/include/linux/platform_data/ina2xx.h
index 9abc0ca..9f0aa1b 100644
--- a/include/linux/platform_data/ina2xx.h
+++ b/include/linux/platform_data/ina2xx.h
@@ -1,7 +1,7 @@
/*
* Driver for Texas Instruments INA219, INA226 power monitor chips
*
- * Copyright (C) 2012 Lothar Felten <l-felten@ti.com>
+ * Copyright (C) 2012 Lothar Felten <lothar.felten@gmail.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
diff --git a/include/linux/posix-timers.h b/include/linux/posix-timers.h
index 62d44c1..e4b8678 100644
--- a/include/linux/posix-timers.h
+++ b/include/linux/posix-timers.h
@@ -65,8 +65,8 @@ struct k_itimer {
spinlock_t it_lock;
clockid_t it_clock; /* which timer type */
timer_t it_id; /* timer id */
- int it_overrun; /* overrun on pending signal */
- int it_overrun_last; /* overrun on last delivered signal */
+ s64 it_overrun; /* overrun on pending signal */
+ s64 it_overrun_last; /* overrun on last delivered signal */
int it_requeue_pending; /* waiting to requeue this timer */
#define REQUEUE_PENDING 1
int it_sigev_notify; /* notify word of sigevent struct */
diff --git a/include/linux/power_supply.h b/include/linux/power_supply.h
index 78c2a9f..9d45bfe 100644
--- a/include/linux/power_supply.h
+++ b/include/linux/power_supply.h
@@ -125,6 +125,12 @@ enum {
POWER_SUPPLY_PL_NON_STACKED_BATFET,
};
+enum {
+ POWER_SUPPLY_TAPER_CONTROL_OFF = 0,
+ POWER_SUPPLY_TAPER_CONTROL_MODE_STEPPER = 1,
+ POWER_SUPPLY_TAPER_CONTROL_MODE_IMMEDIATE = 2,
+};
+
enum power_supply_property {
/* Properties of type `int' */
POWER_SUPPLY_PROP_STATUS = 0,
@@ -267,6 +273,7 @@ enum power_supply_property {
POWER_SUPPLY_PROP_CC_STEP,
POWER_SUPPLY_PROP_CC_STEP_SEL,
POWER_SUPPLY_PROP_SW_JEITA_ENABLED,
+ POWER_SUPPLY_PROP_TAPER_CONTROL,
POWER_SUPPLY_PROP_PD_VOLTAGE_MAX,
POWER_SUPPLY_PROP_PD_VOLTAGE_MIN,
POWER_SUPPLY_PROP_SDP_CURRENT_MAX,
@@ -286,6 +293,9 @@ enum power_supply_property {
POWER_SUPPLY_PROP_BATTERY_INFO,
POWER_SUPPLY_PROP_BATTERY_INFO_ID,
POWER_SUPPLY_PROP_ENABLE_JEITA_DETECTION,
+ POWER_SUPPLY_PROP_OTG_FASTROLESWAP,
+ POWER_SUPPLY_PROP_CHARGE_DISABLE,
+ POWER_SUPPLY_PROP_PD_IN_EXPLICIT_CONTRACT,
/* Local extensions of type int64_t */
POWER_SUPPLY_PROP_CHARGE_COUNTER_EXT,
/* Properties of type `const char *' */
@@ -342,6 +352,7 @@ enum power_supply_typec_power_role {
POWER_SUPPLY_TYPEC_PR_DUAL,
POWER_SUPPLY_TYPEC_PR_SINK,
POWER_SUPPLY_TYPEC_PR_SOURCE,
+ POWER_SUPPLY_TYPEC_PR_SOURCE_1_5,
};
enum power_supply_notifier_events {
@@ -426,6 +437,7 @@ struct power_supply {
spinlock_t changed_lock;
bool changed;
bool initialized;
+ bool removing;
atomic_t use_cnt;
#ifdef CONFIG_THERMAL
struct thermal_zone_device *tzd;
diff --git a/include/linux/property.h b/include/linux/property.h
index 338f9b7..459337f 100644
--- a/include/linux/property.h
+++ b/include/linux/property.h
@@ -187,7 +187,7 @@ struct property_entry {
*/
#define PROPERTY_ENTRY_INTEGER_ARRAY(_name_, _type_, _val_) \
-{ \
+(struct property_entry) { \
.name = _name_, \
.length = ARRAY_SIZE(_val_) * sizeof(_type_), \
.is_array = true, \
@@ -205,7 +205,7 @@ struct property_entry {
PROPERTY_ENTRY_INTEGER_ARRAY(_name_, u64, _val_)
#define PROPERTY_ENTRY_STRING_ARRAY(_name_, _val_) \
-{ \
+(struct property_entry) { \
.name = _name_, \
.length = ARRAY_SIZE(_val_) * sizeof(const char *), \
.is_array = true, \
@@ -214,7 +214,7 @@ struct property_entry {
}
#define PROPERTY_ENTRY_INTEGER(_name_, _type_, _val_) \
-{ \
+(struct property_entry) { \
.name = _name_, \
.length = sizeof(_type_), \
.is_string = false, \
@@ -231,7 +231,7 @@ struct property_entry {
PROPERTY_ENTRY_INTEGER(_name_, u64, _val_)
#define PROPERTY_ENTRY_STRING(_name_, _val_) \
-{ \
+(struct property_entry) { \
.name = _name_, \
.length = sizeof(_val_), \
.is_string = true, \
@@ -239,7 +239,7 @@ struct property_entry {
}
#define PROPERTY_ENTRY_BOOL(_name_) \
-{ \
+(struct property_entry) { \
.name = _name_, \
}
diff --git a/include/linux/psi.h b/include/linux/psi.h
new file mode 100644
index 0000000..8e0725a
--- /dev/null
+++ b/include/linux/psi.h
@@ -0,0 +1,53 @@
+#ifndef _LINUX_PSI_H
+#define _LINUX_PSI_H
+
+#include <linux/psi_types.h>
+#include <linux/sched.h>
+
+struct seq_file;
+struct css_set;
+
+#ifdef CONFIG_PSI
+
+extern bool psi_disabled;
+
+void psi_init(void);
+
+void psi_task_change(struct task_struct *task, int clear, int set);
+
+void psi_memstall_tick(struct task_struct *task, int cpu);
+void psi_memstall_enter(unsigned long *flags);
+void psi_memstall_leave(unsigned long *flags);
+
+int psi_show(struct seq_file *s, struct psi_group *group, enum psi_res res);
+
+#ifdef CONFIG_CGROUPS
+int psi_cgroup_alloc(struct cgroup *cgrp);
+void psi_cgroup_free(struct cgroup *cgrp);
+void cgroup_move_task(struct task_struct *p, struct css_set *to);
+#endif
+
+#else /* CONFIG_PSI */
+
+static inline void psi_init(void) {}
+
+static inline void psi_memstall_enter(unsigned long *flags) {}
+static inline void psi_memstall_leave(unsigned long *flags) {}
+
+#ifdef CONFIG_CGROUPS
+static inline int psi_cgroup_alloc(struct cgroup *cgrp)
+{
+ return 0;
+}
+static inline void psi_cgroup_free(struct cgroup *cgrp)
+{
+}
+static inline void cgroup_move_task(struct task_struct *p, struct css_set *to)
+{
+ rcu_assign_pointer(p->cgroups, to);
+}
+#endif
+
+#endif /* CONFIG_PSI */
+
+#endif /* _LINUX_PSI_H */
diff --git a/include/linux/psi_types.h b/include/linux/psi_types.h
new file mode 100644
index 0000000..2cf422d
--- /dev/null
+++ b/include/linux/psi_types.h
@@ -0,0 +1,92 @@
+#ifndef _LINUX_PSI_TYPES_H
+#define _LINUX_PSI_TYPES_H
+
+#include <linux/seqlock.h>
+#include <linux/types.h>
+
+#ifdef CONFIG_PSI
+
+/* Tracked task states */
+enum psi_task_count {
+ NR_IOWAIT,
+ NR_MEMSTALL,
+ NR_RUNNING,
+ NR_PSI_TASK_COUNTS,
+};
+
+/* Task state bitmasks */
+#define TSK_IOWAIT (1 << NR_IOWAIT)
+#define TSK_MEMSTALL (1 << NR_MEMSTALL)
+#define TSK_RUNNING (1 << NR_RUNNING)
+
+/* Resources that workloads could be stalled on */
+enum psi_res {
+ PSI_IO,
+ PSI_MEM,
+ PSI_CPU,
+ NR_PSI_RESOURCES,
+};
+
+/*
+ * Pressure states for each resource:
+ *
+ * SOME: Stalled tasks & working tasks
+ * FULL: Stalled tasks & no working tasks
+ */
+enum psi_states {
+ PSI_IO_SOME,
+ PSI_IO_FULL,
+ PSI_MEM_SOME,
+ PSI_MEM_FULL,
+ PSI_CPU_SOME,
+ /* Only per-CPU, to weigh the CPU in the global average: */
+ PSI_NONIDLE,
+ NR_PSI_STATES,
+};
+
+struct psi_group_cpu {
+ /* 1st cacheline updated by the scheduler */
+
+ /* Aggregator needs to know of concurrent changes */
+ seqcount_t seq ____cacheline_aligned_in_smp;
+
+ /* States of the tasks belonging to this group */
+ unsigned int tasks[NR_PSI_TASK_COUNTS];
+
+ /* Period time sampling buckets for each state of interest (ns) */
+ u32 times[NR_PSI_STATES];
+
+ /* Time of last task change in this group (rq_clock) */
+ u64 state_start;
+
+ /* 2nd cacheline updated by the aggregator */
+
+ /* Delta detection against the sampling buckets */
+ u32 times_prev[NR_PSI_STATES] ____cacheline_aligned_in_smp;
+};
+
+struct psi_group {
+ /* Protects data updated during an aggregation */
+ struct mutex stat_lock;
+
+ /* Per-cpu task state & time tracking */
+ struct psi_group_cpu __percpu *pcpu;
+
+ /* Periodic aggregation state */
+ u64 total_prev[NR_PSI_STATES - 1];
+ u64 last_update;
+ u64 next_update;
+ struct delayed_work clock_work;
+
+ /* Total stall times and sampled pressure averages */
+ u64 total[NR_PSI_STATES - 1];
+ unsigned long avg[NR_PSI_STATES - 1][3];
+};
+
+#else /* CONFIG_PSI */
+
+struct psi_group { };
+
+#endif /* CONFIG_PSI */
+
+#endif /* _LINUX_PSI_TYPES_H */
diff --git a/include/linux/pstore_ram.h b/include/linux/pstore_ram.h
index cb5edd6..3051bb5 100644
--- a/include/linux/pstore_ram.h
+++ b/include/linux/pstore_ram.h
@@ -44,8 +44,16 @@ struct persistent_ram_ecc_info {
struct persistent_ram_zone {
phys_addr_t paddr;
size_t size;
+ int memtype;
void *vaddr;
struct persistent_ram_buffer *buffer;
+
+ /* Alternate buffer */
+ phys_addr_t alt_paddr;
+ void *alt_vaddr;
+ struct persistent_ram_buffer *alt_buffer;
+
+ /* Common buffer size */
size_t buffer_size;
u32 flags;
raw_spinlock_t buffer_lock;
@@ -60,20 +68,22 @@ struct persistent_ram_zone {
char *old_log;
size_t old_log_size;
+ size_t old_log_alloc_size;
};
-struct persistent_ram_zone *persistent_ram_new(phys_addr_t start, size_t size,
+struct persistent_ram_zone *persistent_ram_new(phys_addr_t start,
+ phys_addr_t alt_start, size_t size,
u32 sig, struct persistent_ram_ecc_info *ecc_info,
unsigned int memtype, u32 flags);
void persistent_ram_free(struct persistent_ram_zone *prz);
-void persistent_ram_zap(struct persistent_ram_zone *prz);
+void persistent_ram_zap(struct persistent_ram_zone *prz, bool use_alt);
int persistent_ram_write(struct persistent_ram_zone *prz, const void *s,
unsigned int count);
int persistent_ram_write_user(struct persistent_ram_zone *prz,
const void __user *s, unsigned int count);
-void persistent_ram_save_old(struct persistent_ram_zone *prz);
+void persistent_ram_save_old(struct persistent_ram_zone *prz, bool use_alt);
size_t persistent_ram_old_size(struct persistent_ram_zone *prz);
void *persistent_ram_old(struct persistent_ram_zone *prz);
void persistent_ram_free_old(struct persistent_ram_zone *prz);
@@ -92,6 +102,7 @@ struct ramoops_platform_data {
unsigned long mem_size;
phys_addr_t mem_address;
unsigned int mem_type;
+ unsigned long alt_mem_address;
unsigned long record_size;
unsigned long console_size;
unsigned long ftrace_size;
diff --git a/include/linux/ptr_ring.h b/include/linux/ptr_ring.h
index 05c6d20..597b84d 100644
--- a/include/linux/ptr_ring.h
+++ b/include/linux/ptr_ring.h
@@ -351,7 +351,7 @@ static inline void *ptr_ring_consume_bh(struct ptr_ring *r)
static inline void **__ptr_ring_init_queue_alloc(unsigned int size, gfp_t gfp)
{
- if (size * sizeof(void *) > KMALLOC_MAX_SIZE)
+ if (size > KMALLOC_MAX_SIZE / sizeof(void *))
return NULL;
return kcalloc(size, sizeof(void *), gfp);
}
@@ -384,6 +384,8 @@ static inline void **__ptr_ring_swap_queue(struct ptr_ring *r, void **queue,
else if (destroy)
destroy(ptr);
+ if (producer >= size)
+ producer = 0;
r->size = size;
r->producer = producer;
r->consumer = 0;
diff --git a/include/linux/reset.h b/include/linux/reset.h
index 5daff15..7e99690 100644
--- a/include/linux/reset.h
+++ b/include/linux/reset.h
@@ -13,76 +13,82 @@ int reset_control_deassert(struct reset_control *rstc);
int reset_control_status(struct reset_control *rstc);
struct reset_control *__of_reset_control_get(struct device_node *node,
- const char *id, int index, int shared);
+ const char *id, int index, bool shared,
+ bool optional);
+struct reset_control *__reset_control_get(struct device *dev, const char *id,
+ int index, bool shared,
+ bool optional);
void reset_control_put(struct reset_control *rstc);
+int __device_reset(struct device *dev, bool optional);
struct reset_control *__devm_reset_control_get(struct device *dev,
- const char *id, int index, int shared);
-
-int __must_check device_reset(struct device *dev);
-
-static inline int device_reset_optional(struct device *dev)
-{
- return device_reset(dev);
-}
+ const char *id, int index, bool shared,
+ bool optional);
#else
static inline int reset_control_reset(struct reset_control *rstc)
{
- WARN_ON(1);
return 0;
}
static inline int reset_control_assert(struct reset_control *rstc)
{
- WARN_ON(1);
return 0;
}
static inline int reset_control_deassert(struct reset_control *rstc)
{
- WARN_ON(1);
return 0;
}
static inline int reset_control_status(struct reset_control *rstc)
{
- WARN_ON(1);
return 0;
}
static inline void reset_control_put(struct reset_control *rstc)
{
- WARN_ON(1);
}
-static inline int __must_check device_reset(struct device *dev)
+static inline int __device_reset(struct device *dev, bool optional)
{
- WARN_ON(1);
- return -ENOTSUPP;
-}
-
-static inline int device_reset_optional(struct device *dev)
-{
- return -ENOTSUPP;
+ return optional ? 0 : -ENOTSUPP;
}
static inline struct reset_control *__of_reset_control_get(
struct device_node *node,
- const char *id, int index, int shared)
+ const char *id, int index, bool shared,
+ bool optional)
{
- return ERR_PTR(-ENOTSUPP);
+ return optional ? NULL : ERR_PTR(-ENOTSUPP);
+}
+
+static inline struct reset_control *__reset_control_get(
+ struct device *dev, const char *id,
+ int index, bool shared, bool optional)
+{
+ return optional ? NULL : ERR_PTR(-ENOTSUPP);
}
static inline struct reset_control *__devm_reset_control_get(
- struct device *dev,
- const char *id, int index, int shared)
+ struct device *dev, const char *id,
+ int index, bool shared, bool optional)
{
- return ERR_PTR(-ENOTSUPP);
+ return optional ? NULL : ERR_PTR(-ENOTSUPP);
}
#endif /* CONFIG_RESET_CONTROLLER */
+static inline int __must_check device_reset(struct device *dev)
+{
+ return __device_reset(dev, false);
+}
+
+static inline int device_reset_optional(struct device *dev)
+{
+ return __device_reset(dev, true);
+}
+
/**
* reset_control_get_exclusive - Lookup and obtain an exclusive reference
* to a reset controller.
@@ -101,10 +107,7 @@ static inline struct reset_control *__devm_reset_control_get(
static inline struct reset_control *
__must_check reset_control_get_exclusive(struct device *dev, const char *id)
{
-#ifndef CONFIG_RESET_CONTROLLER
- WARN_ON(1);
-#endif
- return __of_reset_control_get(dev ? dev->of_node : NULL, id, 0, 0);
+ return __reset_control_get(dev, id, 0, false, false);
}
/**
@@ -132,19 +135,19 @@ __must_check reset_control_get_exclusive(struct device *dev, const char *id)
static inline struct reset_control *reset_control_get_shared(
struct device *dev, const char *id)
{
- return __of_reset_control_get(dev ? dev->of_node : NULL, id, 0, 1);
+ return __reset_control_get(dev, id, 0, true, false);
}
static inline struct reset_control *reset_control_get_optional_exclusive(
struct device *dev, const char *id)
{
- return __of_reset_control_get(dev ? dev->of_node : NULL, id, 0, 0);
+ return __reset_control_get(dev, id, 0, false, true);
}
static inline struct reset_control *reset_control_get_optional_shared(
struct device *dev, const char *id)
{
- return __of_reset_control_get(dev ? dev->of_node : NULL, id, 0, 1);
+ return __reset_control_get(dev, id, 0, true, true);
}
/**
@@ -160,7 +163,7 @@ static inline struct reset_control *reset_control_get_optional_shared(
static inline struct reset_control *of_reset_control_get_exclusive(
struct device_node *node, const char *id)
{
- return __of_reset_control_get(node, id, 0, 0);
+ return __of_reset_control_get(node, id, 0, false, false);
}
/**
@@ -185,7 +188,7 @@ static inline struct reset_control *of_reset_control_get_exclusive(
static inline struct reset_control *of_reset_control_get_shared(
struct device_node *node, const char *id)
{
- return __of_reset_control_get(node, id, 0, 1);
+ return __of_reset_control_get(node, id, 0, true, false);
}
/**
@@ -202,7 +205,7 @@ static inline struct reset_control *of_reset_control_get_shared(
static inline struct reset_control *of_reset_control_get_exclusive_by_index(
struct device_node *node, int index)
{
- return __of_reset_control_get(node, NULL, index, 0);
+ return __of_reset_control_get(node, NULL, index, false, false);
}
/**
@@ -230,7 +233,7 @@ static inline struct reset_control *of_reset_control_get_exclusive_by_index(
static inline struct reset_control *of_reset_control_get_shared_by_index(
struct device_node *node, int index)
{
- return __of_reset_control_get(node, NULL, index, 1);
+ return __of_reset_control_get(node, NULL, index, true, false);
}
/**
@@ -249,10 +252,7 @@ static inline struct reset_control *
__must_check devm_reset_control_get_exclusive(struct device *dev,
const char *id)
{
-#ifndef CONFIG_RESET_CONTROLLER
- WARN_ON(1);
-#endif
- return __devm_reset_control_get(dev, id, 0, 0);
+ return __devm_reset_control_get(dev, id, 0, false, false);
}
/**
@@ -267,19 +267,19 @@ __must_check devm_reset_control_get_exclusive(struct device *dev,
static inline struct reset_control *devm_reset_control_get_shared(
struct device *dev, const char *id)
{
- return __devm_reset_control_get(dev, id, 0, 1);
+ return __devm_reset_control_get(dev, id, 0, true, false);
}
static inline struct reset_control *devm_reset_control_get_optional_exclusive(
struct device *dev, const char *id)
{
- return __devm_reset_control_get(dev, id, 0, 0);
+ return __devm_reset_control_get(dev, id, 0, false, true);
}
static inline struct reset_control *devm_reset_control_get_optional_shared(
struct device *dev, const char *id)
{
- return __devm_reset_control_get(dev, id, 0, 1);
+ return __devm_reset_control_get(dev, id, 0, true, true);
}
/**
@@ -297,7 +297,7 @@ static inline struct reset_control *devm_reset_control_get_optional_shared(
static inline struct reset_control *
devm_reset_control_get_exclusive_by_index(struct device *dev, int index)
{
- return __devm_reset_control_get(dev, NULL, index, 0);
+ return __devm_reset_control_get(dev, NULL, index, false, false);
}
/**
@@ -313,7 +313,7 @@ devm_reset_control_get_exclusive_by_index(struct device *dev, int index)
static inline struct reset_control *
devm_reset_control_get_shared_by_index(struct device *dev, int index)
{
- return __devm_reset_control_get(dev, NULL, index, 1);
+ return __devm_reset_control_get(dev, NULL, index, true, false);
}
/*
diff --git a/include/linux/rhashtable.h b/include/linux/rhashtable.h
index 85d1ffc..4421e5c 100644
--- a/include/linux/rhashtable.h
+++ b/include/linux/rhashtable.h
@@ -138,7 +138,6 @@ struct rhashtable_params {
/**
* struct rhashtable - Hash table handle
* @tbl: Bucket table
- * @nelems: Number of elements in table
* @key_len: Key length for hashfn
* @elasticity: Maximum chain length before rehash
* @p: Configuration parameters
@@ -146,10 +145,10 @@ struct rhashtable_params {
* @run_work: Deferred worker to expand/shrink asynchronously
* @mutex: Mutex to protect current/future table swapping
* @lock: Spin lock to protect walker list
+ * @nelems: Number of elements in table
*/
struct rhashtable {
struct bucket_table __rcu *tbl;
- atomic_t nelems;
unsigned int key_len;
unsigned int elasticity;
struct rhashtable_params p;
@@ -157,6 +156,7 @@ struct rhashtable {
struct work_struct run_work;
struct mutex mutex;
spinlock_t lock;
+ atomic_t nelems;
};
/**
diff --git a/include/linux/ring_buffer.h b/include/linux/ring_buffer.h
index 4acc552..19d0778 100644
--- a/include/linux/ring_buffer.h
+++ b/include/linux/ring_buffer.h
@@ -162,6 +162,7 @@ void ring_buffer_record_enable(struct ring_buffer *buffer);
void ring_buffer_record_off(struct ring_buffer *buffer);
void ring_buffer_record_on(struct ring_buffer *buffer);
int ring_buffer_record_is_on(struct ring_buffer *buffer);
+int ring_buffer_record_is_set_on(struct ring_buffer *buffer);
void ring_buffer_record_disable_cpu(struct ring_buffer *buffer, int cpu);
void ring_buffer_record_enable_cpu(struct ring_buffer *buffer, int cpu);
diff --git a/include/linux/rmap.h b/include/linux/rmap.h
index 71fd2b3..cd8430e 100644
--- a/include/linux/rmap.h
+++ b/include/linux/rmap.h
@@ -12,8 +12,7 @@
extern int isolate_lru_page(struct page *page);
extern void putback_lru_page(struct page *page);
-extern unsigned long reclaim_pages_from_list(struct list_head *page_list,
- struct vm_area_struct *vma);
+extern unsigned long reclaim_pages(struct list_head *page_list);
/*
* The anon_vma heads a list of private "related" vmas, to scan if
@@ -191,8 +190,7 @@ int page_referenced(struct page *, int is_locked,
#define TTU_ACTION(x) ((x) & TTU_ACTION_MASK)
-int try_to_unmap(struct page *, enum ttu_flags flags,
- struct vm_area_struct *vma);
+int try_to_unmap(struct page *, enum ttu_flags flags);
/*
* Used by uprobes to replace a userspace page safely
@@ -269,7 +267,6 @@ int page_mapped_in_vma(struct page *page, struct vm_area_struct *vma);
*/
struct rmap_walk_control {
void *arg;
- struct vm_area_struct *target_vma;
int (*rmap_one)(struct page *page, struct vm_area_struct *vma,
unsigned long addr, void *arg);
int (*done)(struct page *page);
@@ -294,7 +291,7 @@ static inline int page_referenced(struct page *page, int is_locked,
return 0;
}
-#define try_to_unmap(page, refs, vma) SWAP_FAIL
+#define try_to_unmap(page, refs) SWAP_FAIL
static inline int page_mkclean(struct page *page)
{
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 0748b7b..ad1816b 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -26,6 +26,7 @@ struct sched_param {
#include <linux/nodemask.h>
#include <linux/mm_types.h>
#include <linux/preempt.h>
+#include <linux/mm_event.h>
#include <asm/page.h>
#include <asm/ptrace.h>
@@ -51,6 +52,7 @@ struct sched_param {
#include <linux/timer.h>
#include <linux/hrtimer.h>
#include <linux/kcov.h>
+#include <linux/psi_types.h>
#include <linux/task_io_accounting.h>
#include <linux/latencytop.h>
#include <linux/cred.h>
@@ -139,31 +141,6 @@ struct nameidata;
#define VMACACHE_SIZE (1U << VMACACHE_BITS)
#define VMACACHE_MASK (VMACACHE_SIZE - 1)
-/*
- * These are the constant used to fake the fixed-point load-average
- * counting. Some notes:
- * - 11 bit fractions expand to 22 bits by the multiplies: this gives
- * a load-average precision of 10 bits integer + 11 bits fractional
- * - if you want to count load-averages more often, you need more
- * precision, or rounding will get you. With 2-second counting freq,
- * the EXP_n values would be 1981, 2034 and 2043 if still using only
- * 11 bit fractions.
- */
-extern unsigned long avenrun[]; /* Load averages */
-extern void get_avenrun(unsigned long *loads, unsigned long offset, int shift);
-
-#define FSHIFT 11 /* nr of bits of precision */
-#define FIXED_1 (1<<FSHIFT) /* 1.0 as fixed-point */
-#define LOAD_FREQ (5*HZ+1) /* 5 sec intervals */
-#define EXP_1 1884 /* 1/exp(5sec/1min) as fixed-point */
-#define EXP_5 2014 /* 1/exp(5sec/5min) */
-#define EXP_15 2037 /* 1/exp(5sec/15min) */
-
-#define CALC_LOAD(load,exp,n) \
- load *= exp; \
- load += n*(FIXED_1-exp); \
- load >>= FSHIFT;
-
extern unsigned long total_forks;
extern int nr_threads;
DECLARE_PER_CPU(unsigned long, process_counts);
@@ -214,8 +191,6 @@ static inline int sched_set_wake_up_idle(struct task_struct *p,
}
#endif
-extern void calc_global_load(unsigned long ticks);
-
#if defined(CONFIG_SMP) && defined(CONFIG_NO_HZ_COMMON)
extern void cpu_load_update_nohz_start(void);
extern void cpu_load_update_nohz_stop(void);
@@ -1019,39 +994,7 @@ struct sched_info {
};
#endif /* CONFIG_SCHED_INFO */
-#ifdef CONFIG_TASK_DELAY_ACCT
-struct task_delay_info {
- spinlock_t lock;
- unsigned int flags; /* Private per-task flags */
-
- /* For each stat XXX, add following, aligned appropriately
- *
- * struct timespec XXX_start, XXX_end;
- * u64 XXX_delay;
- * u32 XXX_count;
- *
- * Atomicity of updates to XXX_delay, XXX_count protected by
- * single lock above (split into XXX_lock if contention is an issue).
- */
-
- /*
- * XXX_count is incremented on every XXX operation, the delay
- * associated with the operation is added to XXX_delay.
- * XXX_delay contains the accumulated delay time in nanoseconds.
- */
- u64 blkio_start; /* Shared by blkio, swapin */
- u64 blkio_delay; /* wait for sync block io completion */
- u64 swapin_delay; /* wait for swapin block io completion */
- u32 blkio_count; /* total count of the number of sync block */
- /* io operations performed */
- u32 swapin_count; /* total count of the number of swapin block */
- /* io operations performed */
-
- u64 freepages_start;
- u64 freepages_delay; /* wait for memory reclaim */
- u32 freepages_count; /* total count of memory reclaim */
-};
-#endif /* CONFIG_TASK_DELAY_ACCT */
+struct task_delay_info;
static inline int sched_info_on(void)
{
@@ -1230,7 +1173,6 @@ struct eas_stats {
/* select_energy_cpu_brute() stats */
u64 secb_attempts;
- u64 secb_sync;
u64 secb_idle_bt;
u64 secb_insuff_cap;
u64 secb_no_nrg_sav;
@@ -1529,7 +1471,6 @@ struct sched_statistics {
/* energy_aware_wake_cpu() */
u64 nr_wakeups_secb_attempts;
- u64 nr_wakeups_secb_sync;
u64 nr_wakeups_secb_idle_bt;
u64 nr_wakeups_secb_insuff_cap;
u64 nr_wakeups_secb_no_nrg_sav;
@@ -1769,7 +1710,7 @@ struct task_struct {
struct sched_entity se;
struct sched_rt_entity rt;
u64 last_sleep_ts;
- u64 last_cpu_selected_ts;
+ u64 last_cpu_deselected_ts;
#ifdef CONFIG_SCHED_WALT
struct ravg ravg;
/*
@@ -1829,7 +1770,7 @@ struct task_struct {
struct mm_struct *mm, *active_mm;
/* per-thread vma caching */
- u32 vmacache_seqnum;
+ u64 vmacache_seqnum;
struct vm_area_struct *vmacache[VMACACHE_SIZE];
#if defined(SPLIT_RSS_COUNTING)
struct task_rss_stat rss_stat;
@@ -1848,6 +1789,10 @@ struct task_struct {
unsigned sched_contributes_to_load:1;
unsigned sched_migrated:1;
unsigned sched_remote_wakeup:1;
+#ifdef CONFIG_PSI
+ unsigned sched_psi_wake_requeue:1;
+#endif
+
unsigned :0; /* force alignment to the next boundary */
/* unserialized, strictly 'current' */
@@ -1917,6 +1862,8 @@ struct task_struct {
#ifdef CONFIG_CPU_FREQ_TIMES
u64 *time_in_state;
unsigned int max_state;
+ u64 *concurrent_active_time;
+ u64 *concurrent_policy_time;
#endif
struct prev_cputime prev_cputime;
#ifdef CONFIG_VIRT_CPU_ACCOUNTING_GEN
@@ -2013,6 +1960,10 @@ struct task_struct {
struct rt_mutex_waiter *pi_blocked_on;
#endif
+#ifdef CONFIG_MM_EVENT_STAT
+ struct mm_event_task mm_event[MM_TYPE_NUM];
+ unsigned long next_period;
+#endif
#ifdef CONFIG_DEBUG_MUTEXES
/* mutex deadlock detection */
struct mutex_waiter *blocked_on;
@@ -2065,6 +2016,10 @@ struct task_struct {
unsigned long ptrace_message;
siginfo_t *last_siginfo; /* For ptrace use. */
struct task_io_accounting ioac;
+#ifdef CONFIG_PSI
+ /* Pressure stall state */
+ unsigned int psi_flags;
+#endif
#if defined(CONFIG_TASK_XACCT)
u64 acct_rss_mem1; /* accumulated rss usage */
u64 acct_vm_mem1; /* accumulated virtual memory usage */
@@ -2158,9 +2113,10 @@ struct task_struct {
struct page_frag task_frag;
-#ifdef CONFIG_TASK_DELAY_ACCT
- struct task_delay_info *delays;
+#ifdef CONFIG_TASK_DELAY_ACCT
+ struct task_delay_info *delays;
#endif
+
#ifdef CONFIG_FAULT_INJECTION
int make_it_fail;
#endif
@@ -2571,6 +2527,7 @@ extern void thread_group_cputime_adjusted(struct task_struct *p, cputime_t *ut,
#define PF_KTHREAD 0x00200000 /* I am a kernel thread */
#define PF_RANDOMIZE 0x00400000 /* randomize virtual address space */
#define PF_SWAPWRITE 0x00800000 /* Allowed to write to swap */
+#define PF_MEMSTALL 0x01000000 /* Stalled due to lack of memory */
#define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_allowed */
#define PF_MCE_EARLY 0x08000000 /* Early kill for mce process policy */
#define PF_MUTEX_TESTER 0x20000000 /* Thread belongs to the rt mutex tester */
@@ -2629,6 +2586,8 @@ static inline void memalloc_noio_restore(unsigned int flags)
#define PFA_SPREAD_PAGE 1 /* Spread page cache over cpuset */
#define PFA_SPREAD_SLAB 2 /* Spread some slab caches over cpuset */
#define PFA_LMK_WAITING 3 /* Lowmemorykiller is waiting */
+#define PFA_SPEC_SSB_DISABLE 4 /* Speculative Store Bypass disabled */
+#define PFA_SPEC_SSB_FORCE_DISABLE 5 /* Speculative Store Bypass force disabled*/
#define TASK_PFA_TEST(name, func) \
@@ -2655,6 +2614,13 @@ TASK_PFA_CLEAR(SPREAD_SLAB, spread_slab)
TASK_PFA_TEST(LMK_WAITING, lmk_waiting)
TASK_PFA_SET(LMK_WAITING, lmk_waiting)
+TASK_PFA_TEST(SPEC_SSB_DISABLE, spec_ssb_disable)
+TASK_PFA_SET(SPEC_SSB_DISABLE, spec_ssb_disable)
+TASK_PFA_CLEAR(SPEC_SSB_DISABLE, spec_ssb_disable)
+
+TASK_PFA_TEST(SPEC_SSB_FORCE_DISABLE, spec_ssb_force_disable)
+TASK_PFA_SET(SPEC_SSB_FORCE_DISABLE, spec_ssb_force_disable)
+
/*
* task->jobctl flags
*/
@@ -3340,7 +3306,11 @@ static inline void set_task_comm(struct task_struct *tsk, const char *from)
{
__set_task_comm(tsk, from, false);
}
-extern char *get_task_comm(char *to, struct task_struct *tsk);
+extern char *__get_task_comm(char *to, size_t len, struct task_struct *tsk);
+#define get_task_comm(buf, tsk) ({ \
+ BUILD_BUG_ON(sizeof(buf) != TASK_COMM_LEN); \
+ __get_task_comm(buf, sizeof(buf), tsk); \
+})
#ifdef CONFIG_SMP
void scheduler_ipi(void);
@@ -3974,7 +3944,7 @@ static inline unsigned long rlimit_max(unsigned int limit)
#define SCHED_CPUFREQ_DL (1U << 1)
#define SCHED_CPUFREQ_IOWAIT (1U << 2)
#define SCHED_CPUFREQ_INTERCLUSTER_MIG (1U << 3)
-#define SCHED_CPUFREQ_WALT (1U << 4)
+#define SCHED_CPUFREQ_RESERVED (1U << 4)
#define SCHED_CPUFREQ_PL (1U << 5)
#define SCHED_CPUFREQ_EARLY_DET (1U << 6)
#define SCHED_CPUFREQ_FORCE_UPDATE (1U << 7)
diff --git a/include/linux/sched/loadavg.h b/include/linux/sched/loadavg.h
new file mode 100644
index 0000000..208bbf2
--- /dev/null
+++ b/include/linux/sched/loadavg.h
@@ -0,0 +1,47 @@
+#ifndef _LINUX_SCHED_LOADAVG_H
+#define _LINUX_SCHED_LOADAVG_H
+
+/*
+ * These are the constant used to fake the fixed-point load-average
+ * counting. Some notes:
+ * - 11 bit fractions expand to 22 bits by the multiplies: this gives
+ * a load-average precision of 10 bits integer + 11 bits fractional
+ * - if you want to count load-averages more often, you need more
+ * precision, or rounding will get you. With 2-second counting freq,
+ * the EXP_n values would be 1981, 2034 and 2043 if still using only
+ * 11 bit fractions.
+ */
+extern unsigned long avenrun[]; /* Load averages */
+extern void get_avenrun(unsigned long *loads, unsigned long offset, int shift);
+
+#define FSHIFT 11 /* nr of bits of precision */
+#define FIXED_1 (1<<FSHIFT) /* 1.0 as fixed-point */
+#define LOAD_FREQ (5*HZ+1) /* 5 sec intervals */
+#define EXP_1 1884 /* 1/exp(5sec/1min) as fixed-point */
+#define EXP_5 2014 /* 1/exp(5sec/5min) */
+#define EXP_15 2037 /* 1/exp(5sec/15min) */
+
+/*
+ * a1 = a0 * e + a * (1 - e)
+ */
+static inline unsigned long
+calc_load(unsigned long load, unsigned long exp, unsigned long active)
+{
+ unsigned long newload;
+
+ newload = load * exp + active * (FIXED_1 - exp);
+ if (active >= load)
+ newload += FIXED_1-1;
+
+ return newload / FIXED_1;
+}
+
+extern unsigned long calc_load_n(unsigned long load, unsigned long exp,
+ unsigned long active, unsigned int n);
+
+#define LOAD_INT(x) ((x) >> FSHIFT)
+#define LOAD_FRAC(x) LOAD_INT(((x) & (FIXED_1-1)) * 100)
+
+extern void calc_global_load(unsigned long ticks);
+
+#endif /* _LINUX_SCHED_LOADAVG_H */
diff --git a/include/linux/scs.h b/include/linux/scs.h
new file mode 100644
index 0000000..8203485f
--- /dev/null
+++ b/include/linux/scs.h
@@ -0,0 +1,77 @@
+/*
+ * Shadow Call Stack support.
+ *
+ * Copyright (C) 2018 Google LLC
+ */
+
+#ifndef _LINUX_SCS_H
+#define _LINUX_SCS_H
+
+#ifdef CONFIG_SHADOW_CALL_STACK
+
+#include <linux/gfp.h>
+#include <linux/sched.h>
+#include <asm/page.h>
+
+#define SCS_SIZE 1024
+#define SCS_GFP (GFP_KERNEL | __GFP_ZERO)
+
+extern unsigned long init_shadow_call_stack[];
+
+static inline void *task_scs(struct task_struct *tsk)
+{
+ return task_thread_info(tsk)->shadow_call_stack;
+}
+
+static inline void task_set_scs(struct task_struct *tsk, void *s)
+{
+ task_thread_info(tsk)->shadow_call_stack = s;
+}
+
+extern void scs_set_init_magic(struct task_struct *tsk);
+extern void scs_task_init(struct task_struct *tsk);
+extern void scs_task_reset(struct task_struct *tsk);
+extern int scs_prepare(struct task_struct *tsk, int node);
+extern bool scs_corrupted(struct task_struct *tsk);
+extern void scs_release(struct task_struct *tsk);
+
+#else /* CONFIG_SHADOW_CALL_STACK */
+
+static inline void *task_scs(struct task_struct *tsk)
+{
+ return 0;
+}
+
+static inline void task_set_scs(struct task_struct *tsk, void *s)
+{
+}
+
+static inline void scs_set_init_magic(struct task_struct *tsk)
+{
+}
+
+static inline void scs_task_init(struct task_struct *tsk)
+{
+}
+
+static inline void scs_task_reset(struct task_struct *tsk)
+{
+}
+
+static inline int scs_prepare(struct task_struct *tsk, int node)
+{
+ return 0;
+}
+
+static inline bool scs_corrupted(struct task_struct *tsk)
+{
+ return false;
+}
+
+static inline void scs_release(struct task_struct *tsk)
+{
+}
+
+#endif /* CONFIG_SHADOW_CALL_STACK */
+
+#endif /* _LINUX_SCS_H */
diff --git a/include/linux/seccomp.h b/include/linux/seccomp.h
index ecc296c..50c460a 100644
--- a/include/linux/seccomp.h
+++ b/include/linux/seccomp.h
@@ -3,7 +3,8 @@
#include <uapi/linux/seccomp.h>
-#define SECCOMP_FILTER_FLAG_MASK (SECCOMP_FILTER_FLAG_TSYNC)
+#define SECCOMP_FILTER_FLAG_MASK (SECCOMP_FILTER_FLAG_TSYNC | \
+ SECCOMP_FILTER_FLAG_SPEC_ALLOW)
#ifdef CONFIG_SECCOMP
diff --git a/include/linux/serial_core.h b/include/linux/serial_core.h
index 1a94397..c7dff69 100644
--- a/include/linux/serial_core.h
+++ b/include/linux/serial_core.h
@@ -345,13 +345,14 @@ struct earlycon_device {
};
struct earlycon_id {
- char name[16];
+ char name[15];
+ char name_term; /* In case compiler didn't '\0' term name */
char compatible[128];
int (*setup)(struct earlycon_device *, const char *options);
-} __aligned(32);
+};
-extern const struct earlycon_id __earlycon_table[];
-extern const struct earlycon_id __earlycon_table_end[];
+extern const struct earlycon_id *__earlycon_table[];
+extern const struct earlycon_id *__earlycon_table_end[];
#if defined(CONFIG_SERIAL_EARLYCON) && !defined(MODULE)
#define EARLYCON_USED_OR_UNUSED __used
@@ -359,12 +360,19 @@ extern const struct earlycon_id __earlycon_table_end[];
#define EARLYCON_USED_OR_UNUSED __maybe_unused
#endif
-#define OF_EARLYCON_DECLARE(_name, compat, fn) \
- static const struct earlycon_id __UNIQUE_ID(__earlycon_##_name) \
- EARLYCON_USED_OR_UNUSED __section(__earlycon_table) \
+#define _OF_EARLYCON_DECLARE(_name, compat, fn, unique_id) \
+ static const struct earlycon_id unique_id \
+ EARLYCON_USED_OR_UNUSED __initconst \
= { .name = __stringify(_name), \
.compatible = compat, \
- .setup = fn }
+ .setup = fn }; \
+ static const struct earlycon_id EARLYCON_USED_OR_UNUSED \
+ __section(__earlycon_table) \
+ * const __PASTE(__p, unique_id) = &unique_id
+
+#define OF_EARLYCON_DECLARE(_name, compat, fn) \
+ _OF_EARLYCON_DECLARE(_name, compat, fn, \
+ __UNIQUE_ID(__earlycon_##_name))
#define EARLYCON_DECLARE(_name, fn) OF_EARLYCON_DECLARE(_name, "", fn)
diff --git a/include/linux/signal.h b/include/linux/signal.h
index b63f63e..5308304 100644
--- a/include/linux/signal.h
+++ b/include/linux/signal.h
@@ -97,6 +97,23 @@ static inline int sigisemptyset(sigset_t *set)
}
}
+static inline int sigequalsets(const sigset_t *set1, const sigset_t *set2)
+{
+ switch (_NSIG_WORDS) {
+ case 4:
+ return (set1->sig[3] == set2->sig[3]) &&
+ (set1->sig[2] == set2->sig[2]) &&
+ (set1->sig[1] == set2->sig[1]) &&
+ (set1->sig[0] == set2->sig[0]);
+ case 2:
+ return (set1->sig[1] == set2->sig[1]) &&
+ (set1->sig[0] == set2->sig[0]);
+ case 1:
+ return set1->sig[0] == set2->sig[0];
+ }
+ return 0;
+}
+
#define sigmask(sig) (1UL << ((sig) - 1))
#ifndef __HAVE_ARCH_SIG_SETOPS
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index 3fbfe96..11bb7cb 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -602,6 +602,7 @@ static inline bool skb_mstamp_after(const struct skb_mstamp *t1,
* @hash: the packet hash
* @queue_mapping: Queue mapping for multiqueue devices
* @xmit_more: More SKBs are pending for this queue
+ * @pfmemalloc: skbuff was allocated from PFMEMALLOC reserves
* @ndisc_nodetype: router type (from link layer)
* @ooo_okay: allow the mapping of a socket to a queue to be changed
* @l4_hash: indicate hash is a canonical 4-tuple hash over transport
@@ -642,9 +643,14 @@ struct sk_buff {
struct skb_mstamp skb_mstamp;
};
};
- struct rb_node rbnode; /* used in netem & tcp stack */
+ struct rb_node rbnode; /* used in netem, ip4 defrag, and tcp stack */
};
- struct sock *sk;
+
+ union {
+ struct sock *sk;
+ int ip_defrag_offset;
+ };
+
struct net_device *dev;
/*
@@ -692,7 +698,7 @@ struct sk_buff {
peeked:1,
head_frag:1,
xmit_more:1,
- __unused:1; /* one bit hole */
+ pfmemalloc:1;
kmemcheck_bitfield_end(flags1);
/* fields enclosed in headers_start/headers_end are copied
@@ -712,19 +718,18 @@ struct sk_buff {
__u8 __pkt_type_offset[0];
__u8 pkt_type:3;
- __u8 pfmemalloc:1;
__u8 ignore_df:1;
__u8 nfctinfo:3;
-
__u8 nf_trace:1;
+
__u8 ip_summed:2;
__u8 ooo_okay:1;
__u8 l4_hash:1;
__u8 sw_hash:1;
__u8 wifi_acked_valid:1;
__u8 wifi_acked:1;
-
__u8 no_fcs:1;
+
/* Indicates the inner headers are valid in the skbuff. */
__u8 encapsulation:1;
__u8 encap_hdr_csum:1;
@@ -732,11 +737,11 @@ struct sk_buff {
__u8 csum_complete_sw:1;
__u8 csum_level:2;
__u8 csum_bad:1;
-
#ifdef CONFIG_IPV6_NDISC_NODETYPE
__u8 ndisc_nodetype:2;
#endif
__u8 ipvs_property:1;
+
__u8 inner_protocol_type:1;
__u8 fast_forwarded:1;
__u8 remcsum_offload:1;
@@ -2417,7 +2422,7 @@ static inline void __skb_queue_purge(struct sk_buff_head *list)
kfree_skb(skb);
}
-void skb_rbtree_purge(struct rb_root *root);
+unsigned int skb_rbtree_purge(struct rb_root *root);
void *netdev_alloc_frag(unsigned int fragsz);
@@ -2953,6 +2958,7 @@ static inline unsigned char *skb_push_rcsum(struct sk_buff *skb,
return skb->data;
}
+int pskb_trim_rcsum_slow(struct sk_buff *skb, unsigned int len);
/**
* pskb_trim_rcsum - trim received skb and update checksum
* @skb: buffer to trim
@@ -2966,9 +2972,7 @@ static inline int pskb_trim_rcsum(struct sk_buff *skb, unsigned int len)
{
if (likely(len >= skb->len))
return 0;
- if (skb->ip_summed == CHECKSUM_COMPLETE)
- skb->ip_summed = CHECKSUM_NONE;
- return __pskb_trim(skb, len);
+ return pskb_trim_rcsum_slow(skb, len);
}
static inline int __skb_trim_rcsum(struct sk_buff *skb, unsigned int len)
@@ -2986,6 +2990,14 @@ static inline int __skb_grow_rcsum(struct sk_buff *skb, unsigned int len)
return __skb_grow(skb, len);
}
+#define rb_to_skb(rb) rb_entry_safe(rb, struct sk_buff, rbnode)
+
+#define rb_to_skb(rb) rb_entry_safe(rb, struct sk_buff, rbnode)
+#define skb_rb_first(root) rb_to_skb(rb_first(root))
+#define skb_rb_last(root) rb_to_skb(rb_last(root))
+#define skb_rb_next(skb) rb_to_skb(rb_next(&(skb)->rbnode))
+#define skb_rb_prev(skb) rb_to_skb(rb_prev(&(skb)->rbnode))
+
#define skb_queue_walk(queue, skb) \
for (skb = (queue)->next; \
skb != (struct sk_buff *)(queue); \
@@ -3000,6 +3012,18 @@ static inline int __skb_grow_rcsum(struct sk_buff *skb, unsigned int len)
for (; skb != (struct sk_buff *)(queue); \
skb = skb->next)
+#define skb_rbtree_walk(skb, root) \
+ for (skb = skb_rb_first(root); skb != NULL; \
+ skb = skb_rb_next(skb))
+
+#define skb_rbtree_walk_from(skb) \
+ for (; skb != NULL; \
+ skb = skb_rb_next(skb))
+
+#define skb_rbtree_walk_from_safe(skb, tmp) \
+ for (; tmp = skb ? skb_rb_next(skb) : NULL, (skb != NULL); \
+ skb = tmp)
+
#define skb_queue_walk_from_safe(queue, skb, tmp) \
for (tmp = skb->next; \
skb != (struct sk_buff *)(queue); \
diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h
index 75f56c2..b6a59e8 100644
--- a/include/linux/slub_def.h
+++ b/include/linux/slub_def.h
@@ -67,7 +67,8 @@ struct kmem_cache {
int size; /* The size of an object including meta data */
int object_size; /* The size of an object without meta data */
int offset; /* Free pointer offset. */
- int cpu_partial; /* Number of per cpu partial objects to keep around */
+ /* Number of per cpu partial objects to keep around */
+ unsigned int cpu_partial;
struct kmem_cache_order_objects oo;
/* Allocation and freeing of slabs */
diff --git a/include/linux/string.h b/include/linux/string.h
index b7e8f42..76eb56e 100644
--- a/include/linux/string.h
+++ b/include/linux/string.h
@@ -27,7 +27,7 @@ extern char * strncpy(char *,const char *, __kernel_size_t);
size_t strlcpy(char *, const char *, size_t);
#endif
#ifndef __HAVE_ARCH_STRSCPY
-ssize_t __must_check strscpy(char *, const char *, size_t);
+ssize_t strscpy(char *, const char *, size_t);
#endif
#ifndef __HAVE_ARCH_STRCAT
extern char * strcat(char *, const char *);
@@ -136,6 +136,16 @@ static inline int strtobool(const char *s, bool *res)
}
int match_string(const char * const *array, size_t n, const char *string);
+int __sysfs_match_string(const char * const *array, size_t n, const char *s);
+
+/**
+ * sysfs_match_string - matches given string in an array
+ * @_a: array of strings
+ * @_s: string to match with
+ *
+ * Helper for __sysfs_match_string(). Calculates the size of @a automatically.
+ */
+#define sysfs_match_string(_a, _s) __sysfs_match_string(_a, ARRAY_SIZE(_a), _s)
#ifdef CONFIG_BINARY_PRINTF
int vbin_printf(u32 *bin_buf, size_t size, const char *fmt, va_list args);
diff --git a/include/linux/sunrpc/clnt.h b/include/linux/sunrpc/clnt.h
index 333ad11..44161a4 100644
--- a/include/linux/sunrpc/clnt.h
+++ b/include/linux/sunrpc/clnt.h
@@ -155,6 +155,7 @@ int rpc_switch_client_transport(struct rpc_clnt *,
void rpc_shutdown_client(struct rpc_clnt *);
void rpc_release_client(struct rpc_clnt *);
+void rpc_task_release_transport(struct rpc_task *);
void rpc_task_release_client(struct rpc_task *);
int rpcb_create_local(struct net *);
diff --git a/include/linux/suspend.h b/include/linux/suspend.h
index 448321b..90d8569 100644
--- a/include/linux/suspend.h
+++ b/include/linux/suspend.h
@@ -378,6 +378,8 @@ extern int swsusp_page_is_forbidden(struct page *);
extern void swsusp_set_page_free(struct page *);
extern void swsusp_unset_page_free(struct page *);
extern unsigned long get_safe_page(gfp_t gfp_mask);
+extern asmlinkage int swsusp_arch_suspend(void);
+extern asmlinkage int swsusp_arch_resume(void);
extern void hibernation_set_ops(const struct platform_hibernation_ops *ops);
extern int hibernate(void);
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 92d1fde..a358f92 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -249,7 +249,7 @@ struct swap_info_struct {
/* linux/mm/workingset.c */
void *workingset_eviction(struct address_space *mapping, struct page *page);
-bool workingset_refault(void *shadow);
+void workingset_refault(struct page *page, void *shadow);
void workingset_activation(struct page *page);
extern struct list_lru workingset_shadow_nodes;
diff --git a/include/linux/swapfile.h b/include/linux/swapfile.h
index ed2a9c9..ac912f2 100644
--- a/include/linux/swapfile.h
+++ b/include/linux/swapfile.h
@@ -11,6 +11,8 @@ extern spinlock_t swap_avail_lock;
extern struct plist_head swap_avail_head;
extern struct swap_info_struct *swap_info[];
extern int try_to_unuse(unsigned int, bool, unsigned long);
+extern unsigned long generic_max_swapfile_size(void);
+extern unsigned long max_swapfile_size(void);
extern int swap_ratio(struct swap_info_struct **si);
extern void setup_swap_ratio(struct swap_info_struct *p, int prio);
extern bool is_swap_ratio_group(int prio);
diff --git a/include/linux/sysctl.h b/include/linux/sysctl.h
index 8f84c84..2ce7675 100644
--- a/include/linux/sysctl.h
+++ b/include/linux/sysctl.h
@@ -146,6 +146,7 @@ struct ctl_table_header
struct ctl_table_set *set;
struct ctl_dir *parent;
struct ctl_node *node;
+ struct hlist_head inodes; /* head for proc_inode->sysctl_inodes */
};
struct ctl_dir {
diff --git a/include/linux/sysfs.h b/include/linux/sysfs.h
index 00a1f33..d3c19f8 100644
--- a/include/linux/sysfs.h
+++ b/include/linux/sysfs.h
@@ -238,6 +238,9 @@ int __must_check sysfs_create_files(struct kobject *kobj,
const struct attribute **attr);
int __must_check sysfs_chmod_file(struct kobject *kobj,
const struct attribute *attr, umode_t mode);
+struct kernfs_node *sysfs_break_active_protection(struct kobject *kobj,
+ const struct attribute *attr);
+void sysfs_unbreak_active_protection(struct kernfs_node *kn);
void sysfs_remove_file_ns(struct kobject *kobj, const struct attribute *attr,
const void *ns);
bool sysfs_remove_file_self(struct kobject *kobj, const struct attribute *attr);
@@ -351,6 +354,17 @@ static inline int sysfs_chmod_file(struct kobject *kobj,
return 0;
}
+static inline struct kernfs_node *
+sysfs_break_active_protection(struct kobject *kobj,
+ const struct attribute *attr)
+{
+ return NULL;
+}
+
+static inline void sysfs_unbreak_active_protection(struct kernfs_node *kn)
+{
+}
+
static inline void sysfs_remove_file_ns(struct kobject *kobj,
const struct attribute *attr,
const void *ns)
diff --git a/include/linux/syslog.h b/include/linux/syslog.h
index c3a7f0c..e1c3632 100644
--- a/include/linux/syslog.h
+++ b/include/linux/syslog.h
@@ -49,13 +49,4 @@
int do_syslog(int type, char __user *buf, int count, int source);
-#ifdef CONFIG_PRINTK
-int check_syslog_permissions(int type, int source);
-#else
-static inline int check_syslog_permissions(int type, int source)
-{
- return 0;
-}
-#endif
-
#endif /* _LINUX_SYSLOG_H */
diff --git a/include/linux/tc.h b/include/linux/tc.h
index f92511e..a60639f 100644
--- a/include/linux/tc.h
+++ b/include/linux/tc.h
@@ -84,6 +84,7 @@ struct tc_dev {
device. */
struct device dev; /* Generic device interface. */
struct resource resource; /* Address space of this device. */
+ u64 dma_mask; /* DMA addressable range. */
char vendor[9];
char name[9];
char firmware[9];
diff --git a/include/linux/tcp.h b/include/linux/tcp.h
index fc11641..b8ea15a 100644
--- a/include/linux/tcp.h
+++ b/include/linux/tcp.h
@@ -338,7 +338,7 @@ struct tcp_sock {
/* Receiver queue space */
struct {
- int space;
+ u32 space;
u32 seq;
u32 time;
} rcvq_space;
diff --git a/include/linux/thread_info.h b/include/linux/thread_info.h
index 5837387..ce98a8e 100644
--- a/include/linux/thread_info.h
+++ b/include/linux/thread_info.h
@@ -26,12 +26,7 @@
#ifdef __KERNEL__
-#ifdef CONFIG_DEBUG_STACK_USAGE
-# define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_NOTRACK | \
- __GFP_ZERO)
-#else
-# define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_NOTRACK)
-#endif
+#define THREADINFO_GFP (GFP_KERNEL_ACCOUNT | __GFP_NOTRACK | __GFP_ZERO)
/*
* flag set/clear/test wrappers
diff --git a/include/linux/tty.h b/include/linux/tty.h
index 6f1ee85..fe1b862 100644
--- a/include/linux/tty.h
+++ b/include/linux/tty.h
@@ -657,7 +657,7 @@ extern int tty_unregister_ldisc(int disc);
extern int tty_set_ldisc(struct tty_struct *tty, int disc);
extern int tty_ldisc_setup(struct tty_struct *tty, struct tty_struct *o_tty);
extern void tty_ldisc_release(struct tty_struct *tty);
-extern void tty_ldisc_init(struct tty_struct *tty);
+extern int __must_check tty_ldisc_init(struct tty_struct *tty);
extern void tty_ldisc_deinit(struct tty_struct *tty);
extern int tty_ldisc_receive_buf(struct tty_ldisc *ld, unsigned char *p,
char *f, int count);
diff --git a/include/linux/uaccess.h b/include/linux/uaccess.h
index f30c187..9442423 100644
--- a/include/linux/uaccess.h
+++ b/include/linux/uaccess.h
@@ -2,6 +2,9 @@
#define __LINUX_UACCESS_H__
#include <linux/sched.h>
+
+#define uaccess_kernel() segment_eq(get_fs(), KERNEL_DS)
+
#include <asm/uaccess.h>
static __always_inline void pagefault_disabled_inc(void)
diff --git a/include/linux/usb.h b/include/linux/usb.h
index 81e8469..47d29d8 100644
--- a/include/linux/usb.h
+++ b/include/linux/usb.h
@@ -338,11 +338,11 @@ struct usb_host_bos {
};
int __usb_get_extra_descriptor(char *buffer, unsigned size,
- unsigned char type, void **ptr);
+ unsigned char type, void **ptr, size_t min);
#define usb_get_extra_descriptor(ifpoint, type, ptr) \
__usb_get_extra_descriptor((ifpoint)->extra, \
(ifpoint)->extralen, \
- type, (void **)ptr)
+ type, (void **)ptr, sizeof(**(ptr)))
/* ----------------------------------------------------------------------- */
diff --git a/include/linux/usb/audio-v2.h b/include/linux/usb/audio-v2.h
index c5f2158..a877836b 100644
--- a/include/linux/usb/audio-v2.h
+++ b/include/linux/usb/audio-v2.h
@@ -35,12 +35,12 @@
static inline bool uac2_control_is_readable(u32 bmControls, u8 control)
{
- return (bmControls >> (control * 2)) & 0x1;
+ return (bmControls >> ((control - 1) * 2)) & 0x1;
}
static inline bool uac2_control_is_writeable(u32 bmControls, u8 control)
{
- return (bmControls >> (control * 2)) & 0x2;
+ return (bmControls >> ((control - 1) * 2)) & 0x2;
}
/* 4.7.2 Class-Specific AC Interface Descriptor */
diff --git a/include/linux/usb/composite.h b/include/linux/usb/composite.h
index dbf6240..87e97bb 100644
--- a/include/linux/usb/composite.h
+++ b/include/linux/usb/composite.h
@@ -60,6 +60,9 @@
/* big enough to hold our biggest descriptor */
#define USB_COMP_EP0_BUFSIZ 4096
+/* OS feature descriptor length <= 4kB */
+#define USB_COMP_EP0_OS_DESC_BUFSIZ 4096
+
#define USB_MS_TO_HS_INTERVAL(x) (ilog2((x * 1000 / 125)) + 1)
struct usb_configuration;
diff --git a/include/linux/usb/pd.h b/include/linux/usb/pd.h
new file mode 100644
index 0000000..406f18d
--- /dev/null
+++ b/include/linux/usb/pd.h
@@ -0,0 +1,301 @@
+/*
+ * Copyright 2015-2017 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __LINUX_USB_PD_H
+#define __LINUX_USB_PD_H
+
+#include <linux/types.h>
+#include <linux/usb/typec.h>
+
+/* USB PD Messages */
+enum pd_ctrl_msg_type {
+ /* 0 Reserved */
+ PD_CTRL_GOOD_CRC = 1,
+ PD_CTRL_GOTO_MIN = 2,
+ PD_CTRL_ACCEPT = 3,
+ PD_CTRL_REJECT = 4,
+ PD_CTRL_PING = 5,
+ PD_CTRL_PS_RDY = 6,
+ PD_CTRL_GET_SOURCE_CAP = 7,
+ PD_CTRL_GET_SINK_CAP = 8,
+ PD_CTRL_DR_SWAP = 9,
+ PD_CTRL_PR_SWAP = 10,
+ PD_CTRL_VCONN_SWAP = 11,
+ PD_CTRL_WAIT = 12,
+ PD_CTRL_SOFT_RESET = 13,
+ /* 14-15 Reserved */
+};
+
+enum pd_data_msg_type {
+ /* 0 Reserved */
+ PD_DATA_SOURCE_CAP = 1,
+ PD_DATA_REQUEST = 2,
+ PD_DATA_BIST = 3,
+ PD_DATA_SINK_CAP = 4,
+ /* 5-14 Reserved */
+ PD_DATA_VENDOR_DEF = 15,
+};
+
+#define PD_REV10 0x0
+#define PD_REV20 0x1
+
+#define PD_HEADER_CNT_SHIFT 12
+#define PD_HEADER_CNT_MASK 0x7
+#define PD_HEADER_ID_SHIFT 9
+#define PD_HEADER_ID_MASK 0x7
+#define PD_HEADER_PWR_ROLE BIT(8)
+#define PD_HEADER_REV_SHIFT 6
+#define PD_HEADER_REV_MASK 0x3
+#define PD_HEADER_DATA_ROLE BIT(5)
+#define PD_HEADER_TYPE_SHIFT 0
+#define PD_HEADER_TYPE_MASK 0xf
+
+#define PD_HEADER(type, pwr, data, id, cnt) \
+ ((((type) & PD_HEADER_TYPE_MASK) << PD_HEADER_TYPE_SHIFT) | \
+ ((pwr) == TYPEC_SOURCE ? PD_HEADER_PWR_ROLE : 0) | \
+ ((data) == TYPEC_HOST ? PD_HEADER_DATA_ROLE : 0) | \
+ (PD_REV20 << PD_HEADER_REV_SHIFT) | \
+ (((id) & PD_HEADER_ID_MASK) << PD_HEADER_ID_SHIFT) | \
+ (((cnt) & PD_HEADER_CNT_MASK) << PD_HEADER_CNT_SHIFT))
+
+#define PD_HEADER_LE(type, pwr, data, id, cnt) \
+ cpu_to_le16(PD_HEADER((type), (pwr), (data), (id), (cnt)))
+
+static inline unsigned int pd_header_cnt(u16 header)
+{
+ return (header >> PD_HEADER_CNT_SHIFT) & PD_HEADER_CNT_MASK;
+}
+
+static inline unsigned int pd_header_cnt_le(__le16 header)
+{
+ return pd_header_cnt(le16_to_cpu(header));
+}
+
+static inline unsigned int pd_header_type(u16 header)
+{
+ return (header >> PD_HEADER_TYPE_SHIFT) & PD_HEADER_TYPE_MASK;
+}
+
+static inline unsigned int pd_header_type_le(__le16 header)
+{
+ return pd_header_type(le16_to_cpu(header));
+}
+
+static inline unsigned int pd_header_msgid(u16 header)
+{
+ return (header >> PD_HEADER_ID_SHIFT) & PD_HEADER_ID_MASK;
+}
+
+static inline unsigned int pd_header_msgid_le(__le16 header)
+{
+ return pd_header_msgid(le16_to_cpu(header));
+}
+
+#define PD_MAX_PAYLOAD 7
+
+/**
+ * struct pd_message - PD message as seen on wire
+ * @header: PD message header
+ * @payload: PD message payload
+ */
+struct pd_message {
+ __le16 header;
+ __le32 payload[PD_MAX_PAYLOAD];
+} __packed;
+
+/* PDO: Power Data Object */
+#define PDO_MAX_OBJECTS 7
+
+enum pd_pdo_type {
+ PDO_TYPE_FIXED = 0,
+ PDO_TYPE_BATT = 1,
+ PDO_TYPE_VAR = 2,
+};
+
+#define PDO_TYPE_SHIFT 30
+#define PDO_TYPE_MASK 0x3
+
+#define PDO_TYPE(t) ((t) << PDO_TYPE_SHIFT)
+
+#define PDO_VOLT_MASK 0x3ff
+#define PDO_CURR_MASK 0x3ff
+#define PDO_PWR_MASK 0x3ff
+
+#define PDO_FIXED_DUAL_ROLE BIT(29) /* Power role swap supported */
+#define PDO_FIXED_SUSPEND BIT(28) /* USB Suspend supported (Source) */
+#define PDO_FIXED_HIGHER_CAP BIT(28) /* Requires more than vSafe5V (Sink) */
+#define PDO_FIXED_EXTPOWER BIT(27) /* Externally powered */
+#define PDO_FIXED_USB_COMM BIT(26) /* USB communications capable */
+#define PDO_FIXED_DATA_SWAP BIT(25) /* Data role swap supported */
+#define PDO_FIXED_VOLT_SHIFT 10 /* 50mV units */
+#define PDO_FIXED_CURR_SHIFT 0 /* 10mA units */
+
+#define PDO_FIXED_VOLT(mv) ((((mv) / 50) & PDO_VOLT_MASK) << PDO_FIXED_VOLT_SHIFT)
+#define PDO_FIXED_CURR(ma) ((((ma) / 10) & PDO_CURR_MASK) << PDO_FIXED_CURR_SHIFT)
+
+#define PDO_FIXED(mv, ma, flags) \
+ (PDO_TYPE(PDO_TYPE_FIXED) | (flags) | \
+ PDO_FIXED_VOLT(mv) | PDO_FIXED_CURR(ma))
+
+#define VSAFE5V 5000 /* mv units */
+
+#define PDO_BATT_MAX_VOLT_SHIFT 20 /* 50mV units */
+#define PDO_BATT_MIN_VOLT_SHIFT 10 /* 50mV units */
+#define PDO_BATT_MAX_PWR_SHIFT 0 /* 250mW units */
+
+#define PDO_BATT_MIN_VOLT(mv) ((((mv) / 50) & PDO_VOLT_MASK) << PDO_BATT_MIN_VOLT_SHIFT)
+#define PDO_BATT_MAX_VOLT(mv) ((((mv) / 50) & PDO_VOLT_MASK) << PDO_BATT_MAX_VOLT_SHIFT)
+#define PDO_BATT_MAX_POWER(mw) ((((mw) / 250) & PDO_PWR_MASK) << PDO_BATT_MAX_PWR_SHIFT)
+
+#define PDO_BATT(min_mv, max_mv, max_mw) \
+ (PDO_TYPE(PDO_TYPE_BATT) | PDO_BATT_MIN_VOLT(min_mv) | \
+ PDO_BATT_MAX_VOLT(max_mv) | PDO_BATT_MAX_POWER(max_mw))
+
+#define PDO_VAR_MAX_VOLT_SHIFT 20 /* 50mV units */
+#define PDO_VAR_MIN_VOLT_SHIFT 10 /* 50mV units */
+#define PDO_VAR_MAX_CURR_SHIFT 0 /* 10mA units */
+
+#define PDO_VAR_MIN_VOLT(mv) ((((mv) / 50) & PDO_VOLT_MASK) << PDO_VAR_MIN_VOLT_SHIFT)
+#define PDO_VAR_MAX_VOLT(mv) ((((mv) / 50) & PDO_VOLT_MASK) << PDO_VAR_MAX_VOLT_SHIFT)
+#define PDO_VAR_MAX_CURR(ma) ((((ma) / 10) & PDO_CURR_MASK) << PDO_VAR_MAX_CURR_SHIFT)
+
+#define PDO_VAR(min_mv, max_mv, max_ma) \
+ (PDO_TYPE(PDO_TYPE_VAR) | PDO_VAR_MIN_VOLT(min_mv) | \
+ PDO_VAR_MAX_VOLT(max_mv) | PDO_VAR_MAX_CURR(max_ma))
+
+static inline enum pd_pdo_type pdo_type(u32 pdo)
+{
+ return (pdo >> PDO_TYPE_SHIFT) & PDO_TYPE_MASK;
+}
+
+static inline unsigned int pdo_fixed_voltage(u32 pdo)
+{
+ return ((pdo >> PDO_FIXED_VOLT_SHIFT) & PDO_VOLT_MASK) * 50;
+}
+
+static inline unsigned int pdo_min_voltage(u32 pdo)
+{
+ return ((pdo >> PDO_VAR_MIN_VOLT_SHIFT) & PDO_VOLT_MASK) * 50;
+}
+
+static inline unsigned int pdo_max_voltage(u32 pdo)
+{
+ return ((pdo >> PDO_VAR_MAX_VOLT_SHIFT) & PDO_VOLT_MASK) * 50;
+}
+
+static inline unsigned int pdo_max_current(u32 pdo)
+{
+ return ((pdo >> PDO_VAR_MAX_CURR_SHIFT) & PDO_CURR_MASK) * 10;
+}
+
+static inline unsigned int pdo_max_power(u32 pdo)
+{
+ return ((pdo >> PDO_BATT_MAX_PWR_SHIFT) & PDO_PWR_MASK) * 250;
+}
+
+/* RDO: Request Data Object */
+#define RDO_OBJ_POS_SHIFT 28
+#define RDO_OBJ_POS_MASK 0x7
+#define RDO_GIVE_BACK BIT(27) /* Supports reduced operating current */
+#define RDO_CAP_MISMATCH BIT(26) /* Not satisfied by source caps */
+#define RDO_USB_COMM BIT(25) /* USB communications capable */
+#define RDO_NO_SUSPEND BIT(24) /* USB Suspend not supported */
+
+#define RDO_PWR_MASK 0x3ff
+#define RDO_CURR_MASK 0x3ff
+
+#define RDO_FIXED_OP_CURR_SHIFT 10
+#define RDO_FIXED_MAX_CURR_SHIFT 0
+
+#define RDO_OBJ(idx) (((idx) & RDO_OBJ_POS_MASK) << RDO_OBJ_POS_SHIFT)
+
+#define PDO_FIXED_OP_CURR(ma) ((((ma) / 10) & RDO_CURR_MASK) << RDO_FIXED_OP_CURR_SHIFT)
+#define PDO_FIXED_MAX_CURR(ma) ((((ma) / 10) & RDO_CURR_MASK) << RDO_FIXED_MAX_CURR_SHIFT)
+
+#define RDO_FIXED(idx, op_ma, max_ma, flags) \
+ (RDO_OBJ(idx) | (flags) | \
+ PDO_FIXED_OP_CURR(op_ma) | PDO_FIXED_MAX_CURR(max_ma))
+
+#define RDO_BATT_OP_PWR_SHIFT 10 /* 250mW units */
+#define RDO_BATT_MAX_PWR_SHIFT 0 /* 250mW units */
+
+#define RDO_BATT_OP_PWR(mw) ((((mw) / 250) & RDO_PWR_MASK) << RDO_BATT_OP_PWR_SHIFT)
+#define RDO_BATT_MAX_PWR(mw) ((((mw) / 250) & RDO_PWR_MASK) << RDO_BATT_MAX_PWR_SHIFT)
+
+#define RDO_BATT(idx, op_mw, max_mw, flags) \
+ (RDO_OBJ(idx) | (flags) | \
+ RDO_BATT_OP_PWR(op_mw) | RDO_BATT_MAX_PWR(max_mw))
+
+static inline unsigned int rdo_index(u32 rdo)
+{
+ return (rdo >> RDO_OBJ_POS_SHIFT) & RDO_OBJ_POS_MASK;
+}
+
+static inline unsigned int rdo_op_current(u32 rdo)
+{
+ return ((rdo >> RDO_FIXED_OP_CURR_SHIFT) & RDO_CURR_MASK) * 10;
+}
+
+static inline unsigned int rdo_max_current(u32 rdo)
+{
+ return ((rdo >> RDO_FIXED_MAX_CURR_SHIFT) &
+ RDO_CURR_MASK) * 10;
+}
+
+static inline unsigned int rdo_op_power(u32 rdo)
+{
+ return ((rdo >> RDO_BATT_OP_PWR_SHIFT) & RDO_PWR_MASK) * 250;
+}
+
+static inline unsigned int rdo_max_power(u32 rdo)
+{
+ return ((rdo >> RDO_BATT_MAX_PWR_SHIFT) & RDO_PWR_MASK) * 250;
+}
+
+/* USB PD timers and counters */
+#define PD_T_NO_RESPONSE 5000 /* 4.5 - 5.5 seconds */
+#define PD_T_DB_DETECT 10000 /* 10 - 15 seconds */
+#define PD_T_SEND_SOURCE_CAP 150 /* 100 - 200 ms */
+#define PD_T_SENDER_RESPONSE 60 /* 24 - 30 ms, relaxed */
+#define PD_T_SOURCE_ACTIVITY 45
+#define PD_T_SINK_ACTIVITY 135
+#define PD_T_SINK_WAIT_CAP 310
+#define PD_T_PS_TRANSITION 500
+#define PD_T_SRC_TRANSITION 35
+#define PD_T_DRP_SNK 40
+#define PD_T_DRP_SRC 30
+#define PD_T_PS_SOURCE_OFF 920
+#define PD_T_PS_SOURCE_ON 480
+#define PD_T_PS_HARD_RESET 30
+#define PD_T_SRC_RECOVER 760
+#define PD_T_SRC_RECOVER_MAX 1000
+#define PD_T_SRC_TURN_ON 275
+#define PD_T_SAFE_0V 650
+#define PD_T_VCONN_SOURCE_ON 100
+#define PD_T_SINK_REQUEST 100 /* 100 ms minimum */
+#define PD_T_ERROR_RECOVERY 100 /* minimum 25 is insufficient */
+#define PD_T_SRCSWAPSTDBY 625 /* Maximum of 650ms */
+#define PD_T_NEWSRC 250 /* Maximum of 275ms */
+
+#define PD_T_DRP_TRY 100 /* 75 - 150 ms */
+#define PD_T_DRP_TRYWAIT 600 /* 400 - 800 ms */
+
+#define PD_T_CC_DEBOUNCE 200 /* 100 - 200 ms */
+#define PD_T_PD_DEBOUNCE 20 /* 10 - 20 ms */
+
+#define PD_N_CAPS_COUNT (PD_T_NO_RESPONSE / PD_T_SEND_SOURCE_CAP)
+#define PD_N_HARD_RESET_COUNT 2
+
+#define PD_P_SNKSTDBY 2500 /* in mw */
+#endif /* __LINUX_USB_PD_H */
diff --git a/include/linux/usb/pd_bdo.h b/include/linux/usb/pd_bdo.h
new file mode 100644
index 0000000..90b94d9
--- /dev/null
+++ b/include/linux/usb/pd_bdo.h
@@ -0,0 +1,31 @@
+/*
+ * Copyright 2015-2017 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __LINUX_USB_PD_BDO_H
+#define __LINUX_USB_PD_BDO_H
+
+/* BDO : BIST Data Object */
+#define BDO_MODE_RECV (0 << 28)
+#define BDO_MODE_TRANSMIT (1 << 28)
+#define BDO_MODE_COUNTERS (2 << 28)
+#define BDO_MODE_CARRIER0 (3 << 28)
+#define BDO_MODE_CARRIER1 (4 << 28)
+#define BDO_MODE_CARRIER2 (5 << 28)
+#define BDO_MODE_CARRIER3 (6 << 28)
+#define BDO_MODE_EYE (7 << 28)
+#define BDO_MODE_TESTDATA (8 << 28)
+
+#define BDO_MODE_MASK(mode) ((mode) & 0xf0000000)
+
+#endif
diff --git a/include/linux/usb/pd_vdo.h b/include/linux/usb/pd_vdo.h
new file mode 100644
index 0000000..d92259f
--- /dev/null
+++ b/include/linux/usb/pd_vdo.h
@@ -0,0 +1,251 @@
+/*
+ * Copyright 2015-2017 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __LINUX_USB_PD_VDO_H
+#define __LINUX_USB_PD_VDO_H
+
+#include "pd.h"
+
+/*
+ * VDO : Vendor Defined Message Object
+ * VDM object is minimum of VDM header + 6 additional data objects.
+ */
+
+#define VDO_MAX_OBJECTS 6
+#define VDO_MAX_SIZE (VDO_MAX_OBJECTS + 1)
+
+/*
+ * VDM header
+ * ----------
+ * <31:16> :: SVID
+ * <15> :: VDM type ( 1b == structured, 0b == unstructured )
+ * <14:13> :: Structured VDM version (can only be 00 == 1.0 currently)
+ * <12:11> :: reserved
+ * <10:8> :: object position (1-7 valid ... used for enter/exit mode only)
+ * <7:6> :: command type (SVDM only?)
+ * <5> :: reserved (SVDM), command type (UVDM)
+ * <4:0> :: command
+ */
+#define VDO(vid, type, custom) \
+ (((vid) << 16) | \
+ ((type) << 15) | \
+ ((custom) & 0x7FFF))
+
+#define VDO_SVDM_TYPE (1 << 15)
+#define VDO_SVDM_VERS(x) ((x) << 13)
+#define VDO_OPOS(x) ((x) << 8)
+#define VDO_CMDT(x) ((x) << 6)
+#define VDO_OPOS_MASK VDO_OPOS(0x7)
+#define VDO_CMDT_MASK VDO_CMDT(0x3)
+
+#define CMDT_INIT 0
+#define CMDT_RSP_ACK 1
+#define CMDT_RSP_NAK 2
+#define CMDT_RSP_BUSY 3
+
+/* reserved for SVDM ... for Google UVDM */
+#define VDO_SRC_INITIATOR (0 << 5)
+#define VDO_SRC_RESPONDER (1 << 5)
+
+#define CMD_DISCOVER_IDENT 1
+#define CMD_DISCOVER_SVID 2
+#define CMD_DISCOVER_MODES 3
+#define CMD_ENTER_MODE 4
+#define CMD_EXIT_MODE 5
+#define CMD_ATTENTION 6
+
+#define VDO_CMD_VENDOR(x) (((10 + (x)) & 0x1f))
+
+/* ChromeOS specific commands */
+#define VDO_CMD_VERSION VDO_CMD_VENDOR(0)
+#define VDO_CMD_SEND_INFO VDO_CMD_VENDOR(1)
+#define VDO_CMD_READ_INFO VDO_CMD_VENDOR(2)
+#define VDO_CMD_REBOOT VDO_CMD_VENDOR(5)
+#define VDO_CMD_FLASH_ERASE VDO_CMD_VENDOR(6)
+#define VDO_CMD_FLASH_WRITE VDO_CMD_VENDOR(7)
+#define VDO_CMD_ERASE_SIG VDO_CMD_VENDOR(8)
+#define VDO_CMD_PING_ENABLE VDO_CMD_VENDOR(10)
+#define VDO_CMD_CURRENT VDO_CMD_VENDOR(11)
+#define VDO_CMD_FLIP VDO_CMD_VENDOR(12)
+#define VDO_CMD_GET_LOG VDO_CMD_VENDOR(13)
+#define VDO_CMD_CCD_EN VDO_CMD_VENDOR(14)
+
+#define PD_VDO_VID(vdo) ((vdo) >> 16)
+#define PD_VDO_SVDM(vdo) (((vdo) >> 15) & 1)
+#define PD_VDO_OPOS(vdo) (((vdo) >> 8) & 0x7)
+#define PD_VDO_CMD(vdo) ((vdo) & 0x1f)
+#define PD_VDO_CMDT(vdo) (((vdo) >> 6) & 0x3)
+
+/*
+ * SVDM Identity request -> response
+ *
+ * Request is simply properly formatted SVDM header
+ *
+ * Response is 4 data objects:
+ * [0] :: SVDM header
+ * [1] :: Identitiy header
+ * [2] :: Cert Stat VDO
+ * [3] :: (Product | Cable) VDO
+ * [4] :: AMA VDO
+ *
+ */
+#define VDO_INDEX_HDR 0
+#define VDO_INDEX_IDH 1
+#define VDO_INDEX_CSTAT 2
+#define VDO_INDEX_CABLE 3
+#define VDO_INDEX_PRODUCT 3
+#define VDO_INDEX_AMA 4
+
+/*
+ * SVDM Identity Header
+ * --------------------
+ * <31> :: data capable as a USB host
+ * <30> :: data capable as a USB device
+ * <29:27> :: product type
+ * <26> :: modal operation supported (1b == yes)
+ * <25:16> :: Reserved, Shall be set to zero
+ * <15:0> :: USB-IF assigned VID for this cable vendor
+ */
+#define IDH_PTYPE_UNDEF 0
+#define IDH_PTYPE_HUB 1
+#define IDH_PTYPE_PERIPH 2
+#define IDH_PTYPE_PCABLE 3
+#define IDH_PTYPE_ACABLE 4
+#define IDH_PTYPE_AMA 5
+
+#define VDO_IDH(usbh, usbd, ptype, is_modal, vid) \
+ ((usbh) << 31 | (usbd) << 30 | ((ptype) & 0x7) << 27 \
+ | (is_modal) << 26 | ((vid) & 0xffff))
+
+#define PD_IDH_PTYPE(vdo) (((vdo) >> 27) & 0x7)
+#define PD_IDH_VID(vdo) ((vdo) & 0xffff)
+#define PD_IDH_MODAL_SUPP(vdo) ((vdo) & (1 << 26))
+
+/*
+ * Cert Stat VDO
+ * -------------
+ * <31:0> : USB-IF assigned XID for this cable
+ */
+#define PD_CSTAT_XID(vdo) (vdo)
+
+/*
+ * Product VDO
+ * -----------
+ * <31:16> : USB Product ID
+ * <15:0> : USB bcdDevice
+ */
+#define VDO_PRODUCT(pid, bcd) (((pid) & 0xffff) << 16 | ((bcd) & 0xffff))
+#define PD_PRODUCT_PID(vdo) (((vdo) >> 16) & 0xffff)
+
+/*
+ * Cable VDO
+ * ---------
+ * <31:28> :: Cable HW version
+ * <27:24> :: Cable FW version
+ * <23:20> :: Reserved, Shall be set to zero
+ * <19:18> :: type-C to Type-A/B/C (00b == A, 01 == B, 10 == C)
+ * <17> :: Type-C to Plug/Receptacle (0b == plug, 1b == receptacle)
+ * <16:13> :: cable latency (0001 == <10ns(~1m length))
+ * <12:11> :: cable termination type (11b == both ends active VCONN req)
+ * <10> :: SSTX1 Directionality support (0b == fixed, 1b == cfgable)
+ * <9> :: SSTX2 Directionality support
+ * <8> :: SSRX1 Directionality support
+ * <7> :: SSRX2 Directionality support
+ * <6:5> :: Vbus current handling capability
+ * <4> :: Vbus through cable (0b == no, 1b == yes)
+ * <3> :: SOP" controller present? (0b == no, 1b == yes)
+ * <2:0> :: USB SS Signaling support
+ */
+#define CABLE_ATYPE 0
+#define CABLE_BTYPE 1
+#define CABLE_CTYPE 2
+#define CABLE_PLUG 0
+#define CABLE_RECEPTACLE 1
+#define CABLE_CURR_1A5 0
+#define CABLE_CURR_3A 1
+#define CABLE_CURR_5A 2
+#define CABLE_USBSS_U2_ONLY 0
+#define CABLE_USBSS_U31_GEN1 1
+#define CABLE_USBSS_U31_GEN2 2
+#define VDO_CABLE(hw, fw, cbl, gdr, lat, term, tx1d, tx2d, rx1d, rx2d, cur,\
+ vps, sopp, usbss) \
+ (((hw) & 0x7) << 28 | ((fw) & 0x7) << 24 | ((cbl) & 0x3) << 18 \
+ | (gdr) << 17 | ((lat) & 0x7) << 13 | ((term) & 0x3) << 11 \
+ | (tx1d) << 10 | (tx2d) << 9 | (rx1d) << 8 | (rx2d) << 7 \
+ | ((cur) & 0x3) << 5 | (vps) << 4 | (sopp) << 3 \
+ | ((usbss) & 0x7))
+
+/*
+ * AMA VDO
+ * ---------
+ * <31:28> :: Cable HW version
+ * <27:24> :: Cable FW version
+ * <23:12> :: Reserved, Shall be set to zero
+ * <11> :: SSTX1 Directionality support (0b == fixed, 1b == cfgable)
+ * <10> :: SSTX2 Directionality support
+ * <9> :: SSRX1 Directionality support
+ * <8> :: SSRX2 Directionality support
+ * <7:5> :: Vconn power
+ * <4> :: Vconn power required
+ * <3> :: Vbus power required
+ * <2:0> :: USB SS Signaling support
+ */
+#define VDO_AMA(hw, fw, tx1d, tx2d, rx1d, rx2d, vcpwr, vcr, vbr, usbss) \
+ (((hw) & 0x7) << 28 | ((fw) & 0x7) << 24 \
+ | (tx1d) << 11 | (tx2d) << 10 | (rx1d) << 9 | (rx2d) << 8 \
+ | ((vcpwr) & 0x7) << 5 | (vcr) << 4 | (vbr) << 3 \
+ | ((usbss) & 0x7))
+
+#define PD_VDO_AMA_VCONN_REQ(vdo) (((vdo) >> 4) & 1)
+#define PD_VDO_AMA_VBUS_REQ(vdo) (((vdo) >> 3) & 1)
+
+#define AMA_VCONN_PWR_1W 0
+#define AMA_VCONN_PWR_1W5 1
+#define AMA_VCONN_PWR_2W 2
+#define AMA_VCONN_PWR_3W 3
+#define AMA_VCONN_PWR_4W 4
+#define AMA_VCONN_PWR_5W 5
+#define AMA_VCONN_PWR_6W 6
+#define AMA_USBSS_U2_ONLY 0
+#define AMA_USBSS_U31_GEN1 1
+#define AMA_USBSS_U31_GEN2 2
+#define AMA_USBSS_BBONLY 3
+
+/*
+ * SVDM Discover SVIDs request -> response
+ *
+ * Request is properly formatted VDM Header with discover SVIDs command.
+ * Response is a set of SVIDs of all all supported SVIDs with all zero's to
+ * mark the end of SVIDs. If more than 12 SVIDs are supported command SHOULD be
+ * repeated.
+ */
+#define VDO_SVID(svid0, svid1) (((svid0) & 0xffff) << 16 | ((svid1) & 0xffff))
+#define PD_VDO_SVID_SVID0(vdo) ((vdo) >> 16)
+#define PD_VDO_SVID_SVID1(vdo) ((vdo) & 0xffff)
+
+/* USB-IF SIDs */
+#define USB_SID_PD 0xff00 /* power delivery */
+#define USB_SID_DISPLAYPORT 0xff01
+#define USB_SID_MHL 0xff02 /* Mobile High-Definition Link */
+
+/* VDM command timeouts (in ms) */
+
+#define PD_T_VDM_UNSTRUCTURED 500
+#define PD_T_VDM_BUSY 100
+#define PD_T_VDM_WAIT_MODE_E 100
+#define PD_T_VDM_SNDR_RSP 30
+#define PD_T_VDM_E_MODE 25
+#define PD_T_VDM_RCVR_RSP 15
+
+#endif /* __LINUX_USB_PD_VDO_H */
diff --git a/include/linux/usb/tcpm.h b/include/linux/usb/tcpm.h
new file mode 100644
index 0000000..b30483d
--- /dev/null
+++ b/include/linux/usb/tcpm.h
@@ -0,0 +1,222 @@
+/*
+ * Copyright 2015-2017 Google, Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef __LINUX_USB_TCPM_H
+#define __LINUX_USB_TCPM_H
+
+#include <linux/bitops.h>
+#include <linux/usb/typec.h>
+#include "pd.h"
+
+enum typec_cc_status {
+ TYPEC_CC_OPEN,
+ TYPEC_CC_RA,
+ TYPEC_CC_RD,
+ TYPEC_CC_RP_DEF,
+ TYPEC_CC_RP_1_5,
+ TYPEC_CC_RP_3_0,
+};
+
+enum typec_cc_polarity {
+ TYPEC_POLARITY_CC1,
+ TYPEC_POLARITY_CC2,
+};
+
+/* Time to wait for TCPC to complete transmit */
+#define PD_T_TCPC_TX_TIMEOUT 100 /* in ms */
+#define PD_ROLE_SWAP_TIMEOUT (MSEC_PER_SEC * 10)
+
+enum tcpm_transmit_status {
+ TCPC_TX_SUCCESS = 0,
+ TCPC_TX_DISCARDED = 1,
+ TCPC_TX_FAILED = 2,
+};
+
+enum tcpm_transmit_type {
+ TCPC_TX_SOP = 0,
+ TCPC_TX_SOP_PRIME = 1,
+ TCPC_TX_SOP_PRIME_PRIME = 2,
+ TCPC_TX_SOP_DEBUG_PRIME = 3,
+ TCPC_TX_SOP_DEBUG_PRIME_PRIME = 4,
+ TCPC_TX_HARD_RESET = 5,
+ TCPC_TX_CABLE_RESET = 6,
+ TCPC_TX_BIST_MODE_2 = 7
+};
+
+/**
+ * struct tcpc_config - Port configuration
+ * @src_pdo: PDO parameters sent to port partner as response to
+ * PD_CTRL_GET_SOURCE_CAP message
+ * @nr_src_pdo: Number of entries in @src_pdo
+ * @snk_pdo: PDO parameters sent to partner as response to
+ * PD_CTRL_GET_SINK_CAP message
+ * @nr_snk_pdo: Number of entries in @snk_pdo
+ * @max_snk_mv: Maximum acceptable sink voltage in mV
+ * @max_snk_ma: Maximum sink current in mA
+ * @max_snk_mw: Maximum required sink power in mW
+ * @operating_snk_mw:
+ * Required operating sink power in mW
+ * @type: Port type (TYPEC_PORT_DFP, TYPEC_PORT_UFP, or
+ * TYPEC_PORT_DRP)
+ * @default_role:
+ * Default port role (TYPEC_SINK or TYPEC_SOURCE).
+ * Set to TYPEC_NO_PREFERRED_ROLE if no default role.
+ * @try_role_hw:True if try.{Src,Snk} is implemented in hardware
+ * @alt_modes: List of supported alternate modes
+ */
+struct tcpc_config {
+ u32 *src_pdo;
+ unsigned int nr_src_pdo;
+
+ u32 *snk_pdo;
+ unsigned int nr_snk_pdo;
+
+ const u32 *snk_vdo;
+ unsigned int nr_snk_vdo;
+
+ unsigned int max_snk_mv;
+ unsigned int max_snk_ma;
+ unsigned int max_snk_mw;
+ unsigned int operating_snk_mw;
+
+ enum typec_port_type type;
+ enum typec_role default_role;
+ bool try_role_hw; /* try.{src,snk} implemented in hardware */
+ bool self_powered; /* port belongs to a self powered device */
+
+ const struct typec_altmode_desc *alt_modes;
+};
+
+enum tcpc_usb_switch {
+ TCPC_USB_SWITCH_CONNECT,
+ TCPC_USB_SWITCH_DISCONNECT,
+};
+
+/* Mux state attributes */
+#define TCPC_MUX_USB_ENABLED BIT(0) /* USB enabled */
+#define TCPC_MUX_DP_ENABLED BIT(1) /* DP enabled */
+#define TCPC_MUX_POLARITY_INVERTED BIT(2) /* Polarity inverted */
+
+/* Mux modes, decoded to attributes */
+enum tcpc_mux_mode {
+ TYPEC_MUX_NONE = 0, /* Open switch */
+ TYPEC_MUX_USB = TCPC_MUX_USB_ENABLED, /* USB only */
+ TYPEC_MUX_DP = TCPC_MUX_DP_ENABLED, /* DP only */
+ TYPEC_MUX_DOCK = TCPC_MUX_USB_ENABLED | /* Both USB and DP */
+ TCPC_MUX_DP_ENABLED,
+};
+
+struct tcpc_mux_dev {
+ int (*set)(struct tcpc_mux_dev *dev, enum tcpc_mux_mode mux_mode,
+ enum tcpc_usb_switch usb_config,
+ enum typec_cc_polarity polarity);
+ bool dfp_only;
+ void *priv_data;
+};
+
+/**
+ * struct tcpc_dev - Port configuration and callback functions
+ * @config: Pointer to port configuration
+ * @get_vbus: Called to read current VBUS state
+ * @get_current_limit:
+ * Optional; called by the tcpm core when configured as a snk
+ * and cc=Rp-def. This allows the tcpm to provide a fallback
+ * current-limit detection method for the cc=Rp-def case.
+ * For example, some tcpcs may include BC1.2 charger detection
+ * and use that in this case.
+ * @set_cc: Called to set value of CC pins
+ * @get_cc: Called to read current CC pin values
+ * @set_polarity:
+ * Called to set polarity
+ * @set_vconn: Called to enable or disable VCONN
+ * @set_vbus: Called to enable or disable VBUS
+ * @set_current_limit:
+ * Optional; called to set current limit as negotiated
+ * with partner.
+ * @set_pd_rx: Called to enable or disable reception of PD messages
+ * @set_roles: Called to set power and data roles
+ * @start_drp_toggling:
+ * Optional; if supported by hardware, called to start DRP
+ * toggling. DRP toggling is stopped automatically if
+ * a connection is established.
+ * @try_role: Optional; called to set a preferred role
+ * @pd_transmit:Called to transmit PD message
+ * @set_pd_capable:
+ * Optional; Called to notify that pd capable partner has been
+ * detected.
+ * @set_in_hard_reset:
+ * Optional; Called to notify that hard reset is in progress.
+ * @in_pd_contract:
+ * Optional; Called to notify is when pd contract is established.
+ * @mux: Pointer to multiplexer data
+ */
+struct tcpc_dev {
+ struct tcpc_config *config;
+
+ int (*init)(struct tcpc_dev *dev);
+ int (*get_vbus)(struct tcpc_dev *dev);
+ int (*get_current_limit)(struct tcpc_dev *dev);
+ int (*set_cc)(struct tcpc_dev *dev, enum typec_cc_status cc);
+ int (*get_cc)(struct tcpc_dev *dev, enum typec_cc_status *cc1,
+ enum typec_cc_status *cc2);
+ int (*set_polarity)(struct tcpc_dev *dev,
+ enum typec_cc_polarity polarity);
+ int (*set_vconn)(struct tcpc_dev *dev, bool on);
+ int (*set_vbus)(struct tcpc_dev *dev, bool on, bool charge);
+ int (*set_current_limit)(struct tcpc_dev *dev, u32 max_ma, u32 min_mv,
+ u32 max_mv);
+ int (*set_pd_rx)(struct tcpc_dev *dev, bool on);
+ int (*set_roles)(struct tcpc_dev *dev, bool attached,
+ enum typec_role role, enum typec_data_role data,
+ bool usb_comm_capable);
+ int (*start_drp_toggling)(struct tcpc_dev *dev,
+ enum typec_cc_status cc);
+ int (*try_role)(struct tcpc_dev *dev, int role);
+ int (*pd_transmit)(struct tcpc_dev *dev, enum tcpm_transmit_type type,
+ const struct pd_message *msg);
+ int (*set_in_pr_swap)(struct tcpc_dev *dev, bool pr_swap);
+ void (*set_pd_capable)(struct tcpc_dev *dev, bool capable);
+ void (*set_in_hard_reset)(struct tcpc_dev *dev, bool status);
+ void (*log_rtc)(struct tcpc_dev *dev);
+ int (*set_suspend_supported)(struct tcpc_dev *dev,
+ bool suspend_supported);
+ int (*in_pd_contract)(struct tcpc_dev *dev, bool status);
+ struct tcpc_mux_dev *mux;
+};
+
+struct tcpm_port;
+
+struct tcpm_port *tcpm_register_port(struct device *dev, struct tcpc_dev *tcpc);
+void tcpm_unregister_port(struct tcpm_port *port);
+
+int tcpm_update_source_capabilities(struct tcpm_port *port, const u32 *pdo,
+ unsigned int nr_pdo);
+int tcpm_update_sink_capabilities(struct tcpm_port *port, const u32 *pdo,
+ unsigned int nr_pdo,
+ unsigned int max_snk_mv,
+ unsigned int max_snk_ma,
+ unsigned int max_snk_mw,
+ unsigned int operating_snk_mw);
+
+void tcpm_vbus_change(struct tcpm_port *port);
+void tcpm_cc_change(struct tcpm_port *port);
+void tcpm_pd_receive(struct tcpm_port *port,
+ const struct pd_message *msg);
+void tcpm_pd_transmit_complete(struct tcpm_port *port,
+ enum tcpm_transmit_status status);
+void tcpm_pd_hard_reset(struct tcpm_port *port);
+void tcpm_tcpc_reset(struct tcpm_port *port);
+void tcpm_port_reset(struct tcpm_port *port);
+
+#endif /* __LINUX_USB_TCPM_H */
diff --git a/include/linux/usb/typec.h b/include/linux/usb/typec.h
new file mode 100644
index 0000000..ffe7487
--- /dev/null
+++ b/include/linux/usb/typec.h
@@ -0,0 +1,247 @@
+
+#ifndef __LINUX_USB_TYPEC_H
+#define __LINUX_USB_TYPEC_H
+
+#include <linux/types.h>
+
+/* XXX: Once we have a header for USB Power Delivery, this belongs there */
+#define ALTMODE_MAX_MODES 6
+
+/* USB Type-C Specification releases */
+#define USB_TYPEC_REV_1_0 0x100 /* 1.0 */
+#define USB_TYPEC_REV_1_1 0x110 /* 1.1 */
+#define USB_TYPEC_REV_1_2 0x120 /* 1.2 */
+
+struct typec_altmode;
+struct typec_partner;
+struct typec_cable;
+struct typec_plug;
+struct typec_port;
+
+struct fwnode_handle;
+
+enum typec_port_type {
+ TYPEC_PORT_DFP,
+ TYPEC_PORT_UFP,
+ TYPEC_PORT_DRP,
+};
+
+enum typec_plug_type {
+ USB_PLUG_NONE,
+ USB_PLUG_TYPE_A,
+ USB_PLUG_TYPE_B,
+ USB_PLUG_TYPE_C,
+ USB_PLUG_CAPTIVE,
+};
+
+enum typec_data_role {
+ TYPEC_DEVICE,
+ TYPEC_HOST,
+};
+
+enum typec_role {
+ TYPEC_SINK,
+ TYPEC_SOURCE,
+};
+
+enum typec_pwr_opmode {
+ TYPEC_PWR_MODE_USB,
+ TYPEC_PWR_MODE_1_5A,
+ TYPEC_PWR_MODE_3_0A,
+ TYPEC_PWR_MODE_PD,
+};
+
+enum typec_accessory {
+ TYPEC_ACCESSORY_NONE,
+ TYPEC_ACCESSORY_AUDIO,
+ TYPEC_ACCESSORY_DEBUG,
+};
+
+#define TYPEC_MAX_ACCESSORY 3
+
+/*
+ * struct usb_pd_identity - USB Power Delivery identity data
+ * @id_header: ID Header VDO
+ * @cert_stat: Cert Stat VDO
+ * @product: Product VDO
+ *
+ * USB power delivery Discover Identity command response data.
+ *
+ * REVISIT: This is USB Power Delivery specific information, so this structure
+ * probable belongs to USB Power Delivery header file once we have them.
+ */
+struct usb_pd_identity {
+ u32 id_header;
+ u32 cert_stat;
+ u32 product;
+};
+
+int typec_partner_set_identity(struct typec_partner *partner);
+int typec_cable_set_identity(struct typec_cable *cable);
+
+/*
+ * struct typec_mode_desc - Individual Mode of an Alternate Mode
+ * @index: Index of the Mode within the SVID
+ * @vdo: VDO returned by Discover Modes USB PD command
+ * @desc: Optional human readable description of the mode
+ * @roles: Only for ports. DRP if the mode is available in both roles
+ *
+ * Description of a mode of an Alternate Mode which a connector, cable plug or
+ * partner supports. Every mode will have it's own sysfs group. The details are
+ * the VDO returned by discover modes command, description for the mode and
+ * active flag telling has the mode being entered or not.
+ */
+struct typec_mode_desc {
+ int index;
+ u32 vdo;
+ char *desc;
+ /* Only used with ports */
+ enum typec_port_type roles;
+};
+
+/*
+ * struct typec_altmode_desc - USB Type-C Alternate Mode Descriptor
+ * @svid: Standard or Vendor ID
+ * @n_modes: Number of modes
+ * @modes: Array of modes supported by the Alternate Mode
+ *
+ * Representation of an Alternate Mode that has SVID assigned by USB-IF. The
+ * array of modes will list the modes of a particular SVID that are supported by
+ * a connector, partner of a cable plug.
+ */
+struct typec_altmode_desc {
+ u16 svid;
+ int n_modes;
+ struct typec_mode_desc modes[ALTMODE_MAX_MODES];
+};
+
+struct typec_altmode
+*typec_partner_register_altmode(struct typec_partner *partner,
+ const struct typec_altmode_desc *desc);
+struct typec_altmode
+*typec_plug_register_altmode(struct typec_plug *plug,
+ const struct typec_altmode_desc *desc);
+struct typec_altmode
+*typec_port_register_altmode(struct typec_port *port,
+ const struct typec_altmode_desc *desc);
+void typec_unregister_altmode(struct typec_altmode *altmode);
+
+struct typec_port *typec_altmode2port(struct typec_altmode *alt);
+
+void typec_altmode_update_active(struct typec_altmode *alt, int mode,
+ bool active);
+
+enum typec_plug_index {
+ TYPEC_PLUG_SOP_P,
+ TYPEC_PLUG_SOP_PP,
+};
+
+/*
+ * struct typec_plug_desc - USB Type-C Cable Plug Descriptor
+ * @index: SOP Prime for the plug connected to DFP and SOP Double Prime for the
+ * plug connected to UFP
+ *
+ * Represents USB Type-C Cable Plug.
+ */
+struct typec_plug_desc {
+ enum typec_plug_index index;
+};
+
+/*
+ * struct typec_cable_desc - USB Type-C Cable Descriptor
+ * @type: The plug type from USB PD Cable VDO
+ * @active: Is the cable active or passive
+ * @identity: Result of Discover Identity command
+ *
+ * Represents USB Type-C Cable attached to USB Type-C port.
+ */
+struct typec_cable_desc {
+ enum typec_plug_type type;
+ unsigned int active:1;
+ struct usb_pd_identity *identity;
+};
+
+/*
+ * struct typec_partner_desc - USB Type-C Partner Descriptor
+ * @usb_pd: USB Power Delivery support
+ * @accessory: Audio, Debug or none.
+ * @identity: Discover Identity command data
+ *
+ * Details about a partner that is attached to USB Type-C port. If @identity
+ * member exists when partner is registered, a directory named "identity" is
+ * created to sysfs for the partner device.
+ */
+struct typec_partner_desc {
+ unsigned int usb_pd:1;
+ enum typec_accessory accessory;
+ struct usb_pd_identity *identity;
+};
+
+/*
+ * struct typec_capability - USB Type-C Port Capabilities
+ * @role: DFP (Host-only), UFP (Device-only) or DRP (Dual Role)
+ * @revision: USB Type-C Specification release. Binary coded decimal
+ * @pd_revision: USB Power Delivery Specification revision if supported
+ * @prefer_role: Initial role preference
+ * @accessory: Supported Accessory Modes
+ * @fwnode: Optional fwnode of the port
+ * @try_role: Set data role preference for DRP port
+ * @dr_set: Set Data Role
+ * @pr_set: Set Power Role
+ * @vconn_set: Set VCONN Role
+ * @activate_mode: Enter/exit given Alternate Mode
+ * @port_type_set: Set port type
+ *
+ * Static capabilities of a single USB Type-C port.
+ */
+struct typec_capability {
+ enum typec_port_type type;
+ u16 revision; /* 0120H = "1.2" */
+ u16 pd_revision; /* 0300H = "3.0" */
+ int prefer_role;
+ enum typec_accessory accessory[TYPEC_MAX_ACCESSORY];
+
+ struct fwnode_handle *fwnode;
+
+ int (*try_role)(const struct typec_capability *,
+ int role);
+
+ int (*dr_set)(const struct typec_capability *,
+ enum typec_data_role);
+ int (*pr_set)(const struct typec_capability *,
+ enum typec_role);
+ int (*vconn_set)(const struct typec_capability *,
+ enum typec_role);
+
+ int (*activate_mode)(const struct typec_capability *,
+ int mode, int activate);
+ int (*port_type_set)(const struct typec_capability *,
+ enum typec_port_type);
+
+};
+
+/* Specific to try_role(). Indicates the user want's to clear the preference. */
+#define TYPEC_NO_PREFERRED_ROLE (-1)
+
+struct typec_port *typec_register_port(struct device *parent,
+ const struct typec_capability *cap);
+void typec_unregister_port(struct typec_port *port);
+
+struct typec_partner *typec_register_partner(struct typec_port *port,
+ struct typec_partner_desc *desc);
+void typec_unregister_partner(struct typec_partner *partner);
+
+struct typec_cable *typec_register_cable(struct typec_port *port,
+ struct typec_cable_desc *desc);
+void typec_unregister_cable(struct typec_cable *cable);
+
+struct typec_plug *typec_register_plug(struct typec_cable *cable,
+ struct typec_plug_desc *desc);
+void typec_unregister_plug(struct typec_plug *plug);
+
+void typec_set_data_role(struct typec_port *port, enum typec_data_role role);
+void typec_set_pwr_role(struct typec_port *port, enum typec_role role);
+void typec_set_vconn_role(struct typec_port *port, enum typec_role role);
+void typec_set_pwr_opmode(struct typec_port *port, enum typec_pwr_opmode mode);
+
+#endif /* __LINUX_USB_TYPEC_H */
diff --git a/include/linux/verification.h b/include/linux/verification.h
index a10549a..cfa4730 100644
--- a/include/linux/verification.h
+++ b/include/linux/verification.h
@@ -13,6 +13,12 @@
#define _LINUX_VERIFICATION_H
/*
+ * Indicate that both builtin trusted keys and secondary trusted keys
+ * should be used.
+ */
+#define VERIFY_USE_SECONDARY_KEYRING ((struct key *)1UL)
+
+/*
* The use to which an asymmetric key is being put.
*/
enum key_being_used_for {
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index d5eb547..3f8f3505 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -143,6 +143,9 @@ int virtio_device_freeze(struct virtio_device *dev);
int virtio_device_restore(struct virtio_device *dev);
#endif
+#define virtio_device_for_each_vq(vdev, vq) \
+ list_for_each_entry(vq, &vdev->vqs, list)
+
/**
* virtio_driver - operations for a virtio I/O driver
* @driver: underlying device driver (populate name and owner).
diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h
index a9c2e4c..9febfd2 100644
--- a/include/linux/vm_event_item.h
+++ b/include/linux/vm_event_item.h
@@ -97,7 +97,6 @@ enum vm_event_item { PGPGIN, PGPGOUT, PGPGOUTCLEAN, PSWPIN, PSWPOUT,
#ifdef CONFIG_DEBUG_VM_VMACACHE
VMACACHE_FIND_CALLS,
VMACACHE_FIND_HITS,
- VMACACHE_FULL_FLUSHES,
#endif
NR_VM_EVENT_ITEMS
};
diff --git a/include/linux/vmacache.h b/include/linux/vmacache.h
index c3fa0fd4..4f58ff2 100644
--- a/include/linux/vmacache.h
+++ b/include/linux/vmacache.h
@@ -15,7 +15,6 @@ static inline void vmacache_flush(struct task_struct *tsk)
memset(tsk->vmacache, 0, sizeof(tsk->vmacache));
}
-extern void vmacache_flush_all(struct mm_struct *mm);
extern void vmacache_update(unsigned long addr, struct vm_area_struct *newvma);
extern struct vm_area_struct *vmacache_find(struct mm_struct *mm,
unsigned long addr);
@@ -29,10 +28,6 @@ extern struct vm_area_struct *vmacache_find_exact(struct mm_struct *mm,
static inline void vmacache_invalidate(struct mm_struct *mm)
{
mm->vmacache_seqnum++;
-
- /* deal with overflows */
- if (unlikely(mm->vmacache_seqnum == 0))
- vmacache_flush_all(mm);
}
#endif /* __LINUX_VMACACHE_H */
diff --git a/include/linux/wakeup_reason.h b/include/linux/wakeup_reason.h
index d84d8c3..a015ce5 100644
--- a/include/linux/wakeup_reason.h
+++ b/include/linux/wakeup_reason.h
@@ -18,15 +18,87 @@
#ifndef _LINUX_WAKEUP_REASON_H
#define _LINUX_WAKEUP_REASON_H
+#include <linux/types.h>
+#include <linux/completion.h>
+
#define MAX_SUSPEND_ABORT_LEN 256
-void log_wakeup_reason(int irq);
-int check_wakeup_reason(int irq);
+struct wakeup_irq_node {
+ /* @leaf is a linked list of all leaf nodes in the interrupts trees.
+ */
+ struct list_head next;
+ /* @irq: IRQ number of this node.
+ */
+ int irq;
+ struct irq_desc *desc;
-#ifdef CONFIG_SUSPEND
-void log_suspend_abort_reason(const char *fmt, ...);
+ /* @siblings contains the list of irq nodes at the same depth; at a
+ * depth of zero, this is the list of base wakeup interrupts.
+ */
+ struct list_head siblings;
+ /* @parent: only one node in a siblings list has a pointer to the
+ * parent; that node is the head of the list of siblings.
+ */
+ struct wakeup_irq_node *parent;
+ /* @child: any node can have one child
+ */
+ struct wakeup_irq_node *child;
+ /* @handled: this flag is set to true when the interrupt handler (one of
+ * handle_.*_irq in kernel/irq/handle.c) for this node gets called; it is set
+ * to false otherwise. We use this flag to determine whether a subtree rooted
+ * at a node has been handled. When all trees rooted at
+ * base-wakeup-interrupt nodes have been handled, we stop logging
+ * potential wakeup interrupts, and construct the list of actual
+ * wakeups from the leaves of these trees.
+ */
+ bool handled;
+};
+
+#ifdef CONFIG_DEDUCE_WAKEUP_REASONS
+
+/* Called in the resume path, with interrupts and nonboot cpus disabled; on
+ * need for a spinlock.
+ */
+static inline void start_logging_wakeup_reasons(void)
+{
+ extern bool log_wakeups;
+ extern struct completion wakeups_completion;
+ ACCESS_ONCE(log_wakeups) = true;
+ init_completion(&wakeups_completion);
+}
+
+static inline bool logging_wakeup_reasons_nosync(void)
+{
+ extern bool log_wakeups;
+ return ACCESS_ONCE(log_wakeups);
+}
+
+static inline bool logging_wakeup_reasons(void)
+{
+ smp_rmb();
+ return logging_wakeup_reasons_nosync();
+}
+
+bool log_possible_wakeup_reason(int irq,
+ struct irq_desc *desc,
+ bool (*handler)(struct irq_desc *));
+
#else
-static inline void log_suspend_abort_reason(const char *fmt, ...) { }
+
+static inline void start_logging_wakeup_reasons(void) {}
+static inline bool logging_wakeup_reasons_nosync(void) { return false; }
+static inline bool logging_wakeup_reasons(void) { return false; }
+static inline bool log_possible_wakeup_reason(int irq,
+ struct irq_desc *desc,
+ bool (*handler)(struct irq_desc *)) { return true; }
+
#endif
+const struct list_head*
+get_wakeup_reasons(unsigned long timeout, struct list_head *unfinished);
+void log_base_wakeup_reason(int irq);
+void clear_wakeup_reasons(void);
+void log_suspend_abort_reason(const char *fmt, ...);
+int check_wakeup_reason(int irq);
+
#endif /* _LINUX_WAKEUP_REASON_H */
diff --git a/include/linux/workqueue.h b/include/linux/workqueue.h
index 1def337..b95c511 100644
--- a/include/linux/workqueue.h
+++ b/include/linux/workqueue.h
@@ -106,9 +106,9 @@ struct work_struct {
#endif
};
-#define WORK_DATA_INIT() ATOMIC_LONG_INIT(WORK_STRUCT_NO_POOL)
+#define WORK_DATA_INIT() ATOMIC_LONG_INIT((unsigned long)WORK_STRUCT_NO_POOL)
#define WORK_DATA_STATIC_INIT() \
- ATOMIC_LONG_INIT(WORK_STRUCT_NO_POOL | WORK_STRUCT_STATIC)
+ ATOMIC_LONG_INIT((unsigned long)(WORK_STRUCT_NO_POOL | WORK_STRUCT_STATIC))
struct delayed_work {
struct work_struct work;
@@ -359,6 +359,8 @@ extern struct workqueue_struct *system_freezable_wq;
extern struct workqueue_struct *system_power_efficient_wq;
extern struct workqueue_struct *system_freezable_power_efficient_wq;
+extern bool wq_online;
+
extern struct workqueue_struct *
__alloc_workqueue_key(const char *fmt, unsigned int flags, int max_active,
struct lock_class_key *key, const char *lock_name, ...) __printf(1, 6);
@@ -598,7 +600,7 @@ static inline bool schedule_delayed_work(struct delayed_work *dwork,
*/
static inline bool keventd_up(void)
{
- return system_wq != NULL;
+ return wq_online;
}
#ifndef CONFIG_SMP
@@ -635,4 +637,7 @@ int workqueue_online_cpu(unsigned int cpu);
int workqueue_offline_cpu(unsigned int cpu);
#endif
+int __init workqueue_init_early(void);
+int __init workqueue_init(void);
+
#endif
diff --git a/include/linux/writeback.h b/include/linux/writeback.h
index 3eed4f1..6341e78 100644
--- a/include/linux/writeback.h
+++ b/include/linux/writeback.h
@@ -363,7 +363,6 @@ void global_dirty_limits(unsigned long *pbackground, unsigned long *pdirty);
unsigned long wb_calc_thresh(struct bdi_writeback *wb, unsigned long thresh);
void wb_update_bandwidth(struct bdi_writeback *wb, unsigned long start_time);
-void page_writeback_init(void);
void balance_dirty_pages_ratelimited(struct address_space *mapping);
bool wb_over_bg_thresh(struct bdi_writeback *wb);
diff --git a/include/media/v4l2-fh.h b/include/media/v4l2-fh.h
index e19e624..d267160 100644
--- a/include/media/v4l2-fh.h
+++ b/include/media/v4l2-fh.h
@@ -42,10 +42,13 @@ struct v4l2_ctrl_handler;
* @prio: priority of the file handler, as defined by &enum v4l2_priority
*
* @wait: event' s wait queue
+ * @subscribe_lock: serialise changes to the subscribed list; guarantee that
+ * the add and del event callbacks are orderly called
* @subscribed: list of subscribed events
* @available: list of events waiting to be dequeued
* @navailable: number of available events at @available list
* @sequence: event sequence number
+ *
* @m2m_ctx: pointer to &struct v4l2_m2m_ctx
*/
struct v4l2_fh {
@@ -56,6 +59,7 @@ struct v4l2_fh {
/* Events */
wait_queue_head_t wait;
+ struct mutex subscribe_lock;
struct list_head subscribed;
struct list_head available;
unsigned int navailable;
diff --git a/include/net/af_vsock.h b/include/net/af_vsock.h
index f32ed9a..f38fe1c 100644
--- a/include/net/af_vsock.h
+++ b/include/net/af_vsock.h
@@ -62,7 +62,8 @@ struct vsock_sock {
struct list_head pending_links;
struct list_head accept_queue;
bool rejected;
- struct delayed_work dwork;
+ struct delayed_work connect_work;
+ struct delayed_work pending_work;
struct delayed_work close_work;
bool close_work_scheduled;
u32 peer_shutdown;
@@ -75,7 +76,6 @@ struct vsock_sock {
s64 vsock_stream_has_data(struct vsock_sock *vsk);
s64 vsock_stream_has_space(struct vsock_sock *vsk);
-void vsock_pending_work(struct work_struct *work);
struct sock *__vsock_create(struct net *net,
struct socket *sock,
struct sock *parent,
diff --git a/include/net/bonding.h b/include/net/bonding.h
index f32f7ef..8750c2c 100644
--- a/include/net/bonding.h
+++ b/include/net/bonding.h
@@ -139,12 +139,6 @@ struct bond_parm_tbl {
int mode;
};
-struct netdev_notify_work {
- struct delayed_work work;
- struct net_device *dev;
- struct netdev_bonding_info bonding_info;
-};
-
struct slave {
struct net_device *dev; /* first - useful for panic debug */
struct bonding *bond; /* our master */
@@ -171,6 +165,7 @@ struct slave {
#ifdef CONFIG_NET_POLL_CONTROLLER
struct netpoll *np;
#endif
+ struct delayed_work notify_work;
struct kobject kobj;
struct rtnl_link_stats64 slave_stats;
};
@@ -197,6 +192,7 @@ struct bonding {
struct slave __rcu *primary_slave;
struct bond_up_slave __rcu *slave_arr; /* Array of usable slaves */
bool force_primary;
+ u32 nest_level;
s32 slave_cnt; /* never change this value outside the attach/detach wrappers */
int (*recv_probe)(const struct sk_buff *, struct bonding *,
struct slave *);
@@ -276,6 +272,11 @@ static inline bool bond_is_lb(const struct bonding *bond)
BOND_MODE(bond) == BOND_MODE_ALB;
}
+static inline bool bond_needs_speed_duplex(const struct bonding *bond)
+{
+ return BOND_MODE(bond) == BOND_MODE_8023AD || bond_is_lb(bond);
+}
+
static inline bool bond_is_nondyn_tlb(const struct bonding *bond)
{
return (BOND_MODE(bond) == BOND_MODE_TLB) &&
diff --git a/include/net/gro_cells.h b/include/net/gro_cells.h
index 2a1abbf..95f33ee 100644
--- a/include/net/gro_cells.h
+++ b/include/net/gro_cells.h
@@ -86,6 +86,7 @@ static inline void gro_cells_destroy(struct gro_cells *gcells)
for_each_possible_cpu(i) {
struct gro_cell *cell = per_cpu_ptr(gcells->cells, i);
+ napi_disable(&cell->napi);
netif_napi_del(&cell->napi);
__skb_queue_purge(&cell->napi_skbs);
}
diff --git a/include/net/inet_frag.h b/include/net/inet_frag.h
index 634d192..a3812e9 100644
--- a/include/net/inet_frag.h
+++ b/include/net/inet_frag.h
@@ -1,14 +1,20 @@
#ifndef __NET_FRAG_H__
#define __NET_FRAG_H__
+#include <linux/rhashtable.h>
+
struct netns_frags {
- /* Keep atomic mem on separate cachelines in structs that include it */
- atomic_t mem ____cacheline_aligned_in_smp;
/* sysctls */
+ long high_thresh;
+ long low_thresh;
int timeout;
- int high_thresh;
- int low_thresh;
int max_dist;
+ struct inet_frags *f;
+
+ struct rhashtable rhashtable ____cacheline_aligned_in_smp;
+
+ /* Keep atomic mem on separate cachelines in structs that include it */
+ atomic_long_t mem ____cacheline_aligned_in_smp;
};
/**
@@ -24,130 +30,115 @@ enum {
INET_FRAG_COMPLETE = BIT(2),
};
+struct frag_v4_compare_key {
+ __be32 saddr;
+ __be32 daddr;
+ u32 user;
+ u32 vif;
+ __be16 id;
+ u16 protocol;
+};
+
+struct frag_v6_compare_key {
+ struct in6_addr saddr;
+ struct in6_addr daddr;
+ u32 user;
+ __be32 id;
+ u32 iif;
+};
+
/**
* struct inet_frag_queue - fragment queue
*
- * @lock: spinlock protecting the queue
+ * @node: rhash node
+ * @key: keys identifying this frag.
* @timer: queue expiration timer
- * @list: hash bucket list
+ * @lock: spinlock protecting this frag
* @refcnt: reference count of the queue
* @fragments: received fragments head
+ * @rb_fragments: received fragments rb-tree root
* @fragments_tail: received fragments tail
+ * @last_run_head: the head of the last "run". see ip_fragment.c
* @stamp: timestamp of the last received fragment
* @len: total length of the original datagram
* @meat: length of received fragments so far
* @flags: fragment queue flags
* @max_size: maximum received fragment size
* @net: namespace that this frag belongs to
- * @list_evictor: list of queues to forcefully evict (e.g. due to low memory)
+ * @rcu: rcu head for freeing deferall
*/
struct inet_frag_queue {
- spinlock_t lock;
+ struct rhash_head node;
+ union {
+ struct frag_v4_compare_key v4;
+ struct frag_v6_compare_key v6;
+ } key;
struct timer_list timer;
- struct hlist_node list;
+ spinlock_t lock;
atomic_t refcnt;
- struct sk_buff *fragments;
+ struct sk_buff *fragments; /* Used in IPv6. */
+ struct rb_root rb_fragments; /* Used in IPv4. */
struct sk_buff *fragments_tail;
+ struct sk_buff *last_run_head;
ktime_t stamp;
int len;
int meat;
__u8 flags;
u16 max_size;
- struct netns_frags *net;
- struct hlist_node list_evictor;
-};
-
-#define INETFRAGS_HASHSZ 1024
-
-/* averaged:
- * max_depth = default ipfrag_high_thresh / INETFRAGS_HASHSZ /
- * rounded up (SKB_TRUELEN(0) + sizeof(struct ipq or
- * struct frag_queue))
- */
-#define INETFRAGS_MAXDEPTH 128
-
-struct inet_frag_bucket {
- struct hlist_head chain;
- spinlock_t chain_lock;
+ struct netns_frags *net;
+ struct rcu_head rcu;
};
struct inet_frags {
- struct inet_frag_bucket hash[INETFRAGS_HASHSZ];
-
- struct work_struct frags_work;
- unsigned int next_bucket;
- unsigned long last_rebuild_jiffies;
- bool rebuild;
-
- /* The first call to hashfn is responsible to initialize
- * rnd. This is best done with net_get_random_once.
- *
- * rnd_seqlock is used to let hash insertion detect
- * when it needs to re-lookup the hash chain to use.
- */
- u32 rnd;
- seqlock_t rnd_seqlock;
int qsize;
- unsigned int (*hashfn)(const struct inet_frag_queue *);
- bool (*match)(const struct inet_frag_queue *q,
- const void *arg);
void (*constructor)(struct inet_frag_queue *q,
const void *arg);
void (*destructor)(struct inet_frag_queue *);
void (*frag_expire)(unsigned long data);
struct kmem_cache *frags_cachep;
const char *frags_cache_name;
+ struct rhashtable_params rhash_params;
};
int inet_frags_init(struct inet_frags *);
void inet_frags_fini(struct inet_frags *);
-static inline void inet_frags_init_net(struct netns_frags *nf)
+static inline int inet_frags_init_net(struct netns_frags *nf)
{
- atomic_set(&nf->mem, 0);
+ atomic_long_set(&nf->mem, 0);
+ return rhashtable_init(&nf->rhashtable, &nf->f->rhash_params);
}
-void inet_frags_exit_net(struct netns_frags *nf, struct inet_frags *f);
+void inet_frags_exit_net(struct netns_frags *nf);
-void inet_frag_kill(struct inet_frag_queue *q, struct inet_frags *f);
-void inet_frag_destroy(struct inet_frag_queue *q, struct inet_frags *f);
-struct inet_frag_queue *inet_frag_find(struct netns_frags *nf,
- struct inet_frags *f, void *key, unsigned int hash);
+void inet_frag_kill(struct inet_frag_queue *q);
+void inet_frag_destroy(struct inet_frag_queue *q);
+struct inet_frag_queue *inet_frag_find(struct netns_frags *nf, void *key);
-void inet_frag_maybe_warn_overflow(struct inet_frag_queue *q,
- const char *prefix);
+/* Free all skbs in the queue; return the sum of their truesizes. */
+unsigned int inet_frag_rbtree_purge(struct rb_root *root);
-static inline void inet_frag_put(struct inet_frag_queue *q, struct inet_frags *f)
+static inline void inet_frag_put(struct inet_frag_queue *q)
{
if (atomic_dec_and_test(&q->refcnt))
- inet_frag_destroy(q, f);
-}
-
-static inline bool inet_frag_evicting(struct inet_frag_queue *q)
-{
- return !hlist_unhashed(&q->list_evictor);
+ inet_frag_destroy(q);
}
/* Memory Tracking Functions. */
-static inline int frag_mem_limit(struct netns_frags *nf)
+static inline long frag_mem_limit(const struct netns_frags *nf)
{
- return atomic_read(&nf->mem);
+ return atomic_long_read(&nf->mem);
}
-static inline void sub_frag_mem_limit(struct netns_frags *nf, int i)
+static inline void sub_frag_mem_limit(struct netns_frags *nf, long val)
{
- atomic_sub(i, &nf->mem);
+ atomic_long_sub(val, &nf->mem);
}
-static inline void add_frag_mem_limit(struct netns_frags *nf, int i)
+static inline void add_frag_mem_limit(struct netns_frags *nf, long val)
{
- atomic_add(i, &nf->mem);
-}
-
-static inline int sum_frag_mem_limit(struct netns_frags *nf)
-{
- return atomic_read(&nf->mem);
+ atomic_long_add(val, &nf->mem);
}
/* RFC 3168 support :
diff --git a/include/net/inet_sock.h b/include/net/inet_sock.h
index a2f3a49..c2865f5 100644
--- a/include/net/inet_sock.h
+++ b/include/net/inet_sock.h
@@ -132,12 +132,6 @@ static inline int inet_request_bound_dev_if(const struct sock *sk,
return sk->sk_bound_dev_if;
}
-static inline struct ip_options_rcu *ireq_opt_deref(const struct inet_request_sock *ireq)
-{
- return rcu_dereference_check(ireq->ireq_opt,
- atomic_read(&ireq->req.rsk_refcnt) > 0);
-}
-
struct inet_cork {
unsigned int flags;
__be32 addr;
diff --git a/include/net/inet_timewait_sock.h b/include/net/inet_timewait_sock.h
index c9b3eb7..567017b 100644
--- a/include/net/inet_timewait_sock.h
+++ b/include/net/inet_timewait_sock.h
@@ -55,6 +55,7 @@ struct inet_timewait_sock {
#define tw_family __tw_common.skc_family
#define tw_state __tw_common.skc_state
#define tw_reuse __tw_common.skc_reuse
+#define tw_reuseport __tw_common.skc_reuseport
#define tw_ipv6only __tw_common.skc_ipv6only
#define tw_bound_dev_if __tw_common.skc_bound_dev_if
#define tw_node __tw_common.skc_nulls_node
diff --git a/include/net/ip.h b/include/net/ip.h
index ad065a4..0623529 100644
--- a/include/net/ip.h
+++ b/include/net/ip.h
@@ -307,6 +307,13 @@ int ip_decrease_ttl(struct iphdr *iph)
return --iph->ttl;
}
+static inline int ip_mtu_locked(const struct dst_entry *dst)
+{
+ const struct rtable *rt = (const struct rtable *)dst;
+
+ return rt->rt_mtu_locked || dst_metric_locked(dst, RTAX_MTU);
+}
+
static inline
int ip_dont_fragment(const struct sock *sk, const struct dst_entry *dst)
{
@@ -314,7 +321,7 @@ int ip_dont_fragment(const struct sock *sk, const struct dst_entry *dst)
return pmtudisc == IP_PMTUDISC_DO ||
(pmtudisc == IP_PMTUDISC_WANT &&
- !(dst_metric_locked(dst, RTAX_MTU)));
+ !ip_mtu_locked(dst));
}
static inline bool ip_sk_accept_pmtu(const struct sock *sk)
@@ -340,7 +347,7 @@ static inline unsigned int ip_dst_mtu_maybe_forward(const struct dst_entry *dst,
struct net *net = dev_net(dst->dev);
if (net->ipv4.sysctl_ip_fwd_use_pmtu ||
- dst_metric_locked(dst, RTAX_MTU) ||
+ ip_mtu_locked(dst) ||
!forwarding)
return dst_mtu(dst);
@@ -544,7 +551,6 @@ static inline struct sk_buff *ip_check_defrag(struct net *net, struct sk_buff *s
return skb;
}
#endif
-int ip_frag_mem(struct net *net);
/*
* Functions provided by ip_forward.c
diff --git a/include/net/ip_fib.h b/include/net/ip_fib.h
index aa75828..a6446d7 100644
--- a/include/net/ip_fib.h
+++ b/include/net/ip_fib.h
@@ -57,6 +57,7 @@ struct fib_nh_exception {
int fnhe_genid;
__be32 fnhe_daddr;
u32 fnhe_pmtu;
+ bool fnhe_mtu_locked;
__be32 fnhe_gw;
unsigned long fnhe_expires;
struct rtable __rcu *fnhe_rth_input;
@@ -362,6 +363,7 @@ int ip_fib_check_default(__be32 gw, struct net_device *dev);
int fib_sync_down_dev(struct net_device *dev, unsigned long event, bool force);
int fib_sync_down_addr(struct net_device *dev, __be32 local);
int fib_sync_up(struct net_device *dev, unsigned int nh_flags);
+void fib_sync_mtu(struct net_device *dev, u32 orig_mtu);
extern u32 fib_multipath_secret __read_mostly;
diff --git a/include/net/ipv6.h b/include/net/ipv6.h
index e64210c..7cb100d 100644
--- a/include/net/ipv6.h
+++ b/include/net/ipv6.h
@@ -312,14 +312,7 @@ struct ipv6_txoptions *ipv6_dup_options(struct sock *sk,
struct ipv6_txoptions *ipv6_renew_options(struct sock *sk,
struct ipv6_txoptions *opt,
int newtype,
- struct ipv6_opt_hdr __user *newopt,
- int newoptlen);
-struct ipv6_txoptions *
-ipv6_renew_options_kern(struct sock *sk,
- struct ipv6_txoptions *opt,
- int newtype,
- struct ipv6_opt_hdr *newopt,
- int newoptlen);
+ struct ipv6_opt_hdr *newopt);
struct ipv6_txoptions *ipv6_fixup_options(struct ipv6_txoptions *opt_space,
struct ipv6_txoptions *opt);
@@ -337,13 +330,6 @@ static inline bool ipv6_accept_ra(struct inet6_dev *idev)
idev->cnf.accept_ra;
}
-#if IS_ENABLED(CONFIG_IPV6)
-static inline int ip6_frag_mem(struct net *net)
-{
- return sum_frag_mem_limit(&net->ipv6.frags);
-}
-#endif
-
#define IPV6_FRAG_HIGH_THRESH (4 * 1024*1024) /* 4194304 */
#define IPV6_FRAG_LOW_THRESH (3 * 1024*1024) /* 3145728 */
#define IPV6_FRAG_TIMEOUT (60 * HZ) /* 60 seconds */
@@ -537,17 +523,8 @@ enum ip6_defrag_users {
__IP6_DEFRAG_CONNTRACK_BRIDGE_IN = IP6_DEFRAG_CONNTRACK_BRIDGE_IN + USHRT_MAX,
};
-struct ip6_create_arg {
- __be32 id;
- u32 user;
- const struct in6_addr *src;
- const struct in6_addr *dst;
- int iif;
- u8 ecn;
-};
-
void ip6_frag_init(struct inet_frag_queue *q, const void *a);
-bool ip6_frag_match(const struct inet_frag_queue *q, const void *a);
+extern const struct rhashtable_params ip6_rhash_params;
/*
* Equivalent of ipv4 struct ip
@@ -555,19 +532,13 @@ bool ip6_frag_match(const struct inet_frag_queue *q, const void *a);
struct frag_queue {
struct inet_frag_queue q;
- __be32 id; /* fragment id */
- u32 user;
- struct in6_addr saddr;
- struct in6_addr daddr;
-
int iif;
unsigned int csum;
__u16 nhoffset;
u8 ecn;
};
-void ip6_expire_frag_queue(struct net *net, struct frag_queue *fq,
- struct inet_frags *frags);
+void ip6_expire_frag_queue(struct net *net, struct frag_queue *fq);
static inline bool ipv6_addr_any(const struct in6_addr *a)
{
@@ -794,7 +765,7 @@ static inline __be32 ip6_make_flowlabel(struct net *net, struct sk_buff *skb,
* to minimize possbility that any useful information to an
* attacker is leaked. Only lower 20 bits are relevant.
*/
- rol32(hash, 16);
+ hash = rol32(hash, 16);
flowlabel = (__force __be32)hash & IPV6_FLOWLABEL_MASK;
diff --git a/include/net/llc.h b/include/net/llc.h
index e8e61d4..82d9899 100644
--- a/include/net/llc.h
+++ b/include/net/llc.h
@@ -116,6 +116,11 @@ static inline void llc_sap_hold(struct llc_sap *sap)
atomic_inc(&sap->refcnt);
}
+static inline bool llc_sap_hold_safe(struct llc_sap *sap)
+{
+ return atomic_inc_not_zero(&sap->refcnt);
+}
+
void llc_sap_close(struct llc_sap *sap);
static inline void llc_sap_put(struct llc_sap *sap)
diff --git a/include/net/llc_conn.h b/include/net/llc_conn.h
index fe994d2..df528a6 100644
--- a/include/net/llc_conn.h
+++ b/include/net/llc_conn.h
@@ -97,13 +97,14 @@ static __inline__ char llc_backlog_type(struct sk_buff *skb)
struct sock *llc_sk_alloc(struct net *net, int family, gfp_t priority,
struct proto *prot, int kern);
+void llc_sk_stop_all_timers(struct sock *sk, bool sync);
void llc_sk_free(struct sock *sk);
void llc_sk_reset(struct sock *sk);
/* Access to a connection */
int llc_conn_state_process(struct sock *sk, struct sk_buff *skb);
-void llc_conn_send_pdu(struct sock *sk, struct sk_buff *skb);
+int llc_conn_send_pdu(struct sock *sk, struct sk_buff *skb);
void llc_conn_rtn_pdu(struct sock *sk, struct sk_buff *skb);
void llc_conn_resend_i_pdu_as_cmd(struct sock *sk, u8 nr, u8 first_p_bit);
void llc_conn_resend_i_pdu_as_rsp(struct sock *sk, u8 nr, u8 first_f_bit);
diff --git a/include/net/mac80211.h b/include/net/mac80211.h
index 8fd61bc..920a771 100644
--- a/include/net/mac80211.h
+++ b/include/net/mac80211.h
@@ -4091,7 +4091,7 @@ void ieee80211_sta_uapsd_trigger(struct ieee80211_sta *sta, u8 tid);
* The TX headroom reserved by mac80211 for its own tx_status functions.
* This is enough for the radiotap header.
*/
-#define IEEE80211_TX_STATUS_HEADROOM 14
+#define IEEE80211_TX_STATUS_HEADROOM ALIGN(14, 4)
/**
* ieee80211_sta_set_buffered - inform mac80211 about driver-buffered frames
diff --git a/include/net/neighbour.h b/include/net/neighbour.h
index ddfb742..bd82a2b 100644
--- a/include/net/neighbour.h
+++ b/include/net/neighbour.h
@@ -449,6 +449,7 @@ static inline int neigh_hh_bridge(struct hh_cache *hh, struct sk_buff *skb)
static inline int neigh_hh_output(const struct hh_cache *hh, struct sk_buff *skb)
{
+ unsigned int hh_alen = 0;
unsigned int seq;
int hh_len;
@@ -456,16 +457,33 @@ static inline int neigh_hh_output(const struct hh_cache *hh, struct sk_buff *skb
seq = read_seqbegin(&hh->hh_lock);
hh_len = hh->hh_len;
if (likely(hh_len <= HH_DATA_MOD)) {
- /* this is inlined by gcc */
- memcpy(skb->data - HH_DATA_MOD, hh->hh_data, HH_DATA_MOD);
- } else {
- int hh_alen = HH_DATA_ALIGN(hh_len);
+ hh_alen = HH_DATA_MOD;
- memcpy(skb->data - hh_alen, hh->hh_data, hh_alen);
+ /* skb_push() would proceed silently if we have room for
+ * the unaligned size but not for the aligned size:
+ * check headroom explicitly.
+ */
+ if (likely(skb_headroom(skb) >= HH_DATA_MOD)) {
+ /* this is inlined by gcc */
+ memcpy(skb->data - HH_DATA_MOD, hh->hh_data,
+ HH_DATA_MOD);
+ }
+ } else {
+ hh_alen = HH_DATA_ALIGN(hh_len);
+
+ if (likely(skb_headroom(skb) >= hh_alen)) {
+ memcpy(skb->data - hh_alen, hh->hh_data,
+ hh_alen);
+ }
}
} while (read_seqretry(&hh->hh_lock, seq));
- skb_push(skb, hh_len);
+ if (WARN_ON_ONCE(skb_headroom(skb) < hh_alen)) {
+ kfree_skb(skb);
+ return NET_XMIT_DROP;
+ }
+
+ __skb_push(skb, hh_len);
return dev_queue_xmit(skb);
}
diff --git a/include/net/net_namespace.h b/include/net/net_namespace.h
index 23102da..c05db6f 100644
--- a/include/net/net_namespace.h
+++ b/include/net/net_namespace.h
@@ -116,6 +116,7 @@ struct net {
#endif
#if IS_ENABLED(CONFIG_NF_DEFRAG_IPV6)
struct netns_nf_frag nf_frag;
+ struct ctl_table_header *nf_frag_frags_hdr;
#endif
struct sock *nfnl;
struct sock *nfnl_stash;
diff --git a/include/net/netns/ipv6.h b/include/net/netns/ipv6.h
index 10d0848..5cae575 100644
--- a/include/net/netns/ipv6.h
+++ b/include/net/netns/ipv6.h
@@ -89,7 +89,6 @@ struct netns_ipv6 {
#if IS_ENABLED(CONFIG_NF_DEFRAG_IPV6)
struct netns_nf_frag {
- struct netns_sysctl_ipv6 sysctl;
struct netns_frags frags;
};
#endif
diff --git a/include/net/nexthop.h b/include/net/nexthop.h
index 3334dbf..7fc7866 100644
--- a/include/net/nexthop.h
+++ b/include/net/nexthop.h
@@ -6,7 +6,7 @@
static inline int rtnh_ok(const struct rtnexthop *rtnh, int remaining)
{
- return remaining >= sizeof(*rtnh) &&
+ return remaining >= (int)sizeof(*rtnh) &&
rtnh->rtnh_len >= sizeof(*rtnh) &&
rtnh->rtnh_len <= remaining;
}
diff --git a/include/net/nfc/hci.h b/include/net/nfc/hci.h
index 316694d..008f466 100644
--- a/include/net/nfc/hci.h
+++ b/include/net/nfc/hci.h
@@ -87,7 +87,7 @@ struct nfc_hci_pipe {
* According to specification 102 622 chapter 4.4 Pipes,
* the pipe identifier is 7 bits long.
*/
-#define NFC_HCI_MAX_PIPES 127
+#define NFC_HCI_MAX_PIPES 128
struct nfc_hci_init_data {
u8 gate_count;
struct nfc_hci_gate gates[NFC_HCI_MAX_CUSTOM_GATES];
diff --git a/include/net/regulatory.h b/include/net/regulatory.h
index ebc5a2e..f83cacc 100644
--- a/include/net/regulatory.h
+++ b/include/net/regulatory.h
@@ -78,7 +78,7 @@ struct regulatory_request {
int wiphy_idx;
enum nl80211_reg_initiator initiator;
enum nl80211_user_reg_hint_type user_reg_hint_type;
- char alpha2[2];
+ char alpha2[3];
enum nl80211_dfs_regions dfs_region;
bool intersect;
bool processed;
diff --git a/include/net/route.h b/include/net/route.h
index c0874c8..2702b7a 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -63,7 +63,8 @@ struct rtable {
__be32 rt_gateway;
/* Miscellaneous cached information */
- u32 rt_pmtu;
+ u32 rt_mtu_locked:1,
+ rt_pmtu:31;
u32 rt_table_id;
diff --git a/include/net/sock.h b/include/net/sock.h
index 31309b3..e7756fe 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -284,6 +284,7 @@ struct sock_common {
* @sk_filter: socket filtering instructions
* @sk_timer: sock cleanup timer
* @sk_stamp: time stamp of last packet received
+ * @sk_stamp_seq: lock for accessing sk_stamp on 32 bit architectures only
* @sk_tsflags: SO_TIMESTAMPING socket options
* @sk_tskey: counter to disambiguate concurrent tstamp requests
* @sk_socket: Identd and reporting IO signals
@@ -426,6 +427,9 @@ struct sock {
long sk_sndtimeo;
struct timer_list sk_timer;
ktime_t sk_stamp;
+#if BITS_PER_LONG==32
+ seqlock_t sk_stamp_seq;
+#endif
u16 sk_tsflags;
u8 sk_shutdown;
u32 sk_tskey;
@@ -2121,6 +2125,34 @@ static inline void sk_drops_add(struct sock *sk, const struct sk_buff *skb)
atomic_add(segs, &sk->sk_drops);
}
+static inline ktime_t sock_read_timestamp(struct sock *sk)
+{
+#if BITS_PER_LONG==32
+ unsigned int seq;
+ ktime_t kt;
+
+ do {
+ seq = read_seqbegin(&sk->sk_stamp_seq);
+ kt = sk->sk_stamp;
+ } while (read_seqretry(&sk->sk_stamp_seq, seq));
+
+ return kt;
+#else
+ return sk->sk_stamp;
+#endif
+}
+
+static inline void sock_write_timestamp(struct sock *sk, ktime_t kt)
+{
+#if BITS_PER_LONG==32
+ write_seqlock(&sk->sk_stamp_seq);
+ sk->sk_stamp = kt;
+ write_sequnlock(&sk->sk_stamp_seq);
+#else
+ sk->sk_stamp = kt;
+#endif
+}
+
void __sock_recv_timestamp(struct msghdr *msg, struct sock *sk,
struct sk_buff *skb);
void __sock_recv_wifi_status(struct msghdr *msg, struct sock *sk,
@@ -2145,7 +2177,7 @@ sock_recv_timestamp(struct msghdr *msg, struct sock *sk, struct sk_buff *skb)
(sk->sk_tsflags & SOF_TIMESTAMPING_RAW_HARDWARE)))
__sock_recv_timestamp(msg, sk, skb);
else
- sk->sk_stamp = kt;
+ sock_write_timestamp(sk, kt);
if (sock_flag(sk, SOCK_WIFI_STATUS) && skb->wifi_acked_valid)
__sock_recv_wifi_status(msg, sk, skb);
@@ -2165,7 +2197,7 @@ static inline void sock_recv_ts_and_drops(struct msghdr *msg, struct sock *sk,
if (sk->sk_flags & FLAGS_TS_OR_DROPS || sk->sk_tsflags & TSFLAGS_ANY)
__sock_recv_ts_and_drops(msg, sk, skb);
else
- sk->sk_stamp = skb->tstamp;
+ sock_write_timestamp(sk, skb->tstamp);
}
void __sock_tx_timestamp(__u16 tsflags, __u8 *tx_flags);
diff --git a/include/net/tc_act/tc_tunnel_key.h b/include/net/tc_act/tc_tunnel_key.h
index 253f8da..2dcd80d 100644
--- a/include/net/tc_act/tc_tunnel_key.h
+++ b/include/net/tc_act/tc_tunnel_key.h
@@ -16,7 +16,6 @@
struct tcf_tunnel_key_params {
struct rcu_head rcu;
int tcft_action;
- int action;
struct metadata_dst *tcft_enc_metadata;
};
diff --git a/include/net/tcp.h b/include/net/tcp.h
index 76d7c97..1f5ddaf 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -379,6 +379,7 @@ extern int tcp_use_userconfig_sysctl_handler(struct ctl_table *table,
extern int tcp_proc_delayed_ack_control(struct ctl_table *table, int write,
void __user *buffer, size_t *length,
loff_t *ppos);
+void tcp_enter_quickack_mode(struct sock *sk, unsigned int max_quickacks);
static inline void tcp_dec_quickack_mode(struct sock *sk,
const unsigned int pkts)
{
@@ -569,6 +570,7 @@ void tcp_send_fin(struct sock *sk);
void tcp_send_active_reset(struct sock *sk, gfp_t priority);
int tcp_send_synack(struct sock *);
void tcp_push_one(struct sock *, unsigned int mss_now);
+void __tcp_send_ack(struct sock *sk, u32 rcv_nxt);
void tcp_send_ack(struct sock *sk);
void tcp_send_delayed_ack(struct sock *sk);
void tcp_send_loss_probe(struct sock *sk);
@@ -864,8 +866,6 @@ enum tcp_ca_event {
CA_EVENT_LOSS, /* loss timeout */
CA_EVENT_ECN_NO_CE, /* ECT set, but not CE marked */
CA_EVENT_ECN_IS_CE, /* received CE marked IP packet */
- CA_EVENT_DELAYED_ACK, /* Delayed ack is sent */
- CA_EVENT_NON_DELAYED_ACK,
};
/* Information about inbound ACK, passed to cong_ops->in_ack_event() */
diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 5ad43a4..a42535f 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -3308,6 +3308,20 @@ static inline int ib_check_mr_access(int flags)
return 0;
}
+static inline bool ib_access_writable(int access_flags)
+{
+ /*
+ * We have writable memory backing the MR if any of the following
+ * access flags are set. "Local write" and "remote write" obviously
+ * require write access. "Remote atomic" can do things like fetch and
+ * add, which will modify memory, and "MW bind" can change permissions
+ * by binding a window.
+ */
+ return access_flags &
+ (IB_ACCESS_LOCAL_WRITE | IB_ACCESS_REMOTE_WRITE |
+ IB_ACCESS_REMOTE_ATOMIC | IB_ACCESS_MW_BIND);
+}
+
/**
* ib_check_mr_status: lightweight check of MR status.
* This routine may provide status checks on a selected
diff --git a/include/scsi/scsi_cmnd.h b/include/scsi/scsi_cmnd.h
index 9fc1aec..5ef894d 100644
--- a/include/scsi/scsi_cmnd.h
+++ b/include/scsi/scsi_cmnd.h
@@ -294,6 +294,11 @@ static inline sector_t scsi_get_lba(struct scsi_cmnd *scmd)
return blk_rq_pos(scmd->request);
}
+static inline u32 scsi_get_bytes(struct scsi_cmnd *scmd)
+{
+ return blk_rq_bytes(scmd->request);
+}
+
static inline unsigned int scsi_prot_interval(struct scsi_cmnd *scmd)
{
return scmd->device->sector_size;
diff --git a/include/soc/qcom/restart.h b/include/soc/qcom/restart.h
index bd0f139..38127ff 100644
--- a/include/soc/qcom/restart.h
+++ b/include/soc/qcom/restart.h
@@ -18,6 +18,7 @@
#define RESTART_DLOAD 0x1
void msm_set_restart_mode(int mode);
+void set_restart_msg(const char *msg);
extern int pmic_reset_irq;
#endif
diff --git a/include/soc/qcom/subsystem_restart.h b/include/soc/qcom/subsystem_restart.h
index 9a0a53e..c547ed9 100644
--- a/include/soc/qcom/subsystem_restart.h
+++ b/include/soc/qcom/subsystem_restart.h
@@ -1,4 +1,4 @@
-/* Copyright (c) 2014-2017, The Linux Foundation. All rights reserved.
+/* Copyright (c) 2014-2016, The Linux Foundation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 and
@@ -17,8 +17,10 @@
#include <linux/spinlock.h>
#include <linux/interrupt.h>
+#define MAX_SSR_REASON_LEN 256U
+#define MAX_CRASH_TIMESTAMP_LEN 30U
+
struct subsys_device;
-extern struct bus_type subsys_bus_type;
enum {
RESET_SOC = 0,
@@ -59,6 +61,8 @@ struct module;
* @ignore_ssr_failure: SSR failures are usually fatal and results in panic. If
* set will ignore failure.
* @edge: GLINK logical name of the subsystem
+ * @last_crash_reason: reason of the last crash
+ * @last_crash_timestamp: timestamp of the last crash
*/
struct subsys_desc {
const char *name;
@@ -95,6 +99,8 @@ struct subsys_desc {
bool system_debug;
bool ignore_ssr_failure;
const char *edge;
+ char last_crash_reason[MAX_SSR_REASON_LEN];
+ char last_crash_timestamp[MAX_CRASH_TIMESTAMP_LEN];
};
/**
diff --git a/include/soc/tegra/mc.h b/include/soc/tegra/mc.h
index 44202ff..f759e09 100644
--- a/include/soc/tegra/mc.h
+++ b/include/soc/tegra/mc.h
@@ -99,6 +99,8 @@ struct tegra_mc_soc {
u8 client_id_mask;
const struct tegra_smmu_soc *smmu;
+
+ u32 intmask;
};
struct tegra_mc {
diff --git a/include/sound/control.h b/include/sound/control.h
index 21d047f..4142757 100644
--- a/include/sound/control.h
+++ b/include/sound/control.h
@@ -22,6 +22,7 @@
*
*/
+#include <linux/nospec.h>
#include <sound/asound.h>
#define snd_kcontrol_chip(kcontrol) ((kcontrol)->private_data)
@@ -147,12 +148,14 @@ int snd_ctl_get_preferred_subdevice(struct snd_card *card, int type);
static inline unsigned int snd_ctl_get_ioffnum(struct snd_kcontrol *kctl, struct snd_ctl_elem_id *id)
{
- return id->numid - kctl->id.numid;
+ unsigned int ioff = id->numid - kctl->id.numid;
+ return array_index_nospec(ioff, kctl->count);
}
static inline unsigned int snd_ctl_get_ioffidx(struct snd_kcontrol *kctl, struct snd_ctl_elem_id *id)
{
- return id->index - kctl->id.index;
+ unsigned int ioff = id->index - kctl->id.index;
+ return array_index_nospec(ioff, kctl->count);
}
static inline unsigned int snd_ctl_get_ioff(struct snd_kcontrol *kctl, struct snd_ctl_elem_id *id)
diff --git a/include/sound/pcm_params.h b/include/sound/pcm_params.h
index 747c94a..91f6abf 100644
--- a/include/sound/pcm_params.h
+++ b/include/sound/pcm_params.h
@@ -251,11 +251,13 @@ static inline int snd_interval_empty(const struct snd_interval *i)
static inline int snd_interval_single(const struct snd_interval *i)
{
return (i->min == i->max ||
- (i->min + 1 == i->max && i->openmax));
+ (i->min + 1 == i->max && (i->openmin || i->openmax)));
}
static inline int snd_interval_value(const struct snd_interval *i)
{
+ if (i->openmin && !i->openmax)
+ return i->max;
return i->min;
}
diff --git a/include/trace/events/ext4.h b/include/trace/events/ext4.h
index 09c71e9..215668b 100644
--- a/include/trace/events/ext4.h
+++ b/include/trace/events/ext4.h
@@ -223,6 +223,26 @@ TRACE_EVENT(ext4_drop_inode,
(unsigned long) __entry->ino, __entry->drop)
);
+TRACE_EVENT(ext4_nfs_commit_metadata,
+ TP_PROTO(struct inode *inode),
+
+ TP_ARGS(inode),
+
+ TP_STRUCT__entry(
+ __field( dev_t, dev )
+ __field( ino_t, ino )
+ ),
+
+ TP_fast_assign(
+ __entry->dev = inode->i_sb->s_dev;
+ __entry->ino = inode->i_ino;
+ ),
+
+ TP_printk("dev %d,%d ino %lu",
+ MAJOR(__entry->dev), MINOR(__entry->dev),
+ (unsigned long) __entry->ino)
+);
+
TRACE_EVENT(ext4_mark_inode_dirty,
TP_PROTO(struct inode *inode, unsigned long IP),
diff --git a/include/trace/events/irq.h b/include/trace/events/irq.h
index 1c41b74..478d148 100644
--- a/include/trace/events/irq.h
+++ b/include/trace/events/irq.h
@@ -159,6 +159,54 @@ DEFINE_EVENT(softirq, softirq_raise,
TP_ARGS(vec_nr)
);
+/**
+ * tasklet_entry - called immediately before the tasklet handler
+ * @t: pointer to struct tasklet_struct
+ *
+ * When used in combination with the tasklet_exit tracepoint we can
+ * determine the tasklet latency.
+ */
+TRACE_EVENT(tasklet_entry,
+
+ TP_PROTO(struct tasklet_struct *t),
+
+ TP_ARGS(t),
+
+ TP_STRUCT__entry(
+ __field(void *, func)
+ ),
+
+ TP_fast_assign(
+ __entry->func = t->func;
+ ),
+
+ TP_printk("func=%pf", __entry->func)
+);
+
+/**
+ * tasklet_exit - called immediately after the tasklet handler returns
+ * @t: pointer to struct tasklet_struct
+ *
+ * When used in combination with the tasklet_entry tracepoint we can
+ * determine the tasklet latency.
+ */
+TRACE_EVENT(tasklet_exit,
+
+ TP_PROTO(struct tasklet_struct *t),
+
+ TP_ARGS(t),
+
+ TP_STRUCT__entry(
+ __field(void *, func)
+ ),
+
+ TP_fast_assign(
+ __entry->func = t->func;
+ ),
+
+ TP_printk("func=%pf", __entry->func)
+);
+
#endif /* _TRACE_IRQ_H */
/* This part must be outside protection */
diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index 5b8c6f8..0782cbf 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -329,6 +329,54 @@ TRACE_EVENT(mm_page_alloc_extfrag,
__entry->change_ownership)
);
+TRACE_EVENT(ion_heap_shrink,
+
+ TP_PROTO(const char *heap_name,
+ size_t len,
+ long total_allocated),
+
+ TP_ARGS(heap_name, len, total_allocated),
+
+ TP_STRUCT__entry(
+ __string(heap_name, heap_name)
+ __field(size_t, len)
+ __field(long, total_allocated)
+ ),
+
+ TP_fast_assign(
+ __assign_str(heap_name, heap_name);
+ __entry->len = len;
+ __entry->total_allocated = total_allocated;
+ ),
+
+ TP_printk("heap_name=%s, len=%zu, total_allocated=%ld",
+ __get_str(heap_name), __entry->len, __entry->total_allocated)
+);
+
+TRACE_EVENT(ion_heap_grow,
+
+ TP_PROTO(const char *heap_name,
+ size_t len,
+ long total_allocated),
+
+ TP_ARGS(heap_name, len, total_allocated),
+
+ TP_STRUCT__entry(
+ __string(heap_name, heap_name)
+ __field(size_t, len)
+ __field(long, total_allocated)
+ ),
+
+ TP_fast_assign(
+ __assign_str(heap_name, heap_name);
+ __entry->len = len;
+ __entry->total_allocated = total_allocated;
+ ),
+
+ TP_printk("heap_name=%s, len=%zu, total_allocated=%ld",
+ __get_str(heap_name), __entry->len, __entry->total_allocated)
+ );
+
DECLARE_EVENT_CLASS(ion_alloc,
@@ -342,7 +390,7 @@ DECLARE_EVENT_CLASS(ion_alloc,
TP_STRUCT__entry(
__array(char, client_name, 64)
- __field(const char *, heap_name)
+ __string(heap_name, heap_name)
__field(size_t, len)
__field(unsigned int, mask)
__field(unsigned int, flags)
@@ -350,7 +398,7 @@ DECLARE_EVENT_CLASS(ion_alloc,
TP_fast_assign(
strlcpy(__entry->client_name, client_name, 64);
- __entry->heap_name = heap_name;
+ __assign_str(heap_name, heap_name);
__entry->len = len;
__entry->mask = mask;
__entry->flags = flags;
@@ -358,7 +406,7 @@ DECLARE_EVENT_CLASS(ion_alloc,
TP_printk("client_name=%s heap_name=%s len=%zu mask=0x%x flags=0x%x",
__entry->client_name,
- __entry->heap_name,
+ __get_str(heap_name),
__entry->len,
__entry->mask,
__entry->flags)
@@ -399,7 +447,7 @@ DECLARE_EVENT_CLASS(ion_alloc_error,
TP_STRUCT__entry(
__field(const char *, client_name)
- __field(const char *, heap_name)
+ __string(heap_name, heap_name)
__field(size_t, len)
__field(unsigned int, mask)
__field(unsigned int, flags)
@@ -408,7 +456,7 @@ DECLARE_EVENT_CLASS(ion_alloc_error,
TP_fast_assign(
__entry->client_name = client_name;
- __entry->heap_name = heap_name;
+ __assign_str(heap_name, heap_name);
__entry->len = len;
__entry->mask = mask;
__entry->flags = flags;
@@ -418,7 +466,7 @@ DECLARE_EVENT_CLASS(ion_alloc_error,
TP_printk(
"client_name=%s heap_name=%s len=%zu mask=0x%x flags=0x%x error=%ld",
__entry->client_name,
- __entry->heap_name,
+ __get_str(heap_name),
__entry->len,
__entry->mask,
__entry->flags,
@@ -732,21 +780,21 @@ DECLARE_EVENT_CLASS(ion_secure_cma_allocate,
TP_ARGS(heap_name, len, align, flags),
TP_STRUCT__entry(
- __field(const char *, heap_name)
+ __string(heap_name, heap_name)
__field(unsigned long, len)
__field(unsigned long, align)
__field(unsigned long, flags)
),
TP_fast_assign(
- __entry->heap_name = heap_name;
+ __assign_str(heap_name, heap_name);
__entry->len = len;
__entry->align = align;
__entry->flags = flags;
),
TP_printk("heap_name=%s len=%lx align=%lx flags=%lx",
- __entry->heap_name,
+ __get_str(heap_name),
__entry->len,
__entry->align,
__entry->flags)
@@ -780,21 +828,21 @@ DECLARE_EVENT_CLASS(ion_cp_secure_buffer,
TP_ARGS(heap_name, len, align, flags),
TP_STRUCT__entry(
- __field(const char *, heap_name)
+ __string(heap_name, heap_name)
__field(unsigned long, len)
__field(unsigned long, align)
__field(unsigned long, flags)
),
TP_fast_assign(
- __entry->heap_name = heap_name;
+ __assign_str(heap_name, heap_name);
__entry->len = len;
__entry->align = align;
__entry->flags = flags;
),
TP_printk("heap_name=%s len=%lx align=%lx flags=%lx",
- __entry->heap_name,
+ __get_str(heap_name),
__entry->len,
__entry->align,
__entry->flags)
@@ -873,6 +921,28 @@ DEFINE_EVENT(iommu_sec_ptbl_map_range, iommu_sec_ptbl_map_range_end,
TP_ARGS(sec_id, num, va, pa, len)
);
+
+TRACE_EVENT(rss_stat,
+
+ TP_PROTO(int member,
+ long count),
+
+ TP_ARGS(member, count),
+
+ TP_STRUCT__entry(
+ __field(int, member)
+ __field(long, size)
+ ),
+
+ TP_fast_assign(
+ __entry->member = member;
+ __entry->size = (count << PAGE_SHIFT);
+ ),
+
+ TP_printk("member=%d size=%ldB",
+ __entry->member,
+ __entry->size)
+ );
#endif /* _TRACE_KMEM_H */
/* This part must be outside protection */
diff --git a/include/trace/events/mm_event.h b/include/trace/events/mm_event.h
new file mode 100644
index 0000000..fd7a536
--- /dev/null
+++ b/include/trace/events/mm_event.h
@@ -0,0 +1,108 @@
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM mm_event
+
+#if !defined(_TRACE_MM_EVENT_H) || defined(TRACE_HEADER_MULTI_READ)
+#define _TRACE_MM_EVENT_H
+
+#include <linux/types.h>
+#include <linux/tracepoint.h>
+#include <linux/mm.h>
+#include <linux/mm_event.h>
+
+struct mm_event_task;
+struct mm_event_vmstat;
+
+#define show_mm_event_type(type) \
+ __print_symbolic(type, \
+ { MM_MIN_FAULT, "min_flt" }, \
+ { MM_MAJ_FAULT, "maj_flt" }, \
+ { MM_READ_IO, "read_io" }, \
+ { MM_COMPACTION, "compaction" }, \
+ { MM_RECLAIM, "reclaim" }, \
+ { MM_SWP_FAULT, "swp_flt" }, \
+ { MM_KERN_ALLOC, "kern_alloc" })
+
+TRACE_EVENT(mm_event_record,
+
+ TP_PROTO(enum mm_event_type type, struct mm_event_task *record),
+
+ TP_ARGS(type, record),
+
+ TP_STRUCT__entry(
+ __field(enum mm_event_type, type)
+ __field(unsigned int, count)
+ __field(unsigned int, avg_lat)
+ __field(unsigned int, max_lat)
+ ),
+
+ TP_fast_assign(
+ __entry->type = type;
+ __entry->count = record->count;
+ __entry->avg_lat = record->accm_lat / record->count;
+ __entry->max_lat = record->max_lat;
+ ),
+
+ TP_printk("%s count=%d avg_lat=%u max_lat=%u",
+ show_mm_event_type(__entry->type),
+ __entry->count, __entry->avg_lat,
+ __entry->max_lat)
+);
+
+TRACE_EVENT(mm_event_vmstat_record,
+
+ TP_PROTO(struct mm_event_vmstat *vmstat),
+
+ TP_ARGS(vmstat),
+
+ TP_STRUCT__entry(
+ __field(unsigned long, free)
+ __field(unsigned long, file)
+ __field(unsigned long, anon)
+ __field(unsigned long, ion)
+ __field(unsigned long, slab)
+ __field(unsigned long, ws_refault)
+ __field(unsigned long, ws_activate)
+ __field(unsigned long, mapped)
+ __field(unsigned long, pgin)
+ __field(unsigned long, pgout)
+ __field(unsigned long, swpin)
+ __field(unsigned long, swpout)
+ __field(unsigned long, reclaim_steal)
+ __field(unsigned long, reclaim_scan)
+ __field(unsigned long, compact_scan)
+ ),
+
+ TP_fast_assign(
+ __entry->free = vmstat->free;
+ __entry->file = vmstat->file;
+ __entry->anon = vmstat->anon;
+ __entry->ion = vmstat->ion;
+ __entry->slab = vmstat->slab;
+ __entry->ws_refault = vmstat->ws_refault;
+ __entry->ws_activate = vmstat->ws_activate;
+ __entry->mapped = vmstat->mapped;
+ __entry->pgin = vmstat->pgin;
+ __entry->pgout = vmstat->pgout;
+ __entry->swpin = vmstat->swpin;
+ __entry->swpout = vmstat->swpout;
+ __entry->reclaim_steal = vmstat->reclaim_steal;
+ __entry->reclaim_scan = vmstat->reclaim_scan;
+ __entry->compact_scan = vmstat->compact_scan;
+ ),
+
+ TP_printk("free=%lu file=%lu anon=%lu ion=%lu slab=%lu ws_refault=%lu "
+ "ws_activate=%lu mapped=%lu pgin=%lu pgout=%lu, swpin=%lu "
+ "swpout=%lu reclaim_steal=%lu reclaim_scan=%lu compact_scan=%lu",
+ __entry->free, __entry->file,
+ __entry->anon, __entry->ion, __entry->slab,
+ __entry->ws_refault, __entry->ws_activate,
+ __entry->mapped, __entry->pgin, __entry->pgout,
+ __entry->swpin, __entry->swpout,
+ __entry->reclaim_steal, __entry->reclaim_scan,
+ __entry->compact_scan)
+);
+
+#endif /* _TRACE_MM_EVENT_H */
+
+/* This part must be outside protection */
+#include <trace/define_trace.h>
diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h
index 5a81ab4..61d41f2 100644
--- a/include/trace/events/mmflags.h
+++ b/include/trace/events/mmflags.h
@@ -81,12 +81,14 @@
#define __def_pageflag_names \
{1UL << PG_locked, "locked" }, \
+ {1UL << PG_waiters, "waiters" }, \
{1UL << PG_error, "error" }, \
{1UL << PG_referenced, "referenced" }, \
{1UL << PG_uptodate, "uptodate" }, \
{1UL << PG_dirty, "dirty" }, \
{1UL << PG_lru, "lru" }, \
{1UL << PG_active, "active" }, \
+ {1UL << PG_workingset, "workingset" }, \
{1UL << PG_slab, "slab" }, \
{1UL << PG_owner_priv_1, "owner_priv_1" }, \
{1UL << PG_arch_1, "arch_1" }, \
diff --git a/include/trace/events/power.h b/include/trace/events/power.h
index a29c76f..e3270a8 100644
--- a/include/trace/events/power.h
+++ b/include/trace/events/power.h
@@ -331,7 +331,6 @@ DEFINE_EVENT(wakeup_source, wakeup_source_deactivate,
* The clock events are used for clock enable/disable and for
* clock rate change
*/
-#if defined(CONFIG_COMMON_CLK_MSM)
DECLARE_EVENT_CLASS(clock,
TP_PROTO(const char *name, unsigned int state, unsigned int cpu_id),
@@ -425,7 +424,6 @@ TRACE_EVENT(clock_state,
__entry->count, __entry->rate)
);
-#endif /* CONFIG_COMMON_CLK_MSM */
/*
* The power domain events are used for power domains transitions
diff --git a/include/trace/events/process_reclaim.h b/include/trace/events/process_reclaim.h
deleted file mode 100644
index 6fcede7..0000000
--- a/include/trace/events/process_reclaim.h
+++ /dev/null
@@ -1,85 +0,0 @@
-/* Copyright (c) 2015, The Linux Foundation. All rights reserved.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 and
- * only version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
-
-#undef TRACE_SYSTEM
-#define TRACE_SYSTEM process_reclaim
-
-#if !defined(_TRACE_EVENT_PROCESSRECLAIM_H) || defined(TRACE_HEADER_MULTI_READ)
-#define _TRACE_EVENT_PROCESSRECLAIM_H
-
-#include <linux/tracepoint.h>
-#include <linux/types.h>
-#include <linux/sched.h>
-
-TRACE_EVENT(process_reclaim,
-
- TP_PROTO(int tasksize,
- short oom_score_adj,
- int nr_scanned, int nr_reclaimed,
- int per_swap_size, int total_sz,
- int nr_to_reclaim),
-
- TP_ARGS(tasksize, oom_score_adj, nr_scanned,
- nr_reclaimed, per_swap_size,
- total_sz, nr_to_reclaim),
-
- TP_STRUCT__entry(
- __field(int, tasksize)
- __field(short, oom_score_adj)
- __field(int, nr_scanned)
- __field(int, nr_reclaimed)
- __field(int, per_swap_size)
- __field(int, total_sz)
- __field(int, nr_to_reclaim)
- ),
-
- TP_fast_assign(
- __entry->tasksize = tasksize;
- __entry->oom_score_adj = oom_score_adj;
- __entry->nr_scanned = nr_scanned;
- __entry->nr_reclaimed = nr_reclaimed;
- __entry->per_swap_size = per_swap_size;
- __entry->total_sz = total_sz;
- __entry->nr_to_reclaim = nr_to_reclaim;
- ),
-
- TP_printk("%d, %hd, %d, %d, %d, %d, %d",
- __entry->tasksize, __entry->oom_score_adj,
- __entry->nr_scanned, __entry->nr_reclaimed,
- __entry->per_swap_size, __entry->total_sz,
- __entry->nr_to_reclaim)
-);
-
-TRACE_EVENT(process_reclaim_eff,
-
- TP_PROTO(int efficiency, int reclaim_avg_efficiency),
-
- TP_ARGS(efficiency, reclaim_avg_efficiency),
-
- TP_STRUCT__entry(
- __field(int, efficiency)
- __field(int, reclaim_avg_efficiency)
- ),
-
- TP_fast_assign(
- __entry->efficiency = efficiency;
- __entry->reclaim_avg_efficiency = reclaim_avg_efficiency;
- ),
-
- TP_printk("%d, %d", __entry->efficiency,
- __entry->reclaim_avg_efficiency)
-);
-
-#endif
-
-#include <trace/define_trace.h>
-
diff --git a/include/trace/events/sched.h b/include/trace/events/sched.h
index b355ebf..9fbf30a 100644
--- a/include/trace/events/sched.h
+++ b/include/trace/events/sched.h
@@ -713,10 +713,10 @@ TRACE_EVENT(sched_energy_diff,
TRACE_EVENT(sched_task_util,
TP_PROTO(struct task_struct *p, int next_cpu, int backup_cpu,
- int target_cpu, bool sync, bool need_idle, int fastpath,
+ int target_cpu, bool need_idle, int fastpath,
bool placement_boost, int rtg_cpu, u64 start_t),
- TP_ARGS(p, next_cpu, backup_cpu, target_cpu, sync, need_idle, fastpath,
+ TP_ARGS(p, next_cpu, backup_cpu, target_cpu, need_idle, fastpath,
placement_boost, rtg_cpu, start_t),
TP_STRUCT__entry(
@@ -727,7 +727,6 @@ TRACE_EVENT(sched_task_util,
__field(int, next_cpu )
__field(int, backup_cpu )
__field(int, target_cpu )
- __field(bool, sync )
__field(bool, need_idle )
__field(int, fastpath )
__field(bool, placement_boost )
@@ -743,7 +742,6 @@ TRACE_EVENT(sched_task_util,
__entry->next_cpu = next_cpu;
__entry->backup_cpu = backup_cpu;
__entry->target_cpu = target_cpu;
- __entry->sync = sync;
__entry->need_idle = need_idle;
__entry->fastpath = fastpath;
__entry->placement_boost = placement_boost;
@@ -751,8 +749,8 @@ TRACE_EVENT(sched_task_util,
__entry->latency = (sched_clock() - start_t);
),
- TP_printk("pid=%d comm=%s util=%lu prev_cpu=%d next_cpu=%d backup_cpu=%d target_cpu=%d sync=%d need_idle=%d fastpath=%d placement_boost=%d rtg_cpu=%d latency=%llu",
- __entry->pid, __entry->comm, __entry->util, __entry->prev_cpu, __entry->next_cpu, __entry->backup_cpu, __entry->target_cpu, __entry->sync, __entry->need_idle, __entry->fastpath, __entry->placement_boost, __entry->rtg_cpu, __entry->latency)
+ TP_printk("pid=%d comm=%s util=%lu prev_cpu=%d next_cpu=%d backup_cpu=%d target_cpu=%d need_idle=%d fastpath=%d placement_boost=%d rtg_cpu=%d latency=%llu",
+ __entry->pid, __entry->comm, __entry->util, __entry->prev_cpu, __entry->next_cpu, __entry->backup_cpu, __entry->target_cpu, __entry->need_idle, __entry->fastpath, __entry->placement_boost, __entry->rtg_cpu, __entry->latency)
);
#endif
diff --git a/include/trace/events/timer.h b/include/trace/events/timer.h
index 28c5da6..3411da7 100644
--- a/include/trace/events/timer.h
+++ b/include/trace/events/timer.h
@@ -125,6 +125,20 @@ DEFINE_EVENT(timer_class, timer_cancel,
TP_ARGS(timer)
);
+#define decode_clockid(type) \
+ __print_symbolic(type, \
+ { CLOCK_REALTIME, "CLOCK_REALTIME" }, \
+ { CLOCK_MONOTONIC, "CLOCK_MONOTONIC" }, \
+ { CLOCK_BOOTTIME, "CLOCK_BOOTTIME" }, \
+ { CLOCK_TAI, "CLOCK_TAI" })
+
+#define decode_hrtimer_mode(mode) \
+ __print_symbolic(mode, \
+ { HRTIMER_MODE_ABS, "ABS" }, \
+ { HRTIMER_MODE_REL, "REL" }, \
+ { HRTIMER_MODE_ABS_PINNED, "ABS|PINNED" }, \
+ { HRTIMER_MODE_REL_PINNED, "REL|PINNED" })
+
/**
* hrtimer_init - called when the hrtimer is initialized
* @hrtimer: pointer to struct hrtimer
@@ -151,10 +165,8 @@ TRACE_EVENT(hrtimer_init,
),
TP_printk("hrtimer=%p clockid=%s mode=%s", __entry->hrtimer,
- __entry->clockid == CLOCK_REALTIME ?
- "CLOCK_REALTIME" : "CLOCK_MONOTONIC",
- __entry->mode == HRTIMER_MODE_ABS ?
- "HRTIMER_MODE_ABS" : "HRTIMER_MODE_REL")
+ decode_clockid(__entry->clockid),
+ decode_hrtimer_mode(__entry->mode))
);
/**
diff --git a/include/trace/events/ufs.h b/include/trace/events/ufs.h
index aaa5cf7..a3b7038 100644
--- a/include/trace/events/ufs.h
+++ b/include/trace/events/ufs.h
@@ -197,7 +197,7 @@ DEFINE_EVENT(ufshcd_template, ufshcd_init,
TRACE_EVENT(ufshcd_command,
TP_PROTO(const char *dev_name, const char *str, unsigned int tag,
- u32 doorbell, int transfer_len, u32 intr, u64 lba,
+ u32 doorbell, u32 transfer_len, u32 intr, u64 lba,
u8 opcode),
TP_ARGS(dev_name, str, tag, doorbell, transfer_len, intr, lba, opcode),
@@ -207,7 +207,7 @@ TRACE_EVENT(ufshcd_command,
__string(str, str)
__field(unsigned int, tag)
__field(u32, doorbell)
- __field(int, transfer_len)
+ __field(u32, transfer_len)
__field(u32, intr)
__field(u64, lba)
__field(u8, opcode)
@@ -225,7 +225,7 @@ TRACE_EVENT(ufshcd_command,
),
TP_printk(
- "%s: %14s: tag: %-2u cmd: 0x%-2x lba: %-9llu size: %-7d DB: 0x%-8x IS: 0x%x",
+ "%s: %14s: tag: %-2u cmd: 0x%-2x lba: %-9llu size: %-7u DB: 0x%-8x IS: 0x%x",
__get_str(dev_name), __get_str(str), __entry->tag,
(u32)__entry->opcode, __entry->lba, __entry->transfer_len,
__entry->doorbell, __entry->intr
diff --git a/include/trace/events/xen.h b/include/trace/events/xen.h
index bce990f..d6be935 100644
--- a/include/trace/events/xen.h
+++ b/include/trace/events/xen.h
@@ -377,22 +377,6 @@ DECLARE_EVENT_CLASS(xen_mmu_pgd,
DEFINE_XEN_MMU_PGD_EVENT(xen_mmu_pgd_pin);
DEFINE_XEN_MMU_PGD_EVENT(xen_mmu_pgd_unpin);
-TRACE_EVENT(xen_mmu_flush_tlb_all,
- TP_PROTO(int x),
- TP_ARGS(x),
- TP_STRUCT__entry(__array(char, x, 0)),
- TP_fast_assign((void)x),
- TP_printk("%s", "")
- );
-
-TRACE_EVENT(xen_mmu_flush_tlb,
- TP_PROTO(int x),
- TP_ARGS(x),
- TP_STRUCT__entry(__array(char, x, 0)),
- TP_fast_assign((void)x),
- TP_printk("%s", "")
- );
-
TRACE_EVENT(xen_mmu_flush_tlb_single,
TP_PROTO(unsigned long addr),
TP_ARGS(addr),
diff --git a/include/uapi/drm/virtgpu_drm.h b/include/uapi/drm/virtgpu_drm.h
index 91a31ff..9a781f0 100644
--- a/include/uapi/drm/virtgpu_drm.h
+++ b/include/uapi/drm/virtgpu_drm.h
@@ -63,6 +63,7 @@ struct drm_virtgpu_execbuffer {
};
#define VIRTGPU_PARAM_3D_FEATURES 1 /* do we have 3D features in the hw */
+#define VIRTGPU_PARAM_CAPSET_QUERY_FIX 2 /* do we have the capset fix */
struct drm_virtgpu_getparam {
__u64 param;
diff --git a/include/uapi/linux/Kbuild b/include/uapi/linux/Kbuild
index 5da3634..d718e68 100644
--- a/include/uapi/linux/Kbuild
+++ b/include/uapi/linux/Kbuild
@@ -517,3 +517,4 @@
header-y += msm_rotator.h
header-y += bgcom_interface.h
header-y += nfc/
+header-y += citadel.h
diff --git a/include/uapi/linux/android/binder.h b/include/uapi/linux/android/binder.h
index 5539933..0631c50 100644
--- a/include/uapi/linux/android/binder.h
+++ b/include/uapi/linux/android/binder.h
@@ -87,6 +87,14 @@ enum flat_binder_object_flags {
* scheduling policy from the caller (for synchronous transactions).
*/
FLAT_BINDER_FLAG_INHERIT_RT = 0x800,
+
+ /**
+ * @FLAT_BINDER_FLAG_TXN_SECURITY_CTX: request security contexts
+ *
+ * Only when set, causes senders to include their security
+ * context
+ */
+ FLAT_BINDER_FLAG_TXN_SECURITY_CTX = 0x1000,
};
#ifdef BINDER_IPC_32BIT
@@ -246,6 +254,15 @@ struct binder_node_debug_info {
__u32 has_weak_ref;
};
+struct binder_node_info_for_ref {
+ __u32 handle;
+ __u32 strong_count;
+ __u32 weak_count;
+ __u32 reserved1;
+ __u32 reserved2;
+ __u32 reserved3;
+};
+
#define BINDER_WRITE_READ _IOWR('b', 1, struct binder_write_read)
#define BINDER_SET_IDLE_TIMEOUT _IOW('b', 3, __s64)
#define BINDER_SET_MAX_THREADS _IOW('b', 5, __u32)
@@ -254,6 +271,8 @@ struct binder_node_debug_info {
#define BINDER_THREAD_EXIT _IOW('b', 8, __s32)
#define BINDER_VERSION _IOWR('b', 9, struct binder_version)
#define BINDER_GET_NODE_DEBUG_INFO _IOWR('b', 11, struct binder_node_debug_info)
+#define BINDER_GET_NODE_INFO_FOR_REF _IOWR('b', 12, struct binder_node_info_for_ref)
+#define BINDER_SET_CONTEXT_MGR_EXT _IOW('b', 13, struct flat_binder_object)
/*
* NOTE: Two special error codes you should check for when calling
@@ -312,6 +331,11 @@ struct binder_transaction_data {
} data;
};
+struct binder_transaction_data_secctx {
+ struct binder_transaction_data transaction_data;
+ binder_uintptr_t secctx;
+};
+
struct binder_transaction_data_sg {
struct binder_transaction_data transaction_data;
binder_size_t buffers_size;
@@ -347,6 +371,11 @@ enum binder_driver_return_protocol {
BR_OK = _IO('r', 1),
/* No parameters! */
+ BR_TRANSACTION_SEC_CTX = _IOR('r', 2,
+ struct binder_transaction_data_secctx),
+ /*
+ * binder_transaction_data_secctx: the received command.
+ */
BR_TRANSACTION = _IOR('r', 2, struct binder_transaction_data),
BR_REPLY = _IOR('r', 3, struct binder_transaction_data),
/*
diff --git a/include/uapi/linux/btrfs_tree.h b/include/uapi/linux/btrfs_tree.h
index d5ad15a..a1ded2a 100644
--- a/include/uapi/linux/btrfs_tree.h
+++ b/include/uapi/linux/btrfs_tree.h
@@ -452,6 +452,7 @@ struct btrfs_free_space_header {
#define BTRFS_SUPER_FLAG_SEEDING (1ULL << 32)
#define BTRFS_SUPER_FLAG_METADUMP (1ULL << 33)
+#define BTRFS_SUPER_FLAG_METADUMP_V2 (1ULL << 34)
/*
@@ -729,6 +730,7 @@ struct btrfs_balance_item {
#define BTRFS_FILE_EXTENT_INLINE 0
#define BTRFS_FILE_EXTENT_REG 1
#define BTRFS_FILE_EXTENT_PREALLOC 2
+#define BTRFS_FILE_EXTENT_TYPES 2
struct btrfs_file_extent_item {
/*
diff --git a/include/uapi/linux/citadel.h b/include/uapi/linux/citadel.h
new file mode 100644
index 0000000..256091f
--- /dev/null
+++ b/include/uapi/linux/citadel.h
@@ -0,0 +1,35 @@
+/*
+ * include/linux/citadel.h
+ *
+ * Copyright (C) 2017 Google Inc
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ */
+
+#ifndef CITADEL_H
+#define CITADEL_H
+
+#include <linux/types.h>
+
+#define CITADEL_IOC_MAGIC 'c'
+
+struct citadel_ioc_tpm_datagram {
+ __u64 buf;
+ __u32 len;
+ __u32 command;
+};
+
+#define CITADEL_IOC_TPM_DATAGRAM _IOW(CITADEL_IOC_MAGIC, 1, \
+ struct citadel_ioc_tpm_datagram)
+#define CITADEL_IOC_RESET _IO(CITADEL_IOC_MAGIC, 2)
+
+#endif /* CITADEL_H */
diff --git a/include/uapi/linux/cryptouser.h b/include/uapi/linux/cryptouser.h
index 79b5ded..11d21fc 100644
--- a/include/uapi/linux/cryptouser.h
+++ b/include/uapi/linux/cryptouser.h
@@ -46,6 +46,7 @@ enum crypto_attr_type_t {
CRYPTOCFGA_REPORT_CIPHER, /* struct crypto_report_cipher */
CRYPTOCFGA_REPORT_AKCIPHER, /* struct crypto_report_akcipher */
CRYPTOCFGA_REPORT_KPP, /* struct crypto_report_kpp */
+ CRYPTOCFGA_REPORT_ACOMP, /* struct crypto_report_acomp */
__CRYPTOCFGA_MAX
#define CRYPTOCFGA_MAX (__CRYPTOCFGA_MAX - 1)
@@ -112,5 +113,9 @@ struct crypto_report_kpp {
char type[CRYPTO_MAX_NAME];
};
+struct crypto_report_acomp {
+ char type[CRYPTO_MAX_NAME];
+};
+
#define CRYPTO_REPORT_MAXSIZE (sizeof(struct crypto_user_alg) + \
sizeof(struct crypto_report_blkcipher))
diff --git a/include/uapi/linux/dma-buf.h b/include/uapi/linux/dma-buf.h
index fb0dedb..59ec672 100644
--- a/include/uapi/linux/dma-buf.h
+++ b/include/uapi/linux/dma-buf.h
@@ -34,7 +34,11 @@ struct dma_buf_sync {
#define DMA_BUF_SYNC_VALID_FLAGS_MASK \
(DMA_BUF_SYNC_RW | DMA_BUF_SYNC_END)
+#define DMA_BUF_NAME_LEN 32
+
#define DMA_BUF_BASE 'b'
#define DMA_BUF_IOCTL_SYNC _IOW(DMA_BUF_BASE, 0, struct dma_buf_sync)
+#define DMA_BUF_SET_NAME _IOW(DMA_BUF_BASE, 5, const char *)
+#define DMA_BUF_GET_NAME _IOR(DMA_BUF_BASE, 6, char *)
#endif
diff --git a/include/uapi/linux/ethtool.h b/include/uapi/linux/ethtool.h
index 5c22e8c..8c5335b 100644
--- a/include/uapi/linux/ethtool.h
+++ b/include/uapi/linux/ethtool.h
@@ -882,13 +882,13 @@ struct ethtool_rx_flow_spec {
static inline __u64 ethtool_get_flow_spec_ring(__u64 ring_cookie)
{
return ETHTOOL_RX_FLOW_SPEC_RING & ring_cookie;
-};
+}
static inline __u64 ethtool_get_flow_spec_ring_vf(__u64 ring_cookie)
{
return (ETHTOOL_RX_FLOW_SPEC_RING_VF & ring_cookie) >>
ETHTOOL_RX_FLOW_SPEC_RING_VF_OFF;
-};
+}
/**
* struct ethtool_rxnfc - command to get or set RX flow classification rules
diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h
index dfcf371..70978a9 100644
--- a/include/uapi/linux/fs.h
+++ b/include/uapi/linux/fs.h
@@ -263,7 +263,8 @@ struct fsxattr {
#define FS_POLICY_FLAGS_PAD_16 0x02
#define FS_POLICY_FLAGS_PAD_32 0x03
#define FS_POLICY_FLAGS_PAD_MASK 0x03
-#define FS_POLICY_FLAGS_VALID 0x03
+#define FS_POLICY_FLAG_DIRECT_KEY 0x04 /* use master key directly */
+#define FS_POLICY_FLAGS_VALID 0x07
/* Encryption algorithms */
#define FS_ENCRYPTION_MODE_INVALID 0
@@ -273,6 +274,9 @@ struct fsxattr {
#define FS_ENCRYPTION_MODE_AES_256_CTS 4
#define FS_ENCRYPTION_MODE_AES_128_CBC 5
#define FS_ENCRYPTION_MODE_AES_128_CTS 6
+#define FS_ENCRYPTION_MODE_SPECK128_256_XTS 7 /* Removed, do not use. */
+#define FS_ENCRYPTION_MODE_SPECK128_256_CTS 8 /* Removed, do not use. */
+#define FS_ENCRYPTION_MODE_ADIANTUM 9
#define FS_ENCRYPTION_MODE_PRIVATE 127
struct fscrypt_policy {
@@ -300,6 +304,22 @@ struct fscrypt_key {
__u32 size;
};
+/* file-based verity support */
+
+#define FS_VERITY_ALG_SHA256 1
+#define FS_VERITY_ALG_CRC32 2
+
+struct fsverity_measurement {
+ __u16 digest_algorithm;
+ __u16 digest_size;
+ __u32 reserved1;
+ __u64 reserved2[3];
+ __u8 digest[];
+};
+
+#define FS_IOC_ENABLE_VERITY _IO('f', 133)
+#define FS_IOC_SET_VERITY_MEASUREMENT _IOW('f', 134, struct fsverity_measurement)
+
/*
* Inode flags (FS_IOC_GETFLAGS / FS_IOC_SETFLAGS)
*
diff --git a/include/uapi/linux/google-easel-comm.h b/include/uapi/linux/google-easel-comm.h
new file mode 100644
index 0000000..627a11c
--- /dev/null
+++ b/include/uapi/linux/google-easel-comm.h
@@ -0,0 +1,229 @@
+/*
+ * Android/Easel coprocessor communication.
+ *
+ * Copyright 2016 Google Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef _UAPI__GOOGLE_EASEL_COMM_H
+#define _UAPI__GOOGLE_EASEL_COMM_H
+
+#include <linux/compiler.h>
+#include <linux/types.h>
+
+/* Maximum message data size 12KB */
+#define EASELCOMM_MAX_MESSAGE_SIZE (12 * 1024)
+
+/* Maximum service count */
+#define EASELCOMM_SERVICE_COUNT 64
+
+/* Easel message identifier. Compatible with libeasel defines. */
+typedef __u64 easelcomm_msgid_t;
+
+struct easelcomm_wait {
+ __s32 timeout_ms; /* timeout in ms; -1 means indefinite */
+};
+
+/*
+ * Userspace/kernel interface message descriptor. libeasel converts
+ * between its representation of messages and these descriptors, plus the
+ * buffer descriptors below, when passing messages between the kernel and
+ * userspace.
+ */
+struct easelcomm_kmsg_desc {
+ /* 64-bit IDs go first for 32/64-bit struct packing conformity */
+ easelcomm_msgid_t message_id; /* message ID */
+ easelcomm_msgid_t in_reply_to; /* msg ID replied to if non-zero */
+ __u32 message_size; /* size in bytes of the message data */
+ __u32 dma_buf_size; /* size of the DMA buffer transfer */
+ __u32 need_reply; /* non-zero if reply requested */
+ __u32 replycode; /* replycode if in_reply_to != 0 */
+ struct easelcomm_wait wait;
+};
+
+enum easelcomm_dma_buffer_type {
+ EASELCOMM_DMA_BUFFER_UNUSED = 0,
+ EASELCOMM_DMA_BUFFER_USER,
+ EASELCOMM_DMA_BUFFER_DMA_BUF
+};
+
+/*
+ * Local buffer descriptor argument for ioctls that read and write message
+ * or DMA buffers. The message_id describes an in-progress local outgoing
+ * message for which the message data is being written or the DMA source
+ * buffer is being set, or a remote incoming message for which the message
+ * data is being read or the DMA destination buffer is being set.
+ * The amount and location of the data being transferred is affected by
+ * the following parameters:
+ * offset: Refers to the byte offset where data starts.
+ * size: Refers to the total byte size of actual data.
+ * width: Refers to the a contiguous set of bytes.
+ * In general: 1 <= width <= size
+ * stride: Refers to the byte offset from the beginning of a contiguous set of
+ * data specified by "width" to the next contiguous set of data.
+ * At the moment: stride >= width.
+ * Parameter examples (assume a 10x10 byte buffer):
+ * 1) Transfer a rectangular 3x3 sub-region from a 10x10 buffer, starting at
+ * offset 5:
+ * CFG: offset=5, size=9, width=3, stride=10
+ * 2) Transfer a full 10x10 buffer (sub-region is same size as buffer):
+ * CFG: offset=0, size=100, width=100, stride=100
+ * 3) Transfer every even row in a 10x10 buffer:
+ * CFG: offset=0, size=50, width=10, stride=20
+ * 4) Transfer every even column in a 10x10 buffer:
+ * CFG: offset=0, size=50, width=1, stride=2
+ *
+ * Note that when using a dma buf:
+ * offset + [ceil(size/width)*stride - (stride-width)] <= buf_size
+ */
+struct easelcomm_kbuf_desc {
+ easelcomm_msgid_t message_id; /* ID of message for this transfer */
+ void __user *buf; /* local buffer source or dest */
+ int dma_buf_fd; /* fd of local dma_buf */
+ int buf_type; /* use enum easelcomm_dma_buffer_type */
+ __u32 buf_size; /* size of the local buffer */
+ struct easelcomm_wait wait;
+ __u32 dma_buf_off; /* offset into the local dma_buf */
+ __u32 dma_buf_width; /* width of region */
+ __u32 dma_buf_stride; /* stride width */
+};
+
+/*
+ * Legacy local buffer descriptor for backwards compatibility with clients which
+ * haven't migrated to the new descriptor/ioctls.
+ */
+struct easelcomm_kbuf_desc_legacy {
+ easelcomm_msgid_t message_id; /* ID of message for this transfer */
+ void __user *buf; /* local buffer source or dest */
+ int dma_buf_fd; /* fd of local dma_buf */
+ int buf_type; /* use enum easelcomm_dma_buffer_type */
+ __u32 buf_size; /* size of the local buffer */
+ struct easelcomm_wait wait;
+};
+
+/*
+ * Kernel driver ioctls. All ioctls return zero for success or -1 for
+ * error with errno set: EINVAL for invalid parameters (dmesg may contain
+ * further explanation), ENOMEM for out of memory, or other codes as described
+ * below.
+ */
+#define EASELCOMM_IOC_MAGIC 0xEA
+
+/*
+ * Register the file descriptor for an easelcomm service. The AP client will
+ * flush any local client and remote server messages associated with the
+ * service at this time, assuming any previous traffic is stale. Registration
+ * by the Easel server does not flush state and will commence processing of
+ * any queued incoming messages previously sent to the service by a client.
+ * Only one fd may be registered for a service on each of the AP and Easel
+ * sides.
+ */
+#define EASELCOMM_IOC_REGISTER _IOW(EASELCOMM_IOC_MAGIC, 0, int)
+
+/*
+ * Start sending an outgoing message. Reads a message descriptor and
+ * returns a modified descriptor with a local message ID assigned. That
+ * ID is then supplied in the buffer descriptors of WRITEBUF and SENDDMA
+ * ioctls as appropriate to supply the message data and local DMA source
+ * information.
+ */
+#define EASELCOMM_IOC_SENDMSG _IOWR(EASELCOMM_IOC_MAGIC, 1, \
+ struct easelcomm_kmsg_desc *)
+/*
+ * Read the message data for an incoming message received from the
+ * remote. The supplied remote message ID was returned by a previous
+ * WAITMSG or WAITREPLY ioctl. The data is read to the specified buffer
+ * in the calling process. If no DMA transfer is requested by the
+ * associated message then successful completion of this ioctl frees the
+ * local kernel copy of the message.
+ */
+#define EASELCOMM_IOC_READDATA _IOW(EASELCOMM_IOC_MAGIC, 10, \
+ struct easelcomm_kbuf_desc)
+#define EASELCOMM_IOC_READDATA_LEGACY _IOW(EASELCOMM_IOC_MAGIC, 2, \
+ struct easelcomm_kbuf_desc_legacy *)
+/*
+ * Write the message data for an outgoing message being sent to the
+ * remote. The supplied remote message ID was returned by a previous
+ * SENDMSG ioctl. The data is written from the specified buffer in the
+ * calling process. This ioctl sends the message to the remote upon
+ * successful completion; even if the message data length is zero, this
+ * ioctl must still follow a SENDMSG to actually send the message. If no
+ * DMA transfer is requested by the associated message then successful
+ * completion of this ioctl frees the local kernel copy of the message.
+ */
+#define EASELCOMM_IOC_WRITEDATA _IOW(EASELCOMM_IOC_MAGIC, 11, \
+ struct easelcomm_kbuf_desc)
+#define EASELCOMM_IOC_WRITEDATA_LEGACY _IOW(EASELCOMM_IOC_MAGIC, 3, \
+ struct easelcomm_kbuf_desc_legacy *)
+/*
+ * Initiate a DMA write for an outgoing message that includes a DMA
+ * transfer. The supplied local message ID was returned by a previous
+ * SENDMSG ioctl. The DMA data will be read from the specified buffer in the
+ * calling process. The ioctl returns once the DMA transfer is complete (or
+ * skipped due to being discarded by the remote). Successful completion of
+ * this ioctl frees the local kernel copy of the message.
+ */
+#define EASELCOMM_IOC_SENDDMA _IOW(EASELCOMM_IOC_MAGIC, 12, \
+ struct easelcomm_kbuf_desc)
+#define EASELCOMM_IOC_SENDDMA_LEGACY _IOW(EASELCOMM_IOC_MAGIC, 4, \
+ struct easelcomm_kbuf_desc_legacy *)
+/*
+ * Specify the local destination for a DMA transfer requested by an incoming
+ * message. The supplied remote message ID was returned by a previous
+ * WAITMSG or WAITREPLY ioctl. The DMA data will be written to the specified
+ * buffer in the calling process; or if a NULL buffer pointer is supplied then
+ * the DMA transfer is discarded. The ioctl returns once the DMA transfer is
+ * complete (or discarded per the preceding). Successful completion of this
+ * ioctl frees the local kernel copy of the message.
+ */
+#define EASELCOMM_IOC_RECVDMA _IOW(EASELCOMM_IOC_MAGIC, 13, \
+ struct easelcomm_kbuf_desc)
+#define EASELCOMM_IOC_RECVDMA_LEGACY _IOW(EASELCOMM_IOC_MAGIC, 5, \
+ struct easelcomm_kbuf_desc_legacy *)
+/*
+ * Wait for and return a descriptor for a reply from the remote to a
+ * local message that requests a reply. The message ID in the supplied
+ * descriptor was returned by a previous SENDMSG ioctl. This ioctl waits
+ * for the remote's reply message and returns a desciptor for that message.
+ * A READDATA ioctl should then be issued to read the message data, followed
+ * by a RECVDMA if a DMA transfer is requested by the reply.
+ */
+#define EASELCOMM_IOC_WAITREPLY _IOWR(EASELCOMM_IOC_MAGIC, 6, \
+ struct easelcomm_kmsg_desc *)
+/*
+ * Wait for and return a descriptor for an incoming message from the remote
+ * (that is not a reply to a local message). This ioctl waits for the next
+ * non-reply message from the remote and returns a desciptor for that message.
+ * A READDATA ioctl should then be issued to read the message data, followed
+ * by a RECVDMA if a DMA transfer is requested by the message.
+ * Returns error ESHUTDOWN if a SHUTDOWN ioctl is issued for the file
+ * descriptor or the file descriptor has been closed.
+ */
+#define EASELCOMM_IOC_WAITMSG _IOWR(EASELCOMM_IOC_MAGIC, 7, \
+ struct easelcomm_kmsg_desc *)
+/*
+ * Shut down the local easelcomm connection for the given file descriptor.
+ * Any other threads blocked on a WAITMSG using the same file descriptor will
+ * return an ESHUTDOWN. Any local in-progress outgoing or incoming messages
+ * for the registered Easel service are discarded.
+ */
+#define EASELCOMM_IOC_SHUTDOWN _IO(EASELCOMM_IOC_MAGIC, 8)
+
+/*
+ * Any local in-progress outgoing or incoming messages for the registered Easel
+ * service are discarded, and the remote side is sent a command to flush its
+ * local messages for the service. Returns when the remote acknowledges the
+ * flush complete. No userspace handler is required to be registered on the
+ * remote to perform the flush.
+ */
+#define EASELCOMM_IOC_FLUSH _IO(EASELCOMM_IOC_MAGIC, 9)
+
+/*
+ * The last close() of an fd also flushes any local messages for the
+ * registered service.
+ */
+
+#endif /* _UAPI__GOOGLE_EASEL_COMM_H */
diff --git a/include/uapi/linux/if_ether.h b/include/uapi/linux/if_ether.h
index cae866f..6475a16 100644
--- a/include/uapi/linux/if_ether.h
+++ b/include/uapi/linux/if_ether.h
@@ -29,6 +29,7 @@
*/
#define ETH_ALEN 6 /* Octets in one ethernet addr */
+#define ETH_TLEN 2 /* Octets in ethernet type field */
#define ETH_HLEN 14 /* Total octets in header. */
#define ETH_ZLEN 60 /* Min. octets in frame sans FCS */
#define ETH_DATA_LEN 1500 /* Max. octets in payload */
diff --git a/include/uapi/linux/input-event-codes.h b/include/uapi/linux/input-event-codes.h
index c1ea699..6313abc 100644
--- a/include/uapi/linux/input-event-codes.h
+++ b/include/uapi/linux/input-event-codes.h
@@ -516,6 +516,11 @@
#define KEY_DEL_EOS 0x1c1
#define KEY_INS_LINE 0x1c2
#define KEY_DEL_LINE 0x1c3
+#define KEY_SIDE_GESTURE 0x1c6
+#define KEY_BLACK_UI_GESTURE 0x1c7
+
+#define KEY_SIDE_GESTURE_RIGHT 0x1ca
+#define KEY_SIDE_GESTURE_LEFT 0x1cb
#define KEY_FN 0x1d0
#define KEY_FN_ESC 0x1d1
@@ -747,6 +752,15 @@
#define ABS_MISC 0x28
+/*
+ * 0x2e is reserved and should not be used in input drivers.
+ * It was used by HID as ABS_MISC+6 and userspace needs to detect if
+ * the next ABS_* event is correct or is just ABS_MISC + n.
+ * We define here ABS_RESERVED so userspace can rely on it and detect
+ * the situation described above.
+ */
+#define ABS_RESERVED 0x2e
+
#define ABS_MT_SLOT 0x2f /* MT slot being modified */
#define ABS_MT_TOUCH_MAJOR 0x30 /* Major axis of touching ellipse */
#define ABS_MT_TOUCH_MINOR 0x31 /* Minor axis (omit if circular) */
@@ -763,6 +777,8 @@
#define ABS_MT_TOOL_X 0x3c /* Center X tool position */
#define ABS_MT_TOOL_Y 0x3d /* Center Y tool position */
+#define ABS_MT_CUSTOM 0x3e /* custom event */
+#define ABS_MT_GRIP 0x3f /* grip touch */
#define ABS_MAX 0x3f
#define ABS_CNT (ABS_MAX+1)
@@ -788,12 +804,15 @@
#define SW_ROTATE_LOCK 0x0c /* set = rotate locked/disabled */
#define SW_LINEIN_INSERT 0x0d /* set = inserted */
#define SW_MUTE_DEVICE 0x0e /* set = device disabled */
+#define SW_GLOVE 0x0f /* set = glove mode */
#define SW_PEN_INSERTED 0x0f /* set = pen inserted */
#define SW_HPHL_OVERCURRENT 0x10 /* set = over current on left hph */
#define SW_HPHR_OVERCURRENT 0x11 /* set = over current on right hph */
#define SW_MICROPHONE2_INSERT 0x12 /* set = inserted */
#define SW_UNSUPPORT_INSERT 0x13 /* set = unsupported device inserted */
-#define SW_MAX 0x20
+#define SW_FLIP 0x15 /* set = flip cover */
+#define SW_CERTIFYHALL 0x1b /* set = certify_hall... */
+#define SW_MAX 0x1F
#define SW_CNT (SW_MAX+1)
/*
diff --git a/include/uapi/linux/input.h b/include/uapi/linux/input.h
index e794f7b..44d8e0d 100644
--- a/include/uapi/linux/input.h
+++ b/include/uapi/linux/input.h
@@ -19,6 +19,81 @@
#include "input-event-codes.h"
/*
+ * sec Log
+ */
+#define SECLOG "[sec_input]"
+#define INPUT_LOG_BUF_SIZE 512
+
+#ifdef CONFIG_SEC_DEBUG_TSP_LOG
+#include <linux/sec_debug.h>
+
+#define input_dbg(mode, dev, fmt, ...) \
+({ \
+ static char input_log_buf[INPUT_LOG_BUF_SIZE]; \
+ snprintf(input_log_buf, sizeof(input_log_buf), "%s %s", SECLOG, fmt); \
+ dev_dbg(dev, input_log_buf, ## __VA_ARGS__); \
+ if (mode) { \
+ if (dev) \
+ snprintf(input_log_buf, sizeof(input_log_buf), "%s %s", \
+ dev_driver_string(dev), dev_name(dev)); \
+ else \
+ snprintf(input_log_buf, sizeof(input_log_buf), "NULL"); \
+ sec_debug_tsp_log_msg(input_log_buf, fmt, ## __VA_ARGS__); \
+ } \
+})
+#define input_info(mode, dev, fmt, ...) \
+({ \
+ static char input_log_buf[INPUT_LOG_BUF_SIZE]; \
+ snprintf(input_log_buf, sizeof(input_log_buf), "%s %s", SECLOG, fmt); \
+ dev_info(dev, input_log_buf, ## __VA_ARGS__); \
+ if (mode) { \
+ if (dev) \
+ snprintf(input_log_buf, sizeof(input_log_buf), "%s %s", \
+ dev_driver_string(dev), dev_name(dev)); \
+ else \
+ snprintf(input_log_buf, sizeof(input_log_buf), "NULL"); \
+ sec_debug_tsp_log_msg(input_log_buf, fmt, ## __VA_ARGS__); \
+ } \
+})
+#define input_err(mode, dev, fmt, ...) \
+({ \
+ static char input_log_buf[INPUT_LOG_BUF_SIZE]; \
+ snprintf(input_log_buf, sizeof(input_log_buf), "%s %s", SECLOG, fmt); \
+ dev_err(dev, input_log_buf, ## __VA_ARGS__); \
+ if (mode) { \
+ if (dev) \
+ snprintf(input_log_buf, sizeof(input_log_buf), "%s %s", \
+ dev_driver_string(dev), dev_name(dev)); \
+ else \
+ snprintf(input_log_buf, sizeof(input_log_buf), "NULL"); \
+ sec_debug_tsp_log_msg(input_log_buf, fmt, ## __VA_ARGS__); \
+ } \
+})
+#define input_log_fix() {}
+#else
+#define input_dbg(mode, dev, fmt, ...) \
+({ \
+ static char input_log_buf[INPUT_LOG_BUF_SIZE]; \
+ snprintf(input_log_buf, sizeof(input_log_buf), "%s %s", SECLOG, fmt); \
+ dev_dbg(dev, input_log_buf, ## __VA_ARGS__); \
+})
+#define input_info(mode, dev, fmt, ...) \
+({ \
+ static char input_log_buf[INPUT_LOG_BUF_SIZE]; \
+ snprintf(input_log_buf, sizeof(input_log_buf), "%s %s", SECLOG, fmt); \
+ dev_info(dev, input_log_buf, ## __VA_ARGS__); \
+})
+#define input_err(mode, dev, fmt, ...) \
+({ \
+ static char input_log_buf[INPUT_LOG_BUF_SIZE]; \
+ snprintf(input_log_buf, sizeof(input_log_buf), "%s %s", SECLOG, fmt); \
+ dev_err(dev, input_log_buf, ## __VA_ARGS__); \
+})
+#define input_log_fix() {}
+#endif
+
+
+/*
* The event structure itself
*/
@@ -61,9 +136,14 @@ struct input_id {
* Note that input core does not clamp reported values to the
* [minimum, maximum] limits, such task is left to userspace.
*
- * Resolution for main axes (ABS_X, ABS_Y, ABS_Z) is reported in
- * units per millimeter (units/mm), resolution for rotational axes
- * (ABS_RX, ABS_RY, ABS_RZ) is reported in units per radian.
+ * The default resolution for main axes (ABS_X, ABS_Y, ABS_Z)
+ * is reported in units per millimeter (units/mm), resolution
+ * for rotational axes (ABS_RX, ABS_RY, ABS_RZ) is reported
+ * in units per radian.
+ * When INPUT_PROP_ACCELEROMETER is set the resolution changes.
+ * The main axes (ABS_X, ABS_Y, ABS_Z) are then reported in
+ * in units per g (units/g) and in units per degree per second
+ * (units/deg/s) for rotational axes (ABS_RX, ABS_RY, ABS_RZ).
*/
struct input_absinfo {
__s32 value;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 4ee67cb9..a0a365c 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -717,6 +717,7 @@ struct kvm_ppc_smmu_info {
#define KVM_TRACE_PAUSE __KVM_DEPRECATED_MAIN_0x07
#define KVM_TRACE_DISABLE __KVM_DEPRECATED_MAIN_0x08
#define KVM_GET_EMULATED_CPUID _IOWR(KVMIO, 0x09, struct kvm_cpuid2)
+#define KVM_GET_MSR_FEATURE_INDEX_LIST _IOWR(KVMIO, 0x0a, struct kvm_msr_list)
/*
* Extension capability list.
@@ -870,6 +871,8 @@ struct kvm_ppc_smmu_info {
#define KVM_CAP_S390_USER_INSTR0 130
#define KVM_CAP_MSI_DEVID 131
#define KVM_CAP_PPC_HTM 132
+#define KVM_CAP_S390_BPB 152
+#define KVM_CAP_GET_MSR_FEATURES 153
#ifdef KVM_CAP_IRQ_ROUTING
diff --git a/include/uapi/linux/loop.h b/include/uapi/linux/loop.h
index c8125ec..23158db 100644
--- a/include/uapi/linux/loop.h
+++ b/include/uapi/linux/loop.h
@@ -88,6 +88,7 @@ struct loop_info64 {
#define LOOP_CHANGE_FD 0x4C06
#define LOOP_SET_CAPACITY 0x4C07
#define LOOP_SET_DIRECT_IO 0x4C08
+#define LOOP_SET_BLOCK_SIZE 0x4C09
/* /dev/loop-control interface */
#define LOOP_CTL_ADD 0x4C80
diff --git a/include/uapi/linux/magic.h b/include/uapi/linux/magic.h
index bd01769..ebb9020 100644
--- a/include/uapi/linux/magic.h
+++ b/include/uapi/linux/magic.h
@@ -85,5 +85,6 @@
#define UDF_SUPER_MAGIC 0x15013346
#define BALLOON_KVM_MAGIC 0x13661366
#define ZSMALLOC_MAGIC 0x58295829
+#define DMA_BUF_MAGIC 0x444d4142 /* "DMAB" */
#endif /* __LINUX_MAGIC_H__ */
diff --git a/include/uapi/linux/mnh-sm.h b/include/uapi/linux/mnh-sm.h
new file mode 100644
index 0000000..7ba9f19
--- /dev/null
+++ b/include/uapi/linux/mnh-sm.h
@@ -0,0 +1,106 @@
+/*
+ *
+ * MNH State Manager Configuration.
+ * Copyright (c) 2016, Intel Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for
+ * more details.
+ *
+ */
+
+#ifndef _UAPI__MNH_SM_H
+#define _UAPI__MNH_SM_H
+
+#define MIPI_TX0 0
+#define MIPI_TX1 1
+#define MIPI_TX_IPU 2
+#define MIPI_RX0 0
+#define MIPI_RX1 1
+#define MIPI_RX2 2
+#define MIPI_RX_IPU 3
+
+#define MIPI_MODE_BYPASS 0
+#define MIPI_MODE_BYPASS_W_IPU 1
+#define MIPI_MODE_FUNCTIONAL 2
+
+#define MNH_MIPI_VC0_EN_MASK 0x1
+#define MNH_MIPI_VC1_EN_MASK 0x2
+#define MNH_MIPI_VC2_EN_MASK 0x4
+#define MNH_MIPI_VC3_EN_MASK 0x8
+#define MNH_MIPI_VC_ALL_EN_MASK 0xf
+
+#define MNH_SM_IOC_MAGIC 'T'
+#define MNH_SM_MAX 8
+
+#define MNH_ION_BUFFER_SIZE SZ_32M
+#define FW_VER_SIZE 24
+
+#define MNH_SM_IOC_GET_STATE \
+ _IOR(MNH_SM_IOC_MAGIC, 1, int *)
+#define MNH_SM_IOC_SET_STATE \
+ _IOW(MNH_SM_IOC_MAGIC, 2, int)
+#define MNH_SM_IOC_WAIT_FOR_STATE \
+ _IOW(MNH_SM_IOC_MAGIC, 3, int)
+#define MNH_SM_IOC_CONFIG_MIPI \
+ _IOW(MNH_SM_IOC_MAGIC, 4, struct mnh_mipi_config *)
+#define MNH_SM_IOC_STOP_MIPI \
+ _IOW(MNH_SM_IOC_MAGIC, 5, struct mnh_mipi_config *)
+#define MNH_SM_IOC_WAIT_FOR_POWER \
+ _IO(MNH_SM_IOC_MAGIC, 6)
+#define MNH_SM_IOC_GET_UPDATE_BUF \
+ _IOR(MNH_SM_IOC_MAGIC, 7, int *)
+#define MNH_SM_IOC_POST_UPDATE_BUF \
+ _IOW(MNH_SM_IOC_MAGIC, 8, struct mnh_update_configs *)
+#define MNH_SM_IOC_GET_FW_VER \
+ _IOR(MNH_SM_IOC_MAGIC, 9, char [FW_VER_SIZE])
+
+enum mnh_sm_state {
+ MNH_STATE_OFF, /* powered off */
+ MNH_STATE_ACTIVE, /* powered on and booted */
+ MNH_STATE_SUSPEND, /* suspended, ddr in self-refresh */
+ MNH_STATE_MAX,
+};
+
+enum mnh_fw_slot {
+ MNH_FW_SLOT_SBL = 0,
+ MNH_FW_SLOT_KERNEL,
+ MNH_FW_SLOT_DTB,
+ MNH_FW_SLOT_RAMDISK,
+ MAX_NR_MNH_FW_SLOTS
+};
+
+struct mnh_mipi_config {
+ /* Tx dev (MIPI sink) MNH_MUX_DEVICE_TX* */
+ int txdev;
+ /* Rx dev (MIPI source) MNH_MUX_DEVICE_RX* */
+ int rxdev;
+ /* RX MIPI transfer rate */
+ int rx_rate;
+ /* TX MIPI transfer rate */
+ int tx_rate;
+ /* Mux mode */
+ int mode;
+ /* virtual channel enable mask */
+ int vc_en_mask;
+};
+
+struct mnh_update_config {
+ /* slot type (dtb, kernel, ramdisk) */
+ enum mnh_fw_slot slot_type;
+ /* slot offset in the ion buffer */
+ unsigned long offset;
+ /* slot size */
+ size_t size;
+};
+
+struct mnh_update_configs {
+ struct mnh_update_config config[MAX_NR_MNH_FW_SLOTS];
+};
+
+#endif /* _UAPI__MNH_SM_H */
diff --git a/include/uapi/linux/nl80211.h b/include/uapi/linux/nl80211.h
index 9399a35..8143af6 100644
--- a/include/uapi/linux/nl80211.h
+++ b/include/uapi/linux/nl80211.h
@@ -2603,6 +2603,8 @@ enum nl80211_attrs {
#define NL80211_ATTR_KEYS NL80211_ATTR_KEYS
#define NL80211_ATTR_FEATURE_FLAGS NL80211_ATTR_FEATURE_FLAGS
+#define NL80211_WIPHY_NAME_MAXLEN 64
+
#define NL80211_MAX_SUPP_RATES 32
#define NL80211_MAX_SUPP_HT_RATES 77
#define NL80211_MAX_SUPP_REG_RULES 64
diff --git a/include/uapi/linux/prctl.h b/include/uapi/linux/prctl.h
index c1af9b3..0ba7cc7 100644
--- a/include/uapi/linux/prctl.h
+++ b/include/uapi/linux/prctl.h
@@ -197,6 +197,18 @@ struct prctl_mm_map {
# define PR_CAP_AMBIENT_LOWER 3
# define PR_CAP_AMBIENT_CLEAR_ALL 4
+/* Per task speculation control */
+#define PR_GET_SPECULATION_CTRL 52
+#define PR_SET_SPECULATION_CTRL 53
+/* Speculation control variants */
+# define PR_SPEC_STORE_BYPASS 0
+/* Return and control values for PR_SET/GET_SPECULATION_CTRL */
+# define PR_SPEC_NOT_AFFECTED 0
+# define PR_SPEC_PRCTL (1UL << 0)
+# define PR_SPEC_ENABLE (1UL << 1)
+# define PR_SPEC_DISABLE (1UL << 2)
+# define PR_SPEC_FORCE_DISABLE (1UL << 3)
+
/* Sets the timerslack for arbitrary threads
* arg2 slack value, 0 means "use default"
* arg3 pid of the thread whose timer slack needs to be set
diff --git a/include/uapi/linux/qseecom.h b/include/uapi/linux/qseecom.h
index f0a26b2..55c71dd 100644
--- a/include/uapi/linux/qseecom.h
+++ b/include/uapi/linux/qseecom.h
@@ -277,17 +277,6 @@ struct qseecom_ce_info_req {
struct qseecom_ce_pipe_entry ce_pipe_entry[MAX_CE_PIPE_PAIR_PER_UNIT];
};
-struct qseecom_ice_data_t {
- int flag;
-};
-
-struct qseecom_encdec_conf_t {
- __le64 start_sector;
- size_t fs_size;
- int index;
- int mode;
-};
-
#define SG_ENTRY_SZ sizeof(struct qseecom_sg_entry)
#define SG_ENTRY_SZ_64BIT sizeof(struct qseecom_sg_entry_64bit)
@@ -396,10 +385,5 @@ struct file;
#define QSEECOM_IOCTL_QUERY_CE_PIPE_INFO \
_IOWR(QSEECOM_IOC_MAGIC, 42, struct qseecom_ce_info_req)
-#define QSEECOM_IOCTL_SET_ICE_INFO \
- _IOWR(QSEECOM_IOC_MAGIC, 43, struct qseecom_ice_data_t)
-
-#define QSEECOM_IOCTL_SET_ENCDEC_INFO \
- _IOWR(QSEECOM_IOC_MAGIC, 44, struct qseecom_encdec_conf_t)
#endif /* _UAPI_QSEECOM_H_ */
diff --git a/include/uapi/linux/seccomp.h b/include/uapi/linux/seccomp.h
index 0f238a4..e4acb61 100644
--- a/include/uapi/linux/seccomp.h
+++ b/include/uapi/linux/seccomp.h
@@ -15,7 +15,9 @@
#define SECCOMP_SET_MODE_FILTER 1
/* Valid flags for SECCOMP_SET_MODE_FILTER */
-#define SECCOMP_FILTER_FLAG_TSYNC 1
+#define SECCOMP_FILTER_FLAG_TSYNC (1UL << 0)
+/* In v4.14+ SECCOMP_FILTER_FLAG_LOG is (1UL << 1) */
+#define SECCOMP_FILTER_FLAG_SPEC_ALLOW (1UL << 2)
/*
* All BPF programs must return a 32-bit value.
diff --git a/include/uapi/linux/snmp.h b/include/uapi/linux/snmp.h
index e7a31f8..3442a26 100644
--- a/include/uapi/linux/snmp.h
+++ b/include/uapi/linux/snmp.h
@@ -55,6 +55,7 @@ enum
IPSTATS_MIB_ECT1PKTS, /* InECT1Pkts */
IPSTATS_MIB_ECT0PKTS, /* InECT0Pkts */
IPSTATS_MIB_CEPKTS, /* InCEPkts */
+ IPSTATS_MIB_REASM_OVERLAPS, /* ReasmOverlaps */
__IPSTATS_MIB_MAX
};
diff --git a/include/uapi/linux/taskstats.h b/include/uapi/linux/taskstats.h
index 2466e55..b48f747 100644
--- a/include/uapi/linux/taskstats.h
+++ b/include/uapi/linux/taskstats.h
@@ -33,7 +33,7 @@
*/
-#define TASKSTATS_VERSION 8
+#define TASKSTATS_VERSION 9
#define TS_COMM_LEN 32 /* should be >= TASK_COMM_LEN
* in linux/sched.h */
@@ -163,6 +163,10 @@ struct taskstats {
/* Delay waiting for memory reclaim */
__u64 freepages_count;
__u64 freepages_delay_total;
+
+ /* Delay waiting for thrashing page */
+ __u64 thrashing_count;
+ __u64 thrashing_delay_total;
};
diff --git a/include/uapi/media/cam_req_mgr.h b/include/uapi/media/cam_req_mgr.h
index 841c40a..a24f126 100644
--- a/include/uapi/media/cam_req_mgr.h
+++ b/include/uapi/media/cam_req_mgr.h
@@ -44,6 +44,7 @@
#define V4L_EVENT_CAM_REQ_MGR_SOF 0
#define V4L_EVENT_CAM_REQ_MGR_ERROR 1
#define V4L_EVENT_CAM_REQ_MGR_SOF_BOOT_TS 2
+#define V4L_EVENT_CAM_REQ_MGR_MAX 3
/* SOF Event status */
#define CAM_REQ_MGR_SOF_EVENT_SUCCESS 0
diff --git a/include/uapi/media/cam_sensor.h b/include/uapi/media/cam_sensor.h
index f5af604..2fcd6f5 100644
--- a/include/uapi/media/cam_sensor.h
+++ b/include/uapi/media/cam_sensor.h
@@ -9,6 +9,25 @@
#define CAM_FLASH_MAX_LED_TRIGGERS 3
#define MAX_OIS_NAME_SIZE 32
#define CAM_CSIPHY_SECURE_MODE_ENABLED 1
+#define MAX_RAINBOW_CONFIG_SIZE 32
+
+enum rainbow_op_type {
+ RAINBOW_SEQ_READ,
+ RAINBOW_RANDOM_READ,
+ RAINBOW_SEQ_WRITE,
+ RAINBOW_RANDOM_WRITE
+};
+
+struct rainbow_config {
+ enum rainbow_op_type operation;
+ uint32_t size;
+ uint32_t reg_addr[MAX_RAINBOW_CONFIG_SIZE];
+ uint32_t reg_data[MAX_RAINBOW_CONFIG_SIZE];
+} __attribute__((packed));
+
+#define RAINBOW_CONFIG \
+ _IOWR('R', 1, struct rainbow_config)
+
/**
* struct cam_sensor_query_cap - capabilities info for sensor
*
@@ -100,6 +119,34 @@ struct cam_cmd_i2c_info {
} __attribute__((packed));
/**
+ * struct cam_cmd_get_ois_data - Contains OIS data read cmd
+ *
+ * @reg_addr : register addr to read data from
+ * @reg_data : number of bytes to read
+ * @query_size_handle : handle to user space query_size address
+ * @query_data_handle : handle to user space query_data address
+ */
+struct cam_cmd_get_ois_data {
+ uint32_t reg_addr;
+ uint32_t reg_data;
+ uint64_t query_size_handle;
+ uint64_t query_data_handle;
+} __attribute__((packed));
+
+/**
+ * struct cam_ois_shift - Contains OIS shift data
+ *
+ * @ois_shift_x : shift in x dim
+ * @ois_shift_y : shift in y dim
+ * @time_readout : time that the shift is read out
+ */
+struct cam_ois_shift {
+ int16_t ois_shift_x;
+ int16_t ois_shift_y;
+ int64_t time_readout;
+} __attribute__((packed));
+
+/**
* struct cam_ois_opcode - Contains OIS opcode
*
* @prog : OIS FW prog register address
@@ -147,6 +194,7 @@ struct cam_cmd_ois_info {
* @data_mask : Data mask if only few bits are valid
* @camera_id : Indicates the slot to which camera
* needs to be probed
+ * @fw_update_flag : Update OIS firmware
* @reserved
*/
struct cam_cmd_probe {
@@ -158,6 +206,7 @@ struct cam_cmd_probe {
uint32_t expected_data;
uint32_t data_mask;
uint16_t camera_id;
+ uint8_t fw_update_flag;
uint16_t reserved;
} __attribute__((packed));
@@ -200,7 +249,7 @@ struct cam_cmd_power {
* @ cmd_type : Command buffer type
* @ data_type : I2C data type
* @ addr_type : I2C address type
- * @ reserved
+ * @ slave_addr : Slave address
*/
struct i2c_rdwr_header {
uint16_t count;
@@ -208,7 +257,7 @@ struct i2c_rdwr_header {
uint8_t cmd_type;
uint8_t data_type;
uint8_t addr_type;
- uint16_t reserved;
+ uint16_t slave_addr;
} __attribute__((packed));
/**
diff --git a/include/uapi/scsi/ufs/ufs.h b/include/uapi/scsi/ufs/ufs.h
index cd82b76..6d8ef80 100644
--- a/include/uapi/scsi/ufs/ufs.h
+++ b/include/uapi/scsi/ufs/ufs.h
@@ -1,7 +1,7 @@
#ifndef UAPI_UFS_H_
#define UAPI_UFS_H_
-#define MAX_QUERY_IDN 0x12
+#define MAX_QUERY_IDN 0x18
/* Flag idn for Query Requests*/
enum flag_idn {
@@ -14,6 +14,8 @@ enum flag_idn {
QUERY_FLAG_IDN_RESERVED2 = 0x07,
QUERY_FLAG_IDN_FPHYRESOURCEREMOVAL = 0x08,
QUERY_FLAG_IDN_BUSY_RTC = 0x09,
+ /* use one reserved bit */
+ QUERY_FLAG_IDN_MANUAL_GC_CONT = 0x0E,
};
/* Attribute idn for Query requests */
@@ -36,6 +38,8 @@ enum attr_idn {
QUERY_ATTR_IDN_SECONDS_PASSED = 0x0F,
QUERY_ATTR_IDN_CNTX_CONF = 0x10,
QUERY_ATTR_IDN_CORR_PRG_BLK_NUM = 0x11,
+ /* use one reserved bit */
+ QUERY_ATTR_IDN_MANUAL_GC_STATUS = 0x17,
};
#define QUERY_ATTR_IDN_BOOT_LU_EN_MAX 0x02
@@ -51,7 +55,8 @@ enum desc_idn {
QUERY_DESC_IDN_RFU_1 = 0x6,
QUERY_DESC_IDN_GEOMETRY = 0x7,
QUERY_DESC_IDN_POWER = 0x8,
- QUERY_DESC_IDN_RFU_2 = 0x9,
+ QUERY_DESC_IDN_HEALTH = 0x9,
+ QUERY_DESC_IDN_RFU_2 = 0xA,
QUERY_DESC_IDN_MAX,
};
@@ -68,4 +73,13 @@ enum query_opcode {
UPIU_QUERY_OPCODE_TOGGLE_FLAG = 0x8,
UPIU_QUERY_OPCODE_MAX,
};
+
+/*
+ * high 16 bits for HPB. E.g.,
+ * opcode = (UFS_IOCTL_QUERY_OPCODE << 16) | UPIU_QUERY_OPCODE_READ_DESC
+ */
+#define UPIU_QUERY_OPCODE_HIGH_HPB 0x5500
+#define UPIU_QUERY_OPCODE_HIGH(opcode) ((opcode) >> 16)
+#define UPIU_QUERY_OPCODE_LOW(opcode) ((opcode) & 0xffff)
+
#endif /* UAPI_UFS_H_ */
diff --git a/include/video/udlfb.h b/include/video/udlfb.h
index f9466fa..2ad9a6d 100644
--- a/include/video/udlfb.h
+++ b/include/video/udlfb.h
@@ -87,7 +87,7 @@ struct dlfb_data {
#define MIN_RAW_PIX_BYTES 2
#define MIN_RAW_CMD_BYTES (RAW_HEADER_BYTES + MIN_RAW_PIX_BYTES)
-#define DL_DEFIO_WRITE_DELAY 5 /* fb_deferred_io.delay in jiffies */
+#define DL_DEFIO_WRITE_DELAY msecs_to_jiffies(HZ <= 300 ? 4 : 10) /* optimal value for 720p video */
#define DL_DEFIO_WRITE_DISABLE (HZ*60) /* "disable" with long delay */
/* remove these once align.h patch is taken into kernel */
diff --git a/init/Kconfig b/init/Kconfig
index e3929edd..faff7b0 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -484,6 +484,25 @@
Say N if unsure.
+config PSI
+ bool "Pressure stall information tracking"
+ help
+ Collect metrics that indicate how overcommitted the CPU, memory,
+ and IO capacity are in the system.
+
+ If you say Y here, the kernel will create /proc/pressure/ with the
+ pressure statistics files cpu, memory, and io. These will indicate
+ the share of walltime in which some or all tasks in the system are
+ delayed due to contention of the respective resource.
+
+ In kernels with cgroup support, cgroups (cgroup2 only) will
+ have cpu.pressure, memory.pressure, and io.pressure files,
+ which aggregate pressure stalls for the grouped tasks only.
+
+ For more details see Documentation/accounting/psi.txt.
+
+ Say N if unsure.
+
endmenu # "CPU/Task time and stats accounting"
menu "RCU Subsystem"
diff --git a/init/init_task.c b/init/init_task.c
index 11f83be1..7b200aa 100644
--- a/init/init_task.c
+++ b/init/init_task.c
@@ -7,6 +7,7 @@
#include <linux/init.h>
#include <linux/fs.h>
#include <linux/mm.h>
+#include <linux/scs.h>
#include <asm/pgtable.h>
#include <asm/uaccess.h>
@@ -14,6 +15,11 @@
static struct signal_struct init_signals = INIT_SIGNALS(init_signals);
static struct sighand_struct init_sighand = INIT_SIGHAND(init_sighand);
+#ifdef CONFIG_SHADOW_CALL_STACK
+unsigned long init_shadow_call_stack[SCS_SIZE / sizeof(long)]
+ __init_task_data __aligned(SCS_SIZE);
+#endif
+
/* Initial task structure */
struct task_struct init_task = INIT_TASK(init_task);
EXPORT_SYMBOL(init_task);
diff --git a/init/main.c b/init/main.c
index f9cd3f0..2aca149 100644
--- a/init/main.c
+++ b/init/main.c
@@ -82,6 +82,7 @@
#include <linux/io.h>
#include <linux/kaiser.h>
#include <linux/cache.h>
+#include <linux/scs.h>
#include <asm/io.h>
#include <asm/bugs.h>
@@ -484,6 +485,8 @@ asmlinkage __visible void __init start_kernel(void)
char *after_dashes;
set_task_stack_end_magic(&init_task);
+ scs_set_init_magic(&init_task);
+
smp_setup_processor_id();
debug_objects_early_init();
@@ -508,8 +511,8 @@ asmlinkage __visible void __init start_kernel(void)
setup_command_line(command_line);
setup_nr_cpu_ids();
setup_per_cpu_areas();
- boot_cpu_state_init();
smp_prepare_boot_cpu(); /* arch-specific boot-cpu hooks */
+ boot_cpu_hotplug_init();
build_all_zonelists(NULL, NULL);
page_alloc_init();
@@ -552,6 +555,14 @@ asmlinkage __visible void __init start_kernel(void)
"Interrupts were enabled *very* early, fixing it\n"))
local_irq_disable();
idr_init_cache();
+
+ /*
+ * Allow workqueue creation and work item queueing/cancelling
+ * early. Work item execution depends on kthreads and starts after
+ * workqueue_init().
+ */
+ workqueue_init_early();
+
rcu_init();
/* trace_printk() and trace points may be used after this */
@@ -637,9 +648,8 @@ asmlinkage __visible void __init start_kernel(void)
security_init();
dbg_late_init();
vfs_caches_init();
+ pagecache_init();
signals_init();
- /* rootfs populating might need page-writeback */
- page_writeback_init();
proc_root_init();
nsfs_init();
cpuset_init();
@@ -849,8 +859,11 @@ static void __init do_initcalls(void)
{
int level;
- for (level = 0; level < ARRAY_SIZE(initcall_levels) - 1; level++)
+ for (level = 0; level < ARRAY_SIZE(initcall_levels) - 1; level++) {
do_initcall_level(level);
+ /* need to finish all async calls before going into next level */
+ async_synchronize_full();
+ }
}
/*
@@ -914,7 +927,7 @@ static int try_to_run_init_process(const char *init_filename)
static noinline void __init kernel_init_freeable(void);
-#if defined(CONFIG_DEBUG_RODATA) || defined(CONFIG_SET_MODULE_RONX)
+#if defined(CONFIG_DEBUG_RODATA) || defined(CONFIG_DEBUG_SET_MODULE_RONX)
bool rodata_enabled __ro_after_init = true;
static int __init set_debug_rodata(char *str)
{
@@ -1006,6 +1019,8 @@ static noinline void __init kernel_init_freeable(void)
smp_prepare_cpus(setup_max_cpus);
+ workqueue_init();
+
do_pre_smp_initcalls();
lockup_detector_init();
diff --git a/ipc/shm.c b/ipc/shm.c
index b626745..9c687cd 100644
--- a/ipc/shm.c
+++ b/ipc/shm.c
@@ -1127,14 +1127,17 @@ long do_shmat(int shmid, char __user *shmaddr, int shmflg,
goto out;
else if ((addr = (ulong)shmaddr)) {
if (addr & (shmlba - 1)) {
- /*
- * Round down to the nearest multiple of shmlba.
- * For sane do_mmap_pgoff() parameters, avoid
- * round downs that trigger nil-page and MAP_FIXED.
- */
- if ((shmflg & SHM_RND) && addr >= shmlba)
- addr &= ~(shmlba - 1);
- else
+ if (shmflg & SHM_RND) {
+ addr &= ~(shmlba - 1); /* round down */
+
+ /*
+ * Ensure that the round-down is non-nil
+ * when remapping. This can happen for
+ * cases when addr < shmlba.
+ */
+ if (!addr && (shmflg & SHM_REMAP))
+ goto out;
+ } else
#ifndef __ARCH_FORCE_SHMLBA
if (addr & ~PAGE_MASK)
#endif
diff --git a/kernel/Makefile b/kernel/Makefile
index 108f5be..b2dbd75 100644
--- a/kernel/Makefile
+++ b/kernel/Makefile
@@ -61,6 +61,7 @@
obj-$(CONFIG_MODULES) += module.o
obj-$(CONFIG_MODULE_SIG) += module_signing.o
obj-$(CONFIG_KALLSYMS) += kallsyms.o
+obj-$(CONFIG_BLDR_DEBUG_LOG) += kdebuginfo.o
obj-$(CONFIG_BSD_PROCESS_ACCT) += acct.o
obj-$(CONFIG_KEXEC_CORE) += kexec_core.o
obj-$(CONFIG_KEXEC) += kexec.o
@@ -105,6 +106,7 @@
obj-$(CONFIG_CPU_PM) += cpu_pm.o
obj-$(CONFIG_BPF) += bpf/
obj-$(CONFIG_CFI_CLANG) += cfi.o
+obj-$(CONFIG_SHADOW_CALL_STACK) += scs.o
obj-$(CONFIG_PERF_EVENTS) += events/
diff --git a/kernel/audit.c b/kernel/audit.c
index da4e7c0e..3461a3d 100644
--- a/kernel/audit.c
+++ b/kernel/audit.c
@@ -742,6 +742,8 @@ static void audit_log_feature_change(int which, u32 old_feature, u32 new_feature
return;
ab = audit_log_start(NULL, GFP_KERNEL, AUDIT_FEATURE_CHANGE);
+ if (!ab)
+ return;
audit_log_task_info(ab, current);
audit_log_format(ab, " feature=%s old=%u new=%u old_lock=%u new_lock=%u res=%d",
audit_feature_names[which], !!old_feature, !!new_feature,
diff --git a/kernel/audit_watch.c b/kernel/audit_watch.c
index 690e1e3..f036b6a 100644
--- a/kernel/audit_watch.c
+++ b/kernel/audit_watch.c
@@ -419,6 +419,13 @@ int audit_add_watch(struct audit_krule *krule, struct list_head **list)
struct path parent_path;
int h, ret = 0;
+ /*
+ * When we will be calling audit_add_to_parent, krule->watch might have
+ * been updated and watch might have been freed.
+ * So we need to keep a reference of watch.
+ */
+ audit_get_watch(watch);
+
mutex_unlock(&audit_filter_mutex);
/* Avoid calling path_lookup under audit_filter_mutex. */
@@ -427,8 +434,10 @@ int audit_add_watch(struct audit_krule *krule, struct list_head **list)
/* caller expects mutex locked */
mutex_lock(&audit_filter_mutex);
- if (ret)
+ if (ret) {
+ audit_put_watch(watch);
return ret;
+ }
/* either find an old parent or attach a new one */
parent = audit_find_parent(d_backing_inode(parent_path.dentry));
@@ -446,6 +455,7 @@ int audit_add_watch(struct audit_krule *krule, struct list_head **list)
*list = &audit_inode_hash[h];
error:
path_put(&parent_path);
+ audit_put_watch(watch);
return ret;
}
diff --git a/kernel/auditfilter.c b/kernel/auditfilter.c
index 85d9cac..cd4f413 100644
--- a/kernel/auditfilter.c
+++ b/kernel/auditfilter.c
@@ -406,7 +406,7 @@ static int audit_field_valid(struct audit_entry *entry, struct audit_field *f)
return -EINVAL;
break;
case AUDIT_EXE:
- if (f->op != Audit_equal)
+ if (f->op != Audit_not_equal && f->op != Audit_equal)
return -EINVAL;
if (entry->rule.listnr != AUDIT_FILTER_EXIT)
return -EINVAL;
diff --git a/kernel/auditsc.c b/kernel/auditsc.c
index 2cd5256..c2aaf53 100644
--- a/kernel/auditsc.c
+++ b/kernel/auditsc.c
@@ -469,6 +469,8 @@ static int audit_filter_rules(struct task_struct *tsk,
break;
case AUDIT_EXE:
result = audit_exe_compare(tsk, rule->exe);
+ if (f->op == Audit_not_equal)
+ result = !result;
break;
case AUDIT_UID:
result = audit_uid_comparator(cred->uid, f->op, f->uid);
diff --git a/kernel/bounds.c b/kernel/bounds.c
index e1d1d195..c37f68d 100644
--- a/kernel/bounds.c
+++ b/kernel/bounds.c
@@ -12,7 +12,7 @@
#include <linux/log2.h>
#include <linux/spinlock_types.h>
-void foo(void)
+int main(void)
{
/* The enum constants to put into include/generated/bounds.h */
DEFINE(NR_PAGEFLAGS, __NR_PAGEFLAGS);
@@ -22,4 +22,6 @@ void foo(void)
#endif
DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t));
/* End of constants */
+
+ return 0;
}
diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 4db6a67..b30ca0f 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -194,7 +194,7 @@ int bpf_percpu_array_copy(struct bpf_map *map, void *key, void *value)
static int array_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
{
struct bpf_array *array = container_of(map, struct bpf_array, map);
- u32 index = *(u32 *)key;
+ u32 index = key ? *(u32 *)key : U32_MAX;
u32 *next = (u32 *)next_key;
if (index >= array->map.max_entries) {
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 27f4f2c..9c86d5d 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -328,12 +328,15 @@ static int htab_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
struct hlist_head *head;
struct htab_elem *l, *next_l;
u32 hash, key_size;
- int i;
+ int i = 0;
WARN_ON_ONCE(!rcu_read_lock_held());
key_size = map->key_size;
+ if (!key)
+ goto find_first_elem;
+
hash = htab_map_hash(key, key_size);
head = select_bucket(htab, hash);
@@ -341,10 +344,8 @@ static int htab_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
/* lookup the key */
l = lookup_elem_raw(head, hash, key, key_size);
- if (!l) {
- i = 0;
+ if (!l)
goto find_first_elem;
- }
/* key was found, get next key in the same bucket */
next_l = hlist_entry_safe(rcu_dereference_raw(hlist_next_rcu(&l->hash_node)),
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index 41aa664..85ea598 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -579,14 +579,18 @@ static int map_get_next_key(union bpf_attr *attr)
goto err_put;
}
- err = -ENOMEM;
- key = kmalloc(map->key_size, GFP_USER);
- if (!key)
- goto err_put;
+ if (ukey) {
+ err = -ENOMEM;
+ key = kmalloc(map->key_size, GFP_USER);
+ if (!key)
+ goto err_put;
- err = -EFAULT;
- if (copy_from_user(key, ukey, map->key_size) != 0)
- goto free_key;
+ err = -EFAULT;
+ if (copy_from_user(key, ukey, map->key_size) != 0)
+ goto free_key;
+ } else {
+ key = NULL;
+ }
err = -ENOMEM;
next_key = kmalloc(map->key_size, GFP_USER);
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index 076e4a0..335c002 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -540,10 +540,11 @@ static bool is_spillable_regtype(enum bpf_reg_type type)
/* check_stack_read/write functions track spill/fill of registers,
* stack boundary and alignment are checked in check_mem_access()
*/
-static int check_stack_write(struct bpf_verifier_state *state, int off,
- int size, int value_regno)
+static int check_stack_write(struct bpf_verifier_env *env,
+ struct bpf_verifier_state *state, int off,
+ int size, int value_regno, int insn_idx)
{
- int i;
+ int i, spi = (MAX_BPF_STACK + off) / BPF_REG_SIZE;
/* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0,
* so it's aligned access and [off, off + size) are within stack limits
*/
@@ -558,15 +559,37 @@ static int check_stack_write(struct bpf_verifier_state *state, int off,
}
/* save register state */
- state->spilled_regs[(MAX_BPF_STACK + off) / BPF_REG_SIZE] =
- state->regs[value_regno];
+ state->spilled_regs[spi] = state->regs[value_regno];
- for (i = 0; i < BPF_REG_SIZE; i++)
+ for (i = 0; i < BPF_REG_SIZE; i++) {
+ if (state->stack_slot_type[MAX_BPF_STACK + off + i] == STACK_MISC &&
+ !env->allow_ptr_leaks) {
+ int *poff = &env->insn_aux_data[insn_idx].sanitize_stack_off;
+ int soff = (-spi - 1) * BPF_REG_SIZE;
+
+ /* detected reuse of integer stack slot with a pointer
+ * which means either llvm is reusing stack slot or
+ * an attacker is trying to exploit CVE-2018-3639
+ * (speculative store bypass)
+ * Have to sanitize that slot with preemptive
+ * store of zero.
+ */
+ if (*poff && *poff != soff) {
+ /* disallow programs where single insn stores
+ * into two different stack slots, since verifier
+ * cannot sanitize them
+ */
+ verbose("insn %d cannot access two stack slots fp%d and fp%d",
+ insn_idx, *poff, soff);
+ return -EINVAL;
+ }
+ *poff = soff;
+ }
state->stack_slot_type[MAX_BPF_STACK + off + i] = STACK_SPILL;
+ }
} else {
/* regular write of data into stack */
- state->spilled_regs[(MAX_BPF_STACK + off) / BPF_REG_SIZE] =
- (struct bpf_reg_state) {};
+ state->spilled_regs[spi] = (struct bpf_reg_state) {};
for (i = 0; i < size; i++)
state->stack_slot_type[MAX_BPF_STACK + off + i] = STACK_MISC;
@@ -747,7 +770,7 @@ static int check_ptr_alignment(struct bpf_verifier_env *env,
* if t==write && value_regno==-1, some unknown value is stored into memory
* if t==read && value_regno==-1, don't care what we read from memory
*/
-static int check_mem_access(struct bpf_verifier_env *env, u32 regno, int off,
+static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regno, int off,
int bpf_size, enum bpf_access_type t,
int value_regno)
{
@@ -843,7 +866,8 @@ static int check_mem_access(struct bpf_verifier_env *env, u32 regno, int off,
verbose("attempt to corrupt spilled pointer on stack\n");
return -EACCES;
}
- err = check_stack_write(state, off, size, value_regno);
+ err = check_stack_write(env, state, off, size,
+ value_regno, insn_idx);
} else {
err = check_stack_read(state, off, size, value_regno);
}
@@ -877,7 +901,7 @@ static int check_mem_access(struct bpf_verifier_env *env, u32 regno, int off,
return err;
}
-static int check_xadd(struct bpf_verifier_env *env, struct bpf_insn *insn)
+static int check_xadd(struct bpf_verifier_env *env, int insn_idx, struct bpf_insn *insn)
{
struct bpf_reg_state *regs = env->cur_state.regs;
int err;
@@ -910,13 +934,13 @@ static int check_xadd(struct bpf_verifier_env *env, struct bpf_insn *insn)
}
/* check whether atomic_add can read the memory */
- err = check_mem_access(env, insn->dst_reg, insn->off,
+ err = check_mem_access(env, insn_idx, insn->dst_reg, insn->off,
BPF_SIZE(insn->code), BPF_READ, -1);
if (err)
return err;
/* check whether atomic_add can write into the same memory */
- return check_mem_access(env, insn->dst_reg, insn->off,
+ return check_mem_access(env, insn_idx, insn->dst_reg, insn->off,
BPF_SIZE(insn->code), BPF_WRITE, -1);
}
@@ -1272,7 +1296,7 @@ static int check_call(struct bpf_verifier_env *env, int func_id, int insn_idx)
* is inferred from register state.
*/
for (i = 0; i < meta.access_size; i++) {
- err = check_mem_access(env, meta.regno, i, BPF_B, BPF_WRITE, -1);
+ err = check_mem_access(env, insn_idx, meta.regno, i, BPF_B, BPF_WRITE, -1);
if (err)
return err;
}
@@ -2895,6 +2919,9 @@ static int do_check(struct bpf_verifier_env *env)
goto process_bpf_exit;
}
+ if (signal_pending(current))
+ return -EAGAIN;
+
if (need_resched())
cond_resched();
@@ -2938,7 +2965,7 @@ static int do_check(struct bpf_verifier_env *env)
/* check that memory (src_reg + off) is readable,
* the state of dst_reg will be updated by this func
*/
- err = check_mem_access(env, insn->src_reg, insn->off,
+ err = check_mem_access(env, insn_idx, insn->src_reg, insn->off,
BPF_SIZE(insn->code), BPF_READ,
insn->dst_reg);
if (err)
@@ -2978,7 +3005,7 @@ static int do_check(struct bpf_verifier_env *env)
enum bpf_reg_type *prev_dst_type, dst_reg_type;
if (BPF_MODE(insn->code) == BPF_XADD) {
- err = check_xadd(env, insn);
+ err = check_xadd(env, insn_idx, insn);
if (err)
return err;
insn_idx++;
@@ -2997,7 +3024,7 @@ static int do_check(struct bpf_verifier_env *env)
dst_reg_type = regs[insn->dst_reg].type;
/* check that memory (dst_reg + off) is writeable */
- err = check_mem_access(env, insn->dst_reg, insn->off,
+ err = check_mem_access(env, insn_idx, insn->dst_reg, insn->off,
BPF_SIZE(insn->code), BPF_WRITE,
insn->src_reg);
if (err)
@@ -3032,7 +3059,7 @@ static int do_check(struct bpf_verifier_env *env)
}
/* check that memory (dst_reg + off) is writeable */
- err = check_mem_access(env, insn->dst_reg, insn->off,
+ err = check_mem_access(env, insn_idx, insn->dst_reg, insn->off,
BPF_SIZE(insn->code), BPF_WRITE,
-1);
if (err)
@@ -3225,7 +3252,7 @@ static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env)
/* hold the map. If the program is rejected by verifier,
* the map will be released by release_maps() or it
* will be used by the valid program until it's unloaded
- * and all maps are released in free_bpf_prog_info()
+ * and all maps are released in free_used_maps()
*/
map = bpf_map_inc(map, false);
if (IS_ERR(map)) {
@@ -3369,6 +3396,34 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
else
continue;
+ if (type == BPF_WRITE &&
+ env->insn_aux_data[i + delta].sanitize_stack_off) {
+ struct bpf_insn patch[] = {
+ /* Sanitize suspicious stack slot with zero.
+ * There are no memory dependencies for this store,
+ * since it's only using frame pointer and immediate
+ * constant of zero
+ */
+ BPF_ST_MEM(BPF_DW, BPF_REG_FP,
+ env->insn_aux_data[i + delta].sanitize_stack_off,
+ 0),
+ /* the original STX instruction will immediately
+ * overwrite the same stack slot with appropriate value
+ */
+ *insn,
+ };
+
+ cnt = ARRAY_SIZE(patch);
+ new_prog = bpf_patch_insn_data(env, i + delta, patch, cnt);
+ if (!new_prog)
+ return -ENOMEM;
+
+ delta += cnt - 1;
+ env->prog = new_prog;
+ insn = new_prog->insnsi + i + delta;
+ continue;
+ }
+
if (env->insn_aux_data[i + delta].ptr_type != PTR_TO_CTX)
continue;
@@ -3629,7 +3684,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr)
vfree(log_buf);
if (!env->prog->aux->used_maps)
/* if we didn't copy map pointers into bpf_prog_info, release
- * them now. Otherwise free_bpf_prog_info() will release them.
+ * them now. Otherwise free_used_maps() will release them.
*/
release_maps(env);
*prog = env->prog;
diff --git a/kernel/cfi.c b/kernel/cfi.c
index 87053e2..6951c25 100644
--- a/kernel/cfi.c
+++ b/kernel/cfi.c
@@ -23,12 +23,12 @@
#define cfi_slowpath_handler __cfi_slowpath
#endif /* CONFIG_CFI_PERMISSIVE */
-static inline void handle_cfi_failure()
+static inline void handle_cfi_failure(void *ptr)
{
#ifdef CONFIG_CFI_PERMISSIVE
- WARN_RATELIMIT(1, "CFI failure:\n");
+ WARN_RATELIMIT(1, "CFI failure (target: [<%px>] %pF):\n", ptr, ptr);
#else
- pr_err("CFI failure:\n");
+ pr_err("CFI failure (target: [<%px>] %pF):\n", ptr, ptr);
BUG();
#endif
}
@@ -282,18 +282,18 @@ void cfi_slowpath_handler(uint64_t id, void *ptr, void *diag)
if (likely(check))
check(id, ptr, diag);
else /* Don't allow unchecked modules */
- handle_cfi_failure();
+ handle_cfi_failure(ptr);
}
EXPORT_SYMBOL(cfi_slowpath_handler);
#endif /* CONFIG_MODULES */
-void cfi_failure_handler(void *data, void *value, void *vtable)
+void cfi_failure_handler(void *data, void *ptr, void *vtable)
{
- handle_cfi_failure();
+ handle_cfi_failure(ptr);
}
EXPORT_SYMBOL(cfi_failure_handler);
-void __cfi_check_fail(void *data, void *value)
+void __cfi_check_fail(void *data, void *ptr)
{
- handle_cfi_failure();
+ handle_cfi_failure(ptr);
}
diff --git a/kernel/cgroup.c b/kernel/cgroup.c
index a83771f..bbee5d4 100644
--- a/kernel/cgroup.c
+++ b/kernel/cgroup.c
@@ -62,6 +62,7 @@
#include <linux/proc_ns.h>
#include <linux/nsproxy.h>
#include <linux/file.h>
+#include <linux/psi.h>
#include <net/sock.h>
#define CREATE_TRACE_POINTS
@@ -361,15 +362,6 @@ static void cgroup_idr_remove(struct idr *idr, int id)
spin_unlock_bh(&cgroup_idr_lock);
}
-static struct cgroup *cgroup_parent(struct cgroup *cgrp)
-{
- struct cgroup_subsys_state *parent_css = cgrp->self.parent;
-
- if (parent_css)
- return container_of(parent_css, struct cgroup, self);
- return NULL;
-}
-
/* subsystems visibly enabled on a cgroup */
static u16 cgroup_control(struct cgroup *cgrp)
{
@@ -487,17 +479,6 @@ static inline bool cgroup_is_dead(const struct cgroup *cgrp)
return !(cgrp->self.flags & CSS_ONLINE);
}
-static void cgroup_get(struct cgroup *cgrp)
-{
- WARN_ON_ONCE(cgroup_is_dead(cgrp));
- css_get(&cgrp->self);
-}
-
-static bool cgroup_tryget(struct cgroup *cgrp)
-{
- return css_tryget(&cgrp->self);
-}
-
struct cgroup_subsys_state *of_css(struct kernfs_open_file *of)
{
struct cgroup *cgrp = of->kn->parent->priv;
@@ -781,7 +762,7 @@ static void css_set_move_task(struct task_struct *task,
*/
WARN_ON_ONCE(task->flags & PF_EXITING);
- rcu_assign_pointer(task->cgroups, to_cset);
+ cgroup_move_task(task, to_cset);
list_add_tail(&task->cg_list, use_mg_tasks ? &to_cset->mg_tasks :
&to_cset->tasks);
}
@@ -3525,6 +3506,21 @@ static int cgroup_events_show(struct seq_file *seq, void *v)
return 0;
}
+#ifdef CONFIG_PSI
+static int cgroup_io_pressure_show(struct seq_file *seq, void *v)
+{
+ return psi_show(seq, &seq_css(seq)->cgroup->psi, PSI_IO);
+}
+static int cgroup_memory_pressure_show(struct seq_file *seq, void *v)
+{
+ return psi_show(seq, &seq_css(seq)->cgroup->psi, PSI_MEM);
+}
+static int cgroup_cpu_pressure_show(struct seq_file *seq, void *v)
+{
+ return psi_show(seq, &seq_css(seq)->cgroup->psi, PSI_CPU);
+}
+#endif
+
static ssize_t cgroup_file_write(struct kernfs_open_file *of, char *buf,
size_t nbytes, loff_t off)
{
@@ -4940,6 +4936,23 @@ static struct cftype cgroup_dfl_base_files[] = {
.file_offset = offsetof(struct cgroup, events_file),
.seq_show = cgroup_events_show,
},
+#ifdef CONFIG_PSI
+ {
+ .name = "io.pressure",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = cgroup_io_pressure_show,
+ },
+ {
+ .name = "memory.pressure",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = cgroup_memory_pressure_show,
+ },
+ {
+ .name = "cpu.pressure",
+ .flags = CFTYPE_NOT_ON_ROOT,
+ .seq_show = cgroup_cpu_pressure_show,
+ },
+#endif
{ } /* terminate */
};
@@ -5045,6 +5058,8 @@ static void css_free_work_fn(struct work_struct *work)
*/
cgroup_put(cgroup_parent(cgrp));
kernfs_put(cgrp->kn);
+ if (cgroup_on_dfl(cgrp))
+ psi_cgroup_free(cgrp);
kfree(cgrp);
} else {
/*
@@ -5314,6 +5329,12 @@ static struct cgroup *cgroup_create(struct cgroup *parent)
if (!cgroup_on_dfl(cgrp))
cgrp->subtree_control = cgroup_control(cgrp);
+ if (cgroup_on_dfl(cgrp)) {
+ ret = psi_cgroup_alloc(cgrp);
+ if (ret)
+ goto out_idr_free;
+ }
+
if (parent)
cgroup_bpf_inherit(cgrp, parent);
@@ -5321,6 +5342,8 @@ static struct cgroup *cgroup_create(struct cgroup *parent)
return cgrp;
+out_idr_free:
+ cgroup_idr_remove(&root->cgroup_idr, cgrp->id);
out_cancel_ref:
percpu_ref_exit(&cgrp->self.refcnt);
out_free_cgrp:
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 3f3b7f8..d9d11f0 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -55,6 +55,7 @@ struct cpuhp_cpu_state {
bool rollback;
bool single;
bool bringup;
+ bool booted_once;
struct hlist_node *node;
enum cpuhp_state cb_state;
int result;
@@ -246,12 +247,6 @@ static struct {
#define cpuhp_lock_acquire() lock_map_acquire(&cpu_hotplug.dep_map)
#define cpuhp_lock_release() lock_map_release(&cpu_hotplug.dep_map)
-void cpu_hotplug_mutex_held(void)
-{
- lockdep_assert_held(&cpu_hotplug.lock);
-}
-EXPORT_SYMBOL(cpu_hotplug_mutex_held);
-
void get_online_cpus(void)
{
might_sleep();
@@ -361,6 +356,85 @@ void cpu_hotplug_enable(void)
EXPORT_SYMBOL_GPL(cpu_hotplug_enable);
#endif /* CONFIG_HOTPLUG_CPU */
+#ifdef CONFIG_HOTPLUG_SMT
+enum cpuhp_smt_control cpu_smt_control __read_mostly = CPU_SMT_ENABLED;
+EXPORT_SYMBOL_GPL(cpu_smt_control);
+
+static bool cpu_smt_available __read_mostly;
+
+void __init cpu_smt_disable(bool force)
+{
+ if (cpu_smt_control == CPU_SMT_FORCE_DISABLED ||
+ cpu_smt_control == CPU_SMT_NOT_SUPPORTED)
+ return;
+
+ if (force) {
+ pr_info("SMT: Force disabled\n");
+ cpu_smt_control = CPU_SMT_FORCE_DISABLED;
+ } else {
+ cpu_smt_control = CPU_SMT_DISABLED;
+ }
+}
+
+/*
+ * The decision whether SMT is supported can only be done after the full
+ * CPU identification. Called from architecture code before non boot CPUs
+ * are brought up.
+ */
+void __init cpu_smt_check_topology_early(void)
+{
+ if (!topology_smt_supported())
+ cpu_smt_control = CPU_SMT_NOT_SUPPORTED;
+}
+
+/*
+ * If SMT was disabled by BIOS, detect it here, after the CPUs have been
+ * brought online. This ensures the smt/l1tf sysfs entries are consistent
+ * with reality. cpu_smt_available is set to true during the bringup of non
+ * boot CPUs when a SMT sibling is detected. Note, this may overwrite
+ * cpu_smt_control's previous setting.
+ */
+void __init cpu_smt_check_topology(void)
+{
+ if (!cpu_smt_available)
+ cpu_smt_control = CPU_SMT_NOT_SUPPORTED;
+}
+
+static int __init smt_cmdline_disable(char *str)
+{
+ cpu_smt_disable(str && !strcmp(str, "force"));
+ return 0;
+}
+early_param("nosmt", smt_cmdline_disable);
+
+static inline bool cpu_smt_allowed(unsigned int cpu)
+{
+ if (topology_is_primary_thread(cpu))
+ return true;
+
+ /*
+ * If the CPU is not a 'primary' thread and the booted_once bit is
+ * set then the processor has SMT support. Store this information
+ * for the late check of SMT support in cpu_smt_check_topology().
+ */
+ if (per_cpu(cpuhp_state, cpu).booted_once)
+ cpu_smt_available = true;
+
+ if (cpu_smt_control == CPU_SMT_ENABLED)
+ return true;
+
+ /*
+ * On x86 it's required to boot all logical CPUs at least once so
+ * that the init code can get a chance to set CR4.MCE on each
+ * CPU. Otherwise, a broadacasted MCE observing CR4.MCE=0b on any
+ * core will shutdown the machine.
+ */
+ return !per_cpu(cpuhp_state, cpu).booted_once;
+}
+#else
+static inline bool cpu_smt_allowed(unsigned int cpu) { return true; }
+#endif
+
/* Need to know about CPUs going up/down? */
int register_cpu_notifier(struct notifier_block *nb)
{
@@ -438,6 +512,16 @@ static int bringup_wait_for_ap(unsigned int cpu)
stop_machine_unpark(cpu);
kthread_unpark(st->thread);
+ /*
+ * SMT soft disabling on X86 requires to bring the CPU out of the
+ * BIOS 'wait for SIPI' state in order to set the CR4.MCE bit. The
+ * CPU marked itself as booted_once in cpu_notify_starting() so the
+ * cpu_smt_allowed() check will now return false if this is not the
+ * primary sibling.
+ */
+ if (!cpu_smt_allowed(cpu))
+ return -ECANCELED;
+
/* Should we go further up ? */
if (st->target > CPUHP_AP_ONLINE_IDLE) {
__cpuhp_kick_ap_work(st);
@@ -826,7 +910,6 @@ static int takedown_cpu(unsigned int cpu)
/* Park the smpboot threads */
kthread_park(per_cpu_ptr(&cpuhp_state, cpu)->thread);
- smpboot_park_threads(cpu);
/*
* Prevent irq alloc/free while the dying cpu reorganizes the
@@ -972,20 +1055,19 @@ static int __ref _cpu_down(unsigned int cpu, int tasks_frozen,
return ret;
}
+static int cpu_down_maps_locked(unsigned int cpu, enum cpuhp_state target)
+{
+ if (cpu_hotplug_disabled)
+ return -EBUSY;
+ return _cpu_down(cpu, 0, target);
+}
+
static int do_cpu_down(unsigned int cpu, enum cpuhp_state target)
{
int err;
cpu_maps_update_begin();
-
- if (cpu_hotplug_disabled) {
- err = -EBUSY;
- goto out;
- }
-
- err = _cpu_down(cpu, 0, target);
-
-out:
+ err = cpu_down_maps_locked(cpu, target);
cpu_maps_update_done();
return err;
}
@@ -1009,6 +1091,7 @@ void notify_cpu_starting(unsigned int cpu)
enum cpuhp_state target = min((int)st->target, CPUHP_AP_ONLINE);
rcu_cpu_starting(cpu); /* Enables RCU usage on this CPU. */
+ st->booted_once = true;
while (st->state < target) {
st->state++;
cpuhp_invoke_callback(cpu, st->state, true, NULL);
@@ -1150,6 +1233,10 @@ static int do_cpu_up(unsigned int cpu, enum cpuhp_state target)
err = -EBUSY;
goto out;
}
+ if (!cpu_smt_allowed(cpu)) {
+ err = -EPERM;
+ goto out;
+ }
err = _cpu_up(cpu, 0, target);
out:
@@ -1461,7 +1548,7 @@ static struct cpuhp_step cpuhp_ap_states[] = {
[CPUHP_AP_SMPBOOT_THREADS] = {
.name = "smpboot/threads:online",
.startup.single = smpboot_unpark_threads,
- .teardown.single = NULL,
+ .teardown.single = smpboot_park_threads,
},
[CPUHP_AP_PERF_ONLINE] = {
.name = "perf:online",
@@ -1916,10 +2003,172 @@ static struct attribute_group cpuhp_cpu_root_attr_group = {
NULL
};
+#ifdef CONFIG_HOTPLUG_SMT
+
+static const char *smt_states[] = {
+ [CPU_SMT_ENABLED] = "on",
+ [CPU_SMT_DISABLED] = "off",
+ [CPU_SMT_FORCE_DISABLED] = "forceoff",
+ [CPU_SMT_NOT_SUPPORTED] = "notsupported",
+};
+
+static ssize_t
+show_smt_control(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ return snprintf(buf, PAGE_SIZE - 2, "%s\n", smt_states[cpu_smt_control]);
+}
+
+static void cpuhp_offline_cpu_device(unsigned int cpu)
+{
+ struct device *dev = get_cpu_device(cpu);
+
+ dev->offline = true;
+ /* Tell user space about the state change */
+ kobject_uevent(&dev->kobj, KOBJ_OFFLINE);
+}
+
+static void cpuhp_online_cpu_device(unsigned int cpu)
+{
+ struct device *dev = get_cpu_device(cpu);
+
+ dev->offline = false;
+ /* Tell user space about the state change */
+ kobject_uevent(&dev->kobj, KOBJ_ONLINE);
+}
+
+static int cpuhp_smt_disable(enum cpuhp_smt_control ctrlval)
+{
+ int cpu, ret = 0;
+
+ cpu_maps_update_begin();
+ for_each_online_cpu(cpu) {
+ if (topology_is_primary_thread(cpu))
+ continue;
+ ret = cpu_down_maps_locked(cpu, CPUHP_OFFLINE);
+ if (ret)
+ break;
+ /*
+ * As this needs to hold the cpu maps lock it's impossible
+ * to call device_offline() because that ends up calling
+ * cpu_down() which takes cpu maps lock. cpu maps lock
+ * needs to be held as this might race against in kernel
+ * abusers of the hotplug machinery (thermal management).
+ *
+ * So nothing would update device:offline state. That would
+ * leave the sysfs entry stale and prevent onlining after
+ * smt control has been changed to 'off' again. This is
+ * called under the sysfs hotplug lock, so it is properly
+ * serialized against the regular offline usage.
+ */
+ cpuhp_offline_cpu_device(cpu);
+ }
+ if (!ret)
+ cpu_smt_control = ctrlval;
+ cpu_maps_update_done();
+ return ret;
+}
+
+static int cpuhp_smt_enable(void)
+{
+ int cpu, ret = 0;
+
+ cpu_maps_update_begin();
+ cpu_smt_control = CPU_SMT_ENABLED;
+ for_each_present_cpu(cpu) {
+ /* Skip online CPUs and CPUs on offline nodes */
+ if (cpu_online(cpu) || !node_online(cpu_to_node(cpu)))
+ continue;
+ ret = _cpu_up(cpu, 0, CPUHP_ONLINE);
+ if (ret)
+ break;
+ /* See comment in cpuhp_smt_disable() */
+ cpuhp_online_cpu_device(cpu);
+ }
+ cpu_maps_update_done();
+ return ret;
+}
+
+static ssize_t
+store_smt_control(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ int ctrlval, ret;
+
+ if (sysfs_streq(buf, "on"))
+ ctrlval = CPU_SMT_ENABLED;
+ else if (sysfs_streq(buf, "off"))
+ ctrlval = CPU_SMT_DISABLED;
+ else if (sysfs_streq(buf, "forceoff"))
+ ctrlval = CPU_SMT_FORCE_DISABLED;
+ else
+ return -EINVAL;
+
+ if (cpu_smt_control == CPU_SMT_FORCE_DISABLED)
+ return -EPERM;
+
+ if (cpu_smt_control == CPU_SMT_NOT_SUPPORTED)
+ return -ENODEV;
+
+ ret = lock_device_hotplug_sysfs();
+ if (ret)
+ return ret;
+
+ if (ctrlval != cpu_smt_control) {
+ switch (ctrlval) {
+ case CPU_SMT_ENABLED:
+ ret = cpuhp_smt_enable();
+ break;
+ case CPU_SMT_DISABLED:
+ case CPU_SMT_FORCE_DISABLED:
+ ret = cpuhp_smt_disable(ctrlval);
+ break;
+ }
+ }
+
+ unlock_device_hotplug();
+ return ret ? ret : count;
+}
+static DEVICE_ATTR(control, 0644, show_smt_control, store_smt_control);
+
+static ssize_t
+show_smt_active(struct device *dev, struct device_attribute *attr, char *buf)
+{
+ bool active = topology_max_smt_threads() > 1;
+
+ return snprintf(buf, PAGE_SIZE - 2, "%d\n", active);
+}
+static DEVICE_ATTR(active, 0444, show_smt_active, NULL);
+
+static struct attribute *cpuhp_smt_attrs[] = {
+ &dev_attr_control.attr,
+ &dev_attr_active.attr,
+ NULL
+};
+
+static const struct attribute_group cpuhp_smt_attr_group = {
+ .attrs = cpuhp_smt_attrs,
+ .name = "smt",
+ NULL
+};
+
+static int __init cpu_smt_state_init(void)
+{
+ return sysfs_create_group(&cpu_subsys.dev_root->kobj,
+ &cpuhp_smt_attr_group);
+}
+
+#else
+static inline int cpu_smt_state_init(void) { return 0; }
+#endif
+
static int __init cpuhp_sysfs_init(void)
{
int cpu, ret;
+ ret = cpu_smt_state_init();
+ if (ret)
+ return ret;
+
ret = sysfs_create_group(&cpu_subsys.dev_root->kobj,
&cpuhp_cpu_root_attr_group);
if (ret)
@@ -2024,9 +2273,12 @@ void __init boot_cpu_init(void)
/*
* Must be called _AFTER_ setting up the per_cpu areas
*/
-void __init boot_cpu_state_init(void)
+void __init boot_cpu_hotplug_init(void)
{
- per_cpu_ptr(&cpuhp_state, smp_processor_id())->state = CPUHP_ONLINE;
+#ifdef CONFIG_SMP
+ this_cpu_write(cpuhp_state.booted_once, true);
+#endif
+ this_cpu_write(cpuhp_state.state, CPUHP_ONLINE);
}
static ATOMIC_NOTIFIER_HEAD(idle_notifier);
diff --git a/kernel/cpuset.c b/kernel/cpuset.c
index af9159a..fa5dbbc 100644
--- a/kernel/cpuset.c
+++ b/kernel/cpuset.c
@@ -815,7 +815,6 @@ static void rebuild_sched_domains_unlocked(void)
cpumask_var_t *doms;
int ndoms;
- cpu_hotplug_mutex_held();
lockdep_assert_held(&cpuset_mutex);
/*
diff --git a/kernel/debug/kdb/kdb_io.c b/kernel/debug/kdb/kdb_io.c
index 3990c1f..d3c5b15 100644
--- a/kernel/debug/kdb/kdb_io.c
+++ b/kernel/debug/kdb/kdb_io.c
@@ -215,7 +215,7 @@ static char *kdb_read(char *buffer, size_t bufsize)
int count;
int i;
int diag, dtab_count;
- int key;
+ int key, buf_size, ret;
static int last_crlf;
diag = kdbgetintenv("DTABCOUNT", &dtab_count);
@@ -343,9 +343,8 @@ static char *kdb_read(char *buffer, size_t bufsize)
else
p_tmp = tmpbuffer;
len = strlen(p_tmp);
- count = kallsyms_symbol_complete(p_tmp,
- sizeof(tmpbuffer) -
- (p_tmp - tmpbuffer));
+ buf_size = sizeof(tmpbuffer) - (p_tmp - tmpbuffer);
+ count = kallsyms_symbol_complete(p_tmp, buf_size);
if (tab == 2 && count > 0) {
kdb_printf("\n%d symbols are found.", count);
if (count > dtab_count) {
@@ -357,9 +356,13 @@ static char *kdb_read(char *buffer, size_t bufsize)
}
kdb_printf("\n");
for (i = 0; i < count; i++) {
- if (WARN_ON(!kallsyms_symbol_next(p_tmp, i)))
+ ret = kallsyms_symbol_next(p_tmp, i, buf_size);
+ if (WARN_ON(!ret))
break;
- kdb_printf("%s ", p_tmp);
+ if (ret != -E2BIG)
+ kdb_printf("%s ", p_tmp);
+ else
+ kdb_printf("%s... ", p_tmp);
*(p_tmp + len) = '\0';
}
if (i >= dtab_count)
diff --git a/kernel/debug/kdb/kdb_main.c b/kernel/debug/kdb/kdb_main.c
index 2a20c0d..12076ff 100644
--- a/kernel/debug/kdb/kdb_main.c
+++ b/kernel/debug/kdb/kdb_main.c
@@ -18,6 +18,7 @@
#include <linux/kmsg_dump.h>
#include <linux/reboot.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/sysrq.h>
#include <linux/smp.h>
#include <linux/utsname.h>
@@ -1564,6 +1565,7 @@ static int kdb_md(int argc, const char **argv)
int symbolic = 0;
int valid = 0;
int phys = 0;
+ int raw = 0;
kdbgetintenv("MDCOUNT", &mdcount);
kdbgetintenv("RADIX", &radix);
@@ -1573,9 +1575,10 @@ static int kdb_md(int argc, const char **argv)
repeat = mdcount * 16 / bytesperword;
if (strcmp(argv[0], "mdr") == 0) {
- if (argc != 2)
+ if (argc == 2 || (argc == 0 && last_addr != 0))
+ valid = raw = 1;
+ else
return KDB_ARGCOUNT;
- valid = 1;
} else if (isdigit(argv[0][2])) {
bytesperword = (int)(argv[0][2] - '0');
if (bytesperword == 0) {
@@ -1611,7 +1614,10 @@ static int kdb_md(int argc, const char **argv)
radix = last_radix;
bytesperword = last_bytesperword;
repeat = last_repeat;
- mdcount = ((repeat * bytesperword) + 15) / 16;
+ if (raw)
+ mdcount = repeat;
+ else
+ mdcount = ((repeat * bytesperword) + 15) / 16;
}
if (argc) {
@@ -1628,7 +1634,10 @@ static int kdb_md(int argc, const char **argv)
diag = kdbgetularg(argv[nextarg], &val);
if (!diag) {
mdcount = (int) val;
- repeat = mdcount * 16 / bytesperword;
+ if (raw)
+ repeat = mdcount;
+ else
+ repeat = mdcount * 16 / bytesperword;
}
}
if (argc >= nextarg+1) {
@@ -1638,8 +1647,15 @@ static int kdb_md(int argc, const char **argv)
}
}
- if (strcmp(argv[0], "mdr") == 0)
- return kdb_mdr(addr, mdcount);
+ if (strcmp(argv[0], "mdr") == 0) {
+ int ret;
+ last_addr = addr;
+ ret = kdb_mdr(addr, mdcount);
+ last_addr += mdcount;
+ last_repeat = mdcount;
+ last_bytesperword = bytesperword; // to make REPEAT happy
+ return ret;
+ }
switch (radix) {
case 10:
@@ -2569,16 +2585,11 @@ static int kdb_summary(int argc, const char **argv)
}
kdb_printf("%02ld:%02ld\n", val.uptime/(60*60), (val.uptime/60)%60);
- /* lifted from fs/proc/proc_misc.c::loadavg_read_proc() */
-
-#define LOAD_INT(x) ((x) >> FSHIFT)
-#define LOAD_FRAC(x) LOAD_INT(((x) & (FIXED_1-1)) * 100)
kdb_printf("load avg %ld.%02ld %ld.%02ld %ld.%02ld\n",
LOAD_INT(val.loads[0]), LOAD_FRAC(val.loads[0]),
LOAD_INT(val.loads[1]), LOAD_FRAC(val.loads[1]),
LOAD_INT(val.loads[2]), LOAD_FRAC(val.loads[2]));
-#undef LOAD_INT
-#undef LOAD_FRAC
+
/* Display in kilobytes */
#define K(x) ((x) << (PAGE_SHIFT - 10))
kdb_printf("\nMemTotal: %8lu kB\nMemFree: %8lu kB\n"
diff --git a/kernel/debug/kdb/kdb_private.h b/kernel/debug/kdb/kdb_private.h
index 75014d7..533e04e 100644
--- a/kernel/debug/kdb/kdb_private.h
+++ b/kernel/debug/kdb/kdb_private.h
@@ -83,7 +83,7 @@ typedef struct __ksymtab {
unsigned long sym_start;
unsigned long sym_end;
} kdb_symtab_t;
-extern int kallsyms_symbol_next(char *prefix_name, int flag);
+extern int kallsyms_symbol_next(char *prefix_name, int flag, int buf_size);
extern int kallsyms_symbol_complete(char *prefix_name, int max_len);
/* Exported Symbols for kernel loadable modules to use. */
diff --git a/kernel/debug/kdb/kdb_support.c b/kernel/debug/kdb/kdb_support.c
index d35cc2d..61cd704 100644
--- a/kernel/debug/kdb/kdb_support.c
+++ b/kernel/debug/kdb/kdb_support.c
@@ -129,13 +129,13 @@ int kdbnearsym(unsigned long addr, kdb_symtab_t *symtab)
}
if (i >= ARRAY_SIZE(kdb_name_table)) {
debug_kfree(kdb_name_table[0]);
- memcpy(kdb_name_table, kdb_name_table+1,
+ memmove(kdb_name_table, kdb_name_table+1,
sizeof(kdb_name_table[0]) *
(ARRAY_SIZE(kdb_name_table)-1));
} else {
debug_kfree(knt1);
knt1 = kdb_name_table[i];
- memcpy(kdb_name_table+i, kdb_name_table+i+1,
+ memmove(kdb_name_table+i, kdb_name_table+i+1,
sizeof(kdb_name_table[0]) *
(ARRAY_SIZE(kdb_name_table)-i-1));
}
@@ -221,11 +221,13 @@ int kallsyms_symbol_complete(char *prefix_name, int max_len)
* Parameters:
* prefix_name prefix of a symbol name to lookup
* flag 0 means search from the head, 1 means continue search.
+ * buf_size maximum length that can be written to prefix_name
+ * buffer
* Returns:
* 1 if a symbol matches the given prefix.
* 0 if no string found
*/
-int kallsyms_symbol_next(char *prefix_name, int flag)
+int kallsyms_symbol_next(char *prefix_name, int flag, int buf_size)
{
int prefix_len = strlen(prefix_name);
static loff_t pos;
@@ -235,10 +237,8 @@ int kallsyms_symbol_next(char *prefix_name, int flag)
pos = 0;
while ((name = kdb_walk_kallsyms(&pos))) {
- if (strncmp(name, prefix_name, prefix_len) == 0) {
- strncpy(prefix_name, name, strlen(name)+1);
- return 1;
- }
+ if (!strncmp(name, prefix_name, prefix_len))
+ return strscpy(prefix_name, name, buf_size);
}
return 0;
}
diff --git a/kernel/delayacct.c b/kernel/delayacct.c
index 435c14a..3fde7df 100644
--- a/kernel/delayacct.c
+++ b/kernel/delayacct.c
@@ -124,9 +124,12 @@ int __delayacct_add_tsk(struct taskstats *d, struct task_struct *tsk)
d->swapin_delay_total = (tmp < d->swapin_delay_total) ? 0 : tmp;
tmp = d->freepages_delay_total + tsk->delays->freepages_delay;
d->freepages_delay_total = (tmp < d->freepages_delay_total) ? 0 : tmp;
+ tmp = d->thrashing_delay_total + tsk->delays->thrashing_delay;
+ d->thrashing_delay_total = (tmp < d->thrashing_delay_total) ? 0 : tmp;
d->blkio_count += tsk->delays->blkio_count;
d->swapin_count += tsk->delays->swapin_count;
d->freepages_count += tsk->delays->freepages_count;
+ d->thrashing_count += tsk->delays->thrashing_count;
spin_unlock_irqrestore(&tsk->delays->lock, flags);
return 0;
@@ -156,3 +159,14 @@ void __delayacct_freepages_end(void)
¤t->delays->freepages_count);
}
+void __delayacct_thrashing_start(void)
+{
+ current->delays->thrashing_start = ktime_get_ns();
+}
+
+void __delayacct_thrashing_end(void)
+{
+ delayacct_end(¤t->delays->thrashing_start,
+ ¤t->delays->thrashing_delay,
+ ¤t->delays->thrashing_count);
+}
diff --git a/kernel/events/callchain.c b/kernel/events/callchain.c
index 411226b..c265f1c 100644
--- a/kernel/events/callchain.c
+++ b/kernel/events/callchain.c
@@ -117,23 +117,20 @@ int get_callchain_buffers(int event_max_stack)
goto exit;
}
- if (count > 1) {
- /* If the allocation failed, give up */
- if (!callchain_cpus_entries)
- err = -ENOMEM;
- /*
- * If requesting per event more than the global cap,
- * return a different error to help userspace figure
- * this out.
- *
- * And also do it here so that we have &callchain_mutex held.
- */
- if (event_max_stack > sysctl_perf_event_max_stack)
- err = -EOVERFLOW;
+ /*
+ * If requesting per event more than the global cap,
+ * return a different error to help userspace figure
+ * this out.
+ *
+ * And also do it here so that we have &callchain_mutex held.
+ */
+ if (event_max_stack > sysctl_perf_event_max_stack) {
+ err = -EOVERFLOW;
goto exit;
}
- err = alloc_callchain_buffers();
+ if (count == 1)
+ err = alloc_callchain_buffers();
exit:
if (err)
atomic_dec(&nr_callchain_events);
diff --git a/kernel/events/core.c b/kernel/events/core.c
index f4e4bf5c..ca84209 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -486,7 +486,7 @@ int perf_cpu_time_max_percent_handler(struct ctl_table *table, int write,
void __user *buffer, size_t *lenp,
loff_t *ppos)
{
- int ret = proc_dointvec(table, write, buffer, lenp, ppos);
+ int ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
if (ret || !write)
return ret;
@@ -667,9 +667,15 @@ static inline void __update_cgrp_time(struct perf_cgroup *cgrp)
static inline void update_cgrp_time_from_cpuctx(struct perf_cpu_context *cpuctx)
{
- struct perf_cgroup *cgrp_out = cpuctx->cgrp;
- if (cgrp_out)
- __update_cgrp_time(cgrp_out);
+ struct perf_cgroup *cgrp = cpuctx->cgrp;
+ struct cgroup_subsys_state *css;
+
+ if (cgrp) {
+ for (css = &cgrp->css; css; css = css->parent) {
+ cgrp = container_of(css, struct perf_cgroup, css);
+ __update_cgrp_time(cgrp);
+ }
+ }
}
static inline void update_cgrp_time_from_event(struct perf_event *event)
@@ -697,6 +703,7 @@ perf_cgroup_set_timestamp(struct task_struct *task,
{
struct perf_cgroup *cgrp;
struct perf_cgroup_info *info;
+ struct cgroup_subsys_state *css;
/*
* ctx->lock held by caller
@@ -707,8 +714,12 @@ perf_cgroup_set_timestamp(struct task_struct *task,
return;
cgrp = perf_cgroup_from_task(task, ctx);
- info = this_cpu_ptr(cgrp->info);
- info->timestamp = ctx->timestamp;
+
+ for (css = &cgrp->css; css; css = css->parent) {
+ cgrp = container_of(css, struct perf_cgroup, css);
+ info = this_cpu_ptr(cgrp->info);
+ info->timestamp = ctx->timestamp;
+ }
}
#define PERF_CGROUP_SWOUT 0x1 /* cgroup switch out every event */
@@ -4278,7 +4289,7 @@ static DEFINE_SPINLOCK(zombie_list_lock);
* object, it will not preserve its functionality. Once the last 'user'
* gives up the object, we'll destroy the thing.
*/
-static int __perf_event_release_kernel(struct perf_event *event)
+static int perf_event_release(struct perf_event *event)
{
struct perf_event_context *ctx = event->ctx;
struct perf_event *child, *tmp;
@@ -4409,13 +4420,10 @@ static int __perf_event_release_kernel(struct perf_event *event)
int perf_event_release_kernel(struct perf_event *event)
{
- int ret;
-
- mutex_lock(&pmus_lock);
- ret = __perf_event_release_kernel(event);
- mutex_unlock(&pmus_lock);
-
- return ret;
+ get_online_cpus();
+ perf_event_release(event);
+ put_online_cpus();
+ return 0;
}
EXPORT_SYMBOL_GPL(perf_event_release_kernel);
@@ -4461,7 +4469,9 @@ EXPORT_SYMBOL_GPL(perf_event_read_value);
static int __perf_read_group_add(struct perf_event *leader,
u64 read_format, u64 *values)
{
+ struct perf_event_context *ctx = leader->ctx;
struct perf_event *sub;
+ unsigned long flags;
int n = 1; /* skip @nr */
int ret;
@@ -4491,12 +4501,15 @@ static int __perf_read_group_add(struct perf_event *leader,
if (read_format & PERF_FORMAT_ID)
values[n++] = primary_event_id(leader);
+ raw_spin_lock_irqsave(&ctx->lock, flags);
+
list_for_each_entry(sub, &leader->sibling_list, group_entry) {
values[n++] += perf_event_count(sub);
if (read_format & PERF_FORMAT_ID)
values[n++] = primary_event_id(sub);
}
+ raw_spin_unlock_irqrestore(&ctx->lock, flags);
return 0;
}
@@ -5693,6 +5706,7 @@ perf_output_sample_ustack(struct perf_output_handle *handle, u64 dump_size,
unsigned long sp;
unsigned int rem;
u64 dyn_size;
+ mm_segment_t fs;
/*
* We dump:
@@ -5710,7 +5724,10 @@ perf_output_sample_ustack(struct perf_output_handle *handle, u64 dump_size,
/* Data. */
sp = perf_user_stack_pointer(regs);
+ fs = get_fs();
+ set_fs(USER_DS);
rem = __output_copy_user(handle, (void *) sp, dump_size);
+ set_fs(fs);
dyn_size = dump_size - rem;
perf_output_skip(handle, rem);
@@ -5830,7 +5847,8 @@ static void perf_output_read_group(struct perf_output_handle *handle,
if (read_format & PERF_FORMAT_TOTAL_TIME_RUNNING)
values[n++] = running;
- if (leader != event)
+ if ((leader != event) &&
+ (leader->state == PERF_EVENT_STATE_ACTIVE))
leader->pmu->read(leader);
values[n++] = perf_event_count(leader);
@@ -7863,6 +7881,8 @@ void perf_tp_event(u16 event_type, u64 count, void *record, int entry_size,
goto unlock;
list_for_each_entry_rcu(event, &ctx->event_list, event_entry) {
+ if (event->cpu != smp_processor_id())
+ continue;
if (event->attr.type != PERF_TYPE_TRACEPOINT)
continue;
if (event->attr.config != entry->type)
@@ -9719,9 +9739,9 @@ static int perf_copy_attr(struct perf_event_attr __user *uattr,
* __u16 sample size limit.
*/
if (attr->sample_stack_user >= USHRT_MAX)
- ret = -EINVAL;
+ return -EINVAL;
else if (!IS_ALIGNED(attr->sample_stack_user, sizeof(u64)))
- ret = -EINVAL;
+ return -EINVAL;
}
if (attr->sample_type & PERF_SAMPLE_REGS_INTR)
@@ -11129,7 +11149,7 @@ static void perf_event_zombie_cleanup(unsigned int cpu)
* PMU expects it to be in an active state
*/
event->state = PERF_EVENT_STATE_ACTIVE;
- __perf_event_release_kernel(event);
+ perf_event_release(event);
spin_lock(&zombie_list_lock);
}
diff --git a/kernel/events/ring_buffer.c b/kernel/events/ring_buffer.c
index 257fa46..017f793 100644
--- a/kernel/events/ring_buffer.c
+++ b/kernel/events/ring_buffer.c
@@ -14,6 +14,7 @@
#include <linux/slab.h>
#include <linux/circ_buf.h>
#include <linux/poll.h>
+#include <linux/nospec.h>
#include "internal.h"
@@ -844,8 +845,10 @@ perf_mmap_to_page(struct ring_buffer *rb, unsigned long pgoff)
return NULL;
/* AUX space */
- if (pgoff >= rb->aux_pgoff)
- return virt_to_page(rb->aux_pages[pgoff - rb->aux_pgoff]);
+ if (pgoff >= rb->aux_pgoff) {
+ int aux_pgoff = array_index_nospec(pgoff - rb->aux_pgoff, rb->aux_nr_pages);
+ return virt_to_page(rb->aux_pages[aux_pgoff]);
+ }
}
return __perf_mmap_to_page(rb, pgoff);
diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c
index a1de021..fbfab57 100644
--- a/kernel/events/uprobes.c
+++ b/kernel/events/uprobes.c
@@ -608,7 +608,7 @@ static int prepare_uprobe(struct uprobe *uprobe, struct file *file,
BUG_ON((uprobe->offset & ~PAGE_MASK) +
UPROBE_SWBP_INSN_SIZE > PAGE_SIZE);
- smp_wmb(); /* pairs with rmb() in find_active_uprobe() */
+ smp_wmb(); /* pairs with the smp_rmb() in handle_swbp() */
set_bit(UPROBE_COPY_INSN, &uprobe->flags);
out:
@@ -1902,10 +1902,18 @@ static void handle_swbp(struct pt_regs *regs)
* After we hit the bp, _unregister + _register can install the
* new and not-yet-analyzed uprobe at the same address, restart.
*/
- smp_rmb(); /* pairs with wmb() in install_breakpoint() */
if (unlikely(!test_bit(UPROBE_COPY_INSN, &uprobe->flags)))
goto out;
+ /*
+ * Pairs with the smp_wmb() in prepare_uprobe().
+ *
+ * Guarantees that if we see the UPROBE_COPY_INSN bit set, then
+ * we must also see the stores to &uprobe->arch performed by the
+ * prepare_uprobe() call.
+ */
+ smp_rmb();
+
/* Tracing handlers use ->utask to communicate with fetch methods */
if (!get_utask())
goto out;
diff --git a/kernel/exit.c b/kernel/exit.c
index 4b4f03a..dd8d6b3 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -54,7 +54,6 @@
#include <linux/writeback.h>
#include <linux/shm.h>
#include <linux/kcov.h>
-#include <linux/cpufreq_times.h>
#include "sched/tune.h"
@@ -172,9 +171,6 @@ void release_task(struct task_struct *p)
{
struct task_struct *leader;
int zap_leader;
-#ifdef CONFIG_CPU_FREQ_TIMES
- cpufreq_task_times_exit(p);
-#endif
repeat:
/* don't need to get the RCU readlock here - the process is dead and
* can't be modifying its own credentials. But shut RCU-lockdep up */
@@ -1680,6 +1676,10 @@ SYSCALL_DEFINE4(wait4, pid_t, upid, int __user *, stat_addr,
__WNOTHREAD|__WCLONE|__WALL))
return -EINVAL;
+ /* -INT_MIN is not defined */
+ if (upid == INT_MIN)
+ return -ESRCH;
+
if (upid == -1)
type = PIDTYPE_MAX;
else if (upid < 0) {
diff --git a/kernel/fork.c b/kernel/fork.c
index 79fdfd8..ae3e276 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -77,6 +77,8 @@
#include <linux/compiler.h>
#include <linux/sysctl.h>
#include <linux/kcov.h>
+#include <linux/cpufreq_times.h>
+#include <linux/scs.h>
#include <asm/pgtable.h>
#include <asm/pgalloc.h>
@@ -184,6 +186,9 @@ static unsigned long *alloc_thread_stack_node(struct task_struct *tsk, int node)
continue;
this_cpu_write(cached_stacks[i], NULL);
+ /* Clear stale pointers from reused stack. */
+ memset(s->addr, 0, THREAD_SIZE);
+
tsk->stack_vm_area = s;
local_irq_enable();
return s->addr;
@@ -339,6 +344,8 @@ void put_task_stack(struct task_struct *tsk)
void free_task(struct task_struct *tsk)
{
+ scs_release(tsk);
+
#ifndef CONFIG_THREAD_INFO_IN_TASK
/*
* The task is finally done with both the stack and thread_info,
@@ -352,6 +359,9 @@ void free_task(struct task_struct *tsk)
*/
WARN_ON_ONCE(atomic_read(&tsk->stack_refcount) != 0);
#endif
+#ifdef CONFIG_CPU_FREQ_TIMES
+ cpufreq_task_times_exit(tsk);
+#endif
rt_mutex_debug_task_free(tsk);
ftrace_graph_exit_task(tsk);
put_seccomp_filter(tsk);
@@ -525,6 +535,7 @@ static struct task_struct *dup_task_struct(struct task_struct *orig, int node)
clear_user_return_notifier(tsk);
clear_tsk_need_resched(tsk);
set_task_stack_end_magic(tsk);
+ scs_task_init(tsk);
#ifdef CONFIG_CC_STACKPROTECTOR
tsk->stack_canary = get_random_long();
@@ -1179,6 +1190,7 @@ static int copy_mm(unsigned long clone_flags, struct task_struct *tsk)
tsk->min_flt = tsk->maj_flt = 0;
tsk->nvcsw = tsk->nivcsw = 0;
+ mm_event_task_init(tsk);
#ifdef CONFIG_DETECT_HUNG_TASK
tsk->last_switch_count = tsk->nvcsw + tsk->nivcsw;
#endif
@@ -1305,7 +1317,9 @@ static int copy_sighand(unsigned long clone_flags, struct task_struct *tsk)
return -ENOMEM;
atomic_set(&sig->count, 1);
+ spin_lock_irq(¤t->sighand->siglock);
memcpy(sig->action, current->sighand->action, sizeof(sig->action));
+ spin_unlock_irq(¤t->sighand->siglock);
return 0;
}
@@ -1543,6 +1557,10 @@ static __latent_entropy struct task_struct *copy_process(
*/
p->clear_child_tid = (clone_flags & CLONE_CHILD_CLEARTID) ? child_tidptr : NULL;
+#ifdef CONFIG_CPU_FREQ_TIMES
+ cpufreq_task_times_init(p);
+#endif
+
ftrace_graph_init_task(p);
rt_mutex_init_task(p);
@@ -1600,13 +1618,15 @@ static __latent_entropy struct task_struct *copy_process(
p->default_timer_slack_ns = current->timer_slack_ns;
+#ifdef CONFIG_PSI
+ p->psi_flags = 0;
+#endif
+
task_io_accounting_init(&p->ioac);
acct_clear_integrals(p);
posix_cpu_timers_init(p);
- p->start_time = ktime_get_ns();
- p->real_start_time = ktime_get_boot_ns();
p->io_context = NULL;
p->audit_context = NULL;
cgroup_fork(p);
@@ -1695,6 +1715,9 @@ static __latent_entropy struct task_struct *copy_process(
retval = copy_thread_tls(clone_flags, stack_start, stack_size, p, tls);
if (retval)
goto bad_fork_cleanup_io;
+ retval = scs_prepare(p, node);
+ if (retval)
+ goto bad_fork_cleanup_thread;
if (pid != &init_struct_pid) {
pid = alloc_pid(p->nsproxy->pid_ns_for_children);
@@ -1767,6 +1790,17 @@ static __latent_entropy struct task_struct *copy_process(
goto bad_fork_free_pid;
/*
+ * From this point on we must avoid any synchronous user-space
+ * communication until we take the tasklist-lock. In particular, we do
+ * not want user-space to be able to predict the process start-time by
+ * stalling fork(2) after we recorded the start_time but before it is
+ * visible to the system.
+ */
+
+ p->start_time = ktime_get_ns();
+ p->real_start_time = ktime_get_boot_ns();
+
+ /*
* Make it visible to the rest of the system, but dont wake it up yet.
* Need tasklist lock for parent etc handling!
*/
diff --git a/kernel/futex.c b/kernel/futex.c
index bb2265a..1b9b37f 100644
--- a/kernel/futex.c
+++ b/kernel/futex.c
@@ -1458,6 +1458,53 @@ futex_wake(u32 __user *uaddr, unsigned int flags, int nr_wake, u32 bitset)
return ret;
}
+static int futex_atomic_op_inuser(unsigned int encoded_op, u32 __user *uaddr)
+{
+ unsigned int op = (encoded_op & 0x70000000) >> 28;
+ unsigned int cmp = (encoded_op & 0x0f000000) >> 24;
+ int oparg = sign_extend32((encoded_op & 0x00fff000) >> 12, 11);
+ int cmparg = sign_extend32(encoded_op & 0x00000fff, 11);
+ int oldval, ret;
+
+ if (encoded_op & (FUTEX_OP_OPARG_SHIFT << 28)) {
+ if (oparg < 0 || oparg > 31) {
+ char comm[sizeof(current->comm)];
+ /*
+ * kill this print and return -EINVAL when userspace
+ * is sane again
+ */
+ pr_info_ratelimited("futex_wake_op: %s tries to shift op by %d; fix this program\n",
+ get_task_comm(comm, current), oparg);
+ oparg &= 31;
+ }
+ oparg = 1 << oparg;
+ }
+
+ if (!access_ok(VERIFY_WRITE, uaddr, sizeof(u32)))
+ return -EFAULT;
+
+ ret = arch_futex_atomic_op_inuser(op, oparg, &oldval, uaddr);
+ if (ret)
+ return ret;
+
+ switch (cmp) {
+ case FUTEX_OP_CMP_EQ:
+ return oldval == cmparg;
+ case FUTEX_OP_CMP_NE:
+ return oldval != cmparg;
+ case FUTEX_OP_CMP_LT:
+ return oldval < cmparg;
+ case FUTEX_OP_CMP_GE:
+ return oldval >= cmparg;
+ case FUTEX_OP_CMP_LE:
+ return oldval <= cmparg;
+ case FUTEX_OP_CMP_GT:
+ return oldval > cmparg;
+ default:
+ return -ENOSYS;
+ }
+}
+
/*
* Wake up all waiters hashed on the physical page that is mapped
* to this virtual address:
@@ -3284,6 +3331,9 @@ SYSCALL_DEFINE6(futex, u32 __user *, uaddr, int, op, u32, val,
}
static void __init futex_detect_cmpxchg(void)
+#if defined(__clang__) && IS_ENABLED(CONFIG_ARM64)
+__attribute__((optnone))
+#endif
{
#ifndef CONFIG_HAVE_FUTEX_CMPXCHG
u32 curval;
diff --git a/kernel/gcov/Kconfig b/kernel/gcov/Kconfig
index 1276aab..0396bd8 100644
--- a/kernel/gcov/Kconfig
+++ b/kernel/gcov/Kconfig
@@ -78,6 +78,11 @@
---help---
Select this option to use the format defined by GCC 4.7.
+config GCOV_FORMAT_CLANG
+ bool "Clang format"
+ ---help---
+ Select this option to use the format defined by Clang.
+
endchoice
endmenu
diff --git a/kernel/gcov/Makefile b/kernel/gcov/Makefile
index 752d648..0879dcf 100644
--- a/kernel/gcov/Makefile
+++ b/kernel/gcov/Makefile
@@ -1,7 +1,8 @@
ccflags-y := -DSRCTREE='"$(srctree)"' -DOBJTREE='"$(objtree)"'
obj-y := base.o fs.o
-obj-$(CONFIG_GCOV_FORMAT_3_4) += gcc_3_4.o
-obj-$(CONFIG_GCOV_FORMAT_4_7) += gcc_4_7.o
+obj-$(CONFIG_GCOV_FORMAT_3_4) += gcc_base.o gcc_3_4.o
+obj-$(CONFIG_GCOV_FORMAT_4_7) += gcc_base.o gcc_4_7.o
+obj-$(CONFIG_GCOV_FORMAT_CLANG) += clang.o
obj-$(CONFIG_GCOV_FORMAT_AUTODETECT) += $(call cc-ifversion, -lt, 0407, \
gcc_3_4.o, gcc_4_7.o)
diff --git a/kernel/gcov/base.c b/kernel/gcov/base.c
index c51a49c..1afef6a 100644
--- a/kernel/gcov/base.c
+++ b/kernel/gcov/base.c
@@ -21,82 +21,8 @@
#include <linux/sched.h>
#include "gcov.h"
-static int gcov_events_enabled;
-static DEFINE_MUTEX(gcov_lock);
-
-/*
- * __gcov_init is called by gcc-generated constructor code for each object
- * file compiled with -fprofile-arcs.
- */
-void __gcov_init(struct gcov_info *info)
-{
- static unsigned int gcov_version;
-
- mutex_lock(&gcov_lock);
- if (gcov_version == 0) {
- gcov_version = gcov_info_version(info);
- /*
- * Printing gcc's version magic may prove useful for debugging
- * incompatibility reports.
- */
- pr_info("version magic: 0x%x\n", gcov_version);
- }
- /*
- * Add new profiling data structure to list and inform event
- * listener.
- */
- gcov_info_link(info);
- if (gcov_events_enabled)
- gcov_event(GCOV_ADD, info);
- mutex_unlock(&gcov_lock);
-}
-EXPORT_SYMBOL(__gcov_init);
-
-/*
- * These functions may be referenced by gcc-generated profiling code but serve
- * no function for kernel profiling.
- */
-void __gcov_flush(void)
-{
- /* Unused. */
-}
-EXPORT_SYMBOL(__gcov_flush);
-
-void __gcov_merge_add(gcov_type *counters, unsigned int n_counters)
-{
- /* Unused. */
-}
-EXPORT_SYMBOL(__gcov_merge_add);
-
-void __gcov_merge_single(gcov_type *counters, unsigned int n_counters)
-{
- /* Unused. */
-}
-EXPORT_SYMBOL(__gcov_merge_single);
-
-void __gcov_merge_delta(gcov_type *counters, unsigned int n_counters)
-{
- /* Unused. */
-}
-EXPORT_SYMBOL(__gcov_merge_delta);
-
-void __gcov_merge_ior(gcov_type *counters, unsigned int n_counters)
-{
- /* Unused. */
-}
-EXPORT_SYMBOL(__gcov_merge_ior);
-
-void __gcov_merge_time_profile(gcov_type *counters, unsigned int n_counters)
-{
- /* Unused. */
-}
-EXPORT_SYMBOL(__gcov_merge_time_profile);
-
-void __gcov_merge_icall_topn(gcov_type *counters, unsigned int n_counters)
-{
- /* Unused. */
-}
-EXPORT_SYMBOL(__gcov_merge_icall_topn);
+int gcov_events_enabled;
+DEFINE_MUTEX(gcov_lock);
void __gcov_exit(void)
{
diff --git a/kernel/gcov/clang.c b/kernel/gcov/clang.c
new file mode 100644
index 0000000..ed6f32e
--- /dev/null
+++ b/kernel/gcov/clang.c
@@ -0,0 +1,548 @@
+/*
+ * Copyright (C) 2017 Google, Inc.
+ * modified from kernel/gcov/gcc_4_7.c
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ *
+ * LLVM uses profiling data that's deliberately similar to GCC, but has a
+ * very different way of exporting that data. LLVM calls llvm_gcov_init() once
+ * per module, and provides a couple of callbacks that we can use to ask for
+ * more data.
+ *
+ * We care about the "writeout" callback, which in turn calls back into
+ * compiler-rt/this module to dump all the gathered coverage data to disk:
+ *
+ * llvm_gcda_start_file()
+ * llvm_gcda_emit_function()
+ * llvm_gcda_emit_arcs()
+ * llvm_gcda_emit_function()
+ * llvm_gcda_emit_arcs()
+ * [... repeats for each function ...]
+ * llvm_gcda_summary_info()
+ * llvm_gcda_end_file()
+ *
+ * This design is much more stateless and unstructured than gcc's, and is
+ * intended to run at process exit. This forces us to keep some local state
+ * about which module we're dealing with at the moment. On the other hand, it
+ * also means we don't depend as much on how LLVM represents profiling data
+ * internally.
+ *
+ * See LLVM's lib/Transforms/Instrumentation/GCOVProfiling.cpp for more
+ * details on how this works, particularly GCOVProfiler::emitProfileArcs(),
+ * GCOVProfiler::insertCounterWriteout(), and
+ * GCOVProfiler::insertFlush().
+ */
+
+#define pr_fmt(fmt) "gcov: " fmt
+
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/printk.h>
+#include <linux/ratelimit.h>
+#include <linux/seq_file.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include "gcov.h"
+
+#define GCOV_TAG_FUNCTION_LENGTH 3
+
+typedef void (*llvm_gcov_callback)(void);
+
+struct gcov_info {
+ struct list_head head;
+
+ const char *filename;
+ unsigned int version;
+ u32 checksum;
+
+ struct list_head functions;
+};
+
+struct gcov_fn_info {
+ struct list_head head;
+
+ u32 ident;
+ u32 checksum;
+ u8 use_extra_checksum;
+ u32 cfg_checksum;
+
+ u32 num_counters;
+ u64 *counters;
+ const char *function_name;
+};
+
+static struct gcov_info *current_info;
+
+static LIST_HEAD(clang_gcov_list);
+
+void llvm_gcov_init(llvm_gcov_callback writeout, llvm_gcov_callback flush)
+{
+ struct gcov_info *info = kzalloc(sizeof(*info), GFP_KERNEL);
+ if (!info) {
+ pr_warn_ratelimited("failed to allocate gcov info\n");
+ return;
+ }
+
+ INIT_LIST_HEAD(&info->head);
+ INIT_LIST_HEAD(&info->functions);
+
+ mutex_lock(&gcov_lock);
+
+ list_add_tail(&info->head, &clang_gcov_list);
+ current_info = info;
+ writeout();
+ current_info = NULL;
+ if (gcov_events_enabled)
+ gcov_event(GCOV_ADD, info);
+
+ mutex_unlock(&gcov_lock);
+}
+EXPORT_SYMBOL(llvm_gcov_init);
+
+void llvm_gcda_start_file(const char *orig_filename, const char version[4],
+ u32 checksum)
+{
+ current_info->filename = orig_filename;
+ memcpy(¤t_info->version, version, sizeof(current_info->version));
+ current_info->checksum = checksum;
+}
+EXPORT_SYMBOL(llvm_gcda_start_file);
+
+void llvm_gcda_emit_function(u32 ident, const char *function_name,
+ u32 func_checksum, u8 use_extra_checksum, u32 cfg_checksum)
+{
+ struct gcov_fn_info *info = kzalloc(sizeof(*info), GFP_KERNEL);
+ if (!info) {
+ pr_warn_ratelimited("failed to allocate gcov function info for %s\n",
+ function_name ?: "UNKNOWN");
+ return;
+ }
+
+ INIT_LIST_HEAD(&info->head);
+ info->ident = ident;
+ info->checksum = func_checksum;
+ info->use_extra_checksum = use_extra_checksum;
+ info->cfg_checksum = cfg_checksum;
+ if (function_name)
+ info->function_name = kstrdup(function_name, GFP_KERNEL);
+
+ list_add_tail(&info->head, ¤t_info->functions);
+}
+EXPORT_SYMBOL(llvm_gcda_emit_function);
+
+void llvm_gcda_emit_arcs(u32 num_counters, u64 *counters)
+{
+ struct gcov_fn_info *info = list_last_entry(¤t_info->functions,
+ struct gcov_fn_info, head);
+
+ info->num_counters = num_counters;
+ info->counters = counters;
+}
+EXPORT_SYMBOL(llvm_gcda_emit_arcs);
+
+void llvm_gcda_summary_info(void)
+{
+}
+EXPORT_SYMBOL(llvm_gcda_summary_info);
+
+void llvm_gcda_end_file(void)
+{
+}
+EXPORT_SYMBOL(llvm_gcda_end_file);
+
+/**
+ * gcov_info_filename - return info filename
+ * @info: profiling data set
+ */
+const char *gcov_info_filename(struct gcov_info *info)
+{
+ return info->filename;
+}
+
+/**
+ * gcov_info_version - return info version
+ * @info: profiling data set
+ */
+unsigned int gcov_info_version(struct gcov_info *info)
+{
+ return info->version;
+}
+
+/**
+ * gcov_info_next - return next profiling data set
+ * @info: profiling data set
+ *
+ * Returns next gcov_info following @info or first gcov_info in the chain if
+ * @info is %NULL.
+ */
+struct gcov_info *gcov_info_next(struct gcov_info *info)
+{
+ if (!info)
+ return list_first_entry_or_null(&clang_gcov_list,
+ struct gcov_info, head);
+ if (list_is_last(&info->head, &clang_gcov_list))
+ return NULL;
+ return list_next_entry(info, head);
+}
+
+/**
+ * gcov_info_link - link/add profiling data set to the list
+ * @info: profiling data set
+ */
+void gcov_info_link(struct gcov_info *info)
+{
+ list_add_tail(&info->head, &clang_gcov_list);
+}
+
+/**
+ * gcov_info_unlink - unlink/remove profiling data set from the list
+ * @prev: previous profiling data set
+ * @info: profiling data set
+ */
+void gcov_info_unlink(struct gcov_info *prev, struct gcov_info *info)
+{
+ if (prev)
+ list_del(&prev->head);
+}
+
+/* Symbolic links to be created for each profiling data file. */
+const struct gcov_link gcov_link[] = {
+ { OBJ_TREE, "gcno" }, /* Link to .gcno file in $(objtree). */
+ { 0, NULL},
+};
+
+/**
+ * gcov_info_reset - reset profiling data to zero
+ * @info: profiling data set
+ */
+void gcov_info_reset(struct gcov_info *info)
+{
+ struct gcov_fn_info *fn;
+
+ list_for_each_entry(fn, &info->functions, head)
+ memset(fn->counters, 0,
+ sizeof(fn->counters[0]) * fn->num_counters);
+}
+
+/**
+ * gcov_info_is_compatible - check if profiling data can be added
+ * @info1: first profiling data set
+ * @info2: second profiling data set
+ *
+ * Returns non-zero if profiling data can be added, zero otherwise.
+ */
+int gcov_info_is_compatible(struct gcov_info *info1, struct gcov_info *info2)
+{
+ return (info1->checksum == info2->checksum);
+}
+
+/**
+ * gcov_info_add - add up profiling data
+ * @dest: profiling data set to which data is added
+ * @source: profiling data set which is added
+ *
+ * Adds profiling counts of @source to @dest.
+ */
+void gcov_info_add(struct gcov_info *dst, struct gcov_info *src)
+{
+ struct gcov_fn_info *dfn_ptr;
+ struct gcov_fn_info *sfn_ptr = list_first_entry_or_null(&src->functions,
+ struct gcov_fn_info, head);
+
+ list_for_each_entry(dfn_ptr, &dst->functions, head) {
+ u32 i;
+ for (i = 0; i < sfn_ptr->num_counters; i++)
+ dfn_ptr->counters[i] += sfn_ptr->counters[i];
+
+ if (!list_is_last(&sfn_ptr->head, &src->functions))
+ sfn_ptr = list_next_entry(sfn_ptr, head);
+ }
+}
+
+static struct gcov_fn_info *gcov_fn_info_dup(struct gcov_fn_info *fn)
+{
+ struct gcov_fn_info *fn_dup = kmemdup(fn, sizeof(*fn),
+ GFP_KERNEL);
+ if (!fn_dup)
+ return NULL;
+ INIT_LIST_HEAD(&fn_dup->head);
+
+ fn_dup->function_name = kstrdup(fn->function_name, GFP_KERNEL);
+ if (!fn_dup->function_name)
+ goto err_name;
+
+ fn_dup->counters = kmemdup(fn->counters,
+ fn->num_counters * sizeof(fn->counters[0]),
+ GFP_KERNEL);
+ if (!fn_dup->counters)
+ goto err_counters;
+
+ return fn_dup;
+
+err_counters:
+ kfree(fn_dup->function_name);
+err_name:
+ kfree(fn_dup);
+ return NULL;
+}
+
+/**
+ * gcov_info_dup - duplicate profiling data set
+ * @info: profiling data set to duplicate
+ *
+ * Return newly allocated duplicate on success, %NULL on error.
+ */
+struct gcov_info *gcov_info_dup(struct gcov_info *info)
+{
+ struct gcov_info *dup;
+ struct gcov_fn_info *fn, *tmp;
+
+ dup = kmemdup(info, sizeof(*dup), GFP_KERNEL);
+ if (!dup)
+ return NULL;
+ INIT_LIST_HEAD(&dup->head);
+ INIT_LIST_HEAD(&dup->functions);
+ dup->filename = kstrdup(info->filename, GFP_KERNEL);
+
+ list_for_each_entry_safe(fn, tmp, &info->functions, head) {
+ struct gcov_fn_info *fn_dup = gcov_fn_info_dup(fn);
+ if (!fn_dup)
+ goto err;
+ list_add_tail(&fn_dup->head, &dup->functions);
+ }
+
+ return dup;
+
+err:
+ gcov_info_free(dup);
+ return NULL;
+}
+
+/**
+ * gcov_info_free - release memory for profiling data set duplicate
+ * @info: profiling data set duplicate to free
+ */
+void gcov_info_free(struct gcov_info *info)
+{
+ struct gcov_fn_info *fn, *tmp;
+
+ list_for_each_entry_safe(fn, tmp, &info->functions, head) {
+ kfree(fn->function_name);
+ kfree(fn->counters);
+ list_del(&fn->head);
+ kfree(fn);
+ }
+ kfree(info->filename);
+ kfree(info);
+}
+
+#define ITER_STRIDE PAGE_SIZE
+
+/**
+ * struct gcov_iterator - specifies current file position in logical records
+ * @info: associated profiling data
+ * @buffer: buffer containing file data
+ * @size: size of buffer
+ * @pos: current position in file
+ */
+struct gcov_iterator {
+ struct gcov_info *info;
+ void *buffer;
+ size_t size;
+ loff_t pos;
+};
+
+/**
+ * store_gcov_u32 - store 32 bit number in gcov format to buffer
+ * @buffer: target buffer or NULL
+ * @off: offset into the buffer
+ * @v: value to be stored
+ *
+ * Number format defined by gcc: numbers are recorded in the 32 bit
+ * unsigned binary form of the endianness of the machine generating the
+ * file. Returns the number of bytes stored. If @buffer is %NULL, doesn't
+ * store anything.
+ */
+static size_t store_gcov_u32(void *buffer, size_t off, u32 v)
+{
+ u32 *data;
+
+ if (buffer) {
+ data = buffer + off;
+ *data = v;
+ }
+
+ return sizeof(*data);
+}
+
+/**
+ * store_gcov_u64 - store 64 bit number in gcov format to buffer
+ * @buffer: target buffer or NULL
+ * @off: offset into the buffer
+ * @v: value to be stored
+ *
+ * Number format defined by gcc: numbers are recorded in the 32 bit
+ * unsigned binary form of the endianness of the machine generating the
+ * file. 64 bit numbers are stored as two 32 bit numbers, the low part
+ * first. Returns the number of bytes stored. If @buffer is %NULL, doesn't store
+ * anything.
+ */
+static size_t store_gcov_u64(void *buffer, size_t off, u64 v)
+{
+ u32 *data;
+
+ if (buffer) {
+ data = buffer + off;
+
+ data[0] = (v & 0xffffffffUL);
+ data[1] = (v >> 32);
+ }
+
+ return sizeof(*data) * 2;
+}
+
+/**
+ * convert_to_gcda - convert profiling data set to gcda file format
+ * @buffer: the buffer to store file data or %NULL if no data should be stored
+ * @info: profiling data set to be converted
+ *
+ * Returns the number of bytes that were/would have been stored into the buffer.
+ */
+static size_t convert_to_gcda(char *buffer, struct gcov_info *info)
+{
+ struct gcov_fn_info *fi_ptr;
+ size_t pos = 0;
+
+ /* File header. */
+ pos += store_gcov_u32(buffer, pos, GCOV_DATA_MAGIC);
+ pos += store_gcov_u32(buffer, pos, info->version);
+ pos += store_gcov_u32(buffer, pos, info->checksum);
+
+ list_for_each_entry(fi_ptr, &info->functions, head) {
+ u32 i;
+ u32 len = 2;
+ if (fi_ptr->use_extra_checksum)
+ len++;
+
+ pos += store_gcov_u32(buffer, pos, GCOV_TAG_FUNCTION);
+ pos += store_gcov_u32(buffer, pos, len);
+ pos += store_gcov_u32(buffer, pos, fi_ptr->ident);
+ pos += store_gcov_u32(buffer, pos, fi_ptr->checksum);
+ if (fi_ptr->use_extra_checksum)
+ pos += store_gcov_u32(buffer, pos, fi_ptr->cfg_checksum);
+
+ pos += store_gcov_u32(buffer, pos, GCOV_TAG_COUNTER_BASE);
+ pos += store_gcov_u32(buffer, pos, fi_ptr->num_counters * 2);
+ for (i = 0; i < fi_ptr->num_counters; i++)
+ pos += store_gcov_u64(buffer, pos, fi_ptr->counters[i]);
+ }
+
+ return pos;
+}
+
+/**
+ * gcov_iter_new - allocate and initialize profiling data iterator
+ * @info: profiling data set to be iterated
+ *
+ * Return file iterator on success, %NULL otherwise.
+ */
+struct gcov_iterator *gcov_iter_new(struct gcov_info *info)
+{
+ struct gcov_iterator *iter;
+
+ iter = kzalloc(sizeof(struct gcov_iterator), GFP_KERNEL);
+ if (!iter)
+ goto err_free;
+
+ iter->info = info;
+ /* Dry-run to get the actual buffer size. */
+ iter->size = convert_to_gcda(NULL, info);
+ iter->buffer = vmalloc(iter->size);
+ if (!iter->buffer)
+ goto err_free;
+
+ convert_to_gcda(iter->buffer, info);
+
+ return iter;
+
+err_free:
+ kfree(iter);
+ return NULL;
+}
+
+
+/**
+ * gcov_iter_get_info - return profiling data set for given file iterator
+ * @iter: file iterator
+ */
+void gcov_iter_free(struct gcov_iterator *iter)
+{
+ vfree(iter->buffer);
+ kfree(iter);
+}
+
+/**
+ * gcov_iter_get_info - return profiling data set for given file iterator
+ * @iter: file iterator
+ */
+struct gcov_info *gcov_iter_get_info(struct gcov_iterator *iter)
+{
+ return iter->info;
+}
+
+/**
+ * gcov_iter_start - reset file iterator to starting position
+ * @iter: file iterator
+ */
+void gcov_iter_start(struct gcov_iterator *iter)
+{
+ iter->pos = 0;
+}
+
+/**
+ * gcov_iter_next - advance file iterator to next logical record
+ * @iter: file iterator
+ *
+ * Return zero if new position is valid, non-zero if iterator has reached end.
+ */
+int gcov_iter_next(struct gcov_iterator *iter)
+{
+ if (iter->pos < iter->size)
+ iter->pos += ITER_STRIDE;
+
+ if (iter->pos >= iter->size)
+ return -EINVAL;
+
+ return 0;
+}
+
+/**
+ * gcov_iter_write - write data for current pos to seq_file
+ * @iter: file iterator
+ * @seq: seq_file handle
+ *
+ * Return zero on success, non-zero otherwise.
+ */
+int gcov_iter_write(struct gcov_iterator *iter, struct seq_file *seq)
+{
+ size_t len;
+
+ if (iter->pos >= iter->size)
+ return -EINVAL;
+
+ len = ITER_STRIDE;
+ if (iter->pos + len > iter->size)
+ len = iter->size - iter->pos;
+
+ seq_write(seq, iter->buffer + iter->pos, len);
+
+ return 0;
+}
diff --git a/kernel/gcov/gcc_base.c b/kernel/gcov/gcc_base.c
new file mode 100644
index 0000000..93efe4f
--- /dev/null
+++ b/kernel/gcov/gcc_base.c
@@ -0,0 +1,77 @@
+#include <linux/kernel.h>
+#include <linux/mutex.h>
+#include "gcov.h"
+
+/*
+ * __gcov_init is called by gcc-generated constructor code for each object
+ * file compiled with -fprofile-arcs.
+ */
+void __gcov_init(struct gcov_info *info)
+{
+ static unsigned int gcov_version;
+
+ mutex_lock(&gcov_lock);
+ if (gcov_version == 0) {
+ gcov_version = gcov_info_version(info);
+ /*
+ * Printing gcc's version magic may prove useful for debugging
+ * incompatibility reports.
+ */
+ pr_info("version magic: 0x%x\n", gcov_version);
+ }
+ /*
+ * Add new profiling data structure to list and inform event
+ * listener.
+ */
+ gcov_info_link(info);
+ if (gcov_events_enabled)
+ gcov_event(GCOV_ADD, info);
+ mutex_unlock(&gcov_lock);
+}
+EXPORT_SYMBOL(__gcov_init);
+
+/*
+ * These functions may be referenced by gcc-generated profiling code but serve
+ * no function for kernel profiling.
+ */
+void __gcov_flush(void)
+{
+ /* Unused. */
+}
+EXPORT_SYMBOL(__gcov_flush);
+
+void __gcov_merge_add(gcov_type *counters, unsigned int n_counters)
+{
+ /* Unused. */
+}
+EXPORT_SYMBOL(__gcov_merge_add);
+
+void __gcov_merge_single(gcov_type *counters, unsigned int n_counters)
+{
+ /* Unused. */
+}
+EXPORT_SYMBOL(__gcov_merge_single);
+
+void __gcov_merge_delta(gcov_type *counters, unsigned int n_counters)
+{
+ /* Unused. */
+}
+EXPORT_SYMBOL(__gcov_merge_delta);
+
+void __gcov_merge_ior(gcov_type *counters, unsigned int n_counters)
+{
+ /* Unused. */
+}
+EXPORT_SYMBOL(__gcov_merge_ior);
+
+void __gcov_merge_time_profile(gcov_type *counters, unsigned int n_counters)
+{
+ /* Unused. */
+}
+EXPORT_SYMBOL(__gcov_merge_time_profile);
+
+void __gcov_merge_icall_topn(gcov_type *counters, unsigned int n_counters)
+{
+ /* Unused. */
+}
+EXPORT_SYMBOL(__gcov_merge_icall_topn);
diff --git a/kernel/gcov/gcov.h b/kernel/gcov/gcov.h
index 92c8e22..fbec3b8 100644
--- a/kernel/gcov/gcov.h
+++ b/kernel/gcov/gcov.h
@@ -82,4 +82,7 @@ struct gcov_link {
};
extern const struct gcov_link gcov_link[];
+extern int gcov_events_enabled;
+extern struct mutex gcov_lock;
+
#endif /* GCOV_H */
diff --git a/kernel/irq/chip.c b/kernel/irq/chip.c
index f30110e..d7d2b00 100644
--- a/kernel/irq/chip.c
+++ b/kernel/irq/chip.c
@@ -326,11 +326,12 @@ void unmask_threaded_irq(struct irq_desc *desc)
* handler. The handler function is called inside the calling
* threads context.
*/
-void handle_nested_irq(unsigned int irq)
+bool handle_nested_irq(unsigned int irq)
{
struct irq_desc *desc = irq_to_desc(irq);
struct irqaction *action;
irqreturn_t action_ret;
+ bool handled = false;
might_sleep();
@@ -355,8 +356,11 @@ void handle_nested_irq(unsigned int irq)
raw_spin_lock_irq(&desc->lock);
irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
+ handled = true;
+
out_unlock:
raw_spin_unlock_irq(&desc->lock);
+ return handled;
}
EXPORT_SYMBOL_GPL(handle_nested_irq);
@@ -403,8 +407,10 @@ static bool irq_may_run(struct irq_desc *desc)
* Note: The caller is expected to handle the ack, clear, mask and
* unmask issues if necessary.
*/
-void handle_simple_irq(struct irq_desc *desc)
+bool handle_simple_irq(struct irq_desc *desc)
{
+ bool handled = false;
+
raw_spin_lock(&desc->lock);
if (!irq_may_run(desc))
@@ -420,8 +426,11 @@ void handle_simple_irq(struct irq_desc *desc)
kstat_incr_irqs_this_cpu(desc);
handle_irq_event(desc);
+ handled = true;
+
out_unlock:
raw_spin_unlock(&desc->lock);
+ return handled;
}
EXPORT_SYMBOL_GPL(handle_simple_irq);
@@ -438,9 +447,10 @@ EXPORT_SYMBOL_GPL(handle_simple_irq);
* Note: Like handle_simple_irq, the caller is expected to handle
* the ack, clear, mask and unmask issues if necessary.
*/
-void handle_untracked_irq(struct irq_desc *desc)
+bool handle_untracked_irq(struct irq_desc *desc)
{
unsigned int flags = 0;
+ bool handled = false;
raw_spin_lock(&desc->lock);
@@ -462,9 +472,11 @@ void handle_untracked_irq(struct irq_desc *desc)
raw_spin_lock(&desc->lock);
irqd_clear(&desc->irq_data, IRQD_IRQ_INPROGRESS);
+ handled = true;
out_unlock:
raw_spin_unlock(&desc->lock);
+ return handled;
}
EXPORT_SYMBOL_GPL(handle_untracked_irq);
@@ -495,8 +507,10 @@ static void cond_unmask_irq(struct irq_desc *desc)
* it after the associated handler has acknowledged the device, so the
* interrupt line is back to inactive.
*/
-void handle_level_irq(struct irq_desc *desc)
+bool handle_level_irq(struct irq_desc *desc)
{
+ bool handled = false;
+
raw_spin_lock(&desc->lock);
mask_ack_irq(desc);
@@ -519,8 +533,11 @@ void handle_level_irq(struct irq_desc *desc)
cond_unmask_irq(desc);
+ handled = true;
+
out_unlock:
raw_spin_unlock(&desc->lock);
+ return handled;
}
EXPORT_SYMBOL_GPL(handle_level_irq);
@@ -564,9 +581,10 @@ static void cond_unmask_eoi_irq(struct irq_desc *desc, struct irq_chip *chip)
* for modern forms of interrupt handlers, which handle the flow
* details in hardware, transparently.
*/
-void handle_fasteoi_irq(struct irq_desc *desc)
+bool handle_fasteoi_irq(struct irq_desc *desc)
{
struct irq_chip *chip = desc->irq_data.chip;
+ bool handled = false;
raw_spin_lock(&desc->lock);
@@ -594,12 +612,15 @@ void handle_fasteoi_irq(struct irq_desc *desc)
cond_unmask_eoi_irq(desc, chip);
+ handled = true;
+
raw_spin_unlock(&desc->lock);
- return;
+ return handled;
out:
if (!(chip->flags & IRQCHIP_EOI_IF_HANDLED))
chip->irq_eoi(&desc->irq_data);
raw_spin_unlock(&desc->lock);
+ return handled;
}
EXPORT_SYMBOL_GPL(handle_fasteoi_irq);
@@ -618,8 +639,10 @@ EXPORT_SYMBOL_GPL(handle_fasteoi_irq);
* the handler was running. If all pending interrupts are handled, the
* loop is left.
*/
-void handle_edge_irq(struct irq_desc *desc)
+bool handle_edge_irq(struct irq_desc *desc)
{
+ bool handled = false;
+
raw_spin_lock(&desc->lock);
desc->istate &= ~(IRQS_REPLAY | IRQS_WAITING);
@@ -663,12 +686,14 @@ void handle_edge_irq(struct irq_desc *desc)
}
handle_irq_event(desc);
+ handled = true;
} while ((desc->istate & IRQS_PENDING) &&
!irqd_irq_disabled(&desc->irq_data));
out_unlock:
raw_spin_unlock(&desc->lock);
+ return handled;
}
EXPORT_SYMBOL(handle_edge_irq);
@@ -680,8 +705,9 @@ EXPORT_SYMBOL(handle_edge_irq);
* Similar as the above handle_edge_irq, but using eoi and w/o the
* mask/unmask logic.
*/
-void handle_edge_eoi_irq(struct irq_desc *desc)
+bool handle_edge_eoi_irq(struct irq_desc *desc)
{
+ bool handled = false;
struct irq_chip *chip = irq_desc_get_chip(desc);
raw_spin_lock(&desc->lock);
@@ -709,6 +735,7 @@ void handle_edge_eoi_irq(struct irq_desc *desc)
goto out_eoi;
handle_irq_event(desc);
+ handled = true;
} while ((desc->istate & IRQS_PENDING) &&
!irqd_irq_disabled(&desc->irq_data));
@@ -716,6 +743,7 @@ void handle_edge_eoi_irq(struct irq_desc *desc)
out_eoi:
chip->irq_eoi(&desc->irq_data);
raw_spin_unlock(&desc->lock);
+ return handled;
}
#endif
@@ -725,7 +753,7 @@ void handle_edge_eoi_irq(struct irq_desc *desc)
*
* Per CPU interrupts on SMP machines without locking requirements
*/
-void handle_percpu_irq(struct irq_desc *desc)
+bool handle_percpu_irq(struct irq_desc *desc)
{
struct irq_chip *chip = irq_desc_get_chip(desc);
@@ -738,6 +766,8 @@ void handle_percpu_irq(struct irq_desc *desc)
if (chip->irq_eoi)
chip->irq_eoi(&desc->irq_data);
+
+ return true;
}
/**
@@ -751,7 +781,7 @@ void handle_percpu_irq(struct irq_desc *desc)
* contain the real device id for the cpu on which this handler is
* called
*/
-void handle_percpu_devid_irq(struct irq_desc *desc)
+bool handle_percpu_devid_irq(struct irq_desc *desc)
{
struct irq_chip *chip = irq_desc_get_chip(desc);
struct irqaction *action = desc->action;
@@ -780,6 +810,8 @@ void handle_percpu_devid_irq(struct irq_desc *desc)
if (chip->irq_eoi)
chip->irq_eoi(&desc->irq_data);
+
+ return true;
}
static void
diff --git a/kernel/irq/handle.c b/kernel/irq/handle.c
index d3f2490..039607b 100644
--- a/kernel/irq/handle.c
+++ b/kernel/irq/handle.c
@@ -26,13 +26,14 @@
*
* Handles spurious and unhandled IRQ's. It also prints a debugmessage.
*/
-void handle_bad_irq(struct irq_desc *desc)
+bool handle_bad_irq(struct irq_desc *desc)
{
unsigned int irq = irq_desc_get_irq(desc);
print_irq_desc(irq, desc);
kstat_incr_irqs_this_cpu(desc);
ack_bad_irq(irq);
+ return true;
}
EXPORT_SYMBOL_GPL(handle_bad_irq);
diff --git a/kernel/irq/irqdesc.c b/kernel/irq/irqdesc.c
index 77977f55df..a0a625a 100644
--- a/kernel/irq/irqdesc.c
+++ b/kernel/irq/irqdesc.c
@@ -16,6 +16,7 @@
#include <linux/bitmap.h>
#include <linux/irqdomain.h>
#include <linux/sysfs.h>
+#include <linux/wakeup_reason.h>
#include "internals.h"
@@ -579,16 +580,25 @@ void irq_init_desc(unsigned int irq)
/**
* generic_handle_irq - Invoke the handler for a particular irq
* @irq: The irq number to handle
- *
+ * returns:
+ * negative on error
+ * 0 when the interrupt handler was not called
+ * 1 when the interrupt handler was called
*/
+
int generic_handle_irq(unsigned int irq)
{
struct irq_desc *desc = irq_to_desc(irq);
if (!desc)
return -EINVAL;
- generic_handle_irq_desc(desc);
- return 0;
+
+ if (unlikely(logging_wakeup_reasons_nosync()))
+ return log_possible_wakeup_reason(irq,
+ desc,
+ generic_handle_irq_desc);
+
+ return generic_handle_irq_desc(desc);
}
EXPORT_SYMBOL_GPL(generic_handle_irq);
diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index c93d4df..15d609b 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -881,6 +881,9 @@ irq_forced_thread_fn(struct irq_desc *desc, struct irqaction *action)
local_bh_disable();
ret = action->thread_fn(action->irq, action->dev_id);
+ if (ret == IRQ_HANDLED)
+ atomic_inc(&desc->threads_handled);
+
irq_finalize_oneshot(desc, action);
local_bh_enable();
return ret;
@@ -897,6 +900,9 @@ static irqreturn_t irq_thread_fn(struct irq_desc *desc,
irqreturn_t ret;
ret = action->thread_fn(action->irq, action->dev_id);
+ if (ret == IRQ_HANDLED)
+ atomic_inc(&desc->threads_handled);
+
irq_finalize_oneshot(desc, action);
return ret;
}
@@ -974,8 +980,6 @@ static int irq_thread(void *data)
irq_thread_check_affinity(desc, action);
action_ret = handler_fn(desc, action);
- if (action_ret == IRQ_HANDLED)
- atomic_inc(&desc->threads_handled);
if (action_ret == IRQ_WAKE_THREAD)
irq_wake_secondary(desc, action);
@@ -1029,6 +1033,13 @@ static int irq_setup_forced_threading(struct irqaction *new)
if (new->flags & (IRQF_NO_THREAD | IRQF_PERCPU | IRQF_ONESHOT))
return 0;
+ /*
+ * No further action required for interrupts which are requested as
+ * threaded interrupts already
+ */
+ if (new->handler == irq_default_primary_handler)
+ return 0;
+
new->flags |= IRQF_ONESHOT;
/*
@@ -1036,7 +1047,7 @@ static int irq_setup_forced_threading(struct irqaction *new)
* thread handler. We force thread them as well by creating a
* secondary action.
*/
- if (new->handler != irq_default_primary_handler && new->thread_fn) {
+ if (new->handler && new->thread_fn) {
/* Allocate the secondary action */
new->secondary = kzalloc(sizeof(struct irqaction), GFP_KERNEL);
if (!new->secondary)
diff --git a/kernel/irq/pm.c b/kernel/irq/pm.c
index cea1de0..de1aecb 100644
--- a/kernel/irq/pm.c
+++ b/kernel/irq/pm.c
@@ -11,7 +11,6 @@
#include <linux/interrupt.h>
#include <linux/suspend.h>
#include <linux/syscore_ops.h>
-
#include "internals.h"
bool irq_pm_check_wakeup(struct irq_desc *desc)
diff --git a/kernel/kcov.c b/kernel/kcov.c
index 154a80d..fad5144 100644
--- a/kernel/kcov.c
+++ b/kernel/kcov.c
@@ -240,7 +240,8 @@ static void kcov_put(struct kcov *kcov)
void kcov_task_init(struct task_struct *t)
{
- t->kcov_mode = KCOV_MODE_DISABLED;
+ WRITE_ONCE(t->kcov_mode, KCOV_MODE_DISABLED);
+ barrier();
t->kcov_size = 0;
t->kcov_area = NULL;
t->kcov = NULL;
diff --git a/kernel/kdebuginfo.c b/kernel/kdebuginfo.c
new file mode 100644
index 0000000..d1a9d93
--- /dev/null
+++ b/kernel/kdebuginfo.c
@@ -0,0 +1,174 @@
+/* Copyright 2018 Google, Inc.
+ *
+ * This software is licensed under the terms of the GNU General Public
+ * License version 2, as published by the Free Software Foundation, and
+ * may be copied, distributed, and modified under those terms.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+#include <linux/kallsyms.h>
+#include <linux/module.h>
+#include <linux/io.h>
+#include <linux/of_address.h>
+#include <linux/bldr_debug_tools.h>
+
+/*
+ * These will be re-linked against their real values
+ * during the second link stage.
+ */
+extern const unsigned long kallsyms_addresses[] __weak;
+extern const int kallsyms_offsets[] __weak;
+extern const u8 kallsyms_names[] __weak;
+
+/*
+ * Tell the compiler that the count isn't in the small data section if the arch
+ * has one (eg: FRV).
+ */
+extern const unsigned long kallsyms_num_syms
+__attribute__((weak, section(".rodata")));
+
+extern const unsigned long kallsyms_relative_base
+__attribute__((weak, section(".rodata")));
+
+extern const u8 kallsyms_token_table[] __weak;
+extern const u16 kallsyms_token_index[] __weak;
+
+extern const unsigned long kallsyms_markers[] __weak;
+
+struct kernel_info {
+ /* For kallsyms */
+ u8 enabled_all;
+ u8 enabled_base_relative;
+ u8 enabled_absolute_percpu;
+ u8 enabled_cfi_clang;
+ u32 num_syms;
+ u16 name_len;
+ u16 bit_per_long;
+ u16 module_name_len;
+ u16 symbol_len;
+ phys_addr_t _addresses_va;
+ phys_addr_t _relative_va;
+ phys_addr_t _stext_va;
+ phys_addr_t _etext_va;
+ phys_addr_t _sinittext_va;
+ phys_addr_t _einittext_va;
+ phys_addr_t _end_va;
+ phys_addr_t _offsets_va;
+ phys_addr_t _names_va;
+ phys_addr_t _token_table_va;
+ phys_addr_t _token_index_va;
+ phys_addr_t _markers_va;
+
+ /* For frame pointer */
+ u32 thread_size;
+
+ /* For virt_to_phys */
+ u32 va_bits;
+ u64 page_offset;
+ u64 phys_offset;
+ u64 kimage_voffset;
+};
+
+struct kernel_all_info {
+ u32 magic_number;
+ u32 combined_checksum;
+ struct kernel_info info;
+};
+
+static void backup_kernel_info(void)
+{
+ struct device_node *np;
+ struct resource res;
+ struct kernel_all_info *all_info;
+ struct kernel_info *info;
+ u32 *checksum_info;
+ int num_reg = 0;
+ int ret, index;
+
+ np = of_find_compatible_node(NULL, NULL, "bootloader_kinfo");
+ if (!np) {
+ pr_warn("%s: bootloader_kinfo node does not exist\n", __func__);
+ return;
+ }
+
+ ret = of_address_to_resource(np, num_reg, &res);
+ if(ret) {
+ pr_warn("%s: invalid argument, ret %d\n", __func__, ret);
+ return;
+ }
+
+ if ((!res.start) ||
+ (resource_size(&res) < sizeof(struct kernel_all_info))) {
+ pr_warn("%s: unexpected resource start %llx and size %llx\n",
+ __func__, res.start, resource_size(&res));
+ return;
+ }
+
+ ret = strcmp(res.name, "kinfo");
+ if (ret) {
+ pr_warn("%s: unexpected resource name %s, ret %d\n", __func__,
+ res.name, ret);
+ return;
+ }
+
+ all_info = (struct kernel_all_info *)
+ ioremap(res.start, resource_size(&res));
+ if (!all_info) {
+ pr_warn("%s: failed to map kernel_all_info\n", __func__);
+ return;
+ }
+
+ memset_io(all_info, 0, resource_size(&res));
+ info = &(all_info->info);
+ info->enabled_all = IS_ENABLED(CONFIG_KALLSYMS_ALL);
+ info->enabled_base_relative = IS_ENABLED(CONFIG_KALLSYMS_BASE_RELATIVE);
+ info->enabled_absolute_percpu =
+ IS_ENABLED(CONFIG_KALLSYMS_ABSOLUTE_PERCPU);
+ info->enabled_cfi_clang = IS_ENABLED(CONFIG_CFI_CLANG);
+ info->num_syms = kallsyms_num_syms;
+ info->name_len = KSYM_NAME_LEN;
+ info->bit_per_long = BITS_PER_LONG;
+ info->module_name_len = MODULE_NAME_LEN;
+ info->symbol_len = KSYM_SYMBOL_LEN;
+ info->_addresses_va = (phys_addr_t)kallsyms_addresses;
+ info->_relative_va = (phys_addr_t)kallsyms_relative_base;
+ info->_stext_va = (phys_addr_t)_stext;
+ info->_etext_va = (phys_addr_t)_etext;
+ info->_sinittext_va = (phys_addr_t)_sinittext;
+ info->_einittext_va = (phys_addr_t)_einittext;
+ info->_end_va = (phys_addr_t)_end;
+ info->_offsets_va = (phys_addr_t)kallsyms_offsets;
+ info->_names_va = (phys_addr_t)kallsyms_names;
+ info->_token_table_va = (phys_addr_t)kallsyms_token_table;
+ info->_token_index_va = (phys_addr_t)kallsyms_token_index;
+ info->_markers_va = (phys_addr_t)kallsyms_markers;
+ info->thread_size = THREAD_SIZE;
+ info->va_bits = VA_BITS;
+ info->page_offset = PAGE_OFFSET;
+ info->phys_offset = PHYS_OFFSET;
+ info->kimage_voffset = kimage_voffset;
+
+ for (index = 0; index < sizeof(struct kernel_info)/sizeof(u32);
+ index++) {
+ checksum_info = (u32 *)info;
+ all_info->combined_checksum ^= checksum_info[index];
+ }
+
+ all_info->magic_number = BOOT_DEBUG_MAGIC;
+
+ /* Ensure all information written to kernel_info is completed */
+ mb();
+ iounmap(all_info);
+}
+
+static int __init kdebuginfo_init(void)
+{
+ /* Backup kernel information for bootloader */
+ backup_kernel_info();
+
+ return 0;
+}
+device_initcall(kdebuginfo_init);
diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c
index 5616755..f5ab72e 100644
--- a/kernel/kexec_core.c
+++ b/kernel/kexec_core.c
@@ -38,6 +38,7 @@
#include <linux/syscore_ops.h>
#include <linux/compiler.h>
#include <linux/hugetlb.h>
+#include <linux/frame.h>
#include <asm/page.h>
#include <asm/sections.h>
@@ -878,7 +879,7 @@ int kexec_load_disabled;
* only when panic_cpu holds the current CPU number; this is the only CPU
* which processes crash_kexec routines.
*/
-void __crash_kexec(struct pt_regs *regs)
+void __noclone __crash_kexec(struct pt_regs *regs)
{
/* Take the kexec_mutex here to prevent sys_kexec_load
* running on one cpu from replacing the crash kernel
@@ -900,6 +901,7 @@ void __crash_kexec(struct pt_regs *regs)
mutex_unlock(&kexec_mutex);
}
}
+STACK_FRAME_NON_STANDARD(__crash_kexec);
void crash_kexec(struct pt_regs *regs)
{
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index 6948518..f580352 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -665,9 +665,10 @@ static void unoptimize_kprobe(struct kprobe *p, bool force)
}
/* Cancel unoptimizing for reusing */
-static void reuse_unused_kprobe(struct kprobe *ap)
+static int reuse_unused_kprobe(struct kprobe *ap)
{
struct optimized_kprobe *op;
+ int ret;
BUG_ON(!kprobe_unused(ap));
/*
@@ -681,8 +682,12 @@ static void reuse_unused_kprobe(struct kprobe *ap)
/* Enable the probe again */
ap->flags &= ~KPROBE_FLAG_DISABLED;
/* Optimize it again (remove from op->list) */
- BUG_ON(!kprobe_optready(ap));
+ ret = kprobe_optready(ap);
+ if (ret)
+ return ret;
+
optimize_kprobe(ap);
+ return 0;
}
/* Remove optimized instructions */
@@ -894,11 +899,16 @@ static void __disarm_kprobe(struct kprobe *p, bool reopt)
#define kprobe_disarmed(p) kprobe_disabled(p)
#define wait_for_kprobe_optimizer() do {} while (0)
-/* There should be no unused kprobes can be reused without optimization */
-static void reuse_unused_kprobe(struct kprobe *ap)
+static int reuse_unused_kprobe(struct kprobe *ap)
{
+ /*
+ * If the optimized kprobe is NOT supported, the aggr kprobe is
+ * released at the same time that the last aggregated kprobe is
+ * unregistered.
+ * Thus there should be no chance to reuse unused kprobe.
+ */
printk(KERN_ERR "Error: There should be no unused kprobe here.\n");
- BUG_ON(kprobe_unused(ap));
+ return -EINVAL;
}
static void free_aggr_kprobe(struct kprobe *p)
@@ -1276,9 +1286,12 @@ static int register_aggr_kprobe(struct kprobe *orig_p, struct kprobe *p)
goto out;
}
init_aggr_kprobe(ap, orig_p);
- } else if (kprobe_unused(ap))
+ } else if (kprobe_unused(ap)) {
/* This probe is going to die. Rescue it */
- reuse_unused_kprobe(ap);
+ ret = reuse_unused_kprobe(ap);
+ if (ret)
+ goto out;
+ }
if (kprobe_gone(ap)) {
/*
@@ -2441,7 +2454,7 @@ static int __init debugfs_kprobe_init(void)
if (!dir)
return -ENOMEM;
- file = debugfs_create_file("list", 0444, dir, NULL,
+ file = debugfs_create_file("list", 0400, dir, NULL,
&debugfs_kprobes_operations);
if (!file)
goto error;
@@ -2451,7 +2464,7 @@ static int __init debugfs_kprobe_init(void)
if (!file)
goto error;
- file = debugfs_create_file("blacklist", 0444, dir, NULL,
+ file = debugfs_create_file("blacklist", 0400, dir, NULL,
&debugfs_kprobe_blacklist_ops);
if (!file)
goto error;
diff --git a/kernel/kthread.c b/kernel/kthread.c
index b65854c..981ebe9 100644
--- a/kernel/kthread.c
+++ b/kernel/kthread.c
@@ -290,8 +290,14 @@ static struct task_struct *__kthread_create_on_node(int (*threadfn)(void *data),
task = create->result;
if (!IS_ERR(task)) {
static const struct sched_param param = { .sched_priority = 0 };
+ char name[TASK_COMM_LEN];
- vsnprintf(task->comm, sizeof(task->comm), namefmt, args);
+ /*
+ * task is already visible to other tasks, so updating
+ * COMM must be protected.
+ */
+ vsnprintf(name, sizeof(name), namefmt, args);
+ set_task_comm(task, name);
/*
* root may have changed our (kthreadd's) priority or CPU mask.
* The kernel thread should not inherit these properties.
diff --git a/kernel/locking/lockdep.c b/kernel/locking/lockdep.c
index 6599c7f..26fc428 100644
--- a/kernel/locking/lockdep.c
+++ b/kernel/locking/lockdep.c
@@ -1240,11 +1240,11 @@ unsigned long lockdep_count_forward_deps(struct lock_class *class)
this.parent = NULL;
this.class = class;
- local_irq_save(flags);
+ raw_local_irq_save(flags);
arch_spin_lock(&lockdep_lock);
ret = __lockdep_count_forward_deps(&this);
arch_spin_unlock(&lockdep_lock);
- local_irq_restore(flags);
+ raw_local_irq_restore(flags);
return ret;
}
@@ -1267,11 +1267,11 @@ unsigned long lockdep_count_backward_deps(struct lock_class *class)
this.parent = NULL;
this.class = class;
- local_irq_save(flags);
+ raw_local_irq_save(flags);
arch_spin_lock(&lockdep_lock);
ret = __lockdep_count_backward_deps(&this);
arch_spin_unlock(&lockdep_lock);
- local_irq_restore(flags);
+ raw_local_irq_restore(flags);
return ret;
}
@@ -4010,7 +4010,7 @@ void lock_contended(struct lockdep_map *lock, unsigned long ip)
{
unsigned long flags;
- if (unlikely(!lock_stat))
+ if (unlikely(!lock_stat || !debug_locks))
return;
if (unlikely(current->lockdep_recursion))
@@ -4030,7 +4030,7 @@ void lock_acquired(struct lockdep_map *lock, unsigned long ip)
{
unsigned long flags;
- if (unlikely(!lock_stat))
+ if (unlikely(!lock_stat || !debug_locks))
return;
if (unlikely(current->lockdep_recursion))
@@ -4273,7 +4273,7 @@ void debug_check_no_locks_freed(const void *mem_from, unsigned long mem_len)
if (unlikely(!debug_locks))
return;
- local_irq_save(flags);
+ raw_local_irq_save(flags);
for (i = 0; i < curr->lockdep_depth; i++) {
hlock = curr->held_locks + i;
@@ -4284,7 +4284,7 @@ void debug_check_no_locks_freed(const void *mem_from, unsigned long mem_len)
print_freed_lock_bug(curr, mem_from, mem_from + mem_len, hlock);
break;
}
- local_irq_restore(flags);
+ raw_local_irq_restore(flags);
}
EXPORT_SYMBOL_GPL(debug_check_no_locks_freed);
diff --git a/kernel/locking/qspinlock.c b/kernel/locking/qspinlock.c
index b2caec7..0ed478e 100644
--- a/kernel/locking/qspinlock.c
+++ b/kernel/locking/qspinlock.c
@@ -76,6 +76,18 @@
#endif
/*
+ * The pending bit spinning loop count.
+ * This heuristic is used to limit the number of lockword accesses
+ * made by atomic_cond_read_relaxed when waiting for the lock to
+ * transition out of the "== _Q_PENDING_VAL" state. We don't spin
+ * indefinitely because there's no guarantee that we'll make forward
+ * progress.
+ */
+#ifndef _Q_PENDING_LOOPS
+#define _Q_PENDING_LOOPS 1
+#endif
+
+/*
* Per-CPU queue node structures; we can never have more than 4 nested
* contexts: task, softirq, hardirq, nmi.
*
@@ -113,42 +125,19 @@ static inline __pure struct mcs_spinlock *decode_tail(u32 tail)
#define _Q_LOCKED_PENDING_MASK (_Q_LOCKED_MASK | _Q_PENDING_MASK)
-/*
- * By using the whole 2nd least significant byte for the pending bit, we
- * can allow better optimization of the lock acquisition for the pending
- * bit holder.
- *
- * This internal structure is also used by the set_locked function which
- * is not restricted to _Q_PENDING_BITS == 8.
- */
-struct __qspinlock {
- union {
- atomic_t val;
-#ifdef __LITTLE_ENDIAN
- struct {
- u8 locked;
- u8 pending;
- };
- struct {
- u16 locked_pending;
- u16 tail;
- };
-#else
- struct {
- u16 tail;
- u16 locked_pending;
- };
- struct {
- u8 reserved[2];
- u8 pending;
- u8 locked;
- };
-#endif
- };
-};
-
#if _Q_PENDING_BITS == 8
/**
+ * clear_pending - clear the pending bit.
+ * @lock: Pointer to queued spinlock structure
+ *
+ * *,1,* -> *,0,*
+ */
+static __always_inline void clear_pending(struct qspinlock *lock)
+{
+ WRITE_ONCE(lock->pending, 0);
+}
+
+/**
* clear_pending_set_locked - take ownership and clear the pending bit.
* @lock: Pointer to queued spinlock structure
*
@@ -158,9 +147,7 @@ struct __qspinlock {
*/
static __always_inline void clear_pending_set_locked(struct qspinlock *lock)
{
- struct __qspinlock *l = (void *)lock;
-
- WRITE_ONCE(l->locked_pending, _Q_LOCKED_VAL);
+ WRITE_ONCE(lock->locked_pending, _Q_LOCKED_VAL);
}
/*
@@ -169,25 +156,34 @@ static __always_inline void clear_pending_set_locked(struct qspinlock *lock)
* @tail : The new queue tail code word
* Return: The previous queue tail code word
*
- * xchg(lock, tail)
+ * xchg(lock, tail), which heads an address dependency
*
* p,*,* -> n,*,* ; prev = xchg(lock, node)
*/
static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail)
{
- struct __qspinlock *l = (void *)lock;
-
/*
* Use release semantics to make sure that the MCS node is properly
* initialized before changing the tail code.
*/
- return (u32)xchg_release(&l->tail,
+ return (u32)xchg_release(&lock->tail,
tail >> _Q_TAIL_OFFSET) << _Q_TAIL_OFFSET;
}
#else /* _Q_PENDING_BITS == 8 */
/**
+ * clear_pending - clear the pending bit.
+ * @lock: Pointer to queued spinlock structure
+ *
+ * *,1,* -> *,0,*
+ */
+static __always_inline void clear_pending(struct qspinlock *lock)
+{
+ atomic_andnot(_Q_PENDING_VAL, &lock->val);
+}
+
+/**
* clear_pending_set_locked - take ownership and clear the pending bit.
* @lock: Pointer to queued spinlock structure
*
@@ -229,6 +225,20 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail)
#endif /* _Q_PENDING_BITS == 8 */
/**
+ * queued_fetch_set_pending_acquire - fetch the whole lock value and set pending
+ * @lock : Pointer to queued spinlock structure
+ * Return: The previous lock value
+ *
+ * *,*,* -> *,1,*
+ */
+#ifndef queued_fetch_set_pending_acquire
+static __always_inline u32 queued_fetch_set_pending_acquire(struct qspinlock *lock)
+{
+ return atomic_fetch_or_acquire(_Q_PENDING_VAL, &lock->val);
+}
+#endif
+
+/**
* set_locked - Set the lock bit and own the lock
* @lock: Pointer to queued spinlock structure
*
@@ -236,9 +246,7 @@ static __always_inline u32 xchg_tail(struct qspinlock *lock, u32 tail)
*/
static __always_inline void set_locked(struct qspinlock *lock)
{
- struct __qspinlock *l = (void *)lock;
-
- WRITE_ONCE(l->locked, _Q_LOCKED_VAL);
+ WRITE_ONCE(lock->locked, _Q_LOCKED_VAL);
}
@@ -410,7 +418,7 @@ EXPORT_SYMBOL(queued_spin_unlock_wait);
void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
{
struct mcs_spinlock *prev, *next, *node;
- u32 new, old, tail;
+ u32 old, tail;
int idx;
BUILD_BUG_ON(CONFIG_NR_CPUS >= (1U << _Q_TAIL_CPU_BITS));
@@ -422,65 +430,58 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
return;
/*
- * wait for in-progress pending->locked hand-overs
+ * Wait for in-progress pending->locked hand-overs with a bounded
+ * number of spins so that we guarantee forward progress.
*
* 0,1,0 -> 0,0,1
*/
if (val == _Q_PENDING_VAL) {
- while ((val = atomic_read(&lock->val)) == _Q_PENDING_VAL)
- cpu_relax();
+ int cnt = _Q_PENDING_LOOPS;
+ val = smp_cond_load_acquire(&lock->val.counter,
+ (VAL != _Q_PENDING_VAL) || !cnt--);
}
/*
+ * If we observe any contention; queue.
+ */
+ if (val & ~_Q_LOCKED_MASK)
+ goto queue;
+
+ /*
* trylock || pending
*
* 0,0,0 -> 0,0,1 ; trylock
* 0,0,1 -> 0,1,1 ; pending
*/
- for (;;) {
- /*
- * If we observe any contention; queue.
- */
- if (val & ~_Q_LOCKED_MASK)
- goto queue;
+ val = queued_fetch_set_pending_acquire(lock);
- new = _Q_LOCKED_VAL;
- if (val == new)
- new |= _Q_PENDING_VAL;
-
- /*
- * Acquire semantic is required here as the function may
- * return immediately if the lock was free.
- */
- old = atomic_cmpxchg_acquire(&lock->val, val, new);
- if (old == val)
- break;
-
- val = old;
+ /*
+ * If we observe any contention; undo and queue.
+ */
+ if (unlikely(val & ~_Q_LOCKED_MASK)) {
+ if (!(val & _Q_PENDING_MASK))
+ clear_pending(lock);
+ goto queue;
}
/*
- * we won the trylock
- */
- if (new == _Q_LOCKED_VAL)
- return;
-
- /*
- * we're pending, wait for the owner to go away.
+ * We're pending, wait for the owner to go away.
*
- * *,1,1 -> *,1,0
+ * 0,1,1 -> 0,1,0
*
* this wait loop must be a load-acquire such that we match the
* store-release that clears the locked bit and create lock
- * sequentiality; this is because not all clear_pending_set_locked()
- * implementations imply full barriers.
+ * sequentiality; this is because not all
+ * clear_pending_set_locked() implementations imply full
+ * barriers.
*/
- smp_cond_load_acquire(&lock->val.counter, !(VAL & _Q_LOCKED_MASK));
+ if (val & _Q_LOCKED_MASK)
+ smp_cond_load_acquire(&lock->val.counter, !(VAL & _Q_LOCKED_MASK));
/*
* take ownership and clear the pending bit.
*
- * *,1,0 -> *,0,1
+ * 0,1,0 -> 0,0,1
*/
clear_pending_set_locked(lock);
return;
@@ -495,6 +496,14 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
tail = encode_tail(smp_processor_id(), idx);
node += idx;
+
+ /*
+ * Ensure that we increment the head node->count before initialising
+ * the actual node. If the compiler is kind enough to reorder these
+ * stores, then an IRQ could overwrite our assignments.
+ */
+ barrier();
+
node->locked = 0;
node->next = NULL;
pv_init_node(node);
@@ -524,16 +533,15 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
*/
if (old & _Q_TAIL_MASK) {
prev = decode_tail(old);
- /*
- * The above xchg_tail() is also a load of @lock which generates,
- * through decode_tail(), a pointer.
- *
- * The address dependency matches the RELEASE of xchg_tail()
- * such that the access to @prev must happen after.
- */
- smp_read_barrier_depends();
- WRITE_ONCE(prev->next, node);
+ /*
+ * We must ensure that the stores to @node are observed before
+ * the write to prev->next. The address dependency from
+ * xchg_tail is not sufficient to ensure this because the read
+ * component of xchg_tail is unordered with respect to the
+ * initialisation of @node.
+ */
+ smp_store_release(&prev->next, node);
pv_wait_node(node, prev);
arch_mcs_spin_lock_contended(&node->locked);
@@ -580,30 +588,27 @@ void queued_spin_lock_slowpath(struct qspinlock *lock, u32 val)
* claim the lock:
*
* n,0,0 -> 0,0,1 : lock, uncontended
- * *,0,0 -> *,0,1 : lock, contended
+ * *,*,0 -> *,*,1 : lock, contended
*
- * If the queue head is the only one in the queue (lock value == tail),
- * clear the tail code and grab the lock. Otherwise, we only need
- * to grab the lock.
+ * If the queue head is the only one in the queue (lock value == tail)
+ * and nobody is pending, clear the tail code and grab the lock.
+ * Otherwise, we only need to grab the lock.
*/
- for (;;) {
- /* In the PV case we might already have _Q_LOCKED_VAL set */
- if ((val & _Q_TAIL_MASK) != tail) {
- set_locked(lock);
- break;
- }
+
+ /* In the PV case we might already have _Q_LOCKED_VAL set */
+ if ((val & _Q_TAIL_MASK) == tail) {
/*
* The smp_cond_load_acquire() call above has provided the
- * necessary acquire semantics required for locking. At most
- * two iterations of this loop may be ran.
+ * necessary acquire semantics required for locking.
*/
old = atomic_cmpxchg_relaxed(&lock->val, val, _Q_LOCKED_VAL);
if (old == val)
- goto release; /* No contention */
-
- val = old;
+ goto release; /* No contention */
}
+ /* Either somebody is queued behind us or _Q_PENDING_VAL is set */
+ set_locked(lock);
+
/*
* contended path; wait for next if not observed yet, release.
*/
diff --git a/kernel/locking/qspinlock_paravirt.h b/kernel/locking/qspinlock_paravirt.h
index e3b5520..af2a24d 100644
--- a/kernel/locking/qspinlock_paravirt.h
+++ b/kernel/locking/qspinlock_paravirt.h
@@ -69,10 +69,8 @@ struct pv_node {
#define queued_spin_trylock(l) pv_queued_spin_steal_lock(l)
static inline bool pv_queued_spin_steal_lock(struct qspinlock *lock)
{
- struct __qspinlock *l = (void *)lock;
-
if (!(atomic_read(&lock->val) & _Q_LOCKED_PENDING_MASK) &&
- (cmpxchg(&l->locked, 0, _Q_LOCKED_VAL) == 0)) {
+ (cmpxchg(&lock->locked, 0, _Q_LOCKED_VAL) == 0)) {
qstat_inc(qstat_pv_lock_stealing, true);
return true;
}
@@ -87,16 +85,7 @@ static inline bool pv_queued_spin_steal_lock(struct qspinlock *lock)
#if _Q_PENDING_BITS == 8
static __always_inline void set_pending(struct qspinlock *lock)
{
- struct __qspinlock *l = (void *)lock;
-
- WRITE_ONCE(l->pending, 1);
-}
-
-static __always_inline void clear_pending(struct qspinlock *lock)
-{
- struct __qspinlock *l = (void *)lock;
-
- WRITE_ONCE(l->pending, 0);
+ WRITE_ONCE(lock->pending, 1);
}
/*
@@ -106,10 +95,8 @@ static __always_inline void clear_pending(struct qspinlock *lock)
*/
static __always_inline int trylock_clear_pending(struct qspinlock *lock)
{
- struct __qspinlock *l = (void *)lock;
-
- return !READ_ONCE(l->locked) &&
- (cmpxchg(&l->locked_pending, _Q_PENDING_VAL, _Q_LOCKED_VAL)
+ return !READ_ONCE(lock->locked) &&
+ (cmpxchg(&lock->locked_pending, _Q_PENDING_VAL, _Q_LOCKED_VAL)
== _Q_PENDING_VAL);
}
#else /* _Q_PENDING_BITS == 8 */
@@ -118,11 +105,6 @@ static __always_inline void set_pending(struct qspinlock *lock)
atomic_or(_Q_PENDING_VAL, &lock->val);
}
-static __always_inline void clear_pending(struct qspinlock *lock)
-{
- atomic_andnot(_Q_PENDING_VAL, &lock->val);
-}
-
static __always_inline int trylock_clear_pending(struct qspinlock *lock)
{
int val = atomic_read(&lock->val);
@@ -353,7 +335,6 @@ static void pv_wait_node(struct mcs_spinlock *node, struct mcs_spinlock *prev)
static void pv_kick_node(struct qspinlock *lock, struct mcs_spinlock *node)
{
struct pv_node *pn = (struct pv_node *)node;
- struct __qspinlock *l = (void *)lock;
/*
* If the vCPU is indeed halted, advance its state to match that of
@@ -372,7 +353,7 @@ static void pv_kick_node(struct qspinlock *lock, struct mcs_spinlock *node)
* the hash table later on at unlock time, no atomic instruction is
* needed.
*/
- WRITE_ONCE(l->locked, _Q_SLOW_VAL);
+ WRITE_ONCE(lock->locked, _Q_SLOW_VAL);
(void)pv_hash(lock, pn);
}
@@ -387,7 +368,6 @@ static u32
pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node)
{
struct pv_node *pn = (struct pv_node *)node;
- struct __qspinlock *l = (void *)lock;
struct qspinlock **lp = NULL;
int waitcnt = 0;
int loop;
@@ -438,13 +418,13 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node)
*
* Matches the smp_rmb() in __pv_queued_spin_unlock().
*/
- if (xchg(&l->locked, _Q_SLOW_VAL) == 0) {
+ if (xchg(&lock->locked, _Q_SLOW_VAL) == 0) {
/*
* The lock was free and now we own the lock.
* Change the lock value back to _Q_LOCKED_VAL
* and unhash the table.
*/
- WRITE_ONCE(l->locked, _Q_LOCKED_VAL);
+ WRITE_ONCE(lock->locked, _Q_LOCKED_VAL);
WRITE_ONCE(*lp, NULL);
goto gotlock;
}
@@ -452,7 +432,7 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node)
WRITE_ONCE(pn->state, vcpu_hashed);
qstat_inc(qstat_pv_wait_head, true);
qstat_inc(qstat_pv_wait_again, waitcnt);
- pv_wait(&l->locked, _Q_SLOW_VAL);
+ pv_wait(&lock->locked, _Q_SLOW_VAL);
/*
* Because of lock stealing, the queue head vCPU may not be
@@ -477,7 +457,6 @@ pv_wait_head_or_lock(struct qspinlock *lock, struct mcs_spinlock *node)
__visible void
__pv_queued_spin_unlock_slowpath(struct qspinlock *lock, u8 locked)
{
- struct __qspinlock *l = (void *)lock;
struct pv_node *node;
if (unlikely(locked != _Q_SLOW_VAL)) {
@@ -506,7 +485,7 @@ __pv_queued_spin_unlock_slowpath(struct qspinlock *lock, u8 locked)
* Now that we have a reference to the (likely) blocked pv_node,
* release the lock.
*/
- smp_store_release(&l->locked, 0);
+ smp_store_release(&lock->locked, 0);
/*
* At this point the memory pointed at by lock can be freed/reused,
@@ -532,7 +511,6 @@ __pv_queued_spin_unlock_slowpath(struct qspinlock *lock, u8 locked)
#ifndef __pv_queued_spin_unlock
__visible void __pv_queued_spin_unlock(struct qspinlock *lock)
{
- struct __qspinlock *l = (void *)lock;
u8 locked;
/*
@@ -540,7 +518,7 @@ __visible void __pv_queued_spin_unlock(struct qspinlock *lock)
* unhash. Otherwise it would be possible to have multiple @lock
* entries, which would be BAD.
*/
- locked = cmpxchg_release(&l->locked, _Q_LOCKED_VAL, 0);
+ locked = cmpxchg_release(&lock->locked, _Q_LOCKED_VAL, 0);
if (likely(locked == _Q_LOCKED_VAL))
return;
diff --git a/kernel/memremap.c b/kernel/memremap.c
index f61a8c3..9a8b594 100644
--- a/kernel/memremap.c
+++ b/kernel/memremap.c
@@ -305,15 +305,12 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
is_ram = region_intersects(align_start, align_size,
IORESOURCE_SYSTEM_RAM, IORES_DESC_NONE);
- if (is_ram == REGION_MIXED) {
- WARN_ONCE(1, "%s attempted on mixed region %pr\n",
- __func__, res);
+ if (is_ram != REGION_DISJOINT) {
+ WARN_ONCE(1, "%s attempted on %s region %pr\n", __func__,
+ is_ram == REGION_MIXED ? "mixed" : "ram", res);
return ERR_PTR(-ENXIO);
}
- if (is_ram == REGION_INTERSECTS)
- return __va(res->start);
-
if (!ref)
return ERR_PTR(-EINVAL);
@@ -399,7 +396,7 @@ void *devm_memremap_pages(struct device *dev, struct resource *res,
devres_free(page_map);
return ERR_PTR(error);
}
-EXPORT_SYMBOL(devm_memremap_pages);
+EXPORT_SYMBOL_GPL(devm_memremap_pages);
unsigned long vmem_altmap_offset(struct vmem_altmap *altmap)
{
diff --git a/kernel/module.c b/kernel/module.c
index 13ad2f8..b63d342 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -4022,7 +4022,7 @@ static unsigned long mod_find_symname(struct module *mod, const char *name)
for (i = 0; i < kallsyms->num_symtab; i++)
if (strcmp(name, symname(kallsyms, i)) == 0 &&
- kallsyms->symtab[i].st_info != 'U')
+ kallsyms->symtab[i].st_shndx != SHN_UNDEF)
return kallsyms->symtab[i].st_value;
return 0;
}
@@ -4068,6 +4068,10 @@ int module_kallsyms_on_each_symbol(int (*fn)(void *, const char *,
if (mod->state == MODULE_STATE_UNFORMED)
continue;
for (i = 0; i < kallsyms->num_symtab; i++) {
+
+ if (kallsyms->symtab[i].st_shndx == SHN_UNDEF)
+ continue;
+
ret = fn(data, symname(kallsyms, i),
mod, kallsyms->symtab[i].st_value);
if (ret != 0)
diff --git a/kernel/panic.c b/kernel/panic.c
index d797170..278d64b 100644
--- a/kernel/panic.c
+++ b/kernel/panic.c
@@ -13,6 +13,7 @@
#include <linux/kmsg_dump.h>
#include <linux/kallsyms.h>
#include <linux/notifier.h>
+#include <linux/vt_kern.h>
#include <linux/module.h>
#include <linux/random.h>
#include <linux/ftrace.h>
@@ -234,7 +235,10 @@ void panic(const char *fmt, ...)
if (_crash_kexec_post_notifiers)
__crash_kexec(NULL);
- bust_spinlocks(0);
+#ifdef CONFIG_VT
+ unblank_screen();
+#endif
+ console_unblank();
/*
* We may have ended up stopping the CPU holding the lock (in
diff --git a/kernel/power/Kconfig b/kernel/power/Kconfig
index bf60b37..fbc9251 100644
--- a/kernel/power/Kconfig
+++ b/kernel/power/Kconfig
@@ -133,6 +133,7 @@
def_bool y
depends on SUSPEND || HIBERNATE_CALLBACKS
select PM
+ select SRCU
config PM_SLEEP_SMP
def_bool y
@@ -339,3 +340,7 @@
config CPU_PM
bool
+
+config DEDUCE_WAKEUP_REASONS
+ bool
+ default n
diff --git a/kernel/power/main.c b/kernel/power/main.c
index 281a697..d401c21 100644
--- a/kernel/power/main.c
+++ b/kernel/power/main.c
@@ -78,6 +78,78 @@ static ssize_t pm_async_store(struct kobject *kobj, struct kobj_attribute *attr,
power_attr(pm_async);
+#ifdef CONFIG_SUSPEND
+static ssize_t mem_sleep_show(struct kobject *kobj, struct kobj_attribute *attr,
+ char *buf)
+{
+ char *s = buf;
+ suspend_state_t i;
+
+ for (i = PM_SUSPEND_MIN; i < PM_SUSPEND_MAX; i++)
+ if (mem_sleep_states[i]) {
+ const char *label = mem_sleep_states[i];
+
+ if (mem_sleep_current == i)
+ s += sprintf(s, "[%s] ", label);
+ else
+ s += sprintf(s, "%s ", label);
+ }
+
+ /* Convert the last space to a newline if needed. */
+ if (s != buf)
+ *(s-1) = '\n';
+
+ return (s - buf);
+}
+
+static suspend_state_t decode_suspend_state(const char *buf, size_t n)
+{
+ suspend_state_t state;
+ char *p;
+ int len;
+
+ p = memchr(buf, '\n', n);
+ len = p ? p - buf : n;
+
+ for (state = PM_SUSPEND_MIN; state < PM_SUSPEND_MAX; state++) {
+ const char *label = mem_sleep_states[state];
+
+ if (label && len == strlen(label) && !strncmp(buf, label, len))
+ return state;
+ }
+
+ return PM_SUSPEND_ON;
+}
+
+static ssize_t mem_sleep_store(struct kobject *kobj, struct kobj_attribute *attr,
+ const char *buf, size_t n)
+{
+ suspend_state_t state;
+ int error;
+
+ error = pm_autosleep_lock();
+ if (error)
+ return error;
+
+ if (pm_autosleep_state() > PM_SUSPEND_ON) {
+ error = -EBUSY;
+ goto out;
+ }
+
+ state = decode_suspend_state(buf, n);
+ if (state < PM_SUSPEND_MAX && state > PM_SUSPEND_ON)
+ mem_sleep_current = state;
+ else
+ error = -EINVAL;
+
+ out:
+ pm_autosleep_unlock();
+ return error ? error : n;
+}
+
+power_attr(mem_sleep);
+#endif /* CONFIG_SUSPEND */
+
#ifdef CONFIG_PM_DEBUG
int pm_test_level = TEST_NONE;
@@ -368,12 +440,16 @@ static ssize_t state_store(struct kobject *kobj, struct kobj_attribute *attr,
}
state = decode_state(buf, n);
- if (state < PM_SUSPEND_MAX)
+ if (state < PM_SUSPEND_MAX) {
+ if (state == PM_SUSPEND_MEM)
+ state = mem_sleep_current;
+
error = pm_suspend(state);
- else if (state == PM_SUSPEND_MAX)
+ } else if (state == PM_SUSPEND_MAX) {
error = hibernate();
- else
+ } else {
error = -EINVAL;
+ }
out:
pm_autosleep_unlock();
@@ -485,6 +561,9 @@ static ssize_t autosleep_store(struct kobject *kobj,
&& strcmp(buf, "off") && strcmp(buf, "off\n"))
return -EINVAL;
+ if (state == PM_SUSPEND_MEM)
+ state = mem_sleep_current;
+
error = pm_autosleep_set_state(state);
return error ? error : n;
}
@@ -602,6 +681,9 @@ static struct attribute * g[] = {
#ifdef CONFIG_PM_SLEEP
&pm_async_attr.attr,
&wakeup_count_attr.attr,
+#ifdef CONFIG_SUSPEND
+ &mem_sleep_attr.attr,
+#endif
#ifdef CONFIG_PM_AUTOSLEEP
&autosleep_attr.attr,
#endif
diff --git a/kernel/power/power.h b/kernel/power/power.h
index 56d1d0d..c647bfb 100644
--- a/kernel/power/power.h
+++ b/kernel/power/power.h
@@ -103,9 +103,6 @@ extern int in_suspend;
extern dev_t swsusp_resume_device;
extern sector_t swsusp_resume_block;
-extern asmlinkage int swsusp_arch_suspend(void);
-extern asmlinkage int swsusp_arch_resume(void);
-
extern int create_basic_memory_bitmaps(void);
extern void free_basic_memory_bitmaps(void);
extern int hibernate_preallocate_memory(void);
@@ -189,8 +186,9 @@ extern void swsusp_show_speed(ktime_t, ktime_t, unsigned int, char *);
#ifdef CONFIG_SUSPEND
/* kernel/power/suspend.c */
-extern const char *pm_labels[];
extern const char *pm_states[];
+extern const char *mem_sleep_states[];
+extern suspend_state_t mem_sleep_current;
extern int suspend_devices_and_enter(suspend_state_t state);
#else /* !CONFIG_SUSPEND */
diff --git a/kernel/power/process.c b/kernel/power/process.c
index 9a12c83..4a0c7b1 100644
--- a/kernel/power/process.c
+++ b/kernel/power/process.c
@@ -36,9 +36,6 @@ static int try_to_freeze_tasks(bool user_only)
unsigned int elapsed_msecs;
bool wakeup = false;
int sleep_usecs = USEC_PER_MSEC;
-#ifdef CONFIG_PM_SLEEP
- char suspend_abort[MAX_SUSPEND_ABORT_LEN];
-#endif
start = ktime_get_boottime();
@@ -68,11 +65,6 @@ static int try_to_freeze_tasks(bool user_only)
break;
if (pm_wakeup_pending()) {
-#ifdef CONFIG_PM_SLEEP
- pm_get_active_wakeup_sources(suspend_abort,
- MAX_SUSPEND_ABORT_LEN);
- log_suspend_abort_reason(suspend_abort);
-#endif
wakeup = true;
break;
}
diff --git a/kernel/power/suspend.c b/kernel/power/suspend.c
index 2d0c99b..3f8104e 100644
--- a/kernel/power/suspend.c
+++ b/kernel/power/suspend.c
@@ -34,8 +34,19 @@
#include "power.h"
-const char *pm_labels[] = { "mem", "standby", "freeze", NULL };
-const char *pm_states[PM_SUSPEND_MAX];
+const char *pm_states[PM_SUSPEND_MAX] = {
+ [PM_SUSPEND_FREEZE] = "freeze",
+ [PM_SUSPEND_MEM] = "mem",
+};
+const char * const mem_sleep_labels[] = {
+ [PM_SUSPEND_FREEZE] = "s2idle",
+ [PM_SUSPEND_STANDBY] = "shallow",
+ [PM_SUSPEND_MEM] = "deep",
+};
+const char *mem_sleep_states[PM_SUSPEND_MAX];
+
+suspend_state_t mem_sleep_current = PM_SUSPEND_FREEZE;
+static suspend_state_t mem_sleep_default = PM_SUSPEND_MEM;
unsigned int pm_suspend_global_flags;
EXPORT_SYMBOL_GPL(pm_suspend_global_flags);
@@ -112,30 +123,29 @@ static bool valid_state(suspend_state_t state)
return suspend_ops && suspend_ops->valid && suspend_ops->valid(state);
}
-/*
- * If this is set, the "mem" label always corresponds to the deepest sleep state
- * available, the "standby" label corresponds to the second deepest sleep state
- * available (if any), and the "freeze" label corresponds to the remaining
- * available sleep state (if there is one).
- */
-static bool relative_states;
-
void __init pm_states_init(void)
{
/*
- * freeze state should be supported even without any suspend_ops,
- * initialize pm_states accordingly here
+ * Suspend-to-idle should be supported even without any suspend_ops,
+ * initialize mem_sleep_states[] accordingly here.
*/
- pm_states[PM_SUSPEND_FREEZE] = pm_labels[relative_states ? 0 : 2];
+ mem_sleep_states[PM_SUSPEND_FREEZE] = mem_sleep_labels[PM_SUSPEND_FREEZE];
}
-static int __init sleep_states_setup(char *str)
+static int __init mem_sleep_default_setup(char *str)
{
- relative_states = !strncmp(str, "1", 1);
+ suspend_state_t state;
+
+ for (state = PM_SUSPEND_FREEZE; state <= PM_SUSPEND_MEM; state++)
+ if (mem_sleep_labels[state] &&
+ !strcmp(str, mem_sleep_labels[state])) {
+ mem_sleep_default = state;
+ break;
+ }
+
return 1;
}
-
-__setup("relative_sleep_states=", sleep_states_setup);
+__setup("mem_sleep_default=", mem_sleep_default_setup);
/**
* suspend_set_ops - Set the global suspend method table.
@@ -143,21 +153,21 @@ __setup("relative_sleep_states=", sleep_states_setup);
*/
void suspend_set_ops(const struct platform_suspend_ops *ops)
{
- suspend_state_t i;
- int j = 0;
-
lock_system_sleep();
suspend_ops = ops;
- for (i = PM_SUSPEND_MEM; i >= PM_SUSPEND_STANDBY; i--)
- if (valid_state(i)) {
- pm_states[i] = pm_labels[j++];
- } else if (!relative_states) {
- pm_states[i] = NULL;
- j++;
- }
- pm_states[PM_SUSPEND_FREEZE] = pm_labels[j];
+ if (valid_state(PM_SUSPEND_STANDBY)) {
+ mem_sleep_states[PM_SUSPEND_STANDBY] = mem_sleep_labels[PM_SUSPEND_STANDBY];
+ pm_states[PM_SUSPEND_STANDBY] = "standby";
+ if (mem_sleep_default == PM_SUSPEND_STANDBY)
+ mem_sleep_current = PM_SUSPEND_STANDBY;
+ }
+ if (valid_state(PM_SUSPEND_MEM)) {
+ mem_sleep_states[PM_SUSPEND_MEM] = mem_sleep_labels[PM_SUSPEND_MEM];
+ if (mem_sleep_default == PM_SUSPEND_MEM)
+ mem_sleep_current = PM_SUSPEND_MEM;
+ }
unlock_system_sleep();
}
@@ -295,6 +305,7 @@ static int suspend_prepare(suspend_state_t state)
if (!error)
return 0;
+ log_suspend_abort_reason("One or more tasks refusing to freeze");
suspend_stats.failed_freeze++;
dpm_save_failed_step(SUSPEND_FREEZE);
Finish:
@@ -324,7 +335,6 @@ void __weak arch_suspend_enable_irqs(void)
*/
static int suspend_enter(suspend_state_t state, bool *wakeup)
{
- char suspend_abort[MAX_SUSPEND_ABORT_LEN];
int error, last_dev;
error = platform_suspend_prepare(state);
@@ -393,11 +403,10 @@ static int suspend_enter(suspend_state_t state, bool *wakeup)
state, false);
events_check_enabled = false;
} else if (*wakeup) {
- pm_get_active_wakeup_sources(suspend_abort,
- MAX_SUSPEND_ABORT_LEN);
- log_suspend_abort_reason(suspend_abort);
error = -EBUSY;
}
+
+ start_logging_wakeup_reasons();
syscore_resume();
}
diff --git a/kernel/power/user.c b/kernel/power/user.c
index 35310b6..bc6dde1 100644
--- a/kernel/power/user.c
+++ b/kernel/power/user.c
@@ -186,6 +186,11 @@ static ssize_t snapshot_write(struct file *filp, const char __user *buf,
res = PAGE_SIZE - pg_offp;
}
+ if (!data_of(data->handle)) {
+ res = -EINVAL;
+ goto unlock;
+ }
+
res = simple_write_to_buffer(data_of(data->handle), res, &pg_offp,
buf, count);
if (res > 0)
diff --git a/kernel/power/wakeup_reason.c b/kernel/power/wakeup_reason.c
index 252611f..8f1505b 100644
--- a/kernel/power/wakeup_reason.c
+++ b/kernel/power/wakeup_reason.c
@@ -26,42 +26,232 @@
#include <linux/spinlock.h>
#include <linux/notifier.h>
#include <linux/suspend.h>
-
+#include <linux/slab.h>
#define MAX_WAKEUP_REASON_IRQS 32
-static int irq_list[MAX_WAKEUP_REASON_IRQS];
-static int irqcount;
static bool suspend_abort;
static char abort_reason[MAX_SUSPEND_ABORT_LEN];
+
+static struct wakeup_irq_node *base_irq_nodes;
+static struct wakeup_irq_node *cur_irq_tree;
+static int cur_irq_tree_depth;
+static LIST_HEAD(wakeup_irqs);
+
+static struct kmem_cache *wakeup_irq_nodes_cache;
static struct kobject *wakeup_reason;
-static DEFINE_SPINLOCK(resume_reason_lock);
+static spinlock_t resume_reason_lock;
+bool log_wakeups __read_mostly;
+struct completion wakeups_completion;
static ktime_t last_monotime; /* monotonic time before last suspend */
static ktime_t curr_monotime; /* monotonic time after last suspend */
static ktime_t last_stime; /* monotonic boottime offset before last suspend */
static ktime_t curr_stime; /* monotonic boottime offset after last suspend */
-static ssize_t last_resume_reason_show(struct kobject *kobj, struct kobj_attribute *attr,
- char *buf)
+static void init_wakeup_irq_node(struct wakeup_irq_node *p, int irq)
{
- int irq_no, buf_offset = 0;
- struct irq_desc *desc;
- spin_lock(&resume_reason_lock);
- if (suspend_abort) {
- buf_offset = sprintf(buf, "Abort: %s", abort_reason);
- } else {
- for (irq_no = 0; irq_no < irqcount; irq_no++) {
- desc = irq_to_desc(irq_list[irq_no]);
- if (desc && desc->action && desc->action->name)
- buf_offset += sprintf(buf + buf_offset, "%d %s\n",
- irq_list[irq_no], desc->action->name);
- else
- buf_offset += sprintf(buf + buf_offset, "%d\n",
- irq_list[irq_no]);
+ p->irq = irq;
+ p->desc = irq_to_desc(irq);
+ p->child = NULL;
+ p->parent = NULL;
+ p->handled = false;
+ INIT_LIST_HEAD(&p->siblings);
+ INIT_LIST_HEAD(&p->next);
+}
+
+static struct wakeup_irq_node* alloc_irq_node(int irq)
+{
+ struct wakeup_irq_node *n;
+
+ n = kmem_cache_alloc(wakeup_irq_nodes_cache, GFP_ATOMIC);
+ if (!n) {
+ pr_warning("Failed to log chained wakeup IRQ %d\n",
+ irq);
+ return NULL;
+ }
+
+ init_wakeup_irq_node(n, irq);
+ return n;
+}
+
+static struct wakeup_irq_node *
+search_siblings(struct wakeup_irq_node *root, int irq)
+{
+ bool found = false;
+ struct wakeup_irq_node *n = NULL;
+ BUG_ON(!root);
+
+ if (root->irq == irq)
+ return root;
+
+ list_for_each_entry(n, &root->siblings, siblings) {
+ if (n->irq == irq) {
+ found = true;
+ break;
}
}
- spin_unlock(&resume_reason_lock);
- return buf_offset;
+
+ return found ? n : NULL;
+}
+
+static struct wakeup_irq_node *
+add_to_siblings(struct wakeup_irq_node *root, int irq)
+{
+ struct wakeup_irq_node *n;
+ if (root) {
+ n = search_siblings(root, irq);
+ if (n)
+ return n;
+ }
+ n = alloc_irq_node(irq);
+
+ if (n && root)
+ list_add(&n->siblings, &root->siblings);
+ return n;
+}
+
+#ifdef CONFIG_DEDUCE_WAKEUP_REASONS
+static struct wakeup_irq_node* add_child(struct wakeup_irq_node *root, int irq)
+{
+ if (!root->child) {
+ root->child = alloc_irq_node(irq);
+ if (!root->child)
+ return NULL;
+ root->child->parent = root;
+ return root->child;
+ }
+
+ return add_to_siblings(root->child, irq);
+}
+
+static struct wakeup_irq_node *find_first_sibling(struct wakeup_irq_node *node)
+{
+ struct wakeup_irq_node *n;
+ if (node->parent)
+ return node;
+ list_for_each_entry(n, &node->siblings, siblings) {
+ if (n->parent)
+ return n;
+ }
+ return NULL;
+}
+
+static struct wakeup_irq_node *
+get_base_node(struct wakeup_irq_node *node, unsigned depth)
+{
+ if (!node)
+ return NULL;
+
+ while (depth) {
+ node = find_first_sibling(node);
+ BUG_ON(!node);
+ node = node->parent;
+ depth--;
+ }
+
+ return node;
+}
+#endif /* CONFIG_DEDUCE_WAKEUP_REASONS */
+
+static const struct list_head* get_wakeup_reasons_nosync(void);
+
+static void print_wakeup_sources(void)
+{
+ struct wakeup_irq_node *n;
+ const struct list_head *wakeups;
+
+ if (suspend_abort) {
+ pr_info("Abort: %s\n", abort_reason);
+ return;
+ }
+
+ wakeups = get_wakeup_reasons_nosync();
+ list_for_each_entry(n, wakeups, next) {
+ if (n->desc && n->desc->action && n->desc->action->name)
+ pr_info("Resume caused by IRQ %d, %s\n", n->irq,
+ n->desc->action->name);
+ else
+ pr_info("Resume caused by IRQ %d\n", n->irq);
+ }
+}
+
+static bool walk_irq_node_tree(struct wakeup_irq_node *root,
+ bool (*visit)(struct wakeup_irq_node *, void *),
+ void *cookie)
+{
+ struct wakeup_irq_node *n, *t;
+
+ if (!root)
+ return true;
+
+ list_for_each_entry_safe(n, t, &root->siblings, siblings) {
+ if (!walk_irq_node_tree(n->child, visit, cookie))
+ return false;
+ if (!visit(n, cookie))
+ return false;
+ }
+
+ if (!walk_irq_node_tree(root->child, visit, cookie))
+ return false;
+ return visit(root, cookie);
+}
+
+#ifdef CONFIG_DEDUCE_WAKEUP_REASONS
+static bool is_node_handled(struct wakeup_irq_node *n, void *_p)
+{
+ return n->handled;
+}
+
+static bool base_irq_nodes_done(void)
+{
+ return walk_irq_node_tree(base_irq_nodes, is_node_handled, NULL);
+}
+#endif
+
+struct buf_cookie {
+ char *buf;
+ int buf_offset;
+};
+
+static bool print_leaf_node(struct wakeup_irq_node *n, void *_p)
+{
+ struct buf_cookie *b = _p;
+ if (!n->child) {
+ if (n->desc && n->desc->action && n->desc->action->name)
+ b->buf_offset +=
+ snprintf(b->buf + b->buf_offset,
+ PAGE_SIZE - b->buf_offset,
+ "%d %s\n",
+ n->irq, n->desc->action->name);
+ else
+ b->buf_offset +=
+ snprintf(b->buf + b->buf_offset,
+ PAGE_SIZE - b->buf_offset,
+ "%d\n",
+ n->irq);
+ }
+ return true;
+}
+
+static ssize_t last_resume_reason_show(struct kobject *kobj,
+ struct kobj_attribute *attr,
+ char *buf)
+{
+ unsigned long flags;
+
+ struct buf_cookie b = {
+ .buf = buf,
+ .buf_offset = 0
+ };
+
+ spin_lock_irqsave(&resume_reason_lock, flags);
+ if (suspend_abort)
+ b.buf_offset = snprintf(buf, PAGE_SIZE, "Abort: %s", abort_reason);
+ else
+ walk_irq_node_tree(base_irq_nodes, print_leaf_node, &b);
+ spin_unlock_irqrestore(&resume_reason_lock, flags);
+
+ return b.buf_offset;
}
static ssize_t last_suspend_time_show(struct kobject *kobj,
@@ -104,56 +294,156 @@ static struct attribute_group attr_group = {
.attrs = attrs,
};
+static inline void stop_logging_wakeup_reasons(void)
+{
+ ACCESS_ONCE(log_wakeups) = false;
+ smp_wmb();
+}
+
/*
- * logs all the wake up reasons to the kernel
- * stores the irqs to expose them to the userspace via sysfs
+ * stores the immediate wakeup irqs; these often aren't the ones seen by
+ * the drivers that registered them, due to chained interrupt controllers,
+ * and multiple-interrupt dispatch.
*/
-void log_wakeup_reason(int irq)
+void log_base_wakeup_reason(int irq)
{
- struct irq_desc *desc;
- desc = irq_to_desc(irq);
- if (desc && desc->action && desc->action->name)
- printk(KERN_INFO "Resume caused by IRQ %d, %s\n", irq,
- desc->action->name);
- else
- printk(KERN_INFO "Resume caused by IRQ %d\n", irq);
+ unsigned long flags;
- spin_lock(&resume_reason_lock);
- if (irqcount == MAX_WAKEUP_REASON_IRQS) {
- spin_unlock(&resume_reason_lock);
- printk(KERN_WARNING "Resume caused by more than %d IRQs\n",
- MAX_WAKEUP_REASON_IRQS);
+ spin_lock_irqsave(&resume_reason_lock, flags);
+ base_irq_nodes = add_to_siblings(base_irq_nodes, irq);
+ BUG_ON(!base_irq_nodes);
+#ifndef CONFIG_DEDUCE_WAKEUP_REASONS
+ base_irq_nodes->handled = true;
+#endif
+ spin_unlock_irqrestore(&resume_reason_lock, flags);
+}
+
+#ifdef CONFIG_DEDUCE_WAKEUP_REASONS
+
+/* This function is called by generic_handle_irq, which may call itself
+ * recursively. This happens with interrupts disabled. Using
+ * log_possible_wakeup_reason, we build a tree of interrupts, tracing the call
+ * stack of generic_handle_irq, for each wakeup source containing the
+ * interrupts actually handled.
+ *
+ * Most of these "trees" would either have a single node (in the event that the
+ * wakeup source is the final interrupt), or consist of a list of two
+ * interrupts, with the wakeup source at the root, and the final dispatched
+ * interrupt at the leaf.
+ *
+ * When *all* wakeup sources have been thusly spoken for, this function will
+ * clear the log_wakeups flag, and print the wakeup reasons.
+
+ TODO: percpu
+
+ */
+
+static struct wakeup_irq_node *
+log_possible_wakeup_reason_start(int irq, struct irq_desc *desc, unsigned depth)
+{
+ BUG_ON(!irqs_disabled());
+ BUG_ON((signed)depth < 0);
+
+ /* This function can race with a call to stop_logging_wakeup_reasons()
+ * from a thread context. If this happens, just exit silently, as we are no
+ * longer interested in logging interrupts.
+ */
+ if (!logging_wakeup_reasons())
+ return NULL;
+
+ /* If suspend was aborted, the base IRQ nodes are missing, and we stop
+ * logging interrupts immediately.
+ */
+ if (!base_irq_nodes) {
+ stop_logging_wakeup_reasons();
+ return NULL;
+ }
+
+ /* We assume wakeup interrupts are handlerd only by the first core. */
+ /* TODO: relax this by having percpu versions of the irq tree */
+ if (smp_processor_id() != 0) {
+ return NULL;
+ }
+
+ if (depth == 0) {
+ cur_irq_tree_depth = 0;
+ cur_irq_tree = search_siblings(base_irq_nodes, irq);
+ }
+ else if (cur_irq_tree) {
+ if (depth > cur_irq_tree_depth) {
+ BUG_ON(depth - cur_irq_tree_depth > 1);
+ cur_irq_tree = add_child(cur_irq_tree, irq);
+ if (cur_irq_tree)
+ cur_irq_tree_depth++;
+ }
+ else {
+ cur_irq_tree = get_base_node(cur_irq_tree,
+ cur_irq_tree_depth - depth);
+ cur_irq_tree_depth = depth;
+ cur_irq_tree = add_to_siblings(cur_irq_tree, irq);
+ }
+ }
+
+ return cur_irq_tree;
+}
+
+static void log_possible_wakeup_reason_complete(struct wakeup_irq_node *n,
+ unsigned depth,
+ bool handled)
+{
+ if (!n)
return;
+ n->handled = handled;
+ if (depth == 0) {
+ if (base_irq_nodes_done()) {
+ stop_logging_wakeup_reasons();
+ complete(&wakeups_completion);
+ print_wakeup_sources();
+ }
}
-
- irq_list[irqcount++] = irq;
- spin_unlock(&resume_reason_lock);
}
-int check_wakeup_reason(int irq)
+bool log_possible_wakeup_reason(int irq,
+ struct irq_desc *desc,
+ bool (*handler)(struct irq_desc *))
{
- int irq_no;
- int ret = false;
+ static DEFINE_PER_CPU(unsigned int, depth);
- spin_lock(&resume_reason_lock);
- for (irq_no = 0; irq_no < irqcount; irq_no++)
- if (irq_list[irq_no] == irq) {
- ret = true;
- break;
- }
- spin_unlock(&resume_reason_lock);
- return ret;
+ struct wakeup_irq_node *n;
+ bool handled;
+ unsigned d;
+
+ d = get_cpu_var(depth)++;
+ put_cpu_var(depth);
+
+ n = log_possible_wakeup_reason_start(irq, desc, d);
+
+ handled = handler(desc);
+
+ d = --get_cpu_var(depth);
+ put_cpu_var(depth);
+
+ if (!handled && desc && desc->action)
+ pr_debug("%s: irq %d action %pF not handled\n", __func__,
+ irq, desc->action->handler);
+
+ log_possible_wakeup_reason_complete(n, d, handled);
+
+ return handled;
}
+#endif /* CONFIG_DEDUCE_WAKEUP_REASONS */
+
void log_suspend_abort_reason(const char *fmt, ...)
{
va_list args;
+ unsigned long flags;
- spin_lock(&resume_reason_lock);
+ spin_lock_irqsave(&resume_reason_lock, flags);
//Suspend abort reason has already been logged.
if (suspend_abort) {
- spin_unlock(&resume_reason_lock);
+ spin_unlock_irqrestore(&resume_reason_lock, flags);
return;
}
@@ -161,29 +451,128 @@ void log_suspend_abort_reason(const char *fmt, ...)
va_start(args, fmt);
vsnprintf(abort_reason, MAX_SUSPEND_ABORT_LEN, fmt, args);
va_end(args);
- spin_unlock(&resume_reason_lock);
+
+ spin_unlock_irqrestore(&resume_reason_lock, flags);
+}
+
+static bool match_node(struct wakeup_irq_node *n, void *_p)
+{
+ int irq = *((int *)_p);
+ return n->irq != irq;
+}
+
+int check_wakeup_reason(int irq)
+{
+ bool found;
+ unsigned long flags;
+ spin_lock_irqsave(&resume_reason_lock, flags);
+ found = !walk_irq_node_tree(base_irq_nodes, match_node, &irq);
+ spin_unlock_irqrestore(&resume_reason_lock, flags);
+ return found;
+}
+
+static bool build_leaf_nodes(struct wakeup_irq_node *n, void *_p)
+{
+ struct list_head *wakeups = _p;
+ if (!n->child)
+ list_add(&n->next, wakeups);
+ return true;
+}
+
+static const struct list_head* get_wakeup_reasons_nosync(void)
+{
+ BUG_ON(logging_wakeup_reasons());
+ INIT_LIST_HEAD(&wakeup_irqs);
+ walk_irq_node_tree(base_irq_nodes, build_leaf_nodes, &wakeup_irqs);
+ return &wakeup_irqs;
+}
+
+static bool build_unfinished_nodes(struct wakeup_irq_node *n, void *_p)
+{
+ struct list_head *unfinished = _p;
+ if (!n->handled) {
+ pr_warning("%s: wakeup irq %d was not handled\n",
+ __func__, n->irq);
+ list_add(&n->next, unfinished);
+ }
+ return true;
+}
+
+const struct list_head* get_wakeup_reasons(unsigned long timeout,
+ struct list_head *unfinished)
+{
+ INIT_LIST_HEAD(unfinished);
+
+ if (logging_wakeup_reasons()) {
+ unsigned long signalled = 0;
+ if (timeout)
+ signalled = wait_for_completion_timeout(&wakeups_completion, timeout);
+ if (WARN_ON(!signalled)) {
+ stop_logging_wakeup_reasons();
+ walk_irq_node_tree(base_irq_nodes, build_unfinished_nodes, unfinished);
+ return NULL;
+ }
+ pr_info("%s: waited for %u ms\n",
+ __func__,
+ jiffies_to_msecs(timeout - signalled));
+ }
+
+ return get_wakeup_reasons_nosync();
+}
+
+static bool delete_node(struct wakeup_irq_node *n, void *unused)
+{
+ list_del(&n->siblings);
+ kmem_cache_free(wakeup_irq_nodes_cache, n);
+ return true;
+}
+
+void clear_wakeup_reasons(void)
+{
+ unsigned long flags;
+ spin_lock_irqsave(&resume_reason_lock, flags);
+
+ BUG_ON(logging_wakeup_reasons());
+ walk_irq_node_tree(base_irq_nodes, delete_node, NULL);
+ base_irq_nodes = NULL;
+ cur_irq_tree = NULL;
+ cur_irq_tree_depth = 0;
+ INIT_LIST_HEAD(&wakeup_irqs);
+ suspend_abort = false;
+
+ spin_unlock_irqrestore(&resume_reason_lock, flags);
}
/* Detects a suspend and clears all the previous wake up reasons*/
static int wakeup_reason_pm_event(struct notifier_block *notifier,
unsigned long pm_event, void *unused)
{
+ unsigned long flags;
switch (pm_event) {
case PM_SUSPEND_PREPARE:
- spin_lock(&resume_reason_lock);
- irqcount = 0;
+ spin_lock_irqsave(&resume_reason_lock, flags);
suspend_abort = false;
- spin_unlock(&resume_reason_lock);
+ spin_unlock_irqrestore(&resume_reason_lock, flags);
/* monotonic time since boot */
last_monotime = ktime_get();
/* monotonic time since boot including the time spent in suspend */
last_stime = ktime_get_boottime();
+ clear_wakeup_reasons();
break;
case PM_POST_SUSPEND:
/* monotonic time since boot */
curr_monotime = ktime_get();
/* monotonic time since boot including the time spent in suspend */
curr_stime = ktime_get_boottime();
+#ifdef CONFIG_DEDUCE_WAKEUP_REASONS
+ /* log_wakeups should have been cleared by now. */
+ if (WARN_ON(logging_wakeup_reasons())) {
+ stop_logging_wakeup_reasons();
+ print_wakeup_sources();
+ }
+#else
+ print_wakeup_sources();
+#endif
break;
default:
break;
@@ -195,31 +584,46 @@ static struct notifier_block wakeup_reason_pm_notifier_block = {
.notifier_call = wakeup_reason_pm_event,
};
-/* Initializes the sysfs parameter
- * registers the pm_event notifier
- */
int __init wakeup_reason_init(void)
{
- int retval;
+ spin_lock_init(&resume_reason_lock);
- retval = register_pm_notifier(&wakeup_reason_pm_notifier_block);
- if (retval)
- printk(KERN_WARNING "[%s] failed to register PM notifier %d\n",
- __func__, retval);
+ if (register_pm_notifier(&wakeup_reason_pm_notifier_block)) {
+ pr_warning("[%s] failed to register PM notifier\n",
+ __func__);
+ goto fail;
+ }
wakeup_reason = kobject_create_and_add("wakeup_reasons", kernel_kobj);
if (!wakeup_reason) {
- printk(KERN_WARNING "[%s] failed to create a sysfs kobject\n",
+ pr_warning("[%s] failed to create a sysfs kobject\n",
__func__);
- return 1;
+ goto fail_unregister_pm_notifier;
}
- retval = sysfs_create_group(wakeup_reason, &attr_group);
- if (retval) {
- kobject_put(wakeup_reason);
- printk(KERN_WARNING "[%s] failed to create a sysfs group %d\n",
- __func__, retval);
+
+ if (sysfs_create_group(wakeup_reason, &attr_group)) {
+ pr_warning("[%s] failed to create a sysfs group\n",
+ __func__);
+ goto fail_kobject_put;
}
+
+ wakeup_irq_nodes_cache =
+ kmem_cache_create("wakeup_irq_node_cache",
+ sizeof(struct wakeup_irq_node), 0,
+ 0, NULL);
+ if (!wakeup_irq_nodes_cache)
+ goto fail_remove_group;
+
return 0;
+
+fail_remove_group:
+ sysfs_remove_group(wakeup_reason, &attr_group);
+fail_kobject_put:
+ kobject_put(wakeup_reason);
+fail_unregister_pm_notifier:
+ unregister_pm_notifier(&wakeup_reason_pm_notifier_block);
+fail:
+ return 1;
}
late_initcall(wakeup_reason_init);
diff --git a/kernel/printk/nmi.c b/kernel/printk/nmi.c
index 16bab47..2c3e7f0 100644
--- a/kernel/printk/nmi.c
+++ b/kernel/printk/nmi.c
@@ -63,6 +63,7 @@ static int vprintk_nmi(const char *fmt, va_list args)
struct nmi_seq_buf *s = this_cpu_ptr(&nmi_print_seq);
int add = 0;
size_t len;
+ va_list ap;
again:
len = atomic_read(&s->len);
@@ -79,7 +80,9 @@ static int vprintk_nmi(const char *fmt, va_list args)
if (!len)
smp_rmb();
- add = vsnprintf(s->buffer + len, sizeof(s->buffer) - len, fmt, args);
+ va_copy(ap, args);
+ add = vsnprintf(s->buffer + len, sizeof(s->buffer) - len, fmt, ap);
+ va_end(ap);
/*
* Do it once again if the buffer has been flushed in the meantime.
@@ -257,12 +260,12 @@ void __init printk_nmi_init(void)
printk_nmi_flush();
}
-void printk_nmi_enter(void)
+void notrace printk_nmi_enter(void)
{
this_cpu_write(printk_func, vprintk_nmi);
}
-void printk_nmi_exit(void)
+void notrace printk_nmi_exit(void)
{
this_cpu_write(printk_func, vprintk_default);
}
diff --git a/kernel/printk/printk.c b/kernel/printk/printk.c
index 396b020..71ad376 100644
--- a/kernel/printk/printk.c
+++ b/kernel/printk/printk.c
@@ -342,12 +342,16 @@ struct printk_log {
u8 facility; /* syslog facility */
u8 flags:5; /* internal record flags */
u8 level:3; /* syslog level */
+ unsigned int cpu;
+ pid_t pid;
}
#ifdef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
__packed __aligned(4)
#endif
;
+static struct printk_log *last_msg;
+
/*
* The logbuf_lock protects kmsg buffer, indices, counters. This can be taken
* within the scheduler's rq lock. It must be released before calling
@@ -589,6 +593,9 @@ static int log_store(int facility, int level,
log_next_idx += msg->len;
log_next_seq++;
+ /* record last_msg for extension */
+ last_msg = msg;
+
return msg->text_len;
}
@@ -606,7 +613,7 @@ static int syslog_action_restricted(int type)
type != SYSLOG_ACTION_SIZE_BUFFER;
}
-int check_syslog_permissions(int type, int source)
+static int check_syslog_permissions(int type, int source)
{
/*
* If this is from /proc/kmsg and we've already opened it, then we've
@@ -634,7 +641,6 @@ int check_syslog_permissions(int type, int source)
ok:
return security_syslog(type);
}
-EXPORT_SYMBOL_GPL(check_syslog_permissions);
static void append_char(char **pp, char *e, char c)
{
@@ -1014,7 +1020,12 @@ static void __init log_buf_len_update(unsigned size)
/* save requested log_buf_len since it's too early to process it */
static int __init log_buf_len_setup(char *str)
{
- unsigned size = memparse(str, &str);
+ unsigned int size;
+
+ if (!str)
+ return -EINVAL;
+
+ size = memparse(str, &str);
log_buf_len_update(size);
@@ -1191,6 +1202,43 @@ static size_t print_time(u64 ts, char *buf)
(unsigned long)ts, rem_nsec / 1000);
}
+static bool printk_cpu = IS_ENABLED(CONFIG_PRINTK_CPU_ID);
+module_param_named(cpu, printk_cpu, bool, S_IRUGO | S_IWUSR);
+
+static bool printk_pid = IS_ENABLED(CONFIG_PRINTK_PID);
+module_param_named(pid, printk_pid, bool, S_IRUGO | S_IWUSR);
+
+static size_t print_cpu(unsigned int cpu, char *buf)
+{
+ if (!printk_cpu)
+ return 0;
+
+ if (!buf)
+ return snprintf(NULL, 0, "c%u ", cpu);
+
+ return sprintf(buf, "c%u ", cpu);
+}
+
+static size_t print_pid(pid_t pid, char *buf)
+{
+ if (!printk_pid)
+ return 0;
+
+ if (!buf)
+ return snprintf(NULL, 0, "%6u ", pid);
+
+ return sprintf(buf, "%6u ", pid);
+}
+
+static void update_msg_ext(unsigned int cpu, pid_t pid)
+{
+ if (!last_msg)
+ return;
+
+ last_msg->cpu = cpu;
+ last_msg->pid = pid;
+}
+
static size_t print_prefix(const struct printk_log *msg, bool syslog, char *buf)
{
size_t len = 0;
@@ -1211,6 +1259,8 @@ static size_t print_prefix(const struct printk_log *msg, bool syslog, char *buf)
}
len += print_time(msg->ts_nsec, buf ? buf + len : NULL);
+ len += print_cpu(msg->cpu, buf ? buf + len : NULL);
+ len += print_pid(msg->pid, buf ? buf + len : NULL);
return len;
}
@@ -1642,6 +1692,8 @@ static struct cont {
u8 facility; /* log facility of first message */
enum log_flags flags; /* prefix, newline flags */
bool flushed:1; /* buffer sealed and committed */
+ unsigned int cpu;
+ pid_t pid;
} cont;
static void cont_flush(void)
@@ -1659,6 +1711,7 @@ static void cont_flush(void)
log_store(cont.facility, cont.level, cont.flags | LOG_NOCONS,
cont.ts_nsec, NULL, 0, cont.buf, cont.len);
cont.flushed = true;
+ update_msg_ext(cont.cpu, cont.pid);
} else {
/*
* If no fragment of this line ever reached the console,
@@ -1666,6 +1719,7 @@ static void cont_flush(void)
*/
log_store(cont.facility, cont.level, cont.flags, 0,
NULL, 0, cont.buf, cont.len);
+ update_msg_ext(cont.cpu, cont.pid);
cont.len = 0;
}
}
@@ -1693,6 +1747,8 @@ static bool cont_add(int facility, int level, enum log_flags flags, const char *
cont.flags = flags;
cont.cons = 0;
cont.flushed = false;
+ cont.cpu = smp_processor_id();
+ cont.pid = current->pid;
}
memcpy(cont.buf + cont.len, text, len);
@@ -1829,6 +1885,7 @@ asmlinkage int vprintk_emit(int facility, int level,
printed_len += log_store(0, 2, LOG_PREFIX|LOG_NEWLINE, 0,
NULL, 0, recursion_msg,
strlen(recursion_msg));
+ update_msg_ext(logbuf_cpu, current->pid);
}
nmi_message_lost = get_nmi_message_lost();
@@ -1838,6 +1895,7 @@ asmlinkage int vprintk_emit(int facility, int level,
nmi_message_lost);
printed_len += log_store(0, 2, LOG_PREFIX|LOG_NEWLINE, 0,
NULL, 0, textbuf, text_len);
+ update_msg_ext(logbuf_cpu, current->pid);
}
/*
@@ -1885,6 +1943,7 @@ asmlinkage int vprintk_emit(int facility, int level,
lflags |= LOG_PREFIX|LOG_NEWLINE;
printed_len += log_output(facility, level, lflags, dict, dictlen, text, text_len);
+ update_msg_ext(logbuf_cpu, current->pid);
logbuf_cpu = UINT_MAX;
raw_spin_unlock(&logbuf_lock);
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index e3944c4..554ea54 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -521,8 +521,14 @@ static void rcu_print_detail_task_stall_rnp(struct rcu_node *rnp)
}
t = list_entry(rnp->gp_tasks->prev,
struct task_struct, rcu_node_entry);
- list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry)
+ list_for_each_entry_continue(t, &rnp->blkd_tasks, rcu_node_entry) {
+ /*
+ * We could be printing a lot while holding a spinlock.
+ * Avoid triggering hard lockup.
+ */
+ touch_nmi_watchdog();
sched_show_task(t);
+ }
raw_spin_unlock_irqrestore_rcu_node(rnp, flags);
}
@@ -1629,6 +1635,12 @@ static void print_cpu_stall_info(struct rcu_state *rsp, int cpu)
char *ticks_title;
unsigned long ticks_value;
+ /*
+ * We could be printing a lot while holding a spinlock. Avoid
+ * triggering hard lockup.
+ */
+ touch_nmi_watchdog();
+
if (rsp->gpnum == rdp->gpnum) {
ticks_title = "ticks this GP";
ticks_value = rdp->ticks_this_gp;
diff --git a/kernel/relay.c b/kernel/relay.c
index 2603e04..91e8fbf 100644
--- a/kernel/relay.c
+++ b/kernel/relay.c
@@ -163,7 +163,7 @@ static struct rchan_buf *relay_create_buf(struct rchan *chan)
{
struct rchan_buf *buf;
- if (chan->n_subbufs > UINT_MAX / sizeof(size_t *))
+ if (chan->n_subbufs > KMALLOC_MAX_SIZE / sizeof(size_t *))
return NULL;
buf = kzalloc(sizeof(struct rchan_buf), GFP_KERNEL);
diff --git a/kernel/sched/Makefile b/kernel/sched/Makefile
index 38cbc0d..499d53b 100644
--- a/kernel/sched/Makefile
+++ b/kernel/sched/Makefile
@@ -15,6 +15,10 @@
CFLAGS_core.o := $(PROFILING) -fno-omit-frame-pointer
endif
+ifeq ($(call clang-ifversion, -ge, 0600, y), y)
+ccflags-y += -Wno-section
+endif
+
obj-y += core.o loadavg.o clock.o cputime.o
obj-y += idle_task.o fair.o rt.o deadline.o stop_task.o
obj-y += wait.o swait.o completion.o idle.o
@@ -28,3 +32,4 @@
obj-$(CONFIG_CPU_FREQ) += cpufreq.o
obj-$(CONFIG_CPU_FREQ_GOV_SCHEDUTIL) += cpufreq_schedutil.o
obj-$(CONFIG_SCHED_CORE_CTL) += core_ctl.o
+obj-$(CONFIG_PSI) += psi.o
diff --git a/kernel/sched/boost.c b/kernel/sched/boost.c
index 09ad1f0..9c2269f 100644
--- a/kernel/sched/boost.c
+++ b/kernel/sched/boost.c
@@ -202,8 +202,14 @@ int sched_boost_handler(struct ctl_table *table, int write,
if (verify_boost_params(old_val, *data)) {
_sched_set_boost(old_val, *data);
} else {
- *data = old_val;
- ret = -EINVAL;
+ /*
+ * Only return error when switching from one boost type
+ * to another.
+ */
+ if (old_val != *data) {
+ *data = old_val;
+ ret = -EINVAL;
+ }
}
done:
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index fb2e56c..f97e8f6 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -77,6 +77,8 @@
#include <linux/prefetch.h>
#include <linux/irq.h>
#include <linux/cpufreq_times.h>
+#include <linux/sched/loadavg.h>
+#include <linux/scs.h>
#include <asm/switch_to.h>
#include <asm/tlb.h>
@@ -176,21 +178,6 @@ unlock_rq_of(struct rq *rq, struct task_struct *p, struct rq_flags *flags)
}
/*
- * this_rq_lock - lock this runqueue and disable interrupts.
- */
-static struct rq *this_rq_lock(void)
- __acquires(rq->lock)
-{
- struct rq *rq;
-
- local_irq_disable();
- rq = this_rq();
- raw_spin_lock(&rq->lock);
-
- return rq;
-}
-
-/*
* __task_rq_lock - lock the rq @p resides on.
*/
struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)
@@ -204,7 +191,7 @@ struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)
rq = task_rq(p);
raw_spin_lock(&rq->lock);
if (likely(rq == task_rq(p) && !task_on_rq_migrating(p))) {
- rf->cookie = lockdep_pin_lock(&rq->lock);
+ rq_pin_lock(rq, rf);
return rq;
}
raw_spin_unlock(&rq->lock);
@@ -244,7 +231,7 @@ struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)
* pair with the WMB to ensure we must then also see migrating.
*/
if (likely(rq == task_rq(p) && !task_on_rq_migrating(p))) {
- rf->cookie = lockdep_pin_lock(&rq->lock);
+ rq_pin_lock(rq, rf);
return rq;
}
raw_spin_unlock(&rq->lock);
@@ -774,8 +761,10 @@ static void set_load_weight(struct task_struct *p)
static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
{
update_rq_clock(rq);
- if (!(flags & ENQUEUE_RESTORE))
+ if (!(flags & ENQUEUE_RESTORE)) {
sched_info_queued(rq, p);
+ psi_enqueue(p, flags & ENQUEUE_WAKEUP);
+ }
p->sched_class->enqueue_task(rq, p, flags);
walt_update_last_enqueue(p);
trace_sched_enq_deq_task(p, 1, cpumask_bits(&p->cpus_allowed)[0]);
@@ -784,8 +773,10 @@ static inline void enqueue_task(struct rq *rq, struct task_struct *p, int flags)
static inline void dequeue_task(struct rq *rq, struct task_struct *p, int flags)
{
update_rq_clock(rq);
- if (!(flags & DEQUEUE_SAVE))
+ if (!(flags & DEQUEUE_SAVE)) {
sched_info_dequeued(rq, p);
+ psi_dequeue(p, flags & DEQUEUE_SLEEP);
+ }
p->sched_class->dequeue_task(rq, p, flags);
trace_sched_enq_deq_task(p, 0, cpumask_bits(&p->cpus_allowed)[0]);
}
@@ -1239,9 +1230,9 @@ static int __set_cpus_allowed_ptr(struct task_struct *p,
* OK, since we're going to drop the lock immediately
* afterwards anyway.
*/
- lockdep_unpin_lock(&rq->lock, rf.cookie);
+ rq_unpin_lock(rq, &rf);
rq = move_queued_task(rq, p, dest_cpu);
- lockdep_repin_lock(&rq->lock, rf.cookie);
+ rq_repin_lock(rq, &rf);
}
out:
task_rq_unlock(rq, p, &rf);
@@ -1757,7 +1748,7 @@ static inline void ttwu_activate(struct rq *rq, struct task_struct *p, int en_fl
* Mark the task runnable and perform wakeup-preemption.
*/
static void ttwu_do_wakeup(struct rq *rq, struct task_struct *p, int wake_flags,
- struct pin_cookie cookie)
+ struct rq_flags *rf)
{
check_preempt_curr(rq, p, wake_flags);
p->state = TASK_RUNNING;
@@ -1769,9 +1760,9 @@ static void ttwu_do_wakeup(struct rq *rq, struct task_struct *p, int wake_flags,
* Our task @p is fully woken up and running; so its safe to
* drop the rq->lock, hereafter rq is only used for statistics.
*/
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_unpin_lock(rq, rf);
p->sched_class->task_woken(rq, p);
- lockdep_repin_lock(&rq->lock, cookie);
+ rq_repin_lock(rq, rf);
}
if (rq->idle_stamp) {
@@ -1790,7 +1781,7 @@ static void ttwu_do_wakeup(struct rq *rq, struct task_struct *p, int wake_flags,
static void
ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
- struct pin_cookie cookie)
+ struct rq_flags *rf)
{
int en_flags = ENQUEUE_WAKEUP;
@@ -1805,7 +1796,7 @@ ttwu_do_activate(struct rq *rq, struct task_struct *p, int wake_flags,
#endif
ttwu_activate(rq, p, en_flags);
- ttwu_do_wakeup(rq, p, wake_flags, cookie);
+ ttwu_do_wakeup(rq, p, wake_flags, rf);
}
/*
@@ -1824,7 +1815,7 @@ static int ttwu_remote(struct task_struct *p, int wake_flags)
if (task_on_rq_queued(p)) {
/* check_preempt_curr() may use rq clock */
update_rq_clock(rq);
- ttwu_do_wakeup(rq, p, wake_flags, rf.cookie);
+ ttwu_do_wakeup(rq, p, wake_flags, &rf);
ret = 1;
}
__task_rq_unlock(rq, &rf);
@@ -1837,15 +1828,15 @@ void sched_ttwu_pending(void)
{
struct rq *rq = this_rq();
struct llist_node *llist = llist_del_all(&rq->wake_list);
- struct pin_cookie cookie;
struct task_struct *p;
unsigned long flags;
+ struct rq_flags rf;
if (!llist)
return;
raw_spin_lock_irqsave(&rq->lock, flags);
- cookie = lockdep_pin_lock(&rq->lock);
+ rq_pin_lock(rq, &rf);
while (llist) {
int wake_flags = 0;
@@ -1856,10 +1847,10 @@ void sched_ttwu_pending(void)
if (p->sched_remote_wakeup)
wake_flags = WF_MIGRATED;
- ttwu_do_activate(rq, p, wake_flags, cookie);
+ ttwu_do_activate(rq, p, wake_flags, &rf);
}
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_unpin_lock(rq, &rf);
raw_spin_unlock_irqrestore(&rq->lock, flags);
}
@@ -1959,7 +1950,7 @@ bool cpus_share_cache(int this_cpu, int that_cpu)
static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
{
struct rq *rq = cpu_rq(cpu);
- struct pin_cookie cookie;
+ struct rq_flags rf;
#if defined(CONFIG_SMP)
if (sched_feat(TTWU_QUEUE) && !cpus_share_cache(smp_processor_id(), cpu)) {
@@ -1970,9 +1961,9 @@ static void ttwu_queue(struct task_struct *p, int cpu, int wake_flags)
#endif
raw_spin_lock(&rq->lock);
- cookie = lockdep_pin_lock(&rq->lock);
- ttwu_do_activate(rq, p, wake_flags, cookie);
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_pin_lock(rq, &rf);
+ ttwu_do_activate(rq, p, wake_flags, &rf);
+ rq_unpin_lock(rq, &rf);
raw_spin_unlock(&rq->lock);
}
@@ -2191,6 +2182,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
src_cpu = task_cpu(p);
if (src_cpu != cpu) {
wake_flags |= WF_MIGRATED;
+ psi_ttwu_dequeue(p);
set_task_cpu(p, cpu);
notif_required = true;
}
@@ -2207,9 +2199,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
if (success && sched_predl) {
raw_spin_lock_irqsave(&cpu_rq(cpu)->lock, flags);
if (do_pl_notif(cpu_rq(cpu)))
- cpufreq_update_util(cpu_rq(cpu),
- SCHED_CPUFREQ_WALT |
- SCHED_CPUFREQ_PL);
+ cpufreq_update_util(cpu_rq(cpu), SCHED_CPUFREQ_PL);
raw_spin_unlock_irqrestore(&cpu_rq(cpu)->lock, flags);
}
@@ -2225,7 +2215,7 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
* ensure that this_rq() is locked, @p is bound to this_rq() and not
* the current task.
*/
-static void try_to_wake_up_local(struct task_struct *p, struct pin_cookie cookie)
+static void try_to_wake_up_local(struct task_struct *p, struct rq_flags *rf)
{
struct rq *rq = task_rq(p);
@@ -2246,11 +2236,11 @@ static void try_to_wake_up_local(struct task_struct *p, struct pin_cookie cookie
* disabled avoiding further scheduler activity on it and we've
* not yet picked a replacement task.
*/
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_unpin_lock(rq, rf);
raw_spin_unlock(&rq->lock);
raw_spin_lock(&p->pi_lock);
raw_spin_lock(&rq->lock);
- lockdep_repin_lock(&rq->lock, cookie);
+ rq_repin_lock(rq, rf);
}
if (!(p->state & TASK_NORMAL))
@@ -2267,7 +2257,7 @@ static void try_to_wake_up_local(struct task_struct *p, struct pin_cookie cookie
note_task_waking(p, wallclock);
}
- ttwu_do_wakeup(rq, p, 0, cookie);
+ ttwu_do_wakeup(rq, p, 0, rf);
ttwu_stat(p, smp_processor_id(), 0);
out:
raw_spin_unlock(&p->pi_lock);
@@ -2331,7 +2321,7 @@ static void __sched_fork(unsigned long clone_flags, struct task_struct *p)
p->se.nr_migrations = 0;
p->se.vruntime = 0;
p->last_sleep_ts = 0;
- p->last_cpu_selected_ts = 0;
+ p->last_cpu_deselected_ts = 0;
INIT_LIST_HEAD(&p->se.group_node);
@@ -2509,6 +2499,9 @@ int sched_fork(unsigned long clone_flags, struct task_struct *p)
cpu = get_cpu();
__sched_fork(clone_flags, p);
+#ifdef CONFIG_CPU_FREQ_TIMES
+ cpufreq_task_times_alloc(p);
+#endif
/*
* We mark the process as NEW here. This guarantees that
* nobody will actually run it, and a signal or other external
@@ -2728,9 +2721,9 @@ void wake_up_new_task(struct task_struct *p)
* Nothing relies on rq->lock after this, so its fine to
* drop it.
*/
- lockdep_unpin_lock(&rq->lock, rf.cookie);
+ rq_unpin_lock(rq, &rf);
p->sched_class->task_woken(rq, p);
- lockdep_repin_lock(&rq->lock, rf.cookie);
+ rq_repin_lock(rq, &rf);
}
#endif
task_rq_unlock(rq, p, &rf);
@@ -2999,7 +2992,7 @@ asmlinkage __visible void schedule_tail(struct task_struct *prev)
*/
static __always_inline struct rq *
context_switch(struct rq *rq, struct task_struct *prev,
- struct task_struct *next, struct pin_cookie cookie)
+ struct task_struct *next, struct rq_flags *rf)
{
struct mm_struct *mm, *oldmm;
@@ -3031,7 +3024,7 @@ context_switch(struct rq *rq, struct task_struct *prev,
* of the scheduler it's an obvious special-case), so we
* do an early lockdep release here:
*/
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_unpin_lock(rq, rf);
spin_release(&rq->lock.dep_map, 1, _THIS_IP_);
/* Here we just switch the register state and the stack. */
@@ -3252,7 +3245,6 @@ void scheduler_tick(void)
bool early_notif;
u32 old_load;
struct related_thread_group *grp;
- unsigned int flag = 0;
sched_clock_tick();
@@ -3268,12 +3260,12 @@ void scheduler_tick(void)
curr->sched_class->task_tick(rq, curr, 0);
cpu_load_update_active(rq);
calc_global_load_tick(rq);
+ psi_task_tick(rq);
early_notif = early_detection_notify(rq, wallclock);
if (early_notif)
- flag = SCHED_CPUFREQ_WALT | SCHED_CPUFREQ_EARLY_DET;
+ cpufreq_update_util(rq, SCHED_CPUFREQ_EARLY_DET);
- cpufreq_update_util(rq, flag);
raw_spin_unlock(&rq->lock);
if (early_notif)
@@ -3495,7 +3487,7 @@ static inline void schedule_debug(struct task_struct *prev)
* Pick up the highest-prio task:
*/
static inline struct task_struct *
-pick_next_task(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie)
+pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
{
const struct sched_class *class = &fair_sched_class;
struct task_struct *p;
@@ -3506,20 +3498,20 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie
*/
if (likely(prev->sched_class == class &&
rq->nr_running == rq->cfs.h_nr_running)) {
- p = fair_sched_class.pick_next_task(rq, prev, cookie);
+ p = fair_sched_class.pick_next_task(rq, prev, rf);
if (unlikely(p == RETRY_TASK))
goto again;
/* assumes fair_sched_class->next == idle_sched_class */
if (unlikely(!p))
- p = idle_sched_class.pick_next_task(rq, prev, cookie);
+ p = idle_sched_class.pick_next_task(rq, prev, rf);
return p;
}
again:
for_each_class(class) {
- p = class->pick_next_task(rq, prev, cookie);
+ p = class->pick_next_task(rq, prev, rf);
if (p) {
if (unlikely(p == RETRY_TASK))
goto again;
@@ -3573,7 +3565,7 @@ static void __sched notrace __schedule(bool preempt)
{
struct task_struct *prev, *next;
unsigned long *switch_count;
- struct pin_cookie cookie;
+ struct rq_flags rf;
struct rq *rq;
int cpu;
u64 wallclock;
@@ -3597,7 +3589,7 @@ static void __sched notrace __schedule(bool preempt)
*/
smp_mb__before_spinlock();
raw_spin_lock(&rq->lock);
- cookie = lockdep_pin_lock(&rq->lock);
+ rq_pin_lock(rq, &rf);
rq->clock_skip_update <<= 1; /* promote REQ to ACT */
@@ -3619,7 +3611,7 @@ static void __sched notrace __schedule(bool preempt)
to_wakeup = wq_worker_sleeping(prev);
if (to_wakeup)
- try_to_wake_up_local(to_wakeup, cookie);
+ try_to_wake_up_local(to_wakeup, &rf);
}
}
switch_count = &prev->nvcsw;
@@ -3628,13 +3620,14 @@ static void __sched notrace __schedule(bool preempt)
if (task_on_rq_queued(prev))
update_rq_clock(rq);
- next = pick_next_task(rq, prev, cookie);
+ next = pick_next_task(rq, prev, &rf);
clear_tsk_need_resched(prev);
clear_preempt_need_resched();
rq->clock_skip_update = 0;
wallclock = ktime_get_ns();
if (likely(prev != next)) {
+ prev->last_cpu_deselected_ts = wallclock;
if (!prev->on_rq)
prev->last_sleep_ts = wallclock;
@@ -3645,10 +3638,10 @@ static void __sched notrace __schedule(bool preempt)
++*switch_count;
trace_sched_switch(preempt, prev, next);
- rq = context_switch(rq, prev, next, cookie); /* unlocks the rq */
+ rq = context_switch(rq, prev, next, &rf); /* unlocks the rq */
} else {
update_task_ravg(prev, rq, TASK_UPDATE, wallclock, 0);
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_unpin_lock(rq, &rf);
raw_spin_unlock_irq(&rq->lock);
}
@@ -4332,9 +4325,8 @@ static int __sched_setscheduler(struct task_struct *p,
int queue_flags = DEQUEUE_SAVE | DEQUEUE_MOVE;
struct rq *rq;
- /* may grab non-irq protected spin_locks */
- if (pi)
- BUG_ON(in_interrupt());
+ /* The pi code expects interrupts enabled */
+ BUG_ON(pi && in_interrupt());
recheck:
/* double check policy once rq lock held */
if (policy < 0) {
@@ -5145,7 +5137,10 @@ SYSCALL_DEFINE3(sched_getaffinity, pid_t, pid, unsigned int, len,
*/
SYSCALL_DEFINE0(sched_yield)
{
- struct rq *rq = this_rq_lock();
+ struct rq_flags rf;
+ struct rq *rq;
+
+ rq = this_rq_lock_irq(&rf);
schedstat_inc(rq->yld_count);
current->sched_class->yield_task(rq);
@@ -5154,9 +5149,8 @@ SYSCALL_DEFINE0(sched_yield)
* Since we are going to call schedule() anyway, there's
* no need to preempt or enable interrupts:
*/
- __release(rq->lock);
- spin_release(&rq->lock.dep_map, 1, _THIS_IP_);
- do_raw_spin_unlock(&rq->lock);
+ preempt_disable();
+ rq_unlock(rq, &rf);
sched_preempt_enable_no_resched();
schedule();
@@ -5539,6 +5533,8 @@ void init_idle(struct task_struct *idle, int cpu, bool cpu_up)
struct rq *rq = cpu_rq(cpu);
unsigned long flags;
+ scs_task_reset(idle);
+
__sched_fork(0, idle);
if (!cpu_up)
@@ -5820,7 +5816,7 @@ static void migrate_tasks(struct rq *dead_rq, bool migrate_pinned_tasks)
{
struct rq *rq = dead_rq;
struct task_struct *next, *stop = rq->stop;
- struct pin_cookie cookie;
+ struct rq_flags rf;
int dest_cpu;
unsigned int num_pinned_kthreads = 1; /* this thread */
LIST_HEAD(tasks);
@@ -5857,8 +5853,8 @@ static void migrate_tasks(struct rq *dead_rq, bool migrate_pinned_tasks)
/*
* pick_next_task assumes pinned rq->lock.
*/
- cookie = lockdep_pin_lock(&rq->lock);
- next = pick_next_task(rq, &fake_task, cookie);
+ rq_pin_lock(rq, &rf);
+ next = pick_next_task(rq, &fake_task, &rf);
BUG_ON(!next);
next->sched_class->put_prev_task(rq, next);
@@ -5866,7 +5862,7 @@ static void migrate_tasks(struct rq *dead_rq, bool migrate_pinned_tasks)
!cpumask_intersects(&avail_cpus, &next->cpus_allowed)) {
detach_one_task(next, rq, &tasks);
num_pinned_kthreads += 1;
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_unpin_lock(rq, &rf);
continue;
}
@@ -5879,7 +5875,7 @@ static void migrate_tasks(struct rq *dead_rq, bool migrate_pinned_tasks)
* because !cpu_active at this point, which means load-balance
* will not interfere. Also, stop-machine.
*/
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_unpin_lock(rq, &rf);
raw_spin_unlock(&rq->lock);
raw_spin_lock(&next->pi_lock);
raw_spin_lock(&rq->lock);
@@ -7740,17 +7736,6 @@ static int build_sched_domains(const struct cpumask *cpu_map,
/* Attach the domains */
rcu_read_lock();
for_each_cpu(i, cpu_map) {
- int max_cpu = READ_ONCE(d.rd->max_cap_orig_cpu);
- int min_cpu = READ_ONCE(d.rd->min_cap_orig_cpu);
-
- if ((max_cpu < 0) || (cpu_rq(i)->cpu_capacity_orig >
- cpu_rq(max_cpu)->cpu_capacity_orig))
- WRITE_ONCE(d.rd->max_cap_orig_cpu, i);
-
- if ((min_cpu < 0) || (cpu_rq(i)->cpu_capacity_orig <
- cpu_rq(min_cpu)->cpu_capacity_orig))
- WRITE_ONCE(d.rd->min_cap_orig_cpu, i);
-
sd = *per_cpu_ptr(d.sd, i);
cpu_attach_domain(sd, d.rd, i);
@@ -8384,6 +8369,8 @@ void __init sched_init(void)
init_schedstats();
+ psi_init();
+
scheduler_running = 1;
}
diff --git a/kernel/sched/cpuacct.c b/kernel/sched/cpuacct.c
index bc0b309c..4c88279 100644
--- a/kernel/sched/cpuacct.c
+++ b/kernel/sched/cpuacct.c
@@ -297,7 +297,7 @@ static int cpuacct_stats_show(struct seq_file *sf, void *v)
for (stat = 0; stat < CPUACCT_STAT_NSTATS; stat++) {
seq_printf(sf, "%s %lld\n",
cpuacct_stat_desc[stat],
- cputime64_to_clock_t(val[stat]));
+ nsec_to_clock_t(val[stat]));
}
return 0;
diff --git a/kernel/sched/cpufreq_schedutil.c b/kernel/sched/cpufreq_schedutil.c
index 895c63f..938f49d 100644
--- a/kernel/sched/cpufreq_schedutil.c
+++ b/kernel/sched/cpufreq_schedutil.c
@@ -19,14 +19,22 @@
#include "sched.h"
#include "tune.h"
+/* Stub out fast switch routines present on mainline to reduce the backport
+ * overhead. */
+#define cpufreq_driver_fast_switch(x, y) 0
+#define cpufreq_enable_fast_switch(x)
+#define cpufreq_disable_fast_switch(x)
+#define LATENCY_MULTIPLIER (1000)
#define SUGOV_KTHREAD_PRIORITY 50
struct sugov_tunables {
struct gov_attr_set attr_set;
- unsigned int rate_limit_us;
+ unsigned int up_rate_limit_us;
+ unsigned int down_rate_limit_us;
unsigned int hispeed_load;
unsigned int hispeed_freq;
bool pl;
+ bool iowait_boost_enable;
};
struct sugov_policy {
@@ -37,7 +45,9 @@ struct sugov_policy {
raw_spinlock_t update_lock; /* For shared policies */
u64 last_freq_update_time;
- s64 freq_update_delay_ns;
+ s64 min_rate_limit_ns;
+ s64 up_rate_delay_ns;
+ s64 down_rate_delay_ns;
u64 last_ws;
u64 curr_cycles;
u64 last_cyc_update_time;
@@ -101,7 +111,27 @@ static bool sugov_should_update_freq(struct sugov_policy *sg_policy, u64 time)
}
delta_ns = time - sg_policy->last_freq_update_time;
- return delta_ns >= sg_policy->freq_update_delay_ns;
+
+ /* No need to recalculate next freq for min_rate_limit_us at least */
+ return delta_ns >= sg_policy->min_rate_limit_ns;
+}
+
+static bool sugov_up_down_rate_limit(struct sugov_policy *sg_policy, u64 time,
+ unsigned int next_freq)
+{
+ s64 delta_ns;
+
+ delta_ns = time - sg_policy->last_freq_update_time;
+
+ if (next_freq > sg_policy->next_freq &&
+ delta_ns < sg_policy->up_rate_delay_ns)
+ return true;
+
+ if (next_freq < sg_policy->next_freq &&
+ delta_ns < sg_policy->down_rate_delay_ns)
+ return true;
+
+ return false;
}
static void sugov_update_commit(struct sugov_policy *sg_policy, u64 time,
@@ -109,6 +139,12 @@ static void sugov_update_commit(struct sugov_policy *sg_policy, u64 time,
{
struct cpufreq_policy *policy = sg_policy->policy;
+ if (sugov_up_down_rate_limit(sg_policy, time, next_freq)) {
+ /* Don't cache a raw freq that didn't become next_freq */
+ sg_policy->cached_raw_freq = 0;
+ return;
+ }
+
if (sg_policy->next_freq == next_freq)
return;
@@ -184,6 +220,11 @@ static void sugov_get_util(unsigned long *util, unsigned long *max, int cpu)
static void sugov_set_iowait_boost(struct sugov_cpu *sg_cpu, u64 time,
unsigned int flags)
{
+ struct sugov_policy *sg_policy = sg_cpu->sg_policy;
+
+ if (!sg_policy->tunables->iowait_boost_enable)
+ return;
+
if (flags & SCHED_CPUFREQ_IOWAIT) {
sg_cpu->iowait_boost = sg_cpu->iowait_boost_max;
} else if (sg_cpu->iowait_boost) {
@@ -246,7 +287,7 @@ static void sugov_calc_avg_cap(struct sugov_policy *sg_policy, u64 curr_ws,
if (unlikely(!sysctl_sched_use_walt_cpu_util))
return;
- BUG_ON(curr_ws < last_ws);
+ WARN_ON(curr_ws < last_ws);
if (curr_ws <= last_ws)
return;
@@ -331,6 +372,8 @@ static void sugov_update_single(struct update_util_data *hook, u64 time,
raw_spin_lock(&sg_policy->update_lock);
if (flags & SCHED_CPUFREQ_RT_DL) {
+ /* clear cache when it's bypassed */
+ sg_policy->cached_raw_freq = 0;
next_f = policy->cpuinfo.max_freq;
} else {
sugov_get_util(&util, &max, sg_cpu->cpu);
@@ -358,18 +401,20 @@ static void sugov_update_single(struct update_util_data *hook, u64 time,
* Do not reduce the frequency if the CPU has not been idle
* recently, as the reduction is likely to be premature then.
*/
- if (busy && next_f < sg_policy->next_freq)
+ if (busy && next_f < sg_policy->next_freq) {
next_f = sg_policy->next_freq;
+ /* clear cache when it's bypassed */
+ sg_policy->cached_raw_freq = 0;
+ }
}
sugov_update_commit(sg_policy, time, next_f);
raw_spin_unlock(&sg_policy->update_lock);
}
-static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu)
+static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu, u64 time)
{
struct sugov_policy *sg_policy = sg_cpu->sg_policy;
struct cpufreq_policy *policy = sg_policy->policy;
- u64 last_freq_update_time = sg_policy->last_freq_update_time;
unsigned long util = 0, max = 1;
unsigned int j;
@@ -385,13 +430,16 @@ static unsigned int sugov_next_freq_shared(struct sugov_cpu *sg_cpu)
* enough, don't take the CPU into account as it probably is
* idle now (and clear iowait_boost for it).
*/
- delta_ns = last_freq_update_time - j_sg_cpu->last_update;
+ delta_ns = time - j_sg_cpu->last_update;
if (delta_ns > stale_ns) {
j_sg_cpu->iowait_boost = 0;
continue;
}
- if (j_sg_cpu->flags & SCHED_CPUFREQ_RT_DL)
+ if (j_sg_cpu->flags & SCHED_CPUFREQ_RT_DL) {
+ /* clear cache when it's bypassed */
+ sg_policy->cached_raw_freq = 0;
return policy->cpuinfo.max_freq;
+ }
j_util = j_sg_cpu->util;
j_max = j_sg_cpu->max;
@@ -447,10 +495,13 @@ static void sugov_update_shared(struct update_util_data *hook, u64 time,
sg_cpu->walt_load.pl, flags);
if (sugov_should_update_freq(sg_policy, time)) {
- if (flags & SCHED_CPUFREQ_RT_DL)
+ if (flags & SCHED_CPUFREQ_RT_DL) {
next_f = sg_policy->policy->cpuinfo.max_freq;
- else
- next_f = sugov_next_freq_shared(sg_cpu);
+ /* clear cache when it's bypassed */
+ sg_policy->cached_raw_freq = 0;
+ } else {
+ next_f = sugov_next_freq_shared(sg_cpu, time);
+ }
sugov_update_commit(sg_policy, time, next_f);
}
@@ -507,15 +558,32 @@ static inline struct sugov_tunables *to_sugov_tunables(struct gov_attr_set *attr
return container_of(attr_set, struct sugov_tunables, attr_set);
}
-static ssize_t rate_limit_us_show(struct gov_attr_set *attr_set, char *buf)
+static DEFINE_MUTEX(min_rate_lock);
+
+static void update_min_rate_limit_us(struct sugov_policy *sg_policy)
+{
+ mutex_lock(&min_rate_lock);
+ sg_policy->min_rate_limit_ns = min(sg_policy->up_rate_delay_ns,
+ sg_policy->down_rate_delay_ns);
+ mutex_unlock(&min_rate_lock);
+}
+
+static ssize_t up_rate_limit_us_show(struct gov_attr_set *attr_set, char *buf)
{
struct sugov_tunables *tunables = to_sugov_tunables(attr_set);
- return sprintf(buf, "%u\n", tunables->rate_limit_us);
+ return scnprintf(buf, PAGE_SIZE, "%u\n", tunables->up_rate_limit_us);
}
-static ssize_t rate_limit_us_store(struct gov_attr_set *attr_set, const char *buf,
- size_t count)
+static ssize_t down_rate_limit_us_show(struct gov_attr_set *attr_set, char *buf)
+{
+ struct sugov_tunables *tunables = to_sugov_tunables(attr_set);
+
+ return scnprintf(buf, PAGE_SIZE, "%u\n", tunables->down_rate_limit_us);
+}
+
+static ssize_t up_rate_limit_us_store(struct gov_attr_set *attr_set,
+ const char *buf, size_t count)
{
struct sugov_tunables *tunables = to_sugov_tunables(attr_set);
struct sugov_policy *sg_policy;
@@ -524,10 +592,32 @@ static ssize_t rate_limit_us_store(struct gov_attr_set *attr_set, const char *bu
if (kstrtouint(buf, 10, &rate_limit_us))
return -EINVAL;
- tunables->rate_limit_us = rate_limit_us;
+ tunables->up_rate_limit_us = rate_limit_us;
- list_for_each_entry(sg_policy, &attr_set->policy_list, tunables_hook)
- sg_policy->freq_update_delay_ns = rate_limit_us * NSEC_PER_USEC;
+ list_for_each_entry(sg_policy, &attr_set->policy_list, tunables_hook) {
+ sg_policy->up_rate_delay_ns = rate_limit_us * NSEC_PER_USEC;
+ update_min_rate_limit_us(sg_policy);
+ }
+
+ return count;
+}
+
+static ssize_t down_rate_limit_us_store(struct gov_attr_set *attr_set,
+ const char *buf, size_t count)
+{
+ struct sugov_tunables *tunables = to_sugov_tunables(attr_set);
+ struct sugov_policy *sg_policy;
+ unsigned int rate_limit_us;
+
+ if (kstrtouint(buf, 10, &rate_limit_us))
+ return -EINVAL;
+
+ tunables->down_rate_limit_us = rate_limit_us;
+
+ list_for_each_entry(sg_policy, &attr_set->policy_list, tunables_hook) {
+ sg_policy->down_rate_delay_ns = rate_limit_us * NSEC_PER_USEC;
+ update_min_rate_limit_us(sg_policy);
+ }
return count;
}
@@ -598,20 +688,45 @@ static ssize_t pl_store(struct gov_attr_set *attr_set, const char *buf,
if (kstrtobool(buf, &tunables->pl))
return -EINVAL;
+ return count;
+}
+
+static ssize_t iowait_boost_enable_show(struct gov_attr_set *attr_set,
+ char *buf)
+{
+ struct sugov_tunables *tunables = to_sugov_tunables(attr_set);
+
+ return sprintf(buf, "%u\n", tunables->iowait_boost_enable);
+}
+
+static ssize_t iowait_boost_enable_store(struct gov_attr_set *attr_set,
+ const char *buf, size_t count)
+{
+ struct sugov_tunables *tunables = to_sugov_tunables(attr_set);
+ bool enable;
+
+ if (kstrtobool(buf, &enable))
+ return -EINVAL;
+
+ tunables->iowait_boost_enable = enable;
return count;
}
-static struct governor_attr rate_limit_us = __ATTR_RW(rate_limit_us);
+static struct governor_attr up_rate_limit_us = __ATTR_RW(up_rate_limit_us);
+static struct governor_attr down_rate_limit_us = __ATTR_RW(down_rate_limit_us);
static struct governor_attr hispeed_load = __ATTR_RW(hispeed_load);
static struct governor_attr hispeed_freq = __ATTR_RW(hispeed_freq);
static struct governor_attr pl = __ATTR_RW(pl);
+static struct governor_attr iowait_boost_enable = __ATTR_RW(iowait_boost_enable);
static struct attribute *sugov_attributes[] = {
- &rate_limit_us.attr,
+ &up_rate_limit_us.attr,
+ &down_rate_limit_us.attr,
&hispeed_load.attr,
&hispeed_freq.attr,
&pl.attr,
+ &iowait_boost_enable.attr,
NULL
};
@@ -726,7 +841,8 @@ static void sugov_tunables_save(struct cpufreq_policy *policy,
cached->pl = tunables->pl;
cached->hispeed_load = tunables->hispeed_load;
cached->hispeed_freq = tunables->hispeed_freq;
- cached->rate_limit_us = tunables->rate_limit_us;
+ cached->up_rate_limit_us = tunables->up_rate_limit_us;
+ cached->down_rate_limit_us = tunables->down_rate_limit_us;
}
static void sugov_tunables_free(struct sugov_tunables *tunables)
@@ -749,9 +865,11 @@ static void sugov_tunables_restore(struct cpufreq_policy *policy)
tunables->pl = cached->pl;
tunables->hispeed_load = cached->hispeed_load;
tunables->hispeed_freq = cached->hispeed_freq;
- tunables->rate_limit_us = cached->rate_limit_us;
- sg_policy->freq_update_delay_ns =
- tunables->rate_limit_us * NSEC_PER_USEC;
+ tunables->up_rate_limit_us = cached->up_rate_limit_us;
+ tunables->down_rate_limit_us = cached->down_rate_limit_us;
+ sg_policy->up_rate_delay_ns = cached->up_rate_limit_us;
+ sg_policy->down_rate_delay_ns = cached->down_rate_limit_us;
+ update_min_rate_limit_us(sg_policy);
}
static int sugov_init(struct cpufreq_policy *policy)
@@ -797,12 +915,17 @@ static int sugov_init(struct cpufreq_policy *policy)
goto stop_kthread;
}
- tunables->rate_limit_us = LATENCY_MULTIPLIER;
+ tunables->up_rate_limit_us = LATENCY_MULTIPLIER;
+ tunables->down_rate_limit_us = LATENCY_MULTIPLIER;
tunables->hispeed_load = DEFAULT_HISPEED_LOAD;
tunables->hispeed_freq = 0;
lat = policy->cpuinfo.transition_latency / NSEC_PER_USEC;
- if (lat)
- tunables->rate_limit_us *= lat;
+ if (lat) {
+ tunables->up_rate_limit_us *= lat;
+ tunables->down_rate_limit_us *= lat;
+ }
+
+ tunables->iowait_boost_enable = false;
policy->governor_data = sg_policy;
sg_policy->tunables = tunables;
@@ -866,7 +989,11 @@ static int sugov_start(struct cpufreq_policy *policy)
struct sugov_policy *sg_policy = policy->governor_data;
unsigned int cpu;
- sg_policy->freq_update_delay_ns = sg_policy->tunables->rate_limit_us * NSEC_PER_USEC;
+ sg_policy->up_rate_delay_ns =
+ sg_policy->tunables->up_rate_limit_us * NSEC_PER_USEC;
+ sg_policy->down_rate_delay_ns =
+ sg_policy->tunables->down_rate_limit_us * NSEC_PER_USEC;
+ update_min_rate_limit_us(sg_policy);
sg_policy->last_freq_update_time = 0;
sg_policy->next_freq = UINT_MAX;
sg_policy->work_in_progress = false;
diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c
index 50f2ea89..5d99d8b 100644
--- a/kernel/sched/cputime.c
+++ b/kernel/sched/cputime.c
@@ -38,6 +38,18 @@ void disable_sched_clock_irqtime(void)
sched_clock_irqtime = 0;
}
+static void irqtime_account_delta(struct irqtime *irqtime, u64 delta,
+ enum cpu_usage_stat idx)
+{
+ u64 *cpustat = kcpustat_this_cpu->cpustat;
+
+ u64_stats_update_begin(&irqtime->sync);
+ cpustat[idx] += delta;
+ irqtime->total += delta;
+ irqtime->tick_delta += delta;
+ u64_stats_update_end(&irqtime->sync);
+}
+
/*
* Called before incrementing preempt_count on {soft,}irq_enter
* and before decrementing preempt_count on {soft,}irq_exit.
@@ -58,7 +70,6 @@ void irqtime_account_irq(struct task_struct *curr)
delta = wallclock - irqtime->irq_start_time;
irqtime->irq_start_time += delta;
- u64_stats_update_begin(&irqtime->sync);
/*
* We do not account for softirq time from ksoftirqd here.
* We want to continue accounting softirq time to ksoftirqd thread
@@ -66,9 +77,9 @@ void irqtime_account_irq(struct task_struct *curr)
* that do not consume any time, but still wants to run.
*/
if (hardirq_count())
- irqtime->hardirq_time += delta;
+ irqtime_account_delta(irqtime, delta, CPUTIME_IRQ);
else if (in_serving_softirq() && curr != this_cpu_ksoftirqd())
- irqtime->softirq_time += delta;
+ irqtime_account_delta(irqtime, delta, CPUTIME_SOFTIRQ);
else
account = false;
@@ -81,40 +92,23 @@ void irqtime_account_irq(struct task_struct *curr)
}
EXPORT_SYMBOL_GPL(irqtime_account_irq);
-static cputime_t irqtime_account_update(u64 irqtime, int idx, cputime_t maxtime)
+static cputime_t irqtime_tick_accounted(cputime_t maxtime)
{
- u64 *cpustat = kcpustat_this_cpu->cpustat;
- cputime_t irq_cputime;
+ struct irqtime *irqtime = this_cpu_ptr(&cpu_irqtime);
+ cputime_t delta;
- irq_cputime = nsecs_to_cputime64(irqtime) - cpustat[idx];
- irq_cputime = min(irq_cputime, maxtime);
- cpustat[idx] += irq_cputime;
+ delta = nsecs_to_cputime(irqtime->tick_delta);
+ delta = min(delta, maxtime);
+ irqtime->tick_delta -= cputime_to_nsecs(delta);
- return irq_cputime;
-}
-
-static cputime_t irqtime_account_hi_update(cputime_t maxtime)
-{
- return irqtime_account_update(__this_cpu_read(cpu_irqtime.hardirq_time),
- CPUTIME_IRQ, maxtime);
-}
-
-static cputime_t irqtime_account_si_update(cputime_t maxtime)
-{
- return irqtime_account_update(__this_cpu_read(cpu_irqtime.softirq_time),
- CPUTIME_SOFTIRQ, maxtime);
+ return delta;
}
#else /* CONFIG_IRQ_TIME_ACCOUNTING */
#define sched_clock_irqtime (0)
-static cputime_t irqtime_account_hi_update(cputime_t dummy)
-{
- return 0;
-}
-
-static cputime_t irqtime_account_si_update(cputime_t dummy)
+static cputime_t irqtime_tick_accounted(cputime_t dummy)
{
return 0;
}
@@ -154,7 +148,7 @@ void account_user_time(struct task_struct *p, cputime_t cputime,
index = (task_nice(p) > 0) ? CPUTIME_NICE : CPUTIME_USER;
/* Add user time to cpustat. */
- task_group_account_field(p, index, (__force u64) cputime);
+ task_group_account_field(p, index, cputime_to_nsecs(cputime));
/* Account for user time used */
acct_account_cputime(p);
@@ -184,11 +178,11 @@ static void account_guest_time(struct task_struct *p, cputime_t cputime,
/* Add guest time to cpustat. */
if (task_nice(p) > 0) {
- cpustat[CPUTIME_NICE] += (__force u64) cputime;
- cpustat[CPUTIME_GUEST_NICE] += (__force u64) cputime;
+ cpustat[CPUTIME_NICE] += cputime_to_nsecs(cputime);
+ cpustat[CPUTIME_GUEST_NICE] += cputime_to_nsecs(cputime);
} else {
- cpustat[CPUTIME_USER] += (__force u64) cputime;
- cpustat[CPUTIME_GUEST] += (__force u64) cputime;
+ cpustat[CPUTIME_USER] += cputime_to_nsecs(cputime);
+ cpustat[CPUTIME_GUEST] += cputime_to_nsecs(cputime);
}
}
@@ -209,7 +203,7 @@ void __account_system_time(struct task_struct *p, cputime_t cputime,
account_group_system_time(p, cputime);
/* Add system time to cpustat. */
- task_group_account_field(p, index, (__force u64) cputime);
+ task_group_account_field(p, index, cputime_to_nsecs(cputime));
/* Account for system time used */
acct_account_cputime(p);
@@ -254,7 +248,7 @@ void account_steal_time(cputime_t cputime)
{
u64 *cpustat = kcpustat_this_cpu->cpustat;
- cpustat[CPUTIME_STEAL] += (__force u64) cputime;
+ cpustat[CPUTIME_STEAL] += cputime_to_nsecs(cputime);
}
/*
@@ -267,9 +261,9 @@ void account_idle_time(cputime_t cputime)
struct rq *rq = this_rq();
if (atomic_read(&rq->nr_iowait) > 0)
- cpustat[CPUTIME_IOWAIT] += (__force u64) cputime;
+ cpustat[CPUTIME_IOWAIT] += cputime_to_nsecs(cputime);
else
- cpustat[CPUTIME_IDLE] += (__force u64) cputime;
+ cpustat[CPUTIME_IDLE] += cputime_to_nsecs(cputime);
}
/*
@@ -310,10 +304,7 @@ static inline cputime_t account_other_time(cputime_t max)
accounted = steal_account_process_time(max);
if (accounted < max)
- accounted += irqtime_account_hi_update(max - accounted);
-
- if (accounted < max)
- accounted += irqtime_account_si_update(max - accounted);
+ accounted += irqtime_tick_accounted(max - accounted);
return accounted;
}
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index d7c7dd6..ba9dce4 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -749,9 +749,9 @@ static enum hrtimer_restart dl_task_timer(struct hrtimer *timer)
* Nothing relies on rq->lock after this, so its safe to drop
* rq->lock.
*/
- lockdep_unpin_lock(&rq->lock, rf.cookie);
+ rq_unpin_lock(rq, &rf);
push_dl_task(rq);
- lockdep_repin_lock(&rq->lock, rf.cookie);
+ rq_repin_lock(rq, &rf);
}
#endif
@@ -1248,7 +1248,7 @@ static struct sched_dl_entity *pick_next_dl_entity(struct rq *rq,
}
struct task_struct *
-pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie)
+pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
{
struct sched_dl_entity *dl_se;
struct task_struct *p;
@@ -1263,9 +1263,9 @@ pick_next_task_dl(struct rq *rq, struct task_struct *prev, struct pin_cookie coo
* disabled avoiding further scheduler activity on it and we're
* being very careful to re-start the picking loop.
*/
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_unpin_lock(rq, rf);
pull_dl_task(rq);
- lockdep_repin_lock(&rq->lock, cookie);
+ rq_repin_lock(rq, rf);
/*
* pull_rt_task() can drop (and re-acquire) rq->lock; this
* means a stop task can slip in, in which case we need to
diff --git a/kernel/sched/debug.c b/kernel/sched/debug.c
index 16065b2..3d53ab7 100644
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -1017,7 +1017,6 @@ void proc_sched_show_task(struct task_struct *p, struct seq_file *m)
P_SCHEDSTAT(se.statistics.nr_wakeups_sis_count);
/* select_energy_cpu_brute() */
P_SCHEDSTAT(se.statistics.nr_wakeups_secb_attempts);
- P_SCHEDSTAT(se.statistics.nr_wakeups_secb_sync);
P_SCHEDSTAT(se.statistics.nr_wakeups_secb_idle_bt);
P_SCHEDSTAT(se.statistics.nr_wakeups_secb_insuff_cap);
P_SCHEDSTAT(se.statistics.nr_wakeups_secb_no_nrg_sav);
diff --git a/kernel/sched/energy.c b/kernel/sched/energy.c
index 77d8361..b75b13a 100644
--- a/kernel/sched/energy.c
+++ b/kernel/sched/energy.c
@@ -269,6 +269,22 @@ static int sched_energy_probe(struct platform_device *pdev)
cpu_max_cap);
arch_update_cpu_capacity(cpu);
+
+ cpu_rq(cpu)->cpu_capacity_orig = cpu_max_cap;
+ }
+
+ for_each_possible_cpu(cpu) {
+ struct rq *rq = cpu_rq(cpu);
+ int max_cpu = READ_ONCE(rq->rd->max_cap_orig_cpu);
+ int min_cpu = READ_ONCE(rq->rd->min_cap_orig_cpu);
+
+ if ((max_cpu < 0) || rq->cpu_capacity_orig >
+ cpu_rq(max_cpu)->cpu_capacity_orig)
+ WRITE_ONCE(rq->rd->max_cap_orig_cpu, cpu);
+
+ if ((min_cpu < 0) || rq->cpu_capacity_orig <
+ cpu_rq(min_cpu)->cpu_capacity_orig)
+ WRITE_ONCE(rq->rd->min_cap_orig_cpu, cpu);
}
kfree(max_frequencies);
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index fff4170..b2be181 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3814,7 +3814,7 @@ dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
* put back on, and if we advance min_vruntime, we'll be placed back
* further than we started -- ie. we'll be penalized.
*/
- if ((flags & (DEQUEUE_SAVE | DEQUEUE_MOVE)) == DEQUEUE_SAVE)
+ if ((flags & (DEQUEUE_SAVE | DEQUEUE_MOVE)) != DEQUEUE_SAVE)
update_min_vruntime(cfs_rq);
}
@@ -4291,9 +4291,13 @@ static void throttle_cfs_rq(struct cfs_rq *cfs_rq)
/*
* Add to the _head_ of the list, so that an already-started
- * distribute_cfs_runtime will not see us
+ * distribute_cfs_runtime will not see us. If disribute_cfs_runtime is
+ * not running add to the tail so that later runqueues don't get starved.
*/
- list_add_rcu(&cfs_rq->throttled_list, &cfs_b->throttled_cfs_rq);
+ if (cfs_b->distribute_running)
+ list_add_rcu(&cfs_rq->throttled_list, &cfs_b->throttled_cfs_rq);
+ else
+ list_add_tail_rcu(&cfs_rq->throttled_list, &cfs_b->throttled_cfs_rq);
/*
* If we're the first throttled task, make sure the bandwidth
@@ -4440,14 +4444,16 @@ static int do_sched_cfs_period_timer(struct cfs_bandwidth *cfs_b, int overrun)
* in us over-using our runtime if it is all used during this loop, but
* only by limited amounts in that extreme case.
*/
- while (throttled && cfs_b->runtime > 0) {
+ while (throttled && cfs_b->runtime > 0 && !cfs_b->distribute_running) {
runtime = cfs_b->runtime;
+ cfs_b->distribute_running = 1;
raw_spin_unlock(&cfs_b->lock);
/* we can't nest cfs_b->lock while distributing bandwidth */
runtime = distribute_cfs_runtime(cfs_b, runtime,
runtime_expires);
raw_spin_lock(&cfs_b->lock);
+ cfs_b->distribute_running = 0;
throttled = !list_empty(&cfs_b->throttled_cfs_rq);
cfs_b->runtime -= min(runtime, cfs_b->runtime);
@@ -4558,6 +4564,11 @@ static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
/* confirm we're still not at a refresh boundary */
raw_spin_lock(&cfs_b->lock);
+ if (cfs_b->distribute_running) {
+ raw_spin_unlock(&cfs_b->lock);
+ return;
+ }
+
if (runtime_refresh_within(cfs_b, min_bandwidth_expiration)) {
raw_spin_unlock(&cfs_b->lock);
return;
@@ -4567,6 +4578,9 @@ static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
runtime = cfs_b->runtime;
expires = cfs_b->runtime_expires;
+ if (runtime)
+ cfs_b->distribute_running = 1;
+
raw_spin_unlock(&cfs_b->lock);
if (!runtime)
@@ -4577,6 +4591,7 @@ static void do_sched_cfs_slack_timer(struct cfs_bandwidth *cfs_b)
raw_spin_lock(&cfs_b->lock);
if (expires == cfs_b->runtime_expires)
cfs_b->runtime -= min(runtime, cfs_b->runtime);
+ cfs_b->distribute_running = 0;
raw_spin_unlock(&cfs_b->lock);
}
@@ -4685,6 +4700,7 @@ void init_cfs_bandwidth(struct cfs_bandwidth *cfs_b)
cfs_b->period_timer.function = sched_cfs_period_timer;
hrtimer_init(&cfs_b->slack_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
cfs_b->slack_timer.function = sched_cfs_slack_timer;
+ cfs_b->distribute_running = 0;
}
static void init_cfs_rq_runtime(struct cfs_rq *cfs_rq)
@@ -7305,6 +7321,29 @@ bias_to_waker_cpu(struct task_struct *p, int cpu, struct cpumask *rtg_target)
task_fits_max(p, cpu);
}
+static inline bool
+task_is_boosted(struct task_struct *p) {
+#ifdef CONFIG_CGROUP_SCHEDTUNE
+ return schedtune_task_boost(p) > 0;
+#else
+ return get_sysctl_sched_cfs_boost() > 0;
+#endif
+}
+
+/*
+ * Check whether cpu is in the fastest set of cpu's that p should run on.
+ * If p is boosted, prefer that p runs on a faster cpu; otherwise, allow p
+ * to run on any cpu.
+ */
+static inline bool
+cpu_is_in_target_set(struct task_struct *p, int cpu)
+{
+ int first_cpu = start_cpu(task_is_boosted(p));
+ int next_usable_cpu = cpumask_next(first_cpu - 1, tsk_cpus_allowed(p));
+ return cpu >= next_usable_cpu || next_usable_cpu >= nr_cpu_ids;
+}
+
+
#define SCHED_SELECT_PREV_CPU_NSEC 2000000
#define SCHED_FORCE_CPU_SELECTION_NSEC 20000000
@@ -7318,14 +7357,17 @@ bias_to_prev_cpu(struct task_struct *p, struct cpumask *rtg_target)
u64 ms = sched_clock();
#endif
+ if (!cpu_is_in_target_set(p, prev_cpu)) {
+ return false;
+ }
+
if (cpu_isolated(prev_cpu) || !idle_cpu(prev_cpu))
return false;
if (!ms)
return false;
- if (ms - p->last_cpu_selected_ts >= SCHED_SELECT_PREV_CPU_NSEC) {
- p->last_cpu_selected_ts = ms;
+ if (ms - p->last_cpu_deselected_ts >= SCHED_SELECT_PREV_CPU_NSEC) {
return false;
}
@@ -7372,7 +7414,8 @@ enum fastpaths {
PREV_CPU_BIAS,
};
-static int select_energy_cpu_brute(struct task_struct *p, int prev_cpu, int sync)
+static int select_energy_cpu_brute(struct task_struct *p, int prev_cpu,
+ int sync_boost)
{
bool boosted, prefer_idle;
struct sched_domain *sd;
@@ -7390,11 +7433,11 @@ static int select_energy_cpu_brute(struct task_struct *p, int prev_cpu, int sync
schedstat_inc(p->se.statistics.nr_wakeups_secb_attempts);
schedstat_inc(this_rq()->eas_stats.secb_attempts);
+ rcu_read_lock();
+ boosted = task_is_boosted(p);
#ifdef CONFIG_CGROUP_SCHEDTUNE
- boosted = schedtune_task_boost(p) > 0;
prefer_idle = schedtune_prefer_idle(p) > 0;
#else
- boosted = get_sysctl_sched_cfs_boost() > 0;
prefer_idle = 0;
#endif
@@ -7410,29 +7453,12 @@ static int select_energy_cpu_brute(struct task_struct *p, int prev_cpu, int sync
false;
fbt_env.avoid_prev_cpu = false;
- if (prefer_idle || fbt_env.need_idle)
- sync = 0;
-
- if (sysctl_sched_sync_hint_enable && sync) {
- int cpu = smp_processor_id();
-
- if (bias_to_waker_cpu(p, cpu, rtg_target)) {
- schedstat_inc(p->se.statistics.nr_wakeups_secb_sync);
- schedstat_inc(this_rq()->eas_stats.secb_sync);
- target_cpu = cpu;
- fastpath = SYNC_WAKEUP;
- goto out;
- }
- }
-
if (bias_to_prev_cpu(p, rtg_target)) {
target_cpu = prev_cpu;
fastpath = PREV_CPU_BIAS;
- goto out;
+ goto unlock;
}
- rcu_read_lock();
-
sd = rcu_dereference(per_cpu(sd_ea, prev_cpu));
if (!sd) {
target_cpu = prev_cpu;
@@ -7442,8 +7468,8 @@ static int select_energy_cpu_brute(struct task_struct *p, int prev_cpu, int sync
sync_entity_load_avg(&p->se);
/* Find a cpu with sufficient capacity */
- next_cpu = find_best_target(p, &backup_cpu, boosted, prefer_idle,
- &fbt_env);
+ next_cpu = find_best_target(p, &backup_cpu, boosted || sync_boost,
+ prefer_idle, &fbt_env);
if (next_cpu == -1) {
target_cpu = prev_cpu;
goto unlock;
@@ -7517,8 +7543,7 @@ static int select_energy_cpu_brute(struct task_struct *p, int prev_cpu, int sync
unlock:
rcu_read_unlock();
-out:
- trace_sched_task_util(p, next_cpu, backup_cpu, target_cpu, sync,
+ trace_sched_task_util(p, next_cpu, backup_cpu, target_cpu,
fbt_env.need_idle, fastpath,
fbt_env.placement_boost, rtg_target ?
cpumask_first(rtg_target) : -1, start_t);
@@ -7547,13 +7572,34 @@ select_task_rq_fair(struct task_struct *p, int prev_cpu, int sd_flag, int wake_f
int sync = wake_flags & WF_SYNC;
if (sd_flag & SD_BALANCE_WAKE) {
+ int _wake_cap = wake_cap(p, cpu, prev_cpu);
+
+ if (cpumask_test_cpu(cpu, tsk_cpus_allowed(p))) {
+ bool about_to_idle = (cpu_rq(cpu)->nr_running < 2);
+
+ if (sysctl_sched_sync_hint_enable && sync &&
+ !_wake_cap && about_to_idle &&
+ cpu_is_in_target_set(p, cpu))
+ return cpu;
+ }
+
record_wakee(p);
- want_affine = (!wake_wide(p) && !wake_cap(p, cpu, prev_cpu) &&
+ want_affine = (!wake_wide(p) && !_wake_cap &&
cpumask_test_cpu(cpu, tsk_cpus_allowed(p)));
}
- if (energy_aware())
- return select_energy_cpu_brute(p, prev_cpu, sync);
+ if (energy_aware() && !(cpu_rq(prev_cpu)->rd->overutilized)) {
+ /*
+ * If the sync flag is set but ignored, prefer to
+ * select cpu in the same cluster as current. So
+ * if current is a big cpu and sync is set, indicate
+ * that the selection algorithm for a boosted task
+ * should be used.
+ */
+ bool sync_boost = sync && cpu >= start_cpu(true);
+
+ return select_energy_cpu_brute(p, prev_cpu, sync_boost);
+ }
rcu_read_lock();
for_each_domain(cpu, tmp) {
@@ -7821,7 +7867,7 @@ static void check_preempt_wakeup(struct rq *rq, struct task_struct *p, int wake_
}
static struct task_struct *
-pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie)
+pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
{
struct cfs_rq *cfs_rq = &rq->cfs;
struct sched_entity *se;
@@ -7939,9 +7985,9 @@ pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct pin_cookie c
* further scheduler activity on it and we're being very careful to
* re-start the picking loop.
*/
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_unpin_lock(rq, rf);
new_tasks = idle_balance(rq);
- lockdep_repin_lock(&rq->lock, cookie);
+ rq_repin_lock(rq, rf);
/*
* Because idle_balance() releases (and re-acquires) rq->lock, it is
* possible for any higher priority task to appear. In that case we
@@ -11095,7 +11141,8 @@ static inline bool vruntime_normalized(struct task_struct *p)
* - A task which has been woken up by try_to_wake_up() and
* waiting for actually being woken up by sched_ttwu_pending().
*/
- if (!se->sum_exec_runtime || p->state == TASK_WAKING)
+ if (!se->sum_exec_runtime ||
+ (p->state == TASK_WAKING && p->sched_remote_wakeup))
return true;
return false;
diff --git a/kernel/sched/features.h b/kernel/sched/features.h
index ef52be4..7855434 100644
--- a/kernel/sched/features.h
+++ b/kernel/sched/features.h
@@ -49,7 +49,7 @@ SCHED_FEAT(NONTASK_CAPACITY, true)
* Queue remote wakeups on the target CPU and process them
* using the scheduler IPI. Reduces rq->lock contention/bounces.
*/
-SCHED_FEAT(TTWU_QUEUE, false)
+SCHED_FEAT(TTWU_QUEUE, true)
/*
* When doing wakeups, attempt to limit superfluous scans of the LLC domain.
@@ -103,4 +103,4 @@ SCHED_FEAT(FBT_STRICT_ORDER, false)
* ON: schedtune.prefer_idle is replaced with need_idle
* OFF: schedtune.prefer_idle is honored as is.
*/
-SCHED_FEAT(EAS_USE_NEED_IDLE, true)
+SCHED_FEAT(EAS_USE_NEED_IDLE, false)
diff --git a/kernel/sched/idle_task.c b/kernel/sched/idle_task.c
index 5405d3f..0c00172 100644
--- a/kernel/sched/idle_task.c
+++ b/kernel/sched/idle_task.c
@@ -24,7 +24,7 @@ static void check_preempt_curr_idle(struct rq *rq, struct task_struct *p, int fl
}
static struct task_struct *
-pick_next_task_idle(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie)
+pick_next_task_idle(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
{
put_prev_task(rq, prev);
update_idle_core(rq);
diff --git a/kernel/sched/loadavg.c b/kernel/sched/loadavg.c
index ec91fcc..5f3812c 100644
--- a/kernel/sched/loadavg.c
+++ b/kernel/sched/loadavg.c
@@ -7,6 +7,7 @@
*/
#include <linux/export.h>
+#include <linux/sched/loadavg.h>
#include "sched.h"
@@ -93,19 +94,73 @@ long calc_load_fold_active(struct rq *this_rq, long adjust)
return delta;
}
-/*
- * a1 = a0 * e + a * (1 - e)
+/**
+ * fixed_power_int - compute: x^n, in O(log n) time
+ *
+ * @x: base of the power
+ * @frac_bits: fractional bits of @x
+ * @n: power to raise @x to.
+ *
+ * By exploiting the relation between the definition of the natural power
+ * function: x^n := x*x*...*x (x multiplied by itself for n times), and
+ * the binary encoding of numbers used by computers: n := \Sum n_i * 2^i,
+ * (where: n_i \elem {0, 1}, the binary vector representing n),
+ * we find: x^n := x^(\Sum n_i * 2^i) := \Prod x^(n_i * 2^i), which is
+ * of course trivially computable in O(log_2 n), the length of our binary
+ * vector.
*/
static unsigned long
-calc_load(unsigned long load, unsigned long exp, unsigned long active)
+fixed_power_int(unsigned long x, unsigned int frac_bits, unsigned int n)
{
- unsigned long newload;
+ unsigned long result = 1UL << frac_bits;
- newload = load * exp + active * (FIXED_1 - exp);
- if (active >= load)
- newload += FIXED_1-1;
+ if (n) {
+ for (;;) {
+ if (n & 1) {
+ result *= x;
+ result += 1UL << (frac_bits - 1);
+ result >>= frac_bits;
+ }
+ n >>= 1;
+ if (!n)
+ break;
+ x *= x;
+ x += 1UL << (frac_bits - 1);
+ x >>= frac_bits;
+ }
+ }
- return newload / FIXED_1;
+ return result;
+}
+
+/*
+ * a1 = a0 * e + a * (1 - e)
+ *
+ * a2 = a1 * e + a * (1 - e)
+ * = (a0 * e + a * (1 - e)) * e + a * (1 - e)
+ * = a0 * e^2 + a * (1 - e) * (1 + e)
+ *
+ * a3 = a2 * e + a * (1 - e)
+ * = (a0 * e^2 + a * (1 - e) * (1 + e)) * e + a * (1 - e)
+ * = a0 * e^3 + a * (1 - e) * (1 + e + e^2)
+ *
+ * ...
+ *
+ * an = a0 * e^n + a * (1 - e) * (1 + e + ... + e^n-1) [1]
+ * = a0 * e^n + a * (1 - e) * (1 - e^n)/(1 - e)
+ * = a0 * e^n + a * (1 - e^n)
+ *
+ * [1] application of the geometric series:
+ *
+ * n 1 - x^(n+1)
+ * S_n := \Sum x^i = -------------
+ * i=0 1 - x
+ */
+unsigned long
+calc_load_n(unsigned long load, unsigned long exp,
+ unsigned long active, unsigned int n)
+{
+ return calc_load(load, fixed_power_int(exp, FSHIFT, n), active);
}
#ifdef CONFIG_NO_HZ_COMMON
@@ -227,75 +282,6 @@ static long calc_load_fold_idle(void)
return delta;
}
-/**
- * fixed_power_int - compute: x^n, in O(log n) time
- *
- * @x: base of the power
- * @frac_bits: fractional bits of @x
- * @n: power to raise @x to.
- *
- * By exploiting the relation between the definition of the natural power
- * function: x^n := x*x*...*x (x multiplied by itself for n times), and
- * the binary encoding of numbers used by computers: n := \Sum n_i * 2^i,
- * (where: n_i \elem {0, 1}, the binary vector representing n),
- * we find: x^n := x^(\Sum n_i * 2^i) := \Prod x^(n_i * 2^i), which is
- * of course trivially computable in O(log_2 n), the length of our binary
- * vector.
- */
-static unsigned long
-fixed_power_int(unsigned long x, unsigned int frac_bits, unsigned int n)
-{
- unsigned long result = 1UL << frac_bits;
-
- if (n) {
- for (;;) {
- if (n & 1) {
- result *= x;
- result += 1UL << (frac_bits - 1);
- result >>= frac_bits;
- }
- n >>= 1;
- if (!n)
- break;
- x *= x;
- x += 1UL << (frac_bits - 1);
- x >>= frac_bits;
- }
- }
-
- return result;
-}
-
-/*
- * a1 = a0 * e + a * (1 - e)
- *
- * a2 = a1 * e + a * (1 - e)
- * = (a0 * e + a * (1 - e)) * e + a * (1 - e)
- * = a0 * e^2 + a * (1 - e) * (1 + e)
- *
- * a3 = a2 * e + a * (1 - e)
- * = (a0 * e^2 + a * (1 - e) * (1 + e)) * e + a * (1 - e)
- * = a0 * e^3 + a * (1 - e) * (1 + e + e^2)
- *
- * ...
- *
- * an = a0 * e^n + a * (1 - e) * (1 + e + ... + e^n-1) [1]
- * = a0 * e^n + a * (1 - e) * (1 - e^n)/(1 - e)
- * = a0 * e^n + a * (1 - e^n)
- *
- * [1] application of the geometric series:
- *
- * n 1 - x^(n+1)
- * S_n := \Sum x^i = -------------
- * i=0 1 - x
- */
-static unsigned long
-calc_load_n(unsigned long load, unsigned long exp,
- unsigned long active, unsigned int n)
-{
- return calc_load(load, fixed_power_int(exp, FSHIFT, n), active);
-}
-
/*
* NO_HZ can leave us missing all per-cpu ticks calling
* calc_load_account_active(), but since an idle CPU folds its delta into
diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
new file mode 100644
index 0000000..2d8a1f2
--- /dev/null
+++ b/kernel/sched/psi.c
@@ -0,0 +1,759 @@
+/*
+ * Pressure stall information for CPU, memory and IO
+ *
+ * Copyright (c) 2018 Facebook, Inc.
+ * Author: Johannes Weiner <hannes@cmpxchg.org>
+ *
+ * When CPU, memory and IO are contended, tasks experience delays that
+ * reduce throughput and introduce latencies into the workload. Memory
+ * and IO contention, in addition, can cause a full loss of forward
+ * progress in which the CPU goes idle.
+ *
+ * This code aggregates individual task delays into resource pressure
+ * metrics that indicate problems with both workload health and
+ * resource utilization.
+ *
+ * Model
+ *
+ * The time in which a task can execute on a CPU is our baseline for
+ * productivity. Pressure expresses the amount of time in which this
+ * potential cannot be realized due to resource contention.
+ *
+ * This concept of productivity has two components: the workload and
+ * the CPU. To measure the impact of pressure on both, we define two
+ * contention states for a resource: SOME and FULL.
+ *
+ * In the SOME state of a given resource, one or more tasks are
+ * delayed on that resource. This affects the workload's ability to
+ * perform work, but the CPU may still be executing other tasks.
+ *
+ * In the FULL state of a given resource, all non-idle tasks are
+ * delayed on that resource such that nobody is advancing and the CPU
+ * goes idle. This leaves both workload and CPU unproductive.
+ *
+ * (Naturally, the FULL state doesn't exist for the CPU resource.)
+ *
+ * SOME = nr_delayed_tasks != 0
+ * FULL = nr_delayed_tasks != 0 && nr_running_tasks == 0
+ *
+ * The percentage of wallclock time spent in those compound stall
+ * states gives pressure numbers between 0 and 100 for each resource,
+ * where the SOME percentage indicates workload slowdowns and the FULL
+ * percentage indicates reduced CPU utilization:
+ *
+ * %SOME = time(SOME) / period
+ * %FULL = time(FULL) / period
+ *
+ * Multiple CPUs
+ *
+ * The more tasks and available CPUs there are, the more work can be
+ * performed concurrently. This means that the potential that can go
+ * unrealized due to resource contention *also* scales with non-idle
+ * tasks and CPUs.
+ *
+ * Consider a scenario where 257 number crunching tasks are trying to
+ * run concurrently on 256 CPUs. If we simply aggregated the task
+ * states, we would have to conclude a CPU SOME pressure number of
+ * 100%, since *somebody* is waiting on a runqueue at all
+ * times. However, that is clearly not the amount of contention the
+ * workload is experiencing: only one out of 256 possible exceution
+ * threads will be contended at any given time, or about 0.4%.
+ *
+ * Conversely, consider a scenario of 4 tasks and 4 CPUs where at any
+ * given time *one* of the tasks is delayed due to a lack of memory.
+ * Again, looking purely at the task state would yield a memory FULL
+ * pressure number of 0%, since *somebody* is always making forward
+ * progress. But again this wouldn't capture the amount of execution
+ * potential lost, which is 1 out of 4 CPUs, or 25%.
+ *
+ * To calculate wasted potential (pressure) with multiple processors,
+ * we have to base our calculation on the number of non-idle tasks in
+ * conjunction with the number of available CPUs, which is the number
+ * of potential execution threads. SOME becomes then the proportion of
+ * delayed tasks to possibe threads, and FULL is the share of possible
+ * threads that are unproductive due to delays:
+ *
+ * threads = min(nr_nonidle_tasks, nr_cpus)
+ * SOME = min(nr_delayed_tasks / threads, 1)
+ * FULL = (threads - min(nr_running_tasks, threads)) / threads
+ *
+ * For the 257 number crunchers on 256 CPUs, this yields:
+ *
+ * threads = min(257, 256)
+ * SOME = min(1 / 256, 1) = 0.4%
+ * FULL = (256 - min(257, 256)) / 256 = 0%
+ *
+ * For the 1 out of 4 memory-delayed tasks, this yields:
+ *
+ * threads = min(4, 4)
+ * SOME = min(1 / 4, 1) = 25%
+ * FULL = (4 - min(3, 4)) / 4 = 25%
+ *
+ * [ Substitute nr_cpus with 1, and you can see that it's a natural
+ * extension of the single-CPU model. ]
+ *
+ * Implementation
+ *
+ * To assess the precise time spent in each such state, we would have
+ * to freeze the system on task changes and start/stop the state
+ * clocks accordingly. Obviously that doesn't scale in practice.
+ *
+ * Because the scheduler aims to distribute the compute load evenly
+ * among the available CPUs, we can track task state locally to each
+ * CPU and, at much lower frequency, extrapolate the global state for
+ * the cumulative stall times and the running averages.
+ *
+ * For each runqueue, we track:
+ *
+ * tSOME[cpu] = time(nr_delayed_tasks[cpu] != 0)
+ * tFULL[cpu] = time(nr_delayed_tasks[cpu] && !nr_running_tasks[cpu])
+ * tNONIDLE[cpu] = time(nr_nonidle_tasks[cpu] != 0)
+ *
+ * and then periodically aggregate:
+ *
+ * tNONIDLE = sum(tNONIDLE[i])
+ *
+ * tSOME = sum(tSOME[i] * tNONIDLE[i]) / tNONIDLE
+ * tFULL = sum(tFULL[i] * tNONIDLE[i]) / tNONIDLE
+ *
+ * %SOME = tSOME / period
+ * %FULL = tFULL / period
+ *
+ * This gives us an approximation of pressure that is practical
+ * cost-wise, yet way more sensitive and accurate than periodic
+ * sampling of the aggregate task states would be.
+ */
+
+#include <linux/sched/loadavg.h>
+#include <linux/seq_file.h>
+#include <linux/proc_fs.h>
+#include <linux/seqlock.h>
+#include <linux/cgroup.h>
+#include <linux/module.h>
+#include <linux/sched.h>
+#include <linux/psi.h>
+#include "sched.h"
+
+static int psi_bug __read_mostly;
+
+bool psi_disabled __read_mostly;
+core_param(psi_disabled, psi_disabled, bool, 0644);
+
+/* Running averages - we need to be higher-res than loadavg */
+#define PSI_FREQ (2*HZ+1) /* 2 sec intervals */
+#define EXP_10s 1677 /* 1/exp(2s/10s) as fixed-point */
+#define EXP_60s 1981 /* 1/exp(2s/60s) */
+#define EXP_300s 2034 /* 1/exp(2s/300s) */
+
+/* Sampling frequency in nanoseconds */
+static u64 psi_period __read_mostly;
+
+/* System-level pressure and stall tracking */
+static DEFINE_PER_CPU(struct psi_group_cpu, system_group_pcpu);
+static struct psi_group psi_system = {
+ .pcpu = &system_group_pcpu,
+};
+
+static void psi_update_work(struct work_struct *work);
+
+static void group_init(struct psi_group *group)
+{
+ int cpu;
+
+ for_each_possible_cpu(cpu)
+ seqcount_init(&per_cpu_ptr(group->pcpu, cpu)->seq);
+ group->next_update = sched_clock() + psi_period;
+ INIT_DELAYED_WORK(&group->clock_work, psi_update_work);
+ mutex_init(&group->stat_lock);
+}
+
+void __init psi_init(void)
+{
+ if (psi_disabled)
+ return;
+
+ psi_period = jiffies_to_nsecs(PSI_FREQ);
+ group_init(&psi_system);
+}
+
+static bool test_state(unsigned int *tasks, enum psi_states state)
+{
+ switch (state) {
+ case PSI_IO_SOME:
+ return tasks[NR_IOWAIT];
+ case PSI_IO_FULL:
+ return tasks[NR_IOWAIT] && !tasks[NR_RUNNING];
+ case PSI_MEM_SOME:
+ return tasks[NR_MEMSTALL];
+ case PSI_MEM_FULL:
+ return tasks[NR_MEMSTALL] && !tasks[NR_RUNNING];
+ case PSI_CPU_SOME:
+ return tasks[NR_RUNNING] > 1;
+ case PSI_NONIDLE:
+ return tasks[NR_IOWAIT] || tasks[NR_MEMSTALL] ||
+ tasks[NR_RUNNING];
+ default:
+ return false;
+ }
+}
+
+static void get_recent_times(struct psi_group *group, int cpu, u32 *times)
+{
+ struct psi_group_cpu *groupc = per_cpu_ptr(group->pcpu, cpu);
+ unsigned int tasks[NR_PSI_TASK_COUNTS];
+ u64 now, state_start;
+ unsigned int seq;
+ int s;
+
+ /* Snapshot a coherent view of the CPU state */
+ do {
+ seq = read_seqcount_begin(&groupc->seq);
+ now = cpu_clock(cpu);
+ memcpy(times, groupc->times, sizeof(groupc->times));
+ memcpy(tasks, groupc->tasks, sizeof(groupc->tasks));
+ state_start = groupc->state_start;
+ } while (read_seqcount_retry(&groupc->seq, seq));
+
+ /* Calculate state time deltas against the previous snapshot */
+ for (s = 0; s < NR_PSI_STATES; s++) {
+ u32 delta;
+ /*
+ * In addition to already concluded states, we also
+ * incorporate currently active states on the CPU,
+ * since states may last for many sampling periods.
+ *
+ * This way we keep our delta sampling buckets small
+ * (u32) and our reported pressure close to what's
+ * actually happening.
+ */
+ if (test_state(tasks, s))
+ times[s] += now - state_start;
+
+ delta = times[s] - groupc->times_prev[s];
+ groupc->times_prev[s] = times[s];
+
+ times[s] = delta;
+ }
+}
+
+static void calc_avgs(unsigned long avg[3], int missed_periods,
+ u64 time, u64 period)
+{
+ unsigned long pct;
+
+ /* Fill in zeroes for periods of no activity */
+ if (missed_periods) {
+ avg[0] = calc_load_n(avg[0], EXP_10s, 0, missed_periods);
+ avg[1] = calc_load_n(avg[1], EXP_60s, 0, missed_periods);
+ avg[2] = calc_load_n(avg[2], EXP_300s, 0, missed_periods);
+ }
+
+ /* Sample the most recent active period */
+ pct = div_u64(time * 100, period);
+ pct *= FIXED_1;
+ avg[0] = calc_load(avg[0], EXP_10s, pct);
+ avg[1] = calc_load(avg[1], EXP_60s, pct);
+ avg[2] = calc_load(avg[2], EXP_300s, pct);
+}
+
+static bool update_stats(struct psi_group *group)
+{
+ u64 deltas[NR_PSI_STATES - 1] = { 0, };
+ unsigned long missed_periods = 0;
+ unsigned long nonidle_total = 0;
+ u64 now, expires, period;
+ int cpu;
+ int s;
+
+ mutex_lock(&group->stat_lock);
+
+ /*
+ * Collect the per-cpu time buckets and average them into a
+ * single time sample that is normalized to wallclock time.
+ *
+ * For averaging, each CPU is weighted by its non-idle time in
+ * the sampling period. This eliminates artifacts from uneven
+ * loading, or even entirely idle CPUs.
+ */
+ for_each_possible_cpu(cpu) {
+ u32 times[NR_PSI_STATES];
+ u32 nonidle;
+
+ get_recent_times(group, cpu, times);
+
+ nonidle = nsecs_to_jiffies(times[PSI_NONIDLE]);
+ nonidle_total += nonidle;
+
+ for (s = 0; s < PSI_NONIDLE; s++)
+ deltas[s] += (u64)times[s] * nonidle;
+ }
+
+ /*
+ * Integrate the sample into the running statistics that are
+ * reported to userspace: the cumulative stall times and the
+ * decaying averages.
+ *
+ * Pressure percentages are sampled at PSI_FREQ. We might be
+ * called more often when the user polls more frequently than
+ * that; we might be called less often when there is no task
+ * activity, thus no data, and clock ticks are sporadic. The
+ * below handles both.
+ */
+
+ /* total= */
+ for (s = 0; s < NR_PSI_STATES - 1; s++)
+ group->total[s] += div_u64(deltas[s], max(nonidle_total, 1UL));
+
+ /* avgX= */
+ now = sched_clock();
+ expires = group->next_update;
+ if (now < expires)
+ goto out;
+ if (now - expires > psi_period)
+ missed_periods = div_u64(now - expires, psi_period);
+
+ /*
+ * The periodic clock tick can get delayed for various
+ * reasons, especially on loaded systems. To avoid clock
+ * drift, we schedule the clock in fixed psi_period intervals.
+ * But the deltas we sample out of the per-cpu buckets above
+ * are based on the actual time elapsing between clock ticks.
+ */
+ group->next_update = expires + ((1 + missed_periods) * psi_period);
+ period = now - (group->last_update + (missed_periods * psi_period));
+ group->last_update = now;
+
+ for (s = 0; s < NR_PSI_STATES - 1; s++) {
+ u32 sample;
+
+ sample = group->total[s] - group->total_prev[s];
+ /*
+ * Due to the lockless sampling of the time buckets,
+ * recorded time deltas can slip into the next period,
+ * which under full pressure can result in samples in
+ * excess of the period length.
+ *
+ * We don't want to report non-sensical pressures in
+ * excess of 100%, nor do we want to drop such events
+ * on the floor. Instead we punt any overage into the
+ * future until pressure subsides. By doing this we
+ * don't underreport the occurring pressure curve, we
+ * just report it delayed by one period length.
+ *
+ * The error isn't cumulative. As soon as another
+ * delta slips from a period P to P+1, by definition
+ * it frees up its time T in P.
+ */
+ if (sample > period)
+ sample = period;
+ group->total_prev[s] += sample;
+ calc_avgs(group->avg[s], missed_periods, sample, period);
+ }
+out:
+ mutex_unlock(&group->stat_lock);
+ return nonidle_total;
+}
+
+static void psi_update_work(struct work_struct *work)
+{
+ struct delayed_work *dwork;
+ struct psi_group *group;
+ bool nonidle;
+
+ dwork = to_delayed_work(work);
+ group = container_of(dwork, struct psi_group, clock_work);
+
+ /*
+ * If there is task activity, periodically fold the per-cpu
+ * times and feed samples into the running averages. If things
+ * are idle and there is no data to process, stop the clock.
+ * Once restarted, we'll catch up the running averages in one
+ * go - see calc_avgs() and missed_periods.
+ */
+
+ nonidle = update_stats(group);
+
+ if (nonidle) {
+ unsigned long delay = 0;
+ u64 now;
+
+ now = sched_clock();
+ if (group->next_update > now)
+ delay = nsecs_to_jiffies(group->next_update - now) + 1;
+ schedule_delayed_work(dwork, delay);
+ }
+}
+
+static void record_times(struct psi_group_cpu *groupc, int cpu,
+ bool memstall_tick)
+{
+ u32 delta;
+ u64 now;
+
+ now = cpu_clock(cpu);
+ delta = now - groupc->state_start;
+ groupc->state_start = now;
+
+ if (test_state(groupc->tasks, PSI_IO_SOME)) {
+ groupc->times[PSI_IO_SOME] += delta;
+ if (test_state(groupc->tasks, PSI_IO_FULL))
+ groupc->times[PSI_IO_FULL] += delta;
+ }
+
+ if (test_state(groupc->tasks, PSI_MEM_SOME)) {
+ groupc->times[PSI_MEM_SOME] += delta;
+ if (test_state(groupc->tasks, PSI_MEM_FULL))
+ groupc->times[PSI_MEM_FULL] += delta;
+ else if (memstall_tick) {
+ u32 sample;
+ /*
+ * Since we care about lost potential, a
+ * memstall is FULL when there are no other
+ * working tasks, but also when the CPU is
+ * actively reclaiming and nothing productive
+ * could run even if it were runnable.
+ *
+ * When the timer tick sees a reclaiming CPU,
+ * regardless of runnable tasks, sample a FULL
+ * tick (or less if it hasn't been a full tick
+ * since the last state change).
+ */
+ sample = min(delta, (u32)jiffies_to_nsecs(1));
+ groupc->times[PSI_MEM_FULL] += sample;
+ }
+ }
+
+ if (test_state(groupc->tasks, PSI_CPU_SOME))
+ groupc->times[PSI_CPU_SOME] += delta;
+
+ if (test_state(groupc->tasks, PSI_NONIDLE))
+ groupc->times[PSI_NONIDLE] += delta;
+}
+
+static void psi_group_change(struct psi_group *group, int cpu,
+ unsigned int clear, unsigned int set)
+{
+ struct psi_group_cpu *groupc;
+ unsigned int t, m;
+
+ groupc = per_cpu_ptr(group->pcpu, cpu);
+
+ /*
+ * First we assess the aggregate resource states this CPU's
+ * tasks have been in since the last change, and account any
+ * SOME and FULL time these may have resulted in.
+ *
+ * Then we update the task counts according to the state
+ * change requested through the @clear and @set bits.
+ */
+ write_seqcount_begin(&groupc->seq);
+
+ record_times(groupc, cpu, false);
+
+ for (t = 0, m = clear; m; m &= ~(1 << t), t++) {
+ if (!(m & (1 << t)))
+ continue;
+ if (groupc->tasks[t] == 0 && !psi_bug) {
+ printk_deferred(KERN_ERR "psi: task underflow! cpu=%d t=%d tasks=[%u %u %u] clear=%x set=%x\n",
+ cpu, t, groupc->tasks[0],
+ groupc->tasks[1], groupc->tasks[2],
+ clear, set);
+ psi_bug = 1;
+ }
+ groupc->tasks[t]--;
+ }
+
+ for (t = 0; set; set &= ~(1 << t), t++)
+ if (set & (1 << t))
+ groupc->tasks[t]++;
+
+ write_seqcount_end(&groupc->seq);
+
+ if (!delayed_work_pending(&group->clock_work))
+ schedule_delayed_work(&group->clock_work, PSI_FREQ);
+}
+
+static struct psi_group *iterate_groups(struct task_struct *task, void **iter)
+{
+#ifdef CONFIG_CGROUPS
+ struct cgroup *cgroup = NULL;
+
+ if (!*iter)
+ cgroup = task->cgroups->dfl_cgrp;
+ else if (*iter == &psi_system)
+ return NULL;
+ else
+ cgroup = cgroup_parent(*iter);
+
+ if (cgroup && cgroup_parent(cgroup)) {
+ *iter = cgroup;
+ return cgroup_psi(cgroup);
+ }
+#else
+ if (*iter)
+ return NULL;
+#endif
+ *iter = &psi_system;
+ return &psi_system;
+}
+
+void psi_task_change(struct task_struct *task, int clear, int set)
+{
+ int cpu = task_cpu(task);
+ struct psi_group *group;
+ void *iter = NULL;
+
+ if (!task->pid)
+ return;
+
+ if (((task->psi_flags & set) ||
+ (task->psi_flags & clear) != clear) &&
+ !psi_bug) {
+ printk_deferred(KERN_ERR "psi: inconsistent task state! task=%d:%s cpu=%d psi_flags=%x clear=%x set=%x\n",
+ task->pid, task->comm, cpu,
+ task->psi_flags, clear, set);
+ psi_bug = 1;
+ }
+
+ task->psi_flags &= ~clear;
+ task->psi_flags |= set;
+
+ while ((group = iterate_groups(task, &iter)))
+ psi_group_change(group, cpu, clear, set);
+}
+
+void psi_memstall_tick(struct task_struct *task, int cpu)
+{
+ struct psi_group *group;
+ void *iter = NULL;
+
+ while ((group = iterate_groups(task, &iter))) {
+ struct psi_group_cpu *groupc;
+
+ groupc = per_cpu_ptr(group->pcpu, cpu);
+ write_seqcount_begin(&groupc->seq);
+ record_times(groupc, cpu, true);
+ write_seqcount_end(&groupc->seq);
+ }
+}
+
+/**
+ * psi_memstall_enter - mark the beginning of a memory stall section
+ * @flags: flags to handle nested sections
+ *
+ * Marks the calling task as being stalled due to a lack of memory,
+ * such as waiting for a refault or performing reclaim.
+ */
+void psi_memstall_enter(unsigned long *flags)
+{
+ struct rq_flags rf;
+ struct rq *rq;
+
+ if (psi_disabled)
+ return;
+
+ *flags = current->flags & PF_MEMSTALL;
+ if (*flags)
+ return;
+ /*
+ * PF_MEMSTALL setting & accounting needs to be atomic wrt
+ * changes to the task's scheduling state, otherwise we can
+ * race with CPU migration.
+ */
+ rq = this_rq_lock_irq(&rf);
+
+ current->flags |= PF_MEMSTALL;
+ psi_task_change(current, 0, TSK_MEMSTALL);
+
+ rq_unlock_irq(rq, &rf);
+}
+
+/**
+ * psi_memstall_leave - mark the end of an memory stall section
+ * @flags: flags to handle nested memdelay sections
+ *
+ * Marks the calling task as no longer stalled due to lack of memory.
+ */
+void psi_memstall_leave(unsigned long *flags)
+{
+ struct rq_flags rf;
+ struct rq *rq;
+
+ if (psi_disabled)
+ return;
+
+ if (*flags)
+ return;
+ /*
+ * PF_MEMSTALL clearing & accounting needs to be atomic wrt
+ * changes to the task's scheduling state, otherwise we could
+ * race with CPU migration.
+ */
+ rq = this_rq_lock_irq(&rf);
+
+ current->flags &= ~PF_MEMSTALL;
+ psi_task_change(current, TSK_MEMSTALL, 0);
+
+ rq_unlock_irq(rq, &rf);
+}
+
+#ifdef CONFIG_CGROUPS
+int psi_cgroup_alloc(struct cgroup *cgroup)
+{
+ if (psi_disabled)
+ return 0;
+
+ cgroup->psi.pcpu = alloc_percpu(struct psi_group_cpu);
+ if (!cgroup->psi.pcpu)
+ return -ENOMEM;
+ group_init(&cgroup->psi);
+ return 0;
+}
+
+void psi_cgroup_free(struct cgroup *cgroup)
+{
+ if (psi_disabled)
+ return;
+
+ cancel_delayed_work_sync(&cgroup->psi.clock_work);
+ free_percpu(cgroup->psi.pcpu);
+}
+
+/**
+ * cgroup_move_task - move task to a different cgroup
+ * @task: the task
+ * @to: the target css_set
+ *
+ * Move task to a new cgroup and safely migrate its associated stall
+ * state between the different groups.
+ *
+ * This function acquires the task's rq lock to lock out concurrent
+ * changes to the task's scheduling state and - in case the task is
+ * running - concurrent changes to its stall state.
+ */
+void cgroup_move_task(struct task_struct *task, struct css_set *to)
+{
+ bool move_psi = !psi_disabled;
+ unsigned int task_flags = 0;
+ struct rq_flags rf;
+ struct rq *rq = NULL;
+
+ if (move_psi) {
+ rq = task_rq_lock(task, &rf);
+
+ if (task_on_rq_queued(task))
+ task_flags = TSK_RUNNING;
+ else if (task->in_iowait)
+ task_flags = TSK_IOWAIT;
+
+ if (task->flags & PF_MEMSTALL)
+ task_flags |= TSK_MEMSTALL;
+
+ if (task_flags)
+ psi_task_change(task, task_flags, 0);
+ }
+
+ /*
+ * Lame to do this here, but the scheduler cannot be locked
+ * from the outside, so we move cgroups from inside sched/.
+ */
+ rcu_assign_pointer(task->cgroups, to);
+
+ if (move_psi) {
+ if (task_flags)
+ psi_task_change(task, 0, task_flags);
+
+ task_rq_unlock(rq, task, &rf);
+ }
+}
+#endif /* CONFIG_CGROUPS */
+
+int psi_show(struct seq_file *m, struct psi_group *group, enum psi_res res)
+{
+ int full;
+
+ if (psi_disabled)
+ return -EOPNOTSUPP;
+
+ update_stats(group);
+
+ for (full = 0; full < 2 - (res == PSI_CPU); full++) {
+ unsigned long avg[3];
+ u64 total;
+ int w;
+
+ for (w = 0; w < 3; w++)
+ avg[w] = group->avg[res * 2 + full][w];
+ total = div_u64(group->total[res * 2 + full], NSEC_PER_USEC);
+
+ seq_printf(m, "%s avg10=%lu.%02lu avg60=%lu.%02lu avg300=%lu.%02lu total=%llu\n",
+ full ? "full" : "some",
+ LOAD_INT(avg[0]), LOAD_FRAC(avg[0]),
+ LOAD_INT(avg[1]), LOAD_FRAC(avg[1]),
+ LOAD_INT(avg[2]), LOAD_FRAC(avg[2]),
+ total);
+ }
+
+ return 0;
+}
+
+static int psi_io_show(struct seq_file *m, void *v)
+{
+ return psi_show(m, &psi_system, PSI_IO);
+}
+
+static int psi_memory_show(struct seq_file *m, void *v)
+{
+ return psi_show(m, &psi_system, PSI_MEM);
+}
+
+static int psi_cpu_show(struct seq_file *m, void *v)
+{
+ return psi_show(m, &psi_system, PSI_CPU);
+}
+
+static int psi_io_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, psi_io_show, NULL);
+}
+
+static int psi_memory_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, psi_memory_show, NULL);
+}
+
+static int psi_cpu_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, psi_cpu_show, NULL);
+}
+
+static const struct file_operations psi_io_fops = {
+ .open = psi_io_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static const struct file_operations psi_memory_fops = {
+ .open = psi_memory_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static const struct file_operations psi_cpu_fops = {
+ .open = psi_cpu_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static int __init psi_proc_init(void)
+{
+ proc_mkdir("pressure", NULL);
+ proc_create("pressure/io", 0, NULL, &psi_io_fops);
+ proc_create("pressure/memory", 0, NULL, &psi_memory_fops);
+ proc_create("pressure/cpu", 0, NULL, &psi_cpu_fops);
+ return 0;
+}
+module_init(psi_proc_init);
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index f23f040..eb0e9c0 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -839,6 +839,8 @@ static int do_sched_rt_period_timer(struct rt_bandwidth *rt_b, int overrun)
struct rq *rq = rq_of_rt_rq(rt_rq);
raw_spin_lock(&rq->lock);
+ update_rq_clock(rq);
+
if (rt_rq->rt_time) {
u64 runtime;
@@ -1623,7 +1625,7 @@ static struct task_struct *_pick_next_task_rt(struct rq *rq)
}
static struct task_struct *
-pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie)
+pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
{
struct task_struct *p;
struct rt_rq *rt_rq = &rq->rt;
@@ -1635,9 +1637,9 @@ pick_next_task_rt(struct rq *rq, struct task_struct *prev, struct pin_cookie coo
* disabled avoiding further scheduler activity on it and we're
* being very careful to re-start the picking loop.
*/
- lockdep_unpin_lock(&rq->lock, cookie);
+ rq_unpin_lock(rq, rf);
pull_rt_task(rq);
- lockdep_repin_lock(&rq->lock, cookie);
+ rq_repin_lock(rq, rf);
/*
* pull_rt_task() can drop (and re-acquire) rq->lock; this
* means a dl or stop task can slip in, in which case we need
@@ -1770,10 +1772,8 @@ static int find_lowest_rq(struct task_struct *task)
sched_boost_policy() : SCHED_BOOST_NONE;
best_capacity = placement_boost ? 0 : ULONG_MAX;
- rcu_read_lock();
sd = rcu_dereference(per_cpu(sd_ea, start_cpu));
if (!sd) {
- rcu_read_unlock();
goto noea;
}
@@ -1803,7 +1803,6 @@ static int find_lowest_rq(struct task_struct *task)
}
}
} while (sg = sg->next, sg != sd->groups);
- rcu_read_unlock();
if (sg_target) {
cpumask_and(&search_cpu, lowest_mask,
@@ -1922,7 +1921,6 @@ static int find_lowest_rq(struct task_struct *task)
if (!cpumask_test_cpu(this_cpu, lowest_mask))
this_cpu = -1; /* Skip this_cpu opt if not among lowest */
- rcu_read_lock();
for_each_domain(cpu, sd) {
if (sd->flags & SD_WAKE_AFFINE) {
int best_cpu;
@@ -1933,19 +1931,16 @@ static int find_lowest_rq(struct task_struct *task)
*/
if (this_cpu != -1 &&
cpumask_test_cpu(this_cpu, sched_domain_span(sd))) {
- rcu_read_unlock();
return this_cpu;
}
best_cpu = cpumask_first_and(lowest_mask,
sched_domain_span(sd));
if (best_cpu < nr_cpu_ids) {
- rcu_read_unlock();
return best_cpu;
}
}
}
- rcu_read_unlock();
/*
* And finally, if there were no matches within the domains
@@ -1969,7 +1964,9 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
int cpu;
for (tries = 0; tries < RT_MAX_TRIES; tries++) {
+ rcu_read_lock();
cpu = find_lowest_rq(task);
+ rcu_read_unlock();
if ((cpu == -1) || (cpu == rq->cpu))
break;
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index d73dfae..e52915e 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -4,8 +4,10 @@
#include <linux/sched/rt.h>
#include <linux/u64_stats_sync.h>
#include <linux/sched/deadline.h>
+#include <linux/kernel_stat.h>
#include <linux/binfmts.h>
#include <linux/mutex.h>
+#include <linux/psi.h>
#include <linux/spinlock.h>
#include <linux/stop_machine.h>
#include <linux/irq_work.h>
@@ -301,6 +303,7 @@ extern struct mutex sched_domains_mutex;
#ifdef CONFIG_CGROUP_SCHED
#include <linux/cgroup.h>
+#include <linux/psi.h>
struct cfs_rq;
struct rt_rq;
@@ -322,6 +325,8 @@ struct cfs_bandwidth {
/* statistics */
int nr_periods, nr_throttled;
u64 throttled_time;
+
+ bool distribute_running;
#endif
};
@@ -897,6 +902,8 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct rq, runqueues);
#define cpu_curr(cpu) (cpu_rq(cpu)->curr)
#define raw_rq() raw_cpu_ptr(&runqueues)
+extern void update_rq_clock(struct rq *rq);
+
static inline u64 __rq_clock_broken(struct rq *rq)
{
return READ_ONCE(rq->clock);
@@ -926,6 +933,118 @@ static inline void rq_clock_skip_update(struct rq *rq, bool skip)
rq->clock_skip_update &= ~RQCF_REQ_SKIP;
}
+struct rq_flags {
+ unsigned long flags;
+ struct pin_cookie cookie;
+};
+
+static inline void rq_pin_lock(struct rq *rq, struct rq_flags *rf)
+{
+ rf->cookie = lockdep_pin_lock(&rq->lock);
+}
+
+static inline void rq_unpin_lock(struct rq *rq, struct rq_flags *rf)
+{
+ lockdep_unpin_lock(&rq->lock, rf->cookie);
+}
+
+static inline void rq_repin_lock(struct rq *rq, struct rq_flags *rf)
+{
+ lockdep_repin_lock(&rq->lock, rf->cookie);
+}
+
+struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)
+ __acquires(rq->lock);
+
+struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)
+ __acquires(p->pi_lock)
+ __acquires(rq->lock);
+
+static inline void __task_rq_unlock(struct rq *rq, struct rq_flags *rf)
+ __releases(rq->lock)
+{
+ rq_unpin_lock(rq, rf);
+ raw_spin_unlock(&rq->lock);
+}
+
+static inline void
+task_rq_unlock(struct rq *rq, struct task_struct *p, struct rq_flags *rf)
+ __releases(rq->lock)
+ __releases(p->pi_lock)
+{
+ rq_unpin_lock(rq, rf);
+ raw_spin_unlock(&rq->lock);
+ raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
+}
+
+static inline void
+rq_lock_irqsave(struct rq *rq, struct rq_flags *rf)
+ __acquires(rq->lock)
+{
+ raw_spin_lock_irqsave(&rq->lock, rf->flags);
+ rq_pin_lock(rq, rf);
+}
+
+static inline void
+rq_lock_irq(struct rq *rq, struct rq_flags *rf)
+ __acquires(rq->lock)
+{
+ raw_spin_lock_irq(&rq->lock);
+ rq_pin_lock(rq, rf);
+}
+
+static inline void
+rq_lock(struct rq *rq, struct rq_flags *rf)
+ __acquires(rq->lock)
+{
+ raw_spin_lock(&rq->lock);
+ rq_pin_lock(rq, rf);
+}
+
+static inline void
+rq_relock(struct rq *rq, struct rq_flags *rf)
+ __acquires(rq->lock)
+{
+ raw_spin_lock(&rq->lock);
+ rq_repin_lock(rq, rf);
+}
+
+static inline void
+rq_unlock_irqrestore(struct rq *rq, struct rq_flags *rf)
+ __releases(rq->lock)
+{
+ rq_unpin_lock(rq, rf);
+ raw_spin_unlock_irqrestore(&rq->lock, rf->flags);
+}
+
+static inline void
+rq_unlock_irq(struct rq *rq, struct rq_flags *rf)
+ __releases(rq->lock)
+{
+ rq_unpin_lock(rq, rf);
+ raw_spin_unlock_irq(&rq->lock);
+}
+
+static inline void
+rq_unlock(struct rq *rq, struct rq_flags *rf)
+ __releases(rq->lock)
+{
+ rq_unpin_lock(rq, rf);
+ raw_spin_unlock(&rq->lock);
+}
+
+static inline struct rq *
+this_rq_lock_irq(struct rq_flags *rf)
+ __acquires(rq->lock)
+{
+ struct rq *rq;
+
+ local_irq_disable();
+ rq = this_rq();
+ rq_lock(rq, rf);
+ return rq;
+}
+
#ifdef CONFIG_NUMA
enum numa_topology_type {
NUMA_DIRECT,
@@ -1400,7 +1519,7 @@ struct sched_class {
*/
struct task_struct * (*pick_next_task) (struct rq *rq,
struct task_struct *prev,
- struct pin_cookie cookie);
+ struct rq_flags *rf);
void (*put_prev_task) (struct rq *rq, struct task_struct *p);
#ifdef CONFIG_SMP
@@ -1652,8 +1771,6 @@ static inline void rq_last_tick_reset(struct rq *rq)
#endif
}
-extern void update_rq_clock(struct rq *rq);
-
extern void activate_task(struct rq *rq, struct task_struct *p, int flags);
extern void deactivate_task(struct rq *rq, struct task_struct *p, int flags);
@@ -1949,34 +2066,6 @@ static inline void sched_rt_avg_update(struct rq *rq, u64 rt_delta) { }
static inline void sched_avg_update(struct rq *rq) { }
#endif
-struct rq_flags {
- unsigned long flags;
- struct pin_cookie cookie;
-};
-
-struct rq *__task_rq_lock(struct task_struct *p, struct rq_flags *rf)
- __acquires(rq->lock);
-struct rq *task_rq_lock(struct task_struct *p, struct rq_flags *rf)
- __acquires(p->pi_lock)
- __acquires(rq->lock);
-
-static inline void __task_rq_unlock(struct rq *rq, struct rq_flags *rf)
- __releases(rq->lock)
-{
- lockdep_unpin_lock(&rq->lock, rf->cookie);
- raw_spin_unlock(&rq->lock);
-}
-
-static inline void
-task_rq_unlock(struct rq *rq, struct task_struct *p, struct rq_flags *rf)
- __releases(rq->lock)
- __releases(p->pi_lock)
-{
- lockdep_unpin_lock(&rq->lock, rf->cookie);
- raw_spin_unlock(&rq->lock);
- raw_spin_unlock_irqrestore(&p->pi_lock, rf->flags);
-}
-
extern struct rq *lock_rq_of(struct task_struct *p, struct rq_flags *flags);
extern void unlock_rq_of(struct rq *rq, struct task_struct *p, struct rq_flags *flags);
@@ -2210,14 +2299,19 @@ static inline void nohz_balance_exit_idle(unsigned int cpu) { }
#ifdef CONFIG_IRQ_TIME_ACCOUNTING
struct irqtime {
- u64 hardirq_time;
- u64 softirq_time;
+ u64 total;
+ u64 tick_delta;
u64 irq_start_time;
struct u64_stats_sync sync;
};
DECLARE_PER_CPU(struct irqtime, cpu_irqtime);
+/*
+ * Returns the irqtime minus the softirq time computed by ksoftirqd.
+ * Otherwise ksoftirqd's sum_exec_runtime is substracted its own runtime
+ * and never move forward.
+ */
static inline u64 irq_time_read(int cpu)
{
struct irqtime *irqtime = &per_cpu(cpu_irqtime, cpu);
@@ -2226,7 +2320,7 @@ static inline u64 irq_time_read(int cpu)
do {
seq = __u64_stats_fetch_begin(&irqtime->sync);
- total = irqtime->softirq_time + irqtime->hardirq_time;
+ total = irqtime->total;
} while (__u64_stats_fetch_retry(&irqtime->sync, seq));
return total;
@@ -2268,11 +2362,6 @@ static inline void cpufreq_update_util(struct rq *rq, unsigned int flags)
{
struct update_util_data *data;
-#ifdef CONFIG_SCHED_WALT
- if (!(flags & SCHED_CPUFREQ_WALT))
- return;
-#endif
-
data = rcu_dereference_sched(*per_cpu_ptr(&cpufreq_update_util_data,
cpu_of(rq)));
if (data)
diff --git a/kernel/sched/stats.c b/kernel/sched/stats.c
index 6d74a7c..9d1637a 100644
--- a/kernel/sched/stats.c
+++ b/kernel/sched/stats.c
@@ -20,8 +20,8 @@ static inline void show_easstat(struct seq_file *seq, struct eas_stats *stats)
stats->sis_attempts, stats->sis_idle, stats->sis_cache_affine,
stats->sis_suff_cap, stats->sis_idle_cpu, stats->sis_count);
- seq_printf(seq, "%llu %llu %llu %llu %llu %llu %llu ",
- stats->secb_attempts, stats->secb_sync, stats->secb_idle_bt,
+ seq_printf(seq, "%llu %llu %llu %llu %llu %llu ",
+ stats->secb_attempts, stats->secb_idle_bt,
stats->secb_insuff_cap, stats->secb_no_nrg_sav,
stats->secb_nrg_sav, stats->secb_count);
diff --git a/kernel/sched/stats.h b/kernel/sched/stats.h
index 34659a8..3bace95 100644
--- a/kernel/sched/stats.h
+++ b/kernel/sched/stats.h
@@ -54,6 +54,92 @@ rq_sched_info_depart(struct rq *rq, unsigned long long delta)
#define schedstat_val_or_zero(var) 0
#endif /* CONFIG_SCHEDSTATS */
+#ifdef CONFIG_PSI
+/*
+ * PSI tracks state that persists across sleeps, such as iowaits and
+ * memory stalls. As a result, it has to distinguish between sleeps,
+ * where a task's runnable state changes, and requeues, where a task
+ * and its state are being moved between CPUs and runqueues.
+ */
+static inline void psi_enqueue(struct task_struct *p, bool wakeup)
+{
+ int clear = 0, set = TSK_RUNNING;
+
+ if (psi_disabled)
+ return;
+
+ if (!wakeup || p->sched_psi_wake_requeue) {
+ if (p->flags & PF_MEMSTALL)
+ set |= TSK_MEMSTALL;
+ if (p->sched_psi_wake_requeue)
+ p->sched_psi_wake_requeue = 0;
+ } else {
+ if (p->in_iowait)
+ clear |= TSK_IOWAIT;
+ }
+
+ psi_task_change(p, clear, set);
+}
+
+static inline void psi_dequeue(struct task_struct *p, bool sleep)
+{
+ int clear = TSK_RUNNING, set = 0;
+
+ if (psi_disabled)
+ return;
+
+ if (!sleep) {
+ if (p->flags & PF_MEMSTALL)
+ clear |= TSK_MEMSTALL;
+ } else {
+ if (p->in_iowait)
+ set |= TSK_IOWAIT;
+ }
+
+ psi_task_change(p, clear, set);
+}
+
+static inline void psi_ttwu_dequeue(struct task_struct *p)
+{
+ if (psi_disabled)
+ return;
+ /*
+ * Is the task being migrated during a wakeup? Make sure to
+ * deregister its sleep-persistent psi states from the old
+ * queue, and let psi_enqueue() know it has to requeue.
+ */
+ if (unlikely(p->in_iowait || (p->flags & PF_MEMSTALL))) {
+ struct rq_flags rf;
+ struct rq *rq;
+ int clear = 0;
+
+ if (p->in_iowait)
+ clear |= TSK_IOWAIT;
+ if (p->flags & PF_MEMSTALL)
+ clear |= TSK_MEMSTALL;
+
+ rq = __task_rq_lock(p, &rf);
+ psi_task_change(p, clear, 0);
+ p->sched_psi_wake_requeue = 1;
+ __task_rq_unlock(rq, &rf);
+ }
+}
+
+static inline void psi_task_tick(struct rq *rq)
+{
+ if (psi_disabled)
+ return;
+
+ if (unlikely(rq->curr->flags & PF_MEMSTALL))
+ psi_memstall_tick(rq->curr, cpu_of(rq));
+}
+#else /* CONFIG_PSI */
+static inline void psi_enqueue(struct task_struct *p, bool wakeup) {}
+static inline void psi_dequeue(struct task_struct *p, bool sleep) {}
+static inline void psi_ttwu_dequeue(struct task_struct *p) {}
+static inline void psi_task_tick(struct rq *rq) {}
+#endif /* CONFIG_PSI */
+
#ifdef CONFIG_SCHED_INFO
static inline void sched_info_reset_dequeued(struct task_struct *t)
{
diff --git a/kernel/sched/stop_task.c b/kernel/sched/stop_task.c
index 11a1888..a6ce92c 100644
--- a/kernel/sched/stop_task.c
+++ b/kernel/sched/stop_task.c
@@ -25,7 +25,7 @@ check_preempt_curr_stop(struct rq *rq, struct task_struct *p, int flags)
}
static struct task_struct *
-pick_next_task_stop(struct rq *rq, struct task_struct *prev, struct pin_cookie cookie)
+pick_next_task_stop(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
{
struct task_struct *stop = rq->stop;
diff --git a/kernel/sched/swait.c b/kernel/sched/swait.c
index 82f0dff..9c2da06 100644
--- a/kernel/sched/swait.c
+++ b/kernel/sched/swait.c
@@ -33,9 +33,6 @@ void swake_up(struct swait_queue_head *q)
{
unsigned long flags;
- if (!swait_active(q))
- return;
-
raw_spin_lock_irqsave(&q->lock, flags);
swake_up_locked(q);
raw_spin_unlock_irqrestore(&q->lock, flags);
@@ -51,9 +48,6 @@ void swake_up_all(struct swait_queue_head *q)
struct swait_queue *curr;
LIST_HEAD(tmp);
- if (!swait_active(q))
- return;
-
raw_spin_lock_irq(&q->lock);
list_splice_init(&q->task_list, &tmp);
while (!list_empty(&tmp)) {
diff --git a/kernel/sched/tune.c b/kernel/sched/tune.c
index 192e8c7..c4c49fa 100644
--- a/kernel/sched/tune.c
+++ b/kernel/sched/tune.c
@@ -723,7 +723,7 @@ prefer_idle_write(struct cgroup_subsys_state *css, struct cftype *cft,
u64 prefer_idle)
{
struct schedtune *st = css_st(css);
- st->prefer_idle = prefer_idle;
+ st->prefer_idle = !!prefer_idle;
return 0;
}
diff --git a/kernel/sched/walt.c b/kernel/sched/walt.c
index 8bacb6f..7bbb6fd 100644
--- a/kernel/sched/walt.c
+++ b/kernel/sched/walt.c
@@ -54,10 +54,15 @@ static void acquire_rq_locks_irqsave(const cpumask_t *cpus,
unsigned long *flags)
{
int cpu;
-
+ int level = 0;
local_irq_save(*flags);
- for_each_cpu(cpu, cpus)
- raw_spin_lock(&cpu_rq(cpu)->lock);
+ for_each_cpu(cpu, cpus) {
+ if (level == 0)
+ raw_spin_lock(&cpu_rq(cpu)->lock);
+ else
+ raw_spin_lock_nested(&cpu_rq(cpu)->lock, level);
+ level++;
+ }
}
static void release_rq_locks_irqrestore(const cpumask_t *cpus,
@@ -665,14 +670,10 @@ static inline void inter_cluster_migration_fixup
BUG_ON((s64)src_rq->nt_curr_runnable_sum < 0);
}
-static int load_to_index(u32 load)
+static u32 load_to_index(u32 load)
{
- if (load < sched_load_granule)
- return 0;
- else if (load >= sched_ravg_window)
- return NUM_LOAD_INDICES - 1;
- else
- return load / sched_load_granule;
+ u32 index = load / sched_load_granule;
+ return min(index, (u32)(NUM_LOAD_INDICES - 1));
}
static void
@@ -3135,15 +3136,20 @@ void walt_irq_work(struct irq_work *irq_work)
struct rq *rq;
int cpu;
u64 wc;
- int flag = SCHED_CPUFREQ_WALT;
bool is_migration = false;
+ int level = 0;
/* Am I the window rollover work or the migration work? */
if (irq_work == &walt_migration_irq_work)
is_migration = true;
- for_each_cpu(cpu, cpu_possible_mask)
- raw_spin_lock(&cpu_rq(cpu)->lock);
+ for_each_cpu(cpu, cpu_possible_mask) {
+ if (level == 0)
+ raw_spin_lock(&cpu_rq(cpu)->lock);
+ else
+ raw_spin_lock_nested(&cpu_rq(cpu)->lock, level);
+ level++;
+ }
wc = ktime_get_ns();
walt_load_reported_window = atomic64_read(&walt_irq_work_lastq_ws);
@@ -3169,16 +3175,16 @@ void walt_irq_work(struct irq_work *irq_work)
for_each_sched_cluster(cluster) {
for_each_cpu(cpu, &cluster->cpus) {
- int nflag = flag;
+ int nflag = 0;
rq = cpu_rq(cpu);
if (is_migration) {
if (rq->notif_pending) {
- nflag |= SCHED_CPUFREQ_INTERCLUSTER_MIG;
+ nflag = SCHED_CPUFREQ_INTERCLUSTER_MIG;
rq->notif_pending = false;
} else {
- nflag |= SCHED_CPUFREQ_FORCE_UPDATE;
+ nflag = SCHED_CPUFREQ_FORCE_UPDATE;
}
}
diff --git a/kernel/scs.c b/kernel/scs.c
new file mode 100644
index 0000000..cec9ab3
--- /dev/null
+++ b/kernel/scs.c
@@ -0,0 +1,143 @@
+/*
+ * Shadow Call Stack support.
+ *
+ * Copyright (C) 2018 Google LLC
+ */
+
+#include <linux/cpuhotplug.h>
+#include <linux/mm.h>
+#include <linux/mmzone.h>
+#include <linux/slab.h>
+#include <linux/scs.h>
+#include <linux/vmstat.h>
+
+#include <asm/scs.h>
+
+#define SCS_END_MAGIC 0xaf0194819b1635f6UL
+
+static inline void *__scs_base(struct task_struct *tsk)
+{
+ return (void *)((uintptr_t)task_scs(tsk) & ~(SCS_SIZE - 1));
+}
+
+static inline void *scs_alloc(int node)
+{
+ return kmalloc(SCS_SIZE, SCS_GFP);
+}
+
+static inline void scs_free(void *s)
+{
+ kfree(s);
+}
+
+static struct page *__scs_page(struct task_struct *tsk)
+{
+ return virt_to_page(__scs_base(tsk));
+}
+
+static inline unsigned long *scs_magic(struct task_struct *tsk)
+{
+ return (unsigned long *)(__scs_base(tsk) + SCS_SIZE - sizeof(long));
+}
+
+static inline void scs_set_magic(struct task_struct *tsk)
+{
+ *scs_magic(tsk) = SCS_END_MAGIC;
+}
+
+void scs_task_init(struct task_struct *tsk)
+{
+ task_set_scs(tsk, NULL);
+}
+
+void scs_task_reset(struct task_struct *tsk)
+{
+ task_set_scs(tsk, __scs_base(tsk));
+}
+
+void scs_set_init_magic(struct task_struct *tsk)
+{
+ scs_save(tsk);
+ scs_set_magic(tsk);
+ scs_load(tsk);
+}
+
+static void scs_account(struct task_struct *tsk, int account)
+{
+ mod_zone_page_state(page_zone(__scs_page(tsk)), NR_KERNEL_SCS_BYTES,
+ account * SCS_SIZE);
+}
+
+int scs_prepare(struct task_struct *tsk, int node)
+{
+ void *s;
+
+ s = scs_alloc(node);
+ if (!s)
+ return -ENOMEM;
+
+ task_set_scs(tsk, s);
+ scs_set_magic(tsk);
+ scs_account(tsk, 1);
+
+ return 0;
+}
+
+#ifdef CONFIG_DEBUG_STACK_USAGE
+static inline unsigned long scs_used(struct task_struct *tsk)
+{
+ unsigned long *p = __scs_base(tsk);
+ unsigned long *end = scs_magic(tsk);
+ uintptr_t s = (uintptr_t)p;
+
+ while (p < end && *p)
+ p++;
+
+ return (uintptr_t)p - s;
+}
+
+static void scs_check_usage(struct task_struct *tsk)
+{
+ static DEFINE_SPINLOCK(lock);
+ static unsigned long highest;
+ unsigned long used = scs_used(tsk);
+
+ if (used <= highest)
+ return;
+
+ spin_lock(&lock);
+
+ if (used > highest) {
+ pr_info("%s: highest shadow stack usage %lu bytes\n",
+ __func__, used);
+ highest = used;
+ }
+
+ spin_unlock(&lock);
+}
+#else
+static inline void scs_check_usage(struct task_struct *tsk)
+{
+}
+#endif
+
+bool scs_corrupted(struct task_struct *tsk)
+{
+ return *scs_magic(tsk) != SCS_END_MAGIC;
+}
+
+void scs_release(struct task_struct *tsk)
+{
+ void *s;
+
+ s = __scs_base(tsk);
+ if (!s)
+ return;
+
+ BUG_ON(scs_corrupted(tsk));
+ scs_check_usage(tsk);
+
+ scs_account(tsk, -1);
+ scs_task_init(tsk);
+ scs_free(s);
+}
diff --git a/kernel/seccomp.c b/kernel/seccomp.c
index af182a6..3975856 100644
--- a/kernel/seccomp.c
+++ b/kernel/seccomp.c
@@ -16,6 +16,8 @@
#include <linux/atomic.h>
#include <linux/audit.h>
#include <linux/compat.h>
+#include <linux/nospec.h>
+#include <linux/prctl.h>
#include <linux/sched.h>
#include <linux/seccomp.h>
#include <linux/slab.h>
@@ -214,8 +216,11 @@ static inline bool seccomp_may_assign_mode(unsigned long seccomp_mode)
return true;
}
+void __weak arch_seccomp_spec_mitigate(struct task_struct *task) { }
+
static inline void seccomp_assign_mode(struct task_struct *task,
- unsigned long seccomp_mode)
+ unsigned long seccomp_mode,
+ unsigned long flags)
{
assert_spin_locked(&task->sighand->siglock);
@@ -225,6 +230,9 @@ static inline void seccomp_assign_mode(struct task_struct *task,
* filter) is set.
*/
smp_mb__before_atomic();
+ /* Assume default seccomp processes want spec flaw mitigation. */
+ if ((flags & SECCOMP_FILTER_FLAG_SPEC_ALLOW) == 0)
+ arch_seccomp_spec_mitigate(task);
set_tsk_thread_flag(task, TIF_SECCOMP);
}
@@ -292,7 +300,7 @@ static inline pid_t seccomp_can_sync_threads(void)
* without dropping the locks.
*
*/
-static inline void seccomp_sync_threads(void)
+static inline void seccomp_sync_threads(unsigned long flags)
{
struct task_struct *thread, *caller;
@@ -333,7 +341,8 @@ static inline void seccomp_sync_threads(void)
* allow one thread to transition the other.
*/
if (thread->seccomp.mode == SECCOMP_MODE_DISABLED)
- seccomp_assign_mode(thread, SECCOMP_MODE_FILTER);
+ seccomp_assign_mode(thread, SECCOMP_MODE_FILTER,
+ flags);
}
}
@@ -452,7 +461,7 @@ static long seccomp_attach_filter(unsigned int flags,
/* Now that the new filter is in place, synchronize to all threads. */
if (flags & SECCOMP_FILTER_FLAG_TSYNC)
- seccomp_sync_threads();
+ seccomp_sync_threads(flags);
return 0;
}
@@ -712,7 +721,7 @@ static long seccomp_set_mode_strict(void)
#ifdef TIF_NOTSC
disable_TSC();
#endif
- seccomp_assign_mode(current, seccomp_mode);
+ seccomp_assign_mode(current, seccomp_mode, 0);
ret = 0;
out:
@@ -770,7 +779,7 @@ static long seccomp_set_mode_filter(unsigned int flags,
/* Do not free the successfully attached filter. */
prepared = NULL;
- seccomp_assign_mode(current, seccomp_mode);
+ seccomp_assign_mode(current, seccomp_mode, flags);
out:
spin_unlock_irq(¤t->sighand->siglock);
if (flags & SECCOMP_FILTER_FLAG_TSYNC)
diff --git a/kernel/signal.c b/kernel/signal.c
index 7ebe236..049929a 100644
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -991,7 +991,7 @@ static int __send_signal(int sig, struct siginfo *info, struct task_struct *t,
result = TRACE_SIGNAL_IGNORED;
if (!prepare_signal(sig, t,
- from_ancestor_ns || (info == SEND_SIG_FORCED)))
+ from_ancestor_ns || (info == SEND_SIG_PRIV) || (info == SEND_SIG_FORCED)))
goto ret;
pending = group ? &t->signal->shared_pending : &t->pending;
@@ -1392,6 +1392,10 @@ static int kill_something_info(int sig, struct siginfo *info, pid_t pid)
return ret;
}
+ /* -INT_MIN is undefined. Exclude this case to avoid a UBSAN warning */
+ if (pid == INT_MIN)
+ return -ESRCH;
+
read_lock(&tasklist_lock);
if (pid != -1) {
ret = __kill_pgrp_info(sig, info,
@@ -2495,6 +2499,13 @@ void __set_current_blocked(const sigset_t *newset)
{
struct task_struct *tsk = current;
+ /*
+ * In case the signal mask hasn't changed, there is nothing we need
+ * to do. The current->blocked shouldn't be modified by other task.
+ */
+ if (sigequalsets(&tsk->blocked, newset))
+ return;
+
spin_lock_irq(&tsk->sighand->siglock);
__set_task_blocked(tsk, newset);
spin_unlock_irq(&tsk->sighand->siglock);
@@ -3105,7 +3116,8 @@ int do_sigaction(int sig, struct k_sigaction *act, struct k_sigaction *oact)
}
static int
-do_sigaltstack (const stack_t __user *uss, stack_t __user *uoss, unsigned long sp)
+do_sigaltstack (const stack_t __user *uss, stack_t __user *uoss, unsigned long sp,
+ size_t min_ss_size)
{
stack_t oss;
int error;
@@ -3144,9 +3156,8 @@ do_sigaltstack (const stack_t __user *uss, stack_t __user *uoss, unsigned long s
ss_size = 0;
ss_sp = NULL;
} else {
- error = -ENOMEM;
- if (ss_size < MINSIGSTKSZ)
- goto out;
+ if (unlikely(ss_size < min_ss_size))
+ return -ENOMEM;
}
current->sas_ss_sp = (unsigned long) ss_sp;
@@ -3169,12 +3180,14 @@ do_sigaltstack (const stack_t __user *uss, stack_t __user *uoss, unsigned long s
}
SYSCALL_DEFINE2(sigaltstack,const stack_t __user *,uss, stack_t __user *,uoss)
{
- return do_sigaltstack(uss, uoss, current_user_stack_pointer());
+ return do_sigaltstack(uss, uoss, current_user_stack_pointer(),
+ MINSIGSTKSZ);
}
int restore_altstack(const stack_t __user *uss)
{
- int err = do_sigaltstack(uss, NULL, current_user_stack_pointer());
+ int err = do_sigaltstack(uss, NULL, current_user_stack_pointer(),
+ MINSIGSTKSZ);
/* squash all but EFAULT for now */
return err == -EFAULT ? err : 0;
}
@@ -3215,7 +3228,8 @@ COMPAT_SYSCALL_DEFINE2(sigaltstack,
set_fs(KERNEL_DS);
ret = do_sigaltstack((stack_t __force __user *) (uss_ptr ? &uss : NULL),
(stack_t __force __user *) &uoss,
- compat_user_stack_pointer());
+ compat_user_stack_pointer(),
+ COMPAT_MINSIGSTKSZ);
set_fs(seg);
if (ret >= 0 && uoss_ptr) {
if (!access_ok(VERIFY_WRITE, uoss_ptr, sizeof(compat_stack_t)) ||
diff --git a/kernel/smp.c b/kernel/smp.c
index 313d9a8..d49e5df 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -597,6 +597,8 @@ void __init smp_init(void)
free_boot_cpu_mask();
+ /* Final decision about SMT support */
+ cpu_smt_check_topology();
/* Any cleanup work */
smp_announce();
smp_cpus_done(setup_max_cpus);
diff --git a/kernel/softirq.c b/kernel/softirq.c
index b593317..ef66c185 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -524,7 +524,9 @@ static __latent_entropy void tasklet_action(struct softirq_action *a)
if (!test_and_clear_bit(TASKLET_STATE_SCHED,
&t->state))
BUG();
+ trace_tasklet_entry(t);
t->func(t->data);
+ trace_tasklet_exit(t);
tasklet_unlock(t);
continue;
}
@@ -560,7 +562,9 @@ static __latent_entropy void tasklet_hi_action(struct softirq_action *a)
if (!test_and_clear_bit(TASKLET_STATE_SCHED,
&t->state))
BUG();
+ trace_tasklet_entry(t);
t->func(t->data);
+ trace_tasklet_exit(t);
tasklet_unlock(t);
continue;
}
diff --git a/kernel/stop_machine.c b/kernel/stop_machine.c
index ec9ab2f..9b8cd7e 100644
--- a/kernel/stop_machine.c
+++ b/kernel/stop_machine.c
@@ -36,7 +36,7 @@ struct cpu_stop_done {
struct cpu_stopper {
struct task_struct *thread;
- spinlock_t lock;
+ raw_spinlock_t lock;
bool enabled; /* is this stopper enabled? */
struct list_head works; /* list of pending works */
@@ -78,13 +78,13 @@ static bool cpu_stop_queue_work(unsigned int cpu, struct cpu_stop_work *work)
unsigned long flags;
bool enabled;
- spin_lock_irqsave(&stopper->lock, flags);
+ raw_spin_lock_irqsave(&stopper->lock, flags);
enabled = stopper->enabled;
if (enabled)
__cpu_stop_queue_work(stopper, work);
else if (work->done)
cpu_stop_signal_done(work->done);
- spin_unlock_irqrestore(&stopper->lock, flags);
+ raw_spin_unlock_irqrestore(&stopper->lock, flags);
return enabled;
}
@@ -231,8 +231,8 @@ static int cpu_stop_queue_two_works(int cpu1, struct cpu_stop_work *work1,
struct cpu_stopper *stopper2 = per_cpu_ptr(&cpu_stopper, cpu2);
int err;
retry:
- spin_lock_irq(&stopper1->lock);
- spin_lock_nested(&stopper2->lock, SINGLE_DEPTH_NESTING);
+ raw_spin_lock_irq(&stopper1->lock);
+ raw_spin_lock_nested(&stopper2->lock, SINGLE_DEPTH_NESTING);
err = -ENOENT;
if (!stopper1->enabled || !stopper2->enabled)
@@ -255,8 +255,8 @@ static int cpu_stop_queue_two_works(int cpu1, struct cpu_stop_work *work1,
__cpu_stop_queue_work(stopper1, work1);
__cpu_stop_queue_work(stopper2, work2);
unlock:
- spin_unlock(&stopper2->lock);
- spin_unlock_irq(&stopper1->lock);
+ raw_spin_unlock(&stopper2->lock);
+ raw_spin_unlock_irq(&stopper1->lock);
if (unlikely(err == -EDEADLK)) {
while (stop_cpus_in_progress)
@@ -448,9 +448,9 @@ static int cpu_stop_should_run(unsigned int cpu)
unsigned long flags;
int run;
- spin_lock_irqsave(&stopper->lock, flags);
+ raw_spin_lock_irqsave(&stopper->lock, flags);
run = !list_empty(&stopper->works);
- spin_unlock_irqrestore(&stopper->lock, flags);
+ raw_spin_unlock_irqrestore(&stopper->lock, flags);
return run;
}
@@ -461,13 +461,13 @@ static void cpu_stopper_thread(unsigned int cpu)
repeat:
work = NULL;
- spin_lock_irq(&stopper->lock);
+ raw_spin_lock_irq(&stopper->lock);
if (!list_empty(&stopper->works)) {
work = list_first_entry(&stopper->works,
struct cpu_stop_work, list);
list_del_init(&work->list);
}
- spin_unlock_irq(&stopper->lock);
+ raw_spin_unlock_irq(&stopper->lock);
if (work) {
cpu_stop_fn_t fn = work->fn;
@@ -541,7 +541,7 @@ static int __init cpu_stop_init(void)
for_each_possible_cpu(cpu) {
struct cpu_stopper *stopper = &per_cpu(cpu_stopper, cpu);
- spin_lock_init(&stopper->lock);
+ raw_spin_lock_init(&stopper->lock);
INIT_LIST_HEAD(&stopper->works);
}
diff --git a/kernel/sys.c b/kernel/sys.c
index 4ccf5f0..e1c2460 100644
--- a/kernel/sys.c
+++ b/kernel/sys.c
@@ -51,10 +51,13 @@
#include <linux/binfmts.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/rcupdate.h>
#include <linux/uidgid.h>
#include <linux/cred.h>
+#include <linux/nospec.h>
+
#include <linux/kmsg_dump.h>
/* Move somewhere else to avoid recompiling? */
#include <generated/utsrelease.h>
@@ -1142,18 +1145,19 @@ static int override_release(char __user *release, size_t len)
SYSCALL_DEFINE1(newuname, struct new_utsname __user *, name)
{
- int errno = 0;
+ struct new_utsname tmp;
down_read(&uts_sem);
- if (copy_to_user(name, utsname(), sizeof *name))
- errno = -EFAULT;
+ memcpy(&tmp, utsname(), sizeof(tmp));
up_read(&uts_sem);
+ if (copy_to_user(name, &tmp, sizeof(tmp)))
+ return -EFAULT;
- if (!errno && override_release(name->release, sizeof(name->release)))
- errno = -EFAULT;
- if (!errno && override_architecture(name))
- errno = -EFAULT;
- return errno;
+ if (override_release(name->release, sizeof(name->release)))
+ return -EFAULT;
+ if (override_architecture(name))
+ return -EFAULT;
+ return 0;
}
#ifdef __ARCH_WANT_SYS_OLD_UNAME
@@ -1162,55 +1166,46 @@ SYSCALL_DEFINE1(newuname, struct new_utsname __user *, name)
*/
SYSCALL_DEFINE1(uname, struct old_utsname __user *, name)
{
- int error = 0;
+ struct old_utsname tmp;
if (!name)
return -EFAULT;
down_read(&uts_sem);
- if (copy_to_user(name, utsname(), sizeof(*name)))
- error = -EFAULT;
+ memcpy(&tmp, utsname(), sizeof(tmp));
up_read(&uts_sem);
+ if (copy_to_user(name, &tmp, sizeof(tmp)))
+ return -EFAULT;
- if (!error && override_release(name->release, sizeof(name->release)))
- error = -EFAULT;
- if (!error && override_architecture(name))
- error = -EFAULT;
- return error;
+ if (override_release(name->release, sizeof(name->release)))
+ return -EFAULT;
+ if (override_architecture(name))
+ return -EFAULT;
+ return 0;
}
SYSCALL_DEFINE1(olduname, struct oldold_utsname __user *, name)
{
- int error;
+ struct oldold_utsname tmp = {};
if (!name)
return -EFAULT;
- if (!access_ok(VERIFY_WRITE, name, sizeof(struct oldold_utsname)))
- return -EFAULT;
down_read(&uts_sem);
- error = __copy_to_user(&name->sysname, &utsname()->sysname,
- __OLD_UTS_LEN);
- error |= __put_user(0, name->sysname + __OLD_UTS_LEN);
- error |= __copy_to_user(&name->nodename, &utsname()->nodename,
- __OLD_UTS_LEN);
- error |= __put_user(0, name->nodename + __OLD_UTS_LEN);
- error |= __copy_to_user(&name->release, &utsname()->release,
- __OLD_UTS_LEN);
- error |= __put_user(0, name->release + __OLD_UTS_LEN);
- error |= __copy_to_user(&name->version, &utsname()->version,
- __OLD_UTS_LEN);
- error |= __put_user(0, name->version + __OLD_UTS_LEN);
- error |= __copy_to_user(&name->machine, &utsname()->machine,
- __OLD_UTS_LEN);
- error |= __put_user(0, name->machine + __OLD_UTS_LEN);
+ memcpy(&tmp.sysname, &utsname()->sysname, __OLD_UTS_LEN);
+ memcpy(&tmp.nodename, &utsname()->nodename, __OLD_UTS_LEN);
+ memcpy(&tmp.release, &utsname()->release, __OLD_UTS_LEN);
+ memcpy(&tmp.version, &utsname()->version, __OLD_UTS_LEN);
+ memcpy(&tmp.machine, &utsname()->machine, __OLD_UTS_LEN);
up_read(&uts_sem);
+ if (copy_to_user(name, &tmp, sizeof(tmp)))
+ return -EFAULT;
- if (!error && override_architecture(name))
- error = -EFAULT;
- if (!error && override_release(name->release, sizeof(name->release)))
- error = -EFAULT;
- return error ? -EFAULT : 0;
+ if (override_architecture(name))
+ return -EFAULT;
+ if (override_release(name->release, sizeof(name->release)))
+ return -EFAULT;
+ return 0;
}
#endif
@@ -1224,17 +1219,18 @@ SYSCALL_DEFINE2(sethostname, char __user *, name, int, len)
if (len < 0 || len > __NEW_UTS_LEN)
return -EINVAL;
- down_write(&uts_sem);
errno = -EFAULT;
if (!copy_from_user(tmp, name, len)) {
- struct new_utsname *u = utsname();
+ struct new_utsname *u;
+ down_write(&uts_sem);
+ u = utsname();
memcpy(u->nodename, tmp, len);
memset(u->nodename + len, 0, sizeof(u->nodename) - len);
errno = 0;
uts_proc_notify(UTS_PROC_HOSTNAME);
+ up_write(&uts_sem);
}
- up_write(&uts_sem);
return errno;
}
@@ -1242,8 +1238,9 @@ SYSCALL_DEFINE2(sethostname, char __user *, name, int, len)
SYSCALL_DEFINE2(gethostname, char __user *, name, int, len)
{
- int i, errno;
+ int i;
struct new_utsname *u;
+ char tmp[__NEW_UTS_LEN + 1];
if (len < 0)
return -EINVAL;
@@ -1252,11 +1249,11 @@ SYSCALL_DEFINE2(gethostname, char __user *, name, int, len)
i = 1 + strlen(u->nodename);
if (i > len)
i = len;
- errno = 0;
- if (copy_to_user(name, u->nodename, i))
- errno = -EFAULT;
+ memcpy(tmp, u->nodename, i);
up_read(&uts_sem);
- return errno;
+ if (copy_to_user(name, tmp, i))
+ return -EFAULT;
+ return 0;
}
#endif
@@ -1275,17 +1272,18 @@ SYSCALL_DEFINE2(setdomainname, char __user *, name, int, len)
if (len < 0 || len > __NEW_UTS_LEN)
return -EINVAL;
- down_write(&uts_sem);
errno = -EFAULT;
if (!copy_from_user(tmp, name, len)) {
- struct new_utsname *u = utsname();
+ struct new_utsname *u;
+ down_write(&uts_sem);
+ u = utsname();
memcpy(u->domainname, tmp, len);
memset(u->domainname + len, 0, sizeof(u->domainname) - len);
errno = 0;
uts_proc_notify(UTS_PROC_DOMAINNAME);
+ up_write(&uts_sem);
}
- up_write(&uts_sem);
return errno;
}
@@ -1313,6 +1311,7 @@ SYSCALL_DEFINE2(old_getrlimit, unsigned int, resource,
if (resource >= RLIM_NLIMITS)
return -EINVAL;
+ resource = array_index_nospec(resource, RLIM_NLIMITS);
task_lock(current->group_leader);
x = current->signal->rlim[resource];
task_unlock(current->group_leader);
@@ -2074,6 +2073,17 @@ static int prctl_get_tid_address(struct task_struct *me, int __user **tid_addr)
}
#endif
+int __weak arch_prctl_spec_ctrl_get(struct task_struct *t, unsigned long which)
+{
+ return -EINVAL;
+}
+
+int __weak arch_prctl_spec_ctrl_set(struct task_struct *t, unsigned long which,
+ unsigned long ctrl)
+{
+ return -EINVAL;
+}
+
#ifdef CONFIG_MMU
static int prctl_update_vma_anon_name(struct vm_area_struct *vma,
struct vm_area_struct **prev,
@@ -2440,6 +2450,16 @@ SYSCALL_DEFINE5(prctl, int, option, unsigned long, arg2, unsigned long, arg3,
case PR_GET_FP_MODE:
error = GET_FP_MODE(me);
break;
+ case PR_GET_SPECULATION_CTRL:
+ if (arg3 || arg4 || arg5)
+ return -EINVAL;
+ error = arch_prctl_spec_ctrl_get(me, arg2);
+ break;
+ case PR_SET_SPECULATION_CTRL:
+ if (arg4 || arg5)
+ return -EINVAL;
+ error = arch_prctl_spec_ctrl_set(me, arg2, arg3);
+ break;
case PR_SET_VMA:
error = prctl_set_vma(arg2, arg3, arg4, arg5);
break;
diff --git a/kernel/sysctl.c b/kernel/sysctl.c
index f41c0e9..1f6800c 100644
--- a/kernel/sysctl.c
+++ b/kernel/sysctl.c
@@ -1975,6 +1975,24 @@ static struct ctl_table fs_table[] = {
.extra2 = &one,
},
{
+ .procname = "protected_fifos",
+ .data = &sysctl_protected_fifos,
+ .maxlen = sizeof(int),
+ .mode = 0600,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = &zero,
+ .extra2 = &two,
+ },
+ {
+ .procname = "protected_regular",
+ .data = &sysctl_protected_regular,
+ .maxlen = sizeof(int),
+ .mode = 0600,
+ .proc_handler = proc_dointvec_minmax,
+ .extra1 = &zero,
+ .extra2 = &two,
+ },
+ {
.procname = "suid_dumpable",
.data = &suid_dumpable,
.maxlen = sizeof(int),
diff --git a/kernel/time/alarmtimer.c b/kernel/time/alarmtimer.c
index 78c0e04..ea27ddf 100644
--- a/kernel/time/alarmtimer.c
+++ b/kernel/time/alarmtimer.c
@@ -15,6 +15,7 @@
* it under the terms of the GNU General Public License version 2 as
* published by the Free Software Foundation.
*/
+#include <linux/delay.h>
#include <linux/time.h>
#include <linux/hrtimer.h>
#include <linux/timerqueue.h>
@@ -436,7 +437,7 @@ int alarm_cancel(struct alarm *alarm)
int ret = alarm_try_to_cancel(alarm);
if (ret >= 0)
return ret;
- cpu_relax();
+ udelay(1);
}
}
EXPORT_SYMBOL_GPL(alarm_cancel);
@@ -815,7 +816,8 @@ static int alarm_timer_nsleep(const clockid_t which_clock, int flags,
/* Convert (if necessary) to absolute time */
if (flags != TIMER_ABSTIME) {
ktime_t now = alarm_bases[type].gettime();
- exp = ktime_add(now, exp);
+
+ exp = ktime_add_safe(now, exp);
}
if (alarmtimer_do_nsleep(&alarm, exp))
diff --git a/kernel/time/hrtimer.c b/kernel/time/hrtimer.c
index 11e4af2f..b44f5ce 100644
--- a/kernel/time/hrtimer.c
+++ b/kernel/time/hrtimer.c
@@ -32,6 +32,7 @@
*/
#include <linux/cpu.h>
+#include <linux/delay.h>
#include <linux/export.h>
#include <linux/percpu.h>
#include <linux/hrtimer.h>
@@ -1042,7 +1043,7 @@ int hrtimer_cancel(struct hrtimer *timer)
if (ret >= 0)
return ret;
- cpu_relax();
+ udelay(1);
}
}
EXPORT_SYMBOL_GPL(hrtimer_cancel);
diff --git a/kernel/time/posix-cpu-timers.c b/kernel/time/posix-cpu-timers.c
index ad538fe..be1cc61 100644
--- a/kernel/time/posix-cpu-timers.c
+++ b/kernel/time/posix-cpu-timers.c
@@ -103,7 +103,7 @@ static void bump_cpu_timer(struct k_itimer *timer,
continue;
timer->it.cpu.expires += incr;
- timer->it_overrun += 1 << i;
+ timer->it_overrun += 1LL << i;
delta -= incr;
}
}
diff --git a/kernel/time/posix-timers.c b/kernel/time/posix-timers.c
index fc7c37a..0e6ed2e 100644
--- a/kernel/time/posix-timers.c
+++ b/kernel/time/posix-timers.c
@@ -355,6 +355,17 @@ static __init int init_posix_timers(void)
__initcall(init_posix_timers);
+/*
+ * The siginfo si_overrun field and the return value of timer_getoverrun(2)
+ * are of type int. Clamp the overrun value to INT_MAX
+ */
+static inline int timer_overrun_to_int(struct k_itimer *timr, int baseval)
+{
+ s64 sum = timr->it_overrun_last + (s64)baseval;
+
+ return sum > (s64)INT_MAX ? INT_MAX : (int)sum;
+}
+
static void schedule_next_timer(struct k_itimer *timr)
{
struct hrtimer *timer = &timr->it.real.timer;
@@ -362,12 +373,11 @@ static void schedule_next_timer(struct k_itimer *timr)
if (timr->it.real.interval.tv64 == 0)
return;
- timr->it_overrun += (unsigned int) hrtimer_forward(timer,
- timer->base->get_time(),
- timr->it.real.interval);
+ timr->it_overrun += hrtimer_forward(timer, timer->base->get_time(),
+ timr->it.real.interval);
timr->it_overrun_last = timr->it_overrun;
- timr->it_overrun = -1;
+ timr->it_overrun = -1LL;
++timr->it_requeue_pending;
hrtimer_restart(timer);
}
@@ -396,7 +406,7 @@ void do_schedule_next_timer(struct siginfo *info)
else
schedule_next_timer(timr);
- info->si_overrun += timr->it_overrun_last;
+ info->si_overrun = timer_overrun_to_int(timr, info->si_overrun);
}
if (timr)
@@ -491,8 +501,7 @@ static enum hrtimer_restart posix_timer_fn(struct hrtimer *timer)
now = ktime_add(now, kj);
}
#endif
- timr->it_overrun += (unsigned int)
- hrtimer_forward(timer, now,
+ timr->it_overrun += hrtimer_forward(timer, now,
timr->it.real.interval);
ret = HRTIMER_RESTART;
++timr->it_requeue_pending;
@@ -633,7 +642,7 @@ SYSCALL_DEFINE3(timer_create, const clockid_t, which_clock,
it_id_set = IT_ID_SET;
new_timer->it_id = (timer_t) new_timer_id;
new_timer->it_clock = which_clock;
- new_timer->it_overrun = -1;
+ new_timer->it_overrun = -1LL;
if (timer_event_spec) {
if (copy_from_user(&event, timer_event_spec, sizeof (event))) {
@@ -762,7 +771,7 @@ common_timer_get(struct k_itimer *timr, struct itimerspec *cur_setting)
*/
if (iv.tv64 && (timr->it_requeue_pending & REQUEUE_PENDING ||
timr->it_sigev_notify == SIGEV_NONE))
- timr->it_overrun += (unsigned int) hrtimer_forward(timer, now, iv);
+ timr->it_overrun += hrtimer_forward(timer, now, iv);
remaining = __hrtimer_expires_remaining_adjusted(timer, now);
/* Return 0 only, when the timer is expired and not pending */
@@ -824,7 +833,7 @@ SYSCALL_DEFINE1(timer_getoverrun, timer_t, timer_id)
if (!timr)
return -EINVAL;
- overrun = timr->it_overrun_last;
+ overrun = timer_overrun_to_int(timr, 0);
unlock_timer(timr, flags);
return overrun;
diff --git a/kernel/time/tick-broadcast.c b/kernel/time/tick-broadcast.c
index d2a20e8..22d7454 100644
--- a/kernel/time/tick-broadcast.c
+++ b/kernel/time/tick-broadcast.c
@@ -610,6 +610,14 @@ static void tick_handle_oneshot_broadcast(struct clock_event_device *dev)
now = ktime_get();
/* Find all expired events */
for_each_cpu(cpu, tick_broadcast_oneshot_mask) {
+ /*
+ * Required for !SMP because for_each_cpu() reports
+ * unconditionally CPU0 as set on UP kernels.
+ */
+ if (!IS_ENABLED(CONFIG_SMP) &&
+ cpumask_empty(tick_broadcast_oneshot_mask))
+ break;
+
td = &per_cpu(tick_cpu_device, cpu);
if (td->evtdev->next_event.tv64 <= now.tv64) {
cpumask_set_cpu(cpu, tmpmask);
diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 4ce4285..8f482de 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -686,7 +686,7 @@ static void tick_nohz_restart(struct tick_sched *ts, ktime_t now)
static inline bool local_timer_softirq_pending(void)
{
- return local_softirq_pending() & TIMER_SOFTIRQ;
+ return local_softirq_pending() & BIT(TIMER_SOFTIRQ);
}
static ktime_t tick_nohz_stop_sched_tick(struct tick_sched *ts,
diff --git a/kernel/time/time.c b/kernel/time/time.c
index 1b2d209..9e5ffd1 100644
--- a/kernel/time/time.c
+++ b/kernel/time/time.c
@@ -28,6 +28,7 @@
*/
#include <linux/export.h>
+#include <linux/kernel.h>
#include <linux/timex.h>
#include <linux/capability.h>
#include <linux/timekeeper_internal.h>
@@ -258,9 +259,10 @@ unsigned int jiffies_to_msecs(const unsigned long j)
return (j + (HZ / MSEC_PER_SEC) - 1)/(HZ / MSEC_PER_SEC);
#else
# if BITS_PER_LONG == 32
- return (HZ_TO_MSEC_MUL32 * j) >> HZ_TO_MSEC_SHR32;
+ return (HZ_TO_MSEC_MUL32 * j + (1ULL << HZ_TO_MSEC_SHR32) - 1) >>
+ HZ_TO_MSEC_SHR32;
# else
- return (j * HZ_TO_MSEC_NUM) / HZ_TO_MSEC_DEN;
+ return DIV_ROUND_UP(j * HZ_TO_MSEC_NUM, HZ_TO_MSEC_DEN);
# endif
#endif
}
@@ -702,6 +704,16 @@ u64 nsec_to_clock_t(u64 x)
#endif
}
+u64 jiffies64_to_nsecs(u64 j)
+{
+#if !(NSEC_PER_SEC % HZ)
+ return (NSEC_PER_SEC / HZ) * j;
+# else
+ return div_u64(j * HZ_TO_NSEC_NUM, HZ_TO_NSEC_DEN);
+#endif
+}
+EXPORT_SYMBOL(jiffies64_to_nsecs);
+
/**
* nsecs_to_jiffies64 - Convert nsecs in u64 to jiffies64
*
diff --git a/kernel/time/timeconst.bc b/kernel/time/timeconst.bc
index c486889..f83bbb8 100644
--- a/kernel/time/timeconst.bc
+++ b/kernel/time/timeconst.bc
@@ -98,6 +98,12 @@
print "#define HZ_TO_USEC_DEN\t\t", hz/cd, "\n"
print "#define USEC_TO_HZ_NUM\t\t", hz/cd, "\n"
print "#define USEC_TO_HZ_DEN\t\t", 1000000/cd, "\n"
+
+ cd=gcd(hz,1000000000)
+ print "#define HZ_TO_NSEC_NUM\t\t", 1000000000/cd, "\n"
+ print "#define HZ_TO_NSEC_DEN\t\t", hz/cd, "\n"
+ print "#define NSEC_TO_HZ_NUM\t\t", hz/cd, "\n"
+ print "#define NSEC_TO_HZ_DEN\t\t", 1000000000/cd, "\n"
print "\n"
print "#endif /* KERNEL_TIMECONST_H */\n"
diff --git a/kernel/time/timekeeping.c b/kernel/time/timekeeping.c
index 5659b40..e6f4ef3 100644
--- a/kernel/time/timekeeping.c
+++ b/kernel/time/timekeeping.c
@@ -15,6 +15,7 @@
#include <linux/init.h>
#include <linux/mm.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/syscore_ops.h>
#include <linux/clocksource.h>
#include <linux/jiffies.h>
diff --git a/kernel/time/timer.c b/kernel/time/timer.c
index d397432..7cc0d6b 100644
--- a/kernel/time/timer.c
+++ b/kernel/time/timer.c
@@ -1263,7 +1263,7 @@ int del_timer_sync(struct timer_list *timer)
int ret = try_to_del_timer_sync(timer);
if (ret >= 0)
return ret;
- cpu_relax();
+ udelay(1);
}
}
EXPORT_SYMBOL(del_timer_sync);
@@ -1657,6 +1657,22 @@ static inline void __run_timers(struct timer_base *base)
spin_lock_irq(&base->lock);
+ /*
+ * timer_base::must_forward_clk must be cleared before running
+ * timers so that any timer functions that call mod_timer() will
+ * not try to forward the base. Idle tracking / clock forwarding
+ * logic is only used with BASE_STD timers.
+ *
+ * The must_forward_clk flag is cleared unconditionally also for
+ * the deferrable base. The deferrable base is not affected by idle
+ * tracking and never forwarded, so clearing the flag is a NOOP.
+ *
+ * The fact that the deferrable base is never forwarded can cause
+ * large variations in granularity for deferrable timers, but they
+ * can be deferred for long periods due to idle anyway.
+ */
+ base->must_forward_clk = false;
+
while (time_after_eq(jiffies, base->clk)) {
levels = collect_expired_timers(base, heads);
@@ -1676,19 +1692,6 @@ static __latent_entropy void run_timer_softirq(struct softirq_action *h)
{
struct timer_base *base = this_cpu_ptr(&timer_bases[BASE_STD]);
- /*
- * must_forward_clk must be cleared before running timers so that any
- * timer functions that call mod_timer will not try to forward the
- * base. idle trcking / clock forwarding logic is only used with
- * BASE_STD timers.
- *
- * The deferrable base does not do idle tracking at all, so we do
- * not forward it. This can result in very large variations in
- * granularity for deferrable timers, but they can be deferred for
- * long periods due to idle.
- */
- base->must_forward_clk = false;
-
__run_timers(base);
if (IS_ENABLED(CONFIG_NO_HZ_COMMON))
__run_timers(this_cpu_ptr(&timer_bases[BASE_DEF]));
diff --git a/kernel/time/timer_list.c b/kernel/time/timer_list.c
index 998d730..d38f8dd 100644
--- a/kernel/time/timer_list.c
+++ b/kernel/time/timer_list.c
@@ -389,7 +389,7 @@ static int __init init_timer_list_procfs(void)
{
struct proc_dir_entry *pe;
- pe = proc_create("timer_list", 0444, NULL, &timer_list_fops);
+ pe = proc_create("timer_list", 0400, NULL, &timer_list_fops);
if (!pe)
return -ENOMEM;
return 0;
diff --git a/kernel/trace/Kconfig b/kernel/trace/Kconfig
index 1f89ca0..2453485 100644
--- a/kernel/trace/Kconfig
+++ b/kernel/trace/Kconfig
@@ -179,6 +179,7 @@
depends on HAVE_FUNCTION_GRAPH_TRACER
depends on FUNCTION_TRACER
depends on !X86_32 || !CC_OPTIMIZE_FOR_SIZE
+ depends on ROP_PROTECTION_NONE
default y
help
Enable the kernel to trace a function at both its return
diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index 4e17d55..bfa8bb3 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -1720,6 +1720,10 @@ static ssize_t sysfs_blk_trace_attr_store(struct device *dev,
mutex_lock(&bdev->bd_mutex);
if (attr == &dev_attr_enable) {
+ if (!!value == !!q->blk_trace) {
+ ret = 0;
+ goto out_unlock_bdev;
+ }
if (value)
ret = blk_trace_setup_queue(q, bdev);
else
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index 41805fb..786009f 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -131,7 +131,7 @@ BPF_CALL_5(bpf_trace_printk, char *, fmt, u32, fmt_size, u64, arg1,
bool str_seen = false;
int mod[3] = {};
int fmt_cnt = 0;
- u64 unsafe_addr;
+ u64 unsafe_addr = 0;
char buf[64];
int i;
@@ -161,11 +161,13 @@ BPF_CALL_5(bpf_trace_printk, char *, fmt, u32, fmt_size, u64, arg1,
i++;
} else if (fmt[i] == 'p' || fmt[i] == 's') {
mod[fmt_cnt]++;
- i++;
- if (!isspace(fmt[i]) && !ispunct(fmt[i]) && fmt[i] != 0)
+ /* disallow any further format extensions */
+ if (fmt[i + 1] != 0 &&
+ !isspace(fmt[i + 1]) &&
+ !ispunct(fmt[i + 1]))
return -EINVAL;
fmt_cnt++;
- if (fmt[i - 1] == 's') {
+ if (fmt[i] == 's') {
if (str_seen)
/* allow only one '%s' per fmt string */
return -EINVAL;
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 063dd22..ae6bdf3 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -120,8 +120,9 @@ static void ftrace_ops_list_func(unsigned long ip, unsigned long parent_ip,
struct ftrace_ops *op, struct pt_regs *regs);
#else
/* See comment below, where ftrace_ops_list_func is defined */
-static void ftrace_ops_no_ops(unsigned long ip, unsigned long parent_ip);
-#define ftrace_ops_list_func ((ftrace_func_t)ftrace_ops_no_ops)
+static void ftrace_ops_no_ops(unsigned long ip, unsigned long parent_ip,
+ struct ftrace_ops *op, struct pt_regs *regs);
+#define ftrace_ops_list_func ftrace_ops_no_ops
#endif
/*
@@ -4836,6 +4837,7 @@ void ftrace_destroy_filter_files(struct ftrace_ops *ops)
if (ops->flags & FTRACE_OPS_FL_ENABLED)
ftrace_shutdown(ops, 0);
ops->flags |= FTRACE_OPS_FL_DELETED;
+ ftrace_free_filter(ops);
mutex_unlock(&ftrace_lock);
}
@@ -5309,7 +5311,8 @@ static void ftrace_ops_list_func(unsigned long ip, unsigned long parent_ip,
__ftrace_ops_list_func(ip, parent_ip, NULL, regs);
}
#else
-static void ftrace_ops_no_ops(unsigned long ip, unsigned long parent_ip)
+static void ftrace_ops_no_ops(unsigned long ip, unsigned long parent_ip,
+ struct ftrace_ops *op, struct pt_regs *regs)
{
__ftrace_ops_list_func(ip, parent_ip, NULL, NULL);
}
@@ -5735,14 +5738,17 @@ void ftrace_graph_graph_time_control(bool enable)
fgraph_graph_time = enable;
}
+void ftrace_graph_return_stub(struct ftrace_graph_ret *trace)
+{
+}
+
int ftrace_graph_entry_stub(struct ftrace_graph_ent *trace)
{
return 0;
}
/* The callbacks that hook a function */
-trace_func_graph_ret_t ftrace_graph_return =
- (trace_func_graph_ret_t)ftrace_stub;
+trace_func_graph_ret_t ftrace_graph_return = ftrace_graph_return_stub;
trace_func_graph_ent_t ftrace_graph_entry = ftrace_graph_entry_stub;
static trace_func_graph_ent_t __ftrace_graph_entry = ftrace_graph_entry_stub;
@@ -5970,7 +5976,7 @@ void unregister_ftrace_graph(void)
goto out;
ftrace_graph_active--;
- ftrace_graph_return = (trace_func_graph_ret_t)ftrace_stub;
+ ftrace_graph_return = ftrace_graph_return_stub;
ftrace_graph_entry = ftrace_graph_entry_stub;
__ftrace_graph_entry = ftrace_graph_entry_stub;
ftrace_shutdown(&graph_ops, FTRACE_STOP_FUNC_RET);
diff --git a/kernel/trace/ring_buffer.c b/kernel/trace/ring_buffer.c
index 3e1d11f..79a4d50 100644
--- a/kernel/trace/ring_buffer.c
+++ b/kernel/trace/ring_buffer.c
@@ -1137,15 +1137,12 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu)
struct buffer_page *bpage, *tmp;
long i;
+ /* Bail out before going to OOM */
+ current->flags |= PF_DUMPCORE;
for (i = 0; i < nr_pages; i++) {
struct page *page;
- /*
- * __GFP_NORETRY flag makes sure that the allocation fails
- * gracefully without invoking oom-killer and the system is
- * not destabilized.
- */
bpage = kzalloc_node(ALIGN(sizeof(*bpage), cache_line_size()),
- GFP_KERNEL | __GFP_NORETRY,
+ GFP_KERNEL,
cpu_to_node(cpu));
if (!bpage)
goto free_pages;
@@ -1153,16 +1150,18 @@ static int __rb_allocate_pages(long nr_pages, struct list_head *pages, int cpu)
list_add(&bpage->list, pages);
page = alloc_pages_node(cpu_to_node(cpu),
- GFP_KERNEL | __GFP_NORETRY, 0);
+ GFP_KERNEL, 0);
if (!page)
goto free_pages;
bpage->page = page_address(page);
rb_init_page(bpage->page);
}
+ current->flags &= ~PF_DUMPCORE;
return 0;
free_pages:
+ current->flags &= ~PF_DUMPCORE;
list_for_each_entry_safe(bpage, tmp, pages, list) {
list_del_init(&bpage->list);
free_buffer_page(bpage);
@@ -1504,6 +1503,8 @@ rb_remove_pages(struct ring_buffer_per_cpu *cpu_buffer, unsigned long nr_pages)
tmp_iter_page = first_page;
do {
+ cond_resched();
+
to_remove_page = tmp_iter_page;
rb_inc_page(cpu_buffer, &tmp_iter_page);
@@ -3137,6 +3138,22 @@ int ring_buffer_record_is_on(struct ring_buffer *buffer)
}
/**
+ * ring_buffer_record_is_set_on - return true if the ring buffer is set writable
+ * @buffer: The ring buffer to see if write is set enabled
+ *
+ * Returns true if the ring buffer is set writable by ring_buffer_record_on().
+ * Note that this does NOT mean it is in a writable state.
+ *
+ * It may return true when the ring buffer has been disabled by
+ * ring_buffer_record_disable(), as that is a temporary disabling of
+ * the ring buffer.
+ */
+int ring_buffer_record_is_set_on(struct ring_buffer *buffer)
+{
+ return !(atomic_read(&buffer->record_disabled) & RB_BUFFER_OFF);
+}
+
+/**
* ring_buffer_record_disable_cpu - stop all writes into the cpu_buffer
* @buffer: The ring buffer to stop writes to.
* @cpu: The CPU buffer to stop
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 046abb0..df29e06 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -1325,6 +1325,12 @@ update_max_tr(struct trace_array *tr, struct task_struct *tsk, int cpu)
arch_spin_lock(&tr->max_lock);
+ /* Inherit the recordable setting from trace_buffer */
+ if (ring_buffer_record_is_set_on(tr->trace_buffer.buffer))
+ ring_buffer_record_on(tr->max_buffer.buffer);
+ else
+ ring_buffer_record_off(tr->max_buffer.buffer);
+
buf = tr->trace_buffer.buffer;
tr->trace_buffer.buffer = tr->max_buffer.buffer;
tr->max_buffer.buffer = buf;
@@ -2566,6 +2572,7 @@ int trace_vbprintk(unsigned long ip, const char *fmt, va_list args)
}
EXPORT_SYMBOL_GPL(trace_vbprintk);
+__printf(3, 0)
static int
__trace_array_vprintk(struct ring_buffer *buffer,
unsigned long ip, const char *fmt, va_list args)
@@ -2621,12 +2628,14 @@ __trace_array_vprintk(struct ring_buffer *buffer,
return len;
}
+__printf(3, 0)
int trace_array_vprintk(struct trace_array *tr,
unsigned long ip, const char *fmt, va_list args)
{
return __trace_array_vprintk(tr->trace_buffer.buffer, ip, fmt, args);
}
+__printf(3, 0)
int trace_array_printk(struct trace_array *tr,
unsigned long ip, const char *fmt, ...)
{
@@ -2642,6 +2651,7 @@ int trace_array_printk(struct trace_array *tr,
return ret;
}
+__printf(3, 4)
int trace_array_printk_buf(struct ring_buffer *buffer,
unsigned long ip, const char *fmt, ...)
{
@@ -2657,6 +2667,7 @@ int trace_array_printk_buf(struct ring_buffer *buffer,
return ret;
}
+__printf(2, 0)
int trace_vprintk(unsigned long ip, const char *fmt, va_list args)
{
return trace_array_vprintk(&global_trace, ip, fmt, args);
@@ -3078,13 +3089,14 @@ static void test_cpu_buff_start(struct trace_iterator *iter)
if (!(iter->iter_flags & TRACE_FILE_ANNOTATE))
return;
- if (iter->started && cpumask_test_cpu(iter->cpu, iter->started))
+ if (cpumask_available(iter->started) &&
+ cpumask_test_cpu(iter->cpu, iter->started))
return;
if (per_cpu_ptr(iter->trace_buffer->data, iter->cpu)->skipped_entries)
return;
- if (iter->started)
+ if (cpumask_available(iter->started))
cpumask_set_cpu(iter->cpu, iter->started);
/* Don't print started cpu buffer for the first entry of the trace */
@@ -7065,7 +7077,9 @@ rb_simple_write(struct file *filp, const char __user *ubuf,
if (buffer) {
mutex_lock(&trace_types_lock);
- if (val) {
+ if (!!val == tracer_tracing_is_on(tr)) {
+ val = 0; /* do nothing */
+ } else if (val) {
tracer_tracing_on(tr);
if (tr->current_trace->start)
tr->current_trace->start(tr);
diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c
index 0193f58..e35a411 100644
--- a/kernel/trace/trace_events_filter.c
+++ b/kernel/trace/trace_events_filter.c
@@ -322,6 +322,9 @@ static int regex_match_full(char *str, struct regex *r, int len)
static int regex_match_front(char *str, struct regex *r, int len)
{
+ if (len < r->len)
+ return 0;
+
if (strncmp(str, r->pattern, r->len) == 0)
return 1;
return 0;
diff --git a/kernel/trace/trace_events_trigger.c b/kernel/trace/trace_events_trigger.c
index 6721a1e8..7e6971b 100644
--- a/kernel/trace/trace_events_trigger.c
+++ b/kernel/trace/trace_events_trigger.c
@@ -481,9 +481,10 @@ clear_event_triggers(struct trace_array *tr)
struct trace_event_file *file;
list_for_each_entry(file, &tr->events, list) {
- struct event_trigger_data *data;
- list_for_each_entry_rcu(data, &file->triggers, list) {
+ struct event_trigger_data *data, *n;
+ list_for_each_entry_safe(data, n, &file->triggers, list) {
trace_event_trigger_enable_disable(file, 0);
+ list_del_rcu(&data->list);
if (data->ops->free)
data->ops->free(data->ops, data);
}
@@ -677,6 +678,8 @@ event_trigger_callback(struct event_command *cmd_ops,
goto out_free;
out_reg:
+ /* Up the trigger_data count to make sure reg doesn't free it on failure */
+ event_trigger_init(trigger_ops, trigger_data);
ret = cmd_ops->reg(glob, trigger_ops, trigger_data, file);
/*
* The above returns on success the # of functions enabled,
@@ -684,11 +687,13 @@ event_trigger_callback(struct event_command *cmd_ops,
* Consider no functions a failure too.
*/
if (!ret) {
+ cmd_ops->unreg(glob, trigger_ops, trigger_data, file);
ret = -ENOENT;
- goto out_free;
- } else if (ret < 0)
- goto out_free;
- ret = 0;
+ } else if (ret > 0)
+ ret = 0;
+
+ /* Down the counter of trigger_data or free it if not used anymore */
+ event_trigger_free(trigger_ops, trigger_data);
out:
return ret;
@@ -737,8 +742,10 @@ int set_trigger_filter(char *filter_str,
/* The filter is for the 'trigger' event, not the triggered event */
ret = create_event_filter(file->event_call, filter_str, false, &filter);
- if (ret)
- goto out;
+ /*
+ * If create_event_filter() fails, filter still needs to be freed.
+ * Which the calling code will do with data->filter.
+ */
assign:
tmp = rcu_access_pointer(data->filter);
@@ -1384,6 +1391,9 @@ int event_enable_trigger_func(struct event_command *cmd_ops,
goto out;
}
+ /* Up the trigger_data count to make sure nothing frees it on failure */
+ event_trigger_init(trigger_ops, trigger_data);
+
if (trigger) {
number = strsep(&trigger, ":");
@@ -1434,6 +1444,7 @@ int event_enable_trigger_func(struct event_command *cmd_ops,
goto out_disable;
/* Just return zero, not the number of enabled functions */
ret = 0;
+ event_trigger_free(trigger_ops, trigger_data);
out:
return ret;
@@ -1444,7 +1455,7 @@ int event_enable_trigger_func(struct event_command *cmd_ops,
out_free:
if (cmd_ops->set_filter)
cmd_ops->set_filter(NULL, trigger_data, NULL);
- kfree(trigger_data);
+ event_trigger_free(trigger_ops, trigger_data);
kfree(enable_data);
goto out;
}
diff --git a/kernel/trace/trace_functions_graph.c b/kernel/trace/trace_functions_graph.c
index 3cb38f1..7461d51 100644
--- a/kernel/trace/trace_functions_graph.c
+++ b/kernel/trace/trace_functions_graph.c
@@ -835,6 +835,7 @@ print_graph_entry_leaf(struct trace_iterator *iter,
struct ftrace_graph_ret *graph_ret;
struct ftrace_graph_ent *call;
unsigned long long duration;
+ int cpu = iter->cpu;
int i;
graph_ret = &ret_entry->ret;
@@ -843,7 +844,6 @@ print_graph_entry_leaf(struct trace_iterator *iter,
if (data) {
struct fgraph_cpu_data *cpu_data;
- int cpu = iter->cpu;
cpu_data = per_cpu_ptr(data->cpu_data, cpu);
@@ -873,6 +873,9 @@ print_graph_entry_leaf(struct trace_iterator *iter,
trace_seq_printf(s, "%ps();\n", (void *)call->func);
+ print_graph_irq(iter, graph_ret->func, TRACE_GRAPH_RET,
+ cpu, iter->ent->pid, flags);
+
return trace_handle_return(s);
}
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index ea3ed03..3b4cd44 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -359,11 +359,10 @@ static struct trace_kprobe *find_trace_kprobe(const char *event,
static int
enable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file)
{
+ struct event_file_link *link = NULL;
int ret = 0;
if (file) {
- struct event_file_link *link;
-
link = kmalloc(sizeof(*link), GFP_KERNEL);
if (!link) {
ret = -ENOMEM;
@@ -383,6 +382,18 @@ enable_trace_kprobe(struct trace_kprobe *tk, struct trace_event_file *file)
else
ret = enable_kprobe(&tk->rp.kp);
}
+
+ if (ret) {
+ if (file) {
+ /* Notice the if is true on not WARN() */
+ if (!WARN_ON_ONCE(!link))
+ list_del_rcu(&link->list);
+ kfree(link);
+ tk->tp.flags &= ~TP_FLAG_TRACE;
+ } else {
+ tk->tp.flags &= ~TP_FLAG_PROFILE;
+ }
+ }
out:
return ret;
}
diff --git a/kernel/trace/trace_uprobe.c b/kernel/trace/trace_uprobe.c
index bc6c6ec5..92e394e 100644
--- a/kernel/trace/trace_uprobe.c
+++ b/kernel/trace/trace_uprobe.c
@@ -149,6 +149,8 @@ static void FETCH_FUNC_NAME(memory, string)(struct pt_regs *regs,
return;
ret = strncpy_from_user(dst, src, maxlen);
+ if (ret == maxlen)
+ dst[--ret] = '\0';
if (ret < 0) { /* Failed to fetch string */
((u8 *)get_rloc_data(dest))[0] = '\0';
@@ -967,7 +969,7 @@ probe_event_disable(struct trace_uprobe *tu, struct trace_event_file *file)
list_del_rcu(&link->list);
/* synchronize with u{,ret}probe_trace_func */
- synchronize_sched();
+ synchronize_rcu();
kfree(link);
if (!list_empty(&tu->tp.files))
diff --git a/kernel/tracepoint.c b/kernel/tracepoint.c
index d0639d9..c8e7cc0 100644
--- a/kernel/tracepoint.c
+++ b/kernel/tracepoint.c
@@ -202,7 +202,7 @@ static int tracepoint_add_func(struct tracepoint *tp,
lockdep_is_held(&tracepoints_mutex));
old = func_add(&tp_funcs, func, prio);
if (IS_ERR(old)) {
- WARN_ON_ONCE(1);
+ WARN_ON_ONCE(PTR_ERR(old) != -ENOMEM);
return PTR_ERR(old);
}
@@ -235,7 +235,7 @@ static int tracepoint_remove_func(struct tracepoint *tp,
lockdep_is_held(&tracepoints_mutex));
old = func_remove(&tp_funcs, func);
if (IS_ERR(old)) {
- WARN_ON_ONCE(1);
+ WARN_ON_ONCE(PTR_ERR(old) != -ENOMEM);
return PTR_ERR(old);
}
diff --git a/kernel/user_namespace.c b/kernel/user_namespace.c
index 86b7854..f789bbb 100644
--- a/kernel/user_namespace.c
+++ b/kernel/user_namespace.c
@@ -649,7 +649,16 @@ static ssize_t map_write(struct file *file, const char __user *buf,
unsigned idx;
struct uid_gid_extent *extent = NULL;
char *kbuf = NULL, *pos, *next_line;
- ssize_t ret = -EINVAL;
+ ssize_t ret;
+
+ /* Only allow < page size writes at the beginning of the file */
+ if ((*ppos != 0) || (count >= PAGE_SIZE))
+ return -EINVAL;
+
+ /* Slurp in the user data */
+ kbuf = memdup_user_nul(buf, count);
+ if (IS_ERR(kbuf))
+ return PTR_ERR(kbuf);
/*
* The userns_state_mutex serializes all writes to any given map.
@@ -683,19 +692,6 @@ static ssize_t map_write(struct file *file, const char __user *buf,
if (cap_valid(cap_setid) && !file_ns_capable(file, ns, CAP_SYS_ADMIN))
goto out;
- /* Only allow < page size writes at the beginning of the file */
- ret = -EINVAL;
- if ((*ppos != 0) || (count >= PAGE_SIZE))
- goto out;
-
- /* Slurp in the user data */
- kbuf = memdup_user_nul(buf, count);
- if (IS_ERR(kbuf)) {
- ret = PTR_ERR(kbuf);
- kbuf = NULL;
- goto out;
- }
-
/* Parse the user data */
ret = -EINVAL;
pos = kbuf;
diff --git a/kernel/utsname_sysctl.c b/kernel/utsname_sysctl.c
index c8eac43..d2b3b29 100644
--- a/kernel/utsname_sysctl.c
+++ b/kernel/utsname_sysctl.c
@@ -17,7 +17,7 @@
#ifdef CONFIG_PROC_SYSCTL
-static void *get_uts(struct ctl_table *table, int write)
+static void *get_uts(struct ctl_table *table)
{
char *which = table->data;
struct uts_namespace *uts_ns;
@@ -25,21 +25,9 @@ static void *get_uts(struct ctl_table *table, int write)
uts_ns = current->nsproxy->uts_ns;
which = (which - (char *)&init_uts_ns) + (char *)uts_ns;
- if (!write)
- down_read(&uts_sem);
- else
- down_write(&uts_sem);
return which;
}
-static void put_uts(struct ctl_table *table, int write, void *which)
-{
- if (!write)
- up_read(&uts_sem);
- else
- up_write(&uts_sem);
-}
-
/*
* Special case of dostring for the UTS structure. This has locks
* to observe. Should this be in kernel/sys.c ????
@@ -49,13 +37,34 @@ static int proc_do_uts_string(struct ctl_table *table, int write,
{
struct ctl_table uts_table;
int r;
- memcpy(&uts_table, table, sizeof(uts_table));
- uts_table.data = get_uts(table, write);
- r = proc_dostring(&uts_table, write, buffer, lenp, ppos);
- put_uts(table, write, uts_table.data);
+ char tmp_data[__NEW_UTS_LEN + 1];
- if (write)
+ memcpy(&uts_table, table, sizeof(uts_table));
+ uts_table.data = tmp_data;
+
+ /*
+ * Buffer the value in tmp_data so that proc_dostring() can be called
+ * without holding any locks.
+ * We also need to read the original value in the write==1 case to
+ * support partial writes.
+ */
+ down_read(&uts_sem);
+ memcpy(tmp_data, get_uts(table), sizeof(tmp_data));
+ up_read(&uts_sem);
+ r = proc_dostring(&uts_table, write, buffer, lenp, ppos);
+
+ if (write) {
+ /*
+ * Write back the new value.
+ * Note that, since we dropped uts_sem, the result can
+ * theoretically be incorrect if there are two parallel writes
+ * at non-zero offsets to the same sysctl.
+ */
+ down_write(&uts_sem);
+ memcpy(get_uts(table), tmp_data, sizeof(tmp_data));
+ up_write(&uts_sem);
proc_sys_poll_notify(table->poll);
+ }
return r;
}
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 04c2289..172fa22 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -293,6 +293,8 @@ module_param_named(disable_numa, wq_disable_numa, bool, 0444);
static bool wq_power_efficient = IS_ENABLED(CONFIG_WQ_POWER_EFFICIENT_DEFAULT);
module_param_named(power_efficient, wq_power_efficient, bool, 0444);
+bool wq_online; /* can kworkers be created yet? */
+
static bool wq_numa_enabled; /* unbound NUMA affinity enabled */
/* buf for wq_update_unbound_numa_attrs(), protected by CPU hotplug exclusion */
@@ -2586,6 +2588,9 @@ void flush_workqueue(struct workqueue_struct *wq)
};
int next_color;
+ if (WARN_ON(!wq_online))
+ return;
+
lock_map_acquire(&wq->lockdep_map);
lock_map_release(&wq->lockdep_map);
@@ -2846,6 +2851,9 @@ bool flush_work(struct work_struct *work)
{
struct wq_barrier barr;
+ if (WARN_ON(!wq_online))
+ return false;
+
lock_map_acquire(&work->lockdep_map);
lock_map_release(&work->lockdep_map);
@@ -2916,7 +2924,13 @@ static bool __cancel_work_timer(struct work_struct *work, bool is_dwork)
mark_work_canceling(work);
local_irq_restore(flags);
- flush_work(work);
+ /*
+ * This allows canceling during early boot. We know that @work
+ * isn't executing.
+ */
+ if (wq_online)
+ flush_work(work);
+
clear_work_data(work);
/*
@@ -3366,7 +3380,7 @@ static struct worker_pool *get_unbound_pool(const struct workqueue_attrs *attrs)
goto fail;
/* create and start the initial worker */
- if (!create_worker(pool))
+ if (wq_online && !create_worker(pool))
goto fail;
/* install */
@@ -3431,6 +3445,7 @@ static void pwq_adjust_max_active(struct pool_workqueue *pwq)
{
struct workqueue_struct *wq = pwq->wq;
bool freezable = wq->flags & WQ_FREEZABLE;
+ unsigned long flags;
/* for @wq->saved_max_active */
lockdep_assert_held(&wq->mutex);
@@ -3439,7 +3454,8 @@ static void pwq_adjust_max_active(struct pool_workqueue *pwq)
if (!freezable && pwq->max_active == wq->saved_max_active)
return;
- spin_lock_irq(&pwq->pool->lock);
+ /* this function can be called during early boot w/ irq disabled */
+ spin_lock_irqsave(&pwq->pool->lock, flags);
/*
* During [un]freezing, the caller is responsible for ensuring that
@@ -3462,7 +3478,7 @@ static void pwq_adjust_max_active(struct pool_workqueue *pwq)
pwq->max_active = 0;
}
- spin_unlock_irq(&pwq->pool->lock);
+ spin_unlock_irqrestore(&pwq->pool->lock, flags);
}
/* initialize newly alloced @pwq which is associated with @wq and @pool */
@@ -5276,7 +5292,7 @@ int workqueue_sysfs_register(struct workqueue_struct *wq)
ret = device_register(&wq_dev->dev);
if (ret) {
- kfree(wq_dev);
+ put_device(&wq_dev->dev);
wq->wq_dev = NULL;
return ret;
}
@@ -5509,7 +5525,17 @@ static void __init wq_numa_init(void)
wq_numa_enabled = true;
}
-static int __init init_workqueues(void)
+/**
+ * workqueue_init_early - early init for workqueue subsystem
+ *
+ * This is the first half of two-staged workqueue subsystem initialization
+ * and invoked as soon as the bare basics - memory allocation, cpumasks and
+ * idr are up. It sets up all the data structures and system workqueues
+ * and allows early boot code to create workqueues and queue/cancel work
+ * items. Actual work item execution starts only after kthreads can be
+ * created and scheduled right before early initcalls.
+ */
+int __init workqueue_init_early(void)
{
int std_nice[NR_STD_WORKER_POOLS] = { 0, HIGHPRI_NICE_LEVEL };
int i, cpu;
@@ -5542,16 +5568,6 @@ static int __init init_workqueues(void)
}
}
- /* create the initial worker */
- for_each_online_cpu(cpu) {
- struct worker_pool *pool;
-
- for_each_cpu_worker_pool(pool, cpu) {
- pool->flags &= ~POOL_DISASSOCIATED;
- BUG_ON(!create_worker(pool));
- }
- }
-
/* create default unbound and ordered wq attrs */
for (i = 0; i < NR_STD_WORKER_POOLS; i++) {
struct workqueue_attrs *attrs;
@@ -5588,8 +5604,36 @@ static int __init init_workqueues(void)
!system_power_efficient_wq ||
!system_freezable_power_efficient_wq);
+ return 0;
+}
+
+/**
+ * workqueue_init - bring workqueue subsystem fully online
+ *
+ * This is the latter half of two-staged workqueue subsystem initialization
+ * and invoked as soon as kthreads can be created and scheduled.
+ * Workqueues have been created and work items queued on them, but there
+ * are no kworkers executing the work items yet. Populate the worker pools
+ * with the initial workers and enable future kworker creations.
+ */
+int __init workqueue_init(void)
+{
+ struct worker_pool *pool;
+ int cpu, bkt;
+
+ /* create the initial workers */
+ for_each_online_cpu(cpu) {
+ for_each_cpu_worker_pool(pool, cpu) {
+ pool->flags &= ~POOL_DISASSOCIATED;
+ BUG_ON(!create_worker(pool));
+ }
+ }
+
+ hash_for_each(unbound_pool_hash, bkt, pool, hash_node)
+ BUG_ON(!create_worker(pool));
+
+ wq_online = true;
wq_watchdog_init();
return 0;
}
-early_initcall(init_workqueues);
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 079d91a..5b0e9fe 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -15,6 +15,22 @@
The behavior is also controlled by the kernel command line
parameter printk.time=1. See Documentation/kernel-parameters.txt
+config PRINTK_CPU_ID
+ bool "Show CPU number on printks"
+ default y
+ depends on PRINTK
+ help
+ Selecting this option adds CPU number to printk() messages.
+ It can be turned off by writing 0 to printk module parameter cpu.
+
+config PRINTK_PID
+ bool "Show process ID on printks"
+ default y
+ depends on PRINTK
+ help
+ Selecting this option adds process ID to printk() messages.
+ It can be turned off by writing 0 to printk module parameter pid.
+
config MESSAGE_LOGLEVEL_DEFAULT
int "Default message log level (1-7)"
range 1 7
@@ -2076,6 +2092,16 @@
recoverable data corruption scenarios to system-halting panics,
for easier detection and debug.
+config CC_WERROR
+ bool "Treat all compile warnings as errors"
+ default n
+ help
+ Select this option to set compiler warnings as errors,
+ to prevent easily-fixable problems from creeping into
+ the codebase.
+
+ If unsure, say N.
+
source "samples/Kconfig"
source "lib/Kconfig.kgdb"
diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan
index bd38aab..12c0683 100644
--- a/lib/Kconfig.kasan
+++ b/lib/Kconfig.kasan
@@ -43,6 +43,13 @@
make kernel's .text size much bigger.
This requires a gcc version of 5.0 or later.
+config KASAN_PANIC_ON_WARN
+ bool "Cause a Kernel Panic When KASan bug is detected"
+ help
+ Select this option to upgrade warnings for potentially memory
+ corruption detected form KASan to panics, for easier detection
+ and debug.
+
endchoice
config TEST_KASAN
diff --git a/lib/debug_locks.c b/lib/debug_locks.c
index 96c4c63..124fdf2 100644
--- a/lib/debug_locks.c
+++ b/lib/debug_locks.c
@@ -37,7 +37,7 @@ EXPORT_SYMBOL_GPL(debug_locks_silent);
*/
int debug_locks_off(void)
{
- if (__debug_locks_off()) {
+ if (debug_locks && __debug_locks_off()) {
if (!debug_locks_silent) {
console_verbose();
return 1;
diff --git a/lib/debugobjects.c b/lib/debugobjects.c
index 19572a4..1c43d4c 100644
--- a/lib/debugobjects.c
+++ b/lib/debugobjects.c
@@ -293,13 +293,13 @@ static void debug_object_is_on_stack(void *addr, int onstack)
return;
limit++;
- if (is_on_stack) {
- pr_warn("object %p is on stack %p, but NOT annotated\n", addr,
- task_stack_page(current));
- } else {
- pr_warn("object %p is NOT on stack %p, but annotated\n", addr,
- task_stack_page(current));
- }
+ if (is_on_stack)
+ pr_warn("object %p is on stack %p, but NOT annotated.\n", addr,
+ task_stack_page(current));
+ else
+ pr_warn("object %p is NOT on stack %p, but annotated.\n", addr,
+ task_stack_page(current));
+
WARN_ON(1);
}
@@ -1110,7 +1110,8 @@ void __init debug_objects_mem_init(void)
obj_cache = kmem_cache_create("debug_objects_cache",
sizeof (struct debug_obj), 0,
- SLAB_DEBUG_OBJECTS, NULL);
+ SLAB_DEBUG_OBJECTS | SLAB_NOLEAKTRACE,
+ NULL);
if (!obj_cache || debug_objects_replace_static_objects()) {
debug_objects_enabled = 0;
diff --git a/lib/decompress_unlz4.c b/lib/decompress_unlz4.c
index 036fc88..1b0baf3 100644
--- a/lib/decompress_unlz4.c
+++ b/lib/decompress_unlz4.c
@@ -72,7 +72,7 @@ STATIC inline int INIT unlz4(u8 *input, long in_len,
error("NULL input pointer and missing fill function");
goto exit_1;
} else {
- inp = large_malloc(lz4_compressbound(uncomp_chunksize));
+ inp = large_malloc(LZ4_compressBound(uncomp_chunksize));
if (!inp) {
error("Could not allocate input buffer");
goto exit_1;
@@ -136,7 +136,7 @@ STATIC inline int INIT unlz4(u8 *input, long in_len,
inp += 4;
size -= 4;
} else {
- if (chunksize > lz4_compressbound(uncomp_chunksize)) {
+ if (chunksize > LZ4_compressBound(uncomp_chunksize)) {
error("chunk length is longer than allocated");
goto exit_2;
}
@@ -152,11 +152,14 @@ STATIC inline int INIT unlz4(u8 *input, long in_len,
out_len -= dest_len;
} else
dest_len = out_len;
- ret = lz4_decompress(inp, &chunksize, outp, dest_len);
+
+ ret = LZ4_decompress_fast(inp, outp, dest_len);
+ chunksize = ret;
#else
dest_len = uncomp_chunksize;
- ret = lz4_decompress_unknownoutputsize(inp, chunksize, outp,
- &dest_len);
+
+ ret = LZ4_decompress_safe(inp, outp, chunksize, dest_len);
+ dest_len = ret;
#endif
if (ret < 0) {
error("Decoding failed");
diff --git a/lib/interval_tree_test.c b/lib/interval_tree_test.c
index 245900b..222c801 100644
--- a/lib/interval_tree_test.c
+++ b/lib/interval_tree_test.c
@@ -1,27 +1,38 @@
#include <linux/module.h>
+#include <linux/moduleparam.h>
#include <linux/interval_tree.h>
#include <linux/random.h>
+#include <linux/slab.h>
#include <asm/timex.h>
-#define NODES 100
-#define PERF_LOOPS 100000
-#define SEARCHES 100
-#define SEARCH_LOOPS 10000
+#define __param(type, name, init, msg) \
+ static type name = init; \
+ module_param(name, type, 0444); \
+ MODULE_PARM_DESC(name, msg);
+
+__param(int, nnodes, 100, "Number of nodes in the interval tree");
+__param(int, perf_loops, 1000, "Number of iterations modifying the tree");
+
+__param(int, nsearches, 100, "Number of searches to the interval tree");
+__param(int, search_loops, 1000, "Number of iterations searching the tree");
+__param(bool, search_all, false, "Searches will iterate all nodes in the tree");
+
+__param(uint, max_endpoint, ~0, "Largest value for the interval's endpoint");
static struct rb_root root = RB_ROOT;
-static struct interval_tree_node nodes[NODES];
-static u32 queries[SEARCHES];
+static struct interval_tree_node *nodes = NULL;
+static u32 *queries = NULL;
static struct rnd_state rnd;
static inline unsigned long
-search(unsigned long query, struct rb_root *root)
+search(struct rb_root *root, unsigned long start, unsigned long last)
{
struct interval_tree_node *node;
unsigned long results = 0;
- for (node = interval_tree_iter_first(root, query, query); node;
- node = interval_tree_iter_next(node, query, query))
+ for (node = interval_tree_iter_first(root, start, last); node;
+ node = interval_tree_iter_next(node, start, last))
results++;
return results;
}
@@ -29,19 +40,22 @@ search(unsigned long query, struct rb_root *root)
static void init(void)
{
int i;
- for (i = 0; i < NODES; i++) {
- u32 a = prandom_u32_state(&rnd);
- u32 b = prandom_u32_state(&rnd);
- if (a <= b) {
- nodes[i].start = a;
- nodes[i].last = b;
- } else {
- nodes[i].start = b;
- nodes[i].last = a;
- }
+
+ for (i = 0; i < nnodes; i++) {
+ u32 b = (prandom_u32_state(&rnd) >> 4) % max_endpoint;
+ u32 a = (prandom_u32_state(&rnd) >> 4) % b;
+
+ nodes[i].start = a;
+ nodes[i].last = b;
}
- for (i = 0; i < SEARCHES; i++)
- queries[i] = prandom_u32_state(&rnd);
+
+ /*
+ * Limit the search scope to what the user defined.
+ * Otherwise we are merely measuring empty walks,
+ * which is pointless.
+ */
+ for (i = 0; i < nsearches; i++)
+ queries[i] = (prandom_u32_state(&rnd) >> 4) % max_endpoint;
}
static int interval_tree_test_init(void)
@@ -50,6 +64,16 @@ static int interval_tree_test_init(void)
unsigned long results;
cycles_t time1, time2, time;
+ nodes = kmalloc(nnodes * sizeof(struct interval_tree_node), GFP_KERNEL);
+ if (!nodes)
+ return -ENOMEM;
+
+ queries = kmalloc(nsearches * sizeof(int), GFP_KERNEL);
+ if (!queries) {
+ kfree(nodes);
+ return -ENOMEM;
+ }
+
printk(KERN_ALERT "interval tree insert/remove");
prandom_seed_state(&rnd, 3141592653589793238ULL);
@@ -57,39 +81,46 @@ static int interval_tree_test_init(void)
time1 = get_cycles();
- for (i = 0; i < PERF_LOOPS; i++) {
- for (j = 0; j < NODES; j++)
+ for (i = 0; i < perf_loops; i++) {
+ for (j = 0; j < nnodes; j++)
interval_tree_insert(nodes + j, &root);
- for (j = 0; j < NODES; j++)
+ for (j = 0; j < nnodes; j++)
interval_tree_remove(nodes + j, &root);
}
time2 = get_cycles();
time = time2 - time1;
- time = div_u64(time, PERF_LOOPS);
+ time = div_u64(time, perf_loops);
printk(" -> %llu cycles\n", (unsigned long long)time);
printk(KERN_ALERT "interval tree search");
- for (j = 0; j < NODES; j++)
+ for (j = 0; j < nnodes; j++)
interval_tree_insert(nodes + j, &root);
time1 = get_cycles();
results = 0;
- for (i = 0; i < SEARCH_LOOPS; i++)
- for (j = 0; j < SEARCHES; j++)
- results += search(queries[j], &root);
+ for (i = 0; i < search_loops; i++)
+ for (j = 0; j < nsearches; j++) {
+ unsigned long start = search_all ? 0 : queries[j];
+ unsigned long last = search_all ? max_endpoint : queries[j];
+
+ results += search(&root, start, last);
+ }
time2 = get_cycles();
time = time2 - time1;
- time = div_u64(time, SEARCH_LOOPS);
- results = div_u64(results, SEARCH_LOOPS);
+ time = div_u64(time, search_loops);
+ results = div_u64(results, search_loops);
printk(" -> %llu cycles (%lu results)\n",
(unsigned long long)time, results);
+ kfree(queries);
+ kfree(nodes);
+
return -EAGAIN; /* Fail will directly unload the module */
}
diff --git a/lib/ioremap.c b/lib/ioremap.c
index 5323b59..b946203 100644
--- a/lib/ioremap.c
+++ b/lib/ioremap.c
@@ -84,7 +84,7 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr,
if (ioremap_pmd_enabled() &&
((next - addr) == PMD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PMD_SIZE) &&
- pmd_free_pte_page(pmd)) {
+ pmd_free_pte_page(pmd, addr)) {
if (pmd_set_huge(pmd, phys_addr + addr, prot))
continue;
}
@@ -111,7 +111,7 @@ static inline int ioremap_pud_range(pgd_t *pgd, unsigned long addr,
if (ioremap_pud_enabled() &&
((next - addr) == PUD_SIZE) &&
IS_ALIGNED(phys_addr + addr, PUD_SIZE) &&
- pud_free_pmd_page(pud)) {
+ pud_free_pmd_page(pud, addr)) {
if (pud_set_huge(pud, phys_addr + addr, prot))
continue;
}
diff --git a/lib/klist.c b/lib/klist.c
index 0507fa5..f6b5478 100644
--- a/lib/klist.c
+++ b/lib/klist.c
@@ -336,8 +336,9 @@ struct klist_node *klist_prev(struct klist_iter *i)
void (*put)(struct klist_node *) = i->i_klist->put;
struct klist_node *last = i->i_cur;
struct klist_node *prev;
+ unsigned long flags;
- spin_lock(&i->i_klist->k_lock);
+ spin_lock_irqsave(&i->i_klist->k_lock, flags);
if (last) {
prev = to_klist_node(last->n_node.prev);
@@ -356,7 +357,7 @@ struct klist_node *klist_prev(struct klist_iter *i)
prev = to_klist_node(prev->n_node.prev);
}
- spin_unlock(&i->i_klist->k_lock);
+ spin_unlock_irqrestore(&i->i_klist->k_lock, flags);
if (put && last)
put(last);
@@ -377,8 +378,9 @@ struct klist_node *klist_next(struct klist_iter *i)
void (*put)(struct klist_node *) = i->i_klist->put;
struct klist_node *last = i->i_cur;
struct klist_node *next;
+ unsigned long flags;
- spin_lock(&i->i_klist->k_lock);
+ spin_lock_irqsave(&i->i_klist->k_lock, flags);
if (last) {
next = to_klist_node(last->n_node.next);
@@ -397,7 +399,7 @@ struct klist_node *klist_next(struct klist_iter *i)
next = to_klist_node(next->n_node.next);
}
- spin_unlock(&i->i_klist->k_lock);
+ spin_unlock_irqrestore(&i->i_klist->k_lock, flags);
if (put && last)
put(last);
diff --git a/lib/kobject.c b/lib/kobject.c
index 763d70a..bbbb067 100644
--- a/lib/kobject.c
+++ b/lib/kobject.c
@@ -127,7 +127,7 @@ static void fill_kobj_path(struct kobject *kobj, char *path, int length)
int cur = strlen(kobject_name(parent));
/* back up enough to print this name with '/' */
length -= cur;
- strncpy(path + length, kobject_name(parent), cur);
+ memcpy(path + length, kobject_name(parent), cur);
*(path + --length) = '/';
}
@@ -234,14 +234,12 @@ static int kobject_add_internal(struct kobject *kobj)
/* be noisy on error issues */
if (error == -EEXIST)
- WARN(1, "%s failed for %s with "
- "-EEXIST, don't try to register things with "
- "the same name in the same directory.\n",
- __func__, kobject_name(kobj));
+ pr_err("%s failed for %s with -EEXIST, don't try to register things with the same name in the same directory.\n",
+ __func__, kobject_name(kobj));
else
- WARN(1, "%s failed for %s (error: %d parent: %s)\n",
- __func__, kobject_name(kobj), error,
- parent ? kobject_name(parent) : "'none'");
+ pr_err("%s failed for %s (error: %d parent: %s)\n",
+ __func__, kobject_name(kobj), error,
+ parent ? kobject_name(parent) : "'none'");
} else
kobj->state_in_sysfs = 1;
diff --git a/lib/lz4/Makefile b/lib/lz4/Makefile
index 8085d04..f7b11327 100644
--- a/lib/lz4/Makefile
+++ b/lib/lz4/Makefile
@@ -1,3 +1,5 @@
+ccflags-y += -O3
+
obj-$(CONFIG_LZ4_COMPRESS) += lz4_compress.o
obj-$(CONFIG_LZ4HC_COMPRESS) += lz4hc_compress.o
obj-$(CONFIG_LZ4_DECOMPRESS) += lz4_decompress.o
diff --git a/lib/lz4/lz4_compress.c b/lib/lz4/lz4_compress.c
index 28321d8..668400b 100644
--- a/lib/lz4/lz4_compress.c
+++ b/lib/lz4/lz4_compress.c
@@ -1,19 +1,16 @@
/*
* LZ4 - Fast LZ compression algorithm
- * Copyright (C) 2011-2012, Yann Collet.
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
-
+ * Copyright (C) 2011 - 2016, Yann Collet.
+ * BSD 2 - Clause License (http://www.opensource.org/licenses/bsd - license.php)
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following disclaimer
* in the documentation and/or other materials provided with the
* distribution.
- *
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
@@ -25,417 +22,939 @@
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
* You can contact the author at :
- * - LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
- * - LZ4 source repository : http://code.google.com/p/lz4/
+ * - LZ4 homepage : http://www.lz4.org
+ * - LZ4 source repository : https://github.com/lz4/lz4
*
- * Changed for kernel use by:
- * Chanho Min <chanho.min@lge.com>
+ * Changed for kernel usage by:
+ * Sven Schmidt <4sschmid@informatik.uni-hamburg.de>
*/
+/*-************************************
+ * Dependencies
+ **************************************/
+#include <linux/lz4.h>
+#include "lz4defs.h"
#include <linux/module.h>
#include <linux/kernel.h>
-#include <linux/lz4.h>
#include <asm/unaligned.h>
-#include "lz4defs.h"
+
+static const int LZ4_minLength = (MFLIMIT + 1);
+static const int LZ4_64Klimit = ((64 * KB) + (MFLIMIT - 1));
+
+/*-******************************
+ * Compression functions
+ ********************************/
+static FORCE_INLINE U32 LZ4_hash4(
+ U32 sequence,
+ tableType_t const tableType)
+{
+ if (tableType == byU16)
+ return ((sequence * 2654435761U)
+ >> ((MINMATCH * 8) - (LZ4_HASHLOG + 1)));
+ else
+ return ((sequence * 2654435761U)
+ >> ((MINMATCH * 8) - LZ4_HASHLOG));
+}
+
+static FORCE_INLINE U32 LZ4_hash5(
+ U64 sequence,
+ tableType_t const tableType)
+{
+ const U32 hashLog = (tableType == byU16)
+ ? LZ4_HASHLOG + 1
+ : LZ4_HASHLOG;
+
+#if LZ4_LITTLE_ENDIAN
+ static const U64 prime5bytes = 889523592379ULL;
+
+ return (U32)(((sequence << 24) * prime5bytes) >> (64 - hashLog));
+#else
+ static const U64 prime8bytes = 11400714785074694791ULL;
+
+ return (U32)(((sequence >> 24) * prime8bytes) >> (64 - hashLog));
+#endif
+}
+
+static FORCE_INLINE U32 LZ4_hashPosition(
+ const void *p,
+ tableType_t const tableType)
+{
+#if LZ4_ARCH64
+ if (tableType == byU32)
+ return LZ4_hash5(LZ4_read_ARCH(p), tableType);
+#endif
+
+ return LZ4_hash4(LZ4_read32(p), tableType);
+}
+
+static void LZ4_putPositionOnHash(
+ const BYTE *p,
+ U32 h,
+ void *tableBase,
+ tableType_t const tableType,
+ const BYTE *srcBase)
+{
+ switch (tableType) {
+ case byPtr:
+ {
+ const BYTE **hashTable = (const BYTE **)tableBase;
+
+ hashTable[h] = p;
+ return;
+ }
+ case byU32:
+ {
+ U32 *hashTable = (U32 *) tableBase;
+
+ hashTable[h] = (U32)(p - srcBase);
+ return;
+ }
+ case byU16:
+ {
+ U16 *hashTable = (U16 *) tableBase;
+
+ hashTable[h] = (U16)(p - srcBase);
+ return;
+ }
+ }
+}
+
+static FORCE_INLINE void LZ4_putPosition(
+ const BYTE *p,
+ void *tableBase,
+ tableType_t tableType,
+ const BYTE *srcBase)
+{
+ U32 const h = LZ4_hashPosition(p, tableType);
+
+ LZ4_putPositionOnHash(p, h, tableBase, tableType, srcBase);
+}
+
+static const BYTE *LZ4_getPositionOnHash(
+ U32 h,
+ void *tableBase,
+ tableType_t tableType,
+ const BYTE *srcBase)
+{
+ if (tableType == byPtr) {
+ const BYTE **hashTable = (const BYTE **) tableBase;
+
+ return hashTable[h];
+ }
+
+ if (tableType == byU32) {
+ const U32 * const hashTable = (U32 *) tableBase;
+
+ return hashTable[h] + srcBase;
+ }
+
+ {
+ /* default, to ensure a return */
+ const U16 * const hashTable = (U16 *) tableBase;
+
+ return hashTable[h] + srcBase;
+ }
+}
+
+static FORCE_INLINE const BYTE *LZ4_getPosition(
+ const BYTE *p,
+ void *tableBase,
+ tableType_t tableType,
+ const BYTE *srcBase)
+{
+ U32 const h = LZ4_hashPosition(p, tableType);
+
+ return LZ4_getPositionOnHash(h, tableBase, tableType, srcBase);
+}
+
/*
- * LZ4_compressCtx :
- * -----------------
- * Compress 'isize' bytes from 'source' into an output buffer 'dest' of
- * maximum size 'maxOutputSize'. * If it cannot achieve it, compression
- * will stop, and result of the function will be zero.
- * return : the number of bytes written in buffer 'dest', or 0 if the
- * compression fails
+ * LZ4_compress_generic() :
+ * inlined, to ensure branches are decided at compilation time
*/
-static inline int lz4_compressctx(void *ctx,
- const char *source,
- char *dest,
- int isize,
- int maxoutputsize)
+static FORCE_INLINE int LZ4_compress_generic(
+ LZ4_stream_t_internal * const dictPtr,
+ const char * const source,
+ char * const dest,
+ const int inputSize,
+ const int maxOutputSize,
+ const limitedOutput_directive outputLimited,
+ const tableType_t tableType,
+ const dict_directive dict,
+ const dictIssue_directive dictIssue,
+ const U32 acceleration)
{
- HTYPE *hashtable = (HTYPE *)ctx;
- const u8 *ip = (u8 *)source;
+ const BYTE *ip = (const BYTE *) source;
+ const BYTE *base;
+ const BYTE *lowLimit;
+ const BYTE * const lowRefLimit = ip - dictPtr->dictSize;
+ const BYTE * const dictionary = dictPtr->dictionary;
+ const BYTE * const dictEnd = dictionary + dictPtr->dictSize;
+ const size_t dictDelta = dictEnd - (const BYTE *)source;
+ const BYTE *anchor = (const BYTE *) source;
+ const BYTE * const iend = ip + inputSize;
+ const BYTE * const mflimit = iend - MFLIMIT;
+ const BYTE * const matchlimit = iend - LASTLITERALS;
+
+ BYTE *op = (BYTE *) dest;
+ BYTE * const olimit = op + maxOutputSize;
+
+ U32 forwardH;
+ size_t refDelta = 0;
+
+ /* Init conditions */
+ if ((U32)inputSize > (U32)LZ4_MAX_INPUT_SIZE) {
+ /* Unsupported inputSize, too large (or negative) */
+ return 0;
+ }
+
+ switch (dict) {
+ case noDict:
+ default:
+ base = (const BYTE *)source;
+ lowLimit = (const BYTE *)source;
+ break;
+ case withPrefix64k:
+ base = (const BYTE *)source - dictPtr->currentOffset;
+ lowLimit = (const BYTE *)source - dictPtr->dictSize;
+ break;
+ case usingExtDict:
+ base = (const BYTE *)source - dictPtr->currentOffset;
+ lowLimit = (const BYTE *)source;
+ break;
+ }
+
+ if ((tableType == byU16)
+ && (inputSize >= LZ4_64Klimit)) {
+ /* Size too large (not within 64K limit) */
+ return 0;
+ }
+
+ if (inputSize < LZ4_minLength) {
+ /* Input too small, no compression (all literals) */
+ goto _last_literals;
+ }
+
+ /* First Byte */
+ LZ4_putPosition(ip, dictPtr->hashTable, tableType, base);
+ ip++;
+ forwardH = LZ4_hashPosition(ip, tableType);
+
+ /* Main Loop */
+ for ( ; ; ) {
+ const BYTE *match;
+ BYTE *token;
+
+ /* Find a match */
+ {
+ const BYTE *forwardIp = ip;
+ unsigned int step = 1;
+ unsigned int searchMatchNb = acceleration << LZ4_SKIPTRIGGER;
+
+ do {
+ U32 const h = forwardH;
+
+ ip = forwardIp;
+ forwardIp += step;
+ step = (searchMatchNb++ >> LZ4_SKIPTRIGGER);
+
+ if (unlikely(forwardIp > mflimit))
+ goto _last_literals;
+
+ match = LZ4_getPositionOnHash(h,
+ dictPtr->hashTable,
+ tableType, base);
+
+ if (dict == usingExtDict) {
+ if (match < (const BYTE *)source) {
+ refDelta = dictDelta;
+ lowLimit = dictionary;
+ } else {
+ refDelta = 0;
+ lowLimit = (const BYTE *)source;
+ } }
+
+ forwardH = LZ4_hashPosition(forwardIp,
+ tableType);
+
+ LZ4_putPositionOnHash(ip, h, dictPtr->hashTable,
+ tableType, base);
+ } while (((dictIssue == dictSmall)
+ ? (match < lowRefLimit)
+ : 0)
+ || ((tableType == byU16)
+ ? 0
+ : (match + MAX_DISTANCE < ip))
+ || (LZ4_read32(match + refDelta)
+ != LZ4_read32(ip)));
+ }
+
+ /* Catch up */
+ while (((ip > anchor) & (match + refDelta > lowLimit))
+ && (unlikely(ip[-1] == match[refDelta - 1]))) {
+ ip--;
+ match--;
+ }
+
+ /* Encode Literals */
+ {
+ unsigned const int litLength = (unsigned int)(ip - anchor);
+
+ token = op++;
+
+ if ((outputLimited) &&
+ /* Check output buffer overflow */
+ (unlikely(op + litLength +
+ (2 + 1 + LASTLITERALS) +
+ (litLength / 255) > olimit)))
+ return 0;
+
+ if (litLength >= RUN_MASK) {
+ int len = (int)litLength - RUN_MASK;
+
+ *token = (RUN_MASK << ML_BITS);
+
+ for (; len >= 255; len -= 255)
+ *op++ = 255;
+ *op++ = (BYTE)len;
+ } else
+ *token = (BYTE)(litLength << ML_BITS);
+
+ /* Copy Literals */
+ LZ4_wildCopy(op, anchor, op + litLength);
+ op += litLength;
+ }
+
+_next_match:
+ /* Encode Offset */
+ LZ4_writeLE16(op, (U16)(ip - match));
+ op += 2;
+
+ /* Encode MatchLength */
+ {
+ unsigned int matchCode;
+
+ if ((dict == usingExtDict)
+ && (lowLimit == dictionary)) {
+ const BYTE *limit;
+
+ match += refDelta;
+ limit = ip + (dictEnd - match);
+
+ if (limit > matchlimit)
+ limit = matchlimit;
+
+ matchCode = LZ4_count(ip + MINMATCH,
+ match + MINMATCH, limit);
+
+ ip += MINMATCH + matchCode;
+
+ if (ip == limit) {
+ unsigned const int more = LZ4_count(ip,
+ (const BYTE *)source,
+ matchlimit);
+
+ matchCode += more;
+ ip += more;
+ }
+ } else {
+ matchCode = LZ4_count(ip + MINMATCH,
+ match + MINMATCH, matchlimit);
+ ip += MINMATCH + matchCode;
+ }
+
+ if (outputLimited &&
+ /* Check output buffer overflow */
+ (unlikely(op +
+ (1 + LASTLITERALS) +
+ (matchCode >> 8) > olimit)))
+ return 0;
+
+ if (matchCode >= ML_MASK) {
+ *token += ML_MASK;
+ matchCode -= ML_MASK;
+ LZ4_write32(op, 0xFFFFFFFF);
+
+ while (matchCode >= 4 * 255) {
+ op += 4;
+ LZ4_write32(op, 0xFFFFFFFF);
+ matchCode -= 4 * 255;
+ }
+
+ op += matchCode / 255;
+ *op++ = (BYTE)(matchCode % 255);
+ } else
+ *token += (BYTE)(matchCode);
+ }
+
+ anchor = ip;
+
+ /* Test end of chunk */
+ if (ip > mflimit)
+ break;
+
+ /* Fill table */
+ LZ4_putPosition(ip - 2, dictPtr->hashTable, tableType, base);
+
+ /* Test next position */
+ match = LZ4_getPosition(ip, dictPtr->hashTable,
+ tableType, base);
+
+ if (dict == usingExtDict) {
+ if (match < (const BYTE *)source) {
+ refDelta = dictDelta;
+ lowLimit = dictionary;
+ } else {
+ refDelta = 0;
+ lowLimit = (const BYTE *)source;
+ }
+ }
+
+ LZ4_putPosition(ip, dictPtr->hashTable, tableType, base);
+
+ if (((dictIssue == dictSmall) ? (match >= lowRefLimit) : 1)
+ && (match + MAX_DISTANCE >= ip)
+ && (LZ4_read32(match + refDelta) == LZ4_read32(ip))) {
+ token = op++;
+ *token = 0;
+ goto _next_match;
+ }
+
+ /* Prepare next loop */
+ forwardH = LZ4_hashPosition(++ip, tableType);
+ }
+
+_last_literals:
+ /* Encode Last Literals */
+ {
+ size_t const lastRun = (size_t)(iend - anchor);
+
+ if ((outputLimited) &&
+ /* Check output buffer overflow */
+ ((op - (BYTE *)dest) + lastRun + 1 +
+ ((lastRun + 255 - RUN_MASK) / 255) > (U32)maxOutputSize))
+ return 0;
+
+ if (lastRun >= RUN_MASK) {
+ size_t accumulator = lastRun - RUN_MASK;
+ *op++ = RUN_MASK << ML_BITS;
+ for (; accumulator >= 255; accumulator -= 255)
+ *op++ = 255;
+ *op++ = (BYTE) accumulator;
+ } else {
+ *op++ = (BYTE)(lastRun << ML_BITS);
+ }
+
+ memcpy(op, anchor, lastRun);
+
+ op += lastRun;
+ }
+
+ /* End */
+ return (int) (((char *)op) - dest);
+}
+
+static int LZ4_compress_fast_extState(
+ void *state,
+ const char *source,
+ char *dest,
+ int inputSize,
+ int maxOutputSize,
+ int acceleration)
+{
+ LZ4_stream_t_internal *ctx = &((LZ4_stream_t *)state)->internal_donotuse;
#if LZ4_ARCH64
- const BYTE * const base = ip;
+ const tableType_t tableType = byU32;
#else
- const int base = 0;
+ const tableType_t tableType = byPtr;
#endif
- const u8 *anchor = ip;
- const u8 *const iend = ip + isize;
- const u8 *const mflimit = iend - MFLIMIT;
- #define MATCHLIMIT (iend - LASTLITERALS)
- u8 *op = (u8 *) dest;
- u8 *const oend = op + maxoutputsize;
- int length;
- const int skipstrength = SKIPSTRENGTH;
- u32 forwardh;
- int lastrun;
+ LZ4_resetStream((LZ4_stream_t *)state);
- /* Init */
- if (isize < MINLENGTH)
+ if (acceleration < 1)
+ acceleration = LZ4_ACCELERATION_DEFAULT;
+
+ if (maxOutputSize >= LZ4_COMPRESSBOUND(inputSize)) {
+ if (inputSize < LZ4_64Klimit)
+ return LZ4_compress_generic(ctx, source,
+ dest, inputSize, 0,
+ noLimit, byU16, noDict,
+ noDictIssue, acceleration);
+ else
+ return LZ4_compress_generic(ctx, source,
+ dest, inputSize, 0,
+ noLimit, tableType, noDict,
+ noDictIssue, acceleration);
+ } else {
+ if (inputSize < LZ4_64Klimit)
+ return LZ4_compress_generic(ctx, source,
+ dest, inputSize,
+ maxOutputSize, limitedOutput, byU16, noDict,
+ noDictIssue, acceleration);
+ else
+ return LZ4_compress_generic(ctx, source,
+ dest, inputSize,
+ maxOutputSize, limitedOutput, tableType, noDict,
+ noDictIssue, acceleration);
+ }
+}
+
+int LZ4_compress_fast(const char *source, char *dest, int inputSize,
+ int maxOutputSize, int acceleration, void *wrkmem)
+{
+ return LZ4_compress_fast_extState(wrkmem, source, dest, inputSize,
+ maxOutputSize, acceleration);
+}
+EXPORT_SYMBOL(LZ4_compress_fast);
+
+int LZ4_compress_default(const char *source, char *dest, int inputSize,
+ int maxOutputSize, void *wrkmem)
+{
+ return LZ4_compress_fast(source, dest, inputSize,
+ maxOutputSize, LZ4_ACCELERATION_DEFAULT, wrkmem);
+}
+EXPORT_SYMBOL(LZ4_compress_default);
+
+/*-******************************
+ * *_destSize() variant
+ ********************************/
+static int LZ4_compress_destSize_generic(
+ LZ4_stream_t_internal * const ctx,
+ const char * const src,
+ char * const dst,
+ int * const srcSizePtr,
+ const int targetDstSize,
+ const tableType_t tableType)
+{
+ const BYTE *ip = (const BYTE *) src;
+ const BYTE *base = (const BYTE *) src;
+ const BYTE *lowLimit = (const BYTE *) src;
+ const BYTE *anchor = ip;
+ const BYTE * const iend = ip + *srcSizePtr;
+ const BYTE * const mflimit = iend - MFLIMIT;
+ const BYTE * const matchlimit = iend - LASTLITERALS;
+
+ BYTE *op = (BYTE *) dst;
+ BYTE * const oend = op + targetDstSize;
+ BYTE * const oMaxLit = op + targetDstSize - 2 /* offset */
+ - 8 /* because 8 + MINMATCH == MFLIMIT */ - 1 /* token */;
+ BYTE * const oMaxMatch = op + targetDstSize
+ - (LASTLITERALS + 1 /* token */);
+ BYTE * const oMaxSeq = oMaxLit - 1 /* token */;
+
+ U32 forwardH;
+
+ /* Init conditions */
+ /* Impossible to store anything */
+ if (targetDstSize < 1)
+ return 0;
+ /* Unsupported input size, too large (or negative) */
+ if ((U32)*srcSizePtr > (U32)LZ4_MAX_INPUT_SIZE)
+ return 0;
+ /* Size too large (not within 64K limit) */
+ if ((tableType == byU16) && (*srcSizePtr >= LZ4_64Klimit))
+ return 0;
+ /* Input too small, no compression (all literals) */
+ if (*srcSizePtr < LZ4_minLength)
goto _last_literals;
- memset((void *)hashtable, 0, LZ4_MEM_COMPRESS);
-
/* First Byte */
- hashtable[LZ4_HASH_VALUE(ip)] = ip - base;
- ip++;
- forwardh = LZ4_HASH_VALUE(ip);
+ *srcSizePtr = 0;
+ LZ4_putPosition(ip, ctx->hashTable, tableType, base);
+ ip++; forwardH = LZ4_hashPosition(ip, tableType);
/* Main Loop */
- for (;;) {
- int findmatchattempts = (1U << skipstrength) + 3;
- const u8 *forwardip = ip;
- const u8 *ref;
- u8 *token;
+ for ( ; ; ) {
+ const BYTE *match;
+ BYTE *token;
/* Find a match */
- do {
- u32 h = forwardh;
- int step = findmatchattempts++ >> skipstrength;
- ip = forwardip;
- forwardip = ip + step;
+ {
+ const BYTE *forwardIp = ip;
+ unsigned int step = 1;
+ unsigned int searchMatchNb = 1 << LZ4_SKIPTRIGGER;
- if (unlikely(forwardip > mflimit))
- goto _last_literals;
+ do {
+ U32 h = forwardH;
- forwardh = LZ4_HASH_VALUE(forwardip);
- ref = base + hashtable[h];
- hashtable[h] = ip - base;
- } while ((ref < ip - MAX_DISTANCE) || (A32(ref) != A32(ip)));
+ ip = forwardIp;
+ forwardIp += step;
+ step = (searchMatchNb++ >> LZ4_SKIPTRIGGER);
+
+ if (unlikely(forwardIp > mflimit))
+ goto _last_literals;
+
+ match = LZ4_getPositionOnHash(h, ctx->hashTable,
+ tableType, base);
+ forwardH = LZ4_hashPosition(forwardIp,
+ tableType);
+ LZ4_putPositionOnHash(ip, h,
+ ctx->hashTable, tableType,
+ base);
+
+ } while (((tableType == byU16)
+ ? 0
+ : (match + MAX_DISTANCE < ip))
+ || (LZ4_read32(match) != LZ4_read32(ip)));
+ }
/* Catch up */
- while ((ip > anchor) && (ref > (u8 *)source) &&
- unlikely(ip[-1] == ref[-1])) {
+ while ((ip > anchor)
+ && (match > lowLimit)
+ && (unlikely(ip[-1] == match[-1]))) {
ip--;
- ref--;
+ match--;
}
/* Encode Literal length */
- length = (int)(ip - anchor);
- token = op++;
- /* check output limit */
- if (unlikely(op + length + (2 + 1 + LASTLITERALS) +
- (length >> 8) > oend))
- return 0;
+ {
+ unsigned int litLength = (unsigned int)(ip - anchor);
- if (length >= (int)RUN_MASK) {
- int len;
- *token = (RUN_MASK << ML_BITS);
- len = length - RUN_MASK;
- for (; len > 254 ; len -= 255)
- *op++ = 255;
- *op++ = (u8)len;
- } else
- *token = (length << ML_BITS);
+ token = op++;
+ if (op + ((litLength + 240) / 255)
+ + litLength > oMaxLit) {
+ /* Not enough space for a last match */
+ op--;
+ goto _last_literals;
+ }
+ if (litLength >= RUN_MASK) {
+ unsigned int len = litLength - RUN_MASK;
+ *token = (RUN_MASK<<ML_BITS);
+ for (; len >= 255; len -= 255)
+ *op++ = 255;
+ *op++ = (BYTE)len;
+ } else
+ *token = (BYTE)(litLength << ML_BITS);
- /* Copy Literals */
- LZ4_BLINDCOPY(anchor, op, length);
+ /* Copy Literals */
+ LZ4_wildCopy(op, anchor, op + litLength);
+ op += litLength;
+ }
+
_next_match:
/* Encode Offset */
- LZ4_WRITE_LITTLEENDIAN_16(op, (u16)(ip - ref));
+ LZ4_writeLE16(op, (U16)(ip - match)); op += 2;
- /* Start Counting */
- ip += MINMATCH;
- /* MinMatch verified */
- ref += MINMATCH;
- anchor = ip;
- while (likely(ip < MATCHLIMIT - (STEPSIZE - 1))) {
- #if LZ4_ARCH64
- u64 diff = A64(ref) ^ A64(ip);
- #else
- u32 diff = A32(ref) ^ A32(ip);
- #endif
- if (!diff) {
- ip += STEPSIZE;
- ref += STEPSIZE;
- continue;
- }
- ip += LZ4_NBCOMMONBYTES(diff);
- goto _endcount;
- }
- #if LZ4_ARCH64
- if ((ip < (MATCHLIMIT - 3)) && (A32(ref) == A32(ip))) {
- ip += 4;
- ref += 4;
- }
- #endif
- if ((ip < (MATCHLIMIT - 1)) && (A16(ref) == A16(ip))) {
- ip += 2;
- ref += 2;
- }
- if ((ip < MATCHLIMIT) && (*ref == *ip))
- ip++;
-_endcount:
/* Encode MatchLength */
- length = (int)(ip - anchor);
- /* Check output limit */
- if (unlikely(op + (1 + LASTLITERALS) + (length >> 8) > oend))
- return 0;
- if (length >= (int)ML_MASK) {
- *token += ML_MASK;
- length -= ML_MASK;
- for (; length > 509 ; length -= 510) {
- *op++ = 255;
- *op++ = 255;
- }
- if (length > 254) {
- length -= 255;
- *op++ = 255;
- }
- *op++ = (u8)length;
- } else
- *token += length;
+ {
+ size_t matchLength = LZ4_count(ip + MINMATCH,
+ match + MINMATCH, matchlimit);
- /* Test end of chunk */
- if (ip > mflimit) {
- anchor = ip;
- break;
+ if (op + ((matchLength + 240)/255) > oMaxMatch) {
+ /* Match description too long : reduce it */
+ matchLength = (15 - 1) + (oMaxMatch - op) * 255;
+ }
+ ip += MINMATCH + matchLength;
+
+ if (matchLength >= ML_MASK) {
+ *token += ML_MASK;
+ matchLength -= ML_MASK;
+ while (matchLength >= 255) {
+ matchLength -= 255;
+ *op++ = 255;
+ }
+ *op++ = (BYTE)matchLength;
+ } else
+ *token += (BYTE)(matchLength);
}
+ anchor = ip;
+
+ /* Test end of block */
+ if (ip > mflimit)
+ break;
+ if (op > oMaxSeq)
+ break;
+
/* Fill table */
- hashtable[LZ4_HASH_VALUE(ip-2)] = ip - 2 - base;
+ LZ4_putPosition(ip - 2, ctx->hashTable, tableType, base);
/* Test next position */
- ref = base + hashtable[LZ4_HASH_VALUE(ip)];
- hashtable[LZ4_HASH_VALUE(ip)] = ip - base;
- if ((ref > ip - (MAX_DISTANCE + 1)) && (A32(ref) == A32(ip))) {
- token = op++;
- *token = 0;
+ match = LZ4_getPosition(ip, ctx->hashTable, tableType, base);
+ LZ4_putPosition(ip, ctx->hashTable, tableType, base);
+
+ if ((match + MAX_DISTANCE >= ip)
+ && (LZ4_read32(match) == LZ4_read32(ip))) {
+ token = op++; *token = 0;
goto _next_match;
}
/* Prepare next loop */
- anchor = ip++;
- forwardh = LZ4_HASH_VALUE(ip);
+ forwardH = LZ4_hashPosition(++ip, tableType);
}
_last_literals:
/* Encode Last Literals */
- lastrun = (int)(iend - anchor);
- if (((char *)op - dest) + lastrun + 1
- + ((lastrun + 255 - RUN_MASK) / 255) > (u32)maxoutputsize)
- return 0;
+ {
+ size_t lastRunSize = (size_t)(iend - anchor);
- if (lastrun >= (int)RUN_MASK) {
- *op++ = (RUN_MASK << ML_BITS);
- lastrun -= RUN_MASK;
- for (; lastrun > 254 ; lastrun -= 255)
- *op++ = 255;
- *op++ = (u8)lastrun;
- } else
- *op++ = (lastrun << ML_BITS);
- memcpy(op, anchor, iend - anchor);
- op += iend - anchor;
-
- /* End */
- return (int)(((char *)op) - dest);
-}
-
-static inline int lz4_compress64kctx(void *ctx,
- const char *source,
- char *dest,
- int isize,
- int maxoutputsize)
-{
- u16 *hashtable = (u16 *)ctx;
- const u8 *ip = (u8 *) source;
- const u8 *anchor = ip;
- const u8 *const base = ip;
- const u8 *const iend = ip + isize;
- const u8 *const mflimit = iend - MFLIMIT;
- #define MATCHLIMIT (iend - LASTLITERALS)
-
- u8 *op = (u8 *) dest;
- u8 *const oend = op + maxoutputsize;
- int len, length;
- const int skipstrength = SKIPSTRENGTH;
- u32 forwardh;
- int lastrun;
-
- /* Init */
- if (isize < MINLENGTH)
- goto _last_literals;
-
- memset((void *)hashtable, 0, LZ4_MEM_COMPRESS);
-
- /* First Byte */
- ip++;
- forwardh = LZ4_HASH64K_VALUE(ip);
-
- /* Main Loop */
- for (;;) {
- int findmatchattempts = (1U << skipstrength) + 3;
- const u8 *forwardip = ip;
- const u8 *ref;
- u8 *token;
-
- /* Find a match */
- do {
- u32 h = forwardh;
- int step = findmatchattempts++ >> skipstrength;
- ip = forwardip;
- forwardip = ip + step;
-
- if (forwardip > mflimit)
- goto _last_literals;
-
- forwardh = LZ4_HASH64K_VALUE(forwardip);
- ref = base + hashtable[h];
- hashtable[h] = (u16)(ip - base);
- } while (A32(ref) != A32(ip));
-
- /* Catch up */
- while ((ip > anchor) && (ref > (u8 *)source)
- && (ip[-1] == ref[-1])) {
- ip--;
- ref--;
+ if (op + 1 /* token */
+ + ((lastRunSize + 240) / 255) /* litLength */
+ + lastRunSize /* literals */ > oend) {
+ /* adapt lastRunSize to fill 'dst' */
+ lastRunSize = (oend - op) - 1;
+ lastRunSize -= (lastRunSize + 240) / 255;
}
+ ip = anchor + lastRunSize;
- /* Encode Literal length */
- length = (int)(ip - anchor);
- token = op++;
- /* Check output limit */
- if (unlikely(op + length + (2 + 1 + LASTLITERALS)
- + (length >> 8) > oend))
- return 0;
- if (length >= (int)RUN_MASK) {
- *token = (RUN_MASK << ML_BITS);
- len = length - RUN_MASK;
- for (; len > 254 ; len -= 255)
+ if (lastRunSize >= RUN_MASK) {
+ size_t accumulator = lastRunSize - RUN_MASK;
+
+ *op++ = RUN_MASK << ML_BITS;
+ for (; accumulator >= 255; accumulator -= 255)
*op++ = 255;
- *op++ = (u8)len;
- } else
- *token = (length << ML_BITS);
-
- /* Copy Literals */
- LZ4_BLINDCOPY(anchor, op, length);
-
-_next_match:
- /* Encode Offset */
- LZ4_WRITE_LITTLEENDIAN_16(op, (u16)(ip - ref));
-
- /* Start Counting */
- ip += MINMATCH;
- /* MinMatch verified */
- ref += MINMATCH;
- anchor = ip;
-
- while (ip < MATCHLIMIT - (STEPSIZE - 1)) {
- #if LZ4_ARCH64
- u64 diff = A64(ref) ^ A64(ip);
- #else
- u32 diff = A32(ref) ^ A32(ip);
- #endif
-
- if (!diff) {
- ip += STEPSIZE;
- ref += STEPSIZE;
- continue;
- }
- ip += LZ4_NBCOMMONBYTES(diff);
- goto _endcount;
+ *op++ = (BYTE) accumulator;
+ } else {
+ *op++ = (BYTE)(lastRunSize<<ML_BITS);
}
- #if LZ4_ARCH64
- if ((ip < (MATCHLIMIT - 3)) && (A32(ref) == A32(ip))) {
- ip += 4;
- ref += 4;
- }
- #endif
- if ((ip < (MATCHLIMIT - 1)) && (A16(ref) == A16(ip))) {
- ip += 2;
- ref += 2;
- }
- if ((ip < MATCHLIMIT) && (*ref == *ip))
- ip++;
-_endcount:
-
- /* Encode MatchLength */
- len = (int)(ip - anchor);
- /* Check output limit */
- if (unlikely(op + (1 + LASTLITERALS) + (len >> 8) > oend))
- return 0;
- if (len >= (int)ML_MASK) {
- *token += ML_MASK;
- len -= ML_MASK;
- for (; len > 509 ; len -= 510) {
- *op++ = 255;
- *op++ = 255;
- }
- if (len > 254) {
- len -= 255;
- *op++ = 255;
- }
- *op++ = (u8)len;
- } else
- *token += len;
-
- /* Test end of chunk */
- if (ip > mflimit) {
- anchor = ip;
- break;
- }
-
- /* Fill table */
- hashtable[LZ4_HASH64K_VALUE(ip-2)] = (u16)(ip - 2 - base);
-
- /* Test next position */
- ref = base + hashtable[LZ4_HASH64K_VALUE(ip)];
- hashtable[LZ4_HASH64K_VALUE(ip)] = (u16)(ip - base);
- if (A32(ref) == A32(ip)) {
- token = op++;
- *token = 0;
- goto _next_match;
- }
-
- /* Prepare next loop */
- anchor = ip++;
- forwardh = LZ4_HASH64K_VALUE(ip);
+ memcpy(op, anchor, lastRunSize);
+ op += lastRunSize;
}
-_last_literals:
- /* Encode Last Literals */
- lastrun = (int)(iend - anchor);
- if (op + lastrun + 1 + (lastrun - RUN_MASK + 255) / 255 > oend)
- return 0;
- if (lastrun >= (int)RUN_MASK) {
- *op++ = (RUN_MASK << ML_BITS);
- lastrun -= RUN_MASK;
- for (; lastrun > 254 ; lastrun -= 255)
- *op++ = 255;
- *op++ = (u8)lastrun;
- } else
- *op++ = (lastrun << ML_BITS);
- memcpy(op, anchor, iend - anchor);
- op += iend - anchor;
/* End */
- return (int)(((char *)op) - dest);
+ *srcSizePtr = (int) (((const char *)ip) - src);
+ return (int) (((char *)op) - dst);
}
-int lz4_compress(const unsigned char *src, size_t src_len,
- unsigned char *dst, size_t *dst_len, void *wrkmem)
+static int LZ4_compress_destSize_extState(
+ LZ4_stream_t *state,
+ const char *src,
+ char *dst,
+ int *srcSizePtr,
+ int targetDstSize)
{
- int ret = -1;
- int out_len = 0;
+#if LZ4_ARCH64
+ const tableType_t tableType = byU32;
+#else
+ const tableType_t tableType = byPtr;
+#endif
- if (src_len < LZ4_64KLIMIT)
- out_len = lz4_compress64kctx(wrkmem, src, dst, src_len,
- lz4_compressbound(src_len));
+ LZ4_resetStream(state);
+
+ if (targetDstSize >= LZ4_COMPRESSBOUND(*srcSizePtr)) {
+ /* compression success is guaranteed */
+ return LZ4_compress_fast_extState(
+ state, src, dst, *srcSizePtr,
+ targetDstSize, 1);
+ } else {
+ if (*srcSizePtr < LZ4_64Klimit)
+ return LZ4_compress_destSize_generic(
+ &state->internal_donotuse,
+ src, dst, srcSizePtr,
+ targetDstSize, byU16);
+ else
+ return LZ4_compress_destSize_generic(
+ &state->internal_donotuse,
+ src, dst, srcSizePtr,
+ targetDstSize, tableType);
+ }
+}
+
+
+int LZ4_compress_destSize(
+ const char *src,
+ char *dst,
+ int *srcSizePtr,
+ int targetDstSize,
+ void *wrkmem)
+{
+ return LZ4_compress_destSize_extState(wrkmem, src, dst, srcSizePtr,
+ targetDstSize);
+}
+EXPORT_SYMBOL(LZ4_compress_destSize);
+
+/*-******************************
+ * Streaming functions
+ ********************************/
+void LZ4_resetStream(LZ4_stream_t *LZ4_stream)
+{
+ memset(LZ4_stream, 0, sizeof(LZ4_stream_t));
+}
+
+int LZ4_loadDict(LZ4_stream_t *LZ4_dict,
+ const char *dictionary, int dictSize)
+{
+ LZ4_stream_t_internal *dict = &LZ4_dict->internal_donotuse;
+ const BYTE *p = (const BYTE *)dictionary;
+ const BYTE * const dictEnd = p + dictSize;
+ const BYTE *base;
+
+ if ((dict->initCheck)
+ || (dict->currentOffset > 1 * GB)) {
+ /* Uninitialized structure, or reuse overflow */
+ LZ4_resetStream(LZ4_dict);
+ }
+
+ if (dictSize < (int)HASH_UNIT) {
+ dict->dictionary = NULL;
+ dict->dictSize = 0;
+ return 0;
+ }
+
+ if ((dictEnd - p) > 64 * KB)
+ p = dictEnd - 64 * KB;
+ dict->currentOffset += 64 * KB;
+ base = p - dict->currentOffset;
+ dict->dictionary = p;
+ dict->dictSize = (U32)(dictEnd - p);
+ dict->currentOffset += dict->dictSize;
+
+ while (p <= dictEnd - HASH_UNIT) {
+ LZ4_putPosition(p, dict->hashTable, byU32, base);
+ p += 3;
+ }
+
+ return dict->dictSize;
+}
+EXPORT_SYMBOL(LZ4_loadDict);
+
+static void LZ4_renormDictT(LZ4_stream_t_internal *LZ4_dict,
+ const BYTE *src)
+{
+ if ((LZ4_dict->currentOffset > 0x80000000) ||
+ ((uptrval)LZ4_dict->currentOffset > (uptrval)src)) {
+ /* address space overflow */
+ /* rescale hash table */
+ U32 const delta = LZ4_dict->currentOffset - 64 * KB;
+ const BYTE *dictEnd = LZ4_dict->dictionary + LZ4_dict->dictSize;
+ int i;
+
+ for (i = 0; i < LZ4_HASH_SIZE_U32; i++) {
+ if (LZ4_dict->hashTable[i] < delta)
+ LZ4_dict->hashTable[i] = 0;
+ else
+ LZ4_dict->hashTable[i] -= delta;
+ }
+ LZ4_dict->currentOffset = 64 * KB;
+ if (LZ4_dict->dictSize > 64 * KB)
+ LZ4_dict->dictSize = 64 * KB;
+ LZ4_dict->dictionary = dictEnd - LZ4_dict->dictSize;
+ }
+}
+
+int LZ4_saveDict(LZ4_stream_t *LZ4_dict, char *safeBuffer, int dictSize)
+{
+ LZ4_stream_t_internal * const dict = &LZ4_dict->internal_donotuse;
+ const BYTE * const previousDictEnd = dict->dictionary + dict->dictSize;
+
+ if ((U32)dictSize > 64 * KB) {
+ /* useless to define a dictionary > 64 * KB */
+ dictSize = 64 * KB;
+ }
+ if ((U32)dictSize > dict->dictSize)
+ dictSize = dict->dictSize;
+
+ memmove(safeBuffer, previousDictEnd - dictSize, dictSize);
+
+ dict->dictionary = (const BYTE *)safeBuffer;
+ dict->dictSize = (U32)dictSize;
+
+ return dictSize;
+}
+EXPORT_SYMBOL(LZ4_saveDict);
+
+int LZ4_compress_fast_continue(LZ4_stream_t *LZ4_stream, const char *source,
+ char *dest, int inputSize, int maxOutputSize, int acceleration)
+{
+ LZ4_stream_t_internal *streamPtr = &LZ4_stream->internal_donotuse;
+ const BYTE * const dictEnd = streamPtr->dictionary
+ + streamPtr->dictSize;
+
+ const BYTE *smallest = (const BYTE *) source;
+
+ if (streamPtr->initCheck) {
+ /* Uninitialized structure detected */
+ return 0;
+ }
+
+ if ((streamPtr->dictSize > 0) && (smallest > dictEnd))
+ smallest = dictEnd;
+
+ LZ4_renormDictT(streamPtr, smallest);
+
+ if (acceleration < 1)
+ acceleration = LZ4_ACCELERATION_DEFAULT;
+
+ /* Check overlapping input/dictionary space */
+ {
+ const BYTE *sourceEnd = (const BYTE *) source + inputSize;
+
+ if ((sourceEnd > streamPtr->dictionary)
+ && (sourceEnd < dictEnd)) {
+ streamPtr->dictSize = (U32)(dictEnd - sourceEnd);
+ if (streamPtr->dictSize > 64 * KB)
+ streamPtr->dictSize = 64 * KB;
+ if (streamPtr->dictSize < 4)
+ streamPtr->dictSize = 0;
+ streamPtr->dictionary = dictEnd - streamPtr->dictSize;
+ }
+ }
+
+ /* prefix mode : source data follows dictionary */
+ if (dictEnd == (const BYTE *)source) {
+ int result;
+
+ if ((streamPtr->dictSize < 64 * KB) &&
+ (streamPtr->dictSize < streamPtr->currentOffset)) {
+ result = LZ4_compress_generic(
+ streamPtr, source, dest, inputSize,
+ maxOutputSize, limitedOutput, byU32,
+ withPrefix64k, dictSmall, acceleration);
+ } else {
+ result = LZ4_compress_generic(
+ streamPtr, source, dest, inputSize,
+ maxOutputSize, limitedOutput, byU32,
+ withPrefix64k, noDictIssue, acceleration);
+ }
+ streamPtr->dictSize += (U32)inputSize;
+ streamPtr->currentOffset += (U32)inputSize;
+ return result;
+ }
+
+ /* external dictionary mode */
+ {
+ int result;
+
+ if ((streamPtr->dictSize < 64 * KB) &&
+ (streamPtr->dictSize < streamPtr->currentOffset)) {
+ result = LZ4_compress_generic(
+ streamPtr, source, dest, inputSize,
+ maxOutputSize, limitedOutput, byU32,
+ usingExtDict, dictSmall, acceleration);
+ } else {
+ result = LZ4_compress_generic(
+ streamPtr, source, dest, inputSize,
+ maxOutputSize, limitedOutput, byU32,
+ usingExtDict, noDictIssue, acceleration);
+ }
+ streamPtr->dictionary = (const BYTE *)source;
+ streamPtr->dictSize = (U32)inputSize;
+ streamPtr->currentOffset += (U32)inputSize;
+ return result;
+ }
+}
+EXPORT_SYMBOL(LZ4_compress_fast_continue);
+
+/*-******************************
+ * For backwards compatibility
+ ********************************/
+int lz4_compress(const unsigned char *src, size_t src_len, unsigned char *dst,
+ size_t *dst_len, void *wrkmem) {
+ *dst_len = LZ4_compress_default(src, dst, src_len,
+ LZ4_COMPRESSBOUND(src_len), wrkmem);
+
+ /*
+ * Prior lz4_compress will return -1 in case of error
+ * and 0 on success
+ * while new LZ4_compress_fast/default
+ * returns 0 in case of error
+ * and the output length on success
+ */
+ if (!*dst_len)
+ return -1;
else
- out_len = lz4_compressctx(wrkmem, src, dst, src_len,
- lz4_compressbound(src_len));
-
- if (out_len < 0)
- goto exit;
-
- *dst_len = out_len;
-
- return 0;
-exit:
- return ret;
+ return 0;
}
EXPORT_SYMBOL(lz4_compress);
diff --git a/lib/lz4/lz4_decompress.c b/lib/lz4/lz4_decompress.c
index 6d940c7..b5e00a1 100644
--- a/lib/lz4/lz4_decompress.c
+++ b/lib/lz4/lz4_decompress.c
@@ -1,25 +1,16 @@
/*
- * LZ4 Decompressor for Linux kernel
- *
- * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
- *
- * Based on LZ4 implementation by Yann Collet.
- *
* LZ4 - Fast LZ compression algorithm
- * Copyright (C) 2011-2012, Yann Collet.
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
- *
+ * Copyright (C) 2011 - 2016, Yann Collet.
+ * BSD 2 - Clause License (http://www.opensource.org/licenses/bsd - license.php)
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following disclaimer
* in the documentation and/or other materials provided with the
* distribution.
- *
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
@@ -31,313 +22,529 @@
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ * You can contact the author at :
+ * - LZ4 homepage : http://www.lz4.org
+ * - LZ4 source repository : https://github.com/lz4/lz4
*
- * You can contact the author at :
- * - LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
- * - LZ4 source repository : http://code.google.com/p/lz4/
+ * Changed for kernel usage by:
+ * Sven Schmidt <4sschmid@informatik.uni-hamburg.de>
*/
-#ifndef STATIC
+/*-************************************
+ * Dependencies
+ **************************************/
+#include <linux/lz4.h>
+#include "lz4defs.h"
+#include <linux/init.h>
#include <linux/module.h>
#include <linux/kernel.h>
-#endif
-#include <linux/lz4.h>
-
#include <asm/unaligned.h>
-#include "lz4defs.h"
-
-static const int dec32table[] = {0, 3, 2, 3, 0, 0, 0, 0};
-#if LZ4_ARCH64
-static const int dec64table[] = {0, 0, 0, -1, 0, 1, 2, 3};
-#endif
-
-static int lz4_uncompress(const char *source, char *dest, int osize)
+/*-*****************************
+ * Decompression functions
+ *******************************/
+/* LZ4_decompress_generic() :
+ * This generic decompression function cover all use cases.
+ * It shall be instantiated several times, using different sets of directives
+ * Note that it is important this generic function is really inlined,
+ * in order to remove useless branches during compilation optimization.
+ */
+static FORCE_INLINE int LZ4_decompress_generic(
+ const char * const source,
+ char * const dest,
+ int inputSize,
+ /*
+ * If endOnInput == endOnInputSize,
+ * this value is the max size of Output Buffer.
+ */
+ int outputSize,
+ /* endOnOutputSize, endOnInputSize */
+ int endOnInput,
+ /* full, partial */
+ int partialDecoding,
+ /* only used if partialDecoding == partial */
+ int targetOutputSize,
+ /* noDict, withPrefix64k, usingExtDict */
+ int dict,
+ /* == dest when no prefix */
+ const BYTE * const lowPrefix,
+ /* only if dict == usingExtDict */
+ const BYTE * const dictStart,
+ /* note : = 0 if noDict */
+ const size_t dictSize
+ )
{
+ /* Local Variables */
const BYTE *ip = (const BYTE *) source;
- const BYTE *ref;
+ const BYTE * const iend = ip + inputSize;
+
BYTE *op = (BYTE *) dest;
- BYTE * const oend = op + osize;
+ BYTE * const oend = op + outputSize;
BYTE *cpy;
- unsigned token;
- size_t length;
+ BYTE *oexit = op + targetOutputSize;
+ const BYTE * const lowLimit = lowPrefix - dictSize;
+ const BYTE * const dictEnd = (const BYTE *)dictStart + dictSize;
+ const unsigned int dec32table[] = { 0, 1, 2, 1, 4, 4, 4, 4 };
+ const int dec64table[] = { 0, 0, 0, -1, 0, 1, 2, 3 };
+
+ const int safeDecode = (endOnInput == endOnInputSize);
+ const int checkOffset = ((safeDecode) && (dictSize < (int)(64 * KB)));
+
+ /* Special cases */
+ /* targetOutputSize too high => decode everything */
+ if ((partialDecoding) && (oexit > oend - MFLIMIT))
+ oexit = oend - MFLIMIT;
+
+ /* Empty output buffer */
+ if ((endOnInput) && (unlikely(outputSize == 0)))
+ return ((inputSize == 1) && (*ip == 0)) ? 0 : -1;
+
+ if ((!endOnInput) && (unlikely(outputSize == 0)))
+ return (*ip == 0 ? 1 : -1);
+
+ /* Main Loop : decode sequences */
while (1) {
+ size_t length;
+ const BYTE *match;
+ size_t offset;
- /* get runlength */
- token = *ip++;
- length = (token >> ML_BITS);
+ /* get literal length */
+ unsigned int const token = *ip++;
+
+ length = token>>ML_BITS;
+
if (length == RUN_MASK) {
- size_t len;
+ unsigned int s;
- len = *ip++;
- for (; len == 255; length += 255)
- len = *ip++;
- if (unlikely(length > (size_t)(length + len)))
+ do {
+ s = *ip++;
+ length += s;
+ } while (likely(endOnInput
+ ? ip < iend - RUN_MASK
+ : 1) & (s == 255));
+
+ if ((safeDecode)
+ && unlikely(
+ (size_t)(op + length) < (size_t)(op))) {
+ /* overflow detection */
goto _output_error;
- length += len;
+ }
+ if ((safeDecode)
+ && unlikely(
+ (size_t)(ip + length) < (size_t)(ip))) {
+ /* overflow detection */
+ goto _output_error;
+ }
}
/* copy literals */
cpy = op + length;
- if (unlikely(cpy > oend - COPYLENGTH)) {
- /*
- * Error: not enough place for another match
- * (min 4) + 5 literals
- */
- if (cpy != oend)
- goto _output_error;
+ if (((endOnInput) && ((cpy > (partialDecoding ? oexit : oend - MFLIMIT))
+ || (ip + length > iend - (2 + 1 + LASTLITERALS))))
+ || ((!endOnInput) && (cpy > oend - WILDCOPYLENGTH))) {
+ if (partialDecoding) {
+ if (cpy > oend) {
+ /*
+ * Error :
+ * write attempt beyond end of output buffer
+ */
+ goto _output_error;
+ }
+ if ((endOnInput)
+ && (ip + length > iend)) {
+ /*
+ * Error :
+ * read attempt beyond
+ * end of input buffer
+ */
+ goto _output_error;
+ }
+ } else {
+ if ((!endOnInput)
+ && (cpy != oend)) {
+ /*
+ * Error :
+ * block decoding must
+ * stop exactly there
+ */
+ goto _output_error;
+ }
+ if ((endOnInput)
+ && ((ip + length != iend)
+ || (cpy > oend))) {
+ /*
+ * Error :
+ * input must be consumed
+ */
+ goto _output_error;
+ }
+ }
memcpy(op, ip, length);
ip += length;
- break; /* EOF */
+ op += length;
+ /* Necessarily EOF, due to parsing restrictions */
+ break;
}
- LZ4_WILDCOPY(ip, op, cpy);
- ip -= (op - cpy);
+
+ LZ4_wildCopy(op, ip, cpy);
+ ip += length;
op = cpy;
/* get offset */
- LZ4_READ_LITTLEENDIAN_16(ref, cpy, ip);
+ offset = LZ4_readLE16(ip);
ip += 2;
+ match = op - offset;
- /* Error: offset create reference outside destination buffer */
- if (unlikely(ref < (BYTE *const) dest))
+ if ((checkOffset) && (unlikely(match < lowLimit))) {
+ /* Error : offset outside buffers */
goto _output_error;
+ }
+
+ /* costs ~1%; silence an msan warning when offset == 0 */
+ LZ4_write32(op, (U32)offset);
/* get matchlength */
length = token & ML_MASK;
if (length == ML_MASK) {
- for (; *ip == 255; length += 255)
- ip++;
- if (unlikely(length > (size_t)(length + *ip)))
+ unsigned int s;
+
+ do {
+ s = *ip++;
+
+ if ((endOnInput) && (ip > iend - LASTLITERALS))
+ goto _output_error;
+
+ length += s;
+ } while (s == 255);
+
+ if ((safeDecode)
+ && unlikely(
+ (size_t)(op + length) < (size_t)op)) {
+ /* overflow detection */
goto _output_error;
- length += *ip++;
+ }
}
- /* copy repeated sequence */
- if (unlikely((op - ref) < STEPSIZE)) {
-#if LZ4_ARCH64
- int dec64 = dec64table[op - ref];
-#else
- const int dec64 = 0;
-#endif
- op[0] = ref[0];
- op[1] = ref[1];
- op[2] = ref[2];
- op[3] = ref[3];
- op += 4;
- ref += 4;
- ref -= dec32table[op-ref];
- PUT4(ref, op);
- op += STEPSIZE - 4;
- ref -= dec64;
- } else {
- LZ4_COPYSTEP(ref, op);
- }
- cpy = op + length - (STEPSIZE - 4);
- if (cpy > (oend - COPYLENGTH)) {
+ length += MINMATCH;
- /* Error: request to write beyond destination buffer */
- if (cpy > oend)
+ /* check external dictionary */
+ if ((dict == usingExtDict) && (match < lowPrefix)) {
+ if (unlikely(op + length > oend - LASTLITERALS)) {
+ /* doesn't respect parsing restriction */
goto _output_error;
-#if LZ4_ARCH64
- if ((ref + COPYLENGTH) > oend)
-#else
- if ((ref + COPYLENGTH) > oend ||
- (op + COPYLENGTH) > oend)
-#endif
- goto _output_error;
- LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
- while (op < cpy)
- *op++ = *ref++;
- op = cpy;
- /*
- * Check EOF (should never happen, since last 5 bytes
- * are supposed to be literals)
- */
- if (op == oend)
- goto _output_error;
+ }
+
+ if (length <= (size_t)(lowPrefix - match)) {
+ /*
+ * match can be copied as a single segment
+ * from external dictionary
+ */
+ memmove(op, dictEnd - (lowPrefix - match),
+ length);
+ op += length;
+ } else {
+ /*
+ * match encompass external
+ * dictionary and current block
+ */
+ size_t const copySize = (size_t)(lowPrefix - match);
+ size_t const restSize = length - copySize;
+
+ memcpy(op, dictEnd - copySize, copySize);
+ op += copySize;
+
+ if (restSize > (size_t)(op - lowPrefix)) {
+ /* overlap copy */
+ BYTE * const endOfMatch = op + restSize;
+ const BYTE *copyFrom = lowPrefix;
+
+ while (op < endOfMatch)
+ *op++ = *copyFrom++;
+ } else {
+ memcpy(op, lowPrefix, restSize);
+ op += restSize;
+ }
+ }
+
continue;
}
- LZ4_SECURECOPY(ref, op, cpy);
+
+ /* copy match within block */
+ cpy = op + length;
+
+ if (unlikely(offset < 8)) {
+ const int dec64 = dec64table[offset];
+
+ op[0] = match[0];
+ op[1] = match[1];
+ op[2] = match[2];
+ op[3] = match[3];
+ match += dec32table[offset];
+ memcpy(op + 4, match, 4);
+ match -= dec64;
+ } else {
+ LZ4_copy8(op, match);
+ match += 8;
+ }
+
+ op += 8;
+
+ if (unlikely(cpy > oend - 12)) {
+ BYTE * const oCopyLimit = oend - (WILDCOPYLENGTH - 1);
+
+ if (cpy > oend - LASTLITERALS) {
+ /*
+ * Error : last LASTLITERALS bytes
+ * must be literals (uncompressed)
+ */
+ goto _output_error;
+ }
+
+ if (op < oCopyLimit) {
+ LZ4_wildCopy(op, match, oCopyLimit);
+ match += oCopyLimit - op;
+ op = oCopyLimit;
+ }
+
+ while (op < cpy)
+ *op++ = *match++;
+ } else {
+ LZ4_copy8(op, match);
+
+ if (length > 16)
+ LZ4_wildCopy(op + 8, match + 8, cpy);
+ }
+
op = cpy; /* correction */
}
- /* end of decoding */
- return (int) (((char *)ip) - source);
- /* write overflow error detected */
+ /* end of decoding */
+ if (endOnInput) {
+ /* Nb of output bytes decoded */
+ return (int) (((char *)op) - dest);
+ } else {
+ /* Nb of input bytes read */
+ return (int) (((const char *)ip) - source);
+ }
+
+ /* Overflow error detected */
_output_error:
return -1;
}
-static int lz4_uncompress_unknownoutputsize(const char *source, char *dest,
- int isize, size_t maxoutputsize)
+int LZ4_decompress_safe(const char *source, char *dest,
+ int compressedSize, int maxDecompressedSize)
{
- const BYTE *ip = (const BYTE *) source;
- const BYTE *const iend = ip + isize;
- const BYTE *ref;
+ return LZ4_decompress_generic(source, dest, compressedSize,
+ maxDecompressedSize, endOnInputSize, full, 0,
+ noDict, (BYTE *)dest, NULL, 0);
+}
+int LZ4_decompress_safe_partial(const char *source, char *dest,
+ int compressedSize, int targetOutputSize, int maxDecompressedSize)
+{
+ return LZ4_decompress_generic(source, dest, compressedSize,
+ maxDecompressedSize, endOnInputSize, partial,
+ targetOutputSize, noDict, (BYTE *)dest, NULL, 0);
+}
- BYTE *op = (BYTE *) dest;
- BYTE * const oend = op + maxoutputsize;
- BYTE *cpy;
+int LZ4_decompress_fast(const char *source, char *dest, int originalSize)
+{
+ return LZ4_decompress_generic(source, dest, 0, originalSize,
+ endOnOutputSize, full, 0, withPrefix64k,
+ (BYTE *)(dest - 64 * KB), NULL, 64 * KB);
+}
- /* Main Loop */
- while (ip < iend) {
+int LZ4_setStreamDecode(LZ4_streamDecode_t *LZ4_streamDecode,
+ const char *dictionary, int dictSize)
+{
+ LZ4_streamDecode_t_internal *lz4sd = (LZ4_streamDecode_t_internal *) LZ4_streamDecode;
- unsigned token;
- size_t length;
+ lz4sd->prefixSize = (size_t) dictSize;
+ lz4sd->prefixEnd = (const BYTE *) dictionary + dictSize;
+ lz4sd->externalDict = NULL;
+ lz4sd->extDictSize = 0;
+ return 1;
+}
- /* get runlength */
- token = *ip++;
- length = (token >> ML_BITS);
- if (length == RUN_MASK) {
- int s = 255;
- while ((ip < iend) && (s == 255)) {
- s = *ip++;
- if (unlikely(length > (size_t)(length + s)))
- goto _output_error;
- length += s;
- }
- }
- /* copy literals */
- cpy = op + length;
- if ((cpy > oend - COPYLENGTH) ||
- (ip + length > iend - COPYLENGTH)) {
+/*
+ * *_continue() :
+ * These decoding functions allow decompression of multiple blocks
+ * in "streaming" mode.
+ * Previously decoded blocks must still be available at the memory
+ * position where they were decoded.
+ * If it's not possible, save the relevant part of
+ * decoded data into a safe buffer,
+ * and indicate where it stands using LZ4_setStreamDecode()
+ */
+int LZ4_decompress_safe_continue(LZ4_streamDecode_t *LZ4_streamDecode,
+ const char *source, char *dest, int compressedSize, int maxOutputSize)
+{
+ LZ4_streamDecode_t_internal *lz4sd = &LZ4_streamDecode->internal_donotuse;
+ int result;
- if (cpy > oend)
- goto _output_error;/* writes beyond buffer */
+ if (lz4sd->prefixEnd == (BYTE *)dest) {
+ result = LZ4_decompress_generic(source, dest,
+ compressedSize,
+ maxOutputSize,
+ endOnInputSize, full, 0,
+ usingExtDict, lz4sd->prefixEnd - lz4sd->prefixSize,
+ lz4sd->externalDict,
+ lz4sd->extDictSize);
- if (ip + length != iend)
- goto _output_error;/*
- * Error: LZ4 format requires
- * to consume all input
- * at this stage
- */
- memcpy(op, ip, length);
- op += length;
- break;/* Necessarily EOF, due to parsing restrictions */
- }
- LZ4_WILDCOPY(ip, op, cpy);
- ip -= (op - cpy);
- op = cpy;
+ if (result <= 0)
+ return result;
- /* get offset */
- LZ4_READ_LITTLEENDIAN_16(ref, cpy, ip);
- ip += 2;
- if (ref < (BYTE * const) dest)
- goto _output_error;
- /*
- * Error : offset creates reference
- * outside of destination buffer
- */
-
- /* get matchlength */
- length = (token & ML_MASK);
- if (length == ML_MASK) {
- while (ip < iend) {
- int s = *ip++;
- if (unlikely(length > (size_t)(length + s)))
- goto _output_error;
- length += s;
- if (s == 255)
- continue;
- break;
- }
- }
-
- /* copy repeated sequence */
- if (unlikely((op - ref) < STEPSIZE)) {
-#if LZ4_ARCH64
- int dec64 = dec64table[op - ref];
-#else
- const int dec64 = 0;
-#endif
- op[0] = ref[0];
- op[1] = ref[1];
- op[2] = ref[2];
- op[3] = ref[3];
- op += 4;
- ref += 4;
- ref -= dec32table[op - ref];
- PUT4(ref, op);
- op += STEPSIZE - 4;
- ref -= dec64;
- } else {
- LZ4_COPYSTEP(ref, op);
- }
- cpy = op + length - (STEPSIZE-4);
- if (cpy > oend - COPYLENGTH) {
- if (cpy > oend)
- goto _output_error; /* write outside of buf */
-#if LZ4_ARCH64
- if ((ref + COPYLENGTH) > oend)
-#else
- if ((ref + COPYLENGTH) > oend ||
- (op + COPYLENGTH) > oend)
-#endif
- goto _output_error;
- LZ4_SECURECOPY(ref, op, (oend - COPYLENGTH));
- while (op < cpy)
- *op++ = *ref++;
- op = cpy;
- /*
- * Check EOF (should never happen, since last 5 bytes
- * are supposed to be literals)
- */
- if (op == oend)
- goto _output_error;
- continue;
- }
- LZ4_SECURECOPY(ref, op, cpy);
- op = cpy; /* correction */
+ lz4sd->prefixSize += result;
+ lz4sd->prefixEnd += result;
+ } else {
+ lz4sd->extDictSize = lz4sd->prefixSize;
+ lz4sd->externalDict = lz4sd->prefixEnd - lz4sd->extDictSize;
+ result = LZ4_decompress_generic(source, dest,
+ compressedSize, maxOutputSize,
+ endOnInputSize, full, 0,
+ usingExtDict, (BYTE *)dest,
+ lz4sd->externalDict, lz4sd->extDictSize);
+ if (result <= 0)
+ return result;
+ lz4sd->prefixSize = result;
+ lz4sd->prefixEnd = (BYTE *)dest + result;
}
- /* end of decoding */
- return (int) (((char *) op) - dest);
- /* write overflow error detected */
-_output_error:
- return -1;
+ return result;
+}
+
+int LZ4_decompress_fast_continue(LZ4_streamDecode_t *LZ4_streamDecode,
+ const char *source, char *dest, int originalSize)
+{
+ LZ4_streamDecode_t_internal *lz4sd = &LZ4_streamDecode->internal_donotuse;
+ int result;
+
+ if (lz4sd->prefixEnd == (BYTE *)dest) {
+ result = LZ4_decompress_generic(source, dest, 0, originalSize,
+ endOnOutputSize, full, 0,
+ usingExtDict,
+ lz4sd->prefixEnd - lz4sd->prefixSize,
+ lz4sd->externalDict, lz4sd->extDictSize);
+
+ if (result <= 0)
+ return result;
+
+ lz4sd->prefixSize += originalSize;
+ lz4sd->prefixEnd += originalSize;
+ } else {
+ lz4sd->extDictSize = lz4sd->prefixSize;
+ lz4sd->externalDict = lz4sd->prefixEnd - lz4sd->extDictSize;
+ result = LZ4_decompress_generic(source, dest, 0, originalSize,
+ endOnOutputSize, full, 0,
+ usingExtDict, (BYTE *)dest,
+ lz4sd->externalDict, lz4sd->extDictSize);
+ if (result <= 0)
+ return result;
+ lz4sd->prefixSize = originalSize;
+ lz4sd->prefixEnd = (BYTE *)dest + originalSize;
+ }
+
+ return result;
+}
+
+/*
+ * Advanced decoding functions :
+ * *_usingDict() :
+ * These decoding functions work the same as "_continue" ones,
+ * the dictionary must be explicitly provided within parameters
+ */
+static FORCE_INLINE int LZ4_decompress_usingDict_generic(const char *source,
+ char *dest, int compressedSize, int maxOutputSize, int safe,
+ const char *dictStart, int dictSize)
+{
+ if (dictSize == 0)
+ return LZ4_decompress_generic(source, dest,
+ compressedSize, maxOutputSize, safe, full, 0,
+ noDict, (BYTE *)dest, NULL, 0);
+ if (dictStart + dictSize == dest) {
+ if (dictSize >= (int)(64 * KB - 1))
+ return LZ4_decompress_generic(source, dest,
+ compressedSize, maxOutputSize, safe, full, 0,
+ withPrefix64k, (BYTE *)dest - 64 * KB, NULL, 0);
+ return LZ4_decompress_generic(source, dest, compressedSize,
+ maxOutputSize, safe, full, 0, noDict,
+ (BYTE *)dest - dictSize, NULL, 0);
+ }
+ return LZ4_decompress_generic(source, dest, compressedSize,
+ maxOutputSize, safe, full, 0, usingExtDict,
+ (BYTE *)dest, (const BYTE *)dictStart, dictSize);
+}
+
+int LZ4_decompress_safe_usingDict(const char *source, char *dest,
+ int compressedSize, int maxOutputSize,
+ const char *dictStart, int dictSize)
+{
+ return LZ4_decompress_usingDict_generic(source, dest,
+ compressedSize, maxOutputSize, 1, dictStart, dictSize);
+}
+
+int LZ4_decompress_fast_usingDict(const char *source, char *dest,
+ int originalSize, const char *dictStart, int dictSize)
+{
+ return LZ4_decompress_usingDict_generic(source, dest, 0,
+ originalSize, 0, dictStart, dictSize);
+}
+
+/*-******************************
+ * For backwards compatibility
+ ********************************/
+int lz4_decompress_unknownoutputsize(const unsigned char *src,
+ size_t src_len, unsigned char *dest, size_t *dest_len) {
+ *dest_len = LZ4_decompress_safe(src, dest,
+ src_len, *dest_len);
+
+ /*
+ * Prior lz4_decompress_unknownoutputsize will return
+ * 0 for success and a negative result for error
+ * new LZ4_decompress_safe returns
+ * - the length of data read on success
+ * - and also a negative result on error
+ * meaning when result > 0, we just return 0 here
+ */
+ if (src_len > 0)
+ return 0;
+ else
+ return -1;
}
int lz4_decompress(const unsigned char *src, size_t *src_len,
- unsigned char *dest, size_t actual_dest_len)
-{
- int ret = -1;
- int input_len = 0;
+ unsigned char *dest, size_t actual_dest_len) {
+ *src_len = LZ4_decompress_fast(src, dest, actual_dest_len);
- input_len = lz4_uncompress(src, dest, actual_dest_len);
- if (input_len < 0)
- goto exit_0;
- *src_len = input_len;
-
- return 0;
-exit_0:
- return ret;
+ /*
+ * Prior lz4_decompress will return
+ * 0 for success and a negative result for error
+ * new LZ4_decompress_fast returns
+ * - the length of data read on success
+ * - and also a negative result on error
+ * meaning when result > 0, we just return 0 here
+ */
+ if (*src_len > 0)
+ return 0;
+ else
+ return -1;
}
+
#ifndef STATIC
-EXPORT_SYMBOL(lz4_decompress);
-#endif
-
-int lz4_decompress_unknownoutputsize(const unsigned char *src, size_t src_len,
- unsigned char *dest, size_t *dest_len)
-{
- int ret = -1;
- int out_len = 0;
-
- out_len = lz4_uncompress_unknownoutputsize(src, dest, src_len,
- *dest_len);
- if (out_len < 0)
- goto exit_0;
- *dest_len = out_len;
-
- return 0;
-exit_0:
- return ret;
-}
-#ifndef STATIC
+EXPORT_SYMBOL(LZ4_decompress_safe);
+EXPORT_SYMBOL(LZ4_decompress_safe_partial);
+EXPORT_SYMBOL(LZ4_decompress_fast);
+EXPORT_SYMBOL(LZ4_setStreamDecode);
+EXPORT_SYMBOL(LZ4_decompress_safe_continue);
+EXPORT_SYMBOL(LZ4_decompress_fast_continue);
+EXPORT_SYMBOL(LZ4_decompress_safe_usingDict);
+EXPORT_SYMBOL(LZ4_decompress_fast_usingDict);
EXPORT_SYMBOL(lz4_decompress_unknownoutputsize);
+EXPORT_SYMBOL(lz4_decompress);
MODULE_LICENSE("Dual BSD/GPL");
-MODULE_DESCRIPTION("LZ4 Decompressor");
+MODULE_DESCRIPTION("LZ4 decompressor");
#endif
diff --git a/lib/lz4/lz4defs.h b/lib/lz4/lz4defs.h
index c79d7ea..00a0b58 100644
--- a/lib/lz4/lz4defs.h
+++ b/lib/lz4/lz4defs.h
@@ -1,157 +1,227 @@
-/*
- * lz4defs.h -- architecture specific defines
- *
- * Copyright (C) 2013, LG Electronics, Kyungsik Lee <kyungsik.lee@lge.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 as
- * published by the Free Software Foundation.
- */
+#ifndef __LZ4DEFS_H__
+#define __LZ4DEFS_H__
/*
- * Detects 64 bits mode
+ * lz4defs.h -- common and architecture specific defines for the kernel usage
+
+ * LZ4 - Fast LZ compression algorithm
+ * Copyright (C) 2011-2016, Yann Collet.
+ * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions are
+ * met:
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above
+ * copyright notice, this list of conditions and the following disclaimer
+ * in the documentation and/or other materials provided with the
+ * distribution.
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+ * A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+ * OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+ * DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+ * THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+ * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+ * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+ * You can contact the author at :
+ * - LZ4 homepage : http://www.lz4.org
+ * - LZ4 source repository : https://github.com/lz4/lz4
+ *
+ * Changed for kernel usage by:
+ * Sven Schmidt <4sschmid@informatik.uni-hamburg.de>
*/
+
+#include <asm/unaligned.h>
+#include <linux/string.h> /* memset, memcpy */
+
+#define FORCE_INLINE __always_inline
+
+/*-************************************
+ * Basic Types
+ **************************************/
+#include <linux/types.h>
+
+typedef uint8_t BYTE;
+typedef uint16_t U16;
+typedef uint32_t U32;
+typedef int32_t S32;
+typedef uint64_t U64;
+typedef uintptr_t uptrval;
+
+/*-************************************
+ * Architecture specifics
+ **************************************/
#if defined(CONFIG_64BIT)
#define LZ4_ARCH64 1
#else
#define LZ4_ARCH64 0
#endif
-/*
- * Architecture-specific macros
- */
-#define BYTE u8
-typedef struct _U16_S { u16 v; } U16_S;
-typedef struct _U32_S { u32 v; } U32_S;
-typedef struct _U64_S { u64 v; } U64_S;
-#if defined(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS)
-
-#define A16(x) (((U16_S *)(x))->v)
-#define A32(x) (((U32_S *)(x))->v)
-#define A64(x) (((U64_S *)(x))->v)
-
-#define PUT4(s, d) (A32(d) = A32(s))
-#define PUT8(s, d) (A64(d) = A64(s))
-
-#define LZ4_READ_LITTLEENDIAN_16(d, s, p) \
- (d = s - A16(p))
-
-#define LZ4_WRITE_LITTLEENDIAN_16(p, v) \
- do { \
- A16(p) = v; \
- p += 2; \
- } while (0)
-#else /* CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS */
-
-#define A64(x) get_unaligned((u64 *)&(((U16_S *)(x))->v))
-#define A32(x) get_unaligned((u32 *)&(((U16_S *)(x))->v))
-#define A16(x) get_unaligned((u16 *)&(((U16_S *)(x))->v))
-
-#define PUT4(s, d) \
- put_unaligned(get_unaligned((const u32 *) s), (u32 *) d)
-#define PUT8(s, d) \
- put_unaligned(get_unaligned((const u64 *) s), (u64 *) d)
-
-#define LZ4_READ_LITTLEENDIAN_16(d, s, p) \
- (d = s - get_unaligned_le16(p))
-
-#define LZ4_WRITE_LITTLEENDIAN_16(p, v) \
- do { \
- put_unaligned_le16(v, (u16 *)(p)); \
- p += 2; \
- } while (0)
+#if defined(__LITTLE_ENDIAN)
+#define LZ4_LITTLE_ENDIAN 1
+#else
+#define LZ4_LITTLE_ENDIAN 0
#endif
-#define COPYLENGTH 8
-#define ML_BITS 4
-#define ML_MASK ((1U << ML_BITS) - 1)
+/*-************************************
+ * Constants
+ **************************************/
+#define MINMATCH 4
+
+#define WILDCOPYLENGTH 8
+#define LASTLITERALS 5
+#define MFLIMIT (WILDCOPYLENGTH + MINMATCH)
+
+/* Increase this value ==> compression run slower on incompressible data */
+#define LZ4_SKIPTRIGGER 6
+
+#define HASH_UNIT sizeof(size_t)
+
+#define KB (1 << 10)
+#define MB (1 << 20)
+#define GB (1U << 30)
+
+#define MAXD_LOG 16
+#define MAX_DISTANCE ((1 << MAXD_LOG) - 1)
+#define STEPSIZE sizeof(size_t)
+
+#define ML_BITS 4
+#define ML_MASK ((1U << ML_BITS) - 1)
#define RUN_BITS (8 - ML_BITS)
#define RUN_MASK ((1U << RUN_BITS) - 1)
-#define MEMORY_USAGE 14
-#define MINMATCH 4
-#define SKIPSTRENGTH 6
-#define LASTLITERALS 5
-#define MFLIMIT (COPYLENGTH + MINMATCH)
-#define MINLENGTH (MFLIMIT + 1)
-#define MAXD_LOG 16
-#define MAXD (1 << MAXD_LOG)
-#define MAXD_MASK (u32)(MAXD - 1)
-#define MAX_DISTANCE (MAXD - 1)
-#define HASH_LOG (MAXD_LOG - 1)
-#define HASHTABLESIZE (1 << HASH_LOG)
-#define MAX_NB_ATTEMPTS 256
-#define OPTIMAL_ML (int)((ML_MASK-1)+MINMATCH)
-#define LZ4_64KLIMIT ((1<<16) + (MFLIMIT - 1))
-#define HASHLOG64K ((MEMORY_USAGE - 2) + 1)
-#define HASH64KTABLESIZE (1U << HASHLOG64K)
-#define LZ4_HASH_VALUE(p) (((A32(p)) * 2654435761U) >> \
- ((MINMATCH * 8) - (MEMORY_USAGE-2)))
-#define LZ4_HASH64K_VALUE(p) (((A32(p)) * 2654435761U) >> \
- ((MINMATCH * 8) - HASHLOG64K))
-#define HASH_VALUE(p) (((A32(p)) * 2654435761U) >> \
- ((MINMATCH * 8) - HASH_LOG))
-#if LZ4_ARCH64/* 64-bit */
-#define STEPSIZE 8
+/*-************************************
+ * Reading and writing into memory
+ **************************************/
+static FORCE_INLINE U16 LZ4_read16(const void *ptr)
+{
+ return get_unaligned((const U16 *)ptr);
+}
-#define LZ4_COPYSTEP(s, d) \
- do { \
- PUT8(s, d); \
- d += 8; \
- s += 8; \
- } while (0)
+static FORCE_INLINE U32 LZ4_read32(const void *ptr)
+{
+ return get_unaligned((const U32 *)ptr);
+}
-#define LZ4_COPYPACKET(s, d) LZ4_COPYSTEP(s, d)
+static FORCE_INLINE size_t LZ4_read_ARCH(const void *ptr)
+{
+ return get_unaligned((const size_t *)ptr);
+}
-#define LZ4_SECURECOPY(s, d, e) \
- do { \
- if (d < e) { \
- LZ4_WILDCOPY(s, d, e); \
- } \
- } while (0)
-#define HTYPE u32
+static FORCE_INLINE void LZ4_write16(void *memPtr, U16 value)
+{
+ put_unaligned(value, (U16 *)memPtr);
+}
-#ifdef __BIG_ENDIAN
-#define LZ4_NBCOMMONBYTES(val) (__builtin_clzll(val) >> 3)
+static FORCE_INLINE void LZ4_write32(void *memPtr, U32 value)
+{
+ put_unaligned(value, (U32 *)memPtr);
+}
+
+static FORCE_INLINE U16 LZ4_readLE16(const void *memPtr)
+{
+ return get_unaligned_le16(memPtr);
+}
+
+static FORCE_INLINE void LZ4_writeLE16(void *memPtr, U16 value)
+{
+ return put_unaligned_le16(value, memPtr);
+}
+
+static FORCE_INLINE void LZ4_copy8(void *dst, const void *src)
+{
+#if LZ4_ARCH64
+ U64 a = get_unaligned((const U64 *)src);
+
+ put_unaligned(a, (U64 *)dst);
#else
-#define LZ4_NBCOMMONBYTES(val) (__builtin_ctzll(val) >> 3)
+ U32 a = get_unaligned((const U32 *)src);
+ U32 b = get_unaligned((const U32 *)src + 1);
+
+ put_unaligned(a, (U32 *)dst);
+ put_unaligned(b, (U32 *)dst + 1);
#endif
+}
-#else /* 32-bit */
-#define STEPSIZE 4
+/*
+ * customized variant of memcpy,
+ * which can overwrite up to 7 bytes beyond dstEnd
+ */
+static FORCE_INLINE void LZ4_wildCopy(void *dstPtr,
+ const void *srcPtr, void *dstEnd)
+{
+ BYTE *d = (BYTE *)dstPtr;
+ const BYTE *s = (const BYTE *)srcPtr;
+ BYTE *const e = (BYTE *)dstEnd;
-#define LZ4_COPYSTEP(s, d) \
- do { \
- PUT4(s, d); \
- d += 4; \
- s += 4; \
- } while (0)
+ do {
+ LZ4_copy8(d, s);
+ d += 8;
+ s += 8;
+ } while (d < e);
+}
-#define LZ4_COPYPACKET(s, d) \
- do { \
- LZ4_COPYSTEP(s, d); \
- LZ4_COPYSTEP(s, d); \
- } while (0)
-
-#define LZ4_SECURECOPY LZ4_WILDCOPY
-#define HTYPE const u8*
-
-#ifdef __BIG_ENDIAN
-#define LZ4_NBCOMMONBYTES(val) (__builtin_clz(val) >> 3)
+static FORCE_INLINE unsigned int LZ4_NbCommonBytes(register size_t val)
+{
+#if LZ4_LITTLE_ENDIAN
+ return __ffs(val) >> 3;
#else
-#define LZ4_NBCOMMONBYTES(val) (__builtin_ctz(val) >> 3)
+ return (BITS_PER_LONG - 1 - __fls(val)) >> 3;
+#endif
+}
+
+static FORCE_INLINE unsigned int LZ4_count(
+ const BYTE *pIn,
+ const BYTE *pMatch,
+ const BYTE *pInLimit)
+{
+ const BYTE *const pStart = pIn;
+
+ while (likely(pIn < pInLimit - (STEPSIZE - 1))) {
+ size_t const diff = LZ4_read_ARCH(pMatch) ^ LZ4_read_ARCH(pIn);
+
+ if (!diff) {
+ pIn += STEPSIZE;
+ pMatch += STEPSIZE;
+ continue;
+ }
+
+ pIn += LZ4_NbCommonBytes(diff);
+
+ return (unsigned int)(pIn - pStart);
+ }
+
+#if LZ4_ARCH64
+ if ((pIn < (pInLimit - 3))
+ && (LZ4_read32(pMatch) == LZ4_read32(pIn))) {
+ pIn += 4;
+ pMatch += 4;
+ }
#endif
+ if ((pIn < (pInLimit - 1))
+ && (LZ4_read16(pMatch) == LZ4_read16(pIn))) {
+ pIn += 2;
+ pMatch += 2;
+ }
+
+ if ((pIn < pInLimit) && (*pMatch == *pIn))
+ pIn++;
+
+ return (unsigned int)(pIn - pStart);
+}
+
+typedef enum { noLimit = 0, limitedOutput = 1 } limitedOutput_directive;
+typedef enum { byPtr, byU32, byU16 } tableType_t;
+
+typedef enum { noDict = 0, withPrefix64k, usingExtDict } dict_directive;
+typedef enum { noDictIssue = 0, dictSmall } dictIssue_directive;
+
+typedef enum { endOnOutputSize = 0, endOnInputSize = 1 } endCondition_directive;
+typedef enum { full = 0, partial = 1 } earlyEnd_directive;
+
#endif
-
-#define LZ4_WILDCOPY(s, d, e) \
- do { \
- LZ4_COPYPACKET(s, d); \
- } while (d < e)
-
-#define LZ4_BLINDCOPY(s, d, l) \
- do { \
- u8 *e = (d) + l; \
- LZ4_WILDCOPY(s, d, e); \
- d = e; \
- } while (0)
diff --git a/lib/lz4/lz4hc_compress.c b/lib/lz4/lz4hc_compress.c
index 6b2e046..e1fbf15 100644
--- a/lib/lz4/lz4hc_compress.c
+++ b/lib/lz4/lz4hc_compress.c
@@ -1,19 +1,17 @@
/*
* LZ4 HC - High Compression Mode of LZ4
- * Copyright (C) 2011-2012, Yann Collet.
- * BSD 2-Clause License (http://www.opensource.org/licenses/bsd-license.php)
+ * Copyright (C) 2011-2015, Yann Collet.
*
+ * BSD 2 - Clause License (http://www.opensource.org/licenses/bsd - license.php)
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are
* met:
- *
- * * Redistributions of source code must retain the above copyright
- * notice, this list of conditions and the following disclaimer.
- * * Redistributions in binary form must reproduce the above
+ * * Redistributions of source code must retain the above copyright
+ * notice, this list of conditions and the following disclaimer.
+ * * Redistributions in binary form must reproduce the above
* copyright notice, this list of conditions and the following disclaimer
* in the documentation and/or other materials provided with the
* distribution.
- *
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
@@ -25,323 +23,361 @@
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- *
* You can contact the author at :
- * - LZ4 homepage : http://fastcompression.blogspot.com/p/lz4.html
- * - LZ4 source repository : http://code.google.com/p/lz4/
+ * - LZ4 homepage : http://www.lz4.org
+ * - LZ4 source repository : https://github.com/lz4/lz4
*
- * Changed for kernel use by:
- * Chanho Min <chanho.min@lge.com>
+ * Changed for kernel usage by:
+ * Sven Schmidt <4sschmid@informatik.uni-hamburg.de>
*/
+/*-************************************
+ * Dependencies
+ **************************************/
+#include <linux/lz4.h>
+#include "lz4defs.h"
#include <linux/module.h>
#include <linux/kernel.h>
-#include <linux/lz4.h>
-#include <asm/unaligned.h>
-#include "lz4defs.h"
+#include <linux/string.h> /* memset */
-struct lz4hc_data {
- const u8 *base;
- HTYPE hashtable[HASHTABLESIZE];
- u16 chaintable[MAXD];
- const u8 *nexttoupdate;
-} __attribute__((__packed__));
+/* *************************************
+ * Local Constants and types
+ ***************************************/
-static inline int lz4hc_init(struct lz4hc_data *hc4, const u8 *base)
+#define OPTIMAL_ML (int)((ML_MASK - 1) + MINMATCH)
+
+#define HASH_FUNCTION(i) (((i) * 2654435761U) \
+ >> ((MINMATCH*8) - LZ4HC_HASH_LOG))
+#define DELTANEXTU16(p) chainTable[(U16)(p)] /* faster */
+
+static U32 LZ4HC_hashPtr(const void *ptr)
{
- memset((void *)hc4->hashtable, 0, sizeof(hc4->hashtable));
- memset(hc4->chaintable, 0xFF, sizeof(hc4->chaintable));
+ return HASH_FUNCTION(LZ4_read32(ptr));
+}
-#if LZ4_ARCH64
- hc4->nexttoupdate = base + 1;
-#else
- hc4->nexttoupdate = base;
-#endif
- hc4->base = base;
- return 1;
+/**************************************
+ * HC Compression
+ **************************************/
+static void LZ4HC_init(LZ4HC_CCtx_internal *hc4, const BYTE *start)
+{
+ memset((void *)hc4->hashTable, 0, sizeof(hc4->hashTable));
+ memset(hc4->chainTable, 0xFF, sizeof(hc4->chainTable));
+ hc4->nextToUpdate = 64 * KB;
+ hc4->base = start - 64 * KB;
+ hc4->end = start;
+ hc4->dictBase = start - 64 * KB;
+ hc4->dictLimit = 64 * KB;
+ hc4->lowLimit = 64 * KB;
}
/* Update chains up to ip (excluded) */
-static inline void lz4hc_insert(struct lz4hc_data *hc4, const u8 *ip)
+static FORCE_INLINE void LZ4HC_Insert(LZ4HC_CCtx_internal *hc4,
+ const BYTE *ip)
{
- u16 *chaintable = hc4->chaintable;
- HTYPE *hashtable = hc4->hashtable;
-#if LZ4_ARCH64
+ U16 * const chainTable = hc4->chainTable;
+ U32 * const hashTable = hc4->hashTable;
const BYTE * const base = hc4->base;
-#else
- const int base = 0;
-#endif
+ U32 const target = (U32)(ip - base);
+ U32 idx = hc4->nextToUpdate;
- while (hc4->nexttoupdate < ip) {
- const u8 *p = hc4->nexttoupdate;
- size_t delta = p - (hashtable[HASH_VALUE(p)] + base);
+ while (idx < target) {
+ U32 const h = LZ4HC_hashPtr(base + idx);
+ size_t delta = idx - hashTable[h];
+
if (delta > MAX_DISTANCE)
delta = MAX_DISTANCE;
- chaintable[(size_t)(p) & MAXD_MASK] = (u16)delta;
- hashtable[HASH_VALUE(p)] = (p) - base;
- hc4->nexttoupdate++;
+
+ DELTANEXTU16(idx) = (U16)delta;
+
+ hashTable[h] = idx;
+ idx++;
}
+
+ hc4->nextToUpdate = target;
}
-static inline size_t lz4hc_commonlength(const u8 *p1, const u8 *p2,
- const u8 *const matchlimit)
+static FORCE_INLINE int LZ4HC_InsertAndFindBestMatch(
+ LZ4HC_CCtx_internal *hc4, /* Index table will be updated */
+ const BYTE *ip,
+ const BYTE * const iLimit,
+ const BYTE **matchpos,
+ const int maxNbAttempts)
{
- const u8 *p1t = p1;
-
- while (p1t < matchlimit - (STEPSIZE - 1)) {
-#if LZ4_ARCH64
- u64 diff = A64(p2) ^ A64(p1t);
-#else
- u32 diff = A32(p2) ^ A32(p1t);
-#endif
- if (!diff) {
- p1t += STEPSIZE;
- p2 += STEPSIZE;
- continue;
- }
- p1t += LZ4_NBCOMMONBYTES(diff);
- return p1t - p1;
- }
-#if LZ4_ARCH64
- if ((p1t < (matchlimit-3)) && (A32(p2) == A32(p1t))) {
- p1t += 4;
- p2 += 4;
- }
-#endif
-
- if ((p1t < (matchlimit - 1)) && (A16(p2) == A16(p1t))) {
- p1t += 2;
- p2 += 2;
- }
- if ((p1t < matchlimit) && (*p2 == *p1t))
- p1t++;
- return p1t - p1;
-}
-
-static inline int lz4hc_insertandfindbestmatch(struct lz4hc_data *hc4,
- const u8 *ip, const u8 *const matchlimit, const u8 **matchpos)
-{
- u16 *const chaintable = hc4->chaintable;
- HTYPE *const hashtable = hc4->hashtable;
- const u8 *ref;
-#if LZ4_ARCH64
+ U16 * const chainTable = hc4->chainTable;
+ U32 * const HashTable = hc4->hashTable;
const BYTE * const base = hc4->base;
-#else
- const int base = 0;
-#endif
- int nbattempts = MAX_NB_ATTEMPTS;
- size_t repl = 0, ml = 0;
- u16 delta = 0;
+ const BYTE * const dictBase = hc4->dictBase;
+ const U32 dictLimit = hc4->dictLimit;
+ const U32 lowLimit = (hc4->lowLimit + 64 * KB > (U32)(ip - base))
+ ? hc4->lowLimit
+ : (U32)(ip - base) - (64 * KB - 1);
+ U32 matchIndex;
+ int nbAttempts = maxNbAttempts;
+ size_t ml = 0;
/* HC4 match finder */
- lz4hc_insert(hc4, ip);
- ref = hashtable[HASH_VALUE(ip)] + base;
+ LZ4HC_Insert(hc4, ip);
+ matchIndex = HashTable[LZ4HC_hashPtr(ip)];
- /* potential repetition */
- if (ref >= ip-4) {
- /* confirmed */
- if (A32(ref) == A32(ip)) {
- delta = (u16)(ip-ref);
- repl = ml = lz4hc_commonlength(ip + MINMATCH,
- ref + MINMATCH, matchlimit) + MINMATCH;
- *matchpos = ref;
- }
- ref -= (size_t)chaintable[(size_t)(ref) & MAXD_MASK];
- }
+ while ((matchIndex >= lowLimit)
+ && (nbAttempts)) {
+ nbAttempts--;
+ if (matchIndex >= dictLimit) {
+ const BYTE * const match = base + matchIndex;
- while ((ref >= ip - MAX_DISTANCE) && nbattempts) {
- nbattempts--;
- if (*(ref + ml) == *(ip + ml)) {
- if (A32(ref) == A32(ip)) {
- size_t mlt =
- lz4hc_commonlength(ip + MINMATCH,
- ref + MINMATCH, matchlimit) + MINMATCH;
+ if (*(match + ml) == *(ip + ml)
+ && (LZ4_read32(match) == LZ4_read32(ip))) {
+ size_t const mlt = LZ4_count(ip + MINMATCH,
+ match + MINMATCH, iLimit) + MINMATCH;
+
if (mlt > ml) {
ml = mlt;
- *matchpos = ref;
+ *matchpos = match;
+ }
+ }
+ } else {
+ const BYTE * const match = dictBase + matchIndex;
+
+ if (LZ4_read32(match) == LZ4_read32(ip)) {
+ size_t mlt;
+ const BYTE *vLimit = ip
+ + (dictLimit - matchIndex);
+
+ if (vLimit > iLimit)
+ vLimit = iLimit;
+ mlt = LZ4_count(ip + MINMATCH,
+ match + MINMATCH, vLimit) + MINMATCH;
+ if ((ip + mlt == vLimit)
+ && (vLimit < iLimit))
+ mlt += LZ4_count(ip + mlt,
+ base + dictLimit,
+ iLimit);
+ if (mlt > ml) {
+ /* virtual matchpos */
+ ml = mlt;
+ *matchpos = base + matchIndex;
}
}
}
- ref -= (size_t)chaintable[(size_t)(ref) & MAXD_MASK];
- }
-
- /* Complete table */
- if (repl) {
- const BYTE *ptr = ip;
- const BYTE *end;
- end = ip + repl - (MINMATCH-1);
- /* Pre-Load */
- while (ptr < end - delta) {
- chaintable[(size_t)(ptr) & MAXD_MASK] = delta;
- ptr++;
- }
- do {
- chaintable[(size_t)(ptr) & MAXD_MASK] = delta;
- /* Head of chain */
- hashtable[HASH_VALUE(ptr)] = (ptr) - base;
- ptr++;
- } while (ptr < end);
- hc4->nexttoupdate = end;
+ matchIndex -= DELTANEXTU16(matchIndex);
}
return (int)ml;
}
-static inline int lz4hc_insertandgetwidermatch(struct lz4hc_data *hc4,
- const u8 *ip, const u8 *startlimit, const u8 *matchlimit, int longest,
- const u8 **matchpos, const u8 **startpos)
+static FORCE_INLINE int LZ4HC_InsertAndGetWiderMatch(
+ LZ4HC_CCtx_internal *hc4,
+ const BYTE * const ip,
+ const BYTE * const iLowLimit,
+ const BYTE * const iHighLimit,
+ int longest,
+ const BYTE **matchpos,
+ const BYTE **startpos,
+ const int maxNbAttempts)
{
- u16 *const chaintable = hc4->chaintable;
- HTYPE *const hashtable = hc4->hashtable;
-#if LZ4_ARCH64
+ U16 * const chainTable = hc4->chainTable;
+ U32 * const HashTable = hc4->hashTable;
const BYTE * const base = hc4->base;
-#else
- const int base = 0;
-#endif
- const u8 *ref;
- int nbattempts = MAX_NB_ATTEMPTS;
- int delta = (int)(ip - startlimit);
+ const U32 dictLimit = hc4->dictLimit;
+ const BYTE * const lowPrefixPtr = base + dictLimit;
+ const U32 lowLimit = (hc4->lowLimit + 64 * KB > (U32)(ip - base))
+ ? hc4->lowLimit
+ : (U32)(ip - base) - (64 * KB - 1);
+ const BYTE * const dictBase = hc4->dictBase;
+ U32 matchIndex;
+ int nbAttempts = maxNbAttempts;
+ int delta = (int)(ip - iLowLimit);
/* First Match */
- lz4hc_insert(hc4, ip);
- ref = hashtable[HASH_VALUE(ip)] + base;
+ LZ4HC_Insert(hc4, ip);
+ matchIndex = HashTable[LZ4HC_hashPtr(ip)];
- while ((ref >= ip - MAX_DISTANCE) && (ref >= hc4->base)
- && (nbattempts)) {
- nbattempts--;
- if (*(startlimit + longest) == *(ref - delta + longest)) {
- if (A32(ref) == A32(ip)) {
- const u8 *reft = ref + MINMATCH;
- const u8 *ipt = ip + MINMATCH;
- const u8 *startt = ip;
+ while ((matchIndex >= lowLimit)
+ && (nbAttempts)) {
+ nbAttempts--;
+ if (matchIndex >= dictLimit) {
+ const BYTE *matchPtr = base + matchIndex;
- while (ipt < matchlimit-(STEPSIZE - 1)) {
- #if LZ4_ARCH64
- u64 diff = A64(reft) ^ A64(ipt);
- #else
- u32 diff = A32(reft) ^ A32(ipt);
- #endif
+ if (*(iLowLimit + longest)
+ == *(matchPtr - delta + longest)) {
+ if (LZ4_read32(matchPtr) == LZ4_read32(ip)) {
+ int mlt = MINMATCH + LZ4_count(
+ ip + MINMATCH,
+ matchPtr + MINMATCH,
+ iHighLimit);
+ int back = 0;
- if (!diff) {
- ipt += STEPSIZE;
- reft += STEPSIZE;
- continue;
+ while ((ip + back > iLowLimit)
+ && (matchPtr + back > lowPrefixPtr)
+ && (ip[back - 1] == matchPtr[back - 1]))
+ back--;
+
+ mlt -= back;
+
+ if (mlt > longest) {
+ longest = (int)mlt;
+ *matchpos = matchPtr + back;
+ *startpos = ip + back;
}
- ipt += LZ4_NBCOMMONBYTES(diff);
- goto _endcount;
}
- #if LZ4_ARCH64
- if ((ipt < (matchlimit - 3))
- && (A32(reft) == A32(ipt))) {
- ipt += 4;
- reft += 4;
- }
- ipt += 2;
- #endif
- if ((ipt < (matchlimit - 1))
- && (A16(reft) == A16(ipt))) {
- reft += 2;
- }
- if ((ipt < matchlimit) && (*reft == *ipt))
- ipt++;
-_endcount:
- reft = ref;
+ }
+ } else {
+ const BYTE * const matchPtr = dictBase + matchIndex;
- while ((startt > startlimit)
- && (reft > hc4->base)
- && (startt[-1] == reft[-1])) {
- startt--;
- reft--;
- }
+ if (LZ4_read32(matchPtr) == LZ4_read32(ip)) {
+ size_t mlt;
+ int back = 0;
+ const BYTE *vLimit = ip + (dictLimit - matchIndex);
- if ((ipt - startt) > longest) {
- longest = (int)(ipt - startt);
- *matchpos = reft;
- *startpos = startt;
+ if (vLimit > iHighLimit)
+ vLimit = iHighLimit;
+
+ mlt = LZ4_count(ip + MINMATCH,
+ matchPtr + MINMATCH, vLimit) + MINMATCH;
+
+ if ((ip + mlt == vLimit) && (vLimit < iHighLimit))
+ mlt += LZ4_count(ip + mlt, base + dictLimit,
+ iHighLimit);
+ while ((ip + back > iLowLimit)
+ && (matchIndex + back > lowLimit)
+ && (ip[back - 1] == matchPtr[back - 1]))
+ back--;
+
+ mlt -= back;
+
+ if ((int)mlt > longest) {
+ longest = (int)mlt;
+ *matchpos = base + matchIndex + back;
+ *startpos = ip + back;
}
}
}
- ref -= (size_t)chaintable[(size_t)(ref) & MAXD_MASK];
+
+ matchIndex -= DELTANEXTU16(matchIndex);
}
+
return longest;
}
-static inline int lz4_encodesequence(const u8 **ip, u8 **op, const u8 **anchor,
- int ml, const u8 *ref)
+static FORCE_INLINE int LZ4HC_encodeSequence(
+ const BYTE **ip,
+ BYTE **op,
+ const BYTE **anchor,
+ int matchLength,
+ const BYTE * const match,
+ limitedOutput_directive limitedOutputBuffer,
+ BYTE *oend)
{
- int length, len;
- u8 *token;
+ int length;
+ BYTE *token;
/* Encode Literal length */
length = (int)(*ip - *anchor);
token = (*op)++;
+
+ if ((limitedOutputBuffer)
+ && ((*op + (length>>8)
+ + length + (2 + 1 + LASTLITERALS)) > oend)) {
+ /* Check output limit */
+ return 1;
+ }
if (length >= (int)RUN_MASK) {
- *token = (RUN_MASK << ML_BITS);
+ int len;
+
+ *token = (RUN_MASK<<ML_BITS);
len = length - RUN_MASK;
for (; len > 254 ; len -= 255)
*(*op)++ = 255;
- *(*op)++ = (u8)len;
+ *(*op)++ = (BYTE)len;
} else
- *token = (length << ML_BITS);
+ *token = (BYTE)(length<<ML_BITS);
/* Copy Literals */
- LZ4_BLINDCOPY(*anchor, *op, length);
+ LZ4_wildCopy(*op, *anchor, (*op) + length);
+ *op += length;
/* Encode Offset */
- LZ4_WRITE_LITTLEENDIAN_16(*op, (u16)(*ip - ref));
+ LZ4_writeLE16(*op, (U16)(*ip - match));
+ *op += 2;
/* Encode MatchLength */
- len = (int)(ml - MINMATCH);
- if (len >= (int)ML_MASK) {
+ length = (int)(matchLength - MINMATCH);
+
+ if ((limitedOutputBuffer)
+ && (*op + (length>>8)
+ + (1 + LASTLITERALS) > oend)) {
+ /* Check output limit */
+ return 1;
+ }
+
+ if (length >= (int)ML_MASK) {
*token += ML_MASK;
- len -= ML_MASK;
- for (; len > 509 ; len -= 510) {
+ length -= ML_MASK;
+
+ for (; length > 509 ; length -= 510) {
*(*op)++ = 255;
*(*op)++ = 255;
}
- if (len > 254) {
- len -= 255;
+
+ if (length > 254) {
+ length -= 255;
*(*op)++ = 255;
}
- *(*op)++ = (u8)len;
+
+ *(*op)++ = (BYTE)length;
} else
- *token += len;
+ *token += (BYTE)(length);
/* Prepare next loop */
- *ip += ml;
+ *ip += matchLength;
*anchor = *ip;
return 0;
}
-static int lz4_compresshcctx(struct lz4hc_data *ctx,
- const char *source,
- char *dest,
- int isize)
+static int LZ4HC_compress_generic(
+ LZ4HC_CCtx_internal *const ctx,
+ const char * const source,
+ char * const dest,
+ int const inputSize,
+ int const maxOutputSize,
+ int compressionLevel,
+ limitedOutput_directive limit
+ )
{
- const u8 *ip = (const u8 *)source;
- const u8 *anchor = ip;
- const u8 *const iend = ip + isize;
- const u8 *const mflimit = iend - MFLIMIT;
- const u8 *const matchlimit = (iend - LASTLITERALS);
+ const BYTE *ip = (const BYTE *) source;
+ const BYTE *anchor = ip;
+ const BYTE * const iend = ip + inputSize;
+ const BYTE * const mflimit = iend - MFLIMIT;
+ const BYTE * const matchlimit = (iend - LASTLITERALS);
- u8 *op = (u8 *)dest;
+ BYTE *op = (BYTE *) dest;
+ BYTE * const oend = op + maxOutputSize;
+ unsigned int maxNbAttempts;
int ml, ml2, ml3, ml0;
- const u8 *ref = NULL;
- const u8 *start2 = NULL;
- const u8 *ref2 = NULL;
- const u8 *start3 = NULL;
- const u8 *ref3 = NULL;
- const u8 *start0;
- const u8 *ref0;
- int lastrun;
+ const BYTE *ref = NULL;
+ const BYTE *start2 = NULL;
+ const BYTE *ref2 = NULL;
+ const BYTE *start3 = NULL;
+ const BYTE *ref3 = NULL;
+ const BYTE *start0;
+ const BYTE *ref0;
+
+ /* init */
+ if (compressionLevel > LZ4HC_MAX_CLEVEL)
+ compressionLevel = LZ4HC_MAX_CLEVEL;
+ if (compressionLevel < 1)
+ compressionLevel = LZ4HC_DEFAULT_CLEVEL;
+ maxNbAttempts = 1 << (compressionLevel - 1);
+ ctx->end += inputSize;
ip++;
/* Main Loop */
while (ip < mflimit) {
- ml = lz4hc_insertandfindbestmatch(ctx, ip, matchlimit, (&ref));
+ ml = LZ4HC_InsertAndFindBestMatch(ctx, ip,
+ matchlimit, (&ref), maxNbAttempts);
if (!ml) {
ip++;
continue;
@@ -351,51 +387,59 @@ static int lz4_compresshcctx(struct lz4hc_data *ctx,
start0 = ip;
ref0 = ref;
ml0 = ml;
-_search2:
- if (ip+ml < mflimit)
- ml2 = lz4hc_insertandgetwidermatch(ctx, ip + ml - 2,
- ip + 1, matchlimit, ml, &ref2, &start2);
+
+_Search2:
+ if (ip + ml < mflimit)
+ ml2 = LZ4HC_InsertAndGetWiderMatch(ctx,
+ ip + ml - 2, ip + 0,
+ matchlimit, ml, &ref2,
+ &start2, maxNbAttempts);
else
ml2 = ml;
- /* No better match */
+
if (ml2 == ml) {
- lz4_encodesequence(&ip, &op, &anchor, ml, ref);
+ /* No better match */
+ if (LZ4HC_encodeSequence(&ip, &op,
+ &anchor, ml, ref, limit, oend))
+ return 0;
continue;
}
if (start0 < ip) {
- /* empirical */
if (start2 < ip + ml0) {
+ /* empirical */
ip = start0;
ref = ref0;
ml = ml0;
}
}
- /*
- * Here, start0==ip
- * First Match too small : removed
- */
+
+ /* Here, start0 == ip */
if ((start2 - ip) < 3) {
+ /* First Match too small : removed */
ml = ml2;
ip = start2;
ref = ref2;
- goto _search2;
+ goto _Search2;
}
-_search3:
+_Search3:
/*
- * Currently we have :
- * ml2 > ml1, and
- * ip1+3 <= ip2 (usually < ip1+ml1)
- */
+ * Currently we have :
+ * ml2 > ml1, and
+ * ip1 + 3 <= ip2 (usually < ip1 + ml1)
+ */
if ((start2 - ip) < OPTIMAL_ML) {
int correction;
int new_ml = ml;
+
if (new_ml > OPTIMAL_ML)
new_ml = OPTIMAL_ML;
if (ip + new_ml > start2 + ml2 - MINMATCH)
new_ml = (int)(start2 - ip) + ml2 - MINMATCH;
+
correction = new_ml - (int)(start2 - ip);
+
if (correction > 0) {
start2 += correction;
ref2 += correction;
@@ -403,39 +447,44 @@ static int lz4_compresshcctx(struct lz4hc_data *ctx,
}
}
/*
- * Now, we have start2 = ip+new_ml,
- * with new_ml=min(ml, OPTIMAL_ML=18)
+ * Now, we have start2 = ip + new_ml,
+ * with new_ml = min(ml, OPTIMAL_ML = 18)
*/
+
if (start2 + ml2 < mflimit)
- ml3 = lz4hc_insertandgetwidermatch(ctx,
- start2 + ml2 - 3, start2, matchlimit,
- ml2, &ref3, &start3);
+ ml3 = LZ4HC_InsertAndGetWiderMatch(ctx,
+ start2 + ml2 - 3, start2,
+ matchlimit, ml2, &ref3, &start3,
+ maxNbAttempts);
else
ml3 = ml2;
- /* No better match : 2 sequences to encode */
if (ml3 == ml2) {
+ /* No better match : 2 sequences to encode */
/* ip & ref are known; Now for ml */
- if (start2 < ip+ml)
+ if (start2 < ip + ml)
ml = (int)(start2 - ip);
-
/* Now, encode 2 sequences */
- lz4_encodesequence(&ip, &op, &anchor, ml, ref);
+ if (LZ4HC_encodeSequence(&ip, &op, &anchor,
+ ml, ref, limit, oend))
+ return 0;
ip = start2;
- lz4_encodesequence(&ip, &op, &anchor, ml2, ref2);
+ if (LZ4HC_encodeSequence(&ip, &op, &anchor,
+ ml2, ref2, limit, oend))
+ return 0;
continue;
}
- /* Not enough space for match 2 : remove it */
if (start3 < ip + ml + 3) {
- /*
- * can write Seq1 immediately ==> Seq2 is removed,
- * so Seq3 becomes Seq1
- */
+ /* Not enough space for match 2 : remove it */
if (start3 >= (ip + ml)) {
+ /* can write Seq1 immediately
+ * ==> Seq2 is removed,
+ * so Seq3 becomes Seq1
+ */
if (start2 < ip + ml) {
- int correction =
- (int)(ip + ml - start2);
+ int correction = (int)(ip + ml - start2);
+
start2 += correction;
ref2 += correction;
ml2 -= correction;
@@ -446,35 +495,38 @@ static int lz4_compresshcctx(struct lz4hc_data *ctx,
}
}
- lz4_encodesequence(&ip, &op, &anchor, ml, ref);
- ip = start3;
+ if (LZ4HC_encodeSequence(&ip, &op, &anchor,
+ ml, ref, limit, oend))
+ return 0;
+ ip = start3;
ref = ref3;
- ml = ml3;
+ ml = ml3;
start0 = start2;
ref0 = ref2;
ml0 = ml2;
- goto _search2;
+ goto _Search2;
}
start2 = start3;
ref2 = ref3;
ml2 = ml3;
- goto _search3;
+ goto _Search3;
}
/*
- * OK, now we have 3 ascending matches; let's write at least
- * the first one ip & ref are known; Now for ml
- */
+ * OK, now we have 3 ascending matches;
+ * let's write at least the first one
+ * ip & ref are known; Now for ml
+ */
if (start2 < ip + ml) {
if ((start2 - ip) < (int)ML_MASK) {
int correction;
+
if (ml > OPTIMAL_ML)
ml = OPTIMAL_ML;
if (ip + ml > start2 + ml2 - MINMATCH)
- ml = (int)(start2 - ip) + ml2
- - MINMATCH;
+ ml = (int)(start2 - ip) + ml2 - MINMATCH;
correction = ml - (int)(start2 - ip);
if (correction > 0) {
start2 += correction;
@@ -484,7 +536,9 @@ static int lz4_compresshcctx(struct lz4hc_data *ctx,
} else
ml = (int)(start2 - ip);
}
- lz4_encodesequence(&ip, &op, &anchor, ml, ref);
+ if (LZ4HC_encodeSequence(&ip, &op, &anchor, ml,
+ ref, limit, oend))
+ return 0;
ip = start2;
ref = ref2;
@@ -494,46 +548,245 @@ static int lz4_compresshcctx(struct lz4hc_data *ctx,
ref2 = ref3;
ml2 = ml3;
- goto _search3;
+ goto _Search3;
}
/* Encode Last Literals */
- lastrun = (int)(iend - anchor);
- if (lastrun >= (int)RUN_MASK) {
- *op++ = (RUN_MASK << ML_BITS);
- lastrun -= RUN_MASK;
- for (; lastrun > 254 ; lastrun -= 255)
- *op++ = 255;
- *op++ = (u8) lastrun;
- } else
- *op++ = (lastrun << ML_BITS);
- memcpy(op, anchor, iend - anchor);
- op += iend - anchor;
+ {
+ int lastRun = (int)(iend - anchor);
+
+ if ((limit)
+ && (((char *)op - dest) + lastRun + 1
+ + ((lastRun + 255 - RUN_MASK)/255)
+ > (U32)maxOutputSize)) {
+ /* Check output limit */
+ return 0;
+ }
+ if (lastRun >= (int)RUN_MASK) {
+ *op++ = (RUN_MASK<<ML_BITS);
+ lastRun -= RUN_MASK;
+ for (; lastRun > 254 ; lastRun -= 255)
+ *op++ = 255;
+ *op++ = (BYTE) lastRun;
+ } else
+ *op++ = (BYTE)(lastRun<<ML_BITS);
+ memcpy(op, anchor, iend - anchor);
+ op += iend - anchor;
+ }
+
/* End */
return (int) (((char *)op) - dest);
}
-int lz4hc_compress(const unsigned char *src, size_t src_len,
- unsigned char *dst, size_t *dst_len, void *wrkmem)
+static int LZ4_compress_HC_extStateHC(
+ void *state,
+ const char *src,
+ char *dst,
+ int srcSize,
+ int maxDstSize,
+ int compressionLevel)
{
- int ret = -1;
- int out_len = 0;
+ LZ4HC_CCtx_internal *ctx = &((LZ4_streamHC_t *)state)->internal_donotuse;
- struct lz4hc_data *hc4 = (struct lz4hc_data *)wrkmem;
- lz4hc_init(hc4, (const u8 *)src);
- out_len = lz4_compresshcctx((struct lz4hc_data *)hc4, (const u8 *)src,
- (char *)dst, (int)src_len);
+ if (((size_t)(state)&(sizeof(void *) - 1)) != 0) {
+ /* Error : state is not aligned
+ * for pointers (32 or 64 bits)
+ */
+ return 0;
+ }
- if (out_len < 0)
- goto exit;
+ LZ4HC_init(ctx, (const BYTE *)src);
- *dst_len = out_len;
- return 0;
+ if (maxDstSize < LZ4_compressBound(srcSize))
+ return LZ4HC_compress_generic(ctx, src, dst,
+ srcSize, maxDstSize, compressionLevel, limitedOutput);
+ else
+ return LZ4HC_compress_generic(ctx, src, dst,
+ srcSize, maxDstSize, compressionLevel, noLimit);
+}
-exit:
- return ret;
+int LZ4_compress_HC(const char *src, char *dst, int srcSize,
+ int maxDstSize, int compressionLevel, void *wrkmem)
+{
+ return LZ4_compress_HC_extStateHC(wrkmem, src, dst,
+ srcSize, maxDstSize, compressionLevel);
+}
+EXPORT_SYMBOL(LZ4_compress_HC);
+
+/**************************************
+ * Streaming Functions
+ **************************************/
+void LZ4_resetStreamHC(LZ4_streamHC_t *LZ4_streamHCPtr, int compressionLevel)
+{
+ LZ4_streamHCPtr->internal_donotuse.base = NULL;
+ LZ4_streamHCPtr->internal_donotuse.compressionLevel = (unsigned int)compressionLevel;
+}
+
+int LZ4_loadDictHC(LZ4_streamHC_t *LZ4_streamHCPtr,
+ const char *dictionary,
+ int dictSize)
+{
+ LZ4HC_CCtx_internal *ctxPtr = &LZ4_streamHCPtr->internal_donotuse;
+
+ if (dictSize > 64 * KB) {
+ dictionary += dictSize - 64 * KB;
+ dictSize = 64 * KB;
+ }
+ LZ4HC_init(ctxPtr, (const BYTE *)dictionary);
+ if (dictSize >= 4)
+ LZ4HC_Insert(ctxPtr, (const BYTE *)dictionary + (dictSize - 3));
+ ctxPtr->end = (const BYTE *)dictionary + dictSize;
+ return dictSize;
+}
+EXPORT_SYMBOL(LZ4_loadDictHC);
+
+/* compression */
+
+static void LZ4HC_setExternalDict(
+ LZ4HC_CCtx_internal *ctxPtr,
+ const BYTE *newBlock)
+{
+ if (ctxPtr->end >= ctxPtr->base + 4) {
+ /* Referencing remaining dictionary content */
+ LZ4HC_Insert(ctxPtr, ctxPtr->end - 3);
+ }
+
+ /*
+ * Only one memory segment for extDict,
+ * so any previous extDict is lost at this stage
+ */
+ ctxPtr->lowLimit = ctxPtr->dictLimit;
+ ctxPtr->dictLimit = (U32)(ctxPtr->end - ctxPtr->base);
+ ctxPtr->dictBase = ctxPtr->base;
+ ctxPtr->base = newBlock - ctxPtr->dictLimit;
+ ctxPtr->end = newBlock;
+ /* match referencing will resume from there */
+ ctxPtr->nextToUpdate = ctxPtr->dictLimit;
+}
+EXPORT_SYMBOL(LZ4HC_setExternalDict);
+
+static int LZ4_compressHC_continue_generic(
+ LZ4_streamHC_t *LZ4_streamHCPtr,
+ const char *source,
+ char *dest,
+ int inputSize,
+ int maxOutputSize,
+ limitedOutput_directive limit)
+{
+ LZ4HC_CCtx_internal *ctxPtr = &LZ4_streamHCPtr->internal_donotuse;
+
+ /* auto - init if forgotten */
+ if (ctxPtr->base == NULL)
+ LZ4HC_init(ctxPtr, (const BYTE *) source);
+
+ /* Check overflow */
+ if ((size_t)(ctxPtr->end - ctxPtr->base) > 2 * GB) {
+ size_t dictSize = (size_t)(ctxPtr->end - ctxPtr->base)
+ - ctxPtr->dictLimit;
+ if (dictSize > 64 * KB)
+ dictSize = 64 * KB;
+ LZ4_loadDictHC(LZ4_streamHCPtr,
+ (const char *)(ctxPtr->end) - dictSize, (int)dictSize);
+ }
+
+ /* Check if blocks follow each other */
+ if ((const BYTE *)source != ctxPtr->end)
+ LZ4HC_setExternalDict(ctxPtr, (const BYTE *)source);
+
+ /* Check overlapping input/dictionary space */
+ {
+ const BYTE *sourceEnd = (const BYTE *) source + inputSize;
+ const BYTE * const dictBegin = ctxPtr->dictBase + ctxPtr->lowLimit;
+ const BYTE * const dictEnd = ctxPtr->dictBase + ctxPtr->dictLimit;
+
+ if ((sourceEnd > dictBegin)
+ && ((const BYTE *)source < dictEnd)) {
+ if (sourceEnd > dictEnd)
+ sourceEnd = dictEnd;
+ ctxPtr->lowLimit = (U32)(sourceEnd - ctxPtr->dictBase);
+
+ if (ctxPtr->dictLimit - ctxPtr->lowLimit < 4)
+ ctxPtr->lowLimit = ctxPtr->dictLimit;
+ }
+ }
+
+ return LZ4HC_compress_generic(ctxPtr, source, dest,
+ inputSize, maxOutputSize, ctxPtr->compressionLevel, limit);
+}
+
+int LZ4_compress_HC_continue(
+ LZ4_streamHC_t *LZ4_streamHCPtr,
+ const char *source,
+ char *dest,
+ int inputSize,
+ int maxOutputSize)
+{
+ if (maxOutputSize < LZ4_compressBound(inputSize))
+ return LZ4_compressHC_continue_generic(LZ4_streamHCPtr,
+ source, dest, inputSize, maxOutputSize, limitedOutput);
+ else
+ return LZ4_compressHC_continue_generic(LZ4_streamHCPtr,
+ source, dest, inputSize, maxOutputSize, noLimit);
+}
+EXPORT_SYMBOL(LZ4_compress_HC_continue);
+
+/* dictionary saving */
+
+int LZ4_saveDictHC(
+ LZ4_streamHC_t *LZ4_streamHCPtr,
+ char *safeBuffer,
+ int dictSize)
+{
+ LZ4HC_CCtx_internal *const streamPtr = &LZ4_streamHCPtr->internal_donotuse;
+ int const prefixSize = (int)(streamPtr->end
+ - (streamPtr->base + streamPtr->dictLimit));
+
+ if (dictSize > 64 * KB)
+ dictSize = 64 * KB;
+ if (dictSize < 4)
+ dictSize = 0;
+ if (dictSize > prefixSize)
+ dictSize = prefixSize;
+
+ memmove(safeBuffer, streamPtr->end - dictSize, dictSize);
+
+ {
+ U32 const endIndex = (U32)(streamPtr->end - streamPtr->base);
+
+ streamPtr->end = (const BYTE *)safeBuffer + dictSize;
+ streamPtr->base = streamPtr->end - endIndex;
+ streamPtr->dictLimit = endIndex - dictSize;
+ streamPtr->lowLimit = endIndex - dictSize;
+
+ if (streamPtr->nextToUpdate < streamPtr->dictLimit)
+ streamPtr->nextToUpdate = streamPtr->dictLimit;
+ }
+ return dictSize;
+}
+EXPORT_SYMBOL(LZ4_saveDictHC);
+
+/*-******************************
+ * For backwards compatibility
+ ********************************/
+int lz4hc_compress(const unsigned char *src, size_t src_len,
+ unsigned char *dst, size_t *dst_len, void *wrkmem)
+{
+ *dst_len = LZ4_compress_HC(src, dst, src_len,
+ *dst_len, LZ4HC_DEFAULT_CLEVEL, wrkmem);
+
+ /*
+ * Prior lz4hc_compress will return -1 in case of error
+ * and 0 on success
+ * while new LZ4_compress_HC
+ * returns 0 in case of error
+ * and the output length on success
+ */
+ if (!*dst_len)
+ return -1;
+ else
+ return 0;
}
EXPORT_SYMBOL(lz4hc_compress);
MODULE_LICENSE("Dual BSD/GPL");
-MODULE_DESCRIPTION("LZ4HC compressor");
+MODULE_DESCRIPTION("LZ4 HC compressor");
diff --git a/lib/radix-tree.c b/lib/radix-tree.c
index 8e6d552..380ebb9 100644
--- a/lib/radix-tree.c
+++ b/lib/radix-tree.c
@@ -352,6 +352,15 @@ static int __radix_tree_preload(gfp_t gfp_mask, int nr)
int ret = -ENOMEM;
/*
+ * New allocate node must have node->private_list as INIT_LIST_HEAD
+ * state by workingset shadow memory implementation.
+ * If user pass __GFP_ZERO by mistake, slab allocator will clear
+ * node->private_list, which makes a BUG. Rather than going Oops,
+ * just fix and warn about it.
+ */
+ if (WARN_ON(gfp_mask & __GFP_ZERO))
+ gfp_mask &= ~__GFP_ZERO;
+ /*
* Nodes preloaded by one cgroup can be be used by another cgroup, so
* they should never be accounted to any particular memory cgroup.
*/
diff --git a/lib/raid6/test/Makefile b/lib/raid6/test/Makefile
index 2c7b60e..1faeef0 100644
--- a/lib/raid6/test/Makefile
+++ b/lib/raid6/test/Makefile
@@ -26,7 +26,7 @@
CFLAGS += -I../../../arch/arm/include -mfpu=neon
HAS_NEON = yes
endif
-ifeq ($(ARCH),arm64)
+ifeq ($(ARCH),aarch64)
CFLAGS += -I../../../arch/arm64/include
HAS_NEON = yes
endif
@@ -40,7 +40,7 @@
gcc -c -x assembler - >&/dev/null && \
rm ./-.o && echo -DCONFIG_AS_AVX512=1)
else ifeq ($(HAS_NEON),yes)
- OBJS += neon.o neon1.o neon2.o neon4.o neon8.o
+ OBJS += neon.o neon1.o neon2.o neon4.o neon8.o recov_neon.o recov_neon_inner.o
CFLAGS += -DCONFIG_KERNEL_MODE_NEON=1
else
HAS_ALTIVEC := $(shell printf '\#include <altivec.h>\nvector int a;\n' |\
diff --git a/lib/rbtree_test.c b/lib/rbtree_test.c
index 8b3c9dc8..afedd37 100644
--- a/lib/rbtree_test.c
+++ b/lib/rbtree_test.c
@@ -1,11 +1,18 @@
#include <linux/module.h>
+#include <linux/moduleparam.h>
#include <linux/rbtree_augmented.h>
#include <linux/random.h>
+#include <linux/slab.h>
#include <asm/timex.h>
-#define NODES 100
-#define PERF_LOOPS 100000
-#define CHECK_LOOPS 100
+#define __param(type, name, init, msg) \
+ static type name = init; \
+ module_param(name, type, 0444); \
+ MODULE_PARM_DESC(name, msg);
+
+__param(int, nnodes, 100, "Number of nodes in the rb-tree");
+__param(int, perf_loops, 1000, "Number of iterations modifying the rb-tree");
+__param(int, check_loops, 100, "Number of iterations modifying and verifying the rb-tree");
struct test_node {
u32 key;
@@ -17,7 +24,7 @@ struct test_node {
};
static struct rb_root root = RB_ROOT;
-static struct test_node nodes[NODES];
+static struct test_node *nodes = NULL;
static struct rnd_state rnd;
@@ -95,7 +102,7 @@ static void erase_augmented(struct test_node *node, struct rb_root *root)
static void init(void)
{
int i;
- for (i = 0; i < NODES; i++) {
+ for (i = 0; i < nnodes; i++) {
nodes[i].key = prandom_u32_state(&rnd);
nodes[i].val = prandom_u32_state(&rnd);
}
@@ -177,6 +184,10 @@ static int __init rbtree_test_init(void)
int i, j;
cycles_t time1, time2, time;
+ nodes = kmalloc(nnodes * sizeof(*nodes), GFP_KERNEL);
+ if (!nodes)
+ return -ENOMEM;
+
printk(KERN_ALERT "rbtree testing");
prandom_seed_state(&rnd, 3141592653589793238ULL);
@@ -184,27 +195,27 @@ static int __init rbtree_test_init(void)
time1 = get_cycles();
- for (i = 0; i < PERF_LOOPS; i++) {
- for (j = 0; j < NODES; j++)
+ for (i = 0; i < perf_loops; i++) {
+ for (j = 0; j < nnodes; j++)
insert(nodes + j, &root);
- for (j = 0; j < NODES; j++)
+ for (j = 0; j < nnodes; j++)
erase(nodes + j, &root);
}
time2 = get_cycles();
time = time2 - time1;
- time = div_u64(time, PERF_LOOPS);
+ time = div_u64(time, perf_loops);
printk(" -> %llu cycles\n", (unsigned long long)time);
- for (i = 0; i < CHECK_LOOPS; i++) {
+ for (i = 0; i < check_loops; i++) {
init();
- for (j = 0; j < NODES; j++) {
+ for (j = 0; j < nnodes; j++) {
check(j);
insert(nodes + j, &root);
}
- for (j = 0; j < NODES; j++) {
- check(NODES - j);
+ for (j = 0; j < nnodes; j++) {
+ check(nnodes - j);
erase(nodes + j, &root);
}
check(0);
@@ -216,32 +227,34 @@ static int __init rbtree_test_init(void)
time1 = get_cycles();
- for (i = 0; i < PERF_LOOPS; i++) {
- for (j = 0; j < NODES; j++)
+ for (i = 0; i < perf_loops; i++) {
+ for (j = 0; j < nnodes; j++)
insert_augmented(nodes + j, &root);
- for (j = 0; j < NODES; j++)
+ for (j = 0; j < nnodes; j++)
erase_augmented(nodes + j, &root);
}
time2 = get_cycles();
time = time2 - time1;
- time = div_u64(time, PERF_LOOPS);
+ time = div_u64(time, perf_loops);
printk(" -> %llu cycles\n", (unsigned long long)time);
- for (i = 0; i < CHECK_LOOPS; i++) {
+ for (i = 0; i < check_loops; i++) {
init();
- for (j = 0; j < NODES; j++) {
+ for (j = 0; j < nnodes; j++) {
check_augmented(j);
insert_augmented(nodes + j, &root);
}
- for (j = 0; j < NODES; j++) {
- check_augmented(NODES - j);
+ for (j = 0; j < nnodes; j++) {
+ check_augmented(nnodes - j);
erase_augmented(nodes + j, &root);
}
check_augmented(0);
}
+ kfree(nodes);
+
return -EAGAIN; /* Fail will directly unload the module */
}
diff --git a/lib/rhashtable.c b/lib/rhashtable.c
index 895961c..fdffd62 100644
--- a/lib/rhashtable.c
+++ b/lib/rhashtable.c
@@ -251,8 +251,10 @@ static int rhashtable_rehash_table(struct rhashtable *ht)
if (!new_tbl)
return 0;
- for (old_hash = 0; old_hash < old_tbl->size; old_hash++)
+ for (old_hash = 0; old_hash < old_tbl->size; old_hash++) {
rhashtable_rehash_chain(ht, old_hash);
+ cond_resched();
+ }
/* Publish the new table pointer. */
rcu_assign_pointer(ht->tbl, new_tbl);
@@ -783,8 +785,16 @@ EXPORT_SYMBOL_GPL(rhashtable_walk_stop);
static size_t rounded_hashtable_size(const struct rhashtable_params *params)
{
- return max(roundup_pow_of_two(params->nelem_hint * 4 / 3),
- (unsigned long)params->min_size);
+ size_t retsize;
+
+ if (params->nelem_hint)
+ retsize = max(roundup_pow_of_two(params->nelem_hint * 4 / 3),
+ (unsigned long)params->min_size);
+ else
+ retsize = max(HASH_DEFAULT_SIZE,
+ (unsigned long)params->min_size);
+
+ return retsize;
}
static u32 rhashtable_jhash2(const void *key, u32 length, u32 seed)
@@ -841,8 +851,6 @@ int rhashtable_init(struct rhashtable *ht,
struct bucket_table *tbl;
size_t size;
- size = HASH_DEFAULT_SIZE;
-
if ((!params->key_len && !params->obj_hashfn) ||
(params->obj_hashfn && !params->obj_cmpfn))
return -EINVAL;
@@ -869,8 +877,7 @@ int rhashtable_init(struct rhashtable *ht,
ht->p.min_size = max(ht->p.min_size, HASH_MIN_SIZE);
- if (params->nelem_hint)
- size = rounded_hashtable_size(&ht->p);
+ size = rounded_hashtable_size(&ht->p);
/* The maximum (not average) chain length grows with the
* size of the hash table, at a rate of (log N)/(log log N).
@@ -988,6 +995,7 @@ void rhashtable_free_and_destroy(struct rhashtable *ht,
for (i = 0; i < tbl->size; i++) {
struct rhash_head *pos, *next;
+ cond_resched();
for (pos = rht_dereference(tbl->buckets[i], ht),
next = !rht_is_a_nulls(pos) ?
rht_dereference(pos->next, ht) : NULL;
diff --git a/lib/string.c b/lib/string.c
index ccabe16..71e389a 100644
--- a/lib/string.c
+++ b/lib/string.c
@@ -656,6 +656,32 @@ int match_string(const char * const *array, size_t n, const char *string)
}
EXPORT_SYMBOL(match_string);
+/**
+ * __sysfs_match_string - matches given string in an array
+ * @array: array of strings
+ * @n: number of strings in the array or -1 for NULL terminated arrays
+ * @str: string to match with
+ *
+ * Returns index of @str in the @array or -EINVAL, just like match_string().
+ * Uses sysfs_streq instead of strcmp for matching.
+ */
+int __sysfs_match_string(const char * const *array, size_t n, const char *str)
+{
+ const char *item;
+ int index;
+
+ for (index = 0; index < n; index++) {
+ item = array[index];
+ if (!item)
+ break;
+ if (sysfs_streq(item, str))
+ return index;
+ }
+
+ return -EINVAL;
+}
+EXPORT_SYMBOL(__sysfs_match_string);
+
#ifndef __HAVE_ARCH_MEMSET
/**
* memset - Fill a region of memory with the given value
diff --git a/lib/swiotlb.c b/lib/swiotlb.c
index b7812df..7ff9dc3 100644
--- a/lib/swiotlb.c
+++ b/lib/swiotlb.c
@@ -17,6 +17,8 @@
* 08/12/11 beckyb Add highmem support
*/
+#define pr_fmt(fmt) "software IO TLB: " fmt
+
#include <linux/cache.h>
#include <linux/dma-mapping.h>
#include <linux/mm.h>
@@ -147,20 +149,16 @@ static bool no_iotlb_memory;
void swiotlb_print_info(void)
{
unsigned long bytes = io_tlb_nslabs << IO_TLB_SHIFT;
- unsigned char *vstart, *vend;
if (no_iotlb_memory) {
- pr_warn("software IO TLB: No low mem\n");
+ pr_warn("No low mem\n");
return;
}
- vstart = phys_to_virt(io_tlb_start);
- vend = phys_to_virt(io_tlb_end);
-
- printk(KERN_INFO "software IO TLB [mem %#010llx-%#010llx] (%luMB) mapped at [%p-%p]\n",
+ pr_info("mapped [mem %#010llx-%#010llx] (%luMB)\n",
(unsigned long long)io_tlb_start,
(unsigned long long)io_tlb_end,
- bytes >> 20, vstart, vend - 1);
+ bytes >> 20);
}
int __init swiotlb_init_with_tbl(char *tlb, unsigned long nslabs, int verbose)
@@ -234,7 +232,7 @@ swiotlb_init(int verbose)
if (io_tlb_start)
memblock_free_early(io_tlb_start,
PAGE_ALIGN(io_tlb_nslabs << IO_TLB_SHIFT));
- pr_warn("Cannot allocate SWIOTLB buffer");
+ pr_warn("Cannot allocate buffer");
no_iotlb_memory = true;
}
@@ -276,8 +274,8 @@ swiotlb_late_init_with_default_size(size_t default_size)
return -ENOMEM;
}
if (order != get_order(bytes)) {
- printk(KERN_WARNING "Warning: only able to allocate %ld MB "
- "for software IO TLB\n", (PAGE_SIZE << order) >> 20);
+ pr_warn("only able to allocate %ld MB\n",
+ (PAGE_SIZE << order) >> 20);
io_tlb_nslabs = SLABS_PER_PAGE << order;
}
rc = swiotlb_late_init_with_tbl(vstart, io_tlb_nslabs);
@@ -691,7 +689,7 @@ swiotlb_alloc_coherent(struct device *hwdev, size_t size,
return ret;
err_warn:
- pr_warn("swiotlb: coherent allocation failed for device %s size=%zu\n",
+ pr_warn("coherent allocation failed for device %s size=%zu\n",
dev_name(hwdev), size);
dump_stack();
diff --git a/lib/test_bpf.c b/lib/test_bpf.c
index 98da752..960d4d6 100644
--- a/lib/test_bpf.c
+++ b/lib/test_bpf.c
@@ -83,6 +83,7 @@ struct bpf_test {
__u32 result;
} test[MAX_SUBTESTS];
int (*fill_helper)(struct bpf_test *self);
+ int expected_errcode; /* used when FLAG_EXPECTED_FAIL is set in the aux */
__u8 frag_data[MAX_DATA];
};
@@ -1900,7 +1901,9 @@ static struct bpf_test tests[] = {
},
CLASSIC | FLAG_NO_DATA | FLAG_EXPECTED_FAIL,
{ },
- { }
+ { },
+ .fill_helper = NULL,
+ .expected_errcode = -EINVAL,
},
{
"check: div_k_0",
@@ -1910,7 +1913,9 @@ static struct bpf_test tests[] = {
},
CLASSIC | FLAG_NO_DATA | FLAG_EXPECTED_FAIL,
{ },
- { }
+ { },
+ .fill_helper = NULL,
+ .expected_errcode = -EINVAL,
},
{
"check: unknown insn",
@@ -1921,7 +1926,9 @@ static struct bpf_test tests[] = {
},
CLASSIC | FLAG_EXPECTED_FAIL,
{ },
- { }
+ { },
+ .fill_helper = NULL,
+ .expected_errcode = -EINVAL,
},
{
"check: out of range spill/fill",
@@ -1931,7 +1938,9 @@ static struct bpf_test tests[] = {
},
CLASSIC | FLAG_NO_DATA | FLAG_EXPECTED_FAIL,
{ },
- { }
+ { },
+ .fill_helper = NULL,
+ .expected_errcode = -EINVAL,
},
{
"JUMPS + HOLES",
@@ -2023,6 +2032,8 @@ static struct bpf_test tests[] = {
CLASSIC | FLAG_NO_DATA | FLAG_EXPECTED_FAIL,
{ },
{ },
+ .fill_helper = NULL,
+ .expected_errcode = -EINVAL,
},
{
"check: LDX + RET X",
@@ -2033,6 +2044,8 @@ static struct bpf_test tests[] = {
CLASSIC | FLAG_NO_DATA | FLAG_EXPECTED_FAIL,
{ },
{ },
+ .fill_helper = NULL,
+ .expected_errcode = -EINVAL,
},
{ /* Mainly checking JIT here. */
"M[]: alt STX + LDX",
@@ -2207,6 +2220,8 @@ static struct bpf_test tests[] = {
CLASSIC | FLAG_NO_DATA | FLAG_EXPECTED_FAIL,
{ },
{ },
+ .fill_helper = NULL,
+ .expected_errcode = -EINVAL,
},
{ /* Passes checker but fails during runtime. */
"LD [SKF_AD_OFF-1]",
@@ -4803,6 +4818,7 @@ static struct bpf_test tests[] = {
{ },
{ },
.fill_helper = bpf_fill_maxinsns4,
+ .expected_errcode = -EINVAL,
},
{ /* Mainly checking JIT here. */
"BPF_MAXINSNS: Very long jump",
@@ -4858,10 +4874,15 @@ static struct bpf_test tests[] = {
{
"BPF_MAXINSNS: Jump, gap, jump, ...",
{ },
+#if defined(CONFIG_BPF_JIT_ALWAYS_ON) && defined(CONFIG_X86)
+ CLASSIC | FLAG_NO_DATA | FLAG_EXPECTED_FAIL,
+#else
CLASSIC | FLAG_NO_DATA,
+#endif
{ },
{ { 0, 0xababcbac } },
.fill_helper = bpf_fill_maxinsns11,
+ .expected_errcode = -ENOTSUPP,
},
{
"BPF_MAXINSNS: ld_abs+get_processor_id",
@@ -5632,7 +5653,7 @@ static struct bpf_prog *generate_filter(int which, int *err)
*err = bpf_prog_create(&fp, &fprog);
if (tests[which].aux & FLAG_EXPECTED_FAIL) {
- if (*err == -EINVAL) {
+ if (*err == tests[which].expected_errcode) {
pr_cont("PASS\n");
/* Verifier rejected filter as expected. */
*err = 0;
diff --git a/lib/test_hexdump.c b/lib/test_hexdump.c
index 3f415d8..1c3c513a 100644
--- a/lib/test_hexdump.c
+++ b/lib/test_hexdump.c
@@ -81,7 +81,7 @@ static void __init test_hexdump_prepare_test(size_t len, int rowsize,
const char *q = *result++;
size_t amount = strlen(q);
- strncpy(p, q, amount);
+ memcpy(p, q, amount);
p += amount;
*p++ = ' ';
diff --git a/lib/ubsan.c b/lib/ubsan.c
index 50d1d5c..60e108c 100644
--- a/lib/ubsan.c
+++ b/lib/ubsan.c
@@ -451,8 +451,7 @@ void __ubsan_handle_shift_out_of_bounds(struct shift_out_of_bounds_data *data,
EXPORT_SYMBOL(__ubsan_handle_shift_out_of_bounds);
-void __noreturn
-__ubsan_handle_builtin_unreachable(struct unreachable_data *data)
+void __ubsan_handle_builtin_unreachable(struct unreachable_data *data)
{
unsigned long flags;
diff --git a/lib/vdso/compiler.h b/lib/vdso/compiler.h
new file mode 100644
index 0000000..0e618b7
--- /dev/null
+++ b/lib/vdso/compiler.h
@@ -0,0 +1,24 @@
+/*
+ * Userspace implementations of fallback calls
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __VDSO_COMPILER_H
+#define __VDSO_COMPILER_H
+
+#error "vdso: Provide architectural overrides such as ARCH_PROVIDES_TIMER,"
+#error " DEFINE_FALLBACK and __arch_counter_get or any overrides. eg:"
+#error " vdso entry points or compilation time helpers."
+
+#endif /* __VDSO_COMPILER_H */
diff --git a/lib/vdso/datapage.h b/lib/vdso/datapage.h
new file mode 100644
index 0000000..df4427e
--- /dev/null
+++ b/lib/vdso/datapage.h
@@ -0,0 +1,24 @@
+/*
+ * Userspace implementations of __get_datapage
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __VDSO_DATAPAGE_H
+#define __VDSO_DATAPAGE_H
+
+#error "vdso: Provide a user space architecture specific definition or"
+#error "prototype for struct vdso_data *__get_datapage(void). Also define"
+#error "ARCH_CLOCK_FIXED_MASK if not provided by cs_mask."
+
+#endif /* __VDSO_DATAPAGE_H */
diff --git a/lib/vdso/vgettimeofday.c b/lib/vdso/vgettimeofday.c
new file mode 100644
index 0000000..6feeb82
--- /dev/null
+++ b/lib/vdso/vgettimeofday.c
@@ -0,0 +1,439 @@
+/*
+ * Userspace implementations of gettimeofday() and friends.
+ *
+ * Copyright (C) 2017 Cavium, Inc.
+ * Copyright (C) 2015 Mentor Graphics Corporation
+ * Copyright (C) 2012 ARM Limited
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Author: Will Deacon <will.deacon@arm.com>
+ * Rewriten from arch64 version into C by: Andrew Pinski <apinski@cavium.com>
+ * Reworked and rebased over arm version by: Mark Salyzyn <salyzyn@android.com>
+ */
+
+#include <asm/barrier.h>
+#include <linux/compiler.h> /* for notrace */
+#include <linux/math64.h> /* for __iter_div_u64_rem() */
+#include <uapi/linux/time.h> /* for struct timespec */
+
+#include "compiler.h"
+#include "datapage.h"
+
+#ifdef ARCH_PROVIDES_TIMER
+DEFINE_FALLBACK(gettimeofday, struct timeval *, tv, struct timezone *, tz)
+#endif
+DEFINE_FALLBACK(clock_gettime, clockid_t, clock, struct timespec *, ts)
+DEFINE_FALLBACK(clock_getres, clockid_t, clock, struct timespec *, ts)
+
+#ifdef USE_SYSCALL
+#if defined(__LP64__)
+# define USE_SYSCALL_MASK (USE_SYSCALL | USE_SYSCALL_64)
+#else
+# define USE_SYSCALL_MASK (USE_SYSCALL | USE_SYSCALL_32)
+#endif
+#else
+# define USE_SYSCALL_MASK ((uint32_t)-1)
+#endif
+
+static notrace u32 vdso_read_begin(const struct vdso_data *vd)
+{
+ u32 seq;
+
+ do {
+ seq = READ_ONCE(vd->tb_seq_count);
+
+ if ((seq & 1) == 0)
+ break;
+
+ cpu_relax();
+ } while (true);
+
+ smp_rmb(); /* Pairs with second smp_wmb in update_vsyscall */
+ return seq;
+}
+
+static notrace int vdso_read_retry(const struct vdso_data *vd, u32 start)
+{
+ u32 seq;
+
+ smp_rmb(); /* Pairs with first smp_wmb in update_vsyscall */
+ seq = READ_ONCE(vd->tb_seq_count);
+ return seq != start;
+}
+
+static notrace int do_realtime_coarse(const struct vdso_data *vd,
+ struct timespec *ts)
+{
+ u32 seq;
+
+ do {
+ seq = vdso_read_begin(vd);
+
+ ts->tv_sec = vd->xtime_coarse_sec;
+ ts->tv_nsec = vd->xtime_coarse_nsec;
+
+ } while (vdso_read_retry(vd, seq));
+
+ return 0;
+}
+
+static notrace int do_monotonic_coarse(const struct vdso_data *vd,
+ struct timespec *ts)
+{
+ struct timespec tomono;
+ u32 seq;
+ u64 nsec;
+
+ do {
+ seq = vdso_read_begin(vd);
+
+ ts->tv_sec = vd->xtime_coarse_sec;
+ ts->tv_nsec = vd->xtime_coarse_nsec;
+
+ tomono.tv_sec = vd->wtm_clock_sec;
+ tomono.tv_nsec = vd->wtm_clock_nsec;
+
+ } while (vdso_read_retry(vd, seq));
+
+ ts->tv_sec += tomono.tv_sec;
+ /* open coding timespec_add_ns */
+ ts->tv_sec += __iter_div_u64_rem(ts->tv_nsec + tomono.tv_nsec,
+ NSEC_PER_SEC, &nsec);
+ ts->tv_nsec = nsec;
+
+ return 0;
+}
+
+#ifdef ARCH_PROVIDES_TIMER
+
+/*
+ * Returns the clock delta, in nanoseconds left-shifted by the clock
+ * shift.
+ */
+static notrace u64 get_clock_shifted_nsec(const u64 cycle_last,
+ const u32 mult,
+ const u64 mask)
+{
+ u64 res;
+
+ /* Read the virtual counter. */
+ res = arch_vdso_read_counter();
+
+ res = res - cycle_last;
+
+ res &= mask;
+ return res * mult;
+}
+
+static notrace int do_realtime(const struct vdso_data *vd, struct timespec *ts)
+{
+ u32 seq, mult, shift;
+ u64 nsec, cycle_last;
+#ifdef ARCH_CLOCK_FIXED_MASK
+ static const u64 mask = ARCH_CLOCK_FIXED_MASK;
+#else
+ u64 mask;
+#endif
+ vdso_xtime_clock_sec_t sec;
+
+ do {
+ seq = vdso_read_begin(vd);
+
+ if (vd->use_syscall & USE_SYSCALL_MASK)
+ return -1;
+
+ cycle_last = vd->cs_cycle_last;
+
+ mult = vd->cs_mono_mult;
+ shift = vd->cs_shift;
+#ifndef ARCH_CLOCK_FIXED_MASK
+ mask = vd->cs_mask;
+#endif
+
+ sec = vd->xtime_clock_sec;
+ nsec = vd->xtime_clock_snsec;
+
+ } while (unlikely(vdso_read_retry(vd, seq)));
+
+ nsec += get_clock_shifted_nsec(cycle_last, mult, mask);
+ nsec >>= shift;
+ /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */
+ ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec);
+ ts->tv_nsec = nsec;
+
+ return 0;
+}
+
+static notrace int do_monotonic(const struct vdso_data *vd, struct timespec *ts)
+{
+ u32 seq, mult, shift;
+ u64 nsec, cycle_last;
+#ifdef ARCH_CLOCK_FIXED_MASK
+ static const u64 mask = ARCH_CLOCK_FIXED_MASK;
+#else
+ u64 mask;
+#endif
+ vdso_wtm_clock_nsec_t wtm_nsec;
+ __kernel_time_t sec;
+
+ do {
+ seq = vdso_read_begin(vd);
+
+ if (vd->use_syscall & USE_SYSCALL_MASK)
+ return -1;
+
+ cycle_last = vd->cs_cycle_last;
+
+ mult = vd->cs_mono_mult;
+ shift = vd->cs_shift;
+#ifndef ARCH_CLOCK_FIXED_MASK
+ mask = vd->cs_mask;
+#endif
+
+ sec = vd->xtime_clock_sec;
+ nsec = vd->xtime_clock_snsec;
+
+ sec += vd->wtm_clock_sec;
+ wtm_nsec = vd->wtm_clock_nsec;
+
+ } while (unlikely(vdso_read_retry(vd, seq)));
+
+ nsec += get_clock_shifted_nsec(cycle_last, mult, mask);
+ nsec >>= shift;
+ nsec += wtm_nsec;
+ /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */
+ ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec);
+ ts->tv_nsec = nsec;
+
+ return 0;
+}
+
+static notrace int do_monotonic_raw(const struct vdso_data *vd,
+ struct timespec *ts)
+{
+ u32 seq, mult, shift;
+ u64 nsec, cycle_last;
+#ifdef ARCH_CLOCK_FIXED_MASK
+ static const u64 mask = ARCH_CLOCK_FIXED_MASK;
+#else
+ u64 mask;
+#endif
+ vdso_raw_time_sec_t sec;
+
+ do {
+ seq = vdso_read_begin(vd);
+
+ if (vd->use_syscall & USE_SYSCALL_MASK)
+ return -1;
+
+ cycle_last = vd->cs_cycle_last;
+
+ mult = vd->cs_raw_mult;
+ shift = vd->cs_shift;
+#ifndef ARCH_CLOCK_FIXED_MASK
+ mask = vd->cs_mask;
+#endif
+
+ sec = vd->raw_time_sec;
+ nsec = vd->raw_time_nsec;
+
+ } while (unlikely(vdso_read_retry(vd, seq)));
+
+ nsec += get_clock_shifted_nsec(cycle_last, mult, mask);
+ nsec >>= shift;
+ /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */
+ ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec);
+ ts->tv_nsec = nsec;
+
+ return 0;
+}
+
+static notrace int do_boottime(const struct vdso_data *vd, struct timespec *ts)
+{
+ u32 seq, mult, shift;
+ u64 nsec, cycle_last;
+ vdso_wtm_clock_nsec_t wtm_nsec;
+#ifdef ARCH_CLOCK_FIXED_MASK
+ static const u64 mask = ARCH_CLOCK_FIXED_MASK;
+#else
+ u64 mask;
+#endif
+ __kernel_time_t sec;
+
+ do {
+ seq = vdso_read_begin(vd);
+
+ if (vd->use_syscall & USE_SYSCALL_MASK)
+ return -1;
+
+ cycle_last = vd->cs_cycle_last;
+
+ mult = vd->cs_mono_mult;
+ shift = vd->cs_shift;
+#ifndef ARCH_CLOCK_FIXED_MASK
+ mask = vd->cs_mask;
+#endif
+
+ sec = vd->xtime_clock_sec;
+ nsec = vd->xtime_clock_snsec;
+
+ sec += vd->wtm_clock_sec + vd->btm_sec;
+ wtm_nsec = vd->wtm_clock_nsec + vd->btm_nsec;
+
+ } while (unlikely(vdso_read_retry(vd, seq)));
+
+ nsec += get_clock_shifted_nsec(cycle_last, mult, mask);
+ nsec >>= shift;
+ nsec += wtm_nsec;
+
+ /* open coding timespec_add_ns to save a ts->tv_nsec = 0 */
+ ts->tv_sec = sec + __iter_div_u64_rem(nsec, NSEC_PER_SEC, &nsec);
+ ts->tv_nsec = nsec;
+
+ return 0;
+}
+
+#endif /* ARCH_PROVIDES_TIMER */
+
+notrace int __vdso_clock_gettime(clockid_t clock, struct timespec *ts)
+{
+ const struct vdso_data *vd = __get_datapage();
+
+#ifdef USE_SYSCALL
+ if (vd->use_syscall & USE_SYSCALL_MASK) {
+ goto fallback;
+ }
+#endif
+
+ switch (clock) {
+ case CLOCK_REALTIME_COARSE:
+ do_realtime_coarse(vd, ts);
+ break;
+ case CLOCK_MONOTONIC_COARSE:
+ do_monotonic_coarse(vd, ts);
+ break;
+#ifdef ARCH_PROVIDES_TIMER
+ case CLOCK_REALTIME:
+ if (do_realtime(vd, ts))
+ goto fallback;
+ break;
+ case CLOCK_MONOTONIC:
+ if (do_monotonic(vd, ts))
+ goto fallback;
+ break;
+ case CLOCK_MONOTONIC_RAW:
+ if (do_monotonic_raw(vd, ts))
+ goto fallback;
+ break;
+ case CLOCK_BOOTTIME:
+ if (do_boottime(vd, ts))
+ goto fallback;
+ break;
+#endif
+ default:
+ goto fallback;
+ }
+
+ return 0;
+fallback:
+ return clock_gettime_fallback(clock, ts);
+}
+
+#ifdef ARCH_PROVIDES_TIMER
+notrace int __vdso_gettimeofday(struct timeval *tv, struct timezone *tz)
+{
+ const struct vdso_data *vd = __get_datapage();
+
+ if (likely(tv != NULL)) {
+ struct timespec ts;
+
+ if (do_realtime(vd, &ts))
+ return gettimeofday_fallback(tv, tz);
+
+ tv->tv_sec = ts.tv_sec;
+ tv->tv_usec = ts.tv_nsec / 1000;
+ }
+
+ if (unlikely(tz != NULL)) {
+ tz->tz_minuteswest = vd->tz_minuteswest;
+ tz->tz_dsttime = vd->tz_dsttime;
+ }
+
+ return 0;
+}
+#endif
+
+int __vdso_clock_getres(clockid_t clock, struct timespec *res)
+{
+ long nsec;
+
+#ifdef USE_SYSCALL
+ const struct vdso_data *vd = __get_datapage();
+
+ if (vd->use_syscall & USE_SYSCALL_MASK) {
+ return clock_getres_fallback(clock, res);
+ }
+#endif
+
+ switch (clock) {
+ case CLOCK_REALTIME_COARSE:
+ case CLOCK_MONOTONIC_COARSE:
+ nsec = LOW_RES_NSEC;
+ break;
+#ifdef ARCH_PROVIDES_TIMER
+ case CLOCK_REALTIME:
+ case CLOCK_MONOTONIC:
+ case CLOCK_MONOTONIC_RAW:
+ case CLOCK_BOOTTIME:
+ nsec = MONOTONIC_RES_NSEC;
+ break;
+#endif
+ default:
+ return clock_getres_fallback(clock, res);
+ }
+
+ if (likely(res != NULL)) {
+ res->tv_sec = 0;
+ res->tv_nsec = nsec;
+ }
+
+ return 0;
+}
+
+notrace time_t __vdso_time(time_t *t)
+{
+ const struct vdso_data *vd = __get_datapage();
+
+#ifdef USE_SYSCALL
+ time_t result;
+
+ if (vd->use_syscall & USE_SYSCALL_MASK) {
+ /* Facsimile of syscall implementation (faster by a few ns) */
+ struct timeval tv;
+ int ret = gettimeofday_fallback(&tv, NULL);
+
+ if (ret < 0)
+ return ret;
+ result = tv.tv_sec;
+ } else {
+ result = READ_ONCE(vd->xtime_coarse_sec);
+ }
+#else
+ time_t result = READ_ONCE(vd->xtime_coarse_sec);
+#endif
+
+ if (t)
+ *t = result;
+ return result;
+}
diff --git a/lib/vsprintf.c b/lib/vsprintf.c
index 0967771d..79ba3cc 100644
--- a/lib/vsprintf.c
+++ b/lib/vsprintf.c
@@ -1391,9 +1391,6 @@ char *clock(char *buf, char *end, struct clk *clk, struct printf_spec spec,
return string(buf, end, NULL, spec);
switch (fmt[1]) {
- case 'r':
- return number(buf, end, clk_get_rate(clk), spec);
-
case 'n':
default:
#ifdef CONFIG_COMMON_CLK
diff --git a/mm/Kconfig b/mm/Kconfig
index 3363a70..17c842e 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -271,6 +271,18 @@
it and then we would be really interested to hear about that at
linux-mm@kvack.org.
+config PROCESS_RECLAIM
+ bool "Enable process reclaim"
+ depends on PROC_FS
+ default n
+ help
+ It allows to reclaim pages of the process by /proc/pid/reclaim.
+
+ (echo file > /proc/PID/reclaim) reclaims file-backed pages only.
+ (echo anon > /proc/PID/reclaim) reclaims anonymous pages only.
+ (echo all > /proc/PID/reclaim) reclaims all pages.
+
+ Any other value is ignored.
#
# support for page migration
#
@@ -638,6 +650,22 @@
information to userspace via debugfs.
If unsure, say N.
+config MM_EVENT_STAT
+ bool "Track per-process MM event"
+ depends on MMU
+ help
+ This option enables per-process mm event stat(e.g., fault, reclaim,
+ compaction and so on ) with some interval(Default is 0.5sec).
+ Admin can see the stat from trace file via debugfs(e.g.,
+ /sys/kernel/debug/tracing/trace)
+
+ It includes max/average memory allocation latency for the interval
+ as well as event count so that admin can see what happens in VM side
+ (how many each event happens and how much processes spent time for
+ the MM event). If it's too large, that would be not good situation.
+
+ System can dump the trace into bugreport when user allows the dump.
+
config GENERIC_EARLY_IOREMAP
bool
@@ -675,6 +703,7 @@
depends on ARCH_SUPPORTS_DEFERRED_STRUCT_PAGE_INIT
depends on NO_BOOTMEM && MEMORY_HOTPLUG
depends on !FLATMEM
+ depends on !NEED_PER_CPU_KM
help
Ordinarily all struct pages are initialised during early boot in a
single thread. On very large machines this can take a considerable
@@ -730,19 +759,3 @@
always using ZONE_DMA memory.
If unsure, say "n".
-
-config PROCESS_RECLAIM
- bool "Enable process reclaim"
- depends on PROC_FS
- default n
- help
- It allows to reclaim pages of the process by /proc/pid/reclaim.
-
- (echo file > /proc/PID/reclaim) reclaims file-backed pages only.
- (echo anon > /proc/PID/reclaim) reclaims anonymous pages only.
- (echo all > /proc/PID/reclaim) reclaims all pages.
-
- (echo addr size-byte > /proc/PID/reclaim) reclaims pages in
- (addr, addr + size-bytes) of the process.
-
- Any other vaule is ignored.
diff --git a/mm/Makefile b/mm/Makefile
index 7a9642f..457892d 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -26,6 +26,9 @@
mlock.o mmap.o mprotect.o mremap.o msync.o rmap.o \
vmalloc.o pagewalk.o pgtable-generic.o
+ifdef CONFIG_MM_EVENT_STAT
+mmu-$(CONFIG_MMU) += mm_event.o
+endif
ifdef CONFIG_CROSS_MEMORY_ATTACH
mmu-$(CONFIG_MMU) += process_vm_access.o
endif
@@ -100,4 +103,3 @@
obj-$(CONFIG_FRAME_VECTOR) += frame_vector.o
obj-$(CONFIG_DEBUG_PAGE_REF) += debug_page_ref.o
obj-$(CONFIG_HARDENED_USERCOPY) += usercopy.o
-obj-$(CONFIG_PROCESS_RECLAIM) += process_reclaim.o
diff --git a/mm/compaction.c b/mm/compaction.c
index f002a7f..3d5ae4d 100644
--- a/mm/compaction.c
+++ b/mm/compaction.c
@@ -20,6 +20,7 @@
#include <linux/kthread.h>
#include <linux/freezer.h>
#include <linux/page_owner.h>
+#include <linux/psi.h>
#include "internal.h"
#ifdef CONFIG_COMPACTION
@@ -1507,6 +1508,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro
unsigned long end_pfn = zone_end_pfn(zone);
const int migratetype = gfpflags_to_migratetype(cc->gfp_mask);
const bool sync = cc->mode != MIGRATE_ASYNC;
+ ktime_t event_ts;
ret = compaction_suitable(zone, cc->order, cc->alloc_flags,
cc->classzone_idx);
@@ -1552,6 +1554,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro
cc->last_migrated_pfn = 0;
+ mm_event_start(&event_ts);
trace_mm_compaction_begin(start_pfn, cc->migrate_pfn,
cc->free_pfn, end_pfn, sync);
@@ -1637,6 +1640,7 @@ static enum compact_result compact_zone(struct zone *zone, struct compact_contro
}
out:
+ mm_event_end(MM_COMPACTION, event_ts);
/*
* Release free pages and update where the free scanner should restart,
* so we don't leave any returned pages behind in the next attempt.
@@ -1714,11 +1718,13 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
struct zoneref *z;
struct zone *zone;
enum compact_result rc = COMPACT_SKIPPED;
+ ktime_t event_ts;
/* Check if the GFP flags allow compaction */
if (!may_enter_fs || !may_perform_io)
return COMPACT_SKIPPED;
+ mm_event_start(&event_ts);
trace_mm_compaction_try_to_compact_pages(order, gfp_mask, prio);
/* Compact each zone in the list */
@@ -1768,6 +1774,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,
break;
}
+ mm_event_end(MM_COMPACTION, event_ts);
return rc;
}
@@ -2005,11 +2012,15 @@ static int kcompactd(void *p)
pgdat->kcompactd_classzone_idx = pgdat->nr_zones - 1;
while (!kthread_should_stop()) {
+ unsigned long pflags;
+
trace_mm_compaction_kcompactd_sleep(pgdat->node_id);
wait_event_freezable(pgdat->kcompactd_wait,
kcompactd_work_requested(pgdat));
+ psi_memstall_enter(&pflags);
kcompactd_do_work(pgdat);
+ psi_memstall_leave(&pflags);
}
return 0;
diff --git a/mm/debug.c b/mm/debug.c
index 9feb699..bebe48a 100644
--- a/mm/debug.c
+++ b/mm/debug.c
@@ -95,7 +95,7 @@ EXPORT_SYMBOL(dump_vma);
void dump_mm(const struct mm_struct *mm)
{
- pr_emerg("mm %p mmap %p seqnum %d task_size %lu\n"
+ pr_emerg("mm %p mmap %p seqnum %llu task_size %lu\n"
#ifdef CONFIG_MMU
"get_unmapped_area %p\n"
#endif
@@ -125,7 +125,7 @@ void dump_mm(const struct mm_struct *mm)
#endif
"def_flags: %#lx(%pGv)\n",
- mm, mm->mmap, mm->vmacache_seqnum, mm->task_size,
+ mm, mm->mmap, (long long) mm->vmacache_seqnum, mm->task_size,
#ifdef CONFIG_MMU
mm->get_unmapped_area,
#endif
diff --git a/mm/fadvise.c b/mm/fadvise.c
index 6c707bf..eb3269e 100644
--- a/mm/fadvise.c
+++ b/mm/fadvise.c
@@ -68,8 +68,12 @@ SYSCALL_DEFINE4(fadvise64_64, int, fd, loff_t, offset, loff_t, len, int, advice)
goto out;
}
- /* Careful about overflows. Len == 0 means "as much as possible" */
- endbyte = offset + len;
+ /*
+ * Careful about overflows. Len == 0 means "as much as possible". Use
+ * unsigned math because signed overflows are undefined and UBSan
+ * complains.
+ */
+ endbyte = (u64)offset + (u64)len;
if (!len || endbyte < len)
endbyte = -1;
else
@@ -126,7 +130,15 @@ SYSCALL_DEFINE4(fadvise64_64, int, fd, loff_t, offset, loff_t, len, int, advice)
*/
start_index = (offset+(PAGE_SIZE-1)) >> PAGE_SHIFT;
end_index = (endbyte >> PAGE_SHIFT);
- if ((endbyte & ~PAGE_MASK) != ~PAGE_MASK) {
+ /*
+ * The page at end_index will be inclusively discarded according
+ * by invalidate_mapping_pages(), so subtracting 1 from
+ * end_index means we will skip the last page. But if endbyte
+ * is page aligned or is at the end of file, we should not skip
+ * that page - discarding the last page is safe enough.
+ */
+ if ((endbyte & ~PAGE_MASK) != ~PAGE_MASK &&
+ endbyte != inode->i_size - 1) {
/* First page is tricky as 0 - 1 = -1, but pgoff_t
* is unsigned, so the end_index >= start_index
* check below would be true and we'll discard the whole
diff --git a/mm/filemap.c b/mm/filemap.c
index 211df83..ef3034e 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -35,6 +35,8 @@
#include <linux/memcontrol.h>
#include <linux/cleancache.h>
#include <linux/rmap.h>
+#include <linux/delayacct.h>
+#include <linux/psi.h>
#include "internal.h"
#define CREATE_TRACE_POINTS
@@ -426,41 +428,35 @@ int filemap_flush(struct address_space *mapping)
}
EXPORT_SYMBOL(filemap_flush);
-static int __filemap_fdatawait_range(struct address_space *mapping,
+static void __filemap_fdatawait_range(struct address_space *mapping,
loff_t start_byte, loff_t end_byte)
{
pgoff_t index = start_byte >> PAGE_SHIFT;
pgoff_t end = end_byte >> PAGE_SHIFT;
struct pagevec pvec;
int nr_pages;
- int ret = 0;
if (end_byte < start_byte)
- goto out;
+ return;
pagevec_init(&pvec, 0);
- while ((index <= end) &&
- (nr_pages = pagevec_lookup_tag(&pvec, mapping, &index,
- PAGECACHE_TAG_WRITEBACK,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1)) != 0) {
+ while (index <= end) {
unsigned i;
+ nr_pages = pagevec_lookup_range_tag(&pvec, mapping, &index,
+ end, PAGECACHE_TAG_WRITEBACK);
+ if (!nr_pages)
+ break;
+
for (i = 0; i < nr_pages; i++) {
struct page *page = pvec.pages[i];
- /* until radix tree lookup accepts end_index */
- if (page->index > end)
- continue;
-
wait_on_page_writeback(page);
- if (TestClearPageError(page))
- ret = -EIO;
+ ClearPageError(page);
}
pagevec_release(&pvec);
cond_resched();
}
-out:
- return ret;
}
/**
@@ -480,14 +476,8 @@ static int __filemap_fdatawait_range(struct address_space *mapping,
int filemap_fdatawait_range(struct address_space *mapping, loff_t start_byte,
loff_t end_byte)
{
- int ret, ret2;
-
- ret = __filemap_fdatawait_range(mapping, start_byte, end_byte);
- ret2 = filemap_check_errors(mapping);
- if (!ret)
- ret = ret2;
-
- return ret;
+ __filemap_fdatawait_range(mapping, start_byte, end_byte);
+ return filemap_check_errors(mapping);
}
EXPORT_SYMBOL(filemap_fdatawait_range);
@@ -747,12 +737,9 @@ int add_to_page_cache_lru(struct page *page, struct address_space *mapping,
* data from the working set, only to cache data that will
* get overwritten with something else, is a waste of memory.
*/
- if (!(gfp_mask & __GFP_WRITE) &&
- shadow && workingset_refault(shadow)) {
- SetPageActive(page);
- workingset_activation(page);
- } else
- ClearPageActive(page);
+ WARN_ON_ONCE(PageActive(page));
+ if (!(gfp_mask & __GFP_WRITE) && shadow)
+ workingset_refault(page, shadow);
lru_cache_add(page);
}
return ret;
@@ -790,46 +777,177 @@ EXPORT_SYMBOL(__page_cache_alloc);
* at a cost of "thundering herd" phenomena during rare hash
* collisions.
*/
-wait_queue_head_t *page_waitqueue(struct page *page)
+#define PAGE_WAIT_TABLE_BITS 8
+#define PAGE_WAIT_TABLE_SIZE (1 << PAGE_WAIT_TABLE_BITS)
+static wait_queue_head_t page_wait_table[PAGE_WAIT_TABLE_SIZE] __cacheline_aligned;
+
+static wait_queue_head_t *page_waitqueue(struct page *page)
{
- return bit_waitqueue(page, 0);
+ return &page_wait_table[hash_ptr(page, PAGE_WAIT_TABLE_BITS)];
}
-EXPORT_SYMBOL(page_waitqueue);
+
+void __init pagecache_init(void)
+{
+ int i;
+
+ for (i = 0; i < PAGE_WAIT_TABLE_SIZE; i++)
+ init_waitqueue_head(&page_wait_table[i]);
+
+ page_writeback_init();
+}
+
+struct wait_page_key {
+ struct page *page;
+ int bit_nr;
+ int page_match;
+};
+
+struct wait_page_queue {
+ struct page *page;
+ int bit_nr;
+ wait_queue_t wait;
+};
+
+static int wake_page_function(wait_queue_t *wait, unsigned mode, int sync, void *arg)
+{
+ struct wait_page_key *key = arg;
+ struct wait_page_queue *wait_page
+ = container_of(wait, struct wait_page_queue, wait);
+
+ if (wait_page->page != key->page)
+ return 0;
+ key->page_match = 1;
+
+ if (wait_page->bit_nr != key->bit_nr)
+ return 0;
+ if (test_bit(key->bit_nr, &key->page->flags))
+ return 0;
+
+ return autoremove_wake_function(wait, mode, sync, key);
+}
+
+void wake_up_page_bit(struct page *page, int bit_nr)
+{
+ wait_queue_head_t *q = page_waitqueue(page);
+ struct wait_page_key key;
+ unsigned long flags;
+
+ key.page = page;
+ key.bit_nr = bit_nr;
+ key.page_match = 0;
+
+ spin_lock_irqsave(&q->lock, flags);
+ __wake_up_locked_key(q, TASK_NORMAL, &key);
+ /*
+ * It is possible for other pages to have collided on the waitqueue
+ * hash, so in that case check for a page match. That prevents a long-
+ * term waiter
+ *
+ * It is still possible to miss a case here, when we woke page waiters
+ * and removed them from the waitqueue, but there are still other
+ * page waiters.
+ */
+ if (!waitqueue_active(q) || !key.page_match) {
+ ClearPageWaiters(page);
+ /*
+ * It's possible to miss clearing Waiters here, when we woke
+ * our page waiters, but the hashed waitqueue has waiters for
+ * other pages on it.
+ *
+ * That's okay, it's a rare case. The next waker will clear it.
+ */
+ }
+ spin_unlock_irqrestore(&q->lock, flags);
+}
+EXPORT_SYMBOL(wake_up_page_bit);
+
+static inline int wait_on_page_bit_common(wait_queue_head_t *q,
+ struct page *page, int bit_nr, int state, bool lock)
+{
+ struct wait_page_queue wait_page;
+ wait_queue_t *wait = &wait_page.wait;
+ bool thrashing = false;
+ unsigned long pflags;
+ int ret = 0;
+
+ if (bit_nr == PG_locked &&
+ !PageUptodate(page) && PageWorkingset(page)) {
+ if (!PageSwapBacked(page))
+ delayacct_thrashing_start();
+ psi_memstall_enter(&pflags);
+ thrashing = true;
+ }
+
+ init_wait(wait);
+ wait->func = wake_page_function;
+ wait_page.page = page;
+ wait_page.bit_nr = bit_nr;
+
+ for (;;) {
+ spin_lock_irq(&q->lock);
+
+ if (likely(list_empty(&wait->task_list))) {
+ if (lock)
+ __add_wait_queue_tail_exclusive(q, wait);
+ else
+ __add_wait_queue(q, wait);
+ SetPageWaiters(page);
+ }
+
+ set_current_state(state);
+
+ spin_unlock_irq(&q->lock);
+
+ if (likely(test_bit(bit_nr, &page->flags))) {
+ io_schedule();
+ }
+
+ if (lock) {
+ if (!test_and_set_bit_lock(bit_nr, &page->flags))
+ break;
+ } else {
+ if (!test_bit(bit_nr, &page->flags))
+ break;
+ }
+
+ if (unlikely(signal_pending_state(state, current))) {
+ ret = -EINTR;
+ break;
+ }
+ }
+
+ finish_wait(q, wait);
+
+ if (thrashing) {
+ if (!PageSwapBacked(page))
+ delayacct_thrashing_end();
+ psi_memstall_leave(&pflags);
+ }
+
+ /*
+ * A signal could leave PageWaiters set. Clearing it here if
+ * !waitqueue_active would be possible (by open-coding finish_wait),
+ * but still fail to catch it in the case of wait hash collision. We
+ * already can fail to clear wait hash collision cases, so don't
+ * bother with signals either.
+ */
+
+ return ret;
+}
void wait_on_page_bit(struct page *page, int bit_nr)
{
- DEFINE_WAIT_BIT(wait, &page->flags, bit_nr);
-
- if (test_bit(bit_nr, &page->flags))
- __wait_on_bit(page_waitqueue(page), &wait, bit_wait_io,
- TASK_UNINTERRUPTIBLE);
+ wait_queue_head_t *q = page_waitqueue(page);
+ wait_on_page_bit_common(q, page, bit_nr, TASK_UNINTERRUPTIBLE, false);
}
EXPORT_SYMBOL(wait_on_page_bit);
int wait_on_page_bit_killable(struct page *page, int bit_nr)
{
- DEFINE_WAIT_BIT(wait, &page->flags, bit_nr);
-
- if (!test_bit(bit_nr, &page->flags))
- return 0;
-
- return __wait_on_bit(page_waitqueue(page), &wait,
- bit_wait_io, TASK_KILLABLE);
+ wait_queue_head_t *q = page_waitqueue(page);
+ return wait_on_page_bit_common(q, page, bit_nr, TASK_KILLABLE, false);
}
-int wait_on_page_bit_killable_timeout(struct page *page,
- int bit_nr, unsigned long timeout)
-{
- DEFINE_WAIT_BIT(wait, &page->flags, bit_nr);
-
- wait.key.timeout = jiffies + timeout;
- if (!test_bit(bit_nr, &page->flags))
- return 0;
- return __wait_on_bit(page_waitqueue(page), &wait,
- bit_wait_io_timeout, TASK_KILLABLE);
-}
-EXPORT_SYMBOL_GPL(wait_on_page_bit_killable_timeout);
-
/**
* add_page_wait_queue - Add an arbitrary waiter to a page's wait queue
* @page: Page defining the wait queue of interest
@@ -844,6 +962,7 @@ void add_page_wait_queue(struct page *page, wait_queue_t *waiter)
spin_lock_irqsave(&q->lock, flags);
__add_wait_queue(q, waiter);
+ SetPageWaiters(page);
spin_unlock_irqrestore(&q->lock, flags);
}
EXPORT_SYMBOL_GPL(add_page_wait_queue);
@@ -928,23 +1047,19 @@ EXPORT_SYMBOL_GPL(page_endio);
* __lock_page - get a lock on the page, assuming we need to sleep to get it
* @page: the page to lock
*/
-void __lock_page(struct page *page)
+void __lock_page(struct page *__page)
{
- struct page *page_head = compound_head(page);
- DEFINE_WAIT_BIT(wait, &page_head->flags, PG_locked);
-
- __wait_on_bit_lock(page_waitqueue(page_head), &wait, bit_wait_io,
- TASK_UNINTERRUPTIBLE);
+ struct page *page = compound_head(__page);
+ wait_queue_head_t *q = page_waitqueue(page);
+ wait_on_page_bit_common(q, page, PG_locked, TASK_UNINTERRUPTIBLE, true);
}
EXPORT_SYMBOL(__lock_page);
-int __lock_page_killable(struct page *page)
+int __lock_page_killable(struct page *__page)
{
- struct page *page_head = compound_head(page);
- DEFINE_WAIT_BIT(wait, &page_head->flags, PG_locked);
-
- return __wait_on_bit_lock(page_waitqueue(page_head), &wait,
- bit_wait_io, TASK_KILLABLE);
+ struct page *page = compound_head(__page);
+ wait_queue_head_t *q = page_waitqueue(page);
+ return wait_on_page_bit_common(q, page, PG_locked, TASK_KILLABLE, true);
}
EXPORT_SYMBOL_GPL(__lock_page_killable);
@@ -1190,6 +1305,9 @@ EXPORT_SYMBOL(find_lock_entry);
* @gfp_mask and added to the page cache and the VM's LRU
* list. The page is returned locked and with an increased
* refcount. Otherwise, %NULL is returned.
+ * FGP_FOR_MMAP: Similar to FGP_CREAT, only we want to allow the caller to do
+ * its own locking dance if the page is already in cache, or unlock the page
+ * before returning if we had to add the page to pagecache.
*
* If FGP_LOCK or FGP_CREAT are specified then the function may sleep even
* if the GFP flags specified for FGP_CREAT are atomic.
@@ -1242,7 +1360,7 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
if (!page)
return NULL;
- if (WARN_ON_ONCE(!(fgp_flags & FGP_LOCK)))
+ if (WARN_ON_ONCE(!(fgp_flags & (FGP_LOCK | FGP_FOR_MMAP))))
fgp_flags |= FGP_LOCK;
/* Init accessed so avoid atomic mark_page_accessed later */
@@ -1256,6 +1374,14 @@ struct page *pagecache_get_page(struct address_space *mapping, pgoff_t offset,
if (err == -EEXIST)
goto repeat;
}
+
+ /*
+ * add_to_page_cache_lru lock's the page, and for mmap we expect
+ * a unlocked page.
+ */
+ if (fgp_flags & FGP_FOR_MMAP)
+ unlock_page(page);
+
}
return page;
@@ -1493,9 +1619,10 @@ unsigned find_get_pages_contig(struct address_space *mapping, pgoff_t index,
EXPORT_SYMBOL(find_get_pages_contig);
/**
- * find_get_pages_tag - find and return pages that match @tag
+ * find_get_pages_range_tag - find and return pages in given range matching @tag
* @mapping: the address_space to search
* @index: the starting page index
+ * @end: The final page index (inclusive)
* @tag: the tag index
* @nr_pages: the maximum number of pages
* @pages: where the resulting pages are placed
@@ -1503,8 +1630,9 @@ EXPORT_SYMBOL(find_get_pages_contig);
* Like find_get_pages, except we only return pages which are tagged with
* @tag. We update @index to index the next page for the traversal.
*/
-unsigned find_get_pages_tag(struct address_space *mapping, pgoff_t *index,
- int tag, unsigned int nr_pages, struct page **pages)
+unsigned find_get_pages_range_tag(struct address_space *mapping, pgoff_t *index,
+ pgoff_t end, int tag, unsigned int nr_pages,
+ struct page **pages)
{
struct radix_tree_iter iter;
void **slot;
@@ -1517,6 +1645,9 @@ unsigned find_get_pages_tag(struct address_space *mapping, pgoff_t *index,
radix_tree_for_each_tagged(slot, &mapping->page_tree,
&iter, *index, tag) {
struct page *head, *page;
+
+ if (iter.index > end)
+ break;
repeat:
page = radix_tree_deref_slot(slot);
if (unlikely(!page))
@@ -1558,18 +1689,28 @@ unsigned find_get_pages_tag(struct address_space *mapping, pgoff_t *index,
}
pages[ret] = page;
- if (++ret == nr_pages)
- break;
+ if (++ret == nr_pages) {
+ *index = pages[ret - 1]->index + 1;
+ goto out;
+ }
}
+ /*
+ * We come here when we got at @end. We take care to not overflow the
+ * index @index as it confuses some of the callers. This breaks the
+ * iteration when there is page at index -1 but that is already broken
+ * anyway.
+ */
+ if (end == (pgoff_t)-1)
+ *index = (pgoff_t)-1;
+ else
+ *index = end + 1;
+out:
rcu_read_unlock();
- if (ret)
- *index = pages[ret - 1]->index + 1;
-
return ret;
}
-EXPORT_SYMBOL(find_get_pages_tag);
+EXPORT_SYMBOL(find_get_pages_range_tag);
/**
* find_get_entries_tag - find and return entries that match @tag
@@ -1704,7 +1845,9 @@ static ssize_t do_generic_file_read(struct file *filp, loff_t *ppos,
pgoff_t end_index;
loff_t isize;
unsigned long nr, ret;
+ ktime_t event_ts;
+ event_ts.tv64 = 0;
cond_resched();
find_page:
if (fatal_signal_pending(current)) {
@@ -1714,6 +1857,7 @@ static ssize_t do_generic_file_read(struct file *filp, loff_t *ppos,
page = find_get_page(mapping, index);
if (!page) {
+ mm_event_start(&event_ts);
page_cache_sync_readahead(mapping,
ra, filp,
index, last_index - index);
@@ -1755,6 +1899,8 @@ static ssize_t do_generic_file_read(struct file *filp, loff_t *ppos,
unlock_page(page);
}
page_ok:
+ if (event_ts.tv64 != 0)
+ mm_event_end(MM_READ_IO, event_ts);
/*
* i_size must be checked after we know the page is Uptodate.
*
@@ -1977,62 +2123,91 @@ generic_file_read_iter(struct kiocb *iocb, struct iov_iter *iter)
EXPORT_SYMBOL(generic_file_read_iter);
#ifdef CONFIG_MMU
-/**
- * page_cache_read - adds requested page to the page cache if not already there
- * @file: file to read
- * @offset: page index
- * @gfp_mask: memory allocation flags
- *
- * This adds the requested page to the page cache if it isn't already there,
- * and schedules an I/O to read in its contents from disk.
- */
-static int page_cache_read(struct file *file, pgoff_t offset, gfp_t gfp_mask)
-{
- struct address_space *mapping = file->f_mapping;
- struct page *page;
- int ret;
-
- do {
- page = __page_cache_alloc(gfp_mask|__GFP_COLD);
- if (!page)
- return -ENOMEM;
-
- ret = add_to_page_cache_lru(page, mapping, offset, gfp_mask);
- if (ret == 0)
- ret = mapping->a_ops->readpage(file, page);
- else if (ret == -EEXIST)
- ret = 0; /* losing race to add is OK */
-
- put_page(page);
-
- } while (ret == AOP_TRUNCATED_PAGE);
-
- return ret;
-}
-
#define MMAP_LOTSAMISS (100)
+static struct file *maybe_unlock_mmap_for_io(struct vm_area_struct *vma,
+ unsigned long flags, struct file *fpin)
+{
+ if (fpin)
+ return fpin;
+
+ /*
+ * FAULT_FLAG_RETRY_NOWAIT means we don't want to wait on page locks or
+ * anything, so we only pin the file and drop the mmap_sem if only
+ * FAULT_FLAG_ALLOW_RETRY is set.
+ */
+ if ((flags & (FAULT_FLAG_ALLOW_RETRY | FAULT_FLAG_RETRY_NOWAIT)) ==
+ FAULT_FLAG_ALLOW_RETRY) {
+ fpin = get_file(vma->vm_file);
+ up_read(&vma->vm_mm->mmap_sem);
+ }
+ return fpin;
+}
+
/*
- * Synchronous readahead happens when we don't even find
- * a page in the page cache at all.
+ * lock_page_maybe_drop_mmap - lock the page, possibly dropping the mmap_sem
+ * @vmf - the vm_fault for this fault.
+ * @page - the page to lock.
+ * @fpin - the pointer to the file we may pin (or is already pinned).
+ *
+ * This works similar to lock_page_or_retry in that it can drop the mmap_sem.
+ * It differs in that it actually returns the page locked if it returns 1 and 0
+ * if it couldn't lock the page. If we did have to drop the mmap_sem then fpin
+ * will point to the pinned file and needs to be fput()'ed at a later point.
*/
-static void do_sync_mmap_readahead(struct vm_area_struct *vma,
+static int lock_page_maybe_drop_mmap(struct vm_area_struct *vma,
+ unsigned long flags, struct page *page, struct file **fpin)
+{
+ if (trylock_page(page))
+ return 1;
+
+ if (flags & FAULT_FLAG_RETRY_NOWAIT)
+ return 0;
+ *fpin = maybe_unlock_mmap_for_io(vma, flags, *fpin);
+ if (flags & FAULT_FLAG_KILLABLE) {
+ if (__lock_page_killable(page)) {
+ /*
+ * We didn't have the right flags to drop the mmap_sem,
+ * but all fault_handlers only check for fatal signals
+ * if we return VM_FAULT_RETRY, so we need to drop the
+ * mmap_sem here and return 0 if we don't have a fpin.
+ */
+ if (*fpin == NULL)
+ up_read(&vma->vm_mm->mmap_sem);
+ return 0;
+ }
+ } else
+ __lock_page(page);
+ return 1;
+}
+
+/*
+ * Synchronous readahead happens when we don't even find a page in the page
+ * cache at all. We don't want to perform IO under the mmap sem, so if we have
+ * to drop the mmap sem we return the file that was pinned in order for us to do
+ * that. If we didn't pin a file then we return NULL. The file that is
+ * returned needs to be fput()'ed when we're done with it.
+ */
+static struct file *do_sync_mmap_readahead(struct vm_area_struct *vma,
+ unsigned long flags,
struct file_ra_state *ra,
struct file *file,
pgoff_t offset)
{
+ struct file *fpin = NULL;
struct address_space *mapping = file->f_mapping;
/* If we don't want any read-ahead, don't bother */
if (vma->vm_flags & VM_RAND_READ)
- return;
+ return fpin;
if (!ra->ra_pages)
- return;
+ return fpin;
if (vma->vm_flags & VM_SEQ_READ) {
+ fpin = maybe_unlock_mmap_for_io(vma, flags, fpin);
page_cache_sync_readahead(mapping, ra, file, offset,
ra->ra_pages);
- return;
+ return fpin;
}
/* Avoid banging the cache line if not needed */
@@ -2044,37 +2219,45 @@ static void do_sync_mmap_readahead(struct vm_area_struct *vma,
* stop bothering with read-ahead. It will only hurt.
*/
if (ra->mmap_miss > MMAP_LOTSAMISS)
- return;
+ return fpin;
/*
* mmap read-around
*/
+ fpin = maybe_unlock_mmap_for_io(vma, flags, fpin);
ra->start = max_t(long, 0, offset - ra->ra_pages / 2);
ra->size = ra->ra_pages;
ra->async_size = ra->ra_pages / 4;
ra_submit(ra, mapping, file);
+ return fpin;
}
/*
* Asynchronous readahead happens when we find the page and PG_readahead,
- * so we want to possibly extend the readahead further..
+ * so we want to possibly extend the readahead further. We return the file that
+ * was pinned if we have to drop the mmap_sem in order to do IO.
*/
-static void do_async_mmap_readahead(struct vm_area_struct *vma,
+static struct file *do_async_mmap_readahead(struct vm_area_struct *vma,
+ unsigned long flags,
struct file_ra_state *ra,
struct file *file,
struct page *page,
pgoff_t offset)
{
struct address_space *mapping = file->f_mapping;
+ struct file *fpin = NULL;
/* If we don't want any read-ahead, don't bother */
if (vma->vm_flags & VM_RAND_READ)
- return;
+ return fpin;
if (ra->mmap_miss > 0)
ra->mmap_miss--;
- if (PageReadahead(page))
+ if (PageReadahead(page)) {
+ fpin = maybe_unlock_mmap_for_io(vma, flags, fpin);
page_cache_async_readahead(mapping, ra, file,
page, offset, ra->ra_pages);
+ }
+ return fpin;
}
/**
@@ -2105,6 +2288,7 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
{
int error;
struct file *file = vma->vm_file;
+ struct file *fpin = NULL;
struct address_space *mapping = file->f_mapping;
struct file_ra_state *ra = &file->f_ra;
struct inode *inode = mapping->host;
@@ -2126,23 +2310,27 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
* We found the page, so try async readahead before
* waiting for the lock.
*/
- do_async_mmap_readahead(vma, ra, file, page, offset);
+ fpin = do_async_mmap_readahead(vma, vmf->flags, ra,
+ file, page, offset);
} else if (!page) {
/* No page in the page cache at all */
- do_sync_mmap_readahead(vma, ra, file, offset);
count_vm_event(PGMAJFAULT);
mem_cgroup_count_vm_event(vma->vm_mm, PGMAJFAULT);
ret = VM_FAULT_MAJOR;
+ fpin = do_sync_mmap_readahead(vma, vmf->flags, ra,
+ file, offset);
retry_find:
- page = find_get_page(mapping, offset);
- if (!page)
- goto no_cached_page;
+ page = pagecache_get_page(mapping, offset,
+ FGP_CREAT|FGP_FOR_MMAP,
+ vmf->gfp_mask);
+ if (!page) {
+ if (fpin)
+ goto out_retry;
+ return VM_FAULT_OOM;
+ }
}
-
- if (!lock_page_or_retry(page, vma->vm_mm, vmf->flags)) {
- put_page(page);
- return ret | VM_FAULT_RETRY;
- }
+ if (!lock_page_maybe_drop_mmap(vma, vmf->flags, page, &fpin))
+ goto out_retry;
/* Did it get truncated? */
if (unlikely(page->mapping != mapping)) {
@@ -2160,6 +2348,16 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
goto page_not_uptodate;
/*
+ * We've made it this far and we had to drop our mmap_sem, now is the
+ * time to return to the upper layer and have it re-find the vma and
+ * redo the fault.
+ */
+ if (fpin) {
+ unlock_page(page);
+ goto out_retry;
+ }
+
+ /*
* Found the page and have a reference on it.
* We must recheck i_size under page lock.
*/
@@ -2173,30 +2371,6 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
vmf->page = page;
return ret | VM_FAULT_LOCKED;
-no_cached_page:
- /*
- * We're only likely to ever get here if MADV_RANDOM is in
- * effect.
- */
- error = page_cache_read(file, offset, vmf->gfp_mask);
-
- /*
- * The page we want has now been added to the page cache.
- * In the unlikely event that someone removed it in the
- * meantime, we'll just come back here and read it again.
- */
- if (error >= 0)
- goto retry_find;
-
- /*
- * An error return from page_cache_read can result if the
- * system is low on memory, or a problem occurs while trying
- * to schedule I/O.
- */
- if (error == -ENOMEM)
- return VM_FAULT_OOM;
- return VM_FAULT_SIGBUS;
-
page_not_uptodate:
/*
* Umm, take care of errors if the page isn't up-to-date.
@@ -2205,12 +2379,15 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
* and we need to check for errors.
*/
ClearPageError(page);
+ fpin = maybe_unlock_mmap_for_io(vma, vmf->flags, fpin);
error = mapping->a_ops->readpage(file, page);
if (!error) {
wait_on_page_locked(page);
if (!PageUptodate(page))
error = -EIO;
}
+ if (fpin)
+ goto out_retry;
put_page(page);
if (!error || error == AOP_TRUNCATED_PAGE)
@@ -2219,6 +2396,18 @@ int filemap_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
/* Things didn't work out. Return zero to tell the mm layer so. */
shrink_readahead_size_eio(file, ra);
return VM_FAULT_SIGBUS;
+
+out_retry:
+ /*
+ * We dropped the mmap_sem, we need to return to the fault handler to
+ * re-find the vma and come back and find our hopefully still populated
+ * page.
+ */
+ if (page)
+ put_page(page);
+ if (fpin)
+ fput(fpin);
+ return ret | VM_FAULT_RETRY;
}
EXPORT_SYMBOL(filemap_fault);
diff --git a/mm/frame_vector.c b/mm/frame_vector.c
index 375a103..d73eed0 100644
--- a/mm/frame_vector.c
+++ b/mm/frame_vector.c
@@ -61,8 +61,10 @@ int get_vaddr_frames(unsigned long start, unsigned int nr_frames,
* get_user_pages_longterm() and disallow it for filesystem-dax
* mappings.
*/
- if (vma_is_fsdax(vma))
- return -EOPNOTSUPP;
+ if (vma_is_fsdax(vma)) {
+ ret = -EOPNOTSUPP;
+ goto out;
+ }
if (!(vma->vm_flags & (VM_IO | VM_PFNMAP))) {
vec->got_ref = true;
diff --git a/mm/gup.c b/mm/gup.c
index 6c3b4e8..d71da72 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -430,6 +430,9 @@ static int check_vma_flags(struct vm_area_struct *vma, unsigned long gup_flags)
if (vm_flags & (VM_IO | VM_PFNMAP))
return -EFAULT;
+ if (gup_flags & FOLL_ANON && !vma_is_anonymous(vma))
+ return -EFAULT;
+
if (write) {
if (!(vm_flags & VM_WRITE)) {
if (!(gup_flags & FOLL_FORCE))
@@ -1119,8 +1122,6 @@ int __mm_populate(unsigned long start, unsigned long len, int ignore_errors)
int locked = 0;
long ret = 0;
- VM_BUG_ON(start & ~PAGE_MASK);
- VM_BUG_ON(len != PAGE_ALIGN(len));
end = start + len;
for (nstart = start; nstart < end; nstart = nend) {
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index e2982ea..c6779ac 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -542,7 +542,8 @@ static int __do_huge_pmd_anonymous_page(struct fault_env *fe, struct page *page,
VM_BUG_ON_PAGE(!PageCompound(page), page);
- if (mem_cgroup_try_charge(page, vma->vm_mm, gfp, &memcg, true)) {
+ if (mem_cgroup_try_charge(page, vma->vm_mm, gfp | __GFP_NORETRY, &memcg,
+ true)) {
put_page(page);
count_vm_event(THP_FAULT_FALLBACK);
return VM_FAULT_FALLBACK;
@@ -1060,7 +1061,7 @@ int do_huge_pmd_wp_page(struct fault_env *fe, pmd_t orig_pmd)
}
if (unlikely(mem_cgroup_try_charge(new_page, vma->vm_mm,
- huge_gfp, &memcg, true))) {
+ huge_gfp | __GFP_NORETRY, &memcg, true))) {
put_page(new_page);
split_huge_pmd(vma, fe->pmd, fe->address);
if (page)
@@ -1258,12 +1259,12 @@ int do_huge_pmd_numa_page(struct fault_env *fe, pmd_t pmd)
/* Migration could have started since the pmd_trans_migrating check */
if (!page_locked) {
+ page_nid = -1;
if (!get_page_unless_zero(page))
goto out_unlock;
spin_unlock(fe->ptl);
wait_on_page_locked(page);
put_page(page);
- page_nid = -1;
goto out;
}
@@ -1444,7 +1445,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
unsigned long new_addr, unsigned long old_end,
- pmd_t *old_pmd, pmd_t *new_pmd, bool *need_flush)
+ pmd_t *old_pmd, pmd_t *new_pmd)
{
spinlock_t *old_ptl, *new_ptl;
pmd_t pmd;
@@ -1475,7 +1476,7 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
if (new_ptl != old_ptl)
spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING);
pmd = pmdp_huge_get_and_clear(mm, old_addr, old_pmd);
- if (pmd_present(pmd) && pmd_dirty(pmd))
+ if (pmd_present(pmd))
force_flush = true;
VM_BUG_ON(!pmd_none(*new_pmd));
@@ -1486,12 +1487,10 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
}
set_pmd_at(mm, new_addr, new_pmd, pmd_mksoft_dirty(pmd));
- if (new_ptl != old_ptl)
- spin_unlock(new_ptl);
if (force_flush)
flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
- else
- *need_flush = true;
+ if (new_ptl != old_ptl)
+ spin_unlock(new_ptl);
spin_unlock(old_ptl);
return true;
}
@@ -1641,6 +1640,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
if (vma_is_dax(vma))
return;
page = pmd_page(_pmd);
+ if (!PageDirty(page) && pmd_dirty(_pmd))
+ set_page_dirty(page);
if (!PageReferenced(page) && pmd_young(_pmd))
SetPageReferenced(page);
page_remove_rmap(page, true);
@@ -1838,7 +1839,7 @@ void vma_adjust_trans_huge(struct vm_area_struct *vma,
}
}
-static void freeze_page(struct page *page)
+static void unmap_page(struct page *page)
{
enum ttu_flags ttu_flags = TTU_IGNORE_MLOCK | TTU_IGNORE_ACCESS |
TTU_RMAP_LOCKED;
@@ -1861,7 +1862,7 @@ static void freeze_page(struct page *page)
VM_BUG_ON_PAGE(ret, page + i - 1);
}
-static void unfreeze_page(struct page *page)
+static void remap_page(struct page *page)
{
int i;
@@ -1875,26 +1876,13 @@ static void __split_huge_page_tail(struct page *head, int tail,
struct page *page_tail = head + tail;
VM_BUG_ON_PAGE(atomic_read(&page_tail->_mapcount) != -1, page_tail);
- VM_BUG_ON_PAGE(page_ref_count(page_tail) != 0, page_tail);
/*
- * tail_page->_refcount is zero and not changing from under us. But
- * get_page_unless_zero() may be running from under us on the
- * tail_page. If we used atomic_set() below instead of atomic_inc() or
- * atomic_add(), we would then run atomic_set() concurrently with
- * get_page_unless_zero(), and atomic_set() is implemented in C not
- * using locked ops. spin_unlock on x86 sometime uses locked ops
- * because of PPro errata 66, 92, so unless somebody can guarantee
- * atomic_set() here would be safe on all archs (and not only on x86),
- * it's safer to use atomic_inc()/atomic_add().
+ * Clone page flags before unfreezing refcount.
+ *
+ * After successful get_page_unless_zero() might follow flags change,
+ * for exmaple lock_page() which set PG_waiters.
*/
- if (PageAnon(head)) {
- page_ref_inc(page_tail);
- } else {
- /* Additional pin to radix tree */
- page_ref_add(page_tail, 2);
- }
-
page_tail->flags &= ~PAGE_FLAGS_CHECK_AT_PREP;
page_tail->flags |= (head->flags &
((1L << PG_referenced) |
@@ -1902,40 +1890,47 @@ static void __split_huge_page_tail(struct page *head, int tail,
(1L << PG_mlocked) |
(1L << PG_uptodate) |
(1L << PG_active) |
+ (1L << PG_workingset) |
(1L << PG_locked) |
(1L << PG_unevictable) |
(1L << PG_dirty)));
- /*
- * After clearing PageTail the gup refcount can be released.
- * Page flags also must be visible before we make the page non-compound.
- */
+ /* ->mapping in first tail page is compound_mapcount */
+ VM_BUG_ON_PAGE(tail > 2 && page_tail->mapping != TAIL_MAPPING,
+ page_tail);
+ page_tail->mapping = head->mapping;
+ page_tail->index = head->index + tail;
+
+ /* Page flags must be visible before we make the page non-compound. */
smp_wmb();
+ /*
+ * Clear PageTail before unfreezing page refcount.
+ *
+ * After successful get_page_unless_zero() might follow put_page()
+ * which needs correct compound_head().
+ */
clear_compound_head(page_tail);
+ /* Finally unfreeze refcount. Additional reference from page cache. */
+ page_ref_unfreeze(page_tail, 1 + (!PageAnon(head) ||
+ PageSwapCache(head)));
+
if (page_is_young(head))
set_page_young(page_tail);
if (page_is_idle(head))
set_page_idle(page_tail);
- /* ->mapping in first tail page is compound_mapcount */
- VM_BUG_ON_PAGE(tail > 2 && page_tail->mapping != TAIL_MAPPING,
- page_tail);
- page_tail->mapping = head->mapping;
-
- page_tail->index = head->index + tail;
page_cpupid_xchg_last(page_tail, page_cpupid_last(head));
lru_add_page_tail(head, page_tail, lruvec, list);
}
static void __split_huge_page(struct page *page, struct list_head *list,
- unsigned long flags)
+ pgoff_t end, unsigned long flags)
{
struct page *head = compound_head(page);
struct zone *zone = page_zone(head);
struct lruvec *lruvec;
- pgoff_t end = -1;
int i;
lruvec = mem_cgroup_page_lruvec(head, zone->zone_pgdat);
@@ -1943,9 +1938,6 @@ static void __split_huge_page(struct page *page, struct list_head *list,
/* complete memcg works before add pages to LRU */
mem_cgroup_split_huge_fixup(head);
- if (!PageAnon(page))
- end = DIV_ROUND_UP(i_size_read(head->mapping->host), PAGE_SIZE);
-
for (i = HPAGE_PMD_NR - 1; i >= 1; i--) {
__split_huge_page_tail(head, i, lruvec, list);
/* Some pages can be beyond i_size: drop them from page cache */
@@ -1970,7 +1962,7 @@ static void __split_huge_page(struct page *page, struct list_head *list,
spin_unlock_irqrestore(zone_lru_lock(page_zone(head)), flags);
- unfreeze_page(head);
+ remap_page(head);
for (i = 0; i < HPAGE_PMD_NR; i++) {
struct page *subpage = head + i;
@@ -2098,6 +2090,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
int count, mapcount, extra_pins, ret;
bool mlocked;
unsigned long flags;
+ pgoff_t end;
VM_BUG_ON_PAGE(is_huge_zero_page(page), page);
VM_BUG_ON_PAGE(!PageLocked(page), page);
@@ -2119,6 +2112,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
goto out;
}
extra_pins = 0;
+ end = -1;
mapping = NULL;
anon_vma_lock_write(anon_vma);
} else {
@@ -2134,10 +2128,19 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
extra_pins = HPAGE_PMD_NR;
anon_vma = NULL;
i_mmap_lock_read(mapping);
+
+ /*
+ *__split_huge_page() may need to trim off pages beyond EOF:
+ * but on 32-bit, i_size_read() takes an irq-unsafe seqlock,
+ * which cannot be nested inside the page tree lock. So note
+ * end now: i_size itself may be changed at any moment, but
+ * head page lock is good enough to serialize the trimming.
+ */
+ end = DIV_ROUND_UP(i_size_read(mapping->host), PAGE_SIZE);
}
/*
- * Racy check if we can split the page, before freeze_page() will
+ * Racy check if we can split the page, before unmap_page() will
* split PMDs
*/
if (total_mapcount(head) != page_count(head) - extra_pins - 1) {
@@ -2146,7 +2149,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
}
mlocked = PageMlocked(page);
- freeze_page(head);
+ unmap_page(head);
VM_BUG_ON_PAGE(compound_mapcount(head), head);
/* Make sure the page is not on per-CPU pagevec as it takes pin */
@@ -2183,7 +2186,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
if (mapping)
__dec_node_page_state(page, NR_SHMEM_THPS);
spin_unlock(&pgdata->split_queue_lock);
- __split_huge_page(page, list, flags);
+ __split_huge_page(page, list, end, flags);
ret = 0;
} else {
if (IS_ENABLED(CONFIG_DEBUG_VM) && mapcount) {
@@ -2198,7 +2201,7 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
fail: if (mapping)
spin_unlock(&mapping->tree_lock);
spin_unlock_irqrestore(zone_lru_lock(page_zone(head)), flags);
- unfreeze_page(head);
+ remap_page(head);
ret = -EBUSY;
}
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index 6ff65c4..3e50fcf 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -2171,6 +2171,7 @@ static void __init gather_bootmem_prealloc(void)
*/
if (hstate_is_gigantic(h))
adjust_managed_page_count(page, 1 << h->order);
+ cond_resched();
}
}
@@ -3219,7 +3220,7 @@ static int is_hugetlb_entry_hwpoisoned(pte_t pte)
int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
struct vm_area_struct *vma)
{
- pte_t *src_pte, *dst_pte, entry;
+ pte_t *src_pte, *dst_pte, entry, dst_entry;
struct page *ptepage;
unsigned long addr;
int cow;
@@ -3247,15 +3248,30 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
break;
}
- /* If the pagetables are shared don't copy or take references */
- if (dst_pte == src_pte)
+ /*
+ * If the pagetables are shared don't copy or take references.
+ * dst_pte == src_pte is the common case of src/dest sharing.
+ *
+ * However, src could have 'unshared' and dst shares with
+ * another vma. If dst_pte !none, this implies sharing.
+ * Check here before taking page table lock, and once again
+ * after taking the lock below.
+ */
+ dst_entry = huge_ptep_get(dst_pte);
+ if ((dst_pte == src_pte) || !huge_pte_none(dst_entry))
continue;
dst_ptl = huge_pte_lock(h, dst, dst_pte);
src_ptl = huge_pte_lockptr(h, src, src_pte);
spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
entry = huge_ptep_get(src_pte);
- if (huge_pte_none(entry)) { /* skip none entry */
+ dst_entry = huge_ptep_get(dst_pte);
+ if (huge_pte_none(entry) || !huge_pte_none(dst_entry)) {
+ /*
+ * Skip if src entry none. Also, skip in the
+ * unlikely case dst entry !none as this implies
+ * sharing with another vma.
+ */
;
} else if (unlikely(is_hugetlb_entry_migration(entry) ||
is_hugetlb_entry_hwpoisoned(entry))) {
@@ -3644,6 +3660,12 @@ int huge_add_to_page_cache(struct page *page, struct address_space *mapping,
return err;
ClearPagePrivate(page);
+ /*
+ * set page dirty so that it will not be removed from cache/file
+ * by non-hugetlbfs specific code paths.
+ */
+ set_page_dirty(page);
+
spin_lock(&inode->i_lock);
inode->i_blocks += blocks_per_huge_page(h);
spin_unlock(&inode->i_lock);
@@ -4148,6 +4170,12 @@ int hugetlb_reserve_pages(struct inode *inode,
struct resv_map *resv_map;
long gbl_reserve;
+ /* This should never happen */
+ if (from > to) {
+ VM_WARN(1, "%s called with a negative range\n", __func__);
+ return -EINVAL;
+ }
+
/*
* Only apply hugepage reservation if asked. At fault time, an
* attempt will be made for VM_NORESERVE to allocate a page
@@ -4237,7 +4265,9 @@ int hugetlb_reserve_pages(struct inode *inode,
return 0;
out_err:
if (!vma || vma->vm_flags & VM_MAYSHARE)
- region_abort(resv_map, from, to);
+ /* Don't call region_abort if region_chg failed */
+ if (chg >= 0)
+ region_abort(resv_map, from, to);
if (vma && is_vma_resv_set(vma, HPAGE_RESV_OWNER))
kref_put(&resv_map->refs, resv_map_release);
return ret;
@@ -4311,13 +4341,41 @@ static bool vma_shareable(struct vm_area_struct *vma, unsigned long addr)
/*
* check on proper vm_flags and page table alignment
*/
- if (vma->vm_flags & VM_MAYSHARE &&
- vma->vm_start <= base && end <= vma->vm_end)
+ if (vma->vm_flags & VM_MAYSHARE && range_in_vma(vma, base, end))
return true;
return false;
}
/*
+ * Determine if start,end range within vma could be mapped by shared pmd.
+ * If yes, adjust start and end to cover range associated with possible
+ * shared pmd mappings.
+ */
+void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
+ unsigned long *start, unsigned long *end)
+{
+ unsigned long check_addr = *start;
+
+ if (!(vma->vm_flags & VM_MAYSHARE))
+ return;
+
+ for (check_addr = *start; check_addr < *end; check_addr += PUD_SIZE) {
+ unsigned long a_start = check_addr & PUD_MASK;
+ unsigned long a_end = a_start + PUD_SIZE;
+
+ /*
+ * If sharing is possible, adjust start/end if necessary.
+ */
+ if (range_in_vma(vma, a_start, a_end)) {
+ if (a_start < *start)
+ *start = a_start;
+ if (a_end > *end)
+ *end = a_end;
+ }
+ }
+}
+
+/*
* Search for a shareable pmd page for hugetlb. In any case calls pmd_alloc()
* and returns the corresponding pte. While this is not necessary for the
* !shared pmd case because we can allocate the pmd later as well, it makes the
@@ -4413,6 +4471,11 @@ int huge_pmd_unshare(struct mm_struct *mm, unsigned long *addr, pte_t *ptep)
{
return 0;
}
+
+void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma,
+ unsigned long *start, unsigned long *end)
+{
+}
#define want_pmd_share() (0)
#endif /* CONFIG_ARCH_WANT_HUGE_PMD_SHARE */
diff --git a/mm/internal.h b/mm/internal.h
index 6aa1c51..efcd82f 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -36,6 +36,8 @@
/* Do not use these with a slab allocator */
#define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK)
+void page_writeback_init(void);
+
int do_swap_page(struct fault_env *fe, pte_t orig_pte);
void free_pgtables(struct mmu_gather *tlb, struct vm_area_struct *start_vma,
diff --git a/mm/kasan/kasan.c b/mm/kasan/kasan.c
index ab47d93..627c699 100644
--- a/mm/kasan/kasan.c
+++ b/mm/kasan/kasan.c
@@ -671,12 +671,13 @@ void kasan_kfree_large(const void *ptr)
int kasan_module_alloc(void *addr, size_t size)
{
void *ret;
+ size_t scaled_size;
size_t shadow_size;
unsigned long shadow_start;
shadow_start = (unsigned long)kasan_mem_to_shadow(addr);
- shadow_size = round_up(size >> KASAN_SHADOW_SCALE_SHIFT,
- PAGE_SIZE);
+ scaled_size = (size + KASAN_SHADOW_MASK) >> KASAN_SHADOW_SCALE_SHIFT;
+ shadow_size = round_up(scaled_size, PAGE_SIZE);
if (WARN_ON(!PAGE_ALIGNED(shadow_start)))
return -EINVAL;
@@ -860,5 +861,5 @@ static int __init kasan_memhotplug_init(void)
return 0;
}
-module_init(kasan_memhotplug_init);
+core_initcall(kasan_memhotplug_init);
#endif
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 5f2b9eb..707e3f6 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -176,7 +176,9 @@ static void kasan_end_report(unsigned long *flags)
pr_err("==================================================================\n");
add_taint(TAINT_BAD_PAGE, LOCKDEP_NOW_UNRELIABLE);
spin_unlock_irqrestore(&report_lock, *flags);
+#ifndef CONFIG_KASAN_PANIC_ON_WARN
if (panic_on_warn)
+#endif
panic("panic_on_warn set ...\n");
kasan_enable_current();
}
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index 898eb26..e0cfc3a 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -963,7 +963,9 @@ static void collapse_huge_page(struct mm_struct *mm,
goto out_nolock;
}
- if (unlikely(mem_cgroup_try_charge(new_page, mm, gfp, &memcg, true))) {
+ /* Do not oom kill for khugepaged charges */
+ if (unlikely(mem_cgroup_try_charge(new_page, mm, gfp | __GFP_NORETRY,
+ &memcg, true))) {
result = SCAN_CGROUP_CHARGE_FAIL;
goto out_nolock;
}
@@ -1284,7 +1286,7 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)
* collapse_shmem - collapse small tmpfs/shmem pages into huge one.
*
* Basic scheme is simple, details are more complex:
- * - allocate and freeze a new huge page;
+ * - allocate and lock a new huge page;
* - scan over radix tree replacing old pages the new one
* + swap in pages if necessary;
* + fill in gaps;
@@ -1292,11 +1294,11 @@ static void retract_page_tables(struct address_space *mapping, pgoff_t pgoff)
* - if replacing succeed:
* + copy data over;
* + free old pages;
- * + unfreeze huge page;
+ * + unlock huge page;
* - if replacing failed;
* + put all pages back and unfreeze them;
* + restore gaps in the radix-tree;
- * + free huge page;
+ * + unlock and free huge page;
*/
static void collapse_shmem(struct mm_struct *mm,
struct address_space *mapping, pgoff_t start,
@@ -1323,23 +1325,22 @@ static void collapse_shmem(struct mm_struct *mm,
goto out;
}
- if (unlikely(mem_cgroup_try_charge(new_page, mm, gfp, &memcg, true))) {
+ /* Do not oom kill for khugepaged charges */
+ if (unlikely(mem_cgroup_try_charge(new_page, mm, gfp | __GFP_NORETRY,
+ &memcg, true))) {
result = SCAN_CGROUP_CHARGE_FAIL;
goto out;
}
+ __SetPageLocked(new_page);
+ __SetPageSwapBacked(new_page);
new_page->index = start;
new_page->mapping = mapping;
- __SetPageSwapBacked(new_page);
- __SetPageLocked(new_page);
- BUG_ON(!page_ref_freeze(new_page, 1));
-
/*
- * At this point the new_page is 'frozen' (page_count() is zero), locked
- * and not up-to-date. It's safe to insert it into radix tree, because
- * nobody would be able to map it or use it in other way until we
- * unfreeze it.
+ * At this point the new_page is locked and not up-to-date.
+ * It's safe to insert it into the page cache, because nobody would
+ * be able to map it or use it in another way until we unlock it.
*/
index = start;
@@ -1348,18 +1349,28 @@ static void collapse_shmem(struct mm_struct *mm,
int n = min(iter.index, end) - index;
/*
+ * Stop if extent has been hole-punched, and is now completely
+ * empty (the more obvious i_size_read() check would take an
+ * irq-unsafe seqlock on 32-bit).
+ */
+ if (n >= HPAGE_PMD_NR) {
+ result = SCAN_TRUNCATED;
+ goto tree_locked;
+ }
+
+ /*
* Handle holes in the radix tree: charge it from shmem and
* insert relevant subpage of new_page into the radix-tree.
*/
if (n && !shmem_charge(mapping->host, n)) {
result = SCAN_FAIL;
- break;
+ goto tree_locked;
}
- nr_none += n;
for (; index < min(iter.index, end); index++) {
radix_tree_insert(&mapping->page_tree, index,
new_page + (index % HPAGE_PMD_NR));
}
+ nr_none += n;
/* We are done. */
if (index >= end)
@@ -1375,12 +1386,12 @@ static void collapse_shmem(struct mm_struct *mm,
result = SCAN_FAIL;
goto tree_unlocked;
}
- spin_lock_irq(&mapping->tree_lock);
} else if (trylock_page(page)) {
get_page(page);
+ spin_unlock_irq(&mapping->tree_lock);
} else {
result = SCAN_PAGE_LOCK;
- break;
+ goto tree_locked;
}
/*
@@ -1389,17 +1400,24 @@ static void collapse_shmem(struct mm_struct *mm,
*/
VM_BUG_ON_PAGE(!PageLocked(page), page);
VM_BUG_ON_PAGE(!PageUptodate(page), page);
- VM_BUG_ON_PAGE(PageTransCompound(page), page);
+
+ /*
+ * If file was truncated then extended, or hole-punched, before
+ * we locked the first page, then a THP might be there already.
+ */
+ if (PageTransCompound(page)) {
+ result = SCAN_PAGE_COMPOUND;
+ goto out_unlock;
+ }
if (page_mapping(page) != mapping) {
result = SCAN_TRUNCATED;
goto out_unlock;
}
- spin_unlock_irq(&mapping->tree_lock);
if (isolate_lru_page(page)) {
result = SCAN_DEL_PAGE_LRU;
- goto out_isolate_failed;
+ goto out_unlock;
}
if (page_mapped(page))
@@ -1421,7 +1439,9 @@ static void collapse_shmem(struct mm_struct *mm,
*/
if (!page_ref_freeze(page, 3)) {
result = SCAN_PAGE_COUNT;
- goto out_lru;
+ spin_unlock_irq(&mapping->tree_lock);
+ putback_lru_page(page);
+ goto out_unlock;
}
/*
@@ -1437,17 +1457,10 @@ static void collapse_shmem(struct mm_struct *mm,
slot = radix_tree_iter_next(&iter);
index++;
continue;
-out_lru:
- spin_unlock_irq(&mapping->tree_lock);
- putback_lru_page(page);
-out_isolate_failed:
- unlock_page(page);
- put_page(page);
- goto tree_unlocked;
out_unlock:
unlock_page(page);
put_page(page);
- break;
+ goto tree_unlocked;
}
/*
@@ -1455,14 +1468,18 @@ static void collapse_shmem(struct mm_struct *mm,
* This code only triggers if there's nothing in radix tree
* beyond 'end'.
*/
- if (result == SCAN_SUCCEED && index < end) {
+ if (index < end) {
int n = end - index;
+ /* Stop if extent has been truncated, and is now empty */
+ if (n >= HPAGE_PMD_NR) {
+ result = SCAN_TRUNCATED;
+ goto tree_locked;
+ }
if (!shmem_charge(mapping->host, n)) {
result = SCAN_FAIL;
goto tree_locked;
}
-
for (; index < end; index++) {
radix_tree_insert(&mapping->page_tree, index,
new_page + (index % HPAGE_PMD_NR));
@@ -1470,57 +1487,62 @@ static void collapse_shmem(struct mm_struct *mm,
nr_none += n;
}
+ __inc_node_page_state(new_page, NR_SHMEM_THPS);
+ if (nr_none) {
+ struct zone *zone = page_zone(new_page);
+
+ __mod_node_page_state(zone->zone_pgdat, NR_FILE_PAGES, nr_none);
+ __mod_node_page_state(zone->zone_pgdat, NR_SHMEM, nr_none);
+ }
+
tree_locked:
spin_unlock_irq(&mapping->tree_lock);
tree_unlocked:
if (result == SCAN_SUCCEED) {
- unsigned long flags;
- struct zone *zone = page_zone(new_page);
-
/*
* Replacing old pages with new one has succeed, now we need to
* copy the content and free old pages.
*/
+ index = start;
list_for_each_entry_safe(page, tmp, &pagelist, lru) {
+ while (index < page->index) {
+ clear_highpage(new_page + (index % HPAGE_PMD_NR));
+ index++;
+ }
copy_highpage(new_page + (page->index % HPAGE_PMD_NR),
page);
list_del(&page->lru);
- unlock_page(page);
- page_ref_unfreeze(page, 1);
page->mapping = NULL;
+ page_ref_unfreeze(page, 1);
ClearPageActive(page);
ClearPageUnevictable(page);
+ unlock_page(page);
put_page(page);
+ index++;
+ }
+ while (index < end) {
+ clear_highpage(new_page + (index % HPAGE_PMD_NR));
+ index++;
}
- local_irq_save(flags);
- __inc_node_page_state(new_page, NR_SHMEM_THPS);
- if (nr_none) {
- __mod_node_page_state(zone->zone_pgdat, NR_FILE_PAGES, nr_none);
- __mod_node_page_state(zone->zone_pgdat, NR_SHMEM, nr_none);
- }
- local_irq_restore(flags);
-
- /*
- * Remove pte page tables, so we can re-faulti
- * the page as huge.
- */
- retract_page_tables(mapping, start);
-
- /* Everything is ready, let's unfreeze the new_page */
- set_page_dirty(new_page);
SetPageUptodate(new_page);
- page_ref_unfreeze(new_page, HPAGE_PMD_NR);
+ page_ref_add(new_page, HPAGE_PMD_NR - 1);
+ set_page_dirty(new_page);
mem_cgroup_commit_charge(new_page, memcg, false, true);
lru_cache_add_anon(new_page);
- unlock_page(new_page);
+ /*
+ * Remove pte page tables, so we can re-fault the page as huge.
+ */
+ retract_page_tables(mapping, start);
*hpage = NULL;
} else {
/* Something went wrong: rollback changes to the radix-tree */
- shmem_uncharge(mapping->host, nr_none);
spin_lock_irq(&mapping->tree_lock);
+ mapping->nrpages -= nr_none;
+ shmem_uncharge(mapping->host, nr_none);
+
radix_tree_for_each_slot(slot, &mapping->page_tree, &iter,
start) {
if (iter.index >= end)
@@ -1545,20 +1567,19 @@ static void collapse_shmem(struct mm_struct *mm,
page_ref_unfreeze(page, 2);
radix_tree_replace_slot(slot, page);
spin_unlock_irq(&mapping->tree_lock);
- putback_lru_page(page);
unlock_page(page);
+ putback_lru_page(page);
spin_lock_irq(&mapping->tree_lock);
slot = radix_tree_iter_next(&iter);
}
VM_BUG_ON(nr_none);
spin_unlock_irq(&mapping->tree_lock);
- /* Unfreeze new_page, caller would take care about freeing it */
- page_ref_unfreeze(new_page, 1);
mem_cgroup_cancel_charge(new_page, memcg, true);
- unlock_page(new_page);
new_page->mapping = NULL;
}
+
+ unlock_page(new_page);
out:
VM_BUG_ON(!list_empty(&pagelist));
/* TODO: tracepoints */
@@ -1678,10 +1699,14 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages,
spin_unlock(&khugepaged_mm_lock);
mm = mm_slot->mm;
- down_read(&mm->mmap_sem);
- if (unlikely(khugepaged_test_exit(mm)))
- vma = NULL;
- else
+ /*
+ * Don't wait for semaphore (to avoid long wait times). Just move to
+ * the next mm on the list.
+ */
+ vma = NULL;
+ if (unlikely(!down_read_trylock(&mm->mmap_sem)))
+ goto breakouterloop_mmap_sem;
+ if (likely(!khugepaged_test_exit(mm)))
vma = find_vma(mm, khugepaged_scan.address);
progress++;
diff --git a/mm/ksm.c b/mm/ksm.c
index 927aa34..918e434 100644
--- a/mm/ksm.c
+++ b/mm/ksm.c
@@ -1487,8 +1487,22 @@ static void cmp_and_merge_page(struct page *page, struct rmap_item *rmap_item)
tree_rmap_item =
unstable_tree_search_insert(rmap_item, page, &tree_page);
if (tree_rmap_item) {
+ bool split;
+
kpage = try_to_merge_two_pages(rmap_item, page,
tree_rmap_item, tree_page);
+ /*
+ * If both pages we tried to merge belong to the same compound
+ * page, then we actually ended up increasing the reference
+ * count of the same compound page twice, and split_huge_page
+ * failed.
+ * Here we set a flag if that happened, and we use it later to
+ * try split_huge_page again. Since we call put_page right
+ * afterwards, the reference count will be correct and
+ * split_huge_page should succeed.
+ */
+ split = PageTransCompound(page)
+ && compound_head(page) == compound_head(tree_page);
put_page(tree_page);
if (kpage) {
/*
@@ -1513,6 +1527,20 @@ static void cmp_and_merge_page(struct page *page, struct rmap_item *rmap_item)
break_cow(tree_rmap_item);
break_cow(rmap_item);
}
+ } else if (split) {
+ /*
+ * We are here if we tried to merge two pages and
+ * failed because they both belonged to the same
+ * compound page. We will split the page now, but no
+ * merging will take place.
+ * We do not want to add the cost of a full lock; if
+ * the page is locked, it is better to skip it and
+ * perhaps try again later.
+ */
+ if (!trylock_page(page))
+ return;
+ split_huge_page(page);
+ unlock_page(page);
}
}
}
@@ -1964,7 +1992,6 @@ int rmap_walk_ksm(struct page *page, struct rmap_walk_control *rwc)
stable_node = page_stable_node(page);
if (!stable_node)
return ret;
-
again:
hlist_for_each_entry(rmap_item, &stable_node->hlist, hlist) {
struct anon_vma *anon_vma = rmap_item->anon_vma;
diff --git a/mm/madvise.c b/mm/madvise.c
index 59d1aae..bf654aa 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -83,7 +83,7 @@ static long madvise_behavior(struct vm_area_struct *vma,
new_flags |= VM_DONTDUMP;
break;
case MADV_DODUMP:
- if (new_flags & VM_SPECIAL) {
+ if (!is_vm_hugetlb_page(vma) && new_flags & VM_SPECIAL) {
error = -EINVAL;
goto out;
}
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 34eec18..edaa133 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -895,7 +895,7 @@ static void invalidate_reclaim_iterators(struct mem_cgroup *dead_memcg)
int nid;
int i;
- while ((memcg = parent_mem_cgroup(memcg))) {
+ for (; memcg; memcg = parent_mem_cgroup(memcg)) {
for_each_node(nid) {
mz = mem_cgroup_nodeinfo(memcg, nid);
for (i = 0; i <= DEF_PRIORITY; i++) {
@@ -4091,6 +4091,14 @@ static struct cftype mem_cgroup_legacy_files[] = {
static DEFINE_IDR(mem_cgroup_idr);
+static void mem_cgroup_id_remove(struct mem_cgroup *memcg)
+{
+ if (memcg->id.id > 0) {
+ idr_remove(&mem_cgroup_idr, memcg->id.id);
+ memcg->id.id = 0;
+ }
+}
+
static void mem_cgroup_id_get_many(struct mem_cgroup *memcg, unsigned int n)
{
VM_BUG_ON(atomic_read(&memcg->id.ref) <= 0);
@@ -4101,8 +4109,7 @@ static void mem_cgroup_id_put_many(struct mem_cgroup *memcg, unsigned int n)
{
VM_BUG_ON(atomic_read(&memcg->id.ref) < n);
if (atomic_sub_and_test(n, &memcg->id.ref)) {
- idr_remove(&mem_cgroup_idr, memcg->id.id);
- memcg->id.id = 0;
+ mem_cgroup_id_remove(memcg);
/* Memcg ID pins CSS */
css_put(&memcg->css);
@@ -4227,8 +4234,7 @@ static struct mem_cgroup *mem_cgroup_alloc(void)
idr_replace(&mem_cgroup_idr, memcg, memcg->id.id);
return memcg;
fail:
- if (memcg->id.id > 0)
- idr_remove(&mem_cgroup_idr, memcg->id.id);
+ mem_cgroup_id_remove(memcg);
__mem_cgroup_free(memcg);
return NULL;
}
@@ -4287,6 +4293,7 @@ mem_cgroup_css_alloc(struct cgroup_subsys_state *parent_css)
return &memcg->css;
fail:
+ mem_cgroup_id_remove(memcg);
mem_cgroup_free(memcg);
return ERR_PTR(-ENOMEM);
}
diff --git a/mm/memory-failure.c b/mm/memory-failure.c
index 351d94a..851efb0 100644
--- a/mm/memory-failure.c
+++ b/mm/memory-failure.c
@@ -980,7 +980,7 @@ static int hwpoison_user_mappings(struct page *p, unsigned long pfn,
if (kill)
collect_procs(hpage, &tokill, flags & MF_ACTION_REQUIRED);
- ret = try_to_unmap(hpage, ttu, NULL);
+ ret = try_to_unmap(hpage, ttu);
if (ret != SWAP_SUCCESS)
pr_err("Memory failure: %#lx: failed to unmap page (mapcount=%d)\n",
pfn, page_mapcount(hpage));
diff --git a/mm/memory.c b/mm/memory.c
index cc6ab38..9bfdad6 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -65,6 +65,8 @@
#include <linux/userfaultfd_k.h>
#include <linux/dax.h>
+#include <trace/events/kmem.h>
+
#include <asm/io.h>
#include <asm/mmu_context.h>
#include <asm/pgalloc.h>
@@ -134,6 +136,21 @@ static int __init init_zero_pfn(void)
}
core_initcall(init_zero_pfn);
+/*
+ * This threshold is the boundary in the value space, that the counter has to
+ * advance before we trace it. Should be a power of 2. It is to reduce unwanted
+ * trace overhead. The counter is number of pages.
+ */
+#define TRACE_MM_COUNTER_THRESHOLD 128
+
+void mm_trace_rss_stat(int member, long count, long value)
+{
+ long thresh_mask = ~(TRACE_MM_COUNTER_THRESHOLD - 1);
+
+ /* Threshold roll-over, trace it */
+ if ((count & thresh_mask) != ((count - value) & thresh_mask))
+ trace_rss_stat(member, count);
+}
#if defined(SPLIT_RSS_COUNTING)
@@ -373,15 +390,6 @@ void tlb_remove_table(struct mmu_gather *tlb, void *table)
{
struct mmu_table_batch **batch = &tlb->batch;
- /*
- * When there's less then two users of this mm there cannot be a
- * concurrent page-table walk.
- */
- if (atomic_read(&tlb->mm->mm_users) < 2) {
- __tlb_remove_table(table);
- return;
- }
-
if (*batch == NULL) {
*batch = (struct mmu_table_batch *)__get_free_page(GFP_NOWAIT | __GFP_NOWARN);
if (*batch == NULL) {
@@ -1635,6 +1643,9 @@ int vm_insert_pfn_prot(struct vm_area_struct *vma, unsigned long addr,
if (track_pfn_insert(vma, &pgprot, __pfn_to_pfn_t(pfn, PFN_DEV)))
return -EINVAL;
+ if (!pfn_modify_allowed(pfn, pgprot))
+ return -EACCES;
+
ret = insert_pfn(vma, addr, __pfn_to_pfn_t(pfn, PFN_DEV), pgprot);
return ret;
@@ -1653,6 +1664,9 @@ int vm_insert_mixed(struct vm_area_struct *vma, unsigned long addr,
if (track_pfn_insert(vma, &pgprot, pfn))
return -EINVAL;
+ if (!pfn_modify_allowed(pfn_t_to_pfn(pfn), pgprot))
+ return -EACCES;
+
/*
* If we don't have pte special, then we have to use the pfn_valid()
* based VM_MIXEDMAP scheme (see vm_normal_page), and thus we *must*
@@ -1686,6 +1700,7 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd,
{
pte_t *pte;
spinlock_t *ptl;
+ int err = 0;
pte = pte_alloc_map_lock(mm, pmd, addr, &ptl);
if (!pte)
@@ -1693,12 +1708,16 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd,
arch_enter_lazy_mmu_mode();
do {
BUG_ON(!pte_none(*pte));
+ if (!pfn_modify_allowed(pfn, prot)) {
+ err = -EACCES;
+ break;
+ }
set_pte_at(mm, addr, pte, pte_mkspecial(pfn_pte(pfn, prot)));
pfn++;
} while (pte++, addr += PAGE_SIZE, addr != end);
arch_leave_lazy_mmu_mode();
pte_unmap_unlock(pte - 1, ptl);
- return 0;
+ return err;
}
static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,
@@ -1707,6 +1726,7 @@ static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,
{
pmd_t *pmd;
unsigned long next;
+ int err;
pfn -= addr >> PAGE_SHIFT;
pmd = pmd_alloc(mm, pud, addr);
@@ -1715,9 +1735,10 @@ static inline int remap_pmd_range(struct mm_struct *mm, pud_t *pud,
VM_BUG_ON(pmd_trans_huge(*pmd));
do {
next = pmd_addr_end(addr, end);
- if (remap_pte_range(mm, pmd, addr, next,
- pfn + (addr >> PAGE_SHIFT), prot))
- return -ENOMEM;
+ err = remap_pte_range(mm, pmd, addr, next,
+ pfn + (addr >> PAGE_SHIFT), prot);
+ if (err)
+ return err;
} while (pmd++, addr = next, addr != end);
return 0;
}
@@ -1728,6 +1749,7 @@ static inline int remap_pud_range(struct mm_struct *mm, pgd_t *pgd,
{
pud_t *pud;
unsigned long next;
+ int err;
pfn -= addr >> PAGE_SHIFT;
pud = pud_alloc(mm, pgd, addr);
@@ -1735,9 +1757,10 @@ static inline int remap_pud_range(struct mm_struct *mm, pgd_t *pgd,
return -ENOMEM;
do {
next = pud_addr_end(addr, end);
- if (remap_pmd_range(mm, pud, addr, next,
- pfn + (addr >> PAGE_SHIFT), prot))
- return -ENOMEM;
+ err = remap_pmd_range(mm, pud, addr, next,
+ pfn + (addr >> PAGE_SHIFT), prot);
+ if (err)
+ return err;
} while (pud++, addr = next, addr != end);
return 0;
}
@@ -2678,7 +2701,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
unlock:
pte_unmap_unlock(fe->pte, fe->ptl);
out:
- return ret;
+ return ret | VM_FAULT_SWAP;
out_nomap:
mem_cgroup_cancel_charge(page, memcg, false);
pte_unmap_unlock(fe->pte, fe->ptl);
@@ -2690,7 +2713,7 @@ int do_swap_page(struct fault_env *fe, pte_t orig_pte)
unlock_page(swapcache);
put_page(swapcache);
}
- return ret;
+ return ret | VM_FAULT_SWAP;
}
/*
@@ -3840,6 +3863,9 @@ int generic_access_phys(struct vm_area_struct *vma, unsigned long addr,
return -EINVAL;
maddr = ioremap_prot(phys_addr, PAGE_ALIGN(len + offset), prot);
+ if (!maddr)
+ return -ENOMEM;
+
if (write)
memcpy_toio(maddr + offset, buf, len);
else
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 0dab426..2a3c75c 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -34,6 +34,7 @@
#include <linux/memblock.h>
#include <linux/bootmem.h>
#include <linux/compaction.h>
+#include <linux/rmap.h>
#include <asm/tlbflush.h>
@@ -1512,7 +1513,7 @@ int test_pages_in_a_zone(unsigned long start_pfn, unsigned long end_pfn,
while ((i < MAX_ORDER_NR_PAGES) &&
!pfn_valid_within(pfn + i))
i++;
- if (i == MAX_ORDER_NR_PAGES)
+ if (i == MAX_ORDER_NR_PAGES || pfn + i >= end_pfn)
continue;
page = pfn_to_page(pfn + i);
if (zone && page_zone(page) != zone)
@@ -1526,7 +1527,7 @@ int test_pages_in_a_zone(unsigned long start_pfn, unsigned long end_pfn,
if (zone) {
*valid_start = start;
- *valid_end = end;
+ *valid_end = min(end, end_pfn);
return 1;
} else {
return 0;
@@ -1621,6 +1622,21 @@ do_migrate_range(unsigned long start_pfn, unsigned long end_pfn)
continue;
}
+ /*
+ * HWPoison pages have elevated reference counts so the migration would
+ * fail on them. It also doesn't make any sense to migrate them in the
+ * first place. Still try to unmap such a page in case it is still mapped
+ * (e.g. current hwpoison implementation doesn't unmap KSM pages but keep
+ * the unmap as the catch all safety net).
+ */
+ if (PageHWPoison(page)) {
+ if (WARN_ON(PageLRU(page)))
+ isolate_lru_page(page);
+ if (page_mapped(page))
+ try_to_unmap(page, TTU_IGNORE_MLOCK | TTU_IGNORE_ACCESS);
+ continue;
+ }
+
if (!get_page_unless_zero(page))
continue;
/*
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 9547583..d808982 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1265,6 +1265,7 @@ static int get_nodes(nodemask_t *nodes, const unsigned long __user *nmask,
unsigned long maxnode)
{
unsigned long k;
+ unsigned long t;
unsigned long nlongs;
unsigned long endmask;
@@ -1281,13 +1282,19 @@ static int get_nodes(nodemask_t *nodes, const unsigned long __user *nmask,
else
endmask = (1UL << (maxnode % BITS_PER_LONG)) - 1;
- /* When the user specified more nodes than supported just check
- if the non supported part is all zero. */
+ /*
+ * When the user specified more nodes than supported just check
+ * if the non supported part is all zero.
+ *
+ * If maxnode have more longs than MAX_NUMNODES, check
+ * the bits in that area first. And then go through to
+ * check the rest bits which equal or bigger than MAX_NUMNODES.
+ * Otherwise, just check bits [MAX_NUMNODES, maxnode).
+ */
if (nlongs > BITS_TO_LONGS(MAX_NUMNODES)) {
if (nlongs > PAGE_SIZE/sizeof(long))
return -EINVAL;
for (k = BITS_TO_LONGS(MAX_NUMNODES); k < nlongs; k++) {
- unsigned long t;
if (get_user(t, nmask + k))
return -EFAULT;
if (k == nlongs - 1) {
@@ -1300,6 +1307,16 @@ static int get_nodes(nodemask_t *nodes, const unsigned long __user *nmask,
endmask = ~0UL;
}
+ if (maxnode > MAX_NUMNODES && MAX_NUMNODES % BITS_PER_LONG != 0) {
+ unsigned long valid_mask = endmask;
+
+ valid_mask &= ~((1UL << (MAX_NUMNODES % BITS_PER_LONG)) - 1);
+ if (get_user(t, nmask + nlongs - 1))
+ return -EFAULT;
+ if (t & valid_mask)
+ return -EINVAL;
+ }
+
if (copy_from_user(nodes_addr(*nodes), nmask, nlongs*sizeof(unsigned long)))
return -EFAULT;
nodes_addr(*nodes)[nlongs-1] &= endmask;
@@ -1426,10 +1443,14 @@ SYSCALL_DEFINE4(migrate_pages, pid_t, pid, unsigned long, maxnode,
goto out_put;
}
- if (!nodes_subset(*new, node_states[N_MEMORY])) {
- err = -EINVAL;
+ task_nodes = cpuset_mems_allowed(current);
+ nodes_and(*new, *new, task_nodes);
+ if (nodes_empty(*new))
goto out_put;
- }
+
+ nodes_and(*new, *new, node_states[N_MEMORY]);
+ if (nodes_empty(*new))
+ goto out_put;
err = security_task_movememory(task);
if (err)
@@ -2007,8 +2028,36 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma,
nmask = policy_nodemask(gfp, pol);
if (!nmask || node_isset(hpage_node, *nmask)) {
mpol_cond_put(pol);
- page = __alloc_pages_node(hpage_node,
- gfp | __GFP_THISNODE, order);
+ /*
+ * We cannot invoke reclaim if __GFP_THISNODE
+ * is set. Invoking reclaim with
+ * __GFP_THISNODE set, would cause THP
+ * allocations to trigger heavy swapping
+ * despite there may be tons of free memory
+ * (including potentially plenty of THP
+ * already available in the buddy) on all the
+ * other NUMA nodes.
+ *
+ * At most we could invoke compaction when
+ * __GFP_THISNODE is set (but we would need to
+ * refrain from invoking reclaim even if
+ * compaction returned COMPACT_SKIPPED because
+ * there wasn't not enough memory to succeed
+ * compaction). For now just avoid
+ * __GFP_THISNODE instead of limiting the
+ * allocation path to a strict and single
+ * compaction invocation.
+ *
+ * Supposedly if direct reclaim was enabled by
+ * the caller, the app prefers THP regardless
+ * of the node it comes from so this would be
+ * more desiderable behavior than only
+ * providing THP originated from the local
+ * node in such case.
+ */
+ if (!(gfp & __GFP_DIRECT_RECLAIM))
+ gfp |= __GFP_THISNODE;
+ page = __alloc_pages_node(hpage_node, gfp, order);
goto out;
}
}
@@ -2139,6 +2188,9 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b)
case MPOL_INTERLEAVE:
return !!nodes_equal(a->v.nodes, b->v.nodes);
case MPOL_PREFERRED:
+ /* a's ->flags is the same as b's */
+ if (a->flags & MPOL_F_LOCAL)
+ return true;
return a->v.preferred_node == b->v.preferred_node;
default:
BUG();
diff --git a/mm/migrate.c b/mm/migrate.c
index eb1f043..d46ea1f 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -637,6 +637,8 @@ void migrate_page_copy(struct page *newpage, struct page *page)
SetPageActive(newpage);
} else if (TestClearPageUnevictable(page))
SetPageUnevictable(newpage);
+ if (PageWorkingset(page))
+ SetPageWorkingset(newpage);
if (PageChecked(page))
SetPageChecked(newpage);
if (PageMappedToDisk(page))
@@ -1021,7 +1023,7 @@ static int __unmap_and_move(struct page *page, struct page *newpage,
VM_BUG_ON_PAGE(PageAnon(page) && !PageKsm(page) && !anon_vma,
page);
try_to_unmap(page,
- TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS, NULL);
+ TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);
page_was_mapped = 1;
}
@@ -1239,7 +1241,7 @@ static int unmap_and_move_huge_page(new_page_t get_new_page,
if (page_mapped(hpage)) {
try_to_unmap(hpage,
- TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS, NULL);
+ TTU_MIGRATION|TTU_IGNORE_MLOCK|TTU_IGNORE_ACCESS);
page_was_mapped = 1;
}
diff --git a/mm/mm_event.c b/mm/mm_event.c
new file mode 100644
index 0000000..d4be179
--- /dev/null
+++ b/mm/mm_event.c
@@ -0,0 +1,202 @@
+#include <linux/mm.h>
+#include <linux/mm_event.h>
+#include <linux/sched.h>
+#include <linux/vmalloc.h>
+#include <linux/seq_file.h>
+#include <linux/debugfs.h>
+
+#define CREATE_TRACE_POINTS
+#include <trace/events/mm_event.h>
+/* msec */
+static unsigned long period_ms __read_mostly = 500;
+static unsigned long vmstat_period_ms __read_mostly = 1000;
+static unsigned long vmstat_next_period;
+
+static DEFINE_SPINLOCK(vmstat_lock);
+static DEFINE_RWLOCK(period_lock);
+
+void mm_event_task_init(struct task_struct *tsk)
+{
+ memset(tsk->mm_event, 0, sizeof(tsk->mm_event));
+ tsk->next_period = 0;
+}
+
+static void record_vmstat(void)
+{
+ int cpu;
+ struct mm_event_vmstat vmstat;
+
+ if (time_is_after_jiffies(vmstat_next_period))
+ return;
+
+ /* Need double check under the lock */
+ spin_lock(&vmstat_lock);
+ if (time_is_after_jiffies(vmstat_next_period)) {
+ spin_unlock(&vmstat_lock);
+ return;
+ }
+ vmstat_next_period = jiffies + msecs_to_jiffies(vmstat_period_ms);
+ spin_unlock(&vmstat_lock);
+
+ memset(&vmstat, 0, sizeof(vmstat));
+ vmstat.free = global_page_state(NR_FREE_PAGES);
+ vmstat.slab = global_page_state(NR_SLAB_RECLAIMABLE) +
+ global_page_state(NR_SLAB_UNRECLAIMABLE);
+
+ vmstat.file = global_node_page_state(NR_ACTIVE_FILE) +
+ global_node_page_state(NR_INACTIVE_FILE);
+ vmstat.anon = global_node_page_state(NR_ACTIVE_ANON) +
+ global_node_page_state(NR_INACTIVE_ANON);
+ vmstat.ion = global_node_page_state(NR_ION_HEAP) +
+ (global_node_page_state(NR_INDIRECTLY_RECLAIMABLE_BYTES)
+ >> PAGE_SHIFT);
+
+ vmstat.ws_refault = global_node_page_state(WORKINGSET_REFAULT);
+ vmstat.ws_activate = global_node_page_state(WORKINGSET_ACTIVATE);
+ vmstat.mapped = global_node_page_state(NR_FILE_MAPPED);
+
+ /* No want to make lock dependency between vmstat_lock and hotplug */
+ get_online_cpus();
+ for_each_online_cpu(cpu) {
+ struct vm_event_state *this = &per_cpu(vm_event_states, cpu);
+
+ /* sectors to kbytes for PGPGIN/PGPGOUT */
+ vmstat.pgin += this->event[PGPGIN] / 2;
+ vmstat.pgout += this->event[PGPGOUT] / 2;
+ vmstat.swpin += this->event[PSWPIN];
+ vmstat.swpout += this->event[PSWPOUT];
+ vmstat.reclaim_steal += this->event[PGSTEAL_DIRECT] +
+ this->event[PGSTEAL_KSWAPD];
+ vmstat.reclaim_scan += this->event[PGSCAN_DIRECT] +
+ this->event[PGSCAN_KSWAPD];
+ vmstat.compact_scan += this->event[COMPACTFREE_SCANNED] +
+ this->event[COMPACTFREE_SCANNED];
+ }
+ put_online_cpus();
+ trace_mm_event_vmstat_record(&vmstat);
+}
+
+static void record_stat(void)
+{
+ int i;
+ bool need_vmstat = false;
+
+ if (time_is_after_jiffies(current->next_period))
+ return;
+
+ read_lock(&period_lock);
+ current->next_period = jiffies + msecs_to_jiffies(period_ms);
+ read_unlock(&period_lock);
+
+ for (i = 0; i < MM_TYPE_NUM; i++) {
+ if (current->mm_event[i].count == 0)
+ continue;
+ if (i == MM_COMPACTION || i == MM_RECLAIM)
+ need_vmstat = true;
+ trace_mm_event_record(i, ¤t->mm_event[i]);
+ memset(¤t->mm_event[i], 0,
+ sizeof(struct mm_event_task));
+ }
+
+ if (need_vmstat)
+ record_vmstat();
+}
+
+void mm_event_start(ktime_t *time)
+{
+ *time = ktime_get();
+}
+
+void mm_event_end(enum mm_event_type event, ktime_t start)
+{
+ s64 elapsed = ktime_us_delta(ktime_get(), start);
+
+ current->mm_event[event].count++;
+ current->mm_event[event].accm_lat += elapsed;
+ if (elapsed > current->mm_event[event].max_lat)
+ current->mm_event[event].max_lat = elapsed;
+ record_stat();
+}
+
+void mm_event_count(enum mm_event_type event, int count)
+{
+ current->mm_event[event].count += count;
+ record_stat();
+}
+
+static struct dentry *mm_event_root;
+
+static int period_ms_set(void *data, u64 val)
+{
+ if (val < 1 || val > ULONG_MAX)
+ return -EINVAL;
+
+ write_lock(&period_lock);
+ period_ms = (unsigned long)val;
+ write_unlock(&period_lock);
+ return 0;
+}
+
+static int period_ms_get(void *data, u64 *val)
+{
+ read_lock(&period_lock);
+ *val = period_ms;
+ read_unlock(&period_lock);
+
+ return 0;
+}
+
+static int vmstat_period_ms_set(void *data, u64 val)
+{
+ if (val < 1 || val > ULONG_MAX)
+ return -EINVAL;
+
+ spin_lock(&vmstat_lock);
+ vmstat_period_ms = (unsigned long)val;
+ spin_unlock(&vmstat_lock);
+ return 0;
+}
+
+static int vmstat_period_ms_get(void *data, u64 *val)
+{
+ spin_lock(&vmstat_lock);
+ *val = vmstat_period_ms;
+ spin_unlock(&vmstat_lock);
+ return 0;
+}
+
+DEFINE_SIMPLE_ATTRIBUTE(period_ms_operations, period_ms_get,
+ period_ms_set, "%llu\n");
+DEFINE_SIMPLE_ATTRIBUTE(vmstat_period_ms_operations, vmstat_period_ms_get,
+ vmstat_period_ms_set, "%llu\n");
+
+static int __init mm_event_init(void)
+{
+ struct dentry *entry;
+
+ mm_event_root = debugfs_create_dir("mm_event", NULL);
+ if (!mm_event_root) {
+ pr_warn("debugfs dir <mm_event> creation failed\n");
+ return PTR_ERR(mm_event_root);
+ }
+
+ entry = debugfs_create_file("period_ms", 0644,
+ mm_event_root, NULL, &period_ms_operations);
+
+ if (IS_ERR(entry)) {
+ pr_warn("debugfs file mm_event_task creation failed\n");
+ debugfs_remove_recursive(mm_event_root);
+ return PTR_ERR(entry);
+ }
+
+ entry = debugfs_create_file("vmstat_period_ms", 0644,
+ mm_event_root, NULL, &vmstat_period_ms_operations);
+ if (IS_ERR(entry)) {
+ pr_warn("debugfs file vmstat_mm_event_task creation failed\n");
+ debugfs_remove_recursive(mm_event_root);
+ return PTR_ERR(entry);
+ }
+
+ return 0;
+}
+subsys_initcall(mm_event_init);
diff --git a/mm/mmap.c b/mm/mmap.c
index 5df92da..9744ab4 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1322,6 +1322,35 @@ static inline int mlock_future_check(struct mm_struct *mm,
return 0;
}
+static inline u64 file_mmap_size_max(struct file *file, struct inode *inode)
+{
+ if (S_ISREG(inode->i_mode))
+ return MAX_LFS_FILESIZE;
+
+ if (S_ISBLK(inode->i_mode))
+ return MAX_LFS_FILESIZE;
+
+ /* Special "we do even unsigned file positions" case */
+ if (file->f_mode & FMODE_UNSIGNED_OFFSET)
+ return 0;
+
+ /* Yes, random drivers might want more. But I'm tired of buggy drivers */
+ return ULONG_MAX;
+}
+
+static inline bool file_mmap_ok(struct file *file, struct inode *inode,
+ unsigned long pgoff, unsigned long len)
+{
+ u64 maxsize = file_mmap_size_max(file, inode);
+
+ if (maxsize && len > maxsize)
+ return false;
+ maxsize -= len;
+ if (pgoff > maxsize >> PAGE_SHIFT)
+ return false;
+ return true;
+}
+
/*
* The caller must hold down_write(¤t->mm->mmap_sem).
*/
@@ -1394,6 +1423,9 @@ unsigned long do_mmap(struct file *file, unsigned long addr,
if (file) {
struct inode *inode = file_inode(file);
+ if (!file_mmap_ok(file, inode, pgoff, len))
+ return -EOVERFLOW;
+
switch (flags & MAP_TYPE) {
case MAP_SHARED:
if ((prot&PROT_WRITE) && !(file->f_mode&FMODE_WRITE))
@@ -2855,21 +2887,15 @@ static inline void verify_mm_writelocked(struct mm_struct *mm)
* anonymous maps. eventually we may be able to do some
* brk-specific accounting here.
*/
-static int do_brk(unsigned long addr, unsigned long request)
+static int do_brk(unsigned long addr, unsigned long len)
{
struct mm_struct *mm = current->mm;
struct vm_area_struct *vma, *prev;
- unsigned long flags, len;
+ unsigned long flags;
struct rb_node **rb_link, *rb_parent;
pgoff_t pgoff = addr >> PAGE_SHIFT;
int error;
- len = PAGE_ALIGN(request);
- if (len < request)
- return -ENOMEM;
- if (!len)
- return 0;
-
flags = VM_DATA_DEFAULT_FLAGS | VM_ACCOUNT | mm->def_flags;
error = get_unmapped_area(NULL, addr, len, 0, MAP_FIXED);
@@ -2938,12 +2964,19 @@ static int do_brk(unsigned long addr, unsigned long request)
return 0;
}
-int vm_brk(unsigned long addr, unsigned long len)
+int vm_brk(unsigned long addr, unsigned long request)
{
struct mm_struct *mm = current->mm;
+ unsigned long len;
int ret;
bool populate;
+ len = PAGE_ALIGN(request);
+ if (len < request)
+ return -ENOMEM;
+ if (!len)
+ return 0;
+
if (down_write_killable(&mm->mmap_sem))
return -EINTR;
diff --git a/mm/mprotect.c b/mm/mprotect.c
index 1f2c969..5471f35 100644
--- a/mm/mprotect.c
+++ b/mm/mprotect.c
@@ -260,6 +260,42 @@ unsigned long change_protection(struct vm_area_struct *vma, unsigned long start,
return pages;
}
+static int prot_none_pte_entry(pte_t *pte, unsigned long addr,
+ unsigned long next, struct mm_walk *walk)
+{
+ return pfn_modify_allowed(pte_pfn(*pte), *(pgprot_t *)(walk->private)) ?
+ 0 : -EACCES;
+}
+
+static int prot_none_hugetlb_entry(pte_t *pte, unsigned long hmask,
+ unsigned long addr, unsigned long next,
+ struct mm_walk *walk)
+{
+ return pfn_modify_allowed(pte_pfn(*pte), *(pgprot_t *)(walk->private)) ?
+ 0 : -EACCES;
+}
+
+static int prot_none_test(unsigned long addr, unsigned long next,
+ struct mm_walk *walk)
+{
+ return 0;
+}
+
+static int prot_none_walk(struct vm_area_struct *vma, unsigned long start,
+ unsigned long end, unsigned long newflags)
+{
+ pgprot_t new_pgprot = vm_get_page_prot(newflags);
+ struct mm_walk prot_none_walk = {
+ .pte_entry = prot_none_pte_entry,
+ .hugetlb_entry = prot_none_hugetlb_entry,
+ .test_walk = prot_none_test,
+ .mm = current->mm,
+ .private = &new_pgprot,
+ };
+
+ return walk_page_range(start, end, &prot_none_walk);
+}
+
int
mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
unsigned long start, unsigned long end, unsigned long newflags)
@@ -278,6 +314,19 @@ mprotect_fixup(struct vm_area_struct *vma, struct vm_area_struct **pprev,
}
/*
+ * Do PROT_NONE PFN permission checks here when we can still
+ * bail out without undoing a lot of state. This is a rather
+ * uncommon case, so doesn't need to be very optimized.
+ */
+ if (arch_has_pfn_modify_check() &&
+ (vma->vm_flags & (VM_PFNMAP|VM_MIXEDMAP)) &&
+ (newflags & (VM_READ|VM_WRITE|VM_EXEC)) == 0) {
+ error = prot_none_walk(vma, start, end, newflags);
+ if (error)
+ return error;
+ }
+
+ /*
* If we make a private mapping writable we increase our commit;
* but (without finer accounting) cannot reduce our commit if we
* make it unwritable again. hugetlb mapping were accounted for
diff --git a/mm/mremap.c b/mm/mremap.c
index 1597671..9e60359 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -104,7 +104,7 @@ static pte_t move_soft_dirty_pte(pte_t pte)
static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
unsigned long old_addr, unsigned long old_end,
struct vm_area_struct *new_vma, pmd_t *new_pmd,
- unsigned long new_addr, bool need_rmap_locks, bool *need_flush)
+ unsigned long new_addr, bool need_rmap_locks)
{
struct mm_struct *mm = vma->vm_mm;
pte_t *old_pte, *new_pte, pte;
@@ -152,15 +152,17 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
pte = ptep_get_and_clear(mm, old_addr, old_pte);
/*
- * If we are remapping a dirty PTE, make sure
+ * If we are remapping a valid PTE, make sure
* to flush TLB before we drop the PTL for the
- * old PTE or we may race with page_mkclean().
+ * PTE.
*
- * This check has to be done after we removed the
- * old PTE from page tables or another thread may
- * dirty it after the check and before the removal.
+ * NOTE! Both old and new PTL matter: the old one
+ * for racing with page_mkclean(), the new one to
+ * make sure the physical page stays valid until
+ * the TLB entry for the old mapping has been
+ * flushed.
*/
- if (pte_present(pte) && pte_dirty(pte))
+ if (pte_present(pte))
force_flush = true;
pte = move_pte(pte, new_vma->vm_page_prot, old_addr, new_addr);
pte = move_soft_dirty_pte(pte);
@@ -168,13 +170,11 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
}
arch_leave_lazy_mmu_mode();
+ if (force_flush)
+ flush_tlb_range(vma, old_end - len, old_end);
if (new_ptl != old_ptl)
spin_unlock(new_ptl);
pte_unmap(new_pte - 1);
- if (force_flush)
- flush_tlb_range(vma, old_end - len, old_end);
- else
- *need_flush = true;
pte_unmap_unlock(old_pte - 1, old_ptl);
if (need_rmap_locks)
drop_rmap_locks(vma);
@@ -189,7 +189,6 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
{
unsigned long extent, next, old_end;
pmd_t *old_pmd, *new_pmd;
- bool need_flush = false;
unsigned long mmun_start; /* For mmu_notifiers */
unsigned long mmun_end; /* For mmu_notifiers */
@@ -220,8 +219,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
if (need_rmap_locks)
take_rmap_locks(vma);
moved = move_huge_pmd(vma, old_addr, new_addr,
- old_end, old_pmd, new_pmd,
- &need_flush);
+ old_end, old_pmd, new_pmd);
if (need_rmap_locks)
drop_rmap_locks(vma);
if (moved)
@@ -239,10 +237,8 @@ unsigned long move_page_tables(struct vm_area_struct *vma,
if (extent > LATENCY_LIMIT)
extent = LATENCY_LIMIT;
move_ptes(vma, old_pmd, old_addr, old_addr + extent, new_vma,
- new_pmd, new_addr, need_rmap_locks, &need_flush);
+ new_pmd, new_addr, need_rmap_locks);
}
- if (need_flush)
- flush_tlb_range(vma, old_end-len, old_addr);
mmu_notifier_invalidate_range_end(vma->vm_mm, mmun_start, mmun_end);
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index ca18dc0..756c386 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -2185,30 +2185,14 @@ int write_cache_pages(struct address_space *mapping,
while (!done && (index <= end)) {
int i;
- nr_pages = pagevec_lookup_tag(&pvec, mapping, &index, tag,
- min(end - index, (pgoff_t)PAGEVEC_SIZE-1) + 1);
+ nr_pages = pagevec_lookup_range_tag(&pvec, mapping, &index, end,
+ tag);
if (nr_pages == 0)
break;
for (i = 0; i < nr_pages; i++) {
struct page *page = pvec.pages[i];
- /*
- * At this point, the page may be truncated or
- * invalidated (changing page->mapping to NULL), or
- * even swizzled back from swapper_space to tmpfs file
- * mapping. However, page->index will not change
- * because we have a reference on the page.
- */
- if (page->index > end) {
- /*
- * can't be range_cyclic (1st pass) because
- * end == -1 in that case.
- */
- done = 1;
- break;
- }
-
done_index = page->index;
lock_page(page);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index b74f30e..74e4e1a 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -65,6 +65,7 @@
#include <linux/kthread.h>
#include <linux/memcontrol.h>
#include <linux/show_mem_notifier.h>
+#include <linux/psi.h>
#include <asm/sections.h>
#include <asm/tlbflush.h>
@@ -3247,15 +3248,20 @@ __alloc_pages_direct_compact(gfp_t gfp_mask, unsigned int order,
enum compact_priority prio, enum compact_result *compact_result)
{
struct page *page;
+ unsigned long pflags;
unsigned int noreclaim_flag = current->flags & PF_MEMALLOC;
if (!order)
return NULL;
+ psi_memstall_enter(&pflags);
current->flags |= PF_MEMALLOC;
+
*compact_result = try_to_compact_pages(gfp_mask, order, alloc_flags, ac,
prio);
+
current->flags = (current->flags & ~PF_MEMALLOC) | noreclaim_flag;
+ psi_memstall_leave(&pflags);
if (*compact_result <= COMPACT_INACTIVE)
return NULL;
@@ -3435,11 +3441,13 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,
{
struct reclaim_state reclaim_state;
int progress;
+ unsigned long pflags;
cond_resched();
/* We now go into synchronous reclaim */
cpuset_memory_pressure_bump();
+ psi_memstall_enter(&pflags);
current->flags |= PF_MEMALLOC;
lockdep_set_current_reclaim_state(gfp_mask);
reclaim_state.reclaimed_slab = 0;
@@ -3451,6 +3459,7 @@ __perform_reclaim(gfp_t gfp_mask, unsigned int order,
current->reclaim_state = NULL;
lockdep_clear_current_reclaim_state();
current->flags &= ~PF_MEMALLOC;
+ psi_memstall_leave(&pflags);
cond_resched();
@@ -3666,8 +3675,6 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
enum compact_result compact_result;
int compaction_retries;
int no_progress_loops;
- unsigned long alloc_start = jiffies;
- unsigned int stall_timeout = 10 * HZ;
unsigned int cpuset_mems_cookie;
/*
@@ -3778,7 +3785,6 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
* orientated.
*/
if (!(alloc_flags & ALLOC_CPUSET) || (alloc_flags & ALLOC_NO_WATERMARKS)) {
- ac->zonelist = node_zonelist(numa_node_id(), gfp_mask);
ac->preferred_zoneref = first_zones_zonelist(ac->zonelist,
ac->high_zoneidx, ac->nodemask);
}
@@ -3841,14 +3847,6 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
if (order > PAGE_ALLOC_COSTLY_ORDER && !(gfp_mask & __GFP_REPEAT))
goto nopage;
- /* Make sure we know about allocations which stall for too long */
- if (time_after(jiffies, alloc_start + stall_timeout)) {
- warn_alloc(gfp_mask,
- "page allocation stalls for %ums, order:%u",
- jiffies_to_msecs(jiffies-alloc_start), order);
- stall_timeout += 10 * HZ;
- }
-
if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,
did_some_progress > 0, &no_progress_loops))
goto retry;
@@ -4321,6 +4319,13 @@ long si_mem_available(void)
available += global_page_state(NR_SLAB_RECLAIMABLE) -
min(global_page_state(NR_SLAB_RECLAIMABLE) / 2, wmark_low);
+ /*
+ * Part of the kernel memory, which can be released under memory
+ * pressure.
+ */
+ available += global_node_page_state(NR_INDIRECTLY_RECLAIMABLE_BYTES) >>
+ PAGE_SHIFT;
+
if (available < 0)
available = 0;
return available;
@@ -4451,7 +4456,8 @@ void show_free_areas(unsigned int filter)
" unevictable:%lu dirty:%lu writeback:%lu unstable:%lu\n"
" slab_reclaimable:%lu slab_unreclaimable:%lu\n"
" mapped:%lu shmem:%lu pagetables:%lu bounce:%lu\n"
- " free:%lu free_pcp:%lu free_cma:%lu\n",
+ " free:%lu free_pcp:%lu free_cma:%lu zspages: %lu\n"
+ " ion_heap:%lu ion_heap_pool:%lu\n",
global_node_page_state(NR_ACTIVE_ANON),
global_node_page_state(NR_INACTIVE_ANON),
global_node_page_state(NR_ISOLATED_ANON),
@@ -4470,7 +4476,15 @@ void show_free_areas(unsigned int filter)
global_page_state(NR_BOUNCE),
global_page_state(NR_FREE_PAGES),
free_pcp,
- global_page_state(NR_FREE_CMA_PAGES));
+ global_page_state(NR_FREE_CMA_PAGES),
+#if IS_ENABLED(CONFIG_ZSMALLOC)
+ global_page_state(NR_ZSPAGES),
+#else
+ 0UL,
+#endif
+ global_node_page_state(NR_ION_HEAP),
+ global_node_page_state(NR_INDIRECTLY_RECLAIMABLE_BYTES)
+ >> PAGE_SHIFT);
for_each_online_pgdat(pgdat) {
printk("Node %d"
@@ -4547,6 +4561,9 @@ void show_free_areas(unsigned int filter)
" slab_reclaimable:%lukB"
" slab_unreclaimable:%lukB"
" kernel_stack:%lukB"
+#ifdef CONFIG_SHADOW_CALL_STACK
+ " shadow_call_stack:%lukB"
+#endif
" pagetables:%lukB"
" bounce:%lukB"
" free_pcp:%lukB"
@@ -4570,6 +4587,9 @@ void show_free_areas(unsigned int filter)
K(zone_page_state(zone, NR_SLAB_RECLAIMABLE)),
K(zone_page_state(zone, NR_SLAB_UNRECLAIMABLE)),
zone_page_state(zone, NR_KERNEL_STACK_KB),
+#ifdef CONFIG_SHADOW_CALL_STACK
+ zone_page_state(zone, NR_KERNEL_SCS_BYTES) / 1024,
+#endif
K(zone_page_state(zone, NR_PAGETABLE)),
K(zone_page_state(zone, NR_BOUNCE)),
K(free_pcp),
diff --git a/mm/percpu.c b/mm/percpu.c
index f014ceb..3794cfc 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -70,6 +70,7 @@
#include <linux/vmalloc.h>
#include <linux/workqueue.h>
#include <linux/kmemleak.h>
+#include <linux/sched.h>
#include <asm/cacheflush.h>
#include <asm/sections.h>
diff --git a/mm/process_reclaim.c b/mm/process_reclaim.c
deleted file mode 100644
index 36516eb..0000000
--- a/mm/process_reclaim.c
+++ /dev/null
@@ -1,253 +0,0 @@
-/*
- * Copyright (c) 2015-2016, The Linux Foundation. All rights reserved.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 and
- * only version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
-#include <linux/module.h>
-#include <linux/kernel.h>
-#include <linux/mm.h>
-#include <linux/swap.h>
-#include <linux/sort.h>
-#include <linux/oom.h>
-#include <linux/sched.h>
-#include <linux/rcupdate.h>
-#include <linux/notifier.h>
-#include <linux/vmpressure.h>
-
-#define CREATE_TRACE_POINTS
-#include <trace/events/process_reclaim.h>
-
-#define MAX_SWAP_TASKS SWAP_CLUSTER_MAX
-
-static void swap_fn(struct work_struct *work);
-DECLARE_WORK(swap_work, swap_fn);
-
-/* User knob to enable/disable process reclaim feature */
-static int enable_process_reclaim;
-module_param_named(enable_process_reclaim, enable_process_reclaim, int, 0644);
-
-/* The max number of pages tried to be reclaimed in a single run */
-int per_swap_size = SWAP_CLUSTER_MAX * 32;
-module_param_named(per_swap_size, per_swap_size, int, 0644);
-
-int reclaim_avg_efficiency;
-module_param_named(reclaim_avg_efficiency, reclaim_avg_efficiency, int, 0444);
-
-/* The vmpressure region where process reclaim operates */
-static unsigned long pressure_min = 50;
-static unsigned long pressure_max = 90;
-module_param_named(pressure_min, pressure_min, ulong, 0644);
-module_param_named(pressure_max, pressure_max, ulong, 0644);
-
-static short min_score_adj = 360;
-module_param_named(min_score_adj, min_score_adj, short, 0644);
-
-/*
- * Scheduling process reclaim workqueue unecessarily
- * when the reclaim efficiency is low does not make
- * sense. We try to detect a drop in efficiency and
- * disable reclaim for a time period. This period and the
- * period for which we monitor a drop in efficiency is
- * defined by swap_eff_win. swap_opt_eff is the optimal
- * efficincy used as theshold for this.
- */
-static int swap_eff_win = 2;
-module_param_named(swap_eff_win, swap_eff_win, int, 0644);
-
-static int swap_opt_eff = 50;
-module_param_named(swap_opt_eff, swap_opt_eff, int, 0644);
-
-static atomic_t skip_reclaim = ATOMIC_INIT(0);
-/* Not atomic since only a single instance of swap_fn run at a time */
-static int monitor_eff;
-
-struct selected_task {
- struct task_struct *p;
- int tasksize;
- short oom_score_adj;
-};
-
-int selected_cmp(const void *a, const void *b)
-{
- const struct selected_task *x = a;
- const struct selected_task *y = b;
- int ret;
-
- ret = x->tasksize < y->tasksize ? -1 : 1;
-
- return ret;
-}
-
-static int test_task_flag(struct task_struct *p, int flag)
-{
- struct task_struct *t = p;
-
- rcu_read_lock();
- for_each_thread(p, t) {
- task_lock(t);
- if (test_tsk_thread_flag(t, flag)) {
- task_unlock(t);
- rcu_read_unlock();
- return 1;
- }
- task_unlock(t);
- }
- rcu_read_unlock();
-
- return 0;
-}
-
-static void swap_fn(struct work_struct *work)
-{
- struct task_struct *tsk;
- struct reclaim_param rp;
-
- /* Pick the best MAX_SWAP_TASKS tasks in terms of anon size */
- struct selected_task selected[MAX_SWAP_TASKS] = {{0, 0, 0},};
- int si = 0;
- int i;
- int tasksize;
- int total_sz = 0;
- int total_scan = 0;
- int total_reclaimed = 0;
- int nr_to_reclaim;
- int efficiency;
-
- rcu_read_lock();
- for_each_process(tsk) {
- struct task_struct *p;
- short oom_score_adj;
-
- if (tsk->flags & PF_KTHREAD)
- continue;
-
- if (test_task_flag(tsk, TIF_MEMDIE))
- continue;
-
- p = find_lock_task_mm(tsk);
- if (!p)
- continue;
-
- oom_score_adj = p->signal->oom_score_adj;
- if (oom_score_adj < min_score_adj) {
- task_unlock(p);
- continue;
- }
-
- tasksize = get_mm_counter(p->mm, MM_ANONPAGES);
- task_unlock(p);
-
- if (tasksize <= 0)
- continue;
-
- if (si == MAX_SWAP_TASKS) {
- sort(&selected[0], MAX_SWAP_TASKS,
- sizeof(struct selected_task),
- &selected_cmp, NULL);
- if (tasksize < selected[0].tasksize)
- continue;
- selected[0].p = p;
- selected[0].oom_score_adj = oom_score_adj;
- selected[0].tasksize = tasksize;
- } else {
- selected[si].p = p;
- selected[si].oom_score_adj = oom_score_adj;
- selected[si].tasksize = tasksize;
- si++;
- }
- }
-
- for (i = 0; i < si; i++)
- total_sz += selected[i].tasksize;
-
- /* Skip reclaim if total size is too less */
- if (total_sz < SWAP_CLUSTER_MAX) {
- rcu_read_unlock();
- return;
- }
-
- for (i = 0; i < si; i++)
- get_task_struct(selected[i].p);
-
- rcu_read_unlock();
-
- while (si--) {
- nr_to_reclaim =
- (selected[si].tasksize * per_swap_size) / total_sz;
- /* scan atleast a page */
- if (!nr_to_reclaim)
- nr_to_reclaim = 1;
-
- rp = reclaim_task_anon(selected[si].p, nr_to_reclaim);
-
- trace_process_reclaim(selected[si].tasksize,
- selected[si].oom_score_adj, rp.nr_scanned,
- rp.nr_reclaimed, per_swap_size, total_sz,
- nr_to_reclaim);
- total_scan += rp.nr_scanned;
- total_reclaimed += rp.nr_reclaimed;
- put_task_struct(selected[si].p);
- }
-
- if (total_scan) {
- efficiency = (total_reclaimed * 100) / total_scan;
-
- if (efficiency < swap_opt_eff) {
- if (++monitor_eff == swap_eff_win) {
- atomic_set(&skip_reclaim, swap_eff_win);
- monitor_eff = 0;
- }
- } else {
- monitor_eff = 0;
- }
-
- reclaim_avg_efficiency =
- (efficiency + reclaim_avg_efficiency) / 2;
- trace_process_reclaim_eff(efficiency, reclaim_avg_efficiency);
- }
-}
-
-static int vmpressure_notifier(struct notifier_block *nb,
- unsigned long action, void *data)
-{
- unsigned long pressure = action;
-
- if (!enable_process_reclaim)
- return 0;
-
- if (!current_is_kswapd())
- return 0;
-
- if (atomic_dec_if_positive(&skip_reclaim) >= 0)
- return 0;
-
- if ((pressure >= pressure_min) && (pressure < pressure_max))
- if (!work_pending(&swap_work))
- queue_work(system_unbound_wq, &swap_work);
- return 0;
-}
-
-static struct notifier_block vmpr_nb = {
- .notifier_call = vmpressure_notifier,
-};
-
-static int __init process_reclaim_init(void)
-{
- vmpressure_notifier_register(&vmpr_nb);
- return 0;
-}
-
-static void __exit process_reclaim_exit(void)
-{
- vmpressure_notifier_unregister(&vmpr_nb);
-}
-
-module_init(process_reclaim_init);
-module_exit(process_reclaim_exit);
diff --git a/mm/readahead.c b/mm/readahead.c
index 8b2a755..cf4e180 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -207,12 +207,21 @@ int __do_page_cache_readahead(struct address_space *mapping, struct file *filp,
* memory at once.
*/
int force_page_cache_readahead(struct address_space *mapping, struct file *filp,
- pgoff_t offset, unsigned long nr_to_read)
+ pgoff_t offset, unsigned long nr_to_read)
{
+ struct backing_dev_info *bdi = inode_to_bdi(mapping->host);
+ struct file_ra_state *ra = &filp->f_ra;
+ unsigned long max_pages;
+
if (unlikely(!mapping->a_ops->readpage && !mapping->a_ops->readpages))
return -EINVAL;
- nr_to_read = min(nr_to_read, inode_to_bdi(mapping->host)->ra_pages);
+ /*
+ * If the request exceeds the readahead window, allow the read to
+ * be up to the optimal hardware IO size
+ */
+ max_pages = max_t(unsigned long, bdi->io_pages, ra->ra_pages);
+ nr_to_read = min(nr_to_read, max_pages);
while (nr_to_read) {
int err;
@@ -371,10 +380,19 @@ ondemand_readahead(struct address_space *mapping,
bool hit_readahead_marker, pgoff_t offset,
unsigned long req_size)
{
- unsigned long max = ra->ra_pages;
+ struct backing_dev_info *bdi = inode_to_bdi(mapping->host);
+ unsigned long max_pages = ra->ra_pages;
+ unsigned long add_pages;
pgoff_t prev_offset;
/*
+ * If the request exceeds the readahead window, allow the read to
+ * be up to the optimal hardware IO size
+ */
+ if (req_size > max_pages && bdi->io_pages > max_pages)
+ max_pages = min(req_size, bdi->io_pages);
+
+ /*
* start of file
*/
if (!offset)
@@ -387,7 +405,7 @@ ondemand_readahead(struct address_space *mapping,
if ((offset == (ra->start + ra->size - ra->async_size) ||
offset == (ra->start + ra->size))) {
ra->start += ra->size;
- ra->size = get_next_ra_size(ra, max);
+ ra->size = get_next_ra_size(ra, max_pages);
ra->async_size = ra->size;
goto readit;
}
@@ -402,16 +420,16 @@ ondemand_readahead(struct address_space *mapping,
pgoff_t start;
rcu_read_lock();
- start = page_cache_next_hole(mapping, offset + 1, max);
+ start = page_cache_next_hole(mapping, offset + 1, max_pages);
rcu_read_unlock();
- if (!start || start - offset > max)
+ if (!start || start - offset > max_pages)
return 0;
ra->start = start;
ra->size = start - offset; /* old async_size */
ra->size += req_size;
- ra->size = get_next_ra_size(ra, max);
+ ra->size = get_next_ra_size(ra, max_pages);
ra->async_size = ra->size;
goto readit;
}
@@ -419,7 +437,7 @@ ondemand_readahead(struct address_space *mapping,
/*
* oversize read
*/
- if (req_size > max)
+ if (req_size > max_pages)
goto initial_readahead;
/*
@@ -435,7 +453,7 @@ ondemand_readahead(struct address_space *mapping,
* Query the page cache and look for the traces(cached history pages)
* that a sequential stream would leave behind.
*/
- if (try_context_readahead(mapping, ra, offset, req_size, max))
+ if (try_context_readahead(mapping, ra, offset, req_size, max_pages))
goto readit;
/*
@@ -446,7 +464,7 @@ ondemand_readahead(struct address_space *mapping,
initial_readahead:
ra->start = offset;
- ra->size = get_init_ra_size(req_size, max);
+ ra->size = get_init_ra_size(req_size, max_pages);
ra->async_size = ra->size > req_size ? ra->size - req_size : ra->size;
readit:
@@ -454,10 +472,17 @@ ondemand_readahead(struct address_space *mapping,
* Will this read hit the readahead marker made by itself?
* If so, trigger the readahead marker hit now, and merge
* the resulted next readahead window into the current one.
+ * Take care of maximum IO pages as above.
*/
if (offset == ra->start && ra->size == ra->async_size) {
- ra->async_size = get_next_ra_size(ra, max);
- ra->size += ra->async_size;
+ add_pages = get_next_ra_size(ra, max_pages);
+ if (ra->size + add_pages <= max_pages) {
+ ra->async_size = add_pages;
+ ra->size += add_pages;
+ } else {
+ ra->size = max_pages;
+ ra->async_size = max_pages >> 1;
+ }
}
return ra_submit(ra, mapping, filp);
diff --git a/mm/rmap.c b/mm/rmap.c
index 4d19dd1..a7276d8 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1476,6 +1476,9 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
pte_t pteval;
spinlock_t *ptl;
int ret = SWAP_AGAIN;
+ unsigned long sh_address;
+ bool pmd_sharing_possible = false;
+ unsigned long spmd_start, spmd_end;
struct rmap_private *rp = arg;
enum ttu_flags flags = rp->flags;
@@ -1491,6 +1494,32 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
goto out;
}
+ /*
+ * Only use the range_start/end mmu notifiers if huge pmd sharing
+ * is possible. In the normal case, mmu_notifier_invalidate_page
+ * is sufficient as we only unmap a page. However, if we unshare
+ * a pmd, we will unmap a PUD_SIZE range.
+ */
+ if (PageHuge(page)) {
+ spmd_start = address;
+ spmd_end = spmd_start + vma_mmu_pagesize(vma);
+
+ /*
+ * Check if pmd sharing is possible. If possible, we could
+ * unmap a PUD_SIZE range. spmd_start/spmd_end will be
+ * modified if sharing is possible.
+ */
+ adjust_range_if_pmd_sharing_possible(vma, &spmd_start,
+ &spmd_end);
+ if (spmd_end - spmd_start != vma_mmu_pagesize(vma)) {
+ sh_address = address;
+
+ pmd_sharing_possible = true;
+ mmu_notifier_invalidate_range_start(vma->vm_mm,
+ spmd_start, spmd_end);
+ }
+ }
+
pte = page_check_address(page, mm, address, &ptl,
PageTransCompound(page));
if (!pte)
@@ -1524,6 +1553,30 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
}
}
+ /*
+ * Call huge_pmd_unshare to potentially unshare a huge pmd. Pass
+ * sh_address as it will be modified if unsharing is successful.
+ */
+ if (PageHuge(page) && huge_pmd_unshare(mm, &sh_address, pte)) {
+ /*
+ * huge_pmd_unshare unmapped an entire PMD page. There is
+ * no way of knowing exactly which PMDs may be cached for
+ * this mm, so flush them all. spmd_start/spmd_end cover
+ * this PUD_SIZE range.
+ */
+ flush_cache_range(vma, spmd_start, spmd_end);
+ flush_tlb_range(vma, spmd_start, spmd_end);
+
+ /*
+ * The ref count of the PMD page was dropped which is part
+ * of the way map counting is done for shared PMDs. When
+ * there is no other sharing, huge_pmd_unshare returns false
+ * and we will unmap the actual page and drop map count
+ * to zero.
+ */
+ goto out_unmap;
+ }
+
/* Nuke the page table entry. */
flush_cache_page(vma, address, page_to_pfn(page));
if (should_defer_flush(mm, flags)) {
@@ -1621,6 +1674,9 @@ static int try_to_unmap_one(struct page *page, struct vm_area_struct *vma,
if (ret != SWAP_FAIL && ret != SWAP_MLOCK && !(flags & TTU_MUNLOCK))
mmu_notifier_invalidate_page(mm, address);
out:
+ if (pmd_sharing_possible)
+ mmu_notifier_invalidate_range_end(vma->vm_mm,
+ spmd_start, spmd_end);
return ret;
}
@@ -1652,12 +1708,9 @@ static int page_mapcount_is_zero(struct page *page)
* try_to_unmap - try to remove all page table mappings to a page
* @page: the page to get unmapped
* @flags: action and flags
- * @vma : target vma for reclaim
*
* Tries to remove all the page table entries which are mapping this
* page, used in the pageout path. Caller must hold the page lock.
- * If @vma is not NULL, this function try to remove @page from only @vma
- * without peeking all mapped vma for @page.
* Return values are:
*
* SWAP_SUCCESS - we succeeded in removing all mappings
@@ -1665,8 +1718,7 @@ static int page_mapcount_is_zero(struct page *page)
* SWAP_FAIL - the page is unswappable
* SWAP_MLOCK - page is mlocked.
*/
-int try_to_unmap(struct page *page, enum ttu_flags flags,
- struct vm_area_struct *vma)
+int try_to_unmap(struct page *page, enum ttu_flags flags)
{
int ret;
struct rmap_private rp = {
@@ -1679,7 +1731,6 @@ int try_to_unmap(struct page *page, enum ttu_flags flags,
.arg = &rp,
.done = page_mapcount_is_zero,
.anon_lock = page_lock_anon_vma_read,
- .target_vma = vma,
};
/*
@@ -1739,7 +1790,6 @@ int try_to_munlock(struct page *page)
.arg = &rp,
.done = page_not_mapped,
.anon_lock = page_lock_anon_vma_read,
- .target_vma = NULL,
};
@@ -1802,11 +1852,6 @@ static int rmap_walk_anon(struct page *page, struct rmap_walk_control *rwc,
struct anon_vma_chain *avc;
int ret = SWAP_AGAIN;
- if (rwc->target_vma) {
- unsigned long address = vma_address(page, rwc->target_vma);
- return rwc->rmap_one(page, rwc->target_vma, address, rwc->arg);
- }
-
if (locked) {
anon_vma = page_anon_vma(page);
/* anon_vma disappear under us? */
@@ -1814,7 +1859,6 @@ static int rmap_walk_anon(struct page *page, struct rmap_walk_control *rwc,
} else {
anon_vma = rmap_walk_anon_lock(page, rwc);
}
-
if (!anon_vma)
return ret;
@@ -1859,7 +1903,6 @@ static int rmap_walk_file(struct page *page, struct rmap_walk_control *rwc,
struct address_space *mapping = page_mapping(page);
pgoff_t pgoff;
struct vm_area_struct *vma;
- unsigned long address;
int ret = SWAP_AGAIN;
/*
@@ -1876,13 +1919,6 @@ static int rmap_walk_file(struct page *page, struct rmap_walk_control *rwc,
pgoff = page_to_pgoff(page);
if (!locked)
i_mmap_lock_read(mapping);
-
- if (rwc->target_vma) {
- address = vma_address(page, rwc->target_vma);
- ret = rwc->rmap_one(page, rwc->target_vma, address, rwc->arg);
- goto done;
- }
-
vma_interval_tree_foreach(vma, &mapping->i_mmap, pgoff, pgoff) {
unsigned long address = vma_address(page, vma);
diff --git a/mm/shmem.c b/mm/shmem.c
index 61a39aa..d15851d 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -181,6 +181,38 @@ static inline void shmem_unacct_blocks(unsigned long flags, long pages)
vm_unacct_memory(pages * VM_ACCT(PAGE_SIZE));
}
+static inline bool shmem_inode_acct_block(struct inode *inode, long pages)
+{
+ struct shmem_inode_info *info = SHMEM_I(inode);
+ struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
+
+ if (shmem_acct_block(info->flags, pages))
+ return false;
+
+ if (sbinfo->max_blocks) {
+ if (percpu_counter_compare(&sbinfo->used_blocks,
+ sbinfo->max_blocks - pages) > 0)
+ goto unacct;
+ percpu_counter_add(&sbinfo->used_blocks, pages);
+ }
+
+ return true;
+
+unacct:
+ shmem_unacct_blocks(info->flags, pages);
+ return false;
+}
+
+static inline void shmem_inode_unacct_blocks(struct inode *inode, long pages)
+{
+ struct shmem_inode_info *info = SHMEM_I(inode);
+ struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
+
+ if (sbinfo->max_blocks)
+ percpu_counter_sub(&sbinfo->used_blocks, pages);
+ shmem_unacct_blocks(info->flags, pages);
+}
+
static const struct super_operations shmem_ops;
static const struct address_space_operations shmem_aops;
static const struct file_operations shmem_file_operations;
@@ -237,61 +269,46 @@ static void shmem_recalc_inode(struct inode *inode)
freed = info->alloced - info->swapped - inode->i_mapping->nrpages;
if (freed > 0) {
- struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
- if (sbinfo->max_blocks)
- percpu_counter_add(&sbinfo->used_blocks, -freed);
info->alloced -= freed;
inode->i_blocks -= freed * BLOCKS_PER_PAGE;
- shmem_unacct_blocks(info->flags, freed);
+ shmem_inode_unacct_blocks(inode, freed);
}
}
bool shmem_charge(struct inode *inode, long pages)
{
struct shmem_inode_info *info = SHMEM_I(inode);
- struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
unsigned long flags;
- if (shmem_acct_block(info->flags, pages))
+ if (!shmem_inode_acct_block(inode, pages))
return false;
+
+ /* nrpages adjustment first, then shmem_recalc_inode() when balanced */
+ inode->i_mapping->nrpages += pages;
+
spin_lock_irqsave(&info->lock, flags);
info->alloced += pages;
inode->i_blocks += pages * BLOCKS_PER_PAGE;
shmem_recalc_inode(inode);
spin_unlock_irqrestore(&info->lock, flags);
- inode->i_mapping->nrpages += pages;
- if (!sbinfo->max_blocks)
- return true;
- if (percpu_counter_compare(&sbinfo->used_blocks,
- sbinfo->max_blocks - pages) > 0) {
- inode->i_mapping->nrpages -= pages;
- spin_lock_irqsave(&info->lock, flags);
- info->alloced -= pages;
- shmem_recalc_inode(inode);
- spin_unlock_irqrestore(&info->lock, flags);
- shmem_unacct_blocks(info->flags, pages);
- return false;
- }
- percpu_counter_add(&sbinfo->used_blocks, pages);
return true;
}
void shmem_uncharge(struct inode *inode, long pages)
{
struct shmem_inode_info *info = SHMEM_I(inode);
- struct shmem_sb_info *sbinfo = SHMEM_SB(inode->i_sb);
unsigned long flags;
+ /* nrpages adjustment done by __delete_from_page_cache() or caller */
+
spin_lock_irqsave(&info->lock, flags);
info->alloced -= pages;
inode->i_blocks -= pages * BLOCKS_PER_PAGE;
shmem_recalc_inode(inode);
spin_unlock_irqrestore(&info->lock, flags);
- if (sbinfo->max_blocks)
- percpu_counter_sub(&sbinfo->used_blocks, pages);
- shmem_unacct_blocks(info->flags, pages);
+ shmem_inode_unacct_blocks(inode, pages);
}
/*
@@ -1424,9 +1441,10 @@ static struct page *shmem_alloc_page(gfp_t gfp,
}
static struct page *shmem_alloc_and_acct_page(gfp_t gfp,
- struct shmem_inode_info *info, struct shmem_sb_info *sbinfo,
+ struct inode *inode,
pgoff_t index, bool huge)
{
+ struct shmem_inode_info *info = SHMEM_I(inode);
struct page *page;
int nr;
int err = -ENOSPC;
@@ -1435,14 +1453,8 @@ static struct page *shmem_alloc_and_acct_page(gfp_t gfp,
huge = false;
nr = huge ? HPAGE_PMD_NR : 1;
- if (shmem_acct_block(info->flags, nr))
+ if (!shmem_inode_acct_block(inode, nr))
goto failed;
- if (sbinfo->max_blocks) {
- if (percpu_counter_compare(&sbinfo->used_blocks,
- sbinfo->max_blocks - nr) > 0)
- goto unacct;
- percpu_counter_add(&sbinfo->used_blocks, nr);
- }
if (huge)
page = shmem_alloc_hugepage(gfp, info, index);
@@ -1455,10 +1467,7 @@ static struct page *shmem_alloc_and_acct_page(gfp_t gfp,
}
err = -ENOMEM;
- if (sbinfo->max_blocks)
- percpu_counter_add(&sbinfo->used_blocks, -nr);
-unacct:
- shmem_unacct_blocks(info->flags, nr);
+ shmem_inode_unacct_blocks(inode, nr);
failed:
return ERR_PTR(err);
}
@@ -1485,11 +1494,13 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp,
{
struct page *oldpage, *newpage;
struct address_space *swap_mapping;
+ swp_entry_t entry;
pgoff_t swap_index;
int error;
oldpage = *pagep;
- swap_index = page_private(oldpage);
+ entry.val = page_private(oldpage);
+ swap_index = swp_offset(entry);
swap_mapping = page_mapping(oldpage);
/*
@@ -1508,7 +1519,7 @@ static int shmem_replace_page(struct page **pagep, gfp_t gfp,
__SetPageLocked(newpage);
__SetPageSwapBacked(newpage);
SetPageUptodate(newpage);
- set_page_private(newpage, swap_index);
+ set_page_private(newpage, entry.val);
SetPageSwapCache(newpage);
/*
@@ -1718,10 +1729,9 @@ static int shmem_getpage_gfp(struct inode *inode, pgoff_t index,
}
alloc_huge:
- page = shmem_alloc_and_acct_page(gfp, info, sbinfo,
- index, true);
+ page = shmem_alloc_and_acct_page(gfp, inode, index, true);
if (IS_ERR(page)) {
-alloc_nohuge: page = shmem_alloc_and_acct_page(gfp, info, sbinfo,
+alloc_nohuge: page = shmem_alloc_and_acct_page(gfp, inode,
index, false);
}
if (IS_ERR(page)) {
@@ -1843,10 +1853,7 @@ alloc_nohuge: page = shmem_alloc_and_acct_page(gfp, info, sbinfo,
* Error recovery.
*/
unacct:
- if (sbinfo->max_blocks)
- percpu_counter_sub(&sbinfo->used_blocks,
- 1 << compound_order(page));
- shmem_unacct_blocks(info->flags, 1 << compound_order(page));
+ shmem_inode_unacct_blocks(inode, 1 << compound_order(page));
if (PageTransHuge(page)) {
unlock_page(page);
@@ -2160,6 +2167,8 @@ static struct inode *shmem_get_inode(struct super_block *sb, const struct inode
mpol_shared_policy_init(&info->policy, NULL);
break;
}
+
+ lockdep_annotate_inode_mutex_key(inode);
} else
shmem_free_inode(sb);
return inode;
@@ -2412,9 +2421,7 @@ static loff_t shmem_file_llseek(struct file *file, loff_t offset, int whence)
inode_lock(inode);
/* We're holding i_mutex so we can access i_size directly */
- if (offset < 0)
- offset = -EINVAL;
- else if (offset >= inode->i_size)
+ if (offset < 0 || offset >= inode->i_size)
offset = -ENXIO;
else {
start = offset >> PAGE_SHIFT;
diff --git a/mm/slab.c b/mm/slab.c
index c59844d..263dcda68 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3690,6 +3690,8 @@ __do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller)
struct kmem_cache *cachep;
void *ret;
+ if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
+ return NULL;
cachep = kmalloc_slab(size, flags);
if (unlikely(ZERO_OR_NULL_PTR(cachep)))
return cachep;
@@ -3725,6 +3727,8 @@ static __always_inline void *__do_kmalloc(size_t size, gfp_t flags,
struct kmem_cache *cachep;
void *ret;
+ if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
+ return NULL;
cachep = kmalloc_slab(size, flags);
if (unlikely(ZERO_OR_NULL_PTR(cachep)))
return cachep;
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 0dc614d..467cf11 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -885,18 +885,18 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags)
{
int index;
- if (unlikely(size > KMALLOC_MAX_SIZE)) {
- WARN_ON_ONCE(!(flags & __GFP_NOWARN));
- return NULL;
- }
-
if (size <= 192) {
if (!size)
return ZERO_SIZE_PTR;
index = size_index[size_index_elem(size)];
- } else
+ } else {
+ if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
+ WARN_ON(1);
+ return NULL;
+ }
index = fls(size - 1);
+ }
#ifdef CONFIG_ZONE_DMA
if (unlikely((flags & GFP_DMA)))
diff --git a/mm/slub.c b/mm/slub.c
index 7341005..3efb4f2 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -683,7 +683,7 @@ void object_err(struct kmem_cache *s, struct page *page,
slab_panic(reason);
}
-static void slab_err(struct kmem_cache *s, struct page *page,
+static __printf(3, 4) void slab_err(struct kmem_cache *s, struct page *page,
const char *fmt, ...)
{
va_list args;
@@ -1806,7 +1806,7 @@ static void *get_partial_node(struct kmem_cache *s, struct kmem_cache_node *n,
{
struct page *page, *page2;
void *object = NULL;
- int available = 0;
+ unsigned int available = 0;
int objects;
/*
@@ -4884,10 +4884,10 @@ static ssize_t cpu_partial_show(struct kmem_cache *s, char *buf)
static ssize_t cpu_partial_store(struct kmem_cache *s, const char *buf,
size_t length)
{
- unsigned long objects;
+ unsigned int objects;
int err;
- err = kstrtoul(buf, 10, &objects);
+ err = kstrtouint(buf, 10, &objects);
if (err)
return err;
if (objects && !kmem_cache_has_cpu_partial(s))
diff --git a/mm/swap.c b/mm/swap.c
index 6f22754d..0b01f9d 100644
--- a/mm/swap.c
+++ b/mm/swap.c
@@ -69,6 +69,7 @@ static void __page_cache_release(struct page *page)
del_page_from_lru_list(page, lruvec, page_off_lru(page));
spin_unlock_irqrestore(zone_lru_lock(zone), flags);
}
+ __ClearPageWaiters(page);
mem_cgroup_uncharge(page);
}
@@ -785,6 +786,7 @@ void release_pages(struct page **pages, int nr, bool cold)
/* Clear Active bit in case of parallel mark_page_accessed */
__ClearPageActive(page);
+ __ClearPageWaiters(page);
list_add(&page->lru, &pages_to_free);
}
@@ -955,15 +957,25 @@ unsigned pagevec_lookup(struct pagevec *pvec, struct address_space *mapping,
}
EXPORT_SYMBOL(pagevec_lookup);
-unsigned pagevec_lookup_tag(struct pagevec *pvec, struct address_space *mapping,
- pgoff_t *index, int tag, unsigned nr_pages)
+unsigned pagevec_lookup_range_tag(struct pagevec *pvec,
+ struct address_space *mapping, pgoff_t *index, pgoff_t end,
+ int tag)
{
- pvec->nr = find_get_pages_tag(mapping, index, tag,
- nr_pages, pvec->pages);
+ pvec->nr = find_get_pages_range_tag(mapping, index, end, tag,
+ PAGEVEC_SIZE, pvec->pages);
return pagevec_count(pvec);
}
-EXPORT_SYMBOL(pagevec_lookup_tag);
+EXPORT_SYMBOL(pagevec_lookup_range_tag);
+unsigned pagevec_lookup_range_nr_tag(struct pagevec *pvec,
+ struct address_space *mapping, pgoff_t *index, pgoff_t end,
+ int tag, unsigned max_pages)
+{
+ pvec->nr = find_get_pages_range_tag(mapping, index, end, tag,
+ min_t(unsigned int, max_pages, PAGEVEC_SIZE), pvec->pages);
+ return pagevec_count(pvec);
+}
+EXPORT_SYMBOL(pagevec_lookup_range_nr_tag);
/*
* Perform any setup for the swap system
*/
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 5ac5846..6b6be9c 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -368,6 +368,7 @@ struct page *__read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
/*
* Initiate read into locked page and return.
*/
+ SetPageWorkingset(new_page);
lru_cache_add_anon(new_page);
*new_page_allocated = true;
return new_page;
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 9cf2595..7b996b6 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -2261,6 +2261,35 @@ static int claim_swapfile(struct swap_info_struct *p, struct inode *inode)
return 0;
}
+
+/*
+ * Find out how many pages are allowed for a single swap device. There
+ * are two limiting factors:
+ * 1) the number of bits for the swap offset in the swp_entry_t type, and
+ * 2) the number of bits in the swap pte, as defined by the different
+ * architectures.
+ *
+ * In order to find the largest possible bit mask, a swap entry with
+ * swap type 0 and swap offset ~0UL is created, encoded to a swap pte,
+ * decoded to a swp_entry_t again, and finally the swap offset is
+ * extracted.
+ *
+ * This will mask all the bits from the initial ~0UL mask that can't
+ * be encoded in either the swp_entry_t or the architecture definition
+ * of a swap pte.
+ */
+unsigned long generic_max_swapfile_size(void)
+{
+ return swp_offset(pte_to_swp_entry(
+ swp_entry_to_pte(swp_entry(0, ~0UL)))) + 1;
+}
+
+/* Can be overridden by an architecture for additional checks. */
+__weak unsigned long max_swapfile_size(void)
+{
+ return generic_max_swapfile_size();
+}
+
static unsigned long read_swap_header(struct swap_info_struct *p,
union swap_header *swap_header,
struct inode *inode)
@@ -2296,23 +2325,12 @@ static unsigned long read_swap_header(struct swap_info_struct *p,
p->cluster_next = 1;
p->cluster_nr = 0;
- /*
- * Find out how many pages are allowed for a single swap
- * device. There are two limiting factors: 1) the number
- * of bits for the swap offset in the swp_entry_t type, and
- * 2) the number of bits in the swap pte as defined by the
- * different architectures. In order to find the
- * largest possible bit mask, a swap entry with swap type 0
- * and swap offset ~0UL is created, encoded to a swap pte,
- * decoded to a swp_entry_t again, and finally the swap
- * offset is extracted. This will mask all the bits from
- * the initial ~0UL mask that can't be encoded in either
- * the swp_entry_t or the architecture definition of a
- * swap pte.
- */
- maxpages = swp_offset(pte_to_swp_entry(
- swp_entry_to_pte(swp_entry(0, ~0UL)))) + 1;
+ maxpages = max_swapfile_size();
last_page = swap_header->info.last_page;
+ if (!last_page) {
+ pr_warn("Empty swap-file\n");
+ return 0;
+ }
if (last_page > maxpages) {
pr_warn("Truncating oversized swap area, only using %luk out of %luk\n",
maxpages << (PAGE_SHIFT - 10),
diff --git a/mm/truncate.c b/mm/truncate.c
index 9c809e7..b97359e 100644
--- a/mm/truncate.c
+++ b/mm/truncate.c
@@ -70,6 +70,171 @@ static void clear_exceptional_entry(struct address_space *mapping,
spin_unlock_irq(&mapping->tree_lock);
}
+static void do_truncate_inode_pages_range(struct address_space *mapping,
+ loff_t lstart, loff_t lend, bool fill_zero)
+{
+ pgoff_t start; /* inclusive */
+ pgoff_t end; /* exclusive */
+ unsigned int partial_start; /* inclusive */
+ unsigned int partial_end; /* exclusive */
+ struct pagevec pvec;
+ pgoff_t indices[PAGEVEC_SIZE];
+ pgoff_t index;
+ int i;
+
+ cleancache_invalidate_inode(mapping);
+ if (mapping->nrpages == 0 && mapping->nrexceptional == 0)
+ return;
+
+ /* Offsets within partial pages */
+ partial_start = lstart & (PAGE_SIZE - 1);
+ partial_end = (lend + 1) & (PAGE_SIZE - 1);
+
+ /*
+ * 'start' and 'end' always covers the range of pages to be fully
+ * truncated. Partial pages are covered with 'partial_start' at the
+ * start of the range and 'partial_end' at the end of the range.
+ * Note that 'end' is exclusive while 'lend' is inclusive.
+ */
+ start = (lstart + PAGE_SIZE - 1) >> PAGE_SHIFT;
+ if (lend == -1)
+ /*
+ * lend == -1 indicates end-of-file so we have to set 'end'
+ * to the highest possible pgoff_t and since the type is
+ * unsigned we're using -1.
+ */
+ end = -1;
+ else
+ end = (lend + 1) >> PAGE_SHIFT;
+
+ pagevec_init(&pvec, 0);
+ index = start;
+
+ while (index < end && pagevec_lookup_entries(&pvec, mapping, index,
+ min(end - index, (pgoff_t)PAGEVEC_SIZE),
+ indices)) {
+ for (i = 0; i < pagevec_count(&pvec); i++) {
+ struct page *page = pvec.pages[i];
+
+ /* We rely upon deletion not changing page->index */
+ index = indices[i];
+ if (index >= end)
+ break;
+
+ if (radix_tree_exceptional_entry(page)) {
+ clear_exceptional_entry(mapping, index, page);
+ continue;
+ }
+
+ if (!trylock_page(page))
+ continue;
+ WARN_ON(page->index != index);
+ if (PageWriteback(page)) {
+ unlock_page(page);
+ continue;
+ }
+ truncate_inode_page(mapping, page);
+ if (fill_zero)
+ zero_user(page, 0, PAGE_SIZE);
+ unlock_page(page);
+ }
+ pagevec_remove_exceptionals(&pvec);
+ pagevec_release(&pvec);
+ cond_resched();
+ index++;
+ }
+
+ if (partial_start) {
+ struct page *page = find_lock_page(mapping, start - 1);
+
+ if (page) {
+ unsigned int top = PAGE_SIZE;
+
+ if (start > end) {
+ /* Truncation within a single page */
+ top = partial_end;
+ partial_end = 0;
+ }
+ wait_on_page_writeback(page);
+ zero_user_segment(page, partial_start, top);
+ cleancache_invalidate_page(mapping, page);
+ if (page_has_private(page))
+ do_invalidatepage(page, partial_start,
+ top - partial_start);
+ unlock_page(page);
+ put_page(page);
+ }
+ }
+ if (partial_end) {
+ struct page *page = find_lock_page(mapping, end);
+
+ if (page) {
+ wait_on_page_writeback(page);
+ zero_user_segment(page, 0, partial_end);
+ cleancache_invalidate_page(mapping, page);
+ if (page_has_private(page))
+ do_invalidatepage(page, 0,
+ partial_end);
+ unlock_page(page);
+ put_page(page);
+ }
+ }
+ /*
+ * If the truncation happened within a single page no pages
+ * will be released, just zeroed, so we can bail out now.
+ */
+ if (start >= end)
+ return;
+
+ index = start;
+ for ( ; ; ) {
+ cond_resched();
+ if (!pagevec_lookup_entries(&pvec, mapping, index,
+ min(end - index, (pgoff_t)PAGEVEC_SIZE), indices)) {
+ /* If all gone from start onwards, we're done */
+ if (index == start)
+ break;
+ /* Otherwise restart to make sure all gone */
+ index = start;
+ continue;
+ }
+ if (index == start && indices[0] >= end) {
+ /* All gone out of hole to be punched, we're done */
+ pagevec_remove_exceptionals(&pvec);
+ pagevec_release(&pvec);
+ break;
+ }
+ for (i = 0; i < pagevec_count(&pvec); i++) {
+ struct page *page = pvec.pages[i];
+
+ /* We rely upon deletion not changing page->index */
+ index = indices[i];
+ if (index >= end) {
+ /* Restart punch to make sure all gone */
+ index = start - 1;
+ break;
+ }
+
+ if (radix_tree_exceptional_entry(page)) {
+ clear_exceptional_entry(mapping, index, page);
+ continue;
+ }
+
+ lock_page(page);
+ WARN_ON(page->index != index);
+ wait_on_page_writeback(page);
+ truncate_inode_page(mapping, page);
+ if (fill_zero)
+ zero_user(page, 0, PAGE_SIZE);
+ unlock_page(page);
+ }
+ pagevec_remove_exceptionals(&pvec);
+ pagevec_release(&pvec);
+ index++;
+ }
+ cleancache_invalidate_inode(mapping);
+}
+
/**
* do_invalidatepage - invalidate part or all of a page
* @page: the page which is affected
@@ -229,162 +394,43 @@ int invalidate_inode_page(struct page *page)
void truncate_inode_pages_range(struct address_space *mapping,
loff_t lstart, loff_t lend)
{
- pgoff_t start; /* inclusive */
- pgoff_t end; /* exclusive */
- unsigned int partial_start; /* inclusive */
- unsigned int partial_end; /* exclusive */
- struct pagevec pvec;
- pgoff_t indices[PAGEVEC_SIZE];
- pgoff_t index;
- int i;
-
- cleancache_invalidate_inode(mapping);
- if (mapping->nrpages == 0 && mapping->nrexceptional == 0)
- return;
-
- /* Offsets within partial pages */
- partial_start = lstart & (PAGE_SIZE - 1);
- partial_end = (lend + 1) & (PAGE_SIZE - 1);
-
- /*
- * 'start' and 'end' always covers the range of pages to be fully
- * truncated. Partial pages are covered with 'partial_start' at the
- * start of the range and 'partial_end' at the end of the range.
- * Note that 'end' is exclusive while 'lend' is inclusive.
- */
- start = (lstart + PAGE_SIZE - 1) >> PAGE_SHIFT;
- if (lend == -1)
- /*
- * lend == -1 indicates end-of-file so we have to set 'end'
- * to the highest possible pgoff_t and since the type is
- * unsigned we're using -1.
- */
- end = -1;
- else
- end = (lend + 1) >> PAGE_SHIFT;
-
- pagevec_init(&pvec, 0);
- index = start;
- while (index < end && pagevec_lookup_entries(&pvec, mapping, index,
- min(end - index, (pgoff_t)PAGEVEC_SIZE),
- indices)) {
- for (i = 0; i < pagevec_count(&pvec); i++) {
- struct page *page = pvec.pages[i];
-
- /* We rely upon deletion not changing page->index */
- index = indices[i];
- if (index >= end)
- break;
-
- if (radix_tree_exceptional_entry(page)) {
- clear_exceptional_entry(mapping, index, page);
- continue;
- }
-
- if (!trylock_page(page))
- continue;
- WARN_ON(page_to_index(page) != index);
- if (PageWriteback(page)) {
- unlock_page(page);
- continue;
- }
- truncate_inode_page(mapping, page);
- unlock_page(page);
- }
- pagevec_remove_exceptionals(&pvec);
- pagevec_release(&pvec);
- cond_resched();
- index++;
- }
-
- if (partial_start) {
- struct page *page = find_lock_page(mapping, start - 1);
- if (page) {
- unsigned int top = PAGE_SIZE;
- if (start > end) {
- /* Truncation within a single page */
- top = partial_end;
- partial_end = 0;
- }
- wait_on_page_writeback(page);
- zero_user_segment(page, partial_start, top);
- cleancache_invalidate_page(mapping, page);
- if (page_has_private(page))
- do_invalidatepage(page, partial_start,
- top - partial_start);
- unlock_page(page);
- put_page(page);
- }
- }
- if (partial_end) {
- struct page *page = find_lock_page(mapping, end);
- if (page) {
- wait_on_page_writeback(page);
- zero_user_segment(page, 0, partial_end);
- cleancache_invalidate_page(mapping, page);
- if (page_has_private(page))
- do_invalidatepage(page, 0,
- partial_end);
- unlock_page(page);
- put_page(page);
- }
- }
- /*
- * If the truncation happened within a single page no pages
- * will be released, just zeroed, so we can bail out now.
- */
- if (start >= end)
- return;
-
- index = start;
- for ( ; ; ) {
- cond_resched();
- if (!pagevec_lookup_entries(&pvec, mapping, index,
- min(end - index, (pgoff_t)PAGEVEC_SIZE), indices)) {
- /* If all gone from start onwards, we're done */
- if (index == start)
- break;
- /* Otherwise restart to make sure all gone */
- index = start;
- continue;
- }
- if (index == start && indices[0] >= end) {
- /* All gone out of hole to be punched, we're done */
- pagevec_remove_exceptionals(&pvec);
- pagevec_release(&pvec);
- break;
- }
- for (i = 0; i < pagevec_count(&pvec); i++) {
- struct page *page = pvec.pages[i];
-
- /* We rely upon deletion not changing page->index */
- index = indices[i];
- if (index >= end) {
- /* Restart punch to make sure all gone */
- index = start - 1;
- break;
- }
-
- if (radix_tree_exceptional_entry(page)) {
- clear_exceptional_entry(mapping, index, page);
- continue;
- }
-
- lock_page(page);
- WARN_ON(page_to_index(page) != index);
- wait_on_page_writeback(page);
- truncate_inode_page(mapping, page);
- unlock_page(page);
- }
- pagevec_remove_exceptionals(&pvec);
- pagevec_release(&pvec);
- index++;
- }
- cleancache_invalidate_inode(mapping);
+ do_truncate_inode_pages_range(mapping, lstart, lend, false);
}
EXPORT_SYMBOL(truncate_inode_pages_range);
/**
+ * truncate_inode_pages_range_fill_zero - truncate range of pages
+ * specified by start & end byte offsets and zero them out
+ * @mapping: mapping to truncate
+ * @lstart: offset from which to truncate
+ * @lend: offset to which to truncate (inclusive)
+ *
+ * Truncate the page cache, removing the pages that are between
+ * specified offsets (and zeroing out partial pages
+ * if lstart or lend + 1 is not page aligned).
+ *
+ * Truncate takes two passes - the first pass is nonblocking. It will not
+ * block on page locks and it will not block on writeback. The second pass
+ * will wait. This is to prevent as much IO as possible in the affected region.
+ * The first pass will remove most pages, so the search cost of the second pass
+ * is low.
+ *
+ * We pass down the cache-hot hint to the page freeing code. Even if the
+ * mapping is large, it is probably the case that the final pages are the most
+ * recently touched, and freeing happens in ascending file offset order.
+ *
+ * Note that since ->invalidatepage() accepts range to invalidate
+ * truncate_inode_pages_range is able to handle cases where lend + 1 is not
+ * page aligned properly.
+ */
+void truncate_inode_pages_range_fill_zero(struct address_space *mapping,
+ loff_t lstart, loff_t lend)
+{
+ do_truncate_inode_pages_range(mapping, lstart, lend, true);
+}
+EXPORT_SYMBOL(truncate_inode_pages_range_fill_zero);
+
+/**
* truncate_inode_pages - truncate *all* the pages from an offset
* @mapping: mapping to truncate
* @lstart: offset from which to truncate
@@ -403,6 +449,27 @@ void truncate_inode_pages(struct address_space *mapping, loff_t lstart)
EXPORT_SYMBOL(truncate_inode_pages);
/**
+ * truncate_inode_pages_fill_zero - truncate *all* the pages from an offset
+ * and zero them out
+ * @mapping: mapping to truncate
+ * @lstart: offset from which to truncate
+ *
+ * Called under (and serialised by) inode->i_mutex.
+ *
+ * Note: When this function returns, there can be a page in the process of
+ * deletion (inside __delete_from_page_cache()) in the specified range. Thus
+ * mapping->nrpages can be non-zero when this function returns even after
+ * truncation of the whole mapping.
+ */
+void truncate_inode_pages_fill_zero(struct address_space *mapping,
+ loff_t lstart)
+{
+ truncate_inode_pages_range_fill_zero(mapping, lstart, (loff_t)-1);
+}
+EXPORT_SYMBOL(truncate_inode_pages_fill_zero);
+
+
+/**
* truncate_inode_pages_final - truncate *all* pages before inode dies
* @mapping: mapping to truncate
*
@@ -443,9 +510,13 @@ void truncate_inode_pages_final(struct address_space *mapping)
*/
spin_lock_irq(&mapping->tree_lock);
spin_unlock_irq(&mapping->tree_lock);
-
- truncate_inode_pages(mapping, 0);
}
+
+ /*
+ * Cleancache needs notification even if there are no pages or shadow
+ * entries.
+ */
+ truncate_inode_pages(mapping, 0);
}
EXPORT_SYMBOL(truncate_inode_pages_final);
diff --git a/mm/util.c b/mm/util.c
index 8c755d0..6296de0 100644
--- a/mm/util.c
+++ b/mm/util.c
@@ -576,6 +576,13 @@ int __vm_enough_memory(struct mm_struct *mm, long pages, int cap_sys_admin)
free += global_page_state(NR_SLAB_RECLAIMABLE);
/*
+ * Part of the kernel memory, which can be released
+ * under memory pressure.
+ */
+ free += global_node_page_state(
+ NR_INDIRECTLY_RECLAIMABLE_BYTES) >> PAGE_SHIFT;
+
+ /*
* Leave reserved pages. The pages are not for anonymous pages.
*/
if (free <= totalreserve_pages)
diff --git a/mm/vmacache.c b/mm/vmacache.c
index 035fdeb..c9ca3dd 100644
--- a/mm/vmacache.c
+++ b/mm/vmacache.c
@@ -6,44 +6,6 @@
#include <linux/vmacache.h>
/*
- * Flush vma caches for threads that share a given mm.
- *
- * The operation is safe because the caller holds the mmap_sem
- * exclusively and other threads accessing the vma cache will
- * have mmap_sem held at least for read, so no extra locking
- * is required to maintain the vma cache.
- */
-void vmacache_flush_all(struct mm_struct *mm)
-{
- struct task_struct *g, *p;
-
- count_vm_vmacache_event(VMACACHE_FULL_FLUSHES);
-
- /*
- * Single threaded tasks need not iterate the entire
- * list of process. We can avoid the flushing as well
- * since the mm's seqnum was increased and don't have
- * to worry about other threads' seqnum. Current's
- * flush will occur upon the next lookup.
- */
- if (atomic_read(&mm->mm_users) == 1)
- return;
-
- rcu_read_lock();
- for_each_process_thread(g, p) {
- /*
- * Only flush the vmacache pointers as the
- * mm seqnum is already set and curr's will
- * be set upon invalidation when the next
- * lookup is done.
- */
- if (mm == p->mm)
- vmacache_flush(p);
- }
- rcu_read_unlock();
-}
-
-/*
* This task may be accessing a foreign mm via (for example)
* get_user_pages()->find_vma(). The vmacache is task-local and this
* task's vmacache pertains to a different mm (ie, its own). There is
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index ed89128..bc45dc9 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1568,7 +1568,7 @@ static void __vunmap(const void *addr, int deallocate_pages)
addr))
return;
- area = remove_vm_area(addr);
+ area = find_vmap_area((unsigned long)addr)->vm;
if (unlikely(!area)) {
WARN(1, KERN_ERR "Trying to vfree() nonexistent vm area (%p)\n",
addr);
@@ -1578,6 +1578,7 @@ static void __vunmap(const void *addr, int deallocate_pages)
debug_check_no_locks_freed(addr, get_vm_area_size(area));
debug_check_no_obj_freed(addr, get_vm_area_size(area));
+ remove_vm_area(addr);
if (deallocate_pages) {
int i;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index c8e300c..59061ce 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -47,6 +47,7 @@
#include <linux/prefetch.h>
#include <linux/printk.h>
#include <linux/dax.h>
+#include <linux/psi.h>
#include <asm/tlbflush.h>
#include <asm/div64.h>
@@ -109,13 +110,6 @@ struct scan_control {
/* Number of pages freed so far during a call to shrink_zones() */
unsigned long nr_reclaimed;
-
- /*
- * Reclaim pages from a vma. If the page is shared by other tasks
- * it is zapped from a vma without reclaim so it ends up remaining
- * on memory until last task zap it.
- */
- struct vm_area_struct *target_vma;
};
#ifdef ARCH_HAS_PREFETCH
@@ -951,7 +945,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
unsigned long *ret_nr_congested,
unsigned long *ret_nr_writeback,
unsigned long *ret_nr_immediate,
- bool force_reclaim)
+ bool skip_reference_check)
{
LIST_HEAD(ret_pages);
LIST_HEAD(free_pages);
@@ -983,8 +977,6 @@ static unsigned long shrink_page_list(struct list_head *page_list,
goto keep;
VM_BUG_ON_PAGE(PageActive(page), page);
- if (pgdat)
- VM_BUG_ON_PAGE(page_pgdat(page) != pgdat, page);
sc->nr_scanned++;
@@ -1072,7 +1064,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
/* Case 1 above */
if (current_is_kswapd() &&
PageReclaim(page) &&
- (pgdat && test_bit(PGDAT_WRITEBACK, &pgdat->flags))) {
+ test_bit(PGDAT_WRITEBACK, &pgdat->flags)) {
nr_immediate++;
goto activate_locked;
@@ -1104,7 +1096,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
}
}
- if (!force_reclaim)
+ if (!skip_reference_check)
references = page_check_references(page, sc);
switch (references) {
@@ -1146,8 +1138,7 @@ static unsigned long shrink_page_list(struct list_head *page_list,
if (page_mapped(page) && mapping) {
switch (ret = try_to_unmap(page, lazyfree ?
(ttu_flags | TTU_BATCH_FLUSH | TTU_LZFREE) :
- (ttu_flags | TTU_BATCH_FLUSH),
- sc->target_vma)) {
+ (ttu_flags | TTU_BATCH_FLUSH))) {
case SWAP_FAIL:
goto activate_locked;
case SWAP_AGAIN:
@@ -1290,13 +1281,6 @@ static unsigned long shrink_page_list(struct list_head *page_list,
* appear not as the counts should be low
*/
list_add(&page->lru, &free_pages);
- /*
- * If pagelist are from multiple zones, we should decrease
- * NR_ISOLATED_ANON + x on freed pages in here.
- */
- if (!pgdat)
- dec_node_page_state(page, NR_ISOLATED_ANON +
- page_is_file_cache(page));
continue;
cull_mlocked:
@@ -1342,8 +1326,6 @@ unsigned long reclaim_clean_pages_from_list(struct zone *zone,
.gfp_mask = GFP_KERNEL,
.priority = DEF_PRIORITY,
.may_unmap = 1,
- /* Doesn't allow to write out dirty page */
- .may_writepage = 0,
};
unsigned long ret, dummy1, dummy2, dummy3, dummy4, dummy5;
struct page *page, *next;
@@ -1366,37 +1348,52 @@ unsigned long reclaim_clean_pages_from_list(struct zone *zone,
}
#ifdef CONFIG_PROCESS_RECLAIM
-unsigned long reclaim_pages_from_list(struct list_head *page_list,
- struct vm_area_struct *vma)
+unsigned long reclaim_pages(struct list_head *page_list)
{
+ unsigned long dummy1, dummy2, dummy3, dummy4, dummy5;
+ unsigned long nr_reclaimed;
+ struct page *page;
+ unsigned long nr_isolated[2] = {0, };
+ struct pglist_data *pgdat = NULL;
struct scan_control sc = {
.gfp_mask = GFP_KERNEL,
.priority = DEF_PRIORITY,
.may_writepage = 1,
.may_unmap = 1,
.may_swap = 1,
- .target_vma = vma,
};
- unsigned long nr_reclaimed;
- struct page *page;
- unsigned long dummy1, dummy2, dummy3, dummy4, dummy5;
+ if (list_empty(page_list))
+ return 0;
- list_for_each_entry(page, page_list, lru)
+ list_for_each_entry(page, page_list, lru) {
ClearPageActive(page);
+ if (pgdat == NULL)
+ pgdat = page_pgdat(page);
+ /* XXX: It could be multiple node in other config */
+ WARN_ON_ONCE(pgdat != page_pgdat(page));
+ if (!page_is_file_cache(page))
+ nr_isolated[0]++;
+ else
+ nr_isolated[1]++;
+ }
- nr_reclaimed = shrink_page_list(page_list, NULL, &sc,
+ mod_node_page_state(pgdat, NR_ISOLATED_ANON, nr_isolated[0]);
+ mod_node_page_state(pgdat, NR_ISOLATED_FILE, nr_isolated[1]);
+
+ nr_reclaimed = shrink_page_list(page_list, pgdat, &sc,
TTU_UNMAP|TTU_IGNORE_ACCESS,
&dummy1, &dummy2, &dummy3, &dummy4, &dummy5, true);
while (!list_empty(page_list)) {
page = lru_to_page(page_list);
list_del(&page->lru);
- dec_node_page_state(page, NR_ISOLATED_ANON +
- page_is_file_cache(page));
putback_lru_page(page);
}
+ mod_node_page_state(pgdat, NR_ISOLATED_ANON, -nr_isolated[0]);
+ mod_node_page_state(pgdat, NR_ISOLATED_FILE, -nr_isolated[1]);
+
return nr_reclaimed;
}
#endif
@@ -1440,14 +1437,24 @@ int __isolate_lru_page(struct page *page, isolate_mode_t mode)
if (PageDirty(page)) {
struct address_space *mapping;
+ bool migrate_dirty;
/*
* Only pages without mappings or that have a
* ->migratepage callback are possible to migrate
- * without blocking
+ * without blocking. However, we can be racing with
+ * truncation so it's necessary to lock the page
+ * to stabilise the mapping as truncation holds
+ * the page lock until after the page is removed
+ * from the page cache.
*/
+ if (!trylock_page(page))
+ return ret;
+
mapping = page_mapping(page);
- if (mapping && !mapping->a_ops->migratepage)
+ migrate_dirty = !mapping || mapping->a_ops->migratepage;
+ unlock_page(page);
+ if (!migrate_dirty)
return ret;
}
}
@@ -2084,6 +2091,7 @@ static void shrink_active_list(unsigned long nr_to_scan,
}
ClearPageActive(page); /* we are de-activating */
+ SetPageWorkingset(page);
list_add(&page->lru, &l_inactive);
}
@@ -3015,6 +3023,7 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,
unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
gfp_t gfp_mask, nodemask_t *nodemask)
{
+ ktime_t event_ts;
unsigned long nr_reclaimed;
struct scan_control sc = {
.nr_to_reclaim = SWAP_CLUSTER_MAX,
@@ -3036,6 +3045,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
if (throttle_direct_reclaim(sc.gfp_mask, zonelist, nodemask))
return 1;
+ mm_event_start(&event_ts);
trace_mm_vmscan_direct_reclaim_begin(order,
sc.may_writepage,
sc.gfp_mask,
@@ -3044,6 +3054,7 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,
nr_reclaimed = do_try_to_free_pages(zonelist, &sc);
trace_mm_vmscan_direct_reclaim_end(nr_reclaimed);
+ mm_event_end(MM_RECLAIM, event_ts);
return nr_reclaimed;
}
@@ -3095,6 +3106,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
{
struct zonelist *zonelist;
unsigned long nr_reclaimed;
+ unsigned long pflags;
int nid;
struct scan_control sc = {
.nr_to_reclaim = max(nr_pages, SWAP_CLUSTER_MAX),
@@ -3122,9 +3134,13 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
sc.gfp_mask,
sc.reclaim_idx);
+ psi_memstall_enter(&pflags);
current->flags |= PF_MEMALLOC;
+
nr_reclaimed = do_try_to_free_pages(zonelist, &sc);
+
current->flags &= ~PF_MEMALLOC;
+ psi_memstall_leave(&pflags);
trace_mm_vmscan_memcg_reclaim_end(nr_reclaimed);
@@ -3242,6 +3258,25 @@ static bool kswapd_shrink_node(pg_data_t *pgdat,
struct zone *zone;
int z;
+ if (sc->order) {
+ int ret;
+
+ for (z = 0; z <= sc->reclaim_idx; z++) {
+ zone = pgdat->node_zones + z;
+ if (!managed_zone(zone))
+ continue;
+ ret = compaction_suitable(zone, sc->order, 0,
+ sc->reclaim_idx);
+ if (ret != COMPACT_SUCCESS && ret != COMPACT_CONTINUE)
+ goto reclaim;
+ }
+
+ sc->order = 0;
+ sc->nr_reclaimed = SWAP_CLUSTER_MAX;
+ return true;
+ }
+
+reclaim:
/* Reclaim a number of pages proportional to the number of zones */
sc->nr_to_reclaim = 0;
for (z = 0; z <= sc->reclaim_idx; z++) {
@@ -3289,6 +3324,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)
int i;
unsigned long nr_soft_reclaimed;
unsigned long nr_soft_scanned;
+ unsigned long pflags;
struct zone *zone;
struct scan_control sc = {
.gfp_mask = GFP_KERNEL,
@@ -3298,6 +3334,8 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)
.may_unmap = 1,
.may_swap = 1,
};
+
+ psi_memstall_enter(&pflags);
count_vm_event(PAGEOUTRUN);
do {
@@ -3391,6 +3429,7 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int classzone_idx)
pgdat->kswapd_failures++;
out:
+ psi_memstall_leave(&pflags);
/*
* Return the order kswapd stopped reclaiming at as
* prepare_kswapd_sleep() takes it into account. If another caller
@@ -3547,6 +3586,7 @@ static int kswapd(void *p)
for ( ; ; ) {
bool ret;
+ ktime_t event_ts;
alloc_order = reclaim_order = pgdat->kswapd_order;
classzone_idx = kswapd_classzone_idx(pgdat, classzone_idx);
@@ -3581,7 +3621,9 @@ static int kswapd(void *p)
*/
trace_mm_vmscan_kswapd_wake(pgdat->node_id, classzone_idx,
alloc_order);
+ mm_event_start(&event_ts);
reclaim_order = balance_pgdat(pgdat, alloc_order, classzone_idx);
+ mm_event_end(MM_RECLAIM, event_ts);
if (reclaim_order < alloc_order)
goto kswapd_try_sleep;
}
@@ -3920,7 +3962,13 @@ int node_reclaim(struct pglist_data *pgdat, gfp_t gfp_mask, unsigned int order)
*/
int page_evictable(struct page *page)
{
- return !mapping_unevictable(page_mapping(page)) && !PageMlocked(page);
+ int ret;
+
+ /* Prevent address_space of inode and swap cache from being freed */
+ rcu_read_lock();
+ ret = !mapping_unevictable(page_mapping(page)) && !PageMlocked(page);
+ rcu_read_unlock();
+ return ret;
}
#ifdef CONFIG_SHMEM
diff --git a/mm/vmstat.c b/mm/vmstat.c
index 8bd62ed..a8a25a3 100644
--- a/mm/vmstat.c
+++ b/mm/vmstat.c
@@ -932,6 +932,9 @@ const char * const vmstat_text[] = {
"nr_slab_unreclaimable",
"nr_page_table_pages",
"nr_kernel_stack",
+#if IS_ENABLED(CONFIG_SHADOW_CALL_STACK)
+ "nr_shadow_call_stack_bytes",
+#endif
"nr_overhead",
"nr_bounce",
#if IS_ENABLED(CONFIG_ZSMALLOC)
@@ -957,6 +960,7 @@ const char * const vmstat_text[] = {
"nr_isolated_file",
"workingset_refault",
"workingset_activate",
+ "workingset_restore",
"workingset_nodereclaim",
"nr_anon_pages",
"nr_mapped",
@@ -973,6 +977,8 @@ const char * const vmstat_text[] = {
"nr_vmscan_immediate_reclaim",
"nr_dirtied",
"nr_written",
+ "nr_indirectly_reclaimable",
+ "nr_ion_heap",
/* enum writeback_stat_item counters */
"nr_dirty_threshold",
@@ -1078,6 +1084,9 @@ const char * const vmstat_text[] = {
#ifdef CONFIG_SMP
"nr_tlb_remote_flush",
"nr_tlb_remote_flush_received",
+#else
+ "", /* nr_tlb_remote_flush */
+ "", /* nr_tlb_remote_flush_received */
#endif /* CONFIG_SMP */
"nr_tlb_local_flush_all",
"nr_tlb_local_flush_one",
@@ -1086,7 +1095,6 @@ const char * const vmstat_text[] = {
#ifdef CONFIG_DEBUG_VM_VMACACHE
"vmacache_find_calls",
"vmacache_find_hits",
- "vmacache_full_flushes",
#endif
#endif /* CONFIG_VM_EVENTS_COUNTERS */
};
diff --git a/mm/workingset.c b/mm/workingset.c
index 4c4f056..a697611 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -118,7 +118,7 @@
* the only thing eating into inactive list space is active pages.
*
*
- * Activating refaulting pages
+ * Refaulting inactive pages
*
* All that is known about the active list is that the pages have been
* accessed more than once in the past. This means that at any given
@@ -131,6 +131,10 @@
* used less frequently than the refaulting page - or even not used at
* all anymore.
*
+ * That means if inactive cache is refaulting with a suitable refault
+ * distance, we assume the cache workingset is transitioning and put
+ * pressure on the current active list.
+ *
* If this is wrong and demotion kicks in, the pages which are truly
* used more frequently will be reactivated while the less frequently
* used once will be evicted from memory.
@@ -138,6 +142,14 @@
* But if this is right, the stale pages will be pushed out of memory
* and the used pages get to stay in cache.
*
+ * Refaulting active pages
+ *
+ * If on the other hand the refaulting pages have recently been
+ * deactivated, it means that the active list is no longer protecting
+ * actively used cache from reclaim. The cache is NOT transitioning to
+ * a different workingset; the existing workingset is thrashing in the
+ * space allocated to the page cache.
+ *
*
* Implementation
*
@@ -153,8 +165,7 @@
*/
#define EVICTION_SHIFT (RADIX_TREE_EXCEPTIONAL_ENTRY + \
- NODES_SHIFT + \
- MEM_CGROUP_ID_SHIFT)
+ 1 + NODES_SHIFT + MEM_CGROUP_ID_SHIFT)
#define EVICTION_MASK (~0UL >> EVICTION_SHIFT)
/*
@@ -167,23 +178,28 @@
*/
static unsigned int bucket_order __read_mostly;
-static void *pack_shadow(int memcgid, pg_data_t *pgdat, unsigned long eviction)
+static void *pack_shadow(int memcgid, pg_data_t *pgdat, unsigned long eviction,
+ bool workingset)
{
eviction >>= bucket_order;
eviction = (eviction << MEM_CGROUP_ID_SHIFT) | memcgid;
eviction = (eviction << NODES_SHIFT) | pgdat->node_id;
+ eviction = (eviction << 1) | workingset;
eviction = (eviction << RADIX_TREE_EXCEPTIONAL_SHIFT);
return (void *)(eviction | RADIX_TREE_EXCEPTIONAL_ENTRY);
}
static void unpack_shadow(void *shadow, int *memcgidp, pg_data_t **pgdat,
- unsigned long *evictionp)
+ unsigned long *evictionp, bool *workingsetp)
{
unsigned long entry = (unsigned long)shadow;
int memcgid, nid;
+ bool workingset;
entry >>= RADIX_TREE_EXCEPTIONAL_SHIFT;
+ workingset = entry & 1;
+ entry >>= 1;
nid = entry & ((1UL << NODES_SHIFT) - 1);
entry >>= NODES_SHIFT;
memcgid = entry & ((1UL << MEM_CGROUP_ID_SHIFT) - 1);
@@ -192,6 +208,7 @@ static void unpack_shadow(void *shadow, int *memcgidp, pg_data_t **pgdat,
*memcgidp = memcgid;
*pgdat = NODE_DATA(nid);
*evictionp = entry << bucket_order;
+ *workingsetp = workingset;
}
/**
@@ -204,8 +221,8 @@ static void unpack_shadow(void *shadow, int *memcgidp, pg_data_t **pgdat,
*/
void *workingset_eviction(struct address_space *mapping, struct page *page)
{
- struct mem_cgroup *memcg = page_memcg(page);
struct pglist_data *pgdat = page_pgdat(page);
+ struct mem_cgroup *memcg = page_memcg(page);
int memcgid = mem_cgroup_id(memcg);
unsigned long eviction;
struct lruvec *lruvec;
@@ -217,30 +234,30 @@ void *workingset_eviction(struct address_space *mapping, struct page *page)
lruvec = mem_cgroup_lruvec(pgdat, memcg);
eviction = atomic_long_inc_return(&lruvec->inactive_age);
- return pack_shadow(memcgid, pgdat, eviction);
+ return pack_shadow(memcgid, pgdat, eviction, PageWorkingset(page));
}
/**
* workingset_refault - evaluate the refault of a previously evicted page
+ * @page: the freshly allocated replacement page
* @shadow: shadow entry of the evicted page
*
* Calculates and evaluates the refault distance of the previously
* evicted page in the context of the node it was allocated in.
- *
- * Returns %true if the page should be activated, %false otherwise.
*/
-bool workingset_refault(void *shadow)
+void workingset_refault(struct page *page, void *shadow)
{
unsigned long refault_distance;
+ struct pglist_data *pgdat;
unsigned long active_file;
struct mem_cgroup *memcg;
unsigned long eviction;
struct lruvec *lruvec;
unsigned long refault;
- struct pglist_data *pgdat;
+ bool workingset;
int memcgid;
- unpack_shadow(shadow, &memcgid, &pgdat, &eviction);
+ unpack_shadow(shadow, &memcgid, &pgdat, &eviction, &workingset);
rcu_read_lock();
/*
@@ -260,40 +277,51 @@ bool workingset_refault(void *shadow)
* configurations instead.
*/
memcg = mem_cgroup_from_id(memcgid);
- if (!mem_cgroup_disabled() && !memcg) {
- rcu_read_unlock();
- return false;
- }
+ if (!mem_cgroup_disabled() && !memcg)
+ goto out;
lruvec = mem_cgroup_lruvec(pgdat, memcg);
refault = atomic_long_read(&lruvec->inactive_age);
active_file = lruvec_lru_size(lruvec, LRU_ACTIVE_FILE, MAX_NR_ZONES);
- rcu_read_unlock();
/*
- * The unsigned subtraction here gives an accurate distance
- * across inactive_age overflows in most cases.
+ * Calculate the refault distance
*
- * There is a special case: usually, shadow entries have a
- * short lifetime and are either refaulted or reclaimed along
- * with the inode before they get too old. But it is not
- * impossible for the inactive_age to lap a shadow entry in
- * the field, which can then can result in a false small
- * refault distance, leading to a false activation should this
- * old entry actually refault again. However, earlier kernels
- * used to deactivate unconditionally with *every* reclaim
- * invocation for the longest time, so the occasional
- * inappropriate activation leading to pressure on the active
- * list is not a problem.
+ * The unsigned subtraction here gives an accurate distance
+ * across inactive_age overflows in most cases. There is a
+ * special case: usually, shadow entries have a short lifetime
+ * and are either refaulted or reclaimed along with the inode
+ * before they get too old. But it is not impossible for the
+ * inactive_age to lap a shadow entry in the field, which can
+ * then result in a false small refault distance, leading to a
+ * false activation should this old entry actually refault
+ * again. However, earlier kernels used to deactivate
+ * unconditionally with *every* reclaim invocation for the
+ * longest time, so the occasional inappropriate activation
+ * leading to pressure on the active list is not a problem.
*/
refault_distance = (refault - eviction) & EVICTION_MASK;
inc_node_state(pgdat, WORKINGSET_REFAULT);
- if (refault_distance <= active_file) {
- inc_node_state(pgdat, WORKINGSET_ACTIVATE);
- return true;
+ /*
+ * Compare the distance to the existing workingset size. We
+ * don't act on pages that couldn't stay resident even if all
+ * the memory was available to the page cache.
+ */
+ if (refault_distance > active_file)
+ goto out;
+
+ SetPageActive(page);
+ atomic_long_inc(&lruvec->inactive_age);
+ inc_node_state(pgdat, WORKINGSET_ACTIVATE);
+
+ /* Page was active prior to eviction */
+ if (workingset) {
+ SetPageWorkingset(page);
+ inc_node_state(pgdat, WORKINGSET_RESTORE);
}
- return false;
+out:
+ rcu_read_unlock();
}
/**
diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index d3548c4..cf15851 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -473,7 +473,7 @@ static bool is_zspage_isolated(struct zspage *zspage)
return zspage->isolated;
}
-static int is_first_page(struct page *page)
+static __maybe_unused int is_first_page(struct page *page)
{
return PagePrivate(page);
}
@@ -558,20 +558,23 @@ static int get_size_class_index(int size)
return min(zs_size_classes - 1, idx);
}
+/* type can be of enum type zs_stat_type or fullness_group */
static inline void zs_stat_inc(struct size_class *class,
- enum zs_stat_type type, unsigned long cnt)
+ int type, unsigned long cnt)
{
class->stats.objs[type] += cnt;
}
+/* type can be of enum type zs_stat_type or fullness_group */
static inline void zs_stat_dec(struct size_class *class,
- enum zs_stat_type type, unsigned long cnt)
+ int type, unsigned long cnt)
{
class->stats.objs[type] -= cnt;
}
+/* type can be of enum type zs_stat_type or fullness_group */
static inline unsigned long zs_stat_get(struct size_class *class,
- enum zs_stat_type type)
+ int type)
{
return class->stats.objs[type];
}
diff --git a/mm/zswap.c b/mm/zswap.c
index ded051e..c2b5435 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1018,6 +1018,15 @@ static int zswap_frontswap_store(unsigned type, pgoff_t offset,
ret = -ENOMEM;
goto reject;
}
+
+ /* A second zswap_is_full() check after
+ * zswap_shrink() to make sure it's now
+ * under the max_pool_percent
+ */
+ if (zswap_is_full()) {
+ ret = -ENOMEM;
+ goto reject;
+ }
}
/* allocate entry */
diff --git a/net/6lowpan/iphc.c b/net/6lowpan/iphc.c
index 79f1fa2..23654f1 100644
--- a/net/6lowpan/iphc.c
+++ b/net/6lowpan/iphc.c
@@ -745,6 +745,7 @@ int lowpan_header_decompress(struct sk_buff *skb, const struct net_device *dev,
hdr.hop_limit, &hdr.daddr);
skb_push(skb, sizeof(hdr));
+ skb_reset_mac_header(skb);
skb_reset_network_header(skb);
skb_copy_to_linear_data(skb, &hdr, sizeof(hdr));
diff --git a/net/9p/client.c b/net/9p/client.c
index 1fd6019..142afe70 100644
--- a/net/9p/client.c
+++ b/net/9p/client.c
@@ -156,6 +156,12 @@ static int parse_opts(char *opts, struct p9_client *clnt)
ret = r;
continue;
}
+ if (option < 4096) {
+ p9_debug(P9_DEBUG_ERROR,
+ "msize should be at least 4k\n");
+ ret = -EINVAL;
+ continue;
+ }
clnt->msize = option;
break;
case Opt_trans:
@@ -931,7 +937,7 @@ static int p9_client_version(struct p9_client *c)
{
int err = 0;
struct p9_req_t *req;
- char *version;
+ char *version = NULL;
int msize;
p9_debug(P9_DEBUG_9P, ">>> TVERSION msize %d protocol %d\n",
@@ -972,10 +978,18 @@ static int p9_client_version(struct p9_client *c)
else if (!strncmp(version, "9P2000", 6))
c->proto_version = p9_proto_legacy;
else {
+ p9_debug(P9_DEBUG_ERROR,
+ "server returned an unknown version: %s\n", version);
err = -EREMOTEIO;
goto error;
}
+ if (msize < 4096) {
+ p9_debug(P9_DEBUG_ERROR,
+ "server returned a msize < 4096: %d\n", msize);
+ err = -EREMOTEIO;
+ goto error;
+ }
if (msize < c->msize)
c->msize = msize;
@@ -1040,6 +1054,13 @@ struct p9_client *p9_client_create(const char *dev_name, char *options)
if (clnt->msize > clnt->trans_mod->maxsize)
clnt->msize = clnt->trans_mod->maxsize;
+ if (clnt->msize < 4096) {
+ p9_debug(P9_DEBUG_ERROR,
+ "Please specify a msize of at least 4k\n");
+ err = -EINVAL;
+ goto free_client;
+ }
+
err = p9_client_version(clnt);
if (err)
goto close_trans;
diff --git a/net/9p/protocol.c b/net/9p/protocol.c
index 16d2875..145f805 100644
--- a/net/9p/protocol.c
+++ b/net/9p/protocol.c
@@ -46,10 +46,15 @@ p9pdu_writef(struct p9_fcall *pdu, int proto_version, const char *fmt, ...);
void p9stat_free(struct p9_wstat *stbuf)
{
kfree(stbuf->name);
+ stbuf->name = NULL;
kfree(stbuf->uid);
+ stbuf->uid = NULL;
kfree(stbuf->gid);
+ stbuf->gid = NULL;
kfree(stbuf->muid);
+ stbuf->muid = NULL;
kfree(stbuf->extension);
+ stbuf->extension = NULL;
}
EXPORT_SYMBOL(p9stat_free);
diff --git a/net/9p/trans_fd.c b/net/9p/trans_fd.c
index 7bc2208..aa45866 100644
--- a/net/9p/trans_fd.c
+++ b/net/9p/trans_fd.c
@@ -181,6 +181,8 @@ static void p9_mux_poll_stop(struct p9_conn *m)
spin_lock_irqsave(&p9_poll_lock, flags);
list_del_init(&m->poll_pending_link);
spin_unlock_irqrestore(&p9_poll_lock, flags);
+
+ flush_work(&p9_poll_work);
}
/**
@@ -193,15 +195,14 @@ static void p9_mux_poll_stop(struct p9_conn *m)
static void p9_conn_cancel(struct p9_conn *m, int err)
{
struct p9_req_t *req, *rtmp;
- unsigned long flags;
LIST_HEAD(cancel_list);
p9_debug(P9_DEBUG_ERROR, "mux %p err %d\n", m, err);
- spin_lock_irqsave(&m->client->lock, flags);
+ spin_lock(&m->client->lock);
if (m->err) {
- spin_unlock_irqrestore(&m->client->lock, flags);
+ spin_unlock(&m->client->lock);
return;
}
@@ -213,7 +214,6 @@ static void p9_conn_cancel(struct p9_conn *m, int err)
list_for_each_entry_safe(req, rtmp, &m->unsent_req_list, req_list) {
list_move(&req->req_list, &cancel_list);
}
- spin_unlock_irqrestore(&m->client->lock, flags);
list_for_each_entry_safe(req, rtmp, &cancel_list, req_list) {
p9_debug(P9_DEBUG_ERROR, "call back req %p\n", req);
@@ -222,6 +222,7 @@ static void p9_conn_cancel(struct p9_conn *m, int err)
req->t_err = err;
p9_client_cb(m->client, req, REQ_STATUS_ERROR);
}
+ spin_unlock(&m->client->lock);
}
static int
@@ -377,8 +378,9 @@ static void p9_read_work(struct work_struct *work)
if (m->req->status != REQ_STATUS_ERROR)
status = REQ_STATUS_RCVD;
list_del(&m->req->req_list);
- spin_unlock(&m->client->lock);
+ /* update req->status while holding client->lock */
p9_client_cb(m->client, m->req, status);
+ spin_unlock(&m->client->lock);
m->rc.sdata = NULL;
m->rc.offset = 0;
m->rc.capacity = 0;
@@ -937,7 +939,7 @@ p9_fd_create_tcp(struct p9_client *client, const char *addr, char *args)
if (err < 0)
return err;
- if (valid_ipaddr4(addr) < 0)
+ if (addr == NULL || valid_ipaddr4(addr) < 0)
return -EINVAL;
csocket = NULL;
@@ -985,6 +987,9 @@ p9_fd_create_unix(struct p9_client *client, const char *addr, char *args)
csocket = NULL;
+ if (addr == NULL)
+ return -EINVAL;
+
if (strlen(addr) >= UNIX_PATH_MAX) {
pr_err("%s (%d): address too long: %s\n",
__func__, task_pid_nr(current), addr);
diff --git a/net/9p/trans_rdma.c b/net/9p/trans_rdma.c
index 553ed4e..5a2ad47 100644
--- a/net/9p/trans_rdma.c
+++ b/net/9p/trans_rdma.c
@@ -622,6 +622,9 @@ rdma_create_trans(struct p9_client *client, const char *addr, char *args)
struct rdma_conn_param conn_param;
struct ib_qp_init_attr qp_attr;
+ if (addr == NULL)
+ return -EINVAL;
+
/* Parse the transport specific mount options */
err = parse_opts(args, &opts);
if (err < 0)
diff --git a/net/9p/trans_virtio.c b/net/9p/trans_virtio.c
index 3aa5a93..e73fd64 100644
--- a/net/9p/trans_virtio.c
+++ b/net/9p/trans_virtio.c
@@ -189,7 +189,7 @@ static int pack_sg_list(struct scatterlist *sg, int start,
s = rest_of_page(data);
if (s > count)
s = count;
- BUG_ON(index > limit);
+ BUG_ON(index >= limit);
/* Make sure we don't terminate early. */
sg_unmark_end(&sg[index]);
sg_set_buf(&sg[index++], data, s);
@@ -234,6 +234,7 @@ pack_sg_list_p(struct scatterlist *sg, int start, int limit,
s = PAGE_SIZE - data_off;
if (s > count)
s = count;
+ BUG_ON(index >= limit);
/* Make sure we don't terminate early. */
sg_unmark_end(&sg[index]);
sg_set_page(&sg[index++], pdata[i++], s, data_off);
@@ -406,6 +407,7 @@ p9_virtio_zc_request(struct p9_client *client, struct p9_req_t *req,
p9_debug(P9_DEBUG_TRANS, "virtio request\n");
if (uodata) {
+ __le32 sz;
int n = p9_get_mapped_pages(chan, &out_pages, uodata,
outlen, &offs, &need_drop);
if (n < 0)
@@ -416,6 +418,12 @@ p9_virtio_zc_request(struct p9_client *client, struct p9_req_t *req,
memcpy(&req->tc->sdata[req->tc->size - 4], &v, 4);
outlen = n;
}
+ /* The size field of the message must include the length of the
+ * header and the length of the data. We didn't actually know
+ * the length of the data until this point so add it in now.
+ */
+ sz = cpu_to_le32(req->tc->size + outlen);
+ memcpy(&req->tc->sdata[0], &sz, sizeof(sz));
} else if (uidata) {
int n = p9_get_mapped_pages(chan, &in_pages, uidata,
inlen, &offs, &need_drop);
@@ -563,7 +571,7 @@ static int p9_virtio_probe(struct virtio_device *vdev)
chan->vq = virtio_find_single_vq(vdev, req_done, "requests");
if (IS_ERR(chan->vq)) {
err = PTR_ERR(chan->vq);
- goto out_free_vq;
+ goto out_free_chan;
}
chan->vq->vdev->priv = chan;
spin_lock_init(&chan->lock);
@@ -616,6 +624,7 @@ static int p9_virtio_probe(struct virtio_device *vdev)
kfree(tag);
out_free_vq:
vdev->config->del_vqs(vdev);
+out_free_chan:
kfree(chan);
fail:
return err;
@@ -643,6 +652,9 @@ p9_virtio_create(struct p9_client *client, const char *devname, char *args)
int ret = -ENOENT;
int found = 0;
+ if (devname == NULL)
+ return -EINVAL;
+
mutex_lock(&virtio_9p_lock);
list_for_each_entry(chan, &virtio_chan_list, chan_list) {
if (!strncmp(devname, chan->tag, chan->tag_len) &&
diff --git a/net/atm/lec.c b/net/atm/lec.c
index 5d26938..1e84c52 100644
--- a/net/atm/lec.c
+++ b/net/atm/lec.c
@@ -41,6 +41,9 @@ static unsigned char bridge_ula_lec[] = { 0x01, 0x80, 0xc2, 0x00, 0x00 };
#include <linux/module.h>
#include <linux/init.h>
+/* Hardening for Spectre-v1 */
+#include <linux/nospec.h>
+
#include "lec.h"
#include "lec_arpc.h"
#include "resources.h"
@@ -697,8 +700,10 @@ static int lec_vcc_attach(struct atm_vcc *vcc, void __user *arg)
bytes_left = copy_from_user(&ioc_data, arg, sizeof(struct atmlec_ioc));
if (bytes_left != 0)
pr_info("copy from user failed for %d bytes\n", bytes_left);
- if (ioc_data.dev_num < 0 || ioc_data.dev_num >= MAX_LEC_ITF ||
- !dev_lec[ioc_data.dev_num])
+ if (ioc_data.dev_num < 0 || ioc_data.dev_num >= MAX_LEC_ITF)
+ return -EINVAL;
+ ioc_data.dev_num = array_index_nospec(ioc_data.dev_num, MAX_LEC_ITF);
+ if (!dev_lec[ioc_data.dev_num])
return -EINVAL;
vpriv = kmalloc(sizeof(struct lec_vcc_priv), GFP_KERNEL);
if (!vpriv)
diff --git a/net/ax25/af_ax25.c b/net/ax25/af_ax25.c
index 2fdebab..2772f6a 100644
--- a/net/ax25/af_ax25.c
+++ b/net/ax25/af_ax25.c
@@ -654,15 +654,22 @@ static int ax25_setsockopt(struct socket *sock, int level, int optname,
break;
}
- dev = dev_get_by_name(&init_net, devname);
+ rtnl_lock();
+ dev = __dev_get_by_name(&init_net, devname);
if (!dev) {
+ rtnl_unlock();
res = -ENODEV;
break;
}
ax25->ax25_dev = ax25_dev_ax25dev(dev);
+ if (!ax25->ax25_dev) {
+ rtnl_unlock();
+ res = -ENODEV;
+ break;
+ }
ax25_fillin_cb(ax25, ax25->ax25_dev);
- dev_put(dev);
+ rtnl_unlock();
break;
default:
diff --git a/net/ax25/ax25_dev.c b/net/ax25/ax25_dev.c
index 3d10676..5faca5d 100644
--- a/net/ax25/ax25_dev.c
+++ b/net/ax25/ax25_dev.c
@@ -116,6 +116,7 @@ void ax25_dev_device_down(struct net_device *dev)
if ((s = ax25_dev_list) == ax25_dev) {
ax25_dev_list = s->next;
spin_unlock_bh(&ax25_dev_lock);
+ dev->ax25_ptr = NULL;
dev_put(dev);
kfree(ax25_dev);
return;
@@ -125,6 +126,7 @@ void ax25_dev_device_down(struct net_device *dev)
if (s->next == ax25_dev) {
s->next = ax25_dev->next;
spin_unlock_bh(&ax25_dev_lock);
+ dev->ax25_ptr = NULL;
dev_put(dev);
kfree(ax25_dev);
return;
diff --git a/net/batman-adv/bat_iv_ogm.c b/net/batman-adv/bat_iv_ogm.c
index e2d18d0..1ae8c59f 100644
--- a/net/batman-adv/bat_iv_ogm.c
+++ b/net/batman-adv/bat_iv_ogm.c
@@ -2704,8 +2704,8 @@ static int batadv_iv_gw_dump_entry(struct sk_buff *msg, u32 portid, u32 seq,
{
struct batadv_neigh_ifinfo *router_ifinfo = NULL;
struct batadv_neigh_node *router;
- struct batadv_gw_node *curr_gw;
- int ret = -EINVAL;
+ struct batadv_gw_node *curr_gw = NULL;
+ int ret = 0;
void *hdr;
router = batadv_orig_router_get(gw_node->orig_node, BATADV_IF_DEFAULT);
@@ -2752,6 +2752,8 @@ static int batadv_iv_gw_dump_entry(struct sk_buff *msg, u32 portid, u32 seq,
ret = 0;
out:
+ if (curr_gw)
+ batadv_gw_node_put(curr_gw);
if (router_ifinfo)
batadv_neigh_ifinfo_put(router_ifinfo);
if (router)
diff --git a/net/batman-adv/bat_v.c b/net/batman-adv/bat_v.c
index e79f6f0..4348118 100644
--- a/net/batman-adv/bat_v.c
+++ b/net/batman-adv/bat_v.c
@@ -919,8 +919,8 @@ static int batadv_v_gw_dump_entry(struct sk_buff *msg, u32 portid, u32 seq,
{
struct batadv_neigh_ifinfo *router_ifinfo = NULL;
struct batadv_neigh_node *router;
- struct batadv_gw_node *curr_gw;
- int ret = -EINVAL;
+ struct batadv_gw_node *curr_gw = NULL;
+ int ret = 0;
void *hdr;
router = batadv_orig_router_get(gw_node->orig_node, BATADV_IF_DEFAULT);
@@ -987,6 +987,8 @@ static int batadv_v_gw_dump_entry(struct sk_buff *msg, u32 portid, u32 seq,
ret = 0;
out:
+ if (curr_gw)
+ batadv_gw_node_put(curr_gw);
if (router_ifinfo)
batadv_neigh_ifinfo_put(router_ifinfo);
if (router)
diff --git a/net/batman-adv/bat_v_elp.c b/net/batman-adv/bat_v_elp.c
index ee08540..5d79004 100644
--- a/net/batman-adv/bat_v_elp.c
+++ b/net/batman-adv/bat_v_elp.c
@@ -243,6 +243,7 @@ static void batadv_v_elp_periodic_work(struct work_struct *work)
struct batadv_priv *bat_priv;
struct sk_buff *skb;
u32 elp_interval;
+ bool ret;
bat_v = container_of(work, struct batadv_hard_iface_bat_v, elp_wq.work);
hard_iface = container_of(bat_v, struct batadv_hard_iface, bat_v);
@@ -304,8 +305,11 @@ static void batadv_v_elp_periodic_work(struct work_struct *work)
* may sleep and that is not allowed in an rcu protected
* context. Therefore schedule a task for that.
*/
- queue_work(batadv_event_workqueue,
- &hardif_neigh->bat_v.metric_work);
+ ret = queue_work(batadv_event_workqueue,
+ &hardif_neigh->bat_v.metric_work);
+
+ if (!ret)
+ batadv_hardif_neigh_put(hardif_neigh);
}
rcu_read_unlock();
diff --git a/net/batman-adv/bridge_loop_avoidance.c b/net/batman-adv/bridge_loop_avoidance.c
index 5419b12..8b6f654 100644
--- a/net/batman-adv/bridge_loop_avoidance.c
+++ b/net/batman-adv/bridge_loop_avoidance.c
@@ -1767,6 +1767,7 @@ batadv_bla_loopdetect_check(struct batadv_priv *bat_priv, struct sk_buff *skb,
{
struct batadv_bla_backbone_gw *backbone_gw;
struct ethhdr *ethhdr;
+ bool ret;
ethhdr = eth_hdr(skb);
@@ -1790,8 +1791,13 @@ batadv_bla_loopdetect_check(struct batadv_priv *bat_priv, struct sk_buff *skb,
if (unlikely(!backbone_gw))
return true;
- queue_work(batadv_event_workqueue, &backbone_gw->report_work);
- /* backbone_gw is unreferenced in the report work function function */
+ ret = queue_work(batadv_event_workqueue, &backbone_gw->report_work);
+
+ /* backbone_gw is unreferenced in the report work function function
+ * if queue_work() call was successful
+ */
+ if (!ret)
+ batadv_backbone_gw_put(backbone_gw);
return true;
}
@@ -2149,22 +2155,25 @@ batadv_bla_claim_dump_bucket(struct sk_buff *msg, u32 portid, u32 seq,
{
struct batadv_bla_claim *claim;
int idx = 0;
+ int ret = 0;
rcu_read_lock();
hlist_for_each_entry_rcu(claim, head, hash_entry) {
if (idx++ < *idx_skip)
continue;
- if (batadv_bla_claim_dump_entry(msg, portid, seq,
- primary_if, claim)) {
+
+ ret = batadv_bla_claim_dump_entry(msg, portid, seq,
+ primary_if, claim);
+ if (ret) {
*idx_skip = idx - 1;
goto unlock;
}
}
- *idx_skip = idx;
+ *idx_skip = 0;
unlock:
rcu_read_unlock();
- return 0;
+ return ret;
}
/**
@@ -2379,22 +2388,25 @@ batadv_bla_backbone_dump_bucket(struct sk_buff *msg, u32 portid, u32 seq,
{
struct batadv_bla_backbone_gw *backbone_gw;
int idx = 0;
+ int ret = 0;
rcu_read_lock();
hlist_for_each_entry_rcu(backbone_gw, head, hash_entry) {
if (idx++ < *idx_skip)
continue;
- if (batadv_bla_backbone_dump_entry(msg, portid, seq,
- primary_if, backbone_gw)) {
+
+ ret = batadv_bla_backbone_dump_entry(msg, portid, seq,
+ primary_if, backbone_gw);
+ if (ret) {
*idx_skip = idx - 1;
goto unlock;
}
}
- *idx_skip = idx;
+ *idx_skip = 0;
unlock:
rcu_read_unlock();
- return 0;
+ return ret;
}
/**
diff --git a/net/batman-adv/distributed-arp-table.c b/net/batman-adv/distributed-arp-table.c
index e257efd..df7c6a0 100644
--- a/net/batman-adv/distributed-arp-table.c
+++ b/net/batman-adv/distributed-arp-table.c
@@ -391,7 +391,7 @@ static void batadv_dbg_arp(struct batadv_priv *bat_priv, struct sk_buff *skb,
batadv_arp_hw_src(skb, hdr_size), &ip_src,
batadv_arp_hw_dst(skb, hdr_size), &ip_dst);
- if (hdr_size == 0)
+ if (hdr_size < sizeof(struct batadv_unicast_packet))
return;
unicast_4addr_packet = (struct batadv_unicast_4addr_packet *)skb->data;
diff --git a/net/batman-adv/fragmentation.c b/net/batman-adv/fragmentation.c
index 0934730..a06b604 100644
--- a/net/batman-adv/fragmentation.c
+++ b/net/batman-adv/fragmentation.c
@@ -264,7 +264,7 @@ batadv_frag_merge_packets(struct hlist_head *chain)
kfree(entry);
packet = (struct batadv_frag_packet *)skb_out->data;
- size = ntohs(packet->total_size);
+ size = ntohs(packet->total_size) + hdr_size;
/* Make room for the rest of the fragments. */
if (pskb_expand_head(skb_out, 0, size - skb_out->len, GFP_ATOMIC) < 0) {
@@ -276,7 +276,8 @@ batadv_frag_merge_packets(struct hlist_head *chain)
/* Move the existing MAC header to just before the payload. (Override
* the fragment header.)
*/
- skb_pull_rcsum(skb_out, hdr_size);
+ skb_pull(skb_out, hdr_size);
+ skb_out->ip_summed = CHECKSUM_NONE;
memmove(skb_out->data - ETH_HLEN, skb_mac_header(skb_out), ETH_HLEN);
skb_set_mac_header(skb_out, -ETH_HLEN);
skb_reset_network_header(skb_out);
diff --git a/net/batman-adv/gateway_client.c b/net/batman-adv/gateway_client.c
index de055d6..ed9aaf3 100644
--- a/net/batman-adv/gateway_client.c
+++ b/net/batman-adv/gateway_client.c
@@ -715,6 +715,9 @@ bool batadv_gw_out_of_range(struct batadv_priv *bat_priv,
vid = batadv_get_vid(skb, 0);
+ if (is_multicast_ether_addr(ethhdr->h_dest))
+ goto out;
+
orig_dst_node = batadv_transtable_search(bat_priv, ethhdr->h_source,
ethhdr->h_dest, vid);
if (!orig_dst_node)
diff --git a/net/batman-adv/multicast.c b/net/batman-adv/multicast.c
index 13661f4..5a2aac1 100644
--- a/net/batman-adv/multicast.c
+++ b/net/batman-adv/multicast.c
@@ -527,8 +527,8 @@ static bool batadv_mcast_mla_tvlv_update(struct batadv_priv *bat_priv)
bat_priv->mcast.enabled = true;
}
- return !(mcast_data.flags &
- (BATADV_MCAST_WANT_ALL_IPV4 | BATADV_MCAST_WANT_ALL_IPV6));
+ return !(mcast_data.flags & BATADV_MCAST_WANT_ALL_IPV4 &&
+ mcast_data.flags & BATADV_MCAST_WANT_ALL_IPV6);
}
/**
@@ -769,8 +769,8 @@ static struct batadv_orig_node *
batadv_mcast_forw_tt_node_get(struct batadv_priv *bat_priv,
struct ethhdr *ethhdr)
{
- return batadv_transtable_search(bat_priv, ethhdr->h_source,
- ethhdr->h_dest, BATADV_NO_FLAGS);
+ return batadv_transtable_search(bat_priv, NULL, ethhdr->h_dest,
+ BATADV_NO_FLAGS);
}
/**
diff --git a/net/batman-adv/network-coding.c b/net/batman-adv/network-coding.c
index e3baf69..a7b5cf0 100644
--- a/net/batman-adv/network-coding.c
+++ b/net/batman-adv/network-coding.c
@@ -845,24 +845,6 @@ batadv_nc_get_nc_node(struct batadv_priv *bat_priv,
spinlock_t *lock; /* Used to lock list selected by "int in_coding" */
struct list_head *list;
- /* Check if nc_node is already added */
- nc_node = batadv_nc_find_nc_node(orig_node, orig_neigh_node, in_coding);
-
- /* Node found */
- if (nc_node)
- return nc_node;
-
- nc_node = kzalloc(sizeof(*nc_node), GFP_ATOMIC);
- if (!nc_node)
- return NULL;
-
- /* Initialize nc_node */
- INIT_LIST_HEAD(&nc_node->list);
- kref_init(&nc_node->refcount);
- ether_addr_copy(nc_node->addr, orig_node->orig);
- kref_get(&orig_neigh_node->refcount);
- nc_node->orig_node = orig_neigh_node;
-
/* Select ingoing or outgoing coding node */
if (in_coding) {
lock = &orig_neigh_node->in_coding_list_lock;
@@ -872,13 +854,34 @@ batadv_nc_get_nc_node(struct batadv_priv *bat_priv,
list = &orig_neigh_node->out_coding_list;
}
+ spin_lock_bh(lock);
+
+ /* Check if nc_node is already added */
+ nc_node = batadv_nc_find_nc_node(orig_node, orig_neigh_node, in_coding);
+
+ /* Node found */
+ if (nc_node)
+ goto unlock;
+
+ nc_node = kzalloc(sizeof(*nc_node), GFP_ATOMIC);
+ if (!nc_node)
+ goto unlock;
+
+ /* Initialize nc_node */
+ INIT_LIST_HEAD(&nc_node->list);
+ kref_init(&nc_node->refcount);
+ ether_addr_copy(nc_node->addr, orig_node->orig);
+ kref_get(&orig_neigh_node->refcount);
+ nc_node->orig_node = orig_neigh_node;
+
batadv_dbg(BATADV_DBG_NC, bat_priv, "Adding nc_node %pM -> %pM\n",
nc_node->addr, nc_node->orig_node->orig);
/* Add nc_node to orig_node */
- spin_lock_bh(lock);
kref_get(&nc_node->refcount);
list_add_tail_rcu(&nc_node->list, list);
+
+unlock:
spin_unlock_bh(lock);
return nc_node;
diff --git a/net/batman-adv/routing.c b/net/batman-adv/routing.c
index 7e8dc64..8b98609 100644
--- a/net/batman-adv/routing.c
+++ b/net/batman-adv/routing.c
@@ -724,6 +724,7 @@ static int batadv_route_unicast_packet(struct sk_buff *skb,
/**
* batadv_reroute_unicast_packet - update the unicast header for re-routing
* @bat_priv: the bat priv with all the soft interface information
+ * @skb: unicast packet to process
* @unicast_packet: the unicast header to be updated
* @dst_addr: the payload destination
* @vid: VLAN identifier
@@ -735,7 +736,7 @@ static int batadv_route_unicast_packet(struct sk_buff *skb,
* Return: true if the packet header has been updated, false otherwise
*/
static bool
-batadv_reroute_unicast_packet(struct batadv_priv *bat_priv,
+batadv_reroute_unicast_packet(struct batadv_priv *bat_priv, struct sk_buff *skb,
struct batadv_unicast_packet *unicast_packet,
u8 *dst_addr, unsigned short vid)
{
@@ -764,8 +765,10 @@ batadv_reroute_unicast_packet(struct batadv_priv *bat_priv,
}
/* update the packet header */
+ skb_postpull_rcsum(skb, unicast_packet, sizeof(*unicast_packet));
ether_addr_copy(unicast_packet->dest, orig_addr);
unicast_packet->ttvn = orig_ttvn;
+ skb_postpush_rcsum(skb, unicast_packet, sizeof(*unicast_packet));
ret = true;
out:
@@ -806,7 +809,7 @@ static bool batadv_check_unicast_ttvn(struct batadv_priv *bat_priv,
* the packet to
*/
if (batadv_tt_local_client_is_roaming(bat_priv, ethhdr->h_dest, vid)) {
- if (batadv_reroute_unicast_packet(bat_priv, unicast_packet,
+ if (batadv_reroute_unicast_packet(bat_priv, skb, unicast_packet,
ethhdr->h_dest, vid))
batadv_dbg_ratelimited(BATADV_DBG_TT,
bat_priv,
@@ -852,7 +855,7 @@ static bool batadv_check_unicast_ttvn(struct batadv_priv *bat_priv,
* destination can possibly be updated and forwarded towards the new
* target host
*/
- if (batadv_reroute_unicast_packet(bat_priv, unicast_packet,
+ if (batadv_reroute_unicast_packet(bat_priv, skb, unicast_packet,
ethhdr->h_dest, vid)) {
batadv_dbg_ratelimited(BATADV_DBG_TT, bat_priv,
"Rerouting unicast packet to %pM (dst=%pM): TTVN mismatch old_ttvn=%u new_ttvn=%u\n",
@@ -875,12 +878,14 @@ static bool batadv_check_unicast_ttvn(struct batadv_priv *bat_priv,
if (!primary_if)
return false;
+ /* update the packet header */
+ skb_postpull_rcsum(skb, unicast_packet, sizeof(*unicast_packet));
ether_addr_copy(unicast_packet->dest, primary_if->net_dev->dev_addr);
+ unicast_packet->ttvn = curr_ttvn;
+ skb_postpush_rcsum(skb, unicast_packet, sizeof(*unicast_packet));
batadv_hardif_put(primary_if);
- unicast_packet->ttvn = curr_ttvn;
-
return true;
}
diff --git a/net/batman-adv/soft-interface.c b/net/batman-adv/soft-interface.c
index 49e16b6..05bc176 100644
--- a/net/batman-adv/soft-interface.c
+++ b/net/batman-adv/soft-interface.c
@@ -448,13 +448,7 @@ void batadv_interface_rx(struct net_device *soft_iface,
/* skb->dev & skb->pkt_type are set here */
skb->protocol = eth_type_trans(skb, soft_iface);
-
- /* should not be necessary anymore as we use skb_pull_rcsum()
- * TODO: please verify this and remove this TODO
- * -- Dec 21st 2009, Simon Wunderlich
- */
-
- /* skb->ip_summed = CHECKSUM_UNNECESSARY; */
+ skb_postpull_rcsum(skb, eth_hdr(skb), ETH_HLEN);
batadv_inc_counter(bat_priv, BATADV_CNT_RX);
batadv_add_counter(bat_priv, BATADV_CNT_RX_BYTES,
@@ -571,15 +565,20 @@ int batadv_softif_create_vlan(struct batadv_priv *bat_priv, unsigned short vid)
struct batadv_softif_vlan *vlan;
int err;
+ spin_lock_bh(&bat_priv->softif_vlan_list_lock);
+
vlan = batadv_softif_vlan_get(bat_priv, vid);
if (vlan) {
batadv_softif_vlan_put(vlan);
+ spin_unlock_bh(&bat_priv->softif_vlan_list_lock);
return -EEXIST;
}
vlan = kzalloc(sizeof(*vlan), GFP_ATOMIC);
- if (!vlan)
+ if (!vlan) {
+ spin_unlock_bh(&bat_priv->softif_vlan_list_lock);
return -ENOMEM;
+ }
vlan->bat_priv = bat_priv;
vlan->vid = vid;
@@ -587,17 +586,23 @@ int batadv_softif_create_vlan(struct batadv_priv *bat_priv, unsigned short vid)
atomic_set(&vlan->ap_isolation, 0);
- err = batadv_sysfs_add_vlan(bat_priv->soft_iface, vlan);
- if (err) {
- kfree(vlan);
- return err;
- }
-
- spin_lock_bh(&bat_priv->softif_vlan_list_lock);
kref_get(&vlan->refcount);
hlist_add_head_rcu(&vlan->list, &bat_priv->softif_vlan_list);
spin_unlock_bh(&bat_priv->softif_vlan_list_lock);
+ /* batadv_sysfs_add_vlan cannot be in the spinlock section due to the
+ * sleeping behavior of the sysfs functions and the fs_reclaim lock
+ */
+ err = batadv_sysfs_add_vlan(bat_priv->soft_iface, vlan);
+ if (err) {
+ /* ref for the function */
+ batadv_softif_vlan_put(vlan);
+
+ /* ref for the list */
+ batadv_softif_vlan_put(vlan);
+ return err;
+ }
+
/* add a new TT local entry. This one will be marked with the NOPURGE
* flag
*/
diff --git a/net/batman-adv/sysfs.c b/net/batman-adv/sysfs.c
index 02d96f2..31d7e23 100644
--- a/net/batman-adv/sysfs.c
+++ b/net/batman-adv/sysfs.c
@@ -187,7 +187,8 @@ ssize_t batadv_store_##_name(struct kobject *kobj, \
\
return __batadv_store_uint_attr(buff, count, _min, _max, \
_post_func, attr, \
- &bat_priv->_var, net_dev); \
+ &bat_priv->_var, net_dev, \
+ NULL); \
}
#define BATADV_ATTR_SIF_SHOW_UINT(_name, _var) \
@@ -261,7 +262,9 @@ ssize_t batadv_store_##_name(struct kobject *kobj, \
\
length = __batadv_store_uint_attr(buff, count, _min, _max, \
_post_func, attr, \
- &hard_iface->_var, net_dev); \
+ &hard_iface->_var, \
+ hard_iface->soft_iface, \
+ net_dev); \
\
batadv_hardif_put(hard_iface); \
return length; \
@@ -355,10 +358,12 @@ __batadv_store_bool_attr(char *buff, size_t count,
static int batadv_store_uint_attr(const char *buff, size_t count,
struct net_device *net_dev,
+ struct net_device *slave_dev,
const char *attr_name,
unsigned int min, unsigned int max,
atomic_t *attr)
{
+ char ifname[IFNAMSIZ + 3] = "";
unsigned long uint_val;
int ret;
@@ -384,8 +389,11 @@ static int batadv_store_uint_attr(const char *buff, size_t count,
if (atomic_read(attr) == uint_val)
return count;
- batadv_info(net_dev, "%s: Changing from: %i to: %lu\n",
- attr_name, atomic_read(attr), uint_val);
+ if (slave_dev)
+ snprintf(ifname, sizeof(ifname), "%s: ", slave_dev->name);
+
+ batadv_info(net_dev, "%s: %sChanging from: %i to: %lu\n",
+ attr_name, ifname, atomic_read(attr), uint_val);
atomic_set(attr, uint_val);
return count;
@@ -396,12 +404,13 @@ static ssize_t __batadv_store_uint_attr(const char *buff, size_t count,
void (*post_func)(struct net_device *),
const struct attribute *attr,
atomic_t *attr_store,
- struct net_device *net_dev)
+ struct net_device *net_dev,
+ struct net_device *slave_dev)
{
int ret;
- ret = batadv_store_uint_attr(buff, count, net_dev, attr->name, min, max,
- attr_store);
+ ret = batadv_store_uint_attr(buff, count, net_dev, slave_dev,
+ attr->name, min, max, attr_store);
if (post_func && ret)
post_func(net_dev);
@@ -570,7 +579,7 @@ static ssize_t batadv_store_gw_sel_class(struct kobject *kobj,
return __batadv_store_uint_attr(buff, count, 1, BATADV_TQ_MAX_VALUE,
batadv_post_gw_reselect, attr,
&bat_priv->gw.sel_class,
- bat_priv->soft_iface);
+ bat_priv->soft_iface, NULL);
}
static ssize_t batadv_show_gw_bwidth(struct kobject *kobj,
@@ -1084,8 +1093,9 @@ static ssize_t batadv_store_throughput_override(struct kobject *kobj,
if (old_tp_override == tp_override)
goto out;
- batadv_info(net_dev, "%s: Changing from: %u.%u MBit to: %u.%u MBit\n",
- "throughput_override",
+ batadv_info(hard_iface->soft_iface,
+ "%s: %s: Changing from: %u.%u MBit to: %u.%u MBit\n",
+ "throughput_override", net_dev->name,
old_tp_override / 10, old_tp_override % 10,
tp_override / 10, tp_override % 10);
diff --git a/net/batman-adv/translation-table.c b/net/batman-adv/translation-table.c
index 0dc85eb..b9f9a31 100644
--- a/net/batman-adv/translation-table.c
+++ b/net/batman-adv/translation-table.c
@@ -1550,6 +1550,8 @@ batadv_tt_global_orig_entry_add(struct batadv_tt_global_entry *tt_global,
{
struct batadv_tt_orig_list_entry *orig_entry;
+ spin_lock_bh(&tt_global->list_lock);
+
orig_entry = batadv_tt_global_orig_entry_find(tt_global, orig_node);
if (orig_entry) {
/* refresh the ttvn: the current value could be a bogus one that
@@ -1570,16 +1572,16 @@ batadv_tt_global_orig_entry_add(struct batadv_tt_global_entry *tt_global,
orig_entry->ttvn = ttvn;
kref_init(&orig_entry->refcount);
- spin_lock_bh(&tt_global->list_lock);
kref_get(&orig_entry->refcount);
hlist_add_head_rcu(&orig_entry->list,
&tt_global->orig_list);
- spin_unlock_bh(&tt_global->list_lock);
atomic_inc(&tt_global->orig_list_count);
out:
if (orig_entry)
batadv_tt_orig_list_entry_put(orig_entry);
+
+ spin_unlock_bh(&tt_global->list_lock);
}
/**
diff --git a/net/batman-adv/tvlv.c b/net/batman-adv/tvlv.c
index 77654f0..8e91a26 100644
--- a/net/batman-adv/tvlv.c
+++ b/net/batman-adv/tvlv.c
@@ -528,15 +528,20 @@ void batadv_tvlv_handler_register(struct batadv_priv *bat_priv,
{
struct batadv_tvlv_handler *tvlv_handler;
+ spin_lock_bh(&bat_priv->tvlv.handler_list_lock);
+
tvlv_handler = batadv_tvlv_handler_get(bat_priv, type, version);
if (tvlv_handler) {
+ spin_unlock_bh(&bat_priv->tvlv.handler_list_lock);
batadv_tvlv_handler_put(tvlv_handler);
return;
}
tvlv_handler = kzalloc(sizeof(*tvlv_handler), GFP_ATOMIC);
- if (!tvlv_handler)
+ if (!tvlv_handler) {
+ spin_unlock_bh(&bat_priv->tvlv.handler_list_lock);
return;
+ }
tvlv_handler->ogm_handler = optr;
tvlv_handler->unicast_handler = uptr;
@@ -546,7 +551,6 @@ void batadv_tvlv_handler_register(struct batadv_priv *bat_priv,
kref_init(&tvlv_handler->refcount);
INIT_HLIST_NODE(&tvlv_handler->list);
- spin_lock_bh(&bat_priv->tvlv.handler_list_lock);
kref_get(&tvlv_handler->refcount);
hlist_add_head_rcu(&tvlv_handler->list, &bat_priv->tvlv.handler_list);
spin_unlock_bh(&bat_priv->tvlv.handler_list_lock);
diff --git a/net/bluetooth/hidp/core.c b/net/bluetooth/hidp/core.c
index 1811f8e..552e00b0 100644
--- a/net/bluetooth/hidp/core.c
+++ b/net/bluetooth/hidp/core.c
@@ -774,7 +774,7 @@ static int hidp_setup_hid(struct hidp_session *session,
hid->version = req->version;
hid->country = req->country;
- strncpy(hid->name, req->name, sizeof(req->name) - 1);
+ strncpy(hid->name, req->name, sizeof(hid->name));
snprintf(hid->phys, sizeof(hid->phys), "%pMR",
&l2cap_pi(session->ctrl_sock->sk)->chan->src);
diff --git a/net/bluetooth/mgmt.c b/net/bluetooth/mgmt.c
index 1fba2a0..ba24f61 100644
--- a/net/bluetooth/mgmt.c
+++ b/net/bluetooth/mgmt.c
@@ -2298,9 +2298,8 @@ static int unpair_device(struct sock *sk, struct hci_dev *hdev, void *data,
/* LE address type */
addr_type = le_addr_type(cp->addr.type);
- hci_remove_irk(hdev, &cp->addr.bdaddr, addr_type);
-
- err = hci_remove_ltk(hdev, &cp->addr.bdaddr, addr_type);
+ /* Abort any ongoing SMP pairing. Removes ltk and irk if they exist. */
+ err = smp_cancel_and_remove_pairing(hdev, &cp->addr.bdaddr, addr_type);
if (err < 0) {
err = mgmt_cmd_complete(sk, hdev->id, MGMT_OP_UNPAIR_DEVICE,
MGMT_STATUS_NOT_PAIRED, &rp,
@@ -2314,8 +2313,6 @@ static int unpair_device(struct sock *sk, struct hci_dev *hdev, void *data,
goto done;
}
- /* Abort any ongoing SMP pairing */
- smp_cancel_pairing(conn);
/* Defer clearing up the connection parameters until closing to
* give a chance of keeping them if a repairing happens.
diff --git a/net/bluetooth/sco.c b/net/bluetooth/sco.c
index 3125ce6..95fd7a8 100644
--- a/net/bluetooth/sco.c
+++ b/net/bluetooth/sco.c
@@ -392,7 +392,8 @@ static void sco_sock_cleanup_listen(struct sock *parent)
*/
static void sco_sock_kill(struct sock *sk)
{
- if (!sock_flag(sk, SOCK_ZAPPED) || sk->sk_socket)
+ if (!sock_flag(sk, SOCK_ZAPPED) || sk->sk_socket ||
+ sock_flag(sk, SOCK_DEAD))
return;
BT_DBG("sk %p state %d", sk, sk->sk_state);
diff --git a/net/bluetooth/smp.c b/net/bluetooth/smp.c
index ead4d1b..1abfbcd8 100644
--- a/net/bluetooth/smp.c
+++ b/net/bluetooth/smp.c
@@ -2353,30 +2353,51 @@ int smp_conn_security(struct hci_conn *hcon, __u8 sec_level)
return ret;
}
-void smp_cancel_pairing(struct hci_conn *hcon)
+int smp_cancel_and_remove_pairing(struct hci_dev *hdev, bdaddr_t *bdaddr,
+ u8 addr_type)
{
- struct l2cap_conn *conn = hcon->l2cap_data;
+ struct hci_conn *hcon;
+ struct l2cap_conn *conn;
struct l2cap_chan *chan;
struct smp_chan *smp;
+ int err;
+ err = hci_remove_ltk(hdev, bdaddr, addr_type);
+ hci_remove_irk(hdev, bdaddr, addr_type);
+
+ hcon = hci_conn_hash_lookup_le(hdev, bdaddr, addr_type);
+ if (!hcon)
+ goto done;
+
+ conn = hcon->l2cap_data;
if (!conn)
- return;
+ goto done;
chan = conn->smp;
if (!chan)
- return;
+ goto done;
l2cap_chan_lock(chan);
smp = chan->data;
if (smp) {
+ /* Set keys to NULL to make sure smp_failure() does not try to
+ * remove and free already invalidated rcu list entries. */
+ smp->ltk = NULL;
+ smp->slave_ltk = NULL;
+ smp->remote_irk = NULL;
+
if (test_bit(SMP_FLAG_COMPLETE, &smp->flags))
smp_failure(conn, 0);
else
smp_failure(conn, SMP_UNSPECIFIED);
+ err = 0;
}
l2cap_chan_unlock(chan);
+
+done:
+ return err;
}
static int smp_cmd_encrypt_info(struct l2cap_conn *conn, struct sk_buff *skb)
diff --git a/net/bluetooth/smp.h b/net/bluetooth/smp.h
index ffcc70b..993cbd7 100644
--- a/net/bluetooth/smp.h
+++ b/net/bluetooth/smp.h
@@ -180,7 +180,8 @@ enum smp_key_pref {
};
/* SMP Commands */
-void smp_cancel_pairing(struct hci_conn *hcon);
+int smp_cancel_and_remove_pairing(struct hci_dev *hdev, bdaddr_t *bdaddr,
+ u8 addr_type);
bool smp_sufficient_security(struct hci_conn *hcon, u8 sec_level,
enum smp_key_pref key_pref);
int smp_conn_security(struct hci_conn *hcon, __u8 sec_level);
diff --git a/net/bridge/br_if.c b/net/bridge/br_if.c
index 9218931..f57de0a 100644
--- a/net/bridge/br_if.c
+++ b/net/bridge/br_if.c
@@ -504,8 +504,8 @@ int br_add_if(struct net_bridge *br, struct net_device *dev)
if (dev->netdev_ops->ndo_start_xmit == br_dev_xmit)
return -ELOOP;
- /* Device is already being bridged */
- if (br_port_exists(dev))
+ /* Device has master upper dev */
+ if (netdev_master_upper_dev_get(dev))
return -EBUSY;
/* No bridging devices that dislike that (e.g. wireless) */
diff --git a/net/bridge/br_multicast.c b/net/bridge/br_multicast.c
index 2136e45f..4bd5750 100644
--- a/net/bridge/br_multicast.c
+++ b/net/bridge/br_multicast.c
@@ -1287,7 +1287,14 @@ static void br_multicast_query_received(struct net_bridge *br,
return;
br_multicast_update_query_timer(br, query, max_delay);
- br_multicast_mark_router(br, port);
+
+ /* Based on RFC4541, section 2.1.1 IGMP Forwarding Rules,
+ * the arrival port for IGMP Queries where the source address
+ * is 0.0.0.0 should not be added to router port list.
+ */
+ if ((saddr->proto == htons(ETH_P_IP) && saddr->u.ip4) ||
+ saddr->proto == htons(ETH_P_IPV6))
+ br_multicast_mark_router(br, port);
}
static int br_ip4_multicast_query(struct net_bridge *br,
diff --git a/net/bridge/netfilter/ebt_arpreply.c b/net/bridge/netfilter/ebt_arpreply.c
index 070cf13..f2660c1 100644
--- a/net/bridge/netfilter/ebt_arpreply.c
+++ b/net/bridge/netfilter/ebt_arpreply.c
@@ -67,6 +67,9 @@ static int ebt_arpreply_tg_check(const struct xt_tgchk_param *par)
if (e->ethproto != htons(ETH_P_ARP) ||
e->invflags & EBT_IPROTO)
return -EINVAL;
+ if (ebt_invalid_target(info->target))
+ return -EINVAL;
+
return 0;
}
diff --git a/net/bridge/netfilter/ebtables.c b/net/bridge/netfilter/ebtables.c
index 5a89a4a..18c1f07 100644
--- a/net/bridge/netfilter/ebtables.c
+++ b/net/bridge/netfilter/ebtables.c
@@ -406,6 +406,12 @@ ebt_check_watcher(struct ebt_entry_watcher *w, struct xt_tgchk_param *par,
watcher = xt_request_find_target(NFPROTO_BRIDGE, w->u.name, 0);
if (IS_ERR(watcher))
return PTR_ERR(watcher);
+
+ if (watcher->family != NFPROTO_BRIDGE) {
+ module_put(watcher->me);
+ return -ENOENT;
+ }
+
w->u.watcher = watcher;
par->target = watcher;
@@ -704,6 +710,8 @@ ebt_check_entry(struct ebt_entry *e, struct net *net,
}
i = 0;
+ memset(&mtpar, 0, sizeof(mtpar));
+ memset(&tgpar, 0, sizeof(tgpar));
mtpar.net = tgpar.net = net;
mtpar.table = tgpar.table = name;
mtpar.entryinfo = tgpar.entryinfo = e;
@@ -725,6 +733,13 @@ ebt_check_entry(struct ebt_entry *e, struct net *net,
goto cleanup_watchers;
}
+ /* Reject UNSPEC, xtables verdicts/return values are incompatible */
+ if (target->family != NFPROTO_BRIDGE) {
+ module_put(target->me);
+ ret = -ENOENT;
+ goto cleanup_watchers;
+ }
+
t->u.target = target;
if (t->u.target == &ebt_standard_target) {
if (gap < sizeof(struct ebt_standard_target)) {
@@ -1625,7 +1640,8 @@ static int compat_match_to_user(struct ebt_entry_match *m, void __user **dstptr,
int off = ebt_compat_match_offset(match, m->match_size);
compat_uint_t msize = m->match_size - off;
- BUG_ON(off >= m->match_size);
+ if (WARN_ON(off >= m->match_size))
+ return -EINVAL;
if (copy_to_user(cm->u.name, match->name,
strlen(match->name) + 1) || put_user(msize, &cm->match_size))
@@ -1652,7 +1668,8 @@ static int compat_target_to_user(struct ebt_entry_target *t,
int off = xt_compat_target_offset(target);
compat_uint_t tsize = t->target_size - off;
- BUG_ON(off >= t->target_size);
+ if (WARN_ON(off >= t->target_size))
+ return -EINVAL;
if (copy_to_user(cm->u.name, target->name,
strlen(target->name) + 1) || put_user(tsize, &cm->match_size))
@@ -1880,7 +1897,8 @@ static int ebt_buf_add(struct ebt_entries_buf_state *state,
if (state->buf_kern_start == NULL)
goto count_only;
- BUG_ON(state->buf_kern_offset + sz > state->buf_kern_len);
+ if (WARN_ON(state->buf_kern_offset + sz > state->buf_kern_len))
+ return -EINVAL;
memcpy(state->buf_kern_start + state->buf_kern_offset, data, sz);
@@ -1893,7 +1911,8 @@ static int ebt_buf_add_pad(struct ebt_entries_buf_state *state, unsigned int sz)
{
char *b = state->buf_kern_start;
- BUG_ON(b && state->buf_kern_offset > state->buf_kern_len);
+ if (WARN_ON(b && state->buf_kern_offset > state->buf_kern_len))
+ return -EINVAL;
if (b != NULL && sz > 0)
memset(b + state->buf_kern_offset, 0, sz);
@@ -1919,7 +1938,8 @@ static int compat_mtw_from_user(struct compat_ebt_entry_mwt *mwt,
int off, pad = 0;
unsigned int size_kern, match_size = mwt->match_size;
- strlcpy(name, mwt->u.name, sizeof(name));
+ if (strscpy(name, mwt->u.name, sizeof(name)) < 0)
+ return -EINVAL;
if (state->buf_kern_start)
dst = state->buf_kern_start + state->buf_kern_offset;
@@ -1970,8 +1990,10 @@ static int compat_mtw_from_user(struct compat_ebt_entry_mwt *mwt,
pad = XT_ALIGN(size_kern) - size_kern;
if (pad > 0 && dst) {
- BUG_ON(state->buf_kern_len <= pad);
- BUG_ON(state->buf_kern_offset - (match_size + off) + size_kern > state->buf_kern_len - pad);
+ if (WARN_ON(state->buf_kern_len <= pad))
+ return -EINVAL;
+ if (WARN_ON(state->buf_kern_offset - (match_size + off) + size_kern > state->buf_kern_len - pad))
+ return -EINVAL;
memset(dst + size_kern, 0, pad);
}
return off + match_size;
@@ -2021,7 +2043,8 @@ static int ebt_size_mwt(struct compat_ebt_entry_mwt *match32,
if (ret < 0)
return ret;
- BUG_ON(ret < match32->match_size);
+ if (WARN_ON(ret < match32->match_size))
+ return -EINVAL;
growth += ret - match32->match_size;
growth += ebt_compat_entry_padsize();
@@ -2090,8 +2113,12 @@ static int size_entry_mwt(struct ebt_entry *entry, const unsigned char *base,
* offsets are relative to beginning of struct ebt_entry (i.e., 0).
*/
for (i = 0; i < 4 ; ++i) {
- if (offsets[i] >= *total)
+ if (offsets[i] > *total)
return -EINVAL;
+
+ if (i < 3 && offsets[i] == *total)
+ return -EINVAL;
+
if (i == 0)
continue;
if (offsets[i-1] > offsets[i])
@@ -2130,7 +2157,8 @@ static int size_entry_mwt(struct ebt_entry *entry, const unsigned char *base,
startoff = state->buf_user_offset - startoff;
- BUG_ON(*total < startoff);
+ if (WARN_ON(*total < startoff))
+ return -EINVAL;
*total -= startoff;
return 0;
}
@@ -2257,7 +2285,8 @@ static int compat_do_replace(struct net *net, void __user *user,
state.buf_kern_len = size64;
ret = compat_copy_entries(entries_tmp, tmp.entries_size, &state);
- BUG_ON(ret < 0); /* parses same data again */
+ if (WARN_ON(ret < 0))
+ goto out_unlock;
vfree(entries_tmp);
tmp.entries_size = size64;
diff --git a/net/caif/caif_dev.c b/net/caif/caif_dev.c
index d730a0f..a0443d4 100644
--- a/net/caif/caif_dev.c
+++ b/net/caif/caif_dev.c
@@ -131,8 +131,10 @@ static void caif_flow_cb(struct sk_buff *skb)
caifd = caif_get(skb->dev);
WARN_ON(caifd == NULL);
- if (caifd == NULL)
+ if (!caifd) {
+ rcu_read_unlock();
return;
+ }
caifd_hold(caifd);
rcu_read_unlock();
diff --git a/net/ceph/auth.c b/net/ceph/auth.c
index c822b3ae..8e79dca 100644
--- a/net/ceph/auth.c
+++ b/net/ceph/auth.c
@@ -314,14 +314,30 @@ int ceph_auth_update_authorizer(struct ceph_auth_client *ac,
}
EXPORT_SYMBOL(ceph_auth_update_authorizer);
+int ceph_auth_add_authorizer_challenge(struct ceph_auth_client *ac,
+ struct ceph_authorizer *a,
+ void *challenge_buf,
+ int challenge_buf_len)
+{
+ int ret = 0;
+
+ mutex_lock(&ac->mutex);
+ if (ac->ops && ac->ops->add_authorizer_challenge)
+ ret = ac->ops->add_authorizer_challenge(ac, a, challenge_buf,
+ challenge_buf_len);
+ mutex_unlock(&ac->mutex);
+ return ret;
+}
+EXPORT_SYMBOL(ceph_auth_add_authorizer_challenge);
+
int ceph_auth_verify_authorizer_reply(struct ceph_auth_client *ac,
- struct ceph_authorizer *a, size_t len)
+ struct ceph_authorizer *a)
{
int ret = 0;
mutex_lock(&ac->mutex);
if (ac->ops && ac->ops->verify_authorizer_reply)
- ret = ac->ops->verify_authorizer_reply(ac, a, len);
+ ret = ac->ops->verify_authorizer_reply(ac, a);
mutex_unlock(&ac->mutex);
return ret;
}
diff --git a/net/ceph/auth_x.c b/net/ceph/auth_x.c
index b216131..a4896e4 100644
--- a/net/ceph/auth_x.c
+++ b/net/ceph/auth_x.c
@@ -8,6 +8,7 @@
#include <linux/ceph/decode.h>
#include <linux/ceph/auth.h>
+#include <linux/ceph/ceph_features.h>
#include <linux/ceph/libceph.h>
#include <linux/ceph/messenger.h>
@@ -69,25 +70,40 @@ static int ceph_x_encrypt(struct ceph_crypto_key *secret, void *buf,
return sizeof(u32) + ciphertext_len;
}
+static int __ceph_x_decrypt(struct ceph_crypto_key *secret, void *p,
+ int ciphertext_len)
+{
+ struct ceph_x_encrypt_header *hdr = p;
+ int plaintext_len;
+ int ret;
+
+ ret = ceph_crypt(secret, false, p, ciphertext_len, ciphertext_len,
+ &plaintext_len);
+ if (ret)
+ return ret;
+
+ if (le64_to_cpu(hdr->magic) != CEPHX_ENC_MAGIC) {
+ pr_err("%s bad magic\n", __func__);
+ return -EINVAL;
+ }
+
+ return plaintext_len - sizeof(*hdr);
+}
+
static int ceph_x_decrypt(struct ceph_crypto_key *secret, void **p, void *end)
{
- struct ceph_x_encrypt_header *hdr = *p + sizeof(u32);
- int ciphertext_len, plaintext_len;
+ int ciphertext_len;
int ret;
ceph_decode_32_safe(p, end, ciphertext_len, e_inval);
ceph_decode_need(p, end, ciphertext_len, e_inval);
- ret = ceph_crypt(secret, false, *p, end - *p, ciphertext_len,
- &plaintext_len);
- if (ret)
+ ret = __ceph_x_decrypt(secret, *p, ciphertext_len);
+ if (ret < 0)
return ret;
- if (hdr->struct_v != 1 || le64_to_cpu(hdr->magic) != CEPHX_ENC_MAGIC)
- return -EPERM;
-
*p += ciphertext_len;
- return plaintext_len - sizeof(struct ceph_x_encrypt_header);
+ return ret;
e_inval:
return -EINVAL;
@@ -271,6 +287,51 @@ static int ceph_x_proc_ticket_reply(struct ceph_auth_client *ac,
return -EINVAL;
}
+/*
+ * Encode and encrypt the second part (ceph_x_authorize_b) of the
+ * authorizer. The first part (ceph_x_authorize_a) should already be
+ * encoded.
+ */
+static int encrypt_authorizer(struct ceph_x_authorizer *au,
+ u64 *server_challenge)
+{
+ struct ceph_x_authorize_a *msg_a;
+ struct ceph_x_authorize_b *msg_b;
+ void *p, *end;
+ int ret;
+
+ msg_a = au->buf->vec.iov_base;
+ WARN_ON(msg_a->ticket_blob.secret_id != cpu_to_le64(au->secret_id));
+ p = (void *)(msg_a + 1) + le32_to_cpu(msg_a->ticket_blob.blob_len);
+ end = au->buf->vec.iov_base + au->buf->vec.iov_len;
+
+ msg_b = p + ceph_x_encrypt_offset();
+ msg_b->struct_v = 2;
+ msg_b->nonce = cpu_to_le64(au->nonce);
+ if (server_challenge) {
+ msg_b->have_challenge = 1;
+ msg_b->server_challenge_plus_one =
+ cpu_to_le64(*server_challenge + 1);
+ } else {
+ msg_b->have_challenge = 0;
+ msg_b->server_challenge_plus_one = 0;
+ }
+
+ ret = ceph_x_encrypt(&au->session_key, p, end - p, sizeof(*msg_b));
+ if (ret < 0)
+ return ret;
+
+ p += ret;
+ if (server_challenge) {
+ WARN_ON(p != end);
+ } else {
+ WARN_ON(p > end);
+ au->buf->vec.iov_len = p - au->buf->vec.iov_base;
+ }
+
+ return 0;
+}
+
static void ceph_x_authorizer_cleanup(struct ceph_x_authorizer *au)
{
ceph_crypto_key_destroy(&au->session_key);
@@ -287,7 +348,6 @@ static int ceph_x_build_authorizer(struct ceph_auth_client *ac,
int maxlen;
struct ceph_x_authorize_a *msg_a;
struct ceph_x_authorize_b *msg_b;
- void *p, *end;
int ret;
int ticket_blob_len =
(th->ticket_blob ? th->ticket_blob->vec.iov_len : 0);
@@ -331,21 +391,13 @@ static int ceph_x_build_authorizer(struct ceph_auth_client *ac,
dout(" th %p secret_id %lld %lld\n", th, th->secret_id,
le64_to_cpu(msg_a->ticket_blob.secret_id));
- p = msg_a + 1;
- p += ticket_blob_len;
- end = au->buf->vec.iov_base + au->buf->vec.iov_len;
-
- msg_b = p + ceph_x_encrypt_offset();
- msg_b->struct_v = 1;
get_random_bytes(&au->nonce, sizeof(au->nonce));
- msg_b->nonce = cpu_to_le64(au->nonce);
- ret = ceph_x_encrypt(&au->session_key, p, end - p, sizeof(*msg_b));
- if (ret < 0)
+ ret = encrypt_authorizer(au, NULL);
+ if (ret) {
+ pr_err("failed to encrypt authorizer: %d", ret);
goto out_au;
+ }
- p += ret;
- WARN_ON(p > end);
- au->buf->vec.iov_len = p - au->buf->vec.iov_base;
dout(" built authorizer nonce %llx len %d\n", au->nonce,
(int)au->buf->vec.iov_len);
return 0;
@@ -622,8 +674,56 @@ static int ceph_x_update_authorizer(
return 0;
}
+static int decrypt_authorize_challenge(struct ceph_x_authorizer *au,
+ void *challenge_buf,
+ int challenge_buf_len,
+ u64 *server_challenge)
+{
+ struct ceph_x_authorize_challenge *ch =
+ challenge_buf + sizeof(struct ceph_x_encrypt_header);
+ int ret;
+
+ /* no leading len */
+ ret = __ceph_x_decrypt(&au->session_key, challenge_buf,
+ challenge_buf_len);
+ if (ret < 0)
+ return ret;
+ if (ret < sizeof(*ch)) {
+ pr_err("bad size %d for ceph_x_authorize_challenge\n", ret);
+ return -EINVAL;
+ }
+
+ *server_challenge = le64_to_cpu(ch->server_challenge);
+ return 0;
+}
+
+static int ceph_x_add_authorizer_challenge(struct ceph_auth_client *ac,
+ struct ceph_authorizer *a,
+ void *challenge_buf,
+ int challenge_buf_len)
+{
+ struct ceph_x_authorizer *au = (void *)a;
+ u64 server_challenge;
+ int ret;
+
+ ret = decrypt_authorize_challenge(au, challenge_buf, challenge_buf_len,
+ &server_challenge);
+ if (ret) {
+ pr_err("failed to decrypt authorize challenge: %d", ret);
+ return ret;
+ }
+
+ ret = encrypt_authorizer(au, &server_challenge);
+ if (ret) {
+ pr_err("failed to encrypt authorizer w/ challenge: %d", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
static int ceph_x_verify_authorizer_reply(struct ceph_auth_client *ac,
- struct ceph_authorizer *a, size_t len)
+ struct ceph_authorizer *a)
{
struct ceph_x_authorizer *au = (void *)a;
void *p = au->enc_buf;
@@ -633,8 +733,10 @@ static int ceph_x_verify_authorizer_reply(struct ceph_auth_client *ac,
ret = ceph_x_decrypt(&au->session_key, &p, p + CEPHX_AU_ENC_BUF_LEN);
if (ret < 0)
return ret;
- if (ret != sizeof(*reply))
- return -EPERM;
+ if (ret < sizeof(*reply)) {
+ pr_err("bad size %d for ceph_x_authorize_reply\n", ret);
+ return -EINVAL;
+ }
if (au->nonce + 1 != le64_to_cpu(reply->nonce_plus_one))
ret = -EPERM;
@@ -700,26 +802,64 @@ static int calc_signature(struct ceph_x_authorizer *au, struct ceph_msg *msg,
__le64 *psig)
{
void *enc_buf = au->enc_buf;
- struct {
- __le32 len;
- __le32 header_crc;
- __le32 front_crc;
- __le32 middle_crc;
- __le32 data_crc;
- } __packed *sigblock = enc_buf + ceph_x_encrypt_offset();
int ret;
- sigblock->len = cpu_to_le32(4*sizeof(u32));
- sigblock->header_crc = msg->hdr.crc;
- sigblock->front_crc = msg->footer.front_crc;
- sigblock->middle_crc = msg->footer.middle_crc;
- sigblock->data_crc = msg->footer.data_crc;
- ret = ceph_x_encrypt(&au->session_key, enc_buf, CEPHX_AU_ENC_BUF_LEN,
- sizeof(*sigblock));
- if (ret < 0)
- return ret;
+ if (!(msg->con->peer_features & CEPH_FEATURE_CEPHX_V2)) {
+ struct {
+ __le32 len;
+ __le32 header_crc;
+ __le32 front_crc;
+ __le32 middle_crc;
+ __le32 data_crc;
+ } __packed *sigblock = enc_buf + ceph_x_encrypt_offset();
- *psig = *(__le64 *)(enc_buf + sizeof(u32));
+ sigblock->len = cpu_to_le32(4*sizeof(u32));
+ sigblock->header_crc = msg->hdr.crc;
+ sigblock->front_crc = msg->footer.front_crc;
+ sigblock->middle_crc = msg->footer.middle_crc;
+ sigblock->data_crc = msg->footer.data_crc;
+
+ ret = ceph_x_encrypt(&au->session_key, enc_buf,
+ CEPHX_AU_ENC_BUF_LEN, sizeof(*sigblock));
+ if (ret < 0)
+ return ret;
+
+ *psig = *(__le64 *)(enc_buf + sizeof(u32));
+ } else {
+ struct {
+ __le32 header_crc;
+ __le32 front_crc;
+ __le32 front_len;
+ __le32 middle_crc;
+ __le32 middle_len;
+ __le32 data_crc;
+ __le32 data_len;
+ __le32 seq_lower_word;
+ } __packed *sigblock = enc_buf;
+ struct {
+ __le64 a, b, c, d;
+ } __packed *penc = enc_buf;
+ int ciphertext_len;
+
+ sigblock->header_crc = msg->hdr.crc;
+ sigblock->front_crc = msg->footer.front_crc;
+ sigblock->front_len = msg->hdr.front_len;
+ sigblock->middle_crc = msg->footer.middle_crc;
+ sigblock->middle_len = msg->hdr.middle_len;
+ sigblock->data_crc = msg->footer.data_crc;
+ sigblock->data_len = msg->hdr.data_len;
+ sigblock->seq_lower_word = *(__le32 *)&msg->hdr.seq;
+
+ /* no leading len, no ceph_x_encrypt_header */
+ ret = ceph_crypt(&au->session_key, true, enc_buf,
+ CEPHX_AU_ENC_BUF_LEN, sizeof(*sigblock),
+ &ciphertext_len);
+ if (ret)
+ return ret;
+
+ *psig = penc->a ^ penc->b ^ penc->c ^ penc->d;
+ }
+
return 0;
}
@@ -774,6 +914,7 @@ static const struct ceph_auth_client_ops ceph_x_ops = {
.handle_reply = ceph_x_handle_reply,
.create_authorizer = ceph_x_create_authorizer,
.update_authorizer = ceph_x_update_authorizer,
+ .add_authorizer_challenge = ceph_x_add_authorizer_challenge,
.verify_authorizer_reply = ceph_x_verify_authorizer_reply,
.invalidate_authorizer = ceph_x_invalidate_authorizer,
.reset = ceph_x_reset,
diff --git a/net/ceph/auth_x_protocol.h b/net/ceph/auth_x_protocol.h
index 671d305..a7cd203 100644
--- a/net/ceph/auth_x_protocol.h
+++ b/net/ceph/auth_x_protocol.h
@@ -69,6 +69,13 @@ struct ceph_x_authorize_a {
struct ceph_x_authorize_b {
__u8 struct_v;
__le64 nonce;
+ __u8 have_challenge;
+ __le64 server_challenge_plus_one;
+} __attribute__ ((packed));
+
+struct ceph_x_authorize_challenge {
+ __u8 struct_v;
+ __le64 server_challenge;
} __attribute__ ((packed));
struct ceph_x_authorize_reply {
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index 25a30be..5a8075d 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -588,9 +588,15 @@ static int ceph_tcp_sendpage(struct socket *sock, struct page *page,
int ret;
struct kvec iov;
- /* sendpage cannot properly handle pages with page_count == 0,
- * we need to fallback to sendmsg if that's the case */
- if (page_count(page) >= 1)
+ /*
+ * sendpage cannot properly handle pages with page_count == 0,
+ * we need to fall back to sendmsg if that's the case.
+ *
+ * Same goes for slab pages: skb_can_coalesce() allows
+ * coalescing neighboring slab objects into a single frag which
+ * triggers one of hardened usercopy checks.
+ */
+ if (page_count(page) >= 1 && !PageSlab(page))
return __ceph_tcp_sendpage(sock, page, offset, size, more);
iov.iov_base = kmap(page) + offset;
@@ -1388,30 +1394,26 @@ static void prepare_write_keepalive(struct ceph_connection *con)
* Connection negotiation.
*/
-static struct ceph_auth_handshake *get_connect_authorizer(struct ceph_connection *con,
- int *auth_proto)
+static int get_connect_authorizer(struct ceph_connection *con)
{
struct ceph_auth_handshake *auth;
+ int auth_proto;
if (!con->ops->get_authorizer) {
+ con->auth = NULL;
con->out_connect.authorizer_protocol = CEPH_AUTH_UNKNOWN;
con->out_connect.authorizer_len = 0;
- return NULL;
+ return 0;
}
- /* Can't hold the mutex while getting authorizer */
- mutex_unlock(&con->mutex);
- auth = con->ops->get_authorizer(con, auth_proto, con->auth_retry);
- mutex_lock(&con->mutex);
-
+ auth = con->ops->get_authorizer(con, &auth_proto, con->auth_retry);
if (IS_ERR(auth))
- return auth;
- if (con->state != CON_STATE_NEGOTIATING)
- return ERR_PTR(-EAGAIN);
+ return PTR_ERR(auth);
- con->auth_reply_buf = auth->authorizer_reply_buf;
- con->auth_reply_buf_len = auth->authorizer_reply_buf_len;
- return auth;
+ con->auth = auth;
+ con->out_connect.authorizer_protocol = cpu_to_le32(auth_proto);
+ con->out_connect.authorizer_len = cpu_to_le32(auth->authorizer_buf_len);
+ return 0;
}
/*
@@ -1427,12 +1429,22 @@ static void prepare_write_banner(struct ceph_connection *con)
con_flag_set(con, CON_FLAG_WRITE_PENDING);
}
+static void __prepare_write_connect(struct ceph_connection *con)
+{
+ con_out_kvec_add(con, sizeof(con->out_connect), &con->out_connect);
+ if (con->auth)
+ con_out_kvec_add(con, con->auth->authorizer_buf_len,
+ con->auth->authorizer_buf);
+
+ con->out_more = 0;
+ con_flag_set(con, CON_FLAG_WRITE_PENDING);
+}
+
static int prepare_write_connect(struct ceph_connection *con)
{
unsigned int global_seq = get_global_seq(con->msgr, 0);
int proto;
- int auth_proto;
- struct ceph_auth_handshake *auth;
+ int ret;
switch (con->peer_name.type) {
case CEPH_ENTITY_TYPE_MON:
@@ -1459,24 +1471,11 @@ static int prepare_write_connect(struct ceph_connection *con)
con->out_connect.protocol_version = cpu_to_le32(proto);
con->out_connect.flags = 0;
- auth_proto = CEPH_AUTH_UNKNOWN;
- auth = get_connect_authorizer(con, &auth_proto);
- if (IS_ERR(auth))
- return PTR_ERR(auth);
+ ret = get_connect_authorizer(con);
+ if (ret)
+ return ret;
- con->out_connect.authorizer_protocol = cpu_to_le32(auth_proto);
- con->out_connect.authorizer_len = auth ?
- cpu_to_le32(auth->authorizer_buf_len) : 0;
-
- con_out_kvec_add(con, sizeof (con->out_connect),
- &con->out_connect);
- if (auth && auth->authorizer_buf_len)
- con_out_kvec_add(con, auth->authorizer_buf_len,
- auth->authorizer_buf);
-
- con->out_more = 0;
- con_flag_set(con, CON_FLAG_WRITE_PENDING);
-
+ __prepare_write_connect(con);
return 0;
}
@@ -1737,11 +1736,21 @@ static int read_partial_connect(struct ceph_connection *con)
if (ret <= 0)
goto out;
- size = le32_to_cpu(con->in_reply.authorizer_len);
- end += size;
- ret = read_partial(con, end, size, con->auth_reply_buf);
- if (ret <= 0)
- goto out;
+ if (con->auth) {
+ size = le32_to_cpu(con->in_reply.authorizer_len);
+ if (size > con->auth->authorizer_reply_buf_len) {
+ pr_err("authorizer reply too big: %d > %zu\n", size,
+ con->auth->authorizer_reply_buf_len);
+ ret = -EINVAL;
+ goto out;
+ }
+
+ end += size;
+ ret = read_partial(con, end, size,
+ con->auth->authorizer_reply_buf);
+ if (ret <= 0)
+ goto out;
+ }
dout("read_partial_connect %p tag %d, con_seq = %u, g_seq = %u\n",
con, (int)con->in_reply.tag,
@@ -1749,7 +1758,6 @@ static int read_partial_connect(struct ceph_connection *con)
le32_to_cpu(con->in_reply.global_seq));
out:
return ret;
-
}
/*
@@ -2033,13 +2041,28 @@ static int process_connect(struct ceph_connection *con)
dout("process_connect on %p tag %d\n", con, (int)con->in_tag);
- if (con->auth_reply_buf) {
+ if (con->auth) {
/*
* Any connection that defines ->get_authorizer()
- * should also define ->verify_authorizer_reply().
+ * should also define ->add_authorizer_challenge() and
+ * ->verify_authorizer_reply().
+ *
* See get_connect_authorizer().
*/
- ret = con->ops->verify_authorizer_reply(con, 0);
+ if (con->in_reply.tag == CEPH_MSGR_TAG_CHALLENGE_AUTHORIZER) {
+ ret = con->ops->add_authorizer_challenge(
+ con, con->auth->authorizer_reply_buf,
+ le32_to_cpu(con->in_reply.authorizer_len));
+ if (ret < 0)
+ return ret;
+
+ con_out_kvec_reset(con);
+ __prepare_write_connect(con);
+ prepare_read_connect(con);
+ return 0;
+ }
+
+ ret = con->ops->verify_authorizer_reply(con);
if (ret < 0) {
con->error_msg = "bad authorize reply";
return ret;
@@ -2512,6 +2535,11 @@ static int try_write(struct ceph_connection *con)
int ret = 1;
dout("try_write start %p state %lu\n", con, con->state);
+ if (con->state != CON_STATE_PREOPEN &&
+ con->state != CON_STATE_CONNECTING &&
+ con->state != CON_STATE_NEGOTIATING &&
+ con->state != CON_STATE_OPEN)
+ return 0;
more:
dout("try_write out_kvec_bytes %d\n", con->out_kvec_bytes);
@@ -2537,6 +2565,8 @@ static int try_write(struct ceph_connection *con)
}
more_kvec:
+ BUG_ON(!con->sock);
+
/* kvec data queued? */
if (con->out_kvec_left) {
ret = write_partial_kvec(con);
diff --git a/net/ceph/mon_client.c b/net/ceph/mon_client.c
index a8effc8..5004810 100644
--- a/net/ceph/mon_client.c
+++ b/net/ceph/mon_client.c
@@ -209,6 +209,14 @@ static void reopen_session(struct ceph_mon_client *monc)
__open_session(monc);
}
+static void un_backoff(struct ceph_mon_client *monc)
+{
+ monc->hunt_mult /= 2; /* reduce by 50% */
+ if (monc->hunt_mult < 1)
+ monc->hunt_mult = 1;
+ dout("%s hunt_mult now %d\n", __func__, monc->hunt_mult);
+}
+
/*
* Reschedule delayed work timer.
*/
@@ -955,6 +963,7 @@ static void delayed_work(struct work_struct *work)
if (!monc->hunting) {
ceph_con_keepalive(&monc->con);
__validate_auth(monc);
+ un_backoff(monc);
}
if (is_auth) {
@@ -1114,9 +1123,8 @@ static void finish_hunting(struct ceph_mon_client *monc)
dout("%s found mon%d\n", __func__, monc->cur_mon);
monc->hunting = false;
monc->had_a_connection = true;
- monc->hunt_mult /= 2; /* reduce by 50% */
- if (monc->hunt_mult < 1)
- monc->hunt_mult = 1;
+ un_backoff(monc);
+ __schedule_delayed(monc);
}
}
diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
index 0ffeb60..70ccb07 100644
--- a/net/ceph/osd_client.c
+++ b/net/ceph/osd_client.c
@@ -4478,14 +4478,24 @@ static struct ceph_auth_handshake *get_authorizer(struct ceph_connection *con,
return auth;
}
-
-static int verify_authorizer_reply(struct ceph_connection *con, int len)
+static int add_authorizer_challenge(struct ceph_connection *con,
+ void *challenge_buf, int challenge_buf_len)
{
struct ceph_osd *o = con->private;
struct ceph_osd_client *osdc = o->o_osdc;
struct ceph_auth_client *ac = osdc->client->monc.auth;
- return ceph_auth_verify_authorizer_reply(ac, o->o_auth.authorizer, len);
+ return ceph_auth_add_authorizer_challenge(ac, o->o_auth.authorizer,
+ challenge_buf, challenge_buf_len);
+}
+
+static int verify_authorizer_reply(struct ceph_connection *con)
+{
+ struct ceph_osd *o = con->private;
+ struct ceph_osd_client *osdc = o->o_osdc;
+ struct ceph_auth_client *ac = osdc->client->monc.auth;
+
+ return ceph_auth_verify_authorizer_reply(ac, o->o_auth.authorizer);
}
static int invalidate_authorizer(struct ceph_connection *con)
@@ -4519,6 +4529,7 @@ static const struct ceph_connection_operations osd_con_ops = {
.put = put_osd_con,
.dispatch = dispatch,
.get_authorizer = get_authorizer,
+ .add_authorizer_challenge = add_authorizer_challenge,
.verify_authorizer_reply = verify_authorizer_reply,
.invalidate_authorizer = invalidate_authorizer,
.alloc_msg = alloc_msg,
diff --git a/net/compat.c b/net/compat.c
index a96fd2f..633fcf6 100644
--- a/net/compat.c
+++ b/net/compat.c
@@ -372,7 +372,8 @@ static int compat_sock_setsockopt(struct socket *sock, int level, int optname,
optname == SO_ATTACH_REUSEPORT_CBPF)
return do_set_attach_filter(sock, level, optname,
optval, optlen);
- if (optname == SO_RCVTIMEO || optname == SO_SNDTIMEO)
+ if (!COMPAT_USE_64BIT_TIME &&
+ (optname == SO_RCVTIMEO || optname == SO_SNDTIMEO))
return do_set_sock_timeout(sock, level, optname, optval, optlen);
return sock_setsockopt(sock, level, optname, optval, optlen);
@@ -437,7 +438,8 @@ static int do_get_sock_timeout(struct socket *sock, int level, int optname,
static int compat_sock_getsockopt(struct socket *sock, int level, int optname,
char __user *optval, int __user *optlen)
{
- if (optname == SO_RCVTIMEO || optname == SO_SNDTIMEO)
+ if (!COMPAT_USE_64BIT_TIME &&
+ (optname == SO_RCVTIMEO || optname == SO_SNDTIMEO))
return do_get_sock_timeout(sock, level, optname, optval, optlen);
return sock_getsockopt(sock, level, optname, optval, optlen);
}
@@ -455,12 +457,14 @@ int compat_sock_get_timestamp(struct sock *sk, struct timeval __user *userstamp)
err = -ENOENT;
if (!sock_flag(sk, SOCK_TIMESTAMP))
sock_enable_timestamp(sk, SOCK_TIMESTAMP);
- tv = ktime_to_timeval(sk->sk_stamp);
+ tv = ktime_to_timeval(sock_read_timestamp(sk));
+
if (tv.tv_sec == -1)
return err;
if (tv.tv_sec == 0) {
- sk->sk_stamp = ktime_get_real();
- tv = ktime_to_timeval(sk->sk_stamp);
+ ktime_t kt = ktime_get_real();
+ sock_write_timestamp(sk, kt);
+ tv = ktime_to_timeval(kt);
}
err = 0;
if (put_user(tv.tv_sec, &ctv->tv_sec) ||
@@ -483,12 +487,13 @@ int compat_sock_get_timestampns(struct sock *sk, struct timespec __user *usersta
err = -ENOENT;
if (!sock_flag(sk, SOCK_TIMESTAMP))
sock_enable_timestamp(sk, SOCK_TIMESTAMP);
- ts = ktime_to_timespec(sk->sk_stamp);
+ ts = ktime_to_timespec(sock_read_timestamp(sk));
if (ts.tv_sec == -1)
return err;
if (ts.tv_sec == 0) {
- sk->sk_stamp = ktime_get_real();
- ts = ktime_to_timespec(sk->sk_stamp);
+ ktime_t kt = ktime_get_real();
+ sock_write_timestamp(sk, kt);
+ ts = ktime_to_timespec(kt);
}
err = 0;
if (put_user(ts.tv_sec, &ctv->tv_sec) ||
diff --git a/net/core/datagram.c b/net/core/datagram.c
index 4fa4011..146502f 100644
--- a/net/core/datagram.c
+++ b/net/core/datagram.c
@@ -754,8 +754,9 @@ int skb_copy_and_csum_datagram_msg(struct sk_buff *skb,
return -EINVAL;
}
- if (unlikely(skb->ip_summed == CHECKSUM_COMPLETE))
- netdev_rx_csum_fault(skb->dev);
+ if (unlikely(skb->ip_summed == CHECKSUM_COMPLETE) &&
+ !skb->csum_complete_sw)
+ netdev_rx_csum_fault(NULL);
}
return 0;
fault:
diff --git a/net/core/dev.c b/net/core/dev.c
index 802b3fa..932ca2b 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -1666,6 +1666,28 @@ int call_netdevice_notifiers(unsigned long val, struct net_device *dev)
}
EXPORT_SYMBOL(call_netdevice_notifiers);
+/**
+ * call_netdevice_notifiers_mtu - call all network notifier blocks
+ * @val: value passed unmodified to notifier function
+ * @dev: net_device pointer passed unmodified to notifier function
+ * @arg: additional u32 argument passed to the notifier function
+ *
+ * Call all network notifier blocks. Parameters and return value
+ * are as for raw_notifier_call_chain().
+ */
+static int call_netdevice_notifiers_mtu(unsigned long val,
+ struct net_device *dev, u32 arg)
+{
+ struct netdev_notifier_info_ext info = {
+ .info.dev = dev,
+ .ext.mtu = arg,
+ };
+
+ BUILD_BUG_ON(offsetof(struct netdev_notifier_info_ext, info) != 0);
+
+ return call_netdevice_notifiers_info(val, dev, &info.info);
+}
+
#ifdef CONFIG_NET_INGRESS
static struct static_key ingress_needed __read_mostly;
@@ -2873,7 +2895,7 @@ netdev_features_t passthru_features_check(struct sk_buff *skb,
}
EXPORT_SYMBOL(passthru_features_check);
-static netdev_features_t dflt_features_check(const struct sk_buff *skb,
+static netdev_features_t dflt_features_check(struct sk_buff *skb,
struct net_device *dev,
netdev_features_t features)
{
@@ -4837,6 +4859,10 @@ static void napi_reuse_skb(struct napi_struct *napi, struct sk_buff *skb)
skb->vlan_tci = 0;
skb->dev = napi->dev;
skb->skb_iif = 0;
+
+ /* eth_type_trans() assumes pkt_type is PACKET_HOST */
+ skb->pkt_type = PACKET_HOST;
+
skb->encapsulation = 0;
skb_shinfo(skb)->gso_type = 0;
skb->truesize = SKB_TRUESIZE(skb_end_offset(skb));
@@ -6720,14 +6746,16 @@ int dev_set_mtu(struct net_device *dev, int new_mtu)
err = __dev_set_mtu(dev, new_mtu);
if (!err) {
- err = call_netdevice_notifiers(NETDEV_CHANGEMTU, dev);
+ err = call_netdevice_notifiers_mtu(NETDEV_CHANGEMTU, dev,
+ orig_mtu);
err = notifier_to_errno(err);
if (err) {
/* setting mtu back and notifying everyone again,
* so that they have a chance to revert changes.
*/
__dev_set_mtu(dev, orig_mtu);
- call_netdevice_notifiers(NETDEV_CHANGEMTU, dev);
+ call_netdevice_notifiers_mtu(NETDEV_CHANGEMTU, dev,
+ new_mtu);
}
}
return err;
@@ -8076,7 +8104,8 @@ int dev_change_net_namespace(struct net_device *dev, struct net *net, const char
/* We get here if we can't use the current device name */
if (!pat)
goto out;
- if (dev_get_valid_name(net, dev, pat) < 0)
+ err = dev_get_valid_name(net, dev, pat);
+ if (err < 0)
goto out;
}
@@ -8088,7 +8117,6 @@ int dev_change_net_namespace(struct net_device *dev, struct net *net, const char
dev_close(dev);
/* And unlink it from device chain */
- err = -ENODEV;
unlist_netdevice(dev);
synchronize_net();
diff --git a/net/core/dev_addr_lists.c b/net/core/dev_addr_lists.c
index c0548d2..e3e6a3e 100644
--- a/net/core/dev_addr_lists.c
+++ b/net/core/dev_addr_lists.c
@@ -57,8 +57,8 @@ static int __hw_addr_add_ex(struct netdev_hw_addr_list *list,
return -EINVAL;
list_for_each_entry(ha, &list->list, list) {
- if (!memcmp(ha->addr, addr, addr_len) &&
- ha->type == addr_type) {
+ if (ha->type == addr_type &&
+ !memcmp(ha->addr, addr, addr_len)) {
if (global) {
/* check if addr is already used as global */
if (ha->global_use)
diff --git a/net/core/ethtool.c b/net/core/ethtool.c
index 7913771..a8a9938 100644
--- a/net/core/ethtool.c
+++ b/net/core/ethtool.c
@@ -2397,13 +2397,17 @@ static int ethtool_set_per_queue_coalesce(struct net_device *dev,
return ret;
}
-static int ethtool_set_per_queue(struct net_device *dev, void __user *useraddr)
+static int ethtool_set_per_queue(struct net_device *dev,
+ void __user *useraddr, u32 sub_cmd)
{
struct ethtool_per_queue_op per_queue_opt;
if (copy_from_user(&per_queue_opt, useraddr, sizeof(per_queue_opt)))
return -EFAULT;
+ if (per_queue_opt.sub_command != sub_cmd)
+ return -EINVAL;
+
switch (per_queue_opt.sub_command) {
case ETHTOOL_GCOALESCE:
return ethtool_get_per_queue_coalesce(dev, useraddr, &per_queue_opt);
@@ -2669,7 +2673,7 @@ int dev_ethtool(struct net *net, struct ifreq *ifr)
rc = ethtool_get_phy_stats(dev, useraddr);
break;
case ETHTOOL_PERQUEUE:
- rc = ethtool_set_per_queue(dev, useraddr);
+ rc = ethtool_set_per_queue(dev, useraddr, sub_cmd);
break;
case ETHTOOL_GLINKSETTINGS:
rc = ethtool_get_link_ksettings(dev, useraddr);
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index 862d63e..ab7c500 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -538,8 +538,8 @@ bool __skb_flow_dissect(const struct sk_buff *skb,
break;
}
- if (dissector_uses_key(flow_dissector,
- FLOW_DISSECTOR_KEY_PORTS)) {
+ if (dissector_uses_key(flow_dissector, FLOW_DISSECTOR_KEY_PORTS) &&
+ !(key_control->flags & FLOW_DIS_IS_FRAGMENT)) {
key_ports = skb_flow_dissector_target(flow_dissector,
FLOW_DISSECTOR_KEY_PORTS,
target_container);
diff --git a/net/core/gen_stats.c b/net/core/gen_stats.c
index 508e051..18f17e1 100644
--- a/net/core/gen_stats.c
+++ b/net/core/gen_stats.c
@@ -77,8 +77,20 @@ gnet_stats_start_copy_compat(struct sk_buff *skb, int type, int tc_stats_type,
d->lock = lock;
spin_lock_bh(lock);
}
- if (d->tail)
- return gnet_stats_copy(d, type, NULL, 0, padattr);
+ if (d->tail) {
+ int ret = gnet_stats_copy(d, type, NULL, 0, padattr);
+
+ /* The initial attribute added in gnet_stats_copy() may be
+ * preceded by a padding attribute, in which case d->tail will
+ * end up pointing at the padding instead of the real attribute.
+ * Fix this so gnet_stats_finish_copy() adjusts the length of
+ * the right attribute.
+ */
+ if (ret == 0 && d->tail->nla_type == padattr)
+ d->tail = (struct nlattr *)((char *)d->tail +
+ NLA_ALIGN(d->tail->nla_len));
+ return ret;
+ }
return 0;
}
diff --git a/net/core/neighbour.c b/net/core/neighbour.c
index cb9a16b..2cfbe3f 100644
--- a/net/core/neighbour.c
+++ b/net/core/neighbour.c
@@ -54,7 +54,8 @@ do { \
static void neigh_timer_handler(unsigned long arg);
static void __neigh_notify(struct neighbour *n, int type, int flags);
static void neigh_update_notify(struct neighbour *neigh);
-static int pneigh_ifdown(struct neigh_table *tbl, struct net_device *dev);
+static int pneigh_ifdown_and_unlock(struct neigh_table *tbl,
+ struct net_device *dev);
static unsigned int neigh_probe_enable;
#ifdef CONFIG_PROC_FS
@@ -255,8 +256,7 @@ int neigh_ifdown(struct neigh_table *tbl, struct net_device *dev)
{
write_lock_bh(&tbl->lock);
neigh_flush_dev(tbl, dev);
- pneigh_ifdown(tbl, dev);
- write_unlock_bh(&tbl->lock);
+ pneigh_ifdown_and_unlock(tbl, dev);
del_timer_sync(&tbl->proxy_timer);
pneigh_queue_purge(&tbl->proxy_queue);
@@ -646,9 +646,10 @@ int pneigh_delete(struct neigh_table *tbl, struct net *net, const void *pkey,
return -ENOENT;
}
-static int pneigh_ifdown(struct neigh_table *tbl, struct net_device *dev)
+static int pneigh_ifdown_and_unlock(struct neigh_table *tbl,
+ struct net_device *dev)
{
- struct pneigh_entry *n, **np;
+ struct pneigh_entry *n, **np, *freelist = NULL;
u32 h;
for (h = 0; h <= PNEIGH_HASHMASK; h++) {
@@ -656,16 +657,23 @@ static int pneigh_ifdown(struct neigh_table *tbl, struct net_device *dev)
while ((n = *np) != NULL) {
if (!dev || n->dev == dev) {
*np = n->next;
- if (tbl->pdestructor)
- tbl->pdestructor(n);
- if (n->dev)
- dev_put(n->dev);
- kfree(n);
+ n->next = freelist;
+ freelist = n;
continue;
}
np = &n->next;
}
}
+ write_unlock_bh(&tbl->lock);
+ while ((n = freelist)) {
+ freelist = n->next;
+ n->next = NULL;
+ if (tbl->pdestructor)
+ tbl->pdestructor(n);
+ if (n->dev)
+ dev_put(n->dev);
+ kfree(n);
+ }
return -ENOENT;
}
@@ -1139,6 +1147,12 @@ int neigh_update(struct neighbour *neigh, const u8 *lladdr, u8 new,
lladdr = neigh->ha;
}
+ /* Update confirmed timestamp for neighbour entry after we
+ * received ARP packet even if it doesn't change IP to MAC binding.
+ */
+ if (new & NUD_CONNECTED)
+ neigh->confirmed = jiffies;
+
/* If entry was valid and address is not changed,
do not change entry state, if new one is STALE.
*/
@@ -1160,15 +1174,12 @@ int neigh_update(struct neighbour *neigh, const u8 *lladdr, u8 new,
}
}
- /* Update timestamps only once we know we will make a change to the
+ /* Update timestamp only once we know we will make a change to the
* neighbour entry. Otherwise we risk to move the locktime window with
* noop updates and ignore relevant ARP updates.
*/
- if (new != old || lladdr != neigh->ha) {
- if (new & NUD_CONNECTED)
- neigh->confirmed = jiffies;
+ if (new != old || lladdr != neigh->ha)
neigh->updated = jiffies;
- }
if (new != old) {
neigh_del_timer(neigh);
@@ -2299,12 +2310,16 @@ static int neigh_dump_table(struct neigh_table *tbl, struct sk_buff *skb,
err = nlmsg_parse(nlh, sizeof(struct ndmsg), tb, NDA_MAX, NULL);
if (!err) {
- if (tb[NDA_IFINDEX])
+ if (tb[NDA_IFINDEX]) {
+ if (nla_len(tb[NDA_IFINDEX]) != sizeof(u32))
+ return -EINVAL;
filter_idx = nla_get_u32(tb[NDA_IFINDEX]);
-
- if (tb[NDA_MASTER])
+ }
+ if (tb[NDA_MASTER]) {
+ if (nla_len(tb[NDA_MASTER]) != sizeof(u32))
+ return -EINVAL;
filter_master_idx = nla_get_u32(tb[NDA_MASTER]);
-
+ }
if (filter_idx || filter_master_idx)
flags |= NLM_F_DUMP_FILTERED;
}
diff --git a/net/core/netclassid_cgroup.c b/net/core/netclassid_cgroup.c
index 46e8830..2e4eef7 100644
--- a/net/core/netclassid_cgroup.c
+++ b/net/core/netclassid_cgroup.c
@@ -104,6 +104,7 @@ static int write_classid(struct cgroup_subsys_state *css, struct cftype *cft,
iterate_fd(p->files, 0, update_classid_sock,
(void *)(unsigned long)cs->classid);
task_unlock(p);
+ cond_resched();
}
css_task_iter_end(&it);
diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c
index c2339b8..ba72457 100644
--- a/net/core/rtnetlink.c
+++ b/net/core/rtnetlink.c
@@ -1914,6 +1914,10 @@ static int do_setlink(const struct sk_buff *skb,
const struct net_device_ops *ops = dev->netdev_ops;
int err;
+ err = validate_linkmsg(dev, tb);
+ if (err < 0)
+ return err;
+
if (tb[IFLA_NET_NS_PID] || tb[IFLA_NET_NS_FD]) {
struct net *net = rtnl_link_get_net(dev_net(dev), tb);
if (IS_ERR(net)) {
@@ -2234,10 +2238,6 @@ static int rtnl_setlink(struct sk_buff *skb, struct nlmsghdr *nlh)
goto errout;
}
- err = validate_linkmsg(dev, tb);
- if (err < 0)
- goto errout;
-
err = do_setlink(skb, dev, ifm, tb, ifname, 0);
errout:
return err;
@@ -2339,9 +2339,12 @@ int rtnl_configure_link(struct net_device *dev, const struct ifinfomsg *ifm)
return err;
}
- dev->rtnl_link_state = RTNL_LINK_INITIALIZED;
-
- __dev_notify_flags(dev, old_flags, ~0U);
+ if (dev->rtnl_link_state == RTNL_LINK_INITIALIZED) {
+ __dev_notify_flags(dev, old_flags, 0U);
+ } else {
+ dev->rtnl_link_state = RTNL_LINK_INITIALIZED;
+ __dev_notify_flags(dev, old_flags, ~0U);
+ }
return 0;
}
EXPORT_SYMBOL(rtnl_configure_link);
@@ -2365,6 +2368,12 @@ struct net_device *rtnl_create_link(struct net *net,
else if (ops->get_num_rx_queues)
num_rx_queues = ops->get_num_rx_queues();
+ if (num_tx_queues < 1 || num_tx_queues > 4096)
+ return ERR_PTR(-EINVAL);
+
+ if (num_rx_queues < 1 || num_rx_queues > 4096)
+ return ERR_PTR(-EINVAL);
+
err = -ENOMEM;
dev = alloc_netdev_mqs(ops->priv_size, ifname, name_assign_type,
ops->setup, num_tx_queues, num_rx_queues);
@@ -2978,6 +2987,11 @@ static int rtnl_fdb_add(struct sk_buff *skb, struct nlmsghdr *nlh)
return -EINVAL;
}
+ if (dev->type != ARPHRD_ETHER) {
+ pr_info("PF_BRIDGE: FDB add only supported for Ethernet devices");
+ return -EINVAL;
+ }
+
addr = nla_data(tb[NDA_LLADDR]);
err = fdb_vid_parse(tb[NDA_VLAN], &vid);
@@ -3081,6 +3095,11 @@ static int rtnl_fdb_del(struct sk_buff *skb, struct nlmsghdr *nlh)
return -EINVAL;
}
+ if (dev->type != ARPHRD_ETHER) {
+ pr_info("PF_BRIDGE: FDB delete only supported for Ethernet devices");
+ return -EINVAL;
+ }
+
addr = nla_data(tb[NDA_LLADDR]);
err = fdb_vid_parse(tb[NDA_VLAN], &vid);
@@ -3167,6 +3186,9 @@ int ndo_dflt_fdb_dump(struct sk_buff *skb,
{
int err;
+ if (dev->type != ARPHRD_ETHER)
+ return -EINVAL;
+
netif_addr_lock_bh(dev);
err = nlmsg_populate_fdb(skb, cb, dev, idx, &dev->uc);
if (err)
diff --git a/net/core/skbuff.c b/net/core/skbuff.c
index e2136eb..57aefdc 100644
--- a/net/core/skbuff.c
+++ b/net/core/skbuff.c
@@ -909,6 +909,8 @@ static struct sk_buff *__skb_clone(struct sk_buff *n, struct sk_buff *skb)
n->hdr_len = skb->nohdr ? skb_headroom(skb) : skb->hdr_len;
n->cloned = 1;
n->nohdr = 0;
+ n->peeked = 0;
+ C(pfmemalloc);
n->destructor = NULL;
C(tail);
C(end);
@@ -1582,6 +1584,21 @@ int ___pskb_trim(struct sk_buff *skb, unsigned int len)
}
EXPORT_SYMBOL(___pskb_trim);
+/* Note : use pskb_trim_rcsum() instead of calling this directly
+ */
+int pskb_trim_rcsum_slow(struct sk_buff *skb, unsigned int len)
+{
+ if (skb->ip_summed == CHECKSUM_COMPLETE) {
+ int delta = skb->len - len;
+
+ skb->csum = csum_block_sub(skb->csum,
+ skb_checksum(skb, len, delta, 0),
+ len);
+ }
+ return __pskb_trim(skb, len);
+}
+EXPORT_SYMBOL(pskb_trim_rcsum_slow);
+
/**
* __pskb_pull_tail - advance tail of skb header
* @skb: buffer to reallocate
@@ -2429,20 +2446,27 @@ EXPORT_SYMBOL(skb_queue_purge);
/**
* skb_rbtree_purge - empty a skb rbtree
* @root: root of the rbtree to empty
+ * Return value: the sum of truesizes of all purged skbs.
*
* Delete all buffers on an &sk_buff rbtree. Each buffer is removed from
* the list and one reference dropped. This function does not take
* any lock. Synchronization should be handled by the caller (e.g., TCP
* out-of-order queue is protected by the socket lock).
*/
-void skb_rbtree_purge(struct rb_root *root)
+unsigned int skb_rbtree_purge(struct rb_root *root)
{
- struct sk_buff *skb, *next;
+ struct rb_node *p = rb_first(root);
+ unsigned int sum = 0;
- rbtree_postorder_for_each_entry_safe(skb, next, root, rbnode)
+ while (p) {
+ struct sk_buff *skb = rb_entry(p, struct sk_buff, rbnode);
+
+ p = rb_next(p);
+ rb_erase(&skb->rbnode, root);
+ sum += skb->truesize;
kfree_skb(skb);
-
- *root = RB_ROOT;
+ }
+ return sum;
}
/**
@@ -3257,6 +3281,7 @@ struct sk_buff *skb_segment(struct sk_buff *head_skb,
net_warn_ratelimited(
"skb_segment: too many frags: %u %u\n",
pos, mss);
+ err = -EINVAL;
goto err;
}
@@ -3293,11 +3318,10 @@ struct sk_buff *skb_segment(struct sk_buff *head_skb,
perform_csum_check:
if (!csum) {
- if (skb_has_shared_frag(nskb)) {
- err = __skb_linearize(nskb);
- if (err)
- goto err;
- }
+ if (skb_has_shared_frag(nskb) &&
+ __skb_linearize(nskb))
+ goto err;
+
if (!nskb->remcsum_offload)
nskb->ip_summed = CHECKSUM_NONE;
SKB_GSO_CB(nskb)->csum =
@@ -4403,6 +4427,10 @@ void skb_scrub_packet(struct sk_buff *skb, bool xnet)
nf_reset(skb);
nf_reset_trace(skb);
+#ifdef CONFIG_NET_SWITCHDEV
+ skb->offload_fwd_mark = 0;
+#endif
+
if (!xnet)
return;
@@ -4480,13 +4508,18 @@ EXPORT_SYMBOL_GPL(skb_gso_validate_mtu);
static struct sk_buff *skb_reorder_vlan_header(struct sk_buff *skb)
{
+ int mac_len;
+
if (skb_cow(skb, skb_headroom(skb)) < 0) {
kfree_skb(skb);
return NULL;
}
- memmove(skb->data - ETH_HLEN, skb->data - skb->mac_len - VLAN_HLEN,
- 2 * ETH_ALEN);
+ mac_len = skb->data - skb_mac_header(skb);
+ if (likely(mac_len > VLAN_HLEN + ETH_TLEN)) {
+ memmove(skb_mac_header(skb) + VLAN_HLEN, skb_mac_header(skb),
+ mac_len - VLAN_HLEN - ETH_TLEN);
+ }
skb->mac_header += VLAN_HLEN;
return skb;
}
diff --git a/net/core/sock.c b/net/core/sock.c
index 1d88335..7d42594 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1465,7 +1465,7 @@ void sk_destruct(struct sock *sk)
static void __sk_free(struct sock *sk)
{
- if (unlikely(sock_diag_has_destroy_listeners(sk) && sk->sk_net_refcnt))
+ if (unlikely(sk->sk_net_refcnt && sock_diag_has_destroy_listeners(sk)))
sock_diag_broadcast_destroy(sk);
else
sk_destruct(sk);
@@ -2478,6 +2478,9 @@ void sock_init_data(struct socket *sock, struct sock *sk)
sk->sk_sndtimeo = MAX_SCHEDULE_TIMEOUT;
sk->sk_stamp = ktime_set(-1L, 0);
+#if BITS_PER_LONG==32
+ seqlock_init(&sk->sk_stamp_seq);
+#endif
#ifdef CONFIG_NET_RX_BUSY_POLL
sk->sk_napi_id = 0;
diff --git a/net/dcb/dcbnl.c b/net/dcb/dcbnl.c
index 3202d75..a111670 100644
--- a/net/dcb/dcbnl.c
+++ b/net/dcb/dcbnl.c
@@ -1764,7 +1764,7 @@ static struct dcb_app_type *dcb_app_lookup(const struct dcb_app *app,
if (itr->app.selector == app->selector &&
itr->app.protocol == app->protocol &&
itr->ifindex == ifindex &&
- (!prio || itr->app.priority == prio))
+ ((prio == -1) || itr->app.priority == prio))
return itr;
}
@@ -1799,7 +1799,8 @@ u8 dcb_getapp(struct net_device *dev, struct dcb_app *app)
u8 prio = 0;
spin_lock_bh(&dcb_lock);
- if ((itr = dcb_app_lookup(app, dev->ifindex, 0)))
+ itr = dcb_app_lookup(app, dev->ifindex, -1);
+ if (itr)
prio = itr->app.priority;
spin_unlock_bh(&dcb_lock);
@@ -1827,7 +1828,8 @@ int dcb_setapp(struct net_device *dev, struct dcb_app *new)
spin_lock_bh(&dcb_lock);
/* Search for existing match and replace */
- if ((itr = dcb_app_lookup(new, dev->ifindex, 0))) {
+ itr = dcb_app_lookup(new, dev->ifindex, -1);
+ if (itr) {
if (new->priority)
itr->app.priority = new->priority;
else {
@@ -1860,7 +1862,8 @@ u8 dcb_ieee_getapp_mask(struct net_device *dev, struct dcb_app *app)
u8 prio = 0;
spin_lock_bh(&dcb_lock);
- if ((itr = dcb_app_lookup(app, dev->ifindex, 0)))
+ itr = dcb_app_lookup(app, dev->ifindex, -1);
+ if (itr)
prio |= 1 << itr->app.priority;
spin_unlock_bh(&dcb_lock);
diff --git a/net/dccp/ccids/ccid2.c b/net/dccp/ccids/ccid2.c
index 7753681..161dfcf 100644
--- a/net/dccp/ccids/ccid2.c
+++ b/net/dccp/ccids/ccid2.c
@@ -126,6 +126,16 @@ static void ccid2_change_l_seq_window(struct sock *sk, u64 val)
DCCPF_SEQ_WMAX));
}
+static void dccp_tasklet_schedule(struct sock *sk)
+{
+ struct tasklet_struct *t = &dccp_sk(sk)->dccps_xmitlet;
+
+ if (!test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
+ sock_hold(sk);
+ __tasklet_schedule(t);
+ }
+}
+
static void ccid2_hc_tx_rto_expire(unsigned long data)
{
struct sock *sk = (struct sock *)data;
@@ -166,7 +176,7 @@ static void ccid2_hc_tx_rto_expire(unsigned long data)
/* if we were blocked before, we may now send cwnd=1 packet */
if (sender_was_blocked)
- tasklet_schedule(&dccp_sk(sk)->dccps_xmitlet);
+ dccp_tasklet_schedule(sk);
/* restart backed-off timer */
sk_reset_timer(sk, &hc->tx_rtotimer, jiffies + hc->tx_rto);
out:
@@ -218,14 +228,16 @@ static void ccid2_cwnd_restart(struct sock *sk, const u32 now)
struct ccid2_hc_tx_sock *hc = ccid2_hc_tx_sk(sk);
u32 cwnd = hc->tx_cwnd, restart_cwnd,
iwnd = rfc3390_bytes_to_packets(dccp_sk(sk)->dccps_mss_cache);
+ s32 delta = now - hc->tx_lsndtime;
hc->tx_ssthresh = max(hc->tx_ssthresh, (cwnd >> 1) + (cwnd >> 2));
/* don't reduce cwnd below the initial window (IW) */
restart_cwnd = min(cwnd, iwnd);
- cwnd >>= (now - hc->tx_lsndtime) / hc->tx_rto;
- hc->tx_cwnd = max(cwnd, restart_cwnd);
+ while ((delta -= hc->tx_rto) >= 0 && cwnd > restart_cwnd)
+ cwnd >>= 1;
+ hc->tx_cwnd = max(cwnd, restart_cwnd);
hc->tx_cwnd_stamp = now;
hc->tx_cwnd_used = 0;
@@ -706,7 +718,7 @@ static void ccid2_hc_tx_packet_recv(struct sock *sk, struct sk_buff *skb)
done:
/* check if incoming Acks allow pending packets to be sent */
if (sender_was_blocked && !ccid2_cwnd_network_limited(hc))
- tasklet_schedule(&dccp_sk(sk)->dccps_xmitlet);
+ dccp_tasklet_schedule(sk);
dccp_ackvec_parsed_cleanup(&hc->tx_av_chunks);
}
diff --git a/net/dccp/ccids/ccid3.c b/net/dccp/ccids/ccid3.c
index 119c043..03fcf3e 100644
--- a/net/dccp/ccids/ccid3.c
+++ b/net/dccp/ccids/ccid3.c
@@ -599,7 +599,7 @@ static void ccid3_hc_rx_send_feedback(struct sock *sk,
{
struct ccid3_hc_rx_sock *hc = ccid3_hc_rx_sk(sk);
struct dccp_sock *dp = dccp_sk(sk);
- ktime_t now = ktime_get_real();
+ ktime_t now = ktime_get();
s64 delta = 0;
switch (fbtype) {
@@ -624,15 +624,14 @@ static void ccid3_hc_rx_send_feedback(struct sock *sk,
case CCID3_FBACK_PERIODIC:
delta = ktime_us_delta(now, hc->rx_tstamp_last_feedback);
if (delta <= 0)
- DCCP_BUG("delta (%ld) <= 0", (long)delta);
- else
- hc->rx_x_recv = scaled_div32(hc->rx_bytes_recv, delta);
+ delta = 1;
+ hc->rx_x_recv = scaled_div32(hc->rx_bytes_recv, delta);
break;
default:
return;
}
- ccid3_pr_debug("Interval %ldusec, X_recv=%u, 1/p=%u\n", (long)delta,
+ ccid3_pr_debug("Interval %lldusec, X_recv=%u, 1/p=%u\n", delta,
hc->rx_x_recv, hc->rx_pinv);
hc->rx_tstamp_last_feedback = now;
@@ -679,7 +678,8 @@ static int ccid3_hc_rx_insert_options(struct sock *sk, struct sk_buff *skb)
static u32 ccid3_first_li(struct sock *sk)
{
struct ccid3_hc_rx_sock *hc = ccid3_hc_rx_sk(sk);
- u32 x_recv, p, delta;
+ u32 x_recv, p;
+ s64 delta;
u64 fval;
if (hc->rx_rtt == 0) {
@@ -687,7 +687,9 @@ static u32 ccid3_first_li(struct sock *sk)
hc->rx_rtt = DCCP_FALLBACK_RTT;
}
- delta = ktime_to_us(net_timedelta(hc->rx_tstamp_last_feedback));
+ delta = ktime_us_delta(ktime_get(), hc->rx_tstamp_last_feedback);
+ if (delta <= 0)
+ delta = 1;
x_recv = scaled_div32(hc->rx_bytes_recv, delta);
if (x_recv == 0) { /* would also trigger divide-by-zero */
DCCP_WARN("X_recv==0\n");
diff --git a/net/dccp/input.c b/net/dccp/input.c
index 4a05d78..84ff43a 100644
--- a/net/dccp/input.c
+++ b/net/dccp/input.c
@@ -605,11 +605,13 @@ int dccp_rcv_state_process(struct sock *sk, struct sk_buff *skb,
if (sk->sk_state == DCCP_LISTEN) {
if (dh->dccph_type == DCCP_PKT_REQUEST) {
/* It is possible that we process SYN packets from backlog,
- * so we need to make sure to disable BH right there.
+ * so we need to make sure to disable BH and RCU right there.
*/
+ rcu_read_lock();
local_bh_disable();
acceptable = inet_csk(sk)->icsk_af_ops->conn_request(sk, skb) >= 0;
local_bh_enable();
+ rcu_read_unlock();
if (!acceptable)
return 1;
consume_skb(skb);
diff --git a/net/dccp/ipv4.c b/net/dccp/ipv4.c
index 8c7799cd..28ad6f1 100644
--- a/net/dccp/ipv4.c
+++ b/net/dccp/ipv4.c
@@ -493,9 +493,11 @@ static int dccp_v4_send_response(const struct sock *sk, struct request_sock *req
dh->dccph_checksum = dccp_v4_csum_finish(skb, ireq->ir_loc_addr,
ireq->ir_rmt_addr);
+ rcu_read_lock();
err = ip_build_and_send_pkt(skb, sk, ireq->ir_loc_addr,
ireq->ir_rmt_addr,
- ireq_opt_deref(ireq));
+ rcu_dereference(ireq->ireq_opt));
+ rcu_read_unlock();
err = net_xmit_eval(err);
}
@@ -620,6 +622,7 @@ int dccp_v4_conn_request(struct sock *sk, struct sk_buff *skb)
ireq = inet_rsk(req);
sk_rcv_saddr_set(req_to_sk(req), ip_hdr(skb)->daddr);
sk_daddr_set(req_to_sk(req), ip_hdr(skb)->saddr);
+ ireq->ir_mark = inet_request_mark(sk, skb);
ireq->ireq_family = AF_INET;
ireq->ir_iif = sk->sk_bound_dev_if;
diff --git a/net/dccp/ipv6.c b/net/dccp/ipv6.c
index 28e8252..6cbcf39 100644
--- a/net/dccp/ipv6.c
+++ b/net/dccp/ipv6.c
@@ -349,6 +349,7 @@ static int dccp_v6_conn_request(struct sock *sk, struct sk_buff *skb)
ireq->ir_v6_rmt_addr = ipv6_hdr(skb)->saddr;
ireq->ir_v6_loc_addr = ipv6_hdr(skb)->daddr;
ireq->ireq_family = AF_INET6;
+ ireq->ir_mark = inet_request_mark(sk, skb);
if (ipv6_opt_accepted(sk, skb, IP6CB(skb)) ||
np->rxopt.bits.rxinfo || np->rxopt.bits.rxoinfo ||
diff --git a/net/dccp/proto.c b/net/dccp/proto.c
index ff3b058..936dab1 100644
--- a/net/dccp/proto.c
+++ b/net/dccp/proto.c
@@ -280,9 +280,7 @@ int dccp_disconnect(struct sock *sk, int flags)
dccp_clear_xmit_timers(sk);
ccid_hc_rx_delete(dp->dccps_hc_rx_ccid, sk);
- ccid_hc_tx_delete(dp->dccps_hc_tx_ccid, sk);
dp->dccps_hc_rx_ccid = NULL;
- dp->dccps_hc_tx_ccid = NULL;
__skb_queue_purge(&sk->sk_receive_queue);
__skb_queue_purge(&sk->sk_write_queue);
diff --git a/net/dccp/timer.c b/net/dccp/timer.c
index 3a2c340..2a952cb 100644
--- a/net/dccp/timer.c
+++ b/net/dccp/timer.c
@@ -230,12 +230,12 @@ static void dccp_write_xmitlet(unsigned long data)
else
dccp_write_xmit(sk);
bh_unlock_sock(sk);
+ sock_put(sk);
}
static void dccp_write_xmit_timer(unsigned long data)
{
dccp_write_xmitlet(data);
- sock_put((struct sock *)data);
}
void dccp_init_xmit_timers(struct sock *sk)
diff --git a/net/dns_resolver/dns_key.c b/net/dns_resolver/dns_key.c
index e1d4d89..5f5d9ea 100644
--- a/net/dns_resolver/dns_key.c
+++ b/net/dns_resolver/dns_key.c
@@ -25,6 +25,7 @@
#include <linux/moduleparam.h>
#include <linux/slab.h>
#include <linux/string.h>
+#include <linux/ratelimit.h>
#include <linux/kernel.h>
#include <linux/keyctl.h>
#include <linux/err.h>
@@ -86,35 +87,39 @@ dns_resolver_preparse(struct key_preparsed_payload *prep)
opt++;
kdebug("options: '%s'", opt);
do {
+ int opt_len, opt_nlen;
const char *eq;
- int opt_len, opt_nlen, opt_vlen, tmp;
+ char optval[128];
next_opt = memchr(opt, '#', end - opt) ?: end;
opt_len = next_opt - opt;
- if (!opt_len) {
- printk(KERN_WARNING
- "Empty option to dns_resolver key\n");
+ if (opt_len <= 0 || opt_len > sizeof(optval)) {
+ pr_warn_ratelimited("Invalid option length (%d) for dns_resolver key\n",
+ opt_len);
return -EINVAL;
}
- eq = memchr(opt, '=', opt_len) ?: end;
- opt_nlen = eq - opt;
- eq++;
- opt_vlen = next_opt - eq; /* will be -1 if no value */
+ eq = memchr(opt, '=', opt_len);
+ if (eq) {
+ opt_nlen = eq - opt;
+ eq++;
+ memcpy(optval, eq, next_opt - eq);
+ optval[next_opt - eq] = '\0';
+ } else {
+ opt_nlen = opt_len;
+ optval[0] = '\0';
+ }
- tmp = opt_vlen >= 0 ? opt_vlen : 0;
- kdebug("option '%*.*s' val '%*.*s'",
- opt_nlen, opt_nlen, opt, tmp, tmp, eq);
+ kdebug("option '%*.*s' val '%s'",
+ opt_nlen, opt_nlen, opt, optval);
/* see if it's an error number representing a DNS error
* that's to be recorded as the result in this key */
if (opt_nlen == sizeof(DNS_ERRORNO_OPTION) - 1 &&
memcmp(opt, DNS_ERRORNO_OPTION, opt_nlen) == 0) {
kdebug("dns error number option");
- if (opt_vlen <= 0)
- goto bad_option_value;
- ret = kstrtoul(eq, 10, &derrno);
+ ret = kstrtoul(optval, 10, &derrno);
if (ret < 0)
goto bad_option_value;
@@ -127,10 +132,8 @@ dns_resolver_preparse(struct key_preparsed_payload *prep)
}
bad_option_value:
- printk(KERN_WARNING
- "Option '%*.*s' to dns_resolver key:"
- " bad/missing value\n",
- opt_nlen, opt_nlen, opt);
+ pr_warn_ratelimited("Option '%*.*s' to dns_resolver key: bad/missing value\n",
+ opt_nlen, opt_nlen, opt);
return -EINVAL;
} while (opt = next_opt + 1, opt < end);
}
diff --git a/net/dsa/slave.c b/net/dsa/slave.c
index 5000e6f..339d9c6 100644
--- a/net/dsa/slave.c
+++ b/net/dsa/slave.c
@@ -1199,6 +1199,9 @@ int dsa_slave_suspend(struct net_device *slave_dev)
{
struct dsa_slave_priv *p = netdev_priv(slave_dev);
+ if (!netif_running(slave_dev))
+ return 0;
+
netif_device_detach(slave_dev);
if (p->phy) {
@@ -1216,6 +1219,9 @@ int dsa_slave_resume(struct net_device *slave_dev)
{
struct dsa_slave_priv *p = netdev_priv(slave_dev);
+ if (!netif_running(slave_dev))
+ return 0;
+
netif_device_attach(slave_dev);
if (p->phy) {
diff --git a/net/ieee802154/6lowpan/6lowpan_i.h b/net/ieee802154/6lowpan/6lowpan_i.h
index 5ac7789..78916c5 100644
--- a/net/ieee802154/6lowpan/6lowpan_i.h
+++ b/net/ieee802154/6lowpan/6lowpan_i.h
@@ -16,37 +16,19 @@ typedef unsigned __bitwise__ lowpan_rx_result;
#define LOWPAN_DISPATCH_FRAG1 0xc0
#define LOWPAN_DISPATCH_FRAGN 0xe0
-struct lowpan_create_arg {
+struct frag_lowpan_compare_key {
u16 tag;
u16 d_size;
- const struct ieee802154_addr *src;
- const struct ieee802154_addr *dst;
+ struct ieee802154_addr src;
+ struct ieee802154_addr dst;
};
-/* Equivalent of ipv4 struct ip
+/* Equivalent of ipv4 struct ipq
*/
struct lowpan_frag_queue {
struct inet_frag_queue q;
-
- u16 tag;
- u16 d_size;
- struct ieee802154_addr saddr;
- struct ieee802154_addr daddr;
};
-static inline u32 ieee802154_addr_hash(const struct ieee802154_addr *a)
-{
- switch (a->mode) {
- case IEEE802154_ADDR_LONG:
- return (((__force u64)a->extended_addr) >> 32) ^
- (((__force u64)a->extended_addr) & 0xffffffff);
- case IEEE802154_ADDR_SHORT:
- return (__force u32)(a->short_addr + (a->pan_id << 16));
- default:
- return 0;
- }
-}
-
int lowpan_frag_rcv(struct sk_buff *skb, const u8 frag_type);
void lowpan_net_frag_exit(void);
int lowpan_net_frag_init(void);
diff --git a/net/ieee802154/6lowpan/core.c b/net/ieee802154/6lowpan/core.c
index 83af533..ba8bd24 100644
--- a/net/ieee802154/6lowpan/core.c
+++ b/net/ieee802154/6lowpan/core.c
@@ -90,12 +90,18 @@ static int lowpan_neigh_construct(struct net_device *dev, struct neighbour *n)
return 0;
}
+static int lowpan_get_iflink(const struct net_device *dev)
+{
+ return lowpan_802154_dev(dev)->wdev->ifindex;
+}
+
static const struct net_device_ops lowpan_netdev_ops = {
.ndo_init = lowpan_dev_init,
.ndo_start_xmit = lowpan_xmit,
.ndo_open = lowpan_open,
.ndo_stop = lowpan_stop,
.ndo_neigh_construct = lowpan_neigh_construct,
+ .ndo_get_iflink = lowpan_get_iflink,
};
static void lowpan_setup(struct net_device *ldev)
diff --git a/net/ieee802154/6lowpan/reassembly.c b/net/ieee802154/6lowpan/reassembly.c
index f85b08b..aab1e2d 100644
--- a/net/ieee802154/6lowpan/reassembly.c
+++ b/net/ieee802154/6lowpan/reassembly.c
@@ -37,47 +37,15 @@ static struct inet_frags lowpan_frags;
static int lowpan_frag_reasm(struct lowpan_frag_queue *fq,
struct sk_buff *prev, struct net_device *ldev);
-static unsigned int lowpan_hash_frag(u16 tag, u16 d_size,
- const struct ieee802154_addr *saddr,
- const struct ieee802154_addr *daddr)
-{
- net_get_random_once(&lowpan_frags.rnd, sizeof(lowpan_frags.rnd));
- return jhash_3words(ieee802154_addr_hash(saddr),
- ieee802154_addr_hash(daddr),
- (__force u32)(tag + (d_size << 16)),
- lowpan_frags.rnd);
-}
-
-static unsigned int lowpan_hashfn(const struct inet_frag_queue *q)
-{
- const struct lowpan_frag_queue *fq;
-
- fq = container_of(q, struct lowpan_frag_queue, q);
- return lowpan_hash_frag(fq->tag, fq->d_size, &fq->saddr, &fq->daddr);
-}
-
-static bool lowpan_frag_match(const struct inet_frag_queue *q, const void *a)
-{
- const struct lowpan_frag_queue *fq;
- const struct lowpan_create_arg *arg = a;
-
- fq = container_of(q, struct lowpan_frag_queue, q);
- return fq->tag == arg->tag && fq->d_size == arg->d_size &&
- ieee802154_addr_equal(&fq->saddr, arg->src) &&
- ieee802154_addr_equal(&fq->daddr, arg->dst);
-}
-
static void lowpan_frag_init(struct inet_frag_queue *q, const void *a)
{
- const struct lowpan_create_arg *arg = a;
+ const struct frag_lowpan_compare_key *key = a;
struct lowpan_frag_queue *fq;
fq = container_of(q, struct lowpan_frag_queue, q);
- fq->tag = arg->tag;
- fq->d_size = arg->d_size;
- fq->saddr = *arg->src;
- fq->daddr = *arg->dst;
+ BUILD_BUG_ON(sizeof(*key) > sizeof(q->key));
+ memcpy(&q->key, key, sizeof(*key));
}
static void lowpan_frag_expire(unsigned long data)
@@ -93,10 +61,10 @@ static void lowpan_frag_expire(unsigned long data)
if (fq->q.flags & INET_FRAG_COMPLETE)
goto out;
- inet_frag_kill(&fq->q, &lowpan_frags);
+ inet_frag_kill(&fq->q);
out:
spin_unlock(&fq->q.lock);
- inet_frag_put(&fq->q, &lowpan_frags);
+ inet_frag_put(&fq->q);
}
static inline struct lowpan_frag_queue *
@@ -104,25 +72,20 @@ fq_find(struct net *net, const struct lowpan_802154_cb *cb,
const struct ieee802154_addr *src,
const struct ieee802154_addr *dst)
{
- struct inet_frag_queue *q;
- struct lowpan_create_arg arg;
- unsigned int hash;
struct netns_ieee802154_lowpan *ieee802154_lowpan =
net_ieee802154_lowpan(net);
+ struct frag_lowpan_compare_key key = {};
+ struct inet_frag_queue *q;
- arg.tag = cb->d_tag;
- arg.d_size = cb->d_size;
- arg.src = src;
- arg.dst = dst;
+ key.tag = cb->d_tag;
+ key.d_size = cb->d_size;
+ key.src = *src;
+ key.dst = *dst;
- hash = lowpan_hash_frag(cb->d_tag, cb->d_size, src, dst);
-
- q = inet_frag_find(&ieee802154_lowpan->frags,
- &lowpan_frags, &arg, hash);
- if (IS_ERR_OR_NULL(q)) {
- inet_frag_maybe_warn_overflow(q, pr_fmt());
+ q = inet_frag_find(&ieee802154_lowpan->frags, &key);
+ if (!q)
return NULL;
- }
+
return container_of(q, struct lowpan_frag_queue, q);
}
@@ -229,7 +192,7 @@ static int lowpan_frag_reasm(struct lowpan_frag_queue *fq, struct sk_buff *prev,
struct sk_buff *fp, *head = fq->q.fragments;
int sum_truesize;
- inet_frag_kill(&fq->q, &lowpan_frags);
+ inet_frag_kill(&fq->q);
/* Make the one we just received the head. */
if (prev) {
@@ -408,7 +371,7 @@ int lowpan_frag_rcv(struct sk_buff *skb, u8 frag_type)
struct lowpan_frag_queue *fq;
struct net *net = dev_net(skb->dev);
struct lowpan_802154_cb *cb = lowpan_802154_cb(skb);
- struct ieee802154_hdr hdr;
+ struct ieee802154_hdr hdr = {};
int err;
if (ieee802154_hdr_peek_addrs(skb, &hdr) < 0)
@@ -437,7 +400,7 @@ int lowpan_frag_rcv(struct sk_buff *skb, u8 frag_type)
ret = lowpan_frag_queue(fq, skb, frag_type);
spin_unlock(&fq->q.lock);
- inet_frag_put(&fq->q, &lowpan_frags);
+ inet_frag_put(&fq->q);
return ret;
}
@@ -447,24 +410,22 @@ int lowpan_frag_rcv(struct sk_buff *skb, u8 frag_type)
}
#ifdef CONFIG_SYSCTL
-static int zero;
static struct ctl_table lowpan_frags_ns_ctl_table[] = {
{
.procname = "6lowpanfrag_high_thresh",
.data = &init_net.ieee802154_lowpan.frags.high_thresh,
- .maxlen = sizeof(int),
+ .maxlen = sizeof(unsigned long),
.mode = 0644,
- .proc_handler = proc_dointvec_minmax,
+ .proc_handler = proc_doulongvec_minmax,
.extra1 = &init_net.ieee802154_lowpan.frags.low_thresh
},
{
.procname = "6lowpanfrag_low_thresh",
.data = &init_net.ieee802154_lowpan.frags.low_thresh,
- .maxlen = sizeof(int),
+ .maxlen = sizeof(unsigned long),
.mode = 0644,
- .proc_handler = proc_dointvec_minmax,
- .extra1 = &zero,
+ .proc_handler = proc_doulongvec_minmax,
.extra2 = &init_net.ieee802154_lowpan.frags.high_thresh
},
{
@@ -580,14 +541,20 @@ static int __net_init lowpan_frags_init_net(struct net *net)
{
struct netns_ieee802154_lowpan *ieee802154_lowpan =
net_ieee802154_lowpan(net);
+ int res;
ieee802154_lowpan->frags.high_thresh = IPV6_FRAG_HIGH_THRESH;
ieee802154_lowpan->frags.low_thresh = IPV6_FRAG_LOW_THRESH;
ieee802154_lowpan->frags.timeout = IPV6_FRAG_TIMEOUT;
+ ieee802154_lowpan->frags.f = &lowpan_frags;
- inet_frags_init_net(&ieee802154_lowpan->frags);
-
- return lowpan_frags_ns_sysctl_register(net);
+ res = inet_frags_init_net(&ieee802154_lowpan->frags);
+ if (res < 0)
+ return res;
+ res = lowpan_frags_ns_sysctl_register(net);
+ if (res < 0)
+ inet_frags_exit_net(&ieee802154_lowpan->frags);
+ return res;
}
static void __net_exit lowpan_frags_exit_net(struct net *net)
@@ -596,7 +563,7 @@ static void __net_exit lowpan_frags_exit_net(struct net *net)
net_ieee802154_lowpan(net);
lowpan_frags_ns_sysctl_unregister(net);
- inet_frags_exit_net(&ieee802154_lowpan->frags, &lowpan_frags);
+ inet_frags_exit_net(&ieee802154_lowpan->frags);
}
static struct pernet_operations lowpan_frags_ops = {
@@ -604,32 +571,63 @@ static struct pernet_operations lowpan_frags_ops = {
.exit = lowpan_frags_exit_net,
};
+static u32 lowpan_key_hashfn(const void *data, u32 len, u32 seed)
+{
+ return jhash2(data,
+ sizeof(struct frag_lowpan_compare_key) / sizeof(u32), seed);
+}
+
+static u32 lowpan_obj_hashfn(const void *data, u32 len, u32 seed)
+{
+ const struct inet_frag_queue *fq = data;
+
+ return jhash2((const u32 *)&fq->key,
+ sizeof(struct frag_lowpan_compare_key) / sizeof(u32), seed);
+}
+
+static int lowpan_obj_cmpfn(struct rhashtable_compare_arg *arg, const void *ptr)
+{
+ const struct frag_lowpan_compare_key *key = arg->key;
+ const struct inet_frag_queue *fq = ptr;
+
+ return !!memcmp(&fq->key, key, sizeof(*key));
+}
+
+static const struct rhashtable_params lowpan_rhash_params = {
+ .head_offset = offsetof(struct inet_frag_queue, node),
+ .hashfn = lowpan_key_hashfn,
+ .obj_hashfn = lowpan_obj_hashfn,
+ .obj_cmpfn = lowpan_obj_cmpfn,
+ .automatic_shrinking = true,
+};
+
int __init lowpan_net_frag_init(void)
{
int ret;
+ lowpan_frags.constructor = lowpan_frag_init;
+ lowpan_frags.destructor = NULL;
+ lowpan_frags.qsize = sizeof(struct frag_queue);
+ lowpan_frags.frag_expire = lowpan_frag_expire;
+ lowpan_frags.frags_cache_name = lowpan_frags_cache_name;
+ lowpan_frags.rhash_params = lowpan_rhash_params;
+ ret = inet_frags_init(&lowpan_frags);
+ if (ret)
+ goto out;
+
ret = lowpan_frags_sysctl_register();
if (ret)
- return ret;
+ goto err_sysctl;
ret = register_pernet_subsys(&lowpan_frags_ops);
if (ret)
goto err_pernet;
-
- lowpan_frags.hashfn = lowpan_hashfn;
- lowpan_frags.constructor = lowpan_frag_init;
- lowpan_frags.destructor = NULL;
- lowpan_frags.qsize = sizeof(struct frag_queue);
- lowpan_frags.match = lowpan_frag_match;
- lowpan_frags.frag_expire = lowpan_frag_expire;
- lowpan_frags.frags_cache_name = lowpan_frags_cache_name;
- ret = inet_frags_init(&lowpan_frags);
- if (ret)
- goto err_pernet;
-
+out:
return ret;
err_pernet:
lowpan_frags_sysctl_unregister();
+err_sysctl:
+ inet_frags_fini(&lowpan_frags);
return ret;
}
diff --git a/net/ieee802154/6lowpan/tx.c b/net/ieee802154/6lowpan/tx.c
index dbb476d..34d20a2 100644
--- a/net/ieee802154/6lowpan/tx.c
+++ b/net/ieee802154/6lowpan/tx.c
@@ -48,6 +48,9 @@ int lowpan_header_create(struct sk_buff *skb, struct net_device *ldev,
const struct ipv6hdr *hdr = ipv6_hdr(skb);
struct neighbour *n;
+ if (!daddr)
+ return -EINVAL;
+
/* TODO:
* if this package isn't ipv6 one, where should it be routed?
*/
@@ -266,9 +269,24 @@ netdev_tx_t lowpan_xmit(struct sk_buff *skb, struct net_device *ldev)
/* We must take a copy of the skb before we modify/replace the ipv6
* header as the header could be used elsewhere
*/
- skb = skb_unshare(skb, GFP_ATOMIC);
- if (!skb)
- return NET_XMIT_DROP;
+ if (unlikely(skb_headroom(skb) < ldev->needed_headroom ||
+ skb_tailroom(skb) < ldev->needed_tailroom)) {
+ struct sk_buff *nskb;
+
+ nskb = skb_copy_expand(skb, ldev->needed_headroom,
+ ldev->needed_tailroom, GFP_ATOMIC);
+ if (likely(nskb)) {
+ consume_skb(skb);
+ skb = nskb;
+ } else {
+ kfree_skb(skb);
+ return NET_XMIT_DROP;
+ }
+ } else {
+ skb = skb_unshare(skb, GFP_ATOMIC);
+ if (!skb)
+ return NET_XMIT_DROP;
+ }
ret = lowpan_header(skb, ldev, &dgram_size, &dgram_offset);
if (ret < 0) {
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index 90c91a7..275ef13 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -1315,6 +1315,7 @@ struct sk_buff *inet_gso_segment(struct sk_buff *skb,
if (encap)
skb_reset_inner_headers(skb);
skb->network_header = (u8 *)iph - skb->head;
+ skb_reset_mac_len(skb);
} while ((skb = skb->next));
out:
diff --git a/net/ipv4/cipso_ipv4.c b/net/ipv4/cipso_ipv4.c
index 972353c..571d079 100644
--- a/net/ipv4/cipso_ipv4.c
+++ b/net/ipv4/cipso_ipv4.c
@@ -1512,7 +1512,7 @@ static int cipso_v4_parsetag_loc(const struct cipso_v4_doi *doi_def,
*
* Description:
* Parse the packet's IP header looking for a CIPSO option. Returns a pointer
- * to the start of the CIPSO option on success, NULL if one if not found.
+ * to the start of the CIPSO option on success, NULL if one is not found.
*
*/
unsigned char *cipso_v4_optptr(const struct sk_buff *skb)
@@ -1522,10 +1522,21 @@ unsigned char *cipso_v4_optptr(const struct sk_buff *skb)
int optlen;
int taglen;
- for (optlen = iph->ihl*4 - sizeof(struct iphdr); optlen > 0; ) {
+ for (optlen = iph->ihl*4 - sizeof(struct iphdr); optlen > 1; ) {
+ switch (optptr[0]) {
+ case IPOPT_END:
+ return NULL;
+ case IPOPT_NOOP:
+ taglen = 1;
+ break;
+ default:
+ taglen = optptr[1];
+ }
+ if (!taglen || taglen > optlen)
+ return NULL;
if (optptr[0] == IPOPT_CIPSO)
return optptr;
- taglen = optptr[1];
+
optlen -= taglen;
optptr += taglen;
}
diff --git a/net/ipv4/fib_frontend.c b/net/ipv4/fib_frontend.c
index 08b7260..8e556fa 100644
--- a/net/ipv4/fib_frontend.c
+++ b/net/ipv4/fib_frontend.c
@@ -282,18 +282,19 @@ __be32 fib_compute_spec_dst(struct sk_buff *skb)
return ip_hdr(skb)->daddr;
in_dev = __in_dev_get_rcu(dev);
- BUG_ON(!in_dev);
net = dev_net(dev);
scope = RT_SCOPE_UNIVERSE;
if (!ipv4_is_zeronet(ip_hdr(skb)->saddr)) {
+ bool vmark = in_dev && IN_DEV_SRC_VMARK(in_dev);
struct flowi4 fl4 = {
.flowi4_iif = LOOPBACK_IFINDEX,
+ .flowi4_oif = l3mdev_master_ifindex_rcu(dev),
.daddr = ip_hdr(skb)->saddr,
.flowi4_tos = RT_TOS(ip_hdr(skb)->tos),
.flowi4_scope = scope,
- .flowi4_mark = IN_DEV_SRC_VMARK(in_dev) ? skb->mark : 0,
+ .flowi4_mark = vmark ? skb->mark : 0,
};
if (!fib_lookup(net, &fl4, &res, 0))
return FIB_RES_PREFSRC(net, res);
@@ -1171,7 +1172,8 @@ static int fib_inetaddr_event(struct notifier_block *this, unsigned long event,
static int fib_netdev_event(struct notifier_block *this, unsigned long event, void *ptr)
{
struct net_device *dev = netdev_notifier_info_to_dev(ptr);
- struct netdev_notifier_changeupper_info *info;
+ struct netdev_notifier_changeupper_info *upper_info = ptr;
+ struct netdev_notifier_info_ext *info_ext = ptr;
struct in_device *in_dev;
struct net *net = dev_net(dev);
unsigned int flags;
@@ -1206,16 +1208,19 @@ static int fib_netdev_event(struct notifier_block *this, unsigned long event, vo
fib_sync_up(dev, RTNH_F_LINKDOWN);
else
fib_sync_down_dev(dev, event, false);
- /* fall through */
+ rt_cache_flush(net);
+ break;
case NETDEV_CHANGEMTU:
+ fib_sync_mtu(dev, info_ext->ext.mtu);
rt_cache_flush(net);
break;
case NETDEV_CHANGEUPPER:
- info = ptr;
+ upper_info = ptr;
/* flush all routes if dev is linked to or unlinked from
* an L3 master device (e.g., VRF)
*/
- if (info->upper_dev && netif_is_l3_master(info->upper_dev))
+ if (upper_info->upper_dev &&
+ netif_is_l3_master(upper_info->upper_dev))
fib_disable_ip(dev, NETDEV_DOWN, true);
break;
}
diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
index e1be244..90c6540 100644
--- a/net/ipv4/fib_semantics.c
+++ b/net/ipv4/fib_semantics.c
@@ -979,6 +979,8 @@ fib_convert_metrics(struct fib_info *fi, const struct fib_config *cfg)
if (val == TCP_CA_UNSPEC)
return -EINVAL;
} else {
+ if (nla_len(nla) != sizeof(u32))
+ return -EINVAL;
val = nla_get_u32(nla);
}
if (type == RTAX_ADVMSS && val > 65535 - 40)
@@ -1375,6 +1377,56 @@ int fib_sync_down_addr(struct net_device *dev, __be32 local)
return ret;
}
+/* Update the PMTU of exceptions when:
+ * - the new MTU of the first hop becomes smaller than the PMTU
+ * - the old MTU was the same as the PMTU, and it limited discovery of
+ * larger MTUs on the path. With that limit raised, we can now
+ * discover larger MTUs
+ * A special case is locked exceptions, for which the PMTU is smaller
+ * than the minimal accepted PMTU:
+ * - if the new MTU is greater than the PMTU, don't make any change
+ * - otherwise, unlock and set PMTU
+ */
+static void nh_update_mtu(struct fib_nh *nh, u32 new, u32 orig)
+{
+ struct fnhe_hash_bucket *bucket;
+ int i;
+
+ bucket = rcu_dereference_protected(nh->nh_exceptions, 1);
+ if (!bucket)
+ return;
+
+ for (i = 0; i < FNHE_HASH_SIZE; i++) {
+ struct fib_nh_exception *fnhe;
+
+ for (fnhe = rcu_dereference_protected(bucket[i].chain, 1);
+ fnhe;
+ fnhe = rcu_dereference_protected(fnhe->fnhe_next, 1)) {
+ if (fnhe->fnhe_mtu_locked) {
+ if (new <= fnhe->fnhe_pmtu) {
+ fnhe->fnhe_pmtu = new;
+ fnhe->fnhe_mtu_locked = false;
+ }
+ } else if (new < fnhe->fnhe_pmtu ||
+ orig == fnhe->fnhe_pmtu) {
+ fnhe->fnhe_pmtu = new;
+ }
+ }
+ }
+}
+
+void fib_sync_mtu(struct net_device *dev, u32 orig_mtu)
+{
+ unsigned int hash = fib_devindex_hashfn(dev->ifindex);
+ struct hlist_head *head = &fib_info_devhash[hash];
+ struct fib_nh *nh;
+
+ hlist_for_each_entry(nh, head, nh_hash) {
+ if (nh->nh_dev == dev)
+ nh_update_mtu(nh, dev->mtu, orig_mtu);
+ }
+}
+
/* Event force Flags Description
* NETDEV_CHANGE 0 LINKDOWN Carrier OFF, not for scope host
* NETDEV_DOWN 0 LINKDOWN|DEAD Link down, not for scope host
diff --git a/net/ipv4/igmp.c b/net/ipv4/igmp.c
index 7f5fe07..f2e6e87 100644
--- a/net/ipv4/igmp.c
+++ b/net/ipv4/igmp.c
@@ -1193,8 +1193,7 @@ static void igmpv3_del_delrec(struct in_device *in_dev, struct ip_mc_list *im)
if (pmc) {
im->interface = pmc->interface;
im->crcount = in_dev->mr_qrv ?: net->ipv4.sysctl_igmp_qrv;
- im->sfmode = pmc->sfmode;
- if (pmc->sfmode == MCAST_INCLUDE) {
+ if (im->sfmode == MCAST_INCLUDE) {
im->tomb = pmc->tomb;
im->sources = pmc->sources;
for (psf = im->sources; psf; psf = psf->sf_next)
diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index 11558ca..9453180 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -417,7 +417,8 @@ struct dst_entry *inet_csk_route_req(const struct sock *sk,
struct ip_options_rcu *opt;
struct rtable *rt;
- opt = ireq_opt_deref(ireq);
+ rcu_read_lock();
+ opt = rcu_dereference(ireq->ireq_opt);
flowi4_init_output(fl4, ireq->ir_iif, ireq->ir_mark,
RT_CONN_FLAGS(sk), RT_SCOPE_UNIVERSE,
@@ -431,11 +432,13 @@ struct dst_entry *inet_csk_route_req(const struct sock *sk,
goto no_route;
if (opt && opt->opt.is_strictroute && rt->rt_uses_gateway)
goto route_err;
+ rcu_read_unlock();
return &rt->dst;
route_err:
ip_rt_put(rt);
no_route:
+ rcu_read_unlock();
__IP_INC_STATS(net, IPSTATS_MIB_OUTNOROUTES);
return NULL;
}
diff --git a/net/ipv4/inet_fragment.c b/net/ipv4/inet_fragment.c
index 8effac0..5a8c26c 100644
--- a/net/ipv4/inet_fragment.c
+++ b/net/ipv4/inet_fragment.c
@@ -25,12 +25,6 @@
#include <net/inet_frag.h>
#include <net/inet_ecn.h>
-#define INETFRAGS_EVICT_BUCKETS 128
-#define INETFRAGS_EVICT_MAX 512
-
-/* don't rebuild inetfrag table with new secret more often than this */
-#define INETFRAGS_MIN_REBUILD_INTERVAL (5 * HZ)
-
/* Given the OR values of all fragments, apply RFC 3168 5.3 requirements
* Value : 0xff if frame should be dropped.
* 0 or INET_ECN_CE value, to be ORed in to final iph->tos field
@@ -52,157 +46,8 @@ const u8 ip_frag_ecn_table[16] = {
};
EXPORT_SYMBOL(ip_frag_ecn_table);
-static unsigned int
-inet_frag_hashfn(const struct inet_frags *f, const struct inet_frag_queue *q)
-{
- return f->hashfn(q) & (INETFRAGS_HASHSZ - 1);
-}
-
-static bool inet_frag_may_rebuild(struct inet_frags *f)
-{
- return time_after(jiffies,
- f->last_rebuild_jiffies + INETFRAGS_MIN_REBUILD_INTERVAL);
-}
-
-static void inet_frag_secret_rebuild(struct inet_frags *f)
-{
- int i;
-
- write_seqlock_bh(&f->rnd_seqlock);
-
- if (!inet_frag_may_rebuild(f))
- goto out;
-
- get_random_bytes(&f->rnd, sizeof(u32));
-
- for (i = 0; i < INETFRAGS_HASHSZ; i++) {
- struct inet_frag_bucket *hb;
- struct inet_frag_queue *q;
- struct hlist_node *n;
-
- hb = &f->hash[i];
- spin_lock(&hb->chain_lock);
-
- hlist_for_each_entry_safe(q, n, &hb->chain, list) {
- unsigned int hval = inet_frag_hashfn(f, q);
-
- if (hval != i) {
- struct inet_frag_bucket *hb_dest;
-
- hlist_del(&q->list);
-
- /* Relink to new hash chain. */
- hb_dest = &f->hash[hval];
-
- /* This is the only place where we take
- * another chain_lock while already holding
- * one. As this will not run concurrently,
- * we cannot deadlock on hb_dest lock below, if its
- * already locked it will be released soon since
- * other caller cannot be waiting for hb lock
- * that we've taken above.
- */
- spin_lock_nested(&hb_dest->chain_lock,
- SINGLE_DEPTH_NESTING);
- hlist_add_head(&q->list, &hb_dest->chain);
- spin_unlock(&hb_dest->chain_lock);
- }
- }
- spin_unlock(&hb->chain_lock);
- }
-
- f->rebuild = false;
- f->last_rebuild_jiffies = jiffies;
-out:
- write_sequnlock_bh(&f->rnd_seqlock);
-}
-
-static bool inet_fragq_should_evict(const struct inet_frag_queue *q)
-{
- if (!hlist_unhashed(&q->list_evictor))
- return false;
-
- return q->net->low_thresh == 0 ||
- frag_mem_limit(q->net) >= q->net->low_thresh;
-}
-
-static unsigned int
-inet_evict_bucket(struct inet_frags *f, struct inet_frag_bucket *hb)
-{
- struct inet_frag_queue *fq;
- struct hlist_node *n;
- unsigned int evicted = 0;
- HLIST_HEAD(expired);
-
- spin_lock(&hb->chain_lock);
-
- hlist_for_each_entry_safe(fq, n, &hb->chain, list) {
- if (!inet_fragq_should_evict(fq))
- continue;
-
- if (!del_timer(&fq->timer))
- continue;
-
- hlist_add_head(&fq->list_evictor, &expired);
- ++evicted;
- }
-
- spin_unlock(&hb->chain_lock);
-
- hlist_for_each_entry_safe(fq, n, &expired, list_evictor)
- f->frag_expire((unsigned long) fq);
-
- return evicted;
-}
-
-static void inet_frag_worker(struct work_struct *work)
-{
- unsigned int budget = INETFRAGS_EVICT_BUCKETS;
- unsigned int i, evicted = 0;
- struct inet_frags *f;
-
- f = container_of(work, struct inet_frags, frags_work);
-
- BUILD_BUG_ON(INETFRAGS_EVICT_BUCKETS >= INETFRAGS_HASHSZ);
-
- local_bh_disable();
-
- for (i = ACCESS_ONCE(f->next_bucket); budget; --budget) {
- evicted += inet_evict_bucket(f, &f->hash[i]);
- i = (i + 1) & (INETFRAGS_HASHSZ - 1);
- if (evicted > INETFRAGS_EVICT_MAX)
- break;
- }
-
- f->next_bucket = i;
-
- local_bh_enable();
-
- if (f->rebuild && inet_frag_may_rebuild(f))
- inet_frag_secret_rebuild(f);
-}
-
-static void inet_frag_schedule_worker(struct inet_frags *f)
-{
- if (unlikely(!work_pending(&f->frags_work)))
- schedule_work(&f->frags_work);
-}
-
int inet_frags_init(struct inet_frags *f)
{
- int i;
-
- INIT_WORK(&f->frags_work, inet_frag_worker);
-
- for (i = 0; i < INETFRAGS_HASHSZ; i++) {
- struct inet_frag_bucket *hb = &f->hash[i];
-
- spin_lock_init(&hb->chain_lock);
- INIT_HLIST_HEAD(&hb->chain);
- }
-
- seqlock_init(&f->rnd_seqlock);
- f->last_rebuild_jiffies = 0;
f->frags_cachep = kmem_cache_create(f->frags_cache_name, f->qsize, 0, 0,
NULL);
if (!f->frags_cachep)
@@ -214,83 +59,75 @@ EXPORT_SYMBOL(inet_frags_init);
void inet_frags_fini(struct inet_frags *f)
{
- cancel_work_sync(&f->frags_work);
+ /* We must wait that all inet_frag_destroy_rcu() have completed. */
+ rcu_barrier();
+
kmem_cache_destroy(f->frags_cachep);
+ f->frags_cachep = NULL;
}
EXPORT_SYMBOL(inet_frags_fini);
-void inet_frags_exit_net(struct netns_frags *nf, struct inet_frags *f)
+static void inet_frags_free_cb(void *ptr, void *arg)
{
- unsigned int seq;
- int i;
+ struct inet_frag_queue *fq = ptr;
- nf->low_thresh = 0;
+ /* If we can not cancel the timer, it means this frag_queue
+ * is already disappearing, we have nothing to do.
+ * Otherwise, we own a refcount until the end of this function.
+ */
+ if (!del_timer(&fq->timer))
+ return;
-evict_again:
- local_bh_disable();
- seq = read_seqbegin(&f->rnd_seqlock);
+ spin_lock_bh(&fq->lock);
+ if (!(fq->flags & INET_FRAG_COMPLETE)) {
+ fq->flags |= INET_FRAG_COMPLETE;
+ atomic_dec(&fq->refcnt);
+ }
+ spin_unlock_bh(&fq->lock);
- for (i = 0; i < INETFRAGS_HASHSZ ; i++)
- inet_evict_bucket(f, &f->hash[i]);
+ inet_frag_put(fq);
+}
- local_bh_enable();
- cond_resched();
+void inet_frags_exit_net(struct netns_frags *nf)
+{
+ nf->low_thresh = 0; /* prevent creation of new frags */
- if (read_seqretry(&f->rnd_seqlock, seq) ||
- sum_frag_mem_limit(nf))
- goto evict_again;
+ rhashtable_free_and_destroy(&nf->rhashtable, inet_frags_free_cb, NULL);
}
EXPORT_SYMBOL(inet_frags_exit_net);
-static struct inet_frag_bucket *
-get_frag_bucket_locked(struct inet_frag_queue *fq, struct inet_frags *f)
-__acquires(hb->chain_lock)
-{
- struct inet_frag_bucket *hb;
- unsigned int seq, hash;
-
- restart:
- seq = read_seqbegin(&f->rnd_seqlock);
-
- hash = inet_frag_hashfn(f, fq);
- hb = &f->hash[hash];
-
- spin_lock(&hb->chain_lock);
- if (read_seqretry(&f->rnd_seqlock, seq)) {
- spin_unlock(&hb->chain_lock);
- goto restart;
- }
-
- return hb;
-}
-
-static inline void fq_unlink(struct inet_frag_queue *fq, struct inet_frags *f)
-{
- struct inet_frag_bucket *hb;
-
- hb = get_frag_bucket_locked(fq, f);
- hlist_del(&fq->list);
- fq->flags |= INET_FRAG_COMPLETE;
- spin_unlock(&hb->chain_lock);
-}
-
-void inet_frag_kill(struct inet_frag_queue *fq, struct inet_frags *f)
+void inet_frag_kill(struct inet_frag_queue *fq)
{
if (del_timer(&fq->timer))
atomic_dec(&fq->refcnt);
if (!(fq->flags & INET_FRAG_COMPLETE)) {
- fq_unlink(fq, f);
+ struct netns_frags *nf = fq->net;
+
+ fq->flags |= INET_FRAG_COMPLETE;
+ rhashtable_remove_fast(&nf->rhashtable, &fq->node, nf->f->rhash_params);
atomic_dec(&fq->refcnt);
}
}
EXPORT_SYMBOL(inet_frag_kill);
-void inet_frag_destroy(struct inet_frag_queue *q, struct inet_frags *f)
+static void inet_frag_destroy_rcu(struct rcu_head *head)
+{
+ struct inet_frag_queue *q = container_of(head, struct inet_frag_queue,
+ rcu);
+ struct inet_frags *f = q->net->f;
+
+ if (f->destructor)
+ f->destructor(q);
+ kmem_cache_free(f->frags_cachep, q);
+}
+
+void inet_frag_destroy(struct inet_frag_queue *q)
{
struct sk_buff *fp;
struct netns_frags *nf;
unsigned int sum, sum_truesize = 0;
+ struct inet_frags *f;
WARN_ON(!(q->flags & INET_FRAG_COMPLETE));
WARN_ON(del_timer(&q->timer) != 0);
@@ -298,68 +135,34 @@ void inet_frag_destroy(struct inet_frag_queue *q, struct inet_frags *f)
/* Release all fragment data. */
fp = q->fragments;
nf = q->net;
- while (fp) {
- struct sk_buff *xp = fp->next;
+ f = nf->f;
+ if (fp) {
+ do {
+ struct sk_buff *xp = fp->next;
- sum_truesize += fp->truesize;
- kfree_skb(fp);
- fp = xp;
+ sum_truesize += fp->truesize;
+ kfree_skb(fp);
+ fp = xp;
+ } while (fp);
+ } else {
+ sum_truesize = inet_frag_rbtree_purge(&q->rb_fragments);
}
sum = sum_truesize + f->qsize;
- if (f->destructor)
- f->destructor(q);
- kmem_cache_free(f->frags_cachep, q);
+ call_rcu(&q->rcu, inet_frag_destroy_rcu);
sub_frag_mem_limit(nf, sum);
}
EXPORT_SYMBOL(inet_frag_destroy);
-static struct inet_frag_queue *inet_frag_intern(struct netns_frags *nf,
- struct inet_frag_queue *qp_in,
- struct inet_frags *f,
- void *arg)
-{
- struct inet_frag_bucket *hb = get_frag_bucket_locked(qp_in, f);
- struct inet_frag_queue *qp;
-
-#ifdef CONFIG_SMP
- /* With SMP race we have to recheck hash table, because
- * such entry could have been created on other cpu before
- * we acquired hash bucket lock.
- */
- hlist_for_each_entry(qp, &hb->chain, list) {
- if (qp->net == nf && f->match(qp, arg)) {
- atomic_inc(&qp->refcnt);
- spin_unlock(&hb->chain_lock);
- qp_in->flags |= INET_FRAG_COMPLETE;
- inet_frag_put(qp_in, f);
- return qp;
- }
- }
-#endif
- qp = qp_in;
- if (!mod_timer(&qp->timer, jiffies + nf->timeout))
- atomic_inc(&qp->refcnt);
-
- atomic_inc(&qp->refcnt);
- hlist_add_head(&qp->list, &hb->chain);
-
- spin_unlock(&hb->chain_lock);
-
- return qp;
-}
-
static struct inet_frag_queue *inet_frag_alloc(struct netns_frags *nf,
struct inet_frags *f,
void *arg)
{
struct inet_frag_queue *q;
- if (!nf->high_thresh || frag_mem_limit(nf) > nf->high_thresh) {
- inet_frag_schedule_worker(f);
+ if (!nf->high_thresh || frag_mem_limit(nf) > nf->high_thresh)
return NULL;
- }
q = kmem_cache_zalloc(f->frags_cachep, GFP_ATOMIC);
if (!q)
@@ -371,70 +174,52 @@ static struct inet_frag_queue *inet_frag_alloc(struct netns_frags *nf,
setup_timer(&q->timer, f->frag_expire, (unsigned long)q);
spin_lock_init(&q->lock);
- atomic_set(&q->refcnt, 1);
+ atomic_set(&q->refcnt, 3);
return q;
}
static struct inet_frag_queue *inet_frag_create(struct netns_frags *nf,
- struct inet_frags *f,
- void *arg)
+ void *arg,
+ struct inet_frag_queue **prev)
{
+ struct inet_frags *f = nf->f;
struct inet_frag_queue *q;
q = inet_frag_alloc(nf, f, arg);
- if (!q)
+ if (!q) {
+ *prev = ERR_PTR(-ENOMEM);
return NULL;
+ }
+ mod_timer(&q->timer, jiffies + nf->timeout);
- return inet_frag_intern(nf, q, f, arg);
+ *prev = rhashtable_lookup_get_insert_key(&nf->rhashtable, &q->key,
+ &q->node, f->rhash_params);
+ if (*prev) {
+ q->flags |= INET_FRAG_COMPLETE;
+ inet_frag_kill(q);
+ inet_frag_destroy(q);
+ return NULL;
+ }
+ return q;
}
+EXPORT_SYMBOL(inet_frag_create);
-struct inet_frag_queue *inet_frag_find(struct netns_frags *nf,
- struct inet_frags *f, void *key,
- unsigned int hash)
+/* TODO : call from rcu_read_lock() and no longer use refcount_inc_not_zero() */
+struct inet_frag_queue *inet_frag_find(struct netns_frags *nf, void *key)
{
- struct inet_frag_bucket *hb;
- struct inet_frag_queue *q;
- int depth = 0;
+ struct inet_frag_queue *fq = NULL, *prev;
- if (frag_mem_limit(nf) > nf->low_thresh)
- inet_frag_schedule_worker(f);
-
- hash &= (INETFRAGS_HASHSZ - 1);
- hb = &f->hash[hash];
-
- spin_lock(&hb->chain_lock);
- hlist_for_each_entry(q, &hb->chain, list) {
- if (q->net == nf && f->match(q, key)) {
- atomic_inc(&q->refcnt);
- spin_unlock(&hb->chain_lock);
- return q;
- }
- depth++;
+ rcu_read_lock();
+ prev = rhashtable_lookup(&nf->rhashtable, key, nf->f->rhash_params);
+ if (!prev)
+ fq = inet_frag_create(nf, key, &prev);
+ if (prev && !IS_ERR(prev)) {
+ fq = prev;
+ if (!atomic_inc_not_zero(&fq->refcnt))
+ fq = NULL;
}
- spin_unlock(&hb->chain_lock);
-
- if (depth <= INETFRAGS_MAXDEPTH)
- return inet_frag_create(nf, f, key);
-
- if (inet_frag_may_rebuild(f)) {
- if (!f->rebuild)
- f->rebuild = true;
- inet_frag_schedule_worker(f);
- }
-
- return ERR_PTR(-ENOBUFS);
+ rcu_read_unlock();
+ return fq;
}
EXPORT_SYMBOL(inet_frag_find);
-
-void inet_frag_maybe_warn_overflow(struct inet_frag_queue *q,
- const char *prefix)
-{
- static const char msg[] = "inet_frag_find: Fragment hash bucket"
- " list length grew over limit " __stringify(INETFRAGS_MAXDEPTH)
- ". Dropping fragment.\n";
-
- if (PTR_ERR(q) == -ENOBUFS)
- net_dbg_ratelimited("%s%s", prefix, msg);
-}
-EXPORT_SYMBOL(inet_frag_maybe_warn_overflow);
diff --git a/net/ipv4/inet_timewait_sock.c b/net/ipv4/inet_timewait_sock.c
index ddcd56c..a6b34ac 100644
--- a/net/ipv4/inet_timewait_sock.c
+++ b/net/ipv4/inet_timewait_sock.c
@@ -182,6 +182,7 @@ struct inet_timewait_sock *inet_twsk_alloc(const struct sock *sk,
tw->tw_dport = inet->inet_dport;
tw->tw_family = sk->sk_family;
tw->tw_reuse = sk->sk_reuse;
+ tw->tw_reuseport = sk->sk_reuseport;
tw->tw_hash = sk->sk_hash;
tw->tw_ipv6only = 0;
tw->tw_transparent = inet->transparent;
diff --git a/net/ipv4/ip_fragment.c b/net/ipv4/ip_fragment.c
index 4bf3b8a..496f8d86b 100644
--- a/net/ipv4/ip_fragment.c
+++ b/net/ipv4/ip_fragment.c
@@ -56,27 +56,64 @@
*/
static const char ip_frag_cache_name[] = "ip4-frags";
-struct ipfrag_skb_cb
-{
+/* Use skb->cb to track consecutive/adjacent fragments coming at
+ * the end of the queue. Nodes in the rb-tree queue will
+ * contain "runs" of one or more adjacent fragments.
+ *
+ * Invariants:
+ * - next_frag is NULL at the tail of a "run";
+ * - the head of a "run" has the sum of all fragment lengths in frag_run_len.
+ */
+struct ipfrag_skb_cb {
struct inet_skb_parm h;
- int offset;
+ struct sk_buff *next_frag;
+ int frag_run_len;
};
-#define FRAG_CB(skb) ((struct ipfrag_skb_cb *)((skb)->cb))
+#define FRAG_CB(skb) ((struct ipfrag_skb_cb *)((skb)->cb))
+
+static void ip4_frag_init_run(struct sk_buff *skb)
+{
+ BUILD_BUG_ON(sizeof(struct ipfrag_skb_cb) > sizeof(skb->cb));
+
+ FRAG_CB(skb)->next_frag = NULL;
+ FRAG_CB(skb)->frag_run_len = skb->len;
+}
+
+/* Append skb to the last "run". */
+static void ip4_frag_append_to_last_run(struct inet_frag_queue *q,
+ struct sk_buff *skb)
+{
+ RB_CLEAR_NODE(&skb->rbnode);
+ FRAG_CB(skb)->next_frag = NULL;
+
+ FRAG_CB(q->last_run_head)->frag_run_len += skb->len;
+ FRAG_CB(q->fragments_tail)->next_frag = skb;
+ q->fragments_tail = skb;
+}
+
+/* Create a new "run" with the skb. */
+static void ip4_frag_create_run(struct inet_frag_queue *q, struct sk_buff *skb)
+{
+ if (q->last_run_head)
+ rb_link_node(&skb->rbnode, &q->last_run_head->rbnode,
+ &q->last_run_head->rbnode.rb_right);
+ else
+ rb_link_node(&skb->rbnode, NULL, &q->rb_fragments.rb_node);
+ rb_insert_color(&skb->rbnode, &q->rb_fragments);
+
+ ip4_frag_init_run(skb);
+ q->fragments_tail = skb;
+ q->last_run_head = skb;
+}
/* Describe an entry in the "incomplete datagrams" queue. */
struct ipq {
struct inet_frag_queue q;
- u32 user;
- __be32 saddr;
- __be32 daddr;
- __be16 id;
- u8 protocol;
u8 ecn; /* RFC3168 support */
u16 max_df_size; /* largest frag with DF set seen */
int iif;
- int vif; /* L3 master device index */
unsigned int rid;
struct inet_peer *peer;
};
@@ -88,49 +125,9 @@ static u8 ip4_frag_ecn(u8 tos)
static struct inet_frags ip4_frags;
-int ip_frag_mem(struct net *net)
-{
- return sum_frag_mem_limit(&net->ipv4.frags);
-}
+static int ip_frag_reasm(struct ipq *qp, struct sk_buff *skb,
+ struct sk_buff *prev_tail, struct net_device *dev);
-static int ip_frag_reasm(struct ipq *qp, struct sk_buff *prev,
- struct net_device *dev);
-
-struct ip4_create_arg {
- struct iphdr *iph;
- u32 user;
- int vif;
-};
-
-static unsigned int ipqhashfn(__be16 id, __be32 saddr, __be32 daddr, u8 prot)
-{
- net_get_random_once(&ip4_frags.rnd, sizeof(ip4_frags.rnd));
- return jhash_3words((__force u32)id << 16 | prot,
- (__force u32)saddr, (__force u32)daddr,
- ip4_frags.rnd);
-}
-
-static unsigned int ip4_hashfn(const struct inet_frag_queue *q)
-{
- const struct ipq *ipq;
-
- ipq = container_of(q, struct ipq, q);
- return ipqhashfn(ipq->id, ipq->saddr, ipq->daddr, ipq->protocol);
-}
-
-static bool ip4_frag_match(const struct inet_frag_queue *q, const void *a)
-{
- const struct ipq *qp;
- const struct ip4_create_arg *arg = a;
-
- qp = container_of(q, struct ipq, q);
- return qp->id == arg->iph->id &&
- qp->saddr == arg->iph->saddr &&
- qp->daddr == arg->iph->daddr &&
- qp->protocol == arg->iph->protocol &&
- qp->user == arg->user &&
- qp->vif == arg->vif;
-}
static void ip4_frag_init(struct inet_frag_queue *q, const void *a)
{
@@ -139,17 +136,12 @@ static void ip4_frag_init(struct inet_frag_queue *q, const void *a)
frags);
struct net *net = container_of(ipv4, struct net, ipv4);
- const struct ip4_create_arg *arg = a;
+ const struct frag_v4_compare_key *key = a;
- qp->protocol = arg->iph->protocol;
- qp->id = arg->iph->id;
- qp->ecn = ip4_frag_ecn(arg->iph->tos);
- qp->saddr = arg->iph->saddr;
- qp->daddr = arg->iph->daddr;
- qp->vif = arg->vif;
- qp->user = arg->user;
+ q->key.v4 = *key;
+ qp->ecn = 0;
qp->peer = q->net->max_dist ?
- inet_getpeer_v4(net->ipv4.peers, arg->iph->saddr, arg->vif, 1) :
+ inet_getpeer_v4(net->ipv4.peers, key->saddr, key->vif, 1) :
NULL;
}
@@ -167,7 +159,7 @@ static void ip4_frag_free(struct inet_frag_queue *q)
static void ipq_put(struct ipq *ipq)
{
- inet_frag_put(&ipq->q, &ip4_frags);
+ inet_frag_put(&ipq->q);
}
/* Kill ipq entry. It is not destroyed immediately,
@@ -175,7 +167,7 @@ static void ipq_put(struct ipq *ipq)
*/
static void ipq_kill(struct ipq *ipq)
{
- inet_frag_kill(&ipq->q, &ip4_frags);
+ inet_frag_kill(&ipq->q);
}
static bool frag_expire_skip_icmp(u32 user)
@@ -192,8 +184,11 @@ static bool frag_expire_skip_icmp(u32 user)
*/
static void ip_expire(unsigned long arg)
{
- struct ipq *qp;
+ const struct iphdr *iph;
+ struct sk_buff *head = NULL;
struct net *net;
+ struct ipq *qp;
+ int err;
qp = container_of((struct inet_frag_queue *) arg, struct ipq, q);
net = container_of(qp->q.net, struct net, ipv4.frags);
@@ -206,51 +201,65 @@ static void ip_expire(unsigned long arg)
ipq_kill(qp);
__IP_INC_STATS(net, IPSTATS_MIB_REASMFAILS);
+ __IP_INC_STATS(net, IPSTATS_MIB_REASMTIMEOUT);
- if (!inet_frag_evicting(&qp->q)) {
- struct sk_buff *clone, *head = qp->q.fragments;
- const struct iphdr *iph;
- int err;
+ if (!(qp->q.flags & INET_FRAG_FIRST_IN))
+ goto out;
- __IP_INC_STATS(net, IPSTATS_MIB_REASMTIMEOUT);
-
- if (!(qp->q.flags & INET_FRAG_FIRST_IN) || !qp->q.fragments)
+ /* sk_buff::dev and sk_buff::rbnode are unionized. So we
+ * pull the head out of the tree in order to be able to
+ * deal with head->dev.
+ */
+ if (qp->q.fragments) {
+ head = qp->q.fragments;
+ qp->q.fragments = head->next;
+ } else {
+ head = skb_rb_first(&qp->q.rb_fragments);
+ if (!head)
goto out;
-
- head->dev = dev_get_by_index_rcu(net, qp->iif);
- if (!head->dev)
- goto out;
-
-
- /* skb has no dst, perform route lookup again */
- iph = ip_hdr(head);
- err = ip_route_input_noref(head, iph->daddr, iph->saddr,
- iph->tos, head->dev);
- if (err)
- goto out;
-
- /* Only an end host needs to send an ICMP
- * "Fragment Reassembly Timeout" message, per RFC792.
- */
- if (frag_expire_skip_icmp(qp->user) &&
- (skb_rtable(head)->rt_type != RTN_LOCAL))
- goto out;
-
- clone = skb_clone(head, GFP_ATOMIC);
-
- /* Send an ICMP "Fragment Reassembly Timeout" message. */
- if (clone) {
- spin_unlock(&qp->q.lock);
- icmp_send(clone, ICMP_TIME_EXCEEDED,
- ICMP_EXC_FRAGTIME, 0);
- consume_skb(clone);
- goto out_rcu_unlock;
- }
+ if (FRAG_CB(head)->next_frag)
+ rb_replace_node(&head->rbnode,
+ &FRAG_CB(head)->next_frag->rbnode,
+ &qp->q.rb_fragments);
+ else
+ rb_erase(&head->rbnode, &qp->q.rb_fragments);
+ memset(&head->rbnode, 0, sizeof(head->rbnode));
+ barrier();
}
+ if (head == qp->q.fragments_tail)
+ qp->q.fragments_tail = NULL;
+
+ sub_frag_mem_limit(qp->q.net, head->truesize);
+
+ head->dev = dev_get_by_index_rcu(net, qp->iif);
+ if (!head->dev)
+ goto out;
+
+
+ /* skb has no dst, perform route lookup again */
+ iph = ip_hdr(head);
+ err = ip_route_input_noref(head, iph->daddr, iph->saddr,
+ iph->tos, head->dev);
+ if (err)
+ goto out;
+
+ /* Only an end host needs to send an ICMP
+ * "Fragment Reassembly Timeout" message, per RFC792.
+ */
+ if (frag_expire_skip_icmp(qp->q.key.v4.user) &&
+ (skb_rtable(head)->rt_type != RTN_LOCAL))
+ goto out;
+
+ spin_unlock(&qp->q.lock);
+ icmp_send(head, ICMP_TIME_EXCEEDED, ICMP_EXC_FRAGTIME, 0);
+ goto out_rcu_unlock;
+
out:
spin_unlock(&qp->q.lock);
out_rcu_unlock:
rcu_read_unlock();
+ if (head)
+ kfree_skb(head);
ipq_put(qp);
}
@@ -260,21 +269,20 @@ static void ip_expire(unsigned long arg)
static struct ipq *ip_find(struct net *net, struct iphdr *iph,
u32 user, int vif)
{
+ struct frag_v4_compare_key key = {
+ .saddr = iph->saddr,
+ .daddr = iph->daddr,
+ .user = user,
+ .vif = vif,
+ .id = iph->id,
+ .protocol = iph->protocol,
+ };
struct inet_frag_queue *q;
- struct ip4_create_arg arg;
- unsigned int hash;
- arg.iph = iph;
- arg.user = user;
- arg.vif = vif;
-
- hash = ipqhashfn(iph->id, iph->saddr, iph->daddr, iph->protocol);
-
- q = inet_frag_find(&net->ipv4.frags, &ip4_frags, &arg, hash);
- if (IS_ERR_OR_NULL(q)) {
- inet_frag_maybe_warn_overflow(q, pr_fmt());
+ q = inet_frag_find(&net->ipv4.frags, &key);
+ if (!q)
return NULL;
- }
+
return container_of(q, struct ipq, q);
}
@@ -294,7 +302,7 @@ static int ip_frag_too_far(struct ipq *qp)
end = atomic_inc_return(&peer->rid);
qp->rid = end;
- rc = qp->q.fragments && (end - start) > max;
+ rc = qp->q.fragments_tail && (end - start) > max;
if (rc) {
struct net *net;
@@ -308,7 +316,6 @@ static int ip_frag_too_far(struct ipq *qp)
static int ip_frag_reinit(struct ipq *qp)
{
- struct sk_buff *fp;
unsigned int sum_truesize = 0;
if (!mod_timer(&qp->q.timer, jiffies + qp->q.net->timeout)) {
@@ -316,21 +323,16 @@ static int ip_frag_reinit(struct ipq *qp)
return -ETIMEDOUT;
}
- fp = qp->q.fragments;
- do {
- struct sk_buff *xp = fp->next;
-
- sum_truesize += fp->truesize;
- kfree_skb(fp);
- fp = xp;
- } while (fp);
+ sum_truesize = inet_frag_rbtree_purge(&qp->q.rb_fragments);
sub_frag_mem_limit(qp->q.net, sum_truesize);
qp->q.flags = 0;
qp->q.len = 0;
qp->q.meat = 0;
qp->q.fragments = NULL;
+ qp->q.rb_fragments = RB_ROOT;
qp->q.fragments_tail = NULL;
+ qp->q.last_run_head = NULL;
qp->iif = 0;
qp->ecn = 0;
@@ -340,11 +342,13 @@ static int ip_frag_reinit(struct ipq *qp)
/* Add new segment to existing queue. */
static int ip_frag_queue(struct ipq *qp, struct sk_buff *skb)
{
- struct sk_buff *prev, *next;
+ struct net *net = container_of(qp->q.net, struct net, ipv4.frags);
+ struct rb_node **rbn, *parent;
+ struct sk_buff *skb1, *prev_tail;
+ int ihl, end, skb1_run_end;
struct net_device *dev;
unsigned int fragsize;
int flags, offset;
- int ihl, end;
int err = -ENOENT;
u8 ecn;
@@ -403,94 +407,67 @@ static int ip_frag_queue(struct ipq *qp, struct sk_buff *skb)
if (err)
goto err;
- /* Find out which fragments are in front and at the back of us
- * in the chain of fragments so far. We must know where to put
- * this fragment, right?
- */
- prev = qp->q.fragments_tail;
- if (!prev || FRAG_CB(prev)->offset < offset) {
- next = NULL;
- goto found;
- }
- prev = NULL;
- for (next = qp->q.fragments; next != NULL; next = next->next) {
- if (FRAG_CB(next)->offset >= offset)
- break; /* bingo! */
- prev = next;
- }
-
-found:
- /* We found where to put this one. Check for overlap with
- * preceding fragment, and, if needed, align things so that
- * any overlaps are eliminated.
- */
- if (prev) {
- int i = (FRAG_CB(prev)->offset + prev->len) - offset;
-
- if (i > 0) {
- offset += i;
- err = -EINVAL;
- if (end <= offset)
- goto err;
- err = -ENOMEM;
- if (!pskb_pull(skb, i))
- goto err;
- if (skb->ip_summed != CHECKSUM_UNNECESSARY)
- skb->ip_summed = CHECKSUM_NONE;
- }
- }
-
- err = -ENOMEM;
-
- while (next && FRAG_CB(next)->offset < end) {
- int i = end - FRAG_CB(next)->offset; /* overlap is 'i' bytes */
-
- if (i < next->len) {
- /* Eat head of the next overlapped fragment
- * and leave the loop. The next ones cannot overlap.
- */
- if (!pskb_pull(next, i))
- goto err;
- FRAG_CB(next)->offset += i;
- qp->q.meat -= i;
- if (next->ip_summed != CHECKSUM_UNNECESSARY)
- next->ip_summed = CHECKSUM_NONE;
- break;
- } else {
- struct sk_buff *free_it = next;
-
- /* Old fragment is completely overridden with
- * new one drop it.
- */
- next = next->next;
-
- if (prev)
- prev->next = next;
- else
- qp->q.fragments = next;
-
- qp->q.meat -= free_it->len;
- sub_frag_mem_limit(qp->q.net, free_it->truesize);
- kfree_skb(free_it);
- }
- }
-
- FRAG_CB(skb)->offset = offset;
-
- /* Insert this fragment in the chain of fragments. */
- skb->next = next;
- if (!next)
- qp->q.fragments_tail = skb;
- if (prev)
- prev->next = skb;
- else
- qp->q.fragments = skb;
-
+ /* Note : skb->rbnode and skb->dev share the same location. */
dev = skb->dev;
- if (dev) {
- qp->iif = dev->ifindex;
- skb->dev = NULL;
+ /* Makes sure compiler wont do silly aliasing games */
+ barrier();
+
+ /* RFC5722, Section 4, amended by Errata ID : 3089
+ * When reassembling an IPv6 datagram, if
+ * one or more its constituent fragments is determined to be an
+ * overlapping fragment, the entire datagram (and any constituent
+ * fragments) MUST be silently discarded.
+ *
+ * We do the same here for IPv4 (and increment an snmp counter) but
+ * we do not want to drop the whole queue in response to a duplicate
+ * fragment.
+ */
+
+ /* Find out where to put this fragment. */
+ prev_tail = qp->q.fragments_tail;
+ if (!prev_tail)
+ ip4_frag_create_run(&qp->q, skb); /* First fragment. */
+ else if (prev_tail->ip_defrag_offset + prev_tail->len < end) {
+ /* This is the common case: skb goes to the end. */
+ /* Detect and discard overlaps. */
+ if (offset < prev_tail->ip_defrag_offset + prev_tail->len)
+ goto discard_qp;
+ if (offset == prev_tail->ip_defrag_offset + prev_tail->len)
+ ip4_frag_append_to_last_run(&qp->q, skb);
+ else
+ ip4_frag_create_run(&qp->q, skb);
+ } else {
+ /* Binary search. Note that skb can become the first fragment,
+ * but not the last (covered above).
+ */
+ rbn = &qp->q.rb_fragments.rb_node;
+ do {
+ parent = *rbn;
+ skb1 = rb_to_skb(parent);
+ skb1_run_end = skb1->ip_defrag_offset +
+ FRAG_CB(skb1)->frag_run_len;
+ if (end <= skb1->ip_defrag_offset)
+ rbn = &parent->rb_left;
+ else if (offset >= skb1_run_end)
+ rbn = &parent->rb_right;
+ else if (offset >= skb1->ip_defrag_offset &&
+ end <= skb1_run_end)
+ goto err; /* No new data, potential duplicate */
+ else
+ goto discard_qp; /* Found an overlap */
+ } while (*rbn);
+ /* Here we have parent properly set, and rbn pointing to
+ * one of its NULL left/right children. Insert skb.
+ */
+ ip4_frag_init_run(skb);
+ rb_link_node(&skb->rbnode, parent, rbn);
+ rb_insert_color(&skb->rbnode, &qp->q.rb_fragments);
}
+
+ if (dev)
+ qp->iif = dev->ifindex;
+ skb->ip_defrag_offset = offset;
+
qp->q.stamp = skb->tstamp;
qp->q.meat += skb->len;
qp->ecn |= ecn;
@@ -512,7 +489,7 @@ static int ip_frag_queue(struct ipq *qp, struct sk_buff *skb)
unsigned long orefdst = skb->_skb_refdst;
skb->_skb_refdst = 0UL;
- err = ip_frag_reasm(qp, prev, dev);
+ err = ip_frag_reasm(qp, skb, prev_tail, dev);
skb->_skb_refdst = orefdst;
return err;
}
@@ -520,20 +497,24 @@ static int ip_frag_queue(struct ipq *qp, struct sk_buff *skb)
skb_dst_drop(skb);
return -EINPROGRESS;
+discard_qp:
+ inet_frag_kill(&qp->q);
+ err = -EINVAL;
+ __IP_INC_STATS(net, IPSTATS_MIB_REASM_OVERLAPS);
err:
kfree_skb(skb);
return err;
}
-
/* Build a new IP datagram from all its fragments. */
-
-static int ip_frag_reasm(struct ipq *qp, struct sk_buff *prev,
- struct net_device *dev)
+static int ip_frag_reasm(struct ipq *qp, struct sk_buff *skb,
+ struct sk_buff *prev_tail, struct net_device *dev)
{
struct net *net = container_of(qp->q.net, struct net, ipv4.frags);
struct iphdr *iph;
- struct sk_buff *fp, *head = qp->q.fragments;
+ struct sk_buff *fp, *head = skb_rb_first(&qp->q.rb_fragments);
+ struct sk_buff **nextp; /* To build frag_list. */
+ struct rb_node *rbn;
int len;
int ihlen;
int err;
@@ -547,26 +528,27 @@ static int ip_frag_reasm(struct ipq *qp, struct sk_buff *prev,
goto out_fail;
}
/* Make the one we just received the head. */
- if (prev) {
- head = prev->next;
- fp = skb_clone(head, GFP_ATOMIC);
+ if (head != skb) {
+ fp = skb_clone(skb, GFP_ATOMIC);
if (!fp)
goto out_nomem;
-
- fp->next = head->next;
- if (!fp->next)
+ FRAG_CB(fp)->next_frag = FRAG_CB(skb)->next_frag;
+ if (RB_EMPTY_NODE(&skb->rbnode))
+ FRAG_CB(prev_tail)->next_frag = fp;
+ else
+ rb_replace_node(&skb->rbnode, &fp->rbnode,
+ &qp->q.rb_fragments);
+ if (qp->q.fragments_tail == skb)
qp->q.fragments_tail = fp;
- prev->next = fp;
-
- skb_morph(head, qp->q.fragments);
- head->next = qp->q.fragments->next;
-
- consume_skb(qp->q.fragments);
- qp->q.fragments = head;
+ skb_morph(skb, head);
+ FRAG_CB(skb)->next_frag = FRAG_CB(head)->next_frag;
+ rb_replace_node(&head->rbnode, &skb->rbnode,
+ &qp->q.rb_fragments);
+ consume_skb(head);
+ head = skb;
}
- WARN_ON(!head);
- WARN_ON(FRAG_CB(head)->offset != 0);
+ WARN_ON(head->ip_defrag_offset != 0);
/* Allocate a new buffer for the datagram. */
ihlen = ip_hdrlen(head);
@@ -590,35 +572,61 @@ static int ip_frag_reasm(struct ipq *qp, struct sk_buff *prev,
clone = alloc_skb(0, GFP_ATOMIC);
if (!clone)
goto out_nomem;
- clone->next = head->next;
- head->next = clone;
skb_shinfo(clone)->frag_list = skb_shinfo(head)->frag_list;
skb_frag_list_init(head);
for (i = 0; i < skb_shinfo(head)->nr_frags; i++)
plen += skb_frag_size(&skb_shinfo(head)->frags[i]);
clone->len = clone->data_len = head->data_len - plen;
- head->data_len -= clone->len;
- head->len -= clone->len;
+ head->truesize += clone->truesize;
clone->csum = 0;
clone->ip_summed = head->ip_summed;
add_frag_mem_limit(qp->q.net, clone->truesize);
+ skb_shinfo(head)->frag_list = clone;
+ nextp = &clone->next;
+ } else {
+ nextp = &skb_shinfo(head)->frag_list;
}
- skb_shinfo(head)->frag_list = head->next;
skb_push(head, head->data - skb_network_header(head));
- for (fp=head->next; fp; fp = fp->next) {
- head->data_len += fp->len;
- head->len += fp->len;
- if (head->ip_summed != fp->ip_summed)
- head->ip_summed = CHECKSUM_NONE;
- else if (head->ip_summed == CHECKSUM_COMPLETE)
- head->csum = csum_add(head->csum, fp->csum);
- head->truesize += fp->truesize;
+ /* Traverse the tree in order, to build frag_list. */
+ fp = FRAG_CB(head)->next_frag;
+ rbn = rb_next(&head->rbnode);
+ rb_erase(&head->rbnode, &qp->q.rb_fragments);
+ while (rbn || fp) {
+ /* fp points to the next sk_buff in the current run;
+ * rbn points to the next run.
+ */
+ /* Go through the current run. */
+ while (fp) {
+ *nextp = fp;
+ nextp = &fp->next;
+ fp->prev = NULL;
+ memset(&fp->rbnode, 0, sizeof(fp->rbnode));
+ fp->sk = NULL;
+ head->data_len += fp->len;
+ head->len += fp->len;
+ if (head->ip_summed != fp->ip_summed)
+ head->ip_summed = CHECKSUM_NONE;
+ else if (head->ip_summed == CHECKSUM_COMPLETE)
+ head->csum = csum_add(head->csum, fp->csum);
+ head->truesize += fp->truesize;
+ fp = FRAG_CB(fp)->next_frag;
+ }
+ /* Move to the next run. */
+ if (rbn) {
+ struct rb_node *rbnext = rb_next(rbn);
+
+ fp = rb_to_skb(rbn);
+ rb_erase(rbn, &qp->q.rb_fragments);
+ rbn = rbnext;
+ }
}
sub_frag_mem_limit(qp->q.net, head->truesize);
+ *nextp = NULL;
head->next = NULL;
+ head->prev = NULL;
head->dev = dev;
head->tstamp = qp->q.stamp;
IPCB(head)->frag_max_size = max(qp->max_df_size, qp->q.max_size);
@@ -646,7 +654,9 @@ static int ip_frag_reasm(struct ipq *qp, struct sk_buff *prev,
__IP_INC_STATS(net, IPSTATS_MIB_REASMOKS);
qp->q.fragments = NULL;
+ qp->q.rb_fragments = RB_ROOT;
qp->q.fragments_tail = NULL;
+ qp->q.last_run_head = NULL;
return 0;
out_nomem:
@@ -654,7 +664,7 @@ static int ip_frag_reasm(struct ipq *qp, struct sk_buff *prev,
err = -ENOMEM;
goto out_fail;
out_oversize:
- net_info_ratelimited("Oversized IP packet from %pI4\n", &qp->saddr);
+ net_info_ratelimited("Oversized IP packet from %pI4\n", &qp->q.key.v4.saddr);
out_fail:
__IP_INC_STATS(net, IPSTATS_MIB_REASMFAILS);
return err;
@@ -714,10 +724,14 @@ struct sk_buff *ip_check_defrag(struct net *net, struct sk_buff *skb, u32 user)
if (ip_is_fragment(&iph)) {
skb = skb_share_check(skb, GFP_ATOMIC);
if (skb) {
- if (!pskb_may_pull(skb, netoff + iph.ihl * 4))
- return skb;
- if (pskb_trim_rcsum(skb, netoff + len))
- return skb;
+ if (!pskb_may_pull(skb, netoff + iph.ihl * 4)) {
+ kfree_skb(skb);
+ return NULL;
+ }
+ if (pskb_trim_rcsum(skb, netoff + len)) {
+ kfree_skb(skb);
+ return NULL;
+ }
memset(IPCB(skb), 0, sizeof(struct inet_skb_parm));
if (ip_defrag(net, skb, user))
return NULL;
@@ -728,25 +742,46 @@ struct sk_buff *ip_check_defrag(struct net *net, struct sk_buff *skb, u32 user)
}
EXPORT_SYMBOL(ip_check_defrag);
+unsigned int inet_frag_rbtree_purge(struct rb_root *root)
+{
+ struct rb_node *p = rb_first(root);
+ unsigned int sum = 0;
+
+ while (p) {
+ struct sk_buff *skb = rb_entry(p, struct sk_buff, rbnode);
+
+ p = rb_next(p);
+ rb_erase(&skb->rbnode, root);
+ while (skb) {
+ struct sk_buff *next = FRAG_CB(skb)->next_frag;
+
+ sum += skb->truesize;
+ kfree_skb(skb);
+ skb = next;
+ }
+ }
+ return sum;
+}
+EXPORT_SYMBOL(inet_frag_rbtree_purge);
+
#ifdef CONFIG_SYSCTL
-static int zero;
+static int dist_min;
static struct ctl_table ip4_frags_ns_ctl_table[] = {
{
.procname = "ipfrag_high_thresh",
.data = &init_net.ipv4.frags.high_thresh,
- .maxlen = sizeof(int),
+ .maxlen = sizeof(unsigned long),
.mode = 0644,
- .proc_handler = proc_dointvec_minmax,
+ .proc_handler = proc_doulongvec_minmax,
.extra1 = &init_net.ipv4.frags.low_thresh
},
{
.procname = "ipfrag_low_thresh",
.data = &init_net.ipv4.frags.low_thresh,
- .maxlen = sizeof(int),
+ .maxlen = sizeof(unsigned long),
.mode = 0644,
- .proc_handler = proc_dointvec_minmax,
- .extra1 = &zero,
+ .proc_handler = proc_doulongvec_minmax,
.extra2 = &init_net.ipv4.frags.high_thresh
},
{
@@ -762,7 +797,7 @@ static struct ctl_table ip4_frags_ns_ctl_table[] = {
.maxlen = sizeof(int),
.mode = 0644,
.proc_handler = proc_dointvec_minmax,
- .extra1 = &zero
+ .extra1 = &dist_min,
},
{ }
};
@@ -844,6 +879,8 @@ static void __init ip4_frags_ctl_register(void)
static int __net_init ipv4_frags_init_net(struct net *net)
{
+ int res;
+
/* Fragment cache limits.
*
* The fragment memory accounting code, (tries to) account for
@@ -868,16 +905,21 @@ static int __net_init ipv4_frags_init_net(struct net *net)
net->ipv4.frags.timeout = IP_FRAG_TIME;
net->ipv4.frags.max_dist = 64;
+ net->ipv4.frags.f = &ip4_frags;
- inet_frags_init_net(&net->ipv4.frags);
-
- return ip4_frags_ns_ctl_register(net);
+ res = inet_frags_init_net(&net->ipv4.frags);
+ if (res < 0)
+ return res;
+ res = ip4_frags_ns_ctl_register(net);
+ if (res < 0)
+ inet_frags_exit_net(&net->ipv4.frags);
+ return res;
}
static void __net_exit ipv4_frags_exit_net(struct net *net)
{
ip4_frags_ns_ctl_unregister(net);
- inet_frags_exit_net(&net->ipv4.frags, &ip4_frags);
+ inet_frags_exit_net(&net->ipv4.frags);
}
static struct pernet_operations ip4_frags_ops = {
@@ -885,17 +927,49 @@ static struct pernet_operations ip4_frags_ops = {
.exit = ipv4_frags_exit_net,
};
+
+static u32 ip4_key_hashfn(const void *data, u32 len, u32 seed)
+{
+ return jhash2(data,
+ sizeof(struct frag_v4_compare_key) / sizeof(u32), seed);
+}
+
+static u32 ip4_obj_hashfn(const void *data, u32 len, u32 seed)
+{
+ const struct inet_frag_queue *fq = data;
+
+ return jhash2((const u32 *)&fq->key.v4,
+ sizeof(struct frag_v4_compare_key) / sizeof(u32), seed);
+}
+
+static int ip4_obj_cmpfn(struct rhashtable_compare_arg *arg, const void *ptr)
+{
+ const struct frag_v4_compare_key *key = arg->key;
+ const struct inet_frag_queue *fq = ptr;
+
+ return !!memcmp(&fq->key, key, sizeof(*key));
+}
+
+static const struct rhashtable_params ip4_rhash_params = {
+ .head_offset = offsetof(struct inet_frag_queue, node),
+ .key_offset = offsetof(struct inet_frag_queue, key),
+ .key_len = sizeof(struct frag_v4_compare_key),
+ .hashfn = ip4_key_hashfn,
+ .obj_hashfn = ip4_obj_hashfn,
+ .obj_cmpfn = ip4_obj_cmpfn,
+ .automatic_shrinking = true,
+};
+
void __init ipfrag_init(void)
{
- ip4_frags_ctl_register();
- register_pernet_subsys(&ip4_frags_ops);
- ip4_frags.hashfn = ip4_hashfn;
ip4_frags.constructor = ip4_frag_init;
ip4_frags.destructor = ip4_frag_free;
ip4_frags.qsize = sizeof(struct ipq);
- ip4_frags.match = ip4_frag_match;
ip4_frags.frag_expire = ip_expire;
ip4_frags.frags_cache_name = ip_frag_cache_name;
+ ip4_frags.rhash_params = ip4_rhash_params;
if (inet_frags_init(&ip4_frags))
panic("IP: failed to allocate ip4_frags cache\n");
+ ip4_frags_ctl_register();
+ register_pernet_subsys(&ip4_frags_ops);
}
diff --git a/net/ipv4/ip_output.c b/net/ipv4/ip_output.c
index e60f9fa..5fcafc8 100644
--- a/net/ipv4/ip_output.c
+++ b/net/ipv4/ip_output.c
@@ -519,6 +519,8 @@ static void ip_copy_metadata(struct sk_buff *to, struct sk_buff *from)
to->dev = from->dev;
to->mark = from->mark;
+ skb_copy_hash(to, from);
+
/* Copy the flags to each fragment. */
IPCB(to)->flags = IPCB(from)->flags;
@@ -1098,7 +1100,8 @@ static int __ip_append_data(struct sock *sk,
if (copy > length)
copy = length;
- if (!(rt->dst.dev->features&NETIF_F_SG)) {
+ if (!(rt->dst.dev->features&NETIF_F_SG) &&
+ skb_tailroom(skb) >= copy) {
unsigned int off;
off = skb->len;
diff --git a/net/ipv4/ip_sockglue.c b/net/ipv4/ip_sockglue.c
index 5ddd649..a5851c0 100644
--- a/net/ipv4/ip_sockglue.c
+++ b/net/ipv4/ip_sockglue.c
@@ -134,19 +134,21 @@ static void ip_cmsg_recv_security(struct msghdr *msg, struct sk_buff *skb)
static void ip_cmsg_recv_dstaddr(struct msghdr *msg, struct sk_buff *skb)
{
struct sockaddr_in sin;
- const struct iphdr *iph = ip_hdr(skb);
- __be16 *ports = (__be16 *)skb_transport_header(skb);
+ __be16 *ports;
+ int end;
- if (skb_transport_offset(skb) + 4 > (int)skb->len)
+ end = skb_transport_offset(skb) + 4;
+ if (end > 0 && !pskb_may_pull(skb, end))
return;
/* All current transport protocols have the port numbers in the
* first four bytes of the transport header and this function is
* written with this assumption in mind.
*/
+ ports = (__be16 *)skb_transport_header(skb);
sin.sin_family = AF_INET;
- sin.sin_addr.s_addr = iph->daddr;
+ sin.sin_addr.s_addr = ip_hdr(skb)->daddr;
sin.sin_port = ports[1];
memset(sin.sin_zero, 0, sizeof(sin.sin_zero));
@@ -503,8 +505,6 @@ int ip_recv_error(struct sock *sk, struct msghdr *msg, int len, int *addr_len)
int err;
int copied;
- WARN_ON_ONCE(sk->sk_family == AF_INET6);
-
err = -EAGAIN;
skb = sock_dequeue_err_skb(sk);
if (!skb)
diff --git a/net/ipv4/ip_tunnel.c b/net/ipv4/ip_tunnel.c
index e1271e75..e6ee6ac 100644
--- a/net/ipv4/ip_tunnel.c
+++ b/net/ipv4/ip_tunnel.c
@@ -261,8 +261,8 @@ static struct net_device *__ip_tunnel_create(struct net *net,
} else {
if (strlen(ops->kind) > (IFNAMSIZ - 3))
goto failed;
- strlcpy(name, ops->kind, IFNAMSIZ);
- strncat(name, "%d", 2);
+ strcpy(name, ops->kind);
+ strcat(name, "%d");
}
ASSERT_RTNL();
@@ -627,6 +627,7 @@ void ip_tunnel_xmit(struct sk_buff *skb, struct net_device *dev,
const struct iphdr *tnl_params, u8 protocol)
{
struct ip_tunnel *tunnel = netdev_priv(dev);
+ unsigned int inner_nhdr_len = 0;
const struct iphdr *inner_iph;
struct flowi4 fl4;
u8 tos, ttl;
@@ -636,6 +637,14 @@ void ip_tunnel_xmit(struct sk_buff *skb, struct net_device *dev,
__be32 dst;
bool connected;
+ /* ensure we can access the inner net header, for several users below */
+ if (skb->protocol == htons(ETH_P_IP))
+ inner_nhdr_len = sizeof(struct iphdr);
+ else if (skb->protocol == htons(ETH_P_IPV6))
+ inner_nhdr_len = sizeof(struct ipv6hdr);
+ if (unlikely(!pskb_may_pull(skb, inner_nhdr_len)))
+ goto tx_error;
+
inner_iph = (const struct iphdr *)skb_inner_network_header(skb);
connected = (tunnel->parms.iph.daddr != 0);
diff --git a/net/ipv4/ip_tunnel_core.c b/net/ipv4/ip_tunnel_core.c
index 0fd1976..2220a1b 100644
--- a/net/ipv4/ip_tunnel_core.c
+++ b/net/ipv4/ip_tunnel_core.c
@@ -80,7 +80,7 @@ void iptunnel_xmit(struct sock *sk, struct rtable *rt, struct sk_buff *skb,
iph->version = 4;
iph->ihl = sizeof(struct iphdr) >> 2;
- iph->frag_off = df;
+ iph->frag_off = ip_mtu_locked(&rt->dst) ? 0 : df;
iph->protocol = proto;
iph->tos = tos;
iph->daddr = dst;
diff --git a/net/ipv4/ip_vti.c b/net/ipv4/ip_vti.c
index b120b9b..cbff0d6 100644
--- a/net/ipv4/ip_vti.c
+++ b/net/ipv4/ip_vti.c
@@ -396,7 +396,6 @@ static int vti_tunnel_init(struct net_device *dev)
memcpy(dev->dev_addr, &iph->saddr, 4);
memcpy(dev->broadcast, &iph->daddr, 4);
- dev->hard_header_len = LL_MAX_HEADER + sizeof(struct iphdr);
dev->mtu = ETH_DATA_LEN;
dev->flags = IFF_NOARP;
dev->addr_len = 4;
diff --git a/net/ipv4/ipconfig.c b/net/ipv4/ipconfig.c
index b23464d..d278b06 100644
--- a/net/ipv4/ipconfig.c
+++ b/net/ipv4/ipconfig.c
@@ -779,6 +779,11 @@ static void __init ic_bootp_init_ext(u8 *e)
*/
static inline void __init ic_bootp_init(void)
{
+ /* Re-initialise all name servers to NONE, in case any were set via the
+ * "ip=" or "nfsaddrs=" kernel command line parameters: any IP addresses
+ * specified there will already have been decoded but are no longer
+ * needed
+ */
ic_nameservers_predef();
dev_add_pack(&bootp_packet_type);
@@ -1401,6 +1406,13 @@ static int __init ip_auto_config(void)
int err;
unsigned int i;
+ /* Initialise all name servers to NONE (but only if the "ip=" or
+ * "nfsaddrs=" kernel command line parameters weren't decoded, otherwise
+ * we'll overwrite the IP addresses specified there)
+ */
+ if (ic_set_manually == 0)
+ ic_nameservers_predef();
+
#ifdef CONFIG_PROC_FS
proc_create("pnp", S_IRUGO, init_net.proc_net, &pnp_seq_fops);
#endif /* CONFIG_PROC_FS */
@@ -1621,6 +1633,7 @@ static int __init ip_auto_config_setup(char *addrs)
return 1;
}
+ /* Initialise all name servers to NONE */
ic_nameservers_predef();
/* Parse string for static IP assignment. */
diff --git a/net/ipv4/ipmr.c b/net/ipv4/ipmr.c
index 742a343..354926e 100644
--- a/net/ipv4/ipmr.c
+++ b/net/ipv4/ipmr.c
@@ -68,6 +68,8 @@
#include <linux/netconf.h>
#include <net/nexthop.h>
+#include <linux/nospec.h>
+
struct ipmr_rule {
struct fib_rule common;
};
@@ -1562,6 +1564,7 @@ int ipmr_compat_ioctl(struct sock *sk, unsigned int cmd, void __user *arg)
return -EFAULT;
if (vr.vifi >= mrt->maxvif)
return -EINVAL;
+ vr.vifi = array_index_nospec(vr.vifi, mrt->maxvif);
read_lock(&mrt_lock);
vif = &mrt->vif_table[vr.vifi];
if (VIF_EXISTS(mrt, vr.vifi)) {
diff --git a/net/ipv4/netfilter/ip_tables.c b/net/ipv4/netfilter/ip_tables.c
index e78f652..4822459 100644
--- a/net/ipv4/netfilter/ip_tables.c
+++ b/net/ipv4/netfilter/ip_tables.c
@@ -554,6 +554,7 @@ find_check_entry(struct ipt_entry *e, struct net *net, const char *name,
return -ENOMEM;
j = 0;
+ memset(&mtpar, 0, sizeof(mtpar));
mtpar.net = net;
mtpar.table = name;
mtpar.entryinfo = &e->ip;
@@ -1912,6 +1913,7 @@ static struct xt_match ipt_builtin_mt[] __read_mostly = {
.checkentry = icmp_checkentry,
.proto = IPPROTO_ICMP,
.family = NFPROTO_IPV4,
+ .me = THIS_MODULE,
},
};
diff --git a/net/ipv4/ping.c b/net/ipv4/ping.c
index 93bfadf..8fa153c 100644
--- a/net/ipv4/ping.c
+++ b/net/ipv4/ping.c
@@ -775,8 +775,10 @@ static int ping_v4_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
ipc.addr = faddr = daddr;
if (ipc.opt && ipc.opt->opt.srr) {
- if (!daddr)
- return -EINVAL;
+ if (!daddr) {
+ err = -EINVAL;
+ goto out_free;
+ }
faddr = ipc.opt->opt.faddr;
}
tos = get_rttos(&ipc, inet);
@@ -842,6 +844,7 @@ static int ping_v4_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
out:
ip_rt_put(rt);
+out_free:
if (free)
kfree(ipc.opt);
if (!err) {
diff --git a/net/ipv4/proc.c b/net/ipv4/proc.c
index 7143ca1..ec48d8ea 100644
--- a/net/ipv4/proc.c
+++ b/net/ipv4/proc.c
@@ -54,7 +54,6 @@
static int sockstat_seq_show(struct seq_file *seq, void *v)
{
struct net *net = seq->private;
- unsigned int frag_mem;
int orphans, sockets;
local_bh_disable();
@@ -74,8 +73,9 @@ static int sockstat_seq_show(struct seq_file *seq, void *v)
sock_prot_inuse_get(net, &udplite_prot));
seq_printf(seq, "RAW: inuse %d\n",
sock_prot_inuse_get(net, &raw_prot));
- frag_mem = ip_frag_mem(net);
- seq_printf(seq, "FRAG: inuse %u memory %u\n", !!frag_mem, frag_mem);
+ seq_printf(seq, "FRAG: inuse %u memory %lu\n",
+ atomic_read(&net->ipv4.frags.rhashtable.nelems),
+ frag_mem_limit(&net->ipv4.frags));
return 0;
}
@@ -134,6 +134,7 @@ static const struct snmp_mib snmp4_ipextstats_list[] = {
SNMP_MIB_ITEM("InECT1Pkts", IPSTATS_MIB_ECT1PKTS),
SNMP_MIB_ITEM("InECT0Pkts", IPSTATS_MIB_ECT0PKTS),
SNMP_MIB_ITEM("InCEPkts", IPSTATS_MIB_CEPKTS),
+ SNMP_MIB_ITEM("ReasmOverlaps", IPSTATS_MIB_REASM_OVERLAPS),
SNMP_MIB_SENTINEL
};
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index e83fdf6..583967b 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -621,6 +621,7 @@ static inline u32 fnhe_hashfun(__be32 daddr)
static void fill_route_from_fnhe(struct rtable *rt, struct fib_nh_exception *fnhe)
{
rt->rt_pmtu = fnhe->fnhe_pmtu;
+ rt->rt_mtu_locked = fnhe->fnhe_mtu_locked;
rt->dst.expires = fnhe->fnhe_expires;
if (fnhe->fnhe_gw) {
@@ -631,7 +632,7 @@ static void fill_route_from_fnhe(struct rtable *rt, struct fib_nh_exception *fnh
}
static void update_or_create_fnhe(struct fib_nh *nh, __be32 daddr, __be32 gw,
- u32 pmtu, unsigned long expires)
+ u32 pmtu, bool lock, unsigned long expires)
{
struct fnhe_hash_bucket *hash;
struct fib_nh_exception *fnhe;
@@ -668,8 +669,10 @@ static void update_or_create_fnhe(struct fib_nh *nh, __be32 daddr, __be32 gw,
fnhe->fnhe_genid = genid;
if (gw)
fnhe->fnhe_gw = gw;
- if (pmtu)
+ if (pmtu) {
fnhe->fnhe_pmtu = pmtu;
+ fnhe->fnhe_mtu_locked = lock;
+ }
fnhe->fnhe_expires = max(1UL, expires);
/* Update all cached dsts too */
rt = rcu_dereference(fnhe->fnhe_rth_input);
@@ -693,6 +696,7 @@ static void update_or_create_fnhe(struct fib_nh *nh, __be32 daddr, __be32 gw,
fnhe->fnhe_daddr = daddr;
fnhe->fnhe_gw = gw;
fnhe->fnhe_pmtu = pmtu;
+ fnhe->fnhe_mtu_locked = lock;
fnhe->fnhe_expires = expires;
/* Exception created; mark the cached routes for the nexthop
@@ -774,7 +778,8 @@ static void __ip_do_redirect(struct rtable *rt, struct sk_buff *skb, struct flow
struct fib_nh *nh = &FIB_RES_NH(res);
update_or_create_fnhe(nh, fl4->daddr, new_gw,
- 0, jiffies + ip_rt_gc_timeout);
+ 0, false,
+ jiffies + ip_rt_gc_timeout);
}
if (kill_route)
rt->dst.obsolete = DST_OBSOLETE_KILL;
@@ -987,15 +992,18 @@ static void __ip_rt_update_pmtu(struct rtable *rt, struct flowi4 *fl4, u32 mtu)
{
struct dst_entry *dst = &rt->dst;
struct fib_result res;
+ bool lock = false;
- if (dst_metric_locked(dst, RTAX_MTU))
+ if (ip_mtu_locked(dst))
return;
if (ipv4_mtu(dst) < mtu)
return;
- if (mtu < ip_rt_min_pmtu)
+ if (mtu < ip_rt_min_pmtu) {
+ lock = true;
mtu = ip_rt_min_pmtu;
+ }
if (rt->rt_pmtu == mtu &&
time_before(jiffies, dst->expires - ip_rt_mtu_expires / 2))
@@ -1005,7 +1013,7 @@ static void __ip_rt_update_pmtu(struct rtable *rt, struct flowi4 *fl4, u32 mtu)
if (fib_lookup(dev_net(dst->dev), fl4, &res, 0) == 0) {
struct fib_nh *nh = &FIB_RES_NH(res);
- update_or_create_fnhe(nh, fl4->daddr, 0, mtu,
+ update_or_create_fnhe(nh, fl4->daddr, 0, mtu, lock,
jiffies + ip_rt_mtu_expires);
}
rcu_read_unlock();
@@ -1262,7 +1270,7 @@ static unsigned int ipv4_mtu(const struct dst_entry *dst)
mtu = READ_ONCE(dst->dev->mtu);
- if (unlikely(dst_metric_locked(dst, RTAX_MTU))) {
+ if (unlikely(ip_mtu_locked(dst))) {
if (rt->rt_uses_gateway && mtu > 576)
mtu = 576;
}
@@ -1487,6 +1495,7 @@ struct rtable *rt_dst_alloc(struct net_device *dev,
rt->rt_is_input = 0;
rt->rt_iif = 0;
rt->rt_pmtu = 0;
+ rt->rt_mtu_locked = 0;
rt->rt_gateway = 0;
rt->rt_uses_gateway = 0;
rt->rt_table_id = 0;
@@ -2410,6 +2419,7 @@ struct dst_entry *ipv4_blackhole_route(struct net *net, struct dst_entry *dst_or
rt->rt_is_input = ort->rt_is_input;
rt->rt_iif = ort->rt_iif;
rt->rt_pmtu = ort->rt_pmtu;
+ rt->rt_mtu_locked = ort->rt_mtu_locked;
rt->rt_genid = rt_genid_ipv4(net);
rt->rt_flags = ort->rt_flags;
@@ -2512,6 +2522,8 @@ static int rt_fill_info(struct net *net, __be32 dst, __be32 src, u32 table_id,
memcpy(metrics, dst_metrics_ptr(&rt->dst), sizeof(metrics));
if (rt->rt_pmtu && expires)
metrics[RTAX_MTU - 1] = rt->rt_pmtu;
+ if (rt->rt_mtu_locked && expires)
+ metrics[RTAX_LOCK - 1] |= BIT(RTAX_MTU);
if (rtnetlink_put_metrics(skb, metrics) < 0)
goto nla_put_failure;
diff --git a/net/ipv4/sysctl_net_ipv4.c b/net/ipv4/sysctl_net_ipv4.c
index ccc484a..adc9ccc 100644
--- a/net/ipv4/sysctl_net_ipv4.c
+++ b/net/ipv4/sysctl_net_ipv4.c
@@ -144,8 +144,9 @@ static int ipv4_ping_group_range(struct ctl_table *table, int write,
if (write && ret == 0) {
low = make_kgid(user_ns, urange[0]);
high = make_kgid(user_ns, urange[1]);
- if (!gid_valid(low) || !gid_valid(high) ||
- (urange[1] < urange[0]) || gid_lt(high, low)) {
+ if (!gid_valid(low) || !gid_valid(high))
+ return -EINVAL;
+ if (urange[1] < urange[0] || gid_lt(high, low)) {
low = make_kgid(&init_user_ns, 1);
high = make_kgid(&init_user_ns, 0);
}
@@ -231,8 +232,9 @@ static int proc_tcp_fastopen_key(struct ctl_table *ctl, int write,
{
struct ctl_table tbl = { .maxlen = (TCP_FASTOPEN_KEY_LENGTH * 2 + 10) };
struct tcp_fastopen_context *ctxt;
- int ret;
u32 user_key[4]; /* 16 bytes, matching TCP_FASTOPEN_KEY_LENGTH */
+ __le32 key[4];
+ int ret, i;
tbl.data = kmalloc(tbl.maxlen, GFP_KERNEL);
if (!tbl.data)
@@ -241,11 +243,14 @@ static int proc_tcp_fastopen_key(struct ctl_table *ctl, int write,
rcu_read_lock();
ctxt = rcu_dereference(tcp_fastopen_ctx);
if (ctxt)
- memcpy(user_key, ctxt->key, TCP_FASTOPEN_KEY_LENGTH);
+ memcpy(key, ctxt->key, TCP_FASTOPEN_KEY_LENGTH);
else
- memset(user_key, 0, sizeof(user_key));
+ memset(key, 0, sizeof(key));
rcu_read_unlock();
+ for (i = 0; i < ARRAY_SIZE(key); i++)
+ user_key[i] = le32_to_cpu(key[i]);
+
snprintf(tbl.data, tbl.maxlen, "%08x-%08x-%08x-%08x",
user_key[0], user_key[1], user_key[2], user_key[3]);
ret = proc_dostring(&tbl, write, buffer, lenp, ppos);
@@ -261,12 +266,16 @@ static int proc_tcp_fastopen_key(struct ctl_table *ctl, int write,
* first invocation of tcp_fastopen_cookie_gen
*/
tcp_fastopen_init_key_once(false);
- tcp_fastopen_reset_cipher(user_key, TCP_FASTOPEN_KEY_LENGTH);
+
+ for (i = 0; i < ARRAY_SIZE(user_key); i++)
+ key[i] = cpu_to_le32(user_key[i]);
+
+ tcp_fastopen_reset_cipher(key, TCP_FASTOPEN_KEY_LENGTH);
}
bad_key:
pr_debug("proc FO key set 0x%x-%x-%x-%x <- 0x%s: %u\n",
- user_key[0], user_key[1], user_key[2], user_key[3],
+ user_key[0], user_key[1], user_key[2], user_key[3],
(char *)tbl.data, ret);
kfree(tbl.data);
return ret;
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index af0b324..452d59b 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -1146,7 +1146,8 @@ int tcp_sendmsg(struct sock *sk, struct msghdr *msg, size_t size)
lock_sock(sk);
flags = msg->msg_flags;
- if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect)) {
+ if (unlikely(flags & MSG_FASTOPEN || inet_sk(sk)->defer_connect) &&
+ !tp->repair) {
err = tcp_sendmsg_fastopen(sk, msg, &copied_syn, size);
if (err == -EINPROGRESS && copied_syn > 0)
goto out;
@@ -1727,7 +1728,7 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
* shouldn't happen.
*/
if (WARN(before(*seq, TCP_SKB_CB(skb)->seq),
- "recvmsg bug: copied %X seq %X rcvnxt %X fl %X\n",
+ "TCP recvmsg seq # bug: copied %X, seq %X, rcvnxt %X, fl %X\n",
*seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt,
flags))
break;
@@ -1742,7 +1743,7 @@ int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, int nonblock,
if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
goto found_fin_ok;
WARN(!(flags & MSG_PEEK),
- "recvmsg bug 2: copied %X seq %X rcvnxt %X fl %X\n",
+ "TCP recvmsg seq # bug 2: copied %X, seq %X, rcvnxt %X, fl %X\n",
*seq, TCP_SKB_CB(skb)->seq, tp->rcv_nxt, flags);
}
@@ -2558,7 +2559,7 @@ static int do_tcp_setsockopt(struct sock *sk, int level,
case TCP_REPAIR_QUEUE:
if (!tp->repair)
err = -EPERM;
- else if (val < TCP_QUEUES_NR)
+ else if ((unsigned int)val < TCP_QUEUES_NR)
tp->repair_queue = val;
else
err = -EINVAL;
@@ -2697,8 +2698,10 @@ static int do_tcp_setsockopt(struct sock *sk, int level,
#ifdef CONFIG_TCP_MD5SIG
case TCP_MD5SIG:
- /* Read the IP->Key mappings from userspace */
- err = tp->af_specific->md5_parse(sk, optval, optlen);
+ if ((1 << sk->sk_state) & (TCPF_CLOSE | TCPF_LISTEN))
+ err = tp->af_specific->md5_parse(sk, optval, optlen);
+ else
+ err = -EINVAL;
break;
#endif
case TCP_USER_TIMEOUT:
@@ -3287,8 +3290,7 @@ int tcp_abort(struct sock *sk, int err)
struct request_sock *req = inet_reqsk(sk);
local_bh_disable();
- inet_csk_reqsk_queue_drop_and_put(req->rsk_listener,
- req);
+ inet_csk_reqsk_queue_drop(req->rsk_listener, req);
local_bh_enable();
return 0;
}
diff --git a/net/ipv4/tcp_bbr.c b/net/ipv4/tcp_bbr.c
index 8ec6053..7e44d23 100644
--- a/net/ipv4/tcp_bbr.c
+++ b/net/ipv4/tcp_bbr.c
@@ -324,6 +324,10 @@ static u32 bbr_target_cwnd(struct sock *sk, u32 bw, int gain)
/* Reduce delayed ACKs by rounding up cwnd to the next even number. */
cwnd = (cwnd + 1) & ~1U;
+ /* Ensure gain cycling gets inflight above BDP even for small BDPs. */
+ if (bbr->mode == BBR_PROBE_BW && gain > BBR_UNIT)
+ cwnd += 2;
+
return cwnd;
}
@@ -773,7 +777,9 @@ static void bbr_update_min_rtt(struct sock *sk, const struct rate_sample *rs)
}
}
}
- bbr->idle_restart = 0;
+ /* Restart after idle ends only once we process a new S/ACK for data */
+ if (rs->delivered > 0)
+ bbr->idle_restart = 0;
}
static void bbr_update_model(struct sock *sk, const struct rate_sample *rs)
diff --git a/net/ipv4/tcp_dctcp.c b/net/ipv4/tcp_dctcp.c
index ab37c67..a08cedf 100644
--- a/net/ipv4/tcp_dctcp.c
+++ b/net/ipv4/tcp_dctcp.c
@@ -55,7 +55,6 @@ struct dctcp {
u32 dctcp_alpha;
u32 next_seq;
u32 ce_state;
- u32 delayed_ack_reserved;
u32 loss_cwnd;
};
@@ -96,7 +95,6 @@ static void dctcp_init(struct sock *sk)
ca->dctcp_alpha = min(dctcp_alpha_on_init, DCTCP_MAX_ALPHA);
- ca->delayed_ack_reserved = 0;
ca->loss_cwnd = 0;
ca->ce_state = 0;
@@ -131,23 +129,14 @@ static void dctcp_ce_state_0_to_1(struct sock *sk)
struct dctcp *ca = inet_csk_ca(sk);
struct tcp_sock *tp = tcp_sk(sk);
- /* State has changed from CE=0 to CE=1 and delayed
- * ACK has not sent yet.
- */
- if (!ca->ce_state && ca->delayed_ack_reserved) {
- u32 tmp_rcv_nxt;
-
- /* Save current rcv_nxt. */
- tmp_rcv_nxt = tp->rcv_nxt;
-
- /* Generate previous ack with CE=0. */
- tp->ecn_flags &= ~TCP_ECN_DEMAND_CWR;
- tp->rcv_nxt = ca->prior_rcv_nxt;
-
- tcp_send_ack(sk);
-
- /* Recover current rcv_nxt. */
- tp->rcv_nxt = tmp_rcv_nxt;
+ if (!ca->ce_state) {
+ /* State has changed from CE=0 to CE=1, force an immediate
+ * ACK to reflect the new CE state. If an ACK was delayed,
+ * send that first to reflect the prior CE state.
+ */
+ if (inet_csk(sk)->icsk_ack.pending & ICSK_ACK_TIMER)
+ __tcp_send_ack(sk, ca->prior_rcv_nxt);
+ tcp_enter_quickack_mode(sk, 1);
}
ca->prior_rcv_nxt = tp->rcv_nxt;
@@ -161,23 +150,14 @@ static void dctcp_ce_state_1_to_0(struct sock *sk)
struct dctcp *ca = inet_csk_ca(sk);
struct tcp_sock *tp = tcp_sk(sk);
- /* State has changed from CE=1 to CE=0 and delayed
- * ACK has not sent yet.
- */
- if (ca->ce_state && ca->delayed_ack_reserved) {
- u32 tmp_rcv_nxt;
-
- /* Save current rcv_nxt. */
- tmp_rcv_nxt = tp->rcv_nxt;
-
- /* Generate previous ack with CE=1. */
- tp->ecn_flags |= TCP_ECN_DEMAND_CWR;
- tp->rcv_nxt = ca->prior_rcv_nxt;
-
- tcp_send_ack(sk);
-
- /* Recover current rcv_nxt. */
- tp->rcv_nxt = tmp_rcv_nxt;
+ if (ca->ce_state) {
+ /* State has changed from CE=1 to CE=0, force an immediate
+ * ACK to reflect the new CE state. If an ACK was delayed,
+ * send that first to reflect the prior CE state.
+ */
+ if (inet_csk(sk)->icsk_ack.pending & ICSK_ACK_TIMER)
+ __tcp_send_ack(sk, ca->prior_rcv_nxt);
+ tcp_enter_quickack_mode(sk, 1);
}
ca->prior_rcv_nxt = tp->rcv_nxt;
@@ -248,25 +228,6 @@ static void dctcp_state(struct sock *sk, u8 new_state)
}
}
-static void dctcp_update_ack_reserved(struct sock *sk, enum tcp_ca_event ev)
-{
- struct dctcp *ca = inet_csk_ca(sk);
-
- switch (ev) {
- case CA_EVENT_DELAYED_ACK:
- if (!ca->delayed_ack_reserved)
- ca->delayed_ack_reserved = 1;
- break;
- case CA_EVENT_NON_DELAYED_ACK:
- if (ca->delayed_ack_reserved)
- ca->delayed_ack_reserved = 0;
- break;
- default:
- /* Don't care for the rest. */
- break;
- }
-}
-
static void dctcp_cwnd_event(struct sock *sk, enum tcp_ca_event ev)
{
switch (ev) {
@@ -276,10 +237,6 @@ static void dctcp_cwnd_event(struct sock *sk, enum tcp_ca_event ev)
case CA_EVENT_ECN_NO_CE:
dctcp_ce_state_1_to_0(sk);
break;
- case CA_EVENT_DELAYED_ACK:
- case CA_EVENT_NON_DELAYED_ACK:
- dctcp_update_ack_reserved(sk, ev);
- break;
default:
/* Don't care for the rest. */
break;
diff --git a/net/ipv4/tcp_illinois.c b/net/ipv4/tcp_illinois.c
index c8e6d86..95ca887 100644
--- a/net/ipv4/tcp_illinois.c
+++ b/net/ipv4/tcp_illinois.c
@@ -6,7 +6,7 @@
* The algorithm is described in:
* "TCP-Illinois: A Loss and Delay-Based Congestion Control Algorithm
* for High-Speed Networks"
- * http://www.ifp.illinois.edu/~srikant/Papers/liubassri06perf.pdf
+ * http://tamerbasar.csl.illinois.edu/LiuBasarSrikantPerfEvalArtJun2008.pdf
*
* Implemented from description in paper and ns-2 simulation.
* Copyright (C) 2007 Stephen Hemminger <shemminger@linux-foundation.org>
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 16d3619..5f6dc5f 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -199,24 +199,27 @@ static void tcp_measure_rcv_mss(struct sock *sk, const struct sk_buff *skb)
}
}
-static void tcp_incr_quickack(struct sock *sk)
+static void tcp_incr_quickack(struct sock *sk, unsigned int max_quickacks)
{
struct inet_connection_sock *icsk = inet_csk(sk);
unsigned int quickacks = tcp_sk(sk)->rcv_wnd / (2 * icsk->icsk_ack.rcv_mss);
if (quickacks == 0)
quickacks = 2;
+ quickacks = min(quickacks, max_quickacks);
if (quickacks > icsk->icsk_ack.quick)
- icsk->icsk_ack.quick = min(quickacks, TCP_MAX_QUICKACKS);
+ icsk->icsk_ack.quick = quickacks;
}
-static void tcp_enter_quickack_mode(struct sock *sk)
+void tcp_enter_quickack_mode(struct sock *sk, unsigned int max_quickacks)
{
struct inet_connection_sock *icsk = inet_csk(sk);
- tcp_incr_quickack(sk);
+
+ tcp_incr_quickack(sk, max_quickacks);
icsk->icsk_ack.pingpong = 0;
icsk->icsk_ack.ato = TCP_ATO_MIN;
}
+EXPORT_SYMBOL(tcp_enter_quickack_mode);
/* Send ACKs quickly, if "quick" count is not exhausted
* and the session is not interactive.
@@ -248,8 +251,10 @@ static void tcp_ecn_withdraw_cwr(struct tcp_sock *tp)
tp->ecn_flags &= ~TCP_ECN_DEMAND_CWR;
}
-static void __tcp_ecn_check_ce(struct tcp_sock *tp, const struct sk_buff *skb)
+static void __tcp_ecn_check_ce(struct sock *sk, const struct sk_buff *skb)
{
+ struct tcp_sock *tp = tcp_sk(sk);
+
switch (TCP_SKB_CB(skb)->ip_dsfield & INET_ECN_MASK) {
case INET_ECN_NOT_ECT:
/* Funny extension: if ECT is not set on a segment,
@@ -257,31 +262,31 @@ static void __tcp_ecn_check_ce(struct tcp_sock *tp, const struct sk_buff *skb)
* it is probably a retransmit.
*/
if (tp->ecn_flags & TCP_ECN_SEEN)
- tcp_enter_quickack_mode((struct sock *)tp);
+ tcp_enter_quickack_mode(sk, 2);
break;
case INET_ECN_CE:
- if (tcp_ca_needs_ecn((struct sock *)tp))
- tcp_ca_event((struct sock *)tp, CA_EVENT_ECN_IS_CE);
+ if (tcp_ca_needs_ecn(sk))
+ tcp_ca_event(sk, CA_EVENT_ECN_IS_CE);
if (!(tp->ecn_flags & TCP_ECN_DEMAND_CWR)) {
/* Better not delay acks, sender can have a very low cwnd */
- tcp_enter_quickack_mode((struct sock *)tp);
+ tcp_enter_quickack_mode(sk, 2);
tp->ecn_flags |= TCP_ECN_DEMAND_CWR;
}
tp->ecn_flags |= TCP_ECN_SEEN;
break;
default:
- if (tcp_ca_needs_ecn((struct sock *)tp))
- tcp_ca_event((struct sock *)tp, CA_EVENT_ECN_NO_CE);
+ if (tcp_ca_needs_ecn(sk))
+ tcp_ca_event(sk, CA_EVENT_ECN_NO_CE);
tp->ecn_flags |= TCP_ECN_SEEN;
break;
}
}
-static void tcp_ecn_check_ce(struct tcp_sock *tp, const struct sk_buff *skb)
+static void tcp_ecn_check_ce(struct sock *sk, const struct sk_buff *skb)
{
- if (tp->ecn_flags & TCP_ECN_OK)
- __tcp_ecn_check_ce(tp, skb);
+ if (tcp_sk(sk)->ecn_flags & TCP_ECN_OK)
+ __tcp_ecn_check_ce(sk, skb);
}
static void tcp_ecn_rcv_synack(struct tcp_sock *tp, const struct tcphdr *th)
@@ -582,8 +587,8 @@ static inline void tcp_rcv_rtt_measure_ts(struct sock *sk,
void tcp_rcv_space_adjust(struct sock *sk)
{
struct tcp_sock *tp = tcp_sk(sk);
+ u32 copied;
int time;
- int copied;
time = tcp_time_stamp - tp->rcvq_space.time;
if (time < (tp->rcv_rtt_est.rtt >> 3) || tp->rcv_rtt_est.rtt == 0)
@@ -605,12 +610,13 @@ void tcp_rcv_space_adjust(struct sock *sk)
if (sysctl_tcp_moderate_rcvbuf &&
!(sk->sk_userlocks & SOCK_RCVBUF_LOCK)) {
- int rcvwin, rcvmem, rcvbuf;
+ int rcvmem, rcvbuf;
+ u64 rcvwin;
/* minimal window to cope with packet losses, assuming
* steady state. Add some cushion because of small variations.
*/
- rcvwin = (copied << 1) + 16 * tp->advmss;
+ rcvwin = ((u64)copied << 1) + 16 * tp->advmss;
/* If rate increased by 25%,
* assume slow start, rcvwin = 3 * copied
@@ -630,12 +636,13 @@ void tcp_rcv_space_adjust(struct sock *sk)
while (tcp_win_from_space(rcvmem) < tp->advmss)
rcvmem += 128;
- rcvbuf = min(rcvwin / tp->advmss * rcvmem, sysctl_tcp_rmem[2]);
+ do_div(rcvwin, tp->advmss);
+ rcvbuf = min_t(u64, rcvwin * rcvmem, sysctl_tcp_rmem[2]);
if (rcvbuf > sk->sk_rcvbuf) {
sk->sk_rcvbuf = rcvbuf;
/* Make the window clamp follow along. */
- tp->window_clamp = rcvwin;
+ tp->window_clamp = tcp_win_from_space(rcvbuf);
}
}
tp->rcvq_space.space = copied;
@@ -673,7 +680,7 @@ static void tcp_event_data_recv(struct sock *sk, struct sk_buff *skb)
/* The _first_ data packet received, initialize
* delayed ACK engine.
*/
- tcp_incr_quickack(sk);
+ tcp_incr_quickack(sk, TCP_MAX_QUICKACKS);
icsk->icsk_ack.ato = TCP_ATO_MIN;
} else {
int m = now - icsk->icsk_ack.lrcvtime;
@@ -689,13 +696,13 @@ static void tcp_event_data_recv(struct sock *sk, struct sk_buff *skb)
/* Too long gap. Apparently sender failed to
* restart window, so that we send ACKs quickly.
*/
- tcp_incr_quickack(sk);
+ tcp_incr_quickack(sk, TCP_MAX_QUICKACKS);
sk_mem_reclaim(sk);
}
}
icsk->icsk_ack.lrcvtime = now;
- tcp_ecn_check_ce(tp, skb);
+ tcp_ecn_check_ce(sk, skb);
if (skb->len >= 128)
tcp_grow_window(sk, skb);
@@ -3235,6 +3242,15 @@ static int tcp_clean_rtx_queue(struct sock *sk, int prior_fackets,
if (tcp_is_reno(tp)) {
tcp_remove_reno_sacks(sk, pkts_acked);
+
+ /* If any of the cumulatively ACKed segments was
+ * retransmitted, non-SACK case cannot confirm that
+ * progress was due to original transmission due to
+ * lack of TCPCB_SACKED_ACKED bits even if some of
+ * the packets may have been never retransmitted.
+ */
+ if (flag & FLAG_RETRANS_DATA_ACKED)
+ flag &= ~FLAG_ORIG_SACK_ACKED;
} else {
int delta;
@@ -3944,11 +3960,8 @@ const u8 *tcp_parse_md5sig_option(const struct tcphdr *th)
int length = (th->doff << 2) - sizeof(*th);
const u8 *ptr = (const u8 *)(th + 1);
- /* If the TCP option is too short, we can short cut */
- if (length < TCPOLEN_MD5SIG)
- return NULL;
-
- while (length > 0) {
+ /* If not enough data remaining, we can short cut */
+ while (length >= TCPOLEN_MD5SIG) {
int opcode = *ptr++;
int opsize;
@@ -4202,7 +4215,7 @@ static void tcp_send_dupack(struct sock *sk, const struct sk_buff *skb)
if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt)) {
NET_INC_STATS(sock_net(sk), LINUX_MIB_DELAYEDACKLOST);
- tcp_enter_quickack_mode(sk);
+ tcp_enter_quickack_mode(sk, TCP_MAX_QUICKACKS);
if (tcp_is_sack(tp) && sysctl_tcp_dsack) {
u32 end_seq = TCP_SKB_CB(skb)->end_seq;
@@ -4358,6 +4371,23 @@ static bool tcp_try_coalesce(struct sock *sk,
return true;
}
+static bool tcp_ooo_try_coalesce(struct sock *sk,
+ struct sk_buff *to,
+ struct sk_buff *from,
+ bool *fragstolen)
+{
+ bool res = tcp_try_coalesce(sk, to, from, fragstolen);
+
+ /* In case tcp_drop() is called later, update to->gso_segs */
+ if (res) {
+ u32 gso_segs = max_t(u16, 1, skb_shinfo(to)->gso_segs) +
+ max_t(u16, 1, skb_shinfo(from)->gso_segs);
+
+ skb_shinfo(to)->gso_segs = min_t(u32, gso_segs, 0xFFFF);
+ }
+ return res;
+}
+
static void tcp_drop(struct sock *sk, struct sk_buff *skb)
{
sk_drops_add(sk, skb);
@@ -4377,7 +4407,7 @@ static void tcp_ofo_queue(struct sock *sk)
p = rb_first(&tp->out_of_order_queue);
while (p) {
- skb = rb_entry(p, struct sk_buff, rbnode);
+ skb = rb_to_skb(p);
if (after(TCP_SKB_CB(skb)->seq, tp->rcv_nxt))
break;
@@ -4441,12 +4471,12 @@ static int tcp_try_rmem_schedule(struct sock *sk, struct sk_buff *skb,
static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
{
struct tcp_sock *tp = tcp_sk(sk);
- struct rb_node **p, *q, *parent;
+ struct rb_node **p, *parent;
struct sk_buff *skb1;
u32 seq, end_seq;
bool fragstolen;
- tcp_ecn_check_ce(tp, skb);
+ tcp_ecn_check_ce(sk, skb);
if (unlikely(tcp_try_rmem_schedule(sk, skb, skb->truesize))) {
NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPOFODROP);
@@ -4481,7 +4511,8 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
/* In the typical case, we are adding an skb to the end of the list.
* Use of ooo_last_skb avoids the O(Log(N)) rbtree lookup.
*/
- if (tcp_try_coalesce(sk, tp->ooo_last_skb, skb, &fragstolen)) {
+ if (tcp_ooo_try_coalesce(sk, tp->ooo_last_skb,
+ skb, &fragstolen)) {
coalesce_done:
tcp_grow_window(sk, skb);
kfree_skb_partial(skb, fragstolen);
@@ -4499,7 +4530,7 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
parent = NULL;
while (*p) {
parent = *p;
- skb1 = rb_entry(parent, struct sk_buff, rbnode);
+ skb1 = rb_to_skb(parent);
if (before(seq, TCP_SKB_CB(skb1)->seq)) {
p = &parent->rb_left;
continue;
@@ -4509,7 +4540,7 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
/* All the bits are present. Drop. */
NET_INC_STATS(sock_net(sk),
LINUX_MIB_TCPOFOMERGE);
- __kfree_skb(skb);
+ tcp_drop(sk, skb);
skb = NULL;
tcp_dsack_set(sk, seq, end_seq);
goto add_sack;
@@ -4528,10 +4559,11 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
TCP_SKB_CB(skb1)->end_seq);
NET_INC_STATS(sock_net(sk),
LINUX_MIB_TCPOFOMERGE);
- __kfree_skb(skb1);
+ tcp_drop(sk, skb1);
goto merge_right;
}
- } else if (tcp_try_coalesce(sk, skb1, skb, &fragstolen)) {
+ } else if (tcp_ooo_try_coalesce(sk, skb1,
+ skb, &fragstolen)) {
goto coalesce_done;
}
p = &parent->rb_right;
@@ -4543,9 +4575,7 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
merge_right:
/* Remove other segments covered by skb. */
- while ((q = rb_next(&skb->rbnode)) != NULL) {
- skb1 = rb_entry(q, struct sk_buff, rbnode);
-
+ while ((skb1 = skb_rb_next(skb)) != NULL) {
if (!after(end_seq, TCP_SKB_CB(skb1)->seq))
break;
if (before(end_seq, TCP_SKB_CB(skb1)->end_seq)) {
@@ -4560,7 +4590,7 @@ static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
tcp_drop(sk, skb1);
}
/* If there is no skb after us, we are the last_skb ! */
- if (!q)
+ if (!skb1)
tp->ooo_last_skb = skb;
add_sack:
@@ -4726,7 +4756,7 @@ static void tcp_data_queue(struct sock *sk, struct sk_buff *skb)
tcp_dsack_set(sk, TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq);
out_of_window:
- tcp_enter_quickack_mode(sk);
+ tcp_enter_quickack_mode(sk, TCP_MAX_QUICKACKS);
inet_csk_schedule_ack(sk);
drop:
tcp_drop(sk, skb);
@@ -4737,8 +4767,6 @@ static void tcp_data_queue(struct sock *sk, struct sk_buff *skb)
if (!before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt + tcp_receive_window(tp)))
goto out_of_window;
- tcp_enter_quickack_mode(sk);
-
if (before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt)) {
/* Partial packet, seq < rcv_next < end_seq */
SOCK_DEBUG(sk, "partial packet: rcv_next %X seq %X - %X\n",
@@ -4763,7 +4791,7 @@ static struct sk_buff *tcp_skb_next(struct sk_buff *skb, struct sk_buff_head *li
if (list)
return !skb_queue_is_last(list, skb) ? skb->next : NULL;
- return rb_entry_safe(rb_next(&skb->rbnode), struct sk_buff, rbnode);
+ return skb_rb_next(skb);
}
static struct sk_buff *tcp_collapse_one(struct sock *sk, struct sk_buff *skb,
@@ -4792,7 +4820,7 @@ static void tcp_rbtree_insert(struct rb_root *root, struct sk_buff *skb)
while (*p) {
parent = *p;
- skb1 = rb_entry(parent, struct sk_buff, rbnode);
+ skb1 = rb_to_skb(parent);
if (before(TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb1)->seq))
p = &parent->rb_left;
else
@@ -4910,27 +4938,22 @@ tcp_collapse(struct sock *sk, struct sk_buff_head *list, struct rb_root *root,
static void tcp_collapse_ofo_queue(struct sock *sk)
{
struct tcp_sock *tp = tcp_sk(sk);
+ u32 range_truesize, sum_tiny = 0;
struct sk_buff *skb, *head;
- struct rb_node *p;
u32 start, end;
- p = rb_first(&tp->out_of_order_queue);
- skb = rb_entry_safe(p, struct sk_buff, rbnode);
+ skb = skb_rb_first(&tp->out_of_order_queue);
new_range:
if (!skb) {
- p = rb_last(&tp->out_of_order_queue);
- /* Note: This is possible p is NULL here. We do not
- * use rb_entry_safe(), as ooo_last_skb is valid only
- * if rbtree is not empty.
- */
- tp->ooo_last_skb = rb_entry(p, struct sk_buff, rbnode);
+ tp->ooo_last_skb = skb_rb_last(&tp->out_of_order_queue);
return;
}
start = TCP_SKB_CB(skb)->seq;
end = TCP_SKB_CB(skb)->end_seq;
+ range_truesize = skb->truesize;
for (head = skb;;) {
- skb = tcp_skb_next(skb, NULL);
+ skb = skb_rb_next(skb);
/* Range is terminated when we see a gap or when
* we are at the queue end.
@@ -4938,11 +4961,20 @@ static void tcp_collapse_ofo_queue(struct sock *sk)
if (!skb ||
after(TCP_SKB_CB(skb)->seq, end) ||
before(TCP_SKB_CB(skb)->end_seq, start)) {
- tcp_collapse(sk, NULL, &tp->out_of_order_queue,
- head, skb, start, end);
+ /* Do not attempt collapsing tiny skbs */
+ if (range_truesize != head->truesize ||
+ end - start >= SKB_WITH_OVERHEAD(SK_MEM_QUANTUM)) {
+ tcp_collapse(sk, NULL, &tp->out_of_order_queue,
+ head, skb, start, end);
+ } else {
+ sum_tiny += range_truesize;
+ if (sum_tiny > sk->sk_rcvbuf >> 3)
+ return;
+ }
goto new_range;
}
+ range_truesize += skb->truesize;
if (unlikely(before(TCP_SKB_CB(skb)->seq, start)))
start = TCP_SKB_CB(skb)->seq;
if (after(TCP_SKB_CB(skb)->end_seq, end))
@@ -4957,6 +4989,7 @@ static void tcp_collapse_ofo_queue(struct sock *sk)
* 2) not add too big latencies if thousands of packets sit there.
* (But if application shrinks SO_RCVBUF, we could still end up
* freeing whole queue here)
+ * 3) Drop at least 12.5 % of sk_rcvbuf to avoid malicious attacks.
*
* Return true if queue has shrunk.
*/
@@ -4964,23 +4997,29 @@ static bool tcp_prune_ofo_queue(struct sock *sk)
{
struct tcp_sock *tp = tcp_sk(sk);
struct rb_node *node, *prev;
+ int goal;
if (RB_EMPTY_ROOT(&tp->out_of_order_queue))
return false;
NET_INC_STATS(sock_net(sk), LINUX_MIB_OFOPRUNED);
+ goal = sk->sk_rcvbuf >> 3;
node = &tp->ooo_last_skb->rbnode;
do {
prev = rb_prev(node);
rb_erase(node, &tp->out_of_order_queue);
- tcp_drop(sk, rb_entry(node, struct sk_buff, rbnode));
- sk_mem_reclaim(sk);
- if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf &&
- !tcp_under_memory_pressure(sk))
- break;
+ goal -= rb_to_skb(node)->truesize;
+ tcp_drop(sk, rb_to_skb(node));
+ if (!prev || goal <= 0) {
+ sk_mem_reclaim(sk);
+ if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf &&
+ !tcp_under_memory_pressure(sk))
+ break;
+ goal = sk->sk_rcvbuf >> 3;
+ }
node = prev;
} while (node);
- tp->ooo_last_skb = rb_entry(prev, struct sk_buff, rbnode);
+ tp->ooo_last_skb = rb_to_skb(prev);
/* Reset SACK state. A conforming SACK implementation will
* do the same at a timeout based retransmit. When a connection
@@ -5012,6 +5051,9 @@ static int tcp_prune_queue(struct sock *sk)
else if (tcp_under_memory_pressure(sk))
tp->rcv_ssthresh = min(tp->rcv_ssthresh, 4U * tp->advmss);
+ if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf)
+ return 0;
+
tcp_collapse_ofo_queue(sk);
if (!skb_queue_empty(&sk->sk_receive_queue))
tcp_collapse(sk, &sk->sk_receive_queue, NULL,
@@ -5802,7 +5844,7 @@ static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
* to stand against the temptation 8) --ANK
*/
inet_csk_schedule_ack(sk);
- tcp_enter_quickack_mode(sk);
+ tcp_enter_quickack_mode(sk, TCP_MAX_QUICKACKS);
inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
TCP_DELACK_MAX, TCP_RTO_MAX);
@@ -5929,11 +5971,13 @@ int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)
if (th->fin)
goto discard;
/* It is possible that we process SYN packets from backlog,
- * so we need to make sure to disable BH right there.
+ * so we need to make sure to disable BH and RCU right there.
*/
+ rcu_read_lock();
local_bh_disable();
acceptable = icsk->icsk_af_ops->conn_request(sk, skb) >= 0;
local_bh_enable();
+ rcu_read_unlock();
if (!acceptable)
return 1;
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 71335ac..9bfa876 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -867,9 +867,11 @@ static int tcp_v4_send_synack(const struct sock *sk, struct dst_entry *dst,
if (skb) {
__tcp_v4_send_check(skb, ireq->ir_loc_addr, ireq->ir_rmt_addr);
+ rcu_read_lock();
err = ip_build_and_send_pkt(skb, sk, ireq->ir_loc_addr,
ireq->ir_rmt_addr,
- ireq_opt_deref(ireq));
+ rcu_dereference(ireq->ireq_opt));
+ rcu_read_unlock();
err = net_xmit_eval(err);
}
@@ -1669,6 +1671,10 @@ int tcp_v4_rcv(struct sk_buff *skb)
reqsk_put(req);
goto discard_it;
}
+ if (tcp_checksum_complete(skb)) {
+ reqsk_put(req);
+ goto csum_error;
+ }
if (unlikely(sk->sk_state != TCP_LISTEN)) {
inet_csk_reqsk_queue_drop_and_put(sk, req);
goto lookup;
@@ -2450,6 +2456,12 @@ static int __net_init tcp_sk_init(struct net *net)
if (res)
goto fail;
sock_set_flag(sk, SOCK_USE_WRITE_QUEUE);
+
+ /* Please enforce IP_DF and IPID==0 for RST and
+ * ACK sent in SYN-RECV and TIME-WAIT state.
+ */
+ inet_sk(sk)->pmtudisc = IP_PMTUDISC_DO;
+
*per_cpu_ptr(net->ipv4.tcp_sk, cpu) = sk;
}
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 830a564..a501b45 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -194,8 +194,9 @@ tcp_timewait_state_process(struct inet_timewait_sock *tw, struct sk_buff *skb,
inet_twsk_deschedule_put(tw);
return TCP_TW_SUCCESS;
}
+ } else {
+ inet_twsk_reschedule(tw, TCP_TIMEWAIT_LEN);
}
- inet_twsk_reschedule(tw, TCP_TIMEWAIT_LEN);
if (tmp_opt.saw_tstamp) {
tcptw->tw_ts_recent = tmp_opt.rcv_tsval;
diff --git a/net/ipv4/tcp_nv.c b/net/ipv4/tcp_nv.c
index e45e2c41..37a3cb9 100644
--- a/net/ipv4/tcp_nv.c
+++ b/net/ipv4/tcp_nv.c
@@ -338,7 +338,7 @@ static void tcpnv_acked(struct sock *sk, const struct ack_sample *sample)
*/
cwnd_by_slope = (u32)
div64_u64(((u64)ca->nv_rtt_max_rate) * ca->nv_min_rtt,
- (u64)(80000 * tp->mss_cache));
+ 80000ULL * tp->mss_cache);
max_win = cwnd_by_slope + nv_pad;
/* If cwnd > max_win, decrease cwnd
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 16a473a..8d31f60 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -174,8 +174,13 @@ static void tcp_event_data_sent(struct tcp_sock *tp,
}
/* Account for an ACK we sent. */
-static inline void tcp_event_ack_sent(struct sock *sk, unsigned int pkts)
+static inline void tcp_event_ack_sent(struct sock *sk, unsigned int pkts,
+ u32 rcv_nxt)
{
+ struct tcp_sock *tp = tcp_sk(sk);
+
+ if (unlikely(rcv_nxt != tp->rcv_nxt))
+ return; /* Special ACK sent by DCTCP to reflect ECN */
tcp_dec_quickack_mode(sk, pkts);
inet_csk_clear_xmit_timer(sk, ICSK_TIME_DACK);
}
@@ -905,8 +910,8 @@ void tcp_wfree(struct sk_buff *skb)
* We are working here with either a clone of the original
* SKB, or a fresh unique copy made by the retransmit engine.
*/
-static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
- gfp_t gfp_mask)
+static int __tcp_transmit_skb(struct sock *sk, struct sk_buff *skb,
+ int clone_it, gfp_t gfp_mask, u32 rcv_nxt)
{
const struct inet_connection_sock *icsk = inet_csk(sk);
struct inet_sock *inet;
@@ -969,7 +974,7 @@ static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
th->source = inet->inet_sport;
th->dest = inet->inet_dport;
th->seq = htonl(tcb->seq);
- th->ack_seq = htonl(tp->rcv_nxt);
+ th->ack_seq = htonl(rcv_nxt);
*(((__be16 *)th) + 6) = htons(((tcp_header_size >> 2) << 12) |
tcb->tcp_flags);
@@ -1010,7 +1015,7 @@ static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
icsk->icsk_af_ops->send_check(sk, skb);
if (likely(tcb->tcp_flags & TCPHDR_ACK))
- tcp_event_ack_sent(sk, tcp_skb_pcount(skb));
+ tcp_event_ack_sent(sk, tcp_skb_pcount(skb), rcv_nxt);
if (skb->len != tcp_header_size) {
tcp_event_data_sent(tp, sk);
@@ -1046,6 +1051,13 @@ static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
return err;
}
+static int tcp_transmit_skb(struct sock *sk, struct sk_buff *skb, int clone_it,
+ gfp_t gfp_mask)
+{
+ return __tcp_transmit_skb(sk, skb, clone_it, gfp_mask,
+ tcp_sk(sk)->rcv_nxt);
+}
+
/* This routine just queues the buffer for sending.
*
* NOTE: probe0 timer is not checked, do not forget tcp_push_pending_frames,
@@ -2334,14 +2346,18 @@ void tcp_send_loss_probe(struct sock *sk)
skb = tcp_write_queue_tail(sk);
}
+ if (unlikely(!skb)) {
+ WARN_ONCE(tp->packets_out,
+ "invalid inflight: %u state %u cwnd %u mss %d\n",
+ tp->packets_out, sk->sk_state, tp->snd_cwnd, mss);
+ inet_csk(sk)->icsk_pending = 0;
+ return;
+ }
+
/* At most one outstanding TLP retransmission. */
if (tp->tlp_high_seq)
goto rearm_timer;
- /* Retransmit last segment. */
- if (WARN_ON(!skb))
- goto rearm_timer;
-
if (skb_still_in_host_queue(sk, skb))
goto rearm_timer;
@@ -2691,8 +2707,10 @@ int __tcp_retransmit_skb(struct sock *sk, struct sk_buff *skb, int segs)
return -EBUSY;
if (before(TCP_SKB_CB(skb)->seq, tp->snd_una)) {
- if (before(TCP_SKB_CB(skb)->end_seq, tp->snd_una))
- BUG();
+ if (unlikely(before(TCP_SKB_CB(skb)->end_seq, tp->snd_una))) {
+ WARN_ON_ONCE(1);
+ return -EINVAL;
+ }
if (tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq))
return -ENOMEM;
}
@@ -3236,6 +3254,7 @@ static void tcp_connect_init(struct sock *sk)
sock_reset_flag(sk, SOCK_DONE);
tp->snd_wnd = 0;
tcp_init_wl(tp, 0);
+ tcp_write_queue_purge(sk);
tp->snd_una = tp->write_seq;
tp->snd_sml = tp->write_seq;
tp->snd_up = tp->write_seq;
@@ -3417,8 +3436,6 @@ void tcp_send_delayed_ack(struct sock *sk)
int ato = icsk->icsk_ack.ato;
unsigned long timeout;
- tcp_ca_event(sk, CA_EVENT_DELAYED_ACK);
-
if (ato > TCP_DELACK_MIN) {
const struct tcp_sock *tp = tcp_sk(sk);
int max_ato = HZ / 2;
@@ -3467,7 +3484,7 @@ void tcp_send_delayed_ack(struct sock *sk)
}
/* This routine sends an ack and also updates the window. */
-void tcp_send_ack(struct sock *sk)
+void __tcp_send_ack(struct sock *sk, u32 rcv_nxt)
{
struct sk_buff *buff;
@@ -3475,8 +3492,6 @@ void tcp_send_ack(struct sock *sk)
if (sk->sk_state == TCP_CLOSE)
return;
- tcp_ca_event(sk, CA_EVENT_NON_DELAYED_ACK);
-
/* We are not putting this on the write queue, so
* tcp_transmit_skb() will set the ownership to this
* sock.
@@ -3505,9 +3520,14 @@ void tcp_send_ack(struct sock *sk)
/* Send it off, this clears delayed acks for us. */
skb_mstamp_get(&buff->skb_mstamp);
- tcp_transmit_skb(sk, buff, 0, (__force gfp_t)0);
+ __tcp_transmit_skb(sk, buff, 0, (__force gfp_t)0, rcv_nxt);
}
-EXPORT_SYMBOL_GPL(tcp_send_ack);
+EXPORT_SYMBOL_GPL(__tcp_send_ack);
+
+void tcp_send_ack(struct sock *sk)
+{
+ __tcp_send_ack(sk, tcp_sk(sk)->rcv_nxt);
+}
/* This routine sends a packet with an out of date sequence
* number. It assumes the other end will try to ack it.
diff --git a/net/ipv4/tcp_probe.c b/net/ipv4/tcp_probe.c
index 3d063eb..f6c50af 100644
--- a/net/ipv4/tcp_probe.c
+++ b/net/ipv4/tcp_probe.c
@@ -117,7 +117,7 @@ static void jtcp_rcv_established(struct sock *sk, struct sk_buff *skb,
(fwmark > 0 && skb->mark == fwmark)) &&
(full || tp->snd_cwnd != tcp_probe.lastcwnd)) {
- spin_lock_bh(&tcp_probe.lock);
+ spin_lock(&tcp_probe.lock);
/* If log fills, just silently drop */
if (tcp_probe_avail() > 1) {
struct tcp_log *p = tcp_probe.log + tcp_probe.head;
@@ -157,7 +157,7 @@ static void jtcp_rcv_established(struct sock *sk, struct sk_buff *skb,
tcp_probe.head = (tcp_probe.head + 1) & (bufsize - 1);
}
tcp_probe.lastcwnd = tp->snd_cwnd;
- spin_unlock_bh(&tcp_probe.lock);
+ spin_unlock(&tcp_probe.lock);
wake_up(&tcp_probe.wait);
}
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index 5af27b9..306d34a 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -987,8 +987,10 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
sock_tx_timestamp(sk, ipc.sockc.tsflags, &ipc.tx_flags);
if (ipc.opt && ipc.opt->opt.srr) {
- if (!daddr)
- return -EINVAL;
+ if (!daddr) {
+ err = -EINVAL;
+ goto out_free;
+ }
faddr = ipc.opt->opt.faddr;
connected = 0;
}
@@ -1096,6 +1098,7 @@ int udp_sendmsg(struct sock *sk, struct msghdr *msg, size_t len)
out:
ip_rt_put(rt);
+out_free:
if (free)
kfree(ipc.opt);
if (!err)
@@ -1733,8 +1736,46 @@ static inline int udp4_csum_init(struct sk_buff *skb, struct udphdr *uh,
/* Note, we are only interested in != 0 or == 0, thus the
* force to int.
*/
- return (__force int)skb_checksum_init_zero_check(skb, proto, uh->check,
- inet_compute_pseudo);
+ err = (__force int)skb_checksum_init_zero_check(skb, proto, uh->check,
+ inet_compute_pseudo);
+ if (err)
+ return err;
+
+ if (skb->ip_summed == CHECKSUM_COMPLETE && !skb->csum_valid) {
+ /* If SW calculated the value, we know it's bad */
+ if (skb->csum_complete_sw)
+ return 1;
+
+ /* HW says the value is bad. Let's validate that.
+ * skb->csum is no longer the full packet checksum,
+ * so don't treat it as such.
+ */
+ skb_checksum_complete_unset(skb);
+ }
+
+ return 0;
+}
+
+/* wrapper for udp_queue_rcv_skb tacking care of csum conversion and
+ * return code conversion for ip layer consumption
+ */
+static int udp_unicast_rcv_skb(struct sock *sk, struct sk_buff *skb,
+ struct udphdr *uh)
+{
+ int ret;
+
+ if (inet_get_convert_csum(sk) && uh->check && !IS_UDPLITE(sk))
+ skb_checksum_try_convert(skb, IPPROTO_UDP, uh->check,
+ inet_compute_pseudo);
+
+ ret = udp_queue_rcv_skb(sk, skb);
+
+ /* a return value > 0 means to resubmit the input, but
+ * it wants the return to be -protocol, or 0
+ */
+ if (ret > 0)
+ return -ret;
+ return 0;
}
/*
@@ -1783,14 +1824,9 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
if (unlikely(sk->sk_rx_dst != dst))
udp_sk_rx_dst_set(sk, dst);
- ret = udp_queue_rcv_skb(sk, skb);
+ ret = udp_unicast_rcv_skb(sk, skb, uh);
sock_put(sk);
- /* a return value > 0 means to resubmit the input, but
- * it wants the return to be -protocol, or 0
- */
- if (ret > 0)
- return -ret;
- return 0;
+ return ret;
}
if (rt->rt_flags & (RTCF_BROADCAST|RTCF_MULTICAST))
@@ -1798,22 +1834,8 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
saddr, daddr, udptable, proto);
sk = __udp4_lib_lookup_skb(skb, uh->source, uh->dest, udptable);
- if (sk) {
- int ret;
-
- if (inet_get_convert_csum(sk) && uh->check && !IS_UDPLITE(sk))
- skb_checksum_try_convert(skb, IPPROTO_UDP, uh->check,
- inet_compute_pseudo);
-
- ret = udp_queue_rcv_skb(sk, skb);
-
- /* a return value > 0 means to resubmit the input, but
- * it wants the return to be -protocol, or 0
- */
- if (ret > 0)
- return -ret;
- return 0;
- }
+ if (sk)
+ return udp_unicast_rcv_skb(sk, skb, uh);
if (!xfrm4_policy_check(NULL, XFRM_POLICY_IN, skb))
goto drop;
diff --git a/net/ipv4/xfrm4_policy.c b/net/ipv4/xfrm4_policy.c
index 7f9a8df..e62d76c9 100644
--- a/net/ipv4/xfrm4_policy.c
+++ b/net/ipv4/xfrm4_policy.c
@@ -101,6 +101,7 @@ static int xfrm4_fill_dst(struct xfrm_dst *xdst, struct net_device *dev,
xdst->u.rt.rt_gateway = rt->rt_gateway;
xdst->u.rt.rt_uses_gateway = rt->rt_uses_gateway;
xdst->u.rt.rt_pmtu = rt->rt_pmtu;
+ xdst->u.rt.rt_mtu_locked = rt->rt_mtu_locked;
xdst->u.rt.rt_table_id = rt->rt_table_id;
INIT_LIST_HEAD(&xdst->u.rt.rt_uncached);
diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c
index 9077060..9877dfa 100644
--- a/net/ipv6/addrconf.c
+++ b/net/ipv6/addrconf.c
@@ -4113,7 +4113,6 @@ static struct inet6_ifaddr *if6_get_first(struct seq_file *seq, loff_t pos)
p++;
continue;
}
- state->offset++;
return ifa;
}
@@ -4137,13 +4136,12 @@ static struct inet6_ifaddr *if6_get_next(struct seq_file *seq,
return ifa;
}
+ state->offset = 0;
while (++state->bucket < IN6_ADDR_HSIZE) {
- state->offset = 0;
hlist_for_each_entry_rcu_bh(ifa,
&inet6_addr_lst[state->bucket], addr_lst) {
if (!net_eq(dev_net(ifa->idev->dev), net))
continue;
- state->offset++;
return ifa;
}
}
@@ -4768,8 +4766,8 @@ static int in6_dump_addrs(struct inet6_dev *idev, struct sk_buff *skb,
/* unicast address incl. temp addr */
list_for_each_entry(ifa, &idev->addr_list, if_list) {
- if (++ip_idx < s_ip_idx)
- continue;
+ if (ip_idx < s_ip_idx)
+ goto next;
err = inet6_fill_ifaddr(skb, ifa,
NETLINK_CB(cb->skb).portid,
cb->nlh->nlmsg_seq,
@@ -4778,6 +4776,8 @@ static int in6_dump_addrs(struct inet6_dev *idev, struct sk_buff *skb,
if (err < 0)
break;
nl_dump_check_consistent(cb, nlmsg_hdr(skb));
+next:
+ ip_idx++;
}
break;
}
diff --git a/net/ipv6/af_inet6.c b/net/ipv6/af_inet6.c
index 1111684..c789a3e 100644
--- a/net/ipv6/af_inet6.c
+++ b/net/ipv6/af_inet6.c
@@ -1062,11 +1062,11 @@ static int __init inet6_init(void)
igmp_fail:
ndisc_cleanup();
ndisc_fail:
- ip6_mr_cleanup();
-icmp_fail:
- unregister_pernet_subsys(&inet6_net_ops);
-ipmr_fail:
icmpv6_cleanup();
+icmp_fail:
+ ip6_mr_cleanup();
+ipmr_fail:
+ unregister_pernet_subsys(&inet6_net_ops);
register_pernet_fail:
sock_unregister(PF_INET6);
rtnl_unregister_all(PF_INET6);
diff --git a/net/ipv6/calipso.c b/net/ipv6/calipso.c
index 8d772fe..9742abf 100644
--- a/net/ipv6/calipso.c
+++ b/net/ipv6/calipso.c
@@ -799,8 +799,7 @@ static int calipso_opt_update(struct sock *sk, struct ipv6_opt_hdr *hop)
{
struct ipv6_txoptions *old = txopt_get(inet6_sk(sk)), *txopts;
- txopts = ipv6_renew_options_kern(sk, old, IPV6_HOPOPTS,
- hop, hop ? ipv6_optlen(hop) : 0);
+ txopts = ipv6_renew_options(sk, old, IPV6_HOPOPTS, hop);
txopt_put(old);
if (IS_ERR(txopts))
return PTR_ERR(txopts);
@@ -1222,8 +1221,7 @@ static int calipso_req_setattr(struct request_sock *req,
if (IS_ERR(new))
return PTR_ERR(new);
- txopts = ipv6_renew_options_kern(sk, req_inet->ipv6_opt, IPV6_HOPOPTS,
- new, new ? ipv6_optlen(new) : 0);
+ txopts = ipv6_renew_options(sk, req_inet->ipv6_opt, IPV6_HOPOPTS, new);
kfree(new);
@@ -1260,8 +1258,7 @@ static void calipso_req_delattr(struct request_sock *req)
if (calipso_opt_del(req_inet->ipv6_opt->hopopt, &new))
return; /* Nothing to do */
- txopts = ipv6_renew_options_kern(sk, req_inet->ipv6_opt, IPV6_HOPOPTS,
- new, new ? ipv6_optlen(new) : 0);
+ txopts = ipv6_renew_options(sk, req_inet->ipv6_opt, IPV6_HOPOPTS, new);
if (!IS_ERR(txopts)) {
txopts = xchg(&req_inet->ipv6_opt, txopts);
diff --git a/net/ipv6/datagram.c b/net/ipv6/datagram.c
index 1778af9..83ce5ac 100644
--- a/net/ipv6/datagram.c
+++ b/net/ipv6/datagram.c
@@ -701,13 +701,16 @@ void ip6_datagram_recv_specific_ctl(struct sock *sk, struct msghdr *msg,
}
if (np->rxopt.bits.rxorigdstaddr) {
struct sockaddr_in6 sin6;
- __be16 *ports = (__be16 *) skb_transport_header(skb);
+ __be16 *ports;
+ int end;
- if (skb_transport_offset(skb) + 4 <= (int)skb->len) {
+ end = skb_transport_offset(skb) + 4;
+ if (end <= 0 || pskb_may_pull(skb, end)) {
/* All current transport protocols have the port numbers in the
* first four bytes of the transport header and this function is
* written with this assumption in mind.
*/
+ ports = (__be16 *)skb_transport_header(skb);
sin6.sin6_family = AF_INET6;
sin6.sin6_addr = ipv6_hdr(skb)->daddr;
diff --git a/net/ipv6/exthdrs.c b/net/ipv6/exthdrs.c
index 139ceb6..b909c77 100644
--- a/net/ipv6/exthdrs.c
+++ b/net/ipv6/exthdrs.c
@@ -760,29 +760,21 @@ ipv6_dup_options(struct sock *sk, struct ipv6_txoptions *opt)
}
EXPORT_SYMBOL_GPL(ipv6_dup_options);
-static int ipv6_renew_option(void *ohdr,
- struct ipv6_opt_hdr __user *newopt, int newoptlen,
- int inherit,
- struct ipv6_opt_hdr **hdr,
- char **p)
+static void ipv6_renew_option(int renewtype,
+ struct ipv6_opt_hdr **dest,
+ struct ipv6_opt_hdr *old,
+ struct ipv6_opt_hdr *new,
+ int newtype, char **p)
{
- if (inherit) {
- if (ohdr) {
- memcpy(*p, ohdr, ipv6_optlen((struct ipv6_opt_hdr *)ohdr));
- *hdr = (struct ipv6_opt_hdr *)*p;
- *p += CMSG_ALIGN(ipv6_optlen(*hdr));
- }
- } else {
- if (newopt) {
- if (copy_from_user(*p, newopt, newoptlen))
- return -EFAULT;
- *hdr = (struct ipv6_opt_hdr *)*p;
- if (ipv6_optlen(*hdr) > newoptlen)
- return -EINVAL;
- *p += CMSG_ALIGN(newoptlen);
- }
- }
- return 0;
+ struct ipv6_opt_hdr *src;
+
+ src = (renewtype == newtype ? new : old);
+ if (!src)
+ return;
+
+ memcpy(*p, src, ipv6_optlen(src));
+ *dest = (struct ipv6_opt_hdr *)*p;
+ *p += CMSG_ALIGN(ipv6_optlen(*dest));
}
/**
@@ -808,13 +800,11 @@ static int ipv6_renew_option(void *ohdr,
*/
struct ipv6_txoptions *
ipv6_renew_options(struct sock *sk, struct ipv6_txoptions *opt,
- int newtype,
- struct ipv6_opt_hdr __user *newopt, int newoptlen)
+ int newtype, struct ipv6_opt_hdr *newopt)
{
int tot_len = 0;
char *p;
struct ipv6_txoptions *opt2;
- int err;
if (opt) {
if (newtype != IPV6_HOPOPTS && opt->hopopt)
@@ -827,8 +817,8 @@ ipv6_renew_options(struct sock *sk, struct ipv6_txoptions *opt,
tot_len += CMSG_ALIGN(ipv6_optlen(opt->dst1opt));
}
- if (newopt && newoptlen)
- tot_len += CMSG_ALIGN(newoptlen);
+ if (newopt)
+ tot_len += CMSG_ALIGN(ipv6_optlen(newopt));
if (!tot_len)
return NULL;
@@ -843,29 +833,19 @@ ipv6_renew_options(struct sock *sk, struct ipv6_txoptions *opt,
opt2->tot_len = tot_len;
p = (char *)(opt2 + 1);
- err = ipv6_renew_option(opt ? opt->hopopt : NULL, newopt, newoptlen,
- newtype != IPV6_HOPOPTS,
- &opt2->hopopt, &p);
- if (err)
- goto out;
-
- err = ipv6_renew_option(opt ? opt->dst0opt : NULL, newopt, newoptlen,
- newtype != IPV6_RTHDRDSTOPTS,
- &opt2->dst0opt, &p);
- if (err)
- goto out;
-
- err = ipv6_renew_option(opt ? opt->srcrt : NULL, newopt, newoptlen,
- newtype != IPV6_RTHDR,
- (struct ipv6_opt_hdr **)&opt2->srcrt, &p);
- if (err)
- goto out;
-
- err = ipv6_renew_option(opt ? opt->dst1opt : NULL, newopt, newoptlen,
- newtype != IPV6_DSTOPTS,
- &opt2->dst1opt, &p);
- if (err)
- goto out;
+ ipv6_renew_option(IPV6_HOPOPTS, &opt2->hopopt,
+ (opt ? opt->hopopt : NULL),
+ newopt, newtype, &p);
+ ipv6_renew_option(IPV6_RTHDRDSTOPTS, &opt2->dst0opt,
+ (opt ? opt->dst0opt : NULL),
+ newopt, newtype, &p);
+ ipv6_renew_option(IPV6_RTHDR,
+ (struct ipv6_opt_hdr **)&opt2->srcrt,
+ (opt ? (struct ipv6_opt_hdr *)opt->srcrt : NULL),
+ newopt, newtype, &p);
+ ipv6_renew_option(IPV6_DSTOPTS, &opt2->dst1opt,
+ (opt ? opt->dst1opt : NULL),
+ newopt, newtype, &p);
opt2->opt_nflen = (opt2->hopopt ? ipv6_optlen(opt2->hopopt) : 0) +
(opt2->dst0opt ? ipv6_optlen(opt2->dst0opt) : 0) +
@@ -873,37 +853,6 @@ ipv6_renew_options(struct sock *sk, struct ipv6_txoptions *opt,
opt2->opt_flen = (opt2->dst1opt ? ipv6_optlen(opt2->dst1opt) : 0);
return opt2;
-out:
- sock_kfree_s(sk, opt2, opt2->tot_len);
- return ERR_PTR(err);
-}
-
-/**
- * ipv6_renew_options_kern - replace a specific ext hdr with a new one.
- *
- * @sk: sock from which to allocate memory
- * @opt: original options
- * @newtype: option type to replace in @opt
- * @newopt: new option of type @newtype to replace (kernel-mem)
- * @newoptlen: length of @newopt
- *
- * See ipv6_renew_options(). The difference is that @newopt is
- * kernel memory, rather than user memory.
- */
-struct ipv6_txoptions *
-ipv6_renew_options_kern(struct sock *sk, struct ipv6_txoptions *opt,
- int newtype, struct ipv6_opt_hdr *newopt,
- int newoptlen)
-{
- struct ipv6_txoptions *ret_val;
- const mm_segment_t old_fs = get_fs();
-
- set_fs(KERNEL_DS);
- ret_val = ipv6_renew_options(sk, opt, newtype,
- (struct ipv6_opt_hdr __user *)newopt,
- newoptlen);
- set_fs(old_fs);
- return ret_val;
}
struct ipv6_txoptions *ipv6_fixup_options(struct ipv6_txoptions *opt_space,
diff --git a/net/ipv6/ip6_checksum.c b/net/ipv6/ip6_checksum.c
index 1dc023c..9d9a16e 100644
--- a/net/ipv6/ip6_checksum.c
+++ b/net/ipv6/ip6_checksum.c
@@ -87,8 +87,24 @@ int udp6_csum_init(struct sk_buff *skb, struct udphdr *uh, int proto)
* Note, we are only interested in != 0 or == 0, thus the
* force to int.
*/
- return (__force int)skb_checksum_init_zero_check(skb, proto, uh->check,
- ip6_compute_pseudo);
+ err = (__force int)skb_checksum_init_zero_check(skb, proto, uh->check,
+ ip6_compute_pseudo);
+ if (err)
+ return err;
+
+ if (skb->ip_summed == CHECKSUM_COMPLETE && !skb->csum_valid) {
+ /* If SW calculated the value, we know it's bad */
+ if (skb->csum_complete_sw)
+ return 1;
+
+ /* HW says the value is bad. Let's validate that.
+ * skb->csum is no longer the full packet checksum,
+ * so don't treat is as such.
+ */
+ skb_checksum_complete_unset(skb);
+ }
+
+ return 0;
}
EXPORT_SYMBOL(udp6_csum_init);
diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c
index 3cdf4dc5..7c539de 100644
--- a/net/ipv6/ip6_offload.c
+++ b/net/ipv6/ip6_offload.c
@@ -113,6 +113,7 @@ static struct sk_buff *ipv6_gso_segment(struct sk_buff *skb,
payload_len = skb->len - nhoff - sizeof(*ipv6h);
ipv6h->payload_len = htons(payload_len);
skb->network_header = (u8 *)ipv6h - skb->head;
+ skb_reset_mac_len(skb);
if (udpfrag) {
int err = ip6_find_1stfragopt(skb, &prevhdr);
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index de0188e..510a83f 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -186,39 +186,37 @@ int ip6_xmit(const struct sock *sk, struct sk_buff *skb, struct flowi6 *fl6,
const struct ipv6_pinfo *np = inet6_sk(sk);
struct in6_addr *first_hop = &fl6->daddr;
struct dst_entry *dst = skb_dst(skb);
+ unsigned int head_room;
struct ipv6hdr *hdr;
u8 proto = fl6->flowi6_proto;
int seg_len = skb->len;
int hlimit = -1;
u32 mtu;
- if (opt) {
- unsigned int head_room;
+ head_room = sizeof(struct ipv6hdr) + LL_RESERVED_SPACE(dst->dev);
+ if (opt)
+ head_room += opt->opt_nflen + opt->opt_flen;
- /* First: exthdrs may take lots of space (~8K for now)
- MAX_HEADER is not enough.
- */
- head_room = opt->opt_nflen + opt->opt_flen;
- seg_len += head_room;
- head_room += sizeof(struct ipv6hdr) + LL_RESERVED_SPACE(dst->dev);
-
- if (skb_headroom(skb) < head_room) {
- struct sk_buff *skb2 = skb_realloc_headroom(skb, head_room);
- if (!skb2) {
- IP6_INC_STATS(net, ip6_dst_idev(skb_dst(skb)),
- IPSTATS_MIB_OUTDISCARDS);
- kfree_skb(skb);
- return -ENOBUFS;
- }
- consume_skb(skb);
- skb = skb2;
- /* skb_set_owner_w() changes sk->sk_wmem_alloc atomically,
- * it is safe to call in our context (socket lock not held)
- */
- skb_set_owner_w(skb, (struct sock *)sk);
+ if (unlikely(skb_headroom(skb) < head_room)) {
+ struct sk_buff *skb2 = skb_realloc_headroom(skb, head_room);
+ if (!skb2) {
+ IP6_INC_STATS(net, ip6_dst_idev(skb_dst(skb)),
+ IPSTATS_MIB_OUTDISCARDS);
+ kfree_skb(skb);
+ return -ENOBUFS;
}
+ if (skb->sk)
+ skb_set_owner_w(skb2, skb->sk);
+ consume_skb(skb);
+ skb = skb2;
+ }
+
+ if (opt) {
+ seg_len += opt->opt_nflen + opt->opt_flen;
+
if (opt->opt_flen)
ipv6_push_frag_opts(skb, opt, &proto);
+
if (opt->opt_nflen)
ipv6_push_nfrag_opts(skb, opt, &proto, &first_hop);
}
@@ -496,7 +494,8 @@ int ip6_forward(struct sk_buff *skb)
send redirects to source routed frames.
We don't send redirects to frames decapsulated from IPsec.
*/
- if (skb->dev == dst->dev && opt->srcrt == 0 && !skb_sec_path(skb)) {
+ if (IP6CB(skb)->iif == dst->dev->ifindex &&
+ opt->srcrt == 0 && !skb_sec_path(skb)) {
struct in6_addr *target = NULL;
struct inet_peer *peer;
struct rt6_info *rt;
@@ -584,6 +583,8 @@ static void ip6_copy_metadata(struct sk_buff *to, struct sk_buff *from)
to->dev = from->dev;
to->mark = from->mark;
+ skb_copy_hash(to, from);
+
#ifdef CONFIG_NET_SCHED
to->tc_index = from->tc_index;
#endif
@@ -1553,7 +1554,8 @@ static int __ip6_append_data(struct sock *sk,
if (copy > length)
copy = length;
- if (!(rt->dst.dev->features&NETIF_F_SG)) {
+ if (!(rt->dst.dev->features&NETIF_F_SG) &&
+ skb_tailroom(skb) >= copy) {
unsigned int off;
off = skb->len;
diff --git a/net/ipv6/ip6_tunnel.c b/net/ipv6/ip6_tunnel.c
index f338848..8855e89 100644
--- a/net/ipv6/ip6_tunnel.c
+++ b/net/ipv6/ip6_tunnel.c
@@ -907,6 +907,7 @@ static int ipxip6_rcv(struct sk_buff *skb, u8 ipproto,
goto drop;
if (!xfrm6_policy_check(NULL, XFRM_POLICY_IN, skb))
goto drop;
+ ipv6h = ipv6_hdr(skb);
if (!ip6_tnl_rcv_ctl(t, &ipv6h->daddr, &ipv6h->saddr))
goto drop;
if (iptunnel_pull_header(skb, 0, tpi->proto, false))
@@ -1133,12 +1134,8 @@ int ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev, __u8 dsfield,
max_headroom += 8;
mtu -= 8;
}
- if (skb->protocol == htons(ETH_P_IPV6)) {
- if (mtu < IPV6_MIN_MTU)
- mtu = IPV6_MIN_MTU;
- } else if (mtu < 576) {
- mtu = 576;
- }
+ mtu = max(mtu, skb->protocol == htons(ETH_P_IPV6) ?
+ IPV6_MIN_MTU : IPV4_MIN_MTU);
if (skb_dst(skb) && !t->parms.collect_md)
skb_dst(skb)->ops->update_pmtu(skb_dst(skb), NULL, skb, mtu);
@@ -1189,11 +1186,6 @@ int ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev, __u8 dsfield,
}
skb_dst_set(skb, dst);
- if (encap_limit >= 0) {
- init_tel_txopt(&opt, encap_limit);
- ipv6_push_nfrag_opts(skb, &opt.ops, &proto, NULL);
- }
-
/* Calculate max headroom for all the headers and adjust
* needed_headroom if necessary.
*/
@@ -1206,6 +1198,11 @@ int ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev, __u8 dsfield,
if (err)
return err;
+ if (encap_limit >= 0) {
+ init_tel_txopt(&opt, encap_limit);
+ ipv6_push_nfrag_opts(skb, &opt.ops, &proto, NULL);
+ }
+
skb_push(skb, sizeof(struct ipv6hdr));
skb_reset_network_header(skb);
ipv6h = ipv6_hdr(skb);
@@ -1230,7 +1227,7 @@ static inline int
ip4ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct ip6_tnl *t = netdev_priv(dev);
- const struct iphdr *iph = ip_hdr(skb);
+ const struct iphdr *iph;
int encap_limit = -1;
struct flowi6 fl6;
__u8 dsfield;
@@ -1238,6 +1235,11 @@ ip4ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev)
u8 tproto;
int err;
+ /* ensure we can access the full inner ip header */
+ if (!pskb_may_pull(skb, sizeof(struct iphdr)))
+ return -1;
+
+ iph = ip_hdr(skb);
memset(&(IPCB(skb)->opt), 0, sizeof(IPCB(skb)->opt));
tproto = ACCESS_ONCE(t->parms.proto);
@@ -1257,7 +1259,7 @@ ip4ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev)
fl6.flowi6_proto = IPPROTO_IPIP;
fl6.daddr = key->u.ipv6.dst;
fl6.flowlabel = key->label;
- dsfield = ip6_tclass(key->label);
+ dsfield = key->tos;
} else {
if (!(t->parms.flags & IP6_TNL_F_IGN_ENCAP_LIMIT))
encap_limit = t->parms.encap_limit;
@@ -1299,7 +1301,7 @@ static inline int
ip6ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev)
{
struct ip6_tnl *t = netdev_priv(dev);
- struct ipv6hdr *ipv6h = ipv6_hdr(skb);
+ struct ipv6hdr *ipv6h;
int encap_limit = -1;
__u16 offset;
struct flowi6 fl6;
@@ -1308,6 +1310,10 @@ ip6ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev)
u8 tproto;
int err;
+ if (unlikely(!pskb_may_pull(skb, sizeof(*ipv6h))))
+ return -1;
+
+ ipv6h = ipv6_hdr(skb);
tproto = ACCESS_ONCE(t->parms.proto);
if ((tproto != IPPROTO_IPV6 && tproto != 0) ||
ip6_tnl_addr_conflict(t, ipv6h))
@@ -1326,7 +1332,7 @@ ip6ip6_tnl_xmit(struct sk_buff *skb, struct net_device *dev)
fl6.flowi6_proto = IPPROTO_IPV6;
fl6.daddr = key->u.ipv6.dst;
fl6.flowlabel = key->label;
- dsfield = ip6_tclass(key->label);
+ dsfield = key->tos;
} else {
offset = ip6_tnl_parse_tlv_enc_lim(skb, skb_network_header(skb));
/* ip6_tnl_parse_tlv_enc_lim() might have reallocated skb->head */
@@ -1976,14 +1982,14 @@ static int ip6_tnl_newlink(struct net *src_net, struct net_device *dev,
{
struct net *net = dev_net(dev);
struct ip6_tnl_net *ip6n = net_generic(net, ip6_tnl_net_id);
- struct ip6_tnl *nt, *t;
struct ip_tunnel_encap ipencap;
+ struct ip6_tnl *nt, *t;
+ int err;
nt = netdev_priv(dev);
if (ip6_tnl_netlink_encap_parms(data, &ipencap)) {
- int err = ip6_tnl_encap_setup(nt, &ipencap);
-
+ err = ip6_tnl_encap_setup(nt, &ipencap);
if (err < 0)
return err;
}
@@ -1999,7 +2005,11 @@ static int ip6_tnl_newlink(struct net *src_net, struct net_device *dev,
return -EEXIST;
}
- return ip6_tnl_create2(dev);
+ err = ip6_tnl_create2(dev);
+ if (!err && tb[IFLA_MTU])
+ ip6_tnl_change_mtu(dev, nla_get_u32(tb[IFLA_MTU]));
+
+ return err;
}
static int ip6_tnl_changelink(struct net_device *dev, struct nlattr *tb[],
diff --git a/net/ipv6/ip6_udp_tunnel.c b/net/ipv6/ip6_udp_tunnel.c
index b283f29..caad40d 100644
--- a/net/ipv6/ip6_udp_tunnel.c
+++ b/net/ipv6/ip6_udp_tunnel.c
@@ -15,7 +15,7 @@
int udp_sock_create6(struct net *net, struct udp_port_cfg *cfg,
struct socket **sockp)
{
- struct sockaddr_in6 udp6_addr;
+ struct sockaddr_in6 udp6_addr = {};
int err;
struct socket *sock = NULL;
@@ -42,6 +42,7 @@ int udp_sock_create6(struct net *net, struct udp_port_cfg *cfg,
goto error;
if (cfg->peer_udp_port) {
+ memset(&udp6_addr, 0, sizeof(udp6_addr));
udp6_addr.sin6_family = AF_INET6;
memcpy(&udp6_addr.sin6_addr, &cfg->peer_ip6,
sizeof(udp6_addr.sin6_addr));
diff --git a/net/ipv6/ip6_vti.c b/net/ipv6/ip6_vti.c
index 5ae1681..c88505d 100644
--- a/net/ipv6/ip6_vti.c
+++ b/net/ipv6/ip6_vti.c
@@ -318,6 +318,7 @@ static int vti6_rcv(struct sk_buff *skb)
return 0;
}
+ ipv6h = ipv6_hdr(skb);
if (!ip6_tnl_rcv_ctl(t, &ipv6h->daddr, &ipv6h->saddr)) {
t->dev->stats.rx_dropped++;
rcu_read_unlock();
@@ -480,12 +481,8 @@ vti6_xmit(struct sk_buff *skb, struct net_device *dev, struct flowi *fl)
goto tx_err_dst_release;
}
- skb_scrub_packet(skb, !net_eq(t->net, dev_net(dev)));
- skb_dst_set(skb, dst);
- skb->dev = skb_dst(skb)->dev;
-
mtu = dst_mtu(dst);
- if (!skb->ignore_df && skb->len > mtu) {
+ if (skb->len > mtu) {
skb_dst(skb)->ops->update_pmtu(dst, NULL, skb, mtu);
if (skb->protocol == htons(ETH_P_IPV6)) {
@@ -498,9 +495,14 @@ vti6_xmit(struct sk_buff *skb, struct net_device *dev, struct flowi *fl)
htonl(mtu));
}
- return -EMSGSIZE;
+ err = -EMSGSIZE;
+ goto tx_err_dst_release;
}
+ skb_scrub_packet(skb, !net_eq(t->net, dev_net(dev)));
+ skb_dst_set(skb, dst);
+ skb->dev = skb_dst(skb)->dev;
+
err = dst_output(t->net, skb->sk, skb);
if (net_xmit_eval(err) == 0) {
struct pcpu_sw_netstats *tstats = this_cpu_ptr(dev->tstats);
diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c
index a30e7e9..ad597b4 100644
--- a/net/ipv6/ip6mr.c
+++ b/net/ipv6/ip6mr.c
@@ -72,6 +72,8 @@ struct mr6_table {
#endif
};
+#include <linux/nospec.h>
+
struct ip6mr_rule {
struct fib_rule common;
};
@@ -1789,7 +1791,8 @@ int ip6_mroute_setsockopt(struct sock *sk, int optname, char __user *optval, uns
ret = 0;
if (!ip6mr_new_table(net, v))
ret = -ENOMEM;
- raw6_sk(sk)->ip6mr_table = v;
+ else
+ raw6_sk(sk)->ip6mr_table = v;
rtnl_unlock();
return ret;
}
@@ -1872,6 +1875,7 @@ int ip6mr_ioctl(struct sock *sk, int cmd, void __user *arg)
return -EFAULT;
if (vr.mifi >= mrt->maxvif)
return -EINVAL;
+ vr.mifi = array_index_nospec(vr.mifi, mrt->maxvif);
read_lock(&mrt_lock);
vif = &mrt->vif6_table[vr.mifi];
if (MIF_EXISTS(mrt, vr.mifi)) {
@@ -1946,6 +1950,7 @@ int ip6mr_compat_ioctl(struct sock *sk, unsigned int cmd, void __user *arg)
return -EFAULT;
if (vr.mifi >= mrt->maxvif)
return -EINVAL;
+ vr.mifi = array_index_nospec(vr.mifi, mrt->maxvif);
read_lock(&mrt_lock);
vif = &mrt->vif6_table[vr.mifi];
if (MIF_EXISTS(mrt, vr.mifi)) {
diff --git a/net/ipv6/ipv6_sockglue.c b/net/ipv6/ipv6_sockglue.c
index c66b9a8..81fd35e 100644
--- a/net/ipv6/ipv6_sockglue.c
+++ b/net/ipv6/ipv6_sockglue.c
@@ -390,6 +390,12 @@ static int do_ipv6_setsockopt(struct sock *sk, int level, int optname,
case IPV6_DSTOPTS:
{
struct ipv6_txoptions *opt;
+ struct ipv6_opt_hdr *new = NULL;
+
+ /* hop-by-hop / destination options are privileged option */
+ retv = -EPERM;
+ if (optname != IPV6_RTHDR && !ns_capable(net->user_ns, CAP_NET_RAW))
+ break;
/* remove any sticky options header with a zero option
* length, per RFC3542.
@@ -401,17 +407,22 @@ static int do_ipv6_setsockopt(struct sock *sk, int level, int optname,
else if (optlen < sizeof(struct ipv6_opt_hdr) ||
optlen & 0x7 || optlen > 8 * 255)
goto e_inval;
-
- /* hop-by-hop / destination options are privileged option */
- retv = -EPERM;
- if (optname != IPV6_RTHDR && !ns_capable(net->user_ns, CAP_NET_RAW))
- break;
+ else {
+ new = memdup_user(optval, optlen);
+ if (IS_ERR(new)) {
+ retv = PTR_ERR(new);
+ break;
+ }
+ if (unlikely(ipv6_optlen(new) > optlen)) {
+ kfree(new);
+ goto e_inval;
+ }
+ }
opt = rcu_dereference_protected(np->opt,
lockdep_sock_is_held(sk));
- opt = ipv6_renew_options(sk, opt, optname,
- (struct ipv6_opt_hdr __user *)optval,
- optlen);
+ opt = ipv6_renew_options(sk, opt, optname, new);
+ kfree(new);
if (IS_ERR(opt)) {
retv = PTR_ERR(opt);
break;
diff --git a/net/ipv6/mcast.c b/net/ipv6/mcast.c
index ca8fac6..40262ab 100644
--- a/net/ipv6/mcast.c
+++ b/net/ipv6/mcast.c
@@ -771,8 +771,7 @@ static void mld_del_delrec(struct inet6_dev *idev, struct ifmcaddr6 *im)
if (pmc) {
im->idev = pmc->idev;
im->mca_crcount = idev->mc_qrv;
- im->mca_sfmode = pmc->mca_sfmode;
- if (pmc->mca_sfmode == MCAST_INCLUDE) {
+ if (im->mca_sfmode == MCAST_INCLUDE) {
im->mca_tomb = pmc->mca_tomb;
im->mca_sources = pmc->mca_sources;
for (psf = im->mca_sources; psf; psf = psf->sf_next)
@@ -2085,7 +2084,8 @@ void ipv6_mc_dad_complete(struct inet6_dev *idev)
mld_send_initial_cr(idev);
idev->mc_dad_count--;
if (idev->mc_dad_count)
- mld_dad_start_timer(idev, idev->mc_maxdelay);
+ mld_dad_start_timer(idev,
+ unsolicited_report_interval(idev));
}
}
@@ -2097,7 +2097,8 @@ static void mld_dad_timer_expire(unsigned long data)
if (idev->mc_dad_count) {
idev->mc_dad_count--;
if (idev->mc_dad_count)
- mld_dad_start_timer(idev, idev->mc_maxdelay);
+ mld_dad_start_timer(idev,
+ unsolicited_report_interval(idev));
}
in6_dev_put(idev);
}
@@ -2412,17 +2413,17 @@ static int ip6_mc_leave_src(struct sock *sk, struct ipv6_mc_socklist *iml,
{
int err;
- /* callers have the socket lock and rtnl lock
- * so no other readers or writers of iml or its sflist
- */
+ write_lock_bh(&iml->sflock);
if (!iml->sflist) {
/* any-source empty exclude case */
- return ip6_mc_del_src(idev, &iml->addr, iml->sfmode, 0, NULL, 0);
+ err = ip6_mc_del_src(idev, &iml->addr, iml->sfmode, 0, NULL, 0);
+ } else {
+ err = ip6_mc_del_src(idev, &iml->addr, iml->sfmode,
+ iml->sflist->sl_count, iml->sflist->sl_addr, 0);
+ sock_kfree_s(sk, iml->sflist, IP6_SFLSIZE(iml->sflist->sl_max));
+ iml->sflist = NULL;
}
- err = ip6_mc_del_src(idev, &iml->addr, iml->sfmode,
- iml->sflist->sl_count, iml->sflist->sl_addr, 0);
- sock_kfree_s(sk, iml->sflist, IP6_SFLSIZE(iml->sflist->sl_max));
- iml->sflist = NULL;
+ write_unlock_bh(&iml->sflock);
return err;
}
@@ -2455,7 +2456,8 @@ static void mld_ifc_timer_expire(unsigned long data)
if (idev->mc_ifc_count) {
idev->mc_ifc_count--;
if (idev->mc_ifc_count)
- mld_ifc_start_timer(idev, idev->mc_maxdelay);
+ mld_ifc_start_timer(idev,
+ unsolicited_report_interval(idev));
}
in6_dev_put(idev);
}
diff --git a/net/ipv6/ndisc.c b/net/ipv6/ndisc.c
index 52236be..14750ba 100644
--- a/net/ipv6/ndisc.c
+++ b/net/ipv6/ndisc.c
@@ -1540,6 +1540,12 @@ void ndisc_send_redirect(struct sk_buff *skb, const struct in6_addr *target)
ops_data_buf[NDISC_OPS_REDIRECT_DATA_SPACE], *ops_data = NULL;
bool ret;
+ if (netif_is_l3_master(skb->dev)) {
+ dev = __dev_get_by_index(dev_net(skb->dev), IPCB(skb)->iif);
+ if (!dev)
+ return;
+ }
+
if (ipv6_get_lladdr(dev, &saddr_buf, IFA_F_TENTATIVE)) {
ND_PRINTK(2, warn, "Redirect: no link-local address on %s\n",
dev->name);
@@ -1688,10 +1694,9 @@ int ndisc_rcv(struct sk_buff *skb)
return 0;
}
- memset(NEIGH_CB(skb), 0, sizeof(struct neighbour_cb));
-
switch (msg->icmph.icmp6_type) {
case NDISC_NEIGHBOUR_SOLICITATION:
+ memset(NEIGH_CB(skb), 0, sizeof(struct neighbour_cb));
ndisc_recv_ns(skb);
break;
diff --git a/net/ipv6/netfilter/ip6_tables.c b/net/ipv6/netfilter/ip6_tables.c
index aa82858..671eb32 100644
--- a/net/ipv6/netfilter/ip6_tables.c
+++ b/net/ipv6/netfilter/ip6_tables.c
@@ -593,6 +593,7 @@ find_check_entry(struct ip6t_entry *e, struct net *net, const char *name,
return -ENOMEM;
j = 0;
+ memset(&mtpar, 0, sizeof(mtpar));
mtpar.net = net;
mtpar.table = name;
mtpar.entryinfo = &e->ipv6;
@@ -1942,6 +1943,7 @@ static struct xt_match ip6t_builtin_mt[] __read_mostly = {
.checkentry = icmp6_checkentry,
.proto = IPPROTO_ICMPV6,
.family = NFPROTO_IPV6,
+ .me = THIS_MODULE,
},
};
diff --git a/net/ipv6/netfilter/nf_conntrack_reasm.c b/net/ipv6/netfilter/nf_conntrack_reasm.c
index 64ec233..e461853 100644
--- a/net/ipv6/netfilter/nf_conntrack_reasm.c
+++ b/net/ipv6/netfilter/nf_conntrack_reasm.c
@@ -63,7 +63,6 @@ struct nf_ct_frag6_skb_cb
static struct inet_frags nf_frags;
#ifdef CONFIG_SYSCTL
-static int zero;
static struct ctl_table nf_ct_frag6_sysctl_table[] = {
{
@@ -76,18 +75,17 @@ static struct ctl_table nf_ct_frag6_sysctl_table[] = {
{
.procname = "nf_conntrack_frag6_low_thresh",
.data = &init_net.nf_frag.frags.low_thresh,
- .maxlen = sizeof(unsigned int),
+ .maxlen = sizeof(unsigned long),
.mode = 0644,
- .proc_handler = proc_dointvec_minmax,
- .extra1 = &zero,
+ .proc_handler = proc_doulongvec_minmax,
.extra2 = &init_net.nf_frag.frags.high_thresh
},
{
.procname = "nf_conntrack_frag6_high_thresh",
.data = &init_net.nf_frag.frags.high_thresh,
- .maxlen = sizeof(unsigned int),
+ .maxlen = sizeof(unsigned long),
.mode = 0644,
- .proc_handler = proc_dointvec_minmax,
+ .proc_handler = proc_doulongvec_minmax,
.extra1 = &init_net.nf_frag.frags.low_thresh
},
{ }
@@ -117,7 +115,7 @@ static int nf_ct_frag6_sysctl_register(struct net *net)
if (hdr == NULL)
goto err_reg;
- net->nf_frag.sysctl.frags_hdr = hdr;
+ net->nf_frag_frags_hdr = hdr;
return 0;
err_reg:
@@ -131,8 +129,8 @@ static void __net_exit nf_ct_frags6_sysctl_unregister(struct net *net)
{
struct ctl_table *table;
- table = net->nf_frag.sysctl.frags_hdr->ctl_table_arg;
- unregister_net_sysctl_table(net->nf_frag.sysctl.frags_hdr);
+ table = net->nf_frag_frags_hdr->ctl_table_arg;
+ unregister_net_sysctl_table(net->nf_frag_frags_hdr);
if (!net_eq(net, &init_net))
kfree(table);
}
@@ -152,23 +150,6 @@ static inline u8 ip6_frag_ecn(const struct ipv6hdr *ipv6h)
return 1 << (ipv6_get_dsfield(ipv6h) & INET_ECN_MASK);
}
-static unsigned int nf_hash_frag(__be32 id, const struct in6_addr *saddr,
- const struct in6_addr *daddr)
-{
- net_get_random_once(&nf_frags.rnd, sizeof(nf_frags.rnd));
- return jhash_3words(ipv6_addr_hash(saddr), ipv6_addr_hash(daddr),
- (__force u32)id, nf_frags.rnd);
-}
-
-
-static unsigned int nf_hashfn(const struct inet_frag_queue *q)
-{
- const struct frag_queue *nq;
-
- nq = container_of(q, struct frag_queue, q);
- return nf_hash_frag(nq->id, &nq->saddr, &nq->daddr);
-}
-
static void nf_ct_frag6_expire(unsigned long data)
{
struct frag_queue *fq;
@@ -177,34 +158,26 @@ static void nf_ct_frag6_expire(unsigned long data)
fq = container_of((struct inet_frag_queue *)data, struct frag_queue, q);
net = container_of(fq->q.net, struct net, nf_frag.frags);
- ip6_expire_frag_queue(net, fq, &nf_frags);
+ ip6_expire_frag_queue(net, fq);
}
/* Creation primitives. */
-static inline struct frag_queue *fq_find(struct net *net, __be32 id,
- u32 user, struct in6_addr *src,
- struct in6_addr *dst, int iif, u8 ecn)
+static struct frag_queue *fq_find(struct net *net, __be32 id, u32 user,
+ const struct ipv6hdr *hdr, int iif)
{
+ struct frag_v6_compare_key key = {
+ .id = id,
+ .saddr = hdr->saddr,
+ .daddr = hdr->daddr,
+ .user = user,
+ .iif = iif,
+ };
struct inet_frag_queue *q;
- struct ip6_create_arg arg;
- unsigned int hash;
- arg.id = id;
- arg.user = user;
- arg.src = src;
- arg.dst = dst;
- arg.iif = iif;
- arg.ecn = ecn;
-
- local_bh_disable();
- hash = nf_hash_frag(id, src, dst);
-
- q = inet_frag_find(&net->nf_frag.frags, &nf_frags, &arg, hash);
- local_bh_enable();
- if (IS_ERR_OR_NULL(q)) {
- inet_frag_maybe_warn_overflow(q, pr_fmt());
+ q = inet_frag_find(&net->nf_frag.frags, &key);
+ if (!q)
return NULL;
- }
+
return container_of(q, struct frag_queue, q);
}
@@ -263,7 +236,7 @@ static int nf_ct_frag6_queue(struct frag_queue *fq, struct sk_buff *skb,
* this case. -DaveM
*/
pr_debug("end of fragment not rounded to 8 bytes.\n");
- inet_frag_kill(&fq->q, &nf_frags);
+ inet_frag_kill(&fq->q);
return -EPROTO;
}
if (end > fq->q.len) {
@@ -356,7 +329,7 @@ static int nf_ct_frag6_queue(struct frag_queue *fq, struct sk_buff *skb,
return 0;
discard_fq:
- inet_frag_kill(&fq->q, &nf_frags);
+ inet_frag_kill(&fq->q);
err:
return -EINVAL;
}
@@ -378,7 +351,7 @@ nf_ct_frag6_reasm(struct frag_queue *fq, struct sk_buff *prev, struct net_devic
int payload_len;
u8 ecn;
- inet_frag_kill(&fq->q, &nf_frags);
+ inet_frag_kill(&fq->q);
WARN_ON(head == NULL);
WARN_ON(NFCT_FRAG6_CB(head)->offset != 0);
@@ -479,6 +452,7 @@ nf_ct_frag6_reasm(struct frag_queue *fq, struct sk_buff *prev, struct net_devic
else if (head->ip_summed == CHECKSUM_COMPLETE)
head->csum = csum_add(head->csum, fp->csum);
head->truesize += fp->truesize;
+ fp->sk = NULL;
}
sub_frag_mem_limit(fq->q.net, head->truesize);
@@ -497,6 +471,7 @@ nf_ct_frag6_reasm(struct frag_queue *fq, struct sk_buff *prev, struct net_devic
head->csum);
fq->q.fragments = NULL;
+ fq->q.rb_fragments = RB_ROOT;
fq->q.fragments_tail = NULL;
return true;
@@ -591,9 +566,13 @@ int nf_ct_frag6_gather(struct net *net, struct sk_buff *skb, u32 user)
hdr = ipv6_hdr(skb);
fhdr = (struct frag_hdr *)skb_transport_header(skb);
+ if (skb->len - skb_network_offset(skb) < IPV6_MIN_MTU &&
+ fhdr->frag_off & htons(IP6_MF))
+ return -EINVAL;
+
skb_orphan(skb);
- fq = fq_find(net, fhdr->identification, user, &hdr->saddr, &hdr->daddr,
- skb->dev ? skb->dev->ifindex : 0, ip6_frag_ecn(hdr));
+ fq = fq_find(net, fhdr->identification, user, hdr,
+ skb->dev ? skb->dev->ifindex : 0);
if (fq == NULL) {
pr_debug("Can't find and can't create new queue\n");
return -ENOMEM;
@@ -621,25 +600,33 @@ int nf_ct_frag6_gather(struct net *net, struct sk_buff *skb, u32 user)
out_unlock:
spin_unlock_bh(&fq->q.lock);
- inet_frag_put(&fq->q, &nf_frags);
+ inet_frag_put(&fq->q);
return ret;
}
EXPORT_SYMBOL_GPL(nf_ct_frag6_gather);
static int nf_ct_net_init(struct net *net)
{
+ int res;
+
net->nf_frag.frags.high_thresh = IPV6_FRAG_HIGH_THRESH;
net->nf_frag.frags.low_thresh = IPV6_FRAG_LOW_THRESH;
net->nf_frag.frags.timeout = IPV6_FRAG_TIMEOUT;
- inet_frags_init_net(&net->nf_frag.frags);
+ net->nf_frag.frags.f = &nf_frags;
- return nf_ct_frag6_sysctl_register(net);
+ res = inet_frags_init_net(&net->nf_frag.frags);
+ if (res < 0)
+ return res;
+ res = nf_ct_frag6_sysctl_register(net);
+ if (res < 0)
+ inet_frags_exit_net(&net->nf_frag.frags);
+ return res;
}
static void nf_ct_net_exit(struct net *net)
{
nf_ct_frags6_sysctl_unregister(net);
- inet_frags_exit_net(&net->nf_frag.frags, &nf_frags);
+ inet_frags_exit_net(&net->nf_frag.frags);
}
static struct pernet_operations nf_ct_net_ops = {
@@ -651,13 +638,12 @@ int nf_ct_frag6_init(void)
{
int ret = 0;
- nf_frags.hashfn = nf_hashfn;
nf_frags.constructor = ip6_frag_init;
nf_frags.destructor = NULL;
nf_frags.qsize = sizeof(struct frag_queue);
- nf_frags.match = ip6_frag_match;
nf_frags.frag_expire = nf_ct_frag6_expire;
nf_frags.frags_cache_name = nf_frags_cache_name;
+ nf_frags.rhash_params = ip6_rhash_params;
ret = inet_frags_init(&nf_frags);
if (ret)
goto out;
diff --git a/net/ipv6/proc.c b/net/ipv6/proc.c
index e88bcb8..dc04c02 100644
--- a/net/ipv6/proc.c
+++ b/net/ipv6/proc.c
@@ -38,7 +38,6 @@
static int sockstat6_seq_show(struct seq_file *seq, void *v)
{
struct net *net = seq->private;
- unsigned int frag_mem = ip6_frag_mem(net);
seq_printf(seq, "TCP6: inuse %d\n",
sock_prot_inuse_get(net, &tcpv6_prot));
@@ -48,7 +47,9 @@ static int sockstat6_seq_show(struct seq_file *seq, void *v)
sock_prot_inuse_get(net, &udplitev6_prot));
seq_printf(seq, "RAW6: inuse %d\n",
sock_prot_inuse_get(net, &rawv6_prot));
- seq_printf(seq, "FRAG6: inuse %u memory %u\n", !!frag_mem, frag_mem);
+ seq_printf(seq, "FRAG6: inuse %u memory %lu\n",
+ atomic_read(&net->ipv6.frags.rhashtable.nelems),
+ frag_mem_limit(&net->ipv6.frags));
return 0;
}
diff --git a/net/ipv6/raw.c b/net/ipv6/raw.c
index ff701ce..55d284a 100644
--- a/net/ipv6/raw.c
+++ b/net/ipv6/raw.c
@@ -645,8 +645,6 @@ static int rawv6_send_hdrinc(struct sock *sk, struct msghdr *msg, int length,
skb->protocol = htons(ETH_P_IPV6);
skb->priority = sk->sk_priority;
skb->mark = sk->sk_mark;
- skb_dst_set(skb, &rt->dst);
- *dstp = NULL;
skb_put(skb, length);
skb_reset_network_header(skb);
@@ -656,8 +654,14 @@ static int rawv6_send_hdrinc(struct sock *sk, struct msghdr *msg, int length,
skb->transport_header = skb->network_header;
err = memcpy_from_msg(iph, msg, length);
- if (err)
- goto error_fault;
+ if (err) {
+ err = -EFAULT;
+ kfree_skb(skb);
+ goto error;
+ }
+
+ skb_dst_set(skb, &rt->dst);
+ *dstp = NULL;
/* if egress device is enslaved to an L3 master device pass the
* skb to its handler for processing
@@ -666,21 +670,28 @@ static int rawv6_send_hdrinc(struct sock *sk, struct msghdr *msg, int length,
if (unlikely(!skb))
return 0;
+ /* Acquire rcu_read_lock() in case we need to use rt->rt6i_idev
+ * in the error path. Since skb has been freed, the dst could
+ * have been queued for deletion.
+ */
+ rcu_read_lock();
IP6_UPD_PO_STATS(net, rt->rt6i_idev, IPSTATS_MIB_OUT, skb->len);
err = NF_HOOK(NFPROTO_IPV6, NF_INET_LOCAL_OUT, net, sk, skb,
NULL, rt->dst.dev, dst_output);
if (err > 0)
err = net_xmit_errno(err);
- if (err)
- goto error;
+ if (err) {
+ IP6_INC_STATS(net, rt->rt6i_idev, IPSTATS_MIB_OUTDISCARDS);
+ rcu_read_unlock();
+ goto error_check;
+ }
+ rcu_read_unlock();
out:
return 0;
-error_fault:
- err = -EFAULT;
- kfree_skb(skb);
error:
IP6_INC_STATS(net, rt->rt6i_idev, IPSTATS_MIB_OUTDISCARDS);
+error_check:
if (err == -ENOBUFS && !np->recverr)
err = 0;
return err;
diff --git a/net/ipv6/reassembly.c b/net/ipv6/reassembly.c
index e585c0a..74ffbcb 100644
--- a/net/ipv6/reassembly.c
+++ b/net/ipv6/reassembly.c
@@ -79,94 +79,58 @@ static struct inet_frags ip6_frags;
static int ip6_frag_reasm(struct frag_queue *fq, struct sk_buff *prev,
struct net_device *dev);
-/*
- * callers should be careful not to use the hash value outside the ipfrag_lock
- * as doing so could race with ipfrag_hash_rnd being recalculated.
- */
-static unsigned int inet6_hash_frag(__be32 id, const struct in6_addr *saddr,
- const struct in6_addr *daddr)
-{
- net_get_random_once(&ip6_frags.rnd, sizeof(ip6_frags.rnd));
- return jhash_3words(ipv6_addr_hash(saddr), ipv6_addr_hash(daddr),
- (__force u32)id, ip6_frags.rnd);
-}
-
-static unsigned int ip6_hashfn(const struct inet_frag_queue *q)
-{
- const struct frag_queue *fq;
-
- fq = container_of(q, struct frag_queue, q);
- return inet6_hash_frag(fq->id, &fq->saddr, &fq->daddr);
-}
-
-bool ip6_frag_match(const struct inet_frag_queue *q, const void *a)
-{
- const struct frag_queue *fq;
- const struct ip6_create_arg *arg = a;
-
- fq = container_of(q, struct frag_queue, q);
- return fq->id == arg->id &&
- fq->user == arg->user &&
- ipv6_addr_equal(&fq->saddr, arg->src) &&
- ipv6_addr_equal(&fq->daddr, arg->dst) &&
- (arg->iif == fq->iif ||
- !(ipv6_addr_type(arg->dst) & (IPV6_ADDR_MULTICAST |
- IPV6_ADDR_LINKLOCAL)));
-}
-EXPORT_SYMBOL(ip6_frag_match);
-
void ip6_frag_init(struct inet_frag_queue *q, const void *a)
{
struct frag_queue *fq = container_of(q, struct frag_queue, q);
- const struct ip6_create_arg *arg = a;
+ const struct frag_v6_compare_key *key = a;
- fq->id = arg->id;
- fq->user = arg->user;
- fq->saddr = *arg->src;
- fq->daddr = *arg->dst;
- fq->ecn = arg->ecn;
+ q->key.v6 = *key;
+ fq->ecn = 0;
}
EXPORT_SYMBOL(ip6_frag_init);
-void ip6_expire_frag_queue(struct net *net, struct frag_queue *fq,
- struct inet_frags *frags)
+void ip6_expire_frag_queue(struct net *net, struct frag_queue *fq)
{
struct net_device *dev = NULL;
+ struct sk_buff *head;
+ rcu_read_lock();
spin_lock(&fq->q.lock);
if (fq->q.flags & INET_FRAG_COMPLETE)
goto out;
- inet_frag_kill(&fq->q, frags);
+ inet_frag_kill(&fq->q);
- rcu_read_lock();
dev = dev_get_by_index_rcu(net, fq->iif);
if (!dev)
- goto out_rcu_unlock;
+ goto out;
__IP6_INC_STATS(net, __in6_dev_get(dev), IPSTATS_MIB_REASMFAILS);
-
- if (inet_frag_evicting(&fq->q))
- goto out_rcu_unlock;
-
__IP6_INC_STATS(net, __in6_dev_get(dev), IPSTATS_MIB_REASMTIMEOUT);
/* Don't send error if the first segment did not arrive. */
- if (!(fq->q.flags & INET_FRAG_FIRST_IN) || !fq->q.fragments)
- goto out_rcu_unlock;
+ head = fq->q.fragments;
+ if (!(fq->q.flags & INET_FRAG_FIRST_IN) || !head)
+ goto out;
/* But use as source device on which LAST ARRIVED
* segment was received. And do not use fq->dev
* pointer directly, device might already disappeared.
*/
- fq->q.fragments->dev = dev;
- icmpv6_send(fq->q.fragments, ICMPV6_TIME_EXCEED, ICMPV6_EXC_FRAGTIME, 0);
-out_rcu_unlock:
- rcu_read_unlock();
+ head->dev = dev;
+ skb_get(head);
+ spin_unlock(&fq->q.lock);
+
+ icmpv6_send(head, ICMPV6_TIME_EXCEED, ICMPV6_EXC_FRAGTIME, 0);
+ kfree_skb(head);
+ goto out_rcu_unlock;
+
out:
spin_unlock(&fq->q.lock);
- inet_frag_put(&fq->q, frags);
+out_rcu_unlock:
+ rcu_read_unlock();
+ inet_frag_put(&fq->q);
}
EXPORT_SYMBOL(ip6_expire_frag_queue);
@@ -178,31 +142,29 @@ static void ip6_frag_expire(unsigned long data)
fq = container_of((struct inet_frag_queue *)data, struct frag_queue, q);
net = container_of(fq->q.net, struct net, ipv6.frags);
- ip6_expire_frag_queue(net, fq, &ip6_frags);
+ ip6_expire_frag_queue(net, fq);
}
static struct frag_queue *
-fq_find(struct net *net, __be32 id, const struct in6_addr *src,
- const struct in6_addr *dst, int iif, u8 ecn)
+fq_find(struct net *net, __be32 id, const struct ipv6hdr *hdr, int iif)
{
+ struct frag_v6_compare_key key = {
+ .id = id,
+ .saddr = hdr->saddr,
+ .daddr = hdr->daddr,
+ .user = IP6_DEFRAG_LOCAL_DELIVER,
+ .iif = iif,
+ };
struct inet_frag_queue *q;
- struct ip6_create_arg arg;
- unsigned int hash;
- arg.id = id;
- arg.user = IP6_DEFRAG_LOCAL_DELIVER;
- arg.src = src;
- arg.dst = dst;
- arg.iif = iif;
- arg.ecn = ecn;
+ if (!(ipv6_addr_type(&hdr->daddr) & (IPV6_ADDR_MULTICAST |
+ IPV6_ADDR_LINKLOCAL)))
+ key.iif = 0;
- hash = inet6_hash_frag(id, src, dst);
-
- q = inet_frag_find(&net->ipv6.frags, &ip6_frags, &arg, hash);
- if (IS_ERR_OR_NULL(q)) {
- inet_frag_maybe_warn_overflow(q, pr_fmt());
+ q = inet_frag_find(&net->ipv6.frags, &key);
+ if (!q)
return NULL;
- }
+
return container_of(q, struct frag_queue, q);
}
@@ -359,7 +321,7 @@ static int ip6_frag_queue(struct frag_queue *fq, struct sk_buff *skb,
return -1;
discard_fq:
- inet_frag_kill(&fq->q, &ip6_frags);
+ inet_frag_kill(&fq->q);
err:
__IP6_INC_STATS(net, ip6_dst_idev(skb_dst(skb)),
IPSTATS_MIB_REASMFAILS);
@@ -386,7 +348,7 @@ static int ip6_frag_reasm(struct frag_queue *fq, struct sk_buff *prev,
int sum_truesize;
u8 ecn;
- inet_frag_kill(&fq->q, &ip6_frags);
+ inet_frag_kill(&fq->q);
ecn = ip_frag_ecn_table[fq->ecn];
if (unlikely(ecn == 0xff))
@@ -504,6 +466,7 @@ static int ip6_frag_reasm(struct frag_queue *fq, struct sk_buff *prev,
__IP6_INC_STATS(net, __in6_dev_get(dev), IPSTATS_MIB_REASMOKS);
rcu_read_unlock();
fq->q.fragments = NULL;
+ fq->q.rb_fragments = RB_ROOT;
fq->q.fragments_tail = NULL;
return 1;
@@ -525,6 +488,7 @@ static int ipv6_frag_rcv(struct sk_buff *skb)
struct frag_queue *fq;
const struct ipv6hdr *hdr = ipv6_hdr(skb);
struct net *net = dev_net(skb_dst(skb)->dev);
+ int iif;
if (IP6CB(skb)->flags & IP6SKB_FRAGMENTED)
goto fail_hdr;
@@ -553,17 +517,22 @@ static int ipv6_frag_rcv(struct sk_buff *skb)
return 1;
}
- fq = fq_find(net, fhdr->identification, &hdr->saddr, &hdr->daddr,
- skb->dev ? skb->dev->ifindex : 0, ip6_frag_ecn(hdr));
+ if (skb->len - skb_network_offset(skb) < IPV6_MIN_MTU &&
+ fhdr->frag_off & htons(IP6_MF))
+ goto fail_hdr;
+
+ iif = skb->dev ? skb->dev->ifindex : 0;
+ fq = fq_find(net, fhdr->identification, hdr, iif);
if (fq) {
int ret;
spin_lock(&fq->q.lock);
+ fq->iif = iif;
ret = ip6_frag_queue(fq, skb, fhdr, IP6CB(skb)->nhoff);
spin_unlock(&fq->q.lock);
- inet_frag_put(&fq->q, &ip6_frags);
+ inet_frag_put(&fq->q);
return ret;
}
@@ -584,24 +553,22 @@ static const struct inet6_protocol frag_protocol = {
};
#ifdef CONFIG_SYSCTL
-static int zero;
static struct ctl_table ip6_frags_ns_ctl_table[] = {
{
.procname = "ip6frag_high_thresh",
.data = &init_net.ipv6.frags.high_thresh,
- .maxlen = sizeof(int),
+ .maxlen = sizeof(unsigned long),
.mode = 0644,
- .proc_handler = proc_dointvec_minmax,
+ .proc_handler = proc_doulongvec_minmax,
.extra1 = &init_net.ipv6.frags.low_thresh
},
{
.procname = "ip6frag_low_thresh",
.data = &init_net.ipv6.frags.low_thresh,
- .maxlen = sizeof(int),
+ .maxlen = sizeof(unsigned long),
.mode = 0644,
- .proc_handler = proc_dointvec_minmax,
- .extra1 = &zero,
+ .proc_handler = proc_doulongvec_minmax,
.extra2 = &init_net.ipv6.frags.high_thresh
},
{
@@ -644,10 +611,6 @@ static int __net_init ip6_frags_ns_sysctl_register(struct net *net)
table[1].data = &net->ipv6.frags.low_thresh;
table[1].extra2 = &net->ipv6.frags.high_thresh;
table[2].data = &net->ipv6.frags.timeout;
-
- /* Don't export sysctls to unprivileged users */
- if (net->user_ns != &init_user_ns)
- table[0].procname = NULL;
}
hdr = register_net_sysctl(net, "net/ipv6", table);
@@ -709,19 +672,27 @@ static void ip6_frags_sysctl_unregister(void)
static int __net_init ipv6_frags_init_net(struct net *net)
{
+ int res;
+
net->ipv6.frags.high_thresh = IPV6_FRAG_HIGH_THRESH;
net->ipv6.frags.low_thresh = IPV6_FRAG_LOW_THRESH;
net->ipv6.frags.timeout = IPV6_FRAG_TIMEOUT;
+ net->ipv6.frags.f = &ip6_frags;
- inet_frags_init_net(&net->ipv6.frags);
+ res = inet_frags_init_net(&net->ipv6.frags);
+ if (res < 0)
+ return res;
- return ip6_frags_ns_sysctl_register(net);
+ res = ip6_frags_ns_sysctl_register(net);
+ if (res < 0)
+ inet_frags_exit_net(&net->ipv6.frags);
+ return res;
}
static void __net_exit ipv6_frags_exit_net(struct net *net)
{
ip6_frags_ns_sysctl_unregister(net);
- inet_frags_exit_net(&net->ipv6.frags, &ip6_frags);
+ inet_frags_exit_net(&net->ipv6.frags);
}
static struct pernet_operations ip6_frags_ops = {
@@ -729,14 +700,55 @@ static struct pernet_operations ip6_frags_ops = {
.exit = ipv6_frags_exit_net,
};
+static u32 ip6_key_hashfn(const void *data, u32 len, u32 seed)
+{
+ return jhash2(data,
+ sizeof(struct frag_v6_compare_key) / sizeof(u32), seed);
+}
+
+static u32 ip6_obj_hashfn(const void *data, u32 len, u32 seed)
+{
+ const struct inet_frag_queue *fq = data;
+
+ return jhash2((const u32 *)&fq->key.v6,
+ sizeof(struct frag_v6_compare_key) / sizeof(u32), seed);
+}
+
+static int ip6_obj_cmpfn(struct rhashtable_compare_arg *arg, const void *ptr)
+{
+ const struct frag_v6_compare_key *key = arg->key;
+ const struct inet_frag_queue *fq = ptr;
+
+ return !!memcmp(&fq->key, key, sizeof(*key));
+}
+
+const struct rhashtable_params ip6_rhash_params = {
+ .head_offset = offsetof(struct inet_frag_queue, node),
+ .hashfn = ip6_key_hashfn,
+ .obj_hashfn = ip6_obj_hashfn,
+ .obj_cmpfn = ip6_obj_cmpfn,
+ .automatic_shrinking = true,
+};
+EXPORT_SYMBOL(ip6_rhash_params);
+
int __init ipv6_frag_init(void)
{
int ret;
- ret = inet6_add_protocol(&frag_protocol, IPPROTO_FRAGMENT);
+ ip6_frags.constructor = ip6_frag_init;
+ ip6_frags.destructor = NULL;
+ ip6_frags.qsize = sizeof(struct frag_queue);
+ ip6_frags.frag_expire = ip6_frag_expire;
+ ip6_frags.frags_cache_name = ip6_frag_cache_name;
+ ip6_frags.rhash_params = ip6_rhash_params;
+ ret = inet_frags_init(&ip6_frags);
if (ret)
goto out;
+ ret = inet6_add_protocol(&frag_protocol, IPPROTO_FRAGMENT);
+ if (ret)
+ goto err_protocol;
+
ret = ip6_frags_sysctl_register();
if (ret)
goto err_sysctl;
@@ -745,16 +757,6 @@ int __init ipv6_frag_init(void)
if (ret)
goto err_pernet;
- ip6_frags.hashfn = ip6_hashfn;
- ip6_frags.constructor = ip6_frag_init;
- ip6_frags.destructor = NULL;
- ip6_frags.qsize = sizeof(struct frag_queue);
- ip6_frags.match = ip6_frag_match;
- ip6_frags.frag_expire = ip6_frag_expire;
- ip6_frags.frags_cache_name = ip6_frag_cache_name;
- ret = inet_frags_init(&ip6_frags);
- if (ret)
- goto err_pernet;
out:
return ret;
@@ -762,6 +764,8 @@ int __init ipv6_frag_init(void)
ip6_frags_sysctl_unregister();
err_sysctl:
inet6_del_protocol(&frag_protocol, IPPROTO_FRAGMENT);
+err_protocol:
+ inet_frags_fini(&ip6_frags);
goto out;
}
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 96be019..c7dcf40 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -1364,9 +1364,6 @@ static void __ip6_rt_update_pmtu(struct dst_entry *dst, const struct sock *sk,
{
struct rt6_info *rt6 = (struct rt6_info *)dst;
- if (rt6->rt6i_flags & RTF_LOCAL)
- return;
-
if (dst_metric_locked(dst, RTAX_MTU))
return;
@@ -1434,10 +1431,13 @@ EXPORT_SYMBOL_GPL(ip6_update_pmtu);
void ip6_sk_update_pmtu(struct sk_buff *skb, struct sock *sk, __be32 mtu)
{
+ int oif = sk->sk_bound_dev_if;
struct dst_entry *dst;
- ip6_update_pmtu(skb, sock_net(sk), mtu,
- sk->sk_bound_dev_if, sk->sk_mark, sk->sk_uid);
+ if (!oif && skb->dev)
+ oif = l3mdev_master_ifindex(skb->dev);
+
+ ip6_update_pmtu(skb, sock_net(sk), mtu, oif, sk->sk_mark, sk->sk_uid);
dst = __sk_dst_get(sk);
if (!dst || !dst->obsolete ||
@@ -2776,6 +2776,7 @@ void rt6_mtu_change(struct net_device *dev, unsigned int mtu)
static const struct nla_policy rtm_ipv6_policy[RTA_MAX+1] = {
[RTA_GATEWAY] = { .len = sizeof(struct in6_addr) },
+ [RTA_PREFSRC] = { .len = sizeof(struct in6_addr) },
[RTA_OIF] = { .type = NLA_U32 },
[RTA_IIF] = { .type = NLA_U32 },
[RTA_PRIORITY] = { .type = NLA_U32 },
@@ -2785,6 +2786,7 @@ static const struct nla_policy rtm_ipv6_policy[RTA_MAX+1] = {
[RTA_ENCAP_TYPE] = { .type = NLA_U16 },
[RTA_ENCAP] = { .type = NLA_NESTED },
[RTA_EXPIRES] = { .type = NLA_U32 },
+ [RTA_TABLE] = { .type = NLA_U32 },
[RTA_UID] = { .type = NLA_U32 },
};
diff --git a/net/ipv6/sit.c b/net/ipv6/sit.c
index dcb2921..fc7ca1e 100644
--- a/net/ipv6/sit.c
+++ b/net/ipv6/sit.c
@@ -659,7 +659,6 @@ static int ipip6_rcv(struct sk_buff *skb)
if (iptunnel_pull_header(skb, 0, htons(ETH_P_IPV6),
!net_eq(tunnel->net, dev_net(tunnel->dev))))
goto out;
- iph = ip_hdr(skb);
err = IP_ECN_decapsulate(iph, skb);
if (unlikely(err)) {
@@ -1572,6 +1571,13 @@ static int ipip6_newlink(struct net *src_net, struct net_device *dev,
if (err < 0)
return err;
+ if (tb[IFLA_MTU]) {
+ u32 mtu = nla_get_u32(tb[IFLA_MTU]);
+
+ if (mtu >= IPV6_MIN_MTU && mtu <= 0xFFF8 - dev->hard_header_len)
+ dev->mtu = mtu;
+ }
+
#ifdef CONFIG_IPV6_SIT_6RD
if (ipip6_netlink_6rd_parms(data, &ip6rd))
err = ipip6_tunnel_update_6rd(nt, &ip6rd);
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 8efeff6..ed5d635 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -1436,6 +1436,10 @@ static int tcp_v6_rcv(struct sk_buff *skb)
reqsk_put(req);
goto discard_it;
}
+ if (tcp_checksum_complete(skb)) {
+ reqsk_put(req);
+ goto csum_error;
+ }
if (unlikely(sk->sk_state != TCP_LISTEN)) {
inet_csk_reqsk_queue_drop_and_put(sk, req);
goto lookup;
diff --git a/net/ipv6/xfrm6_output.c b/net/ipv6/xfrm6_output.c
index 4d09ce6..64862c5 100644
--- a/net/ipv6/xfrm6_output.c
+++ b/net/ipv6/xfrm6_output.c
@@ -165,9 +165,11 @@ static int __xfrm6_output(struct net *net, struct sock *sk, struct sk_buff *skb)
if (toobig && xfrm6_local_dontfrag(skb)) {
xfrm6_local_rxpmtu(skb, mtu);
+ kfree_skb(skb);
return -EMSGSIZE;
} else if (!skb->ignore_df && toobig && skb->sk) {
xfrm_local_error(skb, mtu);
+ kfree_skb(skb);
return -EMSGSIZE;
}
diff --git a/net/ipv6/xfrm6_policy.c b/net/ipv6/xfrm6_policy.c
index 4003b28..d82f427 100644
--- a/net/ipv6/xfrm6_policy.c
+++ b/net/ipv6/xfrm6_policy.c
@@ -124,7 +124,7 @@ _decode_session6(struct sk_buff *skb, struct flowi *fl, int reverse)
struct flowi6 *fl6 = &fl->u.ip6;
int onlyproto = 0;
const struct ipv6hdr *hdr = ipv6_hdr(skb);
- u16 offset = sizeof(*hdr);
+ u32 offset = sizeof(*hdr);
struct ipv6_opt_hdr *exthdr;
const unsigned char *nh = skb_network_header(skb);
u16 nhoff = IP6CB(skb)->nhoff;
diff --git a/net/irda/af_irda.c b/net/irda/af_irda.c
index 101ed6c..0a78f17 100644
--- a/net/irda/af_irda.c
+++ b/net/irda/af_irda.c
@@ -774,6 +774,13 @@ static int irda_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
return -EINVAL;
lock_sock(sk);
+
+ /* Ensure that the socket is not already bound */
+ if (self->ias_obj) {
+ err = -EINVAL;
+ goto out;
+ }
+
#ifdef CONFIG_IRDA_ULTRA
/* Special care for Ultra sockets */
if ((sk->sk_type == SOCK_DGRAM) &&
@@ -2016,7 +2023,11 @@ static int irda_setsockopt(struct socket *sock, int level, int optname,
err = -EINVAL;
goto out;
}
- irias_insert_object(ias_obj);
+
+ /* Only insert newly allocated objects */
+ if (free_ias)
+ irias_insert_object(ias_obj);
+
kfree(ias_opt);
break;
case IRLMP_IAS_DEL:
diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c
index 63e6d08..553d0ad 100644
--- a/net/kcm/kcmsock.c
+++ b/net/kcm/kcmsock.c
@@ -1424,6 +1424,7 @@ static int kcm_attach(struct socket *sock, struct socket *csock,
*/
if (csk->sk_user_data) {
write_unlock_bh(&csk->sk_callback_lock);
+ strp_stop(&psock->strp);
strp_done(&psock->strp);
kmem_cache_free(kcm_psockp, psock);
err = -EALREADY;
@@ -1670,7 +1671,7 @@ static struct file *kcm_clone(struct socket *osock)
__module_get(newsock->ops->owner);
newsk = sk_alloc(sock_net(osock->sk), PF_KCM, GFP_KERNEL,
- &kcm_proto, true);
+ &kcm_proto, false);
if (!newsk) {
sock_release(newsock);
return ERR_PTR(-ENOMEM);
diff --git a/net/key/af_key.c b/net/key/af_key.c
index ff4100f..58c045d 100644
--- a/net/key/af_key.c
+++ b/net/key/af_key.c
@@ -427,6 +427,24 @@ static int verify_address_len(const void *p)
return 0;
}
+static inline int sadb_key_len(const struct sadb_key *key)
+{
+ int key_bytes = DIV_ROUND_UP(key->sadb_key_bits, 8);
+
+ return DIV_ROUND_UP(sizeof(struct sadb_key) + key_bytes,
+ sizeof(uint64_t));
+}
+
+static int verify_key_len(const void *p)
+{
+ const struct sadb_key *key = p;
+
+ if (sadb_key_len(key) > key->sadb_key_len)
+ return -EINVAL;
+
+ return 0;
+}
+
static inline int pfkey_sec_ctx_len(const struct sadb_x_sec_ctx *sec_ctx)
{
return DIV_ROUND_UP(sizeof(struct sadb_x_sec_ctx) +
@@ -523,16 +541,25 @@ static int parse_exthdrs(struct sk_buff *skb, const struct sadb_msg *hdr, void *
return -EINVAL;
if (ext_hdrs[ext_type-1] != NULL)
return -EINVAL;
- if (ext_type == SADB_EXT_ADDRESS_SRC ||
- ext_type == SADB_EXT_ADDRESS_DST ||
- ext_type == SADB_EXT_ADDRESS_PROXY ||
- ext_type == SADB_X_EXT_NAT_T_OA) {
+ switch (ext_type) {
+ case SADB_EXT_ADDRESS_SRC:
+ case SADB_EXT_ADDRESS_DST:
+ case SADB_EXT_ADDRESS_PROXY:
+ case SADB_X_EXT_NAT_T_OA:
if (verify_address_len(p))
return -EINVAL;
- }
- if (ext_type == SADB_X_EXT_SEC_CTX) {
+ break;
+ case SADB_X_EXT_SEC_CTX:
if (verify_sec_ctx_len(p))
return -EINVAL;
+ break;
+ case SADB_EXT_KEY_AUTH:
+ case SADB_EXT_KEY_ENCRYPT:
+ if (verify_key_len(p))
+ return -EINVAL;
+ break;
+ default:
+ break;
}
ext_hdrs[ext_type-1] = (void *) p;
}
@@ -1101,14 +1128,12 @@ static struct xfrm_state * pfkey_msg2xfrm_state(struct net *net,
key = ext_hdrs[SADB_EXT_KEY_AUTH - 1];
if (key != NULL &&
sa->sadb_sa_auth != SADB_X_AALG_NULL &&
- ((key->sadb_key_bits+7) / 8 == 0 ||
- (key->sadb_key_bits+7) / 8 > key->sadb_key_len * sizeof(uint64_t)))
+ key->sadb_key_bits == 0)
return ERR_PTR(-EINVAL);
key = ext_hdrs[SADB_EXT_KEY_ENCRYPT-1];
if (key != NULL &&
sa->sadb_sa_encrypt != SADB_EALG_NULL &&
- ((key->sadb_key_bits+7) / 8 == 0 ||
- (key->sadb_key_bits+7) / 8 > key->sadb_key_len * sizeof(uint64_t)))
+ key->sadb_key_bits == 0)
return ERR_PTR(-EINVAL);
x = xfrm_state_alloc(net);
diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c
index ead98e8..b96dbe3 100644
--- a/net/l2tp/l2tp_core.c
+++ b/net/l2tp/l2tp_core.c
@@ -845,10 +845,8 @@ void l2tp_recv_common(struct l2tp_session *session, struct sk_buff *skb,
}
}
- /* Session data offset is handled differently for L2TPv2 and
- * L2TPv3. For L2TPv2, there is an optional 16-bit value in
- * the header. For L2TPv3, the offset is negotiated using AVPs
- * in the session setup control protocol.
+ /* Session data offset is defined only for L2TPv2 and is
+ * indicated by an optional 16-bit value in the header.
*/
if (tunnel->version == L2TP_HDR_VER_2) {
/* If offset bit set, skip it. */
@@ -856,8 +854,7 @@ void l2tp_recv_common(struct l2tp_session *session, struct sk_buff *skb,
offset = ntohs(*(__be16 *)ptr);
ptr += 2 + offset;
}
- } else
- ptr += session->offset;
+ }
offset = ptr - optr;
if (!pskb_may_pull(skb, offset))
@@ -1141,8 +1138,6 @@ static int l2tp_build_l2tpv3_header(struct l2tp_session *session, void *buf)
}
bufp += session->l2specific_len;
}
- if (session->offset)
- bufp += session->offset;
return bufp - optr;
}
@@ -1239,7 +1234,7 @@ int l2tp_xmit_skb(struct l2tp_session *session, struct sk_buff *skb, int hdr_len
/* Get routing info from the tunnel socket */
skb_dst_drop(skb);
- skb_dst_set(skb, dst_clone(__sk_dst_check(sk, 0)));
+ skb_dst_set(skb, sk_dst_check(sk, 0));
inet = inet_sk(sk);
fl = &inet->cork.fl;
@@ -1827,7 +1822,7 @@ void l2tp_session_set_header_len(struct l2tp_session *session, int version)
if (session->send_seq)
session->hdr_len += 4;
} else {
- session->hdr_len = 4 + session->cookie_len + session->l2specific_len + session->offset;
+ session->hdr_len = 4 + session->cookie_len + session->l2specific_len;
if (session->tunnel->encap == L2TP_ENCAPTYPE_UDP)
session->hdr_len += 4;
}
@@ -1878,7 +1873,6 @@ struct l2tp_session *l2tp_session_create(int priv_size, struct l2tp_tunnel *tunn
session->recv_seq = cfg->recv_seq;
session->lns_mode = cfg->lns_mode;
session->reorder_timeout = cfg->reorder_timeout;
- session->offset = cfg->offset;
session->l2specific_type = cfg->l2specific_type;
session->l2specific_len = cfg->l2specific_len;
session->cookie_len = cfg->cookie_len;
diff --git a/net/l2tp/l2tp_core.h b/net/l2tp/l2tp_core.h
index 42419f1..86356a2 100644
--- a/net/l2tp/l2tp_core.h
+++ b/net/l2tp/l2tp_core.h
@@ -68,7 +68,6 @@ struct l2tp_session_cfg {
int debug; /* bitmask of debug message
* categories */
u16 vlan_id; /* VLAN pseudowire only */
- u16 offset; /* offset to payload */
u16 l2specific_len; /* Layer 2 specific length */
u16 l2specific_type; /* Layer 2 specific type */
u8 cookie[8]; /* optional cookie */
@@ -94,8 +93,6 @@ struct l2tp_session {
int cookie_len;
u8 peer_cookie[8];
int peer_cookie_len;
- u16 offset; /* offset from end of L2TP header
- to beginning of data */
u16 l2specific_len;
u16 l2specific_type;
u16 hdr_len;
diff --git a/net/l2tp/l2tp_debugfs.c b/net/l2tp/l2tp_debugfs.c
index d100aed..2d2a732 100644
--- a/net/l2tp/l2tp_debugfs.c
+++ b/net/l2tp/l2tp_debugfs.c
@@ -181,8 +181,8 @@ static void l2tp_dfs_seq_session_show(struct seq_file *m, void *v)
session->lns_mode ? "LNS" : "LAC",
session->debug,
jiffies_to_msecs(session->reorder_timeout));
- seq_printf(m, " offset %hu l2specific %hu/%hu\n",
- session->offset, session->l2specific_type, session->l2specific_len);
+ seq_printf(m, " offset 0 l2specific %hu/%hu\n",
+ session->l2specific_type, session->l2specific_len);
if (session->cookie_len) {
seq_printf(m, " cookie %02x%02x%02x%02x",
session->cookie[0], session->cookie[1],
diff --git a/net/l2tp/l2tp_netlink.c b/net/l2tp/l2tp_netlink.c
index ce12384..d6fccfd 100644
--- a/net/l2tp/l2tp_netlink.c
+++ b/net/l2tp/l2tp_netlink.c
@@ -536,9 +536,6 @@ static int l2tp_nl_cmd_session_create(struct sk_buff *skb, struct genl_info *inf
}
if (tunnel->version > 2) {
- if (info->attrs[L2TP_ATTR_OFFSET])
- cfg.offset = nla_get_u16(info->attrs[L2TP_ATTR_OFFSET]);
-
if (info->attrs[L2TP_ATTR_DATA_SEQ])
cfg.data_seq = nla_get_u8(info->attrs[L2TP_ATTR_DATA_SEQ]);
@@ -750,8 +747,6 @@ static int l2tp_nl_session_send(struct sk_buff *skb, u32 portid, u32 seq, int fl
if ((session->ifname[0] &&
nla_put_string(skb, L2TP_ATTR_IFNAME, session->ifname)) ||
- (session->offset &&
- nla_put_u16(skb, L2TP_ATTR_OFFSET, session->offset)) ||
(session->cookie_len &&
nla_put(skb, L2TP_ATTR_COOKIE, session->cookie_len,
&session->cookie[0])) ||
diff --git a/net/l2tp/l2tp_ppp.c b/net/l2tp/l2tp_ppp.c
index 163f1fa..9b214f3 100644
--- a/net/l2tp/l2tp_ppp.c
+++ b/net/l2tp/l2tp_ppp.c
@@ -590,6 +590,13 @@ static int pppol2tp_connect(struct socket *sock, struct sockaddr *uservaddr,
lock_sock(sk);
error = -EINVAL;
+
+ if (sockaddr_len != sizeof(struct sockaddr_pppol2tp) &&
+ sockaddr_len != sizeof(struct sockaddr_pppol2tpv3) &&
+ sockaddr_len != sizeof(struct sockaddr_pppol2tpin6) &&
+ sockaddr_len != sizeof(struct sockaddr_pppol2tpv3in6))
+ goto end;
+
if (sp->sa_protocol != PX_PROTO_OL2TP)
goto end;
diff --git a/net/llc/af_llc.c b/net/llc/af_llc.c
index f7caf0f..789e66b 100644
--- a/net/llc/af_llc.c
+++ b/net/llc/af_llc.c
@@ -197,9 +197,19 @@ static int llc_ui_release(struct socket *sock)
llc->laddr.lsap, llc->daddr.lsap);
if (!llc_send_disc(sk))
llc_ui_wait_for_disc(sk, sk->sk_rcvtimeo);
- if (!sock_flag(sk, SOCK_ZAPPED))
+ if (!sock_flag(sk, SOCK_ZAPPED)) {
+ struct llc_sap *sap = llc->sap;
+
+ /* Hold this for release_sock(), so that llc_backlog_rcv()
+ * could still use it.
+ */
+ llc_sap_hold(sap);
llc_sap_remove_socket(llc->sap, sk);
- release_sock(sk);
+ release_sock(sk);
+ llc_sap_put(sap);
+ } else {
+ release_sock(sk);
+ }
if (llc->dev)
dev_put(llc->dev);
sock_put(sk);
@@ -716,7 +726,6 @@ static int llc_ui_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
struct sk_buff *skb = NULL;
struct sock *sk = sock->sk;
struct llc_sock *llc = llc_sk(sk);
- unsigned long cpu_flags;
size_t copied = 0;
u32 peek_seq = 0;
u32 *seq, skb_len;
@@ -841,9 +850,8 @@ static int llc_ui_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
goto copy_uaddr;
if (!(flags & MSG_PEEK)) {
- spin_lock_irqsave(&sk->sk_receive_queue.lock, cpu_flags);
- sk_eat_skb(sk, skb);
- spin_unlock_irqrestore(&sk->sk_receive_queue.lock, cpu_flags);
+ skb_unlink(skb, &sk->sk_receive_queue);
+ kfree_skb(skb);
*seq = 0;
}
@@ -864,9 +872,8 @@ static int llc_ui_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
llc_cmsg_rcv(msg, skb);
if (!(flags & MSG_PEEK)) {
- spin_lock_irqsave(&sk->sk_receive_queue.lock, cpu_flags);
- sk_eat_skb(sk, skb);
- spin_unlock_irqrestore(&sk->sk_receive_queue.lock, cpu_flags);
+ skb_unlink(skb, &sk->sk_receive_queue);
+ kfree_skb(skb);
*seq = 0;
}
@@ -916,6 +923,9 @@ static int llc_ui_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
if (size > llc->dev->mtu)
size = llc->dev->mtu;
copied = size - hdrlen;
+ rc = -EINVAL;
+ if (copied < 0)
+ goto release;
release_sock(sk);
skb = sock_alloc_send_skb(sk, size, noblock, &rc);
lock_sock(sk);
diff --git a/net/llc/llc_c_ac.c b/net/llc/llc_c_ac.c
index ea225bd..4b60f68 100644
--- a/net/llc/llc_c_ac.c
+++ b/net/llc/llc_c_ac.c
@@ -389,7 +389,7 @@ static int llc_conn_ac_send_i_cmd_p_set_0(struct sock *sk, struct sk_buff *skb)
llc_pdu_init_as_i_cmd(skb, 0, llc->vS, llc->vR);
rc = llc_mac_hdr_init(skb, llc->dev->dev_addr, llc->daddr.mac);
if (likely(!rc)) {
- llc_conn_send_pdu(sk, skb);
+ rc = llc_conn_send_pdu(sk, skb);
llc_conn_ac_inc_vs_by_1(sk, skb);
}
return rc;
@@ -916,7 +916,7 @@ static int llc_conn_ac_send_i_rsp_f_set_ackpf(struct sock *sk,
llc_pdu_init_as_i_cmd(skb, llc->ack_pf, llc->vS, llc->vR);
rc = llc_mac_hdr_init(skb, llc->dev->dev_addr, llc->daddr.mac);
if (likely(!rc)) {
- llc_conn_send_pdu(sk, skb);
+ rc = llc_conn_send_pdu(sk, skb);
llc_conn_ac_inc_vs_by_1(sk, skb);
}
return rc;
@@ -935,14 +935,17 @@ static int llc_conn_ac_send_i_rsp_f_set_ackpf(struct sock *sk,
int llc_conn_ac_send_i_as_ack(struct sock *sk, struct sk_buff *skb)
{
struct llc_sock *llc = llc_sk(sk);
+ int ret;
if (llc->ack_must_be_send) {
- llc_conn_ac_send_i_rsp_f_set_ackpf(sk, skb);
+ ret = llc_conn_ac_send_i_rsp_f_set_ackpf(sk, skb);
llc->ack_must_be_send = 0 ;
llc->ack_pf = 0;
- } else
- llc_conn_ac_send_i_cmd_p_set_0(sk, skb);
- return 0;
+ } else {
+ ret = llc_conn_ac_send_i_cmd_p_set_0(sk, skb);
+ }
+
+ return ret;
}
/**
@@ -1096,14 +1099,7 @@ int llc_conn_ac_inc_tx_win_size(struct sock *sk, struct sk_buff *skb)
int llc_conn_ac_stop_all_timers(struct sock *sk, struct sk_buff *skb)
{
- struct llc_sock *llc = llc_sk(sk);
-
- del_timer(&llc->pf_cycle_timer.timer);
- del_timer(&llc->ack_timer.timer);
- del_timer(&llc->rej_sent_timer.timer);
- del_timer(&llc->busy_state_timer.timer);
- llc->ack_must_be_send = 0;
- llc->ack_pf = 0;
+ llc_sk_stop_all_timers(sk, false);
return 0;
}
diff --git a/net/llc/llc_conn.c b/net/llc/llc_conn.c
index 8bc5a1b..b9290a1 100644
--- a/net/llc/llc_conn.c
+++ b/net/llc/llc_conn.c
@@ -30,7 +30,7 @@
#endif
static int llc_find_offset(int state, int ev_type);
-static void llc_conn_send_pdus(struct sock *sk);
+static int llc_conn_send_pdus(struct sock *sk, struct sk_buff *skb);
static int llc_conn_service(struct sock *sk, struct sk_buff *skb);
static int llc_exec_conn_trans_actions(struct sock *sk,
struct llc_conn_state_trans *trans,
@@ -193,11 +193,11 @@ int llc_conn_state_process(struct sock *sk, struct sk_buff *skb)
return rc;
}
-void llc_conn_send_pdu(struct sock *sk, struct sk_buff *skb)
+int llc_conn_send_pdu(struct sock *sk, struct sk_buff *skb)
{
/* queue PDU to send to MAC layer */
skb_queue_tail(&sk->sk_write_queue, skb);
- llc_conn_send_pdus(sk);
+ return llc_conn_send_pdus(sk, skb);
}
/**
@@ -255,7 +255,7 @@ void llc_conn_resend_i_pdu_as_cmd(struct sock *sk, u8 nr, u8 first_p_bit)
if (howmany_resend > 0)
llc->vS = (llc->vS + 1) % LLC_2_SEQ_NBR_MODULO;
/* any PDUs to re-send are queued up; start sending to MAC */
- llc_conn_send_pdus(sk);
+ llc_conn_send_pdus(sk, NULL);
out:;
}
@@ -296,7 +296,7 @@ void llc_conn_resend_i_pdu_as_rsp(struct sock *sk, u8 nr, u8 first_f_bit)
if (howmany_resend > 0)
llc->vS = (llc->vS + 1) % LLC_2_SEQ_NBR_MODULO;
/* any PDUs to re-send are queued up; start sending to MAC */
- llc_conn_send_pdus(sk);
+ llc_conn_send_pdus(sk, NULL);
out:;
}
@@ -340,12 +340,16 @@ int llc_conn_remove_acked_pdus(struct sock *sk, u8 nr, u16 *how_many_unacked)
/**
* llc_conn_send_pdus - Sends queued PDUs
* @sk: active connection
+ * @hold_skb: the skb held by caller, or NULL if does not care
*
- * Sends queued pdus to MAC layer for transmission.
+ * Sends queued pdus to MAC layer for transmission. When @hold_skb is
+ * NULL, always return 0. Otherwise, return 0 if @hold_skb is sent
+ * successfully, or 1 for failure.
*/
-static void llc_conn_send_pdus(struct sock *sk)
+static int llc_conn_send_pdus(struct sock *sk, struct sk_buff *hold_skb)
{
struct sk_buff *skb;
+ int ret = 0;
while ((skb = skb_dequeue(&sk->sk_write_queue)) != NULL) {
struct llc_pdu_sn *pdu = llc_pdu_sn_hdr(skb);
@@ -357,10 +361,20 @@ static void llc_conn_send_pdus(struct sock *sk)
skb_queue_tail(&llc_sk(sk)->pdu_unack_q, skb);
if (!skb2)
break;
- skb = skb2;
+ dev_queue_xmit(skb2);
+ } else {
+ bool is_target = skb == hold_skb;
+ int rc;
+
+ if (is_target)
+ skb_get(skb);
+ rc = dev_queue_xmit(skb);
+ if (is_target)
+ ret = rc;
}
- dev_queue_xmit(skb);
}
+
+ return ret;
}
/**
@@ -720,6 +734,7 @@ void llc_sap_add_socket(struct llc_sap *sap, struct sock *sk)
llc_sk(sk)->sap = sap;
spin_lock_bh(&sap->sk_lock);
+ sock_set_flag(sk, SOCK_RCU_FREE);
sap->sk_count++;
sk_nulls_add_node_rcu(sk, laddr_hb);
hlist_add_head(&llc->dev_hash_node, dev_hb);
@@ -951,6 +966,26 @@ struct sock *llc_sk_alloc(struct net *net, int family, gfp_t priority, struct pr
return sk;
}
+void llc_sk_stop_all_timers(struct sock *sk, bool sync)
+{
+ struct llc_sock *llc = llc_sk(sk);
+
+ if (sync) {
+ del_timer_sync(&llc->pf_cycle_timer.timer);
+ del_timer_sync(&llc->ack_timer.timer);
+ del_timer_sync(&llc->rej_sent_timer.timer);
+ del_timer_sync(&llc->busy_state_timer.timer);
+ } else {
+ del_timer(&llc->pf_cycle_timer.timer);
+ del_timer(&llc->ack_timer.timer);
+ del_timer(&llc->rej_sent_timer.timer);
+ del_timer(&llc->busy_state_timer.timer);
+ }
+
+ llc->ack_must_be_send = 0;
+ llc->ack_pf = 0;
+}
+
/**
* llc_sk_free - Frees a LLC socket
* @sk - socket to free
@@ -963,7 +998,7 @@ void llc_sk_free(struct sock *sk)
llc->state = LLC_CONN_OUT_OF_SVC;
/* Stop all (possibly) running timers */
- llc_conn_ac_stop_all_timers(sk, NULL);
+ llc_sk_stop_all_timers(sk, true);
#ifdef DEBUG_LLC_CONN_ALLOC
printk(KERN_INFO "%s: unackq=%d, txq=%d\n", __func__,
skb_queue_len(&llc->pdu_unack_q),
diff --git a/net/llc/llc_core.c b/net/llc/llc_core.c
index 842851c..e896a2c 100644
--- a/net/llc/llc_core.c
+++ b/net/llc/llc_core.c
@@ -73,8 +73,8 @@ struct llc_sap *llc_sap_find(unsigned char sap_value)
rcu_read_lock_bh();
sap = __llc_sap_find(sap_value);
- if (sap)
- llc_sap_hold(sap);
+ if (!sap || !llc_sap_hold_safe(sap))
+ sap = NULL;
rcu_read_unlock_bh();
return sap;
}
diff --git a/net/mac80211/agg-tx.c b/net/mac80211/agg-tx.c
index 45319cc..80c45567 100644
--- a/net/mac80211/agg-tx.c
+++ b/net/mac80211/agg-tx.c
@@ -7,7 +7,7 @@
* Copyright 2006-2007 Jiri Benc <jbenc@suse.cz>
* Copyright 2007, Michael Wu <flamingice@sourmilk.net>
* Copyright 2007-2010, Intel Corporation
- * Copyright(c) 2015 Intel Deutschland GmbH
+ * Copyright(c) 2015-2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -741,7 +741,47 @@ static void ieee80211_agg_tx_operational(struct ieee80211_local *local,
ieee80211_agg_start_txq(sta, tid, true);
}
-void ieee80211_start_tx_ba_cb(struct ieee80211_vif *vif, u8 *ra, u16 tid)
+void ieee80211_start_tx_ba_cb(struct sta_info *sta, int tid,
+ struct tid_ampdu_tx *tid_tx)
+{
+ struct ieee80211_sub_if_data *sdata = sta->sdata;
+ struct ieee80211_local *local = sdata->local;
+
+ if (WARN_ON(test_and_set_bit(HT_AGG_STATE_DRV_READY, &tid_tx->state)))
+ return;
+
+ if (test_bit(HT_AGG_STATE_RESPONSE_RECEIVED, &tid_tx->state))
+ ieee80211_agg_tx_operational(local, sta, tid);
+}
+
+static struct tid_ampdu_tx *
+ieee80211_lookup_tid_tx(struct ieee80211_sub_if_data *sdata,
+ const u8 *ra, u16 tid, struct sta_info **sta)
+{
+ struct tid_ampdu_tx *tid_tx;
+
+ if (tid >= IEEE80211_NUM_TIDS) {
+ ht_dbg(sdata, "Bad TID value: tid = %d (>= %d)\n",
+ tid, IEEE80211_NUM_TIDS);
+ return NULL;
+ }
+
+ *sta = sta_info_get_bss(sdata, ra);
+ if (!*sta) {
+ ht_dbg(sdata, "Could not find station: %pM\n", ra);
+ return NULL;
+ }
+
+ tid_tx = rcu_dereference((*sta)->ampdu_mlme.tid_tx[tid]);
+
+ if (WARN_ON(!tid_tx))
+ ht_dbg(sdata, "addBA was not requested!\n");
+
+ return tid_tx;
+}
+
+void ieee80211_start_tx_ba_cb_irqsafe(struct ieee80211_vif *vif,
+ const u8 *ra, u16 tid)
{
struct ieee80211_sub_if_data *sdata = vif_to_sdata(vif);
struct ieee80211_local *local = sdata->local;
@@ -750,57 +790,15 @@ void ieee80211_start_tx_ba_cb(struct ieee80211_vif *vif, u8 *ra, u16 tid)
trace_api_start_tx_ba_cb(sdata, ra, tid);
- if (tid >= IEEE80211_NUM_TIDS) {
- ht_dbg(sdata, "Bad TID value: tid = %d (>= %d)\n",
- tid, IEEE80211_NUM_TIDS);
- return;
- }
+ rcu_read_lock();
+ tid_tx = ieee80211_lookup_tid_tx(sdata, ra, tid, &sta);
+ if (!tid_tx)
+ goto out;
- mutex_lock(&local->sta_mtx);
- sta = sta_info_get_bss(sdata, ra);
- if (!sta) {
- mutex_unlock(&local->sta_mtx);
- ht_dbg(sdata, "Could not find station: %pM\n", ra);
- return;
- }
-
- mutex_lock(&sta->ampdu_mlme.mtx);
- tid_tx = rcu_dereference_protected_tid_tx(sta, tid);
-
- if (WARN_ON(!tid_tx)) {
- ht_dbg(sdata, "addBA was not requested!\n");
- goto unlock;
- }
-
- if (WARN_ON(test_and_set_bit(HT_AGG_STATE_DRV_READY, &tid_tx->state)))
- goto unlock;
-
- if (test_bit(HT_AGG_STATE_RESPONSE_RECEIVED, &tid_tx->state))
- ieee80211_agg_tx_operational(local, sta, tid);
-
- unlock:
- mutex_unlock(&sta->ampdu_mlme.mtx);
- mutex_unlock(&local->sta_mtx);
-}
-
-void ieee80211_start_tx_ba_cb_irqsafe(struct ieee80211_vif *vif,
- const u8 *ra, u16 tid)
-{
- struct ieee80211_sub_if_data *sdata = vif_to_sdata(vif);
- struct ieee80211_local *local = sdata->local;
- struct ieee80211_ra_tid *ra_tid;
- struct sk_buff *skb = dev_alloc_skb(0);
-
- if (unlikely(!skb))
- return;
-
- ra_tid = (struct ieee80211_ra_tid *) &skb->cb;
- memcpy(&ra_tid->ra, ra, ETH_ALEN);
- ra_tid->tid = tid;
-
- skb->pkt_type = IEEE80211_SDATA_QUEUE_AGG_START;
- skb_queue_tail(&sdata->skb_queue, skb);
- ieee80211_queue_work(&local->hw, &sdata->work);
+ set_bit(HT_AGG_STATE_START_CB, &tid_tx->state);
+ ieee80211_queue_work(&local->hw, &sta->ampdu_mlme.work);
+ out:
+ rcu_read_unlock();
}
EXPORT_SYMBOL(ieee80211_start_tx_ba_cb_irqsafe);
@@ -860,37 +858,18 @@ int ieee80211_stop_tx_ba_session(struct ieee80211_sta *pubsta, u16 tid)
}
EXPORT_SYMBOL(ieee80211_stop_tx_ba_session);
-void ieee80211_stop_tx_ba_cb(struct ieee80211_vif *vif, u8 *ra, u8 tid)
+void ieee80211_stop_tx_ba_cb(struct sta_info *sta, int tid,
+ struct tid_ampdu_tx *tid_tx)
{
- struct ieee80211_sub_if_data *sdata = vif_to_sdata(vif);
- struct ieee80211_local *local = sdata->local;
- struct sta_info *sta;
- struct tid_ampdu_tx *tid_tx;
+ struct ieee80211_sub_if_data *sdata = sta->sdata;
bool send_delba = false;
- trace_api_stop_tx_ba_cb(sdata, ra, tid);
+ ht_dbg(sdata, "Stopping Tx BA session for %pM tid %d\n",
+ sta->sta.addr, tid);
- if (tid >= IEEE80211_NUM_TIDS) {
- ht_dbg(sdata, "Bad TID value: tid = %d (>= %d)\n",
- tid, IEEE80211_NUM_TIDS);
- return;
- }
-
- ht_dbg(sdata, "Stopping Tx BA session for %pM tid %d\n", ra, tid);
-
- mutex_lock(&local->sta_mtx);
-
- sta = sta_info_get_bss(sdata, ra);
- if (!sta) {
- ht_dbg(sdata, "Could not find station: %pM\n", ra);
- goto unlock;
- }
-
- mutex_lock(&sta->ampdu_mlme.mtx);
spin_lock_bh(&sta->lock);
- tid_tx = rcu_dereference_protected_tid_tx(sta, tid);
- if (!tid_tx || !test_bit(HT_AGG_STATE_STOPPING, &tid_tx->state)) {
+ if (!test_bit(HT_AGG_STATE_STOPPING, &tid_tx->state)) {
ht_dbg(sdata,
"unexpected callback to A-MPDU stop for %pM tid %d\n",
sta->sta.addr, tid);
@@ -906,12 +885,8 @@ void ieee80211_stop_tx_ba_cb(struct ieee80211_vif *vif, u8 *ra, u8 tid)
spin_unlock_bh(&sta->lock);
if (send_delba)
- ieee80211_send_delba(sdata, ra, tid,
+ ieee80211_send_delba(sdata, sta->sta.addr, tid,
WLAN_BACK_INITIATOR, WLAN_REASON_QSTA_NOT_USE);
-
- mutex_unlock(&sta->ampdu_mlme.mtx);
- unlock:
- mutex_unlock(&local->sta_mtx);
}
void ieee80211_stop_tx_ba_cb_irqsafe(struct ieee80211_vif *vif,
@@ -919,19 +894,20 @@ void ieee80211_stop_tx_ba_cb_irqsafe(struct ieee80211_vif *vif,
{
struct ieee80211_sub_if_data *sdata = vif_to_sdata(vif);
struct ieee80211_local *local = sdata->local;
- struct ieee80211_ra_tid *ra_tid;
- struct sk_buff *skb = dev_alloc_skb(0);
+ struct sta_info *sta;
+ struct tid_ampdu_tx *tid_tx;
- if (unlikely(!skb))
- return;
+ trace_api_stop_tx_ba_cb(sdata, ra, tid);
- ra_tid = (struct ieee80211_ra_tid *) &skb->cb;
- memcpy(&ra_tid->ra, ra, ETH_ALEN);
- ra_tid->tid = tid;
+ rcu_read_lock();
+ tid_tx = ieee80211_lookup_tid_tx(sdata, ra, tid, &sta);
+ if (!tid_tx)
+ goto out;
- skb->pkt_type = IEEE80211_SDATA_QUEUE_AGG_STOP;
- skb_queue_tail(&sdata->skb_queue, skb);
- ieee80211_queue_work(&local->hw, &sdata->work);
+ set_bit(HT_AGG_STATE_STOP_CB, &tid_tx->state);
+ ieee80211_queue_work(&local->hw, &sta->ampdu_mlme.work);
+ out:
+ rcu_read_unlock();
}
EXPORT_SYMBOL(ieee80211_stop_tx_ba_cb_irqsafe);
diff --git a/net/mac80211/cfg.c b/net/mac80211/cfg.c
index d7801f6..6ef9d32 100644
--- a/net/mac80211/cfg.c
+++ b/net/mac80211/cfg.c
@@ -386,7 +386,7 @@ static int ieee80211_add_key(struct wiphy *wiphy, struct net_device *dev,
case NL80211_IFTYPE_AP:
case NL80211_IFTYPE_AP_VLAN:
/* Keys without a station are used for TX only */
- if (key->sta && test_sta_flag(key->sta, WLAN_STA_MFP))
+ if (sta && test_sta_flag(sta, WLAN_STA_MFP))
key->conf.flags |= IEEE80211_KEY_FLAG_RX_MGMT;
break;
case NL80211_IFTYPE_ADHOC:
@@ -454,7 +454,7 @@ static int ieee80211_del_key(struct wiphy *wiphy, struct net_device *dev,
goto out_unlock;
}
- ieee80211_key_free(key, true);
+ ieee80211_key_free(key, sdata->vif.type == NL80211_IFTYPE_STATION);
ret = 0;
out_unlock:
diff --git a/net/mac80211/ht.c b/net/mac80211/ht.c
index f4a5287..6ca5442 100644
--- a/net/mac80211/ht.c
+++ b/net/mac80211/ht.c
@@ -7,6 +7,7 @@
* Copyright 2006-2007 Jiri Benc <jbenc@suse.cz>
* Copyright 2007, Michael Wu <flamingice@sourmilk.net>
* Copyright 2007-2010, Intel Corporation
+ * Copyright 2017 Intel Deutschland GmbH
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -289,8 +290,6 @@ void ieee80211_sta_tear_down_BA_sessions(struct sta_info *sta,
{
int i;
- cancel_work_sync(&sta->ampdu_mlme.work);
-
for (i = 0; i < IEEE80211_NUM_TIDS; i++) {
__ieee80211_stop_tx_ba_session(sta, i, reason);
__ieee80211_stop_rx_ba_session(sta, i, WLAN_BACK_RECIPIENT,
@@ -298,6 +297,9 @@ void ieee80211_sta_tear_down_BA_sessions(struct sta_info *sta,
reason != AGG_STOP_DESTROY_STA &&
reason != AGG_STOP_PEER_REQUEST);
}
+
+ /* stopping might queue the work again - so cancel only afterwards */
+ cancel_work_sync(&sta->ampdu_mlme.work);
}
void ieee80211_ba_session_work(struct work_struct *work)
@@ -352,10 +354,16 @@ void ieee80211_ba_session_work(struct work_struct *work)
spin_unlock_bh(&sta->lock);
tid_tx = rcu_dereference_protected_tid_tx(sta, tid);
- if (tid_tx && test_and_clear_bit(HT_AGG_STATE_WANT_STOP,
- &tid_tx->state))
+ if (!tid_tx)
+ continue;
+
+ if (test_and_clear_bit(HT_AGG_STATE_START_CB, &tid_tx->state))
+ ieee80211_start_tx_ba_cb(sta, tid, tid_tx);
+ if (test_and_clear_bit(HT_AGG_STATE_WANT_STOP, &tid_tx->state))
___ieee80211_stop_tx_ba_session(sta, tid,
AGG_STOP_LOCAL_REQUEST);
+ if (test_and_clear_bit(HT_AGG_STATE_STOP_CB, &tid_tx->state))
+ ieee80211_stop_tx_ba_cb(sta, tid, tid_tx);
}
mutex_unlock(&sta->ampdu_mlme.mtx);
}
diff --git a/net/mac80211/ibss.c b/net/mac80211/ibss.c
index a5acaf1..0c0695e 100644
--- a/net/mac80211/ibss.c
+++ b/net/mac80211/ibss.c
@@ -948,8 +948,8 @@ static void ieee80211_rx_mgmt_deauth_ibss(struct ieee80211_sub_if_data *sdata,
if (len < IEEE80211_DEAUTH_FRAME_LEN)
return;
- ibss_dbg(sdata, "RX DeAuth SA=%pM DA=%pM BSSID=%pM (reason: %d)\n",
- mgmt->sa, mgmt->da, mgmt->bssid, reason);
+ ibss_dbg(sdata, "RX DeAuth SA=%pM DA=%pM\n", mgmt->sa, mgmt->da);
+ ibss_dbg(sdata, "\tBSSID=%pM (reason: %d)\n", mgmt->bssid, reason);
sta_info_destroy_addr(sdata, mgmt->sa);
}
@@ -967,9 +967,9 @@ static void ieee80211_rx_mgmt_auth_ibss(struct ieee80211_sub_if_data *sdata,
auth_alg = le16_to_cpu(mgmt->u.auth.auth_alg);
auth_transaction = le16_to_cpu(mgmt->u.auth.auth_transaction);
- ibss_dbg(sdata,
- "RX Auth SA=%pM DA=%pM BSSID=%pM (auth_transaction=%d)\n",
- mgmt->sa, mgmt->da, mgmt->bssid, auth_transaction);
+ ibss_dbg(sdata, "RX Auth SA=%pM DA=%pM\n", mgmt->sa, mgmt->da);
+ ibss_dbg(sdata, "\tBSSID=%pM (auth_transaction=%d)\n",
+ mgmt->bssid, auth_transaction);
if (auth_alg != WLAN_AUTH_OPEN || auth_transaction != 1)
return;
@@ -1176,10 +1176,10 @@ static void ieee80211_rx_bss_info(struct ieee80211_sub_if_data *sdata,
rx_timestamp = drv_get_tsf(local, sdata);
}
- ibss_dbg(sdata,
- "RX beacon SA=%pM BSSID=%pM TSF=0x%llx BCN=0x%llx diff=%lld @%lu\n",
+ ibss_dbg(sdata, "RX beacon SA=%pM BSSID=%pM TSF=0x%llx\n",
mgmt->sa, mgmt->bssid,
- (unsigned long long)rx_timestamp,
+ (unsigned long long)rx_timestamp);
+ ibss_dbg(sdata, "\tBCN=0x%llx diff=%lld @%lu\n",
(unsigned long long)beacon_timestamp,
(unsigned long long)(rx_timestamp - beacon_timestamp),
jiffies);
@@ -1538,9 +1538,9 @@ static void ieee80211_rx_mgmt_probe_req(struct ieee80211_sub_if_data *sdata,
tx_last_beacon = drv_tx_last_beacon(local);
- ibss_dbg(sdata,
- "RX ProbeReq SA=%pM DA=%pM BSSID=%pM (tx_last_beacon=%d)\n",
- mgmt->sa, mgmt->da, mgmt->bssid, tx_last_beacon);
+ ibss_dbg(sdata, "RX ProbeReq SA=%pM DA=%pM\n", mgmt->sa, mgmt->da);
+ ibss_dbg(sdata, "\tBSSID=%pM (tx_last_beacon=%d)\n",
+ mgmt->bssid, tx_last_beacon);
if (!tx_last_beacon && is_multicast_ether_addr(mgmt->da))
return;
diff --git a/net/mac80211/ieee80211_i.h b/net/mac80211/ieee80211_i.h
index 7fd544d..8a690eb 100644
--- a/net/mac80211/ieee80211_i.h
+++ b/net/mac80211/ieee80211_i.h
@@ -1026,8 +1026,6 @@ struct ieee80211_rx_agg {
enum sdata_queue_type {
IEEE80211_SDATA_QUEUE_TYPE_FRAME = 0,
- IEEE80211_SDATA_QUEUE_AGG_START = 1,
- IEEE80211_SDATA_QUEUE_AGG_STOP = 2,
IEEE80211_SDATA_QUEUE_RX_AGG_START = 3,
IEEE80211_SDATA_QUEUE_RX_AGG_STOP = 4,
};
@@ -1416,12 +1414,6 @@ ieee80211_get_sband(struct ieee80211_sub_if_data *sdata)
return local->hw.wiphy->bands[band];
}
-/* this struct represents 802.11n's RA/TID combination */
-struct ieee80211_ra_tid {
- u8 ra[ETH_ALEN];
- u16 tid;
-};
-
/* this struct holds the value parsing from channel switch IE */
struct ieee80211_csa_ie {
struct cfg80211_chan_def chandef;
@@ -1765,8 +1757,10 @@ int __ieee80211_stop_tx_ba_session(struct sta_info *sta, u16 tid,
enum ieee80211_agg_stop_reason reason);
int ___ieee80211_stop_tx_ba_session(struct sta_info *sta, u16 tid,
enum ieee80211_agg_stop_reason reason);
-void ieee80211_start_tx_ba_cb(struct ieee80211_vif *vif, u8 *ra, u16 tid);
-void ieee80211_stop_tx_ba_cb(struct ieee80211_vif *vif, u8 *ra, u8 tid);
+void ieee80211_start_tx_ba_cb(struct sta_info *sta, int tid,
+ struct tid_ampdu_tx *tid_tx);
+void ieee80211_stop_tx_ba_cb(struct sta_info *sta, int tid,
+ struct tid_ampdu_tx *tid_tx);
void ieee80211_ba_session_work(struct work_struct *work);
void ieee80211_tx_ba_session_handle_start(struct sta_info *sta, int tid);
void ieee80211_release_reorder_timeout(struct sta_info *sta, int tid);
diff --git a/net/mac80211/iface.c b/net/mac80211/iface.c
index fa7d757..5768560 100644
--- a/net/mac80211/iface.c
+++ b/net/mac80211/iface.c
@@ -1025,6 +1025,8 @@ static void ieee80211_do_stop(struct ieee80211_sub_if_data *sdata,
if (local->open_count == 0)
ieee80211_clear_tx_pending(local);
+ sdata->vif.bss_conf.beacon_int = 0;
+
/*
* If the interface goes down while suspended, presumably because
* the device was unplugged and that happens before our resume,
@@ -1248,7 +1250,6 @@ static void ieee80211_iface_work(struct work_struct *work)
struct ieee80211_local *local = sdata->local;
struct sk_buff *skb;
struct sta_info *sta;
- struct ieee80211_ra_tid *ra_tid;
struct ieee80211_rx_agg *rx_agg;
if (!ieee80211_sdata_running(sdata))
@@ -1264,15 +1265,7 @@ static void ieee80211_iface_work(struct work_struct *work)
while ((skb = skb_dequeue(&sdata->skb_queue))) {
struct ieee80211_mgmt *mgmt = (void *)skb->data;
- if (skb->pkt_type == IEEE80211_SDATA_QUEUE_AGG_START) {
- ra_tid = (void *)&skb->cb;
- ieee80211_start_tx_ba_cb(&sdata->vif, ra_tid->ra,
- ra_tid->tid);
- } else if (skb->pkt_type == IEEE80211_SDATA_QUEUE_AGG_STOP) {
- ra_tid = (void *)&skb->cb;
- ieee80211_stop_tx_ba_cb(&sdata->vif, ra_tid->ra,
- ra_tid->tid);
- } else if (skb->pkt_type == IEEE80211_SDATA_QUEUE_RX_AGG_START) {
+ if (skb->pkt_type == IEEE80211_SDATA_QUEUE_RX_AGG_START) {
rx_agg = (void *)&skb->cb;
mutex_lock(&local->sta_mtx);
sta = sta_info_get_bss(sdata, rx_agg->addr);
diff --git a/net/mac80211/key.c b/net/mac80211/key.c
index 4c625a3..6e02f8d 100644
--- a/net/mac80211/key.c
+++ b/net/mac80211/key.c
@@ -648,11 +648,15 @@ int ieee80211_key_link(struct ieee80211_key *key,
{
struct ieee80211_local *local = sdata->local;
struct ieee80211_key *old_key;
- int idx, ret;
- bool pairwise;
-
- pairwise = key->conf.flags & IEEE80211_KEY_FLAG_PAIRWISE;
- idx = key->conf.keyidx;
+ int idx = key->conf.keyidx;
+ bool pairwise = key->conf.flags & IEEE80211_KEY_FLAG_PAIRWISE;
+ /*
+ * We want to delay tailroom updates only for station - in that
+ * case it helps roaming speed, but in other cases it hurts and
+ * can cause warnings to appear.
+ */
+ bool delay_tailroom = sdata->vif.type == NL80211_IFTYPE_STATION;
+ int ret;
mutex_lock(&sdata->local->key_mtx);
@@ -680,14 +684,14 @@ int ieee80211_key_link(struct ieee80211_key *key,
increment_tailroom_need_count(sdata);
ieee80211_key_replace(sdata, sta, pairwise, old_key, key);
- ieee80211_key_destroy(old_key, true);
+ ieee80211_key_destroy(old_key, delay_tailroom);
ieee80211_debugfs_key_add(key);
if (!local->wowlan) {
ret = ieee80211_key_enable_hw_accel(key);
if (ret)
- ieee80211_key_free(key, true);
+ ieee80211_key_free(key, delay_tailroom);
} else {
ret = 0;
}
@@ -922,7 +926,8 @@ void ieee80211_free_sta_keys(struct ieee80211_local *local,
ieee80211_key_replace(key->sdata, key->sta,
key->conf.flags & IEEE80211_KEY_FLAG_PAIRWISE,
key, NULL);
- __ieee80211_key_destroy(key, true);
+ __ieee80211_key_destroy(key, key->sdata->vif.type ==
+ NL80211_IFTYPE_STATION);
}
for (i = 0; i < NUM_DEFAULT_KEYS; i++) {
@@ -932,7 +937,8 @@ void ieee80211_free_sta_keys(struct ieee80211_local *local,
ieee80211_key_replace(key->sdata, key->sta,
key->conf.flags & IEEE80211_KEY_FLAG_PAIRWISE,
key, NULL);
- __ieee80211_key_destroy(key, true);
+ __ieee80211_key_destroy(key, key->sdata->vif.type ==
+ NL80211_IFTYPE_STATION);
}
mutex_unlock(&local->key_mtx);
diff --git a/net/mac80211/main.c b/net/mac80211/main.c
index 2bb6899..e3bbfb2 100644
--- a/net/mac80211/main.c
+++ b/net/mac80211/main.c
@@ -254,8 +254,27 @@ static void ieee80211_restart_work(struct work_struct *work)
"%s called with hardware scan in progress\n", __func__);
rtnl_lock();
- list_for_each_entry(sdata, &local->interfaces, list)
+ list_for_each_entry(sdata, &local->interfaces, list) {
+ /*
+ * XXX: there may be more work for other vif types and even
+ * for station mode: a good thing would be to run most of
+ * the iface type's dependent _stop (ieee80211_mg_stop,
+ * ieee80211_ibss_stop) etc...
+ * For now, fix only the specific bug that was seen: race
+ * between csa_connection_drop_work and us.
+ */
+ if (sdata->vif.type == NL80211_IFTYPE_STATION) {
+ /*
+ * This worker is scheduled from the iface worker that
+ * runs on mac80211's workqueue, so we can't be
+ * scheduling this worker after the cancel right here.
+ * The exception is ieee80211_chswitch_done.
+ * Then we can have a race...
+ */
+ cancel_work_sync(&sdata->u.mgd.csa_connection_drop_work);
+ }
flush_delayed_work(&sdata->dec_tailroom_needed_wk);
+ }
ieee80211_scan_cancel(local);
/* make sure any new ROC will consider local->in_reconfig */
@@ -466,10 +485,7 @@ static const struct ieee80211_vht_cap mac80211_vht_capa_mod_mask = {
cpu_to_le32(IEEE80211_VHT_CAP_RXLDPC |
IEEE80211_VHT_CAP_SHORT_GI_80 |
IEEE80211_VHT_CAP_SHORT_GI_160 |
- IEEE80211_VHT_CAP_RXSTBC_1 |
- IEEE80211_VHT_CAP_RXSTBC_2 |
- IEEE80211_VHT_CAP_RXSTBC_3 |
- IEEE80211_VHT_CAP_RXSTBC_4 |
+ IEEE80211_VHT_CAP_RXSTBC_MASK |
IEEE80211_VHT_CAP_TXSTBC |
IEEE80211_VHT_CAP_SU_BEAMFORMER_CAPABLE |
IEEE80211_VHT_CAP_SU_BEAMFORMEE_CAPABLE |
@@ -1164,6 +1180,7 @@ void ieee80211_unregister_hw(struct ieee80211_hw *hw)
#if IS_ENABLED(CONFIG_IPV6)
unregister_inet6addr_notifier(&local->ifa6_notifier);
#endif
+ ieee80211_txq_teardown_flows(local);
rtnl_lock();
@@ -1191,7 +1208,6 @@ void ieee80211_unregister_hw(struct ieee80211_hw *hw)
skb_queue_purge(&local->skb_queue);
skb_queue_purge(&local->skb_queue_unreliable);
skb_queue_purge(&local->skb_queue_tdls_chsw);
- ieee80211_txq_teardown_flows(local);
destroy_workqueue(local->workqueue);
wiphy_unregister(local->hw.wiphy);
diff --git a/net/mac80211/mesh_hwmp.c b/net/mac80211/mesh_hwmp.c
index fed598a..b0acb29 100644
--- a/net/mac80211/mesh_hwmp.c
+++ b/net/mac80211/mesh_hwmp.c
@@ -563,6 +563,10 @@ static void hwmp_preq_frame_process(struct ieee80211_sub_if_data *sdata,
forward = false;
reply = true;
target_metric = 0;
+
+ if (SN_GT(target_sn, ifmsh->sn))
+ ifmsh->sn = target_sn;
+
if (time_after(jiffies, ifmsh->last_sn_update +
net_traversal_jiffies(sdata)) ||
time_before(jiffies, ifmsh->last_sn_update)) {
diff --git a/net/mac80211/mlme.c b/net/mac80211/mlme.c
index 973adf3..a37fc00 100644
--- a/net/mac80211/mlme.c
+++ b/net/mac80211/mlme.c
@@ -989,6 +989,10 @@ static void ieee80211_chswitch_work(struct work_struct *work)
*/
if (sdata->reserved_chanctx) {
+ struct ieee80211_supported_band *sband = NULL;
+ struct sta_info *mgd_sta = NULL;
+ enum ieee80211_sta_rx_bandwidth bw = IEEE80211_STA_RX_BW_20;
+
/*
* with multi-vif csa driver may call ieee80211_csa_finish()
* many times while waiting for other interfaces to use their
@@ -997,6 +1001,48 @@ static void ieee80211_chswitch_work(struct work_struct *work)
if (sdata->reserved_ready)
goto out;
+ if (sdata->vif.bss_conf.chandef.width !=
+ sdata->csa_chandef.width) {
+ /*
+ * For managed interface, we need to also update the AP
+ * station bandwidth and align the rate scale algorithm
+ * on the bandwidth change. Here we only consider the
+ * bandwidth of the new channel definition (as channel
+ * switch flow does not have the full HT/VHT/HE
+ * information), assuming that if additional changes are
+ * required they would be done as part of the processing
+ * of the next beacon from the AP.
+ */
+ switch (sdata->csa_chandef.width) {
+ case NL80211_CHAN_WIDTH_20_NOHT:
+ case NL80211_CHAN_WIDTH_20:
+ default:
+ bw = IEEE80211_STA_RX_BW_20;
+ break;
+ case NL80211_CHAN_WIDTH_40:
+ bw = IEEE80211_STA_RX_BW_40;
+ break;
+ case NL80211_CHAN_WIDTH_80:
+ bw = IEEE80211_STA_RX_BW_80;
+ break;
+ case NL80211_CHAN_WIDTH_80P80:
+ case NL80211_CHAN_WIDTH_160:
+ bw = IEEE80211_STA_RX_BW_160;
+ break;
+ }
+
+ mgd_sta = sta_info_get(sdata, ifmgd->bssid);
+ sband =
+ local->hw.wiphy->bands[sdata->csa_chandef.chan->band];
+ }
+
+ if (sdata->vif.bss_conf.chandef.width >
+ sdata->csa_chandef.width) {
+ mgd_sta->sta.bandwidth = bw;
+ rate_control_rate_update(local, sband, mgd_sta,
+ IEEE80211_RC_BW_CHANGED);
+ }
+
ret = ieee80211_vif_use_reserved_context(sdata);
if (ret) {
sdata_info(sdata,
@@ -1007,6 +1053,13 @@ static void ieee80211_chswitch_work(struct work_struct *work)
goto out;
}
+ if (sdata->vif.bss_conf.chandef.width <
+ sdata->csa_chandef.width) {
+ mgd_sta->sta.bandwidth = bw;
+ rate_control_rate_update(local, sband, mgd_sta,
+ IEEE80211_RC_BW_CHANGED);
+ }
+
goto out;
}
@@ -1229,6 +1282,16 @@ ieee80211_sta_process_chanswitch(struct ieee80211_sub_if_data *sdata,
cbss->beacon_interval));
return;
drop_connection:
+ /*
+ * This is just so that the disconnect flow will know that
+ * we were trying to switch channel and failed. In case the
+ * mode is 1 (we are not allowed to Tx), we will know not to
+ * send a deauthentication frame. Those two fields will be
+ * reset when the disconnection worker runs.
+ */
+ sdata->vif.csa_active = true;
+ sdata->csa_block_tx = csa_ie.mode;
+
ieee80211_queue_work(&local->hw, &ifmgd->csa_connection_drop_work);
mutex_unlock(&local->chanctx_mtx);
mutex_unlock(&local->mtx);
@@ -1804,7 +1867,8 @@ static bool ieee80211_sta_wmm_params(struct ieee80211_local *local,
params[ac].acm = acm;
params[ac].uapsd = uapsd;
- if (params[ac].cw_min > params[ac].cw_max) {
+ if (params[ac].cw_min == 0 ||
+ params[ac].cw_min > params[ac].cw_max) {
sdata_info(sdata,
"AP has invalid WMM params (CWmin/max=%d/%d for ACI %d), using defaults\n",
params[ac].cw_min, params[ac].cw_max, aci);
@@ -2401,6 +2465,7 @@ static void __ieee80211_disconnect(struct ieee80211_sub_if_data *sdata)
struct ieee80211_local *local = sdata->local;
struct ieee80211_if_managed *ifmgd = &sdata->u.mgd;
u8 frame_buf[IEEE80211_DEAUTH_FRAME_LEN];
+ bool tx;
sdata_lock(sdata);
if (!ifmgd->associated) {
@@ -2408,6 +2473,8 @@ static void __ieee80211_disconnect(struct ieee80211_sub_if_data *sdata)
return;
}
+ tx = !sdata->csa_block_tx;
+
/* AP is probably out of range (or not reachable for another reason) so
* remove the bss struct for that AP.
*/
@@ -2415,7 +2482,7 @@ static void __ieee80211_disconnect(struct ieee80211_sub_if_data *sdata)
ieee80211_set_disassoc(sdata, IEEE80211_STYPE_DEAUTH,
WLAN_REASON_DISASSOC_DUE_TO_INACTIVITY,
- true, frame_buf);
+ tx, frame_buf);
mutex_lock(&local->mtx);
sdata->vif.csa_active = false;
ifmgd->csa_waiting_bcn = false;
@@ -2426,7 +2493,7 @@ static void __ieee80211_disconnect(struct ieee80211_sub_if_data *sdata)
}
mutex_unlock(&local->mtx);
- ieee80211_report_disconnect(sdata, frame_buf, sizeof(frame_buf), true,
+ ieee80211_report_disconnect(sdata, frame_buf, sizeof(frame_buf), tx,
WLAN_REASON_DISASSOC_DUE_TO_INACTIVITY);
sdata_unlock(sdata);
diff --git a/net/mac80211/rx.c b/net/mac80211/rx.c
index 404284a..93c3327 100644
--- a/net/mac80211/rx.c
+++ b/net/mac80211/rx.c
@@ -1230,6 +1230,7 @@ ieee80211_rx_h_check_dup(struct ieee80211_rx_data *rx)
return RX_CONTINUE;
if (ieee80211_is_ctl(hdr->frame_control) ||
+ ieee80211_is_nullfunc(hdr->frame_control) ||
ieee80211_is_qos_nullfunc(hdr->frame_control) ||
is_multicast_ether_addr(hdr->addr1))
return RX_CONTINUE;
@@ -3907,7 +3908,7 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx,
if ((hdr->frame_control & cpu_to_le16(IEEE80211_FCTL_FROMDS |
IEEE80211_FCTL_TODS)) !=
fast_rx->expected_ds_bits)
- goto drop;
+ return false;
/* assign the key to drop unencrypted frames (later)
* and strip the IV/MIC if necessary
diff --git a/net/mac80211/spectmgmt.c b/net/mac80211/spectmgmt.c
index 97f4c9d..9249712 100644
--- a/net/mac80211/spectmgmt.c
+++ b/net/mac80211/spectmgmt.c
@@ -8,6 +8,7 @@
* Copyright 2007, Michael Wu <flamingice@sourmilk.net>
* Copyright 2007-2008, Intel Corporation
* Copyright 2008, Johannes Berg <johannes@sipsolutions.net>
+ * Copyright (C) 2018 Intel Corporation
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 2 as
@@ -27,7 +28,7 @@ int ieee80211_parse_ch_switch_ie(struct ieee80211_sub_if_data *sdata,
u32 sta_flags, u8 *bssid,
struct ieee80211_csa_ie *csa_ie)
{
- enum nl80211_band new_band;
+ enum nl80211_band new_band = current_band;
int new_freq;
u8 new_chan_no;
struct ieee80211_channel *new_chan;
@@ -53,15 +54,13 @@ int ieee80211_parse_ch_switch_ie(struct ieee80211_sub_if_data *sdata,
elems->ext_chansw_ie->new_operating_class,
&new_band)) {
sdata_info(sdata,
- "cannot understand ECSA IE operating class %d, disconnecting\n",
+ "cannot understand ECSA IE operating class, %d, ignoring\n",
elems->ext_chansw_ie->new_operating_class);
- return -EINVAL;
}
new_chan_no = elems->ext_chansw_ie->new_ch_num;
csa_ie->count = elems->ext_chansw_ie->count;
csa_ie->mode = elems->ext_chansw_ie->mode;
} else if (elems->ch_switch_ie) {
- new_band = current_band;
new_chan_no = elems->ch_switch_ie->new_ch_num;
csa_ie->count = elems->ch_switch_ie->count;
csa_ie->mode = elems->ch_switch_ie->mode;
diff --git a/net/mac80211/sta_info.c b/net/mac80211/sta_info.c
index 1ecf3d0..892c392 100644
--- a/net/mac80211/sta_info.c
+++ b/net/mac80211/sta_info.c
@@ -313,7 +313,7 @@ struct sta_info *sta_info_alloc(struct ieee80211_sub_if_data *sdata,
if (ieee80211_hw_check(hw, USES_RSS)) {
sta->pcpu_rx_stats =
- alloc_percpu(struct ieee80211_sta_rx_stats);
+ alloc_percpu_gfp(struct ieee80211_sta_rx_stats, gfp);
if (!sta->pcpu_rx_stats)
goto free;
}
@@ -433,6 +433,7 @@ struct sta_info *sta_info_alloc(struct ieee80211_sub_if_data *sdata,
if (sta->sta.txq[0])
kfree(to_txq_info(sta->sta.txq[0]));
free:
+ free_percpu(sta->pcpu_rx_stats);
#ifdef CONFIG_MAC80211_MESH
kfree(sta->mesh);
#endif
diff --git a/net/mac80211/sta_info.h b/net/mac80211/sta_info.h
index 15599c7..cc808ac 100644
--- a/net/mac80211/sta_info.h
+++ b/net/mac80211/sta_info.h
@@ -115,6 +115,8 @@ enum ieee80211_sta_info_flags {
#define HT_AGG_STATE_STOPPING 3
#define HT_AGG_STATE_WANT_START 4
#define HT_AGG_STATE_WANT_STOP 5
+#define HT_AGG_STATE_START_CB 6
+#define HT_AGG_STATE_STOP_CB 7
enum ieee80211_agg_stop_reason {
AGG_STOP_DECLINED,
diff --git a/net/mac80211/status.c b/net/mac80211/status.c
index 72fe9bc..246d113 100644
--- a/net/mac80211/status.c
+++ b/net/mac80211/status.c
@@ -472,11 +472,6 @@ static void ieee80211_report_ack_skb(struct ieee80211_local *local,
if (!skb)
return;
- if (dropped) {
- dev_kfree_skb_any(skb);
- return;
- }
-
if (info->flags & IEEE80211_TX_INTFL_NL80211_FRAME_TX) {
u64 cookie = IEEE80211_SKB_CB(skb)->ack.cookie;
struct ieee80211_sub_if_data *sdata;
@@ -498,6 +493,8 @@ static void ieee80211_report_ack_skb(struct ieee80211_local *local,
rcu_read_unlock();
dev_kfree_skb_any(skb);
+ } else if (dropped) {
+ dev_kfree_skb_any(skb);
} else {
/* consumes skb */
skb_complete_wifi_ack(skb, acked);
@@ -663,6 +660,8 @@ void ieee80211_tx_status_noskb(struct ieee80211_hw *hw,
/* Track when last TDLS packet was ACKed */
if (test_sta_flag(sta, WLAN_STA_TDLS_PEER_AUTH))
sta->status_stats.last_tdls_pkt_time = jiffies;
+ } else if (test_sta_flag(sta, WLAN_STA_PS_STA)) {
+ return;
} else {
ieee80211_lost_packet(sta, info);
}
diff --git a/net/mac80211/tdls.c b/net/mac80211/tdls.c
index f20dcf1..c64ae68 100644
--- a/net/mac80211/tdls.c
+++ b/net/mac80211/tdls.c
@@ -16,6 +16,7 @@
#include "ieee80211_i.h"
#include "driver-ops.h"
#include "rate.h"
+#include "wme.h"
/* give usermode some time for retries in setting up the TDLS session */
#define TDLS_PEER_SETUP_TIMEOUT (15 * HZ)
@@ -1019,14 +1020,13 @@ ieee80211_tdls_prep_mgmt_packet(struct wiphy *wiphy, struct net_device *dev,
switch (action_code) {
case WLAN_TDLS_SETUP_REQUEST:
case WLAN_TDLS_SETUP_RESPONSE:
- skb_set_queue_mapping(skb, IEEE80211_AC_BK);
- skb->priority = 2;
+ skb->priority = 256 + 2;
break;
default:
- skb_set_queue_mapping(skb, IEEE80211_AC_VI);
- skb->priority = 5;
+ skb->priority = 256 + 5;
break;
}
+ skb_set_queue_mapping(skb, ieee80211_select_queue(sdata, skb));
/*
* Set the WLAN_TDLS_TEARDOWN flag to indicate a teardown in progress.
diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c
index 8458299..6a0fb9d 100644
--- a/net/mac80211/tx.c
+++ b/net/mac80211/tx.c
@@ -434,8 +434,8 @@ ieee80211_tx_h_multicast_ps_buf(struct ieee80211_tx_data *tx)
if (ieee80211_hw_check(&tx->local->hw, QUEUE_CONTROL))
info->hw_queue = tx->sdata->vif.cab_queue;
- /* no stations in PS mode */
- if (!atomic_read(&ps->num_sta_ps))
+ /* no stations in PS mode and no buffered packets */
+ if (!atomic_read(&ps->num_sta_ps) && skb_queue_empty(&ps->bc_buf))
return TX_CONTINUE;
info->flags |= IEEE80211_TX_CTL_SEND_AFTER_DTIM;
@@ -1833,7 +1833,7 @@ static bool ieee80211_tx(struct ieee80211_sub_if_data *sdata,
sdata->vif.hw_queue[skb_get_queue_mapping(skb)];
if (invoke_tx_handlers_early(&tx))
- return false;
+ return true;
if (ieee80211_queue_skb(local, sdata, tx.sta, tx.skb))
return true;
diff --git a/net/mac80211/util.c b/net/mac80211/util.c
index ae91a3c..0bb144c 100644
--- a/net/mac80211/util.c
+++ b/net/mac80211/util.c
@@ -2061,7 +2061,8 @@ int ieee80211_reconfig(struct ieee80211_local *local)
if (!sta->uploaded)
continue;
- if (sta->sdata->vif.type != NL80211_IFTYPE_AP)
+ if (sta->sdata->vif.type != NL80211_IFTYPE_AP &&
+ sta->sdata->vif.type != NL80211_IFTYPE_AP_VLAN)
continue;
for (state = IEEE80211_STA_NOTEXIST;
diff --git a/net/mac802154/tx.c b/net/mac802154/tx.c
index 7e25345..bcd1a5e 100644
--- a/net/mac802154/tx.c
+++ b/net/mac802154/tx.c
@@ -63,8 +63,21 @@ ieee802154_tx(struct ieee802154_local *local, struct sk_buff *skb)
int ret;
if (!(local->hw.flags & IEEE802154_HW_TX_OMIT_CKSUM)) {
- u16 crc = crc_ccitt(0, skb->data, skb->len);
+ struct sk_buff *nskb;
+ u16 crc;
+ if (unlikely(skb_tailroom(skb) < IEEE802154_FCS_LEN)) {
+ nskb = skb_copy_expand(skb, 0, IEEE802154_FCS_LEN,
+ GFP_ATOMIC);
+ if (likely(nskb)) {
+ consume_skb(skb);
+ skb = nskb;
+ } else {
+ goto err_tx;
+ }
+ }
+
+ crc = crc_ccitt(0, skb->data, skb->len);
put_unaligned_le16(crc, skb_put(skb, 2));
}
diff --git a/net/netfilter/Makefile b/net/netfilter/Makefile
index 8b29a57..512c9ec 100644
--- a/net/netfilter/Makefile
+++ b/net/netfilter/Makefile
@@ -188,3 +188,7 @@
# IPVS
obj-$(CONFIG_IP_VS) += ipvs/
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-pointer-bool-conversion \
diff --git a/net/netfilter/ipset/ip_set_hash_netportnet.c b/net/netfilter/ipset/ip_set_hash_netportnet.c
index 9a14c23..b259a58 100644
--- a/net/netfilter/ipset/ip_set_hash_netportnet.c
+++ b/net/netfilter/ipset/ip_set_hash_netportnet.c
@@ -213,13 +213,13 @@ hash_netportnet4_uadt(struct ip_set *set, struct nlattr *tb[],
if (tb[IPSET_ATTR_CIDR]) {
e.cidr[0] = nla_get_u8(tb[IPSET_ATTR_CIDR]);
- if (!e.cidr[0] || e.cidr[0] > HOST_MASK)
+ if (e.cidr[0] > HOST_MASK)
return -IPSET_ERR_INVALID_CIDR;
}
if (tb[IPSET_ATTR_CIDR2]) {
e.cidr[1] = nla_get_u8(tb[IPSET_ATTR_CIDR2]);
- if (!e.cidr[1] || e.cidr[1] > HOST_MASK)
+ if (e.cidr[1] > HOST_MASK)
return -IPSET_ERR_INVALID_CIDR;
}
@@ -492,13 +492,13 @@ hash_netportnet6_uadt(struct ip_set *set, struct nlattr *tb[],
if (tb[IPSET_ATTR_CIDR]) {
e.cidr[0] = nla_get_u8(tb[IPSET_ATTR_CIDR]);
- if (!e.cidr[0] || e.cidr[0] > HOST_MASK)
+ if (e.cidr[0] > HOST_MASK)
return -IPSET_ERR_INVALID_CIDR;
}
if (tb[IPSET_ATTR_CIDR2]) {
e.cidr[1] = nla_get_u8(tb[IPSET_ATTR_CIDR2]);
- if (!e.cidr[1] || e.cidr[1] > HOST_MASK)
+ if (e.cidr[1] > HOST_MASK)
return -IPSET_ERR_INVALID_CIDR;
}
diff --git a/net/netfilter/ipset/ip_set_list_set.c b/net/netfilter/ipset/ip_set_list_set.c
index a2a89e4..e821572 100644
--- a/net/netfilter/ipset/ip_set_list_set.c
+++ b/net/netfilter/ipset/ip_set_list_set.c
@@ -518,8 +518,8 @@ list_set_list(const struct ip_set *set,
ret = -EMSGSIZE;
} else {
cb->args[IPSET_CB_ARG0] = i;
+ ipset_nest_end(skb, atd);
}
- ipset_nest_end(skb, atd);
out:
rcu_read_unlock();
return ret;
diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
index e34d3f6..fd186b0 100644
--- a/net/netfilter/ipvs/ip_vs_core.c
+++ b/net/netfilter/ipvs/ip_vs_core.c
@@ -1968,13 +1968,20 @@ ip_vs_in(struct netns_ipvs *ipvs, unsigned int hooknum, struct sk_buff *skb, int
if (cp->dest && !(cp->dest->flags & IP_VS_DEST_F_AVAILABLE)) {
/* the destination server is not available */
- if (sysctl_expire_nodest_conn(ipvs)) {
+ __u32 flags = cp->flags;
+
+ /* when timer already started, silently drop the packet.*/
+ if (timer_pending(&cp->timer))
+ __ip_vs_conn_put(cp);
+ else
+ ip_vs_conn_put(cp);
+
+ if (sysctl_expire_nodest_conn(ipvs) &&
+ !(flags & IP_VS_CONN_F_ONE_PACKET)) {
/* try to expire the connection immediately */
ip_vs_conn_expire_now(cp);
}
- /* don't restart its timer, and silently
- drop the packet. */
- __ip_vs_conn_put(cp);
+
return NF_DROP;
}
diff --git a/net/netfilter/ipvs/ip_vs_ctl.c b/net/netfilter/ipvs/ip_vs_ctl.c
index 74d1195..8382b78 100644
--- a/net/netfilter/ipvs/ip_vs_ctl.c
+++ b/net/netfilter/ipvs/ip_vs_ctl.c
@@ -2390,14 +2390,12 @@ do_ip_vs_set_ctl(struct sock *sk, int cmd, void __user *user, unsigned int len)
struct ipvs_sync_daemon_cfg cfg;
memset(&cfg, 0, sizeof(cfg));
- strlcpy(cfg.mcast_ifn, dm->mcast_ifn,
- sizeof(cfg.mcast_ifn));
+ ret = -EINVAL;
+ if (strscpy(cfg.mcast_ifn, dm->mcast_ifn,
+ sizeof(cfg.mcast_ifn)) <= 0)
+ goto out_dec;
cfg.syncid = dm->syncid;
- rtnl_lock();
- mutex_lock(&ipvs->sync_mutex);
ret = start_sync_thread(ipvs, &cfg, dm->state);
- mutex_unlock(&ipvs->sync_mutex);
- rtnl_unlock();
} else {
mutex_lock(&ipvs->sync_mutex);
ret = stop_sync_thread(ipvs, dm->state);
@@ -2433,12 +2431,19 @@ do_ip_vs_set_ctl(struct sock *sk, int cmd, void __user *user, unsigned int len)
}
}
+ if ((cmd == IP_VS_SO_SET_ADD || cmd == IP_VS_SO_SET_EDIT) &&
+ strnlen(usvc.sched_name, IP_VS_SCHEDNAME_MAXLEN) ==
+ IP_VS_SCHEDNAME_MAXLEN) {
+ ret = -EINVAL;
+ goto out_unlock;
+ }
+
/* Check for valid protocol: TCP or UDP or SCTP, even for fwmark!=0 */
if (usvc.protocol != IPPROTO_TCP && usvc.protocol != IPPROTO_UDP &&
usvc.protocol != IPPROTO_SCTP) {
- pr_err("set_ctl: invalid protocol: %d %pI4:%d %s\n",
+ pr_err("set_ctl: invalid protocol: %d %pI4:%d\n",
usvc.protocol, &usvc.addr.ip,
- ntohs(usvc.port), usvc.sched_name);
+ ntohs(usvc.port));
ret = -EFAULT;
goto out_unlock;
}
@@ -2867,7 +2872,7 @@ static const struct nla_policy ip_vs_cmd_policy[IPVS_CMD_ATTR_MAX + 1] = {
static const struct nla_policy ip_vs_daemon_policy[IPVS_DAEMON_ATTR_MAX + 1] = {
[IPVS_DAEMON_ATTR_STATE] = { .type = NLA_U32 },
[IPVS_DAEMON_ATTR_MCAST_IFN] = { .type = NLA_NUL_STRING,
- .len = IP_VS_IFNAME_MAXLEN },
+ .len = IP_VS_IFNAME_MAXLEN - 1 },
[IPVS_DAEMON_ATTR_SYNC_ID] = { .type = NLA_U32 },
[IPVS_DAEMON_ATTR_SYNC_MAXLEN] = { .type = NLA_U16 },
[IPVS_DAEMON_ATTR_MCAST_GROUP] = { .type = NLA_U32 },
@@ -2885,7 +2890,7 @@ static const struct nla_policy ip_vs_svc_policy[IPVS_SVC_ATTR_MAX + 1] = {
[IPVS_SVC_ATTR_PORT] = { .type = NLA_U16 },
[IPVS_SVC_ATTR_FWMARK] = { .type = NLA_U32 },
[IPVS_SVC_ATTR_SCHED_NAME] = { .type = NLA_NUL_STRING,
- .len = IP_VS_SCHEDNAME_MAXLEN },
+ .len = IP_VS_SCHEDNAME_MAXLEN - 1 },
[IPVS_SVC_ATTR_PE_NAME] = { .type = NLA_NUL_STRING,
.len = IP_VS_PENAME_MAXLEN },
[IPVS_SVC_ATTR_FLAGS] = { .type = NLA_BINARY,
@@ -3495,12 +3500,8 @@ static int ip_vs_genl_new_daemon(struct netns_ipvs *ipvs, struct nlattr **attrs)
if (ipvs->mixed_address_family_dests > 0)
return -EINVAL;
- rtnl_lock();
- mutex_lock(&ipvs->sync_mutex);
ret = start_sync_thread(ipvs, &c,
nla_get_u32(attrs[IPVS_DAEMON_ATTR_STATE]));
- mutex_unlock(&ipvs->sync_mutex);
- rtnl_unlock();
return ret;
}
@@ -4012,6 +4013,9 @@ static void __net_exit ip_vs_control_net_cleanup_sysctl(struct netns_ipvs *ipvs)
static struct notifier_block ip_vs_dst_notifier = {
.notifier_call = ip_vs_dst_event,
+#ifdef CONFIG_IP_VS_IPV6
+ .priority = ADDRCONF_NOTIFY_PRIORITY + 5,
+#endif
};
int __net_init ip_vs_control_net_init(struct netns_ipvs *ipvs)
diff --git a/net/netfilter/ipvs/ip_vs_sync.c b/net/netfilter/ipvs/ip_vs_sync.c
index 9350530..5fbf4b2 100644
--- a/net/netfilter/ipvs/ip_vs_sync.c
+++ b/net/netfilter/ipvs/ip_vs_sync.c
@@ -48,6 +48,7 @@
#include <linux/kthread.h>
#include <linux/wait.h>
#include <linux/kernel.h>
+#include <linux/sched.h>
#include <asm/unaligned.h> /* Used for ntoh_seq and hton_seq */
@@ -1359,15 +1360,9 @@ static void set_mcast_pmtudisc(struct sock *sk, int val)
/*
* Specifiy default interface for outgoing multicasts
*/
-static int set_mcast_if(struct sock *sk, char *ifname)
+static int set_mcast_if(struct sock *sk, struct net_device *dev)
{
- struct net_device *dev;
struct inet_sock *inet = inet_sk(sk);
- struct net *net = sock_net(sk);
-
- dev = __dev_get_by_name(net, ifname);
- if (!dev)
- return -ENODEV;
if (sk->sk_bound_dev_if && dev->ifindex != sk->sk_bound_dev_if)
return -EINVAL;
@@ -1395,19 +1390,14 @@ static int set_mcast_if(struct sock *sk, char *ifname)
* in the in_addr structure passed in as a parameter.
*/
static int
-join_mcast_group(struct sock *sk, struct in_addr *addr, char *ifname)
+join_mcast_group(struct sock *sk, struct in_addr *addr, struct net_device *dev)
{
- struct net *net = sock_net(sk);
struct ip_mreqn mreq;
- struct net_device *dev;
int ret;
memset(&mreq, 0, sizeof(mreq));
memcpy(&mreq.imr_multiaddr, addr, sizeof(struct in_addr));
- dev = __dev_get_by_name(net, ifname);
- if (!dev)
- return -ENODEV;
if (sk->sk_bound_dev_if && dev->ifindex != sk->sk_bound_dev_if)
return -EINVAL;
@@ -1422,15 +1412,10 @@ join_mcast_group(struct sock *sk, struct in_addr *addr, char *ifname)
#ifdef CONFIG_IP_VS_IPV6
static int join_mcast_group6(struct sock *sk, struct in6_addr *addr,
- char *ifname)
+ struct net_device *dev)
{
- struct net *net = sock_net(sk);
- struct net_device *dev;
int ret;
- dev = __dev_get_by_name(net, ifname);
- if (!dev)
- return -ENODEV;
if (sk->sk_bound_dev_if && dev->ifindex != sk->sk_bound_dev_if)
return -EINVAL;
@@ -1442,24 +1427,18 @@ static int join_mcast_group6(struct sock *sk, struct in6_addr *addr,
}
#endif
-static int bind_mcastif_addr(struct socket *sock, char *ifname)
+static int bind_mcastif_addr(struct socket *sock, struct net_device *dev)
{
- struct net *net = sock_net(sock->sk);
- struct net_device *dev;
__be32 addr;
struct sockaddr_in sin;
- dev = __dev_get_by_name(net, ifname);
- if (!dev)
- return -ENODEV;
-
addr = inet_select_addr(dev, 0, RT_SCOPE_UNIVERSE);
if (!addr)
pr_err("You probably need to specify IP address on "
"multicast interface.\n");
IP_VS_DBG(7, "binding socket with (%s) %pI4\n",
- ifname, &addr);
+ dev->name, &addr);
/* Now bind the socket with the address of multicast interface */
sin.sin_family = AF_INET;
@@ -1492,7 +1471,8 @@ static void get_mcast_sockaddr(union ipvs_sockaddr *sa, int *salen,
/*
* Set up sending multicast socket over UDP
*/
-static struct socket *make_send_sock(struct netns_ipvs *ipvs, int id)
+static int make_send_sock(struct netns_ipvs *ipvs, int id,
+ struct net_device *dev, struct socket **sock_ret)
{
/* multicast addr */
union ipvs_sockaddr mcast_addr;
@@ -1504,9 +1484,10 @@ static struct socket *make_send_sock(struct netns_ipvs *ipvs, int id)
IPPROTO_UDP, &sock);
if (result < 0) {
pr_err("Error during creation of socket; terminating\n");
- return ERR_PTR(result);
+ goto error;
}
- result = set_mcast_if(sock->sk, ipvs->mcfg.mcast_ifn);
+ *sock_ret = sock;
+ result = set_mcast_if(sock->sk, dev);
if (result < 0) {
pr_err("Error setting outbound mcast interface\n");
goto error;
@@ -1521,7 +1502,7 @@ static struct socket *make_send_sock(struct netns_ipvs *ipvs, int id)
set_sock_size(sock->sk, 1, result);
if (AF_INET == ipvs->mcfg.mcast_af)
- result = bind_mcastif_addr(sock, ipvs->mcfg.mcast_ifn);
+ result = bind_mcastif_addr(sock, dev);
else
result = 0;
if (result < 0) {
@@ -1537,19 +1518,18 @@ static struct socket *make_send_sock(struct netns_ipvs *ipvs, int id)
goto error;
}
- return sock;
+ return 0;
error:
- sock_release(sock);
- return ERR_PTR(result);
+ return result;
}
/*
* Set up receiving multicast socket over UDP
*/
-static struct socket *make_receive_sock(struct netns_ipvs *ipvs, int id,
- int ifindex)
+static int make_receive_sock(struct netns_ipvs *ipvs, int id,
+ struct net_device *dev, struct socket **sock_ret)
{
/* multicast addr */
union ipvs_sockaddr mcast_addr;
@@ -1561,8 +1541,9 @@ static struct socket *make_receive_sock(struct netns_ipvs *ipvs, int id,
IPPROTO_UDP, &sock);
if (result < 0) {
pr_err("Error during creation of socket; terminating\n");
- return ERR_PTR(result);
+ goto error;
}
+ *sock_ret = sock;
/* it is equivalent to the REUSEADDR option in user-space */
sock->sk->sk_reuse = SK_CAN_REUSE;
result = sysctl_sync_sock_size(ipvs);
@@ -1570,7 +1551,7 @@ static struct socket *make_receive_sock(struct netns_ipvs *ipvs, int id,
set_sock_size(sock->sk, 0, result);
get_mcast_sockaddr(&mcast_addr, &salen, &ipvs->bcfg, id);
- sock->sk->sk_bound_dev_if = ifindex;
+ sock->sk->sk_bound_dev_if = dev->ifindex;
result = sock->ops->bind(sock, (struct sockaddr *)&mcast_addr, salen);
if (result < 0) {
pr_err("Error binding to the multicast addr\n");
@@ -1581,21 +1562,20 @@ static struct socket *make_receive_sock(struct netns_ipvs *ipvs, int id,
#ifdef CONFIG_IP_VS_IPV6
if (ipvs->bcfg.mcast_af == AF_INET6)
result = join_mcast_group6(sock->sk, &mcast_addr.in6.sin6_addr,
- ipvs->bcfg.mcast_ifn);
+ dev);
else
#endif
result = join_mcast_group(sock->sk, &mcast_addr.in.sin_addr,
- ipvs->bcfg.mcast_ifn);
+ dev);
if (result < 0) {
pr_err("Error joining to the multicast group\n");
goto error;
}
- return sock;
+ return 0;
error:
- sock_release(sock);
- return ERR_PTR(result);
+ return result;
}
@@ -1780,13 +1760,12 @@ static int sync_thread_backup(void *data)
int start_sync_thread(struct netns_ipvs *ipvs, struct ipvs_sync_daemon_cfg *c,
int state)
{
- struct ip_vs_sync_thread_data *tinfo;
+ struct ip_vs_sync_thread_data *tinfo = NULL;
struct task_struct **array = NULL, *task;
- struct socket *sock;
struct net_device *dev;
char *name;
int (*threadfn)(void *data);
- int id, count, hlen;
+ int id = 0, count, hlen;
int result = -ENOMEM;
u16 mtu, min_mtu;
@@ -1794,6 +1773,18 @@ int start_sync_thread(struct netns_ipvs *ipvs, struct ipvs_sync_daemon_cfg *c,
IP_VS_DBG(7, "Each ip_vs_sync_conn entry needs %Zd bytes\n",
sizeof(struct ip_vs_sync_conn_v0));
+ /* Do not hold one mutex and then to block on another */
+ for (;;) {
+ rtnl_lock();
+ if (mutex_trylock(&ipvs->sync_mutex))
+ break;
+ rtnl_unlock();
+ mutex_lock(&ipvs->sync_mutex);
+ if (rtnl_trylock())
+ break;
+ mutex_unlock(&ipvs->sync_mutex);
+ }
+
if (!ipvs->sync_state) {
count = clamp(sysctl_sync_ports(ipvs), 1, IPVS_SYNC_PORTS_MAX);
ipvs->threads_mask = count - 1;
@@ -1812,7 +1803,8 @@ int start_sync_thread(struct netns_ipvs *ipvs, struct ipvs_sync_daemon_cfg *c,
dev = __dev_get_by_name(ipvs->net, c->mcast_ifn);
if (!dev) {
pr_err("Unknown mcast interface: %s\n", c->mcast_ifn);
- return -ENODEV;
+ result = -ENODEV;
+ goto out_early;
}
hlen = (AF_INET6 == c->mcast_af) ?
sizeof(struct ipv6hdr) + sizeof(struct udphdr) :
@@ -1829,26 +1821,30 @@ int start_sync_thread(struct netns_ipvs *ipvs, struct ipvs_sync_daemon_cfg *c,
c->sync_maxlen = mtu - hlen;
if (state == IP_VS_STATE_MASTER) {
+ result = -EEXIST;
if (ipvs->ms)
- return -EEXIST;
+ goto out_early;
ipvs->mcfg = *c;
name = "ipvs-m:%d:%d";
threadfn = sync_thread_master;
} else if (state == IP_VS_STATE_BACKUP) {
+ result = -EEXIST;
if (ipvs->backup_threads)
- return -EEXIST;
+ goto out_early;
ipvs->bcfg = *c;
name = "ipvs-b:%d:%d";
threadfn = sync_thread_backup;
} else {
- return -EINVAL;
+ result = -EINVAL;
+ goto out_early;
}
if (state == IP_VS_STATE_MASTER) {
struct ipvs_master_sync_state *ms;
+ result = -ENOMEM;
ipvs->ms = kzalloc(count * sizeof(ipvs->ms[0]), GFP_KERNEL);
if (!ipvs->ms)
goto out;
@@ -1864,39 +1860,38 @@ int start_sync_thread(struct netns_ipvs *ipvs, struct ipvs_sync_daemon_cfg *c,
} else {
array = kzalloc(count * sizeof(struct task_struct *),
GFP_KERNEL);
+ result = -ENOMEM;
if (!array)
goto out;
}
- tinfo = NULL;
for (id = 0; id < count; id++) {
- if (state == IP_VS_STATE_MASTER)
- sock = make_send_sock(ipvs, id);
- else
- sock = make_receive_sock(ipvs, id, dev->ifindex);
- if (IS_ERR(sock)) {
- result = PTR_ERR(sock);
- goto outtinfo;
- }
+ result = -ENOMEM;
tinfo = kmalloc(sizeof(*tinfo), GFP_KERNEL);
if (!tinfo)
- goto outsocket;
+ goto out;
tinfo->ipvs = ipvs;
- tinfo->sock = sock;
+ tinfo->sock = NULL;
if (state == IP_VS_STATE_BACKUP) {
tinfo->buf = kmalloc(ipvs->bcfg.sync_maxlen,
GFP_KERNEL);
if (!tinfo->buf)
- goto outtinfo;
+ goto out;
} else {
tinfo->buf = NULL;
}
tinfo->id = id;
+ if (state == IP_VS_STATE_MASTER)
+ result = make_send_sock(ipvs, id, dev, &tinfo->sock);
+ else
+ result = make_receive_sock(ipvs, id, dev, &tinfo->sock);
+ if (result < 0)
+ goto out;
task = kthread_run(threadfn, tinfo, name, ipvs->gen, id);
if (IS_ERR(task)) {
result = PTR_ERR(task);
- goto outtinfo;
+ goto out;
}
tinfo = NULL;
if (state == IP_VS_STATE_MASTER)
@@ -1913,20 +1908,20 @@ int start_sync_thread(struct netns_ipvs *ipvs, struct ipvs_sync_daemon_cfg *c,
ipvs->sync_state |= state;
spin_unlock_bh(&ipvs->sync_buff_lock);
+ mutex_unlock(&ipvs->sync_mutex);
+ rtnl_unlock();
+
/* increase the module use count */
ip_vs_use_count_inc();
return 0;
-outsocket:
- sock_release(sock);
-
-outtinfo:
- if (tinfo) {
- sock_release(tinfo->sock);
- kfree(tinfo->buf);
- kfree(tinfo);
- }
+out:
+ /* We do not need RTNL lock anymore, release it here so that
+ * sock_release below and in the kthreads can use rtnl_lock
+ * to leave the mcast group.
+ */
+ rtnl_unlock();
count = id;
while (count-- > 0) {
if (state == IP_VS_STATE_MASTER)
@@ -1934,13 +1929,23 @@ int start_sync_thread(struct netns_ipvs *ipvs, struct ipvs_sync_daemon_cfg *c,
else
kthread_stop(array[count]);
}
- kfree(array);
-
-out:
if (!(ipvs->sync_state & IP_VS_STATE_MASTER)) {
kfree(ipvs->ms);
ipvs->ms = NULL;
}
+ mutex_unlock(&ipvs->sync_mutex);
+ if (tinfo) {
+ if (tinfo->sock)
+ sock_release(tinfo->sock);
+ kfree(tinfo->buf);
+ kfree(tinfo);
+ }
+ kfree(array);
+ return result;
+
+out_early:
+ mutex_unlock(&ipvs->sync_mutex);
+ rtnl_unlock();
return result;
}
diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c
index 18e96a2..e7c1135 100644
--- a/net/netfilter/nf_conntrack_core.c
+++ b/net/netfilter/nf_conntrack_core.c
@@ -955,19 +955,22 @@ static unsigned int early_drop_list(struct net *net,
return drops;
}
-static noinline int early_drop(struct net *net, unsigned int _hash)
+static noinline int early_drop(struct net *net, unsigned int hash)
{
- unsigned int i;
+ unsigned int i, bucket;
for (i = 0; i < NF_CT_EVICTION_RANGE; i++) {
struct hlist_nulls_head *ct_hash;
- unsigned int hash, hsize, drops;
+ unsigned int hsize, drops;
rcu_read_lock();
nf_conntrack_get_ht(&ct_hash, &hsize);
- hash = reciprocal_scale(_hash++, hsize);
+ if (!i)
+ bucket = reciprocal_scale(hash, hsize);
+ else
+ bucket = (bucket + 1) % hsize;
- drops = early_drop_list(net, &ct_hash[hash]);
+ drops = early_drop_list(net, &ct_hash[bucket]);
rcu_read_unlock();
if (drops) {
@@ -1901,7 +1904,7 @@ int nf_conntrack_set_hashsize(const char *val, const struct kernel_param *kp)
return -EOPNOTSUPP;
/* On boot, we can set this without any fancy locking. */
- if (!nf_conntrack_htable_size)
+ if (!nf_conntrack_hash)
return param_set_uint(val, kp);
rc = kstrtouint(val, 0, &hashsize);
diff --git a/net/netfilter/nf_conntrack_proto_dccp.c b/net/netfilter/nf_conntrack_proto_dccp.c
index a45bee5..d5560ae 100644
--- a/net/netfilter/nf_conntrack_proto_dccp.c
+++ b/net/netfilter/nf_conntrack_proto_dccp.c
@@ -244,14 +244,14 @@ dccp_state_table[CT_DCCP_ROLE_MAX + 1][DCCP_PKT_SYNCACK + 1][CT_DCCP_MAX + 1] =
* We currently ignore Sync packets
*
* sNO, sRQ, sRS, sPO, sOP, sCR, sCG, sTW */
- sIG, sIG, sIG, sIG, sIG, sIG, sIG, sIG,
+ sIV, sIG, sIG, sIG, sIG, sIG, sIG, sIG,
},
[DCCP_PKT_SYNCACK] = {
/*
* We currently ignore SyncAck packets
*
* sNO, sRQ, sRS, sPO, sOP, sCR, sCG, sTW */
- sIG, sIG, sIG, sIG, sIG, sIG, sIG, sIG,
+ sIV, sIG, sIG, sIG, sIG, sIG, sIG, sIG,
},
},
[CT_DCCP_ROLE_SERVER] = {
@@ -372,14 +372,14 @@ dccp_state_table[CT_DCCP_ROLE_MAX + 1][DCCP_PKT_SYNCACK + 1][CT_DCCP_MAX + 1] =
* We currently ignore Sync packets
*
* sNO, sRQ, sRS, sPO, sOP, sCR, sCG, sTW */
- sIG, sIG, sIG, sIG, sIG, sIG, sIG, sIG,
+ sIV, sIG, sIG, sIG, sIG, sIG, sIG, sIG,
},
[DCCP_PKT_SYNCACK] = {
/*
* We currently ignore SyncAck packets
*
* sNO, sRQ, sRS, sPO, sOP, sCR, sCG, sTW */
- sIG, sIG, sIG, sIG, sIG, sIG, sIG, sIG,
+ sIV, sIG, sIG, sIG, sIG, sIG, sIG, sIG,
},
},
};
diff --git a/net/netfilter/nf_conntrack_seqadj.c b/net/netfilter/nf_conntrack_seqadj.c
index ef7063e..dad08b9 100644
--- a/net/netfilter/nf_conntrack_seqadj.c
+++ b/net/netfilter/nf_conntrack_seqadj.c
@@ -115,12 +115,12 @@ static void nf_ct_sack_block_adjust(struct sk_buff *skb,
/* TCP SACK sequence number adjustment */
static unsigned int nf_ct_sack_adjust(struct sk_buff *skb,
unsigned int protoff,
- struct tcphdr *tcph,
struct nf_conn *ct,
enum ip_conntrack_info ctinfo)
{
- unsigned int dir, optoff, optend;
+ struct tcphdr *tcph = (void *)skb->data + protoff;
struct nf_conn_seqadj *seqadj = nfct_seqadj(ct);
+ unsigned int dir, optoff, optend;
optoff = protoff + sizeof(struct tcphdr);
optend = protoff + tcph->doff * 4;
@@ -128,6 +128,7 @@ static unsigned int nf_ct_sack_adjust(struct sk_buff *skb,
if (!skb_make_writable(skb, optend))
return 0;
+ tcph = (void *)skb->data + protoff;
dir = CTINFO2DIR(ctinfo);
while (optoff < optend) {
@@ -207,7 +208,7 @@ int nf_ct_seq_adjust(struct sk_buff *skb,
ntohl(newack));
tcph->ack_seq = newack;
- res = nf_ct_sack_adjust(skb, protoff, tcph, ct, ctinfo);
+ res = nf_ct_sack_adjust(skb, protoff, ct, ctinfo);
out:
spin_unlock_bh(&ct->lock);
diff --git a/net/netfilter/nf_log.c b/net/netfilter/nf_log.c
index ffb9e8a..42938f9 100644
--- a/net/netfilter/nf_log.c
+++ b/net/netfilter/nf_log.c
@@ -426,6 +426,10 @@ static int nf_log_proc_dostring(struct ctl_table *table, int write,
if (write) {
struct ctl_table tmp = *table;
+ /* proc_dostring() can append to existing strings, so we need to
+ * initialize it as an empty string.
+ */
+ buf[0] = '\0';
tmp.data = buf;
r = proc_dostring(&tmp, write, buffer, lenp, ppos);
if (r)
@@ -444,14 +448,17 @@ static int nf_log_proc_dostring(struct ctl_table *table, int write,
rcu_assign_pointer(net->nf.nf_loggers[tindex], logger);
mutex_unlock(&nf_log_mutex);
} else {
+ struct ctl_table tmp = *table;
+
+ tmp.data = buf;
mutex_lock(&nf_log_mutex);
logger = nft_log_dereference(net->nf.nf_loggers[tindex]);
if (!logger)
- table->data = "NONE";
+ strlcpy(buf, "NONE", sizeof(buf));
else
- table->data = logger->name;
- r = proc_dostring(table, write, buffer, lenp, ppos);
+ strlcpy(buf, logger->name, sizeof(buf));
mutex_unlock(&nf_log_mutex);
+ r = proc_dostring(&tmp, write, buffer, lenp, ppos);
}
return r;
diff --git a/net/netfilter/nf_nat_core.c b/net/netfilter/nf_nat_core.c
index 624d6e4..51b0d83 100644
--- a/net/netfilter/nf_nat_core.c
+++ b/net/netfilter/nf_nat_core.c
@@ -421,7 +421,7 @@ nf_nat_setup_info(struct nf_conn *ct,
else
ct->status |= IPS_DST_NAT;
- if (nfct_help(ct))
+ if (nfct_help(ct) && !nfct_seqadj(ct))
if (!nfct_seqadj_ext_add(ct))
return NF_DROP;
}
diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c
index fa3ef25..a3fb30f 100644
--- a/net/netfilter/nf_tables_api.c
+++ b/net/netfilter/nf_tables_api.c
@@ -2200,41 +2200,46 @@ static int nf_tables_newrule(struct net *net, struct sock *nlsk,
}
if (nlh->nlmsg_flags & NLM_F_REPLACE) {
- if (nft_is_active_next(net, old_rule)) {
- trans = nft_trans_rule_add(&ctx, NFT_MSG_DELRULE,
- old_rule);
- if (trans == NULL) {
- err = -ENOMEM;
- goto err2;
- }
- nft_deactivate_next(net, old_rule);
- chain->use--;
- list_add_tail_rcu(&rule->list, &old_rule->list);
- } else {
+ if (!nft_is_active_next(net, old_rule)) {
err = -ENOENT;
goto err2;
}
- } else if (nlh->nlmsg_flags & NLM_F_APPEND)
- if (old_rule)
- list_add_rcu(&rule->list, &old_rule->list);
- else
- list_add_tail_rcu(&rule->list, &chain->rules);
- else {
- if (old_rule)
- list_add_tail_rcu(&rule->list, &old_rule->list);
- else
- list_add_rcu(&rule->list, &chain->rules);
- }
+ trans = nft_trans_rule_add(&ctx, NFT_MSG_DELRULE,
+ old_rule);
+ if (trans == NULL) {
+ err = -ENOMEM;
+ goto err2;
+ }
+ nft_deactivate_next(net, old_rule);
+ chain->use--;
- if (nft_trans_rule_add(&ctx, NFT_MSG_NEWRULE, rule) == NULL) {
- err = -ENOMEM;
- goto err3;
+ if (nft_trans_rule_add(&ctx, NFT_MSG_NEWRULE, rule) == NULL) {
+ err = -ENOMEM;
+ goto err2;
+ }
+
+ list_add_tail_rcu(&rule->list, &old_rule->list);
+ } else {
+ if (nft_trans_rule_add(&ctx, NFT_MSG_NEWRULE, rule) == NULL) {
+ err = -ENOMEM;
+ goto err2;
+ }
+
+ if (nlh->nlmsg_flags & NLM_F_APPEND) {
+ if (old_rule)
+ list_add_rcu(&rule->list, &old_rule->list);
+ else
+ list_add_tail_rcu(&rule->list, &chain->rules);
+ } else {
+ if (old_rule)
+ list_add_tail_rcu(&rule->list, &old_rule->list);
+ else
+ list_add_rcu(&rule->list, &chain->rules);
+ }
}
chain->use++;
return 0;
-err3:
- list_del_rcu(&rule->list);
err2:
nf_tables_rule_destroy(&ctx, rule);
err1:
@@ -2471,12 +2476,13 @@ struct nft_set *nf_tables_set_lookup_byid(const struct net *net,
u32 id = ntohl(nla_get_be32(nla));
list_for_each_entry(trans, &net->nft.commit_list, list) {
- struct nft_set *set = nft_trans_set(trans);
+ if (trans->msg_type == NFT_MSG_NEWSET) {
+ struct nft_set *set = nft_trans_set(trans);
- if (trans->msg_type == NFT_MSG_NEWSET &&
- id == nft_trans_set_id(trans) &&
- nft_active_genmask(set, genmask))
- return set;
+ if (id == nft_trans_set_id(trans) &&
+ nft_active_genmask(set, genmask))
+ return set;
+ }
}
return ERR_PTR(-ENOENT);
}
diff --git a/net/netfilter/nf_tables_core.c b/net/netfilter/nf_tables_core.c
index 0dd5c69..9d593ec 100644
--- a/net/netfilter/nf_tables_core.c
+++ b/net/netfilter/nf_tables_core.c
@@ -185,7 +185,8 @@ nft_do_chain(struct nft_pktinfo *pkt, void *priv)
switch (regs.verdict.code) {
case NFT_JUMP:
- BUG_ON(stackptr >= NFT_JUMP_STACK_SIZE);
+ if (WARN_ON_ONCE(stackptr >= NFT_JUMP_STACK_SIZE))
+ return NF_DROP;
jumpstack[stackptr].chain = chain;
jumpstack[stackptr].rule = rule;
jumpstack[stackptr].rulenum = rulenum;
diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c
index 5efb402..2a811b5 100644
--- a/net/netfilter/nfnetlink_queue.c
+++ b/net/netfilter/nfnetlink_queue.c
@@ -1210,6 +1210,9 @@ static int nfqnl_recv_unsupp(struct net *net, struct sock *ctnl,
static const struct nla_policy nfqa_cfg_policy[NFQA_CFG_MAX+1] = {
[NFQA_CFG_CMD] = { .len = sizeof(struct nfqnl_msg_config_cmd) },
[NFQA_CFG_PARAMS] = { .len = sizeof(struct nfqnl_msg_config_params) },
+ [NFQA_CFG_QUEUE_MAXLEN] = { .type = NLA_U32 },
+ [NFQA_CFG_MASK] = { .type = NLA_U32 },
+ [NFQA_CFG_FLAGS] = { .type = NLA_U32 },
};
static const struct nf_queue_handler nfqh = {
diff --git a/net/netfilter/x_tables.c b/net/netfilter/x_tables.c
index 59be898..751fec7 100644
--- a/net/netfilter/x_tables.c
+++ b/net/netfilter/x_tables.c
@@ -877,7 +877,7 @@ void *xt_copy_counters_from_user(const void __user *user, unsigned int len,
if (copy_from_user(&compat_tmp, user, sizeof(compat_tmp)) != 0)
return ERR_PTR(-EFAULT);
- strlcpy(info->name, compat_tmp.name, sizeof(info->name));
+ memcpy(info->name, compat_tmp.name, sizeof(info->name) - 1);
info->num_counters = compat_tmp.num_counters;
user += sizeof(compat_tmp);
} else
@@ -890,9 +890,9 @@ void *xt_copy_counters_from_user(const void __user *user, unsigned int len,
if (copy_from_user(info, user, sizeof(*info)) != 0)
return ERR_PTR(-EFAULT);
- info->name[sizeof(info->name) - 1] = '\0';
user += sizeof(*info);
}
+ info->name[sizeof(info->name) - 1] = '\0';
size = sizeof(struct xt_counters);
size *= info->num_counters;
diff --git a/net/netfilter/xt_IDLETIMER.c b/net/netfilter/xt_IDLETIMER.c
index 3d871c4..736cd0f 100644
--- a/net/netfilter/xt_IDLETIMER.c
+++ b/net/netfilter/xt_IDLETIMER.c
@@ -283,6 +283,22 @@ static int idletimer_resume(struct notifier_block *notifier,
return NOTIFY_DONE;
}
+static int idletimer_check_sysfs_name(const char *name, unsigned int size)
+{
+ int ret;
+
+ ret = xt_check_proc_name(name, size);
+ if (ret < 0)
+ return ret;
+
+ if (!strcmp(name, "power") ||
+ !strcmp(name, "subsystem") ||
+ !strcmp(name, "uevent"))
+ return -EINVAL;
+
+ return 0;
+}
+
static int idletimer_tg_create(struct idletimer_tg_info *info)
{
int ret;
@@ -293,6 +309,10 @@ static int idletimer_tg_create(struct idletimer_tg_info *info)
goto out;
}
+ ret = idletimer_check_sysfs_name(info->label, sizeof(info->label));
+ if (ret < 0)
+ goto out_free_timer;
+
sysfs_attr_init(&info->timer->attr.attr);
info->timer->attr.attr.name = kstrdup(info->label, GFP_KERNEL);
if (!info->timer->attr.attr.name) {
diff --git a/net/netfilter/xt_qtaguid.c b/net/netfilter/xt_qtaguid.c
index d2141a6..3dfa5a0 100644
--- a/net/netfilter/xt_qtaguid.c
+++ b/net/netfilter/xt_qtaguid.c
@@ -1197,7 +1197,7 @@ static void get_dev_and_dir(const struct sk_buff *skb,
BUG();
}
if (skb->dev && *el_dev != skb->dev) {
- MT_DEBUG("qtaguid[%d]: skb->dev=%pK %s vs par->%s=%pK %s\n",
+ MT_DEBUG("qtaguid[%d]: skb->dev=%p %s vs par->%s=%p %s\n",
par->hooknum, skb->dev, skb->dev->name,
*direction == IFS_RX ? "in" : "out", *el_dev,
(*el_dev)->name);
@@ -1307,7 +1307,7 @@ static void if_tag_stat_update(const char *ifname, uid_t uid,
}
/* It is ok to process data when an iface_entry is inactive */
- MT_DEBUG("qtaguid: tag_stat: stat_update() dev=%s entry=%pK\n",
+ MT_DEBUG("qtaguid: tag_stat: stat_update() dev=%s entry=%p\n",
ifname, iface_entry);
/*
@@ -1325,7 +1325,7 @@ static void if_tag_stat_update(const char *ifname, uid_t uid,
uid_tag = make_tag_from_uid(uid);
}
MT_DEBUG("qtaguid: tag_stat: stat_update(): "
- " looking for tag=0x%llx (uid=%u) in ife=%pK\n",
+ " looking for tag=0x%llx (uid=%u) in ife=%p\n",
tag, get_uid_from_tag(tag), iface_entry);
/* Loop over tag list under this interface for {acct_tag,uid_tag} */
spin_lock_bh(&iface_entry->tag_stat_list_lock);
@@ -1584,7 +1584,7 @@ static struct sock *qtaguid_find_sk(const struct sk_buff *skb,
struct sock *sk;
unsigned int hook_mask = (1 << par->hooknum);
- MT_DEBUG("qtaguid[%d]: find_sk(skb=%pK) family=%d\n",
+ MT_DEBUG("qtaguid[%d]: find_sk(skb=%p) family=%d\n",
par->hooknum, skb, par->family);
/*
@@ -1606,7 +1606,7 @@ static struct sock *qtaguid_find_sk(const struct sk_buff *skb,
}
if (sk) {
- MT_DEBUG("qtaguid[%d]: %pK->sk_proto=%u->sk_state=%d\n",
+ MT_DEBUG("qtaguid[%d]: %p->sk_proto=%u->sk_state=%d\n",
par->hooknum, sk, sk->sk_protocol, sk->sk_state);
}
return sk;
@@ -1763,7 +1763,7 @@ static bool qtaguid_mt(const struct sk_buff *skb, struct xt_action_param *par)
kgid_t gid_max = make_kgid(&init_user_ns, info->gid_max);
set_sk_callback_lock = true;
read_lock_bh(&sk->sk_callback_lock);
- MT_DEBUG("qtaguid[%d]: sk=%pK->sk_socket=%pK->file=%pK\n",
+ MT_DEBUG("qtaguid[%d]: sk=%p->sk_socket=%p->file=%p\n",
par->hooknum, sk, sk->sk_socket,
sk->sk_socket ? sk->sk_socket->file : (void *)-1LL);
filp = sk->sk_socket ? sk->sk_socket->file : NULL;
diff --git a/net/netlabel/netlabel_unlabeled.c b/net/netlabel/netlabel_unlabeled.c
index 4528cff..053ba86 100644
--- a/net/netlabel/netlabel_unlabeled.c
+++ b/net/netlabel/netlabel_unlabeled.c
@@ -787,7 +787,8 @@ static int netlbl_unlabel_addrinfo_get(struct genl_info *info,
{
u32 addr_len;
- if (info->attrs[NLBL_UNLABEL_A_IPV4ADDR]) {
+ if (info->attrs[NLBL_UNLABEL_A_IPV4ADDR] &&
+ info->attrs[NLBL_UNLABEL_A_IPV4MASK]) {
addr_len = nla_len(info->attrs[NLBL_UNLABEL_A_IPV4ADDR]);
if (addr_len != sizeof(struct in_addr) &&
addr_len != nla_len(info->attrs[NLBL_UNLABEL_A_IPV4MASK]))
@@ -1469,6 +1470,16 @@ int netlbl_unlabel_getattr(const struct sk_buff *skb,
iface = rcu_dereference(netlbl_unlhsh_def);
if (iface == NULL || !iface->valid)
goto unlabel_getattr_nolabel;
+
+#if IS_ENABLED(CONFIG_IPV6)
+ /* When resolving a fallback label, check the sk_buff version as
+ * it is possible (e.g. SCTP) to have family = PF_INET6 while
+ * receiving ip_hdr(skb)->version = 4.
+ */
+ if (family == PF_INET6 && ip_hdr(skb)->version == 4)
+ family = PF_INET;
+#endif /* IPv6 */
+
switch (family) {
case PF_INET: {
struct iphdr *hdr4;
diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c
index 1e97b8d..0254874 100644
--- a/net/netlink/af_netlink.c
+++ b/net/netlink/af_netlink.c
@@ -62,6 +62,7 @@
#include <asm/cacheflush.h>
#include <linux/hash.h>
#include <linux/genetlink.h>
+#include <linux/nospec.h>
#include <net/net_namespace.h>
#include <net/sock.h>
@@ -654,6 +655,7 @@ static int netlink_create(struct net *net, struct socket *sock, int protocol,
if (protocol < 0 || protocol >= MAX_LINKS)
return -EPROTONOSUPPORT;
+ protocol = array_index_nospec(protocol, MAX_LINKS);
netlink_lock_table();
#ifdef CONFIG_MODULES
@@ -984,6 +986,11 @@ static int netlink_bind(struct socket *sock, struct sockaddr *addr,
return err;
}
+ if (nlk->ngroups == 0)
+ groups = 0;
+ else if (nlk->ngroups < 8*sizeof(groups))
+ groups &= (1UL << nlk->ngroups) - 1;
+
bound = nlk->bound;
if (bound) {
/* Ensure nlk->portid is up-to-date. */
@@ -1795,6 +1802,8 @@ static int netlink_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
if (msg->msg_namelen) {
err = -EINVAL;
+ if (msg->msg_namelen < sizeof(struct sockaddr_nl))
+ goto out;
if (addr->nl_family != AF_NETLINK)
goto out;
dst_portid = addr->nl_pid;
diff --git a/net/netrom/af_netrom.c b/net/netrom/af_netrom.c
index ed212ff..046ae1c 100644
--- a/net/netrom/af_netrom.c
+++ b/net/netrom/af_netrom.c
@@ -153,7 +153,7 @@ static struct sock *nr_find_listener(ax25_address *addr)
sk_for_each(s, &nr_list)
if (!ax25cmp(&nr_sk(s)->source_addr, addr) &&
s->sk_state == TCP_LISTEN) {
- bh_lock_sock(s);
+ sock_hold(s);
goto found;
}
s = NULL;
@@ -174,7 +174,7 @@ static struct sock *nr_find_socket(unsigned char index, unsigned char id)
struct nr_sock *nr = nr_sk(s);
if (nr->my_index == index && nr->my_id == id) {
- bh_lock_sock(s);
+ sock_hold(s);
goto found;
}
}
@@ -198,7 +198,7 @@ static struct sock *nr_find_peer(unsigned char index, unsigned char id,
if (nr->your_index == index && nr->your_id == id &&
!ax25cmp(&nr->dest_addr, dest)) {
- bh_lock_sock(s);
+ sock_hold(s);
goto found;
}
}
@@ -224,7 +224,7 @@ static unsigned short nr_find_next_circuit(void)
if (i != 0 && j != 0) {
if ((sk=nr_find_socket(i, j)) == NULL)
break;
- bh_unlock_sock(sk);
+ sock_put(sk);
}
id++;
@@ -918,6 +918,7 @@ int nr_rx_frame(struct sk_buff *skb, struct net_device *dev)
}
if (sk != NULL) {
+ bh_lock_sock(sk);
skb_reset_transport_header(skb);
if (frametype == NR_CONNACK && skb->len == 22)
@@ -927,6 +928,7 @@ int nr_rx_frame(struct sk_buff *skb, struct net_device *dev)
ret = nr_process_rx_frame(sk, skb);
bh_unlock_sock(sk);
+ sock_put(sk);
return ret;
}
@@ -958,10 +960,12 @@ int nr_rx_frame(struct sk_buff *skb, struct net_device *dev)
(make = nr_make_new(sk)) == NULL) {
nr_transmit_refusal(skb, 0);
if (sk)
- bh_unlock_sock(sk);
+ sock_put(sk);
return 0;
}
+ bh_lock_sock(sk);
+
window = skb->data[20];
skb->sk = make;
@@ -1014,6 +1018,7 @@ int nr_rx_frame(struct sk_buff *skb, struct net_device *dev)
sk->sk_data_ready(sk);
bh_unlock_sock(sk);
+ sock_put(sk);
nr_insert_socket(make);
diff --git a/net/nfc/llcp_commands.c b/net/nfc/llcp_commands.c
index c5959ce..04759a0 100644
--- a/net/nfc/llcp_commands.c
+++ b/net/nfc/llcp_commands.c
@@ -149,6 +149,10 @@ struct nfc_llcp_sdp_tlv *nfc_llcp_build_sdreq_tlv(u8 tid, char *uri,
pr_debug("uri: %s, len: %zu\n", uri, uri_len);
+ /* sdreq->tlv_len is u8, takes uri_len, + 3 for header, + 1 for NULL */
+ if (WARN_ON_ONCE(uri_len > U8_MAX - 4))
+ return NULL;
+
sdreq = kzalloc(sizeof(struct nfc_llcp_sdp_tlv), GFP_KERNEL);
if (sdreq == NULL)
return NULL;
@@ -749,11 +753,14 @@ int nfc_llcp_send_ui_frame(struct nfc_llcp_sock *sock, u8 ssap, u8 dsap,
pr_debug("Fragment %zd bytes remaining %zd",
frag_len, remaining_len);
- pdu = nfc_alloc_send_skb(sock->dev, &sock->sk, MSG_DONTWAIT,
+ pdu = nfc_alloc_send_skb(sock->dev, &sock->sk, 0,
frag_len + LLCP_HEADER_SIZE, &err);
if (pdu == NULL) {
- pr_err("Could not allocate PDU\n");
- continue;
+ pr_err("Could not allocate PDU (error=%d)\n", err);
+ len -= remaining_len;
+ if (len == 0)
+ len = err;
+ break;
}
pdu = llcp_add_header(pdu, dsap, ssap, LLCP_PDU_UI);
diff --git a/net/nfc/netlink.c b/net/nfc/netlink.c
index 102c681..dbf74af 100644
--- a/net/nfc/netlink.c
+++ b/net/nfc/netlink.c
@@ -68,7 +68,8 @@ static const struct nla_policy nfc_genl_policy[NFC_ATTR_MAX + 1] = {
};
static const struct nla_policy nfc_sdp_genl_policy[NFC_SDP_ATTR_MAX + 1] = {
- [NFC_SDP_ATTR_URI] = { .type = NLA_STRING },
+ [NFC_SDP_ATTR_URI] = { .type = NLA_STRING,
+ .len = U8_MAX - 4 },
[NFC_SDP_ATTR_SAP] = { .type = NLA_U8 },
};
diff --git a/net/openvswitch/conntrack.c b/net/openvswitch/conntrack.c
index 4663939..f135814 100644
--- a/net/openvswitch/conntrack.c
+++ b/net/openvswitch/conntrack.c
@@ -906,6 +906,36 @@ static int ovs_ct_commit(struct net *net, struct sw_flow_key *key,
return 0;
}
+/* Trim the skb to the length specified by the IP/IPv6 header,
+ * removing any trailing lower-layer padding. This prepares the skb
+ * for higher-layer processing that assumes skb->len excludes padding
+ * (such as nf_ip_checksum). The caller needs to pull the skb to the
+ * network header, and ensure ip_hdr/ipv6_hdr points to valid data.
+ */
+static int ovs_skb_network_trim(struct sk_buff *skb)
+{
+ unsigned int len;
+ int err;
+
+ switch (skb->protocol) {
+ case htons(ETH_P_IP):
+ len = ntohs(ip_hdr(skb)->tot_len);
+ break;
+ case htons(ETH_P_IPV6):
+ len = sizeof(struct ipv6hdr)
+ + ntohs(ipv6_hdr(skb)->payload_len);
+ break;
+ default:
+ len = skb->len;
+ }
+
+ err = pskb_trim_rcsum(skb, len);
+ if (err)
+ kfree_skb(skb);
+
+ return err;
+}
+
/* Returns 0 on success, -EINPROGRESS if 'skb' is stolen, or other nonzero
* value if 'skb' is freed.
*/
@@ -920,6 +950,10 @@ int ovs_ct_execute(struct net *net, struct sk_buff *skb,
nh_ofs = skb_network_offset(skb);
skb_pull_rcsum(skb, nh_ofs);
+ err = ovs_skb_network_trim(skb);
+ if (err)
+ return err;
+
if (key->ip.frag != OVS_FRAG_TYPE_NONE) {
err = handle_fragments(net, key, info->zone.id, skb);
if (err)
diff --git a/net/openvswitch/flow_netlink.c b/net/openvswitch/flow_netlink.c
index 1668916..326945d 100644
--- a/net/openvswitch/flow_netlink.c
+++ b/net/openvswitch/flow_netlink.c
@@ -1296,13 +1296,10 @@ static void nlattr_set(struct nlattr *attr, u8 val,
/* The nlattr stream should already have been validated */
nla_for_each_nested(nla, attr, rem) {
- if (tbl[nla_type(nla)].len == OVS_ATTR_NESTED) {
- if (tbl[nla_type(nla)].next)
- tbl = tbl[nla_type(nla)].next;
- nlattr_set(nla, val, tbl);
- } else {
+ if (tbl[nla_type(nla)].len == OVS_ATTR_NESTED)
+ nlattr_set(nla, val, tbl[nla_type(nla)].next ? : tbl);
+ else
memset(nla_data(nla), val, nla_len(nla));
- }
if (nla_type(nla) == OVS_KEY_ATTR_CT_STATE)
*(u32 *)nla_data(nla) &= CT_SUPPORTED_MASK;
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 1d7ff6f..a9d0358 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -333,11 +333,11 @@ static void packet_pick_tx_queue(struct net_device *dev, struct sk_buff *skb)
skb_set_queue_mapping(skb, queue_index);
}
-/* register_prot_hook must be invoked with the po->bind_lock held,
+/* __register_prot_hook must be invoked through register_prot_hook
* or from a context in which asynchronous accesses to the packet
* socket is not possible (packet_create()).
*/
-static void register_prot_hook(struct sock *sk)
+static void __register_prot_hook(struct sock *sk)
{
struct packet_sock *po = pkt_sk(sk);
@@ -352,8 +352,13 @@ static void register_prot_hook(struct sock *sk)
}
}
-/* {,__}unregister_prot_hook() must be invoked with the po->bind_lock
- * held. If the sync parameter is true, we will temporarily drop
+static void register_prot_hook(struct sock *sk)
+{
+ lockdep_assert_held_once(&pkt_sk(sk)->bind_lock);
+ __register_prot_hook(sk);
+}
+
+/* If the sync parameter is true, we will temporarily drop
* the po->bind_lock and do a synchronize_net to make sure no
* asynchronous packet processing paths still refer to the elements
* of po->prot_hook. If the sync parameter is false, it is the
@@ -363,6 +368,8 @@ static void __unregister_prot_hook(struct sock *sk, bool sync)
{
struct packet_sock *po = pkt_sk(sk);
+ lockdep_assert_held_once(&po->bind_lock);
+
po->running = 0;
if (po->fanout)
@@ -2258,6 +2265,12 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
if (po->stats.stats1.tp_drops)
status |= TP_STATUS_LOSING;
}
+
+ if (do_vnet &&
+ __packet_rcv_vnet(skb, h.raw + macoff -
+ sizeof(struct virtio_net_hdr)))
+ goto drop_n_account;
+
po->stats.stats1.tp_packets++;
if (copy_skb) {
status |= TP_STATUS_COPY;
@@ -2265,14 +2278,6 @@ static int tpacket_rcv(struct sk_buff *skb, struct net_device *dev,
}
spin_unlock(&sk->sk_receive_queue.lock);
- if (do_vnet) {
- if (__packet_rcv_vnet(skb, h.raw + macoff -
- sizeof(struct virtio_net_hdr))) {
- spin_lock(&sk->sk_receive_queue.lock);
- goto drop_n_account;
- }
- }
-
skb_copy_bits(skb, 0, h.raw + macoff, snaplen);
if (!(ts_status = tpacket_get_timestamp(skb, &ts, po->tp_tstamp)))
@@ -2655,8 +2660,10 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg)
sll_addr)))
goto out;
proto = saddr->sll_protocol;
- addr = saddr->sll_addr;
+ addr = saddr->sll_halen ? saddr->sll_addr : NULL;
dev = dev_get_by_index(sock_net(&po->sk), saddr->sll_ifindex);
+ if (addr && dev && saddr->sll_halen < dev->addr_len)
+ goto out;
}
err = -ENXIO;
@@ -2852,8 +2859,10 @@ static int packet_snd(struct socket *sock, struct msghdr *msg, size_t len)
if (msg->msg_namelen < (saddr->sll_halen + offsetof(struct sockaddr_ll, sll_addr)))
goto out;
proto = saddr->sll_protocol;
- addr = saddr->sll_addr;
+ addr = saddr->sll_halen ? saddr->sll_addr : NULL;
dev = dev_get_by_index(sock_net(sk), saddr->sll_ifindex);
+ if (addr && dev && saddr->sll_halen < dev->addr_len)
+ goto out;
}
err = -ENXIO;
@@ -2912,6 +2921,8 @@ static int packet_snd(struct socket *sock, struct msghdr *msg, size_t len)
goto out_free;
} else if (reserve) {
skb_reserve(skb, -reserve);
+ if (len < reserve)
+ skb_reset_network_header(skb);
}
/* Returns -EFAULT on error */
@@ -3019,6 +3030,7 @@ static int packet_release(struct socket *sock)
packet_flush_mclist(sk);
+ lock_sock(sk);
if (po->rx_ring.pg_vec) {
memset(&req_u, 0, sizeof(req_u));
packet_set_ring(sk, &req_u, 1, 0);
@@ -3028,6 +3040,7 @@ static int packet_release(struct socket *sock)
memset(&req_u, 0, sizeof(req_u));
packet_set_ring(sk, &req_u, 1, 1);
}
+ release_sock(sk);
f = fanout_release(sk);
@@ -3261,7 +3274,7 @@ static int packet_create(struct net *net, struct socket *sock, int protocol,
if (proto) {
po->prot_hook.type = proto;
- register_prot_hook(sk);
+ __register_prot_hook(sk);
}
mutex_lock(&net->packet.sklist_lock);
@@ -3656,6 +3669,7 @@ packet_setsockopt(struct socket *sock, int level, int optname, char __user *optv
union tpacket_req_u req_u;
int len;
+ lock_sock(sk);
switch (po->tp_version) {
case TPACKET_V1:
case TPACKET_V2:
@@ -3666,12 +3680,17 @@ packet_setsockopt(struct socket *sock, int level, int optname, char __user *optv
len = sizeof(req_u.req3);
break;
}
- if (optlen < len)
- return -EINVAL;
- if (copy_from_user(&req_u.req, optval, len))
- return -EFAULT;
- return packet_set_ring(sk, &req_u, 0,
- optname == PACKET_TX_RING);
+ if (optlen < len) {
+ ret = -EINVAL;
+ } else {
+ if (copy_from_user(&req_u.req, optval, len))
+ ret = -EFAULT;
+ else
+ ret = packet_set_ring(sk, &req_u, 0,
+ optname == PACKET_TX_RING);
+ }
+ release_sock(sk);
+ return ret;
}
case PACKET_COPY_THRESH:
{
@@ -3737,12 +3756,18 @@ packet_setsockopt(struct socket *sock, int level, int optname, char __user *optv
if (optlen != sizeof(val))
return -EINVAL;
- if (po->rx_ring.pg_vec || po->tx_ring.pg_vec)
- return -EBUSY;
if (copy_from_user(&val, optval, sizeof(val)))
return -EFAULT;
- po->tp_loss = !!val;
- return 0;
+
+ lock_sock(sk);
+ if (po->rx_ring.pg_vec || po->tx_ring.pg_vec) {
+ ret = -EBUSY;
+ } else {
+ po->tp_loss = !!val;
+ ret = 0;
+ }
+ release_sock(sk);
+ return ret;
}
case PACKET_AUXDATA:
{
@@ -3753,7 +3778,9 @@ packet_setsockopt(struct socket *sock, int level, int optname, char __user *optv
if (copy_from_user(&val, optval, sizeof(val)))
return -EFAULT;
+ lock_sock(sk);
po->auxdata = !!val;
+ release_sock(sk);
return 0;
}
case PACKET_ORIGDEV:
@@ -3765,7 +3792,9 @@ packet_setsockopt(struct socket *sock, int level, int optname, char __user *optv
if (copy_from_user(&val, optval, sizeof(val)))
return -EFAULT;
+ lock_sock(sk);
po->origdev = !!val;
+ release_sock(sk);
return 0;
}
case PACKET_VNET_HDR:
@@ -3774,15 +3803,20 @@ packet_setsockopt(struct socket *sock, int level, int optname, char __user *optv
if (sock->type != SOCK_RAW)
return -EINVAL;
- if (po->rx_ring.pg_vec || po->tx_ring.pg_vec)
- return -EBUSY;
if (optlen < sizeof(val))
return -EINVAL;
if (copy_from_user(&val, optval, sizeof(val)))
return -EFAULT;
- po->has_vnet_hdr = !!val;
- return 0;
+ lock_sock(sk);
+ if (po->rx_ring.pg_vec || po->tx_ring.pg_vec) {
+ ret = -EBUSY;
+ } else {
+ po->has_vnet_hdr = !!val;
+ ret = 0;
+ }
+ release_sock(sk);
+ return ret;
}
case PACKET_TIMESTAMP:
{
@@ -3820,11 +3854,17 @@ packet_setsockopt(struct socket *sock, int level, int optname, char __user *optv
if (optlen != sizeof(val))
return -EINVAL;
- if (po->rx_ring.pg_vec || po->tx_ring.pg_vec)
- return -EBUSY;
if (copy_from_user(&val, optval, sizeof(val)))
return -EFAULT;
- po->tp_tx_has_off = !!val;
+
+ lock_sock(sk);
+ if (po->rx_ring.pg_vec || po->tx_ring.pg_vec) {
+ ret = -EBUSY;
+ } else {
+ po->tp_tx_has_off = !!val;
+ ret = 0;
+ }
+ release_sock(sk);
return 0;
}
case PACKET_QDISC_BYPASS:
@@ -4221,7 +4261,6 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
/* Added to avoid minimal code churn */
struct tpacket_req *req = &req_u->req;
- lock_sock(sk);
/* Opening a Tx-ring is NOT supported in TPACKET_V3 */
if (!closing && tx_ring && (po->tp_version > TPACKET_V2)) {
net_warn_ratelimited("Tx-ring is not supported.\n");
@@ -4240,6 +4279,8 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
}
if (req->tp_block_nr) {
+ unsigned int min_frame_size;
+
/* Sanity tests and some calculations */
err = -EBUSY;
if (unlikely(rb->pg_vec))
@@ -4262,12 +4303,12 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
goto out;
if (unlikely(!PAGE_ALIGNED(req->tp_block_size)))
goto out;
+ min_frame_size = po->tp_hdrlen + po->tp_reserve;
if (po->tp_version >= TPACKET_V3 &&
- req->tp_block_size <=
- BLK_PLUS_PRIV((u64)req_u->req3.tp_sizeof_priv))
+ req->tp_block_size <
+ BLK_PLUS_PRIV((u64)req_u->req3.tp_sizeof_priv) + min_frame_size)
goto out;
- if (unlikely(req->tp_frame_size < po->tp_hdrlen +
- po->tp_reserve))
+ if (unlikely(req->tp_frame_size < min_frame_size))
goto out;
if (unlikely(req->tp_frame_size & (TPACKET_ALIGNMENT - 1)))
goto out;
@@ -4357,7 +4398,6 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
if (pg_vec)
free_pg_vec(pg_vec, order, req->tp_block_nr);
out:
- release_sock(sk);
return err;
}
diff --git a/net/packet/internal.h b/net/packet/internal.h
index d55bfc3..1309e2a 100644
--- a/net/packet/internal.h
+++ b/net/packet/internal.h
@@ -109,10 +109,12 @@ struct packet_sock {
int copy_thresh;
spinlock_t bind_lock;
struct mutex pg_vec_lock;
- unsigned int running:1, /* prot_hook is attached*/
- auxdata:1,
+ unsigned int running; /* bind_lock must be held */
+ unsigned int auxdata:1, /* writer must hold sock lock */
origdev:1,
- has_vnet_hdr:1;
+ has_vnet_hdr:1,
+ tp_loss:1,
+ tp_tx_has_off:1;
int pressure;
int ifindex; /* bound device */
__be16 num;
@@ -122,8 +124,6 @@ struct packet_sock {
enum tpacket_versions tp_version;
unsigned int tp_hdrlen;
unsigned int tp_reserve;
- unsigned int tp_loss:1;
- unsigned int tp_tx_has_off:1;
unsigned int tp_tstamp;
struct net_device __rcu *cached_dev;
int (*xmit)(struct sk_buff *skb);
diff --git a/net/qrtr/qrtr.c b/net/qrtr/qrtr.c
index ae5ac17..7b670a9 100644
--- a/net/qrtr/qrtr.c
+++ b/net/qrtr/qrtr.c
@@ -621,6 +621,10 @@ static int qrtr_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
node = NULL;
if (addr->sq_node == QRTR_NODE_BCAST) {
enqueue_fn = qrtr_bcast_enqueue;
+ if (addr->sq_port != QRTR_PORT_CTRL) {
+ release_sock(sk);
+ return -ENOTCONN;
+ }
} else if (addr->sq_node == ipc->us.sq_node) {
enqueue_fn = qrtr_local_enqueue;
} else {
diff --git a/net/qrtr/smd.c b/net/qrtr/smd.c
index 0d11132..ff0112b 100644
--- a/net/qrtr/smd.c
+++ b/net/qrtr/smd.c
@@ -116,5 +116,6 @@ static struct qcom_smd_driver qcom_smd_qrtr_driver = {
module_qcom_smd_driver(qcom_smd_qrtr_driver);
+MODULE_ALIAS("rpmsg:IPCRTR");
MODULE_DESCRIPTION("Qualcomm IPC-Router SMD interface driver");
MODULE_LICENSE("GPL v2");
diff --git a/net/rds/bind.c b/net/rds/bind.c
index adb53ae..cc7e3a1 100644
--- a/net/rds/bind.c
+++ b/net/rds/bind.c
@@ -60,11 +60,13 @@ struct rds_sock *rds_find_bound(__be32 addr, __be16 port)
u64 key = ((u64)addr << 32) | port;
struct rds_sock *rs;
- rs = rhashtable_lookup_fast(&bind_hash_table, &key, ht_parms);
+ rcu_read_lock();
+ rs = rhashtable_lookup(&bind_hash_table, &key, ht_parms);
if (rs && !sock_flag(rds_rs_to_sk(rs), SOCK_DEAD))
rds_sock_addref(rs);
else
rs = NULL;
+ rcu_read_unlock();
rdsdebug("returning rs %p for %pI4:%u\n", rs, &addr,
ntohs(port));
@@ -157,6 +159,7 @@ int rds_bind(struct socket *sock, struct sockaddr *uaddr, int addr_len)
goto out;
}
+ sock_set_flag(sk, SOCK_RCU_FREE);
ret = rds_add_bound(rs, sin->sin_addr.s_addr, &sin->sin_port);
if (ret)
goto out;
diff --git a/net/rds/ib.c b/net/rds/ib.c
index 5680d90..0efb3d2 100644
--- a/net/rds/ib.c
+++ b/net/rds/ib.c
@@ -336,7 +336,8 @@ static int rds_ib_laddr_check(struct net *net, __be32 addr)
/* Create a CMA ID and try to bind it. This catches both
* IB and iWARP capable NICs.
*/
- cm_id = rdma_create_id(&init_net, NULL, NULL, RDMA_PS_TCP, IB_QPT_RC);
+ cm_id = rdma_create_id(&init_net, rds_rdma_cm_event_handler,
+ NULL, RDMA_PS_TCP, IB_QPT_RC);
if (IS_ERR(cm_id))
return PTR_ERR(cm_id);
diff --git a/net/rds/ib_cm.c b/net/rds/ib_cm.c
index 169156c..96e61ea 100644
--- a/net/rds/ib_cm.c
+++ b/net/rds/ib_cm.c
@@ -505,7 +505,7 @@ static int rds_ib_setup_qp(struct rds_connection *conn)
rdsdebug("conn %p pd %p cq %p %p\n", conn, ic->i_pd,
ic->i_send_cq, ic->i_recv_cq);
- return ret;
+ goto out;
sends_out:
vfree(ic->i_sends);
@@ -530,6 +530,7 @@ static int rds_ib_setup_qp(struct rds_connection *conn)
ic->i_send_cq = NULL;
rds_ibdev_out:
rds_ib_remove_conn(rds_ibdev, conn);
+out:
rds_ib_dev_put(rds_ibdev);
return ret;
diff --git a/net/rds/ib_frmr.c b/net/rds/ib_frmr.c
index 66b3d62..3d9c4c6 100644
--- a/net/rds/ib_frmr.c
+++ b/net/rds/ib_frmr.c
@@ -61,6 +61,7 @@ static struct rds_ib_mr *rds_ib_alloc_frmr(struct rds_ib_device *rds_ibdev,
pool->fmr_attr.max_pages);
if (IS_ERR(frmr->mr)) {
pr_warn("RDS/IB: %s failed to allocate MR", __func__);
+ err = PTR_ERR(frmr->mr);
goto out_no_cigar;
}
diff --git a/net/rds/loop.c b/net/rds/loop.c
index f2bf78d..dac6218 100644
--- a/net/rds/loop.c
+++ b/net/rds/loop.c
@@ -193,4 +193,5 @@ struct rds_transport rds_loop_transport = {
.inc_copy_to_user = rds_message_inc_copy_to_user,
.inc_free = rds_loop_inc_free,
.t_name = "loopback",
+ .t_type = RDS_TRANS_LOOP,
};
diff --git a/net/rds/rds.h b/net/rds/rds.h
index 30a51fe..edfc339 100644
--- a/net/rds/rds.h
+++ b/net/rds/rds.h
@@ -440,6 +440,11 @@ struct rds_notifier {
int n_status;
};
+/* Available as part of RDS core, so doesn't need to participate
+ * in get_preferred transport etc
+ */
+#define RDS_TRANS_LOOP 3
+
/**
* struct rds_transport - transport specific behavioural hooks
*
diff --git a/net/rds/recv.c b/net/rds/recv.c
index cbfabdf..f16ee1b 100644
--- a/net/rds/recv.c
+++ b/net/rds/recv.c
@@ -94,6 +94,11 @@ static void rds_recv_rcvbuf_delta(struct rds_sock *rs, struct sock *sk,
return;
rs->rs_rcv_bytes += delta;
+
+ /* loop transport doesn't send/recv congestion updates */
+ if (rs->rs_transport->t_type == RDS_TRANS_LOOP)
+ return;
+
now_congested = rs->rs_rcv_bytes > rds_sk_rcvbuf(rs);
rdsdebug("rs %p (%pI4:%u) recv bytes %d buf %d "
diff --git a/net/rfkill/rfkill-gpio.c b/net/rfkill/rfkill-gpio.c
index 76c01cb..d6d8b34 100644
--- a/net/rfkill/rfkill-gpio.c
+++ b/net/rfkill/rfkill-gpio.c
@@ -138,13 +138,18 @@ static int rfkill_gpio_probe(struct platform_device *pdev)
ret = rfkill_register(rfkill->rfkill_dev);
if (ret < 0)
- return ret;
+ goto err_destroy;
platform_set_drvdata(pdev, rfkill);
dev_info(&pdev->dev, "%s device registered.\n", rfkill->name);
return 0;
+
+err_destroy:
+ rfkill_destroy(rfkill->rfkill_dev);
+
+ return ret;
}
static int rfkill_gpio_remove(struct platform_device *pdev)
diff --git a/net/rxrpc/input.c b/net/rxrpc/input.c
index 1060d14..a4380e1 100644
--- a/net/rxrpc/input.c
+++ b/net/rxrpc/input.c
@@ -216,10 +216,11 @@ static void rxrpc_send_ping(struct rxrpc_call *call, struct sk_buff *skb,
/*
* Apply a hard ACK by advancing the Tx window.
*/
-static void rxrpc_rotate_tx_window(struct rxrpc_call *call, rxrpc_seq_t to,
+static bool rxrpc_rotate_tx_window(struct rxrpc_call *call, rxrpc_seq_t to,
struct rxrpc_ack_summary *summary)
{
struct sk_buff *skb, *list = NULL;
+ bool rot_last = false;
int ix;
u8 annotation;
@@ -243,15 +244,17 @@ static void rxrpc_rotate_tx_window(struct rxrpc_call *call, rxrpc_seq_t to,
skb->next = list;
list = skb;
- if (annotation & RXRPC_TX_ANNO_LAST)
+ if (annotation & RXRPC_TX_ANNO_LAST) {
set_bit(RXRPC_CALL_TX_LAST, &call->flags);
+ rot_last = true;
+ }
if ((annotation & RXRPC_TX_ANNO_MASK) != RXRPC_TX_ANNO_ACK)
summary->nr_rot_new_acks++;
}
spin_unlock(&call->lock);
- trace_rxrpc_transmit(call, (test_bit(RXRPC_CALL_TX_LAST, &call->flags) ?
+ trace_rxrpc_transmit(call, (rot_last ?
rxrpc_transmit_rotate_last :
rxrpc_transmit_rotate));
wake_up(&call->waitq);
@@ -262,6 +265,8 @@ static void rxrpc_rotate_tx_window(struct rxrpc_call *call, rxrpc_seq_t to,
skb->next = NULL;
rxrpc_free_skb(skb, rxrpc_skb_tx_freed);
}
+
+ return rot_last;
}
/*
@@ -332,11 +337,11 @@ static bool rxrpc_receiving_reply(struct rxrpc_call *call)
ktime_get_real());
}
- if (!test_bit(RXRPC_CALL_TX_LAST, &call->flags))
- rxrpc_rotate_tx_window(call, top, &summary);
if (!test_bit(RXRPC_CALL_TX_LAST, &call->flags)) {
- rxrpc_proto_abort("TXL", call, top);
- return false;
+ if (!rxrpc_rotate_tx_window(call, top, &summary)) {
+ rxrpc_proto_abort("TXL", call, top);
+ return false;
+ }
}
if (!rxrpc_end_tx_phase(call, true, "ETD"))
return false;
@@ -803,6 +808,16 @@ static void rxrpc_input_ack(struct rxrpc_call *call, struct sk_buff *skb,
rxrpc_propose_ack_respond_to_ack);
}
+ /* Discard any out-of-order or duplicate ACKs. */
+ if (before_eq(sp->hdr.serial, call->acks_latest)) {
+ _debug("discard ACK %d <= %d",
+ sp->hdr.serial, call->acks_latest);
+ return;
+ }
+ call->acks_latest_ts = skb->tstamp;
+ call->acks_latest = sp->hdr.serial;
+
+ /* Parse rwind and mtu sizes if provided. */
ioffset = offset + nr_acks + 3;
if (skb->len >= ioffset + sizeof(buf.info)) {
if (skb_copy_bits(skb, ioffset, &buf.info, sizeof(buf.info)) < 0)
@@ -824,23 +839,18 @@ static void rxrpc_input_ack(struct rxrpc_call *call, struct sk_buff *skb,
return;
}
- /* Discard any out-of-order or duplicate ACKs. */
- if (before_eq(sp->hdr.serial, call->acks_latest)) {
- _debug("discard ACK %d <= %d",
- sp->hdr.serial, call->acks_latest);
- return;
- }
- call->acks_latest_ts = skb->tstamp;
- call->acks_latest = sp->hdr.serial;
-
if (before(hard_ack, call->tx_hard_ack) ||
after(hard_ack, call->tx_top))
return rxrpc_proto_abort("AKW", call, 0);
if (nr_acks > call->tx_top - hard_ack)
return rxrpc_proto_abort("AKN", call, 0);
- if (after(hard_ack, call->tx_hard_ack))
- rxrpc_rotate_tx_window(call, hard_ack, &summary);
+ if (after(hard_ack, call->tx_hard_ack)) {
+ if (rxrpc_rotate_tx_window(call, hard_ack, &summary)) {
+ rxrpc_end_tx_phase(call, false, "ETA");
+ return;
+ }
+ }
if (nr_acks > 0) {
if (skb_copy_bits(skb, offset, buf.acks, nr_acks) < 0)
@@ -849,11 +859,6 @@ static void rxrpc_input_ack(struct rxrpc_call *call, struct sk_buff *skb,
&summary);
}
- if (test_bit(RXRPC_CALL_TX_LAST, &call->flags)) {
- rxrpc_end_tx_phase(call, false, "ETA");
- return;
- }
-
if (call->rxtx_annotations[call->tx_top & RXRPC_RXTX_BUFF_MASK] &
RXRPC_TX_ANNO_LAST &&
summary.nr_acks == call->tx_top - hard_ack &&
@@ -875,8 +880,7 @@ static void rxrpc_input_ackall(struct rxrpc_call *call, struct sk_buff *skb)
_proto("Rx ACKALL %%%u", sp->hdr.serial);
- rxrpc_rotate_tx_window(call, call->tx_top, &summary);
- if (test_bit(RXRPC_CALL_TX_LAST, &call->flags))
+ if (rxrpc_rotate_tx_window(call, call->tx_top, &summary))
rxrpc_end_tx_phase(call, false, "ETL");
}
@@ -1166,16 +1170,19 @@ void rxrpc_data_ready(struct sock *udp_sk)
goto discard_unlock;
if (sp->hdr.callNumber == chan->last_call) {
- /* For the previous service call, if completed successfully, we
- * discard all further packets.
- */
- if (rxrpc_conn_is_service(conn) &&
- (chan->last_type == RXRPC_PACKET_TYPE_ACK ||
- sp->hdr.type == RXRPC_PACKET_TYPE_ABORT))
+ if (chan->call ||
+ sp->hdr.type == RXRPC_PACKET_TYPE_ABORT)
goto discard_unlock;
- /* But otherwise we need to retransmit the final packet from
- * data cached in the connection record.
+ /* For the previous service call, if completed
+ * successfully, we discard all further packets.
+ */
+ if (rxrpc_conn_is_service(conn) &&
+ chan->last_type == RXRPC_PACKET_TYPE_ACK)
+ goto discard_unlock;
+
+ /* But otherwise we need to retransmit the final packet
+ * from data cached in the connection record.
*/
rxrpc_post_packet_to_conn(conn, skb);
goto out_unlock;
diff --git a/net/rxrpc/recvmsg.c b/net/rxrpc/recvmsg.c
index c29362d..3e52b7fd 100644
--- a/net/rxrpc/recvmsg.c
+++ b/net/rxrpc/recvmsg.c
@@ -493,9 +493,10 @@ int rxrpc_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
ret = put_cmsg(msg, SOL_RXRPC, RXRPC_USER_CALL_ID,
sizeof(unsigned int), &id32);
} else {
+ unsigned long idl = call->user_call_ID;
+
ret = put_cmsg(msg, SOL_RXRPC, RXRPC_USER_CALL_ID,
- sizeof(unsigned long),
- &call->user_call_ID);
+ sizeof(unsigned long), &idl);
}
if (ret < 0)
goto error;
diff --git a/net/rxrpc/sendmsg.c b/net/rxrpc/sendmsg.c
index b214a4d..1de27c3 100644
--- a/net/rxrpc/sendmsg.c
+++ b/net/rxrpc/sendmsg.c
@@ -78,7 +78,9 @@ static inline void rxrpc_instant_resend(struct rxrpc_call *call, int ix)
spin_lock_bh(&call->lock);
if (call->state < RXRPC_CALL_COMPLETE) {
- call->rxtx_annotations[ix] = RXRPC_TX_ANNO_RETRANS;
+ call->rxtx_annotations[ix] =
+ (call->rxtx_annotations[ix] & RXRPC_TX_ANNO_LAST) |
+ RXRPC_TX_ANNO_RETRANS;
if (!test_and_set_bit(RXRPC_CALL_EV_RESEND, &call->events))
rxrpc_queue_call(call);
}
diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
index 95c463c..d2932dc 100644
--- a/net/sched/act_ife.c
+++ b/net/sched/act_ife.c
@@ -267,10 +267,8 @@ static int ife_validate_metatype(struct tcf_meta_ops *ops, void *val, int len)
}
/* called when adding new meta information
- * under ife->tcf_lock for existing action
*/
-static int load_metaops_and_vet(struct tcf_ife_info *ife, u32 metaid,
- void *val, int len, bool exists)
+static int load_metaops_and_vet(u32 metaid, void *val, int len)
{
struct tcf_meta_ops *ops = find_ife_oplist(metaid);
int ret = 0;
@@ -278,13 +276,9 @@ static int load_metaops_and_vet(struct tcf_ife_info *ife, u32 metaid,
if (!ops) {
ret = -ENOENT;
#ifdef CONFIG_MODULES
- if (exists)
- spin_unlock_bh(&ife->tcf_lock);
rtnl_unlock();
request_module("ifemeta%u", metaid);
rtnl_lock();
- if (exists)
- spin_lock_bh(&ife->tcf_lock);
ops = find_ife_oplist(metaid);
#endif
}
@@ -301,24 +295,17 @@ static int load_metaops_and_vet(struct tcf_ife_info *ife, u32 metaid,
}
/* called when adding new meta information
- * under ife->tcf_lock for existing action
*/
-static int add_metainfo(struct tcf_ife_info *ife, u32 metaid, void *metaval,
- int len, bool atomic)
+static int __add_metainfo(const struct tcf_meta_ops *ops,
+ struct tcf_ife_info *ife, u32 metaid, void *metaval,
+ int len, bool atomic, bool exists)
{
struct tcf_meta_info *mi = NULL;
- struct tcf_meta_ops *ops = find_ife_oplist(metaid);
int ret = 0;
- if (!ops)
- return -ENOENT;
-
mi = kzalloc(sizeof(*mi), atomic ? GFP_ATOMIC : GFP_KERNEL);
- if (!mi) {
- /*put back what find_ife_oplist took */
- module_put(ops->owner);
+ if (!mi)
return -ENOMEM;
- }
mi->metaid = metaid;
mi->ops = ops;
@@ -326,17 +313,49 @@ static int add_metainfo(struct tcf_ife_info *ife, u32 metaid, void *metaval,
ret = ops->alloc(mi, metaval, atomic ? GFP_ATOMIC : GFP_KERNEL);
if (ret != 0) {
kfree(mi);
- module_put(ops->owner);
return ret;
}
}
+ if (exists)
+ spin_lock_bh(&ife->tcf_lock);
list_add_tail(&mi->metalist, &ife->metalist);
+ if (exists)
+ spin_unlock_bh(&ife->tcf_lock);
return ret;
}
-static int use_all_metadata(struct tcf_ife_info *ife)
+static int add_metainfo_and_get_ops(const struct tcf_meta_ops *ops,
+ struct tcf_ife_info *ife, u32 metaid,
+ bool exists)
+{
+ int ret;
+
+ if (!try_module_get(ops->owner))
+ return -ENOENT;
+ ret = __add_metainfo(ops, ife, metaid, NULL, 0, true, exists);
+ if (ret)
+ module_put(ops->owner);
+ return ret;
+}
+
+static int add_metainfo(struct tcf_ife_info *ife, u32 metaid, void *metaval,
+ int len, bool exists)
+{
+ const struct tcf_meta_ops *ops = find_ife_oplist(metaid);
+ int ret;
+
+ if (!ops)
+ return -ENOENT;
+ ret = __add_metainfo(ops, ife, metaid, metaval, len, false, exists);
+ if (ret)
+ /*put back what find_ife_oplist took */
+ module_put(ops->owner);
+ return ret;
+}
+
+static int use_all_metadata(struct tcf_ife_info *ife, bool exists)
{
struct tcf_meta_ops *o;
int rc = 0;
@@ -344,7 +363,7 @@ static int use_all_metadata(struct tcf_ife_info *ife)
read_lock(&ife_mod_lock);
list_for_each_entry(o, &ifeoplist, list) {
- rc = add_metainfo(ife, o->metaid, NULL, 0, true);
+ rc = add_metainfo_and_get_ops(o, ife, o->metaid, exists);
if (rc == 0)
installed += 1;
}
@@ -395,7 +414,6 @@ static void _tcf_ife_cleanup(struct tc_action *a, int bind)
struct tcf_meta_info *e, *n;
list_for_each_entry_safe(e, n, &ife->metalist, metalist) {
- module_put(e->ops->owner);
list_del(&e->metalist);
if (e->metaval) {
if (e->ops->release)
@@ -403,6 +421,7 @@ static void _tcf_ife_cleanup(struct tc_action *a, int bind)
else
kfree(e->metaval);
}
+ module_put(e->ops->owner);
kfree(e);
}
}
@@ -416,7 +435,6 @@ static void tcf_ife_cleanup(struct tc_action *a, int bind)
spin_unlock_bh(&ife->tcf_lock);
}
-/* under ife->tcf_lock for existing action */
static int populate_metalist(struct tcf_ife_info *ife, struct nlattr **tb,
bool exists)
{
@@ -430,7 +448,7 @@ static int populate_metalist(struct tcf_ife_info *ife, struct nlattr **tb,
val = nla_data(tb[i]);
len = nla_len(tb[i]);
- rc = load_metaops_and_vet(ife, i, val, len, exists);
+ rc = load_metaops_and_vet(i, val, len);
if (rc != 0)
return rc;
@@ -510,6 +528,8 @@ static int tcf_ife_init(struct net *net, struct nlattr *nla,
if (exists)
spin_lock_bh(&ife->tcf_lock);
ife->tcf_action = parm->action;
+ if (exists)
+ spin_unlock_bh(&ife->tcf_lock);
if (parm->flags & IFE_ENCODE) {
if (daddr)
@@ -537,9 +557,6 @@ static int tcf_ife_init(struct net *net, struct nlattr *nla,
tcf_hash_release(*a, bind);
if (ret == ACT_P_CREATED)
_tcf_ife_cleanup(*a, bind);
-
- if (exists)
- spin_unlock_bh(&ife->tcf_lock);
return err;
}
@@ -553,20 +570,14 @@ static int tcf_ife_init(struct net *net, struct nlattr *nla,
* as we can. You better have at least one else we are
* going to bail out
*/
- err = use_all_metadata(ife);
+ err = use_all_metadata(ife, exists);
if (err) {
if (ret == ACT_P_CREATED)
_tcf_ife_cleanup(*a, bind);
-
- if (exists)
- spin_unlock_bh(&ife->tcf_lock);
return err;
}
}
- if (exists)
- spin_unlock_bh(&ife->tcf_lock);
-
if (ret == ACT_P_CREATED)
tcf_hash_insert(tn, *a);
@@ -634,7 +645,7 @@ int find_decode_metaid(struct sk_buff *skb, struct tcf_ife_info *ife,
}
}
- return 0;
+ return -ENOENT;
}
struct ifeheadr {
diff --git a/net/sched/act_simple.c b/net/sched/act_simple.c
index 289af6f..8b2e87e 100644
--- a/net/sched/act_simple.c
+++ b/net/sched/act_simple.c
@@ -55,22 +55,22 @@ static void tcf_simp_release(struct tc_action *a, int bind)
kfree(d->tcfd_defdata);
}
-static int alloc_defdata(struct tcf_defact *d, char *defdata)
+static int alloc_defdata(struct tcf_defact *d, const struct nlattr *defdata)
{
d->tcfd_defdata = kzalloc(SIMP_MAX_DATA, GFP_KERNEL);
if (unlikely(!d->tcfd_defdata))
return -ENOMEM;
- strlcpy(d->tcfd_defdata, defdata, SIMP_MAX_DATA);
+ nla_strlcpy(d->tcfd_defdata, defdata, SIMP_MAX_DATA);
return 0;
}
-static void reset_policy(struct tcf_defact *d, char *defdata,
+static void reset_policy(struct tcf_defact *d, const struct nlattr *defdata,
struct tc_defact *p)
{
spin_lock_bh(&d->tcf_lock);
d->tcf_action = p->action;
memset(d->tcfd_defdata, 0, SIMP_MAX_DATA);
- strlcpy(d->tcfd_defdata, defdata, SIMP_MAX_DATA);
+ nla_strlcpy(d->tcfd_defdata, defdata, SIMP_MAX_DATA);
spin_unlock_bh(&d->tcf_lock);
}
@@ -89,7 +89,6 @@ static int tcf_simp_init(struct net *net, struct nlattr *nla,
struct tcf_defact *d;
bool exists = false;
int ret = 0, err;
- char *defdata;
if (nla == NULL)
return -EINVAL;
@@ -112,8 +111,6 @@ static int tcf_simp_init(struct net *net, struct nlattr *nla,
return -EINVAL;
}
- defdata = nla_data(tb[TCA_DEF_DATA]);
-
if (!exists) {
ret = tcf_hash_create(tn, parm->index, est, a,
&act_simp_ops, bind, false);
@@ -121,7 +118,7 @@ static int tcf_simp_init(struct net *net, struct nlattr *nla,
return ret;
d = to_defact(*a);
- ret = alloc_defdata(d, defdata);
+ ret = alloc_defdata(d, tb[TCA_DEF_DATA]);
if (ret < 0) {
tcf_hash_cleanup(*a, est);
return ret;
@@ -135,7 +132,7 @@ static int tcf_simp_init(struct net *net, struct nlattr *nla,
if (!ovr)
return -EEXIST;
- reset_policy(d, defdata, parm);
+ reset_policy(d, tb[TCA_DEF_DATA], parm);
}
if (ret == ACT_P_CREATED)
diff --git a/net/sched/act_tunnel_key.c b/net/sched/act_tunnel_key.c
index 901fb8b..41835f6 100644
--- a/net/sched/act_tunnel_key.c
+++ b/net/sched/act_tunnel_key.c
@@ -39,7 +39,7 @@ static int tunnel_key_act(struct sk_buff *skb, const struct tc_action *a,
tcf_lastuse_update(&t->tcf_tm);
bstats_cpu_update(this_cpu_ptr(t->common.cpu_bstats), skb);
- action = params->action;
+ action = READ_ONCE(t->tcf_action);
switch (params->tcft_action) {
case TCA_TUNNEL_KEY_ACT_RELEASE:
@@ -170,7 +170,7 @@ static int tunnel_key_init(struct net *net, struct nlattr *nla,
params_old = rtnl_dereference(t->params);
- params_new->action = parm->action;
+ t->tcf_action = parm->action;
params_new->tcft_action = parm->t_action;
params_new->tcft_enc_metadata = metadata;
@@ -242,13 +242,13 @@ static int tunnel_key_dump(struct sk_buff *skb, struct tc_action *a,
.index = t->tcf_index,
.refcnt = t->tcf_refcnt - ref,
.bindcnt = t->tcf_bindcnt - bind,
+ .action = t->tcf_action,
};
struct tcf_t tm;
params = rtnl_dereference(t->params);
opt.t_action = params->tcft_action;
- opt.action = params->action;
if (nla_put(skb, TCA_TUNNEL_KEY_PARMS, sizeof(opt), &opt))
goto nla_put_failure;
diff --git a/net/sched/cls_matchall.c b/net/sched/cls_matchall.c
index e75fb650..61ddfba 100644
--- a/net/sched/cls_matchall.c
+++ b/net/sched/cls_matchall.c
@@ -94,6 +94,8 @@ static bool mall_destroy(struct tcf_proto *tp, bool force)
if (!head)
return true;
+ tcf_unbind_filter(tp, &head->res);
+
if (tc_should_offload(dev, tp, head->flags))
mall_destroy_hw_filter(tp, head, (unsigned long) head);
diff --git a/net/sched/cls_tcindex.c b/net/sched/cls_tcindex.c
index 0751245..db80a64 100644
--- a/net/sched/cls_tcindex.c
+++ b/net/sched/cls_tcindex.c
@@ -414,11 +414,6 @@ tcindex_set_parms(struct net *net, struct tcf_proto *tp, unsigned long base,
tcf_bind_filter(tp, &cr.res, base);
}
- if (old_r)
- tcf_exts_change(tp, &r->exts, &e);
- else
- tcf_exts_change(tp, &cr.exts, &e);
-
if (old_r && old_r != r) {
err = tcindex_filter_result_init(old_r);
if (err < 0) {
@@ -429,12 +424,15 @@ tcindex_set_parms(struct net *net, struct tcf_proto *tp, unsigned long base,
oldp = p;
r->res = cr.res;
+ tcf_exts_change(tp, &r->exts, &e);
+
rcu_assign_pointer(tp->root, cp);
if (r == &new_filter_result) {
struct tcindex_filter *nfp;
struct tcindex_filter __rcu **fp;
+ f->result.res = r->res;
tcf_exts_change(tp, &f->result.exts, &r->exts);
fp = cp->h + (handle % cp->hash);
diff --git a/net/sched/cls_u32.c b/net/sched/cls_u32.c
index da574a1..e377dd5 100644
--- a/net/sched/cls_u32.c
+++ b/net/sched/cls_u32.c
@@ -851,6 +851,7 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
struct nlattr *opt = tca[TCA_OPTIONS];
struct nlattr *tb[TCA_U32_MAX + 1];
u32 htid, flags = 0;
+ size_t sel_size;
int err;
#ifdef CONFIG_CLS_U32_PERF
size_t size;
@@ -967,8 +968,11 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
return -EINVAL;
s = nla_data(tb[TCA_U32_SEL]);
+ sel_size = sizeof(*s) + sizeof(*s->keys) * s->nkeys;
+ if (nla_len(tb[TCA_U32_SEL]) < sel_size)
+ return -EINVAL;
- n = kzalloc(sizeof(*n) + s->nkeys*sizeof(struct tc_u32_key), GFP_KERNEL);
+ n = kzalloc(offsetof(typeof(*n), sel) + sel_size, GFP_KERNEL);
if (n == NULL)
return -ENOBUFS;
@@ -981,7 +985,7 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
}
#endif
- memcpy(&n->sel, s, sizeof(*s) + s->nkeys*sizeof(struct tc_u32_key));
+ memcpy(&n->sel, s, sel_size);
RCU_INIT_POINTER(n->ht_up, ht);
n->handle = handle;
n->fshift = s->hmask ? ffs(ntohl(s->hmask)) - 1 : 0;
diff --git a/net/sched/em_meta.c b/net/sched/em_meta.c
index a309a07..909848fb 100644
--- a/net/sched/em_meta.c
+++ b/net/sched/em_meta.c
@@ -63,6 +63,7 @@
#include <linux/types.h>
#include <linux/kernel.h>
#include <linux/sched.h>
+#include <linux/sched/loadavg.h>
#include <linux/string.h>
#include <linux/skbuff.h>
#include <linux/random.h>
diff --git a/net/sched/sch_blackhole.c b/net/sched/sch_blackhole.c
index c98a61e..9c4c2bb 100644
--- a/net/sched/sch_blackhole.c
+++ b/net/sched/sch_blackhole.c
@@ -21,7 +21,7 @@ static int blackhole_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
qdisc_drop(skb, sch, to_free);
- return NET_XMIT_SUCCESS;
+ return NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
}
static struct sk_buff *blackhole_dequeue(struct Qdisc *sch)
diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c
index 18e7524..b57b4de 100644
--- a/net/sched/sch_fq.c
+++ b/net/sched/sch_fq.c
@@ -128,6 +128,28 @@ static bool fq_flow_is_detached(const struct fq_flow *f)
return f->next == &detached;
}
+static bool fq_flow_is_throttled(const struct fq_flow *f)
+{
+ return f->next == &throttled;
+}
+
+static void fq_flow_add_tail(struct fq_flow_head *head, struct fq_flow *flow)
+{
+ if (head->first)
+ head->last->next = flow;
+ else
+ head->first = flow;
+ head->last = flow;
+ flow->next = NULL;
+}
+
+static void fq_flow_unset_throttled(struct fq_sched_data *q, struct fq_flow *f)
+{
+ rb_erase(&f->rate_node, &q->delayed);
+ q->throttled_flows--;
+ fq_flow_add_tail(&q->old_flows, f);
+}
+
static void fq_flow_set_throttled(struct fq_sched_data *q, struct fq_flow *f)
{
struct rb_node **p = &q->delayed.rb_node, *parent = NULL;
@@ -155,15 +177,6 @@ static void fq_flow_set_throttled(struct fq_sched_data *q, struct fq_flow *f)
static struct kmem_cache *fq_flow_cachep __read_mostly;
-static void fq_flow_add_tail(struct fq_flow_head *head, struct fq_flow *flow)
-{
- if (head->first)
- head->last->next = flow;
- else
- head->first = flow;
- head->last = flow;
- flow->next = NULL;
-}
/* limit number of collected flows per round */
#define FQ_GC_MAX 8
@@ -267,6 +280,8 @@ static struct fq_flow *fq_classify(struct sk_buff *skb, struct fq_sched_data *q)
f->socket_hash != sk->sk_hash)) {
f->credit = q->initial_quantum;
f->socket_hash = sk->sk_hash;
+ if (fq_flow_is_throttled(f))
+ fq_flow_unset_throttled(q, f);
f->time_next_packet = 0ULL;
}
return f;
@@ -430,9 +445,7 @@ static void fq_check_throttled(struct fq_sched_data *q, u64 now)
q->time_next_delayed_flow = f->time_next_packet;
break;
}
- rb_erase(p, &q->delayed);
- q->throttled_flows--;
- fq_flow_add_tail(&q->old_flows, f);
+ fq_flow_unset_throttled(q, f);
}
}
diff --git a/net/sched/sch_gred.c b/net/sched/sch_gred.c
index 44941e2..729c0e4 100644
--- a/net/sched/sch_gred.c
+++ b/net/sched/sch_gred.c
@@ -411,7 +411,7 @@ static int gred_change(struct Qdisc *sch, struct nlattr *opt)
if (tb[TCA_GRED_PARMS] == NULL && tb[TCA_GRED_STAB] == NULL) {
if (tb[TCA_GRED_LIMIT] != NULL)
sch->limit = nla_get_u32(tb[TCA_GRED_LIMIT]);
- return gred_change_table_def(sch, opt);
+ return gred_change_table_def(sch, tb[TCA_GRED_DPS]);
}
if (tb[TCA_GRED_PARMS] == NULL ||
diff --git a/net/sched/sch_hhf.c b/net/sched/sch_hhf.c
index 2fae8b5..f4b2d69 100644
--- a/net/sched/sch_hhf.c
+++ b/net/sched/sch_hhf.c
@@ -492,6 +492,9 @@ static void hhf_destroy(struct Qdisc *sch)
hhf_free(q->hhf_valid_bits[i]);
}
+ if (!q->hh_flows)
+ return;
+
for (i = 0; i < HH_FLOWS_CNT; i++) {
struct hh_flow_state *flow, *next;
struct list_head *head = &q->hh_flows[i];
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index c798d0d..95fe75d 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -1013,6 +1013,9 @@ static int htb_init(struct Qdisc *sch, struct nlattr *opt)
int err;
int i;
+ qdisc_watchdog_init(&q->watchdog, sch);
+ INIT_WORK(&q->work, htb_work_func);
+
if (!opt)
return -EINVAL;
@@ -1033,8 +1036,6 @@ static int htb_init(struct Qdisc *sch, struct nlattr *opt)
for (i = 0; i < TC_HTB_NUMPRIO; i++)
INIT_LIST_HEAD(q->drops + i);
- qdisc_watchdog_init(&q->watchdog, sch);
- INIT_WORK(&q->work, htb_work_func);
qdisc_skb_head_init(&q->direct_queue);
if (tb[TCA_HTB_DIRECT_QLEN])
diff --git a/net/sched/sch_multiq.c b/net/sched/sch_multiq.c
index 9ffbb02..66b6e80 100644
--- a/net/sched/sch_multiq.c
+++ b/net/sched/sch_multiq.c
@@ -234,7 +234,7 @@ static int multiq_tune(struct Qdisc *sch, struct nlattr *opt)
static int multiq_init(struct Qdisc *sch, struct nlattr *opt)
{
struct multiq_sched_data *q = qdisc_priv(sch);
- int i, err;
+ int i;
q->queues = NULL;
@@ -249,12 +249,7 @@ static int multiq_init(struct Qdisc *sch, struct nlattr *opt)
for (i = 0; i < q->max_bands; i++)
q->queues[i] = &noop_qdisc;
- err = multiq_tune(sch, opt);
-
- if (err)
- kfree(q->queues);
-
- return err;
+ return multiq_tune(sch, opt);
}
static int multiq_dump(struct Qdisc *sch, struct sk_buff *skb)
diff --git a/net/sched/sch_netem.c b/net/sched/sch_netem.c
index e899d9e..2e417c90 100644
--- a/net/sched/sch_netem.c
+++ b/net/sched/sch_netem.c
@@ -442,6 +442,9 @@ static int netem_enqueue(struct sk_buff *skb, struct Qdisc *sch,
int count = 1;
int rc = NET_XMIT_SUCCESS;
+ /* Do not fool qdisc_drop_all() */
+ skb->prev = NULL;
+
/* Random duplication */
if (q->duplicate && q->duplicate >= get_crandom(&q->dup_cor))
++count;
@@ -937,11 +940,11 @@ static int netem_init(struct Qdisc *sch, struct nlattr *opt)
struct netem_sched_data *q = qdisc_priv(sch);
int ret;
+ qdisc_watchdog_init(&q->watchdog, sch);
+
if (!opt)
return -EINVAL;
- qdisc_watchdog_init(&q->watchdog, sch);
-
q->loss_model = CLG_RANDOM;
ret = netem_change(sch, opt);
if (ret)
diff --git a/net/sched/sch_tbf.c b/net/sched/sch_tbf.c
index 303355c..b3f7980 100644
--- a/net/sched/sch_tbf.c
+++ b/net/sched/sch_tbf.c
@@ -423,12 +423,13 @@ static int tbf_init(struct Qdisc *sch, struct nlattr *opt)
{
struct tbf_sched_data *q = qdisc_priv(sch);
+ qdisc_watchdog_init(&q->watchdog, sch);
+ q->qdisc = &noop_qdisc;
+
if (opt == NULL)
return -EINVAL;
q->t_c = ktime_get_ns();
- qdisc_watchdog_init(&q->watchdog, sch);
- q->qdisc = &noop_qdisc;
return tbf_change(sch, opt);
}
diff --git a/net/sctp/associola.c b/net/sctp/associola.c
index f10d339..7e127cd 100644
--- a/net/sctp/associola.c
+++ b/net/sctp/associola.c
@@ -488,8 +488,9 @@ void sctp_assoc_set_primary(struct sctp_association *asoc,
void sctp_assoc_rm_peer(struct sctp_association *asoc,
struct sctp_transport *peer)
{
- struct list_head *pos;
- struct sctp_transport *transport;
+ struct sctp_transport *transport;
+ struct list_head *pos;
+ struct sctp_chunk *ch;
pr_debug("%s: association:%p addr:%pISpc\n",
__func__, asoc, &peer->ipaddr.sa);
@@ -547,7 +548,6 @@ void sctp_assoc_rm_peer(struct sctp_association *asoc,
*/
if (!list_empty(&peer->transmitted)) {
struct sctp_transport *active = asoc->peer.active_path;
- struct sctp_chunk *ch;
/* Reset the transport of each chunk on this list */
list_for_each_entry(ch, &peer->transmitted,
@@ -569,6 +569,10 @@ void sctp_assoc_rm_peer(struct sctp_association *asoc,
sctp_transport_hold(active);
}
+ list_for_each_entry(ch, &asoc->outqueue.out_chunk_list, list)
+ if (ch->transport == peer)
+ ch->transport = NULL;
+
asoc->peer.transport_count--;
sctp_transport_free(peer);
@@ -1006,9 +1010,10 @@ static void sctp_assoc_bh_rcv(struct work_struct *work)
struct sctp_endpoint *ep;
struct sctp_chunk *chunk;
struct sctp_inq *inqueue;
- int state;
sctp_subtype_t subtype;
+ int first_time = 1; /* is this the first time through the loop */
int error = 0;
+ int state;
/* The association should be held so we should be safe. */
ep = asoc->ep;
@@ -1019,6 +1024,30 @@ static void sctp_assoc_bh_rcv(struct work_struct *work)
state = asoc->state;
subtype = SCTP_ST_CHUNK(chunk->chunk_hdr->type);
+ /* If the first chunk in the packet is AUTH, do special
+ * processing specified in Section 6.3 of SCTP-AUTH spec
+ */
+ if (first_time && subtype.chunk == SCTP_CID_AUTH) {
+ struct sctp_chunkhdr *next_hdr;
+
+ next_hdr = sctp_inq_peek(inqueue);
+ if (!next_hdr)
+ goto normal;
+
+ /* If the next chunk is COOKIE-ECHO, skip the AUTH
+ * chunk while saving a pointer to it so we can do
+ * Authentication later (during cookie-echo
+ * processing).
+ */
+ if (next_hdr->type == SCTP_CID_COOKIE_ECHO) {
+ chunk->auth_chunk = skb_clone(chunk->skb,
+ GFP_ATOMIC);
+ chunk->auth = 1;
+ continue;
+ }
+ }
+
+normal:
/* SCTP-AUTH, Section 6.3:
* The receiver has a list of chunk types which it expects
* to be received only after an AUTH-chunk. This list has
@@ -1057,6 +1086,9 @@ static void sctp_assoc_bh_rcv(struct work_struct *work)
/* If there is an error on chunk, discard this packet. */
if (error && chunk)
chunk->pdiscard = 1;
+
+ if (first_time)
+ first_time = 0;
}
sctp_association_put(asoc);
}
diff --git a/net/sctp/inqueue.c b/net/sctp/inqueue.c
index f731de3e8..e06083c 100644
--- a/net/sctp/inqueue.c
+++ b/net/sctp/inqueue.c
@@ -217,7 +217,7 @@ struct sctp_chunk *sctp_inq_pop(struct sctp_inq *queue)
skb_pull(chunk->skb, sizeof(sctp_chunkhdr_t));
chunk->subh.v = NULL; /* Subheader is no longer valid. */
- if (chunk->chunk_end + sizeof(sctp_chunkhdr_t) <
+ if (chunk->chunk_end + sizeof(sctp_chunkhdr_t) <=
skb_tail_pointer(chunk->skb)) {
/* This is not a singleton */
chunk->singleton = 0;
diff --git a/net/sctp/ipv6.c b/net/sctp/ipv6.c
index 355d95a7..e7866d4 100644
--- a/net/sctp/ipv6.c
+++ b/net/sctp/ipv6.c
@@ -101,6 +101,7 @@ static int sctp_inet6addr_event(struct notifier_block *this, unsigned long ev,
if (addr) {
addr->a.v6.sin6_family = AF_INET6;
addr->a.v6.sin6_port = 0;
+ addr->a.v6.sin6_flowinfo = 0;
addr->a.v6.sin6_addr = ifa->addr;
addr->a.v6.sin6_scope_id = ifa->idev->dev->ifindex;
addr->valid = 1;
@@ -521,44 +522,47 @@ static void sctp_v6_to_addr(union sctp_addr *addr, struct in6_addr *saddr,
addr->v6.sin6_scope_id = 0;
}
+static int __sctp_v6_cmp_addr(const union sctp_addr *addr1,
+ const union sctp_addr *addr2)
+{
+ if (addr1->sa.sa_family != addr2->sa.sa_family) {
+ if (addr1->sa.sa_family == AF_INET &&
+ addr2->sa.sa_family == AF_INET6 &&
+ ipv6_addr_v4mapped(&addr2->v6.sin6_addr) &&
+ addr2->v6.sin6_addr.s6_addr32[3] ==
+ addr1->v4.sin_addr.s_addr)
+ return 1;
+
+ if (addr2->sa.sa_family == AF_INET &&
+ addr1->sa.sa_family == AF_INET6 &&
+ ipv6_addr_v4mapped(&addr1->v6.sin6_addr) &&
+ addr1->v6.sin6_addr.s6_addr32[3] ==
+ addr2->v4.sin_addr.s_addr)
+ return 1;
+
+ return 0;
+ }
+
+ if (!ipv6_addr_equal(&addr1->v6.sin6_addr, &addr2->v6.sin6_addr))
+ return 0;
+
+ /* If this is a linklocal address, compare the scope_id. */
+ if ((ipv6_addr_type(&addr1->v6.sin6_addr) & IPV6_ADDR_LINKLOCAL) &&
+ addr1->v6.sin6_scope_id && addr2->v6.sin6_scope_id &&
+ addr1->v6.sin6_scope_id != addr2->v6.sin6_scope_id)
+ return 0;
+
+ return 1;
+}
+
/* Compare addresses exactly.
* v4-mapped-v6 is also in consideration.
*/
static int sctp_v6_cmp_addr(const union sctp_addr *addr1,
const union sctp_addr *addr2)
{
- if (addr1->sa.sa_family != addr2->sa.sa_family) {
- if (addr1->sa.sa_family == AF_INET &&
- addr2->sa.sa_family == AF_INET6 &&
- ipv6_addr_v4mapped(&addr2->v6.sin6_addr)) {
- if (addr2->v6.sin6_port == addr1->v4.sin_port &&
- addr2->v6.sin6_addr.s6_addr32[3] ==
- addr1->v4.sin_addr.s_addr)
- return 1;
- }
- if (addr2->sa.sa_family == AF_INET &&
- addr1->sa.sa_family == AF_INET6 &&
- ipv6_addr_v4mapped(&addr1->v6.sin6_addr)) {
- if (addr1->v6.sin6_port == addr2->v4.sin_port &&
- addr1->v6.sin6_addr.s6_addr32[3] ==
- addr2->v4.sin_addr.s_addr)
- return 1;
- }
- return 0;
- }
- if (addr1->v6.sin6_port != addr2->v6.sin6_port)
- return 0;
- if (!ipv6_addr_equal(&addr1->v6.sin6_addr, &addr2->v6.sin6_addr))
- return 0;
- /* If this is a linklocal address, compare the scope_id. */
- if (ipv6_addr_type(&addr1->v6.sin6_addr) & IPV6_ADDR_LINKLOCAL) {
- if (addr1->v6.sin6_scope_id && addr2->v6.sin6_scope_id &&
- (addr1->v6.sin6_scope_id != addr2->v6.sin6_scope_id)) {
- return 0;
- }
- }
-
- return 1;
+ return __sctp_v6_cmp_addr(addr1, addr2) &&
+ addr1->v6.sin6_port == addr2->v6.sin6_port;
}
/* Initialize addr struct to INADDR_ANY. */
@@ -844,8 +848,8 @@ static int sctp_inet6_cmp_addr(const union sctp_addr *addr1,
const union sctp_addr *addr2,
struct sctp_sock *opt)
{
- struct sctp_af *af1, *af2;
struct sock *sk = sctp_opt2sk(opt);
+ struct sctp_af *af1, *af2;
af1 = sctp_get_af_specific(addr1->sa.sa_family);
af2 = sctp_get_af_specific(addr2->sa.sa_family);
@@ -861,10 +865,10 @@ static int sctp_inet6_cmp_addr(const union sctp_addr *addr1,
if (sctp_is_any(sk, addr1) || sctp_is_any(sk, addr2))
return 1;
- if (addr1->sa.sa_family != addr2->sa.sa_family)
- return 0;
+ if (addr1->sa.sa_family == AF_INET && addr2->sa.sa_family == AF_INET)
+ return addr1->v4.sin_addr.s_addr == addr2->v4.sin_addr.s_addr;
- return af1->cmp_addr(addr1, addr2);
+ return __sctp_v6_cmp_addr(addr1, addr2);
}
/* Verify that the provided sockaddr looks bindable. Common verification,
diff --git a/net/sctp/proc.c b/net/sctp/proc.c
index 206377f..fd7f235 100644
--- a/net/sctp/proc.c
+++ b/net/sctp/proc.c
@@ -337,8 +337,6 @@ static int sctp_assocs_seq_show(struct seq_file *seq, void *v)
}
transport = (struct sctp_transport *)v;
- if (!sctp_transport_hold(transport))
- return 0;
assoc = transport->asoc;
epb = &assoc->base;
sk = epb->sk;
@@ -428,8 +426,6 @@ static int sctp_remaddr_seq_show(struct seq_file *seq, void *v)
}
transport = (struct sctp_transport *)v;
- if (!sctp_transport_hold(transport))
- return 0;
assoc = transport->asoc;
list_for_each_entry_rcu(tsp, &assoc->peer.transport_addr_list,
diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c
index 8ec20a6..bfd0686 100644
--- a/net/sctp/sm_statefuns.c
+++ b/net/sctp/sm_statefuns.c
@@ -144,10 +144,8 @@ static sctp_disposition_t sctp_sf_violation_chunk(
void *arg,
sctp_cmd_seq_t *commands);
-static sctp_ierror_t sctp_sf_authenticate(struct net *net,
- const struct sctp_endpoint *ep,
+static sctp_ierror_t sctp_sf_authenticate(
const struct sctp_association *asoc,
- const sctp_subtype_t type,
struct sctp_chunk *chunk);
static sctp_disposition_t __sctp_sf_do_9_1_abort(struct net *net,
@@ -615,6 +613,38 @@ sctp_disposition_t sctp_sf_do_5_1C_ack(struct net *net,
return SCTP_DISPOSITION_CONSUME;
}
+static bool sctp_auth_chunk_verify(struct net *net, struct sctp_chunk *chunk,
+ const struct sctp_association *asoc)
+{
+ struct sctp_chunk auth;
+
+ if (!chunk->auth_chunk)
+ return true;
+
+ /* SCTP-AUTH: auth_chunk pointer is only set when the cookie-echo
+ * is supposed to be authenticated and we have to do delayed
+ * authentication. We've just recreated the association using
+ * the information in the cookie and now it's much easier to
+ * do the authentication.
+ */
+
+ /* Make sure that we and the peer are AUTH capable */
+ if (!net->sctp.auth_enable || !asoc->peer.auth_capable)
+ return false;
+
+ /* set-up our fake chunk so that we can process it */
+ auth.skb = chunk->auth_chunk;
+ auth.asoc = chunk->asoc;
+ auth.sctp_hdr = chunk->sctp_hdr;
+ auth.chunk_hdr = (struct sctp_chunkhdr *)
+ skb_push(chunk->auth_chunk,
+ sizeof(struct sctp_chunkhdr));
+ skb_pull(chunk->auth_chunk, sizeof(struct sctp_chunkhdr));
+ auth.transport = chunk->transport;
+
+ return sctp_sf_authenticate(asoc, &auth) == SCTP_IERROR_NO_ERROR;
+}
+
/*
* Respond to a normal COOKIE ECHO chunk.
* We are the side that is being asked for an association.
@@ -751,36 +781,9 @@ sctp_disposition_t sctp_sf_do_5_1D_ce(struct net *net,
if (error)
goto nomem_init;
- /* SCTP-AUTH: auth_chunk pointer is only set when the cookie-echo
- * is supposed to be authenticated and we have to do delayed
- * authentication. We've just recreated the association using
- * the information in the cookie and now it's much easier to
- * do the authentication.
- */
- if (chunk->auth_chunk) {
- struct sctp_chunk auth;
- sctp_ierror_t ret;
-
- /* Make sure that we and the peer are AUTH capable */
- if (!net->sctp.auth_enable || !new_asoc->peer.auth_capable) {
- sctp_association_free(new_asoc);
- return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
- }
-
- /* set-up our fake chunk so that we can process it */
- auth.skb = chunk->auth_chunk;
- auth.asoc = chunk->asoc;
- auth.sctp_hdr = chunk->sctp_hdr;
- auth.chunk_hdr = (sctp_chunkhdr_t *)skb_push(chunk->auth_chunk,
- sizeof(sctp_chunkhdr_t));
- skb_pull(chunk->auth_chunk, sizeof(sctp_chunkhdr_t));
- auth.transport = chunk->transport;
-
- ret = sctp_sf_authenticate(net, ep, new_asoc, type, &auth);
- if (ret != SCTP_IERROR_NO_ERROR) {
- sctp_association_free(new_asoc);
- return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
- }
+ if (!sctp_auth_chunk_verify(net, chunk, new_asoc)) {
+ sctp_association_free(new_asoc);
+ return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
}
repl = sctp_make_cookie_ack(new_asoc, chunk);
@@ -1717,13 +1720,15 @@ static sctp_disposition_t sctp_sf_do_dupcook_a(struct net *net,
GFP_ATOMIC))
goto nomem;
+ if (!sctp_auth_chunk_verify(net, chunk, new_asoc))
+ return SCTP_DISPOSITION_DISCARD;
+
/* Make sure no new addresses are being added during the
* restart. Though this is a pretty complicated attack
* since you'd have to get inside the cookie.
*/
- if (!sctp_sf_check_restart_addrs(new_asoc, asoc, chunk, commands)) {
+ if (!sctp_sf_check_restart_addrs(new_asoc, asoc, chunk, commands))
return SCTP_DISPOSITION_CONSUME;
- }
/* If the endpoint is in the SHUTDOWN-ACK-SENT state and recognizes
* the peer has restarted (Action A), it MUST NOT setup a new
@@ -1828,6 +1833,9 @@ static sctp_disposition_t sctp_sf_do_dupcook_b(struct net *net,
GFP_ATOMIC))
goto nomem;
+ if (!sctp_auth_chunk_verify(net, chunk, new_asoc))
+ return SCTP_DISPOSITION_DISCARD;
+
/* Update the content of current association. */
sctp_add_cmd_sf(commands, SCTP_CMD_UPDATE_ASSOC, SCTP_ASOC(new_asoc));
sctp_add_cmd_sf(commands, SCTP_CMD_NEW_STATE,
@@ -1920,6 +1928,9 @@ static sctp_disposition_t sctp_sf_do_dupcook_d(struct net *net,
* a COOKIE ACK.
*/
+ if (!sctp_auth_chunk_verify(net, chunk, asoc))
+ return SCTP_DISPOSITION_DISCARD;
+
/* Don't accidentally move back into established state. */
if (asoc->state < SCTP_STATE_ESTABLISHED) {
sctp_add_cmd_sf(commands, SCTP_CMD_TIMER_STOP,
@@ -1959,7 +1970,7 @@ static sctp_disposition_t sctp_sf_do_dupcook_d(struct net *net,
}
}
- repl = sctp_make_cookie_ack(new_asoc, chunk);
+ repl = sctp_make_cookie_ack(asoc, chunk);
if (!repl)
goto nomem;
@@ -3981,10 +3992,8 @@ sctp_disposition_t sctp_sf_eat_fwd_tsn_fast(
*
* The return value is the disposition of the chunk.
*/
-static sctp_ierror_t sctp_sf_authenticate(struct net *net,
- const struct sctp_endpoint *ep,
+static sctp_ierror_t sctp_sf_authenticate(
const struct sctp_association *asoc,
- const sctp_subtype_t type,
struct sctp_chunk *chunk)
{
struct sctp_authhdr *auth_hdr;
@@ -4083,7 +4092,7 @@ sctp_disposition_t sctp_sf_eat_auth(struct net *net,
commands);
auth_hdr = (struct sctp_authhdr *)chunk->skb->data;
- error = sctp_sf_authenticate(net, ep, asoc, type, chunk);
+ error = sctp_sf_authenticate(asoc, chunk);
switch (error) {
case SCTP_IERROR_AUTH_BAD_HMAC:
/* Generate the ERROR chunk and discard the rest
diff --git a/net/sctp/socket.c b/net/sctp/socket.c
index 78f3805..93e6006 100644
--- a/net/sctp/socket.c
+++ b/net/sctp/socket.c
@@ -185,13 +185,13 @@ static void sctp_for_each_tx_datachunk(struct sctp_association *asoc,
list_for_each_entry(chunk, &t->transmitted, transmitted_list)
cb(chunk);
- list_for_each_entry(chunk, &q->retransmit, list)
+ list_for_each_entry(chunk, &q->retransmit, transmitted_list)
cb(chunk);
- list_for_each_entry(chunk, &q->sacked, list)
+ list_for_each_entry(chunk, &q->sacked, transmitted_list)
cb(chunk);
- list_for_each_entry(chunk, &q->abandoned, list)
+ list_for_each_entry(chunk, &q->abandoned, transmitted_list)
cb(chunk);
list_for_each_entry(chunk, &q->out_chunk_list, list)
@@ -248,11 +248,10 @@ struct sctp_association *sctp_id2assoc(struct sock *sk, sctp_assoc_t id)
spin_lock_bh(&sctp_assocs_id_lock);
asoc = (struct sctp_association *)idr_find(&sctp_assocs_id, (int)id);
+ if (asoc && (asoc->base.sk != sk || asoc->base.dead))
+ asoc = NULL;
spin_unlock_bh(&sctp_assocs_id_lock);
- if (!asoc || (asoc->base.sk != sk) || asoc->base.dead)
- return NULL;
-
return asoc;
}
@@ -3733,32 +3732,16 @@ static int sctp_setsockopt_pr_supported(struct sock *sk,
unsigned int optlen)
{
struct sctp_assoc_value params;
- struct sctp_association *asoc;
- int retval = -EINVAL;
if (optlen != sizeof(params))
- goto out;
+ return -EINVAL;
- if (copy_from_user(¶ms, optval, optlen)) {
- retval = -EFAULT;
- goto out;
- }
+ if (copy_from_user(¶ms, optval, optlen))
+ return -EFAULT;
- asoc = sctp_id2assoc(sk, params.assoc_id);
- if (asoc) {
- asoc->prsctp_enable = !!params.assoc_value;
- } else if (!params.assoc_id) {
- struct sctp_sock *sp = sctp_sk(sk);
+ sctp_sk(sk)->ep->prsctp_enable = !!params.assoc_value;
- sp->ep->prsctp_enable = !!params.assoc_value;
- } else {
- goto out;
- }
-
- retval = 0;
-
-out:
- return retval;
+ return 0;
}
static int sctp_setsockopt_default_prinfo(struct sock *sk,
@@ -4476,9 +4459,14 @@ struct sctp_transport *sctp_transport_get_next(struct net *net,
break;
}
+ if (!sctp_transport_hold(t))
+ continue;
+
if (net_eq(sock_net(t->asoc->base.sk), net) &&
t->asoc->peer.primary_path == t)
break;
+
+ sctp_transport_put(t);
}
return t;
@@ -4488,13 +4476,18 @@ struct sctp_transport *sctp_transport_get_idx(struct net *net,
struct rhashtable_iter *iter,
int pos)
{
- void *obj = SEQ_START_TOKEN;
+ struct sctp_transport *t;
- while (pos && (obj = sctp_transport_get_next(net, iter)) &&
- !IS_ERR(obj))
- pos--;
+ if (!pos)
+ return SEQ_START_TOKEN;
- return obj;
+ while ((t = sctp_transport_get_next(net, iter)) && !IS_ERR(t)) {
+ if (!--pos)
+ break;
+ sctp_transport_put(t);
+ }
+
+ return t;
}
int sctp_for_each_endpoint(int (*cb)(struct sctp_endpoint *, void *),
@@ -4556,8 +4549,6 @@ int sctp_for_each_transport(int (*cb)(struct sctp_transport *, void *),
for (; !IS_ERR_OR_NULL(obj); obj = sctp_transport_get_next(net, &hti)) {
struct sctp_transport *transport = obj;
- if (!sctp_transport_hold(transport))
- continue;
err = cb(transport, p);
sctp_transport_put(transport);
if (err)
diff --git a/net/sctp/transport.c b/net/sctp/transport.c
index ce54dce..03d71cd 100644
--- a/net/sctp/transport.c
+++ b/net/sctp/transport.c
@@ -608,7 +608,7 @@ unsigned long sctp_transport_timeout(struct sctp_transport *trans)
trans->state != SCTP_PF)
timeout += trans->hbinterval;
- return timeout;
+ return max_t(unsigned long, timeout, HZ / 5);
}
/* Reset transport variables to their initial values */
diff --git a/net/sctp/ulpevent.c b/net/sctp/ulpevent.c
index bea0005..6825e05 100644
--- a/net/sctp/ulpevent.c
+++ b/net/sctp/ulpevent.c
@@ -723,7 +723,6 @@ struct sctp_ulpevent *sctp_ulpevent_make_rcvmsg(struct sctp_association *asoc,
return event;
fail_mark:
- sctp_chunk_put(chunk);
kfree_skb(skb);
fail:
return NULL;
diff --git a/net/socket.c b/net/socket.c
index 539755b..6af62e2 100644
--- a/net/socket.c
+++ b/net/socket.c
@@ -89,6 +89,7 @@
#include <linux/magic.h>
#include <linux/slab.h>
#include <linux/xattr.h>
+#include <linux/nospec.h>
#include <linux/seemp_api.h>
#include <linux/seemp_instrumentation.h>
@@ -2377,6 +2378,7 @@ SYSCALL_DEFINE2(socketcall, int, call, unsigned long __user *, args)
if (call < 1 || call > SYS_SENDMMSG)
return -EINVAL;
+ call = array_index_nospec(call, SYS_SENDMMSG + 1);
len = nargs[call];
if (len > sizeof(a))
@@ -2811,9 +2813,14 @@ static int ethtool_ioctl(struct net *net, struct compat_ifreq __user *ifr32)
copy_in_user(&rxnfc->fs.ring_cookie,
&compat_rxnfc->fs.ring_cookie,
(void __user *)(&rxnfc->fs.location + 1) -
- (void __user *)&rxnfc->fs.ring_cookie) ||
- copy_in_user(&rxnfc->rule_cnt, &compat_rxnfc->rule_cnt,
- sizeof(rxnfc->rule_cnt)))
+ (void __user *)&rxnfc->fs.ring_cookie))
+ return -EFAULT;
+ if (ethcmd == ETHTOOL_GRXCLSRLALL) {
+ if (put_user(rule_cnt, &rxnfc->rule_cnt))
+ return -EFAULT;
+ } else if (copy_in_user(&rxnfc->rule_cnt,
+ &compat_rxnfc->rule_cnt,
+ sizeof(rxnfc->rule_cnt)))
return -EFAULT;
}
diff --git a/net/strparser/strparser.c b/net/strparser/strparser.c
index 6cbc935..bbee334 100644
--- a/net/strparser/strparser.c
+++ b/net/strparser/strparser.c
@@ -285,9 +285,9 @@ static int strp_recv(read_descriptor_t *desc, struct sk_buff *orig_skb,
strp_start_rx_timer(strp);
}
+ rxm->accum_len += cand_len;
strp->rx_need_bytes = rxm->strp.full_len -
rxm->accum_len;
- rxm->accum_len += cand_len;
rxm->early_eaten = cand_len;
STRP_STATS_ADD(strp->stats.rx_bytes, cand_len);
desc->count = 0; /* Stop reading socket */
@@ -310,6 +310,7 @@ static int strp_recv(read_descriptor_t *desc, struct sk_buff *orig_skb,
/* Hurray, we have a new message! */
del_timer(&strp->rx_msg_timer);
strp->rx_skb_head = NULL;
+ strp->rx_need_bytes = 0;
STRP_STATS_INCR(strp->stats.rx_msgs);
/* Give skb to upper layer */
@@ -374,9 +375,7 @@ void strp_data_ready(struct strparser *strp)
return;
if (strp->rx_need_bytes) {
- if (strp_peek_len(strp) >= strp->rx_need_bytes)
- strp->rx_need_bytes = 0;
- else
+ if (strp_peek_len(strp) < strp->rx_need_bytes)
return;
}
diff --git a/net/sunrpc/auth_generic.c b/net/sunrpc/auth_generic.c
index f1df983..1ac08dc 100644
--- a/net/sunrpc/auth_generic.c
+++ b/net/sunrpc/auth_generic.c
@@ -281,13 +281,7 @@ static bool generic_key_to_expire(struct rpc_cred *cred)
{
struct auth_cred *acred = &container_of(cred, struct generic_cred,
gc_base)->acred;
- bool ret;
-
- get_rpccred(cred);
- ret = test_bit(RPC_CRED_KEY_EXPIRE_SOON, &acred->ac_flags);
- put_rpccred(cred);
-
- return ret;
+ return test_bit(RPC_CRED_KEY_EXPIRE_SOON, &acred->ac_flags);
}
static const struct rpc_credops generic_credops = {
diff --git a/net/sunrpc/auth_gss/auth_gss.c b/net/sunrpc/auth_gss/auth_gss.c
index 16cea00..591d378 100644
--- a/net/sunrpc/auth_gss/auth_gss.c
+++ b/net/sunrpc/auth_gss/auth_gss.c
@@ -1736,6 +1736,7 @@ priv_release_snd_buf(struct rpc_rqst *rqstp)
for (i=0; i < rqstp->rq_enc_pages_num; i++)
__free_page(rqstp->rq_enc_pages[i]);
kfree(rqstp->rq_enc_pages);
+ rqstp->rq_release_snd_buf = NULL;
}
static int
@@ -1744,6 +1745,9 @@ alloc_enc_pages(struct rpc_rqst *rqstp)
struct xdr_buf *snd_buf = &rqstp->rq_snd_buf;
int first, last, i;
+ if (rqstp->rq_release_snd_buf)
+ rqstp->rq_release_snd_buf(rqstp);
+
if (snd_buf->page_len == 0) {
rqstp->rq_enc_pages_num = 0;
return 0;
diff --git a/net/sunrpc/auth_gss/gss_krb5_crypto.c b/net/sunrpc/auth_gss/gss_krb5_crypto.c
index 4afd414..bad69e9 100644
--- a/net/sunrpc/auth_gss/gss_krb5_crypto.c
+++ b/net/sunrpc/auth_gss/gss_krb5_crypto.c
@@ -169,7 +169,7 @@ make_checksum_hmac_md5(struct krb5_ctx *kctx, char *header, int hdrlen,
struct scatterlist sg[1];
int err = -1;
u8 *checksumdata;
- u8 rc4salt[4];
+ u8 *rc4salt;
struct crypto_ahash *md5;
struct crypto_ahash *hmac_md5;
struct ahash_request *req;
@@ -183,14 +183,18 @@ make_checksum_hmac_md5(struct krb5_ctx *kctx, char *header, int hdrlen,
return GSS_S_FAILURE;
}
+ rc4salt = kmalloc_array(4, sizeof(*rc4salt), GFP_NOFS);
+ if (!rc4salt)
+ return GSS_S_FAILURE;
+
if (arcfour_hmac_md5_usage_to_salt(usage, rc4salt)) {
dprintk("%s: invalid usage value %u\n", __func__, usage);
- return GSS_S_FAILURE;
+ goto out_free_rc4salt;
}
checksumdata = kmalloc(GSS_KRB5_MAX_CKSUM_LEN, GFP_NOFS);
if (!checksumdata)
- return GSS_S_FAILURE;
+ goto out_free_rc4salt;
md5 = crypto_alloc_ahash("md5", 0, CRYPTO_ALG_ASYNC);
if (IS_ERR(md5))
@@ -258,6 +262,8 @@ make_checksum_hmac_md5(struct krb5_ctx *kctx, char *header, int hdrlen,
crypto_free_ahash(md5);
out_free_cksum:
kfree(checksumdata);
+out_free_rc4salt:
+ kfree(rc4salt);
return err ? GSS_S_FAILURE : 0;
}
diff --git a/net/sunrpc/auth_gss/svcauth_gss.c b/net/sunrpc/auth_gss/svcauth_gss.c
index 6a08bc4..b4b68c6 100644
--- a/net/sunrpc/auth_gss/svcauth_gss.c
+++ b/net/sunrpc/auth_gss/svcauth_gss.c
@@ -1112,7 +1112,7 @@ static int svcauth_gss_legacy_init(struct svc_rqst *rqstp,
struct kvec *resv = &rqstp->rq_res.head[0];
struct rsi *rsip, rsikey;
int ret;
- struct sunrpc_net *sn = net_generic(rqstp->rq_xprt->xpt_net, sunrpc_net_id);
+ struct sunrpc_net *sn = net_generic(SVC_NET(rqstp), sunrpc_net_id);
memset(&rsikey, 0, sizeof(rsikey));
ret = gss_read_verf(gc, argv, authp,
@@ -1223,7 +1223,7 @@ static int svcauth_gss_proxy_init(struct svc_rqst *rqstp,
uint64_t handle;
int status;
int ret;
- struct net *net = rqstp->rq_xprt->xpt_net;
+ struct net *net = SVC_NET(rqstp);
struct sunrpc_net *sn = net_generic(net, sunrpc_net_id);
memset(&ud, 0, sizeof(ud));
@@ -1414,7 +1414,7 @@ svcauth_gss_accept(struct svc_rqst *rqstp, __be32 *authp)
__be32 *rpcstart;
__be32 *reject_stat = resv->iov_base + resv->iov_len;
int ret;
- struct sunrpc_net *sn = net_generic(rqstp->rq_xprt->xpt_net, sunrpc_net_id);
+ struct sunrpc_net *sn = net_generic(SVC_NET(rqstp), sunrpc_net_id);
dprintk("RPC: svcauth_gss: argv->iov_len = %zd\n",
argv->iov_len);
@@ -1702,7 +1702,7 @@ svcauth_gss_release(struct svc_rqst *rqstp)
struct rpc_gss_wire_cred *gc = &gsd->clcred;
struct xdr_buf *resbuf = &rqstp->rq_res;
int stat = -EINVAL;
- struct sunrpc_net *sn = net_generic(rqstp->rq_xprt->xpt_net, sunrpc_net_id);
+ struct sunrpc_net *sn = net_generic(SVC_NET(rqstp), sunrpc_net_id);
if (gc->gc_proc != RPC_GSS_PROC_DATA)
goto out;
diff --git a/net/sunrpc/cache.c b/net/sunrpc/cache.c
index 8aabe12..cab50ec 100644
--- a/net/sunrpc/cache.c
+++ b/net/sunrpc/cache.c
@@ -54,6 +54,11 @@ static void cache_init(struct cache_head *h, struct cache_detail *detail)
h->last_refresh = now;
}
+static void cache_fresh_locked(struct cache_head *head, time_t expiry,
+ struct cache_detail *detail);
+static void cache_fresh_unlocked(struct cache_head *head,
+ struct cache_detail *detail);
+
struct cache_head *sunrpc_cache_lookup(struct cache_detail *detail,
struct cache_head *key, int hash)
{
@@ -95,6 +100,7 @@ struct cache_head *sunrpc_cache_lookup(struct cache_detail *detail,
if (cache_is_expired(detail, tmp)) {
hlist_del_init(&tmp->cache_list);
detail->entries --;
+ cache_fresh_locked(tmp, 0, detail);
freeme = tmp;
break;
}
@@ -110,8 +116,10 @@ struct cache_head *sunrpc_cache_lookup(struct cache_detail *detail,
cache_get(new);
write_unlock(&detail->hash_lock);
- if (freeme)
+ if (freeme) {
+ cache_fresh_unlocked(freeme, detail);
cache_put(freeme, detail);
+ }
return new;
}
EXPORT_SYMBOL_GPL(sunrpc_cache_lookup);
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index b2ae4f1..244eac1 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -965,10 +965,20 @@ struct rpc_clnt *rpc_bind_new_program(struct rpc_clnt *old,
}
EXPORT_SYMBOL_GPL(rpc_bind_new_program);
+void rpc_task_release_transport(struct rpc_task *task)
+{
+ struct rpc_xprt *xprt = task->tk_xprt;
+
+ if (xprt) {
+ task->tk_xprt = NULL;
+ xprt_put(xprt);
+ }
+}
+EXPORT_SYMBOL_GPL(rpc_task_release_transport);
+
void rpc_task_release_client(struct rpc_task *task)
{
struct rpc_clnt *clnt = task->tk_client;
- struct rpc_xprt *xprt = task->tk_xprt;
if (clnt != NULL) {
/* Remove from client task list */
@@ -979,12 +989,14 @@ void rpc_task_release_client(struct rpc_task *task)
rpc_release_client(clnt);
}
+ rpc_task_release_transport(task);
+}
- if (xprt != NULL) {
- task->tk_xprt = NULL;
-
- xprt_put(xprt);
- }
+static
+void rpc_task_set_transport(struct rpc_task *task, struct rpc_clnt *clnt)
+{
+ if (!task->tk_xprt)
+ task->tk_xprt = xprt_iter_get_next(&clnt->cl_xpi);
}
static
@@ -992,8 +1004,7 @@ void rpc_task_set_client(struct rpc_task *task, struct rpc_clnt *clnt)
{
if (clnt != NULL) {
- if (task->tk_xprt == NULL)
- task->tk_xprt = xprt_iter_get_next(&clnt->cl_xpi);
+ rpc_task_set_transport(task, clnt);
task->tk_client = clnt;
atomic_inc(&clnt->cl_count);
if (clnt->cl_softrtry)
@@ -1550,6 +1561,7 @@ call_start(struct rpc_task *task)
task->tk_msg.rpc_proc->p_count++;
clnt->cl_stats->rpccnt++;
task->tk_action = call_reserve;
+ rpc_task_set_transport(task, clnt);
}
/*
diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c
index 9c9db55..064f20b 100644
--- a/net/sunrpc/svc_xprt.c
+++ b/net/sunrpc/svc_xprt.c
@@ -1038,7 +1038,7 @@ static void call_xpt_users(struct svc_xprt *xprt)
spin_lock(&xprt->xpt_lock);
while (!list_empty(&xprt->xpt_users)) {
u = list_first_entry(&xprt->xpt_users, struct svc_xpt_user, list);
- list_del(&u->list);
+ list_del_init(&u->list);
u->callback(u);
}
spin_unlock(&xprt->xpt_lock);
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index 266a30c..33f599c 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -572,7 +572,7 @@ static int svc_udp_recvfrom(struct svc_rqst *rqstp)
/* Don't enable netstamp, sunrpc doesn't
need that much accuracy */
}
- svsk->sk_sk->sk_stamp = skb->tstamp;
+ sock_write_timestamp(svsk->sk_sk, skb->tstamp);
set_bit(XPT_DATA, &svsk->sk_xprt.xpt_flags); /* there may be more data... */
len = skb->len;
diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
index 7f1071e..69846c6 100644
--- a/net/sunrpc/xdr.c
+++ b/net/sunrpc/xdr.c
@@ -512,7 +512,7 @@ EXPORT_SYMBOL_GPL(xdr_commit_encode);
static __be32 *xdr_get_next_encode_buffer(struct xdr_stream *xdr,
size_t nbytes)
{
- static __be32 *p;
+ __be32 *p;
int space_left;
int frag1bytes, frag2bytes;
@@ -639,11 +639,10 @@ void xdr_truncate_encode(struct xdr_stream *xdr, size_t len)
WARN_ON_ONCE(xdr->iov);
return;
}
- if (fraglen) {
+ if (fraglen)
xdr->end = head->iov_base + head->iov_len;
- xdr->page_ptr--;
- }
/* (otherwise assume xdr->end is already set) */
+ xdr->page_ptr--;
head->iov_len = len;
buf->len = len;
xdr->p = head->iov_base + head->iov_len;
diff --git a/net/sunrpc/xprt.c b/net/sunrpc/xprt.c
index 685e6d2..1a8df24 100644
--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -778,8 +778,15 @@ void xprt_connect(struct rpc_task *task)
return;
if (xprt_test_and_set_connecting(xprt))
return;
- xprt->stat.connect_start = jiffies;
- xprt->ops->connect(xprt, task);
+ /* Race breaker */
+ if (!xprt_connected(xprt)) {
+ xprt->stat.connect_start = jiffies;
+ xprt->ops->connect(xprt, task);
+ } else {
+ xprt_clear_connecting(xprt);
+ task->tk_status = 0;
+ rpc_wake_up_queued_task(&xprt->pending, task);
+ }
}
xprt_release_write(xprt, task);
}
diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c
index f57c9f0..0287734 100644
--- a/net/sunrpc/xprtrdma/rpc_rdma.c
+++ b/net/sunrpc/xprtrdma/rpc_rdma.c
@@ -229,7 +229,7 @@ rpcrdma_convert_iovs(struct rpcrdma_xprt *r_xprt, struct xdr_buf *xdrbuf,
/* alloc the pagelist for receiving buffer */
ppages[p] = alloc_page(GFP_ATOMIC);
if (!ppages[p])
- return -EAGAIN;
+ return -ENOBUFS;
}
seg[n].mr_page = ppages[p];
seg[n].mr_offset = (void *)(unsigned long) page_base;
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index 1bf9153..280fb31 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -2209,8 +2209,8 @@ static void xs_udp_setup_socket(struct work_struct *work)
trace_rpc_socket_connect(xprt, sock, 0);
status = 0;
out:
- xprt_unlock_connect(xprt, transport);
xprt_clear_connecting(xprt);
+ xprt_unlock_connect(xprt, transport);
xprt_wake_pending_tasks(xprt, status);
}
@@ -2395,8 +2395,8 @@ static void xs_tcp_setup_socket(struct work_struct *work)
}
status = -EAGAIN;
out:
- xprt_unlock_connect(xprt, transport);
xprt_clear_connecting(xprt);
+ xprt_unlock_connect(xprt, transport);
xprt_wake_pending_tasks(xprt, status);
}
diff --git a/net/tipc/netlink.c b/net/tipc/netlink.c
index 3200059..9ba3c46 100644
--- a/net/tipc/netlink.c
+++ b/net/tipc/netlink.c
@@ -79,7 +79,8 @@ const struct nla_policy tipc_nl_sock_policy[TIPC_NLA_SOCK_MAX + 1] = {
const struct nla_policy tipc_nl_net_policy[TIPC_NLA_NET_MAX + 1] = {
[TIPC_NLA_NET_UNSPEC] = { .type = NLA_UNSPEC },
- [TIPC_NLA_NET_ID] = { .type = NLA_U32 }
+ [TIPC_NLA_NET_ID] = { .type = NLA_U32 },
+ [TIPC_NLA_NET_ADDR] = { .type = NLA_U32 },
};
const struct nla_policy tipc_nl_link_policy[TIPC_NLA_LINK_MAX + 1] = {
diff --git a/net/tipc/socket.c b/net/tipc/socket.c
index 25bc5c3..57df99c 100644
--- a/net/tipc/socket.c
+++ b/net/tipc/socket.c
@@ -2277,17 +2277,21 @@ void tipc_sk_reinit(struct net *net)
do {
tsk = ERR_PTR(rhashtable_walk_start(&iter));
- if (tsk)
- continue;
+ if (IS_ERR(tsk))
+ goto walk_stop;
while ((tsk = rhashtable_walk_next(&iter)) && !IS_ERR(tsk)) {
- spin_lock_bh(&tsk->sk.sk_lock.slock);
+ sock_hold(&tsk->sk);
+ rhashtable_walk_stop(&iter);
+ lock_sock(&tsk->sk);
msg = &tsk->phdr;
msg_set_prevnode(msg, tn->own_addr);
msg_set_orignode(msg, tn->own_addr);
- spin_unlock_bh(&tsk->sk.sk_lock.slock);
+ release_sock(&tsk->sk);
+ rhashtable_walk_start(&iter);
+ sock_put(&tsk->sk);
}
-
+walk_stop:
rhashtable_walk_stop(&iter);
} while (tsk == ERR_PTR(-EAGAIN));
}
diff --git a/net/tipc/subscr.c b/net/tipc/subscr.c
index 271cd66..d62affe 100644
--- a/net/tipc/subscr.c
+++ b/net/tipc/subscr.c
@@ -256,7 +256,9 @@ static void tipc_subscrp_delete(struct tipc_subscription *sub)
static void tipc_subscrp_cancel(struct tipc_subscr *s,
struct tipc_subscriber *subscriber)
{
+ tipc_subscrb_get(subscriber);
tipc_subscrb_subscrp_delete(subscriber, s);
+ tipc_subscrb_put(subscriber);
}
static struct tipc_subscription *tipc_subscrp_create(struct net *net,
@@ -387,7 +389,7 @@ int tipc_topsrv_start(struct net *net)
topsrv->tipc_conn_new = tipc_subscrb_connect_cb;
topsrv->tipc_conn_release = tipc_subscrb_release_cb;
- strncpy(topsrv->name, name, strlen(name) + 1);
+ strscpy(topsrv->name, name, sizeof(topsrv->name));
tn->topsrv = topsrv;
atomic_set(&tn->subscription_count, 0);
diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c
index 107375d..133e726 100644
--- a/net/tipc/udp_media.c
+++ b/net/tipc/udp_media.c
@@ -243,10 +243,8 @@ static int tipc_udp_send_msg(struct net *net, struct sk_buff *skb,
}
err = tipc_udp_xmit(net, _skb, ub, src, &rcast->addr);
- if (err) {
- kfree_skb(_skb);
+ if (err)
goto out;
- }
}
err = 0;
out:
@@ -676,6 +674,11 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b,
if (err)
goto err;
+ if (remote.proto != local.proto) {
+ err = -EINVAL;
+ goto err;
+ }
+
b->bcast_addr.media_id = TIPC_MEDIA_TYPE_UDP;
b->bcast_addr.broadcast = 1;
rcu_assign_pointer(b->media_ptr, ub);
diff --git a/net/vmw_vsock/af_vsock.c b/net/vmw_vsock/af_vsock.c
index ee12e17..7566395 100644
--- a/net/vmw_vsock/af_vsock.c
+++ b/net/vmw_vsock/af_vsock.c
@@ -448,14 +448,14 @@ static int vsock_send_shutdown(struct sock *sk, int mode)
return transport->shutdown(vsock_sk(sk), mode);
}
-void vsock_pending_work(struct work_struct *work)
+static void vsock_pending_work(struct work_struct *work)
{
struct sock *sk;
struct sock *listener;
struct vsock_sock *vsk;
bool cleanup;
- vsk = container_of(work, struct vsock_sock, dwork.work);
+ vsk = container_of(work, struct vsock_sock, pending_work.work);
sk = sk_vsock(vsk);
listener = vsk->listener;
cleanup = true;
@@ -495,7 +495,6 @@ void vsock_pending_work(struct work_struct *work)
sock_put(sk);
sock_put(listener);
}
-EXPORT_SYMBOL_GPL(vsock_pending_work);
/**** SOCKET OPERATIONS ****/
@@ -594,6 +593,8 @@ static int __vsock_bind(struct sock *sk, struct sockaddr_vm *addr)
return retval;
}
+static void vsock_connect_timeout(struct work_struct *work);
+
struct sock *__vsock_create(struct net *net,
struct socket *sock,
struct sock *parent,
@@ -636,6 +637,8 @@ struct sock *__vsock_create(struct net *net,
vsk->sent_request = false;
vsk->ignore_connecting_rst = false;
vsk->peer_shutdown = 0;
+ INIT_DELAYED_WORK(&vsk->connect_work, vsock_connect_timeout);
+ INIT_DELAYED_WORK(&vsk->pending_work, vsock_pending_work);
psk = parent ? vsock_sk(parent) : NULL;
if (parent) {
@@ -1115,7 +1118,7 @@ static void vsock_connect_timeout(struct work_struct *work)
struct vsock_sock *vsk;
int cancel = 0;
- vsk = container_of(work, struct vsock_sock, dwork.work);
+ vsk = container_of(work, struct vsock_sock, connect_work.work);
sk = sk_vsock(vsk);
lock_sock(sk);
@@ -1219,9 +1222,7 @@ static int vsock_stream_connect(struct socket *sock, struct sockaddr *addr,
* timeout fires.
*/
sock_hold(sk);
- INIT_DELAYED_WORK(&vsk->dwork,
- vsock_connect_timeout);
- schedule_delayed_work(&vsk->dwork, timeout);
+ schedule_delayed_work(&vsk->connect_work, timeout);
/* Skip ahead to preserve error code set above. */
goto out_wait;
diff --git a/net/vmw_vsock/vmci_transport.c b/net/vmw_vsock/vmci_transport.c
index 4be4fbb..008f342 100644
--- a/net/vmw_vsock/vmci_transport.c
+++ b/net/vmw_vsock/vmci_transport.c
@@ -273,6 +273,31 @@ vmci_transport_send_control_pkt_bh(struct sockaddr_vm *src,
}
static int
+vmci_transport_alloc_send_control_pkt(struct sockaddr_vm *src,
+ struct sockaddr_vm *dst,
+ enum vmci_transport_packet_type type,
+ u64 size,
+ u64 mode,
+ struct vmci_transport_waiting_info *wait,
+ u16 proto,
+ struct vmci_handle handle)
+{
+ struct vmci_transport_packet *pkt;
+ int err;
+
+ pkt = kmalloc(sizeof(*pkt), GFP_KERNEL);
+ if (!pkt)
+ return -ENOMEM;
+
+ err = __vmci_transport_send_control_pkt(pkt, src, dst, type, size,
+ mode, wait, proto, handle,
+ true);
+ kfree(pkt);
+
+ return err;
+}
+
+static int
vmci_transport_send_control_pkt(struct sock *sk,
enum vmci_transport_packet_type type,
u64 size,
@@ -281,9 +306,7 @@ vmci_transport_send_control_pkt(struct sock *sk,
u16 proto,
struct vmci_handle handle)
{
- struct vmci_transport_packet *pkt;
struct vsock_sock *vsk;
- int err;
vsk = vsock_sk(sk);
@@ -293,17 +316,10 @@ vmci_transport_send_control_pkt(struct sock *sk,
if (!vsock_addr_bound(&vsk->remote_addr))
return -EINVAL;
- pkt = kmalloc(sizeof(*pkt), GFP_KERNEL);
- if (!pkt)
- return -ENOMEM;
-
- err = __vmci_transport_send_control_pkt(pkt, &vsk->local_addr,
- &vsk->remote_addr, type, size,
- mode, wait, proto, handle,
- true);
- kfree(pkt);
-
- return err;
+ return vmci_transport_alloc_send_control_pkt(&vsk->local_addr,
+ &vsk->remote_addr,
+ type, size, mode,
+ wait, proto, handle);
}
static int vmci_transport_send_reset_bh(struct sockaddr_vm *dst,
@@ -321,12 +337,29 @@ static int vmci_transport_send_reset_bh(struct sockaddr_vm *dst,
static int vmci_transport_send_reset(struct sock *sk,
struct vmci_transport_packet *pkt)
{
+ struct sockaddr_vm *dst_ptr;
+ struct sockaddr_vm dst;
+ struct vsock_sock *vsk;
+
if (pkt->type == VMCI_TRANSPORT_PACKET_TYPE_RST)
return 0;
- return vmci_transport_send_control_pkt(sk,
- VMCI_TRANSPORT_PACKET_TYPE_RST,
- 0, 0, NULL, VSOCK_PROTO_INVALID,
- VMCI_INVALID_HANDLE);
+
+ vsk = vsock_sk(sk);
+
+ if (!vsock_addr_bound(&vsk->local_addr))
+ return -EINVAL;
+
+ if (vsock_addr_bound(&vsk->remote_addr)) {
+ dst_ptr = &vsk->remote_addr;
+ } else {
+ vsock_addr_init(&dst, pkt->dg.src.context,
+ pkt->src_port);
+ dst_ptr = &dst;
+ }
+ return vmci_transport_alloc_send_control_pkt(&vsk->local_addr, dst_ptr,
+ VMCI_TRANSPORT_PACKET_TYPE_RST,
+ 0, 0, NULL, VSOCK_PROTO_INVALID,
+ VMCI_INVALID_HANDLE);
}
static int vmci_transport_send_negotiate(struct sock *sk, size_t size)
@@ -1099,8 +1132,7 @@ static int vmci_transport_recv_listen(struct sock *sk,
vpending->listener = sk;
sock_hold(sk);
sock_hold(pending);
- INIT_DELAYED_WORK(&vpending->dwork, vsock_pending_work);
- schedule_delayed_work(&vpending->dwork, HZ);
+ schedule_delayed_work(&vpending->pending_work, HZ);
out:
return err;
diff --git a/net/wireless/core.c b/net/wireless/core.c
index 3181b07..c88874f 100644
--- a/net/wireless/core.c
+++ b/net/wireless/core.c
@@ -95,6 +95,9 @@ static int cfg80211_dev_check_name(struct cfg80211_registered_device *rdev,
ASSERT_RTNL();
+ if (strlen(newname) > NL80211_WIPHY_NAME_MAXLEN)
+ return -EINVAL;
+
/* prohibit calling the thing phy%d when %d is not its number */
sscanf(newname, PHY_NAME "%d%n", &wiphy_idx, &taken);
if (taken == strlen(newname) && wiphy_idx != rdev->wiphy_idx) {
diff --git a/net/wireless/nl80211.c b/net/wireless/nl80211.c
index e086950..d899a49 100644
--- a/net/wireless/nl80211.c
+++ b/net/wireless/nl80211.c
@@ -3448,6 +3448,7 @@ static bool ht_rateset_to_mask(struct ieee80211_supported_band *sband,
return false;
/* check availability */
+ ridx = array_index_nospec(ridx, IEEE80211_HT_MCS_MASK_LEN);
if (sband->ht_cap.mcs.rx_mask[ridx] & rbit)
mcs[ridx] |= rbit;
else
@@ -4110,6 +4111,7 @@ static int parse_station_flags(struct genl_info *info,
params->sta_flags_mask = BIT(NL80211_STA_FLAG_AUTHENTICATED) |
BIT(NL80211_STA_FLAG_MFP) |
BIT(NL80211_STA_FLAG_AUTHORIZED);
+ break;
default:
return -EINVAL;
}
@@ -5899,7 +5901,7 @@ do { \
nl80211_check_s32);
/*
* Check HT operation mode based on
- * IEEE 802.11 2012 8.4.2.59 HT Operation element.
+ * IEEE 802.11-2016 9.4.2.57 HT Operation element.
*/
if (tb[NL80211_MESHCONF_HT_OPMODE]) {
ht_opmode = nla_get_u16(tb[NL80211_MESHCONF_HT_OPMODE]);
@@ -5909,22 +5911,9 @@ do { \
IEEE80211_HT_OP_MODE_NON_HT_STA_PRSNT))
return -EINVAL;
- if ((ht_opmode & IEEE80211_HT_OP_MODE_NON_GF_STA_PRSNT) &&
- (ht_opmode & IEEE80211_HT_OP_MODE_NON_HT_STA_PRSNT))
- return -EINVAL;
+ /* NON_HT_STA bit is reserved, but some programs set it */
+ ht_opmode &= ~IEEE80211_HT_OP_MODE_NON_HT_STA_PRSNT;
- switch (ht_opmode & IEEE80211_HT_OP_MODE_PROTECTION) {
- case IEEE80211_HT_OP_MODE_PROTECTION_NONE:
- case IEEE80211_HT_OP_MODE_PROTECTION_20MHZ:
- if (ht_opmode & IEEE80211_HT_OP_MODE_NON_HT_STA_PRSNT)
- return -EINVAL;
- break;
- case IEEE80211_HT_OP_MODE_PROTECTION_NONMEMBER:
- case IEEE80211_HT_OP_MODE_PROTECTION_NONHT_MIXED:
- if (!(ht_opmode & IEEE80211_HT_OP_MODE_NON_HT_STA_PRSNT))
- return -EINVAL;
- break;
- }
cfg->ht_opmode = ht_opmode;
mask |= (1 << (NL80211_MESHCONF_HT_OPMODE - 1));
}
diff --git a/net/wireless/reg.c b/net/wireless/reg.c
index 9195f23..34adb9d 100644
--- a/net/wireless/reg.c
+++ b/net/wireless/reg.c
@@ -2324,6 +2324,7 @@ static int regulatory_hint_core(const char *alpha2)
request->alpha2[0] = alpha2[0];
request->alpha2[1] = alpha2[1];
request->initiator = NL80211_REGDOM_SET_BY_CORE;
+ request->wiphy_idx = WIPHY_IDX_INVALID;
queue_regulatory_request(request);
diff --git a/net/wireless/scan.c b/net/wireless/scan.c
index ad8611be1..7ea2eba 100644
--- a/net/wireless/scan.c
+++ b/net/wireless/scan.c
@@ -981,13 +981,23 @@ cfg80211_bss_update(struct cfg80211_registered_device *rdev,
return NULL;
}
+/*
+ * Update RX channel information based on the available frame payload
+ * information. This is mainly for the 2.4 GHz band where frames can be received
+ * from neighboring channels and the Beacon frames use the DSSS Parameter Set
+ * element to indicate the current (transmitting) channel, but this might also
+ * be needed on other bands if RX frequency does not match with the actual
+ * operating channel of a BSS.
+ */
static struct ieee80211_channel *
cfg80211_get_bss_channel(struct wiphy *wiphy, const u8 *ie, size_t ielen,
- struct ieee80211_channel *channel)
+ struct ieee80211_channel *channel,
+ enum nl80211_bss_scan_width scan_width)
{
const u8 *tmp;
u32 freq;
int channel_number = -1;
+ struct ieee80211_channel *alt_channel;
tmp = cfg80211_find_ie(WLAN_EID_DS_PARAMS, ie, ielen);
if (tmp && tmp[1] == 1) {
@@ -1001,16 +1011,45 @@ cfg80211_get_bss_channel(struct wiphy *wiphy, const u8 *ie, size_t ielen,
}
}
- if (channel_number < 0)
+ if (channel_number < 0) {
+ /* No channel information in frame payload */
return channel;
+ }
freq = ieee80211_channel_to_frequency(channel_number, channel->band);
- channel = ieee80211_get_channel(wiphy, freq);
- if (!channel)
+ alt_channel = ieee80211_get_channel(wiphy, freq);
+ if (!alt_channel) {
+ if (channel->band == NL80211_BAND_2GHZ) {
+ /*
+ * Better not allow unexpected channels when that could
+ * be going beyond the 1-11 range (e.g., discovering
+ * BSS on channel 12 when radio is configured for
+ * channel 11.
+ */
+ return NULL;
+ }
+
+ /* No match for the payload channel number - ignore it */
+ return channel;
+ }
+
+ if (scan_width == NL80211_BSS_CHAN_WIDTH_10 ||
+ scan_width == NL80211_BSS_CHAN_WIDTH_5) {
+ /*
+ * Ignore channel number in 5 and 10 MHz channels where there
+ * may not be an n:1 or 1:n mapping between frequencies and
+ * channel numbers.
+ */
+ return channel;
+ }
+
+ /*
+ * Use the channel determined through the payload channel number
+ * instead of the RX channel reported by the driver.
+ */
+ if (alt_channel->flags & IEEE80211_CHAN_DISABLED)
return NULL;
- if (channel->flags & IEEE80211_CHAN_DISABLED)
- return NULL;
- return channel;
+ return alt_channel;
}
/* Returned bss is reference counted and must be cleaned up appropriately. */
@@ -1035,7 +1074,8 @@ cfg80211_inform_bss_data(struct wiphy *wiphy,
(data->signal < 0 || data->signal > 100)))
return NULL;
- channel = cfg80211_get_bss_channel(wiphy, ie, ielen, data->chan);
+ channel = cfg80211_get_bss_channel(wiphy, ie, ielen, data->chan,
+ data->scan_width);
if (!channel)
return NULL;
@@ -1133,7 +1173,7 @@ cfg80211_inform_bss_frame_data(struct wiphy *wiphy,
return NULL;
channel = cfg80211_get_bss_channel(wiphy, mgmt->u.beacon.variable,
- ielen, data->chan);
+ ielen, data->chan, data->scan_width);
if (!channel)
return NULL;
diff --git a/net/wireless/util.c b/net/wireless/util.c
index 13ff407..c934189 100644
--- a/net/wireless/util.c
+++ b/net/wireless/util.c
@@ -1310,7 +1310,7 @@ bool ieee80211_chandef_to_operating_class(struct cfg80211_chan_def *chandef,
u8 *op_class)
{
u8 vht_opclass;
- u16 freq = chandef->center_freq1;
+ u32 freq = chandef->center_freq1;
if (freq >= 2412 && freq <= 2472) {
if (chandef->width > NL80211_CHAN_WIDTH_40)
diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
index 791ad6e..0bad618 100644
--- a/net/xfrm/xfrm_policy.c
+++ b/net/xfrm/xfrm_policy.c
@@ -659,9 +659,9 @@ static void xfrm_hash_rebuild(struct work_struct *work)
break;
}
if (newpos)
- hlist_add_behind(&policy->bydst, newpos);
+ hlist_add_behind_rcu(&policy->bydst, newpos);
else
- hlist_add_head(&policy->bydst, chain);
+ hlist_add_head_rcu(&policy->bydst, chain);
}
spin_unlock_bh(&net->xfrm.xfrm_policy_lock);
@@ -800,9 +800,9 @@ int xfrm_policy_insert(int dir, struct xfrm_policy *policy, int excl)
break;
}
if (newpos)
- hlist_add_behind(&policy->bydst, newpos);
+ hlist_add_behind_rcu(&policy->bydst, newpos);
else
- hlist_add_head(&policy->bydst, chain);
+ hlist_add_head_rcu(&policy->bydst, chain);
__xfrm_policy_link(policy, dir);
atomic_inc(&net->xfrm.flow_cache_genid);
@@ -1874,7 +1874,10 @@ xfrm_resolve_and_create_bundle(struct xfrm_policy **pols, int num_pols,
/* Try to instantiate a bundle */
err = xfrm_tmpl_resolve(pols, num_pols, fl, xfrm, family);
if (err <= 0) {
- if (err != 0 && err != -EAGAIN)
+ if (err == 0)
+ return NULL;
+
+ if (err != -EAGAIN)
XFRM_INC_STATS(net, LINUX_MIB_XFRMOUTPOLERROR);
return ERR_PTR(err);
}
@@ -2355,6 +2358,9 @@ struct dst_entry *xfrm_lookup_route(struct net *net, struct dst_entry *dst_orig,
if (IS_ERR(dst) && PTR_ERR(dst) == -EREMOTE)
return make_blackhole(net, dst_orig->ops->family, dst_orig);
+ if (IS_ERR(dst))
+ dst_release(dst_orig);
+
return dst;
}
EXPORT_SYMBOL(xfrm_lookup_route);
diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index d869b1d..5306e97 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -641,7 +641,7 @@ void xfrm_sad_getinfo(struct net *net, struct xfrmk_sadinfo *si)
{
spin_lock_bh(&net->xfrm.xfrm_state_lock);
si->sadcnt = net->xfrm.state_num;
- si->sadhcnt = net->xfrm.state_hmask;
+ si->sadhcnt = net->xfrm.state_hmask + 1;
si->sadhmcnt = xfrm_state_hashmax;
spin_unlock_bh(&net->xfrm.xfrm_state_lock);
}
@@ -1197,6 +1197,7 @@ static struct xfrm_state *xfrm_state_clone(struct xfrm_state *orig)
if (orig->aead) {
x->aead = xfrm_algo_aead_clone(orig->aead);
+ x->geniv = orig->geniv;
if (!x->aead)
goto error;
}
diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c
index 49b061b..2c92586 100644
--- a/net/xfrm/xfrm_user.c
+++ b/net/xfrm/xfrm_user.c
@@ -151,10 +151,16 @@ static int verify_newsa_info(struct xfrm_usersa_info *p,
err = -EINVAL;
switch (p->family) {
case AF_INET:
+ if (p->sel.prefixlen_d > 32 || p->sel.prefixlen_s > 32)
+ goto out;
+
break;
case AF_INET6:
#if IS_ENABLED(CONFIG_IPV6)
+ if (p->sel.prefixlen_d > 128 || p->sel.prefixlen_s > 128)
+ goto out;
+
break;
#else
err = -EAFNOSUPPORT;
@@ -992,10 +998,12 @@ static inline int xfrm_nlmsg_multicast(struct net *net, struct sk_buff *skb,
{
struct sock *nlsk = rcu_dereference(net->xfrm.nlsk);
- if (nlsk)
- return nlmsg_multicast(nlsk, skb, pid, group, GFP_ATOMIC);
- else
- return -1;
+ if (!nlsk) {
+ kfree_skb(skb);
+ return -EPIPE;
+ }
+
+ return nlmsg_multicast(nlsk, skb, pid, group, GFP_ATOMIC);
}
static inline size_t xfrm_spdinfo_msgsize(void)
@@ -1322,10 +1330,16 @@ static int verify_newpolicy_info(struct xfrm_userpolicy_info *p)
switch (p->sel.family) {
case AF_INET:
+ if (p->sel.prefixlen_d > 32 || p->sel.prefixlen_s > 32)
+ return -EINVAL;
+
break;
case AF_INET6:
#if IS_ENABLED(CONFIG_IPV6)
+ if (p->sel.prefixlen_d > 128 || p->sel.prefixlen_s > 128)
+ return -EINVAL;
+
break;
#else
return -EAFNOSUPPORT;
@@ -1639,9 +1653,11 @@ static inline size_t userpolicy_type_attrsize(void)
#ifdef CONFIG_XFRM_SUB_POLICY
static int copy_to_user_policy_type(u8 type, struct sk_buff *skb)
{
- struct xfrm_userpolicy_type upt = {
- .type = type,
- };
+ struct xfrm_userpolicy_type upt;
+
+ /* Sadly there are two holes in struct xfrm_userpolicy_type */
+ memset(&upt, 0, sizeof(upt));
+ upt.type = type;
return nla_put(skb, XFRMA_POLICY_TYPE, sizeof(upt), &upt);
}
diff --git a/samples/bpf/parse_varlen.c b/samples/bpf/parse_varlen.c
index 95c1632..0b6f22f 100644
--- a/samples/bpf/parse_varlen.c
+++ b/samples/bpf/parse_varlen.c
@@ -6,6 +6,7 @@
*/
#define KBUILD_MODNAME "foo"
#include <linux/if_ether.h>
+#include <linux/if_vlan.h>
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <linux/in.h>
@@ -108,11 +109,6 @@ static int parse_ipv6(void *data, uint64_t nh_off, void *data_end)
return 0;
}
-struct vlan_hdr {
- uint16_t h_vlan_TCI;
- uint16_t h_vlan_encapsulated_proto;
-};
-
SEC("varlen")
int handle_ingress(struct __sk_buff *skb)
{
diff --git a/samples/bpf/test_overhead_user.c b/samples/bpf/test_overhead_user.c
index d291167f..7dad9a3 100644
--- a/samples/bpf/test_overhead_user.c
+++ b/samples/bpf/test_overhead_user.c
@@ -6,6 +6,7 @@
*/
#define _GNU_SOURCE
#include <sched.h>
+#include <errno.h>
#include <stdio.h>
#include <sys/types.h>
#include <asm/unistd.h>
@@ -44,8 +45,13 @@ static void test_task_rename(int cpu)
exit(1);
}
start_time = time_get_ns();
- for (i = 0; i < MAX_CNT; i++)
- write(fd, buf, sizeof(buf));
+ for (i = 0; i < MAX_CNT; i++) {
+ if (write(fd, buf, sizeof(buf)) < 0) {
+ printf("task rename failed: %s\n", strerror(errno));
+ close(fd);
+ return;
+ }
+ }
printf("task_rename:%d: %lld events per sec\n",
cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
close(fd);
@@ -63,8 +69,13 @@ static void test_urandom_read(int cpu)
exit(1);
}
start_time = time_get_ns();
- for (i = 0; i < MAX_CNT; i++)
- read(fd, buf, sizeof(buf));
+ for (i = 0; i < MAX_CNT; i++) {
+ if (read(fd, buf, sizeof(buf)) < 0) {
+ printf("failed to read from /dev/urandom: %s\n", strerror(errno));
+ close(fd);
+ return;
+ }
+ }
printf("urandom_read:%d: %lld events per sec\n",
cpu, MAX_CNT * 1000000000ll / (time_get_ns() - start_time));
close(fd);
diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 0f101f7..8334276 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -8,6 +8,7 @@
empty :=
space := $(empty) $(empty)
space_escape := _-_SPACE_-_
+pound := \#
###
# Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
@@ -192,12 +193,13 @@
# cc-ldoption
# Usage: ldflags += $(call cc-ldoption, -Wl$(comma)--hash-style=both)
cc-ldoption = $(call try-run,\
- $(CC) $(1) -nostdlib -x c /dev/null -o "$$TMP",$(1),$(2))
+ $(CC) $(CLANG_TARGET) $(1) $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) -nostdlib -x c /dev/null -o "$$TMP",$(1),$(2))
# ld-option
# Usage: LDFLAGS += $(call ld-option, -X)
ld-option = $(call try-run,\
- $(CC) -x c /dev/null -c -o "$$TMPO" ; $(LD) $(1) "$$TMPO" -o "$$TMP",$(1),$(2))
+ $(CC) $(CLANG_TARGET) $(KBUILD_CPPFLAGS) $(CC_OPTION_CFLAGS) -x c /dev/null -c -o "$$TMPO"; \
+ $(LD) $(1) "$$TMPO" -o "$$TMP",$(1),$(2))
# ar-option
# Usage: KBUILD_ARFLAGS := $(call ar-option,D)
@@ -298,11 +300,11 @@
# Replace >$< with >$$< to preserve $ when reloading the .cmd file
# (needed for make)
-# Replace >#< with >\#< to avoid starting a comment in the .cmd file
+# Replace >#< with >$(pound)< to avoid starting a comment in the .cmd file
# (needed for make)
# Replace >'< with >'\''< to be able to enclose the whole string in '...'
# (needed for the shell)
-make-cmd = $(call escsq,$(subst \#,\\\#,$(subst $$,$$$$,$(cmd_$(1)))))
+make-cmd = $(call escsq,$(subst $(pound),$$(pound),$(subst $$,$$$$,$(cmd_$(1)))))
# Find any prerequisites that is newer than target or that does not exist.
# PHONY targets skipped in both cases.
@@ -450,3 +452,6 @@
endef
#
###############################################################################
+
+# delete partially updated (i.e. corrupted) files on error
+.DELETE_ON_ERROR:
diff --git a/scripts/Makefile.build b/scripts/Makefile.build
index a9c1da5..fcffce4 100644
--- a/scripts/Makefile.build
+++ b/scripts/Makefile.build
@@ -296,6 +296,9 @@
ifndef CONFIG_FRAME_POINTER
objtool_args += --no-fp
endif
+ifdef CONFIG_GCOV_KERNEL
+objtool_args += --no-unreachable
+endif
# 'OBJECT_FILES_NON_STANDARD := y': skip objtool checking for a directory
# 'OBJECT_FILES_NON_STANDARD_foo.o := 'y': skip objtool checking for a file
diff --git a/scripts/Makefile.dtboapply b/scripts/Makefile.dtboapply
new file mode 100644
index 0000000..87a5b30
--- /dev/null
+++ b/scripts/Makefile.dtboapply
@@ -0,0 +1,9 @@
+# Treble kerenl requires the SoC device tree and the device specific
+# device tree overlay to be built separately. Before the bootloader
+# supports applying overlay, the overlay step to form the dtb passed
+# to the linux kernel is done on the host side with this target.
+
+.PHONY: all
+
+all:
+ $(if $(KBUILD_SRC),$(srctree)/)scripts/dtbo_apply_overlay.sh $(obj)/${KERNEL_DTB} $(obj)/${DTBO}
\ No newline at end of file
diff --git a/scripts/Makefile.extrawarn b/scripts/Makefile.extrawarn
index fb3522f..d08b6fb 100644
--- a/scripts/Makefile.extrawarn
+++ b/scripts/Makefile.extrawarn
@@ -10,6 +10,8 @@
# are not supported by all versions of the compiler
# ==========================================================================
+KBUILD_CFLAGS += $(call cc-disable-warning, packed-not-aligned)
+
ifeq ("$(origin W)", "command line")
export KBUILD_ENABLE_EXTRA_GCC_CHECKS := $(W)
endif
@@ -25,6 +27,7 @@
warning-1 += $(call cc-option, -Wmissing-include-dirs)
warning-1 += $(call cc-option, -Wunused-but-set-variable)
warning-1 += $(call cc-option, -Wunused-const-variable)
+warning-1 += $(call cc-option, -Wpacked-not-aligned)
warning-1 += $(call cc-disable-warning, missing-field-initializers)
warning-1 += $(call cc-disable-warning, sign-compare)
diff --git a/scripts/Makefile.kasan b/scripts/Makefile.kasan
index 8c69cd1..d809e00 100644
--- a/scripts/Makefile.kasan
+++ b/scripts/Makefile.kasan
@@ -36,4 +36,7 @@
endif
endif
+
+CFLAGS_KASAN_NOSANITIZE := -fno-builtin
+
endif
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index ea98308..8f583b6 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -127,7 +127,7 @@
ifeq ($(CONFIG_KASAN),y)
_c_flags += $(if $(patsubst n%,, \
$(KASAN_SANITIZE_$(basetarget).o)$(KASAN_SANITIZE)y), \
- $(CFLAGS_KASAN))
+ $(CFLAGS_KASAN), $(CFLAGS_KASAN_NOSANITIZE))
endif
ifeq ($(CONFIG_UBSAN),y)
@@ -301,16 +301,10 @@
$(obj)/%.dtb.S: $(obj)/%.dtb
$(call cmd,dt_S_dtb)
-ifneq ($(DTC_EXT),)
-DTC = $(DTC_EXT)
-else
-DTC = $(objtree)/scripts/dtc/dtc
-endif
-
quiet_cmd_dtc = DTC $@
cmd_dtc = mkdir -p $(dir ${dtc-tmp}) ; \
$(CPP) $(dtc_cpp_flags) -x assembler-with-cpp -o $(dtc-tmp) $< ; \
- $(DTC) -O dtb -o $@ -b 0 \
+ $(DTC) -q -O dtb -o $@ -b 0 \
-i $(dir $<) $(DTC_FLAGS) \
-d $(depfile).dtc.tmp $(dtc-tmp) ; \
cat $(depfile).pre.tmp $(depfile).dtc.tmp > $(depfile)
@@ -318,8 +312,16 @@
$(obj)/%.dtb: $(src)/%.dts FORCE
$(call if_changed_dep,dtc)
+$(obj)/%.dtbo: $(src)/%.dts FORCE
+ $(call if_changed_dep,dtc)
+
dtc-tmp = $(subst $(comma),_,$(dot-target).dts.tmp)
+# mkdtimg
+#----------------------------------------------------------------------------
+quiet_cmd_mkdtimg = DTBOIMG $@
+cmd_mkdtimg = mkdtimg cfg_create $@ $<
+
# cat
# ---------------------------------------------------------------------------
# Concatentate multiple files together
@@ -366,7 +368,7 @@
quiet_cmd_lz4 = LZ4 $@
cmd_lz4 = (cat $(filter-out FORCE,$^) | \
- lz4c -l -c1 stdin stdout && $(call size_append, $(filter-out FORCE,$^))) > $@ || \
+ lz4c -9) > $@ || \
(rm -f $@ ; false)
# U-Boot mkimage
diff --git a/scripts/Makefile.modpost b/scripts/Makefile.modpost
index b7c50cb4..ea36c40 100644
--- a/scripts/Makefile.modpost
+++ b/scripts/Makefile.modpost
@@ -88,8 +88,10 @@
ifdef CONFIG_LTO_CLANG
quiet_cmd_cc_lto_link_modules = LD [M] $@
+bad_ld_flags = -plugin-opt=-function-sections -plugin-opt=-data-sections
+filtered_ld_flags = $(filter-out $(bad_ld_flags),$(ld_flags))
cmd_cc_lto_link_modules = \
- $(LD) $(ld_flags) -r -o $(@) \
+ $(LD) $(filtered_ld_flags) -r -o $(@) \
$(shell [ -s $(@:$(modpost-ext).o=.o.symversions) ] && \
echo -T $(@:$(modpost-ext).o=.o.symversions)) \
--whole-archive $(filter-out FORCE,$^)
@@ -136,8 +138,10 @@
quiet_cmd_ld_ko_o = LD [M] $@
ifdef CONFIG_LTO_CLANG
- cmd_ld_ko_o = \
- $(LD) -r $(LDFLAGS) \
+bad_ld_flags = -plugin-opt=-function-sections -plugin-opt=-data-sections
+filtered_ldflags = $(filter-out $(bad_ld_flags),$(LDFLAGS))
+cmd_ld_ko_o = \
+ $(LD) -r $(filtered_ld_flags) \
$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE) \
$(shell [ -s $(@:.ko=.o.symversions) ] && \
echo -T $(@:.ko=.o.symversions)) \
diff --git a/scripts/adjust_autoksyms.sh b/scripts/adjust_autoksyms.sh
index 8dc1918..564db35 100755
--- a/scripts/adjust_autoksyms.sh
+++ b/scripts/adjust_autoksyms.sh
@@ -83,6 +83,13 @@
depfile="include/config/ksym/${sympath}.h"
mkdir -p "$(dirname "$depfile")"
touch "$depfile"
+ # Filesystems with coarse time precision may create timestamps
+ # equal to the one from a file that was very recently built and that
+ # needs to be rebuild. Let's guard against that by making sure our
+ # dep files are always newer than the first file we created here.
+ while [ ! "$depfile" -nt "$new_ksyms_file" ]; do
+ touch "$depfile"
+ done
echo $((count += 1))
done | tail -1 )
changed=${changed:-0}
diff --git a/scripts/checkstack.pl b/scripts/checkstack.pl
index dd83978..12a6940 100755
--- a/scripts/checkstack.pl
+++ b/scripts/checkstack.pl
@@ -46,8 +46,8 @@
$xs = "[0-9a-f ]"; # hex character or space
$funcre = qr/^$x* <(.*)>:$/;
if ($arch eq 'aarch64') {
- #ffffffc0006325cc: a9bb7bfd stp x29, x30, [sp,#-80]!
- $re = qr/^.*stp.*sp,\#-([0-9]{1,8})\]\!/o;
+ #ffffffc0006325cc: a9bb7bfd stp x29, x30, [sp, #-80]!
+ $re = qr/^.*stp.*sp, \#-([0-9]{1,8})\]\!/o;
} elsif ($arch eq 'arm') {
#c0008ffc: e24dd064 sub sp, sp, #100 ; 0x64
$re = qr/.*sub.*sp, sp, #(([0-9]{2}|[3-9])[0-9]{2})/o;
diff --git a/scripts/depmod.sh b/scripts/depmod.sh
index 122599b..baedaef 100755
--- a/scripts/depmod.sh
+++ b/scripts/depmod.sh
@@ -10,7 +10,13 @@
KERNELRELEASE=$2
SYMBOL_PREFIX=$3
-if ! test -r System.map -a -x "$DEPMOD"; then
+if ! test -r System.map ; then
+ exit 0
+fi
+
+if [ -z $(command -v $DEPMOD) ]; then
+ echo "Warning: 'make modules_install' requires $DEPMOD. Please install it." >&2
+ echo "This is probably in the kmod package." >&2
exit 0
fi
diff --git a/scripts/dtbo_apply_overlay.sh b/scripts/dtbo_apply_overlay.sh
new file mode 100755
index 0000000..339e94d
--- /dev/null
+++ b/scripts/dtbo_apply_overlay.sh
@@ -0,0 +1,56 @@
+#!/bin/bash
+
+# Treble kerenl requires the SoC device tree and the device specific
+# device tree overlay to be built separately. Before the bootloader
+# supports applying overlay, the overlay step to form the dtb passed
+# to the linux kernel is done on the host side with this script.
+
+set -x
+
+# prepare overlay workspace
+overlay_dir=`mktemp -d -t overlay.XXXXXXXXXX`
+TOP=${PWD}
+KERNEL_DTB=${TOP}/$1
+DTBO=${TOP}/$2
+
+trap "rm -rf ${overlay_dir}; set +x; exit" SIGINT
+
+cd ${overlay_dir}
+
+# extract soc dtb
+extract_dtb ${KERNEL_DTB} soc.dtb Image.lz4
+
+soc_cnt=$(ls -l soc.dtb* | wc -l)
+if [ ${soc_cnt} -ne 1 ]; then
+ echo "Error: ${soc_cnt} soc.dtb(s) appended to Image.lz4-dtb" >&2
+ exit 1
+fi
+
+# extract dtbo, rev and id
+mkdtimg dump ${DTBO} -b overlay.dtbo > dtbo.info
+id_arr=(`grep dtbo.info -e "id" | sed 's/.*id = \([0-9a-f]\+\)/\1/'`)
+rev_arr=(`grep dtbo.info -e "rev" | sed 's/.*rev = \([0-9a-f]\+\)/\1/'`)
+
+# add number if there's only one dtbo entry
+if [ ${#id_arr[*]} -eq 1 ]; then
+ mv overlay.dtbo overlay.dtbo.0
+fi
+
+for idx in ${!id_arr[*]}; do
+ echo "apply overlay for device id=${id_arr[$idx]} rev=${rev_arr[$idx]}"
+ ufdt_apply_overlay soc.dtb overlay.dtbo.${idx} combined-${idx}.dtb
+ dtc -q -O dts -o combined-${idx}.dts combined-${idx}.dtb
+ echo "/{qcom,board-id=<0x${id_arr[$idx]} 0x${rev_arr[$idx]}>;};" >> \
+ combined-${idx}.dts
+ dtc -q -O dtb -o combined-${idx}.dtb combined-${idx}.dts
+done
+
+# ls -v is used to make sure dtb is combined as the order specified in
+# dtboimg.cfg, where muskie/walley dtbs are before taimen dtbs. Otherwise,
+# muskie/walleye's bootloader may give up looking for dtbs once it sees
+# an invalid dtb, i.e. taimen dtb.
+cat Image.lz4 `ls -v combined-*.dtb` > Image.lz4-dtb
+cp Image.lz4-dtb ${KERNEL_DTB}
+
+rm -rf ${overlay_dir}
+set +x
diff --git a/scripts/dtc/Makefile b/scripts/dtc/Makefile
index 2eb4aec..2a48022 100644
--- a/scripts/dtc/Makefile
+++ b/scripts/dtc/Makefile
@@ -1,9 +1,7 @@
# scripts/dtc makefile
hostprogs-y := dtc
-ifeq ($(DTC_EXT),)
always := $(hostprogs-y)
-endif
dtc-objs := dtc.o flattree.o fstree.o data.o livetree.o treesource.o \
srcpos.o checks.o util.o
diff --git a/scripts/kconfig/confdata.c b/scripts/kconfig/confdata.c
index 297b079..27aac27 100644
--- a/scripts/kconfig/confdata.c
+++ b/scripts/kconfig/confdata.c
@@ -745,7 +745,7 @@ int conf_write(const char *name)
struct menu *menu;
const char *basename;
const char *str;
- char dirname[PATH_MAX+1], tmpname[PATH_MAX+1], newname[PATH_MAX+1];
+ char dirname[PATH_MAX+1], tmpname[PATH_MAX+22], newname[PATH_MAX+8];
char *env;
dirname[0] = 0;
diff --git a/scripts/kconfig/expr.c b/scripts/kconfig/expr.c
index cbf4996..ed29bad 100644
--- a/scripts/kconfig/expr.c
+++ b/scripts/kconfig/expr.c
@@ -113,7 +113,7 @@ void expr_free(struct expr *e)
break;
case E_NOT:
expr_free(e->left.expr);
- return;
+ break;
case E_EQUAL:
case E_GEQ:
case E_GTH:
diff --git a/scripts/kconfig/menu.c b/scripts/kconfig/menu.c
index aed678e..4a61636 100644
--- a/scripts/kconfig/menu.c
+++ b/scripts/kconfig/menu.c
@@ -364,6 +364,7 @@ void menu_finalize(struct menu *parent)
menu->parent = parent;
last_menu = menu;
}
+ expr_free(basedep);
if (last_menu) {
parent->list = parent->next;
parent->next = last_menu->next;
diff --git a/scripts/kconfig/zconf.y b/scripts/kconfig/zconf.y
index 71bf8bf..5122ed2 100644
--- a/scripts/kconfig/zconf.y
+++ b/scripts/kconfig/zconf.y
@@ -107,7 +107,27 @@
%%
input: nl start | start;
-start: mainmenu_stmt stmt_list | stmt_list;
+start: mainmenu_stmt stmt_list | no_mainmenu_stmt stmt_list;
+
+/* mainmenu entry */
+
+mainmenu_stmt: T_MAINMENU prompt nl
+{
+ menu_add_prompt(P_MENU, $2, NULL);
+};
+
+/* Default main menu, if there's no mainmenu entry */
+
+no_mainmenu_stmt: /* empty */
+{
+ /*
+ * Hack: Keep the main menu title on the heap so we can safely free it
+ * later regardless of whether it comes from the 'prompt' in
+ * mainmenu_stmt or here
+ */
+ menu_add_prompt(P_MENU, strdup("Linux Kernel Configuration"), NULL);
+};
+
stmt_list:
/* empty */
@@ -344,13 +364,6 @@
| if_block choice_stmt
;
-/* mainmenu entry */
-
-mainmenu_stmt: T_MAINMENU prompt nl
-{
- menu_add_prompt(P_MENU, $2, NULL);
-};
-
/* menu entry */
menu: T_MENU prompt T_EOL
@@ -495,6 +508,7 @@
void conf_parse(const char *name)
{
+ const char *tmp;
struct symbol *sym;
int i;
@@ -502,7 +516,6 @@
sym_init();
_menu_init();
- rootmenu.prompt = menu_add_prompt(P_MENU, "Linux Kernel Configuration", NULL);
if (getenv("ZCONF_DEBUG"))
zconfdebug = 1;
@@ -512,8 +525,10 @@
if (!modules_sym)
modules_sym = sym_find( "n" );
+ tmp = rootmenu.prompt->text;
rootmenu.prompt->text = _(rootmenu.prompt->text);
rootmenu.prompt->text = sym_expand_string_value(rootmenu.prompt->text);
+ free((char*)tmp);
menu_finalize(&rootmenu);
for_all_symbols(i, sym) {
diff --git a/scripts/mod/modpost.c b/scripts/mod/modpost.c
index 238db4f..e5c4114 100644
--- a/scripts/mod/modpost.c
+++ b/scripts/mod/modpost.c
@@ -649,7 +649,7 @@ static void handle_modversions(struct module *mod, struct elf_info *info,
if (ELF_ST_TYPE(sym->st_info) == STT_SPARC_REGISTER)
break;
if (symname[0] == '.') {
- char *munged = strdup(symname);
+ char *munged = NOFAIL(strdup(symname));
munged[0] = '_';
munged[1] = toupper(munged[1]);
symname = munged;
@@ -936,6 +936,7 @@ static const char *const head_sections[] = { ".head.text*", NULL };
static const char *const linker_symbols[] =
{ "__init_begin", "_sinittext", "_einittext", NULL };
static const char *const optim_symbols[] = { "*.constprop.*", NULL };
+static const char *const cfi_symbols[] = { "*.cfi", NULL };
enum mismatch {
TEXT_TO_ANY_INIT,
@@ -1157,6 +1158,16 @@ static const struct sectioncheck *section_mismatch(
* fromsec = text section
* refsymname = *.constprop.*
*
+ * Pattern 6:
+ * With CONFIG_CFI_CLANG, clang appends .cfi to all indirectly called
+ * functions and creates a function stub with the original name. This
+ * stub is always placed in .text, even if the actual function with the
+ * .cfi postfix is in .init.text or .exit.text.
+ * This pattern is identified by
+ * tosec = init or exit section
+ * fromsec = text section
+ * tosym = *.cfi
+ *
**/
static int secref_whitelist(const struct sectioncheck *mismatch,
const char *fromsec, const char *fromsym,
@@ -1195,9 +1206,39 @@ static int secref_whitelist(const struct sectioncheck *mismatch,
match(fromsym, optim_symbols))
return 0;
+ /* Check for pattern 6 */
+ if (match(fromsec, text_sections) &&
+ match(tosec, init_exit_sections) &&
+ match(tosym, cfi_symbols))
+ return 0;
+
return 1;
}
+static inline int is_arm_mapping_symbol(const char *str)
+{
+ return str[0] == '$' && strchr("axtd", str[1])
+ && (str[2] == '\0' || str[2] == '.');
+}
+
+/*
+ * If there's no name there, ignore it; likewise, ignore it if it's
+ * one of the magic symbols emitted used by current ARM tools.
+ *
+ * Otherwise if find_symbols_between() returns those symbols, they'll
+ * fail the whitelist tests and cause lots of false alarms ... fixable
+ * only by merging __exit and __init sections into __text, bloating
+ * the kernel (which is especially evil on embedded platforms).
+ */
+static inline int is_valid_name(struct elf_info *elf, Elf_Sym *sym)
+{
+ const char *name = elf->strtab + sym->st_name;
+
+ if (!name || !strlen(name))
+ return 0;
+ return !is_arm_mapping_symbol(name);
+}
+
/**
* Find symbol based on relocation record info.
* In some cases the symbol supplied is a valid symbol so
@@ -1223,6 +1264,8 @@ static Elf_Sym *find_elf_symbol(struct elf_info *elf, Elf64_Sword addr,
continue;
if (ELF_ST_TYPE(sym->st_info) == STT_SECTION)
continue;
+ if (!is_valid_name(elf, sym))
+ continue;
if (sym->st_value == addr)
return sym;
/* Find a symbol nearby - addr are maybe negative */
@@ -1241,30 +1284,6 @@ static Elf_Sym *find_elf_symbol(struct elf_info *elf, Elf64_Sword addr,
return NULL;
}
-static inline int is_arm_mapping_symbol(const char *str)
-{
- return str[0] == '$' && strchr("axtd", str[1])
- && (str[2] == '\0' || str[2] == '.');
-}
-
-/*
- * If there's no name there, ignore it; likewise, ignore it if it's
- * one of the magic symbols emitted used by current ARM tools.
- *
- * Otherwise if find_symbols_between() returns those symbols, they'll
- * fail the whitelist tests and cause lots of false alarms ... fixable
- * only by merging __exit and __init sections into __text, bloating
- * the kernel (which is especially evil on embedded platforms).
- */
-static inline int is_valid_name(struct elf_info *elf, Elf_Sym *sym)
-{
- const char *name = elf->strtab + sym->st_name;
-
- if (!name || !strlen(name))
- return 0;
- return !is_arm_mapping_symbol(name);
-}
-
/*
* Find symbols before or equal addr and after addr - in the section sec.
* If we find two symbols with equal offset prefer one with a valid name.
@@ -1312,7 +1331,7 @@ static Elf_Sym *find_elf_symbol2(struct elf_info *elf, Elf_Addr addr,
static char *sec2annotation(const char *s)
{
if (match(s, init_exit_sections)) {
- char *p = malloc(20);
+ char *p = NOFAIL(malloc(20));
char *r = p;
*p++ = '_';
@@ -1332,7 +1351,7 @@ static char *sec2annotation(const char *s)
strcat(p, " ");
return r;
} else {
- return strdup("");
+ return NOFAIL(strdup(""));
}
}
@@ -2033,7 +2052,7 @@ void buf_write(struct buffer *buf, const char *s, int len)
{
if (buf->size - buf->pos < len) {
buf->size += len + SZ;
- buf->p = realloc(buf->p, buf->size);
+ buf->p = NOFAIL(realloc(buf->p, buf->size));
}
strncpy(buf->p + buf->pos, s, len);
buf->pos += len;
diff --git a/scripts/package/builddeb b/scripts/package/builddeb
index 3c575cd0..0a2a737 100755
--- a/scripts/package/builddeb
+++ b/scripts/package/builddeb
@@ -325,7 +325,7 @@
# Build kernel header package
(cd $srctree; find . -name Makefile\* -o -name Kconfig\* -o -name \*.pl) > "$objtree/debian/hdrsrcfiles"
-(cd $srctree; find arch/*/include include scripts -type f) >> "$objtree/debian/hdrsrcfiles"
+(cd $srctree; find arch/*/include include scripts -type f -o -type l) >> "$objtree/debian/hdrsrcfiles"
(cd $srctree; find arch/$SRCARCH -name module.lds -o -name Kbuild.platforms -o -name Platform) >> "$objtree/debian/hdrsrcfiles"
(cd $srctree; find $(find arch/$SRCARCH -name include -o -name scripts -type d) -type f) >> "$objtree/debian/hdrsrcfiles"
if grep -q '^CONFIG_STACK_VALIDATION=y' $KCONFIG_CONFIG ; then
diff --git a/scripts/setlocalversion b/scripts/setlocalversion
index 966dd39..298c170 100755
--- a/scripts/setlocalversion
+++ b/scripts/setlocalversion
@@ -28,16 +28,11 @@
usage
fi
-scm_version()
+_scm_version()
{
local short
short=false
- cd "$srctree"
- if test -e .scmversion; then
- cat .scmversion
- return
- fi
if test "$1" = "--short"; then
short=true
fi
@@ -115,6 +110,33 @@
fi
}
+scm_version()
+{
+ local git_path res git_all_proj name
+ cd "$srctree"
+ if test -e .scmversion; then
+ cat .scmversion
+ return
+ fi
+ git_all_proj=$(find . -name .git -type d | sort)
+ if [ ! -z "$git_all_proj" ] ; then
+ res=""
+ for git_path in $git_all_proj; do
+ git_path=${git_path%.git}
+ name=$(basename ${git_path})
+ if [ $name = "." ] ; then
+ name=""
+ else
+ name="_$name"
+ fi
+ res="$res$name$(cd $git_path;_scm_version)"
+ done
+ printf -- '%s' "$res"
+ else
+ _scm_version
+ fi
+}
+
collect_files()
{
local file res
diff --git a/scripts/unifdef.c b/scripts/unifdef.c
index 7493c0e..db00e3e3 100644
--- a/scripts/unifdef.c
+++ b/scripts/unifdef.c
@@ -395,7 +395,7 @@ usage(void)
* When we have processed a group that starts off with a known-false
* #if/#elif sequence (which has therefore been deleted) followed by a
* #elif that we don't understand and therefore must keep, we edit the
- * latter into a #if to keep the nesting correct. We use strncpy() to
+ * latter into a #if to keep the nesting correct. We use memcpy() to
* overwrite the 4 byte token "elif" with "if " without a '\0' byte.
*
* When we find a true #elif in a group, the following block will
@@ -450,7 +450,7 @@ static void Idrop (void) { Fdrop(); ignoreon(); }
static void Itrue (void) { Ftrue(); ignoreon(); }
static void Ifalse(void) { Ffalse(); ignoreon(); }
/* modify this line */
-static void Mpass (void) { strncpy(keyword, "if ", 4); Pelif(); }
+static void Mpass (void) { memcpy(keyword, "if ", 4); Pelif(); }
static void Mtrue (void) { keywordedit("else"); state(IS_TRUE_MIDDLE); }
static void Melif (void) { keywordedit("endif"); state(IS_FALSE_TRAILER); }
static void Melse (void) { keywordedit("endif"); state(IS_FALSE_ELSE); }
diff --git a/security/integrity/digsig.c b/security/integrity/digsig.c
index 4304372..95433ac 100644
--- a/security/integrity/digsig.c
+++ b/security/integrity/digsig.c
@@ -18,6 +18,7 @@
#include <linux/cred.h>
#include <linux/key-type.h>
#include <linux/digsig.h>
+#include <linux/vmalloc.h>
#include <crypto/public_key.h>
#include <keys/system_keyring.h>
diff --git a/security/integrity/evm/evm.h b/security/integrity/evm/evm.h
index f5f1272..2ff0245 100644
--- a/security/integrity/evm/evm.h
+++ b/security/integrity/evm/evm.h
@@ -48,7 +48,7 @@ int evm_calc_hmac(struct dentry *dentry, const char *req_xattr_name,
size_t req_xattr_value_len, char *digest);
int evm_calc_hash(struct dentry *dentry, const char *req_xattr_name,
const char *req_xattr_value,
- size_t req_xattr_value_len, char *digest);
+ size_t req_xattr_value_len, char type, char *digest);
int evm_init_hmac(struct inode *inode, const struct xattr *xattr,
char *hmac_val);
int evm_init_secfs(void);
diff --git a/security/integrity/evm/evm_crypto.c b/security/integrity/evm/evm_crypto.c
index bf66391..c783fef 100644
--- a/security/integrity/evm/evm_crypto.c
+++ b/security/integrity/evm/evm_crypto.c
@@ -94,7 +94,8 @@ static struct shash_desc *init_desc(char type)
mutex_lock(&mutex);
if (*tfm)
goto out;
- *tfm = crypto_alloc_shash(algo, 0, CRYPTO_ALG_ASYNC);
+ *tfm = crypto_alloc_shash(algo, 0,
+ CRYPTO_ALG_ASYNC | CRYPTO_NOLOAD);
if (IS_ERR(*tfm)) {
rc = PTR_ERR(*tfm);
pr_err("Can not allocate %s (reason: %ld)\n", algo, rc);
@@ -138,7 +139,7 @@ static struct shash_desc *init_desc(char type)
* protection.)
*/
static void hmac_add_misc(struct shash_desc *desc, struct inode *inode,
- char *digest)
+ char type, char *digest)
{
struct h_misc {
unsigned long ino;
@@ -149,13 +150,27 @@ static void hmac_add_misc(struct shash_desc *desc, struct inode *inode,
} hmac_misc;
memset(&hmac_misc, 0, sizeof(hmac_misc));
- hmac_misc.ino = inode->i_ino;
- hmac_misc.generation = inode->i_generation;
- hmac_misc.uid = from_kuid(inode->i_sb->s_user_ns, inode->i_uid);
- hmac_misc.gid = from_kgid(inode->i_sb->s_user_ns, inode->i_gid);
+ /* Don't include the inode or generation number in portable
+ * signatures
+ */
+ if (type != EVM_XATTR_PORTABLE_DIGSIG) {
+ hmac_misc.ino = inode->i_ino;
+ hmac_misc.generation = inode->i_generation;
+ }
+ /* The hmac uid and gid must be encoded in the initial user
+ * namespace (not the filesystems user namespace) as encoding
+ * them in the filesystems user namespace allows an attack
+ * where first they are written in an unprivileged fuse mount
+ * of a filesystem and then the system is tricked to mount the
+ * filesystem for real on next boot and trust it because
+ * everything is signed.
+ */
+ hmac_misc.uid = from_kuid(&init_user_ns, inode->i_uid);
+ hmac_misc.gid = from_kgid(&init_user_ns, inode->i_gid);
hmac_misc.mode = inode->i_mode;
crypto_shash_update(desc, (const u8 *)&hmac_misc, sizeof(hmac_misc));
- if (evm_hmac_attrs & EVM_ATTR_FSUUID)
+ if ((evm_hmac_attrs & EVM_ATTR_FSUUID) &&
+ type != EVM_XATTR_PORTABLE_DIGSIG)
crypto_shash_update(desc, inode->i_sb->s_uuid,
sizeof(inode->i_sb->s_uuid));
crypto_shash_final(desc, digest);
@@ -181,6 +196,7 @@ static int evm_calc_hmac_or_hash(struct dentry *dentry,
char *xattr_value = NULL;
int error;
int size;
+ bool ima_present = false;
if (!(inode->i_opflags & IOP_XATTR))
return -EOPNOTSUPP;
@@ -191,11 +207,18 @@ static int evm_calc_hmac_or_hash(struct dentry *dentry,
error = -ENODATA;
for (xattrname = evm_config_xattrnames; *xattrname != NULL; xattrname++) {
+ bool is_ima = false;
+
+ if (strcmp(*xattrname, XATTR_NAME_IMA) == 0)
+ is_ima = true;
+
if ((req_xattr_name && req_xattr_value)
&& !strcmp(*xattrname, req_xattr_name)) {
error = 0;
crypto_shash_update(desc, (const u8 *)req_xattr_value,
req_xattr_value_len);
+ if (is_ima)
+ ima_present = true;
continue;
}
size = vfs_getxattr_alloc(dentry, *xattrname,
@@ -210,9 +233,14 @@ static int evm_calc_hmac_or_hash(struct dentry *dentry,
error = 0;
xattr_size = size;
crypto_shash_update(desc, (const u8 *)xattr_value, xattr_size);
+ if (is_ima)
+ ima_present = true;
}
- hmac_add_misc(desc, inode, digest);
+ hmac_add_misc(desc, inode, type, digest);
+ /* Portable EVM signatures must include an IMA hash */
+ if (type == EVM_XATTR_PORTABLE_DIGSIG && !ima_present)
+ return -EPERM;
out:
kfree(xattr_value);
kfree(desc);
@@ -224,17 +252,45 @@ int evm_calc_hmac(struct dentry *dentry, const char *req_xattr_name,
char *digest)
{
return evm_calc_hmac_or_hash(dentry, req_xattr_name, req_xattr_value,
- req_xattr_value_len, EVM_XATTR_HMAC, digest);
+ req_xattr_value_len, EVM_XATTR_HMAC, digest);
}
int evm_calc_hash(struct dentry *dentry, const char *req_xattr_name,
const char *req_xattr_value, size_t req_xattr_value_len,
- char *digest)
+ char type, char *digest)
{
return evm_calc_hmac_or_hash(dentry, req_xattr_name, req_xattr_value,
- req_xattr_value_len, IMA_XATTR_DIGEST, digest);
+ req_xattr_value_len, type, digest);
}
+static int evm_is_immutable(struct dentry *dentry, struct inode *inode)
+{
+ const struct evm_ima_xattr_data *xattr_data = NULL;
+ struct integrity_iint_cache *iint;
+ int rc = 0;
+
+ iint = integrity_iint_find(inode);
+ if (iint && (iint->flags & EVM_IMMUTABLE_DIGSIG))
+ return 1;
+
+ /* Do this the hard way */
+ rc = vfs_getxattr_alloc(dentry, XATTR_NAME_EVM, (char **)&xattr_data, 0,
+ GFP_NOFS);
+ if (rc <= 0) {
+ if (rc == -ENODATA)
+ return 0;
+ return rc;
+ }
+ if (xattr_data->type == EVM_XATTR_PORTABLE_DIGSIG)
+ rc = 1;
+ else
+ rc = 0;
+
+ kfree(xattr_data);
+ return rc;
+}
+
+
/*
* Calculate the hmac and update security.evm xattr
*
@@ -247,6 +303,16 @@ int evm_update_evmxattr(struct dentry *dentry, const char *xattr_name,
struct evm_ima_xattr_data xattr_data;
int rc = 0;
+ /*
+ * Don't permit any transformation of the EVM xattr if the signature
+ * is of an immutable type
+ */
+ rc = evm_is_immutable(dentry, inode);
+ if (rc < 0)
+ return rc;
+ if (rc)
+ return -EPERM;
+
rc = evm_calc_hmac(dentry, xattr_name, xattr_value,
xattr_value_len, xattr_data.digest);
if (rc == 0) {
@@ -272,7 +338,7 @@ int evm_init_hmac(struct inode *inode, const struct xattr *lsm_xattr,
}
crypto_shash_update(desc, lsm_xattr->value, lsm_xattr->value_len);
- hmac_add_misc(desc, inode, hmac_val);
+ hmac_add_misc(desc, inode, EVM_XATTR_HMAC, hmac_val);
kfree(desc);
return 0;
}
diff --git a/security/integrity/evm/evm_main.c b/security/integrity/evm/evm_main.c
index ba86155..976b8dc 100644
--- a/security/integrity/evm/evm_main.c
+++ b/security/integrity/evm/evm_main.c
@@ -29,7 +29,7 @@
int evm_initialized;
static char *integrity_status_msg[] = {
- "pass", "fail", "no_label", "no_xattrs", "unknown"
+ "pass", "pass_immutable", "fail", "no_label", "no_xattrs", "unknown"
};
char *evm_hmac = "hmac(sha1)";
char *evm_hash = "sha1";
@@ -118,7 +118,8 @@ static enum integrity_status evm_verify_hmac(struct dentry *dentry,
enum integrity_status evm_status = INTEGRITY_PASS;
int rc, xattr_len;
- if (iint && iint->evm_status == INTEGRITY_PASS)
+ if (iint && (iint->evm_status == INTEGRITY_PASS ||
+ iint->evm_status == INTEGRITY_PASS_IMMUTABLE))
return iint->evm_status;
/* if status is not PASS, try to check again - against -ENOMEM */
@@ -155,22 +156,26 @@ static enum integrity_status evm_verify_hmac(struct dentry *dentry,
rc = -EINVAL;
break;
case EVM_IMA_XATTR_DIGSIG:
+ case EVM_XATTR_PORTABLE_DIGSIG:
rc = evm_calc_hash(dentry, xattr_name, xattr_value,
- xattr_value_len, calc.digest);
+ xattr_value_len, xattr_data->type,
+ calc.digest);
if (rc)
break;
rc = integrity_digsig_verify(INTEGRITY_KEYRING_EVM,
(const char *)xattr_data, xattr_len,
calc.digest, sizeof(calc.digest));
if (!rc) {
- /* Replace RSA with HMAC if not mounted readonly and
- * not immutable
- */
- if (!IS_RDONLY(d_backing_inode(dentry)) &&
- !IS_IMMUTABLE(d_backing_inode(dentry)))
+ if (xattr_data->type == EVM_XATTR_PORTABLE_DIGSIG) {
+ if (iint)
+ iint->flags |= EVM_IMMUTABLE_DIGSIG;
+ evm_status = INTEGRITY_PASS_IMMUTABLE;
+ } else if (!IS_RDONLY(d_backing_inode(dentry)) &&
+ !IS_IMMUTABLE(d_backing_inode(dentry))) {
evm_update_evmxattr(dentry, xattr_name,
xattr_value,
xattr_value_len);
+ }
}
break;
default:
@@ -271,7 +276,7 @@ static enum integrity_status evm_verify_current_integrity(struct dentry *dentry)
* affect security.evm. An interesting side affect of writing posix xattr
* acls is their modifying of the i_mode, which is included in security.evm.
* For posix xattr acls only, permit security.evm, even if it currently
- * doesn't exist, to be updated.
+ * doesn't exist, to be updated unless the EVM signature is immutable.
*/
static int evm_protect_xattr(struct dentry *dentry, const char *xattr_name,
const void *xattr_value, size_t xattr_value_len)
@@ -339,7 +344,8 @@ int evm_inode_setxattr(struct dentry *dentry, const char *xattr_name,
if (strcmp(xattr_name, XATTR_NAME_EVM) == 0) {
if (!xattr_value_len)
return -EINVAL;
- if (xattr_data->type != EVM_IMA_XATTR_DIGSIG)
+ if (xattr_data->type != EVM_IMA_XATTR_DIGSIG &&
+ xattr_data->type != EVM_XATTR_PORTABLE_DIGSIG)
return -EPERM;
}
return evm_protect_xattr(dentry, xattr_name, xattr_value,
@@ -416,6 +422,9 @@ void evm_inode_post_removexattr(struct dentry *dentry, const char *xattr_name)
/**
* evm_inode_setattr - prevent updating an invalid EVM extended attribute
* @dentry: pointer to the affected dentry
+ *
+ * Permit update of file attributes when files have a valid EVM signature,
+ * except in the case of them having an immutable portable signature.
*/
int evm_inode_setattr(struct dentry *dentry, struct iattr *attr)
{
diff --git a/security/integrity/iint.c b/security/integrity/iint.c
index c710d22..7ea39b1 100644
--- a/security/integrity/iint.c
+++ b/security/integrity/iint.c
@@ -74,6 +74,7 @@ static void iint_free(struct integrity_iint_cache *iint)
iint->ima_hash = NULL;
iint->version = 0;
iint->flags = 0UL;
+ iint->atomic_flags = 0UL;
iint->ima_file_status = INTEGRITY_UNKNOWN;
iint->ima_mmap_status = INTEGRITY_UNKNOWN;
iint->ima_bprm_status = INTEGRITY_UNKNOWN;
@@ -155,12 +156,14 @@ static void init_once(void *foo)
memset(iint, 0, sizeof(*iint));
iint->version = 0;
iint->flags = 0UL;
+ iint->atomic_flags = 0;
iint->ima_file_status = INTEGRITY_UNKNOWN;
iint->ima_mmap_status = INTEGRITY_UNKNOWN;
iint->ima_bprm_status = INTEGRITY_UNKNOWN;
iint->ima_read_status = INTEGRITY_UNKNOWN;
iint->evm_status = INTEGRITY_UNKNOWN;
iint->measured_pcrs = 0;
+ mutex_init(&iint->mutex);
}
static int __init integrity_iintcache_init(void)
diff --git a/security/integrity/ima/ima_api.c b/security/integrity/ima/ima_api.c
index d01a52f..3b43057 100644
--- a/security/integrity/ima/ima_api.c
+++ b/security/integrity/ima/ima_api.c
@@ -198,42 +198,59 @@ int ima_collect_measurement(struct integrity_iint_cache *iint,
struct inode *inode = file_inode(file);
const char *filename = file->f_path.dentry->d_name.name;
int result = 0;
+ int length;
+ void *tmpbuf;
+ u64 i_version;
struct {
struct ima_digest_data hdr;
char digest[IMA_MAX_DIGEST_SIZE];
} hash;
- if (!(iint->flags & IMA_COLLECTED)) {
- u64 i_version = file_inode(file)->i_version;
+ if (iint->flags & IMA_COLLECTED)
+ goto out;
- if (file->f_flags & O_DIRECT) {
- audit_cause = "failed(directio)";
- result = -EACCES;
- goto out;
- }
+ /*
+ * Dectecting file change is based on i_version. On filesystems
+ * which do not support i_version, support is limited to an initial
+ * measurement/appraisal/audit.
+ */
+ i_version = file_inode(file)->i_version;
+ hash.hdr.algo = algo;
- hash.hdr.algo = algo;
+ /* Initialize hash digest to 0's in case of failure */
+ memset(&hash.digest, 0, sizeof(hash.digest));
- result = (!buf) ? ima_calc_file_hash(file, &hash.hdr) :
- ima_calc_buffer_hash(buf, size, &hash.hdr);
- if (!result) {
- int length = sizeof(hash.hdr) + hash.hdr.length;
- void *tmpbuf = krealloc(iint->ima_hash, length,
- GFP_NOFS);
- if (tmpbuf) {
- iint->ima_hash = tmpbuf;
- memcpy(iint->ima_hash, &hash, length);
- iint->version = i_version;
- iint->flags |= IMA_COLLECTED;
- } else
- result = -ENOMEM;
- }
+ if (buf)
+ result = ima_calc_buffer_hash(buf, size, &hash.hdr);
+ else
+ result = ima_calc_file_hash(file, &hash.hdr);
+
+ if (result && result != -EBADF && result != -EINVAL)
+ goto out;
+
+ length = sizeof(hash.hdr) + hash.hdr.length;
+ tmpbuf = krealloc(iint->ima_hash, length, GFP_NOFS);
+ if (!tmpbuf) {
+ result = -ENOMEM;
+ goto out;
}
+
+ iint->ima_hash = tmpbuf;
+ memcpy(iint->ima_hash, &hash, length);
+ iint->version = i_version;
+
+ /* Possibly temporary failure due to type of read (eg. O_DIRECT) */
+ if (!result)
+ iint->flags |= IMA_COLLECTED;
out:
- if (result)
+ if (result) {
+ if (file->f_flags & O_DIRECT)
+ audit_cause = "failed(directio)";
+
integrity_audit_msg(AUDIT_INTEGRITY_DATA, inode,
filename, "collect_data", audit_cause,
result, 0);
+ }
return result;
}
@@ -277,7 +294,7 @@ void ima_store_measurement(struct integrity_iint_cache *iint,
}
result = ima_store_template(entry, violation, inode, filename, pcr);
- if (!result || result == -EEXIST) {
+ if ((!result || result == -EEXIST) && !(file->f_flags & O_DIRECT)) {
iint->flags |= IMA_MEASURED;
iint->measured_pcrs |= (0x1 << pcr);
}
diff --git a/security/integrity/ima/ima_appraise.c b/security/integrity/ima/ima_appraise.c
index 7bf8b00..af55c31 100644
--- a/security/integrity/ima/ima_appraise.c
+++ b/security/integrity/ima/ima_appraise.c
@@ -214,7 +214,9 @@ int ima_appraise_measurement(enum ima_hooks func,
}
status = evm_verifyxattr(dentry, XATTR_NAME_IMA, xattr_value, rc, iint);
- if ((status != INTEGRITY_PASS) && (status != INTEGRITY_UNKNOWN)) {
+ if ((status != INTEGRITY_PASS) &&
+ (status != INTEGRITY_PASS_IMMUTABLE) &&
+ (status != INTEGRITY_UNKNOWN)) {
if ((status == INTEGRITY_NOLABEL)
|| (status == INTEGRITY_NOXATTRS))
cause = "missing-HMAC";
@@ -232,6 +234,7 @@ int ima_appraise_measurement(enum ima_hooks func,
status = INTEGRITY_FAIL;
break;
}
+ clear_bit(IMA_DIGSIG, &iint->atomic_flags);
if (xattr_len - sizeof(xattr_value->type) - hash_start >=
iint->ima_hash->length)
/* xattr length may be longer. md5 hash in previous
@@ -250,7 +253,7 @@ int ima_appraise_measurement(enum ima_hooks func,
status = INTEGRITY_PASS;
break;
case EVM_IMA_XATTR_DIGSIG:
- iint->flags |= IMA_DIGSIG;
+ set_bit(IMA_DIGSIG, &iint->atomic_flags);
rc = integrity_digsig_verify(INTEGRITY_KEYRING_IMA,
(const char *)xattr_value, rc,
iint->ima_hash->digest,
@@ -301,7 +304,7 @@ void ima_update_xattr(struct integrity_iint_cache *iint, struct file *file)
int rc = 0;
/* do not collect and update hash for digital signatures */
- if (iint->flags & IMA_DIGSIG)
+ if (test_bit(IMA_DIGSIG, &iint->atomic_flags))
return;
if (iint->ima_file_status != INTEGRITY_PASS)
@@ -311,7 +314,9 @@ void ima_update_xattr(struct integrity_iint_cache *iint, struct file *file)
if (rc < 0)
return;
+ inode_lock(file_inode(file));
ima_fix_xattr(dentry, iint);
+ inode_unlock(file_inode(file));
}
/**
@@ -334,16 +339,14 @@ void ima_inode_post_setattr(struct dentry *dentry)
return;
must_appraise = ima_must_appraise(inode, MAY_ACCESS, POST_SETATTR);
- iint = integrity_iint_find(inode);
- if (iint) {
- iint->flags &= ~(IMA_APPRAISE | IMA_APPRAISED |
- IMA_APPRAISE_SUBMASK | IMA_APPRAISED_SUBMASK |
- IMA_ACTION_RULE_FLAGS);
- if (must_appraise)
- iint->flags |= IMA_APPRAISE;
- }
if (!must_appraise)
__vfs_removexattr(dentry, XATTR_NAME_IMA);
+ iint = integrity_iint_find(inode);
+ if (iint) {
+ set_bit(IMA_CHANGE_ATTR, &iint->atomic_flags);
+ if (!must_appraise)
+ clear_bit(IMA_UPDATE_XATTR, &iint->atomic_flags);
+ }
}
/*
@@ -372,12 +375,12 @@ static void ima_reset_appraise_flags(struct inode *inode, int digsig)
iint = integrity_iint_find(inode);
if (!iint)
return;
-
- iint->flags &= ~IMA_DONE_MASK;
iint->measured_pcrs = 0;
+ set_bit(IMA_CHANGE_XATTR, &iint->atomic_flags);
if (digsig)
- iint->flags |= IMA_DIGSIG;
- return;
+ set_bit(IMA_DIGSIG, &iint->atomic_flags);
+ else
+ clear_bit(IMA_DIGSIG, &iint->atomic_flags);
}
int ima_inode_setxattr(struct dentry *dentry, const char *xattr_name,
@@ -389,14 +392,10 @@ int ima_inode_setxattr(struct dentry *dentry, const char *xattr_name,
result = ima_protect_xattr(dentry, xattr_name, xattr_value,
xattr_value_len);
if (result == 1) {
- bool digsig;
-
if (!xattr_value_len || (xvalue->type >= IMA_XATTR_LAST))
return -EINVAL;
- digsig = (xvalue->type == EVM_IMA_XATTR_DIGSIG);
- if (!digsig && (ima_appraise & IMA_APPRAISE_ENFORCE))
- return -EPERM;
- ima_reset_appraise_flags(d_backing_inode(dentry), digsig);
+ ima_reset_appraise_flags(d_backing_inode(dentry),
+ (xvalue->type == EVM_IMA_XATTR_DIGSIG) ? 1 : 0);
result = 0;
}
return result;
diff --git a/security/integrity/ima/ima_crypto.c b/security/integrity/ima/ima_crypto.c
index 38f2ed8..20e6629 100644
--- a/security/integrity/ima/ima_crypto.c
+++ b/security/integrity/ima/ima_crypto.c
@@ -78,6 +78,8 @@ int __init ima_init_crypto(void)
hash_algo_name[ima_hash_algo], rc);
return rc;
}
+ pr_info("Allocated hash algorithm: %s\n",
+ hash_algo_name[ima_hash_algo]);
return 0;
}
@@ -441,6 +443,16 @@ int ima_calc_file_hash(struct file *file, struct ima_digest_data *hash)
loff_t i_size;
int rc;
+ /*
+ * For consistency, fail file's opened with the O_DIRECT flag on
+ * filesystems mounted with/without DAX option.
+ */
+ if (file->f_flags & O_DIRECT) {
+ hash->length = hash_digest_size[ima_hash_algo];
+ hash->algo = ima_hash_algo;
+ return -EINVAL;
+ }
+
i_size = i_size_read(file_inode(file));
if (ima_ahash_minsize && i_size >= ima_ahash_minsize) {
diff --git a/security/integrity/ima/ima_fs.c b/security/integrity/ima/ima_fs.c
index 3df4690..44b44d7 100644
--- a/security/integrity/ima/ima_fs.c
+++ b/security/integrity/ima/ima_fs.c
@@ -29,14 +29,14 @@
static DEFINE_MUTEX(ima_write_mutex);
static int valid_policy = 1;
-#define TMPBUFLEN 12
+
static ssize_t ima_show_htable_value(char __user *buf, size_t count,
loff_t *ppos, atomic_long_t *val)
{
- char tmpbuf[TMPBUFLEN];
+ char tmpbuf[32]; /* greater than largest 'long' string value */
ssize_t len;
- len = scnprintf(tmpbuf, TMPBUFLEN, "%li\n", atomic_long_read(val));
+ len = scnprintf(tmpbuf, sizeof(tmpbuf), "%li\n", atomic_long_read(val));
return simple_read_from_buffer(buf, count, ppos, tmpbuf, len);
}
diff --git a/security/integrity/ima/ima_main.c b/security/integrity/ima/ima_main.c
index 2b3def1..ea1e629 100644
--- a/security/integrity/ima/ima_main.c
+++ b/security/integrity/ima/ima_main.c
@@ -16,6 +16,9 @@
* implements the IMA hooks: ima_bprm_check, ima_file_mmap,
* and ima_file_check.
*/
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
#include <linux/module.h>
#include <linux/file.h>
#include <linux/binfmts.h>
@@ -96,10 +99,13 @@ static void ima_rdwr_violation_check(struct file *file,
if (!iint)
iint = integrity_iint_find(inode);
/* IMA_MEASURE is set from reader side */
- if (iint && (iint->flags & IMA_MEASURE))
+ if (iint && test_bit(IMA_MUST_MEASURE,
+ &iint->atomic_flags))
send_tomtou = true;
}
} else {
+ if (must_measure)
+ set_bit(IMA_MUST_MEASURE, &iint->atomic_flags);
if ((atomic_read(&inode->i_writecount) > 0) && must_measure)
send_writers = true;
}
@@ -121,21 +127,24 @@ static void ima_check_last_writer(struct integrity_iint_cache *iint,
struct inode *inode, struct file *file)
{
fmode_t mode = file->f_mode;
+ bool update;
if (!(mode & FMODE_WRITE))
return;
- inode_lock(inode);
+ mutex_lock(&iint->mutex);
if (atomic_read(&inode->i_writecount) == 1) {
+ update = test_and_clear_bit(IMA_UPDATE_XATTR,
+ &iint->atomic_flags);
if ((iint->version != inode->i_version) ||
(iint->flags & IMA_NEW_FILE)) {
iint->flags &= ~(IMA_DONE_MASK | IMA_NEW_FILE);
iint->measured_pcrs = 0;
- if (iint->flags & IMA_APPRAISE)
+ if (update)
ima_update_xattr(iint, file);
}
}
- inode_unlock(inode);
+ mutex_unlock(&iint->mutex);
}
/**
@@ -168,7 +177,7 @@ static int process_measurement(struct file *file, char *buf, loff_t size,
char *pathbuf = NULL;
char filename[NAME_MAX];
const char *pathname = NULL;
- int rc = -ENOMEM, action, must_appraise;
+ int rc = 0, action, must_appraise = 0;
int pcr = CONFIG_IMA_MEASURE_PCR_IDX;
struct evm_ima_xattr_data *xattr_value = NULL;
int xattr_len = 0;
@@ -199,17 +208,31 @@ static int process_measurement(struct file *file, char *buf, loff_t size,
if (action) {
iint = integrity_inode_get(inode);
if (!iint)
- goto out;
+ rc = -ENOMEM;
}
- if (violation_check) {
+ if (!rc && violation_check)
ima_rdwr_violation_check(file, iint, action & IMA_MEASURE,
&pathbuf, &pathname);
- if (!action) {
- rc = 0;
- goto out_free;
- }
- }
+
+ inode_unlock(inode);
+
+ if (rc)
+ goto out;
+ if (!action)
+ goto out;
+
+ mutex_lock(&iint->mutex);
+
+ if (test_and_clear_bit(IMA_CHANGE_ATTR, &iint->atomic_flags))
+ /* reset appraisal flags if ima_inode_post_setattr was called */
+ iint->flags &= ~(IMA_APPRAISE | IMA_APPRAISED |
+ IMA_APPRAISE_SUBMASK | IMA_APPRAISED_SUBMASK |
+ IMA_ACTION_FLAGS);
+
+ if (test_and_clear_bit(IMA_CHANGE_XATTR, &iint->atomic_flags))
+ /* reset all flags if ima_inode_setxattr was called */
+ iint->flags &= ~IMA_DONE_MASK;
/* Determine if already appraised/measured based on bitmask
* (IMA_MEASURE, IMA_MEASURED, IMA_XXXX_APPRAISE, IMA_XXXX_APPRAISED,
@@ -227,7 +250,7 @@ static int process_measurement(struct file *file, char *buf, loff_t size,
if (!action) {
if (must_appraise)
rc = ima_get_cache_status(iint, func);
- goto out_digsig;
+ goto out_locked;
}
template_desc = ima_template_desc_current();
@@ -239,11 +262,8 @@ static int process_measurement(struct file *file, char *buf, loff_t size,
hash_algo = ima_get_hash_algo(xattr_value, xattr_len);
rc = ima_collect_measurement(iint, file, buf, size, hash_algo);
- if (rc != 0) {
- if (file->f_flags & O_DIRECT)
- rc = (iint->flags & IMA_PERMIT_DIRECTIO) ? 0 : -EACCES;
- goto out_digsig;
- }
+ if (rc != 0 && rc != -EBADF && rc != -EINVAL)
+ goto out_locked;
if (!pathbuf) /* ima_rdwr_violation possibly pre-fetched */
pathname = ima_d_path(&file->f_path, &pathbuf, filename);
@@ -251,24 +271,32 @@ static int process_measurement(struct file *file, char *buf, loff_t size,
if (action & IMA_MEASURE)
ima_store_measurement(iint, file, pathname,
xattr_value, xattr_len, pcr);
- if (action & IMA_APPRAISE_SUBMASK)
+ if (rc == 0 && (action & IMA_APPRAISE_SUBMASK)) {
+ inode_lock(inode);
rc = ima_appraise_measurement(func, iint, file, pathname,
xattr_value, xattr_len, opened);
+ inode_unlock(inode);
+ }
if (action & IMA_AUDIT)
ima_audit_measurement(iint, pathname);
-out_digsig:
- if ((mask & MAY_WRITE) && (iint->flags & IMA_DIGSIG) &&
+ if ((file->f_flags & O_DIRECT) && (iint->flags & IMA_PERMIT_DIRECTIO))
+ rc = 0;
+out_locked:
+ if ((mask & MAY_WRITE) && test_bit(IMA_DIGSIG, &iint->atomic_flags) &&
!(iint->flags & IMA_NEW_FILE))
rc = -EACCES;
+ mutex_unlock(&iint->mutex);
kfree(xattr_value);
-out_free:
+out:
if (pathbuf)
__putname(pathbuf);
-out:
- inode_unlock(inode);
- if ((rc && must_appraise) && (ima_appraise & IMA_APPRAISE_ENFORCE))
- return -EACCES;
+ if (must_appraise) {
+ if (rc && (ima_appraise & IMA_APPRAISE_ENFORCE))
+ return -EACCES;
+ if (file->f_mode & FMODE_WRITE)
+ set_bit(IMA_UPDATE_XATTR, &iint->atomic_flags);
+ }
return 0;
}
@@ -376,6 +404,7 @@ int ima_read_file(struct file *file, enum kernel_read_file_id read_id)
static int read_idmap[READING_MAX_ID] = {
[READING_FIRMWARE] = FIRMWARE_CHECK,
+ [READING_FIRMWARE_PREALLOC_BUFFER] = FIRMWARE_CHECK,
[READING_MODULE] = MODULE_CHECK,
[READING_KEXEC_IMAGE] = KEXEC_KERNEL_CHECK,
[READING_KEXEC_INITRAMFS] = KEXEC_INITRAMFS_CHECK,
@@ -426,6 +455,16 @@ static int __init init_ima(void)
hash_setup(CONFIG_IMA_DEFAULT_HASH);
error = ima_init();
+
+ if (error && strcmp(hash_algo_name[ima_hash_algo],
+ CONFIG_IMA_DEFAULT_HASH) != 0) {
+ pr_info("Allocating %s failed, going to use default hash algorithm %s\n",
+ hash_algo_name[ima_hash_algo], CONFIG_IMA_DEFAULT_HASH);
+ hash_setup_done = 0;
+ hash_setup(CONFIG_IMA_DEFAULT_HASH);
+ error = ima_init();
+ }
+
if (!error) {
ima_initialized = 1;
ima_update_policy_flag();
diff --git a/security/integrity/integrity.h b/security/integrity/integrity.h
index 24520b4..2f7e236 100644
--- a/security/integrity/integrity.h
+++ b/security/integrity/integrity.h
@@ -29,10 +29,10 @@
/* iint cache flags */
#define IMA_ACTION_FLAGS 0xff000000
#define IMA_ACTION_RULE_FLAGS 0x06000000
-#define IMA_DIGSIG 0x01000000
-#define IMA_DIGSIG_REQUIRED 0x02000000
-#define IMA_PERMIT_DIRECTIO 0x04000000
-#define IMA_NEW_FILE 0x08000000
+#define IMA_DIGSIG_REQUIRED 0x01000000
+#define IMA_PERMIT_DIRECTIO 0x02000000
+#define IMA_NEW_FILE 0x04000000
+#define EVM_IMMUTABLE_DIGSIG 0x08000000
#define IMA_DO_MASK (IMA_MEASURE | IMA_APPRAISE | IMA_AUDIT | \
IMA_APPRAISE_SUBMASK)
@@ -53,11 +53,19 @@
#define IMA_APPRAISED_SUBMASK (IMA_FILE_APPRAISED | IMA_MMAP_APPRAISED | \
IMA_BPRM_APPRAISED | IMA_READ_APPRAISED)
+/* iint cache atomic_flags */
+#define IMA_CHANGE_XATTR 0
+#define IMA_UPDATE_XATTR 1
+#define IMA_CHANGE_ATTR 2
+#define IMA_DIGSIG 3
+#define IMA_MUST_MEASURE 4
+
enum evm_ima_xattr_type {
IMA_XATTR_DIGEST = 0x01,
EVM_XATTR_HMAC,
EVM_IMA_XATTR_DIGSIG,
IMA_XATTR_DIGEST_NG,
+ EVM_XATTR_PORTABLE_DIGSIG,
IMA_XATTR_LAST
};
@@ -100,10 +108,12 @@ struct signature_v2_hdr {
/* integrity data associated with an inode */
struct integrity_iint_cache {
struct rb_node rb_node; /* rooted in integrity_iint_tree */
+ struct mutex mutex; /* protects: version, flags, digest */
struct inode *inode; /* back pointer to inode in question */
u64 version; /* track inode changes */
unsigned long flags;
unsigned long measured_pcrs;
+ unsigned long atomic_flags;
enum integrity_status ima_file_status:4;
enum integrity_status ima_mmap_status:4;
enum integrity_status ima_bprm_status:4;
diff --git a/security/pfe/Kconfig b/security/pfe/Kconfig
index 0cd9e81..a1d80ae 100644
--- a/security/pfe/Kconfig
+++ b/security/pfe/Kconfig
@@ -1,17 +1,6 @@
menu "Qualcomm Technologies, Inc Per File Encryption security device drivers"
depends on ARCH_QCOM
-config PFT
- bool "Per-File-Tagger driver"
- depends on SECURITY
- default n
- help
- This driver is used for tagging enterprise files.
- It is part of the Per-File-Encryption (PFE) feature.
- The driver is tagging files when created by
- registered application.
- Tagged files are encrypted using the dm-req-crypt driver.
-
config PFK
bool "Per-File-Key driver"
depends on SECURITY
diff --git a/security/pfe/Makefile b/security/pfe/Makefile
index 4096aad..92903c2 100644
--- a/security/pfe/Makefile
+++ b/security/pfe/Makefile
@@ -2,9 +2,11 @@
# Makefile for the MSM specific security device drivers.
#
-ccflags-y += -Isecurity/selinux -Isecurity/selinux/include
-#ccflags-y += -Ifs/ext4
+ccflags-y += -Isecurity/selinux -Isecurity/selinux/include -Ifs/ecryptfs
ccflags-y += -Ifs/crypto
-obj-$(CONFIG_PFT) += pft.o
-obj-$(CONFIG_PFK) += pfk.o pfk_kc.o pfk_ice.o pfk_ext4.o pfk_f2fs.o
+obj-$(CONFIG_PFK) += pfk.o pfk_kc.o pfk_ice.o pfk_fscrypt.o pfk_ecryptfs.o
+
+# TODO: remove me b/62058353
+subdir-ccflags-y += \
+ -Wno-ignored-attributes
diff --git a/security/pfe/pfk.c b/security/pfe/pfk.c
index 0e153f2..91019f9 100644
--- a/security/pfe/pfk.c
+++ b/security/pfe/pfk.c
@@ -37,7 +37,6 @@
*
*/
-
/* Uncomment the line below to enable debug messages */
/* #define DEBUG 1 */
#define pr_fmt(fmt) "pfk [%s]: " fmt, __func__
@@ -52,73 +51,71 @@
#include <crypto/ice.h>
#include <linux/pfk.h>
+#include <linux/ecryptfs.h>
#include "pfk_kc.h"
#include "objsec.h"
+#include "ecryptfs_kernel.h"
#include "pfk_ice.h"
-#include "pfk_ext4.h"
-#include "pfk_f2fs.h"
+#include "pfk_fscrypt.h"
+#include "pfk_ecryptfs.h"
#include "pfk_internal.h"
-//#include "ext4.h"
static bool pfk_ready;
-
/* might be replaced by a table when more than one cipher is supported */
#define PFK_SUPPORTED_KEY_SIZE 32
#define PFK_SUPPORTED_SALT_SIZE 32
/* Various PFE types and function tables to support each one of them */
-enum pfe_type {EXT4_CRYPT_PFE, F2FS_CRYPT_PFE, INVALID_PFE};
+enum pfe_type {ECRYPTFS_PFE, FSCRYPT_PFE, INVALID_PFE};
typedef int (*pfk_parse_inode_type)(const struct bio *bio,
- const struct inode *inode,
- struct pfk_key_info *key_info,
- enum ice_cryto_algo_mode *algo,
- bool *is_pfe,
- unsigned int *data_unit,
- const char *storage_type);
+ const struct inode *inode,
+ struct pfk_key_info *key_info,
+ enum ice_cryto_algo_mode *algo,
+ bool *is_pfe);
typedef bool (*pfk_allow_merge_bio_type)(const struct bio *bio1,
- const struct bio *bio2, const struct inode *inode1,
- const struct inode *inode2);
+ const struct bio *bio2, const struct inode *inode1,
+ const struct inode *inode2);
static const pfk_parse_inode_type pfk_parse_inode_ftable[] = {
- /* EXT4_CRYPT_PFE */ &pfk_ext4_parse_inode,
- /* F2FS_CRYPT_PFE */ &pfk_f2fs_parse_inode,
+ /* ECRYPTFS_PFE */ &pfk_ecryptfs_parse_inode,
+ /* FSCRYPT_PFE */ &pfk_fscrypt_parse_inode,
};
static const pfk_allow_merge_bio_type pfk_allow_merge_bio_ftable[] = {
- /* EXT4_CRYPT_PFE */ &pfk_ext4_allow_merge_bio,
- /* F2FS_CRYPT_PFE */ &pfk_f2fs_allow_merge_bio,
+ /* ECRYPTFS_PFE */ &pfk_ecryptfs_allow_merge_bio,
+ /* FSCRYPT_PFE */ &pfk_fscrypt_allow_merge_bio,
};
static void __exit pfk_exit(void)
{
pfk_ready = false;
- pfk_ext4_deinit();
- pfk_f2fs_deinit();
+ pfk_fscrypt_deinit();
pfk_kc_deinit();
}
static int __init pfk_init(void)
{
-
int ret = 0;
- ret = pfk_ext4_init();
+ ret = pfk_ecryptfs_init();
if (ret != 0)
goto fail;
- ret = pfk_f2fs_init();
- if (ret != 0)
+ ret = pfk_fscrypt_init();
+ if (ret != 0) {
+ pfk_ecryptfs_deinit();
goto fail;
+ }
ret = pfk_kc_init();
if (ret != 0) {
pr_err("could init pfk key cache, error %d\n", ret);
- pfk_ext4_deinit();
- pfk_f2fs_deinit();
+ pfk_fscrypt_deinit();
+ pfk_ecryptfs_deinit();
goto fail;
}
@@ -141,41 +138,16 @@ static enum pfe_type pfk_get_pfe_type(const struct inode *inode)
if (!inode)
return INVALID_PFE;
- if (pfk_is_ext4_type(inode))
- return EXT4_CRYPT_PFE;
+ if (pfk_is_ecryptfs_type(inode))
+ return ECRYPTFS_PFE;
- if (pfk_is_f2fs_type(inode))
- return F2FS_CRYPT_PFE;
+ if (pfk_is_fscrypt_type(inode))
+ return FSCRYPT_PFE;
return INVALID_PFE;
}
/**
- * inode_to_filename() - get the filename from inode pointer.
- * @inode: inode pointer
- *
- * it is used for debug prints.
- *
- * Return: filename string or "unknown".
- */
-char *inode_to_filename(const struct inode *inode)
-{
- struct dentry *dentry = NULL;
- char *filename = NULL;
-
- if (!inode)
- return "NULL";
-
- if (hlist_empty(&inode->i_dentry))
- return "unknown";
-
- dentry = hlist_entry(inode->i_dentry.first, struct dentry, d_u.d_alias);
- filename = dentry->d_iname;
-
- return filename;
-}
-
-/**
* pfk_is_ready() - driver is initialized and ready.
*
* Return: true if the driver is ready.
@@ -214,7 +186,7 @@ static struct inode *pfk_bio_get_inode(const struct bio *bio)
/* Using direct-io (O_DIRECT) without page cache */
inode = dio_bio_get_inode((struct bio *)bio);
- pr_debug("inode on direct-io, inode = 0x%pK.\n", inode);
+ pr_debug("inode on direct-io, inode = 0x%p.\n", inode);
return inode;
}
@@ -233,14 +205,13 @@ static struct inode *pfk_bio_get_inode(const struct bio *bio)
* return 0 in case of success, error otherwise (i.e not supported key size)
*/
int pfk_key_size_to_key_type(size_t key_size,
- enum ice_crpto_key_size *key_size_type)
+ enum ice_crpto_key_size *key_size_type)
{
/*
* currently only 32 bit key size is supported
* in the future, table with supported key sizes might
* be introduced
*/
-
if (key_size != PFK_SUPPORTED_KEY_SIZE) {
pr_err("not supported key size %zu\n", key_size);
return -EINVAL;
@@ -256,7 +227,7 @@ int pfk_key_size_to_key_type(size_t key_size,
* Retrieves filesystem type from inode's superblock
*/
bool pfe_is_inode_filesystem_type(const struct inode *inode,
- const char *fs_type)
+ const char *fs_type)
{
if (!inode || !fs_type)
return false;
@@ -283,31 +254,30 @@ bool pfe_is_inode_filesystem_type(const struct inode *inode,
static int pfk_get_key_for_bio(const struct bio *bio,
struct pfk_key_info *key_info,
enum ice_cryto_algo_mode *algo_mode,
- bool *is_pfe, unsigned int *data_unit)
+ bool *is_pfe)
{
const struct inode *inode;
enum pfe_type which_pfe;
- const struct blk_encryption_key *key;
- char *s_type = NULL;
+ const struct blk_encryption_key *key = NULL;
inode = pfk_bio_get_inode(bio);
which_pfe = pfk_get_pfe_type(inode);
- s_type = (char *)pfk_kc_get_storage_type();
if (which_pfe != INVALID_PFE) {
/* Encrypted file; override ->bi_crypt_key */
pr_debug("parsing inode %lu with PFE type %d\n",
inode->i_ino, which_pfe);
return (*(pfk_parse_inode_ftable[which_pfe]))
- (bio, inode, key_info, algo_mode, is_pfe,
- data_unit, (const char *)s_type);
+ (bio, inode, key_info, algo_mode, is_pfe);
}
/*
* bio is not for an encrypted file. Use ->bi_crypt_key if it was set.
* Otherwise, don't encrypt/decrypt the bio.
*/
+#ifdef CONFIG_DM_DEFAULT_KEY
key = bio->bi_crypt_key;
+#endif
if (!key) {
*is_pfe = false;
return -EINVAL;
@@ -350,10 +320,9 @@ int pfk_load_key_start(const struct bio *bio,
bool async)
{
int ret = 0;
- struct pfk_key_info key_info = {NULL, NULL, 0, 0};
+ struct pfk_key_info key_info = {0};
enum ice_cryto_algo_mode algo_mode = ICE_CRYPTO_ALGO_MODE_AES_XTS;
enum ice_crpto_key_size key_size_type = 0;
- unsigned int data_unit = 1 << ICE_CRYPTO_DATA_UNIT_512_B;
u32 key_index = 0;
if (!is_pfe) {
@@ -376,9 +345,7 @@ int pfk_load_key_start(const struct bio *bio,
return -EINVAL;
}
- ret = pfk_get_key_for_bio(bio, &key_info, &algo_mode, is_pfe,
- &data_unit);
-
+ ret = pfk_get_key_for_bio(bio, &key_info, &algo_mode, is_pfe);
if (ret != 0)
return ret;
@@ -387,8 +354,7 @@ int pfk_load_key_start(const struct bio *bio,
return ret;
ret = pfk_kc_load_key_start(key_info.key, key_info.key_size,
- key_info.salt, key_info.salt_size, &key_index, async,
- data_unit);
+ key_info.salt, key_info.salt_size, &key_index, async);
if (ret) {
if (ret != -EBUSY && ret != -EAGAIN)
pr_err("start: could not load key into pfk key cache, error %d\n",
@@ -403,9 +369,6 @@ int pfk_load_key_start(const struct bio *bio,
ice_setting->key_mode = ICE_CRYPTO_USE_LUT_SW_KEY;
ice_setting->key_index = key_index;
- pr_debug("loaded key for file %s key_index %d\n",
- inode_to_filename(pfk_bio_get_inode(bio)), key_index);
-
return 0;
}
@@ -423,7 +386,7 @@ int pfk_load_key_start(const struct bio *bio,
int pfk_load_key_end(const struct bio *bio, bool *is_pfe)
{
int ret = 0;
- struct pfk_key_info key_info = {NULL, NULL, 0, 0};
+ struct pfk_key_info key_info = {0};
if (!is_pfe) {
pr_err("is_pfe is NULL\n");
@@ -439,16 +402,13 @@ int pfk_load_key_end(const struct bio *bio, bool *is_pfe)
if (!pfk_is_ready())
return -ENODEV;
- ret = pfk_get_key_for_bio(bio, &key_info, NULL, is_pfe, NULL);
+ ret = pfk_get_key_for_bio(bio, &key_info, NULL, is_pfe);
if (ret != 0)
return ret;
pfk_kc_load_key_end(key_info.key, key_info.key_size,
key_info.salt, key_info.salt_size);
- pr_debug("finished using key for file %s\n",
- inode_to_filename(pfk_bio_get_inode(bio)));
-
return 0;
}
@@ -469,13 +429,18 @@ int pfk_load_key_end(const struct bio *bio, bool *is_pfe)
*/
bool pfk_allow_merge_bio(const struct bio *bio1, const struct bio *bio2)
{
- const struct blk_encryption_key *key1;
- const struct blk_encryption_key *key2;
+ const struct blk_encryption_key *key1 = NULL;
+ const struct blk_encryption_key *key2 = NULL;
const struct inode *inode1;
const struct inode *inode2;
enum pfe_type which_pfe1;
enum pfe_type which_pfe2;
+#ifdef CONFIG_DM_DEFAULT_KEY
+ key1 = bio1->bi_crypt_key;
+ key2 = bio2->bi_crypt_key;
+#endif
+
if (!pfk_is_ready())
return false;
@@ -485,9 +450,6 @@ bool pfk_allow_merge_bio(const struct bio *bio1, const struct bio *bio2)
if (bio1 == bio2)
return true;
- key1 = bio1->bi_crypt_key;
- key2 = bio2->bi_crypt_key;
-
inode1 = pfk_bio_get_inode(bio1);
inode2 = pfk_bio_get_inode(bio2);
@@ -504,8 +466,8 @@ bool pfk_allow_merge_bio(const struct bio *bio1, const struct bio *bio2)
if (which_pfe1 != INVALID_PFE) {
/* Both bios are for the same type of encrypted file. */
- return (*(pfk_allow_merge_bio_ftable[which_pfe1]))(bio1, bio2,
- inode1, inode2);
+ return (*(pfk_allow_merge_bio_ftable[which_pfe1]))(bio1, bio2,
+ inode1, inode2);
}
/*
diff --git a/security/pfe/pfk_ecryptfs.c b/security/pfe/pfk_ecryptfs.c
new file mode 100644
index 0000000..f55ee83
--- /dev/null
+++ b/security/pfe/pfk_ecryptfs.c
@@ -0,0 +1,592 @@
+/*
+ * Copyright (c) 2015-2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+/*
+ * Per-File-Key (PFK) - eCryptfs.
+ *
+ * This driver is used for storing eCryptfs information (mainly file
+ * encryption key) in file node as part of eCryptfs hardware enhanced solution
+ * provided by Qualcomm Technologies, Inc.
+ *
+ * The information is stored in node when file is first opened (eCryptfs
+ * will fire a callback notifying PFK about this event) and will be later
+ * accessed by Block Device Driver to actually load the key to encryption hw.
+ *
+ * PFK exposes API's for loading and removing keys from encryption hw
+ * and also API to determine whether 2 adjacent blocks can be agregated by
+ * Block Layer in one request to encryption hw.
+ * PFK is only supposed to be used by eCryptfs, except the below.
+ *
+ */
+
+
+/* Uncomment the line below to enable debug messages */
+/* #define DEBUG 1 */
+#define pr_fmt(fmt) "pfk_ecryptfs [%s]: " fmt, __func__
+
+#include <linux/module.h>
+#include <linux/fs.h>
+#include <linux/errno.h>
+#include <linux/printk.h>
+#include <linux/bio.h>
+#include <linux/security.h>
+#include <linux/lsm_hooks.h>
+#include <crypto/ice.h>
+
+#include <linux/pfk.h>
+#include <linux/ecryptfs.h>
+
+#include "pfk_ecryptfs.h"
+#include "pfk_kc.h"
+#include "objsec.h"
+#include "ecryptfs_kernel.h"
+#include "pfk_ice.h"
+
+static DEFINE_MUTEX(pfk_ecryptfs_lock);
+static bool pfk_ecryptfs_ready;
+static int g_events_handle;
+
+
+/* might be replaced by a table when more than one cipher is supported */
+#define PFK_SUPPORTED_CIPHER "aes_xts"
+#define PFK_SUPPORTED_SALT_SIZE 32
+
+static void *pfk_ecryptfs_get_data(const struct inode *inode);
+static void pfk_ecryptfs_open_cb(struct inode *inode, void *ecryptfs_data);
+static void pfk_ecryptfs_release_cb(struct inode *inode);
+static bool pfk_ecryptfs_is_cipher_supported_cb(const void *ecryptfs_data);
+static size_t pfk_ecryptfs_get_salt_key_size_cb(const void *ecryptfs_data);
+static bool pfk_ecryptfs_is_hw_crypt_cb(void);
+
+
+/**
+ * pfk_is_ecryptfs_type() - return true if inode belongs to ICE ecryptfs PFE
+ * @inode: inode pointer
+ */
+bool pfk_is_ecryptfs_type(const struct inode *inode)
+{
+ void *ecryptfs_data = NULL;
+
+ /*
+ * the actual filesystem of an inode is still ext4, eCryptfs never
+ * reaches bio
+ */
+ if (!pfe_is_inode_filesystem_type(inode, "ext4"))
+ return false;
+
+ ecryptfs_data = pfk_ecryptfs_get_data(inode);
+
+ if (!ecryptfs_data)
+ return false;
+
+ return true;
+}
+
+/*
+ * pfk_ecryptfs_lsm_init() - makes sure either se-linux is
+ * registered as security module as it is required by pfk_ecryptfs.
+ *
+ * This is required because ecryptfs uses a field inside security struct in
+ * inode to store its info
+ */
+static int __init pfk_ecryptfs_lsm_init(void)
+{
+ if (!selinux_is_enabled()) {
+ pr_err("PFE eCryptfs requires se linux to be enabled\n");
+ return -ENODEV;
+ }
+
+ return 0;
+}
+
+/*
+ * pfk_ecryptfs_deinit() - Deinit function, should be invoked by upper PFK layer
+ */
+void pfk_ecryptfs_deinit(void)
+{
+ pfk_ecryptfs_ready = false;
+ ecryptfs_unregister_from_events(g_events_handle);
+}
+
+/*
+ * pfk_ecryptfs_init() - Init function, should be invoked by upper PFK layer
+ */
+int __init pfk_ecryptfs_init(void)
+{
+ int ret = 0;
+ struct ecryptfs_events events = {0};
+
+ events.open_cb = pfk_ecryptfs_open_cb;
+ events.release_cb = pfk_ecryptfs_release_cb;
+ events.is_cipher_supported_cb = pfk_ecryptfs_is_cipher_supported_cb;
+ events.is_hw_crypt_cb = pfk_ecryptfs_is_hw_crypt_cb;
+ events.get_salt_key_size_cb = pfk_ecryptfs_get_salt_key_size_cb;
+
+ g_events_handle = ecryptfs_register_to_events(&events);
+ if (g_events_handle == 0) {
+ pr_err("could not register with eCryptfs, error %d\n", ret);
+ goto fail;
+ }
+
+ ret = pfk_ecryptfs_lsm_init();
+ if (ret != 0) {
+ pr_debug("neither pfk nor se-linux sec modules are enabled\n");
+ pr_debug("not an error, just don't enable PFK ecryptfs\n");
+ ecryptfs_unregister_from_events(g_events_handle);
+ return 0;
+ }
+
+ pfk_ecryptfs_ready = true;
+ pr_info("PFK ecryptfs inited successfully\n");
+
+ return 0;
+
+fail:
+ pr_err("Failed to init PFK ecryptfs\n");
+ return -ENODEV;
+}
+
+/**
+ * pfk_ecryptfs_is_ready() - driver is initialized and ready.
+ *
+ * Return: true if the driver is ready.
+ */
+static inline bool pfk_ecryptfs_is_ready(void)
+{
+ return pfk_ecryptfs_ready;
+}
+
+/**
+ * pfk_ecryptfs_get_page_index() - get the inode from a bio.
+ * @bio: Pointer to BIO structure.
+ *
+ * Walk the bio struct links to get the inode.
+ * Please note, that in general bio may consist of several pages from
+ * several files, but in our case we always assume that all pages come
+ * from the same file, since our logic ensures it. That is why we only
+ * walk through the first page to look for inode.
+ *
+ * Return: pointer to the inode struct if successful, or NULL otherwise.
+ *
+ */
+static int pfk_ecryptfs_get_page_index(const struct bio *bio,
+ pgoff_t *page_index)
+{
+ if (!bio || !page_index)
+ return -EINVAL;
+ if (!bio_has_data((struct bio *)bio))
+ return -EINVAL;
+ if (!bio->bi_io_vec)
+ return -EINVAL;
+ if (!bio->bi_io_vec->bv_page)
+ return -EINVAL;
+
+ *page_index = bio->bi_io_vec->bv_page->index;
+
+ return 0;
+}
+
+/**
+ * pfk_ecryptfs_get_data() - retrieves ecryptfs data stored inside node
+ * @inode: inode
+ *
+ * Return the data or NULL if there isn't any or in case of error
+ * Should be invoked under lock
+ */
+static void *pfk_ecryptfs_get_data(const struct inode *inode)
+{
+ struct inode_security_struct *isec = NULL;
+
+ if (!inode)
+ return NULL;
+
+ isec = inode->i_security;
+
+ if (!isec) {
+ pr_debug("i_security is NULL, could be irrelevant file\n");
+ return NULL;
+ }
+
+ return isec->pfk_data;
+}
+
+/**
+ * pfk_ecryptfs_set_data() - stores ecryptfs data inside node
+ * @inode: inode to update
+ * @data: data to put inside the node
+ *
+ * Returns 0 in case of success, error otherwise
+ * Should be invoked under lock
+ */
+static int pfk_ecryptfs_set_data(struct inode *inode, void *ecryptfs_data)
+{
+ struct inode_security_struct *isec = NULL;
+
+ if (!inode)
+ return -EINVAL;
+
+ isec = inode->i_security;
+
+ if (!isec) {
+ pr_err("i_security is NULL, not ready yet\n");
+ return -EINVAL;
+ }
+
+ isec->pfk_data = ecryptfs_data;
+
+ return 0;
+}
+
+
+/**
+ * pfk_ecryptfs_parse_cipher() - parse cipher from ecryptfs to enum
+ * @ecryptfs_data: ecrypfs data
+ * @algo: pointer to store the output enum (can be null)
+ *
+ * return 0 in case of success, error otherwise (i.e not supported cipher)
+ */
+static int pfk_ecryptfs_parse_cipher(const void *ecryptfs_data,
+ enum ice_cryto_algo_mode *algo)
+{
+ /*
+ * currently only AES XTS algo is supported
+ * in the future, table with supported ciphers might
+ * be introduced
+ */
+
+ if (!ecryptfs_data)
+ return -EINVAL;
+
+ if (!ecryptfs_cipher_match(ecryptfs_data,
+ PFK_SUPPORTED_CIPHER, sizeof(PFK_SUPPORTED_CIPHER))) {
+ pr_debug("ecryptfs alghoritm is not supported by pfk\n");
+ return -EINVAL;
+ }
+
+ if (algo)
+ *algo = ICE_CRYPTO_ALGO_MODE_AES_XTS;
+
+ return 0;
+}
+
+/*
+ * pfk_ecryptfs_parse_inode() - parses key and algo information from inode
+ *
+ * Should be invoked by upper pfk layer
+ * @bio: bio
+ * @inode: inode to be parsed
+ * @key_info: out, key and salt information to be stored
+ * @algo: out, algorithm to be stored (can be null)
+ * @is_pfe: out, will be false if inode is not relevant to PFE, in such a case
+ * it should be treated as non PFE by the block layer
+ */
+int pfk_ecryptfs_parse_inode(const struct bio *bio,
+ const struct inode *inode,
+ struct pfk_key_info *key_info,
+ enum ice_cryto_algo_mode *algo,
+ bool *is_pfe)
+{
+ int ret = 0;
+ void *ecryptfs_data = NULL;
+ pgoff_t offset;
+ bool is_metadata = false;
+
+ if (!is_pfe)
+ return -EINVAL;
+
+ /*
+ * only a few errors below can indicate that
+ * this function was not invoked within PFE context,
+ * otherwise we will consider it PFE
+ */
+ *is_pfe = true;
+
+ if (!pfk_ecryptfs_is_ready())
+ return -ENODEV;
+
+ if (!inode)
+ return -EINVAL;
+
+ if (!key_info)
+ return -EINVAL;
+
+ ecryptfs_data = pfk_ecryptfs_get_data(inode);
+ if (!ecryptfs_data) {
+ pr_err("internal error, no ecryptfs data\n");
+ return -EINVAL;
+ }
+
+ ret = pfk_ecryptfs_get_page_index(bio, &offset);
+ if (ret != 0) {
+ pr_err("could not get page index from bio, probably bug %d\n",
+ ret);
+ return -EINVAL;
+ }
+
+ is_metadata = ecryptfs_is_page_in_metadata(ecryptfs_data, offset);
+ if (is_metadata == true) {
+ pr_debug("ecryptfs metadata, bypassing ICE\n");
+ *is_pfe = false;
+ return -EPERM;
+ }
+
+ key_info->key = ecryptfs_get_key(ecryptfs_data);
+ if (!key_info->key) {
+ pr_err("could not parse key from ecryptfs\n");
+ return -EINVAL;
+ }
+
+ key_info->key_size = ecryptfs_get_key_size(ecryptfs_data);
+ if (!key_info->key_size) {
+ pr_err("could not parse key size from ecryptfs\n");
+ return -EINVAL;
+ }
+
+ key_info->salt = ecryptfs_get_salt(ecryptfs_data);
+ if (!key_info->salt) {
+ pr_err("could not parse salt from ecryptfs\n");
+ return -EINVAL;
+ }
+
+ key_info->salt_size = ecryptfs_get_salt_size(ecryptfs_data);
+ if (!key_info->salt_size) {
+ pr_err("could not parse salt size from ecryptfs\n");
+ return -EINVAL;
+ }
+
+ ret = pfk_ecryptfs_parse_cipher(ecryptfs_data, algo);
+ if (ret != 0) {
+ pr_err("not supported cipher\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+/**
+ * pfk_ecryptfs_allow_merge_bio() - Check if 2 bios can be merged.
+ *
+ * Should be invoked by upper pfk layer
+ *
+ * @bio1: Pointer to first BIO structure.
+ * @bio2: Pointer to second BIO structure.
+ * @inode1: Pointer to inode from first bio
+ * @inode2: Pointer to inode from second bio
+ *
+ * Prevent merging of BIOs from encrypted and non-encrypted
+ * files, or files encrypted with different key.
+ * Also prevent non encrypted and encrypted data from the same file
+ * to be merged (ecryptfs header if stored inside file should be non
+ * encrypted)
+ *
+ * Return: true if the BIOs allowed to be merged, false
+ * otherwise.
+ */
+bool pfk_ecryptfs_allow_merge_bio(const struct bio *bio1,
+ const struct bio *bio2, const struct inode *inode1,
+ const struct inode *inode2)
+{
+ int ret;
+ void *ecryptfs_data1 = NULL;
+ void *ecryptfs_data2 = NULL;
+ pgoff_t offset1, offset2;
+
+ /* if there is no ecryptfs pfk, don't disallow merging blocks */
+ if (!pfk_ecryptfs_is_ready())
+ return true;
+
+ if (!inode1 || !inode2)
+ return false;
+
+ ecryptfs_data1 = pfk_ecryptfs_get_data(inode1);
+ ecryptfs_data2 = pfk_ecryptfs_get_data(inode2);
+
+ if (!ecryptfs_data1 || !ecryptfs_data2) {
+ pr_err("internal error, ecryptfs data should not be null");
+ return false;
+ }
+
+ /*
+ * if we have 2 different encrypted files merge is not allowed
+ */
+ if (!ecryptfs_is_data_equal(ecryptfs_data1, ecryptfs_data2))
+ return false;
+
+ /*
+ * at this point both bio's are in the same file which is probably
+ * encrypted, last thing to check is header vs data
+ * We are assuming that we are not working in O_DIRECT mode,
+ * since it is not currently supported by eCryptfs
+ */
+ ret = pfk_ecryptfs_get_page_index(bio1, &offset1);
+ if (ret != 0) {
+ pr_err("could not get page index from bio1, probably bug %d\n",
+ ret);
+ return false;
+ }
+
+ ret = pfk_ecryptfs_get_page_index(bio2, &offset2);
+ if (ret != 0) {
+ pr_err("could not get page index from bio2, bug %d\n", ret);
+ return false;
+ }
+
+ return (ecryptfs_is_page_in_metadata(ecryptfs_data1, offset1) ==
+ ecryptfs_is_page_in_metadata(ecryptfs_data2, offset2));
+}
+
+/**
+ * pfk_ecryptfs_open_cb() - callback function for file open event
+ * @inode: file inode
+ * @data: data provided by eCryptfs
+ *
+ * Will be invoked from eCryptfs in case of file open event
+ */
+static void pfk_ecryptfs_open_cb(struct inode *inode, void *ecryptfs_data)
+{
+ size_t key_size;
+
+ if (!pfk_ecryptfs_is_ready())
+ return;
+
+ if (!inode) {
+ pr_err("inode is null\n");
+ return;
+ }
+
+ key_size = ecryptfs_get_key_size(ecryptfs_data);
+ if (!(key_size)) {
+ pr_err("could not parse key size from ecryptfs\n");
+ return;
+ }
+
+ if (pfk_ecryptfs_parse_cipher(ecryptfs_data, NULL) != 0) {
+ pr_debug("open_cb: not supported cipher\n");
+ return;
+ }
+
+ if (pfk_key_size_to_key_type(key_size, NULL) != 0)
+ return;
+
+ mutex_lock(&pfk_ecryptfs_lock);
+ pfk_ecryptfs_set_data(inode, ecryptfs_data);
+ mutex_unlock(&pfk_ecryptfs_lock);
+}
+
+/**
+ * pfk_ecryptfs_release_cb() - callback function for file release event
+ * @inode: file inode
+ *
+ * Will be invoked from eCryptfs in case of file release event
+ */
+static void pfk_ecryptfs_release_cb(struct inode *inode)
+{
+ const unsigned char *key = NULL;
+ const unsigned char *salt = NULL;
+ size_t key_size = 0;
+ size_t salt_size = 0;
+ void *data = NULL;
+
+ if (!pfk_ecryptfs_is_ready())
+ return;
+
+ if (!inode) {
+ pr_err("inode is null\n");
+ return;
+ }
+
+ data = pfk_ecryptfs_get_data(inode);
+ if (!data) {
+ pr_debug("could not get ecryptfs data from inode\n");
+ return;
+ }
+
+ key = ecryptfs_get_key(data);
+ if (!key) {
+ pr_err("could not parse key from ecryptfs\n");
+ return;
+ }
+
+ key_size = ecryptfs_get_key_size(data);
+ if (!(key_size)) {
+ pr_err("could not parse key size from ecryptfs\n");
+ return;
+ }
+
+ salt = ecryptfs_get_salt(data);
+ if (!salt) {
+ pr_err("could not parse salt from ecryptfs\n");
+ return;
+ }
+
+ salt_size = ecryptfs_get_salt_size(data);
+ if (!salt_size) {
+ pr_err("could not parse salt size from ecryptfs\n");
+ return;
+ }
+
+ pfk_kc_remove_key_with_salt(key, key_size, salt, salt_size);
+
+ mutex_lock(&pfk_ecryptfs_lock);
+ pfk_ecryptfs_set_data(inode, NULL);
+ mutex_unlock(&pfk_ecryptfs_lock);
+}
+
+/*
+ * pfk_ecryptfs_is_cipher_supported_cb() - callback function to determine
+ * whether a particular cipher (stored in ecryptfs_data) is cupported by pfk
+ *
+ * Ecryptfs should invoke this callback whenever it needs to determine whether
+ * pfk supports the particular cipher mode
+ *
+ * @ecryptfs_data: ecryptfs data
+ */
+static bool pfk_ecryptfs_is_cipher_supported_cb(const void *ecryptfs_data)
+{
+ if (!pfk_ecryptfs_is_ready())
+ return false;
+
+ if (!ecryptfs_data)
+ return false;
+
+ return (pfk_ecryptfs_parse_cipher(ecryptfs_data, NULL)) == 0;
+}
+
+/*
+ * pfk_ecryptfs_is_hw_crypt_cb() - callback function that implements a query
+ * by ecryptfs whether PFK supports HW encryption
+ */
+static bool pfk_ecryptfs_is_hw_crypt_cb(void)
+{
+ if (!pfk_ecryptfs_is_ready())
+ return false;
+
+ return true;
+}
+
+/*
+ * pfk_ecryptfs_get_salt_key_size_cb() - callback function to determine
+ * what is the salt size supported by PFK
+ *
+ * @ecryptfs_data: ecryptfs data
+ */
+static size_t pfk_ecryptfs_get_salt_key_size_cb(const void *ecryptfs_data)
+{
+ if (!pfk_ecryptfs_is_ready())
+ return 0;
+
+ if (!pfk_ecryptfs_is_cipher_supported_cb(ecryptfs_data))
+ return 0;
+
+ return PFK_SUPPORTED_SALT_SIZE;
+}
diff --git a/security/pfe/pfk_ecryptfs.h b/security/pfe/pfk_ecryptfs.h
new file mode 100644
index 0000000..bd3a8f28
--- /dev/null
+++ b/security/pfe/pfk_ecryptfs.h
@@ -0,0 +1,39 @@
+/* Copyright (c) 2015-2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _PFK_ECRYPTFS_H_
+#define _PFK_ECRYPTFS_H_
+
+#include <linux/types.h>
+#include <linux/fs.h>
+#include <linux/init.h>
+#include <crypto/ice.h>
+#include "pfk_internal.h"
+
+
+bool pfk_is_ecryptfs_type(const struct inode *inode);
+
+int pfk_ecryptfs_parse_inode(const struct bio *bio,
+ const struct inode *inode,
+ struct pfk_key_info *key_info,
+ enum ice_cryto_algo_mode *algo,
+ bool *is_pfe);
+
+bool pfk_ecryptfs_allow_merge_bio(const struct bio *bio1,
+ const struct bio *bio2, const struct inode *inode1,
+ const struct inode *inode2);
+
+int __init pfk_ecryptfs_init(void);
+
+void pfk_ecryptfs_deinit(void);
+
+#endif /* _PFK_ECRYPTFS_H_ */
diff --git a/security/pfe/pfk_ext4.h b/security/pfe/pfk_ext4.h
deleted file mode 100644
index e39d04d..0000000
--- a/security/pfe/pfk_ext4.h
+++ /dev/null
@@ -1,39 +0,0 @@
-/* Copyright (c) 2015-2018, The Linux Foundation. All rights reserved.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 and
- * only version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
-
-#ifndef _PFK_EXT4_H_
-#define _PFK_EXT4_H_
-
-#include <linux/types.h>
-#include <linux/fs.h>
-#include <crypto/ice.h>
-#include "pfk_internal.h"
-
-bool pfk_is_ext4_type(const struct inode *inode);
-
-int pfk_ext4_parse_inode(const struct bio *bio,
- const struct inode *inode,
- struct pfk_key_info *key_info,
- enum ice_cryto_algo_mode *algo,
- bool *is_pfe,
- unsigned int *data_unit,
- const char *storage_type);
-
-bool pfk_ext4_allow_merge_bio(const struct bio *bio1,
- const struct bio *bio2, const struct inode *inode1,
- const struct inode *inode2);
-
-int __init pfk_ext4_init(void);
-
-void pfk_ext4_deinit(void);
-
-#endif /* _PFK_EXT4_H_ */
diff --git a/security/pfe/pfk_f2fs.c b/security/pfe/pfk_f2fs.c
deleted file mode 100644
index 2076267..0000000
--- a/security/pfe/pfk_f2fs.c
+++ /dev/null
@@ -1,214 +0,0 @@
-/*
- * Copyright (c) 2015-2018, The Linux Foundation. All rights reserved.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 and
- * only version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
-
-/*
- * Per-File-Key (PFK) - f2fs
- *
- * This driver is used for working with EXT4/F2FS crypt extension
- *
- * The key information is stored in node by EXT4/F2FS when file is first opened
- * and will be later accessed by Block Device Driver to actually load the key
- * to encryption hw.
- *
- * PFK exposes API's for loading and removing keys from encryption hw
- * and also API to determine whether 2 adjacent blocks can be agregated by
- * Block Layer in one request to encryption hw.
- *
- */
-
-
-/* Uncomment the line below to enable debug messages */
-#define DEBUG 1
-#define pr_fmt(fmt) "pfk_f2fs [%s]: " fmt, __func__
-
-#include <linux/module.h>
-#include <linux/fs.h>
-#include <linux/errno.h>
-#include <linux/printk.h>
-
-#include "fscrypt_ice.h"
-#include "pfk_f2fs.h"
-
-static bool pfk_f2fs_ready;
-
-/*
- * pfk_f2fs_deinit() - Deinit function, should be invoked by upper PFK layer
- */
-void pfk_f2fs_deinit(void)
-{
- pfk_f2fs_ready = false;
-}
-
-/*
- * pfk_f2fs_init() - Init function, should be invoked by upper PFK layer
- */
-int __init pfk_f2fs_init(void)
-{
- pfk_f2fs_ready = true;
- pr_info("PFK F2FS inited successfully\n");
-
- return 0;
-}
-
-/**
- * pfk_f2fs_is_ready() - driver is initialized and ready.
- *
- * Return: true if the driver is ready.
- */
-static inline bool pfk_f2fs_is_ready(void)
-{
- return pfk_f2fs_ready;
-}
-
-/**
- * pfk_is_f2fs_type() - return true if inode belongs to ICE F2FS PFE
- * @inode: inode pointer
- */
-bool pfk_is_f2fs_type(const struct inode *inode)
-{
- if (!pfe_is_inode_filesystem_type(inode, "f2fs"))
- return false;
-
- return fscrypt_should_be_processed_by_ice(inode);
-}
-
-/**
- * pfk_f2fs_parse_cipher() - parse cipher from inode to enum
- * @inode: inode
- * @algo: pointer to store the output enum (can be null)
- *
- * return 0 in case of success, error otherwise (i.e not supported cipher)
- */
-static int pfk_f2fs_parse_cipher(const struct inode *inode,
- enum ice_cryto_algo_mode *algo)
-{
- /*
- * currently only AES XTS algo is supported
- * in the future, table with supported ciphers might
- * be introduced
- */
- if (!inode)
- return -EINVAL;
-
- if (!fscrypt_is_aes_xts_cipher(inode)) {
- pr_err("f2fs alghoritm is not supported by pfk\n");
- return -EINVAL;
- }
-
- if (algo)
- *algo = ICE_CRYPTO_ALGO_MODE_AES_XTS;
-
- return 0;
-}
-
-
-int pfk_f2fs_parse_inode(const struct bio *bio,
- const struct inode *inode,
- struct pfk_key_info *key_info,
- enum ice_cryto_algo_mode *algo,
- bool *is_pfe,
- unsigned int *data_unit,
- const char *storage_type)
-{
- int ret = 0;
-
- if (!is_pfe)
- return -EINVAL;
-
- /*
- * only a few errors below can indicate that
- * this function was not invoked within PFE context,
- * otherwise we will consider it PFE
- */
- *is_pfe = true;
-
- /* Update the dun based upon storage type.
- * Right now both UFS and eMMC storage uses 4KB dun
- * for F2FS
- */
- if (storage_type && data_unit) {
- if (!memcmp(storage_type, "ufs", strlen("ufs")) ||
- !memcmp(storage_type, "sdcc", strlen("sdcc")))
- *data_unit = 1 << ICE_CRYPTO_DATA_UNIT_4_KB;
- else
- return -EINVAL;
- }
-
- if (!pfk_f2fs_is_ready())
- return -ENODEV;
-
- if (!inode)
- return -EINVAL;
-
- if (!key_info)
- return -EINVAL;
-
- key_info->key = fscrypt_get_ice_encryption_key(inode);
- if (!key_info->key) {
- pr_err("could not parse key from f2fs\n");
- return -EINVAL;
- }
-
- key_info->key_size = fscrypt_get_ice_encryption_key_size(inode);
- if (!key_info->key_size) {
- pr_err("could not parse key size from f2fs\n");
- return -EINVAL;
- }
-
- key_info->salt = fscrypt_get_ice_encryption_salt(inode);
- if (!key_info->salt) {
- pr_err("could not parse salt from f2fs\n");
- return -EINVAL;
- }
-
- key_info->salt_size = fscrypt_get_ice_encryption_salt_size(inode);
- if (!key_info->salt_size) {
- pr_err("could not parse salt size from f2fs\n");
- return -EINVAL;
- }
-
- ret = pfk_f2fs_parse_cipher(inode, algo);
- if (ret != 0) {
- pr_err("not supported cipher\n");
- return ret;
- }
-
- return 0;
-}
-
-bool pfk_f2fs_allow_merge_bio(const struct bio *bio1,
- const struct bio *bio2, const struct inode *inode1,
- const struct inode *inode2)
-{
- bool mergeable;
-
- /* if there is no f2fs pfk, don't disallow merging blocks */
- if (!pfk_f2fs_is_ready())
- return true;
-
- if (!inode1 || !inode2)
- return false;
-
- mergeable = fscrypt_is_ice_encryption_info_equal(inode1, inode2);
- if (!mergeable)
- return false;
-
-
- /* ICE allows only consecutive iv_key stream. */
- if (!bio_dun(bio1) && !bio_dun(bio2))
- return true;
- else if (!bio_dun(bio1) || !bio_dun(bio2))
- return false;
-
- return bio_end_dun(bio1) == bio_dun(bio2);
-}
diff --git a/security/pfe/pfk_f2fs.h b/security/pfe/pfk_f2fs.h
deleted file mode 100644
index 2e0c21d..0000000
--- a/security/pfe/pfk_f2fs.h
+++ /dev/null
@@ -1,39 +0,0 @@
-/* Copyright (c) 2015-2018, The Linux Foundation. All rights reserved.
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License version 2 and
- * only version 2 as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- */
-
-#ifndef _PFK_F2FS_H_
-#define _PFK_F2FS_H_
-
-#include <linux/types.h>
-#include <linux/fs.h>
-#include <crypto/ice.h>
-#include "pfk_internal.h"
-
-bool pfk_is_f2fs_type(const struct inode *inode);
-
-int pfk_f2fs_parse_inode(const struct bio *bio,
- const struct inode *inode,
- struct pfk_key_info *key_info,
- enum ice_cryto_algo_mode *algo,
- bool *is_pfe,
- unsigned int *data_unit,
- const char *storage_type);
-
-bool pfk_f2fs_allow_merge_bio(const struct bio *bio1,
- const struct bio *bio2, const struct inode *inode1,
- const struct inode *inode2);
-
-int __init pfk_f2fs_init(void);
-
-void pfk_f2fs_deinit(void);
-
-#endif /* _PFK_F2FS_H_ */
diff --git a/security/pfe/pfk_fscrypt.c b/security/pfe/pfk_fscrypt.c
new file mode 100644
index 0000000..e7a88ab
--- /dev/null
+++ b/security/pfe/pfk_fscrypt.c
@@ -0,0 +1,199 @@
+/*
+ * Copyright (c) 2015-2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+/*
+ * Per-File-Key (PFK) - FSCRYPT
+ *
+ * This driver is used for working with F2FS crypt extension
+ *
+ * The key information is stored in node by F2FS when file is first opened
+ * and will be later accessed by Block Device Driver to actually load the key
+ * to encryption hw.
+ *
+ * PFK exposes API's for loading and removing keys from encryption hw
+ * and also API to determine whether 2 adjacent blocks can be agregated by
+ * Block Layer in one request to encryption hw.
+ *
+ */
+
+
+/* Uncomment the line below to enable debug messages */
+#define DEBUG 1
+#define pr_fmt(fmt) "pfk_fscrypt [%s]: " fmt, __func__
+
+#include <linux/module.h>
+#include <linux/fs.h>
+#include <linux/errno.h>
+#include <linux/printk.h>
+
+#include "fscrypt_ice.h"
+#include "pfk_fscrypt.h"
+
+static bool pfk_fscrypt_ready;
+
+/*
+ * pfk_fscrypt_deinit() - Deinit function, should be invoked by upper PFK layer
+ */
+void pfk_fscrypt_deinit(void)
+{
+ pfk_fscrypt_ready = false;
+}
+
+/*
+ * pfk_fscrypt_init() - Init function, should be invoked by upper PFK layer
+ */
+int __init pfk_fscrypt_init(void)
+{
+ pfk_fscrypt_ready = true;
+ pr_info("PFK FSCRYPT inited successfully\n");
+
+ return 0;
+}
+
+/**
+ * pfk_fscrypt_is_ready() - driver is initialized and ready.
+ *
+ * Return: true if the driver is ready.
+ */
+static inline bool pfk_fscrypt_is_ready(void)
+{
+ return pfk_fscrypt_ready;
+}
+
+/**
+ * pfk_is_f2fs_type() - return true if inode belongs to ICE F2FS PFE
+ * @inode: inode pointer
+ */
+bool pfk_is_fscrypt_type(const struct inode *inode)
+{
+ if (!pfe_is_inode_filesystem_type(inode, "f2fs"))
+ return false;
+
+ return fscrypt_should_be_processed_by_ice(inode);
+}
+
+/**
+ * pfk_fscrypt_parse_cipher() - parse cipher from inode to enum
+ * @inode: inode
+ * @algo: pointer to store the output enum (can be null)
+ *
+ * return 0 in case of success, error otherwise (i.e not supported cipher)
+ */
+static int pfk_fscrypt_parse_cipher(const struct inode *inode,
+ enum ice_cryto_algo_mode *algo)
+{
+ /*
+ * currently only AES XTS algo is supported
+ * in the future, table with supported ciphers might
+ * be introduced
+ */
+ if (!inode)
+ return -EINVAL;
+
+ if (!fscrypt_is_aes_xts_cipher(inode)) {
+ pr_err("f2fs alghoritm is not supported by pfk\n");
+ return -EINVAL;
+ }
+
+ if (algo)
+ *algo = ICE_CRYPTO_ALGO_MODE_AES_XTS;
+
+ return 0;
+}
+
+
+int pfk_fscrypt_parse_inode(const struct bio *bio,
+ const struct inode *inode,
+ struct pfk_key_info *key_info,
+ enum ice_cryto_algo_mode *algo,
+ bool *is_pfe)
+{
+ int ret = 0;
+
+ if (!is_pfe)
+ return -EINVAL;
+
+ /*
+ * only a few errors below can indicate that
+ * this function was not invoked within PFE context,
+ * otherwise we will consider it PFE
+ */
+ *is_pfe = true;
+
+ if (!pfk_fscrypt_is_ready())
+ return -ENODEV;
+
+ if (!inode)
+ return -EINVAL;
+
+ if (!key_info)
+ return -EINVAL;
+
+ key_info->key = fscrypt_get_ice_encryption_key(inode);
+ if (!key_info->key) {
+ pr_err("could not parse key from f2fs\n");
+ return -EINVAL;
+ }
+
+ key_info->key_size = fscrypt_get_ice_encryption_key_size(inode);
+ if (!key_info->key_size) {
+ pr_err("could not parse key size from f2fs\n");
+ return -EINVAL;
+ }
+
+ key_info->salt = fscrypt_get_ice_encryption_salt(inode);
+ if (!key_info->salt) {
+ pr_err("could not parse salt from f2fs\n");
+ return -EINVAL;
+ }
+
+ key_info->salt_size = fscrypt_get_ice_encryption_salt_size(inode);
+ if (!key_info->salt_size) {
+ pr_err("could not parse salt size from f2fs\n");
+ return -EINVAL;
+ }
+
+ ret = pfk_fscrypt_parse_cipher(inode, algo);
+ if (ret != 0) {
+ pr_err("not supported cipher\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+bool pfk_fscrypt_allow_merge_bio(const struct bio *bio1,
+ const struct bio *bio2, const struct inode *inode1,
+ const struct inode *inode2)
+{
+ bool mergeable;
+
+ /* if there is no fscrypt pfk, don't disallow merging blocks */
+ if (!pfk_fscrypt_is_ready())
+ return true;
+
+ if (!inode1 || !inode2)
+ return false;
+
+ mergeable = fscrypt_is_ice_encryption_info_equal(inode1, inode2);
+ if (!mergeable)
+ return false;
+
+ /* ICE allows only consecutive iv_key stream. */
+ if (!bio_dun(bio1) && !bio_dun(bio2))
+ return true;
+ else if(!bio_dun(bio1) || !bio_dun(bio2))
+ return false;
+
+ return bio_end_dun(bio1) == bio_dun(bio2);
+}
diff --git a/security/pfe/pfk_fscrypt.h b/security/pfe/pfk_fscrypt.h
new file mode 100644
index 0000000..f0a6242
--- /dev/null
+++ b/security/pfe/pfk_fscrypt.h
@@ -0,0 +1,37 @@
+/* Copyright (c) 2015-2017, The Linux Foundation. All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 and
+ * only version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _PFK_FSCRYPT_H_
+#define _PFK_FSCRYPT_H_
+
+#include <linux/types.h>
+#include <linux/fs.h>
+#include <crypto/ice.h>
+#include "pfk_internal.h"
+
+bool pfk_is_fscrypt_type(const struct inode *inode);
+
+int pfk_fscrypt_parse_inode(const struct bio *bio,
+ const struct inode *inode,
+ struct pfk_key_info *key_info,
+ enum ice_cryto_algo_mode *algo,
+ bool *is_pfe);
+
+bool pfk_fscrypt_allow_merge_bio(const struct bio *bio1,
+ const struct bio *bio2, const struct inode *inode1,
+ const struct inode *inode2);
+
+int __init pfk_fscrypt_init(void);
+
+void pfk_fscrypt_deinit(void);
+
+#endif /* _PFK_FSCRYPT_H_ */
diff --git a/security/pfe/pfk_ice.c b/security/pfe/pfk_ice.c
index 958075b..16ed516 100644
--- a/security/pfe/pfk_ice.c
+++ b/security/pfe/pfk_ice.c
@@ -26,7 +26,11 @@
#include "pfk_ice.h"
-#define TZ_ES_CONFIG_SET_ICE_KEY 0x4
+/**********************************/
+/** global definitions **/
+/**********************************/
+
+#define TZ_ES_SET_ICE_KEY 0x2
#define TZ_ES_INVALIDATE_ICE_KEY 0x3
/* index 0 and 1 is reserved for FDE */
@@ -34,45 +38,44 @@
#define MAX_ICE_KEY_INDEX 31
-#define TZ_ES_CONFIG_SET_ICE_KEY_ID \
- TZ_SYSCALL_CREATE_SMC_ID(TZ_OWNER_SIP, TZ_SVC_ES, \
- TZ_ES_CONFIG_SET_ICE_KEY)
+
+#define TZ_ES_SET_ICE_KEY_ID \
+ TZ_SYSCALL_CREATE_SMC_ID(TZ_OWNER_SIP, TZ_SVC_ES, TZ_ES_SET_ICE_KEY)
+
#define TZ_ES_INVALIDATE_ICE_KEY_ID \
TZ_SYSCALL_CREATE_SMC_ID(TZ_OWNER_SIP, \
TZ_SVC_ES, TZ_ES_INVALIDATE_ICE_KEY)
-#define TZ_ES_CONFIG_SET_ICE_KEY_PARAM_ID \
+
+#define TZ_ES_SET_ICE_KEY_PARAM_ID \
TZ_SYSCALL_CREATE_PARAM_ID_5( \
TZ_SYSCALL_PARAM_TYPE_VAL, \
TZ_SYSCALL_PARAM_TYPE_BUF_RW, TZ_SYSCALL_PARAM_TYPE_VAL, \
- TZ_SYSCALL_PARAM_TYPE_VAL, TZ_SYSCALL_PARAM_TYPE_VAL)
+ TZ_SYSCALL_PARAM_TYPE_BUF_RW, TZ_SYSCALL_PARAM_TYPE_VAL)
#define TZ_ES_INVALIDATE_ICE_KEY_PARAM_ID \
TZ_SYSCALL_CREATE_PARAM_ID_1( \
TZ_SYSCALL_PARAM_TYPE_VAL)
-#define ICE_BUFFER_SIZE 64
+#define ICE_KEY_SIZE 32
+#define ICE_SALT_SIZE 32
-enum {
- TZ_CIPHER_MODE_XTS_128 = 0,
- TZ_CIPHER_MODE_CBC_128 = 1,
- TZ_CIPHER_MODE_XTS_256 = 3,
- TZ_CIPHER_MODE_CBC_256 = 4
-};
-
-static uint8_t ice_buffer[ICE_BUFFER_SIZE];
+static uint8_t ice_key[ICE_KEY_SIZE];
+static uint8_t ice_salt[ICE_KEY_SIZE];
int qti_pfk_ice_set_key(uint32_t index, uint8_t *key, uint8_t *salt,
- char *storage_type, unsigned int data_unit)
+ char *storage_type)
{
struct scm_desc desc = {0};
int ret, ret1;
- char *tzbuf = (char *)ice_buffer;
+ char *tzbuf_key = (char *)ice_key;
+ char *tzbuf_salt = (char *)ice_salt;
char *s_type = storage_type;
uint32_t smc_id = 0;
- u32 size = ICE_BUFFER_SIZE / 2;
+ u32 tzbuflen_key = sizeof(ice_key);
+ u32 tzbuflen_salt = sizeof(ice_salt);
if (index < MIN_ICE_KEY_INDEX || index > MAX_ICE_KEY_INDEX) {
pr_err("%s Invalid index %d\n", __func__, index);
@@ -83,7 +86,7 @@ int qti_pfk_ice_set_key(uint32_t index, uint8_t *key, uint8_t *salt,
return -EINVAL;
}
- if (!tzbuf) {
+ if (!tzbuf_key || !tzbuf_salt) {
pr_err("%s No Memory\n", __func__);
return -ENOMEM;
}
@@ -93,21 +96,23 @@ int qti_pfk_ice_set_key(uint32_t index, uint8_t *key, uint8_t *salt,
return -EINVAL;
}
- memset(tzbuf, 0, ICE_BUFFER_SIZE);
+ memset(tzbuf_key, 0, tzbuflen_key);
+ memset(tzbuf_salt, 0, tzbuflen_salt);
- memcpy(ice_buffer, key, size);
- memcpy(ice_buffer + size, salt, size);
+ memcpy(ice_key, key, tzbuflen_key);
+ memcpy(ice_salt, salt, tzbuflen_salt);
- dmac_flush_range(tzbuf, tzbuf + ICE_BUFFER_SIZE);
+ dmac_flush_range(tzbuf_key, tzbuf_key + tzbuflen_key);
+ dmac_flush_range(tzbuf_salt, tzbuf_salt + tzbuflen_salt);
- smc_id = TZ_ES_CONFIG_SET_ICE_KEY_ID;
+ smc_id = TZ_ES_SET_ICE_KEY_ID;
- desc.arginfo = TZ_ES_CONFIG_SET_ICE_KEY_PARAM_ID;
+ desc.arginfo = TZ_ES_SET_ICE_KEY_PARAM_ID;
desc.args[0] = index;
- desc.args[1] = virt_to_phys(tzbuf);
- desc.args[2] = ICE_BUFFER_SIZE;
- desc.args[3] = TZ_CIPHER_MODE_XTS_256;
- desc.args[4] = data_unit;
+ desc.args[1] = virt_to_phys(tzbuf_key);
+ desc.args[2] = tzbuflen_key;
+ desc.args[3] = virt_to_phys(tzbuf_salt);
+ desc.args[4] = tzbuflen_salt;
ret = qcom_ice_setup_ice_hw((const char *)s_type, true);
diff --git a/security/pfe/pfk_ice.h b/security/pfe/pfk_ice.h
index 8fd0d83..31772e7 100644
--- a/security/pfe/pfk_ice.h
+++ b/security/pfe/pfk_ice.h
@@ -26,7 +26,7 @@ int pfk_ice_init(void);
int pfk_ice_deinit(void);
int qti_pfk_ice_set_key(uint32_t index, uint8_t *key, uint8_t *salt,
- char *storage_type, unsigned int data_unit);
+ char *storage_type);
int qti_pfk_ice_invalidate_key(uint32_t index, char *storage_type);
diff --git a/security/pfe/pfk_internal.h b/security/pfe/pfk_internal.h
index 3214327..7a8509c 100644
--- a/security/pfe/pfk_internal.h
+++ b/security/pfe/pfk_internal.h
@@ -29,6 +29,4 @@ int pfk_key_size_to_key_type(size_t key_size,
bool pfe_is_inode_filesystem_type(const struct inode *inode,
const char *fs_type);
-char *inode_to_filename(const struct inode *inode);
-
#endif /* _PFK_INTERNAL_H_ */
diff --git a/security/pfe/pfk_kc.c b/security/pfe/pfk_kc.c
index a8e9909..3065875 100644
--- a/security/pfe/pfk_kc.c
+++ b/security/pfe/pfk_kc.c
@@ -37,7 +37,6 @@
#include "pfk_kc.h"
#include "pfk_ice.h"
-
/** the first available index in ice engine */
#define PFK_KC_STARTING_INDEX 2
@@ -132,16 +131,6 @@ static inline void kc_spin_unlock(void)
}
/**
- * pfk_kc_get_storage_type() - return the hardware storage type.
- *
- * Return: storage type queried during bootup.
- */
-const char *pfk_kc_get_storage_type(void)
-{
- return s_type;
-}
-
-/**
* kc_entry_is_available() - checks whether the entry is available
*
* Return true if it is , false otherwise or if invalid
@@ -399,15 +388,13 @@ static void kc_clear_entry(struct kc_entry *entry)
* @key_size: key_size
* @salt: salt
* @salt_size: salt_size
- * @data_unit: dun size
*
* The previous key is securely released and wiped, the new one is loaded
* to ICE.
* Should be invoked under spinlock
*/
static int kc_update_entry(struct kc_entry *entry, const unsigned char *key,
- size_t key_size, const unsigned char *salt, size_t salt_size,
- unsigned int data_unit)
+ size_t key_size, const unsigned char *salt, size_t salt_size)
{
int ret;
@@ -424,7 +411,7 @@ static int kc_update_entry(struct kc_entry *entry, const unsigned char *key,
kc_spin_unlock();
ret = qti_pfk_ice_set_key(entry->key_index, entry->key,
- entry->salt, s_type, data_unit);
+ entry->salt, s_type);
kc_spin_lock();
return ret;
@@ -447,7 +434,6 @@ int pfk_kc_init(void)
}
kc_ready = true;
kc_spin_unlock();
-
return 0;
}
@@ -459,8 +445,8 @@ int pfk_kc_init(void)
int pfk_kc_deinit(void)
{
int res = pfk_kc_clear();
- kc_ready = false;
+ kc_ready = false;
return res;
}
@@ -490,7 +476,7 @@ int pfk_kc_deinit(void)
*/
int pfk_kc_load_key_start(const unsigned char *key, size_t key_size,
const unsigned char *salt, size_t salt_size, u32 *key_index,
- bool async, unsigned int data_unit)
+ bool async)
{
int ret = 0;
struct kc_entry *entry = NULL;
@@ -555,8 +541,7 @@ int pfk_kc_load_key_start(const unsigned char *key, size_t key_size,
break;
}
case (FREE):
- ret = kc_update_entry(entry, key, key_size, salt, salt_size,
- data_unit);
+ ret = kc_update_entry(entry, key, key_size, salt, salt_size);
if (ret) {
entry->state = SCM_ERROR;
entry->scm_error = ret;
diff --git a/security/pfe/pfk_kc.h b/security/pfe/pfk_kc.h
index 89d40be..901017f 100644
--- a/security/pfe/pfk_kc.h
+++ b/security/pfe/pfk_kc.h
@@ -19,7 +19,7 @@ int pfk_kc_init(void);
int pfk_kc_deinit(void);
int pfk_kc_load_key_start(const unsigned char *key, size_t key_size,
const unsigned char *salt, size_t salt_size, u32 *key_index,
- bool async, unsigned int data_unit);
+ bool async);
void pfk_kc_load_key_end(const unsigned char *key, size_t key_size,
const unsigned char *salt, size_t salt_size);
int pfk_kc_remove_key_with_salt(const unsigned char *key, size_t key_size,
@@ -27,8 +27,6 @@ int pfk_kc_remove_key_with_salt(const unsigned char *key, size_t key_size,
int pfk_kc_remove_key(const unsigned char *key, size_t key_size);
int pfk_kc_clear(void);
void pfk_kc_clear_on_reset(void);
-const char *pfk_kc_get_storage_type(void);
extern char *saved_command_line;
-
#endif /* PFK_KC_H_ */
diff --git a/security/selinux/include/objsec.h b/security/selinux/include/objsec.h
index 60cdcf4..717b9ca 100644
--- a/security/selinux/include/objsec.h
+++ b/security/selinux/include/objsec.h
@@ -25,8 +25,6 @@
#include <linux/in.h>
#include <linux/spinlock.h>
#include <net/net_namespace.h>
-//#include "flask.h"
-//#include "avc.h"
#include "security.h"
struct task_security_struct {
@@ -53,8 +51,7 @@ struct inode_security_struct {
u32 sid; /* SID of this object */
u16 sclass; /* security class of this object */
unsigned char initialized; /* initialization flag */
- u32 tag; /* Per-File-Encryption tag */
- void *pfk_data; /* Per-File-Key data from ecryptfs */
+ void *pfk_data; /* Per-File-Key data from ecryptfs */
struct mutex lock;
};
diff --git a/security/selinux/ss/policydb.c b/security/selinux/ss/policydb.c
index d719db4..175e4dc 100644
--- a/security/selinux/ss/policydb.c
+++ b/security/selinux/ss/policydb.c
@@ -1097,7 +1097,7 @@ static int str_read(char **strp, gfp_t flags, void *fp, u32 len)
if ((len == 0) || (len == (u32)-1))
return -EINVAL;
- str = kmalloc(len + 1, flags);
+ str = kmalloc(len + 1, flags | __GFP_NOWARN);
if (!str)
return -ENOMEM;
diff --git a/security/smack/smack_lsm.c b/security/smack/smack_lsm.c
index 530ed9b..13b0d56 100644
--- a/security/smack/smack_lsm.c
+++ b/security/smack/smack_lsm.c
@@ -2307,6 +2307,7 @@ static void smack_task_to_inode(struct task_struct *p, struct inode *inode)
struct smack_known *skp = smk_of_task_struct(p);
isp->smk_inode = skp;
+ isp->smk_flags |= SMK_INODE_INSTANT;
}
/*
@@ -3965,15 +3966,19 @@ static int smack_socket_sock_rcv_skb(struct sock *sk, struct sk_buff *skb)
struct smack_known *skp = NULL;
int rc = 0;
struct smk_audit_info ad;
+ u16 family = sk->sk_family;
#ifdef CONFIG_AUDIT
struct lsm_network_audit net;
#endif
#if IS_ENABLED(CONFIG_IPV6)
struct sockaddr_in6 sadd;
int proto;
+
+ if (family == PF_INET6 && skb->protocol == htons(ETH_P_IP))
+ family = PF_INET;
#endif /* CONFIG_IPV6 */
- switch (sk->sk_family) {
+ switch (family) {
case PF_INET:
#ifdef CONFIG_SECURITY_SMACK_NETFILTER
/*
@@ -3991,7 +3996,7 @@ static int smack_socket_sock_rcv_skb(struct sock *sk, struct sk_buff *skb)
*/
netlbl_secattr_init(&secattr);
- rc = netlbl_skbuff_getattr(skb, sk->sk_family, &secattr);
+ rc = netlbl_skbuff_getattr(skb, family, &secattr);
if (rc == 0)
skp = smack_from_secattr(&secattr, ssp);
else
@@ -4004,7 +4009,7 @@ static int smack_socket_sock_rcv_skb(struct sock *sk, struct sk_buff *skb)
#endif
#ifdef CONFIG_AUDIT
smk_ad_init_net(&ad, __func__, LSM_AUDIT_DATA_NET, &net);
- ad.a.u.net->family = sk->sk_family;
+ ad.a.u.net->family = family;
ad.a.u.net->netif = skb->skb_iif;
ipv4_skb_to_auditdata(skb, &ad.a, NULL);
#endif
@@ -4018,7 +4023,7 @@ static int smack_socket_sock_rcv_skb(struct sock *sk, struct sk_buff *skb)
rc = smk_bu_note("IPv4 delivery", skp, ssp->smk_in,
MAY_WRITE, rc);
if (rc != 0)
- netlbl_skbuff_err(skb, sk->sk_family, rc, 0);
+ netlbl_skbuff_err(skb, family, rc, 0);
break;
#if IS_ENABLED(CONFIG_IPV6)
case PF_INET6:
@@ -4034,7 +4039,7 @@ static int smack_socket_sock_rcv_skb(struct sock *sk, struct sk_buff *skb)
skp = smack_net_ambient;
#ifdef CONFIG_AUDIT
smk_ad_init_net(&ad, __func__, LSM_AUDIT_DATA_NET, &net);
- ad.a.u.net->family = sk->sk_family;
+ ad.a.u.net->family = family;
ad.a.u.net->netif = skb->skb_iif;
ipv6_skb_to_auditdata(skb, &ad.a, NULL);
#endif /* CONFIG_AUDIT */
diff --git a/sound/aoa/core/gpio-feature.c b/sound/aoa/core/gpio-feature.c
index 7196008..6555742 100644
--- a/sound/aoa/core/gpio-feature.c
+++ b/sound/aoa/core/gpio-feature.c
@@ -88,8 +88,10 @@ static struct device_node *get_gpio(char *name,
}
reg = of_get_property(np, "reg", NULL);
- if (!reg)
+ if (!reg) {
+ of_node_put(np);
return NULL;
+ }
*gpioptr = *reg;
diff --git a/sound/core/control.c b/sound/core/control.c
index 995cde4..511368f 100644
--- a/sound/core/control.c
+++ b/sound/core/control.c
@@ -346,6 +346,40 @@ static int snd_ctl_find_hole(struct snd_card *card, unsigned int count)
return 0;
}
+/* add a new kcontrol object; call with card->controls_rwsem locked */
+static int __snd_ctl_add(struct snd_card *card, struct snd_kcontrol *kcontrol)
+{
+ struct snd_ctl_elem_id id;
+ unsigned int idx;
+ unsigned int count;
+
+ id = kcontrol->id;
+ if (id.index > UINT_MAX - kcontrol->count)
+ return -EINVAL;
+
+ if (snd_ctl_find_id(card, &id)) {
+ dev_err(card->dev,
+ "control %i:%i:%i:%s:%i is already present\n",
+ id.iface, id.device, id.subdevice, id.name, id.index);
+ return -EBUSY;
+ }
+
+ if (snd_ctl_find_hole(card, kcontrol->count) < 0)
+ return -ENOMEM;
+
+ list_add_tail(&kcontrol->list, &card->controls);
+ card->controls_count += kcontrol->count;
+ kcontrol->id.numid = card->last_numid + 1;
+ card->last_numid += kcontrol->count;
+
+ id = kcontrol->id;
+ count = kcontrol->count;
+ for (idx = 0; idx < count; idx++, id.index++, id.numid++)
+ snd_ctl_notify(card, SNDRV_CTL_EVENT_MASK_ADD, &id);
+
+ return 0;
+}
+
/**
* snd_ctl_add - add the control instance to the card
* @card: the card instance
@@ -362,45 +396,18 @@ static int snd_ctl_find_hole(struct snd_card *card, unsigned int count)
*/
int snd_ctl_add(struct snd_card *card, struct snd_kcontrol *kcontrol)
{
- struct snd_ctl_elem_id id;
- unsigned int idx;
- unsigned int count;
int err = -EINVAL;
if (! kcontrol)
return err;
if (snd_BUG_ON(!card || !kcontrol->info))
goto error;
- id = kcontrol->id;
- if (id.index > UINT_MAX - kcontrol->count)
- goto error;
down_write(&card->controls_rwsem);
- if (snd_ctl_find_id(card, &id)) {
- up_write(&card->controls_rwsem);
- dev_err(card->dev, "control %i:%i:%i:%s:%i is already present\n",
- id.iface,
- id.device,
- id.subdevice,
- id.name,
- id.index);
- err = -EBUSY;
- goto error;
- }
- if (snd_ctl_find_hole(card, kcontrol->count) < 0) {
- up_write(&card->controls_rwsem);
- err = -ENOMEM;
- goto error;
- }
- list_add_tail(&kcontrol->list, &card->controls);
- card->controls_count += kcontrol->count;
- kcontrol->id.numid = card->last_numid + 1;
- card->last_numid += kcontrol->count;
- id = kcontrol->id;
- count = kcontrol->count;
+ err = __snd_ctl_add(card, kcontrol);
up_write(&card->controls_rwsem);
- for (idx = 0; idx < count; idx++, id.index++, id.numid++)
- snd_ctl_notify(card, SNDRV_CTL_EVENT_MASK_ADD, &id);
+ if (err < 0)
+ goto error;
return 0;
error:
@@ -1354,9 +1361,12 @@ static int snd_ctl_elem_add(struct snd_ctl_file *file,
kctl->tlv.c = snd_ctl_elem_user_tlv;
/* This function manage to free the instance on failure. */
- err = snd_ctl_add(card, kctl);
- if (err < 0)
- return err;
+ down_write(&card->controls_rwsem);
+ err = __snd_ctl_add(card, kctl);
+ if (err < 0) {
+ snd_ctl_free_one(kctl);
+ goto unlock;
+ }
offset = snd_ctl_get_ioff(kctl, &info->id);
snd_ctl_build_ioff(&info->id, kctl, offset);
/*
@@ -1367,10 +1377,10 @@ static int snd_ctl_elem_add(struct snd_ctl_file *file,
* which locks the element.
*/
- down_write(&card->controls_rwsem);
card->user_ctl_count++;
- up_write(&card->controls_rwsem);
+ unlock:
+ up_write(&card->controls_rwsem);
return 0;
}
diff --git a/sound/core/control_compat.c b/sound/core/control_compat.c
index 1fa7076..84ee29c 100644
--- a/sound/core/control_compat.c
+++ b/sound/core/control_compat.c
@@ -400,8 +400,7 @@ static int snd_ctl_elem_add_compat(struct snd_ctl_file *file,
if (copy_from_user(&data->id, &data32->id, sizeof(data->id)) ||
copy_from_user(&data->type, &data32->type, 3 * sizeof(u32)))
goto error;
- if (get_user(data->owner, &data32->owner) ||
- get_user(data->type, &data32->type))
+ if (get_user(data->owner, &data32->owner))
goto error;
switch (data->type) {
case SNDRV_CTL_ELEM_TYPE_BOOLEAN:
diff --git a/sound/core/memalloc.c b/sound/core/memalloc.c
index f05cb6a..78ffe44 100644
--- a/sound/core/memalloc.c
+++ b/sound/core/memalloc.c
@@ -239,16 +239,12 @@ int snd_dma_alloc_pages_fallback(int type, struct device *device, size_t size,
int err;
while ((err = snd_dma_alloc_pages(type, device, size, dmab)) < 0) {
- size_t aligned_size;
if (err != -ENOMEM)
return err;
if (size <= PAGE_SIZE)
return -ENOMEM;
- aligned_size = PAGE_SIZE << get_order(size);
- if (size != aligned_size)
- size = aligned_size;
- else
- size >>= 1;
+ size >>= 1;
+ size = PAGE_SIZE << get_order(size);
}
if (! dmab->area)
return -ENOMEM;
diff --git a/sound/core/pcm.c b/sound/core/pcm.c
index d79a04e..8239ebc 100644
--- a/sound/core/pcm.c
+++ b/sound/core/pcm.c
@@ -25,6 +25,7 @@
#include <linux/time.h>
#include <linux/mutex.h>
#include <linux/device.h>
+#include <linux/nospec.h>
#include <sound/core.h>
#include <sound/minors.h>
#include <sound/pcm.h>
@@ -125,6 +126,7 @@ static int snd_pcm_control_ioctl(struct snd_card *card,
return -EFAULT;
if (stream < 0 || stream > 1)
return -EINVAL;
+ stream = array_index_nospec(stream, 2);
if (get_user(subdevice, &info->subdevice))
return -EFAULT;
mutex_lock(®ister_mutex);
diff --git a/sound/core/pcm_compat.c b/sound/core/pcm_compat.c
index e1512ae..0c81e265 100644
--- a/sound/core/pcm_compat.c
+++ b/sound/core/pcm_compat.c
@@ -426,6 +426,8 @@ static int snd_pcm_ioctl_xfern_compat(struct snd_pcm_substream *substream,
return -ENOTTY;
if (substream->stream != dir)
return -EINVAL;
+ if (substream->runtime->status->state == SNDRV_PCM_STATE_OPEN)
+ return -EBADFD;
if ((ch = substream->runtime->channels) > 128)
return -EINVAL;
diff --git a/sound/core/pcm_lib.c b/sound/core/pcm_lib.c
index 9ccf6a5..2bae2a3 100644
--- a/sound/core/pcm_lib.c
+++ b/sound/core/pcm_lib.c
@@ -652,27 +652,33 @@ EXPORT_SYMBOL(snd_interval_refine);
static int snd_interval_refine_first(struct snd_interval *i)
{
+ const unsigned int last_max = i->max;
+
if (snd_BUG_ON(snd_interval_empty(i)))
return -EINVAL;
if (snd_interval_single(i))
return 0;
i->max = i->min;
- i->openmax = i->openmin;
- if (i->openmax)
+ if (i->openmin)
i->max++;
+ /* only exclude max value if also excluded before refine */
+ i->openmax = (i->openmax && i->max >= last_max);
return 1;
}
static int snd_interval_refine_last(struct snd_interval *i)
{
+ const unsigned int last_min = i->min;
+
if (snd_BUG_ON(snd_interval_empty(i)))
return -EINVAL;
if (snd_interval_single(i))
return 0;
i->min = i->max;
- i->openmin = i->openmax;
- if (i->openmin)
+ if (i->openmax)
i->min--;
+ /* only exclude min value if also excluded before refine */
+ i->openmin = (i->openmin && i->min <= last_min);
return 1;
}
diff --git a/sound/core/pcm_native.c b/sound/core/pcm_native.c
index 019f60b..a6fdf3a 100644
--- a/sound/core/pcm_native.c
+++ b/sound/core/pcm_native.c
@@ -36,6 +36,7 @@
#include <sound/timer.h>
#include <sound/minors.h>
#include <linux/uio.h>
+#include <linux/delay.h>
/*
* Compatibility
@@ -79,12 +80,12 @@ static DECLARE_RWSEM(snd_pcm_link_rwsem);
* and this may lead to a deadlock when the code path takes read sem
* twice (e.g. one in snd_pcm_action_nonatomic() and another in
* snd_pcm_stream_lock()). As a (suboptimal) workaround, let writer to
- * spin until it gets the lock.
+ * sleep until all the readers are completed without blocking by writer.
*/
-static inline void down_write_nonblock(struct rw_semaphore *lock)
+static inline void down_write_nonfifo(struct rw_semaphore *lock)
{
while (!down_write_trylock(lock))
- cond_resched();
+ msleep(1);
}
/**
@@ -1849,7 +1850,7 @@ static int snd_pcm_link(struct snd_pcm_substream *substream, int fd)
res = -ENOMEM;
goto _nolock;
}
- down_write_nonblock(&snd_pcm_link_rwsem);
+ down_write_nonfifo(&snd_pcm_link_rwsem);
write_lock_irq(&snd_pcm_link_rwlock);
if (substream->runtime->status->state == SNDRV_PCM_STATE_OPEN ||
substream->runtime->status->state != substream1->runtime->status->state ||
@@ -1896,7 +1897,7 @@ static int snd_pcm_unlink(struct snd_pcm_substream *substream)
struct snd_pcm_substream *s;
int res = 0;
- down_write_nonblock(&snd_pcm_link_rwsem);
+ down_write_nonfifo(&snd_pcm_link_rwsem);
write_lock_irq(&snd_pcm_link_rwlock);
if (!snd_pcm_stream_linked(substream)) {
res = -EALREADY;
@@ -2249,7 +2250,8 @@ int snd_pcm_hw_constraints_complete(struct snd_pcm_substream *substream)
static void pcm_release_private(struct snd_pcm_substream *substream)
{
- snd_pcm_unlink(substream);
+ if (snd_pcm_stream_linked(substream))
+ snd_pcm_unlink(substream);
}
void snd_pcm_release_substream(struct snd_pcm_substream *substream)
@@ -2765,6 +2767,7 @@ static int snd_pcm_sync_ptr(struct snd_pcm_substream *substream,
sync_ptr.s.status.hw_ptr = status->hw_ptr;
sync_ptr.s.status.tstamp = status->tstamp;
sync_ptr.s.status.suspended_state = status->suspended_state;
+ sync_ptr.s.status.audio_tstamp = status->audio_tstamp;
snd_pcm_stream_unlock_irq(substream);
if (copy_to_user(_sync_ptr, &sync_ptr, sizeof(sync_ptr)))
return -EFAULT;
diff --git a/sound/core/rawmidi.c b/sound/core/rawmidi.c
index 180261d..5b04850 100644
--- a/sound/core/rawmidi.c
+++ b/sound/core/rawmidi.c
@@ -636,11 +636,9 @@ static int snd_rawmidi_info_select_user(struct snd_card *card,
int snd_rawmidi_output_params(struct snd_rawmidi_substream *substream,
struct snd_rawmidi_params * params)
{
- char *newbuf;
- char *oldbuf;
+ char *newbuf, *oldbuf;
struct snd_rawmidi_runtime *runtime = substream->runtime;
- unsigned long flags;
-
+
if (substream->append && substream->use_count > 1)
return -EBUSY;
snd_rawmidi_drain_output(substream);
@@ -651,22 +649,17 @@ int snd_rawmidi_output_params(struct snd_rawmidi_substream *substream,
return -EINVAL;
}
if (params->buffer_size != runtime->buffer_size) {
- mutex_lock(&runtime->realloc_mutex);
- newbuf = __krealloc(runtime->buffer, params->buffer_size,
- GFP_KERNEL);
- if (!newbuf) {
- mutex_unlock(&runtime->realloc_mutex);
+ newbuf = kmalloc(params->buffer_size, GFP_KERNEL);
+ if (!newbuf)
return -ENOMEM;
- }
- spin_lock_irqsave(&runtime->lock, flags);
+ spin_lock_irq(&runtime->lock);
oldbuf = runtime->buffer;
runtime->buffer = newbuf;
runtime->buffer_size = params->buffer_size;
runtime->avail = runtime->buffer_size;
- spin_unlock_irqrestore(&runtime->lock, flags);
- if (oldbuf != newbuf)
- kfree(oldbuf);
- mutex_unlock(&runtime->realloc_mutex);
+ runtime->appl_ptr = runtime->hw_ptr = 0;
+ spin_unlock_irq(&runtime->lock);
+ kfree(oldbuf);
}
runtime->avail_min = params->avail_min;
substream->active_sensing = !params->no_active_sensing;
@@ -677,10 +670,8 @@ EXPORT_SYMBOL(snd_rawmidi_output_params);
int snd_rawmidi_input_params(struct snd_rawmidi_substream *substream,
struct snd_rawmidi_params * params)
{
- char *newbuf;
- char *oldbuf;
+ char *newbuf, *oldbuf;
struct snd_rawmidi_runtime *runtime = substream->runtime;
- unsigned long flags;
snd_rawmidi_drain_input(substream);
if (params->buffer_size < 32 || params->buffer_size > 1024L * 1024L) {
@@ -690,21 +681,16 @@ int snd_rawmidi_input_params(struct snd_rawmidi_substream *substream,
return -EINVAL;
}
if (params->buffer_size != runtime->buffer_size) {
- mutex_lock(&runtime->realloc_mutex);
- newbuf = __krealloc(runtime->buffer, params->buffer_size,
- GFP_KERNEL);
- if (!newbuf) {
- mutex_unlock(&runtime->realloc_mutex);
+ newbuf = kmalloc(params->buffer_size, GFP_KERNEL);
+ if (!newbuf)
return -ENOMEM;
- }
- spin_lock_irqsave(&runtime->lock, flags);
+ spin_lock_irq(&runtime->lock);
oldbuf = runtime->buffer;
runtime->buffer = newbuf;
runtime->buffer_size = params->buffer_size;
- spin_unlock_irqrestore(&runtime->lock, flags);
- if (oldbuf != newbuf)
- kfree(oldbuf);
- mutex_unlock(&runtime->realloc_mutex);
+ runtime->appl_ptr = runtime->hw_ptr = 0;
+ spin_unlock_irq(&runtime->lock);
+ kfree(oldbuf);
}
runtime->avail_min = params->avail_min;
return 0;
diff --git a/sound/core/seq/oss/seq_oss_event.c b/sound/core/seq/oss/seq_oss_event.c
index c390886..86ca584 100644
--- a/sound/core/seq/oss/seq_oss_event.c
+++ b/sound/core/seq/oss/seq_oss_event.c
@@ -26,6 +26,7 @@
#include <sound/seq_oss_legacy.h>
#include "seq_oss_readq.h"
#include "seq_oss_writeq.h"
+#include <linux/nospec.h>
/*
@@ -287,10 +288,10 @@ note_on_event(struct seq_oss_devinfo *dp, int dev, int ch, int note, int vel, st
{
struct seq_oss_synthinfo *info;
- if (!snd_seq_oss_synth_is_valid(dp, dev))
+ info = snd_seq_oss_synth_info(dp, dev);
+ if (!info)
return -ENXIO;
- info = &dp->synths[dev];
switch (info->arg.event_passing) {
case SNDRV_SEQ_OSS_PROCESS_EVENTS:
if (! info->ch || ch < 0 || ch >= info->nr_voices) {
@@ -298,6 +299,7 @@ note_on_event(struct seq_oss_devinfo *dp, int dev, int ch, int note, int vel, st
return set_note_event(dp, dev, SNDRV_SEQ_EVENT_NOTEON, ch, note, vel, ev);
}
+ ch = array_index_nospec(ch, info->nr_voices);
if (note == 255 && info->ch[ch].note >= 0) {
/* volume control */
int type;
@@ -347,10 +349,10 @@ note_off_event(struct seq_oss_devinfo *dp, int dev, int ch, int note, int vel, s
{
struct seq_oss_synthinfo *info;
- if (!snd_seq_oss_synth_is_valid(dp, dev))
+ info = snd_seq_oss_synth_info(dp, dev);
+ if (!info)
return -ENXIO;
- info = &dp->synths[dev];
switch (info->arg.event_passing) {
case SNDRV_SEQ_OSS_PROCESS_EVENTS:
if (! info->ch || ch < 0 || ch >= info->nr_voices) {
@@ -358,6 +360,7 @@ note_off_event(struct seq_oss_devinfo *dp, int dev, int ch, int note, int vel, s
return set_note_event(dp, dev, SNDRV_SEQ_EVENT_NOTEON, ch, note, vel, ev);
}
+ ch = array_index_nospec(ch, info->nr_voices);
if (info->ch[ch].note >= 0) {
note = info->ch[ch].note;
info->ch[ch].vel = 0;
@@ -381,7 +384,7 @@ note_off_event(struct seq_oss_devinfo *dp, int dev, int ch, int note, int vel, s
static int
set_note_event(struct seq_oss_devinfo *dp, int dev, int type, int ch, int note, int vel, struct snd_seq_event *ev)
{
- if (! snd_seq_oss_synth_is_valid(dp, dev))
+ if (!snd_seq_oss_synth_info(dp, dev))
return -ENXIO;
ev->type = type;
@@ -399,7 +402,7 @@ set_note_event(struct seq_oss_devinfo *dp, int dev, int type, int ch, int note,
static int
set_control_event(struct seq_oss_devinfo *dp, int dev, int type, int ch, int param, int val, struct snd_seq_event *ev)
{
- if (! snd_seq_oss_synth_is_valid(dp, dev))
+ if (!snd_seq_oss_synth_info(dp, dev))
return -ENXIO;
ev->type = type;
diff --git a/sound/core/seq/oss/seq_oss_midi.c b/sound/core/seq/oss/seq_oss_midi.c
index b30b213..9debd1b 100644
--- a/sound/core/seq/oss/seq_oss_midi.c
+++ b/sound/core/seq/oss/seq_oss_midi.c
@@ -29,6 +29,7 @@
#include "../seq_lock.h"
#include <linux/init.h>
#include <linux/slab.h>
+#include <linux/nospec.h>
/*
@@ -315,6 +316,7 @@ get_mididev(struct seq_oss_devinfo *dp, int dev)
{
if (dev < 0 || dev >= dp->max_mididev)
return NULL;
+ dev = array_index_nospec(dev, dp->max_mididev);
return get_mdev(dev);
}
diff --git a/sound/core/seq/oss/seq_oss_synth.c b/sound/core/seq/oss/seq_oss_synth.c
index cd0e0eb..278ebb9 100644
--- a/sound/core/seq/oss/seq_oss_synth.c
+++ b/sound/core/seq/oss/seq_oss_synth.c
@@ -26,6 +26,7 @@
#include <linux/init.h>
#include <linux/module.h>
#include <linux/slab.h>
+#include <linux/nospec.h>
/*
* constants
@@ -339,17 +340,13 @@ snd_seq_oss_synth_cleanup(struct seq_oss_devinfo *dp)
dp->max_synthdev = 0;
}
-/*
- * check if the specified device is MIDI mapped device
- */
-static int
-is_midi_dev(struct seq_oss_devinfo *dp, int dev)
+static struct seq_oss_synthinfo *
+get_synthinfo_nospec(struct seq_oss_devinfo *dp, int dev)
{
if (dev < 0 || dev >= dp->max_synthdev)
- return 0;
- if (dp->synths[dev].is_midi)
- return 1;
- return 0;
+ return NULL;
+ dev = array_index_nospec(dev, SNDRV_SEQ_OSS_MAX_SYNTH_DEVS);
+ return &dp->synths[dev];
}
/*
@@ -359,14 +356,20 @@ static struct seq_oss_synth *
get_synthdev(struct seq_oss_devinfo *dp, int dev)
{
struct seq_oss_synth *rec;
- if (dev < 0 || dev >= dp->max_synthdev)
+ struct seq_oss_synthinfo *info = get_synthinfo_nospec(dp, dev);
+
+ if (!info)
return NULL;
- if (! dp->synths[dev].opened)
+ if (!info->opened)
return NULL;
- if (dp->synths[dev].is_midi)
- return &midi_synth_dev;
- if ((rec = get_sdev(dev)) == NULL)
- return NULL;
+ if (info->is_midi) {
+ rec = &midi_synth_dev;
+ snd_use_lock_use(&rec->use_lock);
+ } else {
+ rec = get_sdev(dev);
+ if (!rec)
+ return NULL;
+ }
if (! rec->opened) {
snd_use_lock_free(&rec->use_lock);
return NULL;
@@ -402,10 +405,8 @@ snd_seq_oss_synth_reset(struct seq_oss_devinfo *dp, int dev)
struct seq_oss_synth *rec;
struct seq_oss_synthinfo *info;
- if (snd_BUG_ON(dev < 0 || dev >= dp->max_synthdev))
- return;
- info = &dp->synths[dev];
- if (! info->opened)
+ info = get_synthinfo_nospec(dp, dev);
+ if (!info || !info->opened)
return;
if (info->sysex)
info->sysex->len = 0; /* reset sysex */
@@ -454,12 +455,14 @@ snd_seq_oss_synth_load_patch(struct seq_oss_devinfo *dp, int dev, int fmt,
const char __user *buf, int p, int c)
{
struct seq_oss_synth *rec;
+ struct seq_oss_synthinfo *info;
int rc;
- if (dev < 0 || dev >= dp->max_synthdev)
+ info = get_synthinfo_nospec(dp, dev);
+ if (!info)
return -ENXIO;
- if (is_midi_dev(dp, dev))
+ if (info->is_midi)
return 0;
if ((rec = get_synthdev(dp, dev)) == NULL)
return -ENXIO;
@@ -467,24 +470,25 @@ snd_seq_oss_synth_load_patch(struct seq_oss_devinfo *dp, int dev, int fmt,
if (rec->oper.load_patch == NULL)
rc = -ENXIO;
else
- rc = rec->oper.load_patch(&dp->synths[dev].arg, fmt, buf, p, c);
+ rc = rec->oper.load_patch(&info->arg, fmt, buf, p, c);
snd_use_lock_free(&rec->use_lock);
return rc;
}
/*
- * check if the device is valid synth device
+ * check if the device is valid synth device and return the synth info
*/
-int
-snd_seq_oss_synth_is_valid(struct seq_oss_devinfo *dp, int dev)
+struct seq_oss_synthinfo *
+snd_seq_oss_synth_info(struct seq_oss_devinfo *dp, int dev)
{
struct seq_oss_synth *rec;
+
rec = get_synthdev(dp, dev);
if (rec) {
snd_use_lock_free(&rec->use_lock);
- return 1;
+ return get_synthinfo_nospec(dp, dev);
}
- return 0;
+ return NULL;
}
@@ -499,16 +503,18 @@ snd_seq_oss_synth_sysex(struct seq_oss_devinfo *dp, int dev, unsigned char *buf,
int i, send;
unsigned char *dest;
struct seq_oss_synth_sysex *sysex;
+ struct seq_oss_synthinfo *info;
- if (! snd_seq_oss_synth_is_valid(dp, dev))
+ info = snd_seq_oss_synth_info(dp, dev);
+ if (!info)
return -ENXIO;
- sysex = dp->synths[dev].sysex;
+ sysex = info->sysex;
if (sysex == NULL) {
sysex = kzalloc(sizeof(*sysex), GFP_KERNEL);
if (sysex == NULL)
return -ENOMEM;
- dp->synths[dev].sysex = sysex;
+ info->sysex = sysex;
}
send = 0;
@@ -553,10 +559,12 @@ snd_seq_oss_synth_sysex(struct seq_oss_devinfo *dp, int dev, unsigned char *buf,
int
snd_seq_oss_synth_addr(struct seq_oss_devinfo *dp, int dev, struct snd_seq_event *ev)
{
- if (! snd_seq_oss_synth_is_valid(dp, dev))
+ struct seq_oss_synthinfo *info = snd_seq_oss_synth_info(dp, dev);
+
+ if (!info)
return -EINVAL;
- snd_seq_oss_fill_addr(dp, ev, dp->synths[dev].arg.addr.client,
- dp->synths[dev].arg.addr.port);
+ snd_seq_oss_fill_addr(dp, ev, info->arg.addr.client,
+ info->arg.addr.port);
return 0;
}
@@ -568,16 +576,18 @@ int
snd_seq_oss_synth_ioctl(struct seq_oss_devinfo *dp, int dev, unsigned int cmd, unsigned long addr)
{
struct seq_oss_synth *rec;
+ struct seq_oss_synthinfo *info;
int rc;
- if (is_midi_dev(dp, dev))
+ info = get_synthinfo_nospec(dp, dev);
+ if (!info || info->is_midi)
return -ENXIO;
if ((rec = get_synthdev(dp, dev)) == NULL)
return -ENXIO;
if (rec->oper.ioctl == NULL)
rc = -ENXIO;
else
- rc = rec->oper.ioctl(&dp->synths[dev].arg, cmd, addr);
+ rc = rec->oper.ioctl(&info->arg, cmd, addr);
snd_use_lock_free(&rec->use_lock);
return rc;
}
@@ -589,7 +599,10 @@ snd_seq_oss_synth_ioctl(struct seq_oss_devinfo *dp, int dev, unsigned int cmd, u
int
snd_seq_oss_synth_raw_event(struct seq_oss_devinfo *dp, int dev, unsigned char *data, struct snd_seq_event *ev)
{
- if (! snd_seq_oss_synth_is_valid(dp, dev) || is_midi_dev(dp, dev))
+ struct seq_oss_synthinfo *info;
+
+ info = snd_seq_oss_synth_info(dp, dev);
+ if (!info || info->is_midi)
return -ENXIO;
ev->type = SNDRV_SEQ_EVENT_OSS;
memcpy(ev->data.raw8.d, data, 8);
diff --git a/sound/core/seq/oss/seq_oss_synth.h b/sound/core/seq/oss/seq_oss_synth.h
index 74ac55f..a63f9e2 100644
--- a/sound/core/seq/oss/seq_oss_synth.h
+++ b/sound/core/seq/oss/seq_oss_synth.h
@@ -37,7 +37,8 @@ void snd_seq_oss_synth_cleanup(struct seq_oss_devinfo *dp);
void snd_seq_oss_synth_reset(struct seq_oss_devinfo *dp, int dev);
int snd_seq_oss_synth_load_patch(struct seq_oss_devinfo *dp, int dev, int fmt,
const char __user *buf, int p, int c);
-int snd_seq_oss_synth_is_valid(struct seq_oss_devinfo *dp, int dev);
+struct seq_oss_synthinfo *snd_seq_oss_synth_info(struct seq_oss_devinfo *dp,
+ int dev);
int snd_seq_oss_synth_sysex(struct seq_oss_devinfo *dp, int dev, unsigned char *buf,
struct snd_seq_event *ev);
int snd_seq_oss_synth_addr(struct seq_oss_devinfo *dp, int dev, struct snd_seq_event *ev);
diff --git a/sound/core/seq/seq_clientmgr.c b/sound/core/seq/seq_clientmgr.c
index ecd1c5f..965473d 100644
--- a/sound/core/seq/seq_clientmgr.c
+++ b/sound/core/seq/seq_clientmgr.c
@@ -2002,7 +2002,8 @@ static int snd_seq_ioctl_query_next_client(struct snd_seq_client *client,
struct snd_seq_client *cptr = NULL;
/* search for next client */
- info->client++;
+ if (info->client < INT_MAX)
+ info->client++;
if (info->client < 0)
info->client = 0;
for (; info->client < SNDRV_SEQ_MAX_CLIENTS; info->client++) {
diff --git a/sound/core/seq/seq_virmidi.c b/sound/core/seq/seq_virmidi.c
index 2007649..1ebb346 100644
--- a/sound/core/seq/seq_virmidi.c
+++ b/sound/core/seq/seq_virmidi.c
@@ -163,6 +163,7 @@ static void snd_virmidi_output_trigger(struct snd_rawmidi_substream *substream,
int count, res;
unsigned char buf[32], *pbuf;
unsigned long flags;
+ bool check_resched = !in_atomic();
if (up) {
vmidi->trigger = 1;
@@ -174,12 +175,12 @@ static void snd_virmidi_output_trigger(struct snd_rawmidi_substream *substream,
}
return;
}
+ spin_lock_irqsave(&substream->runtime->lock, flags);
if (vmidi->event.type != SNDRV_SEQ_EVENT_NONE) {
if (snd_seq_kernel_client_dispatch(vmidi->client, &vmidi->event, in_atomic(), 0) < 0)
- return;
+ goto out;
vmidi->event.type = SNDRV_SEQ_EVENT_NONE;
}
- spin_lock_irqsave(&substream->runtime->lock, flags);
while (1) {
count = __snd_rawmidi_transmit_peek(substream, buf, sizeof(buf));
if (count <= 0)
@@ -200,6 +201,15 @@ static void snd_virmidi_output_trigger(struct snd_rawmidi_substream *substream,
vmidi->event.type = SNDRV_SEQ_EVENT_NONE;
}
}
+ if (!check_resched)
+ continue;
+ /* do temporary unlock & cond_resched() for avoiding
+ * CPU soft lockup, which may happen via a write from
+ * a huge rawmidi buffer
+ */
+ spin_unlock_irqrestore(&substream->runtime->lock, flags);
+ cond_resched();
+ spin_lock_irqsave(&substream->runtime->lock, flags);
}
out:
spin_unlock_irqrestore(&substream->runtime->lock, flags);
diff --git a/sound/core/timer.c b/sound/core/timer.c
index 7622551..d3b6260 100644
--- a/sound/core/timer.c
+++ b/sound/core/timer.c
@@ -547,7 +547,7 @@ static int snd_timer_stop1(struct snd_timer_instance *timeri, bool stop)
else
timeri->flags |= SNDRV_TIMER_IFLG_PAUSED;
snd_timer_notify1(timeri, stop ? SNDRV_TIMER_EVENT_STOP :
- SNDRV_TIMER_EVENT_CONTINUE);
+ SNDRV_TIMER_EVENT_PAUSE);
unlock:
spin_unlock_irqrestore(&timer->lock, flags);
return result;
@@ -569,7 +569,7 @@ static int snd_timer_stop_slave(struct snd_timer_instance *timeri, bool stop)
list_del_init(&timeri->ack_list);
list_del_init(&timeri->active_list);
snd_timer_notify1(timeri, stop ? SNDRV_TIMER_EVENT_STOP :
- SNDRV_TIMER_EVENT_CONTINUE);
+ SNDRV_TIMER_EVENT_PAUSE);
spin_unlock(&timeri->timer->lock);
}
spin_unlock_irqrestore(&slave_active_lock, flags);
diff --git a/sound/core/vmaster.c b/sound/core/vmaster.c
index 6c58e6f..7c6ef87 100644
--- a/sound/core/vmaster.c
+++ b/sound/core/vmaster.c
@@ -68,10 +68,13 @@ static int slave_update(struct link_slave *slave)
return -ENOMEM;
uctl->id = slave->slave.id;
err = slave->slave.get(&slave->slave, uctl);
+ if (err < 0)
+ goto error;
for (ch = 0; ch < slave->info.count; ch++)
slave->vals[ch] = uctl->value.integer.value[ch];
+ error:
kfree(uctl);
- return 0;
+ return err < 0 ? err : 0;
}
/* get the slave ctl info and save the initial values */
diff --git a/sound/drivers/aloop.c b/sound/drivers/aloop.c
index dc91002..847f7034 100644
--- a/sound/drivers/aloop.c
+++ b/sound/drivers/aloop.c
@@ -296,6 +296,8 @@ static int loopback_trigger(struct snd_pcm_substream *substream, int cmd)
cable->pause |= stream;
loopback_timer_stop(dpcm);
spin_unlock(&cable->lock);
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ loopback_active_notify(dpcm);
break;
case SNDRV_PCM_TRIGGER_PAUSE_RELEASE:
case SNDRV_PCM_TRIGGER_RESUME:
@@ -304,6 +306,8 @@ static int loopback_trigger(struct snd_pcm_substream *substream, int cmd)
cable->pause &= ~stream;
loopback_timer_start(dpcm);
spin_unlock(&cable->lock);
+ if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK)
+ loopback_active_notify(dpcm);
break;
default:
return -EINVAL;
@@ -828,9 +832,11 @@ static int loopback_rate_shift_get(struct snd_kcontrol *kcontrol,
{
struct loopback *loopback = snd_kcontrol_chip(kcontrol);
+ mutex_lock(&loopback->cable_lock);
ucontrol->value.integer.value[0] =
loopback->setup[kcontrol->id.subdevice]
[kcontrol->id.device].rate_shift;
+ mutex_unlock(&loopback->cable_lock);
return 0;
}
@@ -862,9 +868,11 @@ static int loopback_notify_get(struct snd_kcontrol *kcontrol,
{
struct loopback *loopback = snd_kcontrol_chip(kcontrol);
+ mutex_lock(&loopback->cable_lock);
ucontrol->value.integer.value[0] =
loopback->setup[kcontrol->id.subdevice]
[kcontrol->id.device].notify;
+ mutex_unlock(&loopback->cable_lock);
return 0;
}
@@ -876,12 +884,14 @@ static int loopback_notify_put(struct snd_kcontrol *kcontrol,
int change = 0;
val = ucontrol->value.integer.value[0] ? 1 : 0;
+ mutex_lock(&loopback->cable_lock);
if (val != loopback->setup[kcontrol->id.subdevice]
[kcontrol->id.device].notify) {
loopback->setup[kcontrol->id.subdevice]
[kcontrol->id.device].notify = val;
change = 1;
}
+ mutex_unlock(&loopback->cable_lock);
return change;
}
@@ -889,13 +899,18 @@ static int loopback_active_get(struct snd_kcontrol *kcontrol,
struct snd_ctl_elem_value *ucontrol)
{
struct loopback *loopback = snd_kcontrol_chip(kcontrol);
- struct loopback_cable *cable = loopback->cables
- [kcontrol->id.subdevice][kcontrol->id.device ^ 1];
+ struct loopback_cable *cable;
+
unsigned int val = 0;
- if (cable != NULL)
- val = (cable->running & (1 << SNDRV_PCM_STREAM_PLAYBACK)) ?
- 1 : 0;
+ mutex_lock(&loopback->cable_lock);
+ cable = loopback->cables[kcontrol->id.subdevice][kcontrol->id.device ^ 1];
+ if (cable != NULL) {
+ unsigned int running = cable->running ^ cable->pause;
+
+ val = (running & (1 << SNDRV_PCM_STREAM_PLAYBACK)) ? 1 : 0;
+ }
+ mutex_unlock(&loopback->cable_lock);
ucontrol->value.integer.value[0] = val;
return 0;
}
@@ -938,9 +953,11 @@ static int loopback_rate_get(struct snd_kcontrol *kcontrol,
{
struct loopback *loopback = snd_kcontrol_chip(kcontrol);
+ mutex_lock(&loopback->cable_lock);
ucontrol->value.integer.value[0] =
loopback->setup[kcontrol->id.subdevice]
[kcontrol->id.device].rate;
+ mutex_unlock(&loopback->cable_lock);
return 0;
}
@@ -960,9 +977,11 @@ static int loopback_channels_get(struct snd_kcontrol *kcontrol,
{
struct loopback *loopback = snd_kcontrol_chip(kcontrol);
+ mutex_lock(&loopback->cable_lock);
ucontrol->value.integer.value[0] =
loopback->setup[kcontrol->id.subdevice]
[kcontrol->id.device].channels;
+ mutex_unlock(&loopback->cable_lock);
return 0;
}
diff --git a/sound/drivers/opl3/opl3_synth.c b/sound/drivers/opl3/opl3_synth.c
index ddcc1a3..42920a2 100644
--- a/sound/drivers/opl3/opl3_synth.c
+++ b/sound/drivers/opl3/opl3_synth.c
@@ -21,6 +21,7 @@
#include <linux/slab.h>
#include <linux/export.h>
+#include <linux/nospec.h>
#include <sound/opl3.h>
#include <sound/asound_fm.h>
@@ -448,7 +449,7 @@ static int snd_opl3_set_voice(struct snd_opl3 * opl3, struct snd_dm_fm_voice * v
{
unsigned short reg_side;
unsigned char op_offset;
- unsigned char voice_offset;
+ unsigned char voice_offset, voice_op;
unsigned short opl3_reg;
unsigned char reg_val;
@@ -473,7 +474,9 @@ static int snd_opl3_set_voice(struct snd_opl3 * opl3, struct snd_dm_fm_voice * v
voice_offset = voice->voice - MAX_OPL2_VOICES;
}
/* Get register offset of operator */
- op_offset = snd_opl3_regmap[voice_offset][voice->op];
+ voice_offset = array_index_nospec(voice_offset, MAX_OPL2_VOICES);
+ voice_op = array_index_nospec(voice->op, 4);
+ op_offset = snd_opl3_regmap[voice_offset][voice_op];
reg_val = 0x00;
/* Set amplitude modulation (tremolo) effect */
diff --git a/sound/firewire/bebob/bebob.c b/sound/firewire/bebob/bebob.c
index 3469ac14..d0dfa82 100644
--- a/sound/firewire/bebob/bebob.c
+++ b/sound/firewire/bebob/bebob.c
@@ -263,6 +263,8 @@ do_registration(struct work_struct *work)
error:
mutex_unlock(&devices_mutex);
snd_bebob_stream_destroy_duplex(bebob);
+ kfree(bebob->maudio_special_quirk);
+ bebob->maudio_special_quirk = NULL;
snd_card_free(bebob->card);
dev_info(&bebob->unit->device,
"Sound card registration failed: %d\n", err);
diff --git a/sound/firewire/bebob/bebob_maudio.c b/sound/firewire/bebob/bebob_maudio.c
index 07e5abd..6dbf047 100644
--- a/sound/firewire/bebob/bebob_maudio.c
+++ b/sound/firewire/bebob/bebob_maudio.c
@@ -96,17 +96,13 @@ int snd_bebob_maudio_load_firmware(struct fw_unit *unit)
struct fw_device *device = fw_parent_device(unit);
int err, rcode;
u64 date;
- __le32 cues[3] = {
- cpu_to_le32(MAUDIO_BOOTLOADER_CUE1),
- cpu_to_le32(MAUDIO_BOOTLOADER_CUE2),
- cpu_to_le32(MAUDIO_BOOTLOADER_CUE3)
- };
+ __le32 *cues;
/* check date of software used to build */
err = snd_bebob_read_block(unit, INFO_OFFSET_SW_DATE,
&date, sizeof(u64));
if (err < 0)
- goto end;
+ return err;
/*
* firmware version 5058 or later has date later than "20070401", but
* 'date' is not null-terminated.
@@ -114,20 +110,28 @@ int snd_bebob_maudio_load_firmware(struct fw_unit *unit)
if (date < 0x3230303730343031LL) {
dev_err(&unit->device,
"Use firmware version 5058 or later\n");
- err = -ENOSYS;
- goto end;
+ return -ENXIO;
}
+ cues = kmalloc_array(3, sizeof(*cues), GFP_KERNEL);
+ if (!cues)
+ return -ENOMEM;
+
+ cues[0] = cpu_to_le32(MAUDIO_BOOTLOADER_CUE1);
+ cues[1] = cpu_to_le32(MAUDIO_BOOTLOADER_CUE2);
+ cues[2] = cpu_to_le32(MAUDIO_BOOTLOADER_CUE3);
+
rcode = fw_run_transaction(device->card, TCODE_WRITE_BLOCK_REQUEST,
device->node_id, device->generation,
device->max_speed, BEBOB_ADDR_REG_REQ,
- cues, sizeof(cues));
+ cues, 3 * sizeof(*cues));
+ kfree(cues);
if (rcode != RCODE_COMPLETE) {
dev_err(&unit->device,
"Failed to send a cue to load firmware\n");
err = -EIO;
}
-end:
+
return err;
}
@@ -290,10 +294,6 @@ snd_bebob_maudio_special_discover(struct snd_bebob *bebob, bool is1814)
bebob->midi_output_ports = 2;
}
end:
- if (err < 0) {
- kfree(params);
- bebob->maudio_special_quirk = NULL;
- }
mutex_unlock(&bebob->mutex);
return err;
}
diff --git a/sound/firewire/dice/dice-stream.c b/sound/firewire/dice/dice-stream.c
index ec4db3a..257cfbf 100644
--- a/sound/firewire/dice/dice-stream.c
+++ b/sound/firewire/dice/dice-stream.c
@@ -425,7 +425,7 @@ int snd_dice_stream_init_duplex(struct snd_dice *dice)
err = init_stream(dice, AMDTP_IN_STREAM, i);
if (err < 0) {
for (; i >= 0; i--)
- destroy_stream(dice, AMDTP_OUT_STREAM, i);
+ destroy_stream(dice, AMDTP_IN_STREAM, i);
goto end;
}
}
diff --git a/sound/firewire/dice/dice.c b/sound/firewire/dice/dice.c
index 25e9f77..0d3d36fb 100644
--- a/sound/firewire/dice/dice.c
+++ b/sound/firewire/dice/dice.c
@@ -14,7 +14,7 @@ MODULE_LICENSE("GPL v2");
#define OUI_WEISS 0x001c6a
#define OUI_LOUD 0x000ff2
#define OUI_FOCUSRITE 0x00130e
-#define OUI_TCELECTRONIC 0x001486
+#define OUI_TCELECTRONIC 0x000166
#define DICE_CATEGORY_ID 0x04
#define WEISS_CATEGORY_ID 0x00
diff --git a/sound/firewire/digi00x/digi00x.c b/sound/firewire/digi00x/digi00x.c
index 1f5e1d2..ef68999 100644
--- a/sound/firewire/digi00x/digi00x.c
+++ b/sound/firewire/digi00x/digi00x.c
@@ -49,6 +49,7 @@ static void dg00x_free(struct snd_dg00x *dg00x)
fw_unit_put(dg00x->unit);
mutex_destroy(&dg00x->mutex);
+ kfree(dg00x);
}
static void dg00x_card_free(struct snd_card *card)
diff --git a/sound/firewire/fireworks/fireworks.c b/sound/firewire/fireworks/fireworks.c
index 71a0613..f2d0733 100644
--- a/sound/firewire/fireworks/fireworks.c
+++ b/sound/firewire/fireworks/fireworks.c
@@ -301,6 +301,8 @@ do_registration(struct work_struct *work)
snd_efw_transaction_remove_instance(efw);
snd_efw_stream_destroy_duplex(efw);
snd_card_free(efw->card);
+ kfree(efw->resp_buf);
+ efw->resp_buf = NULL;
dev_info(&efw->unit->device,
"Sound card registration failed: %d\n", err);
}
diff --git a/sound/firewire/oxfw/oxfw.c b/sound/firewire/oxfw/oxfw.c
index 474b06d..696b6cf 100644
--- a/sound/firewire/oxfw/oxfw.c
+++ b/sound/firewire/oxfw/oxfw.c
@@ -135,6 +135,7 @@ static void oxfw_free(struct snd_oxfw *oxfw)
kfree(oxfw->spec);
mutex_destroy(&oxfw->mutex);
+ kfree(oxfw);
}
/*
@@ -212,6 +213,7 @@ static int detect_quirks(struct snd_oxfw *oxfw)
static void do_registration(struct work_struct *work)
{
struct snd_oxfw *oxfw = container_of(work, struct snd_oxfw, dwork.work);
+ int i;
int err;
if (oxfw->registered)
@@ -274,7 +276,15 @@ static void do_registration(struct work_struct *work)
snd_oxfw_stream_destroy_simplex(oxfw, &oxfw->rx_stream);
if (oxfw->has_output)
snd_oxfw_stream_destroy_simplex(oxfw, &oxfw->tx_stream);
+ for (i = 0; i < SND_OXFW_STREAM_FORMAT_ENTRIES; ++i) {
+ kfree(oxfw->tx_stream_formats[i]);
+ oxfw->tx_stream_formats[i] = NULL;
+ kfree(oxfw->rx_stream_formats[i]);
+ oxfw->rx_stream_formats[i] = NULL;
+ }
snd_card_free(oxfw->card);
+ kfree(oxfw->spec);
+ oxfw->spec = NULL;
dev_info(&oxfw->unit->device,
"Sound card registration failed: %d\n", err);
}
diff --git a/sound/firewire/tascam/tascam.c b/sound/firewire/tascam/tascam.c
index 9dc93a7..4c967ac 100644
--- a/sound/firewire/tascam/tascam.c
+++ b/sound/firewire/tascam/tascam.c
@@ -93,6 +93,7 @@ static void tscm_free(struct snd_tscm *tscm)
fw_unit_put(tscm->unit);
mutex_destroy(&tscm->mutex);
+ kfree(tscm);
}
static void tscm_card_free(struct snd_card *card)
diff --git a/sound/hda/hdac_controller.c b/sound/hda/hdac_controller.c
index 8761877..00c6af2 100644
--- a/sound/hda/hdac_controller.c
+++ b/sound/hda/hdac_controller.c
@@ -40,6 +40,8 @@ static void azx_clear_corbrp(struct hdac_bus *bus)
*/
void snd_hdac_bus_init_cmd_io(struct hdac_bus *bus)
{
+ WARN_ON_ONCE(!bus->rb.area);
+
spin_lock_irq(&bus->reg_lock);
/* CORB set up */
bus->corb.addr = bus->rb.addr;
@@ -478,13 +480,15 @@ bool snd_hdac_bus_init_chip(struct hdac_bus *bus, bool full_reset)
/* reset controller */
azx_reset(bus, full_reset);
- /* initialize interrupts */
+ /* clear interrupts */
azx_int_clear(bus);
- azx_int_enable(bus);
/* initialize the codec command I/O */
snd_hdac_bus_init_cmd_io(bus);
+ /* enable interrupts after CORB/RIRB buffers are initialized above */
+ azx_int_enable(bus);
+
/* program the position buffer */
if (bus->use_posbuf && bus->posbuf.addr) {
snd_hdac_chip_writel(bus, DPLBASE, (u32)bus->posbuf.addr);
diff --git a/sound/isa/msnd/msnd_pinnacle.c b/sound/isa/msnd/msnd_pinnacle.c
index a31ea6c..2d7379d 100644
--- a/sound/isa/msnd/msnd_pinnacle.c
+++ b/sound/isa/msnd/msnd_pinnacle.c
@@ -82,10 +82,10 @@
static void set_default_audio_parameters(struct snd_msnd *chip)
{
- chip->play_sample_size = DEFSAMPLESIZE;
+ chip->play_sample_size = snd_pcm_format_width(DEFSAMPLESIZE);
chip->play_sample_rate = DEFSAMPLERATE;
chip->play_channels = DEFCHANNELS;
- chip->capture_sample_size = DEFSAMPLESIZE;
+ chip->capture_sample_size = snd_pcm_format_width(DEFSAMPLESIZE);
chip->capture_sample_rate = DEFSAMPLERATE;
chip->capture_channels = DEFCHANNELS;
}
diff --git a/sound/isa/wss/wss_lib.c b/sound/isa/wss/wss_lib.c
index 913b731..f40330d 100644
--- a/sound/isa/wss/wss_lib.c
+++ b/sound/isa/wss/wss_lib.c
@@ -1531,7 +1531,6 @@ static int snd_wss_playback_open(struct snd_pcm_substream *substream)
if (err < 0) {
if (chip->release_dma)
chip->release_dma(chip, chip->dma_private_data, chip->dma1);
- snd_free_pages(runtime->dma_area, runtime->dma_bytes);
return err;
}
chip->playback_substream = substream;
@@ -1572,7 +1571,6 @@ static int snd_wss_capture_open(struct snd_pcm_substream *substream)
if (err < 0) {
if (chip->release_dma)
chip->release_dma(chip, chip->dma_private_data, chip->dma2);
- snd_free_pages(runtime->dma_area, runtime->dma_bytes);
return err;
}
chip->capture_substream = substream;
diff --git a/sound/pci/ac97/ac97_codec.c b/sound/pci/ac97/ac97_codec.c
index 82259ca..c4840fd 100644
--- a/sound/pci/ac97/ac97_codec.c
+++ b/sound/pci/ac97/ac97_codec.c
@@ -824,7 +824,7 @@ static int snd_ac97_put_spsa(struct snd_kcontrol *kcontrol, struct snd_ctl_elem_
{
struct snd_ac97 *ac97 = snd_kcontrol_chip(kcontrol);
int reg = kcontrol->private_value & 0xff;
- int shift = (kcontrol->private_value >> 8) & 0xff;
+ int shift = (kcontrol->private_value >> 8) & 0x0f;
int mask = (kcontrol->private_value >> 16) & 0xff;
// int invert = (kcontrol->private_value >> 24) & 0xff;
unsigned short value, old, new;
diff --git a/sound/pci/asihpi/hpimsginit.c b/sound/pci/asihpi/hpimsginit.c
index 7eb6171..a31a70d 100644
--- a/sound/pci/asihpi/hpimsginit.c
+++ b/sound/pci/asihpi/hpimsginit.c
@@ -23,6 +23,7 @@
#include "hpi_internal.h"
#include "hpimsginit.h"
+#include <linux/nospec.h>
/* The actual message size for each object type */
static u16 msg_size[HPI_OBJ_MAXINDEX + 1] = HPI_MESSAGE_SIZE_BY_OBJECT;
@@ -39,10 +40,12 @@ static void hpi_init_message(struct hpi_message *phm, u16 object,
{
u16 size;
- if ((object > 0) && (object <= HPI_OBJ_MAXINDEX))
+ if ((object > 0) && (object <= HPI_OBJ_MAXINDEX)) {
+ object = array_index_nospec(object, HPI_OBJ_MAXINDEX + 1);
size = msg_size[object];
- else
+ } else {
size = sizeof(*phm);
+ }
memset(phm, 0, size);
phm->size = size;
@@ -66,10 +69,12 @@ void hpi_init_response(struct hpi_response *phr, u16 object, u16 function,
{
u16 size;
- if ((object > 0) && (object <= HPI_OBJ_MAXINDEX))
+ if ((object > 0) && (object <= HPI_OBJ_MAXINDEX)) {
+ object = array_index_nospec(object, HPI_OBJ_MAXINDEX + 1);
size = res_size[object];
- else
+ } else {
size = sizeof(*phr);
+ }
memset(phr, 0, sizeof(*phr));
phr->size = size;
diff --git a/sound/pci/asihpi/hpioctl.c b/sound/pci/asihpi/hpioctl.c
index 7e3aa50..3ef9af5 100644
--- a/sound/pci/asihpi/hpioctl.c
+++ b/sound/pci/asihpi/hpioctl.c
@@ -33,6 +33,7 @@
#include <linux/stringify.h>
#include <linux/module.h>
#include <linux/vmalloc.h>
+#include <linux/nospec.h>
#ifdef MODULE_FIRMWARE
MODULE_FIRMWARE("asihpi/dsp5000.bin");
@@ -182,7 +183,8 @@ long asihpi_hpi_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
struct hpi_adapter *pa = NULL;
if (hm->h.adapter_index < ARRAY_SIZE(adapters))
- pa = &adapters[hm->h.adapter_index];
+ pa = &adapters[array_index_nospec(hm->h.adapter_index,
+ ARRAY_SIZE(adapters))];
if (!pa || !pa->adapter || !pa->adapter->type) {
hpi_init_response(&hr->r0, hm->h.object,
diff --git a/sound/pci/ca0106/ca0106.h b/sound/pci/ca0106/ca0106.h
index 04402c1..9847b66 100644
--- a/sound/pci/ca0106/ca0106.h
+++ b/sound/pci/ca0106/ca0106.h
@@ -582,7 +582,7 @@
#define SPI_PL_BIT_R_R (2<<7) /* right channel = right */
#define SPI_PL_BIT_R_C (3<<7) /* right channel = (L+R)/2 */
#define SPI_IZD_REG 2
-#define SPI_IZD_BIT (1<<4) /* infinite zero detect */
+#define SPI_IZD_BIT (0<<4) /* infinite zero detect */
#define SPI_FMT_REG 3
#define SPI_FMT_BIT_RJ (0<<0) /* right justified mode */
diff --git a/sound/pci/cs46xx/dsp_spos.c b/sound/pci/cs46xx/dsp_spos.c
index 4a0cbd2..3191666 100644
--- a/sound/pci/cs46xx/dsp_spos.c
+++ b/sound/pci/cs46xx/dsp_spos.c
@@ -899,6 +899,9 @@ int cs46xx_dsp_proc_done (struct snd_cs46xx *chip)
struct dsp_spos_instance * ins = chip->dsp_spos_instance;
int i;
+ if (!ins)
+ return 0;
+
snd_info_free_entry(ins->proc_sym_info_entry);
ins->proc_sym_info_entry = NULL;
diff --git a/sound/pci/cs5535audio/cs5535audio.h b/sound/pci/cs5535audio/cs5535audio.h
index 0579daa..425d1b6 100644
--- a/sound/pci/cs5535audio/cs5535audio.h
+++ b/sound/pci/cs5535audio/cs5535audio.h
@@ -66,9 +66,9 @@ struct cs5535audio_dma_ops {
};
struct cs5535audio_dma_desc {
- u32 addr;
- u16 size;
- u16 ctlreserved;
+ __le32 addr;
+ __le16 size;
+ __le16 ctlreserved;
};
struct cs5535audio_dma {
diff --git a/sound/pci/cs5535audio/cs5535audio_pcm.c b/sound/pci/cs5535audio/cs5535audio_pcm.c
index c208c1d..b9912ec2 100644
--- a/sound/pci/cs5535audio/cs5535audio_pcm.c
+++ b/sound/pci/cs5535audio/cs5535audio_pcm.c
@@ -158,8 +158,8 @@ static int cs5535audio_build_dma_packets(struct cs5535audio *cs5535au,
lastdesc->addr = cpu_to_le32((u32) dma->desc_buf.addr);
lastdesc->size = 0;
lastdesc->ctlreserved = cpu_to_le16(PRD_JMP);
- jmpprd_addr = cpu_to_le32(lastdesc->addr +
- (sizeof(struct cs5535audio_dma_desc)*periods));
+ jmpprd_addr = (u32)dma->desc_buf.addr +
+ sizeof(struct cs5535audio_dma_desc) * periods;
dma->substream = substream;
dma->period_bytes = period_bytes;
diff --git a/sound/pci/emu10k1/emufx.c b/sound/pci/emu10k1/emufx.c
index 56fc47b..5d422d6 100644
--- a/sound/pci/emu10k1/emufx.c
+++ b/sound/pci/emu10k1/emufx.c
@@ -36,6 +36,7 @@
#include <linux/init.h>
#include <linux/mutex.h>
#include <linux/moduleparam.h>
+#include <linux/nospec.h>
#include <sound/core.h>
#include <sound/tlv.h>
@@ -1000,6 +1001,8 @@ static int snd_emu10k1_ipcm_poke(struct snd_emu10k1 *emu,
if (ipcm->substream >= EMU10K1_FX8010_PCM_COUNT)
return -EINVAL;
+ ipcm->substream = array_index_nospec(ipcm->substream,
+ EMU10K1_FX8010_PCM_COUNT);
if (ipcm->channels > 32)
return -EINVAL;
pcm = &emu->fx8010.pcm[ipcm->substream];
@@ -1046,6 +1049,8 @@ static int snd_emu10k1_ipcm_peek(struct snd_emu10k1 *emu,
if (ipcm->substream >= EMU10K1_FX8010_PCM_COUNT)
return -EINVAL;
+ ipcm->substream = array_index_nospec(ipcm->substream,
+ EMU10K1_FX8010_PCM_COUNT);
pcm = &emu->fx8010.pcm[ipcm->substream];
mutex_lock(&emu->fx8010.lock);
spin_lock_irq(&emu->reg_lock);
@@ -2520,7 +2525,7 @@ static int snd_emu10k1_fx8010_ioctl(struct snd_hwdep * hw, struct file *file, un
emu->support_tlv = 1;
return put_user(SNDRV_EMU10K1_VERSION, (int __user *)argp);
case SNDRV_EMU10K1_IOCTL_INFO:
- info = kmalloc(sizeof(*info), GFP_KERNEL);
+ info = kzalloc(sizeof(*info), GFP_KERNEL);
if (!info)
return -ENOMEM;
snd_emu10k1_fx8010_info(emu, info);
diff --git a/sound/pci/emu10k1/emupcm.c b/sound/pci/emu10k1/emupcm.c
index 37be1e1..0d2bb30 100644
--- a/sound/pci/emu10k1/emupcm.c
+++ b/sound/pci/emu10k1/emupcm.c
@@ -1850,7 +1850,9 @@ int snd_emu10k1_pcm_efx(struct snd_emu10k1 *emu, int device)
if (!kctl)
return -ENOMEM;
kctl->id.device = device;
- snd_ctl_add(emu->card, kctl);
+ err = snd_ctl_add(emu->card, kctl);
+ if (err < 0)
+ return err;
snd_pcm_lib_preallocate_pages_for_all(pcm, SNDRV_DMA_TYPE_DEV, snd_dma_pci_data(emu->pci), 64*1024, 64*1024);
diff --git a/sound/pci/emu10k1/memory.c b/sound/pci/emu10k1/memory.c
index 4f1f69b..8c778fa3 100644
--- a/sound/pci/emu10k1/memory.c
+++ b/sound/pci/emu10k1/memory.c
@@ -237,13 +237,13 @@ search_empty(struct snd_emu10k1 *emu, int size)
static int is_valid_page(struct snd_emu10k1 *emu, dma_addr_t addr)
{
if (addr & ~emu->dma_mask) {
- dev_err(emu->card->dev,
+ dev_err_ratelimited(emu->card->dev,
"max memory size is 0x%lx (addr = 0x%lx)!!\n",
emu->dma_mask, (unsigned long)addr);
return 0;
}
if (addr & (EMUPAGESIZE-1)) {
- dev_err(emu->card->dev, "page is not aligned\n");
+ dev_err_ratelimited(emu->card->dev, "page is not aligned\n");
return 0;
}
return 1;
@@ -334,7 +334,7 @@ snd_emu10k1_alloc_pages(struct snd_emu10k1 *emu, struct snd_pcm_substream *subst
else
addr = snd_pcm_sgbuf_get_addr(substream, ofs);
if (! is_valid_page(emu, addr)) {
- dev_err(emu->card->dev,
+ dev_err_ratelimited(emu->card->dev,
"emu: failure page = %d\n", idx);
mutex_unlock(&hdr->block_mutex);
return NULL;
diff --git a/sound/pci/fm801.c b/sound/pci/fm801.c
index a178e0d..8561f60 100644
--- a/sound/pci/fm801.c
+++ b/sound/pci/fm801.c
@@ -1068,11 +1068,19 @@ static int snd_fm801_mixer(struct fm801 *chip)
if ((err = snd_ac97_mixer(chip->ac97_bus, &ac97, &chip->ac97_sec)) < 0)
return err;
}
- for (i = 0; i < FM801_CONTROLS; i++)
- snd_ctl_add(chip->card, snd_ctl_new1(&snd_fm801_controls[i], chip));
+ for (i = 0; i < FM801_CONTROLS; i++) {
+ err = snd_ctl_add(chip->card,
+ snd_ctl_new1(&snd_fm801_controls[i], chip));
+ if (err < 0)
+ return err;
+ }
if (chip->multichannel) {
- for (i = 0; i < FM801_CONTROLS_MULTI; i++)
- snd_ctl_add(chip->card, snd_ctl_new1(&snd_fm801_controls_multi[i], chip));
+ for (i = 0; i < FM801_CONTROLS_MULTI; i++) {
+ err = snd_ctl_add(chip->card,
+ snd_ctl_new1(&snd_fm801_controls_multi[i], chip));
+ if (err < 0)
+ return err;
+ }
}
return 0;
}
diff --git a/sound/pci/hda/Kconfig b/sound/pci/hda/Kconfig
index 7f3b5ed..f7a492c 100644
--- a/sound/pci/hda/Kconfig
+++ b/sound/pci/hda/Kconfig
@@ -88,7 +88,6 @@
config SND_HDA_CODEC_REALTEK
tristate "Build Realtek HD-audio codec support"
select SND_HDA_GENERIC
- select INPUT
help
Say Y or M here to include Realtek HD-audio codec support in
snd-hda-intel driver, such as ALC880.
diff --git a/sound/pci/hda/hda_codec.c b/sound/pci/hda/hda_codec.c
index e46c561..c6b046d 100644
--- a/sound/pci/hda/hda_codec.c
+++ b/sound/pci/hda/hda_codec.c
@@ -4025,7 +4025,8 @@ void snd_hda_bus_reset_codecs(struct hda_bus *bus)
list_for_each_codec(codec, bus) {
/* FIXME: maybe a better way needed for forced reset */
- cancel_delayed_work_sync(&codec->jackpoll_work);
+ if (current_work() != &codec->jackpoll_work.work)
+ cancel_delayed_work_sync(&codec->jackpoll_work);
#ifdef CONFIG_PM
if (hda_codec_is_power_on(codec)) {
hda_call_codec_suspend(codec);
diff --git a/sound/pci/hda/hda_controller.c b/sound/pci/hda/hda_controller.c
index 0af1132..56af730 100644
--- a/sound/pci/hda/hda_controller.c
+++ b/sound/pci/hda/hda_controller.c
@@ -748,8 +748,10 @@ int snd_hda_attach_pcm_stream(struct hda_bus *_bus, struct hda_codec *codec,
return err;
strlcpy(pcm->name, cpcm->name, sizeof(pcm->name));
apcm = kzalloc(sizeof(*apcm), GFP_KERNEL);
- if (apcm == NULL)
+ if (apcm == NULL) {
+ snd_device_free(chip->card, pcm);
return -ENOMEM;
+ }
apcm->chip = chip;
apcm->pcm = pcm;
apcm->codec = codec;
diff --git a/sound/pci/hda/hda_controller.h b/sound/pci/hda/hda_controller.h
index a50e053..b83feec 100644
--- a/sound/pci/hda/hda_controller.h
+++ b/sound/pci/hda/hda_controller.h
@@ -155,6 +155,7 @@ struct azx {
unsigned int msi:1;
unsigned int probing:1; /* codec probing phase */
unsigned int snoop:1;
+ unsigned int uc_buffer:1; /* non-cached pages for stream buffers */
unsigned int align_buffer_size:1;
unsigned int region_requested:1;
unsigned int disabled:1; /* disabled by vga_switcheroo */
diff --git a/sound/pci/hda/hda_hwdep.c b/sound/pci/hda/hda_hwdep.c
index 57df06e..cc009a4 100644
--- a/sound/pci/hda/hda_hwdep.c
+++ b/sound/pci/hda/hda_hwdep.c
@@ -21,6 +21,7 @@
#include <linux/init.h>
#include <linux/slab.h>
#include <linux/compat.h>
+#include <linux/nospec.h>
#include <sound/core.h>
#include "hda_codec.h"
#include "hda_local.h"
@@ -51,7 +52,16 @@ static int get_wcap_ioctl(struct hda_codec *codec,
if (get_user(verb, &arg->verb))
return -EFAULT;
- res = get_wcaps(codec, verb >> 24);
+ /* open-code get_wcaps(verb>>24) with nospec */
+ verb >>= 24;
+ if (verb < codec->core.start_nid ||
+ verb >= codec->core.start_nid + codec->core.num_nodes) {
+ res = 0;
+ } else {
+ verb -= codec->core.start_nid;
+ verb = array_index_nospec(verb, codec->core.num_nodes);
+ res = codec->wcaps[verb];
+ }
if (put_user(res, &arg->res))
return -EFAULT;
return 0;
diff --git a/sound/pci/hda/hda_intel.c b/sound/pci/hda/hda_intel.c
index 7d3f88d..fcd583e 100644
--- a/sound/pci/hda/hda_intel.c
+++ b/sound/pci/hda/hda_intel.c
@@ -410,7 +410,7 @@ static void __mark_pages_wc(struct azx *chip, struct snd_dma_buffer *dmab, bool
#ifdef CONFIG_SND_DMA_SGBUF
if (dmab->dev.type == SNDRV_DMA_TYPE_DEV_SG) {
struct snd_sg_buf *sgbuf = dmab->private_data;
- if (chip->driver_type == AZX_DRIVER_CMEDIA)
+ if (!chip->uc_buffer)
return; /* deal with only CORB/RIRB buffers */
if (on)
set_pages_array_wc(sgbuf->page_table, sgbuf->pages);
@@ -1503,6 +1503,7 @@ static void azx_check_snoop_available(struct azx *chip)
dev_info(chip->card->dev, "Force to %s mode by module option\n",
snoop ? "snoop" : "non-snoop");
chip->snoop = snoop;
+ chip->uc_buffer = !snoop;
return;
}
@@ -1523,8 +1524,12 @@ static void azx_check_snoop_available(struct azx *chip)
snoop = false;
chip->snoop = snoop;
- if (!snoop)
+ if (!snoop) {
dev_info(chip->card->dev, "Force to non-snoop mode\n");
+ /* C-Media requires non-cached pages only for CORB/RIRB */
+ if (chip->driver_type != AZX_DRIVER_CMEDIA)
+ chip->uc_buffer = true;
+ }
}
static void azx_probe_work(struct work_struct *work)
@@ -1947,7 +1952,7 @@ static void pcm_mmap_prepare(struct snd_pcm_substream *substream,
#ifdef CONFIG_X86
struct azx_pcm *apcm = snd_pcm_substream_chip(substream);
struct azx *chip = apcm->chip;
- if (!azx_snoop(chip) && chip->driver_type != AZX_DRIVER_CMEDIA)
+ if (chip->uc_buffer)
area->vm_page_prot = pgprot_writecombine(area->vm_page_prot);
#endif
}
@@ -2058,9 +2063,11 @@ static int azx_probe(struct pci_dev *pci,
*/
static struct snd_pci_quirk power_save_blacklist[] = {
/* https://bugzilla.redhat.com/show_bug.cgi?id=1525104 */
- SND_PCI_QUIRK(0x1849, 0x0c0c, "Asrock B85M-ITX", 0),
+ SND_PCI_QUIRK(0x1849, 0xc892, "Asrock B85M-ITX", 0),
/* https://bugzilla.redhat.com/show_bug.cgi?id=1525104 */
SND_PCI_QUIRK(0x1043, 0x8733, "Asus Prime X370-Pro", 0),
+ /* https://bugzilla.redhat.com/show_bug.cgi?id=1572975 */
+ SND_PCI_QUIRK(0x17aa, 0x36a7, "Lenovo C50 All in one", 0),
/* https://bugzilla.kernel.org/show_bug.cgi?id=198611 */
SND_PCI_QUIRK(0x17aa, 0x2227, "Lenovo X1 Carbon 3rd Gen", 0),
{}
@@ -2345,9 +2352,14 @@ static const struct pci_device_id azx_ids[] = {
/* AMD Hudson */
{ PCI_DEVICE(0x1022, 0x780d),
.driver_data = AZX_DRIVER_GENERIC | AZX_DCAPS_PRESET_ATI_SB },
+ /* AMD Stoney */
+ { PCI_DEVICE(0x1022, 0x157a),
+ .driver_data = AZX_DRIVER_GENERIC | AZX_DCAPS_PRESET_ATI_SB |
+ AZX_DCAPS_PM_RUNTIME },
/* AMD Raven */
{ PCI_DEVICE(0x1022, 0x15e3),
- .driver_data = AZX_DRIVER_GENERIC | AZX_DCAPS_PRESET_ATI_SB },
+ .driver_data = AZX_DRIVER_GENERIC | AZX_DCAPS_PRESET_ATI_SB |
+ AZX_DCAPS_PM_RUNTIME },
/* ATI HDMI */
{ PCI_DEVICE(0x1002, 0x0002),
.driver_data = AZX_DRIVER_ATIHDMI_NS | AZX_DCAPS_PRESET_ATI_HDMI_NS },
diff --git a/sound/pci/hda/hda_tegra.c b/sound/pci/hda/hda_tegra.c
index 0621920..e85fb04 100644
--- a/sound/pci/hda/hda_tegra.c
+++ b/sound/pci/hda/hda_tegra.c
@@ -249,10 +249,12 @@ static int hda_tegra_suspend(struct device *dev)
struct snd_card *card = dev_get_drvdata(dev);
struct azx *chip = card->private_data;
struct hda_tegra *hda = container_of(chip, struct hda_tegra, chip);
+ struct hdac_bus *bus = azx_bus(chip);
snd_power_change_state(card, SNDRV_CTL_POWER_D3hot);
azx_stop_chip(chip);
+ synchronize_irq(bus->irq);
azx_enter_link_reset(chip);
hda_tegra_disable_clocks(hda);
diff --git a/sound/pci/hda/patch_ca0132.c b/sound/pci/hda/patch_ca0132.c
index 9ec4dba..2809999 100644
--- a/sound/pci/hda/patch_ca0132.c
+++ b/sound/pci/hda/patch_ca0132.c
@@ -38,6 +38,10 @@
/* Enable this to see controls for tuning purpose. */
/*#define ENABLE_TUNING_CONTROLS*/
+#ifdef ENABLE_TUNING_CONTROLS
+#include <sound/tlv.h>
+#endif
+
#define FLOAT_ZERO 0x00000000
#define FLOAT_ONE 0x3f800000
#define FLOAT_TWO 0x40000000
@@ -3067,8 +3071,8 @@ static int equalizer_ctl_put(struct snd_kcontrol *kcontrol,
return 1;
}
-static const DECLARE_TLV_DB_SCALE(voice_focus_db_scale, 2000, 100, 0);
-static const DECLARE_TLV_DB_SCALE(eq_db_scale, -2400, 100, 0);
+static const SNDRV_CTL_TLVD_DECLARE_DB_SCALE(voice_focus_db_scale, 2000, 100, 0);
+static const SNDRV_CTL_TLVD_DECLARE_DB_SCALE(eq_db_scale, -2400, 100, 0);
static int add_tuning_control(struct hda_codec *codec,
hda_nid_t pnid, hda_nid_t nid,
diff --git a/sound/pci/hda/patch_conexant.c b/sound/pci/hda/patch_conexant.c
index b3851b9..ba9cd75 100644
--- a/sound/pci/hda/patch_conexant.c
+++ b/sound/pci/hda/patch_conexant.c
@@ -205,6 +205,7 @@ static void cx_auto_reboot_notify(struct hda_codec *codec)
struct conexant_spec *spec = codec->spec;
switch (codec->core.vendor_id) {
+ case 0x14f12008: /* CX8200 */
case 0x14f150f2: /* CX20722 */
case 0x14f150f4: /* CX20724 */
break;
@@ -212,13 +213,14 @@ static void cx_auto_reboot_notify(struct hda_codec *codec)
return;
}
- /* Turn the CX20722 codec into D3 to avoid spurious noises
+ /* Turn the problematic codec into D3 to avoid spurious noises
from the internal speaker during (and after) reboot */
cx_auto_turn_eapd(codec, spec->num_eapds, spec->eapds, false);
snd_hda_codec_set_power_to_all(codec, codec->core.afg, AC_PWRST_D3);
snd_hda_codec_write(codec, codec->core.afg, 0,
AC_VERB_SET_POWER_STATE, AC_PWRST_D3);
+ msleep(10);
}
static void cx_auto_free(struct hda_codec *codec)
@@ -851,6 +853,9 @@ static const struct snd_pci_quirk cxt5066_fixups[] = {
SND_PCI_QUIRK(0x103c, 0x8079, "HP EliteBook 840 G3", CXT_FIXUP_HP_DOCK),
SND_PCI_QUIRK(0x103c, 0x807C, "HP EliteBook 820 G3", CXT_FIXUP_HP_DOCK),
SND_PCI_QUIRK(0x103c, 0x80FD, "HP ProBook 640 G2", CXT_FIXUP_HP_DOCK),
+ SND_PCI_QUIRK(0x103c, 0x828c, "HP EliteBook 840 G4", CXT_FIXUP_HP_DOCK),
+ SND_PCI_QUIRK(0x103c, 0x83b3, "HP EliteBook 830 G5", CXT_FIXUP_HP_DOCK),
+ SND_PCI_QUIRK(0x103c, 0x83d3, "HP ProBook 640 G4", CXT_FIXUP_HP_DOCK),
SND_PCI_QUIRK(0x103c, 0x8174, "HP Spectre x360", CXT_FIXUP_HP_SPECTRE),
SND_PCI_QUIRK(0x103c, 0x8115, "HP Z1 Gen3", CXT_FIXUP_HP_GATE_MIC),
SND_PCI_QUIRK(0x1043, 0x138d, "Asus", CXT_FIXUP_HEADPHONE_MIC_PIN),
@@ -863,6 +868,7 @@ static const struct snd_pci_quirk cxt5066_fixups[] = {
SND_PCI_QUIRK(0x17aa, 0x21da, "Lenovo X220", CXT_PINCFG_LENOVO_TP410),
SND_PCI_QUIRK(0x17aa, 0x21db, "Lenovo X220-tablet", CXT_PINCFG_LENOVO_TP410),
SND_PCI_QUIRK(0x17aa, 0x38af, "Lenovo IdeaPad Z560", CXT_FIXUP_MUTE_LED_EAPD),
+ SND_PCI_QUIRK(0x17aa, 0x3905, "Lenovo G50-30", CXT_FIXUP_STEREO_DMIC),
SND_PCI_QUIRK(0x17aa, 0x390b, "Lenovo G50-80", CXT_FIXUP_STEREO_DMIC),
SND_PCI_QUIRK(0x17aa, 0x3975, "Lenovo U300s", CXT_FIXUP_STEREO_DMIC),
SND_PCI_QUIRK(0x17aa, 0x3977, "Lenovo IdeaPad U310", CXT_FIXUP_STEREO_DMIC),
diff --git a/sound/pci/hda/patch_hdmi.c b/sound/pci/hda/patch_hdmi.c
index bd65022..76ae627 100644
--- a/sound/pci/hda/patch_hdmi.c
+++ b/sound/pci/hda/patch_hdmi.c
@@ -33,6 +33,7 @@
#include <linux/delay.h>
#include <linux/slab.h>
#include <linux/module.h>
+#include <linux/pm_runtime.h>
#include <sound/core.h>
#include <sound/jack.h>
#include <sound/asoundef.h>
@@ -731,8 +732,10 @@ static void check_presence_and_report(struct hda_codec *codec, hda_nid_t nid)
if (pin_idx < 0)
return;
+ mutex_lock(&spec->pcm_lock);
if (hdmi_present_sense(get_pin(spec, pin_idx), 1))
snd_hda_jack_report_sync(codec);
+ mutex_unlock(&spec->pcm_lock);
}
static void jack_callback(struct hda_codec *codec,
@@ -1521,21 +1524,23 @@ static void sync_eld_via_acomp(struct hda_codec *codec,
static bool hdmi_present_sense(struct hdmi_spec_per_pin *per_pin, int repoll)
{
struct hda_codec *codec = per_pin->codec;
- struct hdmi_spec *spec = codec->spec;
int ret;
/* no temporary power up/down needed for component notifier */
- if (!codec_has_acomp(codec))
- snd_hda_power_up_pm(codec);
+ if (!codec_has_acomp(codec)) {
+ ret = snd_hda_power_up_pm(codec);
+ if (ret < 0 && pm_runtime_suspended(hda_codec_dev(codec))) {
+ snd_hda_power_down_pm(codec);
+ return false;
+ }
+ }
- mutex_lock(&spec->pcm_lock);
if (codec_has_acomp(codec)) {
sync_eld_via_acomp(codec, per_pin);
ret = false; /* don't call snd_hda_jack_report_sync() */
} else {
ret = hdmi_present_sense_via_verbs(per_pin, repoll);
}
- mutex_unlock(&spec->pcm_lock);
if (!codec_has_acomp(codec))
snd_hda_power_down_pm(codec);
@@ -1547,12 +1552,16 @@ static void hdmi_repoll_eld(struct work_struct *work)
{
struct hdmi_spec_per_pin *per_pin =
container_of(to_delayed_work(work), struct hdmi_spec_per_pin, work);
+ struct hda_codec *codec = per_pin->codec;
+ struct hdmi_spec *spec = codec->spec;
if (per_pin->repoll_count++ > 6)
per_pin->repoll_count = 0;
+ mutex_lock(&spec->pcm_lock);
if (hdmi_present_sense(per_pin, per_pin->repoll_count))
snd_hda_jack_report_sync(per_pin->codec);
+ mutex_unlock(&spec->pcm_lock);
}
static void intel_haswell_fixup_connect_list(struct hda_codec *codec,
diff --git a/sound/pci/hda/patch_realtek.c b/sound/pci/hda/patch_realtek.c
index e2230be..0fd31cf 100644
--- a/sound/pci/hda/patch_realtek.c
+++ b/sound/pci/hda/patch_realtek.c
@@ -329,9 +329,11 @@ static void alc_fill_eapd_coef(struct hda_codec *codec)
break;
case 0x10ec0225:
case 0x10ec0233:
+ case 0x10ec0235:
case 0x10ec0236:
case 0x10ec0255:
case 0x10ec0256:
+ case 0x10ec0257:
case 0x10ec0282:
case 0x10ec0283:
case 0x10ec0286:
@@ -2446,6 +2448,7 @@ static const struct snd_pci_quirk alc262_fixup_tbl[] = {
SND_PCI_QUIRK(0x10cf, 0x1397, "Fujitsu Lifebook S7110", ALC262_FIXUP_FSC_S7110),
SND_PCI_QUIRK(0x10cf, 0x142d, "Fujitsu Lifebook E8410", ALC262_FIXUP_BENQ),
SND_PCI_QUIRK(0x10f1, 0x2915, "Tyan Thunder n6650W", ALC262_FIXUP_TYAN),
+ SND_PCI_QUIRK(0x1734, 0x1141, "FSC ESPRIMO U9210", ALC262_FIXUP_FSC_H270),
SND_PCI_QUIRK(0x1734, 0x1147, "FSC Celsius H270", ALC262_FIXUP_FSC_H270),
SND_PCI_QUIRK(0x17aa, 0x384e, "Lenovo 3000", ALC262_FIXUP_LENOVO_3000),
SND_PCI_QUIRK(0x17ff, 0x0560, "Benq ED8", ALC262_FIXUP_BENQ),
@@ -2662,6 +2665,7 @@ enum {
ALC269_TYPE_ALC298,
ALC269_TYPE_ALC255,
ALC269_TYPE_ALC256,
+ ALC269_TYPE_ALC257,
ALC269_TYPE_ALC225,
ALC269_TYPE_ALC294,
ALC269_TYPE_ALC700,
@@ -2694,6 +2698,7 @@ static int alc269_parse_auto_config(struct hda_codec *codec)
case ALC269_TYPE_ALC298:
case ALC269_TYPE_ALC255:
case ALC269_TYPE_ALC256:
+ case ALC269_TYPE_ALC257:
case ALC269_TYPE_ALC225:
case ALC269_TYPE_ALC294:
case ALC269_TYPE_ALC700:
@@ -3494,6 +3499,7 @@ static void alc280_fixup_hp_gpio4(struct hda_codec *codec,
}
}
+#if IS_REACHABLE(CONFIG_INPUT)
static void gpio2_mic_hotkey_event(struct hda_codec *codec,
struct hda_jack_callback *event)
{
@@ -3626,6 +3632,10 @@ static void alc233_fixup_lenovo_line2_mic_hotkey(struct hda_codec *codec,
spec->kb_dev = NULL;
}
}
+#else /* INPUT */
+#define alc280_fixup_hp_gpio2_mic_hotkey NULL
+#define alc233_fixup_lenovo_line2_mic_hotkey NULL
+#endif /* INPUT */
static void alc269_fixup_hp_line1_mic1_led(struct hda_codec *codec,
const struct hda_fixup *fix, int action)
@@ -4464,7 +4474,6 @@ static void alc_fixup_tpt440_dock(struct hda_codec *codec,
struct alc_spec *spec = codec->spec;
if (action == HDA_FIXUP_ACT_PRE_PROBE) {
- spec->shutup = alc_no_shutup; /* reduce click noise */
spec->reboot_notify = alc_d3_at_reboot; /* reduce noise */
spec->parse_flags = HDA_PINCFG_NO_HP_FIXUP;
codec->power_save_node = 0; /* avoid click noises */
@@ -4480,9 +4489,18 @@ static void alc_fixup_tpt470_dock(struct hda_codec *codec,
{ 0x19, 0x21a11010 }, /* dock mic */
{ }
};
+ /* Assure the speaker pin to be coupled with DAC NID 0x03; otherwise
+ * the speaker output becomes too low by some reason on Thinkpads with
+ * ALC298 codec
+ */
+ static hda_nid_t preferred_pairs[] = {
+ 0x14, 0x03, 0x17, 0x02, 0x21, 0x02,
+ 0
+ };
struct alc_spec *spec = codec->spec;
if (action == HDA_FIXUP_ACT_PRE_PROBE) {
+ spec->gen.preferred_dacs = preferred_pairs;
spec->parse_flags = HDA_PINCFG_NO_HP_FIXUP;
snd_hda_apply_pincfgs(codec, pincfgs);
} else if (action == HDA_FIXUP_ACT_INIT) {
@@ -4826,6 +4844,13 @@ static void alc280_fixup_hp_9480m(struct hda_codec *codec,
/* for hda_fixup_thinkpad_acpi() */
#include "thinkpad_helper.c"
+static void alc_fixup_thinkpad_acpi(struct hda_codec *codec,
+ const struct hda_fixup *fix, int action)
+{
+ alc_fixup_no_shutup(codec, fix, action); /* reduce click noise */
+ hda_fixup_thinkpad_acpi(codec, fix, action);
+}
+
/* for dell wmi mic mute led */
#include "dell_wmi_helper.c"
@@ -5341,7 +5366,7 @@ static const struct hda_fixup alc269_fixups[] = {
},
[ALC269_FIXUP_THINKPAD_ACPI] = {
.type = HDA_FIXUP_FUNC,
- .v.func = hda_fixup_thinkpad_acpi,
+ .v.func = alc_fixup_thinkpad_acpi,
.chained = true,
.chain_id = ALC269_FIXUP_SKU_IGNORE,
},
@@ -5682,6 +5707,7 @@ static const struct snd_pci_quirk alc269_fixup_tbl[] = {
SND_PCI_QUIRK(0x1028, 0x0706, "Dell Inspiron 7559", ALC256_FIXUP_DELL_INSPIRON_7559_SUBWOOFER),
SND_PCI_QUIRK(0x1028, 0x0725, "Dell Inspiron 3162", ALC255_FIXUP_DELL_SPK_NOISE),
SND_PCI_QUIRK(0x1028, 0x075b, "Dell XPS 13 9360", ALC256_FIXUP_DELL_XPS_13_HEADPHONE_NOISE),
+ SND_PCI_QUIRK(0x1028, 0x075c, "Dell XPS 27 7760", ALC298_FIXUP_SPK_VOLUME),
SND_PCI_QUIRK(0x1028, 0x075d, "Dell AIO", ALC298_FIXUP_SPK_VOLUME),
SND_PCI_QUIRK(0x1028, 0x07b0, "Dell Precision 7520", ALC295_FIXUP_DISABLE_DAC3),
SND_PCI_QUIRK(0x1028, 0x0798, "Dell Inspiron 17 7000 Gaming", ALC256_FIXUP_DELL_INSPIRON_7559_SUBWOOFER),
@@ -6359,6 +6385,7 @@ static int patch_alc269(struct hda_codec *codec)
case 0x10ec0298:
spec->codec_variant = ALC269_TYPE_ALC298;
break;
+ case 0x10ec0235:
case 0x10ec0255:
spec->codec_variant = ALC269_TYPE_ALC255;
break;
@@ -6368,10 +6395,17 @@ static int patch_alc269(struct hda_codec *codec)
spec->gen.mixer_nid = 0; /* ALC256 does not have any loopback mixer path */
alc_update_coef_idx(codec, 0x36, 1 << 13, 1 << 5); /* Switch pcbeep path to Line in path*/
break;
+ case 0x10ec0257:
+ spec->codec_variant = ALC269_TYPE_ALC257;
+ spec->gen.mixer_nid = 0;
+ break;
case 0x10ec0225:
case 0x10ec0295:
+ spec->codec_variant = ALC269_TYPE_ALC225;
+ break;
case 0x10ec0299:
spec->codec_variant = ALC269_TYPE_ALC225;
+ spec->gen.mixer_nid = 0; /* no loopback on ALC299 */
break;
case 0x10ec0234:
case 0x10ec0274:
@@ -6786,6 +6820,8 @@ enum {
ALC662_FIXUP_ASUS_Nx50,
ALC668_FIXUP_ASUS_Nx51_HEADSET_MODE,
ALC668_FIXUP_ASUS_Nx51,
+ ALC668_FIXUP_MIC_COEF,
+ ALC668_FIXUP_ASUS_G751,
ALC891_FIXUP_HEADSET_MODE,
ALC891_FIXUP_DELL_MIC_NO_PRESENCE,
ALC662_FIXUP_ACER_VERITON,
@@ -7052,6 +7088,23 @@ static const struct hda_fixup alc662_fixups[] = {
.chained = true,
.chain_id = ALC668_FIXUP_ASUS_Nx51_HEADSET_MODE,
},
+ [ALC668_FIXUP_MIC_COEF] = {
+ .type = HDA_FIXUP_VERBS,
+ .v.verbs = (const struct hda_verb[]) {
+ { 0x20, AC_VERB_SET_COEF_INDEX, 0xc3 },
+ { 0x20, AC_VERB_SET_PROC_COEF, 0x4000 },
+ {}
+ },
+ },
+ [ALC668_FIXUP_ASUS_G751] = {
+ .type = HDA_FIXUP_PINS,
+ .v.pins = (const struct hda_pintbl[]) {
+ { 0x16, 0x0421101f }, /* HP */
+ {}
+ },
+ .chained = true,
+ .chain_id = ALC668_FIXUP_MIC_COEF
+ },
[ALC891_FIXUP_HEADSET_MODE] = {
.type = HDA_FIXUP_FUNC,
.v.func = alc_fixup_headset_mode,
@@ -7107,6 +7160,7 @@ static const struct snd_pci_quirk alc662_fixup_tbl[] = {
SND_PCI_QUIRK(0x1043, 0x11cd, "Asus N550", ALC662_FIXUP_ASUS_Nx50),
SND_PCI_QUIRK(0x1043, 0x13df, "Asus N550JX", ALC662_FIXUP_BASS_1A),
SND_PCI_QUIRK(0x1043, 0x129d, "Asus N750", ALC662_FIXUP_ASUS_Nx50),
+ SND_PCI_QUIRK(0x1043, 0x12ff, "ASUS G751", ALC668_FIXUP_ASUS_G751),
SND_PCI_QUIRK(0x1043, 0x1477, "ASUS N56VZ", ALC662_FIXUP_BASS_MODE4_CHMAP),
SND_PCI_QUIRK(0x1043, 0x15a7, "ASUS UX51VZH", ALC662_FIXUP_BASS_16),
SND_PCI_QUIRK(0x1043, 0x177d, "ASUS N551", ALC668_FIXUP_ASUS_Nx51),
@@ -7354,6 +7408,7 @@ static const struct hda_device_id snd_hda_id_realtek[] = {
HDA_CODEC_ENTRY(0x10ec0236, "ALC236", patch_alc269),
HDA_CODEC_ENTRY(0x10ec0255, "ALC255", patch_alc269),
HDA_CODEC_ENTRY(0x10ec0256, "ALC256", patch_alc269),
+ HDA_CODEC_ENTRY(0x10ec0257, "ALC257", patch_alc269),
HDA_CODEC_ENTRY(0x10ec0260, "ALC260", patch_alc260),
HDA_CODEC_ENTRY(0x10ec0262, "ALC262", patch_alc262),
HDA_CODEC_ENTRY(0x10ec0267, "ALC267", patch_alc268),
diff --git a/sound/pci/rme9652/hdsp.c b/sound/pci/rme9652/hdsp.c
index b94fc63..b044dea 100644
--- a/sound/pci/rme9652/hdsp.c
+++ b/sound/pci/rme9652/hdsp.c
@@ -30,6 +30,7 @@
#include <linux/math64.h>
#include <linux/vmalloc.h>
#include <linux/io.h>
+#include <linux/nospec.h>
#include <sound/core.h>
#include <sound/control.h>
@@ -4065,15 +4066,16 @@ static int snd_hdsp_channel_info(struct snd_pcm_substream *substream,
struct snd_pcm_channel_info *info)
{
struct hdsp *hdsp = snd_pcm_substream_chip(substream);
- int mapped_channel;
+ unsigned int channel = info->channel;
- if (snd_BUG_ON(info->channel >= hdsp->max_channels))
+ if (snd_BUG_ON(channel >= hdsp->max_channels))
+ return -EINVAL;
+ channel = array_index_nospec(channel, hdsp->max_channels);
+
+ if (hdsp->channel_map[channel] < 0)
return -EINVAL;
- if ((mapped_channel = hdsp->channel_map[info->channel]) < 0)
- return -EINVAL;
-
- info->offset = mapped_channel * HDSP_CHANNEL_BUFFER_BYTES;
+ info->offset = hdsp->channel_map[channel] * HDSP_CHANNEL_BUFFER_BYTES;
info->first = 0;
info->step = 32;
return 0;
diff --git a/sound/pci/rme9652/hdspm.c b/sound/pci/rme9652/hdspm.c
index 14bbf55..9899ef4 100644
--- a/sound/pci/rme9652/hdspm.c
+++ b/sound/pci/rme9652/hdspm.c
@@ -137,6 +137,7 @@
#include <linux/pci.h>
#include <linux/math64.h>
#include <linux/io.h>
+#include <linux/nospec.h>
#include <sound/core.h>
#include <sound/control.h>
@@ -5692,40 +5693,43 @@ static int snd_hdspm_channel_info(struct snd_pcm_substream *substream,
struct snd_pcm_channel_info *info)
{
struct hdspm *hdspm = snd_pcm_substream_chip(substream);
+ unsigned int channel = info->channel;
if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) {
- if (snd_BUG_ON(info->channel >= hdspm->max_channels_out)) {
+ if (snd_BUG_ON(channel >= hdspm->max_channels_out)) {
dev_info(hdspm->card->dev,
"snd_hdspm_channel_info: output channel out of range (%d)\n",
- info->channel);
+ channel);
return -EINVAL;
}
- if (hdspm->channel_map_out[info->channel] < 0) {
+ channel = array_index_nospec(channel, hdspm->max_channels_out);
+ if (hdspm->channel_map_out[channel] < 0) {
dev_info(hdspm->card->dev,
"snd_hdspm_channel_info: output channel %d mapped out\n",
- info->channel);
+ channel);
return -EINVAL;
}
- info->offset = hdspm->channel_map_out[info->channel] *
+ info->offset = hdspm->channel_map_out[channel] *
HDSPM_CHANNEL_BUFFER_BYTES;
} else {
- if (snd_BUG_ON(info->channel >= hdspm->max_channels_in)) {
+ if (snd_BUG_ON(channel >= hdspm->max_channels_in)) {
dev_info(hdspm->card->dev,
"snd_hdspm_channel_info: input channel out of range (%d)\n",
- info->channel);
+ channel);
return -EINVAL;
}
- if (hdspm->channel_map_in[info->channel] < 0) {
+ channel = array_index_nospec(channel, hdspm->max_channels_in);
+ if (hdspm->channel_map_in[channel] < 0) {
dev_info(hdspm->card->dev,
"snd_hdspm_channel_info: input channel %d mapped out\n",
- info->channel);
+ channel);
return -EINVAL;
}
- info->offset = hdspm->channel_map_in[info->channel] *
+ info->offset = hdspm->channel_map_in[channel] *
HDSPM_CHANNEL_BUFFER_BYTES;
}
diff --git a/sound/pci/rme9652/rme9652.c b/sound/pci/rme9652/rme9652.c
index 55172c6..a76b1f1 100644
--- a/sound/pci/rme9652/rme9652.c
+++ b/sound/pci/rme9652/rme9652.c
@@ -26,6 +26,7 @@
#include <linux/pci.h>
#include <linux/module.h>
#include <linux/io.h>
+#include <linux/nospec.h>
#include <sound/core.h>
#include <sound/control.h>
@@ -2036,9 +2037,10 @@ static int snd_rme9652_channel_info(struct snd_pcm_substream *substream,
if (snd_BUG_ON(info->channel >= RME9652_NCHANNELS))
return -EINVAL;
- if ((chn = rme9652->channel_map[info->channel]) < 0) {
+ chn = rme9652->channel_map[array_index_nospec(info->channel,
+ RME9652_NCHANNELS)];
+ if (chn < 0)
return -EINVAL;
- }
info->offset = chn * RME9652_CHANNEL_BUFFER_BYTES;
info->first = 0;
diff --git a/sound/pci/trident/trident.c b/sound/pci/trident/trident.c
index cedf13b..2f18b1c 100644
--- a/sound/pci/trident/trident.c
+++ b/sound/pci/trident/trident.c
@@ -123,7 +123,7 @@ static int snd_trident_probe(struct pci_dev *pci,
} else {
strcpy(card->shortname, "Trident ");
}
- strcat(card->shortname, card->driver);
+ strcat(card->shortname, str);
sprintf(card->longname, "%s PCI Audio at 0x%lx, irq %d",
card->shortname, trident->port, trident->irq);
diff --git a/sound/pci/vx222/vx222_ops.c b/sound/pci/vx222/vx222_ops.c
index 8e457ea..1997bb0 100644
--- a/sound/pci/vx222/vx222_ops.c
+++ b/sound/pci/vx222/vx222_ops.c
@@ -275,7 +275,7 @@ static void vx2_dma_write(struct vx_core *chip, struct snd_pcm_runtime *runtime,
length >>= 2; /* in 32bit words */
/* Transfer using pseudo-dma. */
for (; length > 0; length--) {
- outl(cpu_to_le32(*addr), port);
+ outl(*addr, port);
addr++;
}
addr = (u32 *)runtime->dma_area;
@@ -285,7 +285,7 @@ static void vx2_dma_write(struct vx_core *chip, struct snd_pcm_runtime *runtime,
count >>= 2; /* in 32bit words */
/* Transfer using pseudo-dma. */
for (; count > 0; count--) {
- outl(cpu_to_le32(*addr), port);
+ outl(*addr, port);
addr++;
}
@@ -313,7 +313,7 @@ static void vx2_dma_read(struct vx_core *chip, struct snd_pcm_runtime *runtime,
length >>= 2; /* in 32bit words */
/* Transfer using pseudo-dma. */
for (; length > 0; length--)
- *addr++ = le32_to_cpu(inl(port));
+ *addr++ = inl(port);
addr = (u32 *)runtime->dma_area;
pipe->hw_ptr = 0;
}
@@ -321,7 +321,7 @@ static void vx2_dma_read(struct vx_core *chip, struct snd_pcm_runtime *runtime,
count >>= 2; /* in 32bit words */
/* Transfer using pseudo-dma. */
for (; count > 0; count--)
- *addr++ = le32_to_cpu(inl(port));
+ *addr++ = inl(port);
vx2_release_pseudo_dma(chip);
}
diff --git a/sound/pcmcia/vx/vxp_ops.c b/sound/pcmcia/vx/vxp_ops.c
index 56aa1ba..49a8833 100644
--- a/sound/pcmcia/vx/vxp_ops.c
+++ b/sound/pcmcia/vx/vxp_ops.c
@@ -375,7 +375,7 @@ static void vxp_dma_write(struct vx_core *chip, struct snd_pcm_runtime *runtime,
length >>= 1; /* in 16bit words */
/* Transfer using pseudo-dma. */
for (; length > 0; length--) {
- outw(cpu_to_le16(*addr), port);
+ outw(*addr, port);
addr++;
}
addr = (unsigned short *)runtime->dma_area;
@@ -385,7 +385,7 @@ static void vxp_dma_write(struct vx_core *chip, struct snd_pcm_runtime *runtime,
count >>= 1; /* in 16bit words */
/* Transfer using pseudo-dma. */
for (; count > 0; count--) {
- outw(cpu_to_le16(*addr), port);
+ outw(*addr, port);
addr++;
}
vx_release_pseudo_dma(chip);
@@ -417,7 +417,7 @@ static void vxp_dma_read(struct vx_core *chip, struct snd_pcm_runtime *runtime,
length >>= 1; /* in 16bit words */
/* Transfer using pseudo-dma. */
for (; length > 0; length--)
- *addr++ = le16_to_cpu(inw(port));
+ *addr++ = inw(port);
addr = (unsigned short *)runtime->dma_area;
pipe->hw_ptr = 0;
}
@@ -425,12 +425,12 @@ static void vxp_dma_read(struct vx_core *chip, struct snd_pcm_runtime *runtime,
count >>= 1; /* in 16bit words */
/* Transfer using pseudo-dma. */
for (; count > 1; count--)
- *addr++ = le16_to_cpu(inw(port));
+ *addr++ = inw(port);
/* Disable DMA */
pchip->regDIALOG &= ~VXP_DLG_DMAREAD_SEL_MASK;
vx_outb(chip, DIALOG, pchip->regDIALOG);
/* Read the last word (16 bits) */
- *addr = le16_to_cpu(inw(port));
+ *addr = inw(port);
/* Disable 16-bit accesses */
pchip->regDIALOG &= ~VXP_DLG_DMA16_SEL_MASK;
vx_outb(chip, DIALOG, pchip->regDIALOG);
diff --git a/sound/soc/au1x/ac97c.c b/sound/soc/au1x/ac97c.c
index 29a97d5..66d6c52 100644
--- a/sound/soc/au1x/ac97c.c
+++ b/sound/soc/au1x/ac97c.c
@@ -91,8 +91,8 @@ static unsigned short au1xac97c_ac97_read(struct snd_ac97 *ac97,
do {
mutex_lock(&ctx->lock);
- tmo = 5;
- while ((RD(ctx, AC97_STATUS) & STAT_CP) && tmo--)
+ tmo = 6;
+ while ((RD(ctx, AC97_STATUS) & STAT_CP) && --tmo)
udelay(21); /* wait an ac97 frame time */
if (!tmo) {
pr_debug("ac97rd timeout #1\n");
@@ -105,7 +105,7 @@ static unsigned short au1xac97c_ac97_read(struct snd_ac97 *ac97,
* poll, Forrest, poll...
*/
tmo = 0x10000;
- while ((RD(ctx, AC97_STATUS) & STAT_CP) && tmo--)
+ while ((RD(ctx, AC97_STATUS) & STAT_CP) && --tmo)
asm volatile ("nop");
data = RD(ctx, AC97_CMDRESP);
diff --git a/sound/soc/cirrus/edb93xx.c b/sound/soc/cirrus/edb93xx.c
index 85962657..517963e 100644
--- a/sound/soc/cirrus/edb93xx.c
+++ b/sound/soc/cirrus/edb93xx.c
@@ -67,7 +67,7 @@ static struct snd_soc_dai_link edb93xx_dai = {
.cpu_dai_name = "ep93xx-i2s",
.codec_name = "spi0.0",
.codec_dai_name = "cs4271-hifi",
- .dai_fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_IF |
+ .dai_fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF |
SND_SOC_DAIFMT_CBS_CFS,
.ops = &edb93xx_ops,
};
diff --git a/sound/soc/cirrus/ep93xx-i2s.c b/sound/soc/cirrus/ep93xx-i2s.c
index 934f8ae..0dc3852 100644
--- a/sound/soc/cirrus/ep93xx-i2s.c
+++ b/sound/soc/cirrus/ep93xx-i2s.c
@@ -51,7 +51,9 @@
#define EP93XX_I2S_WRDLEN_24 (1 << 0)
#define EP93XX_I2S_WRDLEN_32 (2 << 0)
-#define EP93XX_I2S_LINCTRLDATA_R_JUST (1 << 2) /* Right justify */
+#define EP93XX_I2S_RXLINCTRLDATA_R_JUST BIT(1) /* Right justify */
+
+#define EP93XX_I2S_TXLINCTRLDATA_R_JUST BIT(2) /* Right justify */
#define EP93XX_I2S_CLKCFG_LRS (1 << 0) /* lrclk polarity */
#define EP93XX_I2S_CLKCFG_CKP (1 << 1) /* Bit clock polarity */
@@ -170,25 +172,25 @@ static int ep93xx_i2s_set_dai_fmt(struct snd_soc_dai *cpu_dai,
unsigned int fmt)
{
struct ep93xx_i2s_info *info = snd_soc_dai_get_drvdata(cpu_dai);
- unsigned int clk_cfg, lin_ctrl;
+ unsigned int clk_cfg;
+ unsigned int txlin_ctrl = 0;
+ unsigned int rxlin_ctrl = 0;
clk_cfg = ep93xx_i2s_read_reg(info, EP93XX_I2S_RXCLKCFG);
- lin_ctrl = ep93xx_i2s_read_reg(info, EP93XX_I2S_RXLINCTRLDATA);
switch (fmt & SND_SOC_DAIFMT_FORMAT_MASK) {
case SND_SOC_DAIFMT_I2S:
clk_cfg |= EP93XX_I2S_CLKCFG_REL;
- lin_ctrl &= ~EP93XX_I2S_LINCTRLDATA_R_JUST;
break;
case SND_SOC_DAIFMT_LEFT_J:
clk_cfg &= ~EP93XX_I2S_CLKCFG_REL;
- lin_ctrl &= ~EP93XX_I2S_LINCTRLDATA_R_JUST;
break;
case SND_SOC_DAIFMT_RIGHT_J:
clk_cfg &= ~EP93XX_I2S_CLKCFG_REL;
- lin_ctrl |= EP93XX_I2S_LINCTRLDATA_R_JUST;
+ rxlin_ctrl |= EP93XX_I2S_RXLINCTRLDATA_R_JUST;
+ txlin_ctrl |= EP93XX_I2S_TXLINCTRLDATA_R_JUST;
break;
default:
@@ -213,32 +215,32 @@ static int ep93xx_i2s_set_dai_fmt(struct snd_soc_dai *cpu_dai,
switch (fmt & SND_SOC_DAIFMT_INV_MASK) {
case SND_SOC_DAIFMT_NB_NF:
/* Negative bit clock, lrclk low on left word */
- clk_cfg &= ~(EP93XX_I2S_CLKCFG_CKP | EP93XX_I2S_CLKCFG_REL);
+ clk_cfg &= ~(EP93XX_I2S_CLKCFG_CKP | EP93XX_I2S_CLKCFG_LRS);
break;
case SND_SOC_DAIFMT_NB_IF:
/* Negative bit clock, lrclk low on right word */
clk_cfg &= ~EP93XX_I2S_CLKCFG_CKP;
- clk_cfg |= EP93XX_I2S_CLKCFG_REL;
+ clk_cfg |= EP93XX_I2S_CLKCFG_LRS;
break;
case SND_SOC_DAIFMT_IB_NF:
/* Positive bit clock, lrclk low on left word */
clk_cfg |= EP93XX_I2S_CLKCFG_CKP;
- clk_cfg &= ~EP93XX_I2S_CLKCFG_REL;
+ clk_cfg &= ~EP93XX_I2S_CLKCFG_LRS;
break;
case SND_SOC_DAIFMT_IB_IF:
/* Positive bit clock, lrclk low on right word */
- clk_cfg |= EP93XX_I2S_CLKCFG_CKP | EP93XX_I2S_CLKCFG_REL;
+ clk_cfg |= EP93XX_I2S_CLKCFG_CKP | EP93XX_I2S_CLKCFG_LRS;
break;
}
/* Write new register values */
ep93xx_i2s_write_reg(info, EP93XX_I2S_RXCLKCFG, clk_cfg);
ep93xx_i2s_write_reg(info, EP93XX_I2S_TXCLKCFG, clk_cfg);
- ep93xx_i2s_write_reg(info, EP93XX_I2S_RXLINCTRLDATA, lin_ctrl);
- ep93xx_i2s_write_reg(info, EP93XX_I2S_TXLINCTRLDATA, lin_ctrl);
+ ep93xx_i2s_write_reg(info, EP93XX_I2S_RXLINCTRLDATA, rxlin_ctrl);
+ ep93xx_i2s_write_reg(info, EP93XX_I2S_TXLINCTRLDATA, txlin_ctrl);
return 0;
}
diff --git a/sound/soc/cirrus/snappercl15.c b/sound/soc/cirrus/snappercl15.c
index 98089df..c6737a5 100644
--- a/sound/soc/cirrus/snappercl15.c
+++ b/sound/soc/cirrus/snappercl15.c
@@ -72,7 +72,7 @@ static struct snd_soc_dai_link snappercl15_dai = {
.codec_dai_name = "tlv320aic23-hifi",
.codec_name = "tlv320aic23-codec.0-001a",
.platform_name = "ep93xx-i2s",
- .dai_fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_IF |
+ .dai_fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF |
SND_SOC_DAIFMT_CBS_CFS,
.ops = &snappercl15_ops,
};
diff --git a/sound/soc/codecs/Kconfig b/sound/soc/codecs/Kconfig
index c67667b..45f21ca 100644
--- a/sound/soc/codecs/Kconfig
+++ b/sound/soc/codecs/Kconfig
@@ -86,6 +86,7 @@
select SND_SOC_MAX9867 if I2C
select SND_SOC_MAX98925 if I2C
select SND_SOC_MAX98926 if I2C
+ select SND_SOC_MAX98927 if I2C
select SND_SOC_MAX9850 if I2C
select SND_SOC_MAX9860 if I2C
select SND_SOC_MAX9768 if I2C
@@ -573,6 +574,10 @@
config SND_SOC_MAX98926
tristate
+config SND_SOC_MAX98927
+ tristate "Maxim Integrated MAX98927 Speaker Amplifier"
+ depends on I2C
+
config SND_SOC_MAX9850
tristate
diff --git a/sound/soc/codecs/Makefile b/sound/soc/codecs/Makefile
index 958cd49..1f5fe2c 100644
--- a/sound/soc/codecs/Makefile
+++ b/sound/soc/codecs/Makefile
@@ -82,6 +82,7 @@
snd-soc-max9867-objs := max9867.o
snd-soc-max98925-objs := max98925.o
snd-soc-max98926-objs := max98926.o
+snd-soc-max98927-objs := max98927.o
snd-soc-max9850-objs := max9850.o
snd-soc-max9860-objs := max9860.o
snd-soc-mc13783-objs := mc13783.o
@@ -306,6 +307,7 @@
obj-$(CONFIG_SND_SOC_MAX9867) += snd-soc-max9867.o
obj-$(CONFIG_SND_SOC_MAX98925) += snd-soc-max98925.o
obj-$(CONFIG_SND_SOC_MAX98926) += snd-soc-max98926.o
+obj-$(CONFIG_SND_SOC_MAX98927) += snd-soc-max98927.o
obj-$(CONFIG_SND_SOC_MAX9850) += snd-soc-max9850.o
obj-$(CONFIG_SND_SOC_MAX9860) += snd-soc-max9860.o
obj-$(CONFIG_SND_SOC_MC13783) += snd-soc-mc13783.o
diff --git a/sound/soc/codecs/cs4265.c b/sound/soc/codecs/cs4265.c
index fd966bb..6e8eb1f 100644
--- a/sound/soc/codecs/cs4265.c
+++ b/sound/soc/codecs/cs4265.c
@@ -157,8 +157,8 @@ static const struct snd_kcontrol_new cs4265_snd_controls[] = {
SOC_SINGLE("Validity Bit Control Switch", CS4265_SPDIF_CTL2,
3, 1, 0),
SOC_ENUM("SPDIF Mono/Stereo", spdif_mono_stereo_enum),
- SOC_SINGLE("MMTLR Data Switch", 0,
- 1, 1, 0),
+ SOC_SINGLE("MMTLR Data Switch", CS4265_SPDIF_CTL2,
+ 0, 1, 0),
SOC_ENUM("Mono Channel Select", spdif_mono_select_enum),
SND_SOC_BYTES("C Data Buffer", CS4265_C_DATA_BUFF, 24),
};
diff --git a/sound/soc/codecs/max98927.c b/sound/soc/codecs/max98927.c
new file mode 100644
index 0000000..1ca4007
--- /dev/null
+++ b/sound/soc/codecs/max98927.c
@@ -0,0 +1,1061 @@
+/*
+ * max98927.c -- MAX98927 ALSA Soc Audio driver
+ *
+ * Copyright (C) 2016 Maxim Integrated Products
+ * Author: Ryan Lee <ryans.lee@maximintegrated.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ */
+
+#include <linux/acpi.h>
+#include <linux/i2c.h>
+#include <linux/module.h>
+#include <linux/regmap.h>
+#include <linux/slab.h>
+#include <linux/cdev.h>
+#include <sound/pcm.h>
+#include <sound/pcm_params.h>
+#include <sound/soc.h>
+#include <linux/gpio.h>
+#include <linux/of_gpio.h>
+#include <sound/tlv.h>
+#include "max98927.h"
+
+static struct reg_default max98927_reg[] = {
+ {MAX98927_R0001_INT_RAW1, 0x00},
+ {MAX98927_R0002_INT_RAW2, 0x00},
+ {MAX98927_R0003_INT_RAW3, 0x00},
+ {MAX98927_R0004_INT_STATE1, 0x00},
+ {MAX98927_R0005_INT_STATE2, 0x00},
+ {MAX98927_R0006_INT_STATE3, 0x00},
+ {MAX98927_R0007_INT_FLAG1, 0x00},
+ {MAX98927_R0008_INT_FLAG2, 0x00},
+ {MAX98927_R0009_INT_FLAG3, 0x00},
+ {MAX98927_R000A_INT_EN1, 0x00},
+ {MAX98927_R000B_INT_EN2, 0x00},
+ {MAX98927_R000C_INT_EN3, 0x00},
+ {MAX98927_R000D_INT_FLAG_CLR1, 0x00},
+ {MAX98927_R000E_INT_FLAG_CLR2, 0x00},
+ {MAX98927_R000F_INT_FLAG_CLR3, 0x00},
+ {MAX98927_R0010_IRQ_CTRL, 0x00},
+ {MAX98927_R0011_CLK_MON, 0x00},
+ {MAX98927_R0012_WDOG_CTRL, 0x00},
+ {MAX98927_R0013_WDOG_RST, 0x00},
+ {MAX98927_R0014_MEAS_ADC_THERM_WARN_THRESH, 0x00},
+ {MAX98927_R0015_MEAS_ADC_THERM_SHDN_THRESH, 0x00},
+ {MAX98927_R0016_MEAS_ADC_THERM_HYSTERESIS, 0x00},
+ {MAX98927_R0017_PIN_CFG, 0x55},
+ {MAX98927_R0018_PCM_RX_EN_A, 0x00},
+ {MAX98927_R0019_PCM_RX_EN_B, 0x00},
+ {MAX98927_R001A_PCM_TX_EN_A, 0x00},
+ {MAX98927_R001B_PCM_TX_EN_B, 0x00},
+ {MAX98927_R001C_PCM_TX_HIZ_CTRL_A, 0x00},
+ {MAX98927_R001D_PCM_TX_HIZ_CTRL_B, 0x00},
+ {MAX98927_R001E_PCM_TX_CH_SRC_A, 0x00},
+ {MAX98927_R001F_PCM_TX_CH_SRC_B, 0x00},
+ {MAX98927_R0020_PCM_MODE_CFG, 0x40},
+ {MAX98927_R0021_PCM_MASTER_MODE, 0x00},
+ {MAX98927_R0022_PCM_CLK_SETUP, 0x22},
+ {MAX98927_R0023_PCM_SR_SETUP1, 0x00},
+ {MAX98927_R0024_PCM_SR_SETUP2, 0x00},
+ {MAX98927_R0025_PCM_TO_SPK_MONOMIX_A, 0x00},
+ {MAX98927_R0026_PCM_TO_SPK_MONOMIX_B, 0x00},
+ {MAX98927_R0027_ICC_RX_EN_A, 0x00},
+ {MAX98927_R0028_ICC_RX_EN_B, 0x00},
+ {MAX98927_R002B_ICC_TX_EN_A, 0x00},
+ {MAX98927_R002C_ICC_TX_EN_B, 0x00},
+ {MAX98927_R002E_ICC_HIZ_MANUAL_MODE, 0x00},
+ {MAX98927_R002F_ICC_TX_HIZ_EN_A, 0x00},
+ {MAX98927_R0030_ICC_TX_HIZ_EN_B, 0x00},
+ {MAX98927_R0031_ICC_LNK_EN, 0x00},
+ {MAX98927_R0032_PDM_TX_EN, 0x00},
+ {MAX98927_R0033_PDM_TX_HIZ_CTRL, 0x00},
+ {MAX98927_R0034_PDM_TX_CTRL, 0x00},
+ {MAX98927_R0035_PDM_RX_CTRL, 0x00},
+ {MAX98927_R0036_AMP_VOL_CTRL, 0x00},
+ {MAX98927_R0037_AMP_DSP_CFG, 0x02},
+ {MAX98927_R0038_TONE_GEN_DC_CFG, 0x00},
+ {MAX98927_R0039_DRE_CTRL, 0x01},
+ {MAX98927_R003A_AMP_EN, 0x00},
+ {MAX98927_R003B_SPK_SRC_SEL, 0x00},
+ {MAX98927_R003C_SPK_GAIN, 0x00},
+ {MAX98927_R003D_SSM_CFG, 0x01},
+ {MAX98927_R003E_MEAS_EN, 0x00},
+ {MAX98927_R003F_MEAS_DSP_CFG, 0x04},
+ {MAX98927_R0040_BOOST_CTRL0, 0x00},
+ {MAX98927_R0041_BOOST_CTRL3, 0x00},
+ {MAX98927_R0042_BOOST_CTRL1, 0x00},
+ {MAX98927_R0043_MEAS_ADC_CFG, 0x00},
+ {MAX98927_R0044_MEAS_ADC_BASE_MSB, 0x00},
+ {MAX98927_R0045_MEAS_ADC_BASE_LSB, 0x00},
+ {MAX98927_R0046_ADC_CH0_DIVIDE, 0x00},
+ {MAX98927_R0047_ADC_CH1_DIVIDE, 0x00},
+ {MAX98927_R0048_ADC_CH2_DIVIDE, 0x00},
+ {MAX98927_R0049_ADC_CH0_FILT_CFG, 0x00},
+ {MAX98927_R004A_ADC_CH1_FILT_CFG, 0x00},
+ {MAX98927_R004B_ADC_CH2_FILT_CFG, 0x00},
+ {MAX98927_R004C_MEAS_ADC_CH0_READ, 0x00},
+ {MAX98927_R004D_MEAS_ADC_CH1_READ, 0x00},
+ {MAX98927_R004E_MEAS_ADC_CH2_READ, 0x00},
+ {MAX98927_R0051_BROWNOUT_STATUS, 0x00},
+ {MAX98927_R0052_BROWNOUT_EN, 0x00},
+ {MAX98927_R0053_BROWNOUT_INFINITE_HOLD, 0x00},
+ {MAX98927_R0054_BROWNOUT_INFINITE_HOLD_CLR, 0x00},
+ {MAX98927_R0055_BROWNOUT_LVL_HOLD, 0x00},
+ {MAX98927_R005A_BROWNOUT_LVL1_THRESH, 0x00},
+ {MAX98927_R005B_BROWNOUT_LVL2_THRESH, 0x00},
+ {MAX98927_R005C_BROWNOUT_LVL3_THRESH, 0x00},
+ {MAX98927_R005D_BROWNOUT_LVL4_THRESH, 0x00},
+ {MAX98927_R005E_BROWNOUT_THRESH_HYSTERYSIS, 0x00},
+ {MAX98927_R005F_BROWNOUT_AMP_LIMITER_ATK_REL, 0x00},
+ {MAX98927_R0060_BROWNOUT_AMP_GAIN_ATK_REL, 0x00},
+ {MAX98927_R0061_BROWNOUT_AMP1_CLIP_MODE, 0x00},
+ {MAX98927_R0072_BROWNOUT_LVL1_CUR_LIMIT, 0x00},
+ {MAX98927_R0073_BROWNOUT_LVL1_AMP1_CTRL1, 0x00},
+ {MAX98927_R0074_BROWNOUT_LVL1_AMP1_CTRL2, 0x00},
+ {MAX98927_R0075_BROWNOUT_LVL1_AMP1_CTRL3, 0x00},
+ {MAX98927_R0076_BROWNOUT_LVL2_CUR_LIMIT, 0x00},
+ {MAX98927_R0077_BROWNOUT_LVL2_AMP1_CTRL1, 0x00},
+ {MAX98927_R0078_BROWNOUT_LVL2_AMP1_CTRL2, 0x00},
+ {MAX98927_R0079_BROWNOUT_LVL2_AMP1_CTRL3, 0x00},
+ {MAX98927_R007A_BROWNOUT_LVL3_CUR_LIMIT, 0x00},
+ {MAX98927_R007B_BROWNOUT_LVL3_AMP1_CTRL1, 0x00},
+ {MAX98927_R007C_BROWNOUT_LVL3_AMP1_CTRL2, 0x00},
+ {MAX98927_R007D_BROWNOUT_LVL3_AMP1_CTRL3, 0x00},
+ {MAX98927_R007E_BROWNOUT_LVL4_CUR_LIMIT, 0x00},
+ {MAX98927_R007F_BROWNOUT_LVL4_AMP1_CTRL1, 0x00},
+ {MAX98927_R0080_BROWNOUT_LVL4_AMP1_CTRL2, 0x00},
+ {MAX98927_R0081_BROWNOUT_LVL4_AMP1_CTRL3, 0x00},
+ {MAX98927_R0082_ENV_TRACK_VOUT_HEADROOM, 0x00},
+ {MAX98927_R0083_ENV_TRACK_BOOST_VOUT_DELAY, 0x00},
+ {MAX98927_R0084_ENV_TRACK_REL_RATE, 0x00},
+ {MAX98927_R0085_ENV_TRACK_HOLD_RATE, 0x00},
+ {MAX98927_R0086_ENV_TRACK_CTRL, 0x00},
+ {MAX98927_R0087_ENV_TRACK_BOOST_VOUT_READ, 0x00},
+ {MAX98927_R00FF_GLOBAL_SHDN, 0x00},
+ {MAX98927_R0100_SOFT_RESET, 0x00},
+ {MAX98927_R01FF_REV_ID, 0x40},
+};
+
+static int max98927_reg_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol, unsigned int reg,
+ unsigned int mask, unsigned int shift)
+{
+ struct snd_soc_codec *codec = snd_soc_kcontrol_codec(kcontrol);
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+ int data;
+
+ regmap_read(max98927->regmap, reg, &data);
+ ucontrol->value.integer.value[0] = (data & mask) >> shift;
+ return 0;
+}
+
+static int max98927_reg_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol, unsigned int reg,
+ unsigned int mask, unsigned int shift)
+{
+ struct snd_soc_codec *codec = snd_soc_kcontrol_codec(kcontrol);
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+ unsigned int sel = ucontrol->value.integer.value[0];
+
+ regmap_update_bits(max98927->regmap, reg, mask, sel << shift);
+ dev_dbg(codec->dev, "%s: register 0x%02X, value 0x%02X\n",
+ __func__, reg, sel);
+ return 0;
+}
+
+static int max98927_dai_set_fmt(struct snd_soc_dai *codec_dai, unsigned int fmt)
+{
+ struct snd_soc_codec *codec = codec_dai->codec;
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+ unsigned int mode = 0;
+ unsigned int invert = 0;
+
+ dev_dbg(codec->dev, "%s: fmt 0x%08X\n", __func__, fmt);
+
+ switch (fmt & SND_SOC_DAIFMT_MASTER_MASK) {
+ case SND_SOC_DAIFMT_CBS_CFS:
+ mode = MAX98927_PCM_MASTER_MODE_SLAVE;
+ break;
+ case SND_SOC_DAIFMT_CBM_CFM:
+ max98927->master = true;
+ mode = MAX98927_PCM_MASTER_MODE_MASTER;
+ break;
+ case SND_SOC_DAIFMT_CBS_CFM:
+ mode = MAX98927_PCM_MASTER_MODE_HYBRID;
+ default:
+ dev_err(codec->dev, "DAI clock mode unsupported");
+ return -EINVAL;
+ }
+
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R0021_PCM_MASTER_MODE,
+ MAX98927_PCM_MASTER_MODE_MASK,
+ mode);
+
+ switch (fmt & SND_SOC_DAIFMT_INV_MASK) {
+ case SND_SOC_DAIFMT_NB_NF:
+ break;
+ case SND_SOC_DAIFMT_IB_NF:
+ invert = MAX98927_PCM_MODE_CFG_PCM_BCLKEDGE;
+ break;
+ default:
+ dev_err(codec->dev, "DAI invert mode unsupported");
+ return -EINVAL;
+ }
+
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R0020_PCM_MODE_CFG,
+ MAX98927_PCM_MODE_CFG_PCM_BCLKEDGE,
+ invert);
+
+ /* interface format */
+ switch (fmt & SND_SOC_DAIFMT_FORMAT_MASK) {
+ case SND_SOC_DAIFMT_I2S:
+ max98927->iface |= SND_SOC_DAIFMT_I2S;
+
+ break;
+ case SND_SOC_DAIFMT_LEFT_J:
+ max98927->iface |= SND_SOC_DAIFMT_LEFT_J;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R0020_PCM_MODE_CFG,
+ max98927->iface, max98927->iface);
+
+ /* pcm channel configuration */
+ if (max98927->iface & (SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_LEFT_J))
+ regmap_write(max98927->regmap,
+ MAX98927_R0018_PCM_RX_EN_A,
+ MAX98927_PCM_RX_CH0_EN|
+ MAX98927_PCM_RX_CH1_EN);
+
+ return 0;
+}
+
+/* codec MCLK rate in master mode */
+static const int rate_table[] = {
+ 5644800, 6000000, 6144000, 6500000,
+ 9600000, 11289600, 12000000, 12288000,
+ 13000000, 19200000,
+};
+
+static int max98927_set_clock(struct max98927_priv *max98927,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_soc_codec *codec = max98927->codec;
+ /* BCLK/LRCLK ratio calculation */
+ int blr_clk_ratio = params_channels(params) * max98927->ch_size;
+ int reg = MAX98927_R0022_PCM_CLK_SETUP;
+ int mask = MAX98927_PCM_CLK_SETUP_BSEL_MASK;
+ int value;
+
+ if (max98927->master) {
+ int i;
+ /* match rate to closest value */
+ for (i = 0; i < ARRAY_SIZE(rate_table); i++) {
+ if (rate_table[i] >= max98927->sysclk)
+ break;
+ }
+ if (i == ARRAY_SIZE(rate_table)) {
+ dev_err(codec->dev, "failed to find proper clock rate.\n");
+ return -EINVAL;
+ }
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R0021_PCM_MASTER_MODE,
+ MAX98927_PCM_MASTER_MODE_MCLK_MASK,
+ i << MAX98927_PCM_MASTER_MODE_MCLK_RATE_SHIFT);
+ }
+
+ switch (blr_clk_ratio) {
+ case 32:
+ value = 2;
+ break;
+ case 48:
+ value = 3;
+ break;
+ case 64:
+ value = 4;
+ break;
+ default:
+ return -EINVAL;
+ }
+ regmap_update_bits(max98927->regmap, reg, mask, value);
+ return 0;
+}
+
+static int max98927_dai_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params,
+ struct snd_soc_dai *dai)
+{
+ struct snd_soc_codec *codec = dai->codec;
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+ unsigned int sampling_rate = 0;
+ unsigned int chan_sz = 0;
+
+ /* pcm mode configuration */
+ switch (snd_pcm_format_width(params_format(params))) {
+ case 16:
+ chan_sz = MAX98927_PCM_MODE_CFG_CHANSZ_16;
+ max98927->ch_size = 16;
+ break;
+ case 24:
+ chan_sz = MAX98927_PCM_MODE_CFG_CHANSZ_24;
+ max98927->ch_size = 24;
+ break;
+ case 32:
+ chan_sz = MAX98927_PCM_MODE_CFG_CHANSZ_32;
+ max98927->ch_size = 32;
+ break;
+ default:
+ dev_err(codec->dev, "format unsupported %d",
+ params_format(params));
+ goto err;
+ }
+
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R0020_PCM_MODE_CFG,
+ MAX98927_PCM_MODE_CFG_CHANSZ_MASK, chan_sz);
+
+ dev_dbg(codec->dev, "format supported %d",
+ params_format(params));
+
+ /* sampling rate configuration */
+ switch (params_rate(params)) {
+ case 8000:
+ sampling_rate |= MAX98927_PCM_SR_SET1_SR_8000;
+ break;
+ case 11025:
+ sampling_rate |= MAX98927_PCM_SR_SET1_SR_11025;
+ break;
+ case 12000:
+ sampling_rate |= MAX98927_PCM_SR_SET1_SR_12000;
+ break;
+ case 16000:
+ sampling_rate |= MAX98927_PCM_SR_SET1_SR_16000;
+ break;
+ case 22050:
+ sampling_rate |= MAX98927_PCM_SR_SET1_SR_22050;
+ break;
+ case 24000:
+ sampling_rate |= MAX98927_PCM_SR_SET1_SR_24000;
+ break;
+ case 32000:
+ sampling_rate |= MAX98927_PCM_SR_SET1_SR_32000;
+ break;
+ case 44100:
+ sampling_rate |= MAX98927_PCM_SR_SET1_SR_44100;
+ break;
+ case 48000:
+ sampling_rate |= MAX98927_PCM_SR_SET1_SR_48000;
+ break;
+ default:
+ dev_err(codec->dev, "rate %d not supported\n",
+ params_rate(params));
+ goto err;
+ }
+ /* set DAI_SR to correct LRCLK frequency */
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R0023_PCM_SR_SETUP1,
+ MAX98927_PCM_SR_SET1_SR_MASK,
+ sampling_rate);
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R0024_PCM_SR_SETUP2,
+ MAX98927_PCM_SR_SET2_SR_MASK,
+ sampling_rate << MAX98927_PCM_SR_SET2_SR_SHIFT);
+
+ /* set sampling rate of IV */
+ if (max98927->interleave_mode &&
+ sampling_rate > MAX98927_PCM_SR_SET1_SR_16000)
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R0024_PCM_SR_SETUP2,
+ MAX98927_PCM_SR_SET2_IVADC_SR_MASK,
+ sampling_rate - 3);
+ else
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R0024_PCM_SR_SETUP2,
+ MAX98927_PCM_SR_SET2_IVADC_SR_MASK,
+ sampling_rate);
+ return max98927_set_clock(max98927, params);
+err:
+ return -EINVAL;
+}
+
+#define MAX98927_RATES SNDRV_PCM_RATE_8000_48000
+
+#define MAX98927_FORMATS (SNDRV_PCM_FMTBIT_S16_LE | \
+ SNDRV_PCM_FMTBIT_S24_LE | SNDRV_PCM_FMTBIT_S32_LE)
+
+static int max98927_dai_set_sysclk(struct snd_soc_dai *dai,
+ int clk_id, unsigned int freq, int dir)
+{
+ struct snd_soc_codec *codec = dai->codec;
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+
+ max98927->sysclk = freq;
+ return 0;
+}
+
+static const struct snd_soc_dai_ops max98927_dai_ops = {
+ .set_sysclk = max98927_dai_set_sysclk,
+ .set_fmt = max98927_dai_set_fmt,
+ .hw_params = max98927_dai_hw_params,
+};
+
+static int max98927_dac_event(struct snd_soc_dapm_widget *w,
+ struct snd_kcontrol *kcontrol, int event)
+{
+ struct snd_soc_codec *codec = snd_soc_dapm_to_codec(w->dapm);
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+
+ switch (event) {
+ case SND_SOC_DAPM_POST_PMU:
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R003A_AMP_EN,
+ MAX98927_AMP_EN_MASK, 1);
+ /* enable VMON and IMON */
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R003E_MEAS_EN,
+ MAX98927_MEAS_V_EN | MAX98927_MEAS_I_EN,
+ MAX98927_MEAS_V_EN | MAX98927_MEAS_I_EN);
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R00FF_GLOBAL_SHDN,
+ MAX98927_GLOBAL_EN_MASK, 1);
+ break;
+ case SND_SOC_DAPM_POST_PMD:
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R00FF_GLOBAL_SHDN,
+ MAX98927_GLOBAL_EN_MASK, 0);
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R003A_AMP_EN,
+ MAX98927_AMP_EN_MASK, 0);
+ /* disable VMON and IMON */
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R003E_MEAS_EN,
+ MAX98927_MEAS_V_EN | MAX98927_MEAS_I_EN, 0);
+ break;
+ default:
+ return 0;
+ }
+ return 0;
+}
+
+static const char * const max98927_switch_text[] = {
+ "Left", "Right", "LeftRight"};
+
+static const struct soc_enum dai_sel_enum =
+ SOC_ENUM_SINGLE(MAX98927_R0025_PCM_TO_SPK_MONOMIX_A,
+ MAX98927_PCM_TO_SPK_MONOMIX_CFG_SHIFT,
+ 3, max98927_switch_text);
+
+static const struct snd_kcontrol_new max98927_dai_controls =
+ SOC_DAPM_ENUM("DAI Sel", dai_sel_enum);
+
+static const struct snd_soc_dapm_widget max98927_dapm_widgets[] = {
+ SND_SOC_DAPM_AIF_IN("DAI_OUT", "HiFi Playback", 0, SND_SOC_NOPM, 0, 0),
+ SND_SOC_DAPM_DAC_E("Amp Enable", "HiFi Playback", MAX98927_R003A_AMP_EN,
+ 0, 0, max98927_dac_event,
+ SND_SOC_DAPM_POST_PMU | SND_SOC_DAPM_POST_PMD),
+ SND_SOC_DAPM_MUX("DAI Sel Mux", SND_SOC_NOPM, 0, 0,
+ &max98927_dai_controls),
+ SND_SOC_DAPM_OUTPUT("BE_OUT"),
+};
+
+static DECLARE_TLV_DB_SCALE(max98927_spk_tlv, 300, 300, 0);
+static DECLARE_TLV_DB_SCALE(max98927_digital_tlv, -1600, 25, 0);
+
+static int max98927_spk_gain_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct snd_soc_codec *codec = snd_soc_kcontrol_codec(kcontrol);
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+
+ ucontrol->value.integer.value[0] = max98927->spk_gain;
+ dev_dbg(codec->dev, "%s: spk_gain setting returned %d\n", __func__,
+ (int) ucontrol->value.integer.value[0]);
+ return 0;
+}
+
+static int max98927_spk_gain_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct snd_soc_codec *codec = snd_soc_kcontrol_codec(kcontrol);
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+ unsigned int sel = ucontrol->value.integer.value[0];
+
+ /* 0x7 is reserved */
+ if (sel > 6)
+ return -EINVAL;
+
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R003C_SPK_GAIN,
+ MAX98927_SPK_PCM_GAIN_MASK, sel);
+ max98927->spk_gain = sel;
+ return 0;
+}
+
+static int max98927_digital_gain_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct snd_soc_codec *codec = snd_soc_kcontrol_codec(kcontrol);
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+
+ ucontrol->value.integer.value[0] = max98927->digital_gain;
+ dev_dbg(codec->dev, "%s: digital_gain setting returned %d\n", __func__,
+ (int) ucontrol->value.integer.value[0]);
+ return 0;
+}
+
+static int max98927_digital_gain_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct snd_soc_codec *codec = snd_soc_kcontrol_codec(kcontrol);
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+ unsigned int sel = ucontrol->value.integer.value[0];
+
+ if (sel < ((1 << MAX98927_AMP_VOL_WIDTH) - 1)) {
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R0036_AMP_VOL_CTRL,
+ MAX98927_AMP_VOL_MASK, sel);
+ max98927->digital_gain = sel;
+ }
+ return 0;
+}
+
+static int max98927_boost_voltage_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_get(kcontrol, ucontrol,
+ MAX98927_R0040_BOOST_CTRL0,
+ MAX98927_BOOST_CTRL0_VOUT_MASK, 0);
+}
+
+static int max98927_boost_voltage_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_put(kcontrol, ucontrol,
+ MAX98927_R0040_BOOST_CTRL0,
+ MAX98927_BOOST_CTRL0_VOUT_MASK, 0);
+}
+
+static int max98927_amp_vol_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_get(kcontrol, ucontrol,
+ MAX98927_R0036_AMP_VOL_CTRL,
+ MAX98927_AMP_VOL_SEL,
+ MAX98927_AMP_VOL_SHIFT);
+}
+
+static int max98927_amp_vol_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_put(kcontrol, ucontrol,
+ MAX98927_R0036_AMP_VOL_CTRL,
+ MAX98927_AMP_VOL_SEL,
+ MAX98927_AMP_VOL_SHIFT);
+}
+
+static int max98927_amp_dsp_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_put(kcontrol, ucontrol,
+ MAX98927_R0052_BROWNOUT_EN,
+ MAX98927_BROWNOUT_DSP_EN,
+ MAX98927_BROWNOUT_DSP_SHIFT);
+}
+
+static int max98927_amp_dsp_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_get(kcontrol, ucontrol,
+ MAX98927_R0052_BROWNOUT_EN,
+ MAX98927_BROWNOUT_DSP_EN,
+ MAX98927_BROWNOUT_DSP_SHIFT);
+}
+
+static int max98927_ramp_switch_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_put(kcontrol, ucontrol,
+ MAX98927_R0037_AMP_DSP_CFG,
+ MAX98927_AMP_DSP_CFG_RMP_BYPASS,
+ MAX98927_AMP_DSP_CFG_RMP_SHIFT);
+}
+static int max98927_ramp_switch_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_get(kcontrol, ucontrol,
+ MAX98927_R0037_AMP_DSP_CFG,
+ MAX98927_AMP_DSP_CFG_RMP_BYPASS,
+ MAX98927_AMP_DSP_CFG_RMP_SHIFT);
+}
+
+static int max98927_dre_en_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_put(kcontrol, ucontrol,
+ MAX98927_R0039_DRE_CTRL,
+ MAX98927_DRE_CTRL_DRE_EN, 0);
+}
+static int max98927_dre_en_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_get(kcontrol, ucontrol,
+ MAX98927_R0039_DRE_CTRL,
+ MAX98927_DRE_CTRL_DRE_EN, 0);
+}
+static int max98927_spk_src_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_get(kcontrol, ucontrol,
+ MAX98927_R003B_SPK_SRC_SEL,
+ MAX98927_SPK_SRC_MASK, 0);
+}
+
+static int max98927_spk_src_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_put(kcontrol, ucontrol,
+ MAX98927_R003B_SPK_SRC_SEL,
+ MAX98927_SPK_SRC_MASK, 0);
+}
+
+static int max98927_mono_out_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_get(kcontrol, ucontrol,
+ MAX98927_R0025_PCM_TO_SPK_MONOMIX_A,
+ MAX98927_PCM_TO_SPK_MONOMIX_CFG_MASK,
+ MAX98927_PCM_TO_SPK_MONOMIX_CFG_SHIFT);
+}
+
+static int max98927_mono_out_put(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ return max98927_reg_put(kcontrol, ucontrol,
+ MAX98927_R0025_PCM_TO_SPK_MONOMIX_A,
+ MAX98927_PCM_TO_SPK_MONOMIX_CFG_MASK,
+ MAX98927_PCM_TO_SPK_MONOMIX_CFG_SHIFT);
+}
+
+static bool max98927_readable_register(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case MAX98927_R0001_INT_RAW1 ... MAX98927_R0028_ICC_RX_EN_B:
+ case MAX98927_R002B_ICC_TX_EN_A ... MAX98927_R002C_ICC_TX_EN_B:
+ case MAX98927_R002E_ICC_HIZ_MANUAL_MODE
+ ... MAX98927_R004E_MEAS_ADC_CH2_READ:
+ case MAX98927_R0051_BROWNOUT_STATUS
+ ... MAX98927_R0055_BROWNOUT_LVL_HOLD:
+ case MAX98927_R005A_BROWNOUT_LVL1_THRESH
+ ... MAX98927_R0061_BROWNOUT_AMP1_CLIP_MODE:
+ case MAX98927_R0072_BROWNOUT_LVL1_CUR_LIMIT
+ ... MAX98927_R0087_ENV_TRACK_BOOST_VOUT_READ:
+ case MAX98927_R00FF_GLOBAL_SHDN:
+ case MAX98927_R0100_SOFT_RESET:
+ case MAX98927_R01FF_REV_ID:
+ return true;
+ }
+ return false;
+};
+
+static bool max98927_volatile_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case MAX98927_R0001_INT_RAW1 ... MAX98927_R0009_INT_FLAG3:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static const char * const max98927_boost_voltage_text[] = {
+ "6.5V", "6.625V", "6.75V", "6.875V", "7V", "7.125V", "7.25V", "7.375V",
+ "7.5V", "7.625V", "7.75V", "7.875V", "8V", "8.125V", "8.25V", "8.375V",
+ "8.5V", "8.625V", "8.75V", "8.875V", "9V", "9.125V", "9.25V", "9.375V",
+ "9.5V", "9.625V", "9.75V", "9.875V", "10V"
+};
+
+static const char * const max98927_speaker_source_text[] = {
+ "i2s", "reserved", "tone", "pdm"
+};
+
+static const char * const max98927_monomix_output_text[] = {
+ "ch_0", "ch_1", "ch_1_2_div"
+};
+
+static const struct soc_enum max98927_enum[] = {
+ SOC_ENUM_SINGLE_EXT(ARRAY_SIZE(max98927_monomix_output_text),
+ max98927_monomix_output_text),
+ SOC_ENUM_SINGLE_EXT(ARRAY_SIZE(max98927_speaker_source_text),
+ max98927_speaker_source_text),
+ SOC_ENUM_SINGLE_EXT(ARRAY_SIZE(max98927_boost_voltage_text),
+ max98927_boost_voltage_text),
+};
+
+static const struct snd_kcontrol_new max98927_snd_controls[] = {
+ SOC_SINGLE_EXT_TLV("Speaker Volume",
+ MAX98927_R003C_SPK_GAIN,
+ 0, 6, 0,
+ max98927_spk_gain_get, max98927_spk_gain_put,
+ max98927_spk_tlv),
+ SOC_SINGLE_EXT_TLV("Digital Gain",
+ MAX98927_R0036_AMP_VOL_CTRL,
+ 0, (1<<MAX98927_AMP_VOL_WIDTH)-1, 0,
+ max98927_digital_gain_get, max98927_digital_gain_put,
+ max98927_digital_tlv),
+ SOC_SINGLE_EXT("Amp DSP Enable",
+ MAX98927_R0052_BROWNOUT_EN,
+ MAX98927_BROWNOUT_DSP_SHIFT, 1, 0,
+ max98927_amp_dsp_get, max98927_amp_dsp_put),
+ SOC_SINGLE_EXT("Ramp Switch",
+ MAX98927_R0037_AMP_DSP_CFG,
+ MAX98927_AMP_DSP_CFG_RMP_SHIFT, 1, 1,
+ max98927_ramp_switch_get, max98927_ramp_switch_put),
+ SOC_SINGLE_EXT("DRE EN",
+ MAX98927_R0039_DRE_CTRL,
+ MAX98927_DRE_EN_SHIFT, 1, 0,
+ max98927_dre_en_get, max98927_dre_en_put),
+ SOC_SINGLE_EXT("Amp Volume Location",
+ MAX98927_R0036_AMP_VOL_CTRL,
+ MAX98927_AMP_VOL_SEL_SHIFT, 1, 0,
+ max98927_amp_vol_get, max98927_amp_vol_put),
+ SOC_ENUM_EXT("Boost Output Voltage", max98927_enum[2],
+ max98927_boost_voltage_get, max98927_boost_voltage_put),
+ SOC_ENUM_EXT("Speaker Source", max98927_enum[1],
+ max98927_spk_src_get, max98927_spk_src_put),
+ SOC_ENUM_EXT("Monomix Output", max98927_enum[0],
+ max98927_mono_out_get, max98927_mono_out_put),
+};
+
+static const struct snd_soc_dapm_route max98927_audio_map[] = {
+ {"Amp Enable", NULL, "DAI_OUT"},
+ {"DAI Sel Mux", "Left", "Amp Enable"},
+ {"DAI Sel Mux", "Right", "Amp Enable"},
+ {"DAI Sel Mux", "LeftRight", "Amp Enable"},
+ {"BE_OUT", NULL, "DAI Sel Mux"},
+};
+
+static struct snd_soc_dai_driver max98927_dai[] = {
+ {
+ .name = "max98927-aif1",
+ .playback = {
+ .stream_name = "HiFi Playback",
+ .channels_min = 1,
+ .channels_max = 2,
+ .rates = MAX98927_RATES,
+ .formats = MAX98927_FORMATS,
+ },
+ .capture = {
+ .stream_name = "HiFi Capture",
+ .channels_min = 1,
+ .channels_max = 2,
+ .rates = MAX98927_RATES,
+ .formats = MAX98927_FORMATS,
+ },
+ .ops = &max98927_dai_ops,
+ }
+};
+
+static int max98927_probe(struct snd_soc_codec *codec)
+{
+ struct max98927_priv *max98927 = snd_soc_codec_get_drvdata(codec);
+ int ret = 0, reg = 0;
+
+ max98927->codec = codec;
+ codec->control_data = max98927->regmap;
+ codec->cache_bypass = 1;
+
+ /* Software Reset */
+ regmap_write(max98927->regmap,
+ MAX98927_R0100_SOFT_RESET, MAX98927_SOFT_RESET);
+
+ /* Check Revision ID */
+ ret = regmap_read(max98927->regmap,
+ MAX98927_R01FF_REV_ID, ®);
+ if (ret < 0)
+ dev_err(codec->dev,
+ "Failed to read: 0x%02X\n", MAX98927_R01FF_REV_ID);
+ else
+ dev_info(codec->dev,
+ "MAX98927 revisionID: 0x%02X\n", reg);
+
+ /* IV default slot configuration */
+ regmap_write(max98927->regmap,
+ MAX98927_R001C_PCM_TX_HIZ_CTRL_A,
+ 0xFF);
+ regmap_write(max98927->regmap,
+ MAX98927_R001D_PCM_TX_HIZ_CTRL_B,
+ 0xFF);
+ regmap_write(max98927->regmap,
+ MAX98927_R0025_PCM_TO_SPK_MONOMIX_A,
+ 0x80);
+ regmap_write(max98927->regmap,
+ MAX98927_R0026_PCM_TO_SPK_MONOMIX_B,
+ 0x1);
+ /* Set inital volume (+13dB) */
+ regmap_write(max98927->regmap,
+ MAX98927_R0036_AMP_VOL_CTRL,
+ 0x38);
+ regmap_write(max98927->regmap,
+ MAX98927_R003C_SPK_GAIN,
+ 0x05);
+ /* Enable DC blocker */
+ regmap_write(max98927->regmap,
+ MAX98927_R0037_AMP_DSP_CFG,
+ 0x03);
+ /* Enable IMON VMON DC blocker */
+ regmap_write(max98927->regmap,
+ MAX98927_R003F_MEAS_DSP_CFG,
+ 0xF7);
+ /* Boost Output Voltage & Current limit */
+ regmap_write(max98927->regmap,
+ MAX98927_R0040_BOOST_CTRL0,
+ 0x1C);
+ regmap_write(max98927->regmap,
+ MAX98927_R0042_BOOST_CTRL1,
+ 0x3E);
+ /* Measurement ADC config */
+ regmap_write(max98927->regmap,
+ MAX98927_R0043_MEAS_ADC_CFG,
+ 0x04);
+ regmap_write(max98927->regmap,
+ MAX98927_R0044_MEAS_ADC_BASE_MSB,
+ 0x00);
+ regmap_write(max98927->regmap,
+ MAX98927_R0045_MEAS_ADC_BASE_LSB,
+ 0x24);
+ /* Brownout Level */
+ regmap_write(max98927->regmap,
+ MAX98927_R007F_BROWNOUT_LVL4_AMP1_CTRL1,
+ 0x06);
+ /* Envelope Tracking configuration */
+ regmap_write(max98927->regmap,
+ MAX98927_R0082_ENV_TRACK_VOUT_HEADROOM,
+ 0x08);
+ regmap_write(max98927->regmap,
+ MAX98927_R0086_ENV_TRACK_CTRL,
+ 0x01);
+ regmap_write(max98927->regmap,
+ MAX98927_R0087_ENV_TRACK_BOOST_VOUT_READ,
+ 0x10);
+
+ /* voltage, current slot configuration */
+ regmap_write(max98927->regmap,
+ MAX98927_R001E_PCM_TX_CH_SRC_A,
+ (max98927->i_l_slot<<MAX98927_PCM_TX_CH_SRC_A_I_SHIFT|
+ max98927->v_l_slot)&0xFF);
+
+ if (max98927->v_l_slot < 8) {
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R001C_PCM_TX_HIZ_CTRL_A,
+ 1 << max98927->v_l_slot, 0);
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R001A_PCM_TX_EN_A,
+ 1 << max98927->v_l_slot,
+ 1 << max98927->v_l_slot);
+ } else {
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R001D_PCM_TX_HIZ_CTRL_B,
+ 1 << (max98927->v_l_slot - 8), 0);
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R001B_PCM_TX_EN_B,
+ 1 << (max98927->v_l_slot - 8),
+ 1 << (max98927->v_l_slot - 8));
+ }
+
+ if (max98927->i_l_slot < 8) {
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R001C_PCM_TX_HIZ_CTRL_A,
+ 1 << max98927->i_l_slot, 0);
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R001A_PCM_TX_EN_A,
+ 1 << max98927->i_l_slot,
+ 1 << max98927->i_l_slot);
+ } else {
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R001D_PCM_TX_HIZ_CTRL_B,
+ 1 << (max98927->i_l_slot - 8), 0);
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R001B_PCM_TX_EN_B,
+ 1 << (max98927->i_l_slot - 8),
+ 1 << (max98927->i_l_slot - 8));
+ }
+
+ /* Set interleave mode */
+ if (max98927->interleave_mode)
+ regmap_update_bits(max98927->regmap,
+ MAX98927_R001F_PCM_TX_CH_SRC_B,
+ MAX98927_PCM_TX_CH_INTERLEAVE_MASK,
+ MAX98927_PCM_TX_CH_INTERLEAVE_MASK);
+ return ret;
+}
+
+static const struct snd_soc_codec_driver soc_codec_dev_max98927 = {
+ .probe = max98927_probe,
+ .component_driver = {
+ .controls = max98927_snd_controls,
+ .num_controls = ARRAY_SIZE(max98927_snd_controls),
+ .dapm_widgets = max98927_dapm_widgets,
+ .num_dapm_widgets = ARRAY_SIZE(max98927_dapm_widgets),
+ .dapm_routes = max98927_audio_map,
+ .num_dapm_routes = ARRAY_SIZE(max98927_audio_map),
+ },
+};
+
+static const struct regmap_config max98927_regmap = {
+ .reg_bits = 16,
+ .val_bits = 8,
+ .max_register = MAX98927_R01FF_REV_ID,
+ .reg_defaults = max98927_reg,
+ .num_reg_defaults = ARRAY_SIZE(max98927_reg),
+ .readable_reg = max98927_readable_register,
+ .volatile_reg = max98927_volatile_reg,
+ .cache_type = REGCACHE_RBTREE,
+};
+
+static void max98927_slot_config(struct i2c_client *i2c,
+ struct max98927_priv *max98927)
+{
+ int value;
+
+ if (!of_property_read_u32(i2c->dev.of_node,
+ "maxim,vmon-slot-no", &value))
+ max98927->v_l_slot = value & 0xF;
+ else
+ max98927->v_l_slot = 0;
+ if (!of_property_read_u32(i2c->dev.of_node,
+ "maxim,imon-slot-no", &value))
+ max98927->i_l_slot = value & 0xF;
+ else
+ max98927->i_l_slot = 1;
+}
+
+static int max98927_i2c_probe(struct i2c_client *i2c,
+ const struct i2c_device_id *id)
+{
+
+ int ret = 0, value;
+ struct max98927_priv *max98927 = NULL;
+
+ max98927 = devm_kzalloc(&i2c->dev,
+ sizeof(*max98927), GFP_KERNEL);
+
+ if (!max98927) {
+ ret = -ENOMEM;
+ goto err;
+ }
+ i2c_set_clientdata(i2c, max98927);
+
+ max98927->reset_gpio = devm_gpiod_get_optional(&i2c->dev, "reset",
+ GPIOD_OUT_LOW);
+ if (IS_ERR(max98927->reset_gpio)) {
+ ret = PTR_ERR(max98927->reset_gpio);
+ max98927->reset_gpio = NULL;
+ if (ret == -EBUSY) {
+ dev_info(&i2c->dev,
+ "Reset line busy, assuming shared reset\n");
+ } else {
+ dev_err(&i2c->dev, "Failed to get reset GPIO: %d\n", ret);
+ goto err;
+ }
+ }
+
+ if (max98927->reset_gpio) {
+ gpiod_set_value_cansleep(max98927->reset_gpio, 1);
+ usleep_range(1000, 2100);
+ gpiod_set_value_cansleep(max98927->reset_gpio, 0);
+ usleep_range(10, 20);
+ gpiod_set_value_cansleep(max98927->reset_gpio, 1);
+ }
+
+ /* update interleave mode info */
+ if (!of_property_read_u32(i2c->dev.of_node,
+ "maxim,interleave_mode", &value)) {
+ if (value > 0)
+ max98927->interleave_mode = 1;
+ else
+ max98927->interleave_mode = 0;
+ } else
+ max98927->interleave_mode = 0;
+
+ /* regmap initialization */
+ max98927->regmap
+ = devm_regmap_init_i2c(i2c, &max98927_regmap);
+ if (IS_ERR(max98927->regmap)) {
+ ret = PTR_ERR(max98927->regmap);
+ dev_err(&i2c->dev,
+ "Failed to allocate regmap: %d\n", ret);
+ goto err;
+ }
+
+ /* voltage/current slot configuration */
+ max98927_slot_config(i2c, max98927);
+
+ /* codec registeration */
+ ret = snd_soc_register_codec(&i2c->dev, &soc_codec_dev_max98927,
+ max98927_dai, ARRAY_SIZE(max98927_dai));
+ if (ret < 0)
+ dev_err(&i2c->dev, "Failed to register codec: %d\n", ret);
+
+ return ret;
+
+err:
+ if (max98927)
+ devm_kfree(&i2c->dev, max98927);
+ return ret;
+}
+
+static int max98927_i2c_remove(struct i2c_client *client)
+{
+ snd_soc_unregister_codec(&client->dev);
+ return 0;
+}
+
+static const struct i2c_device_id max98927_i2c_id[] = {
+ { "max98927", 0},
+ { },
+};
+
+MODULE_DEVICE_TABLE(i2c, max98927_i2c_id);
+
+#if defined(CONFIG_OF)
+static const struct of_device_id max98927_of_match[] = {
+ { .compatible = "maxim,max98927", },
+ { }
+};
+MODULE_DEVICE_TABLE(of, max98927_of_match);
+#endif
+
+#ifdef CONFIG_ACPI
+static const struct acpi_device_id max98927_acpi_match[] = {
+ { "MX98927", 0 },
+ {},
+};
+MODULE_DEVICE_TABLE(acpi, max98927_acpi_match);
+#endif
+
+static struct i2c_driver max98927_i2c_driver = {
+ .driver = {
+ .name = "max98927",
+ .of_match_table = of_match_ptr(max98927_of_match),
+ .acpi_match_table = ACPI_PTR(max98927_acpi_match),
+ .pm = NULL,
+ },
+ .probe = max98927_i2c_probe,
+ .remove = max98927_i2c_remove,
+ .id_table = max98927_i2c_id,
+};
+
+module_i2c_driver(max98927_i2c_driver)
+
+MODULE_DESCRIPTION("ALSA SoC MAX98927 driver");
+MODULE_AUTHOR("Ryan Lee <ryans.lee@maximintegrated.com>");
+MODULE_LICENSE("GPL");
diff --git a/sound/soc/codecs/max98927.h b/sound/soc/codecs/max98927.h
new file mode 100644
index 0000000..4dee02a
--- /dev/null
+++ b/sound/soc/codecs/max98927.h
@@ -0,0 +1,268 @@
+/*
+ * max98927.h -- MAX98927 ALSA Soc Audio driver
+ *
+ * Copyright 2013-15 Maxim Integrated Products
+ * Author: Ryan Lee <ryans.lee@maximintegrated.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation; either version 2 of the License, or (at your
+ * option) any later version.
+ *
+ */
+#ifndef _MAX98927_H
+#define _MAX98927_H
+
+/* Register Values */
+#define MAX98927_R0001_INT_RAW1 0x0001
+#define MAX98927_R0002_INT_RAW2 0x0002
+#define MAX98927_R0003_INT_RAW3 0x0003
+#define MAX98927_R0004_INT_STATE1 0x0004
+#define MAX98927_R0005_INT_STATE2 0x0005
+#define MAX98927_R0006_INT_STATE3 0x0006
+#define MAX98927_R0007_INT_FLAG1 0x0007
+#define MAX98927_R0008_INT_FLAG2 0x0008
+#define MAX98927_R0009_INT_FLAG3 0x0009
+#define MAX98927_R000A_INT_EN1 0x000A
+#define MAX98927_R000B_INT_EN2 0x000B
+#define MAX98927_R000C_INT_EN3 0x000C
+#define MAX98927_R000D_INT_FLAG_CLR1 0x000D
+#define MAX98927_R000E_INT_FLAG_CLR2 0x000E
+#define MAX98927_R000F_INT_FLAG_CLR3 0x000F
+#define MAX98927_R0010_IRQ_CTRL 0x0010
+#define MAX98927_R0011_CLK_MON 0x0011
+#define MAX98927_R0012_WDOG_CTRL 0x0012
+#define MAX98927_R0013_WDOG_RST 0x0013
+#define MAX98927_R0014_MEAS_ADC_THERM_WARN_THRESH 0x0014
+#define MAX98927_R0015_MEAS_ADC_THERM_SHDN_THRESH 0x0015
+#define MAX98927_R0016_MEAS_ADC_THERM_HYSTERESIS 0x0016
+#define MAX98927_R0017_PIN_CFG 0x0017
+#define MAX98927_R0018_PCM_RX_EN_A 0x0018
+#define MAX98927_R0019_PCM_RX_EN_B 0x0019
+#define MAX98927_R001A_PCM_TX_EN_A 0x001A
+#define MAX98927_R001B_PCM_TX_EN_B 0x001B
+#define MAX98927_R001C_PCM_TX_HIZ_CTRL_A 0x001C
+#define MAX98927_R001D_PCM_TX_HIZ_CTRL_B 0x001D
+#define MAX98927_R001E_PCM_TX_CH_SRC_A 0x001E
+#define MAX98927_R001F_PCM_TX_CH_SRC_B 0x001F
+#define MAX98927_R0020_PCM_MODE_CFG 0x0020
+#define MAX98927_R0021_PCM_MASTER_MODE 0x0021
+#define MAX98927_R0022_PCM_CLK_SETUP 0x0022
+#define MAX98927_R0023_PCM_SR_SETUP1 0x0023
+#define MAX98927_R0024_PCM_SR_SETUP2 0x0024
+#define MAX98927_R0025_PCM_TO_SPK_MONOMIX_A 0x0025
+#define MAX98927_R0026_PCM_TO_SPK_MONOMIX_B 0x0026
+#define MAX98927_R0027_ICC_RX_EN_A 0x0027
+#define MAX98927_R0028_ICC_RX_EN_B 0x0028
+#define MAX98927_R002B_ICC_TX_EN_A 0x002B
+#define MAX98927_R002C_ICC_TX_EN_B 0x002C
+#define MAX98927_R002E_ICC_HIZ_MANUAL_MODE 0x002E
+#define MAX98927_R002F_ICC_TX_HIZ_EN_A 0x002F
+#define MAX98927_R0030_ICC_TX_HIZ_EN_B 0x0030
+#define MAX98927_R0031_ICC_LNK_EN 0x0031
+#define MAX98927_R0032_PDM_TX_EN 0x0032
+#define MAX98927_R0033_PDM_TX_HIZ_CTRL 0x0033
+#define MAX98927_R0034_PDM_TX_CTRL 0x0034
+#define MAX98927_R0035_PDM_RX_CTRL 0x0035
+#define MAX98927_R0036_AMP_VOL_CTRL 0x0036
+#define MAX98927_R0037_AMP_DSP_CFG 0x0037
+#define MAX98927_R0038_TONE_GEN_DC_CFG 0x0038
+#define MAX98927_R0039_DRE_CTRL 0x0039
+#define MAX98927_R003A_AMP_EN 0x003A
+#define MAX98927_R003B_SPK_SRC_SEL 0x003B
+#define MAX98927_R003C_SPK_GAIN 0x003C
+#define MAX98927_R003D_SSM_CFG 0x003D
+#define MAX98927_R003E_MEAS_EN 0x003E
+#define MAX98927_R003F_MEAS_DSP_CFG 0x003F
+#define MAX98927_R0040_BOOST_CTRL0 0x0040
+#define MAX98927_R0041_BOOST_CTRL3 0x0041
+#define MAX98927_R0042_BOOST_CTRL1 0x0042
+#define MAX98927_R0043_MEAS_ADC_CFG 0x0043
+#define MAX98927_R0044_MEAS_ADC_BASE_MSB 0x0044
+#define MAX98927_R0045_MEAS_ADC_BASE_LSB 0x0045
+#define MAX98927_R0046_ADC_CH0_DIVIDE 0x0046
+#define MAX98927_R0047_ADC_CH1_DIVIDE 0x0047
+#define MAX98927_R0048_ADC_CH2_DIVIDE 0x0048
+#define MAX98927_R0049_ADC_CH0_FILT_CFG 0x0049
+#define MAX98927_R004A_ADC_CH1_FILT_CFG 0x004A
+#define MAX98927_R004B_ADC_CH2_FILT_CFG 0x004B
+#define MAX98927_R004C_MEAS_ADC_CH0_READ 0x004C
+#define MAX98927_R004D_MEAS_ADC_CH1_READ 0x004D
+#define MAX98927_R004E_MEAS_ADC_CH2_READ 0x004E
+#define MAX98927_R0051_BROWNOUT_STATUS 0x0051
+#define MAX98927_R0052_BROWNOUT_EN 0x0052
+#define MAX98927_R0053_BROWNOUT_INFINITE_HOLD 0x0053
+#define MAX98927_R0054_BROWNOUT_INFINITE_HOLD_CLR 0x0054
+#define MAX98927_R0055_BROWNOUT_LVL_HOLD 0x0055
+#define MAX98927_R005A_BROWNOUT_LVL1_THRESH 0x005A
+#define MAX98927_R005B_BROWNOUT_LVL2_THRESH 0x005B
+#define MAX98927_R005C_BROWNOUT_LVL3_THRESH 0x005C
+#define MAX98927_R005D_BROWNOUT_LVL4_THRESH 0x005D
+#define MAX98927_R005E_BROWNOUT_THRESH_HYSTERYSIS 0x005E
+#define MAX98927_R005F_BROWNOUT_AMP_LIMITER_ATK_REL 0x005F
+#define MAX98927_R0060_BROWNOUT_AMP_GAIN_ATK_REL 0x0060
+#define MAX98927_R0061_BROWNOUT_AMP1_CLIP_MODE 0x0061
+#define MAX98927_R0072_BROWNOUT_LVL1_CUR_LIMIT 0x0072
+#define MAX98927_R0073_BROWNOUT_LVL1_AMP1_CTRL1 0x0073
+#define MAX98927_R0074_BROWNOUT_LVL1_AMP1_CTRL2 0x0074
+#define MAX98927_R0075_BROWNOUT_LVL1_AMP1_CTRL3 0x0075
+#define MAX98927_R0076_BROWNOUT_LVL2_CUR_LIMIT 0x0076
+#define MAX98927_R0077_BROWNOUT_LVL2_AMP1_CTRL1 0x0077
+#define MAX98927_R0078_BROWNOUT_LVL2_AMP1_CTRL2 0x0078
+#define MAX98927_R0079_BROWNOUT_LVL2_AMP1_CTRL3 0x0079
+#define MAX98927_R007A_BROWNOUT_LVL3_CUR_LIMIT 0x007A
+#define MAX98927_R007B_BROWNOUT_LVL3_AMP1_CTRL1 0x007B
+#define MAX98927_R007C_BROWNOUT_LVL3_AMP1_CTRL2 0x007C
+#define MAX98927_R007D_BROWNOUT_LVL3_AMP1_CTRL3 0x007D
+#define MAX98927_R007E_BROWNOUT_LVL4_CUR_LIMIT 0x007E
+#define MAX98927_R007F_BROWNOUT_LVL4_AMP1_CTRL1 0x007F
+#define MAX98927_R0080_BROWNOUT_LVL4_AMP1_CTRL2 0x0080
+#define MAX98927_R0081_BROWNOUT_LVL4_AMP1_CTRL3 0x0081
+#define MAX98927_R0082_ENV_TRACK_VOUT_HEADROOM 0x0082
+#define MAX98927_R0083_ENV_TRACK_BOOST_VOUT_DELAY 0x0083
+#define MAX98927_R0084_ENV_TRACK_REL_RATE 0x0084
+#define MAX98927_R0085_ENV_TRACK_HOLD_RATE 0x0085
+#define MAX98927_R0086_ENV_TRACK_CTRL 0x0086
+#define MAX98927_R0087_ENV_TRACK_BOOST_VOUT_READ 0x0087
+#define MAX98927_R00FF_GLOBAL_SHDN 0x00FF
+#define MAX98927_R0100_SOFT_RESET 0x0100
+#define MAX98927_R01FF_REV_ID 0x01FF
+
+/* MAX98927_R0018_PCM_RX_EN_A */
+#define MAX98927_PCM_RX_CH0_EN (0x1 << 0)
+#define MAX98927_PCM_RX_CH1_EN (0x1 << 1)
+#define MAX98927_PCM_RX_CH2_EN (0x1 << 2)
+#define MAX98927_PCM_RX_CH3_EN (0x1 << 3)
+#define MAX98927_PCM_RX_CH4_EN (0x1 << 4)
+#define MAX98927_PCM_RX_CH5_EN (0x1 << 5)
+#define MAX98927_PCM_RX_CH6_EN (0x1 << 6)
+#define MAX98927_PCM_RX_CH7_EN (0x1 << 7)
+
+/* MAX98927_R001A_PCM_TX_EN_A */
+#define MAX98927_PCM_TX_CH0_EN (0x1 << 0)
+#define MAX98927_PCM_TX_CH1_EN (0x1 << 1)
+#define MAX98927_PCM_TX_CH2_EN (0x1 << 2)
+#define MAX98927_PCM_TX_CH3_EN (0x1 << 3)
+#define MAX98927_PCM_TX_CH4_EN (0x1 << 4)
+#define MAX98927_PCM_TX_CH5_EN (0x1 << 5)
+#define MAX98927_PCM_TX_CH6_EN (0x1 << 6)
+#define MAX98927_PCM_TX_CH7_EN (0x1 << 7)
+
+/* MAX98927_R001E_PCM_TX_CH_SRC_A */
+#define MAX98927_PCM_TX_CH_SRC_A_V_SHIFT (0)
+#define MAX98927_PCM_TX_CH_SRC_A_I_SHIFT (4)
+
+/* MAX98927_R001F_PCM_TX_CH_SRC_B */
+#define MAX98927_PCM_TX_CH_INTERLEAVE_MASK (0x1 << 5)
+
+/* MAX98927_R0020_PCM_MODE_CFG */
+#define MAX98927_PCM_MODE_CFG_PCM_BCLKEDGE (0x1 << 2)
+#define MAX98927_PCM_MODE_CFG_FORMAT_MASK (0x7 << 3)
+
+#define MAX98927_PCM_MODE_CFG_CHANSZ_MASK (0x3 << 6)
+#define MAX98927_PCM_MODE_CFG_CHANSZ_16 (0x1 << 6)
+#define MAX98927_PCM_MODE_CFG_CHANSZ_24 (0x2 << 6)
+#define MAX98927_PCM_MODE_CFG_CHANSZ_32 (0x3 << 6)
+
+/* MAX98927_R0021_PCM_MASTER_MODE */
+#define MAX98927_PCM_MASTER_MODE_MASK (0x3 << 0)
+#define MAX98927_PCM_MASTER_MODE_SLAVE (0x0 << 0)
+#define MAX98927_PCM_MASTER_MODE_MASTER (0x3 << 0)
+#define MAX98927_PCM_MASTER_MODE_HYBRID (0x1 << 0)
+
+#define MAX98927_PCM_MASTER_MODE_MCLK_MASK (0xF << 2)
+#define MAX98927_PCM_MASTER_MODE_MCLK_RATE_SHIFT (2)
+
+/* MAX98927_R0022_PCM_CLK_SETUP */
+#define MAX98927_PCM_CLK_SETUP_BSEL_MASK (0xF << 0)
+
+/* MAX98927_R0023_PCM_SR_SETUP1 */
+#define MAX98927_PCM_SR_SET1_SR_MASK (0xF << 0)
+
+#define MAX98927_PCM_SR_SET1_SR_8000 (0x0 << 0)
+#define MAX98927_PCM_SR_SET1_SR_11025 (0x1 << 0)
+#define MAX98927_PCM_SR_SET1_SR_12000 (0x2 << 0)
+#define MAX98927_PCM_SR_SET1_SR_16000 (0x3 << 0)
+#define MAX98927_PCM_SR_SET1_SR_22050 (0x4 << 0)
+#define MAX98927_PCM_SR_SET1_SR_24000 (0x5 << 0)
+#define MAX98927_PCM_SR_SET1_SR_32000 (0x6 << 0)
+#define MAX98927_PCM_SR_SET1_SR_44100 (0x7 << 0)
+#define MAX98927_PCM_SR_SET1_SR_48000 (0x8 << 0)
+
+/* MAX98927_R0024_PCM_SR_SETUP2 */
+#define MAX98927_PCM_SR_SET2_SR_MASK (0xF << 4)
+#define MAX98927_PCM_SR_SET2_SR_SHIFT (4)
+#define MAX98927_PCM_SR_SET2_IVADC_SR_MASK (0xf << 0)
+
+/* MAX98927_R0025_PCM_TO_SPK_MONOMIX_A */
+#define MAX98927_PCM_TO_SPK_MONOMIX_CFG_MASK (0x3 << 6)
+#define MAX98927_PCM_TO_SPK_MONOMIX_CFG_SHIFT (6)
+
+/* MAX98927_R0036_AMP_VOL_CTRL */
+#define MAX98927_AMP_VOL_SEL (0x1 << 7)
+#define MAX98927_AMP_VOL_SEL_WIDTH (1)
+#define MAX98927_AMP_VOL_SEL_SHIFT (7)
+#define MAX98927_AMP_VOL_MASK (0x7f << 0)
+#define MAX98927_AMP_VOL_WIDTH (7)
+#define MAX98927_AMP_VOL_SHIFT (0)
+
+/* MAX98927_R0037_AMP_DSP_CFG */
+#define MAX98927_AMP_DSP_CFG_DCBLK_EN (0x1 << 0)
+#define MAX98927_AMP_DSP_CFG_DITH_EN (0x1 << 1)
+#define MAX98927_AMP_DSP_CFG_RMP_BYPASS (0x1 << 4)
+#define MAX98927_AMP_DSP_CFG_DAC_INV (0x1 << 5)
+#define MAX98927_AMP_DSP_CFG_RMP_SHIFT (4)
+
+/* MAX98927_R0039_DRE_CTRL */
+#define MAX98927_DRE_CTRL_DRE_EN (0x1 << 0)
+#define MAX98927_DRE_EN_SHIFT 0x1
+
+/* MAX98927_R003A_AMP_EN */
+#define MAX98927_AMP_EN_MASK (0x1 << 0)
+
+/* MAX98927_R003B_SPK_SRC_SEL */
+#define MAX98927_SPK_SRC_MASK (0x3 << 0)
+
+/* MAX98927_R003C_SPK_GAIN */
+#define MAX98927_SPK_PCM_GAIN_MASK (0x7 << 0)
+#define MAX98927_SPK_PDM_GAIN_MASK (0x7 << 4)
+#define MAX98927_SPK_GAIN_WIDTH (3)
+
+/* MAX98927_R003E_MEAS_EN */
+#define MAX98927_MEAS_V_EN (0x1 << 0)
+#define MAX98927_MEAS_I_EN (0x1 << 1)
+
+/* MAX98927_R0040_BOOST_CTRL0 */
+#define MAX98927_BOOST_CTRL0_VOUT_MASK (0x1f << 0)
+#define MAX98927_BOOST_CTRL0_PVDD_MASK (0x1 << 7)
+#define MAX98927_BOOST_CTRL0_PVDD_EN_SHIFT (7)
+
+/* MAX98927_R0052_BROWNOUT_EN */
+#define MAX98927_BROWNOUT_BDE_EN (0x1 << 0)
+#define MAX98927_BROWNOUT_AMP_EN (0x1 << 1)
+#define MAX98927_BROWNOUT_DSP_EN (0x1 << 2)
+#define MAX98927_BROWNOUT_DSP_SHIFT (2)
+
+/* MAX98927_R0100_SOFT_RESET */
+#define MAX98927_SOFT_RESET (0x1 << 0)
+
+/* MAX98927_R00FF_GLOBAL_SHDN */
+#define MAX98927_GLOBAL_EN_MASK (0x1 << 0)
+
+struct max98927_priv {
+ struct regmap *regmap;
+ struct snd_soc_codec *codec;
+ struct max98927_pdata *pdata;
+ unsigned int spk_gain;
+ unsigned int sysclk;
+ unsigned int v_l_slot;
+ unsigned int i_l_slot;
+ bool interleave_mode;
+ unsigned int ch_size;
+ unsigned int rate;
+ unsigned int iface;
+ unsigned int master;
+ unsigned int digital_gain;
+ struct gpio_desc *reset_gpio;
+};
+#endif
diff --git a/sound/soc/codecs/sigmadsp.c b/sound/soc/codecs/sigmadsp.c
index d53680a..6df1586 100644
--- a/sound/soc/codecs/sigmadsp.c
+++ b/sound/soc/codecs/sigmadsp.c
@@ -117,8 +117,7 @@ static int sigmadsp_ctrl_write(struct sigmadsp *sigmadsp,
struct sigmadsp_control *ctrl, void *data)
{
/* safeload loads up to 20 bytes in a atomic operation */
- if (ctrl->num_bytes > 4 && ctrl->num_bytes <= 20 && sigmadsp->ops &&
- sigmadsp->ops->safeload)
+ if (ctrl->num_bytes <= 20 && sigmadsp->ops && sigmadsp->ops->safeload)
return sigmadsp->ops->safeload(sigmadsp, ctrl->addr, data,
ctrl->num_bytes);
else
diff --git a/sound/soc/codecs/wm8804-i2c.c b/sound/soc/codecs/wm8804-i2c.c
index f27464c..7954196 100644
--- a/sound/soc/codecs/wm8804-i2c.c
+++ b/sound/soc/codecs/wm8804-i2c.c
@@ -13,6 +13,7 @@
#include <linux/init.h>
#include <linux/module.h>
#include <linux/i2c.h>
+#include <linux/acpi.h>
#include "wm8804.h"
@@ -40,17 +41,29 @@ static const struct i2c_device_id wm8804_i2c_id[] = {
};
MODULE_DEVICE_TABLE(i2c, wm8804_i2c_id);
+#if defined(CONFIG_OF)
static const struct of_device_id wm8804_of_match[] = {
{ .compatible = "wlf,wm8804", },
{ }
};
MODULE_DEVICE_TABLE(of, wm8804_of_match);
+#endif
+
+#ifdef CONFIG_ACPI
+static const struct acpi_device_id wm8804_acpi_match[] = {
+ { "1AEC8804", 0 }, /* Wolfson PCI ID + part ID */
+ { "10138804", 0 }, /* Cirrus Logic PCI ID + part ID */
+ { },
+};
+MODULE_DEVICE_TABLE(acpi, wm8804_acpi_match);
+#endif
static struct i2c_driver wm8804_i2c_driver = {
.driver = {
.name = "wm8804",
.pm = &wm8804_pm,
- .of_match_table = wm8804_of_match,
+ .of_match_table = of_match_ptr(wm8804_of_match),
+ .acpi_match_table = ACPI_PTR(wm8804_acpi_match),
},
.probe = wm8804_i2c_probe,
.remove = wm8804_i2c_remove,
diff --git a/sound/soc/codecs/wm8994.c b/sound/soc/codecs/wm8994.c
index 3896523..f289762 100644
--- a/sound/soc/codecs/wm8994.c
+++ b/sound/soc/codecs/wm8994.c
@@ -2431,6 +2431,7 @@ static int wm8994_set_dai_sysclk(struct snd_soc_dai *dai,
snd_soc_update_bits(codec, WM8994_POWER_MANAGEMENT_2,
WM8994_OPCLK_ENA, 0);
}
+ break;
default:
return -EINVAL;
diff --git a/sound/soc/fsl/fsl_esai.c b/sound/soc/fsl/fsl_esai.c
index 38bfd46..3ef1745 100644
--- a/sound/soc/fsl/fsl_esai.c
+++ b/sound/soc/fsl/fsl_esai.c
@@ -145,6 +145,13 @@ static int fsl_esai_divisor_cal(struct snd_soc_dai *dai, bool tx, u32 ratio,
psr = ratio <= 256 * maxfp ? ESAI_xCCR_xPSR_BYPASS : ESAI_xCCR_xPSR_DIV8;
+ /* Do not loop-search if PM (1 ~ 256) alone can serve the ratio */
+ if (ratio <= 256) {
+ pm = ratio;
+ fp = 1;
+ goto out;
+ }
+
/* Set the max fluctuation -- 0.1% of the max devisor */
savesub = (psr ? 1 : 8) * 256 * maxfp / 1000;
diff --git a/sound/soc/intel/boards/cht_bsw_max98090_ti.c b/sound/soc/intel/boards/cht_bsw_max98090_ti.c
index cdcced9..b7c1e3d 100644
--- a/sound/soc/intel/boards/cht_bsw_max98090_ti.c
+++ b/sound/soc/intel/boards/cht_bsw_max98090_ti.c
@@ -128,23 +128,19 @@ static int cht_codec_init(struct snd_soc_pcm_runtime *runtime)
struct cht_mc_private *ctx = snd_soc_card_get_drvdata(runtime->card);
struct snd_soc_jack *jack = &ctx->jack;
- /**
- * TI supports 4 butons headset detection
- * KEY_MEDIA
- * KEY_VOICECOMMAND
- * KEY_VOLUMEUP
- * KEY_VOLUMEDOWN
- */
- if (ctx->ts3a227e_present)
- jack_type = SND_JACK_HEADPHONE | SND_JACK_MICROPHONE |
- SND_JACK_BTN_0 | SND_JACK_BTN_1 |
- SND_JACK_BTN_2 | SND_JACK_BTN_3;
- else
- jack_type = SND_JACK_HEADPHONE | SND_JACK_MICROPHONE;
+ if (ctx->ts3a227e_present) {
+ /*
+ * The jack has already been created in the
+ * cht_max98090_headset_init() function.
+ */
+ snd_soc_jack_notifier_register(jack, &cht_jack_nb);
+ return 0;
+ }
+
+ jack_type = SND_JACK_HEADPHONE | SND_JACK_MICROPHONE;
ret = snd_soc_card_jack_new(runtime->card, "Headset Jack",
jack_type, jack, NULL, 0);
-
if (ret) {
dev_err(runtime->dev, "Headset Jack creation failed %d\n", ret);
return ret;
@@ -200,6 +196,27 @@ static int cht_max98090_headset_init(struct snd_soc_component *component)
{
struct snd_soc_card *card = component->card;
struct cht_mc_private *ctx = snd_soc_card_get_drvdata(card);
+ struct snd_soc_jack *jack = &ctx->jack;
+ int jack_type;
+ int ret;
+
+ /*
+ * TI supports 4 butons headset detection
+ * KEY_MEDIA
+ * KEY_VOICECOMMAND
+ * KEY_VOLUMEUP
+ * KEY_VOLUMEDOWN
+ */
+ jack_type = SND_JACK_HEADPHONE | SND_JACK_MICROPHONE |
+ SND_JACK_BTN_0 | SND_JACK_BTN_1 |
+ SND_JACK_BTN_2 | SND_JACK_BTN_3;
+
+ ret = snd_soc_card_jack_new(card, "Headset Jack", jack_type,
+ jack, NULL, 0);
+ if (ret) {
+ dev_err(card->dev, "Headset Jack creation failed %d\n", ret);
+ return ret;
+ }
return ts3a227e_enable_jack_detect(component, &ctx->jack);
}
diff --git a/sound/soc/intel/common/sst-firmware.c b/sound/soc/intel/common/sst-firmware.c
index a086c35..79a9fdf94d 100644
--- a/sound/soc/intel/common/sst-firmware.c
+++ b/sound/soc/intel/common/sst-firmware.c
@@ -274,7 +274,6 @@ int sst_dma_new(struct sst_dsp *sst)
struct sst_pdata *sst_pdata = sst->pdata;
struct sst_dma *dma;
struct resource mem;
- const char *dma_dev_name;
int ret = 0;
if (sst->pdata->resindex_dma_base == -1)
@@ -285,7 +284,6 @@ int sst_dma_new(struct sst_dsp *sst)
* is attached to the ADSP IP. */
switch (sst->pdata->dma_engine) {
case SST_DMA_TYPE_DW:
- dma_dev_name = "dw_dmac";
break;
default:
dev_err(sst->dev, "error: invalid DMA engine %d\n",
diff --git a/sound/soc/intel/skylake/skl-topology.c b/sound/soc/intel/skylake/skl-topology.c
index bef8a45..76e5bb4 100644
--- a/sound/soc/intel/skylake/skl-topology.c
+++ b/sound/soc/intel/skylake/skl-topology.c
@@ -1780,6 +1780,7 @@ static int skl_tplg_get_token(struct device *dev,
case SKL_TKN_U8_CORE_ID:
mconfig->core_id = tkn_elem->value;
+ break;
case SKL_TKN_U8_MOD_TYPE:
mconfig->m_type = tkn_elem->value;
@@ -2325,7 +2326,7 @@ static int skl_tplg_get_manifest_tkn(struct device *dev,
if (ret < 0)
return ret;
- tkn_count += ret;
+ tkn_count = ret;
tuple_size += tkn_count *
sizeof(struct snd_soc_tplg_vendor_string_elem);
diff --git a/sound/soc/omap/omap-abe-twl6040.c b/sound/soc/omap/omap-abe-twl6040.c
index 89fe95e..07af300 100644
--- a/sound/soc/omap/omap-abe-twl6040.c
+++ b/sound/soc/omap/omap-abe-twl6040.c
@@ -36,6 +36,8 @@
#include "../codecs/twl6040.h"
struct abe_twl6040 {
+ struct snd_soc_card card;
+ struct snd_soc_dai_link dai_links[2];
int jack_detection; /* board can detect jack events */
int mclk_freq; /* MCLK frequency speed for twl6040 */
};
@@ -208,40 +210,10 @@ static int omap_abe_dmic_init(struct snd_soc_pcm_runtime *rtd)
ARRAY_SIZE(dmic_audio_map));
}
-/* Digital audio interface glue - connects codec <--> CPU */
-static struct snd_soc_dai_link abe_twl6040_dai_links[] = {
- {
- .name = "TWL6040",
- .stream_name = "TWL6040",
- .codec_dai_name = "twl6040-legacy",
- .codec_name = "twl6040-codec",
- .init = omap_abe_twl6040_init,
- .ops = &omap_abe_ops,
- },
- {
- .name = "DMIC",
- .stream_name = "DMIC Capture",
- .codec_dai_name = "dmic-hifi",
- .codec_name = "dmic-codec",
- .init = omap_abe_dmic_init,
- .ops = &omap_abe_dmic_ops,
- },
-};
-
-/* Audio machine driver */
-static struct snd_soc_card omap_abe_card = {
- .owner = THIS_MODULE,
-
- .dapm_widgets = twl6040_dapm_widgets,
- .num_dapm_widgets = ARRAY_SIZE(twl6040_dapm_widgets),
- .dapm_routes = audio_map,
- .num_dapm_routes = ARRAY_SIZE(audio_map),
-};
-
static int omap_abe_probe(struct platform_device *pdev)
{
struct device_node *node = pdev->dev.of_node;
- struct snd_soc_card *card = &omap_abe_card;
+ struct snd_soc_card *card;
struct device_node *dai_node;
struct abe_twl6040 *priv;
int num_links = 0;
@@ -252,12 +224,18 @@ static int omap_abe_probe(struct platform_device *pdev)
return -ENODEV;
}
- card->dev = &pdev->dev;
-
priv = devm_kzalloc(&pdev->dev, sizeof(struct abe_twl6040), GFP_KERNEL);
if (priv == NULL)
return -ENOMEM;
+ card = &priv->card;
+ card->dev = &pdev->dev;
+ card->owner = THIS_MODULE;
+ card->dapm_widgets = twl6040_dapm_widgets;
+ card->num_dapm_widgets = ARRAY_SIZE(twl6040_dapm_widgets);
+ card->dapm_routes = audio_map;
+ card->num_dapm_routes = ARRAY_SIZE(audio_map);
+
if (snd_soc_of_parse_card_name(card, "ti,model")) {
dev_err(&pdev->dev, "Card name is not provided\n");
return -ENODEV;
@@ -274,14 +252,27 @@ static int omap_abe_probe(struct platform_device *pdev)
dev_err(&pdev->dev, "McPDM node is not provided\n");
return -EINVAL;
}
- abe_twl6040_dai_links[0].cpu_of_node = dai_node;
- abe_twl6040_dai_links[0].platform_of_node = dai_node;
+
+ priv->dai_links[0].name = "DMIC";
+ priv->dai_links[0].stream_name = "TWL6040";
+ priv->dai_links[0].cpu_of_node = dai_node;
+ priv->dai_links[0].platform_of_node = dai_node;
+ priv->dai_links[0].codec_dai_name = "twl6040-legacy";
+ priv->dai_links[0].codec_name = "twl6040-codec";
+ priv->dai_links[0].init = omap_abe_twl6040_init;
+ priv->dai_links[0].ops = &omap_abe_ops;
dai_node = of_parse_phandle(node, "ti,dmic", 0);
if (dai_node) {
num_links = 2;
- abe_twl6040_dai_links[1].cpu_of_node = dai_node;
- abe_twl6040_dai_links[1].platform_of_node = dai_node;
+ priv->dai_links[1].name = "TWL6040";
+ priv->dai_links[1].stream_name = "DMIC Capture";
+ priv->dai_links[1].cpu_of_node = dai_node;
+ priv->dai_links[1].platform_of_node = dai_node;
+ priv->dai_links[1].codec_dai_name = "dmic-hifi";
+ priv->dai_links[1].codec_name = "dmic-codec";
+ priv->dai_links[1].init = omap_abe_dmic_init;
+ priv->dai_links[1].ops = &omap_abe_dmic_ops;
} else {
num_links = 1;
}
@@ -300,7 +291,7 @@ static int omap_abe_probe(struct platform_device *pdev)
return -ENODEV;
}
- card->dai_link = abe_twl6040_dai_links;
+ card->dai_link = priv->dai_links;
card->num_links = num_links;
snd_soc_card_set_drvdata(card, priv);
diff --git a/sound/soc/omap/omap-dmic.c b/sound/soc/omap/omap-dmic.c
index 09db2aec..776e809 100644
--- a/sound/soc/omap/omap-dmic.c
+++ b/sound/soc/omap/omap-dmic.c
@@ -48,6 +48,8 @@ struct omap_dmic {
struct device *dev;
void __iomem *io_base;
struct clk *fclk;
+ struct pm_qos_request pm_qos_req;
+ int latency;
int fclk_freq;
int out_freq;
int clk_div;
@@ -124,6 +126,8 @@ static void omap_dmic_dai_shutdown(struct snd_pcm_substream *substream,
mutex_lock(&dmic->mutex);
+ pm_qos_remove_request(&dmic->pm_qos_req);
+
if (!dai->active)
dmic->active = 0;
@@ -226,6 +230,8 @@ static int omap_dmic_dai_hw_params(struct snd_pcm_substream *substream,
/* packet size is threshold * channels */
dma_data = snd_soc_dai_get_dma_data(dai, substream);
dma_data->maxburst = dmic->threshold * channels;
+ dmic->latency = (OMAP_DMIC_THRES_MAX - dmic->threshold) * USEC_PER_SEC /
+ params_rate(params);
return 0;
}
@@ -236,6 +242,9 @@ static int omap_dmic_dai_prepare(struct snd_pcm_substream *substream,
struct omap_dmic *dmic = snd_soc_dai_get_drvdata(dai);
u32 ctrl;
+ if (pm_qos_request_active(&dmic->pm_qos_req))
+ pm_qos_update_request(&dmic->pm_qos_req, dmic->latency);
+
/* Configure uplink threshold */
omap_dmic_write(dmic, OMAP_DMIC_FIFO_CTRL_REG, dmic->threshold);
diff --git a/sound/soc/omap/omap-mcpdm.c b/sound/soc/omap/omap-mcpdm.c
index 64609c7..44ffeb7 100644
--- a/sound/soc/omap/omap-mcpdm.c
+++ b/sound/soc/omap/omap-mcpdm.c
@@ -54,6 +54,8 @@ struct omap_mcpdm {
unsigned long phys_base;
void __iomem *io_base;
int irq;
+ struct pm_qos_request pm_qos_req;
+ int latency[2];
struct mutex mutex;
@@ -277,6 +279,9 @@ static void omap_mcpdm_dai_shutdown(struct snd_pcm_substream *substream,
struct snd_soc_dai *dai)
{
struct omap_mcpdm *mcpdm = snd_soc_dai_get_drvdata(dai);
+ int tx = (substream->stream == SNDRV_PCM_STREAM_PLAYBACK);
+ int stream1 = tx ? SNDRV_PCM_STREAM_PLAYBACK : SNDRV_PCM_STREAM_CAPTURE;
+ int stream2 = tx ? SNDRV_PCM_STREAM_CAPTURE : SNDRV_PCM_STREAM_PLAYBACK;
mutex_lock(&mcpdm->mutex);
@@ -289,6 +294,14 @@ static void omap_mcpdm_dai_shutdown(struct snd_pcm_substream *substream,
}
}
+ if (mcpdm->latency[stream2])
+ pm_qos_update_request(&mcpdm->pm_qos_req,
+ mcpdm->latency[stream2]);
+ else if (mcpdm->latency[stream1])
+ pm_qos_remove_request(&mcpdm->pm_qos_req);
+
+ mcpdm->latency[stream1] = 0;
+
mutex_unlock(&mcpdm->mutex);
}
@@ -300,7 +313,7 @@ static int omap_mcpdm_dai_hw_params(struct snd_pcm_substream *substream,
int stream = substream->stream;
struct snd_dmaengine_dai_dma_data *dma_data;
u32 threshold;
- int channels;
+ int channels, latency;
int link_mask = 0;
channels = params_channels(params);
@@ -340,14 +353,25 @@ static int omap_mcpdm_dai_hw_params(struct snd_pcm_substream *substream,
dma_data->maxburst =
(MCPDM_DN_THRES_MAX - threshold) * channels;
+ latency = threshold;
} else {
/* If playback is not running assume a stereo stream to come */
if (!mcpdm->config[!stream].link_mask)
mcpdm->config[!stream].link_mask = (0x3 << 3);
dma_data->maxburst = threshold * channels;
+ latency = (MCPDM_DN_THRES_MAX - threshold);
}
+ /*
+ * The DMA must act to a DMA request within latency time (usec) to avoid
+ * under/overflow
+ */
+ mcpdm->latency[stream] = latency * USEC_PER_SEC / params_rate(params);
+
+ if (!mcpdm->latency[stream])
+ mcpdm->latency[stream] = 10;
+
/* Check if we need to restart McPDM with this stream */
if (mcpdm->config[stream].link_mask &&
mcpdm->config[stream].link_mask != link_mask)
@@ -362,6 +386,20 @@ static int omap_mcpdm_prepare(struct snd_pcm_substream *substream,
struct snd_soc_dai *dai)
{
struct omap_mcpdm *mcpdm = snd_soc_dai_get_drvdata(dai);
+ struct pm_qos_request *pm_qos_req = &mcpdm->pm_qos_req;
+ int tx = (substream->stream == SNDRV_PCM_STREAM_PLAYBACK);
+ int stream1 = tx ? SNDRV_PCM_STREAM_PLAYBACK : SNDRV_PCM_STREAM_CAPTURE;
+ int stream2 = tx ? SNDRV_PCM_STREAM_CAPTURE : SNDRV_PCM_STREAM_PLAYBACK;
+ int latency = mcpdm->latency[stream2];
+
+ /* Prevent omap hardware from hitting off between FIFO fills */
+ if (!latency || mcpdm->latency[stream1] < latency)
+ latency = mcpdm->latency[stream1];
+
+ if (pm_qos_request_active(pm_qos_req))
+ pm_qos_update_request(pm_qos_req, latency);
+ else if (latency)
+ pm_qos_add_request(pm_qos_req, PM_QOS_CPU_DMA_LATENCY, latency);
if (!omap_mcpdm_active(mcpdm)) {
omap_mcpdm_start(mcpdm);
@@ -423,6 +461,9 @@ static int omap_mcpdm_remove(struct snd_soc_dai *dai)
free_irq(mcpdm->irq, (void *)mcpdm);
pm_runtime_disable(mcpdm->dev);
+ if (pm_qos_request_active(&mcpdm->pm_qos_req))
+ pm_qos_remove_request(&mcpdm->pm_qos_req);
+
return 0;
}
diff --git a/sound/soc/samsung/i2s.c b/sound/soc/samsung/i2s.c
index 85324e6..2d14e37 100644
--- a/sound/soc/samsung/i2s.c
+++ b/sound/soc/samsung/i2s.c
@@ -642,8 +642,12 @@ static int i2s_set_fmt(struct snd_soc_dai *dai,
tmp |= mod_slave;
break;
case SND_SOC_DAIFMT_CBS_CFS:
- /* Set default source clock in Master mode */
- if (i2s->rclk_srcrate == 0)
+ /*
+ * Set default source clock in Master mode, only when the
+ * CLK_I2S_RCLK_SRC clock is not exposed so we ensure any
+ * clock configuration assigned in DT is not overwritten.
+ */
+ if (i2s->rclk_srcrate == 0 && i2s->clk_data.clks == NULL)
i2s_set_sysclk(dai, SAMSUNG_I2S_RCLKSRC_0,
0, SND_SOC_CLOCK_IN);
break;
@@ -858,6 +862,11 @@ static int config_setup(struct i2s_dai *i2s)
return 0;
if (!(i2s->quirks & QUIRK_NO_MUXPSR)) {
+ struct clk *rclksrc = i2s->clk_table[CLK_I2S_RCLK_SRC];
+
+ if (i2s->rclk_srcrate == 0 && rclksrc && !IS_ERR(rclksrc))
+ i2s->rclk_srcrate = clk_get_rate(rclksrc);
+
psr = i2s->rclk_srcrate / i2s->frmclk / rfs;
writel(((psr - 1) << 8) | PSR_PSREN, i2s->addr + I2SPSR);
dev_dbg(&i2s->pdev->dev,
diff --git a/sound/soc/sirf/sirf-usp.c b/sound/soc/sirf/sirf-usp.c
index 45fc06c..6b504f4 100644
--- a/sound/soc/sirf/sirf-usp.c
+++ b/sound/soc/sirf/sirf-usp.c
@@ -367,10 +367,9 @@ static int sirf_usp_pcm_probe(struct platform_device *pdev)
platform_set_drvdata(pdev, usp);
mem_res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
- base = devm_ioremap(&pdev->dev, mem_res->start,
- resource_size(mem_res));
- if (base == NULL)
- return -ENOMEM;
+ base = devm_ioremap_resource(&pdev->dev, mem_res);
+ if (IS_ERR(base))
+ return PTR_ERR(base);
usp->regmap = devm_regmap_init_mmio(&pdev->dev, base,
&sirf_usp_regmap_config);
if (IS_ERR(usp->regmap))
diff --git a/sound/soc/soc-core.c b/sound/soc/soc-core.c
index b761761..f8110b7 100644
--- a/sound/soc/soc-core.c
+++ b/sound/soc/soc-core.c
@@ -2036,6 +2036,7 @@ static int snd_soc_instantiate_card(struct snd_soc_card *card)
}
card->instantiated = 1;
+ dapm_mark_endpoints_dirty(card);
snd_soc_dapm_sync(&card->dapm);
mutex_unlock(&card->mutex);
mutex_unlock(&client_mutex);
diff --git a/sound/soc/soc-dapm.c b/sound/soc/soc-dapm.c
index 172af54..b36c856 100644
--- a/sound/soc/soc-dapm.c
+++ b/sound/soc/soc-dapm.c
@@ -427,6 +427,8 @@ static int dapm_kcontrol_data_alloc(struct snd_soc_dapm_widget *widget,
static void dapm_kcontrol_free(struct snd_kcontrol *kctl)
{
struct dapm_kcontrol_data *data = snd_kcontrol_chip(kctl);
+
+ list_del(&data->paths);
kfree(data->wlist);
kfree(data);
}
@@ -3927,6 +3929,13 @@ int snd_soc_dapm_link_dai_widgets(struct snd_soc_card *card)
continue;
}
+ /* let users know there is no DAI to link */
+ if (!dai_w->priv) {
+ dev_dbg(card->dev, "dai widget %s has no DAI\n",
+ dai_w->name);
+ continue;
+ }
+
dai = dai_w->priv;
/* ...find all widgets with the same stream and link them */
diff --git a/sound/soc/soc-pcm.c b/sound/soc/soc-pcm.c
index bd8f34a..7fb237b 100644
--- a/sound/soc/soc-pcm.c
+++ b/sound/soc/soc-pcm.c
@@ -1739,6 +1739,14 @@ static u64 dpcm_runtime_base_format(struct snd_pcm_substream *substream)
int i;
for (i = 0; i < be->num_codecs; i++) {
+ /*
+ * Skip CODECs which don't support the current stream
+ * type. See soc_pcm_init_runtime_hw() for more details
+ */
+ if (!snd_soc_dai_stream_valid(be->codec_dais[i],
+ stream))
+ continue;
+
codec_dai_drv = be->codec_dais[i]->driver;
if (stream == SNDRV_PCM_STREAM_PLAYBACK)
codec_stream = &codec_dai_drv->playback;
@@ -1911,8 +1919,10 @@ int dpcm_be_dai_shutdown(struct snd_soc_pcm_runtime *fe, int stream)
continue;
if ((be->dpcm[stream].state != SND_SOC_DPCM_STATE_HW_FREE) &&
- (be->dpcm[stream].state != SND_SOC_DPCM_STATE_OPEN))
- continue;
+ (be->dpcm[stream].state != SND_SOC_DPCM_STATE_OPEN)) {
+ soc_pcm_hw_free(be_substream);
+ be->dpcm[stream].state = SND_SOC_DPCM_STATE_HW_FREE;
+ }
dev_dbg(be->dev, "ASoC: close BE %s\n",
be->dai_link->name);
diff --git a/sound/soc/soc-topology.c b/sound/soc/soc-topology.c
index 8a758c9..d6b48c7 100644
--- a/sound/soc/soc-topology.c
+++ b/sound/soc/soc-topology.c
@@ -1180,6 +1180,9 @@ static struct snd_kcontrol_new *soc_tplg_dapm_widget_dmixer_create(
kfree(sm);
continue;
}
+
+ /* create any TLV data */
+ soc_tplg_create_tlv(tplg, &kc[i], &mc->hdr);
}
return kc;
diff --git a/sound/sparc/cs4231.c b/sound/sparc/cs4231.c
index 30bdc97..017e241 100644
--- a/sound/sparc/cs4231.c
+++ b/sound/sparc/cs4231.c
@@ -1146,10 +1146,8 @@ static int snd_cs4231_playback_open(struct snd_pcm_substream *substream)
runtime->hw = snd_cs4231_playback;
err = snd_cs4231_open(chip, CS4231_MODE_PLAY);
- if (err < 0) {
- snd_free_pages(runtime->dma_area, runtime->dma_bytes);
+ if (err < 0)
return err;
- }
chip->playback_substream = substream;
chip->p_periods_sent = 0;
snd_pcm_set_sync(substream);
@@ -1167,10 +1165,8 @@ static int snd_cs4231_capture_open(struct snd_pcm_substream *substream)
runtime->hw = snd_cs4231_capture;
err = snd_cs4231_open(chip, CS4231_MODE_RECORD);
- if (err < 0) {
- snd_free_pages(runtime->dma_area, runtime->dma_bytes);
+ if (err < 0)
return err;
- }
chip->capture_substream = substream;
chip->c_periods_sent = 0;
snd_pcm_set_sync(substream);
diff --git a/sound/synth/emux/emux_hwdep.c b/sound/synth/emux/emux_hwdep.c
index e557946..d9fcae0 100644
--- a/sound/synth/emux/emux_hwdep.c
+++ b/sound/synth/emux/emux_hwdep.c
@@ -22,9 +22,9 @@
#include <sound/core.h>
#include <sound/hwdep.h>
#include <linux/uaccess.h>
+#include <linux/nospec.h>
#include "emux_voice.h"
-
#define TMP_CLIENT_ID 0x1001
/*
@@ -66,13 +66,16 @@ snd_emux_hwdep_misc_mode(struct snd_emux *emu, void __user *arg)
return -EFAULT;
if (info.mode < 0 || info.mode >= EMUX_MD_END)
return -EINVAL;
+ info.mode = array_index_nospec(info.mode, EMUX_MD_END);
if (info.port < 0) {
for (i = 0; i < emu->num_ports; i++)
emu->portptrs[i]->ctrls[info.mode] = info.value;
} else {
- if (info.port < emu->num_ports)
+ if (info.port < emu->num_ports) {
+ info.port = array_index_nospec(info.port, emu->num_ports);
emu->portptrs[info.port]->ctrls[info.mode] = info.value;
+ }
}
return 0;
}
diff --git a/sound/usb/card.c b/sound/usb/card.c
index f029f8c..8abe8bd 100644
--- a/sound/usb/card.c
+++ b/sound/usb/card.c
@@ -742,9 +742,12 @@ static int usb_audio_probe(struct usb_interface *intf,
__error:
if (chip) {
+ /* chip->active is inside the chip->card object,
+ * decrement before memory is possibly returned.
+ */
+ atomic_dec(&chip->active);
if (!chip->num_interfaces)
snd_card_free(chip->card);
- atomic_dec(&chip->active);
}
mutex_unlock(®ister_mutex);
return err;
diff --git a/sound/usb/clock.c b/sound/usb/clock.c
index 09c6e29..4cd1341 100644
--- a/sound/usb/clock.c
+++ b/sound/usb/clock.c
@@ -151,7 +151,7 @@ static bool uac_clock_source_is_valid(struct snd_usb_audio *chip, int source_id)
/* If a clock source can't tell us whether it's valid, we assume it is */
if (!uac2_control_is_readable(cs_desc->bmControls,
- UAC2_CS_CONTROL_CLOCK_VALID - 1))
+ UAC2_CS_CONTROL_CLOCK_VALID))
return 1;
err = snd_usb_ctl_msg(dev, usb_rcvctrlpipe(dev, 0), UAC2_CS_CUR,
@@ -376,7 +376,8 @@ static int set_sample_rate_v2(struct snd_usb_audio *chip, int iface,
return 0;
cs_desc = snd_usb_find_clock_source(chip->ctrl_intf, clock);
- writeable = uac2_control_is_writeable(cs_desc->bmControls, UAC2_CS_CONTROL_SAM_FREQ - 1);
+ writeable = uac2_control_is_writeable(cs_desc->bmControls,
+ UAC2_CS_CONTROL_SAM_FREQ);
if (writeable) {
data = cpu_to_le32(rate);
err = snd_usb_ctl_msg(dev, usb_sndctrlpipe(dev, 0), UAC2_CS_CUR,
diff --git a/sound/usb/mixer.c b/sound/usb/mixer.c
index ce11cc9..f61ffe8 100644
--- a/sound/usb/mixer.c
+++ b/sound/usb/mixer.c
@@ -888,26 +888,27 @@ static int check_input_term(struct mixer_build *state, int id,
/* feature unit control information */
struct usb_feature_control_info {
+ int control;
const char *name;
- int type; /* data type for uac1 */
+ int type; /* data type for uac1 */
int type_uac2; /* data type for uac2 if different from uac1, else -1 */
};
static struct usb_feature_control_info audio_feature_info[] = {
- { "Mute", USB_MIXER_INV_BOOLEAN, -1 },
- { "Volume", USB_MIXER_S16, -1 },
- { "Tone Control - Bass", USB_MIXER_S8, -1 },
- { "Tone Control - Mid", USB_MIXER_S8, -1 },
- { "Tone Control - Treble", USB_MIXER_S8, -1 },
- { "Graphic Equalizer", USB_MIXER_S8, -1 }, /* FIXME: not implemeted yet */
- { "Auto Gain Control", USB_MIXER_BOOLEAN, -1 },
- { "Delay Control", USB_MIXER_U16, USB_MIXER_U32 },
- { "Bass Boost", USB_MIXER_BOOLEAN, -1 },
- { "Loudness", USB_MIXER_BOOLEAN, -1 },
+ { UAC_FU_MUTE, "Mute", USB_MIXER_INV_BOOLEAN, -1 },
+ { UAC_FU_VOLUME, "Volume", USB_MIXER_S16, -1 },
+ { UAC_FU_BASS, "Tone Control - Bass", USB_MIXER_S8, -1 },
+ { UAC_FU_MID, "Tone Control - Mid", USB_MIXER_S8, -1 },
+ { UAC_FU_TREBLE, "Tone Control - Treble", USB_MIXER_S8, -1 },
+ { UAC_FU_GRAPHIC_EQUALIZER, "Graphic Equalizer", USB_MIXER_S8, -1 }, /* FIXME: not implemented yet */
+ { UAC_FU_AUTOMATIC_GAIN, "Auto Gain Control", USB_MIXER_BOOLEAN, -1 },
+ { UAC_FU_DELAY, "Delay Control", USB_MIXER_U16, USB_MIXER_U32 },
+ { UAC_FU_BASS_BOOST, "Bass Boost", USB_MIXER_BOOLEAN, -1 },
+ { UAC_FU_LOUDNESS, "Loudness", USB_MIXER_BOOLEAN, -1 },
/* UAC2 specific */
- { "Input Gain Control", USB_MIXER_S16, -1 },
- { "Input Gain Pad Control", USB_MIXER_S16, -1 },
- { "Phase Inverter Control", USB_MIXER_BOOLEAN, -1 },
+ { UAC2_FU_INPUT_GAIN, "Input Gain Control", USB_MIXER_S16, -1 },
+ { UAC2_FU_INPUT_GAIN_PAD, "Input Gain Pad Control", USB_MIXER_S16, -1 },
+ { UAC2_FU_PHASE_INVERTER, "Phase Inverter Control", USB_MIXER_BOOLEAN, -1 },
};
/* private_free callback */
@@ -975,6 +976,14 @@ static void volume_control_quirks(struct usb_mixer_elem_info *cval,
}
break;
+ case USB_ID(0x0d8c, 0x0103):
+ if (!strcmp(kctl->id.name, "PCM Playback Volume")) {
+ usb_audio_info(chip,
+ "set volume quirk for CM102-A+/102S+\n");
+ cval->min = -256;
+ }
+ break;
+
case USB_ID(0x0471, 0x0101):
case USB_ID(0x0471, 0x0104):
case USB_ID(0x0471, 0x0105):
@@ -1256,6 +1265,21 @@ static int mixer_ctl_feature_put(struct snd_kcontrol *kcontrol,
return changed;
}
+/* get the current value from a mixer element */
+static int mixer_ctl_connector_get(struct snd_kcontrol *kcontrol,
+ struct snd_ctl_elem_value *ucontrol)
+{
+ struct usb_mixer_elem_info *cval = kcontrol->private_data;
+ int val, err;
+
+ err = snd_usb_get_cur_mix_value(cval, 0, 0, &val);
+ if (err < 0)
+ return filter_error(cval, err);
+ val = (val != 0);
+ ucontrol->value.integer.value[0] = val;
+ return 0;
+}
+
static struct snd_kcontrol_new usb_feature_unit_ctl = {
.iface = SNDRV_CTL_ELEM_IFACE_MIXER,
.name = "", /* will be filled later manually */
@@ -1273,6 +1297,16 @@ static struct snd_kcontrol_new usb_feature_unit_ctl_ro = {
.put = NULL,
};
+/* A UAC control mixer control */
+static struct snd_kcontrol_new usb_connector_ctl_ro = {
+ .iface = SNDRV_CTL_ELEM_IFACE_CARD,
+ .name = "", /* will be filled later manually */
+ .access = SNDRV_CTL_ELEM_ACCESS_READ,
+ .info = snd_ctl_boolean_mono_info,
+ .get = mixer_ctl_connector_get,
+ .put = NULL,
+};
+
/*
* This symbol is exported in order to allow the mixer quirks to
* hook up to the standard feature unit control mechanism
@@ -1315,6 +1349,17 @@ static void check_no_speaker_on_headset(struct snd_kcontrol *kctl,
strlcpy(kctl->id.name, "Headphone", sizeof(kctl->id.name));
}
+static struct usb_feature_control_info *get_feature_control_info(int control)
+{
+ int i;
+
+ for (i = 0; i < ARRAY_SIZE(audio_feature_info); ++i) {
+ if (audio_feature_info[i].control == control)
+ return &audio_feature_info[i];
+ }
+ return NULL;
+}
+
static void build_feature_ctl(struct mixer_build *state, void *raw_desc,
unsigned int ctl_mask, int control,
struct usb_audio_term *iterm, int unitid,
@@ -1336,8 +1381,6 @@ static void build_feature_ctl(struct mixer_build *state, void *raw_desc,
else
nameid = uac_feature_unit_iFeature(desc);
- control++; /* change from zero-based to 1-based value */
-
if (control == UAC_FU_GRAPHIC_EQUALIZER) {
/* FIXME: not supported yet */
return;
@@ -1353,7 +1396,11 @@ static void build_feature_ctl(struct mixer_build *state, void *raw_desc,
snd_usb_mixer_elem_init_std(&cval->head, state->mixer, unitid);
cval->control = control;
cval->cmask = ctl_mask;
- ctl_info = &audio_feature_info[control-1];
+ ctl_info = get_feature_control_info(control);
+ if (!ctl_info) {
+ kfree(cval);
+ return;
+ }
if (state->mixer->protocol == UAC_VERSION_1)
cval->val_type = ctl_info->type;
else /* UAC_VERSION_2 or UAC_VERSION_3*/
@@ -1479,6 +1526,62 @@ static void build_feature_ctl(struct mixer_build *state, void *raw_desc,
snd_usb_mixer_add_control(&cval->head, kctl);
}
+static void get_connector_control_name(struct mixer_build *state,
+ struct usb_audio_term *term,
+ bool is_input, char *name, int name_size)
+{
+ int name_len = get_term_name(state, term, name, name_size, 0);
+
+ if (name_len == 0)
+ strlcpy(name, "Unknown", name_size);
+
+ /*
+ * sound/core/ctljack.c has a convention of naming jack controls
+ * by ending in " Jack". Make it slightly more useful by
+ * indicating Input or Output after the terminal name.
+ */
+ if (is_input)
+ strlcat(name, " - Input Jack", name_size);
+ else
+ strlcat(name, " - Output Jack", name_size);
+}
+
+/* Build a mixer control for a UAC connector control (jack-detect) */
+static void build_connector_control(struct mixer_build *state,
+ struct usb_audio_term *term, bool is_input)
+{
+ struct snd_kcontrol *kctl;
+ struct usb_mixer_elem_info *cval;
+
+ cval = kzalloc(sizeof(*cval), GFP_KERNEL);
+ if (!cval)
+ return;
+ snd_usb_mixer_elem_init_std(&cval->head, state->mixer, term->id);
+ /*
+ * The first byte from reading the UAC2_TE_CONNECTOR control returns the
+ * number of channels connected. This boolean ctl will simply report
+ * if any channels are connected or not.
+ * (Audio20_final.pdf Table 5-10: Connector Control CUR Parameter Block)
+ */
+ cval->control = UAC2_TE_CONNECTOR;
+ cval->val_type = USB_MIXER_BOOLEAN;
+ cval->channels = 1; /* report true if any channel is connected */
+ cval->master_readonly = 1;
+ cval->min = 0;
+ cval->max = 1;
+ kctl = snd_ctl_new1(&usb_connector_ctl_ro, cval);
+ if (!kctl) {
+ usb_audio_err(state->chip, "cannot malloc kcontrol\n");
+ kfree(cval);
+ return;
+ }
+ get_connector_control_name(state, term, is_input, kctl->id.name,
+ sizeof(kctl->id.name));
+ kctl->private_free = snd_usb_mixer_elem_free;
+ snd_usb_mixer_add_control(&cval->head, kctl);
+}
+
+
static int parse_clock_source_unit(struct mixer_build *state, int unitid,
void *_ftr)
{
@@ -1503,7 +1606,7 @@ static int parse_clock_source_unit(struct mixer_build *state, int unitid,
* clock source validity. If that isn't readable, just bail out.
*/
if (!uac2_control_is_readable(hdr->bmControls,
- ilog2(UAC2_CS_CONTROL_CLOCK_VALID)))
+ UAC2_CS_CONTROL_CLOCK_VALID))
return 0;
cval = kzalloc(sizeof(*cval), GFP_KERNEL);
@@ -1519,7 +1622,7 @@ static int parse_clock_source_unit(struct mixer_build *state, int unitid,
cval->control = UAC2_CS_CONTROL_CLOCK_VALID;
if (uac2_control_is_writeable(hdr->bmControls,
- ilog2(UAC2_CS_CONTROL_CLOCK_VALID)))
+ UAC2_CS_CONTROL_CLOCK_VALID))
kctl = snd_ctl_new1(&usb_feature_unit_ctl, cval);
else {
cval->master_readonly = 1;
@@ -1798,6 +1901,8 @@ static int parse_audio_feature_unit(struct mixer_build *state, int unitid,
/* check all control types */
for (i = 0; i < 10; i++) {
unsigned int ch_bits = 0;
+ int control = audio_feature_info[i].control;
+
for (j = 0; j < channels; j++) {
unsigned int mask;
@@ -1813,26 +1918,29 @@ static int parse_audio_feature_unit(struct mixer_build *state, int unitid,
* (for ease of programming).
*/
if (ch_bits & 1)
- build_feature_ctl(state, _ftr, ch_bits, i,
+ build_feature_ctl(state, _ftr, ch_bits, control,
&iterm, unitid, 0);
if (master_bits & (1 << i))
- build_feature_ctl(state, _ftr, 0, i, &iterm,
- unitid, 0);
+ build_feature_ctl(state, _ftr, 0, control,
+ &iterm, unitid, 0);
}
} else { /* UAC_VERSION_2 or UAC_VERSION_3*/
for (i = 0; i < ARRAY_SIZE(audio_feature_info); i++) {
unsigned int ch_bits = 0;
unsigned int ch_read_only = 0;
+ int control = audio_feature_info[i].control;
for (j = 0; j < channels; j++) {
unsigned int mask;
mask = snd_usb_combine_bytes(bmaControls +
csize * (j+1), csize);
- if (uac2_control_is_readable(mask, i)) {
+ if (uac2_control_is_readable(mask, control)) {
ch_bits |= (1 << j);
- if (!uac2_control_is_writeable(mask, i))
+ if (!uac2_control_is_writeable(mask,
+ control)) {
ch_read_only |= (1 << j);
+ }
}
}
@@ -1850,11 +1958,14 @@ static int parse_audio_feature_unit(struct mixer_build *state, int unitid,
* (for ease of programming).
*/
if (ch_bits & 1)
- build_feature_ctl(state, _ftr, ch_bits, i,
+ build_feature_ctl(state, _ftr, ch_bits, control,
&iterm, unitid, ch_read_only);
- if (uac2_control_is_readable(master_bits, i))
- build_feature_ctl(state, _ftr, 0, i, &iterm, unitid,
- !uac2_control_is_writeable(master_bits, i));
+ if (uac2_control_is_readable(master_bits, control)) {
+ int wr = uac2_control_is_writeable(master_bits,
+ control);
+ build_feature_ctl(state, _ftr, 0, control,
+ &iterm, unitid, !wr);
+ }
}
}
@@ -1926,6 +2037,25 @@ static void build_mixer_unit_ctl(struct mixer_build *state,
snd_usb_mixer_add_control(&cval->head, kctl);
}
+static int parse_audio_input_terminal(struct mixer_build *state, int unitid,
+ void *raw_desc)
+{
+ struct usb_audio_term iterm;
+
+ struct uac2_input_terminal_descriptor *d = raw_desc;
+ /* determine the input source type and name */
+ check_input_term(state, d->bTerminalID, &iterm);
+
+ if (state->mixer->protocol == UAC_VERSION_2) {
+ /* Check for jack detection. */
+ if (uac2_control_is_readable(d->bmControls,
+ UAC2_TE_CONNECTOR)) {
+ build_connector_control(state, &iterm, true);
+ }
+ }
+ return 0;
+}
+
/*
* parse a mixer unit
*/
@@ -2139,7 +2269,7 @@ static int build_audio_procunit(struct mixer_build *state, int unitid,
char *name)
{
struct uac_processing_unit_descriptor *desc = raw_desc;
- int num_ins = desc->bNrInPins;
+ int num_ins;
struct usb_mixer_elem_info *cval;
struct snd_kcontrol *kctl;
int i, err, nameid, type, len;
@@ -2154,7 +2284,13 @@ static int build_audio_procunit(struct mixer_build *state, int unitid,
0, NULL, default_value_info
};
- if (desc->bLength < 13 || desc->bLength < 13 + num_ins ||
+ if (desc->bLength < 13) {
+ usb_audio_err(state->chip, "invalid %s descriptor (id %d)\n", name, unitid);
+ return -EINVAL;
+ }
+
+ num_ins = desc->bNrInPins;
+ if (desc->bLength < 13 + num_ins ||
desc->bLength < num_ins + uac_processing_unit_bControlSize(desc, state->mixer->protocol)) {
usb_audio_err(state->chip, "invalid %s descriptor (id %d)\n", name, unitid);
return -EINVAL;
@@ -2487,7 +2623,7 @@ static int parse_audio_unit(struct mixer_build *state, int unitid)
switch (p1[2]) {
case UAC_INPUT_TERMINAL:
- return 0; /* NOP */
+ return parse_audio_input_terminal(state, unitid, p1);
case UAC_MIXER_UNIT:
return parse_audio_mixer_unit(state, unitid, p1);
case UAC2_CLOCK_SOURCE:
@@ -2697,6 +2833,13 @@ static int snd_usb_mixer_controls(struct usb_mixer_interface *mixer)
desc->bCSourceID);
if (err < 0 && err != -EINVAL)
return err;
+
+ if (uac2_control_is_readable(desc->bmControls,
+ UAC2_TE_CONNECTOR)) {
+ build_connector_control(&state,
+ &state.oterm,
+ false);
+ }
}
}
}
diff --git a/sound/usb/mixer_maps.c b/sound/usb/mixer_maps.c
index 9038b2e..eaa03ac 100644
--- a/sound/usb/mixer_maps.c
+++ b/sound/usb/mixer_maps.c
@@ -353,8 +353,11 @@ static struct usbmix_name_map bose_companion5_map[] = {
/*
* Dell usb dock with ALC4020 codec had a firmware problem where it got
* screwed up when zero volume is passed; just skip it as a workaround
+ *
+ * Also the extension unit gives an access error, so skip it as well.
*/
static const struct usbmix_name_map dell_alc4020_map[] = {
+ { 4, NULL }, /* extension unit */
{ 16, NULL },
{ 19, NULL },
{ 0 }
diff --git a/sound/usb/pcm.c b/sound/usb/pcm.c
index dc0a9ef..b57d1d4 100644
--- a/sound/usb/pcm.c
+++ b/sound/usb/pcm.c
@@ -1379,7 +1379,7 @@ static void retire_capture_urb(struct snd_usb_substream *subs,
if (bytes % (runtime->sample_bits >> 3) != 0) {
int oldbytes = bytes;
bytes = frames * stride;
- dev_warn(&subs->dev->dev,
+ dev_warn_ratelimited(&subs->dev->dev,
"Corrected urb data len. %d->%d\n",
oldbytes, bytes);
}
diff --git a/sound/usb/quirks-table.h b/sound/usb/quirks-table.h
index 69bf5cf..d32727c 100644
--- a/sound/usb/quirks-table.h
+++ b/sound/usb/quirks-table.h
@@ -2875,7 +2875,8 @@ YAMAHA_DEVICE(0x7010, "UB99"),
*/
#define AU0828_DEVICE(vid, pid, vname, pname) { \
- USB_DEVICE_VENDOR_SPEC(vid, pid), \
+ .idVendor = vid, \
+ .idProduct = pid, \
.match_flags = USB_DEVICE_ID_MATCH_DEVICE | \
USB_DEVICE_ID_MATCH_INT_CLASS | \
USB_DEVICE_ID_MATCH_INT_SUBCLASS, \
@@ -3320,6 +3321,9 @@ AU0828_DEVICE(0x2040, 0x7270, "Hauppauge", "HVR-950Q"),
}
}
},
+ {
+ .ifnum = -1
+ },
}
}
},
diff --git a/sound/usb/quirks.c b/sound/usb/quirks.c
index 45655b9..da9fc08 100644
--- a/sound/usb/quirks.c
+++ b/sound/usb/quirks.c
@@ -1153,24 +1153,27 @@ bool snd_usb_get_sample_rate_quirk(struct snd_usb_audio *chip)
return false;
}
-/* Marantz/Denon USB DACs need a vendor cmd to switch
+/* ITF-USB DSD based DACs need a vendor cmd to switch
* between PCM and native DSD mode
+ * (2 altsets version)
*/
-static bool is_marantz_denon_dac(unsigned int id)
+static bool is_itf_usb_dsd_2alts_dac(unsigned int id)
{
switch (id) {
case USB_ID(0x154e, 0x1003): /* Denon DA-300USB */
case USB_ID(0x154e, 0x3005): /* Marantz HD-DAC1 */
case USB_ID(0x154e, 0x3006): /* Marantz SA-14S1 */
+ case USB_ID(0x1852, 0x5065): /* Luxman DA-06 */
return true;
}
return false;
}
-/* TEAC UD-501/UD-503/NT-503 USB DACs need a vendor cmd to switch
- * between PCM/DOP and native DSD mode
+/* ITF-USB DSD based DACs need a vendor cmd to switch
+ * between PCM and native DSD mode
+ * (3 altsets version)
*/
-static bool is_teac_dsd_dac(unsigned int id)
+static bool is_itf_usb_dsd_3alts_dac(unsigned int id)
{
switch (id) {
case USB_ID(0x0644, 0x8043): /* TEAC UD-501/UD-503/NT-503 */
@@ -1187,7 +1190,7 @@ int snd_usb_select_mode_quirk(struct snd_usb_substream *subs,
struct usb_device *dev = subs->dev;
int err;
- if (is_marantz_denon_dac(subs->stream->chip->usb_id)) {
+ if (is_itf_usb_dsd_2alts_dac(subs->stream->chip->usb_id)) {
/* First switch to alt set 0, otherwise the mode switch cmd
* will not be accepted by the DAC
*/
@@ -1208,7 +1211,7 @@ int snd_usb_select_mode_quirk(struct snd_usb_substream *subs,
break;
}
mdelay(20);
- } else if (is_teac_dsd_dac(subs->stream->chip->usb_id)) {
+ } else if (is_itf_usb_dsd_3alts_dac(subs->stream->chip->usb_id)) {
/* Vendor mode switch cmd is required. */
switch (fmt->altsetting) {
case 3: /* DSD mode (DSD_U32) requested */
@@ -1304,10 +1307,10 @@ void snd_usb_ctl_msg_quirk(struct usb_device *dev, unsigned int pipe,
(requesttype & USB_TYPE_MASK) == USB_TYPE_CLASS)
mdelay(20);
- /* Marantz/Denon devices with USB DAC functionality need a delay
+ /* ITF-USB DSD based DACs functionality need a delay
* after each class compliant request
*/
- if (is_marantz_denon_dac(chip->usb_id)
+ if (is_itf_usb_dsd_2alts_dac(chip->usb_id)
&& (requesttype & USB_TYPE_MASK) == USB_TYPE_CLASS)
mdelay(20);
@@ -1371,14 +1374,14 @@ u64 snd_usb_interface_dsd_format_quirks(struct snd_usb_audio *chip,
break;
}
- /* Denon/Marantz devices with USB DAC functionality */
- if (is_marantz_denon_dac(chip->usb_id)) {
+ /* ITF-USB DSD based DACs (2 altsets version) */
+ if (is_itf_usb_dsd_2alts_dac(chip->usb_id)) {
if (fp->altsetting == 2)
return SNDRV_PCM_FMTBIT_DSD_U32_BE;
}
- /* TEAC devices with USB DAC functionality */
- if (is_teac_dsd_dac(chip->usb_id)) {
+ /* ITF-USB DSD based DACs (3 altsets version) */
+ if (is_itf_usb_dsd_3alts_dac(chip->usb_id)) {
if (fp->altsetting == 3)
return SNDRV_PCM_FMTBIT_DSD_U32_BE;
}
diff --git a/tools/accounting/getdelays.c b/tools/accounting/getdelays.c
index b5ca536..961e473 100644
--- a/tools/accounting/getdelays.c
+++ b/tools/accounting/getdelays.c
@@ -202,6 +202,8 @@ static void print_delayacct(struct taskstats *t)
"SWAP %15s%15s%15s\n"
" %15llu%15llu%15llums\n"
"RECLAIM %12s%15s%15s\n"
+ " %15llu%15llu%15llums\n"
+ "THRASHING%12s%15s%15s\n"
" %15llu%15llu%15llums\n",
"count", "real total", "virtual total",
"delay total", "delay average",
@@ -221,7 +223,11 @@ static void print_delayacct(struct taskstats *t)
"count", "delay total", "delay average",
(unsigned long long)t->freepages_count,
(unsigned long long)t->freepages_delay_total,
- average_ms(t->freepages_delay_total, t->freepages_count));
+ average_ms(t->freepages_delay_total, t->freepages_count),
+ "count", "delay total", "delay average",
+ (unsigned long long)t->thrashing_count,
+ (unsigned long long)t->thrashing_delay_total,
+ average_ms(t->thrashing_delay_total, t->thrashing_count));
}
static void task_context_switch_counts(struct taskstats *t)
diff --git a/tools/arch/arm/include/uapi/asm/kvm.h b/tools/arch/arm/include/uapi/asm/kvm.h
index a2b3eb3..0b8cf31 100644
--- a/tools/arch/arm/include/uapi/asm/kvm.h
+++ b/tools/arch/arm/include/uapi/asm/kvm.h
@@ -84,6 +84,13 @@ struct kvm_regs {
#define KVM_VGIC_V2_DIST_SIZE 0x1000
#define KVM_VGIC_V2_CPU_SIZE 0x2000
+/* Supported VGICv3 address types */
+#define KVM_VGIC_V3_ADDR_TYPE_DIST 2
+#define KVM_VGIC_V3_ADDR_TYPE_REDIST 3
+
+#define KVM_VGIC_V3_DIST_SIZE SZ_64K
+#define KVM_VGIC_V3_REDIST_SIZE (2 * SZ_64K)
+
#define KVM_ARM_VCPU_POWER_OFF 0 /* CPU is started in OFF state */
#define KVM_ARM_VCPU_PSCI_0_2 1 /* CPU uses PSCI v0.2 */
@@ -166,6 +173,12 @@ struct kvm_arch_memory_slot {
#define KVM_REG_ARM_VFP_FPINST 0x1009
#define KVM_REG_ARM_VFP_FPINST2 0x100A
+/* KVM-as-firmware specific pseudo-registers */
+#define KVM_REG_ARM_FW (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
+#define KVM_REG_ARM_FW_REG(r) (KVM_REG_ARM | KVM_REG_SIZE_U64 | \
+ KVM_REG_ARM_FW | ((r) & 0xffff))
+#define KVM_REG_ARM_PSCI_VERSION KVM_REG_ARM_FW_REG(0)
+
/* Device Control API: ARM VGIC */
#define KVM_DEV_ARM_VGIC_GRP_ADDR 0
#define KVM_DEV_ARM_VGIC_GRP_DIST_REGS 1
diff --git a/tools/arch/arm64/include/uapi/asm/kvm.h b/tools/arch/arm64/include/uapi/asm/kvm.h
index 3051f86..702de7a 100644
--- a/tools/arch/arm64/include/uapi/asm/kvm.h
+++ b/tools/arch/arm64/include/uapi/asm/kvm.h
@@ -195,6 +195,12 @@ struct kvm_arch_memory_slot {
#define KVM_REG_ARM_TIMER_CNT ARM64_SYS_REG(3, 3, 14, 3, 2)
#define KVM_REG_ARM_TIMER_CVAL ARM64_SYS_REG(3, 3, 14, 0, 2)
+/* KVM-as-firmware specific pseudo-registers */
+#define KVM_REG_ARM_FW (0x0014 << KVM_REG_ARM_COPROC_SHIFT)
+#define KVM_REG_ARM_FW_REG(r) (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \
+ KVM_REG_ARM_FW | ((r) & 0xffff))
+#define KVM_REG_ARM_PSCI_VERSION KVM_REG_ARM_FW_REG(0)
+
/* Device Control API: ARM VGIC */
#define KVM_DEV_ARM_VGIC_GRP_ADDR 0
#define KVM_DEV_ARM_VGIC_GRP_DIST_REGS 1
diff --git a/tools/arch/powerpc/include/uapi/asm/kvm.h b/tools/arch/powerpc/include/uapi/asm/kvm.h
index c93cf35..0fb1326 100644
--- a/tools/arch/powerpc/include/uapi/asm/kvm.h
+++ b/tools/arch/powerpc/include/uapi/asm/kvm.h
@@ -596,6 +596,7 @@ struct kvm_get_htab_header {
#define KVM_REG_PPC_TM_VSCR (KVM_REG_PPC_TM | KVM_REG_SIZE_U32 | 0x67)
#define KVM_REG_PPC_TM_DSCR (KVM_REG_PPC_TM | KVM_REG_SIZE_U64 | 0x68)
#define KVM_REG_PPC_TM_TAR (KVM_REG_PPC_TM | KVM_REG_SIZE_U64 | 0x69)
+#define KVM_REG_PPC_TM_XER (KVM_REG_PPC_TM | KVM_REG_SIZE_U64 | 0x6a)
/* PPC64 eXternal Interrupt Controller Specification */
#define KVM_DEV_XICS_GRP_SOURCES 1 /* 64-bit source attributes */
diff --git a/tools/arch/s390/include/uapi/asm/kvm.h b/tools/arch/s390/include/uapi/asm/kvm.h
index a2ffec4..81c02e1 100644
--- a/tools/arch/s390/include/uapi/asm/kvm.h
+++ b/tools/arch/s390/include/uapi/asm/kvm.h
@@ -197,6 +197,7 @@ struct kvm_guest_debug_arch {
#define KVM_SYNC_VRS (1UL << 6)
#define KVM_SYNC_RICCB (1UL << 7)
#define KVM_SYNC_FPRS (1UL << 8)
+#define KVM_SYNC_BPBC (1UL << 10)
/* definition of registers in kvm_run */
struct kvm_sync_regs {
__u64 prefix; /* prefix register */
@@ -217,7 +218,9 @@ struct kvm_sync_regs {
};
__u8 reserved[512]; /* for future vector expansion */
__u32 fpc; /* valid on KVM_SYNC_VRS or KVM_SYNC_FPRS */
- __u8 padding[52]; /* riccb needs to be 64byte aligned */
+ __u8 bpbc : 1; /* bp mode */
+ __u8 reserved2 : 7;
+ __u8 padding1[51]; /* riccb needs to be 64byte aligned */
__u8 riccb[64]; /* runtime instrumentation controls block */
};
diff --git a/tools/arch/x86/include/asm/cpufeatures.h b/tools/arch/x86/include/asm/cpufeatures.h
index f79669a..f6d1bc9 100644
--- a/tools/arch/x86/include/asm/cpufeatures.h
+++ b/tools/arch/x86/include/asm/cpufeatures.h
@@ -12,7 +12,7 @@
/*
* Defines x86 CPU feature bits
*/
-#define NCAPINTS 18 /* N 32-bit words worth of info */
+#define NCAPINTS 19 /* N 32-bit words worth of info */
#define NBUGINTS 1 /* N 32-bit bug flags */
/*
@@ -104,7 +104,6 @@
#define X86_FEATURE_EXTD_APICID ( 3*32+26) /* has extended APICID (8 bits) */
#define X86_FEATURE_AMD_DCM ( 3*32+27) /* multi-node processor */
#define X86_FEATURE_APERFMPERF ( 3*32+28) /* APERFMPERF */
-#define X86_FEATURE_EAGER_FPU ( 3*32+29) /* "eagerfpu" Non lazy FPU restore */
#define X86_FEATURE_NONSTOP_TSC_S3 ( 3*32+30) /* TSC doesn't stop in S3 state */
/* Intel-defined CPU features, CPUID level 0x00000001 (ecx), word 4 */
@@ -189,17 +188,32 @@
#define X86_FEATURE_CPB ( 7*32+ 2) /* AMD Core Performance Boost */
#define X86_FEATURE_EPB ( 7*32+ 3) /* IA32_ENERGY_PERF_BIAS support */
+#define X86_FEATURE_INVPCID_SINGLE ( 7*32+ 4) /* Effectively INVPCID && CR4.PCIDE=1 */
#define X86_FEATURE_HW_PSTATE ( 7*32+ 8) /* AMD HW-PState */
#define X86_FEATURE_PROC_FEEDBACK ( 7*32+ 9) /* AMD ProcFeedbackInterface */
-#define X86_FEATURE_INTEL_PT ( 7*32+15) /* Intel Processor Trace */
-#define X86_FEATURE_AVX512_4VNNIW (7*32+16) /* AVX-512 Neural Network Instructions */
-#define X86_FEATURE_AVX512_4FMAPS (7*32+17) /* AVX-512 Multiply Accumulation Single precision */
+#define X86_FEATURE_RETPOLINE ( 7*32+12) /* "" Generic Retpoline mitigation for Spectre variant 2 */
+#define X86_FEATURE_RETPOLINE_AMD ( 7*32+13) /* "" AMD Retpoline mitigation for Spectre variant 2 */
+
+#define X86_FEATURE_MSR_SPEC_CTRL ( 7*32+16) /* "" MSR SPEC_CTRL is implemented */
+#define X86_FEATURE_SSBD ( 7*32+17) /* Speculative Store Bypass Disable */
+
+#define X86_FEATURE_RSB_CTXSW ( 7*32+19) /* "" Fill RSB on context switches */
/* Because the ALTERNATIVE scheme is for members of the X86_FEATURE club... */
#define X86_FEATURE_KAISER ( 7*32+31) /* CONFIG_PAGE_TABLE_ISOLATION w/o nokaiser */
+#define X86_FEATURE_USE_IBPB ( 7*32+21) /* "" Indirect Branch Prediction Barrier enabled */
+#define X86_FEATURE_USE_IBRS_FW ( 7*32+22) /* "" Use IBRS during runtime firmware calls */
+#define X86_FEATURE_SPEC_STORE_BYPASS_DISABLE ( 7*32+23) /* "" Disable Speculative Store Bypass. */
+#define X86_FEATURE_LS_CFG_SSBD ( 7*32+24) /* "" AMD SSBD implementation */
+#define X86_FEATURE_IBRS ( 7*32+25) /* Indirect Branch Restricted Speculation */
+#define X86_FEATURE_IBPB ( 7*32+26) /* Indirect Branch Prediction Barrier */
+#define X86_FEATURE_STIBP ( 7*32+27) /* Single Thread Indirect Branch Predictors */
+#define X86_FEATURE_ZEN ( 7*32+28) /* "" CPU is AMD family 0x17 (Zen) */
+#define X86_FEATURE_L1TF_PTEINV ( 7*32+29) /* "" L1TF workaround PTE inversion */
+
/* Virtualization flags: Linux defined, word 8 */
#define X86_FEATURE_TPR_SHADOW ( 8*32+ 0) /* Intel TPR Shadow */
#define X86_FEATURE_VNMI ( 8*32+ 1) /* Intel Virtual NMI */
@@ -231,6 +245,7 @@
#define X86_FEATURE_SMAP ( 9*32+20) /* Supervisor Mode Access Prevention */
#define X86_FEATURE_CLFLUSHOPT ( 9*32+23) /* CLFLUSHOPT instruction */
#define X86_FEATURE_CLWB ( 9*32+24) /* CLWB instruction */
+#define X86_FEATURE_INTEL_PT ( 9*32+25) /* Intel Processor Trace */
#define X86_FEATURE_AVX512PF ( 9*32+26) /* AVX-512 Prefetch */
#define X86_FEATURE_AVX512ER ( 9*32+27) /* AVX-512 Exponential and Reciprocal */
#define X86_FEATURE_AVX512CD ( 9*32+28) /* AVX-512 Conflict Detection */
@@ -255,6 +270,10 @@
/* AMD-defined CPU features, CPUID level 0x80000008 (ebx), word 13 */
#define X86_FEATURE_CLZERO (13*32+0) /* CLZERO instruction */
#define X86_FEATURE_IRPERF (13*32+1) /* Instructions Retired Count */
+#define X86_FEATURE_AMD_IBPB (13*32+12) /* Indirect Branch Prediction Barrier */
+#define X86_FEATURE_AMD_IBRS (13*32+14) /* Indirect Branch Restricted Speculation */
+#define X86_FEATURE_AMD_STIBP (13*32+15) /* Single Thread Indirect Branch Predictors */
+#define X86_FEATURE_VIRT_SSBD (13*32+25) /* Virtualized Speculative Store Bypass Disable */
/* Thermal and Power Management Leaf, CPUID level 0x00000006 (eax), word 14 */
#define X86_FEATURE_DTHERM (14*32+ 0) /* Digital Thermal Sensor */
@@ -290,6 +309,17 @@
#define X86_FEATURE_SUCCOR (17*32+1) /* Uncorrectable error containment and recovery */
#define X86_FEATURE_SMCA (17*32+3) /* Scalable MCA */
+
+/* Intel-defined CPU features, CPUID level 0x00000007:0 (EDX), word 18 */
+#define X86_FEATURE_AVX512_4VNNIW (18*32+ 2) /* AVX-512 Neural Network Instructions */
+#define X86_FEATURE_AVX512_4FMAPS (18*32+ 3) /* AVX-512 Multiply Accumulation Single precision */
+#define X86_FEATURE_PCONFIG (18*32+18) /* Intel PCONFIG */
+#define X86_FEATURE_SPEC_CTRL (18*32+26) /* "" Speculation Control (IBRS + IBPB) */
+#define X86_FEATURE_INTEL_STIBP (18*32+27) /* "" Single Thread Indirect Branch Predictors */
+#define X86_FEATURE_FLUSH_L1D (18*32+28) /* Flush L1D cache */
+#define X86_FEATURE_ARCH_CAPABILITIES (18*32+29) /* IA32_ARCH_CAPABILITIES MSR (Intel) */
+#define X86_FEATURE_SPEC_CTRL_SSBD (18*32+31) /* "" Speculative Store Bypass Disable */
+
/*
* BUG word(s)
*/
@@ -314,4 +344,11 @@
#define X86_BUG_NULL_SEG X86_BUG(10) /* Nulling a selector preserves the base */
#define X86_BUG_SWAPGS_FENCE X86_BUG(11) /* SWAPGS without input dep on GS */
#define X86_BUG_MONITOR X86_BUG(12) /* IPI required to wake up remote CPU */
+#define X86_BUG_AMD_E400 X86_BUG(13) /* CPU is among the affected by Erratum 400 */
+#define X86_BUG_CPU_MELTDOWN X86_BUG(14) /* CPU is affected by meltdown attack and needs kernel page table isolation */
+#define X86_BUG_SPECTRE_V1 X86_BUG(15) /* CPU is affected by Spectre variant 1 attack with conditional branches */
+#define X86_BUG_SPECTRE_V2 X86_BUG(16) /* CPU is affected by Spectre variant 2 attack with indirect branches */
+#define X86_BUG_SPEC_STORE_BYPASS X86_BUG(17) /* CPU is affected by speculative store bypass attack */
+#define X86_BUG_L1TF X86_BUG(18) /* CPU is affected by L1 Terminal Fault */
+
#endif /* _ASM_X86_CPUFEATURES_H */
diff --git a/tools/arch/x86/include/asm/disabled-features.h b/tools/arch/x86/include/asm/disabled-features.h
index 85599ad..1f8cca4 100644
--- a/tools/arch/x86/include/asm/disabled-features.h
+++ b/tools/arch/x86/include/asm/disabled-features.h
@@ -21,11 +21,13 @@
# define DISABLE_K6_MTRR (1<<(X86_FEATURE_K6_MTRR & 31))
# define DISABLE_CYRIX_ARR (1<<(X86_FEATURE_CYRIX_ARR & 31))
# define DISABLE_CENTAUR_MCR (1<<(X86_FEATURE_CENTAUR_MCR & 31))
+# define DISABLE_PCID 0
#else
# define DISABLE_VME 0
# define DISABLE_K6_MTRR 0
# define DISABLE_CYRIX_ARR 0
# define DISABLE_CENTAUR_MCR 0
+# define DISABLE_PCID (1<<(X86_FEATURE_PCID & 31))
#endif /* CONFIG_X86_64 */
#ifdef CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS
@@ -43,7 +45,7 @@
#define DISABLED_MASK1 0
#define DISABLED_MASK2 0
#define DISABLED_MASK3 (DISABLE_CYRIX_ARR|DISABLE_CENTAUR_MCR|DISABLE_K6_MTRR)
-#define DISABLED_MASK4 0
+#define DISABLED_MASK4 (DISABLE_PCID)
#define DISABLED_MASK5 0
#define DISABLED_MASK6 0
#define DISABLED_MASK7 0
@@ -57,6 +59,7 @@
#define DISABLED_MASK15 0
#define DISABLED_MASK16 (DISABLE_PKU|DISABLE_OSPKE)
#define DISABLED_MASK17 0
-#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 18)
+#define DISABLED_MASK18 0
+#define DISABLED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
#endif /* _ASM_X86_DISABLED_FEATURES_H */
diff --git a/tools/arch/x86/include/asm/required-features.h b/tools/arch/x86/include/asm/required-features.h
index fac9a5c..6847d85 100644
--- a/tools/arch/x86/include/asm/required-features.h
+++ b/tools/arch/x86/include/asm/required-features.h
@@ -100,6 +100,7 @@
#define REQUIRED_MASK15 0
#define REQUIRED_MASK16 0
#define REQUIRED_MASK17 0
-#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 18)
+#define REQUIRED_MASK18 0
+#define REQUIRED_MASK_CHECK BUILD_BUG_ON_ZERO(NCAPINTS != 19)
#endif /* _ASM_X86_REQUIRED_FEATURES_H */
diff --git a/tools/build/Build.include b/tools/build/Build.include
index 1dcb95e..ab02f8d 100644
--- a/tools/build/Build.include
+++ b/tools/build/Build.include
@@ -12,6 +12,7 @@
# Convenient variables
comma := ,
squote := '
+pound := \#
###
# Name of target with a '.' as filename prefix. foo/bar.o => foo/.bar.o
@@ -43,11 +44,11 @@
###
# Replace >$< with >$$< to preserve $ when reloading the .cmd file
# (needed for make)
-# Replace >#< with >\#< to avoid starting a comment in the .cmd file
+# Replace >#< with >$(pound)< to avoid starting a comment in the .cmd file
# (needed for make)
# Replace >'< with >'\''< to be able to enclose the whole string in '...'
# (needed for the shell)
-make-cmd = $(call escsq,$(subst \#,\\\#,$(subst $$,$$$$,$(cmd_$(1)))))
+make-cmd = $(call escsq,$(subst $(pound),$$(pound),$(subst $$,$$$$,$(cmd_$(1)))))
###
# Find any prerequisites that is newer than target or that does not exist.
@@ -62,8 +63,8 @@
$(fixdep) $(depfile) $@ '$(make-cmd)' > $(dot-target).tmp; \
rm -f $(depfile); \
mv -f $(dot-target).tmp $(dot-target).cmd, \
- printf '\# cannot find fixdep (%s)\n' $(fixdep) > $(dot-target).cmd; \
- printf '\# using basic dep data\n\n' >> $(dot-target).cmd; \
+ printf '$(pound) cannot find fixdep (%s)\n' $(fixdep) > $(dot-target).cmd; \
+ printf '$(pound) using basic dep data\n\n' >> $(dot-target).cmd; \
cat $(depfile) >> $(dot-target).cmd; \
printf '%s\n' 'cmd_$@ := $(make-cmd)' >> $(dot-target).cmd)
diff --git a/tools/build/Makefile b/tools/build/Makefile
index 8332959..20d9ae1 100644
--- a/tools/build/Makefile
+++ b/tools/build/Makefile
@@ -42,7 +42,7 @@
$(Q)$(MAKE) $(build)=fixdep
$(OUTPUT)fixdep: $(OUTPUT)fixdep-in.o
- $(QUIET_LINK)$(HOSTCC) $(LDFLAGS) -o $@ $<
+ $(QUIET_LINK)$(HOSTCC) $(HOSTLDFLAGS) -o $@ $<
FORCE:
diff --git a/tools/hv/hv_kvp_daemon.c b/tools/hv/hv_kvp_daemon.c
index 60a94b3..1774800 100644
--- a/tools/hv/hv_kvp_daemon.c
+++ b/tools/hv/hv_kvp_daemon.c
@@ -286,7 +286,7 @@ static int kvp_key_delete(int pool, const __u8 *key, int key_size)
* Found a match; just move the remaining
* entries up.
*/
- if (i == num_records) {
+ if (i == (num_records - 1)) {
kvp_file_info[pool].num_records--;
kvp_update_file(pool);
return 0;
diff --git a/tools/include/asm-generic/bitops.h b/tools/include/asm-generic/bitops.h
index 653d1ba..0304600 100644
--- a/tools/include/asm-generic/bitops.h
+++ b/tools/include/asm-generic/bitops.h
@@ -13,6 +13,7 @@
*/
#include <asm-generic/bitops/__ffs.h>
+#include <asm-generic/bitops/__ffz.h>
#include <asm-generic/bitops/fls.h>
#include <asm-generic/bitops/__fls.h>
#include <asm-generic/bitops/fls64.h>
diff --git a/tools/include/asm-generic/bitops/__ffz.h b/tools/include/asm-generic/bitops/__ffz.h
new file mode 100644
index 0000000..6744bd4
--- /dev/null
+++ b/tools/include/asm-generic/bitops/__ffz.h
@@ -0,0 +1,12 @@
+#ifndef _ASM_GENERIC_BITOPS_FFZ_H_
+#define _ASM_GENERIC_BITOPS_FFZ_H_
+
+/*
+ * ffz - find first zero in word.
+ * @word: The word to search
+ *
+ * Undefined if no zero exists, so code should check against ~0UL first.
+ */
+#define ffz(x) __ffs(~(x))
+
+#endif /* _ASM_GENERIC_BITOPS_FFZ_H_ */
diff --git a/tools/include/asm-generic/bitops/find.h b/tools/include/asm-generic/bitops/find.h
index 31f5154..5538ecd 100644
--- a/tools/include/asm-generic/bitops/find.h
+++ b/tools/include/asm-generic/bitops/find.h
@@ -15,6 +15,21 @@ extern unsigned long find_next_bit(const unsigned long *addr, unsigned long
size, unsigned long offset);
#endif
+#ifndef find_next_zero_bit
+
+/**
+ * find_next_zero_bit - find the next cleared bit in a memory region
+ * @addr: The address to base the search on
+ * @offset: The bitnumber to start searching at
+ * @size: The bitmap size in bits
+ *
+ * Returns the bit number of the next zero bit
+ * If no bits are zero, returns @size.
+ */
+unsigned long find_next_zero_bit(const unsigned long *addr, unsigned long size,
+ unsigned long offset);
+#endif
+
#ifndef find_first_bit
/**
@@ -30,4 +45,17 @@ extern unsigned long find_first_bit(const unsigned long *addr,
#endif /* find_first_bit */
+#ifndef find_first_zero_bit
+
+/**
+ * find_first_zero_bit - find the first cleared bit in a memory region
+ * @addr: The address to start the search at
+ * @size: The maximum number of bits to search
+ *
+ * Returns the bit number of the first cleared bit.
+ * If no bits are zero, returns @size.
+ */
+unsigned long find_first_zero_bit(const unsigned long *addr, unsigned long size);
+#endif
+
#endif /*_TOOLS_LINUX_ASM_GENERIC_BITOPS_FIND_H_ */
diff --git a/tools/include/linux/atomic.h b/tools/include/linux/atomic.h
index 4e3d3d1..9f21fc2 100644
--- a/tools/include/linux/atomic.h
+++ b/tools/include/linux/atomic.h
@@ -3,4 +3,10 @@
#include <asm/atomic.h>
+/* atomic_cmpxchg_relaxed */
+#ifndef atomic_cmpxchg_relaxed
+#define atomic_cmpxchg_relaxed atomic_cmpxchg
+#define atomic_cmpxchg_release atomic_cmpxchg
+#endif /* atomic_cmpxchg_relaxed */
+
#endif /* __TOOLS_LINUX_ATOMIC_H */
diff --git a/tools/include/linux/bitmap.h b/tools/include/linux/bitmap.h
index 43c1c50..eef41d5 100644
--- a/tools/include/linux/bitmap.h
+++ b/tools/include/linux/bitmap.h
@@ -35,6 +35,32 @@ static inline void bitmap_zero(unsigned long *dst, int nbits)
}
}
+static inline void bitmap_fill(unsigned long *dst, unsigned int nbits)
+{
+ unsigned int nlongs = BITS_TO_LONGS(nbits);
+ if (!small_const_nbits(nbits)) {
+ unsigned int len = (nlongs - 1) * sizeof(unsigned long);
+ memset(dst, 0xff, len);
+ }
+ dst[nlongs - 1] = BITMAP_LAST_WORD_MASK(nbits);
+}
+
+static inline int bitmap_empty(const unsigned long *src, unsigned nbits)
+{
+ if (small_const_nbits(nbits))
+ return ! (*src & BITMAP_LAST_WORD_MASK(nbits));
+
+ return find_first_bit(src, nbits) == nbits;
+}
+
+static inline int bitmap_full(const unsigned long *src, unsigned int nbits)
+{
+ if (small_const_nbits(nbits))
+ return ! (~(*src) & BITMAP_LAST_WORD_MASK(nbits));
+
+ return find_first_zero_bit(src, nbits) == nbits;
+}
+
static inline int bitmap_weight(const unsigned long *src, int nbits)
{
if (small_const_nbits(nbits))
diff --git a/tools/include/linux/bitops.h b/tools/include/linux/bitops.h
index 49c929a..fc446343 100644
--- a/tools/include/linux/bitops.h
+++ b/tools/include/linux/bitops.h
@@ -39,6 +39,11 @@ extern unsigned long __sw_hweight64(__u64 w);
(bit) < (size); \
(bit) = find_next_bit((addr), (size), (bit) + 1))
+#define for_each_clear_bit(bit, addr, size) \
+ for ((bit) = find_first_zero_bit((addr), (size)); \
+ (bit) < (size); \
+ (bit) = find_next_zero_bit((addr), (size), (bit) + 1))
+
/* same as for_each_set_bit() but use bit as value to start with */
#define for_each_set_bit_from(bit, addr, size) \
for ((bit) = find_next_bit((addr), (size), (bit)); \
diff --git a/tools/include/linux/bug.h b/tools/include/linux/bug.h
new file mode 100644
index 0000000..8e4a4f4
--- /dev/null
+++ b/tools/include/linux/bug.h
@@ -0,0 +1,10 @@
+#ifndef _TOOLS_PERF_LINUX_BUG_H
+#define _TOOLS_PERF_LINUX_BUG_H
+
+/* Force a compilation error if condition is true, but also produce a
+ result (of value 0 and type size_t), so the expression can be used
+ e.g. in a structure initializer (or where-ever else comma expressions
+ aren't permitted). */
+#define BUILD_BUG_ON_ZERO(e) (sizeof(struct { int:-!!(e); }))
+
+#endif /* _TOOLS_PERF_LINUX_BUG_H */
diff --git a/tools/include/linux/compiler-gcc.h b/tools/include/linux/compiler-gcc.h
new file mode 100644
index 0000000..825d44f
--- /dev/null
+++ b/tools/include/linux/compiler-gcc.h
@@ -0,0 +1,21 @@
+#ifndef _TOOLS_LINUX_COMPILER_H_
+#error "Please don't include <linux/compiler-gcc.h> directly, include <linux/compiler.h> instead."
+#endif
+
+/*
+ * Common definitions for all gcc versions go here.
+ */
+#define GCC_VERSION (__GNUC__ * 10000 \
+ + __GNUC_MINOR__ * 100 \
+ + __GNUC_PATCHLEVEL__)
+
+#if GCC_VERSION >= 70000 && !defined(__CHECKER__)
+# define __fallthrough __attribute__ ((fallthrough))
+#endif
+
+#if GCC_VERSION >= 40300
+# define __compiletime_error(message) __attribute__((error(message)))
+#endif /* GCC_VERSION >= 40300 */
+
+/* &a[0] degrades to a pointer: a different type from an array */
+#define __must_be_array(a) BUILD_BUG_ON_ZERO(__same_type((a), &(a)[0]))
diff --git a/tools/include/linux/compiler.h b/tools/include/linux/compiler.h
index d94179f..23299d7 100644
--- a/tools/include/linux/compiler.h
+++ b/tools/include/linux/compiler.h
@@ -1,6 +1,14 @@
#ifndef _TOOLS_LINUX_COMPILER_H_
#define _TOOLS_LINUX_COMPILER_H_
+#ifdef __GNUC__
+#include <linux/compiler-gcc.h>
+#endif
+
+#ifndef __compiletime_error
+# define __compiletime_error(message)
+#endif
+
/* Optimization barrier */
/* The "volatile" is due to gcc bugs */
#define barrier() __asm__ __volatile__("": : :"memory")
@@ -9,6 +17,11 @@
# define __always_inline inline __attribute__((always_inline))
#endif
+/* Are two types/vars the same type (ignoring qualifiers)? */
+#ifndef __same_type
+# define __same_type(a, b) __builtin_types_compatible_p(typeof(a), typeof(b))
+#endif
+
#ifdef __ANDROID__
/*
* FIXME: Big hammer to get rid of tons of:
@@ -21,6 +34,8 @@
#endif
#define __user
+#define __rcu
+#define __read_mostly
#ifndef __attribute_const__
# define __attribute_const__
@@ -50,6 +65,8 @@
# define unlikely(x) __builtin_expect(!!(x), 0)
#endif
+#define uninitialized_var(x) x = *(&(x))
+
#define ACCESS_ONCE(x) (*(volatile typeof(x) *)&(x))
#include <linux/types.h>
@@ -128,11 +145,7 @@ static __always_inline void __write_once_size(volatile void *p, void *res, int s
#ifndef __fallthrough
-# if defined(__GNUC__) && __GNUC__ >= 7
-# define __fallthrough __attribute__ ((fallthrough))
-# else
-# define __fallthrough
-# endif
+# define __fallthrough
#endif
#endif /* _TOOLS_LINUX_COMPILER_H */
diff --git a/tools/include/linux/hashtable.h b/tools/include/linux/hashtable.h
index c65cc0a..251eabf 100644
--- a/tools/include/linux/hashtable.h
+++ b/tools/include/linux/hashtable.h
@@ -13,10 +13,6 @@
#include <linux/hash.h>
#include <linux/log2.h>
-#ifndef ARRAY_SIZE
-#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
-#endif
-
#define DEFINE_HASHTABLE(name, bits) \
struct hlist_head name[1 << (bits)] = \
{ [0 ... ((1 << (bits)) - 1)] = HLIST_HEAD_INIT }
diff --git a/tools/include/linux/kernel.h b/tools/include/linux/kernel.h
index 28607db..73ccc481 100644
--- a/tools/include/linux/kernel.h
+++ b/tools/include/linux/kernel.h
@@ -4,6 +4,11 @@
#include <stdarg.h>
#include <stddef.h>
#include <assert.h>
+#include <linux/compiler.h>
+
+#ifndef UINT_MAX
+#define UINT_MAX (~0U)
+#endif
#define DIV_ROUND_UP(n,d) (((n) + (d) - 1) / (d))
@@ -72,6 +77,8 @@
int vscnprintf(char *buf, size_t size, const char *fmt, va_list args);
int scnprintf(char * buf, size_t size, const char * fmt, ...);
+#define ARRAY_SIZE(arr) (sizeof(arr) / sizeof((arr)[0]) + __must_be_array(arr))
+
/*
* This looks more complex than it should be. But we need to
* get the type for the ~ right in round_down (it needs to be
diff --git a/tools/include/linux/log2.h b/tools/include/linux/log2.h
index d5677d3..0325cef 100644
--- a/tools/include/linux/log2.h
+++ b/tools/include/linux/log2.h
@@ -12,6 +12,9 @@
#ifndef _TOOLS_LINUX_LOG2_H
#define _TOOLS_LINUX_LOG2_H
+#include <linux/bitops.h>
+#include <linux/types.h>
+
/*
* non-constant log of base 2 calculators
* - the arch may override these in asm/bitops.h if they can be implemented
diff --git a/tools/include/linux/refcount.h b/tools/include/linux/refcount.h
new file mode 100644
index 0000000..a0177c1
--- /dev/null
+++ b/tools/include/linux/refcount.h
@@ -0,0 +1,151 @@
+#ifndef _TOOLS_LINUX_REFCOUNT_H
+#define _TOOLS_LINUX_REFCOUNT_H
+
+/*
+ * Variant of atomic_t specialized for reference counts.
+ *
+ * The interface matches the atomic_t interface (to aid in porting) but only
+ * provides the few functions one should use for reference counting.
+ *
+ * It differs in that the counter saturates at UINT_MAX and will not move once
+ * there. This avoids wrapping the counter and causing 'spurious'
+ * use-after-free issues.
+ *
+ * Memory ordering rules are slightly relaxed wrt regular atomic_t functions
+ * and provide only what is strictly required for refcounts.
+ *
+ * The increments are fully relaxed; these will not provide ordering. The
+ * rationale is that whatever is used to obtain the object we're increasing the
+ * reference count on will provide the ordering. For locked data structures,
+ * its the lock acquire, for RCU/lockless data structures its the dependent
+ * load.
+ *
+ * Do note that inc_not_zero() provides a control dependency which will order
+ * future stores against the inc, this ensures we'll never modify the object
+ * if we did not in fact acquire a reference.
+ *
+ * The decrements will provide release order, such that all the prior loads and
+ * stores will be issued before, it also provides a control dependency, which
+ * will order us against the subsequent free().
+ *
+ * The control dependency is against the load of the cmpxchg (ll/sc) that
+ * succeeded. This means the stores aren't fully ordered, but this is fine
+ * because the 1->0 transition indicates no concurrency.
+ *
+ * Note that the allocator is responsible for ordering things between free()
+ * and alloc().
+ *
+ */
+
+#include <linux/atomic.h>
+#include <linux/kernel.h>
+
+#ifdef NDEBUG
+#define REFCOUNT_WARN(cond, str) (void)(cond)
+#define __refcount_check
+#else
+#define REFCOUNT_WARN(cond, str) BUG_ON(cond)
+#define __refcount_check __must_check
+#endif
+
+typedef struct refcount_struct {
+ atomic_t refs;
+} refcount_t;
+
+#define REFCOUNT_INIT(n) { .refs = ATOMIC_INIT(n), }
+
+static inline void refcount_set(refcount_t *r, unsigned int n)
+{
+ atomic_set(&r->refs, n);
+}
+
+static inline unsigned int refcount_read(const refcount_t *r)
+{
+ return atomic_read(&r->refs);
+}
+
+/*
+ * Similar to atomic_inc_not_zero(), will saturate at UINT_MAX and WARN.
+ *
+ * Provides no memory ordering, it is assumed the caller has guaranteed the
+ * object memory to be stable (RCU, etc.). It does provide a control dependency
+ * and thereby orders future stores. See the comment on top.
+ */
+static inline __refcount_check
+bool refcount_inc_not_zero(refcount_t *r)
+{
+ unsigned int old, new, val = atomic_read(&r->refs);
+
+ for (;;) {
+ new = val + 1;
+
+ if (!val)
+ return false;
+
+ if (unlikely(!new))
+ return true;
+
+ old = atomic_cmpxchg_relaxed(&r->refs, val, new);
+ if (old == val)
+ break;
+
+ val = old;
+ }
+
+ REFCOUNT_WARN(new == UINT_MAX, "refcount_t: saturated; leaking memory.\n");
+
+ return true;
+}
+
+/*
+ * Similar to atomic_inc(), will saturate at UINT_MAX and WARN.
+ *
+ * Provides no memory ordering, it is assumed the caller already has a
+ * reference on the object, will WARN when this is not so.
+ */
+static inline void refcount_inc(refcount_t *r)
+{
+ REFCOUNT_WARN(!refcount_inc_not_zero(r), "refcount_t: increment on 0; use-after-free.\n");
+}
+
+/*
+ * Similar to atomic_dec_and_test(), it will WARN on underflow and fail to
+ * decrement when saturated at UINT_MAX.
+ *
+ * Provides release memory ordering, such that prior loads and stores are done
+ * before, and provides a control dependency such that free() must come after.
+ * See the comment on top.
+ */
+static inline __refcount_check
+bool refcount_sub_and_test(unsigned int i, refcount_t *r)
+{
+ unsigned int old, new, val = atomic_read(&r->refs);
+
+ for (;;) {
+ if (unlikely(val == UINT_MAX))
+ return false;
+
+ new = val - i;
+ if (new > val) {
+ REFCOUNT_WARN(new > val, "refcount_t: underflow; use-after-free.\n");
+ return false;
+ }
+
+ old = atomic_cmpxchg_release(&r->refs, val, new);
+ if (old == val)
+ break;
+
+ val = old;
+ }
+
+ return !new;
+}
+
+static inline __refcount_check
+bool refcount_dec_and_test(refcount_t *r)
+{
+ return refcount_sub_and_test(1, r);
+}
+
+
+#endif /* _ATOMIC_LINUX_REFCOUNT_H */
diff --git a/tools/include/linux/spinlock.h b/tools/include/linux/spinlock.h
new file mode 100644
index 0000000..58397dc
--- /dev/null
+++ b/tools/include/linux/spinlock.h
@@ -0,0 +1,5 @@
+#define spinlock_t pthread_mutex_t
+#define DEFINE_SPINLOCK(x) pthread_mutex_t x = PTHREAD_MUTEX_INITIALIZER;
+
+#define spin_lock_irqsave(x, f) (void)f, pthread_mutex_lock(x)
+#define spin_unlock_irqrestore(x, f) (void)f, pthread_mutex_unlock(x)
diff --git a/tools/include/linux/types.h b/tools/include/linux/types.h
index 8ebf627..77a28a2 100644
--- a/tools/include/linux/types.h
+++ b/tools/include/linux/types.h
@@ -7,6 +7,7 @@
#define __SANE_USERSPACE_TYPES__ /* For PPC64, to get LL64 types */
#include <asm/types.h>
+#include <asm/posix_types.h>
struct page;
struct kmem_cache;
@@ -42,11 +43,7 @@ typedef __s8 s8;
#else
#define __bitwise__
#endif
-#ifdef __CHECK_ENDIAN__
#define __bitwise __bitwise__
-#else
-#define __bitwise
-#endif
#define __force
#define __user
diff --git a/tools/include/uapi/asm-generic/mman-common.h b/tools/include/uapi/asm-generic/mman-common.h
index 5827438..8c27db0 100644
--- a/tools/include/uapi/asm-generic/mman-common.h
+++ b/tools/include/uapi/asm-generic/mman-common.h
@@ -72,4 +72,9 @@
#define MAP_HUGE_SHIFT 26
#define MAP_HUGE_MASK 0x3f
+#define PKEY_DISABLE_ACCESS 0x1
+#define PKEY_DISABLE_WRITE 0x2
+#define PKEY_ACCESS_MASK (PKEY_DISABLE_ACCESS |\
+ PKEY_DISABLE_WRITE)
+
#endif /* __ASM_GENERIC_MMAN_COMMON_H */
diff --git a/tools/include/uapi/linux/bpf.h b/tools/include/uapi/linux/bpf.h
index 42dfbeb..d560fd8 100644
--- a/tools/include/uapi/linux/bpf.h
+++ b/tools/include/uapi/linux/bpf.h
@@ -95,6 +95,7 @@ enum bpf_prog_type {
BPF_PROG_TYPE_SCHED_ACT,
BPF_PROG_TYPE_TRACEPOINT,
BPF_PROG_TYPE_XDP,
+ BPF_PROG_TYPE_PERF_EVENT,
};
#define BPF_PSEUDO_MAP_FD 1
@@ -376,6 +377,17 @@ enum bpf_func_id {
BPF_FUNC_probe_write_user,
/**
+ * bpf_current_task_under_cgroup(map, index) - Check cgroup2 membership of current task
+ * @map: pointer to bpf_map in BPF_MAP_TYPE_CGROUP_ARRAY type
+ * @index: index of the cgroup in the bpf_map
+ * Return:
+ * == 0 current failed the cgroup2 descendant test
+ * == 1 current succeeded the cgroup2 descendant test
+ * < 0 error
+ */
+ BPF_FUNC_current_task_under_cgroup,
+
+ /**
* int bpf_skb_change_tail(skb, len, flags)
* The helper will resize the skb to the given new size, to be used f.e.
* with control messages.
diff --git a/tools/include/uapi/linux/fcntl.h b/tools/include/uapi/linux/fcntl.h
new file mode 100644
index 0000000..beed138
--- /dev/null
+++ b/tools/include/uapi/linux/fcntl.h
@@ -0,0 +1,67 @@
+#ifndef _UAPI_LINUX_FCNTL_H
+#define _UAPI_LINUX_FCNTL_H
+
+#include <asm/fcntl.h>
+
+#define F_SETLEASE (F_LINUX_SPECIFIC_BASE + 0)
+#define F_GETLEASE (F_LINUX_SPECIFIC_BASE + 1)
+
+/*
+ * Cancel a blocking posix lock; internal use only until we expose an
+ * asynchronous lock api to userspace:
+ */
+#define F_CANCELLK (F_LINUX_SPECIFIC_BASE + 5)
+
+/* Create a file descriptor with FD_CLOEXEC set. */
+#define F_DUPFD_CLOEXEC (F_LINUX_SPECIFIC_BASE + 6)
+
+/*
+ * Request nofications on a directory.
+ * See below for events that may be notified.
+ */
+#define F_NOTIFY (F_LINUX_SPECIFIC_BASE+2)
+
+/*
+ * Set and get of pipe page size array
+ */
+#define F_SETPIPE_SZ (F_LINUX_SPECIFIC_BASE + 7)
+#define F_GETPIPE_SZ (F_LINUX_SPECIFIC_BASE + 8)
+
+/*
+ * Set/Get seals
+ */
+#define F_ADD_SEALS (F_LINUX_SPECIFIC_BASE + 9)
+#define F_GET_SEALS (F_LINUX_SPECIFIC_BASE + 10)
+
+/*
+ * Types of seals
+ */
+#define F_SEAL_SEAL 0x0001 /* prevent further seals from being set */
+#define F_SEAL_SHRINK 0x0002 /* prevent file from shrinking */
+#define F_SEAL_GROW 0x0004 /* prevent file from growing */
+#define F_SEAL_WRITE 0x0008 /* prevent writes */
+/* (1U << 31) is reserved for signed error codes */
+
+/*
+ * Types of directory notifications that may be requested.
+ */
+#define DN_ACCESS 0x00000001 /* File accessed */
+#define DN_MODIFY 0x00000002 /* File modified */
+#define DN_CREATE 0x00000004 /* File created */
+#define DN_DELETE 0x00000008 /* File removed */
+#define DN_RENAME 0x00000010 /* File renamed */
+#define DN_ATTRIB 0x00000020 /* File changed attibutes */
+#define DN_MULTISHOT 0x80000000 /* Don't remove notifier */
+
+#define AT_FDCWD -100 /* Special value used to indicate
+ openat should use the current
+ working directory. */
+#define AT_SYMLINK_NOFOLLOW 0x100 /* Do not follow symbolic links. */
+#define AT_REMOVEDIR 0x200 /* Remove directory instead of
+ unlinking file. */
+#define AT_SYMLINK_FOLLOW 0x400 /* Follow symbolic links. */
+#define AT_NO_AUTOMOUNT 0x800 /* Suppress terminal automount traversal */
+#define AT_EMPTY_PATH 0x1000 /* Allow empty relative pathname */
+
+
+#endif /* _UAPI_LINUX_FCNTL_H */
diff --git a/tools/include/uapi/linux/stat.h b/tools/include/uapi/linux/stat.h
new file mode 100644
index 0000000..7fec7e3
--- /dev/null
+++ b/tools/include/uapi/linux/stat.h
@@ -0,0 +1,45 @@
+#ifndef _UAPI_LINUX_STAT_H
+#define _UAPI_LINUX_STAT_H
+
+
+#if defined(__KERNEL__) || !defined(__GLIBC__) || (__GLIBC__ < 2)
+
+#define S_IFMT 00170000
+#define S_IFSOCK 0140000
+#define S_IFLNK 0120000
+#define S_IFREG 0100000
+#define S_IFBLK 0060000
+#define S_IFDIR 0040000
+#define S_IFCHR 0020000
+#define S_IFIFO 0010000
+#define S_ISUID 0004000
+#define S_ISGID 0002000
+#define S_ISVTX 0001000
+
+#define S_ISLNK(m) (((m) & S_IFMT) == S_IFLNK)
+#define S_ISREG(m) (((m) & S_IFMT) == S_IFREG)
+#define S_ISDIR(m) (((m) & S_IFMT) == S_IFDIR)
+#define S_ISCHR(m) (((m) & S_IFMT) == S_IFCHR)
+#define S_ISBLK(m) (((m) & S_IFMT) == S_IFBLK)
+#define S_ISFIFO(m) (((m) & S_IFMT) == S_IFIFO)
+#define S_ISSOCK(m) (((m) & S_IFMT) == S_IFSOCK)
+
+#define S_IRWXU 00700
+#define S_IRUSR 00400
+#define S_IWUSR 00200
+#define S_IXUSR 00100
+
+#define S_IRWXG 00070
+#define S_IRGRP 00040
+#define S_IWGRP 00020
+#define S_IXGRP 00010
+
+#define S_IRWXO 00007
+#define S_IROTH 00004
+#define S_IWOTH 00002
+#define S_IXOTH 00001
+
+#endif
+
+
+#endif /* _UAPI_LINUX_STAT_H */
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index b699aea..7788cfb 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -590,6 +590,24 @@ bpf_object__init_maps_name(struct bpf_object *obj)
return 0;
}
+static bool section_have_execinstr(struct bpf_object *obj, int idx)
+{
+ Elf_Scn *scn;
+ GElf_Shdr sh;
+
+ scn = elf_getscn(obj->efile.elf, idx);
+ if (!scn)
+ return false;
+
+ if (gelf_getshdr(scn, &sh) != &sh)
+ return false;
+
+ if (sh.sh_flags & SHF_EXECINSTR)
+ return true;
+
+ return false;
+}
+
static int bpf_object__elf_collect(struct bpf_object *obj)
{
Elf *elf = obj->efile.elf;
@@ -673,6 +691,14 @@ static int bpf_object__elf_collect(struct bpf_object *obj)
} else if (sh.sh_type == SHT_REL) {
void *reloc = obj->efile.reloc;
int nr_reloc = obj->efile.nr_reloc + 1;
+ int sec = sh.sh_info; /* points to other section */
+
+ /* Only do relo for section with exec instructions */
+ if (!section_have_execinstr(obj, sec)) {
+ pr_debug("skip relo %s(%d) for section(%d)\n",
+ name, idx, sec);
+ continue;
+ }
reloc = realloc(reloc,
sizeof(*obj->efile.reloc) * nr_reloc);
diff --git a/tools/lib/find_bit.c b/tools/lib/find_bit.c
index 9122a9e..6d8b8f2 100644
--- a/tools/lib/find_bit.c
+++ b/tools/lib/find_bit.c
@@ -82,3 +82,28 @@ unsigned long find_first_bit(const unsigned long *addr, unsigned long size)
return size;
}
#endif
+
+#ifndef find_first_zero_bit
+/*
+ * Find the first cleared bit in a memory region.
+ */
+unsigned long find_first_zero_bit(const unsigned long *addr, unsigned long size)
+{
+ unsigned long idx;
+
+ for (idx = 0; idx * BITS_PER_LONG < size; idx++) {
+ if (addr[idx] != ~0UL)
+ return min(idx * BITS_PER_LONG + ffz(addr[idx]), size);
+ }
+
+ return size;
+}
+#endif
+
+#ifndef find_next_zero_bit
+unsigned long find_next_zero_bit(const unsigned long *addr, unsigned long size,
+ unsigned long offset)
+{
+ return _find_next_bit(addr, size, offset, ~0UL);
+}
+#endif
diff --git a/tools/lib/str_error_r.c b/tools/lib/str_error_r.c
index 503ae07..9ab2d0ad 100644
--- a/tools/lib/str_error_r.c
+++ b/tools/lib/str_error_r.c
@@ -21,6 +21,6 @@ char *str_error_r(int errnum, char *buf, size_t buflen)
{
int err = strerror_r(errnum, buf, buflen);
if (err)
- snprintf(buf, buflen, "INTERNAL ERROR: strerror_r(%d, %p, %zd)=%d", errnum, buf, buflen, err);
+ snprintf(buf, buflen, "INTERNAL ERROR: strerror_r(%d, [buf], %zd)=%d", errnum, buflen, err);
return buf;
}
diff --git a/tools/lib/subcmd/pager.c b/tools/lib/subcmd/pager.c
index 6518bea..68af60f 100644
--- a/tools/lib/subcmd/pager.c
+++ b/tools/lib/subcmd/pager.c
@@ -29,10 +29,13 @@ static void pager_preexec(void)
* have real input
*/
fd_set in;
+ fd_set exception;
FD_ZERO(&in);
+ FD_ZERO(&exception);
FD_SET(0, &in);
- select(1, &in, NULL, &in, NULL);
+ FD_SET(0, &exception);
+ select(1, &in, NULL, &exception, NULL);
setenv("LESS", "FRSX", 0);
}
diff --git a/tools/lib/traceevent/event-parse.c b/tools/lib/traceevent/event-parse.c
index 664c90c..6694753 100644
--- a/tools/lib/traceevent/event-parse.c
+++ b/tools/lib/traceevent/event-parse.c
@@ -4927,21 +4927,22 @@ static void pretty_print(struct trace_seq *s, void *data, int size, struct event
else
ls = 2;
- if (*(ptr+1) == 'F' || *(ptr+1) == 'f' ||
- *(ptr+1) == 'S' || *(ptr+1) == 's') {
+ if (isalnum(ptr[1]))
ptr++;
+
+ if (*ptr == 'F' || *ptr == 'f' ||
+ *ptr == 'S' || *ptr == 's') {
show_func = *ptr;
- } else if (*(ptr+1) == 'M' || *(ptr+1) == 'm') {
- print_mac_arg(s, *(ptr+1), data, size, event, arg);
- ptr++;
+ } else if (*ptr == 'M' || *ptr == 'm') {
+ print_mac_arg(s, *ptr, data, size, event, arg);
arg = arg->next;
break;
- } else if (*(ptr+1) == 'I' || *(ptr+1) == 'i') {
+ } else if (*ptr == 'I' || *ptr == 'i') {
int n;
- n = print_ip_arg(s, ptr+1, data, size, event, arg);
+ n = print_ip_arg(s, ptr, data, size, event, arg);
if (n > 0) {
- ptr += n;
+ ptr += n - 1;
arg = arg->next;
break;
}
diff --git a/tools/lib/traceevent/parse-filter.c b/tools/lib/traceevent/parse-filter.c
index 7c214ce..5e10ba7 100644
--- a/tools/lib/traceevent/parse-filter.c
+++ b/tools/lib/traceevent/parse-filter.c
@@ -1879,17 +1879,25 @@ static const char *get_field_str(struct filter_arg *arg, struct pevent_record *r
struct pevent *pevent;
unsigned long long addr;
const char *val = NULL;
+ unsigned int size;
char hex[64];
/* If the field is not a string convert it */
if (arg->str.field->flags & FIELD_IS_STRING) {
val = record->data + arg->str.field->offset;
+ size = arg->str.field->size;
+
+ if (arg->str.field->flags & FIELD_IS_DYNAMIC) {
+ addr = *(unsigned int *)val;
+ val = record->data + (addr & 0xffff);
+ size = addr >> 16;
+ }
/*
* We need to copy the data since we can't be sure the field
* is null terminated.
*/
- if (*(val + arg->str.field->size - 1)) {
+ if (*(val + size - 1)) {
/* copy it */
memcpy(arg->str.buffer, val, arg->str.field->size);
/* the buffer is already NULL terminated */
diff --git a/tools/objtool/.gitignore b/tools/objtool/.gitignore
index d3102c8..914cff1 100644
--- a/tools/objtool/.gitignore
+++ b/tools/objtool/.gitignore
@@ -1,3 +1,3 @@
-arch/x86/insn/inat-tables.c
+arch/x86/lib/inat-tables.c
objtool
fixdep
diff --git a/tools/objtool/Build b/tools/objtool/Build
index d6cdece..749becd 100644
--- a/tools/objtool/Build
+++ b/tools/objtool/Build
@@ -1,5 +1,9 @@
objtool-y += arch/$(SRCARCH)/
objtool-y += builtin-check.o
+objtool-y += builtin-orc.o
+objtool-y += check.o
+objtool-y += orc_gen.o
+objtool-y += orc_dump.o
objtool-y += elf.o
objtool-y += special.o
objtool-y += objtool.o
diff --git a/tools/objtool/Documentation/stack-validation.txt b/tools/objtool/Documentation/stack-validation.txt
index 05536d8..1fff237 100644
--- a/tools/objtool/Documentation/stack-validation.txt
+++ b/tools/objtool/Documentation/stack-validation.txt
@@ -11,9 +11,6 @@
It enforces a set of rules on asm code and C inline assembly code so
that stack traces can be reliable.
-Currently it only checks frame pointer usage, but there are plans to add
-CFI validation for C files and CFI generation for asm files.
-
For each function, it recursively follows all possible code paths and
validates the correct frame pointer state at each instruction.
@@ -23,6 +20,10 @@
instructions). Similarly, it knows how to follow switch statements, for
which gcc sometimes uses jump tables.
+(Objtool also has an 'orc generate' subcommand which generates debuginfo
+for the ORC unwinder. See Documentation/x86/orc-unwinder.txt in the
+kernel tree for more details.)
+
Why do we need stack metadata validation?
-----------------------------------------
@@ -93,62 +94,24 @@
or at the very end of the function after the stack frame has been
destroyed. This is an inherent limitation of frame pointers.
-b) 100% reliable stack traces for DWARF enabled kernels
+b) ORC (Oops Rewind Capability) unwind table generation
- (NOTE: This is not yet implemented)
+ An alternative to frame pointers and DWARF, ORC unwind data can be
+ used to walk the stack. Unlike frame pointers, ORC data is out of
+ band. So it doesn't affect runtime performance and it can be
+ reliable even when interrupts or exceptions are involved.
- As an alternative to frame pointers, DWARF Call Frame Information
- (CFI) metadata can be used to walk the stack. Unlike frame pointers,
- CFI metadata is out of band. So it doesn't affect runtime
- performance and it can be reliable even when interrupts or exceptions
- are involved.
-
- For C code, gcc automatically generates DWARF CFI metadata. But for
- asm code, generating CFI is a tedious manual approach which requires
- manually placed .cfi assembler macros to be scattered throughout the
- code. It's clumsy and very easy to get wrong, and it makes the real
- code harder to read.
-
- Stacktool will improve this situation in several ways. For code
- which already has CFI annotations, it will validate them. For code
- which doesn't have CFI annotations, it will generate them. So an
- architecture can opt to strip out all the manual .cfi annotations
- from their asm code and have objtool generate them instead.
-
- We might also add a runtime stack validation debug option where we
- periodically walk the stack from schedule() and/or an NMI to ensure
- that the stack metadata is sane and that we reach the bottom of the
- stack.
-
- So the benefit of objtool here will be that external tooling should
- always show perfect stack traces. And the same will be true for
- kernel warning/oops traces if the architecture has a runtime DWARF
- unwinder.
+ For more details, see Documentation/x86/orc-unwinder.txt.
c) Higher live patching compatibility rate
- (NOTE: This is not yet implemented)
+ Livepatch has an optional "consistency model", which is needed for
+ more complex patches. In order for the consistency model to work,
+ stack traces need to be reliable (or an unreliable condition needs to
+ be detectable). Objtool makes that possible.
- Currently with CONFIG_LIVEPATCH there's a basic live patching
- framework which is safe for roughly 85-90% of "security" fixes. But
- patches can't have complex features like function dependency or
- prototype changes, or data structure changes.
-
- There's a strong need to support patches which have the more complex
- features so that the patch compatibility rate for security fixes can
- eventually approach something resembling 100%. To achieve that, a
- "consistency model" is needed, which allows tasks to be safely
- transitioned from an unpatched state to a patched state.
-
- One of the key requirements of the currently proposed livepatch
- consistency model [*] is that it needs to walk the stack of each
- sleeping task to determine if it can be transitioned to the patched
- state. If objtool can ensure that stack traces are reliable, this
- consistency model can be used and the live patching compatibility
- rate can be improved significantly.
-
- [*] https://lkml.kernel.org/r/cover.1423499826.git.jpoimboe@redhat.com
-
+ For more details, see the livepatch documentation in the Linux kernel
+ source tree at Documentation/livepatch/livepatch.txt.
Rules
-----
@@ -201,80 +164,84 @@
return normally.
-Errors in .S files
-------------------
+Objtool warnings
+----------------
-If you're getting an error in a compiled .S file which you don't
-understand, first make sure that the affected code follows the above
-rules.
+For asm files, if you're getting an error which doesn't make sense,
+first make sure that the affected code follows the above rules.
+
+For C files, the common culprits are inline asm statements and calls to
+"noreturn" functions. See below for more details.
+
+Another possible cause for errors in C code is if the Makefile removes
+-fno-omit-frame-pointer or adds -fomit-frame-pointer to the gcc options.
Here are some examples of common warnings reported by objtool, what
they mean, and suggestions for how to fix them.
-1. asm_file.o: warning: objtool: func()+0x128: call without frame pointer save/setup
+1. file.o: warning: objtool: func()+0x128: call without frame pointer save/setup
The func() function made a function call without first saving and/or
- updating the frame pointer.
+ updating the frame pointer, and CONFIG_FRAME_POINTER is enabled.
- If func() is indeed a callable function, add proper frame pointer
- logic using the FRAME_BEGIN and FRAME_END macros. Otherwise, remove
- its ELF function annotation by changing ENDPROC to END.
+ If the error is for an asm file, and func() is indeed a callable
+ function, add proper frame pointer logic using the FRAME_BEGIN and
+ FRAME_END macros. Otherwise, if it's not a callable function, remove
+ its ELF function annotation by changing ENDPROC to END, and instead
+ use the manual unwind hint macros in asm/unwind_hints.h.
- If you're getting this error in a .c file, see the "Errors in .c
- files" section.
+ If it's a GCC-compiled .c file, the error may be because the function
+ uses an inline asm() statement which has a "call" instruction. An
+ asm() statement with a call instruction must declare the use of the
+ stack pointer in its output operand. On x86_64, this means adding
+ the ASM_CALL_CONSTRAINT as an output constraint:
+
+ asm volatile("call func" : ASM_CALL_CONSTRAINT);
+
+ Otherwise the stack frame may not get created before the call.
-2. asm_file.o: warning: objtool: .text+0x53: return instruction outside of a callable function
+2. file.o: warning: objtool: .text+0x53: unreachable instruction
- A return instruction was detected, but objtool couldn't find a way
- for a callable function to reach the instruction.
+ Objtool couldn't find a code path to reach the instruction.
- If the return instruction is inside (or reachable from) a callable
- function, the function needs to be annotated with the ENTRY/ENDPROC
- macros.
+ If the error is for an asm file, and the instruction is inside (or
+ reachable from) a callable function, the function should be annotated
+ with the ENTRY/ENDPROC macros (ENDPROC is the important one).
+ Otherwise, the code should probably be annotated with the unwind hint
+ macros in asm/unwind_hints.h so objtool and the unwinder can know the
+ stack state associated with the code.
- If you _really_ need a return instruction outside of a function, and
- are 100% sure that it won't affect stack traces, you can tell
- objtool to ignore it. See the "Adding exceptions" section below.
-
-
-3. asm_file.o: warning: objtool: func()+0x9: function has unreachable instruction
-
- The instruction lives inside of a callable function, but there's no
- possible control flow path from the beginning of the function to the
- instruction.
-
- If the instruction is actually needed, and it's actually in a
- callable function, ensure that its function is properly annotated
- with ENTRY/ENDPROC.
+ If you're 100% sure the code won't affect stack traces, or if you're
+ a just a bad person, you can tell objtool to ignore it. See the
+ "Adding exceptions" section below.
If it's not actually in a callable function (e.g. kernel entry code),
change ENDPROC to END.
-4. asm_file.o: warning: objtool: func(): can't find starting instruction
+4. file.o: warning: objtool: func(): can't find starting instruction
or
- asm_file.o: warning: objtool: func()+0x11dd: can't decode instruction
+ file.o: warning: objtool: func()+0x11dd: can't decode instruction
- Did you put data in a text section? If so, that can confuse
+ Does the file have data in a text section? If so, that can confuse
objtool's instruction decoder. Move the data to a more appropriate
section like .data or .rodata.
-5. asm_file.o: warning: objtool: func()+0x6: kernel entry/exit from callable instruction
+5. file.o: warning: objtool: func()+0x6: unsupported instruction in callable function
- This is a kernel entry/exit instruction like sysenter or sysret.
- Such instructions aren't allowed in a callable function, and are most
- likely part of the kernel entry code.
-
- If the instruction isn't actually in a callable function, change
- ENDPROC to END.
+ This is a kernel entry/exit instruction like sysenter or iret. Such
+ instructions aren't allowed in a callable function, and are most
+ likely part of the kernel entry code. They should usually not have
+ the callable function annotation (ENDPROC) and should always be
+ annotated with the unwind hint macros in asm/unwind_hints.h.
-6. asm_file.o: warning: objtool: func()+0x26: sibling call from callable instruction with changed frame pointer
+6. file.o: warning: objtool: func()+0x26: sibling call from callable instruction with modified stack frame
- This is a dynamic jump or a jump to an undefined symbol. Stacktool
+ This is a dynamic jump or a jump to an undefined symbol. Objtool
assumed it's a sibling call and detected that the frame pointer
wasn't first restored to its original state.
@@ -282,24 +249,28 @@
destination code to the local file.
If the instruction is not actually in a callable function (e.g.
- kernel entry code), change ENDPROC to END.
+ kernel entry code), change ENDPROC to END and annotate manually with
+ the unwind hint macros in asm/unwind_hints.h.
-7. asm_file: warning: objtool: func()+0x5c: frame pointer state mismatch
+7. file: warning: objtool: func()+0x5c: stack state mismatch
The instruction's frame pointer state is inconsistent, depending on
which execution path was taken to reach the instruction.
- Make sure the function pushes and sets up the frame pointer (for
- x86_64, this means rbp) at the beginning of the function and pops it
- at the end of the function. Also make sure that no other code in the
- function touches the frame pointer.
+ Make sure that, when CONFIG_FRAME_POINTER is enabled, the function
+ pushes and sets up the frame pointer (for x86_64, this means rbp) at
+ the beginning of the function and pops it at the end of the function.
+ Also make sure that no other code in the function touches the frame
+ pointer.
+
+ Another possibility is that the code has some asm or inline asm which
+ does some unusual things to the stack or the frame pointer. In such
+ cases it's probably appropriate to use the unwind hint macros in
+ asm/unwind_hints.h.
-Errors in .c files
-------------------
-
-1. c_file.o: warning: objtool: funcA() falls through to next function funcB()
+8. file.o: warning: objtool: funcA() falls through to next function funcB()
This means that funcA() doesn't end with a return instruction or an
unconditional jump, and that objtool has determined that the function
diff --git a/tools/objtool/Makefile b/tools/objtool/Makefile
index 041b493..8ae824d 100644
--- a/tools/objtool/Makefile
+++ b/tools/objtool/Makefile
@@ -1,3 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
include ../scripts/Makefile.include
include ../scripts/Makefile.arch
@@ -6,17 +7,19 @@
endif
# always use the host compiler
-CC = gcc
-LD = ld
-AR = ar
+HOSTCC ?= gcc
+HOSTLD ?= ld
+CC = $(HOSTCC)
+LD = $(HOSTLD)
+AR = ar
ifeq ($(srctree),)
-srctree := $(patsubst %/,%,$(dir $(shell pwd)))
+srctree := $(patsubst %/,%,$(dir $(CURDIR)))
srctree := $(patsubst %/,%,$(dir $(srctree)))
endif
SUBCMD_SRCDIR = $(srctree)/tools/lib/subcmd/
-LIBSUBCMD_OUTPUT = $(if $(OUTPUT),$(OUTPUT),$(PWD)/)
+LIBSUBCMD_OUTPUT = $(if $(OUTPUT),$(OUTPUT),$(CURDIR)/)
LIBSUBCMD = $(LIBSUBCMD_OUTPUT)libsubcmd.a
OBJTOOL := $(OUTPUT)objtool
@@ -24,12 +27,15 @@
all: $(OBJTOOL)
-INCLUDES := -I$(srctree)/tools/include -I$(srctree)/tools/arch/$(HOSTARCH)/include/uapi
-CFLAGS += -Wall -Werror $(EXTRA_WARNINGS) -fomit-frame-pointer -O2 -g $(INCLUDES)
+INCLUDES := -I$(srctree)/tools/include \
+ -I$(srctree)/tools/arch/$(HOSTARCH)/include/uapi \
+ -I$(srctree)/tools/objtool/arch/$(ARCH)/include
+WARNINGS := $(EXTRA_WARNINGS) -Wno-switch-default -Wno-switch-enum -Wno-packed
+CFLAGS += -Wall -Werror $(WARNINGS) -fomit-frame-pointer -O2 -g $(INCLUDES)
LDFLAGS += -lelf $(LIBSUBCMD)
# Allow old libelf to be used:
-elfshdr := $(shell echo '\#include <libelf.h>' | $(CC) $(CFLAGS) -x c -E - | grep elf_getshdr)
+elfshdr := $(shell echo '$(pound)include <libelf.h>' | $(CC) $(CFLAGS) -x c -E - | grep elf_getshdr)
CFLAGS += $(if $(elfshdr),,-DLIBELF_USE_DEPRECATED)
AWK = awk
@@ -39,19 +45,8 @@
$(OBJTOOL_IN): fixdep FORCE
@$(MAKE) $(build)=objtool
-# Busybox's diff doesn't have -I, avoid warning in that case
-#
$(OBJTOOL): $(LIBSUBCMD) $(OBJTOOL_IN)
- @(diff -I 2>&1 | grep -q 'option requires an argument' && \
- test -d ../../kernel -a -d ../../tools -a -d ../objtool && (( \
- diff -I'^#include' arch/x86/insn/insn.c ../../arch/x86/lib/insn.c >/dev/null && \
- diff -I'^#include' arch/x86/insn/inat.c ../../arch/x86/lib/inat.c >/dev/null && \
- diff arch/x86/insn/x86-opcode-map.txt ../../arch/x86/lib/x86-opcode-map.txt >/dev/null && \
- diff arch/x86/insn/gen-insn-attr-x86.awk ../../arch/x86/tools/gen-insn-attr-x86.awk >/dev/null && \
- diff -I'^#include' arch/x86/insn/insn.h ../../arch/x86/include/asm/insn.h >/dev/null && \
- diff -I'^#include' arch/x86/insn/inat.h ../../arch/x86/include/asm/inat.h >/dev/null && \
- diff -I'^#include' arch/x86/insn/inat_types.h ../../arch/x86/include/asm/inat_types.h >/dev/null) \
- || echo "warning: objtool: x86 instruction decoder differs from kernel" >&2 )) || true
+ @$(CONFIG_SHELL) ./sync-check.sh
$(QUIET_LINK)$(CC) $(OBJTOOL_IN) $(LDFLAGS) -o $@
@@ -61,7 +56,7 @@
clean:
$(call QUIET_CLEAN, objtool) $(RM) $(OBJTOOL)
$(Q)find $(OUTPUT) -name '*.o' -delete -o -name '\.*.cmd' -delete -o -name '\.*.d' -delete
- $(Q)$(RM) $(OUTPUT)arch/x86/insn/inat-tables.c $(OUTPUT)fixdep
+ $(Q)$(RM) $(OUTPUT)arch/x86/lib/inat-tables.c $(OUTPUT)fixdep
FORCE:
diff --git a/tools/objtool/arch.h b/tools/objtool/arch.h
index f7350fc..b0d7dc3 100644
--- a/tools/objtool/arch.h
+++ b/tools/objtool/arch.h
@@ -19,26 +19,64 @@
#define _ARCH_H
#include <stdbool.h>
+#include <linux/list.h>
#include "elf.h"
+#include "cfi.h"
-#define INSN_FP_SAVE 1
-#define INSN_FP_SETUP 2
-#define INSN_FP_RESTORE 3
-#define INSN_JUMP_CONDITIONAL 4
-#define INSN_JUMP_UNCONDITIONAL 5
-#define INSN_JUMP_DYNAMIC 6
-#define INSN_CALL 7
-#define INSN_CALL_DYNAMIC 8
-#define INSN_RETURN 9
-#define INSN_CONTEXT_SWITCH 10
-#define INSN_BUG 11
-#define INSN_NOP 12
-#define INSN_OTHER 13
+#define INSN_JUMP_CONDITIONAL 1
+#define INSN_JUMP_UNCONDITIONAL 2
+#define INSN_JUMP_DYNAMIC 3
+#define INSN_CALL 4
+#define INSN_CALL_DYNAMIC 5
+#define INSN_RETURN 6
+#define INSN_CONTEXT_SWITCH 7
+#define INSN_STACK 8
+#define INSN_BUG 9
+#define INSN_NOP 10
+#define INSN_OTHER 11
#define INSN_LAST INSN_OTHER
+enum op_dest_type {
+ OP_DEST_REG,
+ OP_DEST_REG_INDIRECT,
+ OP_DEST_MEM,
+ OP_DEST_PUSH,
+ OP_DEST_LEAVE,
+};
+
+struct op_dest {
+ enum op_dest_type type;
+ unsigned char reg;
+ int offset;
+};
+
+enum op_src_type {
+ OP_SRC_REG,
+ OP_SRC_REG_INDIRECT,
+ OP_SRC_CONST,
+ OP_SRC_POP,
+ OP_SRC_ADD,
+ OP_SRC_AND,
+};
+
+struct op_src {
+ enum op_src_type type;
+ unsigned char reg;
+ int offset;
+};
+
+struct stack_op {
+ struct op_dest dest;
+ struct op_src src;
+};
+
+void arch_initial_func_cfi_state(struct cfi_state *state);
+
int arch_decode_instruction(struct elf *elf, struct section *sec,
unsigned long offset, unsigned int maxlen,
unsigned int *len, unsigned char *type,
- unsigned long *displacement);
+ unsigned long *immediate, struct stack_op *op);
+
+bool arch_callee_saved_reg(unsigned char reg);
#endif /* _ARCH_H */
diff --git a/tools/objtool/arch/x86/Build b/tools/objtool/arch/x86/Build
index debbdb0..b998412c 100644
--- a/tools/objtool/arch/x86/Build
+++ b/tools/objtool/arch/x86/Build
@@ -1,12 +1,12 @@
objtool-y += decode.o
-inat_tables_script = arch/x86/insn/gen-insn-attr-x86.awk
-inat_tables_maps = arch/x86/insn/x86-opcode-map.txt
+inat_tables_script = arch/x86/tools/gen-insn-attr-x86.awk
+inat_tables_maps = arch/x86/lib/x86-opcode-map.txt
-$(OUTPUT)arch/x86/insn/inat-tables.c: $(inat_tables_script) $(inat_tables_maps)
+$(OUTPUT)arch/x86/lib/inat-tables.c: $(inat_tables_script) $(inat_tables_maps)
$(call rule_mkdir)
$(Q)$(call echo-cmd,gen)$(AWK) -f $(inat_tables_script) $(inat_tables_maps) > $@
-$(OUTPUT)arch/x86/decode.o: $(OUTPUT)arch/x86/insn/inat-tables.c
+$(OUTPUT)arch/x86/decode.o: $(OUTPUT)arch/x86/lib/inat-tables.c
-CFLAGS_decode.o += -I$(OUTPUT)arch/x86/insn
+CFLAGS_decode.o += -I$(OUTPUT)arch/x86/lib
diff --git a/tools/objtool/arch/x86/decode.c b/tools/objtool/arch/x86/decode.c
index 039636f..540a209 100644
--- a/tools/objtool/arch/x86/decode.c
+++ b/tools/objtool/arch/x86/decode.c
@@ -19,14 +19,25 @@
#include <stdlib.h>
#define unlikely(cond) (cond)
-#include "insn/insn.h"
-#include "insn/inat.c"
-#include "insn/insn.c"
+#include <asm/insn.h>
+#include "lib/inat.c"
+#include "lib/insn.c"
#include "../../elf.h"
#include "../../arch.h"
#include "../../warn.h"
+static unsigned char op_to_cfi_reg[][2] = {
+ {CFI_AX, CFI_R8},
+ {CFI_CX, CFI_R9},
+ {CFI_DX, CFI_R10},
+ {CFI_BX, CFI_R11},
+ {CFI_SP, CFI_R12},
+ {CFI_BP, CFI_R13},
+ {CFI_SI, CFI_R14},
+ {CFI_DI, CFI_R15},
+};
+
static int is_x86_64(struct elf *elf)
{
switch (elf->ehdr.e_machine) {
@@ -40,24 +51,50 @@ static int is_x86_64(struct elf *elf)
}
}
+bool arch_callee_saved_reg(unsigned char reg)
+{
+ switch (reg) {
+ case CFI_BP:
+ case CFI_BX:
+ case CFI_R12:
+ case CFI_R13:
+ case CFI_R14:
+ case CFI_R15:
+ return true;
+
+ case CFI_AX:
+ case CFI_CX:
+ case CFI_DX:
+ case CFI_SI:
+ case CFI_DI:
+ case CFI_SP:
+ case CFI_R8:
+ case CFI_R9:
+ case CFI_R10:
+ case CFI_R11:
+ case CFI_RA:
+ default:
+ return false;
+ }
+}
+
int arch_decode_instruction(struct elf *elf, struct section *sec,
unsigned long offset, unsigned int maxlen,
unsigned int *len, unsigned char *type,
- unsigned long *immediate)
+ unsigned long *immediate, struct stack_op *op)
{
struct insn insn;
- int x86_64;
- unsigned char op1, op2, ext;
+ int x86_64, sign;
+ unsigned char op1, op2, rex = 0, rex_b = 0, rex_r = 0, rex_w = 0,
+ rex_x = 0, modrm = 0, modrm_mod = 0, modrm_rm = 0,
+ modrm_reg = 0, sib = 0;
x86_64 = is_x86_64(elf);
if (x86_64 == -1)
return -1;
- insn_init(&insn, (void *)(sec->data + offset), maxlen, x86_64);
+ insn_init(&insn, sec->data->d_buf + offset, maxlen, x86_64);
insn_get_length(&insn);
- insn_get_opcode(&insn);
- insn_get_modrm(&insn);
- insn_get_immediate(&insn);
if (!insn_complete(&insn)) {
WARN_FUNC("can't decode instruction", sec, offset);
@@ -73,70 +110,317 @@ int arch_decode_instruction(struct elf *elf, struct section *sec,
op1 = insn.opcode.bytes[0];
op2 = insn.opcode.bytes[1];
+ if (insn.rex_prefix.nbytes) {
+ rex = insn.rex_prefix.bytes[0];
+ rex_w = X86_REX_W(rex) >> 3;
+ rex_r = X86_REX_R(rex) >> 2;
+ rex_x = X86_REX_X(rex) >> 1;
+ rex_b = X86_REX_B(rex);
+ }
+
+ if (insn.modrm.nbytes) {
+ modrm = insn.modrm.bytes[0];
+ modrm_mod = X86_MODRM_MOD(modrm);
+ modrm_reg = X86_MODRM_REG(modrm);
+ modrm_rm = X86_MODRM_RM(modrm);
+ }
+
+ if (insn.sib.nbytes)
+ sib = insn.sib.bytes[0];
+
switch (op1) {
- case 0x55:
- if (!insn.rex_prefix.nbytes)
- /* push rbp */
- *type = INSN_FP_SAVE;
+
+ case 0x1:
+ case 0x29:
+ if (rex_w && !rex_b && modrm_mod == 3 && modrm_rm == 4) {
+
+ /* add/sub reg, %rsp */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = op_to_cfi_reg[modrm_reg][rex_r];
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+ }
break;
- case 0x5d:
- if (!insn.rex_prefix.nbytes)
- /* pop rbp */
- *type = INSN_FP_RESTORE;
+ case 0x50 ... 0x57:
+
+ /* push reg */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG;
+ op->src.reg = op_to_cfi_reg[op1 & 0x7][rex_b];
+ op->dest.type = OP_DEST_PUSH;
+
+ break;
+
+ case 0x58 ... 0x5f:
+
+ /* pop reg */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_POP;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = op_to_cfi_reg[op1 & 0x7][rex_b];
+
+ break;
+
+ case 0x68:
+ case 0x6a:
+ /* push immediate */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_CONST;
+ op->dest.type = OP_DEST_PUSH;
break;
case 0x70 ... 0x7f:
*type = INSN_JUMP_CONDITIONAL;
break;
+ case 0x81:
+ case 0x83:
+ if (rex != 0x48)
+ break;
+
+ if (modrm == 0xe4) {
+ /* and imm, %rsp */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_AND;
+ op->src.reg = CFI_SP;
+ op->src.offset = insn.immediate.value;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+ break;
+ }
+
+ if (modrm == 0xc4)
+ sign = 1;
+ else if (modrm == 0xec)
+ sign = -1;
+ else
+ break;
+
+ /* add/sub imm, %rsp */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = CFI_SP;
+ op->src.offset = insn.immediate.value * sign;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+ break;
+
case 0x89:
- if (insn.rex_prefix.nbytes == 1 &&
- insn.rex_prefix.bytes[0] == 0x48 &&
- insn.modrm.nbytes && insn.modrm.bytes[0] == 0xe5)
- /* mov rsp, rbp */
- *type = INSN_FP_SETUP;
+ if (rex_w && !rex_r && modrm_mod == 3 && modrm_reg == 4) {
+
+ /* mov %rsp, reg */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG;
+ op->src.reg = CFI_SP;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = op_to_cfi_reg[modrm_rm][rex_b];
+ break;
+ }
+
+ if (rex_w && !rex_b && modrm_mod == 3 && modrm_rm == 4) {
+
+ /* mov reg, %rsp */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG;
+ op->src.reg = op_to_cfi_reg[modrm_reg][rex_r];
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+ break;
+ }
+
+ /* fallthrough */
+ case 0x88:
+ if (!rex_b &&
+ (modrm_mod == 1 || modrm_mod == 2) && modrm_rm == 5) {
+
+ /* mov reg, disp(%rbp) */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG;
+ op->src.reg = op_to_cfi_reg[modrm_reg][rex_r];
+ op->dest.type = OP_DEST_REG_INDIRECT;
+ op->dest.reg = CFI_BP;
+ op->dest.offset = insn.displacement.value;
+
+ } else if (rex_w && !rex_b && modrm_rm == 4 && sib == 0x24) {
+
+ /* mov reg, disp(%rsp) */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG;
+ op->src.reg = op_to_cfi_reg[modrm_reg][rex_r];
+ op->dest.type = OP_DEST_REG_INDIRECT;
+ op->dest.reg = CFI_SP;
+ op->dest.offset = insn.displacement.value;
+ }
+
+ break;
+
+ case 0x8b:
+ if (rex_w && !rex_b && modrm_mod == 1 && modrm_rm == 5) {
+
+ /* mov disp(%rbp), reg */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG_INDIRECT;
+ op->src.reg = CFI_BP;
+ op->src.offset = insn.displacement.value;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = op_to_cfi_reg[modrm_reg][rex_r];
+
+ } else if (rex_w && !rex_b && sib == 0x24 &&
+ modrm_mod != 3 && modrm_rm == 4) {
+
+ /* mov disp(%rsp), reg */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_REG_INDIRECT;
+ op->src.reg = CFI_SP;
+ op->src.offset = insn.displacement.value;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = op_to_cfi_reg[modrm_reg][rex_r];
+ }
+
break;
case 0x8d:
- if (insn.rex_prefix.nbytes &&
- insn.rex_prefix.bytes[0] == 0x48 &&
- insn.modrm.nbytes && insn.modrm.bytes[0] == 0x2c &&
- insn.sib.nbytes && insn.sib.bytes[0] == 0x24)
- /* lea %(rsp), %rbp */
- *type = INSN_FP_SETUP;
+ if (sib == 0x24 && rex_w && !rex_b && !rex_x) {
+
+ *type = INSN_STACK;
+ if (!insn.displacement.value) {
+ /* lea (%rsp), reg */
+ op->src.type = OP_SRC_REG;
+ } else {
+ /* lea disp(%rsp), reg */
+ op->src.type = OP_SRC_ADD;
+ op->src.offset = insn.displacement.value;
+ }
+ op->src.reg = CFI_SP;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = op_to_cfi_reg[modrm_reg][rex_r];
+
+ } else if (rex == 0x48 && modrm == 0x65) {
+
+ /* lea disp(%rbp), %rsp */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = CFI_BP;
+ op->src.offset = insn.displacement.value;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+
+ } else if (rex == 0x49 && modrm == 0x62 &&
+ insn.displacement.value == -8) {
+
+ /*
+ * lea -0x8(%r10), %rsp
+ *
+ * Restoring rsp back to its original value after a
+ * stack realignment.
+ */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = CFI_R10;
+ op->src.offset = -8;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+
+ } else if (rex == 0x49 && modrm == 0x65 &&
+ insn.displacement.value == -16) {
+
+ /*
+ * lea -0x10(%r13), %rsp
+ *
+ * Restoring rsp back to its original value after a
+ * stack realignment.
+ */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_ADD;
+ op->src.reg = CFI_R13;
+ op->src.offset = -16;
+ op->dest.type = OP_DEST_REG;
+ op->dest.reg = CFI_SP;
+ }
+
+ break;
+
+ case 0x8f:
+ /* pop to mem */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_POP;
+ op->dest.type = OP_DEST_MEM;
break;
case 0x90:
*type = INSN_NOP;
break;
+ case 0x9c:
+ /* pushf */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_CONST;
+ op->dest.type = OP_DEST_PUSH;
+ break;
+
+ case 0x9d:
+ /* popf */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_POP;
+ op->dest.type = OP_DEST_MEM;
+ break;
+
case 0x0f:
- if (op2 >= 0x80 && op2 <= 0x8f)
+
+ if (op2 >= 0x80 && op2 <= 0x8f) {
+
*type = INSN_JUMP_CONDITIONAL;
- else if (op2 == 0x05 || op2 == 0x07 || op2 == 0x34 ||
- op2 == 0x35)
+
+ } else if (op2 == 0x05 || op2 == 0x07 || op2 == 0x34 ||
+ op2 == 0x35) {
+
/* sysenter, sysret */
*type = INSN_CONTEXT_SWITCH;
- else if (op2 == 0x0b || op2 == 0xb9)
+
+ } else if (op2 == 0x0b || op2 == 0xb9) {
+
/* ud2 */
*type = INSN_BUG;
- else if (op2 == 0x0d || op2 == 0x1f)
+
+ } else if (op2 == 0x0d || op2 == 0x1f) {
+
/* nopl/nopw */
*type = INSN_NOP;
- else if (op2 == 0x01 && insn.modrm.nbytes &&
- (insn.modrm.bytes[0] == 0xc2 ||
- insn.modrm.bytes[0] == 0xd8))
- /* vmlaunch, vmrun */
- *type = INSN_CONTEXT_SWITCH;
+
+ } else if (op2 == 0xa0 || op2 == 0xa8) {
+
+ /* push fs/gs */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_CONST;
+ op->dest.type = OP_DEST_PUSH;
+
+ } else if (op2 == 0xa1 || op2 == 0xa9) {
+
+ /* pop fs/gs */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_POP;
+ op->dest.type = OP_DEST_MEM;
+ }
break;
- case 0xc9: /* leave */
- *type = INSN_FP_RESTORE;
+ case 0xc9:
+ /*
+ * leave
+ *
+ * equivalent to:
+ * mov bp, sp
+ * pop bp
+ */
+ *type = INSN_STACK;
+ op->dest.type = OP_DEST_LEAVE;
+
break;
- case 0xe3: /* jecxz/jrcxz */
+ case 0xe3:
+ /* jecxz/jrcxz */
*type = INSN_JUMP_CONDITIONAL;
break;
@@ -161,14 +445,27 @@ int arch_decode_instruction(struct elf *elf, struct section *sec,
break;
case 0xff:
- ext = X86_MODRM_REG(insn.modrm.bytes[0]);
- if (ext == 2 || ext == 3)
+ if (modrm_reg == 2 || modrm_reg == 3)
+
*type = INSN_CALL_DYNAMIC;
- else if (ext == 4)
+
+ else if (modrm_reg == 4)
+
*type = INSN_JUMP_DYNAMIC;
- else if (ext == 5) /*jmpf */
+
+ else if (modrm_reg == 5)
+
+ /* jmpf */
*type = INSN_CONTEXT_SWITCH;
+ else if (modrm_reg == 6) {
+
+ /* push from mem */
+ *type = INSN_STACK;
+ op->src.type = OP_SRC_CONST;
+ op->dest.type = OP_DEST_PUSH;
+ }
+
break;
default:
@@ -179,3 +476,21 @@ int arch_decode_instruction(struct elf *elf, struct section *sec,
return 0;
}
+
+void arch_initial_func_cfi_state(struct cfi_state *state)
+{
+ int i;
+
+ for (i = 0; i < CFI_NUM_REGS; i++) {
+ state->regs[i].base = CFI_UNDEFINED;
+ state->regs[i].offset = 0;
+ }
+
+ /* initial CFA (call frame address) */
+ state->cfa.base = CFI_SP;
+ state->cfa.offset = 8;
+
+ /* initial RA (return address) */
+ state->regs[16].base = CFI_CFA;
+ state->regs[16].offset = -8;
+}
diff --git a/tools/objtool/arch/x86/include/asm/inat.h b/tools/objtool/arch/x86/include/asm/inat.h
new file mode 100644
index 0000000..02aff08
--- /dev/null
+++ b/tools/objtool/arch/x86/include/asm/inat.h
@@ -0,0 +1,234 @@
+#ifndef _ASM_X86_INAT_H
+#define _ASM_X86_INAT_H
+/*
+ * x86 instruction attributes
+ *
+ * Written by Masami Hiramatsu <mhiramat@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ */
+#include <asm/inat_types.h>
+
+/*
+ * Internal bits. Don't use bitmasks directly, because these bits are
+ * unstable. You should use checking functions.
+ */
+
+#define INAT_OPCODE_TABLE_SIZE 256
+#define INAT_GROUP_TABLE_SIZE 8
+
+/* Legacy last prefixes */
+#define INAT_PFX_OPNDSZ 1 /* 0x66 */ /* LPFX1 */
+#define INAT_PFX_REPE 2 /* 0xF3 */ /* LPFX2 */
+#define INAT_PFX_REPNE 3 /* 0xF2 */ /* LPFX3 */
+/* Other Legacy prefixes */
+#define INAT_PFX_LOCK 4 /* 0xF0 */
+#define INAT_PFX_CS 5 /* 0x2E */
+#define INAT_PFX_DS 6 /* 0x3E */
+#define INAT_PFX_ES 7 /* 0x26 */
+#define INAT_PFX_FS 8 /* 0x64 */
+#define INAT_PFX_GS 9 /* 0x65 */
+#define INAT_PFX_SS 10 /* 0x36 */
+#define INAT_PFX_ADDRSZ 11 /* 0x67 */
+/* x86-64 REX prefix */
+#define INAT_PFX_REX 12 /* 0x4X */
+/* AVX VEX prefixes */
+#define INAT_PFX_VEX2 13 /* 2-bytes VEX prefix */
+#define INAT_PFX_VEX3 14 /* 3-bytes VEX prefix */
+#define INAT_PFX_EVEX 15 /* EVEX prefix */
+
+#define INAT_LSTPFX_MAX 3
+#define INAT_LGCPFX_MAX 11
+
+/* Immediate size */
+#define INAT_IMM_BYTE 1
+#define INAT_IMM_WORD 2
+#define INAT_IMM_DWORD 3
+#define INAT_IMM_QWORD 4
+#define INAT_IMM_PTR 5
+#define INAT_IMM_VWORD32 6
+#define INAT_IMM_VWORD 7
+
+/* Legacy prefix */
+#define INAT_PFX_OFFS 0
+#define INAT_PFX_BITS 4
+#define INAT_PFX_MAX ((1 << INAT_PFX_BITS) - 1)
+#define INAT_PFX_MASK (INAT_PFX_MAX << INAT_PFX_OFFS)
+/* Escape opcodes */
+#define INAT_ESC_OFFS (INAT_PFX_OFFS + INAT_PFX_BITS)
+#define INAT_ESC_BITS 2
+#define INAT_ESC_MAX ((1 << INAT_ESC_BITS) - 1)
+#define INAT_ESC_MASK (INAT_ESC_MAX << INAT_ESC_OFFS)
+/* Group opcodes (1-16) */
+#define INAT_GRP_OFFS (INAT_ESC_OFFS + INAT_ESC_BITS)
+#define INAT_GRP_BITS 5
+#define INAT_GRP_MAX ((1 << INAT_GRP_BITS) - 1)
+#define INAT_GRP_MASK (INAT_GRP_MAX << INAT_GRP_OFFS)
+/* Immediates */
+#define INAT_IMM_OFFS (INAT_GRP_OFFS + INAT_GRP_BITS)
+#define INAT_IMM_BITS 3
+#define INAT_IMM_MASK (((1 << INAT_IMM_BITS) - 1) << INAT_IMM_OFFS)
+/* Flags */
+#define INAT_FLAG_OFFS (INAT_IMM_OFFS + INAT_IMM_BITS)
+#define INAT_MODRM (1 << (INAT_FLAG_OFFS))
+#define INAT_FORCE64 (1 << (INAT_FLAG_OFFS + 1))
+#define INAT_SCNDIMM (1 << (INAT_FLAG_OFFS + 2))
+#define INAT_MOFFSET (1 << (INAT_FLAG_OFFS + 3))
+#define INAT_VARIANT (1 << (INAT_FLAG_OFFS + 4))
+#define INAT_VEXOK (1 << (INAT_FLAG_OFFS + 5))
+#define INAT_VEXONLY (1 << (INAT_FLAG_OFFS + 6))
+#define INAT_EVEXONLY (1 << (INAT_FLAG_OFFS + 7))
+/* Attribute making macros for attribute tables */
+#define INAT_MAKE_PREFIX(pfx) (pfx << INAT_PFX_OFFS)
+#define INAT_MAKE_ESCAPE(esc) (esc << INAT_ESC_OFFS)
+#define INAT_MAKE_GROUP(grp) ((grp << INAT_GRP_OFFS) | INAT_MODRM)
+#define INAT_MAKE_IMM(imm) (imm << INAT_IMM_OFFS)
+
+/* Attribute search APIs */
+extern insn_attr_t inat_get_opcode_attribute(insn_byte_t opcode);
+extern int inat_get_last_prefix_id(insn_byte_t last_pfx);
+extern insn_attr_t inat_get_escape_attribute(insn_byte_t opcode,
+ int lpfx_id,
+ insn_attr_t esc_attr);
+extern insn_attr_t inat_get_group_attribute(insn_byte_t modrm,
+ int lpfx_id,
+ insn_attr_t esc_attr);
+extern insn_attr_t inat_get_avx_attribute(insn_byte_t opcode,
+ insn_byte_t vex_m,
+ insn_byte_t vex_pp);
+
+/* Attribute checking functions */
+static inline int inat_is_legacy_prefix(insn_attr_t attr)
+{
+ attr &= INAT_PFX_MASK;
+ return attr && attr <= INAT_LGCPFX_MAX;
+}
+
+static inline int inat_is_address_size_prefix(insn_attr_t attr)
+{
+ return (attr & INAT_PFX_MASK) == INAT_PFX_ADDRSZ;
+}
+
+static inline int inat_is_operand_size_prefix(insn_attr_t attr)
+{
+ return (attr & INAT_PFX_MASK) == INAT_PFX_OPNDSZ;
+}
+
+static inline int inat_is_rex_prefix(insn_attr_t attr)
+{
+ return (attr & INAT_PFX_MASK) == INAT_PFX_REX;
+}
+
+static inline int inat_last_prefix_id(insn_attr_t attr)
+{
+ if ((attr & INAT_PFX_MASK) > INAT_LSTPFX_MAX)
+ return 0;
+ else
+ return attr & INAT_PFX_MASK;
+}
+
+static inline int inat_is_vex_prefix(insn_attr_t attr)
+{
+ attr &= INAT_PFX_MASK;
+ return attr == INAT_PFX_VEX2 || attr == INAT_PFX_VEX3 ||
+ attr == INAT_PFX_EVEX;
+}
+
+static inline int inat_is_evex_prefix(insn_attr_t attr)
+{
+ return (attr & INAT_PFX_MASK) == INAT_PFX_EVEX;
+}
+
+static inline int inat_is_vex3_prefix(insn_attr_t attr)
+{
+ return (attr & INAT_PFX_MASK) == INAT_PFX_VEX3;
+}
+
+static inline int inat_is_escape(insn_attr_t attr)
+{
+ return attr & INAT_ESC_MASK;
+}
+
+static inline int inat_escape_id(insn_attr_t attr)
+{
+ return (attr & INAT_ESC_MASK) >> INAT_ESC_OFFS;
+}
+
+static inline int inat_is_group(insn_attr_t attr)
+{
+ return attr & INAT_GRP_MASK;
+}
+
+static inline int inat_group_id(insn_attr_t attr)
+{
+ return (attr & INAT_GRP_MASK) >> INAT_GRP_OFFS;
+}
+
+static inline int inat_group_common_attribute(insn_attr_t attr)
+{
+ return attr & ~INAT_GRP_MASK;
+}
+
+static inline int inat_has_immediate(insn_attr_t attr)
+{
+ return attr & INAT_IMM_MASK;
+}
+
+static inline int inat_immediate_size(insn_attr_t attr)
+{
+ return (attr & INAT_IMM_MASK) >> INAT_IMM_OFFS;
+}
+
+static inline int inat_has_modrm(insn_attr_t attr)
+{
+ return attr & INAT_MODRM;
+}
+
+static inline int inat_is_force64(insn_attr_t attr)
+{
+ return attr & INAT_FORCE64;
+}
+
+static inline int inat_has_second_immediate(insn_attr_t attr)
+{
+ return attr & INAT_SCNDIMM;
+}
+
+static inline int inat_has_moffset(insn_attr_t attr)
+{
+ return attr & INAT_MOFFSET;
+}
+
+static inline int inat_has_variant(insn_attr_t attr)
+{
+ return attr & INAT_VARIANT;
+}
+
+static inline int inat_accept_vex(insn_attr_t attr)
+{
+ return attr & INAT_VEXOK;
+}
+
+static inline int inat_must_vex(insn_attr_t attr)
+{
+ return attr & (INAT_VEXONLY | INAT_EVEXONLY);
+}
+
+static inline int inat_must_evex(insn_attr_t attr)
+{
+ return attr & INAT_EVEXONLY;
+}
+#endif
diff --git a/tools/objtool/arch/x86/insn/inat_types.h b/tools/objtool/arch/x86/include/asm/inat_types.h
similarity index 100%
rename from tools/objtool/arch/x86/insn/inat_types.h
rename to tools/objtool/arch/x86/include/asm/inat_types.h
diff --git a/tools/objtool/arch/x86/include/asm/insn.h b/tools/objtool/arch/x86/include/asm/insn.h
new file mode 100644
index 0000000..b3e32b0
--- /dev/null
+++ b/tools/objtool/arch/x86/include/asm/insn.h
@@ -0,0 +1,211 @@
+#ifndef _ASM_X86_INSN_H
+#define _ASM_X86_INSN_H
+/*
+ * x86 instruction analysis
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * Copyright (C) IBM Corporation, 2009
+ */
+
+/* insn_attr_t is defined in inat.h */
+#include <asm/inat.h>
+
+struct insn_field {
+ union {
+ insn_value_t value;
+ insn_byte_t bytes[4];
+ };
+ /* !0 if we've run insn_get_xxx() for this field */
+ unsigned char got;
+ unsigned char nbytes;
+};
+
+struct insn {
+ struct insn_field prefixes; /*
+ * Prefixes
+ * prefixes.bytes[3]: last prefix
+ */
+ struct insn_field rex_prefix; /* REX prefix */
+ struct insn_field vex_prefix; /* VEX prefix */
+ struct insn_field opcode; /*
+ * opcode.bytes[0]: opcode1
+ * opcode.bytes[1]: opcode2
+ * opcode.bytes[2]: opcode3
+ */
+ struct insn_field modrm;
+ struct insn_field sib;
+ struct insn_field displacement;
+ union {
+ struct insn_field immediate;
+ struct insn_field moffset1; /* for 64bit MOV */
+ struct insn_field immediate1; /* for 64bit imm or off16/32 */
+ };
+ union {
+ struct insn_field moffset2; /* for 64bit MOV */
+ struct insn_field immediate2; /* for 64bit imm or seg16 */
+ };
+
+ insn_attr_t attr;
+ unsigned char opnd_bytes;
+ unsigned char addr_bytes;
+ unsigned char length;
+ unsigned char x86_64;
+
+ const insn_byte_t *kaddr; /* kernel address of insn to analyze */
+ const insn_byte_t *end_kaddr; /* kernel address of last insn in buffer */
+ const insn_byte_t *next_byte;
+};
+
+#define MAX_INSN_SIZE 15
+
+#define X86_MODRM_MOD(modrm) (((modrm) & 0xc0) >> 6)
+#define X86_MODRM_REG(modrm) (((modrm) & 0x38) >> 3)
+#define X86_MODRM_RM(modrm) ((modrm) & 0x07)
+
+#define X86_SIB_SCALE(sib) (((sib) & 0xc0) >> 6)
+#define X86_SIB_INDEX(sib) (((sib) & 0x38) >> 3)
+#define X86_SIB_BASE(sib) ((sib) & 0x07)
+
+#define X86_REX_W(rex) ((rex) & 8)
+#define X86_REX_R(rex) ((rex) & 4)
+#define X86_REX_X(rex) ((rex) & 2)
+#define X86_REX_B(rex) ((rex) & 1)
+
+/* VEX bit flags */
+#define X86_VEX_W(vex) ((vex) & 0x80) /* VEX3 Byte2 */
+#define X86_VEX_R(vex) ((vex) & 0x80) /* VEX2/3 Byte1 */
+#define X86_VEX_X(vex) ((vex) & 0x40) /* VEX3 Byte1 */
+#define X86_VEX_B(vex) ((vex) & 0x20) /* VEX3 Byte1 */
+#define X86_VEX_L(vex) ((vex) & 0x04) /* VEX3 Byte2, VEX2 Byte1 */
+/* VEX bit fields */
+#define X86_EVEX_M(vex) ((vex) & 0x03) /* EVEX Byte1 */
+#define X86_VEX3_M(vex) ((vex) & 0x1f) /* VEX3 Byte1 */
+#define X86_VEX2_M 1 /* VEX2.M always 1 */
+#define X86_VEX_V(vex) (((vex) & 0x78) >> 3) /* VEX3 Byte2, VEX2 Byte1 */
+#define X86_VEX_P(vex) ((vex) & 0x03) /* VEX3 Byte2, VEX2 Byte1 */
+#define X86_VEX_M_MAX 0x1f /* VEX3.M Maximum value */
+
+extern void insn_init(struct insn *insn, const void *kaddr, int buf_len, int x86_64);
+extern void insn_get_prefixes(struct insn *insn);
+extern void insn_get_opcode(struct insn *insn);
+extern void insn_get_modrm(struct insn *insn);
+extern void insn_get_sib(struct insn *insn);
+extern void insn_get_displacement(struct insn *insn);
+extern void insn_get_immediate(struct insn *insn);
+extern void insn_get_length(struct insn *insn);
+
+/* Attribute will be determined after getting ModRM (for opcode groups) */
+static inline void insn_get_attribute(struct insn *insn)
+{
+ insn_get_modrm(insn);
+}
+
+/* Instruction uses RIP-relative addressing */
+extern int insn_rip_relative(struct insn *insn);
+
+/* Init insn for kernel text */
+static inline void kernel_insn_init(struct insn *insn,
+ const void *kaddr, int buf_len)
+{
+#ifdef CONFIG_X86_64
+ insn_init(insn, kaddr, buf_len, 1);
+#else /* CONFIG_X86_32 */
+ insn_init(insn, kaddr, buf_len, 0);
+#endif
+}
+
+static inline int insn_is_avx(struct insn *insn)
+{
+ if (!insn->prefixes.got)
+ insn_get_prefixes(insn);
+ return (insn->vex_prefix.value != 0);
+}
+
+static inline int insn_is_evex(struct insn *insn)
+{
+ if (!insn->prefixes.got)
+ insn_get_prefixes(insn);
+ return (insn->vex_prefix.nbytes == 4);
+}
+
+/* Ensure this instruction is decoded completely */
+static inline int insn_complete(struct insn *insn)
+{
+ return insn->opcode.got && insn->modrm.got && insn->sib.got &&
+ insn->displacement.got && insn->immediate.got;
+}
+
+static inline insn_byte_t insn_vex_m_bits(struct insn *insn)
+{
+ if (insn->vex_prefix.nbytes == 2) /* 2 bytes VEX */
+ return X86_VEX2_M;
+ else if (insn->vex_prefix.nbytes == 3) /* 3 bytes VEX */
+ return X86_VEX3_M(insn->vex_prefix.bytes[1]);
+ else /* EVEX */
+ return X86_EVEX_M(insn->vex_prefix.bytes[1]);
+}
+
+static inline insn_byte_t insn_vex_p_bits(struct insn *insn)
+{
+ if (insn->vex_prefix.nbytes == 2) /* 2 bytes VEX */
+ return X86_VEX_P(insn->vex_prefix.bytes[1]);
+ else
+ return X86_VEX_P(insn->vex_prefix.bytes[2]);
+}
+
+/* Get the last prefix id from last prefix or VEX prefix */
+static inline int insn_last_prefix_id(struct insn *insn)
+{
+ if (insn_is_avx(insn))
+ return insn_vex_p_bits(insn); /* VEX_p is a SIMD prefix id */
+
+ if (insn->prefixes.bytes[3])
+ return inat_get_last_prefix_id(insn->prefixes.bytes[3]);
+
+ return 0;
+}
+
+/* Offset of each field from kaddr */
+static inline int insn_offset_rex_prefix(struct insn *insn)
+{
+ return insn->prefixes.nbytes;
+}
+static inline int insn_offset_vex_prefix(struct insn *insn)
+{
+ return insn_offset_rex_prefix(insn) + insn->rex_prefix.nbytes;
+}
+static inline int insn_offset_opcode(struct insn *insn)
+{
+ return insn_offset_vex_prefix(insn) + insn->vex_prefix.nbytes;
+}
+static inline int insn_offset_modrm(struct insn *insn)
+{
+ return insn_offset_opcode(insn) + insn->opcode.nbytes;
+}
+static inline int insn_offset_sib(struct insn *insn)
+{
+ return insn_offset_modrm(insn) + insn->modrm.nbytes;
+}
+static inline int insn_offset_displacement(struct insn *insn)
+{
+ return insn_offset_sib(insn) + insn->sib.nbytes;
+}
+static inline int insn_offset_immediate(struct insn *insn)
+{
+ return insn_offset_displacement(insn) + insn->displacement.nbytes;
+}
+
+#endif /* _ASM_X86_INSN_H */
diff --git a/tools/objtool/arch/x86/include/asm/orc_types.h b/tools/objtool/arch/x86/include/asm/orc_types.h
new file mode 100644
index 0000000..7dc777a
--- /dev/null
+++ b/tools/objtool/arch/x86/include/asm/orc_types.h
@@ -0,0 +1,107 @@
+/*
+ * Copyright (C) 2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef _ORC_TYPES_H
+#define _ORC_TYPES_H
+
+#include <linux/types.h>
+#include <linux/compiler.h>
+
+/*
+ * The ORC_REG_* registers are base registers which are used to find other
+ * registers on the stack.
+ *
+ * ORC_REG_PREV_SP, also known as DWARF Call Frame Address (CFA), is the
+ * address of the previous frame: the caller's SP before it called the current
+ * function.
+ *
+ * ORC_REG_UNDEFINED means the corresponding register's value didn't change in
+ * the current frame.
+ *
+ * The most commonly used base registers are SP and BP -- which the previous SP
+ * is usually based on -- and PREV_SP and UNDEFINED -- which the previous BP is
+ * usually based on.
+ *
+ * The rest of the base registers are needed for special cases like entry code
+ * and GCC realigned stacks.
+ */
+#define ORC_REG_UNDEFINED 0
+#define ORC_REG_PREV_SP 1
+#define ORC_REG_DX 2
+#define ORC_REG_DI 3
+#define ORC_REG_BP 4
+#define ORC_REG_SP 5
+#define ORC_REG_R10 6
+#define ORC_REG_R13 7
+#define ORC_REG_BP_INDIRECT 8
+#define ORC_REG_SP_INDIRECT 9
+#define ORC_REG_MAX 15
+
+/*
+ * ORC_TYPE_CALL: Indicates that sp_reg+sp_offset resolves to PREV_SP (the
+ * caller's SP right before it made the call). Used for all callable
+ * functions, i.e. all C code and all callable asm functions.
+ *
+ * ORC_TYPE_REGS: Used in entry code to indicate that sp_reg+sp_offset points
+ * to a fully populated pt_regs from a syscall, interrupt, or exception.
+ *
+ * ORC_TYPE_REGS_IRET: Used in entry code to indicate that sp_reg+sp_offset
+ * points to the iret return frame.
+ *
+ * The UNWIND_HINT macros are used only for the unwind_hint struct. They
+ * aren't used in struct orc_entry due to size and complexity constraints.
+ * Objtool converts them to real types when it converts the hints to orc
+ * entries.
+ */
+#define ORC_TYPE_CALL 0
+#define ORC_TYPE_REGS 1
+#define ORC_TYPE_REGS_IRET 2
+#define UNWIND_HINT_TYPE_SAVE 3
+#define UNWIND_HINT_TYPE_RESTORE 4
+
+#ifndef __ASSEMBLY__
+/*
+ * This struct is more or less a vastly simplified version of the DWARF Call
+ * Frame Information standard. It contains only the necessary parts of DWARF
+ * CFI, simplified for ease of access by the in-kernel unwinder. It tells the
+ * unwinder how to find the previous SP and BP (and sometimes entry regs) on
+ * the stack for a given code address. Each instance of the struct corresponds
+ * to one or more code locations.
+ */
+struct orc_entry {
+ s16 sp_offset;
+ s16 bp_offset;
+ unsigned sp_reg:4;
+ unsigned bp_reg:4;
+ unsigned type:2;
+};
+
+/*
+ * This struct is used by asm and inline asm code to manually annotate the
+ * location of registers on the stack for the ORC unwinder.
+ *
+ * Type can be either ORC_TYPE_* or UNWIND_HINT_TYPE_*.
+ */
+struct unwind_hint {
+ u32 ip;
+ s16 sp_offset;
+ u8 sp_reg;
+ u8 type;
+};
+#endif /* __ASSEMBLY__ */
+
+#endif /* _ORC_TYPES_H */
diff --git a/tools/objtool/arch/x86/insn/inat.c b/tools/objtool/arch/x86/insn/inat.c
deleted file mode 100644
index e4bf28e..0000000
--- a/tools/objtool/arch/x86/insn/inat.c
+++ /dev/null
@@ -1,97 +0,0 @@
-/*
- * x86 instruction attribute tables
- *
- * Written by Masami Hiramatsu <mhiramat@redhat.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
- *
- */
-#include "insn.h"
-
-/* Attribute tables are generated from opcode map */
-#include "inat-tables.c"
-
-/* Attribute search APIs */
-insn_attr_t inat_get_opcode_attribute(insn_byte_t opcode)
-{
- return inat_primary_table[opcode];
-}
-
-int inat_get_last_prefix_id(insn_byte_t last_pfx)
-{
- insn_attr_t lpfx_attr;
-
- lpfx_attr = inat_get_opcode_attribute(last_pfx);
- return inat_last_prefix_id(lpfx_attr);
-}
-
-insn_attr_t inat_get_escape_attribute(insn_byte_t opcode, int lpfx_id,
- insn_attr_t esc_attr)
-{
- const insn_attr_t *table;
- int n;
-
- n = inat_escape_id(esc_attr);
-
- table = inat_escape_tables[n][0];
- if (!table)
- return 0;
- if (inat_has_variant(table[opcode]) && lpfx_id) {
- table = inat_escape_tables[n][lpfx_id];
- if (!table)
- return 0;
- }
- return table[opcode];
-}
-
-insn_attr_t inat_get_group_attribute(insn_byte_t modrm, int lpfx_id,
- insn_attr_t grp_attr)
-{
- const insn_attr_t *table;
- int n;
-
- n = inat_group_id(grp_attr);
-
- table = inat_group_tables[n][0];
- if (!table)
- return inat_group_common_attribute(grp_attr);
- if (inat_has_variant(table[X86_MODRM_REG(modrm)]) && lpfx_id) {
- table = inat_group_tables[n][lpfx_id];
- if (!table)
- return inat_group_common_attribute(grp_attr);
- }
- return table[X86_MODRM_REG(modrm)] |
- inat_group_common_attribute(grp_attr);
-}
-
-insn_attr_t inat_get_avx_attribute(insn_byte_t opcode, insn_byte_t vex_m,
- insn_byte_t vex_p)
-{
- const insn_attr_t *table;
- if (vex_m > X86_VEX_M_MAX || vex_p > INAT_LSTPFX_MAX)
- return 0;
- /* At first, this checks the master table */
- table = inat_avx_tables[vex_m][0];
- if (!table)
- return 0;
- if (!inat_is_group(table[opcode]) && vex_p) {
- /* If this is not a group, get attribute directly */
- table = inat_avx_tables[vex_m][vex_p];
- if (!table)
- return 0;
- }
- return table[opcode];
-}
-
diff --git a/tools/objtool/arch/x86/insn/inat.h b/tools/objtool/arch/x86/insn/inat.h
deleted file mode 100644
index 125ecd2..0000000
--- a/tools/objtool/arch/x86/insn/inat.h
+++ /dev/null
@@ -1,234 +0,0 @@
-#ifndef _ASM_X86_INAT_H
-#define _ASM_X86_INAT_H
-/*
- * x86 instruction attributes
- *
- * Written by Masami Hiramatsu <mhiramat@redhat.com>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
- *
- */
-#include "inat_types.h"
-
-/*
- * Internal bits. Don't use bitmasks directly, because these bits are
- * unstable. You should use checking functions.
- */
-
-#define INAT_OPCODE_TABLE_SIZE 256
-#define INAT_GROUP_TABLE_SIZE 8
-
-/* Legacy last prefixes */
-#define INAT_PFX_OPNDSZ 1 /* 0x66 */ /* LPFX1 */
-#define INAT_PFX_REPE 2 /* 0xF3 */ /* LPFX2 */
-#define INAT_PFX_REPNE 3 /* 0xF2 */ /* LPFX3 */
-/* Other Legacy prefixes */
-#define INAT_PFX_LOCK 4 /* 0xF0 */
-#define INAT_PFX_CS 5 /* 0x2E */
-#define INAT_PFX_DS 6 /* 0x3E */
-#define INAT_PFX_ES 7 /* 0x26 */
-#define INAT_PFX_FS 8 /* 0x64 */
-#define INAT_PFX_GS 9 /* 0x65 */
-#define INAT_PFX_SS 10 /* 0x36 */
-#define INAT_PFX_ADDRSZ 11 /* 0x67 */
-/* x86-64 REX prefix */
-#define INAT_PFX_REX 12 /* 0x4X */
-/* AVX VEX prefixes */
-#define INAT_PFX_VEX2 13 /* 2-bytes VEX prefix */
-#define INAT_PFX_VEX3 14 /* 3-bytes VEX prefix */
-#define INAT_PFX_EVEX 15 /* EVEX prefix */
-
-#define INAT_LSTPFX_MAX 3
-#define INAT_LGCPFX_MAX 11
-
-/* Immediate size */
-#define INAT_IMM_BYTE 1
-#define INAT_IMM_WORD 2
-#define INAT_IMM_DWORD 3
-#define INAT_IMM_QWORD 4
-#define INAT_IMM_PTR 5
-#define INAT_IMM_VWORD32 6
-#define INAT_IMM_VWORD 7
-
-/* Legacy prefix */
-#define INAT_PFX_OFFS 0
-#define INAT_PFX_BITS 4
-#define INAT_PFX_MAX ((1 << INAT_PFX_BITS) - 1)
-#define INAT_PFX_MASK (INAT_PFX_MAX << INAT_PFX_OFFS)
-/* Escape opcodes */
-#define INAT_ESC_OFFS (INAT_PFX_OFFS + INAT_PFX_BITS)
-#define INAT_ESC_BITS 2
-#define INAT_ESC_MAX ((1 << INAT_ESC_BITS) - 1)
-#define INAT_ESC_MASK (INAT_ESC_MAX << INAT_ESC_OFFS)
-/* Group opcodes (1-16) */
-#define INAT_GRP_OFFS (INAT_ESC_OFFS + INAT_ESC_BITS)
-#define INAT_GRP_BITS 5
-#define INAT_GRP_MAX ((1 << INAT_GRP_BITS) - 1)
-#define INAT_GRP_MASK (INAT_GRP_MAX << INAT_GRP_OFFS)
-/* Immediates */
-#define INAT_IMM_OFFS (INAT_GRP_OFFS + INAT_GRP_BITS)
-#define INAT_IMM_BITS 3
-#define INAT_IMM_MASK (((1 << INAT_IMM_BITS) - 1) << INAT_IMM_OFFS)
-/* Flags */
-#define INAT_FLAG_OFFS (INAT_IMM_OFFS + INAT_IMM_BITS)
-#define INAT_MODRM (1 << (INAT_FLAG_OFFS))
-#define INAT_FORCE64 (1 << (INAT_FLAG_OFFS + 1))
-#define INAT_SCNDIMM (1 << (INAT_FLAG_OFFS + 2))
-#define INAT_MOFFSET (1 << (INAT_FLAG_OFFS + 3))
-#define INAT_VARIANT (1 << (INAT_FLAG_OFFS + 4))
-#define INAT_VEXOK (1 << (INAT_FLAG_OFFS + 5))
-#define INAT_VEXONLY (1 << (INAT_FLAG_OFFS + 6))
-#define INAT_EVEXONLY (1 << (INAT_FLAG_OFFS + 7))
-/* Attribute making macros for attribute tables */
-#define INAT_MAKE_PREFIX(pfx) (pfx << INAT_PFX_OFFS)
-#define INAT_MAKE_ESCAPE(esc) (esc << INAT_ESC_OFFS)
-#define INAT_MAKE_GROUP(grp) ((grp << INAT_GRP_OFFS) | INAT_MODRM)
-#define INAT_MAKE_IMM(imm) (imm << INAT_IMM_OFFS)
-
-/* Attribute search APIs */
-extern insn_attr_t inat_get_opcode_attribute(insn_byte_t opcode);
-extern int inat_get_last_prefix_id(insn_byte_t last_pfx);
-extern insn_attr_t inat_get_escape_attribute(insn_byte_t opcode,
- int lpfx_id,
- insn_attr_t esc_attr);
-extern insn_attr_t inat_get_group_attribute(insn_byte_t modrm,
- int lpfx_id,
- insn_attr_t esc_attr);
-extern insn_attr_t inat_get_avx_attribute(insn_byte_t opcode,
- insn_byte_t vex_m,
- insn_byte_t vex_pp);
-
-/* Attribute checking functions */
-static inline int inat_is_legacy_prefix(insn_attr_t attr)
-{
- attr &= INAT_PFX_MASK;
- return attr && attr <= INAT_LGCPFX_MAX;
-}
-
-static inline int inat_is_address_size_prefix(insn_attr_t attr)
-{
- return (attr & INAT_PFX_MASK) == INAT_PFX_ADDRSZ;
-}
-
-static inline int inat_is_operand_size_prefix(insn_attr_t attr)
-{
- return (attr & INAT_PFX_MASK) == INAT_PFX_OPNDSZ;
-}
-
-static inline int inat_is_rex_prefix(insn_attr_t attr)
-{
- return (attr & INAT_PFX_MASK) == INAT_PFX_REX;
-}
-
-static inline int inat_last_prefix_id(insn_attr_t attr)
-{
- if ((attr & INAT_PFX_MASK) > INAT_LSTPFX_MAX)
- return 0;
- else
- return attr & INAT_PFX_MASK;
-}
-
-static inline int inat_is_vex_prefix(insn_attr_t attr)
-{
- attr &= INAT_PFX_MASK;
- return attr == INAT_PFX_VEX2 || attr == INAT_PFX_VEX3 ||
- attr == INAT_PFX_EVEX;
-}
-
-static inline int inat_is_evex_prefix(insn_attr_t attr)
-{
- return (attr & INAT_PFX_MASK) == INAT_PFX_EVEX;
-}
-
-static inline int inat_is_vex3_prefix(insn_attr_t attr)
-{
- return (attr & INAT_PFX_MASK) == INAT_PFX_VEX3;
-}
-
-static inline int inat_is_escape(insn_attr_t attr)
-{
- return attr & INAT_ESC_MASK;
-}
-
-static inline int inat_escape_id(insn_attr_t attr)
-{
- return (attr & INAT_ESC_MASK) >> INAT_ESC_OFFS;
-}
-
-static inline int inat_is_group(insn_attr_t attr)
-{
- return attr & INAT_GRP_MASK;
-}
-
-static inline int inat_group_id(insn_attr_t attr)
-{
- return (attr & INAT_GRP_MASK) >> INAT_GRP_OFFS;
-}
-
-static inline int inat_group_common_attribute(insn_attr_t attr)
-{
- return attr & ~INAT_GRP_MASK;
-}
-
-static inline int inat_has_immediate(insn_attr_t attr)
-{
- return attr & INAT_IMM_MASK;
-}
-
-static inline int inat_immediate_size(insn_attr_t attr)
-{
- return (attr & INAT_IMM_MASK) >> INAT_IMM_OFFS;
-}
-
-static inline int inat_has_modrm(insn_attr_t attr)
-{
- return attr & INAT_MODRM;
-}
-
-static inline int inat_is_force64(insn_attr_t attr)
-{
- return attr & INAT_FORCE64;
-}
-
-static inline int inat_has_second_immediate(insn_attr_t attr)
-{
- return attr & INAT_SCNDIMM;
-}
-
-static inline int inat_has_moffset(insn_attr_t attr)
-{
- return attr & INAT_MOFFSET;
-}
-
-static inline int inat_has_variant(insn_attr_t attr)
-{
- return attr & INAT_VARIANT;
-}
-
-static inline int inat_accept_vex(insn_attr_t attr)
-{
- return attr & INAT_VEXOK;
-}
-
-static inline int inat_must_vex(insn_attr_t attr)
-{
- return attr & (INAT_VEXONLY | INAT_EVEXONLY);
-}
-
-static inline int inat_must_evex(insn_attr_t attr)
-{
- return attr & INAT_EVEXONLY;
-}
-#endif
diff --git a/tools/objtool/arch/x86/insn/insn.c b/tools/objtool/arch/x86/insn/insn.c
deleted file mode 100644
index ca983e2..0000000
--- a/tools/objtool/arch/x86/insn/insn.c
+++ /dev/null
@@ -1,606 +0,0 @@
-/*
- * x86 instruction analysis
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
- *
- * Copyright (C) IBM Corporation, 2002, 2004, 2009
- */
-
-#ifdef __KERNEL__
-#include <linux/string.h>
-#else
-#include <string.h>
-#endif
-#include "inat.h"
-#include "insn.h"
-
-/* Verify next sizeof(t) bytes can be on the same instruction */
-#define validate_next(t, insn, n) \
- ((insn)->next_byte + sizeof(t) + n <= (insn)->end_kaddr)
-
-#define __get_next(t, insn) \
- ({ t r = *(t*)insn->next_byte; insn->next_byte += sizeof(t); r; })
-
-#define __peek_nbyte_next(t, insn, n) \
- ({ t r = *(t*)((insn)->next_byte + n); r; })
-
-#define get_next(t, insn) \
- ({ if (unlikely(!validate_next(t, insn, 0))) goto err_out; __get_next(t, insn); })
-
-#define peek_nbyte_next(t, insn, n) \
- ({ if (unlikely(!validate_next(t, insn, n))) goto err_out; __peek_nbyte_next(t, insn, n); })
-
-#define peek_next(t, insn) peek_nbyte_next(t, insn, 0)
-
-/**
- * insn_init() - initialize struct insn
- * @insn: &struct insn to be initialized
- * @kaddr: address (in kernel memory) of instruction (or copy thereof)
- * @x86_64: !0 for 64-bit kernel or 64-bit app
- */
-void insn_init(struct insn *insn, const void *kaddr, int buf_len, int x86_64)
-{
- /*
- * Instructions longer than MAX_INSN_SIZE (15 bytes) are invalid
- * even if the input buffer is long enough to hold them.
- */
- if (buf_len > MAX_INSN_SIZE)
- buf_len = MAX_INSN_SIZE;
-
- memset(insn, 0, sizeof(*insn));
- insn->kaddr = kaddr;
- insn->end_kaddr = kaddr + buf_len;
- insn->next_byte = kaddr;
- insn->x86_64 = x86_64 ? 1 : 0;
- insn->opnd_bytes = 4;
- if (x86_64)
- insn->addr_bytes = 8;
- else
- insn->addr_bytes = 4;
-}
-
-/**
- * insn_get_prefixes - scan x86 instruction prefix bytes
- * @insn: &struct insn containing instruction
- *
- * Populates the @insn->prefixes bitmap, and updates @insn->next_byte
- * to point to the (first) opcode. No effect if @insn->prefixes.got
- * is already set.
- */
-void insn_get_prefixes(struct insn *insn)
-{
- struct insn_field *prefixes = &insn->prefixes;
- insn_attr_t attr;
- insn_byte_t b, lb;
- int i, nb;
-
- if (prefixes->got)
- return;
-
- nb = 0;
- lb = 0;
- b = peek_next(insn_byte_t, insn);
- attr = inat_get_opcode_attribute(b);
- while (inat_is_legacy_prefix(attr)) {
- /* Skip if same prefix */
- for (i = 0; i < nb; i++)
- if (prefixes->bytes[i] == b)
- goto found;
- if (nb == 4)
- /* Invalid instruction */
- break;
- prefixes->bytes[nb++] = b;
- if (inat_is_address_size_prefix(attr)) {
- /* address size switches 2/4 or 4/8 */
- if (insn->x86_64)
- insn->addr_bytes ^= 12;
- else
- insn->addr_bytes ^= 6;
- } else if (inat_is_operand_size_prefix(attr)) {
- /* oprand size switches 2/4 */
- insn->opnd_bytes ^= 6;
- }
-found:
- prefixes->nbytes++;
- insn->next_byte++;
- lb = b;
- b = peek_next(insn_byte_t, insn);
- attr = inat_get_opcode_attribute(b);
- }
- /* Set the last prefix */
- if (lb && lb != insn->prefixes.bytes[3]) {
- if (unlikely(insn->prefixes.bytes[3])) {
- /* Swap the last prefix */
- b = insn->prefixes.bytes[3];
- for (i = 0; i < nb; i++)
- if (prefixes->bytes[i] == lb)
- prefixes->bytes[i] = b;
- }
- insn->prefixes.bytes[3] = lb;
- }
-
- /* Decode REX prefix */
- if (insn->x86_64) {
- b = peek_next(insn_byte_t, insn);
- attr = inat_get_opcode_attribute(b);
- if (inat_is_rex_prefix(attr)) {
- insn->rex_prefix.value = b;
- insn->rex_prefix.nbytes = 1;
- insn->next_byte++;
- if (X86_REX_W(b))
- /* REX.W overrides opnd_size */
- insn->opnd_bytes = 8;
- }
- }
- insn->rex_prefix.got = 1;
-
- /* Decode VEX prefix */
- b = peek_next(insn_byte_t, insn);
- attr = inat_get_opcode_attribute(b);
- if (inat_is_vex_prefix(attr)) {
- insn_byte_t b2 = peek_nbyte_next(insn_byte_t, insn, 1);
- if (!insn->x86_64) {
- /*
- * In 32-bits mode, if the [7:6] bits (mod bits of
- * ModRM) on the second byte are not 11b, it is
- * LDS or LES or BOUND.
- */
- if (X86_MODRM_MOD(b2) != 3)
- goto vex_end;
- }
- insn->vex_prefix.bytes[0] = b;
- insn->vex_prefix.bytes[1] = b2;
- if (inat_is_evex_prefix(attr)) {
- b2 = peek_nbyte_next(insn_byte_t, insn, 2);
- insn->vex_prefix.bytes[2] = b2;
- b2 = peek_nbyte_next(insn_byte_t, insn, 3);
- insn->vex_prefix.bytes[3] = b2;
- insn->vex_prefix.nbytes = 4;
- insn->next_byte += 4;
- if (insn->x86_64 && X86_VEX_W(b2))
- /* VEX.W overrides opnd_size */
- insn->opnd_bytes = 8;
- } else if (inat_is_vex3_prefix(attr)) {
- b2 = peek_nbyte_next(insn_byte_t, insn, 2);
- insn->vex_prefix.bytes[2] = b2;
- insn->vex_prefix.nbytes = 3;
- insn->next_byte += 3;
- if (insn->x86_64 && X86_VEX_W(b2))
- /* VEX.W overrides opnd_size */
- insn->opnd_bytes = 8;
- } else {
- /*
- * For VEX2, fake VEX3-like byte#2.
- * Makes it easier to decode vex.W, vex.vvvv,
- * vex.L and vex.pp. Masking with 0x7f sets vex.W == 0.
- */
- insn->vex_prefix.bytes[2] = b2 & 0x7f;
- insn->vex_prefix.nbytes = 2;
- insn->next_byte += 2;
- }
- }
-vex_end:
- insn->vex_prefix.got = 1;
-
- prefixes->got = 1;
-
-err_out:
- return;
-}
-
-/**
- * insn_get_opcode - collect opcode(s)
- * @insn: &struct insn containing instruction
- *
- * Populates @insn->opcode, updates @insn->next_byte to point past the
- * opcode byte(s), and set @insn->attr (except for groups).
- * If necessary, first collects any preceding (prefix) bytes.
- * Sets @insn->opcode.value = opcode1. No effect if @insn->opcode.got
- * is already 1.
- */
-void insn_get_opcode(struct insn *insn)
-{
- struct insn_field *opcode = &insn->opcode;
- insn_byte_t op;
- int pfx_id;
- if (opcode->got)
- return;
- if (!insn->prefixes.got)
- insn_get_prefixes(insn);
-
- /* Get first opcode */
- op = get_next(insn_byte_t, insn);
- opcode->bytes[0] = op;
- opcode->nbytes = 1;
-
- /* Check if there is VEX prefix or not */
- if (insn_is_avx(insn)) {
- insn_byte_t m, p;
- m = insn_vex_m_bits(insn);
- p = insn_vex_p_bits(insn);
- insn->attr = inat_get_avx_attribute(op, m, p);
- if ((inat_must_evex(insn->attr) && !insn_is_evex(insn)) ||
- (!inat_accept_vex(insn->attr) &&
- !inat_is_group(insn->attr)))
- insn->attr = 0; /* This instruction is bad */
- goto end; /* VEX has only 1 byte for opcode */
- }
-
- insn->attr = inat_get_opcode_attribute(op);
- while (inat_is_escape(insn->attr)) {
- /* Get escaped opcode */
- op = get_next(insn_byte_t, insn);
- opcode->bytes[opcode->nbytes++] = op;
- pfx_id = insn_last_prefix_id(insn);
- insn->attr = inat_get_escape_attribute(op, pfx_id, insn->attr);
- }
- if (inat_must_vex(insn->attr))
- insn->attr = 0; /* This instruction is bad */
-end:
- opcode->got = 1;
-
-err_out:
- return;
-}
-
-/**
- * insn_get_modrm - collect ModRM byte, if any
- * @insn: &struct insn containing instruction
- *
- * Populates @insn->modrm and updates @insn->next_byte to point past the
- * ModRM byte, if any. If necessary, first collects the preceding bytes
- * (prefixes and opcode(s)). No effect if @insn->modrm.got is already 1.
- */
-void insn_get_modrm(struct insn *insn)
-{
- struct insn_field *modrm = &insn->modrm;
- insn_byte_t pfx_id, mod;
- if (modrm->got)
- return;
- if (!insn->opcode.got)
- insn_get_opcode(insn);
-
- if (inat_has_modrm(insn->attr)) {
- mod = get_next(insn_byte_t, insn);
- modrm->value = mod;
- modrm->nbytes = 1;
- if (inat_is_group(insn->attr)) {
- pfx_id = insn_last_prefix_id(insn);
- insn->attr = inat_get_group_attribute(mod, pfx_id,
- insn->attr);
- if (insn_is_avx(insn) && !inat_accept_vex(insn->attr))
- insn->attr = 0; /* This is bad */
- }
- }
-
- if (insn->x86_64 && inat_is_force64(insn->attr))
- insn->opnd_bytes = 8;
- modrm->got = 1;
-
-err_out:
- return;
-}
-
-
-/**
- * insn_rip_relative() - Does instruction use RIP-relative addressing mode?
- * @insn: &struct insn containing instruction
- *
- * If necessary, first collects the instruction up to and including the
- * ModRM byte. No effect if @insn->x86_64 is 0.
- */
-int insn_rip_relative(struct insn *insn)
-{
- struct insn_field *modrm = &insn->modrm;
-
- if (!insn->x86_64)
- return 0;
- if (!modrm->got)
- insn_get_modrm(insn);
- /*
- * For rip-relative instructions, the mod field (top 2 bits)
- * is zero and the r/m field (bottom 3 bits) is 0x5.
- */
- return (modrm->nbytes && (modrm->value & 0xc7) == 0x5);
-}
-
-/**
- * insn_get_sib() - Get the SIB byte of instruction
- * @insn: &struct insn containing instruction
- *
- * If necessary, first collects the instruction up to and including the
- * ModRM byte.
- */
-void insn_get_sib(struct insn *insn)
-{
- insn_byte_t modrm;
-
- if (insn->sib.got)
- return;
- if (!insn->modrm.got)
- insn_get_modrm(insn);
- if (insn->modrm.nbytes) {
- modrm = (insn_byte_t)insn->modrm.value;
- if (insn->addr_bytes != 2 &&
- X86_MODRM_MOD(modrm) != 3 && X86_MODRM_RM(modrm) == 4) {
- insn->sib.value = get_next(insn_byte_t, insn);
- insn->sib.nbytes = 1;
- }
- }
- insn->sib.got = 1;
-
-err_out:
- return;
-}
-
-
-/**
- * insn_get_displacement() - Get the displacement of instruction
- * @insn: &struct insn containing instruction
- *
- * If necessary, first collects the instruction up to and including the
- * SIB byte.
- * Displacement value is sign-expanded.
- */
-void insn_get_displacement(struct insn *insn)
-{
- insn_byte_t mod, rm, base;
-
- if (insn->displacement.got)
- return;
- if (!insn->sib.got)
- insn_get_sib(insn);
- if (insn->modrm.nbytes) {
- /*
- * Interpreting the modrm byte:
- * mod = 00 - no displacement fields (exceptions below)
- * mod = 01 - 1-byte displacement field
- * mod = 10 - displacement field is 4 bytes, or 2 bytes if
- * address size = 2 (0x67 prefix in 32-bit mode)
- * mod = 11 - no memory operand
- *
- * If address size = 2...
- * mod = 00, r/m = 110 - displacement field is 2 bytes
- *
- * If address size != 2...
- * mod != 11, r/m = 100 - SIB byte exists
- * mod = 00, SIB base = 101 - displacement field is 4 bytes
- * mod = 00, r/m = 101 - rip-relative addressing, displacement
- * field is 4 bytes
- */
- mod = X86_MODRM_MOD(insn->modrm.value);
- rm = X86_MODRM_RM(insn->modrm.value);
- base = X86_SIB_BASE(insn->sib.value);
- if (mod == 3)
- goto out;
- if (mod == 1) {
- insn->displacement.value = get_next(signed char, insn);
- insn->displacement.nbytes = 1;
- } else if (insn->addr_bytes == 2) {
- if ((mod == 0 && rm == 6) || mod == 2) {
- insn->displacement.value =
- get_next(short, insn);
- insn->displacement.nbytes = 2;
- }
- } else {
- if ((mod == 0 && rm == 5) || mod == 2 ||
- (mod == 0 && base == 5)) {
- insn->displacement.value = get_next(int, insn);
- insn->displacement.nbytes = 4;
- }
- }
- }
-out:
- insn->displacement.got = 1;
-
-err_out:
- return;
-}
-
-/* Decode moffset16/32/64. Return 0 if failed */
-static int __get_moffset(struct insn *insn)
-{
- switch (insn->addr_bytes) {
- case 2:
- insn->moffset1.value = get_next(short, insn);
- insn->moffset1.nbytes = 2;
- break;
- case 4:
- insn->moffset1.value = get_next(int, insn);
- insn->moffset1.nbytes = 4;
- break;
- case 8:
- insn->moffset1.value = get_next(int, insn);
- insn->moffset1.nbytes = 4;
- insn->moffset2.value = get_next(int, insn);
- insn->moffset2.nbytes = 4;
- break;
- default: /* opnd_bytes must be modified manually */
- goto err_out;
- }
- insn->moffset1.got = insn->moffset2.got = 1;
-
- return 1;
-
-err_out:
- return 0;
-}
-
-/* Decode imm v32(Iz). Return 0 if failed */
-static int __get_immv32(struct insn *insn)
-{
- switch (insn->opnd_bytes) {
- case 2:
- insn->immediate.value = get_next(short, insn);
- insn->immediate.nbytes = 2;
- break;
- case 4:
- case 8:
- insn->immediate.value = get_next(int, insn);
- insn->immediate.nbytes = 4;
- break;
- default: /* opnd_bytes must be modified manually */
- goto err_out;
- }
-
- return 1;
-
-err_out:
- return 0;
-}
-
-/* Decode imm v64(Iv/Ov), Return 0 if failed */
-static int __get_immv(struct insn *insn)
-{
- switch (insn->opnd_bytes) {
- case 2:
- insn->immediate1.value = get_next(short, insn);
- insn->immediate1.nbytes = 2;
- break;
- case 4:
- insn->immediate1.value = get_next(int, insn);
- insn->immediate1.nbytes = 4;
- break;
- case 8:
- insn->immediate1.value = get_next(int, insn);
- insn->immediate1.nbytes = 4;
- insn->immediate2.value = get_next(int, insn);
- insn->immediate2.nbytes = 4;
- break;
- default: /* opnd_bytes must be modified manually */
- goto err_out;
- }
- insn->immediate1.got = insn->immediate2.got = 1;
-
- return 1;
-err_out:
- return 0;
-}
-
-/* Decode ptr16:16/32(Ap) */
-static int __get_immptr(struct insn *insn)
-{
- switch (insn->opnd_bytes) {
- case 2:
- insn->immediate1.value = get_next(short, insn);
- insn->immediate1.nbytes = 2;
- break;
- case 4:
- insn->immediate1.value = get_next(int, insn);
- insn->immediate1.nbytes = 4;
- break;
- case 8:
- /* ptr16:64 is not exist (no segment) */
- return 0;
- default: /* opnd_bytes must be modified manually */
- goto err_out;
- }
- insn->immediate2.value = get_next(unsigned short, insn);
- insn->immediate2.nbytes = 2;
- insn->immediate1.got = insn->immediate2.got = 1;
-
- return 1;
-err_out:
- return 0;
-}
-
-/**
- * insn_get_immediate() - Get the immediates of instruction
- * @insn: &struct insn containing instruction
- *
- * If necessary, first collects the instruction up to and including the
- * displacement bytes.
- * Basically, most of immediates are sign-expanded. Unsigned-value can be
- * get by bit masking with ((1 << (nbytes * 8)) - 1)
- */
-void insn_get_immediate(struct insn *insn)
-{
- if (insn->immediate.got)
- return;
- if (!insn->displacement.got)
- insn_get_displacement(insn);
-
- if (inat_has_moffset(insn->attr)) {
- if (!__get_moffset(insn))
- goto err_out;
- goto done;
- }
-
- if (!inat_has_immediate(insn->attr))
- /* no immediates */
- goto done;
-
- switch (inat_immediate_size(insn->attr)) {
- case INAT_IMM_BYTE:
- insn->immediate.value = get_next(signed char, insn);
- insn->immediate.nbytes = 1;
- break;
- case INAT_IMM_WORD:
- insn->immediate.value = get_next(short, insn);
- insn->immediate.nbytes = 2;
- break;
- case INAT_IMM_DWORD:
- insn->immediate.value = get_next(int, insn);
- insn->immediate.nbytes = 4;
- break;
- case INAT_IMM_QWORD:
- insn->immediate1.value = get_next(int, insn);
- insn->immediate1.nbytes = 4;
- insn->immediate2.value = get_next(int, insn);
- insn->immediate2.nbytes = 4;
- break;
- case INAT_IMM_PTR:
- if (!__get_immptr(insn))
- goto err_out;
- break;
- case INAT_IMM_VWORD32:
- if (!__get_immv32(insn))
- goto err_out;
- break;
- case INAT_IMM_VWORD:
- if (!__get_immv(insn))
- goto err_out;
- break;
- default:
- /* Here, insn must have an immediate, but failed */
- goto err_out;
- }
- if (inat_has_second_immediate(insn->attr)) {
- insn->immediate2.value = get_next(signed char, insn);
- insn->immediate2.nbytes = 1;
- }
-done:
- insn->immediate.got = 1;
-
-err_out:
- return;
-}
-
-/**
- * insn_get_length() - Get the length of instruction
- * @insn: &struct insn containing instruction
- *
- * If necessary, first collects the instruction up to and including the
- * immediates bytes.
- */
-void insn_get_length(struct insn *insn)
-{
- if (insn->length)
- return;
- if (!insn->immediate.got)
- insn_get_immediate(insn);
- insn->length = (unsigned char)((unsigned long)insn->next_byte
- - (unsigned long)insn->kaddr);
-}
diff --git a/tools/objtool/arch/x86/insn/insn.h b/tools/objtool/arch/x86/insn/insn.h
deleted file mode 100644
index e23578c..0000000
--- a/tools/objtool/arch/x86/insn/insn.h
+++ /dev/null
@@ -1,211 +0,0 @@
-#ifndef _ASM_X86_INSN_H
-#define _ASM_X86_INSN_H
-/*
- * x86 instruction analysis
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
- *
- * Copyright (C) IBM Corporation, 2009
- */
-
-/* insn_attr_t is defined in inat.h */
-#include "inat.h"
-
-struct insn_field {
- union {
- insn_value_t value;
- insn_byte_t bytes[4];
- };
- /* !0 if we've run insn_get_xxx() for this field */
- unsigned char got;
- unsigned char nbytes;
-};
-
-struct insn {
- struct insn_field prefixes; /*
- * Prefixes
- * prefixes.bytes[3]: last prefix
- */
- struct insn_field rex_prefix; /* REX prefix */
- struct insn_field vex_prefix; /* VEX prefix */
- struct insn_field opcode; /*
- * opcode.bytes[0]: opcode1
- * opcode.bytes[1]: opcode2
- * opcode.bytes[2]: opcode3
- */
- struct insn_field modrm;
- struct insn_field sib;
- struct insn_field displacement;
- union {
- struct insn_field immediate;
- struct insn_field moffset1; /* for 64bit MOV */
- struct insn_field immediate1; /* for 64bit imm or off16/32 */
- };
- union {
- struct insn_field moffset2; /* for 64bit MOV */
- struct insn_field immediate2; /* for 64bit imm or seg16 */
- };
-
- insn_attr_t attr;
- unsigned char opnd_bytes;
- unsigned char addr_bytes;
- unsigned char length;
- unsigned char x86_64;
-
- const insn_byte_t *kaddr; /* kernel address of insn to analyze */
- const insn_byte_t *end_kaddr; /* kernel address of last insn in buffer */
- const insn_byte_t *next_byte;
-};
-
-#define MAX_INSN_SIZE 15
-
-#define X86_MODRM_MOD(modrm) (((modrm) & 0xc0) >> 6)
-#define X86_MODRM_REG(modrm) (((modrm) & 0x38) >> 3)
-#define X86_MODRM_RM(modrm) ((modrm) & 0x07)
-
-#define X86_SIB_SCALE(sib) (((sib) & 0xc0) >> 6)
-#define X86_SIB_INDEX(sib) (((sib) & 0x38) >> 3)
-#define X86_SIB_BASE(sib) ((sib) & 0x07)
-
-#define X86_REX_W(rex) ((rex) & 8)
-#define X86_REX_R(rex) ((rex) & 4)
-#define X86_REX_X(rex) ((rex) & 2)
-#define X86_REX_B(rex) ((rex) & 1)
-
-/* VEX bit flags */
-#define X86_VEX_W(vex) ((vex) & 0x80) /* VEX3 Byte2 */
-#define X86_VEX_R(vex) ((vex) & 0x80) /* VEX2/3 Byte1 */
-#define X86_VEX_X(vex) ((vex) & 0x40) /* VEX3 Byte1 */
-#define X86_VEX_B(vex) ((vex) & 0x20) /* VEX3 Byte1 */
-#define X86_VEX_L(vex) ((vex) & 0x04) /* VEX3 Byte2, VEX2 Byte1 */
-/* VEX bit fields */
-#define X86_EVEX_M(vex) ((vex) & 0x03) /* EVEX Byte1 */
-#define X86_VEX3_M(vex) ((vex) & 0x1f) /* VEX3 Byte1 */
-#define X86_VEX2_M 1 /* VEX2.M always 1 */
-#define X86_VEX_V(vex) (((vex) & 0x78) >> 3) /* VEX3 Byte2, VEX2 Byte1 */
-#define X86_VEX_P(vex) ((vex) & 0x03) /* VEX3 Byte2, VEX2 Byte1 */
-#define X86_VEX_M_MAX 0x1f /* VEX3.M Maximum value */
-
-extern void insn_init(struct insn *insn, const void *kaddr, int buf_len, int x86_64);
-extern void insn_get_prefixes(struct insn *insn);
-extern void insn_get_opcode(struct insn *insn);
-extern void insn_get_modrm(struct insn *insn);
-extern void insn_get_sib(struct insn *insn);
-extern void insn_get_displacement(struct insn *insn);
-extern void insn_get_immediate(struct insn *insn);
-extern void insn_get_length(struct insn *insn);
-
-/* Attribute will be determined after getting ModRM (for opcode groups) */
-static inline void insn_get_attribute(struct insn *insn)
-{
- insn_get_modrm(insn);
-}
-
-/* Instruction uses RIP-relative addressing */
-extern int insn_rip_relative(struct insn *insn);
-
-/* Init insn for kernel text */
-static inline void kernel_insn_init(struct insn *insn,
- const void *kaddr, int buf_len)
-{
-#ifdef CONFIG_X86_64
- insn_init(insn, kaddr, buf_len, 1);
-#else /* CONFIG_X86_32 */
- insn_init(insn, kaddr, buf_len, 0);
-#endif
-}
-
-static inline int insn_is_avx(struct insn *insn)
-{
- if (!insn->prefixes.got)
- insn_get_prefixes(insn);
- return (insn->vex_prefix.value != 0);
-}
-
-static inline int insn_is_evex(struct insn *insn)
-{
- if (!insn->prefixes.got)
- insn_get_prefixes(insn);
- return (insn->vex_prefix.nbytes == 4);
-}
-
-/* Ensure this instruction is decoded completely */
-static inline int insn_complete(struct insn *insn)
-{
- return insn->opcode.got && insn->modrm.got && insn->sib.got &&
- insn->displacement.got && insn->immediate.got;
-}
-
-static inline insn_byte_t insn_vex_m_bits(struct insn *insn)
-{
- if (insn->vex_prefix.nbytes == 2) /* 2 bytes VEX */
- return X86_VEX2_M;
- else if (insn->vex_prefix.nbytes == 3) /* 3 bytes VEX */
- return X86_VEX3_M(insn->vex_prefix.bytes[1]);
- else /* EVEX */
- return X86_EVEX_M(insn->vex_prefix.bytes[1]);
-}
-
-static inline insn_byte_t insn_vex_p_bits(struct insn *insn)
-{
- if (insn->vex_prefix.nbytes == 2) /* 2 bytes VEX */
- return X86_VEX_P(insn->vex_prefix.bytes[1]);
- else
- return X86_VEX_P(insn->vex_prefix.bytes[2]);
-}
-
-/* Get the last prefix id from last prefix or VEX prefix */
-static inline int insn_last_prefix_id(struct insn *insn)
-{
- if (insn_is_avx(insn))
- return insn_vex_p_bits(insn); /* VEX_p is a SIMD prefix id */
-
- if (insn->prefixes.bytes[3])
- return inat_get_last_prefix_id(insn->prefixes.bytes[3]);
-
- return 0;
-}
-
-/* Offset of each field from kaddr */
-static inline int insn_offset_rex_prefix(struct insn *insn)
-{
- return insn->prefixes.nbytes;
-}
-static inline int insn_offset_vex_prefix(struct insn *insn)
-{
- return insn_offset_rex_prefix(insn) + insn->rex_prefix.nbytes;
-}
-static inline int insn_offset_opcode(struct insn *insn)
-{
- return insn_offset_vex_prefix(insn) + insn->vex_prefix.nbytes;
-}
-static inline int insn_offset_modrm(struct insn *insn)
-{
- return insn_offset_opcode(insn) + insn->opcode.nbytes;
-}
-static inline int insn_offset_sib(struct insn *insn)
-{
- return insn_offset_modrm(insn) + insn->modrm.nbytes;
-}
-static inline int insn_offset_displacement(struct insn *insn)
-{
- return insn_offset_sib(insn) + insn->sib.nbytes;
-}
-static inline int insn_offset_immediate(struct insn *insn)
-{
- return insn_offset_displacement(insn) + insn->displacement.nbytes;
-}
-
-#endif /* _ASM_X86_INSN_H */
diff --git a/tools/objtool/arch/x86/lib/inat.c b/tools/objtool/arch/x86/lib/inat.c
new file mode 100644
index 0000000..c1f01a8
--- /dev/null
+++ b/tools/objtool/arch/x86/lib/inat.c
@@ -0,0 +1,97 @@
+/*
+ * x86 instruction attribute tables
+ *
+ * Written by Masami Hiramatsu <mhiramat@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ */
+#include <asm/insn.h>
+
+/* Attribute tables are generated from opcode map */
+#include "inat-tables.c"
+
+/* Attribute search APIs */
+insn_attr_t inat_get_opcode_attribute(insn_byte_t opcode)
+{
+ return inat_primary_table[opcode];
+}
+
+int inat_get_last_prefix_id(insn_byte_t last_pfx)
+{
+ insn_attr_t lpfx_attr;
+
+ lpfx_attr = inat_get_opcode_attribute(last_pfx);
+ return inat_last_prefix_id(lpfx_attr);
+}
+
+insn_attr_t inat_get_escape_attribute(insn_byte_t opcode, int lpfx_id,
+ insn_attr_t esc_attr)
+{
+ const insn_attr_t *table;
+ int n;
+
+ n = inat_escape_id(esc_attr);
+
+ table = inat_escape_tables[n][0];
+ if (!table)
+ return 0;
+ if (inat_has_variant(table[opcode]) && lpfx_id) {
+ table = inat_escape_tables[n][lpfx_id];
+ if (!table)
+ return 0;
+ }
+ return table[opcode];
+}
+
+insn_attr_t inat_get_group_attribute(insn_byte_t modrm, int lpfx_id,
+ insn_attr_t grp_attr)
+{
+ const insn_attr_t *table;
+ int n;
+
+ n = inat_group_id(grp_attr);
+
+ table = inat_group_tables[n][0];
+ if (!table)
+ return inat_group_common_attribute(grp_attr);
+ if (inat_has_variant(table[X86_MODRM_REG(modrm)]) && lpfx_id) {
+ table = inat_group_tables[n][lpfx_id];
+ if (!table)
+ return inat_group_common_attribute(grp_attr);
+ }
+ return table[X86_MODRM_REG(modrm)] |
+ inat_group_common_attribute(grp_attr);
+}
+
+insn_attr_t inat_get_avx_attribute(insn_byte_t opcode, insn_byte_t vex_m,
+ insn_byte_t vex_p)
+{
+ const insn_attr_t *table;
+ if (vex_m > X86_VEX_M_MAX || vex_p > INAT_LSTPFX_MAX)
+ return 0;
+ /* At first, this checks the master table */
+ table = inat_avx_tables[vex_m][0];
+ if (!table)
+ return 0;
+ if (!inat_is_group(table[opcode]) && vex_p) {
+ /* If this is not a group, get attribute directly */
+ table = inat_avx_tables[vex_m][vex_p];
+ if (!table)
+ return 0;
+ }
+ return table[opcode];
+}
+
diff --git a/tools/objtool/arch/x86/lib/insn.c b/tools/objtool/arch/x86/lib/insn.c
new file mode 100644
index 0000000..1088eb8
--- /dev/null
+++ b/tools/objtool/arch/x86/lib/insn.c
@@ -0,0 +1,606 @@
+/*
+ * x86 instruction analysis
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ * Copyright (C) IBM Corporation, 2002, 2004, 2009
+ */
+
+#ifdef __KERNEL__
+#include <linux/string.h>
+#else
+#include <string.h>
+#endif
+#include <asm/inat.h>
+#include <asm/insn.h>
+
+/* Verify next sizeof(t) bytes can be on the same instruction */
+#define validate_next(t, insn, n) \
+ ((insn)->next_byte + sizeof(t) + n <= (insn)->end_kaddr)
+
+#define __get_next(t, insn) \
+ ({ t r = *(t*)insn->next_byte; insn->next_byte += sizeof(t); r; })
+
+#define __peek_nbyte_next(t, insn, n) \
+ ({ t r = *(t*)((insn)->next_byte + n); r; })
+
+#define get_next(t, insn) \
+ ({ if (unlikely(!validate_next(t, insn, 0))) goto err_out; __get_next(t, insn); })
+
+#define peek_nbyte_next(t, insn, n) \
+ ({ if (unlikely(!validate_next(t, insn, n))) goto err_out; __peek_nbyte_next(t, insn, n); })
+
+#define peek_next(t, insn) peek_nbyte_next(t, insn, 0)
+
+/**
+ * insn_init() - initialize struct insn
+ * @insn: &struct insn to be initialized
+ * @kaddr: address (in kernel memory) of instruction (or copy thereof)
+ * @x86_64: !0 for 64-bit kernel or 64-bit app
+ */
+void insn_init(struct insn *insn, const void *kaddr, int buf_len, int x86_64)
+{
+ /*
+ * Instructions longer than MAX_INSN_SIZE (15 bytes) are invalid
+ * even if the input buffer is long enough to hold them.
+ */
+ if (buf_len > MAX_INSN_SIZE)
+ buf_len = MAX_INSN_SIZE;
+
+ memset(insn, 0, sizeof(*insn));
+ insn->kaddr = kaddr;
+ insn->end_kaddr = kaddr + buf_len;
+ insn->next_byte = kaddr;
+ insn->x86_64 = x86_64 ? 1 : 0;
+ insn->opnd_bytes = 4;
+ if (x86_64)
+ insn->addr_bytes = 8;
+ else
+ insn->addr_bytes = 4;
+}
+
+/**
+ * insn_get_prefixes - scan x86 instruction prefix bytes
+ * @insn: &struct insn containing instruction
+ *
+ * Populates the @insn->prefixes bitmap, and updates @insn->next_byte
+ * to point to the (first) opcode. No effect if @insn->prefixes.got
+ * is already set.
+ */
+void insn_get_prefixes(struct insn *insn)
+{
+ struct insn_field *prefixes = &insn->prefixes;
+ insn_attr_t attr;
+ insn_byte_t b, lb;
+ int i, nb;
+
+ if (prefixes->got)
+ return;
+
+ nb = 0;
+ lb = 0;
+ b = peek_next(insn_byte_t, insn);
+ attr = inat_get_opcode_attribute(b);
+ while (inat_is_legacy_prefix(attr)) {
+ /* Skip if same prefix */
+ for (i = 0; i < nb; i++)
+ if (prefixes->bytes[i] == b)
+ goto found;
+ if (nb == 4)
+ /* Invalid instruction */
+ break;
+ prefixes->bytes[nb++] = b;
+ if (inat_is_address_size_prefix(attr)) {
+ /* address size switches 2/4 or 4/8 */
+ if (insn->x86_64)
+ insn->addr_bytes ^= 12;
+ else
+ insn->addr_bytes ^= 6;
+ } else if (inat_is_operand_size_prefix(attr)) {
+ /* oprand size switches 2/4 */
+ insn->opnd_bytes ^= 6;
+ }
+found:
+ prefixes->nbytes++;
+ insn->next_byte++;
+ lb = b;
+ b = peek_next(insn_byte_t, insn);
+ attr = inat_get_opcode_attribute(b);
+ }
+ /* Set the last prefix */
+ if (lb && lb != insn->prefixes.bytes[3]) {
+ if (unlikely(insn->prefixes.bytes[3])) {
+ /* Swap the last prefix */
+ b = insn->prefixes.bytes[3];
+ for (i = 0; i < nb; i++)
+ if (prefixes->bytes[i] == lb)
+ prefixes->bytes[i] = b;
+ }
+ insn->prefixes.bytes[3] = lb;
+ }
+
+ /* Decode REX prefix */
+ if (insn->x86_64) {
+ b = peek_next(insn_byte_t, insn);
+ attr = inat_get_opcode_attribute(b);
+ if (inat_is_rex_prefix(attr)) {
+ insn->rex_prefix.value = b;
+ insn->rex_prefix.nbytes = 1;
+ insn->next_byte++;
+ if (X86_REX_W(b))
+ /* REX.W overrides opnd_size */
+ insn->opnd_bytes = 8;
+ }
+ }
+ insn->rex_prefix.got = 1;
+
+ /* Decode VEX prefix */
+ b = peek_next(insn_byte_t, insn);
+ attr = inat_get_opcode_attribute(b);
+ if (inat_is_vex_prefix(attr)) {
+ insn_byte_t b2 = peek_nbyte_next(insn_byte_t, insn, 1);
+ if (!insn->x86_64) {
+ /*
+ * In 32-bits mode, if the [7:6] bits (mod bits of
+ * ModRM) on the second byte are not 11b, it is
+ * LDS or LES or BOUND.
+ */
+ if (X86_MODRM_MOD(b2) != 3)
+ goto vex_end;
+ }
+ insn->vex_prefix.bytes[0] = b;
+ insn->vex_prefix.bytes[1] = b2;
+ if (inat_is_evex_prefix(attr)) {
+ b2 = peek_nbyte_next(insn_byte_t, insn, 2);
+ insn->vex_prefix.bytes[2] = b2;
+ b2 = peek_nbyte_next(insn_byte_t, insn, 3);
+ insn->vex_prefix.bytes[3] = b2;
+ insn->vex_prefix.nbytes = 4;
+ insn->next_byte += 4;
+ if (insn->x86_64 && X86_VEX_W(b2))
+ /* VEX.W overrides opnd_size */
+ insn->opnd_bytes = 8;
+ } else if (inat_is_vex3_prefix(attr)) {
+ b2 = peek_nbyte_next(insn_byte_t, insn, 2);
+ insn->vex_prefix.bytes[2] = b2;
+ insn->vex_prefix.nbytes = 3;
+ insn->next_byte += 3;
+ if (insn->x86_64 && X86_VEX_W(b2))
+ /* VEX.W overrides opnd_size */
+ insn->opnd_bytes = 8;
+ } else {
+ /*
+ * For VEX2, fake VEX3-like byte#2.
+ * Makes it easier to decode vex.W, vex.vvvv,
+ * vex.L and vex.pp. Masking with 0x7f sets vex.W == 0.
+ */
+ insn->vex_prefix.bytes[2] = b2 & 0x7f;
+ insn->vex_prefix.nbytes = 2;
+ insn->next_byte += 2;
+ }
+ }
+vex_end:
+ insn->vex_prefix.got = 1;
+
+ prefixes->got = 1;
+
+err_out:
+ return;
+}
+
+/**
+ * insn_get_opcode - collect opcode(s)
+ * @insn: &struct insn containing instruction
+ *
+ * Populates @insn->opcode, updates @insn->next_byte to point past the
+ * opcode byte(s), and set @insn->attr (except for groups).
+ * If necessary, first collects any preceding (prefix) bytes.
+ * Sets @insn->opcode.value = opcode1. No effect if @insn->opcode.got
+ * is already 1.
+ */
+void insn_get_opcode(struct insn *insn)
+{
+ struct insn_field *opcode = &insn->opcode;
+ insn_byte_t op;
+ int pfx_id;
+ if (opcode->got)
+ return;
+ if (!insn->prefixes.got)
+ insn_get_prefixes(insn);
+
+ /* Get first opcode */
+ op = get_next(insn_byte_t, insn);
+ opcode->bytes[0] = op;
+ opcode->nbytes = 1;
+
+ /* Check if there is VEX prefix or not */
+ if (insn_is_avx(insn)) {
+ insn_byte_t m, p;
+ m = insn_vex_m_bits(insn);
+ p = insn_vex_p_bits(insn);
+ insn->attr = inat_get_avx_attribute(op, m, p);
+ if ((inat_must_evex(insn->attr) && !insn_is_evex(insn)) ||
+ (!inat_accept_vex(insn->attr) &&
+ !inat_is_group(insn->attr)))
+ insn->attr = 0; /* This instruction is bad */
+ goto end; /* VEX has only 1 byte for opcode */
+ }
+
+ insn->attr = inat_get_opcode_attribute(op);
+ while (inat_is_escape(insn->attr)) {
+ /* Get escaped opcode */
+ op = get_next(insn_byte_t, insn);
+ opcode->bytes[opcode->nbytes++] = op;
+ pfx_id = insn_last_prefix_id(insn);
+ insn->attr = inat_get_escape_attribute(op, pfx_id, insn->attr);
+ }
+ if (inat_must_vex(insn->attr))
+ insn->attr = 0; /* This instruction is bad */
+end:
+ opcode->got = 1;
+
+err_out:
+ return;
+}
+
+/**
+ * insn_get_modrm - collect ModRM byte, if any
+ * @insn: &struct insn containing instruction
+ *
+ * Populates @insn->modrm and updates @insn->next_byte to point past the
+ * ModRM byte, if any. If necessary, first collects the preceding bytes
+ * (prefixes and opcode(s)). No effect if @insn->modrm.got is already 1.
+ */
+void insn_get_modrm(struct insn *insn)
+{
+ struct insn_field *modrm = &insn->modrm;
+ insn_byte_t pfx_id, mod;
+ if (modrm->got)
+ return;
+ if (!insn->opcode.got)
+ insn_get_opcode(insn);
+
+ if (inat_has_modrm(insn->attr)) {
+ mod = get_next(insn_byte_t, insn);
+ modrm->value = mod;
+ modrm->nbytes = 1;
+ if (inat_is_group(insn->attr)) {
+ pfx_id = insn_last_prefix_id(insn);
+ insn->attr = inat_get_group_attribute(mod, pfx_id,
+ insn->attr);
+ if (insn_is_avx(insn) && !inat_accept_vex(insn->attr))
+ insn->attr = 0; /* This is bad */
+ }
+ }
+
+ if (insn->x86_64 && inat_is_force64(insn->attr))
+ insn->opnd_bytes = 8;
+ modrm->got = 1;
+
+err_out:
+ return;
+}
+
+
+/**
+ * insn_rip_relative() - Does instruction use RIP-relative addressing mode?
+ * @insn: &struct insn containing instruction
+ *
+ * If necessary, first collects the instruction up to and including the
+ * ModRM byte. No effect if @insn->x86_64 is 0.
+ */
+int insn_rip_relative(struct insn *insn)
+{
+ struct insn_field *modrm = &insn->modrm;
+
+ if (!insn->x86_64)
+ return 0;
+ if (!modrm->got)
+ insn_get_modrm(insn);
+ /*
+ * For rip-relative instructions, the mod field (top 2 bits)
+ * is zero and the r/m field (bottom 3 bits) is 0x5.
+ */
+ return (modrm->nbytes && (modrm->value & 0xc7) == 0x5);
+}
+
+/**
+ * insn_get_sib() - Get the SIB byte of instruction
+ * @insn: &struct insn containing instruction
+ *
+ * If necessary, first collects the instruction up to and including the
+ * ModRM byte.
+ */
+void insn_get_sib(struct insn *insn)
+{
+ insn_byte_t modrm;
+
+ if (insn->sib.got)
+ return;
+ if (!insn->modrm.got)
+ insn_get_modrm(insn);
+ if (insn->modrm.nbytes) {
+ modrm = (insn_byte_t)insn->modrm.value;
+ if (insn->addr_bytes != 2 &&
+ X86_MODRM_MOD(modrm) != 3 && X86_MODRM_RM(modrm) == 4) {
+ insn->sib.value = get_next(insn_byte_t, insn);
+ insn->sib.nbytes = 1;
+ }
+ }
+ insn->sib.got = 1;
+
+err_out:
+ return;
+}
+
+
+/**
+ * insn_get_displacement() - Get the displacement of instruction
+ * @insn: &struct insn containing instruction
+ *
+ * If necessary, first collects the instruction up to and including the
+ * SIB byte.
+ * Displacement value is sign-expanded.
+ */
+void insn_get_displacement(struct insn *insn)
+{
+ insn_byte_t mod, rm, base;
+
+ if (insn->displacement.got)
+ return;
+ if (!insn->sib.got)
+ insn_get_sib(insn);
+ if (insn->modrm.nbytes) {
+ /*
+ * Interpreting the modrm byte:
+ * mod = 00 - no displacement fields (exceptions below)
+ * mod = 01 - 1-byte displacement field
+ * mod = 10 - displacement field is 4 bytes, or 2 bytes if
+ * address size = 2 (0x67 prefix in 32-bit mode)
+ * mod = 11 - no memory operand
+ *
+ * If address size = 2...
+ * mod = 00, r/m = 110 - displacement field is 2 bytes
+ *
+ * If address size != 2...
+ * mod != 11, r/m = 100 - SIB byte exists
+ * mod = 00, SIB base = 101 - displacement field is 4 bytes
+ * mod = 00, r/m = 101 - rip-relative addressing, displacement
+ * field is 4 bytes
+ */
+ mod = X86_MODRM_MOD(insn->modrm.value);
+ rm = X86_MODRM_RM(insn->modrm.value);
+ base = X86_SIB_BASE(insn->sib.value);
+ if (mod == 3)
+ goto out;
+ if (mod == 1) {
+ insn->displacement.value = get_next(signed char, insn);
+ insn->displacement.nbytes = 1;
+ } else if (insn->addr_bytes == 2) {
+ if ((mod == 0 && rm == 6) || mod == 2) {
+ insn->displacement.value =
+ get_next(short, insn);
+ insn->displacement.nbytes = 2;
+ }
+ } else {
+ if ((mod == 0 && rm == 5) || mod == 2 ||
+ (mod == 0 && base == 5)) {
+ insn->displacement.value = get_next(int, insn);
+ insn->displacement.nbytes = 4;
+ }
+ }
+ }
+out:
+ insn->displacement.got = 1;
+
+err_out:
+ return;
+}
+
+/* Decode moffset16/32/64. Return 0 if failed */
+static int __get_moffset(struct insn *insn)
+{
+ switch (insn->addr_bytes) {
+ case 2:
+ insn->moffset1.value = get_next(short, insn);
+ insn->moffset1.nbytes = 2;
+ break;
+ case 4:
+ insn->moffset1.value = get_next(int, insn);
+ insn->moffset1.nbytes = 4;
+ break;
+ case 8:
+ insn->moffset1.value = get_next(int, insn);
+ insn->moffset1.nbytes = 4;
+ insn->moffset2.value = get_next(int, insn);
+ insn->moffset2.nbytes = 4;
+ break;
+ default: /* opnd_bytes must be modified manually */
+ goto err_out;
+ }
+ insn->moffset1.got = insn->moffset2.got = 1;
+
+ return 1;
+
+err_out:
+ return 0;
+}
+
+/* Decode imm v32(Iz). Return 0 if failed */
+static int __get_immv32(struct insn *insn)
+{
+ switch (insn->opnd_bytes) {
+ case 2:
+ insn->immediate.value = get_next(short, insn);
+ insn->immediate.nbytes = 2;
+ break;
+ case 4:
+ case 8:
+ insn->immediate.value = get_next(int, insn);
+ insn->immediate.nbytes = 4;
+ break;
+ default: /* opnd_bytes must be modified manually */
+ goto err_out;
+ }
+
+ return 1;
+
+err_out:
+ return 0;
+}
+
+/* Decode imm v64(Iv/Ov), Return 0 if failed */
+static int __get_immv(struct insn *insn)
+{
+ switch (insn->opnd_bytes) {
+ case 2:
+ insn->immediate1.value = get_next(short, insn);
+ insn->immediate1.nbytes = 2;
+ break;
+ case 4:
+ insn->immediate1.value = get_next(int, insn);
+ insn->immediate1.nbytes = 4;
+ break;
+ case 8:
+ insn->immediate1.value = get_next(int, insn);
+ insn->immediate1.nbytes = 4;
+ insn->immediate2.value = get_next(int, insn);
+ insn->immediate2.nbytes = 4;
+ break;
+ default: /* opnd_bytes must be modified manually */
+ goto err_out;
+ }
+ insn->immediate1.got = insn->immediate2.got = 1;
+
+ return 1;
+err_out:
+ return 0;
+}
+
+/* Decode ptr16:16/32(Ap) */
+static int __get_immptr(struct insn *insn)
+{
+ switch (insn->opnd_bytes) {
+ case 2:
+ insn->immediate1.value = get_next(short, insn);
+ insn->immediate1.nbytes = 2;
+ break;
+ case 4:
+ insn->immediate1.value = get_next(int, insn);
+ insn->immediate1.nbytes = 4;
+ break;
+ case 8:
+ /* ptr16:64 is not exist (no segment) */
+ return 0;
+ default: /* opnd_bytes must be modified manually */
+ goto err_out;
+ }
+ insn->immediate2.value = get_next(unsigned short, insn);
+ insn->immediate2.nbytes = 2;
+ insn->immediate1.got = insn->immediate2.got = 1;
+
+ return 1;
+err_out:
+ return 0;
+}
+
+/**
+ * insn_get_immediate() - Get the immediates of instruction
+ * @insn: &struct insn containing instruction
+ *
+ * If necessary, first collects the instruction up to and including the
+ * displacement bytes.
+ * Basically, most of immediates are sign-expanded. Unsigned-value can be
+ * get by bit masking with ((1 << (nbytes * 8)) - 1)
+ */
+void insn_get_immediate(struct insn *insn)
+{
+ if (insn->immediate.got)
+ return;
+ if (!insn->displacement.got)
+ insn_get_displacement(insn);
+
+ if (inat_has_moffset(insn->attr)) {
+ if (!__get_moffset(insn))
+ goto err_out;
+ goto done;
+ }
+
+ if (!inat_has_immediate(insn->attr))
+ /* no immediates */
+ goto done;
+
+ switch (inat_immediate_size(insn->attr)) {
+ case INAT_IMM_BYTE:
+ insn->immediate.value = get_next(signed char, insn);
+ insn->immediate.nbytes = 1;
+ break;
+ case INAT_IMM_WORD:
+ insn->immediate.value = get_next(short, insn);
+ insn->immediate.nbytes = 2;
+ break;
+ case INAT_IMM_DWORD:
+ insn->immediate.value = get_next(int, insn);
+ insn->immediate.nbytes = 4;
+ break;
+ case INAT_IMM_QWORD:
+ insn->immediate1.value = get_next(int, insn);
+ insn->immediate1.nbytes = 4;
+ insn->immediate2.value = get_next(int, insn);
+ insn->immediate2.nbytes = 4;
+ break;
+ case INAT_IMM_PTR:
+ if (!__get_immptr(insn))
+ goto err_out;
+ break;
+ case INAT_IMM_VWORD32:
+ if (!__get_immv32(insn))
+ goto err_out;
+ break;
+ case INAT_IMM_VWORD:
+ if (!__get_immv(insn))
+ goto err_out;
+ break;
+ default:
+ /* Here, insn must have an immediate, but failed */
+ goto err_out;
+ }
+ if (inat_has_second_immediate(insn->attr)) {
+ insn->immediate2.value = get_next(signed char, insn);
+ insn->immediate2.nbytes = 1;
+ }
+done:
+ insn->immediate.got = 1;
+
+err_out:
+ return;
+}
+
+/**
+ * insn_get_length() - Get the length of instruction
+ * @insn: &struct insn containing instruction
+ *
+ * If necessary, first collects the instruction up to and including the
+ * immediates bytes.
+ */
+void insn_get_length(struct insn *insn)
+{
+ if (insn->length)
+ return;
+ if (!insn->immediate.got)
+ insn_get_immediate(insn);
+ insn->length = (unsigned char)((unsigned long)insn->next_byte
+ - (unsigned long)insn->kaddr);
+}
diff --git a/tools/objtool/arch/x86/insn/x86-opcode-map.txt b/tools/objtool/arch/x86/lib/x86-opcode-map.txt
similarity index 100%
rename from tools/objtool/arch/x86/insn/x86-opcode-map.txt
rename to tools/objtool/arch/x86/lib/x86-opcode-map.txt
diff --git a/tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk b/tools/objtool/arch/x86/tools/gen-insn-attr-x86.awk
similarity index 100%
rename from tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk
rename to tools/objtool/arch/x86/tools/gen-insn-attr-x86.awk
diff --git a/tools/objtool/builtin-check.c b/tools/objtool/builtin-check.c
index a688a85..694abc6 100644
--- a/tools/objtool/builtin-check.c
+++ b/tools/objtool/builtin-check.c
@@ -1,5 +1,5 @@
/*
- * Copyright (C) 2015 Josh Poimboeuf <jpoimboe@redhat.com>
+ * Copyright (C) 2015-2017 Josh Poimboeuf <jpoimboe@redhat.com>
*
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
@@ -25,1300 +25,35 @@
* For more information, see tools/objtool/Documentation/stack-validation.txt.
*/
-#include <string.h>
-#include <stdlib.h>
#include <subcmd/parse-options.h>
-
#include "builtin.h"
-#include "elf.h"
-#include "special.h"
-#include "arch.h"
-#include "warn.h"
+#include "check.h"
-#include <linux/hashtable.h>
+bool no_fp, no_unreachable, retpoline, module;
-#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
-
-#define STATE_FP_SAVED 0x1
-#define STATE_FP_SETUP 0x2
-#define STATE_FENTRY 0x4
-
-struct instruction {
- struct list_head list;
- struct hlist_node hash;
- struct section *sec;
- unsigned long offset;
- unsigned int len, state;
- unsigned char type;
- unsigned long immediate;
- bool alt_group, visited, ignore_alts;
- struct symbol *call_dest;
- struct instruction *jump_dest;
- struct list_head alts;
- struct symbol *func;
-};
-
-struct alternative {
- struct list_head list;
- struct instruction *insn;
-};
-
-struct objtool_file {
- struct elf *elf;
- struct list_head insn_list;
- DECLARE_HASHTABLE(insn_hash, 16);
- struct section *rodata, *whitelist;
- bool ignore_unreachables, c_file;
-};
-
-const char *objname;
-static bool nofp;
-
-static struct instruction *find_insn(struct objtool_file *file,
- struct section *sec, unsigned long offset)
-{
- struct instruction *insn;
-
- hash_for_each_possible(file->insn_hash, insn, hash, offset)
- if (insn->sec == sec && insn->offset == offset)
- return insn;
-
- return NULL;
-}
-
-static struct instruction *next_insn_same_sec(struct objtool_file *file,
- struct instruction *insn)
-{
- struct instruction *next = list_next_entry(insn, list);
-
- if (&next->list == &file->insn_list || next->sec != insn->sec)
- return NULL;
-
- return next;
-}
-
-static bool gcov_enabled(struct objtool_file *file)
-{
- struct section *sec;
- struct symbol *sym;
-
- list_for_each_entry(sec, &file->elf->sections, list)
- list_for_each_entry(sym, &sec->symbol_list, list)
- if (!strncmp(sym->name, "__gcov_.", 8))
- return true;
-
- return false;
-}
-
-#define for_each_insn(file, insn) \
- list_for_each_entry(insn, &file->insn_list, list)
-
-#define func_for_each_insn(file, func, insn) \
- for (insn = find_insn(file, func->sec, func->offset); \
- insn && &insn->list != &file->insn_list && \
- insn->sec == func->sec && \
- insn->offset < func->offset + func->len; \
- insn = list_next_entry(insn, list))
-
-#define func_for_each_insn_continue_reverse(file, func, insn) \
- for (insn = list_prev_entry(insn, list); \
- &insn->list != &file->insn_list && \
- insn->sec == func->sec && insn->offset >= func->offset; \
- insn = list_prev_entry(insn, list))
-
-#define sec_for_each_insn_from(file, insn) \
- for (; insn; insn = next_insn_same_sec(file, insn))
-
-
-/*
- * Check if the function has been manually whitelisted with the
- * STACK_FRAME_NON_STANDARD macro, or if it should be automatically whitelisted
- * due to its use of a context switching instruction.
- */
-static bool ignore_func(struct objtool_file *file, struct symbol *func)
-{
- struct rela *rela;
- struct instruction *insn;
-
- /* check for STACK_FRAME_NON_STANDARD */
- if (file->whitelist && file->whitelist->rela)
- list_for_each_entry(rela, &file->whitelist->rela->rela_list, list) {
- if (rela->sym->type == STT_SECTION &&
- rela->sym->sec == func->sec &&
- rela->addend == func->offset)
- return true;
- if (rela->sym->type == STT_FUNC && rela->sym == func)
- return true;
- }
-
- /* check if it has a context switching instruction */
- func_for_each_insn(file, func, insn)
- if (insn->type == INSN_CONTEXT_SWITCH)
- return true;
-
- return false;
-}
-
-/*
- * This checks to see if the given function is a "noreturn" function.
- *
- * For global functions which are outside the scope of this object file, we
- * have to keep a manual list of them.
- *
- * For local functions, we have to detect them manually by simply looking for
- * the lack of a return instruction.
- *
- * Returns:
- * -1: error
- * 0: no dead end
- * 1: dead end
- */
-static int __dead_end_function(struct objtool_file *file, struct symbol *func,
- int recursion)
-{
- int i;
- struct instruction *insn;
- bool empty = true;
-
- /*
- * Unfortunately these have to be hard coded because the noreturn
- * attribute isn't provided in ELF data.
- */
- static const char * const global_noreturns[] = {
- "__stack_chk_fail",
- "panic",
- "do_exit",
- "do_task_dead",
- "__module_put_and_exit",
- "complete_and_exit",
- "kvm_spurious_fault",
- "__reiserfs_panic",
- "lbug_with_loc"
- };
-
- if (func->bind == STB_WEAK)
- return 0;
-
- if (func->bind == STB_GLOBAL)
- for (i = 0; i < ARRAY_SIZE(global_noreturns); i++)
- if (!strcmp(func->name, global_noreturns[i]))
- return 1;
-
- if (!func->sec)
- return 0;
-
- func_for_each_insn(file, func, insn) {
- empty = false;
-
- if (insn->type == INSN_RETURN)
- return 0;
- }
-
- if (empty)
- return 0;
-
- /*
- * A function can have a sibling call instead of a return. In that
- * case, the function's dead-end status depends on whether the target
- * of the sibling call returns.
- */
- func_for_each_insn(file, func, insn) {
- if (insn->sec != func->sec ||
- insn->offset >= func->offset + func->len)
- break;
-
- if (insn->type == INSN_JUMP_UNCONDITIONAL) {
- struct instruction *dest = insn->jump_dest;
- struct symbol *dest_func;
-
- if (!dest)
- /* sibling call to another file */
- return 0;
-
- if (dest->sec != func->sec ||
- dest->offset < func->offset ||
- dest->offset >= func->offset + func->len) {
- /* local sibling call */
- dest_func = find_symbol_by_offset(dest->sec,
- dest->offset);
- if (!dest_func)
- continue;
-
- if (recursion == 5) {
- WARN_FUNC("infinite recursion (objtool bug!)",
- dest->sec, dest->offset);
- return -1;
- }
-
- return __dead_end_function(file, dest_func,
- recursion + 1);
- }
- }
-
- if (insn->type == INSN_JUMP_DYNAMIC && list_empty(&insn->alts))
- /* sibling call */
- return 0;
- }
-
- return 1;
-}
-
-static int dead_end_function(struct objtool_file *file, struct symbol *func)
-{
- return __dead_end_function(file, func, 0);
-}
-
-/*
- * Call the arch-specific instruction decoder for all the instructions and add
- * them to the global instruction list.
- */
-static int decode_instructions(struct objtool_file *file)
-{
- struct section *sec;
- struct symbol *func;
- unsigned long offset;
- struct instruction *insn;
- int ret;
-
- list_for_each_entry(sec, &file->elf->sections, list) {
-
- if (!(sec->sh.sh_flags & SHF_EXECINSTR))
- continue;
-
- for (offset = 0; offset < sec->len; offset += insn->len) {
- insn = malloc(sizeof(*insn));
- memset(insn, 0, sizeof(*insn));
-
- INIT_LIST_HEAD(&insn->alts);
- insn->sec = sec;
- insn->offset = offset;
-
- ret = arch_decode_instruction(file->elf, sec, offset,
- sec->len - offset,
- &insn->len, &insn->type,
- &insn->immediate);
- if (ret)
- return ret;
-
- if (!insn->type || insn->type > INSN_LAST) {
- WARN_FUNC("invalid instruction type %d",
- insn->sec, insn->offset, insn->type);
- return -1;
- }
-
- hash_add(file->insn_hash, &insn->hash, insn->offset);
- list_add_tail(&insn->list, &file->insn_list);
- }
-
- list_for_each_entry(func, &sec->symbol_list, list) {
- if (func->type != STT_FUNC)
- continue;
-
- if (!find_insn(file, sec, func->offset)) {
- WARN("%s(): can't find starting instruction",
- func->name);
- return -1;
- }
-
- func_for_each_insn(file, func, insn)
- if (!insn->func)
- insn->func = func;
- }
- }
-
- return 0;
-}
-
-/*
- * Warnings shouldn't be reported for ignored functions.
- */
-static void add_ignores(struct objtool_file *file)
-{
- struct instruction *insn;
- struct section *sec;
- struct symbol *func;
-
- list_for_each_entry(sec, &file->elf->sections, list) {
- list_for_each_entry(func, &sec->symbol_list, list) {
- if (func->type != STT_FUNC)
- continue;
-
- if (!ignore_func(file, func))
- continue;
-
- func_for_each_insn(file, func, insn)
- insn->visited = true;
- }
- }
-}
-
-/*
- * FIXME: For now, just ignore any alternatives which add retpolines. This is
- * a temporary hack, as it doesn't allow ORC to unwind from inside a retpoline.
- * But it at least allows objtool to understand the control flow *around* the
- * retpoline.
- */
-static int add_nospec_ignores(struct objtool_file *file)
-{
- struct section *sec;
- struct rela *rela;
- struct instruction *insn;
-
- sec = find_section_by_name(file->elf, ".rela.discard.nospec");
- if (!sec)
- return 0;
-
- list_for_each_entry(rela, &sec->rela_list, list) {
- if (rela->sym->type != STT_SECTION) {
- WARN("unexpected relocation symbol type in %s", sec->name);
- return -1;
- }
-
- insn = find_insn(file, rela->sym->sec, rela->addend);
- if (!insn) {
- WARN("bad .discard.nospec entry");
- return -1;
- }
-
- insn->ignore_alts = true;
- }
-
- return 0;
-}
-
-/*
- * Find the destination instructions for all jumps.
- */
-static int add_jump_destinations(struct objtool_file *file)
-{
- struct instruction *insn;
- struct rela *rela;
- struct section *dest_sec;
- unsigned long dest_off;
-
- for_each_insn(file, insn) {
- if (insn->type != INSN_JUMP_CONDITIONAL &&
- insn->type != INSN_JUMP_UNCONDITIONAL)
- continue;
-
- /* skip ignores */
- if (insn->visited)
- continue;
-
- rela = find_rela_by_dest_range(insn->sec, insn->offset,
- insn->len);
- if (!rela) {
- dest_sec = insn->sec;
- dest_off = insn->offset + insn->len + insn->immediate;
- } else if (rela->sym->type == STT_SECTION) {
- dest_sec = rela->sym->sec;
- dest_off = rela->addend + 4;
- } else if (rela->sym->sec->idx) {
- dest_sec = rela->sym->sec;
- dest_off = rela->sym->sym.st_value + rela->addend + 4;
- } else if (strstr(rela->sym->name, "_indirect_thunk_")) {
- /*
- * Retpoline jumps are really dynamic jumps in
- * disguise, so convert them accordingly.
- */
- insn->type = INSN_JUMP_DYNAMIC;
- continue;
- } else {
- /* sibling call */
- insn->jump_dest = 0;
- continue;
- }
-
- insn->jump_dest = find_insn(file, dest_sec, dest_off);
- if (!insn->jump_dest) {
-
- /*
- * This is a special case where an alt instruction
- * jumps past the end of the section. These are
- * handled later in handle_group_alt().
- */
- if (!strcmp(insn->sec->name, ".altinstr_replacement"))
- continue;
-
- WARN_FUNC("can't find jump dest instruction at %s+0x%lx",
- insn->sec, insn->offset, dest_sec->name,
- dest_off);
- return -1;
- }
- }
-
- return 0;
-}
-
-/*
- * Find the destination instructions for all calls.
- */
-static int add_call_destinations(struct objtool_file *file)
-{
- struct instruction *insn;
- unsigned long dest_off;
- struct rela *rela;
-
- for_each_insn(file, insn) {
- if (insn->type != INSN_CALL)
- continue;
-
- rela = find_rela_by_dest_range(insn->sec, insn->offset,
- insn->len);
- if (!rela) {
- dest_off = insn->offset + insn->len + insn->immediate;
- insn->call_dest = find_symbol_by_offset(insn->sec,
- dest_off);
- /*
- * FIXME: Thanks to retpolines, it's now considered
- * normal for a function to call within itself. So
- * disable this warning for now.
- */
-#if 0
- if (!insn->call_dest) {
- WARN_FUNC("can't find call dest symbol at offset 0x%lx",
- insn->sec, insn->offset, dest_off);
- return -1;
- }
-#endif
- } else if (rela->sym->type == STT_SECTION) {
- insn->call_dest = find_symbol_by_offset(rela->sym->sec,
- rela->addend+4);
- if (!insn->call_dest ||
- insn->call_dest->type != STT_FUNC) {
- WARN_FUNC("can't find call dest symbol at %s+0x%x",
- insn->sec, insn->offset,
- rela->sym->sec->name,
- rela->addend + 4);
- return -1;
- }
- } else
- insn->call_dest = rela->sym;
- }
-
- return 0;
-}
-
-/*
- * The .alternatives section requires some extra special care, over and above
- * what other special sections require:
- *
- * 1. Because alternatives are patched in-place, we need to insert a fake jump
- * instruction at the end so that validate_branch() skips all the original
- * replaced instructions when validating the new instruction path.
- *
- * 2. An added wrinkle is that the new instruction length might be zero. In
- * that case the old instructions are replaced with noops. We simulate that
- * by creating a fake jump as the only new instruction.
- *
- * 3. In some cases, the alternative section includes an instruction which
- * conditionally jumps to the _end_ of the entry. We have to modify these
- * jumps' destinations to point back to .text rather than the end of the
- * entry in .altinstr_replacement.
- *
- * 4. It has been requested that we don't validate the !POPCNT feature path
- * which is a "very very small percentage of machines".
- */
-static int handle_group_alt(struct objtool_file *file,
- struct special_alt *special_alt,
- struct instruction *orig_insn,
- struct instruction **new_insn)
-{
- struct instruction *last_orig_insn, *last_new_insn, *insn, *fake_jump;
- unsigned long dest_off;
-
- last_orig_insn = NULL;
- insn = orig_insn;
- sec_for_each_insn_from(file, insn) {
- if (insn->offset >= special_alt->orig_off + special_alt->orig_len)
- break;
-
- if (special_alt->skip_orig)
- insn->type = INSN_NOP;
-
- insn->alt_group = true;
- last_orig_insn = insn;
- }
-
- if (!next_insn_same_sec(file, last_orig_insn)) {
- WARN("%s: don't know how to handle alternatives at end of section",
- special_alt->orig_sec->name);
- return -1;
- }
-
- fake_jump = malloc(sizeof(*fake_jump));
- if (!fake_jump) {
- WARN("malloc failed");
- return -1;
- }
- memset(fake_jump, 0, sizeof(*fake_jump));
- INIT_LIST_HEAD(&fake_jump->alts);
- fake_jump->sec = special_alt->new_sec;
- fake_jump->offset = -1;
- fake_jump->type = INSN_JUMP_UNCONDITIONAL;
- fake_jump->jump_dest = list_next_entry(last_orig_insn, list);
-
- if (!special_alt->new_len) {
- *new_insn = fake_jump;
- return 0;
- }
-
- last_new_insn = NULL;
- insn = *new_insn;
- sec_for_each_insn_from(file, insn) {
- if (insn->offset >= special_alt->new_off + special_alt->new_len)
- break;
-
- last_new_insn = insn;
-
- if (insn->type != INSN_JUMP_CONDITIONAL &&
- insn->type != INSN_JUMP_UNCONDITIONAL)
- continue;
-
- if (!insn->immediate)
- continue;
-
- dest_off = insn->offset + insn->len + insn->immediate;
- if (dest_off == special_alt->new_off + special_alt->new_len)
- insn->jump_dest = fake_jump;
-
- if (!insn->jump_dest) {
- WARN_FUNC("can't find alternative jump destination",
- insn->sec, insn->offset);
- return -1;
- }
- }
-
- if (!last_new_insn) {
- WARN_FUNC("can't find last new alternative instruction",
- special_alt->new_sec, special_alt->new_off);
- return -1;
- }
-
- list_add(&fake_jump->list, &last_new_insn->list);
-
- return 0;
-}
-
-/*
- * A jump table entry can either convert a nop to a jump or a jump to a nop.
- * If the original instruction is a jump, make the alt entry an effective nop
- * by just skipping the original instruction.
- */
-static int handle_jump_alt(struct objtool_file *file,
- struct special_alt *special_alt,
- struct instruction *orig_insn,
- struct instruction **new_insn)
-{
- if (orig_insn->type == INSN_NOP)
- return 0;
-
- if (orig_insn->type != INSN_JUMP_UNCONDITIONAL) {
- WARN_FUNC("unsupported instruction at jump label",
- orig_insn->sec, orig_insn->offset);
- return -1;
- }
-
- *new_insn = list_next_entry(orig_insn, list);
- return 0;
-}
-
-/*
- * Read all the special sections which have alternate instructions which can be
- * patched in or redirected to at runtime. Each instruction having alternate
- * instruction(s) has them added to its insn->alts list, which will be
- * traversed in validate_branch().
- */
-static int add_special_section_alts(struct objtool_file *file)
-{
- struct list_head special_alts;
- struct instruction *orig_insn, *new_insn;
- struct special_alt *special_alt, *tmp;
- struct alternative *alt;
- int ret;
-
- ret = special_get_alts(file->elf, &special_alts);
- if (ret)
- return ret;
-
- list_for_each_entry_safe(special_alt, tmp, &special_alts, list) {
-
- orig_insn = find_insn(file, special_alt->orig_sec,
- special_alt->orig_off);
- if (!orig_insn) {
- WARN_FUNC("special: can't find orig instruction",
- special_alt->orig_sec, special_alt->orig_off);
- ret = -1;
- goto out;
- }
-
- /* Ignore retpoline alternatives. */
- if (orig_insn->ignore_alts)
- continue;
-
- new_insn = NULL;
- if (!special_alt->group || special_alt->new_len) {
- new_insn = find_insn(file, special_alt->new_sec,
- special_alt->new_off);
- if (!new_insn) {
- WARN_FUNC("special: can't find new instruction",
- special_alt->new_sec,
- special_alt->new_off);
- ret = -1;
- goto out;
- }
- }
-
- if (special_alt->group) {
- ret = handle_group_alt(file, special_alt, orig_insn,
- &new_insn);
- if (ret)
- goto out;
- } else if (special_alt->jump_or_nop) {
- ret = handle_jump_alt(file, special_alt, orig_insn,
- &new_insn);
- if (ret)
- goto out;
- }
-
- alt = malloc(sizeof(*alt));
- if (!alt) {
- WARN("malloc failed");
- ret = -1;
- goto out;
- }
-
- alt->insn = new_insn;
- list_add_tail(&alt->list, &orig_insn->alts);
-
- list_del(&special_alt->list);
- free(special_alt);
- }
-
-out:
- return ret;
-}
-
-static int add_switch_table(struct objtool_file *file, struct symbol *func,
- struct instruction *insn, struct rela *table,
- struct rela *next_table)
-{
- struct rela *rela = table;
- struct instruction *alt_insn;
- struct alternative *alt;
-
- list_for_each_entry_from(rela, &file->rodata->rela->rela_list, list) {
- if (rela == next_table)
- break;
-
- if (rela->sym->sec != insn->sec ||
- rela->addend <= func->offset ||
- rela->addend >= func->offset + func->len)
- break;
-
- alt_insn = find_insn(file, insn->sec, rela->addend);
- if (!alt_insn) {
- WARN("%s: can't find instruction at %s+0x%x",
- file->rodata->rela->name, insn->sec->name,
- rela->addend);
- return -1;
- }
-
- alt = malloc(sizeof(*alt));
- if (!alt) {
- WARN("malloc failed");
- return -1;
- }
-
- alt->insn = alt_insn;
- list_add_tail(&alt->list, &insn->alts);
- }
-
- return 0;
-}
-
-/*
- * find_switch_table() - Given a dynamic jump, find the switch jump table in
- * .rodata associated with it.
- *
- * There are 3 basic patterns:
- *
- * 1. jmpq *[rodata addr](,%reg,8)
- *
- * This is the most common case by far. It jumps to an address in a simple
- * jump table which is stored in .rodata.
- *
- * 2. jmpq *[rodata addr](%rip)
- *
- * This is caused by a rare GCC quirk, currently only seen in three driver
- * functions in the kernel, only with certain obscure non-distro configs.
- *
- * As part of an optimization, GCC makes a copy of an existing switch jump
- * table, modifies it, and then hard-codes the jump (albeit with an indirect
- * jump) to use a single entry in the table. The rest of the jump table and
- * some of its jump targets remain as dead code.
- *
- * In such a case we can just crudely ignore all unreachable instruction
- * warnings for the entire object file. Ideally we would just ignore them
- * for the function, but that would require redesigning the code quite a
- * bit. And honestly that's just not worth doing: unreachable instruction
- * warnings are of questionable value anyway, and this is such a rare issue.
- *
- * 3. mov [rodata addr],%reg1
- * ... some instructions ...
- * jmpq *(%reg1,%reg2,8)
- *
- * This is a fairly uncommon pattern which is new for GCC 6. As of this
- * writing, there are 11 occurrences of it in the allmodconfig kernel.
- *
- * TODO: Once we have DWARF CFI and smarter instruction decoding logic,
- * ensure the same register is used in the mov and jump instructions.
- */
-static struct rela *find_switch_table(struct objtool_file *file,
- struct symbol *func,
- struct instruction *insn)
-{
- struct rela *text_rela, *rodata_rela;
- struct instruction *orig_insn = insn;
-
- text_rela = find_rela_by_dest_range(insn->sec, insn->offset, insn->len);
- if (text_rela && text_rela->sym == file->rodata->sym) {
- /* case 1 */
- rodata_rela = find_rela_by_dest(file->rodata,
- text_rela->addend);
- if (rodata_rela)
- return rodata_rela;
-
- /* case 2 */
- rodata_rela = find_rela_by_dest(file->rodata,
- text_rela->addend + 4);
- if (!rodata_rela)
- return NULL;
- file->ignore_unreachables = true;
- return rodata_rela;
- }
-
- /* case 3 */
- func_for_each_insn_continue_reverse(file, func, insn) {
- if (insn->type == INSN_JUMP_DYNAMIC)
- break;
-
- /* allow small jumps within the range */
- if (insn->type == INSN_JUMP_UNCONDITIONAL &&
- insn->jump_dest &&
- (insn->jump_dest->offset <= insn->offset ||
- insn->jump_dest->offset > orig_insn->offset))
- break;
-
- /* look for a relocation which references .rodata */
- text_rela = find_rela_by_dest_range(insn->sec, insn->offset,
- insn->len);
- if (!text_rela || text_rela->sym != file->rodata->sym)
- continue;
-
- /*
- * Make sure the .rodata address isn't associated with a
- * symbol. gcc jump tables are anonymous data.
- */
- if (find_symbol_containing(file->rodata, text_rela->addend))
- continue;
-
- return find_rela_by_dest(file->rodata, text_rela->addend);
- }
-
- return NULL;
-}
-
-static int add_func_switch_tables(struct objtool_file *file,
- struct symbol *func)
-{
- struct instruction *insn, *prev_jump = NULL;
- struct rela *rela, *prev_rela = NULL;
- int ret;
-
- func_for_each_insn(file, func, insn) {
- if (insn->type != INSN_JUMP_DYNAMIC)
- continue;
-
- rela = find_switch_table(file, func, insn);
- if (!rela)
- continue;
-
- /*
- * We found a switch table, but we don't know yet how big it
- * is. Don't add it until we reach the end of the function or
- * the beginning of another switch table in the same function.
- */
- if (prev_jump) {
- ret = add_switch_table(file, func, prev_jump, prev_rela,
- rela);
- if (ret)
- return ret;
- }
-
- prev_jump = insn;
- prev_rela = rela;
- }
-
- if (prev_jump) {
- ret = add_switch_table(file, func, prev_jump, prev_rela, NULL);
- if (ret)
- return ret;
- }
-
- return 0;
-}
-
-/*
- * For some switch statements, gcc generates a jump table in the .rodata
- * section which contains a list of addresses within the function to jump to.
- * This finds these jump tables and adds them to the insn->alts lists.
- */
-static int add_switch_table_alts(struct objtool_file *file)
-{
- struct section *sec;
- struct symbol *func;
- int ret;
-
- if (!file->rodata || !file->rodata->rela)
- return 0;
-
- list_for_each_entry(sec, &file->elf->sections, list) {
- list_for_each_entry(func, &sec->symbol_list, list) {
- if (func->type != STT_FUNC)
- continue;
-
- ret = add_func_switch_tables(file, func);
- if (ret)
- return ret;
- }
- }
-
- return 0;
-}
-
-static int decode_sections(struct objtool_file *file)
-{
- int ret;
-
- ret = decode_instructions(file);
- if (ret)
- return ret;
-
- add_ignores(file);
-
- ret = add_nospec_ignores(file);
- if (ret)
- return ret;
-
- ret = add_jump_destinations(file);
- if (ret)
- return ret;
-
- ret = add_call_destinations(file);
- if (ret)
- return ret;
-
- ret = add_special_section_alts(file);
- if (ret)
- return ret;
-
- ret = add_switch_table_alts(file);
- if (ret)
- return ret;
-
- return 0;
-}
-
-static bool is_fentry_call(struct instruction *insn)
-{
- if (insn->type == INSN_CALL &&
- insn->call_dest->type == STT_NOTYPE &&
- !strcmp(insn->call_dest->name, "__fentry__"))
- return true;
-
- return false;
-}
-
-static bool has_modified_stack_frame(struct instruction *insn)
-{
- return (insn->state & STATE_FP_SAVED) ||
- (insn->state & STATE_FP_SETUP);
-}
-
-static bool has_valid_stack_frame(struct instruction *insn)
-{
- return (insn->state & STATE_FP_SAVED) &&
- (insn->state & STATE_FP_SETUP);
-}
-
-static unsigned int frame_state(unsigned long state)
-{
- return (state & (STATE_FP_SAVED | STATE_FP_SETUP));
-}
-
-/*
- * Follow the branch starting at the given instruction, and recursively follow
- * any other branches (jumps). Meanwhile, track the frame pointer state at
- * each instruction and validate all the rules described in
- * tools/objtool/Documentation/stack-validation.txt.
- */
-static int validate_branch(struct objtool_file *file,
- struct instruction *first, unsigned char first_state)
-{
- struct alternative *alt;
- struct instruction *insn;
- struct section *sec;
- struct symbol *func = NULL;
- unsigned char state;
- int ret;
-
- insn = first;
- sec = insn->sec;
- state = first_state;
-
- if (insn->alt_group && list_empty(&insn->alts)) {
- WARN_FUNC("don't know how to handle branch to middle of alternative instruction group",
- sec, insn->offset);
- return 1;
- }
-
- while (1) {
- if (file->c_file && insn->func) {
- if (func && func != insn->func) {
- WARN("%s() falls through to next function %s()",
- func->name, insn->func->name);
- return 1;
- }
-
- func = insn->func;
- }
-
- if (insn->visited) {
- if (frame_state(insn->state) != frame_state(state)) {
- WARN_FUNC("frame pointer state mismatch",
- sec, insn->offset);
- return 1;
- }
-
- return 0;
- }
-
- insn->visited = true;
- insn->state = state;
-
- list_for_each_entry(alt, &insn->alts, list) {
- ret = validate_branch(file, alt->insn, state);
- if (ret)
- return 1;
- }
-
- switch (insn->type) {
-
- case INSN_FP_SAVE:
- if (!nofp) {
- if (state & STATE_FP_SAVED) {
- WARN_FUNC("duplicate frame pointer save",
- sec, insn->offset);
- return 1;
- }
- state |= STATE_FP_SAVED;
- }
- break;
-
- case INSN_FP_SETUP:
- if (!nofp) {
- if (state & STATE_FP_SETUP) {
- WARN_FUNC("duplicate frame pointer setup",
- sec, insn->offset);
- return 1;
- }
- state |= STATE_FP_SETUP;
- }
- break;
-
- case INSN_FP_RESTORE:
- if (!nofp) {
- if (has_valid_stack_frame(insn))
- state &= ~STATE_FP_SETUP;
-
- state &= ~STATE_FP_SAVED;
- }
- break;
-
- case INSN_RETURN:
- if (!nofp && has_modified_stack_frame(insn)) {
- WARN_FUNC("return without frame pointer restore",
- sec, insn->offset);
- return 1;
- }
- return 0;
-
- case INSN_CALL:
- if (is_fentry_call(insn)) {
- state |= STATE_FENTRY;
- break;
- }
-
- ret = dead_end_function(file, insn->call_dest);
- if (ret == 1)
- return 0;
- if (ret == -1)
- return 1;
-
- /* fallthrough */
- case INSN_CALL_DYNAMIC:
- if (!nofp && !has_valid_stack_frame(insn)) {
- WARN_FUNC("call without frame pointer save/setup",
- sec, insn->offset);
- return 1;
- }
- break;
-
- case INSN_JUMP_CONDITIONAL:
- case INSN_JUMP_UNCONDITIONAL:
- if (insn->jump_dest) {
- ret = validate_branch(file, insn->jump_dest,
- state);
- if (ret)
- return 1;
- } else if (has_modified_stack_frame(insn)) {
- WARN_FUNC("sibling call from callable instruction with changed frame pointer",
- sec, insn->offset);
- return 1;
- } /* else it's a sibling call */
-
- if (insn->type == INSN_JUMP_UNCONDITIONAL)
- return 0;
-
- break;
-
- case INSN_JUMP_DYNAMIC:
- if (list_empty(&insn->alts) &&
- has_modified_stack_frame(insn)) {
- WARN_FUNC("sibling call from callable instruction with changed frame pointer",
- sec, insn->offset);
- return 1;
- }
-
- return 0;
-
- case INSN_BUG:
- return 0;
-
- default:
- break;
- }
-
- insn = next_insn_same_sec(file, insn);
- if (!insn) {
- WARN("%s: unexpected end of section", sec->name);
- return 1;
- }
- }
-
- return 0;
-}
-
-static bool is_kasan_insn(struct instruction *insn)
-{
- return (insn->type == INSN_CALL &&
- !strcmp(insn->call_dest->name, "__asan_handle_no_return"));
-}
-
-static bool is_ubsan_insn(struct instruction *insn)
-{
- return (insn->type == INSN_CALL &&
- !strcmp(insn->call_dest->name,
- "__ubsan_handle_builtin_unreachable"));
-}
-
-static bool ignore_unreachable_insn(struct symbol *func,
- struct instruction *insn)
-{
- int i;
-
- if (insn->type == INSN_NOP)
- return true;
-
- /*
- * Check if this (or a subsequent) instruction is related to
- * CONFIG_UBSAN or CONFIG_KASAN.
- *
- * End the search at 5 instructions to avoid going into the weeds.
- */
- for (i = 0; i < 5; i++) {
-
- if (is_kasan_insn(insn) || is_ubsan_insn(insn))
- return true;
-
- if (insn->type == INSN_JUMP_UNCONDITIONAL && insn->jump_dest) {
- insn = insn->jump_dest;
- continue;
- }
-
- if (insn->offset + insn->len >= func->offset + func->len)
- break;
- insn = list_next_entry(insn, list);
- }
-
- return false;
-}
-
-static int validate_functions(struct objtool_file *file)
-{
- struct section *sec;
- struct symbol *func;
- struct instruction *insn;
- int ret, warnings = 0;
-
- list_for_each_entry(sec, &file->elf->sections, list) {
- list_for_each_entry(func, &sec->symbol_list, list) {
- if (func->type != STT_FUNC)
- continue;
-
- insn = find_insn(file, sec, func->offset);
- if (!insn)
- continue;
-
- ret = validate_branch(file, insn, 0);
- warnings += ret;
- }
- }
-
- list_for_each_entry(sec, &file->elf->sections, list) {
- list_for_each_entry(func, &sec->symbol_list, list) {
- if (func->type != STT_FUNC)
- continue;
-
- func_for_each_insn(file, func, insn) {
- if (insn->visited)
- continue;
-
- insn->visited = true;
-
- if (file->ignore_unreachables || warnings ||
- ignore_unreachable_insn(func, insn))
- continue;
-
- /*
- * gcov produces a lot of unreachable
- * instructions. If we get an unreachable
- * warning and the file has gcov enabled, just
- * ignore it, and all other such warnings for
- * the file.
- */
- if (!file->ignore_unreachables &&
- gcov_enabled(file)) {
- file->ignore_unreachables = true;
- continue;
- }
-
- WARN_FUNC("function has unreachable instruction", insn->sec, insn->offset);
- warnings++;
- }
- }
- }
-
- return warnings;
-}
-
-static int validate_uncallable_instructions(struct objtool_file *file)
-{
- struct instruction *insn;
- int warnings = 0;
-
- for_each_insn(file, insn) {
- if (!insn->visited && insn->type == INSN_RETURN) {
-
- /*
- * Don't warn about call instructions in unvisited
- * retpoline alternatives.
- */
- if (!strcmp(insn->sec->name, ".altinstr_replacement"))
- continue;
-
- WARN_FUNC("return instruction outside of a callable function",
- insn->sec, insn->offset);
- warnings++;
- }
- }
-
- return warnings;
-}
-
-static void cleanup(struct objtool_file *file)
-{
- struct instruction *insn, *tmpinsn;
- struct alternative *alt, *tmpalt;
-
- list_for_each_entry_safe(insn, tmpinsn, &file->insn_list, list) {
- list_for_each_entry_safe(alt, tmpalt, &insn->alts, list) {
- list_del(&alt->list);
- free(alt);
- }
- list_del(&insn->list);
- hash_del(&insn->hash);
- free(insn);
- }
- elf_close(file->elf);
-}
-
-const char * const check_usage[] = {
+static const char * const check_usage[] = {
"objtool check [<options>] file.o",
NULL,
};
+const struct option check_options[] = {
+ OPT_BOOLEAN('f', "no-fp", &no_fp, "Skip frame pointer validation"),
+ OPT_BOOLEAN('u', "no-unreachable", &no_unreachable, "Skip 'unreachable instruction' warnings"),
+ OPT_BOOLEAN('r', "retpoline", &retpoline, "Validate retpoline assumptions"),
+ OPT_BOOLEAN('m', "module", &module, "Indicates the object will be part of a kernel module"),
+ OPT_END(),
+};
+
int cmd_check(int argc, const char **argv)
{
- struct objtool_file file;
- int ret, warnings = 0;
+ const char *objname;
- const struct option options[] = {
- OPT_BOOLEAN('f', "no-fp", &nofp, "Skip frame pointer validation"),
- OPT_END(),
- };
-
- argc = parse_options(argc, argv, options, check_usage, 0);
+ argc = parse_options(argc, argv, check_options, check_usage, 0);
if (argc != 1)
- usage_with_options(check_usage, options);
+ usage_with_options(check_usage, check_options);
objname = argv[0];
- file.elf = elf_open(objname);
- if (!file.elf) {
- fprintf(stderr, "error reading elf file %s\n", objname);
- return 1;
- }
-
- INIT_LIST_HEAD(&file.insn_list);
- hash_init(file.insn_hash);
- file.whitelist = find_section_by_name(file.elf, ".discard.func_stack_frame_non_standard");
- file.rodata = find_section_by_name(file.elf, ".rodata");
- file.ignore_unreachables = false;
- file.c_file = find_section_by_name(file.elf, ".comment");
-
- ret = decode_sections(&file);
- if (ret < 0)
- goto out;
- warnings += ret;
-
- ret = validate_functions(&file);
- if (ret < 0)
- goto out;
- warnings += ret;
-
- ret = validate_uncallable_instructions(&file);
- if (ret < 0)
- goto out;
- warnings += ret;
-
-out:
- cleanup(&file);
-
- /* ignore warnings for now until we get all the code cleaned up */
- if (ret || warnings)
- return 0;
- return 0;
+ return check(objname, false);
}
diff --git a/tools/objtool/builtin-orc.c b/tools/objtool/builtin-orc.c
new file mode 100644
index 0000000..77ea2b9
--- /dev/null
+++ b/tools/objtool/builtin-orc.c
@@ -0,0 +1,68 @@
+/*
+ * Copyright (C) 2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+/*
+ * objtool orc:
+ *
+ * This command analyzes a .o file and adds .orc_unwind and .orc_unwind_ip
+ * sections to it, which is used by the in-kernel ORC unwinder.
+ *
+ * This command is a superset of "objtool check".
+ */
+
+#include <string.h>
+#include "builtin.h"
+#include "check.h"
+
+
+static const char *orc_usage[] = {
+ "objtool orc generate [<options>] file.o",
+ "objtool orc dump file.o",
+ NULL,
+};
+
+int cmd_orc(int argc, const char **argv)
+{
+ const char *objname;
+
+ argc--; argv++;
+ if (argc <= 0)
+ usage_with_options(orc_usage, check_options);
+
+ if (!strncmp(argv[0], "gen", 3)) {
+ argc = parse_options(argc, argv, check_options, orc_usage, 0);
+ if (argc != 1)
+ usage_with_options(orc_usage, check_options);
+
+ objname = argv[0];
+
+ return check(objname, true);
+ }
+
+ if (!strcmp(argv[0], "dump")) {
+ if (argc != 2)
+ usage_with_options(orc_usage, check_options);
+
+ objname = argv[1];
+
+ return orc_dump(objname);
+ }
+
+ usage_with_options(orc_usage, check_options);
+
+ return 0;
+}
diff --git a/tools/objtool/builtin.h b/tools/objtool/builtin.h
index 34d2ba7..28ff40e 100644
--- a/tools/objtool/builtin.h
+++ b/tools/objtool/builtin.h
@@ -17,6 +17,12 @@
#ifndef _BUILTIN_H
#define _BUILTIN_H
+#include <subcmd/parse-options.h>
+
+extern const struct option check_options[];
+extern bool no_fp, no_unreachable, retpoline, module;
+
extern int cmd_check(int argc, const char **argv);
+extern int cmd_orc(int argc, const char **argv);
#endif /* _BUILTIN_H */
diff --git a/tools/objtool/cfi.h b/tools/objtool/cfi.h
new file mode 100644
index 0000000..2fe883c
--- /dev/null
+++ b/tools/objtool/cfi.h
@@ -0,0 +1,55 @@
+/*
+ * Copyright (C) 2015-2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef _OBJTOOL_CFI_H
+#define _OBJTOOL_CFI_H
+
+#define CFI_UNDEFINED -1
+#define CFI_CFA -2
+#define CFI_SP_INDIRECT -3
+#define CFI_BP_INDIRECT -4
+
+#define CFI_AX 0
+#define CFI_DX 1
+#define CFI_CX 2
+#define CFI_BX 3
+#define CFI_SI 4
+#define CFI_DI 5
+#define CFI_BP 6
+#define CFI_SP 7
+#define CFI_R8 8
+#define CFI_R9 9
+#define CFI_R10 10
+#define CFI_R11 11
+#define CFI_R12 12
+#define CFI_R13 13
+#define CFI_R14 14
+#define CFI_R15 15
+#define CFI_RA 16
+#define CFI_NUM_REGS 17
+
+struct cfi_reg {
+ int base;
+ int offset;
+};
+
+struct cfi_state {
+ struct cfi_reg cfa;
+ struct cfi_reg regs[CFI_NUM_REGS];
+};
+
+#endif /* _OBJTOOL_CFI_H */
diff --git a/tools/objtool/check.c b/tools/objtool/check.c
new file mode 100644
index 0000000..e128d1c
--- /dev/null
+++ b/tools/objtool/check.c
@@ -0,0 +1,2209 @@
+/*
+ * Copyright (C) 2015-2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <string.h>
+#include <stdlib.h>
+
+#include "builtin.h"
+#include "check.h"
+#include "elf.h"
+#include "special.h"
+#include "arch.h"
+#include "warn.h"
+
+#include <linux/hashtable.h>
+#include <linux/kernel.h>
+
+struct alternative {
+ struct list_head list;
+ struct instruction *insn;
+};
+
+const char *objname;
+struct cfi_state initial_func_cfi;
+
+struct instruction *find_insn(struct objtool_file *file,
+ struct section *sec, unsigned long offset)
+{
+ struct instruction *insn;
+
+ hash_for_each_possible(file->insn_hash, insn, hash, offset)
+ if (insn->sec == sec && insn->offset == offset)
+ return insn;
+
+ return NULL;
+}
+
+static struct instruction *next_insn_same_sec(struct objtool_file *file,
+ struct instruction *insn)
+{
+ struct instruction *next = list_next_entry(insn, list);
+
+ if (!next || &next->list == &file->insn_list || next->sec != insn->sec)
+ return NULL;
+
+ return next;
+}
+
+static struct instruction *next_insn_same_func(struct objtool_file *file,
+ struct instruction *insn)
+{
+ struct instruction *next = list_next_entry(insn, list);
+ struct symbol *func = insn->func;
+
+ if (!func)
+ return NULL;
+
+ if (&next->list != &file->insn_list && next->func == func)
+ return next;
+
+ /* Check if we're already in the subfunction: */
+ if (func == func->cfunc)
+ return NULL;
+
+ /* Move to the subfunction: */
+ return find_insn(file, func->cfunc->sec, func->cfunc->offset);
+}
+
+#define func_for_each_insn_all(file, func, insn) \
+ for (insn = find_insn(file, func->sec, func->offset); \
+ insn; \
+ insn = next_insn_same_func(file, insn))
+
+#define func_for_each_insn(file, func, insn) \
+ for (insn = find_insn(file, func->sec, func->offset); \
+ insn && &insn->list != &file->insn_list && \
+ insn->sec == func->sec && \
+ insn->offset < func->offset + func->len; \
+ insn = list_next_entry(insn, list))
+
+#define func_for_each_insn_continue_reverse(file, func, insn) \
+ for (insn = list_prev_entry(insn, list); \
+ &insn->list != &file->insn_list && \
+ insn->sec == func->sec && insn->offset >= func->offset; \
+ insn = list_prev_entry(insn, list))
+
+#define sec_for_each_insn_from(file, insn) \
+ for (; insn; insn = next_insn_same_sec(file, insn))
+
+#define sec_for_each_insn_continue(file, insn) \
+ for (insn = next_insn_same_sec(file, insn); insn; \
+ insn = next_insn_same_sec(file, insn))
+
+/*
+ * Check if the function has been manually whitelisted with the
+ * STACK_FRAME_NON_STANDARD macro, or if it should be automatically whitelisted
+ * due to its use of a context switching instruction.
+ */
+static bool ignore_func(struct objtool_file *file, struct symbol *func)
+{
+ struct rela *rela;
+
+ /* check for STACK_FRAME_NON_STANDARD */
+ if (file->whitelist && file->whitelist->rela)
+ list_for_each_entry(rela, &file->whitelist->rela->rela_list, list) {
+ if (rela->sym->type == STT_SECTION &&
+ rela->sym->sec == func->sec &&
+ rela->addend == func->offset)
+ return true;
+ if (rela->sym->type == STT_FUNC && rela->sym == func)
+ return true;
+ }
+
+ return false;
+}
+
+/*
+ * This checks to see if the given function is a "noreturn" function.
+ *
+ * For global functions which are outside the scope of this object file, we
+ * have to keep a manual list of them.
+ *
+ * For local functions, we have to detect them manually by simply looking for
+ * the lack of a return instruction.
+ *
+ * Returns:
+ * -1: error
+ * 0: no dead end
+ * 1: dead end
+ */
+static int __dead_end_function(struct objtool_file *file, struct symbol *func,
+ int recursion)
+{
+ int i;
+ struct instruction *insn;
+ bool empty = true;
+
+ /*
+ * Unfortunately these have to be hard coded because the noreturn
+ * attribute isn't provided in ELF data.
+ */
+ static const char * const global_noreturns[] = {
+ "__stack_chk_fail",
+ "panic",
+ "do_exit",
+ "do_task_dead",
+ "__module_put_and_exit",
+ "complete_and_exit",
+ "kvm_spurious_fault",
+ "__reiserfs_panic",
+ "lbug_with_loc",
+ "fortify_panic",
+ };
+
+ if (func->bind == STB_WEAK)
+ return 0;
+
+ if (func->bind == STB_GLOBAL)
+ for (i = 0; i < ARRAY_SIZE(global_noreturns); i++)
+ if (!strcmp(func->name, global_noreturns[i]))
+ return 1;
+
+ if (!func->len)
+ return 0;
+
+ insn = find_insn(file, func->sec, func->offset);
+ if (!insn->func)
+ return 0;
+
+ func_for_each_insn_all(file, func, insn) {
+ empty = false;
+
+ if (insn->type == INSN_RETURN)
+ return 0;
+ }
+
+ if (empty)
+ return 0;
+
+ /*
+ * A function can have a sibling call instead of a return. In that
+ * case, the function's dead-end status depends on whether the target
+ * of the sibling call returns.
+ */
+ func_for_each_insn_all(file, func, insn) {
+ if (insn->type == INSN_JUMP_UNCONDITIONAL) {
+ struct instruction *dest = insn->jump_dest;
+
+ if (!dest)
+ /* sibling call to another file */
+ return 0;
+
+ if (dest->func && dest->func->pfunc != insn->func->pfunc) {
+
+ /* local sibling call */
+ if (recursion == 5) {
+ /*
+ * Infinite recursion: two functions
+ * have sibling calls to each other.
+ * This is a very rare case. It means
+ * they aren't dead ends.
+ */
+ return 0;
+ }
+
+ return __dead_end_function(file, dest->func,
+ recursion + 1);
+ }
+ }
+
+ if (insn->type == INSN_JUMP_DYNAMIC && list_empty(&insn->alts))
+ /* sibling call */
+ return 0;
+ }
+
+ return 1;
+}
+
+static int dead_end_function(struct objtool_file *file, struct symbol *func)
+{
+ return __dead_end_function(file, func, 0);
+}
+
+static void clear_insn_state(struct insn_state *state)
+{
+ int i;
+
+ memset(state, 0, sizeof(*state));
+ state->cfa.base = CFI_UNDEFINED;
+ for (i = 0; i < CFI_NUM_REGS; i++) {
+ state->regs[i].base = CFI_UNDEFINED;
+ state->vals[i].base = CFI_UNDEFINED;
+ }
+ state->drap_reg = CFI_UNDEFINED;
+ state->drap_offset = -1;
+}
+
+/*
+ * Call the arch-specific instruction decoder for all the instructions and add
+ * them to the global instruction list.
+ */
+static int decode_instructions(struct objtool_file *file)
+{
+ struct section *sec;
+ struct symbol *func;
+ unsigned long offset;
+ struct instruction *insn;
+ int ret;
+
+ for_each_sec(file, sec) {
+
+ if (!(sec->sh.sh_flags & SHF_EXECINSTR))
+ continue;
+
+ if (strcmp(sec->name, ".altinstr_replacement") &&
+ strcmp(sec->name, ".altinstr_aux") &&
+ strncmp(sec->name, ".discard.", 9))
+ sec->text = true;
+
+ for (offset = 0; offset < sec->len; offset += insn->len) {
+ insn = malloc(sizeof(*insn));
+ if (!insn) {
+ WARN("malloc failed");
+ return -1;
+ }
+ memset(insn, 0, sizeof(*insn));
+ INIT_LIST_HEAD(&insn->alts);
+ clear_insn_state(&insn->state);
+
+ insn->sec = sec;
+ insn->offset = offset;
+
+ ret = arch_decode_instruction(file->elf, sec, offset,
+ sec->len - offset,
+ &insn->len, &insn->type,
+ &insn->immediate,
+ &insn->stack_op);
+ if (ret)
+ goto err;
+
+ if (!insn->type || insn->type > INSN_LAST) {
+ WARN_FUNC("invalid instruction type %d",
+ insn->sec, insn->offset, insn->type);
+ ret = -1;
+ goto err;
+ }
+
+ hash_add(file->insn_hash, &insn->hash, insn->offset);
+ list_add_tail(&insn->list, &file->insn_list);
+ }
+
+ list_for_each_entry(func, &sec->symbol_list, list) {
+ if (func->type != STT_FUNC)
+ continue;
+
+ if (!find_insn(file, sec, func->offset)) {
+ WARN("%s(): can't find starting instruction",
+ func->name);
+ return -1;
+ }
+
+ func_for_each_insn(file, func, insn)
+ if (!insn->func)
+ insn->func = func;
+ }
+ }
+
+ return 0;
+
+err:
+ free(insn);
+ return ret;
+}
+
+/*
+ * Mark "ud2" instructions and manually annotated dead ends.
+ */
+static int add_dead_ends(struct objtool_file *file)
+{
+ struct section *sec;
+ struct rela *rela;
+ struct instruction *insn;
+ bool found;
+
+ /*
+ * By default, "ud2" is a dead end unless otherwise annotated, because
+ * GCC 7 inserts it for certain divide-by-zero cases.
+ */
+ for_each_insn(file, insn)
+ if (insn->type == INSN_BUG)
+ insn->dead_end = true;
+
+ /*
+ * Check for manually annotated dead ends.
+ */
+ sec = find_section_by_name(file->elf, ".rela.discard.unreachable");
+ if (!sec)
+ goto reachable;
+
+ list_for_each_entry(rela, &sec->rela_list, list) {
+ if (rela->sym->type != STT_SECTION) {
+ WARN("unexpected relocation symbol type in %s", sec->name);
+ return -1;
+ }
+ insn = find_insn(file, rela->sym->sec, rela->addend);
+ if (insn)
+ insn = list_prev_entry(insn, list);
+ else if (rela->addend == rela->sym->sec->len) {
+ found = false;
+ list_for_each_entry_reverse(insn, &file->insn_list, list) {
+ if (insn->sec == rela->sym->sec) {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found) {
+ WARN("can't find unreachable insn at %s+0x%x",
+ rela->sym->sec->name, rela->addend);
+ return -1;
+ }
+ } else {
+ WARN("can't find unreachable insn at %s+0x%x",
+ rela->sym->sec->name, rela->addend);
+ return -1;
+ }
+
+ insn->dead_end = true;
+ }
+
+reachable:
+ /*
+ * These manually annotated reachable checks are needed for GCC 4.4,
+ * where the Linux unreachable() macro isn't supported. In that case
+ * GCC doesn't know the "ud2" is fatal, so it generates code as if it's
+ * not a dead end.
+ */
+ sec = find_section_by_name(file->elf, ".rela.discard.reachable");
+ if (!sec)
+ return 0;
+
+ list_for_each_entry(rela, &sec->rela_list, list) {
+ if (rela->sym->type != STT_SECTION) {
+ WARN("unexpected relocation symbol type in %s", sec->name);
+ return -1;
+ }
+ insn = find_insn(file, rela->sym->sec, rela->addend);
+ if (insn)
+ insn = list_prev_entry(insn, list);
+ else if (rela->addend == rela->sym->sec->len) {
+ found = false;
+ list_for_each_entry_reverse(insn, &file->insn_list, list) {
+ if (insn->sec == rela->sym->sec) {
+ found = true;
+ break;
+ }
+ }
+
+ if (!found) {
+ WARN("can't find reachable insn at %s+0x%x",
+ rela->sym->sec->name, rela->addend);
+ return -1;
+ }
+ } else {
+ WARN("can't find reachable insn at %s+0x%x",
+ rela->sym->sec->name, rela->addend);
+ return -1;
+ }
+
+ insn->dead_end = false;
+ }
+
+ return 0;
+}
+
+/*
+ * Warnings shouldn't be reported for ignored functions.
+ */
+static void add_ignores(struct objtool_file *file)
+{
+ struct instruction *insn;
+ struct section *sec;
+ struct symbol *func;
+
+ for_each_sec(file, sec) {
+ list_for_each_entry(func, &sec->symbol_list, list) {
+ if (func->type != STT_FUNC)
+ continue;
+
+ if (!ignore_func(file, func))
+ continue;
+
+ func_for_each_insn_all(file, func, insn)
+ insn->ignore = true;
+ }
+ }
+}
+
+/*
+ * FIXME: For now, just ignore any alternatives which add retpolines. This is
+ * a temporary hack, as it doesn't allow ORC to unwind from inside a retpoline.
+ * But it at least allows objtool to understand the control flow *around* the
+ * retpoline.
+ */
+static int add_nospec_ignores(struct objtool_file *file)
+{
+ struct section *sec;
+ struct rela *rela;
+ struct instruction *insn;
+
+ sec = find_section_by_name(file->elf, ".rela.discard.nospec");
+ if (!sec)
+ return 0;
+
+ list_for_each_entry(rela, &sec->rela_list, list) {
+ if (rela->sym->type != STT_SECTION) {
+ WARN("unexpected relocation symbol type in %s", sec->name);
+ return -1;
+ }
+
+ insn = find_insn(file, rela->sym->sec, rela->addend);
+ if (!insn) {
+ WARN("bad .discard.nospec entry");
+ return -1;
+ }
+
+ insn->ignore_alts = true;
+ }
+
+ return 0;
+}
+
+/*
+ * Find the destination instructions for all jumps.
+ */
+static int add_jump_destinations(struct objtool_file *file)
+{
+ struct instruction *insn;
+ struct rela *rela;
+ struct section *dest_sec;
+ unsigned long dest_off;
+
+ for_each_insn(file, insn) {
+ if (insn->type != INSN_JUMP_CONDITIONAL &&
+ insn->type != INSN_JUMP_UNCONDITIONAL)
+ continue;
+
+ if (insn->ignore)
+ continue;
+
+ rela = find_rela_by_dest_range(insn->sec, insn->offset,
+ insn->len);
+ if (!rela) {
+ dest_sec = insn->sec;
+ dest_off = insn->offset + insn->len + insn->immediate;
+ } else if (rela->sym->type == STT_SECTION) {
+ dest_sec = rela->sym->sec;
+ dest_off = rela->addend + 4;
+ } else if (rela->sym->sec->idx) {
+ dest_sec = rela->sym->sec;
+ dest_off = rela->sym->sym.st_value + rela->addend + 4;
+ } else if (strstr(rela->sym->name, "_indirect_thunk_")) {
+ /*
+ * Retpoline jumps are really dynamic jumps in
+ * disguise, so convert them accordingly.
+ */
+ insn->type = INSN_JUMP_DYNAMIC;
+ insn->retpoline_safe = true;
+ continue;
+ } else {
+ /* sibling call */
+ insn->jump_dest = 0;
+ continue;
+ }
+
+ insn->jump_dest = find_insn(file, dest_sec, dest_off);
+ if (!insn->jump_dest) {
+
+ /*
+ * This is a special case where an alt instruction
+ * jumps past the end of the section. These are
+ * handled later in handle_group_alt().
+ */
+ if (!strcmp(insn->sec->name, ".altinstr_replacement"))
+ continue;
+
+ WARN_FUNC("can't find jump dest instruction at %s+0x%lx",
+ insn->sec, insn->offset, dest_sec->name,
+ dest_off);
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+/*
+ * Find the destination instructions for all calls.
+ */
+static int add_call_destinations(struct objtool_file *file)
+{
+ struct instruction *insn;
+ unsigned long dest_off;
+ struct rela *rela;
+
+ for_each_insn(file, insn) {
+ if (insn->type != INSN_CALL)
+ continue;
+
+ rela = find_rela_by_dest_range(insn->sec, insn->offset,
+ insn->len);
+ if (!rela) {
+ dest_off = insn->offset + insn->len + insn->immediate;
+ insn->call_dest = find_symbol_by_offset(insn->sec,
+ dest_off);
+
+ if (!insn->call_dest && !insn->ignore) {
+ WARN_FUNC("unsupported intra-function call",
+ insn->sec, insn->offset);
+ if (retpoline)
+ WARN("If this is a retpoline, please patch it in with alternatives and annotate it with ANNOTATE_NOSPEC_ALTERNATIVE.");
+ return -1;
+ }
+
+ } else if (rela->sym->type == STT_SECTION) {
+ insn->call_dest = find_symbol_by_offset(rela->sym->sec,
+ rela->addend+4);
+ if (!insn->call_dest ||
+ insn->call_dest->type != STT_FUNC) {
+ WARN_FUNC("can't find call dest symbol at %s+0x%x",
+ insn->sec, insn->offset,
+ rela->sym->sec->name,
+ rela->addend + 4);
+ return -1;
+ }
+ } else
+ insn->call_dest = rela->sym;
+ }
+
+ return 0;
+}
+
+/*
+ * The .alternatives section requires some extra special care, over and above
+ * what other special sections require:
+ *
+ * 1. Because alternatives are patched in-place, we need to insert a fake jump
+ * instruction at the end so that validate_branch() skips all the original
+ * replaced instructions when validating the new instruction path.
+ *
+ * 2. An added wrinkle is that the new instruction length might be zero. In
+ * that case the old instructions are replaced with noops. We simulate that
+ * by creating a fake jump as the only new instruction.
+ *
+ * 3. In some cases, the alternative section includes an instruction which
+ * conditionally jumps to the _end_ of the entry. We have to modify these
+ * jumps' destinations to point back to .text rather than the end of the
+ * entry in .altinstr_replacement.
+ *
+ * 4. It has been requested that we don't validate the !POPCNT feature path
+ * which is a "very very small percentage of machines".
+ */
+static int handle_group_alt(struct objtool_file *file,
+ struct special_alt *special_alt,
+ struct instruction *orig_insn,
+ struct instruction **new_insn)
+{
+ struct instruction *last_orig_insn, *last_new_insn, *insn, *fake_jump = NULL;
+ unsigned long dest_off;
+
+ last_orig_insn = NULL;
+ insn = orig_insn;
+ sec_for_each_insn_from(file, insn) {
+ if (insn->offset >= special_alt->orig_off + special_alt->orig_len)
+ break;
+
+ if (special_alt->skip_orig)
+ insn->type = INSN_NOP;
+
+ insn->alt_group = true;
+ last_orig_insn = insn;
+ }
+
+ if (next_insn_same_sec(file, last_orig_insn)) {
+ fake_jump = malloc(sizeof(*fake_jump));
+ if (!fake_jump) {
+ WARN("malloc failed");
+ return -1;
+ }
+ memset(fake_jump, 0, sizeof(*fake_jump));
+ INIT_LIST_HEAD(&fake_jump->alts);
+ clear_insn_state(&fake_jump->state);
+
+ fake_jump->sec = special_alt->new_sec;
+ fake_jump->offset = -1;
+ fake_jump->type = INSN_JUMP_UNCONDITIONAL;
+ fake_jump->jump_dest = list_next_entry(last_orig_insn, list);
+ fake_jump->ignore = true;
+ }
+
+ if (!special_alt->new_len) {
+ if (!fake_jump) {
+ WARN("%s: empty alternative at end of section",
+ special_alt->orig_sec->name);
+ return -1;
+ }
+
+ *new_insn = fake_jump;
+ return 0;
+ }
+
+ last_new_insn = NULL;
+ insn = *new_insn;
+ sec_for_each_insn_from(file, insn) {
+ if (insn->offset >= special_alt->new_off + special_alt->new_len)
+ break;
+
+ last_new_insn = insn;
+
+ insn->ignore = orig_insn->ignore_alts;
+
+ if (insn->type != INSN_JUMP_CONDITIONAL &&
+ insn->type != INSN_JUMP_UNCONDITIONAL)
+ continue;
+
+ if (!insn->immediate)
+ continue;
+
+ dest_off = insn->offset + insn->len + insn->immediate;
+ if (dest_off == special_alt->new_off + special_alt->new_len) {
+ if (!fake_jump) {
+ WARN("%s: alternative jump to end of section",
+ special_alt->orig_sec->name);
+ return -1;
+ }
+ insn->jump_dest = fake_jump;
+ }
+
+ if (!insn->jump_dest) {
+ WARN_FUNC("can't find alternative jump destination",
+ insn->sec, insn->offset);
+ return -1;
+ }
+ }
+
+ if (!last_new_insn) {
+ WARN_FUNC("can't find last new alternative instruction",
+ special_alt->new_sec, special_alt->new_off);
+ return -1;
+ }
+
+ if (fake_jump)
+ list_add(&fake_jump->list, &last_new_insn->list);
+
+ return 0;
+}
+
+/*
+ * A jump table entry can either convert a nop to a jump or a jump to a nop.
+ * If the original instruction is a jump, make the alt entry an effective nop
+ * by just skipping the original instruction.
+ */
+static int handle_jump_alt(struct objtool_file *file,
+ struct special_alt *special_alt,
+ struct instruction *orig_insn,
+ struct instruction **new_insn)
+{
+ if (orig_insn->type == INSN_NOP)
+ return 0;
+
+ if (orig_insn->type != INSN_JUMP_UNCONDITIONAL) {
+ WARN_FUNC("unsupported instruction at jump label",
+ orig_insn->sec, orig_insn->offset);
+ return -1;
+ }
+
+ *new_insn = list_next_entry(orig_insn, list);
+ return 0;
+}
+
+/*
+ * Read all the special sections which have alternate instructions which can be
+ * patched in or redirected to at runtime. Each instruction having alternate
+ * instruction(s) has them added to its insn->alts list, which will be
+ * traversed in validate_branch().
+ */
+static int add_special_section_alts(struct objtool_file *file)
+{
+ struct list_head special_alts;
+ struct instruction *orig_insn, *new_insn;
+ struct special_alt *special_alt, *tmp;
+ struct alternative *alt;
+ int ret;
+
+ ret = special_get_alts(file->elf, &special_alts);
+ if (ret)
+ return ret;
+
+ list_for_each_entry_safe(special_alt, tmp, &special_alts, list) {
+
+ orig_insn = find_insn(file, special_alt->orig_sec,
+ special_alt->orig_off);
+ if (!orig_insn) {
+ WARN_FUNC("special: can't find orig instruction",
+ special_alt->orig_sec, special_alt->orig_off);
+ ret = -1;
+ goto out;
+ }
+
+ new_insn = NULL;
+ if (!special_alt->group || special_alt->new_len) {
+ new_insn = find_insn(file, special_alt->new_sec,
+ special_alt->new_off);
+ if (!new_insn) {
+ WARN_FUNC("special: can't find new instruction",
+ special_alt->new_sec,
+ special_alt->new_off);
+ ret = -1;
+ goto out;
+ }
+ }
+
+ if (special_alt->group) {
+ ret = handle_group_alt(file, special_alt, orig_insn,
+ &new_insn);
+ if (ret)
+ goto out;
+ } else if (special_alt->jump_or_nop) {
+ ret = handle_jump_alt(file, special_alt, orig_insn,
+ &new_insn);
+ if (ret)
+ goto out;
+ }
+
+ alt = malloc(sizeof(*alt));
+ if (!alt) {
+ WARN("malloc failed");
+ ret = -1;
+ goto out;
+ }
+
+ alt->insn = new_insn;
+ list_add_tail(&alt->list, &orig_insn->alts);
+
+ list_del(&special_alt->list);
+ free(special_alt);
+ }
+
+out:
+ return ret;
+}
+
+static int add_switch_table(struct objtool_file *file, struct instruction *insn,
+ struct rela *table, struct rela *next_table)
+{
+ struct rela *rela = table;
+ struct instruction *alt_insn;
+ struct alternative *alt;
+ struct symbol *pfunc = insn->func->pfunc;
+ unsigned int prev_offset = 0;
+
+ list_for_each_entry_from(rela, &file->rodata->rela->rela_list, list) {
+ if (rela == next_table)
+ break;
+
+ /* Make sure the switch table entries are consecutive: */
+ if (prev_offset && rela->offset != prev_offset + 8)
+ break;
+
+ /* Detect function pointers from contiguous objects: */
+ if (rela->sym->sec == pfunc->sec &&
+ rela->addend == pfunc->offset)
+ break;
+
+ alt_insn = find_insn(file, rela->sym->sec, rela->addend);
+ if (!alt_insn)
+ break;
+
+ /* Make sure the jmp dest is in the function or subfunction: */
+ if (alt_insn->func->pfunc != pfunc)
+ break;
+
+ alt = malloc(sizeof(*alt));
+ if (!alt) {
+ WARN("malloc failed");
+ return -1;
+ }
+
+ alt->insn = alt_insn;
+ list_add_tail(&alt->list, &insn->alts);
+ prev_offset = rela->offset;
+ }
+
+ if (!prev_offset) {
+ WARN_FUNC("can't find switch jump table",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ return 0;
+}
+
+/*
+ * find_switch_table() - Given a dynamic jump, find the switch jump table in
+ * .rodata associated with it.
+ *
+ * There are 3 basic patterns:
+ *
+ * 1. jmpq *[rodata addr](,%reg,8)
+ *
+ * This is the most common case by far. It jumps to an address in a simple
+ * jump table which is stored in .rodata.
+ *
+ * 2. jmpq *[rodata addr](%rip)
+ *
+ * This is caused by a rare GCC quirk, currently only seen in three driver
+ * functions in the kernel, only with certain obscure non-distro configs.
+ *
+ * As part of an optimization, GCC makes a copy of an existing switch jump
+ * table, modifies it, and then hard-codes the jump (albeit with an indirect
+ * jump) to use a single entry in the table. The rest of the jump table and
+ * some of its jump targets remain as dead code.
+ *
+ * In such a case we can just crudely ignore all unreachable instruction
+ * warnings for the entire object file. Ideally we would just ignore them
+ * for the function, but that would require redesigning the code quite a
+ * bit. And honestly that's just not worth doing: unreachable instruction
+ * warnings are of questionable value anyway, and this is such a rare issue.
+ *
+ * 3. mov [rodata addr],%reg1
+ * ... some instructions ...
+ * jmpq *(%reg1,%reg2,8)
+ *
+ * This is a fairly uncommon pattern which is new for GCC 6. As of this
+ * writing, there are 11 occurrences of it in the allmodconfig kernel.
+ *
+ * As of GCC 7 there are quite a few more of these and the 'in between' code
+ * is significant. Esp. with KASAN enabled some of the code between the mov
+ * and jmpq uses .rodata itself, which can confuse things.
+ *
+ * TODO: Once we have DWARF CFI and smarter instruction decoding logic,
+ * ensure the same register is used in the mov and jump instructions.
+ *
+ * NOTE: RETPOLINE made it harder still to decode dynamic jumps.
+ */
+static struct rela *find_switch_table(struct objtool_file *file,
+ struct symbol *func,
+ struct instruction *insn)
+{
+ struct rela *text_rela, *rodata_rela;
+ struct instruction *orig_insn = insn;
+ unsigned long table_offset;
+
+ /*
+ * Backward search using the @first_jump_src links, these help avoid
+ * much of the 'in between' code. Which avoids us getting confused by
+ * it.
+ */
+ for (;
+ &insn->list != &file->insn_list &&
+ insn->sec == func->sec &&
+ insn->offset >= func->offset;
+
+ insn = insn->first_jump_src ?: list_prev_entry(insn, list)) {
+
+ if (insn != orig_insn && insn->type == INSN_JUMP_DYNAMIC)
+ break;
+
+ /* allow small jumps within the range */
+ if (insn->type == INSN_JUMP_UNCONDITIONAL &&
+ insn->jump_dest &&
+ (insn->jump_dest->offset <= insn->offset ||
+ insn->jump_dest->offset > orig_insn->offset))
+ break;
+
+ /* look for a relocation which references .rodata */
+ text_rela = find_rela_by_dest_range(insn->sec, insn->offset,
+ insn->len);
+ if (!text_rela || text_rela->sym != file->rodata->sym)
+ continue;
+
+ table_offset = text_rela->addend;
+ if (text_rela->type == R_X86_64_PC32)
+ table_offset += 4;
+
+ /*
+ * Make sure the .rodata address isn't associated with a
+ * symbol. gcc jump tables are anonymous data.
+ */
+ if (find_symbol_containing(file->rodata, table_offset))
+ continue;
+
+ rodata_rela = find_rela_by_dest(file->rodata, table_offset);
+ if (rodata_rela) {
+ /*
+ * Use of RIP-relative switch jumps is quite rare, and
+ * indicates a rare GCC quirk/bug which can leave dead
+ * code behind.
+ */
+ if (text_rela->type == R_X86_64_PC32)
+ file->ignore_unreachables = true;
+
+ return rodata_rela;
+ }
+ }
+
+ return NULL;
+}
+
+
+static int add_func_switch_tables(struct objtool_file *file,
+ struct symbol *func)
+{
+ struct instruction *insn, *last = NULL, *prev_jump = NULL;
+ struct rela *rela, *prev_rela = NULL;
+ int ret;
+
+ func_for_each_insn_all(file, func, insn) {
+ if (!last)
+ last = insn;
+
+ /*
+ * Store back-pointers for unconditional forward jumps such
+ * that find_switch_table() can back-track using those and
+ * avoid some potentially confusing code.
+ */
+ if (insn->type == INSN_JUMP_UNCONDITIONAL && insn->jump_dest &&
+ insn->offset > last->offset &&
+ insn->jump_dest->offset > insn->offset &&
+ !insn->jump_dest->first_jump_src) {
+
+ insn->jump_dest->first_jump_src = insn;
+ last = insn->jump_dest;
+ }
+
+ if (insn->type != INSN_JUMP_DYNAMIC)
+ continue;
+
+ rela = find_switch_table(file, func, insn);
+ if (!rela)
+ continue;
+
+ /*
+ * We found a switch table, but we don't know yet how big it
+ * is. Don't add it until we reach the end of the function or
+ * the beginning of another switch table in the same function.
+ */
+ if (prev_jump) {
+ ret = add_switch_table(file, prev_jump, prev_rela, rela);
+ if (ret)
+ return ret;
+ }
+
+ prev_jump = insn;
+ prev_rela = rela;
+ }
+
+ if (prev_jump) {
+ ret = add_switch_table(file, prev_jump, prev_rela, NULL);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
+}
+
+/*
+ * For some switch statements, gcc generates a jump table in the .rodata
+ * section which contains a list of addresses within the function to jump to.
+ * This finds these jump tables and adds them to the insn->alts lists.
+ */
+static int add_switch_table_alts(struct objtool_file *file)
+{
+ struct section *sec;
+ struct symbol *func;
+ int ret;
+
+ if (!file->rodata || !file->rodata->rela)
+ return 0;
+
+ for_each_sec(file, sec) {
+ list_for_each_entry(func, &sec->symbol_list, list) {
+ if (func->type != STT_FUNC)
+ continue;
+
+ ret = add_func_switch_tables(file, func);
+ if (ret)
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int read_unwind_hints(struct objtool_file *file)
+{
+ struct section *sec, *relasec;
+ struct rela *rela;
+ struct unwind_hint *hint;
+ struct instruction *insn;
+ struct cfi_reg *cfa;
+ int i;
+
+ sec = find_section_by_name(file->elf, ".discard.unwind_hints");
+ if (!sec)
+ return 0;
+
+ relasec = sec->rela;
+ if (!relasec) {
+ WARN("missing .rela.discard.unwind_hints section");
+ return -1;
+ }
+
+ if (sec->len % sizeof(struct unwind_hint)) {
+ WARN("struct unwind_hint size mismatch");
+ return -1;
+ }
+
+ file->hints = true;
+
+ for (i = 0; i < sec->len / sizeof(struct unwind_hint); i++) {
+ hint = (struct unwind_hint *)sec->data->d_buf + i;
+
+ rela = find_rela_by_dest(sec, i * sizeof(*hint));
+ if (!rela) {
+ WARN("can't find rela for unwind_hints[%d]", i);
+ return -1;
+ }
+
+ insn = find_insn(file, rela->sym->sec, rela->addend);
+ if (!insn) {
+ WARN("can't find insn for unwind_hints[%d]", i);
+ return -1;
+ }
+
+ cfa = &insn->state.cfa;
+
+ if (hint->type == UNWIND_HINT_TYPE_SAVE) {
+ insn->save = true;
+ continue;
+
+ } else if (hint->type == UNWIND_HINT_TYPE_RESTORE) {
+ insn->restore = true;
+ insn->hint = true;
+ continue;
+ }
+
+ insn->hint = true;
+
+ switch (hint->sp_reg) {
+ case ORC_REG_UNDEFINED:
+ cfa->base = CFI_UNDEFINED;
+ break;
+ case ORC_REG_SP:
+ cfa->base = CFI_SP;
+ break;
+ case ORC_REG_BP:
+ cfa->base = CFI_BP;
+ break;
+ case ORC_REG_SP_INDIRECT:
+ cfa->base = CFI_SP_INDIRECT;
+ break;
+ case ORC_REG_R10:
+ cfa->base = CFI_R10;
+ break;
+ case ORC_REG_R13:
+ cfa->base = CFI_R13;
+ break;
+ case ORC_REG_DI:
+ cfa->base = CFI_DI;
+ break;
+ case ORC_REG_DX:
+ cfa->base = CFI_DX;
+ break;
+ default:
+ WARN_FUNC("unsupported unwind_hint sp base reg %d",
+ insn->sec, insn->offset, hint->sp_reg);
+ return -1;
+ }
+
+ cfa->offset = hint->sp_offset;
+ insn->state.type = hint->type;
+ }
+
+ return 0;
+}
+
+static int read_retpoline_hints(struct objtool_file *file)
+{
+ struct section *sec;
+ struct instruction *insn;
+ struct rela *rela;
+
+ sec = find_section_by_name(file->elf, ".rela.discard.retpoline_safe");
+ if (!sec)
+ return 0;
+
+ list_for_each_entry(rela, &sec->rela_list, list) {
+ if (rela->sym->type != STT_SECTION) {
+ WARN("unexpected relocation symbol type in %s", sec->name);
+ return -1;
+ }
+
+ insn = find_insn(file, rela->sym->sec, rela->addend);
+ if (!insn) {
+ WARN("bad .discard.retpoline_safe entry");
+ return -1;
+ }
+
+ if (insn->type != INSN_JUMP_DYNAMIC &&
+ insn->type != INSN_CALL_DYNAMIC) {
+ WARN_FUNC("retpoline_safe hint not an indirect jump/call",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ insn->retpoline_safe = true;
+ }
+
+ return 0;
+}
+
+static int decode_sections(struct objtool_file *file)
+{
+ int ret;
+
+ ret = decode_instructions(file);
+ if (ret)
+ return ret;
+
+ ret = add_dead_ends(file);
+ if (ret)
+ return ret;
+
+ add_ignores(file);
+
+ ret = add_nospec_ignores(file);
+ if (ret)
+ return ret;
+
+ ret = add_jump_destinations(file);
+ if (ret)
+ return ret;
+
+ ret = add_special_section_alts(file);
+ if (ret)
+ return ret;
+
+ ret = add_call_destinations(file);
+ if (ret)
+ return ret;
+
+ ret = add_switch_table_alts(file);
+ if (ret)
+ return ret;
+
+ ret = read_unwind_hints(file);
+ if (ret)
+ return ret;
+
+ ret = read_retpoline_hints(file);
+ if (ret)
+ return ret;
+
+ return 0;
+}
+
+static bool is_fentry_call(struct instruction *insn)
+{
+ if (insn->type == INSN_CALL &&
+ insn->call_dest->type == STT_NOTYPE &&
+ !strcmp(insn->call_dest->name, "__fentry__"))
+ return true;
+
+ return false;
+}
+
+static bool has_modified_stack_frame(struct insn_state *state)
+{
+ int i;
+
+ if (state->cfa.base != initial_func_cfi.cfa.base ||
+ state->cfa.offset != initial_func_cfi.cfa.offset ||
+ state->stack_size != initial_func_cfi.cfa.offset ||
+ state->drap)
+ return true;
+
+ for (i = 0; i < CFI_NUM_REGS; i++)
+ if (state->regs[i].base != initial_func_cfi.regs[i].base ||
+ state->regs[i].offset != initial_func_cfi.regs[i].offset)
+ return true;
+
+ return false;
+}
+
+static bool has_valid_stack_frame(struct insn_state *state)
+{
+ if (state->cfa.base == CFI_BP && state->regs[CFI_BP].base == CFI_CFA &&
+ state->regs[CFI_BP].offset == -16)
+ return true;
+
+ if (state->drap && state->regs[CFI_BP].base == CFI_BP)
+ return true;
+
+ return false;
+}
+
+static int update_insn_state_regs(struct instruction *insn, struct insn_state *state)
+{
+ struct cfi_reg *cfa = &state->cfa;
+ struct stack_op *op = &insn->stack_op;
+
+ if (cfa->base != CFI_SP)
+ return 0;
+
+ /* push */
+ if (op->dest.type == OP_DEST_PUSH)
+ cfa->offset += 8;
+
+ /* pop */
+ if (op->src.type == OP_SRC_POP)
+ cfa->offset -= 8;
+
+ /* add immediate to sp */
+ if (op->dest.type == OP_DEST_REG && op->src.type == OP_SRC_ADD &&
+ op->dest.reg == CFI_SP && op->src.reg == CFI_SP)
+ cfa->offset -= op->src.offset;
+
+ return 0;
+}
+
+static void save_reg(struct insn_state *state, unsigned char reg, int base,
+ int offset)
+{
+ if (arch_callee_saved_reg(reg) &&
+ state->regs[reg].base == CFI_UNDEFINED) {
+ state->regs[reg].base = base;
+ state->regs[reg].offset = offset;
+ }
+}
+
+static void restore_reg(struct insn_state *state, unsigned char reg)
+{
+ state->regs[reg].base = CFI_UNDEFINED;
+ state->regs[reg].offset = 0;
+}
+
+/*
+ * A note about DRAP stack alignment:
+ *
+ * GCC has the concept of a DRAP register, which is used to help keep track of
+ * the stack pointer when aligning the stack. r10 or r13 is used as the DRAP
+ * register. The typical DRAP pattern is:
+ *
+ * 4c 8d 54 24 08 lea 0x8(%rsp),%r10
+ * 48 83 e4 c0 and $0xffffffffffffffc0,%rsp
+ * 41 ff 72 f8 pushq -0x8(%r10)
+ * 55 push %rbp
+ * 48 89 e5 mov %rsp,%rbp
+ * (more pushes)
+ * 41 52 push %r10
+ * ...
+ * 41 5a pop %r10
+ * (more pops)
+ * 5d pop %rbp
+ * 49 8d 62 f8 lea -0x8(%r10),%rsp
+ * c3 retq
+ *
+ * There are some variations in the epilogues, like:
+ *
+ * 5b pop %rbx
+ * 41 5a pop %r10
+ * 41 5c pop %r12
+ * 41 5d pop %r13
+ * 41 5e pop %r14
+ * c9 leaveq
+ * 49 8d 62 f8 lea -0x8(%r10),%rsp
+ * c3 retq
+ *
+ * and:
+ *
+ * 4c 8b 55 e8 mov -0x18(%rbp),%r10
+ * 48 8b 5d e0 mov -0x20(%rbp),%rbx
+ * 4c 8b 65 f0 mov -0x10(%rbp),%r12
+ * 4c 8b 6d f8 mov -0x8(%rbp),%r13
+ * c9 leaveq
+ * 49 8d 62 f8 lea -0x8(%r10),%rsp
+ * c3 retq
+ *
+ * Sometimes r13 is used as the DRAP register, in which case it's saved and
+ * restored beforehand:
+ *
+ * 41 55 push %r13
+ * 4c 8d 6c 24 10 lea 0x10(%rsp),%r13
+ * 48 83 e4 f0 and $0xfffffffffffffff0,%rsp
+ * ...
+ * 49 8d 65 f0 lea -0x10(%r13),%rsp
+ * 41 5d pop %r13
+ * c3 retq
+ */
+static int update_insn_state(struct instruction *insn, struct insn_state *state)
+{
+ struct stack_op *op = &insn->stack_op;
+ struct cfi_reg *cfa = &state->cfa;
+ struct cfi_reg *regs = state->regs;
+
+ /* stack operations don't make sense with an undefined CFA */
+ if (cfa->base == CFI_UNDEFINED) {
+ if (insn->func) {
+ WARN_FUNC("undefined stack state", insn->sec, insn->offset);
+ return -1;
+ }
+ return 0;
+ }
+
+ if (state->type == ORC_TYPE_REGS || state->type == ORC_TYPE_REGS_IRET)
+ return update_insn_state_regs(insn, state);
+
+ switch (op->dest.type) {
+
+ case OP_DEST_REG:
+ switch (op->src.type) {
+
+ case OP_SRC_REG:
+ if (op->src.reg == CFI_SP && op->dest.reg == CFI_BP &&
+ cfa->base == CFI_SP &&
+ regs[CFI_BP].base == CFI_CFA &&
+ regs[CFI_BP].offset == -cfa->offset) {
+
+ /* mov %rsp, %rbp */
+ cfa->base = op->dest.reg;
+ state->bp_scratch = false;
+ }
+
+ else if (op->src.reg == CFI_SP &&
+ op->dest.reg == CFI_BP && state->drap) {
+
+ /* drap: mov %rsp, %rbp */
+ regs[CFI_BP].base = CFI_BP;
+ regs[CFI_BP].offset = -state->stack_size;
+ state->bp_scratch = false;
+ }
+
+ else if (op->src.reg == CFI_SP && cfa->base == CFI_SP) {
+
+ /*
+ * mov %rsp, %reg
+ *
+ * This is needed for the rare case where GCC
+ * does:
+ *
+ * mov %rsp, %rax
+ * ...
+ * mov %rax, %rsp
+ */
+ state->vals[op->dest.reg].base = CFI_CFA;
+ state->vals[op->dest.reg].offset = -state->stack_size;
+ }
+
+ else if (op->src.reg == CFI_BP && op->dest.reg == CFI_SP &&
+ cfa->base == CFI_BP) {
+
+ /*
+ * mov %rbp, %rsp
+ *
+ * Restore the original stack pointer (Clang).
+ */
+ state->stack_size = -state->regs[CFI_BP].offset;
+ }
+
+ else if (op->dest.reg == cfa->base) {
+
+ /* mov %reg, %rsp */
+ if (cfa->base == CFI_SP &&
+ state->vals[op->src.reg].base == CFI_CFA) {
+
+ /*
+ * This is needed for the rare case
+ * where GCC does something dumb like:
+ *
+ * lea 0x8(%rsp), %rcx
+ * ...
+ * mov %rcx, %rsp
+ */
+ cfa->offset = -state->vals[op->src.reg].offset;
+ state->stack_size = cfa->offset;
+
+ } else {
+ cfa->base = CFI_UNDEFINED;
+ cfa->offset = 0;
+ }
+ }
+
+ break;
+
+ case OP_SRC_ADD:
+ if (op->dest.reg == CFI_SP && op->src.reg == CFI_SP) {
+
+ /* add imm, %rsp */
+ state->stack_size -= op->src.offset;
+ if (cfa->base == CFI_SP)
+ cfa->offset -= op->src.offset;
+ break;
+ }
+
+ if (op->dest.reg == CFI_SP && op->src.reg == CFI_BP) {
+
+ /* lea disp(%rbp), %rsp */
+ state->stack_size = -(op->src.offset + regs[CFI_BP].offset);
+ break;
+ }
+
+ if (op->src.reg == CFI_SP && cfa->base == CFI_SP) {
+
+ /* drap: lea disp(%rsp), %drap */
+ state->drap_reg = op->dest.reg;
+
+ /*
+ * lea disp(%rsp), %reg
+ *
+ * This is needed for the rare case where GCC
+ * does something dumb like:
+ *
+ * lea 0x8(%rsp), %rcx
+ * ...
+ * mov %rcx, %rsp
+ */
+ state->vals[op->dest.reg].base = CFI_CFA;
+ state->vals[op->dest.reg].offset = \
+ -state->stack_size + op->src.offset;
+
+ break;
+ }
+
+ if (state->drap && op->dest.reg == CFI_SP &&
+ op->src.reg == state->drap_reg) {
+
+ /* drap: lea disp(%drap), %rsp */
+ cfa->base = CFI_SP;
+ cfa->offset = state->stack_size = -op->src.offset;
+ state->drap_reg = CFI_UNDEFINED;
+ state->drap = false;
+ break;
+ }
+
+ if (op->dest.reg == state->cfa.base) {
+ WARN_FUNC("unsupported stack register modification",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ break;
+
+ case OP_SRC_AND:
+ if (op->dest.reg != CFI_SP ||
+ (state->drap_reg != CFI_UNDEFINED && cfa->base != CFI_SP) ||
+ (state->drap_reg == CFI_UNDEFINED && cfa->base != CFI_BP)) {
+ WARN_FUNC("unsupported stack pointer realignment",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ if (state->drap_reg != CFI_UNDEFINED) {
+ /* drap: and imm, %rsp */
+ cfa->base = state->drap_reg;
+ cfa->offset = state->stack_size = 0;
+ state->drap = true;
+ }
+
+ /*
+ * Older versions of GCC (4.8ish) realign the stack
+ * without DRAP, with a frame pointer.
+ */
+
+ break;
+
+ case OP_SRC_POP:
+ if (!state->drap && op->dest.type == OP_DEST_REG &&
+ op->dest.reg == cfa->base) {
+
+ /* pop %rbp */
+ cfa->base = CFI_SP;
+ }
+
+ if (state->drap && cfa->base == CFI_BP_INDIRECT &&
+ op->dest.type == OP_DEST_REG &&
+ op->dest.reg == state->drap_reg &&
+ state->drap_offset == -state->stack_size) {
+
+ /* drap: pop %drap */
+ cfa->base = state->drap_reg;
+ cfa->offset = 0;
+ state->drap_offset = -1;
+
+ } else if (regs[op->dest.reg].offset == -state->stack_size) {
+
+ /* pop %reg */
+ restore_reg(state, op->dest.reg);
+ }
+
+ state->stack_size -= 8;
+ if (cfa->base == CFI_SP)
+ cfa->offset -= 8;
+
+ break;
+
+ case OP_SRC_REG_INDIRECT:
+ if (state->drap && op->src.reg == CFI_BP &&
+ op->src.offset == state->drap_offset) {
+
+ /* drap: mov disp(%rbp), %drap */
+ cfa->base = state->drap_reg;
+ cfa->offset = 0;
+ state->drap_offset = -1;
+ }
+
+ if (state->drap && op->src.reg == CFI_BP &&
+ op->src.offset == regs[op->dest.reg].offset) {
+
+ /* drap: mov disp(%rbp), %reg */
+ restore_reg(state, op->dest.reg);
+
+ } else if (op->src.reg == cfa->base &&
+ op->src.offset == regs[op->dest.reg].offset + cfa->offset) {
+
+ /* mov disp(%rbp), %reg */
+ /* mov disp(%rsp), %reg */
+ restore_reg(state, op->dest.reg);
+ }
+
+ break;
+
+ default:
+ WARN_FUNC("unknown stack-related instruction",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ break;
+
+ case OP_DEST_PUSH:
+ state->stack_size += 8;
+ if (cfa->base == CFI_SP)
+ cfa->offset += 8;
+
+ if (op->src.type != OP_SRC_REG)
+ break;
+
+ if (state->drap) {
+ if (op->src.reg == cfa->base && op->src.reg == state->drap_reg) {
+
+ /* drap: push %drap */
+ cfa->base = CFI_BP_INDIRECT;
+ cfa->offset = -state->stack_size;
+
+ /* save drap so we know when to restore it */
+ state->drap_offset = -state->stack_size;
+
+ } else if (op->src.reg == CFI_BP && cfa->base == state->drap_reg) {
+
+ /* drap: push %rbp */
+ state->stack_size = 0;
+
+ } else if (regs[op->src.reg].base == CFI_UNDEFINED) {
+
+ /* drap: push %reg */
+ save_reg(state, op->src.reg, CFI_BP, -state->stack_size);
+ }
+
+ } else {
+
+ /* push %reg */
+ save_reg(state, op->src.reg, CFI_CFA, -state->stack_size);
+ }
+
+ /* detect when asm code uses rbp as a scratch register */
+ if (!no_fp && insn->func && op->src.reg == CFI_BP &&
+ cfa->base != CFI_BP)
+ state->bp_scratch = true;
+ break;
+
+ case OP_DEST_REG_INDIRECT:
+
+ if (state->drap) {
+ if (op->src.reg == cfa->base && op->src.reg == state->drap_reg) {
+
+ /* drap: mov %drap, disp(%rbp) */
+ cfa->base = CFI_BP_INDIRECT;
+ cfa->offset = op->dest.offset;
+
+ /* save drap offset so we know when to restore it */
+ state->drap_offset = op->dest.offset;
+ }
+
+ else if (regs[op->src.reg].base == CFI_UNDEFINED) {
+
+ /* drap: mov reg, disp(%rbp) */
+ save_reg(state, op->src.reg, CFI_BP, op->dest.offset);
+ }
+
+ } else if (op->dest.reg == cfa->base) {
+
+ /* mov reg, disp(%rbp) */
+ /* mov reg, disp(%rsp) */
+ save_reg(state, op->src.reg, CFI_CFA,
+ op->dest.offset - state->cfa.offset);
+ }
+
+ break;
+
+ case OP_DEST_LEAVE:
+ if ((!state->drap && cfa->base != CFI_BP) ||
+ (state->drap && cfa->base != state->drap_reg)) {
+ WARN_FUNC("leave instruction with modified stack frame",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ /* leave (mov %rbp, %rsp; pop %rbp) */
+
+ state->stack_size = -state->regs[CFI_BP].offset - 8;
+ restore_reg(state, CFI_BP);
+
+ if (!state->drap) {
+ cfa->base = CFI_SP;
+ cfa->offset -= 8;
+ }
+
+ break;
+
+ case OP_DEST_MEM:
+ if (op->src.type != OP_SRC_POP) {
+ WARN_FUNC("unknown stack-related memory operation",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ /* pop mem */
+ state->stack_size -= 8;
+ if (cfa->base == CFI_SP)
+ cfa->offset -= 8;
+
+ break;
+
+ default:
+ WARN_FUNC("unknown stack-related instruction",
+ insn->sec, insn->offset);
+ return -1;
+ }
+
+ return 0;
+}
+
+static bool insn_state_match(struct instruction *insn, struct insn_state *state)
+{
+ struct insn_state *state1 = &insn->state, *state2 = state;
+ int i;
+
+ if (memcmp(&state1->cfa, &state2->cfa, sizeof(state1->cfa))) {
+ WARN_FUNC("stack state mismatch: cfa1=%d%+d cfa2=%d%+d",
+ insn->sec, insn->offset,
+ state1->cfa.base, state1->cfa.offset,
+ state2->cfa.base, state2->cfa.offset);
+
+ } else if (memcmp(&state1->regs, &state2->regs, sizeof(state1->regs))) {
+ for (i = 0; i < CFI_NUM_REGS; i++) {
+ if (!memcmp(&state1->regs[i], &state2->regs[i],
+ sizeof(struct cfi_reg)))
+ continue;
+
+ WARN_FUNC("stack state mismatch: reg1[%d]=%d%+d reg2[%d]=%d%+d",
+ insn->sec, insn->offset,
+ i, state1->regs[i].base, state1->regs[i].offset,
+ i, state2->regs[i].base, state2->regs[i].offset);
+ break;
+ }
+
+ } else if (state1->type != state2->type) {
+ WARN_FUNC("stack state mismatch: type1=%d type2=%d",
+ insn->sec, insn->offset, state1->type, state2->type);
+
+ } else if (state1->drap != state2->drap ||
+ (state1->drap && state1->drap_reg != state2->drap_reg) ||
+ (state1->drap && state1->drap_offset != state2->drap_offset)) {
+ WARN_FUNC("stack state mismatch: drap1=%d(%d,%d) drap2=%d(%d,%d)",
+ insn->sec, insn->offset,
+ state1->drap, state1->drap_reg, state1->drap_offset,
+ state2->drap, state2->drap_reg, state2->drap_offset);
+
+ } else
+ return true;
+
+ return false;
+}
+
+/*
+ * Follow the branch starting at the given instruction, and recursively follow
+ * any other branches (jumps). Meanwhile, track the frame pointer state at
+ * each instruction and validate all the rules described in
+ * tools/objtool/Documentation/stack-validation.txt.
+ */
+static int validate_branch(struct objtool_file *file, struct instruction *first,
+ struct insn_state state)
+{
+ struct alternative *alt;
+ struct instruction *insn, *next_insn;
+ struct section *sec;
+ struct symbol *func = NULL;
+ int ret;
+
+ insn = first;
+ sec = insn->sec;
+
+ if (insn->alt_group && list_empty(&insn->alts)) {
+ WARN_FUNC("don't know how to handle branch to middle of alternative instruction group",
+ sec, insn->offset);
+ return 1;
+ }
+
+ while (1) {
+ next_insn = next_insn_same_sec(file, insn);
+
+ if (file->c_file && func && insn->func && func != insn->func->pfunc) {
+ WARN("%s() falls through to next function %s()",
+ func->name, insn->func->name);
+ return 1;
+ }
+
+ func = insn->func ? insn->func->pfunc : NULL;
+
+ if (func && insn->ignore) {
+ WARN_FUNC("BUG: why am I validating an ignored function?",
+ sec, insn->offset);
+ return 1;
+ }
+
+ if (insn->visited) {
+ if (!insn->hint && !insn_state_match(insn, &state))
+ return 1;
+
+ return 0;
+ }
+
+ if (insn->hint) {
+ if (insn->restore) {
+ struct instruction *save_insn, *i;
+
+ i = insn;
+ save_insn = NULL;
+ func_for_each_insn_continue_reverse(file, insn->func, i) {
+ if (i->save) {
+ save_insn = i;
+ break;
+ }
+ }
+
+ if (!save_insn) {
+ WARN_FUNC("no corresponding CFI save for CFI restore",
+ sec, insn->offset);
+ return 1;
+ }
+
+ if (!save_insn->visited) {
+ /*
+ * Oops, no state to copy yet.
+ * Hopefully we can reach this
+ * instruction from another branch
+ * after the save insn has been
+ * visited.
+ */
+ if (insn == first)
+ return 0;
+
+ WARN_FUNC("objtool isn't smart enough to handle this CFI save/restore combo",
+ sec, insn->offset);
+ return 1;
+ }
+
+ insn->state = save_insn->state;
+ }
+
+ state = insn->state;
+
+ } else
+ insn->state = state;
+
+ insn->visited = true;
+
+ if (!insn->ignore_alts) {
+ list_for_each_entry(alt, &insn->alts, list) {
+ ret = validate_branch(file, alt->insn, state);
+ if (ret)
+ return 1;
+ }
+ }
+
+ switch (insn->type) {
+
+ case INSN_RETURN:
+ if (func && has_modified_stack_frame(&state)) {
+ WARN_FUNC("return with modified stack frame",
+ sec, insn->offset);
+ return 1;
+ }
+
+ if (state.bp_scratch) {
+ WARN("%s uses BP as a scratch register",
+ insn->func->name);
+ return 1;
+ }
+
+ return 0;
+
+ case INSN_CALL:
+ if (is_fentry_call(insn))
+ break;
+
+ ret = dead_end_function(file, insn->call_dest);
+ if (ret == 1)
+ return 0;
+ if (ret == -1)
+ return 1;
+
+ /* fallthrough */
+ case INSN_CALL_DYNAMIC:
+ if (!no_fp && func && !has_valid_stack_frame(&state)) {
+ WARN_FUNC("call without frame pointer save/setup",
+ sec, insn->offset);
+ return 1;
+ }
+ break;
+
+ case INSN_JUMP_CONDITIONAL:
+ case INSN_JUMP_UNCONDITIONAL:
+ if (insn->jump_dest &&
+ (!func || !insn->jump_dest->func ||
+ insn->jump_dest->func->pfunc == func)) {
+ ret = validate_branch(file, insn->jump_dest,
+ state);
+ if (ret)
+ return 1;
+
+ } else if (func && has_modified_stack_frame(&state)) {
+ WARN_FUNC("sibling call from callable instruction with modified stack frame",
+ sec, insn->offset);
+ return 1;
+ }
+
+ if (insn->type == INSN_JUMP_UNCONDITIONAL)
+ return 0;
+
+ break;
+
+ case INSN_JUMP_DYNAMIC:
+ if (func && list_empty(&insn->alts) &&
+ has_modified_stack_frame(&state)) {
+ WARN_FUNC("sibling call from callable instruction with modified stack frame",
+ sec, insn->offset);
+ return 1;
+ }
+
+ return 0;
+
+ case INSN_CONTEXT_SWITCH:
+ if (func && (!next_insn || !next_insn->hint)) {
+ WARN_FUNC("unsupported instruction in callable function",
+ sec, insn->offset);
+ return 1;
+ }
+ return 0;
+
+ case INSN_STACK:
+ if (update_insn_state(insn, &state))
+ return 1;
+
+ break;
+
+ default:
+ break;
+ }
+
+ if (insn->dead_end)
+ return 0;
+
+ if (!next_insn) {
+ if (state.cfa.base == CFI_UNDEFINED)
+ return 0;
+ WARN("%s: unexpected end of section", sec->name);
+ return 1;
+ }
+
+ insn = next_insn;
+ }
+
+ return 0;
+}
+
+static int validate_unwind_hints(struct objtool_file *file)
+{
+ struct instruction *insn;
+ int ret, warnings = 0;
+ struct insn_state state;
+
+ if (!file->hints)
+ return 0;
+
+ clear_insn_state(&state);
+
+ for_each_insn(file, insn) {
+ if (insn->hint && !insn->visited) {
+ ret = validate_branch(file, insn, state);
+ warnings += ret;
+ }
+ }
+
+ return warnings;
+}
+
+static int validate_retpoline(struct objtool_file *file)
+{
+ struct instruction *insn;
+ int warnings = 0;
+
+ for_each_insn(file, insn) {
+ if (insn->type != INSN_JUMP_DYNAMIC &&
+ insn->type != INSN_CALL_DYNAMIC)
+ continue;
+
+ if (insn->retpoline_safe)
+ continue;
+
+ /*
+ * .init.text code is ran before userspace and thus doesn't
+ * strictly need retpolines, except for modules which are
+ * loaded late, they very much do need retpoline in their
+ * .init.text
+ */
+ if (!strcmp(insn->sec->name, ".init.text") && !module)
+ continue;
+
+ WARN_FUNC("indirect %s found in RETPOLINE build",
+ insn->sec, insn->offset,
+ insn->type == INSN_JUMP_DYNAMIC ? "jump" : "call");
+
+ warnings++;
+ }
+
+ return warnings;
+}
+
+static bool is_kasan_insn(struct instruction *insn)
+{
+ return (insn->type == INSN_CALL &&
+ !strcmp(insn->call_dest->name, "__asan_handle_no_return"));
+}
+
+static bool is_ubsan_insn(struct instruction *insn)
+{
+ return (insn->type == INSN_CALL &&
+ !strcmp(insn->call_dest->name,
+ "__ubsan_handle_builtin_unreachable"));
+}
+
+static bool ignore_unreachable_insn(struct instruction *insn)
+{
+ int i;
+
+ if (insn->ignore || insn->type == INSN_NOP)
+ return true;
+
+ /*
+ * Ignore any unused exceptions. This can happen when a whitelisted
+ * function has an exception table entry.
+ *
+ * Also ignore alternative replacement instructions. This can happen
+ * when a whitelisted function uses one of the ALTERNATIVE macros.
+ */
+ if (!strcmp(insn->sec->name, ".fixup") ||
+ !strcmp(insn->sec->name, ".altinstr_replacement") ||
+ !strcmp(insn->sec->name, ".altinstr_aux"))
+ return true;
+
+ /*
+ * Check if this (or a subsequent) instruction is related to
+ * CONFIG_UBSAN or CONFIG_KASAN.
+ *
+ * End the search at 5 instructions to avoid going into the weeds.
+ */
+ if (!insn->func)
+ return false;
+ for (i = 0; i < 5; i++) {
+
+ if (is_kasan_insn(insn) || is_ubsan_insn(insn))
+ return true;
+
+ if (insn->type == INSN_JUMP_UNCONDITIONAL) {
+ if (insn->jump_dest &&
+ insn->jump_dest->func == insn->func) {
+ insn = insn->jump_dest;
+ continue;
+ }
+
+ break;
+ }
+
+ if (insn->offset + insn->len >= insn->func->offset + insn->func->len)
+ break;
+
+ insn = list_next_entry(insn, list);
+ }
+
+ return false;
+}
+
+static int validate_functions(struct objtool_file *file)
+{
+ struct section *sec;
+ struct symbol *func;
+ struct instruction *insn;
+ struct insn_state state;
+ int ret, warnings = 0;
+
+ clear_insn_state(&state);
+
+ state.cfa = initial_func_cfi.cfa;
+ memcpy(&state.regs, &initial_func_cfi.regs,
+ CFI_NUM_REGS * sizeof(struct cfi_reg));
+ state.stack_size = initial_func_cfi.cfa.offset;
+
+ for_each_sec(file, sec) {
+ list_for_each_entry(func, &sec->symbol_list, list) {
+ if (func->type != STT_FUNC || func->pfunc != func)
+ continue;
+
+ insn = find_insn(file, sec, func->offset);
+ if (!insn || insn->ignore)
+ continue;
+
+ ret = validate_branch(file, insn, state);
+ warnings += ret;
+ }
+ }
+
+ return warnings;
+}
+
+static int validate_reachable_instructions(struct objtool_file *file)
+{
+ struct instruction *insn;
+
+ if (file->ignore_unreachables)
+ return 0;
+
+ for_each_insn(file, insn) {
+ if (insn->visited || ignore_unreachable_insn(insn))
+ continue;
+
+ WARN_FUNC("unreachable instruction", insn->sec, insn->offset);
+ return 1;
+ }
+
+ return 0;
+}
+
+static void cleanup(struct objtool_file *file)
+{
+ struct instruction *insn, *tmpinsn;
+ struct alternative *alt, *tmpalt;
+
+ list_for_each_entry_safe(insn, tmpinsn, &file->insn_list, list) {
+ list_for_each_entry_safe(alt, tmpalt, &insn->alts, list) {
+ list_del(&alt->list);
+ free(alt);
+ }
+ list_del(&insn->list);
+ hash_del(&insn->hash);
+ free(insn);
+ }
+ elf_close(file->elf);
+}
+
+int check(const char *_objname, bool orc)
+{
+ struct objtool_file file;
+ int ret, warnings = 0;
+
+ objname = _objname;
+
+ file.elf = elf_open(objname, orc ? O_RDWR : O_RDONLY);
+ if (!file.elf)
+ return 1;
+
+ INIT_LIST_HEAD(&file.insn_list);
+ hash_init(file.insn_hash);
+ file.whitelist = find_section_by_name(file.elf, ".discard.func_stack_frame_non_standard");
+ file.rodata = find_section_by_name(file.elf, ".rodata");
+ file.c_file = find_section_by_name(file.elf, ".comment");
+ file.ignore_unreachables = no_unreachable;
+ file.hints = false;
+
+ arch_initial_func_cfi_state(&initial_func_cfi);
+
+ ret = decode_sections(&file);
+ if (ret < 0)
+ goto out;
+ warnings += ret;
+
+ if (list_empty(&file.insn_list))
+ goto out;
+
+ if (retpoline) {
+ ret = validate_retpoline(&file);
+ if (ret < 0)
+ return ret;
+ warnings += ret;
+ }
+
+ ret = validate_functions(&file);
+ if (ret < 0)
+ goto out;
+ warnings += ret;
+
+ ret = validate_unwind_hints(&file);
+ if (ret < 0)
+ goto out;
+ warnings += ret;
+
+ if (!warnings) {
+ ret = validate_reachable_instructions(&file);
+ if (ret < 0)
+ goto out;
+ warnings += ret;
+ }
+
+ if (orc) {
+ ret = create_orc(&file);
+ if (ret < 0)
+ goto out;
+
+ ret = create_orc_sections(&file);
+ if (ret < 0)
+ goto out;
+
+ ret = elf_write(file.elf);
+ if (ret < 0)
+ goto out;
+ }
+
+out:
+ cleanup(&file);
+
+ /* ignore warnings for now until we get all the code cleaned up */
+ if (ret || warnings)
+ return 0;
+ return 0;
+}
diff --git a/tools/objtool/check.h b/tools/objtool/check.h
new file mode 100644
index 0000000..c6b68fc
--- /dev/null
+++ b/tools/objtool/check.h
@@ -0,0 +1,82 @@
+/*
+ * Copyright (C) 2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef _CHECK_H
+#define _CHECK_H
+
+#include <stdbool.h>
+#include "elf.h"
+#include "cfi.h"
+#include "arch.h"
+#include "orc.h"
+#include <linux/hashtable.h>
+
+struct insn_state {
+ struct cfi_reg cfa;
+ struct cfi_reg regs[CFI_NUM_REGS];
+ int stack_size;
+ unsigned char type;
+ bool bp_scratch;
+ bool drap;
+ int drap_reg, drap_offset;
+ struct cfi_reg vals[CFI_NUM_REGS];
+};
+
+struct instruction {
+ struct list_head list;
+ struct hlist_node hash;
+ struct section *sec;
+ unsigned long offset;
+ unsigned int len;
+ unsigned char type;
+ unsigned long immediate;
+ bool alt_group, visited, dead_end, ignore, hint, save, restore, ignore_alts;
+ bool retpoline_safe;
+ struct symbol *call_dest;
+ struct instruction *jump_dest;
+ struct instruction *first_jump_src;
+ struct list_head alts;
+ struct symbol *func;
+ struct stack_op stack_op;
+ struct insn_state state;
+ struct orc_entry orc;
+};
+
+struct objtool_file {
+ struct elf *elf;
+ struct list_head insn_list;
+ DECLARE_HASHTABLE(insn_hash, 16);
+ struct section *rodata, *whitelist;
+ bool ignore_unreachables, c_file, hints;
+};
+
+int check(const char *objname, bool orc);
+
+struct instruction *find_insn(struct objtool_file *file,
+ struct section *sec, unsigned long offset);
+
+#define for_each_insn(file, insn) \
+ list_for_each_entry(insn, &file->insn_list, list)
+
+#define sec_for_each_insn(file, sec, insn) \
+ for (insn = find_insn(file, sec, 0); \
+ insn && &insn->list != &file->insn_list && \
+ insn->sec == sec; \
+ insn = list_next_entry(insn, list))
+
+
+#endif /* _CHECK_H */
diff --git a/tools/objtool/elf.c b/tools/objtool/elf.c
index faacf0c..dd4ed7c 100644
--- a/tools/objtool/elf.c
+++ b/tools/objtool/elf.c
@@ -31,12 +31,7 @@
#include "elf.h"
#include "warn.h"
-/*
- * Fallback for systems without this "read, mmaping if possible" cmd.
- */
-#ifndef ELF_C_READ_MMAP
-#define ELF_C_READ_MMAP ELF_C_READ
-#endif
+#define MAX_NAME_LEN 128
struct section *find_section_by_name(struct elf *elf, const char *name)
{
@@ -86,6 +81,19 @@ struct symbol *find_symbol_by_offset(struct section *sec, unsigned long offset)
return NULL;
}
+struct symbol *find_symbol_by_name(struct elf *elf, const char *name)
+{
+ struct section *sec;
+ struct symbol *sym;
+
+ list_for_each_entry(sec, &elf->sections, list)
+ list_for_each_entry(sym, &sec->symbol_list, list)
+ if (!strcmp(sym->name, name))
+ return sym;
+
+ return NULL;
+}
+
struct symbol *find_symbol_containing(struct section *sec, unsigned long offset)
{
struct symbol *sym;
@@ -140,12 +148,12 @@ static int read_sections(struct elf *elf)
int i;
if (elf_getshdrnum(elf->elf, §ions_nr)) {
- perror("elf_getshdrnum");
+ WARN_ELF("elf_getshdrnum");
return -1;
}
if (elf_getshdrstrndx(elf->elf, &shstrndx)) {
- perror("elf_getshdrstrndx");
+ WARN_ELF("elf_getshdrstrndx");
return -1;
}
@@ -166,37 +174,37 @@ static int read_sections(struct elf *elf)
s = elf_getscn(elf->elf, i);
if (!s) {
- perror("elf_getscn");
+ WARN_ELF("elf_getscn");
return -1;
}
sec->idx = elf_ndxscn(s);
if (!gelf_getshdr(s, &sec->sh)) {
- perror("gelf_getshdr");
+ WARN_ELF("gelf_getshdr");
return -1;
}
sec->name = elf_strptr(elf->elf, shstrndx, sec->sh.sh_name);
if (!sec->name) {
- perror("elf_strptr");
+ WARN_ELF("elf_strptr");
return -1;
}
- sec->elf_data = elf_getdata(s, NULL);
- if (!sec->elf_data) {
- perror("elf_getdata");
- return -1;
+ if (sec->sh.sh_size != 0) {
+ sec->data = elf_getdata(s, NULL);
+ if (!sec->data) {
+ WARN_ELF("elf_getdata");
+ return -1;
+ }
+ if (sec->data->d_off != 0 ||
+ sec->data->d_size != sec->sh.sh_size) {
+ WARN("unexpected data attributes for %s",
+ sec->name);
+ return -1;
+ }
}
-
- if (sec->elf_data->d_off != 0 ||
- sec->elf_data->d_size != sec->sh.sh_size) {
- WARN("unexpected data attributes for %s", sec->name);
- return -1;
- }
-
- sec->data = (unsigned long)sec->elf_data->d_buf;
- sec->len = sec->elf_data->d_size;
+ sec->len = sec->sh.sh_size;
}
/* sanity check, one more call to elf_nextscn() should return NULL */
@@ -210,10 +218,11 @@ static int read_sections(struct elf *elf)
static int read_symbols(struct elf *elf)
{
- struct section *symtab;
- struct symbol *sym;
+ struct section *symtab, *sec;
+ struct symbol *sym, *pfunc;
struct list_head *entry, *tmp;
int symbols_nr, i;
+ char *coldstr;
symtab = find_section_by_name(elf, ".symtab");
if (!symtab) {
@@ -233,15 +242,15 @@ static int read_symbols(struct elf *elf)
sym->idx = i;
- if (!gelf_getsym(symtab->elf_data, i, &sym->sym)) {
- perror("gelf_getsym");
+ if (!gelf_getsym(symtab->data, i, &sym->sym)) {
+ WARN_ELF("gelf_getsym");
goto err;
}
sym->name = elf_strptr(elf->elf, symtab->sh.sh_link,
sym->sym.st_name);
if (!sym->name) {
- perror("elf_strptr");
+ WARN_ELF("elf_strptr");
goto err;
}
@@ -288,6 +297,54 @@ static int read_symbols(struct elf *elf)
hash_add(sym->sec->symbol_hash, &sym->hash, sym->idx);
}
+ /* Create parent/child links for any cold subfunctions */
+ list_for_each_entry(sec, &elf->sections, list) {
+ list_for_each_entry(sym, &sec->symbol_list, list) {
+ char pname[MAX_NAME_LEN + 1];
+ size_t pnamelen;
+ if (sym->type != STT_FUNC)
+ continue;
+ sym->pfunc = sym->cfunc = sym;
+ coldstr = strstr(sym->name, ".cold.");
+ if (!coldstr)
+ continue;
+
+ pnamelen = coldstr - sym->name;
+ if (pnamelen > MAX_NAME_LEN) {
+ WARN("%s(): parent function name exceeds maximum length of %d characters",
+ sym->name, MAX_NAME_LEN);
+ return -1;
+ }
+
+ strncpy(pname, sym->name, pnamelen);
+ pname[pnamelen] = '\0';
+ pfunc = find_symbol_by_name(elf, pname);
+
+ if (!pfunc) {
+ WARN("%s(): can't find parent function",
+ sym->name);
+ return -1;
+ }
+
+ sym->pfunc = pfunc;
+ pfunc->cfunc = sym;
+
+ /*
+ * Unfortunately, -fnoreorder-functions puts the child
+ * inside the parent. Remove the overlap so we can
+ * have sane assumptions.
+ *
+ * Note that pfunc->len now no longer matches
+ * pfunc->sym.st_size.
+ */
+ if (sym->sec == pfunc->sec &&
+ sym->offset >= pfunc->offset &&
+ sym->offset + sym->len == pfunc->offset + pfunc->len) {
+ pfunc->len -= sym->len;
+ }
+ }
+ }
+
return 0;
err:
@@ -323,8 +380,8 @@ static int read_relas(struct elf *elf)
}
memset(rela, 0, sizeof(*rela));
- if (!gelf_getrela(sec->elf_data, i, &rela->rela)) {
- perror("gelf_getrela");
+ if (!gelf_getrela(sec->data, i, &rela->rela)) {
+ WARN_ELF("gelf_getrela");
return -1;
}
@@ -348,9 +405,10 @@ static int read_relas(struct elf *elf)
return 0;
}
-struct elf *elf_open(const char *name)
+struct elf *elf_open(const char *name, int flags)
{
struct elf *elf;
+ Elf_Cmd cmd;
elf_version(EV_CURRENT);
@@ -363,27 +421,28 @@ struct elf *elf_open(const char *name)
INIT_LIST_HEAD(&elf->sections);
- elf->name = strdup(name);
- if (!elf->name) {
- perror("strdup");
- goto err;
- }
-
- elf->fd = open(name, O_RDONLY);
+ elf->fd = open(name, flags);
if (elf->fd == -1) {
fprintf(stderr, "objtool: Can't open '%s': %s\n",
name, strerror(errno));
goto err;
}
- elf->elf = elf_begin(elf->fd, ELF_C_READ_MMAP, NULL);
+ if ((flags & O_ACCMODE) == O_RDONLY)
+ cmd = ELF_C_READ_MMAP;
+ else if ((flags & O_ACCMODE) == O_RDWR)
+ cmd = ELF_C_RDWR;
+ else /* O_WRONLY */
+ cmd = ELF_C_WRITE;
+
+ elf->elf = elf_begin(elf->fd, cmd, NULL);
if (!elf->elf) {
- perror("elf_begin");
+ WARN_ELF("elf_begin");
goto err;
}
if (!gelf_getehdr(elf->elf, &elf->ehdr)) {
- perror("gelf_getehdr");
+ WARN_ELF("gelf_getehdr");
goto err;
}
@@ -403,12 +462,212 @@ struct elf *elf_open(const char *name)
return NULL;
}
+struct section *elf_create_section(struct elf *elf, const char *name,
+ size_t entsize, int nr)
+{
+ struct section *sec, *shstrtab;
+ size_t size = entsize * nr;
+ struct Elf_Scn *s;
+ Elf_Data *data;
+
+ sec = malloc(sizeof(*sec));
+ if (!sec) {
+ perror("malloc");
+ return NULL;
+ }
+ memset(sec, 0, sizeof(*sec));
+
+ INIT_LIST_HEAD(&sec->symbol_list);
+ INIT_LIST_HEAD(&sec->rela_list);
+ hash_init(sec->rela_hash);
+ hash_init(sec->symbol_hash);
+
+ list_add_tail(&sec->list, &elf->sections);
+
+ s = elf_newscn(elf->elf);
+ if (!s) {
+ WARN_ELF("elf_newscn");
+ return NULL;
+ }
+
+ sec->name = strdup(name);
+ if (!sec->name) {
+ perror("strdup");
+ return NULL;
+ }
+
+ sec->idx = elf_ndxscn(s);
+ sec->len = size;
+ sec->changed = true;
+
+ sec->data = elf_newdata(s);
+ if (!sec->data) {
+ WARN_ELF("elf_newdata");
+ return NULL;
+ }
+
+ sec->data->d_size = size;
+ sec->data->d_align = 1;
+
+ if (size) {
+ sec->data->d_buf = malloc(size);
+ if (!sec->data->d_buf) {
+ perror("malloc");
+ return NULL;
+ }
+ memset(sec->data->d_buf, 0, size);
+ }
+
+ if (!gelf_getshdr(s, &sec->sh)) {
+ WARN_ELF("gelf_getshdr");
+ return NULL;
+ }
+
+ sec->sh.sh_size = size;
+ sec->sh.sh_entsize = entsize;
+ sec->sh.sh_type = SHT_PROGBITS;
+ sec->sh.sh_addralign = 1;
+ sec->sh.sh_flags = SHF_ALLOC;
+
+
+ /* Add section name to .shstrtab */
+ shstrtab = find_section_by_name(elf, ".shstrtab");
+ if (!shstrtab) {
+ WARN("can't find .shstrtab section");
+ return NULL;
+ }
+
+ s = elf_getscn(elf->elf, shstrtab->idx);
+ if (!s) {
+ WARN_ELF("elf_getscn");
+ return NULL;
+ }
+
+ data = elf_newdata(s);
+ if (!data) {
+ WARN_ELF("elf_newdata");
+ return NULL;
+ }
+
+ data->d_buf = sec->name;
+ data->d_size = strlen(name) + 1;
+ data->d_align = 1;
+
+ sec->sh.sh_name = shstrtab->len;
+
+ shstrtab->len += strlen(name) + 1;
+ shstrtab->changed = true;
+
+ return sec;
+}
+
+struct section *elf_create_rela_section(struct elf *elf, struct section *base)
+{
+ char *relaname;
+ struct section *sec;
+
+ relaname = malloc(strlen(base->name) + strlen(".rela") + 1);
+ if (!relaname) {
+ perror("malloc");
+ return NULL;
+ }
+ strcpy(relaname, ".rela");
+ strcat(relaname, base->name);
+
+ sec = elf_create_section(elf, relaname, sizeof(GElf_Rela), 0);
+ free(relaname);
+ if (!sec)
+ return NULL;
+
+ base->rela = sec;
+ sec->base = base;
+
+ sec->sh.sh_type = SHT_RELA;
+ sec->sh.sh_addralign = 8;
+ sec->sh.sh_link = find_section_by_name(elf, ".symtab")->idx;
+ sec->sh.sh_info = base->idx;
+ sec->sh.sh_flags = SHF_INFO_LINK;
+
+ return sec;
+}
+
+int elf_rebuild_rela_section(struct section *sec)
+{
+ struct rela *rela;
+ int nr, idx = 0, size;
+ GElf_Rela *relas;
+
+ nr = 0;
+ list_for_each_entry(rela, &sec->rela_list, list)
+ nr++;
+
+ size = nr * sizeof(*relas);
+ relas = malloc(size);
+ if (!relas) {
+ perror("malloc");
+ return -1;
+ }
+
+ sec->data->d_buf = relas;
+ sec->data->d_size = size;
+
+ sec->sh.sh_size = size;
+
+ idx = 0;
+ list_for_each_entry(rela, &sec->rela_list, list) {
+ relas[idx].r_offset = rela->offset;
+ relas[idx].r_addend = rela->addend;
+ relas[idx].r_info = GELF_R_INFO(rela->sym->idx, rela->type);
+ idx++;
+ }
+
+ return 0;
+}
+
+int elf_write(struct elf *elf)
+{
+ struct section *sec;
+ Elf_Scn *s;
+
+ /* Update section headers for changed sections: */
+ list_for_each_entry(sec, &elf->sections, list) {
+ if (sec->changed) {
+ s = elf_getscn(elf->elf, sec->idx);
+ if (!s) {
+ WARN_ELF("elf_getscn");
+ return -1;
+ }
+ if (!gelf_update_shdr(s, &sec->sh)) {
+ WARN_ELF("gelf_update_shdr");
+ return -1;
+ }
+ }
+ }
+
+ /* Make sure the new section header entries get updated properly. */
+ elf_flagelf(elf->elf, ELF_C_SET, ELF_F_DIRTY);
+
+ /* Write all changes to the file. */
+ if (elf_update(elf->elf, ELF_C_WRITE) < 0) {
+ WARN_ELF("elf_update");
+ return -1;
+ }
+
+ return 0;
+}
+
void elf_close(struct elf *elf)
{
struct section *sec, *tmpsec;
struct symbol *sym, *tmpsym;
struct rela *rela, *tmprela;
+ if (elf->elf)
+ elf_end(elf->elf);
+
+ if (elf->fd > 0)
+ close(elf->fd);
+
list_for_each_entry_safe(sec, tmpsec, &elf->sections, list) {
list_for_each_entry_safe(sym, tmpsym, &sec->symbol_list, list) {
list_del(&sym->list);
@@ -423,11 +682,6 @@ void elf_close(struct elf *elf)
list_del(&sec->list);
free(sec);
}
- if (elf->name)
- free(elf->name);
- if (elf->fd > 0)
- close(elf->fd);
- if (elf->elf)
- elf_end(elf->elf);
+
free(elf);
}
diff --git a/tools/objtool/elf.h b/tools/objtool/elf.h
index 731973e..de5cd2d 100644
--- a/tools/objtool/elf.h
+++ b/tools/objtool/elf.h
@@ -28,6 +28,13 @@
# define elf_getshdrstrndx elf_getshstrndx
#endif
+/*
+ * Fallback for systems without this "read, mmaping if possible" cmd.
+ */
+#ifndef ELF_C_READ_MMAP
+#define ELF_C_READ_MMAP ELF_C_READ
+#endif
+
struct section {
struct list_head list;
GElf_Shdr sh;
@@ -37,11 +44,11 @@ struct section {
DECLARE_HASHTABLE(rela_hash, 16);
struct section *base, *rela;
struct symbol *sym;
- Elf_Data *elf_data;
+ Elf_Data *data;
char *name;
int idx;
- unsigned long data;
unsigned int len;
+ bool changed, text;
};
struct symbol {
@@ -54,6 +61,7 @@ struct symbol {
unsigned char bind, type;
unsigned long offset;
unsigned int len;
+ struct symbol *pfunc, *cfunc;
};
struct rela {
@@ -76,16 +84,23 @@ struct elf {
};
-struct elf *elf_open(const char *name);
+struct elf *elf_open(const char *name, int flags);
struct section *find_section_by_name(struct elf *elf, const char *name);
struct symbol *find_symbol_by_offset(struct section *sec, unsigned long offset);
+struct symbol *find_symbol_by_name(struct elf *elf, const char *name);
struct symbol *find_symbol_containing(struct section *sec, unsigned long offset);
struct rela *find_rela_by_dest(struct section *sec, unsigned long offset);
struct rela *find_rela_by_dest_range(struct section *sec, unsigned long offset,
unsigned int len);
struct symbol *find_containing_func(struct section *sec, unsigned long offset);
+struct section *elf_create_section(struct elf *elf, const char *name, size_t
+ entsize, int nr);
+struct section *elf_create_rela_section(struct elf *elf, struct section *base);
+int elf_rebuild_rela_section(struct section *sec);
+int elf_write(struct elf *elf);
void elf_close(struct elf *elf);
-
+#define for_each_sec(file, sec) \
+ list_for_each_entry(sec, &file->elf->sections, list)
#endif /* _OBJTOOL_ELF_H */
diff --git a/tools/objtool/objtool.c b/tools/objtool/objtool.c
index 46c326d..07f3299 100644
--- a/tools/objtool/objtool.c
+++ b/tools/objtool/objtool.c
@@ -31,11 +31,10 @@
#include <stdlib.h>
#include <subcmd/exec-cmd.h>
#include <subcmd/pager.h>
+#include <linux/kernel.h>
#include "builtin.h"
-#define ARRAY_SIZE(x) (sizeof(x)/sizeof(x[0]))
-
struct cmd_struct {
const char *name;
int (*fn)(int, const char **);
@@ -43,10 +42,11 @@ struct cmd_struct {
};
static const char objtool_usage_string[] =
- "objtool [OPTIONS] COMMAND [ARGS]";
+ "objtool COMMAND [ARGS]";
static struct cmd_struct objtool_cmds[] = {
{"check", cmd_check, "Perform stack metadata validation on an object file" },
+ {"orc", cmd_orc, "Generate in-place ORC unwind tables for an object file" },
};
bool help;
@@ -70,7 +70,7 @@ static void cmd_usage(void)
printf("\n");
- exit(1);
+ exit(129);
}
static void handle_options(int *argc, const char ***argv)
@@ -86,9 +86,7 @@ static void handle_options(int *argc, const char ***argv)
break;
} else {
fprintf(stderr, "Unknown option: %s\n", cmd);
- fprintf(stderr, "\n Usage: %s\n",
- objtool_usage_string);
- exit(1);
+ cmd_usage();
}
(*argv)++;
diff --git a/tools/objtool/orc.h b/tools/objtool/orc.h
new file mode 100644
index 0000000..b0e92a6
--- /dev/null
+++ b/tools/objtool/orc.h
@@ -0,0 +1,30 @@
+/*
+ * Copyright (C) 2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef _ORC_H
+#define _ORC_H
+
+#include <asm/orc_types.h>
+
+struct objtool_file;
+
+int create_orc(struct objtool_file *file);
+int create_orc_sections(struct objtool_file *file);
+
+int orc_dump(const char *objname);
+
+#endif /* _ORC_H */
diff --git a/tools/objtool/orc_dump.c b/tools/objtool/orc_dump.c
new file mode 100644
index 0000000..c334382
--- /dev/null
+++ b/tools/objtool/orc_dump.c
@@ -0,0 +1,213 @@
+/*
+ * Copyright (C) 2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <unistd.h>
+#include "orc.h"
+#include "warn.h"
+
+static const char *reg_name(unsigned int reg)
+{
+ switch (reg) {
+ case ORC_REG_PREV_SP:
+ return "prevsp";
+ case ORC_REG_DX:
+ return "dx";
+ case ORC_REG_DI:
+ return "di";
+ case ORC_REG_BP:
+ return "bp";
+ case ORC_REG_SP:
+ return "sp";
+ case ORC_REG_R10:
+ return "r10";
+ case ORC_REG_R13:
+ return "r13";
+ case ORC_REG_BP_INDIRECT:
+ return "bp(ind)";
+ case ORC_REG_SP_INDIRECT:
+ return "sp(ind)";
+ default:
+ return "?";
+ }
+}
+
+static const char *orc_type_name(unsigned int type)
+{
+ switch (type) {
+ case ORC_TYPE_CALL:
+ return "call";
+ case ORC_TYPE_REGS:
+ return "regs";
+ case ORC_TYPE_REGS_IRET:
+ return "iret";
+ default:
+ return "?";
+ }
+}
+
+static void print_reg(unsigned int reg, int offset)
+{
+ if (reg == ORC_REG_BP_INDIRECT)
+ printf("(bp%+d)", offset);
+ else if (reg == ORC_REG_SP_INDIRECT)
+ printf("(sp%+d)", offset);
+ else if (reg == ORC_REG_UNDEFINED)
+ printf("(und)");
+ else
+ printf("%s%+d", reg_name(reg), offset);
+}
+
+int orc_dump(const char *_objname)
+{
+ int fd, nr_entries, i, *orc_ip = NULL, orc_size = 0;
+ struct orc_entry *orc = NULL;
+ char *name;
+ size_t nr_sections;
+ Elf64_Addr orc_ip_addr = 0;
+ size_t shstrtab_idx;
+ Elf *elf;
+ Elf_Scn *scn;
+ GElf_Shdr sh;
+ GElf_Rela rela;
+ GElf_Sym sym;
+ Elf_Data *data, *symtab = NULL, *rela_orc_ip = NULL;
+
+
+ objname = _objname;
+
+ elf_version(EV_CURRENT);
+
+ fd = open(objname, O_RDONLY);
+ if (fd == -1) {
+ perror("open");
+ return -1;
+ }
+
+ elf = elf_begin(fd, ELF_C_READ_MMAP, NULL);
+ if (!elf) {
+ WARN_ELF("elf_begin");
+ return -1;
+ }
+
+ if (elf_getshdrnum(elf, &nr_sections)) {
+ WARN_ELF("elf_getshdrnum");
+ return -1;
+ }
+
+ if (elf_getshdrstrndx(elf, &shstrtab_idx)) {
+ WARN_ELF("elf_getshdrstrndx");
+ return -1;
+ }
+
+ for (i = 0; i < nr_sections; i++) {
+ scn = elf_getscn(elf, i);
+ if (!scn) {
+ WARN_ELF("elf_getscn");
+ return -1;
+ }
+
+ if (!gelf_getshdr(scn, &sh)) {
+ WARN_ELF("gelf_getshdr");
+ return -1;
+ }
+
+ name = elf_strptr(elf, shstrtab_idx, sh.sh_name);
+ if (!name) {
+ WARN_ELF("elf_strptr");
+ return -1;
+ }
+
+ data = elf_getdata(scn, NULL);
+ if (!data) {
+ WARN_ELF("elf_getdata");
+ return -1;
+ }
+
+ if (!strcmp(name, ".symtab")) {
+ symtab = data;
+ } else if (!strcmp(name, ".orc_unwind")) {
+ orc = data->d_buf;
+ orc_size = sh.sh_size;
+ } else if (!strcmp(name, ".orc_unwind_ip")) {
+ orc_ip = data->d_buf;
+ orc_ip_addr = sh.sh_addr;
+ } else if (!strcmp(name, ".rela.orc_unwind_ip")) {
+ rela_orc_ip = data;
+ }
+ }
+
+ if (!symtab || !orc || !orc_ip)
+ return 0;
+
+ if (orc_size % sizeof(*orc) != 0) {
+ WARN("bad .orc_unwind section size");
+ return -1;
+ }
+
+ nr_entries = orc_size / sizeof(*orc);
+ for (i = 0; i < nr_entries; i++) {
+ if (rela_orc_ip) {
+ if (!gelf_getrela(rela_orc_ip, i, &rela)) {
+ WARN_ELF("gelf_getrela");
+ return -1;
+ }
+
+ if (!gelf_getsym(symtab, GELF_R_SYM(rela.r_info), &sym)) {
+ WARN_ELF("gelf_getsym");
+ return -1;
+ }
+
+ scn = elf_getscn(elf, sym.st_shndx);
+ if (!scn) {
+ WARN_ELF("elf_getscn");
+ return -1;
+ }
+
+ if (!gelf_getshdr(scn, &sh)) {
+ WARN_ELF("gelf_getshdr");
+ return -1;
+ }
+
+ name = elf_strptr(elf, shstrtab_idx, sh.sh_name);
+ if (!name || !*name) {
+ WARN_ELF("elf_strptr");
+ return -1;
+ }
+
+ printf("%s+%llx:", name, (unsigned long long)rela.r_addend);
+
+ } else {
+ printf("%llx:", (unsigned long long)(orc_ip_addr + (i * sizeof(int)) + orc_ip[i]));
+ }
+
+
+ printf(" sp:");
+
+ print_reg(orc[i].sp_reg, orc[i].sp_offset);
+
+ printf(" bp:");
+
+ print_reg(orc[i].bp_reg, orc[i].bp_offset);
+
+ printf(" type:%s\n", orc_type_name(orc[i].type));
+ }
+
+ elf_end(elf);
+ close(fd);
+
+ return 0;
+}
diff --git a/tools/objtool/orc_gen.c b/tools/objtool/orc_gen.c
new file mode 100644
index 0000000..18384d9
--- /dev/null
+++ b/tools/objtool/orc_gen.c
@@ -0,0 +1,221 @@
+/*
+ * Copyright (C) 2017 Josh Poimboeuf <jpoimboe@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version 2
+ * of the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <stdlib.h>
+#include <string.h>
+
+#include "orc.h"
+#include "check.h"
+#include "warn.h"
+
+int create_orc(struct objtool_file *file)
+{
+ struct instruction *insn;
+
+ for_each_insn(file, insn) {
+ struct orc_entry *orc = &insn->orc;
+ struct cfi_reg *cfa = &insn->state.cfa;
+ struct cfi_reg *bp = &insn->state.regs[CFI_BP];
+
+ if (cfa->base == CFI_UNDEFINED) {
+ orc->sp_reg = ORC_REG_UNDEFINED;
+ continue;
+ }
+
+ switch (cfa->base) {
+ case CFI_SP:
+ orc->sp_reg = ORC_REG_SP;
+ break;
+ case CFI_SP_INDIRECT:
+ orc->sp_reg = ORC_REG_SP_INDIRECT;
+ break;
+ case CFI_BP:
+ orc->sp_reg = ORC_REG_BP;
+ break;
+ case CFI_BP_INDIRECT:
+ orc->sp_reg = ORC_REG_BP_INDIRECT;
+ break;
+ case CFI_R10:
+ orc->sp_reg = ORC_REG_R10;
+ break;
+ case CFI_R13:
+ orc->sp_reg = ORC_REG_R13;
+ break;
+ case CFI_DI:
+ orc->sp_reg = ORC_REG_DI;
+ break;
+ case CFI_DX:
+ orc->sp_reg = ORC_REG_DX;
+ break;
+ default:
+ WARN_FUNC("unknown CFA base reg %d",
+ insn->sec, insn->offset, cfa->base);
+ return -1;
+ }
+
+ switch(bp->base) {
+ case CFI_UNDEFINED:
+ orc->bp_reg = ORC_REG_UNDEFINED;
+ break;
+ case CFI_CFA:
+ orc->bp_reg = ORC_REG_PREV_SP;
+ break;
+ case CFI_BP:
+ orc->bp_reg = ORC_REG_BP;
+ break;
+ default:
+ WARN_FUNC("unknown BP base reg %d",
+ insn->sec, insn->offset, bp->base);
+ return -1;
+ }
+
+ orc->sp_offset = cfa->offset;
+ orc->bp_offset = bp->offset;
+ orc->type = insn->state.type;
+ }
+
+ return 0;
+}
+
+static int create_orc_entry(struct section *u_sec, struct section *ip_relasec,
+ unsigned int idx, struct section *insn_sec,
+ unsigned long insn_off, struct orc_entry *o)
+{
+ struct orc_entry *orc;
+ struct rela *rela;
+
+ if (!insn_sec->sym) {
+ WARN("missing symbol for section %s", insn_sec->name);
+ return -1;
+ }
+
+ /* populate ORC data */
+ orc = (struct orc_entry *)u_sec->data->d_buf + idx;
+ memcpy(orc, o, sizeof(*orc));
+
+ /* populate rela for ip */
+ rela = malloc(sizeof(*rela));
+ if (!rela) {
+ perror("malloc");
+ return -1;
+ }
+ memset(rela, 0, sizeof(*rela));
+
+ rela->sym = insn_sec->sym;
+ rela->addend = insn_off;
+ rela->type = R_X86_64_PC32;
+ rela->offset = idx * sizeof(int);
+
+ list_add_tail(&rela->list, &ip_relasec->rela_list);
+ hash_add(ip_relasec->rela_hash, &rela->hash, rela->offset);
+
+ return 0;
+}
+
+int create_orc_sections(struct objtool_file *file)
+{
+ struct instruction *insn, *prev_insn;
+ struct section *sec, *u_sec, *ip_relasec;
+ unsigned int idx;
+
+ struct orc_entry empty = {
+ .sp_reg = ORC_REG_UNDEFINED,
+ .bp_reg = ORC_REG_UNDEFINED,
+ .type = ORC_TYPE_CALL,
+ };
+
+ sec = find_section_by_name(file->elf, ".orc_unwind");
+ if (sec) {
+ WARN("file already has .orc_unwind section, skipping");
+ return -1;
+ }
+
+ /* count the number of needed orcs */
+ idx = 0;
+ for_each_sec(file, sec) {
+ if (!sec->text)
+ continue;
+
+ prev_insn = NULL;
+ sec_for_each_insn(file, sec, insn) {
+ if (!prev_insn ||
+ memcmp(&insn->orc, &prev_insn->orc,
+ sizeof(struct orc_entry))) {
+ idx++;
+ }
+ prev_insn = insn;
+ }
+
+ /* section terminator */
+ if (prev_insn)
+ idx++;
+ }
+ if (!idx)
+ return -1;
+
+
+ /* create .orc_unwind_ip and .rela.orc_unwind_ip sections */
+ sec = elf_create_section(file->elf, ".orc_unwind_ip", sizeof(int), idx);
+ if (!sec)
+ return -1;
+
+ ip_relasec = elf_create_rela_section(file->elf, sec);
+ if (!ip_relasec)
+ return -1;
+
+ /* create .orc_unwind section */
+ u_sec = elf_create_section(file->elf, ".orc_unwind",
+ sizeof(struct orc_entry), idx);
+
+ /* populate sections */
+ idx = 0;
+ for_each_sec(file, sec) {
+ if (!sec->text)
+ continue;
+
+ prev_insn = NULL;
+ sec_for_each_insn(file, sec, insn) {
+ if (!prev_insn || memcmp(&insn->orc, &prev_insn->orc,
+ sizeof(struct orc_entry))) {
+
+ if (create_orc_entry(u_sec, ip_relasec, idx,
+ insn->sec, insn->offset,
+ &insn->orc))
+ return -1;
+
+ idx++;
+ }
+ prev_insn = insn;
+ }
+
+ /* section terminator */
+ if (prev_insn) {
+ if (create_orc_entry(u_sec, ip_relasec, idx,
+ prev_insn->sec,
+ prev_insn->offset + prev_insn->len,
+ &empty))
+ return -1;
+
+ idx++;
+ }
+ }
+
+ if (elf_rebuild_rela_section(ip_relasec))
+ return -1;
+
+ return 0;
+}
diff --git a/tools/objtool/special.c b/tools/objtool/special.c
index bff8abb..84f001d 100644
--- a/tools/objtool/special.c
+++ b/tools/objtool/special.c
@@ -91,16 +91,16 @@ static int get_alt_entry(struct elf *elf, struct special_entry *entry,
alt->jump_or_nop = entry->jump_or_nop;
if (alt->group) {
- alt->orig_len = *(unsigned char *)(sec->data + offset +
+ alt->orig_len = *(unsigned char *)(sec->data->d_buf + offset +
entry->orig_len);
- alt->new_len = *(unsigned char *)(sec->data + offset +
+ alt->new_len = *(unsigned char *)(sec->data->d_buf + offset +
entry->new_len);
}
if (entry->feature) {
unsigned short feature;
- feature = *(unsigned short *)(sec->data + offset +
+ feature = *(unsigned short *)(sec->data->d_buf + offset +
entry->feature);
/*
diff --git a/tools/objtool/sync-check.sh b/tools/objtool/sync-check.sh
new file mode 100755
index 0000000..1470e74
--- /dev/null
+++ b/tools/objtool/sync-check.sh
@@ -0,0 +1,29 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+
+FILES='
+arch/x86/lib/insn.c
+arch/x86/lib/inat.c
+arch/x86/lib/x86-opcode-map.txt
+arch/x86/tools/gen-insn-attr-x86.awk
+arch/x86/include/asm/insn.h
+arch/x86/include/asm/inat.h
+arch/x86/include/asm/inat_types.h
+arch/x86/include/asm/orc_types.h
+'
+
+check()
+{
+ local file=$1
+
+ diff $file ../../$file > /dev/null ||
+ echo "Warning: synced file at 'tools/objtool/$file' differs from latest kernel version at '$file'"
+}
+
+if [ ! -d ../../kernel ] || [ ! -d ../../tools ] || [ ! -d ../objtool ]; then
+ exit 0
+fi
+
+for i in $FILES; do
+ check $i
+done
diff --git a/tools/objtool/warn.h b/tools/objtool/warn.h
index ac7e075..afd9f7a 100644
--- a/tools/objtool/warn.h
+++ b/tools/objtool/warn.h
@@ -18,6 +18,13 @@
#ifndef _WARN_H
#define _WARN_H
+#include <stdlib.h>
+#include <string.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include "elf.h"
+
extern const char *objname;
static inline char *offstr(struct section *sec, unsigned long offset)
@@ -57,4 +64,7 @@ static inline char *offstr(struct section *sec, unsigned long offset)
free(_str); \
})
+#define WARN_ELF(format, ...) \
+ WARN(format ": %s", ##__VA_ARGS__, elf_errmsg(-1))
+
#endif /* _WARN_H */
diff --git a/tools/perf/MANIFEST b/tools/perf/MANIFEST
index 0bda2cc..a4f98e1 100644
--- a/tools/perf/MANIFEST
+++ b/tools/perf/MANIFEST
@@ -51,6 +51,7 @@
tools/include/asm-generic/bitops/atomic.h
tools/include/asm-generic/bitops/const_hweight.h
tools/include/asm-generic/bitops/__ffs.h
+tools/include/asm-generic/bitops/__ffz.h
tools/include/asm-generic/bitops/__fls.h
tools/include/asm-generic/bitops/find.h
tools/include/asm-generic/bitops/fls64.h
@@ -60,7 +61,9 @@
tools/include/linux/atomic.h
tools/include/linux/bitops.h
tools/include/linux/compiler.h
+tools/include/linux/compiler-gcc.h
tools/include/linux/coresight-pmu.h
+tools/include/linux/bug.h
tools/include/linux/filter.h
tools/include/linux/hash.h
tools/include/linux/kernel.h
@@ -70,12 +73,15 @@
tools/include/uapi/asm-generic/mman.h
tools/include/uapi/linux/bpf.h
tools/include/uapi/linux/bpf_common.h
+tools/include/uapi/linux/fcntl.h
tools/include/uapi/linux/hw_breakpoint.h
tools/include/uapi/linux/mman.h
tools/include/uapi/linux/perf_event.h
+tools/include/uapi/linux/stat.h
tools/include/linux/poison.h
tools/include/linux/rbtree.h
tools/include/linux/rbtree_augmented.h
+tools/include/linux/refcount.h
tools/include/linux/string.h
tools/include/linux/stringify.h
tools/include/linux/types.h
diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index 32a64e6..cd86fd7 100644
--- a/tools/perf/Makefile
+++ b/tools/perf/Makefile
@@ -83,10 +83,10 @@
endif # MAKECMDGOALS
#
-# The clean target is not really parallel, don't print the jobs info:
+# Explicitly disable parallelism for the clean target.
#
clean:
- $(make)
+ $(make) -j1
#
# The build-test target is not really parallel, don't print the jobs info,
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 2b92ffe..ad3726c 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -177,6 +177,36 @@
endif
endif
+# The fixdep build - we force fixdep tool to be built as
+# the first target in the separate make session not to be
+# disturbed by any parallel make jobs. Once fixdep is done
+# we issue the requested build with FIXDEP=1 variable.
+#
+# The fixdep build is disabled for $(NON_CONFIG_TARGETS)
+# targets, because it's not necessary.
+
+ifdef FIXDEP
+ force_fixdep := 0
+else
+ force_fixdep := $(config)
+endif
+
+export srctree OUTPUT RM CC CXX LD AR CFLAGS CXXFLAGS V BISON FLEX AWK
+export HOSTCC HOSTLD HOSTAR
+
+include $(srctree)/tools/build/Makefile.include
+
+ifeq ($(force_fixdep),1)
+goals := $(filter-out all sub-make, $(MAKECMDGOALS))
+
+$(goals) all: sub-make
+
+sub-make: fixdep
+ @./check-headers.sh
+ $(Q)$(MAKE) FIXDEP=1 -f Makefile.perf $(goals)
+
+else # force_fixdep
+
# Set FEATURE_TESTS to 'all' so all possible feature checkers are executed.
# Without this setting the output feature dump file misses some features, for
# example, liberty. Select all checkers so we won't get an incomplete feature
@@ -348,10 +378,6 @@
PERF_IN := $(OUTPUT)perf-in.o
-export srctree OUTPUT RM CC LD AR CFLAGS V BISON FLEX AWK
-export HOSTCC HOSTLD HOSTAR
-include $(srctree)/tools/build/Makefile.include
-
JEVENTS := $(OUTPUT)pmu-events/jevents
JEVENTS_IN := $(OUTPUT)pmu-events/jevents-in.o
@@ -362,99 +388,6 @@
build := -f $(srctree)/tools/build/Makefile.build dir=. obj
$(PERF_IN): prepare FORCE
- @(test -f ../../include/uapi/linux/perf_event.h && ( \
- (diff -B ../include/uapi/linux/perf_event.h ../../include/uapi/linux/perf_event.h >/dev/null) \
- || echo "Warning: tools/include/uapi/linux/perf_event.h differs from kernel" >&2 )) || true
- @(test -f ../../include/linux/hash.h && ( \
- (diff -B ../include/linux/hash.h ../../include/linux/hash.h >/dev/null) \
- || echo "Warning: tools/include/linux/hash.h differs from kernel" >&2 )) || true
- @(test -f ../../include/uapi/linux/hw_breakpoint.h && ( \
- (diff -B ../include/uapi/linux/hw_breakpoint.h ../../include/uapi/linux/hw_breakpoint.h >/dev/null) \
- || echo "Warning: tools/include/uapi/linux/hw_breakpoint.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/x86/include/asm/disabled-features.h && ( \
- (diff -B ../arch/x86/include/asm/disabled-features.h ../../arch/x86/include/asm/disabled-features.h >/dev/null) \
- || echo "Warning: tools/arch/x86/include/asm/disabled-features.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/x86/include/asm/required-features.h && ( \
- (diff -B ../arch/x86/include/asm/required-features.h ../../arch/x86/include/asm/required-features.h >/dev/null) \
- || echo "Warning: tools/arch/x86/include/asm/required-features.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/x86/include/asm/cpufeatures.h && ( \
- (diff -B ../arch/x86/include/asm/cpufeatures.h ../../arch/x86/include/asm/cpufeatures.h >/dev/null) \
- || echo "Warning: tools/arch/x86/include/asm/cpufeatures.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/x86/lib/memcpy_64.S && ( \
- (diff -B ../arch/x86/lib/memcpy_64.S ../../arch/x86/lib/memcpy_64.S >/dev/null) \
- || echo "Warning: tools/arch/x86/lib/memcpy_64.S differs from kernel" >&2 )) || true
- @(test -f ../../arch/x86/lib/memset_64.S && ( \
- (diff -B ../arch/x86/lib/memset_64.S ../../arch/x86/lib/memset_64.S >/dev/null) \
- || echo "Warning: tools/arch/x86/lib/memset_64.S differs from kernel" >&2 )) || true
- @(test -f ../../arch/arm/include/uapi/asm/perf_regs.h && ( \
- (diff -B ../arch/arm/include/uapi/asm/perf_regs.h ../../arch/arm/include/uapi/asm/perf_regs.h >/dev/null) \
- || echo "Warning: tools/arch/arm/include/uapi/asm/perf_regs.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/arm64/include/uapi/asm/perf_regs.h && ( \
- (diff -B ../arch/arm64/include/uapi/asm/perf_regs.h ../../arch/arm64/include/uapi/asm/perf_regs.h >/dev/null) \
- || echo "Warning: tools/arch/arm64/include/uapi/asm/perf_regs.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/powerpc/include/uapi/asm/perf_regs.h && ( \
- (diff -B ../arch/powerpc/include/uapi/asm/perf_regs.h ../../arch/powerpc/include/uapi/asm/perf_regs.h >/dev/null) \
- || echo "Warning: tools/arch/powerpc/include/uapi/asm/perf_regs.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/x86/include/uapi/asm/perf_regs.h && ( \
- (diff -B ../arch/x86/include/uapi/asm/perf_regs.h ../../arch/x86/include/uapi/asm/perf_regs.h >/dev/null) \
- || echo "Warning: tools/arch/x86/include/uapi/asm/perf_regs.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/x86/include/uapi/asm/kvm.h && ( \
- (diff -B ../arch/x86/include/uapi/asm/kvm.h ../../arch/x86/include/uapi/asm/kvm.h >/dev/null) \
- || echo "Warning: tools/arch/x86/include/uapi/asm/kvm.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/x86/include/uapi/asm/kvm_perf.h && ( \
- (diff -B ../arch/x86/include/uapi/asm/kvm_perf.h ../../arch/x86/include/uapi/asm/kvm_perf.h >/dev/null) \
- || echo "Warning: tools/arch/x86/include/uapi/asm/kvm_perf.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/x86/include/uapi/asm/svm.h && ( \
- (diff -B ../arch/x86/include/uapi/asm/svm.h ../../arch/x86/include/uapi/asm/svm.h >/dev/null) \
- || echo "Warning: tools/arch/x86/include/uapi/asm/svm.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/x86/include/uapi/asm/vmx.h && ( \
- (diff -B ../arch/x86/include/uapi/asm/vmx.h ../../arch/x86/include/uapi/asm/vmx.h >/dev/null) \
- || echo "Warning: tools/arch/x86/include/uapi/asm/vmx.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/powerpc/include/uapi/asm/kvm.h && ( \
- (diff -B ../arch/powerpc/include/uapi/asm/kvm.h ../../arch/powerpc/include/uapi/asm/kvm.h >/dev/null) \
- || echo "Warning: tools/arch/powerpc/include/uapi/asm/kvm.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/s390/include/uapi/asm/kvm.h && ( \
- (diff -B ../arch/s390/include/uapi/asm/kvm.h ../../arch/s390/include/uapi/asm/kvm.h >/dev/null) \
- || echo "Warning: tools/arch/s390/include/uapi/asm/kvm.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/s390/include/uapi/asm/kvm_perf.h && ( \
- (diff -B ../arch/s390/include/uapi/asm/kvm_perf.h ../../arch/s390/include/uapi/asm/kvm_perf.h >/dev/null) \
- || echo "Warning: tools/arch/s390/include/uapi/asm/kvm_perf.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/s390/include/uapi/asm/sie.h && ( \
- (diff -B ../arch/s390/include/uapi/asm/sie.h ../../arch/s390/include/uapi/asm/sie.h >/dev/null) \
- || echo "Warning: tools/arch/s390/include/uapi/asm/sie.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/arm/include/uapi/asm/kvm.h && ( \
- (diff -B ../arch/arm/include/uapi/asm/kvm.h ../../arch/arm/include/uapi/asm/kvm.h >/dev/null) \
- || echo "Warning: tools/arch/arm/include/uapi/asm/kvm.h differs from kernel" >&2 )) || true
- @(test -f ../../arch/arm64/include/uapi/asm/kvm.h && ( \
- (diff -B ../arch/arm64/include/uapi/asm/kvm.h ../../arch/arm64/include/uapi/asm/kvm.h >/dev/null) \
- || echo "Warning: tools/arch/arm64/include/uapi/asm/kvm.h differs from kernel" >&2 )) || true
- @(test -f ../../include/asm-generic/bitops/arch_hweight.h && ( \
- (diff -B ../include/asm-generic/bitops/arch_hweight.h ../../include/asm-generic/bitops/arch_hweight.h >/dev/null) \
- || echo "Warning: tools/include/asm-generic/bitops/arch_hweight.h differs from kernel" >&2 )) || true
- @(test -f ../../include/asm-generic/bitops/const_hweight.h && ( \
- (diff -B ../include/asm-generic/bitops/const_hweight.h ../../include/asm-generic/bitops/const_hweight.h >/dev/null) \
- || echo "Warning: tools/include/asm-generic/bitops/const_hweight.h differs from kernel" >&2 )) || true
- @(test -f ../../include/asm-generic/bitops/__fls.h && ( \
- (diff -B ../include/asm-generic/bitops/__fls.h ../../include/asm-generic/bitops/__fls.h >/dev/null) \
- || echo "Warning: tools/include/asm-generic/bitops/__fls.h differs from kernel" >&2 )) || true
- @(test -f ../../include/asm-generic/bitops/fls.h && ( \
- (diff -B ../include/asm-generic/bitops/fls.h ../../include/asm-generic/bitops/fls.h >/dev/null) \
- || echo "Warning: tools/include/asm-generic/bitops/fls.h differs from kernel" >&2 )) || true
- @(test -f ../../include/asm-generic/bitops/fls64.h && ( \
- (diff -B ../include/asm-generic/bitops/fls64.h ../../include/asm-generic/bitops/fls64.h >/dev/null) \
- || echo "Warning: tools/include/asm-generic/bitops/fls64.h differs from kernel" >&2 )) || true
- @(test -f ../../include/linux/coresight-pmu.h && ( \
- (diff -B ../include/linux/coresight-pmu.h ../../include/linux/coresight-pmu.h >/dev/null) \
- || echo "Warning: tools/include/linux/coresight-pmu.h differs from kernel" >&2 )) || true
- @(test -f ../../include/uapi/asm-generic/mman-common.h && ( \
- (diff -B ../include/uapi/asm-generic/mman-common.h ../../include/uapi/asm-generic/mman-common.h >/dev/null) \
- || echo "Warning: tools/include/uapi/asm-generic/mman-common.h differs from kernel" >&2 )) || true
- @(test -f ../../include/uapi/asm-generic/mman.h && ( \
- (diff -B -I "^#include <\(uapi/\)*asm-generic/mman-common.h>$$" ../include/uapi/asm-generic/mman.h ../../include/uapi/asm-generic/mman.h >/dev/null) \
- || echo "Warning: tools/include/uapi/asm-generic/mman.h differs from kernel" >&2 )) || true
- @(test -f ../../include/uapi/linux/mman.h && ( \
- (diff -B -I "^#include <\(uapi/\)*asm/mman.h>$$" ../include/uapi/linux/mman.h ../../include/uapi/linux/mman.h >/dev/null) \
- || echo "Warning: tools/include/uapi/linux/mman.h differs from kernel" >&2 )) || true
$(Q)$(MAKE) $(build)=perf
$(JEVENTS_IN): FORCE
@@ -470,7 +403,7 @@
$(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) $(LIBTRACEEVENT_DYNAMIC_LIST_LDFLAGS) \
$(PERF_IN) $(PMU_EVENTS_IN) $(LIBS) -o $@
-$(GTK_IN): fixdep FORCE
+$(GTK_IN): FORCE
$(Q)$(MAKE) $(build)=gtk
$(OUTPUT)libperf-gtk.so: $(GTK_IN) $(PERFLIBS)
@@ -515,7 +448,7 @@
__build-dir = $(subst $(OUTPUT),,$(dir $@))
build-dir = $(if $(__build-dir),$(__build-dir),.)
-prepare: $(OUTPUT)PERF-VERSION-FILE $(OUTPUT)common-cmds.h fixdep archheaders
+prepare: $(OUTPUT)PERF-VERSION-FILE $(OUTPUT)common-cmds.h archheaders
$(OUTPUT)%.o: %.c prepare FORCE
$(Q)$(MAKE) -f $(srctree)/tools/build/Makefile.build dir=$(build-dir) $@
@@ -555,7 +488,7 @@
LIBPERF_IN := $(OUTPUT)libperf-in.o
-$(LIBPERF_IN): prepare fixdep FORCE
+$(LIBPERF_IN): prepare FORCE
$(Q)$(MAKE) $(build)=libperf
$(LIB_FILE): $(LIBPERF_IN)
@@ -563,10 +496,10 @@
LIBTRACEEVENT_FLAGS += plugin_dir=$(plugindir_SQ)
-$(LIBTRACEEVENT): fixdep FORCE
+$(LIBTRACEEVENT): FORCE
$(Q)$(MAKE) -C $(TRACE_EVENT_DIR) $(LIBTRACEEVENT_FLAGS) O=$(OUTPUT) $(OUTPUT)libtraceevent.a
-libtraceevent_plugins: fixdep FORCE
+libtraceevent_plugins: FORCE
$(Q)$(MAKE) -C $(TRACE_EVENT_DIR) $(LIBTRACEEVENT_FLAGS) O=$(OUTPUT) plugins
$(LIBTRACEEVENT_DYNAMIC_LIST): libtraceevent_plugins
@@ -579,21 +512,21 @@
install-traceevent-plugins: libtraceevent_plugins
$(Q)$(MAKE) -C $(TRACE_EVENT_DIR) $(LIBTRACEEVENT_FLAGS) O=$(OUTPUT) install_plugins
-$(LIBAPI): fixdep FORCE
+$(LIBAPI): FORCE
$(Q)$(MAKE) -C $(LIB_DIR) O=$(OUTPUT) $(OUTPUT)libapi.a
$(LIBAPI)-clean:
$(call QUIET_CLEAN, libapi)
$(Q)$(MAKE) -C $(LIB_DIR) O=$(OUTPUT) clean >/dev/null
-$(LIBBPF): fixdep FORCE
+$(LIBBPF): FORCE
$(Q)$(MAKE) -C $(BPF_DIR) O=$(OUTPUT) $(OUTPUT)libbpf.a FEATURES_DUMP=$(FEATURE_DUMP_EXPORT)
$(LIBBPF)-clean:
$(call QUIET_CLEAN, libbpf)
$(Q)$(MAKE) -C $(BPF_DIR) O=$(OUTPUT) clean >/dev/null
-$(LIBSUBCMD): fixdep FORCE
+$(LIBSUBCMD): FORCE
$(Q)$(MAKE) -C $(SUBCMD_DIR) O=$(OUTPUT) $(OUTPUT)libsubcmd.a
$(LIBSUBCMD)-clean:
@@ -790,3 +723,4 @@
.PHONY: $(GIT-HEAD-PHONY) TAGS tags cscope FORCE prepare
.PHONY: libtraceevent_plugins archheaders
+endif # force_fixdep
diff --git a/tools/perf/arch/powerpc/util/skip-callchain-idx.c b/tools/perf/arch/powerpc/util/skip-callchain-idx.c
index 0c370f8..9a53f6e 100644
--- a/tools/perf/arch/powerpc/util/skip-callchain-idx.c
+++ b/tools/perf/arch/powerpc/util/skip-callchain-idx.c
@@ -58,9 +58,13 @@ static int check_return_reg(int ra_regno, Dwarf_Frame *frame)
}
/*
- * Check if return address is on the stack.
+ * Check if return address is on the stack. If return address
+ * is in a register (typically R0), it is yet to be saved on
+ * the stack.
*/
- if (nops != 0 || ops != NULL)
+ if ((nops != 0 || ops != NULL) &&
+ !(nops == 1 && ops[0].atom == DW_OP_regx &&
+ ops[0].number2 == 0 && ops[0].offset == 0))
return 0;
/*
@@ -243,10 +247,10 @@ int arch_skip_callchain_idx(struct thread *thread, struct ip_callchain *chain)
u64 ip;
u64 skip_slot = -1;
- if (chain->nr < 3)
+ if (!chain || chain->nr < 3)
return skip_slot;
- ip = chain->ips[2];
+ ip = chain->ips[1];
thread__find_addr_location(thread, PERF_RECORD_MISC_USER,
MAP__FUNCTION, ip, &al);
diff --git a/tools/perf/arch/powerpc/util/sym-handling.c b/tools/perf/arch/powerpc/util/sym-handling.c
index 1030a6e..01a288c 100644
--- a/tools/perf/arch/powerpc/util/sym-handling.c
+++ b/tools/perf/arch/powerpc/util/sym-handling.c
@@ -21,15 +21,16 @@ bool elf__needs_adjust_symbols(GElf_Ehdr ehdr)
#endif
-#if !defined(_CALL_ELF) || _CALL_ELF != 2
int arch__choose_best_symbol(struct symbol *syma,
struct symbol *symb __maybe_unused)
{
char *sym = syma->name;
+#if !defined(_CALL_ELF) || _CALL_ELF != 2
/* Skip over any initial dot */
if (*sym == '.')
sym++;
+#endif
/* Avoid "SyS" kernel syscall aliases */
if (strlen(sym) >= 3 && !strncmp(sym, "SyS", 3))
@@ -40,6 +41,7 @@ int arch__choose_best_symbol(struct symbol *syma,
return SYMBOL_A;
}
+#if !defined(_CALL_ELF) || _CALL_ELF != 2
/* Allow matching against dot variants */
int arch__compare_symbol_names(const char *namea, const char *nameb)
{
@@ -115,8 +117,10 @@ void arch__post_process_probe_trace_events(struct perf_probe_event *pev,
for (i = 0; i < ntevs; i++) {
tev = &pev->tevs[i];
map__for_each_symbol(map, sym, tmp) {
- if (map->unmap_ip(map, sym->start) == tev->point.address)
+ if (map->unmap_ip(map, sym->start) == tev->point.address) {
arch__fix_tev_from_maps(pev, tev, map, sym);
+ break;
+ }
}
}
}
diff --git a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
index 555263e..e93ef0b 100644
--- a/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
+++ b/tools/perf/arch/x86/entry/syscalls/syscall_64.tbl
@@ -335,6 +335,9 @@
326 common copy_file_range sys_copy_file_range
327 64 preadv2 sys_preadv2
328 64 pwritev2 sys_pwritev2
+329 common pkey_mprotect sys_pkey_mprotect
+330 common pkey_alloc sys_pkey_alloc
+331 common pkey_free sys_pkey_free
#
# x32-specific system call numbers start at 512 to avoid cache impact
@@ -374,5 +377,5 @@
543 x32 io_setup compat_sys_io_setup
544 x32 io_submit compat_sys_io_submit
545 x32 execveat compat_sys_execveat/ptregs
-534 x32 preadv2 compat_sys_preadv2
-535 x32 pwritev2 compat_sys_pwritev2
+546 x32 preadv2 compat_sys_preadv64v2
+547 x32 pwritev2 compat_sys_pwritev64v2
diff --git a/tools/perf/arch/x86/util/header.c b/tools/perf/arch/x86/util/header.c
index a74a48d..2eb1154 100644
--- a/tools/perf/arch/x86/util/header.c
+++ b/tools/perf/arch/x86/util/header.c
@@ -69,7 +69,7 @@ get_cpuid_str(void)
{
char *buf = malloc(128);
- if (__get_cpuid(buf, 128, "%s-%u-%X$") < 0) {
+ if (buf && __get_cpuid(buf, 128, "%s-%u-%X$") < 0) {
free(buf);
return NULL;
}
diff --git a/tools/perf/bench/numa.c b/tools/perf/bench/numa.c
index 23cce5e..ee9565a 100644
--- a/tools/perf/bench/numa.c
+++ b/tools/perf/bench/numa.c
@@ -1093,7 +1093,7 @@ static void *worker_thread(void *__tdata)
u8 *global_data;
u8 *process_data;
u8 *thread_data;
- u64 bytes_done;
+ u64 bytes_done, secs;
long work_done;
u32 l;
struct rusage rusage;
@@ -1249,7 +1249,8 @@ static void *worker_thread(void *__tdata)
timersub(&stop, &start0, &diff);
td->runtime_ns = diff.tv_sec * NSEC_PER_SEC;
td->runtime_ns += diff.tv_usec * NSEC_PER_USEC;
- td->speed_gbs = bytes_done / (td->runtime_ns / NSEC_PER_SEC) / 1e9;
+ secs = td->runtime_ns / NSEC_PER_SEC;
+ td->speed_gbs = secs ? bytes_done / secs / 1e9 : 0;
getrusage(RUSAGE_THREAD, &rusage);
td->system_time_ns = rusage.ru_stime.tv_sec * NSEC_PER_SEC;
diff --git a/tools/perf/builtin-stat.c b/tools/perf/builtin-stat.c
index 68861e8..43d5f35 100644
--- a/tools/perf/builtin-stat.c
+++ b/tools/perf/builtin-stat.c
@@ -2042,11 +2042,16 @@ static int add_default_attributes(void)
return 0;
if (transaction_run) {
+ struct parse_events_error errinfo;
+
if (pmu_have_event("cpu", "cycles-ct") &&
pmu_have_event("cpu", "el-start"))
- err = parse_events(evsel_list, transaction_attrs, NULL);
+ err = parse_events(evsel_list, transaction_attrs,
+ &errinfo);
else
- err = parse_events(evsel_list, transaction_limited_attrs, NULL);
+ err = parse_events(evsel_list,
+ transaction_limited_attrs,
+ &errinfo);
if (err) {
fprintf(stderr, "Cannot set up transaction events\n");
return -1;
diff --git a/tools/perf/builtin-top.c b/tools/perf/builtin-top.c
index c61e012..e68c866 100644
--- a/tools/perf/builtin-top.c
+++ b/tools/perf/builtin-top.c
@@ -1061,8 +1061,10 @@ parse_callchain_opt(const struct option *opt, const char *arg, int unset)
static int perf_top_config(const char *var, const char *value, void *cb __maybe_unused)
{
- if (!strcmp(var, "top.call-graph"))
- var = "call-graph.record-mode"; /* fall-through */
+ if (!strcmp(var, "top.call-graph")) {
+ var = "call-graph.record-mode";
+ return perf_default_config(var, value, cb);
+ }
if (!strcmp(var, "top.children")) {
symbol_conf.cumulate_callchain = perf_config_bool(var, value);
return 0;
diff --git a/tools/perf/check-headers.sh b/tools/perf/check-headers.sh
new file mode 100755
index 0000000..83fe220
--- /dev/null
+++ b/tools/perf/check-headers.sh
@@ -0,0 +1,61 @@
+#!/bin/sh
+
+HEADERS='
+include/uapi/linux/fcntl.h
+include/uapi/linux/perf_event.h
+include/uapi/linux/stat.h
+include/linux/hash.h
+include/uapi/linux/hw_breakpoint.h
+arch/x86/include/asm/disabled-features.h
+arch/x86/include/asm/required-features.h
+arch/x86/include/asm/cpufeatures.h
+arch/arm/include/uapi/asm/perf_regs.h
+arch/arm64/include/uapi/asm/perf_regs.h
+arch/powerpc/include/uapi/asm/perf_regs.h
+arch/x86/include/uapi/asm/perf_regs.h
+arch/x86/include/uapi/asm/kvm.h
+arch/x86/include/uapi/asm/kvm_perf.h
+arch/x86/include/uapi/asm/svm.h
+arch/x86/include/uapi/asm/vmx.h
+arch/powerpc/include/uapi/asm/kvm.h
+arch/s390/include/uapi/asm/kvm.h
+arch/s390/include/uapi/asm/kvm_perf.h
+arch/s390/include/uapi/asm/sie.h
+arch/arm/include/uapi/asm/kvm.h
+arch/arm64/include/uapi/asm/kvm.h
+include/asm-generic/bitops/arch_hweight.h
+include/asm-generic/bitops/const_hweight.h
+include/asm-generic/bitops/__fls.h
+include/asm-generic/bitops/fls.h
+include/asm-generic/bitops/fls64.h
+include/linux/coresight-pmu.h
+include/uapi/asm-generic/mman-common.h
+'
+
+check () {
+ file=$1
+ opts=
+
+ shift
+ while [ -n "$*" ]; do
+ opts="$opts \"$1\""
+ shift
+ done
+
+ cmd="diff $opts ../$file ../../$file > /dev/null"
+
+ test -f ../../$file &&
+ eval $cmd || echo "Warning: $file differs from kernel" >&2
+}
+
+
+# simple diff check
+for i in $HEADERS; do
+ check $i -B
+done
+
+# diff with extra ignore lines
+check arch/x86/lib/memcpy_64.S -B -I "^EXPORT_SYMBOL" -I "^#include <asm/export.h>"
+check arch/x86/lib/memset_64.S -B -I "^EXPORT_SYMBOL" -I "^#include <asm/export.h>"
+check include/uapi/asm-generic/mman.h -B -I "^#include <\(uapi/\)*asm-generic/mman-common.h>"
+check include/uapi/linux/mman.h -B -I "^#include <\(uapi/\)*asm/mman.h>"
diff --git a/tools/perf/perf.h b/tools/perf/perf.h
index 9a0236a..8f8d895 100644
--- a/tools/perf/perf.h
+++ b/tools/perf/perf.h
@@ -22,7 +22,9 @@ static inline unsigned long long rdclock(void)
return ts.tv_sec * 1000000000ULL + ts.tv_nsec;
}
+#ifndef MAX_NR_CPUS
#define MAX_NR_CPUS 1024
+#endif
extern const char *input_name;
extern bool perf_host, perf_guest;
diff --git a/tools/perf/scripts/python/export-to-postgresql.py b/tools/perf/scripts/python/export-to-postgresql.py
index 7656ff8..c001d5a 100644
--- a/tools/perf/scripts/python/export-to-postgresql.py
+++ b/tools/perf/scripts/python/export-to-postgresql.py
@@ -204,14 +204,23 @@
libpq = CDLL("libpq.so.5")
PQconnectdb = libpq.PQconnectdb
PQconnectdb.restype = c_void_p
+PQconnectdb.argtypes = [ c_char_p ]
PQfinish = libpq.PQfinish
+PQfinish.argtypes = [ c_void_p ]
PQstatus = libpq.PQstatus
+PQstatus.restype = c_int
+PQstatus.argtypes = [ c_void_p ]
PQexec = libpq.PQexec
PQexec.restype = c_void_p
+PQexec.argtypes = [ c_void_p, c_char_p ]
PQresultStatus = libpq.PQresultStatus
+PQresultStatus.restype = c_int
+PQresultStatus.argtypes = [ c_void_p ]
PQputCopyData = libpq.PQputCopyData
+PQputCopyData.restype = c_int
PQputCopyData.argtypes = [ c_void_p, c_void_p, c_int ]
PQputCopyEnd = libpq.PQputCopyEnd
+PQputCopyEnd.restype = c_int
PQputCopyEnd.argtypes = [ c_void_p, c_void_p ]
sys.path.append(os.environ['PERF_EXEC_PATH'] + \
diff --git a/tools/perf/tests/builtin-test.c b/tools/perf/tests/builtin-test.c
index 778668a..0323995 100644
--- a/tools/perf/tests/builtin-test.c
+++ b/tools/perf/tests/builtin-test.c
@@ -335,7 +335,7 @@ static int test_and_print(struct test *t, bool force_skip, int subtest)
if (!t->subtest.get_nr)
pr_debug("%s:", t->desc);
else
- pr_debug("%s subtest %d:", t->desc, subtest);
+ pr_debug("%s subtest %d:", t->desc, subtest + 1);
switch (err) {
case TEST_OK:
diff --git a/tools/perf/tests/task-exit.c b/tools/perf/tests/task-exit.c
index 01a5ba2..b0d005d 100644
--- a/tools/perf/tests/task-exit.c
+++ b/tools/perf/tests/task-exit.c
@@ -82,7 +82,7 @@ int test__task_exit(int subtest __maybe_unused)
evsel = perf_evlist__first(evlist);
evsel->attr.task = 1;
- evsel->attr.sample_freq = 0;
+ evsel->attr.sample_freq = 1;
evsel->attr.inherit = 0;
evsel->attr.watermark = 0;
evsel->attr.wakeup_events = 1;
diff --git a/tools/perf/tests/topology.c b/tools/perf/tests/topology.c
index 98fe69a..3e7cdef 100644
--- a/tools/perf/tests/topology.c
+++ b/tools/perf/tests/topology.c
@@ -42,6 +42,7 @@ static int session_write_header(char *path)
perf_header__set_feat(&session->header, HEADER_CPU_TOPOLOGY);
perf_header__set_feat(&session->header, HEADER_NRCPUS);
+ perf_header__set_feat(&session->header, HEADER_ARCH);
session->header.data_size += DATA_SIZE;
diff --git a/tools/perf/tests/vmlinux-kallsyms.c b/tools/perf/tests/vmlinux-kallsyms.c
index a508233..2aabf0a 100644
--- a/tools/perf/tests/vmlinux-kallsyms.c
+++ b/tools/perf/tests/vmlinux-kallsyms.c
@@ -123,7 +123,7 @@ int test__vmlinux_matches_kallsyms(int subtest __maybe_unused)
if (pair && UM(pair->start) == mem_start) {
next_pair:
- if (strcmp(sym->name, pair->name) == 0) {
+ if (arch__compare_symbol_names(sym->name, pair->name) == 0) {
/*
* kallsyms don't have the symbol end, so we
* set that by using the next symbol start - 1,
diff --git a/tools/perf/util/annotate.c b/tools/perf/util/annotate.c
index a38227e..3336cbc 100644
--- a/tools/perf/util/annotate.c
+++ b/tools/perf/util/annotate.c
@@ -495,9 +495,19 @@ static struct ins *ins__find(const char *name)
int symbol__alloc_hist(struct symbol *sym)
{
struct annotation *notes = symbol__annotation(sym);
- const size_t size = symbol__size(sym);
+ size_t size = symbol__size(sym);
size_t sizeof_sym_hist;
+ /*
+ * Add buffer of one element for zero length symbol.
+ * When sample is taken from first instruction of
+ * zero length symbol, perf still resolves it and
+ * shows symbol name in perf report and allows to
+ * annotate it.
+ */
+ if (size == 0)
+ size = 1;
+
/* Check for overflow when calculating sizeof_sym_hist */
if (size > (SIZE_MAX - sizeof(struct sym_hist)) / sizeof(u64))
return -1;
diff --git a/tools/perf/util/auxtrace.c b/tools/perf/util/auxtrace.c
index 78bd632..29d015e 100644
--- a/tools/perf/util/auxtrace.c
+++ b/tools/perf/util/auxtrace.c
@@ -195,6 +195,9 @@ static int auxtrace_queues__grow(struct auxtrace_queues *queues,
for (i = 0; i < queues->nr_queues; i++) {
list_splice_tail(&queues->queue_array[i].head,
&queue_array[i].head);
+ queue_array[i].tid = queues->queue_array[i].tid;
+ queue_array[i].cpu = queues->queue_array[i].cpu;
+ queue_array[i].set = queues->queue_array[i].set;
queue_array[i].priv = queues->queue_array[i].priv;
}
diff --git a/tools/perf/util/dso.c b/tools/perf/util/dso.c
index 4bc5882..8bec053 100644
--- a/tools/perf/util/dso.c
+++ b/tools/perf/util/dso.c
@@ -253,6 +253,8 @@ int __kmod_path__parse(struct kmod_path *m, const char *path,
if ((strncmp(name, "[kernel.kallsyms]", 17) == 0) ||
(strncmp(name, "[guest.kernel.kallsyms", 22) == 0) ||
(strncmp(name, "[vdso]", 6) == 0) ||
+ (strncmp(name, "[vdso32]", 8) == 0) ||
+ (strncmp(name, "[vdsox32]", 9) == 0) ||
(strncmp(name, "[vsyscall]", 10) == 0)) {
m->kmod = false;
@@ -366,23 +368,7 @@ static int __open_dso(struct dso *dso, struct machine *machine)
if (!is_regular_file(name))
return -EINVAL;
- if (dso__needs_decompress(dso)) {
- char newpath[KMOD_DECOMP_LEN];
- size_t len = sizeof(newpath);
-
- if (dso__decompress_kmodule_path(dso, name, newpath, len) < 0) {
- free(name);
- return -dso->load_errno;
- }
-
- strcpy(name, newpath);
- }
-
fd = do_open(name);
-
- if (dso__needs_decompress(dso))
- unlink(name);
-
free(name);
return fd;
}
diff --git a/tools/perf/util/event.c b/tools/perf/util/event.c
index 6631923..2d5744d 100644
--- a/tools/perf/util/event.c
+++ b/tools/perf/util/event.c
@@ -839,6 +839,7 @@ void *cpu_map_data__alloc(struct cpu_map *map, size_t *size, u16 *type, int *max
}
*size += sizeof(struct cpu_map_data);
+ *size = PERF_ALIGN(*size, sizeof(u64));
return zalloc(*size);
}
diff --git a/tools/perf/util/evsel.c b/tools/perf/util/evsel.c
index bce80f8..f7128c2 100644
--- a/tools/perf/util/evsel.c
+++ b/tools/perf/util/evsel.c
@@ -241,8 +241,9 @@ struct perf_evsel *perf_evsel__new_idx(struct perf_event_attr *attr, int idx)
{
struct perf_evsel *evsel = zalloc(perf_evsel__object.size);
- if (evsel != NULL)
- perf_evsel__init(evsel, attr, idx);
+ if (!evsel)
+ return NULL;
+ perf_evsel__init(evsel, attr, idx);
if (perf_evsel__is_bpf_output(evsel)) {
evsel->attr.sample_type |= (PERF_SAMPLE_RAW | PERF_SAMPLE_TIME |
@@ -262,8 +263,20 @@ struct perf_evsel *perf_evsel__new_cycles(void)
struct perf_evsel *evsel;
event_attr_init(&attr);
+ /*
+ * Unnamed union member, not supported as struct member named
+ * initializer in older compilers such as gcc 4.4.7
+ *
+ * Just for probing the precise_ip:
+ */
+ attr.sample_period = 1;
perf_event_attr__set_max_precise_ip(&attr);
+ /*
+ * Now let the usual logic to set up the perf_event_attr defaults
+ * to kick in when we return and before perf_evsel__open() is called.
+ */
+ attr.sample_period = 0;
evsel = perf_evsel__new(&attr);
if (evsel == NULL)
@@ -681,14 +694,14 @@ static void apply_config_terms(struct perf_evsel *evsel,
struct perf_evsel_config_term *term;
struct list_head *config_terms = &evsel->config_terms;
struct perf_event_attr *attr = &evsel->attr;
- struct callchain_param param;
+ /* callgraph default */
+ struct callchain_param param = {
+ .record_mode = callchain_param.record_mode,
+ };
u32 dump_size = 0;
int max_stack = 0;
const char *callgraph_buf = NULL;
- /* callgraph default */
- param.record_mode = callchain_param.record_mode;
-
list_for_each_entry(term, config_terms, list) {
switch (term->type) {
case PERF_EVSEL__CONFIG_TERM_PERIOD:
diff --git a/tools/perf/util/hist.c b/tools/perf/util/hist.c
index 10849a0..ad613ea5 100644
--- a/tools/perf/util/hist.c
+++ b/tools/perf/util/hist.c
@@ -865,7 +865,7 @@ iter_prepare_cumulative_entry(struct hist_entry_iter *iter,
* cumulated only one time to prevent entries more than 100%
* overhead.
*/
- he_cache = malloc(sizeof(*he_cache) * (iter->max_stack + 1));
+ he_cache = malloc(sizeof(*he_cache) * (callchain_cursor.nr + 1));
if (he_cache == NULL)
return -ENOMEM;
@@ -1030,8 +1030,6 @@ int hist_entry_iter__add(struct hist_entry_iter *iter, struct addr_location *al,
if (err)
return err;
- iter->max_stack = max_stack_depth;
-
err = iter->ops->prepare_entry(iter, al);
if (err)
goto out;
diff --git a/tools/perf/util/hist.h b/tools/perf/util/hist.h
index a440a04..159d616 100644
--- a/tools/perf/util/hist.h
+++ b/tools/perf/util/hist.h
@@ -102,7 +102,6 @@ struct hist_entry_iter {
int curr;
bool hide_unresolved;
- int max_stack;
struct perf_evsel *evsel;
struct perf_sample *sample;
diff --git a/tools/perf/util/intel-pt-decoder/intel-pt-decoder.c b/tools/perf/util/intel-pt-decoder/intel-pt-decoder.c
index cac3953..d27715ff 100644
--- a/tools/perf/util/intel-pt-decoder/intel-pt-decoder.c
+++ b/tools/perf/util/intel-pt-decoder/intel-pt-decoder.c
@@ -112,6 +112,7 @@ struct intel_pt_decoder {
bool have_cyc;
bool fixup_last_mtc;
bool have_last_ip;
+ enum intel_pt_param_flags flags;
uint64_t pos;
uint64_t last_ip;
uint64_t ip;
@@ -215,6 +216,8 @@ struct intel_pt_decoder *intel_pt_decoder_new(struct intel_pt_params *params)
decoder->data = params->data;
decoder->return_compression = params->return_compression;
+ decoder->flags = params->flags;
+
decoder->period = params->period;
decoder->period_type = params->period_type;
@@ -1012,6 +1015,15 @@ static int intel_pt_walk_insn(struct intel_pt_decoder *decoder,
return err;
}
+static inline bool intel_pt_fup_with_nlip(struct intel_pt_decoder *decoder,
+ struct intel_pt_insn *intel_pt_insn,
+ uint64_t ip, int err)
+{
+ return decoder->flags & INTEL_PT_FUP_WITH_NLIP && !err &&
+ intel_pt_insn->branch == INTEL_PT_BR_INDIRECT &&
+ ip == decoder->ip + intel_pt_insn->length;
+}
+
static int intel_pt_walk_fup(struct intel_pt_decoder *decoder)
{
struct intel_pt_insn intel_pt_insn;
@@ -1024,7 +1036,8 @@ static int intel_pt_walk_fup(struct intel_pt_decoder *decoder)
err = intel_pt_walk_insn(decoder, &intel_pt_insn, ip);
if (err == INTEL_PT_RETURN)
return 0;
- if (err == -EAGAIN) {
+ if (err == -EAGAIN ||
+ intel_pt_fup_with_nlip(decoder, &intel_pt_insn, ip, err)) {
if (decoder->set_fup_tx_flags) {
decoder->set_fup_tx_flags = false;
decoder->tx_flags = decoder->fup_tx_flags;
@@ -1034,7 +1047,7 @@ static int intel_pt_walk_fup(struct intel_pt_decoder *decoder)
decoder->state.flags = decoder->fup_tx_flags;
return 0;
}
- return err;
+ return -EAGAIN;
}
decoder->set_fup_tx_flags = false;
if (err)
@@ -1298,7 +1311,6 @@ static int intel_pt_overflow(struct intel_pt_decoder *decoder)
{
intel_pt_log("ERROR: Buffer overflow\n");
intel_pt_clear_tx_flags(decoder);
- decoder->have_tma = false;
decoder->cbr = 0;
decoder->timestamp_insn_cnt = 0;
decoder->pkt_state = INTEL_PT_STATE_ERR_RESYNC;
@@ -1517,7 +1529,6 @@ static int intel_pt_walk_fup_tip(struct intel_pt_decoder *decoder)
case INTEL_PT_PSB:
case INTEL_PT_TSC:
case INTEL_PT_TMA:
- case INTEL_PT_CBR:
case INTEL_PT_MODE_TSX:
case INTEL_PT_BAD:
case INTEL_PT_PSBEND:
@@ -1526,6 +1537,10 @@ static int intel_pt_walk_fup_tip(struct intel_pt_decoder *decoder)
decoder->pkt_step = 0;
return -ENOENT;
+ case INTEL_PT_CBR:
+ intel_pt_calc_cbr(decoder);
+ break;
+
case INTEL_PT_OVF:
return intel_pt_overflow(decoder);
diff --git a/tools/perf/util/intel-pt-decoder/intel-pt-decoder.h b/tools/perf/util/intel-pt-decoder/intel-pt-decoder.h
index 9ae4df1..2fe8f4c 100644
--- a/tools/perf/util/intel-pt-decoder/intel-pt-decoder.h
+++ b/tools/perf/util/intel-pt-decoder/intel-pt-decoder.h
@@ -53,6 +53,14 @@ enum {
INTEL_PT_ERR_MAX,
};
+enum intel_pt_param_flags {
+ /*
+ * FUP packet can contain next linear instruction pointer instead of
+ * current linear instruction pointer.
+ */
+ INTEL_PT_FUP_WITH_NLIP = 1 << 0,
+};
+
struct intel_pt_state {
enum intel_pt_sample_type type;
int err;
@@ -92,6 +100,7 @@ struct intel_pt_params {
unsigned int mtc_period;
uint32_t tsc_ctc_ratio_n;
uint32_t tsc_ctc_ratio_d;
+ enum intel_pt_param_flags flags;
};
struct intel_pt_decoder;
diff --git a/tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.c b/tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.c
index 7528ae4..e5c6caf 100644
--- a/tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.c
+++ b/tools/perf/util/intel-pt-decoder/intel-pt-pkt-decoder.c
@@ -281,7 +281,7 @@ static int intel_pt_get_cyc(unsigned int byte, const unsigned char *buf,
if (len < offs)
return INTEL_PT_NEED_MORE_BYTES;
byte = buf[offs++];
- payload |= (byte >> 1) << shift;
+ payload |= ((uint64_t)byte >> 1) << shift;
}
packet->type = INTEL_PT_CYC;
diff --git a/tools/perf/util/intel-pt.c b/tools/perf/util/intel-pt.c
index b1161d7..d40ab4c 100644
--- a/tools/perf/util/intel-pt.c
+++ b/tools/perf/util/intel-pt.c
@@ -752,6 +752,7 @@ static struct intel_pt_queue *intel_pt_alloc_queue(struct intel_pt *pt,
unsigned int queue_nr)
{
struct intel_pt_params params = { .get_trace = 0, };
+ struct perf_env *env = pt->machine->env;
struct intel_pt_queue *ptq;
ptq = zalloc(sizeof(struct intel_pt_queue));
@@ -832,6 +833,9 @@ static struct intel_pt_queue *intel_pt_alloc_queue(struct intel_pt *pt,
}
}
+ if (env->cpuid && !strncmp(env->cpuid, "GenuineIntel,6,92,", 18))
+ params.flags |= INTEL_PT_FUP_WITH_NLIP;
+
ptq->decoder = intel_pt_decoder_new(¶ms);
if (!ptq->decoder)
goto out_free;
@@ -1344,6 +1348,7 @@ static int intel_pt_sample(struct intel_pt_queue *ptq)
if (intel_pt_is_switch_ip(ptq, state->to_ip)) {
switch (ptq->switch_state) {
+ case INTEL_PT_SS_NOT_TRACING:
case INTEL_PT_SS_UNKNOWN:
case INTEL_PT_SS_EXPECTING_SWITCH_IP:
err = intel_pt_next_tid(pt, ptq);
diff --git a/tools/perf/util/llvm-utils.c b/tools/perf/util/llvm-utils.c
index bf7216b..621f652 100644
--- a/tools/perf/util/llvm-utils.c
+++ b/tools/perf/util/llvm-utils.c
@@ -260,16 +260,16 @@ static const char *kinc_fetch_script =
"#!/usr/bin/env sh\n"
"if ! test -d \"$KBUILD_DIR\"\n"
"then\n"
-" exit -1\n"
+" exit 1\n"
"fi\n"
"if ! test -f \"$KBUILD_DIR/include/generated/autoconf.h\"\n"
"then\n"
-" exit -1\n"
+" exit 1\n"
"fi\n"
"TMPDIR=`mktemp -d`\n"
"if test -z \"$TMPDIR\"\n"
"then\n"
-" exit -1\n"
+" exit 1\n"
"fi\n"
"cat << EOF > $TMPDIR/Makefile\n"
"obj-y := dummy.o\n"
diff --git a/tools/perf/util/parse-events.y b/tools/perf/util/parse-events.y
index 879115f9..98a4205a 100644
--- a/tools/perf/util/parse-events.y
+++ b/tools/perf/util/parse-events.y
@@ -68,6 +68,7 @@
%type <num> value_sym
%type <head> event_config
%type <head> opt_event_config
+%type <head> opt_pmu_config
%type <term> event_term
%type <head> event_pmu
%type <head> event_legacy_symbol
@@ -219,7 +220,7 @@
event_bpf_file
event_pmu:
-PE_NAME opt_event_config
+PE_NAME opt_pmu_config
{
struct parse_events_evlist *data = _data;
struct list_head *list;
@@ -482,6 +483,17 @@
$$ = NULL;
}
+opt_pmu_config:
+'/' event_config '/'
+{
+ $$ = $2;
+}
+|
+'/' '/'
+{
+ $$ = NULL;
+}
+
start_terms: event_config
{
struct parse_events_terms *data = _data;
diff --git a/tools/perf/util/pmu.c b/tools/perf/util/pmu.c
index b1474dc..c86c1d5 100644
--- a/tools/perf/util/pmu.c
+++ b/tools/perf/util/pmu.c
@@ -103,7 +103,7 @@ static int perf_pmu__parse_scale(struct perf_pmu_alias *alias, char *dir, char *
char path[PATH_MAX];
char *lc;
- snprintf(path, PATH_MAX, "%s/%s.scale", dir, name);
+ scnprintf(path, PATH_MAX, "%s/%s.scale", dir, name);
fd = open(path, O_RDONLY);
if (fd == -1)
@@ -163,7 +163,7 @@ static int perf_pmu__parse_unit(struct perf_pmu_alias *alias, char *dir, char *n
ssize_t sret;
int fd;
- snprintf(path, PATH_MAX, "%s/%s.unit", dir, name);
+ scnprintf(path, PATH_MAX, "%s/%s.unit", dir, name);
fd = open(path, O_RDONLY);
if (fd == -1)
@@ -193,7 +193,7 @@ perf_pmu__parse_per_pkg(struct perf_pmu_alias *alias, char *dir, char *name)
char path[PATH_MAX];
int fd;
- snprintf(path, PATH_MAX, "%s/%s.per-pkg", dir, name);
+ scnprintf(path, PATH_MAX, "%s/%s.per-pkg", dir, name);
fd = open(path, O_RDONLY);
if (fd == -1)
@@ -211,7 +211,7 @@ static int perf_pmu__parse_snapshot(struct perf_pmu_alias *alias,
char path[PATH_MAX];
int fd;
- snprintf(path, PATH_MAX, "%s/%s.snapshot", dir, name);
+ scnprintf(path, PATH_MAX, "%s/%s.snapshot", dir, name);
fd = open(path, O_RDONLY);
if (fd == -1)
@@ -685,13 +685,14 @@ static void pmu_format_value(unsigned long *format, __u64 value, __u64 *v,
static __u64 pmu_format_max_value(const unsigned long *format)
{
- __u64 w = 0;
- int fbit;
+ int w;
- for_each_set_bit(fbit, format, PERF_PMU_FORMAT_BITS)
- w |= (1ULL << fbit);
-
- return w;
+ w = bitmap_weight(format, PERF_PMU_FORMAT_BITS);
+ if (!w)
+ return 0;
+ if (w < 64)
+ return (1ULL << w) - 1;
+ return -1;
}
/*
diff --git a/tools/perf/util/probe-event.c b/tools/perf/util/probe-event.c
index c93daccec..a7452fd 100644
--- a/tools/perf/util/probe-event.c
+++ b/tools/perf/util/probe-event.c
@@ -615,7 +615,7 @@ static int post_process_probe_trace_point(struct probe_trace_point *tp,
struct map *map, unsigned long offs)
{
struct symbol *sym;
- u64 addr = tp->address + tp->offset - offs;
+ u64 addr = tp->address - offs;
sym = map__find_symbol(map, addr);
if (!sym)
diff --git a/tools/perf/util/strbuf.c b/tools/perf/util/strbuf.c
index 81759390..842cf3f 100644
--- a/tools/perf/util/strbuf.c
+++ b/tools/perf/util/strbuf.c
@@ -105,19 +105,25 @@ static int strbuf_addv(struct strbuf *sb, const char *fmt, va_list ap)
va_copy(ap_saved, ap);
len = vsnprintf(sb->buf + sb->len, sb->alloc - sb->len, fmt, ap);
- if (len < 0)
+ if (len < 0) {
+ va_end(ap_saved);
return len;
+ }
if (len > strbuf_avail(sb)) {
ret = strbuf_grow(sb, len);
- if (ret)
+ if (ret) {
+ va_end(ap_saved);
return ret;
+ }
len = vsnprintf(sb->buf + sb->len, sb->alloc - sb->len, fmt, ap_saved);
va_end(ap_saved);
if (len > strbuf_avail(sb)) {
pr_debug("this should not happen, your vsnprintf is broken");
+ va_end(ap_saved);
return -EINVAL;
}
}
+ va_end(ap_saved);
return strbuf_setlen(sb, sb->len + len);
}
diff --git a/tools/perf/util/trace-event-info.c b/tools/perf/util/trace-event-info.c
index d995743..58ce620 100644
--- a/tools/perf/util/trace-event-info.c
+++ b/tools/perf/util/trace-event-info.c
@@ -507,12 +507,14 @@ struct tracing_data *tracing_data_get(struct list_head *pattrs,
"/tmp/perf-XXXXXX");
if (!mkstemp(tdata->temp_file)) {
pr_debug("Can't make temp file");
+ free(tdata);
return NULL;
}
temp_fd = open(tdata->temp_file, O_RDWR);
if (temp_fd < 0) {
pr_debug("Can't read '%s'", tdata->temp_file);
+ free(tdata);
return NULL;
}
diff --git a/tools/perf/util/trace-event-read.c b/tools/perf/util/trace-event-read.c
index b67a0cc..23baee7 100644
--- a/tools/perf/util/trace-event-read.c
+++ b/tools/perf/util/trace-event-read.c
@@ -334,9 +334,12 @@ static int read_event_files(struct pevent *pevent)
for (x=0; x < count; x++) {
size = read8(pevent);
ret = read_event_file(pevent, sys, size);
- if (ret)
+ if (ret) {
+ free(sys);
return ret;
+ }
}
+ free(sys);
}
return 0;
}
diff --git a/tools/perf/util/util.h b/tools/perf/util/util.h
index 43899e0..e72d370 100644
--- a/tools/perf/util/util.h
+++ b/tools/perf/util/util.h
@@ -23,8 +23,6 @@
#endif
#endif
-#define ARRAY_SIZE(x) (sizeof(x)/sizeof(x[0]))
-
#ifdef __GNUC__
#define TYPEOF(x) (__typeof__(x))
#else
diff --git a/tools/power/cpupower/bench/Makefile b/tools/power/cpupower/bench/Makefile
index 3e59f1a..8a285bc 100644
--- a/tools/power/cpupower/bench/Makefile
+++ b/tools/power/cpupower/bench/Makefile
@@ -8,7 +8,7 @@
ifeq ($(strip $(STATIC)),true)
LIBS = -L../ -L$(OUTPUT) -lm
OBJS = $(OUTPUT)main.o $(OUTPUT)parse.o $(OUTPUT)system.o $(OUTPUT)benchmark.o \
- $(OUTPUT)../lib/cpufreq.o $(OUTPUT)../lib/sysfs.o
+ $(OUTPUT)../lib/cpufreq.o $(OUTPUT)../lib/cpupower.o
else
LIBS = -L../ -L$(OUTPUT) -lm -lcpupower
OBJS = $(OUTPUT)main.o $(OUTPUT)parse.o $(OUTPUT)system.o $(OUTPUT)benchmark.o
diff --git a/tools/power/cpupower/lib/cpufreq.c b/tools/power/cpupower/lib/cpufreq.c
index 1b993fe..0c0f3e3 100644
--- a/tools/power/cpupower/lib/cpufreq.c
+++ b/tools/power/cpupower/lib/cpufreq.c
@@ -28,7 +28,7 @@ static unsigned int sysfs_cpufreq_read_file(unsigned int cpu, const char *fname,
snprintf(path, sizeof(path), PATH_TO_CPU "cpu%u/cpufreq/%s",
cpu, fname);
- return sysfs_read_file(path, buf, buflen);
+ return cpupower_read_sysfs(path, buf, buflen);
}
/* helper function to write a new value to a /sys file */
diff --git a/tools/power/cpupower/lib/cpuidle.c b/tools/power/cpupower/lib/cpuidle.c
index 9bd4c76..852d254 100644
--- a/tools/power/cpupower/lib/cpuidle.c
+++ b/tools/power/cpupower/lib/cpuidle.c
@@ -319,7 +319,7 @@ static unsigned int sysfs_cpuidle_read_file(const char *fname, char *buf,
snprintf(path, sizeof(path), PATH_TO_CPU "cpuidle/%s", fname);
- return sysfs_read_file(path, buf, buflen);
+ return cpupower_read_sysfs(path, buf, buflen);
}
diff --git a/tools/power/cpupower/lib/cpupower.c b/tools/power/cpupower/lib/cpupower.c
index 9c395ec9..9711d62 100644
--- a/tools/power/cpupower/lib/cpupower.c
+++ b/tools/power/cpupower/lib/cpupower.c
@@ -15,7 +15,7 @@
#include "cpupower.h"
#include "cpupower_intern.h"
-unsigned int sysfs_read_file(const char *path, char *buf, size_t buflen)
+unsigned int cpupower_read_sysfs(const char *path, char *buf, size_t buflen)
{
int fd;
ssize_t numread;
@@ -95,7 +95,7 @@ static int sysfs_topology_read_file(unsigned int cpu, const char *fname, int *re
snprintf(path, sizeof(path), PATH_TO_CPU "cpu%u/topology/%s",
cpu, fname);
- if (sysfs_read_file(path, linebuf, MAX_LINE_LEN) == 0)
+ if (cpupower_read_sysfs(path, linebuf, MAX_LINE_LEN) == 0)
return -1;
*result = strtol(linebuf, &endp, 0);
if (endp == linebuf || errno == ERANGE)
diff --git a/tools/power/cpupower/lib/cpupower_intern.h b/tools/power/cpupower/lib/cpupower_intern.h
index f8ec400..433fa86 100644
--- a/tools/power/cpupower/lib/cpupower_intern.h
+++ b/tools/power/cpupower/lib/cpupower_intern.h
@@ -2,4 +2,4 @@
#define MAX_LINE_LEN 4096
#define SYSFS_PATH_MAX 255
-unsigned int sysfs_read_file(const char *path, char *buf, size_t buflen);
+unsigned int cpupower_read_sysfs(const char *path, char *buf, size_t buflen);
diff --git a/tools/power/x86/turbostat/turbostat.c b/tools/power/x86/turbostat/turbostat.c
index 9664b1f..5ec2de8 100644
--- a/tools/power/x86/turbostat/turbostat.c
+++ b/tools/power/x86/turbostat/turbostat.c
@@ -733,9 +733,7 @@ void format_all_counters(struct thread_data *t, struct core_data *c, struct pkg_
if (!printed || !summary_only)
print_header();
- if (topo.num_cpus > 1)
- format_counters(&average.threads, &average.cores,
- &average.packages);
+ format_counters(&average.threads, &average.cores, &average.packages);
printed = 1;
@@ -3202,7 +3200,9 @@ void process_cpuid()
family = (fms >> 8) & 0xf;
model = (fms >> 4) & 0xf;
stepping = fms & 0xf;
- if (family == 6 || family == 0xf)
+ if (family == 0xf)
+ family += (fms >> 20) & 0xff;
+ if (family >= 6)
model += ((fms >> 16) & 0xf) << 4;
if (debug) {
diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
index 19edc1a..7ea4438 100644
--- a/tools/scripts/Makefile.include
+++ b/tools/scripts/Makefile.include
@@ -92,3 +92,5 @@
QUIET_INSTALL = @printf ' INSTALL %s\n' $1;
endif
endif
+
+pound := \#
diff --git a/tools/testing/nvdimm/test/iomap.c b/tools/testing/nvdimm/test/iomap.c
index 64cae1a..5c7ad24 100644
--- a/tools/testing/nvdimm/test/iomap.c
+++ b/tools/testing/nvdimm/test/iomap.c
@@ -114,7 +114,7 @@ void *__wrap_devm_memremap_pages(struct device *dev, struct resource *res,
return nfit_res->buf + offset - nfit_res->res.start;
return devm_memremap_pages(dev, res, ref, altmap);
}
-EXPORT_SYMBOL(__wrap_devm_memremap_pages);
+EXPORT_SYMBOL_GPL(__wrap_devm_memremap_pages);
pfn_t __wrap_phys_to_pfn_t(phys_addr_t addr, unsigned long flags)
{
diff --git a/tools/testing/selftests/Makefile b/tools/testing/selftests/Makefile
index a899ef81..d37dfc6 100644
--- a/tools/testing/selftests/Makefile
+++ b/tools/testing/selftests/Makefile
@@ -15,6 +15,7 @@
TARGETS += mount
TARGETS += mqueue
TARGETS += net
+TARGETS += netfilter
TARGETS += nsfs
TARGETS += powerpc
TARGETS += pstore
@@ -94,6 +95,7 @@
for TARGET in $(TARGETS); do \
echo "echo ; echo Running tests in $$TARGET" >> $(ALL_SCRIPT); \
echo "echo ========================================" >> $(ALL_SCRIPT); \
+ echo "[ -w /dev/kmsg ] && echo \"kselftest: Running tests in $$TARGET\" >> /dev/kmsg" >> $(ALL_SCRIPT); \
echo "cd $$TARGET" >> $(ALL_SCRIPT); \
make -s --no-print-directory -C $$TARGET emit_tests >> $(ALL_SCRIPT); \
echo "cd \$$ROOT" >> $(ALL_SCRIPT); \
diff --git a/tools/testing/selftests/efivarfs/config b/tools/testing/selftests/efivarfs/config
new file mode 100644
index 0000000..4e151f1
--- /dev/null
+++ b/tools/testing/selftests/efivarfs/config
@@ -0,0 +1 @@
+CONFIG_EFIVAR_FS=y
diff --git a/tools/testing/selftests/firmware/fw_filesystem.sh b/tools/testing/selftests/firmware/fw_filesystem.sh
index 17e16fc..99d7f13 100755
--- a/tools/testing/selftests/firmware/fw_filesystem.sh
+++ b/tools/testing/selftests/firmware/fw_filesystem.sh
@@ -29,9 +29,11 @@
echo "$OLD_TIMEOUT" >/sys/class/firmware/timeout
fi
if [ "$OLD_FWPATH" = "" ]; then
- OLD_FWPATH=" "
+ # A zero-length write won't work; write a null byte
+ printf '\000' >/sys/module/firmware_class/parameters/path
+ else
+ echo -n "$OLD_FWPATH" >/sys/module/firmware_class/parameters/path
fi
- echo -n "$OLD_FWPATH" >/sys/module/firmware_class/parameters/path
rm -f "$FW"
rmdir "$FWPATH"
}
diff --git a/tools/testing/selftests/ftrace/test.d/00basic/snapshot.tc b/tools/testing/selftests/ftrace/test.d/00basic/snapshot.tc
new file mode 100644
index 0000000..3b1f45e
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/00basic/snapshot.tc
@@ -0,0 +1,28 @@
+#!/bin/sh
+# description: Snapshot and tracing setting
+# flags: instance
+
+[ ! -f snapshot ] && exit_unsupported
+
+echo "Set tracing off"
+echo 0 > tracing_on
+
+echo "Allocate and take a snapshot"
+echo 1 > snapshot
+
+# Since trace buffer is empty, snapshot is also empty, but allocated
+grep -q "Snapshot is allocated" snapshot
+
+echo "Ensure keep tracing off"
+test `cat tracing_on` -eq 0
+
+echo "Set tracing on"
+echo 1 > tracing_on
+
+echo "Take a snapshot again"
+echo 1 > snapshot
+
+echo "Ensure keep tracing on"
+test `cat tracing_on` -eq 1
+
+exit 0
diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args_string.tc b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args_string.tc
new file mode 100644
index 0000000..5ba7303
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args_string.tc
@@ -0,0 +1,46 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: Kprobe event string type argument
+
+[ -f kprobe_events ] || exit_unsupported # this is configurable
+
+echo 0 > events/enable
+echo > kprobe_events
+
+case `uname -m` in
+x86_64)
+ ARG2=%si
+ OFFS=8
+;;
+i[3456]86)
+ ARG2=%cx
+ OFFS=4
+;;
+aarch64)
+ ARG2=%x1
+ OFFS=8
+;;
+arm*)
+ ARG2=%r1
+ OFFS=4
+;;
+*)
+ echo "Please implement other architecture here"
+ exit_untested
+esac
+
+: "Test get argument (1)"
+echo "p:testprobe create_trace_kprobe arg1=+0(+0(${ARG2})):string" > kprobe_events
+echo 1 > events/kprobes/testprobe/enable
+! echo test >> kprobe_events
+tail -n 1 trace | grep -qe "testprobe.* arg1=\"test\""
+
+echo 0 > events/kprobes/testprobe/enable
+: "Test get argument (2)"
+echo "p:testprobe create_trace_kprobe arg1=+0(+0(${ARG2})):string arg2=+0(+${OFFS}(${ARG2})):string" > kprobe_events
+echo 1 > events/kprobes/testprobe/enable
+! echo test1 test2 >> kprobe_events
+tail -n 1 trace | grep -qe "testprobe.* arg1=\"test1\" arg2=\"test2\""
+
+echo 0 > events/enable
+echo > kprobe_events
diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args_syntax.tc b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args_syntax.tc
new file mode 100644
index 0000000..231bcd2
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/kprobe/kprobe_args_syntax.tc
@@ -0,0 +1,97 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: Kprobe event argument syntax
+
+[ -f kprobe_events ] || exit_unsupported # this is configurable
+
+grep "x8/16/32/64" README > /dev/null || exit_unsupported # version issue
+
+echo 0 > events/enable
+echo > kprobe_events
+
+PROBEFUNC="vfs_read"
+GOODREG=
+BADREG=
+GOODSYM="_sdata"
+if ! grep -qw ${GOODSYM} /proc/kallsyms ; then
+ GOODSYM=$PROBEFUNC
+fi
+BADSYM="deaqswdefr"
+SYMADDR=0x`grep -w ${GOODSYM} /proc/kallsyms | cut -f 1 -d " "`
+GOODTYPE="x16"
+BADTYPE="y16"
+
+case `uname -m` in
+x86_64|i[3456]86)
+ GOODREG=%ax
+ BADREG=%ex
+;;
+aarch64)
+ GOODREG=%x0
+ BADREG=%ax
+;;
+arm*)
+ GOODREG=%r0
+ BADREG=%ax
+;;
+esac
+
+test_goodarg() # Good-args
+{
+ while [ "$1" ]; do
+ echo "p ${PROBEFUNC} $1" > kprobe_events
+ shift 1
+ done;
+}
+
+test_badarg() # Bad-args
+{
+ while [ "$1" ]; do
+ ! echo "p ${PROBEFUNC} $1" > kprobe_events
+ shift 1
+ done;
+}
+
+echo > kprobe_events
+
+: "Register access"
+test_goodarg ${GOODREG}
+test_badarg ${BADREG}
+
+: "Symbol access"
+test_goodarg "@${GOODSYM}" "@${SYMADDR}" "@${GOODSYM}+10" "@${GOODSYM}-10"
+test_badarg "@" "@${BADSYM}" "@${GOODSYM}*10" "@${GOODSYM}/10" \
+ "@${GOODSYM}%10" "@${GOODSYM}&10" "@${GOODSYM}|10"
+
+: "Stack access"
+test_goodarg "\$stack" "\$stack0" "\$stack1"
+test_badarg "\$stackp" "\$stack0+10" "\$stack1-10"
+
+: "Retval access"
+echo "r ${PROBEFUNC} \$retval" > kprobe_events
+! echo "p ${PROBEFUNC} \$retval" > kprobe_events
+
+: "Comm access"
+test_goodarg "\$comm"
+
+: "Indirect memory access"
+test_goodarg "+0(${GOODREG})" "-0(${GOODREG})" "+10(\$stack)" \
+ "+0(\$stack1)" "+10(@${GOODSYM}-10)" "+0(+10(+20(\$stack)))"
+test_badarg "+(${GOODREG})" "(${GOODREG}+10)" "-(${GOODREG})" "(${GOODREG})" \
+ "+10(\$comm)" "+0(${GOODREG})+10"
+
+: "Name assignment"
+test_goodarg "varname=${GOODREG}"
+test_badarg "varname=varname2=${GOODREG}"
+
+: "Type syntax"
+test_goodarg "${GOODREG}:${GOODTYPE}"
+test_badarg "${GOODREG}::${GOODTYPE}" "${GOODREG}:${BADTYPE}" \
+ "${GOODTYPE}:${GOODREG}"
+
+: "Combination check"
+
+test_goodarg "\$comm:string" "+0(\$stack):string"
+test_badarg "\$comm:x64" "\$stack:string" "${GOODREG}:string"
+
+echo > kprobe_events
diff --git a/tools/testing/selftests/ftrace/test.d/kprobe/probepoint.tc b/tools/testing/selftests/ftrace/test.d/kprobe/probepoint.tc
new file mode 100644
index 0000000..4fda01a
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/kprobe/probepoint.tc
@@ -0,0 +1,43 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: Kprobe events - probe points
+
+[ -f kprobe_events ] || exit_unsupported # this is configurable
+
+TARGET_FUNC=create_trace_kprobe
+
+dec_addr() { # hexaddr
+ printf "%d" "0x"`echo $1 | tail -c 8`
+}
+
+set_offs() { # prev target next
+ A1=`dec_addr $1`
+ A2=`dec_addr $2`
+ A3=`dec_addr $3`
+ TARGET="0x$2" # an address
+ PREV=`expr $A1 - $A2` # offset to previous symbol
+ NEXT=+`expr $A3 - $A2` # offset to next symbol
+ OVERFLOW=+`printf "0x%x" ${PREV}` # overflow offset to previous symbol
+}
+
+# We have to decode symbol addresses to get correct offsets.
+# If the offset is not an instruction boundary, it cause -EILSEQ.
+set_offs `grep -A1 -B1 ${TARGET_FUNC} /proc/kallsyms | cut -f 1 -d " " | xargs`
+
+UINT_TEST=no
+# printf "%x" -1 returns (unsigned long)-1.
+if [ `printf "%x" -1 | wc -c` != 9 ]; then
+ UINT_TEST=yes
+fi
+
+echo 0 > events/enable
+echo > kprobe_events
+echo "p:testprobe ${TARGET_FUNC}" > kprobe_events
+echo "p:testprobe ${TARGET}" > kprobe_events
+echo "p:testprobe ${TARGET_FUNC}${NEXT}" > kprobe_events
+! echo "p:testprobe ${TARGET_FUNC}${PREV}" > kprobe_events
+if [ "${UINT_TEST}" = yes ]; then
+! echo "p:testprobe ${TARGET_FUNC}${OVERFLOW}" > kprobe_events
+fi
+echo > kprobe_events
+clear_trace
diff --git a/tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-syntax.tc b/tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-syntax.tc
new file mode 100644
index 0000000..88e6c3f
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/trigger/inter-event/trigger-synthetic-event-syntax.tc
@@ -0,0 +1,80 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: event trigger - test synthetic_events syntax parser
+
+do_reset() {
+ reset_trigger
+ echo > set_event
+ clear_trace
+}
+
+fail() { #msg
+ do_reset
+ echo $1
+ exit_fail
+}
+
+if [ ! -f set_event ]; then
+ echo "event tracing is not supported"
+ exit_unsupported
+fi
+
+if [ ! -f synthetic_events ]; then
+ echo "synthetic event is not supported"
+ exit_unsupported
+fi
+
+reset_tracer
+do_reset
+
+echo "Test synthetic_events syntax parser"
+
+echo > synthetic_events
+
+# synthetic event must have a field
+! echo "myevent" >> synthetic_events
+echo "myevent u64 var1" >> synthetic_events
+
+# synthetic event must be found in synthetic_events
+grep "myevent[[:space:]]u64 var1" synthetic_events
+
+# it is not possible to add same name event
+! echo "myevent u64 var2" >> synthetic_events
+
+# Non-append open will cleanup all events and add new one
+echo "myevent u64 var2" > synthetic_events
+
+# multiple fields with different spaces
+echo "myevent u64 var1; u64 var2;" > synthetic_events
+grep "myevent[[:space:]]u64 var1; u64 var2" synthetic_events
+echo "myevent u64 var1 ; u64 var2 ;" > synthetic_events
+grep "myevent[[:space:]]u64 var1; u64 var2" synthetic_events
+echo "myevent u64 var1 ;u64 var2" > synthetic_events
+grep "myevent[[:space:]]u64 var1; u64 var2" synthetic_events
+
+# test field types
+echo "myevent u32 var" > synthetic_events
+echo "myevent u16 var" > synthetic_events
+echo "myevent u8 var" > synthetic_events
+echo "myevent s64 var" > synthetic_events
+echo "myevent s32 var" > synthetic_events
+echo "myevent s16 var" > synthetic_events
+echo "myevent s8 var" > synthetic_events
+
+echo "myevent char var" > synthetic_events
+echo "myevent int var" > synthetic_events
+echo "myevent long var" > synthetic_events
+echo "myevent pid_t var" > synthetic_events
+
+echo "myevent unsigned char var" > synthetic_events
+echo "myevent unsigned int var" > synthetic_events
+echo "myevent unsigned long var" > synthetic_events
+grep "myevent[[:space:]]unsigned long var" synthetic_events
+
+# test string type
+echo "myevent char var[10]" > synthetic_events
+grep "myevent[[:space:]]char\[10\] var" synthetic_events
+
+do_reset
+
+exit 0
diff --git a/tools/testing/selftests/intel_pstate/run.sh b/tools/testing/selftests/intel_pstate/run.sh
index 7868c10..b62876f 100755
--- a/tools/testing/selftests/intel_pstate/run.sh
+++ b/tools/testing/selftests/intel_pstate/run.sh
@@ -48,11 +48,12 @@
echo "sleeping for 5 seconds"
sleep 5
- num_freqs=$(cat /proc/cpuinfo | grep MHz | sort -u | wc -l)
- if [ $num_freqs -le 2 ]; then
- cat /proc/cpuinfo | grep MHz | sort -u | tail -1 > /tmp/result.$1
+ grep MHz /proc/cpuinfo | sort -u > /tmp/result.freqs
+ num_freqs=$(wc -l /tmp/result.freqs | awk ' { print $1 } ')
+ if [ $num_freqs -ge 2 ]; then
+ tail -n 1 /tmp/result.freqs > /tmp/result.$1
else
- cat /proc/cpuinfo | grep MHz | sort -u > /tmp/result.$1
+ cp /tmp/result.freqs /tmp/result.$1
fi
./msr 0 >> /tmp/result.$1
@@ -82,21 +83,20 @@
max_freq=$(($_max_freq / 1000))
-for freq in `seq $max_freq -100 $min_freq`
+[ $EVALUATE_ONLY -eq 0 ] && for freq in `seq $max_freq -100 $min_freq`
do
echo "Setting maximum frequency to $freq"
cpupower frequency-set -g powersave --max=${freq}MHz >& /dev/null
- [ $EVALUATE_ONLY -eq 0 ] && run_test $freq
+ run_test $freq
done
-echo "=============================================================================="
+[ $EVALUATE_ONLY -eq 0 ] && cpupower frequency-set -g powersave --max=${max_freq}MHz >& /dev/null
+echo "=============================================================================="
echo "The marketing frequency of the cpu is $mkt_freq MHz"
echo "The maximum frequency of the cpu is $max_freq MHz"
echo "The minimum frequency of the cpu is $min_freq MHz"
-cpupower frequency-set -g powersave --max=${max_freq}MHz >& /dev/null
-
# make a pretty table
echo "Target Actual Difference MSR(0x199) max_perf_pct"
for freq in `seq $max_freq -100 $min_freq`
@@ -104,10 +104,6 @@
result_freq=$(cat /tmp/result.${freq} | grep "cpu MHz" | awk ' { print $4 } ' | awk -F "." ' { print $1 } ')
msr=$(cat /tmp/result.${freq} | grep "msr" | awk ' { print $3 } ')
max_perf_pct=$(cat /tmp/result.${freq} | grep "max_perf_pct" | awk ' { print $2 } ' )
- if [ $result_freq -eq $freq ]; then
- echo " $freq $result_freq 0 $msr $(($max_perf_pct*3300))"
- else
- echo " $freq $result_freq $(($result_freq-$freq)) $msr $(($max_perf_pct*$max_freq))"
- fi
+ echo " $freq $result_freq $(($result_freq-$freq)) $msr $(($max_perf_pct*$max_freq))"
done
exit 0
diff --git a/tools/testing/selftests/memfd/config b/tools/testing/selftests/memfd/config
new file mode 100644
index 0000000..835c7f4d
--- /dev/null
+++ b/tools/testing/selftests/memfd/config
@@ -0,0 +1 @@
+CONFIG_FUSE_FS=m
diff --git a/tools/testing/selftests/memory-hotplug/config b/tools/testing/selftests/memory-hotplug/config
index 2fde301..a7e8cd5 100644
--- a/tools/testing/selftests/memory-hotplug/config
+++ b/tools/testing/selftests/memory-hotplug/config
@@ -2,3 +2,4 @@
CONFIG_MEMORY_HOTPLUG_SPARSE=y
CONFIG_NOTIFIER_ERROR_INJECTION=y
CONFIG_MEMORY_NOTIFIER_ERROR_INJECT=m
+CONFIG_MEMORY_HOTREMOVE=y
diff --git a/tools/testing/selftests/net/psock_fanout.c b/tools/testing/selftests/net/psock_fanout.c
index 4124593..9b654a0 100644
--- a/tools/testing/selftests/net/psock_fanout.c
+++ b/tools/testing/selftests/net/psock_fanout.c
@@ -97,6 +97,8 @@ static int sock_fanout_open(uint16_t typeflags, int num_packets)
static void sock_fanout_set_ebpf(int fd)
{
+ static char log_buf[65536];
+
const int len_off = __builtin_offsetof(struct __sk_buff, len);
struct bpf_insn prog[] = {
{ BPF_ALU64 | BPF_MOV | BPF_X, 6, 1, 0, 0 },
@@ -109,7 +111,6 @@ static void sock_fanout_set_ebpf(int fd)
{ BPF_ALU | BPF_MOV | BPF_K, 0, 0, 0, 0 },
{ BPF_JMP | BPF_EXIT, 0, 0, 0, 0 }
};
- char log_buf[512];
union bpf_attr attr;
int pfd;
diff --git a/tools/testing/selftests/net/reuseport_bpf.c b/tools/testing/selftests/net/reuseport_bpf.c
index 4a82174..b527710 100644
--- a/tools/testing/selftests/net/reuseport_bpf.c
+++ b/tools/testing/selftests/net/reuseport_bpf.c
@@ -21,6 +21,7 @@
#include <sys/epoll.h>
#include <sys/types.h>
#include <sys/socket.h>
+#include <sys/resource.h>
#include <unistd.h>
#ifndef ARRAY_SIZE
@@ -190,11 +191,14 @@ static void send_from(struct test_params p, uint16_t sport, char *buf,
struct sockaddr * const saddr = new_any_sockaddr(p.send_family, sport);
struct sockaddr * const daddr =
new_loopback_sockaddr(p.send_family, p.recv_port);
- const int fd = socket(p.send_family, p.protocol, 0);
+ const int fd = socket(p.send_family, p.protocol, 0), one = 1;
if (fd < 0)
error(1, errno, "failed to create send socket");
+ if (setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)))
+ error(1, errno, "failed to set reuseaddr");
+
if (bind(fd, saddr, sockaddr_size()))
error(1, errno, "failed to bind send socket");
@@ -433,6 +437,26 @@ void enable_fastopen(void)
}
}
+static struct rlimit rlim_old;
+
+static __attribute__((constructor)) void main_ctor(void)
+{
+ getrlimit(RLIMIT_MEMLOCK, &rlim_old);
+
+ if (rlim_old.rlim_cur != RLIM_INFINITY) {
+ struct rlimit rlim_new;
+
+ rlim_new.rlim_cur = rlim_old.rlim_cur + (1UL << 20);
+ rlim_new.rlim_max = rlim_old.rlim_max + (1UL << 20);
+ setrlimit(RLIMIT_MEMLOCK, &rlim_new);
+ }
+}
+
+static __attribute__((destructor)) void main_dtor(void)
+{
+ setrlimit(RLIMIT_MEMLOCK, &rlim_old);
+}
+
int main(void)
{
fprintf(stderr, "---- IPv4 UDP ----\n");
diff --git a/tools/testing/selftests/netfilter/Makefile b/tools/testing/selftests/netfilter/Makefile
new file mode 100644
index 0000000..47ed6ce
--- /dev/null
+++ b/tools/testing/selftests/netfilter/Makefile
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: GPL-2.0
+# Makefile for netfilter selftests
+
+TEST_PROGS := nft_trans_stress.sh
+
+include ../lib.mk
diff --git a/tools/testing/selftests/netfilter/config b/tools/testing/selftests/netfilter/config
new file mode 100644
index 0000000..1017313
--- /dev/null
+++ b/tools/testing/selftests/netfilter/config
@@ -0,0 +1,2 @@
+CONFIG_NET_NS=y
+NF_TABLES_INET=y
diff --git a/tools/testing/selftests/netfilter/nft_trans_stress.sh b/tools/testing/selftests/netfilter/nft_trans_stress.sh
new file mode 100755
index 0000000..f1affd1
--- /dev/null
+++ b/tools/testing/selftests/netfilter/nft_trans_stress.sh
@@ -0,0 +1,78 @@
+#!/bin/bash
+#
+# This test is for stress-testing the nf_tables config plane path vs.
+# packet path processing: Make sure we never release rules that are
+# still visible to other cpus.
+#
+# set -e
+
+# Kselftest framework requirement - SKIP code is 4.
+ksft_skip=4
+
+testns=testns1
+tables="foo bar baz quux"
+
+nft --version > /dev/null 2>&1
+if [ $? -ne 0 ];then
+ echo "SKIP: Could not run test without nft tool"
+ exit $ksft_skip
+fi
+
+ip -Version > /dev/null 2>&1
+if [ $? -ne 0 ];then
+ echo "SKIP: Could not run test without ip tool"
+ exit $ksft_skip
+fi
+
+tmp=$(mktemp)
+
+for table in $tables; do
+ echo add table inet "$table" >> "$tmp"
+ echo flush table inet "$table" >> "$tmp"
+
+ echo "add chain inet $table INPUT { type filter hook input priority 0; }" >> "$tmp"
+ echo "add chain inet $table OUTPUT { type filter hook output priority 0; }" >> "$tmp"
+ for c in $(seq 1 400); do
+ chain=$(printf "chain%03u" "$c")
+ echo "add chain inet $table $chain" >> "$tmp"
+ done
+
+ for c in $(seq 1 400); do
+ chain=$(printf "chain%03u" "$c")
+ for BASE in INPUT OUTPUT; do
+ echo "add rule inet $table $BASE counter jump $chain" >> "$tmp"
+ done
+ echo "add rule inet $table $chain counter return" >> "$tmp"
+ done
+done
+
+ip netns add "$testns"
+ip -netns "$testns" link set lo up
+
+lscpu | grep ^CPU\(s\): | ( read cpu cpunum ;
+cpunum=$((cpunum-1))
+for i in $(seq 0 $cpunum);do
+ mask=$(printf 0x%x $((1<<$i)))
+ ip netns exec "$testns" taskset $mask ping -4 127.0.0.1 -fq > /dev/null &
+ ip netns exec "$testns" taskset $mask ping -6 ::1 -fq > /dev/null &
+done)
+
+sleep 1
+
+for i in $(seq 1 10) ; do ip netns exec "$testns" nft -f "$tmp" & done
+
+for table in $tables;do
+ randsleep=$((RANDOM%10))
+ sleep $randsleep
+ ip netns exec "$testns" nft delete table inet $table 2>/dev/null
+done
+
+randsleep=$((RANDOM%10))
+sleep $randsleep
+
+pkill -9 ping
+
+wait
+
+rm -f "$tmp"
+ip netns del "$testns"
diff --git a/tools/testing/selftests/powerpc/harness.c b/tools/testing/selftests/powerpc/harness.c
index 66d31de..9d7166d 100644
--- a/tools/testing/selftests/powerpc/harness.c
+++ b/tools/testing/selftests/powerpc/harness.c
@@ -85,13 +85,13 @@ int run_test(int (test_function)(void), char *name)
return status;
}
-static void alarm_handler(int signum)
+static void sig_handler(int signum)
{
- /* Jut wake us up from waitpid */
+ /* Just wake us up from waitpid */
}
-static struct sigaction alarm_action = {
- .sa_handler = alarm_handler,
+static struct sigaction sig_action = {
+ .sa_handler = sig_handler,
};
void test_harness_set_timeout(uint64_t time)
@@ -106,8 +106,14 @@ int test_harness(int (test_function)(void), char *name)
test_start(name);
test_set_git_version(GIT_VERSION);
- if (sigaction(SIGALRM, &alarm_action, NULL)) {
- perror("sigaction");
+ if (sigaction(SIGINT, &sig_action, NULL)) {
+ perror("sigaction (sigint)");
+ test_error(name);
+ return 1;
+ }
+
+ if (sigaction(SIGALRM, &sig_action, NULL)) {
+ perror("sigaction (sigalrm)");
test_error(name);
return 1;
}
diff --git a/tools/testing/selftests/powerpc/mm/subpage_prot.c b/tools/testing/selftests/powerpc/mm/subpage_prot.c
index 35ade74..3ae77ba 100644
--- a/tools/testing/selftests/powerpc/mm/subpage_prot.c
+++ b/tools/testing/selftests/powerpc/mm/subpage_prot.c
@@ -135,6 +135,16 @@ static int run_test(void *addr, unsigned long size)
return 0;
}
+static int syscall_available(void)
+{
+ int rc;
+
+ errno = 0;
+ rc = syscall(__NR_subpage_prot, 0, 0, 0);
+
+ return rc == 0 || (errno != ENOENT && errno != ENOSYS);
+}
+
int test_anon(void)
{
unsigned long align;
@@ -145,6 +155,8 @@ int test_anon(void)
void *mallocblock;
unsigned long mallocsize;
+ SKIP_IF(!syscall_available());
+
if (getpagesize() != 0x10000) {
fprintf(stderr, "Kernel page size must be 64K!\n");
return 1;
@@ -180,6 +192,8 @@ int test_file(void)
off_t filesize;
int fd;
+ SKIP_IF(!syscall_available());
+
fd = open(file_name, O_RDWR);
if (fd == -1) {
perror("failed to open file");
diff --git a/tools/testing/selftests/powerpc/tm/tm-tmspr.c b/tools/testing/selftests/powerpc/tm/tm-tmspr.c
index 2bda81c..df1d7d4 100644
--- a/tools/testing/selftests/powerpc/tm/tm-tmspr.c
+++ b/tools/testing/selftests/powerpc/tm/tm-tmspr.c
@@ -98,7 +98,7 @@ void texasr(void *in)
int test_tmspr()
{
- pthread_t thread;
+ pthread_t *thread;
int thread_num;
unsigned long i;
@@ -107,21 +107,28 @@ int test_tmspr()
/* To cause some context switching */
thread_num = 10 * sysconf(_SC_NPROCESSORS_ONLN);
- /* Test TFIAR and TFHAR */
- for (i = 0 ; i < thread_num ; i += 2){
- if (pthread_create(&thread, NULL, (void*)tfiar_tfhar, (void *)i))
- return EXIT_FAILURE;
- }
- if (pthread_join(thread, NULL) != 0)
+ thread = malloc(thread_num * sizeof(pthread_t));
+ if (thread == NULL)
return EXIT_FAILURE;
- /* Test TEXASR */
- for (i = 0 ; i < thread_num ; i++){
- if (pthread_create(&thread, NULL, (void*)texasr, (void *)i))
+ /* Test TFIAR and TFHAR */
+ for (i = 0; i < thread_num; i += 2) {
+ if (pthread_create(&thread[i], NULL, (void *)tfiar_tfhar,
+ (void *)i))
return EXIT_FAILURE;
}
- if (pthread_join(thread, NULL) != 0)
- return EXIT_FAILURE;
+ /* Test TEXASR */
+ for (i = 1; i < thread_num; i += 2) {
+ if (pthread_create(&thread[i], NULL, (void *)texasr, (void *)i))
+ return EXIT_FAILURE;
+ }
+
+ for (i = 0; i < thread_num; i++) {
+ if (pthread_join(thread[i], NULL) != 0)
+ return EXIT_FAILURE;
+ }
+
+ free(thread);
if (passed)
return 0;
diff --git a/tools/testing/selftests/pstore/config b/tools/testing/selftests/pstore/config
index 6a8e5a9..d148f9f 100644
--- a/tools/testing/selftests/pstore/config
+++ b/tools/testing/selftests/pstore/config
@@ -2,3 +2,4 @@
CONFIG_PSTORE=y
CONFIG_PSTORE_PMSG=y
CONFIG_PSTORE_CONSOLE=y
+CONFIG_PSTORE_RAM=m
diff --git a/tools/testing/selftests/pstore/pstore_post_reboot_tests b/tools/testing/selftests/pstore/pstore_post_reboot_tests
index 6ccb154..22f8df1 100755
--- a/tools/testing/selftests/pstore/pstore_post_reboot_tests
+++ b/tools/testing/selftests/pstore/pstore_post_reboot_tests
@@ -7,13 +7,16 @@
#
# Released under the terms of the GPL v2.
+# Kselftest framework requirement - SKIP code is 4.
+ksft_skip=4
+
. ./common_tests
if [ -e $REBOOT_FLAG ]; then
rm $REBOOT_FLAG
else
prlog "pstore_crash_test has not been executed yet. we skip further tests."
- exit 0
+ exit $ksft_skip
fi
prlog -n "Mounting pstore filesystem ... "
diff --git a/tools/testing/selftests/seccomp/seccomp_bpf.c b/tools/testing/selftests/seccomp/seccomp_bpf.c
index f689981..d5be7b5 100644
--- a/tools/testing/selftests/seccomp/seccomp_bpf.c
+++ b/tools/testing/selftests/seccomp/seccomp_bpf.c
@@ -1692,7 +1692,11 @@ TEST_F_SIGNAL(TRACE_syscall, kill_after_ptrace, SIGSYS)
#endif
#ifndef SECCOMP_FILTER_FLAG_TSYNC
-#define SECCOMP_FILTER_FLAG_TSYNC 1
+#define SECCOMP_FILTER_FLAG_TSYNC (1UL << 0)
+#endif
+
+#ifndef SECCOMP_FILTER_FLAG_SPEC_ALLOW
+#define SECCOMP_FILTER_FLAG_SPEC_ALLOW (1UL << 2)
#endif
#ifndef seccomp
@@ -1791,6 +1795,78 @@ TEST(seccomp_syscall_mode_lock)
}
}
+/*
+ * Test detection of known and unknown filter flags. Userspace needs to be able
+ * to check if a filter flag is supported by the current kernel and a good way
+ * of doing that is by attempting to enter filter mode, with the flag bit in
+ * question set, and a NULL pointer for the _args_ parameter. EFAULT indicates
+ * that the flag is valid and EINVAL indicates that the flag is invalid.
+ */
+TEST(detect_seccomp_filter_flags)
+{
+ unsigned int flags[] = { SECCOMP_FILTER_FLAG_TSYNC,
+ SECCOMP_FILTER_FLAG_SPEC_ALLOW };
+ unsigned int flag, all_flags;
+ int i;
+ long ret;
+
+ /* Test detection of known-good filter flags */
+ for (i = 0, all_flags = 0; i < ARRAY_SIZE(flags); i++) {
+ int bits = 0;
+
+ flag = flags[i];
+ /* Make sure the flag is a single bit! */
+ while (flag) {
+ if (flag & 0x1)
+ bits ++;
+ flag >>= 1;
+ }
+ ASSERT_EQ(1, bits);
+ flag = flags[i];
+
+ ret = seccomp(SECCOMP_SET_MODE_FILTER, flag, NULL);
+ ASSERT_NE(ENOSYS, errno) {
+ TH_LOG("Kernel does not support seccomp syscall!");
+ }
+ EXPECT_EQ(-1, ret);
+ EXPECT_EQ(EFAULT, errno) {
+ TH_LOG("Failed to detect that a known-good filter flag (0x%X) is supported!",
+ flag);
+ }
+
+ all_flags |= flag;
+ }
+
+ /* Test detection of all known-good filter flags */
+ ret = seccomp(SECCOMP_SET_MODE_FILTER, all_flags, NULL);
+ EXPECT_EQ(-1, ret);
+ EXPECT_EQ(EFAULT, errno) {
+ TH_LOG("Failed to detect that all known-good filter flags (0x%X) are supported!",
+ all_flags);
+ }
+
+ /* Test detection of an unknown filter flag */
+ flag = -1;
+ ret = seccomp(SECCOMP_SET_MODE_FILTER, flag, NULL);
+ EXPECT_EQ(-1, ret);
+ EXPECT_EQ(EINVAL, errno) {
+ TH_LOG("Failed to detect that an unknown filter flag (0x%X) is unsupported!",
+ flag);
+ }
+
+ /*
+ * Test detection of an unknown filter flag that may simply need to be
+ * added to this test
+ */
+ flag = flags[ARRAY_SIZE(flags) - 1] << 1;
+ ret = seccomp(SECCOMP_SET_MODE_FILTER, flag, NULL);
+ EXPECT_EQ(-1, ret);
+ EXPECT_EQ(EINVAL, errno) {
+ TH_LOG("Failed to detect that an unknown filter flag (0x%X) is unsupported! Does a new flag need to be added to this test?",
+ flag);
+ }
+}
+
TEST(TSYNC_first)
{
struct sock_filter filter[] = {
diff --git a/tools/testing/selftests/static_keys/test_static_keys.sh b/tools/testing/selftests/static_keys/test_static_keys.sh
index 1261e3f..5bba779 100755
--- a/tools/testing/selftests/static_keys/test_static_keys.sh
+++ b/tools/testing/selftests/static_keys/test_static_keys.sh
@@ -1,6 +1,19 @@
#!/bin/sh
# Runs static keys kernel module tests
+# Kselftest framework requirement - SKIP code is 4.
+ksft_skip=4
+
+if ! /sbin/modprobe -q -n test_static_key_base; then
+ echo "static_key: module test_static_key_base is not found [SKIP]"
+ exit $ksft_skip
+fi
+
+if ! /sbin/modprobe -q -n test_static_keys; then
+ echo "static_key: module test_static_keys is not found [SKIP]"
+ exit $ksft_skip
+fi
+
if /sbin/modprobe -q test_static_key_base; then
if /sbin/modprobe -q test_static_keys; then
echo "static_key: ok"
diff --git a/tools/testing/selftests/sync/config b/tools/testing/selftests/sync/config
new file mode 100644
index 0000000..1ab7e81
--- /dev/null
+++ b/tools/testing/selftests/sync/config
@@ -0,0 +1,4 @@
+CONFIG_STAGING=y
+CONFIG_ANDROID=y
+CONFIG_SYNC=y
+CONFIG_SW_SYNC=y
diff --git a/tools/testing/selftests/timers/raw_skew.c b/tools/testing/selftests/timers/raw_skew.c
index 30906bf..0ab937a 100644
--- a/tools/testing/selftests/timers/raw_skew.c
+++ b/tools/testing/selftests/timers/raw_skew.c
@@ -146,6 +146,11 @@ int main(int argv, char **argc)
printf(" %lld.%i(act)", ppm/1000, abs((int)(ppm%1000)));
if (llabs(eppm - ppm) > 1000) {
+ if (tx1.offset || tx2.offset ||
+ tx1.freq != tx2.freq || tx1.tick != tx2.tick) {
+ printf(" [SKIP]\n");
+ return ksft_exit_skip("The clock was adjusted externally. Shutdown NTPd or other time sync daemons\n");
+ }
printf(" [FAILED]\n");
return ksft_exit_fail();
}
diff --git a/tools/testing/selftests/user/test_user_copy.sh b/tools/testing/selftests/user/test_user_copy.sh
index 350107f..0409270 100755
--- a/tools/testing/selftests/user/test_user_copy.sh
+++ b/tools/testing/selftests/user/test_user_copy.sh
@@ -1,6 +1,13 @@
#!/bin/sh
# Runs copy_to/from_user infrastructure using test_user_copy kernel module
+# Kselftest framework requirement - SKIP code is 4.
+ksft_skip=4
+
+if ! /sbin/modprobe -q -n test_user_copy; then
+ echo "user: module test_user_copy is not found [SKIP]"
+ exit $ksft_skip
+fi
if /sbin/modprobe -q test_user_copy; then
/sbin/modprobe -q -r test_user_copy
echo "user_copy: ok"
diff --git a/tools/testing/selftests/x86/sigreturn.c b/tools/testing/selftests/x86/sigreturn.c
index 246145b..4d9dc3f 100644
--- a/tools/testing/selftests/x86/sigreturn.c
+++ b/tools/testing/selftests/x86/sigreturn.c
@@ -610,21 +610,41 @@ static int test_valid_sigreturn(int cs_bits, bool use_16bit_ss, int force_ss)
*/
for (int i = 0; i < NGREG; i++) {
greg_t req = requested_regs[i], res = resulting_regs[i];
+
if (i == REG_TRAPNO || i == REG_IP)
continue; /* don't care */
- if (i == REG_SP) {
- printf("\tSP: %llx -> %llx\n", (unsigned long long)req,
- (unsigned long long)res);
+ if (i == REG_SP) {
/*
- * In many circumstances, the high 32 bits of rsp
- * are zeroed. For example, we could be a real
- * 32-bit program, or we could hit any of a number
- * of poorly-documented IRET or segmented ESP
- * oddities. If this happens, it's okay.
+ * If we were using a 16-bit stack segment, then
+ * the kernel is a bit stuck: IRET only restores
+ * the low 16 bits of ESP/RSP if SS is 16-bit.
+ * The kernel uses a hack to restore bits 31:16,
+ * but that hack doesn't help with bits 63:32.
+ * On Intel CPUs, bits 63:32 end up zeroed, and, on
+ * AMD CPUs, they leak the high bits of the kernel
+ * espfix64 stack pointer. There's very little that
+ * the kernel can do about it.
+ *
+ * Similarly, if we are returning to a 32-bit context,
+ * the CPU will often lose the high 32 bits of RSP.
*/
- if (res == (req & 0xFFFFFFFF))
- continue; /* OK; not expected to work */
+
+ if (res == req)
+ continue;
+
+ if (cs_bits != 64 && ((res ^ req) & 0xFFFFFFFF) == 0) {
+ printf("[NOTE]\tSP: %llx -> %llx\n",
+ (unsigned long long)req,
+ (unsigned long long)res);
+ continue;
+ }
+
+ printf("[FAIL]\tSP mismatch: requested 0x%llx; got 0x%llx\n",
+ (unsigned long long)requested_regs[i],
+ (unsigned long long)resulting_regs[i]);
+ nerrs++;
+ continue;
}
bool ignore_reg = false;
@@ -654,25 +674,18 @@ static int test_valid_sigreturn(int cs_bits, bool use_16bit_ss, int force_ss)
#endif
/* Sanity check on the kernel */
- if (i == REG_CX && requested_regs[i] != resulting_regs[i]) {
+ if (i == REG_CX && req != res) {
printf("[FAIL]\tCX (saved SP) mismatch: requested 0x%llx; got 0x%llx\n",
- (unsigned long long)requested_regs[i],
- (unsigned long long)resulting_regs[i]);
+ (unsigned long long)req,
+ (unsigned long long)res);
nerrs++;
continue;
}
- if (requested_regs[i] != resulting_regs[i] && !ignore_reg) {
- /*
- * SP is particularly interesting here. The
- * usual cause of failures is that we hit the
- * nasty IRET case of returning to a 16-bit SS,
- * in which case bits 16:31 of the *kernel*
- * stack pointer persist in ESP.
- */
+ if (req != res && !ignore_reg) {
printf("[FAIL]\tReg %d mismatch: requested 0x%llx; got 0x%llx\n",
- i, (unsigned long long)requested_regs[i],
- (unsigned long long)resulting_regs[i]);
+ i, (unsigned long long)req,
+ (unsigned long long)res);
nerrs++;
}
}
diff --git a/tools/testing/selftests/zram/zram.sh b/tools/testing/selftests/zram/zram.sh
index 683a292..9399c4a 100755
--- a/tools/testing/selftests/zram/zram.sh
+++ b/tools/testing/selftests/zram/zram.sh
@@ -1,6 +1,9 @@
#!/bin/bash
TCID="zram.sh"
+# Kselftest framework requirement - SKIP code is 4.
+ksft_skip=4
+
. ./zram_lib.sh
run_zram () {
@@ -23,5 +26,5 @@
else
echo "$TCID : No zram.ko module or /dev/zram0 device file not found"
echo "$TCID : CONFIG_ZRAM is not set"
- exit 1
+ exit $ksft_skip
fi
diff --git a/tools/testing/selftests/zram/zram_lib.sh b/tools/testing/selftests/zram/zram_lib.sh
index f6a9c73..9e73a4f 100755
--- a/tools/testing/selftests/zram/zram_lib.sh
+++ b/tools/testing/selftests/zram/zram_lib.sh
@@ -18,6 +18,9 @@
dev_makeswap=-1
dev_mounted=-1
+# Kselftest framework requirement - SKIP code is 4.
+ksft_skip=4
+
trap INT
check_prereqs()
@@ -27,7 +30,7 @@
if [ $uid -ne 0 ]; then
echo $msg must be run as root >&2
- exit 0
+ exit $ksft_skip
fi
}
diff --git a/tools/thermal/tmon/sysfs.c b/tools/thermal/tmon/sysfs.c
index 1c12536..18f5235 100644
--- a/tools/thermal/tmon/sysfs.c
+++ b/tools/thermal/tmon/sysfs.c
@@ -486,6 +486,7 @@ int zone_instance_to_index(int zone_inst)
int update_thermal_data()
{
int i;
+ int next_thermal_record = cur_thermal_record + 1;
char tz_name[256];
static unsigned long samples;
@@ -495,9 +496,9 @@ int update_thermal_data()
}
/* circular buffer for keeping historic data */
- if (cur_thermal_record >= NR_THERMAL_RECORDS)
- cur_thermal_record = 0;
- gettimeofday(&trec[cur_thermal_record].tv, NULL);
+ if (next_thermal_record >= NR_THERMAL_RECORDS)
+ next_thermal_record = 0;
+ gettimeofday(&trec[next_thermal_record].tv, NULL);
if (tmon_log) {
fprintf(tmon_log, "%lu ", ++samples);
fprintf(tmon_log, "%3.1f ", p_param.t_target);
@@ -507,11 +508,12 @@ int update_thermal_data()
snprintf(tz_name, 256, "%s/%s%d", THERMAL_SYSFS, TZONE,
ptdata.tzi[i].instance);
sysfs_get_ulong(tz_name, "temp",
- &trec[cur_thermal_record].temp[i]);
+ &trec[next_thermal_record].temp[i]);
if (tmon_log)
fprintf(tmon_log, "%lu ",
- trec[cur_thermal_record].temp[i]/1000);
+ trec[next_thermal_record].temp[i] / 1000);
}
+ cur_thermal_record = next_thermal_record;
for (i = 0; i < ptdata.nr_cooling_dev; i++) {
char cdev_name[256];
unsigned long val;
diff --git a/tools/thermal/tmon/tmon.c b/tools/thermal/tmon/tmon.c
index 9aa1965..b43138f 100644
--- a/tools/thermal/tmon/tmon.c
+++ b/tools/thermal/tmon/tmon.c
@@ -336,7 +336,6 @@ int main(int argc, char **argv)
show_data_w();
show_cooling_device();
}
- cur_thermal_record++;
time_elapsed += ticktime;
controller_handler(trec[0].temp[target_tz_index] / 1000,
&yk);
diff --git a/tools/usb/ffs-test.c b/tools/usb/ffs-test.c
index 88d5e71..47dfa0b 100644
--- a/tools/usb/ffs-test.c
+++ b/tools/usb/ffs-test.c
@@ -44,12 +44,25 @@
/******************** Little Endian Handling ********************************/
-#define cpu_to_le16(x) htole16(x)
-#define cpu_to_le32(x) htole32(x)
+/*
+ * cpu_to_le16/32 are used when initializing structures, a context where a
+ * function call is not allowed. To solve this, we code cpu_to_le16/32 in a way
+ * that allows them to be used when initializing structures.
+ */
+
+#if __BYTE_ORDER == __LITTLE_ENDIAN
+#define cpu_to_le16(x) (x)
+#define cpu_to_le32(x) (x)
+#else
+#define cpu_to_le16(x) ((((x) >> 8) & 0xffu) | (((x) & 0xffu) << 8))
+#define cpu_to_le32(x) \
+ ((((x) & 0xff000000u) >> 24) | (((x) & 0x00ff0000u) >> 8) | \
+ (((x) & 0x0000ff00u) << 8) | (((x) & 0x000000ffu) << 24))
+#endif
+
#define le32_to_cpu(x) le32toh(x)
#define le16_to_cpu(x) le16toh(x)
-
/******************** Messages and Errors ***********************************/
static const char argv0[] = "ffs-test";
diff --git a/tools/usb/usbip/src/usbip_detach.c b/tools/usb/usbip/src/usbip_detach.c
index 9db9d21..6a8db85 100644
--- a/tools/usb/usbip/src/usbip_detach.c
+++ b/tools/usb/usbip/src/usbip_detach.c
@@ -43,7 +43,7 @@ void usbip_detach_usage(void)
static int detach_port(char *port)
{
- int ret;
+ int ret = 0;
uint8_t portnum;
char path[PATH_MAX+1];
@@ -73,9 +73,12 @@ static int detach_port(char *port)
}
ret = usbip_vhci_detach_device(portnum);
- if (ret < 0)
- return -1;
+ if (ret < 0) {
+ ret = -1;
+ goto call_driver_close;
+ }
+call_driver_close:
usbip_vhci_driver_close();
return ret;
diff --git a/tools/virtio/ringtest/ptr_ring.c b/tools/virtio/ringtest/ptr_ring.c
index 635b07b..b4a2e6a 100644
--- a/tools/virtio/ringtest/ptr_ring.c
+++ b/tools/virtio/ringtest/ptr_ring.c
@@ -15,24 +15,41 @@
#define unlikely(x) (__builtin_expect(!!(x), 0))
#define likely(x) (__builtin_expect(!!(x), 1))
#define ALIGN(x, a) (((x) + (a) - 1) / (a) * (a))
+#define SIZE_MAX (~(size_t)0)
+
typedef pthread_spinlock_t spinlock_t;
typedef int gfp_t;
-static void *kmalloc(unsigned size, gfp_t gfp)
-{
- return memalign(64, size);
-}
+#define __GFP_ZERO 0x1
-static void *kzalloc(unsigned size, gfp_t gfp)
+static void *kmalloc(unsigned size, gfp_t gfp)
{
void *p = memalign(64, size);
if (!p)
return p;
- memset(p, 0, size);
+ if (gfp & __GFP_ZERO)
+ memset(p, 0, size);
return p;
}
+static inline void *kzalloc(unsigned size, gfp_t flags)
+{
+ return kmalloc(size, flags | __GFP_ZERO);
+}
+
+static inline void *kmalloc_array(size_t n, size_t size, gfp_t flags)
+{
+ if (size != 0 && n > SIZE_MAX / size)
+ return NULL;
+ return kmalloc(n * size, flags);
+}
+
+static inline void *kcalloc(size_t n, size_t size, gfp_t flags)
+{
+ return kmalloc_array(n, size, flags | __GFP_ZERO);
+}
+
static void kfree(void *p)
{
if (p)
diff --git a/tools/vm/page-types.c b/tools/vm/page-types.c
index e92903f..6d5bcba 100644
--- a/tools/vm/page-types.c
+++ b/tools/vm/page-types.c
@@ -155,12 +155,6 @@ static const char * const page_flag_names[] = {
};
-static const char * const debugfs_known_mountpoints[] = {
- "/sys/kernel/debug",
- "/debug",
- 0,
-};
-
/*
* data structures
*/
diff --git a/tools/vm/slabinfo.c b/tools/vm/slabinfo.c
index b9d34b37..6975ec4 100644
--- a/tools/vm/slabinfo.c
+++ b/tools/vm/slabinfo.c
@@ -29,8 +29,8 @@ struct slabinfo {
int alias;
int refs;
int aliases, align, cache_dma, cpu_slabs, destroy_by_rcu;
- int hwcache_align, object_size, objs_per_slab;
- int sanity_checks, slab_size, store_user, trace;
+ unsigned int hwcache_align, object_size, objs_per_slab;
+ unsigned int sanity_checks, slab_size, store_user, trace;
int order, poison, reclaim_account, red_zone;
unsigned long partial, objects, slabs, objects_partial, objects_total;
unsigned long alloc_fastpath, alloc_slowpath;
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/virt/kvm/arm/hyp/vgic-v2-sr.c
index 9502124..3d6dbdf 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v2-sr.c
@@ -203,7 +203,7 @@ int __hyp_text __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu)
return -1;
rd = kvm_vcpu_dabt_get_rd(vcpu);
- addr = kern_hyp_va((kern_hyp_va(&kvm_vgic_global_state))->vcpu_base_va);
+ addr = kern_hyp_va(hyp_symbol_addr(kvm_vgic_global_state)->vcpu_base_va);
addr += fault_ipa - vgic->vgic_cpu_base;
if (kvm_vcpu_dabt_iswrite(vcpu)) {
diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
index 539d3f5..80d8888 100644
--- a/virt/kvm/arm/vgic/vgic-init.c
+++ b/virt/kvm/arm/vgic/vgic-init.c
@@ -241,6 +241,10 @@ int vgic_init(struct kvm *kvm)
if (vgic_initialized(kvm))
return 0;
+ /* Are we also in the middle of creating a VCPU? */
+ if (kvm->created_vcpus != atomic_read(&kvm->online_vcpus))
+ return -EBUSY;
+
/* freeze the number of spis */
if (!dist->nr_spis)
dist->nr_spis = VGIC_NR_IRQS_LEGACY - VGIC_NR_PRIVATE_IRQS;
diff --git a/virt/kvm/arm/vgic/vgic-its.c b/virt/kvm/arm/vgic/vgic-its.c
index 31f5625..1ebbf23 100644
--- a/virt/kvm/arm/vgic/vgic-its.c
+++ b/virt/kvm/arm/vgic/vgic-its.c
@@ -208,8 +208,8 @@ static int update_lpi_config(struct kvm *kvm, struct vgic_irq *irq,
u8 prop;
int ret;
- ret = kvm_read_guest(kvm, propbase + irq->intid - GIC_LPI_OFFSET,
- &prop, 1);
+ ret = kvm_read_guest_lock(kvm, propbase + irq->intid - GIC_LPI_OFFSET,
+ &prop, 1);
if (ret)
return ret;
@@ -339,8 +339,9 @@ static int its_sync_lpi_pending_table(struct kvm_vcpu *vcpu)
* this very same byte in the last iteration. Reuse that.
*/
if (byte_offset != last_byte_offset) {
- ret = kvm_read_guest(vcpu->kvm, pendbase + byte_offset,
- &pendmask, 1);
+ ret = kvm_read_guest_lock(vcpu->kvm,
+ pendbase + byte_offset,
+ &pendmask, 1);
if (ret) {
kfree(intids);
return ret;
@@ -628,7 +629,7 @@ static bool vgic_its_check_id(struct vgic_its *its, u64 baser, int id)
return false;
/* Each 1st level entry is represented by a 64-bit value. */
- if (kvm_read_guest(its->dev->kvm,
+ if (kvm_read_guest_lock(its->dev->kvm,
BASER_ADDRESS(baser) + index * sizeof(indirect_ptr),
&indirect_ptr, sizeof(indirect_ptr)))
return false;
@@ -1152,8 +1153,8 @@ static void vgic_its_process_commands(struct kvm *kvm, struct vgic_its *its)
cbaser = CBASER_ADDRESS(its->cbaser);
while (its->cwriter != its->creadr) {
- int ret = kvm_read_guest(kvm, cbaser + its->creadr,
- cmd_buf, ITS_CMD_SIZE);
+ int ret = kvm_read_guest_lock(kvm, cbaser + its->creadr,
+ cmd_buf, ITS_CMD_SIZE);
/*
* If kvm_read_guest() fails, this could be due to the guest
* programming a bogus value in CBASER or something else going
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index f132006..c792471 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -337,11 +337,6 @@ int vgic_v3_probe(const struct gic_kvm_info *info)
pr_warn("GICV physical address 0x%llx not page aligned\n",
(unsigned long long)info->vcpu.start);
kvm_vgic_global_state.vcpu_base = 0;
- } else if (!PAGE_ALIGNED(resource_size(&info->vcpu))) {
- pr_warn("GICV size 0x%llx not a multiple of page size 0x%lx\n",
- (unsigned long long)resource_size(&info->vcpu),
- PAGE_SIZE);
- kvm_vgic_global_state.vcpu_base = 0;
} else {
kvm_vgic_global_state.vcpu_base = info->vcpu.start;
kvm_vgic_global_state.can_emulate_gicv2 = true;
diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c
index 4d28a9d..16e17ea 100644
--- a/virt/kvm/eventfd.c
+++ b/virt/kvm/eventfd.c
@@ -119,8 +119,12 @@ irqfd_shutdown(struct work_struct *work)
{
struct kvm_kernel_irqfd *irqfd =
container_of(work, struct kvm_kernel_irqfd, shutdown);
+ struct kvm *kvm = irqfd->kvm;
u64 cnt;
+ /* Make sure irqfd has been initalized in assign path. */
+ synchronize_srcu(&kvm->irq_srcu);
+
/*
* Synchronize with the wait-queue and unhook ourselves to prevent
* further events.
@@ -387,7 +391,6 @@ kvm_irqfd_assign(struct kvm *kvm, struct kvm_irqfd *args)
idx = srcu_read_lock(&kvm->irq_srcu);
irqfd_update(kvm, irqfd);
- srcu_read_unlock(&kvm->irq_srcu, idx);
list_add_tail(&irqfd->list, &kvm->irqfds.items);
@@ -402,11 +405,6 @@ kvm_irqfd_assign(struct kvm *kvm, struct kvm_irqfd *args)
if (events & POLLIN)
schedule_work(&irqfd->inject);
- /*
- * do not drop the file until the irqfd is fully initialized, otherwise
- * we might race against the POLLHUP
- */
- fdput(f);
#ifdef CONFIG_HAVE_KVM_IRQ_BYPASS
if (kvm_arch_has_irq_bypass()) {
irqfd->consumer.token = (void *)irqfd->eventfd;
@@ -421,6 +419,13 @@ kvm_irqfd_assign(struct kvm *kvm, struct kvm_irqfd *args)
}
#endif
+ srcu_read_unlock(&kvm->irq_srcu, idx);
+
+ /*
+ * do not drop the file until the irqfd is fully initialized, otherwise
+ * we might race against the POLLHUP
+ */
+ fdput(f);
return 0;
fail:
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index eaae725..4f2a2df 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -1466,7 +1466,8 @@ static bool vma_is_valid(struct vm_area_struct *vma, bool write_fault)
static int hva_to_pfn_remapped(struct vm_area_struct *vma,
unsigned long addr, bool *async,
- bool write_fault, kvm_pfn_t *p_pfn)
+ bool write_fault, bool *writable,
+ kvm_pfn_t *p_pfn)
{
unsigned long pfn;
int r;
@@ -1492,6 +1493,8 @@ static int hva_to_pfn_remapped(struct vm_area_struct *vma,
}
+ if (writable)
+ *writable = true;
/*
* Get a reference here because callers of *hva_to_pfn* and
@@ -1557,7 +1560,7 @@ static kvm_pfn_t hva_to_pfn(unsigned long addr, bool atomic, bool *async,
if (vma == NULL)
pfn = KVM_PFN_ERR_FAULT;
else if (vma->vm_flags & (VM_IO | VM_PFNMAP)) {
- r = hva_to_pfn_remapped(vma, addr, async, write_fault, &pfn);
+ r = hva_to_pfn_remapped(vma, addr, async, write_fault, writable, &pfn);
if (r == -EAGAIN)
goto retry;
if (r < 0)
